Psychophysiology 48(1)

Psychophysiology, 48 (2011), 1. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophysiolog...

Author: Robert F. Simons

44 downloads 1526 Views 10MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Psychophysiology, 48 (2011), 1. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophysiological Research DOI: 10.1111/j.1469-8986.2010.01161.x

ANNOUNCEMENT

Full-length manuscripts should not exceed 30 pages of text (including references). It is suggested that the introductory section be limited to approximately 1000 words and the discussion to 2000 words. To qualify for consideration as a Brief Report, the entire manuscript should not exceed 3000 words, including references, tables, figures, figure legends, and abstract. Each table and each figure corresponds to 250 words. Unless otherwise specified, the guideline for preparation of manuscripts is the Publication Manual of the American Psychological Association (6th edition).

Starting with this January 2011 issue, Psychophysiology will increase its publication frequency from 6 to 12 issues per year. Over the past several years, submissions to the journal have more than doubled, resulting in an abundance of high quality manuscripts. The Society for Psychophysiological Research, which publishes the journal through Wiley-Blackwell, has decided to increase the number of issues and total pages to accommodate the high number of quality submissions to the journal, providing authors with more opportunities to publish and shorter publication lag times, and scholars with access to more high quality papers in a growing research area. At the same time, the Journal has made the following changes to the Author Guidelines regarding the length of submitted manuscripts and the preparation style:

1

Psychophysiology, 48 (2011), 2–3. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophysiological Research DOI: 10.1111/j.1469-8986.2010.01142.x

In Memoriam John A. Stern (1925–2010) JOHN W. ROHRBAUGH Washington University School of Medicine

John Alexander Stern died on April 3, 2010, several months after being diagnosed with pancreatic cancer, in his home and with his family gathered. Upon his death, the Society for Psychophysiological Research lost a founding member, one who remained as an active contributor to the Society throughout his long and distinguished career. John was the Society’s seventh president, hosted its annual meeting in Saint Louis in 1971, served on a number of committees, and was a frequent contributor and reviewer for the journal Psychophysiology. In 1993 he was awarded the Society’s highest honor, the award for Distinguished Contributions to Psychophysiology. This service to the Society was part of his larger role in the development of psychophysiology as an integrative discipline. John was born January 4, 1925, in Montabour, Germany. In advance of the repression of Jews in Germany (combined with his self-described ‘‘youthful outspokenness’’), he was sent at age 9 to live with relatives in Holland, then on to New York in early 1936. Upon graduation from Peter Stuyvesant High School in 1943, he was drafted into the Army of the United States in which he saw heavy combat in the Mediterranean and European theaters. He was wounded on several occasions and was awarded three Bronze Stars. Although he was reticent to talk about his military service, it clearly marked his temperament and professional career. He related how his wartime experiences fostered his interest in psychologyFthe impressions left on him by the extremes of behavior seen under combat conditions and the seeming impossibility of predicting them on the basis of comportment during less challenging times. And, somewhat paradoxically, they left him with a certain kind of optimism, that nothing in comparison would be beyond him going forward. After returning from the war, John enrolled under GI benefits at Hunter College in New York, where he received his B.A. degree in 1949. This was followed by graduate studies in psychology at the University of Illinois in Urbana–Champaign, leading to a Ph.D. in 1953. It was there that he met his wife Carolyn, who was working in the University’s Student Counseling Center and who was to remain his partner through life. He is survived by Carolyn, a twin sister Helen Silber, daughters Julie Stern and Nancy Mozier, son John Stern, and their spouses, and five grandchildren. John spent his entire career at Washington University, first in the School of Medicine, where he headed the Division of Medical Psychology in the Department of Psychiatry from 1961 to 1969, after which he moved to the Department of Psychology, where he served as chair from 1987–1996. His tenure as chair was marked by a restoration of intradepartmental cooperation and rise in productivity. He oversaw the design and construction of a new psychology building, which laid the groundwork for substantial

subsequent expansion of the department. He retained an office and laboratory throughout the remainder of his career, becoming Professor Emeritus in 2000. He helped to form the company BioBehavioral Analysis Systems, as a vehicle for supporting his ongoing research. In the very first issue of this publication, John contributed a brief piece that aimed to define the domain of appropriate content for the new journal. While much discussed over the years, the thoughts expressed there have served in a much broader context to establish an identity for the discipline and to foster a research agenda. A read through his professional bibliography offers a tour of many of the themes that shaped psychophysiology as a scientific discipline and that persist as core issues. In an era of ever increasing specialization and focus, it is remarkable to reflect on a time that accommodated such breadth. 2

In Memoriam John’s early studies focused on the effects of electroconvulsive shock in rats, but with a clear emphasis at that seminal stage on the relationships among physiological and behavioral effects. It is to the benefit of human psychophysiology that John was forced to move away from these studies because he developed allergies to his rat subjects and so shifted to human research. Along the way, there were studies relating to the psychiatric issues of mental disorders, stress, and acute and chronic effects of drugs and common substances, but ranging widely to include studies of acupuncture and hypnosis, development and aging, hemispheric specialization, sleep, and sexual behavior. This work served to crystallize his interests in the integrative aspects of physiological responses in multiple response systems and the degree to which they supported differentiation among emotions, cognitions, and motivational states. His focus on the basic processes of conditioning, orienting, and habituation evolved to a broader perspective encompassing complex behaviors, including reading and other processes involved in the acquisition and processing of information. The final three decades of his career emphasized oculomotor activity. He was fond of saying that this interest evolved from a series of electroencephalogram (EEG) studies in which he became concerned with the then customary procedure of requiring subjects to suppress eye movements and blinks (in the interest of minimizing the attendant artifact). John felt that this risked artificially truncating the associated mental processes and that the ocular ‘‘artifact’’ was of interest in its own right. He went on to demonstrate in a landmark series of studies and reviews that even the simple eyeblink was not randomly distributed in time, but could be used to infer aspects regarding the timing and quality of cognitive processes. Late in his career he increasingly turned to using oculomotor measures in applied settings, to study problems of fatigue, situational awareness, air traffic control, driving, piloting, and the detection of deception. Even as a graduate student, he had carried aloft a primitive forensic polygraph to record physiological responses in pilots during simulated emergencies. He was instrumental in 1993 in the founding of the special interest group, Psychophysiology in Ergonomics, to foster this type of research. Throughout, his career was marked by a level of technical sophisticationFa ‘‘gadgeteering’’ that he admired in his pre-

3 decessors. His publications include an early description of a new stabilometer, and he readily adopted then novel methods for sensing skin temperature and blood volume. He was among the pioneers in applying computerized methods to the analysis of the EEG and eye movements, capitalizing on the availability of the LINC computer at Washington University (as a principal development site). He enthusiastically adopted camera-based hardware and software methods for recording oculomotor activity and interacted with major commercial developers to improve their products. In recognition that head movements are an essential aspect of the gaze control system, he developed a simple laboratory apparatus that allowed him to record head movements and enfold them into his measures. The dry facts of his curriculum vitae, impressive as they are, fail to capture the lively essence of his spirit as a friend and colleague. He was extremely generousFcertainly to me but also to a wide net of associates around the world. Over his career, he had intense and lengthy collaborations with Czech, Russian, Japanese, and Chinese investigators, as well as those in the United States, often hosting them in Saint Louis for extended stays. He was a committed teacher and mentor; one of his legacies is the John Stern Fund for Undergraduate Research at Washington University, which he established with a substantial donation. He was always optimistic, always supportive, always available. He was also committed to the arts and a number of social causes in the community beyond Washington University. I had the wonderful privilege of working closely with John for the past 20 years, often on a daily basis. If there was any disengagement as he aged, any dimming of the force of his intellect, I never saw it. He remained a productive scientist, continued to publish, secured research funding, and served as reviewer and consultant. His absences from the laboratory grew a bit longer and a bit more frequent because he followed his love of travel with Carolyn, but there was never any real winding down. Upon the passing of Chester Darrow, John eulogized in this journal many years ago (1967) that, ‘‘though he may have been grandfatherly in age at the time of his death he was always young in ideas and actively involved in research.’’ I think that John would see this, with pride, as a fitting tribute to him as well.


Auditory processing that leads to conscious perception: A unique window to central auditory processing opened by the mismatch negativity and related responses

RISTO NAÄ¨TA¨NEN,a,b,c TEIJA KUJALA,c and ISTVAŃ WINKLER,c,d,e a

Department of Psychology, University of Tartu, Tartu, Estonia Centre of Integrative Neuroscience (CFIN), University of Aarhus, Aarhus, Denmark Cognitive Brain Research Unit, Institute of Behavioural Sciences, University of Helsinki, Helsinki, Finland d Department of Experimental Psychology, Institute for Psychology, Hungarian Academy of Sciences, Budapest, Hungary e Institute of Psychology, University of Szeged, Hungary b c

Abstract In this review, we will present a model of brain events leading to conscious perception in audition. This represents an updated version of Naä¨ta¨nen’s previous model of automatic and attentive central auditory processing. This revised model is mainly based on the mismatch negativity (MMN) and N1 indices of automatic processing, the processing negativity (PN) index of selective attention, and their magnetoencephalographic (MEG) and functional magnetic resonance imaging (fMRI) equivalents. Special attention is paid to determining the neural processes that might underlie conscious perception and the borderline between automatic and attention-dependent processes in audition. Descriptors: Central auditory processing, Event-related potential (ERP), Mismatch negativity (MMN), N1

will also be used in specifying this model.Before introducing the model, it is necessary to examine the functional significance and separability of these overlapping brain responses, because these are central to the model. Moreover, this discussion will help in interpreting the roles that the processes that generate these responses play in the model.

One of the most exciting issues of modern cognitive neuroscience is the division to, and borderline between, brain processes that underly or do not underly conscious experience. For example, how much of auditory processing occurs outside of our attention and conscious experience and, further, what are the brain mechanisms that determine whether conscious perception occurs or not? These questions will be the scope of the present review which will present a model of preconscious and conscious processes in audition that aims at determining the functional borderline between the two processing modes. This model is an extension of Naä¨ta¨nen’s (1990) model on attention and automaticity in auditory processing. The principal tool that has developed this model is the mismatch negativity (MMN) (Naä¨ta¨nen, Gaillard, & Ma¨ntysalo, 1978). The (auditory) MMN is a fronto–centrally negative event-related potential (ERP) component that is elicited by sounds that violate the automatic predictions of the central auditory system. The MMN and its magnetoencephalographic (MEG) equivalent, the MMNm (Hari et al., 1984) provide a unique window to preconscious central auditory processing. Results obtained for the N1 (Naä¨ta¨nen & Picton, 1987) and the processing negativity (PN) described by Naä¨ta¨nen et al. (1978)

The mismatch negativity (MMN). The MMN was initially isolated from the ‘‘N2’’ (Ford, Kopell, et al., 1976a, 1976b; Simson, Vaughan, & Ritter, 1976, 1977; Squires, Squires, & Hillyard, 1975; Squires, Wickens, Squires, & Donchin, 1976;) and the ‘‘N2-P3a’’ (Snyder & Hillyard, 1976) wave complexes, that are typically elicited in auditory oddball sequences, by Naä¨ta¨nen et al. (1978; see also Naä¨ta¨nen, 1975) through the use of deviantstandard difference waveforms. In contrast, the N1, an obligatory fronto–centrally negative-polarity response that peaks at about 100 ms from sound onset, manifests as a separate ERP peak. The MMN and its magnetic counterpart MMNm usually become clearly visible only through a subtraction procedure, in which the ERP response to some control stimulus, such as the frequent stimulus (‘‘standard’’), is subtracted from the response elicited by the infrequent stimulus ‘‘deviant.’’ For a review of the proper control for deriving the MMN, see Kujala, Tervaniemi, and Schro¨ger (2007). Further, Naä¨ta¨nen, Simpson, and Loveless (1982) showed that, after the MMN (‘‘N2a’’) is removed from the N2 wave complex by subtraction, the remaining waveform can be identified as the ‘‘N2b’’ response. The N2b response has a somewhat posterior topography compared to the N1 and also to

This research was supported by The Philips Nordic Prize for 2007 for Achieved and Continued Research Work on Neurodevelopmental Disorders, the Academy of Finland (projects 1122745 and 1128840), and the European Commission (ICT-FP7-231168). Address correspondence to: Risto Naä¨ta¨nen, Cognitive Brain Research Unit, Institute of Behavioural Sciences, P. O. Box 9 (Siltavuorenpenger 1B), 00014 University of Helsinki, Finland. E-mail: [email protected] 4

Auditory processing that leads to conscious perception

5

the MMN and, together with the accompanying P3a, forms the N2-P3a or the N2b-P3a complex. The N2b-P3a complex is elicited by deviants when the stimulus sequence is attended or when there is an ‘‘attention leak’’ to the to-be-ignored channel, which has been reviewed by Naä¨ta¨nen and Gaillard (1983). For a

schematic illustration of the different ERP components obtained in the oddball paradigm, see Figure 1. The MMN was initially interpreted as being generated by an automatic memory-based change-detection mechanism that operates independently of the listener’s attention or behavioral

IGNORE CONDITION Components

ERP wave form standard deviant

N1-P2 MMN

MMN

Fz

MMN

N1 Cz P2

Pz − −100

0 stim

400

200

−100

[ms] +

0 stim

400

200

[ms]

ATTEND CONDITION N1-P2 MMN N2b-P3a P3b

Components MMN

ERP wave form

standard deviant

“N2”

“frontal slow wave” Fz

“N1”

N2b Cz P165

“P2”

P3a

Pz

−100

0 stim

200

400

“parietal slow wave”

−

P3 or P3b

“P3” or “P3b” −100

[ms] +

“P3a”

0 stim

200

400

[ms]

Figure 1. Top left: Different ERP components (contributions to the scalp-recorded ERP of separable generator processes; Naä¨ta¨nen & Picton, 1987) in the oddball paradigm are schematically illustrated, separately for standards and deviants, for midline electrodes Fz, Cz, and Pz in the IGNORE condition. These components are elicited even in the absence of attention and also when attention is strictly controlled, which reflects fully automatic processing in audition. Note that the N1 and P2 components might differ between deviants and standards, depending on the nature and probability of the stimuli used. Top right: ERP waveforms recorded from these midline electrodes composed of the components illustrated on the left are shown. Note also the much more frontal midline scalp topography of the MMN relative to that of the N1. Bottom left: The component structure of the ERPs recorded on the same midline electrodes is illustrated in the ATTEND condition. The most important difference to the IGNORE condition is the addition of the N2b-P3a complex, which is often preceded by a P165 (Goodin, Squires, Henderson, & Starr, 1978), and the slow frontal negative and parietal positive waves. Depending on instructions, attention may also enhance the N1 and MMN amplitudes. Bottom right: ERP waveforms recorded from these midline electrodes composed of the components illustrated on the left are shown. Source: Naä¨ta¨nen (1986).

6 goals (Naä¨ta¨nen et al., 1978; Naä¨ta¨nen & Michie, 1979; Naä¨ta¨nen, 1975), even though some studies (Woldorff, Hackley, & Hillyard, 1991; Woldorff, Hillyard, Gallen, Hampson & Bloom, 1998) showed that under some conditions, the MMN amplitude can be attenuated by strongly focusing attention to some other stimulus sequence. For two recent reviews of the attention-MMN relationship, see Haroush, Hochstein, and Deouell (2010) and Sussman (2007). Further, according to this prevailing interpretation (Naä¨ta¨nen, Paavilainen, Rinne, & Alho, 2007; Winkler, 2007), the MMN is based on a memory trace that encodes the repetitive aspects (termed regularity) of the most recent auditory stimulation. The MMN is elicited when the auditory input does not match the actual or predicted sensory information encoded in this trace (Grimm & Schro¨ger, 2007; Tervaniemi, Maury, & Naä¨ta¨nen, 1994a). The most recent interpretation of the MMN emphasizes the active role of the memory trace assumed to be used in MMN generation. The MMN is elicited by a mismatch between the auditory input and the predictions formed on the basis of the trends or rules that are automatically detected in the recent auditory stimulation (Naä¨ta¨nen, 1992; Naä¨ta¨nen & Winkler, 1999; Winkler, Karmos, & Naä¨ta¨nen, 1996; Winkler, Denham, & Nelken, 2009a; Winkler, 2007). The biological significance of the MMN-generation process might be the automatic switching of the organism’s attention to auditory change. This interpretation is supported by transient deteriorations in primary-task performance that accompany MMN elicitation by changes in irrelevant auditory background stimulation (Escera, Corral, & Yago, 2002; Escera, Yago, Corral, Corbera, & Nun˜ez, 2003; Schro¨ger, 1996, 1997; Yago, Escera, Alho, & Giard, 2001). It is possibly the frontal MMN subcomponent (Deouell, 2007; Giard, Perrin, Pernier, & Bouchet, 1990; Gomot, Giard, Roux, Barthelemy, & Bruneau, 2000; Jaä¨skelaïnen, Alho, Escera, Winkler, Sillanaukee, & Naä¨ta¨nen, 1996b; Jaä¨skelaïnen, Pekkonen, Hirvonen, Sillanaukee, & Naä¨ta¨nen, 1996a; Jaä¨skelaïnen, Varonen, Naä¨ta¨nen, & Pekkonen, 1999; Molholm, Martinez, Ritter, Javitt, & Foxe, 2005; Rinne, Alho, Ilmoniemi, Virtanen, & Naä¨ta¨nen, 2000; Tse and Penney, 2008), with an onset that follows that of the auditory-cortical MMN subcomponent by 10–20 ms (Rinne et al., 2000; Tse & Penney, 2008), that is generated by the attention-call process (O¨hman, 1979) to auditory deviance, as suggested by Giard et al. (1990). This is supported by, among other things, the fact that one of the important frontal-lobe functions controls the direction of attention (Fuster, 1989; Knight, 1991; Stuss & Benson, 1986). Furthermore, the involvement of the frontal cortex in MMN generation is also supported by results that show that lesions of dorsolateral prefrontal cortex attenuate the MMN amplitude (Alain, Woods, & Knight, 1998; Alho, Woods, Algazi, Knight, Naä¨ta¨nen, et al., 1994). Jaä¨skelaïnen, Alho, et al. (1996), Jaä¨skelaïnen, Pekkonen, et al. (1996), and Jaä¨skelaïnen et al. (1999) also demonstrated the role of the generator process of the frontal MMN component in attention switching. They found that even a moderate dose of alcohol selectively eliminated this frontal component, which leaves the auditory-cortex component intact and, simultaneously, abolishes the distracting effect of noise on the hit rate in the primary task that was observed in the absence of alcohol. Hence, ethanol blocks the route of auditory distraction to the involuntary attention-switching system reflected by the frontal MMN component. Further evidence implicating the role of the frontal MMN subcomponent in attention switching was pro-

R. Naä¨ta¨nen et al. vided by data obtained from closed head injury patients. These show an association between a pathologically strong frontal MMN-generator process and a pathologically sensitized involuntary attention switching (Kaipio et al., 2000). However, the auditory-cortical component was unaffected (Kaipio et al., 2000). The memory trace that encodes sensory information of the preceding stimuli assumed to be involved in MMN generation usually lasts for a few seconds (Bo¨ttscher-Gandor & Ullsperger, et al., 1992; Cheour et al., 2002; Cooper, Todd, McGill, & Michie, 2006; Glass, Sachse, & von Suchodoletz, 2008a, 2008b; Gomes et al., 1999; Grau, Escera, Yago, & Polo, 1998; Pekkonen et al., 1996; Ritter, Deacon, Gomes, Javitt, & Vaughan, 1995; Sams, Hari, Rif, & Knuutila, 1993). Thereafter no MMN can be elicited, unless the trace is reactivated by a ‘‘reminder’’ stimulus (Winkler & Cowan, 2005). Very importantly, no MMN can be elicited before this trace has been developed, that is, before the regular aspects of the auditory input have been extracted from the sound sequence (Bendixen, Roeber, & Schro¨ger, 2007; Bendixen & Schro¨ger, 2008; Cowan, Winkler, Teder, & Naä¨ta¨nen, 1993; Sams et al., 1985). Further, deviance based on any feature difference or combination of feature differences, that the listener is able to discriminate elicits the MMN (Deacon, Nousak, Pilotti, Ritter, & Yang, 1998; Naä¨ta¨nen & Alho, 1995, 1997). This suggests that the memory trace in question encodes the results of the full analysis of the acoustic features, including their integration into a unitary sensory-memory representation. In contrast, the refractoriness patterns of similar duration that account for the N1 adaptation effects probably encode acoustic features separately, and thus serve as buffers to the sensory data provided by the different feature detectors as a necessary prerequisite for auditory feature integration (Naä¨ta¨nen & Winkler, 1999).

Separability of N1 and MMN As to whether the MMN and the N1 are separable has been discussed since the discovery of the MMN. Recently, within the framework of this debate, it has been suggested (Jaä¨skelaïnen et al., 2004; Jaä¨skelaïnen, Ahveninen, Belliveau, Raij, & Sams, 2007; May & Tiitinen, 2009) that the deviant-standard difference wave can be fully explained by the N1 difference between deviants and standards. In contrast, new computational modeling results clearly separate N1- and MMN-related neural activity within the deviant-minus-standard difference wave (Friston & Kiebel, 2009; Garrido et al., 2008; Garrido, Kilner, Kiebel, & Friston, 2009a). The phenomenon of separability raises two questions: (1) Can the observable deviant-minus-control subtraction waveform be explained by differences in the N1 components elicited by the two stimulus events? (2) Does one need to assume the existence of a memory trace to account for the MMN results obtained during the past ca. 30 years? In the following, we shall show that the N1 and MMN ERP responses can be separated. Further, that the two discrete responses reflect different types of memory traces, both of which are important for understanding preconscious and conscious central auditory processing in the human brain. These will be described in the model we present. Furthermore, these processes can be separated from voluntary (conscious) operations on auditory information, as shown by a third important ERP response, the PN. We shall start by reviewing ERP studies that indicate that the MMN can be observed under conditions in which there can be no systematic N1 differences between deviants and standards. This is the case when deviants differ from stan-

Auditory processing that leads to conscious perception dards in terms of higher-order categories, or when deviants violate higher-order sequential contingency rules. In the next section, we will review evidence that shows that MMN is elicited or enhanced with no systematic acoustic difference. For instance, this occurs when speech-sound deviants are presented to listeners who speak or do not speak the language involved, or when deviant sounds violate linguistic or musicrelated rules. Language-specific MMNs. Several studies compared MMNs elicited by acoustically identical speech stimuli between native speakers of a language and control subjects who did not speak that language (Cheour et al., 1998; Dehaene-Lambertz, 1997; Dehaene-Lambertz, Dupoux, & Gout, 2000; Naä¨ta¨nen et al., 1997; Pulvermu¨ller et al., 2001; Sharma & Dorman, 2000; Winkler et al., 1999). The MMN difference obtained between the two groups cannot be explained by acoustic N1-related factors that were the same for the two groups. For instance, when Pulvermu¨ller et al. (2001) instructed Finnish subjects to ignore sounds and to watch a silent movie, they found that the MMNm to the same spoken Finnish syllable as a deviant stimulus was larger in amplitude when it ended a Finnish word than when it ended a pseudoword. In contrast, this effect did not occur in foreign participants who understood no Finnish. Moreover, the major intracranial source of this word-related MMNm was located in the left superior temporal lobe and it was clearly separable from the N1m locus, which demonstrates an MMN (MMNm) that could not be derived from the N1 response. For further MMNm data supporting this conclusion, see Shestakova et al. (2002) who found a left-hemispheric vowel-category MMNm with 150 randomized, acoustically varying exemplars in each vowel category. MMN to syntactic and semantic violations. The N1-independent generation of the MMN is also shown by studies that demonstrate the automatic processing of grammar. For instance, in Pulvermu¨ller and Shtyrov’s (2003) study that used grammatical and ungrammatical items as deviant stimuli, the MMNm was enhanced in amplitude for grammatical violations as compared with that elicited by grammatically correct deviants. This MMNm, with its main source in the left frontal cortex, indicated that the MMN mechanism was engaged when these grammar effects were elicited. The authors related this syntactic MMNm to the differential activation of neuronal memory traces for grammatical word sequences called ‘‘sequence detectors’’ (Bonte, Mitterer, Zellagui, Poelmans, & Blomert, 2005; Mitterer & Blomert, 2003; Pulvermu¨ller & Shtyrov, 2003). Subsequent studies confirmed and extended this initial finding to different kinds of syntactic and even to semantic violations (Gunter, Friederici, & Hahne, 1999; Hasting, Kotz, & Friederici, 2007; Menning et al., 2005; Pulvermu¨ller & Assadollahi, 2007; Shtyrov, Pulvermu¨ller, Naä¨ta¨nen, & Ilmoniemi, 2003; for reviews, see Pulvermu¨ller, 2001; Pulvermu¨ller & Knoblauch, 2009; Pulvermu¨ller & Shtyrov, 2006; Pulvermu¨ller, Shtyrov, & Hauk, 2009). See also the ‘‘early left anterior negativity’’ (ELAN) described by Friederici and her colleagues (Eckstein & Friederici, 2006; Friederici, 1995, 2002, 2004; Friederici et al. 1993, 1996, 2004; Rossi et al., 2006), which was elicited by syntactic violations as early as at 100–150 ms from the violation onset and which was not affected by attentional factors (Hahne & Friederici, 1999), hence closely resembling the MMNs to syntactic violations reviewed above.

7 In addition, evidence converging with results from language studies was obtained in the research on the automatic processing of musical syntax (Koelsch, Gunter, Schro¨ger, & Friederici, 2003; Koelsch, Grossman, et al., 2003; Leino et al., 2007; Loui et al., 2005). In these studies, chords with an irregular harmonic function that violated the rules of the Western music, presented within sequences of in-key chords, elicited the ‘‘early right anterior negativity’’ (ERAN). This, in turn, has been denoted as the ‘‘music-syntactic MMN’’ (Koelsch, Gunter, et al., 2003; Koelsch, Grossman, et al., 2003; Koelsch & Siebel, 2005; Mu¨nte, Altenmu¨ller, & Ja¨ncke, 2002; Tervaniemi & Brattico, 2004). In many further studies, the MMN was elicited by violating some musical regularity, while acoustic deviance, and its related N1 response difference were controlled. For example, in a study by Tervaniemi, Rytko¨nen, Schro¨ger, Ilmoniemi, and Naä¨ta¨nen (2001), subjects were presented with standard stimuli that consisted of melodic patterns that randomly occurred at very different frequency levels, which simulates a melody transposed to different keys. Nevertheless, occasional slight contour changes in patterns that widely varied in frequency also elicited the MMN but only in ‘‘musical’’ subjects. This finding was subsequently confirmed and extended to different types of musical material (Fujioka, Trainor, Ross, Kakigi, & Pantev, 2004; Tervaniemi, Castaneda, Knoll, & Uther, 2006; Trainor, McDonald, & Alain, 2002) and was even observed in newborn babies (Stefanics et al., 2009). In addition, violating a rhythm elicits an earlier MMN response in adults when the rhythm is violated at a more salient position of the metric hierarchy even when the acoustic deviation is equalized (Ladinig, Honing, Ha´den, & Winkler, 2009). In newborn infants, rhythmic violations occurring at salient metric positions elicited the MMN, whereas violations at non–salient positions did not (Winkler, Ha´den, Ladining, Sziller, & Honing, 2009b). MMN to violations of complex sequential stimulus-contingency rules. In this section, we review evidence that shows that MMN is elicited by violating complex sequential rules when no acoustic change is associated with the deviance. In the first of these studies, Paavilainen, Araja¨rvi, and Takegata (2007) presented their subjects, instructed to ignore sounds, with sounds that varied in two dimensions, duration and frequency, with stimuli being: short (50 ms) or long (150 ms), and low (1000 Hz) or high (1500 Hz). All combinations (short-low, short-high, long-low, longhigh) were equiprobably presented along with a silent inter-stimulus interval (ISI) of 300 ms. The stimulus sequences were constructed so that the duration of each stimulus, which was randomly either short or long, predicted the frequency of the next stimulus so that: (1) if the present stimulus is short in duration, then the subsequent stimulus will be low in frequency; and (2) if the present stimulus is long in duration, then the subsequent stimulus will be high in frequency. Occasional deviant events broke these rules. For example, a high-pitched stimulus following a short stimulus. In this design, all the four different stimulus combinations used could be perceived as either a standard or as a deviant event, depending on the duration of the preceding stimulus. Only the deviant events elicited the MMN. This MMN reversed its polarity at the mastoids, which suggested a source in the auditory cortex. Corroborating results were obtained from studies that used even more complex stimulus-sequence rules (Bendixen, Roeber, & Schro¨ger, 2007; Bendixen, Prinz, Horva´th, Trujillo-Barreto, & Schro¨ger, 2008; Bendixen & Schro¨ger, 2008; Schro¨ger, Bendixen, Trujillo-Barreto, & Roeber, 2007).

8 In the subsequent attend condition described by Paavilainen et al. (2007), subjects who had received no prior information regarding the rules used in constructing the sound sequences were asked to press a button upon hearing any sound they judged to be ‘‘strange’’ or ‘‘deviant.’’ Although they could detect only about 15% of the deviant events, and none of them could verbally express the rules in the later interviews, the MMN was nevertheless elicited. Hence, these results suggest that the neural mechanism that models the auditory environment may automatically learn the co-variation between the features of the successive events and make predictions of the properties of the forthcoming stimuli. If the predictions are not fulfilled, then the MMN is generated. Very recent additional MMN evidence for the predictive nature of the central auditory system has been reported by Bendixen, Schro¨ger, and Winkler (2009) and Todd, Myers, Pirillo, and Drysdale (2010). Furthermore, Sculthorpe, Ouellet, and Campbell (2009) found that sound patterns are extracted from acoustically varying stimuli and their violations detected even in REM sleep. Consequently, the information extracted by the sensorymemory mechanisms often is in an implicit form that is not directly available to conscious processes and difficult to express verbally, which was also confirmed by van Zuijen, Simoens, Paavilainen, Naä¨ta¨nen, and Tervaniemi (2006). Hence these results are consistent with the framework originally outlined by Winkler, Karmos, and Naä¨ta¨nen (1996), according to which the main function of the MMN process is to adjust a neural model to the various regularities of the auditory environment. This enables the central auditory system to manage a large part of its subsequent input automatically, i.e., without requiring the limited resources of the controlled-processing system. (For recent reviews, see Winkler, 2007, and Winkler, Denham, & Nelken, 2009a; for a very recent review of automatic sensory cognition in audition, see Naä¨ta¨nen et al., 2010.) The MMN-N1 generator loci differences in humans. Evidence for the separability of the MMN and the N1 is also provided by localization data that suggest that although they are adjacent to one another the N1 and MMN generator loci are clearly separable from one another. A large number of studies (Alho et al., 1993, 1998; Cse´pe, Pantev, Hoke, Hampson, & Ross, 1992; Korzyukov et al., 1999; Kropotov et al., 1995; Leva¨nen, Hari, McEvoy, & Sams, 1993; Leva¨nen, Ahonen, Hari, McEvoy, & Sams, 1996; Rosburg, 2003; Rosburg et al., 2004; Sams et al., 1985; Sams, Kaukoranta, Ha¨ma¨laïnen, & Naä¨ta¨nen, 1991; Scherg, Vajsar, & Picton, 1989; Tiitinen et al., 1993) obtained separable generation loci for the N1 and MMN, with the MMN (or MMNm) equivalent current dipole (ECD) in several studies being located about 1 cm anteriorly to that for the N1 (for a review, see Alho, 1995). For example, Scherg et al. (1989) found that the difference-wave negativity to a small frequency change could be modeled with one dipole source in the supratemporal auditory cortex of each hemisphere, whereas two dipoles in each hemisphere were needed to explain the negativity elicited by a large frequency change. According to the authors, one of these two dipoles was probably the genuine MMN generator, which was even activated by small frequency changes. The other dipole, which was activated somewhat earlier, was located posteriorly to the MMN generator and appeared to indicate enhanced activity of the supratemporal N1 generator. Thus, these results suggest that in the deviant-standard difference waves the early negativity may be composed of, or enhanced by, the release-from-refrac-

R. Naä¨ta¨nen et al. toriness of the N1 neurons, whereas the later part represents a genuine MMN. Consistent with these results, optical-imaging data showed separate generators for the N1 and MMN, which corresponded to the source locations previously found with the MEG (Rinne et al., 1999; Tse & Penney, 2008). Furthermore, corroborating fMRI results were obtained by Opitz, Schro¨ger, and von Cramon (2005) and MEG results by Maess, Jacobsen, Schro¨ger, and Friederici (2007). Opitz et al. (2005) combined the event-related fMRI with an experimental protocol that controlled for the refractoriness effects (Campbell, Winkler, & Kujala, 2007; Jacobsen & Schro¨ger, 2001; Jacobsen, Schro¨ger, Horenkamp, & Winkler, 2003), and found N1-related activity in the primary auditory cortex, whereas MMN-related activity originated in nonprimary auditory areas in the anterior part of Heschl’s gyrus. The authors concluded that their experiment succeeded in delineating the cognitive mechanism. This was based on delineating the memory-comparison processes generating a genuine MMN for frequency change and subserved by nonprimary auditory areas in the anterior part of Heschl’s gyrus, from the contribution of the sensory mechanism associated with a differential state of refractoriness in the primary auditory cortex. These results were corroborated by Maess et al. (2007) on the basis of their MEG data that showed opposite orientations of the early and late effects. These authors concluded that the early part of the deviant-minus-standard difference for frequency change is mainly due to the sensorial, N1mrelated mechanism, whereas the later part of the difference wave is mainly due to the cognitive MMNm-related mechanism. Inverse modeling revealed that sources for both contributions were bilaterally located in the gyrus temporales transversus. These MEG results suggested distinct but temporally and spatially partially overlapping activities of sensorial (non-comparator-based) and cognitive (comparator-based) mechanisms of automatic frequency-change detection in the auditory cortex (as also reported by Rosburg et al., 2004). According to Schro¨ger (1997), the function of those non-primary areas that generate the genuine MMN might be to establish a sparse representation of simple and complex invariants inherent in the recent stimulation, thereby providing the neural basis for memory comparison. Both sensorial and cognitive mechanisms contribute to pitch-change detection in the classic oddball paradigm. According to Schro¨ger (1997), this parallel use of two different mechanisms in the service of the same function underlines the biological importance of preattentively detecting changes in the auditory environment. Furthermore, Alho et al. (1996) observed the MMNm response to a change within a sound pattern, in addition to that elicited by a change in one frequency element of a music chord, with supratemporal sources anterior to the N1 source. Moreover, a number of further studies (Alain, Achim, & Woods, 1999; Escera et al., 2002; Frodl-Bauch, Kathman, Mo¨ller, & Hegerl, 1997; Giard et al., 1995; Leva¨nen et al., 1996; Paavilainen et al., 1991; Rosburg, 2003; Sysoeva, Takegata, & Naä¨ta¨nen, 2006; Takegata et al., 2001) also showed that the MMNs (MMNms) and their fMRI equivalents (Molholm et al., 2005) for different auditory features are generated in separate loci of the auditory cortex, which necessarily dissociates at least some of the MMN loci from that of the N1 to the (common) standard. Thus, it appears that, in the deviant-minus-standard difference waves, the early negativity may be enhanced by the releasefrom-refractoriness of the N1 neurons, whereas its later part is fully accounted for by the ‘‘genuine’’ MMN. This is also evident in the study of Tiitinen, May, Reinikainen, and Naä¨ta¨nen

Auditory processing that leads to conscious perception (1994), which showed the deviant-minus-standard difference wave as a function of the magnitude of frequency change. With very small changes, the MMN is clearly separate from the N1 enhancement whereas, with an increasing frequency difference, the MMN commences earlier, and there is an increasing temporal overlap between the two responses. (For a delineation of the genuine MMN part of the deviant-minus-standard difference, see also Horva´th et al., 2008.) In addition, an analogous data pattern was obtained with intensity increments, whereas intensity decrements appeared to elicit a genuine MMN only, because the N1 amplitude decreases with decreasing sound intensities (Naä¨ta¨nen, 1992; Naä¨ta¨nen, Paavilainen, Alho, Reinikainen, & Sams, 1989). Importantly, Tiitinen et al. (1993) also found that the MMNm generator mechanism seemed to be tonotopically organized but differed from the tonotopy of the N1m generator by its anterior locus. Tiitinen et al. (1993) therefore concluded that this MMNm tonotopy ‘‘is presumably that of the neuronal population(s) underlying frequency representation in sensory memory,’’ and that ‘‘the separability of this memory tonotopy from the afferent tonotopy of the neuronal population underlying N1m generation is suggested by the clearly different loci of the two responses’’ (Tiitinen et al., 1993, p. 539). Furthermore, the frontal subcomponent of the MMN is predominantly right-hemispheric (Giard et al., 1990), whereas the frontal N1 subcomponent is bilaterally generated (Giard et al., 1994). In addition, there also appears to exist a parietal MMN subcomponent (Gomot et al., 2000; Leva¨nen et al., 1996), which is probably generated in the posterior parietal cortex (Gomot et al., 2000) where no N1 generator seems to exist (Naä¨ta¨nen & Picton, 1987). Intracranial animal recordings. The stimulus-specific adaptation (SSA) found in animal recordings from different levels of the auditory pathway (Moore, 2003; Nelken & Ulanovsky, 2007; Ulanovsky, Las, & Nelken, 2003) has been suggested to fully explain the generation of the MMN. This view has been recently rejected, however. In their single- and multi-unit, evoked local field potential recordings that were obtained from the primary auditory cortex of the awake rat, von der Behrens et al. (2009) found that both neurons and evoked local field potentials adapted in a stimulus-specific manner. However, no MMN kinds of response, with characteristics matching those of the human MMN or those of the MMNs demonstrated in a cat (Cse´pe, Karmos, & Molna´r, 1987; Pincze, Lakatos, Rajkai, Ulbert, & Karmos, 2001, 2002) or monkey (Javitt, Schroeder, Steinschneider, Arezzo, & Vaughan, 1992; Javitt, Steinschneider, Schroeder, Vaughan, & Arezzo, 1994; Javitt, Steinschneider, Schroeder, & Arezzo, 1996), were found. Instead, the researchers concluded that the stimulus-specific adaptation of isolated units in the rat primary auditory cortex profoundly contributed to changes in the P1-N1 complex. Furthermore, in a recent review, Winkler, Denham, and Nelken (2009a) suggested that the SSAexhibiting neurons observed in all previous experiments lie upstream from those generating the MMN and, further, that the SSA alone cannot fully explain the MMN response. Pharmacological effects on the MMN and N1. There are also opposite effects of psychopharmacological manipulations on the MMN and N1 that implicate separate mechanisms of these two components. For instance, Umbricht et al. (2000) found that ketamine, an NMDA receptor antagonist, diminished the MMN amplitude but enhanced the N1 amplitude. Moreover, in their

9 study on monkeys, Javitt et al. (1996) observed that the NMDAreceptor antagonist MK-801 had no effect on the N1, whereas the MMN was abolished. MMN elicitation to sound omission. As further evidence for the N1-MMN separability, of particular importance are results that show MMN elicitation even with no afferent input. This occurs when a stimulus is omitted from a stimulus sequence presented at short constant (Yabe, Tervaniemi, Reinikainen, & Naä¨ta¨nen, 1997; Yabe, Tervaniemi, Sinkkonen, Huotilainen, Ilmoniemi, & Naä¨ta¨nen, 1998; Yabe, Koyoma, Kakigi, Gunji, Tervaniemi, Sato, & Kaneko, 2001; Yabe, Matsuoka, Sato, Hiruma, Sutoh, & Koyama, 2005; Yabe, Winkler, Czigler, Koyama, Kakigi, Suto, et al., 2001; Yabe, Sutoh, Matsuoka, Asai, Hiruma, Sato, et al., 2005) or varied SOAs (Ocea´k et al., 2006), when the second of two closely paced paired tones is occasionally omitted (Tervaniemi, Saarinen, Paavilainen, Danilova, & Naä¨ta¨nen, 1994), or when a stimulus is partially omitted (Winkler & Naä¨ta¨nen, 1993).These results suggest an N1-independent elicitation of the MMN, as no afferent elements could be involved in the generation of the MMN to stimulus omission. Different Developmental Time Courses of the MMN and N1 The MMN generator process is recordable even in the fetus (Draganova et al., 2005, 2007; Huotilainen et al., 2005), whereas N1 shows a considerably later developmental time course (Cse´pe, 1995; Pasman, Rotteveel, Maassen, & Visco, 1999; Ponton, Eggermont, Kwong, & Don, 2000; Ponton, Eggermont, Khosla, Kwong, & Don, 2002; Sharma, Kraus, McGee, & Nicol, 1997). MMN-N1 dissociations in patients. In some patient groups, MMN can be present with no N1. This can be the case for comatose patients (Fischer et al. 1999) or subjects who have cochlear implants (Ponton, Don, et al., 2000). The MMN with acoustically identical standards and deviants. The MMN process can be elicited by an auditory phoneme stimulus paired with an occasional incongruent visual stimulus in a sequence of identical auditory stimuli paired with congruent visual stimuli (Mo¨tto¨nen, Krause, Tiippana, & Sams, 2002; Mo¨tto¨nen, Schurman, & Sams, 2004; Sams et al., 1991; Tiippana et al., 2004). This result was even obtained in 5-month-old infants (Kushnerenko, Teinonen, Volein, & Csibra, 2008). Different sensory or perceptual correlates of the MMN and N1. The memory trace reflected by the MMN corresponds to the feature- and temporally integrated auditory event, whereas the sensory information that is encoded by the N1 generator does not appear to correspond to the subjective contents of perception (Butler, 1972; Parasuraman & Beatty, 1980; Winkler, Tervaniemi, & Naä¨ta¨nen, 1997) but rather to its attention-catching properties (Rinne et al., 2006). Consistent with this, the N1 seems better at indexing detection rather than discrimination, judging from the result that the N1 amplitude correlated with the detection of the occurrence of a faint signal but did not correlate with its recognition (Parasuraman & Beatty, 1980), whereas the MMN appears to be the best objective index of auditory discrimination currently available (Kraus et al., 1995, 1996; Lang et al., 1990; Naä¨ta¨nen & Alho, 1997; Naä¨ta¨nen et al., 2007). Furthermore, the N1 generator encodes stimulus information over the first 40–50 ms from stimulus onset only; therefore, it is unable to integrate stimulus energy long enough for perceived loudness to emerge (Gage & Roberts, 2000; Scharf, 1978; Scharf

10 & Houtsma, 1986). This, in turn, results in a clear dissociation between the sensory magnitude and the N1 amplitude (Picton, Goodman, & Bryce, 1970; Picton, Woods, & Proulx, 1978; Pratt & Sohmer, 1977). In a similar vein, Woods and Elmasian (1986) observed that the strong attenuation of the N1 amplitude at the beginning of a stimulus block is not directly related to loudness (see also Donald, 1979), but rather to its attention-catching properties or disruptiveness (Campbell, 2005; Campbell et al., 2003, 2005; Rinne et al., 2006; Valtonen et al., 2003). For the same reason, the N1 generator process does not seem to be involved in feature integration. Moreover, in contrast to the traces reflected by N1, those used in MMN elicitation can even encode long-duration auditory stimulus patterns that last for several hundreds of ms (Schro¨ger, Naä¨ta¨nen, & Paavilainen, et al., 1992) (although the first 300 ms from stimulus onset seem to be the most accurately represented; Grimm & Schro¨ger, 2005).

Memory Reflected by the MMN and the N1 In this section, we compare the kinds of memories reflected by the MMN and N1 with each other and show that these two responses are associated with very different kinds of sensory-memory information. As described in the introduction, the MMN is traditionally interpreted in terms of a memory-dependent effect. In the literature, the closest correspondence can be found in the so-called echoic memory, a form of auditory sensory memory with perception kind of vivacity lasting ca. 10 s in young adult participants (Cowan, 1984, 1988; Kallman & Massaro, 1979; Massaro, 1970, 1976). Several studies (Winkler et al., 1992, 1995; Winkler & Naä¨ta¨nen, 1994) show that the subjective contents of the memory involved in the MMN generation indeed correspond to that in perception and sensory memory (for a review, see Naä¨ta¨nen & Winkler, 1999). However, the MMN is not a direct index of sensory-memory traces, as a deviant after a single standard does not elicit the MMN (Cowan et al., 1993), and usually a sequence of 2–3 standards is needed before the MMN can be elicited (Bendixen et al., 2007; Cowan et al., 1993; Winkler, Cowan, Cs´ epe, Czigler, & Naä¨ta¨nen, 1996). Moreover, a very large N1 but no MMN is elicited by the first stimulus in a sequence after a long period of silence (Naä¨ta¨nen et al., 1989; Sams et al., 1985). This is because the elicitation of the MMN is not directly related to the sensorymemory trace of a single sound, but rather to the memory that encodes the regular sensory and higher-order features of a sequence of sounds (Cowan et al., 1993; Winkler, 2007; Winkler, Karmos, et al., 1996, 2009a). Consequently, rather than forming an index of memory-trace formation, the MMN indexes sensorymemory updating. For example, when a deviant event suddenly starts to repeat with no intervening standards, it in fact becomes a new standard against which deviants start to elicit the MMN (Winkler, Karmos, et al., 1996; Naä¨ta¨nen & Rinne, 2002; Bendixen et al., 2008). Such data support Naä¨ta¨nen’s (2009) suggestion of the MMN being a universal index of the second of the brain’s two main tasks with regard to environmental information, namely, updating the system of environmental stimulus representations. The first main task of the brain is the initial formation of the stimulus representations. The N1 adaptation reflects the refractoriness of the corresponding feature trace(s), whereas the MMN indicates the presence of feature-integrated stimulus representations that

R. Naä¨ta¨nen et al. correspond to the subjective contents of perception (Naä¨ta¨nen & Winkler, 1999). On the basis of the afore-reviewed differences between the MMN and N1 responses, we conclude that they are clearly separate, and represent different steps in central auditory processing, with the N1 generator process being related to the processing of separate auditory stimulus features. In contrast, the MMN response reflects the representation of inter-sound regularities based on feature- and temporally integrated sensory stimulus information (Naä¨ta¨nen & Winkler, 1999). Consequently, these two responses are associated with very different kinds of sensorymemory information. Selective Attention Effects on N1: The Separability of the N1 and the Processing Negativity (PN) According to Naä¨ta¨nen’s (1975) review, the first valid demonstration of ‘‘the N1 effect’’ of selective attention was provided by Hillyard, Hink, Schwent, Picton, et al. (1973). In their selective dichotic-listening task with very short, irregular inter-stimulus intervals (ISIs), the left-ear tones were of a considerably higher pitch than the right-ear tones. In addition, both sequences included occasional, randomly placed, slightly higher tones. The subject’s task was to count these deviants among the standards in the designated ear and to ignore all the input to the opposite ear. Hillyard and his colleagues found that the vertex N1 showed a higher amplitude for the attended than for the ignored stimuli. The authors regarded their effect as an enhancement of the ‘‘N1 component’’ and suggested that it reflected Broadbent’s (1970, 1971) stimulus-set mode of attention. ‘‘A stimulus set preferentially admits all sensory input to an attended channel (stimuli having in common a simple sensory attribute, such as pitch, position in space, receptor surface, or the like) for further perceptual analysis while blocking or attenuating input arriving over irrelevant channels (for example, the unattended ear) at an early stage of processing’’ (Hillyard et al., 1973, p. 180). The authors stressed the short onset latency of the effect as critical evidence: ‘‘The early latency of the attention effects upon N1 (evident at 60–70 ms in most subjects) suggests that the underlying attentional process is a tonically maintained set favoring one ear over the other rather than an active discrimination and recognition of each individual stimulus’’ (Hillyard et al., 1973, p. 179). Subsequently, by using a considerably longer and constant (800 ms) ISI in an otherwise quite similar experimental condition, Naä¨ta¨nen et al. (1978) found a slow negative shift which they termed the processing negativity (PN). The effect, recorded over the vertex and both left and right auditory cortices, appeared to represent no modulation of any obligatory ERP component but was rather a new component that emerged during selective attention. The peak amplitude of the N1 deflection was not affected. However, the N1 peak was followed by a low-amplitude (1–2 mV) negative displacement of the ERP of the attended standards compared with the unattended standards. Further, this displacement began at 150 ms, during the descending limb of the N1 deflection, and persisted for at least 500 ms. In addition, in their subsequent study, Naä¨ta¨nen, Gaillard, and Ma¨ntysalo (1980) obtained PNs over the temporal areas that were as large in amplitude as those over the vertex, which suggests that at least a part of the PN was generated in the sensory-specific auditory regions. Naä¨ta¨nen and Michie (1979) also proposed that the PN has a frontal generator. Naä¨ta¨nen et al. (1978) suggested that the PN is an endogenous component that was generated by a cerebral mechanism

Auditory processing that leads to conscious perception different from that of the N1 component. They also proposed that the N1 effect reported by Hillyard et al. (1973) might have been caused by the PN rather than by an amplification of the generator process of the N1 component. Namely, their considerably shorter ISIs might have shortened the PN latency so that the PN overlapped the N1 component, and caused an artificial increase in its measured amplitudes (Hillyard et al., 1973). Subsequently, the existence of the PN was verified by several further studies (Alho, Donauer, Paavilainen, Reinikainen, Sams, & Naä¨ta¨nen, 1987; Alho, To¨tto¨la¨, Reinikainen, Sams, & Naä¨ta¨nen, 1987; Okita, 1979; Okita, Konishi, & Inamori, 1983; Parasuraman, 1978), and its MEG equivalent was described by Hari et al. (1989). The PN was also observed by Hillyard and his colleagues (Hansen & Hillyard, 1980, 1983, 1984; Hillyard & Hansen, 1986; Hillyard & Kutas, 1983). Nonetheless, Hansen and Hillyard (1980) also suggested that a ‘‘genuine’’ N1 enhancement, too, may have been present in their data. Subsequent studies have indeed shown that, during very strongly focused selective attention, both the PN and the ‘‘genuine’’ enhancement of the N1 component may co-occur, thus supporting Hillyard’s position. Hence, the N1 effect of selective attention cannot be fully explained by the early onset of the PN (Naä¨ta¨nen, 1990, 1992; Naä¨ta¨nen, Schro¨ger, & Alho, 2002). Subsequent studies confirmed the two-component structure of the PN, proposed by Naä¨ta¨nen et al. (1978) and Naä¨ta¨nen & Michie (1979), with the sensory-specific auditory-cortex component that has a slightly earlier onset than that of the frontal component. The sensory-specific component was interpreted as being elicited by an on-line comparison between the incoming input and the so-called ‘‘attentional trace,’’ a voluntarily maintained representation of the to-be-attended stimulus (Naä¨ta¨nen 1982). It is developed by using fresh sensory-memory data of this stimulus (Donald & Young, 1982; Donald & Nugent, 1986) for tuning it to exactly correspond to its critical features (Naä¨ta¨nen, 1982, 1990). Further, this matching process between the incoming stimulus and the attentional trace, and hence the PN generation, terminates the sooner, the more different the stimulus is from the to-be-attended one. The PN runs its full course only in the case of a perfect match, with the generating selection process accepting the input to the prepared further-processing stages or for an immediate response (Alho et al., 1987a; Naä¨ta¨nen, 1982, 1990). The frontal component of the PN, in turn, might be related to the maintenance or control processes of the selectiveattention state (Naä¨ta¨nen, 1975, 1990, 1992). Model of Preconscious and Conscious Perceptual Processing in Audition The afore-reviewed data can be regarded as showing that the N1, MMN, and PN represent separate brain responses, each of which reflect its own auditory processing stages and separate properties of the storage of auditory sensory information. The N1 component is associated with the afferent response to sound onset (transient detection that subserves conscious stimuli perception) and also associated with feature analysis beyond that accomplished by the lower-level mechanisms (Banai et al., 2005, 2007; Galbraith et al., 1995, 1997; Johnson et al., 2007, 2008; King et al., 2002; Kraus & Nicol, 2005). The MMN, in turn, is elicited by auditory change. In more general terms, it is elicited by the violation of detected auditory regularities, which include the fully integrated auditory sensory information of the stimulus embedded in its sequential context. This violation is usually consciously

11 Table 1. Auditory ERP Components, Functions Reflected, and Roles of their Generators in Attention Component N1 MMN PN

Function

Role in attention

Onset detection and feature encoding Sensory-memory updating and change/rule violation detection Template (attentional-trace) matching

Conscious stimulus perception Conscious change detection Stimulus selection

perceived because of the attention-triggering property of the MMN process (Naä¨ta¨nen, 1990; Winkler, 2007). Finally, the PN is associated with attentional stimulus selection (see Table 1) and is based on the voluntarily maintained memory representation of the critical features of the to-be-attended sound (Alho et al., 1987a; Naä¨ta¨nen, 1982). The three ERP responses and their MEG in addition to fMRI equivalents, and the related behavioral data, constitute the empirical justification of the model presented in Figure 2. This is an updated and considerably elaborated version of that described by Naä¨ta¨nen (1990), and has been developed with the aim of defining the borderline between the automatic and attention-dependent processes in audition and to illustrate the emergence of conscious auditory perception. With a very short latency, sound (S) onset activates the feature-detector neuronal networks that correspond to the different stimulus features, such as the frequency-specific neurons along the afferent pathway. These early stimulus-specific processes mainly occur well before the N1 onset, and generate the auditory brainstem (ABR; Picton, Stapells, & Campbell, 1981; Starr & Don, 1988; Vaughan & Arezzo, 1988) and middle-latency responses (MLR; Picton, Hillyard, Krausz, & Galambos, 1974). Further, even though a large proportion of the N1 neurons are nonspecific (cf. the three N1 components described by Naä¨ta¨nen & Picton, 1987) or relatively nonspecific, because they have wide receptive fields (Woods & Elmasian, 1986), there is also some evidence for the N1 generator containing highly stimulus-specific neuronal populations (Butler, 1968; Naä¨ta¨nen et al., 1988; Picton et al., 1978). The outputs of the different feature detectors are then automatically integrated in time (the temporal window of integration; TWI) with a duration of approximately 200 ms (Atienza et al., 2003; Naä¨ta¨nen & Winkler, 1999; Nousak, Deacon, Ritter, & Vaughan, 1996; Ocea´k et al., 2006; Tervaniemi, Saarinen, et al., 1994; Yabe et al., 1997, 1998; Yabe, Koyoma, Kakigi, Gunji, Tervaniemi, Sato, & Kaneko, 2001; Yabe, Winkler, Czigler, Koyoma, Kakigi, Suto, et al., 2001; Yabe, Matsuoka, Sato, Hiruma, Sutoh, Koyoma, et al., 2005; Yabe, Sutoh, Matsuoka, Asai, Hiruma, Sato, et al., 2005) and across the different features (Gomes et al., 1995, 1997; Ritter et al., 2000; Takegata & Morotomi, 1999; Takegata et al., 1999, 2001, 2005; Winkler et al., 2005). For instance, loudness integration continues for 200 ms from stimulus onset, with the outcome of this process determining the loudness, the perceived intensity of the sound (Moore, 1989; Scharf & Houtsma, 1986; Zwicker & Fastl, 1990), which provides an estimate of the duration of the TWI. During the TWI, masking may also occur, with a subsequent stimulus often preventing the accurate perception of the preceding stimulus (Bazana & Stelmack, 2002; Cowan, 1984; Foyle & Watson, 1984; Hawkins & Presson, 1977, 1986; Massaro, 1970). This can also

12

R. Naä¨ta¨nen et al.

State Sensory Analysis (0 – 100 ms)

Transient Detectors

Sensory Memory (100 – 200 ms)

Att.Call (N1) EXCITABILITY

S

Feature Detectors

CONSC. PERC.

TWI

Sensory Memory

REHEARSAL

Temporary Feature Recognizers (attentional trace)

Post-sensory analysis and control (200 ms +) A T T E N T I O N C O N T R O L

EXEC. MECHANISMS + LTM

. .

Matching (AC-PN)

Figure 2. A model of conscious and unconscious processes in audition. The sound stimulus is first very rapidly analyzed by the different feature detectors. Thereafter, the outputs from the different feature detectors are temporarily integrated and with each other in the Temporal Window of Integration. The accumulation of this integrated sensory information in the mechanisms of Sensory Memory that evolves in time provides the sensory data of subjective contents of percepts, i.e., the central sound representation (Naä¨ta¨nen & Winkler, 1999). This central representation becomes consciously experienced, depending on the strength of the attention-call signal elicited by the dynamogenic stimulus features indexed by the N1 amplitude. Further, if some discernible change in auditory stimulation occurs, then this change results in the updating of auditory representations in Sensory Memory, eliciting the auditory-cortex MMN component. This, in turn, activates the frontal-cortex mechanisms generating the frontal MMN component (representing an attention-call signal to auditory change). During selective attention, the Executive Mechanisms use fresh sensory-memory data to set up and tune the Attentional Trace, a temporary template for the rapid selection of the to-be-attended input for further processing or response. This selection mechanism continuously depends on the active maintenance and rehearsal of the aspects of sensory input that very rapidly enable the listener to distinguish the relevant sensory input stream among the concurrent stimulus streams.

be shown by using the MMN, which is abolished when a masking stimulus follows each stimulus of the oddball paradigm with a very short interval. Therefore the MMN can also be used for determining the TWI duration (Winkler & Naä¨ta¨nen, 1994; Winkler, Reinikainen, & Naä¨ta¨nen, 1993). The outputs from the TWI process accumulate in the neural populations that subserve sensory memory. Further, the phase of the rapid accumulation of this stimulus-specific information underlies the stimulus perception. This, in turn, becomes conscious if the N1 transient-detector system, activated by the same stimulus, generates a signal (attention call) that is strong enough to exceed some temporally varying threshold. This mainly depends, if stimuli are ignored, on the following: the strength of attentional focus elsewhere, the rise time of the stimulus (Kodera, Hink, Yamada, & Suzuki, 1979; Onishi & Davis, 1968; Ostroff, McDonald, Schneider, & Alain, 2003; Pedersen & Salomon, 1977), and the degree of refractoriness of the neuronal population involved in the generation of this signal (Escera, Alho, Schro¨ger, & Winkler, 2000). The N1 amplitude apparently reflects the magnitude of the sensory ‘‘refreshment’’ of the feature trace involved (Naä¨ta¨nen, 1984). The biological significance of the long duration of these N1 refractoriness patterns might lie in ‘‘optimizing’’ the strength and frequency of the attention-call signals elicited. This transient-detector system (Graham, 1979; Loveless, 1983; MacMillan, 1973;

Newstead & Dennis, 1979; Phillips, 2001; Walter, 1964) is mainly composed of N1 neurons of non-specific or relatively nonspecific type, and probably also include the neurons that generate the frontal-cortical N1 component (Alcaini, Giard, Echallier, & Pernier, 1995; Giard et al., 1994). Consistent with this notion, previously, Walter (1964) suggested that the ‘‘vertex potential’’ notified the brain that something was happening while the specific sensory areas determined what it was (see also Davis & Zerlin, 1966; Gersuni, 1971; Naä¨ta¨nen, 1975). A second major cerebral route to attention switch/conscious perception is provided for violations of the automatic predictions that are based on the regularities extracted from the preceding sequence. Similarly, the MMN generator process also causes attention switch to the eliciting auditory event when this signal exceeds some momentary threshold, as in the case of the route from the transient detectors (N1). This is mediated by the auditory-cortex MMN process activating the frontal-cortex MMN process. Furthermore, in many cases, a deviant stimulus may, in addition to activating the MMN attention-call mechanisms, also enhance the generator process of the N1 component. This results in the attention-call signal triggered by the stimulus onset, thus increasing the probability of the conscious perception of stimulus change (Rinne et al., 2006). In either case, exceeding the threshold results in the conscious perception of the parallel featureand temporally integrated sensory contents incorporated in the

Auditory processing that leads to conscious perception memory trace (for a description of the parallel processing of features and the integrated stimulus representation, see Ritter et al., 1995). Hence, similar to the N1, depending on the strength of this MMN attention-call signal, it may lead to an attention switch to, and conscious perception of, auditory deviation or regularity violation that elicits the P3a (Escera et al., 1998, 2001; Friedman, Cycowicz, & Gaeta, 2001; Squires et al., 1975) or the N2b-P3a responses (Naä¨ta¨nen et al., 1982; for a review, see Naä¨ta¨nen, 1992). It is also possible that even some P3 (Sutton, Baren, Zubin, & John, 1965) and slower positivity are elicited, when the stimulus is recognized as a target. In addition, autonomic nervous system (ANS) responses may also be observed (Lyytinen, Blomberg, & Naä¨ta¨nen, 1992; Lyytinen & Naä¨ta¨nen, 1987). This attention switch is also manifested by transient deteriorations in primary-task performance that accompanies the MMN, as already reviewed (Escera et al., 1998). The conscious perception/experience of auditory stimulus representations (perception or rehearsal and imagination) are indicated by the yellow coloring in Figure 2. The conscious perception/awareness of the contents of sensory memory occurs either when one of the attentional-call processes is strong enough to exceed some momentarily varying threshold (Naä¨ta¨nen, 1990, 1992) or when the stimulus features of the to-be-attended stimulus are maintained in the attentional trace. The presence of the attentional trace continuously depends on its conscious, voluntary maintenance by the attentional control mechanisms reflected by the frontal PN component (FR-PN) (Hansen & Hillyard, 1980, 1983, 1984; Naä¨ta¨nen, 1982; Okita et al., 1983). During the lifetime of the attentional trace, each stimulus initiates a comparison process that is reflected by the auditory-cortex PN. The more discernable the stimulus is from that represented by the trace, the sooner the comparison process terminates. This comparison process runs its full time course only when the input fully matches with the stimulus represented by the attentional trace (Alho, To¨tto¨la¨, Reinikainen, Sams, & Naä¨ta¨nen, 1987; Alho, Donauer, Paavilainen, Reinikainen, Sams, & Naä¨ta¨nen, 1987). This is illustrated by the arrow in the bottom of Figure 2. See also Table 1. Illustrated in the figure is also another type of attention effect (EXCITABILITY) that is channel- rather than stimulus-specific. This is supported by the very early Hillyard type of N1 effect found in the condition in which the subject attends to stimuli presented to the designated ear at a very rapid rate (Hillyard et al., 1973). In this case, this effect expresses ‘‘a tonically maintained set rather than an active discrimination and recognition of each individual stimulus’’ (Hillyard et al., 1973, p. 179). Furthermore, even earlier selective-attention effects of this type were subsequently reported (Hackley, Wolldorf, & Hillyard, 1987, 1990; McCallum et al., 1983; Michie et al., 1993; Rinne et al., 2008; Woldorff, Hansen, & Hillyard, 1987; Woldorff, Hackley, & Hillyard, 1991; Woldorff & Hillyard, 1991), supporting the presence of attentional control over the input-channel excitability (exogenous attention effects; EXOG. AE; see Figure 2). The part of the model in which perception can become conscious closely corresponds to Naä¨ta¨nen and Winkler’s (1999) distinction between the representational/pre-representational systems. According to these authors, the representational system contrasts with the pre-representational system in that the stimulus code: (a) is stable, even though it is subject to decay and interference; (b) it contains the outcome of complete sensory analysis, has temporal properties, and corresponds to the per-

13 cept; (c) it can be brought into conscious experience by an attentional-call process or subject-initiated attention, imagination, or rehearsal; hence these codes are accessible to top-down operations; and (d) depending on the outcome of (c), the stimulus code can contact the LTM, which may result in the recognition of the stimulus and semantic activation (Massaro, 1976; Posner & Snyder, 1975; Pulvermu¨ller & Shtyrov, 2006; Pulvermu¨ller et al., 2009). The present model is consistent with these suggestions, but it can also accommodate the very early attention effects on auditory processing. Even though these top-down selective-attention effects are manifested peripherally from the borderline between the representational and pre-representational systems, the nature of these effects nevertheless is channel-specific, rather than stimulus-specific, which is in agreement with the suggested borderline between the representational and pre-representational systems. Finally, the general vigilance state of the organism is also illustrated. The excitability of the Transient Detectors depends on the subject’s state (Eason et al., 1964; Eason & Dudley, 1971; Fruhstorfer & Bergstro¨m, 1969; Hermanutz, Cohen, Sommer, 1981; Naä¨ta¨nen, 1970, 1975; Naä¨ta¨nen & Picton, 1987), but such effects might involve Feature Detectors, too (for reviews, see Naä¨ta¨nen & Picton, 1987; Sokolov et al., 2002). Furthermore, the Transient-Detector activation probably also contributes to the increased vigilance of the subject (Lindsley, 1960).

Concluding Discussion In the aforegoing, an updated version of Naä¨ta¨nen’s (1990) model of attention and automaticity in central auditory processing, which was developed more than 20 years ago, is introduced. First, the present model focuses on the dynamics of stimulus perception by incorporating the temporal and feature integration mechanism called the Temporal Window of Integration (TWI). The TWI integrates Feature-Detector outputs that form the neural basis for auditory event perception. Second, the present model acknowledges the very early selective-attention effects reported during the last two decades, starting from the now classic study of Hillyard et al. (1973). Consequently, it is now endowed with mechanisms of general centrifugal sensory excitability control of the Transient-Detector and Feature-Detector systems. These modulate all inputs through these channels in the same way rather than in a stimulus-specific manner. Third, the model also specifies brain events associated with change detection by suggesting that the attention-call signal elicited by auditory deviance specifically originates from the frontal mechanisms of MMN generation that is triggered by the auditory-cortex MMN generator process within a slightly earlier time course (Rinne et al., 2000; Tse & Penney, 2008). Fourth, the present model also separates the frontal Attention Control mechanisms within the Executive Mechanisms, which can also be commanded by internal attention-call signals that are generated during the automatic processing of auditory input. Fifth, state factors are now also represented in the model. Sixth, most importantly, the present model explicitly illustrates the stages or aspects of central auditory processing that can be consciously experienced. Consequently, the present revised model can contribute to the reconciliation between the two major competing lines of behavioral and ERP evidence pertinent to the role of attention in

14

R. Naä¨ta¨nen et al.

auditory processing. On the one hand, a large bulk of the results suggest automaticity even at the highest levels of central auditory processing (Deutsch & Deutsch, 1963; Holender, 1986; Kahneman & Treisman, 1984; Norman, 1968). On the other hand, a number of more recent studies (Alcaini et al., 1995; Hackley et al., 1987, 1990; McCallum et al., 1983; Woldorff et al., 1991; Woldorff & Hillyard, 1991) point to selective-attention effects even at the peripheral levels of auditory processing. Hence, results that stress high-level automaticity might be, at least partially, accounted for by the powerful automatic attention-switching mechanisms that are controlled by stimulus onsets, offsets, and changes. These transient and change (regularity violation) detectors cause the release of fully analyzed and integrated sensory information from sensory memory to the LTM system. This in turn leads to semantic activation (Escera et al., 2003; Naä¨ta¨nen, 1990, 1992). In this way, these automatic mechanisms could account for the data interpreted in terms of the ‘‘break-through of the unattended’’ that is found in selectiveattention experiments (Broadbent, 1982; Kahneman & Treisman, 1984; Moray, 1959; Treisman, 1960) even under the strict control of the attentional focus. The presence of such powerful attention-switching mechanisms serves the vital biological function of securing the rapid conscious evaluation of the significance of the eliciting event and a prompt response to it. It is to be stressed that, for most of the time, the far-reaching automaticity of stimulus processing that is endowed with powerful attention-switching mechanisms to potentially significant events is absolutely necessary in the auditory domain, in view of the presence of multiple concurrent auditory (Winkler, 2007; Winkler et al., 2009a) and other sensory-modality input streams. Moreover, the focus of attention is often directed to the visual domain. Therefore, it is of vital importance that auditory stimulation can alert one to potentially significant events that occur outside the focus of attention. The present model can also account for the experimental demonstrations of very early selective-attention effects on auditory ERPs by postulating centrifugal gain (excitability) control mechanisms through which the Executive Mechanisms can extend the attentional inflow control far down towards the periphery. In contrast to the Attentional-Trace mechanism, this very early selection process does not use specific stimulus representations in input selection but is rather based on selective inputchannel facilitation, with the stimulus set described as a tonic set of facilitation of inputs that arrives from a designated ear (Hillyard et al., 1973). Therefore, it appears possible to draw the borderline between representational and non-representational central auditory processing (Naä¨ta¨nen & Winkler, 1999) at the level of sensory-memory representations. More specifically, the borderline can be drawn at the input to this stage, where the memory-trace formation that underlies the emergence of auditory percepts occurs. This borderline also constrains the locus or loci of the possible conscious processes in the central auditory system. Depending on attentional factors, neural events that

subserve such conscious processes may occur at the entry of temporarily and feature-integrated feature detector inputs to the sensory memory. This occurs as a built-up phase of the central auditory representation for perception (Naä¨ta¨nen & Winkler, 1999), but does not occur more peripherally. Furthermore, this borderline is also essential for understanding the relationship between the MMN and the N1. These two responses are probably generated by neural events on the opposite sides of this critical borderline, and imply that even the highest afferent mechanisms, reflected by obligatory afferent ERP components, such as the N1 and P2, do not encode featureand temporally integrated stimulus information. Consequently, it appears that the sufficient immediate, direct neural basis of feature-integrated auditory event perception is only formed at the level of sensory-memory mechanisms, where the MMN is generated as an expression of memory updating and associated alarm functions. Finally, it might also be possible to develop a more general information-processing model by using these principles to explain the interplay between the voluntary (top-down) and involuntary (bottom-up) factors that compete for the moment-tomoment control of the direction of attention. In addition to the different modalities that have N1-types of responses to stimulus onset, including visual (Vogel & Luck, 2000) and somatosensory modalities (Kekoni et al., 1997), recent studies conducted on visual modality (Czigler, 2007; Pazo-Alvarez, Cadaveira, & Amendo, 2003) demonstrated the presence of a visual MMN (vMMN). This vMMN is, as the auditory MMN, generated in the modality-specific cortex (Astikainen, Ruusuvirta, Wikgren, & Korhonen, 2004; Astikainen & Hietanen, 2009; Kremlacek et al., 2004, 2006; Pazo-Alvarez, Amendo, & Cadaveira, 2004), it is memory-dependent (Astikainen, Lillstrang, & Ruusuvirta, 2008; Czigler, Balazs, & Winkler, 2002; Czigler, Winkler, Pato´, Va´rnagy, Weisz, & Bala´zs, 2006; Pazo-Alvarez et al., 2004), and is also independent of attention (Mu¨ller et al., 2010; Kremlacek et al., 2006). On the basis of their vMMN data, Kremlacek et al. (2006) concluded that the sensory information extracted by the magnocellular system undergoes processing capable of detecting changes in sequences of unattended peripheral motion stimuli. Moreover, the presence of the MMN with a sensoryspecific topography has also been demonstrated in the somatosensory modalities (Kekoni et al., 1997) and olfactory (Krauel et al., 1999) modalities, and also for integrated audio–visual stimuli (Widman et al., 2004; Winkler et al., 2009c). Furthermore, very recent vMMN data (Mu¨ller et al., 2010; Winkler et al., 2009c) showed, analogously to the auditory modality, the occurrence of attention-independent feature integration in visual object formation. Hence these results suggest that, by forming object representations early on, our perceptual system prepares the stage for higher cognitive processes and, generally, for successful adaptation to the ever-changing environment.

REFERENCES Alain, C., Achim, A., & Woods, D. L. (1999). Separate memory-related processing for auditory frequency and patterns. Psychophysiology, 36, 737–744. Alain, C., Woods, D. L., & Knight, R. T. (1998). A distributed cortical network for auditory sensory memory in humans. Brain Research, 812, 23–37.

Alcaini, M., Giard, M. H., Echallier, J. F., & Pernier, J. (1995). Selective auditory attention effects in tonotopically organized cortical areas: A topographic ERP study. Human Brain Mapping, 2, 159–169. Alcaini, M., Giard, M. H., Thevenet, M., & Pernier, J. (1994). Two separate frontal components in the N1 wave of the human auditory evoked response. Psychophysiology, 31, 611–615.

Auditory processing that leads to conscious perception Alho, K. (1995). Cerebral generators of mismatch negativity (MMN) and its magnetic counterpart (MMNm) elicited by sound change. Ear and Hearing, 16, 38–51. Alho, K., Donauer, N., Paavilainen, P., Reinikainen, K., Sams, M., & Naä¨ta¨nen, R. (1987). Stimulus selection during auditory spatial attention as expressed by event-related potentials. Biological Psychology, 24, 153–162. Alho, K., Huotilainen, M., Tiitinen, H., Ilmoniemi, R. J., Knuutila, J., & Naä¨ta¨nen, R. (1993). Memory-related processing of complex sound patterns in human auditory cortex: A MEG study. NeuroReport, 4, 391–394. Alho, K., Tervaniemi, M., Huotilainen, M., Lavikainen, J., Tiitinen, H., Ilmoniemi, R. J., et al. (1996). Processing of complex sounds in the human auditory cortex as revealed by magnetic brain responses. Psychophysiology, 33, 369–375. Alho, K., To¨tto¨la¨, K., Reinikainen, K., Sams, M., & Naä¨ta¨nen, R. (1987). Brain mechanism of selective listening reflected by eventrelated potentials. Electroencephalography and Clinical Neurophysiology, 68, 458–470. Alho, K., Winkler, I., Escera, C., Huotilainen, M., Virtanen, J., Jaä¨skelaïnen, et al. (1998). Processing of novel sounds and frequency changes in the human auditory cortex: Magnetoencephalographic recordings. Psychophysiology, 35, 211–224. Alho, K., Woods, D. L., Algazi, A., Knight, R. T., & Naä¨ta¨nen, R. (1994). Lesions of frontal cortex diminish the auditory mismatch negativity. Electroencephalography and Clinical Neurophysiology, 82, 356–368. Astikainen, P., & Hietanen, J. K. (2009). Event-related potentials to taskirrelevant changes in facial expressions. Behavioural and Brain Functions, 5, 30 doi:10.1186/1744-9081-5-30. Astikainen, P., Lillstrang, E., & Ruusuvirta, T. (2008). Visual mismatch negativity for changes in orientation – sensory memory-dependent response. European Journal of Neuroscience, 28, 2319–2324. Astikainen, P., Ruusuvirta, T., Wikgren, J., & Korhonen, T. (2004). The human brain processes visual changes that are not cued by attended auditory stimulation. Neuroscience Letters, 368, 231–4. Atienza, M., Cantero, J. L., Grau, C., Gomez, C., Dominguez-Marin, E., & Escera, C. (2003). Effects of temporal encoding on auditory object formation: A mismatch negativity study. Cognitive Brain Research, 16, 359–371. Banai, K., Abrams, D., & Kraus, N. (2007). Sensory-based learning disability: Insights from brainstem processing of speech sounds. International Journal of Audiology, 46, 524–532. Banai, K., Nicol, T., Zecker, S. G., & Kraus, N. (2005). Brainstem timing: Implications for cortical processing and literacy. Journal of Neuroscience, 25, 9850–9857. Bazana, P. G., & Stelmack, R. M. (2002). Intelligence and information processing during an auditory discrimination task with backward masking: An event-related potential analysis. Journal of Personality and Social Psychology, 4, 998–1008. Bendixen, A., Prinz, W., Horva´th, J., Trujillo-Barreto, N. J., & Schro¨ger, E. (2008). Rapid extraction of auditory feature contingencies. NeuroImage, 41, 1111–1119. Bendixen, A., Roeber, U., & Schro¨ger, E. (2007). Regularity extraction and application in dynamic auditory stimulus sequences. Journal of Cognitive Neuroscience, 19, 1664–1677. Bendixen, A., & Schro¨ger, E. (2008). Memory trace formation for abstract auditory features and its consequences in different attentional contexts. Biological Psychology, 78, 231–241. Bendixen, A., Schro¨ger, E., & Winkler, I. (2009). I heard that coming: Event-related potential evidence for stimulus-driven prediction in the auditory system. The Journal of Neuroscience, 29, 8447–8451. Bonte, M., Mitterer, H., Zellagui, N., Poelmans, H., & Blomert, L. (2005). Auditory cortical tuning to statistical regularities in phonology. Clinical Neurophysiology, 116, 2765–2774. Bo¨ttcher-Gandor, C., & Ullsperger, P. (1992). Mismatch negativity in event-related potentials of auditory stimuli as a function of varying interstimulus interval. Psychophysiology, 29, 546–550. Broadbent, D. E. (1970). Stimulus set and response set: Two kinds of selective attention. In D. I. Mostofsky (Ed.), Attention: Contemporary theory and analysis (pp. 51–60). Appleton-Century-Crofts, New York. Broadbent, D. E. (1971). Decision and stress. Academic Press, London. Broadbent, D. E. (1982). Task combination and selective intake of information. Acta Psychologica, 50, 253–290.

15 Butler, R. A. (1968). Effect of changes in stimulus frequency and intensity on habituation of the human vertex potential. Journal of the Acoustical Society of America, 44, 945–950. Butler, R. A. (1972). The auditory evoked response to stimuli producing periodicity pitch. Psychophysiology, 9, 233–237. Campbell, T. A. (2005). The cognitive neuroscience of auditory distraction. Trends in Cognitive Sciences, 9, 3–5. Campbell, T., Winkler, I., & Kujala, T. (2005). Disruption of immediate memory and brain processes: An auditory ERP protocol. Brain Research Protocols, 14, 77–86. Campbell, T., Winkler, I., & Kujala, T. (2007). N1 and the mismatch negativity are spatiotemporally distinct ERP components: Disruption of immediate memory by auditory distraction can be related to N1. Psychophysiology, 44, 530–540. Campbell, T., Winkler, I., Kujala, T., & Naä¨ta¨nen, R. (2003). The N1 hypothesis and irrelevant sound: Evidence from token set size effects. Cognitive Brain Research, 18, 39–47. Cheour, M., Ceponiene, R., Lehtokoski, A., Luuk, A., Allik, J., Alho, K., & Naä¨ta¨nen, R. (1998). Development of language-specific phoneme representations in the infant brain. Nature Neuroscience, 1, 351–353. Cheour, M., Ceponiene, R., Leppa¨nen, P., Alho, K., Kujala, T., Renlund, M., et al. (2002). The auditory sensory memory trace decays rapidly in newborns. Scandinavian Journal of Psychology, 43, 33–39. Cooper, R. J., Todd, J., McGill, K., & Michie, P. T. (2006). Auditory sensory memory and the aging brain: A mismatch negativity study. Neurobiology of Aging, 27, 752–762. Cowan, N. (1984). On short and long auditory stores. Psychological Bulletin, 96, 341–370. Cowan, N. (1988). Evolving conceptions of memory storage, selective attention, and their mutual constraints within the human information-processing system. Psychological Bulletin, 104, 163–191. Cowan, N., Winkler, I., Teder, W., & Naä¨ta¨nen, R. (1993). Memory prerequisites of the mismatch negativity in the auditory event-related potential (ERP). Journal of Experimental Psychology: Human Perception and Performance, 19, 909–921. Cse´pe, V. (1995). On the origin and development of the mismatch negativity. Ear & Hearing, 16, 91–104. Cse´pe, V., Karmos, G., & Molna´r, M. (1987). Evoked potential correlates of stimulus deviance during wakefulness and sleep in cat: Animal model of mismatch negativity. Electroencephalography and Clinical Neurophysiology, 66, 571–578. Cse´pe, V., Pantev, C., Hoke, M., Hampson, S., & Ross, B. (1992). Evoked magnetic responses of the human auditory cortex to minor pitch changes: Localization of the mismatch field. Electroencephalography and Clinical Neurophysiology, 84, 538–548. Czigler, I. (2007). Visual mismatch negativity. Violation of nonattended environmental regularities. Journal of Psychophysiology, 21, 224–230. Czigler, I., Balazs, L., & Winkler, I. (2002). Memory-based detection of task-irrelevant visual changes. Psychophysiology, 39, 869–873. Czigler, I., Winkler, I., Pato´, L., Va´rnagy, A., Weisz, J., & Bala´zs, L. (2006). Visual temporal window of integration as revealed by the visual mismatch negativity event-related potential to stimulus omission. Brain Research, 1104, 129–140. Davis, H., & Zerlin, S. (1966). Acoustic relations of the human vertex potential. Journal of the Acoustical Society of America, 39, 109–116. Deacon, D., Nousak, J. M., Pilotti, M., Ritter, W., & Yang, C.-M. (1998). Automatic change detection: Does the auditory system use representations of individual stimulus features or gestalts? Psychophysiology, 35, 413–419. Dehaene-Lambertz, G. (1997). Electrophysiological correlates of categorical phoneme perception in adults. NeuroReport, 8, 919–924. Dehaene-Lambertz, G., Dupoux, E., & Gout, A. (2000). Electrophysiological correlates of phonological processing: A cross-linguistic study. Journal of Cognitive Neuroscience, 12, 635–647. Deouell, L. Y. (2007). The frontal generator of the mismatch negativity revisited. Journal of Psychophysiology, 21, 188–203. Deutsch, J. A., & Deutsch, D. (1963). Attention: Some theoretical considerations. Psychological Review, 70, 80–90. Donald, M. W. (1979). Limits on current theories of transient evoked potentials. In J. Desmedt (Ed.), Cognitive components in cerebral event-related potentials and selective attention. Progress in Clinical Neurophysiology (Vol. 6, pp. 187–199). Basel, Switzerland: Karger. Donald, M. W., & Nugent, P. M. (1986). Attentional tuning of a spatially specific auditory component. In W. C. McCallum, R. Zappoli, & F.

16 Denoth (Eds.), Cerebral psychophysiology: Studies in event-related potentials (pp. 38–42; Suppl 38 Electroencephalography and Clinical Neurophysiology). Amsterdam: Elsevier. Donald, M. W., & Young, M. J. (1982). The time course of selective neural tuning in auditory attention. Experimental Brain Research, 46, 357–367. Draganova, R., Eswaran, H., Murphy, P., Huotilainen, M., Lowery, C., & Preissl, H. (2005). Sound frequency change detection in fetuses and newborns, a magnetoencephalographic study. NeuroImage, 28, 354– 361. Draganova, R., Eswaran, H., Murphy, P., Lowery, C., & Preissl, H. (2007). Serial magnetoencephalographic study of fetal and newborn auditory discriminative evoked responses. Early Human Development, 83, 199–207. Eason, R. G., Aiken, L. R. Jr., White, C. T., & Lichtenstein, M. (1964). Activation and behavior: II. Visually evoked cortical potentials in man as indicates of activation level. Perceptual and Motor Skills, 19, 875–895. Eason, R. G., & Dudley, L. M. (1971). Physiological and behavioral indicates of activation. Psychophysiology, 7, 223–232. Eckstein, K., & Friederici, A. D. (2006). It´s early: Event-related potential evidence for initial interaction of syntax and prosody in speech comprehension. Journal of Cognitive Neuroscience, 18, 1696–1711. Escera, C., Alho, K., Schro¨ger, E., & Winkler, I. (2000). Involuntary attention and distractibility as evaluated with event-related brain potentials. Audiology & Neuro-Otology, 5, 151–166. Escera, C., Alho, K., Winkler, I., & Naä¨ta¨nen, R. (1998). Neural mechanisms of involuntary attention switching to novelty and change in the acoustic environment. Journal of Cognitive Neuroscience, 10, 590– 604. Escera, C., Corral, M.-J., & Yago, H. (2002). An electrophysiological and behavioral investigation of involuntary attention towards auditory frequency, duration and intensity changes. Cognitive Brain Research, 14, 325–332. Escera, C., Yago, E., & Alho, K. (2001). Electrical responses reveal the temporal dynamics of brain events during involuntary attention switching. European Journal of Neuroscience, 14, 877–883. Escera, C., Yago, E., Corral, M.-J., Corbera, S., & Nun˜ez, M. I. (2003). Attention capture by auditory significant stimuli: Semantic analysis follows attention switching. European Journal of Neuroscience, 18, 2408–2412. Fischer, C., Morlet, D., Bouchet, P., Luaute, J., Jourdan, C., & Salord, F. (1999). Mismatch negativity and late auditory evoked potentials in comatose patients. Clinical Neurophysiology, 110, 1601–1610. Ford, J. M., Roth, W. T., & Kopell, B. S. (1976a). Auditory evoked potentials to unpredictable shifts in pitch. Psychpophysiology, 13, 32– 39. Ford, J. M., Roth, W. T., & Kopell, B. S. (1976b). Attention effects on auditory evoked potentials to infrequent events. Biological Psychology, 4, 65–77. Foyle, D. C., & Watson, C. S. (1984). Stimulus-based versus performance-based measurement of auditory backward recognition masking. Perception and Psychophysics, 36, 515–522. Friederici, A. D. (1995). The time course of syntactic activation during language processing: A model based on neuropsychological and neurophysiological data. Brain and Language, 50, 259–281. Friederici, A. D. (2002). Towards a neural basis of auditory sentence processing. Trends in Cognitive Science, 6, 78–84. Friederici, A. D. (2004). Event-related brain potential studies in language. Current Neurology and Neuroscience Reports, 4, 466–470. Friederici, A. D., Gunter, T. C., Hahne, A., & Mauth, K. (2004). The relative timing of syntactic and semantic processes in sentence comprehension. NeuroReport, 15, 165–169. Friederici, A. D., Hahne, A., & Mecklinger, A. (1996). The temporal structure of syntactic parsing: Early and late event-related brain potential effects elicited by syntactic anomalies. Journal of Experimental Psychology: Learning, Memory and Cognition, 22, 1219–1248. Friederici, A. D., Pfeifer, E., & Hahne, A. (1993). Event-related brain potentials during natural speech processing: Effects of semantic, morphological and syntactic violations. Cognitive Brain Research, 1, 183–192. Friedman, D., Cycowicz, Y. M., & Gaeta, H. (2001). The novelty P3: An event-related brain potential (ERP) sign of the brain’s evaluation of novelty. Neuroscience and Biobehavioral Reviews, 25, 355–373.

R. Naä¨ta¨nen et al. Friston, K., & Kiebel, S. (2009). Cortical circuits for perceptual inference. Neural Networks, 22, 1093–1104. Frodl-Bauch, T., Kathmann, N., Mo¨ller, H. J., & Hegerl, U. (1997). Dipole localization and test-retest reliability of frequency and duration mismatch negativity generator processes. Brain Topography, 10, 3–8. Fruhstorfer, H., & Bergstro¨m, R. M. (1969). Human vigilance and auditory evoked responses. Electroencephalography & Clinical Neurophysiology, 27, 346–355. Fujioka, T., Trainor, L. J., Ross, B., Kakigi, R., & Pantev, C. (2004). Musical training enhances automatic encoding of melodic contour and interval structure. Journal of Cognitive Neuroscience, 16, 1010–1021. Fuster, J. (1989). The prefrontal cortex: Anatomy, physiology, and neuropsychology of the frontal lobe. New York: Raven Press. Gage, N. M., & Roberts, T. P. (2000). Temporal integration: Reflections in the M100 of the auditory evoked field. NeuroReport, 11, 2723–2726. Galbraith, G. C., Arbagey, P. W., Branski, R., Comerci, N., & Rector, P. M. (1995). Intelligible speech encoded in the human brain stem frequency-following response. NeuroReport, 6, 2363–2367. Galbraith, G. C., Jhaveri, S. P., & Kuo, J. (1997). Speech-evoked brainstem frequency-following responses during verbal transformations due to word repetition. Electroencephalography & Clinical Neurophysiology, 102, 46–53. Garrido, M. I., Friston, K. J., Kiebel, S. J., Stephan, K. E., Baldeweg, T., & Kilner, J. M. (2008). The functional anatomy of the MMN: A DCM study of the roving paradigm. NeuroImage, 42, 936–944. Garrido, M. I., Kilner, J. M., Kiebel, S. J., & Friston, K. J. (2009a). Dynamic causal modeling of the response to frequency deviants. Journal of Neurophysiology, 101, 2620–2631. Garrido, M., Kilner, J. M., Stephan, K. E., & Friston, K. J. (2009b). The mismatch negativity: A review of underlying mechanisms. Clinical Neurophysiology, 120, 453–463. Gersuni, G. V. (1971). Temporal organization of the auditory function. In Gersuni, G. V. (Ed.), Sensory processes at the neuronal and behavioral levels (pp. 85–114). New York: Academic Press. Giard, M. H., Lavikainen, J., Reinikainen, K., Perrin, F., Bertrand, O., Pernier, J., & Naä¨ta¨nen, R. (1995). Separate representation of stimulus frequency, intensity, and duration in auditory sensory memory: An event-related potential and dipole-model analysis. Journal of Cognitive Neuroscience, 7, 133–143. Giard, M. H., Perrin, F., Echallier, J. F., The´venet, M., Froment, J. C., & Pernier, J. (1994). Dissociation of temporal and frontal components in the human auditory N1 wave: A scalp current density and dipole model analysis. Electroencephalography and Clinical Neurophysiology, 92, 238–252. Giard, M. H., Perrin, F., Pernier, J., & Bouchet, P. (1990). Brain generators implicated in processing of auditory stimulus deviance: A topographic event-related potential study. Psychophysiology, 27, 627–640. Glass, E., Sachse, S., & von Suchodoletz, W. (2008a). Development of auditory sensory memory from 2 to 6 years: An MMN study. Journal of Neural Transmission, 115, 1221–1229. Glass, E., Sachse, S., & von Suchodoletz, W. (2008b). Auditory sensory memory in 2-year-old children: An event-related potential study. NeuroReport, 19, 569–573. Gomes, H., Bernstein, R., Ritter, W., Vaughan Jr., H. G., & Miller, J. (1997). Storage of feature conjunctions in transient auditory memory. Psychophysiology, 34, 712–716. Gomes, H., Ritter, W., & Vaughan Jr., H. G. (1995). The nature of preattentive storage in the auditory system. Journal of Cognitive Neuroscience, 7, 81–94. Gomes, H., Sussman, E, Ritter, W., Kurtzberg, D., Cowan, N., & Vaughan, H. G. Jr. (1999). Electrophysiological evidence of developmental changes in the duration of auditory sensory memory. Developmental Psychology, 35, 294–302. Gomot, M., Giard, M.-H., Roux, S., Barthelemy, C., & Bruneau, N. (2000). Maturation of frontal and temporal components of mismatch negativity (MMN) in children. NeuroReport, 11, 3109–3112. Goodin, D. S., Squires, K. C., Henderson, B. H., & Starr, A. (1978). An early event-related cortical potential. Psychophysiology, 15, 360–365. Graham, F. K. (1979). Distinguishing among orienting, defence and startle reflexes. In H. D. Kimmel, E. H. van Olst, & J. F. Orlebeke

Auditory processing that leads to conscious perception (Eds.), The orienting reflex in humans (pp. 137–167). Hillsdale, NJ: Erlbaum. Grau, C., Escera, C., Yago, E., & Polo, D. (1998). Mismatch negativity and auditory memory evaluation: A new faster paradigm. NeuroReport, 9, 2451–2456. Grimm, S., & Schro¨ger, E. (2005). Preattentive and attentive processing of temporal and frequency characteristics within long sounds. Cognitive Brain Research, 25, 711–721. Grimm, S., & Schro¨ger, E. (2007). The processing of frequency deviations within sounds: Evidence for the predictive nature of the mismatch negativity (MMN) system. Restorative Neurology and Neuroscience, 25, 241–249. Gunter, T. C., Friederici, A. D., & Hahne, A. (1999). Brain responses during sentence reading: Visual input affects central processing. NeuroReport, 10, 3175–3178. Hackley, S. A., Woldorff, M., & Hillyard, S. A. (1987). Combined use of microreflexes and event-related brain potentials as measures of auditory selective attention. Psychophysiology, 24, 632–647. Hackley, S. A., Woldorff, M., & Hillyard, S. A. (1990). Cross-model selective attention effects on retinal, myogenic, brainstem, and cerebral evoked potentials. Psychophysiology, 27, 195–208. Hahne, A., & Friederici, A. D. (1999). Electrophysiological evidence for two steps in syntactic analysis: Early automatic and late controlled processes. Journal of Cognitive Neuroscience, 11, 194–205. Hansen, J. C., & Hillyard, S. A. (1980). Endogenous brain potentials associated with selective auditory attention. Electroencephalography & Clinical Neurophysiology, 49, 277–290. Hansen, J. C., & Hillyard, S. A. (1983). Selective attention to multidimensional auditory stimuli in man. Journal of Experimental Psychology: Human Perception and Performance, 9, 1–9. Hansen, J. C., & Hillyard, S. A. (1984). Effects of stimulation rate and attribute cuing on event-related potentials during selective auditory attention. Psychophysiology, 21, 394–405. Hari, R., Ha¨ma¨laïnen, M., Ilmoniemi, R. J., Kaukoranta, E., Reinikainen, K., Salminen, J., et al. (1984). Responses of the primary auditory cortex to pitch changes in a sequence of tone pips: Neuromagnetic recordings in man. Neuroscience Letters, 50, 127–132. Hari, R., Ha¨ma¨laïnen, M., Kaukoranta, E., Ma¨kela¨, J., Joutsiniemi, S. L., & Tiihonen, J. (1989). Selective listening modifies activity of the human auditory cortex. Experimental Brain Research, 74, 463–470. Haroush, K., Hochstein, S., & Deouell, L. Y. (2010). Momentary fluctuations in allocation of attention: Cross-modal effects of visual task load on auditory discrimination. Journal of Cognitive Neuroscience, 22, 1440–1451. Hasting, A. S., Kotz, S. A., & Friederici, A. D. (2007). Setting the stage for automatic syntax processing: The mismatch negativity as an indicator of syntactic priming. Journal of Cognitive Neuroscience, 19, 386–400. Hawkins, H. L., & Presson, J. C. (1977). Masking and pereptual selectivity in auditory recognition. In S. Dornic (Ed.), Attention and performance VI (pp. 195–211). Hillsdale, NJ: Erlbaum. Hawkins, H. L., & Presson, J. C. (1986). Auditory information processing. In K. R. Boff, L. Kaufman, & J. P. Thomas (Eds.), Handbook of Perception and Human Performance (Vol. 2, pp. 261–264). New York: Wiley. Hermanutz, M., Cohen, R., & Sommer, W. (1981). The effects of serial order in long sequences of auditory stimuli on event-related potentials. Psychophysiology, 18, 415–423. Hillyard, S. A., & Hansen, J. C. (1986). Attention: Electrophysiological approaches. In M. G. H. Coles, E. Donchin, & S. W. Porges (Eds.), Psychophysiology: Systems, processes, and applications (pp. 227–243). New York: The Guilford Press. Hillyard, S. A., Hink, R. F., Schwent, V. L., & Picton, T. W. (1973). Electrical signs of selective attention in the human brain. Science, 182, 177–180. Hillyard, S. A., & Kutas, M. (1983). Electrophysiology of cognitive processing. Annual Review of Psychology, 34, 33–61. Holender, D. (1986). Semantic activation without conscious identification in dichotic listening, parafoveal vision, and visual masking: A survey and appraisal. Behavioral and Brain Sciences, 9, 1–66. Horva´th, J., Czigler, I., Jacobsen, T., Maess, B., Schro¨ger, E., & Winkler, I. (2008). MMN or no MMN: No magnitude of deviance effect on the MMN amplitude. Psychophysiology, 45, 60–69. Huotilainen, M., Kujala, A., Hotakainen, M., Parkkonen, L., Taulu, S., Simola, J., et al. (2005). Short-term memory functions of the human

17 fetus recorded with magnetoencephalography. NeuroReport, 16, 81–84. Jaä¨skelaïnen, I. P., Ahveninen, J., Belliveau, J. W., Raij, T., & Sams, M. (2007). Short-term plasticity in auditory cognition. Trends in Neurosciences, 30, 653–661. Jaä¨skelaïnen, I. P., Ahveninen, J., Bonmassar, G., Dale, A. M., Ilmoniemi, R. J., Leva¨nen, S., et al. (2004). Human posterior auditory cortex gates novel sounds to consciousness. Proceedings of the National Academy of Sciences, 101, 6809–6814. Jaä¨skelaïnen, I. P., Alho, K., Escera, C., Winkler, I., Sillanaukee, P., & Naä¨ta¨nen, R. (1996b). Effects of ethanol and auditory distraction on forced choice reaction time. Alcohol, 13, 153–156. Jaä¨skelaïnen, I. P., Pekkonen, E., Hirvonen, J., Sillanaukee, P., & Naä¨ta¨nen, R. (1996a). Mismatch negativity subcomponents and ethyl alcohol. Biological Psychology, 43, 13–25. Jaä¨skelaïnen, I. P., Varonen, R., Naä¨ta¨nen, R., & Pekkonen, E. (1999). Decay of cortical pre-attentive sound discrimination in middle-age. NeuroReport, 10, 123–126. Jacobsen, T., & Schro¨ger, E. (2001). Is there pre-attentive memory-based comparison of pitch? Psychophysiology, 38, 723–727. Jacobsen, T., Schro¨ger, E., Horenkamp, T., & Winkler, I. (2003). Mismatch negativity to pitch change: Varied stimulus proportions in controlling effects of neural refractoriness on human auditory event related brain potentials. Neuroscience Letters, 344, 79–82. Javitt, D. C., Schroeder, C. E., Steinschneider, M., Arezzo, J. C., & Vaughan Jr., H. G. (1992). Demonstration of mismatch negativity in the monkey. Electroencephalography and Clinical Neurophysiology, 83, 87–90. Javitt, D. C., Steinschneider, M., Schroeder, C. E., & Arezzo, J. C. (1996). Role of cortical N-methyl-D-aspartate receptors in auditory sensory memory and mismatch negativity generation: Implications for schizophrenia. Proceedings of the National Academy of Sciences of the USA, 93, 11962–11967. Javitt, D. C., Steinschneider, M., Schroeder, C. E., Vaughan Jr., H. G., & Arezzo, J. C. (1994). Intracortical mechanisms of mismatch negativity (MMN) generation. Brain Research, 667, 192–200. Johnson, K. L., Nicol, T., Zecker, S. G., Bradlow, A. R., Skoe, E., & Kraus, N. (2008). Brainstem encoding of voiced consonant-vowel stop syllables. Clinical Neurophysiology, 119, 2613–2635. Johnson, K. L., Nicol, T., Zecker, S. G., & Kraus, N. (2007). Auditory brainstem correlates of perceptual timing deficits. Journal of Cognitive Neuroscience, 19, 376–385. Kahneman, D., & Treisman, A. (1984). Changing views of attention and automaticity. In R. Parasuraman & D. R. Davies (Eds.), Varieties of attention (pp. 29–61). New York: Academic Press. Kaipio, M-L., Cheour, M., Ceponiene, R., O¨hman, J., Alku, P., & Naä¨ta¨nen, R. (2000). Increased distractibility in closed head injury as revealed by event-related potentials. NeuroReport, 11, 1463–1468. Kallman, H. J., & Massaro, D. W. (1979). Similarity effects in backward recognition masking. Journal of Experimental Psychology: Human Perception and Performance, 5, 110–128. Kekoni, J., Ha¨ma¨laïnen, H., Saarinen, M., Gro¨hn, J., Reinikainen, K., Lehtokoski, A., & Naä¨ta¨nen, R. (1997). Rate effect and mismatch responses in the somatosensory system: ERP-recordings in humans. Biological Psychology, 46, 125–142. King, C., Warrier, C. M., Hayes, E., & Kraus, N. (2002). Deficits in auditory brainstem pathway encoding of speech sounds in children with learning problems. Neuroscience Letters, 319, 111–115. Knight, R. T. (1991). Evoked potential studies of attention capacity in human frontal lobe lesions. In H. Levin, H. Eisenberg, & F. Benton (Eds.), Frontal lobe function and dysfunction (pp. 139–153). Oxford: Oxford University Press. Kodera, K., Hink, R. F., Yamada, O., & Suzuki, J. (1979). Effects of rise time on simultaneously recorded auditory-evoked potentials from the early, middle and late ranges. Audiology, 18, 395–402. Koelsch, S., Gunter, T. C., Schro¨ger, E., & Friederici, A. D. (2003). Processing tonal modulations: An ERP study. Journal of Cognitive Neuroscience, 15, 1149–1159. Koelsch, S., Grossmann, T., Gunter, T. C., Hahne, A., Schro¨ger, E., & Friederici, A. D. (2003). Children processing music: Electric brain responses reveal musical competence and gender differences. Journal of Cognitive Neuroscience, 15, 683–693. Koelsch, S., & Siebel, W. A. (2005). Towards a neural basis of music perception. Trends in Cognitive Sciences, 9, 578–584.

18 Korzyukov, O., Alho, K., Kujala, A., Gumenyuk, V., Ilmoniemi, R. J., Virtanen, J., et al.. (1999). Electromagnetic responses of the human auditory cortex generated by sensory-memory based processing of tone-frequency changes. Neuroscience Letters, 276, 169–172. Krauel, K., Schott, P., Sojka, B., Pause, B. M., & Ferstl, R. (1999). Is there a mismatch negativity analogue in the olfactory event-related potential? Journal of Psychophysiology, 13, 49–55. Kraus, N., McGee, T. J., Carrell, T. D., King, C., Tremblay, K., & Nicol, T. G. (1995). Central auditory plasticity associated with speech discrimination training. Journal of Cognitive Neuroscience, 7, 25–32. Kraus, N., McGee, T. J., Carrell, T. D., Zecker, S. G., Nicol, T. G., & Koch, D. B. (1996). Auditory neurophysiologic response and discrimination deficits in children with learning problems. Science, 273, 971–973. Kraus, N., & Nicol, T. G. (2005). Brainstem origins for cortical ‘‘what’’ and ‘‘where’’ pathways in the auditory system. Trends in Neuroscience, 28, 176–181. Kremlacek, J., Kuba, M., Chlubnova, J., & Kubova, Z. (2004). Effect of stimulus localization on motion-onset VEP. Vision Research, 44, 2989–3000. Kremlacek, J., Kuba, M., Kubova, Z., & Langrova, J. (2006). Visual mismatch negativity elicited by magnocellular system activation. Vision Research, 46, 485–490. Kropotov, J. D., Naä¨ta¨nen, R., Sevostianov, A. V., Alho, K., Reinikainen, K., & Kropotova, O. V. (1995). Mismatch negativity to auditory stimulus change recorded directly from the human temporal cortex. Psychophysiology, 32, 418–422. Kujala, T., Tervaniemi, M., & Schro¨ger, E. (2007). The mismatch negativity in cognitive and clinical neuroscience: Theoretical and methodological considerations. Biological Psychology, 74, 1–19. Kushnerenko, E., Teinonen, T., Volein, A., & Csibra, G. (2008). Electrophysiological evidence of illusory audiovisual speech percept in human infants. Proceedings of the National Academy of Sciences USA., 105, 11442–11445. Ladinig, O., Honing, H., Ha´den, G., & Winkler, I. (2009). Probing attentive and pre–attentive emergent meter in adult listeners without extensive music training. Music Perception, 26, 377–386. Lang, H. A., Nyrke, T., Ek, M., Aaltonen, O., Raimo, I., & Naä¨ta¨nen, R. (1990). Pitch discrimination performance and auditory event-related potentials. In C. H. M. Brunia, A. W. K. Gaillard, A. Kok, G. Mulder, & M. N. Verbaten (Eds.), Psychophysiological brain research (Vol. 1, pp. 294–298). Tilburg, The Netherlands: Tilburg University Press. Leino, S., Brattico, E., Tervaniemi, M., & Vuust, P. (2007). Representation of harmony rules in the human brain: Further evidence from event-related potentials. Brain Research, 1142, 169–177. Leva¨nen, S., Ahonen, A., Hari, R., McEvoy, L., & Sams, M. (1996). Deviant auditory stimuli activate human left and right auditory cortex differently. Cerebral Cortex, 6, 288–296. Leva¨nen, S., Hari, R., McEvoy, L., & Sams, M. (1993). Responses of the human auditory cortex to changes in one versus two stimulus features. Experimental Brain Research, 97, 177–183. Lindsley, D. B. (1960). Attention, consciousness, sleep and wakefulness. In J. Field, et al. (Eds.) Handbook of Physiology, Neurophysiology (vol. III, Section 1, pp. 1553–1593). Washington, D.C.: American Physiological Society. Loui, P., Grent-t-Jong, T., Torpey, D., & Woldorff, M. (2005). Effects of attention on the neural processing of harmonic syntax in Western music. Cognitive Brain Research, 25, 678–687. Loveless, N. (1983). The orienting response and evoked potentials in man. In D. Siddle (Ed.) Orienting and habituation: Perspectives in human research (pp. 71–108). New York: John Wiley & Sons. Lyytinen, H., Blomberg, A. P., & Naä¨ta¨nen, R. (1992). Event-related potentials and autonomic responses to a change in unattended auditory stimuli. Psychophysiology, 29, 523–534. Lyytinen, H., & Naä¨ta¨nen, R. (1987). Autonomic and ERP responses to deviant stimuli: Analysis of covariation. In R. Johnson, J. W. Rohrbaugh, & R. Parasuraman (Eds.) Current Trends in EventRelated Brain Potential Research. Electroencephalography and Clinical Neurophysiology. Suppl., 40, 108–117. MacMillan, N. A. (1973). Detection and recognition of intensity changes in tone and noise: The detection-recognition disparity. Perception and Psychophysics, 13, 65–75.

R. Naä¨ta¨nen et al. Maess, B., Jacobsen, T., Schro¨ger, E., & Friederici, A. D. (2007). Localizing pre-attentive auditory memory-based comparison: Magnetic mismatch negativity to pitch change. NeuroImage, 37, 561–571. Massaro, D. W. (1970). Retroactive interference in short-term recognition memory for pitch. Journal of Experimental Psychology, 83, 32–39. Massaro, D. W. (1976). Auditory information processing. In W. K. Estes (Ed.), Handbook of learning and cognitive processes (pp. 275–320). Lawrence Erlbaum Associates, Hillsdale, NJ. May, P. J. C., & Tiitinen, H. (2009). Mismatch negativity (MMN), the deviance-elicited auditory deflection, explained. Psychophysiology, 46, 1–57. McCallum, W. C., Curry, S. H., Cooper, R., Pocock, P. V., & Papakostopoulos, D. (1983). Brain event-related potentials as indicators of early selective processes in auditory target localization. Psychophysiology, 20, 1–17. Menning, H., Zwitserlood, P., Scho¨ning, S., Hihn, H., Bo¨lte, J., Dobel, C., et al. (2005). Pre-attentive detection of syntactic and semantic errors. NeuroReport, 16, 77–80. Michie, P. T., Solowij, N., Crawford, J. M., & Glue, L. C. (1993). The effects of between-source discriminability on attended and unattended auditory ERPs. Psychophysiology, 30, 205–220. Mitterer, H., & Blomert, L. (2003). Coping with phonological assimilation in speech perception: Evidence for early compensation. Perception & Psychophysics, 65, 956–969. Molholm, S., Martinez, A., Ritter, W., Javitt, D. C., & Foxe, J. J. (2005). The neural circuitry of pre-attentive auditory change-detection: An fMRI study of pitch and duration mismatch negativity generators. Cerebral Cortex, 15, 545–551. Moore, B. C. J. (1989). An Introduction to the Psychology of Hearing. London: Academic Press. Moore, D. R. (2003). Cortical neurons signal sound novelty. Nature Neuroscience, 6, 330–332. Moray, N. (1959). Attention in dichotic listening: Affective cues and the influence of instruction. The Quarterly Journal of Experimental Psychology, 9, 56–60. Mo¨tto¨nen, R., Krause, C. M., Tiippana, K., & Sams, M. (2002). Processing of changes in visual speech in the human auditory cortex. Cognitive Brain Research, 13, 417–425. Mo¨tto¨nen, R., Schurman, M., & Sams, M. (2004). Time course of multisensory interactions during audiovisual speech perception in humans: A magnetoencephalographic study. Neuroscience Letters, 363, 112–115. Mu¨ller, D., Winkler, I., Roeber, U., Schaffer, S., Czigler, I., & Schro¨ger, E. (2010). Visual object representations can be formed outside the focus of voluntary attention: Evidence from event-related potentials. Journal of Cognitive Neuroscience, 22, 1179–1188. Mu¨nte, T. F., Altenmu¨ller, E., & Ja¨ncke, L. (2002). The musician’s brain as a model of plasticity. Nature Reviews Neuroscience, 3, 473–478. Naä¨ta¨nen, R. (1970). EEG, slow potential and evoked potential correlates of selective attention. In A. F. Sanders (Ed.), Attention and performance III. Acta Psychologica (33, 178–192). Naä¨ta¨nen, R. (1975). Selective attention and evoked potentials in humans - a critical review. Biological Psychology, 2, 237–307. Naä¨ta¨nen, R. (1982). Processing negativity: An evoked-potential reflection of selective attention. Psychological Bulletin, 92, 605–640. Naä¨ta¨nen, R. (1984). In search of a short-duration memory trace of a stimulus in the human brain. In L. Pulkkinen & P. Lyytinen (Eds.), Human action and personality. Essays in honour of Martti Takala (pp. 29–43). Jyva¨skyla¨: University of Jyva¨skyla¨. Naä¨ta¨nen, R. (1986). N2 and automatic vs. controlled processes: A classification of N2 kinds of ERP components. Electroencephalography and Clinical Neurophysiology, Suppl., 38, 169–171. Naä¨ta¨nen, R. (1990). The role of attention in auditory information processing as revealed by event-related potentials and other brain measures of cognitive function. The Behavioral and Brain Sciences, 13, 201–288. Naä¨ta¨nen, R. (1992). Attention and brain function. Hillsdale, NJ: Lawrence Erlbaum Associates. Naä¨ta¨nen, R. (2009). Somatosensory mismatch negativity: A new clinical tool for developmental neurological research? Developmental Medicine & Child Neurology, 51, 927–931. Naä¨ta¨nen, R., & Alho, K. (1995). Mismatch negativity: A unique measure of sensory processing in audition. International Journal of Neuroscience, 80, 317–337.

Auditory processing that leads to conscious perception Naä¨ta¨nen, R., & Alho, K. (1997). Mismatch negativity (MMN) - the measure for central sound representation accuracy. Audiology & Neuro-Otology, 2, 341–353. Naä¨ta¨nen, R., Astikainen, P., Ruusuvirta, T., & Huotilainen, M. (2010). Automatic auditory intelligence: An expression of the sensory-cognitive core of cognitive processes. Brain Research, 64, 123–136. Naä¨ta¨nen, R., & Gaillard, A. W. K. (1983). The orienting reflex and the N2 deflection of the event-related potential (ERP). In A. W. K. Gaillard & W. Ritter (Eds.), Tutorials in Event Related Potential Research: Endogenous components (pp. 119–141). Amsterdam: North Holland Publishing. Naä¨ta¨nen, R., Gaillard, A. W. K., & Ma¨ntysalo, S. (1978). Early selective-attention effect on evoked potential reinterpreted. Acta Psychologica, 42, 313–329. Naä¨ta¨nen, R., Gaillard, A. W. K., & Ma¨ntysalo, S. (1980). Brain potential correlates of voluntary and involuntary attention. In H. H. Kornhuber & L. Deecke (Eds.), Motivation, motor and sensory processes of the brain: Electrical potentials, behavior and clinical use. Progress in Brain Research, 54, 343–348. Amsterdam: Elsevier. Naä¨ta¨nen, R., Lehtokoski, A., Lennes, M., Cheour, M., Huotilainen, M., Iivonen, A., et al. (1997). Language-specific phoneme representations revealed by electric and magnetic brain responses. Nature, 385, 432–434. Naä¨ta¨nen, R., & Michie, P. T. (1979). Early selective attention effects on the evoked potential: A critical review and reinterpretation. Biological Psychology, 8, 81–136. Naä¨ta¨nen, R., Paavilainen, P., Alho, K., Reinikainen, K., & Sams, M. (1989). Do event-related potentials reveal the mechanism of the auditory sensory memory in the human brain? Neuroscience Letters, 98, 217–221. Naä¨ta¨nen, R., Paavilainen, P., Rinne, T., & Alho, K. (2007). The mismatch negativity (MMN) in basic research of central auditory processing: A review. Clinical Neurophysiology, 118, 2544–2590. Naä¨ta¨nen, R., & Picton, T. (1987). The N1 wave of the human electric and magnetic response to sound: A review and an analysis of the component structure. Psychophysiology, 24, 375–425. Naä¨ta¨nen, R., & Rinne, T. (2002). Electric brain response to sound repetition in humans: An index of long-term-memory-trace formation? Neuroscience Letters, 318, 49–51. Naä¨ta¨nen, R., Sams, M., Alho, K., Paavilainen, P., Reinikainen, K., & Sokolov, E. N. (1988). Frequency and location specificity of the human vertex N1 wave. Electroencephalography and Clinical Neurophysiology, 69, 523–531. Naä¨ta¨nen, R., Schro¨ger, E., & Alho, K. (2002). Electrophysiology of attention. In J. Wixted (Ed.), Stevens’ Handbook of Experimental Psychology (3rd edition, pp. 601–653). New York: John Wiley & Sons. Inc. Naä¨ta¨nen, R., Simpson, M., & Loveless, N. E. (1982). Stimulus deviance and evoked potentials. Biological Psychology, 14, 53–98. Naä¨ta¨nen, R., & Winkler, I. (1999). The concept of auditory stimulus representation in cognitive neuroscience. Psychological Bulletin, 6, 826–859. Nelken, I., & Ulanovsky, N. (2007). Mismatch negativity and stimulusspecific adaptation in animal models. Journal of Psychophysiology, 21, 214–223. Newstead, S. E., & Dennis, I. (1979). Lexical and grammatical processing of unshadowed messages: A reexamination of the McKay effect. Quarterly Journal of Experimental Psychology, 31, 477–488. Norman, D. A. (1968). Toward a theory of memory and attention. Psychological Review, 75, 522–536. Nousak, J. M. K., Deacon, D., Ritter, W., & Vaughan Jr., H. G. (1996). Storage of information in transient auditory memory. Cognitive Brain Research, 4, 305–317. Ocea´k, A., Winkler, I., Sussman, E., & Alho, K. (2006). Loudness summation and the mismatch negativity event related brain potential in humans. Psychophysiology, 43, 13–20. O¨hman, A. (1979). The orienting response, attention and learning: An information-processing perspective. In H. D. Kimmel, E. H. van Olst, & J. F. Orlebeke (Eds.), The orienting reflex in humans (pp. 443–471). Hillsdale, NJ: Lawrence Erlbaum Associates. Okita, T. (1979). Event-related potentials and selective attention to auditory stimuli varying in pitch and localization. Biological Psychology, 9, 271–284.

19 Okita, T., Konishi, K., & Inamori, R. (1983). Attention-related negative brain potential for speech words and pure tones. Biological Psychology, 16, 29–47. Onishi, S., & Davis, H. (1968). Effects of duration and rise time of tonebursts on evoked V-potentials. Journal of the Acoustical Society of America, 44, 582–591. Opitz, B., Schro¨ger, E., & von Cramon, D. Y. (2005). Sensory and cognitive mechanisms for preattentive change detection in auditory cortex. European Journal of Neuroscience, 21, 531–535. Ostroff, J. M., McDonald, K. L., Schneider, B. A., & Alain, C. (2003). Aging and the processing of sound duration in human auditory cortex. Hearing Research, 181, 1–7. Paavilainen, P., Alho, K., Reinikainen, K., Sams, M., & Naä¨ta¨nen, R. (1991). Right-hemisphere dominance of different mismatch negativities. Electroencephalography and Clinical Neurophysiology, 78, 466–479. Paavilainen, P., Araja¨rvi, P., & Takegata, R. (2007). Preattentive detection of nonsalient contingencies between auditory features. NeuroReport, 18, 159–163. Parasuraman, R. (1978). Auditory potentials and divided attention. Psychophysiology, 15, 460–465. Parasuraman, R., & Beatty, J. (1980). Brain events underlying detection and recognition of weak sensory signals. Science, 210, 80–83. Pasman, J. W., Rotteveel, J. J., Maassen, B., & Visco, Y. M. (1999). The maturation of auditory cortical evoked responses between (preterm) birth and 14 years of age. European Journal of Neurology, 3, 79–82. Pazo-Alvarez, P., Amenedo, E., & Cadaveira, F. (2004). Automatic detection of motion direction changes in the human brain. European Journal of Neuroscience, 19, 1978–1986. Pazo-Alvarez, P., Cadaveira, F., & Amenedo, E. (2003). MMN in visual modality: A review. Biological Psychology, 63, 199–236. Pedersen, C. B., & Salomon, G. (1977). Temporal integration of acoustic energy. Acta Otolaryngology, 83, 417–423. Pekkonen, E., Rinne, T., Reinikainen, K., Kujala, T., Alho, K., & Naä¨ta¨nen, R. (1996). Aging effects on auditory processing: An eventrelated potential study. Experimental Aging Research, 22, 171–184. Phillips, C. (2001). Levels of representation in the electrophysiology of speech perception. Cognitive Science, 25, 711–731. Picton, T. W., Goodman, W. S., & Bryce, D. P. (1970). Amplitude of evoked responses to tones of high intensity. Acta Otolaryngology, 70, 77–82. Picton, T. W., Hillyard, S. A., Krausz, H. I., & Galambos, R. (1974). Human auditory evoked potentials: I. Evaluation of components. Electroencephalography & Clinical Neurophysiology, 36, 179–190. Picton, T. W., Stapells, D. R., & Campbell, K. N. (1981). Auditory evoked potentials from the human cochlea and brainstem. Journal of Otolaryngology, 9(Suppl.), 1–41. Picton, T. W., Woods, D. L., & Proulx, G. B. (1978). Human auditory sustained potentials. II. Stimulus relationships. Electroencephalography & Clinical Neurophysiology, 45, 198–210. Pincze, Z., Lakatos, P., Rajkai, C., Ulbert, I., & Karmos, G. (2001). Separation of mismatch negativity and the N1 wave in the auditory cortex of the cat: A topographic study. Clinical Neurophysiology, 112, 778–784. Pincze, Z., Lakatos, P., Rajkai, C., Ulbert, I., & Karmos, G. (2002). Effect of deviant probability and interstimulus/interdeviant interval on the auditory N1 and mismatch negativity in the cat auditory cortex. Cognitive Brain Research, 13, 249–253. Ponton, C. W., Don, M., Eggermont, J. J., Waring, M. D., Kwong, B., Cunningham, J., & Trautwein, P. (2000). Maturation of the mismatch negativity: Effects of profound deafness and cochlear implant use. Audiology & Neuro-Otology, 5, 167–185. Ponton, C., Eggermont, J. J., Khosla, D., Kwong, B., & Don, M. (2002). Maturation of human central auditory system activity: Separating auditory evoked potentials by dipole source modeling. Clinical Neurophysiology, 113, 407–420. Ponton, C., Eggermont, J. J., Kwong, B., & Don, M. (2000a). Maturation of human central auditory system activity: Evidence from multi-channel evoked potentials. Clinical Neurophysiology, 111, 220–236. Posner, M. I., & Snyder, C. R. R. (1975). Attention and cognitive control. In R. L. Solso (Ed.), Information processing and cognition: The Loyola Symposium (pp. 205–223). New Jersey: Erlbaum. Pratt, H., & Sohmer, H. (1977). Correlations between psychophysical magnitude estimates and simultaneously obtained auditory nerve,

20 brain stem and cortical responses to click stimuli in man. Electroencephalography & Clinical Neurophysiology, 43, 802–812. Pulvermu¨ller, F. (2001). Brain reflections of words and their meaning. Trends in Cognitive Sciences, 5, 517–524. Pulvermu¨ller, F., & Assadollahi, R. (2007). Grammar or serial order?: Discrete combinatorial brain mechanisms reflected by the syntactic mismatch negativity. Journal of Cognitive Neuroscience, 19, 971–980. Pulvermu¨ller, F., & Knoblauch, A. (2009). Discrete combinatorial circuits emerging in neural networks: A mechanism for rules grammar in the human brain? Neural Networks, 22, 161–172. Pulvermu¨ller, F., Kujala, T., Shtyrov, Y., Simola, J., Tiitinen, H., Alku, P., et al. (2001). Memory traces for words as revealed by the mismatch negativity. NeuroImage, 14, 607–616. Pulvermu¨ller, F., & Shtyrov, Y. (2003). Automatic processing of grammar in the human brain as revealed by the mismatch negativity. NeuroImage, 20, 159–172. Pulvermu¨ller, F., & Shtyrov, Y. (2006). Language outside the focus of attention: The mismatch negativity as a tool for studying higher cognitive processes. Progress in Neurobiology, 79, 49–71. Pulvermu¨ller, F., Shtyrov, Y., & Hauk, O. (2009). Understanding in an instant: Neurophysiological evidence for mechanistic language circuits in the brain. Brain & Language, 110, 81–94. Rinne, T., Alho, K., Ilmoniemi, R. J., Virtanen, J., & Naä¨ta¨nen, R. (2000). Separate time behaviors of the temporal and frontal MMN sources. NeuroImage, 12, 14–19. Rinne, T., Balk, M. H., Koistinen, S., Autti, T., Alho, K., & Sams, M. (2008). Auditory selective attention modulates activation of human inferior collicus. Journal of Neurophysiology, 100, 3323–3327. Rinne, T., Gratton, G., Fabiani, M., Cowan, N., Maclin, E., Stinard, A., et al. (1999). Scalp-recorded optical signals make sound processing in the auditory cortex visible. NeuroImage, 10, 620–624. Rinne, T., Sa¨rkka¨, A., Degerman, A., Schro¨ger, E., & Alho, K. (2006). Two separate mechanisms underlie auditory change detection and involuntary control of attention. Brain Research, 1077, 135–143. Ritter, W., Deacon, D., Gomes, H., Javitt, D. C., & Vaughan Jr., H. G. (1995). The mismatch negativity of event-related potentials as a probe of transient auditory memory: A review. Ear and Hearing, 16, 52–67. Ritter, W., Sussman, E., & Molholm, S. (2000). Evidence that the mismatch negativity system works on the basis of objects. NeuroReport, 11, 61–63. Rosburg, T. (2003). Left hemispheric dipole locations of the neuromagnetic mismatch negativity to frequency, intensity and duration deviants. Cognitive Brain Research, 16, 83–90. Rosburg, T., Haueisen, J., & Kreitschmann-Andermahr, I. (2004). The dipole location shift within the auditory evoked neuromagnetic field components N100m and mismatch negativity (MMNm). Clinical Neurophysiology, 115, 906–913. Rossi, S., Gugler, M. F., Friederici, A. D., & Hahne, A. (2006). The impact of proficiency on syntactic second-language processing of German and Italian: Evidence from event-related potentials. Journal of Cognitive Neuroscience, 18, 2030–2048. Sams, M., Ha¨ma¨laïnen, M., Antervo, A., Kaukoranta, E., Reinikainen, K., & Hari, R. (1985). Cerebral neuromagnetic responses evoked by short auditory stimuli. Electroencephalography and Clinical Neurophysiology, 61, 254–266. Sams, M., Hari, R., Rif, J., & Knuutila, J. (1993). The human auditory sensory memory trace persists about 10 sec: Neuromagnetic evidence. Journal of Cognitive Neuroscience, 5, 363–370. Sams, M., Kaukoranta, E., Ha¨ma¨laïnen, M., & Naä¨ta¨nen, R. (1991). Cortical activity elicited by changes in auditory stimuli: Different sources for the magnetic N100m and mismatch responses. Psychophysiology, 28, 21–29. Scharf, B. (1978). Loudness. In E. C. Carterette & N. P. Friedman (Eds.), Handbook of perception, vol. IV, Hearing (pp. 187–242). New York: Academic Press. Scharf, B., & Houtsma, A. J. (1986). Audition II. Loudness, pitch, localization, aural disortion, pathology. In K. R. Boff, L. Kaufman, & J. P. Thomas (Eds.), Handbook of perception and human performance: vol. 1. Sensory processes and perception (pp. 15.1–60). New York: Wiley.

R. Naä¨ta¨nen et al. Scherg, M., Vajsar, J., & Picton, T. W. (1989). A source analysis of the late human auditory evoked potentials. Journal of Cognitive Neuroscience, 1, 336–355. Schro¨ger, E. (1996). A neural mechanism for involuntary attention shifts to changes in auditory stimulation. Journal of Cognitive Neuroscience, 8, 527–539. Schro¨ger, E. (1997). On the detection of auditory deviants: A pre-attentive activation model. Psychophysiology, 34, 245–257. Schro¨ger, E., Bendixen, A., Trujillo-Barreto, N. J., & Roeber, U. (2007). Processing of abstract rule violations in audition. PLoS One, 11, e1131. Schro¨ger, E., Naä¨ta¨nen, R., & Paavilainen, P. (1992). Event-related potentials reveal how non-attended complex sound patterns are represented by the human brain. Neuroscience Letters, 146, 183–186. Sculthorpe, L. D., Ouellet, D. R., & Campbell, K. B. (2009). MMN elicitation during natural sleep to violations of an auditory pattern. Brain Research, 1290, 52–62. Sharma, A., & Dorman, M. F. (2000). Neurophysiologic correlates of cross-language phonetic perception. Journal of the Acoustical Society of America, 107, 2697–2703. Sharma, A., Kraus, N., McGee, T. J., & Nicol, T. G. (1997). Developmental changes in P1 and N1 central auditory responses elicited by consonant-vowel syllables. Electroencephalography and Clinical Neurophysiology, 104, 540–545. Shestakova, A., Brattico, E., Huotilainen, M., Galunov, V., Soloviev, A., Sams, M., et al. (2002). Abstract phoneme representations in the left temporal cortex: Magnetic mismatch negativity study. NeuroReport, 13, 1813–1816. Shinozaki, N., Yabe, H., Sato, Y., Hiruma, T., Sutoh, T., Matsuoka, T., & Kaneko, S. (2003). Spectrotemporal window of integration of auditory information in the human brain. Cognitive Brain Research, 17, 563–571. Shtyrov, Y., Pulvermu¨ller, F., Naä¨ta¨nen, R., & Ilmoniemi, R. J. (2003). Grammar processing outside the focus of attention: An MEG study. Journal of Cognitive Neuroscience, 15, 1195–1206. Simson, R., Vaughan Jr., H. G., & Ritter, W. (1976). The scalp topography of potentials associated with missing visual or auditory stimuli. Electroencephalography and Clinical Neurophysiology, 40, 33–42. Simson, R., Vaughan Jr., H. G., & Ritter, W. (1977). The scalp topography of potentials in auditory and visual discrimination tasks. Electroencephalography and Clinical Neurophysiology, 42, 528–535. Snyder, E., & Hillyard, S. A. (1976). Long latency evoked potentials to irrelevant, deviant stimuli. Behavioural Biology, 16, 319–331. Sokolov, E. N., Spinks, J. A., Naä¨ta¨nen, R., & Lyytinen, H. (2002). The orienting response in information processing. Mahwah, NJ: Erlbaum. Squires, K. C., Wickens, C., Squires, N. K., & Donchin, E. (1976). The effect of stimulus sequence on the waveform of the cortical eventrelated potential. Science, 193, 1142–1146. Squires, N. K., Squires, K. C., & Hillyard, S. A. (1975). Two varieties of long-latency positive waves evoked by unpredictable auditory stimuli in man. Electroencephalography and Clinical Neurophysiology, 38, 387–401. Starr, A., & Don, M. (1988). Brain potentials evoked by acoustic stimuli. In T. W. Picton (Ed.), Human event-related potentials (Vol. 3, pp. 97– 157). Amsterdam: Elsevier. Stefanics, G., Ha´den, G. P., Sziller, I., Bala´zs, L., Beke, A., & Winkler, I. (2009). Newborn infants process pitch intervals. Clinical Neurophysiology, 120, 304–308. Stuss, D. T., & Benson, D. F. (1986). The frontal lobes. New York: Raven Press. Sussman, E. S. (2007). A new view on the MMN and attention debate: The role of context in processing auditory events. Journal of Psychophysiology, 21, 164–175. Sutton, S., Braren, M., Zubin, J., & John, E. R. (1965). Evoked-potential correlates of stimulus uncertainty. Science, 150, 1187–1188. Sysoeva, O., Takegata, R., & Naä¨ta¨nen, R. (2006). Pre-attentive representation of sound duration in the human brain. Psychophysiology, 43, 272–276. Takegata, R., Brattico, E., Tervaniemi, M., Varyagina, O., Naä¨ta¨nen, R., & Winkler, I. (2005). Preattentive representation of feature conjunctions for simultaneous, spatially distributed auditory objects. Cognitive Brain Research, 25, 169–179. Takegata, R., Huotilainen, M., Rinne, T., Naä¨ta¨nen, R., & Winkler, I. (2001). Changes in acoustic features and their conjunctions are processed by separate neuronal populations. NeuroReport, 12, 525–529.

Auditory processing that leads to conscious perception Takegata, R., & Morotomi, T. (1999). Integrated neural representation of sound and temporal features in human auditory sensory memory: An event-related potential study. Neuroscience Letters, 274, 29 207– 29 210. Takegata, R., Paavilainen, P., Naä¨ta¨nen, R., & Winkler, I. (1999). Independent processing of changes in auditory single features and feature conjunctions in humans as indexed by the mismatch negativity (MMN). Neuroscience Letters, 266, 109–112. Tervaniemi, M., & Brattico, E. (2004). From sounds to music. Towards understanding the neurocognition of musical sound perception. Journal of Consciousness Studies, 11, 9–27. Tervaniemi, M., Castaneda, A., Knoll, M., & Uther, M. (2006). Sound processing in amateur musicians and nonmusicians: Event-related potential and behavioral indices. NeuroReport, 17, 1225–1228. Tervaniemi, M., Maury, S., & Naä¨ta¨nen, R. (1994). Neural representations of abstract stimulus features in the human brain as reflected by the mismatch negativity. NeuroReport, 5, 844–846. Tervaniemi, M., Rytko¨nen, M., Schro¨ger, E., Ilmoniemi, R. J., & Naä¨ta¨nen, R. (2001). Superior formation of cortical memory traces for melodic patterns in musicians. Learning & Memory, 8, 295–300. Tervaniemi, M., Saarinen, J., Paavilainen, P., Danilova, N., & Naä¨ta¨nen, R. (1994). Temporal integration of auditory information in sensory memory as reflected by the mismatch negativity. Biological Psychology, 38, 157–167. Tiippana, K., Andersen, T. S., & Sams, M. (2004). Visual attention modulates audiovisual speech perception. European Journal of Cognitive Psychology, 16, 457–472. Tiitinen, H., Alho, K., Huotilainen, M., Ilmoniemi, R. J., Simola, J., & Naä¨ta¨nen, R. (1993). Tonotopic auditory cortex and the magnetoencephalographic (MEG) equivalent of the mismatch negativity. Psychophysiology, 30, 537–540. Tiitinen, H., May, P., Reinikainen, K., & Naä¨ta¨nen, R. (1994). Attentive novelty detection in humans is governed by pre-attentive sensory memory. Nature, 370, 90–92. Todd, J., Myers, R., Pirillo, R., & Drysdale, K. (2010). Neuropsychological correlates of auditory perceptual inference: A mismatch negativity (MMN) study. Brain Research, 1310, 113–123. Trainor, L. J., McDonald, K. L., & Alain, C. (2002). Automatic and controlled processing of melodic contour and interval information measured by electrical brain activity. Journal of Cognitive Neuroscience, 14, 1–13. Treisman, A. M. (1960). Contextual cues in selective listening. The Quarterly Journal of Experimental Psychology, 12, 242–248. Tse, C.-Y., & Penney, T. B. (2008). On the functional role of temporal and frontal cortex activation in passive detection of auditory deviance. NeuroImage, 41, 1462–1470. Ulanovsky, N., Las, L., & Nelken, I. (2003). Processing of low-probability sounds by cortical neurons. Nature Neuroscience, 6, 391–398. Umbricht, D., Schmid, L., Koller, R., Vollenweider, F. X., Hell, D., & Javitt, D. C. (2000). Ketamine-induced deficits in auditory and visual context-dependent processing in healthy volunteers. Archives of General Psychiatry, 57, 1139–1147. Valtonen, J., May, P., Ma¨kinen, V., & Tiitinen, H. (2003). Visual short-term memory load affects sensory processing of irrelevant sounds in human auditory cortex. Cognitive Brain Research, 17, 358–367. van Zuijen, T., Simoens, V. L., Paavilainen, P., Naä¨ta¨nen, R., & Tervaniemi, M. (2006). Implicit, intuitive, and explicit knowledge of abstract regularities in a sound sequence: An event-related brain potential study. Journal of Cognitive Neuroscience, 18, 1292–1303. Vaughan, H. G. Jr., & Arezzo, J. C. (1988). The neural basis of eventrelated potentials. In T. W. Picton (Ed.), Human event-related potentials (Vol. 3, pp. 45–96). Amsterdam: Elsevier. Vogel, E. K., & Luck, S. J. (2000). The visual N1 component as an index of a discrimination process. Psychophysiology, 37, 190–203. Behrens, von der W., Baüerle, P., Ko¨ssl, M., & Gaese, B. H. (2009). Corrrelating stimulus-specific adaptation of cortical neurons and local field potentials in the awake rat. The Journal of Neuroscience, 29, 13837–13849. Walter, W. G. (1964). The convergence and interaction of visual, auditory and tactile responses in human non-specific cortex. Annals of the New York Academy of Science, 112, 320–361.

21 Wang, W., Datta, H., & Sussman, E. (2005). The development of the length of the temporal window of integration for rapidly presented auditory information as indexed by MMN. Clinical Neurophysiology, 116, 1695–1706. Widmann, A., Kujala, T., Tervaniemi, M., Kujala, A., & Schro¨ger, E. (2004). From symbols to sounds: Visual symbolic information activates sound representations. Psychophysiology, 41, 709–715. Winkler, I. (2007). Interpreting the mismatch negativity. Journal of Psychophysiology, 21, 147–163. Winkler, I., & Cowan, N. (2005). From sensory to long–term memory: Evidence from auditory memory reactivation studies. Experimental Psychology, 52, 3–20. Winkler, I., Cowan, N., Cse´pe, V., Czigler, I., & Naä¨ta¨nen, R. (1996). Interactions between transient and long-term auditory memory as reflected by the mismatch negativity. Journal of Cognitive Neuroscience, 8, 403–415. Winkler, I., Czigler, I., Sussman, E., Horva´th, J., & Bala´zs, L. (2005). Preattentive binding of auditory and visual stimulus features. Journal of Cognitive Neuroscience, 17, 320–339. Winkler, I., Denham, S. L., & Nelken, I. (2009a). Modeling the auditory scene: Predictive regularity representations and perceptual objects. Trends in Cognitive Sciences, doi:10.1016/j.tics.2009.09.003. Winkler, I., Ha´den, G. P., Ladinig, O., Sziller, I., & Honing, H. (2009b). Newborn infants detect the beat in music. Proceedings of the National Academy of Sciences USA, 106, 2468–2471. Winkler, I., Horva´th, J., Weisz, J., & Trejo, L. (2009c). Deviance detection in congruent audiovisual speech: Evidence for implicit integrated audiovisual memory representations. Biological Psychology, 82, 281–292. Winkler, I., Karmos, G., & Naä¨ta¨nen, R. (1996). Adaptive modeling of the unattended acoustic environment reflected in the mismatch negativity event-related potential. Brain Research, 742, 239–252. Winkler, I., Kujala, T., Tiitinen, H., Sivonen, P., Alku, P., Lehtokoski, A., et al. (1999). Brain responses reveal the learning of foreign language phonemes. Psychophysiology, 36, 638–642. Winkler, I., & Naä¨ta¨nen, R. (1993). Event-related brain potentials to infrequent partial omissions in series of auditory stimuli. In H.-J. Heinze, G. R. Mangun, & T. F. Mu¨nte (Eds.), New developments in event-related potentials (pp. 219–226). Boston-Basel-Berlin: Birkhaüser. Winkler, I., & Naä¨ta¨nen, R. (1994). The effects of auditory backward masking on event-related brain potentials. Electroencephalography and Clinical Neurophysiology, 44, 185–189. Winkler, I., Paavilainen, P., & Naä¨ta¨nen, R. (1992). Can echoic memory store two traces simultaneously? A study of event-related brain potentials. Psychophysiology, 29, 337–349. Winkler, I., Reinikainen, K., & Naä¨ta¨nen, R. (1993). Event related brain potentials reflect traces of the echoic memory in humans. Perception & Psychophysics, 53, 443–449. Winkler, I., Tervaniemi, M., Huotilainen, M., Ilmoniemi, R., Ahonen, A., Salonen, O., et al. (1995). From objective to subjective: Pitch representation in the human auditory cortex. NeuroReport, 6, 2317–2320. Winkler, I., Tervaniemi, M., & Naä¨ta¨nen, R. (1997). Two separate codes for missing-fundamental pitch in the human auditory cortex. Journal of the Acoustical Society of America, 102, 1072–1082. Woldorff, M. G., Hackley, S. A., & Hillyard, S. A. (1991). The effects of channel-selective attention on the mismatch negativity wave elicited by deviant tones. Psychophysiology, 28, 30–42. Woldorff, M. G., Hansen, J. C., & Hillyard, S. A. (1987). Evidence for effects of selective attention in the mid-latency range of the human auditory event-related potential. In R. Johnson Jr., J. W. Rohrbaugh, & R. Parasuraman (Eds.), Current trends in event-related brain potential research (Suppl. 40 to Electroencephalography and Clinical Neurophysiology; pp. 146–154). Amsterdam: Elsevier. Woldorff, M. G., Hillyard, S. A., Gallen, C. C., Hampson, S. R., & Bloom, F. E. (1998). Magnetoencephalographic recordings demonstrate attentional modulation of mismatch-related neural activity in human auditory cortex. Psychophysiology, 35, 283–292. Woldorff, M. G., & Hillyard, S. A. (1991). Modulation of early auditory processing during selective listening to rapidly presented tones. Electroencephalography and Clinical Neurophysiology, 79, 170–191.

22 Woods, D. L., & Elmasian, R. (1986). The habituation of event-related potentials to speech sounds and tones. Electroencephalography & Clinical Neurophysiology, 65, 447–459. Yabe, H., Koyoma, S., Kakigi, R., Gunji, A., Tervaniemi, M., Sato, Y., & Kaneko, S. (2001). Automatic discriminative sensitivity inside temporal window of sensory memory as a function of time. Cognitive Brain Research, 12, 39–48. Yabe, H., Matsuoka, T., Sato, Y., Hiruma, T., Sutoh, T., Koyama, S., et al. (2005). Time may be compressed in sound representation as replicated in sensory memory. NeuroReport, 16, 95–98. Yabe, H., Sutoh, T., Matsuoka, T., Asai, R., Hiruma, T., Sato, Y., et al. (2005). Transient gamma-band response is dissociated from sensory memory as reflected by MMN. Neuroscience Letters, 380, 80–82. Yabe, H., Tervaniemi, M., Reinikainen, K., & Naä¨ta¨nen, R. (1997). Temporal window of integration revealed by MMN to sound omission. NeuroReport, 8, 1971–1974.

R. Naä¨ta¨nen et al. Yabe, H., Tervaniemi, M., Sinkkonen, J., Huotilainen, M., Ilmoniemi, R. J., & Naä¨ta¨nen, R. (1998). Temporal window of integration of auditory information in the human brain. Psychophysiology, 35, 615–619. Yabe, H., Winkler, I., Czigler, I., Koyama, S., Kakigi, R., Suto, T., et al. (2001). Organizing sound sequences in the human brain: the interplay of auditory streaming and temporal integration. Brain Research, 897, 222–227. Yago, E., Escera, C., Alho, K., & Giard, M.-H. 2001. Cerebral mechanisms underlying orienting of attention towards auditory frequency changes. NeuroReport, 12, 2583–7. Zwicker, E., & Fastl, H. (1990). Psychoacoustics: Facts and models. Berlin: Springer-Verlag. (Received December 17, 2009; Accepted June 7, 2010)


Change-related responses in the human auditory cortex: An MEG study

KOYA YAMASHIRO,a KOJI INUI,a,b NAOFUMI OTSURU,a,b and RYUSUKE KAKIGIa,b a

Department of Integrative Physiology, National Institute for Physiological Sciences, Okazaki, Japan Department of Physiological Sciences, School of Life Sciences, Graduate University for Advanced Studies, Hayama, Japan

b

Abstract We recorded cortical activity in response to the onset, offset, and frequency change of a pure tone using magnetencephalograms (MEGs) to clarify the physiological significance of N1m relating to the detection of changes. Four interstimulus intervals (ISIs) (0.5, 1.5, 3, and 6 s) were used for each of the three auditory events. Results showed that (i) all three auditory events elicited N1m with a similar topography and similar temporal profile, (ii) the source of N1m was located in the superior temporal gyrus (STG) for all events under all ISI conditions, (iii) the amplitude of the STG activity as a function of the duration of the steady state preceding the change was similar among the three events, and (iv) there was a significant positive correlation in amplitude between on-N1m and off-N1m and between on-N1m and change-N1m. These results suggested that N1m for the three events has a similar physiological significance relating to the detection of changes. Descriptors: N1m, ISIs, Auditory change detection

for detecting changes was automatically activated by comparing the new event (on or off) with the preceding condition (silent or continuous stimuli) using memory trace. Thus, if the N1m components are elicited by a memory trace by comparing any change with the steady state preceding the change’s occurrence, they will be larger with a longer preceding steady state. In fact, the amplitude of auditory N1m (Hari, Kaila, Katila, Tuomisto, & Varpula, 1982; Sams et al., 1993) and the activity in the superior temporal gyrus (STG) (Howard et al., 2000; Tanaka et al., 2008) are sensitive to the stimulus rate in the auditory modality. That is, its amplitude increases with a decrease in the stimulus rate. Some previous studies (Hari et al., 1987; Hillyard & Picton, 1978; Pfefferbaum, Buchsbaum, & Gips, 1971) also showed that the amplitude of N1m (N1) in response to both the onset (ON-N1) and offset (OFF-N1) of a sound stimulus was dependent on the duration of preceding intervals (sound or silent). Given that the ON- and OFF-N1 components are a response to auditory changes, this finding is quite natural since a longer preceding interval indicates an abrupt break of a longer silent period (ON) or longer sound (OFF). The idea that N1 is an automatic response to an auditory change can also explain why only a single N1 component of a similar duration is evoked by various types of auditory stimuli (brief, long, continuous, and repetitive). Previous studies with magnetoencephalograms (MEGs) (Hari et al., 1987; Noda et al., 1998; Pantev et al., 1996; Yamashiro et al., 2009) showed the time course of activation, location, and orientation to be very similar between the activities for ON-N1m and OFF-N1m. Therefore, we speculate that they originate from a similar group of neurons, or even identical neurons, sensitive to auditory changes. Supporting this idea, a study

One of the most important functions of sensory processing in animals is to quickly detect and respond to changes or new events in the surrounding environment. Automatic shifts of attention to an event lead to the facilitation of subsequent processes to execute appropriate behavior. In humans, a cortical network sensitive to sensory changes of various modalities is known (Downar, Crawley, Mikulis, & Davis, 2000; Tanaka, Kida, Inui, & Kakigi, 2009). Previous studies (Hari et al., 1987; Noda et al., 1998; Pantev, Eulitz, Hampson, Ross, & Roberts, 1996; Yamashiro, Inui, Otsuru, Kida, & Kakigi, 2009) showed that auditory on- and offevents elicit similar components peaking at around 100 ms, that is, N1m. The N1m (N1) component is elicited by various types of sounds including pure tone (Pantev et al., 1996; Sams, Hari, Rif, & Knuutila, 1993), clicks (Joutsiniemi, Hari, & Vilkman, 1989; Picton, Hillyard, Krausz, & Galambos, 1974), the human voice (Hari & Lounasmaa, 1989), animal sounds (Altmann et al., 2008), and the offset of a sound (Hari et al., 1987; Noda et al., 1998; Pantev et al., 1996; Yamashiro et al., 2009). Therefore, N1m seems to be a result of any abrupt auditory event. In addition to these findings, our previous study (Yamashiro et al., 2009) showed that the latency of N1m for the off-response could be determined precisely using the offset-discriminating point (ODP; the latency of the last pulse plus the interstimulus interval). These results suggested that a similar cortical network We are very grateful to Mr. Y. Takeshima for technical help during this study. Address correspondence to: Koya Yamashiro, Department of Integrative Physiology, National Institute for Physiological Sciences, Okazaki 444-8585, Japan. E-mail: [email protected] 23

24 using an abrupt change in frequency from 988 to 1108 Hz found a larger N1m than the control N1m elicited by brief (100 ms) discrete deviant stimuli of 1108 Hz (Lavikainen, Huotilainen, Ilmoniemi, Simola, & Naä¨ta¨nen, 1995). Although results of numerous auditory studies (Altmann et al., 2008; Hari & Lounasmaa, 1989; Hari et al., 1987; Joutsiniemi et al., 1989; Lavikainen et al., 1995; Noda et al., 1998; Pantev et al., 1996; Picton et al., 1974; Yamashiro et al., 2009) appear to support the notion that a similar N1m component is elicited by any abrupt auditory change, whether this indicates an indicator of change-detection based on sensory memory remains to be elucidated. The present study aimed to clarify the physiological significance of the N1m component to the detection of changes. For this purpose, we recorded auditory evoked magnetic fields and compared the N1m components elicited by three different auditory events (onset, offset, and frequency change) under four different interstimulus interval (ISI) conditions (0.5 ! 6 s).

Methods Subjects Three experiments were performed on twelve (three females and nine males for Experiments 1 and 2) and eleven (two females and nine males for Experiment 3) normal hearing, right-handed volunteers (25–45 years). The same subjects took part in all three experiments. The study was in accordance with the declaration of Helsinki and approved in advance by the Ethics Committee of the National Institute for Physiological Sciences, Okazaki, Japan, and written informed consent was obtained from all the subjects. Auditory Stimulation and Paradigm Auditory evoked magnetic fields (AEFs) were elicited with a pure tone presented binaurally through a plastic tube and ear pieces (E-A-Rtone 3A, Aero Company, Indianapolis, IN). The intensity of the pure tone was adjusted to 60 db above the threshold for each subject. In Experiment 1 (ON), AEFs were elicited by the onset of a 1000 Hz pure tone (duration of 300 ms including 5 ms rise and fall times) presented at four different ISIs, 0.5, 1.5, 3, and 6 s. That is, the onset of the sound was preceded by a silence lasting 0.5, 1.5, 3, and 6 s in each ISI condition. In Experiment 2 (OFF), AEFs elicited by the offset of the sound were recorded by replacing the tone and silence in Experiment 1 (Figure 1). That is, a 1000 Hz pure tone 0.5. 1.5, 3, or 6 s in duration was presented with a subsequent silent blank of 300 ms, and AEFs were recorded using the offset point as a trigger. In this experiment, therefore, the offset of the sound was preceded by a 0.5, 1.5, 3, or 6 s sound. In Experiment 3 (CHANGE), AEFs elicited by an abrupt change in sound frequency were recorded by replacing the 300 ms of silence in Experiment 2 with a 1100 Hz pure tone 300 ms in duration (including 5 ms rise and fall times). As in Experiments 1 and 2, the duration of the 1000 Hz tone prior to the change to the 1100 Hz tone was either 0.5, 1.5, 3, or 6 s. In all experiments, AEFs for different ISIs were recorded in separate sessions. The sessions with four different ISIs were performed in random order for each subject. All the experiments were conducted on a single day. MEG Recording and Analysis The experiments were carried out in a magnetically shielded room. Subjects were instructed to watch a silent movie through-

K. Yamashiro et al. onset

ON

ISI offset

OFF

ISI 1000 Hz tone change

CHANGE

ISI 1100 Hz tone

Figure 1. Stimulation paradigm. Trigger points are shown by arrows.

out the experiment. AEFs were recorded with a helmet-shaped 306-channel MEG system (Vector-view, ELEKTA Neuromag, Helsinki, Finland), which comprised 102 identical triple sensor elements. Each sensor element consisted of two orthogonal planar gradiometers and one magnetometer coupled to a multi-superconducting quantum interference device (SQUID), and thus provided 3 independent measurements of the magnetic fields. In this study, we analyzed MEG signals recorded from 204 planartype gradiometers. These planar gradiometers can detect the largest signal just over local cerebral sources. The signals were recorded with a bandpass of 0.1–200 Hz and digitized at 997 Hz. The period of analysis for the on-, off-, and change-responses was 350 ms, including a period of 50 ms before the trigger that was used as the baseline. Trials with noise (42700 fT/cm) were rejected from the analysis automatically. For each on-, off-, and change-response, 100 artifact-free trials were recorded in each ISI condition. The average data was filtered with a 1–50 Hz bandpass filter and then used for the analysis (Yabe et al., 2004, 2005). To identify sources of the evoked activities, the equivalent current dipole (ECD), which best explains the measured data, was computed by using a least-squares search. A subset of 14–20 channels including the local signal maxima was used to estimate ECDs (Forss & Jousmaki, 1998; Nakata, Inui, Wasaka, Akatsuka, & Kakigi, 2005; Wasaka et al., 2005). These calculations gave the three-dimensional (3D) location, orientation, and strength of the ECD in a spherical conductor model, which was based on each subject’s magnetic resonance imaging (MRI) to show the source’s location. The goodness-of-fit value of an ECD was calculated to indicate in percentage terms how much the dipole accounted for the measured field variance. A single dipole can explain activity from several dipolar sources with a goodness of fit of 80%. Only ECDs explaining more than 80% of the field variance for selected periods of time were used for further analysis. The period of analysis was extended to the entire time period, and all channels were taken into account when computing a time-varying multi-dipole model. The strength of the previously found ECDs was allowed to change, while locations and orientations were kept fixed. The data acquisition and analysis followed Ha¨ma¨laïnen, Hari, Illmoniemi, Knuutila, and Lounasmaa (1993). MRI scans were obtained from all subjects with a 3.0-T MRI system (Allegra; Siemens, Erlangen, Germany). T1-weighted coronal, axial, and sagittal image slices obtained every 1.5 mm were used to render the 3D reconstruction

Change-related responses in the human auditory cortex

25

of the brain’s surface. Prior to the recording, a current was fed to four head position indicator (HPI) coils placed at known sites to obtain the exact location of the head with respect to the sensor, and the resulting magnetic fields were measured with the magnetometer, which allowed for aligning the individual head coordinate systems with the magnetometer coordinate system. The four HPI coils attached to the subject’s head were measured with respect to the three anatomical landmarks using a 3D digitizer to allow alignment of the MEG and MRI. The x-axis was fixed with the preauricular points, the positive direction being to the right. The positive y-axis passed through the nasion and the z-axis thus pointed upward. The peak latency and peak amplitude of each cortical activity were subjected to a three-way repeated measure analysis of variance (ANOVA) (event ! hemisphere ! ISI) between ON and OFF and between ON and CHANGE. The main focus of this study was ON-N1m. Therefore, we compared ON-N1m and OFF-N1m, and ON-N1m and CH-N1m separately. The Greenhouse-Geisser epsilon was used to correct the degrees of freedom.

A

The statistical significance of the source’s location was assessed by a discriminant analysis using x, y, and z coordinates as variables for each condition. The relationship of the amplitude of the ECD between ON and OFF across all subjects was assessed under three conditions (ISI: 1.5, 3, and 6 s) by determining a Pearson product-moment correlation coefficient, r. The relationship in amplitude between ON and CHANGE across ten subjects was assessed under four conditions (ISI: 0.5, 1.5, 3, and 6 s), again by determining a Pearson product-moment correlation coefficient, r.

Results On- and Off-Responses Figure 2A shows superimposed waveforms of on- and offresponses at an ISI of 6 s in a representative subject. Both the onset and offset of the tone elicited a magnetic component peaking at around 100 ms (ON-N1m and OFF-N1m) in the temporal

ON vs OFF

ON vs CHANGE

a’

a

b’

b

100ft / mm 100ms a

b

a’

ON

OFF

b’

CHANGE

B

ISI 0.5 S

ISI 1.5 S

ISI 3.0 S

ISI 6.0 S

Figure 2. Magnetic responses to the onset (ON), offset (OFF), and frequency change (CHANGE) of a pure tone. Data from a representative subject. (A) The top view trace of all sensors. (B) Location of dipoles superimposed on the subject’s own MR images.

26

K. Yamashiro et al.

area of each hemisphere. Clear N1m components were evoked in all the subjects in all four ISI conditions, except for the 0.5 s condition of OFF. The dipoles responsible for these responses were estimated to be located in the STG of both hemispheres. The location of the source did not differ significantly between ON and OFF (Table 1, Figure 2B). In a comparison between ON and OFF, a three-way ANOVA indicated the ISI (F(2,22) 5 15.6, po.01, e 5 0.87) to be a significant factor determining the peak latency of the STG activities. That is, the peak latency increased with an increase in ISI (Table 2). Although the peak latency was shorter for the offresponse than the on-response, the difference only tended to be significant (event factor, F(1,11) 5 4.6, p 5 .056). As for peak amplitude, results of the ANOVA indicated event (F(1,11) 5 127.3, po.001, e 5 1), hemisphere (F(1,11) 5 7.0, po.05, e 5 1) and ISI (F(2,22) 5 76.4, po.001, e 5 0.87) to be significant factors (Table 2). The overall peak amplitude was greater for ON than OFF, greater for Rt-STG than Lt-STG, and greater for the longer ISI condition.

Amplitude of the STG Activity as a Function of the Duration of the Steady State Preceding the Change Figure 3A shows the effects of the ISI on the amplitude of the STG activity. The activity increased in amplitude as the ISI increased for all three events, suggesting that its amplitude was determined by the duration of the prior state, the silence (ON) and tone (OFF and CHANGE). As shown in Figure 3B, the amplitude was linearly correlated with the log of the duration of the steady state preceding the change for all three events. In individual subjects, the correlation efficient, r, was 0.84–0.99, 0.58–0.99, and 0.90-0.98 for ON, OFF, and CHANGE, respectively.

On- and Change-Response Figure 2A shows superimposed waveforms of on- and changeresponses at an ISI of 6 s in a representative subject. The abrupt change in tone frequency elicited a clear component (CH-N1m) similar to On- and Off-N1m in the temporal area bilaterally except for one subject. The dipoles responsible for CH-N1m were also estimated to be located in the STG bilaterally. The location of the source did not differ significantly between ON and CHANGE (Table 1, Figure 2B). A three-way ANOVA indicated the events (F(1,9) 5 13.0, po.01, e 5 1) and ISI (F(3,27) 5 6.4, po.01, e 5 0.654) to be significant factors determining the peak latency of the STG activity. That is, the peak latency of the change-response was longer than that of the on-response (Table 2). Like the on- and offresponse, the change-response in the STG increased in amplitude with an increase in ISI. As for the peak amplitude of the STG activity, results of ANOVA indicated that the events (F(1,9) 5 5.2, po.05, e 5 1) and ISI (F(3,27) 5 178.7, po.001, e 5 0.556) were significant factors and hemisphere (F(1,9) 5 4.5, p 5 .062, e 5 1) tended to be a significant factor. The peak amplitude of the activity was significantly greater for ON than CHANGE. Like the on- and off-responses, the amplitude of the change-response increased with an increase in ISI, as described in detail below.

Discussion

Relationship of the STG Amplitude Between the On- and Off- or Change-Response Among the Subjects When the relationship of the amplitude of the STG activity between ON and OFF was compared across subjects, the correlation coefficient, r, was 0.79 (po.0001) (Figure 4A). The slope of the regression line (off/on) was 0.54. Likewise, r was 0.82 (po.0001) for the relationship between ON and CHANGE with a slope (change/on) of 0.74 (Figure 4B).

In the present study, we investigated the auditory N1m component to clarify its physiological significance to the detection of changes. Results showed that (i) a similar N1m was elicited by the on-, off-, and change-events, and the location and orientation of the source in the STG did not differ significantly among the three events, (ii) the amplitude of the activity as a function of the duration of the steady state preceding the change was similar among the three events, and (iii) there was a significant positive correlation in amplitude between ON and OFF, and between ON and CHANGE among subjects. Based on these findings, we consider that N1m is automatically elicited with any abrupt change, and N1m is elicited by various types of auditory events that would have similar physiological significance relating to the detection of changes. Similar STG Activity in On-, Off-, and Change-Events Similar activity in the STG was elicited by the on-, off-, and change-events (Figure 2) in terms of location, orientation, and time course. Therefore, it is possible that the activity in on-, off-, and change-events was elicited by a similar group of neurons and has a similar function related to the detection of changes. In support of this, the source of the activity was estimated to be

Table 1. Locations of the Dipoles for the On- and Off-Responses Under the Three Conditions ON

OFF

Lt-STG

x

ISI 0.5 s ISI 1.5 s ISI 3.0 s ISI 6.0 s Rt-STG ISI 0.5 s ISI 1.5 s ISI 3.0 s ISI 6.0 s

! 55 ! 53 ! 55 ! 54

" " " "

6 4 4 4

11.1 12.6 14.3 13.2

y " " " "

5.7 5.6 6.4 6.6

60.8 58.3 58.1 58.4

z " " " "

4.6 4.1 4.7 4.1

x

53 53 51 50

" " " "

5 6 5 4

16.0 19.7 19.6 21.4

" " " "

4.7 5.1 4.5 4.7

56.6 56.4 56.2 56.9

" " " "

5.8 4.7 5.6 4.0

y

CHANGE z

! 54.2 " 6.8 ! 57.1 " 5.4 ! 54.2 " 3.3

15.1 " 8.2 13.5 " 6.8 13.4 " 6.2

57.6 " 5.5 60.0 " 4.3 57.7 " 5.1

52.9 " 5.4 51.9 " 5.9 50.5 " 5.1

18.5 " 6.3 18.3 " 5.1 19.0 " 4.5

57.4 " 5.0 57.1 " 5.4 58.5 " 4.3

x

y

z

! 57.5 ! 53.2 ! 55.1 ! 52.6

" " " "

7.7 5.1 .4 4.9

14.2 18.5 16.4 14.6

" " " "

5.0 6.3 8.2 8.0

56.7 59.1 57.6 56.1

" " " "

4.0 3.4 4.0 4.5

51.3 51.2 50.9 49.8

" " " "

4.9 4.8 5.2 4.7

20.0 23.5 22.2 20.7

" " " "

6.1 4.2 7.4 5.3

55.6 57.1 57.3 55.2

" " " "

3.6 3.4 2.7 4.0

Note: The x-axis was fixed with the preauricular points, the positive direction being to the right. The positive y-axis passed through the nasion, and the z-axis thus pointed upward. Rt-STG, right superior temporal gyrus; Lt-STG, left superior temporal gyrus.

Change-related responses in the human auditory cortex

27

Table 2. The Peak Latency and Amplitude of Each Cortical Source Under the Four Conditions ON (n 5 12) Latency (ms) ISI 0.5 s ISI 1.5 s ISI 3.0 s ISI 6.0 s Amplitude (nAm) ISI 0.5 s ISI 1.5 s ISI 3.0 s ISI 6.0 s

Lt-STG

OFF (n 5 12) Rt-STG

90.5 92.0 93.5 95.8

! ! ! !

10.2 7.9 7.9 6.5

89.3 92.6 94.8 95.4

! ! ! !

7.9 9.2 5.8 6.2

12.6 35.1 47.1 59.0

! ! ! !

7.2 15.9 17.5 22.1

17.0 41.4 63.2 74.2

! ! ! !

9.1 10 14.1 19.5

Lt-STG

CHANGE (n 5 10) Rt-STG

83.3 ! 10.6 90.3 ! 10.8 91.9 ! 12.3

85.0 ! 8.8 84.7 ! 8.6 89.4 ! 9.4

12.4 ! 8.0 15.4 ! 8.0 23.8 ! 10.5

14.3 ! 7.2 27.7 ! 12.7 41.6 ! 14.7

Lt-STG

Rt-STG

99.2 98.7 102.2 106

! ! ! !

15.5 10.3 10.4 7.2

98 97 99.9 105.8

! ! ! !

14.8 13.2 9.1 11.8

10.6 30 41.9 49.9

! ! ! !

7.6 9.8 18.0 21.0

16.8 42.6 59.7 63.1

! ! ! !

12.7 15.6 18.7 17.4

Note: Data are expressed as the mean ! SD. Rt-STG, right superior temporal gyrus; Lt-STG, left superior temporal gyrus.

within the change-sensitive area in an fMRI study (Downar, Crawley, Mikulis, & Davis, 2000). They used the change of continuous sounds of running water and croaking frogs in turn, and suggested the activation of the primary and secondary auditory cortices in response to changes in continuous auditory input. The idea that the N1 component is involved in the detection of changes was consistent with a report (Hari et al., 1987) that N1m seems to reflect cortical activities related to any abrupt change in the auditory environment. In fact, for the on-response, a similar N1m component was elicited by various types of sounds in addition to a pure tone including a click (Joutsiniemi et al., 1989; Picton et al., 1974), human voice (Hari & Lounasmaa, 1989) or animal sound (Altmann et al., 2008). However, the difference in the latency of the STG activity among the three events or among the four ISI conditions needs some discussion. At first, the peak latency tended to be shorter for OFF than ON. This phenomenon seems to be caused by the

A ISI

0.5 s

physical difference in stimuli between ON and OFF. That is, the ON-STG activity involves at least two factors, the on-event and physical features of the stimulus, while the OFF-STG activity only involves the off-event. The N1 component is known to consist of a number of different cerebral processes with a peak latency of between 50 and 150 ms (Naä¨ta¨nen & Picton, 1987). The ON-STG activity was considered to reflect a complex process including the frequency, intensity, and location of the sound, while the OFF-STG activity reflects a more simple process of the off-event. Therefore, the peak latency of the OFF-STG activity may be shorter than that of the ON-STG activity because of the lack of the latter part of its activity relating to processing of the sound feature. Second, the latency of the CH-STG activity was significantly longer than that of the ON-STG activity, perhaps due to the period necessary for processing the change in frequency from 1000 Hz to 1100 Hz. Given that the CH-STG activity in this

1.5 s

3.0 s

6.0 s

ON

CHANGE 40nAm OFF 0

300 0

300 0

300 0

300

B 80 R=0.99

80 R=0.97

60

60

40

40

20

20

20

10

0

0 1.0

3.0 ON

6.0 (s)

0

40 R=0.99 30

0 1.0

3.0 CHANGE

6.0 (s)

0

0 1.0

3.0 OFF

6.0 (s)

Figure 3. Effects of the interstimulus interval (ISI) on the amplitude of the evoked response. (A) superimposed waveforms of the activity in the right superior temporal gyrus (Rt-STG) for all the subjects (gray) and respective grand-averaged waveforms (black). (B) the mean amplitude of the Rt-STG activity plotted against the ISI.

28

K. Yamashiro et al.

A

B

(nAm) r=0.79 P

Psychophysiology 48(1)

Psychophysiology 48(2)

Handbook of Psychophysiology

Psychophysiology 47(6)

Psychophysiology 48(3)

Merovingian Military Organization, 481-751

Chariton: Callirhoe (Loeb Classical Library No. 481)

Developmental Psychophysiology: Theory, Systems, and Methods

Ciel et Espace 481 juin.2010

Erlauterungen zu Arthur Schnitzler: Traumnovelle, 2. Auflage (Konigs Erlauterungen und Materialien, Band 481)

Perception-based data processing in acoustics: applications to music information retrieval and psychophysiology of hearing

Methods in Enzymology 481 - Cryo-EM Part A: Sample Preparation and Data Collection

Case Studies in Applied Psychophysiology: Neurofeedback and Biofeedback Treatments for Advances in Human Performance

Perception-Based Data Processing in Acoustics: Applications to Music Information Retrieval and Psychophysiology of Hearing (Studies in Computational Intelligence)

Psychophysiology 48(1)

Psychophysiology 48(2)

Handbook of Psychophysiology

Psychophysiology 47(6)

Psychophysiology 48(3)

Merovingian Military Organization, 481-751

Chariton: Callirhoe (Loeb Classical Library No. 481)

Developmental Psychophysiology: Theory, Systems, and Methods

Ciel et Espace 481 juin.2010

Erlauterungen zu Arthur Schnitzler: Traumnovelle, 2. Auflage (Konigs Erlauterungen und Materialien, Band 481)

Perception-based data processing in acoustics: applications to music information retrieval and psychophysiology of hearing

Methods in Enzymology 481 - Cryo-EM Part A: Sample Preparation and Data Collection

Case Studies in Applied Psychophysiology: Neurofeedback and Biofeedback Treatments for Advances in Human Performance

Perception-Based Data Processing in Acoustics: Applications to Music Information Retrieval and Psychophysiology of Hearing (Studies in Computational Intelligence)

Recommend Documents