Psychophysiology 48(2)

Psychophysiology, 48 (2011), 149–154. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophy...

Author: Robert F. Simons (Ed.)

62 downloads 1393 Views 9MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Psychophysiology, 48 (2011), 149–154. Wiley Periodicals, Inc. Printed in the USA. Copyright r 2010 Society for Psychophysiological Research DOI: 10.1111/j.1469-8986.2010.01050.x

A mock terrorism application of the P300-based concealed information test

JOHN B. MEIXNER AND J. PETER ROSENFELD Department of Psychology, Northwestern University, Evanston, Ilinois, USA

Abstract Previous studies examining the P300-based concealed information test typically tested for mock crime or autobiographical details, but no studies have used this test in a counterterrorism scenario. Subjects in the present study covertly planned a mock terrorist attack on a major city. They were then given three separate blocks of concealed information testing, examining for knowledge of the location, method, and date of the planned terrorist attack, using the Complex Trial Protocol (Rosenfeld et al., 2008). With prior knowledge of the probe items, we detected 12/12 guilty subjects as having knowledge of the planned terrorist attack with no false positives among 12 innocent subjects. Additionally, we were able to identify 10/12 subjects and among them 20/30 crime-related details with no false positives using restricted a priori knowledge of the crime details, suggesting that the protocol could potentially identify future terrorist activity. Descriptors: EEG/ERP, P300, Deception, Complex Trial Protocol, Concealed information test (CIT)

Like CITs using other physiological measures, most P300based CITexperiments have used personally relevant information as the probe item (Rosenfeld, Shue, & Singer, 2007; Rosenfeld et al., 2008) or asked subjects to actively memorize words, testing for concealed knowledge of those words (Allen et al., 1992). Additionally, some experiments have involved the commission of mock crimes, testing for details that were central to a mock theft or espionage that subjects either actually committed (Farwell & Donchin, 1991; Lui & Rosenfeld, 2008; Rosenfeld et al., 1988) or committed in a virtual reality environment (Mertens & Allen, 2008, where subjects also had to learn the probe items verbatim prior to the commission of the virtual mock crime). However, no ERP-based CIT has yet tested subjects on crime-related information that they were not instructed to memorize for a crime that they have not yet carried out. Previous work has shown that mere exposure to new stimuli will elicit unique ERPs for those stimuli as compared to unstudied items, even when a subject is not explicitly asked to remember the stimuli (Cycowicz & Friedman, 1999; Paller, Kutas, & McIssac, 1999). Additionally, subjects repeatedly exposed to words as they read text show differential ERPs to repeated and nonrepeated words, even when there is no instruction to memorize the repeated words (Joyce, Paller, Schwartz, & Kutas, 1999; Rugg, 1985; Van Petten, Kutas, Kluender, Mitchiner, & McIsaac, 1991). These results suggest that incidental exposure to crimerelated information, without any memorization, could lead to differences between guilty and innocent participants in a CIT, though these studies did not examine individual differences, which are necessary for success in a diagnostic test like the CIT. These results suggest potential applications of the CIT that have not been investigated, for example, as an antiterrorism tool, both as a way to identify terrorists and as a way to identify details

Research on the concealed information test (CIT) has typically focused on the detection of mock crime knowledge (Ben-Shakhar & Dolev, 1996; Farwell & Donchin, 1991; Lui & Rosenfeld, 2008; Lykken, 1959; Mertens & Allen, 2008; Rosenfeld et al., 1988) or personally relevant information (Lykken, 1960; Rosenfeld, Soskins, Bosh, & Ryan, 2004; Rosenfeld et al., 2008). The CIT presents subjects with various stimuli, one of which is a crime-related item (the probe, such as the gun used to commit a murder). Other stimuli consist of control items that are of the same class (irrelevants, such as other potentially deadly weapons: a knife, a bat, etc.) such that an innocent person would be unable to discriminate them from the crime-related item. If the subject’s physiological response is greater for the probe item than for irrelevant items, then knowledge of the crime or other event is inferred. The CIT has since been adapted to use the P300 component as the key response (Allen, Iacono, & Danielson, 1992; Farwell & Donchin, 1991; Rosenfeld et al., 1988). P300-based CITs have typically shown 80% to 95% accuracy in detecting guilty participants, with a 0% to 10% false positive rate (Allen et al., 1992; Farwell & Donchin, 1991; Rosenfeld et al., 1988, 2008), though one recent study using a virtual mock crime method found guilty detection rates near 50% (Mertens & Allen, 2008). Additionally, one recent protocol has shown resistance to physical and mental countermeasures directed at irrelevant items, which have traditionally defeated the original P300-based CITs (Rosenfeld et al., 2008). This research was supported by the Defense Academy for Credibility Assessment grants DODP198-P-0001, DODPI04-P-0002, and W74V8H-6-01-0001 awarded to J. Peter Rosenfeld. Address correspondence to: John B. Meixner, Department of Psychology, Northwestern University, 2021 Sheridan Road, Evanston, IL 60208-2700, USA. E-mail: [email protected] 149

150

J.B. Meixner & J.P. Rosenfeld

of a planned terrorist attack. The current study shows the effectiveness of the P300-based CIT in identifying subjects who have planned, but not yet executed, a mock terrorist attack. Subjects spent approximately 30 min planning aspects of a terrorist attack and then underwent a CIT based on the Complex Trial Protocol as described by Rosenfeld et al. (2008). Analyses were done both given advance knowledge of the probe (to identify individuals as knowledgeable about the attack) and without advance knowledge (to identify the details of the planned attack in addition to individuals involved in the attack). Additionally, the current study uses three separate test blocks for three different categories of concealed information (location, date, and method of the planned attack) in an attempt to increase detection accuracy and reduce false positives.

Methods Participants Twenty-nine students (average age: 18.7 years; 14 men) at Northwestern University were recruited and gave informed consent. All subjects were right-handed. Participants received course credit for participation. All participants had normal or corrected vision. Trial Structure Trial structure was modeled after Rosenfeld et al. (2008). Each trial began with a 100-ms baseline period of black screen during which prestimulus electroencephalogram (EEG) was recorded. Next, depending on the block, a date, city, or method of terrorist attack was presented in white text on a black background for 300 ms (see Figure 1). Cities and methods of attack were presented as single words; months were presented as three letter abbreviations (Apr, Jan, etc). Upon seeing the stimulus, subjects pressed one of five response buttons at random, regardless of the stimulus seen. Responses were made using a five-button box where subjects placed each digit of the left hand on a separate button. The first stimulus (probe or irrelevant) was followed by a randomly varying interstimulus interval of 1400 ms to 1850 ms,

Figure 1. Trial structure.

during which a black screen appeared. Following this interval, a string of six identical numbers ranging from 1 to 5 (i.e., 111111, 222222, etc.) was presented for 300 ms. Subjects were instructed to press the left mouse button with the index finger of the right hand when they saw the string of ones (the target) and the right mouse button with the middle finger of the right hand when they saw any other string (nontargets). All stimuli were shown in white font 0.7 cm high on a monitor 70 cm in front of the subject.

Procedure After signing consent, subjects were seated in a comfortable chair and given written instructions outlining the study. Among these written instructions was a form listing the cities, months, and types of terrorist attacks that would later appear in the CIT. Subjects were asked to circle any items that had personal relevance to them (e.g., the money during which month during which the subject was born, the city where the subject had lived, etc.), though they were not told they would be viewing these items later. If any of the irrelevant items had personal relevance to the subject, these items were replaced with similar items of no personal relevance. Subjects in the guilty group (n 5 12) were given a briefing document explaining that they were to play the role of a terrorist agent and plan a mock terrorist attack on the United States. The document detailed several different options they could choose regarding how to carry out the attack. Subjects read detailed descriptions of four types of bombs that could be used, four locations in the city of Houston that could be attacked, and four dates in July when the attack could take place. The descriptions contained pros and cons of each potential choice and instructed subjects to choose one type of bomb, one location in Houston, and one date on which to attack. After reading the briefing document, subjects were instructed to compose a letter to their superior in the terrorist organization describing the choices they had made. Note that there was no explicit formal training or instructed item memorization in this protocol. Subjects in the innocent group (n 5 12) completed a similar task planning a vacation instead of a terrorist attack. Subjects completed 5 min of practice, in which they performed 30 trials of a task identical to the full task as described above, except subjects viewed random first names rather than items relevant to the planned attack. The target/nontarget task in the practice was identical to that of the full task. After the practice, subjects completed three separate blocks of the task, with each block testing for a separate concealed information item. Subjects were shown potential cities (e.g., Detroit, Atlanta, etc.) where the terrorist attack could occur (with Houston to be used as the probe), potential types of terrorist attacks (with Bomb to be used as the probe) and potential months in which the attack could occur (with July to be used as the probe). Order was counterbalanced. After every 50 trials, the task was paused and subjects were asked to verbally repeat the previous item seen (to help ensure attention). Two subjects were removed from the final analysis for more than five such errors across all three blocks, and 3 subjects were removed because one of the probe items had personal relevance to them (‘‘July’’ for 2 subjects, ‘‘Houston’’ for 1 subject), creating a confound. Each block contained 300 trials and lasted 25 min. There were five irrelevant items and one probe in each block; the ratio of probe to irrelevant trials was 1:5. Targets occurred on 10% of all trials and were equally likely to occur after either a probe or an irrelevant stimulus.

Mock terrorism concealed information test At the end of the experiment (and following all data collection), subjects were asked which city, method of attack (innocent subjects were asked about their vacation activity), and month were associated with the briefing they had read, to ensure subjects had actually read the briefing document. Data Acquisition EEG was recorded using Ag/AgCl electrodes attached to midline sites Fz, Cz, and Pz. Scalp electrodes were referenced to linked mastoids. Electrode impedances were held below 10 kO. Electrooculogram (EOG) was recorded differentially via Ag/AgCl electrodes placed above and below the left eye. EOG electrodes were placed diagonally to allow for the recording of both vertical and horizontal eye movements as well as eyeblinks. Artifact rejection criteria varied based on each subject’s artifact amplitudes, but was always less than 50 mV. Trials for which this threshold was exceeded were removed from both the ERP and reaction time analyses. Two subjects with fewer than 25 nonartifacted trials per stimulus were removed from the final analysis. The forehead was connected to the chassis of the isolated side of the amplifier system (‘‘ground’’). Signals were passed through Grass P511K amplifiers with a 30-Hz low-pass filter setting, and highpass filters set (3 db) at 0.3 Hz. Amplifier output was passed through a 16-bit A/D converter sampling at 500 Hz. After initial recording, single sweeps and averages were digitally filtered offline to remove higher frequencies; 3 db point 5 6 Hz. Analysis Methods P300 amplitude, our main dependent variable, was measured using the peak–peak method as described by Soskins, Rosenfeld, and Niendam (2001). We and others have found this analysis method to be more sensitive for the detection of deception than the standard base–peak method as used in earlier studies (Meijer, Smulders, Merckelbach, & Wolf, 2007; Soskins et al., 2001). Using in-house software designed for the Matlab platform, an algorithm searched a window of 400 ms to 650 ms to find the maximally positive segment of 100 ms, with the midpoint of this segment defined as P300 latency and its average amplitude defined as the positive P300 peak. Next, the algorithm searched a window from the P300 latency to 1300 ms to find the maximally negative segment of 100 ms. The peak–peak amplitude of the P300 was defined as the difference between the positive P300 peak and the maximally negative voltage following the P300 peak. ERP analysis was only performed on the probe/irrelevant half of the trial and not on the attention enforcing target/nontarget task. Within-Individuals Bootstrap Analysis To determine whether the P300 evoked by a given stimulus is greater than that evoked by another stimulus within an individual in each block, the bootstrap method (Wasserman & Bockenholt, 1989) was used at the Pz site, where P300 is usually largest (Fabiani, Gratton, Karis, & Donchin, 1987). Because the actual distributions of probe and irrelevant waves are not available, they must be bootstrapped from the existing data. To do this, a computer program draws, with replacement, a set of individual probe waveforms equal to the number of accepted probe trials in each block and also draws (with replacement) an equal number of irrelevant waveforms, selected randomly from all five irrelevant items in each block. The program then subtracts the mean irrelevant P300 from the mean probe P300 and then repeats the process 1,000 times to create a distribution of bootstrapped

151 probe minus irrelevant averages. This bootstrap test is referred to as the Iall test, because it compares the probe to the average of all irrelevants to determine the probability that the true difference between the average probe P300 and average irrelevant P300 is greater than zero in each block. In reporting bootstrap values, we report the number of iterations (out of 1,000) in which the probe average exceeded the irrelevant average in each block. Individual detection rates were reported based on the average number of iterations in which the probe average exceeded the irrelevant average across all three blocks. So, the bootstrap just described is conducted for each block, and each subject’s three blocks are then averaged to yield the subject’s bootstrap value across blocks as seen in Table 1. The maximum bootstrap value per block is 1,000, or 3,000 over the three blocks per subject. The maximum average value per subject is 3,000/3 5 1,000. For the Iall test, a .9 confidence interval cut point was used as the criterion for guilt, as in previous studies (Farwell & Donchin, 1991; Rosenfeld et al., 2004, 2008). Thus, a subject is detected as guilty if, across all three blocks, the probe average exceeds the irrelevant average (both over three blocks) on at least 900 out of 1,000 iterations of the bootstrap process. A second, more rigorous test compared the probe P300 to the largest maximum irrelevant stimulus P300 (Imax). This process is identical to the Iall method, except irrelevant waveforms were drawn only from the irrelevant item that yielded the largest individual P300 amplitude. So, for example, on the city block, if ‘‘Detroit’’ was the irrelevant item that yielded the largest P300 amplitude for the entire block, the program would draw only from trials in which ‘‘Detroit’’ was the stimulus, effectively comparing the probe to the irrelevant item that generated the largest P300 amplitude. Many studies have arbitrarily used .9 confidence intervals for both Iall and Imax tests. As can be seen in the first two columns of Table 1, this may be unnecessarily stringent, as any value from .7 to .9 would yield perfect guilty–innocent Table 1. Individual Bootstrap Detection Rates Iall

Imax

Blind Imax

Guilty

Innocent

Guilty

Innocent

Guilty

Innocent

1,000 1,000 955 996 994 909 945 997 999 985 912 903 966

648 610 598 611 150 475 600 555 586 690 390 644 546

985 999 889 898 946 698 677 959 908 888 667 837 863

287 416 476 430 17 284 365 250 217 382 129 215 289

985 998 892 893 943 761 702 961 907 886 698 842 872

603 602 649 605 689 547 536 569 565 706 650 702 619

12/12 0/12 AUC 5 1.0

12/12 0/12 AUC 5 1.0

10/12 0/12 AUC 5 .979

Note: Numbers indicate the average number of iterations (across all three blocks) of the bootstrap process in which probe was greater than Iall or Imax for each of the twelve guilty and twelve innocent subjects. Blind Imax numbers indicate the average number of iterations in which the largest single item (probe or irrelevant) was greater than the second largest single item. Mean values for each column are displayed in bold above detection rates. AUC: area under the curve in the ROC analysis. Iall detection rates are based on a .9 confidence interval, Imax on a .5 interval, and Blind Imax on a .75 interval.

152


discrimination. Here, we use a .5 confidence interval for the Imax test (500 significant iterations or greater yields a guilty diagnosis; second two columns, Table 1) though any cut point between .5 and .65 would yield perfect guilty–innocent discrimination, as it is evident that for both guilty and especially innocent subjects, positive bootstrap iteration totals are much lower than with Iall. This is as expected, because in an Iall bootstrap, we compare the probe item (which for an innocent subject is essentially another irrelevant item) to the average of all irrelevant items. For an innocent subject, the Iall bootstrap value should approach 500 out of 1,000 iterations because we are essentially comparing one irrelevant item to the average of the other five irrelevant items; there should be little to no difference between these values. Contrastingly, in the Imax bootstrap, we are comparing the probe item to the irrelevant item that is by definition the largest. Thus, for the innocent subject, we are comparing what is essentially a randomly picked irrelevant item (the probe) to the largest irrelevant item. Though these two items should be similar in size theoretically, individual variations in the ERPs for each stimulus may cause one irrelevant to be larger than another, and the Imax test by definition selects the irrelevant item that is largest, which must cause the Imax bootstrap value to be smaller as compared to Iall. Finally, a third test (Blind Imax) was conducted to determine guilt or innocence if one does not know the probe a priori. For example, authorities may not know the city of a planned terrorist attack and might want to test a suspected terrorist conspirator for this information. To do this, one must conduct the analyses with no advance knowledge of the probe. In this test, the stimulus with the largest P300 (whether probe or irrelevant) is assumed to be the probe, and its P300 is compared with the next largest stimulus’s P300, which is assumed to be the largest irrelevant P300. For this also very demanding test, it was necessary to use a lower cut point of .75 for optimal guilty–innocent discrimination. For detection of individual items in single blocks using the Blind Imax bootstrap, a .9 confidence interval was used. It is important to note that the confidence intervals we used provide, as all such intervals should, the best guilty–innocent discrimination in the current experiment, but may not be universally ideal. The confidence intervals presented here simply indicate that there is perfect discrimination between guilty and innocent subjects when details of the attack are known ahead of time. These confidence intervals or cut points must be established with replication across many subjects. In the field, where ground truth is unknown, the ideal cut point would have to be based on Guilty

well-established norms because it could not be selected a posteriori as was done here. It should be appreciated that there is no absolute ideal cut point in a given situation; one may reasonably use any cut point that provides an acceptable sensitivity (guilty detection rate) with an acceptable level of specificity (false positive rate). Results All within-subjects analysis of variance (ANOVA) p values reported are Greenhouse–Geisser (GG) corrected if df41. Partial eta squared values (Z) are reported where applicable. All subjects correctly recalled all relevant details from the briefing (e.g., city, method of attack, month). Figure 2 shows grand average waveforms at site Pz. Waveforms are shown for the probe item and the average of all irrelevant items (Iall) for both groups. Probe P300 amplitude (peak–peak) is clearly larger than Iall amplitude in the guilty group, whereas the probe and Iall amplitudes are nearly identical in the innocent group. A 2 (Stimulus: probe vs. irrelevant) ! 2 (Group: guilty vs. innocent) ANOVA was run on the peak–peak P300 amplitudes. There was a significant main effect of stimulus, F(1,70) 5 100.31, po.001, Z 5 .589, and a main effect of group, F(1,70) 5 20.36, po.001, Z 5 .225. The interaction was highly significant, F(1,70) 5 77.35, po.001, Z 5 .525. Table 1 shows detection rates within subjects for both groups, as well as the average number of significant iterations out of the maximum possible 1,000 in the bootstrap test for each subject. Detection rates are shown for each of the three tests: Iall, Imax, and Blind Imax, as described above. Perfect 12/12 detection rates with no false positives were attained when the probe item was known a priori in the Iall and Imax bootstrap tests. To examine the detection efficiency of each analysis method, receiver operating characteristic (ROC) analyses were conducted. The input statistic for the ROC analysis was the bootstrap statistic as displayed in Table 1. Because there is no overlap between the guilty and innocent groups for either the Iall or Imax analysis methods, the area under the curve (AUC) is 1.0, as seen in Table 1. The Blind Imax method yielded an AUC of .979. The Blind Imax bootstrap results for each individual block are shown in Figure 3. With no a priori knowledge of the probe item, we were able to successfully identify 21/36 details associated with the planned terrorist attack at a .9 confidence level with no Innocent

Figure 2. Grand average probe and irrelevant ERPs at Pz.

Mock terrorism concealed information test

153

1000

Bootstrap Value

900 Guilty

800

Innocent

700 600 500 400

Figure 3. Bootstrap results of each individual block (12 subjects ! 3 blocks 5 36 guilty blocks136 innocent blocks for 72 total blocks) using the blind Imax method. Twenty-one of 36 items associated with the terrorist attack were successfully identified at a .9 confidence level with no false positives.

false positives. ROC analysis revealed that the AUC for this analysis method was .873. Among the 10 subjects whom we successfully identified as possessing attack-related knowledge using the Blind Imax method, we were able to successfully identify 20/30 possible items as being relevant to the terrorist attack.

Discussion The data reported here demonstrate that the Complex Trial Protocol version of the P300-based concealed information test could be highly effective in detecting an individual’s knowledge of a planned terrorist attack. To the best of our knowledge, this is the first such report of a mock terrorism-based CIT. These data differ from previously reported mock crime studies (e.g., Lui & Rosenfeld, 2008; Mertens & Allen, 2008; Rosenfeld et al., 1988), because subjects in this experiment did not commit any crimeFthey only planned a crime that was to occur at a future date. Additionally, unlike previous studies, subjects here were not formally trained or explicitly instructed to memorize items. One mock crime study (Carmel, Dayan, Naveh, Raveh, & BenShakhar, 2003) was more similar to the current study, using a CIT to test subjects for knowledge of mock crime details that were not directly involved in the crime itself (e.g., a portrait on the wall where the crime was committed). This provided mere exposure to the probe items without explicit instructions from the experimenters to pay attention to those specific details. The results showed reduced accuracy in this more realistic type of mock crime, but this study was conducted using skin conductance response as the primary dependent measure, rather than ERPs. However, it should be noted that the depth of processing for those incidental items in Carmel et al. (2003) was likely less than that of the current study, where subjects carefully reviewed information about the planned terrorist attack. Additionally, this study is the first to make use of multiple blocks of testing to successfully increase the sensitivity of the P300-based CIT (Rosenfeld, Shue, & Singer, 2007, used multiple blocks but achieved only 55% sensitivity using the older, nonComplex Trial Protocol version of the P300-based CIT). Though the use of multiple blocks of testing with multiple questions regarding different crime-related items is common in polygraphbased CITs (Elaad, 1990; Elaad & Ben-Shakhar, 1991; Elaad, Ginton, & Jungman, 1992), P300-based CITs have primarily focused on a single guilty knowledge item (Rosenfeld et al., 1988,

2004, 2008) or several guilty knowledge items in a single block of testing (Farwell & Donchin, 1991; Farwell & Smith, 2001; Mertens & Allen, 2008), which is problematic (Rosenfeld et al., 2004). It appears that the combination of three separate blocks of data contributed to the high individual detection accuracy in the current experiment. In the Iall and Imax bootstrap tests with 100% individual detection, many subjects had individual blocks in which the probe P300 amplitude was not significantly greater than the irrelevant amplitude such that the single block would not, by itself, yield an accurate detection. However, when the three blocks are combined, the effects of occasionally inadequate blocks are reduced, effectively increasing the signal-to-noise ratio by sampling more information from each subject. Perhaps the most potentially useful result reported here is the moderately high rate of detection of individual blocks without specific a priori knowledge of the probe items (Figure 3). Allen et al. (1992) also utilized a blind Bayesian approach to the identification of learned versus unlearned lists of words. A blind approach is inherent in the Bayesian approach utilized by Allen et al., which asks about the conditional probability that a list is learned, given that it elicits a P300. This approach could have been applied to our present data set, though their approach first developed Bayesian parameters on a nonblinded basis from one sample of 20 participants and then applied those parameters to subsequent samples of subjects. The terrorist scenario used here does not readily lend itself to such a preliminary model-building phase. Additionally, subjects in the Allen et al. study were specificially instructed to memorize the lists of words that were eventually tested, which was not the case here. The lack of such explicit learning to perfection may result in ERPs with relatively lower signal-to-noise ratios pertaining to the terrorist act details that were studied only briefly. Application of the relatively more direct assumption that the actual key knowledge detail will elicit the largest P300 (measured as simple amplitude) compared in a simple bootstrap to the next largest P300 is a more direct approach that is original in the present context. Our results suggest that one might be able to identify locations or times of terrorist attacks if the location or time is restricted to a small enough set to perform a test similar to that of the current experiment. In the field, generating such a small set may be simple for the month of the attack, as there are only 12 possible months, whereas determining the city and type of attack would be considerably more difficult because of the multitude of possibilities. In determining the city where the attack is planned to occur, one could attempt a type of partition test, where the subject is presented several potential large locations (such as the Northeast, Midwest, etc.). Using the blind analysis method demonstrated here, it should be possible to determine which of these larger areas is the planned location of attack and subsequently separate that area into smaller and smaller partitions until the location is discovered. This may not be effective, however, as a single error at any level of the process would lead the examiner astray. This type of test has not been attempted using P300 and would be an interesting future experiment. It should be noted that 3 subjects were removed because one of the probe items had personal relevance to them. In the field, obviously, we cannot just throw out suspects for whom selected probe items have personal relevance. This is a limitation of the current study. However, there ideally would be more than three blocks of testing in the field, and extra blocks could be used to compensate for a block that is confounded by personal relevance. For example, if the probe is known to be Houston, but the sus-

154


pect was born in Houston, this block could not be used, but other blocks should not be affected, still potentially allowing detection of the individual. One does not know how well these results will translate to a field scenario, but it is likely that details that are central to a planned attack will be well rehearsed and remembered by terrorist conspirators. Although the level of encoding in the current study was comprehensive, resulting in perfect recall of the

crime-relevant items when subjects were asked after the experiment, it is likely that our subjects, who spent only about 30 min learning about the attack and planning details, did not attach the same level of meaning to these items that a real terrorist would, having likely spent hours reviewing the attack plans. This increase in familiarity with the probe items could translate to larger P300s and thus greater detection efficiency in the field.

REFERENCES Allen, J., Iacono, W. G., & Danielson, K. D. (1992). The identification of concealed memories using the event-related potential and implicit behavioral measures: A methodology for prediction in the face of individual differences. Psychophysiology, 29, 504–522. Ben-Shakhar, G., & Dolev, K. (1996). Psychophysiological detection through the guilty knowledge technique: Effects of mental countermeasures. Journal of Applied Psychology, 81, 273–281. Carmel, D., Dayan, E., Naveh, A., Raveh, O., & Ben-Shakhar, G. (2003). Estimating the validity of the guilty knowledge test from simulated experiments: The external validity of mock crime studies. Journal of Experimental Psychology: Applied, 9, 261–269. Cycowicz, Y. M., & Friedman, D. (1999). ERP recordings during a picture fragment completion task: Effects of memory instructions. Cognitive Brain Research, 8, 271–288. Elaad, E. (1990). Detection of guilty knowledge in real-life criminal investigations. Journal of Applied Psychology, 75, 521–529. Elaad, E., & Ben-Shakhar, G. (1991). Effects of mental countermeasures on psychophysiological detection in the guilty knowledge test. International Journal of Psychophysiology, 11, 99–108. Elaad, E., Ginton, A., & Jungman, N. (1992). Detection measures in real-life criminal guilty knowledge tests. Journal of Applied Psychology, 77, 757–767. Fabiani, M., Gratton, G., Karis, D., & Donchin, E. (1987). The definition, identification, and reliability of measurement of the P300 component of the event-related brain potential. In P. K. Ackles, J. R. Jennings, & M. G. H. Coles (Eds.), Advances in Psychophysiology (Vol. 2, pp. 1–78). Greenwich, CT: JAI Press. Farwell, L. A., & Donchin, E. (1991). The truth will out: Interrogative polygraphy (‘‘lie detection’’) with event-related potentials. Psychophysiology, 28, 531–547. Farwell, L. A., & Smith, S. S. (2001). Using brain MERMER testing to detect knowledge despite efforts to conceal. Journal of Forensic Sciences, 46, 135–143. Joyce, C. A., Paller, K. A., Schwartz, T. J., & Kutas, M. (1999). An electrophysiological analysis of modality-specific aspects of word repetition. Psychophysiology, 36, 655–665. Lui, M., & Rosenfeld, J. P. (2008). Detection of deception about multiple, concealed, mock crime items, based on a spatial-temporal analysis of ERP amplitude and scalp distribution. Psychophysiology, 45, 721–730. Lykken, D. T. (1959). The GSR in the detection of guilt. Journal of Applied Psychology, 43, 385–388.

Lykken, D. T. (1960). The validity of the guilty knowledge technique: The effects of faking. Journal of Applied Psychhology, 44, 258–262. Meijer, E. H., Smulders, F. T. Y., Merckelbach, H. L. G. J., & Wolf, A. G. (2007). The P300 is sensitive to face recognition. International Journal of Psychophysiology, 66, 231–237. Mertens, R., & Allen, J. B. (2008). The role of psychophysiology in forensic assessments: Deception detection, ERPs, and virtual mock crime scenarios. Psychophysiology, 45, 286–298. Paller, K. A., Kutas, M., & McIsaac, H. K. (1999). Monitoring conscious recollection via the electrical activity of the brain. Psychological Science, 6, 107–111. Rosenfeld, J. P., Cantwell, G., Nasman, V. T., Wojdac, V., Ivanov, S., & Mazzeri, L. (1988). A modified, event-related potential-based guilty knowledge test. International Journal of Neuroscience, 24, 157–161. Rosenfeld, J. P., Labkovsky, E., Lui, M. A., Winograd, M., Vandenboom, C., & Chedid, K. (2008). The Complex Trial Protocol (CTP): A new, countermeasure-resistant, accurate, P300-based method for detection of concealed information. Psychophysiology, 45, 906–919. Rosenfeld, J. P., Shue, E., & Singer, E. (2007). Single versus multiple proble blocks of P300-based concealed information tests for self-referring versus incidentally obtained information. Biological Psychology, 74, 394–404. Rosenfeld, J. P., Soskins, M., Bosh, G., & Ryan, A. (2004). Simple, effective countermeasures to P300-based tests of detection of concealed information. Psychophysiology, 41, 205–219. Rugg, M. D. (1985). The effects of semantic priming and word repetition on event-related potentials. Psychophysiology, 22, 642–647. Soskins, M., Rosenfeld, J. P., & Niendam, T. (2001). The case for peakto-peak measurement of P300 recorded at .3 hz high pass filter settings in detection of deception. International Journal of Psychophysiology, 40, 173–180. Van Petten, C., Kutas, M., Kluender, R., Mitchiner, M., & McIsaac, H. (1991). Fractioning the word repetition effect with event-related potentials. Journal of Cognitive Neuroscience, 3, 131–150. Wasserman, S., & Bockenholt, U. (1989). Bootstrapping: Applications to psychophysiology. Psychophysiology, 26, 208–221.

(Received July 14, 2009; Accepted January 30, 2010)


Mock crime application of the Complex Trial Protocol (CTP) P300-based concealed information test

MICHAEL R. WINOGRAD and J. PETER ROSENFELD Department of Psychology, Northwestern University, Evanston, Illinois

Abstract The Complex Trial Protocol (CTP), was shown to be an improvement over the previous ‘‘three stimulus’’ P300-based concealed information tests (CITs). Not only was it highly accurate with autobiographical information but was also resistant to the use of countermeasures (CMs). The current study applied the CTP to the detection of incidentally acquired information in a mock crime scenario. In previous ‘‘three stimulus’’ mock crime studies utilizing P300-based CITs, participants memorized a guilty knowledge item(s). Special care was taken in the current study to ensure that participants’ knowledge of the guilty item in the mock crime was obtained only during the commission of the act in order to bolster ecological validity. Overall, 92% of all participants in guilty, innocent, and countermeasure conditions were correctly classified. CM use was again indexed by reaction times (RTs). Descriptors: Mock crime, P300, Concealed information test, Deception, Complex Trial Protocol, CIT

ject sees a rare and meaningful ‘‘guilty’’ item during questioning. During CIT questioning, a subject is asked a question relevant to a crime or incident in which he is suspected of having participated, such as, ‘‘What was used to kill the victim?’’ Then, the examiner presents the suspect serially with a number of possible answers, with pauses in between each, to measure autonomic arousal: ‘‘Was it a rope? Was it a gun? Was it a knife? etc., . . .’’ The subject is expected to react most strongly when he recognizes the correct answer, a detail of the crime that is known ideally only by the police and the guilty party. MacLaren (2001) conducted the most comprehensive meta-analysis of autonomic nervous system (ANS)-based CIT mock crime studies to date and found an average correct guilty classification rate of 76% with a falsepositive rate of 17%. This methodology lends itself very well to use with the P300, an endogenous positive ERP evoked between 300–900 ms poststimulus. It is evoked by the oddball recognition response, which has been alternatively described as an orienting response (Vrij, 2008) or a sign of context updating (Donchin & Coles, 1988). To an innocent person, each potential murder weapon would be irrelevant. However, a guilty suspect would recognize the one correct alternative, thus making it an oddball capable of evoking a P300 response. This application of the P300 was first explored around 20 years ago (Farwell & Donchin, 1991; Rosenfeld, Angell, Johnson, & Qian, 1991; Rosenfeld et al., 1988). Since then, the majority of P300-based CITs used the three-stimulus method, which consists of trials with either a probe, one of four or more irrelevants, or a target stimulus. These three stimuli are presented in a random sequence. In this method, the probe is a guilty knowledge item (such as the subject’s name, birth date, or an object used in a crime). The irrelevants are also dates, names,

Rosenfeld, Labkovsky, Winograd, Lui, Vandenboom, and Chedid (2008) introduced the Complex Trial Protocol (CTP), a new P300-based concealed information test (CIT). In contrast to previous CITs (Allen, Iacnono, & Danielson, 1992; Farwell & Donchin, 1991; Rosenfeld, Cantwell, Nasman, Wojdac, Ivanov, & Mazzeri, 1988; Rosenfeld, Soskins, Bosh, & Ryan, 2004), which utilized the P300 event-related potential (ERP), the CTP was resistant to the use of countermeasures (CMs), covert responses executed by the subject in an attempt to beat the test. While these results are encouraging for the future use of P300based CITs in the detection of concealed autobiographical information (such as the probes used in Rosenfeld et al. (2008), namely, participants’ birthdates, hometowns, and mothers’ first names), it was unknown if the CTP would perform as well using incidentally acquired or episodic information, such as details from a mock crime. Indeed, recent applications of the earlier three-stimulus P300 CITs were not sensitive to incidentally acquired information (Rosenfeld, Biroshak, & Furedy, 2006). The CIT was created by David Lykken and originally referred to as the ‘‘Guilty Knowledge Test’’ as an alternative to the Comparison Question Test (CQT) polygraph method (Lykken, 1959). Unlike the case in the CQT method where one looks for arousal driven by the act of lying and fear of detection, the CIT relies upon arousal driven by an orienting response when a sub-

This research was supported by the Department of Defense Polygraph Institute Grants DoDPI98-P-0001 and DoDPI04-P-0002 awarded to J.P.R. Address correspondence to: Michael R. Winograd, Department of Psychology, Northwestern University, 2021 Sheridan Rd., Evanston, IL, 60208. E-mail: [email protected] 155

156 or objects, but which are not meaningful to the subject. The target stimulus is simply another irrelevant, but it is assigned a unique response that is used to force attention to all the stimuli. Despite not making a unique response to the probes, the simple recognition of a ‘‘guilty’’ item is ordinarily enough to evoke a P300 and also a slower reaction time (Allen, Iacnono, & Danielson, 1992; Farwell & Donchin, 1991; Rosenfeld et al., 2004, 2008; Verschuere, Rosenfeld, Winograd, Labkovsky, & Wiersema, 2009). Previous research has shown that reaction times (RTs) alone can also be an indicator of guilty knowledge in CITs (Seymour, Seifert, Shafto, & Mosmann, 2000; Verschuere, Crombez, Degrootte, & Rosseel, 2009), with longer reaction times for probe items. Vrij (2008) reviewed current P300-based CITs and found a mean overall correct classification rate of 82.3% (range 51– 100%) for guilty participants and 87.5% (range 72–100%) for innocent participants. Two of the fourteen studies cited (Allen & Iacono, 1997, and Farwell & Donchin, 1991) classified some participants as ‘‘inconclusive.’’ Across these studies, an average of 8.8% of innocent participants were incorrectly classified as guilty and 16.2% of guilty participants were incorrectly diagnosed as innocent; there are somewhat better results than with the ANS-based CIT. These studies used various types of stimuli, including autobiographical information, studied words, and incidentally acquired knowledge such as an experimenter’s name. While fairly accurate with guilty and innocent participants at a level comparable to or better than that of CIT polygraphs (Vrij, 2008), these three-stimulus, P300-based CITs were shown to be vulnerable to CMs (Mertens & Allen, 2008; Rosenfeld et al., 2004). In Rosenfeld et al. (2004), using a three-stimulus protocol, the detection rate for participants using CMs dropped to a 50% level (from 92%) with an autobiographical probe and to just 18% (from 82%) with an incidentally acquired item as the probe. This vulnerability to CMs is also found in the ANS-based polygraph literature. Honts & Amato (2002) reviewed a number of polygraph studies and found false negative outcomes in 47 to 70% of participants trained in the use of CMs and a 10% to 40% reduction in correct classifications in CITs. However, CM use was detectable through an increase in irrelevant item RT in the majority of CM participants. In the CTP, the target/non-target presentation and decision response is separated from the presentation of probe or irrelevant in a two-stimulus trial. Initially, the subject is presented with either a probe or one of multiple irrelevants. After a delay of about 1.5 s, a second (target or non-target) stimulus is presented, and the subject makes a classification decision. Using this new protocol with autobiographical probes, guilty and innocent participants were correctly diagnosed with 100% accuracy. In contrast to previous ANS and P300-based CITs, 92% of CM participants were correctly classified as guilty, a dramatic improvement over the three-stimulus method (Rosenfeld et al., 2008). Analysis of RTs clearly indicated CM use by identifying slower RTs to irrelevants than probes within a testing block. The goal of the current study was to show that the CTP would be a sensitive measure for the detection of concealed, purely incidentally acquired information in a mock crime scenario, and that RTs could continue to be used in the detection of CM use. In addition to the previous method of identifying CM use by comparing RTs between probe and irrelevant stimuli within a block (Rosenfeld et al., 2004, 2008), it was hypothesized that an increase in RT from a baseline block to an experimental block would also be indicative of CM use.

M. R. Winograd & J. P. Rosenfeld Methods Participants Participants were recruited through an online recruiting service (researchchicago.com) and with flyers placed around the Northwestern University campus. Data from 36 participants (24 female, age range 18–35) were used in the final analyses. Four participants were excluded from the analyses for failing to follow various aspects of the instructions. Participants were offered $20 for two hours of participation. All participants had normal or corrected-to-normal vision. Procedures Participants were randomly assigned to one of three conditions, Simple Guilty (SG), Countermeasure (CM), or Innocent Control (IC). The CTP structure was used in both the baseline and experimental blocks: For each trial, stimulus one (S1), which was the probe or one of six irrelevants, was presented 100 ms after trial onset and remained for 300 ms. Participants were instructed to press the left button on the button box as soon as they saw this first stimulus, whichever it was for each trial, to acknowledge having seen it. This response has therefore been named the ‘‘I Saw It’’ response. After a randomly varying delay of 1400, 1550, 1700, or 1850 ms, stimulus two (S2) appeared, also for 300 ms. Stimulus two was a string of one of five identical numbers (11111, 22222, 33333, 44444, or 55555). The string of ones was designated as a target stimulus for each subject, to which they were instructed to press the right button. The left button was pressed for each non-target S2. Stimulus probabilities are presented in Table 1. Note that there is a difference in conditional probabilities of targets following probes and targets following irrelevants. Rosenfeld, Tang, Meixner, Winograd, and Labkovsky (2009) showed no significant differences in P300 amplitudes between an asymmetric and symmetric matrix of conditional target probabilities following probes and irrelevants. For this reason, asymmetric probabilities as in Rosenfeld et al. (2008) were utilized here again. Stimuli were presented in 0.5 cm tall uppercase white text on a black background. Participants were seated about 1 m from the computer monitor. To help ensure attention, testing was interrupted every 20–40 trials and participants were asked what the last stimulus (S1) they saw was. Participants were informed prior to the testing session that missing more than two of these identifications was grounds for exclusion from analysis, as such errors showed that participants were not paying proper attention to the stimuli. Baseline Block After giving informed consent, all participants in all groups first participated in a baseline block, which was identical in structure to the experimental block, so as to determine a range for each subject’s normal RTs. Participants were seated in a chair, the same one in which the later ERP testing would take place, and were presented with seven playing cards face down on the table. Participants were told to choose one card and memorize the number or face value (the suit of the card was not used in the subsequent RT test). A hidden closed-circuit video camera was used to see which card the participants chose and to set up the subsequent baseline block. Participants then participated in 300 CTP trials, with the probe being the card they chose and memorized. RTs were recorded during the baseline block, but electroencephalogram (EEG) data were not.

Mock crime CTP P300 concealed information test

157

Table 1. Stimulus Probabilities Stimulus type Probe Target Probe Nontarget Irrelevant Target Irrelevant Nontarget All Probes All Irrelevants

Number

Probability

22 21 22 235 43 257

0.073 0.073 0.073 0.783 0.143 0.857

Note: Target/Nontarget means the previous stimulus (Probe or Irrelevant) was followed by a Target or Nontarget stimulus.

Mock Crime Upon completion of the baseline block, participants in the SG and CM conditions participated in a mock crime. Participants were instructed that they were to carry a large manila envelope to the office of the Psychology department and to tell the secretary that they needed to place a document in Dr. Rosenfeld’s mail box. They were told that there would be an identical envelope labeled ‘‘For Dr. Rosenfeld’’ already in the mail box. They were instructed to place their envelope in the box, and surreptitiously remove the other envelope from the box. They were told that there would be an item in the other envelope which they needed to steal and bring back to the lab. Participants were never told what the item in the envelope was to ensure that any guilty knowledge of the mock crime was obtained solely from the episodic memory of the act. In an attempt to increase the arousal and sense of realism participants felt during the commission of the mock crime, they were told that the secretaries were not aware of the study that they were participating in. They were told to try their best not to get caught stealing the item out of the envelope and that if they were caught and confronted to contact the lab. While subjective arousal was not measured, many participants initially expressed hesitation to commit the mock crime before deciding with informed consent to participate. In a follow-up study, two participants declined to commit the mock crime after receiving the directions, which suggests that the scenario is realistic enough to make participants anxious. After taking the item (a ring), participants were told to hide it on their person and come back to the lab for their CTP test. Unknown to the participants, the secretaries in the office were always informed when a participant was on the way to complete the mock crime. Those participants assigned to the innocent control condition were given instructions to walk down to the office and come back without doing anything else. Experimental Block The protocols for the experimental block were identical to those of the baseline block, but the stimuli were possible items stolen in the mock crime. The word ‘‘RING’’ was the probe and other items, WALLET, EARRING, WATCH, LOCKET, NECKLACE, and BRACELET, were the six irrelevant stimuli. Participants were run in the experimental block until 300 artifactfree trials were collected. All participants were told that the first stimulus in each trial could be an item that was stolen from the office. Those participants assigned to the CM condition were taught countermeasures prior to the beginning of the test. They were given a list of six countermeasures (left index finger pressure on the leg, left thumb pressure on the leg, left big toe wiggle, right big toe wiggle, both big toe wiggles, and imagining the experimenter slapping you), one for each irrelevant, to execute prior to

the ‘‘I Saw It’’ response. The point of the CMs is to evoke P300s to the irrelevant stimuli in an attempt to defeat the probe-irrelevant comparison, since guilt is defined by the finding of probe amplitude being greater than the average irrelevant amplitude. These specific CMs were utilized because they had previously been shown to be effective at defeating the three-stimulus P300 CIT (Rosenfeld et al., 2004). Moreover, Sasaki, Hira, and Matsuda (2001) showed that passive CMs were ineffective in a P300based CIT. After the experimenter was finished attaching the electrodes to the subject’s face and scalp, participants were asked to repeat the six CMs to make sure they had memorized them. Instead of assigning these CMs to stimuli prior to the test starting, participants were told to assign them to stimuli as they came up during the test, and to assign one CM to each of the meaningless or irrelevant stimuli they saw, and to repeat that CM each time that item appeared. IC and SG participants were not given CM instructions. It is noted that, for this type of CM (one that converts an irrelevant into a covert target) to be effective, each irrelevant must have a unique assigned CM. Data Acquisition The baseline EEG level was defined as a 100-ms pre-trial voltage average. EEG recordings were taken using Ag/AgCl electrodes at the Fz, Cz, and Pz sites. Analysis was limited to the Pz site. EEG electrodes were referenced to linked mastoids. Electrooculogram (EOG) was differentially amplified and recorded diagonally across the top and bottom of the right eye to pick up both horizontal and vertical eye artifacts. A grounded electrode was attached to the middle of the forehead. The artifact rejection criterion was set at 80 uV. Artifact filtering was done on-line, and trials in which an artifact was detected were discarded. Signals were passed through Grass P511K amplifiers (Astro-Med, Inc., West Warwick, RI) with a 30-Hz low-pass filter setting, and highpass filters set (3 db) at 0.3Hz. Amplifier output was passed to a 12-bit Keithly Metrabyte A/D converter (Keithley Instruments, Inc., Cleveland, OH) sampling at 100 Hz. All trials used in the analysis were filtered off-line to remove higher frequencies; 3 db point 5 4.23 Hz. P300 amplitude was measured using a peak-to-peak measure, which was defined as the difference between the maximum positivity in the P300 component of the ERP minus the minimum negativity in the portion of the ERP immediately following the P300. This measure has been shown in the past to provide more accurate detection rates than a baseline-to-peak measure (Meijer, Smulders, Merckelbach, & Wolf, 2007; Soskins, Rosenfeld, & Niendam, 2001). A standard look window was used for all participants. It searched for the largest 100 ms average positivity between 300–700 ms. The midpoint of this 100-ms segment was defined as P300 latency. After finding this segment, the algorithm searches from P300 latency to 1500 ms for the maximum 100 ms negativity. The p-p measure is defined as the difference between the maximum positivity and negativity. The look windows used did not extend past the time for the earliest S2 presentation, to ensure that the ERP being measured was driven by the critical first stimulus.

Results All within-subjects analysis of variance (ANOVA) with df41 are reported with Greenhouse-Geisser (GG) p-value correction. Partial eta squared (Z2) values are presented where relevant.

158

M. R. Winograd & J. P. Rosenfeld

Behavioral I Saw It Reaction TimesFGroup Effects Average probe and average irrelevant (Iall) RTs are presented in Table 2. For analysis, irrelevant RTs were collapsed over all stimuli. In general, probe and Iall RTs did not appear to be different within a group, except for the experimental block in the CM group. RTs for SG and IC participants appeared to be quicker in the experimental block, likely due to a practice or habituation effect. A one-way (six levels) repeated measures ANOVA across all groups showed no significant RT differences across individual irrelevants, F(5,165) 5 1.53, p4.2. Separate 3 ! 2 mixed-model ANOVAs (Group ! Stimulus) were run on both the Baseline and Experimental blocks. The Baseline block yielded a significant main effect of stimulus, F(1,33) 5 22.0, po.001, Z2 5 .40, with probe RTs greater than Iall RTs. There was no main effect of condition, p4.13, or stimulus ! group interaction, p4.3. This shows that there were no differences between groups during the baseline block. In the Experimental block, main effects of group, F(2,33) 5 58.3, po.001, Z2 5 .78, and stimulus, F(1,33) 5 16.4, po.001, Z2 5 .33, were found. On average, Iall RTs were longer than probe RTs, clearly (Table 2) due to the Iall RT increase in the CM group. There was also a significant interaction of stimulus ! group, F(2,33) 5 15.9, po.001, Z2 5 .49. Tukey’s HSD (honestly significant differences) post-hoc tests revealed significant differences between the CM group and both the SG and IC, both po.001, with RTs in the CM group being larger in both cases, with no significant differences in post-hoc tests between the RTs in the SG and IC groups. It therefore appears that the main effect of stimulus and the interaction were driven by the differential increase in RTs between probe and Iall in the CM group, which can be seen in Table 2. ‘‘I Saw It’’ RT differences between the Baseline and Experimental blocks clearly identified which participants were using CMs (see Figure 1). RTs from the baseline block were subtracted from RTs in the experimental block to yield an RT difference for each stimulus type (probe and Iall). A 3 ! 2 mixed-model ANOVA (Group ! Stimulus RT difference) yielded a main effect of group, F(2,33) 5 45.5, po.001, Z2 5 .7, and stimulus, F(1,33) 5 23.7, po.001, Z2 5 .42. Tukey’s HSD post-hoc comparisons showed significant differences between the CM block and both the SG and IC (both po.001), with the CM group having larger ‘‘I Saw It’’ RT differences. There were no differences between the SG and IC groups. The interaction term was also significant, F(2,33) 5 17.7, po.001, Z2 5 .52. RTs decreased from the baseline to the experimental block for the SG and IC groups while RTs increased for the CM group, with a larger increase in RT for Iall than for probe. I Saw It Reaction TimesFIndividual Effects When the previous individual ‘‘I Saw It’’ RT differences for probe and Iall are summed, there is no overlap in distributions

Table 2. Mean Probe and Iall ‘‘I Saw It’’ RTs in Baseline and Experimental Blocks by Group Group SG CM IC

Base-P

Base-Iall

Exp-P

Exp-Iall

401.8 483.9 469.1

390.8 460.3 442.2

392.8 806.5 393.7

393.4 1196.9 397.3

Figure 1. Distributions of combined RT differences between experimental and baseline blocks by group. Combined RT difference was computed by subtracting the RTs from the baseline block from the experimental block for probe and Iall, and then summing these values together.

(see Figure 1) between the CM and either the SG or IC groups, suggesting that RTs are a usable metric for diagnosing this physical and mental assigned response method of CM use. Mean experimental–baseline RT differences for each group were " 6.5 ms (SG), 1059.2 ms (CM), and " 120.3 ms (IC). The difference between the slowest SG subject and fastest CM subject was 77.6 ms. However, this CM subject’s combined RT difference (230.8 ms) was 353.8 ms faster than the next slowest, suggesting this subject struggled to properly execute the CMs. This subject was correctly diagnosed as guilty. CM use was also identifiable utilizing a comparison of probe and Iall RTs within the experimental block. All but one of the CM participants showed an RT difference (Iall RT–probe RT) of at least 70 ms (M 5 390.4 ms). The one subject that did not was the aforementioned subject, who had a difference of " 115.7 ms. The greatest differences in the SG and IC groups were 36.8 ms and 50.8 ms, respectively. While the one CM subject did not show the expected pattern of slower irrelevant responses, his increase in RTs between the baseline and experimental blocks suggest he was attempting to do the CMs. Excluding this subject, there is once again no overlap in RT differences (Iall–probe) between the CM and either the SG (M 5 0.6 ms) or IC (M 5 3.7 ms) groups. ERPs Qualitative Grand-average P300 waveforms (from the Pz site) are presented in Figure 2. Waveforms are shown for the probe and Iall stimuli in each group. Grand averages are shown through the first 1500 ms of each trial. There are clear differences between probe and Iall stimuli in the SG and CM groups, with amplitudes for both stimulus types increasing during the use of CMs. P300 amplitudes in the IC group appear to be equal for probe and Iall. The grand average of probes from each block shows how there is virtually no P300 to probes in the IC block, a noticeable one for the SG participants, and a larger, more prominent P300 to probes in the CM group.


159

Figure 2. Grand-average ERPs at Pz site.

QuantitativeFP300 Amplitude All ERP analyses were completed using the data collected from the Pz site, where P300 amplitude is known to be largest (Fabiani, Gratton, Karis, & Donchin, 1987). However, the Cz site was used for one subject whose Pz electrode failed during EEG recording. Mean calculated p-p probe and Iall P300 amplitudes are presented in Figure 3. The P300 amplitude values were analyzed using a 3 ! 2 mixed-model ANOVA (Group ! Stimulus) which yielded a significant main effect of group,

F(2,33) 5 5.2, po.05, Z2 5 .24, and stimulus, F(1,33) 5 63.0, po.001, Z2 5 .66, with average probe amplitudes greater than Iall. The test for a group ! stimulus interaction was also significant, F(2,33) 5 23.8, po.001, Z2 5 .59. In order to break down the significant interaction, post-hoc t-tests were performed comparing probe and Iall amplitude within each group. These tests revealed significant differences in probe (M 5 8.55, SD 5 3.99) and Iall (M 5 5.12, SD 5 1.73) amplitude in the SG, t(11) 5 4.8, po.001, and the CM groups (Probe M 5 14.72, SD 5 5.35; Iall M 5 8.57, SD 5 4.51), t(11) 5 7.6, po.001. The post-hoc test in the innocent group revealed no difference between probe (M 5 7.73, SD 5 3.73) and Iall (M 5 8.09, SD 5 3.77). The significant interaction was thus due to larger probe amplitudes than Iall amplitudes in the two guilty conditions but not in the IC condition. Individual Detection Rates Detection rates for each condition are presented in Table 3. Individual diagnostic classifications were made using the same bootstrap amplitude difference procedure as Rosenfeld et al. (2008) (see also Wasserman & Bockenholt, 1989). In this bootstrap procedure, individual probe and irrelevant trials are sampled, with replacement, from all the non-artifacted probe and Iall sweeps, with separate averages made for each stimulus type. This

Table 3. Individual Bootstrap Diagnostic Hit Rates by Group at 90% Confidence Level Group SG CM IC Figure 3. Mean probe and Iall p-p P300 amplitudes by group.

Detections

Percent correct

10/12 12/12 1/12

83% 100% 92%

160 procedure is repeated 100 times. Within each iteration, a comparison of probe versus Iall P300 amplitude is made, and if the probe is found to be larger than Iall in at least 90% of iterations, the subject is classified as guilty. Overall, 92% (33/36) of participants across the three conditions were correctly classified. In the SG group, 10/12 (83%) were diagnosed as being guilty. Detection rates improved in the CM condition, as 12/12 (100%) of participants were correctly classified as guilty. There was one false positive in the IC group yielding a correct classification rate of 11/12 (92%). The correct classification rate for SG participants (83%) and low false-positive rate (8%) closely match previous detection rates for previous CITs (Vrij, 2008) but there is here an increased resistance to CMs. Grier A 0 Values The Grier A 0 parameter was calculated to determine the overall discriminative ability of the CTP in the mock crime situation. A 0 is calculated based upon the formula by Grier (1971), A 0 5 .51{(y ! x) n (11y ! x)/[4 n y n (1 ! x)]}, with y representing the guilty hit rate and x the false-positive rate. Values for A 0 range from 1.0, representing perfect discrimination, and 0.5, representing chance discrimination. A 0 values were calculated separately for the SG and CM groups using the IC group as the false-positive rate for both. Using the SG group, A 0 5 .93. For the CM group, A 0 5 .98. Overall, these values suggest that the CTP was successful at discriminating guilty and innocent participants in a mock-crime scenario. Discussion The CTP proved to be sensitive to the detection of concealed information incidentally acquired during a mock crime. This is in contrast to Rosenfeld et al. (2006), who found that the older ‘‘three stimulus’’ protocol was not as effective with incidentally acquired information as with autobiographical information. The fact that the CTP is also highly resistant to CMs using assigned responses to irrelevant items that previously had been shown to be highly effective against the three-stimulus CIT (Rosenfeld et al., 2004) suggests that it could be a useful tool in the detection of concealed information in police investigations. With an episodic memory-based probe, the pattern of results found in this CTP study closely matched that which was found when using autobiographical probes (Rosenfeld et al., 2008). The probes in the SG condition were reliably larger than the irrelevants on both an individual and group basis, and this difference was strong enough to accurately diagnose guilt. In the CM condition, the amplitude of the probe increased, rendering this method of CMs ineffective. Additionally, along with high accuracy in detecting guilt, the CTP once again was shown to have a low false-positive rate. Countermeasure use was readily identifiable by an increase in ‘‘I Saw It’’ RTs for both the probe and Iall stimuli in the CM block. The non-overlapping RT distributions, between participants who executed CMs and those who didn’t, support the usage of RTs as a method for identifying this method of CM use. One curious result was observed with P300 amplitudes. Predicting results based upon the oddball effect, it would be expected that one would see an increase in probe amplitude for SG participants. As expected, at both the group and individual levels, the probe in the SG guilty group was reliably larger than Iall. Likewise no differences were found between probe and Iall at the individual or group level in the IC group. However, unexpectedly, the probe and Iall amplitudes in the IC group were the same

M. R. Winograd & J. P. Rosenfeld size as the probe in the SG group (see Figure 3). The only difference between the two groups is decreased Iall amplitude for SG participants. While this could be due to simple group differences between the SG and IC groups, random assignment makes this unlikely. Another possibility is that there may be a limited amount of processing resources available in the brain at any given time. For guilty, but not innocent, participants, the observation of a probe may draw resources away from processing the irrelevant stimuli, reducing their P300s. Amplitudes can still be modulated by attention and test-relevant tasks, such as the additional covert responses made by CM participants. One of the main focuses of this study was to ensure that any reaction the subject had to the probe was unambiguously due to the subject committing the mock crime. In previous P300-based mock crime CITs (Farwell & Donchin, 1991; Mertens & Allen, 2008; Rosenfeld et al., 2004), participants were told in the instructions what item they would be stealing or other details on which they would later be tested. In these studies, the detection of the guilty participants cannot be assumed to be due solely to the commission of the mock crime, as simple knowledge of the item to be stolen or other details may have been enough to evoke P300s. For example, in Mertens and Allen (2008), only one of the twelve probes used in the analysis was unlearned prior to the virtual mock crime. In the present study, however, we can assume that the large P300s observed in the SG and CM participants are due to their recognition of the ring that they stole during the commission of the mock crime, as they had no prior knowledge of the item before completing the mock crime portion of the experiment. A serious problem with laboratory studies, especially those involving deception or lie detection, is a lack of ecological validity. This is particularly relevant in studies where participants are tested on details that were learned prior to the commission of a mock crime. A number of CITs involving mock crimes have been done using ANS polygraph measures (Carmel, Dayan, Ayelet, Raveh, & Ben-Shakhar, 2003; Jokinen, Santtila, Ravaja, & Puttonen, 2006; for a review, see Ben-Shakhar & Elaad, 2003). While the measures obviously differ between CITs using ANS processes and P300s, the principles behind the CIT remain the same, along with the characteristics which impact the validity of any mock crimes used in either situation. These studies found that a detail central to a mock crime, such as a stolen item, is more readily detected than are peripheral details, such as surrounding items in the environment, and that other factors affecting item salience, such as the time delay between a mock crime and testing, could directly influence detection rates in ANS-based CITs. In short, the more salient an item, the better it will be remembered, therefore making it a more sensitive detail for use in a CIT. Taking this into account, one would expect that details that are rehearsed or learned through instructions prior to a mock crime will be more salient than other details that are incidentally acquired through the execution of the crime itself. The sensitivity of the CTP at detecting incidentally acquired knowledge in a mock crime is promising. Even so, more work needs to be done before this or any other method proves to by truly ecologically valid. As noted by Ben-Shakhar and Elaad (2003), many CIT studies involve testing immediately after the commission of a mock crime, which was true in the present study. In a real investigation, a suspect would likely not be tested for at least a few days after a crime has been discovered. However, it has been found that time-delayed testing does not negatively impact detection rates in a P300-based CIT in an eyewitness


161

identification task, even if participants cannot accurately identify a probe with as much accuracy over 1-h, 1-day, or 1-week testing conditions (Lefebvre, Marchand, Smith, & Connolly, 2007). It was found that a P300-based CIT can be sensitive to ‘‘guilty knowledge’’ over differential time delays, even when a subject cannot consciously recall a detail accurately. This finding is encouraging for the potential application of the CTP in these situations. However, our study, as with many others, did not employ double-blind controls, so participants were likely aware that the experimenter knew whether or not they had committed the mock crime, a circumstance less likely in a real investigation. Another limitation of the current study is that only one detail of the mock crime was used as a probe. In a real investigation, one would not convict a suspect based upon a detection on a single item in a CIT. While the low false-positive rate (8%) and 92% overall guilty (combining SG and CM) detection rate suggest that the CTP is indeed accurate at discriminating between innocent and guilty participants, more details would need to be used for field use, as one would not want to determine guilt based

upon just one detail (to which participants in this study had a .2 probability of reacting). Finally, although participants were not offered a financial bonus for attempting to beat the test, the fact that participants assigned to the CM condition showed the predicted increase in RT and increased Iall P300 suggest that they indeed did make a conscious effort to appear innocent. Despite these concerns, we have shown here that the CTP is effective for the detection of episodically learned concealed information. Additionally, the CTP remained resistant to CM responses assigned to irrelevant items, and which were readily detectable through RT analysis. To more fully examine the applicability and validity of the CTP to real-life situations, the CTP will need to be further tested using multiple probes under a variety of time-delayed testing conditions. While laboratory studies will never fully replicate the characteristics of a real crime (such as subject arousal and punishment threat), the current results of the CTP are encouraging. Future studies focusing on the changes noted above will go a long way in establishing the CTP as being an ecologically valid test for the detection of guilty knowledge.

REFERENCES Allen, J. J. B., & Iacono, W. G. (1997). A comparison of methods for the analysis of event-related potentials in deception detection. Psychophysiology, 34, 234–240. Allen, J., Iacono, W. G., & Danielson, K. D. (1992). The indentification of concealed memories using the event-related potential and implicit behavioral measures: A methodology for prediction in the face of individual differences. Psychophysiology, 29, 504–522. Ben-Shakhar, G., & Elaad, E. (2003). The validity of psychophysiological detection of information with the Guilty Knowledge Test: A metaanalytic review. Journal of Applied Psychology, 88, 131–151. Carmel, D., Dayan, E., Ayelet, N., Raveh, O., & Ben-Shakhar, G. (2003). Estimating the validity of the Guilty Knowledge Test from simulated experiments: The external validity of mock crime studies. Journal of Experimental Psychology: Applied, 9, 261–269. Donchin, E., & Coles, M. G. H. (1988). Is the P300 component a manifestation of context updating? Behavioral and Brain Sciences, 11, 355–372. Grier, J. B. (1971). Non-parametric indexes for sensitivity and bias: Computing formulas. Psychological Bulletin, 75, 424–429. Fabiani, M., Gratton, G., Karis, D., & Donchin, E. (1987). The definition, identification and reliability of measurement of the P300 component of the event-related brain potential. In P. K. Ackles, J. R. Jennings, & M. G. H. Coles (Eds.), Advances in psychophysiology (Vol. 2, pp. 1–78). Greenwich, CT: JAI Press. Farwell, L. A., & Donchin, E. (1991). The truth will out: Interrogative polygraphy (‘‘lie detection’’) with event-related potentials. Psychophysiology, 28, 531–547. Honts, C. R., & Amato, S. L. (2002). Countermeasures. In M. Kleiner (Ed.), Handbook of polygraph testing (pp. 251–264). San Diego, CA: Academic Press. Jokinen, A., Santtila, P., Ravaja, N., & Puttonen, S. (2006). Salience of guilty knowledge test items affects accuracy in realistic mock crimes. International Journal of Psychophysiology, 62, 175–184. Lefebvre, C. D., Marchand, Y., Smith, S. M., & Connolly, J. (2007). Determining eyewitness identification accuracy using event-related potentials (ERPs). Psychophysiology, 44, 894–904. Lykken, D. T. (1959). The GSR in the detection of guilt. Journal of Applied Psychology, 43, 385–388. MacLaren, V. V. (2001). A quantitative review of the guilty knowledge test. Journal of Applied Psychology, 86–4, 674–683. Meijer, E. H., Smulders, F. T. Y., Merckelbach, H. L. G. J., & Wolf, A. G. (2007). The P300 is sensitive to face recognition. International Journal of Psychophysiology, 66, 231–237. Mertens, R., & Allen, J. B. (2008). The role of psychophysiology in forensic assessments: Deception detection, ERPs, and virtual reality mock crime scenarios. Psychophysiology, 45, 286–298.

Rosenfeld, J. P., Angell, A., Johnson, M., & Qian, J. (1991). An ERPbased control-question lie detector analog: Algorithms for discriminating effects within individuals’ average waveforms. Psychophysiology, 38, 319–335. Rosenfeld, J. P., Biroshak, J. R., & Furedy, J. J. (2006). P-300-based detection of concealed autobiographical versus incidentally acquired information in target and non-target paradigms. International Journal of Psychophysiology, 60, 251–259. Rosenfeld, J. P., Cantwell, G., Nasman, V. T., Wojdac, V., Ivanov, S., & Mazzeri, L. (1988). A modified, event-related potential-based guilty knowledge test. International Journal of Neuroscience, 24, 157–161. Rosenfeld, J. P., Labkovsky, E., Winograd, M., Lui, M., Vandenboom, C., & Chedid, E. (2008). The Complex Trial Protocol (CTP): A new, countermeasure-resistant, accurate P300-based method for detection of concealed information. Psychophysiology, 45, 906–919. Rosenfeld, J. P., Tang, M., Meixner, J. B., Winograd, M. R., & Labkovsky, E. (2009). The effects of asymmetric versus symmetric probability of targets following probe and irrelevant stimuli in the Complex Trial Protocol for detection of concealed information with P300. Physiology & Behavior, 98, 10–16. Rosenfeld, J. P., Soskins, M., Bosh, G., & Ryan, A. (2004). Simple effective countermeasures to P300-based tests of detection of concealed information. Psychophysiology, 41, 205–219. Sasaki, M., Hira, S., & Matsuda, T. (2001). Effects of mental countermeasure on the physiological detection of deception using the eventrelated brain potentials. Japanese Journal of Psychology, 72, 322–328. Seymour, T. L., Seifert, C. M., Shafto, M. G., & Mosmann, A. L. (2000). Using response time measures to assess ‘‘guilty knowledge.’’ Journal of Applied Psychology, 85, 30–37. Soskins, M., Rosenfeld, J. P., & Niendam, T. (2001). The case for peakto-peak measurement of P300 recorded at .3 Hz high pass filter settings in detection of deception. International Journal of Psychophysiology, 40, 173–180. Verschuere, B., Crombez, G., Degrootte, T., & Rosseel, Y. (2009). Detecting concealed information with reaction times: Validity and comparison with the polygraph. Applied Cognitive Psychology, 23, 1–11. Verschuere, B., Rosenfeld, J. P., Winograd, M., Labkovsky, E., & Wiersema, J. R. (2009). The role of deception in P300 memory detection. Legal and Criminal Psychology, 14–2, 253–262. Vrij, A. (2008). Detecting lies and deceit: Pitfalls and opportunities (Second Edition). Reading, MA: Wiley and Sons, Ltd. Wasserman, S., & Bockenholt, U. (1989). Bootstrapping: Applications to psychophysiology. Psychophysiology, 26, 208–221. (Received July 15, 2009; Accepted February 12, 2010)


REVIEW

The anatomical and functional relationship between the P3 and autonomic components of the orienting response

SANDER NIEUWENHUIS,a,b ECO J. DE GEUS,c and GARY ASTON-JONESd a

Leiden Institute for Brain and Cognition, Leiden University, Leiden, The Netherlands Institute of Psychology, Leiden University, Leiden, The Netherlands Department of Biological Psychology, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands d Department of Neurosciences, Medical University of South Carolina, Charleston, South Carolina b c

Abstract Many psychophysiologists have noted the striking similarities between the antecedent conditions for the P3 component of the event-related potential and the orienting response: both are typically elicited by salient, unexpected, novel, taskrelevant, and other motivationally significant stimuli. Although the close coupling of the P3 and orienting response has been well documented, the neural basis and functional role of this relationship is still poorly understood. Here we propose that the simultaneous occurrence of the P3 and autonomic components of the orienting response reflects the co-activation of the locus coeruleus-norepinephrine system and the peripheral sympathetic nervous system by their common major afferent: the rostral ventrolateral medulla, a key sympathoexcitatory region. A comparison of the functional significance of the locus coeruleus-norepinephrine system and the peripheral sympathetic nervous system suggests that the P3 and orienting response reflect complementary cognitive and physical contributions to the mobilization for action following motivationally significant stimuli. Descriptors: P300, P3, Orienting response, Norepinephrine, Noradrenergic, Rostral ventrolateral medulla, Arousal, Reticular formation

The orienting response is a collection of physiological changes that are elicited by the occurrence of motivationally significant events (Lynn, 1966; Pavlov, 1927; Sokolov, 1963). These changes include a temporary dilation of the pupils, a rise in skin conductance, a momentary change in heart rate, and a range of other short-lived changes in organ activity. Although the orienting response should probably be regarded as a convenient grouping of physiological changes rather than as a unitary construct (e.g., Barry, 1979, 2009; Kahneman, 1973), these changes generally occur together, typically accompanied by a shift of attention toward the eliciting event. While the precise functional significance of the orienting response has been a topic of much debate, there appears to be consensus that it serves to potentiate information processing and to prepare or facilitate a rapid behavioral response to the eliciting stimulus (even if such action is not always undertaken; e.g., Donchin, Heffley, Hillyard, Loveless, Maltzman, et al., 1984; Lynn, 1966). An important question that occupied many psychophysiologists in the late 1970s and early 1980s concerned the neural correlates of the orienting response. In particular, they debated whether the electrophysiological P3 should be seen as the central nervous system counterpart to the autonomic components of the orienting response, and how explanations of the functions of these phenomena might be integrated within one theoretical framework (Donchin et al., 1984; Friedman, 1978; Kimmel, Van

In the past decade, the use of functional neuroimaging techniques has greatly aided our understanding of the cortical and subcortical brain structures involved in regulating the autonomic nervous system, and in representing bodily states (Berntson, Sarter, & Cacioppo, 2003; Critchley, 2005). It has long been known that the interplay between the central and autonomic nervous system is not just important for homeostatic regulation, but also an essential component of active, stimulus-driven behavior. As an important example, motivationally significant stimuli1 are typically followed by a phasic response of the autonomic nervous system, often referred to as the orienting response, along with a transient change in brain activity known as the P3 or P300. However, although the close coupling between these two phenomena has been well documented, the functional role and neural basis of this relationship is still poorly understood. Here, we propose an hypothesis that attempts to explain this relationship.

This research was supported by the Netherlands Organization for Scientific Research. Address correspondence to: Sander Nieuwenhuis, Cognitive Psychology Unit, Leiden University, Wassenaarseweg 52, 2333 AK, Leiden, the Netherlands. E-mail: [email protected] 1 By motivationally significant stimuli, we mean stimuli that are either relevant to the current task or that have the potential to be associated with some form of utility (positive or negative). 162

P3 and orienting response Olst, & Orlebeke, 1979). It was clear to most researchers that the P3 and orienting response have very similar antecedent conditions (Ritter, Vaughan, & Costa, 1968). These antecedent conditions indicate that both phenomena reflect more closely the motivational significance of eliciting stimuli (as determined by their inherent value, task instructions, recent experience, and other factors) than their physical characteristics per se. To illustrate, both the P3 (Sutton, Tueting, Zubin, & John, 1967) and the orienting response (Sokolov, 1963) can be elicited by the absence of a stimulus when that absence delivers information to the subject. These and other similarities as well as some apparent discrepancies fuelled discussion about the functional relationship between the P3 and the orienting response. However, we believe it is fair to say that no satisfactory integrative theoretical framework emerged from these years of discussion, and in the second half of the 1980s interest in the link between P3 and the orienting response quickly diminished. There seem to be at least three reasons for this course of events. First, a fair amount of initial confusion, which impeded theoretical progress, was caused by the fact that the paradigms used to study the P3 generally differed from those traditionally employed to study the autonomic components of the orienting response. Specifically, the former studies tended to focus on the P3 to taskrelevant stimuli (which shows very little habituation over the course of an experiment), whereas the latter studies focused on the orienting response to task-irrelevant stimuli in passive observers (which habituates rapidly). Fortunately, several researchers noted the artifactual nature of the ensuing discrepancies in results, and this issue of debate was resolved (for a more detailed summary, see Donchin et al., 1984). A second challenge for the development of an integrative theoretical framework concerned the gap between prevalent conceptions of the functional role of the orienting response and the most influential theory of the P3: the context-updating hypothesis (Donchin & Coles, 1988). As noted above, a common and natural interpretation of the orienting response is that it serves to prepare or facilitate rapid action in response to the eliciting stimulus. This action-oriented view stands in marked contrast with the context-updating hypothesis, which posits a strategic role for the process underlying the P3: updating of a cognitive schema of the environment. Although, according to these interpretations, the orienting response (action preparation) and the P3 (context updating) may be triggered by very similar antecedent conditions, and hence could be considered correlates of each other, their action-based vs. memory-based contributions to goal-directed behavior are of a distinct nature (Donchin, 1981). Finally, evidence from intracranial recordings (in humans and animals) and functional imaging suggested the involvement of multiple, and diverse brain areas in generating the P3 (reviewed in Soltani & Knight, 2000), and it proved difficult to integrate this evidence in a comprehensive account of the neural basis of the P3. Therefore, researchers were lacking a neurobiological framework with which to correlate the at the time sparse knowledge of the brain areas involved in eliciting the orienting response. Thus, for a long time, the available knowledge and theoretical conceptions of the P3 made it hard to understand the link between this component and the orienting response. However, recent research has led to a new, detailed theory of the neural basis and functional significance of the P3 (Nieuwenhuis, AstonJones, & Cohen, 2005). As we will argue, this theory sheds new

163 light on the relationship between the P3 and the orienting response, suggesting a close correspondence between the two phenomena at both the neurobiological and functional levels. The remainder of this article is organized as follows. We first review the empirical evidence for a close link between the P3 and the orienting response, focusing in particular on the similarity in antecedent conditions. The scope of this review is modest, in particular in the sense that it does not cover many of the subtleties of the orienting response, which are discussed in detail elsewhere (e.g., Sokolov, Spinks, Naä¨ta¨nen, & Lyytinen, 2002). Furthermore, we limit our discussion to two of the autonomic components of the orienting response that have received the most attention in the context of the P3: the skin conductance response (SCR) and the pupil dilation response (PDR). The SCR is entirely driven by changes in the sympathetic nervous system (SNS) whereas the PDR is subserved by closely coupled SNS and parasympathetic inputs. Although phasic heart rate changes have also been a popular measure of the autonomic orienting response to motivationally significant stimuli (for excellent reviews, see Graham & Clifton, 1966; Simons, 1988), they mainly reflect parasympathetic (or vagal) inhibitory influences, which are strongly modulated by respiratory behavior and baroreflex activation (e.g., Berntson, Cacioppo, & Quigley, 1993). Because these influences complicate interpretation, phasic heart rate changes will not be discussed in our review. Following this brief review, we will summarize the theory of the P3 recently proposed by Nieuwenhuis, Aston-Jones, and Cohen (2005). According to this theory, the P3 reflects the response of the neuromodulatory locus coeruleus–norepinephrine system to the outcome of stimulus evaluation and perceptual decision making. Furthermore, the theory holds that the observed properties of the P3 reflect an important information processing function of the locus coeruleus–norepinephrine system, which is to potentiate the response to motivationally significant events. In the final sections of the article, we suggest how this theory and knowledge about the anatomy of the locus coeruleus–norepinephrine system can be used to leverage our understanding of the neurobiological and functional relationship between the P3 and the orienting response.

Similarities Between the P3 and Autonomic Components of the Orienting Response An extensive review of the P3, SCR, and PDR is beyond the scope of this paper, and can be found elsewhere (e.g., Janisse, 1977; Nieuwenhuis, Aston-Jones, & Cohen, 2005; Pritchard, 1981; Roth, 1983). Our goal here is merely to illustrate the point that the P3 shares many properties with phasic sympathetic responses reflected in the SCR and the PDR. The P3 The P3 is a broad, positive, large-amplitude potential with a parieto-central scalp distribution, and a typical peak latency between 300 and 400 ms following presentation of stimuli in any sensory modality (Sutton, Braren, Zubin, & John, 1965). An important factor affecting the amplitude of the P3 is the subjective probability of the eliciting stimulus (Donchin & Coles, 1988). The effect of stimulus probability on P3 amplitude has been thoroughly documented using the oddball task. In this task, lowfrequency target stimuli (‘‘oddballs’’) are embedded in a train of non-target stimuli (‘‘standards’’), and the subject’s task is either

164 to actively respond to each target stimulus, or to passively attend to the stimulus sequence. Using this task, it has been found that the amplitude of the P3 associated with targets and non-target stimuli is inversely related to their probability of occurrence (Duncan-Johnson and Donchin, 1977). The effect of target probability is mediated, at least in part, by differences in targetto-target interval (Croft, Gonsalvez, Gabriel, & Barry, 2003). Furthermore, the amplitude of the P3 to the oddball stimulus is proportional to the degree of deviation from the standards (e.g., in terms of tone pitch; Ford, Roth, & Kopell, 1976). Although both targets (i.e., stimuli requiring a response) and non-target stimuli can elicit a P3, when equated for frequency of occurrence, targets typically elicit somewhat higher P3 amplitudes than nontarget stimuli (e.g., Duncan-Johnson & Donchin, 1977). This indicates that P3 amplitude is also sensitive to the motivational significance of the eliciting stimulus. In laboratory contexts, such as the oddball task, stimuli often derive their motivational significance from a set of, in some sense, arbitrary task instructions. However, the P3 is also sensitive to stimuli with more intrinsic significance. For example, emotionally valent stimuli, whether experienced as positive or negative, are associated with larger P3s than emotionally neutral stimuli (Johnston, Miller, & Burleson, 1986; Yeung & Sanfey, 2004). Moreover, the P3 can be elicited by the absence of a stimulus when that absence delivers important information to the subject (Sutton et al., 1967), which further illustrates that the P3 is sensitive to the significance rather than physical properties of a stimulus. The effects of subjective probability and motivational significance on P3 amplitude are modulated by a third variable, the amount of attention paid to the stimulus (Johnson, 1993). Specifically, the same stimuli that would under normal circumstances elicit a robust P3, elicit no P3 or a P3 of much smaller amplitude when they are deliberately ignored or when subjects’ attention is occupied by another, secondary task (Duncan-Johnson & Donchin, 1977). A P3 will be observed only if an initially unattended stimulus has sufficient intensity to capture attention and intrude into consciousness (Ritter et al., 1968). Indeed, the only physical property that systematically affects the P3 is stimulus intensity, which is positively correlated with P3 amplitude (Covington & Polich, 1996; Roth, Dorato, & Kopell, 1984). Highly deviant or salient task-irrelevant stimuli, such as infrequently presented loud sounds, can be regarded as a specific class of motivationally significant, attention-capturing stimuli. The P3 elicited by this class of stimuli (often labeled P3a) has a number of properties that distinguish it from the typical P3 (or P3b) associated with task-relevant stimuli (Friedman, Cycowicz, & Gaeta, 2001; Simons, Graham, Miles, & Chen, 2001; Spencer, Dien, & Donchin, 2001): (i) its scalp distribution has a prominent fronto-central focus; (ii) it peaks 60–80 ms earlier than the P3b; and (iii) its amplitude shows rapid habituation as the novelty or salience of task-irrelevant stimuli decreases with repeated presentations (Courchesne, Hillyard, & Galambos, 1975; Roth, 1973; see Rushby & Barry, 2007 for more stringent habituation criteria), although the evidence for such habituation at long interstimulus intervals (430 s in traditional orienting response studies) is mixed (e.g., Rushby & Barry, 2009; Rust, 1977; Simons, Rockstroh, Elbert, Fiorito, Lutzenberger, & Birbaumer, 1987). Various lines of evidence indicate that the P3 is intimately related to task performance. For example, under the attentiondemanding circumstances presented by signal-detection tasks, P3 amplitude varies directly with detection and recognition performance on signal-present trials (e.g., Hillyard, Squires, Bauer, &

S. Nieuwenhuis et al. Lindsay, 1971). More specifically, stimuli that elicit a large P3 have a higher chance of being accurately discriminated. Similar findings have been obtained with the attentional blink paradigm (cf. Nieuwenhuis, Gilzenrat, Holmes, & Cohen, 2005). Furthermore, in speeded two-choice reaction time tasks, P3 latency and reaction time generally covary across trials (Makeig, Delorme, Westerfield, Jung, Townsend, et al., 2004; Pfefferbaum, Ford, Roth, & Kopell, 1980; Ritter, Simson, & Vaughan, 1972) with the peak of the P3 generally occurring around the time of the response. They also covary across task conditions when these affect the duration of stimulus encoding or the decision process (for a review and some exceptions, see Verleger, 1997). Finally, P3 amplitude and reaction time are negatively correlated across trials in the oddball task (Holm, Ranta-aho, Sallinen, Karjalainen, & Mu¨ller, 2006; Li, Keil, & Principe, 2009; Suwazono, Shibasaki, Nishida, Nakamura, Honda, et al., 1994). Together, these results are consistent with the notion that the P3 process serves to facilitate behavioral responses (Nieuwenhuis, AstonJones, & Cohen, 2005). The P3 process may also facilitate internal ‘‘responses’’ such as the encoding of information into long-term memory. For example, P3 amplitude to a stimulus is predictive of later recall of that stimulus (Karis, Fabiani, & Donchin, 1984). To summarize, the process underlying the P3 is driven by the motivational significance and frequency of task-relevant stimuli while being relatively insensitive to their physical attributes. In addition, a P3 with a slightly earlier timing and a more frontal scalp distribution occurs to task-irrelevant stimuli that are salient by virtue of novelty or intensity. Finally, the P3 is closely associated with the speed and accuracy of responding. The Skin Conductance Response The eccrine sweat glands are innervated by efferent neurons from the sympathetic axis of the autonomic nervous system. The primary function of most eccrine sweat glands is thermoregulation, but the eccrine glands located on the palms and soles of the feet may be more concerned with grasping behavior than with evaporative cooling (Edelberg, 1972). Furthermore, it has been suggested that these glands are more responsive to emotional stimuli than to thermal stimuli (Dawson, Schell, & Filion, 2000). Transient changes in SNS activity are reflected in measurable changes in skin conductance at the surface, where the activity modulates the conductance of an applied current to the skin. The SCR to external stimuli consists of a rise in conductance beginning more than 1 s following stimulation, a peak around 1 s following onset, and a slow recovery to baseline. This pattern can be readily measured on single trials. The antecedent conditions for the SCR have been thoroughly investigated. In this research, subjects are typically passive observers, and the intervals between consecutive stimuli are relatively long so that the SCR to one stimulus has time to evolve before the next stimulus appears. No study has systematically examined the relationship between stimulus probability and SCR amplitude, presumably because of the large number of (longduration) trials required. However, one robust finding is that unexpected stimulus change is sufficient to produce a reliable SCR (Siddle, O’Gorman, & Wood, 1979). For example, in the study of Siddle, Remington, and Churchill (1984) subjects watched a sequence of 41 stimuli, spaced at random intervals, all longer than 20 s. Half of the subjects saw 40 letter stimuli (H and F), followed by a shape stimulus (diamond or triangle). The other half of the subjects saw 40 shape stimuli followed by a

P3 and orienting response letter. The data indicated a substantially increased SCR on the change trial compared to the two preceding non-change trials, and this increase was independent of the identity of the change stimulus (letter or shape). Other studies have found that the size of the SCR is proportional to the degree of (e.g., physical or semantic) mismatch between the standards and the oddball (e.g., Siddle & Heron, 1976). Motivational significance (e.g., task relevance) is also an important determinant of SCR amplitude (Bernstein, 1979; Maltzman, 1979a, b). Instructing the subject to perform a voluntary response to a stimulus generally increases the corresponding SCR compared to when the stimulus is not associated with any task (Bernstein & Taylor, 1979; Siddle et al., 1979). Bernstein, Taylor, and Weinstein (1975) reported enhanced SCRs when subjects were required to respond to a designated class of auditory stimulus in a sequence of tones, compared to when they were asked to merely listen to the sequence. In addition, when subjects were instructed to respond only when the target was presented to one of their ears, the target-evoked SCR was larger for targets presented to the task-relevant compared to the task-irrelevant ear. Enhanced SCRs have also been observed in response to conditioned stimuli that indicate a high probability or intensity of physical punishment (O¨hman, Bjorkstrand, & Ellstrom, 1973), and to other stimuli with learned significance, such as one’s own name (Siddle et al., 1979). Finally, the unexpected omission of an unconditioned stimulus can elicit a SCR, again illustrating that it is the meaning rather than the physical properties of an event that elicits the orienting response. Like P3 amplitude, SCR amplitude increases monotonically with stimulus intensity (Barry, 1975; Jackson, 1974; Turpin & Siddle, 1979). This effect appears independently of whether subjects are passively observing or actively responding to the stimuli (Roth et al., 1984). The initial SCR to salient, task-irrelevant stimuli (often referred to as involuntary orienting response; Maltzman, 1979a) habituates with repeated stimulus presentations (Barry, Feldman, Gordon, Cocker, & Rennie, 1993), with the rate of habituation being slower for more salient stimuli (Raskin, Kotses, & Bever, 1969). In contrast, the SCR to task-relevant stimuli (the voluntary orienting response) usually shows little or no habituation (e.g., Van Olst, Heemstra, & Ten Kortenaar, 1979). The amplitude of the SCR response upon initial presentation of a stimulus is directly related with the probability of long-term recall of that stimulus (Kleinsmith & Kaplan, 1964; Maltzman, Kantor, & Langdon, 1966), mirroring the relationship between P3 amplitude and recall. Although the antecedent conditions for the P3 and SCR are highly similar across different experiments, the few studies that directly compared these measures within the same experiment offer mixed results. Such direct comparisons have been uncommon because they are limited by methodological factors. The low signal-to-noise ratio of event-related potentials (ERPs) demands that they are averaged across many trials. ERPs are rapid (o1 s) and interstimulus intervals are usually short to allow for many repeated trials. In contrast, the SCR does not require averaging and is typically investigated using long interstimulus intervals (410 s) to allow for its protracted time course, thus limiting the total number of trials that can be obtained. Verbaten (1983) measured the P3 and SCR to repeated presentations of schematic pictures while requiring the subjects to either passively watch stimuli or memorize them. Regardless of the instruction, the amplitude of the frontocentral P3 and the SCR showed a significant decrease over multiple stimulus

165 presentations, whereas the posterior P3 did not. Two other studies, both using an active auditory oddball task, compared the P3 across trials with and without a SCR (Bahramali, Gordon, Lim, Li, Lagopoulos et al., 1997; Halgren & Marinkovic, 1995). In both studies, the P3 was reliably larger for SCR-present than for SCR-absent trials, but only at frontocentral electrodes; at posterior electrodes the P3 showed very little difference between these trial groups. Lyytinen, Blomberg, and Naä¨ta¨nen (1992) have reported similar results with a passive auditory oddball task. Roth, Blowers, Doyle, and Kopell (1982) obtained singletrial estimates of P3 amplitude (after low-pass filtering) and SCR amplitude using a passive auditory oddball paradigm, and found no significant correlation between these measures. However, as we will discuss below, this null result might be attributable to extraneous sources of variance inherent to both signals. Finally, two recent studies have compared habituation of the SCR and P3 to repetitive task-irrelevant stimuli in a typical orienting response paradigm with long interstimulus intervals of 8 s (Rushby, Barry, & Doherty, 2005) and 2 min (Rushby & Barry, 2009). In Rushby et al. (2005), SCR amplitude and P3 amplitude both showed clear habituation, response recovery (to a change stimulus) and enhanced responding (or dishabituation) to a re-presentation of the original stimulus. In Rushby and Barry (2009), SCR amplitude showed habituation over the first few trials of the stimulus train while the P3 showed a nonsignificant decreasing trend across all 12 presented tones. In both studies, principal component analysis was used to investigate habituation of subcomponents of the P3. The 3 extracted phasic subcomponents of the P3 in each study differed widely in terms of their correlation with the SCR across trials. Poor correlations were found between the SCR and a subcomponent corresponding with the P3a (both studies); moderate correlations between SCR and a subcomponent corresponding with the P3b (Rushby et al., 2005); and high correlations between the SCR and a relatively late, frontally distributed subcomponent that the authors labeled ‘‘novelty P3’’ (both studies). The Pupil Dilation Response The stimulus-evoked PDR reflects contributions of the SNS and parasympathetic nervous system, which act in a relatively straightforward, reciprocal manner, with SNS activation closely coupled to parasympathetic inhibition (Beatty & Lucero-Wagoner, 2000). The SNS enlarges the pupil by direct stimulation of the dilator muscles. The contribution of the parasympathetic pathway is mediated by central inhibition of the Edinger–Westphal complex, resulting in relaxation of the sphincter muscles and hence dilation. The PDR has a typical onset latency between 200–500 ms after the stimulus, peaks about 1 s later, and terminates rapidly upon completion of stimulus processing. This phasic, stimulus-evoked activity can be distinguished from the more protracted pupil effects of mental processing load, which will not be considered here (for review, see Beatty, 1982). Though considerably smaller in size, the literature on the antecedent conditions for the PDR is generally consistent with the P3 and SCR literatures summarized above. Like the P3 and SCR, the PDR is highly sensitive to stimulus probability. For example, Qiyuan and colleagues systematically manipulated stimulus probability in an active auditory oddball task and found that the magnitude of the PDR to both targets and non-target stimuli was inversely proportional to their probability (Qiyuan, Richer, Wagoner, & Beatty, 1985). The unexpected absence of a stimulus also evoked a reliable PDR. Friedman, Hakerem,

166 Sutton, and Fleiss (1973) recorded both pupil diameter and the electroencephalogram (EEG) in a passive auditory oddball task in which the relative probability of the two stimulus categories was systematically varied across conditions. Subjects were either told (certain condition) or were asked to guess (uncertain condition) which stimulus would occur on a given trial. Both the PDR and the P3 increased in amplitude as the probability of the eliciting stimulus decreased, but only in the uncertain condition. Steinhauer and Hakerem (1992) report a similar inverse relation between stimulus probability and the amplitudes of the P3 and PDR, both when subjects were counting auditory oddballs, and when they were responding to both stimulus categories (see also Steinhauer & Zubin, 1982). Regarding the effects of task relevance and stimulus value on the PDR, the available evidence is limited. Peavler (1974) measured the pupillary response while subjects were listening to a string of digits. Significant dilation occurred only when subjects were told they would later be tested for recall, and not when no task was associated with the stimuli. However, it should be noted that the reported pattern (monotonic increase in pupil dilation with each presented digit) was not phasic in nature but instead resembled the tonic dilation effects associated with mental processing load (Beatty, 1982). Van Olst et al. (1979) found larger PDRs to targets than to non-target stimuli in an active auditory oddball task, even though the two stimulus categories were equiprobable (but see Qiyuan et al., 1985). Pleasant stimuli, such as erotic pictures, and stressors typically cause large pupil dilation (Bradley, Miccoli, Escrig, & Lang, 2008, Janisse, 1977), suggesting an important influence of stimulus value. The evidence concerning the effect of stimulus intensity on the PDR is also sparse. One study has reported that PDR amplitude increases with the intensity of auditory noise (80–100 dB; Antikainen & Niemi, 1983). In another study, it was found that stimulus intensity (60, 75, or 100 dB) did not affect PDR amplitude. However, it is worth noting that in this study each level of intensity was presented to a different group of subjects, precluding context-sensitive scaling of pupil responses (Stelmack & Siddle, 1982). Finally, several studies have reported reliable habituation of the PDR across multiple presentations of taskirrelevant stimuli (Antikainen & Niemi, 1983; Maher & Furedy, 1979; Stelmack & Siddle, 1982). The relationship between PDR amplitude and performance accuracy mirrors that observed for P3 amplitude and accuracy. That is, under data-limited conditions, larger pupil dilations are generally associated with better performance. Like for the P3, the main evidence comes from studies using the signal-detection paradigm. Hakerem and Sutton (1966) required subjects to detect near-threshold stimuli in a visual signal-detection task. These stimuli elicited a PDR only on trials in which the stimulus was correctly reported as seen. Beatty and Wagoner (1976, unpublished data; cited in Janisse, 1977) found similar results using an auditory signal-detection task: On signal-present trials, larger PDRs were associated with significantly more correct decisions and with higher confidence that the decisions made were correct. Discussion The above review reveals a striking resemblance between the conditions that evoke the P3 and those associated with two exemplary components of the orienting response, the SCR and the PDR. All three measures are preferentially sensitive to novelty, motivational significance (e.g., task relevance), and other salient stimulus characteristics that are potentially important for

S. Nieuwenhuis et al. survival and goal-directed behavior. In addition, all three measures show evidence of habituation with repeated presentation of task-irrelevant stimuli. The occurrence of a P3, SCR, or PDR is also associated with better task performance. As described above, similar observations by other authors, mainly in the 1970s and 1980s, led to the question of how exactly the P3 might be related to autonomic components of the orienting response. In the next section, we discuss a recently proposed theory of the P3 that suggests a straightforward account of this relationship. Before we turn to this theory, two issues are worth noting. First, the data reviewed above suggest that the SCR is more strongly correlated with P3 activity at frontocentral electrodes than with posterior P3 activity. The possible significance of this finding will be considered in the section ‘‘Summary and Discussion.’’ Second, the few studies that have co-registered the P3 and autonomic nervous system measures have reported small or absent correlations between these phenomena. Although such findings may appear at odds with the remarkable similarities in the antecedents for these responses, it should be kept in mind that the between-subject correlations between the P3 and autonomic nervous system measures will be affected by their differential susceptibility to several types of variables (cf. Steinhauer & Hakerem, 1992). Such variables include, for example, exogenous influences like ambient lighting conditions and temperature, recent smoking behavior and time of last meal that may affect autonomic nervous system measures differently than the P3 (e.g., Polich & Kok, 1995). Studies that examined withinsubject, cross-trial correlations between the P3 and autonomic nervous system responses have struggled with the vast differences in the time scales on which these responses can be assessed. Attempts to overcome these methodological challenges are direly needed. Link Between the P3 and the Locus Coeruleus-Norepinephrine (LC-NE) System Recent research has suggested that the neuromodulatory brainstem nucleus locus coeruleus (LC) is critical for the regulation of cognitive performance (Aston-Jones & Cohen, 2005; Nieuwenhuis & Jepma, in press; Robbins, 1997; Sara, 2009; Yu & Dayan, 2005). The LC exhibits a strong phasic increase in activity during the processing of motivationally relevant stimuli, leading to the release of the neuromodulatory neurotransmitter norepinephrine (NE) in widespread cortical projection areas. This LC-mediated noradrenergic innervation increases the responsivity (or gain) of efferent target neurons (for a review, see Berridge & Waterhouse, 2003). It has been shown that, when applied in a temporally strategic manner (e.g., when driven by the identification and evaluation of motivationally relevant stimuli), increases in gain produce an increase in the signal-to-noise ratio of subsequent processing and a concomitant improvement in the efficiency and reliability of behavioral responses (e.g., Servan-Schreiber, Printz, & Cohen, 1990). Accordingly, it has been found that LC phasic activation reliably precedes and is temporally linked to behavioral responses to attended stimuli (Aston-Jones, Rajkowski, Kubiak, & Alexinsky, 1994; Bouret & Sara, 2004; Clayton, Rajkowski, Cohen, & Aston-Jones, 2004). According to a recent theory, the scalp-recorded P3 is the electrophysiological correlate of LC-induced phasic enhancement of neural responsivity (gain) in the neocortex (Nieuwenhuis, Aston-Jones, & Cohen, 2005); this theory follows from earlier theoretical and experimental work (Desmedt & Debecker,

P3 and orienting response 1979; Pineda, Foote, & Neville, 1989). Strong evidence for subcortical involvement in P3 generation has come from a study showing largely intact P3 components to unilaterally presented visual stimuli over the unstimulated hemisphere of a split-brain patient (Kutas, Hillyard, Volpe, & Gazzaniga, 1990). Given that in split-brain patients interhemispheric transfer of information is not possible at the cortical level, this finding indicates that critical input and/or output signals of the P3 process must have passed through one of the intact subcortical commissures. Here, we briefly summarize the evidence for a specific role of LC phasic activity in P3 generation. (For an extensive review and a comparison with the context-updating hypothesis, see Nieuwenhuis, Aston-Jones, & Cohen, 2005.) First, the distribution and timing of intracranial and scalp-recorded P3 activity are consistent with the anatomical and physiological properties of the LC-NE system. For example, the diffuse P3 scalp distribution and observation of P3 activity in multiple intracranial structures (Soltani & Knight, 2000) are consistent with the widespread projections from the LC to cortical and subcortical areas. Furthermore, P3 onset latency in simple two-alternative forced choice tasks is consistent with the latency of LC phasic activity (!150–200 ms), if one takes into account the relatively slow conduction velocity of LC fibers (Aston-Jones, Foote, & Segal, 1985). Additionally, the relatively early timing of P3 activity in frontal (P3a) and subcortical areas (e.g., thalamus; Klostermann, Wahl, Marzinzik, Schneider, Kupsch, & Curio, 2006) is consistent with the trajectory of LC fibers, which first reach these areas and only then veer backwards to innervate posterior cortical areas (Morrison, Molliver, Grzanna, & Coyle, 1981), where the P3b is generated. Because the neuromodulatory effect of NEFpresumed to be reflected in the P3Fis to enhance processing in target areas, brain areas that are most engaged by a given task should show the greatest increases in activity. This may explain why the relative contribution of the P3a and P3b to the overall P3 scalp topography depends on the antecedent conditions. For example, the large P3a to novel stimuli, leading to a more anterior focus of the P3 scalp distribution, may reflect the greater contribution of prefrontal structures to novelty processing (Soltani & Knight, 2000), an effect that is enhanced by LC-NE engagement. Second, the antecedent conditions for the P3, discussed above, are highly similar to those for the LC phasic response (for review, see Aston-Jones, Rajkowski, & Cohen, 2000). The LC phasic response is preferentially elicited by motivationally significant stimuli, including stimuli that are novel or unexpected, conditioned stimuli that require a response (e.g., in an oddball task), unconditioned auditory startle stimuli, appetive and aversive stimuli. The LC phasic response after task-irrelevant auditory stimuli varies directly with the intensity of those stimuli (Grant, Aston-Jones, & Redmond, 1988). Like the P3, the LC phasic response is relatively insensitive to the physical attributes of stimuli, and habituates as the salience of task-irrelevant stimuli decreases with repeated presentations (Sara, Vankov, & Herve, 1994). However, LC responses do not exhibit habituation for highly salient or task-relevant stimulus events (Aston-Jones & Bloom, 1981; Aston-Jones, Rajkowski, et al., 1994). The parallel between the P3 and the LC phasic response is further supported by a study that simultaneously recorded the two phenomena, and found that their changes in amplitude in response to experimental manipulations followed a very similar time course (Aston-Jones, Chiang, & Alexinsky, 1991). Third, several studies have reported direct evidence for an LC generator of the P3. These include psychopharmacological

167 studies, which have shown that P3 amplitude is modulated in a systematic fashion by noradrenergic agents such as clonidine (Swick, Pineda, & Foote, 1994), and entirely abolished following drug-induced norepinephrine depletion (Glover, Ghilardi, Bodis-Wollner, & Onofrj, 1988). Lesion studies have demonstrated a selective decrease in P3 amplitude in monkeys sustaining LC lesions (Pineda et al., 1989). Also, a recent study has found that individual differences in the noradrenergic gene that affects the activity of the alpha-2a receptor are a key determinant of P3 amplitude (Liu, Kiehl, Pearlson, Perrone-Bizzozero, Eichele, & Calhoun, 2009). Finally, the tight link between the P3 and task performance (i.e., speed and accuracy) is consistent with the functional role ascribed to LC phasic activity, namely, to facilitate post-decisional information processing and behavioral responding. Indeed, LC activity itself is closely related to behavioral performance: Larger LC responses are associated with higher performance accuracy, and the latency of LC phasic responses is positively correlated with the overt reaction times (Aston-Jones, Rajkowski, et al., 1994; Aston-Jones et al., 2000; Clayton et al., 2004)Fa pattern similar to that for the P3. We note that the potentiating influence of the LC-NE system on behavioral responding is likely to be modest in typical laboratory tasks, which use simple stimuli and discrete button-press responses. These tasks are performed so quickly that the noradrenergic modulation of the relevant cortical areas (as reflected in the P3) may sometimes occur too late to facilitate the response. It is plausible that the facilitatory influence of the LC-NE system is more prominent in real-life situations, which are characterized by multimodal, crowded sensory environments and a range of potential, often time-consuming response options. In sum, there is converging evidence from multiple research disciplines that indicates a crucial role for the LC-NE system in generating the P3. As we will discuss below, the tight relationship between the LC-NE system, the P3, and the orienting response is further supported by several findings that suggest a strong temporal correlation between LC-NE activity and SNS activity.

Correlation between LC-NE activity and SNS activity The types of stimuli that are most effective for eliciting LC phasic responses are those that stop ongoing behavior and elicit a behavioral orienting response (Aston-Jones & Bloom, 1981; Aston-Jones, Valentino, Van Bockstaele, & Meyerson, 1994). This observation is supported by several studies that have measured correlations between autonomic nervous system changes and LC activity across various time scales. For example, Elam and colleagues found that noxious and non-noxious sensory stimuli produced parallel changes in LC-NE unit activity and peripheral sympathetic nerve discharge in rats (Elam, Svensson, & Thoren, 1986). Abercrombie and Jacobs (1987) found that changes in LC-NE activity induced by chronically presented stressful stimuli were closely correlated with changes in heart rate in cats. Studies examining LC activity after physiological manipulations that cause autonomic activation found that the two were frequently correlated, with higher LC activity for hypoglycemia, hypotension, hypervolemia, ambient heating, and pyrogen-induced fever (Morilak, Fornal, & Jacobs, 1987a, 1987b, 1987c). Furthermore, Reiner (1986) reported parallel changes in activity of LC neurons and peripheral sympathetic tone across the stages of the sleep-wake cycle in behaving cats.

168 Finally, neurophysiological recordings in the monkey have indicated that tonic changes in pupil diameter closely track the time course of LC activity, and show the same relationship to behavioral performance as tonic LC activity (Aston-Jones & Cohen, 2005; Rajkowski, Kubiak, & Aston-Jones, 1993). Some of these findings have recently been corroborated in a series of human pupillometry experiments (Gilzenrat, Nieuwenhuis, Jepma, & Cohen, 2010; Jepma & Nieuwenhuis, submitted). In these experiments, it was found that the magnitudes of baseline pupil diameter and task-evoked pupil dilations were inversely correlated, corresponding to the reciprocal relationship observed between LC tonic and phasic modes (Aston-Jones et al., 2000; Usher, Cohen, Servan-Schreiber, Rajkowski, & Aston-Jones, 1999). Furthermore, these measures of pupil diameter were sensitive to experimental manipulations of task utility, and predictive of behavioral indices of task (dis)engagement and exploratory behaviors in a manner consistent with predictions of the adaptive gain theory of LC function (Aston-Jones & Cohen, 2005). The observed similarity in antecedent conditions and correlations between autonomic nervous system and LC-NE activity suggest that the LC-NE system in the brain is a central analogue of the peripheral SNS (cf. Amaral & Sinnamon, 1977; AstonJones et al., 1991), and that the two systems often operate in an integrated fashion. They also lend further credence to the notion that autonomic components of the orienting response and the P3 are intimately coupled. The critical question that motivates much of the remainder of this article is how the parallel activation of the SNS and the P3 (as a correlate of phasic LC-NE activity) can be understood in anatomical terms, given existing knowledge about the anatomy of the LC-NE system. Anatomical Link: Parallel Activation of the LC-NE System and Peripheral SNS by the Rostral Ventrolateral Medulla It is unlikely that the parallel activation of the LC-NE system and peripheral SNS reflects a direct influence of one on the other. Contrary to occasional claims in the literature (e.g., Szabadi & Bradshaw, 1996), there is no reliable evidence for a direct projection from the LC to the autonomic nuclei that regulate the pupil, sweat glands, heart, and other organs (Aston-Jones, 2004). Instead, some of these nuclei are innervated by lower medullary NE cell groups (e.g., Levitt & Moore, 1979). Although there is substantial evidence that autonomic (mainly cardiovascular) responses have an influence on LC activity (Berntson, Sarter, & Cacioppo, 1998; Morilak et al., 1987a, 1987b, 1987c; Svensson, 1987), this anatomical route is too slow to explain the rapid, phasic LC responses to motivationally significant stimuli and the resulting P3. As an alternative explanation for the parallel activation of the LC and the SNS, we propose an anatomical model that introduces a third player: the nucleus paragigantocellularis (PGi), a highly integrative nucleus in the rostral ventrolateral medulla that plays a pivotal role in controlling both the LC and the SNS (see Figure 1). The PGi provides one of the major excitatory inputs to the LC (Aston-Jones, Ennis, Pieribone, Nickell, & Shipley, 1986). Furthermore, pharmacologic blockade of the PGi blocks LC responses to somatosensory stimulation (Chiang & Aston-Jones, 1993; Ennis & Aston-Jones, 1988), as does blockade of glutamate transmission in the LC, the major neurotransmitter in the PGi-to-LC pathway (Ennis & Aston-Jones, 1988). These and other studies have indicated that the PGi is a critical

S. Nieuwenhuis et al.

Figure 1. Anatomical model that corresponds to our hypothesis. PGi 5 nucleus paragigantocellularis; LC 5 locus coeruleus; SNS 5 sympathetic nervous system; mPFC 5 medial prefrontal cortex; Orb 5 orbitofrontal cortex; ACC 5 anterior cingulate cortex; Amyg 5 amygdala; PBN 5 parabrachial nucleus; Hypoth 5 hypothalamus; PAG 5 periaqueductal grey; NTS 5 nucleus tractus solitarius; IML 5 intermediolateral cell column of the spinal cord.

relay center for mediating the phasic LC responses evoked by at least certain sensory stimuli. Importantly, the PGi is also a key sympathoexcitatory brain region, with strong projections that directly innervate preganglionic sympathetic neurons of the intermediolateral cell column of the spinal cord (Guyenet, 1990; Loewy, Wallach, & McKellar, 1981). These preganglionic cells send axons to specific ganglia in the peripheral nervous system and synapse on postganglionic neurons, which in turn innervate various peripheral targets such as the dilator muscle of the pupil and the sweat glands. Stimulation of the PGi elicits electrodermal responses, pupil dilations, increases in blood pressure, and other sympathetic responses (Davison & Koss, 1975; Guyenet, 1990; Koss & Wang, 1972). Interestingly, stimulation of PGi neurons can also increase cortical arousal, as indicated by changes in the EEG power spectrum (Wu, Stavarache, Pfaff, & Kow, 2007). The location of the PGi in the medullary reticular formation is consistent with early proposals regarding the origin of the orienting response in the reticular formation (Sokolov, 1963, 1975), and with classic experiments showing that stimulation of the reticular formation by implanted electrodes reproduces the autonomic, behavioral, and EEG components of the orienting response (Moruzzi & Magoun, 1949; Scheibel, 1980). It is noteworthy, however, that evidence indicates that neurons projecting to sympathetic preganglionic areas and to the LC are often distinct but interdigitated cells in the ventrolateral medulla (Huangfu, Verberne, & Guyenet, 1992). Thus, the frequently observed parallel regulation of the LC and the SNS seems to involve similar inputs to parallel but distinct neurons innervating spinal sympathetic areas and LC.

P3 and orienting response Anatomical studies have revealed that the PGi itself receives inputs from a wide variety of brain areas involved in autonomic and visceral regulation, including the periaqueductal grey, the hypothalamus, and the insular cortex, and from multimodal association areas including the medial prefrontal cortex (Van Bockstaele, Aston-Jones, Ennis, Shipley, & Pieribone, 1991; Van Bockstaele, Pieribone, & Aston-Jones, 1989). The PGi is a critical relay center for the descending sympathoexcitatory pathways originating in the hypothalamus (Hilton & Smith, 1984). Furthermore, emotional signals from the amygdala may reach the PGi by way of the hypothalamus and periaqueductal grey. Thus, the PGi integrates several types of autonomic and sensory information, and provides potent parallel activation of the LC and the peripheral SNS (Aston-Jones, Valentino, et al., 1994). These properties are consistent with the simultaneous occurrence of the P3 and orienting response in response to a wide array of motivationally significant events. Alternative Anatomical Routes Our model posits that the LC provides no direct efferent innervation of the nuclei subserving sympathetic tone, because the LC does not project substantially to preganglionic autonomic nuclei. However, experimental manipulation of the LC-NE system can result in changes in SNS activity. An early study found that electrical stimulation of the cat LC elicited an increase in heart rate and blood pressure (Gurtu, Pant, Sinha, & Bhargava, 1984). However, these effects must have been the result of activating other structures, either structures nearby the LC such as the periaqueductal grey, or antidromically activated afferents such as the PGi, because chemical stimulation of the LC decreases both blood pressure and heart rate (Murase, Takayama, & Nosaka, 1993; Sved & Felsten, 1987). These responses were eliminated by chemical destruction of noradrenergic LC neurons using local injection of 6-hydroxydopamine, a selective neurotoxin of catecholamine neurons. Lesions induced by 6-hydroxydopamine in the rat and cat dorsal noradrenergic bundle also resulted in the complete abolition of auditory-evoked SCRs (Yamamoto, Arai, & Nakayama, 1990; Yamamoto, Hoshino, Takahashi, Kaneko, & Ozawa, 1991). In rats, the alpha-2 adrenoceptor agonist clonidine (which inhibits LC activity and decreases NE release) decreased the SCR amplitude, whereas the alpha-2 adrenoceptor antagonist yohimbine (which increases LC activity and NE release) substantially increased the amplitude of the SCR (Yamamoto, Ozawa, Shinba, & Hoshino, 1994). Similarly, Saiers and Campbell (1990) reported that a decrease of noradrenergic activity as a result of clonidine injections in rats disrupted the heart rate component of the orienting response to an auditory stimulus. In contrast, pharmacological modulations of the dopaminergic, cholinergic, and serotonergic systems did not affect heart rate responses. Administration of clonidine and yohimbine have also been found to change baseline pupil diameter (Koss, 1986; Phillips, Szabadi, & Bradshaw, 2000). However, it is hard to determine whether the effects of such noradrenergic agents on autonomic activity are mediated by adrenoceptors located on LC neurons or by other adrenoceptors, for example, located on the lower medullary NE cell groups, which directly innervate autonomic nuclei, or even on autonomic nuclei themselves. Furthermore, some of the reported effects may be a consequence of a reduction in parasympathetic tone, instead of an increase in sympathetic tone (Koss, 1986).

169 As no direct anatomical connections have been documented between the LC and autonomic nuclei, there is no straightforward way to explain the effects of LC lesions/manipulations on SNS activity other than that they are produced via indirect pathways. Indeed, there are a number of possible indirect pathways by which LC manipulation could affect the SNS. In particular, the LC has dense ascending projections to various important higher brain centers involved in SNS control, which in turn project directly or indirectly to autonomic nuclei that regulate peripheral SNS responses (cf. Berntson et al., 1998). Anatomical and physiological studies suggest that these control centers include the anterior cingulate, the insula, amygdala, and hippocampus (Verberne & Owens, 1998). Electrical or chemical stimulation of these LC projection areas elicits a wide range of peripheral sympathetic responses, and lesions damaging these areas tend to reduce or abolish these peripheral responses (Critchley, 2002; Jordan, 1990; Knight, 1996; Verberne & Owens, 1998). Converging evidence from fMRI studies confirms that activity in the anterior cingulate and insula is modulated by stimulus frequency and novelty, consistent with a role for these regions in orienting to motivationally significant stimuli (Ranganath & Rainer, 2003). Finally, the hypothalamus is another major component of the descending pathways that regulate sympathetic and vagal neurons. However, LC projections to the hypothalamus are quite limited (Aston-Jones, 2004), indicating that other areas are the critical links in the circuit connecting the LC with the SNS.

Functional Significance of the P3 and the Orienting Response: Mobilization for Action Theoretical accounts of the function of the orienting response generally distinguish between two components: enhancing the perception of the eliciting event and facilitating action in response to the stimulus. The latter component refers to the energizing quality of the orienting responseFthe mobilization of somatic and autonomic systems for dealing with the immediate consequences of the triggering stimulus. As reviewed by Lynn (1966), somatic responses include the inhibition of ongoing activity, increases in general muscle tone that prepare the muscles for action, and changes in the skeletal muscles that direct the sense organs towards the source of the stimulus. This directional motor activity (e.g., ocular motion, pricking of the ears in animals) likely reflects the interaction of the spatially nonspecific orienting response with brain systems specialized in directing spatial attention. The multifaceted autonomic response, including cardiovascular, respiratory, hormonal, and pupillary changes, likewise seems to prepare the body for efficient action and increased energy expenditure (Lynn, 1966). Other theorists have emphasized the importance of the orienting response for enhancing sensory processing of the eliciting stimulus (Graham, 1979; Pavlov, 1927; Sokolov, 1963). The orienting response is associated with increased sensory receptor sensitivity, lowering perceptual thresholds (Lynn, 1966). However, aside from this, there is little empirical support for the notion that the orienting response enhances information uptake. For example, there is little or no evidence for the argument that a large pupil enhances perceptual sensitivity (cf. Lynn, 1966). Of course, information uptake will be enhanced by the orienting of body and sense organs, but these are motor changes, not changes in the perceptual system or SNS. Some psychophysiologists have

170 also questioned the use of the orienting response for enhancing perceptual processing, given that the elicitation of the orienting response is contingent on a perceptual analysis of the stimulus for determining its motivational significance (e.g., Siddle & Spinks, 1979). This argument forces theorists to be explicit about their model of information processing and the corresponding aspects of perception that are enhanced by the orienting response. These limitations of the sensory-intake hypothesis, along with the apparent consensus that the distinction between the benefits for perception vs. action afforded by the orienting response is largely irrelevant from a selection-for-action perspective (Allport, 1987), have led to the view that the primary role of the orienting response is in the mobilization for action (e.g, Donchin et al., 1984). As discussed earlier, the theoretical integration of the orienting response and P3 literatures was challenged in the 1980s by the discrepancy between the action-oriented view of the orienting response and the reigning theory of the P3: the contextupdating hypothesis (Donchin, 1981; Donchin & Coles, 1988). Donchin’s hypothesis was strongly inspired by Sokolov’s ideas about the antecedent conditions for the orienting response: the P3 was assumed to be elicited when there is a mismatch between a subject’s representation (‘‘neuronal model’’) of the environment (in its broadest sense) and actual experience. However, the context-updating hypothesis attributed a different function to the process manifested by the P3 process, namely, the updating of the representation of the environment (context) to optimize decision making in response to future stimuli. Thus, according to this hypothesis, the P3 reflects a strategic or memory function rather than the facilitation of responses. Unlike the context-updating hypothesis, the recent theory that the P3 reflects phasic activity of the LC-NE system (Nieuwenhuis, Aston-Jones, & Cohen, 2005) suggests a shared functional interpretation of the P3 and the orienting response. For its assumptions about the functional significance of the P3, the theory draws on our current understanding of the function of LC phasic responses, which indicates that these LC responses facilitate behavioral responses to the outcome of task-specific decision processes (Aston-Jones & Cohen, 2005; Bouret & Sara, 2004). In addition, the broad projections of LC-NE neurons indicate that responses in these cells may also augment other processes important for decision execution besides motor activity, including sensory perception and memory (Hurley, Devilbiss, & Waterhouse, 2004; Sara et al., 1994). As reviewed above, the ensuing action-oriented view of the P3 is supported by the tight link between the latency and amplitude of the P3 and corresponding task performance. Taken together, this analysis suggests that the P3 and SNS components of the orienting response can be seen as manifestations of a global sympathetic system specialized in mobilization for action. The central nervous system limb of this system, the LC-NE system (manifested in the P3), facilitates the execution of cognitive decisions concerning proper behaviors in the face of urgent stimulus demand while, at the same time, the autonomic nervous system limb (manifested in the orienting response) facilitates physical execution of the chosen behaviors. This theoretical integration of the orienting response and P3 suggests a highly efficient system for urgent responding: Stimuli are analyzed by cortical (and perhaps subcortical) areas capable of performing the precise computations that determine whether a stimulus is task-relevant or otherwise motivationally significant and what response should be elicited. The output of this analysis

S. Nieuwenhuis et al. is passed down to lower brain areas, including the rostral ventrolateral medulla, which in turn projects to the LC and autonomic nuclei. In case a motivationally significant stimulus is detected and a response decision is reached, the LC is activated and produces a system-wide transient innervation of the brain (P3) that facilitates further processing of the eliciting stimulus and other stimuli related to the decision reached, speeding up the deployment of attention and the execution of a behavioral response. Simultaneously, the SNS is activated (orienting response) to facilitate motor action in response to the stimulus.

Summary and Discussion In this article, we have discussed the similarities between two psychophysiological phenomena: the orienting response, a collection of physiological responses in order to effectively cope with motivationally significant events; and the ubiquitous P3, the single most-studied component of the event-related potential. The orienting response and P3 generally co-occur; as we have reviewed, both are elicited by stimuli with learned or inherent motivational significance. This has raised the question whether the P3 should be seen as the central nervous system counterpart to the SNS components of the orienting response (Donchin et al., 1984; Friedman, 1978; Kimmel et al., 1979). The analysis of anatomy and function presented here suggests that the answer to this question is a cautious ‘yes.’ With regard to anatomy, we have discussed that there is no direct connection between SNS nuclei and the LC, the most probable initial generator of the P3 (Nieuwenhuis, Aston-Jones, & Cohen, 2005). Furthermore, the sensory feedback of actual autonomic responses to the LC is too slow to explain the rapid, phasic LC responses to motivationally significant stimuli. Instead, it is likely that the tight link between P3 and orienting response reflects common afferent projections to the LC and sympathetic preganglionic neurons: the major source of input to the LC is a key sympathoexcitatory region of the rostral ventrolateral medullaFin particular, the PGi. This highly integrative medullary area could be responsible for the observed parallel activation of the LC-NE system and peripheral SNS in response to various types of motivationally significant stimuli. The LC and its ascending projections thus carry efferent copies of the medulla’s command signals to the peripheral SNS. This suggests that the LC-NE system may implement one of Damasio’s (1999) ‘as-if’ loopsFthe notion that somatic markers can reflect not only states of the body (e.g., in the peripheral SNS) but also representations (i.e., copies) of body states in, for example, brainstem neuromodulatory systems. In other words, feedback from the body is short-circuited by direct signals from brainstem areas to regions representing body state. With regard to function, a comparison of the functional significance of the LC-NE system and SNS suggests that the P3 and orienting response reflect complementary contributions to the mobilization for action following motivationally significant stimuli. Phasic LC responses (giving rise to the P3) may optimize information processing following decisions regarding appropriate behavioral responses at the same time that the peripheral sympathetic system prepares the subject physically to execute these responses. The wide range of antecedent conditions for the P3 and orienting response is consistent with our anatomical model, given the integrative properties of the rostral ventrolateral medulla. There has been a lot of debate about the common

P3 and orienting response

171

denominator of these antecedent conditions (e.g., Bernstein, 1979; Maltzman, 1979b; O’Gorman, 1979). We doubt whether this is a useful debate; the brain has evolved to favor the processing of salient, significant, unexpected, and novel stimuli (Corbetta, Patel, & Shulman, 2008; Desimone & Duncan, 1995; Ranganath & Rainer, 2003), and the antecedent conditions for the P3 and orienting response merely reflect this preference. A challenge for our anatomical model, suggested by our review of the similarities between the orienting response and P3, is that the SCR seems more strongly correlated with P3 activity at frontocentral electrodes (P3a) than with posterior P3 activity (P3b). A speculative explanation for this finding is that posterior cortical areas, which are not directly connected to the LC, synapse on frontal neurons that are directly connected to the LC (Arnsten & Goldman-Rakic, 1984; Aston-Jones & Cohen, 2005; Lee, Kim, & Waterhouse, 2005), whereas other frontal neurons connect to the LC via the rostral ventrolateral medulla (Van Bockstaele et al., 1989). As we have discussed, there are also a number of possible indirect pathways by which the LC could affect the SNS. At the level of the cortex, these pathways include frontal areas (e.g., anterior cingulate cortex) and not posterior areas, which may also partly account for the pattern of correlation between P3a, P3b, and orienting response. Future research should address these possibilities. Our theory of the P3 also suggests another, more general, explanation for dissociations between the orienting response and subcomponents of the P3. The theory claims that the P3 reflects the neuromodulatory effect of NE in cortical target areas. Therefore, cortical areas that are most engaged by a given stimulus or task should show the greatest increases in activity, and cortical areas that are not involved should show little or no noradrenergic modulation. The implication of this conjecture is that a putative correlation between the orienting response and a P3 subcomponent may be confounded by variance in the involvement of the cortical areas that directly generate the P3 subcomponent. However, it is implausible that these explanations can account for all of the observed dissociations between the orienting response and P3 subcomponents. Indeed, although our anatomical model offers an attractive account of the similarities between these phenomena, it is unlikely to provide a complete account of

their relationship. For example, both the locus coeruleus and sympathetic preganglionic neurons receive projections from a wide range of areas other than the rostral ventrolateral medulla, including forebrain, hypothalamic and other brainstem areas (Aston-Jones, 2004; Berridge & Waterhouse, 2003; Sved, Cano, & Card, 2001). To the extent that these projections are not shared, they will result in dissociations between the orienting response and P3. Our analysis has several other limitations. First, although our hypothesis about the relationship between the orienting response and P3 is based on a multitude of data from psychophysiological and neurophysiological literatures that have not previously been connected in this context, the exact value of the hypothesis awaits new empirical tests. For example, future animal studies could simultaneously record cell activity in the PGi, the scalp P3, and components of the orienting response, and test the prediction that the corresponding measures should be highly correlated. Second, our analysis is too focused to do justice to some of the complexities of the orienting response literature, for example, the subtle but crucial difference between the orienting response and the defense reflex (Graham, 1979). A third limitation is that our analysis is fully focused on the role of the LC-NE system in generating the P3, even though other neurochemical systems almost certainly influence the P3 as well (Polich & Criado, 2006). A better understanding of the complex interactions between neuromodulatory systems, such as the cholinergic, dopaminergic, and noradrenergic systems, would almost certainly further enhance our understanding of the current topic (Briand, Gritton, Howe, Young, & Sarter, 2007). These limitations notwithstanding, we believe that our analysis is a valuable step towards establishing the precise relationship between the P3 and the orienting response. More broadly, the present research illustrates the value of integrating the psychophysiological and neurophysiological literature: Knowledge of the LC-NE system played a crucial role in developing a novel hypothesis regarding the relationship between the P3 and peripheral manifestations of the orienting responseFthat they reflect the co-activation of the LC-NE system and the peripheral SNS by a common medullary pathway that has evolved to afford rapid action in response to motivationally significant stimuli.

REFERENCES Abercrombie, E. D., & Jacobs, B. L. (1987). Single-unit response of noradrenergic neurons in the locus coeruleus of freely moving cats. II. Adaptation to chronically presented stressful stimuli. Journal of Neuroscience, 7, 2844–2848. Allport, D. A. (1987). Selection for action: Some behavioral and neurophysiological considerations of attention and action. In H. Heuer & A. F. Sanders (Eds), Perspectives on perception and action (pp. 395– 419). Hillsdale, NJ: Lawrence Erlbaum Associates Inc. Amaral, D. G., & Sinnamon, H. M. (1977). The locus coeruleus: Neurobiology of a central noradrenergic nucleus. Progress in Neurobiology, 9, 147–196. Antikainen, J., & Niemi, P. (1983). Neuroticism and the pupillary response to a brief exposure to noise. Biological Psychology, 17, 131– 135. Arnsten, A. F., & Goldman-Rakic, P. S. (1984). Selective prefrontal cortical projections to the region of the locus coeruleus and raphe nuclei in the rhesus monkey. Brain Research, 306, 9–18. Aston-Jones, G. (2004). Locus coeruleus, A5 and A7 noradrenergic cell groups. In G. Paxinos (Ed.), The rat nervous system (3rd edition, pp. 259–294). San Diego: Elsevier Academic Press. Aston-Jones, G., & Bloom, F. E. (1981). Norepinephrine-containing locus coeruleus neurons in behaving rats exhibit pronounced

responses to non-noxious environmental stimuli. Journal of Neuroscience, 1, 887–900. Aston-Jones, G., Chiang, C., & Alexinsky, T. (1991). Discharge of noradrenergic locus coeruleus neurons in behaving rats and monkeys suggests a role in vigilance. Progress in Brain Research, 88, 501–520. Aston-Jones, G., & Cohen, J. D. (2005). An integrative theory of locus coeruleus-norepinephrine function: Adaptive gain and optimal performance. Annual Review of Neuroscience, 28, 403–450. Aston-Jones, G., Ennis, M., Pieribone, V. A., Nickell, W. T., & Shipley, M. T. (1986). The brain nucleus locus coeruleus: Restricted afferent control of a broad efferent network. Science, 234, 734–737. Aston-Jones, G., Foote, S. L., & Segal, M. (1985). Impulse conduction properties of noradrenergic locus coeruleus axons projecting to monkey cerebrocortex. Neuroscience, 15, 765–777. Aston-Jones, G., Rajkowski, J., & Cohen, J. D. (2000). Locus coeruleus and regulation of behavioral flexibility and attention. Progress in Brain Research, 126, 165–182. Aston-Jones, G., Rajkowski, J., Kubiak, P., & Alexinsky, T. (1994). Locus coeruleus neurons in the monkey are selectively activated by attended stimuli in a vigilance task. Journal of Neuroscience, 14, 4467–4480. Aston-Jones, G., Valentino, R. J., Van Bockstaele, E. J., & Meyerson, A. T. (1994). Locus coeruleus, stress, and PTSD: Neurobiological and

172 clinical parallels. In M. M. Murburg (Ed.), Cathecholamine function in post-traumatic stress disorder: Emerging concepts (1st edition, pp. 17– 62). Washington, DC: American Psychiatric Press, Inc. Bahramali, H., Gordon, E., Lim, C. L., Li, W., Lagopoulos, J., Leslie, J., et al. (1997). Evoked related potentials associated with and without an orienting reflex. Neuroreport, 18, 2665–2669. Barry, R. J. (1975). Low-intensity auditory stimulation and the GSR orienting response. Physiological Psychology, 3, 98–100. Barry, R. J. (1979). A factor-analytic examination of the unitary OR concept. Biological Psychology, 8, 161–178. Barry, R. J. (2009). Habituation of the orienting reflex and the development of Preliminary Process Theory. Neurobiology of Learning and Memory, 92, 235–242. Barry, R. J., Feldman, S., Gordon, E., Cocker, K. I., & Rennie, C. (1993). Elicitation and habituation of the electrodermal orienting response in a short interstimulus interval paradigm. International Journal of Psychophysiology, 15, 247–253. Beatty, J. (1982). Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological Bulletin, 91, 276–292. Beatty, J., & Lucero-Wagoner, B. (2000). The pupillary system. In J. Caccioppo, L. G. Tassinary, & G. Berntson (Eds.), The handbook of psychophysiology. Hillsdale, NJ: Cambridge University Press. Bernstein, A. S. (1979). The orienting response as novelty and significance detector: Reply to O’Gorman. Psychophysiology, 16, 263–273. Bernstein, A. S., & Taylor, K. W. (1979). The interaction of stimulus information with potential stimulus significance in eliciting the skin conductance orienting response. In H. Kimmel, E. Van Olst, & J. Orlebeke (Eds.), The orienting reflex in humans (pp. 499–519). Hillsdale, NJ: Lawrence Erlbaum. Bernstein, A. S., Taylor, K. W., & Weinstein, E. (1975). The phasic electrodermal response as a differentiated complex reflecting stimulus significance. Psychophysiology, 12, 158–169. Berntson, G. G., Cacioppo, J. T., & Quigley, K. S. (1993). Respiratory sinus arrhythmia: Autonomic origins, physiological mechanisms, and psychophysiological implications. Psychophysiology, 30, 183–196. Berntson, G. G., Sarter, M., & Cacioppo, J. T. (1998). Anxiety and cardiovascular reactivity: The basal forebrain cholinergic link. Behavioral Brain Research, 94, 225–248. Berntson, G. G., Sarter, M., & Cacioppo, J. T. (2003). Ascending visceral regulation of cortical affective information processing. European Journal of Neuroscience, 18, 2103–2109. Berridge, C. W., & Waterhouse, B. D. (2003). The locus coeruleus–noradrenergic system: Modulation of behavioral state and statedependent cognitive processes. Brain Research Reviews, 42, 33–84. Bouret, S., & Sara, S. J. (2004). Reward expectation, orientation of attention and locus coeruleus–medial frontal cortex interplay during learning. European Journal of Neuroscience, 20, 791–802. Bradley, M. M., Miccoli, L., Escrig, M. A., & Lang, P. J. (2008). The pupil as a measure of emotional arousal and autonomic activation. Psychophysiology, 45, 602–607. Briand, L. A., Gritton, H., Howe, W. M., Young, D. A., & Sarter, M. (2007). Modulators in concert for cognition: Modulator interactions in the prefrontal cortex. Progress in Neurobiology, 83, 69–91. Chiang, C., & Aston-Jones, G. (1993). Response of locus coeruleus neurons to footshock stimulation is mediated by neurons in the rostral ventral medulla. Neuroscience, 53, 705–715. Clayton, E. C., Rajkowski, J., Cohen, J. D., & Aston-Jones, G. (2004). Phasic activation of monkey locus ceruleus neurons by simple decisions in a forced-choice task. Journal of Neuroscience, 24, 9914–9920. Corbetta, M., Patel, G., & Shulman, G. L. (2008). The reorienting system of the human brain: From environment to theory of mind. Neuron, 58, 306–324. Courchesne, E., Hillyard, S. A., & Galambos, R. (1975). Stimulus novelty, task relevance and the visual evoked potential in man. Electroencephalography and Clinical Neurophysiology, 39, 131–143. Covington, J. W., & Polich, J. (1996). P300, stimulus intensity, and modality. Electroencephalography and Clinical Neurophysiology, 100, 579–584. Critchley, H. D. (2002). Electrodermal responses: What happens in the brain. Neuroscientist, 8, 132–142. Critchley, H. D. (2005). Neural mechanisms of autonomic, affective, and cognitive integration. Journal of Comparative Neurology, 493, 154– 166. Croft, R. J., Gonsalvez, C. J., Gabriel, C., & Barry, R. J. (2003). Targetto-target interval versus probability effects on P300 in one- and twotone tasks. Psychophysiology, 40, 322–328.

S. Nieuwenhuis et al. Damasio, A. R. (1999). The feeling of what happens: Body and emotion in the making of consciousness. New York: Harcourt Brace. Davison, M. A., & Koss, M. C. (1975). Brainstem loci for activation of electrodermal response in the cat. American Journal of Physiology, 229, 930–934. Dawson, M. E., Schell, A. M., & Filion, D. L. (2000). The electrodermal system. In J. T. Cacioppo & L. G. Tassinary (Eds.), Handbook of psychophysiology (2nd edition, pp. 200–223). Cambridge, UK: Cambridge University Press. Desimone, R., & Duncan, J. (1995). Neural mechanisms of selective visual attention. Annual Review of Neuroscience, 18, 193–222. Desmedt, J. E., & Debecker, J. (1979). Wave form and neural mechanism of the decision P350 elicited without pre-stimulus CNV or readiness potential in random sequences of near-threshold auditory clicks and finger stimuli. Electroencephalography and Clinical Neurophysiology, 47, 648–70. Donchin, E. (1981). Surprise! . . . Surprise? Psychophysiology, 18, 493– 513. Donchin, E., & Coles, M. G. H. (1988). Is the P300 component a manifestation of context updating? Behavioral and Brain Sciences, 11, 357–374. Donchin, E., Heffley, E., Hillyard, S. A., Loveless, N., Maltzman, I., Ohman, A., et al. (1984). Cognition and event-related potentials. II. The orienting reflex and P300. Annals of the New York Academy of Sciences, 425, 39–57. Duncan-Johnson, C. C., & Donchin, E. (1977). On quantifying surprise: The variation of event-related potentials with subjective probability. Psychophysiology, 14, 456–467. Edelberg, R. (1972). Electrodermal recovery rate, goal-orientation, and aversion. Psychophysiology, 9, 512–520. Elam, M., Svensson, T. H., & Thoren, P. (1986). Locus coeruleus neurons and sympathetic nerves: Activation by cutaneous sensory afferents. Brain Research, 366, 254–261. Ennis, M., & Aston-Jones, G. (1988). Activation of locus coeruleus from nucleus paragigantocellularis: A new excitatory amino acid pathway in brain. Journal of Neuroscience, 8, 3644–3657. Ford, J. M., Roth, W. T., & Kopell, B. S. (1976). Auditory evoked potentials to unpredictable shifts in pitch. Psychophysiology, 13, 32–39. Friedman, D. (1978). The late positive component and orienting behavior. In D. Otto (Ed.), Multidisciplinary perspectives in event-related brain potential research (pp. 178–180). Washington, DC: U.S. EPA. Friedman, D., Cycowicz, Y. M., & Gaeta, H. (2001). The novelty P3: An event-related brain potential (ERP) sign of the brain’s evaluation of novelty. Neuroscience & Biobehavioral Reviews, 5, 355–373. Friedman, D., Hakerem, G., Sutton, S., & Fleiss, J. L. (1973). Effect of stimulus uncertainty on the pupillary dilation response and the vertex evoked potential. Electroencephalography and Clinical Neurophysiology, 34, 475–484. Gilzenrat, M. S., Nieuwenhuis, S., Jepma, M., & Cohen, J. D. (2010). Pupil diameter tracks changes in control state predicted by the adaptive gain theory of locus coeruleus function. Cognitive, Affective, & Behavioral Neuroscience, 10, 252–269. Glover, A., Ghilardi, M. F., Bodis-Wollner, I., & Onofrj, M. (1988). Alterations in event-related potentials (ERPs) of MPTP-treated monkeys. Electroencephalography and Clinical Neurophysiology, 71, 461– 468. Graham, F. K. (1979). Distinguishing among orienting, defense and startle reflexes. In H. D. Kimmel, E. H. Van Olst, & J. F. Orlebeke (Eds.), The orienting reflex in humans (pp. 137–168). New York: Lawrence Erlbaum Associates. Graham, F. K., & Clifton, R. K. (1966). Heart-rate change as a component of the orienting response. Psychological Bulletin, 65, 305–320. Grant, S., Aston-Jones, G., & Redmond, D. E. (1988). Responses of primate locus coeruleus neurons to simple and complex stimuli. Brain Research Bulletin, 21, 401–410. Gurtu, S., Pant, K. K., Sinha, J. N., & Bhargava, K. P. (1984). An investigation into the mechanism of cardiovascular responses elicited by electrical stimulation of locus coeruleus and subcoeruleus in the cat. Brain Research, 301, 59–64. Guyenet, P. G. (1990). Role of the ventral medulla oblongata in blood pressure regulation. In A. Loewy & K. M. Spyer (Eds.), Central regulation of autonomic functions (pp. 145–167). New York: Oxford University Press. Hakerem, G., & Sutton, S. (1966). Pupillary response at visual threshold. Nature, 212, 485–486.

P3 and orienting response Halgren, E., & Marinkovic, K. (1995). Neurophysiological networks integrating human emotions. In M. Gazzaniga (Ed.), The cognitive neurosciences (pp. 1137–1151). Cambridge, MA: MIT Press. Hillyard, S. A., Squires, K. C., Bauer, J. W., & Lindsay, P. H. (1971). Evoked potential correlates of auditory signal detection. Science, 172, 1357–1360. Hilton, S. M., & Smith, P. R. (1984). Ventral medullary neurones excited from the hypothalamic and mid-brain defence areas. Journal of the Autonomic Nervous System, 11, 35–42. Holm, A., Ranta-aho, P. O., Sallinen, M., Karjalainen, P. A., & Mu¨ller, K. (2006). Relationship of P300 single-trial responses with reaction time and preceding stimulus sequence. International Journal of Psychophysiology, 61, 244–252. Huangfu, D., Verberne, A. J., & Guyenet, P. G. (1992). Rostral ventrolateral medullary neurons projecting to locus coeruleus have cardiorespiratory inputs. Brain Research, 598, 67–75. Hurley, L. M., Devilbiss, D. M., & Waterhouse, B. D. (2004). A matter of focus: Monoaminergic modulation of stimulus coding in mammalian sensory networks. Current Opinion in Neurobiology, 14, 488–495. Jackson, J. C. (1974). Amplitude and habituation of the orienting reflex as a function of stimulus intensity. Psychophysiology, 11, 647–659. Janisse, M.-P. (1977). Pupillometry: The psychology of the pupillary response. Washington, DC: Hemisphere Publishing Co. Jepma, M., & Nieuwenhuis, S. (submitted). Pupil diameter predicts changes in the exploration-exploitation trade-off: Evidence for the adaptive gain theory of locus coeruleus function. Johnson, R., Jr. (1993). On the neural generators of the P300 component of the event-related potential. Psychophysiology, 30, 90–97. Johnston, V. S., Miller, D. R., & Burleson, M. H. (1986). Multiple P3s to emotional stimuli and their theoretical significance. Psychophysiology, 23, 684–694. Jordan, D. (1990). Autonomic changes in affective behavior. In A. D. Loewy & K. M. Spyer (Eds.), Central regulation of autonomic functions (pp. 349–366). New York: Oxford University Press. Kahneman, D. (1973). Attention and effort. Englewood Cliffs, NJ: Prentice-Hall. Karis, D., Fabiani, M., & Donchin, E. (1984). ‘‘P300’’ and memory: Individual differences in the von Restorff effect. Cognitive Psychology, 16, 177–216. Kimmel, H., Van Olst, E. H., & Orlebeke, J. F. (1979). The orienting reflex in humans. Hillsdale, NJ: Erlbaum. Kleinsmith, L. J., & Kaplan, S. (1964). Interaction of arousal and recall interval in nonsense syllable paired-associate learning. Journal of Experimental Psychology, 67, 124–126. Klostermann, F., Wahl, M., Marzinzik, F., Schneider, G. H., Kupsch, A., & Curio, G. (2006). Mental chronometry of target detection: Human thalamus leads cortex. Brain, 129, 923–931. Knight, R. (1996). Contribution of human hippocampal region to novelty detection. Nature, 383, 256–259. Koss, M. C. (1986). Pupillary dilation as an index of central nervous system alpha 2-adrenoceptor activation. Journal of Pharmacological Methods, 15, 1–19. Koss, M. C., & Wang, S. C. (1972). Brainstem loci for sympathetic activation of the nictitating membrane and pupil in the cat. American Journal of Physiology, 222, 900–905. Kutas, M., Hillyard, S. A., Volpe, B. T., & Gazzaniga, M. S. (1990). Late positive event-related potentials after commissural section in humans. Journal of Cognitive Neuroscience, 2, 258–271. Lee, H. S., Kim, M. A., & Waterhouse, B. D. (2005). Retrograde doublelabeling study of common afferent projections to the dorsal raphe and the nuclear core of the locus coeruleus in the rat. Journal of Comparative Neurology, 481, 179–193. Levitt, P., & Moore, R. Y. (1979). Origin and organization of brainstem catecholamine innervation in the rat. Journal of Comparative Neurology, 186, 505–528. Li, R., Keil, A., & Principe, J. C. (2009). Single-trial P300 estimation with a spatiotemporal filtering method. Journal of Neuroscience Methods, 177, 488–496. Liu, J., Kiehl, K. A., Pearlson, G., Perrone-Bizzozero, N. I., Eichele, T., & Calhoun, V. D. (2009). Genetic determinants of target and noveltyrelated event-related potentials in the auditory oddball response. NeuroImage, 46, 809–816. Loewy, A. D., Wallach, J. H., & McKellar, S. (1981). Efferent connections of the ventral medulla oblongata in the rat. Brain Research, 228, 63–80.

173 Lynn, R. (1966). Attention, arousal, and the orientation reaction. Oxford: Pergamon Press. Lyytinen, H., Blomberg, A. P., & Naä¨ta¨nen, R. (1992). Event-related potentials and autonomic responses to a change in unattended auditory stimuli. Psychophysiology, 29, 523–534. Maher, T. F., & Furedy, J. J. (1979). A comparison of the pupillary and electrodermal components of the orienting reflex in sensitivity of initial stimulus presentation, repetition and change. In H. D. Kimmel, E. H. Van Olst, & J. F. Orlebeke (Eds.), The orienting reflex in humans (pp. 381–391). New York: Lawrence Erlbaum Associates. Makeig, S., Delorme, A., Westerfield, M., Jung, T.-P., Townsend, J., Courchesne, E., & Sejnowski, T. J. (2004). Electroencephalographic brain dynamics following manually responded visual targets. PloS Biology, 2, 747–762. Maltzman, I. (1979a). Orienting reflexes and classical conditioning in humans. In H. D. Kimmel, E. H. Van Olst, & J. F. Orlebeke (Eds.), The orienting reflex in humans (pp. 323–351). New York: Lawrence Erlbaum Associates. Maltzman, I. (1979b). Orienting reflexes and significance: A reply to O’Gorman. Psychophysiology, 16, 274–283. Maltzman, I., Kantor, W., & Langdon, B. (1966). Immediate and delayed retention, arousal, and the orienting and defensive reflexes. Psychonomic Science, 6, 445–446. Morilak, D. A., Fornal, C. A., & Jacobs, B. L. (1987a). Effects of physiological manipulations on locus coeruleus neuronal activity in freely moving cats. I. Thermoregulatory challenge. Brain Research, 422, 17–23. Morilak, D. A., Fornal, C. A., & Jacobs, B. L. (1987b). Effects of physiological manipulations on locus coeruleus neuronal activity in freely moving cats. II. Cardiovascular challenge. Brain Research, 422, 24–31. Morilak, D. A., Fornal, C. A., & Jacobs, B. L. (1987c). Effects of physiological manipulations on locus coeruleus neuronal activity in freely moving cats. III. Glucoregulatory challenge. Brain Research, 422, 32–39. Morrison, J. H., Molliver, M. E., Grzanna, R., & Coyle, J. T. (1981). The intracortical trajectory of the coeruleo-cortical projection in the rat: A tangentially organized cortical afferent. Neuroscience, 6, 139–158. Moruzzi, G., & Magoun, H. W. (1949). Brain stem reticular formation and activation of the EEG. Electroencephalography and Clinical Neurophysiology, 1, 455–473. Murase, S., Takayama, M., & Nosaka, S. (1993). Chemical stimulation of the nucleus locus coeruleus: Cardiovascular responses and baroreflex modification. Neuroscience Letters, 153, 1–4. Nieuwenhuis, S., Aston-Jones, G., & Cohen, J. D. (2005). Decision making, the P3, and the locus coeruleus-norepinephrine system. Psychological Bulletin, 131, 510–532. Nieuwenhuis, S., Gilzenrat, M. S., Holmes, B. D., & Cohen, J. D. (2005). The role of the locus coeruleus in mediating the attentional blink: A neurocomputational theory. Journal of Experimental Psychology: General, 134, 291–307. Nieuwenhuis, S., & Jepma, M. (in press). Investigating the role of the noradrenergic system in human cognition. In T. Robbins, M. Delgado, & E. Phelps (Eds.), Decision making. Attention & Performance, Vol. XXIII. Oxford: Oxford University Press. O’Gorman, J. (1979). The orienting reflex: Novelty or significance detector? Psychophysiology, 16, 253–262. O¨hman, A., Bjorkstrand, P. A., & Ellstrom, P. E. (1973). Effect of explicit trial-by-trial information about shock probability in long interstimulus interval GSR conditioning. Journal of Experimental Psychology, 98, 145–151. Pavlov, I. P. (1927). Conditioned reflexes. Oxford: Clarendon Press. Peavler, W. S. (1974). Pupil size, information overload, and performance differences. Psychophysiology, 11, 559–566. Pfefferbaum, A., Ford, J. M., Roth, W. T., & Kopell, B. S. (1980). Age differences in P3-reaction time associations. Electroencephalography and Clinical Neurophysiology, 49, 257–265. Phillips, M. A., Szabadi, E., & Bradshaw, C. M. (2000). Comparison of the effects of clonidine and yohimbine on pupillary diameter at different illumination levels. British Journal of Clinical Pharmacology, 50, 65–68. Pineda, J. A., Foote, S. L., & Neville, H. J. (1989). Effects of locus coeruleus lesions on auditory, long-latency, event-related potentials in monkey. Journal of Neuroscience, 9, 81–93. Polich, J., & Criado, J. R. (2006). Neuropsychology and neuropharmacology of P3a and P3b. International Journal of Psychophysiology, 60, 172–185.

174 Polich, J., & Kok, A. (1995). Cognitive and biological determinants of P300: An integrative review. Biological Psychology, 41, 103–146. Pritchard, W. S. (1981). Psychophysiology of P300. Psychological Bulletin, 89, 506–540. Qiyuan, J., Richer, F., Wagoner, B. L., & Beatty, J. (1985). The pupil and stimulus probability. Psychophysiology, 22, 530–534. Rajkowski, J., Kubiak, P., & Aston-Jones, G. (1993). Correlations between locus coeruleus (LC) neural activity, pupil diameter and behavior in monkey support a role of LC in attention [Abstract]. Society for Neuroscience Abstracts, 19, 974. Ranganath, C., & Rainer, G. (2003). Neural mechanisms for detecting and remembering novel events. Nature Reviews Neuroscience, 4, 193–202. Raskin, D. C., Kotses, H., & Bever, J. (1969). Autonomic indicators of orienting and defensive reflexes. Journal of Experimental Psychology, 80, 423–433. Reiner, P. B. (1986). Correlational analysis of central noradrenergic neuronal activity and sympathetic tone in behaving cats. Brain Research, 378, 86–96. Ritter, W., Simson, R., & Vaughan, H. G., Jr. (1972). Association cortex potentials and reaction time in auditory discrimination. Electroencephalography and Clinical Neurophysiology, 33, 547–555. Ritter, W., Vaughan, H. G., Jr., & Costa, L. D. (1968). Orienting and habituation to auditory stimuli: A study of short term changes in average evoked responses. Electroencephalography and Clinical Neurophysiology, 25, 550–556. Robbins, T. W. (1997). Arousal systems and attentional processes. Biological Psychology, 45, 57–71. Roth, W. T. (1973). Auditory evoked responses to unpredictable stimuli. Psychophysiology, 10, 125–138. Roth, W. T. (1983). A comparison of P300 and the skin conductance response. In A. W. K. Gaillard & W. Ritter (Eds.), Tutorials in ERP researchFEndogenous Components (pp. 177–199). Amsterdam: North-Holland Publishing Co. Roth, W. T., Blowers, G. H., Doyle, C. M., & Kopell, B. S. (1982). Auditory stimulus intensity effects on components of the late positive complex. Electroencephalography and Clinical Neurophysiology, 54, 132–146. Roth, W. T., Dorato, K. H., & Kopell, B. S. (1984). Intensity and task effects on evoked physiological responses to noise bursts. Psychophysiology, 21, 466–481. Rushby, J. A., & Barry, R. J. (2007). Event-related potential correlates of phasic and tonic measures of the orienting reflex. Biological Psychology, 75, 248–259. Rushby, J. A., & Barry, R. J. (2009). Single-trial event-related potentials to significant stimuli. International Journal of Psychophysiology, 74, 120–131. Rushby, J. A., Barry, R. J., & Doherty, R. J. (2005). Separation of the components of the late positive complex in an ERP dishabituation paradigm. Clinical Neurophysiology, 116, 2363–2380. Rust, J. (1977). Habituation and the orienting response in the auditory cortical evoked potential. Psychophysiology, 14, 123–126. Saiers, J. A., & Campbell, B. A. (1990). Disruption of noradrenergic, but not serotonergic or opiate, functioning blocks both cardiac and behavioral components of the orienting response in preweanling rats. Behavioral and Neural Biology, 54, 254–270. Sara, S. J. (2009). The locus coeruleus and noradrenergic modulation of cognition. Nature Reviews in Neuroscience, 10, 211–223. Sara, S. J., Vankov, A., & Herve, A. (1994). Locus coeruleus-evoked responses in behaving rats: A clue to the role of noradrenaline in memory. Brain Research Bulletin, 35, 457–465. Scheibel, A. B. (1980). Anatomical and physiological substrate of arousal: A view from the bridge. In J. A. Hobson & M. A. B. Brazier (Eds.), The reticular formation revisited (pp. 55–66). New York: Raven Press. Servan-Schreiber, D., Printz, H., & Cohen, J. D. (1990). A network model of catecholamine effects: Gain, signal-to-noise ratio, and behavior. Science, 249, 892–895. Siddle, A. T., & Heron, P. A. (1976). Effects of length of training and amount of tone frequency change on amplitude of autonomic components of the orienting response. Psychophysiology, 13, 281–287. Siddle, D. A., O’Gorman, J. G., & Wood, L. (1979). Effects of electrodermal lability and stimulus significance on electrodermal response amplitude to stimulus change. Psychophysiology, 16, 520–527.

S. Nieuwenhuis et al. Siddle, D. A., Remington, B., & Churchill, M. (1984). Effects of stimulus change on the electrodermal orienting response. Biological Psychology, 18, 33–39. Siddle, D. A., & Spinks, J. A. (1979). Orienting response and information-processing: Some theoretical and empirical problems. In H. D. Kimmel, E. H. Van Olst, & J. F. Orlebeke (Eds.), The orienting reflex in humans (pp. 473–497). New York: Lawrence Erlbaum Associates. Simons, R. F. (1988). Event-related slow brain potentials: A perspective from ANS psychophysiology. Advances in Psychophysiology, Vol. 3, 223–267. Simons, R. F., Graham, F. K., Miles, M. A., & Chen, X. (2001). On the relationship of P3a and the novelty-P3. Biological Psychology, 56, 207–218. Simons, R. F., Rockstroh, B., Elbert, T., Fiorito, E., Lutzenberger, W., & Birbaumer, N. (1987). Evocation and habituation or autonomic and event-related potential responses in a nonsignal environment. Journal of Psychophysiology, 1, 45–59. Sokolov, E. N. (1963). Perception and the Conditioned Reflex. Oxford: Pergamon Press. Sokolov, E. N. (1975). The neuronal mechanisms of the orienting reflex. In E. N. Sokolov & O. S. Vinogradova (Eds.), Neuronal mechanisms of the orienting reflex (pp. 217–235). Hillsdale, NJ: Erlbaum. Sokolov, E. N., Spinks, J. A., Naä¨ta¨nen, R., & Lyytinen, H. (2002). The orienting response in information processing. London: Lawrence Erlbaum Associates. Soltani, M., & Knight, R. T. (2000). Neural origins of the P300. Critical Reviews in Neurobiology, 14, 199–224. Spencer, K. M., Dien, J., & Donchin, E. (2001). Spatiotemporal analysis of the late ERP responses to deviant stimuli. Psychophysiology, 38, 343–358. Steinhauer, S. R., & Hakerem, G. (1992). The pupillary response in cognitive psychophysiology and schizophrenia. In D. Friedman & G. Bruder (Eds.), Psychophysiology and experimental psychopathology: A tribute to Samuel Sutton. Annals of the New York Academy of Sciences, 658, 182–204. Steinhauer, S. R., & Zubin, J. 1982. Vulnerability to schizophrenia: Information processing in the pupil and event-related potential. In E. Usdin & I. Hanin (Eds.), Biological markers in psychiatry and neurology (pp. 371–385). Oxford: Pergamon Press. Stelmack, R. M., & Siddle, D. A. (1982). Pupillary dilation as an index of the orienting reflex. Psychophysiology, 19, 706–708. Sutton, S., Braren, M., Zubin, J., & John, E. R. (1965). Evoked-potential correlates of stimulus uncertainty. Science, 150, 1187–1188. Sutton, S., Tueting, P., Zubin, J., & John, E.R (1967). Information delivery and the sensory evoked potential. Science, 155, 1436–1439. Suwazono, S., Shibasaki, H., Nishida, S., Nakamura, M., Honda, M., Nagamine, T., et al. (1994). Automatic detection of P300 in single sweep records of auditory event-related potential. Journal of Clinical Neurophysiology, 11, 448–460. Sved, A. F., Cano, G., & Card, J. P. (2001). Neuroanatomical specificity of the circuits controlling sympathetic outflow to different targets. Clinical and Experimental Pharmacology and Physiology, 28, 115–119. Sved, A. F., & Felsten, G. (1987). Stimulation of the locus coeruleus decreases arterial pressure. Brain Research, 414, 119–132. Svensson, T. H. (1987). Peripheral, autonomic regulation of locus coeruleus noradrenergic neurons in brain: Putative implications for psychiatry and psychopharmacology. Psychopharmacology, 92, 1–7. Swick, D., Pineda, J. A., & Foote, S. L. (1994). Effects of systemic clonidine on auditory event-related potentials in squirrel monkeys. Brain Research Bulletin, 33, 79–86. Szabadi, E., & Bradshaw, C. M. (1996). Autonomic pharmacology of a2adrenoceptors. Journal of Psychopharmacology, 10, 6–18. Turpin, G., & Siddle, D. A. (1979). Effects of stimulus intensity on electrodermal activity. Psychophysiology, 16, 582–591. Usher, M., Cohen, J. D., Servan-Schreiber, D., Rajkowski, J., & AstonJones, G. (1999). The role of locus coeruleus in the regulation of cognitive performance. Science, 283, 549–554. Van Bockstaele, E. J., Aston-Jones, G., Ennis, M., Shipley, M. T., & Pieribone, V. A. (1991). Subregions of the periaqueductal gray topographically innervate the rostral ventrolateral medulla in the rat. Journal of Comparative Neurology, 309, 305–327. Van Bockstaele, E. J., Pieribone, V. A., & Aston-Jones, G. (1989). Diverse afferents converge on the nucleus paragigantocellularis in the

P3 and orienting response rat ventrolateral medulla: Retrograde and anterograde tracing studies. Journal of Comparative Neurology, 290, 561–584. Van Olst, E. H., Heemstra, M. L., & Ten Kortenaar, T. (1979). Stimulus significance and the orienting reaction. In H. D. Kimmel, E. H. Van Olst, & J. F. Orlebeke (Eds.), The orienting reflex in humans (pp. 521– 547). New York: Lawrence Erlbaum Associates. Verbaten, M. N. (1983). The influence of information on habituation of cortical, autonomic and behavioral components of the orienting response. In A. W. K. Gailard & W. Ritter (Eds.), Tutorials in ERP Research: Endogenous Components (pp. 201–216). Amsterdam: Elsevier. Verberne, A. J., & Owens, N. C. (1998). Cortical modulation of the cardiovascular system. Progress in Neurobiology, 54, 149–168. Verleger, R. (1997). On the utility of P3 latency as an index of mental chronometry. Psychophysiology, 34, 131–156. Wu, H. B., Stavarache, M., Pfaff, D. W., & Kow, L. M. (2007). Arousal of cerebral cortex electroencephalogram consequent to high-frequency stimulation of ventral medullary reticular formation. Proceedings of the National Academy of Sciences, 104, 18292–18296.

175 Yamamoto, K., Arai, H., & Nakayama, S. (1990). Skin conductance response after 6-hydroxydopamine lesion of central noradrenaline system in cats. Biological Psychiatry, 28, 151–160. Yamamoto, K., Hoshino, T., Takahashi, Y., Kaneko, H., & Ozawa, N. (1991). Skin conductance activity after intraventricular administration of 6-hydroxydopa in rats. Biological Psychiatry, 29, 365–375. Yamamoto, K., Ozawa, N., Shinba, T., & Hoshino, T. (1994). Functional influence of the central noradrenergic system on the skin conductance activity in rats. Schizophrenia Research, 13, 145–150. Yeung, N., & Sanfey, A. G. (2004). Independent coding of reward magnitude and valence in the human brain. Journal of Neuroscience, 24, 6258–6264. Yu, A. J., & Dayan, P. (2005). Uncertainty, neuromodulation, and attention. Neuron, 46, 681–692.

(Received November 5, 2009; Accepted April 2, 2010)


The N400 as a snapshot of interactive processing: Evidence from regression analyses of orthographic neighbor and lexical associate effects

SARAH LASZLOa and KARA D. FEDERMEIERb,c,d a

Department of Psychology, Carnegie Mellon University, Pittsburgh, Pennsylvania Department of Psychology, University of Illinois at Urbana Champaign, Urbana, Illinois c Program in Neuroscience, University of Illinois at Urbana Champaign, Urbana, Illinois d Beckman Institute for Advanced Science and Technology, University of Illinois at Urbana Champaign, Urbana, Illinois b

Abstract Linking print with meaning tends to be divided into subprocesses, such as recognition of an input’s lexical entry and subsequent access of semantics. However, recent results suggest that the set of semantic features activated by an input is broader than implied by a view wherein access serially follows recognition. EEG was collected from participants who viewed items varying in number and frequency of both orthographic neighbors and lexical associates. Regression analysis of single item ERPs replicated past findings, showing that N400 amplitudes are greater for items with more neighbors, and further revealed that N400 amplitudes increase for items with more lexical associates and with higher frequency neighbors or associates. Together, the data suggest that in the N400 time window semantic features of items broadly related to inputs are active, consistent with models in which semantic access takes place in parallel with stimulus recognition. Descriptors: N400, Semantic access, Multiple regression, Orthographic neighborhood

a template- or feature-matching process to be completed before semantic information can be retrieved have also been proposed for face and object recognition (e.g., for faces, Brunelli & Poggio, 1993; for objects, Poggio & Edelman, 1990). Staged models of word processing that involve an isolated recognition process successfully explain a range of behavioral findings, notably the complex, interacting effects of frequency, context, and stimulus quality on lexical decision reaction times (e.g., Borowsky & Besner, 1993; Stolz & Neely, 1995). However, such models make two specific predictions about the semantic processing that follows recognition that seem incongruent with data from the event-related potential (ERP) literature pertaining to the N400 component, a well-established, functionally specific marker of lexico-semantic processing (for review, see Kutas, Van Petten, & Kluender, 2007). First, if semantic access can only proceed after recognition has been successfully completed, then access should never be attempted for items without lexical representations, such as pseudowords or orthographically illegal consonant strings. Second, if semantic access is essentially limited to the process of looking up meaning information associated with a recognized lexical entry, then the largely inconsistent semantics of other orthographically or lexically associated items should never become simultaneously active. That is, for the input FORK, the semantics of the orthographically similar PORK and the lexically associated SPOON should never become active. If the first prediction is true, and semantic access will only be attempted for items corresponding to a known lexical entry, then

A concept that has classically been important to psycholinguistic theories of visual word processing is recognition, a process by which orthographic inputs are compared to internal representationsFoften items in the mental lexiconFin order to find a match to the input that can subsequently be linked with semantics. This type of staged recognition process is exemplified in Forster and colleagues’ Entry Opening Model (e.g., Forster, 1999; Forster & Davis, 1984; Forster & Veres, 1998), wherein information corresponding to an orthographic input cannot be retrieved until a matching lexical entry has been identified. A similar formulation is found in the Multistage Activation Model of Besner and colleagues (e.g., Besner & Chapnik Smith, 1992; Borowsky & Besner, 1993; Plourde & Besner, 1997), in which orthographic analysis of an input must be complete (that is, the input must be recognized) before associated information can be passed forward for subsequent processing. Theories that require The authors wish to acknowledge B. Armstrong, B. Gonsalves, C. Lee, K. Mathewson, G. Miller, D. Plaut, and E. Wlotko for insightful discussion of the single item data set, as well numerous research assistants for their efforts in data collection and processingFespecially P. Anaya, H. Buller, and C. Laguna. This research was supported by National Institute of Mental Health Training grant T32 MH019983 to Carnegie Mellon University, which supported SL, and NIA grant AG26308 to KDF. Address correspondence to: Sarah Laszlo, Carnegie Mellon University, 5000 Forbes Ave, Baker Hall 254T, Pittsburgh, PA 15213. E-mail: [email protected] 176

N400 neighbor and associate affects the N400Fwhich has been established as a marker of attempted or successful semantic access (see, for example, Federmeier & Laszlo, 2009; Kutas & Federmeier, 2000)Fshould only be observed in response to items with lexical representations. However, this is not the case. Clear N400 components and N400 effects (such as reductions in amplitude with repetition) have long been observed in response to pronounceable pseudowords, such as GORK (e.g., Rugg & Nagy, 1987; Deacon, Dynowska, & Grose-Fifer, 2004; Laszlo & Federmeier, 2007). This finding necessitates weakening the proposal that semantic access occurs only for lexically represented items to, at minimum, allow for attempts at semantic access for strings that are very similar to lexically represented items (i.e., pseudowords, which are often created by changing one letter of a real word). However, even this weakened proposal is incompatible with recent work from our lab, which has shown that, at least in a supportive sentence context, even meaningless, illegal letter strings (e.g., NHK) with minimum orthographic neighborhood size (i.e., Coltheart’s N: the number of words that can be formed from a target by changing one of its letters) can elicit clear N400 components and N400 effects (Laszlo & Federmeier, 2008, 2009). We have argued that this pattern indicates that semantic access is attempted for all orthographic inputs, regardless of their lexical status, although the success of that attempted access can vary with contextFexplaining why, for example, unfamiliar, orthographically illegal strings embedded in word lists have been found to not show N400 repetition effects (Rugg & Nagy, 1987), whereas, when embedded in sentences, these same types of strings do elicit N400 effects associated with similarity to a predicted completion (Laszlo & Federmeier, 2009). Although seemingly incompatible with staged recognition models, the ERP findings are compatible with Parallel Distributed Processing (PDP) models of reading that result in some semantic features becoming at least initially active in response to all inputs (e.g., Harm & Seidenberg, 2004). In models of this type, processing that appears staged can result from nonlinear activation dynamics between orthography and semantics (Kello, Plaut, & MacWhinney, 2000). Importantly, however, even when such models exhibit stage-like behavior, this is accomplished without any formal implementation of stages and also without any formal distinction between the processing of lexically represented and unrepresented items (a distinction that is a necessary consequence of strongly staged models). Thus, data that have often been explained with staged models can also be explained with cascaded models, which, additionally, are consistent with ERP findings showing that non-lexical items engage attempts at semantic access that seem identical in timing and in nature to those engaged by lexically represented items. The second prediction of staged models of reading outlined above is also not shared by PDP models. Because staged models assume that items are identified before semantic access begins, there is no reason to predict that semantic features of orthographically similar or lexically associated items should become active (to any significant degree) along with the features of the input stimulus. For example, having recognized an input as FORK, the system would not access semantics associated with the orthographically similar input PORK. In contrast, given the tendency of PDP models to activate similar outputs in response to similar inputs, semantic features associated with a range of items similar to the input stimulus at the orthographic or lexical levels of analysis can become active in parallel with the appropriate semantics for the input, at least transiently. Thus, in a PDP

177 model given the input string ‘‘FORK,’’ both the semantics of FORK and PORK could initially become active, as both are at least partially consistent with the input (i.e., contain ORK), although, of course, the semantics of FORK are more consistent with the input and would eventually become most active. Again, recent ERP data are more in line with the predictions of cascaded models than staged ones. In particular, Holcomb, Grainger, and O’Rourke (2002) found that out-of-context N400 amplitudes were larger to words and pseudowords with higher orthographic neighborhood sizes, and we have replicated that finding and shown that it extends to illegal strings of letters (Laszlo & Federmeier, 2007) and that the amplitude difference is maintained even when items are embedded in sentences (Laszlo & Federmeier, 2008, 2009). We have argued that the larger N400s to items with high N result from there beingFat least initiallyFmore semantic activation for items that are orthographically similar to many other items. That is, a high N input like CAT activates not only its own semantics, but also, briefly, the semantics of all its neighbors, whereas a low N input like OWL results in a less broad activation at the semantic level of representation. Importantly, the fact that effects of N are identical for lexical and non-lexical inputs (Laszlo & Federmeier, 2009) suggests not only that a broader range of semantic features becomes active in response to an input than staged models would predict, but also that lexical status per se is not a determining factor in this semantic level effect. Although effects of neighborhood density on the N400 can be taken to suggest that a range of orthographically similar and lexically associated items become active in response to any given input, it could potentially be argued that N actually reflects a property of the input item itselfFN might instead be a proxy for some information about orthographic regularity that is included in an item’s lexical entry. For example, maybe the number of neighbors an item has could be an abstracted proxy for how similarly to other words that item is pronounced, and thus whether or not it can be pronounced by rule or must be considered an exception. A stronger test of the hypothesis that the N400 reflects the processing of not just an input, but also items similar to that input, could thus potentially come from examining the effect of the frequency of an item’s orthographic neighbors on the magnitude of the N400 elicited by that item. Such an effect, if observed, would indicate directly that properties of items similar to an input affect its semantic processing. In fact, one study has reported such an effect, finding that items with high frequency neighbors elicited more negative N400s than items with lower frequency neighbors (Debruille, 1998). Unfortunately, however, neighbor frequency was confounded with orthographic neighborhood size in that study, making it difficult to strongly conclude that it was the frequency of an item’s neighbors, and not just the number of neighbors, that affected N400 amplitude. Therefore, the first goal of the present study was to determine whether neighbor frequency has an effect on the N400 independent of the effect of N. Further, while neighbor frequency is a property of items orthographically similar to an input, our second goal was to determine whether the properties of items lexically associated to an input might also affect its processing. Specifically, we were interested in examining the effects, if any, of the number of lexical associates and written frequency of the top associate on N400 amplitudes, as these might be considered lexical level analogues of N and neighbor frequency. For example, if FORK can activate some of the semantics of PORK by virtue of their shared

178 orthography, can DOG also activate some of the semantics of BONE by virtue of their lexical association? The cascaded nature of the information flow between representational levels in the modeling framework that has thus far been most consistent with N400 effect patterns would seem to predict such effectsF through spreading activation at the lexico-semantic level of representationFbut, to our knowledge, no N400 data addressing this issue exist. Our two experimental goals thus have much the same flavor: each is aimed at trying to determine whether or not the properties of items similar to (or linked to) an input at the orthographic or lexical levelFand thus likely to become active in parallel during input processingFaffect the semantic processing of that input. Evidence for such effects would support cascaded models over staged models of reading, and, in the case of the orthographic variables, this conclusion could be strengthened by an absence of an interaction with lexicality, as staged models predict important differences between lexically represented and non-represented stimuli at processing stages, such as semantic access, that are assumed to follow recognition. We took a somewhat novel approach to these goals. The typical design of an ERP experiment aiming to examine, for example, the effect of neighbor frequency independent of the effect of N might be a factorial one wherein participants view items high and low in neighbor frequency but matched on N. Although this design would provide information about the impact of neighbor frequency on the ERP, it would do so at the expense of not affording information about the simultaneous effect of NFa downside because, of course, these variables apply to all inputs and are never processed in isolation. To address this problem, some studies have begun moving toward the use of designs that enable multiple regression analyses (e.g., Hauk, Davis, Ford, Pulvermuller, & Marslen-Wilson, 2006; Hauk, Pulvermuller, Ford, Marslen-Wilson, & Davis, 2009; King & Kutas, 1998), in order to attempt to untangle the effects of linguistic variables which tend to be highly correlated (e.g., length and word frequency, in the case of King & Kutas, 1998). Multiple regression when applied to items can afford the identification of independent effects of each variable of interest while avoiding the artificiality of attempting to examine the effects of lexical variables in isolation. Multiple regression can be particularly useful for unraveling effects of intercorrelated item variables when combined with items-based analyses (as opposed to subjects-based analyses, which do not permit generalization across items). Despite this advantage, multiple regression has not often been used to examine dependent variables measured over items in ERP studies, because item data with satisfactory signal-to-noise characteristics is not generally available with the numbers of participants typically run in ERP studies (although, for an interesting exception, see Rey, Dufau, Massol, & Grainger, 2009, who extracted item ERPs representing the response to single letters or pseudoletters). In an approach similar to the one we employ in the present study, Dambacher, Kliegl, Hofman, and Jacobs (2006) used simultaneous multiple regression to model the single trial electroencephalogram (EEG) collected from participants reading sentences, and supported cascaded models of word recognition over staged ones; however, high noise levels in the item ERPsFcollected from only 50 participantsFresulted in relatively low R2 values for their multiple regression models. We, therefore, sought to address this issue by collecting a large scale data set from 120 participants who viewed words, pseudowords,

S. Laszlo & K. D. Federmeier acronyms, and illegal strings that intentionally varied widely in their lexical characteristics (including the four presently of interest). With this data, we could form ERPs representing the responses to single items, averaged across participants (e.g., the response to the word DOG only, consisting of 120 trialsFone from each participant.) This data set enables us to generalize over items in a way that is not possible in a typical ERP design where approximately 40 items per condition for approximately 20–30 participants might be collected. Figure 1 displays an unfiltered example from each item type, showing that these single item ERPs were stable, with good signal-to-noise ratios. With stable ERPs available for individual items, it is then possible to obtain item–level mean N400 amplitude measures (or, of course, any other measure that can be obtained from a more typical, item-aggregated ERP). Those single item means are then eligible for regression analyses that are not possible with subject aggregated data. One drawback of this approach is that items analysis does not permit generalization across subjects. However a substantial benefit of this approach is that regression is a more powerful analysis method than analysis of variance; another is that, with multiple regression, the independent effects

Middle Parietal 5 µV

DOG 900 ms

DAWK

DVD

DSN

Figure 1. Example single item ERPs: Each ERP is an average of one EEG sweep over the middle parietal channel from each of 120 participants in response to a single item: the word DOG, the pseudoword DAWK, the acronym DVD, and the illegal string DSN. In this figure, as in all subsequent ones, negative is plotted up. These ERPs are unfiltered, which makes it evident that the signal-to-noise characteristics of the single item ERPs are satisfactory.

N400 neighbor and associate affects

179 by a given participant were included in the averaged ERPs computed for that participant. Table 1 displays mean lexical characteristics of each item type (i.e., length, frequency, N, orthographic neighborhood frequency, number of lexical associates, and frequency of top associate), along with examples. Orthographic neighborhood size was computed as the total number of words that could be formed by replacing one letter of a target item, as indicated by the Medical College of Wisconsin Orthographic Wordform Database (Medler & Binder, 2005). Neighbor frequency was, in turn, computed as the logarithm of the summed frequency of all of an item’s orthographic neighbors, with frequency estimates drawn from the Wall Street Journal corpus (Marcus, Santorini, & Marcinkiewicz, 1993). An additional analysis of neighbor frequency considered only the log of the maximum frequency neighbor of each item, as opposed to the sum of the frequencies of all neighbors. Number of lexical associates was retrieved from the South Florida Free Association Norms (Nelson, McEvoy, & Schreiber, 1998), and the log frequency of each item’s top lexical associate was again obtained using the Wall Street Journal corpus (Marcus et al., 1993). Critical experimental items (i.e., words, pseudowords, acronyms, and illegal strings) were each repeated one time at a lag of 0, 2, or 3 intervening items, allowing us to examine the stability of any effects we might observe across presentations. Each level of repetition lag occurred an equal number of times both within and across item types. Participants did not respond to the critical items, in order to prevent contamination of the critical ERPs by response potentials. The proper names served as the targets for the behavioral task, and were only presented once. Participants responded to proper names by pressing a button with their right hand. False alarms (i.e., button presses to critical items) were not included in averaged ERPs. The experiment thus included 750 trials (2 ! 300 critical items 1150 proper names). These 750 trials were broken up into 5 blocks of 150 trials with rest breaks between each block. Across the 120 participants, each of the 120 permutations of 5 block orders was presented exactly once.

of multiple variables can be examined simultaneously (e.g., the effects of N and neighbor frequency). Given past results from factorial studies (Holcomb et al., 2002; Laszlo & Federmeier, 2007, 2008, 2009), we predicted that neighborhood size would be positively correlated with N400 magnitude, independent of the lexical status of the input string. We predicted a similar relationship between number of lexical associates and N400 response to words (the only class of items for which lexical association data is available). Critically, although neighbor frequency and neighborhood size tend to be correlated, we also predicted an independent effect of orthographic neighbor frequency on N400 mean amplitude (and a similar effect of frequency of top associate), indicating that the spread of semantic activation elicited by an input is considerably broader than would be suggested under a staged account.

Methods Participants Data were analyzed from 120 participants (58 female, age range 18–24, mean age 19.1). Data from 6 additional participants were discarded due to either unsatisfactory levels of ocular artifact or EEG digitization equipment malfunction. All participants were right-handed, monolingual speakers of English with normal or corrected-to-normal vision and no history of neurological disease or defect. Participants were graduate or undergraduate students at the University of Illinois. The experimental protocol was approved by the Internal Review Board of the University of Illinois, and all participants were compensated with money or course credit.

Stimuli Stimuli were 75 each words (e.g., HAT, MAP), pseudowords (e.g., DAWK, KAK), meaningless, illegal strings (e.g., CKL, KKB), and familiar, orthographically illegal acronyms (e.g., VCR, AAA). Additionally, 150 common proper American first names (e.g., SARA, JOHN) served as targets in the substantive behavioral task, which was to monitor the stream of unconnected text for names and press a button when a name was detected. All items were between 3 and 5 letters long (mean 3.19). Words, pseudowords, illegal strings, and acronyms were the critical experimental items; no response was made to these items. Illegal strings and acronyms were composed of all consonants or all vowels. Acronym familiarity was assessed by a paper and pencil post-test (identical to that described in Laszlo & Federmeier, 2007), and only EEG responses to acronyms correctly identified

Procedure Participants were seated 100 cm away from a computer monitor and instructed that their task was to press a button whenever they were presented with a ‘‘common English proper first name,’’ and to minimize blinks and eye movements except during a blink interval indicated on the screen by the presence of a white cross. After a demonstration of trial structure, participants were presented with a short block of practice trials consisting of items similar to those in the experiment proper. In both the practice and experimental blocks, a fixation arrow was continuously present in the center of the screen. Participants

Table 1. Selected Lexical Characteristics

Item type Word Pseudoword Acronym Illegal string

Examples

Length

Log written frequency

N

Log neighborhood frequency

Number of lexical associates

Log frequency of top associate

HAT, MAP TUL, KAK VCR, AAA CKL, KKB

3.2 3.2 3.2 3.2

2.39 – 0.96 –

12.99 11.04 1.93 2.4

4.32 4.12 2.71 2.96

10.53 – – –

2.45 – – –

Note: By design, the lexical characteristics of the items included in the single item ERP corpus varied broadly. N was estimated from the Medical College of Wisconsin Orthographic Wordform Database (Medler & Binder, 2005). All frequency estimates were drawn from the Wall Street Journal Corpus (Marcus, Santorini, & Marcinkiewicz, 1993). Number of lexical associates was estimated from the South Florida Free Association Norms (Nelson, McEvoy, & Schreiber, 1998).

180 were instructed to keep their eyes on the fixation arrow as much as possible. Stimuli were presented one at a time in white directly above the fixation arrow on the black background of a 22-inch CRT computer monitor with resolution 640 ! 480. Trial structure was as follows: 500 ms warning stimulus (red cross above the fixation arrow), 500 ms stimulus presentation, 1000 ms response interval (fixation arrow present only), 1000 ms blink interval (white cross above the fixation arrow). After the 5 experimental blocks, participants completed the paper and pencil acronym knowledge questionnaire (described in Laszlo & Federmeier, 2007), in order to permit sorting of the acronym items as familiar or unfamiliar on an individual basis. In brief, the questionnaire required participants to indicate whether each of the acronyms and illegal strings presented in the EEG experiment were acronyms or not acronyms. If participants believed an item was an acronym, they had the option of indicating what the letters in the acronym stood for, writing a sentence showing what the acronym meant, or selecting ‘‘Don’t Know,’’ in instances when they ‘‘had heard other people use it before, but [didn’t] know what the letters in it stand for and couldn’t use it [themselves].’’ Only items for which participants could identify all the letters or could write a sentence were included in subsequent ERP analyses. This method has proved reliable in the past for sorting acronym stimuli into classes distinguished in the ERP signal (Laszlo & Federmeier, 2007, 2008). On average, participants were able to correctly identify 83% of acronyms (" 62/75).

EEG Recording EEG was recorded from 6 Ag/AgCl electrodes embedded in an electrocap. We sampled from middle prefrontal, middle parietal, middle central, left middle central, right middle central, and middle occipital electrode sites. This reduced electrode montage was necessary in order to enable the collection of 120 participants in a reasonable period of time. Because our focus was on the N400 component, we chose a montage that provided good coverage of the region of the scalp where N400 effects are typically maximal (i.e., the central posterior scalp), as well as one prefrontal site to confirm the posterior distribution of observed effects. All EEG electrodes were referenced online to the left mastoid process and then digitally re-referenced offline to the average of the left and right mastoids. The electrooculogram (EOG) was recorded using a bipolar montage of electrodes placed at the outer canthi of the left and right eyes; blinks were monitored with an electrode at the suborbital ridge. EEG and EOG were recorded with a bandpass of 0.02 to 100 Hz and sampled at a rate of 250 Hz with a gain of 10,000!. All electrode impedances were kept below 2 kO. Single item ERPs were computed by averaging (across the 120 subjects) at each electrode time-locked to the onset of each of the critical items (resulting in 600 single item ERPs: one for each of two presentations of each of 300 critical items). In addition to the single item ERPs, more traditional ERPs representing the average within-subject response to, for example, all words, were also computed. Trials containing eye movement or drift artifact were rejected with a threshold individualized to each participant by inspection of that participant’s raw waveforms, and blinks were corrected using a procedure described by Dale (1994). Artifact rejection resulted in an average loss of 7% of trials per participant. All ERPs contained a 100 ms pre-stimulus baseline and continued for 920 ms after stimulus

S. Laszlo & K. D. Federmeier onset. Measurement of ERP mean amplitude was conducted on data digitally filtered off-line with a bandpass of 0.2 to 20 Hz. Results Behavioral Data Correct behavioral responses were either to press a button in the right hand in response to a name, or to press nothing in response to any other item type. Thus a hit was a button press for a name, and a correct rejection was no button press for a critical item. Participants made on average 137/150 hits (s 5 10.2), or 91% accuracy, for the names, and on average 589/600 (s 5 16.5) correct rejections, or 98% accuracy, to critical items. Overall, these results indicate that participants were appropriately attending to the substantive behavioral task, and, more importantly, that they were processing each item in the text stream. Electrophysiological Data Three types of analysis are reported: 1) factorial analyses including item Analyses of Variance (ANOVAs) and, where appropriate, non-parametric factorial tests, 2) single regressions over items, and 3) multiple regressions over items. In what follows, we first present factorial analyses and single regressions pertaining to each of the four single lexical factors of interest (i.e., N, neighbor frequency, number of lexical associates, and frequency of top associate). We then present multiple regressions pertaining to combinations of those variables. For all analyses, the N400 was measured as mean amplitude in a 250–450 ms post stimulus onset window, relative to a 100 ms pre-stimulus baseline. The N400 was measured over the middle parietal channel only. The reduced electrode montage made analyses including data from each of the 6 electrode channels relatively uninformative; all reported effects were qualitatively similar across all five central-posterior channels. Orthographic Neighborhood Size We began with a 2 ! 2 item ANOVA with factors of orthographic neighborhood size (high or low) and lexical type (lexical: word and acronym, or nonlexical: pseudoword and illegal string). This ANOVA revealed a main effect of N (F(1, 296) 5 159.7, po.0001), but no effect of lexical type (F 5 .19) and no interaction (F 5 1.1). Indeed, as is depicted in Figure 2, the relationship between N and N400 amplitude is nearly identical for the two lexical types. The single regression correlations of N on N4 mean amplitude for lexical and nonlexical items are r 5 # .61 (r2 5 .37, po.0001) and r 5 # .49 (r2 5 .24, po.0001), respectively. The equivalence of the N effect for lexical and nonlexical itemsFand the strong effect of N on N400 amplitudeFis reiterated in Figure 3, which shows item ERPs for a low, mid, and high N item from each lexical category separately. Because the N effect is so similar across lexical category, in what follows we will sometimes collapse across lexical category when considering N effects (for example, when collapsed across lexical category, the single correlation of N with N400 mean amplitude has r 5 # .55, r2 5 .30, po.0001). The same pattern of N effect was also observed on second presentation. An identical item ANOVA with factors of N (high or low) and lexical type (lexical or nonlexical) revealed a main effect of N (F(1,296) 5 39.6, po.0001), but no main effect of lexical type (F 5 3.23) and no interaction between the two (F 5 .13). The single regression correlations of N with N4 amplitude were mildly reduced but still highly reliable on second presentation. For lexical items, r 5 # .43 (r2 5 .19, po.0001),

N400 neighbor and associate affects –1 0

181

Orthographic Neighborhood Size 5

10

15

20

25

N400 Mean Amplitude (µV)

1 2 3 4 5 6

Lexical Items (r2 = .37) Nonlexical Items (r2 = .24)

7 8

Figure 2. Equivalency of N effect across lexical types: Item N400 mean amplitude (250–450 ms) over the middle parietal channel is plotted against orthographic neighborhood for lexical items (filled circles) and nonlexical items (empty circles). Single regression trend lines for the relationship between N4 mean amplitude and N are also plotted for each item type. The function relating N400 amplitude to N is nearly identical for the two item types.

and for nonlexical items r 5 ! .33 (r2 5 .11, po.0001). Thus, across both first and second presentation, items with high N elicited more negative N400s than did items with low N, regardless of lexical type. Neighbor Frequency Our analysis of neighbor frequency effects mirrored our analysis of N effects. Again, we began with an item ANOVA with factors of (summed) neighbor frequency (high or low) and lexical type (lexical or nonlexical), which revealed a main effect of neighbor frequency (F(1,296) 5 53.0, po.0001), but no main effect of lexical type (F 5 .15) and no interaction (F 5 .81). The single re-

gression correlations of summed neighbor frequency with N4 amplitude were also both strongly reliable (for lexical items, r 5 ! .48, r2 5 .23, po.0001; for nonlexical items, r 5 ! .39, r2 5 .15, po.0001). As was the case with the effect of N, the effect of orthographic neighbor frequency was nearly identical across lexical types. The strikingly similar pattern is displayed in Figure 4. An identical ANOVA conducted with a neighbor frequency measure consisting of the frequency of each item’s highest frequency neighbor (as opposed to the summed frequency of all its neighbors) yielded the same pattern of results, with a main effect of maximum neighbor frequency (F(1,296) 5 21.66, po.0001) but no main effect of lexical type (F 5 .13) and no interaction between the two (Fo0.1). Similarly, the single regressions of maximum neighbor frequency with N4 amplitude were reliable for both lexical types (for lexical items r 5 ! .28, r2 5 .08, po.001; for nonlexical items r 5 ! .25, r2 5 .06, p 5 .002). Figure 5 displays waveforms evincing the neighbor frequency effect, aggregated over items and lexical types. As with N, items with higher neighbor frequencies elicit larger N400s, regardless of lexical type. Also as in the case of N, the same pattern of effects was observed on second presentation. An item ANOVA with factors of neighbor frequency (high or low) and lexical type (lexical or nonlexical) again revealed a strongly reliable main effect of neighbor frequency (F(1,296) 5 27.8, po.0001), but no main effect of lexical type (F 5 3.14) and no interaction (F 5 2.83). Also as with N, on second presentation the correlations between neighbor frequency and N4 amplitude were reduced, but still highly reliable, for both lexical types (for lexical items, r 5 ! .39, r2 5 .15, po.0001; for nonlexical items r 5 ! .34, r2 5 .12, po.0001). Thus, just as with N, items with higher neighbor frequency elicit larger N400s on both first and second presentation, regardless of lexical type. We again conduced an identical set of analyses using the frequency of the most frequent neighbor (as opposed to the summed frequency of all neighbors) on second presentation. An item ANOVA with factors of max neighbor frequency (high or low) and lexical type (lexical or nonlexical) showed that with this measure of neighbor frequency, on second presentation, there

Middle Parietal

5 µV

900 ms

LAD: N = 22, High

BAP: N = 22, High

LCD: N = 5, Mid

BNN: N = 5, Mid

NPR: N = 1, Low

MVH: N = 1, Low

Figure 3. N effect in item ERPs: Left, the ERPs elicited by lexical items with Ns of 1, 5, and 22: the word LAD, and the acronyms LCD and NPR (Liquid Crystal Display and National Public Radio). Right, ERPs elicited by nonlexical items with the same Ns: the pseudoword BAP and the illegal strings of letters BNN and MVH. Individual N is strong predictor of N400 amplitude, regardless of lexical type.

182

S. Laszlo & K. D. Federmeier Summed Log Neighbor Frequency


−1 0

0

1

2

3

4

5

6

tion. The rank sum test was only marginally reliable (p 5 .12). However, the more powerful single regression of number of lexical associates with N4 mean amplitude was reliable (r 5 ! .34, r2 5 .12, p 5 .008). Thus, similar to the analogous effect of orthographic neighborhood size, items with more lexical associates elicit more negative N400s. Figure 6 displays this relationship. On second presentation, an equivalent rank sum test performed on a median split of the N4 mean amplitude data sorted by number of lexical associates was reliable (p 5 .02), as was the single regression of number of lexical associates with N4 mean amplitude (r 5 ! .41, r2 5 .17, p 5 .001). On second presentation, as on first presentation, items with more lexical associates elicited a more negative N400.

7

1 2 3 4 5 6 7 8

Frequency of Top Associate Effects A median split of the item N4 mean amplitude data when sorted by frequency of top lexical associate put 30 items in the ‘‘high frequency of top lexical associate’’ category and 30 items in the ‘‘low frequency of top associate’’ category. Although the number of items in the high and low categories was thus balanced in this comparison, for analogy with the analyses of the effects of number of lexical associates, we again used rank sum tests in our factorial analysis of the effects of frequency of top associate. A rank sum test on the effect of frequency of top associate on N400 amplitude on first presentation was reliable (p 5 .03), and, accordingly, so was the single regression correlation of frequency of top associate with N4 amplitude (r 5 ! .27, r2 5 . 07, p 5 .04). Figure 6 depicts the effect of frequency of top associate side by side with the effect of number of lexical associates. On second presentation, the effect of frequency of top associate on N4 mean amplitude was not reliable either in the factorial analysis (rank sum p 5 .38) or the single regression correlation (r 5 ! .13, p 5 .32).

Figure 4. Equivalency of neighbor frequency effect across lexical types: Item N400 mean amplitude (250–450 ms) over the middle parietal channel is plotted against neighbor frequency for lexical items (filled circles) and nonlexical items (empty circles). Single regression trend lines for the relationship between N4 mean amplitude and neighbor frequency are also plotted for each item type. The function relating N400 amplitude to neighbor frequency is nearly identical for the two item types.

was no main effect of either neighbor frequency or lexical type, and no interaction between the two (for neighbor frequency, F 5 1.72, for lexical type F 5 2.87, for the interaction F 5 0.67). Accordingly, the single correlations between N4 mean amplitude and max neighbor frequency were not reliable for either lexical or nonlexical items (for lexical items, r 5 ! 0.08, p 5 .31; for nonlexical items r 5 ! 0.05, p 5 .56). Number of Lexical Associates Sixty-one of our lexical items were included in the South Florida Free Association Norms (Nelson et al., 1998), and a median split of the N4 mean amplitude data when sorted by number of lexical associates put 27 items in the ‘‘high number of lexical associates’’ category and 28 in the ‘‘low number of lexical associates’’ category. Because there were different numbers of items in the two categories, we used a nonparametric rank sum test (equivalent to a Mann-Whitney U test) to examine whether or not there was a factorial effect of number of lexical associates on first presenta-

Multiple Regressions Of particular interest was to use multiple regression to enable examination of the unique effects of each of our variables of interest. We conducted two multiple regression analyses: one pertaining to orthographic variables and one pertaining to lexical variables. Included in the orthographic analysis were N and

Middle Parietal

5 µV

900 ms

High N

High Neighbor Frequency

Mid N

Mid Neighbor Frequency

Low N

Low Neighbor Frequency

Figure 5. Effects of N and neighbor frequency: Left, grand average ERPs elicited in response to items with high, mid, or low orthographic neighborhood size (N). Right, grand average ERPs elicited in response to items with high, mid, or low neighbor frequency. All ERPs are from the middle parietal channel, and are averaged over both lexical and nonlexical items. In part because the two variables are highly inter-correlated, the effects are quite similar.


183

Number of Lexical Associates 5

10

15

20

Log Frequency of Top Associate 25

1 2 3 4

r2 = .12

5

0 N400 Mean Amplitude (µV)


0

1

2

3

4

1 2 3 4

r2 = .07

5

Figure 6. Effects of number and frequency of lexical associates: Left, a scatter plot showing the relationship of N400 mean amplitude (250–450 ms) and number of lexical associates at the single item level. Right, an identical scatter plot showing the relationship of N400 mean amplitude and log frequency of top lexical associate. Items with more lexical associates and items with more frequent lexical associates both elicit more negative N400s.

neighbor frequency (which are strongly correlated in this dataset: r 5 .64, po.001). Included in the lexical analysis were number of lexical associates and frequency of top associate (which are more weakly correlated in this datasest: r 5 .19, p 5 .14). Because the effects of N and neighbor frequency are so similar across the lexical and nonlexical item types, we collapsed across lexicality in the analysis of orthographic factors. (In addition to the lack of interaction in the N4 window, we also observed no differences between these item types in the immediately preceding P2 (175–225 ms, middle prefrontal channel) window (t(298) 5 .09, p 5 .93).) Automated stepwise multiple regression revealed that the most reliable predictor of N4 amplitude was N, followed by neighbor frequency. Alone, N explained 30.6% of variance in N400 mean amplitude (F(298) 5 131.58, po.0001). With the variance from N already explained, the stepwise procedure did add neighbor frequency to the model, which explained an additional 1.2% of variance (F(297) 5 69.37, po.0001). Thus, when combined, these two factors explain 31.8% of variance in N400 mean amplitude. Neighbor frequency was added to the model after N even when length was additionally added to the pool of lexical variables available to the stepwise procedureFa supplemental analysis we conducted because length and N are strongly correlated in this dataset (r 5 ! .27, po.0001). Additionally, a simultaneous multiple regression including length, N, and neighbor frequency was highly reliable (F(296) 5 56.73, po.0001). Because both N and neighbor frequency also influenced N400 amplitudes for repeated items, we performed the same regression analysis on data from the second presentation of each item, again collapsed across lexicality. Automated stepwise multiple regression revealed that the most reliable predictor of N4 amplitude was again N, followed by neighbor frequency. Alone, N explained 14.8% of variance in N400 mean amplitude (F(298) 5 51.81, po.0001). With the variance from N already explained, the stepwise procedure again added neighbor frequency to the model, which explained an additional 2.5% of variance (F(297) 5 31.10, po.0001). Thus, when combined, these two factors explain 17.3% of variance in N400 mean amplitude to items that have been repeated. An identical automated stepwise multiple regression conducted over lexical variables (number of lexical associates and frequency of top lexical associate for all 61 items for which this information was available) revealed that number of lexical associates was a better predictor of N4 amplitude than was frequency of top associate. Alone, number of lexical associates

explained 11.5% of variance in N4 mean amplitude (F(59) 5 7.65, p 5 .008). With the variance due to number of lexical associates already explained, adding in frequency of top associate explained an additional 4.1% of variance (F(58) 5 5.37, p 5 .007). Thus, when combined, these two factors explained 15.6% of variance in N400 mean amplitude. Number of lexical associates was strongly correlated with written frequency in this dataset (r 5 .35, p 5 .006), but an additional automated stepwise procedure conducted with number of lexical associates, frequency of top associate, and written frequency as predictor variables added number of lexical associates to the model after variance due to frequency was explained. In this case, neighbor frequency was added only if a less conservative entry criterion was used (p o.10). Thus, the independent contributions of item frequency and neighbor frequency are more difficult to disentangle. The simultaneous regression with all three variables was also reliable (F(57) 5 3.60, p 5 .02). Discussion Our goal was to discover whether properties of items related to an input itemFeither orthographically or lexicallyFwould have any effect on the magnitude of the N400 ERP component elicited by that input. We were motivated to this goal by recent evidence suggesting that the range of information activated by a particular input may be considerably broader than is assumed in classical, staged models of readingFespecially at the semantic level of representation. Thus, we looked for N400 effects of orthographic neighborhood size and neighbor frequency (either summed or maximum) and what we thought of as their lexical correlates, namely number of lexical associates and frequency of top associate. We found effects of all four factors, consistent with the hypothesis that orthographic inputs activate not only directly associated semantic information but also information associated with items related to the input on at least two levels of representation (orthographic and lexical), and that this information is accessed in a cascaded, not staged, fashion. We replicated and extended previous findings showing a relationship between orthographic neighborhood size and N4 amplitude (Holcomb et al., 2002; Laszlo & Federmeier, 2007, 2008, 2009) with high N items eliciting larger N400s than low N items. Our use of regression analysis on individual item ERPs showed clearly that this is a graded effect, and a strong one, with just over 30% of unique variance in N400 mean amplitude explained by N

184 alone. Furthermore, this relationship was statistically indistinguishable for lexically represented items (words, acronyms) and items without lexical representation (pseudowords, illegal strings of letters)Falthough, of course, non-lexical items are discriminated from lexical items in portions of the ERP subsequent to the N400 window. The finding that number of orthographic neighbors strongly affects N400 amplitude already hints that semantic information associated with orthographically similar items becomes active in parallel with that for a given input. However, as we described in the introduction, N alone could potentially be thought of as a proxy for some property of lexically represented items such as how likely they are to be pronounced similarly to other wordsFalthough it is then difficult to explain the identical N effects we observed for non-lexical items. Nevertheless, the present data revealed an even stronger finding in support of the hypothesis that items orthographically related to an input affect that input’s semantic processing. In particular, we found that items with orthographic neighbors that are high in lexical frequency tend to elicit N400s with larger amplitude than items with neighbors that are low in frequency. Multiple regression analysis revealed that, even though N and neighbor frequency are strongly correlated, neighbor frequency explains an additional, unique portion of variance in N400 amplitude. To our knowledge, this is the first time that neighbor frequency has been shown to affect the N400 independent of N (c.f. Debruille, 1998). Although lexical frequency effects on the N400 have often been in the form of amplitude reductions (more positivity) to high as opposed to low frequency words, the effect of neighbor frequency we observe here is different, with more negative responses when neighbors of input items have high lexical frequency. However, this pattern may reflect a similar underlying mechanism. Traditional N400 frequency effects are often interpreted as reflecting the ‘‘ease’’ with which an item becomes active, with higher frequency words being easier to activate than lower frequency words. This higher ease of activation may reflect a greater tendency for the neighbor to become active when an item containing some of its orthographic features is encountered. In other words, the neighbor item is a better ‘‘lure’’ when it is of higher frequency. This explanation seems consistent with the finding in the behavioral literature that the lexical decision task takes longer for items with high frequency neighbors than with low frequency neighbors (e.g., Grainger, 1990), which has been interpreted as representing ‘‘interference’’ by the high frequency neighbors (Grainger, 1990). In the data described here, because higher frequency neighbors are more likely to become active in response to a given input, the net amount of semantic information evoked by that input is greater, resulting in larger N400 amplitude. We also found corresponding effects from items lexically associated with an input, which we believe to be novel to the N400 literature. In particular, N400 amplitudes were larger for words with higher numbers of lexical associates, suggesting again that inputs evoke semantic activity associated with a set of items that are similar or interconnected at lower processing levels. Some items elicit a greater spread of activation at the lexical level and, in turn, a greater net level of initial activity in the semantic system. Analogous to the pattern seen for orthographic neighbors, we also found that N400 amplitudes are larger for items whose top associate is higher in lexical frequency. We were not able to statistically disentangle effects that might be due to frequency of a word’s top associate from effects that might arise from the

S. Laszlo & K. D. Federmeier frequency of the word itself; however, it is worth noting the direction of the effect, if due to word frequency rather than frequency of the top associate, goes in the opposite direction from that typically observed, as in this case more frequent words (with more frequent top associates) elicited a larger (rather than a smaller) N400. The effects of lexical association were smaller than the effects of orthographic similarity, perhaps reflecting their second-order nature. That is, whereas it is reasonable to assume that orthographic neighbors are activated directly by the input (i.e., the presence of ‘‘ORK’’ in the input FORK directly causes some co-activation of PORK), the activation of lexical associates must be mediated, such that the activation of, for example, SPOON is dependent on the activity associated with FORK. The distinction between measures that reflect properties of a subset of the networkFsuch as N and number of lexical associatesFand measures that instead represent the properties of single itemsFsuch as neighbor frequency and frequency of top associateFis critical to explaining the different impact of repetition on neighbor or associate effects, as compared with neighbor or associate frequency effects. Effects of both N and number of lexical associates were maintained across repetitions. However, effects of neighbor frequency were only maintained when a measure of lexical frequency summed across all an item’s neighbors was used. When max neighbor frequencyFa measure more similar to the frequency of a single, top associateFwas used, an effect of neighbor frequency was no longer observable on second presentation. Because both N and number of lexical associates are properties of the structure of the comprehension network, it makes sense that these factors would have an impact every time an input is encountered (and, indeed, N effects have been shown to persist even for the final words of highly constraining sentences; Laszlo & Federmeier, 2009)F a single presentation of an item does not affect the entire system it is embedded in in a persistent way. In contrast, effects that arise due to baseline activity of particular itemsFfor example, frequency effectsFcan be over-ridden by the processing context (e.g., Van Petten & Kutas, 1990). Thus, it makes sense that the effect of frequency of an orthographic or lexical associate is reduced (in fact, statistically eliminated) with repetition, as first order lexical frequency effects of input items (i.e., not even second order effects of associates of items) have been found to be similarly context sensitive (Van Petten & Kutas, 1990). Taken all together, the findings that semantic processing, as indexed by the N400, is modulated by the number of items that share orthographic features with an input and the number that are lexically associated with that input, as well as by lexical properties (such as frequency) of those orthographically or lexically related items strongly suggest that semantic access does not serially follow a recognition process in which the input has been mapped onto a single, stored representation. In models of that type (e.g., Borowsky & Besner, 1993; Forster & Davis, 1984), semantic processing should be limited to lexically represented items, and only semantic features directly associated with a recognized input should become active. That is to say, no semantic processing should be observed for nonlexical items like our pseudowords and illegal strings. Instead, it seems that activity is elicited in the semantic system for both lexical and nonlexical inputs, and that this activity is cascaded from lower-level (orthographic, lexical) processes, such that semantic features associated with a range of similar (or associated) inputs become active in parallel, beginning around 250 ms post-stimulus onset.

N400 neighbor and associate affects Although serial models do not predict this pattern, it is entirely consistent with cascaded models, which do not require lexical access to be complete (or ultimately successful) in order for semantic processing to begin. In arguing against models where semantic access is gated by recognition, these data are also inconsistent with views of the N400 that derive from such models, especially those that map the N400 onto some aspect of ‘‘post-lexical’’ processing (e.g., Brown & Hagoort, 1993; Sereno, Rayner, & Posner, 1998). For example, Hagoort, Baggio, and Willems (2009) have linked the N400 with post-recognition processes that integrate the (already accessed) meaning of the current word with sentence- and discourse-level representations. It seems difficult, under this kind of view, to explain how items with no lexical representationFsuch as pseudowords and orthographically illegal stringsFcan show identical, graded N400 effects to those shown by lexically represented items. Furthermore, the N400’s sensitivity to number of neighbors and associates and to properties of those items is inconsistent with the assumption that the meaning information associated with a given input has already been accessed by the time the N400 is measured. Instead, the present dataFin the context of the full set of variables known to modulate the N400 (for a review, see, e.g., Kutas & Federmeier, 2000)Fare more consistent with views that link the N400 to early aspects of semantic access (e.g., Federmeier & Laszlo, 2009; Kutas & Federmeier, 2000; Lau, Phillips, & Poeppel, 2008; Van Berkum, 2009), on the assumption that semantic access takes place in a cascaded processing stream and is distributed over time. Under such views, while linguistic effects can still be observed in the ERP prior to the N400 epochFfor example discrimination between pseudohomophones and orthographically matched controls (Grainger, Kiyonaga, & Holcomb, 2006) or discrimination between words and nonwords in the lexical decision task (Kiyonaga, Midgley, Holcomb, & Grainger, 2007)Fthey are interpreted not as evidence for early lexical access, but instead as representing complex perceptual or formal analysis, with the N400 still representing the first point in time at which amodal, position invariant representations come into contact with semantics (e.g., Grainger & Holcomb, 2009). For example, in the bi-modal interactive activation model (BIAM), which instantiates the principles of interactivity proposed in Rumelhart & McClelland’s Interactive Activation model (McClelland & Rumelhart, 1981; Rumelhart & McClelland, 1982), visual word recognition proceeds first through visual feature analysis around 100 ms post stimulus onset, then subsequently through position dependent and position invariant orthographic analysis at approximately 200 ms and 250 ms respectively. The position invariant orthographic analysis outputs representations akin to visual wordforms around 300 ms, and processing of these visual wordforms is only advanced enough to begin contacting semantics around 400 msFthat is, the time of the N400 (Grainger & Holcomb, 2009, Massol, Grainger, Dufau, & Holcomb, 2010). In addition to being more compatible with the present data than post-lexical views, models like the BIAM are consistent with what is known about the neural generators of early ERP components elicited during word reading (e.g., Marincovic, Dhond, Dale, Glessner, Carr, & Halgren, 2003; Tse, Lee, Sullivan, Garnsey, Dell, et al., 2007)Fan important constraint on cognitive models of any kind. However, even if post-lexical theories of the N400 are correct, and lexical access does take place prior to 400 ms (despite being opaque to ERPs, magnetoencephalography, and the event-re-

185 lated optical signal; although see Shtyrov, Kujala, and Pulvermuller (2010), for counterarguments to this claim), the present data clearly indicate that, during the N400 time window, the system is in a state wherein lexical and nonlexical items are treated identically, as indicated by the indistinguishable effects of N and neighbor frequency we observed for words and nonwords, and wherein activity reflects the structure of the input network, not just the properties of the input item itself. To our knowledge, no implemented or proposed model of serial word recognition would be expected to show such effect patterns in a post-recognition time window. Instead, we have previously suggested (Federmeier & Laszlo, 2009) that the basic temporal properties of the N400 inherently argue against the idea that the processing it indexes is dependent on a discrete recognition process, since recognition, both theoretically and empirically, would seem to take varying amounts of time for different types of stimuli and in different types of contexts, whereas the N400 manifests striking temporal stability. If, then, semantic access is not dependent on recognition, it follows that all types of stimuli might elicit N400 activity to some degreeFas was observed here. Furthermore, the present data suggest that activity in the N400 time window can reflect initial semantic activation states, that is, those which emerge before activity in orthographic levels of processing has reached a stable point. Thus, although the comprehension system will eventually reach a state in which only the orthographic features comprising F-O-R-K, and the corresponding semantic features of FORK are strongly activated (or in which the system has determined, for example, that there is no stable semantic representation associated with the input GORK), activity in the N400 time window is sensitive to points in processing earlier than this, when semantic information associated with a distributed set of co-activated representations comes online in parallel. Within the PDP modeling framework, this same point might be stated as suggesting that the N400 represents activity taking place in the semantic level of representation before either the orthographic or semantic layers have settled. The N400 might thus be well described as providing a temporally delimited ‘‘snapshot’’ of activity elicited by a given input in a distributed, cascaded, semantic system.

Conclusion Using a regression approach to examine effects of correlated variables on ERP responses to single items, we observed strong, independent effects of orthographic neighborhood size, neighbor frequency, number of lexical associates, and frequency of top associate on the amplitude of the N400 componentFthe latter three, to our knowledge, for the first time in the literature. This pattern supports a view of the N400 as indexing fairly early aspects of distributed semantic activation arising in a cascaded processing system. In turn, these data, in combination with the larger literature, are consistent with parallel distributed processing models of language comprehension, which are characterized by interactive dynamics and recurrent architecture. Such models can typically never be said to be doing only ‘‘semantic’’ processing or ‘‘orthographic’’ processing, as activation flows through all levels of representation in a parallel fashion. Thus, a snapshot of the model at any particular moment in time reflects activity in all levels of representationFmuch as, as suggested by the current data, the N400 represents a snapshot of late orthographic and early semantic processing occurring in parallel.

186

S. Laszlo & K. D. Federmeier REFERENCES

Besner, D., & Chapnik Smith, M. (1992). Models of visual word recognition: When obscuring the stimulus yields a clearer view. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18, 468–482. Borowsky, R., & Besner, D. (1993). Visual word recognition: A multistage activation model. Journal of Experimental Psychology: Learning, Memory, and Cognition, 19, 813–840. Brown, C., & Hagoort, P. (1993). The processing nature of the N400: Evidence from masked priming. Journal of Cognitive Neuroscience, 5, 34–44. Brunelli, R., & Poggio, T. (1993). Face recognition: Features versus templates. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15, 1042–1052. Dale, A. M. (1994). Source localization and spatial discriminant analysis of event-related potentials: Linear approaches. Unpublished doctoral dissertation. La Jolla, CA: University of California, San Diego. Dambacher, M., Kliegl, R., Hofman, M., & Jacobs, A. M. (2006). Frequency and predictability effects on event-related potentials during reading. Brain Research, 1084, 89–103. Deacon, D., Dynowska, A., Ritter, W., & Grose-Fifer, J. (2004). Repetition and semantic priming of nonwords: Implications for theories of N400 and word recognition. Psychophysiology, 41, 60–74. Debruille, J. B. (1998). Knowledge inhibition and N400: A study with words that look like common words. Brain and Language, 62, 202–220. Federmeier, K. D., & Laszlo, S. (2009). Time for meaning: Electrophysiology provides insights into the dynamics of representation and processing in semantic memory. In B. Ross (Ed.), Psychology of Learning and Memory, 51, 1–44. Forster, K. I. (1999). The microgenesis of priming effects in lexical access. Brain and Language, 68, 5–15. Forster, K. I., & Davis, C. (1984). Repetition priming and frequency attenuation in lexical access. Journal of Experimental Psychology: Learning, Memory, and Cognition, 10, 680–698. Forster, K. I., & Veres, C. (1998). The prime lexicality effect: Formpriming as a function of prime awareness, lexical status, and discrimination difficulty. Journal of Experimental Psychology: Learning, Memory, and Cognition, 24, 498–514. Grainger, J. (1990). Word frequency and neighborhood frequency effects in lexical decision and naming. Journal of Memory and Language, 29, 228–244. Grainger, J., & Holcomb, P. J. (2009). Watching the word go by: On the time-course of component processes in visual word recognition. Language and Linguistics Compass, 3, 128–156. Grainger, J., Kinyonaga, K., & Holcomb, P. J. (2006). The time-course of orthographic and phonological code activation. Psychological Science, 17, 1021–1026. Hagoort, P., Baggio, G., & Willems, R. M. (2009). Semantic unification. In M. Gazzaniga (Ed.), The Cognitive Neurosciences (4th edition, pp. 819–836). Boston: MIT Press. Harm, M. W., & Seidenberg, M. S. (2004). Computing the meanings of words in reading: Cooperative division of labor between visual and phonological processes. Psychological Review, 111, 662–720. Hauk, O., Davis, M. H., Ford, M., Pulvermuller, F., & Marslen-Wilson, W. D. (2006). The time course of visual word recognition as revealed by linear regression analysis of ERP data. NeuroImage, 30, 1313–1400. Hauk, O., Pulvermuller, F., Ford, M., Marslen-Wilson, W. D., & Davis, M. H. (2009). Can I have a quick word? Early electrophysiological manifestations of psycholinguistic processes revealed by event-related regression analysis of the EEG. Biological Psychology, 80, 64–74. Holcomb, P. J., Grainger, J., & O’Rourke, T. (2002). An electrophysiological study of the effects of orthographic neighborhood size on printed word perception. Journal of Cognitive Neuroscience, 14, 938–950. Kello, C. T., Plaut, D. C., & MacWhinney, B. (2000). The task-dependence of staged versus cascaded processing: An empirical and computational study of Stroop interference on speech production. Journal of Experimental Psychology: General, 129, 340–360. King, J.W, & Kutas, M. (1998). Neural plasticity in the dynamics of human visual word recognition. Neuroscience Letters, 244, 61–64. Kiyonaga, K., Midgley, K. J., Holcomb, P. J., & Grainger, J. (2007). Masked cross-modal repetition priming: An ERP investigation. Language and Cognitive Processes, 22, 337–376. Kutas, M., & Federmeier, K. D. (2000). Electrophysiology reveals semantic memory use in language comprehension. Trends in Cognitive Science, 4, 463–470.

Kutas, M., Van Petten, C. K., & Kluender, R. (2007). Psycholinguistics electrified II (1994–2005). In M. A. Gernsbacher & M. Traxler (Eds.), Handbook of Psycholinguistics (2nd edition, pp. 88–143). San Diego, CA: Academic Press. Laszlo, S., & Federmeier, K. D. (2007). Better the DVL you know: Acronyms reveal the contribution of familiarity to single word reading. Psychological Science, 18, 122–126. Laszlo, S., & Federmeier, K. D. (2008). Minding the PS, queues, and PXQs: Uniformity of semantic processing across multiple stimulus types. Psychophysiology, 45, 458–466. Laszlo, S., & Federmeier, K. D. (2009). A beautiful day in the neighborhood: An event-related potential study of lexical relationships in sentence context. Journal of Memory and Language, 61, 326–338. Lau, E. F., Phillips, C., & Poeppel, D. (2008). A cortical network for semantics: (De)constructing the N400. Nature Reviews Neuroscience, 9, 920–933. Marcus, M., Santorini, B., & Marcinkiewicz, M. (1993). Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19, 313–330. Marincovic, K., Dhond, R., Dale, A. M., Glessner, M., Carr, V., & Halgren, E. (2003). Spatiotemporal dynamics of modality-specific and supramodal word processing. Neuron, 38, 487–497. Massol, S., Grainger, J., Dufau, S., & Holcomb, P. (2010). Masked priming from orthographic neighbors: An ERP investivation. Journal of Experimental Psychology: Human Perception and Performance, 36, 162–174. McClelland, J. L., & Rumelhart, D. E. (1981). An interactive activation model of context effects in letter perception: Part 1. An account of basic findings. Psychological Review, 88, 375–407. Medler, D. A., & Binder, J. R. (2005). MCWord: An on-line orthographic database of the English language. Retrieved from: http:// www.neuro.mcw.edu/mcword/ Nelson, D. L., McEvoy, C. L., & Schreiber, T. A. (1998). The University of South Florida word association, rhyme, and word fragment norms. Retrieved from: http://www.usf.edu/FreeAssociation/ Plourde, C. E., & Besner, D. (1997). On the locus of the word frequency effect in visual word recognition. Canadian Journal of Experimental Psychology, 51, 181–194. Poggio, T., & Edelman, S. (1990). A network that learns to recognize three-dimensional objects. Nature, 343, 263–266. Rey, A., Dufau, S., Massol, S., & Grainger, J. (2009). Testing computational models of letter perception with item-level event-related potentials. Cognitive Neuropsychology, 26, 7–22. Rugg, M. D., & Nagy, M. E. (1987). Lexical contribution to nonwordrepetition effects: Evidence from event-related potentials. Memory & Cognition, 15, 473–481. Rumelhart, D. D., & McClelland, J. L. (1982). An interactive activation model of context effects in letter perception: Part 2. The contextual enhancement effect and some tests and extensions of the model. Psychological Review, 89, 60–94. Sereno, S. C., Rayner, K., & Posner, M. I. (1998). Establishing a timeline of word recognition: Evidence from eye movements and eventrelated potentials. NeuroReport, 9, 2195–2200. Shtyrov, Y., Kujala, T., & Pulvermuller, F. (2010). Interactions between language and attention systems: Early automatic lexical processing? Journal of Cognitive Neuroscience, 22, 1465–1478. Stolz, J. A., & Neely, J. H. (1995). When target degradation does and does not enhance semantic context effects in word recognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21, 596–611. Tse, C. Y., Lee, C. L., Sullivan, J., Garnsey, S. M., Dell, G. S., Fabiani, M., & Gratton, G. (2007). Imaging cortical dynamics of language processing with the event-related optical signal. Proceedings of the National Academy of Sciences of the United States of America, 104, 17157–17161. Van Berkum, J. J. A. (2009). Does the N400 directly reflect compositional sense-making? Psychophysiology, 46(Supplement 2), s2. Van Petten, C., & Kutas, M. (1990). Interactions between sentence context and word frequency in event-related brain potentials. Memory & Cognition, 18, 380–393. (Received January 20, 2010; Accepted April 3, 2010)


The N400 as a snapshot of interactive processing: Evidence from regression analyses of orthographic neighbor and lexical associate effects

SARAH LASZLOa and KARA D. FEDERMEIERb,c,d a

Department of Psychology, Carnegie Mellon University, Pittsburgh, Pennsylvania Department of Psychology, University of Illinois at Urbana Champaign, Urbana, Illinois c Program in Neuroscience, University of Illinois at Urbana Champaign, Urbana, Illinois d Beckman Institute for Advanced Science and Technology, University of Illinois at Urbana Champaign, Urbana, Illinois b

Abstract Linking print with meaning tends to be divided into subprocesses, such as recognition of an input’s lexical entry and subsequent access of semantics. However, recent results suggest that the set of semantic features activated by an input is broader than implied by a view wherein access serially follows recognition. EEG was collected from participants who viewed items varying in number and frequency of both orthographic neighbors and lexical associates. Regression analysis of single item ERPs replicated past findings, showing that N400 amplitudes are greater for items with more neighbors, and further revealed that N400 amplitudes increase for items with more lexical associates and with higher frequency neighbors or associates. Together, the data suggest that in the N400 time window semantic features of items broadly related to inputs are active, consistent with models in which semantic access takes place in parallel with stimulus recognition. Descriptors: N400, Semantic access, Multiple regression, Orthographic neighborhood

a template- or feature-matching process to be completed before semantic information can be retrieved have also been proposed for face and object recognition (e.g., for faces, Brunelli & Poggio, 1993; for objects, Poggio & Edelman, 1990). Staged models of word processing that involve an isolated recognition process successfully explain a range of behavioral findings, notably the complex, interacting effects of frequency, context, and stimulus quality on lexical decision reaction times (e.g., Borowsky & Besner, 1993; Stolz & Neely, 1995). However, such models make two specific predictions about the semantic processing that follows recognition that seem incongruent with data from the event-related potential (ERP) literature pertaining to the N400 component, a well-established, functionally specific marker of lexico-semantic processing (for review, see Kutas, Van Petten, & Kluender, 2007). First, if semantic access can only proceed after recognition has been successfully completed, then access should never be attempted for items without lexical representations, such as pseudowords or orthographically illegal consonant strings. Second, if semantic access is essentially limited to the process of looking up meaning information associated with a recognized lexical entry, then the largely inconsistent semantics of other orthographically or lexically associated items should never become simultaneously active. That is, for the input FORK, the semantics of the orthographically similar PORK and the lexically associated SPOON should never become active. If the first prediction is true, and semantic access will only be attempted for items corresponding to a known lexical entry, then

A concept that has classically been important to psycholinguistic theories of visual word processing is recognition, a process by which orthographic inputs are compared to internal representationsFoften items in the mental lexiconFin order to find a match to the input that can subsequently be linked with semantics. This type of staged recognition process is exemplified in Forster and colleagues’ Entry Opening Model (e.g., Forster, 1999; Forster & Davis, 1984; Forster & Veres, 1998), wherein information corresponding to an orthographic input cannot be retrieved until a matching lexical entry has been identified. A similar formulation is found in the Multistage Activation Model of Besner and colleagues (e.g., Besner & Chapnik Smith, 1992; Borowsky & Besner, 1993; Plourde & Besner, 1997), in which orthographic analysis of an input must be complete (that is, the input must be recognized) before associated information can be passed forward for subsequent processing. Theories that require The authors wish to acknowledge B. Armstrong, B. Gonsalves, C. Lee, K. Mathewson, G. Miller, D. Plaut, and E. Wlotko for insightful discussion of the single item data set, as well numerous research assistants for their efforts in data collection and processingFespecially P. Anaya, H. Buller, and C. Laguna. This research was supported by National Institute of Mental Health Training grant T32 MH019983 to Carnegie Mellon University, which supported SL, and NIA grant AG26308 to KDF. Address correspondence to: Sarah Laszlo, Carnegie Mellon University, 5000 Forbes Ave, Baker Hall 254T, Pittsburgh, PA 15213. E-mail: [email protected] 176

N400 neighbor and associate affects the N400Fwhich has been established as a marker of attempted or successful semantic access (see, for example, Federmeier & Laszlo, 2009; Kutas & Federmeier, 2000)Fshould only be observed in response to items with lexical representations. However, this is not the case. Clear N400 components and N400 effects (such as reductions in amplitude with repetition) have long been observed in response to pronounceable pseudowords, such as GORK (e.g., Rugg & Nagy, 1987; Deacon, Dynowska, & Grose-Fifer, 2004; Laszlo & Federmeier, 2007). This finding necessitates weakening the proposal that semantic access occurs only for lexically represented items to, at minimum, allow for attempts at semantic access for strings that are very similar to lexically represented items (i.e., pseudowords, which are often created by changing one letter of a real word). However, even this weakened proposal is incompatible with recent work from our lab, which has shown that, at least in a supportive sentence context, even meaningless, illegal letter strings (e.g., NHK) with minimum orthographic neighborhood size (i.e., Coltheart’s N: the number of words that can be formed from a target by changing one of its letters) can elicit clear N400 components and N400 effects (Laszlo & Federmeier, 2008, 2009). We have argued that this pattern indicates that semantic access is attempted for all orthographic inputs, regardless of their lexical status, although the success of that attempted access can vary with contextFexplaining why, for example, unfamiliar, orthographically illegal strings embedded in word lists have been found to not show N400 repetition effects (Rugg & Nagy, 1987), whereas, when embedded in sentences, these same types of strings do elicit N400 effects associated with similarity to a predicted completion (Laszlo & Federmeier, 2009). Although seemingly incompatible with staged recognition models, the ERP findings are compatible with Parallel Distributed Processing (PDP) models of reading that result in some semantic features becoming at least initially active in response to all inputs (e.g., Harm & Seidenberg, 2004). In models of this type, processing that appears staged can result from nonlinear activation dynamics between orthography and semantics (Kello, Plaut, & MacWhinney, 2000). Importantly, however, even when such models exhibit stage-like behavior, this is accomplished without any formal implementation of stages and also without any formal distinction between the processing of lexically represented and unrepresented items (a distinction that is a necessary consequence of strongly staged models). Thus, data that have often been explained with staged models can also be explained with cascaded models, which, additionally, are consistent with ERP findings showing that non-lexical items engage attempts at semantic access that seem identical in timing and in nature to those engaged by lexically represented items. The second prediction of staged models of reading outlined above is also not shared by PDP models. Because staged models assume that items are identified before semantic access begins, there is no reason to predict that semantic features of orthographically similar or lexically associated items should become active (to any significant degree) along with the features of the input stimulus. For example, having recognized an input as FORK, the system would not access semantics associated with the orthographically similar input PORK. In contrast, given the tendency of PDP models to activate similar outputs in response to similar inputs, semantic features associated with a range of items similar to the input stimulus at the orthographic or lexical levels of analysis can become active in parallel with the appropriate semantics for the input, at least transiently. Thus, in a PDP

177 model given the input string ‘‘FORK,’’ both the semantics of FORK and PORK could initially become active, as both are at least partially consistent with the input (i.e., contain ORK), although, of course, the semantics of FORK are more consistent with the input and would eventually become most active. Again, recent ERP data are more in line with the predictions of cascaded models than staged ones. In particular, Holcomb, Grainger, and O’Rourke (2002) found that out-of-context N400 amplitudes were larger to words and pseudowords with higher orthographic neighborhood sizes, and we have replicated that finding and shown that it extends to illegal strings of letters (Laszlo & Federmeier, 2007) and that the amplitude difference is maintained even when items are embedded in sentences (Laszlo & Federmeier, 2008, 2009). We have argued that the larger N400s to items with high N result from there beingFat least initiallyFmore semantic activation for items that are orthographically similar to many other items. That is, a high N input like CAT activates not only its own semantics, but also, briefly, the semantics of all its neighbors, whereas a low N input like OWL results in a less broad activation at the semantic level of representation. Importantly, the fact that effects of N are identical for lexical and non-lexical inputs (Laszlo & Federmeier, 2009) suggests not only that a broader range of semantic features becomes active in response to an input than staged models would predict, but also that lexical status per se is not a determining factor in this semantic level effect. Although effects of neighborhood density on the N400 can be taken to suggest that a range of orthographically similar and lexically associated items become active in response to any given input, it could potentially be argued that N actually reflects a property of the input item itselfFN might instead be a proxy for some information about orthographic regularity that is included in an item’s lexical entry. For example, maybe the number of neighbors an item has could be an abstracted proxy for how similarly to other words that item is pronounced, and thus whether or not it can be pronounced by rule or must be considered an exception. A stronger test of the hypothesis that the N400 reflects the processing of not just an input, but also items similar to that input, could thus potentially come from examining the effect of the frequency of an item’s orthographic neighbors on the magnitude of the N400 elicited by that item. Such an effect, if observed, would indicate directly that properties of items similar to an input affect its semantic processing. In fact, one study has reported such an effect, finding that items with high frequency neighbors elicited more negative N400s than items with lower frequency neighbors (Debruille, 1998). Unfortunately, however, neighbor frequency was confounded with orthographic neighborhood size in that study, making it difficult to strongly conclude that it was the frequency of an item’s neighbors, and not just the number of neighbors, that affected N400 amplitude. Therefore, the first goal of the present study was to determine whether neighbor frequency has an effect on the N400 independent of the effect of N. Further, while neighbor frequency is a property of items orthographically similar to an input, our second goal was to determine whether the properties of items lexically associated to an input might also affect its processing. Specifically, we were interested in examining the effects, if any, of the number of lexical associates and written frequency of the top associate on N400 amplitudes, as these might be considered lexical level analogues of N and neighbor frequency. For example, if FORK can activate some of the semantics of PORK by virtue of their shared

178 orthography, can DOG also activate some of the semantics of BONE by virtue of their lexical association? The cascaded nature of the information flow between representational levels in the modeling framework that has thus far been most consistent with N400 effect patterns would seem to predict such effectsF through spreading activation at the lexico-semantic level of representationFbut, to our knowledge, no N400 data addressing this issue exist. Our two experimental goals thus have much the same flavor: each is aimed at trying to determine whether or not the properties of items similar to (or linked to) an input at the orthographic or lexical levelFand thus likely to become active in parallel during input processingFaffect the semantic processing of that input. Evidence for such effects would support cascaded models over staged models of reading, and, in the case of the orthographic variables, this conclusion could be strengthened by an absence of an interaction with lexicality, as staged models predict important differences between lexically represented and non-represented stimuli at processing stages, such as semantic access, that are assumed to follow recognition. We took a somewhat novel approach to these goals. The typical design of an ERP experiment aiming to examine, for example, the effect of neighbor frequency independent of the effect of N might be a factorial one wherein participants view items high and low in neighbor frequency but matched on N. Although this design would provide information about the impact of neighbor frequency on the ERP, it would do so at the expense of not affording information about the simultaneous effect of NFa downside because, of course, these variables apply to all inputs and are never processed in isolation. To address this problem, some studies have begun moving toward the use of designs that enable multiple regression analyses (e.g., Hauk, Davis, Ford, Pulvermuller, & Marslen-Wilson, 2006; Hauk, Pulvermuller, Ford, Marslen-Wilson, & Davis, 2009; King & Kutas, 1998), in order to attempt to untangle the effects of linguistic variables which tend to be highly correlated (e.g., length and word frequency, in the case of King & Kutas, 1998). Multiple regression when applied to items can afford the identification of independent effects of each variable of interest while avoiding the artificiality of attempting to examine the effects of lexical variables in isolation. Multiple regression can be particularly useful for unraveling effects of intercorrelated item variables when combined with items-based analyses (as opposed to subjects-based analyses, which do not permit generalization across items). Despite this advantage, multiple regression has not often been used to examine dependent variables measured over items in ERP studies, because item data with satisfactory signal-to-noise characteristics is not generally available with the numbers of participants typically run in ERP studies (although, for an interesting exception, see Rey, Dufau, Massol, & Grainger, 2009, who extracted item ERPs representing the response to single letters or pseudoletters). In an approach similar to the one we employ in the present study, Dambacher, Kliegl, Hofman, and Jacobs (2006) used simultaneous multiple regression to model the single trial electroencephalogram (EEG) collected from participants reading sentences, and supported cascaded models of word recognition over staged ones; however, high noise levels in the item ERPsFcollected from only 50 participantsFresulted in relatively low R2 values for their multiple regression models. We, therefore, sought to address this issue by collecting a large scale data set from 120 participants who viewed words, pseudowords,

S. Laszlo & K. D. Federmeier acronyms, and illegal strings that intentionally varied widely in their lexical characteristics (including the four presently of interest). With this data, we could form ERPs representing the responses to single items, averaged across participants (e.g., the response to the word DOG only, consisting of 120 trialsFone from each participant.) This data set enables us to generalize over items in a way that is not possible in a typical ERP design where approximately 40 items per condition for approximately 20–30 participants might be collected. Figure 1 displays an unfiltered example from each item type, showing that these single item ERPs were stable, with good signal-to-noise ratios. With stable ERPs available for individual items, it is then possible to obtain item–level mean N400 amplitude measures (or, of course, any other measure that can be obtained from a more typical, item-aggregated ERP). Those single item means are then eligible for regression analyses that are not possible with subject aggregated data. One drawback of this approach is that items analysis does not permit generalization across subjects. However a substantial benefit of this approach is that regression is a more powerful analysis method than analysis of variance; another is that, with multiple regression, the independent effects

Middle Parietal 5 µV

DOG 900 ms

DAWK

DVD

DSN

Figure 1. Example single item ERPs: Each ERP is an average of one EEG sweep over the middle parietal channel from each of 120 participants in response to a single item: the word DOG, the pseudoword DAWK, the acronym DVD, and the illegal string DSN. In this figure, as in all subsequent ones, negative is plotted up. These ERPs are unfiltered, which makes it evident that the signal-to-noise characteristics of the single item ERPs are satisfactory.


179 by a given participant were included in the averaged ERPs computed for that participant. Table 1 displays mean lexical characteristics of each item type (i.e., length, frequency, N, orthographic neighborhood frequency, number of lexical associates, and frequency of top associate), along with examples. Orthographic neighborhood size was computed as the total number of words that could be formed by replacing one letter of a target item, as indicated by the Medical College of Wisconsin Orthographic Wordform Database (Medler & Binder, 2005). Neighbor frequency was, in turn, computed as the logarithm of the summed frequency of all of an item’s orthographic neighbors, with frequency estimates drawn from the Wall Street Journal corpus (Marcus, Santorini, & Marcinkiewicz, 1993). An additional analysis of neighbor frequency considered only the log of the maximum frequency neighbor of each item, as opposed to the sum of the frequencies of all neighbors. Number of lexical associates was retrieved from the South Florida Free Association Norms (Nelson, McEvoy, & Schreiber, 1998), and the log frequency of each item’s top lexical associate was again obtained using the Wall Street Journal corpus (Marcus et al., 1993). Critical experimental items (i.e., words, pseudowords, acronyms, and illegal strings) were each repeated one time at a lag of 0, 2, or 3 intervening items, allowing us to examine the stability of any effects we might observe across presentations. Each level of repetition lag occurred an equal number of times both within and across item types. Participants did not respond to the critical items, in order to prevent contamination of the critical ERPs by response potentials. The proper names served as the targets for the behavioral task, and were only presented once. Participants responded to proper names by pressing a button with their right hand. False alarms (i.e., button presses to critical items) were not included in averaged ERPs. The experiment thus included 750 trials (2 ! 300 critical items 1150 proper names). These 750 trials were broken up into 5 blocks of 150 trials with rest breaks between each block. Across the 120 participants, each of the 120 permutations of 5 block orders was presented exactly once.

of multiple variables can be examined simultaneously (e.g., the effects of N and neighbor frequency). Given past results from factorial studies (Holcomb et al., 2002; Laszlo & Federmeier, 2007, 2008, 2009), we predicted that neighborhood size would be positively correlated with N400 magnitude, independent of the lexical status of the input string. We predicted a similar relationship between number of lexical associates and N400 response to words (the only class of items for which lexical association data is available). Critically, although neighbor frequency and neighborhood size tend to be correlated, we also predicted an independent effect of orthographic neighbor frequency on N400 mean amplitude (and a similar effect of frequency of top associate), indicating that the spread of semantic activation elicited by an input is considerably broader than would be suggested under a staged account.

Methods Participants Data were analyzed from 120 participants (58 female, age range 18–24, mean age 19.1). Data from 6 additional participants were discarded due to either unsatisfactory levels of ocular artifact or EEG digitization equipment malfunction. All participants were right-handed, monolingual speakers of English with normal or corrected-to-normal vision and no history of neurological disease or defect. Participants were graduate or undergraduate students at the University of Illinois. The experimental protocol was approved by the Internal Review Board of the University of Illinois, and all participants were compensated with money or course credit.

Stimuli Stimuli were 75 each words (e.g., HAT, MAP), pseudowords (e.g., DAWK, KAK), meaningless, illegal strings (e.g., CKL, KKB), and familiar, orthographically illegal acronyms (e.g., VCR, AAA). Additionally, 150 common proper American first names (e.g., SARA, JOHN) served as targets in the substantive behavioral task, which was to monitor the stream of unconnected text for names and press a button when a name was detected. All items were between 3 and 5 letters long (mean 3.19). Words, pseudowords, illegal strings, and acronyms were the critical experimental items; no response was made to these items. Illegal strings and acronyms were composed of all consonants or all vowels. Acronym familiarity was assessed by a paper and pencil post-test (identical to that described in Laszlo & Federmeier, 2007), and only EEG responses to acronyms correctly identified

Procedure Participants were seated 100 cm away from a computer monitor and instructed that their task was to press a button whenever they were presented with a ‘‘common English proper first name,’’ and to minimize blinks and eye movements except during a blink interval indicated on the screen by the presence of a white cross. After a demonstration of trial structure, participants were presented with a short block of practice trials consisting of items similar to those in the experiment proper. In both the practice and experimental blocks, a fixation arrow was continuously present in the center of the screen. Participants

Table 1. Selected Lexical Characteristics

Item type Word Pseudoword Acronym Illegal string

Examples

Length

Log written frequency

N

Log neighborhood frequency

Number of lexical associates

Log frequency of top associate

HAT, MAP TUL, KAK VCR, AAA CKL, KKB

3.2 3.2 3.2 3.2

2.39 – 0.96 –

12.99 11.04 1.93 2.4

4.32 4.12 2.71 2.96

10.53 – – –

2.45 – – –

Note: By design, the lexical characteristics of the items included in the single item ERP corpus varied broadly. N was estimated from the Medical College of Wisconsin Orthographic Wordform Database (Medler & Binder, 2005). All frequency estimates were drawn from the Wall Street Journal Corpus (Marcus, Santorini, & Marcinkiewicz, 1993). Number of lexical associates was estimated from the South Florida Free Association Norms (Nelson, McEvoy, & Schreiber, 1998).

180 were instructed to keep their eyes on the fixation arrow as much as possible. Stimuli were presented one at a time in white directly above the fixation arrow on the black background of a 22-inch CRT computer monitor with resolution 640 ! 480. Trial structure was as follows: 500 ms warning stimulus (red cross above the fixation arrow), 500 ms stimulus presentation, 1000 ms response interval (fixation arrow present only), 1000 ms blink interval (white cross above the fixation arrow). After the 5 experimental blocks, participants completed the paper and pencil acronym knowledge questionnaire (described in Laszlo & Federmeier, 2007), in order to permit sorting of the acronym items as familiar or unfamiliar on an individual basis. In brief, the questionnaire required participants to indicate whether each of the acronyms and illegal strings presented in the EEG experiment were acronyms or not acronyms. If participants believed an item was an acronym, they had the option of indicating what the letters in the acronym stood for, writing a sentence showing what the acronym meant, or selecting ‘‘Don’t Know,’’ in instances when they ‘‘had heard other people use it before, but [didn’t] know what the letters in it stand for and couldn’t use it [themselves].’’ Only items for which participants could identify all the letters or could write a sentence were included in subsequent ERP analyses. This method has proved reliable in the past for sorting acronym stimuli into classes distinguished in the ERP signal (Laszlo & Federmeier, 2007, 2008). On average, participants were able to correctly identify 83% of acronyms (" 62/75).

EEG Recording EEG was recorded from 6 Ag/AgCl electrodes embedded in an electrocap. We sampled from middle prefrontal, middle parietal, middle central, left middle central, right middle central, and middle occipital electrode sites. This reduced electrode montage was necessary in order to enable the collection of 120 participants in a reasonable period of time. Because our focus was on the N400 component, we chose a montage that provided good coverage of the region of the scalp where N400 effects are typically maximal (i.e., the central posterior scalp), as well as one prefrontal site to confirm the posterior distribution of observed effects. All EEG electrodes were referenced online to the left mastoid process and then digitally re-referenced offline to the average of the left and right mastoids. The electrooculogram (EOG) was recorded using a bipolar montage of electrodes placed at the outer canthi of the left and right eyes; blinks were monitored with an electrode at the suborbital ridge. EEG and EOG were recorded with a bandpass of 0.02 to 100 Hz and sampled at a rate of 250 Hz with a gain of 10,000!. All electrode impedances were kept below 2 kO. Single item ERPs were computed by averaging (across the 120 subjects) at each electrode time-locked to the onset of each of the critical items (resulting in 600 single item ERPs: one for each of two presentations of each of 300 critical items). In addition to the single item ERPs, more traditional ERPs representing the average within-subject response to, for example, all words, were also computed. Trials containing eye movement or drift artifact were rejected with a threshold individualized to each participant by inspection of that participant’s raw waveforms, and blinks were corrected using a procedure described by Dale (1994). Artifact rejection resulted in an average loss of 7% of trials per participant. All ERPs contained a 100 ms pre-stimulus baseline and continued for 920 ms after stimulus

S. Laszlo & K. D. Federmeier onset. Measurement of ERP mean amplitude was conducted on data digitally filtered off-line with a bandpass of 0.2 to 20 Hz. Results Behavioral Data Correct behavioral responses were either to press a button in the right hand in response to a name, or to press nothing in response to any other item type. Thus a hit was a button press for a name, and a correct rejection was no button press for a critical item. Participants made on average 137/150 hits (s 5 10.2), or 91% accuracy, for the names, and on average 589/600 (s 5 16.5) correct rejections, or 98% accuracy, to critical items. Overall, these results indicate that participants were appropriately attending to the substantive behavioral task, and, more importantly, that they were processing each item in the text stream. Electrophysiological Data Three types of analysis are reported: 1) factorial analyses including item Analyses of Variance (ANOVAs) and, where appropriate, non-parametric factorial tests, 2) single regressions over items, and 3) multiple regressions over items. In what follows, we first present factorial analyses and single regressions pertaining to each of the four single lexical factors of interest (i.e., N, neighbor frequency, number of lexical associates, and frequency of top associate). We then present multiple regressions pertaining to combinations of those variables. For all analyses, the N400 was measured as mean amplitude in a 250–450 ms post stimulus onset window, relative to a 100 ms pre-stimulus baseline. The N400 was measured over the middle parietal channel only. The reduced electrode montage made analyses including data from each of the 6 electrode channels relatively uninformative; all reported effects were qualitatively similar across all five central-posterior channels. Orthographic Neighborhood Size We began with a 2 ! 2 item ANOVA with factors of orthographic neighborhood size (high or low) and lexical type (lexical: word and acronym, or nonlexical: pseudoword and illegal string). This ANOVA revealed a main effect of N (F(1, 296) 5 159.7, po.0001), but no effect of lexical type (F 5 .19) and no interaction (F 5 1.1). Indeed, as is depicted in Figure 2, the relationship between N and N400 amplitude is nearly identical for the two lexical types. The single regression correlations of N on N4 mean amplitude for lexical and nonlexical items are r 5 # .61 (r2 5 .37, po.0001) and r 5 # .49 (r2 5 .24, po.0001), respectively. The equivalence of the N effect for lexical and nonlexical itemsFand the strong effect of N on N400 amplitudeFis reiterated in Figure 3, which shows item ERPs for a low, mid, and high N item from each lexical category separately. Because the N effect is so similar across lexical category, in what follows we will sometimes collapse across lexical category when considering N effects (for example, when collapsed across lexical category, the single correlation of N with N400 mean amplitude has r 5 # .55, r2 5 .30, po.0001). The same pattern of N effect was also observed on second presentation. An identical item ANOVA with factors of N (high or low) and lexical type (lexical or nonlexical) revealed a main effect of N (F(1,296) 5 39.6, po.0001), but no main effect of lexical type (F 5 3.23) and no interaction between the two (F 5 .13). The single regression correlations of N with N4 amplitude were mildly reduced but still highly reliable on second presentation. For lexical items, r 5 # .43 (r2 5 .19, po.0001),

N400 neighbor and associate affects –1 0

181

Orthographic Neighborhood Size 5

10

15

20

25


1 2 3 4 5 6

Lexical Items (r2 = .37) Nonlexical Items (r2 = .24)

7 8

Figure 2. Equivalency of N effect across lexical types: Item N400 mean amplitude (250–450 ms) over the middle parietal channel is plotted against orthographic neighborhood for lexical items (filled circles) and nonlexical items (empty circles). Single regression trend lines for the relationship between N4 mean amplitude and N are also plotted for each item type. The function relating N400 amplitude to N is nearly identical for the two item types.

and for nonlexical items r 5 ! .33 (r2 5 .11, po.0001). Thus, across both first and second presentation, items with high N elicited more negative N400s than did items with low N, regardless of lexical type. Neighbor Frequency Our analysis of neighbor frequency effects mirrored our analysis of N effects. Again, we began with an item ANOVA with factors of (summed) neighbor frequency (high or low) and lexical type (lexical or nonlexical), which revealed a main effect of neighbor frequency (F(1,296) 5 53.0, po.0001), but no main effect of lexical type (F 5 .15) and no interaction (F 5 .81). The single re-

gression correlations of summed neighbor frequency with N4 amplitude were also both strongly reliable (for lexical items, r 5 ! .48, r2 5 .23, po.0001; for nonlexical items, r 5 ! .39, r2 5 .15, po.0001). As was the case with the effect of N, the effect of orthographic neighbor frequency was nearly identical across lexical types. The strikingly similar pattern is displayed in Figure 4. An identical ANOVA conducted with a neighbor frequency measure consisting of the frequency of each item’s highest frequency neighbor (as opposed to the summed frequency of all its neighbors) yielded the same pattern of results, with a main effect of maximum neighbor frequency (F(1,296) 5 21.66, po.0001) but no main effect of lexical type (F 5 .13) and no interaction between the two (Fo0.1). Similarly, the single regressions of maximum neighbor frequency with N4 amplitude were reliable for both lexical types (for lexical items r 5 ! .28, r2 5 .08, po.001; for nonlexical items r 5 ! .25, r2 5 .06, p 5 .002). Figure 5 displays waveforms evincing the neighbor frequency effect, aggregated over items and lexical types. As with N, items with higher neighbor frequencies elicit larger N400s, regardless of lexical type. Also as in the case of N, the same pattern of effects was observed on second presentation. An item ANOVA with factors of neighbor frequency (high or low) and lexical type (lexical or nonlexical) again revealed a strongly reliable main effect of neighbor frequency (F(1,296) 5 27.8, po.0001), but no main effect of lexical type (F 5 3.14) and no interaction (F 5 2.83). Also as with N, on second presentation the correlations between neighbor frequency and N4 amplitude were reduced, but still highly reliable, for both lexical types (for lexical items, r 5 ! .39, r2 5 .15, po.0001; for nonlexical items r 5 ! .34, r2 5 .12, po.0001). Thus, just as with N, items with higher neighbor frequency elicit larger N400s on both first and second presentation, regardless of lexical type. We again conduced an identical set of analyses using the frequency of the most frequent neighbor (as opposed to the summed frequency of all neighbors) on second presentation. An item ANOVA with factors of max neighbor frequency (high or low) and lexical type (lexical or nonlexical) showed that with this measure of neighbor frequency, on second presentation, there

Middle Parietal

5 µV

900 ms

LAD: N = 22, High

BAP: N = 22, High

LCD: N = 5, Mid

BNN: N = 5, Mid

NPR: N = 1, Low

MVH: N = 1, Low

Figure 3. N effect in item ERPs: Left, the ERPs elicited by lexical items with Ns of 1, 5, and 22: the word LAD, and the acronyms LCD and NPR (Liquid Crystal Display and National Public Radio). Right, ERPs elicited by nonlexical items with the same Ns: the pseudoword BAP and the illegal strings of letters BNN and MVH. Individual N is strong predictor of N400 amplitude, regardless of lexical type.

182

S. Laszlo & K. D. Federmeier Summed Log Neighbor Frequency


−1 0

0

1

2

3

4

5

6

tion. The rank sum test was only marginally reliable (p 5 .12). However, the more powerful single regression of number of lexical associates with N4 mean amplitude was reliable (r 5 ! .34, r2 5 .12, p 5 .008). Thus, similar to the analogous effect of orthographic neighborhood size, items with more lexical associates elicit more negative N400s. Figure 6 displays this relationship. On second presentation, an equivalent rank sum test performed on a median split of the N4 mean amplitude data sorted by number of lexical associates was reliable (p 5 .02), as was the single regression of number of lexical associates with N4 mean amplitude (r 5 ! .41, r2 5 .17, p 5 .001). On second presentation, as on first presentation, items with more lexical associates elicited a more negative N400.

7

1 2 3 4 5 6 7 8

Frequency of Top Associate Effects A median split of the item N4 mean amplitude data when sorted by frequency of top lexical associate put 30 items in the ‘‘high frequency of top lexical associate’’ category and 30 items in the ‘‘low frequency of top associate’’ category. Although the number of items in the high and low categories was thus balanced in this comparison, for analogy with the analyses of the effects of number of lexical associates, we again used rank sum tests in our factorial analysis of the effects of frequency of top associate. A rank sum test on the effect of frequency of top associate on N400 amplitude on first presentation was reliable (p 5 .03), and, accordingly, so was the single regression correlation of frequency of top associate with N4 amplitude (r 5 ! .27, r2 5 . 07, p 5 .04). Figure 6 depicts the effect of frequency of top associate side by side with the effect of number of lexical associates. On second presentation, the effect of frequency of top associate on N4 mean amplitude was not reliable either in the factorial analysis (rank sum p 5 .38) or the single regression correlation (r 5 ! .13, p 5 .32).

Figure 4. Equivalency of neighbor frequency effect across lexical types: Item N400 mean amplitude (250–450 ms) over the middle parietal channel is plotted against neighbor frequency for lexical items (filled circles) and nonlexical items (empty circles). Single regression trend lines for the relationship between N4 mean amplitude and neighbor frequency are also plotted for each item type. The function relating N400 amplitude to neighbor frequency is nearly identical for the two item types.

was no main effect of either neighbor frequency or lexical type, and no interaction between the two (for neighbor frequency, F 5 1.72, for lexical type F 5 2.87, for the interaction F 5 0.67). Accordingly, the single correlations between N4 mean amplitude and max neighbor frequency were not reliable for either lexical or nonlexical items (for lexical items, r 5 ! 0.08, p 5 .31; for nonlexical items r 5 ! 0.05, p 5 .56). Number of Lexical Associates Sixty-one of our lexical items were included in the South Florida Free Association Norms (Nelson et al., 1998), and a median split of the N4 mean amplitude data when sorted by number of lexical associates put 27 items in the ‘‘high number of lexical associates’’ category and 28 in the ‘‘low number of lexical associates’’ category. Because there were different numbers of items in the two categories, we used a nonparametric rank sum test (equivalent to a Mann-Whitney U test) to examine whether or not there was a factorial effect of number of lexical associates on first presenta-

Multiple Regressions Of particular interest was to use multiple regression to enable examination of the unique effects of each of our variables of interest. We conducted two multiple regression analyses: one pertaining to orthographic variables and one pertaining to lexical variables. Included in the orthographic analysis were N and

Middle Parietal

5 µV

900 ms

High N

High Neighbor Frequency

Mid N

Mid Neighbor Frequency

Low N

Low Neighbor Frequency

Figure 5. Effects of N and neighbor frequency: Left, grand average ERPs elicited in response to items with high, mid, or low orthographic neighborhood size (N). Right, grand average ERPs elicited in response to items with high, mid, or low neighbor frequency. All ERPs are from the middle parietal channel, and are averaged over both lexical and nonlexical items. In part because the two variables are highly inter-correlated, the effects are quite similar.


183

Number of Lexical Associates 5

10

15

20

Log Frequency of Top Associate 25

1 2 3 4

r2 = .12

5

0 N400 Mean Amplitude (µV)


0

1

2

3

4

1 2 3 4

r2 = .07

5

Figure 6. Effects of number and frequency of lexical associates: Left, a scatter plot showing the relationship of N400 mean amplitude (250–450 ms) and number of lexical associates at the single item level. Right, an identical scatter plot showing the relationship of N400 mean amplitude and log frequency of top lexical associate. Items with more lexical associates and items with more frequent lexical associates both elicit more negative N400s.

neighbor frequency (which are strongly correlated in this dataset: r 5 .64, po.001). Included in the lexical analysis were number of lexical associates and frequency of top associate (which are more weakly correlated in this datasest: r 5 .19, p 5 .14). Because the effects of N and neighbor frequency are so similar across the lexical and nonlexical item types, we collapsed across lexicality in the analysis of orthographic factors. (In addition to the lack of interaction in the N4 window, we also observed no differences between these item types in the immediately preceding P2 (175–225 ms, middle prefrontal channel) window (t(298) 5 .09, p 5 .93).) Automated stepwise multiple regression revealed that the most reliable predictor of N4 amplitude was N, followed by neighbor frequency. Alone, N explained 30.6% of variance in N400 mean amplitude (F(298) 5 131.58, po.0001). With the variance from N already explained, the stepwise procedure did add neighbor frequency to the model, which explained an additional 1.2% of variance (F(297) 5 69.37, po.0001). Thus, when combined, these two factors explain 31.8% of variance in N400 mean amplitude. Neighbor frequency was added to the model after N even when length was additionally added to the pool of lexical variables available to the stepwise procedureFa supplemental analysis we conducted because length and N are strongly correlated in this dataset (r 5 ! .27, po.0001). Additionally, a simultaneous multiple regression including length, N, and neighbor frequency was highly reliable (F(296) 5 56.73, po.0001). Because both N and neighbor frequency also influenced N400 amplitudes for repeated items, we performed the same regression analysis on data from the second presentation of each item, again collapsed across lexicality. Automated stepwise multiple regression revealed that the most reliable predictor of N4 amplitude was again N, followed by neighbor frequency. Alone, N explained 14.8% of variance in N400 mean amplitude (F(298) 5 51.81, po.0001). With the variance from N already explained, the stepwise procedure again added neighbor frequency to the model, which explained an additional 2.5% of variance (F(297) 5 31.10, po.0001). Thus, when combined, these two factors explain 17.3% of variance in N400 mean amplitude to items that have been repeated. An identical automated stepwise multiple regression conducted over lexical variables (number of lexical associates and frequency of top lexical associate for all 61 items for which this information was available) revealed that number of lexical associates was a better predictor of N4 amplitude than was frequency of top associate. Alone, number of lexical associates

explained 11.5% of variance in N4 mean amplitude (F(59) 5 7.65, p 5 .008). With the variance due to number of lexical associates already explained, adding in frequency of top associate explained an additional 4.1% of variance (F(58) 5 5.37, p 5 .007). Thus, when combined, these two factors explained 15.6% of variance in N400 mean amplitude. Number of lexical associates was strongly correlated with written frequency in this dataset (r 5 .35, p 5 .006), but an additional automated stepwise procedure conducted with number of lexical associates, frequency of top associate, and written frequency as predictor variables added number of lexical associates to the model after variance due to frequency was explained. In this case, neighbor frequency was added only if a less conservative entry criterion was used (p o.10). Thus, the independent contributions of item frequency and neighbor frequency are more difficult to disentangle. The simultaneous regression with all three variables was also reliable (F(57) 5 3.60, p 5 .02). Discussion Our goal was to discover whether properties of items related to an input itemFeither orthographically or lexicallyFwould have any effect on the magnitude of the N400 ERP component elicited by that input. We were motivated to this goal by recent evidence suggesting that the range of information activated by a particular input may be considerably broader than is assumed in classical, staged models of readingFespecially at the semantic level of representation. Thus, we looked for N400 effects of orthographic neighborhood size and neighbor frequency (either summed or maximum) and what we thought of as their lexical correlates, namely number of lexical associates and frequency of top associate. We found effects of all four factors, consistent with the hypothesis that orthographic inputs activate not only directly associated semantic information but also information associated with items related to the input on at least two levels of representation (orthographic and lexical), and that this information is accessed in a cascaded, not staged, fashion. We replicated and extended previous findings showing a relationship between orthographic neighborhood size and N4 amplitude (Holcomb et al., 2002; Laszlo & Federmeier, 2007, 2008, 2009) with high N items eliciting larger N400s than low N items. Our use of regression analysis on individual item ERPs showed clearly that this is a graded effect, and a strong one, with just over 30% of unique variance in N400 mean amplitude explained by N

184 alone. Furthermore, this relationship was statistically indistinguishable for lexically represented items (words, acronyms) and items without lexical representation (pseudowords, illegal strings of letters)Falthough, of course, non-lexical items are discriminated from lexical items in portions of the ERP subsequent to the N400 window. The finding that number of orthographic neighbors strongly affects N400 amplitude already hints that semantic information associated with orthographically similar items becomes active in parallel with that for a given input. However, as we described in the introduction, N alone could potentially be thought of as a proxy for some property of lexically represented items such as how likely they are to be pronounced similarly to other wordsFalthough it is then difficult to explain the identical N effects we observed for non-lexical items. Nevertheless, the present data revealed an even stronger finding in support of the hypothesis that items orthographically related to an input affect that input’s semantic processing. In particular, we found that items with orthographic neighbors that are high in lexical frequency tend to elicit N400s with larger amplitude than items with neighbors that are low in frequency. Multiple regression analysis revealed that, even though N and neighbor frequency are strongly correlated, neighbor frequency explains an additional, unique portion of variance in N400 amplitude. To our knowledge, this is the first time that neighbor frequency has been shown to affect the N400 independent of N (c.f. Debruille, 1998). Although lexical frequency effects on the N400 have often been in the form of amplitude reductions (more positivity) to high as opposed to low frequency words, the effect of neighbor frequency we observe here is different, with more negative responses when neighbors of input items have high lexical frequency. However, this pattern may reflect a similar underlying mechanism. Traditional N400 frequency effects are often interpreted as reflecting the ‘‘ease’’ with which an item becomes active, with higher frequency words being easier to activate than lower frequency words. This higher ease of activation may reflect a greater tendency for the neighbor to become active when an item containing some of its orthographic features is encountered. In other words, the neighbor item is a better ‘‘lure’’ when it is of higher frequency. This explanation seems consistent with the finding in the behavioral literature that the lexical decision task takes longer for items with high frequency neighbors than with low frequency neighbors (e.g., Grainger, 1990), which has been interpreted as representing ‘‘interference’’ by the high frequency neighbors (Grainger, 1990). In the data described here, because higher frequency neighbors are more likely to become active in response to a given input, the net amount of semantic information evoked by that input is greater, resulting in larger N400 amplitude. We also found corresponding effects from items lexically associated with an input, which we believe to be novel to the N400 literature. In particular, N400 amplitudes were larger for words with higher numbers of lexical associates, suggesting again that inputs evoke semantic activity associated with a set of items that are similar or interconnected at lower processing levels. Some items elicit a greater spread of activation at the lexical level and, in turn, a greater net level of initial activity in the semantic system. Analogous to the pattern seen for orthographic neighbors, we also found that N400 amplitudes are larger for items whose top associate is higher in lexical frequency. We were not able to statistically disentangle effects that might be due to frequency of a word’s top associate from effects that might arise from the

S. Laszlo & K. D. Federmeier frequency of the word itself; however, it is worth noting the direction of the effect, if due to word frequency rather than frequency of the top associate, goes in the opposite direction from that typically observed, as in this case more frequent words (with more frequent top associates) elicited a larger (rather than a smaller) N400. The effects of lexical association were smaller than the effects of orthographic similarity, perhaps reflecting their second-order nature. That is, whereas it is reasonable to assume that orthographic neighbors are activated directly by the input (i.e., the presence of ‘‘ORK’’ in the input FORK directly causes some co-activation of PORK), the activation of lexical associates must be mediated, such that the activation of, for example, SPOON is dependent on the activity associated with FORK. The distinction between measures that reflect properties of a subset of the networkFsuch as N and number of lexical associatesFand measures that instead represent the properties of single itemsFsuch as neighbor frequency and frequency of top associateFis critical to explaining the different impact of repetition on neighbor or associate effects, as compared with neighbor or associate frequency effects. Effects of both N and number of lexical associates were maintained across repetitions. However, effects of neighbor frequency were only maintained when a measure of lexical frequency summed across all an item’s neighbors was used. When max neighbor frequencyFa measure more similar to the frequency of a single, top associateFwas used, an effect of neighbor frequency was no longer observable on second presentation. Because both N and number of lexical associates are properties of the structure of the comprehension network, it makes sense that these factors would have an impact every time an input is encountered (and, indeed, N effects have been shown to persist even for the final words of highly constraining sentences; Laszlo & Federmeier, 2009)F a single presentation of an item does not affect the entire system it is embedded in in a persistent way. In contrast, effects that arise due to baseline activity of particular itemsFfor example, frequency effectsFcan be over-ridden by the processing context (e.g., Van Petten & Kutas, 1990). Thus, it makes sense that the effect of frequency of an orthographic or lexical associate is reduced (in fact, statistically eliminated) with repetition, as first order lexical frequency effects of input items (i.e., not even second order effects of associates of items) have been found to be similarly context sensitive (Van Petten & Kutas, 1990). Taken all together, the findings that semantic processing, as indexed by the N400, is modulated by the number of items that share orthographic features with an input and the number that are lexically associated with that input, as well as by lexical properties (such as frequency) of those orthographically or lexically related items strongly suggest that semantic access does not serially follow a recognition process in which the input has been mapped onto a single, stored representation. In models of that type (e.g., Borowsky & Besner, 1993; Forster & Davis, 1984), semantic processing should be limited to lexically represented items, and only semantic features directly associated with a recognized input should become active. That is to say, no semantic processing should be observed for nonlexical items like our pseudowords and illegal strings. Instead, it seems that activity is elicited in the semantic system for both lexical and nonlexical inputs, and that this activity is cascaded from lower-level (orthographic, lexical) processes, such that semantic features associated with a range of similar (or associated) inputs become active in parallel, beginning around 250 ms post-stimulus onset.

N400 neighbor and associate affects Although serial models do not predict this pattern, it is entirely consistent with cascaded models, which do not require lexical access to be complete (or ultimately successful) in order for semantic processing to begin. In arguing against models where semantic access is gated by recognition, these data are also inconsistent with views of the N400 that derive from such models, especially those that map the N400 onto some aspect of ‘‘post-lexical’’ processing (e.g., Brown & Hagoort, 1993; Sereno, Rayner, & Posner, 1998). For example, Hagoort, Baggio, and Willems (2009) have linked the N400 with post-recognition processes that integrate the (already accessed) meaning of the current word with sentence- and discourse-level representations. It seems difficult, under this kind of view, to explain how items with no lexical representationFsuch as pseudowords and orthographically illegal stringsFcan show identical, graded N400 effects to those shown by lexically represented items. Furthermore, the N400’s sensitivity to number of neighbors and associates and to properties of those items is inconsistent with the assumption that the meaning information associated with a given input has already been accessed by the time the N400 is measured. Instead, the present dataFin the context of the full set of variables known to modulate the N400 (for a review, see, e.g., Kutas & Federmeier, 2000)Fare more consistent with views that link the N400 to early aspects of semantic access (e.g., Federmeier & Laszlo, 2009; Kutas & Federmeier, 2000; Lau, Phillips, & Poeppel, 2008; Van Berkum, 2009), on the assumption that semantic access takes place in a cascaded processing stream and is distributed over time. Under such views, while linguistic effects can still be observed in the ERP prior to the N400 epochFfor example discrimination between pseudohomophones and orthographically matched controls (Grainger, Kiyonaga, & Holcomb, 2006) or discrimination between words and nonwords in the lexical decision task (Kiyonaga, Midgley, Holcomb, & Grainger, 2007)Fthey are interpreted not as evidence for early lexical access, but instead as representing complex perceptual or formal analysis, with the N400 still representing the first point in time at which amodal, position invariant representations come into contact with semantics (e.g., Grainger & Holcomb, 2009). For example, in the bi-modal interactive activation model (BIAM), which instantiates the principles of interactivity proposed in Rumelhart & McClelland’s Interactive Activation model (McClelland & Rumelhart, 1981; Rumelhart & McClelland, 1982), visual word recognition proceeds first through visual feature analysis around 100 ms post stimulus onset, then subsequently through position dependent and position invariant orthographic analysis at approximately 200 ms and 250 ms respectively. The position invariant orthographic analysis outputs representations akin to visual wordforms around 300 ms, and processing of these visual wordforms is only advanced enough to begin contacting semantics around 400 msFthat is, the time of the N400 (Grainger & Holcomb, 2009, Massol, Grainger, Dufau, & Holcomb, 2010). In addition to being more compatible with the present data than post-lexical views, models like the BIAM are consistent with what is known about the neural generators of early ERP components elicited during word reading (e.g., Marincovic, Dhond, Dale, Glessner, Carr, & Halgren, 2003; Tse, Lee, Sullivan, Garnsey, Dell, et al., 2007)Fan important constraint on cognitive models of any kind. However, even if post-lexical theories of the N400 are correct, and lexical access does take place prior to 400 ms (despite being opaque to ERPs, magnetoencephalography, and the event-re-

185 lated optical signal; although see Shtyrov, Kujala, and Pulvermuller (2010), for counterarguments to this claim), the present data clearly indicate that, during the N400 time window, the system is in a state wherein lexical and nonlexical items are treated identically, as indicated by the indistinguishable effects of N and neighbor frequency we observed for words and nonwords, and wherein activity reflects the structure of the input network, not just the properties of the input item itself. To our knowledge, no implemented or proposed model of serial word recognition would be expected to show such effect patterns in a post-recognition time window. Instead, we have previously suggested (Federmeier & Laszlo, 2009) that the basic temporal properties of the N400 inherently argue against the idea that the processing it indexes is dependent on a discrete recognition process, since recognition, both theoretically and empirically, would seem to take varying amounts of time for different types of stimuli and in different types of contexts, whereas the N400 manifests striking temporal stability. If, then, semantic access is not dependent on recognition, it follows that all types of stimuli might elicit N400 activity to some degreeFas was observed here. Furthermore, the present data suggest that activity in the N400 time window can reflect initial semantic activation states, that is, those which emerge before activity in orthographic levels of processing has reached a stable point. Thus, although the comprehension system will eventually reach a state in which only the orthographic features comprising F-O-R-K, and the corresponding semantic features of FORK are strongly activated (or in which the system has determined, for example, that there is no stable semantic representation associated with the input GORK), activity in the N400 time window is sensitive to points in processing earlier than this, when semantic information associated with a distributed set of co-activated representations comes online in parallel. Within the PDP modeling framework, this same point might be stated as suggesting that the N400 represents activity taking place in the semantic level of representation before either the orthographic or semantic layers have settled. The N400 might thus be well described as providing a temporally delimited ‘‘snapshot’’ of activity elicited by a given input in a distributed, cascaded, semantic system.

Conclusion Using a regression approach to examine effects of correlated variables on ERP responses to single items, we observed strong, independent effects of orthographic neighborhood size, neighbor frequency, number of lexical associates, and frequency of top associate on the amplitude of the N400 componentFthe latter three, to our knowledge, for the first time in the literature. This pattern supports a view of the N400 as indexing fairly early aspects of distributed semantic activation arising in a cascaded processing system. In turn, these data, in combination with the larger literature, are consistent with parallel distributed processing models of language comprehension, which are characterized by interactive dynamics and recurrent architecture. Such models can typically never be said to be doing only ‘‘semantic’’ processing or ‘‘orthographic’’ processing, as activation flows through all levels of representation in a parallel fashion. Thus, a snapshot of the model at any particular moment in time reflects activity in all levels of representationFmuch as, as suggested by the current data, the N400 represents a snapshot of late orthographic and early semantic processing occurring in parallel.

186

S. Laszlo & K. D. Federmeier REFERENCES

Besner, D., & Chapnik Smith, M. (1992). Models of visual word recognition: When obscuring the stimulus yields a clearer view. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18, 468–482. Borowsky, R., & Besner, D. (1993). Visual word recognition: A multistage activation model. Journal of Experimental Psychology: Learning, Memory, and Cognition, 19, 813–840. Brown, C., & Hagoort, P. (1993). The processing nature of the N400: Evidence from masked priming. Journal of Cognitive Neuroscience, 5, 34–44. Brunelli, R., & Poggio, T. (1993). Face recognition: Features versus templates. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15, 1042–1052. Dale, A. M. (1994). Source localization and spatial discriminant analysis of event-related potentials: Linear approaches. Unpublished doctoral dissertation. La Jolla, CA: University of California, San Diego. Dambacher, M., Kliegl, R., Hofman, M., & Jacobs, A. M. (2006). Frequency and predictability effects on event-related potentials during reading. Brain Research, 1084, 89–103. Deacon, D., Dynowska, A., Ritter, W., & Grose-Fifer, J. (2004). Repetition and semantic priming of nonwords: Implications for theories of N400 and word recognition. Psychophysiology, 41, 60–74. Debruille, J. B. (1998). Knowledge inhibition and N400: A study with words that look like common words. Brain and Language, 62, 202–220. Federmeier, K. D., & Laszlo, S. (2009). Time for meaning: Electrophysiology provides insights into the dynamics of representation and processing in semantic memory. In B. Ross (Ed.), Psychology of Learning and Memory, 51, 1–44. Forster, K. I. (1999). The microgenesis of priming effects in lexical access. Brain and Language, 68, 5–15. Forster, K. I., & Davis, C. (1984). Repetition priming and frequency attenuation in lexical access. Journal of Experimental Psychology: Learning, Memory, and Cognition, 10, 680–698. Forster, K. I., & Veres, C. (1998). The prime lexicality effect: Formpriming as a function of prime awareness, lexical status, and discrimination difficulty. Journal of Experimental Psychology: Learning, Memory, and Cognition, 24, 498–514. Grainger, J. (1990). Word frequency and neighborhood frequency effects in lexical decision and naming. Journal of Memory and Language, 29, 228–244. Grainger, J., & Holcomb, P. J. (2009). Watching the word go by: On the time-course of component processes in visual word recognition. Language and Linguistics Compass, 3, 128–156. Grainger, J., Kinyonaga, K., & Holcomb, P. J. (2006). The time-course of orthographic and phonological code activation. Psychological Science, 17, 1021–1026. Hagoort, P., Baggio, G., & Willems, R. M. (2009). Semantic unification. In M. Gazzaniga (Ed.), The Cognitive Neurosciences (4th edition, pp. 819–836). Boston: MIT Press. Harm, M. W., & Seidenberg, M. S. (2004). Computing the meanings of words in reading: Cooperative division of labor between visual and phonological processes. Psychological Review, 111, 662–720. Hauk, O., Davis, M. H., Ford, M., Pulvermuller, F., & Marslen-Wilson, W. D. (2006). The time course of visual word recognition as revealed by linear regression analysis of ERP data. NeuroImage, 30, 1313–1400. Hauk, O., Pulvermuller, F., Ford, M., Marslen-Wilson, W. D., & Davis, M. H. (2009). Can I have a quick word? Early electrophysiological manifestations of psycholinguistic processes revealed by event-related regression analysis of the EEG. Biological Psychology, 80, 64–74. Holcomb, P. J., Grainger, J., & O’Rourke, T. (2002). An electrophysiological study of the effects of orthographic neighborhood size on printed word perception. Journal of Cognitive Neuroscience, 14, 938–950. Kello, C. T., Plaut, D. C., & MacWhinney, B. (2000). The task-dependence of staged versus cascaded processing: An empirical and computational study of Stroop interference on speech production. Journal of Experimental Psychology: General, 129, 340–360. King, J.W, & Kutas, M. (1998). Neural plasticity in the dynamics of human visual word recognition. Neuroscience Letters, 244, 61–64. Kiyonaga, K., Midgley, K. J., Holcomb, P. J., & Grainger, J. (2007). Masked cross-modal repetition priming: An ERP investigation. Language and Cognitive Processes, 22, 337–376. Kutas, M., & Federmeier, K. D. (2000). Electrophysiology reveals semantic memory use in language comprehension. Trends in Cognitive Science, 4, 463–470.

Kutas, M., Van Petten, C. K., & Kluender, R. (2007). Psycholinguistics electrified II (1994–2005). In M. A. Gernsbacher & M. Traxler (Eds.), Handbook of Psycholinguistics (2nd edition, pp. 88–143). San Diego, CA: Academic Press. Laszlo, S., & Federmeier, K. D. (2007). Better the DVL you know: Acronyms reveal the contribution of familiarity to single word reading. Psychological Science, 18, 122–126. Laszlo, S., & Federmeier, K. D. (2008). Minding the PS, queues, and PXQs: Uniformity of semantic processing across multiple stimulus types. Psychophysiology, 45, 458–466. Laszlo, S., & Federmeier, K. D. (2009). A beautiful day in the neighborhood: An event-related potential study of lexical relationships in sentence context. Journal of Memory and Language, 61, 326–338. Lau, E. F., Phillips, C., & Poeppel, D. (2008). A cortical network for semantics: (De)constructing the N400. Nature Reviews Neuroscience, 9, 920–933. Marcus, M., Santorini, B., & Marcinkiewicz, M. (1993). Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19, 313–330. Marincovic, K., Dhond, R., Dale, A. M., Glessner, M., Carr, V., & Halgren, E. (2003). Spatiotemporal dynamics of modality-specific and supramodal word processing. Neuron, 38, 487–497. Massol, S., Grainger, J., Dufau, S., & Holcomb, P. (2010). Masked priming from orthographic neighbors: An ERP investivation. Journal of Experimental Psychology: Human Perception and Performance, 36, 162–174. McClelland, J. L., & Rumelhart, D. E. (1981). An interactive activation model of context effects in letter perception: Part 1. An account of basic findings. Psychological Review, 88, 375–407. Medler, D. A., & Binder, J. R. (2005). MCWord: An on-line orthographic database of the English language. Retrieved from: http:// www.neuro.mcw.edu/mcword/ Nelson, D. L., McEvoy, C. L., & Schreiber, T. A. (1998). The University of South Florida word association, rhyme, and word fragment norms. Retrieved from: http://www.usf.edu/FreeAssociation/ Plourde, C. E., & Besner, D. (1997). On the locus of the word frequency effect in visual word recognition. Canadian Journal of Experimental Psychology, 51, 181–194. Poggio, T., & Edelman, S. (1990). A network that learns to recognize three-dimensional objects. Nature, 343, 263–266. Rey, A., Dufau, S., Massol, S., & Grainger, J. (2009). Testing computational models of letter perception with item-level event-related potentials. Cognitive Neuropsychology, 26, 7–22. Rugg, M. D., & Nagy, M. E. (1987). Lexical contribution to nonwordrepetition effects: Evidence from event-related potentials. Memory & Cognition, 15, 473–481. Rumelhart, D. D., & McClelland, J. L. (1982). An interactive activation model of context effects in letter perception: Part 2. The contextual enhancement effect and some tests and extensions of the model. Psychological Review, 89, 60–94. Sereno, S. C., Rayner, K., & Posner, M. I. (1998). Establishing a timeline of word recognition: Evidence from eye movements and eventrelated potentials. NeuroReport, 9, 2195–2200. Shtyrov, Y., Kujala, T., & Pulvermuller, F. (2010). Interactions between language and attention systems: Early automatic lexical processing? Journal of Cognitive Neuroscience, 22, 1465–1478. Stolz, J. A., & Neely, J. H. (1995). When target degradation does and does not enhance semantic context effects in word recognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21, 596–611. Tse, C. Y., Lee, C. L., Sullivan, J., Garnsey, S. M., Dell, G. S., Fabiani, M., & Gratton, G. (2007). Imaging cortical dynamics of language processing with the event-related optical signal. Proceedings of the National Academy of Sciences of the United States of America, 104, 17157–17161. Van Berkum, J. J. A. (2009). Does the N400 directly reflect compositional sense-making? Psychophysiology, 46(Supplement 2), s2. Van Petten, C., & Kutas, M. (1990). Interactions between sentence context and word frequency in event-related brain potentials. Memory & Cognition, 18, 380–393. (Received January 20, 2010; Accepted April 3, 2010)


Listening strategy for auditory rhythms modulates neural correlates of expectancy and cognitive processing

JOEL S. SNYDER,a AMANDA C. PASINSKI,a and J. DEVIN McAULEYb,c a

Department of Psychology, University of Nevada Las Vegas, Las Vegas, Nevada Department of Psychology, Bowling Green State University, Bowling Green, Ohio c Department of Psychology, Michigan State University, East Lansing, Michigan b

Abstract A recently described auditory tempo perception paradigm revealed individual differences in perceived stimulus timing for identical stimulus sequences. The current study takes advantage of this paradigm by recording event-related potentials (ERPs) concurrent with task performance in order to reveal brain responses that reflect individual differences in timing strategy. No strategy-related differences were observed in sensory encoding of tones, as measured by the P1-N1-P2 complex. However, the contingent negative variation (CNV) leading up to the final tone of the sequence varied as a function of strategy, as did a parietal-maximum late positive component (P3b) that occurred following the final tone. These data suggest that temporal expectancy for and cognitive processing of the final tone of rhythmic sequences underlies differences in strategy during rhythm perception. Descriptors: Rhythm perception, Tempo, Expectancy, Individual differences, Event-related potentials

Once identified, such neural correlates can provide clues as to the importance of particular stages of processing or particular brain structures in timing behavior. McAuley, Frater, Janke, and Miller (2006) studied perception of a five-tone rhythm (see Figure 1), predicted to be perceived differently depending on strategy. One group of listeners heard the pattern speeding up or slowing down when the final interval was shorter or longer than 600 ms, respectively, possibly the result of imposing a 600-ms beat on the three tones at the beginning of the pattern, continuing the beat, and comparing it with the timing of the final tone. Another group of listeners always heard the pattern slowing down, possibly the result of implicitly calculating a pair of 300ms intervals at the beginning of the pattern and comparing this stored 300-ms interval with the much longer final interval. The two groups did not differ on a four-tone rhythm (see Figure 1), predicted to result in the same pattern of behavior regardless of strategy. Although McAuley et al. (2006) interpreted the observed individual differences as reflecting the extent of engaging beat-based (Large & Jones, 1999; Large & Snyder, 2009; McAuley, 1995; Schulze, 1978) versus interval-based (Treisman, 1963) mechanisms, it should be noted that it is also possible that the individual differences arose from different tempo preferences. For example, it is possible that, when presented with a sequence containing an ambiguous reference interval duration (i.e., 300 or 600 ms), some participants prefer to attend to the long interval whereas others prefer to attend to the short interval. Regardless of the precise nature of the individual differences revealed by the ambiguous tempo perception paradigm, these individual differences present a unique opportunity to identify brain activity related to different timing strategies, while controlling for stimulus effects. Recently, Grahn and McAuley (2009)

Processing temporal structure is a fundamental aspect of sensory, motor, and cognitive function in many complex organisms. In humans, temporal structure serves as an important cue to distinguishing different languages from early in life (Nazzi, Bertoncini, & Mehler, 1998), and in identifying the beat from music in order to coordinate movement in time during tapping, dancing, and music-making (Drake, Penel, & Bigand, 2000; Hannon, Snyder, Eerola, & Krumhansl, 2004; Snyder & Krumhansl, 2001; Toiviainen & Snyder, 2003; van Noorden & Moelants, 1999). But despite a long history of empirical and theoretical work, the mechanisms of timing ability in the human brain are still not well understood (Ivry & Schlerf, 2008; Zatorre, Chen, & Penhune, 2007). For example, the precise role of individual brain structures such as the cerebellum, basal ganglia, premotor cortex, supplementary motor area, and sensory cortices and how they interact in timing tasks is still a matter of debate. And what type of computational mechanism is used by humans to encode time is also unclear (Keele, Nicoletti, Ivry, & Pokorny, 1989; Martin, Egly, Houck, Bish, Barrera, et al., 2005; McAuley & Jones, 2003; Pashler, 2001; Schulze, 1978). One new approach to studying timing mechanisms is to identify neural correlates of individual differences in timing strategy. This work was supported by a summer research stipend from the College of Liberal Arts and a research development award from the Office for Research and Graduate Studies at the University of Nevada Las Vegas (J.S.S.) and grants from the GRAMMY Foundation and the National Science Foundation (J.D.M.). The authors thank Jessica Grahn for comments on an earlier draft of the paper. Address correspondence to: Joel S. Snyder, Department of Psychology, University of Nevada Las Vegas, 4505 Maryland ParkwayFMail Stop 5030, Las Vegas, NV 89154-5030. E-mail: [email protected] 198

Listening to rhythms

199 5-Tone Sequence 300 ms

300 ms

1200 ms

474, 526, 642, or 726 ms S

L

4-Tone Sequence 600 ms

1200 ms

474, 526, 642, or 726 ms

= Tones with fixed onset times = Possible onset times of final tone S = Long-interval-based expectancy L = Short-interval-based expectancy = Long- and Short-interval-based expectancy Figure 1. Stimuli. Two types of sequences were presented to participants, consisting of an initial group of three or two tones (for the five-tone and fourtone sequences, respectively) and a final group of two tones with a variable final IOI (474, 526, 642, or 726 ms). The task was to indicate whether the sequence was speeding up or slowing down at the end. If a listener adopts a weakly long-interval-based strategy, five-tone sequences would be expected to be perceived as always slowing down. In contrast, if a listener adopts a strongly long-interval-based strategy, then five-tone sequences would be expected to be heard as speeding up or slowing down depending on the final interval. Regardless of strategy, four-tone sequences would be expected to be heard as speeding up or slowing down depending on the final interval.

used this tempo paradigm in a functional magnetic resonance imaging (fMRI) study. They identified a number of specific brain areas activated during the task and found differences between long- and short-interval attending groups in mostly left-hemisphere structures (Grahn & McAuley, 2009). In particular, the inferior frontal gyrus, supplementary motor area, medial prefrontal cortex, and insula/ventrolateral prefrontal cortex, all on the left side, showed more activity for the long-interval group. The left superior temporal gyrus, left middle temporal gyrus, and right premotor cortex showed more activity for the short-interval group. The activity differences occurred while participants listened to both the five- and four-tone sequences, providing strong evidence that differences in neural activity related to timing strategy reflect general processing differences, even when no behavioral differences are apparent in the four-tone sequence. The limited temporal resolution of fMRI, however, prevented any strong conclusions about when, in the time course of processing, strategy-related differences occurred. Although group differences in prefrontal brain regions suggest the importance of higher-order cognitive aspects of processing, differences in the superior temporal lobe suggest the importance of lower-level auditory sensory processing in explaining differences in perception. The current study used the ambiguous tempo paradigm while recording event-related brain potentials (ERPs) in human listeners. Using ERPs specifically allowed us to examine whether strategy-related differences in neural processing were the result of sensory processing of stimulus events, forming expectancies for the final (variable) interval, or comparing the final interval with previous intervals and making a decision. We hypothesized that ERP correlates of early sensory processing, namely the P1-N1P2 long-latency auditory responses, might explain group differences in superior temporal lobe activations observed using fMRI (Grahn & McAuley, 2009). This could arise from differential attention to particular tones of the rhythmic sequence (Picton & Hillyard, 1974). For example, listeners with a preference for the long 600 ms intervals might attend more to the first and third tones of the five-tone sequence because these tones mark the

perceived 600-ms beat; this would result in larger long-latency sensory-evoked responses to the first and third tones compared to the other groups. On the other hand, the differential activations in the supplementary motor area suggest the importance of expectancy processes, which can be measured using the contingent negative variation (Macar, Vidal, & Casini, 1999; Pfeuty, Ragot, & Pouthas, 2003; Walter, Winter, Cooper, McCallum, & Aldridge, 1964). In particular, it is possible that attending to a 300-ms interval might result in contingent negative variation (CNV) time course that ends earlier or is smaller overall than if attending to a 600-ms interval. Finally, differential activations observed in other frontal areas suggest the involvement of expectancy violation or memory updating mechanisms, as indexed by P3a and P3b, respectively (Polich, 2007), which have been previously observed to be important in rhythm perception tasks (e.g., Besson, Faita, & Requin, 1994; Brochard, Abecasis, Potter, Ragot, & Drake, 2003; Ford & Hillyard, 1981; Jongsma, Desain, & Honing, 2004).

Materials and Methods Participants Thirty-eight adults (15 men and 23 women, age range 5 18–42 years, mean age 5 23.76 years) with normal hearing (! 30 dB HL from 250–4000 Hz) from the University of Nevada, Las Vegas Psychology subject pool participated for course credit after giving written informed consent. An additional eight participants were tested but were not included in the final sample because of poor behavioral performance in the four-tone control condition (5 participants) or because of poor-quality electrophysiological data (3 participants). Stimuli, Design, and Procedure The stimulus used to construct the rhythmic sequences was generated off-line in MATLAB (The MathWorks, Inc., Natick, MA) and consisted of a single pure tone (424 Hz, 50 ms in

200 duration, including 10-ms rise/fall times). Sequences using this tone were presented binaurally through ER3A headphones (Etymotic Research, Inc., Elk Grove Village, IL) at 70 dB SPL. Behavioral responses were made on a RB-830 button box (Cedrus Corporation, San Pedro, CA). All aspects of stimulus presentation and behavioral response collection were controlled by a custom program written in Presentation (Neurobehavioral Systems, Inc., Albany, CA), running on a Pentium 4 computer with a SB X-Fi sound card (Creative Technology, Ltd.). Two types of sequences were presented using the stimulus tone (Figure 1). Five-tone sequences consisted of 3 initial tones marking two 300-ms inter-onset intervals (IOIs) followed by 2 tones that specified a variable final IOI (600 ms ! DT). The key aspect of the five-tone sequence is that a periodic 600-ms beat is implied (but not explicitly emphasized) by the temporal structure of the sequence (Povel & Essens, 1985). Four-tone sequences, in contrast, consisted of 2 initial tones that specified a 600-ms IOI followed by 2 tones marking the same variable final IOI (600 ms ! DT). Thus, the only physical difference between the two types of sequences is that the four-tone sequence condition does not include the 2nd tone from the five-tone sequence condition. For both sequence types, the initial group of tones was separated from the final group of tones by an IOI of 1200 ms. Final IOIs of the sequences were 600 ms ! DT, where DT equaled ! 7% or ! 21% of the final IOI (474, 558, 642, or 726 ms). Each participant listened to both the five-tone and four-tone sequences with all four final IOIs, resulting in two within-subjects factors (sequence type and final IOI). Six blocks were presented, each containing 136 trials (17 of each trial type). Thus, each trial type was presented 102 times to each participant. Eight practice trials, one of each trial type, were presented prior to beginning the experiment. Participants were seated in a comfortable chair in a single-walled sound-attenuated room (Industrial Acoustics Corp., Bronx, NY) and were asked to maintain fixation on a white cross on a black background in the center of a computer screen throughout the experiment. Participants were asked to listen to the stimuli during electrophysiological recording, and to avoid moving their eyes, head, or other body parts while the sequences were presented. At the end of each sequence, participants indicated by pressing one of two buttons whether they perceived the pattern ‘‘slowing down’’ or ‘‘speeding up.’’ There was a 2-s interstimulus interval during which participants made their responses before a new trial began. Electrophysiological Recording Electroencephalographic (EEG) signals were recorded from an array of 72 electrodes, with a Ag-AgCl Common Mode Sense (CMS) active electrode and a Ag-AgCl Driven Right Leg (DRL) passive electrode serving as ground (see http://www.biosemi.com/faq/cms&drl.htm), placed at 64 points based on the 10/20 system in a Biosemi electrode cap and 8 additional points below the hair line (both mastoids, both pre-auricular points, outer canthus of each eye, and inferior orbit of each eye) and recorded onto a PC desktop computer for offline analysis. EEG signals were digitized continuously (512 Hz sampling rate and a 104 Hz bandwidth) using a Biosemi ActiveTwo system (http:// www.biosemi.com). Before EEG recording, conducting gel was applied to the skin at each electrode site with the cap on and sintered Ag-AgCl pin-type electrodes were fit into place at each site. Sintered Ag-AgCl flat-type electrodes were attached with adhesive to sites below the hair line. No abrading of the skin was performed. Voltage offsets were below 40 mV prior to recording

J. S. Snyder et al. and the resting EEG was checked for any problematic electrodes prior to and throughout the recording session. Data Analysis Proportion of ‘speeding up’ responses was calculated for each participant for each of the 8 trial types (2 sequence types " 4 final IOIs). To quantify the extent of long-interval-based responding, Response proportions for five-tone (test) sequences were fit with a simple contrast model in order to assess the extent to which participants’ tempo judgments about the five-tone (test) sequences were based on a 300-ms referent interval or a 600-ms referent interval; see Grahn and McAuley (2009) for full details of the model. In the model, binary (‘speeding up’/’slowing down’) judgments on a given trial are assumed to be based on one of two temporal referents: P 5 300 ms corresponding to the explicit time interval marked by the first three tones of the sequences (short temporal referent), and P 5 600 ms corresponding to the implied beat (long temporal referent). For each final IOI of the sequence, Ti, a temporal contrast metric, and Ci, is calculated, which measures the normalized difference between the final IOI and each referent, P: ðTi % PÞ P Previous work has shown that the temporal contrast metric is a good index of the information that participants use to make time judgment decisions (McAuley & Jones, 2003). Because there are two possible temporal referents, each final IOI, Ti, results in two values of Ci, labeled here as Ci300 for the P 5 300-ms referent and Ci600 for the P 5 600-ms referent. In line with standard signal detection assumptions (Macmillan & Creelman, 1991), values of temporal contrast for each referent are assumed to be normally distributed with standard deviation, s; the values of Ci300 and Ci600 were then z-transformed and combined using a simple weighted average: Ci ¼

z ¼ ð1 % wÞzi300 þ wzi600 Predicted proportions of ‘speeding up’ responses, P(‘Speeding Up’), for each final interval, Ti, are then generated using cumulative normal distribution function: Pð‘Speeding UpÞ ¼ 1 % fðzÞ Model fits allowed both wA[0, 1] and s to vary, minimizing the root-mean-square error (RMSE) between the observed and predicted response proportions. Most important for the present purposes, the continuous value of w provided an estimate of the extent to which tempo judgments for each participant about the five-tone (test) sequences were based on the explicit 300-ms (short) temporal referent or the implied 600-ms (long) temporal referent, with the latter consistent with a beat-based listening strategy. Thus, a participant with larger values of w shows greater tendency to judge tempo using the long temporal referent interval. Participants were divided into three roughly equal groups according to w, yielding strongly long-interval based (SLI, n 5 12), moderately long-interval based (MLI, n 5 13) and weakly longinterval based (WLI, n 5 13) listener groups. Response proportions were then entered into a mixed-measures analysis of variance (ANOVA) to test for differences in perception depending on the within-subjects variables sequence (4 tones vs. 5 tones) and final IOI ( % 21, % 7,17, 121%), and the between-subjects variable group (SLI, MLI, WLI). The degrees of freedom were adjusted with the Greenhouse-Geisser epsilon (e) when


had statistically different w values, F(2,35) 5 24.11, po.001 (SLI 5 0.99, MLI 5 0.92, WLI 5 0.55). In order to assess overall temporal sensitivity regardless of any strategy differences, discrimination thresholds were measured for each participant for the four-tone (control) sequences. Discrimination thresholds correlated negatively with w, r(36) 5 ! 0.675, po.001, indicating that shorter-interval listeners had poorer temporal discrimination than longer-interval listeners in the four-tone condition, even though no differences were expected depending on strategy in this condition. Because differences in discrimination thresholds for four-tone (control) sequences represents a possible confound in assessing effects of listening strategy on behavioral and brain responses to the five-tone (test) sequences, analyses reported below were run with and without the discrimination threshold included as a covariate. Tests that became non-significant when the covariate was included are indicated below. Otherwise, results are reported from analyses without the covariate. Figure 2 shows the proportion of trials in which participants perceived speeding up for the five-tone and four-tone sequences. The behavioral results are best understood by dissecting the significant three-way interaction between the factors, F(6,105) 5 7.09, po.001. This occurred because the stronger long-interval-based listening groups showed a steeper decline in

1 Proportion "Speeding Up"

0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 400

1

Behavioral Data Estimates of w showed a range of values across participants but an overall tendency for most participants to perceive the five-tone sequences in a long-interval-based manner. Despite the overall tendency for long-interval-based responding, the three groups

500 600 700 Final Interval Duration (ms)

800

4 Tones

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 400

Results

5 Tones

0.9

Proportion "Speeding Up"

appropriate, and all reported probability estimates were based on the reduced degrees of freedom. This adjustment was applied to all ANOVAs. All off-line ERP analyses were performed using Brain Electrical Source Analysis software (BESA, MEGIS Software GmbH, Gra¨felfing, Germany), except for baseline correction and amplitude measurements, which were performed by custom scripts in MATLAB (The MathWorks, Inc.). Electrodes that were noted during the recording as being noisy throughout the experiment were interpolated prior to analysis. Ocular artifacts (blinks, saccades, and smooth movements) were corrected automatically with a Principal Component Analysis method. Epochs contaminated by artifacts (amplitude4150 mV, gradient475 mV, low signalo0.10 mV) were automatically rejected before averaging. EEG epochs were averaged separately across all nonartifact trials for each of the sequence types (five-tone and fourtone) and for each electrode site, and re-referenced to the average of all electrodes. To examine ERPs related to processing the initial three or two tones of the sequences (for the five- and four-tone sequences, respectively), epochs were segmented with time 0 at the onset of the first tone of the sequence, with a 1226-ms pre-trigger baseline period and a 1000-ms post-trigger active period, and baseline corrected by subtracting the mean of the ! 26 to 0 ms portion of the baseline from each point in the epoch. These epochs were digitally band-pass filtered to attenuate frequencies below 0.5 Hz (6 dB/octave attenuation, forward) and above 30 Hz (24 dB/ octave attenuation, symmetrical). To examine ERPs related to processing the final two tones of the sequence, epochs were segmented with time 0 at the onset of the last tone of the sequence, with a 1226-ms pre-trigger baseline period and a 1000-ms post-trigger active period, and baseline corrected by subtracting the mean of the ! 26 to 0 ms portion of the baseline from each point in the epoch. These epochs were digitally band-pass filtered to attenuate frequencies below 0.5 Hz (6 dB/octave attenuation, forward) and above 30 Hz (24 dB/ octave attenuation, symmetrical). To quantify the CNV (which was expected to occur leading up to the final tone), these epochs were digitally band-pass filtered to attenuate frequencies below 0.1 Hz (6 dB/octave attenuation, forward) and above 30 Hz (24 dB/octave attenuation, symmetrical) and baseline corrected by subtracting the mean of the ! 752 to ! 726 ms portion of the baseline (time before penultimate tone onset) from each point in the epoch. ERP mean amplitudes were calculated in time ranges showing maximal differences in the grand-averaged waveforms between conditions of interest at electrodes showing the maximal difference. Mean amplitudes were averaged across a small set of 6 or 9 (depending on the particular ERP component) electrode sites for each participant and submitted to mixed-measures ANOVAs with Greenhouse-Geisser corrections when appropriate, with sequence (five-tone vs. four-tone) as the within-subjects factor and group (SLI, MLI, and WLI) as the between-subjects factor.

201

500 600 700 Final Interval Duration (ms)

800

Figure 2. Behavioral results. Participants were divided into strongly longinterval-based (SLI), moderately long-interval-based (MLI), or weakly long-interval-based (WLI) according to an index of long-interval-based listening. The three groups showed a similar pattern of behavior in the four-tone condition, but distinct patterns of behavior in the five-tone condition.

202

J. S. Snyder et al.

Figure 3. Sensory-evoked ERPs for initial tones. Responses are averaged across participants in the three groups for the five-tone and four-tone sequences. (A) Scalp topographical patterns of voltage at the P2 peak, showing maxima at central midline electrodes in all groups (electrodes labeled as filled circles were used for ERP amplitude measurement; filled black circle indicates electrode shown in panel B). (B) ERP traces showing responses at Cz, with P1, N1, P2, and N2 responses to Tone 1 labeled.

perceiving speeding up as a function of final IOI, especially for five-tone sequences. When the data from the five-tone and fourtone sequences were analyzed separately, significant interactions occurred between group and final IOI for five-tone sequences, F(6,105) 5 28.58, po.001, and four-tone sequences, F(6,105) 5 4.65, po.005, but the interaction for four-tone sequences became non-significant when the covariate was added, F(6,102) 5 1.77, p 5 .15. This result simply indicates that the difference in slope of the psychometric curve for the four-tone condition reflects a difference in temporal discrimination ability. The results for the SLI (and to a lesser extent the MLI) listeners in the five-tone condition can be explained if these listeners were more likely to judge tempo using the implied 600-ms beat and consistently expected events to occur every 600 ms, causing them to perceive speeding up when the final IOI was shorter than 600 ms and slowing down when the final IOI was longer than 600 ms. The results for the WLI listeners in the five-tone condition can be explained if these participants tended to perceive the mean of the first two IOIs to be 300 ms and perceived the final IOI to be longer than 300 ms. In summary, even when controlling for differences in sensitivity to temporal changes for the four-tone control pattern, the two groups of participants showed reliable differences in perception of the five-tone pattern. Specifically, the SLI and MLI listeners were much more affected by the final IOI in the five-tone sequence than the WLI listeners; SLI and MLI listeners appeared to base tempo judgments for five-tone sequences on the implied 600-ms beat, whereas WLI listeners did so to a much less extent. Electrophysiological Data One possible difference between groups in neural processing of the rhythms is the initial sensory encoding of the stimuli. Figure 3 shows grand-averaged ERPs of the three groups overlaid on top

of each other for the five-tone and four-tone sequences. Clear long-latency N1, P2, and N2 responses (measured at electrodes FCz/1/2, Cz/1/2, and CPz/1/2 from 95–120, 145–200, and 290– 330 ms, respectively) at central midline electrodes occurred in response to the first tone of both sequences. No significant differences were observed as a function of group for any of these responses, F(2,35) 5 0.65, 0.29, 0.53 and, p 5 .53, .75, and .60 for the N1, P2, and N2, respectively. Additional smaller responses occurred to subsequent tones in the two stimulus sequences but no group differences were apparent, and lack of reliable measurement of these responses precluded quantitative analysis. As shown in Figure 4, the P2 (measured at electrodes FCz/1/2, Cz/1/2, and CPz/1/2 from 140–200) to the final tone of the sequence showed similar amplitude as a function of group, F(2,35) 5 0.58, p 5 .57, as with the sensory evoked responses to the initial tone of the sequences. These negative results suggest that group behavioral differences are not likely to be related to modulation of sensory encoding. Another possible difference between groups is how they compute time intervals and form expectancies for the final tone in the sequences. As shown in Figure 5, a CNV at central midline electrodes appeared between the onsets of the penultimate and final tones of the sequence in all groups of listeners (measured at electrodes FCz/1/2, Cz/1/2, and CPz/1/2 from –400–0 ms relative to the final tone). The CNV we observed likely reflected expectancy for the final tone evoked by the penultimate tone and the preceding initial tones of the sequence, as a result of explicit computation of temporal intervals (Pfeuty et al., 2003; Walter et al., 1964). There was a trend in the data for a larger CNV in listeners more likely to judge tempo using a long temporal referent, but the main effect of group was not significant, F(2,35) 5 0.86, p 5 .43. However, there was a significant group ! sequence interaction, F(2,35) 5 5.25, po.025, due to larger CNV with more long-


203

Figure 4. Sensory-evoked ERPs for final tone. (A) Scalp topographical patterns of voltage at the P2 peak, showing maxima at central midline electrodes in all groups (electrodes labeled as filled circles were used for ERP amplitude measurement; filled black circle indicates electrode shown in panel B). (B) ERP traces showing responses at Cz, with P2 response labeled.

interval-based listening for the five-tone condition, but less of an effect of group in the four-tone condition. A final possible difference between listeners is in brain activity related to detecting the deviant in the final interval, prior to responding speeding up or slowing down. Such activity would be predicted to occur after the final tone in the sequences. In response to the final tone of both sequences, Figure 6 shows the presence of a late positive component occurring in all three listener groups (Donchin & Coles, 1988; Polich, 2007; Sutton, Braren, Zubin, & John, 1965). Prominent differences between the groups (measured at electrodes Pz/1/2 and POz/3/4 from 270–430 ms), F(2,35) 5 7.78, po.005, occurred at parietal electrodes, consistent with a P3b response (Polich, 2007), with more positive activity as a function of stronger long-interval-based listening in both the five-tone and four-tone sequences. The extended temporal course of the positive wave suggests that other components besides P3b might also be activated. No group ! sequence interaction was present. Thus, the P3b reflects differences in the strength of long-interval-based listening that are present for both the four-tone and five-tone sequence conditions, unlike the CNV differences that were only present in the five-tone condition. Finally, the topographic distribution of scalp voltage for the P3b response showed larger amplitude in right-hemisphere electrodes, as indicated by a significant main-effect of hemisphere (measured at electrodes P1/2 and PO3/4 from 270– 430 ms), F(1,35) 5 13.73, po.001. However, this right-hemisphere bias was similar for the three groups, as indicated by a non-significant group ! hemisphere interaction term. Discussion Differences between participants that can be attributed to rhythm processing strategy were reflected in brain activity related to timing and expectancy for the final tone of the

five-tone sequence that was designed to elicit strategy-related differences in behavioral judgments. The CNV has been clearly linked with temporal processing because it is elicited by a stimulus that temporally predicts a later stimulus (Besson, Faita, Czternasty, & Kutas, 1997; Macar et al., 1999; Martin, Houck, Kicic, & Tesche, 2008; Pfeuty et al., 2003; Pouthas, Garnero, Ferrandez, & Renault, 2000; Walter et al., 1964). Furthermore, the CNV correlates with the well-known behavioral observation that presenting multiple time intervals in succession enhances the precision of temporal judgments (Pfeuty et al., 2003). The finding that CNV differences between groups were only observed for the five-tone condition suggests that it indexes the active engagement of a particular strategy during rhythmic pattern processing. Whether the larger CNV in more long-interval-based participants is due to the engagement of stronger beat-based processing or simply due to greater attention to the implied 600-ms temporal referent per se is difficult to resolve from the current data. Although previous researchers have suggested that the CNV is likely to index the buildup of a time estimation process during interval-based processing (Martin et al., 2008; Pfeuty et al., 2003), it is equally likely that the CNVcould reflect the activation of an oscillatory mechanism in beat-based processing. Besides the CNV, other EEG-based measures of temporal expectancy are long-latency ERPs (Hughes, Darcey, Barkan, Williamson, Roberts, & Asline, 2001; Janata, 2001; Raij, McEvoy, Makela, & Hari, 1997; Simson, Vaughan, & Ritter, 1976; Weinberg, Walter, Cooper, & Aldridge, 1973) and high-frequency activity (Fujioka, Large, Trainor, & Ross, 2009; Iversen, Repp, & Patel, 2009; Snyder & Large, 2005; Zanto, Large, Fuchs, & Kelso, 2005; also see Lakatos, Karmos, Mehta, Ulbert, & Schroeder, 2008) evoked by missing expected events, which future studies may also show to distinguish between different listening strategies. The current study took an individual-differences approach to distinguishing activity related to timing strategy, specifically by dividing participants into groups depending on a behavioral

204

J. S. Snyder et al.

Figure 5. Contingent negative variation (CNV) following penultimate tone. (A) Scalp topographical patterns of voltage at ! 50 ms, prior to the final tone, showing maxima at frontocentral midline electrodes in all groups (electrodes labeled as filled circles were used for ERP amplitude measurement; filled black circles indicate electrodes shown in panel B). (B) ERP traces showing responses at FCz, Cz, and CPz electrodes, with CNV labeled. There is a larger CNV in stronger long-interval-based listening groups, mainly due to the difference in the five-tone sequence.

measure of sensitivity to an implied 600-ms temporal referent. However, another approach to distinguish between different processes in the brain related to timing is to look for physiological markers of these processes in different brain areas. For example, a recent study used source modeling of magnetoencephalography (MEG) data to identify activity that correlated with the positive effect of a prior warning stimulus during a visual timing task (Martin et al., 2008). The authors interpreted observed CNV-like slow-wave activity as reflecting the buildup of an interval-based mechanism (Treisman, 1963), and stimulus-related phase of activity as reflecting a phase-correcting beat-based mechanism (Large & Jones, 1999; McAuley, 1995; McAuley & Jones, 2003). Sources in the parietal lobe and cerebellum exhibited slow-wave activity similar to a CNV that correlated with behavioral performance enhancement, while the cerebellum and somatomotor cortex exhibited stimulus-related phase that also correlated with behavior. These results suggest not only that beat- and intervalbased timing strategies are both observable in different individuals, but that both types of mechanism may be operating in parallel in the same individuals. In the current study, strategy-related differences were observed in late positive brain activity (i.e., the P3b response) following the final tone of the two sequences. This was likely

indicative of a difference in cognitive processing of the final interval. A previous fMRI study using the same paradigm as the current study found brain activation differences in participants with high vs. low w values (Grahn & McAuley, 2009). Consistent with the current P3b differences, these differences occurred while participants listened to both the five- and four-tone sequences, providing strong converging evidence that some differences in neural activity related to timing strategy reflect general processing differences, even when no behavioral differences are apparent. The P3b difference between groups is also consistent with the previous fMRI data showing group differences in frontal and superior temporal brain regions, major generators of the P3b (Halgren, Marinkovic, & Chauvel, 1998). The group difference in P3b occurred for both five-tone and four-tone sequences and was present even when controlling for discrimination threshold in the four-tone condition (which is not predicted to differ depending on listening strategy). These results suggest that P3b differences do not simply reflect overall differences in temporal sensitivity or differences in temporal sensitivity specifically for the five-tone sequence. It is also unlikely that the difference in P3b occurred because weakly long-interval-based listeners were simply experiencing much larger deviations from the expected final interval than strongly long-interval-based


205

Figure 6. Late positive component (P3b) following final tone. (A) Scalp topographical patterns of voltage at P3b peak, showing maxima at parietal electrodes (electrodes labeled as filled circles were used for ERP amplitude measurement; filled black circle indicates electrode shown in panel B). (B) ERP traces showing responses at POz electrodes, with P3b labeled. There is a larger P3b in stronger long-interval-based listening groups for both five- and four-tone sequences.

participants because this would actually predict larger P3b amplitude in weakly long-interval-based participants and only in the five-tone condition, which was not observed. Finally, it is unlikely that individual differences occurred because the final interval was so far from an expected 300-ms interval that weakly long-interval-based listeners did not need to pay attention, because this would also predict group differences only for the fivetone condition. The fact that the P3b showed reliable differences between groups suggests the importance of this component in processing temporally-structured patterns, consistent with previous ERP studies. Following intervals that were shorter or longer than the previous intervals in isochronous sequences or following a break of a pattern of intervals, late positive responses occurred (Ford & Hillyard, 1981; Nordby, Roth, & Pfefferbaum, 1988a, 1988b) that were larger in participants who performed better at detecting temporal change (Jongsma, Meeuwissen, Vos, & Maes, 2007). Studies also showed that late positive components during rhythm tasks were larger in participants with extensive training in musical rhythms (Jongsma et al., 2004), although one study found no difference between musicians and non-musicians during detection of late notes in familiar and unfamiliar melodies (Besson et al., 1994). In addition to indicating detection of temporal irregularities and individual differences in rhythm perception, late positive components index the illusory perception of alternating accents in non-accented isochronous sequences (Brochard et al., 2003), and is enhanced when a temporal interval is accurately cued by a warning stimulus (Miniussi, Wilding, Coull, & Nobre, 1999). Late positive components also have been observed in response to deviants of non-rhythmic aspects of musical structure such as melody, harmony, and lyrics (Besson, Faita, Peretz, Bonnel, & Requin, 1998; Janata, 1995; Patel, Gibson, Ratner, Besson, & Holcomb, 1998; Trainor, McDonald, & Alain, 2002), suggesting a general role in processing sequential patterns.

It is perhaps surprising that the sensory-evoked P1-N1-P2 responses did not differ between listeners, especially for the first group of tones in the five-tone sequence because it was predicted that listeners sensitive to the 600-ms interval might attend more to the first and third tones compared to the second tone, which would result in enhanced activity (Picton & Hillyard, 1974). This negative result is all the more surprising given that sensory-evoked responses are sensitive to temporal structure, showing larger responses for larger time intervals (Carver, Fuchs, Jantzen, & Kelso, 2002; Hari, Kaila, Katila, Tuomisto, & Varpula, 1982; Lu, Williamson, & Kaufman, 1992; Mayville et al., 2001; Snyder & Large, 2004), sensitivity to the grouping structure of simple acoustic sequences (Loveless, Levanen, Jousmaki, Sams, & Hari, 1996; Loveless & Hari, 1993; Skrandies & Rammsayer, 1995), and modulation by early or late events during sensory-motor synchronization (Praamstra, Turgeon, Hesse, Wing, & Perryer, 2003; Tecchio, Salustri, Thaut, Pasqualetti, & Rossini, 2000). It is possible that the lack of individual differences in sensory-evoked responses is related to the fact that even participants in the weakly long-interval-based group had relatively high w scores. Thus, future ERP studies could screen participants in order to have listeners with maximally different rhythm processing strategies. The behavioral data from our study and the recent fMRI study using the same paradigm (Grahn & McAuley, 2009) suggest that there is no clear-cut distinction between listener groups. Rather, listeners appear to vary continuously in listening strategy, as revealed by the range of w scores. The current ERP data further suggest a lack of clear distinction between groups because similar neural responses were observed regardless of listening strategy, with only quantitative differences in amplitude for the elicited responses. The ERP data showed highly similar timecourses of brain activity at all stages of processing, with similarsized sensory-evoked responses to the tones and CNV and P3b responses that showed strategy-related processing of the final

206

J. S. Snyder et al.

time interval. Importantly, no qualitatively different pattern of brain responses was observed as a function of listening strategy. Thus, although the behavioral data do suggest that listeners can

have distinct patterns of responding, the ERP data suggest that such a pattern of behavioral data is generated by modulating the amount of activity in similar brain processes.

REFERENCES Besson, M., Faita, F., Czternasty, C., & Kutas, M. (1997). What’s in a pause: Event-related potential analysis of temporal disruptions in written and spoken sentences. Biological Psychology, 46, 3–23. Besson, M., Faita, F., Peretz, I., Bonnel, A. M., & Requin, J. (1998). Singing in the brain: Independence of lyrics and tunes. Psychological Science, 9, 494–498. Besson, M., Faita, F., & Requin, J. (1994). Brain waves associated with musical incongruities differ for musicians and non-musicians. Neuroscience Letters, 168, 101–105. Brochard, R., Abecasis, D., Potter, D., Ragot, R., & Drake, C. (2003). The ‘‘ticktock’’ of our internal clock: Direct brain evidence of subjective accents in isochronous sequences. Psychological Science, 14, 362–366. Carver, F. W., Fuchs, A., Jantzen, K. J., & Kelso, J. A. S. (2002). Spatiotemporal analysis of the neuromagnetic response to rhythmic auditory stimulation: Rate dependence and transient to steady-state transition. Clinical Neurophysiology, 113, 1921–1931. Donchin, E., & Coles, M. G. H. (1988). Is the P300 component a manifestation of context updating. Behavioral and Brain Sciences, 11, 357–374. Drake, C., Penel, A., & Bigand, E. (2000). Tapping in time with mechanically and expressively performed music. Music Perception, 18, 1–23. Ford, J. M., & Hillyard, S. A. (1981). Event-related potentials (ERPs) to interruptions of a steady rhythm. Psychophysiology, 18, 322–330. Fujioka, T., Large, E. W., Trainor, L. J., & Ross, B. (2009). Beta and gamma rhythms in human auditory cortex during musical beat processing. Annals of the New York Academy of Sciences, 1169, 89–92. Grahn, J. A., & McAuley, J. D. (2009). Neural bases of individual differences in beat perception. NeuroImage, 47, 1894–1903. Hannon, E. E., Snyder, J. S., Eerola, T., & Krumhansl, C. L. (2004). The role of melodic and temporal cues in perceiving musical meter. Journal of Experimental Psychology: Human Perception and Performance, 30, 956–974. Halgren, E., Marinkovic, K., & Chauvel, P. (1998). Generators of the late cognitive potentials in auditory and visual oddball tasks. Electroencephalography and Clinincal Neurophysiology, 106, 156–164. Hari, R., Kaila, K., Katila, T., Tuomisto, T., & Varpula, T. (1982). Interstimulus-interval dependence of the auditory vertex response and its magnetic counterpart: Implications for their neural generation. Electroencephalography and Clinical Neurophysiology, 54, 561–569. Hughes, H. C., Darcey, T. M., Barkan, H. I., Williamson, P. D., Roberts, D. W., & Aslin, C. H. (2001). Responses of human auditory association cortex to the omission of an expected acoustic event. NeuroImage, 13, 1073–1089. Ivry, R. B., & Schlerf, J. E. (2008). Dedicated and intrinsic models of time perception. Trends in Cognitive Sciences, 12, 273–280. Iversen, J. R., Repp, B. H., & Patel, A. D. (2009). Top-down control of rhythm perception modulates early auditory responses. Annals of the New York Academy of Sciences, 1169, 58–73. Janata, P. (1995). ERP measures assay the degree of expectancy violation of harmonic contexts in music. Journal of Cognitive Neuroscience, 7, 153–164. Janata, P. (2001). Brain electrical activity evoked by mental formation of auditory expectations and images. Brain Topography, 13, 169–193. Jongsma, M. L., Desain, P., & Honing, H. (2004). Rhythmic context influences the auditory evoked potentials of musicians and non-musicians. Biological Psychology, 66, 129–152. Jongsma, M. L., Meeuwissen, E., Vos, P. G., & Maes, R. (2007). Rhythm perception: Speeding up or slowing down affects different subcomponents of the ERP P3 complex. Biological Psychology, 75, 219–228. Keele, S. W., Nicoletti, R., Ivry, R. B., & Pokorny, R. A. (1989). Mechanisms of perceptual timing: Beat-based or interval-based judgements? Psychological Research, 50, 251–256. Lakatos, P., Karmos, G., Mehta, A. D., Ulbert, I., & Schroeder, C. E. (2008). Entrainment of neuronal oscillations as a mechanism of attentional selection. Science, 320, 110–113.

Large, E. W., & Jones, M. R. (1999). The dynamics of attending: How people track time-varying events. Psychological Review, 106, 119–159. Large, E. W., & Snyder, J. S. (2009). Pulse and meter as neural resonance. Annals of the New York Academy of Sciences, 1169, 46–57. Loveless, N., Levanen, S., Jousmaki, V., Sams, M., & Hari, R. (1996). Temporal integration in auditory sensory memory: Neuromagnetic evidence. Electroencephalography and Clinical Neurophysiology, 100, 220–228. Loveless, N. E., & Hari, R. (1993). Auditory evoked fields covary with perceptual grouping. Biological Psychology, 35, 1–15. Lu, Z. L., Williamson, S. J., & Kaufman, L. (1992). Behavioral lifetime of human auditory sensory memory predicted by physiological measures. Science, 258, 1668–1670. Macar, F., Vidal, F., & Casini, L. (1999). The supplementary motor area in motor and sensory timing: Evidence from slow brain potential changes. Experimental Brain Research, 125, 271–280. Macmillan, N. A., & Creelman, C. D. (1991). Detection Theory: A User’s Guide. New York: Cambridge University Press. Martin, T., Egly, R., Houck, J. M., Bish, J. P., Barrera, B. D., Lee, D. C., et al. (2005). Chronometric evidence for entrained attention. Perception and Psychophysics, 67, 168–184. Martin, T., Houck, J. M., Kicic, D., & Tesche, C. D. (2008). Interval timers and coupled oscillators both mediate the effect of temporally structured cueing. NeuroImage, 40, 1798–1806. Mayville, J. M., Fuchs, A., Ding, M. Z., Cheyne, D., Deecke, L., & Kelso, J. A. S. (2001). Event-related changes in neuromagnetic activity associated with syncopation and synchronization timing tasks. Human Brain Mapping, 14, 65–80. McAuley, J. D. (1995). Perception of time as phase: toward an adaptiveoscillator model of rhythmic pattern processing. Unpublished doctoral dissertation, Indiana University, Bloomington. McAuley, J. D., Frater, D., Janke, K., & Miller, N. S. (2006). Detecting changes in timing: Evidence for two modes of listening. The Proceedings of the 9th International Conference on Music Perception and Cognition, 188–189. McAuley, J. D., & Jones, M. R. (2003). Modeling effects of rhythmic context on perceived duration: A comparison of interval and entrainment approaches to short-interval timing. J Exp Psychol Hum Percept Perform, 29, 1102–1125. Miniussi, C., Wilding, E. L., Coull, J. T., & Nobre, A. C. (1999). Orienting attention in time. Modulation of brain potentials. Brain, 122, 1507–1518. Nazzi, T., Bertoncini, J., & Mehler, J. (1998). Language discrimination by newborns: Toward an understanding of the role of rhythm. Journal of Experimental Psychology: Human Perception and Performance, 24, 756–766. Nordby, H., Roth, W. T., & Pfefferbaum, A. (1988a). Event-related potentials to breaks in sequences of alternating pitches or interstimulus intervals. Psychophysiology, 25, 262–268. Nordby, H., Roth, W. T., & Pfefferbaum, A. (1988b). Event-related potentials to time-deviant and pitch-deviant tones. Psychophysiology, 25, 249–261. Pashler, H. (2001). Perception and production of brief durations: Beatbased versus interval-based timing. Journal of Experimental Psychology: Human Perception and Performance, 27, 485–493. Patel, A. D., Gibson, E., Ratner, J., Besson, M., & Holcomb, P. J. (1998). Processing syntactic relations in language and music: An event-related potential study. Journal of Cognitive Neuroscience, 10, 717–733. Pfeuty, M., Ragot, R., & Pouthas, V. (2003). Processes involved in tempo perception: A CNV analysis. Psychophysiology, 40, 69–76. Picton, T. W., & Hillyard, S. A. (1974). Human auditory evoked-potentials 2: Effects of attention. Electroencephalography and Clinical Neurophysiology, 36, 191–199. Polich, J. (2007). Updating P300: An integrative theory of P3a and P3b. Clinical Neurophysiology, 118, 2128–2148.

Listening to rhythms Pouthas, V., Garnero, L., Ferrandez, A. M., & Renault, B. (2000). ERPs and PET analysis of time perception: Spatial and temporal brain mapping during visual discrimination tasks. Human Brain Mapping, 10, 49–60. Povel, D. J., & Essens, P. J. (1985). Perception of temporal patterns. Music Perception, 2, 411–440. Praamstra, P., Turgeon, A., Hesse, C. W., Wing, A. M., & Perryer, L. (2003). Neurophysiological correlates of error correction in sensorimotor-synchronization. NeuroImage, 20, 1283–1297. Raij, T., McEvoy, L., Makela, J. P., & Hari, R. (1997). Human auditory cortex is activated by omissions of auditory stimuli. Brain Research, 745, 134–143. Schulze, H. H. (1978). The detectability of local and global displacements in regular rhythmic patterns. Psychological Research, 40, 173–181. Simson, R., Vaughan, H. G., & Ritter, W. (1976). Scalp topography of potentials associated with missing visual or auditory stimuli. Electroencephalography and Clinical Neurophysiology, 40, 33–42. Skrandies, W., & Rammsayer, T. (1995). The perception of temporal structure and auditory evoked brain activity. Biological Psychology, 40, 267–280. Snyder, J., & Krumhansl, C. L. (2001). Tapping to ragtime: Cues to pulse finding. Music Perception, 18, 455–489. Snyder, J. S., & Large, E. W. (2004). Tempo dependence of middle- and long-latency auditory responses: Power and phase modulation of the EEG at multiple time-scales. Clinical Neurophysiology, 115, 1885– 1895. Snyder, J. S., & Large, E. W. (2005). Gamma-band activity reflects the metric structure of rhythmic tone sequences. Cognitive Brain Research, 24, 117–126. Sutton, S., Braren, M., Zubin, J., & John, E. R. (1965). Evoked-potential correlates of stimulus uncertainty. Science, 150, 1187–1188.

207 Tecchio, F., Salustri, C., Thaut, M. H., Pasqualetti, P., & Rossini, P. M. (2000). Conscious and preconscious adaptation to rhythmic auditory stimuli: A magnetoencephalographic study of human brain responses. Experimental Brain Research, 135, 222–230. Trainor, L. J., McDonald, K. L., & Alain, C. (2002). Automatic and controlled processing of melodic contour and interval information measured by electrical brain activity. Journal of Cognitive Neuroscience, 14, 430–442. Treisman, M. (1963). Temporal discrimination and the indifference interval: Implications for a model of the ‘‘internal clock.’’ Psychological Monographs, 77, 1–31. Toiviainen, P., & Snyder, J. S. (2003). Tapping to Bach: Resonance-based modeling of pulse. Music Perception, 21, 43–80. van Noorden, L. P. A. S., & Moelants, D. (1999). Resonance in the perception of musical pulse. Journal of New Music Research, 28, 43–66. Walter, W. G., Winter, A. L., Cooper, R., McCallum, W. C., & Aldridge, V. J. (1964). Contingent negative variation: Electric sign of sensorimotor association and expectancy in the human brain. Nature, 203, 380–384. Weinberg, H., Walter, W. G., Cooper, R., & Aldridge, V. J. (1973). Emitted cerebral events. Electroencephalography and Clinical Neurophysiology, 34, 752–752. Zanto, T. P., Large, E. W., Fuchs, A., & Kelso, J. A. S. (2005). Gammaband responses to perturbed auditory sequences: Evidence for synchronization of perceptual processes. Music Perception, 22, 531–547. Zatorre, R. J., Chen, J. L., & Penhune, V. B. (2007). When the brain plays music: Auditory-motor interactions in music perception and production. Nature Reviews Neuroscience, 8, 547–558. (Received January 12, 2010; Accepted February 22, 2010)


Unintentional covert motor activations predict behavioral effects: Multilevel modeling of trial-level electrophysiological motor activations

VIVIAN ZAYAS,a ANTHONY G. GREENWALD,b and LEE OSTERHOUTb a

Department of Psychology, Cornell University, Ithaca, New York Department of Psychology, University of Washington, Seattle, Washington

b

Abstract The present experiment measured an EEG indicator of motor cortex activation, the lateralized readiness potential (LRP), while participants performed a speeded category classification task. The LRP data showed that visually masked words triggered covert motor activations. These prime-induced motor activations preceded motor activations by subsequent (to-be-classified) visible target words. Multilevel statistical analyses of trial-level effects, applied here for the first time with electrophysiological data, revealed that accuracy and latency of classifying target words was affected by both (a) covert motor activations caused by visually masked primes and (b) spontaneous fluctuations in covert motor activations. Spontaneous covert motor fluctuations were unobserved with standard subject-level (multi-trial) analyses of grand-averaged LRPs, highlighting the utility of multilevel modeling of trial-level effects. Descriptors: Priming, Motor activations, Event-related potentials

Despite extensive theorizing and research, the effects of these preparatory motor activations on task performance have not been clearly established. Past work has relied on standard subject-level analyses that aggregate across trials and across subjects. Such analytic techniques do not afford a clear test of the role of preparatory motor activations on subsequent performance. The present research is the first to use multilevel modeling (MLM) statistical techniques (e.g., Bryk & Raudenbush, 1992; Raudenbush & Bryk, 2002) on single-trial electrophysiological data to establish the impact of unintentional preparatory covert motor activations, both those triggered by environmental stimuli as well as those that occur spontaneously, on subsequent behavior.

In the absence of intentional effort or conscious awareness, environmental stimuli to which participants are instructed to ignore may trigger preparatory motor activations (e.g., Coles, Gratton, Bashore, Eriksen, & Donchin, 1985; Miller & Hackley, 1992; Osman, Bashore, Coles, Donchin, & Meyer, 1988, 1992). Moreover, even in the absence of an environmental trigger, spontaneous (i.e., more or less random and involuntary) fluctuations in preparatory motor activations may occur (Gratton, Coles, Sirevaag, Eriksen, & Donchin, 1988). Such preparatory covert motor activations may not be sufficient to trigger overt behavioral responses. Nonetheless, it is widely assumed that they influence the subsequent execution of appropriate responses (e.g., Dehaene, Naccache, Le Clec’H, Koechlin, Mueller, et al., 1998; Eimer & Schlaghecken, 1998, 2003). For example, in speeded two-choice classification tasks, unintentional preparatory motor activations favoring appropriate responses are expected to facilitate the actual execution of such responses. However, those favoring inappropriate responses are expected to interfere with the execution of suitable responses.

Category Priming Priming tasks are routinely used to assess processes automatically and unintentionally triggered by environmental stimuli (e.g., Ferguson & Zayas, 2009). In the standard category priming paradigm, participants classify visible target words that are dichotomized on a dimension (e.g., gender) into one of two categories (i.e., male vs. female). Prior to the presentation of targets, masked prime words from one of the two categories are presented. A robust finding from studies using this kind of procedure is the priming effect (e.g., Dehaene et al., 1998; Fazio, 2001; Greenwald, Draine, & Abrams, 1996). When primes and targets are congruent (both belong to the same category), classification of targets is facilitated, as reflected by higher accuracy, faster reaction times, or both. When primes and targets are incongruent (belong to opposing categories), classification of targets is more difficult.

The study was supported by Grants MH39349 from the National Institute of Mental Health and DC01947 from the National Institute for Deafness and Other Communication Disorders. We are grateful to Albert Kim and Judith McLaughlin for their assistance in data collection and analyses, and to Melissa Ferguson, Walter Mischel, and Yuichi Shoda for their thoughtful comments and suggestions on earlier drafts of the manuscript. Address correspondence to: Vivian Zayas, Department of Psychology, Cornell University, 240 Uris Hall, Ithaca, NY 14850. E-mail: [email protected] 208

Trial-level electrophysiological motor activations Historically, priming effects, especially those obtained from lexical decision tasks that call for word versus nonword judgments, have been interpreted as reflecting spreading semantic activation (Fazio, 2001). Upon the presentation of a prime, corresponding nodes within a person’s lexical–semantic network automatically become activated. In turn, this activation spreads throughout the network to associated nodes, including those corresponding to concepts that are related to the prime. Thus, a target that belongs to the same category as the preceding prime will be identified with greater ease, because nodes associated with the target have already been activated to some extent. A target that belongs to the opposing category as the preceding prime will not benefit from the spreading semantic activation caused by the prime. Although spreading semantic activation is still the most widely accepted account for priming, a growing body of research (e.g., Dehaene et al., 1998; Klinger, Burton, & Pitts, 2000; Praamstra & Seiss, 2005) supports the idea that category priming effects are caused, at least in part, by a process of response competition (see also Fazio, 2001). When the target classification task has a limited set of response options (male vs. female) and when primes are classifiable using those same responses, subjects will unintentionally apply the task instructions to the primes. As a result, the primes themselves will trigger preparatory motor activations that either facilitate or interfere with the classification of subsequently presented targets. Lateralize Readiness Potential (LRP) as a Measure of Covert Motor Activations Increasingly, researchers have been using electrophysiological measures to investigate motor activations triggered by environmental stimuli. The lateralized readiness potential (LRP) is an electroencephalographic (EEG), millisecond-to-millisecond, continuous record of the differential activation of the motor circuits responsible for controlling hand movements (e.g., Van Turennout, Hagoort, & Brown, 1998). Critically, the LRP captures ‘‘subthreshold’’ response activations, that is, low levels of preparatory covert motor activations that precede overt behavioral responses (Gratton, Coles, Sirevaag, Eriksen, & Donchin, 1988). Research using LRP consistently shows that stimuli that participants are instructed to ignore trigger covert motor activations (Coles et al., 1985; Miller & Hackley, 1992; Osman et al., 1988, 1992). Most relevant, using a masked priming task, Dehaene and colleagues (1998) showed that the direction of the initial covert motor activation was driven by the information provided by the prime. That is, congruent primes, which cued the same response as targets, triggered preparatory covert motor activations towards correct target responses. In contrast, incongruent primes, which cued the opposite response as targets, triggered preparatory motor activations towards incorrect target responses. These prime-induced preparatory motor activations occurred prior to the motor activations triggered by the presentation of the target. Moreover, analyses of reaction times revealed the expected behavioral priming effect (i.e., slower RTs on incongruent trials than congruent trials). Jointly, these LRP and behavioral data suggest that prime-induced preparatory covert motor activations interfered with or facilitated the execution of appropriate responses to subsequent stimuli (e.g., Eimer & Schlaghecken, 1998, 2003; Minelli, Marzi, & Girelli, 2007; Praamstra & Seiss, 2005). Even more, Gratton and colleagues (1988) have shown that prestimulus preparatory covert motor activations also influence subsequent behavioral responses. Specifically, in a two-choice con-

209 flict task (i.e., flanker; Eriksen & Eriksen, 1979), participants classify the central letter of a five-letter array (e.g., HHHHH, HHSHH). On ‘‘fast guess’’ trials in which participants responded within 150–199 ms of the presentation of the array, covert motor activations during the 100-ms fore period (the time preceding the onset of the stimulus array) predicted the subsequent behavioral response. Collectively, these findings suggest that changes in preparatory covert motor activations, as indexed by the LRP, may be unintentionally triggered by a prime stimulus, or reflect spontaneous (more or less random) fluctuations in motor readiness. Most importantly, they appear to play a role in the execution of subsequent behavioral responses. Trial-Level Effects of Covert Motor Activations on Subsequent Performance Although the existing work is consistent with the hypothesis that preparatory covert motor activations, whether prime-induced or spontaneous (more or less random), precede and impact behavioral performance on individual trials, the analytic techniques that have been used do not afford a clear test of this hypothesis. Research to date has used standard statistical procedures of aggregating data across participants and across trials for a given participant. However, such subject-level analyses do not unequivocally establish that preparatory covert motor activations and subsequent behavioral performance occur in known sequence on individual trials. It is possible, for example, that they are parallel effects with covert motor activations occurring for a subset of participants and behavioral priming effects occurring for a different subset, or that within the same participant, covert motor activations occur on a subset of trials and behavioral priming effects occur on a different subset.1 According to various models of continuous (vs. discrete) human information processing models (McClelland, 1979; Miller, 1988; Sternberg, 1969), environmental stimuli trigger a number of psychological processes (e.g., motor, semantic) likely operating, at least at times, in parallel. Thus, it is possible that the extent to which primes impact subsequent performance via motor activation, semantic activation, or a combination of both may vary from trial to trial within an individual or across individuals. This, in turn, may lead to behavioral effects occurring on some trials and covert motor activations occurring on different trials. For example, prime-induced covert motor activations may quickly dissipate given the short-lived nature of prime influence (Greenwald et al., 1996) and return to baseline levels before influencing performance on the subsequent target classification task. In this case, behavioral effects may still emerge, if primes exert their influence through other mechanisms (e.g., semantic activation). Present Research A test of the hypothesis that covert motor activations influence subsequent behavioral performance requires a shift from traditional subject-level, multitrial-aggregation analyses to trial-level analyses. In the present research, we used MLMs, a statistical technique well suited for investigating trial-level effects of motor processes on subsequent response. Although MLM techniques have been used extensively in a number of psychological domains 1 Analyses that involve sorting trials into bins (e.g., as a function of response time) also rely on standard subject-aggregate analyses, because data are averaged across subjects. Moreover, a characteristic of binning trials is that multiple trials from a given subject may be included in a bin, thereby violating assumptions of independence that are central to standard subject-aggregate statistical techniques.

210 (e.g., Zayas & Shoda, 2007; see also Bolger, Zuckerman, & Kessler, 2000), they have not been applied to electrophysiological data. The present paper is the first application of MLM to electrophysiological data for the purpose of modeling the effect of covert motor activations, on a given trial, on later performance outcomes; how these covert motor activations interact with, depend on, prime information; and the strength of these associations as they unfold over time. Method Overview of Procedures Participants completed all procedures individually on an IBM compatible desktop running Inquisit psychological software (Millisecond Software, LLC, Seattle WA). CRT monitors operated at a 120-Hz refresh rate. Participants completed a category priming task followed by a perceptibility task while their EEG was recorded. Participants Sixteen participants (9 female) completed the experiment in exchange for extra credit applied towards their introductory psychology courses. Participants had normal or corrected-tonormal vision. Category Priming Task Participants’ task was to classify the gender of visible male or female proper names (targets) presented in the center of the computer screen by pressing the E key with a finger from their right hand and the I key with a finger from their left hand. As

V. Zayas et al. shown in Figure 1, each target was preceded by a prime stimulus for 75 ms, which itself was preceded by a forward mask presented for 75 ms. There were no additional time intervals between stimuli. Participants indicated their response to the target within a 133-ms interval called the response window. The response window obliges participants to classify targets within a specified window of time, thus making individuals respond more quickly than they normally would be inclined to do. Thus, the response window increases participants’ reliance on prime information and increases the likelihood that a response will occur before the rapid decay of activation from the masked prime (e.g., Greenwald, Abrams, Naccache, & Dehaene, 2003). In block 1, the onset of the response window was 333 ms post target onset, and was delayed by 33 ms in each subsequent block.2 During the response window, a gray exclamation point (‘‘!’’) appeared on the computer screen. If participants indicated their response within the response window, the exclamation point turned red. On average, participants indicated their response within the response window on 62% of the trials (SD 5 4%) and indicated their response prior to the offset of the response window on 87% of the trials (SD 5 13%). The intertrial interval (ITI) was 1500 ms, during which a focus point (1) appeared on the computer screen. These specifications (Stimulus Onset Asynchrony (SOA) of 75 ms, prime duration of 75 ms, and response window procedure) have been shown to maximize the effect of the prime on the target classification task (Greenwald et al., 1996; Klinger et al., 2000). The category priming task consisted of three types of trials: congruent (prime and target belonged to the same gender category), incongruent (prime and target belonged to opposite gender categories), or no-information (prime was a letter string

Figure 1. Schematic representation of temporal structure of category priming task for congruent, incongruent, and no-information trials. Duration of each stimulus (in milliseconds) is in parentheses. Participants indicated their response to the target within a 133-ms interval called the response window. During the response window, a gray exclamation point appeared on the computer screen. If participants indicated their response within the response window, the exclamation point turned red. In block 1, the onset of the response window was 333 ms post target onset, and was delayed by 33 ms in each subsequent block.

Trial-level electrophysiological motor activations ‘‘XXXX’’). Participants completed two 24-trial and two 48-trial practice blocks followed by six 48-trial data blocks. Targets were presented in lowercase Arial font, and primes were presented in uppercase Arial font. Two sets of 12 male and 12 female proper names were used as stimuli. Stimulus set and response key assignment were counterbalanced across participants. To minimize blinking and other motor movements that would produce artifacts in the EEG recording, participants were instructed to minimize blinking throughout the experimental procedures and to blink in between blocks and in between trials (during the ITI). Perceptibility Task To assess the extent to which the masked primes were perceptible, participants completed a perceptibility task. This task was identical to the category priming task except that the participants’ task was to classify the gender of the prime and make their response after the end of the response window. The perceptibility task consisted of two 48-trial practice blocks and six 48-trial data blocks. Analyses of accuracy (excluding trials with reaction times (RTs)45000 ms) showed that primes were partially perceptible, but clearly difficult to identify (accuracy 5 63%, d 0 5 .76, t(15) 5 5.07, p 5 .0001). EEG Recording EEG was recorded using tin electrodes attached to an elastic cap (Electro-Cap International, Eaton, OH) placed over the left and right pre-frontal (Fp1, Fp2), frontal (F3, F4), inferior frontal (F7, F8), temporal (T7, T8), central (C3, C4), parietal (P3, P4), posterior parietal (P7, P8), and occipital (O1, O2) locations, and from three midline locations (Fz, Cz, Pz). Given the aims of the present research, we focused specifically on C3 and C4. Vertical and horizontal eye movements were recorded via electrodes placed below the left eye and to the right of the right eye, respectively. The double subtraction method used to derive the grand-averaged LRPs and the single subtraction method used to derive the trial-level covert motor activations (both described in the Data Reduction and Analytic Strategy section) alleviate activity caused by eye, muscle, and other motor-related artifacts. All channels were referenced to an electrode placed over the left mastoid bone. Activity recorded over the right mastoid was not affected by trial type. The EEG was amplified (SAI bioamplifier system) with a bandpass of .01–100Hz (3dB cutoff). The EEG and stimulus trigger codes were digitized on-line by a Data Translation 2801-A board at a sampling frequency of 200 Hz. Data Reduction and Analytic Strategy Behavioral data. Reaction times and accuracy were recorded for each trial. Trials with response latencies outside the normal range of time needed to categorize a single trial (i.e., greater than 1000 ms for the category priming tasks and greater than 5000 ms 2 The 33-ms incremental delay in the response window was included to investigate whether allowing participants more time to respond would weaken the influence of the prime. Analyses of the grand-averaged LRPs as well as the trial-level covert motor activations revealed no appreciable effect of delaying the response window. At the behavioral level, the magnitude of the priming effect decreased linearly as the response window was delayed (error rates: (F(1,15) 5 4.91, p 5 .043, Z2 5 .25; reaction times: (F(1,15) 5 34.51, p 5 o10 # 4, Z2 5 .70), although priming effects were statistically significant in each data collection block. Because the delay of the response window did not reliably influence covert motor activations, which are the focus of the present research, we report results of analyses collapsing across block.

211 for the perceptibility task) were excluded from all analysis. Analyses of reaction times were based on log-transformed reaction times for correctly classified targets only. Reaction times transformed back to milliseconds are reported for illustrative purposes. To investigate category priming effects on accuracy, we analyzed proportion incorrect as a function of trial type. Additionally, we used a signal detection approach, which takes into account responding biases, to corroborate results from analyses based on proportion incorrect. We computed signal detection theory’s sensitivity (d 0 ) measure by treating trials for which the prime belonged to the category female as signal trials and those for which the prime belonged to the category male as noise trials. The hit rate was thus the proportion of signal trials classified as female, and the false alarm rate was the proportion of noise trials classified as female. Similar analyses were conducted to assess performance on the perceptibility task. Grand-averaged LRPs. Given the focus on prime-induced activations, grand-averaged LRPs were time-locked to primes, and the 100-ms period preceding prime onset was used as a baseline. We computed the LRP following procedures described by Van Turennout and colleagues (1998): LRP ¼ meanðC3 # C4ÞRH #meanðC3 # C4ÞLH

ð1:0Þ

where rh represents trials in which the correct target response cued the right hand, and lh represents trials in which the correct target response cued the left hand. On each trial, for each sample point, the difference between potentials recorded from electrode sites placed over the left and right central medial-lateral sites (C3, C4) were averaged separately for trials in which the target stimulus called for left- and right-hand responses. The difference waveform obtained for left-cued trials was subtracted from the difference waveform obtained for right-cued trials.3 Thus, negative-going LRPs indicate covert activations of the correct response, and positive-going LRPs indicate covert activations of the incorrect response. Two-tailed t-tests were performed on voltage sampled every 5 ms. The LRP was defined as being present if the t-tests performed on five or more consecutive 5-ms samples were statistically different from zero in the same direction. The onset of the LRP was taken as the beginning of such a sequence (Van Turennout et al., 1998). Trial-level covert motor activations. Trial-level covert motor activations (referred to hereafter as TCMA) were assessed by computing, for each trial, the difference between potentials at C3 minus potentials at C4 (Gratton et al., 1988).4 Trials in which the target cued the left-hand were multiplied by # 1. Sixty-ms moving averages of this difference were computed, each shifted by 5 3 There are a number of methods available for computing the LRP. The method used in the present research is equivalent to procedures used by De Jong, Wierda, Mulder, and Mulder (1988) and Eimer and Schlaghecken (1998), except that it reverses the polarity. It is also equivalent to other methods (e.g., Coles, 1989; Gratton et al., 1988) that divide the entire sum by 2, thus halving the amplitude. 4 Conventional subject-level LRP involves the subtraction of average voltages recorded at C3 and C4 with left and right hand responses (or equivalent procedures). This subtraction removes lateralizations caused by structural and functional differences between the two hemispheres that are not related to motor lateralizations. To compute TCMA, we followed Gratton et al. (1988)’s method of assessing laterality on a given trial. Specifically, we subtracted voltages recorded at C3 and C4 on a given trial.

212

V. Zayas et al.

ms (0–59 ms, 5–64 ms, etc.). Temporal positions of TCMA are identified by the midpoints of these 60-ms intervals. Each trial consisted of 220 TCMA 60-ms intervals, with the first of these intervals starting at ! 100 ms ( ! 130 ms to ! 71 ms) post prime onset and the last ending at 1000 ms (970 ms to 1029 ms) post prime onset.

Accordingly, the level-2 models from these analyses estimated the average effects for the entire sample, and provide a test of whether the effect of TCMA varies across trial types.

MLM of trial-level effects of covert motor activations. The data from the present study are multilevel in that all trials, the level-1 units, were presented to each participant, the level-2 units. MLM (e.g., Bryk & Raudenbush, 1992; Raudenbush & Bryk, 2002) can estimate the relations among constructs at level 1 and level 2 simultaneously, while taking into account that the observations at level 1 are not independent. Accordingly, MLM is appropriate for estimating the effect of TCMA on subsequent response (accuracy and response time). Below, we describe the MLM analyses with accuracy as the outcome. We repeated the MLM analyses treating raw reaction time as the outcome variable (results using log-transformed reaction time were highly similar to those observed with raw reaction time). We first performed MLM to estimate the effect of each 60-ms TCMA interval on accuracy on a given trial for each of the three trial types separately. The level-1 model estimated, for each participant j (j 5 1–16), a regression line that predicted each participant’s accuracy (correct 5 0; error 5 1) on each trial i from the subject-centered 60-ms TCMA interval. This model was represented as follows:

Behavioral Priming Effects Analyses of the behavioral data showed the expected priming effects; task performance was facilitated on congruent trials and hindered on incongruent trials (Table 1). Compared to no-information trials, incongruent trials produced more errors, t(15) 5 10.77, p 5 10 ! 7, and slower responses, t(15) 5 9.57, p 5 10 ! 7. In contrast, compared to no-information trials, congruent trials produced fewer errors, t(15) 5 2.16, p 5 .048, and faster responses, t(15) 5 3.93, p 5 .001. To assess the magnitude of masked priming effects when prime perceptibility was zero, we followed Draine and Greenwald’s (1998) procedure of regressing priming effects on perceptibility effects, both measured in d 0 . The unstandardized intercept, an index of priming in the absence of prime perceptibility, was .56 and statistically greater than zero, t(15) 5 8.27, po.0001.

½errori #j ¼ b0j þ b1j ½TCMAi # þ rij

ð2:0Þ

where b0j, the intercept, is interpreted as participant j’s mean error rate for a given trial type (because all level-1 predictors were subject-centered); b1j, the slope, represents the effect of the 60-ms TCMA interval on accuracy for each participant j (positive coefficients represent the predictive magnitude of TCMA on correct target response for a given participant); and rij is the residual error term.5 The level-2 models estimated the average effects for the entire sample. The models were represented as follows: b0j ¼ g00 þm0j

ð2:1Þ

b1j ¼g10 þm1j

ð2:2Þ

where the intercept, g00, is interpreted as the average error rate for the entire sample; g10 is the average effect of TCMA on error (positive coefficients represent the predictive magnitude of TCMA on correct target response for the sample as a whole), and thus the primary estimate of interest; and m0j and m1j are the residual error terms. Note that, because we did not hypothesize differences among participants, no predictor variables were included in the level-2 models. Second, we performed MLM to investigate how the effect of TCMA on subsequent behavioral response varied across trial types. A set of priori contrasts was created to test the effect of each trial type relative to the other two. For these analyses, the level-1 model was identical to the previous model (2.0), except that it also included a contrast code for trial type and the trial type contrast ( TCMA interaction term as level-1 predictors. 5

A fixed slope effect for TCMA was specified for all models after ascertaining, using the log likelihood ratio test (Bryk & Raudenbush, 1992), that allowing individual-subject slopes to vary did not significantly increase model fit.

Results

Evidence of Prime-Induced Covert Motor Activations: GrandAveraged LRPs Before investigating TCMA effects, the grand-averaged LRPs, which reflect motor activations averaged across trials and across participants, were examined. Figure 2a shows the grand-averaged LRPs for all (i.e., correct and incorrect) trials as a function of priming conditions. Negative-going (upward) LRPs indicate activation of the correct response. Positive-going (downward) LRPs indicate activation of the incorrect response (see Method section for a description of the statistical analyses). On congruent trials, primes tended to trigger covert motor activations associated with the correct target response, whereas on incongruent trials, primes triggered covert motor activations associated with the incorrect target response. Evidence for this inference is based on the onset and initial direction of the LRPs for the three trial types. Specifically, the LRP for congruent trials was negative-going, indicative of covert activations of the correct target response, starting at 335 ms post prime onset. In contrast, the LRP for incongruent trials was positive-going, indicative of covert activations of the incorrect target response, starting at 340 ms post prime onset. Moreover, the LRP onsets for congruent

Table 1. Mean Error Rate (i.e., Proportion Incorrect) and Reaction Times (in Milliseconds) as a Function of Trial Type (Congruent, No-Information, and Incongruent) Error Rate (proportion incorrect)

Reaction Times (ms)

Trial Type

M

(SD)

M

(SD)

Congruent No-information Incongruent

.12a .15b .29c

(.07) (.08) (.10)

472.72a 486.71b 518.02c

(29.65) (26.35) (32.24)

Note: Means with different subscripts differ significantly (ps for differences between means ranged from 10 ! 7 to .05). Analyses of reaction times were based on log-transformed reaction times for correctly classified targets only. Reaction times transformed back to milliseconds are reported for illustrative purposes. SD: standard deviation.

Trial-level electrophysiological motor activations

213

and incongruent trials occurred approximately 65–70 ms earlier than the LRP onset for no-information trials, which was negative-going at 405 ms post prime onset. The 65–70 ms delay in LRP onset for no-information trials relative to the onset for congruent and incongruent trials corresponds with the 75 ms prime–target SOA.

a

prime onset

target onset

–3

All trials (correct and incorrect)

–2 –1 0

Lateraized Readiness Potential (µV)

+1

no information congruent

incongruent

0 75

b

300

600

900 [ms]

Correctly classified trials only

–3 –2

incongruent

–1

no information congruent

0 +1

0 75

c

300

600

900 [ms]

Incorrectly classified trials only

–1 0 +1 +2 +3

0 75 congruent

300

600 incongruent

900 [ms] no information

Figure 2. Grand-averaged lateralized readiness potential (LRP) waveforms. LRP waveforms are shown for (a) all trials (correctly and incorrectly classified), (b) correctly classified only, and (c) incorrectly classified only. In each panel, LRPs are plotted as a function of trial type: Congruent, incongruent, and no-information. Arrowheads mark the onset of the LRP for each trial type (i.e., the first of five or more consecutive 5 ms samples in which the LRP was statistically different from zero in the same direction). LRPs are time-locked to prime onset, which is marked by a vertical line at 0 ms. The vertical line at 75 ms post prime onset marks the onset of the target. LRPs deviate from the baseline (zero) as response preparation occurs. LRPs in the negative (upward) direction reflect activation of the contralateral motor cortex, indicative of preparing to make a correct target response. LRPs in the positive (downward) direction reflect activation of the ipsilateral motor cortex, indicative of preparing to make an incorrect target response. LRP waveforms were derived by (1) computing the difference between potentials recorded from electrode sites placed over the left and right central medial-lateral sites (C3–C4), (2) creating separate averages for trials in which the target stimulus called for left- and right-hand responses, and (3) subtracting the difference waveform obtained for left-cued trials from the difference waveform obtained for right-cued trials.

Prime-induced covert motor activations were substantial, as indicated by comparing the grand-average LRP for congruent and incongruent trials to that for no-information trials. At 405 ms post prime onset, the time at which activation of the correct response begins to be detectable on no-information trials, the incorrect response was still activated on incongruent trials as reflected by a statistically significant positive-going LRP. Moreover, at 405 ms post prime onset, the correct response was activated to a greater extent on congruent than no-information trials, t(15) 5 3.74, p 5 .002. The pattern of the grand-averaged LRP data on correctly classified trials (Figure 2b) was similar to those based on all trials, with one exception: analysis of the LRP revealed that correctly classified incongruent trials did not show significant activations of incorrect responses. The LRP for correctly classified incongruent trials was never significantly greater than zero. Nonetheless, the temporal order of LRP onsets as a function of trial type was similar to the pattern observed for analyses involving all (correctly and incorrectly classified) trials. Covert motor activations of the correct response occurred at 330 ms post prime onset on congruent trials, at 370 ms on no-information trials, and 430 ms on incongruent trials. On incorrectly classified trials, the pattern of the grand-averaged LRPs was approximately the mirror image of that for correctly classified trials (Figure 2c). Because LRPs for incorrectly classified trials were based on considerably fewer trials (see Table 1), they were less statistically detectable. Thus, the fluctuations in the LRP waveformo300 ms post prime onset were not statistically different from zero. Nonetheless, the sequence of covert motor activations suggests prime influence. On incongruent trials, in which primes cued the incorrect target response, the LRP onset of the incorrect response was positive-going and statistically reliable at 340 ms, whereas, on no-information and congruent trials, the LRP onset of the incorrect response was delayed by 40 ms and 50 ms, respectively. Multilevel Analyses of Covert Motor Activations on Performance Analyses of grand-averaged LRPs are based on covert motor activations aggregated across trials and across participants. The standard practice of multitrial aggregation allows for the possibility that prime-induced covert motor activations occur on a subset of trials (or a subset of participants), and behavioral priming effects occur on a different subset (or other participants). In contrast to standard subject-level aggregation analyses, MLM estimates, for each trial, the effect of covert motor activations on subsequent response. In the present research, MLM was used to examine the effect of covert motor activations on subsequent response, how covert motor activations interact with prime information to influence behavioral responding, and the strength of these associations as they unfold over time (see Method section for a description of the analyses). Predicting accuracy from trial-level covert motor activations. The level-2 model provides an estimate (g10 in equation 2.2) of the average effect of TCMA on error. Figure 3a plots the t values corresponding to g10, and reflects the predictive ability of each 60-ms TCMA interval over time for each trial type separately. A positive t value indicates that greater covert motor activations of the correct (vs. incorrect) response predicted greater accuracy. A negative t value indicates that greater covert motor activations of the correct response predicted decreased accuracy. A noticeable effect in Figure 3a is the difference in the predictive ability of covert motor activations on accuracy between

214

V. Zayas et al.

a

Predicting Accuracy from Trial-Level LRP prime onset

target onset

Effect of Trial-Level LRP on Accuracy (t)

10

congruent slope > incongruent & no-information slope (p < .05) congruent & incongruent slope > no-information slope (p < .05)

8 6 4 2 0 –2

no-information slope < 0 (p < .05)

–4 –6

0

75

b

300

600

900 [ms]

Predicting Reaction Time from Trial-Level LRP prime onset

Effect of Trial-Level LRP on Reaction Time (t)

congruent incongruent no-information

target onset

8

congruent & no-information slope > incongruent slope (p < .05)

6 4 2 0 –2

congruent incongruent no-information

–4 –6

incongruent slope < 0 (p < .05)

0

75

300

600

900 [ms]

Figure 3. The effect of trial-level covert motor activation (TCMA) on (a) accuracy and (b) response times for correctly classified targets, as a function of trial type (congruent, incongruent, and no-information). T values correspond to the level-2 estimate (g10), representing the average effect of TCMA on behavioral performance. T values are plotted across time post prime onset. A positive t value, reflecting a positive slope, indicates that covert motor activation associated with the correct (vs. incorrect) response predicted better (more accurate, faster) behavioral performance. A negative t value reflects a negative slope and indicates that covert motor activation associated with the correct (vs. incorrect) response predicted worse (less accurate, slower) behavioral performance. TCMA was computed by taking, for each trial, the difference between potentials at C3 minus potentials at C4, multiplying these differences by ! 1 for trials in which the target cued the left-hand, and averaging across 60-ms intervals that were shifted by 5 ms (220–279 ms, 225–284 ms, etc.). Each time interval of the trial-level LRP is referred to by its midpoint. Thus, in the figure above, the point associated with trial-level LRP at 220 ms corresponds to covert motor activation occurring between 190–249 ms.

no-information trials, on the one hand, and congruent and incongruent trials, on the other. Starting as early as 70 ms post prime onset, greater covert motor activations of the correct response predicted decreased accuracy on no-information trials, and no such pattern emerged on congruent and incongruent trials. Thus, whereas analyses of the subject-level grand-averaged LRPs indicated that the mean level of activation on no-information trials did not differ from zero until 340 ms post prime onset (see Figure 2a), the trial-level analyses indicated that the

variability of covert motor activations, which is centered around zero, is in fact meaningful. Furthermore, at approximately 105 ms post prime onset, trial-level covert motor activations of the correct response predicted greater accuracy on congruent trials than incongruent and no-information trials. These trial-level effects were observed even though analyses of the subject-level, grandaveraged LRPs indicated no significant covert motor activation during the same period of time (i.e., the grand-averaged LRP

Trial-level electrophysiological motor activations on congruent trials at approximately 105 ms did not differ significantly from zero). Finally, as time post prime onset elapsed (4300 ms post prime onset), trial-level covert motor activations of the correct response significantly predicted higher accuracy for all trial types. The onsets of these later occurring trial-level effects were 365 ms, 365 ms, and 340 ms for congruent, no-information, and incongruent trials, respectively. These onsets correspond approximately with the onsets for the grand-averaged LRPs (Figure 2a). Predicting reaction times from trial-level covert motor activations. The same MLM strategy (see MLM Analyses of TCMA Effects in the Method section) was used to predict RT on correctly classified trials. Figure 3b plots the t values corresponding to g10, and reflects the predictive ability of each 60-ms TCMA interval over time for each trial type separately. A positive t value indicates that greater covert motor activations of the correct (vs. incorrect) response predicted faster RTs. A negative t value indicates that greater covert motor activations of the correct response predicted slower RTs. A noticeable effect is the difference in predictive ability between incongruent trials, on the one hand, and congruent and no-information trials, on the other. Starting at 30 ms post prime onset, triallevel covert motor activations of the correct response was associated with slower RTs on incongruent trials. This pattern was not observed on congruent and no-information trials. Thus, even though analyses of the grand-averaged LRPs indicated no significant covert motor activation during the same period of time, the MLM analyses showed that variability in covert motor activations on incongruent trials significantly predicted RTs in classifying targets. In addition, as time post prime onset elapsed (4300 ms post prime onset), trial-level covert motor activations of the correct response predicted faster RTs for all trial types. The onsets of these effects were 320 ms for congruent and 310 ms for no-information trials and delayed by 70–80 ms on incongruent trials (390 ms). They also correspond approximately with the onsets for the grand-averaged LRPs (Figure 2b). Discussion Behavioral Priming Effects Analyses of behavioral responses showed that primes facilitated, as well as hindered, subsequent behavioral performance on the target classification task. Specifically, on trials in which the gender of the prime matched the gender of the target (congruent), classification of targets was both more accurate and faster, compared to trials in which primes provided no information about gender. Conversely, on trials in which the gender of the prime was opposite of the gender of the target (incongruent), classification of targets was both less accurate and slower, compared to trials with no-information primes. Thus, even though participants were instructed to classify targets and ignore primes, primes affected the ease with which participants were able to perform the subsequent target classification task. Evidence of Prime-Induced Covert Motor Activations: GrandAveraged LRPs Results from the standard subject-level analyses of the grandaveraged LRPs showed that masked primes triggered preparatory covert motor activations, consistent with past findings (Dehaene et al., 1998). Support for this claim is based on two findings: the initial direction of the LRP for congruent and incongruent trials

215 and the delayed LRP onset for no-information trials. As shown in Figure 2a, the LRP for congruent trials was initially negative (upward going), indicating greater covert preparatory motor activations of the correct target response. In contrast, the LRP for incongruent trials was initially positive (downward going), indicating greater preparatory motor activations of the incorrect target response. Moreover, compared to congruent and incongruent trials, the LRP onset for no-information trials was delayed by approximately 70 ms, which is approximately the same time interval by which targets followed primes. It is worth noting that in the present experiment prime-induced covert motor activations persisted long enough to overlap with motor activations triggered as a response to the target. Specifically, the time interval in which the LRP onset for noinformation trials became statistically significant is assumed to reflect the onset of activation triggered primarily by the target. During this time interval, congruent trials showed greater covert activations of the correct response, whereas incongruent trials showed greater covert activations of the incorrect response. Evidence of Spontaneous Covert Motor Activations: Multilevel Analyses of Covert Motor Activations on Performance Subject-level analyses do not unequivocally establish that preparatory covert motor activations, indexed by the LRP, and subsequent behavioral performance (accuracy, RT) occur in known sequence on individual trials. In contrast, MLM is appropriate for estimating trial-level effects of covert motor activations on subsequent behavioral performance, how these covert motor activations interact and depend on incoming information, as well as the strength of these effects over time. In some respects, the results from the MLM converged with the results from the grand-averaged LRP analyses. Specifically, MLM analyses revealed that for the three trial types covert motor activations of the correct response occurring approximately 300–400 ms post prime onset was associated with enhanced behavioral performance, as indexed by higher accuracy (Figure 3a) and faster RTs (Figure 3b). However, most important, the findings from the MLM analyses provide information that goes beyond the analyses of the grand-averaged LRP waveforms. With regard to the later occurring trial-level effects (4300 ms post prime onset), the MLM analyses indicate that within each trial type, spontaneous activations of the correct response predicted higher accuracy and faster RT. Whereas analyses of the grand-averaged LRPs focus on differences in covert motor activations between the trial types (e.g., activation of the correct response occurs earlier for congruent than no-information trials), the MLM analyses focus on within trial type variation. The results of MLM show that, even when controlling for differences across trial types, covert motor activation predicted accuracy and RT. To provide a more concrete illustration, grand-averaged LRPs showed an earlier onset of covert motor activations of the correct response on congruent trials (compared to no-information and incongruent trials). The trial-level analyses show that, among congruent trials, spontaneous covert motor activations of the correct response, occurring approximately 300–400 ms post prime onset, predicted faster reaction times and greater accuracy on the target classification task. Although past work (Gratton et al., 1988) has shown that prestimulus covert motor activations predict responses on ‘‘fast guess’’ trials in which responses occur within 150–199 ms, the present findings indicate that within trial variation of covert motor activations predicts responses on trials with longer RTs as well.

216 Moreover, the MLM analyses also revealed that fluctuations in covert motor activations early on in the stimulus stream (o150 ms post prime onset) interacted with, and depended on, the prime information encountered. As illustrated in Figure 3a, if early in the stimulus stream (o150 ms post prime onset), the motor circuits associated with the correct response were activated, (a) encountering a congruent prime was associated with increased accuracy in classifying the target, (b) encountering a no-information prime, which is not relevant to the target classification task, was associated with decreased accuracy, and (c) encountering an incongruent prime was relatively unassociated with subsequent accuracy. The findings that spontaneous covert motor activations interacted with prime information suggest a variant of response competition processes. More concretely, if covert motor activations occurring early in the stimulus stream are followed by information (primes) that reinforces the initial activations, then the initial activations are further enhanced. If covert motor activations are followed by information that is not consistent with, or opposite of, the initial activation, then the initial activation may not be enhanced, and may even be inhibited. These findings are consistent with models of human information processing that emphasize the partial accumulation of evidence over time (e.g., Osman et al., 1992) as well as models assuming continuous (versus discrete or all-or-none) patterns of response activations underlying overt behavioral responses and decisionmaking (e.g., Coles et al., 1985; Dale, Kehoe, & Spivey, 2007; Eriksen & Schultz, 1979; Eriksen, Coles, Morris, & O’Hara, 1985; McKinstry, Dale, & Spivey, 2008; Nosofsky & Palmeri, 1997). Most important, the findings of spontaneous fluctuations in covert motor activation occurring o150 ms post prime onset influencing later behavioral responses are not observed in the grand-averaged LRP waveforms (Figure 2a). That is, whereas the analyses of the aggregate subject-level grand-averaged LRP indicate that covert motor activations were not significantly different from zero in the periodo150 ms post prime onset, the trial-level covert motor activations, indexed by TCMA, indicate that variation in the activity is meaningful and influential. Further, highlighting the utility of applying MLM to complement the grand-average analyses, the MLM analyses, but not analyses of the grand-averaged LRPs, provide information about the role of covert motor activations in the behavioral priming effects observed in the RTs for correctly classified trials. Specifically, on correctly classified trials, priming effects emerged in RTs (i.e., slower RTs on incongruent than congruent and noinformation trials; see Table 1). Results from the MLM analyses suggest that covert motor activations may be playing a role in the slowing of RT for incongruent trials. As shown in Figure 3b, on incongruent trials, starting at 30 ms post prime onset, greater covert activations of the correct response predicted longer RTs. This suggests that, if early in the stimulus stream covert motor activations favor the correct target response, encountering incongruent information (prime) may inhibit these initial activations, leading to slower RTs in classifying targets. Most important, analyses of the grand-averaged LRPs for correctly classified incongruent trials did not reliably show activations of the incorrect response (i.e., at no point was the LRP significantly different from zero in the positive direction; Figure 2b). Outstanding Questions and Future Directions It is noteworthy that early (o150 ms post prime onset) trial-level covert motor fluctuations were differentially related to accuracy and RTs. For example, on incongruent trials, covert motor ac-

V. Zayas et al. tivations occurring early in the stimulus stream were relatively unassociated with subsequent accuracy (Figure 3a), but were negatively associated with RTs (Figure 3b). On no-information trials, early covert motor activations were inversely associated with accuracy (Figure 3a), but were not reliably associated with RTs (Figure 3b). Finally, on congruent trials, early covert motor activations (starting at 70 ms post prime onset) were related to greater accuracy (Figure 3a), but were not significantly related to RTs (Figure 3b). The source of these dissociations observed in the early occurring (o150 ms post prime onset) trial-level effects is unclear. One speculation based on observations of dissociations between reaction times and accuracy observed in other tasks, such as letter recognition (Santee & Egeth, 1982) and spatial cueing tasks (Prinzmetal, McCool, & Park, 2005), is that different neural and cognitive mechanisms operate at different stages of conflict resolution (Casey, Thomas, Welsh, Badgaiyan, Eccard, Jennings, & Crone, 2000). Moreover, a central aim of the present research was to investigate the role of spontaneous covert motor activations on subsequent behavioral responding. Although spontaneous covert motor activations were defined in the present experiment as covert motor activations not driven by an external stimulus (i.e., prime) and appearing to be more or less random, a valuable next step is to assess whether these variations may be predicted by the preceding response or expectations about the upcoming target. Nonetheless, irrespective of the sources influencing spontaneous fluctuations, we believe that their ability to predict subsequent behavior (accuracy and RT) and interactions with prime information are likely to extend to other tasks. Another promising avenue for future work is to assess the generalizability of these findings to other paradigms. As a final point, the present experiment shows that trial-level motor activations, both those occurring late in the stimulus stream (4300 ms post prime onset) as well as early in the stimulus stream (o150 ms post prime onset), are associated with the ease with which the target classification is performed. It is possible that the response window procedure used in the present experiment, which obliged participants to respond more rapidly than they would naturally be inclined to do, predisposes participants to use any cues, external and internal, available to them for making the target classification judgment within the response window. As such, participants may increase their reliance on not only the prime information (external cue), but also on differential levels of covert motor activation that predispose one motor movement over another (internal cue). Although we believe that the effects observed in the present research reflect a more general phenomenonFthat is, that fluctuations in covert motor activations influence the processing of incoming stimuli as well as subsequent behavioral responseFfuture research is needed to establish whether such effects would be reliably observed using priming paradigms that do not employ a response window.

Conclusions This research is the first to apply MLM statistical techniques to electrophysiological data to establish the role of unintentional covert motor activations on subsequent behavior. The trial-level analyses indicated that spontaneous covert motor activations (4300 ms post prime onset) of the correct response occur to a greater extent on trials in which targets are correctly classified, as well as on trials in which targets are classified more quickly.

Trial-level electrophysiological motor activations

217

Moreover, the present work identifies that early occurring spontaneous fluctuations in covert motor activations are a source of influence on behavioral performance. Trial-level analyses revealed that early (o150 ms post prime onset) covert motor activations interacted with, and depended on, prime information to predict subsequent accuracy and reaction times. Critically, these findings of spontaneous covert motor fluctuations would be otherwise unobserved with standard subject-level, multitrial-aggregation analyses.

The present findings are consistent with models assuming continuous (versus discrete or all-or-none) patterns of response activations underlying overt behavioral responses and decisionmaking (e.g., Coles et al., 1985; Dale et al., 2007; Eriksen & Schultz, 1979; Eriksen et al., 1985). Moreover, whereas past research has focused on how sensory information affects the motor system without conscious awareness, the present findings suggest that covert motor activations may impact the processing and effect of initial perceptual information.

REFERENCES Bolger, N., Zuckerman, A., & Kessler, R. C. (2000). Invisible support and adjustment to stress. Journal of Personality and Social Psychology, 79, 953–961. Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical Linear Models. Applications and Data Analysis Methods. Newbury Park, CA: Sage. Casey, B. J., Thomas, K. M., Welsh, T., Badgaiyan, R. D., Eccard, C. H., Jennings, J. R., & Crone, E. A. (2000). Dissociation of response conflict, attentional selection, and expectancy with functional magnetic resonance imaging. Proceedings of the National Academy of Sciences, 97, 8728–8733. Coles, M. G. H., Gratton, G., Bashore, T. R., Eriksen, C. W., & Donchin, E. (1985). A psychophysiological investigation of the continuous flow model of human information processing. Journal of Experimental Psychology: Human Perception and Performance, 11, 529–553. Coles, M. G. H. (1989). Modern mind-brain reading: Psychophysiology, physiology, and cognition. Psychophysiology, 26, 251–269. Dale, R., Kehoe, C., & Spivey, M. (2007). Graded motor responses in the time course of categorizing atypical exemplars. Memory and Cognition, 35, 15–28. Dehaene, S., Naccache, L., Le Clec’H, G., Koechlin, E., Mueller, M., Dehaene-Lambertz, G., et al. (1998). Imaging unconscious semantic priming. Nature, 395, 597–600. De Jong, R., Wierda, M., Mulder, G., & Mulder, L. J. M. (1988). Use of partial information in responding. Journal of Experimental Psychology: Human Perception and Performance, 14, 682–692. Draine, S. C., & Greenwald, A. G. (1998). Replicable unconscious semantic priming. Journal of Experimental Psychology: General, 127, 286–303. Eimer, M., & Schlaghecken, F. (1998). Effects of masked stimuli on motor activation: Behavioral and electrophysiological evidence. Journal of Experimental Psychology: Human Perception and Performance, 24, 1737–1747. Eimer, M., & Schlaghecken, F. (2003). Response facilitation and inhibition in subliminal priming. Biological Psychology, 64, 7–26. Eriksen, C. W., & Eriksen, B. A. (1979). Target redundancy in visual search: Do repetitions of the target within the display impair processing? Perception & Psychophysics, 26, 195–205. Eriksen, C. W., & Schultz, D. W. (1979). Information processing in visual search: A continuous flow conception and experimental results. Perception and Psychophysics, 25, 249–263. Eriksen, C. W., Coles, M. G. H., Morris, L. R., & O’Hara, W. P. (1985). An electromyographic examination of response competition. Bulletin of the Psychonomic Society, 23, 165–168. Fazio, R. H. (2001). On the automatic activation of associated evaluations: An overview. Cognition and Emotion, 15, 115–141. Ferguson, M., & Zayas, V. (2009). Automatic evaluation. Current Directions in Psychological Science, 18, 362–366. Gratton, G., Coles, M. G. H., Sirevaag, E. J., Eriksen, C. W., & Donchin, E. (1988). re- and poststimulus activation of response channels: A psychophysiological analysis. Journal of Experimental Psychology: Human Perception and Performance, 14, 331–344. Greenwald, A. G., Abrams, R. L., Naccache, L., & Dehaene, S. (2003). Long-term semantic memory versus contextual memory in unconscious number processing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 29, 235–247. Greenwald, A. G., Draine, S. C., & Abrams, R. L. (1996). Three cognitive markers of unconscious semantic activation. Science, 273, 1699–1702.

Inquisit 1.33 [Computer software]. (2003). Seattle, WA: Millisecond Software. Klinger, M. R., Burton, P. C., & Pitts, G. S. (2000). Mechanisms of priming I: Response competition, not spreading activation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 26, 441–445. McClelland, J. L. (1979). On the time relations of mental processes: An examination of systems of processes in cascade. Psychological Review, 86, 287–330. Miller, J. O. (1988). Discrete and continuous models of humans information processing: Theoretical distinctions and empirical results. Acta Psychologica, 67, 191–257. Miller, J. O., & Hackley, S. A. (1992). Electrophysiological evidence for temporal overlap among contingent mental processes. Journal of Experimental Psychology: General, 121, 195–209. McKinstry, C., Dale, R., & Spivey, M. (2008). Action dynamics reveal parallel competition in decision making. Psychological Science, 19, 22–24. Minelli, A., Marzi, C. A., & Girelli, M. (2007). Lateralized readiness potential elicited by undetected visual stimuli. Experimental Brain Research, 179, 683–690. Nosofsky, R. M., & Palmeri, T. J. (1997). An exemplar-based random walk model of speeded classification. Psychological Review, 104, 266– 300. Osman, A., Bashore, T. R., Coles, M. G. H., Donchin, E., & Meyer, D. E. (1988). Neuronal activity and information processing in motor control: From stages to continuous flow. Biological Psychology Special Issue, 26, 179–198. Osman, A., Bashore, T. R., Coles, M. G. H., Donchin, E., & Meyer, D. E. (1992). On the transmission of partial information: Inferences from movement-related brain potentials. Journal of Experimental Psychology: Human Perception and Performance, 18, 217–232. Praamstra, P., & Seiss, E. (2005). The neurophysiology of response competition: Motor cortex activation and inhibition following subliminal response priming. Journal of Cognitive Neuroscience, 17, 483–493. Prinzmetal, W., McCool, C., & Park, S. (2005). Attention: Reaction time and accuracy reveal different mechanisms. Journal of Experimental Psychology General, 134, 73–92. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd edition). Newbury Park, CA: Sage. Santee, J. L., & Egeth, H. E. (1982). Do reaction time and accuracy measure the same aspects of letter recognition? Journal of Experimental Psychological: Human Perception and Performance, 8, 489–501. Sternberg., S. (1969). The discovery of processing stages: Extensions of Donders’s method. In W. G. Koster (Ed.), Attention and performance II (pp 276–315). Amsterdam: North-Holland. Van Turennout, M., Hagoort, P., & Brown, C. M. (1998). Brain activation during speaking: From syntax to phonology in 40 milliseconds. Science, 280, 572–574. Zayas, V., & Shoda, Y. (2007). Predicting preferences for dating partners from past experiences of psychological abuse: Identifying the ‘psychological ingredients’ of situations. Personality and Social Psychology Bulletin, 33, 123–138. (Received August 20, 2009; Accepted March 9, 2010)


Neural response to action and reward prediction errors: Comparing the error-related negativity to behavioral errors and the feedback-related negativity to reward prediction violations

GEOFFREY F. POTTS,a LAURA E. MARTIN,b SIRI-MARIA KAMP,a and EMANUEL DONCHINa a

Department of Psychology, University of South Florida, Tampa, Florida, USA Hoglund Brain Imaging Center, University of Kansas Medical Center, Kansas City, Kansas, USA

b

Abstract The error-related negativity (ERN) is thought to index an anterior cingulate (ACC) behavioral monitoring system. The feedback ERN (FRN) is elicited to error feedback when the correct response is not known, but also when a choice outcome is suboptimal and to passive reward prediction violation, suggesting that the monitoring system may not be restricted to actions. This study used principal components analysis to show that the ERN consists of a single central component whereas the reward prediction violation FRN is comprised of central and prefrontal components. A prefrontal component is also present in action monitoring but occurs later, at the error positivity latency. This suggests that ACC monitors both actions and events for reward prediction error. Prefrontal cortex may update reward expectation based on the prediction violation with the latency difference due to differential processing time for motor and perceptual information. Descriptors: Error-related negativity, Feedback-related negativity, Behavior monitoring, Reward prediction violation, Anterior cingulate cortex

postdetection corrective process (Falkenstein et al., 1991; Gehring et al., 1993). Source analysis placed the ERN neural source in anterior cingulate cortex (ACC; Dehaene, Posner, & Tucker, 1994). Converging evidence from animal single unit recording (Gemba, Sasaki, & Brooks, 1986) and human hemodynamic neuroimaging (Carter et al., 1998) are consistent with ACC as a core part of a behavior monitoring neural network. The ACC is densely interconnected with premotor cortex and motivational areas of the brain including the amygdala, ventral tegmentum, and ventromedial prefrontal and orbitofrontal cortex (Denvinsky, Morrell, & Vogt, 1995; Paus, 2001), and thus is well situated to evaluate motor plans in the context of motivational goals. Holroyd and Coles (2002) presented a model of the behavior monitoring system and the ERN based on the reinforcement learning model of the ventral tegmental area (VTA; Schultz, Dayan, & Montague, 1997), explicitly linking behavior monitoring to the brain’s appetitive motivation system, the mesotelencephalic dopamine (DA) reward system, originating in the VTA with wide-ranging targets in the striatum, ventromedial prefrontal cortex, orbitofrontal cortex, and the ACC. On the basis of animal self-stimulation studies and assessment of neuron activity and neurotransmitter binding, it was originally theorized that this neural system signaled the presence of items with appetitive motivational value (for reviews, see Spanagel & Weiss,

Monitoring actions and their outcomes is a critical cognitive function for the effective generation of goal-directed behavior. In the absence of information about the impact of actions and choices on motivational goals an organism cannot modify its behavior for optimal performance. Given the importance of action and outcome monitoring, it seems reasonable that the nervous system would have a neural system dedicated to this function. In the early 1990s, Falkenstein, Hohnsbein, Hoormann, and Blanke (1991) and Gehring, Goss, Coles, Meyer, and Donchin (1993) independently reported an event-related potential (ERP) index of a neural behavioral monitoring system, labeled the error negativity (Ne) or error-related negativity (ERN). The ERN is a medial frontocentral ERP component with an onset at or just before the execution of an overt response, peaking about 100 ms postresponse, elicited when the response is erroneous (Falkenstein et al., 1991; Gehring et al., 1993). Initial hypotheses about the specific cognitive operation indexed by the ERN focused on the detection of a response error or some This study was funded by NIH grants DA14073, DA023273 (Potts), and DA018498 (Martin). Address correspondence to: Geoffrey Potts, PCD4118G, 4202 E. Fowler Ave.. Tampa, FL 33620, USA. E-mail: [email protected] 218

Response to action and reward prediction errors 1999; Wise & Rompre, 1989). Schultz et al. (1997) extended this model, reporting that the response of VTA neurons to reward could be conditioned. Using classical conditioning procedures, they showed that if an appetitive stimulus was presented in the absence of any predictive cue, VTA neurons showed enhanced firing, the classic reward response. However, if the appetitive stimulus was repeatedly paired with a previously neutral cue, after conditioning, the VTA neurons showed enhanced firing to the predictive cue but no longer to the appetitive stimulus itself. Also following conditioning, if the predictive cue was presented but the appetitive stimulus was then withheld, VTA neuron response was suppressed below baseline at the time the predicted reward would have been delivered (Schultz et al., 1997). Thus VTA neurons do not code reward itself; rather they code whether a delivered reward violates reward prediction, that is, if there is a reward prediction error, with enhanced firing for outcomes better than predicted and suppressed firing for outcomes worse than predicted. Holroyd and Coles (2002) proposed that the ERN reflects input from the DA reward prediction system to the ACC. When an action fails to achieve the expected motivational outcome, the DA reward system produces a reward prediction error signal that is transmitted to ACC, where it functions as a ‘‘learning signal,’’ biasing the motor planning and production system to acquire new associations. The reduced DA input releases the ACC from inhibition, resulting in increased activity and a scalp recorded ERN (Holroyd & Coles, 2002). There has been a sizable body of work examining the response of the monitoring system to explicit reward expectation violation using an ERP component related to the ERN, the feedback error related negativity (FRN; Miltner, Braun, & Coles, 1997; for review, see Nieuwenhuis, Holroyd, Mol, & Coles, 2004). As originally described, an FRN is elicited when the participant does not know the correct response when making the response, but is subsequently provided with performance feedback. When the participant does not know the correct response, then there is no ERN elicited by the action (e.g., the incorrect key press); rather there is an FRN elicited to the perceptual feedback informing the participant that her or his response was incorrect. The FRN appears to have the same neural source as the motor-related ERN, suggesting that they are the same component, elicited when the participant can determine the accuracy of the response (Miltner et al., 1997). As the participant learns the task, the error effect transfers from the feedback-locked FRN to the responselocked FRN, further supporting the unity of cognitive operation indexed by the ERN and FRN (Holroyd & Coles, 2002; Nieuwenhuis et al., 2002). Following the initial descriptions of the ERN and FRN as components elicited by action errors, numerous studies have expanded on the eliciting conditions required for an FRN. The original designs required feedback informing the participant that her or his response was an explicit error, that is, a response that violated task instructions, to elicit an FRN. Some subsequent designs have shown that feedback signaling suboptimal trial outcome is sufficient to elicit an FRN. Most of these designs have used some form of monetarily motivated choice option task in which monetary gain or loss on a given trial is the outcome of the participant’s choice (e.g., Gehring & Willoughby, 2002; Holroyd, Nieuwenhuis, et al., 2004; Yeung & Sanfey, 2004). In these gambling-like tasks, an FRN is elicited when the choice outcome is less than the best available, given the options. For example, if breaking even is the best available option (i.e., all the other outcomes are monetary losses), then breaking even would not elicit

219 an FRN; however, if the other options are all monetary gains, then breaking even would elicit an FRN (Holroyd, Larsen, & Cohen, 2004). To account for these results, the conception of the FRN has been expanded from indexing simple behavioral errors to indexing actions or choice outcomes that fail to meet motivational goals (Nieuwenhuis, Holroyd, et al., 2004). However, in contrast to the choice outcome evaluation hypothesis, several studies have now reported an FRN in the absence of any overt choice or motor behavior (Donkers, Nieuwenhuis, & van Boxtel, 2005; Martin & Potts, 2004; Potts, Martin, Burton, & Montague, 2006; Yeung, Holroyd, & Cohen, 2005). Some of these experiments have employed designs similar to the choice gambling tasks above, modified so that another player or the computer is making the choice, rather than the participant (Yeung et al., 2005). Other experiments have used slot-machine–like designs in which there is no action or choice involved; participants simply observe the stimuli signaling the trial’s monetary outcome for the participant (Donkers et al., 2005; Martin & Potts, 2004; Potts et al., 2006). An FRN is elicited in these designs in the absence of overt action or choice on the part of the participant, when the outcome is suboptimal or worse than expected. Elicitation of an FRN in the absence of an explicit response or choice lends support to the theory that the monitoring system receives input from the VTA reward prediction system, a system that does not require an explicit action to elicit a reward prediction error response, but not that the system monitors only actions and choice outcomes. In the single-unit studies of the reward system, an active choice or action is not required to elicit a response from VTA neurons; in those designs a stimulus predicts the reward (the conditioned stimulus), then the reward itself is either delivered or not (Schultz et al., 1997). Withholding the predicted reward is sufficient to elicit a VTA neuronal response; no action by the participant is required. Similarly, in the slot-machine-type FRN designs, the failure to deliver a predicted reward is sufficient to elicit an FRN (Donkers et al., 2005; Potts et al., 2006), suggesting that the neural system indexed by the FRN has a function extending beyond the monitoring of actions and choice outcomes. One difficulty in drawing conclusions about the functional relationships between the ERN and the FRN is that the eliciting contexts are different. Whereas the FRN can be examined during purely perceptual tasks, the ERN, as a response-locked component, cannot. The hypothesis that the stimulus- and responselocked ERNs index the same cognitive operation supported in the same neural system derive primarily from studies that elicited the FRN using feedback that signaled an explicit action error, which does not directly address the relationship between the ERN elicited by action error and the FRN elicited by reward prediction violation. Whereas Miltner et al. (1997) reported that the FRN had the same medial frontal scalp distribution as the ERN, Gehring and Willoughby (2004) directly compared the topography of the ERN elicited by behavioral errors in a flanker task with the FRN elicited by monetary losses in a gambling task and found that the FRN to monetary loss had a more anterior scalp distribution than the error ERN and concluded that the ERN and FRN could not both be due to a single common neural generator. However, that study did not attempt to address what those differences in the neural systems engaged by action errors and monetary loss might be. The current study investigated to what extent the same neural systems are engaged by response errors and by reward prediction violations by examining whether the ERNs elicited by action

220 errors and reward prediction violations are best described as a single distinct ERP component or if they are better described as one or more distinct subcomponents, that is, whether either the ERN or FRN has a subcomponent not shared by the other. We applied difference wave analysis to determine if the scalp topographies of the ERN and FRN were distinct when extracted from perceptual- and motor-related ERP activity and principal components analysis (PCA) to investigate whether the unique sources of variance contributing to those scalp distributions were the same or different between the action error-related ERN and the reward prediction violation-related FRN on data from 24 participants who participated in both a flanker task and a passive reward prediction design. Difference waves extract the effects distinct between the two conditions but cannot distinguish which condition contributes which effects to the difference wave and can confound amplitude and latency differences. PCA reduces the dimensionality of the data by extracting coherent sources of variance in the ERP across time, space, conditions, and participants (Donchin, 1966; Ruchkin, Villegas, & John, 1964; for review, see Donchin & Heffley, 1979). In the current data, an equivalent ERN and FRN difference wave scalp topography and the same PCA-derived factor structure would be consistent with the hypothesis that action errors and reward prediction violations both engage the same neural system. If the ERPs elicited by these two violations have distinctly different scalp topographies and are associated with different factor structures, we would conclude that errors and reward prediction violations activate different cognitive operations, implemented in different neural structures. Distinct topographies with partially overlapping factor structure would indicate that the neural systems and cognitive operations engaged by the two tasks share some subcomponent operations but also have additional operations only engaged by one of the tasks.

Methods Participants Twenty-four Rice undergraduate students (15 female, one lefthanded, mean age 20.38 years, SD 3.17) were paid for participation, and all participated in both experimental designs. Note that results from subsets of the participants in the reward prediction design have been previously reported by Potts et al. (2006) and, with fMRI results, by Martin, Potts, Burton, and Montague (2009); thus the results here do not constitute a replication of those findings. Experimental Designs Flanker task. The flanker design was modified from that in the study by Eriksen and Eriksen (1979). Stimuli were five-letter strings consisting of the letters P and R. Participants responded to the center letter in the string in a two-choice forced-alternative manner using the left index finger to respond to one letter and the right to the other (response hand to letter mapping was counterbalanced across participants). Stimulus strings were either congruent (all five letters the same) or incongruent (flanking letters did not match the center letter). A trial consisted of a fixation cross followed by an 800-ms warning asterisk followed by the stimulus string that remained onscreen for 100 ms. Participants had 800 ms to respond, and 1000 ms after the stimulus string a

G.F. Potts et al. feedback screen appeared informing the participant of the outcome of the trial. There were nine blocks of trials with 96 trials per block with stimulus type (P or R central stimulus, congruent or incongruent flankers) chosen randomly and equiprobably within each block, for a total of 214 trials of each stimulus combination. The design contained a motivational aspect in that the center R stimuli were potentially rewarding and the center P stimuli potentially punishing. Participants were given a cash reward for correct responses to Rs with no consequence for errors and lost money for incorrect responses to Ps with no reward for correct responses; thus Rs were potentially rewarding and Ps potentially punishing. Participants started each block with $5 in their bank; correct responses to Rs added $1 and incorrect responses subtracted $1 from the bank. At the end of the experiment, participants were paid their winnings on one of the blocks, chosen at random. Because all participants made more correct than error responses, all blocks were winning blocks; however, participants were told at the beginning of the experiment that they would leave with at least the $5 they started with regardless of performance. The punishment/reward motivation factor had no differential effect on the scalp topography or factor structure of the ERPs, so it will not be discussed further here. Reward prediction design. Stimuli were images of lemons and gold bars in an S1/S2 design in which S1 predicted S2 and S2 delivered or did not deliver a reward. A trial began with 300 ms of fixation followed by S1, an image of either a lemon or a gold bar, remaining on screen for 500 ms, followed by another 300 ms of fixation; then S2 was presented for 500 ms, then another 300 ms of fixation, then a text screen stating the monetary outcome of the trial and the participant’s current bankroll, which stayed on screen for 600 ms. S2 determined the reward amount. If S2 was a gold bar the participant won $1; if it was a lemon the participant won nothing. On 80% of the trials, S1 and S2 were identical, leading to four trial types in a 2 ! 2 design: Prediction (predicted, unpredicted) ! Reward (reward, no reward). There were eight blocks of trials, and participants started each block with $5. Each trial cost $0.25, and that cost was subtracted from and winnings were added to that $5. Each block consisted of 60 trials for 480 trials total. Although the overall predicted/ unpredicted ratio was fixed at 80/20, slight variation was introduced into each block, so the payoff for each block was slightly different. At the end of the experiments, participants drew a number from 1 to 8 and were paid for the outcome of that block. Data Acquisition and Signal Processing Data were acquired with a 128-channel Electrical Geodesics System 200 (EGI, Eugene, OR). Electroencephalogram (EEG) was acquired continuously, sampled at 250 Hz, with a vertex reference. The EEG was digitally filtered at 20 Hz lowpass and segmented from 100 ms preresponse to 400 ms postresponse in the flanker task and from 200 ms pre-S2 to 800 ms post-S2 in the reward prediction task. The segments were screened for noncephalic artifact, and the clean segments averaged into error and correct response conditions in the flanker task and into the predicted and unpredicted reward and no reward conditions in the reward prediction task. The subject averages were baseline corrected over the 100 ms preresponse (flanker) or 200 ms prestimulus (reward prediction) periods and re-referenced into an average reference representation. The subject averages were averaged together to create grand-average files.

Response to action and reward prediction errors Analyses Overt responses in the flanker task were analyzed by computing the percent correct responses for each participant in the congruent and incongruent conditions and averaging the individual reaction times for each participant separated by congruent and incongruent conditions and by correct and incorrect response. The percent correct was tested using a paired t test comparing the congruent and incongruent conditions, and the mean reaction times were analyzed in a repeated measures analysis of variance (ANOVA) with Condition (congruent, incongruent) and Response (correct, error) as factors. To determine if the designs elicited an ERN to error responses and an FRN to reward prediction violation, we extracted the mean amplitude from 0 to 150 ms after the response for the flanker task and from 200 to 350 ms after S2 onset in the reward prediction design from the frontocentral sensor net electrode closest to the frontocentral FCz location. The mean amplitude was analyzed using repeated measures ANOVAS, one ANOVA for each experimental design, with Response (error, correct) as the factor in the flanker design and Prediction (predicted, not predicted) and Reward (reward, no reward) as the factors in the reward prediction design. To examine the scalp distribution of the ERN elicited by errors and the FRN from reward prediction violations, separated from the experiment design-specific motor and stimulus ERP effects, we created difference waves in the flanker task (error minus correct) and the reward prediction design (unpredicted no reward [bar lemon] minus predicted reward [bar bar]). The subtraction in the reward prediction design confounds effects due to reward delivery and prediction violation but avoids potential confound due to differential prediction by holding S1 constant. Interpolated maps were created from these difference waves, and possible differences in scalp distribution were tested parametrically by extracting the difference wave ERN and FRN using the same windows described above (0–150 ms postresponse in the flanker design, 200–350 ms post-S2 in the reward prediction design) at the sensor net midline electrodes corresponding to FPz, Fz, FCz, Cz, CPz, and Pz and using a repeated measures ANOVA with Experiment Design (flanker, reward prediction) and Electrode (FPz, Fz, FCz, Cz, CPz, Pz) as factors. Effects or interactions with the Electrode factor (the only factor with more than two levels) were corrected for deviation from sphericity with the Greenhouse–Geisser epsilon. PCA The ERPs from 100 ms before to 450 ms after the motor response or feedback stimulus for each subject at all electrodes for both error and correct responses in the flanker task and for the predicted reward (bar–bar) and unpredicted no reward (bar–lemon) conditions were submitted to a spatiotemporal PCA (Dien & Frishkoff, 2004; Spencer, Dien, & Donchin, 2001) conducted separately for flanker and reward prediction designs (note that the predicted reward and unpredicted no reward conditions provide the ‘‘outcome worse than predicted’’ contrast relevant to the reward prediction violation theory). Details of the PCA procedures are described by Dien and Frishkoff (2004) and Spencer et al. (2001) as implemented in the Matlab toolbox provided by Dien (v. 1.23; Dien, 2010). Briefly, the input to the PCA for each task consisted of 326 observations (2 conditions ! 163 data points per epoch) for each participant and electrode. Spatial factors were initially extracted to reveal spatially coherent patterns of variance in the ERP, followed by a temporal PCA

221 performed on each spatial factor to examine when in time each spatial factor had coherent temporal structure. Covariance matrices and Promax rotations without the Kaiser correction option were used in the PCA. Paired sample t tests were conducted on the factor scores of each resultant temporal factor to test for differences between experimental conditions. Only those spatiotemporal factors whose factor scores were significantly different between conditions at a level of po.05 and that accounted for more than 3% of the variance in the total solution are discussed here. Results Behavioral Overall accuracy in the flanker task was 72% (so flanker task errors were slightly more frequent than unexpectedly withheld rewards in the reward prediction design). Participants were more accurate in the congruous (mean accuracy 84.1%, SD 13.5) than the incongruous condition (mean 59.7%, SD 11.7), t(23) 5 13.32, po.001. Participants were also faster in the congruous (mean RT 5 356 ms, SD 5 65) than the incongruous condition (mean 5 373 ms, SD 5 65), F(1,23) 5 20.38, po.001. There was no impact on RT of response accuracy. ERP The average waveforms across all participants (grand average) for the flanker (correct, error) and reward prediction (predicted reward [PR], unpredicted reward [UR], predicted no reward [PNR], unpredicted no reward [UNR]) designs at four midline electrodes in the sensor net approximately equivalent to Fz, FCz, Cz, and Pz are shown in Figure 1. Flanker design. The response-locked ERN was more negative on error responses than correct responses, F(1,23) 5 11.99, po.0001 (see Figure 1). Reward prediction design. The S2 stimulus-locked FRN was more negative to an S2 that signaled no reward (lemon) than to stimuli that signaled reward (gold bar), F(1,23) 5 26.44, po.0001. There was a Reward ! Prediction interaction, F(1,23) 5 7.20, po.05, indicating that FRN to the no reward stimuli was larger when that absent reward was unpredicted, that is, when a predicted reward was not delivered (see Figure 1). Difference Wave, Midline Electrodes There was a main effect for Design, F(1,23) 5 13.01, po.005, with the ERN in the flanker design larger than the reward prediction FRN. There was also a main effect for Electrode, F(5,115) 5 4.26, e 5 .375, po.05, with the E/FRN largest at FCz and falling off to the anterior and posterior. The Design ! Electrode interaction was significant, F(5,115) 5 12.15, e 5 .309, po.005, with the flanker ERN largest at FCz but the reward prediction FRN largest at Fz and FPz (see Figure 2), an interaction that was still significant after normalization to correct for the ‘‘misallocation of variance’’ problem (McCarthy & Wood, 1985), F(5,115) 5 11.18, e 5 .394, po.001. The amplitude of the error ERN and predicted reward not delivered FRN were not significantly correlated, r 5 .14, n.s., but the ERN and FRN differences were positively correlated r 5 .55, po.01. PCA Only the first three spatial factors extracted from each design contained temporal factors that varied significantly by condition

222

G.F. Potts et al. 10

6

8

4

6

2

4 0

2 Correct Error

0 –2 –100

0

100

200

300

400

PR PNR UR UNR

–2 –4 –100

0

100

ms

200

300

400

200

300

400

200

300

400

200

300

400

ms

10

6

8

4

6

2

4 0

2 Correct Error

0 –2 –100

0

100

200

300

400

PR PNR UR UNR

–2 –4 –100

0

100

ms

ms

10

6

8

4

6

2

4 0

2

–2 –100

0

100 200 ms

300

PR PNR UR UNR

–2

Correct Error

0

400

–4 –100

0

100 ms

10

6

8

4

6

2

4 0

2 0 –2 –100

0

100

200

300

PR PNR UR UNR

–2

Correct Error 400

ms

–4 –100

0

100 ms

Figure 1. Grand average ERP waveform plots from the midline electrodes closest to Fz, FCz, Cz, and Pz in the sensor net for the flanker design showing the correct (thin line) and error (thick line) responses with the ERN analysis window delimited and for the reward prediction design showing the predicted (P: dashed lines) and unpredicted (U: solid lines) reward (R: thin lines) and no reward (NR: thick lines) conditions with the FRN analysis window delimited.

and accounted for more than 3% of the variance, all with midline scalp distributions, one frontal, one central, and one posterior (see Figure 3, top row). The topographic distributions of the

spatial factors (i.e., the spatial factor loadings on the electrodes) were highly correlated between designs. The spatial loadings of central factor (Spatial Factor 1 of the flanker experiment and

Response to action and reward prediction errors Spatial Factor 2 of the reward prediction experiment) were highly correlated, r 5 .88, po.01, as were the frontal factor (Spatial Factor 2 in the flanker design and Spatial Factor 3 in the reward prediction design), r 5 .98, po.01, and the posterior factor (Spatial Factor 3 of the flanker experiment and Spatial

a

b

c

223 Factor 1 of the reward prediction experiment), r 5 .87, po.01 (see Figure 3, top row). The spatial factor scores were plotted across time points for each condition to create the ‘‘virtual ERPs’’ as described by Spencer et al. (2001) to illustrate the time course of each spatial factor in each condition and are presented in Figure 3, second row. The temporal factors extracted for each spatial factor that differed significantly between conditions are overplotted by condition in Figure 3, third and fourth rows (no temporal factor beyond Factor 6 differed between condition and accounted for more than 3% of the variance in the solution). In the flanker task, for the central factor (Spatial Factor 1), Temporal Factor 1 (SF1TF1) had its highest temporal factor loading and condition difference from 0 to 200 ms, peaking at 100 ms, then crossing over at 200 ms to a second peak just past 300 ms (see Figure 3, Column 1). This factor accounted for 18% of the variance in the solution and differed significantly between error and correct responses, t(24) 5 8.82, po.01. For the frontal factor (Spatial Factor 2), Temporal Factor 3 (SF2TF3) had its primary temporal factor loadings and condition difference rising through the preresponse period with an abrupt return to baseline immediately following the response (3% variance accounted for) and differed between error and correct responses, t(23) 5 3.52, po.01 (see Figure 3, Column 3). SF2TF4 also differed by response, t(23) 5 ! 3.34, po.01 (3% variance accounted for), with the temporal factor loadings largest between 200 and 400 ms postresponse, peaking at about 280 ms. For the posterior spatial factor (SF3), TF1 had an initial large temporal factor loading around response onset and a later positive loading from 200 to 400 ms, peaking at about 300 ms (7% variance accounted for) and differed between correct and error responses, t(23) 5 7.40, po.01 (see Figure 3, Column 5). In the reward prediction task, for the central factor (which was SF2 in this design), SF2TF1 accounted for 5% of the variance and differed between the predicted reward and unpredicted no reward conditions, t(23) 5 3.21, po.01, temporal factor loading occurring primarily after 350 ms (see Figure 3, Column 2). SF2TF2 also accounted for 5% of the variance and differed between the conditions, t(23) 5 4.61, po.01, with the temporal factor loading occurring between 200 and 300 ms poststimulus (see Figure 3, Column 2). For the frontal factor (SF3), TF2 accounted for 5% of the variance and differed between conditions, t(23) 5 4.80, po.01, with the factor loading largest from about 225 to 350 ms (see Figure 3, Column 4). For the posterior factor (SF1), TF2 accounted for 7% of the variance and differed between conditions, t(23) 5 2.59, po.05, mostly after 350 ms. SF1TF6 also differed between conditions, t(23) 5 2.56, po.05,

Figure 2. a: Maps of the scalp field topography of the error-minus-correct difference wave from the flanker design and the predicted reward-minusunpredicted no reward difference wave from the reward prediction design. The maps are oriented with the nose at the top of the figure, back of the head at the bottom and the vertex at the center. Darker is more negative. b: Difference wave waveform plots for six midline electrodes in the sensor net approximately equivalent to FPz, Fz, FCz, Cz, CPz, and Pz in the extended 10–20 system showing the electrodes where the behavioral error ERN in the flanker design (solid lines) or reward prediction violation FRN in the reward prediction design (dashed lines) are largest. c: Histrograph of the Design " Electrode interaction effect showing that the difference wave reward prediction violation FRN has a more anterior distribution than the behavioral error ERN.

Figure 3. Top row: Topographic maps of the first three spatial factors from the flanker and reward prediction designs. Second row: ‘‘Virtual ERPs,’’ showing the temporal course of spatial factors. Third and fourth rows: Temporal factor loadings on time by condition for the temporal factors for each spatial factor that differ significantly by condition.

224 G.F. Potts et al.

Response to action and reward prediction errors with the difference occurring primarily between 180 and 280 ms (see Figure 3, Column 6). Discussion This study addressed the cognitive function of the purported behavior monitoring system located in the ACC and indexed by the ERN (Dehaene et al., 1994; Falkenstein et al., 1991; Gehring et al., 1993) by comparing the component structure of the behavioral error elicited ERN with the reward prediction violation elicited FRN. To the extent that the response-related ERN and the feedback stimulus-related FRN index activity in the same neural system performing the same cognitive operation, contrasting the eliciting conditions for the ERN and FRN can help define the actual cognitive operation performed by this monitoring system, whether it is monitoring for explicit behavioral errors, the outcomes of choices, or some more general monitoring function. The current results indicate that the ERN and FRN share a common central factor, likely the ACC-generated component (Dehaene et al., 1994), but that the FRN contains a second, more anterior factor, perhaps indexing a more frontal medial component. Elicitation of the central component by reward prediction violation lends support to the theory that the error negativity (‘‘error negativity’’ is used here as a general term to describe the shared aspects of the ERN and FRN) is dependent on input from the VTA reward prediction system, but elicitation of the component in the absence of explicit action or choice demonstrates that the motor system involvement or overt choice evaluation are not required to engage the cognitive operation indexed by the error negativity. Both the error minus correct difference ERN and the predicted reward minus unpredicted no reward difference FRN had medial frontal scalp distributions, but the FRN was more anterior than the ERN (see Figure 2), consistent with Gehring and Willoughby (2004), indicating either a more anterior source for the FRN than the ERN or of partially nonoverlapping multiplesource configurations for the two components with one FRN source more anterior. The PCA decomposition results indicate the latter, showing that both behavioral errors and reward prediction violations share a medial central error negativity component, a component that fully accounted for the behavioral error-elicited ERN, but that the reward prediction violation FRN contains an additional coincident prefrontal subcomponent accounting for its more anterior scalp distribution. The PCA of the flanker data extracted a single factor (SF1TF1) that had the temporal, spatial, and response characteristics of the ERN: a temporal distribution between 0 and 200 ms peaking at about 100 ms postresponse, a medial central scalp distribution, and which differed significantly between correct and error responses. This factor accounted for six times as much variance as any other factor that varied by condition in the solution (18 % vs. 3%), suggesting that it reflects the primary ERP index of errors in the flanker task. The ‘‘virtual ERP’’ for the central spatial factor showed a clear correspondence with the ERN waveform (see Figures 1 and 3). Although two other factors had partial temporal overlap with this factor (SF2TF3 and SF3TF1), neither had the same time course as the ERN, indicating that this single medial central component fully accounts for the error negativity elicited to behavioral errors. This component appears to account for the classic ERN, indexing activity in the purported behavior monitoring system in the ACC (Dehaene et al., 1994).

225 The PCA of the reward prediction violation ERP also extracted a factor with a medial central spatial topography with the temporal distribution of the FRN (SF2TF2), from 250 to 350 ms, peaking at about 300 ms, that differed significantly between when rewards were predictably delivered and when rewards were unexpectedly withheld. This factor had the same spatial distribution as the ERN factor (SF1TF1) from the flanker task, suggesting that it reflects the same ‘‘classic ERN’’ component, the ERP index of error detection generated in the ACC (Dehaene et al., 1994; Falkenstein et al., 1991; Gehring et al., 1993). The significant correlation across participants of the difference wave ERN and FRN supports the idea of some commonality between the components. In the reward prediction design, however, this error negativity was elicited by the unexpected withholding of a predicted reward in the absence of any action or choice by the participant. If this ERP component reflects the same cognitive operation in the same neural system indexed by the response-locked ERN, then that system’s function cannot be confined to response conflict mediation (Gehring & Fencsik, 2001), behavioral error detection (Falkenstein et al., 1991; Falkenstein, Hoormann, Christ, & Hohnsbein, 2000), or choice outcome monitoring (Hajcak, Moser, Holroyd, & Simons, 2006; Nieuwenhuis, Yeung, Holroyd, Schurger, & Cohen, 2004) because there were no competing response options, behavioral errors, or alternative choices in the reward prediction design. However, if the concept of ‘‘error’’ is expanded beyond the motor domain to include incorrect predictions about the availability of motivationally relevant items in the environment, then this monitoring system may be conceived as performing a more generalized error monitoring function: monitoring for errors of motivational outcome prediction, applied to actions, choices, or environmental events. In addition to the central factor, the reward prediction violation ERP contained a frontal factor that also had the same time course as the FRN, SF3TF2, indicating that the reward prediction violation FRN is comprised of two subcomponents: one frontal and one central. The frontal factor, like the central factor, distinguished between predicted delivered rewards and when a reward was unexpectedly withheld. Although inferring ERP source localization from scalp topography is speculative, the medial anterior inferior scalp distribution of the frontal component suggests that it may emanate from more anterior portions of medial prefrontal cortex, perhaps ventromedial prefrontal cortex (VMPFC). VMPFC, like ACC, receives projections from the VTA and appears to have an evaluative function in the reward system, integrating reward prediction violation information with information about the environmental context and actions taken, updating estimates of the potential reward values of the current environment and of the current behavioral strategy when a delivered reward exceeds or fails to meet expectation (for reviews, see Bechara, Damasio, & Damasio, 2000; Rolls, 2000). Reward prediction violation has been shown to engage VMPFC using fMRI (Knutson & Cooper, 2005), single-unit recording in monkeys (Schoenbaum, Chiba, & Gallagher, 1998), and depth recording in a human patient (Oya et al., 2005), and patients with damage to VMPFC are unable to use reward prediction violation information to guide future decisions (Bechara, Damasio, Damasio, & Anderson, 1994), indicating the role or VMPFC in the use of motivation information in the formation of strategic models of the environment. Thus, in the reward prediction design, there may be simultaneous detection of an event that fails to meet motivational goals, indexed by the central factor, and an updating of those goal representations, indexed by the frontal factor.

226 There was a frontal factor in the flanker ERP as well that differed between error and correct responses, SF2TF4; however, its temporal distribution was later than the central factor, between 200 and 400 ms postresponse and peaking about 280 ms postresponse, suggesting that the prefrontal evaluative/integrative system was engaged in the flanker task, just later than the central detection system. This later medial frontal factor is in the temporal range of the error positivity (Pe; see Figure 3, Column 3), which follows the ERN and is more positive on error trials (Falkenstein, Hohnsbein, Hoormann, & Blanke, 1990; for reviews, see Arbel & Donchin, 2009; Overbeek, Nieuwenhuis, & Ridderinkhof, 2005). The cognitive operation indexed by the Pe has received less study than the ERN; however, one conception holds that the Pe reflects a more conscious reflection on an error and its consequences, in contrast to the ERN, which reflects early detection (Nieuwenhuis, Ridderinkhof, Blom, Band, & Kok, 2001; but see also Falkenstein et al., 2000). Some researchers have suggested that the Pe is simply a P300 to the relatively rare error events (Davies, Segalowitz, Dywan, & Pailing, 2001), whereas others claim that the Pe appears dissociable into anterior and posterior subcomponents with only the posterior subcomponent reflecting a P300, whereas the anterior subcomponent is more related to the ERN (Arbel & Donchin, 2009). In the current data, the SF2TF4 factor may reflect the more anterior subcomponent of the Pe seen by Arbel and Donchin, perhaps reflecting the same evaluative/integrative function suggested for the frontal subcomponent of the reward prediction violation FRN. The latency difference between the central and frontal components following behaviorally (sequential) and environmentally (simultaneous) signaled errors may be due to substantial error information being available earlier in processing in behavioral errors, in which the motor program is prepared in advance of actual execution, and stimulus representations, which can only be formed after stimulus presentation. There were several factors in the flanker and reward prediction violation ERPs outside the temporal range of the ERN and FRN that differed by condition. The current study was not designed to elicit effects outside of the ERN and FRN temporal range, so the following interpretations are post hoc speculations. A P300 subcomponent of the Pe may be reflected in the current flanker data by the posterior factor SF3TF1 that had a late difference between the error and correct responses. There was also another frontal factor in the flanker task that differed between error and correct responses, SF2TF3, that had most of its temporal distribution prior to the response. This factor may represent motor preparatory activity that differs between preparation or an error and correct response. Premotor cortex can hold multiple competing motor programs prior to execution choice (Cisek & Kalaska, 2002), and the to-be-executed representation gets enhanced and the others suppressed at decision (Cisek & Kalaska, 2005). There are ERP components in the preresponse period that may reflect this preresponse neural activity, including the readiness potential or lateralized readiness potential, a component beginning as early as 800 ms prior to the actual response, reflecting motor program preparation, and a bilateral premotor potential possibly related to motor initiation (Deecke, Gro¨zinger, & Kornhuber, 1976; Kornhuber & Deecke, 1965; Kutas & Donchin, 1980). Gratton, Coles, Sirevaag, Eriksen, and Donchin (1988) demonstrated that response correctness was predicted by premotor ERP activity. An LRP indicating the

G.F. Potts et al. to-be-executed response appeared prior to the appearance of the imperative stimulus; thus, on trials where the LRP was ipsilateral to the correct response indicated by the imperative stimulus, the response would be an error (Gratton et al., 1988). The preresponse differential between errors and correct responses in this frontal factor may reflect an ERP index of these motor activation/inhibition processes. In the reward prediction violation ERP there was a central factor, SF2TF1, that appeared to have most of its condition difference at the end of the epoch, perhaps reflecting a P300 response to the unpredicted stimuli (see Figure 3), although it is unclear why the flanker P300 would be more posterior than the reward prediction P300. There were two posterior factors in the reward prediction violation ERP that differed by condition, SF1TF2 and SF1TF6, but both were earlier than the P300. SF1TF2 differed by condition at about 180 ms poststimulus, and SF2TF6 varied between condition peaking at about 220 ms, perhaps reflecting N1 and P2 indices of differential attention to the unexpected stimulus (Eimer, Holmes, & McGlone, 2003; Johannes, Mu¨nte, Heinze, & Mangun, 1995; Mangun, 1995). Conclusion The current study indicates that the ERN elicited by behavioral errors and FRN elicited by reward prediction violations consist of partially overlapping components: a medial central component and a medial frontal component. For behavioral errors, the central component is the only one present in the temporal range of the ERN, indicating that the ERN consists of a single component, likely indexing the ACC based error detection system (Dehaene et al., 1994; Falkenstein et al., 1991; Gehring et al., 1993). However, because this same factor is also present in reward prediction violations, in the absence of explicit motor response or overt choice, the neurocognitive system indexed by this component cannot be linked solely to behavior or choice, as the dominant theories of the ERN and ACC function posit. Rather the system appears to be engaged by a more general class of prediction errors, errors that include actions, choice outcomes, and environmental events that fail to meet motivational goals. In its response to deviance from motivational expectation, the ACC monitoring system, indexed by the ERN, may join the general class of neural deviance responses, from simple sensory mismatch indexed by the mismatch negativity (Naä¨ta¨nen, Simpson, & Loveless, 1982), semantic expectation violation indexed by the N400 (Kutas & Hillyard, 1980), and violation of contextually constrained stimulus expectation indexed by the P300 (Donchin, 1981). It should be noted that the reward prediction violation FRN elicited here, with its frontal subcomponent, may not be representative of all feedback error-related negativities. In the original (Miltner et al., 1997) description of the FRN to action error feedback, the FRN had the same scalp distribution as the response-locked ERN. It is the FRN to monetary loss that has the more frontal distribution than the ERN, as shown in the current data and by Gehring and Willoughby (2004). Thus the FRNs to performance error feedback and reward prediction violation signal may not be equivalent; the inferior prefrontal subcomponent may only be present in feedback linked to an explicit reward. A study examining the component structure of reward related and performance related FRNs is needed to address that question.

Response to action and reward prediction errors

227 REFERENCES

Arbel, Y., & Donchin, E. (2009). Parsing the componential structure of post error ERPs: A principal component analysis of ERPs following errors. Psychophysiology, 46, 1179–1189. Bechara, A., Damasio, A. R., Damasio, H., & Anderson, S. W. (1994). Insensitivity to future consequences following damage to human prefrontal cortex. Cognition, 50, 7–15. Bechara, A., Damasio, H., & Damasio, A. R. (2000). Emotion, decision making and the orbitofrontal cortex. Cerebral Cortex, 10, 295–307. Carter, C. S., Braver, T. S., Barch, D. M., Botvinick, M. M., Noll, D., & Cohen, J. D. (1998). Anterior cingulate cortex, error detection, and the online monitoring of performance. Science, 280, 747–749. Cisek, P., & Kalaska, J. F. (2002). Simultaneous encoding of multiple potential reach directions in dorsal premotor cortex. Journal of Neurophysiology, 87, 1149–1154. Cisek, P., & Kalaska, J. F. (2005). Neural correlates of reaching decisions in dorsal premotor cortex: Specification of multiple direction choices and final selection of action. Neuron, 45, 801–814. Davies, P. L., Segalowitz, S. J., Dywan, J., & Pailing, P. E. (2001). Errornegativity and positivity as they relate to other ERP indices of attentional control and stimulus processing. Biological Psychology, 56, 191–206. Deecke, L., Gro¨zinger, B., & Kornhuber, H. H. (1976). Voluntary finger movement in man: Cerebral potentials and theory. Biological Cybernetics, 23, 99–119. Dehaene, S., Posner, M. I., & Tucker, D. M. (1994). Localization of a neural system for error detection and compensation. Psychological Science, 5, 303–305. Denvinsky, O., Morrell, M. J., & Vogt, B. A. (1995). Contributions of anterior cingulate cortex to behaviour. Brain, 118, 279–306. Dien, J. (2010). The ERP PCA Toolkit: An open source program for advanced statistical analysis of event-related potential data. Journal of Neuroscience Methods, 187, 138–145. Dien, J., & Frishkoff, G. A. (2004). Principal components analysis of ERP data. In T. C. Handy (Ed.), Event-related potentials: A methods handbook (pp. 184–208). Cambridge, MA: MIT Press. Donchin, E. (1966). A multivariate approach to the analysis of average evoked potentials. IEEE Transactions on Bio-Medical Engineering, BME-13, 131–139. Donchin, E. (1981). Surprise! . . . Surprise? Psychophysiology, 18, 493–513. Donchin, E., & Heffley, E. (1979). Multivariate analysis of event-related potential data: A tutorial review. In D. Otto (Ed.), Multidisciplinary perspectives in event-related potential research (pp. 555–572). Washington, DC: U.S. Government Printing Office. Donkers, F. C. L., Nieuwenhuis, S., & van Boxtel, G. J. M. (2005). Mediofrontal negativities in the absence of responding. Cognitive Brain Research, 25, 777–787. Eimer, M., Holmes, A., & McGlone, F. P. (2003). The role of spatial attention in the processing of facial expression: An ERP study of rapid brain responses to six basic emotions. Cognitive, Affective & Behavioral Neuroscience, 3, 97–110. Eriksen, C., & Eriksen, B. (1979). Target redundancey in visual search: Do repetitions of the target within the display impair processing? Perception and Psychophysics, 26, 195–205. Falkenstein, M., Hohnsbein, J., Hoormann, J., & Blanke, L. (1990). Effects of errors in choice reaction tasks on the ERP under focused and divided attention. In C. H. M. Brunia, A. W. K. Gaillard, & A. Kok (Eds.), Psychophysiological brain research (pp. 192–195). Tilburg, Germany: Tilburg University Press. Falkenstein, M., Hohnsbein, J., Hoormann, J., & Blanke, L. (1991). Effects of crossmodal divided attention on late ERP components. II. Error processing in choice reaction tasks. Electroencephalography & Clinical Neurophysiology, 78, 447–455. Falkenstein, M., Hoormann, J., Christ, S., & Hohnsbein, J. (2000). ERP components on reaction errors and their functional significance: A tutorial. Biological Psychology, 51, 87–107. Gehring, W. J., & Fencsik, D. E. (2001). Functions of the medial frontal cortex in the processing of conflict and errors. Journal of Neuroscience, 21, 9430–9437. Gehring, W. J., Goss, B., Coles, M. G., Meyer, D. E., & Donchin, E. (1993). A neural system for error detection and compensation. Psychological Science, 4, 385–390. Gehring, W. J., & Willoughby, A. R. (2002). The medial frontal cortex and the rapid processing of monetary gains and losses. Science, 295, 2279–2282.

Gehring, W. J., & Willoughby, A. R. (2004). Are all medial frontal negativities created equal? Toward a richer empirical basis for theories of action monitoring. In M. Ullsperger & M. Falkenstein (Eds.), Errors, conflicts, and the brain. Current opinions on performance monitoring (pp. 14–20). Leipzig, Germany: Max Planck Institute of Cognitive Neuroscience. Gemba, H., Sasaki, K., & Brooks, V. (1986). Error potentials in limbic cortex (anterior cingulate area 24) of monkeys during motor learning. Neuroscience Letters, 70, 223–227. Gratton, G., Coles, M. G. H., Sirevaag, E. J., Eriksen, C. W., & Donchin, E. (1988). Pre- and poststimulus activation of response channels: A psychophysiological analysis. Journal of Experimental Psychology: Human Perception and Performance, 14, 331–344. Hajcak, G., Moser, J. S., Holroyd, C. B., & Simons, R. F. (2006). The feedback-related negativity reflects the binary evaluation of good versus bad outcomes. Biological Psychology, 71, 148–154. Holroyd, C. B., & Coles, M. G. (2002). The neural basis of human error processing: Reinforcement learning, dopamine, and the error-related negativity. Psychological Review, 109, 679–709. Holroyd, C. B., Larsen, J. T., & Cohen, J. D. (2004). Context dependence of the event-related brain potential associated with reward and punishment. Psychophysiology, 41, 245–253. Holroyd, C. B., Nieuwenhuis, S., Yeung, N., Nystrom, L., Mars, R. B., Coles, M. G. H., et al. (2004). Dorsal anterior cingulate cortex shows fMRI response to internal and external error signals. Nature Neuroscience, 7, 497–498. Johannes, S., Mu¨nte, T. F., Heinze, H. J., & Mangun, G. R. (1995). Luminance and spatial attention effects on early visual processing. Cognitive Brain Research, 2, 189–205. Knutson, B., & Cooper, J. C. (2005). Functional magnetic resonance imaging of reward prediction. Current Opinion in Neurology, 18, 411– 417. Kornhuber, H., & Deecke, L. (1965). [Changes in the brain potential in voluntary movements and passive movements in man: Readiness potential and reafferent potentials.] (in German). Pflu¨gers Archiv fu¨r die gesamte Physiologie des Menschen und der Tiere, 10, 1–17. Kutas, M., & Donchin, E. (1980). Preparation to respond as manifested by movement-related brain potentials. Brain Research, 202, 95–115. Kutas, M., & Hillyard, S. A. (1980). Reading senseless sentences: Brain potentials reflect semantic incongruity. Science, 207, 203–205. Mangun, G. R. (1995). Neural mechanisms of visual selective attention. Psychophysiology, 32, 4–18. Martin, L., Potts, G., Burton, P., & Montague, P. (2009). Electrophysiological and hemodynamic responses to reward prediction violation. NeuroReport, 20, 1140–1143. Martin, L. E., & Potts, G. F. (2004). Reward sensitivity in impulsivity. NeuroReport, 15, 1519–1522. McCarthy, G., & Wood, C. C. (1985). Scalp distributions of eventrelated potentials: An ambiguity associated with analysis of variance models. Electroencephalography and Clinical Neurophysiology, 62, 203–208. Miltner, W. H. R., Braun, C. H., & Coles, M. G. H. (1997). Eventrelated brain potentials following incorrect feedback in a time-estimation task: Evidence for a ‘‘generic’’ neural system for error detection. Journal of Cognitive Neuroscience, 9, 788–798. Naä¨ta¨nen, R., Simpson, M., & Loveless, N. E. (1982). Stimulus deviance and evoked potentials. Biological Psychology, 14, 53–98. Nieuwenhuis, S., Holroyd, C. B., Mol, N., & Coles, M. G. H. (2004). Reinforcement-related brain potentials from medial frontal cortex: Origins and functional significance. Neuroscience & Biobehavioral Reviews, 28, 441–448. Nieuwenhuis, S., Ridderinkhof, K., Blom, J., Band, G. P., & Kok, A. (2001). Error-related brain potentials are differentially related to awareness of response errors: Evidence from an antisaccade task. Psychophysiology, 38, 752–760. Nieuwenhuis, S., Ridderinkhof, K., Talsma, D., Coles, M. G., Holroyd, C. B., Kok, A., et al. (2002). A computational account of altered error processing in older age: Dopamine and the error-related negativity. Cognitive, Affective & Behavioral Neuroscience, 2, 19–36. Nieuwenhuis, S., Yeung, N., Holroyd, C. B., Schurger, A., & Cohen, J. D. (2004). Sensitivity of electrophysiological activity from medial frontal cortex to utilitarian and performance feedback. Cerebral Cortex, 14, 741–747.

228 Overbeek, T. J. M., Nieuwenhuis, S., & Ridderinkhof, K. R. (2005). Dissociable components of error processing: On the functional significance of the Pe vis-a`-vis the ERN/Ne. Journal of Psychophysiology, 19, 319–329. Oya, H., Adolphs, R., Kawasaki, H., Bechara, A., Damasio, A., & Howard, M. A. (2005). Electrophysiological correlates of reward prediction error recorded in the human prefrontal cortex. Proceedings of the National Academy of Sciences, USA, 102, 8351–8356. Paus, T. (2001). Primate anterior cingulate cortex: Where motor control, drive and cognition interface. Nature Reviews Neuroscience, 2, 417– 424. Potts, G., Martin, L., Burton, P., & Montague, P. (2006). When things are better or worse than expected: Medial frontal cortex and the allocation of processing resources. Journal of Cognitive Neuroscience, 18, 1–8. Rolls, E. T. (2000). The orbitofrontal cortex and reward. Cerebral Cortex, 10, 284–294. Ruchkin, D. S., Villegas, J., & John, E. R. (1964). An analysis of average evoked potentials making use of least mean square techniques. Annals of the New York Academy of Sciences, 115, 799–826.

G.F. Potts et al. Schoenbaum, G., Chiba, A. A., & Gallagher, M. (1998). Orbitofrontal cortex and basolateral amygdala encode expected outcomes during learning. Nature Neuroscience, 1, 155–159. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275, 1593–1599. Spanagel, R., & Weiss, F. (1999). The dopamine hypothesis of reward: Past and current status. Trends in Neurosciences, 22, 521–527. Spencer, K. D., Dien, J., & Donchin, E. (2001). Spatiotemporal analysis of the late ERP responses to deviant stimuli. Psychophysiology, 38, 343–358. Wise, R., & Rompre, P. (1989). Brain dopamine and reward. Annual Review of Psychology, 40, 191–225. Yeung, N., Holroyd, C. B., & Cohen, J. D. (2005). ERP correlates of feedback and reward processing in the presence and absence of response choice. Cerebral Cortex, 15, 535–544. Yeung, N., & Sanfey, A. G. (2004). Independent coding of reward magnitude and valence in the human brain. Journal of Neuroscience, 24, 6258–6264. (Received November 5, 2009; Accepted February 1, 2010)


ADJUST: An automatic EEG artifact detector based on the joint use of spatial and temporal features

ANDREA MOGNON,a,b JORGE JOVICICH,a LORENZO BRUZZONE,c and MARCO BUIATTIa,d,e,f a Functional NeuroImaging Laboratory, Center for Mind/Brain Sciences, Department of Cognitive and Education Sciences, University of Trento, Trento, Italy b NILab, Neuroinformatics Laboratory, Fondazione Bruno Kessler, Trento, Italy c Department of Information Engineering and Computer Science, University of Trento, Trento, Italy d INSERM, U992, Cognitive Neuroimaging Unit, Gif/Yvette, France e CEA, DSV/I2BM, NeuroSpin Center, Gif/Yvette, France f Universite´ Paris-Sud, Cognitive Neuroimaging Unit, Gif/Yvette, France

Abstract A successful method for removing artifacts from electroencephalogram (EEG) recordings is Independent Component Analysis (ICA), but its implementation remains largely user-dependent. Here, we propose a completely automatic algorithm (ADJUST) that identifies artifacted independent components by combining stereotyped artifact-specific spatial and temporal features. Features were optimized to capture blinks, eye movements, and generic discontinuities on a feature selection dataset. Validation on a totally different EEG dataset shows that (1) ADJUST’s classification of independent components largely matches a manual one by experts (agreement on 95.2% of the data variance), and (2) Removal of the artifacted components detected by ADJUST leads to neat reconstruction of visual and auditory eventrelated potentials from heavily artifacted data. These results demonstrate that ADJUST provides a fast, efficient, and automatic way to use ICA for artifact removal. Descriptors: Electroencephalography, Independent component analysis, EEG artifacts, EEG artefacts, Event-related potentials, Ongoing brain activity, Automatic classification, Thresholding

electrodes throughout the scalp. Artifact-free ERPs are then obtained by averaging the data over the remaining epochs, thereby increasing the signal-to-noise ratio. However, this procedure is problematic when only a few epochs are available, or when artifacts are very frequent, as in studies involving patients or children. Moreover, it is inapplicable to studies focusing on slow non-event-locked activity arising from the continuous EEG (e.g., slow brain oscillations (Vanhatalo, Palva, Holmes, Miller, Voipio, & Kaila, 2004) or long-range temporal correlations (Linkenkaer-Hansen, Nikouline, Palva, & Ilmoniemi, 2001)). Alternative procedures consist of modelling the signals generated by blinks or ocular movements and removing them from the data while preserving the remaining activity. Most of these methods are based on regressing out reference signals, usually recorded near the eyes, from the EEG signals with a model of artifact propagation, either in the time domain (Gratton, Coles, & Donchin, 1983; Kenemans, Molenaar, Verbaten, & Slangen, 1991; Verleger, Gasser, & Mocks, 1982) or in the frequency domain (Gasser, Sroka, & Mocks, 1985; Woestenburg, Verbaten, & Slangen, 1983). However, because EEG and ocular activity mix bidirectionally (Oster & Stern, 1980; Peters, 1967), regressing out eye artifacts inevitably involves subtracting relevant neural signals from each recording as well as ocular activity (Croft & Barry, 2002; Jung, Makeig, Westerfield, Townsend, Courchesne,

Due to its excellent temporal resolution, electroencephalography (EEG) is a widely used experimental technique to investigate human brain function by tracking the spatio-temporal neural dynamics correlated to experimentally manipulated events (Niedermeyer & da Silva, 2005). However, a major problem common to all EEG studies is that the activity due to artifacts has typically much higher amplitude than the one generated by neural sources. Artifacts may have a physiological origin as eye movements or muscle contractions, or non-biological causes as electrode high-impedance or electric devices interference (Croft & Barry, 2000). In a typical event-related potential (ERP) paradigm, data are divided in epochs time-locked to the stimulus, and artifacts are removed by discarding epochs in which the EEG activity exceeds some predefined thresholds either in specific electrodes (e.g., electrooculogram (EOG) signals for ocular movements) or in all We thank Mariano Sigman and Stanislas Dehaene for sharing the EEG data, Francesca Bovolo and Michele Dalponte for helpful advice on the use of the thresholding algorithm, and Sara Assecondi for valuable comments on an earlier version of the manuscript. Address correspondence to: Marco Buiatti, CEA/DSV/I2BM/ NeuroSpin, INSERM U992FCognitive Neuroimaging Unit, Baˆt 145FPoint Courrier 156, Gif sur Yvette F-91191, France. E-mail: [email protected] 229

230 & Sejnowski, 2000). Moreover, these methods do not work without reference signals, which are not always present for ocular movements, and very difficult to obtain for other types of artifacts (muscular, non-biological). A recent successful approach to this problem is the use of independent component analysis (ICA) (Jung, Humphries, Lee, Makeig, McKeown, Iragui, & Sejnowski, 1998), a statistical tool that decomposes EEG data in a set of sources with maximally independent time courses. ICA proved very efficient in separating activity related to a large number of artifacts from neural activity by automatically segregating the former in specific independent components (ICs) (Jung, Makeig, Humphries, Lee, McKeown, Iragui, & Sejnowski, 2000; Viga´rio, Sa¨rela¨, Jousma¨ki, Ha¨ma¨laïnen, & Oja, 2000). Since the number of sources is potentially much higher than the number of ICs (Baillet, Mosher, & Leahy, 2001; Liu, Dale, & Belliveau, 2002), this separation will never be perfect (Groppe, Makeig, & Kutas, 2009). However, after removing non-stereotyped artifacts by an accurate preprocessing of the data, it is possible to obtain ‘clean’ ICA decompositions (Onton, Westerfield, Townsend, & Makeig, 2006), such that removing artifacted ICs from the data by simple subtraction generally leads to marginal distortion of the remaining EEG data (Joyce, Gorodnitsky, & Kutas, 2004; Jung, Makeig, Humphries, et al., 2000). Nevertheless, the practical usability of ICA as a tool for artifact rejection has an important limitation: the detection of the ICs associated with artifacts is time-consuming and involves subjective decision making. Several attempts have been made to guide IC classification by using a number of measures to discriminate artifacted from non-artifacted ICs either in the time domain (Barbati, Porcaro, Zappasodi, Rossini, & Tecchio, 2004; Delorme, Sejnowski, & Makeig, 2007; Mantini, Franciotti, Romani, & Pizzella, 2008), in the space domain (Li, Ma, Lu, & Li, 2006; Viola, Thorne, Edmonds, Schneider, Eichele, & Debener, 2009) or in both (Joyce et al., 2004; Okada, Jung, & Kobayashi, 2007). A single discriminative measure may already be very helpful in detecting specific artifacts (blinks (Li et al., 2006; Okada et al., 2007; Viola et al., 2009), lateral eye movements (Viola et al., 2009), heartbeat artifacts (Viola et al., 2009)) or even a wide variety of biological and non-biological artifacts (Mantini et al., 2008). Multiple measures (Barbati et al., 2004) and additional information from EOG signals (when available) (Joyce et al., 2004; Okada et al., 2007) have been used to improve this detection. However, these algorithms are not completely automatic since they either require a training set (Delorme et al., 2007; Li et al., 2006; Mantini et al., 2008), the arbitrary tuning of the thresholds separating artifacted from nonartifacted components (Barbati et al., 2004; Joyce et al., 2004; Okada et al., 2007) or an initial topography template (Viola et al., 2009). Here, a completely automatic ICA-based algorithm for identification of artifact-related components in EEG recordings is proposed. The algorithm is built on the basis of two main observations: (1) for a large number of artifacts, artifact-related ICs are characterized by stereotyped features both in their temporal course and spatial distribution; and (2) while single features may not be accurate enough when discriminating artifact from non-artifact components, their combination can efficiently and systematically achieve this goal. The proposed algorithm is called ADJUST (Automatic EEG artifact Detection based on the Joint Use of Spatial and Temporal features) because it automatically ‘‘adjusts’’ its parameters to the data to compute the set of

A. Mognon et al. artifact-specific spatial and temporal features needed for IC classification without any additional information (e.g., EOG channels). The first step of ADJUST consists of decomposing the EEG data into ICs. Four artifact classes are then considered: three classes are related to ocular artifacts (blinks, vertical, and horizontal eye movements), and a generic artifact class (called discontinuity) is devoted to capturing anomalous activity recorded at single electrodes due to high-impedance conditions or electrical instabilities in the recording device. For each of the four artifact classes, a detector is implemented by computing a class-specific set of spatial and temporal features on all ICs. For each feature, a threshold dividing artifacts from non-artifacts is estimated on the whole set of ICs in a completely automatic way by the Expectation-Maximization automatic thresholding method (Bruzzone & Prieto, 2000). For each detector, ICs are classified as artifacts if all artifact-specific spatial and temporal features belonging to that detector exceed their respective threshold value. ADJUST spatial and temporal features were optimized on a feature selection EEG dataset. ADJUST was then validated on a validation dataset recorded with a different EEG system in a different laboratory and with a different paradigm with respect to the feature selection dataset. Validation consisted of three steps: first, ADJUST classification accuracy was evaluated by comparing it to a manual classification performed by three independent experts (as in Mantini et al., 2008); second, the advantage of using a combination of features was evaluated; third, ADJUST performance in recovering clean ERP topographies from artifacted data was assessed by comparing ERPs computed after ADJUSTcorrection with the ones obtained from uncorrected data and the ones obtained after the manual classification by experts.

Materials and Methods EEG Data Acquisition and Experimental Design Two different datasets were used in this study: a feature selection dataset for selecting and implementing the optimal features for the artifact detection algorithm, and a validation dataset for testing the accuracy of artifact detection. Relative to the feature selection dataset, the validation dataset uses data recorded with a different EEG system in a different laboratory and with a different experimental design. The feature selection dataset consists of EEG recordings drawn from a study investigating serial and parallel processing during dual-task performance (details in Sigman & Dehaene, 2008). Twenty-one right-handed native French speakers (10 women; mean age 24 years, ranging from 20 to 33 years) participated in the experiment. Data from one subject could not be used for the current study because of data corruption. All participants provided informed written consent to take part in the experiment, which was approved by the Comite´ Consultatif pour la Protection des Personnes dans la Recherche Biome`dicale, Hoˆpital de Bicetre (Le Kremlin-Bicetre, France). In brief, participants were asked to perform two tasks with variable onset delay. Here, only trials with a delay onset of 1200 ms will be considered (240 trials per subject) since shorter delays involve complex effects of interference between the two tasks that were not of interest to the current study. Four more subjects were discarded because of a high rate of artifacts in the selected trials.

Automatic spatio-temporal EEG artifact detection Subjects responded to both tasks with key presses, with the right hand for the first task (number-comparison) and with the left hand for the second task (auditory tone discrimination). In the number-comparison task, a number, which varied randomly among four different values (28, 37, 53, and 62), was flashed in the center of the screen for 150 ms, and subjects had to respond whether the number was larger or smaller than 45. In the auditory task, subjects had to respond whether the tone (lasting 150 ms) was high (880 Hz) or low (440 Hz) frequency. Stimuli were presented centrally on a black-and-white display on a 17-inch monitor with a refresh rate of 60 Hz. Subjects sat 1 m from the screen. Auditory stimulation was provided through headphones. During the EEG recording, intertrial intervals (ITIs) were jittered in the range from 3 to 4.2 s (mean ITI, 3.6 s). EEG recordings were sampled at 250 Hz with a 128-electrode geodesic sensor net (EGI, Eugene, OR) referenced to the vertex. Since a high-density channel distribution is not a requirement for the current study, for computationally faster ICA calculation the number of channels was reduced from 128 to 63 while keeping the channels’ distribution uniform throughout the scalp. The validation dataset consists of EEG recordings drawn from a study investigating the neural substrates of semantic representations with a semantic priming paradigm.1 Ten right-handed native Italian speakers participated in the experiment (5 women, mean age 29 years, ranging from 21 to 39 years). On each trial, subjects were presented with two stimuli: a word presented through headphones, and a word presented visually, for a total of 360 auditory-visual pairs. The onset delay between the two stimuli was 800 ms. During the EEG recording, ITIs were jittered in the range from 1.7 to 2.7 s (mean ITI, 2.2 s). Trials comprised equal proportions of semantically related pairs at three different levels of semantic relatedness. Participants were asked to press a button whenever they read the name of a city. Catch trials including a city name were discarded from further analyses. Visual stimuli were presented centrally for 160 ms on a black-and-white display on a 17-inch monitor with a refresh rate of 75 Hz. Subjects sat 1 m from the screen. Auditory stimulation was provided through headphones. All participants provided informed written consent to take part in the experiment, which was approved by the Ethical Committee of the University of Trento (Italy). EEG was recorded at 64 electrodes (BrainAmp, Munich, Germany) referenced to the vertex, and sampled at 500 Hz. EEG Data Preprocessing Data were processed using the EEGLAB toolbox (Delorme & Makeig, 2004) and custom-made software running on Matlab 7.5.0 R2007b (MathWorks, Natick, MA) on a CentOS 5.0 Linux system (Xeon CPU X5365 @3.00 GHz quad core, 32 GB of RAM, Intel, Santa Clara, CA). Continuous data from both datasets were visually inspected to discard paroxysmal portions of artifacted data and high-pass filtered at 0.5 Hz with a basic FIR filter to remove linear trends (EEGLAB tutorial, http://sccn.ucsd.edu/eeglab/) and improve the reliability of ICA decomposition (Groppe et al., 2009). Epochs of 2.5-s duration were extracted starting 500 ms before the onset of the first stimulus. To further remove non-stereotyped artifacts that would significantly affect the quality of the ICA decomposition (Onton et al., 2006), epochs in which recordings at any channel exceeded ! 150 mV were rejected (average num1 Buiatti M., Finocchiaro, C., Mognon, A., Caramazza, A., Dehaene, S., and Piazza, M., in preparation.

231 ber of rejected epochs: 31 ! 9 for the feature selection dataset, 36 ! 10 for the validation dataset). The remaining epochs were low-pass filtered at 40 Hz for the feature selection dataset and at 25 Hz for the validation dataset to minimize muscular artifacts (more frequent in the validation dataset), a class of artifacts that is not considered in the current study. ADJUST: Implementation The three main steps of ADJUSTare illustrated in the scheme of Figure 1, and described in the following. Independent Component Analysis ICA is a well known technique in signal processing literature that detects and separates the information sources associated with multidimensional signals (Hyvarinen, Karhunen, & Oja, 2001). ICA can be used for identifying the information sources mixed in the EEG data (Lee, Girolami, & Sejnowski, 1999). Let us assume that a set of q measured observations of random variables g(t) 5 [g1(t), . . ., gq(t)]T is given by linear combination of p independent source signal components s(t) 5 [s1(t), . . ., sp(t)]T, whose number is at most equal to the number of observations (p " q); source activity is supposed to be non-Gaussian (Lee et al., 1999) or non-white in time (Belouchrani, Abed-Meraim, Cardoso, & Moulines, 1997). The ICA model can be expressed in the general case as gðtÞ ¼ AsðtÞ þ nðtÞ

ð1Þ

indicating that the observations g(t) can be obtained by mixing the sources s(t) via a constant [q ' p] matrix A called mixing matrix and adding the vector of white noise n(t) (which is not considered in some implementations). The mixing matrix is full column-rank (r(A) 5 p). Given these hypotheses, a solution to the problem of the identification of the ICA components can be implemented, and Raw EEG data

Preprocessing IC Topographies

IC Time Courses

Independent Component Analysis

Spatial feature extraction

Temporal feature extraction

Threshold computation

Threshold computation

Thresholder

Thresholder

AND

Artifact ICs removal

Clean EEG data

Figure 1. Architecture of the ADJUST algorithm for a generic detector with one spatial and one temporal feature. Any supplementary spatial or temporal feature can be added in parallel to the existing ones within the same architecture.

232

A. Mognon et al.

the ICs can be estimated by determining a [p ! q] matrix W called unmixing matrix for which the vector s^ðtÞ ¼ WgðtÞ

ð2Þ

is the best estimate of s(t). In this work, the INFOMAX algorithm (Bell & Sejnowski, 1995) implementation for ICA included in the EEGLAB toolbox was used. The INFOMAX algorithm is based on a learning rule which minimizes the mutual information between the source signals estimates, which is equivalent to maximizing the joint entropy between the estimates, in order to estimate the sources, which are assumed to be super-Gaussian. INFOMAX ICA estimates q ICs from a set of q observation vectors. ICA decomposition was computed on all datasets, separately for each subject. The number of epochs was sufficiently large to ensure a good performance of the ICA algorithm, as the (number of time points)/(number of electrodes)2 (considered as a predictor of ICA reliability (Groppe et al., 2009)) ranged between 18.70 and 50.07 (38.8 % 7.2 average % std) for the feature selection dataset, and between 68.02 and 120.93 (91.4 % 12.6 average % standard deviation) for the validation dataset. Features Computation We searched for spatial and temporal features that best captured the behavior of the ICs associated with four different artifact classes: eye blinks, vertical eye movements, horizontal eye movements, and generic discontinuities (see Figure 2 for examples of IC topographies and time courses typical of each artifact class and of a neural component). Since artifact-specific IC topographies are characterized by a particular spatial shape, which is independent of the overall scale

Vertical Eye Movement

1. Eye Blinks: Eye blinks typically generate abrupt amplitude jumps in frontal electrodes. Their time course is well captured by the kurtosis (Barbati et al., 2004; Delorme et al., 2007), a measure that is very sensitive to outliers in the amplitude distribution. Since its sensitivity to abrupt jumps would be hampered by slow amplitude drifts on the whole IC time course, here the kurtosis is computed within each epoch after removing the epoch mean, and then averaged over epochs:

Generic Discontinuity

Horizontal Eye Movement

Neural

Normalized features

ERP image

Topography

Eye Blink

of the topography, IC topography weights were normalized with respect to across the scalp: aðnÞ ¼ qffiP ffiffiffiffitheir ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffinorm ffiffiffiffiffiffiffiffiffi 2 aoriginal ðnÞ= m aoriginal ðmÞ, where a(n) is the topography weight at sensor n, aoriginal (m) is the topography weight originally computed by ICA (the vector of all topography weights aoriginal (n) corresponds to one column of the mixing matrix A (Equation 1)) and the sum is computed across all sensors m. For coherence with the ICA model (Equation 1), each corresponding was multiplied by the same factor ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiIC ffiffiffiffiffiffiffiffiactivation ffiffi qffiP 2 a ðmÞ . m original Ideal features should maximally discriminate artifact from non-artifact ICs, resulting in a bimodal distribution of values, or in case of few artifacts, in artifact IC values falling on the tails (outliers) of the non-artifact IC values distribution. For each artifact class, several measures were tested on the feature selection EEG dataset in a trial-and-error approach following this criterion, and the most effective artifact-specific features were selected for artifact classification. Hereafter we describe the selected features for each artifact class.

1

1

EB

VEM HEM

GD

1

EB

VEM HEM

GD

1

EB

VEM HEM GD

1

EB

VEM HEM

GD

EB

VEM HEM GD

Figure 2. Examples of typical ICs from each artifact class and of a typical neural IC drawn from the validation dataset. Top row: IC topography. Middle row: ERP image illustrating the color-coded amplitude fluctuations (arbitrary units) of the IC in 100 contiguous epochs (time relative to visual word presentation on the x-axis, epochs on the y-axis). Bottom row: histograms of feature values (Spatial Average Difference and Temporal Kurtosis for eye blinks (EB); Spatial Average Difference and Maximum Epoch Variance for vertical eye movements (VEM); Spatial Eye Difference and Maximum Epoch Variance for horizontal eye movements (HEM); Generic Discontinuities Spatial Feature and Maximum Epoch Variance for generic discontinuities (GD)) normalized by the corresponding automatically calculated threshold value. Bars of features belonging to the same artifact class are grouped together, and are marked in red color if they all cross the threshold, indicating that ADJUST classifies the IC as a component of that artifact class.

Automatic spatio-temporal EEG artifact detection

233

Fpz

such that their average is very low, resulting in a spuriously high value of the spatial average difference (SAD).

Fpz F5 C5 P5

Fz Cz Pz Oz

F6 C6 P6

F5 C5 P5

Fz Cz Pz

F6

2. Vertical Eye Movements: Vertical eye movements generate large amplitude fluctuations in frontal channels that are typically slower than those generated by blinks, therefore not efficiently identifiable by the kurtosis. They are well captured by a temporal feature based on the variance of the signal within each epoch:

C6 P6

Oz

Figure 3. Scalp areas used in ADJUST spatial features computation. Left-hand panel: Frontal Area (in green) and Posterior Area (in blue); Right-hand panel: Left Eye area (in yellow) and Right Eye area (in purple). Red dots indicate channel positions in the validation dataset.

Maximum Epoch Variance &D ' E trim and max si ðtÞ2 $hsi ðtÞi2ep ep &D 'i ; ¼ E 2 trim and mean si ðtÞ $hsi ðtÞi2ep ep

1 0D E 4 C B si ðtÞ ep Temporal Kurtosis ¼ trim and meanB E2 $ 3C A; @D si ðtÞ2 ep

i

ð3Þ

where si(t) indicates the time course of the IC as defined by Equation (2) within the epoch i, h. . .i ep indicates the average within an epoch, and trim_and_mean (. . .)i denotes the average across epochs computed after the top 1% of the values have been removed. This measure was preferred to the simple average because the latter would be too sensitive to spurious outliers. To capture the spatial topography of blink ICs, we used a measure specifically sensitive to higher amplitude in frontal areas compared to posterior areas: ! ! ! ! Spatial Average Difference ¼ !haiFA ! $ !haiPA !; ð4Þ

where a is the vector of normalized IC topography weights defined above, h. . .i FA denotes the average over all channels in the frontal area (FA) (radial range: 0.4oro1; angular range from medial line: 0o|y|o601 (if present, electrodes below the eyes would not be included), see Figure 3), and h. . .i PA denotes average over all channels in the posterior area (PA) (radial range: 0oro1; angular range: 1101o|y|o1801, see Figure 3). Two additional controls were imposed: a) The average IC topography weights across the left eye area (LE) (radial range 0.3oro1 and angular range $ 611oy o $ 291, see Figure 3) must have the same sign of the average IC topography weights across the right eye area (RE) (0.3oro1 and 291othetao611, see Figure 3) (to distinguish blinks from horizontal eye movements); b) Variance of scalp weights included in the FA (defined above) must be higher than the variance of scalp weights included in the PA; this control is quantified by the feature $" # % Spatial Variance Difference ¼ a2 FA $hai2FA $" # % $ a2 PA $hai2PA ; ð5Þ

which should be positive for eye blink components. This control is useful against false positives in cases where IC weights across the PA span both positive and negative values

ð6Þ

i

where trim_and_max (. . .)i indicates the maximum of the trimmed vector of variance values over the epochs (as for the kurtosis, this measure was preferred to the simple maximum because the latter would be too sensitive to spurious outliers); this measure is normalized with respect to the average of trimmed variance values (trim_and_mean (. . .)i, see Temporal Kurtosis definition above) in order to better capture the difference from the baseline behavior of the time course. Trim was performed as explained for Temporal Kurtosis. Since the spatial distribution of vertical eye movement artifacts is similar to that of blink artifacts, the same spatial feature (SAD) was used, together with the same additional controls. 3. Horizontal Eye Movements: Since the time course of artifacts caused by horizontal eye movements is similar to the one generated by vertical eye movements, the temporal feature used is the same (Maximum Epoch Variance (Equation 6)). The spatial distribution is characterized by large amplitudes in frontal channels near the eyes, typically in anti-phase (one negative and one positive). A spatial feature sensitive to this pattern is ! ! Spatial Eye Difference ¼ !haiLE $haiRE !; ð7Þ

where h. . .i LE (h. . .i RE) denotes average overall channels in the LE area (RE area) defined above, respectively. To check that amplitudes are in anti-phase, one additional control is added: the average of IC topography weights in the LE and RE must have a different sign. 4. Generic Discontinuities: Artifacts generated by impedance fluctuations or electronic device interference typically involve sudden amplitude fluctuations in one channel, with no spatial preference. The time course of this artifact is captured by Maximum Epoch Variance (Equation 6). Its spatial distribution is captured by a feature sensitive to local spatial discontinuities: Generic Discontinuities Spatial Feature !) (! ¼ max !an $ hkmn am im ! n ;

ð8Þ

where an is the nth topography weight, kmn 5 exp ( $ ||ym $ yn||) decays exponentially with the distance ||ym $ yn|| between channel m and channel n, h. . .i m denotes the average over all channels m6¼n, and max(. . .)n indicates the maximum over all channels n of the scalp.

234

A. Mognon et al.

Automatic Classification For each feature included in the detectors, the threshold value was computed by means of a completely automatic image processing thresholding algorithm based on the Expectation-Maximization (EM) technique (Bruzzone & Prieto, 2000). This algorithm is expected to work in a 1-dimensional feature space where a set O 5 fon,oa g of two information classes on and oa is defined. The classes on and oa represent the cases in which an IC component is not associated to an artifact or is associated to an artifact, respectively. The EM algorithm estimates the a priori probabilities of the classes and their probability density functions. The former model the probability that a random sample belongs to a given class, while the latter describe the distribution of each class’s random variable. Probability density functions are assumed normally distributed, and thus they are modelled by mean values mn, ma and variances sn2, sa2 of the two Gaussian distributions. In the first step, Expectation, an approximation for the two Gaussian distributions is computed from the data. The overall distribution is initially divided into two clusters that approximately contain entries from on and entries from class oa; given the middle value of the histogram MD 5 (maxfXD g1min fXD g)/2, where XD indicates the vector of feature values, two thresholds Tn and Ta equally distant from MD (distance: 0.01 (maxfXD g ! MD) from MD) are used in this initial step to separate the clusters: entries in the interval [minfXD g, Tn] are included in cluster 1, and entries in the interval [Ta, maxfXD g] are included in cluster 2. Mean, variance and prior probability computed from the clusters are assumed to be the statistics of the classes’ distributions at step zero of the following iteration process. The iteration process, named Maximization, refines the statistics of the classes’ distributions by maximizing a log-likelihood measure. At each iteration, the statistics prior probability, mean, and variance of the distributions are updated as: 0 1 X Ptþ1 ðon Þ ¼ @ ½Pt ðon Þpt ðXðiÞ=on Þ=pt ðXðiÞÞ'A=I; ð9Þ XðiÞ2XD

mtþ1 n

0

1 X Pt ðon Þpt ðXðiÞ=on Þ XðiÞA ¼@ pt ðXðiÞÞ XðiÞ2XD 0 1 X Pt ðon Þpt ðXðiÞ=on Þ A; =@ pt ðXðiÞÞ XðiÞ2X

ð10Þ

D

ðs2n Þtþ1

0

1 X Pt ðon Þpt ðXðiÞ=on Þ 2 t ¼@ ½XðiÞ ! mn ' A pt ðXðiÞÞ XðiÞ2XD 0 1 X Pt ðon Þpt ðXðiÞ=on Þ A; =@ pt ðXðiÞÞ XðiÞ2X

ð11Þ

D

where the superscripts t and t11 indicate the current and successive iteration, respectively, X(i) denotes the ith feature value and I is the length of the feature vector XD. Analogous equations can be written for class oa. This iterative process is repeated until the difference between any of the statistics at step i and the same statistic at step I11 is lower than 10 ! 4 times the statistic at step zero. At convergence, the threshold value is computed as the intersection between the estimated Gaussian distribution of class on and the estimated

Gaussian distribution of class oa, where the Gaussian distributions are computed from the statistics estimated in Equations (9)–(11). In the last step of ADJUST, each detector checks whether each IC feature value is above the respective threshold; if this occurs for all the features belonging to that detector, the IC is marked as artifacted IC (it belongs to oa) for that artifact class. Virtually artifact-free EEG data are thus obtained by simply subtracting the artifacted ICs from the data. A free version of the ADJUST software with sample data used in this study will be publicly released in the form of a plug-in toolbox to be run under the EEGLAB software (http:// sccn.ucsd.edu/eeglab/). ADJUST: Validation Procedure The validation procedure was divided into three steps: (1) determination of the accuracy of IC classification; (2) evaluation of the benefit of combining spatial and temporal features compared to their separate use; and (3) determination of the accuracy in ERP reconstruction after removal of the ICs detected as artifacts by ADJUST. Artifact Classification Accuracy In the first step, ADJUST’s IC classification was compared to manual IC classification performed by three independent scorers with proven expertise in the field of EEG analysis and familiarity with ICA decomposition of EEG data. Experts manually classified ICs from the feature selection datasets and the validation datasets by visualizing IC properties via the EEGLAB software package (Delorme & Makeig, 2004). Experts were invited to inspect the IC topography, power spectrum, and ‘ERP image’ (Jung, Makeig, Humphries, et al., 2000), a useful graphic representation displaying the IC time course of all epochs within the same figure by coding amplitudes in color scale (see Figure 2, middle row). Experts were asked to mark the ICs relative to the four classes of artifacts defined above (blinks, vertical and horizontal eye movements, discontinuities), and were invited to do so by looking both at the topography and at the time course of each component. Discontinuities were defined as sudden jumps with localized, non-biological spatial distribution. Experts were asked to mark only components that clearly belonged to one artifact class, and to not mark components also containing some presumably neural portion, as well as ambiguous components. Experts classified a total of 1008 ICs of the feature selection dataset and 630 ICs of the validation dataset. A unique classification, further referred to as ‘manual classification,’ was generated from the three scorers’ classifications by using a majority criterion. Manual and ADJUST classifications were then compared by computing an agreement measure for each class-specific detector. An additional agreement measure was generated for the detection of all types of artifacts (an IC was considered to be artifacted if detected by at least one single detector), which we will refer to as ‘general artifact detection.’ The agreement measure g was computed as the ratio between the variance accounted for by the ICs for which the two classifications agree (IC marked as artifact or non-artifact in both classifications) and the total variance of all ICs:

g¼

P

ni

AgreementICs

P i

ni

ð12Þ


235

where ni indicates the variance accounted for by the ith IC computed (using the EEGLAB function eeg_pvaf()) as the average variance of the IC activations back-projected into the electrode space, and AgreementICs is the list of ICs for which the two classifications agree. This measure is very similar to the agreement measure used in Li et al. (2006), the only difference being that in g each IC is weighted by the variance it accounts for. The rationale behind this difference is that the more variance is explained by the IC, the more it is important to correctly classify that IC as artifacted or not. Effects of Combining Spatial and Temporal Features ADJUST detectors can be thought of as AND-detectors because they identify an artifact only when all associated temporal and spatial features have values higher than the decision threshold. To evaluate the advantage of combining features in an exclusive rather than inclusive way, ADJUST classification was compared to the one obtained by using OR-detectors, which identify an artifact when any of the associated features exceeds the threshold. To evaluate the statistical significance of the difference, a paired t-test was computed between the accuracies of ANDdetectors and OR-detectors for each type of artifact, and for general artifact detection. ERP Accuracy from Artifact Corrected Data The third step consisted of testing the efficiency of ADJUST in reconstructing artifact-free topographies of well known ERPs from sets of artifacted epochs. In order to have a reliable reference, ERPs computed from artifacted epochs after ADJUST correction were compared with artifact-free ERPs from the same subjects in the same experimental sessions. For this purpose, two sets of epochs were extracted for each dataset: ‘Most Contaminated Epochs,’ in which there was at least one channel exceeding 65 mV, and ‘Least Contaminated Epochs,’ selected as artifactfree epochs in the same number of the Most Contaminated Epochs. Three different ERP topographies were computed by averaging the spatiotemporal ERP within an interval centered on the latency of the peak of the grand-averaged ERP: Auditory N1: latency [90–120] ms after the auditory stimulus; Visual P1: latency [80–110] ms after the visual stimulus; Visual N12: latency [160–190] ms after the visual stimulus. These latencies are well matched with the typical latencies found in the literature (e.g., Hine & Debener, 2007, for the Auditory N1; Di Russo, Martinez, Sereno, Pitzalis, & Hillyard, 2001, for the visual P1 and N1). The distortions caused by artifacts on each ERP topography were quantified by the topography error e, computed as the square root of the sum squared difference between the ERP topography calculated from the Least Contaminated Epochs and the one calculated from the Most Contaminated Epochs, normalized with respect to the square root of the sum squared amplitude of the Least Corrupted ERP: e¼

2

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi sffiX sffiX ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ðgLCE # gMCE Þ2 = ðgLCE Þ2 scalp

Posterior Visual N1.

scalp

ð13Þ

where gLCE (gMCE) indicates the ERP topography map calculated from Least Corrupted Epochs (Most Corrupted Epochs), and the sum is computed over all channels of the scalp. Normalization is performed in order to scale the difference by the size of the least polluted topography map; this was done because the degree of distortion introduced by artifacts depends on the magnitude of the ‘‘clean’’ ERP wave. Results EEG recordings from all datasets contained ocular artifacts and artifactual discontinuities in a variable amount across subjects. Several artifactual ICA components could be easily identified by visual inspection in all subjects. Figure 2 (top row) shows examples drawn from the validation set of artifact-specific IC topographies for each artifact class defined in the Methods sections (blinks, horizontal and vertical eye movements, discontinuities). In contrast, neural ICs typically display a smooth dipolar topography (see example of an IC representing a visual evoked potential in Figure 2, last column). The time course of artifactual signals is also archetypical: all artifacted ICs exhibit low-amplitude fluctuations interspersed with high-amplitude jumps occurring in only a few trials, represented by red or blue color spots in the ERP images of Figure 2 (second row). In contrast, the neural IC presents an activity distributed across all trials comprising a clear event-related potential, which is visible in most trials. Once ICA weights are computed, ADJUST is very fast: it takes about 12 s to run the algorithm on a dataset of about 200 MB and display the classification results for further inspection on a standard PC (Microsoft Windows XP Professional SP3, Intel Core 2 Duo CPU E4600 @2.40 GHz, 2.00 GB RAM). The joint use of spatial and temporal features revealed crucial for artifact identification: even though single features may not follow a bimodal distribution, the combination of more features together led to a cluster of artifact-specific ICs clearly separate from the rest of the ICs (see Figure 4 for an example). ADJUST’s artifact-specific detector selectivity is evident from the example of Figure 2 (bottom row): for each artifact-specific IC, all features associated to that artifact visibly cross the threshold (blue horizontal line), triggering the classification of that IC as artifacted (red bars); in contrast, for the detectors specific to the other types of artifacts there is always at least one feature that has a value lower than the threshold, so that ADJUST does not classify that IC as belonging to that artifact class (green bars). The neural IC is not classified as artifact because none of the artifact-specific groups of features crosses the threshold. ADJUST Validation As mentioned in the previous section, the effectiveness of ADJUST was assessed on a validation dataset completely different and uncorrelated with the feature selection dataset used to optimize ADJUST. The validation dataset is different from the feature selection datasets in the EEG recording system, in the laboratory in which it was recorded, and in the experimental paradigm (see Methods for details). This validation was performed by using the same spatial and temporal features that were optimized on the feature selection datasets. Artifact Classification Accuracy In the first validation step, ADJUST’s IC classification was compared to manual IC classification made by three independent expert scorers (see Methods for details). Agreement between scorers’ classification was high (95.3% on all artifacts). Among

236

Figure 4. Scatter plot showing the values of the two features composing the Blink detector (Temporal Kurtosis on the x-axis and Spatial Average Difference on the y-axis) for the ICs belonging to one subject (validation dataset). Distributions of ICs values for each feature are superposed on the relative axis. Red lines indicate the thresholds computed automatically for each feature. Red points indicate the values of ICs marked as artifacted for the Blink detector.

the 630 ICs presented to the experts, 43 were classified as ocular artifacts, 74 as generic discontinuities, and the remaining as neural or low-amplitude noise components. Even though ocular artifact ICs were fewer than discontinuity ICs, they explained a much larger amount of the total data variance (51.5% vs 1.7% of discontinuity ICs), suggesting that most artifacts are captured by a few ICs associated with ocular movements. ADJUST accuracy was computed for each artifact class and for the general artifact detection by means of the agreement measure defined in Equation (12) as the ratio between the variance accounted for by the ICs classified in the same way by ADJUSTand the independent scorers and the variance of all ICs (see Methods). Accuracy relative to ocular artifacts was excellent: 99.0% for blinks, 96.0% for vertical eye movements, 99.2% for horizontal eye movements. Despite the more heterogeneous spatiotemporal features of generic discontinuities compared to ocular artifacts, the associated accuracy was also high (97.7%). Overall, the accuracy of the detection of any type of artifact (general artifact detection) was 95.2%.3 This result is homogeneous across subjects (see black error bars for AND-detectors in Figure 5). Figure 6 summarizes the agreement between ADJUST classification and the manual classification for all artifacts. It is worth noting that false positive errors (components classified as artifact by ADJUST but considered non-artifact by manual classification) are very infrequent for both datasets, meaning that

3 The agreement measure relative to the general artifact detection (i.e., computed on all types of artifacts) is generally lower than the one relative to a single type of artifact because the former comprises disagreements (false alarms and missed alarms) relative to all types of artifacts, while the latter is only penalized by disagreements relative to that type of artifact (see Equation 12). This effect is partially compensated by occasional artifact mislabeling (e.g., a blink labeled as a generic discontinuity by ADJUST), which affects the single artifact agreement measure but not the general artifact detection one.

A. Mognon et al.

Figure 5. Comparison of classification accuracies obtained by ANDdetectors (dark gray bars) and OR-detectors (light gray bars) for each artifact detector and for the general artifact detection for the validation dataset. Black error bars indicate the standard error of the mean.

the probability of removing a neural component is very small (2.5% of the total variance). Effects of Combining Spatial and Temporal Features In the second validation step, we evaluated the benefits of characterizing each artifact class by the combination of spatial and temporal features together with respect to using each feature as a separate artifact-specific detector. To this purpose, performance of ADJUST detectors (here indicated as AND-detectors because they classify an IC as artifacted only when all associated temporal and spatial features have a value higher than their respective threshold) was compared with that obtained by OR-

Figure 6. Classification performance of the general artifact detector (an IC is labeled as an artifact if marked as such by any of the four artifact detectors) compared to the one provided by three independent experts for the validation dataset. Bars represent the amount of True Negative (TN), True Positive (TP), False Negative (FN), and False Positive (FP) ICs weighted by the percent of total variance they account for, respectively. Black error bars represent the standard error of the mean.


237

NO CORRECTION

ADJUST CORRECTION

Auditory N1

ERP topography

ERP time course

Visual P1

ERP topography

ERP time course

Visual N1

ERP topography

ERP time course

Figure 7. Examples of ERP reconstruction at the three selected latencies drawn from three representative subjects. Each panel shows: 1) ERP topographies at peak latency (top row) computed on Least Contaminated epochs (odd columns) and Most Contaminated epochs (even columns) with no correction (left-hand columns) and after ADJUST correction (right-hand columns); 2) ERP time courses (bottom row) at representative electrodes (F3, Fz, F4, C3, Cz, C4, O1, Oz, O2) before and after ADJUST correction averaged over Least Contaminated epochs (blue lines) and Most Contaminated epochs (red lines). Vertical red arrows mark the latency of the ERP components at the electrodes showing highest amplitude.

detectors, which classify an IC as artifacted when any of the associated features exceeds the threshold value. AND-detectors reach significantly higher accuracies than the respective ORdetectors for all types of artifacts and for the general artifact detector (for all t-tests, t(9)419.68, po.001) (Figure 5). The performance of OR-detectors is sometimes near to that of ANDdetectors for the most stereotyped artifact (e.g., horizontal eye movements), but drastically drops for more heterogeneous artifacts, resulting in accuracy for the general artifact detector of about 80%. ERP Accuracy from Artifact Corrected Data The last validation step consisted of testing the efficiency of ADJUST in reconstructing artifact-free topographies of wellknown ERPs from sets of artifacted epochs. In order to have a

reliable reference, ERPs computed from artifacted epochs after ADJUST correction were compared with artifact-free ERPs from the same subjects in the same experimental sessions. For this purpose, two sets of epochs were extracted for each dataset: ‘Least Contaminated Epochs’ were virtually artifact-free, while ‘Most Contaminated Epochs’ were the most contaminated by artifacts (see Methods for selection criteria). The average number of epochs in each set was 90 ! 12 (mean ! standard deviation). ADJUST was evaluated on the efficiency of reconstruction from the Most Contaminated Epochs of the topographies of three different ERPs: auditory N1 (latency 100 ms), visual P1 (latency 100 ms), and visual N1 (latency 175 ms). As expected, artifacts considerably altered ERP topographies and time courses (Figure 7). Anterior electrodes were particularly affected,

238 suggesting that ocular artifacts were the major cause of alteration. ADJUST systematically removed the most important artifacts distortions, reconstructing an ERP from the Most Contaminated Epochs that almost overlaps to the one computed from the Least Contaminated Epochs, both in its topography and in its time course (Figure 7). The amount of distortions caused by the residual artifacts on each ERP topography was quantified by the topography error e (Equation 13). Topography errors were significantly lower for ADJUST than for uncorrected data for all examined ERPs (Wilcoxon signed rank test (Wilcoxon, 1945): p 5 .0097 for auditory N1, p 5 .0019 for visual P1, and p 5 .0019 for visual N1) (Figure 8). Topography error variability across subjects was also remarkably lower for ADJUST than for uncorrected data (error bars in Figure 8), suggesting that ADJUST typically provides a stable performance across subjects. To further validate the performance of ADJUST, topography errors obtained using ADJUST were compared to those relative to manual classification. The difference between the topography errors from the two methods was not significant for all ERPs considered (Wilcoxon signed rank test (Wilcoxon, 1945): p 5 .56 for auditory N1, p 5 .24 for visual P1, and p 5 .70 for visual N1) (Figure 8), suggesting that ADJUST performance is equivalent to that of a manual classification by experts. Discussion In this paper, a completely automatic method for the detection of artifacted ICs from EEG data (ADJUST) has been proposed. The core property of ADJUST is the simultaneous use of multiple spatial and temporal features to detect the artifacted ICs. The key applicative aspect of ADJUST is its completely automatic nature: no trial-and-errors procedures are necessary for tuning parameters, as features are defined a priori and the algorithm that computes feature thresholds is completely unsupervised (Bruzzone & Prieto, 2000). The efficiency of ADJUST was demonstrated by a remarkable classification accuracy (95.2% for all artifacts) and by its ability to reconstruct clean ERP topographies from heavily artifacted data. In the following, we discuss these results in the context of the recent literature, and propose future extensions and applications of our method. Automaticity and Feature Combination As clearly described in Onton et al. (2006), EEG artifacts can be divided in two classes: non-stereotyped artifacts due to movements of the electrodes on the scalp arising from large muscle movements or external sources, and stereotyped artifacts, mainly due to ocular eye movements and blinks. Artifacts from the first class are problematic for ICA because, since their spatial distribution is extremely variable, they introduce a large number of unique scalp maps, leaving few ICs available for capturing brain sources. Accordingly, ADJUST does not attempt to remove these artifacts, and it relies on a suitable pre-processing for removing them before the ICA decomposition. However, stereotyped artifacts belonging to the second class also display a wide spatial and temporal heterogeneity that is testified by the wide range of measures that have been used to identify them: high-order statistics (Barbati et al., 2004; Delorme et al., 2007), entropy measures (Mantini et al., 2008), and spatial templates (Li et al., 2006; Viola et al., 2009). Here we have proposed an algorithm that identifies ICs due to stereotyped artifacts by computing several measures simultaneously. Its effectiveness is a

A. Mognon et al.

Figure 8. Mean topography error (Equation 13) of all subjects of the validation dataset for the three selected ERPs: Auditory N1, Visual P1, and Visual N1, for data with no correction (dark gray), data after ADJUSTcorrection (mid-gray), and data after correction (light gray) by manual classification. Bars indicate standard deviation of the mean. Symbols over thin black lines indicate the results of a Wilcoxon signed rank test performed between topography errors relative to the conditions linked by the same lines. One asterisk (two asterisks) indicate po.05 (po.01), while N.S. indicates non-significant.

demonstration that the diversity of EEG artifacts is limited, and can be fully captured by a reduced set of spatial and temporal features, provided that these are used together (see Figure 5). The efficiency of the automatic procedure implemented in ADJUST is based on this simple property: even though the distribution of single feature values does not clearly separate artifact from artifact-free components, this goal is achieved when combining more features together (Figure 4). Remarkably, this result is obtained with a simple thresholding algorithm (Bruzzone & Prieto, 2000) in a completely unsupervised way. This is important because most of the ICA-based artifact removal algorithms proposed in the literature have a supervised seed, either in the form of a training set (Delorme et al., 2007; Li et al., 2006; Mantini et al., 2008) or in the arbitrary tuning of the thresholds separating artifacted from non-artifacted components (Barbati et al., 2004; Joyce et al., 2004; Okada et al., 2007). ADJUST Potential Extensions The architecture of ADJUST (Figure 1) is intrinsically flexible: extensions to other types of artifacts may be implemented by building new detectors for those artifacts and adding them in parallel to the original ones. One type of artifact that may be included in the future is the one generated by muscle contractions, which may be identified by a spectral feature (Barbati et al., 2004; Delorme et al., 2007; Joyce et al., 2004). Another natural extension is to add features based on correlation with ECG or EOG signals (Joyce et al., 2004; Okada et al., 2007). In particular, EOG signals would improve the identification of blink ICs as they typically display a polarity flip across the eyes (Talsma & Woldorff, 2005), which would be easily captured by an ad-hoc feature. More generally, ADJUST architecture might


239

be used in the future for the classification of neural components from continuous EEG data, for example, by integrating it with algorithms of IC clustering as the one implemented in the EEGLAB software. This approach is potentially promising for studies focused on the relation between event-related and ongoing activity (Buiatti, 2008). Contrary to the case of artifacts, features expressing data regularity and stationarity might be chosen. Potentially, ADJUST can be adapted to MEG data. Spatial filters for spatial features computation can be easily imported

since they are based on scalp areas, which are identified by polar coordinates; sensors involved are automatically detected by inspecting channels coordinates. Due to its ease of application for its automatic nature, absence of constraints on the experimental paradigm, and flexibility to new extensions, we believe that ADJUST is suitable for routine automatic artifact removal in research and clinical settings. Additional tests on populations prone to artifacts (like clinical data, or data on children) will help to further improve the method.

REFERENCES Baillet, S., Mosher, J. C., & Leahy, R. M. (2001). Electromagnetic brain mapping. IEEE Signal Processing Magazine, 18, 14–30. Barbati, G., Porcaro, C., Zappasodi, F., Rossini, P. M., & Tecchio, F. (2004). Optimization of an independent component analysis approach for artifact identification and removal in magnetoencephalographic signals. Clinical Neurophysiology, 115, 1220–1232. Bell, A. J., & Sejnowski, T. J. (1995). An information-maximisation approach to blind separation and blind deconvolution. Neural Computation, 7, 1004–1034. Belouchrani, A., Abed-Meraim, K., Cardoso, J.-F., & Moulines, E. (1997). A blind source separation technique using second-order statistics. IEEE Transactions on Signal Processing, 45, 434–444. Bruzzone, L., & Prieto, D. F. (2000). Automatic analysis of the difference image for unsupervised change detection. IEEE Transactions on Geoscience and Remote Sensing, 38, 1171–1182. Buiatti, M. (2008). The correlated nature of large-scale neural activity unveiled by the resting brain. Rivista Di Biologia-Biology Forum, 101, 353–373. Croft, R., & Barry, R. (2000). Removal of ocular artifact from the EEG: A review. Clinical Neurophysiology, 30, 5–19. Croft, R., & Barry, R. (2002). Issues relating to the subtraction phase in EOG artifact correction of the EEG. International Journal of Psychophysiology, 44, 187–195. Delorme, A., & Makeig, S. (2004). EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of Neuroscience Methods, 134, 9–21. Delorme, A., Sejnowski, T. J., & Makeig, S. (2007). Enhanced detection of artifacts in EEG data using higher-order statistics and independent component analysis. NeuroImage, 34, 1443–1449. Di Russo, F., Martinez, A., Sereno, M. I., Pitzalis, S., & Hillyard, S. A. (2001). Cortical sources of the early components of the visual evoked potential. Human Brain Mapping, 15, 95–111. Gasser, T., Sroka, L., & Mocks, J. (1985). The transfer of EOG activity into the EEG for eyes open and closed. Electroencephalography & Clinical Neurophysiology, 61, 181–193. Gratton, G., Coles, M. G., & Donchin, E. (1983). A new method for offline removal of ocular artifact. Electroencephalography & Clinical Neurophysiology, 55, 468–484. Groppe, D. M., Makeig, S., & Kutas, M. (2009). Identifying reliable independent components via split-half comparisons. NeuroImage, 45, 1199–1211. Hine, J., & Debener, S. (2007). Late auditory evoked potentials asymmetry revisited. Clinical Neurophysiology, 118, 1274–1285. Hyvarinen, A., Karhunen, J., & Oja, E. (2001). Independent component analysis. New York: John Wiley & Sons. Joyce, C. A., Gorodnitsky, I. F., & Kutas, M. (2004). Automatic removal of eye movement and blink artifacts from EEG data using blind component separation. Psychophysiology, 41, 313–325. Jung, T.-P., Humphries, C., Lee, T.-W., Makeig, S., McKeown, M. J., Iragui, V., & Sejnowski, T. J. (1998). Extended ICA removes artifacts from electroencephalographic recordings. In D. Touretzky, M. Mozer, & M. Hasselmo (Eds), Advances in Neural Information Processing Systems, 10, 894–900. Jung, T.-P., Makeig, S., Humphries, C., Lee, T.-W., McKeown, M. J., Iragui, V., & Sejnowski, T. J. (2000). Removing electroencephalographic artifacts by blind source separation. Psychophysiology, 37, 163–178.

Jung, T.-P., Makeig, S., Westerfield, M., Townsend, J., Courchesne, E., & Sejnowski, T. J. (2000). Removal of eye activity artifacts from visual event-related potentials in normal and clinical subjects. Clinical Neurophysiology, 111, 1745–1758. Kenemans, J. L., Molenaar, P. C. M., Verbaten, M. N., & Slangen, J. L. (1991). Removal of the ocular artifact from the EEG: A comparison of time and frequency domain methods with simulated and real data. Psychophysiology, 28, 114–121. Lee, T.-W., Girolami, M., & Sejnowski, T. J. (1999). Independent component analysis using an extended Infomax algorithm for mixed subGaussian and superGaussian sources. Proceedings of the 4th Joint Symposium of Neural Computation, 7, 132–139. Li, Y., Ma, Z., Lu, W., & Li, Y. (2006). Automatic removal of the eye blink artifact from EEG using an ICA-based template matching approach. Physiological Measurement, 27, 425–436. Linkenkaer-Hansen, K., Nikouline, V. V., Palva, J. M., & Ilmoniemi, R. J. (2001). Long-range temporal correlations and scaling behavior in human brain oscillations. Journal of Neuroscience, 21, 1370–1377. Liu, A. K., Dale, A. M., & Belliveau, J. W. (2002). Monte Carlo simulation studies of EEG and MEG localization accuracy. Human Brain Mapping, 16, 47–62. Mantini, D., Franciotti, R., Romani, G. L., & Pizzella, V. (2008). Improving MEG source localizations: An automated method for complete artifact removal based on independent component analysis. NeuroImage, 40, 160–173. Niedermeyer, E., & da Silva, F. H. L. (2005). Electroencephalography: basic principles, clinical applications, and related fields (Fifth edition). Hagerstown, MD: Lippincott Williams & Wilkins. Okada, Y., Jung, J., & Kobayashi, T. (2007). An automatic identification and removal method for eye-blink artifacts in event-related magnetoencephalographic measurements. Physiological Measurements, 28, 1523–1532. Onton, J., Westerfield, M., Townsend, J., & Makeig, S. (2006). Imaging human EEG dynamics using independent component analysis. Neuroscience and Biobehavioral Reviews, 30, 808–822. Oster, P. J., & Stern, J. A. (1980). Measurement of eye movement electrooculography. In I. Matin & P. H. Venables (Eds.). Techniques in Psychophysiology, 275–309. Peters, J. F. (1967). Surface electrical fields generated by eye movement and eye blink potentials over the scalp. Journal of EEG Technology, 7, 27–40. Sigman, M., & Dehaene, S. (2008). Brain mechanisms of serial and parallel processing during dual-task performance. Journal of Neuroscience, 28, 7585–7598. Talsma, D., & Woldorff, M. G. (2005). Methods for the estimation and removal of artifacts and overlap in ERP waveforms. In T. C. Handy (Ed.), Event-related potentials: A methods handbook (pp. 115–148). Cambridge, MA: MIT Press. Vanhatalo, S., Palva, J. M., Holmes, M. D., Miller, J. W., Voipio, J., & Kaila, K. (2004). Infraslow oscillations modulate excitability and interictal epileptic activity in the human cortex during sleep. Proceedings of the National Academy of Sciences USA, 101, 5053–5057. Verleger, R., Gasser, T., & Mocks, J. (1982). Correction of EOG artifacts in event-related potentials of the EEG: Aspects of reliability and validity. Psychophysiology, 19, 472–480.

240 Viga´rio, R., Sa¨rela¨, J., Jousma¨ki, V., Ha¨ma¨laïnen, M., & Oja, E. (2000). Independent component approach to the analysis of EEG and MEG recordings. IEEE Transactions on Biomedical Engineering, 47, 589– 593. Viola, F. C., Thorne, J., Edmonds, B., Schneider, T., Eichele, T., & Debener, S. (2009). Semi-automatic identification of independent components representing EEG artifact. Clinical Neurophysiology, 120, 868–877.

A. Mognon et al. Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics, 1, 80–83. Woestenburg, J. C., Verbaten, M. N., & Slangen, J. L. (1983). The removal of the eye-movement artifact from the EEG by regression analysis in the frequency domain. Biological Psychology, 16, 127–147. (Received March 30, 2009; Accepted March 29, 2010)


An event-related source localization study of response monitoring and social impairments in autism spectrum disorder

DIANE L. SANTESSO,a IRENE E. DRMIC,b MICHELLE K. JETHA,c SUSAN E. BRYSON,d JOEL O. GOLDBERG,b,e GEOFFREY B. HALL,e KAREN J. MATHEWSON,c SIDNEY J. SEGALOWITZ,a and LOUIS A. SCHMIDTc a

Department of Psychology, Brock University, St. Catharines, Ontario, Canada Department of Psychology, York University, Toronto, Ontario, Canada c Department of Psychology, Neuroscience & Behaviour, McMaster University, Hamilton, Ontario, Canada d IWK Health Centre, Departments of Pediatrics and Psychology, Dalhousie University, Halifax, Nova Scotia, Canada e Department of Psychiatry and Behavioural Neuroscience, McMaster University, Hamilton, Ontario, Canada b

Abstract A number of studies suggest anterior cingulate cortex (ACC) abnormalities in autism spectrum disorder (ASD), which might underlie response monitoring and social impairments exhibited by children and adolescents with ASD. The goal of the present study was to extend this work by examining error and correct response monitoring using event-related potentials (ERN, Pe, CRN) and LORETA source localization in high functioning adults with ASD and controls. Adults with ASD showed reduced ERN and Pe amplitudes and reduced rostral ACC activation compared with controls. Adults with ASD also showed less differentiation between error and correct ERP components. Social impairments and higher overall autism symptoms were related to reduced rostral ACC activity at the time of the ERN, particularly in adults with ASD. These findings suggest that reduced ACC activity may reflect a putative brain mechanism involved in the origins and maintenance of social impairments and raise the possibility of the presence of stable brain-behavior relation impairment across development in some individuals with ASD. Descriptors: Response monitoring, Autism spectrum disorder, ERN, Anterior cingulate cortex

evidence over the last decade points to structural and functional abnormalities of the anterior cingulate cortex (ACC) underlying these impairments (Haznedar, Buchsbaum, Metzger, Solimando, Spiegel-Cohen, & Hollander, 1997; Haznedar, Buchsbaum, Wei, Hof, Cartwright, et al., 2000; Mundy, 2003; Thakkar, Polli, Joseph, Tuch, Hadjikhani, et al., 2008; Vlamings, Jonkman, Hoeksma, van Engeland, & Kemner, 2008). Cognitive theories of the ACC emphasize its role in cognitive control of motor behavior, specifically response monitoring and action selection (Botvinick, Braver, Barch, Carter, & Cohen, 2001; Bush, Luu, & Posner, 2000; Devinsky, Morrell, & Vogt, 1995; Holroyd & Coles, 2002) for which the dorsal (cognitive) region of the ACC uses reward prediction error signals conveyed there by the mesencephalic dopamine system to reinforce adaptive behaviors (Holroyd & Coles, 2002). As such, the dorsal ACC is activated by error information: internally generated response errors and negative environmental feedback (Holroyd, Nieuwenhuis, Yeung, Nystrom, Mars, et al., 2004). In contrast, socialcognitive theories emphasize the role of the dorsal ACC in social orienting (Dawson, Meltzoff, Osterling, Rinaldi, & Brown, 1998), joint attention (Henderson, Yoder, Yale, & McDuffie, 2002; Mundy, Card, & Fox, 2000), developing representations of the self (Craik, Moroz, Moscovich, Stuss, Winocur, et al., 1999;

Autism spectrum disorder (ASD) is a pervasive neurodevelopmental condition characterized by a wide range of cognitive and social impairments, including (but not restricted to) monitoring, joint attention, theory of mind (i.e., social perspective taking) and initiation and modification of social behaviors (see Mundy 2003, for a review). Converging theoretical and empirical IED is now at the Autism Research Unit, Hospital for Sick Children, Toronto, Ontario, and MKJ is now at the Department of Psychology, Brock University. This research was supported by a National Alliance for Autism Research (NAAR) Pre-doctoral Fellowship awarded to IED under the direction of SEB, a Natural Sciences and Engineering Research Council of Canada (NSERC) Pre-doctoral Fellowship awarded to MKJ under the direction of LAS, a Community Social and Vocational Rehabilitation (CVSR) Foundation grant awarded to JOG, a Lawson Foundation Post-doctoral Fellowship awarded to KJM under the direction of LAS, and Social Sciences and Humanities Research Council of Canada (SSHRC) and NSERC operating grants awarded to LAS. The authors would like to thank Sue McKee and Stephanie Tak for their help with data collection and the clients and staff at the Woodview Manor Independent Living Program. Address correspondence to: Dr. Louis A. Schmidt, Department of Psychology, Neuroscience & Behaviour, McMaster University, Hamilton, ON L8S 4K1, Canada. E-mail: [email protected] 241

242 Frith & Frith, 1999, 2001; Johnson, Baxter, Wilder, Pipe, Heiserman, & Prigatano, 2002) and relating the self to others (Frith & Frith, 2001). In an examination of the neural basis of social impairments in autism, Mundy (2003) argued that deficits in cognitive functions of the dorsal ACC may contribute to social behavior disturbance in autism. Efficient response monitoring may be essential to the learning and responding to social cues as well as evaluating the reward value of social stimuli (Birrell & Brown, 2000; DiGirolamo, Kramer, Barad, Cepeda, Weissman, et al., 2001; Klin, Jones, Schultz, & Volkmar, 2003; Koechlin, Basso, Peirini, Panzer, & Grafman, 1999; Mundy, 1995) and impairments in response monitoring may affect the normal development of theory of mind (Leslie, 1987), joint attention, and the ability to shift attention between social and nonsocial goals and representations (Mundy et al., 2000; Mundy, Sigman, Ungerer, & Sherman, 1986). Evidence from a variety of neurophysiological studies support theories of ACC abnormalities in autism and ASD. In a recent post-mortem study, Simms and colleagues (Simms, Kemper, Timbie, Bauman, & Blatt, 2009) reported that the ACC of individuals with autism had smaller neurons and reduced neuronal density. Evidence from in vivo studies implicates altered cell membrane metabolism in the ACC (Levitt, O’Neill, Blanton, Smalley, Fadale, et al., 2003), decreased ACC cerebral blood flow (Ohnishi, Matsuda, Hashimoto, Kunihiro, Nishikawa, et al., 2000), and smaller brain volume and hypometabolism relative to controls in the rotstal and dorsal ACC (Haznedar et al., 1997, 2000). Importantly, the latter study demonstrated that glucose metabolism in the rostral and dorsal ACC was positively correlated with social interaction, verbal communication, and nonverbal communication scores in the autism group. In a more recent fMRI study, Thakkar et al. (2008) reported that individuals with ASD had increased activation in the rostral ACC during a performance monitoring task, which was related to repetitive behaviors. ACC-related activity in ASD has also been examined using event-related potentials (ERPs). The error-related negativity (ERN) has been widely used as an electrophysiological index of error detection during response monitoring ERP tasks (i.e., when comparing performance against intended goals; Bernstein, Scheffers, & Coles, 1995; Falkenstein, Hoormann, Christ, & Hohnsbein, 2000). The ERN is a stable (Segalowitz, Santesso, Murphy, Homan, Chantziantoniou, & Khan, 2010) frontocentral component appearing 50 to 100 ms following errors and is associated with activity of the dorsal ACC (Herrmann, Rommler, Ehlis, Heidrich, & Fallgatter, 2004; Holroyd & Coles, 2007; Turken & Swick, 2008) among other medial frontal regions (Stemmer, Segalowitz, Witzke, & Scho¨nle, 2004; Turken & Swick, 2008). The ERN is thought to reflect a reinforcement learning signal originating in the basal ganglia and transmitted to the dorsal ACC via the mesencephalic dopamine system (Holroyd & Coles, 2002). Accordingly, the ERN is assumed to index the automatic initial detection of unfavorable performance outcomes, which then triggers recruitment of prefrontal-based cognitive control. The reinforcement-learning hypothesis has been challenged, however, by the hypothesis that ERN reflects the conflict between response representations or incompatible streams of information (e.g., Botvinick, Braver, Barch, Carter, & Cohen, 2001; Botvinick, Nystrom, Fissell, Carter, & Cohen, 1999). It has also been proposed that the ERN reflects the affective evaluation of errors (e.g., Luu, Tucker, Derryberry, Reed, & Poulsen, 2003) such that the motivational impact (‘‘caring’’ about errors) and/or salience of the error outcome influence the ERN

D. L. Santesso et al. (but this theory is not incompatible with reinforcement learning and/or conflict detection theories of the ERN). Enhanced ERNs have been linked to negative affect (Luu, Collins, & Tucker, 2000), obsessive-compulsiveness (Gehring, Himle, & Nisenson, 2000; Santesso, Segalowitz, & Schmidt, 2006a), depression (Holmes & Pizzagalli, 2008) and anxiety (Olvet & Hajcak, 2009). These studies emphasize that enhanced ERNs might be the result of excessive self-monitoring and/or negative reactions to having erred. Interestingly, activity in the rostral (affective) region of the ACC has been linked to affective response to errors: depressed (Holmes & Pizzagalli, 2008) and obsessive-compulsive (Fitzgerald, Welsh, Gehring, Abelson, Himle, et al., 2005) individuals show rACC hyperactivity and individuals with schizophrenia show rACC hypoactivity during error monitoring (Laurens, Ngan, Bates, Kiehl, & Liddle, 2003). Studies have linked reduced ERNs with compromised social behavior (Dikman & Allen, 2000; Santesso, Segalowitz, & Schmidt, 2005) lack of empathy and risk-taking (Santesso & Segalowitz, 2009), emphasizing the possibility that non-vigilant performance monitoring and/or not caring about having erred influences the ERN amplitude. Other ERPs have been examined in ASD. However, the functional significance is less clear for the error-related positivity (Pe) and correct response negativity (CRN), which are also elicited during response monitoring tasks. The Pe, which occurs 200 to 500 ms following error responses, has been related to error awareness and post-error adjustments in performance (e.g., posterror slowing; Falkenstein et al., 2000; Leuthold & Sommer, 1999). The Pe has a centroparietal distribution but also has been localized to the more rostral regions of the ACC (Luu, Flaisch, & Tucker, 2000; Miltner, Lemke, Weiss, Holroyd, Scheffers, & Coles, 2003; Ullsperger & von Cramon, 2003; van Veen & Carter, 2002). Like the ERN, the CRN is a frontocentrally distributed and dorsal ACC-generated negativity. The CRN, however, occurs after correct responses (Hochman, Eviatar, Breznitz, Nevat, & Shaul, 2009), possibly reflecting uncertainty about the correctness of a response (Pailing & Segalowitz, 2004) or a failure to vigilantly monitor performance, thus leading to a subsequent decrease of executive control and error commission (Allain, Carbonnell, Falkenstein, Burle, & Vidal, 2004). To date, there are no studies examining the ERN in adults with ASD or in relation to social impairments. In children, Henderson, Schwartz, Mundy, Burnette, Sutton, et al. (2006) reported that the ERN was larger for ASD children, and this pattern, together with high IQ, was related to fewer social impairments. In children, Vlamings, Jonkman, Hoeksma, van Engeland, and Kemner (2008) reported that the ERN was reduced for those with ASD compared to controls, with the ERN localized to the ACC using dipole modeling in both groups. The ASD group also showed less differentiation between error and correct trials (smaller CRNERN difference) and reduced Pe amplitudes. Collectively, these results support abnormal ACC-related response monitoring in ASD. It may be, however, the case that maturation of the ACC was delayed in ASD children compared with control children, thus necessitating the examination of ERN/ACC differences in adults who are past this developmental stage. The goal of the present study was to clarify and extend the findings from the recent ERN/autism studies with children (i.e., Henderson et al., 2006; Vlamings et al., 2008) to adults. We used high-density ERP and source localization techniques to examine ACC function in high functioning adults with ASD and controls during a response monitoring task and related this activity to social impairments.

Error monitoring in autism We tested a sample of DSM-IV diagnosed high-functioning adults with ASD and age- and IQ-matched controls performing a version of a flanker task. We predicted that adults with ASD would show diminished ERN and Pe amplitudes following errors, poor differentiation between error and correct responses and a reduction in ACC error-related activity. Given the link between social behavior and the ERN/ACC activity, we also predicted that diminished ERN amplitude and ACC activity would be associated with increased social impairments. Since no link has been made between social behaviors and the Pe or CRN, no specific hypotheses were tested.

Method Participants Autism group. Fifteen high-functioning adults with a clinically confirmed DSM-IV diagnosis of autism or autism spectrum disorder (including Asperger syndrome and pervasive developmental disorder, not otherwise specified) were included in the study. Participants (11 males) ranged in age from 18.8 to 51.6 years (M 5 36.0, SD 5 11.1), and were recruited through Woodview Manor in Hamilton, Ontario. Woodview Manor is an Independent Living Program that serves high-functioning individuals with ASD. Twelve participants met criteria for autism/ASD based on the Autism Diagnostic Observation Schedule (ADOS), Autism Diagnostic Interview/Autism Diagnostic Interview-Revised (ADI/ADI-R), and expert clinical judgment using the DSM-IV (APA, 1994). While specific diagnostic information was not available on the remaining three participants, they did require a formal ASD diagnosis before entering the Independent Living Program; and in two of these cases, the diagnosis was confirmed based on the clinical judgment of one of the authors (SEB, psychologist with extensive experience in ASD). Exclusion criteria included major psychiatric and/or medical disorders and known neurological conditions. Six participants in the ASD group were currently on both atypical antipsychotic and antidepressant medications, and two participants were currently taking only atypical anti-psychotic medications. One ASD participant was excluded because of an IQ score of 64, and another had unusable data because only one error was made on the flanker task, so the final ASD sample size was 13 used in the analyses. Non-clinical comparison group. Sixteen non-clinical adults (12 males), ranging in age from 22.6 to 47.8 years (M 5 35.7, SD 5 10.6) were recruited through McMaster University in Hamilton, Ontario. This control group was matched to the ASD group on sex, age (p-values 4.90) and general cognitive ability (p-values 4.30). Exclusion criteria included a history of learning disabilities, head injury, or chronic medical or neurological disorders. None of the control participants were currently taking medications. Psychometric Measures IQ. The Stanford-Binet Intelligence Scales, Fifth Edition (SB-V), were administered to all participants. The SB-V is an individually-administered assessment of intelligence and cognitive abilities developed for use with adults and children. IQ scores can be obtained from the Abbreviated Battery of the SB-V, which includes a verbal subtest (Vocabulary) and a non-verbal

243 subtest (Object Series/Matrices). The abbreviated battery of the SB-V shows excellent reliability (a 5 .91), similar to the Full Scale Score (.97 to .98) and the Nonverbal (.96) and Verbal (.95) subscale scores (Roid & Pomplun, 2005). Correlations between composite IQ scores of the SB-Vcompare favorably to composite scores of previous SB editions (SB-IV, .85; Form L-M, .90). IQ for the ASD group ranged from 82 to 136 (M 5 101.5, SD 5 14.4), and the control group scores ranged from 76 to 121 (M 5 103.4, SD 5 14.0). No significant differences were found for Total IQ (p4.73). Autism-spectrum quotient (AQ). This self-administered 50item questionnaire was used to assess the level of autism disorder symptoms in individuals with normal or close to normal intelligence along five different domains associated with the autism spectrum: social skills, communication skills, imagination, attention to detail, and attention switching/tolerance of change. Examples of the item are ‘‘I find social situations easy’’ (social skills), and ‘‘I enjoy social chit-chat’’ (communication). The AQ has been shown to accurately identify autistic traits in persons of normal intelligence along a continuum from Asperger syndrome to autism (Baron-Cohen, Wheelwright, Skinner, Martin, & Clubley, 2001). It demonstrates good internal consistency, face and construct validity, and test-retest reliability (Baron-Cohen et al., 2001). Positive scores on this metric indicate higher symptoms (e.g., poor social skills, poor communication skills). As expected, the ASD group scored significantly higher than controls on the social skills [t(27) 5 4.69, po.001], communication skills [t(27) 5 2.85, p 5 .01], and imagination subscales [t(27) 5 4.30, po.001]. Number Flanker Task Participants were presented with an array of three digits and were asked to identify the center number by key press, using a left key if the center number was 3, and a right key for the number 4. This procedure was not counterbalanced across participants, as mapping the lower number to the left key, and the higher number to the right was considered a useful mapping aid. There were a total of four conditions: two congruence (incongruent, congruent) and two distance (close, far). For congruent trials, the flanking numbers matched the centre number (e.g., 333, 444). For incongruent trials, the flanking numbers differed from the center number (e.g., 343, 434). The number arrays were presented either in a close configuration (0.6 degree of visual angle) or in a more widely spaced configuration (1.8 degrees of visual angle).1 The presentation of individual trials was randomized among the four conditions. Target and flanking numbers appeared simultaneously for 200 ms, and each trial ended when a response was made or when 3000 ms had elapsed. The interstimulus interval was 1000 ms. Participants completed 8 blocks of 72 trials each, with a break between each block. The task was presented using E-Prime presented on a 16-inch computer monitor at a fixed 1 Attention has been described as over focused in autism, and evidence of difficulties shifting and disengaging attention has been considered consistent with the notion of a narrow ‘beam’ or ‘spotlight’ of attention (Bryson, Wainwright-Sharp, & Smith, 1990; Landry & Bryson, 2004; Rincover & Ducharme, 1987). However, recent work examining the phenomenon of change blindness has found that individuals with autism show enhanced detection of continuity errors, and argued for an abnormally broad spotlight of attention and superior perceptual skills in individuals with autism (Smith & Milne, 2009). These hypotheses were tested as part of a larger study so we have not addressed the behavioral results here.

244 distance of 57 cm. A chin and forehead rest was used to maintain viewing distance. Procedures Participants were tested at the Child Emotion Laboratory at McMaster University. After completing self-report questionnaires, each participant was administered the IQ and AQ measures. In the case of the AQ measure, the examiner read the instructions and items to the individuals with ASD and, when required, clarification was provided. Controls completed the questionnaires on their own. After completion of these measures, resting electroencephalogram (EEG) and electrocardiogram (ECG) measures were recorded. Both the resting continuous EEG and ECG data and most of the self-report data were collected as part of a larger study. Here we report the ERP data collected during the flanker task. Participants were assessed individually and reimbursed for their time and travel expenses ($40 CAD). The study was approved by the McMaster Health Sciences Research Ethics Board. All participants were deemed cognitively capable of providing informed consent prior to testing. EEG Data Recording and Reduction EEG was recorded continuously using a 128-channel Electrical Geodesics system (EGI Inc., Eugene, OR) at 250 Hz with 0.1 to 100 Hz analog filtering referenced to the vertex. Impedance of all channels was kept below 50 kO. EEG epochs for the midline sites Fz, FCz, Cz, and Pz were extracted beginning 200 ms before and ending 800 ms after errors and correct responses. Data were processed using Brain Vision Analyzer (Brain Products GmbH, Gilching, Germany). Each trial was visually inspected for movement artifacts and automatically removed with a ! 75 mV criterion. Eye-movement artifacts were corrected by an algorithm developed by Gratton, Coles, and Donchin (1983). Data were segmented and re-referenced off-line to an average reference. The amplitude of the ERP was derived from each individual’s average waveform and filtered at 1 to 20 Hz with a pre-response baseline between " 200 to 0 ms. The ERN and CRN were scored separately for each participant and defined as the most negative peak 50 to 100 ms after an error and correct response, respectively. The Pe was defined as the most positive peak 200 to 400 ms after an error response. Source Localization Analyses Low Resolution Electromagnetic Tomography (LORETA; Pascual-Marqui, 1991; Pascual-Marqui, Lehmann, Koenig, Kochi, Merlo, et al., 1999) was used to estimate the intracerebral current density underlying the ERN within a 50 to 100 ms window following error responses, which captured the mean peak latency of the ERN at FCz and Cz (77 ms) in the region of interest in the rostral ACC. A time window of 292 to 340 ms was used to capture the mean peak latency of the Pe at FCz and Cz (315 ms). Current density was computed as the linear, weighted sum of the scalp electric potentials at each voxel (units are scaled to amperes per square meter, A/m2). Current density was derived with values normalized for each participant and log-transformed prior to statistical analysis.

D. L. Santesso et al. Table 1. Means (SD) for the Number of Errors Made during the Flanker Task for the ASD and Control Group

Close Close Far Far

Congruent Incongruent Congruent Incongruent

Behavioral Data Response time was calculated from stimulus onset to key press for each condition (e.g., far, close, congruent, incongruent). For

Control

2.69 (1.8) 7.46 (4.2) 3.07 (2.2) 3.38 (2.6)

5.00 (4.5) 9.25 (7.3) 4.06 (4.4) 6.00 (4.7)

the primary analyses, mixed-model ANOVAs were used to analyze the behavioral performance Distance and Congruency as within-subjects factors and Group as a between-subject factor. Table 1 presents the means (SD) for the number of errors for the ASD and control group. For all participants, more errors were made for close compared to far stimuli [F(1,27) 5 22.81, po.001, Zp2 5 .46] and for incongruent compared to congruent stimuli [F(1,27) 5 12.25, p 5 .002, Zp2 5 .31], but a Distance # Congruency interaction revealed that significantly more errors were made for incongruent compared with congruent stimuli in the close (but not far) condition [F(1,27) 5 13.21, p 5 .001, Zp2 5 .33]. There were no significant main effects or interactions with group. Table 2 presents the means (SD) for response times for the ASD and control group. Participants overall had faster response times (RTs) for congruent compared to incongruent trials [F(1,27) 5 7.32, p 5 .012, Zp2 5 .21] and for error compared to correct trials [F(1,27) 5 12.09, p 5 .002, Zp2 5 .31]. A Distance # Group interaction revealed that, whereas for controls RTs differed little depending on the distance, ASD individuals were significantly faster on far compared with close trials [F(1,27) 5 6.00, p 5 .021, Zp2 5 .18]. Finally, there was a Distance # Accuracy # Group interaction [F(1,27) 5 5.64, p 5 .025, Zp2 5 .17], showing that ASD participants had faster error than correct RTs on far trials only (error and correct RTs were similar for close trials), whereas controls had faster error than correct RTs for both far and close trials. ERP Data Participants did not make enough errors in each individual condition to form stable ERPs (over 15 participants made fewer than 5 errors in each condition, except the close incongruent condition; range ASD: 0–15; range control: 0–26), so ERPs were averaged across all conditions. Paired t-tests indicated that the ERN was maximal at FCz [t(28) 5 8.07, po.001], and the Pe was maximal at FCz [t(28) 5 2.58, p 5 .015], so the analyses were restricted to these sites (see Table 2). Figure 1 displays the (A) averaged ERP waveforms and (B) topographic maps for each Table 2. Means (SD) for Response Time (RT) during the Flanker Task for the ASD and Control Group

Distance Close

Congruency Congruent Incongruent

Results

ASD

Far

Congruent Incongruent

Accuracy Correct Error Correct Error Correct Error Correct Error

ASD

Control

416.62 (55.7) 389.92 (12.8) 455.69 (5.0) 400.08 (5.2) 415.00 (4.9) 333.46 (12.3) 431.15 (53.9) 317.77 (15.6)

463.81 (12.8) 315.50 (19.7) 500.88 (12.3) 363.56 (11.7) 467.69 (13.0) 362.89 (15.5) 481.56 (12.6) 406.44 (9.9)

Error monitoring in autism

245

Figure 1. (A) Grand averaged ERP waveforms for error and correct trials during a flanker task at FCz and (B) topographic maps of error trials for ASD and controls.

group, and Table 3 presents the means (SD) of the ERN, CRN, and Pe. An analysis of variance (ANOVA) with Accuracy (correct, error) and Group (ASD, control) revealed a main effect for Accuracy [F(1,27) 5 129. 65, po.001, Zp2 5 .83]. Importantly, there was an Accuracy ! Group interaction [F(1,27) 5 7.23, p 5 .012, Zp2 5 .22], indicating that the ERN was smaller for the ASD group compared with the control group [t(27) 5 2.28, p 5 .03, Zp2 5 .16]. The topographic map shows a weaker frontalcentral negativity in the ASD compared to the control group. No difference between groups was found for the CRN [t(27) 5 .85, p4.40; see Figure 1A).2 An independent t-test revealed that the

2 A peak-to-peak measure of the ERN was also derived by subtracting the ERN negativity from the preceding positivity. This measure is used to account for any differences in the ERN due the preceding positivity. Groups did not differ on this preceding positivity (p4.90), and the ERN at FCz was again smaller for the ASD compared to the control group using a peak-to-peak measure [t(27) 5 2.0, p 5 .05, Zp2 5 .14]. A CRN minus ERN difference score revealed that the difference between components was smaller for the ASD compared to the control group [t(27) 5 2.19, p 5 .038, Zp2 5 .16].

Pe was larger for the control compared to the ASD group [t(27) 5 2.38, p 5 .025, Zp2 5 .17]. Differences in the ERN and Pe were also examined in ASD individuals currently taking antipsychotic medications (n 5 8) compared to those not taking antipsychotic medication (n 5 5). No differences were found in these subgroups for the amplitude of the ERN (p 5 .20) or the Pe (p 5 .93).3 To ensure that group differences were not due to general attenuation of ERP amplitude in the ASD group (due to stimulus information processing, categorization and/or sedative effects of medication), we examined the N1, P2, and N2 (at sites FCz and Cz), and the P3 (at sites Cz and Pz) time-locked to the presentation of the stimulus for correct trials. We found no significant

3 Research has demonstrated that the amplitude of the ERN changes with age (Santesso, Segalowitz, & Schmidt, 2006b) such that older individuals have reduced ERNs despite equivalent accuracy, Pe amplitude, and relative ERN-CRN amplitude (e.g., Mathalon, Bennett, Askari, Gray, Rosenbloom, & Ford, 2003). An ANOVA with age as a covariate showed that group differences remained for the ERN (p 5 .035, Zp2 5 .16) and the CRN-ERN difference (p 5 .048, Zp2 5 .15) while differences in the Pe were slightly attenuated (p 5 .057, Zp2 5 .15).

246

D. L. Santesso et al.

Table 3. Means (SD) of the ERN, CRN, and Pe Amplitude for the ASD and Control Group

ERN FCz Cz CRN FCz Cz Pe FCz Cz

ASD

Control

! 3.97 (1.71) ! 2.98 (1.91)

! 5.53 (1.93) ! 3.29 (1.57)

! 0.17 (1.29) 0.99 (1.55)

0.29 (1.56) 1.11 (1.50)

3.22 (2.34) 3.03 (2.51)

5.59 (2.90) 4.48 (2.28)

group differences in these components (all p-values 4.23, Zp2 values o.05). LORETA Source Localization Figures 2A and B display the difference in source activity underlying the ERN and Pe (respectively) during error trials for the ASD compared to the control group. As can be seen, at the time of the ERN, controls had higher activity in the rostral ACC and medial prefrontal cortex (BA 10) than ASD individuals (maximum difference was found at co-ordinates X,Y,Z 5 ! 10, 38,1; BA 32; t 5 2.73, p 5 .01; see Figure 2A). Similarly for the Pe, the control group showed greater activity in BA 32 and BA 10 (maximum difference was found at co-ordinates X,Y,Z 5 ! 10, 52,8; BA 10; t 5 3.43, po.001; see Figure 2B). No differences in the current density underlying the ERN were found between ASD individuals currently taking medications and those not currently taking antipsychotic medications (p 5 .54). ERN and Current Density Correlations with Autism Spectrum Quotient (AQ) Scores Pearson correlations revealed that better social skills (r 5 .37, p 5 .05) and fewer overall autism symptoms (i.e., lower total AQ score; r 5 .39, p 5 .04) were related to larger ERNs (see Figures 3A and B, respectively). The direction of the relation was similar in both groups, but no significant relations were found when analyzing the groups separately (r-values o.34, p-values 4.20). No significant correlations were found between the ERN and the Attention to detail, Attention switching, and Imagination subscale scores (p-values 4.22). Neither the CRN, nor the Pe, was related to the AQ scores (all p-values 4.12).4 Current density underlying the ERN was related to social skills, such that better social skills were associated with higher activity of this region (r 5 ! .62, po.001) for all participants. When analyzing the groups separately, this effect was significant only for the ASD group (r 5 ! .72, p 5 .006) but not the control group (r 5 .14, p 5 .62; see Figure 4A). Lower overall AQ scores were also associated with higher activity of this region (r 5 ! .43, p 5 .02) for all participants, with the ASD group (r 5 ! .63, p 5 .02) but not the control group showing a significant relation (r 5 .34, p 5 .19; see Figure 4B). No relations were found between the other AQ subscales and ACC activity (all p-values 4.08). Finally, current density underlying the Pe was not related to AQ scores (all p-values 4.16). 4 Given previous findings linking the ERN with IQ in ASD children, we examined the correlation between the ERN and IQ (verbal, nonverbal, total) scores in adults but failed to find any significant relations (p-values 4.27).

Discussion The present study used an event-related design with high-density source localization techniques to investigate ACC abnormalities during response monitoring in DSM-IV diagnosed high-functioning adults with ASD and controls. Despite comparable behavioral performance on the flanker task, individuals with ASD generated smaller ERN amplitudes than controls and reduced activity in the rostral region of the ACC during error detection. On correct trials, adults with ASD also showed less differentiation between error and correct ERP components (i.e., CRNERN amplitude difference). These results are consistent with a recent ERP source localization study of children with ASD (Vlamings et al., 2008), who used an auditory decision task. Vlamings and colleagues (2008) reported that children with ASD had reduced ERNs compared with control children. We found that adults with ASD also did not distinguish well between correct and error trials, due to reduced ERN activity. More importantly, current density underlying the ERN was reduced in the ASD group. Together, these studies provide converging evidence for ACC hypoactivity underlying general response monitoring in adults and children with ASD, suggesting a persistent or stable problem versus a developmental delay. The ACC not only has a central role in decision making, action selection, and engaging cognitive control on high-conflict and/or error trials (Carter & van Veen, 2007), but it is central to decision making and action within a social context. Structural and functional dissociations of the ACC have been described: a dorsal ‘‘cognitive’’ subdivision is responsible for conflict and error monitoring; the rostral ‘‘affective’’ division is responsible for evaluating the motivational and/or emotional salience of errors and emotion regulation (see Bush et al., 2000; Devinsky et al., 1995 for a review). Consistent with this view and studies linking rostral ACC activity to affective responses to errors in special populations (e.g., Fitzgerald et al., 2005; Holmes & Pizzagalli, 2008; Laurens et al., 2003), we found that poorer social skills and greater overall autism symptom severity were related to rostral ACC hypoactivity as indicated by source analysis, particularly in the ASD group. Although a relation was found between the amplitude of the ERN and social skills and overall symptoms for all participants, this ERP relation was not specific to the ASD group. However, whereas scalp ERP measures such as the ERN reflect the mixture of signals from various neural generating regions, source localization techniques such as LORETA may allow researchers to uncover information not available in traditional scalp measures by focusing on regions of interest that have been shown to have relevant functional distinctions (Pizzagalli, 2007). In this case, the rostral ACC has been implicated in medial frontal negativities including the ERN (Gehring & Willoughby, 2002; van Veen & Carter, 2002), with strong relations to personality, mood, social context, and performance (Pizzagalli, Peccoralo, Davidson, & Cohen, 2006). The source analysis results found in the present study thus imply that reduced rostral ACC function in ASD (sometimes reflected in the ERN itself) may reflect a disturbance in the affective or motivational component of error monitoring that could contribute to the severity of autistic symptoms and social impairments. The results of the present study converge not only with results of other neurophysiological studies of autism (e.g., Vlamings et al., 2008), but also with theories emphasizing affective influences on the ERN. ERP studies have linked reduced ERNs with compromised social behavior (Dikman & Allen, 2000; Santesso et al.,


247

Figure 2. Results of voxel-by-voxel independent t-tests contrasting current density underlying the (A) ERN and (B) the Pe for the ASD and control group during a flanker task. Red: relatively higher activity for the ASD group. Blue: relatively higher activity for the control group. Statistical map is unthresholded and displayed on the MNI template.

2005) and lack of empathy (Santesso & Segalowitz, 2009) consistent with fMRI findings of ACC hypoactivity in low empathic individuals (Singer, Seymour, O’Doherty, Kaube, Dolan, & Frith, 2004; Vollm, Taylor, Richardson, Corcoran, Stirling, et al., 2006). Autistic symptoms (as indexed by the AQ and other measures) have been found to be related to low empathy in a number of independent studies (Baron-Cohen & Wheelwright, 2004; Baron-Cohen et al., 2001; Lombardo, Barnes, Wheelwright, & Baron-Cohen, 2007; Wheelwright, Baron-Cohen, Goldenfeld, Delaney, Fine, et al., 2006), leading some researchers to argue that ASD is an empathy disorder (e.g., Baron-Cohen & Wheelwright, 2004; Blacher, Kraemer, & Schalow, 2003; Gillberg, 1992). However, others have argued that ASD is characterized by a lack of complex emotions and less regulation of emotions (Hill, Berthoz, & Frith, 2004). The present study found that social impairments, particularly in ASD individuals, were related to reduced rostral ACC activity. Taken together, it is clear that heterogeniety exists among ASD individuals in their social and empathic skills (e.g., Rogers, Dziobek, Hassenstab, Wolf, & Convit, 2007). Accordingly, it would be useful to consider the influence of various social skills when examining the ERN and ACC function in ASD and control individuals, whether within or between groups. There is converging evidence for ACC abnormalities in ASD. First, using ERPs only, Henderson et al. (2006) reported that the ERN was enhanced in high-functioning children with ASD, but

this relation was limited to those with high verbal IQ, suggesting that some cognitive processes might be spared in some cases of autism and may moderate the relation between ERN activity and social behavior. Second, in an fMRI study, Thakkar et al. (2008) reported that ASD individuals showed reduced discrimination between error and correct responses; this effect was primarily due to abnormally increased activation on correct trials in the rostral region of the ACC (rACC). In this study, however, ASD individuals made a larger number of errors compared to controls, which might have increased task uncertainty and may have influenced ACC activity on correct trials. The authors also reported that increased activation on correct trials was related to higher ratings of repetitive behavior. Here, we did not include a measure of stereotypical, obsessive-compulsive or repetitive behavior, and thus could not determine whether subtypes of ASD individuals (or characteristically different individuals) would show a different pattern of ACC activation during response monitoring. Thakkar et al. did not report any relation between social impairment and ACC activation. In the present study, ACC abnormalities in ASD individuals were found using the scalp ERN and current density, with the severity of social and overall autism symptoms relating to rostral ACC hypoactivity. The present study also found differences in the Pe between ASD and control individuals, (but when age was accounted for, group differences were attenuated). Additionally, ASD individuals showed reduced rostral ACC and medial prefrontal cortex

248

D. L. Santesso et al.

Figure 3. Scatter plot of the relation between ERN at FCz and the Autism Quotient (AQ) (A) social skills and (B) total AQ scores for the ASD and control group.

Figure 4. Scatter plot of the relation between current source density (log A/m2) in the rostral ACC and the Autism Quotient (AQ) (A) social skills and (B) total AQ scores for the ASD and control group.

activity underlying the Pe. These findings are consistent with Vlamings et al. (2008), who reported a reduction in Pe amplitude in ASD versus control children. These findings suggest that individuals with ASD differ from controls in error awareness, allocation of attention to errors and/or post-error behavioral adjustments (e.g., Falkenstein et al., 2000). However, since we did not have a measure of error awareness or post-error behavior, it is impossible to substantiate this hypothesis. Additional studies (using a narrower age range) are needed to determine whether the Pe is reliably different between ASD and control individuals. Several limitations of the present study should be noted. First, the sample size in the present study was small, so the results within group should be interpreted with caution. Future studies with larger sample sizes are needed in order to determine correlates of performance monitoring, with a particular emphasis on known associations among social skills, IQ, and cognitive/affective empathy in individuals with ASD. Second, although the AQ may serve as a useful instrument in identifying the extent of some

autistic traits shown by adults of intelligence, perhaps a more sensitive measure can be used in order to examine what aspects of social skills deficits relate to the ERN and ACC activity in autism spectrum disorders. Third, we did not include a measure of repetitive behaviors, which is a defining feature of autism/ASD. The inclusion of such a measure might have been useful to determine whether characteristically different individuals with ASD show the same pattern of ERN/ACC activity (see Thakkar et al., 2008). Fourth, some ASD participants were currently taking atypical antipsychotic medication that block dopaminergic signaling (risperidone, quetiapine). It has previously been reported such antipsychotic medications (e.g., haloperidol) reduce the amplitude to the ERN in healthy adult participants (de Bruijn, Sabbe, Hulstijn, Ruigt, & Verkes, 2006). Although we found no differences in ERN amplitude between those currently taking antipsychotic medications and those who were not, the sample size was extremely small in each group possibly limiting reliable comparisons. Future studies should assess the effects of long-term dopamine antagonist use on action monitoring and


249

social functioning in a large sample of ASD individuals. Finally, Schwartz, Henderson, Inge, Zahka, Coman, et al. (2009) recently documented a distinct temperament profile in high-functioning ASD individuals: ASD individuals report lower surgency and effortful control, but report higher fear and negative reactivity than controls. Based on previous findings, temperamental differences may influence affective responses to errors and thus the amplitude of the ERN (and perhaps ACC activation, e.g., Hajcak, McDonald, & Simons, 2004; Santesso & Segalowitz, 2009). Future studies should also include temperamental assessments to determine whether temperament biases ACC/ERN responses in ASD. We found that high-functioning adults with autism spectrum disorder exhibited reduced ACC activity, as reflected in the

attenuated ERN and Pe and current density measures, compared to controls. The present results appear to be the first set of empirical findings linking reduced ACC activity underlying the ERN to social deficits in some adults exhibiting autistic traits. Our findings support general theories emphasizing the affective/ social influences on the ERN and suggest that reduced ACC activity may reflect a putative brain mechanism involved in the origins, maintenance, and the severity of autism impairments, particularly among some ASD individuals. Moreover, given a similar pattern of findings noted recently in children with ASD (Vlamings et al., 2008), our findings with adults raise the possibility of the presence of a stable brain-behavior impairment across development rather than a development delay in some adults with ASD.

REFERENCES Allain, S., Carbonnell, L., Falkenstein, M., Burle, B., & Vidal, F. (2004). The modulation of the Ne-like wave on correct responses foreshadows errors. Neuroscience Letters, 372, 161–166. American Psychiatric Association. (1994). Diagnostic and Statistical Manual of Mental Disorders, 4th ed. (DSM-IV). Washington, DC: Author. Baron-Cohen, S., & Wheelwright, S. (2004). The empathy quotient: An investigation of adults with Asperger syndrome or high functioning autism, and normal sex differences. Journal of Autism and Developmental Disorders, 34, 163–175. Baron-Cohen, S., Wheelwright, S., Skinner, R., Martin, J., & Clubley, E. (2001). The autism spectrum quotient (AQ): Evidence from Asperger syndrome/high functioning autism, males and females, scientists and mathematicians. Journal of Autism and Developmental Disorders, 31, 5–17. Bernstein, P. S., Scheffers, M. K., & Coles, M. G. H. (1995). ‘‘Where did I go wrong?’’ A psychophysiological analysis of error detection. Journal of Experimental Psychology, 21, 1312–1322. Birrell, J., & Brown, V. (2000). Medial-frontal cortex mediates perceptual attention set shifting in the rat. Journal of Neuroscience, 20, 4320– 4324. Blacher, J., Kraemer, B., & Schalow, M. (2003). Asperger syndrome and high functioning autism: Research concerns and emerging foci. Current Opinion in Psychiatry, 16, 535–542. Botvinick, M., Braver, T. S., Barch, D. M., Carter, C. S., & Cohen, J. D. (2001). Conflict monitoring and cognitive control. Psychological Review, 108, 624–652. Botvinick, M., Nystrom, L. E., Fissell, K., Carter, C. S., & Cohen, J. D. (1999). Conflict monitoring versus selection-for-action in anterior cingulate cortex. Nature, 402, 179–181. Bryson, S. E., Wainwright-Sharp, J. A., & Smith, I. M. (1990). Autism: A developmental spatial neglect syndrome? In J. Enns (Ed.), The development of attention: Research and theory (pp. 405–427). Amsterdam: Elsevier. Bush, G., Luu, P., & Posner, M. (2000). Cognitive and emotional influences in the anterior cingulate cortex. Trends in Cognitive Science, 4, 214–222. Carter, C. S., & van Veen, V. (2007). Anterior cingulate cortex and conflict detection: An update of theory and data. Cognitive, Affective and Behavioral Neuroscience, 7, 367–379. Craik, F., Moroz, T., Moscovich, M., Stuss, D., Winocur, G., Tulving, E., & Kapur, S. (1999). In search of the self: A positron emission tomography study. Psychological Science, 10, 26–34. Dawson, G., Meltzoff, A., Osterling, J., Rinaldi, J., & Brown, E. (1998). Children with autism fail to orient to naturally-occurring social stimuli. Journal of Autism and Developmental Disorder, 28, 479–485. de Bruijn, E. R. A., Sabbe, B. G. C., Hulstijn, W., Ruigt, G. S. F., & Verkes, R. J. (2006). Effects of antipsychotic and antidepressant drugs on action monitoring in healthy volunteers. Brain Research, 1105, 122–129. Devinsky, O., Morrell, M. J., & Vogt, B. A. (1995). Contributions of anterior cingulate cortex to behavior. Brain, 118, 279–306.

DiGirolamo, G., Kramer, A., Barad, V., Cepeda, N., Weissman, D., Milham, M., et al. (2001). General and task specific frontal lobe recruitment in older adults during executive processes: A fMRI investigation of task-switching. NeuroReport, 12, 2065–2071. Dikman, Z. V., & Allen, J. J. (2000). Error monitoring during reward and avoidance learning in high- and low-socialized individuals. Psychophysiology, 37, 43–54. Falkenstein, M., Hoormann, J., Christ, S., & Hohnsbein, J. (2000). ERP components on reaction errors and their functional significance: A tutorial. Biological Psychology, 51, 87–107. Fitzgerald, K., Welsh, R., Gehring, W., Abelson, J., Himle, J., Liberzon, I., & Taylor, S. (2005). Error-related hyperactivity of the anterior cingulate cortex in obsessive-compulsive disorder. Biological Psychiatry, 57, 287–294. Frith, C., & Frith, U. (1999). Interacting minds: A biological basis. Science, 286, 1692–1695. Frith, U., & Frith, C. (2001). The biological basis of social interaction. Current Directions in Psychological Science, 10, 151–155. Gehring, W. J., Himle, J., & Nisenson, L. G. (2000). Action-monitoring dysfunction in obsessive-compulsive disorder. Psychological Science, 11, 1–5. Gehring, W. J., & Willoughby, A. R. (2002). The medial frontal cortex and the rapid processing of monetary gains and losses. Science, 295, 2279–2282. Gillberg, C. (1992). Autism and autistic-like conditions: Subclasses among disorders of empathy. Journal of Child Psychology and Psychiatry, 33, 813–842. Gratton, G., Coles, M. G., & Donchin, E. (1983). A new method for offline removal of ocular artifact. Electroencephalography and Clinical Neurophysiology, 55, 468–484. Hajcak, G., McDonald, N., & Simons, R. F. (2004). Error-related psychophysiology and negative affect. Brain and Cognition, 56, 189–197. Haznedar, M. M., Buchsbaum, M. S., Metzger, M., Solimando, A., Spiegel-Cohen, J., & Hollander, E. (1997). Anterior cingulate gyrus volume and glucose metabolism in autistic disorder. American Journal of Psychiatry, 154, 1047–1050. Haznedar, M., Buchsbaum, M., Wei, T., Hof, P., Cartwright, C., Bienstock, C., & Hollander, E. (2000). Limbic circuitry in patients with autism spectrum disorders studied with positron emission tomography and magnetic resonance imaging. American Journal of Psychiatry, 157, 1994–2001. Henderson, H. A., Schwartz, C. B., Mundy, P., Burnette, C., Sutton, S. K., Zahka, N., & Pradella, A. (2006). Response monitoring, the error-related negativity, and differences in social behavior in autism. Brain and Cognition, 61, 96–109. Henderson, L., Yoder, P., Yale, M., & McDuffie, A. (2002). Getting the point: Electrophysiological correlates of protodeclarative pointing. International Journal of Developmental Neuroscience, 20, 449–458. Herrmann, M. J., Rommler, J., Ehlis, A. C., Heidrich, A., & Fallgatter, A. J. (2004). Source localization (LORETA) of the error-relatednegativity (ERN/Ne) and positivity (Pe). Cognitive Brain Research, 20, 294–299.

250 Hill, E., Berthoz, S., & Frith, U. (2004). Brief report: Cognitive processing of own emotions in individuals with autistic spectrum disorder and in their relatives. Journal of Autism and Developmental Disorders, 34, 229–235. Hochman, E. Y., Eviatar, Z., Breznitz, Z., Nevat, M., & Shaul, S. (2009). Source localization of error negativity: Additional source for corrected errors. NeuroReport, 20, 1144–1148. Holmes, J. A., & Pizzagalli, D. A. (2008). Spatiotemporal dynamics of error processing dysfunctions in major depressive disorder. Archives of General Psychiatry, 65, 179–188. Holroyd, C., & Coles, M. (2002). The neural basis of human error processing: Reinforcement learning, dopamine and the error related negativity. Psychological Review, 109, 679–709. Holroyd, C. B., & Coles, M. G. (2007). Dorsal anterior cingulate cortex integrates reinforcement history to guide voluntary behavior. Cortex, 44, 548–559. Holroyd, C. B., Nieuwenhuis, S., Yeung, N., Nystrom, L., Mars, R. B., Coles, M. G., & Cohen, J. D. (2004). Dorsal anterior cingulate cortex shows fMRI response to internal and external error signals. Nature Neuroscience, 7, 497–498. Johnson, S., Baxter, L., Wilder, L., Pipe, J., Heiserman, J., & Prigatano, G. (2002). Neural correlates of self-reflection. Brain, 125, 1808–1814. Klin, A., Jones, W., Schultz, R., & Volkmar, F. (2003). The enactive mind, or from actions to cognition: Lessons from autism. Philosophical Transaction of the Royal Society of London, 10, 1–16. Koechlin, E., Basso, G., Peirini, P., Panzer, S., & Grafman, J. (1999). The role of the anterior prefrontal cortex in human cognition. Nature, 399, 148–151. Landry, R., & Bryson, S. E. (2004). Impaired disengagement of attention in young children with autism. Journal of Child Psychology and Psychiatry, 45, 1115–1122. Laurens, K. R., Ngan, E. T., Bates, A. T., Kiehl, K. A., & Liddle, P. F. (2003). Rostral anterior cingulate cortex dysfunction during error processing in schizophrenia. Brain, 126, 610–622. Leslie, A. (1987). Pretense and representation: The origins of theory of mind. Psychological Review, 94, 412–426. Leuthold, H., & Sommer, W. (1999). ERP correlates of error processing in spatial S-R compatibility tasks. Clinical Neurophysiology, 110, 342–357. Levitt, J. G., O’Neill, J., Blanton, R. E., Smalley, S., Fadale, D., McCracken, J. T., et al. (2003). Proton magnetic resonance spectroscopic imaging of the brain in childhood autism. Biological Psychiatry, 54, 1355–1366. Lombardo, M. V., Barnes, J. L., Wheelwright, S. J., & Baron-Cohen, S. (2007). Self-referential cognition and empathy in autism. PLoS One, 2, e883. Luu, P., Collins, P., & Tucker, D. M. (2000). Mood, personality, and selfmonitoring: Negative affect and emotionality in relation to frontal lobe mechanisms of error monitoring. Journal of Experimental Psychology General, 129, 43–60. Luu, P., Flaisch, T., & Tucker, D. M. (2000). Medial frontal cortex in action monitoring. Journal of Neuroscience, 20, 464–469. Luu, P., Tucker, D. M., Derryberry, D., Reed, M., & Poulsen, C. (2003). Activity in human medial frontal cortex in emotional evaluation and error monitoring. Psychologial Science, 14, 47–53. Mathalon, D. H., Bennett, A., Askari, N., Gray, E. M., Rosenbloom, M. J., & Ford, J. M. (2003). Response-monitoring dysfunction in aging and Alzheimer’s disease: An event-related potential study. Neurobiology of Aging, 24, 675–685. Miltner, W. H. R., Lemke, U., Weiss, T., Holroyd, C., Scheffers, M. K., & Coles, M. G. H. (2003). Implementation of error-processing in the human anterior cingulate cortex: A source analysis of the magnetic equivalent of the error-related negativity. Biological Psychology, 64, 157–166. Mundy, P. (1995). Joint attention and social-emotional approach behavior in children with autism. Development and Psychopathology, 7, 63–82. Mundy, P. (2003). Annotation: The neural basis of social impairments in autism: The role of the dorsal medial-frontal cortex and anterior cingulate system. Journal of Child Psychology and Psychiatry, 44, 793–809. Mundy, P., Card, J., & Fox, N. (2000). EEG correlates of the development of infant joint attention skills. Developmental Psychobiology, 36, 325–338.

D. L. Santesso et al. Mundy, P., Sigman, M., Ungerer, J., & Sherman, T. (1986). Defining the social deficits of autism: The contribution of nonverbal communication measures. Journal of Child Psychology and Psychiatry, 27, 657–669. Ohnishi, T., Matsuda, H., Hashimoto, T., Kunihiro, T., Nishikawa, M., Uema, T., & Sasaki, M. (2000). Abnormal regional cerebral blood flow in childhood autism. Brain, 123, 1838–1844. Olvet, D. M., & Hajcak, G. (2009). The effect of trial-to-trial feedback on the error-related negativity and its relationship with anxiety. Cognitive and Affective Behavioral Neuroscience, 9, 427–433. Pailing, P. E., & Segalowitz, S. J. (2004). The error-related negativity as a state and trait measure: Motivation, personality, and ERPs in response to errors. Psychophysiology, 40, 84–95. Pascual-Marqui, R. D. (1991). Review of methods for solving the EEG inverse problem. International Journal of Bioelectromagnetism, 1, 75– 86. Pascual-Marqui, R. D., Lehmann, D., Koenig, T., Kochi, K., Merlo, M. C., Hell, D., et al. (1999). Low resolution brain electromagnetic tomography LORETA functional imaging in acute, neuroleptic-naive, first-episode, productive schizophrenia. Psychiatry Research: Neuroimaging, 90, 169–179. Pizzagalli, D. A. (2007). Electroencephalography and high-density electrophysiological source localization. In: J. T. Cacioppo, L. G. Tassinaru, & G. G. Berntson (Eds.), Handbook of psychophysiology (3rd ed, pp. 56–84). Cambridge, UK: Cambridge University Press. Pizzagalli, D. A., Peccoralo, L. A., Davidson, R. J., & Cohen, J. D. (2006). Resting anterior cingulate activity and abnormal responses to errors in subjects with elevated depressive symptoms: A 128-channel EEG study. Human Brain Mapping, 27, 185–201. Rincover, A., & Ducharme, J. M. (1987). Variables influencing stimulus over-selectivity and ‘tunnel vision’ in developmentally delayed children. American Journal of Mental Deciency, 91, 422–430. Rogers, K., Dziobek, I., Hassenstab, J., Wolf, O. T., & Convit, A. (2007). Who cares? Revisiting empathy in Asperger Syndrome. Journal of Autism and Developmental Disorders, 37, 709–715. Roid, G., & Pomplun, M. (2005). Interpreting the Stanford-Binet Intelligence Scales. In D. P. Flanagan & P. L. Harrison (Eds.), Contemporary intellectual assessment: Theories, tests and issues (5th ed, pp. 325–343). New York: Guilford Press. Santesso, D. L., & Segalowitz, S. J. (2009). The error-related negativity is related to risk taking and empathy in young men. Psychophysiology, 46, 143–152. Santesso, D. L., Segalowitz, S. J., & Schmidt, L. A. (2005). ERP correlates of error monitoring in 10-year-olds are related to socialization. Biological Psychology, 70, 79–87. Santesso, D. L., Segalowitz, S. J., & Schmidt, L. A. (2006a). Enhanced error-related ERPs in 10-year-old children with high obsessive-compulsive behaviors. Developmental Neuropsychology, 29, 431–445. Santesso, D. L., Segalowitz, S. J., & Schmidt, L. A. (2006b). Errorrelated electrocortical responses in 10-year-old children and young adults. Developmental Science, 9, 473–481. Schwartz, C. B., Henderson, H. A., Inge, A. P., Zahka, N. E., Coman, D. C., Kojkowski, N. M., et al. (2009). Temperament as a predictor of symptomology and adaptive functioning in adolescents with highfunctioning autism. Journal of Autism and Developmental Disorders, 39, 842–855. Segalowitz, S. J., Santesso, D. L., Murphy, T. I., Homan, D. G., Chantziantoniou, D. K., & Khan, S. (2010). Retest reliability of medial frontal negativities during performance monitoring. Psychophysiology, 47, 260–270. Simms, M. L., Kemper, T. L., Timbie, C. M., Bauman, M. L., & Blatt, G. J. (2009). The anterior cingulate cortex in autism: Heterogeneity of qualitative and quantitative cytoarchitectonic features suggests possible subgroups. Acta Neuropathologica, 118, 673–684. Singer, T., Seymour, B., O’Doherty, J., Kaube, H., Dolan, R. J., & Frith, C. D. (2004). Empathy for pain involves the affective but not sensory components of pain. Science, 303, 1157–1162. Smith, H., & Milne, E. (2009). Reduced change blindness suggests enhanced attention to detail in individuals with autism. Journal of Child Psychology and Psychiatry, 50, 300–306. Stemmer, B., Segalowitz, S. J., Witzke, W., & Scho¨nle, P. W. (2004). Error detection in patients with lesions to the medial prefrontal cortex: An ERP study. Neuropsychologia, 42, 118–130. Thakkar, K. N., Polli, F. E., Joseph, R. M., Tuch, D. S., Hadjikhani, N., Barton, J. J., & Manoach, D. S. (2008). Response monitoring, re-

Error monitoring in autism petitive behaviour and anterior cingulate abnormalities in autism spectrum disorders (ASD). Brain, 131, 2464–2478. Turken, A. U., & Swick, D. (2008). The effect of orbitofrontal lesions on the error-related negativity. Neuroscience Letters, 441, 7–10. Ullsperger, M., & von Cramon, D. Y. (2003). Error monitoring using external feedback: Specific roles of the habenular complex, the reward system, and the cingulate motor area revealed by functional magnetic resonance imaging. Journal of Neuroscience, 23, 4308–4314. van Veen, V., & Carter, C. S. (2002). The timing of action-monitoring processes in the anterior cingulate cortex. Journal of Cognitive Neuroscience, 14, 593–602. Vlamings, P. H., Jonkman, L. M., Hoeksma, M. R., van Engeland, H., & Kemner, C. (2008). Reduced error monitoring in children with autism spectrum disorder: An ERP study. European Journal of Neuroscience, 28, 399–406.

251 Vollm, B. A., Taylor, A. N. W., Richardson, P., Corcoran, R., Stirling, J., McKie, S., et al. (2006). Neuronal correlates of theory of mind and empathy: A functional magnetic resonance imaging study in a nonverbal task. NeuroImage, 29, 90–98. Wheelwright, S., Baron-Cohen, S., Goldenfeld, N., Delaney, J., Fine, D., Smith, R., et al. (2006). Predicting Autism-Spectrum Quotient (AQ) from the Systemizing Quotient-Revised (SQ-R) and Empathy Quotient (EQ). Brain Research, 1079, 47–56.

(Received August 18, 2009; Accepted March 10, 2010)


Dynamic causal modeling of spontaneous fluctuations in skin conductance

DOMINIK R. BACH,a JEAN DAUNIZEAU,a NADINE KUELZOW,b KARL J. FRISTON,a and RAYMOND J. DOLANa a

Wellcome Trust Centre for Neuroimaging, University College London, London, United Kingdom Institute for Psychology and Ergonomics, Technical University of Berlin, Berlin, Germany

b

Abstract Spontaneous fluctuations (SF) in skin conductance are often used to index sympathetic arousal and emotional states. SF are caused by sudomotor nerve activity (SNA), which is a direct indicator of sympathetic arousal. Here, we describe a dynamic causal model (DCM) of how SNA causes SF, and apply variational Bayesian model inversion to infer SNA, given empirically observed SF. The estimated SNA bears a relationship to the number of SF as derived from conventional (semi-visual) analysis. Crucially, we show that, during public speaking induced anxiety, the estimated number of SNA bursts is a better predictor of the (known) psychological state than the number of SF. We suggest dynamic causal modeling of SF potentially allows a more precise and informed inference about arousal than purely descriptive methods. Descriptors: SCR, Galvanic skin responses, GSR, Electrodermal activity, EDA

this can, in principle, be realized using model inversion methods that map observed fluctuations in skin conductance to underlying SNA. This approach is now frequently employed in neuroimaging within the framework of dynamic causal modeling (DCM) (Friston, Harrison, & Penny, 2003). At the heart of DCM is a causal model, also referred to as a generative or forward model, which describes a mapping from underlying causes (i.e., neural states) to empirical observations (e.g., BOLD response, EEG waveform, or SF). In the case under consideration here, the model predicts observed SF, given SNA. Inverting the causal model yields the reverse mapping from observations to the (most likely) underlying causes; in our case, the inversion SF ! 7 SNA describes the (most likely) generative sudomotor nerve activity, given observed skin conductance. The key difference between previously proposed models for event-related skin conductance changes where event timing is known (Bach, Flandin, Friston, & Dolan, 2009; Lim et al., 1997) and the model considered here is that both timing and amplitude of SNA bursts have to be estimated from the data. Deconvolution methods afford such estimates, as they try to recover the precise SNA time series from the skin conductance data (Alexander et al., 2005; Benedek & Kaernbach, 2009). Our approach represents an informed Bayesian deconvolution, which rests on parameterizing the SNA in a way that allows a quantitative description of the underlying state. This parameterization places constraints on inferred SNA and decreases the degrees of freedom of the model, which increases the precision of model parameter estimates, especially when analyzing noisy data. In this paper, we describe a DCM for SF, with two goals. First, we wanted to show that a DCM for skin conductance can explain data from different individuals and experiments and to motivate further research into the underlying physiology.

Changes in skin conductance are common indicators of sympathetic arousal whose proximal cause is changing activity of sweat glands innervated by the sympathetic branch of the autonomic nervous system (ANS). The number of spontaneous fluctuations (SF) in skin conductance is among the most widely used measures of tonic ANS activity (for an overview, see Boucsein, 1992) and is thought to reflect variations in arousal stemming from a variety of cognitive and emotional processes. SF are sensitive to small changes in arousal (Boucsein, 1992), and play an important role in inferring stress (Boucsein, 1992) and anxiety (Erdmann & Baumann, 1996). SF occur in the absence of external events, and are preceded by firing bursts of sudomotor nerve activity (SNA), innervating the respective skin region (Macefield & Wallin, 1996; Nishiyama, Sugenoya, Matsumoto, Iwase, & Mano, 2001; Ogawa & Sugenoya, 1993). On this basis, a facility to directly assess SNA instead of SF should provide a closer approximation to underlying autonomic states. In the absence of invasive methods We are grateful to Prof. Gisela Erdmann who provided us with two datasets from her laboratory for reanalysis, and to Guillaume Flandin for stimulating comments on this work. This research was funded by a Programme Grant to R.J.D. from the Wellcome Trust and in part by a Personal Grant to D.R.B. from the Swiss National Science Foundation. The data reanalyzed in this article were acquired while D.R.B. and N.K. were pursuing an MSc degree at the Institute for Psychology and Ergonomics, Technical University of Berlin, Germany. Address correspondence to: Dominik R. Bach, Wellcome Trust Centre for Neuroimaging, 12 Queen Square, London WC1N 3BG, United Kingdom. E-mail: [email protected] Re-use of this article is permitted in accordance with the Terms and Conditions set out at http://wileyonlinelibrary.com/onlineopen#Online Open_Terms 252

DCM of skin conductance fluctuations Second, we sought to establish external validity of the model. We hypothesized that estimates of the underlying autonomic state based on DCM predict (known) psychological states more accurately than estimates from conventional methods. To allow other researchers to perform similar evaluations, the method is included as function scr_sf_dcm.m in the software suite SCRalyze, which is freely available under the GNU general public license from http://scralyze.sourceforge.net.

Methods Forward Neural Model The duration and shape of SNA firing bursts is not well described; one study reported a duration of 637 ! 37 ms (Macefield & Wallin, 1996), although from figures in this and other reports

253 (Nishiyama et al., 2001; Ogawa & Sugenoya, 1993) it seems that burst duration can extend up to 1.5–2 s. The number of SNA bursts differs between these studies, from 3 ! 0.5 per minute (Nishiyama et al., 2001) to 22 ! 4 per minute (Macefield & Wallin, 1996). In the absence of precise knowledge about SNA bursts, we make the simplifying assumption that they differ in amplitude but have a fixed temporal profile, and modeled them as Gaussian bump functions with a standard deviation of 0.3 s and a maximum frequency of 30 bursts per minute. Figure 1A shows a burst with unit amplitude (i.e., that would cause an SF with amplitude of 1 mS). Forward Response Model No simultaneous recordings of SF and SNA have addressed how the shape of the ensuing SF relates to bursting, but there is some indirect evidence that SF have a largely constant shape (Bach, Friston, & Dolan, 2010) and that overlapping skin conductance

Figure 1. (A) Modeled sudomotor nerve firing burst of unit amplitude that is assumed to cause a spontaneous fluctuation of 1 mS amplitude. (B) Green: canonical response function for a single spontaneous fluctuation, derived from the first dataset by using an uninformed finite impulse response model and specifying SF onsets using conventional (semi-visual) analysis. Blue: analytical approximation to this function obtained by optimizing the parameters of a third-order ordinary differential equation using a Bayesian inversion scheme. (C) Estimated SNA for a sample epoch. (D) Empirical skin conductance for this epoch, and estimated skin conductance obtained by DCM using the estimated SNA shown in panel C and the SF function shown in panel B. (E) Correlation between the number of responses revealed by conventional analysis and DCM as a function of the threshold for detecting a response. (F) External validity for the number of responses revealed by conventional analysis and DCM inversion as a function of the threshold for detecting a response.

254 changes build up in a linear fashion (Bach, Flandin, Friston, & Dolan, 2010), such that SF can be regarded as a product of a linear time-invariant system, although this needs to be validated in physiological experiments. The former paper also describes an impulse response function or convolution kernel reflecting the canonical shape of an individual SF at a phenomenological level (i.e., not derived from a biophysical model, but from physiological observations). Note that this canonical SF function (shown in Figure 1B) has a slightly biphasic decay, in line with a recent model of event-related skin conductance responses (Bach, Flandin, Friston, & Dolan, 2010). This biphasic response is predicted by the qualitative pore valve model (Edelberg, 1993), where the steep rise and fall of the skin conductance are caused by rapid opening and closing of sweat duct pores, while a slower recovery is afforded by evaporation of remaining sweat on the skin. Our DCM models the relationship between SNA and SF as a linear time-invariant convolution (Bach, Flandin, Friston, & Dolan, 2009). This was specified in terms of an impulse response function for SF developed previously (Bach, Friston, & Dolan, 2010) and modeled here with a third-order ordinary differential equation (ODE). Figure 1B shows the empirically derived canonical function and its analytic ODE approximation; see the Appendix for details. This ODE is formally equivalent to a biphasic exponentially decaying convolution kernel and therefore captures the biphasic effects described above. The assumed (Gaussian) form of SNA firing bursts and the subsequent ODE convolution that generates observed SF constitute the DCM. The resulting generative model assumes that, in the absence of any SNA, the skin conductance returns to zero, which is not normally the case in SF recordings. We were not interested in this baseline, or its slow drifts, because they are determined not only by SNA but also by peripheral factors (Boucsein, 1992). To remove this confounding data feature, we apply our models to skin conductance time-series that are highpass filtered during recording (where the lowest value of each segment is subtracted). Sustained SF are modeled by repeated, low-amplitude SNA bursts. This is biophysically plausible, but can lead to discrepancies between the estimated number of bursts and the SF number assessed by (semi-)visual methods. Datasets We reanalyzed one previously published (Bach & Erdmann, 2007, 2008) and one unpublished (for a review, see Erdmann & Janke, 2008) dataset from the same laboratory, both of which are based upon a similar paradigm. Dataset 1 served as a training dataset, which we used to optimize the parameters of the ODE that determine the shape of the implicit convolution kernel (see Appendix), and the amplitude threshold for counting responses. Dataset 2 served as an independent validation dataset, which was analyzed using the parameters from the first dataset. Dataset 1 contained four measurements from each of 40 healthy male university students (18–35 years) who participated in a public speaking anticipatory anxiety paradigm with a repeated-measures factorial design. The main focus of this experiment was the interaction of habitual and situational symptom focusing, operationalized as attention towards neck muscle tension. The main experimental manipulation had no effect on indices of skin conductance, and data from the different experimental groups were combined for the present analysis, where we focus on the effect of the public speaking treatment. There were two baseline measurements, one measurement after the announcement of a public speech, and another after dis-

D. R. Bach et al. closure of the speech topic. This manipulation was carried out in order to separate effects of anxiety and cognitive load. Dataset 2 included four measurements for each of 32 healthy female university students (19–29 years) who underwent a similar public speaking experiment in a between-subjects design. That is to say, half of the participants were to deliver a public speech, and the other half a speech without an audience. There was one baseline measurement, one measurement after announcement of the speech, and another after disclosure of the topic. Fourteen of 128 epochs contained motion artefacts and were excluded. Apparatus After skin cleansing with propanol, skin conductance was recorded on thenar/hypothenar of the non-dominant hand using 8 mm Ag/AgCl cup electrodes (Coulbourn, Whitehall, PA) and 0.5% NaCl electrode gel (Par, Berlin, Germany) on thenar/hypothenar of the non-dominant hand; 0.5 V constant voltage was provided by a S77-21 coupler (Coulbourn). The signal was band pass filtered (0.015 and 5 Hz), digitally converted with 10 Hz (Dataset 1) or 100 Hz (Dataset 2) sampling rate (DI-205, Dataq, Akron, OH) and recorded (Windaq, Dataq). Each 60-s epoch was analyzed using a semi-automatic method (Event Detection and Analysis, Trosiener & Kayser, 1993) with a threshold of 0.25 mS. This analysis had already been performed in the context of the original experiments, before the present method was developed, and the corresponding results can be regarded as unbiased. Data Pre-Processing Data analysis was carried out in Matlab (MathWorks, Natick, MA) using custom code that is available from the authors. After import of the 60-s segments into Matlab, the data were again low-pass filtered with a bidirectional first-order Butterworth filter at a cut-off frequency of 5 Hz, and re-sampled to 10 Hz (Dataset 2). No high-pass filtering was applied at this stage (note that data were high-pass filtered during recording). Statistical Analysis The correspondence between conventional and DCM data analyses were summarized with Pearson correlation coefficients between the numbers of detected responses from both methods. Predictive validity was assessed as the correlation between the (known) psychological state and the estimated sympathetic arousal based on the number of responses. This number is estimated by thresholding the continuous estimates of SNA (DCM) or SF (conventional analysis). For the training Dataset 1, the psychological state was defined for each epoch as either baseline or anticipation, and the estimated arousal as number of SNA responses for each epoch. For Dataset 2, which employed a between-subject design (anticipation of public versus anticipation of a non-public speech), psychological state was defined as public or non-public speech with arousal estimated by the mean number of responses in anticipatory epochs minus the number of responses in the baseline epoch. In one participant, the baseline epoch had been excluded such that n 5 31 for this analysis. Relative sensitivity and specificity of the conventional and DCM analyses were quantified using receiver operator characteristics (ROC) curves. Predicting a discrete psychological state from a continuous variable can be reframed by drawing on signal detection theory (Macmillan & Creelman, 2005). Here we tried to classify a given state based on the total number of SF from the conventional analysis and the number of bursts estimated with

DCM of skin conductance fluctuations DCM (both at an amplitude threshold of 0.1 mS). This allowed us to predict the true psychological state (and calculate specificity and sensitivity of that prediction) given the estimated number of responses. Finally, to test whether DCM estimates of autonomic arousal explain more variance in the psychological state than conventional estimates, we computed an F-statistic and its associated p-value by comparing two simple regression models for the two predictor variables. This F-statistic represents the amount of variance in psychological states that is explained by DCM above and beyond the conventional estimates.

Results As described previously, we analyzed the training dataset using an uninformed simple finite impulse response model to estimate a canonical response function (CRF). This requires knowing the onsets of the underlying bursts, which we approximated using a conventional semi-automated analysis of the SF time series (Bach, Friston, & Dolan, 2010). The ensuing CRF was used to optimize the parameters of the DCM so that its implicit convolution kernel matched the CRF (see Appendix). The resulting DCM was then used to deconvolve the time series. Figure 1C and D show DCM inversion for an exemplar epoch and give an overview of data fit and the estimated SNA generating the data. Figure 1E shows the correspondence between the estimated number of (above-threshold) SNA bursts and the SF number estimated by conventional analysis as a function of the threshold used to detect bursts. The correspondence between the two measures increases with increasing threshold and plateaus from a value of about 0.1 mS upwards. In Figure 1F, we depict predictive (external) validity of both methods as the Pearson correlation between the number of estimated responses and the class (baseline or anticipation) to which the epoch belongs. The conventional analysis has better predictive validity at low thresholds. This probably reflects the fact that sustained responses (which could be due to peripheral factors alone) are modeled as sustained SNA. However, from around a threshold of 0.1 mS, validity of DCM response estimates is higher than that of the conventional method. For illustrative purposes, results from both methods for a threshold of 0.1 mS are shown separately for the four measurement periods in Figure 2A. The ROC curves illustrating the trade-off between sensitivity and specificity at a threshold of 0.1 mS are shown in Figure 2C. An ROC curve that is closer to the upper left corner of the diagram indicates better prediction. Thus, the ROC curves point to higher validity of the DCM estimates. We next validated the model using an independent dataset (Dataset 2) using the optimized parameters from Dataset 1. This is important since the CRF used to optimize the DCM parameters was derived from the same dataset to which the DCM was applied. Although the CRF was based on a large number of responses (1153 SF), its generalizability has to be confirmed. Across the second dataset, the correlation between the number of responses (at a threshold of 0.1 mS) detected by conventional analysis and DCM was r 5 .67. Predictive validity (i.e., the ability to predict whether an individual was subjected to public speech anticipation or non-public speech anticipation) was r 5 .29 for the conventional and r 5 .50 for the DCM method. Thus, DCM estimates explained a higher proportion of variance than results from the conventional method (F(1,30) 5 6.6;

255 po.05). Results from both methods are depicted in Figure 2B for illustrative purposes, and Figure 2D corroborates the higher validity for DCM inversion in terms of ROC curves for this dataset. Discussion In this paper, we present a dynamic causal model of skin conductance fluctuations SF and demonstrate that its inversion can be used to predict known psychological states. Crucially, our method showed a significantly higher predictive validity than that afforded by a conventional analysis. This advantage reflects the fact that sudomotor nerve firing is more closely related to the underlying psychological state than the ensuing SF, and suggests that SNA can be inferred from SF, using variational Bayesian inversion of our generative model. We note a high correlation between both methods in the training dataset; an unsurprising observation given that the response function used to optimize the DCM was developed from a conventional analysis. This correspondence between the two methods was much lower in the second dataset, while at the same time the predictive validity of model inversion was relatively higher. Note that, in contradistinction to previously proposed approaches, our goal was not to emulate conventional analysis or perfectly fit the data, but to extract meaningful information about psychological states from the data.1 We were successful in this aim for both datasets, which necessarily led to a lower correlation with conventional methods. Two factors may account for an enhanced predictive validity of our method: one is that any subjective element is removed from analysis, and the other is a suppression of noise through model constraints (i.e., parameterization of the unknown SNA). This contrasts with previous deconvolution approaches that try to recover unconstrained SNA estimates (Alexander et al., 2005; Benedek & Kaernbach, 2009), an approach that might be more susceptible to measurement noise. An interesting extension of the model presented here would be to estimate the parameters of the DCM from the data being analyzed (as opposed to optimizing them using some estimated or assumed CRF, as in this paper). This might enhance the model fit, but possibly reduce the precision of the estimators of the neural states. While inverting a DCM is computationally expensive, the ensuing quantification of the autonomic state is more precise than that afforded by previously proposed simple methods (i.e., area under the curve, Bach, Friston, & Dolan, 2010). Our DCM rests on physiological observations, which in part relate to biophysical models but are not entirely explained by such models. This means that the physiological realism of the DCM could be much improved. Nevertheless, our model can be generalized to any independent dataset acquired from healthy young populations with similar experimental set-ups. The generalizability to qualitatively different populations (e.g., patients) and measurement methods needs to be tested further. We have shown for event-related responses that one canonical response function can fit data from different recording sites (Bach, Flandin, Friston, & Dolan, 2010) and this might even be more tenable for SF, in which response latency is not an issue. On the other hand, since 1 Actually, it is well known in the statistical community that overfitting, i.e., the tendency to fit both the signal and the noise in the data, leads to strongly biased estimation and inference (see, e.g., Carlin & Louis, 2000).

256

D. R. Bach et al.

Figure 2. Number of responses estimated by conventional analysis and the DCM with a threshold of 0.1 mS. (A) Training dataset. BL: baseline measurements, AM1 after announcement of public speech, AM2 after announcement of speech topic. (B) Validation of the method on an independent dataset. BL: baseline measurement, AM1 and AM2 after announcement of speech, AM3 after announcement of speech topic. Solid line: non-public speech (control condition); dashed line: public speech. (C) ROC curves for the training dataset. (D) ROC curves for the second dataset.

filtering influences the shape of the response, it seems crucial to use similar constant voltage measurement and filter settings as the ones applied here, in order to use our DCM parameters. In addition, when quantifying autonomic states from the DCM, the (arbitrary) amplitude threshold used here needs validation for different recording sites and measurement equipment. In general, we would like to encourage other researchers to refine the forward model. Different models can easily be compared in this framework by their likelihood, given the data, and by their predictive validity. To our knowledge, this is the first report of a biophysically motivated generative model for peripheral physiological parameters of psychological states. Such dynamic causal modeling (Daunizeau, Friston, & Kiebel, 2009; Friston et al., 2003) is

becoming standard in neuroimaging, with applications for the analysis of fMRI, EEG/MEG (Chen, Kiebel, & Friston, 2008; Daunizeau, Kiebel, & Friston, 2009; David et al., 2006; Kiebel, Garrido, Moran, Chen, & Friston, 2009; Penny, Litvak, Fuentemilla, Duzel, & Friston, 2009), and electrophysiological data (Moran et al., 2009). The power of such approaches lies in the estimation of causes and unknown (hidden) states by inversion of a mapping from causes to observations. This mapping enables one to place key biophysical constraints on the models and its associated estimators. Furthermore, the parameters and states of these models have a direct and useful biological interpretation. Thus, DCM allows for a wide range of possible implementation in psychophysiology, which we hope to exploit with this work.

REFERENCES Alexander, D. M., Trengove, C., Johnston, P., Cooper, T., August, J. P., & Gordon, E. (2005). Separating individual skin conductance responses in a short interstimulus-interval paradigm. Journal of Neuroscience Methods, 146, 116–123. Bach, D. R., & Erdmann, G. (2007). Influences of habitual and situational bodily symptom focusing on stress responses. Cognition and Emotion, 21, 1091–1101. Bach, D. R., & Erdmann, G. (2008). Subjective bodily symptoms during anticipation of public speaking: Influence of habitual symptom perception and actual symptom focusing [Article in German]. In W. Janke, M. Schmidt-Daffy, & G. Debus (Eds.), Experimentelle Emotionspsychologie (pp. 603–615). Lengerich, Germany: Pabst.

Bach, D. R., Flandin, G., Friston, K., & Dolan, R. J. (2009). Time-series analysis for rapid event-related skin conductance responses. Journal of Neuroscience Methods, 184, 224–234. Bach, D. R., Flandin, G., Friston, K. J., & Dolan, R. J. (2010). Modelling event-related skin conductance responses. International Journal of Psychophysiology, 75, 349–356. Bach, D. R., Friston, K. J., & Dolan, R. J. (2010). Analytic measures for quantification of arousal from spontaneous skin conductance fluctuations. International Journal of Psychophysiology, 76, 52–55. Benedek, M., & Kaernbach, C. (2009). Decomposition of skin conductance data by means of nonnegative deconvolution. Psychophysiology. DOI 10.1111/j.1469-8986.2009.00972.x.

DCM of skin conductance fluctuations Boucsein, W. (1992). Electrodermal activity. Berlin: Springer. Carlin, B. P., & Louis, T. A. (2000). Bayes and empirical Bayes methods for data analysis. New York: Chapman & Hall/CRC Press. Chen, C. C., Kiebel, S. J., & Friston, K. J. (2008). Dynamic causal modelling of induced responses. NeuroImage, 41, 1293–1312. Daunizeau, J., Friston, K. J., & Kiebel, S. J. (2009). Variational Bayesian identification and prediction of stochastic nonlinear dynamic causal models. Physica D, 238, 2089–2118. Daunizeau, J., Kiebel, S. J., & Friston, K. J. (2009). Dynamic causal modelling of distributed electromagnetic responses. NeuroImage, 47, 590–601. David, O., Kiebel, S. J., Harrison, L. M., Mattout, J., Kilner, J. M., & Friston, K. J. (2006). Dynamic causal modeling of evoked responses in EEG and MEG. NeuroImage, 30, 1255–1272. Edelberg, R. (1993). Electrodermal mechanisms: A critique of the twoeffector hypothesis and a proposed replacement. In J. C. Roy, W. Boucsein, D. C. Fowles, & J. H. Gruzelier (Eds.), Progress in electrodermal research (pp. 7–29). New York: Plenum Press. Erdmann, G., & Janke, W. (2008). Public speaking as an anxiety-specific paradigm [Article in German]. In W. Janke, M. Schmidt-Daffy, & G. Debus (Eds.), Experimentelle Emotionspsychologie (pp. 297–322. Lengerich, Germany, Pabst. Erdmann, G., & Baumann, S. (1996). Are psychophysiologic changes in the ‘‘public speaking’’ paradigm an expression of emotional stress? [Article in German]. Zeitschrift fur Experimentelle Psychologie, 43, 224–255. Friston, K., Mattout, J., Trujillo-Barreto, N., Ashburner, J., & Penny, W. (2007). Variational free energy and the Laplace approximation. NeuroImage, 34, 220–234. Friston, K. J., Harrison, L., & Penny, W. (2003). Dynamic causal modelling. NeuroImage, 19, 1273–1302. Kiebel, S. J., Garrido, M. I., Moran, R., Chen, C. C., & Friston, K. J. (2009). Dynamic causal modeling for EEG and MEG. Human Brain Mapping, 30, 1866–1876. Lim, C. L., Rennie, C., Barry, R. J., Bahramali, H., Lazzaro, I., Manor, B., & Gordon, E. (1997). Decomposing skin conductance into tonic and phasic components. International Journal of Psychophysiology, 25, 97–109. Macefield, V. G., & Wallin, B. G. (1996). The discharge behaviour of single sympathetic neurones supplying human sweat glands. Journal of the Autonomic Nervous System, 61, 277–286. Macmillan, N. A., & Creelman, C. D. (2005). Detection theory. New York: Lawrence Erlbaum. Moran, R. J., Stephan, K. E., Seidenbecher, T., Pape, H. C., Dolan, R. J., & Friston, K. J. (2009). Dynamic causal models of steady-state responses. NeuroImage, 44, 796–811. Nishiyama, T., Sugenoya, J., Matsumoto, T., Iwase, S., & Mano, T. (2001). Irregular activation of individual sweat glands in human sole observed by a videomicroscopy. Autonomic Neuroscience, 88, 117–126. Ogawa, T., & Sugenoya, J. (1993). Pulsatile sweating and sympathetic sudomotor activity. Japanese Journal of Physiology, 43, 275–289. Penny, W. D., Litvak, V., Fuentemilla, L., Duzel, E., & Friston, K. (2009). Dynamic causal models for phase coupling. Journal of Neuroscience Methods, 183, 19–30. Trosiener, H.-J., & Kayser, J. (1993). Analysis and interactive control of electrodermal and cardiac activity: A computer program for IBMPCs and compatibles. Journal of Psychophysiology, 7, 171.

257 data. The sum of these bursts is evaluated at each time point t and forms the parameterized input u(t, y) to the skin conductance function: uðt; yÞ ¼

n X

ðt$ti Þ2 2s2

ai e$

i¼1

y ¼ fti ; ai g : i ¼ 1; . . . ; n We assumed a fixed number of n 5 30 SNA bursts per minute. This is therefore the maximum number of detectable responses in the data. If there are fewer than n SN bursts in the data, the amplitude of any extra bursts would be estimated as zero. (2) The skin conductance time series is thought to result from a double convolution operation applied on the sudomotor nerve activity u(t, y). This is modeled as a third-order ordinary differential equation (ODE) with parameters Wi : i 2 1; 2; 3 :::

x þW1 x€ þ W2 x_ þ W3 x $ uðyÞ ¼ 0

ð1Þ

where we have dropped explicit time notation, and x is related to the measured skin conductance time series y using the following (trivial) observation function: y¼xþe

ð2Þ

Here, e is a residual error term. The parameters W were optimized so that they reproduced the canonical SF response function described previously (Bach, Friston, & Dolan, 2010), using the variational Bayes scheme described below: i.e., treating the canonical response function as data y(t) 5 CRF(t) and Wi as unknown parameters with input u(t) 5 d(0). The ensuing posterior estimates of these parameters are: 8 < W1 ¼ 2:1594 W ¼ 3:9210 : 2 W3 ¼ 0:9236 Then, the model was inverted using a variational Bayesian inversion scheme described in Friston, Mattout, Trujillo-Barreto, Ashburner, and Penny (2007). In brief this entails: & Using Gaussian assumptions about the residual errors in the observation process, Equations 1 and 2 are compiled to derive a likelihood function p(y|y), which measures the likelihood of a set of observed SF y, given parameters y. & Defining priors p(y) on the model parameters, which enable one to derive the posterior probability density function (pdf) over the evolution parameters: pðyj yÞ / pðyjyÞpðyÞ:

ð3Þ

(Received October 13, 2009; Accepted February 12, 2010)

APPENDIX Our generative model comprises the following elements: (1) Each of n SNA bursts is modeled as a Gaussian function with a standard deviation of s 5 0.3 s, while the amplitude a and the time of maximum firing t are estimated from the

The posterior pdf p(y|y) measures how likely any particular value of the unknown parameter y is, given the measured times-series of SF. & Having estimated the unknown parameters of the model, we can then define an estimator uˆ of the unknown time-series of sudomotor nerve activity: u^ ¼ E ½uðyÞj y(:

ð4Þ


Listening to urban soundscapes: Physiological validity of perceptual dimensions

AMY IRWIN,a DEBORAH A. HALL,a,b ANDREW PETERS,c and CHRISTOPHER J. PLACKd a

MRC Institute of Hearing Research, University Park, Nottingham, UK Division of Psychology, School of Social Sciences, Nottingham Trent University, Nottingham, UK c Sir Peter Mansfield Magnetic Resonance Centre, School of Physics and Astronomy, University of Nottingham, Nottingham, UK d Division of Human Communication and Deafness, The University of Manchester, Manchester, UK b

Abstract Predominantly, the impact of environmental noise is measured using sound level, ignoring the influence of other factors on subjective experience. The present study tested physiological responses to natural urban soundscapes, using functional magnetic resonance imaging and vector cardiogram. City-based recordings were matched in overall sound level (71 decibel A-weighted scale), but differed on ratings of pleasantness and vibrancy. Listening to soundscapes evoked significant activity in a number of auditory brain regions. Compared with soundscapes that evoked no (neutral) emotional response, those evoking a pleasant or unpleasant emotional response engaged an additional neural circuit including the right amygdala. Ratings of vibrancy had little effect overall, and brain responses were more sensitive to pleasantness than was heart rate. A novel finding is that urban soundscapes with similar loudness can have dramatically different effects on the brain’s response to the environment. Descriptors: Cognition, Sensation/perception, Normal volunteers, fMRA/PET/MRI, Heart rate

ample, increasing level (measured in decibels, dB or Ldn, a composite measurement of noise exposure) has been correlated with increased ratings of annoyance (Fidell, 1978; Kryter, 1982; Vastfjall, Kleiner, & Garling, 2003). It is therefore perhaps unsurprising that sound level (measured using a dB A-weighted scale) is the primary metric required by U.K. legislation (Planning Policy Guidance 24)1 when determining the success of a planning application. However, reducing the negative impact of an urban space by acoustic means presents a real challenge to architects and planners because reduced sound levels are difficult to achieve in practice. It has been reported that 68% of the urban population in England and Wales are still exposed to levels above the 45-dB recommendation of the World Health Organization (Skinner & Grimwood, 2001). An alternative approach considers positive as well as negative reactions to urban soundscapes. This perspective does not simply consider the level of the sound, but also acknowledges the importance of the type of sound source for determining the cognitive and emotional interpretations placed on individual soundscapes (Dubois, Guastavino, & Raimbult, 2006). This approach favors the term ‘‘positive soundscape’’ (Raimbult & Dubois, 2005). For example, natural sounds such as those produced by people (e.g., speech) and animals (e.g., birdsong) tend to be rated as preferable to mechanical sounds (e.g., traffic noise; (Dubois et al., 2006; Nilsson & Berglund, 2005; Yang & Kang, 2005). As yet, however, this knowledge has not been utilized in

The perceptual experience of an urban environment encompasses a range of sensory cues, including those in the visual, acoustic, and olfactory domains. The term ‘‘soundscape’’ is widely used in auditory ecology to refer to the sonic environment (Schafer, 1977). In many ways, it can be considered the auditory equivalent of a visual landscape. The acoustic experience of the environment includes sounds from a variety of sources, including transportation, people, and nature (Raimbult & Dubois, 2005). Soundscapes are frequently considered in terms of their negative impact on the environment such as causing annoyance or sleep disturbance (Botteldooren, Coensel, & De Muer, 2006). The auditory system is highly responsive to sound level (Brechmann, Baumgart, & Scheich, 2002), and several studies have also demonstrated that the cognitive and emotional interpretations of an urban soundscape are strongly determined by its level. For exBoth Irwin and Hall contributed equally to this work. The positive soundscapes project was funded through a multidisciplinary EPSRC grant (EP/E011624/1). John Foster and Kay Head assisted in collecting the physiological data. The authors give special thanks to Ron Coxon at the Sir Peter Mansfield Magnetic Resonance Imaging Centre, University of Nottingham, for his initial assistance with the software for generating the appropriate summary measures from the heart rate data. Some of the fMRI results were presented as preliminary reports at the Experimental Psychology Society meeting, York, UK in July 2009 and the 38th International Congress and Exposition on Noise Control Engineering (Internoise), Ottowa, Canada in August 2009. Address correspondence to: Deborah A. Hall, Division of Psychology, Nottingham Trent University, Burton Street, Nottingham, NG1 4BU, UK. E-mail: [email protected]

1 Planning Policy Guidance 24: Planning and Noise (1994) www. communities.gov.uk/publications/planningandbuilding/ppg24.

258

Listening to urban soundscapes practical terms to develop new approaches for creating pleasing sound aesthetics as a more viable alternative to reducing sound levels (Hill, 2007). The cognitive and emotional responses to a stimulus (be it a soundscape, an individual sound source, or even a picture) are typically measured using visual analog scales. Most of these laboratory-based studies have measured responses to visual stimuli, for example, pictures depicting a pleasant event such as a child playing or an unpleasant event such as a car crash (Bradley & Lang, 1994; Lang, Greenwald, Bradley, & Hamm, 1993). Analogue scales typically use semantic dimensions based on ratings of pleasantness (valence) or vibrancy (arousal) over a 5- or a 9point scale (Bellezza, Greenwald, & Banaji, 1986; Bradley & Lang, 1994; Carles, Barrio, & de Lucio, 1999; Mirz, Gjedde, Sodkilde-Jo¨rgensen, & Pedersen, 2000; Ouis, 2001; Yang & Kang, 2005). The use of such scales is based on past research that has consistently reported the application of these two main factors to describe emotional experience (Bradley & Lang, 2000; see also Cacioppo, Gardner, & Berntson, 1999, for a review). When people are presented with emotionally evocative pictures, reliable patterns of physiological change are found in somatic, visceral, and central nervous systems that differ according to the reported valence and arousal associated with each image. Somatic responses have been measured using facial electromyographic measurements (frowning), whereas recordings of skin conductance (sweating) and heart rate provide measures of visceral change (Anders, Eippert, Weiskopf, & Veit, 2008; Bradley, Codispoti, Cuthbert, & Lang, 2001; Cuthbert, Schupp, Bradley, Birbaumer, & Lang, 2000; Lang et al., 1993). Typically, corrugator muscle activity (indicative of frowning) occurs more when people view unpleasant stimuli compared to pleasant or neutral stimuli (Bradley et al., 2001), skin conductance increases, and heart rate over the stimulus period decelerates more rapidly (Bradley & Lang, 2000). Changes in the central nervous system have been examined using a range of brain mapping techniques, such as event-related brain potentials (Cuthbert et al., 2000; Keil et al., 2002), functional magnetic resonance imaging (fMRI; Lang et al., 1998; Tabert et al., 2001), and positron emission tomography (PET; Lane, Chua, & Dolan, 1999; Mirz et al., 2000). In the majority of these studies, stimuli were separated into three categories of pleasantness (unpleasant, neutral, and pleasant) to assess the influence of valence. Analysis was often further simplified by collapsing unpleasant and pleasant stimuli into one ‘‘affective/emotion-laden’’ category. For example, Cuthbert et al. reported that viewing emotion-laden images prompted a significant late positive potential, increased skin conductance, and significant reduction in heart rate when the images were unpleasant. Such methodology potentially reduces sensitivity by collapsing the stimulus set into broad categories. Similar data have been reported using fMRI; for example, one such study reported an increase in activity in the visual cortex and amygdala when people viewed pleasant and unpleasant pictures relative to neutral pictures (Sabatinelli, Bradley, Fitzsimmons, & Lang, 2005). The amygdala is significant here because it is known to mediate emotional responses to sensory stimuli (Marsh, Fuzessery, Grose, & Wenstrup, 2002; Zald, 2003), and so this finding indicates a functional association between sensory and emotional centers of the brain. A similar study that utilized facial expressions of fear and disgust (Philips et al., 1998) found activity in the amygdala in response to fearful expressions and activation in the right anterior insula in response to disgusted expressions. The anterior insula has also been implicated in the

259 ability to empathize with others by creating an internal representation of their emotional state (Carr, Iacoboni, Dubeau, Mazziotta, & Lenzi, 2003; Singer et al., 2004). It is interesting to note that Carr et al. reported coactivation of insula and amygdala during this condition. The insula and amygdala often seem to coactivate during emotional experiences and so are considered as brain regions of interest in the present study. Further examination of the above studies highlights a potential discrepancy in the effect of emotionally evocative pictures on the visceral system and the central nervous system. Anders et al. (2008) reported no effect on skin conductance but a significant neural response to stimuli evoking positive and negative emotions compared to neutral stimuli. A number of other studies have reported a visceral reaction specific to unpleasant stimuli and have confirmed the neural response to positive as well as to negative valanced stimuli (Bradley et al., 2001; Sabatinelli et al., 2005). The measures might possibly represent different aspects of the perceiver’s response, with visceral changes reflecting an appetitive, or positive, response to pleasant stimuli and a defensive, or negative, response to unpleasant stimuli (Bradley & Lang, 2000), whereas changes in brain activity could reflect a general response of motivated attention toward emotionally evocative stimuli (Lang et al., 1998). Methodological differences in the data sampling might also contribute to the discrepancy. Whereas visceral measures, such as heart rate, tracked changes on a moment-to-moment basis throughout the stimulus period, an integrated measure of brain activity was obtained over the whole stimulus epoch. The body of visual research has prompted a similar examination of the relationship between the perceived pleasantness of sound stimuli and the resulting physiological responses. Although such auditory research is considerably less extensive than in the visual domain, the results are reasonably consistent across modalities (e.g., Bradley & Lang, 2000; Mirz et al., 2000; Sander & Scheich, 2001). One difference may be that visceral changes are not quite as large for sounds as for pictures. For example, Bradley and Lang (2000) reported that the effect of pleasantness (valence) on the change in heart rate was restricted to those stimuli that were rated as highly arousing (high vibrancy). Another difference is that brain imaging studies of emotionally evocative sounds have typically focused on the examination of unpleasant or aversive sounds relative to neutral sounds (Mirz et al., 2000) and have not often considered pleasant sounds. The same is true of animal models of amygdalar functionality, with reports of a feedback loop between the amygdala and inferior colliculus when unpleasant sounds are presented to the animal (Marsh et al., 2002). Reports of an amygdala response to pleasant sounds are less consistent, and often the result reflects an equivalent response for emotionally evocative sounds (Anders et al., 2008; Fecteau, Belin, Joanette, & Armony, 2007; Sander & Scheich, 2001) rather than differential responses according to ratings of valence. Interestingly, one fMRI study selected stimuli according to their properties of valence and arousal (Anders et al., 2008). Results indicated that the amygdala was more highly responsive to stimuli rated for positive or negative valence compared with neutral valence, but there was no significant difference between stimuli rated as arousing or not. Hence, it is suggested that that the amygdala signals stimulus valence rather than arousal. To our knowledge, the present study reports the first evaluation of the neural basis of the cognitive and emotional response to a wide range of soundscape stimuli recorded in naturalistic urban environments. In keeping with the positive soundscape

260 approach, we were concerned to rule out the contribution of sound level to the emotional response, and so our physiological study was conducted using a set of stimuli carefully matched in level. Urban soundscape clips were carefully selected to vary across the perceptual dimensions of pleasantness (valence) and arousal (see Bradley & Lang, 2000). Whereas previous authors (e.g., Anders et al., 2008; Cuthbert et al., 2000) have simply classified pleasantness in three categories (negative, neutral, and positive) and reported series of pairwise comparisons, we used five categories to enable a finer sampling of responses across the pleasantness dimension and tested for a single significant quadratic trend. The main experiment sought to identify the pattern of visceral and neural changes associated with those measures. The metric for the visceral response was heart rate, and the metric for the neural response was brain activity. A priori, significant changes in brain activity were expected within the central auditory system (inferior colliculus, medial geniculate body, and auditory cortex), the amygdala, and the insula cortex. The current literature does not provide clear predictions about the precise nature of the relationship between brain activity and perceptual experience, and so the data set was explored using a broader range of statistical contrasts than have been reported hitherto within individual studies. To rule out some of the methodological differences between previous studies of heart rate and brain activity changes, we measured responses using the same set of sound stimui and the same set of listeners.

Methods Stimuli A set of 219 recordings of British urban soundscapes were acquired from a variety of archival sources.2 A range of different everyday contexts such as the street, market, shopping mall, and park were represented. All sound recordings were sampled at high fidelity (44.1 kHz, 16-bit resolution) and were then digitally edited to an 8-s duration with 50-ms onset and offset ramps. For the small number of clips that contained only a single sound source, urban street noise was added as a background to generate a true soundscape. Stimulus characterization was performed without further digital editing, and so the reader should note that the overall (root-mean-square) sound level ranged by 18 dB across the whole set. For the physiological experiment, it was important that the stimuli spanned a wide range of the pleasantness and vibrancy scales. The strategy used in the present study was to select a subset of stimuli according to their mean pleasantness rating (and also to a lesser extent, vibrancy). To obtain the appropriate ratings data, all 219 clips were rated by 5 participants (aged 21–40 years). The scale was presented as a visual analogue with 9 points (see Appendix 1) and anchor points defined by a set of adjectives (e.g., ‘‘unpleasant, unhappy, unsatisfied’’ and ‘‘pleasant, happy, satisfied’’). Participants were also asked to write down what they judged each soundscape clip to represent.

2 Sources were the personal archive of a soundscape archivist (Peter Cusack), the British Library (www.sounds.bl.uk/Browse.aspx?collection=Soundscapes), the set of International Affective Digital Sounds (www.csea.phhp.ufl.edu/media/iadsmessage.html), a website managed by Trevor Cox (www.sound101.org/badvibes), and a website containing people’s favorite London sounds (www.favouritelondonsounds.org).

A. Irwin et al. The distribution of ratings for pleasantness and vibrancy was normal (mean 5 5.0, SD 5 1.6, and mean 5 5.2, SD 5 1.8, respectively; see Figure 1a). Only two soundscape clips were consistently rated at the negative end of the pleasantness scale (mean rating 5 1, a person vomiting in the street). No clips were rated at the positive end (9) consistently by the 5 participants. However, six clips received a mean rating of 8. Examples were a string quartet playing in a market square and a dawn chorus in the city. This distribution compares favorably with previous sound ratings. For example, Bradley and Lang (2000) reported mean ratings spanning 1.5 (a violent attack) to 7.8 (erotica) on the same 9point scale. As for vibrancy, some people rated a small number of clips (e.g., a door creaking with an airplane flying overhead) at the extreme negative end (1) and others (e.g., a man operating a chainsaw) at the positive (9) end of the scale. However, unlike the ratings for pleasantness, there was rather poor internal reliability in the vibrancy ratings. Prior to the physiological experiment, the set of 219 soundscape clips was reduced to a set of 150 stimuli by discarding many of the clips that had neutral pleasantness ratings in the middle of the distribution and omitting those clips whose informational content appeared rather ambiguous (determined using the written responses of the 5 participants). The 9-point pleasantness scale was reduced to five categories using pragmatic criteria to create 30 soundscape clips in each group (see Figure 1b). The upper and lower cutoffs for each mean score on the 5-point scale were as follows: 1 5 1 to 3.4, 2 5 3.5 to 4.3, 3 5 4.4 to 5.1, 4 5 5.2 to 6, 5 5 6.1 to 8.5. Following others (Anders et al., 2008; Bradley & Lang, 2000), the vibrancy scale was reduced to two categories. The upper and lower cutoffs were as follows: low 5 1 to 5.39 and high 5 5.40 to 9. To provide tighter experimental control, the 150 soundscape clips were adjusted to be presented at a level of 71 dB(A), taken as the overall root-mean-square level averaged over the 8-s duration of each sound clip. Variability in the corresponding stimulus loudness was also minimized (mean 5 22 sones, SD 5 4), according to the sones scale, ISO 532B. Within each pleasantness category, the 30 selected soundscape clips encompassed a range of different urban contexts and contained different dominant sound sources (see Figure 1b). Source types were classified as ‘‘human’’ (e.g., vomiting, talking, giggling), ‘‘natural’’ (e.g., birdsong, wind, thunder), and ‘‘mechanical’’ (e.g., traffic noise, construction). We note that 50% of those soundscape clips rated as unpleasant (! 2) were primarily generated by mechanical sources, whereas the same type of sources accounted for only 18% of the pleasant (" 4) soundscapes. It was not possible to entirely eliminate this sampling bias, although we do not expect it to influence the physiological measures. Participants Sixteen native English-speaking participants were recruited for the physiological experiment. There were 8 women and 8 men, aged 21–55 years. All participants reported normal hearing and none were taking prescription medication or had any history of neurological impairment. The study was approved by the local Medical School ethics committee, and all participants gave informed written consent. Procedure The 150 soundscapes were presented in four sequences to participants in the MR scanner, in scanning blocks that each took

Listening to urban soundscapes

261

Figure 1. A: Two histograms showing the distribution of initial ratings of pleasantness and vibrancy, respectively, made by 5 participants listening to the full set of 219 soundscape clips. B: Selected set of 150 soundscape clips, with the reduced pleasantness scale reduced from 9 points down to 5 points. Each bar is color coded to illustrate the classification of the dominant sound sources.

about 10 min. In all four sequences, two clips were presented in succession, interspersed with 8-s or 16-s periods of silence. The order of presentation was pseudorandomized, with the rule that each pair of clips belonged to the same pleasantness category (e.g., 1,1). Participants were simply instructed to listen to the sounds and remain as still as possible. Our protocol enabled us to record heart beat and brain activity simultaneously while the participant listened to the soundscape clips. To record a vector cardiogram, participants were fitted with four MR-compatible electrodes affixed to the left-hand side of the chest with self-adhesive pads. To record brain activity, participants lay supine in a Philips 3 Tesla MR scanner, equipped with an 8-channel parallel headcoil. Each scan comprised 32 slices oriented in an oblique horizontal direction with an image resolution of 3 ! 3 ! 3 mm. The volume of brain tissue covered by the scan was 192 ! 192 ! 96 mm and it encompassed the main structures of interest (Figure 2). The comparative size of the different regions of interest is reported in Table 1. Between 69 and 79 scans were acquired in each scanning block, depending on the number of stimuli in the sequence. The MR scanner is a rather hostile environment for auditory fMRI because it generates a tonal noise of about 99 dB SPL whenever brain images are acquired. The current experiment used three methods of noise reduction. First, participants wore an MR-compatible, high-fidelity headset built into standard ear defenders that provided up to 40 dB of passive attenuation. Second, the headset also incorporated a bespoke system for active noise cancellation that has been shown to reduce the scanner noise by up at 35 dB, at the ear (Hall et al., 2009). Third, a scanning sequence known as sparse imaging was used. Briefly, this sequence used an 8.55-s interscan interval enabling a set of brain images to be rapidly acquired (acquisition time 5 1971 ms)

at the end of each 8-s soundscape clip with a quiet period when the next clip was presented. This method has been demonstrated to reduce the impact of scanner noise on the pattern of soundevoked brain activity (Hall et al., 1999). On two later occasions, the same participants were asked to listen to the soundscape clips and rate them for pleasantness and vibrancy, respectively, using the original 9-point scales (Appendix 1). The majority of the participant group (n 5 13) did this after the fMRI study to ensure that they were naı¨ ve to the stimuli at the time of the physiological testing. Data Analysis Analysis of the imaging data was conducted using statistical parametric mapping software (SPM5, www.fil.ion.ucl.ac.uk/ spm/) that transforms individual brain images into an internationally recognized spatial coordinate system using a reference scan provided by the Montreal Neurological Institute. Preprocessing included within-subject realignment to correct for head movement during the experiment. To improve the signal-to-noise ratio, data were spatially smoothed by 8 mm, and to reduce the low-frequency drift in the MR signal, data were high-pass filtered using a cutoff of 0.0025 Hz. Analyses were computed using a general linear model approach (Friston, 1997). Individual analysis considered the pattern of sound-related activity (soundscapes4silence) and activity associated with the two perceptual dimensions of pleasantness (five categories) and vibrancy (two categories). Outputs from the individual analyses were pooled together in a repeated measures design to assess the significance of activity with respect to the between-subject variance ( po.05, corrected for multiple comparisons using the false discovery rate; Genovese, Lazar, & Nichols, 2002). All F statistics were appropriately corrected for nonsphericity. The group analysis

262

A. Irwin et al.

Figure 2. Schematic drawing of the human brain shown from two perspectives. A: Ascending auditory pathway shown using a coronal section of the cortex and two horizontal sections of the midbrain. The relative position of the amygdala is also illustrated. B: Positions of the posterior and anterior insula relative to the auditory cortex. These are depicted on a lateral view of the brain with a section of the superior temporal gyrus cut away to reveal the structures within. The vertical dashed line corresponds to the position of the coronal slice shown in A.

contrasts. The criterion for significance was a peak of activity in the region of interest, p ! .05 (corrected using the false discovery rate). Results are reported in Table 2. Sound-related activity was confirmed in all portions of the ascending auditory pathway, including the inferior colliculus, medial geniculate body, and auditory cortex, in both left and right hemispheres. The anterior and posterior insula also responded to the sound stimuli, but crucially no significant activity was found in the amygdala. The incidence map provides further information about the consistency of the sound-related activity across the 16 participants (see Figure 3A). Activity was found predominantly in the auditory cortex, with all participants exhibiting activation in that region of interest. A significant response to sound was also found in the inferior colliculus and posterior insula with 75% (12/16) and 93% (15/16) of participants showing activity at the same voxel location, respectively. Fewer participants exhibited overlapping suprathreshold activity in the medial geniculate body and anterior insula (up to 4/16, 25%). The apparent difference between the results from the group level analysis and the incidence map in these two regions of interest can partly be attributed to the level of subthreshold activity across the other participants. To examine the effect of soundscape listening on mean heart rate, the next step was to contrast the mean heart rate for the sound and the silent conditions (i.e., all soundscapes4silence using a linear mixed effects model). When analyzing these data, it was noted that individual mean heart rate was not stable over time, but declined quite considerably across the four scanning runs (perhaps associated with general reduction in anxiety as the participant became more familiarized with the surroundings).

specifically tested for the presence of sound-related activity and activity associated with pleasantness and vibrancy in a number of anatomically constrained regions of interest (inferior colliculus, medial geniculate body, auditory cortex, insula, and amygdala). Regions of interest are illustrated in Figure 2. In addition, the distribution of activity across individual members of the group was examined by creating ‘‘incidence’’ maps. Incidence maps were generated by summing individual statistical images for all 16 participants, each thresholded at po.01, uncorrected for multiple comparisons (for a full description of this method, see Hall & Plack, 2009). Analysis of the heart rate data was conducted in two ways, first as a measure of mean heart rate for each stimulus presentation and, second, as a measure of heart rate change across the 8s stimulus time period using 2-s time windows (0.1–1.9, 2–3.9, 4– 5.9, 6–7.9; cf. Bradley & Lang, 2000). In both cases, customized software was used to compute the appropriate heart rate measures for each soundscape clip. Results General effects of Soundscape Listening on Brain Activity and Mean Heart Rate The first test for brain activity identified regions in which the response to the soundscapes was significantly greater than the response to the silent condition. This was done by analyzing together the five contrasts (i.e., one category of pleasantness4silence) using a conjunction method (Friston, Penny, & Glaser, 2005). This identified activity that was significant in all five

Table 1. Report of the Comparative Size of the Six Regions of Interest Reported for Left and Right Hemispheres Separately Left hemisphere Region of interest Inferior colliculus Medial geniculate body Auditory cortex Anterior insula Posterior insula Amygdala

Right hemisphere

No. of voxels

Volume (mm3)

No. of voxels

Volume (mm3)

26 17 5,657 8 70 60

208 136 45,256 480 744 560

26 17 5,120 94 89 96

208 136 40,960 752 712 768


263

Table 2. Repeated Measures Group Analysis Examining the Effect of Soundscape Listening on Brain Activity Left hemisphere

Right hemisphere

Coordinates Region of interest Inferior colliculus Medial geniculate body Auditory cortex Anterior insula Posterior insula Amygdala

Coordinates

No. of voxels

x

y

z

Z

p

No. of voxels

x

y

z

Z

p

52 5 3525 33 62 n.s.

2 ! 14 ! 50 ! 38 ! 38 n.s.

! 36 ! 26 ! 24 30 ! 28 n.s.

! 10 !8 6 0 2 n.s.

7.68 2.95 INF (19.47) 2.82 INF (10.90) n.s.

0.0001 0.02 0.0001 0.033 0.0001 n.s.

52 2 3913 11 85 n.s.

2 14 54 48 44 n.s.

! 36 ! 26 ! 14 32 ! 18 n.s.

! 10 !8 4 4 2 n.s.

7.68 3.01 INF (17.49) 3.22 INF (15.24) n.s.

0.0001 0.02 0.0001 0.033 0.0001 n.s.

Statistical outputs are reported for the peak voxel within each region of interest, separately in left and right hemispheres. The three-dimensional coordinates are reported in a standardized stereotaxic brain space. A single voxel measures 2 # 2 # 2 mm in volume. In cases where the Z score is given as infinity (INF), the corresponding T value is also reported in parentheses.

The model therefore included scanning run as an additional control variable. Mean heart rate was significantly increased when participants listened to the soundscapes compared to the silent condition, F(2,27) 5 3.044, po.05, indicating a strong visceral response to soundscape listening (Table 3). The Effect of Pleasantness on Brain Activity and Heart Rate Change A direct statistical comparison between pleasant and unpleasant soundscapes failed to indicate any significant difference (p4.05, corrected using the false discovery rate). An effect that has been

reported in the fMRI literature a number of times is that of a greater response to pleasant and unpleasant stimuli relative to neutral stimuli (Anders et al., 2008; Fecteau et al., 2007; Lang et al., 1998; Sabatinelli et al., 2005). Confirmation of this pattern was sought by using a weighted U-shaped function for the five pleasantness categories. This function was applied to the grouplevel general linear model in SPM5 (i.e., pleasantness dimension: 1, 2, 3, 4, 5; weighting function: 2, 1, ! 6, 1, 2). The amygdala, posterior insula, and auditory cortex were all responsive to soundscapes with a high emotional content (p " .05, corrected using the false discovery rate), although we note that amygdala

Figure 3. Incidence maps showing the distribution of activation across the group of 16 participants. Maps are overlaid onto the mean normalized anatomical scan. A: Contrast between all sound conditions and silence. A range of different slices are shown, with the corresponding axis reported using the standarized coordinate system. The same slices are displayed in B, which shows the result of the analysis assessing the effect of pleasantness. The color scale indicates how many listeners show significant activation at any particular voxel, po.01 uncorrected. Regions of interest are labeled as follows: IC 5 inferior colliculus, MGB 5 medial geniculate body, AC 5 auditory cortex, pAC 5 posterior auditory cortex, ai 5 anterior insula, pi 5 posterior insula, am 5 amygdala. In horizontal and coronal slices the left hemisphere is shown on the right-hand side (radiological convention).

264

A. Irwin et al. However, activity was spatially rather disparate. It occurred in different parts of the amygdala in different participants. The effect of pleasantness on heart rate was assessed using mean heart rate change because this is more resilient to temporal variations in heart rate than is the mean estimate. For comparison with data presented by Bradley and Lang (2000), this analysis was performed on three categories of pleasantness instead of the five used hitherto (unpleasant: 1, 2; neutral: 3; pleasant: 4, 5). Analysis used a univariate ANOVA with pleasantness and time as the main factors and scanning run as an additional control variable. Like the previous study, heart rate change varied significantly over time, F(3,27) 5 2.877, po.05. There was an initial acceleration in heart rate immediately after stimulus onset followed by a sustained reduction in rate after the second time bin, as shown in Figure 5. Heart rate change did not alter significantly as a function of pleasantness, F(4,27) 5 0.214, p4.05, and there was no significant interaction between pleasantness and time, F(12,27) 5 1.404, p4.05. This null result matches that reported by Bradley and Lang (2000). However, these authors reexamined the data set by further splitting the valence (pleasantness) categories according to low and high arousal (vibrancy) with significant results. The next section describes the outcome of this type of analysis applied to both the heart rate and fMRI data.

Table 3. Mean Heart Rate (in Beats per Minute) and Standard Error Measured during the fMRI Experiment for the 16 Participants Condition

Mean

Standard error

Silence 1 (very unpleasant) 2 (unpleasant) 3 (neutral) 4 (pleasant) 5 (very pleasant)

68.29 68.77 68.66 68.59 68.86 68.50

3.393 3.716 3.456 3.573 3.437 3.395

activity did not reach significance in the left hemisphere. An apparent right-hemispheric dominance was seen in terms of more widespread activity in the amygdala and also in the posterior insula (Table 4). A measure of the average response magnitude within the left and right amygdala was extracted for each participant, and these data were submitted to repeated measures analysis of variance (ANOVA), with hemisphere and pleasantness as factors (Figure 4). A test of the within-subjects contrasts confirmed an overall quadratic response to pleasantness, F(1,15) 5 18.745, po.01. Hence, the magnitude of the response appears to be modulated by the intensity of the emotional response. The difference between hemispheres did not reach significance, F(1,15) 5 0.359, p4.05), and there was no significant interaction, F(1,15) 5 0.959, p4.05). Therefore, although the results from the SPM5 analysis appeared to indicate a dominance of the right amygdala in the representation of pleasantness, this did not stand up to scrutiny using a direct statistical test. An incidence map was also generated to visualize the consistency of the U-shaped pattern of activation across participants defined by the SPM5 analysis (Figure 3b). Activity was most consistently found in the posterior part of the auditory cortex (50%, 8/16 participants). Individual suprathreshold activity (po.01, uncorrected) appeared somewhat less consistent in the amygdala and posterior insula (18%–25% of participants, 3–4 out of 16). Again this apparent difference between the grouplevel result and the incidence map can perhaps be attributed to the level of subthreshold activity across participants. One key observation made when we viewed the individual maps of sound-related activity was that the proportion of participants with amygdala activity was somewhat higher than the estimate of consistency reported above. Eight participants (50%) demonstrated activity in the left amygdala, and 7 on the right.

Effects of Pleasantness and Vibrancy on Brain Activity and Heart Rate Change To test the effects of pleasantness and vibrancy on brain activity, the fMRI data set was reanalyzed using a 3 ! 2 mixed-effects ANOVA in SPM5 (Friston, Stephan, Lund, Morcom, & Kiebel, 2005). This group analysis failed to reveal any significant effect of vibrancy (p4.05 FDR corrected). In other words, the neural response within the regions of interest was no different for soundscapes rated as low and high vibrancy (see also Anders et al., 2008). On the other hand, there was a significant effect of pleasantness. This pattern was broadly similar to that shown by the previous analysis. The amygdala and posterior insula in the right hemisphere were highly responsive to the emotional content of the soundscapes, whereas activity in the auditory cortex did not reach statistical significance (Appendix 2). There was no significant interaction between pleasantness and vibrancy. The final analysis of heart-rate change examined the effect of pleasantness using only those soundscapes classified as high vibrancy. The univariate ANOVA confirmed the same pattern of change over time, F(3,29) 5 2.877, po.05 (Figure 5). Contrary

Table 4. Repeated Measures Group Analysis Examining the Effect of Pleasantness on Brain Activity Left hemisphere

Right hemisphere

Coordinates Region of interest Inferior colliculus Medial geniculate body Auditory cortex Anterior insula Posterior insula Amygdala

No. of voxels n.s. n.s. 366 489 48 n.s. n.s. 2

x

" 48 " 60 " 48 " 26

y n.s. n.s. 8 " 44 " 10 n.s. n.s. " 12

Coordinates z

Z

p

No. of voxels

"4 24 " 10

n.s. n.s. 4.19 3.65 2.66 n.s. n.s. 2.35

n.s. n.s. 0.022 0.023 0.035 n.s. n.s. 0.056

n.s. n.s. 808 832

44 68

n.s. 41 42

40 20

" 14

x

y n.s. n.s. "8 " 34 n.s. " 20 "8

z

Z

p

" 10 22

n.s. n.s. 3.96 3.41

n.s. n.s. 0.022 0.026

n.s. 2.98 3.17

n.s. 0.038 0.052

2 " 14

Statistical outputs are reported for the peak voxel within each region of interest, separately in left and right hemispheres. A single voxel measures 2 ! 2 ! 2 mm in volume. The three-dimensional co-ordinates are reported in standardized brain space.


265

Figure 4. Plots of the mean response magnitude for the left and right amygdala across the five categories of pleasantness. The scale on the y-axis represents the parameter estimate of the brain activity, and error bars represent the 95% confidence intervals.

to our expectations, however, there was still no significant effect of pleasantness, F(2,29) 5 1.711, p4.05. The reason for this null effect is unclear, but we might speculate that the testing environment and lying down would have important modulatory effects on the normal heart rate that might mask the rather subtle effects of emotion. Discussion The present study examined physiological responses to the different perceptual and emotional experiences associated with listening to a natural range of urban soundscapes. The most striking outcome demonstrated how the perceived pleasantness

significantly modulated the neural response to the soundscape in a smooth U-shaped manner. Emotionally evocative soundscapes (pleasant and unpleasant) engaged a number of brain centers, namely the amygdala, posterior insula, and posterior auditory cortex. This result differs from previous research examining emotionally evocative auditory stimuli (Anders et al., 2008; Fecteau et al., 2007) because the emotional response was not confined to the amygdala. The auditory system is well known to be highly responsive to sound level (e.g., Brechmann et al., 2002; Hart, Palmer, & Hall, 2002), and sound level is also known to affect the sense of pleasantness (Fidell, 1978; Kryter, 1982; Vastfjall et al., 2003). By carefully normalizing the sound stimuli in the present study, we

Figure 5. The first column illustrates the influence of rated pleasantness upon the mean heart rate change across the full set of soundscapes. The second column illustrates the same split but for the ‘‘high vibrancy’’ items only. Each data point represents the mean change calculated across the defined time window relative to the estimated heart rate at stimulus onset. All graphs are plotted with standard error bars.

266

A. Irwin et al.

are confident that sound level was not a confounding factor affecting the reported perceptual and physiological reactions to pleasantness. Our findings demonstrate that loudness is not the only factor to determine listeners’ experience of their acoustic environment. Emotional interpretations also play a crucial role, and research efforts are now focused on developing appropriate metrics for the eventual routine assessment of soundscapes and their incorporation into environmental and planning policy (Davies et al., 2009). Dissociation between Sensory and Affective Responses to Soundscapes There appears to be a fundamental difference in the way the brain responds to the sensory attributes and the emotional content of the soundscape stimuli. Brain regions in the ascending auditory system (inferior colliculus, medial geniculate body, and auditory cortex) produced a significant response to the acoustic aspects of the stimuli, but not to the rated pleasantness of the same sounds. In contrast, brain regions in the paralimbic system (amygdala and posterior insula) were engaged by the emotional content of the sounds, but not by acoustic input per se. A portion of the posterior auditory cortex responded both to the sounds themselves and to the dimension of pleasantness. These results therefore support a model of sound processing in which there are two networks that are predominantly functionally separate but are nonetheless interactive. Previous research has suggested a functional link between the central auditory system and the amygdala, with the amygdala potentially playing a role in the processing of emotionally evocative acoustic stimuli. As yet, the only published anatomical investigation to confirm such connectivity in the mammalian system has been conducted in pallid bats (Marsh et al., 2002). Cell staining using retrograde tracers confirms a feedback circuit linking the central auditory system and amygdala. Specifically, in this study, the inferior colliculus was found to receive at least two neural projections, one originating in auditory cortex and one in the amygdala. Marsh and colleagues proposed that emotive code projecting from the amygdala could influence the processing of auditory information at a relatively early stage in the ascending auditory pathway. Although the current results should not necessarily be taken as a disproof of this proposal, we saw no evidence in the fMRI data presented here for a modulation of the responses in inferior colliculus according to pleasantness. However, we note that the inferior colliculus is small in size, and the fMRI signal measured in this brain region is rather weak. Although inferior colliculus responses to sound stimuli are often reported for the contrast between sounds and a silent baseline, stimulus-related modulations of the response are generally much less robust (Griffiths, Uppenkamp, Johnsrude, Josephs, & Patterson, 2001). Encoding the Emotional Interpretation of the Stimulus Although our results show a distinct response to the rated pleasantness of the soundscapes, we found no specific response to their

perceived vibrancy. Previous research appears to be divided as to the neural response to vibrancy. Highly arousing (vibrant) visual stimuli have been reported to enhance late positive event-related potentials (Cuthbert et al., 2000). Such stimuli also tend to be recalled with greater success and viewed for longer than their low-arousal counterparts (Lang et al., 1993), leading the authors to conclude that arousal is a determining factor in the brain’s response to affective stimuli. Research using olfactory stimuli has shown amygdala activation to be associated with intensity (akin to arousal and vibrancy), and not pleasantness, of odors (Anderson et al., 2003), leading the authors to suggest that there is a fundamental segregation of the neural representation of arousal and valence. This conclusion is somewhat in opposition to the recent finding by Anders et al. (2008) that the amygdala is sensitive to the valence of pictures and sounds, irrespective of arousal. The results of the present study also support this latter conclusion. Olfaction is perhaps different from vision and audition. First, affective dimensions can be more easily manipulated independently of one another and, second, arousal can be defined relatively free of potentially confounding semantic manipulations, as the concentration of the odor (Anderson et al., 2003). For visual and auditory stimuli, it is difficult to select a stimulus set that is evenly distributed across the two-dimensional space of pleasantness and vibrancy (e.g., Cuthbert et al., 2000). Unpleasant stimuli are typically more arousing and vibrant (e.g., screeching train brakes) than pleasant stimuli (e.g., birdsong). Although Anders et al. (2008) did attempt to balance their emotional categories with respect to individual valence and arousal ratings, they did so in a post hoc manner and did not state how the 40 stimuli were partitioned into the six categories (3 valence ! 2 arousal). Although we used a much larger stimulus set, there were still small numbers of stimuli in the categories for ‘‘unpleasant and vibrant’’ (n 5 12) and ‘‘neutral and vibrant’’ (n 5 7). The rather uneven distribution of stimuli across the multidimensional space does mean that the statistical power was somewhat suboptimal. To resolve the discrepancy about amygdala sensitivities for encoding pleasantness and vibrancy, future research should ensure that stimulus sets are carefully selected in an a priori manner according to their ratings on the two dimensions. To conclude, the present study presents a novel examination of the physiological basis of soundscape perception. The main results demonstrate convincingly how not all soundscapes evoke equal responses in terms of mental activity. Neural response can be differentiated between processing to soundscapes ‘‘as sounds’’ and processing the soundscapes as ‘‘perceptual and emotional entities.’’ For urban architects and planners, the main conclusion is therefore that factors other than sound level influence the perception of urban soundscapes, thus validating the use of a perceptual metric when analyzing the effect of sound on urban dwellers.

REFERENCES Anders, S., Eippert, F., Weiskopf, N., & Veit, R. (2008). The human amygdala is sensitive to the valence of pictures and sounds irrespective of arousal. Social Cognitive and Affective Neuroscience (SCAN), 3, 233–243. Anderson, A. K., Christoff, K., Stappen, I., Panitz, D., Ghahremani, D. G., Glover, G., et al. (2003). Dissociated neural representations of

intensity and valence in human olfaction. Nature Neuroscience, 6, 196–202. Bellezza, F. S., Greenwald, A. G., & Banaji, M. R. (1986). Words high and low in pleasantness as rated by male and female college students. Behaviour Research Methods, Instruments and Computers, 18, 299–303.

Listening to urban soundscapes Botteldooren, D., Coensel, B. D., & De Muer, T. (2006). The temporal structure of urban soundscapes. Journal of Sound and Vibration, 292, 105–123. Bradley, M. M., & Lang, P. J. (1994). Measuring emotion: The selfassessment manikin and the semantic differential. Journal of Behaviour Therapy and Experimental Psychiatry, 25, 49–59. Bradley, M. M., Codispoti, M., Cuthbert, B. N., & Lang, P. (2001). Emotion and motivation I: Defensive and appetitive reactions in picture processing. Emotion, 13, 276–298. Bradley, M. M., & Lang, P. J. (2000). Affective reactions to acoustic stimuli. Psychophysiology, 37, 204–215. Brechmann, A., Baumgart, F., & Scheich, H. (2002). Sound-level-dependent representation of frequency modulations in human auditory cortex: A low-noise fMRI study. Journal of Neurophysiology, 87, 423–433. Cacioppo, J. T., Gardner, W. L., & Berntson, G. G. (1999). Beyond bipolar consceptulizations and measures: The case of attitudes and evaluative space. Personality and Social Psychology Review, 1, 3–25. Carles, J. L., Barrio, I. L., & de Lucio, J. V. (1999). Sound influence on landscape values. Landscape and Urban Planning, 43, 191–200. Carr, L., Iacoboni, M., Dubeau, M. C., Mazziotta, J. C., & Lenzi, G. L. (2003). Neural mechanisms of empathy in humans: A relay from neural systems for imitaion to limbic areas. Proceedings of the National Academy of Sciences, USA, 100, 5497–5502. Cuthbert, B. N., Schupp, H. T., Bradley, M. M., Birbaumer, N., & Lang, P. J. (2000). Brain potentials in affective picture processing: Covariation with autonomic arousal and affective report. Biological Psychology, 52, 95–111. Davies, W. J., Adams, M. D., Bruce, N., Marselle, M., Cain, R., Jennings, P., et al. (2009). The positive soundscape project: A synthesis of results from many disciplines. In Proceedings of Internoise 2009: Innovations in practical noise control. Ottowa, Canada. Dubois, D., Guastavino, C., & Raimbault, M. (2006). A cognitive approach to urban soundscapes: Using verbal data to access everyday life auditory categories. Acta Acoustica United with Acustica, 92, 865–874. Fecteau, S., Belin, P., Joanette, Y., & Armony, J. L. (2007). Amygdala responses to non-linguistic vocalizations. Neuroimage, 36, 480–487. Fidell, S. (1978). Nationwide Urban Noise Survey. Journal of the Acoustical Society of America, 64, 198–206. Friston, K. J. (1997). Testing for anatomically specified regional effects. Human Brain Mapping, 5, 133–136. Friston, K. J., Penny, W. D., & Glaser, D. E. (2005). Conjunction revisited. NeuroImage, 25, 661–667. Friston, K. J., Stephan, K. E., Lund, T. E., Morcom, A., & Kiebel, S. (2005). Mixed effects and fMRI. NeuroImage, 24, 244–252. Genovese, C. E., Lazar, N. A., & Nichols, T. (2002). Thresholding of statistical maps in functional neuroimaging using the false discovery rate. NeuroImage, 15, 870–878. Griffiths, T. D., Uppenkamp, S., Johnsrude, I., Josephs, O., & Patterson, R. D. (2001). Encoding of the temporal regularity of sound in the human brainstem. Nature, 4, 633–637. Hall, D., Haggard, M. P., Akeroyd, M., Palmer, A. R., Summerfield, Q. A., Elliot, M. R., et al. (1999). ‘‘Sparse’’ temporal sampling in auditory fMRI. Human Brain Mapping, 7, 213–223. Hall, D., & Plack, C. (2009). Pitch processing sites in the human auditory brain. Cerebral Cortex, 19, 576–585. Hall, D. A., Chambers, J., Foster, J., Akeroyd, M. A., Coxon, R., & Palmer, A. R. (2009). Acoustic, psychophysical and neuroimaging measurements of the effectiveness of active cancellation during auditory functional magnetic resonance imaging. Journal of the Acoustical Society of America, 125, 347–359. Hart, H., Palmer, A. R., & Hall, D. A. (2002). Heschl’s gyrus is more sensitive to tone level than non-primary auditory cortex. Hearing Research, 171, 177–190.

267 Hill, A. (2007). Why we love sounds of the city jungle. The Observer, September 23. Keil, A., Bradley, M. M., Hauk, O., Rockstroh, B., Elbert, T., & Lang, P. J. (2002). Large-scale neural correlates of affective picture processing. Psychophysiology, 39, 641–649. Kryter, K. D. (1982). Community annoyance from aircraft and ground vehicle noise. Journal of the Acoustical Society of America, 72, 1222– 1242. Lane, R. D., Chua, P. M., & Dolan, R. J. (1999). Common effects of valence, arousal and attention on neural activation during visual processing of pictures. Neuropsychologia, 37, 989–997. Lang, P., Greenwald, M. K., Bradley, M. M., & Hamm, A. O. (1993). Looking at pictures: Affective, visceral and behavioral reactions. Psychophysiology, 30, 261–273. Lang, P. J., Bradley, M. M., Fitzsimmons, J. R., Cuthbert, B. N., Scott, J. D., Moulder, B., et al. (1998). Emotional arousal and activation of the visual cortex: An fMRI analysis. Psychophysiology, 35, 199–210. Marsh, R. A., Fuzessery, Z. M., Grose, C. D., & Wenstrup, J. J. (2002). Projection to the inferior colliculus from the basal nuclues of the amygdala. Journal of Neuroscience, 22, 10449–10460. Mirz, F., Gjedde, A., Sodkilde-Jo¨rgensen, H., & Pedersen, C. B. (2000). Functional brain imaging of tinnitus-like perception induced by aversive auditory stimuli. NeuroReport, 11, 633–637. Nilsson, M. E., & Berglund, B. (2005). Assessment of outdoor soundscapes in quiet areas (A). Journal of the Acoustical Society of America, 117, 2592–2592. Ouis, D. (2001). Annoyance from road traffic noise: A review. Journal of Environmental Psychology, 21, 101–120. Philips, M. L., Young, A. W., Scott, S. K., Calder, A. J., Andrew, C., Giampietro, V., et al. (1998). Neural responses to facial and vocal expressions of fear and disgust. Proceedings of the Royal Sociaty of London: Biological Sciences, 265, 1809–1817. Raimbult, M., & Dubois, D. (2005). Urban soundscapes: Experiences and knowledge. Cities, 22, 339–350. Sabatinelli, D., Bradley, M. M., Fitzsimmons, J. R., & Lang, P. J. (2005). Parellel amygdala and infereotemporal activation reflect emotional intensity and fear relevance. NeuroImage, 24, 1265–1270. Sander, K., & Scheich, H. (2001). Auditory perception of laughing and crying activates human amygdala regardless of attentional state. Cognitive Brain Research, 12, 181–198. Schafer, R. M. (1977). The tuning of the world. New York: Knopf. Singer, T., Seymour, B., O’Doherty, J., Kaube, H., Dolan, R. J., & Frith, C. D. (2004). Empathy for pain involves the affective but not sensory components of pain. Science, 20, 1157–1162. Skinner, C. J., & Grimwood, C. J. (2001). The UK noise climate 1990– 2001: Population exposure and attitudes to environmental noise. Applied Acoustics, 66, 231–243. Tabert, M. H., Borod, J. C., Tang, C. Y., Lange, G., Wei, T. C., Johnson, R., et al. (2001). Differential amygdala activation during emotional decision and recognition tasks using unpleasant words; an fMRI study. Neuropsychologica, 39, 556–573. Vastfjall, D., Kleiner, M., & Garling, T. (2003). Core affective reactions and preference for combinations of interior aircraft sound and vibration. International Journal of Aviation Psychology, 13, 33–47. Yang, W., & Kang, J. (2005). Soundscape and sound preferences in urban squares: A case study in Sheffield. Journal of Urban Design, 10, 61–80. Zald, D. H. (2003). The human amygdala and the emotional evaluation of sensory stimuli. Brain Research Reviews, 41, 88–123. (Received September 9, 2009; Accepted January 29, 2010)

268

A. Irwin et al.

APPENDIX 1 The semantic scales used for rating pleasantness and vibrancy (arousal) of each soundscape stimulus. Pleasantness SOUND 1: …………………………………………………………………………………... Unpleasant, unhappy,

Pleasant, happy,

unsatisfied

satisfied 1

2

3

4

5

6

7

8

9

Vibrancy SOUND 1: ……………………………………………………………………………………... Gloomy, bored, dreary,

Fun, excited, thrilled,

dull, lifeless, tired, artificial

interested, energetic, real

1

2

3

4

5

6

7

8

9

APPENDIX 2 Repeated measures group analysis examining the effect of pleasantness on brain activity for soundscapes rated as ‘‘high vibrancy’’ only. Statistical outputs are reported for the peak voxel within each area of interest, separately in the right and left

hemispheres. The three-dimensional coordinates are reported in standardized brain space with a voxel measuring 2 ! 2 ! 2 mm in volume.

Table A1

Region of interest Inferior colliculus Medial geniculate body Auditory cortex Anterior insula Posterior insula Amygdala

No. of voxels n.s. n.s. n.s. n.s. n.s. n.s.

Left hemisphere

Right hemisphere

Coordinates

Coordinates

x

y n.s. n.s. n.s. n.s. n.s. n.s.

z

Z

p

No. of voxels

n.s. n.s. n.s. n.s. n.s. n.s.

n.s. n.s. n.s. n.s. n.s. n.s.

n.s. n.s. n.s. n.s. 31 22

x

32 20

y n.s. n.s. n.s. n.s. " 18 "8

z

2 " 14

Z

p

n.s. n.s. n.s. n.s. 3.20 2.77

n.s. n.s. n.s. n.s. 0.028 0.052


The emotional startle effect is disrupted by a concurrent working memory task

ROSEMARY KING and ALEXANDRE SCHAEFER Institute of Psychological Sciences, University of Leeds, Leeds, United Kingdom

Abstract Working memory (WM) processes are often thought to play an important role in the cognitive regulation of negative emotions. However, little is known about how they influence emotional processing. We report two experiments that tested whether a concurrent working memory task could modulate the emotional startle eyeblink effect, a well-known index of emotional processing. In both experiments, emotionally negative and neutral pictures were viewed in two conditions: a ‘‘cognitive load’’ (CL) condition, in which participants had to actively maintain information in working memory (WM) while viewing the pictures, and a control ‘‘no load’’ (NL) condition. Picture-viewing instructions were identical across CL and NL. In both experiments, results showed a significant reduction of the emotional modulation of the startle eyeblink reflex in the CL condition compared to the NL condition. These findings suggest that a concurrent WM task disrupts emotional processing even when participants are directing visual focus on emotionally relevant information. Descriptors: Startle, Emotion, Working memory, Attention, Cognitive control

Although prior research has convincingly demonstrated that emotional processing can be altered by cognitive tasks, most previous studies have used attentional paradigms or complex regulation tasks (e.g., ‘‘reappraisal’’ instructions) to investigate the cognitive modulation of emotional processing. Therefore, the role played by other top-down cognitive processes such as working memory (WM), although often cited as an important component of reappraisal processes (Ochsner & Gross, 2005; Schaefer et al., 2003), remains unclear. In particular, available evidence regarding the role of WM in emotional processing is somewhat contradictory. On the one hand, evidence points towards an inhibitory effect of WM processes on emotions. For instance, several tasks thought to involve WM-related processes are often associated with a decrease in the intensity of negative emotions (Philippot et al., 2006, 2003; Schaefer et al., 2003), and WM-related brain areas such as the dorsolateral prefrontal cortex (DLPFC) are often involved in the cognitive down-regulation of negative emotions (Drevets & Raichle, 1998; Johnstone, van Reekum, Urry, Kalin, & Davidson, 2007; Ochsner & Gross, 2005; Schaefer et al., 2003). On the other hand, there is also evidence that WM processes and emotional systems can have links of facilitation rather than inhibition (Banich, Mackiewicz, Depue, Whitmer, Miller, & Heller, 2009; Gray, 2001; Gray & Braver, 2002; Schaefer, Braver, Reynolds, Burgess, Yarkoni, & Gray, 2006; Schaefer & Gray, 2007). The present manuscript reports two studies aiming to contribute towards understanding the role played by WM-related processes in emotional processing. Both studies examined whether WM resources were necessary for the aversive modulation of the startle reflex, a classical psychophysiological index of emotional processing. Beyond the role of WM in emotions, most previous studies approaching the question of top-down influences on emotional

Substantial evidence indicates that emotional responses can be modulated by higher-level cognition, including conscious volitional processes (Jackson, Malmstadt, Larson, & Davidson, 2000). The regulation of negative emotions is a domain where this phenomenon was demonstrated in a robust way. For instance, emotional responses can be intentionally down-regulated by cognitive reappraisal (i.e., changing the interpretation of an emotional situation in order to neutralize its emotional impact) (Gross, 1998; Jackson et al., 2000; Ochsner & Gross, 2005), by distraction (Erk, Abler, & Walter, 2006; Van Dillen & Koole, 2007) and by specification, i.e., a detailed analysis of specific elements of an emotional situation (Philippot, Baeyens, & Douilliez, 2006; Philippot, Schaefer, & Herbette, 2003; Schaefer, Collette, Philippot, Vanderlinden, Laureys, et al., 2003). These phenomena have been shown both at behavioral and neural levels (Gross & Levenson, 1997; Jackson et al., 2000; Ochsner, Bunge, Gross, & Gabrieli, 2002). In addition, the top-down modulation of emotional processing has also been investigated by behavioral and neuroimaging (fMRI and event-related potentials (ERP)) experiments showing that concurrent cognitive tasks could disrupt emotional processing. (Doallo, Holguin, & Cadaveira, 2006; Pessoa, McKenna, Gutierrez, & Ungerleider, 2002; Pessoa, Padmala, & Morland, 2005; Schupp, Stockburger, Bublatzky, Junghofer, Weike, & Hamme, 2007).

This study was supported by the UK Economic and Social Research Council (ESRC), and by the University of Leeds. The authors thank Martin Conway for his comments on an earlier draft of this manuscript, and Denis McKeown for his help with auditory stimulation techniques. Address correspondence to: Alexandre Schaefer, Wolfson Research Institute, University of Durham, Queens Campus, Stockton-on-Tees, TS17 6BH, UK. E-mail: [email protected] 269

270 processing used concurrent tasks that implied differences in viewing instructions between conditions. For instance, many studies have used concurrent attentional tasks requiring participants to direct visual gaze away from the emotional features of the stimuli (e.g., asking participants to look at geometric shapes at the periphery of an emotional stimulus). This approach involves a potential confound between viewing conditions (e.g., looking at emotional information vs. looking at non-emotional information) and actual cognitive load (e.g., the amount of attentional resources available to process emotional information). This is a classic example of a confound between ‘‘viewing’’ and ‘‘attending,’’ which are processes that can be dissociated (Posner, Walker, Friedrich, & Rafal, 1984). In order to avoid this confound, the two studies reported in the present paper used a paradigm in which the effects of WM load on emotional picture processing were manipulated while constant picture viewing instructions were maintained across conditions. We used the emotional modulation of the startle eyeblink reflex to estimate emotional reactions in the present study. This paradigm is based on a rapid defensive startle reflex consisting of the contraction of the orbicularis oculi muscle following an acoustic probe (i.e., a sudden loud noise) which can be measured with electromyography (EMG) electrodes placed beneath the eyes. The amplitude of this reflex can be modulated by the affective context in which it takes place: many studies have shown that the reflex is enhanced when the acoustic probe occurs while a negative emotional picture is being processed, compared to a situation when a neutral or positive picture is being processed (Bradley, Cuthbert, & Lang, 1990; Lang, Bradley, & Cuthbert, 1990). The difference in startle reflex amplitude between a negative and a neutral context (referred hereafter as the ‘‘emotional startle effect’’) is now widely used as a reliable behavioral index of emotional processing (Bradley & Lang, 2000; Grillon & Baas, 2003). The emotional startle effect is stable and widely replicated across a large number of studies (for reviews, see Bradley et al., 1990; Bradley & Lang, 2000). Two experiments are reported in the present article. In both experiments, participants had to view emotionally negative and neutral realistic pictures selected from the International Affective Picture System (IAPS). The picture viewing task was combined with a simple working memory task in which items (words or faces) were displayed before and after each picture. In the ‘‘Cognitive load’’ condition (CL), participants were required to decide if an item matched the item presented just before a preceding emotional picture, so that participants had to retain an item in working memory while viewing each picture. In a ‘‘No load’’ condition (NL), participants had to perform a simple lexical decision task (for words) or a gender discrimination task (for faces). During the display of the pictures, a loud acoustic probe was delivered in order to obtain the startle eyeblink reflex, measurable by EMG. We hypothesized that, if the perception of negative pictures can be disrupted by a concurrent working memory task, then the enhancement of the startle reflex for negative pictures should be attenuated or absent in the CL condition compared to the NL condition.

EXPERIMENT 1 Methods Participants Twenty-two participants (18 females, mean age 5 20.59 years, SD 5 3.39), all undergraduates at the University of Leeds, took

R. King & A. Schaefer part in the study for course credits. One participant failed to complete the study, reducing the final number to twenty-one. Prior to selection, participants were screened with the Centre for Epidemiological Studies Depression Scale (Radloff, 1977) to exclude potentially vulnerable participants from participating (cut-off score 5 16). The study was approved by the local ethics committee, and all participants gave informed consent. Stimuli and Design One hundred and twenty-eight pictures were selected from the IAPS (Bradley, Cuthbert, & Lang, 1999) on the basis of standard normative ratings provided with the IAPS. In these ratings, scales ranging from 1 to 9 reflect valence (negative/positive) and arousal. We selected 64 negative pictures (mean valence 5 2.52, mean arousal 5 6.34) and 64 neutral pictures (mean valence 5 5.17, mean arousal 5 2.98). Negative pictures included depictions of mutilations, accidents, graphic medical conditions, aggression, and dangerous animals. Neutral pictures included depictions of household objects, landscapes, non-threatening animals, vehicles, plants, etc. Sixty-four (32 negative, 32 neutral) pictures were randomly assigned to each condition (i.e., CL and NL) making 64 trials per condition. Forty-two of these (21 negative, 21 neutral) were randomly assigned to probed trials (i.e., trials with a burst of white noise) and 22 (11 negative, 11 neutral) to non-probed trials. On probed trials, the startle reflex was triggered by a 50-ms 100 Db burst of white noise generated at intervals of 2.5, 3.5, and 4.5 s from picture onset. The inclusion of non-probed trials had the aim to attenuate expectation effects. The working memory task used black and white, oval shaped faces1 displaying a neutral facial expression with no extraneous distinguishing features (e.g., hairstyle, clothing, expressions, etc.). Two equal sets of faces were created in which gender, race (e.g., Caucasian, Asian), physical features (e.g., nose shape and size), and image tones (i.e., light exposure levels) were equally distributed across the two groups. Size was standardized via the use of an invisible oval picture frame (7.5 ! 5.5 cm). The assignment of each face set to the experimental conditions (CL vs. NL) was counterbalanced across participants. Procedure Participants sat in a comfortable chair in front of a 17-inch computer screen on which stimuli were displayed throughout the experiment. The task consisted of a continuous sequence of stimuli (faces and pictures) displayed on a screen. Specifically, for each trial, a 1-s fixation point (white screen with a black fixation cross) was followed by a neutral face stimulus displayed for 2000 ms. Next, a negative or neutral IAPS picture was displayed for 5 s. Finally, a 7-point rating scale (1 5 ‘I feel absolutely nothing at all,’ 7 5 ‘I feel extremely strong emotions’) was presented, and participants were instructed to rate their emotional feelings with a button press. Participants were instructed to perform a specific task (gender discrimination task or WM task) as soon as 1 The faces were carefully selected from several face databases: the ‘least verbalizable’ face set database (Sreenivasan & Jha, 2007), The Face and Gesture Recognition Network (FG–NET) Aging Database, Cyprus; The AT and T Laboratories, Cambridge; The Valid Database, University College, Dublin; The Yale Face Database, Yale University; The Essex Face Database, Essex University; The Informatics and Mathematical Modelling (IMM) Face Database, Technical University of Denmark; The University of Manchester Institute of Science and Technology (UMIST) Face Database; The Japanese Female Facial Expression Database (JAFFE); The AR Face Database, University of Barcelona.

Emotion and working memory the face stimulus appeared on the screen, and response times (RT) relative to this task performance were recorded. In the NL condition, participants were instructed to perform a gender discrimination task, (e.g., male/female), for each face using a keypress. In the CL condition, participants had to perform a simple working memory (WM) task: for each face stimulus, participants had to decide by a keypress if the current face matched the face presented before the preceding IAPS picture. Fifty percent of WM trials involved a face stimulus matching the face presented before the preceding IAPS picture. In both CL and NL conditions, participants were asked to respond as quickly as possible after the onset of the face stimulus. In addition, participants were instructed to watch attentively to each picture, and they were instructed not to close their eyes or look away from the picture. The experimenter sat next to the participant to ensure total compliance to the viewing instructions. In addition, participants’ faces were filmed to make sure they were focusing on the screen. For probed pictures, a 50-ms 100 dB burst of white noise (the acoustic probe) was presented binaurally through headphones at 2.5, 3.5, or 4.5 s after picture onset. No more than four pictures of the same valence were displayed consecutively. Both pictures and conditions were fully counterbalanced. When viewing the pictures, participants were instructed to look at the screen for the full 5 s of picture presentation and only register their response when the rating scale appeared on the screen. A practice session preceded both conditions to allow participants time to become familiar with instructions and time constraints. The order of experimental conditions was counterbalanced across participants. Physiological Data Collection and Reduction Electromyographic recordings were taken using a Biopac MP150 system (Goleta, CA). Raw EMG was recorded from the orbicularis oculi muscle beneath the left eye using 4-mm Ag/AgCl electrodes following guidelines proposed by Fridland and Cacioppo (1986) and Blumenthal, Cuthbert, Filion, Hackley, Lipp, and Van Boxtel (2005). The raw EMG signal was amplified using a Biopac EMG 100 C amplifier, filtered (low-pass filter: 500 Hz; high-pass filter: 28 Hz) and digitized using a sample rate of 1 KHz. The amplified signal was rectified and integrated at a time constant of 50 ms. Sampling started just before the experimental block began and continued to the end (continuous recording). Blink amplitude was computed as the peak EMG amplitude in a window between 30 and 150 ms following the onset of each acoustic probe minus the mean EMG value recorded during a 50ms window preceding the acoustic probe. Trials with an excessive baseline (43 standard deviations from the mean, within-subjects) or with low blink amplitude (o20% of the mean, withinsubjects) were rejected, leading to a total rejection of 12% of trials. Participant blink data was then standardized by converting all scores to z scores (calculated on a within-subject basis) in order to attenuate interindividual variance (Bush, Hess, & Wolford, 1993). Z scores were next converted to T scores [(Z score ! 10)150] in order to obtain unidirectional values2 (Bradley, Codispoti, & Lang, 2006). Results were analyzed with repeated-measures analyses of variance (ANOVA). Partial eta2 We also analyzed unstandardized data (raw blink amplitude data), and we obtained results that were similar to those obtained with T scores. Specifically, The Emotion ! Cognitive Load interaction was significant for both Study 1 and Study 2 (pso.05), revealing that the blink amplitude was significantly larger for negative than neutral pictures only in NL (pso.01) for both studies.

271 squares are presented to estimate effect sizes. Statistics were adjusted with the Greenhouse-Geisser correction where necessary. Results Self-Report Data Self-reported arousal ratings to negative and neutral pictures were in line with existing research. A repeated-measures ANOVA revealed a significant main effect of Emotion [F(1,20) 5 218.66 po.001, Z2 5 .91) reflecting significantly higher ratings for negative than neutral pictures [Negative: M 5 5.04, SE 5 0.144; Neutral: M 5 2.69, SE 5 0.122). No other significant difference was found. Accuracy and Reaction Times In both conditions, accuracy (% correct responses) [NL: M 5 90.02, SE 5 1.64, (negative), M 5 90.93, SE 5 1.18 (neutral). CL: M 5 82.54, SE 5 2.66 (negative), M 5 89.34, SE 5 1.64 (neutral)] and mean response times of correct responses (RT) [NL: M 5 944.8, SE 5 35.0, (negative), M 5 975.8, SE 5 29.6 (neutral). CL: M 5 1126.4, SE 5 33.9 (negative), M 5 1058.4, SE 5 33.3 (neutral)] were in line with previous WM research (Gray, Burgess, Schaefer, Yarkoni, Larsen, & Braver, 2005). A main effect of cognitive load was found for both accuracy and RT, [F(1,20) 5 6.84, 15.74; po.02, po.01; Z2 5 .26, Z2 5 .44] revealing that participants were overall slower and less accurate in CL than NL. A significant Emotion ! Load interaction was found on accuracy data [F(1,20) 5 4.74, po.05 Z2 5 .19] revealing that negative pictures were associated with a worse WM accuracy than neutral pictures (NL: Fo1; CL: F(1,20) 5 14.49, po.01, Z2 5 .42). An Emotion ! Load interaction was also observed for RTs [F(1,20) 5 17.40, po.05, Z2 5 .47] suggesting that the effect of emotion was larger in CL, although the difference was significant in both NL and CL conditions (NL: F(1,20) 5 4.42, po.05, Z2 5 .18; CL: F(1,20) 5 18.63, po.001, Z2 5 .48). These findings indicate that (1) the WM task (CL) was overall more difficult than the gender discrimination task; and (2) the WM task was more difficult following negative than neutral pictures, consistent with previous research (Kensinger & Corkin, 2003). Startle Amplitudes An Emotion ! Cognitive Load ! Probe time ! Task Order ANOVA revealed a significant main effect of Emotion [F(1,19) 5 4.94, po.05, Z2 5 .21] reflecting an overall larger blink amplitude during negative compared to neutral picture viewing, and a significant interaction between Emotion and Cognitive Load [F(1,19) 5 4.50, po.05, Z2 5 .19]. Consistent with our predictions, planned comparisons revealed that the effect of Emotion was significant only in the NL condition [F 5 8.2, po.01, Z2 5 .30] showing higher startle magnitudes to negative than neutral pictures. Emotion was not significant in the CL condition (Fo1), indicating that the emotional startle effect was cancelled during the CL condition (Means and SE’s are depicted in Figure 1). These findings were not significantly modulated by condition order or probe time (Fso1). EXPERIMENT 2 The main goal of Experiment 2 was to replicate the results of Experiment 1 in order to ascertain the reliability of the effect. In addition, we examined whether the effect would generalize to

272

R. King & A. Schaefer

Startle Amplitudes (T scores)

55 54

Negative

53

Neutral

52 51 50 49 48

NL

CL

Figure 1. Startle eyeblink responses to negative and neutral pictures in the NL and CL conditions for Experiment 1.

different (verbal) WM contents. Experiment 1 was then repeated with one specific difference: the concurrent tasks used written words and nonwords rather than faces. Specifically, in the CL condition, participants had to perform a WM task in which they had to decide if each word was the same as the last word seen in the sequence. In the NL condition, participants had to perform a simple lexical decision (word vs. nonword).

Methods Participants Fifty-one English-speaking participants, from the University of Leeds (24 female participants, mean age 24.98 years, SD 5.93) took part in the study as paid volunteers. All completed the same questionnaires as in Experiment 1. Participants gave informed consent, and the experiment was approved by the local ethics committee. Materials and Design The stimuli and design were almost the same as for Experiment 1 except that words and non-words replaced faces as working memory items. One hundred and twenty 5-letter words were selected for the lexical tasks from the Medical Research Council’s Psycholinguistic Database (Wilson, 1988) on the basis of similar word length, concreteness, imageability, and familiarity ratings. Next, a pilot study was run to select emotionally neutral words from this initial group of words. Twenty participants were asked to categorize each word as positive, negative, or neutral using a categorical scale, and they also rated each word for arousal using an analog scale (min: 0, max: 100). Sixty-nine words were selected on the basis of these scores: Each selected word had an 85% agreement score for a ‘‘neutral’’ categorization and a score for arousal below 15 in the 100-point arousal scale. Examples of used words are: cover, paste, tower, glass, bench, slope, coach, etc.). The 69 neutral words were used to create the word sequences used for the concurrent task. In the WM task, words were repeated in order to create target trials (trials in which the current word matches the lastseen word) and lure trials (trials in which the current word matches a word previously seen during the experiment but not the last-seen word). Lure trials were introduced to attenuate the potential utilization of strategies based on familiarity (Fales, Barch, Burgess, Schaefer, Mennin, et al., 2008). Forty-one percent of trials were targets, 59% were nontargets of which 45%

were lures (! 27% of the total number of trials). The same levels of repetition were kept for NL condition, in which 13 additional meaningless strings of letters (non-words) were used alongside normal words for a word/non-word decision task. Three different list orders were created for CL and NL conditions, and these list orders were counterbalanced across participants. IAPS pictures were the same as those used in Experiment 1, and the experiment comprised a total number of 128 trials. Procedure The procedure was similar to that described in Experiment 1, with two crucial differences: In the CL condition, a word WM task was used instead of a face WM task. In the NL condition, a lexical decision task was used instead of a face gender discrimination task. Physiological Data Collection and Reduction All physiological data collection and analysis were performed as described in Experiment 1.

Results Self-Report Data As in Experiment 1, self-reported arousal ratings showed a main effect of Emotion [F(1,50) 5 312.10, po.001, Z2 5 .86] in line with existing research. Significantly higher ratings were recorded for negative than neutral stimuli (Negative: M 5 4.31, SE 5 .12; Neutral: M 5 2.39, SE 5 .92). Accuracy and RT Both accuracy [NL: M 5 91.88, SE 5 1.71 (negative), M 5 90.85, SE 5 1.98 (neutral); CL: M 5 89.64, SE 5 1.58 (negative), M 5 89.54, SE 5 1.61 (neutral)] and RT data [NL: M 5 913.76, SE 5 29.88, (negative), M 5 920.91, SE 5 29.80 (neutral). CL: M 5 910.17, SE 5 32.28 (negative), M 5 906.38, SE 5 29.27 (neutral)] were in line with previous research using nback tasks (Gray et al., 2005). ANOVAs revealed no significant differences between conditions. Startle Amplitudes An Emotion " Cognitive Load " Probe time " Task Order ANOVA revealed a significant main effect of Emotion [F(1,49) 5 12.3, po.001, Z2 5 .20] and a significant Emotion" Cognitive Load interaction [F(1,49) 5 17.2, po.0001, Z2 5 .26]. Planned contrasts confirmed that significantly higher blink amplitudes were found for negative pictures than neutral pictures in the NL condition [F(1,49) 5 29.9, po.00001; Z2 5 .38] whereas Emotion was not significant in the CL condition (Fo1). As in Experiment 1, this finding shows that the emotional startle effect was cancelled in the CL condition, as depicted in Figure 2. These findings were not significantly modulated by task order (interactions with order were non-significant, ps4.10). Effects involving probe time revealed a significant Emotion " Probe interaction [F(2,98) 5 5.2, po.01, Z2 5 .09] indicating that the affective modulation of blink amplitude regardless of cognitive load was stronger in later latencies [4.5 s: F(1,49) 5 25.9, po.001, Z2 5 .34] compared to shorter latencies [2.5 s: F(1,49) 5 2.8, p 5 .10, Z2 5 .05, 3.5 s: Fo1], consistent with previous findings (Bradley et al., 2006; Vanman, Boehmelt, Dawson, & Schell, 1996). Descriptive statistics of analyses involving probe time data were included in the online supplementary material.

Emotion and working memory

273

54 Negative

Startle Amplitudes (T scores)

53

Neutral

52 51 50 49 48

NL

CL

Figure 2. Startle eyeblink responses to negative and neutral pictures in the NL and CL.

General Discussion Consistent with our predictions, we observed in both experiments a significant Emotion ! Cognitive Load interaction indicating a cancellation of the emotional startle effect in the CL condition. Specifically, we observed that the increase of the amplitude of the startle reflex during negative picture viewing did not occur when WM was loaded by a concurrent task, whereas it occurred as normal in the NL condition. We observed this pattern of results in two different experiments using concurrent WM tasks with stimuli from different modalities (verbal vs. nonverbal), suggesting that this effect is robust and generalizes to the population from which the samples of participants were drawn. These findings suggest that processes involved in WM performance need to be free or ‘‘switched off’’ during the perception of negative pictures for an optimal processing of emotional information. The precise identity of these processes remains unknown, but attention might be a strong candidate. It is possible that the concurrent WM task in the CL condition might have depleted attentional resources necessary to efficiently process the meaning of negative pictures. This is consistent with the notion that WM and attention are inherently linked (Awh, Vogel, & Oh, 2006; De Fokert, Rees, Frith, & Lavie, 2001). It is consistent with evidence that ERP correlates of emotional processing can be modulated by attention (e.g., Schupp et al., 2007). It is also consistent with fMRI evidence that the amygdala needs some degree of attentional resources in order to process emotional stimuli (Pessoa et al., 2002, 2005). It is noteworthy that our findings were obtained even though viewing instructions were identical between CL and NL. Therefore, our findings also suggest that the effects of a concurrent attentional task on emotional processing (such as the effects shown by Pessoa et al., 2002; Doallo et al., 2006; and Schupp et al., 2007) cannot be easily accounted for by an alternative explanation, suggesting that such effects are observed because concurrent cognitive tasks usually require ‘‘looking away’’ from emotional features of stimuli. Instead, they suggest that cognitive load per se, rather than differences in viewing instructions, is the cause of the cognitive modulation of emotional processing in these tasks. An explanation of our findings based on attention might also have implications for the understanding of the phenomenon of the emotional startle effect. The allocation of attention to foreground stimuli is often thought to inhibit the startle reflex, whereas negative affect is known to facilitate it (Bradley et al., 2006). The current findings suggest instead that attention and

negative affect can cooperate towards the facilitation of the startle reflex. The key to this apparent contradiction might reside in the distinction between early and late attention to emotional contents (Filion, Dawson, & Schell, 1993; Schupp, Junghofer, Weike, & Hamm, 2003; Schupp, Stockburger, Codispoti, Junghofer, Weike, & Hamm, 2007). It might be possible that an inhibition of the startle reflex is caused by early attentional processes, whereas late attentional processes would cooperate with emotional processing and contribute to the facilitation of the startle reflex. The current studies used auditory probes delivered at latencies compatible with late attentional processes, thus our findings could be the result of a depletion of late attentional processes needed to facilitate emotional processing. Consistent with this view, evidence suggests that the inhibition of the startle reflex by attentional processes occurs mainly at very early latencies (Bradley et al., 2006; Bradley, Cuthbert, & Lang, 1993; Filion et al., 1993). Further, there is also evidence that allocation of attentional resources at later stages can actually facilitate the startle reflex (Filion et al., 1993). Therefore, further research will be needed to explore this question. In particular, future studies on the cognitive modulation of the emotional startle reflex should try to use a wider array of probe times including probe times occurring at very early latencies (o300 ms) similar to procedures used in a number of previous studies (Bradley et al., 1993; Globisch, Hamm, Esteves, & Ohman, 1999). Beyond attention, it is possible that the current findings were also driven by processes specific to WM function. In particular, it could be that processes involved in actively maintaining and manipulating information in WM could have inhibited emotional processing. The existence of links of inhibition between emotional systems and WM-related systems has been previously suggested (Drevets & Raichle, 1998; Johnstone et al., 2007; Philippot et al., 2003). Evidence for this idea is mainly provided by functional neuroimaging studies showing an inverse relationship between brain regions associated with emotional processing (e.g., the amygdala) and brain regions usually associated with WM processes (Johnstone et al., 2007; Schaefer, Jackson, Davidson, Aguirre, Kimberg, & Thompson, 2002; Simpson, Snyder, Gusnard, & Raichle, 2001). However, it is unclear whether the observed inhibition of emotional systems is caused by inhibitory projections from WM systems, or because the tasks used in these experiments caused a division of resources that reduced attentional resources necessary for processing emotional stimuli. In addition, it has to be noted that relationships between emotional and WM systems are not always inhibitory. For instance, WM tasks can be associated with an increase in phasic activity in the amygdala (Schaefer et al., 2006). Further research will be needed to investigate more precisely the relationship between WM processes, attention, and emotional generation. In particular, future research should try to break down WM in different subcomponents (Baddeley, 1986) and test the unique effects of different WM subcomponents on emotional processing. These findings also have potential implications for understanding emotion regulation processes. Specifically, it may be the case that a simple cognitive process, division of cognitive resources, can account for at least part of the effectiveness of a wide range of emotion regulation strategies. For example, it is possible that the effect of reappraisal on emotions (Gross & Levenson, 1997; Jackson et al., 2000) might be in part accounted for by a simple division of WM resources. Reappraising emotional contents is thought to recruit several cognitive processes, including

274

R. King & A. Schaefer

WM (Ochsner et al., 2002; Ochsner & Gross, 2005; Schaefer et al., 2002), necessary to actively change the meaning of the emotional contents and to achieve a detachment from the original emotional meaning. Reappraisal might therefore act as a concurrent task taking away cognitive resources that are no longer available for emotional processing, therefore contributing to the resulting attenuation of emotional responses. Similarly, to remember and describe specific details of past emotional events (Philippot et al., 2003) or to analyze the specific details of an emotional mental image (Schaefer et al., 2003) again may reduce emotional responses, at least in part, by reducing cognitive resources available for generating emotion. In the same vein, Holmes (2004) has shown that intrusive memories of an emotional film were attenuated if participants had to perform a concurrent attentional task during film viewing. In summary, our findings suggest the possibility that a simple mechanismFdivision of cognitive resourcesFmight at least partially account for the effects of several emotion regulation strategies. A number of limitations of the current paper have to be noted. First, using faces in the concurrent task in Study 1 could have triggered an affective response due to the inherently social meaning of faces, which might have had an impact on subsequent emotional responses to the IAPS pictures. However, the fact that the effects observed in Study 1 were replicated with neutral words in Study 2 indicates that any potential effect of using faces in Study 1 did not change the observed pattern of cognitive modulation of the emotional startle effect. Second, it could be said that the gender imbalance of Study 1 decreased the generalizability of our results. However, it has to be noted that Study 2 replicated the results of Study 1 using a larger gender-balanced sample. Third, no other psychophysiological measures were used (e.g., heart rate, skin conductance), which could suggest that our results are specific to the emotional startle paradigm. However, our results converge with findings showing that concurrent cognitive tasks do modify ERPs to emotional pictures (Doallo, Cadaveira, & Rodriguez Holguin, 2007; Doallo et al., 2006; Schupp, Stockburger, Bublatzky, et al., 2007) and BOLD activity related to emotional stimuli (Pessoa et al., 2002, 2005). Moreover, results obtained with the emotional startle effect often converge with other psychophysiological measures of emotion (Bradley & Lang, 2000). Fourth, there is an apparent inconsistency between Studies 1 and 2 in that the NL–CL comparison is significant for RTand accuracy in Study 1 but not for Study 2. A possible explanation is that this inconsistency is linked to the fact that the NL condition of Study 2 used a word–non-word decision, which is known to be more difficult than decisions based only on words (as in the WM task used in the CL condition) (Bentin & Moscovitch, 1988; Weekes, 1997). Therefore, it might be possible that this difference in decision difficulty counteracted any potential effect of load on RT and accuracy in Study 2. However, it has to be noted that, despite the absence of a NL–CL difference on RTand accuracy in Study 2, these 2 conditions still differed in the feature which is central to both studies, that is, CL required the maintenance of an item in WM during blink stim-

ulation whereas NL did not. Finally, our self-report measure was a broad measure of emotional intensity, as used in previous research (Schaefer, Nils, Sanchez, & Philippot, in press). Future research should try to use measures that can disentangle arousal and valence. Finally, the findings reported in the present paper can provide additional suggestions of future research directions. First, the main focus of the current paper was on the influence of cognitive processing on the perception of aversive stimuli as a means to model processes of negative emotion regulation. However, future research should investigate whether the effects reported in this paper extend to appetitive, positively valenced stimuli. When positive foreground pictures are used, the startle reflex is typically inhibited compared to neutral pictures (Vrana, Spence, & Lang, 1988). This phenomenon is thought to be caused by an inhibition of defence mechanisms triggered by the perception of appetitive stimuli and/or by a sustained allocation of attentional resources (Bradley et al., 2006). Our findings suggest that allocating cognitive resources on emotional information facilitates the affective modulation of the startle reflex by negative pictures. If this process is valence-independent, we would then predict that a concurrent cognitive task should attenuate or cancel the inhibition of the startle reflex by positive pictures. Second, it would be interesting to investigate whether the current pattern of findings would hold in clinical populations with mood disorders. Our prediction is that the cognitive modulation of the emotional startle effect might be less efficient in such clinical populations because of a potential overreaction to negative stimuli (Sheline, Barch, Ollinger, & Mintun, 2001) compounded by usual deficits in WM performance often observed in patients with mood disorders (Hammar & Ardal, 2009). Third, it could be questioned why no significant effects of cognitive load were found on neutral stimuli, which could suggest that depletion of attentional resources in itself does not affect the startle reflex. However, neutral stimuli tend to mobilize less attentional resources than emotional stimuli (Bradley et al., 2006; Kensinger & Corkin, 2003). Therefore, a mild attentional depletion such as the one caused by our WM task might not have a significant impact on startle reflexes during neutral picture viewing because only very limited attentional resources are needed to process them. To further examine this question, future research should use a methodology in which the strength of cognitive resource depletion is parametrically manipulated (e.g., using and ‘‘N-back’’ task, comparing 1-back vs. 2-back vs. 3-back conditions). In summary, we reported two studies showing that the emotional startle effect can be disrupted by a concurrent WM task. These findings suggest that WM resources need to be free for optimal emotional processing during negative picture viewing. Moreover, these results argue for a deeper understanding of the role of WM and attention in the generation of emotional responses. Further, these results also suggest that a simple mechanismFdivision of cognitive resourcesFmight account for a significant part of the effectiveness of a wide range of emotion regulation strategies.

REFERENCES Awh, E., Vogel, E., & Oh, S.–H (2006). Interactions between attention and working memory. Neuroscience, 139, 201–208. Baddeley, A. D. (1986). Working memory. New York: Oxford University Press.

Banich, M.T, Mackiewicz, K.L, Depue, B.E, Whitmer, A.J, Miller, G.A, & Heller, W. (2009). Cognitive control mechanisms, emotion and memory: A neural perspective with implications for psychopathology. Neuroscience & Biobehavioral Reviews, 33, 613–630.

Emotion and working memory Bentin, S., & Moscovitch, M. (1988). The time course of repetition effects for words and unfamiliar faces. Journal of Experimental Psychology: General, 117, 148–160. Blumenthal, T., Cuthbert, B., Filion, D., Hackley, S., Lipp, O., & Van Boxtel, A. (2005). Committee report: Guidelines for human startle eyeblink electromyographic studies. Psychophysiology, 42, 1–15. Bradley, M., Cuthbert, B., & Lang, P. (1990). Startle reflex modification: Emotion or attention? Psychophysiology, 27, 513–522. Bradley, M., Cuthbert, B., & Lang, P. (1999). Startle modification: Implications for neuroscience, cognitive science, and clinical science. In R. Lane & L. Nadel (Eds.), Cognitive neuroscience of emotion (pp. 296–327). New York: Oxford University Press. Bradley, M. M., Codispoti, M., & Lang, P. J. (2006). A multi-process account of startle modulation during affective perception. Psychophysiology, 43, 486–497. Bradley, M. M., Cuthbert, B. N., & Lang, P. J. (1993). Pictures as prepulse: Attention and emotion in startle modification. Psychophysiology, 30, 541–545. Bradley, M. M., & Lang, P. J. (2000). Measuring emotion: Behavior, feeling and physiology. In R. D. Lane & L. Nadel (Eds.), Cognitive neuroscience of emotion (pp. 242–276). New York: Oxford University Press. Bush, L., Hess, U., & Wolford, G. (1993). Transformations for within– subject designs: A Monte Carlo investigation. Psychological Bulletin, 113, 566–579. De Fokert, J., Rees, G., Frith, C., & Lavie, N. (2001). The role of working memory in visual selective attention. Science, 291, 1803–1806. Doallo, S., Cadaveira, F., & Rodriguez Holguin, S. (2007). Time course of attentional modulations on automatic emotional processing. Neuroscience Letters, 418, 111–116. Doallo, S., Holguin, S. R., & Cadaveira, F. (2006). Attentional load affects automatic emotional processing: Evidence from event-related potentials. NeuroReport, 17, 1797–1801. Drevets, W. C., & Raichle, M. E. (1998). Reciprocal suppression of regional cerebral blood flow during emotional versus higher cognitive processes: Implications for interactions between emotion and cognition. Cognition and Emotion, 12, 353–385. Erk, S., Abler, B., & Walter, H. (2006). Cognitive modulation of emotion anticipation. European Journal of Neuroscience, 24, 1227–1236. Fales, C. L., Barch, D. M., Burgess, G. C., Schaefer, A., Mennin, D. S., Gray, J.R, & Braver, T. S. (2008). Anxiety and cognitive efficiency: Differential modulation of transient and sustained neural activity during a working memory task. Cognitive, Affective, and Behavioral Neuroscience, 8, 239–253. Filion, D. L., Dawson, M. E., & Schell, A. M. (1993). Modification of the acoustic startle–reflex eyeblink: A tool for investigating early and late attentional processes. Biological Psychology, 35, 185–200. Fridland, A., & Cacioppo, J. (1986). Guidelines for human electromyographic research. Psychophysiology, 23, 567–589. Globisch, J., Hamm, A. O., Esteves, F., & Ohman, A. (1999). Fear appears fast: Temporal course of startle reflex potentiation in animal fearful subjects. Psychophysiology, 36, 66–75. Gray, J. R. (2001). Emotional modulation of cognitive control: Approach–withdrawal states double-dissociate spatial from verbal twoback task performance. Journal of Experimental Psychology, General, 130, 436–452. Gray, J. R., & Braver, T. S. (2002). Integration of emotion and cognitive control: A neurocomputational hypothesis of dynamic goal regulation. In S. C. Moore & M. R. Oaksford (Eds.), Emotional cognition (pp. 289–316). Amsterdam: John Benjamins. Gray, J. R., Burgess, G. C., Schaefer, A., Yarkoni, T., Larsen, R. J., & Braver, T. S. (2005). Personality differences in neural processing efficiency revealed using fMRI. Cognitive, Affective, & Behavioral Neuroscience, 5, 182–190. Grillon, C., & Baas, J. (2003). A review of the modulation of the startle reflex by affective states and its application in psychiatry. Clinical Neurophysiology, 114, 1557–1579. Gross, J. J. (1998). Antecedent- and response-focused emotion regulation: Divergent consequences for experience, expression, and physiology. Journal of Personality and Social Psychology, 74, 224–237. Gross, J., & Levenson, R. (1997). Hiding feelings: The acute effects of inhibiting negative and positive emotion. Journal of Abnormal Psychology, 106, 95–103. Hammar, A., & Ardal, G. (2009). Cognitive functioning in major depression––a summary. Frontiers in Human Neuroscience, 3, 26.

275 Holmes, E. (2004). Trauma films, information processing, and intrusive memory development. Journal of Experimental Psychology, General, 133, 3–22. Jackson, D. C., Malmstadt, J. R., Larson, C. L., & Davidson, R. J. (2000). Supression and enhancement of emotional responses to unpleasant pictures. Psychophysiology, 37, 512–522. Johnstone, T., van Reekum, C. M., Urry, H. L., Kalin, N. H., & Davidson, R. J. (2007). Failure to regulate: Counterproductive recruitment of top-down prefrontal–subcortical circuitry in major depression. Journal of Neuroscience, 27, 8877–8884. Kensinger, E. A., & Corkin, S. (2003). Effect of negative emotional content on working memory and long-term memory. Emotion, 3, 378– 393. Lang, P., Bradley, M., & Cuthbert, B. (1990). Emotion, attention, and the startle reflex. Psychological Review, 97, 377–395. Ochsner, K., Bunge, S., Gross, J., & Gabrieli, J. (2002). Rethinking feelings: An fMRI study of the cognitive regulation of emotion. Journal of Cognitive Neuroscience, 14, 1215–1229. Ochsner, K. N., & Gross, J. J. (2005). The cognitive control of emotion. Trends in Cognitive Sciences, 9, 242–249. Pessoa, L., McKenna, M., Gutierrez, E., & Ungerleider, L. G. (2002). Neural processing of emotional faces requires attention. Proceedings of the National Academy of Sciences USA, 99, 11458–11463. Pessoa, L., Padmala, S., & Morland, T. (2005). Fate of unattended fearful faces in the amygdala is determined by both attentional resources and cognitive modulation. NeuroImage, 28, 249–255. Philippot, P., Baeyens, C., & Douilliez, C. (2006). Specifying emotional information: Regulation of emotional intensity via executive processes. Emotion, 6, 560–571. Philippot, P., Schaefer, A., & Herbette, G. (2003). Consequences of specific processing of emotional information: Impact of general versus specific autobiographical memory priming on emotion elicitation. Emotion, 3, 270–283. Posner, M. I., Walker, J. A., Friedrich, F. J., & Rafal, R. D. (1984). Effects of parietal lobe injury on covert orienting of visual attention. Journal of Neuroscience, 4, 1863–1874. Radloff, L. S. (1977). The CES-D Scale: A self report depression scale for research in the general population. Applied Psychological Measurement, 1, 385–401. Schaefer, A., Braver, T. S., Reynolds, J. R., Burgess, G. C., Yarkoni, T., & Gray, J. R. (2006). Individual differences in amygdala activity predict response speed during working memory. Journal of Neuroscience, 26, 10120–10128. Schaefer, A., Collette, F., Philippot, P., Vanderlinden, M., Laureys, S., Delfiore, G., et al. (2003). Neural correlates of ‘‘hot’’ and ‘‘cold’’ emotional processing: A multilevel approach to the functional anatomy of emotions. NeuroImage, 18, 938–949. Schaefer, A., & Gray, J. R. (2007). A role for the amygdala in higher cognition. Reviews in the Neurosciences, 18, 355–363. Schaefer, S. M., Jackson, D. C., Davidson, R. J., Aguirre, G. K., Kimberg, D. Y., & Thompson-Schill, S. L. (2002). Modulation of amygdalar activity by the conscious regulation of negative emotion. Journal of Cognitive Neuroscience, 14, 913–921. Schaefer, A., Nils, X., Sanchez, X., & Philippot, P. (in press). Assessing the effectiveness of a large database of emotion-eliciting films: A new tool for emotion researchers. Cognition and Emotion. Schupp, H. T., Junghofer, M., Weike, A. I., & Hamm, A. O. (2003). Emotional facilitation of sensory processing in the visual cortex. Psychological Science, 14, 7–13. Schupp, H. T., Stockburger, J., Bublatzky, F., Junghofer, M., Weike, A. I., & Hamm, A. O. (2007). Explicit attention interferes with selective emotion processing in human extrastriate cortex. BMC Neuroscience, 8, 16. Schupp, H. T., Stockburger, J., Codispoti, M., Junghofer, M., Weike, A. I., & Hamm, A. O. (2007). Selective visual attention to emotion. Journal of Neuroscience, 27, 1082–1089. Sheline, Y. I., Barch, D. M., Ollinger, J. M., & Mintun, M. A. (2001). Increased amygdala response to masked emotional faces in depressed subjects resolves with antidepressant treatment: An fMRI study. Biological Psychiatry, 50, 651–658. Simpson, J. R., Snyder, A. Z., Gusnard, D. A., & Raichle, M. E. (2001). Emotion-induced changes in human medial prefrontal cortex: I. During cognitive task performance. Proceedings of the National Academy of Sciences USA, 98, 683–687.

276 Sreenivasan, K., & Jha, A. (2007). Selective attention supports working memory maintenance by modulating perceptual processing of distractors. Journal of Cognitive Neuroscience, 19, 32–41. Van Dillen, L., & Koole, S. (2007). Clearing the mind: A working memory model of distraction from negative mood. Emotion, 7, 715–723. Vanman, E. J., Boehmelt, A. H., Dawson, M. E., & Schell, A. M. (1996). The varying time courses of attentional and affective modulation of the startle eyeblink reflex. Psychophysiology, 33, 691–697. Vrana, S. R., Spence, E. L., & Lang, P. J. (1988). The startle probe response: A new measure of emotion? Journal of Abnormal Psychology, 97, 487–491.

R. King & A. Schaefer Weekes, B. (1997). Differential effects of number of letters on word and nonword naming latency. Quarterly Journal of Experimental Psychology, 50, 439–456. Wilson, M. D. (1988). The MRC Psycholinguistic Database: Machine readable dictionary, Version 2. Behavioural Research Methods, Instruments and Computers, 20, 6–11.

(Received November 18, 2009; Accepted March 29, 2010)


Low carbohydrate diet affects the oxygen uptake on-kinetics and rating of perceived exertion in high intensity exercise

ADRIANO E. LIMA-SILVA,a,b FLA´VIO O. PIRES,a ROˆMULO C. M. BERTUZZI,a FA´BIO S. LIRA,c DULCE CASARINI,d and MARIA AUGUSTA P. D. M. KISSa a

School of Physical Education and Sport, University of Saõ Paulo, Saõ Paulo, Brazil Sports Science Research Group, Federal University of Alagoas, Maceio´, Brazil c Department of Physiology, Division of Nutrition Physiology, Federal University of Saõ Paulo, Saõ Paulo, Brazil d Nephrology Division, Hospital of the Kidney and Hypertension, Federal University of Saõ Paulo, Saõ Paulo, Brazil b

Abstract The aim of this study was to determine if the carbohydrate (CHO) availability alters the rate of increase in the rating of perceived exertion (RPE) during high intensity exercise and whether this would be associated with physiological changes. Six males performed high intensity exercise after 48 h of controlled, high CHO (80%) and low CHO (10%) diets. Time to exhaustion was lower in the low compared to high CHO diet. The rate of increase in RPE was greater and the VO2 slow component was lower in the low CHO diet than in the control. There was no significant condition effect for cortisol, insulin, pH, plasma glucose, potassium, or lactate concentrations. Multiple linear regression indicated that the total amplitude of VO2 and perceived muscle strain accounted for the greatest variance in the rate of increase in RPE. These results suggest that cardiorespiratory variables and muscle strain are important afferent signals from the periphery for the RPE calculations. Descriptors: Perceived exertion, Fatigue, Afferent signals, Cardiorespiratory and metabolic systems

2004). Specifically, it has been suggested that a centrally localized governor controls the muscle recruitment pattern in order to prevent any potential metabolic disturbance (Lambert, St. Clair Gibson, & Noakes, 2005; Noakes et al., 2004; Tucker, 2009). This model holds that fatigue is not a simple physical event that must occur whenever a critical metabolic limit is overreached. Rather, fatigue could be considered as a conscious sensation of effort that results from interpretations of multiple physiological and psychological signals integrated in the CNS. As a consequence of this integrated mechanism, the rating of perceived exertion (RPE) may represent the conscious/verbal manifestation when these multiple afferent signals are integrated (Noakes, 2004, 2008; Tucker, 2009). Additionally, the RPE seems to be set at the beginning of the exercise bout as part of a feedforward/feedback mechanism (Crewe, Tucker, & Noakes, 2008). In fact, the scalar behavior of RPE as exercise progresses supports the idea that it is part of a feedforward/feedback mechanism, suggesting that RPE is set as a function of remaining exercise time. Using Baldwin’s data (Baldwin, Snow, Gibala, Garnham, Howarth, & Febbraio, 2003), Noakes (2004) showed that the rate of increase in RPE was higher during exercise with initial low muscle glycogen content than with high muscle glycogen content. However, when plotted against the percentage of the time to exhaustion, the RPE increased at the same rate in both conditions. Similar results have been obtained when comparing fatiguing to non-fatiguing conditions (Eston et al., 2007) and hot to cool environments (Crewe et al., 2008).

In performing a high intensity exercise at a fixed power output, fatigue can be operationally defined as the inability to maintain a pre-determined pedal cadence (Eston, Faulkner, St. Clair Gibson, Noakes, & Parfitt, 2007; Pitsiladis & Maughan, 1999). The traditional theory used to explain the mechanisms involved in fatigue development during high exercise intensity holds that the exercise termination is due to impaired muscle contractile function caused by a failure in homeostasis (Hill & Lupton, 1923; Noakes & St. Clair Gibson, 2004). However, evidence that the homeostasis failure is the specific cause of exercise termination in high exercise intensity still needs to be provided (Kayser, 2003; Noakes & St. Clair Gibson, 2004; Noakes, St. Clair Gibson, & Lambert, 2004). On the other hand, a centrally regulated system model has been recently proposed as an alternative model of fatigue. In this model, the work and metabolic rate during exercise are regulated by the central nervous system (CNS) in a non-linear manner, which prevents the homeostasis failure in bodily systems (Noakes et al., Fla´vio Pires is grateful to Coordenaçaõ de Aperfeiçoamento de Pessoal de Nı´ vel Superior (CAPES) for his PhD scholarship. This study had financial support provided by the Fundaçaõ de Amparo a Pesquisa do Estado de Saõ Paulo (FAPESP) (2006-60641-6). Address correspondence to: Adriano E. Lima-Silva, Sports Science Research Group, Faculty of Nutrition, Federal University of Alagoas, Lorival Melo Mota S/N Avenue, Campus A. C. Simo˜es, Tabuleiro do Martins, Maceio´, Alagoas, Brazil, 7072970. E-mail: [email protected] 277

278 These results reinforce the initial supposition in which the RPE calculation is based on multiple modes of physiological feedback, since the RPE increases at a higher rate when unfavorable metabolic conditions are imposed by experimental manipulation such as depleted muscle glycogen stores, fatigued condition or hot environment (Eston et al., 2007; Crewe et al., 2008; Noakes, 2004). Nevertheless, an alternative model has suggested that RPE is an exclusive result from efferent rather than afferent sensory inputs, so that the increase in RPE over time of exercise could be explained by increases in the central motor input to the locomotor and respiratory muscles (Marcora, 2008). Even with this contestation, studies have demonstrated that manipulations of peripheral muscle conditions can potentially affect RPE (Noakes, 2004; Eston et al., 2007; Crewe et al., 2008). Hampson, St. Clair Gibson, Lambert, and Noakes (2001) have listed several physiological variables that may act as afferent signals during exercise such as heart rate (HR), oxygen uptake (VO2), respiratory rate (RR), ventilatory volume (VE), blood lactate concentrations, pH, mechanical strain, and core temperature. These physiological variables would be all integrated by the brain and used to generate the RPE. Interestingly, the reduction in carbohydrate (CHO) storage can influence some of these physiological afferent signals. The prior reduction in CHO storage changes the ventilatory and heart rate responses to exercise (Heigenhauser, Sutton, & Jones, 1983), alters the VO2 kinetics parameters mainly in high exercise intensity (Bouckaert, Jones, & Koppo, 2004; Carter, Pringle, Boobis, Jones, & Doust, 2004; Krustrup, So¨derlund, Mohr, & Bangsbo, 2004; Osborne & Schneider, 2006), reduces the blood lactate concentrations (Arkinstall, Bruce, Clark, Rickards, Burke, & Hawley, 2004), and increases the plasma potassium concentrations (Busse, Maassen, & Konrad, 1991) and the perceived muscle strain (Johnson, Stannard, Chapman, & Thompson, 2006). Therefore, it can be hypothesized that the reduction in CHO availability can modify most of the afferent signals that could be associated with a higher rate of increase in RPE during high intensity exercise. Therefore, the present study was designed to determine if the rate of increase in RPE during high exercise intensity is altered by CHO manipulation, and whether these alterations are associated with metabolic, hormonal, and cardiorespiratory changes to exercise. Assuming that the CNS integrates different afferent signals for the RPE calculations, we hypothesized that, by altering the CHO availability, we would be able to identify variables that are associated with alteration in the rate of increase in RPE and which may act as afferent signals to promote early fatigue during exercise with lowered CHO availability. Methods Subjects Six healthy men, physically active and accustomed to high intensity exercise (age 25.8 ! 5.8 years, height 176.5 ! 5.2 cm, weight 70.9 ! 6.3 kg, body fat 11.7 ! 4.7% and VO2peak 47.8 ! 7.0 ml " kg # 1 " min # 1), participated in this study. The protocol, benefits, and risks were explained and a written consent was obtained. The study procedures were previously approved by the Ethics Committee of the School of Physical Education and Sport of the University of Saõ Paulo. Experimental Design Subjects reported to the laboratory on six different occasions. During the first visit, subjects underwent anthropometric mea-

A. E. Lima-Silva et al. sures and an incremental test for VO2peak; the first (LT1) and second (LT2) lactate thresholds were also established. In the second visit, subjects performed high intensity exercise until exhaustion as a control test. In the third and fifth visit, the subjects performed an exercise protocol in order to reduce endogenous CHO stores following 48 h of low or high CHO diets. In the fourth and sixth visit, the same experimental protocol used in the control test session was performed. A 1-week interval for washing out any residual effect of fatigue was followed between the fourth and sixth visits. The experimental conditions (low or high CHO diet) were applied in a counterbalanced order. Subjects were asked to refrain from food (8–10 h of overnight fast), exercise, alcohol, tobacco, and caffeine for 48 h before each experimental session. Subjects were blinded to the objectives of this study, manipulated CHO content, and time to exhaustion of second, fourth and sixth experimental trials, until the end of study. Researchers were also blinded to the manipulated CHO content. All experimental trials were performed on an electrical magnetic braked cycle ergometer (Ergo Fit 167, Ergo-Fit GmbH & Co., Pirmasens, Germany), and exhaustion was assumed when the subjects could not sustain a pedal frequency greater than 60 rpm. Preliminary Test The incremental test started at an initial work rate of 50W with 20W increases every 3 min until exhaustion. VE, VO2, and carbon dioxide production (VCO2) were continuously sampled and averaged over 30-s intervals using an on-line breath-by-breath gas analyzer (Quark b2, COSMED, Rome, Italy). The gas analyzer was calibrated according to the manufacturer’s specifications before each test (Quark b2 instruction manual). At the end of each stage, 25 ml of blood were drawn from the ear lobe and immediately analyzed in order to determine the blood lactate concentration (YSI 1500 Sport, Yellow Springs Instruments, Yellow Springs, OH). VO2peak was defined as the highest oxygen uptake obtained during the last 30-s interval during the incremental test. The maximal workload was determined as the highest workload reached with a pedal frequency between 60–70 rpm (Wmax). The LT1 and LT2 lactate breakpoints were determined by linear regression analysis (Ribeiro, Yang, Adams, Kuca, & Knutten, 1986). Control Test Based on previous studies, we chose to determine the high intensity workload through lactate breakpoints rather than VO2peak since the former seem to be a more consistent metabolic strain marker (Lima-Silva, De-Oliveira, Nakamura, & Gevaerd, 2009; Weltman, Weltman, Rutt, Seip, Levine, et al., 1989). After a 5-min warm-up at 50W, the subjects performed a constant workload test until exhaustion at a power output required for 75% (D75%) of the difference between LT2 and Wmax ($ 90% of VO2peak). The calculated D75% was 217.8 ! 51.5W (coefficient of variation [CV] 5 23.7%), corresponding to 94.3 ! 2.4% of VO2peak (CV 5 2.5%). The following variables were recorded: RPE, VE, VO2, VCO2, HR, pH, and blood lactate, plasma glucose, potassium (K1), insulin, cortisol, and epinephrine and norepinephrine concentrations. Manipulation of Carbohydrate Availability For manipulation of CHO availability, subjects reported to the laboratory 48 h before the experimental trials and cycled for 90 min at a power output corresponding to 50% of the difference

Low carbohydrate affects VO2 and RPE response

279

between LT1 and LT2. This procedure was followed by six 1-min exercise bouts at 125% VO2peak interspersed with 1-min rest periods. This protocol was previously validated for reducing the muscle glycogen content in both type I and II fibers (Bergstrom, Hermansen, Hultman, & Saltin, 1967; Gollnick, Armstrong, Sembrowich, Shepherd, & Saltin, 1973; Gollnick, Piehl & Saltin, 1974; Heigenhauser et al., 1983). Subjects followed a low (10% carbohydrate, 35% lipids, and 55% protein) or high CHO (80% carbohydrate, 10% lipids, and 10% protein) diet, for the remaining 48 h. A dietician created all diets using food plans for each subject, taking body mass and food preferences into account. All subjects were given a list with food options in order to describe the allowed content for each food group and provide the recommended daily energy uptake. For each experimental condition, subjects recorded all food intake for 48 h after the exercise depletion protocol until the experimental exercise session. Diet records were subsequently analyzed for the caloric contribution of fat, protein, and carbohydrate. Energy and macronutrient intakes were examined by specific software (DietWin software, Porto Alegre, Brazil). Analysis of the recorded food intake confirmed that subjects conformed to the recommended diets. Experimental Test Forty-eight hours after the CHO availability manipulation, the subjects performed a constant workload test until exhaustion at D75% power output under the same conditions and procedures previously described for the control test. Rating of Perceived Exertion The RPE was recorded every 30 s during the trials using the Borg scale (ranging from 6 to 20) (Borg, 1982). The subjects were asked to report a peripheral RPE using cues from joints and muscles of the legs (RPElegs) and an overall RPE using cues derived from all sensations experienced during exercise (RPEoverall). The individual RPEoverall and RPElegs values were regressed against absolute time (min) and percentage of the time to volitional exhaustion (% time to exhaustion). The slope of RPE against time and % time to exhaustion were computed using a least squares fitting procedure (Origin, Microcal, Piscataway, NJ). Cardiorespiratory Measurements The VE, VO2, VCO2, and respiratory exchange ratio (RER) were measured breath-by-breath throughout trials by an on-line breath-by-breath gas analyzer and converted to 5-s averages for subsequent mathematical modeling (Slawinski, Demarle, Koralsztein, & Billat, 2001). The HR was continuously measured (Polar S810i heart rate monitor, Polar Electro OY, Kempele, Finland) and averaged at the same intervals of gas exchange. Data of VE, VO2, and HR were fitted to single and double exponential models (Barstow & Mole, 1991; O¨zyener, Rossiter, Ward, & Whipp, 2001), using a non-linear least squares fitting procedure (Origin, Microcal). The VE, VO2, and HR kinetics were described as a time function using the following equations: DyðtÞ ¼ A1 ð1 $ e$ðt$d1 Þ=t1 Þ

Blood Sample Analysis Ten milliliters of venous blood were drawn before and immediately after exhaustion in each trial using a Teflon catheter inserted into a brachial vein. The blood samples were immediately transferred to tubes containing EDTA and centrifuged for 10 min (3000 rpm at 41C). The supernatant plasma was stored at $ 801C until analysis. Plasma glucose concentrations were analyzed by enzymatic colorimetric method (Biotećnica, Saõ Paulo, Brazil), plasma insulin and cortisol concentrations by commercial kits RIA (DPC, Los Angeles, CA), and plasma potassium concentrations by ion selective determination (AVL 9180, Roche, Basel, Switzerland). Values for pH were determined by biological paper (Merck, Darmstadt, Germany) while plasma norepinephrine and epinephrine concentrations were measured by HPLC using the ion-pair reverse phase chromatography coupled with electrochemical detection (0.5 V) (Naffah-Mazzacoratti, Casarini, Fernandes, & Cavalheiro, 1992). Micro samples (25 ml) were used for determinations of blood lactate concentrations (YSI 1500 Sport, Yellow Springs Instruments). The hormonal and metabolic variables were expressed as absolute values as well as delta of change (D) (difference between values obtained at exhaustion and at rest). Statistical Analysis The data are presented as means & SD. The differences among the slopes of RPE, the kinetics parameters, and the deltas of physiological parameters for the three experimental conditions were accessed by a mixed model with Bonferroni correction. The different times to exhaustion, which did not show a Gaussian distribution, were compared through a non-parametric Friedman test and a Wilcoxon test. The mixed model with Bonferroni correction was used to assess whether experimental condition and exercise influenced the RPE and physiological variables. Pearson’s correlation coefficient was carried out on pooled data in order to determine the relationship between variables. A stepwise multiple linear regression was applied to verify which variables would predict the slope of RPEoverall. Only variables significantly associated with the slope of RPEoverall were tested as independent variables in the predictive model. Significance was accepted at po.05.

ð1Þ

Results

ð2Þ

Time to Exhaustion The time to exhaustion was significantly lower in low than high CHO, but there was no difference between CHO manipulation and control condition, F(2,10) 5 15.872, po.05 (Figure 1). The CV of the time to exhaustion in control, high, and low CHO conditions was 43.2, 47.4, and 30.7%, respectively.

or DyðtÞ ¼ A1 ð1 $ e$ðt$d1 Þ=t1 Þ þ A2 ð1 $ e$ðt$d2 Þ=t2 Þ

constant, and ‘1’ and ‘2’ denote the fast and slow components, respectively. The double-exponential modeling provided a better characterization of VO2 and HR kinetics, while the mono-exponential modeling better described the VE kinetics. Comparison between models was based on mean square residual tests (F test). Therefore, parameters derived from equation 1 were used to characterize the VE responses, while parameters derived from equation 2 were used to characterize the VO2 and HR responses. The A1 and A11A2 for VO2 were also expressed in terms of functional ‘gain’ (G 5 DVO2/DW) (Perrey, Candau, Rouillon, & Hughson, 2003).

where y(t) is the change in dependent variable (VE, VO2, or HR) at time t, A is the amplitude, d is the time delay, t is the time

280

A. E. Lima-Silva et al.

Figure 1. Time to exhaustion in control, high, and low CHO diet conditions. Values are reported as mean (SD). nSignificantly lower than high CHO diet (po.05).

Rating of Perceived Exertion The RPEoverall increased significantly in each condition as a time function, F(2,36) 5 42.239, po.05 (Figure 2A). There were no differences in RPEoverall between control and CHO manipulation conditions at first 30 s, F(2,10) 5 2.891, p4.05. However, RPEoverall at exhaustion was significantly lower in low CHO than control, F(2,10) 5 4.391, po.05, but the low CHO was not significantly different from the high CHO condition (Figure 3A). There were no differences in RPEoverall at exhaustion between control and high CHO. Similarly, RPElegs increased significantly in each condition as a time effect, F(2,36) 5 21.110, po.05 (Figure 2B). There were no differences in RPElegs among the different conditions neither

Figure 3. Ratings of perceived exertion (RPE) at the first 30 s and the exercise termination in the control, high, and low CHO diet conditions. Values are expressed as mean (SD). Panel A shows the RPEoverall derived from general sensations experienced during exercise, and panel B shows the RPElegs derived from specific sensations from joints and skeletal muscle. The SD of RPEoverall values at the exhaustion point in control condition (Panel A) are not presented since all subjects attained ‘‘20score’’ in the Borg scale. nSignificantly lower than control (po.05).

at first 30 s, F(2,10) 5 2.645, p4.05, nor at exhaustion point, F(2,10) 5 1.585, p4.05 (Figure 3B). When expressed against absolute time, the rate of increase in RPEoverall, F(2,10) 5 4.230, po.05, and RPElegs F(2,10) 5 5.435, po.05, was significantly higher in low CHO than the control condition (Table 1). No difference was found between low CHO and high CHO or between high CHO and the control condition. When RPEoverall, F(2,10) 5 0.360, p4.05, and RPElegs, F(2,10) 5 1.444, p4.05, were expressed as percentage of the time to exhaustion, the rate of increase was similar in all conditions (Table 1).

Cardiorespiratory The VO2 increased continually in all tests until the peak VO2 had been reached (Figure 4). The values for control and high CHO were not significantly different from those attained during the incremental test, but for low CHO they were significantly lower Table 1. Absolute (units ! min " 1) and Relative (units ! % " 1) Values (mean # SD) for Rate of Increase in Rating Perceived Effort (Overall and Legs) in Control, High, and Low CHO Diet Conditions Measure

Figure 2. Rating of perceived exertion (RPE) as a function of exercise duration in control, high, and low CHO diet conditions of a subject. Panel A shows the RPEoverall derived from general sensations experienced during exercise, and panel B shows the RPElegs derived from specific sensations from joints and skeletal muscle.

RPEoverall Absolute (units ! min " 1) Relative (units ! % " 1) RPElegs Absolute (units ! min " 1) Relative (units ! % " 1)

Control

High CHO diet Low CHO diet

1.00 # 0.42 0.08 # 0.02

1.23 # 0.74 0.10 # 0.03

1.49 # 0.70n 0.10 # 0.03

0.80 # 0.36 0.07 # 0.03

0.96 # 0.83 0.09 # 0.06

1.13 # 0.79n 0.09 # 0.05

Significantly higher than control (po.05).

n


281 Table 2. Parameters of the Oxygen Uptake Kinetics in Control, High, and Low CHO Diet Conditions Parameters

Control

High CHO diet Low CHO diet

#1

2.53 ! 0.39 2.44 ! 0.36 A1 (L " min ) 18.1 ! 11.4 15.9 ! 8.3 d1 (s) 41.9 ! 10.6 32.5 ! 7.7 t1 (s) A2 (L " min # 1) 0.31 ! 0.09 0.30 ! 0.11 d2 (s) 158.9 ! 91.6 166.6 ! 97.2 275.9 ! 135.1 198.0 ! 13.7 t2 (s) 2.84 ! 0.41 2.74 ! 0.43 A11A2 (L " min # 1) G1 (ml " min # 1 " W # 1) 10.3 ! 1.3 9.9 ! 1.2 11.3 ! 1.3 Gtot (ml " min # 1 " W # 1) 11.8 ! 1.5

Figure 4. Oxygen uptake (VO2) as a function of exercise duration in control, high, and low CHO diet conditions. Values are expressed as mean (SD). nSignificantly lower than VO2peak (po.05).

than those attained during the maximal incremental test, F(3,15) 5 4.444, po.05 (Figure 4). The A1 (F(2,10) 5 2.230, p4.05), d1 (F(2,10) 5 0.140, p4.05), and G1 (F(2,10) 5 1.978, p4.05) for the primary VO2 transient was similar between conditions (Table 2). However, values of t1 in low CHO were significantly higher than in high CHO, F(2,10) 5 5.168, po.05. The slow component (A2) was considerably lower for low CHO than control or high CHO, F(2,10) 5 12.338, po.05. Consequently, Atot (F(2,10) 5 4.901, po.05) and Gtot (F(2,10) 5 6.432, po.05) were also lower in low than in high CHO. There were no significant differences among the three experimental conditions for d2 (F(2,10) 5 1.077, p4.05) and t2 (F(2,10) 5 1.598, p4.05). The estimated A1 for VE was significantly lower in low CHO than in control or high CHO (F(2,10) 5 5.687, po.05, 84.7 ! 7.5, 97.2 ! 20.3, 93.7 ! 14.9 L " min # 1, respectively). No differences were found for A1 values between control and high CHO conditions. Values of t1 in low CHO were significantly lower than high CHO, but it was not different from control situation (F(2,10) 5 7.217, po.05, 128.5 ! 46.2, 182.1 ! 89.5, 158.3 ! 70.8 s, respectively). No differences were found for t1 values between control and the high CHO condition. The estimated parameters for HR were not different among the three conditions. Metabolic and Hormonal Responses RER increased significantly in each condition as a time effect, F(2,36) 5 44.948, po.05, but was significantly lower in low than

2.18 15.4 47.9 0.07 220.1 161.1 2.24 8.4 8.7

! ! ! ! ! ! ! ! !

0.71 9.4 19.2w 0.08n 19.8 119.8 0.75n 2.5 2.5w

Note: A1, d1, and t1: amplitude, time delay, and time constant of the fast component, respectively. A2, d2, and t2: amplitude, time delay, and time constant of the slow component, respectively. A11A2, G1, and Gtot: total VO2 amplitude, functional gain for fast component, and total VO2 amplitude (functional gain 5 DVO2/DW). n Significantly lower than control and high CHO diet (po.05). w Significantly lower than high CHO diet (po.05).

high CHO, F(2,36) 5 4.440, po.05. No differences were found between low CHO and control or between control and high CHO. Significant exercise effects on blood lactate concentrations, F(1,10) 5 152.426, po.05, and pH, F(1,10) 5 6.775, po.05, were observed in all conditions, but no effects of CHO manipulation or in the interaction of CHO manipulation-exercise were found (Table 3). Plasma norepinephrine concentrations increased over time in all conditions, F(1,10) 5 13.108, po.05, but the values were higher in low CHO than control, F(2,10) 5 4.620, po.05. No significant exercise or condition effect was found for potassium, glucose, epinephrine, insulin, or cortisol concentrations.

Correlations Analyses The slope of RPEoverall was positively associated with the slope of RPElegs (r 5 0.55, po.05), D of insulin (r 5 0.60, po.05) and t1 of HR (r 5 0.49, po.05), and was negatively associated with the A1 (r 5 # 0.52, po.05), A2 (r 5 # 0.57, po.05) and Atot (r 5 # 0.64, po.05) of VO2 (Figure 5). The final predictive model of the slope of RPEoverall was obtained including Atot of VO2 and slope of RPElegs as predictive variables (r2 5 0.54). Time to exhaustion was negatively associated only with the rate of increase in RPEoverall (r2 5 0.61).

Table 3. The Blood Lactate, pH, Potassium, Insulin, Glucose, Epinephrine, Norepinephrine and Cortisol During Rest and Exhaustion in Control, High and Low CHO Diet Conditions. Values Are Mean ! SD Control Rest (mmol " L # 1)nn

Lactate pH (units)nn Potassium (mmol " L # 1) Insulin (uU " mL # 1) Glucose (mmol " L # 1) Epinephrine (pg " mL # 1) Norepinephrine (pg " mL # 1)n,nn Cortisol (nmol " L # 1)

0.50 7.50 3.80 1.90 4.20 62.1 476.1 346.1

! ! ! ! ! ! ! !

0.32 0.42 0.53 0.60 1.33 23.8 240.8 34.2

High CHO diet

Exhaustion 7.93 7.01 4.18 2.58 4.44 67.8 712.4 356.3

! ! ! ! ! ! ! !

2.11 0.26 0.59 0.62 1.52 20.7 293.2 66.5

Rest 0.57 7.30 4.10 3.20 4.03 52.3 354.2 431.4

! ! ! ! ! ! ! !

Significant principal effect of CHO condition (low CHO diet4control, po.05). Significant principal effect of exercise (exhaustion different from the rest, po.05).

n

nn

0.37 0.25 0.32 0.60 0.98 21.6 125.5 97.9

Low CHO diet

Exhaustion 6.71 6.81 4.37 3.53 4.64 66.9 1108.4 519.4

! ! ! ! ! ! ! !

3.07 0.38 0.42 0.63 0.73 44.1 680.1 218.1

Rest 0.61 7.40 3.80 3.58 4.58 49.1 480.1 497.5

! ! ! ! ! ! ! !

0.18 0.71 0.33 0.62 0.61 27.1 106.5 307.4

Eexhaustion 6.03 7.25 4.17 3.22 4.48 77.9 1285.7 354.4

! ! ! ! ! ! ! !

1.43 0.27 0.85 1.34 0.80 34.3 640.1 151.5

282

Figure 5. Correlation coefficient between the rate of increase in RPE (units ! min " 1) and the fast component amplitude of VO2 (A1, panel A), slow component (A2, panel B), and total amplitude of VO2 (Atot, panel C). Correlation coefficient was obtained using the pooled data (control, high, and low CHO diets).

Discussion The major findings of the present study are: (1) reduced endogenous CHO availability caused a slower primary VO2 response and a reduced slow component of the VO2 during high intensity exercise; (2) a remarkable association between slope of the RPEoverall, total amplitude of the VO2 and slope of the RPElegs was found; and (3) time to exhaustion was associated only with slope of the RPEoverall. These findings suggest that an enhanced perception of effort from the periphery can act as afferent signals to the CNS, which could be integrated in the production of the RPE calculations. The linear increase in RPE during exercise corroborates previous findings which compared glycogen-loaded with glycogendepleted states (Noakes, 2004), fatigued with non-fatigued states (Eston et al., 2007) and hot with cool environments (Crewe et al.,

A. E. Lima-Silva et al. 2008), suggesting that the rate of increase in RPE could be set at the initial period of exercise in a feedforward manner. In addition, this also was supported by our data expressed as a percentage of the time to exhaustion, since the RPE increased at the same slope in all conditions. It should be also noted that the highest values of RPEoverall were found in the control and high CHO conditions, whereas the exercise terminated at a lower maximal RPEoverall value in low CHO. However, the values of RPElegs in three experimental conditions did not reach the maximal limit on the Borg scale, and there were no differences between trials. It is likely that the muscle strain is sensed and monitored by CNS during the whole exercise. As a consequence of this sensory feedback, the CNS probably establishes a maximal tolerable RPElegs for prevention of a catastrophic mechanical failure and protection of skeletal muscle integrity, interrupting the exercise at a low CHO level before maximal RPEoverall is reached. This hypothesis is furthermore supported by a significant relationship found between the slope of RPEoverall and the slope of RPElegs. The initial RPEoverall and RPElegs during the first 30 sec were not different between the different trials. However, the low CHO manipulation caused a faster slope in both RPEoverall and RPElegs, presumably because it increased the afferent feedback from the periphery. The significant correlations between the rate of increase of RPEoverall and D for insulin, t1 for HR, and amplitude for VO2 support the conclusion that the RPE is influenced by multiple afferent feedbacks (Hampson et al., 2001). However, a stepwise linear regression model showed that perceived muscle strain and rate of energy production (i.e., VO2) may be important variables monitored by the CNS during high intense exercise. The effect of low CHO on perceived muscle strain has been recently documented (Johnson et al., 2006). Subjects submitted to an exercise-diet protocol for reduction of CHO availability reported greater perception of tiredness and weakness in legs even though they were not aware that their CHO stores had been reduced experimentally. Therefore, it could be suggested that tiredness and weakness sensations could be important for the RPE calculations by the SNC. In fact, a strong correlation between the rate of increase in RPElegs and the rate of increase in RPEoverall was observed, which supports the theory that the muscle strain might play an important role in the perceived exertion generated during high intensity exercise (Hollander, Kilpatrick, Ramadan, Reeves, Francois, et al., 2008). Furthermore, low CHO reduced the amplitude of VO2 slow component. Some studies have demonstrated that the neuromuscular blockade of type I fibers or the reduction in glycogen content of type I fibers can increase the amplitude of the slow VO2 component (Krustrup, Secher, Relu, Hellsten, So¨derlund, & Bangsbo, 2008; Krustrup et al., 2004). Additionally, Carter et al. (2004) found that the amplitude of the VO2 primary component was significantly larger, while the amplitude of the VO2 slow component was significantly reduced after glycogen depletion of type II fibers. The selective depletion of type II skeletal muscle fibers could have shifted the recruitment pattern towards type I muscle fibers. Although we did not measure the muscle glycogen content, the exercise protocol used was designed to reduce the glycogen stores in both fibers (Bergstrom et al., 1967; Gollnick et al., 1973, 1974; Heigenhauser et al., 1983). Thus, some pool of type II fibers may have not been recruited during the last stages of high intense exercise and could not be replaced by type I fiber recruitment. As the motor unit rotation or substitution might act


283

as a compensatory mechanism in reducing the fatigue effects on individual fibers (Jensen, Pilegaard, & Sjogaard, 2000), the afferent signals from depleted glycogen type II fibers could have led the CNS to decrease the recruitment level in these fibers. Thus, the subconscious brain may have ‘shortened’ the period of exercise, matching it to a sustainable level of effort that could be maintained while preserving the muscle integrity. This was corroborated by the negative correlation between the A2 parameter and the slope of RPEoverall, suggesting that a reduced recruitment of Type II fibers as exercise progresses leads to a greater perception of effort. In the present study, the time to exhaustion was strongly associated with the rate of increase in RPEoverall. In contrast, neither the rate of increase in RPElegs nor the total VO2 amplitude was associated with time to exhaustion. Rather, these variables were associated with the rate of increase in RPEoverall. These

results may suggest that multiple afferent feedbacks (VO2, muscle strain, etc.) act as a signals to the CNS, which appear to use this information to modulate the RPEoverall and hence the total exercise duration. In summary, our results show that the low CHO reduces the amplitude of the VO2 response and increases the rate of increase in RPEoverall and RPElegs during high intensity exercise. Although the RPEoverall was strongly correlated with different afferent signals, the total amplitude of VO2 and perceived muscle strain appear to be important afferent signals from the periphery during high intensity exercise. As the time to exhaustion was significantly correlated only with the rate of increase in RPEoverall, it could be suggested that the CNS integrates feedback from multiple peripheral signals and makes the decision about exercise duration and hence exercise termination based on the conscious perception of effort.

REFERENCES Arkinstall, M. J., Bruce, C. R., Clark, S. A., Rickards, C. A., Burke, L. M., & Hawley, J. A. (2004). Regulation of fuel metabolism by preexercise muscle glycogen content and exercise intensity. Journal of Applied Physiology, 97, 2275–2283. Baldwin, J., Snow, R. J., Gibala, M. J., Garnham, A., Howarth, K., & Febbraio, M. A. (2003). Glycogen availability does not affect the TCA cycle or TAN pools during prolonged, fatiguing exercise. Journal of Applied Physiology, 94, 2181–2187. Barstow, T., & Mole, P. (1991). Linear and nonlinear characteristics of oxygen uptake kinetics during heavy exercise. Journal of Applied Physiology, 71, 2099–2106. Bergstrom, J., Hermansen, L., Hultman, E., & Saltin, B. (1967). Diet, muscle glycogen and physical performance. Acta Physiologica Scandinavica, 71, 140–150. Borg, G. A. (1982). Psychophysical bases of perceived exertion. Medicine and Science in Sports and Exercise, 14, 377–381. Bouckaert, J., Jones, A. M., & Koppo, K. (2004). Effect of glycogen depletion on the oxygen uptake slow component in humans. International Journal of Sports Medicine, 25, 351–356. Busse, M. W., Maassen, N., & Konrad, H. (1991). Relation between plasma K1 and ventilation during incremental exercise after glycogen depletion and repletion in man. Journal of Physiology, 443, 469–476. Carter, H., Pringle, J. S., Boobis, L., Jones, A. M., & Doust, J. H. (2004). Muscle glycogen depletion alters oxygen uptake kinetics during heavy exercise. Medicine and Science in Sports and Exercise, 36, 965–972. Crewe, H., Tucker, R., & Noakes, T. D. (2008). The rate of increase in rating of perceived exertion predicts the duration of exercise to fatigue at a fixed power output in different environmental conditions. European Journal of Applied Physiology, 103, 569–577. Eston, R., Faulkner, J., St.Clair Gibson, A., Noakes, T., & Parfitt, G. (2007). The effect of antecedent fatiguing activity on the relationship between perceived exertion and physiological activity during a constant load exercise task. Psychophysiology, 44, 779–786. Gollnick, P. D., Armstrong, R. B., Sembrowich, W. L., Shepherd, R. E., & Saltin, B. (1973). Glycogen depletion pattern in human skeletal muscle fibers after heavy exercise. Journal of Applied Physiology, 34, 615–618. Gollnick, P. D., Piehl, K., & Saltin, B. (1974). Selective glycogen depletion pattern in human muscle fibres after exercise of varying intensity and at varying pedalling rates. Journal of Physiology, 241, 45–57. Hampson, D. B., St.Clair Gibson, A., Lambert, M. I., & Noakes, T. D. (2001). The influence of sensory cues on the perception of exertion during exercise and central regulation of exercise performance. Sports Medicine, 31, 935–952. Heigenhauser, G. J. F., Sutton, J. R., & Jones, N. L. (1983). Effect of glycogen depletion on the ventilatory response to exercise. Journal of Applied Physiology, 54, 470–474. Hill, A. V., & Lupton, H. (1923). Muscular exercise, lactic acid, and the supply and utilization of oxygen. Quarterly Journal of Medicine, 16, 135–171.

Hollander, D. B., Kilpatrick, M. W., Ramadan, Z. G., Reeves, G. V., Francois, M., Blakeney, A., et al. (2008). Load rather than contraction type influences rate of perceived exertion and pain. Journal of Strength and Conditional Research, 22, 1184–1193. Jensen, B. R., Pilegaard, M., & Sjogaard, G. (2000). Motor unit recruitment and rate coding in response to fatiguing shoulder abductions and subsequent recovery. European Journal of Applied Physiology, 83, 190–199. Johnson, N. A., Stannard, S. R., Chapman, P. G., & Thompson, M. W. (2006). Effect of altered pre-exercise carbohydrate availability on selection and perception of effort during prolonged cycling. European Journal of Applied Physiology, 98, 62–70. Kayser, B. (2003). Exercise begins and ends in the brain. European Journal of Applied Physiology, 90, 411–419. Krustrup, P., Secher, N. H., Relu, M. U., Hellsten, Y., So¨derlund, K., & Bangsbo, J. (2008). Neuromuscular blockade of slow twitch muscle fibers elevates muscle oxygen uptake and energy turnover during submaximal exercise in humans. Journal of Physiology, 586, 6037–6048. Krustrup, P., So¨derlund, K., Mohr, M., & Bangsbo, J. (2004). Slowtwitch fiber glycogen depletion elevates moderate-exercise fast-twitch fiber activity and 02 uptake. Medicine and Science in Sports and Exercise, 36, 973–982. Lambert, E. V., St. Clair Gibson, A., & Noakes, T. D. (2005). Complex systems model of fatigue: Integrative homoeostatic control of peripheral physiological systems during exercise in humans. British Journal of Sports Medicine, 39, 52–62. Lima-Silva, A. E., De-Oliveira, F. R., Nakamura, F. Y., & Gevaerd, M. S. (2009). Effect of carbohydrate availability on time to exhaustion in exercise performed at two different intensities. Brazilian Journal of Medical and Biological Research, 42, 404–412. Marcora, S. (2008). Perception of effort during exercise is independent of afferent feedback from skeletal muscles, heart, and lungs. Journal of Applied Physiology, 106, 2060–2062. Naffah-Mazzacoratti, M. G., Casarini, D. E., Fernandes, M. J. S., & Cavalheiro, E. A. (1992). Serum catecholamine levels determined by performance liquid chromatrography coupled with electrochemical detection. Arquivos Brasileiros de Endocrinologia e Metabologia, 36, 119–122. Noakes, T. D. (2004). Linear relationship between the perception of effort and the duration of constant load exercise that remains. Journal of Applied Physiology, 96, 1571–1572. Noakes, T. D. (2008). Rating of perceived exertion as a predictor of the duration of exercise that remains until exhaustion. British Journal of Sports Medicine, 42, 623–624. Noakes, T. D., & St. Clair Gibson, A. (2004). Logical limitations to the ‘‘catastrophe’’ models of fatigue during exercise in humans. British Journal of Sports Medicine, 38, 648–649. Noakes, T. D., St. Clair Gibson, A., & Lambert, E. V. (2004). From catastrophe to complexity: A novel model of integrative central neural regulation of effort and fatigue during exercise in humans. British Journal of Sports Medicine, 38, 511–514.

284 Osborne, M. A., & Schneider, D. A. (2006). Muscle glycogen reduction in man: Relationship between surface EMG activity and oxygen uptake kinetics during heavy exercise. Experimental Physiology, 91, 179– 189. O¨zyener, F., Rossiter, H. B., Ward, S. A., & Whipp, B. J. (2001). Influence of exercise intensity on the on- and off-transient kinetics of pulmonary oxygen uptake in humans. Journal of Physiology, 533, 891–902. Perrey, S., Candau, R., Rouillon, J. D., & Hughson, R. L. (2003). The effect of prolonged submaximal exercise on gas exchange kinetics and ventilation during heavy exercise in humans. European Journal of Applied Physiology, 89, 587–594. Pitsiladis, Y. P., & Maughan, R. J. (1999). The effects of exercise and diet manipulation on the capacity to perform prolonged exercise in the heat and in the cold in trained humans. Journal of Physiology, 517, 919–930. Ribeiro, J. P., Yang, J., Adams, R. P., Kuca, B., & Knutten, H. G. (1986). Effect of different incremental exercise protocols on the de-

A. E. Lima-Silva et al. termination of lactate and ventilatory thresholds. Brazilian Journal of Medical and Biological Research, 19, 109–117. Slawinski, J., Demarle, A., Koralsztein, J. P., & Billat, V. (2001). Effect of supra-lactate threshold training on the relationship between mechanical stride descriptors and aerobic energy cost in trained runners. Archives of Physiology and Biochemistry, 109, 110–116. Tucker, R. (2009). The anticipatory regulation of performance: The physiological basis for pacing strategies and the development of a perception-based model for exercise performance. British Journal of Sports Medicine, 43, 392–400. Weltman, A., Weltman, J., Rutt, R., Seip, R., Levine, S., Snead, D., et al. (1989). Percentages of maximal heart rate, heart rate reserve, and VO2peak for determining endurance training intensity in sedentary women. International Journal of Sports Medicine, 10, 212–216.

(Received December 3, 2008; Accepted April 4, 2010)


Sleep state instabilities in major depressive disorder: Detection and quantification with electrocardiogrambased cardiopulmonary coupling analysis

ALBERT C. YANG,a,b,c CHENG-HUNG YANG,b,d CHEN-JEE HONG,b,d SHIH-JEN TSAI,b,d CHUNG-HSUN KUO,d CHUNG-KANG PENG,e JOSEPH E. MIETUS,e ARY L. GOLDBERGER,e and ROBERT J. THOMASf a

Department of Psychiatry, Chu-Tung Veterans Hospital, Hsin-Chu County, Taiwan Division of Psychiatry, School of Medicine, National Yang-Ming University, Taipei, Taiwan c Institute of Clinical Medicine, National Yang-Ming University, Taipei, Taiwan d Department of Psychiatry, Taipei Veterans General Hospital, Taipei, Taiwan e Division of Interdisciplinary Medicine and Biotechnology and Margret and H. A. Rey Institute for Nonlinear Dynamics in Medicine, Beth Israel Deaconess Medical Center/Harvard Medical School, Boston, MA and Wyss Institute for Biologically Inspired Engineering at Harvard University, Boston, MA f Division of Pulmonary, Critical Care and Sleep, Beth Israel Deaconess Medical Center/Harvard Medical School, Boston, MA b

Abstract Sleep disruption is an important aspect of major depressive disorder but lacks an objective and inexpensive means of assessment. We evaluated the utility of electrocardiogram (ECG)-based cardiopulmonary coupling analysis to quantify physiologic sleep stability in patients with major depression. Relative to controls, unmedicated depressed patients had a reduction in high-frequency coupling, an index of stable sleep, an increase in low-frequency coupling, an index of unstable sleep, and an increase in very-low-frequency coupling, an index of wakefulness/REM sleep. The medicated depressed group showed a restoration of stable sleep to a level comparable with that of the control group. ECG-based cardiopulmonary coupling analysis may provide a simple, cost-efficient point-of-care method to quantify sleep quality/ stability and to objectively evaluate the severity of insomnia in patients with major depression. Descriptors: Depression, Insomnia, Sleep stability, Cyclic alternating pattern, Cardiopulmonary coupling analysis, Electrocardiogram-derived sleep spectrogram

sleep quality. Currently, objective assessments of sleep physiology rely primarily on polysomnography (Buysse, Ancoli-Israel, Edinger, Lichstein, & Morin, 2006). Sleep staging, a key component of polysomnography, is based on arbitrary criteria that relate to the amplitude and morphology of electroencephalographic (EEG) signals (Iber, 2007). The limitations of applying conventional EEG staging to assess insomnia include the findings that (1) EEG staging is poorly correlated with perceived sleep quality (Armitage, Trivedi, Hoffmann, & Rush, 1997; Saletu, 1975), (2) benzodiazepines may reduce ‘‘deep’’ sleep (i.e., decreased delta power in EEG signals) but still improve sleep continuity and subjective sleep quality (Achermann & Borbely, 1987), and (3) the fact that the majority of adult sleep is characterized by second-stage non-REM sleep. A complementary approach used to quantify insomnia is the assessment of sleep ‘‘stability,’’ originally implemented with an EEG morphological marker that has been termed the cyclic alternating pattern (CAP) (Ferre, Guilleminault, & Lopes, 2006; Terzano, Mancia, Salati, Costani, Decembrino, & Parrino, 1985; Terzano & Parrino, 2000). CAP is a state of phasic EEG activity associated with microarousals during sleep and is, therefore, considered to be an index of sleep instability. Increased EEG-CAP during sleep has been

Insomnia, characterized by a perception of poor sleep quality, is a common symptom comorbid with major depression and other mental illness (Sateia & Nowell, 2004). However, unlike other sleep disorders, such as sleep apnea, in which the nature and severity of the illness are quantifiable, the assessment of insomnia is often subjective and is largely based on reports of perceived The authors wish to thank Shan-Ing Chen and Chen-Ru Wang for their excellent technical assistance. This work was supported by the National Science Council of Taiwan (NSC 95-2314-B-075-111), Taipei Veterans General Hospital (V96C1-083, V97C1-132, V97F-005), the G. Harold and Leila Y. Mathers Foundation, the James S. McDonnell Foundation, the NIH-sponsored Research Resource for Complex Physiologic Signals (UO1EB008577), and DynaDx Corporation, Mountain View, California. The algorithm used in the analysis is based on the cardiopulmonary coupling software developed at the Beth Israel Deaconess Medical Center, Boston, MA by CKP, JEM, ALG, and RJT and licensed to Embla, Inc. CKP and JEM have financial interest in DynaDx Corp. Address correspondence to: Albert C. Yang, M.D., Department of Psychiatry, Chu-Tung Veterans Hospital, No. 81, Jhongfong Road, Sec. 1, Chu-Tung Township, Hsin-Chu County 31064, Taiwan. E-mail: [email protected] 285

286 found in patients with major depression and in those with primary insomnia. (Farina, Della Marca, Grochocinski, Mazza, Buysse, et al., 2003; Lopes, Quera-Salva, & Guilleminault, 2007; Terzano, Parrino, Spaggiari, Palomba, Rossi, & Smerieri, 2003). Recently, we demonstrated that the presence of EEG-CAP during sleep is associated with coupled modulations in respiratory and autonomic functions, thus raising the possibility of utilizing a continuous electrocardiographic (ECG) signal alone to quantify sleep stability (Thomas, Mietus, Peng, & Goldberger, 2005). This novel method, termed cardiopulmonary coupling (CPC) analysis, has been developed to measure sleep quality and to detect and phenotype sleep apnea based solely on the continuous ECG signal (Thomas, Mietus, Peng, Gilmartin, Daly, et al., 2007; Thomas et al., 2005). Improved sleep stability has also been quantified by CPC analysis in patients with heart failure undergoing a Tai Chi exercise program (Yeh, Mietus, Peng, Phillips, Davis, et al., 2008). Since depression is associated with reduced sleep stability, as detected by the presence of increased EEG-CAP, and EEG-CAP is correlated with an alteration in autonomic function, an unstable, fragmented sleep pattern may be detectable by CPC analysis in depressed individuals. Consequently, in the present study, we tested the following two hypotheses: (1) stable sleep, as quantified by ECG-derived CPC analysis, is reduced in major depression, and (2) excessive physiologic sleep instability in depression is reversed by the use of hypnotics. We tested these hypotheses by employing the CPC method to investigate and quantify physiologic sleep stability in a group of depressed patients with comorbid insomnia who were either medication-free or were using hypnotics, and in a group of healthy controls. Methods Participants and Clinical Assessment One hundred Han-Chinese patients with major depressive disorder (63% female, aged 42.9 ! 10.7 years), as defined by the Diagnostic and Statistical Manual of Mental Disorders, 4th Edition, were recruited from the Taipei Veterans General Hospital, Taiwan. All subjects gave informed consent before commencement of the study. The protocol was approved by the institutional review board of the Taipei Veterans General Hospital (Taipei, Taiwan). Patient inclusion criteria were: (1) age of 20–65 years; (2) presence of a major depressive episode in either acute or early remission phase; (3) subjective complaints of difficulties initiating or maintaining sleep for a minimum of three nights per week associated with the current depressive episode. Patients were excluded if they met the following criteria: (1) remission of depression for over 2 months; (2) presence of major psychiatric disorders (except personality disorders) other than depression; (3) cardiac arrhythmia, including continuous atrial or ventricular bigeminy, atrial fibrillation, or atrial flutter; (4) severe or acute medical illness within 3 months before the study; (5) other confounding medical conditions, including chronic pain, alcohol/substance abuse or dependence, pregnancy, sleep apnea, and systemic diseases (hypertension and diabetes mellitus), that are known to affect sleep or autonomic function. Moreover, to minimize the confounding effects of medication, patients were also excluded if they used medications which have known effects on autonomic function: anti-cholinergic medication, tricyclic antidepressants with anti-cholinergic effects, and beta-blockers. Psychiatric diagnoses based on DSM-IV criteria were determined by a consensus of at least two psychiatrists. Depression

A. C. Yang et al. severity was evaluated by the self-reported Beck Depression Inventory (BDI) (Beck, Ward, Mendelson, Mock, & Erbaugh, 1961) and the psychiatrist-rated Hamilton Depression Rating Scale (HAM-D, 17 items) (Hamilton, 1960). Subjective sleep quality was assessed by the Pittsburgh Sleep Quality Index (PSQI) (Buysse, Reynolds, Monk, Berman, & Kupfer, 1989) and subjective sleepiness by the Epworth Sleepiness Scale (ESS) (Johns, 1991). Ninety-one age- and sex-matched healthy control subjects (63% female, aged 42.1 ! 12.6 years) were recruited during the same period using identical assessment procedures. Neither the control subjects nor their first-degree relatives had a history of mental illness. The control subjects reported no use of hypnotics in the past month and had no clinically significant insomnia as evaluated by a psychiatrist. Continuous ECG Monitoring Holter recordings (MyECG E3-80 Portable Recorder, Microstar Inc., Taipei, Taiwan) were used to obtain continuous 24-h ECG data. All healthy control subjects and 77% (n 5 77) of the depressed patients received ECG monitoring at home; the other depressed individuals (n 5 23; 23%) received ECG monitoring at an acute psychiatric ward. Participants were asked to maintain their usual daily activities and to avoid smoking and drinking alcoholic beverages when undergoing testing. Half of the depressed subjects (n 5 50, 54% female, aged 41.7 ! 11.3 years) agreed to undergo ECG monitoring before initiating medication treatment and, therefore, were medicationfree on the night of the ECG evaluation. The remainder of the depressed subjects (n 5 50, 72% female, aged 44.2 ! 10.0 years) had received short- (n 5 32), intermediate- (n 5 25), or long-acting benzodiazepines (n 5 24) before entering the study, and continued medication treatment during the night of the ECG monitoring. Most of these patients received a combination of short/intermediate- or short/long-acting hypnotics to treat insomnia. Forty-four of the 50 medicated patients who received hypnotics were also treated with antidepressants. Cardiopulmonary Coupling Analysis The autonomic nervous system has predictable characteristics that vary according to sleep depth and type (Dumont, Jurysta, Lanquart, Migeotte, van de Borne, & Linkowski, 2004; Kuo, Shaw, Lai, & Yang, 2008). CPC analysis is derived from an estimation of the coupling between the autonomic and respiratory drives, using heart rate and respiratory modulation of QRS amplitude, respectively. This dual information can be extracted from a single channel of ECG (Thomas et al., 2005). The ECGderived respiration signal has been described in detail (Moody, Mark, Bump, Berman, Mietus, & Goldberger, 1986), and is highly correlated with the actual respiration waveforms (Yeragani, Appaya, Seema, Kumar, & Tancer, 2005). The algorithm of CPC analysis involves the following steps: the extraction of heart rate and respiration waveforms from the ECG signal, and a subsequent estimation of the cross-spectral power and coherence between the ECG-derived respiration and the heart rate signals to determine sleep state. The analysis window width is 512 s, moving forward in 128-s increments until the entire time series is analyzed. Three physiological sleep states are derived from CPC analysis; namely, stable, unstable, and REM/wakeful states (Thomas et al., 2005). Specifically, stable sleep is associated with highfrequency coupling between the heart rate and the respiration at frequencies of 0.1 to 0.4 Hz, and it is correlated with an EEG

Sleep state instabilities in major depression

287

Figure 1. A comparison of ECG-derived sleep spectrograms from a healthy control subject and a patient with depression. Left panel: A 34-year-old depressed female; high-frequency coupling (stable sleep): 21%; low-frequency coupling (unstable sleep): 46%; and very-low-frequency coupling (REM/ wakefulness): 28%. Right panel: A 30-year-old healthy female; high-frequency coupling: 76%; low-frequency coupling: 13%; and very-low-frequency coupling: 12%. The spectrographic profiles of the two subjects are visually distinguishable: the depressed individual has fragmented sleep states whereas the healthy subject has more continuous high-frequency coupling sleep.

non-CAP sleep state. In contrast, unstable sleep is associated with low-frequency coupling between the heart rate and the respiration over a range of 0.01 to 0.1 Hz, and it is correlated with an EEG CAP sleep state. The wakeful state and REM sleep are associated with the presence of very-low-frequency coupling between the heart rate and the respiration below 0.01 Hz correlates. Without the recording of muscle tone, REM sleep is not distinguishable from the wakeful state, and the detection of verylow-frequency coupling may reflect contributions from both states. Additional information about sleep and wakefulness times reported by the subject was used to constrain the analysis to the approximate sleep period. The ECG signals were automatically processed and analyzed to generate a sleep spectrogram. These CPC-derived sleep states were then used to assess sleep structure. Figure 1 illustrates the comparison of sleep spectrograms between a healthy control subject and a depressed patient. Statistical Analysis The STATISTICA program (Version 8.0; StatSoft, Inc., Tulsa, OK) was used for all statistical analyses. A p value of less than .05 (two-tailed) was required for statistical significance. Fisher’s exact test was used to compare categorical variables. Analysis was performed to compare the entire depression group with the healthy subjects and to compare the two depression groups (medicated and unmedicated) with healthy subjects. Between-group comparisons were performed with a one-way analysis of variance (ANOVA). Tukey’s honest significant difference post hoc test was used to test the significance among groups. Pearson’s correlation was applied to determine the associations between CPC sleep indices and scores from sleep/depression questionnaires. Results Subjects Demographic and clinical assessment data for patient and control groups are presented in Table 1. There was no significant

difference between the depression and control groups with regard to age and gender. Compared with controls, depressed patients had higher PSQI scores (po.001), higher insomnia scores derived from the BDI (po.001), and higher insomnia scores derived from the HAM-D (po.001). Daytime somnolence measured by the ESS score did not differ between the patient and control groups. Of note, 37% of the control subjects were also classified as having poor sleep (PSQI45), compared with 93% of the depressed patients. Cardiopulmonary Coupling Analyses Objective sleep indices derived from the CPC analysis are presented in Table 2. Medication-free depressed patients had a significantly lower percentage of stable sleep compared with healthy subjects (po.001), a higher percentage of unstable sleep (p 5 .007), and a higher percentage of REM/wakeful state (po.001). Furthermore, compared with controls, medicationfree depressed patients had a longer latency to the first epoch of stable sleep (po.001) and a shorter unstable sleep latency (p 5 .02). Both medicated and unmedicated depressed individuals reported significantly more time in bed compared with healthy subjects (both po.001). Depressed and control groups did not differ on the REM/wakeful state latency (p 5 .91). Correlations Between CPC Indices and Age/Questionnaires When considering all study samples (medicated, unmedicated, and healthy controls), there was no significant correlation between CPC sleep indices and questionnaire scores. To reduce the confounding effect of medication on perceived sleep quality, we analyzed only the correlations between CPC sleep indices and questionnaire scores among the 50 medication-free depressed patients and the 91 healthy control subjects. Stable sleep was significantly correlated with the HAM-D (r 5 ! 0.35, p 5 .002) and the BDI insomnia score (r 5 ! 0.33, p 5 .001). REM/wakeful state was significantly correlated with the HAM-D score (r 5 0.42, po.001), the PSQI (r 5 0.31, p 5 .001), the BDI (r 5 0.31, p 5 .001), and the BDI insomnia score (r 5 0.41,

288

A. C. Yang et al.

Table 1. Demographic and Clinical Characteristics Patients with Major Depressive Disorder Variable Age, years Gender, male/female Type of benzodiazepine used Short-acting, n (%) Intermediate, n (%) Long-acting, n (%) Beck Depression Inventory Hamilton Depression Rating Scale, 17 items Pittsburgh Sleep Quality Index Epworth Sleepiness Scale Insomnia score derived from Beck Depression Inventory (item #16) Insomnia score derived from Hamilton Depression Rating Scale (items #4–6)

Medication-free (n 5 50)

Hypnotics (n 5 50)

Total (N 5 100)

41.5 ! 12.9 23/37

44.2 ! 10.0 14/36

42.9 ! 10.7 37/63

N/A

32 (64) 25 (50) 24 (48) 33.3 ! 14.1n 16.8 ! 7.9n 14.0 ! 3.6n 7.2 ! 6.5 2.0 ! 0.9n

32 (32) 25 (25) 24 (24) 27.2 ! 15.7n 15.6 ! 8.1n 13.0 ! 4.0n 8.1 ! 6.3 1.8 ! 1.1n

3.3 ! 2.0n

3.1 ! 1.9n

20.7 14.3 11.8 9.1 1.5

! ! ! ! !

14.9n 8.1n 4.0n 5.9 1.1n

3.0 ! 1.8n

Healthy subjects (N 5 91) 42.1 ! 12.6 34/57 N/A 6.1 1.7 4.7 9.1 0.4

! ! ! ! !

6.6 2.6 2.9 5.1 0.7

0.4 ! 0.8

Note: N/A: not applicable. Data represent mean ! 1 standard deviation unless otherwise noted. n Statistical significance (po.05) compared with healthy controls.

po.001). Daytime somnolence, as measured by the ESS, did not correlate with any of CPC sleep indices. There was no significant correlation between age and CPC sleep states.

Effect of Hypnotics on CPC Sleep Indices Finally, post hoc analyses were conducted to test the differences in CPC sleep indices between the groups of healthy controls and depressed patients (medicated or drug-free). In medicated depressed patients, statistically significant improvements of CPC sleep indices were observed, with increases in stable sleep (po.001) and reductions in both unstable sleep (po.02) and REM/wakeful states (po.001), compared with medication-free patients. Moreover, this restoration was similar to controls in the percentages of stable sleep (p 5 .84), unstable sleep (p 5 .99), and REM/wakeful states (p 5 .38). However, stable sleep latency was not fully restored in medicated depressed patients after receiving hypnotics, compared with controls (p 5 .03). Discussion The key findings of this study, based on the CPC analysis, include the following: 1) reduced stable sleep and increased unstable sleep and wakeful/REM states were found in depressed patients compared with healthy controls; 2) medicated patients demonstrated

partial restoration of stable sleep latency through the use of hypnotics, and 3) certain CPC indices correlated with subjective sleep quality and the severity of depression/insomnia. The CPC analysis described here complements traditional approaches used to assess sleep stability/quality because it objectively incorporates features of physiological dynamics not accounted for by EEG-based techniques (Armitage et al., 1997; Tworoger, Davis, Vitiello, Lentz, & McTiernan, 2005). Our findings of decreased stable sleep and increased unstable sleep and REM/wakeful states in depressed individuals are consistent with well-known features of altered EEG sleep structures in major depression, namely an increase in sleep state fragmentation, a reduction in slow wave sleep, and an increase in REM pressure (Armitage, 1995; Germain, Nofzinger, Kupfer, & Buysse, 2004; Jindal, Thase, Fasiczka, Friedman, Buysse, et al., 2002; Thase, Fasiczka, Berman, Simons, & Reynolds, 1998). Taken together, the results indicate that insomnia in depression may be not only a ‘‘brain’’ symptom but also a systemic phenomenon that represents inter-linked physiological processes, including autonomic, respiratory, and electrocortical functions (Thomas, 2007). Moreover, the CPC indices were associated with subjective sleep quality and the severity of depression, particularly the stable sleep and REM/wakeful components. These findings may enhance the utility of this ECG-based method for evaluating insomnia in depressed patients.

Table 2. Indices Derived from Cardiopulmonary Coupling Analysis Patients with Major Depressive Disorder Variable Stable sleep (high-frequency coupling), %a Unstable sleep (low-frequency coupling), %a REM/wakefulness (very-low-frequency coupling), %a Stable sleep onset latency, minutes Unstable sleep onset latency, minutes REM/wakefulness onset latency, minutes Time in bed, hours

Medication-free (n 5 50) 32.5 37.5 28.0 31.8 8.8 19.9 7.9

! ! ! ! ! ! !

12.0n 10.9n 7.1n 32.3n 13.8n 26.5n 2.1n

Hypnotics (n 5 50) 48.1 29.3 21.2 24.0 19.2 23.8 8.2

Note: Data represent mean !1 standard deviation unless otherwise noted. n Statistical significance (po.05) compared with healthy controls. a Represented as a percent of sampling windows during the sleep log estimated sleep period.

! ! ! ! ! ! !

20.0 18.3 7.1 25.0n 26.8 31.4 1.9n

Total (N 5 100) 40.3 33.4 24.6 27.9 14.0 21.8 8.0

! ! ! ! ! ! !

18.2n 15.6 7.8n 29.0n 21.8 28.9 2.0n

Healthy subjects (N 5 91) 49.9 29.5 19.5 14.4 18.9 21.9 6.7

! ! ! ! ! ! !

18.0 14.9 7.4 21.6 27.9 26.9 1.3


289

Benzodiazepine hypnotics can reduce EEG-CAP states (Ozone, Yagi, Itoh, Tamura, Inoue, et al., 2008; Parrino, Boselli, Spaggiari, Smerieri, & Terzano, 1997; Terzano, Parrino, Boselli, Dell’Orso, Moroni, & Spaggiari, 1995) and, thus, presumably reduce the physiological unstable sleep state. The use of hypnotics may, therefore, produce ‘‘masking’’ effects that could make depressed patients appear to have more stable sleep than they might have without medication. In the present study, despite the restoration of the amount of stable sleep, medicated patients still showed a longer latency to stable sleep than healthy control subjects. However, due to ethical considerations regarding drug regimen modification solely for research purposes, the type and dosage of medication was not rigorously controlled in the present study. Thus, the results of the medicated group may need to be confirmed by future prospective, randomized trials. Reduced High-Frequency Cardiopulmonary Coupling and Depression The mechanisms of stable sleep reduction, as measured by highfrequency coupling in depressed patients, are unclear; however, they may open a new window into the pathophysiology of insomnia in depression. Our prior results suggest that stable sleep is associated with healthy conditions (Thomas et al., 2005). The findings of reduced stable sleep and increased unstable and REM/wakeful state in depression are in line with excessive wakefulness-promoting influences found in depressed patients. These changes could reflect hyperarousal from increased activity in the hypothalamo-pituitary-adrenal axis, enhanced central negative affective processing, or perhaps a personality trait that increases the vulnerability to depression (Adrien, 2002). Higher order autonomic control is mediated by the anterior cingulate, the ventromedial prefrontal cortex, the amygdala, and the insular cortex. Altered activity within this network is well known in depression (Bae, MacFall, Krishnan, Payne, Steffens, & Taylor, 2006; Drevets, Price, & Furey, 2008; van Eijndhoven, van Wingen, van Oijen, Rijpkema, Goraj, et al., 2009). These brain areas are also involved in the regulation of respiratory rate and heart rhythm and may be well positioned to impact the CPC analysis, which reflects the sleep-modulated interaction between heart rate and respiratory dynamics (Liotti, Brennan, Egan, Shade, Madden, et al., 2001; von Leupoldt, Sommer, Kegat, Baumann, Klose, et al., 2008). Relationship of CPC-Derived Sleep Indices and Conventional Spectral Heart Rate Variability Although the CPC analysis involves the measure of autonomic and respiratory drives, the spectrographic coupling metrics are distinct from conventional heart rate variability (HRV) spectral analysis in that CPC analysis incorporates both respiration and heart rate signals and measures the extent of coupling between them. A previous report has suggested that the CPC analysis can detect improved sleep stability in patients with heart failure undergoing Tai-Chi exercise, whereas this improvement was undetectable by conventional HRV techniques (Yeh et al., 2008). In the present analysis, we also found generally low correlations

(r2o0.15) between CPC-derived sleep indices and spectral HRV measures (see Appendix Table A1), suggesting an independent role for CPC analysis in the quantification of sleep physiology. Depression and the Risk of Cardiovascular Disease An emerging body of research suggests that nocturnal autonomic function, measured by HRV, is altered in mood/anxiety disorders and is associated with poor sleep quality (Brosschot, Van Dijk, & Thayer, 2007; Irwin, Valladares, Motivala, Thayer, & Ehlers, 2006; Takahara, Mizuno, Hirose, Sakai, Nishii, et al., 2008). The results of this study are in line with prior reports of increased sympathetic and reduced vagal tone in depression (Nahshoni, Aravot, Aizenberg, Signler, Zalsman, et al., 2004; Udupa, Sathyaprabha, Thirthalli, Kishore, Lavekar, et al., 2007; Yeragani, Rao, Smitha, Pohl, Balon, & Srinivasan, 2002), as the parasympathetic modulation is associated with physiological stable sleep state measured by high-frequency coupling between heart rate and respiration (Thomas et al., 2005). Reduced vagal and heightened sympathetic activity are known to be associated with an increased risk of cardiovascular diseases and cardiovascular mortality (Carney, Blumenthal, Stein, Watkins, Catellier, et al., 2001; Carney & Freedland, 2003; Stein, Carney, Freedland, Skala, Jaffe, Kleiger, & Rottman, 2000). Our finding of reduced high-frequency coupling sleep in depressed patients may indicate a long-term risk factor for adverse cardiovascular events. Limitations The present study has a number of limitations. First, polysomnography was not performed; thus, the exact correlations with conventional sleep indices cannot be made in this population. Second, the detection of ECG-based stable and unstable sleep states reflects only an approximation of EEG non-CAP and CAP states, respectively (Thomas et al., 2005). Third, the role of REM sleep is not assessed in this study, and REM sleep may be attenuated in medicated depressed patients. The difficulty with differentiation of REM sleep from the wakeful state could possibly be addressed if muscle tone recordings are integrated into the algorithm. The addition of actigraphy, which is a simple and widely accepted tool used to assess sleep/wakeful states, may complement the CPC method and strengthen its value by better defining the sleep period. While actigraphy has better ability to estimate total sleep time and circadian sleep distribution, the CPC analysis is better at characterizing the quality of sleep. Conclusions In conclusion, the present study suggests that depressed individuals have disrupted sleep stability and continuity, as quantified by ECG-based CPC analysis. Despite the lack of comparative data from standard polysomnography, our study nevertheless provides a unique viewpoint on sleep stability in the context of cardiovascular physiology. This readily repeatable ECG-based method could provide a simple and objective way to evaluate insomnia in depression, and possibly track treatment effects.

REFERENCES Achermann, P., & Borbely, A. A. (1987). Dynamics of EEG slow wave activity during physiological sleep and after administration of benzodiazepine hypnotics. Human Neurobiology, 6, 203–210.

Adrien, J. (2002). Neurobiological bases for the relation between sleep and depression. Sleep Medicine Review, 6, 341–351. Armitage, R. (1995). Microarchitectural findings in sleep EEG in depression: Diagnostic implications. Biological Psychiatry, 37, 72–84.

290 Armitage, R., Trivedi, M., Hoffmann, R., & Rush, A. J. (1997). Relationship between objective and subjective sleep measures in depressed patients and healthy controls. Depression and Anxiety, 5, 97–102. Bae, J. N., MacFall, J. R., Krishnan, K. R., Payne, M. E., Steffens, D. C., & Taylor, W. D. (2006). Dorsolateral prefrontal cortex and anterior cingulate cortex white matter alterations in late-life depression. Biological Psychiatry, 60, 1356–1363. Beck, A. T., Ward, C. H., Mendelson, M., Mock, J., & Erbaugh, J. (1961). An inventory for measuring depression. Archives of General Psychiatry, 4, 561–571. Brosschot, J. F., Van Dijk, E., & Thayer, J. F. (2007). Daily worry is related to low heart rate variability during waking and the subsequent nocturnal sleep period. International Journal of Psychophysiology, 63, 39–47. Buysse, D. J., Ancoli-Israel, S., Edinger, J. D., Lichstein, K. L., & Morin, C. M. (2006). Recommendations for a standard research assessment of insomnia. Sleep, 29, 1155–1173. Buysse, D. J., Reynolds, C. F. III, Monk, T. H., Berman, S. R., & Kupfer, D. J. (1989). The Pittsburgh Sleep Quality Index: A new instrument for psychiatric practice and research. Psychiatry Research, 28, 193–213. Carney, R. M., Blumenthal, J. A., Stein, P. K., Watkins, L., Catellier, D., Berkman, L. F., et al. (2001). Depression, heart rate variability, and acute myocardial infarction. Circulation, 104, 2024–2028. Carney, R. M., & Freedland, K. E. (2003). Depression, mortality, and medical morbidity in patients with coronary heart disease. Biological Psychiatry, 54, 241–247. Drevets, W. C., Price, J. L., & Furey, M. L. (2008). Brain structural and functional abnormalities in mood disorders: Implications for neurocircuitry models of depression. Brain Structure & Function, 213, 93–118. Dumont, M., Jurysta, F., Lanquart, J. P., Migeotte, P. F., van de Borne, P., & Linkowski, P. (2004). Interdependency between heart rate variability and sleep EEG: Linear/non-linear? Clinical Neurophysiology, 115, 2031–2040. Farina, B., Della Marca, G., Grochocinski, V. J., Mazza, M., Buysse, D. J., Di Giannantonio, M., et al. (2003). Microstructure of sleep in depressed patients according to the cyclic alternating pattern. Journal of Affective Disorders, 77, 227–235. Ferre, A., Guilleminault, C., & Lopes, M. (2006). Cyclic alternating pattern as a sign of brain instability during sleep. Neurologia, 21, 304–311. Germain, A., Nofzinger, E. A., Kupfer, D. J., & Buysse, D. J. (2004). Neurobiology of non-REM sleep in depression: Further evidence for hypofrontality and thalamic dysregulation. The American Journal of Psychiatry, 161, 1856–1863. Hamilton, M. (1960). A rating scale for depression. Journal of Neurology Neurosurgery Psychiatry, 23, 56–62. Iber, C. (2007). AASM manual for the scoring of sleep and associated events: Rules, terminology and technical specification. Westchester, IL: American Academy of Sleep Medicine. Irwin, M. R., Valladares, E. M., Motivala, S., Thayer, J. F., & Ehlers, C. L. (2006). Association between nocturnal vagal tone and sleep depth, sleep quality, and fatigue in alcohol dependence. Psychosomatic Medicine, 68, 159–166. Jindal, R. D., Thase, M. E., Fasiczka, A. L., Friedman, E. S., Buysse, D. J., Frank, E., & Kupfer, D. J. (2002). Electroencephalographic sleep profiles in single-episode and recurrent unipolar forms of major depression: II. Comparison during remission. Biological Psychiatry, 51, 230–236. Johns, M. W. (1991). A new method for measuring daytime sleepiness: The Epworth sleepiness scale. Sleep, 14, 540–545. Kuo, T. B., Shaw, F. Z., Lai, C. J., & Yang, C. C. (2008). Asymmetry in sympathetic and vagal activities during sleep-wake transitions. Sleep, 31, 311–320. Liotti, M., Brannan, S., Egan, G., Shade, R., Madden, L., Abplanalp, B., et al. (2001). Brain responses associated with consciousness of breathlessness (air hunger). Proceedings of the National Academy of Sciences of the United States of America, 98, 2035–2040. Lopes, M. C., Quera-Salva, M. A., & Guilleminault, C. (2007). Non-REM sleep instability in patients with major depressive disorder: Subjective improvement and improvement of non-REM sleep instability with treatment (Agomelatine). Sleep Medicine, 9, 33–41.

A. C. Yang et al. Moody, G. B., Mark, R. G., Bump, M. A., Berman, A. D., Mietus, J. E., & Goldberger, A. L. (1986). Clinical validation of the ECG-derived respiration (EDR) technique. Computing in Cardiology, 13, 507–510. Nahshoni, E., Aravot, D., Aizenberg, D., Sigler, M., Zalsman, G., Strasberg, B., et al. (2004). Heart rate variability in patients with major depression. Psychosomatics, 45, 129–134. Ozone, M., Yagi, T., Itoh, H., Tamura, Y., Inoue, Y., Uchimura, N., et al. (2008). Effects of zolpidem on cyclic alternating pattern, an objective marker of sleep instability, in Japanese patients with psychophysiological insomnia: A randomized crossover comparative study with placebo. Pharmacopsychiatry, 41, 106–114. Parrino, L., Boselli, M., Spaggiari, M. C., Smerieri, A., & Terzano, M. G. (1997). Multidrug comparison (lorazepam, triazolam, zolpidem, and zopiclone) in situational insomnia: Polysomnographic analysis by means of the cyclic alternating pattern. Clinical Neuropharmacology, 20, 253–263. Saletu, B. (1975). Is the subjectively experienced quality of sleep related to objective sleep parameters? Behavioral Biology, 13, 433–444. Sateia, M. J., & Nowell, P. D. (2004). Insomnia. Lancet, 364, 1959–1973. Stein, P. K., Carney, R. M., Freedland, K. E., Skala, J. A., Jaffe, A. S., Kleiger, R. E., & Rottman, J. N. (2000). Severe depression is associated with markedly reduced heart rate variability in patients with stable coronary heart disease. Journal of Psychosomatic Research, 48, 493–500. Takahara, M., Mizuno, K., Hirose, K., Sakai, K., Nishii, K., Onozuka, M., et al. (2008). Continuous recording of autonomic nervous activity at nighttime effectively explains subjective sleep reports in postmenopausal women. Sleep and Biological Rhythms, 6, 215–221. Terzano, M. G., Mancia, D., Salati, M. R., Costani, G., Decembrino, A., & Parrino, L. (1985). The cyclic alternating pattern as a physiologic component of normal NREM sleep. Sleep, 8, 137–145. Terzano, M. G., & Parrino, L. (2000). Origin and significance of the cyclic alternating pattern (CAP). Sleep Medicine Review, 4, 101–123. Terzano, M. G., Parrino, L., Boselli, M., Dell’Orso, S., Moroni, M., & Spaggiari, M. C. (1995). Changes of cyclic alternating pattern (CAP) parameters in situational insomnia under brotizolam and triazolam. Psychopharmacology (Berl), 120, 237–243. Terzano, M. G., Parrino, L., Spaggiari, M. C., Palomba, V., Rossi, M., & Smerieri, A. (2003). CAP variables and arousals as sleep electroencephalogram markers for primary insomnia. Clinical Neurophysiology, 114, 1715–1723. Thase, M. E., Fasiczka, A. L., Berman, S. R., Simons, A. D., & Reynolds, C. F. III. (1998). Electroencephalographic sleep profiles before and after cognitive behavior therapy of depression. Archives of General Psychiatry, 55, 138–144. Thomas, R. J. (2007). Effective sleep homeostasisFOscillations during sleep, and the function of sleep in health and disease. Cellscience Review, 3, 49–62. Thomas, R. J., Mietus, J. E., Peng, C. K., Gilmartin, G., Daly, R. W., Goldberger, A. L., & Gottlieb, D. J. (2007). Differentiating obstructive from central and complex sleep apnea using an automated electrocardiogram-based method. Sleep, 30, 1756–1769. Thomas, R. J., Mietus, J. E., Peng, C. K., & Goldberger, A. L. (2005). An electrocardiogram-based technique to assess cardiopulmonary coupling during sleep. Sleep, 28, 1151–1161. Tworoger, S. S., Davis, S., Vitiello, M. V., Lentz, M. J., & McTiernan, A. (2005). Factors associated with objective (actigraphic) and subjective sleep quality in young adult women. Journal of Psychosomatic Research, 59, 11–19. Udupa, K., Sathyaprabha, T. N., Thirthalli, J., Kishore, K. R., Lavekar, G. S., Raju, T. R., & Gangadhar, B. N. (2007). Alteration of cardiac autonomic functions in patients with major depression: A study using heart rate variability measures. Journal of Affective Disorders, 100, 137–141. van Eijndhoven, P., van Wingen, G., van Oijen, K., Rijpkema, M., Goraj, B., Jan Verkes, R., et al. (2009). Amygdala volume marks the acute state in the early course of depression. Biological Psychiatry, 65, 812–818. von Leupoldt, A., Sommer, T., Kegat, S., Baumann, H. J., Klose, H., Dahme, B., & Buchel, C. (2008). The unpleasantness of perceived dyspnea is processed in the anterior insula and amygdala. American Journal of Respiratory and Critical Care Medicine, 177, 1026–1032. Yeh, G. Y., Mietus, J. E., Peng, C. K., Phillips, R. S., Davis, R. B., Wayne, P. M., et al. (2008). Enhancement of sleep stability with Tai


291

Chi exercise in chronic heart failure: Preliminary findings using an ECG-based spectrogram method. Sleep Medicine, 9, 527–536. Yeragani, V., Appaya, S., Seema, K., Kumar, R., & Tancer, M. (2005). QRS amplitude of ECG in normal humans: Effects of orthostatic challenge on linear and nonlinear measures of beat-to-beat variability. Cardiovascular Engineering, 5, 135–140.

APPENDIX Table A1. Correlation Between CPC-Derived Sleep Indices and Spectral Heart Rate Variability Components Relationship HFC vs. VLF HFC vs. LF HFC vs. HF HFC vs. LF/HF LFC vs. VLF LFC vs. LF LFC vs. HF LFC vs. LF/HF VLFC vs. VLF VLFC vs. LF VLFC vs. HF VLFC vs. LF/HF

r (N 5 191)

r2

p value

! 0.079 ! 0.015 0.207 ! 0.376 0.076 0.021 ! 0.202 0.357 0.021 ! 0.034 ! 0.121 0.169

0.006 o0.001 0.043 0.141 0.006 o0.001 0.041 0.127 o0.001 0.001 0.015 0.029

.276 .834 .004 o.001 .293 .776 .005 o.001 .771 .640 .094 .020

Note: r: Pearson’s correlation coefficient; HFC: high-frequency coupling; LFC: low-frequency coupling; VLFC: very-low frequency coupling; VLF: very-low frequency component of heart rate variability (0.003–0.04 Hz); LF: low-frequency component of heart rate variability (0.04–0.15 Hz); HF: high-frequency component of heart rate variability (0.15–0.4 Hz); LF/HF: low-frequency to high-frequency ratio. The spectral heart rate variability indices were log transformed.

Yeragani, V. K., Rao, K. A., Smitha, M. R., Pohl, R. B., Balon, R., & Srinivasan, K. (2002). Diminished chaos of heart rate time series in patients with major depression. Biological Psychiatry, 51, 733–744. (Received August 7, 2009; Accepted March 29, 2010)

Psychophysiology 48(2)

Handbook of Psychophysiology

Psychophysiology 47(6)

Psychophysiology 48(1)

Psychophysiology 48(3)

Bilim ve Teknik - Ocak 2008 (sayı 482)

Developmental Psychophysiology: Theory, Systems, and Methods

Societal Challenges and Geoinformatics (GSA Special Paper 482)

Trade and cost competitiveness in the Czech Republic, Hungary, Poland, and Slovenia, Volumes 23-482

Perception-based data processing in acoustics: applications to music information retrieval and psychophysiology of hearing

Methods in Enzymology 482 - CRYO-EM, PART B: 3-D RECONSTRUCTION

Case Studies in Applied Psychophysiology: Neurofeedback and Biofeedback Treatments for Advances in Human Performance

Perception-Based Data Processing in Acoustics: Applications to Music Information Retrieval and Psychophysiology of Hearing (Studies in Computational Intelligence)

Psychophysiology 48(2)

Handbook of Psychophysiology

Psychophysiology 47(6)

Psychophysiology 48(1)

Psychophysiology 48(3)

Bilim ve Teknik - Ocak 2008 (sayı 482)

Developmental Psychophysiology: Theory, Systems, and Methods

Societal Challenges and Geoinformatics (GSA Special Paper 482)

Trade and cost competitiveness in the Czech Republic, Hungary, Poland, and Slovenia, Volumes 23-482

Perception-based data processing in acoustics: applications to music information retrieval and psychophysiology of hearing

Methods in Enzymology 482 - CRYO-EM, PART B: 3-D RECONSTRUCTION

Case Studies in Applied Psychophysiology: Neurofeedback and Biofeedback Treatments for Advances in Human Performance

Perception-Based Data Processing in Acoustics: Applications to Music Information Retrieval and Psychophysiology of Hearing (Studies in Computational Intelligence)

Recommend Documents