Stimulated Recall Methodology in Second Language Research

cover

title: author: publisher: isbn10 | asin: print isbn13: ebook isbn13: language: subject publication date: lcc: ddc: subject:

next page >

Stimulated Recall Methodology in Second Language Research Second Language Acquisition Research Gass, Susan M.; Mackey, Alison. Lawrence Erlbaum Associates, Inc. 0805832238 9780805832235 9780585331850 English Second language acquisition--Research, Recollection (Psychology) 2000 P118.2.G376 2000eb 401/.93 Second language acquisition--Research, Recollection (Psychology)

cover

next page >

< previous page

page_i

next page > Page i

Stimulated Recall Methodology in Second Language Research

< previous page

page_i

next page >

< previous page

page_ii

next page > Page ii

Second Language Acquisition Research Theoretical and Methodological Issues Susan Gass and Jacquelyn Schachter, Editors Tarone/Gass/Cohen Research Methodololgy in Second-Language Acquisition Schachter/Gass Second Language Classroom Research: Issues and Opportunities Birdsong Second Language Acquisition and the Critical Period Hypothesis Monographs on Research Methodology Gass/Mackey Stimulated Recall Methodology in Second Language Research Yule Referential Communication Tasks Markee Conversation Analysis

< previous page

page_ii

next page >

< previous page

page_iii

next page > Page iii

Stimulated Recall Methodology in Second Language Research Susan M. Gass Michigan State University Alison Mackey Georgetown University

< previous page

page_iii

next page >

< previous page

page_iv

next page > Page iv

Copyright © 2000 by Lawrence Erlbaum Associates, Inc. All rights reserved. No part of the book may be reproduced in any form, by photostat, microform, retrieval system, or any other means, without prior written permission of the publisher. Lawrence Edbaum Associates, Inc., Publishers 10 Industrial Avenue Mahwah, NJ 07430 Cover design by Kathryn Houghtaling Lacey Library of Congress Cataloging-in-Publication Data Gass, Susan M. Stimulated recall methodology in second language research / Susan M. Gass, Alison Mackey. p. cm. -- (second language acquisition reserch) Includes bibliographical references and index. ISBN 0-8058-3223-8 (c: alk. paper). ISBN 0-8058-3224-6 (pbk. : alk. paper). 1. Second language acquisition-Research. 2. Recollection (Psychology) I. Mackey, Alison. II. Title. III. Series. PG118.2.C376 2000 401'.93dc21 99-36248 CIP Books published by Lawrence Erlbaum Associates are printed on acid-free paper, and their bindings are chosen for strength and durability. The final camera copy for this work was prepared by the author, and therefore the publisher takes no responsibility for consistency or correctness of typographical style. However, this arrangement helps to make publication of this kind of scholarship possible. Printed in the United States of America 10 9 8 7 6 5 4 3 2 1

< previous page

page_iv

next page >

< previous page

next page >

page_v

Page v

Contents Foreword

ix

Preface

xi

Acknowledgments 1 Introduction to Introspective Methods

xiii 1 1

Background 3 Introspective Methods 3 The Usefulness of Reflections on Mental Processes 6 Introspection and Behaviorism 9 Behaviorism and the Study of Language 11 Verbal Reporting 17 Stimulated Recall 20 What Topics Can Be Explored Using Stimulated Recall Methodology? 21 Knowledge Types 22 Knowledge Structures 23 Cognitive Processes and Learner Strategies 24 Conclusion 2 Introspection and Second Language Research

25 26

Classification Scheme for Introspective Research 28 Range of Second Language Studies 35 Conclusion 3 Characterization of Stimulated Recall

37 37

Review of Stimulated Recall Procedure 38 Oral Interaction Research

38 Perceptions of Comprehension 38 Experimental Research 39 Qualitative Research 40 Communication Problems 42 Acceptability Judgments

< previous page

page_v

next page >

< previous page

next page >

page_vi

Page vi 43 Reading/Vocabulary 44 Writing Research 44 Tutorial Sessions 44 Writer's Block 46 Composing Process 46 Pragmatics 47 Classification of Recall Support 48 Færch and Kasper (1987) 48 Relationship to Specific Action 49 Temporal Relationship to Action 50 Participant Training 52 Procedural Structure 53 Stimulus for Recall 53 Initiation of Questions/Recall Interaction 4 Using Stimulated Recall Methodology

57 57

Data Collection: Instructions for Participants and Researchers 63 Data Analysis 63 Interrater Reliability 69 An Extended Example of Rater Training 77 Data Layout and Coding 84 Procedural Pitfalls 84 Timing 85

The Approximate Length of the Recall Support 87 Allocating Time for the Recall Procedure 88 Allocating Time for Set Up and Equipment 88 Verbalization 89 Recall Questions 97 Language of the Recall Session 99 Analysis of Stimulated Recall Data 101 Sampling the Recall Data 102 Preparing the Data for Coding 103 Developing a Coding Scheme 103 Analyzing and Describing the Data 104 Conclusion

< previous page

page_vi

next page >

< previous page

next page >

page_vii

Page vii 5 Limitations and Additional Uses

105 105

Issues of Validity and Reliability 112 Summary 113 Possible Uses for Stimulated Recall 114 Interlanguage Phonology 114 Comprehension of Learner Speech 116 Change over Time 117 Classroom Interaction 118 Oral Production 120 Interlanguage Pragmatics 121 Comprehension 123 Input and Input Processing 123 Reading 124 Amount and Type of Exposure 125 L2 Reading Comprehension 126 Oral Interaction 126 Dialogue 127 Negotiation 128 Syntactic Processing 129 Vocabulary 129 Incidental Vocabulary Learning 131 Acquisition of Words in an Unknown Language 132

Conclusion References

137

Additional References

149

Appendix A

153

Appendix B

157

Appendix C

161

Author Index

165

Subject Index

169

< previous page

page_vii

next page >

< previous page

page_ii

next page >

< previous page

page_ix

next page > Page ix

Foreword This series is born of our belief that to adequately understand conclusions drawn from second language acquisition research, one must understand the methodology that is used to elicit data for that research. The concern with research methodology is common in all fields, but it takes on particular significance in second language research given the interdisciplinary nature of the field and the varying perspectives of second language researchers. Within a single common field, we have psychologists talking to linguists; we have sociolinguists concerned with variation talking to sociolinguists concerned with pragmatics; we have ethnographers talking to generative grammarians. Although all this is healthy for the long-term outcome of the field, it is problematic in that researchers bring with them research traditions from their own disciplines. Again, this is ultimately healthy but can lead to serious misunderstandings, in the short term, of the value of a particular elicitation instrument. As a result, research traditions are attacked through their methodologies without a full understanding of what linguistic or psycholinguistic or sociolinguistic knowledge or abilities a particular instrument is intended to tap. This series of monographs is an attempt to bring these issues to light. The series consists of monographs devoted to particular data-collection methods or instruments. Each monograph probes a specific research method or tool, discussing the history of the instrument as well as its current uses. A major feature of each monograph is an exploration of what the research instrument does and does not purport to tell us about second language acquisition or use. Each monograph addresses the kinds of research questions for which the method or instrument is best suited, its underlying assumptions, a characterization of the method or instrument, and an extended description of its use, including problems associated with its use. It is hoped that the series as a whole will reflect the state of the research in second language acquisition. It is only through a deeper understanding of the strengths and weaknesses, and the advantages and disadvantages of particular research tools, that the field of second language acquisition can progress beyond issues of methodology and begin to work together as a collective whole. SUSAN GASS JACQUELYN SCHACHTER SERIES EDITORS

< previous page

page_ix

next page >

< previous page

page_ii

next page >

< previous page

page_xi

next page > Page xi

Preface Introspective methods have been a common source of data elicitation in second (L2) and foreign language research. One subset of 'introspective methodology is known as verbal reporting, both online (e.g., talk-aloud or think-aloud) and retrospective. Restrospective reports are generally known as stimulated recall or postprocess oral observation. Our concern in this book is with these retrospective reports, which we will refer to as stimulated recall. What is common to introspective methods is that the data come from participants' own statements about the way they organize and understand information. The methods are frequently used as supplements for data gathered based on learners' behavior (i.e., their linguistic and/or nonlinguistic output). Stimulated recall methodology is used to explore learners' thought processes (or strategies) at the time of an activity or task. This is achieved by asking learners to report those thoughts after they have completed a task or activity. Stimulated recall is carried out with some degree of support for the recall. Examples of support include showing a videotape to learners so that they can watch themselves carrying out the original task, or giving learners their L2 written product, so that they can follow the changes they made. Verbal reports, in general, including stimulated recall, are increasingly being used in L2 research, as they have been in fields such as cognitive psychology and educational research where many of these introspective methodologies originated. One thing that is clear from the proliferation of studies that use stimulated recall, and the critiques of verbal report methodology, in general, is that stimulated recall is a methodology that, like many others, must be used with care. Limitations of the data must be recognized, and studies that utilize stimulated recall methodology require carefully structured research designs to avoid pitfalls. This text on stimulated recall is designed so that those conducting research on L2 learning who decide to use the methodology can utilize the book as a comprehensive guide for how to go about conducting stimulated recalls. The text is designed as a jumping-off point for an exploration of the many publications that debate and problematize the various introspective processes that this interesting methodology may uncover. SUSAN GASS ALISON MACKEY

< previous page

page_xi

next page >

< previous page

page_ii

next page >

< previous page

page_xiii

next page > Page xiii

Acknowledgments There are many people to whom we owe a debt of gratitude. First, Judith Amsel of Lawrence Erlbaum Associates assisted us through all phases of the preparation of this manuscript. We are grateful to her for encouragement, support and advice. Second, Jennifer Leeman, Kim McDonough, Ildikê Svetics, and Ian Thornton all helped in important ways in the preparation of this book. In particular, Kim McDonough and Ildikê Svetics tracked down numerous hard to find references. Their sleuthing techniques are to be commended. Third, our spouses were helpful with this manuscript beyond the call of duty. Thanks Josh and David. Josh, in particular, deserves special acknowledgment for his helpful cooking tips to Alisonkeep soaking and lightly roasting those nuts before putting them into the haroseth!! Finally, we are grateful to each other for the many episodes of tearsdown-the-cheeks laughter, especially those in Tokyo where the idea for this book originated over a chicken dinner. Working together was a process we found thoroughly enjoyable. We gratefully acknowledge the following publishers for permission to reprint from their material. 1) Academic Press for permission to reprint original data from The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes by van Someren, Barnard, and Sandberg (1994) and 2) Multilingual Matters Ltd. for permission to reprint a figure from 'Verbal Reports on Thinking' by Ericsson and Simon in Introspection in Second Language research edited by C. Faerch and G. Kasper (1987). Finally, this work was truly a collaborative venture, made possible, in part, by the wonders of e-mail. The order of names reflects alphabetical order only and in no way suggests a greater or lesser contribution by either of the authors.

< previous page

page_xiii

next page >

< previous page

page_1

next page > Page 1

1 Introduction to Introspective Methods Stimulated recall is one subset of a range of introspective methods that represent a means of eliciting data about thought processes involved in carrying out a task or activity. The assumption underlying introspection is that it is possible to observe internal processes in much the same way as one can observe external real-world events. Another assumption is that humans have access to their internal thought processes at some level and can verbalize those processes. This book treats these issues by first providing some background information, including a history of introspection and its place within the fields of philosophy, psychology, and linguistics (chap. 1). The main focus of the book, however, is the specific use of stimulated recall within second language (L2) studies. We detail the ways that L2 researchers have used stimulated recall (chap. 2) and, importantly, we provide information on the do's and do not's of conducting stimulated recall (chaps. 3 and 4). In order to provide readers of this book with a balanced view of the information that stimulated recall can provide to researchers, we present the limitations of stimulated recall (chap. 5). We also describe studies where a follow-up recall procedure might have provided researchers with desirable information, information that was important but not discernible from the original data alone (chap. 5). Background The goal of second language acquisition (SLA) research is to determine: (1) the knowledge that learners have of the second language (L2), and (2) the source of that knowledge. A major source of information about L2 knowledge is production data, that is, utterances produced by

< previous page

page_1

next page >

< previous page

page_2

next page > Page 2

learners. In fact, in the early years of the systematic study of SLA, Selinker (1972: 213214) stated that researchers should ''focus . . . analytical attention upon the only observable data to which we can relate theoretical predictions: the utterances which are produced when the learner attempts to say sentences of a TL." While this view is still maintained by some, it has never been entirely accepted. Corder (1973), for example, argued that forced elicitation data were necessary. In other words, spontaneously produced utterances provide only a part of the picture. If one wants to obtain information about the grammatical knowledge that learners have, then one also must have a means to determine which sentences learners think are possible in a second language (i.e., grammatical) and which are not possible in a second language (i.e., ungrammatical). To accomplish this, data collection from a source other than language production may be necessary (see additional discussion in Gass, 1997). In addition to determining actual knowledge of the second language, one also needs to understand how that knowledge comes about. For this, there are various methodological tools that one can use. The important point is that most processes involved in learning are not directly observable. All that is observable is what a learner produces, in writing or in speech. Various methods have been used in the field of second language research to determine underlying linguistic knowledge, including asking learners to introspect about their knowledge. The focus of this book, stimulated recall, is one such method, generally classified under the broader cover term of introspection. Like most methodological tools, introspection has had a long history and has fallen into disfavor along its path. However, it is now being used once again with some frequency and with increased confidence. In this chapter, we contextualize stimulated recall through an examination of the broader area of introspection, with a focus on verbal reporting and, in particular, on the place of stimulated recall in introspective methods. We begin the examination by briefly considering some of the historical context of introspection. We then present a broad historical and philosophical context, in order to document the rich background work on stimulated recall as an introspective method.

< previous page

page_2

next page >

< previous page

page_3

next page > Page 3

Introspective Methods There is a long history of use of reflections on mental processes, originating in the fields of philosophy and psychology. Lyons (1986) traced this history to Augustine and possibly to Aristotle in western thought. Such mentalistic reflections are often classified as methods of introspection. As mentioned previously, such methods have been in and out of favor in scholarly circles. Introspection assumes that a person can observe what takes place in consciousness in much the same way as one can observe events in the external world. Lyons cited the definition of introspection in the Concie Oxford English Dictionary as "the examination or observation of one's mental processes" (p. 1). This broad definition subsumes a number of different approaches, and, as a definition, has proven to be too general for many scholars, especially those at pains to distance themselves from introspection.1 Our present-day nuances of the term introspection are based on the epitome of introspection, what Lyons (1986) called the "golden age of introspection" (p. 2), covering the time frame from the 17th century to the early part of the 20th century. A seminal figure in introspection is Descartes. A basic premise of Descartes' work (e.g., 1637) was the notion of mind as a separate entity: A person's mind is fundamentally separate from a person's body; a person's mind is also fundamentally separate from the minds of others.2 With this background, we turn now to a discussion of the potential usefulness of reflection. The Usefulness of Reflections on Mental Processes In this section we consider how the use of reflections on mental processes has been conceived. A fundamental question is whether certain types of reflection might be more useful than others. One component 1Introspection has met much the same fate as the term liberal in contemporary American politics. Conservatives classify virtually all their opponents under the broad term of liberalism. The opponents largely adopt a different term for themselves. Those further to the left might call themselves radicals (popular in the 1960s). Those closer to the middle might choose progressive or moderate. Few proudly call themselves liberal. Few scholars in the middle to late 20th century call their method introspection. 2 In today's academic climate, neither distinction is accepted uncritically. For example, there is no evidence in neurology for a mental organ that cannot be affected by somatic factors. Alzheimer's disease and vascular dementia have physical causes.

< previous page

page_3

next page >

< previous page

page_4

next page > Page 4

of usefulness of any investigative method is how likely that method is to produce 'true' results. The issue of truth is a complex one, and it is beyond the scope of this book to examine it in detail. Nonetheless, two tests can be attempted: falsifiability and replicability. If a statement cannot be falsified at all, then some would consider it outside the realm of science, that is, not within the true, at least in Canguilhem's terminology (1989, 1994). Hence, we would want any self-description about mental processes to be falsifiable in practice. Such a result is not always easy to achieve. Consider Wittgenstein's (1958) discussions of how certain we can be about another person's expressions of pain. Wittgenstein likened meaning to a rope made up of many strands twisted together and argued that we can never really know the exact meaning that another person intends to convey. Philosophy apart, presumably, some self-descriptions of mental processes might be more falsifiable than others. Replicability presents similar problems. If one person reports some mental process, how could these processes be replicated at another time in the same person or even in another person? We turn to a consideration of different ways of classifying mental processes. One relevant taxonomy is temporally based. Classical introspection is atemporal. In deciding upon his famous phrase, "Cogito ergo sum," Descartes was not reflecting on his mental functioning at the moment of thinking, or of any particular time, but of his mental functioning in general. We might well question whether it is possible to think about mental processes in general without recalling some experience in particular. Thus, we may question whether atemporal introspection is really a form of retrospection (we elaborate in chap. 3 on different types of introspection and retrospection). It is difficult enough to gain awareness of mental functioning; trying to recall past instances might prove particularly troublesome. Running commentary and stimulated recall, we argue, may be more reliable (e.g., chap. 4).3 3 Consider an analogical argument about evidence in the American legal system. Generally, hearsay evidence (i.e., reports of what was said at some earlier time) is considered unreliable, and rules of evidence largely make hearsay testimony inadmissible in court. There are various exceptions, however, for which hearsay evidence is admissible. The typical rationalization for these exceptions is that hearsay in those particular circumstances is unusually reliable. One exception, for example, is an excited utterance. If someone yells out, "That red car just ran the red light and hit that truck" in an excited manner, then she is quite likely describing what she really thought she saw. The fact that a person was excited when she made her comments is supposed to render testimony reliable because, in some sense, she did not have the opportunity to work (footnote continued on next page)

< previous page

page_4

next page >

< previous page

page_5

next page > Page 5

Another distinction that can be made involves the particulars of the mental processes that we attempt to describe. Perhaps the most accepted descriptions are of judgments, such as acceptability judgments common in linguistic research. These have been aptly described by Habermas (1979) as a reconstructive approach. Native speakers of a language, at least in clear cases, know that something is or is not acceptable in that language. Wilhelm Wundt (1894), one of the founders of experimental psychology, studied language in detail and practiced introspection, but he did not use introspective methods in studying the psychology of language. To the contrary, he felt that introspective methods would not prove successful in investigating language because, in his view, language was a social phenomenon. Clearly, Wundt's view and practice of introspection as a method of self-observation, and the phenomena he described, differ from the verbalizations of thought processes, which are in popular use today and which form the main topic of this book. An interesting model for mental phenomena is Dennett's (1987) analogy of mental processing and magic tricks. According to Dennett, the awareness we have of mental processing is much like the awareness we have of a magic trick. We observe a magic trick and can see what must be explained, but simply observing the trick does not typically lead to an understanding for most people about how the magician pulled off the trick. The trick provides the explanandum, rather than the explicans. As individuals attempting to report our mental processes, we can only report what we are conscious of. We have no access to what is really occurring at any other level.4 One difficulty with the procedure of introspection is that humans are essentially sense-making beings and tend to create explanations, whether such explanations can be justified or not. This is the central thesis of Dennett's (1987) book, The Intentional Stance. He argued that we tend to understand activities as if they are the product of some meaning-producing entity. Recent experiments on split-brain patients (footnote continued from previous page) out a script, to practice or control her statements. She was overcome by the immediacy or emotion of the event. Another exception, also relevant for us, is a present sense impression. Play-byplay commentary, as in a sporting event, is much more likely to be reliable, in the eyes of legal authority, than narrating the event afterward. If this is true in general, it is probably even more true for difficult-to-describe experiences such as mental functioning. 4 People may be trained to be more observant about their reflective processes. We discuss this in more detail in chapter 3.

< previous page

page_5

next page >

< previous page

page_6

next page > Page 6

(i.e., individuals with a severed corpus callosumfor these individuals, the most efficient communication pathway between the two cerebral hemispheres is no longer functional) indicate that the left hemispheres of our brains, are excellent at producing meaningful explanations, even if in error (Gazzaniga, 1998). For split-brain patients, it is possible to show pictures to both hemispheres simultaneously. For most individuals, only the left hemisphere is able to create a narration. In one experiment, patients were asked to point to the picture from a set that best corresponded to what they saw. The right hand is controlled by the left hemisphere and vice versa. Both hands of all the patients were equally adept at the picture-matching test. However, only the left hemisphere (which controls the right hand) was able to construct a coherent narrative of why a particular picture was appropriate. If the individual were asked why the object pointed to by the left hand was chosen, the left hemisphere did not know and the right hemisphere could not say. Gazzaniga found that the left hemisphere constructed a plausible story relating what the left hand pointed to and what the left hemisphere saw. This experiment demonstrates that human beings tend to create explanations for phenomena, even when these explanations may not be warranted. This finding is important when considering introspective methods because clearly there is a danger that individuals may create plausible stories for other descriptions of mental activity, without really knowing what is going on. Introspection and Behaviorism Introspection, as a methodological tool, fell into disfavor with the rise of behaviorism. The main goal of behaviorism was to gain information about human behavior, not through looking inwards, but by observing, measuring, and interpreting human behavior. Watson (1913, as cited in Lyons, 1986) stated: psychology as the behaviorist views it is a purely objective experimental branch of natural science. Its theoretical goal is the prediction and control of behavior. Introspection forms no essential part of its methods, nor is the scientific value of its data dependent upon

< previous page

page_6

next page >

< previous page

page_7

next page > Page 7

the readiness with which they lend themselves to interpretation in terms of consciousness. (p. 23) These comments and other similar ones were a reaction to some of the early work in the field of psychology (such as that by Titchener, 1908, and Wundt, 1894, as discussed by Blumenthal, 1970) that relied heavily on techniques of introspection as a way of gaining insight into the human mind. As Lieberman (1979) pointed out, however, the techniques of introspection used by the early experimentalists were quite different from introspection techniques of today. In earlier years of introspection as a methodology, participants went through significant training before carrying out the introspection process. Lieberman noted that in Wundt's laboratory participants had to "practice at least 10,000 separate introspections" (p. 320) before being considered qualified to participate in an actual introspection. This is clearly problematic in that the mere fact of practice may alter thought processes (see chap. 5), especially practice in such incredible amounts. Patterns of behavior and preconceptions would be established, making it unclear what participants were accessing. The notion of consciousness has always been a concept subject to debate in the field of psychology, and introspection was once seen as a way of access to consciousness. Introspective analysis assumed that the functionings of the mind were in fact accessible to observation. However, Freud's popular perspective clearly made it theoretically impractical to view the human mind in this way. During the years when theories of behaviorism were in ascendance in the field of psychology (from about the turn of the century), consciousness was not a favored or even a valid area of research. The tools used to investigate it were also summarily dismissed. The debate became even sharper in 1920 when Watson (cited by Lyons, 1986) stated: It is a serious misunderstanding of the behavioristic position to say . . ."And of course a behaviorist does not deny that mental states exist. He merely prefers to ignore them." He "ignores" them in the same sense that chemistry ignores alchemy, astronomy horoscopy, and psychology telepathy and psychic manifestations. The behaviorist does not concern himself with them be-

< previous page

page_7

next page >

< previous page

page_8

next page > Page 8

cause as the stream of his science broadens and deepens such older concepts are sucked under, never to reappear. (p. 24) In other words, consciousness was not a serious enterprise and had as much validity as some of the "quack" sciences. Not only were there theoretical difficulties with the concept of introspection as anathema to behaviorism, but also much of the early work using introspection proceeded uncritically and uncontroversially (as noted by Ericsson & Simon, 1993). With the lack of scientific rigor, it was not surprising that the results emanating from introspection were inconsistent. Hence, it was argued that the method itself was at fault and should be eliminated from the repertoire of methodologies used in psychology. This resulted in what can be termed 'throwing out the baby with the bathwater.'5 B. F. Skinner (1953, 1957), the influential psychologist and behaviorist, emphasized the importance of observable behaviors in psychology. As a science, psychology was conducted (as were natural sciences), by controlling variables to determine their effects on, in the case of psychology, behaviors. According to Lyons (1986), behaviorist explanations of internal (mental) events were in terms of covert (inner), reduced (to behavior) and unemitted forms of external behavior. By 1953, however, Skinner acknowledged that inner events could be observed by the possessor of them.6 This, in a sense, went some way to5 There is an interesting parallel from the SLA literature. In the early years of systematic SLA studies, much work was conducted within the framework of behaviorism (see description in Gass & Selinker, 1994). In particular, early work on transfer traditionally focused on behavioral aspects of transfer. With the demise of behaviorism, it was important within an SLA context to show that SLA was not a behavioristic activity. This entailed throwing off the shackles of language transfer. That is, because transfer was strongly associated with behaviorist thought, to show that L2 learning was not a behaviorist activity, researchers argued (Bailey, Madden, & Krashen, 1974; Dulay & Burt, 1974a, 1974b, 1975) that transfer was not a major or even an important factor for L2 learning. The link between behaviorism and its tools of analysis on one hand and its theoretical extensions on the other often went unchallenged with the demise of the theory. As a consequence, one saw a de-emphasis of all transfer research. It was only with work by Gass (1979), Kellerman (1979), Sjoholm (1976), and others, that researchers began to question an inextricable link between behaviorism and the use of the native language. Research during the past two decades on the role of the native language has taken a different view, consistent with a nonbehaviorist position, and has questioned the assumption that language transfer has to be part of behaviorism. 6 This view of Skinnerian psychology is somewhat simplistic, It is clear that Skinner had a much more complex theory in mind. He argued that inner events are limited in their accessibility and do not have any special structure. Hence, they are not the same as the observable events that he (footnote continued on next page)

< previous page

page_8

next page >

< previous page

page_9

next page > Page 9

ward legitimizing introspection as a research tool. Lyons (1986) pointed out that the failure of behaviorists to account for the "problem of privacy" (p. 44) led to "current centralist (brain-centered) psychologies and philosophies" (p. 44). He claims that ''What we gain access to . . . is a private and personal storehouse of myriad public performances, edited and 'replayed' according to largely stereotyped views about our cognitive life" (p. 148). In other words, the access to actual cognitive processes is not direct but is, as Lyons put it, replayed through memory. Behaviorism and the Study of Language Linguistic data in the earlier part of this century were collected by means of observation; linguistic generalizations were made by gathering speech samples from individuals and then analyzing those data in terms of the patterns that they represented. Bloomfield (1933), in a rewrite of his early 1914 work, spoke of mentalistic and mentalist psychology. The mentalists would supplement the facts of language by a version in terms of mind,a version which will differ in the various schools of mentalistic psychology. The mechanists demand that the facts be presented without any assumption of such auxiliary factors. (p. vii) With the advent of cognitive psychologies and their focus on internal events, such as processing (cf. Bruner, Goodnow, & Austin, 1956; Miller, Galanter, & Pribram, 1960; Newell & Simon, 1956) and with Chomsky's (1957, 1959) attack on Skinnerian behaviorism, a new cli(footnote continued from previous page) was interested in. In his 1953 book, Skinner clearly acknowledges the gray area between public (observable) and private (internal): The line between public and private is not fixed. The boundary shifts with every discovery of a technique for making private events public. Behavior which is of such small magnitude that it is not ordinarily observed may be amplified. Covert verbal behavior may be detected in slight movements of the speech apparatus. . . . The problem of privacy may, therefore, eventually be solved by technical advances. But we are still faced with events which occur at the private level and which are important to the organism without instrumental amplification. How the organism reacts to these events will remain an important question, even though the events may some day be made accessible to everyone. (p. 282)

< previous page

page_9

next page >

< previous page

page_10

next page > Page 10

mate that allowed for introspection arose. And, with regard to teaching, the new research paradigms opened new doors. For example, working in the field of general education, Shulman (1986) noted that: to understand adequately the choices teachers make in classrooms, the grounds for their decisions and judgments about pupils, and the cognitive processes through which they select and sequence the actions they have learned to take while teaching, we must study their thought processes before, during, and after teaching. (p. 23) Shulman referred to his earlier work with Elstein (Shulman & Elstein, 1975) in pointing out three main types of cognitive process research when dealing with teaching: judgment and policy, problem solving, and decision making. From such work in education, it is clear that there was an evolving requirement for some sort of verbal reporting, at least when exploring issues such as problem solving and decision making. Uncovering cognitive processes was clearly a complex issue, and the door was opened to a consideration of introspection. In the case of research in linguistics, the rejection of behaviorism concurrently promoted a type of introspection as the main linguistic methodology: Grammaticality judgments7,8 have been used since the early days of transformational grammar. Linguists often use introspection as a source for their own theoretical work. As Bard, Robertson, and Sorace (1996) stated "For many linguists, intuitions about the grammaticality of sentences comprise the primary source of evidence for and against their hypotheses" (p. 32). Not only do linguists use 7 This is the common name given to judgments one makes about the grammaticality of an utterance. Technically speaking, however, when one provides information about sentences or utterances, one is making an acceptability judgment. Grammaticality refers to what is generated by the grammar; acceptability judgments are judgments of wellformedness. Grammaticality, reflecting competence, is not directly accessible; it is inferred through judgments of acceptability. 8 This book is not the place to engage in a detailed commentary on the role of grammaticality judgments in SLA research. Suffice it to say that it is not a straightforward measure and it is not universally regarded as a valid or reliable measure. Some have argued that researchers who use grammaticality judgments do so under the assumption that they are directly tapping competence (Carroll & Meisel, 1990; Ellis, 1990, 1991). This belief is not well-founded. Researchers who have used this measure have typically acknowledged that competence in the linguistic sense is not directly accessible and that it can only be inferred from performance, with acceptability judgments themselves being a performance measure (Cook, 1990; Gass, 1994; White, 1989).

< previous page

page_10

next page >

< previous page

page_11

next page > Page 11

themselves as introspective resources, but when gathering information about languages unknown to them, they ask others for their intuitions about acceptability and unacceptability in their native language. Verbal Reporting Verbal reporting is a special type of introspection and assumes a model of information processing described by Ericsson and Simon (1987). To obtain verbal reports, as new information (thoughts) enters attention, the subjects should verbalize the corresponding thought or thoughts . . . the new incoming information is maintained in attention until the corresponding verbalization of it is completed. (p. 32) Ericsson and Simon illustrated their perspective as shown in Fig. 1.1. In the top panel of their diagram, silent thinking is illustrated. In the middle, talk-aloud appears. This is a vocalization of silent speech. In the bottom panel verbal encoding or think-aloud appears. Ericsson and Simon claimed that the information to be reported does not change depending on the medium of the task (i.e., whether the cognitive work is done silently or verbally). The top panel of Fig. 1.1 represents a normal sequence of states of heeded information (i.e., thoughts). The middle panel, representing talk-aloud data, illustrates the vocalization of silent speech. In the bottom panel, think-aloud, individuals have to convert silent speech into a form that can then be vocalized. Thus, going from the top panel to the bottom panel involves increasing levels of task complexity. What is important for our present purpose is the fact that in a study comparing performance on these three tasks, Ericsson and Simon (1984) found similarities in the responses. Not surprisingly, there was a difference in the time needed to complete each of these task types; think-alouds took longer to perform than the others. Verbal report data has also been subcategorized. For example, Cohen (1998) outlined three primary categories he suggests are used in second language research:

< previous page

page_11

next page >

< previous page

page_12

next page > Page 12

Fig. 1.1. The states of heeded information in a cognitive process and their relation to verbalizations under three different conditions. From Ericcson and Simon (1987). Reprinted with permission of Multilingual Matters. Self-report. With self-report data, one can gain information about general approaches to something. For example, "I am a systematic learner when it comes to learning a second language." This sort of statement might be found on a typical L2 learning questionnaire. Such statements are removed from the event in question and are of less concern here than other types of verbal reporting.

< previous page

page_12

next page >

< previous page

page_13

next page > Page 13

Self-observation. Self-observation data can be introspective (within a short period of the event) or retrospective. A learner reports on what she or he did. An example provided by Cohen (p. 34) is "What I just did was to skim through the incoming oral text as listened, picking out key words and phrases." Such self-observations refer to specific events and are not as generalized as self-report data. Self-revelation. This is what is often described as think-aloud. A participant provides an ongoing report of his or her thought processes while performing some task. In general, introspective reports can be considered as differing along a number of dimensions: currency (i.e., time frame), form (i.e., oral, written), task type (i.e., think-aloud, talk-aloud, retrospective), and support. These introspective types are schematized in Fig. 1.2. The term process tracing is also used to refer to methodologies of verbal reporting. Shavelson, Webb, and Burstein (1986) outlined three types of process tracing. The first is think-aloud or talk-aloud during a task. This is generally known as verbal reporting (cf. van Someren, Barnard, & Sandberg, 1994). The second involves thinking about a previously performed task (i.e., retrospective protocols), and the third involves a prompted interview, for example, watching a video of an event, listening to an audio recording of an event, or even seeing a piece of writing just completed. This latter is known as stimulated recall. van Someren, et al. (1994) were careful to point out that the think-aloud method "is a means to validate or construct theories of cognitive processes, in particular of problem-solving" (p. 9). Despite different terminology, verbal reporting can be seen as gathering data by asking individuals to vocalize what is going through their minds as they are solving a problem or performing a task. Verbal reporting allows researchers to observe how individuals may be similar or different in their approach to problems. The think-aloud protocols illustrated in Fig. 1.3 (van Someren et al., pp. 56) illustrate two very different thought processes during the solving of the same problem. As van Someren et al. (1994) pointed out, these two protocols, produced by two individuals who eventually arrived at the correct answer, reflect very different problem-solving approaches: one algebraic and one hit-or-miss combined with logic (e.g., how old a mother or father

< previous page

page_13

next page >

< previous page

page_14

next page > Page 14

Fig. 1.2. Types of Introspection

< previous page

page_14

next page >

< previous page

next page >

page_15

Page 15 Problem to be solved: A father, a mother and their son are 80 years old together. The father is twice as old as the son. The mother has the same age as the father. How old is the son? Student 1

1. a father, a mother and their son are together 80 years old 2. the father is twice as old as the son 3. the mother is as old as the father 4. how old is the son? 5. Well, that sounds complicated 6. let's have a look 7. I just call them F, M and S 8. F plus M plus S is 80 9. F is 2 times S 10. and M equals F 11. what do we have now? 12. three equations and three unknowns 13. so S . . . 14. 2 times F plus S is 80 15. so 4 times S plus S is 80 16. so 5 times S is 80 17. S is 16 18. yes, that is possible 19. so father and mother are 80 minus 16 20. 64 21. et . . . 32.

Student 2 1. father, mother and son are together 80 years old 2. how is that possible? 3. if such a father is 30 and mother too 4. then the son is 20 5. no, that is not possible 6. if you are 30, you cannot have a son of 20 7. so they should be older 8. about 35, more or less 9. let's have a look 10. the father is twice as old as the son 11. so if he is 35 and the son 17 12. no, that is not possible 13. 36 and 18 14. then the mother is 15. 36 plus 18 is 54 16. 26 . . . 17. well, it might be possible 18. no, then she should have had a child when she was 9 19. oh, no 20. no the father should, the mother should be older 21. for example 30 22. but then I will not have 80 23. 80 minus 30, 50 continued

(text box continued on next page)

< previous page

page_15

next page >

< previous page

page_16

next page > Page 16

(text box continued from previous page) 24. then the father should be nearly 35 and the son nearly 18 25. something like that 26. let's have a look, where am I? 27. the father is twice . . . 28. the mother is as old as the father 29. oh dear 30. my mother, well not my mother 31. but my mother was 30 and my father nearly 35 32. that is not possible 33. if I make them both 33 34. then I have together 66 35. then there is for the son . . . 24 36. no, that is impossible 37. I don't understand it anymore 38. 66, . . . , 80 39. no, wait, the son is 14 40. almost, the parents are too old 41. 32, 32, 64, 16, yes 42. the son is 16 and the parents 32, together 80 Fig. 1.3. Protocol of problem-solving. Reprinted from van Someren et al., The Think Aloud Method, pp. 56, 1994, by permission of the publisher Academic Press. was likely to have been when the child was born). It is only through a think-aloud procedure that these differences in process manifest themselves. Considering simply the outcome would only provide the information that these individuals arrived at the same (correct) answer. As with any methodological tool, there are advantages and limitations to the use of verbal reporting. The major advantage of the use of verbal report is that one can often gain access to processes that are un-

< previous page

page_16

next page >

< previous page

page_17

next page > Page 17

available by other means (as was the case with the mathematical problem-solving transcripts discussed earlier). However, it is also possible to question the extent to which verbal report data are valid and reliable. For example, are the reports given consistent with the behavior of participants? Various researchers (e.g., Ericsson & Simon, 1980 and Lieberman, 1979) have shown that verbal reports are reliable measures and that results obtained using verbal reports do correspond with actual behavior. Not all is rosy, however, for there are also limitations to the use of verbal reports as data. One issue has to do with the accuracy of the reporting. This is particularly the case in self-report and self-observational data. A second issue has to do with the type of memory structure used in recalls. With self-report and self-observational data, when the time between the event reported and the reporting itself is short, there is a greater likelihood that the reporting will be accurate. Stimulated Recall Stimulated recall is one of the introspective methods. As discussed earlier, stimulated recall methodology can be used to prompt participants to recall thoughts they had while performing a task or participating in an event. It is assumed that some tangible (perhaps visual or aural) reminder of an event will stimulate recall of the mental processes in operation during the event itself. In other words, the theoretical foundation for stimulated recall relies on an information-processing approach whereby the use of and access to memory structures is enhanced, if not guaranteed, by a prompt that aids in the recall of information. Why is stimulated recall important for L2 research? What is the function of this methodological tool? As DiPardo (1994) aptly pointed out, "stimulated recall is . . . less a unified approach than a flexible tool that has been adapted to widely varied agendas, and attended by a number of specific methodological choices" (p. 168). Through the use of stimulated recall, "a subject may be enabled to relive an original situation with great vividness and accuracy if he is presented with a large number of the cues or stimuli which occurred during the original situation" (Bloom, 1954, p. 25).

< previous page

page_17

next page >

< previous page

page_18

next page > Page 18

A crucial assumption behind stimulated recall (or any type of recall) is the basic one of recall accuracy. Bloom (1954) attempted to verify the reliability of recall by recording classroom events and asking participants to recall an overt event that occurred immediately following something in the recording. He found that if the recalls were prompted a short period of time after the event (generally 48 hours), recall was 95% accurate. Accuracy declined as a function of the intervening time between the event and the recall. Bloom made the assumption that "the recall of one's own private, conscious thoughts approximates the recall of the overt, observable events" (p. 26). Thus, he argued, the recall method itself is valid for the procurement of information about one's thoughts during an event. It has an advantage over a simple post hoc interview in that the latter relies heavily on memory without any prompts and it has an advantage over think-aloud protocols in that for think-alouds, the researcher needs to train participants, and even after training, not all participants are capable of carrying out a task and simultaneously talking about doing the task. This holds true to a greater extent with speaking activities, for which it is extremely difficult if not impossible to carry out a speaking task and talk about it simultaneously, without the process of think-aloud affecting the task talk. Whereas Bloom's concern was the classroom, the ideas behind stimulated recall have been extended to other areas. For example, Kagan, Krathwohl, and Miller (1963), in a technique that they call Interpersonal Process Recall (part of stimulated recall methodology) investigated interpersonal behavior. In their interesting implementation of the technique, a counselor and client participate in a counseling interview in a closed circuit television studio. The camera is present, and no one but the counselor and client are in the room. At the completion of the interview, the client and counselor are moved to separate rooms; each is accompanied by an interviewer. The videotape of the original session is played back simultaneously in both rooms. The interviewers instruct the participants (counselor and client) to describe what they were feeling during the session, to interpret what they or the other had said, and to translate body movements. Any of the four (i.e., counselor, client, two interviewers) could stop the tape when they wanted in order to comment or to probe. Stimulated recall has also been used as a tool for teacher training and to evaluate teaching effectiveness. Peterson and Clark (1978) video-

< previous page

page_18

next page >

< previous page

page_19

next page > Page 19

taped classrooms and extracted from the videotapes four short (23 minute) segments representing the beginning and ending of the class and two random sequences in between. Specific questions were posed to teachers after they watched each segment: What were you doing and why? What were you noticing about the students? How were the students responding? Were you thinking of any alternative actions or strategies at that time? Did any students' reactions cause you to act differently than you had planned? Variations on this study (Clark & Peterson, 1981; Marx & Peterson, 1981) involved different amounts of teaching time and included the following questions in addition to the preceding ones: Did you have any particular objectives in mind in this segment? If so, what were they? Do you remember any aspects of the situation that might have affected what you did in this segment? Some researchers use stimulated recall to uncover things other than those that are exclusively cognitively oriented. For example, questions about individuals' perspectives on learning (Erickson & Mohatt, 1977) can be explored, as can their impressions of social interactions. Stimulated recall can also be used to explore children's development, including their use of argument skills (Benoit, 1995) or their reasoning abilities (Hample, 1984). One can also use stimulated recall to probe solitary composing processes in L1 or L2 writing (Rose, 1984) or interactions of their social affective and linguistic issues in talk about writing DiPardo, 1994). It can be used to explore readers' lexical retrieval mechanisms or their opinions and impressions about what they have read. Stimulated recall is often used to address questions in research on teachers and their actions, including their decision making and interactive thoughts (Calderhead, 1981a, 1981b). The uncovering of cognitive processes in L2 research is only one area where stimulated recall can be used, although it is probably best known for its uses in more cognitively oriented research. Finally, it should be noted that stimulated recall is often employed in conjunction with other methodologies, as a means of triangulation or further exploration. Within the field of education, there are a number of variations on the theme of stimulated recall, including, for example, how much of a video is watched during the recall session. In Table 1.1, we present some of these variations. There is also variation in terms of who selects which episodes to comment on or whether there are pre-scripted questions to respond to.

< previous page

page_19

next page >

< previous page

page_20

next page > Page 20

TABLE 1.1 A Sample of Studies Using Stimulated Recall in the Field of General Education Author(s) Year What was viewed? Colker 1982 Video (twice) Conners 1978 Video Fogarty, Wang, and 1983 Video Creek Housner and Griffey 1983 Video segments Lowyck 1980 Video Marland 1977 Video McNair 197879Video Morine and Vallance 1975 Video Peterson, Marx, and 1978 Video segments Clark Semmel 1977 Audio, played back entire event Shroyer 1981 Video Woodlinger 1980 Video A variety of specific variations on the theme of stimulated recall methodology are discussed in chapter 3. What Topics Can Be Explored Using Stimulated Recall Methodology? One of the main aims of introspective methodologies in general has been to seek to uncover cognitive processes that are not evident through simple observation. Although full descriptions of the debates surrounding mental processes and introspective verbal reports are beyond the scope of this book on methodology, we provide a brief overview here. As mentioned previously, stimulated recall methodology generally appeals to cognitive psychologists and researchers who are interested in information processing as well as those who are interested in how second languages are learned and taught.

< previous page

page_20

next page >

< previous page

page_21

next page > Page 21

Cognition is concerned with knowledge. In second language research, the focus is on how language-specific knowledge is acquired, organized, and used. As discussed earlier, stimulated recall, as one method of introspection, appeals to researchers because it provides a useful tool that helps uncover cognitive processes which might not be evident through simple observation. More specifically, stimulated recall can be useful for at least three reasons: 1) it can help to isolate particular ''events" from the stream of consciousness. In so doing, it can help to identify the type of knowledge a learner uses when trying to solve particular communicative problems, when making linguistic choices or judgments or just when generally involved in comprehension and/or production; 2) Stimulated recall can also help to determine if this knowledge is being organized in specific ways. Cognitive psychologists have proposed that we employ various types of "cognitive structures" or "mental representations" to help organize the vast amount of information encountered on a daily basis. Some of these structures may be fairly long-lasting, such as the way we organize our mental lexicon, others may be more dynamic and short-lived, such as the structures built during aural comprehension; 3) Stimulated recall can be used to help determine when and if particular cognitive processes, such as search, retrieval or decision making are being employed. In the following sections, we discuss each of these issues in more detail. Knowledge Types An important distinction that is often made when discussing human information processing is that between declarative knowledge and procedural knowledge. In the context of second language learning, declarative knowledge is thought to be comprised of rule knowledge at all linguistic levels, organized in analyzed form. Declarative knowledge is thought to be directly accessible through introspection and thus particularly appropriate for study using stimulated recall. To use or activate declarative knowledge, however, and to extend it through language learning, a second type of knowledge, procedural knowledge, is claimed to exist. Procedural knowledge is comprised of the cognitive and interactional processes involved in the reception, production, and acquisition of language. Unlike declarative knowledge, procedural knowledge is considered automatic and inaccessible via introspection

< previous page

page_21

next page >

< previous page

page_22

next page > Page 22

(Færch & Kasper, 1987). However, breakdowns in automatic processing, such as when the learner does not understand something due to lack of declarative knowledge, may lead to mental states in which some forms of procedural knowledge do become available to introspective report. Knowledge Structures Stimulated recall can help to determine if declarative or procedural knowledge is being organized in specific ways. Two examples that are relevant for second language research are plans and scripts (Nisbett & Wilson, 1977a). Plans, are thought to be mental structures that we build during conscious and deliberate planning or decision making. Such plans have the flavor of declarative knowledge and can be easily self-monitored and reported using stimulated recall. Scripts, on the other hand, are thought to provide the fundamental guidelines we need for much of the routinized or automatic components of our behavior. As discussed above, scripts are clearly related to procedural knowledge, and thus may be harder to explore with introspective techniques. However, it may still be possible to gain some insight into the operation of scripts using stimulated recall methodology. Some of Nisbett's explorations have shown that rather than relying on their actual memories of events for interpretations of their own behaviors, in some circumstances people will rely on their expectations, or scripts, to illustrate what happened. The less recent an event, the more likely expectations rather than memory will be used for interpretation. Ericsson and Simon (1996) provided the following example: "if a picture reminds one of an old friend, it may be tempting to use the stored information about that friend to infer what the person in the picture looked like" (p. 19). To provide a language-related example, we might consider the language learner who receives feedback on the grammaticality of her or his utterance during oral interaction. Because this learner is used to receiving feedback related to meaning or message comprehension in oral discussions with native speakers and not to being grammatically corrected during spontaneous conversation, the learner's script may prompt him or her to report that the feedback provided was related to comprehension, when in fact it was related to grammar.

< previous page

page_22

next page >

< previous page

page_23

next page > Page 23

Stimulated recall techniques may also assist the researcher to gain access to the cognitive processing of scripts (Calderhead, 1981b). Reder (1982), in a study of whether participants used memory retrieval of plausibility judgments to decide whether 10 sample sentences were from a story they had read, found that when memory traces are fresh, retrieval of the exact memory works faster and is easier for participants than considering the plausibility of the sentences. As memory fades, plausibility judgments are easier to make than direct recall of memory. Again, stimulated recall that is carried out immediately after the event and uses a strong stimulus is recommended (cf. Ericsson and Simon, 1980). Cognitive Processes and Learner Strategies Cognitive processes refer to search and storage mechanisms, to inferential mechanisms or to retrieval processes. Such processes are generally thought to operate at an unconscious level. For example, when a person is trying to remember an acquaintance's name, they usually engage in specific cognitive processes, such as trying to remember what the person looks like, the last time they saw him/her, the first letter of their name or the number of syllables in their name. In studies of lexical retrieval, researchers have consistently found that these steps are carried out by almost everyone (Aichison, 1994). However, these processes are usually unconscious unless one is asked to describe exactly the steps during the process of trying to recall the name. Cognitive processes are highly relevant in the field of second language studies, where one needs to investigate the steps learners go through as they search and retrieve lexical items and morphosyntax. Differences and similarities between first and second language processes are obviously of interest. All these topics are of great importance in the field of second language research. Currently, there is a great deal of theoretical debate about unconscious learning, the roles of perception, noticing and attention in second language learning. It is probable that stimulated recall procedures will provide useful data in the ongoing explorations of these topics. Another cognitively oriented aspect of learning that stimulated recall has been used to explore is that of learners' strategies. A great deal of research has been directed at exploring strategy use. Although

< previous page

page_23

next page >

< previous page

page_24

next page > Page 24

strategies are not the focus of this book, we refer the reader to Cohen (1998 and the many references contained in his book) who provides a comprehensive treatment of the topic. Also, in a study that addressed questions about advanced learners, Lennon (1989) explored learners' strategies through introspective methods. He found that some learners demonstrated an orientation towards uncertainty, although they were focused on communication rather than 'correctness.' They reported that their language was very much influenced by that of their interlocutor. As Lennon notes, investigation of learners' beliefs, attitudes, and perceptions about language learning are often explored through stimulated recall. Conclusion As we noted earlier, stimulated recall is an important research tool, but one, like any other research tool, that must be used with full knowledge of its strengths and limitations. Stimulated recall methodologies have been criticized on a number of points, most notably on the memory structures being accessed, and on issues of reliability and validity (see Ericsson & Simon, 1993; Smagorinsky, 1994). Although many of these criticisms have been discussed in the psychological and educational research literature, systematic explorations are seldom found within the second or foreign language literature. With the current increase in the use of stimulated recall methodology, L2 researchers need to be aware of the pitfalls and problems noted in the psychology/educational literature. As van Someren et al. (1994) noted, there are important questions of validity with this methodology. Thus, studies that utilize stimulated recall methodology require carefully structured research designs to avoid problems. These issues are discussed in detail in chapter 4.

< previous page

page_24

next page >

< previous page

page_25

next page > Page 25

2 Introspection and Second Language Research Introspective methods have been a common source of data elicitation in second and foreign language research (cf. Cohen, 1996, 1998; Ericsson & Simon, 1993; see also Færch & Kasper, 1987). Such research has utilized verbal reporting, both on-line (e.g., talk-aloud or think-aloud, sometimes termed concurrent verbal reports) and retrospective. What is common about all these introspections is that the data come from learners' statements about the way they organize and understand information. As mentioned earlier, stimulated recalls are used to explore learners' thought processes or strategies by asking learners to reflect on their thoughts after they have carried out a task. Stimulated recalls are carried out with some degree of support, for example, providing learners with an audio-recording of themselves speaking, or giving them a picture they drew in response to L2 directives. While hearing or seeing these stimuli, learners are asked to recall their motivations and thought processes during the original event. In addition to cognitive psychology and educational research where many of these methodologies originated, verbal reports including stimulated recalls have also been used in the L2 research literature for example, in the interlanguage pragmatics literature (see Cohen & Hosenfeld, 1981; Færch & Kasper, 1987; Kasper & Blum-Kulka, 1993), the L2 writing literature, the L2 reading literature, and currently, the oral interaction literature (Hawkins, 1985; Jourdenais, 1996; Mackey, Gass, & McDonough, in press). As we noted in chapter 1, introspection has been used to gain information about what individuals are doing as they produce language. This is particularly important in the context of second language research because it is often the case that the reasoning behind learners' written or spoken behaviors is inferred by examining only the production data. In L2 research,

< previous page

page_25

next page >

< previous page

page_26

next page > Page 26

understanding the source of second language production is problematic because often there are multiple explanations for production phenomena that can only be assessed by exploring the process phenomena. To take a concrete example, when Spanish speakers produce the English utterance in Example 1 with the intended meaning of something akin to "I am not going," the question arises as to the source for the non-target utterance. Example 1

no go

Are learners producing this utterance because they are following a developmental path, or because they are constrained by the patterns of the native language, or perhaps both (see Schumann, 1978; Zobl, 1980)? Most learners would be unlikely to respond that they were following specific developmental sequences.1 A predicted response to the question of what the learner was thinking about when he produced this utterance might be "I was thinking that I didn't know how to say this in English, so I said it the way it is said in Spanish."2 A response such as this provides some indication that, at least in part, knowledge and use of the native language was one source for the production. Classification Scheme for Introspective Research In this section we discuss some of the most prominent areas where introspective research in general has been used in second language studies.3 Færch and Kasper (1987) offered a useful classification scheme (see Table 2.1) for the collection (not analysis) of introspective data. As they acknowledged, their classificatory system is derived in part from work by Cohen (1987), Cohen and Hosenfeld (1981), Ericsson and Simon (1980, 1984), and Huber and Mandl (1982). 1 See, however, introspective diaries by SLA researchers who are able to reflect on their own developmental progressions (Schmidt & Frota, 1986, Schumann & Schumann, 1977). 2 The astute reader will realize that our hypothetical learner could never have produced such a complex sentence in English if he could not say, "I'm not going." One must assume that this hypothetical recall was done in Sparuish. The language of the recall is further discussed in chapter 4. 3 The term second lanuage studies is being used deliberately here. As Færch and Kasper (1987) noted "we have chosen to refer to the field of study as second language (SL) research, thus avoiding the bias towards developmental issues implicit in the more common term 'second language acquisition research' (p. 5). (see also the diagram in Gass, 1998 representing the diversity of approaches in the field; and Seliger, 1983).

< previous page

page_26

next page >

< previous page

page_27

next page > Page 27

TABLE 2.1 Adaptation of Classification Categories of Introspection Research (Færch & Kasper, 1987)

< previous page

page_27

next page >

< previous page

page_28

next page > Page 28

Considering the categories in column one, the first categorization (i.e., object of introspection) relates to whether one is dealing with linguistic (cognitive), affective, or social factors. Within the category of language-based factors, is the goal to consider knowledge or use? Modality refers to whether the data to be introspected are written or oral. The third. category (i.e., relationship to concrete action) refers to whether or not a specific event is being talked about. This is the case in all stimulated recall studies but is not the case with introspection in general. For example, studies using judgment data are more likely to ask genetic questions, such as, "Can you say X in your language?" Distance from the event refers to the time when introspection takes place vis à vis the original event. Diary studies may be at some distance from the event; in experimental work, such as that found with stimulated recall, the verbalization generally takes place close to the time of the event. Participant training refers to the specialized training that participants need in order to complete the verbalization task. For example, on-line (in real time) verbal reporting generally requires training, whereas diary writing generally requires little training. There are numerous ways of eliciting introspective data. For example, questionnaires may be quite structured, diary writing perhaps much less so. Stimulated recall always requires support (i.e., a record of the event is always present) whereas simple recall of an event does not. Another area to consider is the initiator of the recall. Is it the researcher or experimenter? The participant? All of them? What interaction is there between the researcher and the participant? Finally, is the introspective data supplemented by data of other sorts? We return to this classification scheme in greater detail in chapter 3, where we adapt it to the specific context of stimulated recall. Range of Second Language Studies As noted earlier, within the context of the L2 literature, introspective methods have been used by a number of researchers. In Table 2.2 we list a sampling of such studies, categorized by type,4 focusing in particular on studies that utilize a verbal report. 4 Other studies, such as Roberts (1995), used a video prompt but did not use it as a way of recalling what went on in a classroom. Roberts used it to allow students to do an on-line analysis of what happened in the classroom. We recognize that this may be a fine line; nonetheless, we have not included studies such as these in this categorization.

< previous page

page_28

next page >

< previous page

next page >

page_29

Page 29 As seen in Table 2.2, stimulated recall methodology has been used to address a wide range of research topics. These topics include cognitive processes in general and specifically L2 strategy or inferencing use, L2 teachers' decisions, L2 writing choices and processes, L2 reading and lexical use, and L2 oral interaction amongst other areas. TABLE 2.2 Second Language Studies Using Introspection (na=not available) Author(s) Year Type of Data Method # participants Abraham & Vann 1996L2 test-taking Think-aloud Alanen 1995Reading Rule presentation Think-aloud Anderson 1989L2 test-taking Retrospective think-aloud Think-aloud Anderson 1991L2 test-taking Self-revelation/observation Bartelt 1997Production Introspection (written) Block 1986Reading Think-aloud Bosher 1998Writing Stimulated recall Brice 1995Writing Think-aloud Brown 1993Oral proficiency testing Stimulated recall reports (written)

9 36 28 105 12 3 3 53

(table continued on next page)

< previous page

page_29

next page >

< previous page

next page >

page_30

Page 30 (table continued from previous page) Author(s) Year Type of Data Method # participants Buck 1991Listening Retrospection Cavalcanti 1987Reading Retrospection Think-aloud Chern 1993Vocabulary Think-aloud Cohen 1994L2 test-taking Think-aloud Self-revelation Cohen & Aphek 1979Vocabulary Retrospective reflection Cohen & Aphek 1981Vocabulary Retrospective reflection Cohen & 1987Writing Think-aloud Cavalcanti Cohen & 1990Writing Think-aloud Cavalcanti Cohen & 1993Speaking Stimulated recall Olshtain Cohen, Weaver 1995Strategy use Think-aloud & Li Cumming 1989Writing Compose-aloud Davies & Kaplan1998Grammaticality Think-aloud (not all provided a verbal judgments protocol)

6 na 20 5 17 19 8 12 15 21 23 26

(table continued on next page)

< previous page

page_30

next page >

< previous page

next page >

page_31

Page 31 (table continued from previous page) Author(s) Year Type of Data Method # participants Davies & 1998Grammaticality Think-aloud (not all provided a Kaplan judgments verbal protocol) Dömyei & 1998Oral language Stimulated recall Kormos production Enkvist 1995Translation Think-aloud Fæch & 1986Translation Think-aloud Kasper Retrospection Feldman & 1987L2 test-taking Think-aloud Stemmer Retrospection Fraser 1999Vocabulary Retrospective think-aloud Gass 1994Acceptability Stimulated recall judgments Gatbonton Gerloff Glahn

1999Pedagogical knowledge 1987Translation 1980Gender Communication strategies

Stimulated recall Verbal report Think-aloud Stimulated recall

26 44 47 1 20 8 small subset 7 5 26

(table continued on next page)

< previous page

page_31

next page >

< previous page

next page >

page_32

Page 32 (table continued from previous page) Author(s) Year Type Gordon Gosden Goss, Zhang, & Lantolf Gu

of Data Method # participants 1987L2 test-taking 1996Writing 1994Grammaticality judgments 1994Vocabulary

Haastrup

1987Vocabulary

Hill Höscher & Möhle Hosenfeld Hosenfeld Hosenfeld Hosenfeld

1994L2 test-taking 1987Translation 1976Grammar 1977Reading 1979Reading 1984Reading

Self-revelation/observation Self-reflection/self-report Think-aloud

na 16 15

Think-aloud Retrospective interviews Think-aloud Think-aloud with retrospection Self-revelation/observationn Think-aloud Think-aloud Think-aloud Think-aloud Think-aloud

2 104 na 7 25 40 1 2

(table continued on next page)

< previous page

page_32

next page >

< previous page

next page >

page_33

Page 33 (table continued from previous page) Author(s) Year Type of Data Method # participants Huckin & Bloch 1993 Vocabulary Jones 1985 Writing Jones & Tetroe 1987 Writing Jourdenais, Ota, Stauffer, 1995 Linguistic Boyson, & Doughty knowledge Kern 1994 Reading Krings 1987 Translation Laviosa 1991 Listening Lay Lennon

Think-aloud Stimulated recall Think-aloud Think-aloud

3 9 6 10

Think-aloud Think-aloud Self-observation (immediate retrospection) Think-aloud Self-report

51 8 5

1982 Writing 1989 Acquisition strategies Mackey, Gass, & McDonough in Oral language Stimulated recall press Murphy 1989 Listening Talk-aloud strategies

4 4 17 12

(table continued on next page)

< previous page

page_33

next page >

< previous page

next page >

page_34

Page 34 (table continued from previous page) Author(s) Year Type of Data Method # participants Neubach & Cohen 1988Dictionary use Think-aloud Nevo 1989L2 test-taking Immediate introspective and (Reading) retrospective reports (written) Paribakht & Wesche 1999Vocabulary Think-aloud Retrospective interviews Poulisse 1990Communication Stimulated recall strategies Poulisse, Bongaerts, &1987Communication Stimulated recall Kellerman strategies Raimes 1985Writing Think-aloud Robinson 1991Pragmatics/SpeechThink-aloud Acts Retrospective interviews Skibniewski 1990Writing Think-aloud Stemmer 1991L2 test-taking Think-aloud and immediate retrospection (stimulated recall) Swain & Lapkin 1995Writing Think-aloud Swain & Lapkin 1998Oral interaction Think-aloud

6 42 20 45 45 8 12 3 30 18 35

(table continued on next page)

< previous page

page_34

next page >

< previous page

next page >

page_35

Page 35 (table continued from previous page) Author(s) Year Type of Data Method # participants Tomitch 1999 Reading Think-aloud and retrospective interview Tyler 1995 Discourse Stimulated recall Vignola 1995 Writing Think-aloud Villamil & Guerrero 1998 Writing Think-aloud Warren 1996 L2 test-taking Self-revelation/observation Zamel 1983 Writing Retrospective interviews Zimmermann & Schneider 1987 Vocabulary Think-aloud 10

12 2 17 14 20 6

Conclusion This chapter and the preceding one have presented some of the arguments concerning the cognitive processes stimulated recall may uncover. The chapter also provided evidence that stimulated recall, although generating some interesting criticisms, is, in general, a methodology that has been subjected to both empirical testing and theoretical review (see chaps. 4 and 5 for a discussion of validity and reliability). The following chapters provide greater detail on how stimulated has been and can be used in second language research.

< previous page

page_35

next page >

< previous page

page_ii

next page >

< previous page

page_37

next page > Page 37

3 Characterization of Stimulated Recall In this chapter the stimulated recall procedure is described in detail and contextualized through a review of studies that use it. In the first part of the chapter we present descriptions of studies that have been carried out in a variety of areas of L2 research using different implementations of stimulated recall. These studies have been selected to illustrate a range of topics that can be successfully investigated using stimulated recall. These descriptions include studies in the area of oral interaction, both comprehension and production, and examples from classroom-based, qualitative and experimental research; studies in the areas of acceptability judgments, reading, and vocabulary learning; and studies in different types of L2 writing research. Throughout the remainder of the book we refer back to some of these published studies for concrete examples of many of the constructs we discuss in relation to the stimulated recall procedure. In the final parts of the chapter, we focus on the recall procedure itself, outlining various methods for providing support for the recall. Issues of timing (e.g., how much intervening time occurs between the initial event and the recall of the event) are discussed. Finally, we summarize some typical events that have been subject to investigation using stimulated recalls in the L2 literature. Review of Stimulated Recall Procedure As discussed in chapters 1 and 2, stimulated recall is used primarily in an attempt to explore learners' thought processes and strategies by asking learners to reflect on their thoughts after they have carried out a

< previous page

page_37

next page >

< previous page

page_38

next page > Page 38

predetermined activity. Simulated recall is carried out with some degree of support, for example, showing a videotape to learners so that they can watch themselves carrying out an activity while they vocalize their thought processes at the time of the original activity. Oral Interaction Research Perceptions of Comprehension The first example presented is taken from an early study in the area of oral interaction by Hawkins (1985). The purpose of Hawkins' study was to determine whether replies in non-native speaker discourse, which on the surface appeared to be appropriate conversational responses, did in fact represent comprehension of what had preceded in the discourse. To accomplish this goal, Hawkins utilized stimulated recall methodology. Two dyads of adult participants, each dyad consisting of a native speaker of Spanish and a native speaker of English, carried out four communicative tasks in English. The tasks were designed so that both the non-native speakers and the native speakers would be in possession of information needed by the partner to complete the task. One task was the popular English as a second language (ESL) 'grab bag' game, which consisted of one participant removing and then describing objects from a bag containing common objects (e.g., a plastic knife, a key, a piece of chewing gum). The other participant asked questions about the object in an effort to guess what the object was. The interactions were tape recorded. The tapes of the task-based interaction were played back to the participants, who were asked to ''stop the recorder at any time and comment on what you were thinking at that point in the conversation" (Hawkins, 1985, p. 165). Additionally, Hawkins stated, "The investigator also felt free to stop the recorder and ask questions of the subjects if the subjects themselves did not stop the recorder" (p. 165). The stimulated recall was conducted in each participant's native language. Experimental Research Mackey, Gass, and McDonough (in press) conducted a study of oral language use to investigate the accuracy of learners' perceptions of native speaker feedback. The database consisted of two groups: learners of English as a second language and learners of Italian as a foreign language. Each participant carried out a task-based activity in which the native speaker and the learner both had a picture.

< previous page

page_38

next page >

< previous page

page_39

next page > Page 39

The pictures were similar although not identical. The task was for the two interlocutors to describe the pictures to each other in order to identify the areas of difference. This session lasted for approximately 1520 minutes and was videotaped. During the interaction, the English- and Italian-speaking researchers1 provided different types of implicit negative feedback when the participants produced a non-targetlike utterance. Because various studies have shown that excessive corrective feedback can lead to dysfluencies (Aston, 1986), not all non-targetlike utterances received feedback. The stimulated recall sessions were conducted immediately after completion of the task-based activity. The videotape was rewound and played for the learner by a second researcher (an English speaker for the ESL group and Italian speaker for the Italian group) who discussed the stimulated recall procedure. While watching the videotape, the participants were told that they could pause the tape at any time if they wished to describe their thoughts at any particular point in the interaction. The researcher also paused the tape after episodes where feedback was provided and asked the learner to recall his or her thoughts at the time when the original interaction was going on. These recall sessions were audiotaped. As discussed previously, this stimulated recall procedure was aimed at eliciting learners' perceptions about the feedback episodes at the time when the interaction was in progress. Qualitative Research And finally, we turn to a more qualitative usage of stimulated recall methodology. Tyler (1995) examined a tutoring session between a male native speaker of Korean who was a graduate student in Computer and Information Science and a female native speaker of U.S. English, enrolled in an introductory computer programming class. The tutor had received a score of 235 out of a possible 300 on the SPEAK test a short time before the study, suggesting that he had some problems with grammar and pronunciation but was generally comprehensible. The student went to the tutoring session for help with an assignment. As a part of their university training, interna1 The English-speaking researcher was a native speaker of English; the Italian-speaking researcher was a near-native speaker of Italian.

< previous page

page_39

next page >

< previous page

page_40

next page > Page 40

tional graduate student tutors offered free tutoring, and these tutoring sessions were always videotaped and reviewed with ESL instructors to improve the tutor's English communication skills. Thus, stimulated recall methodology was employed by the ESL course designers to improve the language skills of their tutors, and this stimulated recall interview was utilized by Tyler, along with the original tutoring session videotape, for her research into the sources of miscommunication in interaction. As Tyler pointed out, both participants came to the interaction with clear motivation for their interaction to proceed smoothly. However, at the end of the session, both participants independently complained to the supervisor about the other's uncooperative attitude. These problems were the focus of Tyler's study. Before the stimulated recall was carried out, the videotape of the tutorstudent interaction was viewed independently by two native speakers of U.S. English, who were both involved with the ESL program. Each of these native speakers compiled a list of signs of interlocutors' discomfort, including phenomena such as overlap and volume changes, as well as challenges by the student and refusals by the tutor to provide requested information. A combination of these two lists was then used as a guide during the stimulated recall sessions. There were two sessions, each one with one of the participants and the researcher present. The participants were asked to stop the videotape at any time and comment on anything that made them uncomfortable or confused. The researcher also stopped the tape at other points, basing her choices on the predetermined list. When she stopped the tape she asked the participant to comment on what was going on. These stimulated recall sessions were audiotaped. Tyler's research was based on the videotape of the interaction and the audiotape of the stimulated recall sessions. She carried out an in-depth qualitative analysis of the data to explore why the participants had different interpretations of the exchanges. Communication Problems Another interesting use of stimulated recall data can be seen in Dörnyei and Kormos (1998). They investigated speakers' management of problems in L2 communication. They based their analysis of coping strategies for these problems on Levelt's (1989, 1993, 1995) model of speech production and distinguished four sources of L2 communication problems: resource deficits, processing time

< previous page

page_40

next page >

< previous page

page_41

next page > Page 41

pressure, perceived deficiencies in one's language ouput, and perceived deficiencies in the interlocutor's performance. They described problem-solving devices in terms of the pre- and post-articulatory phases of speech processing, illustrating the mechanisms they identified by examples and retrospective comments taken from L2 learner data. Although the use of stimulated recall is interesting in Dörnyei and Kormos's study, the authors stated clearly that the intent of their study was conceptual. Perhaps because of this, little information is supplied about the mechanics of the retrospection other than the (footnoted) information that participants were asked to listen to the recordings of their own interactions and to answer questions about and make comments on the difficulties they experienced. Dörnyei and Kormos claimed that over 450 manifestations of problem management were discovered or confirmed through stimulated recall. In their study, 44 Hungarian learners of English were asked to perform three communicative tasks. The recall comments supplied are similar to some of those described by Mackey, Gass, and McDonough (in press; see earlier discussion), although the conceptual frameworks in the two studies are different. Example 2 from the Dörnyel and Kormos study illustrates a learner's problem with his or her own output. The authors described this example as self-corrected error repair, making self-initiated corrections of accidental lapses in one's own speech: Example 2 (From Dörnyei & Kormos, 1998) Learner:you have to . . . er rent it er . . . for . . . 35person . . . uhmm it's max . . . minimum, minimum yes Retrospective Interviewer Comment: Have you started saying 'maximum' and what happened then? Retrospective Learner Response: I realized that I was not using the right word. It's not "maximum" but "minimum" because it's a room for 40 people While Dörnyei and Kormos used Levelt's model of speech processing to explain the psycholinguistic mechanisms involved in such repair, stimulated recall methodology was used to supply the empirical evidence.

< previous page

page_41

next page >

< previous page

page_42

next page > Page 42

Acceptability Judgments A study by Gass (1994) illustrates stimulated recall in an investigation of acceptability Judgments. The main purpose of this study was to ascertain the extent to which acceptability judgments are a reliable instrument for gathering second language data. There were two parts to the study. In the first part, non-native speakers of English were asked to give acceptability judgments for 30 English sentences, including 24 sentences with relative clauses, the target of investigation. Participants were first asked to judge each sentence categorically as to whether they felt it was grammatical or ungrammatical and then were asked to assess the degree of confidence they had in their judgment. The answer sheet was as follows: He remembers the man who his brother is a doctor. -3definitely incorrect

-2

-1

0 unsure

+1 +2

+3 definitely correct

A value of [-3] meant that the participants felt that they were 100% certain that their assessment of incorrect was correct; on the other hand, a value of [+3] meant that the participants were 100% certain that their assessment of correct was the correct one. Other values suggested different degrees of certainty or uncertainty. A value of 0 meant that they did not know. The test (with different randomizations) was given to each participant twice with a one-week interval. As mentioned, the purpose of the study was to gain information about the extent to which participants gave similar responses at the two different test administrations. Answer sheets were coded to determine the amount of consistency between Times 1 and 2. Stimulated recall was used with a subset of the participants to uncover reasons why they might have had radically different judgments at the two different times. Gass looked for those responses where the subject had a [+ 3] at Time 1 and [-3] at Time 2 or vice versa. The answer sheets of the participants formed the stimulus for the recall.

< previous page

page_42

next page >

< previous page

page_43

next page > Page 43

Reading/Vocabulary Paribakht and Wesche (1999) investigated learner strategies and knowledge sources that learners use in dealing with unknown words. In their study, they used both concurrent and retrospective think-aloud protocols; the latter are what concern us here. Participants were ESL learners who were given a text to read that contained unknown words (as determined by a pretest). They had access to an English dictionary. After they completed the reading task, they performed a comprehension task and a summary task. In the first, they were asked to answer comprehension questions on a passage they had just read. This task was completed as a think-aloud. Following each question that they answered, they were asked if they had encountered any unknown words during the reading part of the task and if they had, how they had dealt with them. In the second task, which took place during the actual reading, learners were asked to summarize each paragraph as they read it. Immediately following each paragraph summary, learners were asked the same questions; that is, they were asked if they had encountered unknown words and, if so, how they had handled these words. In addition, during the think-aloud parts, researchers put forward other prompts saying, "I can see you're shaking your head, tell me what you're thinking" or, "Why have you put your finger on that word?" or, "You look puzzled" (p. 202). Both the comprehension and summary tasks yielded immediate retrospective protocols. Paribakht and Wesche also collected delayed retrospective protocols by asking participants questions after they had finished the comprehension and the summary tasks. Questions were posed concerning the difficulty of the task, the interest of the task, and, important for their purposes, the words they had worked with and how they had dealt with unknown words. In other work by Paribakht and Wesche (1997), a recall task (in this case, a written questionnaire) was used to determine learners' perceptions of particular vocabulary activities and the extent to which these exercises contributed to their learning of new words. These data were collected alongside data that measured their productive skills in terms of vocabulary learning.

< previous page

page_43

next page >

< previous page

page_44

next page > Page 44

Writing Research Tutorial Sessions DiPardo (1994) conducted research within the context of a university adjunct writing program with a large population of nonAnglo students. In addition to the regular writing course, program participants were enrolled in three additional hours of what were called collaborative tutorial groups. The researcher was interested in the dynamics of the program. In particular, she wanted to assess the webbing of social, affective, and linguistic issues in students' and tutors' talk about writing (Dyson and Freedman, 1991)that is, how the small group conversations related to students' struggles to adjust to life at this predominantly Anglo, middle-class campus, and how tutors' efforts to provide meaningful support related to larger patterns of institutional uncertainty. (p. 169) DiPardo compiled recordings (three audiotaped segments per participant) that she felt were typical of the interactions in the tutorial group sessions that had taken place during the school term. The total playback time was 15 minutes. These composites served as the prompt. There was little consistency apparent across participants concerning the intervening time between the original event and the recall sessions. Before each playback session, DiPardo reminded each participant of the context of each interaction. This included refreshing his or her memory about various points in the assignment, the point of the composing process. Written transcripts were also provided. The researcher stopped the tape after each segment. She also allowed participants to make comments. After each segment the researcher asked open-ended questions such as, "Comments?" "What do you have to say about this?" and "What's your sense of what was going on here?" Writer's Block Rose (1984) investigated writer's block during the composing process. He used two video cameras, one focused on the pad of paper that a participant was using during the writing process and the second facing the student, taping him or her from the waist up. Participants were given the following instructions:

< previous page

page_44

next page >

< previous page

page_45

next page > Page 45

Write this essay as you normally would. Do whatever you usually do when you sit down to compose a school paper. I ask only two things of you: (1) Line out rather than scratch out words you write but choose not to use. (2) Don't rip up any paper you've used. The video cassettes I'm using run for one hour, so after one hour, I'll return, and we'll watch the tape of your essay. If you don't finish, that's o.k. This is not a test. (p. 26) The monitors were in another room where the researcher could watch them, but the student was alone while writing. Before the actual onset of the study, the researcher gave students as much time as necessary to get used to the setting by allowing them time to free-write. This usually took about five minutes. After the videotaped writing session, the researcher and the student sat side by side so that they could view a common monitor. The monitor had a split-screen image (3/4 was of the page that the student was writing on and 1/4 was the student him or herself). After preliminary questions about the student's views of the composing process and the views of how the process compared to those of others, the researcher explained how the stimulated recall worked. As we watch the tape I'll be asking you questions about what you were doing. At times I'll even stop the videotape so we can examine a marginal note, a word choice, a revision and so forth. As you watch your writing unfold, try to recall what you were thinking at the time; try to put your mind back into the task. Anytime you remember something, say it, interrupt me, stop the tape if you want. I'm interested in finding out what you were thinking when you were writing, and it doesn't matter at all to me if those thoughts were silly or profound. I'll audio-record our conversation so I don't have to divide my attention by taking notes. (pp. 2930)

< previous page

page_45

next page >

< previous page

page_46

next page > Page 46

In this study, the researcher stopped the tape when he saw notes, other marginalia, and smooth flow of writing as well as when he saw blank stares, quizzical facial expressions, and other expressions of thinking. Composing Process Bosher (1998) was concerned with the composing process of Southeast Asian students. Her particular focus was differences between students who had graduated from a high school in the United States and those who had completed high school in their home country. Her data collection process consisted of giving participants an article to read followed by a task in which they wrote their opinion about the topic of the article (data from three participants were reported on in her published article, but data were also gathered from others). The article was from a local newspaper and the topic was of concern to at least some of the students (it was about recent school district requirements that students would have to pass competency tests to graduate from high school and the impact that this new requirement would have on the Southeast Asian community). The writing prompt asked them to write about whether or not second-language students should be required to take competency tests before graduating from high school. During the hour of writing, videocameras were focused on the paper so that the movements of pen and paper were visible. While students were writing, they were observed on a monitor situated in another room. In particular, their pause time was noted, as was the time when they referred back to the original reading. Immediately following the writing session, the students were interviewed, with pause times of several seconds used as prompts to probe the students' thought processes during the writing protocol. Their comments were audiotaped. Pragmatics Robinson (1991) investigated interlanguage pragmatics using a stimulated recall. Of specific concern was the speech act of refusal. The participants in her study were 12 Japanese ESL students (all female). Her methodology contained three parts: discourse completion task (DCT), a think-aloud session about the DCT, and stimulated recall about the DCT and think-aloud. Participants completed a 6-item DCT questionnaire with think-aloud, followed immediately by a stimulated recall

< previous page

page_46

next page >

< previous page

page_47

next page > Page 47

based on their think-aloud protocols. At that time, the tapes of their think-alouds were played back "to remind subjects of specific thoughts" (p. 47). The recall probes focused on intentions, cognitions, planning, and evaluations (Ericsson & Simon, 1984). Some questions that were used to probe specific intentions during the recall session were "What did you intend to say?" and "Why did you say that?" Examples of questions designed to probe cognitions included "What did you notice about the situation?'' "What were you paying attention to at that moment?" and "What were you thinking when you said that?" Planning questions included "What did you plan to say? and What did you plan to say first, second, etc.? Finally, evaluation questions included "What were your alternatives?" and "What else could you have said?" (p. 47). In sum, what these studies have shown is that in stimulated recall procedures, as an initial step the researcher can create a context for an event to be recalled later. We have presented examples of events that have been used in L2 research. As in the examples presented here, events can include oral interactions, paper and pencil tasks, reading passages, and L2 written products. We now turn to a discussion of some issues surrounding the recall part of the procedure. Classification of Recall Support In this section we provide information about ways in which various aspects of the stimulated recall procedure can be classified, including details about the relation to the action, participant training, instrument structure, and stimulus for the recall. Obviously, the stimulus is a key part of any methodology in which the data collected rely on participants recalling a previous event. Not wishing to reinvent the wheel, we have based our classification system on the method put forward by Fxrch and Kasper (1987) (described in chap. 2). They presented a broad system, establishing criteria according to which a range of introspective methods can be classified. They based their model on three key interpretations in the literature: Ericsson and Simon's (1980, 1984) cognitive-psychological model of information processing, the description of verbal data used by Huber and Mandl (1982) in the educational and social sciences, and Cohen's (1984, 1987) work on classifying intro-

< previous page

page_47

next page >

< previous page

page_48

next page > Page 48

spective methods in L2 research. Færch and Kasper's criteria for classification appear in the next section. We present a revision and extension of their model that applies directly to verbal protocols obtained through stimulated recall methodology. We focus on the recall support aspects of their model. In each categorization, we also present illustrations for our revised model based on recent L2 studies that utilized stimulated recall methodology. Færch and Kasper (1987) As we noted in chapter 2, Færch and Kasper's classification scheme is frequently used in much of the literature in L2 studies using stimulated recall. In the adaptation in Fig. 3.1, we show exactly how their model applies in the context of the collection of stimulated recall data. In addition we present an extension of their model, situating many of the classificatory terms on a scale and illustrating each term with reference to recent L2 studies also situated on the scale. We conclude with recommendations about the procedure itself. Relationship to Specific Action In chapter 2, where we initially presented Færch and Kasper's (1987) model, two additional categories were noted: object of introspection and modality. In this section we are concerned not so much with what introspection is used for nor the modality of use; rather, we focus on the recall support needed. Thus, our discussion of classification schemes begins with the attempt to identify the cognitive information that is the focus of the recall, in terms of whether or not it is related to a specific action. In the examples given in section 2, concrete actions that were the focus of recalls can be seen in the L2 writing examples and in the task-based interaction. If the cognitive information to be recalled is not related to a concrete action, there is less specificity in the recall. For example, learners might be asked to recall general tendencies in their behaviors, such as their strategy use in the previous few days (e.g., Cohen's self-report, see also chap. 1). There is no concrete situation or action that they are asked to focus on, but rather they are asked to abstract to their thought processes in general.

< previous page

page_48

next page >

< previous page

page_49

next page > Page 49

Fig. 3.1. Adaptation of classification scheme from Faerch and Kasper (1987). Temporal Relationship to Action What Færch and Kasper termed temporal relationship to action is also important when classifying stimulated recall methods. The length of the time period that elapses between the event and the recall, what sort of memory structures are be-

< previous page

page_49

next page >

< previous page

page_50

next page > Page 50

ing accessed, and the efficacy of the support in overcoming any delay are all key issues. With immediate (sometimes known as consecutive) retrospection, which has little or no gap, many researchers have argued that information in memory structures by stimulated recall are still available for access (Ericsson & Simon, 1987). As they noted: Our model predicts that retrospective reports on the immediately preceding cognitive activity can be accessed and specified without the experimenter having to provide the subject with specific information about what to retrieve. In this particular case, the subject will have the necessary retrieval cues in STM after a general instruction is given "to report everything you can remember about your thoughts during the last problem". This form of retrospective verbal report should give us the closest approximation to the actual memory structures. (pp. 4041) Longer periods of time, even when the recall support is very strong, often lead to controversy in terms of what is being accessed and what claims are being made by the researcher. Of course, recalls also exist as delayed recalls, as in diary studies or exit interviews. In these cases the stimulus is often rather weak: "On Fridays, write in a language diary about your experiences in class over the last week" or "In this exit interview please tell us about your feelings about the language program from which you have just graduated." In these cases, the recency effect argument put forward by Cohen (1987) is probably operating in its weak form. Obviously, researchers should take care in making claims and should spell out potential problems with validity when data are based on delayed or weak stimulus recalls. However, Cohen and others (Bloom, 1954) have also pointed out that the majority of loss of memory may occur almost immediately after the event; thus, delays of 3 hours to 3 days may result in similar data. Figure 3.2 illustrates this temporal relation to action. Participant Training When we consider the degree of support offered for the stimulus in a delayed recall, we must also explore how well the

< previous page

page_50

next page >

< previous page

page_51

next page > Page 51

Types of Stimulated Recall Consecutive Recall Example: L2 Writing. Immediately after finishing revisions on an essay draft, participants are interviewed about the changes they made, using! the initial and final written products as stimuli. Delayed Recall Example: L2 Reading. After reading a passage in the L2, participants are given a list of questions about their comprehension of the passage. After the straight comprehension questions, they are asked to write about particular difficulties they may have had with the passage, and how they overcame them. They are asked to take the questions home and bring their answers in the next day. Nonrecent Recall Example: L2 Strategies. After taking a placement test in the middle of the instructional year, one class of participants is divided into groups of successful and less successful students. These students are given email accounts, and are asked to send at least one message a week to a researcher, speculating on the ways in which they are learning vocabulary during the current semester, as opposed to the previous semester. Fig. 3.2. Examples of consecutive, delayed and nonrecent recalls. participants are trained at interacting with the stimulus. There seems to be a lack of consensus in the literature on the topic of training. First, instructions and training need to be distinguished. Of course, in many if not most experiments, participants are generally provided with some form of instructions, however brief. While both are clearly important, in this classification scheme our focus is on training as opposed to instructions. We also need to consider that on-line recalls, or think-alouds, are more difficult to carry out without training than stimulated recall procedures. When asked to vocalize their thoughts during a problem-solving task, many people need some practice as well as a model to follow. Stimulated recall procedures are generally easier to

< previous page

page_51

next page >

< previous page

page_52

next page > Page 52

follow, although, as discussed later in this book, the effect of the instructions and on-going questioning is important. Adequate direction is often needed to keep participants on track and in the "there and then" as opposed to the "here and now" (see chap. 4 for more information on temporal location). Ericsson and Simon (1987, 1993, 1996) claim that participant training does not affect the validity of the verbal report and in effect only serves to increase completeness. We argue that in some designs, training participants by showing them videotapes of others carrying out the procedure or giving them transcripts or diagrams to view may affect the quality of the report data in many ways. Empirical research is still needed to address this issue. For example, priming studies in the psychology literature have shown that participants' verbalizations can be affected by a number of factors in the preceding input. In the L2 context, we need to be particularly vigilant about introducing potentially confounding input variables. The training effect and the effect of memory interference on the recall data are both important issues that should not be underestimated and should be the focus of methodological investigation. In the absence of such detailed empirical work, it seems safe to say that participants should be trained if pilot studies have shown that they need such training in order to provide recalls and that the minimum training necessary should be provided to avoid influencing or affecting the subsequently recalled data. Procedural Structure In terms of the instrument used for collecting the stimulated recall data, we put forward a categorization in terms of structure. Based on Færch and Kasper's (1987) model, we have classified the instrument using a scale that runs from high structure, such as questionnaires with multiple choice items, to low structure, including open interviews not constrained by predetermined researcher questions. Obviously, a low-structure recall procedure allows learners to specify what they verbalize, when they verbalize, and how much they verbalize. The more constraints imposed on the recall in terms of time or researcher control over questions, the more structured it becomes. Conversely, a high-structure multiple-choice questionnaire becomes lower in structure if open-ended questions are used or if a question such as "anything else to report?" follows the structured question.

< previous page

page_52

next page >

< previous page

page_53

next page > Page 53

Stimulus for Recall The purpose of the stimulus is to reactivate or refresh recollection of cognitive processes so that they can be accurately recalled and verbalized. Audiotapes, videotapes, written products, and computer-captured data can all be used as stimuli. Obviously, many caveats apply in terms of the potential effectiveness of the stimuli, particularly when a stimulus with high support is used. It is important to be aware that not all participants respond in the same way to the stimulus and to be cautious about claims for strong levels of support. Since we can only speculate about how strong or weak support is for learners, regardless of researchers' intentions, when considering stimuli, we must also note that retrospective protocols can be obtained without a stimulus. Initiation of Questions/Recall Interaction When considering both the recall stimulus and the procedure, it is important to think about who initiates the stimulus episodes. Who selects them? Who interacts with them? As can be seen in Fig. 3.1, we have categorized these issues in terms of those initiated by the learner, by the researcher, and by both the learner and the researcher. In our study of oral interaction (Mackey, Gass, & McDonough, in press), we designed the stimulus, which was videotapes of learners interacting, to be accessed by both the learner and the researcher. The remote control was set on the table between the two individuals. The learners were initially asked to choose a segment and to pause and replay the videotape to ensure that they knew how to operate the control and to help them feel comfortable doing it. The researchers chose certain segments for replay because they contained implicit negative feedback, which was the focus of the study. The learner-initiated replays represented only about 10% of all the replays, but the situation was made as equal as possible. Factors that may affect stimulated recall interactions include whether the stimulus is discussed and the recall provided in the L1 or the L2 and individual learner variables, which may, of course, be affected by cultural factors. It is often useful to provide learners with models of other-initiated and self-initiated recall support interaction. In considering who initiates various aspects of the recall support, it is also useful to recognize that this often interacts with who controls the verbalization of the recalls. Some recall procedures do not rely on researcher-participant dyads, but participant-participant-researcher, and

< previous page

page_53

next page >

< previous page

page_54

next page > Page 54

even whole-class-teacher-researcher groups. In these cases of more than one participant, the interaction between or among participants must also be considered in terms of its potential effect on the recall data. In this section we have attempted to classify and exemplify issues related to the recall support or stimuli as well as the wider context for the stimulated recall procedure. Based on our classification system, together with theoretical predictions and currently existing research using stimulated recall, we can make some specific recommendations about the procedure. RECOMMENDATIONS Timing Data should be collected as soon as possible after the event which is the focus of the recall. As the event becomes more distant in time and memory, there is a greater chance that participants may say what they think the researcher wants them to say or may create a plausible explanation for themselves because the event is less sharply focused in their memories. Strength The stimulus should be as strong as possible. This may mean using a stimulus of more than one source. For example, participants can watch a video if the recall is immediately after the event. If it is more delayed, they can watch a video and even read a transcript of the relevant episodes as well. Training The participants should be minimally trained, that is, they should be able to carry out the procedure, but should not be cued into experimental goals or unnecessary information. Training can be facilitated by careful pilot testing. (text box continued on next page)

< previous page

page_54

next page >

< previous page

page_55

next page > Page 55

(text box continued from previous page) Often, simple instructions and a direct model will be enough in a stimulated recall procedure. Sometimes, even instructions are not necessary; the collection instrumentation will be sufficient, for example in the case of a questionnaire or a Q-A interview. Structure How much structure is involved in the recall procedure is strongly related to the research question. Generally, if participants are not 'led' or focused, their recalls will be less susceptible to researcher interference. Unstructured situations do not always result in useful data. Also, if learners participate in the selection and control of stimulus episodes and are allowed to initiate recalls themselves, there will again be less likelihood of researcher interference in the data.

< previous page

page_55

next page >

< previous page

page_ii

next page >

< previous page

page_57

next page > Page 57

4 Using Stimulated Recall Methodology This chapter goes into specific detail about ways to carry out stimulated recall. In addition to the detailed step-by-step guide, many of the potential pitfalls involved in carrying out a stimulated recall methodology are discussed, together with our recommendations for avoiding them. Again, we use specific examples from previous studies to illustrate the points. Data Collection: Instructions for Participants and Researchers Before using stimulated recall methodology to collect data, it is advisable to develop a detailed research protocol1 as discussed earlier. The stimulated recall procedure is generally complex. For example, a stimulated recall of oral interaction often involves making at least two separate data recordings, one replay, and two sets of instructions. Thus, the amount of detail specified in the research protocol is important. A detailed protocol helps the researcher to anticipate problems in advance while also acting as a checklist for the many variables and factors the researcher needs to consider and balance while carrying out the procedure. It is also important to think carefully about and pilot test all the procedures laid out in the instructions to the learner, paying particular attention to the effects of the instructions on the procedure. Pilot testing can often lead to revisions and fine tuning of the protocol and can help to avoid costly and time-consuming problems during the data collection procedure. Careful pilot testing can also help to avoid the loss of valuable, potentially useful, and often irreplaceable data. Appendices A and B contain complete example protocols for stimulated recall procedures in oral interaction settings. Figure 4.1 shows a set of instructions for the participant and the researcher, which were laid out in the 1 Note that the use of the word protocol here, as elsewhere in this chapter, refers to a set of instructions, parameters, and details for carrying out an experiment, as opposed to a verbal protocol or report.

< previous page

page_57

next page >

< previous page

page_58

next page > Page 58

protocol contained in Appendix A. These instructions are detailed, clear, and unambiguous. Instructions for stimulated recall procedures are often standardized. For example, they may be tape-recorded. At times they are read from a script. One reason for the standardization of instructions is the importance of orienting the participant to the actual time period under recall. A single word or tense change can affect the nature of the participant's recall. The section on procedural pitfalls deals with this issue in greater detail. It is important to note that if participants are asked not just to vocalize their thoughts but also to explain them, the additional cognitive load may interfere with memory and recall. Hence, Ericsson and Simon's (1993) often-quoted direction is not to ask participants to explain but to ''keep talking." In Fig. 4.1, we provide instructions used in the Mackey, Gass, and McDonough (in press) study of oral interaction. The instructions were as specific as possible. They were developed after two separate pilot studies had revealed problems that the instructions had not previously covered. For example, in the first pilot, one trainee researcher engaged the participants in conversation by making full responses to the learners' statements. Hence, the instructions were expanded to include the comments about backchanelling being preferable to responses. It is important to note that the instructions not only tell the researcher what to say to prompt the recall comments but also provide information about what to say during the recall and after it. A second detailed protocol appears in Appendix B. This protocol was developed for a study that used stimulated recall as a pilot-testing tool. The goal of this stimulated recall procedure was to ensure, in advance of the experiment, that participants in the study perceived the treatment in the way the researcher intended. This preemptive use of stimulated recall is a good example of the versatility of this methodology. This research was carried out by Leeman (1999) to explore the question of how participation in interaction can facilitate SLA by focusing on exactly how recasts promote L2 development. Recasts are targetlike reformulations of a learner's original utterance. They can provide implicit negative evidence, which has been widely discussed in both the L1 and L2 literature. Recasts have been a central issue in debate on the role of negative evidence in language acquisition. Leeman points out that recasts are complex discourse structures that, in addition to negative feedback, also provide positive evidence (Pinker, 1989) and notes that the juxtaposition of the nontarget-like original and the target-like reformulation can also enhance the salience of target forms in the input. This may account for greater gains demonstrated by learners exposed experimentally to recasts than to other forms of positive evidence.

< previous page

page_58

next page >

< previous page

page_59

next page > Page 59

INSTRUCTIONS FOR STIMULATED RECALL Instructions for research participants: What we're going to do now is watch the video. We are interested in what you were thinking at the time you were talking about the pictures. We can hear what you were saying by looking at and listening to the video, but we don't know what you were thinking. So, what Id like you to do is tell me what you were thinking, what was in your mind at that time while you were talking to her. I'm going to put the remote control on the table here and you can pause the video any time that you want. So if you want to tell me something about what you were thinking, you can push pause. If I have a question about what you were thinking, then I will push pause and ask you to talk about that part of the video. Instructions for researcher collecting recall data: After reading the instructions to the participant, model stopping the video and asking a question. For example, choose a segment and stop the video. Ask your question. If they stop the video, listen to what they say. If you stop the video, ask something general like: What were you thinking here/at this point/ right then? Can you tell me what you were thinking at that point? I see you're laughing/looking confused/ something there, what were you thinking then? If the participant says "I don't remember," accept the comment and move on. "Fishing" for recall comments that were not immediately given by the participant will 'increase the likelihood that the recall comments will be based on what participants think now, some other memory/perception, or some flawed or biased recollection. Try not to focus or direct participants' answers beyond "what were you thinking then." It may also be useful to direct participants' attention to the original native speaker utterance or their own production by saying something like: Do you remember thinking anything when she repeated that? Can you remember what you were thinking when she said that/those words? Can you tell me what you thought when she said that? (text box continued on next page)

< previous page

page_59

next page >

< previous page

page_60

next page > Page 60

(text box continued from previous page) If the participant begins to talk over the tape, pause the tape and angle the remote control towards the participant so that she can release the pause when she is finished talking. Additionally, researchers should not give concrete reactions to participants' responses. Backchanelling or non-responses are preferable. For example, oh mhn I see uh-huh ok It is important to avoid extended responses or three-part exchanges, because providing feedback or input to learners may alter the nature of their recall comments. In short, try to be a "warm body," not a conversational partner. Fig. 4.1. Instructions for carrying out a stimulated recall of oral interaction (see Appendix A for full protocol). Leeman's (1999) study addressed the question of how recasts may lead to L2 development by isolating negative evidence and enhanced salience and comparing their effects to those of recasts. She designed an experiment to compare empirically four types of communicative interaction and their effects on learning noun-adjective agreement in Spanish as a foreign language. Seventy second-semester college learners (viewed as beginners) were randomly assigned to four treatment conditions. All participants engaged in communicative interaction individually with a native speaker, and the provision of negative evidence and enhanced salience was experimentally manipulated in the four groups as follows: 1) intensive recasts (i.e., negative evidence and enhanced salience), 2) negative evidence without enhanced salience, 3) enhanced salience of target structures without negative evidence, and 4) control (i.e., neither negative evidence nor enhanced salience). The effects of treatment were evaluated by means of oral pretests, immediate posttests, and delayed posttests consisting of picture-description tasks designed to elicit the target structure. Learners, in a pilot study Leeman carried out before her main data collection effort, participated in semistructured debriefing interviews while listening to themselves speaking on tapes of the treatment. This was to ad-

< previous page

page_60

next page >

< previous page

page_61

next page > Page 61

dress the concern that learners in the enhanced-salience group might also perceive the treatment they received as a form of correction, thereby confounding two of the variables Leeman's study sought to isolate. When salience is enhanced through word stress as in a recast where a learner says "the dog is over the street" and the response is "the dog is ON the street," learners may perceive the stress as corrective (i.e., negative) feedback, not (or possibly also) as simple positive evidence about the form. Results from Leeman's stimulated recall interviews demonstrated that learners in her enhanced-salience group interpreted the researcher's use of stress and intonation simply as a way to highlight the need for noun-adjective agreement rather than as a means of signalling that their original production was non-target-like. Based on the stimulated recalls in her pilot study, Leeman was able to claim that her enhanced-salience group did not receive negative evidence and that her study isolated variables. In viewing Leeman's protocol, it is important to note again that the instructions not only tell the researcher what to say to prompt the recall comments but also deal with what to say during the recall and after it, and that they include provisions for dealing with unexpected eventualities. In many experiments carried out by a research team or pair there is often a principal investigator or more experienced researcher, working with more junior colleagues or graduate students. Having a researcher who is experienced at carrying out stimulated recalls is an excellent way to approach the procedure. The importance of researcher training and instructions when using stimulated recall methodology cannot be overstressed. Stimulated recall procedures, in our experience, although somewhat challenging initially, become easier with practice and the procedure is best observed before the researcher attempts to carry it out. The stimulated recall procedure can go awry very easily. Participants, even when presented with carefully written and pilottested instructions, can often be confused by the procedure. The researcher needs to be able to put participants at ease, convey the impression that the participants are not being asked to do something very difficult or unnatural, and help participants provide recall comments without challenging their preconceived notions of appropriateness and without leading them. It can be a tall order. In some sense, carrying out a stimulated recall procedure is not unlike the sociolinguistic interview technique perfected by Labov (1972) and popularized during the 1970s. As researchers, we need to be aware of the great many variables and pitfalls when collecting stimulated recall data, and we may also need to practice collecting such data more than once. If researchers can work as part of a team, the procedure will be easier to handle. If circumstances dictate that a researcher utilize the

< previous page

page_61

next page >

< previous page

page_62

next page > Page 62

procedure for the first time without prior training, then we cannot overemphasize the importance of careful preparation. Being asked to introspect is particularly difficult for L2 learners from some cultural backgrounds. For example, in many cultures the teacher or native speaker is often considered the expert, and challenges to authority are not encouraged. Thus, when asked to introspect about their actions after a teacher's procedural direction in a classroom study, some speakers may be uncomfortable conveying to researchers that they did not follow a teacher's instruction, such as to read the passage silently, because they had already read it and decided it would be more helpful to prepare for the next class. Admitting to not following a direction, even with this sort of motivation, may be considered a direct challenge to the teacher's authority and could be difficult for some participants. This difficulty illustrates just one of the many challenges faced by researchers carrying out stimulated recalls. An experienced researcher who knows the participant pool and characteristics well might be better able to recognize which problems are likely to be due to stimulated recall instructions and which are related to participant characteristics. Clearly, the research questions dictate the nature and content of the instructions. In concluding this section, we provide general recommendations about instructions. RECOMMENDATIONS Instructions Instructions for both researchers and participants should be carefully drawn up and pilot tested. Instructions for participants should be recorded, read aloud by the researcher, or presented to the participants in written format where appropriate since standardization is important. Instructions for researchers should take into account as many eventualities as can be anticipated. For example, this may include what to say if the participant stops retrospecting and begins to analyze past actions/thoughts in the present. It may also include the scenario where (text box continued on next page)

< previous page

page_62

next page >

< previous page

page_63

next page > Page 63

(text box continued from previous page) the participant begins to talk, providing a recall comment while the tape is still playing* The participant may have forgotten to pause the tape. The researcher needs to stop the tape while facilitating the flow of the participant's speech and to let the participant know she can restart the tape, all without taking the floor from the participant. If the tape is not stopped, the clarity of the recall speech captured may not be good enough for transcription. Instructions for researchers should include information about potential effects of participant characteristics on the recalls and how to minimize effects where possible. Procedures such as the selection of video or written segments as topics for recall comments should be modeled by researchers for participants. When possible, researchers should observe the stimulated recall procedure being carried out before participating in one.** * A relatively frequent scenario in stimulated recall involves participants talking over a tape, even when they have been directed about how to turn on/off the pause button. ** A complete transcript of a stimulated recall procedure involving oral interaction appears in Appendix C. Data Analysis Interrater Reliability As with most data, there are several steps that need to be considered when analysis of stimulated recall data is carried out. These steps include transcription, coding, and description of data, as well as data analysis. Analysis of data obtained through stimulated recall can be qualitative, quantitative, or a combination of both. As with most data, the issue of interrater reliability2 must be considered in relation to transcription and coding of data obtained through stimulated recall. One of the most important issues when carrying out tests for interrater reliability with stimulated recall data is the objectivity of the raters. Often, those who collect the data are the researchers, and usually the re2 For basic information about interrater reliability coefficients, see Hatch and Lazaraton. (1991) and Seliger and Shohamy (1989).

< previous page

page_63

next page >

< previous page

page_64

next page > Page 64

searchers transcribe, code and rate, or analyze the data as well. However, this high level of involvement by the researcher may cause problems with the analysis of such data. There is often a relatively high level of interpretation in relation to data obtained through stimulated recall, and interrater reliability is likely to be affected. Consider, for example, the case found in a study conducted by Mackey and Gum (1997): A participant had taken several turns explaining to the researcher that she ignored the teacher's request for students to read a passage because she was bored by the passage and felt the teacher was not in tune with her L2 learning requirements at that time. The recall data for this study included audiotaped interviews with students who, as a stimulus for the recall interview, were shown a videotape of themselves in class where one of the researchers was also their teacher. One goal in collecting these data was to discover more about students' motivations for not following 'instructions in class and whether teachers' status as a novice or institutionally experienced teacher could be, at least in part, a determining factor in whether or not a student followed a teacher's directions. An extract from a transcript of one of the interviews between the student in question and a researcher follows. Example 3 (From data collected by Mackey & Gum, 1997) Can you tell me why, at that time, what you were thinking while you Researcher:wrote in your notebook? What was going through your head? On the tape we can see that you looked at the teacher while she said, "please read the passage on page 36. Take ten to read the passage," on the tape we can see you wrote in your book for ten minutes or so. What were you thinking? I already read it last time Student: Oh. What were you thinking? Researcher: You know, I thought I read it. I don't need to read it again. Student: So that's what you were thinking Researcher: I thought as I already read it so I'll write out my vocabulary for X's Student: (teacher's name removed) class after this. Then I am making a good use of my time. An independent rater not present during the interaction who read the transcript and rated this interaction was given three categories for coding,

< previous page

page_64

next page >

< previous page

page_65

next page > Page 65

which were predetermined by the researchers. These categories were later determined to be somewhat problematic, and the analysis was not pursued further. However, for the present purpose of illustrating interrater reliability issues in recall data, they work well. A. Participant did not agree with teacher's instruction (e.g., student believed that the use of class time for silent reading was not an effective use of time). B. Participant did not disagree with the instruction but did not want to comply with instruction (for reasons such as student preferred to read another passage or do other activity). C. Participant agreed with teacher's instruction and carried out activity. After reading the transcript, the independent rater coded the students' response as category B. From the transcript it seemed to this rater as though the learner simply preferred to work on her vocabulary, an alternative task. However, if the rater had been present during the interview or had been able to watch a videotape of the interview rather than read a transcript, the rater would have heard backchannels and intonational cues, for example, a strong emphasis on the word already, and drawn-out sighs of displeasure that accompanied this student's verbalization. The student also made a nonverbal hand gesture that could be interpreted as related to the wasted time or effort she attributed to rereading passages. She also spelled out an L for loser with her thumb and index finger, this gesture was not indicated on the transcript of the recall but was recognized by the nonindependent rater-researcher because she had heard other students discussing the gesture after a recent movie they had all seen. Together, these behaviors led the rater-researcher, who watched the recall stimulus and took part in the interview as well as transcribed and coded it, to rate the interaction as category A. This lack of agreement can be seen as an example of inadequate transcription because not indicating nonverbal and backchannelling behavior results in an incomplete transcript. It is also problematic in that the coding system did not allow overlap between categories. Most important, however, the lack of agreement between raters is related to the complex nature of the recall procedure. In this case, the interrater reliability was affected because one rater brought prior knowledge of the recall stimulus to the rating process and the other rater relied only on the transcription of the recall comments. Thus, the incomplete nature of the transcription and the lack of representation of the stimulus combined to produce different ratings. In some cases, researchers want both raters to be in possession of all

< previous page

page_65

next page >

< previous page

page_66

next page > Page 66

potentially influencing information. In other cases, only what was present in the recall comments should be rated. Obviously, the nature of the research question will determine the content of the data to be rated. Researchers who were participants 'in the stimulus, the original activity to be recalled, or who took part in the recall interview with the participant often have extra insight into the learners' verbalizations. In Example 3, the researcher's perceptions about the data were the accurate ones, as seen in the nonverbal cues and the data recalled. However, the main motivation for assessing interrater rehability is to ensure that researchers do not read too much into the interaction. This can happen because researchers may become over-influenced by their research questions and hypotheses and therefore by their expectations about the data. Tests of interrater reliability allow us to have some confidence that what we as researchers see in the data can also be seen by independent raters, or in the case of intrarater reliability, that we as raters are rating each episode consistently. Thus, after carrying out a stimulated recall, we generally recommend finding independent raters who were not participants in the original event to be recalled and who were also not participants in the subsequent recall, or, if this is not possible, finding raters who participated in the same events, either the original or the recall. Independent third-party raters will need to be trained 'in the categories for rating and given the information they need about the stimulated recall procedure and events being recalled. In some designs, raters will need to see the stimulus as they rate the recalls; in other designs this may influence their perceptions about the recall in an undesirable and nonobjective way. In this latter instance, raters will need to rate the recall in isolation. Thus, the data for the interrater reliability test need to be chosen and ordered carefully. All this requires rather more work than usually necessary in many L2 studies. Generally for interrater reliability tests, L2 researchers may simply pull out a random subset of the data, write up a sheet on coding categories, and have the raters turn in their ratings. A simple percentage agreement check on the basis of the researchers' coding can be calculated. To illustrate what we mean by a careful interrater reliability check, we take another extended example. In the study of oral interaction carried out by Mackey, Gass, and McDonough (in press) and described earlier in this book, learners carried out tasks and interacted with native speakers. Non-target-like utterances (or put more simply, learner errors) that triggered feedback episodes were categorized as lexical, phonological, morphosyntactic, or semantic errors. Native speaker feedback provided was also categorized (and in almost all cases was based on the error type such that a mor-

< previous page

page_66

next page >

< previous page

page_67

next page > Page 67

phosyntactic error generated morphosyntactic feedback; occasionally, multierror sequences occurred and generated either single or multierror feedback, again almost always agreeing with one of the error types). Next, learners' perceptions about the feedback they received were categorized in the same way. The aim of the study was to see if learners were accurate in their perceptions. For example, was morphosyntactic feedback perceive as such by learners? In relation to the interrater reliability for this study, it may be obvious that the classification of the initial error and the feedback provided should not be allowed to influence an independent rater's classification of the learners' perceptions. Thus, in Example 4, the learner makes a phonological error, pronouncing the word [sterz] 'stairs' as [staz]. The native speaker provides implicit negative feedback, giving a recast of the correct pronunciation of the word. The error was phonological, and the feedback provided to the learner was also phonological. The stimulated recall comment showed that the learner also perceived the feedback as phonological; the learner even mentions pronunciation. An alternative can be seen in Example 5, where the error was morphosyntactic, the feedback was morphosyntactic, yet the recall comment was focused on meaning. In this case, how would independent raters classify the recall? If they saw the interaction preceding it, they might classify it as morphosyntactic, as it was intended by the native speaker. However, the rater was supposed to be classifying the learner's perception about the recall, and the learner, although in possession of the knowledge of what she had originally said, may or may not have related this to the researcher's comment. So, her feedback should have been classified as semantic, not morphosyntactic. Because they are learners of English, participants may differ from researchers in what they understand about what a native speaker has said. Thus, it is crucially important that just because the rater may understand that the feedback provided by the classification of the native speaker was intended to be morphosyntactic, the rater's understanding should not influence the rating about the learner's perception. Mackey, Gass, and McDonough solved this problem by ordering the ratings. Two independent non-researcher raters both rated half the data. However, they rated the interactions and the recalls separately so that they did not see the interactional feedback and were not influenced by the native speaker's feedback type. The learners' recalls and the interactions that preceded them were rated independently. A reliability check was carried out between the two independent raters and between their ratings and the ratings of the researchers who carried out the study.

< previous page

page_67

next page >

< previous page

page_68

next page > Page 68

Example 4 (From Mackey, Gass, & McDonough, in press) NNS:

[staz], [staz]

NS:

Stars? Oh, stairs?

NNS:

Yeah

Recall:

I know I can't pronounce this word. Oh, no, I need to tell her this word. Inside me, I was laughing, oh my god, just my luck.

Example 5 (From Mackey, Gass, & McDonough, in press) NNS:

Have a wings.

NS:

The bird has wings?

NNS:

Yeah.

Recall:

Maybe she is not sure which bird I saw. She wonder if my bird has wings, she ask me ''the bird in your picture has wings?" I like her question because I think sure, all birds have a wings.

In concluding this section on reliability for coding and and analyzing stimulated recall data, we make the recommendations shown below. RECOMMENDATIONS Interrater reliability Use objective non-researcher raters for checks wherever possible Carefully construct analytical categories, checking subsets of the data with other researchers before coding all data to avoid carrying out large amounts of high inference coding that cannot be replicated. Decide on nature, content, and order/presentation of data for intercoder checks, paying attention to the potential effects of the stimulus along with recalled comments that need to be rated. Having addressed the issue of reliability of the coding of stimulated recall data, we now turn to the issue of data coding. The specification of what to code is, of course, a function of the specific research questions addressed by the study. Nevertheless, there are some general issues related to the coding of stimulated recall data, and it may be helpful for researchers to be aware of these prior to developing a coding system.

< previous page

page_68

next page >

< previous page

page_69

next page > Page 69

An Extended Example of Rater Training In Fig. 4.2a, we present the training protocol that was used for the Mackey, Gass, and McDonough (in press) study to provide some idea of the extent of training that may be necessary. The training sessions took place over a 3-day period, with extensive explanations of concepts and categories as well as numerous opportunities to view examples. Figure 4.2b is an example of the coding sheet used to code the stimulated recall data in the study. Raters were given preselected parts of the stimulated recall interaction and had to code the stimulated recall. Figure 4.2c shows an example of the coding sheet used for the interaction part of the study. When coding the interaction, raters were not given preselected episodes to comment on; rather, they had to select the episodes, transcribe that portion of the tape, and then code the episodes. Figure 4.2d is an example of a coding sheet for the stimulated recall comments. Figure 4.2e shows the rater and task schedule that was developed as an organizational tool to ensure that all tapes were evaluated. Figure 4.2f is a training schedule to ensure that all interactions and stimulated recall were evaluated and that raters had an equal number of evaluations to do. Finally, Fig. 4.2g is the schedule used for interrater reliability. SAMPLE TRAINING PROTOCOL Day One (20 minutes)

Interaction Training Review What is a non-targetlike utterance? Inside the bookshelf, there are books (choice of preposition) The man walk a dog (S-V agreement) There are two cup on the table (plural) I can see a /crowd/ in the sky (pronunciation) The cat is fixing at me (lexical choice) What is interaction? What is negotiation? What is implicit negative feedback (INF)? the third one is a /dawk/ NNS: a what? NS: a dog NNS: What is a recast? he also standing NNS: he's also standing? NS:

(table continued on next page)

< previous page

page_69

next page >

< previous page

page_70

next page > Page 70

(table continued from previous page) yeah NNS: What we aren't looking at (but does occur in the data): 1) explicit explanation of lexical items What color are his fins? NS: Huh? NNS: Fins are like, you know people have arms but fish have fins NS: 2) explicit feedback The cat is playing with wool NNS: Not wool, you mean yarn NS: 3) provision of form It is in front of the fire . . .fire . . .fire . . NNS: place NS: fireplace NNS: (30 minutes)Watch Training Video 1 Trainers use remote, stop and pause at relevant examples of what is INF and what is not, stop and talk throughout video Watch Training Video 2. Watch 35 minutes, take notes individually, no need (30 to write down complete interaction. Pause and discuss. Continue with another minutes)35 minutes. (45 minutes)Rating Session 1 Bev: 1 interaction; Lea: 1 interaction (2 hours) Trainers review ratings, discuss, further training where necessary Day 2 (40 minutes)Stimulated Recall Training What is stimulated recall? Watch video of AB doing recall with K What was the prompt? Classify comments based on content 1) Lexical: specific comments about a known or unknown word, including provision of a synonym and comments about a synonym, for example I am not sure abota feminine; I have heard this word before 2) Semantic: general comments about communicating meaning, creating understanding, being unable to express an intended meaning, providing more detail or elaboration, for example she still doesn't understand; she wants to know more details; we move to focus on bookshelf

(table continued on next page)

< previous page

page_70

next page >

< previous page

page_71

next page > Page 71

(table continued from previous page) 3) Phonological: specific comments about pronunciation, for example I think my pronunciation sounds strange; I have a problem with the /l/ sound 4) Morphosyntax: comments about sentence formation and structure or word order, comments on specific aspects such as S-V agreement, tense, for example because the grammar is not correct; I said people is fishing but I have to say people are fishing 5) Ø The participant has nothing to say, for example Um, no nothing really; It was OK; No, I don't remember anything 6) ?: The participant made some comments about specific content, but the rater cannot classify those comments into a particular category, for example my answer is not suitable; I was thinking that the room looks like my dorm room Potential Problems, resolution mechanism Practice Rating JMO, JFO Rate one SR each (2hours 20 minutes) Rating Session 2 Bev: 2 recalls, 1 interaction Lea: 1 recall, 2 interactions Day 3 (3 hours) Rating Session 3 Bev: 2 recalls, 3 interactions Lea: 3 recalls, 2 interactions Fig. 4.2a. Sample training protocol.

< previous page

page_71

next page >

< previous page

next page >

page_72

Page 72 SAMPLE CODING SHEET FOR STIMULATED RECALL COMMENTS You will be reading some comments that the learner made while the videotape was paused during the stimulated recall procedure. The learners were explaining what they were thinking while the interaction was going on. After you read each comment by the learner, classify the content of that comment as one of the following (see explanations on the training sheet): lexical

L

phonological

P

morpho-syntax

MS

semantic

S

no content

Ø

unable to classify

?

Transcription conventions: underline = emphasis was placed on word by participant. #Stimulated Recall Comment

Type

Sometimes I couldn't catch opponent pronounce but sometimes I can guess 1opponent question but at this time I couldn't understand It's difficult for me to explain about weather. I know it's rain, it's snow, or it's fine, sometimes for now, it's not snow, it's not rain, how can I say? Yeah, I had 2better study about weather's vocabulary 3I just said right side but on THE right hand side, is it correct? 4No, nothing, it's ok 5Oh, it's OK 6Sometimes confusing left and right 7It's ok (table continued on next page)

< previous page

page_72

next page >

< previous page

next page >

page_73

Page 73 (table continued from previous page) 8 At that time I couldn't explain about the road. How can I say this? The boy bring one stick, uh this picture also the a stick, but it's ah different. She said he doesn't bring a stick maybe. At that time she explained the boy doesn't have 9 a stick 10Uh, I have better explain more detail 11Yeah, at that time it's so difficult to explain about between something to something 12At that time I couldn't catch total 13I wrong to explain. I just I say between the trees but I have to say between two trees 14My pronounce is not clearly, especially /r/ and /l/ 15We discuss about more detail, we focus on boots Fig. 4.2b. Sample stimulated recall coding sheet. Sample Coding Sheets for Interaction Episodes When you watch the video, pause every time the NS gives implicit negative feedback to the NNS. Write down what the NNS said and what the NS said in response. Then classify the type of error in the NNS utterance as one of the following: semantic

(problem with meaning)

S

lexical

(problem with lexical choice)

L

phonological

(problem with pronunciation)

P

morphosyntactic

(problem with grammar)

MS

#

NNS utterance

NS Response

Type

Fig. 4.2c. Sample coding sheet for interaction episodes.

< previous page

page_73

next page >

< previous page

next page >

page_74

Page 74 # Stimulated Recall Comment Type

Type

I am thinking where is in front of. In front of means this way and I was 1 thinking this place was in front of. I am thinking I could it's in front of I was confused about below because I don't usually use above, below, I use 2 top 3 I can not easily, right left, if you night side, I was thinking which one 4 I was thinking the name of thing, I don't know the word I was very happy because I didn't know fireplace and I was explaining it look 5 like and she know it 6 In Japanese I call cushion, but I didn't know English I didn't know couch and chair different and when I when I said two couches she didn't understand what I said and I thought I don't know the meaning and I asked how couch mean and she explained. I understood, I felt comfortable 7 because there I didn't understand and I couldn't understand 8 was thinking why she didn't understand 9 My my Enalish is not understandable to native speaker, same as always I didn't know what cage is, I was going to explain it look like jail but I didn't, 10she understood Sounds like John, John is my content teacher, the man in the picture looks like 11John 12I said wrong vocabulary I was thinking how I could explain because I don't know how to form look 13with girl, but I didn't explain and then she said She asked I have or not, I could explain none of them because it was on the 14floor another one is not on the floor so I couldn't explain where it is (table continued on next page)

< previous page

page_74

next page >

< previous page

next page >

page_75

Page 75 (table continued from previous page) 15I said couch but my is not couch so I thought I made a mistake When I'm talking to native speaker and the person doesn't understand my English, I 16always say why this guy doesn't understand me I said of cheese but I have to say a piece of cheese, I thought next time I was going 17to say a piece of cheese 18Because new vocabulary Fig. 4.2d. Sample coding sheet for stimulated recall comments. Schedule for Raters and Tasks Day Rater Learner Task Day 1

Day 2

Bev

1.JF1

Int

Lea

1.MM

Int

Bev

1. MM

SR

2. FF

SR

3. JM2

Int

4. FM1

SR

1. JF1

SR

2. FF

SR

3. JM2

SR

4. FM1

SR

5. TM

SR

6. JM1

SR

7. KM

SR

8. FM2

SR

9. JF2

SR

10. MM

SR

1.TM

Int

2. JM1

SR

3. KM

Int

4. FM2

SR

5. JF2

Int

Lea

Day 3

Bev

(table continued on next page)

< previous page

page_75

next page >

< previous page

page_76

next page > Page 76

(table continued from previous page) Day Rater Learner Task Lea

1. FF

Int

2. FM1

Int

3. KM

Int

4. FM2

Int

5. JF2

Int

6. MM

Int

7. JM2

Int

8. TM

Int

9. JM1 Int Fig. 4.2e. Rater and task schedule. Key: First letter represents Ll (Thai, Japanese, Mandarin, Korean, French), second letter represents gender, SR = Stimulated Recall Coding, Int = Interaction Coding Sample Training Schedule Learner Interaction Rater SR Rater MM

Lea

Bev, Lea

JF1

Bev, Lea

Lea

FF

Lea

Bev, Lea

JM2

Bev, Lea

Lea

FM1

Lea

Bev, Lea

TM

Bev, Lea

Lea

JM1

Lea

Bev, Lea

KM

Bev, Lea

Lea

FM2

Lea

Bev, Lea

JF2 Bev, Lea Lea Fig. 4.2f. Training schedule. Sample Schedule for Interrater Reliability Stimulated Recall Coding Interaction Coding MM

JM

FF

KM

FM1

JF1

JM1

JF2

FM2

JM2

Fig. 4.2g. Schedule for interrater reliability.

< previous page

page_76

next page >

< previous page

page_77

next page > Page 77

Data Layout and Coding Coding sheets for stimulated recall differ from many other types of coding sheets in that one must keep track of two different events. In Figs. 4.3 and 4.4 we present a coding sheet from Hawkins (1985). These coding sheets display the range of relevant coding. As discussed in chapter 3, one purpose of Hawkins' study was to compare the retrospective commentaries by two participants in a conversation, one native speaker and one learner. Hence, Figs. 4.3 and 4.4 are to be read together. The first represents the conversation from the NS point of view and the second the same conversation from the learner's point of view. In the leftmost column is the portion of the data that the stimulated recall focused on. In the second column is the mechanism used (e.g., repetition, comprehension check). The third column provides the reader with information about the topic category, that is, whether it is a new mention (e.g., topic nomination), topic identification (e.g., mention of object function), and so forth. The fourth column identifies the conclusion (AR = appropriate response; C = comprehension) concerning the research project. The fifth column shows the benefit that a particular utterance might have for the possibility of NNS comprehension, and the sixth column identifies the strategy that the NNS used to indicate comprehension or lack of comprehension and the extent to which this strategy affected the NS's realization of the NNS's comprehension or lack thereof. Finally, in the last column are the retrospective comments, carefully numbered and lettered to match the original transcript. Figures 4.5 and 4.6 are from two different NS-NNS pairs and further exemplify the richness of the data that Hawkins (1985) described. In Figs. 4.7 and 4.8 are two examples of a second coding chart. The first example shows an earlier version of the second example. These sheets were developed for the Mackey, Gass, and McDonough (in press) study discussed earlier. We present these two coding sheets as a way of elucidating the problems that an inappropriate coding sheet can bring. The researchers wanted to determine the extent to which learners recognized the purpose of implicit negative feedback. Figure 4.7 is the preliminary version, and Fig. 4.8 is the actual version used in the study. In order to address the research question, the researchers had to know what the original error type was (see the previous section on interrater reliability). In the first sheet, the researcher failed to anticipate the need for coding the type of error and therefore did not leave a column for that information. This was recognized during the coding process, and the sheet was revised.

< previous page

page_77

next page >

< previous page

page_78

next page > Page 78

Fig. 4.3. Transcript from Hawkins (1985).

< previous page

page_78

next page >

< previous page

page_79

next page > Page 79

Fig. 4.4. Transcript from Hawkins (1985).

< previous page

page_79

next page >

< previous page

page_80

next page > Page 80

Fig. 4.5. Transcript from Hawkins (1985).

< previous page

page_80

next page >

< previous page

page_81

next page > Page 81

Fig. 4.6. Transcript from Hawkins (1985).

< previous page

page_81

next page >

< previous page

next page >

page_82

Page 82 Targeted Error L Ph Ms

Feedback

Learner Response

Sem

Stimulated Recall Lex.

Phon.

M/S

Sem

Problems: Four choices for targeted error is not efficient. Need one column. Need space to classify each stimulated recall error in addition to space to transcribe each one. No column for noting initiator of episode. No column for noting learner uptake at the time of the interaction. No space for subject ID or identification of session. Fig. 4.7. Preliminary (flawed) version of coding sheet.

< previous page

page_82

next page >

< previous page

page_83

next page > Page 83

Subject ID/Name:_______________ Date of data collection:__________ Error Targeted Learner Stimulated Initiated by Participant SR Type Error FeedbackResponse Recall or Researcher? Type Uptake Remaining Problems: Is the error column (and category) for the transcription of the error or the classification of the error? Need two columns. How is uptake classified? Again, is response column for transcription of response or classification of uptake or both? Unclear if this coding sheet can account for more than one turn or one sequence involving multiple turns. Fig. 4.8. Final version of coding sheet.

< previous page

page_83

next page >

< previous page

page_84

next page > Page 84

The second column in the revised coding sheet, for reasons of space, eliminates the four choices of error type (i.e., lexical, phonological, morphosyntactic, semantic), allowing the researcher to write the full example. The other major change is the addition of the SR type column (and the concomitant reduction of the four columns) rather than trying to incorporate so much information into a single set of columns, as in the original. This was an important change because at the coding stage, the researchers needed to obtain interrater reliability on precisely these categories. Had they been written in on the coding sheet, it would have been more difficult to obtain interrater reliability using these sheets. For the Mackey, Gass, and McDonough (in press) study, the final column was added to address a specific research question. Other columns that reflect specific research hypotheses can be included, as was the case with the Hawkins study. In concluding this section, we present some recommendations for coding sheets. RECOMMENDATIONS Coding Ensure that coding sheet allows categorization of types of data required to address research questions. Design sheet so that there is easy visual access to simultaneous events. Order sheet so that chronology of non-simultaneous events is easy to follow. Procedural Pitfalls In this section we turn to the many interesting potential problems involved in collecting and analyzing stimulated recall data. First, we deal with the issue of timing and the problems that incorrectly estimating timing can cause. Next, we deal with the specific questions that can be addressed by stimulated recall. Finally, the language of the recall session is discussed. Timing One potential pitfall in conducting stimulated recalls relates to how much time the researcher allocates to the procedure. There are many reasons to be careful about under or overestimating time. For example, for most de-

< previous page

page_84

next page >

< previous page

page_85

next page > Page 85

signs it is desirable to standardize times as much as possible amongst participants. An accurate estimate of time will also allow smooth scheduling and can assist the researcher in avoiding fatiguing the participants unnecessarily. When conducting stimulated recall procedures, it is important to accurately estimate from the outset the time for at least three parts of the procedure. These are described in the following sections. The Approximate Length of the Recall Support In this section we refer to the amount of time allocated during the stimulated recall procedure for the support system, or stimulus, that the researcher uses to aid participants in accurately recalling the event. Recall support systems can include audiotapes or videotapes of the participant, recorded as he or she was carrying out the event. For example, in a stimulated recall of oral interaction, participants may listen to audiotapes of themselves and their partner(s) while they were engaged in the oral interaction. In a stimulated recall of writing processes, the participant may be shown an essay or paper that he or she had rewritten, possibly alongside an earlier draft. The time needed to present the recall support needs to be carefully estimated in calculating the time for the stimulated recall procedure. For example, if a videotape of the event to be recalled is to be shown to participants, the time that the replay will take needs to be calculated and added to the time allocated for the recall procedure. In some designs, the videotape may be replayed in its entirety; in others only segments will be played. In some cases the researcher will control and select the segments and pause the videotape; in other cases the participants themselves may be asked to select and replay segments. Sometimes the participants may replay segments more than once. In some designs participants may ask questions of the researcher about the segments, or may replay the segments as often as they wish. Not all these factors for example, answers to participant questions are completely under the researcher's control, but they need to at least be recognized as time-consuming possibilities. It is often the case with stimulated recall procedures that the researcher underestimates the amount of time the recall support takes. As discussed earlier in this book, the sooner after the event the stimulated recall takes place, the more likely it is that uncorrupted memory structures will be accessed. However, participant fatigue is clearly an issue. Using the videotape of oral interaction, Example 6 is a time estimate for a particular procedure, followed by the actual time taken for the procedure.

< previous page

page_85

next page >

< previous page

page_86

next page > Page 86

Example 6 Research goal:

Conduct a stimulated recall procedure with two learners who carried out oral interaction tasks in a learner-learner dyad.

Initial time estimates: A.

Task-based interaction: 30 minutes.

B.

Stimulated recall procedure with learner 1: 30 minutes.

C.

Stimulated recall procedure with learner 2: 30 minutes. Total researcher time: 90 minutes Time for each learner: 60 minutes

Thus, the whole procedure was estimated to take one hour, with half an hour of down time or break for one learner.3 We compare this original estimate with the actual time taken. Actual time taken: A.

Set-up time: Researcher set up video recorder for the interaction together with backup audio recorder, prepared tasks, had consent forms read and signed by participants, explained initial instructions: 17 minutes. Learners present for 10 of these minutes.

B.

Oral interaction: 28 minutes.

C.

Set-up time: Researcher rewound videotape for replay. Segments were to be selected for playback; researcher reviewed videotape for timestamps and set video to first segment to be replayed (the researcher did not need to watch whole tape as she was present behind video camera, making notes on sections to replay). Researcher set up second video camera to record the stimulated recall: 19 minutes.

D.

Recall procedure. 2 minutes for 'instructions on the recall, 1 minute for Q-A on recall, 11 minutes for recall replays, researcher initiated; 6 minutes for recall replays, learner initiated, 2 minutes for learner questions about specific aspects of the recall support video; 18 minutes for the participants' comments in the stimulated recall; 1 minute for checking of microphones part way: 41 minutes. Total researcher time: 146 minutes Time for each learner: 98 minutes

3 An alternative in terms of time could be the scenario in which two videotapes of the interaction were simultaneously dubbed and two native speaker researchers carried out the recalls simultaneously. This avoids the 30 minutes downtime for one learner. Unfortunately, such resources are beyond the reach of many researchers.

< previous page

page_86

next page >

< previous page

page_87

next page > Page 87

The whole procedure took 2.4 hours for the researcher and 1.6 hours for each learner, with 48 minutes of downtime for one learner. The actual time taken turned out to be a little more than one and a half times as long as the initial time estimate. Learners' schedules were affected. One learner missed lunch, and both missed a part of their ESL class. Both learners were much more taxed than the researchers had anticipated. In this scenario, pilot testing turned out to be essential in accurately estimating time. Allocating Time for the Recall Procedure It is also important to accurately estimate the time that the recall procedurethat is, the researcher-participant interaction minus replayed support cueswill take. As seen in Example 6, the participants' comments took only 18 minutes, but questions about the procedure, questions about the support structure, and opportunities for the participants to access the support were all factors in the time taken for the procedure. This time can also vary from participant to participant. Some participants may be much more forthcoming than others. All the caveats that apply to oral interviews (see Hillway, 1969, pp. 3334) also apply to stimulated recall procedures. Stimulated recalls may also be carried out through the use of questionnaires, with the recalls being written. Constraints such as writing speed and space allocated on the form will have an effect in such circumstances. Because writing takes longer than speaking, the interval between the event and the recall is increased and the validity of the procedure becomes more questionable. In terms of being prepared for the unexpected with reference to the effect individual participant variables have on time estimates, it may be helpful to consider that while conducting stimulated recall procedures, each of the authors of this book has had unfamiliar and unexpected experiences with participants. One of the authors experienced a situation in which a participant opted part way through not to continue with the experiment and left the room abruptly, after stating that he was uncomfortable with the procedure. Fortunately, this participant had clearly understood and internalized the information on his consent form, knew that he had the right to leave, and exercised this right. The other author experienced the case where a procedure, which had ranged between 18 and 30 minutes for each of 10 participants, lasted only 7 minutes. The participant was completely uncooperative and would only say, ''I don't know" or shrug in response to all the recall cues. He also gave a negative response to two direct questions about whether he wanted to stop. This participant was asked three times if he understood the instructions, and the procedure was also modeled for him. He responded positively to those queries about his comprehension of what was

< previous page

page_87

next page >

< previous page

page_88

next page > Page 88

expected or required. He simply seemed to dislike questions about his thought processes or previous verbalizations and was not inhibited about demonstrating this dislike. In our experience, stimulated recall interviews that we have carried out have revealed some of the most interesting data we have seen. However, the episodes described here (that resulted in unusable data) demonstrate the vulnerability of the methodology. To summarize, therefore, although it is desirable to be able to estimate time for a stimulated recall procedure. Factors such as individual learner characteristics, whether the recall is being carried out in the L1 or the L2, who is controlling the interview, and the degree of structure to the questions may all affect the length of the procedure. Allocating Time for Set Up and Equipment Stimulated recall procedures are often among the most time consuming in L2 research in terms of equipment setup and preparation. As in Example 6, the event to be recalled or the product investigated is usually recorded or taped in some way. This recording or product-preservation has been made in a way that can be shown to the learner as a stimulus. It is important to recognize that the stimulated recall of the participant also needs to be recorded in some way. The degree of structure necessary for the support needs to be anticipated when planning the procedure. Reviews of tapes, drafts, or other stimuli and the giving of instructions and perhaps modeling of the procedure all need to be factored into the time calculations. Verbalization Another issue relating to timing in stimulated recall data is the issue of how long a researcher allocates for verbalization on a particular topic during the recall procedure. This issue often comes up in relation to recalls about problem-solving activities. For example, a participant may immediately hit on and describe the mental process that happened to be the focus of the study. Researchers need to take care when this happens that they do not cut off the learner's next verbalization and move immediately to the next episode or problem for which they elicit stimulated recall data. Such immediate moving on could be perceived by the learner as implicit feedback about what the researchers were looking for, especially if the learner had not been stopped in previous episodes. The learner might realize that the previous answer was exactly what the researcher was looking for and modify future answers. Alternatively, a learner might perceive being cut off as a cue that the last response was uninteresting. Clearly, equal time and consistent action by the researcher are both crucial.

< previous page

page_88

next page >

< previous page

page_89

next page > Page 89

Wherever possible, participants in a study should receive similar amounts of time for all topics or episodes being recalled. Consistency across time may not always be practical. As Chi (1997) pointed out, if standard times for recall are not possible, then it may be necessary to analyze the recalls consistently: If one person were cut off after one or two turns in relation to one episode, then all of that learner's episodes could be analyzed up to a similar cutoff point. Also of interest in relation to time is the issue of how much verbalization each learner provides in a standard time frame. Obviously, some individuals are more talkative than others. This is a difficult issue to resolve. On one hand, many researchers have pointed out that verbal protocols should be as full and complete as possible. On the other hand, controlling verbosity is not always possible. Generally, the quality or content, rather than the quantity, of the recall is important for the analysis, but if some recalls are five times the length of others, quality may be affected. It is not usually possible to manipulate individual characteristics such as verbosity. Baseline data for each participant, showing how verbose he or she is in relation to other topics or tasks, is desirable. Then, more wordy recalls for some people may be seen as a function of style. In practice, however, we have found no examples of studies in the literature that have collected such baseline data. Recall Questions This section focuses on the most serious of difficulties, the effects of researcher on-line questions and responses and the potential for compromising the procedure. Given that the goal of stimulated recall is to tap learners' thought processes while they were performing a particular task, the method itself will have no validity unless one can be reasonably sure that accurate recall in fact is taking place. As mentioned in previous chapters, the premise behind stimulated recall is that an event that has taken place is being recalled through the prompt and that the prompt itself helps to ensure that accessible and accurate memory structures are brought into focus and recalled. That is, when a participant is reminded of an event by means of video, audio, or written document accompanied by a researcher's prompt, the event itself and the thoughts that occurred during the event are vividly recalled. One potential pitfall relates to the time frame between the event itself and the stimulated recall. As is obvious from findings of studies of the effects of reporting after timed intervals, one's memory becomes less accurate as time passes. For example, Garner (1988) reported that in an experiment involving participants retrospecting about strategic activity immediately af-

< previous page

page_89

next page >

< previous page

page_90

next page > Page 90

ter task completion and two days later, significantly fewer cognitive events were recalled in the protocols of delayed-report participants' protocols than in the protocols given by same-day report of participants. Thus, it is ideal to have the recall session immediately after the participant performs the language task. As discussed previously, Bloom (1954), in his early studies developing stimulated recall methodology, found as high as 95% accurate recall within two days of the original event, but the accuracy rate declined to about 65% two weeks later. Hence, recall should take place as soon as possible after the original task. There are other less obvious difficulties, however. We present excerpts from a recall session taken from the study by Mackey, Gass, and McDonough (in press). The data reported in Examples 11-15 were thrown out for reasons that we make clear. The participant in these excerpts was a Korean student studying English in the United States. She had just completed a spot-thedifference-task with a native speaker. The native speaker had been instructed to provide indirect negative feedback (including recasts) to the learner when errors were made. The purpose of the study was to determine the extent to which this feedback was perceived as feedback by learners. Types of feedback provided by the native speaker in the Mackey, Gass, and McDonough are provided in Examples 710. Types of feedback provided to the Korean learner and the interviewer's recall prompts are provided in Examples 11-17. Example 7 NNS:

there is a there is a cat. He is a black.

NS:

he is a what?

Example 8 NNS:

the rear, rear [vleks]

NS:

the rear what?

Example 9 NNS:

he also standing

NS:

he's standing?

Example 10 NNS:

people enjoying beach

NS:

people are enjoying the beach

< previous page

page_90

next page >

< previous page

page_91

next page > Page 91

All sessions in the Mackey, Gass, and McDonough study were videotaped and immediately followed by the stimulated recall using the videotape as a prompt. In Examples 11-15, we present the interviewer's prompts as she tried to encourage the Korean learner to recall what she was thinking at the time of the event. Two native speakers participated in the research. Native speaker 1 performed the original task; native speaker 2, who had been operating the video during the original task and who had been paying careful attention to the feedback interaction, carried out the recall session. The crucial episodes are given in bold typeface. Example 11 NNS NS-1 NNS NS-2 NNS Response Original Feedback ResponseStimulated Comments Recall Prompts He just look at-look at me He's what Look at me He's looking at you? Yeah Why was she saying that? It's a um, I cannot cannot cannot answer clearly What do you mean clearly? I have to answer, I have to answer correct because the word is so weird You are saying the word is weird?

< previous page

page_91

next page >

< previous page

page_92

next page > Page 92

The emphasis in stimulated recall must be on the thought processes during the event itself, but the interviewer moved away from that and was focusing on the here and now of the stimulated recall session, saying, "What do you mean clearly?" and later "You are saying the word is weird?" without making it clear that the non-native speaker was supposed to be recalling what she was thinking at the time of the event itself. This entire dataset had to be eliminated from the final analysis of this study given that it was not clear what the learner was actually describing in the recall session. Example 12 NNS Original Comments And the [kaler *] color is yel-yellow

NS-1 Feedback

NNS NS-2 Stimulated Response Recall Prompts

NNS Response

The what is yellow Yellow the color? The color is yellow? What? What?

My voice is maybe low

You're speaking too quietly? Yeah In Example 12, the interviewer should have been attempting to determine if the non-native speaker was aware of why the native speaker was asking about the color yellow. Instead, she asked the question using the present tense, saying, "you're speaking too quietly?" rather than saying something like, "when she said, 'the what is yellow,' what were you thinking?" In other words, the non-native speaker's comments about her low voice are ambiguous. Was she thinking that at the time of the feedback during the original session? Or was she only thinking about it now that her attention was focused on the original exchange? The interviewer did not bring the focus clearly to the time of the recall.

< previous page

page_92

next page >

< previous page

page_93

next page > Page 93

In Example 13 the interviewer continues to focus on the time of the recall rather than on the time of the initial event. The interviewer again, through her use of the present tense and her failure to refer to the past event, continued to prompt the learner to focus on what she was thinking at the time of the recall session rather than what she was thinking at the time of the original picture difference task. Example 13 NNS NS-1 NNS NS-2 Stimulated NNS Response Original FeedbackResponseRecall Prompts Comments He also standing He's standing Yeah Why did she repeat it after you? Because I still speak lower . . . and maybe she don't understand She doesn't hear? Yeah She doesn't hear or doesn't understand Doesn't hear Why does she say that? In Example 14, we see further evidence of the interviewer's focus on the current thinking of the NNS as opposed to the on-line thinking at the time of the original exchange. This example again shows the problematic practice of focusing on the time of the recall and adds another one: The interviewer led the NNS participant toward an answer, suggesting that she had left something out. In the next turn, the interviewer suggested to the NNS why the NS participant in the interaction might have repeated something.

< previous page

page_93

next page >

< previous page

page_94

next page > Page 94

Example 14 NNS NS-1 NNS NS-2 Stimulated Original FeedbackResponseRecall Prompts Comments Right right side Facing to the right

NNS Response

My answer is not suitable Not suitable how? She asked which side fish is is face so I had to answer it is facing to the right side So when you said right side, you left something out? Yeah So, that's why she repeated it? Yeah The interviewer continued in the same vein in Example 15, however, she changed her tactic in Examples 16 and 17 and finally began to ask questions that are reflective of appropriate probing of thought processes during the original task. In Example 16 it is unclear when the NNS realized that people requires a plural verb. It is likely that she thought about it at the time of the stimulated recall rather than at the time of the spot-the-difference task. The interviewer prompts were not eliciting at the desired there and then processing. In Example 16, the interviewer apparently realized her mistake and began to focus on previous thoughts rather than on thoughts during the recall;

< previous page

page_94

next page >

< previous page

page_95

next page > Page 95

the NNS's final response is interesting because it seems that she was not aware of the corrections when they happened. Rather, she only became aware of them during the playback sessions.4 In Example 17, the interviewer asked about what she understood during the original task, but it was clearly too late to elicit useful recall comments given all the previous inappropriate questions. Further, there was still a leading question at the end, "Did you understand that then?" rather than a more neutral question, such as, "What were you thinking when she said 'all three people are in the sea'?" Example 15 NNS Original NS-1 NNS NS-2 Stimulated NNS Response Comments FeedbackResponseRecall Prompts People is fishing on the beach People are what? Fishing Because the grammar is not correct How is it not correct? (replays video) I said people is fishing but I have to say people are fishing. 4 Although it might be interesting to have participants reflect on the differences between what they thought at the time and what they thought later, this was not the focus of the study.

< previous page

page_95

next page >

< previous page

page_96

next page > Page 96

Example 16 NNS Original Comments He's taking picture to his child

NS-1 Feedback NNS NS-2 Stimulated NNS Response ResponseRecall Prompts

He's taking a picture of his child Yeah I think the preposition is not suitable Is that now Yeah Or then? No idea Can't notice back then Example 17 NNS Original NS-1 Comments Feedback

NNS NS-2 NNS Response ResponseStimulated Recall Prompts

Two people, three people is in the sea All three people are in the sea The sea I used I said people is three people is in the sea but I have to say people are Did you understand that then? Yeah?

< previous page

page_96

next page >

< previous page

page_97

next page > Page 97

Language of the Recall Session A final procedural concern that we raise has to do with the language of the recall sessions. Within L2 research, we are faced with a problem not present in research with native speakers, that of the first language of the participants and its potential influence on the recall sessions. If the recall is carried out in the L1, this may allay some concerns, but if the event to be recalled is carried out in the L2, the disparity in events and possibly the locus of knowledge accessed may give rise to a new set of concerns. If both the event and the recall are carried out in the L2, the problem is exacerbated by the fact that we are frequently dealing with learners who are limited in their ability to express themselves in the target language and to understand the target language. Therefore, not only are we hampered by the usual difficulties with verbal report data (i.e., are the reports truly reflective of thought processes?) but we are also confronted with the additional problems of possibly having to interpret what a learner says and making the assumption that a learner correctly understands what is being asked. In Example 16, the NNS says, ''I think the preposition is not suitable." The interviewer probed and was able to determine that the NNS was truly referring to the here and now, as opposed to intending something like "I thought that the preposition wasn't suitable."5 5 That the use of tenses can be misunderstood is apparent in data from Naylor (ms., no date) who worked on a court case that rested on the inappropriate use of and understanding of how the English tense system works. The consequences of the case were far more serious than the consequences of misinterpreting data in a stimulated recall. Nonetheless, the example illustrates how the use of tense can easily be misinterpreted even by reasonably fluent individuals. Naylor, a native speaker of Tagalog, was called by the defense in a trial in which two Filipino nurses had been accused of murdering patients under their care. It was felt that the nurses, in trying to defend themselves during the trial, had contradicted themselves, making them easy targets for conviction. An example from the transcript of the trial is given here: Question: Response:

Would you say that the two of you were close friends during that period of time? I would say we are good friends but we are really not that close because I don't know her and don't know each other that much.

(It is to be noted that in Tagalog, tense and aspect are not marked on the verb. Thus, eats, is eating, and was eating are all expressed by the same verb form.) With regard to the example, the statement might have been true in the past (i.e., they weren't good friends at the time of the incident), but the use of the present tense made this response contradictory. Similar misuse of the present tense can be seen in the following exchange concerning the drug Pavulon, the drug supposedly used to kill the patients. Question:

And what else did you learn about Pavulon, other than it was given at surgery?

(footnote continued on next page)

< previous page

page_97

next page >

< previous page

page_98

next page > Page 98

The issue of recall language became apparent in the Mackey, Gass, and McDonough (in press) study described earlier. This study involved two groups of learners, ESL and Italian as a foreign language (IFL). The recall was conducted in English for the first group (where it was the L2) and in English for the second group (where it was the L1). Of the categories included when analyzing the response types from the stimulated recall sessions, two are relevant for the present discussion: no comment and unclassifiable. What differed was the distribution of comments in these two categories. Whereas the ESL learners were unable to provide any recall comments for 12% of the data, the IFL learners did not produce any recall comments for only 3.5% of the data. We surmise that the use of language in the recall sessions contributed to this difference. The ESL learners, communicating in the L2 and faced with language difficulties, may have been more likely to state simply that they had no comment when they were unable to express their thoughts during an interaction episode. Using a subset from the two datasets (50% of each), we calculated a comparison of the number of words per recall comment. For the ESL learners communicating in the L2 for the recall, the average number of words per recall comment was 16. For the IFL learners, who were using their L1 for the recall (English) the average number of words per recall comment was 26. It is also difficult to determine whether the difference in words per comment is related to the use of the L1/L2 or to cultural differences between the American IFL participants and the ESL participants. Additional studies to explore the differential effects of using the L1 or the L2 during the recall session, as well as the potential effect of cultural background, would help clarify this issue (see Mangubhai, 1992). Another issue is that the limited proficiency of learners operating in their L2s may influence the content of the recalls. They may verbalize what they can, rather than the full version of what they were thinking. Research questions should take into account that when the L2 is used for the recall, some things may be easier for learners to verbalize than others. Pilot testing may ascertain whether some of the expected verbalizations are beyond the linguistic competence (footnote continued from previous page) Response:

Question: Response:

Are you asking me about what I know about Pavulon in the summer of '75 or what I know about Pavulon at the present time, after hearing all these experts? What you knew about Pavulon at the time. I know a little about Pavulon.

The parallels between this example and the use of stimulated recall when the language of the recall is not the native language of the participant are clear. They both assume that the participant has nativelike control over the target language tense system and therefore can accurately understand and produce a system that differentiates between the time of speaking and the time of thinking.

< previous page

page_98

next page >

< previous page

page_99

next page > Page 99

of some learners. We present general recommendations regarding instrumentation (timing and language). RECOMMENDATIONS Timing Carefully pilot all instruments to ensure accurate time estimates. How long will the initial data collection (from which the recall will be derived) take? How long will the recall take? How long does it take to set up the task? Consider of the transition time from the initial task to the recall task. Allow equal amounts of time for all participants across topics or episodes. In the analysis, take into account individual characteristics of verbosity. Make sure that the recall questions focus on the timeframe in question. Language Consider carefully the implications of performing the recall in the L2 versus the L1. Analysis of Stimulated Recall Data Following the discussion concerning the presentation and coding of data and potential pitfalls in carrying out the procedure, we now turn to the analysis of stimulated recall data. Once again, it must be recognized that the analysis is, like the coding categories, highly related to the research questions and hypotheses of the specific second language study. Therefore in this section we present some general issues, using examples from previous studies. We finish with some recommendations. As pointed out in Chi's (1997) detailed article on the analysis of verbal data, quantifying what is usually perceived as subjective coding of the contents of verbal utterances can be challenging. Procedures including tabulating, counting, quantifying, and drawing inferences about relations among different kinds of utterances are often carried out in an attempt to reduce the subjectivity of qualitative coding. Quantification of qualitative data, such as a person's introspective comments about L2 writing revisions, is very

< previous page

page_99

next page >

< previous page

page_100

next page > Page 100

different from quantification of words, clauses, phonemes, grammatical and ungrammatical sentences, t-units, grammatical structures, or many other kinds of L2 data. Analysis of stimulated recall interviews (or other kinds of introspective recall data such as diaries) usually involves quantifying categories where there is no direct one-to-one correspondence between the category and the verbalized data and some inferencing or judgment is needed for classification. In some designs, the highly subjective introspective data need to be linked with objective, sometimes performance-based data. For example, in diary studies that claim to explore issues of noticing (Schmidt & Frota, 1986), introspective comments about language recorded in a diary are linked to objective performance in terms of production of the items recorded in the diary. In such coding schemes challenges to reliability can be especially high. Many stimulated recalls in L2 studies are carried out in order to explore issues about learners' cognitive processing or their use of communication strategies, or sociolingluistic issues related to their L2 use. In analyzing the recall data, the purpose is to identify and classify the verbalizations that shed light on the phenomena being investigated, yet to capture them in a way that is as low in subjectivity as possible. When analyzing stimulated recall data, all the usual caveats in terms of carrying out analyses that are solely quantitative or solely qualitative apply. Many studies benefit by using one approach to supplement the other where possible. Although qualitative analyses can provide a much richer understanding of processes and situational knowledge, controlling for variables that affect the data is impossible, and the analysis is often subjective and nonreplicable. Quantitative analyses on the other hand, are usually objective and replicable but generally only applicable to the specific hypotheses addressed in the study, and problems of generalizability may apply. Thus, when analyzing stimulated recall data, combining analytical techniques is desirable wherever possible. The shortcomings of one type of analysis may be addressed through the strengths of another. There are various ways to combine analytical methods for stimulated recall data. For example, the qualitative data can be used to shed light on the findings of any quantitative analysis. Examples can be found in a study of problem solving by Chi, Feltovich, and Glaser (1981). They asked experts and novices to categorize problems according to their own categories. The categorization patterns were quantitatively analyzed using factor analyses. Participants' explanations for why they chose particular categories were also examined, and these data were used to interpret what the different categories meant to experts and novices. However, as noted by Chi (1997), no main claim was made on the basis of the qualitative data; the emphasis was

< previous page

page_100

next page >

< previous page

page_101

next page > Page 101

on the quantitative data. Another study that uses qualitative data to further interpret quantitative analysis is that of Poulisse, Bongaerts, and Kellerman (1987), who studied communication strategies using retrospective reports and included both a quantitative section that compared categories and a qualitative section that discussed the influence of researcher bias. Finally, Gass (1994; described in greater detail in chap. 2) used a stimulated recall design to aid in the interpretation of a primarily quantitative study. She gathered quantifiable data from learner judgments of sentence acceptability. Judgments were gathered on a 7-point scale and were collected two times within a one-week time period as a way of gathering information on reliability. When judgments seriously differed (e.g., from one end of the scale to another) across the time frames, Gass used recall data in an attempt to explain these differences. Another way to integrate quantitative and qualitative methods in stimulated recall data is to quantitatively analyze categories that are essentially qualitative in nature but for which careful operationalizations have been made and for which high interrater reliability has been obtained. An example can be found in the Mackey, Gass, and McDonough (in press) study, which is discussed at length in the section on interrater reliability. Perhaps the most common integration of methods is to use two analytical techniques side by side. For example, Bosher (1998) studied the composing processes of three ESL writers and carried out quantitative and qualitative analyses of aspects and strategies in their recalls, as well as "themes which emerged from the data" (p. 214). Whichever specific analytical technique or qualitative-quantitative combination is selected, there are certain steps that must be taken in carrying out analyses. We have revised a subset of the techniques put forward by Chi (1997) so that they specifically relate to stimulated recall protocols. The steps are discussed in the following sections. It should be noted that the obvious first step is to transcribe and lay out the data; this process was described earlier. Sampling the Recall Data The first step involves deciding how much or how many of the recalls need to be analyzed. For example, it is not uncommon in the L2 research field to analyze a subset of an interview, such as the middle 10 minutes of a 30-minute interview. Transcribing, coding, and analyzing all 30 minutes is time consuming. It may be unnecessary if 10 minutes is a representative sample of the learner's language. Why choose the middle 10 minutes to sample? The first 10 minutes may be affected by the participant settling down, get-

< previous page

page_101

next page >

< previous page

page_102

next page > Page 102

ting comfortable, and the researcher chit-chatting to warm the learner up and allay any initial nerves. The final 10 minutes may be their winding-down time: The learner may be tired; the interview may finish early. The central 10 minutes will often contain the richest and arguably the most representative data of a particular L2 learner's developmental level. Of course, some studies sample the first 10 minutes, or the first and last 5 minutes. A study of L2 phonology, for example, may focus on the change in vowel quality over the course of an interview and sample all instances of one vowel sound throughout the entire time. Other types of sampling involve making decisions about how much data from each participant to study. If, for example, five recalls exist at different times for one participant, should they all be used, or will two be sufficient? Which two? If an entire class of 30 students carries out a recall, should they all be analyzed or is half the class, selected at random, sufficient? Do all the data contained in the recalls need to be analyzed, or just specific responses that relate only to the research question? If sampling is carried out, how could it affect the analysis as a whole? Whether and how sampling of the data is carried out obviously needs to be decided based on the resources available for the study and the specific research questions addressed by the study. Preparing the Data for Coding Preparation of the data for coding is discussed in this section. Again, this is obviously research-question dependent. If a recall of oral interaction data is to be carried out, the specific interactional features that the coding will isolate need to be selected. This may involve categorizing the data into episodes in the first instance. It may involve separating the data into segments based on turn boundaries or idea units. Alternatively, if a feature such as strategy use is the focus of a study, the data may need to be segmented into units in which each mention of a strategy type is categorized. Obviously, this advice needs to be considered in the context of the research question. Often, the simple segmentation of the data represents the same step as coding. In other cases, the data need to be prepared for coding through segmentation. It can be helpful to think of the notions of segmentation and coding as separate in that segmentation is a much broader category than many coding schemes. It can be thought of as a preparatory stage, in the sense that it avoids having to sift through unnecessary verbalizations during what is probably already painstaking coding. It can also give a useful idea about subjectivity in the data set. If two researchers find it difficult to agree

< previous page

page_102

next page >

< previous page

page_103

next page > Page 103

about broad segmentations in the data set, this may provoke some early questions and assessment in relation to the categorizations. Generally, although not always, if broad categorizations are difficult to define, more finely grained coding schemes based on the broader units will be even more problematic. Developing a Coding Scheme Coding stimulated recall data has been discussed previously. Suffice it to say that coding schemes will be dependent upon the questions and hypotheses to be addressed in the study. In coding stimulated recall data, what often happens is that a top-down coding scheme is developed on the basis of a central research question. This can then be fine tuned after a first pass through the data. In coding stimulated recall data, one often needs to be flexible, as the data can be unpredictable. Thus, coding schemes need to be prepared with the possibility of change and revision in mind. For example, although coding may be first visualized as counting morphosyntactic structures, an analysis of semantic units or discourse units may shed more light on the relations among categories or perceptions. Studies have shown high levels of interrater reliability in the coding of introspective verbal data (cf. Bettman & Park, 1979); see the discussion of complementary coding schemes in Elstein, Shulman, and Spaafka (1978) and our review of concerns about reliability in chapter 5. Analyzing and Describing the Data Having coded the data and considered issues such as interrater reliability and the method of analysis, most researchers consider additional issues such as statistical testing where appropriate, interpretations of the data, and ways to provide descriptions of the data. Whether the data are depicted graphically, pictorially, in a tabulated form, or through example sets will again depend on the research question and on the patterns that need to be illustrated. Although these steps sound straightforward, finding and demonstrating patterns in stimulated recall interviews can be challenging. If the data are coded into categories that can be quantified and the results tested for significance and perhaps graphically illustrated through charts, pattern identification and illustration is relatively straightforward. However, in many purely qualitative analyses, models, perhaps illustrating links, may need to be used. It is often said that researchers who work in a qualitative paradigm need to be able to tell a story convincingly and distinctively. This certainly applies when finding and illustrating patterns in stimulated recall data.

< previous page

page_103

next page >

< previous page

page_104

next page > Page 104

When depicting data, some studies supplement or triangulate stimulated recall data. For example, they may supplement the recall data with performance-based data. One example of a study that uses more than one type of recall data was carried out by Russo, Johnson, and Stephens (1989). Admittedly, their study is a comparison and test of reactivity and veridicality of different oral protocols, yet they triangulate data using different tasks and both on-line verbal protocols (i.e., think-alouds) and stimulated recalls supported by different levels of stimuli.6 Conclusion This chapter has provided detail on the dos and do nots of conducting stimulated recall. In the following chapter, we deal with limitations of stimulated recall and also suggest ways that stimulated recall may be useful in supplementing other types of data. 6 One interesting interpretation they made based on their study was that care must be taken to ensure that people view thinking aloud as secondary to the performance of the task at hand, such as problem solving. Training and the use of warm-up tasks may be helpful in this respect. It is not a problem that needs to be addressed by stimulated recall studies because participants are generally not told that they will be recalling the data, so there is no interference.

< previous page

page_104

next page >

< previous page

page_105

next page > Page 105

5 Limitations and Additional Uses No methodology is without critics. In this final chapter we focus on the criticisms in the literature that have been directed at stimulated recall during its history. We have organized these problem areas into sections that deal with issues of validity and reliability. Procedural problems with the use of the methodology (and recommendations for avoiding problems) have been covered in chapters 3 and 4; interrater reliability was covered in chapter 4. Following the discussion of limitations, we turn the tables and deal conceptually with ways in which stimulated recall can enhance empirical research, particularly research in which questions remain unanswered. Issues of Validity and Reliability In this section, we deal with the most fundamental problems that have to do with the validity and reliability of the methodology. Do the reports actually reflect the thought processes of participants? Are the sequences of states of heeded information (i.e., thoughts), as reflected in the three panels in Fig. 1.1 truly comparable? Are thought processes even relevant to understanding how second languages are learned? To recap, retrospective reports assume that the information that is reported is directly accessible and available for verbal reporting. This assumption is better justified with only a small amount of intervening time between the event and the recall. Nonetheless, participants should be able to access some type of memory structures when instructed to report what they can remember about their thought processes during an event.

< previous page

page_105

next page >

< previous page

page_106

next page > Page 106

Nisbett and Wilson (1977b) argued against the use of verbal reporting as a means of gaining access into cognitive processing, claiming that their participants provided inaccurate reasons for their thoughts. In fact, Nisbett and Wilson claimed that "there may be little or no direct introspective access to higher order cognitive processes" (p. 231). They said that when individuals attempt to report on their cognitive processes, "they do not do so on the basis of any true introspection. Instead, their reports are based on a priori, implicit causal theories, or judgments about the extent to which a particular stimulus is a plausible cause of a given response" (p. 231).1 In other words, the verbal reports, perhaps tainted by inaccurate memory, contain (unknowingly) fabricated mental events. Part of Nisbett and Wilson's argument rests on the fact that conscious awareness can only relate to the products of mental processes; the processes themselves cannot be reached through introspection. White (1980), however, pointed out that this argument becomes circular: "If we decide to use consciousness as the criterion for making the distinction [between process and product], then the product/process viewpoint becomes true by circularity'' (p. 106). Because Nisbett and Wilson's work is frequently cited by those who argue against verbal report data, Ericsson and Simon (1996: 2627) spent considerable time arguing against their, position. Ericcson and Simon presented two excerpts from Nisbett and Wilson (1977b): People often cannot report accurately on the effects of particular stimuli on higher order, inference-based responses. Indeed, sometimes they cannot report on the existence of critical stimuli, sometimes cannot report on the existence of their responses, and sometimes cannot even report that an inferential process of any kind has occurred. (p. 233) When reporting on the effects of stimuli, people may not interrogate a memory of the cognitive processes that operated on the stimuli; instead, they may base their reports on implicit, a priori theories about the 1 Nisbett and Wilson (1977a) further this line of research empirically by showing that global evaluations of an individual can induce altered reports of a person's attributes.

< previous page

page_106

next page >

< previous page

page_107

next page > Page 107

causal connection between stimulus and response. (p. 233) With regard to the first, Ericsson and Simon noted the lack of precision in Nisbett and Wilson's statements with words such as often and sometimes being frequently used. The second statement is more difficult and represents the crux of the matter. However, Nisbett and Wilson did not discriminate between studies that were conducted well and those that were not. If recall is to be accurate and if recall is to reflect processes rather than theories about processes, there have to be safeguards in the procedure itself. As we have pointed out in earlier chapters, the thought processes must be accessible. There are at least two ways to make this more likely. First, and this is the basis of stimulated recall, is the necessity of having recall as opposed to a verbal report and second is the necessity of having as little time lag as possible between the event to be recalled and the probe. In addition to their criticisms of Nisbett and Wilson's perspective, Ericsson and Simon pointed out the need to make distinctions between giving reasons for thoughts or thought sequences and just reporting those thoughts. They provided an example to illustrate this important difference: When people are asked to generate a word that begins with the letter A, most respond with the word apple. When asked for a retrospective report (e.g., "tell me what you were thinking") people report that it "popped up" but cannot provide any intermediate steps or thoughts. However, when asked why, they respond with something like, "In grade school I learned A as in apple.'' Thus, the kinds of responses differ depending on what is being asked. Ericsson and Simon further emphasized the need to ensure that the contents being recalled are in accessible memory structures and were generated orally. This latter requirement ensures that there are no modality translations that might interfere with the recall process. Russo et al. (1989) also pointed out the need to ensure that what is being recalled is not an automated process precisely because little trace of such processes is left in accessible memory. They suggested two ways of determining invalidity of the methodology: reactivity and nonveridicality. Reactivity refers to those instances in which the primary process is altered as a result of verbalization (this is likely to occur in concurrent protocols). Nonveridicality, on the other hand, refers to the lack of correspondence between a protocol and the underly-

< previous page

page_107

next page >

< previous page

page_108

next page > Page 108

ing primary process. Examples of nonveridicality include errors of omission and errors of commission. The former are of little consequences when dealing with stimulated recalls, but the latter are serious and can invalidate the methodology because the protocol itself is taken as veridical. Reactivity is of less concern when doing stimulated recalls but may be important for concurrent protocols. Russo et al.'s data are not conclusive with regard to the issue of reactivity. In some (but not all) of their experimental results, there appears to be evidence of reactivity, although the occurrence of reactivity itself was not predictable. In other words, a task that might have been likely to promote reactivity (e.g., a task that involved recoding from a pictorial to an oral mode) did not, in fact, promote reactivity. With regard to veridicality, the issue is similarly complex. They suggested that cueing the original task (e.g., with the original stimulus or with the individual's performance on the original task) is more likely to result in intrusions than noncued stimuli. However, their participants performed four tasks (only one of which, an anagram task, was a verbal task), and the anagram task was not among those in which intrusions were found to any significant degree. Although the jury is still out on these issues, the question of validity must always be foremost; stimulated recalls must be carried out with care and the data used and interpreted with caution. Russo et al. did not intend to invalidate the procedure. They argued that the instructions given to participants are important in minimizing any threat to the methodology itself. They provide a hierarchy of invalidities from most serious to least serious: disruption of the primary process, omissions in the verbal report, and longer time to complete the task. These issues must all be kept in mind when conducting recall studies. One must also bear in mind the kind of data that are being elicited. Much of the experimental work validating the methodology has dealt with psychological processing rather than with language phenomena. With regard to reactivity, results are conflicting. Anderson (1985) found a decrease in judgment accuracy as a function of verbalization, but Boritz (1986) noted improved performance after verbalization (motivation was hypothesized to be the intermediary factor). Biggs, Rosman, and Sergenian (1993) reported on concurrent verbal protocols, focusing on two issues, reactivity and completeness. Their study was conducted with equity analysts. As a way of testing reactivity and

< previous page

page_108

next page >

< previous page

page_109

next page > Page 109

completeness, researchers had three conditions: verbal reporting, computer search, and both. In the verbal report condition, participants were given information about a company (e.g., financial and nonfinancial information) and were asked to think aloud when examining the data. In the computer condition, information was accessed by means of a computer program. In a condition that included both, participants were asked to think-aloud while conducting a computer search. To determine reactivity they compared the computer condition with the both condition and showed that accuracy of judgments was not affected. To test completeness, they considered the both condition and found that the verbal traces were less complete than the computer search. Even though the computer search condition resulted in greater reliability, it was not able to provide the insight into, in this case, decision-making behavior that can be provided through verbal reporting. One final study that deals with issues of reliability was carried out by Ericsson and Simon (1980). They showed that verbalization affects cognitive processes "only if the instructions require verbalization of information that would not otherwise be attended to" (p. 215). This suggests, as we have suggested throughout the book, that one has to use extreme caution in what and how questions are being asked. As Ericsson and Simon noted ". . . verbal reports, elicited with care and interpreted with full understanding of the circumstances under which they were obtained, are a valuable and thoroughly reliable sort of information about cognitive processes" (p. 247). These studies suggest then that issues of validity and reliability, while controversial, are not serious detractors. The L2 literature has not been without its detractors, both implicit and explicit. As noted in chapter 1, Selinker (1974) made an early claim that "the only observable data from meaningful performance situations we can establish as relevant . . . are . . . IL utterances produced by the learner" (p. 35). He added that this is not an anti-mentalist position, but only that "the analyst in the interlingual domain cannot rely on intuitive grammatical judgments since he will gain information about another system, the one the learner is struggling with, i.e. the TL" (p. 51, fn. 5). In Selinker's view, the most important argument is "that predictions based upon them are not testable in 'meaningful performance situations' . . . a reconstruction of the event upon the part of the learner would have to be made in a perceptual interlingual study. Such difficulties do not exist when predictions are related to the shape of

< previous page

page_109

next page >

< previous page

page_110

next page > Page 110

utterances produced as the result of the learner attempting to express in the TL meanings which he may already have" (p. 51, fn. 9). Selinker was undoubtedly referring to the vexed methodology involving acceptability judgments. Nonetheless, his comments reflect the impressions that the only valuable data are those produced in context. Retrospection or introspection data are tainted because other systems (e.g., memory) may be interfering with linguistic knowledge. Although this view is maintained by some, it has never been entirely accepted. Corder (1973), for example, argued that forced elicitation data were necessary. Elicitation procedures are used to find out something specific about the learner's language. Constraints must be placed on the learner so that she or he is forced to make choices within a severely restricted area of phonological, lexical, or syntactic competence. (Corder, 1973).2 Seliger (1983) questioned "whether the language learner can be used as a linguist in order to describe the process of second language acquisition, or even to describe linguistic processing in a more general sense" (p. 183). He also questioned whether "the verbalizations of learners represent some form of internal reality" (p. 180). He defined introspection as "conscious verbalizations of what we think we know" (p. 183) and argued that in utilizing verbal report data, we are taking "learner pronouncements [as] evidence for "the inner workings of the learner's mind'' (p. 185). In laying out some of the limitations of using verbal report data (in our specific case, stimulated recall data), we have also made it clear that there are ways that some of the limitations of stimulated recall can be minimized (see our recommendations in chap 4). Seliger identified three areas of L2 studies that have relied on verbal report data: monitor studies, diary studies, and introspection studies. We agree that verbal report data may be inappropriate in addressing certain sorts of questions, particularly questions that have to do with abstract knowledge. Seliger pointed out that the major piece of evidence regarding the two independent systems (i.e., acquisition and 2 One of the most common means of gathering introspective data is through acceptability or judgment data. The validity and reliability of judgment data have been amply discussed in the literature and arel not treated here (cf. Bard et al., 1996; Birdsong, 1989; Cowan & Hatasa, 1994; Gass, 1994; Goss, Zhang, & Lantolf, 1994; Munnich, Flynn, & Martohardjono, 1994; Sorace, 1988), as judgment methodology is not directly related to the type of verbal reporting discussed in this book.

< previous page

page_110

next page >

< previous page

page_111

next page > Page 111

learning) posed by Krashen (e.g., 1981) comes from asking learners to retrospect on their performance. Although we do not dispute the fact that retrospection might not be a valid means for gaining information about the status of the proposed acquisition-learning dichotomy, Seliger's reasons for dismissing the methodology, even in this case, are perhaps not valid. He stated, "The learner is asked to report what he was thinking at the time the particular language sample was produced, and this is sometimes days after the actual performance" (p. 186, emphasis ours). As we noted in earlier chapters, the timing between the event to be recalled and the recall itself is crucial. The fact that retrospective comments have been gathered inappropriately (e.g., without the recall prompt and with a significant time delay) does not invalidate the general procedure as a means of gathering important information, perhaps unobtainable by other means, and when carried out according to the guidelines put forward in this book, many of the reported problems of validity can be addressed. Cohen (1996, 1998) and Matsumoto (1993), within the context of the L2 literature, deal seriously with the disadvantages as well as advantages of verbal report data. The advantages they list include the following: Reflect a theoretical framework Reveal the information attended to during task performance Reflect cognitive events Are reliable in that they correlate with behavior Are useful in strategy research Are useful in determining what prior knowledge is used in processing texts Among the disadvantages they discuss are: Unconsciousness of cognitive processes Complexity of cognitive processes Inaccurate reporting on the part of participants Inaccessibility of some information Confounding of introspection and retrospection Intrusive Dependent on verbal skills of participants

< previous page

page_111

next page >

< previous page

page_112

next page > Page 112

For L2 research, the language of processing versus the language of reporting Veridicality Generating verbal reports may alter the nature of the process These and other advantages and disadvantages have been discussed in previous sections of this book and point to the fact that verbal reporting (and stimulated recalls) must be done with care and is, like any methodology, limited in scope and power. The question of reliability has received much less direct attention in the literature, probably because of the difficulty in determining reliability with introspections. Lieberman (1979), in his review of introspection, while acknowledging the limitations of introspection, discussed empirical evidence that shows that introspections can predict future behavior. Pressley and Afflerbach (1995) were careful to point out that the extent to which we can trust verbal report data is, in part, dependent on the amount of interpretation as opposed to pure content of memory that is given. In other words, it is more accurate for the researcher to make an interpretation on the basis of the reporting of content rather than on the basis of participant's interpretation of the content. Summary The previous sections have dealt with the limitations of stimulated recall as a research tool. In sum, there are two important potential limitations in the use of stimulated recall that need to be emphasized in relation to the questions it can be used to address. First, as pointed out by Ericsson and Simon (1987, 1993, 1996), although every effort should be made by the researcher to ensure the recall is carried out as close as possible in time to the actual event, in some cases the memory structures being recalled may not always relate directly to the event that just occurred. For example, if a learner were asked to recall thoughts during a story retelling that he or she performed on-line three days before, the learner may recall thoughts that pre-dated the story telling, knowledge that was laid down earlier but accessed in the same frame. A sec-

< previous page

page_112

next page >

< previous page

page_113

next page > Page 113

ond problem also pointed out by Ericsson and Simon is that participants may have experienced some interference during the period between the event to be recalled and the recall. Even if that time is only half an hour, some memory types can decay rapidly (Posner, 1992), and contamination of memory represents one of the most significant threats to researchers' claims that stimulated recall data can uncover information about participants' cognitive processes. Possible Uses for Stimulated Recall In this section we turn to an exploration of the ways in which stimulated recall methodology can be used as a supplement to empirical L2 research. We do this as a way of demonstrating the potential of this methodology for shedding additional light on issues that may remain unresolved through empirical data alone. This section should not be interpreted as claiming that the researchers whose work we have highlighted should have used recall data or that their studies were in any way weakened without such data. Rather, we intend to show how recall data can be used to reinforce other kinds of data or can be used to shed light on unresolved issues. We approached this task by considering the results of published studies and the issues that they raised. The question we asked was whether the use of stimulated recall could have contributed to our knowledge on this particular topic. The studies themselves were described in published articles in five leading journals in the field: Studies in Second Language Acquisition, TESOL Quarterly, Language Learning, Modern Language Journal, and Applied Linguistics. Our rationale for selecting these journals was that each has a long history of publishing detailed empirical studies using L2 data and that as a whole, they represent the diversity in what might roughly be called the field of applied linguistics. Our selection of articles was limited to those published in 1998. By exploring the ways in which the use of stimulated recall data might enhance the findings of the studies, our intention was to further illustrate the nature of stimulated recall as well as demonstrate its versatility as a tool. Of course, there are many practical issues and challenges to be considered (as detailed in earlier chapters), and although we briefly note some of these in relation to the specific studies we describe here, it is important to point out that we were not privy to the

< previous page

page_113

next page >

< previous page

page_114

next page > Page 114

data collection situation. Further, because not all studies detail their procedures extensively, in some cases our discussion of how the use of stimulated recall may have benefited these studies may be tempered by other concerns, of which we are not aware. It is, of course, the responsibility of individual researchers to carry out a careful cost-benefit analysis when deciding on methodologies. This section, then, details the potential of stimulated recall for triangulation of data and adding insights, and is contextualized within published research. Interlanguage Phonology Comprehension of Learner Speech Munro (1998) explored the effect of background noise on the perception of English spoken by native speakers and by speakers with Mandarin accents. Sets of true and false statements were recorded by the two groups of speakers in a sound-proofed room, spoken at a comfortable rate. Practice time was provided, and the speakers were given the opportunity to produce their best possible renderings of each statement, including the opportunity to repeat their sentences. Volume was kept constant and reading errors, hesitations, and extraneous noises did not occur. Eight productions were selected from each group of 10 native speakers of English and ten non-native speakers with Mandarin L1s. Of the eight productions, four were true and four were false. Half the productions were kept in their original noise-free form, and half included digitally mixed cafeteria noise. Munro hypothesized that between 20 and 30 percent of the utterances would be rendered unintelligible by the cafeteria noise. Counterbalanced stimulus sets containing the following were presented to 24 native English speaker judges: Native English speech without noise, Native English speech with noise, Mandarin-accented English speech without noise, and Mandarin-accented English speech with noise. The judges were undergraduate students of linguistics who were speakers of Canadian English with self-reported normal hearing. There were 40 test items, half with noise and half without. The judges assessed the productions as being true, false, and unintelligible. During a

< previous page

page_114

next page >

< previous page

page_115

next page > Page 115

practice session they heard a random presentation of 10 statements. Their task was to write out the utterance that had been produced as closely as possible using a blank line to indicate unintelligible words. Munro's results were obtained by summing the number of correct true and false responses, treating unintelligible responses as incorrect. Scores were assigned to the transcribed sentences by calculating the percentage of correct words in the sentence. Mandarin-accented utterances were correctly perceived less frequently than native English utterances in the conditions with noise and without noise. However, the Mandarin-accented English without noise was correctly perceived more often than the native English with noise, a finding that was statistically significant. Simply put, noise added in the experiment made the utterances significantly harder to perceive than did the presence of a Mandarin accent alone. In terms of the transcriptions, judges' scores were higher for Mandarin-accented speech without noise than native English speech with noise. More Mandarinaccented utterances with noise were completely unintelligible than any other type of utterance. There was also some variability in the perception of productions by individual speakers. Munro explained that in this experiment, the addition of cafeteria noise had a strong effect on perception for some (but not all) Mandarin-accented speech. He claimed that his experiment provides preliminary evidence that the effect of noise on the intelligibility of foreign-accented speech may be greater than for native-accented speech but suggested a fuller understanding must await additional experimentation: it is . . . noteworthy that individual listeners were apparently greatly affected differently by the presence of noise, with some showing much greater difficulty understanding noisy utterances than others . . . it is beyond the scope of this study to explain why some of the voices were affected more by noise than others . . . It is possible that differences in voice quality . . . and differences in prosodic goodness may have played a role. For instance, the use of a certain vocal pitch or a nonnative rhythmic pattern may make it difficult for listeners to track a voice under adverse listening conditions. (p. 151)

< previous page

page_115

next page >

< previous page

page_116

next page > Page 116

The use of stimulated recall methodology may have provided Munro with an extra window through which to gain perspective on the perception scores. One way to address his call for work to discover effects of vocal pitch and rhythmic patterns may be to extend his own methodology by asking judges about their judgments. Judges could be presented with an opportunity to introspect about their judgments after they complete the task, so as not to affect the initial perception results through ongoing introspection. Each stimulus production and the judgment or transcription could be presented to them immediately after completing the study, with a question about why they had made their judgment and on what criteria the judgment was based. It seems particularly likely that Munro's judges, linguistics students, could articulate problems with prosody, pitch, and rhythm. Of course, it would have to be carefully explained to the judges that the reasons for their judgment at the initial time, the first listening, were of interest, and the effects of subsequent productions and a second hearing would have to be considered. However, when determining the effect of noise on the intelligibility of foreign-accented speech, the insights of the judges about their own perceptions would certainly be an interesting supplement to the data and could be considered in interpretation of these results. Change over Time Riney and Flege (1998) reported on a study of change over time in global foreign accent and liquid (/r/ and /l/) identifiability and accuracy. They carried out three experiments. Their goal was to study variation over time in global accent, intelligibility, and accuracy of phonetic segments and the relations among these constructs. They explored the link between global foreign accent and production of two English consonants, /r/ and /l/, by Japanese college students in their first year (Time 1) and their senior year (Time 2). Most of these learners had spent the majority of this time in Japan. Like Munro, Riney and Flege used native English-speaking judges to rate sentences spoken by other native English speakers and Japanese Ll speakers (n = 11) as well as to rate 25 word onsets containing the target consonants. The native speaking judges assessed how far the consonants produced could be identified as intended at Time 1 and Time 2, and whether the target consonants were produced more accurately at

< previous page

page_116

next page >

< previous page

page_117

next page > Page 117

Time 1 or Time 2. For one of their experiments, they used a group of trained native speaking judges (n = 3). The judges were categorized as trained because they had all carried out postdoctoral research in L2 phonetics and phonology. These trained judges were asked to identify the /r/ and /l/ consonants in the unidentifiable first part of words. Riney and Flege reported that some speakers showed significant improvement in both global foreign accent and liquid identifiability and accuracy. Perhaps unsurprisingly, the two speakers who improved the most were also the two speakers who had spent the most time in an English-speaking environment. The untrained judges used a scale from 1 strong foreign accent to 9 no foreign accent and were asked to rate only pronunciation and to ignore everything else. The trained judges were asked to categorize segments as /r/, /l/, or neither. In their conclusion, Riney and Flege noted that untrained listeners may have resorted to distinguishing sounds on the basis of software-induced noise distortion, that clusters may contain a host of phonotactic variables, and that phonological context may be important. The trained judges, like the undergraduate linguistics students in Munro's study, appear to have been well placed to have commented retrospectively on their judgments. One of the trained listeners was one of the authors, so his insights were no doubt incorporated. Formally collected stimulated recall data about the judgments of the other trained judges with their post-doctoral work in L2 phonetics and phonology would no doubt have added valuable insights that could have refined future studies involving the perception of phonological variables. Classroom Interaction Lyster's (1998) study of recasts, repetition, and ambiguity in the L2 classroom context categorized 377 teacher recasts in 18 hours of classroom interaction. After classifying recasts according to pragmatic functions in classroom discourse, Lyster compared them to teachers' use of noncorrective repetition. This led him to conclude that recasts and noncorrective repetition fulfill identical functions and that the corrective properties of recasts may be overridden by their functional properties in meaning-oriented classrooms. Lyster's measure of corrective properties was learners' immediate uptake of recasts, or what learners actually did with recasts in the turn immediately following the recast.

< previous page

page_117

next page >

< previous page

page_118

next page > Page 118

Mackey and Philp (1998) showed that developmental outcomes of recasts may not show up immediately but may show up in the short term or the longer term, a developmental perspective pointed out by others including Gass and Varonis (1994) and Lightbown (1998). Additionally, Lyster's claims about the way in which recasts are intended by teachers and perceived by learners are difficult to support based on his data. His claim that the corrective function of recasts may be overridden by their positive reinforcing function may be one explanation for why beginning-level learners did not develop as a result of intensive recasts, but advanced-level learners did develop in Mackey and Philp's (1998) laboratory study. Lyster stated: "one may well wonder, first, how L2 learners can distinguish the purpose of recasts from the purpose of noncorrective repetitions and, second, whether the teachers' intention in recasting is indeed to correct form or if their intention has more to do with content" (p. 65) and "in addition, it may be the case that teachers, when their intentions were indeed to correct, provided additional signals, which were not detected in the transcript, that distinguished some recasts from noncorrective repetitions (e.g., waiting longer or looking at students in ways that invited uptake)" (p. 67). These statements clearly indicate that an interesting element could have been added to these classroom data. If the classes had been videotaped and stimulated recall procedures carried out with the teachers, their intentions in using recasts or non-corrective repetition could have been clarified. Also, if the stimulated recall procedure had been carried out with some students, an additional measure to the short-term, immediate uptake score would have shed light on the learners' perceptions about the teachers' recasts. Oral Production Mehnert (1998) carried out a study of the effect of different amounts of planning time on the speech performance of different Ll learners of German. She used two tasks that varied in terms of structure and information familiarity. Both required participants to leave messages on answering machines and were designed so as not to appear too artificial in a language laboratory experimental setting. In the first task, participants were required to explain to a friend that they were not able to meet him at the airport and left directions along with a time to meet at

< previous page

page_118

next page >

< previous page

page_119

next page > Page 119

the university. This task was supposed to be structured, to contain familiar information, and to require the present or future tenses. In the second task the participants were required to apologize and explain to two friends why they had not met them the day before as arranged. Six specified words were to be used. This task was supposed to be based on unfamiliar information, be unstructured, and require the use of the past tense. Arguably, the second task was more complex than the first task, according to Mehnert. Her study utilized three experimental groups with 1, 5, and 10 minutes of planning time. Participants were asked to make written notes to ensure they planned; their oral production was measured in terms of fluency, accuracy, complexity, and lexical density. Results showed that those with 10 minutes of planning time produced more fluent and accurate speech with higher lexical density. Mehnert concluded that planning time results in improved performance but that increase in performance differed according to the four measures she used, meaning that different things are going on at different times. For example, when participants had only 1 minute to plan they gave priority to accuracy, but when they had 10 minutes to plan, they attempted to produce more complex language. She claimed that this result provides support for the competing attentional resources model, that one can focus on only accuracy or complexity at one time. She also claims that task properties at least partly determine the level of fluency, accuracy, and complexity produced. Mehnert began her discussion by saying, "it was assumed that planning time is used by L2 learners to prepare cognitively and linguistically; that is, to decide on what meaning they want to convey and to search for and activate the linguistic resources best suited to express the intended meaning" (p. 99). She ended by saying that "how to approach the planning task was very much left to the subjects themselves" (p. 106) and calls for research on "how subjects' attention can be channeled intentionally into improving either fluency, accuracy, or complexity of speech, or all of them equally'' (p. 106). It would appear that research on planning, and this study in particular, could benefit from the use of stimulated recall methodology. First, carrying out a stimulated recall across different participants for different tasks with different amounts of planning time would result in useful data about what people thought they did with planning time. For example, the assumption Mehnert makes about preparing for linguistic

< previous page

page_119

next page >

< previous page

page_120

next page > Page 120

and cognitive task demands could be explored empirically through introspections. Second, how L2 learners approach planning could be studied through stimulated recall by allowing learners the freedom to plan on paper, but then showing them videotapes of their planning time and asking them about their thought processes while they were planning. This exploration of different learners' approaches would also provide insights into the ways L2 learners' attention could be channeled into using planning time for linguistic improvement. Mehnert's use of different amounts of planning time could also be incorporated into stimulated recall, examining intraparticipant variation in terms of perceptions about use, as well as interparticipant variation. Interlanguage Pragmatics Bardovi-Harlig and Dörnyei (1998) explored the extent to which instructed L2 learners of English (ESL and EFL) were aware of the differences in learners' productions and target language productions in grammar, in terms of the accuracy of utterances, and pragmatics, in terms of the appropriateness of utterances. They used a videotape with 20 scenarios and played the scenarios to 543 learners in three countries. They also considered the learners' teachers in terms of their perceptions about grammatical and pragmatic errors. Eight scenarios featured pragmatically appropriate but ungrammatical sentences, eight scenarios featured grammatically appropriate but pragmatically inappropriate sentences, and four featured sentences that were pragmatically inappropriate and ungrammatical. Bardovi-Harlig and Dörnyei pointed out that grammatical competence often exceeds pragmatic competence and indeed found that EFL learners consistently ranked grammatical errors more seriously than pragmatic errors. The ESL learners showed the opposite pattern, although Bardovi-Harlig and Dörnyei found the ESL learner results more difficult to interpret given research findings that ESL learners exhibit different interlanguage systems than native speakers. They suggested (following Schmidt, 1993) that noticing pragmatic aspects in the input is more likely in situations in which learners are struggling to make themselves understood as well as struggling to establish smooth relationships with native speakers and other learners. They also pointed out that: "clearly any account of the development of interlanguage pragmatics will have

< previous page

page_120

next page >

< previous page

page_121

next page > Page 121

to take into consideration the numerous variables that intervene between the stages of noticing and targetlike production" (p. 255). They noted that in their future research they intend to administer a production questionnaire and supplement the data with respondents' retrospective comments. Stimulating the retrospective comments with judgment data obtained from participants while watching the video scenarios might allow studies such as this one to tap into the learners' perceptions about grammar and pragmatics, as well as to explore their noticing of these variables. Of course, the act of recall could stimulate noticing, so controlled groups of recall and nonrecall participants would be needed. This could be controlled by having some learners (but not others) carry out production exercises. Comprehension Kempe and MacWhinney (1998) carried out a study of the acquisition of the learners' ability in the L2 to comprehend overt morphological case marking in Russian or German as second languages by native English speakers. They contrasted two approaches to learning inflectional morphology: the rule-based approach, which predicts that learning is driven by paradigm complexity, and the associative approach, which predicts that learning is driven by the cue validity of individual inflections. Participants (n = 44, 22 learners of Russian and 22 learners of German) carried out different tasks targeting L2 comprehension. One was a picture choice task consisting of simple active transitive noun-verb-noun sentences that were grammatically correct with cues of case marking, noun configuration, and animacy counterbalanced. A second task required learners to make lexical decisions for words and nonwords. Participants were tested individually. For the lexical decision task they pressed one button if they knew the word and another button if they did not. For the picture choice task, they saw pairs of pictures while the name (descriptive title) of one of the pictures was presented through headphones. Participants were instructed to press the right button or left button depending on whether the name corresponded to the right or left picture. In the main experiment, learners heard simple transitive sentences accompanied by pictures of both nouns, and they chose the agent of the sentence from the two pictures, again pressing a button for the left or right pictures.

< previous page

page_121

next page >

< previous page

page_122

next page > Page 122

The results showed that learners of Russian used case marking earlier than learners of German, and learners of German relied more on animacy to supplement the weaker case marking cue. The competition model correctly simulated the results, supporting the claim that adult L2 learning is associative and driven by the validity of cues in the input. They concluded that It can be argued that the language difference in the use of case markers does not necessarily arise from differences in the statistical distribution of case marking in the two languages but may be related to differences in metalinguistic awareness of case marking . . . More generally speaking, the perceived difference between the Ll and the L2 is smaller for the learners of German, which might encourage transfer and minimize these learners' awareness of the morphological function of the determiner. (p. 580) Stimulated recall might have been profitably used to address the question about metalinguistic awareness. Participants in this study were recruited by advertisement and were college students or recent graduates. Hence, they were sufficiently sophisticated to be able to make appropriate metalinguistic comments. Learners could have been videotaped while carrying out the task, or their task and answers could have been used as the stimulus, given that the videotape would have to include the aural sentences, which might be more complex than simply showing people the task and their responses. Stimulated recalls, carried out in the L1, English, may have shed light on the questions raised by Kempe and MacWhinney about the perceived difference between the L1 and the L2. Learners could have been asked to introspect about these perceptions, which would have provided an additional source of data. The level of learner metalinguistic awareness about their second languages could have been explored by carrying out stimulated recalls too. Of course, stimulated recall protocols need to be used with caution in terms of perceptions, yet in this case they may have shed light on a complex question.

< previous page

page_122

next page >

< previous page

page_123

next page > Page 123

Input and Input Processing Reading Lee (1998) explored whether reading comprehension and input processing were affected by specific morphological characteristics in the input. Seventy-one English Ll university students enrolled in a second-semester Spanish course read a passage in Spanish about a salesperson describing various features of computers. The passage contained 11 subjunctive verb forms, 9 of which were targeted for modification. A subjunctive group read the passage in its original form with the targeted verbs in the subjunctive. An infinitive group read the passage with the targeted verbs in the infinitive form (no morphological encoding of person, gender, etc.). An invented group read the passage with the targeted verbs in an invented form (the verb stem plus -u). This form appeared to convey morphological encoding but in fact did not as it was artificially created. After reading the passage, the students completed a recall exercise in which they wrote down (in the Ll) everything they could remember about the passage. Next, the students completed a word identification activity in which they reviewed a list of 100 verbs and nouns and placed a mark next to each word that had appeared in the story. Lee's results indicate that recall by the subjunctive group was lower than that of the infinitive or invented group (there was no significant difference between the infinitive and invented groups). Additionally, across all groups, general recall for students who mentioned that the passage involved a salesperson was higher than for students who did not mention the salesperson. For the word identification activity, there were no significant differences among the groups. Finally, there was no correlation between performance on the recall activity and the word identification activity. Lee concluded that learners comprehend better when morphological forms in the input are less complex and that learners may detect forms in the input even when they are reading for meaning. He also suggested that the lack of correlation between comprehension and word identification indicates that learners comprehend content and process forms via different processes. He attributed the finding that learners who mentioned the salesperson had higher recall than those who did not to rhetorical organization. There could have been additional exploration through the use of introspective methods. Either think-aloud protocols or stimulated recall

< previous page

page_123

next page >

< previous page

page_124

next page > Page 124

interviews might have revealed more about the learners' processing. By commenting on their thought processes while reading the passage, the learners could have provided insights into the processes they used for comprehension. In addition, the learners' comments may have revealed more information about the input processing of the verbal morphology. It would have been interesting to know how learners reacted to the different presentation of verbs, particularly the invented forms. Amount and Type of Exposure Leow (1998) compared the effects of different amounts and types of exposure to irregular third person singular and plural preterite stem changing on -ir verbs in Spanish. Four groups of 88 English Ll university students enrolled in a first-semester Spanish class (who had had only 7.5 hours of instruction) participated in the study. The single exposure, teacher-centered group received an explanation of the irregular forms, followed by drill exercises. The multiple exposure, teacher-centered group received the same instruction and received it again 3 weeks later. The single exposure, learner-centered group completed a crossword puzzle designed to draw their attention to the irregular forms. The multiple-exposure, learner-centered group completed another crossword puzzle 3 weeks later. All the students in the learner-centered groups completed think-alouds while they were working on the crossword puzzles. Three posttests consisting of a multiple-choice recognition exercise and a fill-inthe-blank activity were conducted immediately after the treatment, 3 weeks later (immediately after second treatment for multiple exposure groups), and 11 weeks later. The results indicate that the multiple-exposure groups outperformed the single-exposure groups on the second and third posttests. Furthermore, the gain by the multiple-exposure groups at the first posttest was maintained over time. In contrast, the single-exposure groups demonstrated a significant decline from Posttest 1 to Posttest 3. With regard to the type of exposure, the learner-centered groups outperformed the teacher-centered groups on all posttests. A great deal of information about the learning processes of the students in the learner-centered groups was obtained through the think-aloud comments, but Leow's study did not elicit information about the learning processes of students in the teachercentered groups. His chosen methodological tool, think-alouds, might not have been possible in

< previous page

page_124

next page >

< previous page

page_125

next page > Page 125

the teacher-centered context; however, stimulated recalls could have been carried out. The teacher-centered sessions could have been videotaped and the tapes played back so that the students could introspect about their thoughts while the instruction was in progress, watching themselves on the videotape for the recall stimulus. It should be noted that carrying out stimulated recalls before all posttests were completely is clearly inadvisable as is a long gap between the original activity and recall. Thus, if such an approach were adopted, it might be best only to obtain recall data from a subset of learners in the teacher-centered group. These learners should then not participate in the posttests. Restricting the participant pool for tests may be why stimulated recall were not carried out in this study. L2 Reading Comprehension Barry and Lazarte (1998) explored domain-related knowledge, syntactic complexity, and reading topic in the context of how they affected inference generation in the written recalls of Spanish learners (with English Lls). Two groups of participants, with high knowledge and low knowledge, read three Spanish passages, each at a different level of (syntactically measured) complexity. Knowledge level was operationalized as students' exposure to a specific content domain and experience with the targeted text types. Barry and Lazarte examined within-text inferences, elaborative inferences, and incorrect inferences. They counted the total number of inferences as providing evidence for the richness of the mental model and the type of inferences as providing information about the nature and accuracy of the mental model. They found that high-level readers generated a richer and more accurate mental model than did low-knowledge readers. A related finding was that the level of complexity and the reading topics indicated a complex pattern of influence on inference generation. On the basis of their findings, Barry and Lazarte suggested (among other conclusions) that high-knowledge readers in an L2 "shift to a top-down or knowledge-driven process when the increased syntactic complexity requires them to maintain clauses in working memory and, simultaneously, to activate information from previous segments of the text" (p. 190). They pointed to the need for further investigations to determine the potential of the resource allocation idea for describing the behavior of second language learners. They

< previous page

page_125

next page >

< previous page

page_126

next page > Page 126

specifically asked what kinds of resources are required for the written recall test. The immediate recall production test could be followed up by a stimulated recall interview, using the reading passage and the written recall of the passage as the stimuli. Learners could also be shown videotapes of themselves completing the written recall and could be asked to introspect about their thought processes while carrying out the task. Researchers could gain potentially valuable insights into the construction of the mental model as well as into any shift in resource allocation and the type of resources learners thought they needed for the written recall of the reading passage. Of course, learners might find recalling the processes they used as they recalled a reading passage in writing to be somewhat challenging. Careful pilot testing and question construction would be essential. Oral Interaction Dialogue Swain and Lapkin (1998) carried out a study of learner-learner dialogue, finding support for the theoretical position that dialogue can be both a means of communication and a cognitive tool. They analyzed language related episodes (LREs) of two Grade 8 French immersion (English Ll) students who were working together to solve a jigsaw task. They defined LREs as any part of a dialogue where students talk about the language they are producing, question their language use, or correct themselves or others. They classified the LREs as lexis-based or form-based. The task involved the students working out a story line, writing it out, and resolving the linguistic problems that arose during the task through discussion. They used both their Ll and their L2 to resolve the problems. Swain and Lapkin claimed that the two students jointly develop the story line, co-construct the knowledge they need to express the meaning they want and co-construct knowledge about language in the process. An example they used shows one student, Rick, finding out the French word for pillow, oreiller, from his partner, Kim, and double checking that he has the correct item by pointing to the picture and saying, "is that l'oreiller?" and writing it down (p. 332). On a posttest, Rick and Kim both got the word for pillow correct. Swain and Lapkin called for "future work to combine an analysis of students' collaborative dialogues with follow-up interviews in order to

< previous page

page_126

next page >

< previous page

page_127

next page > Page 127

derive a more fine-grained understanding of the mental processes" (p. 333). They also questioned whether certain aspects of tasks are found by students to be appealing or unappealing, conducive or not conducive to learning. An excellent way to conduct such follow-up interviews would be to show the students videotapes of themselves carrying out the tasks and ask them to introspect about their own and their perceptions of their partners' contributions to and understanding of the LREs. With such recall support, the actual processes could be tapped more closely in introspection. Students could then be explicitly asked their opinions about certain aspects of the tasks. Negotiation Foster (1998) investigated negotiation of meaning in the second language classroom. The study specifically considered modified interaction in group and pair work as a function of prior negotiation. Twenty-one ESL students from a variety of Lls participated in task-based activities. The dyads engaged in a grammar-based task in which students had to compose questions to elicit particular answers and a picture difference task. This latter constituted a required information exchange. The small groups engaged in a consensus activity in which a problem was given and they had to reach consensus as to the solution and a map activity. This latter was classified as a required information exchange. Analyses were carried out using the c-unit as a basis. Cunits are independent utterances that contain meaning (either referential or pragmatic). The results of Foster's study contradict previous studies in the literature in that the incidence of negotiation was low and, further, there was a low incidence of modified utterances as a result of negotiation. In discussing her results, Foster noted "We now need to explore why so many of the students in this study were disinclined to initiate or pursue negotiation for meaning" (p. 18). We argue that this is precisely the type of information that could be gained through a stimulated recall procedure. For example, Foster presented the following exchanges as examples of the lack of response to a signal of incomprehension: Example 18 (From Foster, 1998, pp. 1516) A:

'the sports field, swimming pool and equipment may be used free of charge.'

B:

Free of charge? What is that?

(table continued on next page)

< previous page

page_127

next page >

< previous page

page_128

next page > Page 128

(table continued from previous page) C:

(laughs) Yes.

A:

sports day.

A:

There is this one, this one, and after to camping site near Oldfield.

B:

Oldfield?

C:

Anyway, the best think I think is er camping.

In both excerpts in Example 18, there is a signal that comprehension is questionable. In the first case, it is direct: "What is that?" and in the second it is indirect: "Oldfield?" Using stimulated recall answers to these questions (and others) may be able to move researchers from the realm of speculation to the realm of greater certainty. Syntactic Processing Hoover and Dwivedi (1998) conducted a study of syntactic processing by fluent L2 speakers. The participants were native speakers of French (n = 48) and English speakers fluent in French (n = 51). Clitics in causative and noncausative sentences, as in Examples 19 and 20, were used to investigate syntactic processing. Example 19 (From Hoover and Dwivedi, 1998) Clitic in non-causative: Il aimait tranquillement le goûter avec son fromage doux préferé. He loved to taste it quietly with his favorite mild cheese. Example 20 (From Hoover and Dwivedi, 1998) Clitic in causative: Il le faisait tranquillement goûter avec son fromage doux préferé. He had it be tasted quietly with his favorite mild cheese. Participants read target sentence pairs. These sentences were embedded in 72 filler sentence pairs. Each pair had a context sentence and a target or filler sentence. The main task was conducted on-line. Participants read sentences word by word, pressing a space bar for each

< previous page

page_128

next page >

< previous page

page_129

next page > Page 129

new word. Sentence pairs were followed by comprehension questions. Following the main experiment, participants were given a standardized reading comprehension test (Wisconsin College-Level Placement Test). The purpose of this test was to divide the L2 readers into two groups (i.e., high and low) according to their reading proficiency. Their specific hypothesis, which was confirmed, was that slow L2 readers would be less efficient in their processing of syntactic information. However, as with many empirical studies, questions remained and alternative explanations were suggested. One question had to do with whether or not the results could have been a factor of word recognition. Stimulated recall, with examples from the task, could shed light on this issue. Videotapes of each participant could have been replayed, and questions relating to the length of time used to press the space bar for the next word could have been asked (e.g., ''I noticed that it took x seconds for you to press the space bar for the next word. What were you thinking about during that time?"). Vocabulary Incidental Vocabulary Learning A study by Joe (1998) on incidental vocabulary learning is an interesting one for the exploration of stimulated recall (see studies by Fraser, 1999, and Paribakht & Wesche, 1999, for examples of verbal protocols in incidental vocabulary learning). Participants were 48 ESL learners of a variety of Lls. They were divided into three groups: experimental, comparison, and control. Prior knowledge was obtained, including knowledge of particular words related to the study, general word knowledge, and overall proficiency. On the basis of background knowledge, participants were divided into two groups (i.e., high and low), and these individuals were distributed evenly across the three experimental groups. A passage on the topic of pain was selected, and from that passage 12 words were targeted. In the experimental group, participants carried out a read-and-retell task after they had done a background-knowledge activation task in which questions based on the text were posed. Participants could not refer to the input text during the retelling phase. The experimental group was given specific instruction on learning strategies. In particular, they were given instruction on ways

< previous page

page_129

next page >

< previous page

page_130

next page > Page 130

(a) to recall prior experience and knowledge to make sense of unfamiliar concepts or words in the text, that is, learners were told to add their own examples, experience, and knowledge to information from the text, and to offer personal opinions and comments; (b) to paraphrase, use synonyms, examples, or analogy; and (c) to discuss why some examples of learners' generative strategies were better than others. (Joe, 1998, p. 367) In the comparison group, the procedure was identical to the experimental group's procedure with two exceptions: They had access to the input text as they retold it, and they did not have any strategy instruction. The control group had no treatment; they continued with regular class instruction during the time of the experiment. They took the pre- and posttests only. The pretest comprised a self-report interview based on 28 words, including the 12 target words, and a read-and-retell task using think-aloud procedures. The posttest included the self-report interview, which focused only on the 12 target words, and two multiple-choice tests. In general, the results suggest that incidental vocabulary learning does take place. The reading and recall tasks without a specific focus on the vocabulary items in question, yielded better results than those of the control group. The study also suggests that prior background knowledge affects vocabulary learning. This study is particularly interesting in that the comparison group had access to the text in the retell task. During retellings, they used think-aloud sorts of phrases such as, "I'm not sure, but I think. . . . " It would be an easy step to turn this into a stimulated recall procedure. Gass (1999) argued that incidental vocabulary learning is often misnamed because researchers investigate the concept from a teacher perspective. That learning appears to be incidental does not mean that there hasn't been a specific focus on the word in question. A stimulated recall procedure (after posttests) might reveal what learners were focusing on. For example, after a comparison of a given learner's performance on the pretest and posttest and the observation that word x was learned, but word y was not learned, the retell (particularly the retell for the comparison group that included access to the original input

< previous page

page_130

next page >

< previous page

page_131

next page > Page 131

text) might be used to probe what learners were thinking about as they encountered words x and y. Acquisition of Words in an Unknown Language Lotto and de Groot (1998) investigated the acquisition of words in a new language (i.e., Italian) by Dutch speakers. The variables considered were learning method, word frequency, and cognate status. Fifty-six native speakers of Dutch participated in their study. None knew any Italian, although some were familiar with other Romance languages. Participants were presented with 80 pictures representing 20 words in each of four categories: high-frequency cognates, high-frequency noncognates, lowfrequency cognates, and low-frequency noncognates. There were two conditions. In the word-learning condition, participants were presented with a Dutch word and the Italian translation. In the picture-learning condition, a stimulus picture was presented with only the Italian word. All stimulus material was presented on a computer. The experimental session included a learning phase and a test phase. The 56 participants were divided into four groups of 14 each. In the learning phase, two of the groups were part of the word-learning group, and the others were in the picture-learning group. Material was presented three times for each stimulus. The test phase resulted in further divisions of the groups. One group in the word-learning condition and one group in the picture-learning condition were presented with material congruent with the learning phase, whereas the others received stimuli that were unlike those in the learning phase. In other words, half the participants in each condition received Dutch words as stimuli and were asked for the equivalent Italian word; the other half received pictures as stimuli and were asked for the appropriate Italian word. In a follow-up test phase, participants did exactly the same task as in the first round of the experiment; they did the same learning task and the same testing task, both of which placed each participant in the same condition as before. In short, the word-learning condition tended to lead to better performance than the picture-learning condition (effects were also found for cognates and for high-frequency words). However, Van Hell and Candia Mahn (1997) presented conflicting results, which might have been affected by learning experience. An additional layer of data collection using a stimulated recall procedure might have been useful. For example, participants could be shown a videotape of themselves per-

< previous page

page_131

next page >

< previous page

page_132

next page > Page 132

forming the testing phase. Examples could be culled from each condition and from each category of word types with questions such as, "You seemed to have taken a long time to respond to this item. What were you thinking about at the time?" In this way, some insight may be gained from all areas of the study, including information from participants who had a different testing as opposed to learning phase and information about different word types (e.g., cognates vs. noncognates; high-frequency vs. lowfrequency words). In Table 5.1 is a summary of the studies discussed in this section. Conclusion We conclude this book with the claim that stimulated recall data can provide valuable information about some of the complex processes involved in learning second languages. The data, however, like all data in second language research, must be carefully elicited and interpreted. Data must be interpreted within the framework of current theoretical concerns, and in conjunction with other compatible and reliable data. We have presented ways of eliciting and interpreting stimulated recall data. We have also discussed pitfalls and areas of controversy. We believe that with informed use, this methodology can make an invaluable contribution to a careful and wellthought out research project.

< previous page

page_132

next page >

< previous page

page_133

next page > Page 133

TABLE 5.1 The Potential of Stimulated Recall Methodology in Second Language Research Research Area Author(s) Overview of Study Additional Issues/Questions Raised Stimulated Recall Suggestions Where Supplementary Data May Be Helpful L2 Munro Perception ofIndividual listeners were affected by Prompting listeners to introspect about perception foreignthe presence of noise when why they made certain judgments (using accented interpreting L2 production. Why? their ratings for stimuli) speech with Did vocal pitch, prosody or speech noise rhythm have an effect? Riney Change over Questions raised about perception of Prompting listeners to introspect about L2 perception and time in listeners who were trained why they made certain judgments Flege global researchers in L2 (ratings as stimuli) foreign phonetics/phonology accent and liquids Classroom Lyster Recasts, Questions raised about intentions of Prompting learners and teachers to interaction repetition teachers in providing recasts and introspect about intentions and and perception of learners about perceptions using videotaped classroom ambiguity corrective function of recasts versus interaction as stimulus noncorrective repetition Oral MehnertDifferent Questions raised about whether Prompting learners to introspect about production/ amounts of planning time is used to prepare how planning time was used utilizing planning planning cognitively or linguistically about videotapes and written plans as the time on task how learners approach planning stimulus: questions focusing on how performance planning time was used (table continued on next page)

< previous page

page_133

next page >

< previous page

page_134

next page > Page 134

(table continued from previous page) Research Area Author(s) Overview of Study Additional Issues/Questions Raised Stimulated Recall Suggestions Where Supplementary Data May Be Helpful Pragmatics Bardovi- Differences Questions raised about the variables Obtaining retrospective Harlig between ESL and that intervene between stages of comments from learners about and EFL learners in noticing and targetlike production intervening variables, using Dörnyei pragmatic and judgment data obtained while grammatical watching the video scenarios as awareness stimulus L2 Kempe Comprehension Questions raised about whether Prompting learners to introspect comprehension and Mac of overt differences between learners of about metalinguistic awareness, Whinneymorphological Russian and of German differed using the task responses as the case marking because of metalinguistic awareness stimuli of case marking Input and Lee Reading Questions raised about whether Carrying out on-line think-aloud input comprehension learners detect forms in input. Even protocols with some learners and processing and input when they read for meaning, some stimulated recalls with others, processing learners' rhetorical organization using videotapes and the task skills were higher responses as the stimuli Input and Leow Effects of amount Questions raised about the teacher- Carrying out stimulated recall input and type of input centered group, who did not protocols with some of the processing exposure on L2 complete the think-aloud tasks of learners using videotapes and develop. the learner-centered groups (it transcripts as the stimuli would have interfered with the lesson) (table continued on next page)

< previous page

page_134

next page >

< previous page

page_135

next page > Page 135

(table continued from previous page) Research Area Author(s) Overview of Study Additional Issues/Questions Raised Stimulated Recall Suggestions Where Supplementary Data May Be Helpful L2 Barry Inference Questions raised about the potential of a Carrying out on-line thinkcomprehension and generation: resource allocation model, where aloud protocols with some Lazarte domainresources are redirected at certain learners and stimulated recalls knowledge, points; also questions about what sort of with others, using videotapes, complexity, and resources learners needed in order to reading passage, and written topic effects carry out the experimental task recall as the stimuli Oral Swain Learner dialogue Questions raised about combining Prompting the two learners to interaction and can be seen as a analyses of students' collaborative introspect about their thoughts Lapkin means of dialogues with follow-up interviews to during the interaction (After communication achieve a better understanding of the post-tests), using and a cognitive mental processes videotapes of the collaborative tool dialogues as the stimuli Negotiation Foster Investigation of Questions remain as to why predictions Posing questions to probe meaning and replication of prior research learners' thoughts during the negotiation in findings of negotiation and resulting time of the original interaction the classroom modifications are not borne out using audiotapes as stimulus Syntactic Hoover Processing of Were the results partly affected by Asking questions in which processing and converging (L1 word recognition? participants are asked to recall Dwivediand L2) and what they were thinking about nonconverging when there were particularly gram-matical fast or slow responses. Use structures and video as reading (table continued on next page)

< previous page

page_135

next page >

< previous page

page_136

next page > Page 136

(table continued from previous page) Research Area Author(s) Overview of Study Additional Issues/Questions Raised Stimulated Recall Suggestions Where Supplementary Data May Be Helpful Incidental Joe Conditions that promote Is there really a lack of Probing differences on gain vocabulary incidental vocabulary learning focus when incidental scores for different words using learning learning takes places? an audio or video stimulus VocabularyLotto The effect of learning method, Questions remain Prompting learners with and deword frequency, and cognate concerning conflicting questions relating to reaction Groot status on the acquisition of new results between this study time. Use task performance as a words and others stimulus

< previous page

page_136

next page >

< previous page

page_137

next page > Page 137

References Abraham, R., & Vann, R. (1996). Using task products to assess second language learning processes. Applied Language Learning, 7, 6189. Aichison, J. (1994). Words in the mind: An introduction to the mental lexicon (2nd ed.). Oxford: Blackwell. Alanen, R. (1995). Input enhancement and rule presentation in second language acquisition. In R. Schmidt (Ed.), Attention and awareness in foreign language learning (pp. 259302). Honolulu: University of Hawai'i, Second Language Teaching and Curriculum Center. Anderson, M. (1985). Some evidence on the effect of verbalization on process: A methodological note. Journal of Accounting Research, 23, 843852. Anderson, N. (1989). Reading comprehension tests versus academic reading: What are Second Language readers doing? Unpublished doctoral dissertation, University of Texas at Austin. Anderson, N. (1991). Individual differences in strategy use in second language reading and testing. Modern Language Journal, 75, 460472. Aston, G. (1986). Trouble-shooting in interaction with learners: The more the merrier? Applied, Linguistics, 7, 128143. Bailey, N., Madden, C., & Krashen, S. (1974). Is there a "natural sequence" in adult second language learning? Language Learning, 24, 235243. Bard, E., Robertson, D., & Sorace, A. (1996). Magnitude estimation of linguistic acceptability. Language, 72, 3268. Bardovi-Harlig, K., & Dörnyei, Z. (1998). Do language learners recognize pragmatic violations? Pragmatic versus grammatical awareness in instructed L2 learning. TESOL Quarterly, 32, 233262. Barry, S., & Lazarte, A. (1998). Evidence for mental models: How do prior knowledge, syntactic complexity, and reading topic affect inference generation in a recall task for nonnative readers of Spanish? Modern Language Journal, 82, 176193. Bartelt, G. (1997). The ethnography of second language production. International Review of Applied Linguistics, 35, 2335. Benoit, W. (1995). Accounts, excuses, and apologies: A theory of image restoration strategies. Albany: State University of New York Press. Bettman, J., & Park, C. (1979). Implications of a constructive view of choice for analysis of protocol data: A coding scheme for elements of choice processes (Working Paper No. 75). Los Angeles: University of California, Center for Marketing Studies. Biggs, S., Rosman, A., & Sergenian, G. (1993). Methodological issues in judgment and decision-making research. Concurrent verbal protocol validity and simultaneous traces of process. Journal of Behavioral Decision Making, 6, 187206. Birdsong, D. (1989). Metalinguistic performance and interlinguistic competence. Berlin: Springer Verlag. Block, E. (1986). The comprehension strategies of second language readers. TESOL Quarterly, 20, 463494. Bloom, B. (1954). The thought processes of students in discussion. In S. J. French (Ed.), Accent on teaching, Experiments in general education (pp. 2346). New York: Harper. Bloomfield, L. (1914). Introduction to the study of language. New York: Henry Holt and Company. Bloomfield, L. (1933). Language. New York: Holt, Rinehart & Winston. Blumenthal, A. (1970). Language and psychology: Historical aspects of psycholinguistics. New York: John Wiley.

< previous page

page_137

next page >

< previous page

page_138

next page > Page 138

Boritz, J. E. (1986). The effect of research method on audit planning and review judgments. Journal of Accounting Research 26, 335348. Bosher, S. (1998). The composing processes of three Southeast Asian writers at the post-secondary level: An exploratory study. Journal of Second Language Writing, 7, 205241. Brice, C. (1995, March). ESL writers' reactions to teacher commentary: A case study. Paper presented at the Annual Meeting of the Teachers of English to Speakers of Other Languages, Long Beach, CA. (ERIC Document ED 394312) Brown, A. (1993). The role of test taker feedback in the test development process: test taker's reactions to a tape-mediated test of proficiency in spoken Japanese. Language Testing 10, 277303. Bruner, J., Goodnow, J., & Austin, G. (1956). A study of thinking. New York: John Wiley. Buck, G. (1991). The testing of listening comprehension: An introspective study. Language Testing, 8, 6791. Calderhead, J. (1981a). A psychological approach to research on teachers' classroom decision making. British Educational Research Journal 7, 5157. Calderhead, J. (1981b). Stimulated recall: A method for research on teaching. British Journal of Educational Psychology 51, 211217. Canguilhem, G. (1989). The normal and the pathological (Translated by C. Fawcett in collaboration with R. Cohen). New York: Zone Books. Canguilhem, G. (1994). Études d'histoire et de philosophie des sciences [Studies in the history and philosophy of science] (7th ed.). Paris: Librarie Philosophique J. Vrin. Carroll, S. & Meisel, J. (1990). Universals and second language acquisition: Some comments on the state of current theory. Studies in Second Language Acquisition, 12, 201208. Cavalcanti, M. (1987). Investigating FL reading performance through pause protocols. In C. Færch & G. Kasper (Eds.), Introspection in second language research (pp. 230250). Clevedon: Multilingual Matters. Chern, C-L. (1993). Chinese students' word solving strategies in reading in English. In T. Huckin, M. Haynes, & J. Coady (Eds.), Second language reading and vocabulary learning (pp. 6785). Norwood, NJ: Ablex. Chi, M. (1997). Quantifying qualitative analyses of verbal data: A practical guide. Journal of the Learning Sciences, 6, 271315. Chi, M., Feltovich, P., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5, 121152. Chomsky, N. (1957). Syntactic structures. Berlin: Mouton. Chomsky, N. (1959). Review of "Verbal Behavior" by B. F. Skinner, Language, 35, 2658. Clark, C., & Peterson, P. (1981). Stimulated-recall. In B. R. Joyce., C. C. Brown, & L. Peck (Eds.), Flexibility in teaching: An excursion into the nature of teaching and training (pp. 256261). New York: Longman. Cohen, A. (1984). On taking tests: What the students report. Language Testing, 1, 7081. Cohen, A. (1987). Using verbal reports in research on language learning. In C. Færch & G. Kasper (Eds.), Introspection in second language research (pp. 8295). Clevedon: Multilingual Matters. Cohen, A. (1994). English for academic purposes in Brazil: The use of summary tasks. In C. Hill & K. Parry (Eds.), From testing to assessment: English as an international language (pp. 174204). London: Longman. Cohen, A. (1996). Verbal reports as a source of insights into second language learner strategies. Applied Language Learning, 7, 524. Cohen, A. (1998). Strategies in learning and using a second language. London: Longman.

< previous page

page_138

next page >

< previous page

page_139

next page > Page 139

Cohen, A., & Aphek, E. (1979). Easifying second language learning (Report submitted to the Jacob Hiatt Institute). Jerusalem: Hebrew University of Jerusalem, School of Education. (ERIC Document ED 163753) Cohen, A., & Aphek, E. (1981). Easifying second language learning. Studies in Second Language Acquisition, 3, 221235. Cohen, A., & Cavalcanti, M. (1987). Giving and getting feedback on compositions: A comparison of teacher and student verbal report. Evaluation and Research in Education, 1, 6373. Cohen, A., & Cavalcanti, M. (1990). Feedback on compositions: Teacher and student verbal reports. In B. Kroll (Ed.), Second language writing: Research insights for the classroom (pp. 155177). Cambridge: Cambridge University Press. Cohen, A., & Hosenfeld, C. (1981). Some uses of mentalistic data in second language research. Language Learning, 31, 285313. Cohen, A., & Olshtain, E. (1993). The production of speech acts by EFL learners. TESOL Quarterly, 27, 3356. Cohen, A., Weaver, S., & Li, T-Y. (1995). The impact of strategies-based instruction on speaking a foreign language (Research Report). Minneapolis: University of Minnesota, Center for Advanced Research on Language Acquisition (CARLA). Colker, L. (1982). Teachers' interactive thoughts about pupil cognition. Unpublished doctoral dissertation, University of Illinois, Urbana-Champaign. Conners, R. D. (1978). Using stimulated recall in naturalistic settings: Some technical procedures (Tech. Paper No. 78-2-1). Edmonton, Canada: University of Alberta, Centre for Research in Teaching. Cook, V. (1990). Timed comprehension of binding in advanced L2 learners of English. Language Learning, 40, 557599. Corder, S. P. (1973). The elicitation of interlanguage. In J. Svartvik (Ed.), Errata: Papers in error analysis (pp. 3648). Lund: CKW Geerup. Cowan, R., & Hatasa, Y. (1994). Investigating the validity and reliability of native speaker and second-language learner judgments about sentences. In E. Tarone, S. Gass, & A. Cohen (Eds.), Research methodology in second-language acquisition (pp. 287302). Mahwah, NJ: Lawrence Erlbaum Associates. Cumming, A. (1989). Writing expertise and second-language proficiency. Language Learning, 39, 81141, Davies, W., & Kaplan, T. (1998). Native speaker vs. L2 learner grammaticality judgements. Applied Linguistics, 19, 183203. Dennett, D. (1987). The international stance. Cambridge, MA: MIT Press. Descartes, R. (1637). Discourse on method (Translated by L. La Fleur, 1960). New York. Bobbs Merrill. DiPardo, A. (1994). Stimulated recall in research on writing: An antidote to ''I don't know, it was fine." In P. Smagorinsky (Ed.), Speaking about writing: Reflections on research methodology (pp 163181). Thousand Oaks, CA: Sage. Dörnyei, Z., & Kormos, J. (1998). Problem-solving mechanisms in L2 communication: A psycholinguistic perspective. Studies in Second Language Acquisition, 20, 349386. Dulay, H., & Burt, M. (1974a). Natural sequences in child second language acquisition. Language Learning, 24, 3753. Dulay, H., & Burt, M. (1974b). You can't learn without goofing. In J. Richards (Ed.), Error analysis: Perspectives on second language acquisition (pp. 95123). London: Longman.

< previous page

page_139

next page >

< previous page

page_140

next page > Page 140

Dulay, H., & Burt, M. (1975). Creative construction in second language learning and teaching. In M. Burt & H. Dulay (Eds.), On TESOL '75: New directions in second language learning (pp. 2132). Washington, DC: TESOL. Dyson, A., & Freedman, S. (1991). Writing. In J. Jensen, J. Flood, D. Lapp, & J. R. Squire (Eds.), Handbook of research on teaching the English language arts (pp. 754774). New York: MacMillan. Ellis, R. (1990). Grammaticality judgments and learner variability. In H. Burmeister and P. Rounds (Eds.), Variability in second language acquisition: Proceedings of Tenth Meeting of the Second Language Research Forum (pp. 2560). Eugene, OR: University of Oregon. Ellis, R. (1991). Grammaticality judgments and second language acquisition. Studies in Second Language Acquisition, 13, 161186. Elstein, A., Shulman, L. S., & Spaafka, A. S. (1978). Medical problem solving: An analysis of clinical reasoning. Cambridge, MA: Harvard University Press. Enkvist, I. (1995). Intellectual and linguistic progress in foreign language students: Students' development during their first year of Spanish at a Swedish university. Studies of Higher Education and Research (Report 34). (ERIC Document ED 390253) Erickson, F., & Mohatt, G. (1977). The social organization of participation structure in two classrooms of Indian students. Ottawa, Canada: Department of Indian Affairs and Northern Development. Ericsson, K., & Simon, H. (1980). Verbal reports as data. Psychological Review, 87, 215251. Ericsson, K., & Simon, H. (1984). Protocol analysis. Cambridge, MA: MIT Press. Ericsson, K., & Simon, H. (1987). Verbal reports on thinking. In C. Færch & G. Kasper (Eds.), Introspection in second language research (pp. 2453). Clevedon: Multilingual Matters. Ericsson, K., & Simon, H. (1993). Protocol analysis: Verbal reports as data (2nd ed.). Cambridge, MA: MIT Press. Ericsson, K., & Simon, H. (1996). Protocol analysis: Verbal reports as data (3rd ed.). Cambridge, MA: MIT Press. Færch, C., & Kasper, G. (1986). One learnertwo languages: Investigating types of interlanguage knowledge. In J. House & S. Blum-Kulka (Eds.), Interlingual and intercultural communication (pp. 211227). Tübingen: Gunter Narr. Færch, C., & Kasper, G. (1987). From product to processIntrospective methods in second language research. In C. Færch & G. Kasper (Eds.), Introspection in second language research (pp. 523). Clevedon: Multilingual Matters. Feldman, U., & Stemmer, B. (1987). Thin__aloud a__retrospective da__in C-te__taking: diff__languages__diff__learners__sa__approaches? In C. Færch & G. Kasper (Eds.), Introspection in second language research (pp, 251266). Clevedon: Clevedon: Multilingual Matters. Fogarty, J., Wang, M., & Creek, R. (1983). A descriptive study of experienced and novice teachers' interactive instructional thoughts and actions. Journal of Educational Experience, 77, 2232. Foster, P. (1998). A classroom perspective on the negotiation of meaning. Applied Linguistics, 19, 123. Fraser, C. (1999). Lexical processing strategy use and vocabulary learning through reading. Studies in Second Language Acquisition, 21, 225241 Garner, R. (1988). Verbal-report data on cognitive and metacognitive strategies. In C. E. Weinstein, E. T. Goetz, & P. A. Alexander (Eds.), Learning and study strategies: Issues in assessment, instruction and evaluation (pp. 6376). New York: Academic Press. Gass, S. (1979). Language transfer and universal grammatical relations. Language Learning, 29, 327344.

< previous page

page_140

next page >

< previous page

page_141

next page > Page 141

Gass, S. (1994). The reliability of second-language grammaticality judgments. In E. Tarone, S. Gass, & A. Cohen (Eds.), Research methodology in second-language acquisition (pp. 303322). Hillsdale, NJ: Lawrence Erlbaum Associates. Gass, S. (1997). Input, interaction, and the second Language leaner. Mahwah, NJ: Lawrence Erlbaum Associates. Gass, S. (1998). Apples and oranges: Or, why apples are not orange and don't need to be. Modem Language Journal, 82, 8390. Gass, S. (1999). Discussion: Incidental vocabulary learning. Studis in Second Language Acquisition, 21, 319333. Gass, S., & Selinker, L. (1994). Second language acquisition: An introductory course. Hillsdale, NJ: Lawrence Erlbaum Associates. Gass, S., & Varonis, E. (1994). Input, interaction and second language production. Studies in Second Language Acquisition, 16, 283302. Gatbonton, E. (1999). Investigating experienced ESL teachers' pedagogical knowledge. Modern Language Journal, 83, 3551. Gazzaniga, M. (1998, July). The split-brain revisited. Scientific American, 5155. Gerloff, P. (1987). Identifying the unit of analysis in translation: Some uses of think-aloud protocol data. In C. Færch & G. Kasper (Eds.), Introspection in second language research (pp. 135158). Clevedon: Multilingual Matters. Glahn, E. (1980). Introspection as a method of elicitation in interlanguage studies. Interlanguage Studies Bulletin, 5, 119128 Gordon, C. (1987). The effect of testing method on achievement in reading comprehension test in English as a fore language. Unpublished master's thesis, Tel Aviv University, Ramat-Aviv, Israel. Gosden, H. (1996). Verbal reports of Japanese novices' research writing practices in English. Journal of Second Language Writing 5, 109128. Goss, N., Zhang, Y-H., & Lantolf, J. (1994). Two heads may be better than one: Mental activity in second-language grammaticality judgments. In E. Tarone, S. Gass, & A. Cohen (Eds.), Research methodology in second-language acquision (pp. 263286). Mahwah, NJ: Lawrence Erlbaum Associates. Gu, Y. (1994). Vocabulary learning strategies of good and poor Chinese EFL learners. In N. Bird, P. Falvey, A. Tsui D. M. Allison, A. McNeill, & R. J. Webb (Eds.), Language and learning (pp. 376401). Hong Kong: Institute of Language and Education, Education Department. Haastrup, K. (1987). Using thinking aloud and retrospection to uncover learners' lexical inferencing procedures. In C. Færch & G. Kasper (Eds.), Introspection in second language research (pp. 197212). Clevedon: Multilingual Matters. Habermas, J. (1979). Communication and the evolution of society. Boston: Beacon Press. Hample, D. (1984). On the use of self-reports. Journal of the American Forensic Association, 20, 140153. Hatch E., & Lazaraton, A. (1991). The research manual: Design and statistics for applied linguistics. Boston: Heinle and Heinle. Hawkins, B. (1985). Is the appropriate response always so appropriate? In S. Gass & C. Madden (Eds.), Input in second language acquisition (pp. 162178). Rowley, MA: Newbury House. Hill, K. (1994). The contribution of multi-informant feedback to the development and validation of an oral proficiency test in two formats. Unpublished master's thesis, Department of Linguistics and Language Studies, University of Melbourne, Australia. Hillway, T. (1969). Handbook of of educational research. A guide to methods and materials. Boston: Houghton Mifflin.

< previous page

page_141

next page >

< previous page

page_142

next page > Page 142

Hölscher, A., & Möhle, D. (1987). Cognitive plans in translation. In C. Færch & G. Kasper (Eds.), Introspection in second language research (pp. 113134). Clevedon: Multilingual Matters. Hoover, M., & Dwivedi, V. (1998). Syntactic processing by skilled bilinguals. Language Learning, 48, 129. Hosenfeld, C. (1976). Learning about learning: Discovering our students' strategies. Foreign Language Annals, 9, 117129. Hosenfeld, C. (1977). A preliminary investigation of the reading strategies of successful and nonsuccessful second language learners. System; 5, 110123. Hosenfeld, C. (1979). Cindy: A learner in today's foreign language classroom. In W. Borne (Ed.), The foreign language leamer in today's classroom environment (pp. 5375). Montpelier: VT: Northwest Conference on the Teaching of Foreign Languages. Hosenfeld, C. (1984). Case studies of ninth grade readers. In J. C. Alderson & A. H. Urquhart (Eds.), Reading in a foreign language (pp. 231249). London: Longman. Housner, L., & Griffey, D. (1983). Teacher cognition: Differences in planning and interactive decision making between experienced and inexperienced teachers. Paper presented at the annual meeting of the American Educational Resarch Association, Montreal, Canada. Huber, G., & Mandl, H. (Eds.). (1982). Verbale daten [Verbal data]. Beltz: Weinheim and Basel. Huckin, T., & Bloch, J. (1993). Strategies for inferring word-meanings in context: A cognitive model. In T. Huckin, M. Haynes, & J. Coady (Eds.), Second language reading and vocabulary learnig (pp. 153178). Norwood, NJ: Ablex. Joe, A. (1998). What effects do text-based tasks promoting generation have on incidental vocabulary acquisition? Applied Linguistics, 19, 357377. Jones, S. (1985). Problems with monitor use in second language composing. In M. Rose (Ed.), When a writer can't write (pp. 96118). New York: Guilford. Jones, S., & Tetroe, J. (1987). Composing in a second language. In A. Matsuhashi (Ed.), Writing in real timeModeling production process (pp. 3457). Norwood, NJ: Ablex. Jourdenais, R. (1996, March). The limitations of thinkalouds. Paper presented at The American Association for Applied Linguistics Conference, Chicago, IL. Jourdenais, R., Ota, M., Stauffer, S., Boyson, B., & Doughty, C. (1995). Does textual enhancement promote noticing? A thinkaloud protocol analysis. In R. Schmidt (Ed.), Attention and awareness in foreign language learning (pp. 183216). Honolulu: University of Hawaii Press. Kagan, N., Krathwohl, D., & Miller, R. (1963). Stimulated recall in therapy using video tape - a case study. Journal of Counseling Psychology, 10, 237243. Kasper, G., & Blum-Kulka, S. (1993). Interlanguage pragmatics. New York: Oxford University Press. Kellerman, E. (1979). Transfer and non-transfer: Where we are now. Studies in Second Language Acquisition, 2, 3757. Kempe, V., & MacWhinney, B. (1998). The acquisition of case marking by adult learners of Russian and German. Studies in Secord Language Acquisition, 20, 543587. Kern, R. (1994). The role of mental translation in second language reading. Studies in Second Language Acquisition, 16, 441461. Krashen, S. (1981). Second language acquisition and second language learning. Oxford: Pergamon. Krings, H. (1987). The use of introspective data in translation. In C. Fxrch & G. Kasper (Eds.), Introspection in second language research (pp. 159176). Clevedon: Multilingual Matters. Labov, W. (1972). Sociolinguistic patterns. Philadelphia: University of Pennsylvania Press.

< previous page

page_142

next page >

< previous page

page_143

next page > Page 143

Laviosa, F. (1991, October). An investigation of the listening strategies of advanced learners of Italian as a second language. Paper presented at the Conference on Bridging Theory and Practice in the Foreign Language Classroom, Baltimore. (ERIC Document ED 345563) Lay, N. (1982). Composing processes of adult ESL learners: A case study. TESOL Quarterly, 16, 406. Lee, J. (1998). The relationship of verb morphology to second language reading comprehension and input processing. Modem Language Journal, 82, 3348. Leeman, J. (1999, September). Recasts in Spanish as a second language: An empirical study of negative evidence and enhanced salience. Paper presented at Second Language Research Forum, Minneapolis. Lennon, P. (1989). Introspection and intentionality in advanced second-language acquisition. Language Learning, 39, 375396. Leow, R. (1998). The effects of amount and type of exposure on adult learners' L2 development in SLA. Modern Language Journal, 82, 4968. Levelt, W. (1989). Speaking: From intenton to articulation. Cambridge, MA: MIT Press. Levelt, W. (1993). Language use in normal speakers and its disorders. In G. Blanken, J. Dittmann, H. Grimm, J.C. Marsha, & C-W. Wallesch (Eds.), Linguistic disorders and pathologies (pp. 115). Berlin: Mouton de Gruyter. Levelt, W. (1995). The ability to speak: From intentions to spoken words. European Review, 3, 1323. Lieberman, D. (1979). Behaviorism and the mind: A limited call for a return to introspection. American Psychologist, 34, 319333. Lightbown, P. (1998). The importance of timing in focus on form. In C. Doughty & J. Williams (Eds.), Focus on form in classroom second language acquisition (pp. 177196). Cambridge, England: Cambridge University Press. Lotto, L., & de Groot, A. (1998). Effects of learning method and word type on acquiring vocabulary in an unfamiliar language. Language Learning 48, 3169. Lowyck, J. (1980). A process analysis of teaching (Report No. 21). Leuven, Belgium: Katholieke Universiteit te Leuven, Department Pedagogische Wetenschappen. Lyons, W. (1986). The disappearance of introspection. Cambridge, MA: MIT Press. Lyster, R. (1998). Recasts, repetition, and ambiguity in L2 classroom discourse. Studies in Second Language Acquisition, 20, 5181. Mackey, A., Gass, S., & McDonough, K. (in press). How do learners perceive implicit negative feedback? Studies in Second Language Acquisition, 22. Mackey, A., & Gum, A. (1997, March). Working into the workplace: Strategies for coping with transition. Paper presented at the Annual Meeting of the Teachers of English to Speakers of Other Languages, Orlando, FL. Mackey, A., & Philp, J. (1998). Conversational interaction and second language development: Recasts, responses, and red herrings. Modern Language Journal, 82, 338356. Mangubhai, F. (1992, July). Going beyad the product: How do we get at the processes of second language acquisition? Paper presented at the Pacific Second Language Research Forum, Sydney. Marland, P. (1977). A study of teachers' interactive thoughts. Unpublished doctoral dissertation, University of Alberta, Edmonton, Canada. Marx, R., & Peterson, P. (1981). The nature of teacher decision making. In B. R. Joyce, C. C. Brown, & L. Peck (Eds.), Flexibility in teaching: An excursion into the nature of teaching and training (pp. 236255). New York: Longman. Matsumoto, K. (1993). Verbal-report data and introspective methods in second language research: State of the art. RELC Journal: A Journal of Language Teaching and Research in Southeast Asia, 24, 3260.

< previous page

page_143

next page >

< previous page

page_144

next page > Page 144

McNair, K. (19781979). Capturing inflight decisions: Thoughts while teaching. Educational Research Quarterly, 3, 2642. Mehnert, U. (1998). The effects of different lengths of time for planning on second language performance. Studies in Second Language Acquisition, 20, 83108. Miller, G., Galanter, E., & Pribram, K. (1960). Plans and the structure of behavior. New York: Holt, Reinhart and Winston. Morine, G., & Vallance, E. (1975). Special study B: A study of teacher and pupil perceptions of classroom interaction (Tech. Rep. No. 75116). San Francisco: Far West Laboratory. Munnich, E., Flynn, S., & Martohardjono, G. (1994). Elicited imitation and grammaticality judgment tasks: What they measure and how they relate to each other. In E. Tarone, S. Gass, & A. Cohen (Eds.), Research methodology in secondlanguage acquisition (pp. 22743). Mahwah, NJ: Lawrence Erlbaum Associates. Munro, M. (1998). The effects of noise on the intelligibility of foreign-accented speech. Studies in Second Language Acquisition, 20, 139154. Murphy, J. (1989). The listening strategies of English as a second language college students. Research and Teaching in Developmetal Education, 4, 2746. Naylor, P. (ms). Legal testimony and the non-native speaker of English: The problem of linguistic and cultural interference in interethnic communication. Neubach, A., & Cohen, A. (1988). Processing strategies and problems encountered in the use of dictionaries. Dictionaries: Journal of the Dictionary Society of North America, 10, 119. Nevo, N. (1989). Test-taking strategies on a multiple-choice test of reading comprehension. Language Testing, 6, 199215. Newell, A., & Simon, H. (1956). The logic theory machine: A complex information processing system. I.R.E. Transactions on Information Theory, 2, 6179. Nisbett, R., & Wilson, T. (1977a). The halo effect: Evidence for unconscious alteration of judgments. Journal of Personality and Social Psychology, 35, 250256. Nisbett, R., & Wilson, T. (1977b). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231259. Paribakht, T. S., & Wesche, M. (1997). Vocabulary enhancement activities and reading for meaning in second language vocabulary acquisition. In J. Coady & T. Huckin (Eds.), Second language vocabulary acquisition: A rationale for pedagogy (pp. 174200). Cambridge, England: Cambridge University Press. Paribakht, T. S., & Wesche, M. (1999). Reading and "incidental" L2 vocabulary acquisition: An introspective study of lexical inferencing. Studies in Second Language Acquisition, 21, 195224. Peterson, P., & Clark, C. (1978). Teachers' reports of their cognitive processes during teaching. American Educational Research Journal, 15, 555565. Peterson, P., Marx, R., & Clark, C. (1978). Teacher planning, teacher behavior and student achievement. American Educational Research Journal, 15, 417432. Pinker, S. (1989). Resolving a learnability paradox in the acquisition of the verb lexicon. In M. L. Rice & R. L. Schiefelbusch (Eds.), The teachability of language (pp. 1362). Baltimore: P.H. Brookes. Posner, M. (1992). Attention as a cognitive and neural system. Current Directions in Psychological Science, 1, 1114. Poulisse, N.. (1990). The use of compensatory strategies by Dutch learners of English. Dordrecht: Foris. Poulisse, N., Bongaerts, T., & Kellerman, E. (1987). The use of retrospective verbal reports in the analysis of compensatory strategies. In C. Færch & G. Kasper (Eds.), Introspection in second language research (pp. 213229). Clevedon: Multilingual Matters.

< previous page

page_144

next page >

< previous page

page_145

next page > Page 145

Pressley, M., & Afflerbach, P. (1995). Verbal protools of reading: The nature of constructively responsive reading. Hillsdale, NJ: Lawrence Erlbaum Associates. Raimes, A. (1985). What unskilled ESL students do as they write: A classroom study of composing. TESOL Quarterly, 19, 229258. Reder, L. (1982). Plausibility judgments versus fact retrieval: Alternative strategies for sentence verification. Psychological Review, 89, 250280. Riney, T., & Flege, J. (1998). Changes over time in global foreign accent and liquid identifiability and accuracy. Studies in Second Language Acquisition, 20, 213243. Roberts, M. (1995). Awareness and the efficacy of error correction. In R. Schmidr (Ed.), Attention and awareness in foreign language learning (Tech. Rep. No. 9. pp. 163182. Honolulu: University of Hawai'i, Second Language Teaching and Curriculum Center. Robinson, M. (1991). Introspective methodology in interlanguage pragmatics research. Pragmatics of Japanese as native and target language (Tech. Rep. No. 3, pp. 2984). Honolulu: University of Hawai'i, Second Language Teaching and Curriculum Center. Rose, M. (1984). Writer's block. Carbondale: Southern Illinois University Press. Russo, J., Johnson, E., & Stephens, D. (1989). The validity of verbal protocols. Memory and Cognition, 17, 759769. Schmidt, R. (1993). Awareness and second language acquisition. Annual Review of Applied Linguistics, 13, 206226. Schmidt, R., & Frota, S. (1986). Developing basic conversational ability in a second language: A case study of an adult learner of Portuguese. In R. Day (Ed.), Talking to Learn: Conversation in Second Language Acquisition (pp. 237326). Rowley, MA: Newbury House. Schumann, J. (1978). The pidginization process. Rowley, MA: Newbury House. Schumann, J., & Schumann, F. (1977). Diary of a language learner: An introspective study of second language learning. In H.D. Brown, C. Yorio, & R. Crymes (Eds.), On TESOL '77 teaching and learning English as a second Language: Trends in research and practice (pp. 241249). Washington, DC: TESOL. Seliger, H. (1983). The language learner as linguist: Of metaphors and realities. Applied Linguistics, 4, 479191. Seliger, H., & Shohamy, E. (1989). Second language research methods. Oxford: Oxford University Press. Selinker, L. (1972). Interlanguage. International Review of Applied Linguistics, 10, 209231. Selinker, L. (1974). Interlanguage. In J. Richards (Ed.), Error analysis: Perspectives on second Language acquisition (pp. 3154). London: Longman. Semmel, D. (1977, April). The effects of training on teacher decision making. Paper presented at the annual meeting of the American Educational Research Association, New York. (ERIC Document ED 138558) Shavelson, R., Webb, N., & Burstein, L. (1986). Measurement of teaching. In M. Wittrock (Ed.), Handbook of research on teaching (pp. 5091). New York: MacMillan. Shroyer, J. (1981). Critical moments in the teaching of mathematics: What makes teaching difficult? Unpublished doctoral dissertation, Michigan State University, East Lansing. Shulman, L. (1986). Paradigms and research programs in the study of teaching: A contemporary perspective. In M. Wittrock (Ed.), Handbook of research on teaching (pp. 336). New York: MacMillan. Shulman, L., & Elstein, A. (1975). Studies of problem solving, judgement, and decision making: Implications for educational research. In F. N. Kerlinger (Ed.), Review of research in education (Vol. 3, pp. 342). Itasca, IL: Peacock. Sjoholm, K. (1976). A comparison of the test results in grammar and vocabulary between Finnish and Swedish-speaking applicants for English. In H. Ringbom & R. Palmberg

< previous page

page_145

next page >

< previous page

page_146

next page > Page 146

(Eds.), Errors made by Finns and Savedish-speaking Finns in the learning of English (AFTIL Vol. 5, pp. 54137). Abö Akademi: Publications of the Department of English. Skibniewski, L. (1990). The writing processes of advanced foreign language learners: Evidence from thinking aloud and behavior protocols. In J. Fisiak (Ed.), Papers and studies in contrastive linguistics. The Polish-English contrastive project (Vol. 25, pp. 193202). Poznan: Adam Mickiewicz University. Skinner, B. F. (1953). Science and human behavior. New York: The Free Press. Skinner, B. F. (1957). Verbal behavior. New York: Appleton-Century-Crofts. Smagorinsky, P. (1994) (Ed.). Speaking about writing: Reflections on resarch methodology. Thousand Oaks, CA: Sage. Someren, M. van, Barnard, Y., & Sandberg, J. (1994). The think aloud method: A practical guide to modelling cognitive processes. London: Academic Press. Sorace, A. (1988). Linguistic intuitions in interlanguage development: The problems of indeterminacy. In J. Pankhurst, M. Sharwod Smith, & P. Van Buren (Eds.), Learnability and second languages: A book of readings (pp. 167190). Dordrecht: Foris. Stemmer, B. (1991). What's on a C-test taker's mind? Mental processes in C-text taking. Bochum: Universitätsverlag Dr N Brockmeyer. Swain, M., & Lapkin, S. (1995). Problems in output and the cognitive processes they generate: A step towards second language learning. Applied Linguistics, 16, 371391. Swain, M., & Lapkin, S. (1998). Interaction and second language learning: Two adolescent French immersion students working together. Modern Language Journal, 82, 320337. Titchener, E. (1908). Lectures on the elementary psychology of feeling and attention. New York: Macmillan. Tomitch, L. (1999, March). Working memory capacity and the recall of predicted elements in the text. Paper presented at the American Association of Applied Linguistics Conference, Stamford, CT. Tyler, A. (1995). The coconstruction of miscommunication. Studies in Second Language Acquisition, 17, 129152. Van Hell, J., & Candia Mahn, A. (1997). Keyword mnemonics versus rote rehearsal: Learning concrete and abstract foreign words by experienced and inexperienced learners. Language Learnings, 47, 507546. Vignola, M. (1995). Les prises de décision lors du processus d'écriture en language maternell et en langue seconde chez les diplômés d'immersion [The decision-making processes in native and second language writing among French immersion graduates]. Quebec, Canada: Université Laval, Centre International de Recherche en Amenagement Linguistique. Villamil, O., & Guerrero, C. de (1998). Assessing the impact of peer revision on L2 writing. Applied Linguistics 19, 491514. Warren, J. (1996). How students pick the right answer: A 'think aloud' study of the French CAT. In J. Burston, M. MonvilleBurston, & J. Warren (Eds.), Issues and innovations in the teaching of French (pp. 7994). Occasional Paper No. 15. Australian Review of Applied Linguistics. Watson, J. (1913). Psychology as the behaviourist views it. Psychological Review, 20, 158177. Watson, J. (1920). Is thinking merely the action of language mechanisms? British Journal of Psychology, 11, 87104. White, L. (1989). Universal grammar and second language acquisition. Amsterdam: John Benjamins. White, P. (1980). Limitations on verbal reports of internal events: A refutation of Nisbett and Wilson and of Bem. Psychological Reveiw, 87, 105112. Wittgenstein, L. (1958). Philosophical investigations (3rd ed.). New York. Macmillan.

< previous page

page_146

next page >

< previous page

page_147

next page > Page 147

Woodlinger, M. (1980). A study of teacher interactive decision making. Unpublished doctoral dissertation, University of Alberta, Edmonton, Canada. Wundt, W. (1894). Lectures on human and animal psychology (Translated by J. E. Creighton & E. B. Titchener, 1896). Swann Sonnenschein. Zamel, V. (1983). The composing processes of advanced ESL students: Six case studies. TESOL Quarterly, 17, 165187. Zimmermann, R., & Schneider, K. (1987). The collective learner tested: Retrospective evidence for a model of lexical search. In C. Færch & G. Kasper (Eds.), Introspection in second language research (pp. 177196). Clevedon: Multilingual Matters. Zobl, H. (1980). Developmental and tranfer errors: Their common bases and (possibly) differential effects on subsequent learning. TESOL Quarterly, 14, 469479.

< previous page

page_147

next page >

< previous page

page_ii

next page >

< previous page

page_149

next page > Page 149

Additional References Allport, A. (1988). What concept of consciousness? In A. Marcel & E. Bisiach (Eds.), Consciousness in contemporary science (pp. 159182). London: Clarendon. Auten, A. (1984). Protocol analysis: Another research tool in the English language arts. English Education, 16, 4851. Barbara, L., & Scott, M. (1994). Reflections on language learning: In honour of Antonieta Celani. Clevedon: Multilingual Matters. Benoit, W., & Benoit, P. (1989). Factors influencing the accuracy of verbal reports of conversational behavior. Central States Speech Bureau, 39, 218232. Bialystok, E. (1990). The competence of processing: Classifying theories of second language acquisition. TESOL Quarterly, 24, 635648. Borsch, S. (1986). Introspective methods in research on interlingual. and intercultural communication. In J. House & S. BlumKulka (Eds.), Interlingual and intercultural communication (pp. 195209). Tübingen: Gunter Narr. Bracewell, R., & Breuleux, A. (1994). Substance and romance in analyzing think-aloud protocols. Thousand Oaks, CA: Sage. Carr, T., & Curran, T. (1994). Cognitive factors in learning about structured sequences: Application to syntax. Studies in Second Language Acquisition, 16, 205230. Chi, M., de Leeuw, N., Chiu, M., & Lavancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18, 439437. Clark, C., & Peterson, P. (1986). Teachers' thought processes. In M. Wittrock (Ed.), Handbook of Research on Teaching (pp. 255296). New York: MacMillan. Cohen, A. (1994). Assessing language ability in the classroom (2nd ed.). Boston: Heinle and Heinle. Cohen, A., & Robbins, M. (1976). Toward assessing interlanguage performance: The relationship between selected errors, learners' characteristics, and learners' explanations. Language Learning, 26, 4566. Cohen, A., Ivry, R., & Keele, S. (1990). Attention and structure in sequence learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16, 1730. Cooper, C., & Odell, L. (1976). Consideration of sound in the composing processes of published writers. Research in the Teaching of English, 10, 103115. Cooper, M., & Holzman, M. (1983). Talking about protocols. College Composition and Communication, 34, 284293. Cowan, N. (1993). Activation, attention and short-term memory. Memory and Cognition, 21, 162167. Cowan, N. (1995). Attention and memory: An integrated framework. New York: Oxford University Press. Cowan, N. (1996). Short-term memory, working memory, and their importance in language processing. Topics in Language Disorders, 17, 118. Crutcher, R. (1994). Telling us what we know: The use of verbal report methodologies in psychological research. Psychological Science, 5, 241244. Curran, T., & Keele, S. (1993). Attentional and nonattentional forms of sequence learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 19, 189202. Cyr, P. (1996). Le point sur les stratégies d'apprentissage d'une langue seconde [The point about second language learner strategies] Anjou (Québec), Canada: Les Éditions CEC.

< previous page

page_149

next page >

< previous page

page_150

next page > Page 150

Dechert, H. (1989). Interlingual processing in students' mental folk models of SLA. In H. Dechert and M. Raupach (Eds.), Interlingual processes (pp. 211228). Tübingen: Gunter Narr Deville, C., & Chalhoub-Deville, M. (1993). Modified scoring, traditional item analysis and Sato's caution index used to investigate the reading recall protocol. Language Testing, 10, 117132. Ellis, N., & Sinclair, S. (1996). Working memory in the acquisition of vocabulary and syntax: Putting language in good order. Quarterly Journal of Experimental Psychology, 49A, 234250. Faigley, L., & Witte, S. (1981). Analyzing revision. College Composition and Communication, 32, 400414. Fairfax, O., & Green, M. (1989). Introspection, language learning, and curriculum development. System, 17, 7182. Fontaine, S. (1989). Using verbal reports to learn about children's audience awareness in writing. Educational Research Quarterly, 13, 2635. Fotos, S. (1992). Consciousness raising and noticing through focus on form. Applied Linguistics, 14, 385407. Gass, S. (1991). Grammar instruction, selective attention and learning processes. In R. Phillipson, E. Kellerman, L. Selinker, M. Sharwood-Smith, & M. Swain (Eds.), Foreign/second language pedagogy research (pp. 134141). Clevedon: Multilingual Matters. Gerloff, P. (1986). Second language learners' reports on the interpretive process: Talk-aloud protocols of translation. In J. House & S. Blum-Kulka (Eds.), Interlingual and intercultural communication (pp. 243262). Tübingen: Gunter Narr. Griffey, D., & Housener, L. (1991). Differences between experienced and inexperienced teachers' planning decisions, interaction, student engagement, and instructional climate. Research Quarterly for Exercise and Sport, 62, 196204. Hayes, J. (1996). A new framework for understanding cognition and affect in writing. In C. Levy & S. Ransdell (Eds.), The science of writing: Theories, methods, individual differences, and applications (pp. 127). Mahwah, NJ: Lawrence Erlbaum Associates. Hayes, J., & Flower, L. (1980). Identifying the organization of writing processes. In L. Gregg and E. Steinberg (Eds.), Cognitive processes in writing (pp. 330). Hillsdale, NJ: Lawrence Erlbaum Associates. Hayes, J., & Flower, L. (1983). Uncovering cognitive processes in writing: An introduction to protocol analysis. In P. Mosenthal, L. Tamor, & S. Walmsley (Eds.), Research on writing: Principles and methods (pp. 207220). New York: Longman. Hyltenstam, K. (1983). Data types and second language variability. In H. Ringbom (Ed.), Psycholinguistics and foreign language learning. Papers from a conference (Stockholm, Swede and Abö, Finland, October 2526, 1982). Abo, Finland: Research Institute of the Abö Akademi Foundation. Janssen D., Waes, L. van, Bergh, H. van den, (1996). Effects of thinking aloud on writing processes. In C. Levy and S. Ransdell (Eds.), The science of writing: Theories, methods, individual differences, and applications (pp. 233250). Mahwah, NJ: Lawrence Erlbaum Associates. Kihlstrom, J. (1984). Conscious, subconscious, unconscious: A cognitive perspective. In K. Bowers & D. Meichenbaum (Eds.), The unconscious reconsidered (pp. 149211). New York: John Wiley. Kroll, B. (Ed.). (1990). Second language writing: Research insights for the classroom. Cambridge, England: Cambridge University Press. Kucan, L., & Beck, I. (1997). Thinking aloud and reading comprehension research: Inquiry, instruction, and social interaction. Review of Educational Research, 67, 271299.

< previous page

page_150

next page >

< previous page

page_151

next page > Page 151

Leow, R. (1997). Attention, awareness and foreign language behaviour. Language Learning, 47, 467505. Mann, S. (1982). Verbal reports as data: A focus on retrospection. In S. Dingwall, S. Mann, & F. Katamba (Eds.), Methods and problems in doing applied linguistic research. Lancaster: Department of Linguistics and Modern English Language. McLaughlin, B. (1990). ''Conscious" vs. "unconscious" learning. TESOL Quarterly, 24, 617634. McLaughlin, B., Rossman, T., & McLeod, B. (1983). Second language learning: An information-processing perspective. Language Learning, 33, 135158. Meichenbaum, D., & Gilmore, J. (1984). The nature of unconscious processes: A cognitive-behavioral perspective. In K. Bowers & D. Meichenbaum (Eds.), The unconscious reconsidered (pp. 273298). New York: Wiley. Morgensen, L. (1984). Transfer of kulturspecifikke begreber. Unpublished master's thesis, University of Aarhus, Denmark. Morine-Dershimer, G. (1979). Teacher plan and classroom reality: The South Bay study: Part 4 (Research Series, No. 60). East Lansing: Michigan State University, Institute for Research on Teaching. Newell, A., & Simon, H. (1972). Human problem solving. Englewood Cliffs, NJ: Prentice-Hall. Norris, S. (1990). Effect of eliciting verbal reports of thinking on critical thinking test performance. Journal of Educational Measurement, 27, 4158. Oxford, R. (1996). Employing a questionnaire to assess the use of language learning strategies. Applied Language Learning, 7, 2546. Payne, J. (1994). Thinking aloud: Insights into information processing. Psychological Science, 5, 245248. Perl, S. (1980). Understanding composing. College Composition and Communication 31, 317336. Posner, M., & Rothbart, M. (1992). Attention and conscious experience. In A. Milner & M. Rugg (Eds.), Foundations of neuropsychology series (pp. 91112). New York: Academic Press. Raimes, A. (1987). Language proficiency, writing ability, and composing strategies: A study of ESL college student writers. Language Learning, 37, 439467. Robinson, P. (1995). Review article: Attention, memory and the 'noticing' hypothesis. Language Learning, 45, 283331. Schmidt, R. (1990). The role of consciousness in second language learning. Applied Linguistics, 11, 129158. Sharwood Smith, M. (1981). Consciousness-raising and the second language learner. Applied Linguistics, 6, 190204. Sharwood Smith, M. (1993). Input enhancement in instructed SLA: Theoretical bases. Studies in Second Language Acquisition, 15, 165179. Stratman, J. F., & Hamp-Lyons, L. (1994). Reactivity in concurrent think-aloud protocols. In P. Smagorinsky (Ed.), Speaking about writing: Reflections on research methodology (pp. 89112).Thousand Oaks, CA: Sage. Tan, S. (1996). Differences between experienced and inexperienced physical education teachers' augmented feedback and interactive teaching decisions. Journal of Teaching in Physical Education, 15, 151170. Tomlin, R., & Villa, V. (1994). Attention in cognitive science and second language acquisition. Studies in Second Language Acquisition, 16, 183203. Trabasso, T., & Suh, S-Y. (1993). Understanding text: Achieving explanatory coherence through on-line inferences and mental operations in working memory. Discourse Processes, 16, 334. Tuckwell, N. (1980a). Content analysis of stimulated recall protocols (Tech. Paper No. 80-2-2). Edmonton, Canada: University of Alberta, Centre for Research in Teaching.

< previous page

page_151

next page >

< previous page

page_152

next page > Page 152

Tuckwell, N. (1980b). Stimulated recall: Theoretical perspectives and practical and technical considerations (Tech. Rep. No. 80-2-3). Edmonton, Canada: University of Alberta, Centre for Research in Teaching. Van Patten, B. (1990). Attending to form and content in the input: An experience in consciousness. Studies in Second Language Acquisition, 12, 287301. Westerman, D. (1991). Expert and novice teacher decision making. Journal of Teacher Education, 42, 292305. Wilson, T. (1994). The proper protocol: Validity and completeness of verbal reports. Psychological Science, 5, 249252.

< previous page

page_152

next page >

< previous page

page_153

next page > Page 153

Appendix A Sample Research Protocol from Stimulated Recall Experiment on Oral Interaction This appendix contains a detailed protocol for a study of learners' perceptions about oral interaction (Mackey, Gass, & McDonough, in press). It is reproduced here as an example. Collection Scenario Two researchers can collect the data, with one doing the interaction and one carrying out the stimulated recall. Or, one researcher can do both the interaction and the recall. Interaction Researcher Responsibilities: Provides instructions to participant (learner) for task activity. Interacts with participant (learner) during task activity. Ensures audiotape is working during task activity. During stimulated recall, a second researcher can sit unobtrusively in the corner and take notes about which topics might be raised with the learner after the recall is concluded. Can ask final questions after stimulated recall is concluded. Interaction Instructions for Researcher: Engage in some chitchat for about 12 minutes, ask participants (learners) to read and sign the consent form so researchers can use the data. Explain that researchers are interested in how language is learned. Give directions for the task. Interact during task, creating breakdowns if necessary so that some negotiation takes place. Target phonological, lexical, and grammatical errors. Try to limit the session to about 1012 minutes in order to get the recall session done in 30 minutes. Stimulated Recall Researcher Responsibilities:

< previous page

page_153

next page >

< previous page

page_154

next page > Page 154

Operates VCR and makes notes during task interaction between participant (learner) and first researcher. Rewinds videotape and plays it during the stimulated recall session. Carries out stimulated recall (selecting segments of tape to examine). Ensures audio tape is working during stimulated recall. Stimulated Recall Instructions: Provide explanation of next step, for example: What we're going to do now is watch the video. We are interested in what you were thinking at the time you were talking about the pictures. We can see what you were doing by looking at the video, but we don't know what you were thinking. So what I'd like you to do is tell me what you were thinking, what was in your mind at that time while you were talking to her. I'm going to put the remote control on the table here and you can pause the video any time that you want. So if you want to tell me something about what you were thinking, you can push pause. If I have a question about what you were thinking, then I will push pause and ask you to talk about that part of the video Demonstrate stopping the video and asking a question for them. If the participant stops the video, listen to what he or she says. If you stop the video, ask something general, for example: What were you thinking here/at this point/right then? Can you tell me what you were thinking at that point? I see you're laughing/looking confused/saying something there, what were you thinking then?" If their response is that they don't remember, do not pursue this because "fishing" for answers that were not immediately provided increases the likelihood that the answer will be based on what the person thinks now or some other memory or perception.

< previous page

page_154

next page >

< previous page

page_155

next page > Page 155

Try not to focus or direct participant responses beyond "what were you thinking then." You might want to focus attention on the NS utterance by saying something like: Do you remember thinking anything when she repeated that? Can you remember what you were thinking when she said that/those word(s)? Can you tell me what you thought when she said that? Try not to react to responses other than providing backchanelling cues or nonresponses: Oh, mhm, great, good, I see, uh-huh, ok When participants have finished the recall, if there are no other experimental steps to conclude, an additional explicit question may elicit useful data to address the research question. You can ask if they have any questions or comments about the video or the task they have done. At that point, the second researcher who has been quietly sitting in the corner can say something like: I was wondering if I could ask you something? I'm just curious. I noticed when you were talking about the video that you mentioned your pronunciation and vocabulary quite a lot. Is that what you are most concerned about when you are speaking? What about grammar? Do you think about grammar at all when you are speaking? There were a few times when I corrected your grammar while you were speaking. Did you notice that I was correcting your grammar?

< previous page

page_155

next page >

< previous page

page_ii

next page >

< previous page

page_157

next page > Page 157

Appendix B This appendix contains a second protocol for stimulated recall procedure. This procedure was used as a pilot test to explore whether enhanced salience in oral interaction was perceived as negative feedback or not (Leeman, 1999). First, the treatment that is being explored through the stimulated recall procedure is described. Then, the detailed protocol appears. Treatment Description: Interaction with Enhanced Input This group received no negative feedback on nontarget utterances. Thus, a sample response to the nontarget utterance given below is: NNS:

La manzana rojo está en la mesa. the apple-[fem.] red-[masc.] is on the table 'The red apple is on the table.'

R:

Um hmm. Um hmm. ¿Qué, más? What else?

However, when the researcher provided directions for object placement (in the second part of each treatment task), stress and intonation were used to enhance the salience of the target structure. Specifically, the gender and number marking of the adjective were stressed: R:

La manzana roJA está en la mesa. the apple [fem.] red [fem.] is on the table 'The red apple is on the table.'

< previous page

page_157

next page >

< previous page

page_158

next page > Page 158

Stimulated Recall Protocol After completion of treatment, stop tape. Begin rewinding. Check if participant needs to get a drink or use restroom. Say: "Now we're going to do a different kind of activity. In this part of our meeting, I'm interested in finding out what you were thinking while we did the last activities. I know what you said, because that's on the tape, but I'd also like to know what was going through your mind while we did the previous activities." "So now we're going to listen to the tape, and I want you try to remember what you were thinking then, not what you think about it now. Put tape player close to participant. Show where pause button is. Say: "I want you to stop the tape whenever you remember what you were thinking. I'll also stop the tape from time to time and ask you to think back and tell me what was going through your mind." Ask if there are questions about procedure. Ask, "Is it clear what we're doing? I want to know what you were thinking as we did the activities." Have participant try using pause button. Possible answers to possible questions: they can stop it as often as they want we don't have to take turns Begin tape. Pause after participant's first utterance and ask, "What were you thinking right then?" After answer, wait few seconds and ask, "Do you remember anything else about what you were thinking at that moment?"

< previous page

page_158

next page >

< previous page

page_159

next page > Page 159

Make sure to pause after utterances containing both target and nontarget agreement. For second half of activities (researcher provides directions to participant), pause after researcher's first utterance and prompt as above. In enhanced salience group, make sure to pause after some utterances with enhanced salience and some without. If participant starts to talk without pausing tape, pause tape for her, then wait for her to unpause it. If she doesn't, wait for a few seconds of silence, then ask, "Do you remember anything else about what you were thinking at that moment?" If participant says he or she made a mistake on agreement, try to maintain orientation to time of production, for example, "were you thinking that at the time?" If participants say they noticed that the researcher corrected them (recast and negative evidence groups), ask what they thought at the timedid they think it was a correction then? Keep them focused on their interpretation at the time it was provided. Ask similar follow-up questions even if participant pauses tape and discusses things not related to research questions (e.g., ser vs. estar, vocabulary, locatives, pronunciation). Emphasize thoughts during activity (not interpretation now or later in the activity). After SR is complete, move to debriefing questions/semistructured interview (so doesn't influence SR). Ask all questions, even if the participant already provided answers during SR: 1. What did you think about these activities? 2. Did you learn anything? (If so, what?) 3. What did you think about my interaction with you?

< previous page

page_159

next page >

< previous page

page_160

next page > Page 160

4. Did you notice anything specific about my interaction with you? If participants mention any interactional features, ask what they thought at the time. "At that time, why did you think I was doing that?" Specifically query whether, at the time of the enhanced salience, they thought the researcher was telling them they had said it wrong previously or whether the enhanced salience was to show them how to say it in Spanish.

< previous page

page_160

next page >

< previous page

page_161

next page > Page 161

Appendix C Examples of Complete Stimulated Recall Data (Collected for a Study Carried out by Mackey, Gass, & McDonough, in Press). NNS (nonnative speaker); INT (interviewer) Recall episode 1

NNS:

un bicchiere pero per il café A glass however for the coffee 'A glass for coffee'

INT

un bicchiere, un bicchiere, un bicchiere? A glass a glass a glass

That I was really off on my answer or on my description like she had no RECALL: clue what I was talking about. Recall episode 2

NNS:

Poi c'è la finestra Then there is the window

INT:

co- no, non ho capito, come? Wh- no, not I have understood, what 'Wh, I-I didn't understand, what?

NNS:

la finestra. The window

RECALL:

Same thing. I was way off.

< previous page

page_161

next page >

< previous page

page_162

next page > Page 162

Recall episode 3

NNS:

c'è un tipo di fior There is a type of flow-

INT:

un tipo di che cosa? 'A type of what'

NNS:

fiore flower

Same thing and I was trying to . . . I don't know my mind was going blank. RECALL: I was trying to think of the vocabulary. Recall episode 4

NNS:

Poi un bicchiere Then a glass

INT:

Un che? Come? A what what

NNS:

Bicchiere glass

I knew it wasn't a glass and then I don't know. I just am getting old or something. I was drawing a blank. Then I thought of a vase, but then I thought that since there was no flowers. Maybe it was just a big glass. So, then I thought I'll see it and see. Then when she said ''come" I knew that it RECALL: was completely wrong. Recall episode 5

NNS:

Ci sono due bottigli There are two bottles (m. pl.)

INT:

Ci sono bottigli?... . due? There are bottles (m. pl.) two? (bottles is feminine)

RECALL:

I was just hoping she didn't hear me the first time.

< previous page

page_162

next page >

< previous page

page_163

next page > Page 163

Recall episode 6

NNS:

Poi sopra i bottiglie ci sono due Then above the (m. pl.) bottles (f. pl.) there are 2 'Then, above the bottles there are 2 . . .'

NT:

sopra che cosa? Above what?

NNS:

di.. of

Well, at first I thought how to explain that it wasn't the next one, but the next one after that and then I couldn't think that's why I abandoned that idea all together and then I was just tryring to focus on how to say that and then RECALL: if need be I would come to that there was a missing shelf. Recall episode 7

NNS:

Poi, a destra di tavolo Then, at right of table (m. ending, should be feminine)

INT:

A destra di? At right of?

NNS:

Tavolo Table

RECALL:

That she didn't hear me.

< previous page

page_163

next page >

< previous page

page_ii

next page >

< previous page

page_165

next page > Page 165

Author Index A Abraham, R., 29 Afflerbach, P., 112 Aichison, J., 23 Alanen, R., 29 Anderson, M., 108 Anderson, N., 29 Aphek, E., 30 Aston, G., 39 Austin, G., 9 B Bailey, N., 8n Bard, E., 10, 110n Bardovi-Harlig, K., 120-121, 134 Barnard, Y., 13, 15-16, 24 Barry, S., 125-126, 135 Bartelt, G., 29 Benoit, W., 19 Bettman, J., 103 Biggs, S., 108-109 Birdsong, D., 110n Bloch, J., 33 Block, E., 29 Bloom, B., 17, 18, 50, 90 Bloomfield, L., 9 Blumenthal, A., 7 Blum-Kulka, S., 25 Bongaerts, T., 34, 101 Boritz, J.E., 108 Bosher, S., 29, 46, 101 Boyson, B., 33 Briice, C., 29 Brown, A., 29 Bruner, J., 9 Buck, G., 30 Burstein, L., 13 Burt, M., 8n C Calderhead, J., 19, 23 Candia Mahn, A., 131 Canguilhem, G., 4

Carroll, S., 10n Cavalcanti, M., 30 Chem, C-L., 30 Chi, M., 89, 99, 100-101 Chomsky, N., 9 Clark, C., 18-19, 20 Cohen, A., 11-13, 24, 25, 26, 30, 34, 47-48, 50, 111 Colker, L., 20 Conners, R.D., 20 Cook, V., 10n Corder, S.P., 2, 110 Cowan, R., 110n Creek, R., 20 Cumming, A., 30 D Davies, W., 30, 31 de Groot, A., 131-132, 136 Dennett, D., 5 DiPardo, A., 17, 19, 44-46, 54 Dôrnyei, Z., 31, 40-41, 120-121, 134 Doughty, C., 33 Dulay, H., 8n

< previous page

page_165

next page >

< previous page

page_166

next page > Page 166

Dwivedi, V., 128-129, 135 Dyson, A., 44 E Ellis, R., 10n Elstein, A., 10, 103 Enkvist, I., 31 Erickson, F., 19 Ericsson, K., 8, 11, 12, 17, 22, 23, 24, 25, 26, 47, 50, 52, 106-107, 109, 112-113 F Færch, C., 21-22, 25, 26-28, 31, 47, 48-54 Feldman, U., 31 Feltovich, P., 100 Flege, J., 116-117, 133 Flynn, S., 110n Fogarty, J., 20 Foster, P., 127-128, 135 Fraser, C., 31, 129 Freedman, S., 44 Frota, S., 26n, 100 G Galanter, E., 9 Garner, R., 89-90 Gass, S., 2, 8n, 10n, 25, 26n, 31, 33, 38-39, 41, 42, 53, 58, 59-60, 66-68, 69-76, 77, 82-84, 90-96, 98, 101, 110n, 118, 130, 153155, 161, 163 Gatbonton, E., 31 Gazzaniga, M., 6 Gerloff, P., 31 Glahn, E., 31 Glaser, R., 100 Goodnow, J., 9 Gordon, C., 32 Gosden, H., 32 Goss, N., 32, 110n Griffey, D., 20 Gu, Y., 32 Guerrero, C. de, 35 Gum, A., 64-66 H Haastrup, K., 32 Habermas, J., 5 Hample, D., 19 Hatasa, Y., 110n Hatch, E, 63n Hawkins, B., 25, 38, 77, 78-81, 84

Hill, K., 32 Hillway, T., 87 Hoôlscher, A., 32 Hoover, M., 128-129, 135 Hosenfeld, C., 25, 26, 32 Housner, L., 20 Huber, G., 26, 47 Huckin, T., 33 J Joe, A., 129-131, 136 Johnson, E., 104, 107-108 Jones, S., 33 Jourdenais, R., 25, 33 K Kagan, N., 18 Kaplan, T., 30, 31 Kasper, G., 21-22, 25, 26-28, 31, 47, 48-54 Kellerman, E., 8n, 34, 101 Kempe, V., 121-122, 134

< previous page

page_166

next page >

< previous page

page_167

next page > Page 167

Kern, R., 33 Kormos, J., 31, 40-41 Krashen, S., 8n, 111 Krathwohl, D., 18 Krings, H., 33 L Labov, W., 61 Lantolf, J., 32, 110n Lapkin, S., 34, 126-127, 135 Laviosa, F., 33 Lay, N., 33 Lazaraton, A., 63n Lazarte, A., 125-126, 135 Lee, J., 123-124, 134 Leeman, J., 58, 60-61, 157-160 Lennon, P., 24, 33 Leow, R., 124-125, 134 Levelt, W., 40, 41 Li, T-Y., 30 Lieberman, D., 7, 17, 112 Lightbown, P., 118 Lotto, L., 131-132, 136 Lowyck, J., 20 Lyons, W., 3, 8-9 Lyster, R., 117-118, 133 M Mackey, A., 25, 33, 38-39, 41, 53, 58, 59-60, 64-66, 66-68, 69-76, 77, 82-84, 90-96, 98, 101, 118, 153-155, 161-163 MacWhinney, B., 121-122, 134 Madden, C., 8n Mandl, H., 26, 47 Mangubhai, F., 98 Marland, P., 20 Martohardjono, G., 110n Marx, R., 19, 20 Matsumoto, K., 111 McDoonough, K., 25, 33, 38-39, 41, 53, 58, 59-60, 66-68, 69-76, 77, 82-84, 90-96, 98, 101, 153-155, 161-163 McNair, K., 20 Mehnert, U., 118-120, 133 Meisel, J., 10n. Miller, G., 9 Miller, R., 18 Mohatt, G., 19 Môhle, D., 32 Morine, G., 20

Munnich, E., 110n. Munro, M., 114-116, 133 Murphy, J., 33 N Naylor, P., 97-98n. Neubach, A., 34 Nevo, N., 34 Newell, A., 9 Nisbett, R., 22, 106-107 O Olshtain, E., 30 Ota, M., 33 P Paribakht, T.S., 34, 43, 129 Park, C., 103 Peterson, P., 18-19, 20 Philp, J., 118 Pinker, S., 58 Posner, M., 113 Poulisse, N., 34, 101 Pressley, M., 112 Pribram, K., 9

< previous page

page_167

next page >

< previous page

page_168

next page > Page 168

R Raimes, A., 34 Reder, L., 23 Riney, T., 116-117, 133 Roberts, M., 28n Robertson, D., 10, 110n Robinson, M., 34, 46-47 Rose, M., 19, 44-46 Rosman, A., 108-109 Russo, J., 104, 107-108 S Sandberg, J., 13, 15-16, 24 Schmidt, R., 26n, 100, 120 Schneider, K., 35 Schumann, F., 26n Schumann, J., 26, 26n Seliger, H., 26n, 63n, 110-111 Selinker, L., 2, 8n, 109-110 Semmel, D., 20 Sergenian, G., 108-109 Shavelsson, R., 13 Shohamy, E., 63n Shroyer, J., 20 Shulman, L.S., 10, 103 Simon, H., 8, 9, 11, 12, 17, 22, 23, 24, 25, 26, 47, 50, 52, 106-107, 109, 112-113 Sjoholm, K., 8n Skibniewski, L., 34 Skinner, B.F., 8-9 Smagorinsky, P., 24 Someren, M. van, 13, 15-16, 24 Sorace, A., 10, 110n Spaafka, A.S., 103 Stauffer, S., 33 Stemmer, B., 31, 34 Stephens, D., 104, 107-108 Swain, M., 34, 126-127, 135 T Tetroe, J., 33 Titchener, E., 7 Tomitch, L., 35 Tyler, A., 35, 39-40 V Vallance, E., 20

Van Hell, J., 131 Vann, R., 29 Varonis, E., 118 Vignola, M., 35 Villarnil, O., 35 W Wang, M., 20 Warren, J., 35 Watson, 6-7, 7-8 Weaver, S., 30 Webb, N., 13 Wesche, M., 34, 43, 129 White, L., 10n White, P., 106 Wilson, T., 22, 106-107 Wittgenstein, L., 4 Woodlinger, M., 20 Wundt, W., 5, 7 Z Zamel, V., 35 Zhang, Y-H., 32, 110n Zimmermann, R., 35 Zobl, H., 26

< previous page

page_168

next page >

< previous page

page_169

next page > Page 169

Subject Index A Accent change in global foreign, 116-117, 133 comprehension of nonnative, 114-116, 113 Acceptability judgements, 5, 10n, 42, 101, 110 Accuracy focus on complexity or, 119 limitations of stimulated recall, 18, 105-113 see also under time lapse Acquisition -learning dichotomy, 111 effect of negative evidence and enhanced salience, 58, 60-61, 157-160 Action, relation of study to concrete, 27, 28, 48, 49 Affective object of introspection, 27, 28 Anagram task, 108 Analysis procedure for stimulated recall data, 63-76, 99-104 qualitive and quantitative, 63, 99-101 see also coding; raters; sampling Animacy, 121, 122 Appropriateness of utterances, 120-121 Aristotle, 3 Attention, 23, 111 competing attentional resources model, 119 Attitudes to tasks, participants' 62, 64-66, 127 Audio recordings, 20, 25, 53, 85 Augustine, St, 3 Authority, paritcipant's attitude to, 62 B Backchannelling, 58, 60, 65 Behaviorism, 6-9, 9-11 Brain, split, 5-6 C C-units, 127 Case marking, morphological, 121-122, 134 Causative/non-causative sentences, clitics in French, 128-129 Child development, 19 Classroom research, 18, 19, 28, 37, 62, 117, 118, 127 classroom interaction, 117-118, 133 see also instruction Clitics, French, 128-129 Coding of stimulated recall data, 63, 68, 69-76, 77-84, 102-103

categorization of types of data, 77, 84 preparation of data, 102-103

< previous page

page_169

next page >

< previous page

page_170

next page > Page 170

sample sheets, 69, 72-73, 77, 78-83 Cognate status and word learning, 131, 136 Cognition and cognitive processes, 9-10, 20 inferential mechanisms, 23, 125, 135 retrieval processes, 23 search and storage mechanisms, 23 stimulated recall applied to, 19, 20, 21, 23, 27, 28, 100 types of cognitive process research, 9-10 verbal reporting, 106-107, 109, 111 Collection procedures for stimulated recall data, 57-63 timing, 84-89, 99 Commentary, running, 4 Communication nonverbal, 65, 118 problems in oral interaction, 39-40, 40-41 stimulated recall data on strategies, 100 Competiton model, 122 Completeness in concurrent verbal protocols, 108-109 Complexity, focus on accuracy or, 119 Composition aloud, 30 written, 19, 46, 101 Comprehension perceptions of, in oral interaction, 38 potential of stimulted recall, 121-122, 134 word identification not correlated, 123 see also reading comprehension Computers, 53, 109 Concurrent protocols, 107 Consciousness, 7-8 Consent, participant's withdrawal of, 87-88 Content/form dichotomy, 118, 123, 133 Context, data produced out of, 109-110 Control of interview, 49, 53-4, 88 Corpus callosum, severing of, 5-6 Court cases, 4-5n, 97-98n Crossword puzzle task, 124 Cueing of stimuli, 108 Cultural influences, 53, 62, 98 Currency of report, 13, 14 D Decision-making research, 10 Declarative knowledge, 21-22 Descartes, René, 3, 4

Description of stimulated recall data, 63, 103-104 Dialogue, learner-learner, 126-127, 135 Diary studies, 26n, 28, 100, 110 Discourse completion task, 46-47 E Elicitation procedure, classification by, 27, 28

< previous page

page_170

next page >

< previous page

page_171

next page > Page 171

Equipment set up and operation, 86, 88, 99 Equity analysts, study with, 108-109 Evidence, legal, 4-5n Example sets, 103 Experimental research in oral interaction, 38-39 Explanations, 58, 107 construction of plausible, 5-6, 54, 106 Exposure, amount and type of, 124-125, 134 F Falsifiability of results, 4 Fatigue, participant, 85, 87 Feedback, 22 inadvertent, 88 learner's perception of nature of, 38-39, 67-68 Forced elicitation data, 2, 110 Form /content dichotomy, 118, 123, 133 and lexis, 126 Frequency and word learning, 131, 136 Freud, Sigmund, 7 G Generalizability of results, 100 Germal case marking, 121-122, 134 Gestures, 65 'grab bag' game, 38 Grammatical errors, learners' attitudes to, 120-121, 134 Grammaticality judgments, 10-11, 109 Graphic depiction of data, 103 H Hand gestures, 65 I Inference generation, 23, 125, 135 Information-processing approach, 17, 20 Initiation of interview, 49, 53-4, 88 Input and input processing, 122, 123-125, 134 Instruction, 127, 129, 130 Instruction to participants and researches, 51, 57-63, 108 standardization, 58, 62 participant's not following, 64-66 procedural recommendations, 62-63 Interaction, see oral interaction Interference in introspection, 5-6, 52, 54, 106, 110, 112-113

L1, in L2 production, 26 Interpersonal behavior, 18, 24 Interpersonal process Recall, 18 Interview techniques, sociolinguistic, 61 Intonational cues, 65 Intropection, 2 atemporal, 4 and behaviorism, 6-9

< previous page

page_171

next page >

< previous page

page_172

next page > Page 172

currency of report, 13, 14 definitions and nuances, 3 and L2 research, 25-36; classification scheme, 26-28; range of studies, 28-35 methods, 3-11 object of, 27, 28, 48 and retrospection, 13, 14, 111 as self-observation, 5 variation by task type 13, 14 see also stimulated recall; talk-aloud; think;-aloud; verbal reporting J Judgement and policy research, 10 K Knowledge background, and word learning, 129,130 declarative and procedural, 21-22 domain-related, 125, 135 and L2 reading comprehension, 125 prior, used in processing text, 111 structures, and stimulated recall, 21, 22-23 L /1/, 116-117, 133 Language, first and second (L1 and L2) choice for recall interview, 53, 88, 97-99, 112 cognitive processes in, 19, 23 see also references to L2 throughout index Language-related episodes in learner-learner dialogue, 126, 135 Learner- and teacher-centered groups, 124-125, 134 Learning -acquisition dichotomy, 111 unconscious, 23 Legal system, 4-5n, 97-98n Lexis -form dichotomy 126 retrieval mechanisms, 19 see also vocabulary Liquid consonants, 116-117, 133 M Magic tricks, analogy of, 5 Meaning, 4 negotiation of, 127-128, 135 Memory, 9, 17, 23, 24

accessibility, 50, 105, 107 contamination and interferencein recall, 52, 112-113 Mental proceses, reflections on, 3-6 Mental representations, 21, 125 Metalinguistic awareness, 122 Methodologies, research combination of, 19, 27, 28, 104 sociolinguistic interview technique, 61

< previous page

page_172

next page >

< previous page

page_173

next page > Page 173

see also particular types, especally introspection; stimulated recall verbal reporting Modality of study, 13, 14 category in introspection research, 27, 28, 48 effect of translation, 107, 108 see also writing Monitor studies, 110 Morphology, 134, 121-124 Motivation, 64-66, 108 Multiple participant studies, 53-54 N Negative evidence, 58, 60-61, 157-160 Negotiation of meaning, 127-128, 135 Noise, background, 114-116, 133 Non-cooperation, participant's, 87-88 Nonverbal communication, 65, 118 Nonveridicality, 107-108, 112 Noticing, 23, 100, 121, 134 O Object of introspection, 27, 28, 48 Objectivity, 63-66 On-line recall see think-aloud Oral interaction research coding of data, 102 communication problems, 39-40, 40-41 experimental research, 38-39 perceptions of comprehension, 38 qualitative research, 39-40 research protocols, 57-60, 153-160 stimulated recall studies, 19, 25, 38-41, 85, 126-128, 135 Oral/written forms, see modality Output, perceived deficiencies in, 41 P Perceptions, stimulated recall studies of, 23, 122 Performance-based data, 104 Phonology, interlanguage, 102, 114-117, 133 see also accent Pictorial depiction of data, 103 Picture-based tasks, 25, 38-39, 121, 131-132 Pilot testing, 57, 58, 60-61, 62, 126 and L1/L2 choice, 98-99 and participant training, 54 stimulated recall as tool for, 58, 157-160 of timing, 87 Pitch, vocal, 115, 116, 133

Planning time and oral production, 118-120, 133 Plans (knowledge structures), 22 Plausibility judgments in stimulated recall, 23 see also under explanations

< previous page

page_173

next page >

< previous page

page_174

next page > Page 174

Pragmatics, interlanguage, 25, 46-47, 120-121, 134 Present sense impression, 5n Priming studies in psychology research, 52 Privacy, 8-9 Problem solving, 10, 13-16, 88 Procedural knowledge, 21-22 Process tracing, 13 Processing, 9 time pressure in L2, 40-41 see also cognitive processes Production, oral, 1-2 effects of planning time, 118-120, 133 source of L2, 25-26 Protocols, research, 57-60, 153-160 Psychology cognitive, 9 mentalistic and mentalist, 9 priming studies in research, 52 Q Qualitative research, 39-40, 63, 100-101 Quantitative analysis, 63, 100-101 Questionnaires, 28, 43, 52, 87 Questions, researcher's, 89-96, 109 R /r/, 116-117, 133 Raters, 63-76 interrater reliability, 63-68, 69, 76, 84, 101, 103 sample schedules, 69, 74-76 training, 66, 69-76 Reactivity, 107-109 Read-and-tell task, 129-131 Reading comprehension, L2, 43, 51, 123-124, 125-126, 134, 135 Reasoning abilities, children's, 19 Recasts, 58, 60-61, 117-118, 133 Reconstructive approaches, 5 Reflections on mental processes, 3-6 Refusals, interlanguage, 46-47 Reliability interrater, 63-68, 69, 76, 84, 101, 103 limitations of stimulated recall, 18, 105-113 Repair, self-corrected error, 41 Repetition, noncorrective, 117-118, 133 Replicability of results, 4, 100 Resistance, participant's, 87-88

Resource allocation model, 125, 126, 135 Resource deficits, 40-41 Responses, researcher's, 58, 60, 89-96, 109 Retrieval processes, 23 Retrospection, 13, 14 atemporal introspection as, 4 immediate/consecutive, 50 interference in, 110 and introspection, 13, 14, 111 L2 studies us Mig, 29, 30, 31, 32, 34, 35 verbal reporting, 25, 111 see also think-aloud (retrospective)

< previous page

page_174

next page >

< previous page

page_175

next page > Page 175

Revision of procedures, 77 Rhythmic patterns, non-native, 115, 116, 133 Rule presentation, 29 Russian case marking, 121-122, 134 S Salience, enhanced, 58, 60-61, 157-160 Sampling of data, 101-102 Scripts (knowledge structures), 22-23 Search and storage mechanisms, 23 Segmentation of data, 102-103 Self-corrected error repair, 41 Self-observation introspection as form of, 5 limitation on accuracy, 17 L2 studies using, 29, 32, 33, 35 verbal reporting, 13 Self-report, 12, 17, 48, 32, 33 Self-revelation, 13, 29, 30, 32, 35 Sentence acceptability, 101 Set up, time allocated to, 86, 88, 99 Skinner, B.F., and behaviorism, 8-9 Social object of introspection, 27, 28 Sociolinguistic interview technique, 61 Sociolinguistic issues, 100 Split-brain patients, 5-6 Spot-the-difference task, 90 Statistical testing, 103 Stimulated recall, 13 characterization, 37-55 classification by support, 47-55 function, 17-20 L2 studies using, 29, 30, 31, 33, 34, 35 limitations, 105-113 operation, 57-104 potential uses, 113-136 procedural recommendations, 62-63, 68, 84, 99 review of procedure, 37-47 topics investigated by, 20-24 types (consecutive, delayed, nonrecent), 51 versatility, 58 Stimtilus for recall, 25, 49, 53 cueing, 108 multiple source, 54 raters' familiarity with, 65, 66

strength, 54 written materials, 53, 54, 85 see also audio recordings; video recordings Strategies, L2, 51, 102, 111 and stimulated recall, 21, 23-24 Structure of task categorization of study by, 27, 28 in planning time study, 119 scale, 52 in stimulated recall, 49, 52, 55 time requirements, 88 Subjectivity, 100, 102-103 Support for introspection and stimulated recall, 13, 14, 25, 38, 47-55

< previous page

page_175

next page >

< previous page

page_176

next page > Page 176

amount, 27, 28 classification of, 26-28, 47-55 initiation of questions, 49, 53-54, 88 recommendations, 54-55 relationship to specific action, 27, 28, 48, 49-50, 54 structure, 49, 52, 55 time allocated to, 85-87 training, 49, 50-52, 54-55 see also stimulus Syntactic processing studies, 128-129, 135 T Tabulation of data, 103 Talk-aloud, 11, 12, 13, 14, 25, 33 Task schedule, 69, 74-75 Teacher training and evaluation, 18-19 Teacher- and learner-centred. groups, 124-125, 134 Temporal location, 52, 90-96, 97, 99 Tenses, 97-98n Think-aloud, 13, 14, 25 concurrent, 43 crossword puzzle task, 124 L2 studies using, 29-35, passim limitations, 124-125, 134 and pragmatics, 46-47 problem-solving, 13-16 retrospective, 29, 31, 32, 43 training for, 18, 51, 104n in verbal reporting, 11, 12, 13 Thinking, silent, 11, 12 Thought processes, access to, 18, 107 Time lapse between event and recall, 13, 14, 42, 49-50, 54 and accuracy of reporting, 17, 18, 89-90, 111, 112-113, 105, 107 classification of studies, 27, 28 and use of expectations or scripts, 22, 23 Time pressure in L2 processing, 40-41 Timing of recall procedure, 84-89, 99 Top-down process, 125 Training instructions distinct from, 51 of participants, 27, 28, 49, 50-52, 54-55 of researchers, 61-62, 63 of raters, 66, 69-76 sample schedule, 69, 75-76 for think-aloud, 18, 51, 104n

Transcription of data, 63, 65 Transfer, 8n, 122 Triangulation of data, 104, 114 Truth of results, 4, 107-108 Tutorial sessions, 44 U Unconscious learning, 23 Unexpected developments in tests, 87-88

< previous page

page_176

next page >

< previous page

page_177 Page 177

Unknown language, acquisition of words in, 131-132, 136 V Variables, individual learner, 53, 62, 63 Verbal encoding, see think-aloud Verbal reporting, 11-17, 25, 31 and cognitive processing, 106-107, 109, 111 concurrent, see talk-aloud; think-aloud limitations, 105-113 retrospection, 25, 111 Verbalization, 88-89, 108, 109 Verbosity, 88-89, 99 Veridicality, 107-108, 112 Versatility of stimulated recall, 58 Video recordings, 19, 20, 38, 39, 53, 85 combined with other stimuli, 54 writing research, 44-46 Vocabulary, 19, 43, 129-32, 136 incidental learning, 129-131, 136 in unknown language, 131-132, 136 word identification, 123, 129 words and nonwords, 121 W Warm-up tasks, 104n Whole-class studies, 53-54 Word identification, 123, 129 Word- and picture-learning conditions, 131-132 Writer's block, 44-46 Writing composing processes, 19, 46, 101 stimulated recall on, 25, 44-46, 51, 85 written introspection, 27, 28, 29, 30, 31, 33, 34 ,35 written stimuli, 53, 54, 85 Wundt, Wilhelm, 5, 7

< previous page

page_177