Assessment and Intervention (Advances in Learning and Behavioral Disabilities)

ASSESSMENT AND INTERVENTION ADVANCES IN LEARNING AND BEHAVIORAL DISABILITIES Series Editors: Thomas E. Scruggs and Ma...

Author: Thomas E Scruggs

251 downloads 2749 Views 2MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

ASSESSMENT AND INTERVENTION

ADVANCES IN LEARNING AND BEHAVIORAL DISABILITIES Series Editors: Thomas E. Scruggs and Margo A. Mastropieri Recent Volumes: Volume 12:

Edited by Thomas E. Scruggs and Margo A. Mastropieri

Volume 13:

Edited by Thomas E. Scruggs and Margo A. Mastropieri

Volume 14:

Educational Interventions – Edited by Thomas E. Scruggs and Margo A. Mastropieri

Volume 15:

Technological Applications – Edited by Thomas E. Scruggs and Margo A. Mastropieri

Volume 16:

Identification and Assessment – Edited by Thomas E. Scruggs and Margo A. Mastropieri

Volume 17:

Research in Secondary Schools – Edited by Thomas E. Scruggs and Margo A. Mastropieri

Volume 18:

Cognition and Learning in Diverse Settings – Edited by Thomas E. Scruggs and Margo A. Mastropieri

Volume 19:

Applications of Research Methodology – Edited by Thomas E. Scruggs and Margo A. Mastropieri

Volume 20:

International Perspectives – Edited by Thomas E. Scruggs and Margo A. Mastropieri

Volume 21:

Personnel Preparation – Edited by Thomas E. Scruggs and Margo A. Mastropieri

Volume 22:

Policy and Practice – Edited by Thomas E. Scruggs and Margo A. Mastropieri

Volume 23:

Literacy and Learning – Edited by Thomas E. Scruggs and Margo A. Mastropieri

ADVANCES IN LEARNING AND BEHAVIORAL DISABILITIES VOLUME 24

ASSESSMENT AND INTERVENTION EDITED BY

THOMAS E. SCRUGGS George Mason University, Fairfax, VA, USA

MARGO A. MASTROPIERI George Mason University, Fairfax, VA, USA

United Kingdom – North America – Japan India – Malaysia – China

Emerald Group Publishing Limited Howard House, Wagon Lane, Bingley BD16 1WA, UK First edition 2011 Copyright r 2011 Emerald Group Publishing Limited Reprints and permission service Contact: [email protected] No part of this book may be reproduced, stored in a retrieval system, transmitted in any form or by any means electronic, mechanical, photocopying, recording or otherwise without either the prior written permission of the publisher or a licence permitting restricted copying issued in the UK by The Copyright Licensing Agency and in the USA by The Copyright Clearance Center. No responsibility is accepted for the accuracy of information contained in the text, illustrations or advertisements. The opinions expressed in these chapters are not necessarily those of the Editor or the publisher. British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN: 978-0-85724-829-9 ISSN: 0735-004X (Series)

Emerald Group Publishing Limited, Howard House, Environmental Management System has been certified by ISOQAR to ISO 14001:2004 standards Awarded in recognition of Emerald’s production department’s adherence to quality systems and processes when preparing scholarly journals for print

CONTENTS LIST OF CONTRIBUTORS

vii

PREDICTIVE VALIDITY OF DYNAMIC TESTING AND WORKING MEMORY AS IT RELATES TO READING GROWTH IN CHILDREN WITH READING DISABILITIES H. Lee Swanson and Michael Orosco

1

APPLICATIONS OF CURRICULUM-BASED MEASURES IN MAKING DECISIONS WITH MULTIPLE REFERENCE POINTS Gerald Tindal and Joseph F. T. Nese

31

EVOLVING STANDARDS OF DIAGNOSTIC ACCURACY IN PREDICTING AND AVOIDING ACADEMIC FAILURE Amanda M. VanDerHeyden

59

HOW TO USE PSYCHOLOGICAL TESTS FOR FUNCTIONAL DIAGNOSIS: THE CASE OF ASSESSMENT OF LEARNING DISABILITIES Giulia Balboni and Roberto Cubelli

79

A COMPARISON OF OBSERVATIONAL TECHNIQUES FOR ASSESSING STUDENTS’ SOCIAL BEHAVIOR Jugnu Agrawal, Dannette Allen-Bronaugh and Margo A. Mastropieri CURRICULUM-BASED MEASUREMENT FOR BEGINNING WRITERS: RECENT DEVELOPMENTS AND FUTURE DIRECTIONS Kristen L. McMaster, Kristen D. Ritchey and Erica Lembke v

93

111

vi

CONTENTS

ISSUES IN ASSESSMENT FOR INTERVENTION IN IMPLEMENTATION OF RESPONSIVENESS TO INTERVENTION MODELS Rollanda E. O’Connor and Victoria Sanchez COMORBIDITY BETWEEN ATTENTION DEFICIT HYPERACTIVITY DISORDER AND READING DISABILITIES: IMPLICATIONS FOR ASSESSMENT AND TREATMENT Ana Miranda, Marıá Jesu´s Presentacioń, Rebeca Siegenthaler, Carla Colomer and Vicente Pinto ASSESSMENT AND INTERVENTION IN SELFDETERMINATION Michael L. Wehmeyer SUCCESS AND FAILURE WITH TIER-2 SRSD FOR TIMED-WRITING TESTS AMONG SECONDTHROUGH FIFTH-GRADE STUDENTS WITH WRITING AND BEHAVIORAL DIFFICULTIES: IMPLICATIONS FOR EVIDENCE-BASED PRACTICE Karin Sandmel, Kristen D. Wilson, Karen R. Harris, Kathleen Lynne Lane, Steve Graham, Wendy P. Oakes, Sharlene A. Kiuhara and Trish D. Steinbrecher

149

171

213

251

DEVELOPING WRITING FLUENCY FOR ADOLESCENTS WITH DISABILITIES Linda H. Mason and Richard M. Kubina

295

ENGAGING YOUTH WITH DISABILITIES IN SCHOOL: BUILDING AND SUSTAINING POSITIVE RELATIONSHIPS Elizabeth Talbott and Lisa S. Cushing

321

IMPROVING INTERVENTION EFFECTIVENESS WITH UNIVERSITY–PUBLIC SCHOOL COHORT PARTNERSHIPS Margo A. Mastropieri, Thomas E. Scruggs, Nicole Conners, Mary Kealy, Nancy Morrison, Tina Diamond and Terry Werner

341

LIST OF CONTRIBUTORS Jugnu Agrawal

College of Education and Human Development, George Mason University, Fairfax, VA, USA

Dannette Allen-Bronaugh


Giulia Balboni

Faculty of Psychology, University of Valle d’Aosta, Aosta, Italy

Carla Colomer

Department of Developmental and Educational Psychology, University of Valencia, Valencia, Spain

Nicole Conners

Fairfax County Public Schools, Fairfax, VA, USA

Roberto Cubelli

Department of Cognitive Sciences and Education, Center for Mind/Brain Sciences, University of Trento, Trento, Italy

Lisa S. Cushing

Department of Special Education, University of Illinois, Chicago, IL, USA

Tina Diamond


Steve Graham

Department of Special Education, Vanderbilt University, Nashville, TN, USA

Karen R. Harris


Mary Kealy

Loudoun County Public Schools, Loudoun, VA, USA vii

viii

LIST OF CONTRIBUTORS

Sharlene A. Kiuhara


Richard M. Kubina

Department of Educational and School Psychology and Special Education, the Pennsylvania State University, University Park, PA, USA

Kathleen Lynne Lane


Erica Lembke

University of Missouri, Columbia, MO, USA

Linda H. Mason

Department of Educational and School Psychology and Special Education, the Pennsylvania State University, University Park, PA, USA

Margo A. Mastropieri


Kristen L. McMaster

University of Minnesota, Minneapolis, MN, USA

Ana Miranda

Department of Developmental and Educational Psychology, University of Valencia, Spain

Nancy Morrison

Fairfax County Public Schools, Fairfax, VA, USA

Joseph F. T. Nese

Department of Special Education, University of Oregon, Eugene, OR, USA

Rollanda E. O’Connor

Graduate School of Education, University of California at Riverside, CA, USA

Wendy P. Oakes


Michael Orosco

Graduate School of Education, University of California at Riverside, Riverside, CA, USA

ix

List of Contributors

Vicente Pinto

Department of Developmental, Educational and Social Psychology, and Methodology, Jaume I University, Castelloń, Spain

Marıá Jesu´s Presentacioń


Kristen D. Ritchey

School of Education, University of Delaware, Newark, DE, USA

Victoria Sanchez

Graduate School of Education, University of California at Riverside, Riverside, CA, USA

Karin Sandmel

School of Education, Johns Hopkins University, Maryland, Baltimore, MD, USA

Thomas E. Scruggs

College of Education and Human Development, George Mason University, Virginia, Fairfax, VA, USA

Rebeca Siegenthaler


Trish D. Steinbrecher


H. Lee Swanson

Graduate School of Education, University of California, Riverside, CA, USA

Elizabeth Talbott

Department of Special Education, University of IL, Chicago, USA

Gerald Tindal

Department of Special Education, University of Oregon, Eugene, OR, USA

Amanda M. VanDerHeyden

Education Research & Consulting, Fairhope, AL, USA

x

LIST OF CONTRIBUTORS

Michael L. Wehmeyer

Department of Special Education, University of Kansas, Lawrence, KS, USA

Terry Werner


Kristen D. Wilson


PREDICTIVE VALIDITY OF DYNAMIC TESTING AND WORKING MEMORY AS IT RELATES TO READING GROWTH IN CHILDREN WITH READING DISABILITIES H. Lee Swanson and Michael Orosco ABSTRACT The purpose of this chapter is to review our findings related to the question ‘‘Do outcomes related to dynamic assessment on a cognitive measure predict reading growth?’’ Our discussion related to the predictive validity of such procedures focused on outcomes related to a battery of memory and reading measures administered over a three-year period to 78 children (11.6 years) with and without reading disabilities (RD). Working memory (WM) tasks were presented under initial, gain, and maintenance testing conditions. The preliminary results suggested that maintenance testing conditions were significant moderators of comprehension and vocabulary growth, whereas probe scores and gain testing

Assessment and Intervention Advances in Learning and Behavioral Disabilities, Volume 24, 1–30 Copyright r 2011 by Emerald Group Publishing Limited All rights of reproduction in any form reserved ISSN: 0735-004X/doi:10.1108/S0735-004X(2011)0000024003

1

2

H. LEE SWANSON AND MICHAEL OROSCO

conditions were significant moderators of nonword fluency growth. Overall, the results suggested that the dynamic assessment of WM added significant variance in predicting later reading performance.

Children with normal intelligence but who suffer from reading disabilities (RD) experience considerable difficulty on working memory (WM) tasks (e.g., see De Beni, Palladino, Pazzaglia, & Cornoldi, 1998; de Jong, 1998; Gathercole, Alloway, Willis, & Adams, 2006; Siegel & Ryan, 1989; Swanson & Siegel, 2001; Willcutt, Pennington, Olson, Chhabildas, & Hulslander, 2005). Although WM is integrally related to a number of academic behaviors (e.g., Gathercole & Pickering, 2000a, 2000b; Pickering, 2006; Swanson, 2008; Swanson, Jerman, & Zheng, 2008), relatively few studies have been undertaken to systematically explore whether WM can be improved upon and whether such improvements predict later academic performance. The purpose of this chapter is to review our findings on whether procedures that attempt to improve WM performance in children with RD predict reading growth. The procedures we use to assess improvements in WM performance come under the rubric of dynamic assessment (DA). Dynamic assessment is one of clearest links between assessment and instruction (e.g., Caffrey, Fuchs, & Fuchs, 2008; Sternberg & Grigorenko, 2002; Swanson & Lussier, 2001) because it incorporates a child’s response to instructional feedback. Although DA is a term used to characterize a number of approaches, its distinctive feature is to determine the learners’ potential for change when given assistance (e.g., Embretson, 1992; see Grigorenko & Sternberg, 1998 for a review), and to provide a prospective measure of performance change independent of assistance (e.g., Brown & Ferrara, 1999). Unlike traditional testing procedures, score changes due to DA methods are not viewed as threatening task validity. In fact, some authors argue that construct validity will increase (e.g., Carlson & Wiedl, 1979; however see Caffrey et al., 2008; Swanson & Lussier, 2001, for a critical review). For example, DA procedures have been found to contribute unique variance to concurrent criterion measures of achievement even when the influence of traditional assessment procedures was partialed out in the analysis (e.g., Swanson, 1999). Although the literature is clear that impairments in WM in children with RD are related to reading performance, whether DA procedures add additional information to the prediction of growth in reading in children with RD beyond traditional testing procedures has not been tested. Perhaps

Predictive Validity of Dynamic Testing

3

the closest study on this issue is a cross-sectional study by Swanson (2003) which determined whether improvements in WM differentiated children with and without RD. The study compared RD and skilled readers (N ¼ 226) across four (7, 10, 13, 20) age groups on WM tasks under various WM conditions: initial (no probes or cues), gain (cues that bring performance to an asymptotic level), and maintenance conditions (asymptotic conditions without cues). The general findings were that skilled readers performed better than children with RD in all processing conditions and that concurrent reading comprehension performance was best predicted by the maintenance testing than the other testing conditions. Further, the magnitude of the difference (effect size) between RD and skilled readers increased on gain and maintenance testing conditions when compared with the initial conditions, suggesting that performance differences between ability groups were enhanced by using DA procedures. However, there was no follow-up testing of these children, raising the questions as to whether the measures were predictive of later reading performance. The work we describe in this chapter outlines our attempts to improve WM performance, via adapting procedures used in DA, and then predict later reading performance. The question of interest was whether DA procedures provide additional information (contributed unique variance) in predicting children’s reading performance relative to traditional testing procedures. To answer this question we assessed WM performance under three conditions across three years. These conditions included: (1) presentation of WM tasks without cues to assess initial performance (initial condition), (2) presentation of graduated cues to help participants access forgotten information from the initial condition and to continue the use of cues until span scores could no longer be improved upon (referred to as the gain or asymptotic testing condition), and (3) presentation of the highest span level achieved for the gain condition after a brief interlude, but without the support of cues (referred to as the maintenance condition). The administration of WM tasks under initial testing conditions were assumed to tap individual differences in idiosyncratic processing, as well as individual differences in accessing newly presented items in storage. To assess individual differences in item accessibility, probes (probes and cues are used interchangeably) were presented to help children reinstate the memory trace and/or retrieve forgotten items. This condition, referred to as the gain testing condition, allowed participants to use as many cues as necessary to access previously forgotten information. The probes were ordered in terms of explicitness, with the general prompts given first and more explicit prompts later. To assess constraints in activating previously retrieved information, the

4


testing procedures reinstated the highest level achieved successfully under gain conditions, but without cues. This condition, referred to as the maintenance condition, presented the same items retrieved in the gain conditions. The presentation of items in the maintenance testing condition was calibrated to the individual’s highest span level and allowed for the assessment of individual processing differences beyond the simple learning of items. Prior to reviewing the outcomes, we briefly describe our sample, procedures, and analysis. In this study, subgroups of children with RD were tested across three testing waves approximately one year apart. We predicted that DA procedures would differentiate WM performance among subgroups of children with RD. For example, a previous study by Swanson and Howard (2005) found that children with RD-only and children with general achievement problems could be separated in terms of performance using DA procedures. They showed that skilled readers and children considered poor readers (children with low IQ and reading scores) were more likely to change and maintain their WM score from initial testing conditions than children with RD-only or children with a combination of math/reading disabilities. Overall, the results suggested that WM deficits were less changeable for children with RD when compared to low IQ readers and skilled readers. To this end, our most recent study included 78 children selected from a school in Southern California. The mean age of the sample in the first testing wave was 11.62 years (SD ¼ 2.28) and included 27 girls and 51 boys. Sixty-six children were Anglo-American, eight AfricanAmerican, and four were of Hispanic heritage. All children were from middle-upper to upper class homes. The criterion for the average reading group was a score above the 45th percentile in reading word recognition, as well as an IQ score above 90. Classification of subgroups of children with RD was based upon standard scores from the word-reading and arithmetic subtests of the Wide Range Achievement Test 3 (WRAT3, Wilkinson, 1993). Operational criteria for children defined as the RD-only group were based upon: (a) IQ scores (in this case scores on the Raven Colored Progressive Matrices Test, Raven, 1976) above 90, (b) reading recognition scores at or below the 25th percentile, and (c) math calculation scores above the 25th percentile. These cut-off score criteria for children with RD-only matched the operational definition of RD outlined by Siegel and Ryan (1989). Children with average fluid intelligence (scores on the Raven Progressive Colored Matrices Test) and verbal IQs but whose performance on the WRAT3 in both word


5

recognition and calculation were below the 25th percentile were classified as RD children with comorbid deficits (RD þ MD). Children whose fluid intelligence (Raven Progressive Colored Matrices Test) were in the average range but tested low (o25th percentile) in both arithmetic and word recognition and obtained low verbal IQs (o85, subtests on the WISCIII that included information and vocabulary), were defined as low verbal IQ readers. All children were monolingual in English. For comparison purposes, participants were divided into four ability groups: children with RD-only (low word recognition/average arithmetic, n ¼ 13), children with both reading and calculation deficits (comorbid group, n ¼ 15), low verbal IQ readers (low verbal IQ, low word recognition, and low arithmetic, n ¼ 27), and skilled readers (high word recognition, high arithmetic, n ¼ 23). No significant differences emerged among subgroups related to gender, ethnicity, or chronological age. Because ability groups were categorized on measures of word recognition and math skills, it was important to follow growth on measures related to other forms of reading achievement independent of the classification measures. Three criterion measures were selected: reading comprehension, nonword reading fluency, and receptive vocabulary. Because real sight words were used in the classification of reading groups, we selected pseudoword reading speed because of its strong association with phonological skills. Further, because expressive vocabulary was used to establish verbal IQ, we selected the receptive language test (PPVT-III) as a criterion measure. Reading comprehension was assessed by the Passage Comprehension subtest from the Woodcock Reading Mastery Test Revised (WRMT-R; Woodcock, 1998). Nonword reading efficiency was assessed by the Phonemic Decoding Efficiency subtest from the Test of Word Reading Efficiency (TOWRE; Wagner & Torgesen, 1999). Receptive vocabulary was assessed by The Peabody Picture Vocabulary Test-Third Edition (PPVT-III; Dunn, Dunn, Dunn, & Dunn 1997).

WORKING MEMORY MEASURES AND TESTING CONDITIONS Four WM subtests from a battery of 11 of the S-Cognitive Processing Test were selected because they represent verbal and visual-spatial processing (Swanson, 1995a, 1995b).

6


Verbal Working Memory Rhyming The purpose of this task was to assess the child’s recall of acoustically similar words. The child listened to sets of words that rhyme. Each successive word in the set was presented every 2 seconds. There were 9 word sets that ranged from 2 to 14 monosyllabic words. The dependent measure was the number of sets recalled. Before recalling the words, the child was asked whether a particular word was included in the set. For example, the child was presented the words ‘‘lip-slip-clip’’ and then asked if ‘‘ship or lip’’ was presented in the word set. They were then asked to recall the previously presented words (lip-slip-clip) in order. The dependent measure was the number of sets recalled correctly (range of 0–9). If the child omitted, inserted, or incorrectly ordered the words, a series of probe responses were presented. Probe responses continued until the child could no longer provide the correct response. For example, consider the sample item ‘‘car-star-bar-far’’ (Item #3) and the process question ‘‘Which word did I say – jar or star?’’ Consider the Probe Sequence: 1. The last word in the sequence was ‘‘far,’’ now can you tell me all the words in order? 2. The first word in the sequence was ‘‘car,’’ now can you tell me all the words in order? 3. The middle words in the sequence are ‘‘star’’ and ‘‘bar,’’ now can you tell me all the words in order? 4. All the words in order are ‘‘car-star-bar-far,’’ now can you tell me all the words? For each set of items not recalled in the correct order or for items left out or substituted, the experimenter provided a series of hints based on the error that was closest to Probe 1. That is, probes went from the least obvious hint (Probe 1) to the next explicit hint that facilitated recall of the answer. Once the appropriate hint had been identified, based on the location of the error, probes were presented in order until the correct sequence was given. For example, suppose the child for Item #3 responded car-bar-far. The child obviously left out a word in the middle so the experimenter would provide a hint related to the middle words (Probe 3 in this case). If Probe 3 did not provide the correct response, the experimenter would then move to Probe 4. In contrast, if a child responded initially by saying only car, the sequence began with Probe 1 and proceeded through all probes until the correct response was given. If a correct response did not occur after probing, the


7

task was discontinued and the next task was administered. If a correct response did occur, the next set of items of increased difficulty was presented. The format of the probe procedures above was used for the other subtests. Digit/Sentence Task The purpose of this task was to assess the participant’s ability to remember numerical information embedded in a short sentence. The administration of items and probes followed the same format as the rhyming task. Prior to stimulus presentation, the participant was shown a figure (see Swanson, 1993, Fig. 1) depicting four strategies for recalling numerical information. These strategies were pictorial representations of rehearsal, chunking, associating, and elaborating of information. The general instructions for introducing the strategies were as follows: I’m going to read you some sentences that have information I want you to remember. All the sentences have to do with remembering an address, but I would like you to pay attention to all the information in the sentence because I will ask you a question about the sentence. After I present this information, and before you recall it, I will ask you to choose a strategy (for children under ten – the phrase, ‘‘A way of remembering the information’’ was used) that you think will best help you remember.

The experimenter then showed four pictures each depicting a person thinking about using one of the four strategies (see Swanson, 1993). As the experimenter explained each strategy, they pointed to the picture that matched the description. The experimenter stated that Some of the ways that may help you remember are: (1) saying the numbers over to yourself. For example, if I say ‘‘2-4-6-3 Bader Street,’’ you would say to yourself ‘‘24-6-3’’ over and over again, or (2) you might say some numbers together in pairs. For example, if I say the numbers ‘‘2-4-6-3 Bader Street,’’ you would say ‘‘24 and 63,’’ or (3) you may just want to remember that the numbers go with a particular street and location. For example, if I say ‘‘2-4-6-3 Bader Street’’ you would remember that 2-4-6-3 and Bader Street goes together, or (4) you might think of other things that go with the numbers. For example, if I say ‘‘2-4-6-3’’ you might think 2-4-6-3 I have to go climb a tree.

These four pictorial representations of strategies generally reflect rehearsal, chunking, associating, and elaborating of information, respectively. After all strategies had been explained, participants were then presented item sets that included numbers in a sentence context. They were then told that they must recall the numbers in the sentence in order shortly after they select from (point to) a pictorial array representing the strategy that best approximates how he or she will attempt to remember the information.

8


No further information about the strategies shown in the picture was provided to the participant. Participants were allowed 10 seconds in which to make a decision. The range of recall difficulty was 3 digits to 14 digits, and the dependent measure was the highest number of sets correctly recalled (range of difficulty 0–9). Thus, the sequence of the steps for administration after the introduction was as follows: (1) The participant was orally read a sentence (the numbers in the sentence were presented at the rate of approximately 1 every 2 seconds), (2) the participant was asked a process question which required them to give the name of the street referred to in the target sentence, (3) the participant was asked to select one of the four strategies that were represented pictorially that were most like the one they would use to remember the order of the street numbers, (4) the participant was asked to recall the numbers of the address in the order in which they were originally presented, and (5) if an error in recall occurred, the probe questions were implemented. Probing procedures followed the same format as the rhyming task: hints were provided sequentially based on the type of error, and ranged from least obvious hint (Probe 1) to the next explicit hint that facilitated recall of the answer. If probing did not elicit a correct response, the task was discontinued and the next task was administered. If a correct response did occur, the next set of items of increased difficulty was presented.

Visual-Spatial Working Memory Visual Matrix The purpose of this task was to assess the participant’s ability to remember visual sequences within a matrix. The participant was presented a series of dots in a matrix and was allowed 5 seconds to study the matrix. The matrix was removed and the participant was asked a discrimination question, i.e., ‘‘Are there any dots in the first column?’’ To ensure the understanding of columns, the experimenter pointed to the first column on a blank matrix (a grid with no dots). After answering the discrimination question, the participant was asked to draw the dots in the correct boxes on the blank matrix. The task difficulty ranged from a matrix of 4 squares and 2 dots to a matrix of 45 squares and 12 dots. The dependent measure was the number of matrices recalled correctly (range of 0–11). If an error occurred, probe questions were started.


9

Probe procedures followed the same format as the verbal subtests, except that the matrices were represented as columns that reflect recency, primacy, and middle positions. For example, when probing for errors that occurred in the recency position, the experimenter drew dots on a blank matrix in the appropriate last columns and said, ‘‘Now can you show me where the rest of the dots go?’’ During probing, the matrices response form in which initial scores were established was turned over. For each hint provided to the participant, a new matrix was presented, the experimenter provided the hints (demonstrates by filling in the appropriate dots) on this matrix and then the participant was asked to fill in the correct remaining dots. If some hints do not have dots in which to demonstrate, the participant was told so. The instructions related to probing were as follows: ‘‘You missed placing some dots in the right boxes. I think you can do it correctly if I provide you with some hints.’’ Probe 1. On the blank matrix, the experimenter correctly draws in the last column(s) of dots. Then the experimenter says, ‘‘Now can you draw where the rest of the dots go?’’ Probe 2. On a blank matrix, the experimenter draws in dots for the first column. Then the experimenter says, ‘‘Now can you draw where the rest of the dots go?’’ Probe 3. On a blank matrix, the experimenter draws in the dots for the middle (between the first and last column) and says, ‘‘Now can you draw where the rest of the dots go?’’ Probe 4. The stimulus card is shown for 2 seconds. The model matrix is removed and the participant is asked to fill in all the dots on a blank matrix.

Mapping and Directions The purpose of this task was to determine whether the participants could remember a sequence of directions on a map that is void of labels (see Swanson, 1995a for detailed instructions). The experimenter presented the participant with a street map with lines connected to a number of dots that illustrated the direction a car would go to get out of the city (see Swanson, 1993). The dots represented stoplights, and the lines represented the direction the car should go. The participant was given 5 seconds to study the map. After the map was removed, the participant was then asked a process question and to point to the strategy (picture) they thought they would use to remember the street directions. Finally, they were asked to draw on a blank map the street directions (lines) and stop lights (dots). The process question was ‘‘Were there any dots in the first street (column)?’’ Using the same pictorial format as the Digit/Sentence Task, strategies were pictorial representations of elemental, global, sectional, or backward processing of patterns. The range of difficulty included dots that ranged

10


in number from 4 to 19. The dependent measure was the number of maps drawn correctly (range of 0–9). If an error was made, a series of probes was presented following the same format as the visual-matrix task procedures. Probing continued until the participant could no longer provide the correct response. The task was subsequently discontinued and the next task was administered.

Validity and Reliability of WM Measures The psychometric qualities of the measures are reviewed in Swanson (1995a, 1995b, 1996). Cronbach alpha measures on the subtest vary from .78 to .96. The reliabilities for the WM measures have been sustained when translated into Spanish (e.g., Swanson, Saez, Gerber, & Leafstedt, 2004) and Japanese (e.g., Hwang et al., 2006). Correlations with each WM subtest vary on intelligence, achievement, and Listening Span (Daneman & Carpenter, 1980) measures from .40 to .80 (e.g., see Swanson, 1992, Tables 4 and 7, Swanson, 1996, see Table 4).

Procedure Individual testing of children was performed by five doctoral students over the course of three years. All items for the WM measures for the initial testing condition were administered until (a) a process question was missed or (b) an error in retrieval occurred. If an error in retrieval occurred (a participant omitted, inserted, or incorrectly ordered the numbers, dots, words, related to the appropriate task), probes were administered. The only stipulation for instituting the probing condition was that the process question be answered correctly. Probes were administered based on the type of error made (i.e., whether the error was related to recency, primacy, or middle items), and probing procedures continued until all targeted items could not be recalled correctly. After all four subtests from the S-CPT were administered under initial and gain testing conditions, participants were re-administered the items for the highest successful set (highest set of items established under gain conditions) for each task. In order to compare ability groups across verbal and visual-spatial WM measures, mean composite z-scores for verbal (rhyming þ auditory sequence)


11

and visual-spatial (matrix þ mapping/direction) WM performance from the initial testing conditions were computed. Z-scores for the gain and maintenance testing conditions were based on the means and SDs from the initial condition. The composite z-scores for wave (year) 1, 2, and 3 as a function of ability group and testing condition are shown in the appendix. The z-scores for wave 2 and 3 were based on the mean and standard deviations of span scores in the initial testing condition at wave 1. As a preliminary analysis of the findings we calculated effect sizes for the gain [(mean gainmean initial)/SDgain] and maintenance testing conditions [(mean gainmean initial)/SDgain] relative to the initial testing conditions. The standard deviations from the DA conditions (in this case gain or maintenance scores) were selected because pretest scores (in this case initial testing scores) are generally inflated (see Swanson & Lussier, 2001, for discussion).1 Further adjustments were made in the calculations because of the high correlations among tasks with repeated measures (see Rosenthal, 1994, p. 241, for formula). Table 1 shows the magnitude of the effect sizes (ES) as a function of gain or maintenance testing conditions. Also shown are the ES corrected for the correlation between the initial and DA conditions. Also shown is the ratio of inflation in standard deviations by dividing the standard deviation for the DA measures (gain or maintenance) by the standard deviation for initial testing measure. A previous synthesis of DA measures (Swanson & Lussier, 2001) suggested that ratios greater than 1.25 reflect practice effects rather than responsiveness to DA techniques. That is, initial testing conditions rather than DA served as the learning event. Using Cohen’s (1988) criterion for large (.80) and moderate (.50) effect sizes, Table 1 shows that all ability groups yielded high effect sizes for the gain testing condition and moderate effect sizes for the maintenance testing conditions. Using 1.25 ratio as a critical value, practice effects from the initial test condition were most notable for the RD þ MD group (4 of 6 values were greater than 1.25). Another important finding was related to the reduction in performance from the gain to maintenance testing condition. The largest average drop in performance from the gain to the maintenance testing condition (ES for gain condition minus ES for maintenance condition) occurred for the RD-only group (.86 ES difference), whereas the remaining groups yielded a reduction in ES that were in the same range [RD þ MD (.54 ES difference), low verbal IQ reader (.56 difference), and skilled readers (.48 difference). Thus, the RD-only group showed the largest drop in performance (3/4 of SD) from the gain to the maintenance testing condition when compared to the other groups.

12

Table 1.

RD-only Wave 1 Wave 2 Wave 3 RD þ MD Wave 1 Wave 2 Wave 3 Low VIQ Wave 1 Wave 2 Wave 3 Skilled Wave 1 Wave 2 Wave 3


Effect Sizes (ES), Retesting Sensitivity, and Corrected Effect Sizes by Reading Group. GainES

MaintenanceES

Gain Ratioa

Maintenance Ratioa

Corrected Gain ES

Corrected Maintenance ES

1.64

1.01

1.02

0.90

1.03

0.63

1.88

0.48

0.86

1.35

1.19

0.3

2.69

0.64

0.92

1.43

1.7

0.4

1.2

0.45

1.67

1.46

0.76

0.28

1.41

0.55

1.98

2.10

0.89

0.35

2.10

1.18

0.92

0.94

1.33

0.74

1.21

0.51

1.32

1.09

0.77

0.32

1.76

0.73

1.33

1.23

1.12

0.46

1.69

0.72

1.12

1.24

1.07

0.45

1.71

0.69

0.89

0.96

1.08

0.43

1.27

0.7

1.74

1.08

0.81

0.44

1.67

1.02

0.99

1.11

1.06

0.64

a

Ratio of inflation in standard deviations by dividing the standard deviation for the DA measures (gain or maintenance) by the standard deviation for initial testing measure.

Summary A descriptive analysis of ES between gain and maintenance conditions suggested that the RD-only group was less likely to maintain performance when compared to the other groups.


13

Repeated Measures Prior to determining whether the various testing conditions of WM contributed to predictions of growth on measures of reading comprehension, nonword fluency, and receptive language, we used mixed modeling with repeated measures across testing waves to determine if significant differences in growth emerged between ability groups as a function of testing conditions. The mean z-scores for each reading group as a function of domain (verbal vs. visual) across the testing waves are shown in the appendix. Because span scores were greater in the verbal than visual-spatial domain under DA conditions at wave 1, only verbal WM span scores were considered in the subsequent analysis. In addition, because initial testing may have served as a learning event, initial testing scores were partialed in the analysis. Further, because chronological age at initial testing in wave 1 may interact with the amount of change across testing waves, chronological age at wave 1 was a covariate in the analysis. The results indicated that skilled readers outperformed the RD subgroups across all testing waves and DA conditions. The DA conditions did not differentiate between subgroups of less skilled readers. Because the RD subgroups could not be differentiated across the testing waves, we combined the performance RD subgroups in the subsequent analyses.

Correlations An important question to address was whether DA procedures in wave 1 were related to achievement measures in wave 3. As shown in Table 2, the majority of verbal WM measures in wave 1 were significantly related to the criterion measures in wave 3. However, it is unclear from the correlation analysis which of the testing conditions in wave 1 uniquely moderated growth on the criterion measures. To address this issue, latent growth modeling was conducted to determine which of the testing conditions in wave 1 moderated growth on the criterion measures. Verbal WM measures were used since the majority of significant correlations shown in Table 2 were related to the verbal WM measures.

Growth Modeling To examine growth in reading, we used a multilevel framework (Singer, 2002; Singer & Willet, 2003). We applied individual growth modeling to the

14


Table 2. Correlations with Working Memory Measures as a Function of Testing Conditions (Initial, Gain, Maintenance) and Criterion Measures for the Total Sample. Wave 3 Measures Passage comprehension Composite wave 1 Verbal WM measures Initial Gain Main Probes

0.51 0.42 0.50 0.13

Visual-spatial WM measures Initial 0.29 Gain 0.23 Maintenance 0.27 Probes 0.06

Nonword fluency

0.27 0.28 0.34 0.02 0.11 0.06 0.05 0.06

Receptive language

0.56 0.51 0.37 0.20 0.31 0.29 0.38 0.17

po.05, po.01, po.001.

study of intra-individual change in reading as function of WM testing conditions over the three-year period via the PROC MIXED program (SAS, 2003). This hierarchical linear modeling (HLM, Bryk & Raudenbush, 2002) allowed us to determine both the average rate of change and individual variability in change over time. Age was the variable that represented the passage of time in our growth model. To interpret the results, we centered age at 12.48 (sample mean chronological age at wave 2), so that intercepts reflected the expected performance at that age. Our growth model yielded parameter estimates that defined both the overall trajectory of the sample (fixed effects) and deviations in the overall trajectories (random effects). This unconditional linear random regression model was expressed as: yij ¼ b0 þ b1 ðageij Þ þ U 0j þ U 1j ðageij Þ þ Rij where yij is the dependent variable (e.g., passage comprehension) measured at time i in child j, ageij is child j’s age at time i, b0 is the average intercept at 12.48. b1 is the linear slope. U0j is the random intercept for child j in the sample as a whole. U1j is the random age slope for child j. Rij is the residual for child j at time i. The between-child variance components, t20 ¼ VarðU 0j Þ; t201 ¼ VarðU 01j Þ and t21 ¼ VarðU 1j Þ reflected individual differences


15

in level of performance for the sample as a whole, and the rate of change between children, respectively. We estimated the association between the outcome (e.g., reading comprehension) and repeated measures of age across the three-year time periods. Both unconditional and conditional models are reported. For the unconditional model, reported are the fixed effects for the intercept value for wave 2 at age 12.48 and the average rate of change across individuals. For the random effects, the estimates are the variance around the sample intercept and the slope related to change overtime. Significant random effects indicated that children differed in intercepts and/or rate of change (slopes). For the conditional model, we tested whether entering WM span scores from the DA conditions into the model explained any statistically significant associations related to fixed and random effects. When one or more predictors are introduced in the conditional model, there are reductions in the magnitude of the various components. These reductions are analogous to effect sizes (Snijders & Bosker, 1999). To evaluate the compatibility of the data with our conditional model, we tested the significance of the model change. This was done by comparing the differences between the deviance values (i.e., the likelihood value for the correspondence between model and data) from the unconditional and conditional growth models. These are chi-square values, and the number of parameters added for the conditional model serves as degrees of freedom. In general, models with lower deviance fit better than models with higher deviance values. For the present study, a significantly lower deviance score for the conditional model indicated that the conditional model showed a better fit to the data than the unconditional model. Prior to the analysis, we converted raw scores for the criterion measures (passage comprehension, nonword fluency, receptive language) to z-scores. Wave 2 and 3 measures were z-scored based upon the means and standard deviations of wave 1. The results were centered on wave 2. The values were centered at wave 2 because group differences were firmly established on a number of measures at wave 1. Age in months at each testing wave was used to measure change. Tables 3–5 show the estimates related to growth and the moderating effects of DA conditions for reading comprehension, fluency, and vocabulary, respectively. For the conditional models, Model 2 included the categorical variable for skilled versus children with RD (combined performance of RD-only, RD þ MD, low verbal IQ poor readers). Model 3 added the testing conditions for the verbal WM measures. Also of interest

16


Hierarchical Linear Modeling for Growth in Passage Comprehension.

Table 3. Model

Passage Comprehension 1

2

3

Random effects Intercepts Slope Residual Deviance

Variance .28 .10 .17 Fit-index 338.0

SE .16 .02 .02

Estimate .15

SE .05

Variance .34 .06 .16 Fit-index 296.1

SE .17 .02 .02

Variance .26 .04 .14

SE .04 .01 .02 Fit-index 255.9

SE .09 .11 .004 .0001

Estimate .30 .23 .01 .001

SE .14 .14 .004 .001

Fixed effects Intercept RD Slope RD Moderators Initial Gain Main. Probe Interaction Gain Main.

0.01

0.001

Estimate .69 .75 .014 .0002

.29 .11 .31 .17 .32 .13

.11 .16 .14 .12 .14 .15

po.05, po.01.

Note: Interactions, ability group dynamic testing condition; main., maintenance testing condition; gain, gain testing condition; initial, initial testing condition; probe, number of probes used to establish gain condition; RD, less skilled readers (combined subgroups of children with RD).

was whether DA testing conditions interacted with ability group performance. Passage Comprehension (WRMT) Table 3 shows the outcomes for one unconditional and two conditional models. To interpret the table, first consider the unconditional model for the fixed effects. The fixed effects model provided an estimate of the intercept (average z-score score at wave 2) for the total sample. As shown, the average

17


Table 4.

Hierarchical Linear Modeling for Growth in Nonword Fluency.

Model

TOWRE: Nonword Fluency 1

2

3

Random effects Intercept Slope Residual Deviance

Variance 1.07 0.27 0.32 Fit-index 447.9

SE 0.29 0.11 0.04

Variance 0.88 0.22 0.24 Fit-index 383.5

SE 0.22 0.08 0.03


Estimate 0.20

SE 0.07

Estimate 1.06 1.20 0.01 0.002

SE 0.12 0.15 0.004 0.005

Estimate .57 .83 .007 .0002

SE .32 .02 .03 Fit-index 365.1

Fixed effects Intercept RD Slope RD Moderators Initial Gain Main. Probe Interaction Gain Main.

.011

.002

SE .20 .19 .005 .002

.29 .57 .14 .35

.16 .22 .19 .18

.34 .18

.20 .21

po.05, po.01. Note: Interactions, ability group dynamic testing condition; main., maintenance testing condition; gain, gain testing condition; initial, initial testing condition; probe, number of probes used to establish gain condition; RD, less skilled readers (combined subgroups of children with RD).

passage comprehension z-score (mean z-score of test administered at time 2) for all children was .15. The slope (.01) indicated that the average child in the total sample improved .01 units for each testing wave. Also presented in Table 3 for the unconditional model were the random effects portions of the model for the total sample. The random effects included the variance of conditions related to the intercept (.28) and slope (.10). The residual error was .17. The random effects reveal that 70% [.28 þ .10)/(.28 þ .10 þ .17)] of the between-subject variance in passage comprehension was related to random effects of child differences at the intercept and slope.

18


Table 5.

Hierarchical Linear Modeling for Growth in Receptive Language.

Model

Receptive Language 1

2

3

Random effects Intercept Slope Residual

Deviance

Variance 0.44 0.08 0.13 Fit-index

SE 0.15 0.02 0.02

320.7

Variance 0.34 0.06 0.13 Fit-index

SE 0.12 0.01 0.02


301.5

SE .11 .01 .01 Fit-index 265.1

Fixed effects Intercept RD Slope RD Moderators Initial Gain Main. Probe Interactions Gain Main.

Estimate 0.17 0.028

SE 0.05 0.002

Estimate 0.55 0.53 0.02 0.001

SE 0.10 0.11 0.003 0.004

Estimate .30 .22 .007 .003

SE .16 .15 .004 .001

.29 .07 .37 .08

.12 .18 .15 .14

.10 .48

.15 .16

p o .05, p o .01.

Note: Interactions, ability group dynamic testing condition; main., maintenance testing condition; gain, gain testing condition; initial, initial testing condition; probe, number of probes used to establish gain condition; RD, less skilled readers (combined subgroups of children with RD).

The next column (Model 2) in Table 3 shows the conditional model that included the entry of the classification variable (skilled vs. children with RD). The model indicated that the average passage comprehension z-score for the children in wave 2 without reading problems was .69. Thus, the skilled reader at wave 2 had gained approximately 3/4 of a standard deviation in passage comprehension performance. Model 2 also showed that without entering moderator variables related to testing conditions, passage


19

reading comprehension performance was .75 z-score points lower (estimate was .75) for children with RD when compared to skilled readers. The growth rate for skilled readers was approximately .02 (.014) units of improvement per testing session. Although the estimate for children with RD showed a lower growth rate (.0002), the estimate was not significantly different from skilled readers. A comparison was made between the unconditional and conditional models on the random intercept (.28 vs. .34) and slope (.10 vs. .06). The addition of the classification measure accounted for approximately 40% of the explainable variance in growth [(.10–.06)/.10)]. However, the variance increased between children on the intercept values, suggesting the model was not a good fit to the data. Model 3 added probe scores and span scores from the WM testing conditions. Also included were potential interactions between reading ability group and the DA condition. The results indicated that WM performance related to initial and maintenance testing for the verbal tasks and the ability group gain testing condition moderated passage comprehension. To interpret these findings it is important to note that predictor variables were centered. That is, the estimate for passage comprehension was .30 when predictors were set to zero. The estimate for the maintenance testing condition was .31. Taken together, the results indicated that children who differ by 1 point on WM for the maintenance testing condition differ by .31 points on passage comprehension. The negative estimates for the interaction indicated that skilled readers were more likely to benefit from the gain conditions than children with RD. A deviance test was computed to determine the fit of Model 3. A deviance test indicated that the Model 3 provided a better fit to the data than Model 2 for passage comprehension, w2 (df ¼ 6) ¼ 40.9 (296.1–255.9), po.001. When compared to Model 2, the measurement of WM across various testing conditions in Model 3 accounted for 24% of the explainable variance in intercepts [(.34–.26)/.34] and 33% of the explainable variance between children in growth [(.06–.04)/.06].

Nonword Fluency (TOWRE) Table 4 shows the results for predicting reading fluency for nonwords. The total sample average z-score (mean z-score of test administered at time 2) for all children was .20. The slope (.01) indicated that children improved .01 units for each testing wave. The random effects for the unconditional model were significant for the intercept (1.07), slope (.27), and the residual error (.32.) The random effects reveal that 81% [1.07 þ .27)/(1.07 þ .27 þ .32)] of

20


the between-subject variance in nonword fluency was related to child differences at the intercept and slope. Model 2 indicated that the nonword fluency z-scores for skilled reading children at wave 2 was 1.06. Children with RD were 1.20 z-score points lower than skilled readers. However, no differences were found between the two groups in growth. The addition of the classification measure in Model 2 explained about 19% of the variance in growth [(.27–.22)/.27)]. Model 3 added probe scores and span scores for the verbal WM testing conditions. As shown for the random effect, the intercepts and slopes decreased when compared to Model 2. The results indicated that span scores from the gain condition and probe scores significantly moderated nonword fluency. The results indicated children who differed by 1.0 with respect to the scores on the gain testing condition for verbal WM tasks differed in intercept values by .57 in the nonword fluency. The deviance test indicated that Model 3 showed a good fit to the data when compared to Model 2, w2 (df ¼ 6) ¼ 18.41 (383.50–365.1), po.001. When compared to Model 2, the measurement of WM across various testing conditions in Model 3 accounted for 17% of the explainable variance in intercepts [(.88–.73)/.88] and 68% of the explainable variance between children in growth [(.22–.07)/.22].

Vocabulary As shown in Table 5, the fixed effects in Model 3 indicated that the slopes were higher for children with RD than skilled readers. The important findings were that the initial and maintenance conditions moderated receptive vocabulary. As shown, the fixed effects for Model 3 eliminated group differences at wave 2 for the intercept estimates. Performance on the maintenance testing condition was a significant moderator of receptive vocabulary as was the ability group maintenance testing condition interaction. The results suggested that children skilled in reading were better able to sustain WM performance at their highest level (based on gain testing conditions) without cues than children with RD. The deviance test indicated that the Model 3 showed a good fit to the data when compared to Model 2, w2 (df ¼ 6) ¼ 36.40 (301.50–265.1), po.001. When compared to Model 2, the measurement of WM across various testing conditions in Model 3 accounted for only 3% of the explainable variance in intercepts [(.34–.33)/.24], but 33% of the explainable variance between children in growth [(.06–.04)/.06].


21

Summary The growth modeling showed that various DA conditions moderated outcomes on the criterion measures. Maintenance testing conditions were important moderators in the growth model of passage comprehension and receptive vocabulary, whereas gain testing conditions and probing moderated nonword fluency. The conditional model that included measures of WM accounted for approximately 33% of the explainable growth variance in passage comprehension and receptive vocabulary and 68% of growth in nonword fluency. Ability group interactions emerged for gain testing conditions for passage comprehension, and maintenance testing conditions for vocabulary (receptive language). These interactions indicated that skilled readers were more likely to benefit from those DA conditions than children with RD.

SUMMARY OF FINDINGS AND IMPLICATIONS We will now consider the two questions that motivated this preliminary work. 1. Does dynamic assessment contribute to later predictions of achievement in children with RD? The results of the HLM analysis support the hypothesis that DA adds important variance in predicting later reading achievement. These results showed that various DA conditions (e.g., conditions related to gain, probing, and/or maintenance) significantly moderated passage comprehension, nonword reading fluency, and receptive language. Interestingly, although intercept values favored skilled readers on all criterion measures, growth on the reading (passage comprehension, nonword fluency) and receptive language measures was statistically comparable between skilled and children with RD. These findings complement some of our earlier work. Our earliest study (Swanson, 1992, Experiment 2) assessed whether dynamic testing of WM predicted concurrent reading recognition performance when initial testing conditions were partialed out in the analysis. In this study, children (mean age 10 years) were administered WM tasks under initial, gain, and maintenance conditions. Effect sizes (ES) for gain and maintenance performance relative to initial conditions (ES .90 for gain scores and .47 for maintenance scores) were in the same range as the present study. The

22


regression analysis found that DA conditions contributed a significant variance (approximately 5%) to concurrent performance on measures of word recognition (see Sternberg & Grigorenko, 2002, for a critical analysis of this finding). This earlier study was broaden to a larger sample ranging in ages 5–85 (Swanson, 1999) and that again found that DA procedures contributed approximately 5% of the variance reading and math performance. So what does this small amount of variance buy us? To answer that question we need to consider the logic of the testing conditions (also see Swanson, 2003, for a more detailed discussion). The initial testing condition reflects the baseline for each participant’s self-initiated processes to access information. The gain testing condition enhances the access to stored items by tailoring cues to help participants reinstate memory traces or to retrieve forgotten items from the initial (or baseline) conditions. Our previous studies (Swanson, 1992, 1999; Swanson, Ashbaker, & Lee, 1996) and the current study have shown that the gain conditions improve performance by as much as 1 standard deviation. We assume this occurs because the systematic cuing procedures emphasize sequential processing strategies and thereby reduces the number of competing strategies employed. We also assume that if the locus of WM problems in children with RD is in the retrieval phase, one would expect a reduction in residual difference between children with and without RD when compared to the initial testing condition. As shown in this study and our earlier studies we have been unable to reduce the residual differences between reading groups in the gain condition. Thus, one could argue that children with RD suffer WM deficits related to retrieval efficiency. One reservation in arguing that WM problems in children with RD are related to problems in retrieval efficiency (i.e., improved access to items previously forgotten), however, is that the manipulations between the initial (noncued) and gain (cued) condition are limited to individual differences as to what was actually attended to and stored by the child. Thus, the maintenance condition allowed for us to examine whether WM deficits reflect capacity constraints in accessing what has been previously stored. For this condition, the same WM tasks that matched each participant’s highest WM span level (gain score) are again administered, but without cues. Thus, each participant is presented items calibrated to their asymptotic level of WM performance. Calibrating this condition allows us to capture processing differences between groups beyond the simple learning of items. The inability to access items that have been previously learned (stored and retrieved) and have been calibrated to the individual span level we assume reflects fundamental constraints in WM capacity.


23

Based on the above logic, we consider the results that are related to the growth curve analysis. The conditional models showed that maintenance testing conditions moderated passage comprehension and receptive language, whereas gain testing conditions moderated nonword fluency. Consistent with the above logic, we assume that individual differences as a function of maintenance testing conditions reflects constraints in WM capacity. These constraints were not directly tied to simple retrieval strategies (e.g., feedback) or previous learning (e.g., item familiarity). In contrast, individual differences in gain conditions reflect limitations in the activation of new information. Thus, DA informs us to some degree that constraints in accessing previously remember information may play a role in some areas of reading (comprehension) and the efficiency in accessing new information in other areas of reading (nonword fluency). However, these results are qualified by the ability group DA interactions. The interaction with reading group gain testing condition when predicting passage comprehension suggests that a failure to activate new information in storage is an important determinant of reading skill. Further it appears from the reading group maintenance interaction that when predicting receptive vocabulary the ability to sustain performance after environmental (probes) support is removed maybe constrained in children with RD. Clearly, these conclusions are speculative on our part but the results do suggest that different processes may be activated as a function of DA when predicting performance on the criterion measures. 2. Do children with RD differ from children with more generalized achievement deficits as a function of dynamic assessment conditions? As expected, skilled readers outperformed children with RD on the WM measures. This occurred even though children were statistically matched on measures of fluid intelligence. Interestingly, no significant differences were found on measures of WM between children with RD who exhibited low verbal IQ from children with RD-only and RD þ MD. This finding is in contrast to other studies (e.g., van der Sluis, van der Leij, & de Jong, 2005) which have suggested that RD-only children exhibit no problems across domains that draw upon executive processing (such as updating as required by WM tasks, see Miyake, Friedman, Emerson, Witzki, & Howerter, 2000), whereas children who show reading plus arithmetic difficulties have problems in executive processing (domain general processing). We did not find support for the notion that more generalized WM deficits occur only in the comorbid group when compared to children with RD only. Further, because more generalized deficits in achievement (problems in both reading

24


and math) may be associated with verbal IQ, we also explored whether group differences in WM measures were merely an artifact of verbal ability. No significant differences were found between poor readers with low verbal IQ and children with RD-only on measures of WM. Thus, we did not find support for the notion that WM deficits in children with RD are merely an artifact of verbal IQ or comorbidity. Although the results indicated that ability group differences were larger on verbal WM tasks than visual-spatial WM tasks, performance among subgroups of children with RD was in the same low range. We now consider in more detail the findings related to the weak differentiation of subgroups as a function of DA and the domain general findings related to WM tasks.

Subgroups and Dynamic Assessment The present findings of this study partially support the previous work of Swanson and Howard (2005), which showed that that WM performance and DA procedures enhanced predictions of reading. The results of Swanson and Howard showed that poor readers and children with RDonly or MD þ RD generally perform in the same low range on verbal WM tasks. These results also coincide with others’ showing that poor readers and children with RD are difficult to separate on cognitive measures (e.g., Siegel, 1992; see Hoskyn & Swanson, 2000, for a review). The Swanson and Howard study showed children with RD and/or RD þ MD were more likely than the other subgroups to return to their initial score performance after presentation of probes (feedback) had been stopped. In contrast, the present study found that effect size scores related to gain and maintenance testing for all ability groups were in the same range. However, the present results did suggest that a drop in performance from the gain to maintenance conditions was more likely to occur for the RD-only group and the RD þ MD group was more likely influenced by initial testing conditions than the actual probing. No doubt, the differences between the previous study and the present study may be due to sampling. The earlier study focused on younger children of middle to low SES background. In contrast, the present sample was older, came from higher SES homes (in terms of parent education), more schooling, and therefore both cognitive and reading performance may have been more stable across the testing sessions. It is important to note that our inability to separate children with RDonly from other children with reading problems is not unique to this study.


25

For example, a meta-analysis comparing children with MD-only, RD-only, and RD þ MD on measures independent of the classification measures placed their performance on cognitive measures in the same range (Swanson & Jerman, 2006). For example, Swanson and Jerman found that the magnitude of ES across cognitive measures were miniscule between children with RD-only and MD-only (.10) and between MDonly and RDMD (M ES ¼ .26). More important, the means ES between RD-only and MD-only groups were .06 for STM of words, .03 for STM digits, .07 verbal WM, and .30 for visual-spatial WM. In general, these results suggested that MD and RD children share common memory deficits. Unraveling how the common memory deficits manifest themselves in different forms on achievement measures, of course, needs to be explored.

Limitations There are, of course, several limitations to our preliminary work. Four are apparent. First, we focused on children who came from average to high SES homes with highly educated parents. We assumed that previous studies that have tested children with RD had failed to control for SES (or focused on low SES participants) and therefore differences in reading in previous studies could be attributed to low SES and/or instructional factors. Regardless, our findings can only be generalized to a select group of children. Second, children with undiagnosed reading problems may yield different results. The children with RD in this study had the diagnosis of learning disabilities and the information from their cumulative file indicated a persistent reading problem in their elementary school years. This study sample was also unique in that all children had fluid intelligence scores (Raven test performance) in the normal range. If fluid intelligence scores were in the same low range as their reading scores, the results may have been different. Third, attrition in the sample occurred at wave 3. Although we did not find significant differences between the retained and nonretained samples across the various measures, the study was limited in terms of sample size when correlating wave 1 and wave 3 testing. Finally, these results need to be established by researchers independent of our research team. Although our DA measures are normed referenced (Swanson, 1995a), the majority of studies independent of our research team using the measures have been isolated to dissertations.

26


Summary In summary, DA procedures moderated later performance on measures of passage comprehension, nonword fluency, and receptive language. The results indicated that children with RD and skilled readers both benefited from DA procedures, but DA procedures did not statistically differentiate between the subgroups of children with RD. A descriptive analysis suggested that greater reductions occur between gain and maintenance testing conditions for the RD only group and the effect sizes related to DA conditions RD þ MD group are more likely related initial testing sensitivity than to procedures that test the limits. Clearly, additional research on the properties of DA and cognitive measures is needed, especially studies linking the malleability of WM to long-term instructional gains.

NOTE 1. Under most training circumstances with no ceiling or floor effects, pretest and posttest variable standard deviations are expected to be similar (see Carlson & Schmidt, 1999, p. 853; for a review). Carlson and Schmidt (1999) have argued that if exposure to the pretest acted as a learning event, then this would have caused posttest training scores to be higher, but responses to be more variable. Posttest dependent variable standard deviations would be systematically larger than pretest standard deviations, and as a result posttest standard deviations would result in smaller effect sizes than those calculated using pretest standard deviations. On the other hand, if changes in performance are a function of treatment (in this case feedback to the examinee) rather than merely retesting, then posttest means would be higher than pretest means, but posttest standard deviations would be comparable to pretest standard deviations (Carlson & Schmidt, 1999). In our case, we assumed that initial scores served as a pretest measure and gain and maintenance reflected posttesting.

ACKNOWLEDGMENTS The authors thank Marshal Raskin and Eleanor Higgins for their administrative assistance in testing children over the three- to five-year period. The authors also thank the various testers (Crystal Howard, Leilani Saez, Olga Jerman, Diane Luxenberg, and Diane Dowds) who as doctoral students played an important role in monitoring children’s progress. The technical aspects of this preliminary work are detailed for reading comprehension (Swanson, in press) and fluency (Swanson, 2010) elsewhere.


27

REFERENCES Brown, A. L., & Ferrara, R. A. (1999). Diagnosing zones of proximal development. In: P. Lloyd (Ed.), L.Vygotsky: Critical assessments: The zones of proximal development (Vol. III, pp. 225–256). New York: Routledge. Bryk, A., & Raudenbush, S. W. (2002). Hierarchical linear models (ed.). Newbury Park, CA: Sage. Caffrey, E., Fuchs, D., & Fuchs, L. S. (2008). The predictive validity of dynamic assessment: A review. The Journal of Special Education, 41, 254–270. Carlson, J. S., & Wiedl, K. H. (1979). Toward a differential testing approach: Testing-the-limits employing the Raven matrices. Intelligence, 3, 323–344. Carlson, K. D., & Schmidt, F. L. (1999). Impact of experimental design on effect size: Findings from the research literature on training. Journal of Applied Psychology, 84, 851–862. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum. Daneman, M., & Carpenter, P. A. (1980). Individual differences in working memory and reading. Journal of Verbal Learning and Verbal Behavior, 19, 450–466. De Beni, R., Palladino, P., Pazzaglia, F., & Cornoldi, C. (1998). Increases in intrusion errors and working memory deficit of poor comprehenders. Quarterly Journal of Experimental Psychology: Human Experimental Psychology, 51, 305–320. De Jong, P. (1998). Working memory deficits of reading disabled children. Journal of Experimental Child Psychology, 70, 75–95. Dunn, L., Dunn, L. M., Dunn, L. M., & Dunn, D. M. (1997). The peabody picture vocabulary test (3rd ed.). Circle Pines, MN: American Guidance Services, Inc. Embretson, S. E. (1992). Measuring and validating cognitive modifiability as an ability: A study in the spatial domain. Journal of Educational Measurement, 29, 25–50. Gathercole, S. E., Alloway, T. P., Willis, C., & Adams, A. (2006). Working memory in children with reading disabilities. Journal of Experimental Child Psychology, 93, 265–281. Gathercole, S. E., & Pickering, S. J. (2000a). Assessment of working memory in six- and sevenyear-old children. Journal of Educational Psychology, 92, 377–390. Gathercole, S. E., & Pickering, S. J. (2000b). Working memory deficits in children with low achievements in the national curriculum at 7 years of age. British Journal of Education Psychology, 70, 177–194. Grigorenko, E. L., & Sternberg, R. J. (1998). Dynamic testing. Psychological Bulletin, 124, 75–111. Hoskyn, M., & Swanson, H. L. (2000). Cognitive processing of low achievers and children with reading disabilities: A selective meta-analytic review of the published literature. School Psychology Review, 29, 102–119. Hwang, Y., Hosokawa, T., Swanson, H. L., Ishizaka, I., Kifune, N., Ohira, D., et al. (2006). A Japanese short form of the Swanson-Cognitive Processing Test to measure working memory: Reliability, validity, and differences in scores between primary school children of the United States and Japan. Psychological Reports, 99, 27–38. Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., & Howerter, A. (2000). The unity and diversity of executive functions and their contributions to complex frontal lobe tasks: A latent variable analysis. Cognitive Psychology, 41, 49–100. Pickering, S. J. (2006). Working memory in dyslexia. In: T. P. Alloway & S. E. Gathercole (Eds), Working memory and neurodevelopmental disorders (pp. 7–40). New York: Psychology Press.

28


Raven, J. C. (1976). Colored progressive matrices. London, England: H. K. Lewis & Co. Ltd. Rosenthal, R. (1994). Parametric measures of effect size. In: H. Cooper & L. V. Hedges (Eds), The handbook of research synthesis (pp. 232–260). New York: Russell Sage Foundation. SAS Institute Inc. (2003). SAS/STAT user’s guide. Cary, NC: SAS Institute. Siegel, L. S. (1992). An evaluation of the discrepancy definition if dyslexia. Journal of Learning Disabilities, 25, 618–629. Siegel, L. S., & Ryan, E. B. (1989). The development of working memory in normally achieving and subtypes of learning disabled. Child Development, 60, 973–980. Singer, J. D. (2002). Fitting individual growth models using SAS PROC MIXED. In: Moskowitz & S. Hersberger (Eds), Modeling intraindividual variability with repeated measures data: Methods and applications (pp. 135–170). Mahwah, NJ: Erlbaum. Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis. New York: Oxford University Press. Snijders, T. A., & Bosker, R. J. (1999). Multilevel analysis. Thousand Oaks, CA: Sage. Sternberg, R. J., & Grigorenko, E. L. (2002). Dynamic testing: The nature and measurement of learning potential. New York: Cambridge University Press. Swanson, H. L. (1992). Generality and modifiability of working memory among skilled and less skilled readers. Journal of Educational Psychology, 84, 473–488. Swanson, H. L. (1993). Working memory in learning disability subgroups. Journal of Experimental Child Psychology, 56, 87–114. Swanson, H. L. (1995a). S-cognitive processing test. Austin, TX: PRO-ED. Swanson, H. L. (1995b). Using the cognitive processing test to assess ability: Development of a dynamic assessment measure. School Psychology Review, 24, 672–693. Swanson, H. L. (1996). Individual and age-related differences in children’s working memory. Memory & Cognition, 24, 70–82. Swanson, H. L. (1999). What develops in working memory? A life span perspective. Developmental Psychology, 35, 986–1000. Swanson, H. L. (2003). Age-related differences in learning disabled and skilled readers’ working memory. Journal of Experimental Child Psychology, 85, 1–31. Swanson, H. L. (2008). Working memory and intelligence in children: What develops? Journal of Educational Psychology, 100, 581–602. Swanson, H. L. (2010). Does the dynamic testing of working memory predict growth in nonword fluency and vocabulary in children with reading disabilities. Journal of Cognitive Education and Psychology, 9, 51–77. Swanson, H. L. (in press). Dynamic testing, working memory and reading comprehension growth in children. Journal of Learning Disabilities. Swanson, H. L., Ashbaker, M., & Lee, C. (1996). Working-memory in learning disabled readers as a function of processing demands. Journal of Child Experimental Psychology, 61, 242–275. Swanson, H. L., & Howard, C. B. (2005). Children with reading disabilities: Does dynamic assessment help in the classification? Learning Disability Quarterly, 28, 17–34. Swanson, H. L., & Jerman, O. (2006). Math disabilities: A selective meta-analysis of the literature. Review of Educational Research, 76, 249–274. Swanson, H. L., Jerman, O., & Zheng, X. (2008). Growth in working memory and mathematical problem solving in children at risk and not at risk for serious math difficulties. Journal of Educational Psychology, 100, 343–379.

29


Swanson, H. L., & Lussier, C. M. (2001). A selective synthesis of the experimental literature on dynamic assessment. Review of Educational Research, 71, 321–363. Swanson, H. L., Saez, L., Gerber, M., & Leafstedt, J. (2004). Literacy and cognitive functioning in bilingual and nonbilingual children at or not at risk for reading disabilities. Journal of Educational Psychology, 96, 3–18. Swanson, H. L., & Siegel, L. (2001). Learning disabilities as a working memory deficit. Issues in Education: Contributions from Educational Psychology, 7, 1–48. van der Sluis, S., van der Leij, A., & de Jong, P. F. (2005). Working memory in Dutch children with reading- and arithmetic-related LD. Journal of Learning Disabilities, 38(3), 207–221. Wagner, R., & Torgesen, J. (1999). Test of word reading efficiency. Austin, TX: Pro-Ed. Wilkinson, G. S. (1993). The Wide Range Achievement Test-3. Wilmington, DE: Wide Range, Inc. Willcutt, E. G., Pennington, B. F., Olson, R. K., Chhabildas, N., & Hulslander, J. (2005). Neuropsychological analyses of comorbidity between reading disability and attention deficit hyperactivity disorder: In search of the common deficit. Developmental Neuropsychology, 27, 35–78. Woodcock, R. W. (1998). Woodcock reading mastery test – Revised. Minneapolis, MN: American Guidance.

APPENDIX. COMPOSITE SCORES (Z-SCORES) FOR WORKING MEMORY TASKS AS A FUNCTION OF TESTING CONDITIONS AND TESTING WAVES Variable

Wave 1 Vinitial Vgain Vmain Vprobe Visinitial Visgain Vismain Visprobe Wave 2 Vinitial Vgain

RD-Only

RD þ MD

Low VIQ Readers

Skilled Readers

Mean SD Mean SD

Mean

SD

Mean

SD

0.83 0.25 0.72 0.85 1.19 1.2 0.75 0.22 1.05 0.76 0.2 1.16 0.65 0.14 0.86 0.83 0.82 1.28 0.69 0.26 1.01 0.74 0.25 1.03

0.13 1.07 0.29 0.01 0.1 0.64 0.28 0.14

0.75 0.99 0.82 0.76 0.75 0.87 0.74 0.55

0.55 1.66 1.03 0.11 0.39 1.0 0.75 0.02

0.73 0.65 0.7 0.67 0.64 0.88 0.81 0.9

0.11 0.84 0.24 0.61 1.46 0.72 1.47 1.21

0.14 1.64

0.64 0.85

0.59 2.02

0.65 1.13

0.39 1.0 0.36 0.01 0.21 0.69 0.01 0.14

30


Variable

RD-Only

RD þ MD

Mean SD Mean SD 0.65 Vmain 0.11 Vprobe Visinitial 0.21 Visgain 0.95 0.76 Vismain Visprobe 0.05

1.13 0.46 1.28 0.61 0.02 0.84 0.69 0.07 1.14 0.9 0.75 1.47 0.93 0.65 1.34 0.65 0.18 0.72

Wave 3 Vinitial 0.26 0.6 0.34 1.74 0.55 1.62 Vgain Vmain 0.81 0.86 1.07 Vprobe 0.06 0.69 0.39 Visinitial 0.57 0.51 0.83 1.39 0.52 1.92 Visgain Vismain 1.15 0.61 1.62 Visprobe 0.04 0.85 0.23

0.66 0.61 0.62 0.7 0.77 0.63 0.7 0.58

Low VIQ Readers

Skilled Readers

Mean

SD

Mean

SD

0.72 0.22 0.39 1.08 0.73 0.25

0.79 0.75 0.83 1.09 1.11 0.67

1.08 0.28 0.68 1.55 1.23 0.12

0.7 0.8 0.76 0.9 0.89 0.67

0.13 1.53 0.79 0.45 0.53 1.73 1.33 0.08

0.74 0.83 0.92 0.73 0.88 1.08 1.15 0.92

0.78 2.37 1.87 0.37 0.77 1.7 1.44 0.35

0.96 0.95 1.07 0.79 0.53 0.86 0.7 0.68

Note: Vinitial, initial testing on verbal measures; Visinitial, initial testing on visual-spatial measures; Vgain, gain testing on verbal measures; Visgain, gain testing on visual-spatial measures; Vmain, maintenance testing on verbal measures; Vismain, maintenance testing on visual-spatial measures.

APPLICATIONS OF CURRICULUMBASED MEASURES IN MAKING DECISIONS WITH MULTIPLE REFERENCE POINTS Gerald Tindal and Joseph F. T. Nese ABSTRACT We write this chapter using a historical discourse, both in the chronology of research and in the development that has occurred over the years with curriculum-based measurement (CBM). More practically, however, we depict the chronology in terms of the sequence of decisions that educators make as they provide special services to students with disabilities. In the first part of the chapter, we begin with a pair of seminal documents that were written in the late 1970s to begin the story of CBM. In the second part of the chapter, we begin with the first decision an educator needs to make in providing special services and then we continue through the chronology of decisions to affect change in learning for individual students. In the end, we conclude with the need to integrate these decisions with multiple references for interpreting data: normative to allocate resources, criterion to diagnose skill deficits, and individual to evaluate instruction.


31

32

GERALD TINDAL AND JOSEPH F. T. NESE

EARLY VIEWS ON INSTRUCTIONAL EVALUATION In the late 1970s, Deno and Mirkin developed a concept they entitled databased program modification, espousing the need for measures that teachers could use to evaluate instruction over time (Deno & Mirkin, 1977). They emphasized specific features of these measures to allow generating alternate forms so that a time series data display could be used to determine if programs were working or needed to be adjusted. For example, the measures had to be easy to create, quick to administer, usable by all, from the curriculum, and with sufficient technical adequacy. In the end, they laid the groundwork for an experimental view of teaching in which any intervention was considered a hypothesis that needed vindication. At nearly the same time, Engelmann, Granzin, and Severson (1979) wrote about the need for diagnosing instruction. Their thesis was that problems in learning were seldom the fault of students; rather they often resulted from ineffective programs even though we spend an inordinate amount of time directly trying to diagnose the problem as the student’s. A central warrant of their thesis was that, while it is impossible to ascertain the certainty of problems ‘‘inside’’ the student, it is very much possible to determine programs are not working, change them, and only upon continued lack of success, make inferences about the problem being with the student. Both approaches relied upon the scientific method in which a null hypothesis (a notion that is not really believed with much certainty) is posited and countered by an alternative hypothesis. Rather than prove anything in a definitive manner, teachers needed to disprove that their actions are ineffective (posit a null hypothesis and then provide data to counter it). For Deno and Mirkin (1977), experimental teaching provided a time series design in which each student’s performance is compared to previous performance (in level, slope, and variation), with the impetus for instructional change residing in the lack of student change. For Engelmann et al. (1979), the null hypothesis is similarly situated with repeated failure to affect change as the only convincing evidence that supports the problem as being with the student (and not an instructional one). Ironically, they approached the problem from opposite ends of the continuum. For Deno and Mirkin (1977), instruction was tentative and ascertained only through measurement. For Engelmann et al. (1979), diagnosis was entirely instructional with measurement only implied. Yet, in an interesting confluence of history, both have become signatures of effective teaching that has current play in any ‘‘response to intervention’’ (RTI) model. In this chapter, we provide a brief history of this perspective

Applications of CBMs in Making Decisions

33

and then discuss the next generation of curriculum-based measurement (CBM) that features alignment to content standards and rigorous statistical modeling analytics. We review the empirical literature that has resulted from more widespread adoption of CBM, in both the kinds of problems being investigated as well as the variety of applications appearing in educational systems. We conclude with classroom and student case studies that echo these developments. CBM research initially focused on measurement validation, different sampling plans, and measurement designs. A number of studies also focused on highlighting the central feature of CBM as reflecting a ‘‘general outcome’’ measure rather than a curriculum-based one. Although efforts to change the language failed and CBM became the parlance in use, more recent efforts with the National Center on Response to Intervention (NCRTI) (www.rti4success.org) reflect a return to more precise language that references progress monitoring as oriented to either mastery monitoring or general outcomes. Mastery monitoring was initially defined by Deno and Mirkin (1977) in terms of the graphic display: ‘‘A progress (or mastery) graph is constructed to display the time it is taking a student to master a set (usually ordered over time in terms of sequence and/or complexity) of instructional objectives’’ (p. 33) with the y-axis reflecting units traversed and the x-axis reflecting time taken (usually days). The most significant feature is that the domain for measurement comes from a constricted sample of items (problems) being taught during the evaluation period. The classic example of mastery monitoring is the weekly spelling list in which 10–15 words are taught during the week, and then students receive a test on this list of words at the end of the week. Another feature of mastery monitoring is the graphic display, reflecting a series of step increases plotted against some definition of mastery; with mastery, performance moves up one unit for the next unit of time, and without mastery, performance is recorded at the same level for the next unit of time. Without incremental movement up in mastery, the graphic display of performance would simply hover at a steady level over time. General outcome measures were likewise described by Deno and Mirkin (1977) in terms of performance graphs: ‘‘designed to display how a student’s behavior changes on a single taskyover time’’ (p. 34) with the y-axis reflecting how well the student performed on the task and the x-axis reflecting time (in days or weeks). In this system, items (problems) are sampled across an alternate unit of instructional time, from the early through late skills that are being introduced over a year (or possibly over years). The sampling plan effectively includes items that preview what is to

34


be taught as well as reviews what has been taught. The graphic display allows raw scores to change without converting them to a common scale. The graphic display includes three metrics to evaluate programs: (a) slope of improvement, (b) variation among successive data values, and (c) immediate changes after a program has been introduced. By looking for increases in slope, decreases in variation, and large changes in level (from pre- to immediately postintervention), teachers can determine whether or not their programs are effective.

THREE DECADES OF RESEARCH ON CBM FROM THE 1980s THROUGH 2010 In one of the early definitions of CBM, Deno (1987) stated: the term curriculum-based assessment, generally refers to any approach that uses direct observation and recording of a student’s performance in the local school curriculum as a basis for gathering information to make instructional decisionsyAs noted earlier, the term curriculum-based measurement refers to a specific set of procedures created through a research and development program y and grew out of the Data-Based Program Modification system developed by Deno and Mirkin (1977) (p. 41).

CBMs, which sample skills related to the curriculum material covered in a given year of instruction, provide teachers with a snapshot of their students’ current level of performance as well as a mechanism for tracking the progress students make in gaining desired academic skills. Historically, CBMs typically are very brief individually administered measures (Deno, 2003). CBM is distinct in two important respects: (a) the procedures reflect technically adequate measures with reliability and validity present to a degree that equals or exceeds that of most achievement tests and (b) ‘‘growth is described by an increasing score on a standard, or constant, task. The most common application of CBM requires that a student’s performance in each curriculum area be measured on a single global task repeatedly across time’’ (Deno, 1987, p. 41). After the initial research conducted at the Institute for Research on Learning Disabilities (IRLD) at the University of Minnesota, scores of the technical reports became publications in the refereed literature; eventually, as the practice of CBM became more widespread, scores of chapters were written with two important books published with Shinn as the editor and most of the original researchers contributing chapters (1989, 1998). Much of this


35

research has been with and on the use of performance measures or general outcome measures and not the mastery measures.

Reading Research The history of research on CBM is rich and rather extensive, from the initial work in the late 1970s and early 1980s on through more recent research in the 2000s. Wallace, Espin, McMaster, and Deno (2007) summarize this research in a recent special series on curriculum-based measurement. Over the initial 20 years of research on reading CBM, a number of technical characteristics were investigated, including: 1. Content-related evidence addressing domain size (Fuchs, Tindal, & Deno, 1984) and difficulty of material (Fuchs & Deno, 1992; Shinn, Gleason, & Tindal, 1989). 2. Criterion-related evidence in the relation of oral reading fluency to reading comprehension (Shinn, Good, Knutson, Tilly, & Colllins, 1992) and predictions of performance (Fewster & Macmillan, 2002). 3. Construct validation with respect to rival hypotheses related to psychological variables (Kranzler, Brownell, & Miller, 1998), as well as other psychometric concerns (Fuchs, Fuchs, & Maxwell, 1988; Mehrens & Clarizio, 1993). 4. The effects of curriculum-based planning on grouping students (Wesson, Vierthaler, & Haubrich, 1989) and making instructional decisions (Allinder, 1996; Fuchs, Fuchs, & Stecker, 1989; Wesson, et al., 1988). As a useful formative assessment method, CBM has been extensively researched, a summary of which is beyond the scope of this chapter; these topics are simply illustrative of the kind of research that has been done over the past 30 years. Much of this research, however, has been conducted with elementary students and has focused on assessment of reading. With the establishment of CBM as a useful means of assessment for decision-making, we can apply its utility to mathematics, as it has been applied to reading.

Mathematics Research The research on mathematics did not begin as early as that done in reading and writing. Rather, work began after the initial research done with the IRLD with the best summary provided by Foegen, Jiban, and Deno (2007).

36


In this publication, the research reported from 32 studies is summarized in terms of three stages (Fuchs, 2004): (a) stage 1, with an emphasis on initial technical adequacy, particularly reliability and criterion validity evidence; (b) stage 2, with the focus on the technical characteristics of slope (change over time); and (c) stage 3, in which instructional utility of the measures is addressed. 1. The dominance of research is on elementary grade levels in which 17 studies provided evidence on the technical characteristics of the measures (stage 1) with monitoring basic skills progress measures (Fuchs, Hamlett, & Fuchs, 1998). The studies were split between progress measures (reflecting curriculum sampling within grades) and performance measures (reflecting robust measures across grades). The results confirmed many of the measures being developed as both reliable and with adequate relations to other measures of achievement. 2. In the early mathematics measures for students in prekindergarten to first grade, all of the studies were published since 2000 and most of them have relied upon performance measures. The focus has been on number identification, quantity discrimination, and missing numbers. 3. The least well-developed research in mathematics has been with middleschool grades. Calhoon (2008) has most aptly phrased the problem at the middle- and high-school level: ‘‘The needs for CBM assessments at the secondary level are limitless’’ (p. 238). Out of 578 articles, dissertations, and reports related to CBM, there have been only four published studies on math progress monitoring for middle school students (Foegen et al., 2007). Of these four, two investigated measures involving facts and estimation (Foegen, 2000; Foegen, et al., 2007), and two focused on measures that reflected a conceptual understanding of math (Helwig, Anderson, & Tindal, 2002; Helwig & Tindal, 2002). All four studies showed promising validity and/or reliability. The essential features of progress monitoring assessments have changed very little since the 1970s. Such measures are still expected to sample from the year’s worth of curriculum. They are meant to provide teachers with meaningful information about the progress students are making in mastering that material. In addition, to enhance their utility, progress monitoring measures are intended to be easy to administer, score, and interpret. However, whereas four decades ago the researchers deemed CBMs as not requiring any particular expertise to develop, the increasing stakes associated with assessment results as well as advances in psychometrics have significantly altered one perspective: We now recognize that the


37

creation of reliable and valid progress monitoring measures requires specialized knowledge beyond what most public school teachers possess. This realization spurred our creation of ‘‘next generation’’ elementary school reading CBMs, measures created using rigorous statistical modeling analytics previously the domain of large-scale assessments.

NEXT GENERATION CBM: ALIGNMENT TO STANDARDS, ITEM SCALING, AND TECHNICAL ADEQUACY Next-generation CBM needs to reflect the changing educational landscape with its emphasis on accountability. We developed easyCBM to reflect this high-stakes environment; in the process, we focused on the need to integrate the decisions teachers would make so that the same data system could be used across multiple decisions and at the same time be predictive of performance on a large-scale summative test. Rather than make a referral decision based on a single measure used only once for one decision (to locate students at risk), we developed screening measures that also would be related to formative progress measures. Not only would the measures be appropriate for all students so teachers could understand their differences, but they also would be useful for an individual student so teachers could understand what instructional programs were effective and make an individual difference (Deno, 1990). To accomplish this outcome and relate decisions using common measures, however, next-generation CBMs were needed that avoided three significant problems represented in the research just cited. The first problem was alignment to standards, the second was proper scaling, and the third and last was the documentation of technical adequacy with the most recent efforts being promulgated by the NCRTI.

Alignment to Standards In the most recent reauthorization of the Elementary and Secondary Education Act (ESEA), the federal government announced that a key priority in education is for all states to develop and adopt standards in reading and mathematics that build through high school (U.S. Department of Education Office of Planning Evaluation and Policy Development, 2010).

38


Academic standards stipulate and define what students must know and be able to do at each grade level. The U.S. Department of Education also pledged support for the development and use of assessments aligned to ‘‘standards in English language arts and mathematics that accurately measure student academic achievement and growth, provide feedback to support and improve teaching, and measure school success and progress’’ (U.S. Department of Education, 2010, p. 11). Thus, the call has been made in education for curriculum standards, and formative assessments aligned to those standards. Progress is meaningful when it is directed, and CBMs that are aligned with content standards may provide more useful information for teachers about student performance and progress. CBM aligned to standards can put the direction and rate of progress into context to best address individual student learning and make informed instructional decisions. Many states and several groups have made concerted efforts to establish curriculum standards, notably the Common Core State Standards Initiative, a state-led effort coordinated by the National Governors Association Center for Best Practices (NGA Center) and the Council of Chief State School Officers (CCSSO). Yet to date, there are no published research studies on the alignment of CBMs to content standards, and easyCBM is the only CBM system that has developed assessments to be aligned to national or state standards (Alonzo, Tindal, Ulmer, & Glasgow, 2006). The easyCBM mathematics system was developed based on the National Council of Teachers of Mathematics (NCTM) Curriculum Focal Points for Kindergarten through Grade 8 Mathematics (National Council of Teachers of Mathematics, 2006). The NCTM Focal Points outline the areas of mathematics instruction focus and objective for each grade level that have been adopted by many states as the basis of their state content standards in mathematics (e.g., Oregon). The easyCBM math items were written directly to the NCTM focal point standards; more specifically, an item was written to a particular objective within a single focal point standard. The results of an alignment study indicated that the easyCBM math items consistently and strongly aligned with the NCTM Focal Points standards (Nese et al., 2010). These results provide content validation of the easyCBM math assessment system, and also lend support to the feasibility and benefits of integrating standards into the process of CBM development. But as content standards become more prevalent and the technical properties of CBMs become more robust, the field needs to better study and understand the importance of alignment of CBM items and systems to content standards.


39

Alignment with content standards also allows CBMs and high-stakes state tests to be complementary. If CBMs and the state achievement test are both developed to be aligned to the same content standards, the predictive and criterion validity of the CBMs will be enhanced, effectively increasing teachers’ instructional decisions. But the purpose of the two tests are discrete, as high-stakes state tests aim to assess year-end content mastery, and CBMs aim to assess academic status and growth across the year. Because CBMs need to be sensitive to academic growth, their ceilings should not be parallel to the knowledge proficiency cut-point associated with state tests. CBMs need to offer a large sample of items with a range of difficulties in order to accurately measure student growth. Thus, an effective CBM system must develop items that are representative of the classroom or state curriculum standards, and also items that represent a wide range of difficulties.

Use of Item Response Theory for Scaling CBM In easyCBM, the potential exists to assist teachers in moving from individual differences to making an individual difference through analysis of item-level data in making diagnostic decisions about what to teach and when. As we noted in the opening review of Deno and Mirkin (1977) and Engelmann et al. (1979), instruction is the focus of any progress monitoring system. However, rather than simply wait for data to be compiled over time as instructional programs are delivered, we have designed a data system that allows teachers to target specific skills based on the data, using item response theory (IRT). IRT has become the theoretical foundation for most current test development given its theoretical measurement principles and its potential to address measurement difficulties (Embretson & Reise, 2000). Embretson and Reise (2000) outlined several advantageous psychometric assessment properties that IRT offers, including shorter, more reliable tests; equating different test forms; providing an unbiased estimates of model parameters that can be obtained from unrepresentative samples; obtaining scores to meaningfully compare students even when initial status scores differ; and providing a standard error of measurement (SEM) that can differ across scores but generalize across populations. This last property is particularly important because it concerns the quality of the test as well as the interpretation of individual student scores. As a consequence, it allows the SEM to vary across students such that the measurement error is smaller (and

40


thus scores are more reliable) for a specific group of students of interest (rather than equally broad for the entire distribution) (Embretson & Reise, 2000). Despite the psychometric advantages that IRT offers, it has not been widely adopted in the research on and development of CBM systems. However, easyCBM capitalizes on the properties of IRT to distribute items appropriately across forms so that each form is of equivalent difficulty. Form equivalency is paramount for a CBM system that purports to offer reliable estimates of academic growth, because it eliminates the test make-up as a reason for score change. easyCBM also uses IRT in developing assessments that provide maximum information for low-achieving students. That is, assessments are comparable and accessible for all students, but are particularly useful in progress monitoring for lower performing students with potential for poor learning outcomes, because these students are at the center of the decision-making process. IRT models the probability of endorsing an item correctly that depends upon a student’s ability and the properties of the item. IRT offers a robust diagnostic analysis of an item, including item difficulty, item discrimination (i.e., how the probability of answering an item correctly changes with student ability), and even a parameter to account for student guessing. easyCBM uses a specific IRT model, the Rasch model, which includes a parameter for individual student ability (e.g., math, vocabulary, or comprehension) and item difficulty and models the relation between the two. For example, as a student’s math ability increases, or as a math item’s difficulty decreases, the probability that a student will answer the item correctly increases. If the student’s math ability is equal to the difficulty of a math item, that student will have a 0.50 probability of answering that question correctly. This is an ideal scenario in test development because such an item affords the most information about a particular student. All of the IRT parameters are scaled in the same metric as individual student ability and thus can be used to immediately target (match) specific items with specific students. easyCBM uses IRT to better understand distributions of both individuals and items as a bridge in moving from individual differences to making an individual difference. Using IRT, all items are displayed on a logit scale that reflects the item difficulty. The advantage of this scale is that items from different measures can be placed on the same scale, allowing teachers to understand student performance relative to the different measures. In this scale a positive number represents a difficult item and a negative number represents an easy item. For example in Table 1, VQ20 (vocabulary question #20) is the most difficult item (with a scaled score of 3.00). In contrast, vocabulary question #1 is the easiest item


Table 1.

41

Item Measure (Difficulty) for Vocabulary and Comprehension Items.

Item

Measure

VQ20 CQ20 VQ7 VQ14 CQ11 VQ9 CQ12 CQ10 VQ24 CQ6 CQ9 CQ3 VQ15 VQ13 CQ14 VQ23 CQ7 VQ5 CQ18 CQ5 VQ18 VQ16 VQ21 CQ4 VQ4 VQ22 CQ19 VQ12 VQ19 CQ16 CQ15 VQ6 VQ8 VQ25 CQ17 VQ10 CQ13 VQ3 VQ11 VQ2 CQ8 VQ17

3.00 1.44 1.26 1.26 1.02 0.84 0.81 0.80 0.71 0.68 0.66 0.56 0.56 0.54 0.52 0.25 0.22 0.19 0.19 0.14 0.14 0.14 0.01 0.06 0.10 0.28 0.35 0.37 0.41 0.42 0.50 0.60 0.67 0.68 0.82 0.91 0.95 0.96 1.04 1.06 1.11 1.15

42


Table 1. (Continued ). Item

Measure

CQ2 CQ1 VQ1

1.16 1.18 1.64

Note: VQ represents vocabulary question, followed by the item number. CQ stands for comprehension question followed by the item number.

(with a scaled score of 1.64). Both vocabulary and (multiple-choice) comprehension items are distributed in between (CQ# stands for comprehension question item number). In all, 22 items are measured below zero (on the easy end of the scale) and 23 items are measured above zero (on the difficult end of the scale). See Table 1. Another advantage of this scale is that items and persons can be placed on the same scale. As in the table, the most difficult items are positive (at the top) and the easiest items are negative (at the bottom). The same group of items as displayed in Table 1 is now displayed in Fig. 1 (with VQ20 on the top right side and VQ1 on the bottom right side). In this figure, however, students are added on the left with each dot (.) representing a student and every seven students depicted as a ‘‘#’’. In this figure, one student is on the scale at each end with VQ20 and VQ1. In addition, M represents the mean, S represents one standard deviation out, and T represents two standard deviations out. Notice that M, S, and T are present for both the students (on the left) and items (on the right). In this particular figure, therefore, it is possible to determine how the students and items distribute themselves (on the same scale). An important conclusion is that the average for students (M on the left side) is higher than the mean of the items (M on the right side). In other words, students are slightly higher in ability than the items, which means that low students are sufficiently low and there should be few false negatives in any screening for determination of risk. Another way to say this is that plenty of items exist, even for the low students (see Fig. 1). The future of IRT for CBMs is as open as it is promising, with potential for improvement and advancement. IRT can offer much beyond scaling and accessing tests. IRT also paves the way for computer-based as well as computer-adaptive tests, which can shorten test length, testing time, and offer a more reliable estimate of ability for all students. This is particularly important in measuring growth within and across years, using CBM benchmark and progress monitoring assessments. Another promising

43

Applications of CBMs in Making Decisions <more> 6

.

5 .

4

3

2

1

0

–1

.# . . ## T . .##### # .##### . .######### .###### S .####### T .######### .###### .######## .########### .######### M ########### S .######### .######### .########## .####### .####### S M .######### .#### .#### .#### .# S .# .# T . . . T

VQ20

CQ20 VQ14

VQ7

CQ11 CQ10 CQ12 VQ9 CQ6 CQ9 VQ24 CQ14 CQ3 VQ13 VQ15 VQTOT VQ23 CQ18 CQ5 CQ7 VQ16 VQ5 CQ4 VQ21 VQ18 VQ4 CQ19 VQ12 VQ19 VQ22 CQ15 CQ16 VQ25 VQ6 VQ8 CQ17 VQ10 CQ13 VQ11 VQ2 VQ3 CQ1 CQ2 CQ8 VQ17

VQ1

–2

. –3

Fig. 1. Item–Person Map (1,358 Persons with 49 Items Measured and 1,198 Persons with 47 Items Measured). Note: Students are added on the left and items on the right. Each dot (.) represents a student and each ‘#’ represents seven students. M represents the mean, S represents one standard deviation out, and T represents two standard deviations out.

44


development lies in the use of distracter options. Typically, IRT allows test developers to analyze items and options in the development of a diagnostic assessment that provide more detailed information about a student’s knowledge and skills in a targeted area. An analysis of patterns of incorrect item responses, based on the construct of the distracters, also provides teachers with critical estimations of specific student misunderstandings. This feedback would offer practical implications and recommendations for both instruction and grouping, an important but missing part of most CBM within RTI.

Technical Adequacy and the National Center on Response to Intervention Criteria The documentation of the technical adequacy of CBM has been intermittently published by peer reviewed journals. These articles typically focus on alternate form, test–retest, or internal item reliability; the reliability of growth rates; benchmark standards for performance or growth; and predictive validity with criterion achievement measures. The NCRTI has created a clearing house for CBM systems by developing a review process of these systems, which has centralized and formalized the documentation of CBM technical adequacy. The mission of the NCRTI, funded by the Office of Special Education Programs (OSEP), is to build the capacity of states to help districts implement sound models for RTI. As such, NCRTI reviews CBM systems designed to identify students at risk for poor learning outcomes and monitor students’ academic progress, particularly in reading and math. The goal of the NCRTI review is to provide technical assistance to participating states, districts, and schools for successful implementation of the progress monitoring tools. Specific criteria are applied to the CBM systems, including different measures of reliability and validity and benchmark criteria for acceptable year-end academic performance and within- and across-year growth. The NCRTI review process exemplifies the increased technical and practical standards to which CBM systems are accountable. As an example, we can describe the validation work for items and forms of the easyCBM system. This work aims to satisfy the criteria set by the NCRTI (column headings in Table 2) for each measurement area of the system (row headings in Table 2). Separate analyses must be done by separate minority groups (e.g., racial minority and those receiving special education services) within each grade level K-8, to satisfy the 10 NCRTI

easyCBM

Tool

Math Reading, comprehension Reading, letter names Reading, letter sounds Passage reading fluency Reading, phone segmenting Word reading fluency

Area

Table 2.

Reliability of the Reliability Validity of the Predictive Alternate Sensitive to End-ofRates of Norms Performance-Level of the Performance-Level Validity of Forms Student Year Improvement Disaggregated Improvement Benchmarks Specified for Diverse Score Slope Score the Slope of Populations Improvement

Disaggregated Reliability and Validity Data

National Center for Response to Intervention Progress Monitoring Tools Chart.

Applications of CBMs in Making Decisions 45

46


criteria for each of the 7 easyCBM measurement areas. Thus, the validation work is expansive and systematic. To demonstrate that the 20 alternate forms of each easyCBM measure were of equal and controlled difficulty, IRT analyses (described earlier in the chapter) were used to present the distribution of item difficulty across forms. In addition, alternate form reliability studies were conducted and indicated, for example, that passage reading fluency alternate form reliability ranged from .87 to .97 (Alonzo, Mariano, & Nese, 2010). The reliability of the performance-level scores were reported through internal reliability analyses of the measures, split-half reliability analyses by full grade-level sample and by ethnic groups, and test–retest reliability. The reliability of the growth slope for each measure was derived from two-level, repeated measures growth analyses conditioned on initial status; results indicated high reliability for most measures (e.g., on average, reliability for the math assessments ranged from .41 to .61, word reading fluency from .62 to .77). Norm-referenced benchmarks for year-end performance and withinyear growth (i.e., the average slope of improvement in measure score) were specified across grade levels and across ethnic groups using both nationally and locally representative samples. These norms have direct and indirect implications for teachers and administrators. For example, the proportion of students correctly classified to meet or not-meet state math proficiency based on these benchmarks ranged from .76 to .88. In other words, these CBM measures can help classify students as at risk of failing the year-end state math test with a high degree of reliability and accuracy. Validity analyses were broad and diverse. The innovative alignment study detailed earlier in the chapter provided robust support for content validity (Nese et al., 2010). In support of construct validity, the easyCBM measures were found to correlate quite highly with each other (generally above .65) and with the state test (generally above .70). In addition, confirmatory factor analyses (CFA) provided support for the theoretical measurement models developed for reading and math assessments by demonstrating excellent fit statistics. Multiple regression of fall, winter, and spring benchmark scores on state achievement test scores demonstrated support for concurrent and predictive validity. Results indicated that the easyCBM spring assessments showed high concurrent validity of the state test scores, and the fall and winter assessments showed high predictive validity of the state test scores. In addition, the separate easyCBM measures each accounted for a moderately high amount of variance in the state reading and math scores. Lastly, predictive validity information for the slope of improvement (i.e., correlation between the slope and achievement outcome) was provided using two-level,


47

repeated measures growth analyses. The interpretation of this criterion in particular is tenuous, because a zero correlation between the two would indicate that all students demonstrated significant growth, regardless of where they began the year academically. Our analyses, however, conditioned on fall achievement, indicated a range of predictive validity estimates for improvement slopes (e.g., on average, passage reading fluency ranged from .10 to .38, and reading comprehension from .46 to .59). In brief, the technical properties of CBM systems must be robust, if only because implications are critical for districts, schools, teachers, and students. The results of a CBM system often are applied in decisions about K-8 student placement intervention and special education eligibility, classroom instruction, school progress, and school and district accountability. For CBMs to be used effectively in the decision-making process of a problem-solving model like RTI, the measures must be valid so that results are meaningful, and the system must offer teachers’ benchmarks for expected performance and growth so that results are useful. The NCRTI process to CBM systems evaluation and review is based on norm-referenced, risk-rating approach. That is, a student’s performance is defined by a comparison to peers’ performance, or individual differences, and scores are often interpreted as standing within a group, or risk of ‘‘failure.’’ We look now from individual student differences to making an individual difference for each student. Essential Data in Moving from Individual Differences to Making an Individual Difference The most significant issue in moving from individual differences to making an individual difference is the change in metric from noting where students exist in a group to helping them progress in their learning. Currently, three systems are available for organizing assignment of students to: (a) tiers of instruction in an RTI model or (b) specific groups for monitoring their progress, and (c) diagnostic error patterns with items. The dominant use is with the former practice, but all three uses shift the emphasis from what has been learned to what are students learning. Benchmarking and Risk within an RTI Model RTI is generally described as a multi-tiered prevention system, but the most common system used in practice and referenced in research consists of three tiers (Fuchs, Mock, Morgan, & Young, 2003). Tier 1 consists of the regular

48


curriculum and instruction delivered to all general education students; Tier 2 consists of supplemental instruction in an identified academic domain, delivered to students identified as not successfully responding to Tier 1 instruction; and Tier 3 consists of intensive instructional intervention delivered to individual students identified as not making satisfactory progress in Tier 2. A main principle of RTI, therefore, is the systematic assessment of students’ response to high quality, research-based instruction to identify students who would benefit most from the supplementary or intensive instruction of a higher tier. Benchmarking generally refers to the assessment process used to identify struggling students, often referred to as universal screening because the assessment is given to all students (including students in Tiers 2 and 3 as a safety check so students have the opportunity to move up or down through the support system). As part of a CBM system, these benchmark assessments are typically administered three times a year, during the fall, winter, and spring to ensure students continue to meet expectations throughout the year. Although the assessments can be of any academic domain, the most frequent assessments across grade levels include different measures of reading (e.g., fluency, comprehension, and vocabulary) and math (e.g., operations, algebra, and geometry). Along with the benchmark assessments, teachers need a complementary method to identify or classify students who may be at risk of low achievement and eligible for Tier 2 services. Most frequently, at-risk students are identified using a cut-point on a CBM benchmark assessment (Hintze & Silberglitt, 2005; Wayman, Wallace, Wiley, Ticha´, & Espin, 2007), and those students below the designated cut-point are candidates for Tier 2 instruction. Establishing these cut-points for risk is a critical matter, as they are used to direct service delivery. The methodological ideal would be for the CBM cutpoint to maximize the proportion of students predicted to be both below and above a cut-point on a criterion measure so that no classification errors are made. An inherent problem in the process of risk, however, is identifying the true status of a condition; asking the educational question: at risk of what? In the field of medicine, for example, the true status of a condition such as mumps can be easily confirmed with a simple blood test. But operationally defining ‘‘low achievement’’ or ‘‘poor learning outcomes’’ is a tenuous status. Within the current context of No Child Left Behind (NCLB) in which students are expected to make Adequate Yearly Progress (AYP) and schools and districts are held accountable, risk is often associated with proficiency on high-stakes state tests. Thus, the answer to the educational question has become: at risk of failing the state reading or math test.


49

Grouping for Monitoring Progress Benchmarking, however, does not need to be attached solely to a specific learning outcome on an external criterion, such as a state test, as the basis for assigning risk potential and assignment to an instructional tier. Districts and schools may also create cut-points to organize resources (financial and human) for monitoring progress. The data produced by CBM benchmarking can be used to create instructional groups based on performance or to assign specific teams of teachers for various instructional programs. For example, a school may target students who score below the 20th percentile on a CBM benchmark reading assessment for an academic intervention. Instructional grouping, whether homogeneous or heterogeneous in skill level, is a controversial topic in education. Proponents claim that it allows teachers to adapt learning outcomes, instructional activities, and pace to better meet the needs of students (Lou et al., 1996; Vaughn et al., 2003). In contrast, opponents claim that students may experience less focused instruction, lowered academic expectations, and less academic engagement (Elbaum, Schumm, & Vaughn, 1997). Although grouping has not been a standard practice, it has been sufficiently common to yield some results. Among a random sample of over 200 primary-grade teachers from across the United States, almost two-thirds of teachers reported using homogenous instruction groups, most reporting that they did so to best meet the needs of their students (Chorzempa & Graham, 2006). Some teachers who did not use homogenous groups reported that it was not allowed by their school or district, or that it was not compatible with their curriculum, but most teachers simply used heterogeneous groupings instead. Groups can be homogeneous or heterogeneous in skill level, depending on need, which may help with allocation of resources within the classroom, including personnel, intervention materials, and time. But homogeneous groupings can help teachers establish appropriate academic goals for each student by providing an appropriate comparative context made up of students of a similar skill level. Silberglitt and Hintze (2007) emphasized the importance of referencing students to the correct group by comparing student rates of within-year reading growth based on initial level of performance. Results indicated that the highest and lowest achieving student groups had significantly lower slopes than did the average student group, and consequently suggested that the academic growth of students may be misinterpreted if viewed in comparison to the growth of a different group.

50


USING EASYCBM TO ENACT THE ENTIRE DECISION-MAKING PROCESS We use real data from the easyCBM system to articulate the intricacies of benchmarking, risk rating, grouping, and progress monitoring (http:// easyCBM.coms). easyCBM is an online benchmark and progress monitoring assessment system, available for free teacher use, that includes multiple different reading and mathematics measures developed for students from K-8. The measures on the system were developed using IRT and represent the psychometric properties expected of modern technically adequate assessments: All items on all measures have good fit (Alonzo, Liu, & Tindal, 2007; Alonzo & Tindal, 2007) as determined by a one-parameter logistic Rasch model with mean square outfits within the acceptable range of 0.5–1.5. Items represent a range of difficulties. Finally, the forms are equivalent in difficulty within each grade. Screening (for Documenting Individual Differences) Many schools, teachers, and administrators use the easyCBM system as part of their RTI programs, and we present here a specific example of a student in such a school. Within this model, all students are administered math and reading CBM benchmark assessments during the fall, winter, and spring. The reading benchmarks include assessments of oral reading fluency (ORF) or passage reading fluency (PRF), vocabulary (VOC), and multiple-choice reading comprehension (MCRC). ORF/PRF is a one-minute test of reading accuracy and fluency in which the total number of correct words per minute (cwpm) is calculated by subtracting the number of incorrectly read words from the total number of words read. The ORF/PRF assessment is administered to each student individually. VOC is a test of students’ knowledge of words in which a student chooses a word’s synonym from three word options. The VOC assessment is group-administered by classroom teachers using an online computer administration format. The MCRC is test of reading comprehension in which a student reads a narrative fiction text and answers 20 questions about the text (literal, inferential, and evaluative). The MCRC assessment is also online and group-administered. A Class Roster When teachers begin the school year, all students are screened, including students in all tiers of instruction if the school is using an RTI model. The


51

purpose of the screener is to determine which students are at risk of failure in reading, using normative benchmarks in various grades to stratify students into three groups (a) strategic (dark gray), (b) intensive (medium gray), and (c) on track (light gray). Although districts are free to establish these groups as they want, most set the strategic group as students below the 10th percentile rank (PR), students between the 11th and 30th PR as intensive, and finally students on track as above the 30th PR. The color coding is applied to each measure and a cumulative risk rating applied overall in the following manner. First, performance on each measure is rated, with strategic students receiving 2 points, intensive students receiving 1 point, and students on track receiving 0 points. Second, the points are added and the following point awards are used to assign students to a final risk rating (though districts are also free to establish these cut-offs): (5–6 points are considered strategic, 3–4 points are considered intensive, and 0–2 points are considered on track). Each teacher receives a class roster with their students listed alphabetically and the option to sort by level of risk (which is depicted in Table 3). As can be seen in this table, the risk rating is applied in the fall and in the spring so teachers may see change in risk groups as well as overall status at the end of the year (blank cells represent students who have moved in during the year and having missing data; therefore they are coded according to the values present for the particular season) (see Table 3). A Case Study Student A began third grade with a benchmark PRF score of 58 cwpm, which was just below the 20th percentile (Fig. 2); a benchmark VOC score which was well below the 10th percentile (Fig. 3); and a benchmark MCRC score which was just above the 20th percentile (Fig. 4). Because this student scored near or below the 20th percentile in all three benchmark assessments, she was targeted for a Tier 2 intervention intended to address her specific academic needs, and placed in a group of students of reading with similar performance and skills for supplemental reading instruction. As a result of her easyCBM benchmark reading scores, beginning on September 15, Student A began an intervention labeled by the school, ‘‘Instructional Differentiation with Progress Monitoring.’’ This intervention is demarcated by a vertical intervention line in Figs. 2–4. In a strictly quantitative analyses of the data (based on a ROC analysis), Student A’s spring benchmark scores were above the cut score and she was predicted to pass the year-end state reading test; in fact, she did not pass this test. But her intervention reference group perhaps provided a more

5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5

Grade

F M M M M F F M F F M F M M F F F M M M F F M F F F F M F M M M M

Gender

50 50 90

50

50 90

50

50

Disab. Code

3 1 1 1 1 3 1 1 1 1 1 3 1 1 3 1 1 1 3 1 1 3 1 1 3 3 1 1 1 3 1 1 1

Ethnic

20 18 16 18 18 21 22

155 130 106 142 124 138 155

143 132 149 103 131 126

11 18 12 9 11 14

13 15 15

20 18 20 15 17

147 155 137 179 139

115 149 127

18 22 16 23 20 15 20

Fall Vocab.

159 184 184 210 118 162 168

Fall PRF

9 6 13 12 15 8

5 9 11

13 16 12 15 16 15 13

18 11 6 13 15

18 16 14 17 15 12 14

Fall MCRC

4 4 3 6 3 5

5 4 5

1 2 5 1 2 0 1

0 3 2 3 1

1 0 2 0 1 4 1

Fall Factor

H H S H S H

H H H

L S H L S L L

L S S S L

L L S L L H L

Fall Risk Category

185 192 186 181 168 178 199 199 185 183 160 207 145 215 179 146 130 175 105 168 170 178 155 152 157 141 157 147 195 138 120 160 134

Spring PRF

22 23 22 22 23 24 22 21 20 20 21 22 21 19 24 20 22 19 23 17 20 18 19 18 18 16 12 12 16 19 20 17 18

Spring Vocab.

16 16 16 16 16 16 18 15 17 18 15 14 16 17 15 19 16 14 17 17 14 15 14 15 13 15 10 15 13 13 14 8 13

Spring MCRC

0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 3 3 3 4 4 4 4 4 4 4 4

Spring Factor

Class Roster to Stratify Students into Three Groups based on Normative Reading Benchmarks.

L L L L L L L L L L L L L L L S S S S S S S S S S H H H H H H H H

Spring Risk Category

Note: Dark gray represents the strategic group, students below the 10th percentile rank (PR). Medium gray represents the intensive group, students between the 11th and 30th PR. Light gray represents the on track group, students above the 30th PR.

1 2 7 8 9 16 17 21 22 23 24 25 26 27 28 40 41 42 43 44 45 46 50 51 52 53 54 55 56 57 58 59 60

Last Name

Table 3. 52 GERALD TINDAL AND JOSEPH F. T. NESE


53

Fig. 2. Student A’s Third Grade easyCBM Passage Reading Fluency Scores. Note: The vertical intervention line represents the implementation of Student A’s Tier 2 intervention. The black diagonal line represents a rough estimate of Student A’s slope, or passage reading fluency growth across the year. easyCBMs graphs also include a dark gray line at the bottom representing students’ scores at the 10th percentile; a medium gray line above the lowest line reflecting scores at the 20th percentile; a line reflecting scores at the 50th percentile; and a light gray line at the top representing scores at the 90th percentile.

appropriate indicator of her yearly progress, and a more promising estimate of her learning for her teachers and parents than the state score might. Student A was correctly identified as needing a reading intervention at the beginning of the year; she was assessed using CBM and monitored to identify specific areas of need, and as a result she made practical gains in reading skills.

Progress Monitoring (for Documenting an Individual Difference) Figs. 2 and 4 show Student A’s academic progress across the school year in reading fluency and comprehension, respectively (no vocabulary progress

54


Fig. 3. Student A’s Third Grade easyCBM Vocabulary Scores. Note: easyCBMs graphs also include a dark gray line at the bottom representing students’ scores at the 10th percentile; a medium gray line above the lowest line reflecting scores at the 20th percentile; a line reflecting scores at the 50th percentile; and a light gray line at the top representing scores at the 90th percentile.

monitoring assessments were available to Student A during the school year). Her spring easyCBM reading benchmark scores (PRF ¼ 95, VOC ¼ 19, MCRC ¼ 18) suggest that she made progress throughout the year, although it is difficult to qualify this progress. In an ipsative approach, Student A’s academic gains during the year were of practical significance, but because she received a group intervention, the teacher or reading specialist could reference her growth with her peers and make a more meaningful interpretation of the data. This grouping method allows both Student A’s status and growth to be qualified to determine whether her response to the intervention is meeting expectations so that instruction can be continued, discontinued, or revised based on appropriate student references. Student A’s spring PRF was above the 25th percentile, and two progress monitoring tests after the spring benchmark showed fluency close to the 50th percentile (Fig. 2). Her scores on the MCRC progress monitoring assessments appeared to increase consistently across the year (Fig. 2), and a high spring


55

Fig. 4. Student A’s Third Grade easyCBM Multiple-Choice Reading Comprehension Scores. Note: The vertical intervention line represents the mplementation of Student A’s Tier 2 intervention. The black diagonal line represents a rough estimate of Student A’s slope, or passage reading fluency growth across the year. easyCBMs graphs also include a dark gray line at the bottom representing students’ scores at the 10th percentile; a medium gray line above the lowest line reflecting scores at the 20th percentile; a line reflecting scores at the 50th percentile; and a light gray line at the top representing scores at the 90th percentile.

comprehension score indicated that she was well above the 50th percentile in reading comprehension at year’s end. No vocabulary progress monitoring measures were administered to Student A throughout the year, so the data provided only a gain score and not a growth trajectory. Student A ended the school year with a score on the spring vocabulary benchmark that was close to the 10th percentile, indicating an area that perhaps needs further attention.

REFERENCES Allinder, R. (1996). When some is not better than none: Effects of differential implementation of curriculum-based measurement. Exceptional Children, 62, 525–535.

56


Alonzo, J., Liu, K., & Tindal, G. (2007). Examining the technical adequacy of reading comprehension measures in a progress monitoring assessment system. Technical Report # 41. Behavioral Research and Teaching, Eugene, OR. Alonzo, J., Mariano, G., & Nese, J. F. (2010). Reliability of the easyCBM reading assessments. Paper presented at the 18th annual meeting of the Pacific Coast Research Conference, Coronado, CA. Alonzo, J., & Tindal, G. (2007). Examining the technical adequacy of word reading and passage reading fluency measures in a progress monitoring assessment system. Technical Report # 40. Behavioral Research and Teaching, Eugene, OR. Alonzo, J., Tindal, G., Ulmer, K., & Glasgow, A. (2006). easyCBM online progress monitoring assessment system. Eugene, OR: Center for Educational Assessment Accountability. Available at http://easycbm.com Calhoon, M. (2008). Curriculum-based measurement for mathmatics at the high school level: What we do not know and what we need to know. Assessment for Effective Intervention, 33, 234–239. doi: 10.1177/15345084073/3488 Chorzempa, B. F., & Graham, S. (2006). Primary-grade teachers’ use of within-class ability grouping in reading. Journal of Educational Psychology, 98, 529–541. doi: 10.1037/ 0022-0663.98.3.529 Deno, S. (1987). Curriculum-based measurement. Teaching Exceptional Children, 20(1), 41–47. Deno, S. (1990). Individual differences and individual difference: The essential difference of special education. The Journal of Special Education, 24(2), 159–173. Deno, S. (2003). Developments in curriculum-based measurement. The Journal of Special Education, 37, 184–192. doi: 10.1177/0022669030370030801 Deno, S., & Mirkin, P. (1977). Data based program modification: A manual. Minneapolis, MN: Leadership Training Institute for Special Education. Elbaum, B. E., Schumm, J. S., & Vaughn, S. (1997). Urban middle elementary students’ perceptions of grouping formats for reading instruction. The Elementary School Journal, 97, 475–500. Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum Associates. Engelmann, S., Granzin, A., & Severson, H. (1979). Diagnosing instruction. The Journal of Special Education, 13(4), 355–363. Fewster, S., & Macmillan, P. (2002). School-based evidence for the validity of curriculum-based measurement of reading and writing. Remedial and Special Education, 23, 149–156. doi: 10.1177/07419325020230030301 Foegen, A. (2000). Technical adequacy of general outcome measures for middle-school mathematics. Assessment for Effective Intervention, 25, 175–203. doi: 10.1177/ 073724770002500301 Foegen, A., Jiban, C., & Deno, S. (2007). Progress monitoring measures in mathematics: A review of the literature. Journal of Special Education, 41, 121–139. doi: 10.1177/ 00224669070410020101 Fuchs, D., Mock, D., Morgan, P. L., & Young, C. L. (2003). Responsiveness-to-intervention: Definitions, evidence, and implications for the learning disabilities construct. Learning Disabilities Research & Practice, 18, 157–171. Fuchs, L. (2004). The past, present, and future of curriculum-based measurement research. School Psychology Review, 33, 188–192.


57

Fuchs, L., & Deno, S. (1992). Effects of curriculum within curriculum-based measurement. Exceptional Children, 58, 232–243. Fuchs, L., Fuchs, D., & Maxwell, L. (1988). The validity of informal reading comprehension measures. Remedial and Special Education, 9(2), 20–28. Fuchs, L., Fuchs, D., & Stecker, P. (1989). Effects of curriculum-based measurement on teachers’ instructional planning. Journal of Learning Disabilities, 22, 51–59. doi: 10.1177/ 002221948902200110 Fuchs, L., Hamlett, C. L., & Fuchs, D. (1998). Monitoring basic skills progress: Basic math contempts and applications [computer program manual]. Austin, TX: Pro Ed. Fuchs, L., Tindal, G., & Deno, S. (1984). Methodological issues in curriculum-based reading assessment. Diagnostique, 9, 191–207. doi: 10.1177/073724778400900401 Helwig, R., Anderson, L., & Tindal, G. (2002). Using a concept-grounded, curriculum-based measure in mathematics to predict statewide test scores for middle school students with LD. Journal of Special Education, 36, 102–112. doi: 10.1177/00224669020360020501 Helwig, R., & Tindal, G. (2002). Using general outcome measures in mathematics to measure adequate yearly progress as mandated by Title I. Assessment for Effective Intervention, 28, 9–18. doi: 10.1177/073724770202800102 Hintze, J., & Silberglitt, B. (2005). A longitudinal examination of the diagnostic accuracy and predictive validity of R-CBM and high stakes testing. School Psychology Review, 34, 372–386. Kranzler, J., Brownell, M., & Miller, M. (1998). The construct validity of curriculum-based measurement of reading: An empirical test of a plausible rival hypothesis. Journal of School Psychology, 36, 399–415. doi: 10.1016/S0022-4405(98)00018-1 Lou, Y., Abrami, P. C., Spence, J. C., Poulsen, C., Chambers, B., & d’Apollonia, S. (1996). Within-class grouping: A meta-analysis. Review of Educational Research, 66, 423–458. Mehrens, W., & Clarizio, H. (1993). Curriculum-based measurement: Conceptual and psychometric considerations. Psychology in the Schools, 30, 214–254. doi: 10.1002/ 1520-6807(199307)30:3o241::AID-PITS2310300307W3.0.CO;2-J National Council of Teachers of Mathematics. (2006). Curriculum focal points for prekindergarten through Grade 8 mathematics: A quest for coherence. Reston, VA: National Council of Teachers of Mathematics. Nese, J. F., Lai, C.F., Anderson, D., Park, B. J., Tindal, G., & Alonso, J., (2010). The alignment of easyCBM math measures to curriculum standards. Technical Report #1002. Behavioral Research and Teaching, Eugene, OR. Shinn, M. (1989). Curriculum-based measurement: Assessing special children. New York, NY: Guilford Press. Shinn, M. (1998). Advanced applications of curriculum-based measurement. New York, NY: Guilford Press. Shinn, M., Gleason, M., & Tindal, G. (1989). Varying the difficulty of testing materials: Implications for curriculum-based measurement. Journal of Special Education, 23, 223–233. doi: 10.1177/002246698902300208 Shinn, M., Good, R., Knutson, N., Tilly, W., & Colllins, V. (1992). Curriculum-based measurement of oral reading fluency: A confirmatory analysis of its relation to reading. School Psychology Review, 21, 459–479. Silberglitt, B., & Hintze, J. (2007). How much growth can we expect? A conditional analysis of R-CBM growth rates by level of performance. Exceptional Children, 74, 71–84.

58


U.S. Department of Education Office of Planning Evaluation and Policy Development. (2010). ESEA Blueprint for Reform. Washington, DC. Vaughn, S., Linan-Thompson, S., Kouzedanani, D., Bryant, P., Dickson, S., & Blazis, S. (2003). Reading instruction grouping for students with reading difficulties. Remedial and Special Education, 24, 301–315. Wallace, T., Espin, C., McMaster, K., & Deno, S. (2007). CBM progress monitoring within a standards-based system. The Journal of Special Education, 41, 66–67. doi: 10.1177/ 00224669070410020201 Wayman, M., Wallace, T., Wiley, H., Ticha´, R., & Espin, C. (2007). Literature synthesis on curriculum-based measurement in reading. Journal of Special Education, 41, 85–120. doi: 10.1177/00224669070410020401 Wesson, C., Deno, S., Mirkin, P., Sevcik, B., Skiba, R., King, R., et al. (1988). A casual analysis of the relationships among on-going measurement and evaluation, the structure of instruction, and student achievement. The Journal of Special Education, 22, 330–343. doi: 10.1177/002246698802200304 Wesson, C., Vierthaler, J., & Haubrich, P. (1989). The discriminative validity of curriculumbased measures for establishing reading groups. Reading Research and Instruction, 29(1), 23–32.

EVOLVING STANDARDS OF DIAGNOSTIC ACCURACY IN PREDICTING AND AVOIDING ACADEMIC FAILURE Amanda M. VanDerHeyden ABSTRACT Since its first appearance in policy, the diagnostic construct of learning disability (LD) has struggled to demonstrate its validity. Any diagnostic construct requires the operationalization of a ‘‘true positive’’ diagnosis to permit strong analyses of diagnostic accuracy and associated intervention outcomes. Because there is no ‘‘true positive’’ definition for LD, diagnostic accuracy and outcome research are disparate and difficult to translate into meaningful actions for diagnosticians and intervention teams in schools. In this chapter, a new framework of decision making centered on consequential validity, evidence-based education, and shared decision making is proposed for evaluating the relative costs and benefits of alternative actions when making decisions about whether to conduct assessment or intervention, and what assessments or interventions to implement.


59

60

AMANDA M. VANDERHEYDEN

EVOLVING STANDARDS OF DIAGNOSTIC ACCURACY IN ASSESSMENT OF LEARNING DISABILITY Some allege that the learning disabilities (LD) construct was born from a desire to explain poor achievement in the absence of readily understood causes in ways that were not (1) stigmatizing to the student and (2) could lead to remediation if only the child were diagnosed and then provided with the ‘‘right’’ instruction (Sleeter, 1986; Zigmond, 1993). Early and strong popular endorsement of LD as a valid diagnostic construct was grounded in what turned out to be the undue belief that diagnosis of LD would lead to specialized interventions that would produce normalized performance for struggling learners (Gresham & Witt, 1997; Kavale & Forness, 1999). Early research primarily came from an advocacy position that the LD construct was valid and subsequent research was designed to provide rich descriptions of individuals diagnosed with LD rather than questioning the discriminant, predictive, and consequential evidence for the LD diagnosis in the first place. The IQ-achievement discrepancy definition of LD was codified in law and regulatory provisions in 1977 by PL 94-142 (the precursor to Individuals with Disabilities in Education Act) and almost immediately received criticism and questions even as the rate of LD diagnosis proliferated in schools (Algozzine, Ysseldyke, & Christenson, 1983; Ysseldyke, Vanderwood, & Shriner, 1997).

OVERRELIANCE ON CORRELATION: ABSENCE OF CONVERGING AND DIVERGING EVIDENCE The LD construct, operationalized as achievement that differs from achievement that is expected based on measures of cognitive processing, has suffered from a lack of convergent and discriminant validity evidence. Valid diagnosis requires that application of diagnostic criteria reliably result in a homogeneous grouping of individuals who share the diagnosed condition and that these individuals differ from others who do not meet diagnostic criteria and do not share the diagnosed condition (Campbell & Fiske, 1959). Consider for a moment the diagnostic criteria for breast cancer. Individuals with breast cancer will share certain symptoms. Some of these symptoms are useful for diagnostic decision making and some are not. Useful symptoms must meet convergent and discriminant validity

Predicting and Avoiding Academic Failure

61

standards. For example, a symptom like fatigue may be a symptom that is shared by most individuals with breast cancer. Fatigue as a diagnostic marker for breast cancer does contribute to homogeneous grouping of individuals with breast cancer because most individuals with breast cancer experience fatigue. A converging pattern is observed: fatigue is positively correlated with breast cancer. The problem is that absence of fatigue does not discriminate well those individuals who do not have breast cancer. In other words, people who do not have breast cancer also commonly experience fatigue. Hence, a diverging pattern is not present: not having fatigue is not correlated with not having breast cancer. Therefore, fatigue has little value as a diagnostic criterion for breast cancer. Useful diagnostic criteria are those measurable symptoms that are shared by individuals with the diagnostic condition (converging evidence) and not shared by those without the diagnostic condition (diverging evidence). To validate the diagnostic construct of LD, it is necessary to demonstrate that students with LD can be readily discriminated from similarly lowperforming students without LD on the basis of proposed measures of the construct. Lack of discriminant validity evidence has been a notable challenge to reliable and valid diagnosis of LD with compelling data demonstrating that errors of classification are prevalent when trying to distinguish among subgroups of low-performing students (Macmann, Barnett, Lombard, Kocher-Belton, & Sharpe, 1989); that measures of IQ are not meaningful in predicting who will successfully respond to intervention and who will not (Lennon & Slesinski, 1999; Vellutino, Scanlon, & Lyon, 2000; Vellutino et al., 1996; Vellutino, Scanlon, & Tanzman, 1998; Vellutino, Scanlon, & Zhang, 2007); and that IQachievement discrepant readers and IQ-achievement consistent readers have not been found to differ systematically on a continuum of measures including phonological awareness, rapid naming, verbal short-term memory, and vocabulary lexical skills (Stuebing et al., 2002). IQ-achievement difference scores (i.e., discrepancies) are unreliable and do not differ predictably between typically achieving learners and low-achieving learners (Kratochwill, Clements, & Kalymon, 2007). Various discrepancy criteria do not result in the same diagnostic decision for the same student with a given set of test scores and different tests may result in different results for the same student given the same discrepancy criterion (Fuchs, Fuchs, & Compton, 2004; Kavale & Flanagan, 2007). These data have led many researchers to conclude that cognitive processing measures are not capable of discriminating students with LD from so-called garden-variety low achievers, especially in the area of reading. Further, specific instructional

62


techniques have not been demonstrated to convey unique benefit to students with LD versus low-performing students without LD (Vellutino et al., 1996, 1998, 2000, 2007) and in practice, special instructional techniques have not been associated with special education services provided to students with LD (Ysseldyke & Christenson, 1987; Ysseldyke, Thurlow, Christenson, & Weiss, 1987; Ysseldyke, O’Sullivan, Thurlow, & Christenson, 1989). The diagnostic challenges associated with figuring out who really has LD and who is ‘‘just’’ a struggling learner led researchers and frontline diagnosticians to adjust diagnostic procedures across settings and time, creating a moving diagnostic target. These diagnostic realities have contributed to a climate where practitioners wish to find help for struggling students, view special education as a way to obtain help, view diagnosis as a necessary means to obtain help through special education, and given the flexibility in the current diagnostic criteria can make a diagnosis of LD with little accountability if that diagnosis is made in error. In fact, advocates have prioritized overdiagnosis of LD in the name of protecting against false negative identification errors. Given the lack of consequential validity evidence for the diagnosis of LD, overdiagnosis of LD is an error of consequence that should be reconsidered by parents, advocates, and school systems.

CONSEQUENTIAL VALIDITY AND THE LD CONSTRUCT AS A POSSIBLE RED HERRING The idea that educational services provided to students should lead to appreciably positive changes in their everyday lives is consonant with the ideals of education and psychology (Wolf, 1978; Hayes, Nelson, & Jarrett, 1987). In 1995, Messick expanded the notion of construct validity to include not just the reliability and validity of scores obtained on measures but also the intended and unintended social consequences of the use of the scores to reach decisions. Academic failure and diagnosis of LD have been associated with high poverty rates and ethnic and gender disproportionality (Donovan & Cross, 2002). When an individual carries a higher probability of receiving a diagnosis of LD because he is male, of a minority ethnicity, or receives free and reduced lunch, the equity of diagnostic procedures must be questioned. Is the diagnosis presumed to legitimately affect children differently based on ethnic, poverty, or gender status? Are the criteria biased in some way? If the condition is presumed to disproportionately affect individuals, is there


63

demonstrated benefit that comes from making the diagnosis and does this benefit outweigh potential for harm? Given that specialized interventions are not available on the basis of diagnosis and outcomes have not been shown to be improved through services in special education (Kavale & Forness, 1999), the consequential validity of the LD diagnosis is empirically unsupported (Barnett, Macmann, & Carey, 1992). If consequential validity is a foundation of our practice, then perhaps focusing on whether or not a child truly does or truly does not have LD is a waste of resources. Perhaps asking whether or not a student really has or really does not have LD is the wrong question and will continue to lead us down fruitless paths of increasingly complicated diagnostic criteria designed to rule out every possible imagined cause of poor performance. Such a path will be counterproductive because increasingly cumbersome rule-out procedures are not likely to be implemented with fidelity in schools and there will always remain the possibility that there is some cause of poor performance that we failed to imagine and assess and rule out when making the diagnosis.

THE DIAGNOSTIC CONUNDRUM OF LD: WHAT IS A TRUE POSITIVE? IS LOW ACHIEVEMENT EVER REALLY UNEXPECTED? Kavale and Flanagan (2007) argued that LD is unexpected underachievement and then described a method for determining when underachievement is indeed unexpected. The problem with their argument is that underachievement is a relative term that is determined on the basis of what is expected, which also must be determined from some measure of student potential for achievement. Kavale and Flanagan state, ‘‘A non-significant discrepancy may be found simply because the cognitive abilities/processes that are responsible for low achievement have attenuated the total test score y If those specific abilities/processes could be removed from the total test score and in so doing a significant discrepancy emerged, then this finding would suggest underachievement. In short, while the finding of a non-significant discrepancy may rule out underachievement in some cases, it does not rule out underachievement in all cases’’ (p. 133). These comments underscore the impossibility of diagnosing LD in a stable, replicable, and meaningful way because the diagnostic criteria as described by Kavale and

64


Flanagan are fluid and are determined in part by the battery of measures to which the child happens to be exposed. At the heart of the problem here is the fact that there has yet to be an operationalized definition of a ‘‘true positive’’ for LD. A ‘‘true positive’’ is a person who truly has LD. Because we have no incontrovertible index, no way of knowing if someone really has or really does not have LD, defining a true positive for LD requires articulating what measures with what scores indicate that a person truly has LD and therein lies the problem. Diagnostic accuracy studies evaluating the utility of various assessment procedures, cutscores, and decision rules require a definition of a true positive and true negative. For example, to conduct diagnostic accuracy studies, researchers will categorize students into one of four cells on a grid. The column headings are criterion positive and criterion negative (these are often referred to as the ‘‘gold standard’’ criteria). The row headings are test positive and test negative. Each child can be sorted into one of the four cells based on his or her scores on test measures and whether or not he or she meets established, agreed-upon criteria for being a true positive (first column) or true negative (second column). Once children are sorted into cells, then classification agreement analyses (e.g., sensitivity, specificity) are possible. Defining a true positive is the first step, otherwise, studies of various symptoms and cutscores will be disparate and chase a moving target. The classification agreement analyses published in the literature over the last 10 years illustrate well the lack of a definition of a true positive for LD. These studies have commonly used outcomes like frequently appearing in the risk range due to low level of performance and slow growth relative to peers (Speece, Case, & Molloy, 2003), failing to respond to short-term intervention (VanDerHeyden, Witt, & Naquin, 2003), or performing below some benchmark on a criterion measure in reading (Hintze, Ryan, & Stoner, 2003). Although these studies represented an important step forward in beginning to quantify the utility of various screening criteria for identifying a need for intervention, from a broader perspective, it seems illogical that frequent debate has centered on the details associated with measuring the presence of LD when we have not yet even defined what LD is in a way that has uniform support. The trouble with LD as a diagnostic construct is that its primary symptom, poor academic achievement, is neither a unique nor specific marker for the presence of LD. So, just like fatigue in the example earlier in this chapter, poor academic achievement, even when it is unexpected, is not uniquely and specifically associated with the presence of LD. Children may demonstrate lower than expected achievement for a


65

number of reasons that have little to do with having LD. For example, children may demonstrate low achievement because they do not like a task and are not highly motivated to give their best performances. Instruction may not have adequately met the needs of a given student. A student may have failed to master the prerequisite skills required to be successful on a given task. Therefore, diagnosis of LD necessitates a series of rule-out decisions. Diagnosticians must rule out alternative explanations of potential causes of poor achievement. After attempting to rule out all possible alternative causes of low achievement, the diagnostician concludes that the child has some condition called LD by default (there are no other tenable explanations to explain low performance and so the child must have a within-child problem called LD). One fundamental assumption in the paradigm of a rule-out diagnosis of LD is that a child who does not attain expected benchmarks in the face of environmental supports to assure high motivation and powerful intervention is a child who truly has LD. As a researcher and practitioner, this gives me qualms. In my scientific framework, there is no flawed learner, but rather there are only flawed interventions. I define a failed RtI as a situation in which we have not yet found the right intervention. I am not ready to conclude that the intervention has successfully ruled-in the presence of some condition we call LD. Again, failed response to intervention has to be a unique and specific marker for the presence of LD and until we can define what that is, we cannot make that leap. Second, interventions fail for many reasons, the most common of which involves poor implementation integrity (Wickstrom, Jones, LaFleur, & Witt, 1998; Witt, Noell, LaFleur, & Mortenson, 1997). Much of the foregoing argument is simply not relevant to the political context in which we operate where diagnosis is required to provide special education services. However, the field could make a paradigm shift as has happened over the years, providing intervention to those students and evaluating the effects of those interventions. Whether we call these students LD or not matters little to me. What we do with those students is what matters and certainly we can do better than what we have done. LD is a diagnosis of exclusion and diagnoses of exclusion are particularly susceptible to error (Hunt & Walker, 1971). Where there is no ‘‘true positive’’ definition and instead diagnosis depends on ruling out hypothesized alternative causes, there is always the possibility that the diagnostician has failed to identify a potential causal variable and/or failed to correctly measure one of the rule-out criteria. Because a ‘‘true positive’’ for LD is

66


unmeasurable, tenable diagnostic accuracy studies are impossible. As a result of discordant definitions of LD, the research base for LD rightfully could be described as disparate. This state of affairs is not just a problem for our science, it is a problem of major consequence ethically and socially. Understanding and operationalizing the construct of LD is fraught with competing contingencies among advocates, policymakers, researchers, practitioners, parents, and students. Fuchs, Fuchs, and Stecker (2010) contextualize RtI as a framework at the center of a host of competing agendas held by various stakeholders. Fuchs and colleagues seem to characterize groups of researchers and organizations as being ‘‘for’’ or ‘‘against’’ special education and then conclude by suggesting that everyone is missing the point. Indeed, it is possible that for all the improvements in using data to filter students into levels of intervention, that we have no tenable data suggesting that those students who have always struggled have shown much improvement as a result of these efforts.

WHAT IS A TRUE POSITIVE IN RTI? Fuchs et al. (2010) assert that there are students who are poorly served in education (general and special) and that these are largely the same students who have always been poorly served (Fuchs, Fuchs, Mathes, & Lipsey, 2000). As for how RtI intersects with this group of low-achieving students, I believe that these children are disproportionately unresponsive to intervention relative to their at-risk peers. On the other hand, their lack of responsiveness is not diagnostic for the presence of LD. Lack of responsiveness might function as a specific marker for ruling in the presence of LD, but only so long as the gold standard in that scenario is lack of responsiveness to X or Y intervention. Reaching broader conclusions about the presence or absence of disability based on failed RtI requires operationalization of a true positive and dealing with the substantial threat that the ‘‘right’’ intervention was not implemented or implemented well. Otherwise, diagnosing LD is akin to proving the negative, which simply cannot be done. With an unreliable or unstable set of diagnostic criteria and a resulting disparate research base, it is no wonder to me that the most vulnerable of students have not been well served by our efforts to reform the systems and rules governing support for struggling students. Selecting measurable outcomes like reduced risk status, improved academic performance, improved equity, accelerated rates of learning, give us, in effect, a set of ‘‘true positive’’ conditions that we can define, aim for,


67

and evaluate. Defining true positive outcomes allows us to focus on attaining the outcomes that matter, and avoid the vulnerabilities associated with making error-prone discriminations between students who fundamentally exhibit the same symptoms, that is, trouble learning. Thus far, efforts to diagnose and serve students with LD have not demonstrated appreciably positive effects (Kavale & Forness, 1999) and so to move toward evidencebased practice in education, we must begin to define the outcomes we care about, align our efforts with those outcomes, and measure progress toward those outcomes, at both the system and the individual student level.

WHAT IS EVIDENCE-BASED PRACTICE AND WHY DOES IT MATTER IN EDUCATION? The integration of science into frontline clinical decision making first occurred in medicine (Sackett, Rosenberg, Gray, Haynes, & Richardson, 1996). Evidence-based medicine gave diagnosticians a framework for quantifying the utility of certain symptoms and diagnostic tests to rule-in or rule-out diagnostic conditions and identifying treatments that were likely to yield the greatest benefit to the patient with the least risk of aversive, painful, or dangerous side effects. Evidence-based medicine was important because it established science as the ‘‘best’’ basis for action and implied a decision system that seeks to consider the cost relative to the benefits at each stage of decision making (initial screening, follow-up assessment, treatment evaluation). It is not a surprise that the framework of evidence-based practice resonated with educators because it shared many similarities to data-based decision-making models that were emerging in schools (Deno & Mirkin, 1977; Marston, 1989). At the core of evidence-based practice models is an emphasis on the consequential or treatment validity of assessments and diagnoses. The validity of an assessment is irrevocably tied to the yield it brings in measurable improvements for a person exposed to that assessment. This measurable yield also considers the cost associated with giving the assessment or making the diagnosis that includes tangible and intangible costs of decision errors and the cost of the assessment itself. Evidence-based practice offers a framework for evaluating not whether an assessment is useful, but rather how much more useful that assessment is compared to available alternatives which include not doing the assessment at all. Importantly, evidence-based practice gives implementers a way to avoid

68


the errors associated with mistaking correlation alone for strong evidence or simple accuracy for correct decision making (Meehl & Rosen, 1955). The question is not whether performance on a given measure correlates with presence of a condition we wish to diagnose, but rather the measurable extent to which use of the assessment leads to a diagnostic conclusion that increases the odds of positive outcomes for the person exposed to the assessment (VanDerHeyden, 2010). This question allows us to focus on the ingredients that matter in a diagnostic framework. Educational decision makers have neglected to consider how prevalence interacts with assessment accuracy in mathematically predictable ways (VanDerHeyden, 2010). Educators must begin to examine whether each additional assessment provides an incremental benefit in decision-making accuracy that makes a measurable difference in the outcome attained by the student. Implicit in such a framework is the notion of efficiency. It is possible that strong universal prevention strategies will outweigh any benefit conveyed by additional assessment for the vast majority of students. Obviously, student performance data collected at routine intervals will be the fuel that drives the car of data-based decision making and multitiered intervention in schools. Bushell and Baer (1994 [2008]) described two worlds in education: One is the world in which we choose everything we want to teach from the universe of measurable goals, and then find the teaching techniques to maximize those measures. In the other world, we choose only how we will teach, as if there were an a priori correct way; whatever we get by teaching that way then must be what we want or should want. It is especially easy to believe that, when what we want is not measurable – or at least not measured. Perhaps the second world is so consistently preferred over the first because the first is doubly embarrassing: It requires us to decide what we want to teach, and it is always embarrassing to discover that in some small part we don’t know and in some small part we don’t agree. And it exposes us to a series of surprises as we discover what techniques teach the goals better than others. (p. 266)

Bushell and Bear recognized the folly of instructing without regard to the learning that either happens or does not happen. A theme in educational reform over the last three decades has been that instructional design cannot be separated from its result (i.e., learning). It is the learning, the quality of the learning, the rate of learning, and the robustness of the learning that must guide the instruction that follows. Consequently, frequent student assessment has become a cornerstone of effective teaching practice (National Council of Teachers of Mathematics, 2000; National Mathematics Advisory Panel, 2008; National Reading Panel, 2000). Yeh (2007) conducted a very compelling analysis of the return for various investments


69

that could be made to enhance learning in schools. In his analysis he computed effect sizes on student achievement for the following conditions: frequent student assessment (2–5 times per week), 10% increase in per pupil spending, voucher programs, charter schools, and increased accountability. He then computed the cost for each approach. Frequent student assessment was 4 times as effective as increased spending per pupil, 6 times as effective as vouchers, 64 times more effective than charter schools, and 6 times more effective than increased accountability even after accounting for the increased costs associated with conducting frequent student assessment. These findings are consonant with familiar data demonstrating the effects of frequent progress monitoring and decision making on student achievement (Fuchs, Fuchs, Hamlett, & Stecker, 1991; Fuchs, Fuchs, Mathes, & Simmons, 1997).

IMPLEMENTATION IS COMPLICATED AND FAILS MORE THAN IT SUCCEEDS Perhaps the most enduring implications of the evidence-based practice movement in education will be the challenges associated with implementation. Implementation failures are not new phenomena. Implementations have failed as long as there have been new recommended programs to try (Fixsen & Blase´, 1993). In education, the framework in science has been to develop and evaluate programs separate from the contexts in which they will be installed or implemented. This framework has resulted in data that are well-controlled but all too often demonstrate no significant effect or have little chance of producing similar effects in the real world (Cook, Tankersley, & Landrum, 2009). In a prophetic article in 1978, Montrose Wolf reflected on implementation failures related to his work developing effective treatment programs for youth exhibiting disruptive behaviors. Wolf commented, ‘‘y the first time we tried to replicate the Achievement Place program in another community, that community gave us feedback in a most drastic manner. Before we really knew they had complaints about our program, they had ‘fired’ us’’ (p. 206). And later in the article, y we were told by many communities that one of the most important characteristics of teaching-parents was ‘‘warmth.’’ When quizzed about ‘‘warmth,’’ the community members indicated that they wanted teaching-parents who ‘‘know how to relate to youths.’’ For some time, our response to this request was to disagree with them. We argued, ‘‘What you really need is someone who knows how to give and take away points at the right time.’’ (p. 207)

70


Wolf concluded, ‘‘A new consensus seems to be developing. It seems that if we aspire to social importance, then we must develop systems that allow our consumers to provide us feedback about how our applications relate to their values, to their reinforcers’’ (p. 213). Wolf recognized then that implementation failures were events that were worthy of analysis, that causes of implementation failures could be identified. His suggestion was to expand the outcomes of behavior-change efforts to include measurement of consumer satisfaction, and this recommendation is pertinent in today’s climate of educational accountability and school system reform. Implementation is a behavior-change process that requires matching of environmental demands with scientific know-how in ways that implementers will have the skill and the motivation to implement (Neef, 1995). Implementation requires, therefore, close and continual contact with the relevant outcome data that should specifically link to the outcomes that are important to or are prized by the system. That is, implementation requires initial equipment, training, and installment, but these conditions are not sufficient for full implementation. Full implementation is incremental and requires constant tending of actual implementation and the effects to adjust the methods used, build environmental capacity to implement, and ensure that those responsible for the implementation experience positive results from their perspective so that implementation will be reinforced and will sustain (Fixsen & Blase´, 1993). There is a long legacy in education of failed innovations. One of the most salient examples of a failed innovation involves direct instruction (Carnine, 2000). In a compelling commentary on what it takes to get evidence-based practices implemented in classrooms, Carnine reiterates the need for analysis and implementation to maintain direct and frequent contact with the student data outcomes that matter. Effective practices will not be adopted if dissemination decisions are politicized and researchers, policy makers, and frontline implementers have not agreed on which outcomes matter most and should be used to guide implementation. Carnine’s commentary highlights the damage of philosophy-driven educational practices at a direct and detrimental cost to datadriven practices where effects on student learning outcomes are the prioritized effects. Carnine writes, ‘‘The best way for a profession to ensure its continued autonomy is to adopt methods that ensure the safety and efficacy of its practices’’ (p. 10). So the key question will remain, once the field has agreed on outcomes related to optimal student learning and satisfaction as the driving force for specialized interventions in schools, how to get those interventions effectively implemented in classrooms. There has been an ever-present


71

tendency in education to focus on intervention selection and to neglect intervention management. This tendency is directly responsible for the majority of intervention failures. Finding an intervention that should work to improve a particular learning problem is not terribly challenging. Getting that intervention implemented in the classroom is substantially more challenging. Research on intervention integrity allows us to reach the following conclusions: (1) intervention failure should be a rare event, but in many settings it is not, (2) most interventions fail because direct measurement of implementation accuracy is not collected or implementation failures are not responded to by adults in the intervention setting. Interventions fail because there is no direct and frequent measurement of both implementation and implementation effects with systematic adjustments to improve effectiveness (Fixsen & Blase´, 1993). There is not enough attention paid to what outcomes merit priority and consensus for how these outcomes will be measured.

WHAT ABOUT SHARED DECISION MAKING IN EDUCATION? GAMBLING AND PROSTRATE SCREENING MAY PAVE THE WAY In medicine, a great deal of attention has been paid to involving patients in medical decision making about what assessments to order and what treatments to provide. Shared decision making has been framed as a way to increase patient buy in with the idea that if patients have a shared role in the decisions, compliance with prescribed treatments will be greater and patients (and systems) will experience superior outcomes. Like many promising initiatives, shared decision making has been more difficult to accomplish in practice. Each decision about an assessment to order carries with it implications related to the accuracy of the test (sensitivity and specificity) in the context in which it is ordered (higher- or lower-risk setting for the condition) and given certain measured symptoms of the patient (symptoms may increase likelihood of a true positive or true negative) and the costs associated with ordering the test (what happens to the patient if there is a false positive decision error? what happens to the patient if there is a false negative decision error?). For highly trained diagnosticians, the systematic influence on decision error that arises from predictable sources like prevalence of a condition in a given setting (e.g., a high-risk setting implies a higher rate of true positive cases in that setting) is frequently, even

72


typically, underestimated and poorly understood (Meehl & Rosen, 1955). Hoffman, Wilkes, Day, Bell, and Higa (2006) developed a decision-making aid that could be used by decision makers to determine relative risk when reaching decisions about what assessments to conduct and what interventions to begin. Their decision aid begins with two side-by-side pie charts (see Fig. 1) that illustrate the probability of a patient (a) doing well (i.e., having no symptoms), (b) experiencing negative side effects (e.g., incontinence), or (c) dying if the 65-year-old patient elects not to have the prostrate screening (chart on left) and if the patient chooses to have the prostrate screening (chart on right). Visually, one can see right away, that the chances of dying are greater if one elects not to have the screening, but only slightly so. Because the test is associated with a possible side effect of incontinence, one can also see that having the test increases chances of becoming incontinent. To make the concept of probability more accessible to decision makers, the same probabilities were used to build two dartboards as in Fig. 2 where a decision maker could choose which dartboard he or she would choose to throw a dart at (i.e., not having the screening on the left or having the screening on the right). Finally, because dartboard games do include more than just random chance (i.e., there is arguably some skill involved in hitting a mark with a dart), the aid was adjusted to represent a roulette wheel as in Fig. 3. Here a decision maker can look at the graph and decide based on probabilities which test to order or not order. This kind of probabilistic thinking is absent in decisions about screening and intervention in schools.

No PSA Screening Asymptomatic

Annual PSA Screening

Risk of incontinence and/or impotence

Death from Prostate Cancer

Fig. 1. Probability of Positive Outcomes (Light Gray), Negative Side Effects (Dark Gray), and Death (White) for a 65-Year Old Patient Who Chooses not to Undergo Screening (On Left) and Who Chooses to Undergo Screening (On Right). Reprinted with Permission from Hoffman et al. (2006).

73



Fig. 2.



Same Probabilities as in Figure 2, Displayed On a Dart Board. Reprinted with Permission from Hoffman et al. (2006).


Outcome:

Fig. 3.





Outcome:

Same Probabilities as in Figure 2, Displayed On a Roulette Wheel. Reprinted with Permission from Hoffman et al. (2006).

In education, there has been an a priori position that more intervention is better even when the data tell us otherwise (Kavale & Forness, 1999). The aid becomes even more useful when one considers the relative risks and benefits of potential treatments. In the example provided by Hoffman and colleagues, knowing that a disease has 40% mortality that can be cut in half with treatment but the treatment carries with it minor side effects in 10% of

74


No Treatment Asymptomatic

With Treatment

Risk of Side Effects

Death from Disease

Fig. 4. Hypothetical Probability of Positive Outcomes (Light Gray), Negative Side Effects (Dark Gray), and Death (White) Given No Treatment (On Left) and Treatment (On Right) for Prostate Cancer. Odds Favor Choosing to Have Treatment. Reprinted with Permission from Hoffman et al. (2006).

No Treatment Asymptomatic

With Treatment

Risk of Side Effects

Death from Disease

Fig. 5. Hypothetical Probabilities for Various Outcomes When Choosing Not to Have Treatment (On Left) and Choosing to Have Treatment (On Right). In This Example, Treatment is Not The Obviously Favored Option. Decision Makers Will Have to Evaluate the Relative Costs of Negative Side Effects Versus the Odds of Death in Electing to Have the Treatment. Reprinted with Permission from Hoffman et al. (2006).

patients, decision makers can readily see that treatment ought to be the favored option based on probability (see Fig. 4). However, doubling the side effects of treatment as in Fig. 5 makes the decision more complicated and


75

requires a judgment about how ‘‘bad’’ the negative side effects are before determining that treatment is worthwhile (and better than dying) because the chances favor that with treatment or without the combined risk of death and/or side effects is the same. In education, we are rarely talking about death. Education’s version of death might be academic failure and dropout from school. Similarly, assessment decisions do carry risks. For example, early reading screening carries substantial false positive error risks (Hintze et al., 2003). Savvy decision makers should question what are the consequences of not screening for reading compared to the consequences of screening for reading? My guess is that empirical evidence would support the decision to screen because of the reduced risk of failing to learn to read if intervention is provided. Choosing one treatment over another, however, complicates the decision. Now decision makers need to understand what are the costs associated with participating in treatment (i.e., loss of instructional time in the classroom, segregation from classroom peers) and the probability of treatment success or failure (child learns to read successfully, child does not learn to read successfully) and make judgments accordingly. For screening, data are available to guide decision makers in knowing whether the benefits outweigh the risks, although they are not typically explained with great precision in this way. In determining whether or not to begin treatment and which treatment to begin, however, the data are much less well developed. Researchers and practitioners should measure student outcomes, measure implementation of interventions designed to improve outcomes, and quantify the effects and choices for consumers in ways that convey the relative risks and benefits of each assessment and treatment option.

REFERENCES Algozzine, B., Ysseldyke, J. E., & Christenson, S. (1983). An analysis of the incidence of special class placement: The masses are burgeoning. The Journal of Special Education, 17, 141–147. Barnett, D. W., Macmann, G. M., & Carey, K. T. (1992). Early intervention and the assessment of developmental skills: Challenges and directions. Topics in Early Childhood Special Education, 12, 21–43. Bushell, D., Jr., & Baer, D. M. (1994 [2008]). Measurably superior instruction means close, continual contact with the relevant outcome data. Revolutionary! In: R. Detrich, R. Keyworth & J. States (Eds), Advances in evidence-based education: A roadmap to evidence-based education (pp. 265–275). Oakland, CA: The Wing Institute. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105. Carnine, D. (2000). Why education experts resist effective practices (and what it would take to make education more like medicine). Available at http://www.edexcellence.net/foundation/ publication/publication.cfm?id ¼ 46. Retrieved on March 16, 2004.

76


Cook, B. G., Tankersley, M., & Landrum, T. J. (2009). Determining evidence-based practices in special education. Exceptional Children, 75, 365–383. Deno, S. L., & Mirkin, P. K. (1977). Data-based program modification: A manual. Reston, VA: Council for Exceptional Children. Donovan, M. S., & Cross, C. T. (2002). Minority students in special and gifted education. Washington, DC: National Academy Press. Fixsen, D. L., & Blase´, K. A. (1993). Creating new realities: Program development and dissemination. Journal of Applied Behavior Analysis, 26, 597–615. Fuchs, D., Fuchs, L. S., & Compton, D. L. (2004). Identifying reading disabilities by responsiveness to instruction: Specifying measures and criteria. Learning Disability Quarterly, 27, 216–228. Fuchs, D., Fuchs, L. S., Mathes, P. G., & Lipsey, M. W. (2000). Reading differences between low-achieving students with and without learning disabilities: A meta-analysis. In: R. Gersten, E. P. Schiller & S. Vaughn (Eds), Contemporary special education research (pp. 81–104). Mahwah, NJ: Erlbaum. Fuchs, D., Fuchs, L. S., Mathes, P. G., & Simmons, D. C. (1997). Peer-assisted learning strategies: Making classrooms more responsive to diversity. American Educational Research Journal, 34, 174–206. Fuchs, D., Fuchs, L. S., & Stecker, P. M. (2010). The ‘‘blurring’’ of special education in a new continuum of general education placements and services. Exceptional Children, 76, 301–323. Fuchs, L. S., Fuchs, D., Hamlett, C. L., & Stecker, P. M. (1991). Effects of curriculum-based measurement and consultation on teacher planning and student achievement in mathematics operations. American Educational Research Journal, 28, 617–641. Gresham, F. M., & Witt, J. C. (1997). Utility of intelligence tests for treatment planning, classification, and placement decisions: Recent empirical findings and future directions. School Psychology Quarterly, 12, 249–267. Hayes, S. C., Nelson, R. O., & Jarrett, R. B. (1987). The treatment utility of assessment: A functional approach to evaluating assessment quality. American Psychologist, 42, 963–974. Hintze, J. M., Ryan, A. L., & Stoner, G. (2003). Concurrent validity and diagnostic accuracy of the dynamic indicators of basic early literacy skills and the comprehensive test of phonological processing. School Psychology Review, 32, 541–556. Hoffman, J. R., Wilkes, M. S., Day, F. C., Bell, D. S., & Higa, J. K. (2006). The roulette wheel: An aid to informed decision making. PLoS Med, 3(6), e137. Hunt, W. A., & Walker, R. E. (1971). Cue utilization in diagnostic judgment. Journal of Clinical Psychology, 62–64. Kavale, K. A., & Flanagan, D. P. (2007). Ability-achievement discrepancy, response to intervention, and assessment of cognitive abilities/processes in specific learning disability identification: Toward a contemporary operational definition. In: S. Jimerson, M. Burns & A. VanDerHeyden (Eds), Handbook of response to intervention: The science and practice of assessment and intervention (pp. 130–147). New York: Springer. Kavale, K. A., & Forness, S. R. (1999). Effectiveness of special education. In: C. R. Reynolds & T. B. Gutkin (Eds), The handbook of school psychology (3rd ed., pp. 984–1024). New York: Wiley. Kratochwill, T. R., Clements, M. A., & Kalymon, K. M. (2007). Response to intervention: Conceptual and methodological issues in implementation. In: S. Jimerson, M. Burns & A. VanDerHeyden (Eds), Handbook of response to intervention: The science and practice of assessment and intervention (pp. 25–52). New York: Springer.


77

Lennon, J. E., & Slesinski, C. (1999). Early intervention in reading: Results of a screening and intervention program for kindergarten students. School Psychology Review, 28, 353–364. Macmann, G. M., Barnett, D. W., Lombard, T. J., Kocher-Belton, E., & Sharpe, M. N. (1989). On the actuarial classification of children: Fundamental studies of classification agreement. The Journal of Special Education, 23, 127–149. Marston, D. B. (1989). A curriculum-based measurement approach to assessing academic performance: What it is and why do it. In: M. R. Shinn (Ed.), Curriculum-based measurement: Assessing special children (pp. 18–78). New York: Guilford Press. Meehl, P. E., & Rosen, A. (1955). Antecedent probability and the efficiency of psychometric signs, patterns, or cutting scores. Psychological Bulletin, 52, 194–215. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749. National Council of Teachers of Mathematics. (2000). Principles and standards for school mathematics. Reston, VA: Author. National Mathematics Advisory Panel. (2008). Foundations for success: The final report of the national mathematics advisory panel. Washington, DC: U.S. Department of Education. National Reading Panel. (2000). Report of the national reading panel. Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction. NIH Publication no. 00-4769. Washington, DC: U.S. Government Printing Office. Neef, N. A. (1995). Research on training trainers in program implementation: An introduction and future directions. Journal of Applied Behavior Analysis, 28, 297–299. Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (Eds). (1996). Editorial: Evidence based medicine: What it is and what it isn’t. British Medical Journal, 312, 71–72. PMid:3527718. Sleeter, C. E. (1986). Learning disabilities: The social construction of a special education category. Exceptional Children, 53, 46–54. Speece, D. L., Case, L. P., & Molloy, D. E. (2003). Responsiveness to general education instruction as the first gate to learning disabilities identification. Learning Disabilities Research and Practice, 18, 147–156. doi:10.1111/1540-5826.00071 Stuebing, K., Fletcher, J., LeDoux, J., Lyon, G. R., Shaywitz, S., & Shaywitz, B. (2002). Validity of IQ-discrepancy classifications of reading disabilities: A meta-analysis. American Educational Research Journal, 39, 469–518. VanDerHeyden, A. M. (2010). Determining early mathematical risk: Ideas for extending the research. School Psychology Review, 39, 196–202. VanDerHeyden, A. M. (2010). Technical adequacy of response to intervention decisions. Exceptional Children, 77, 335–350. VanDerHeyden, A. M., Witt, J. C., & Naquin, G. (2003). Development and validation of a process for screening referrals to special education. School Psychology Review, 32, 204– 227PMid:15505962. Vellutino, F., Scanlon, D., Sipay, E., Small, S., Pratt, A., Chen, R., & Denckla, M. (1996). Cognitive profiles of difficult-to-remediate and readily remediated poor readers: Early intervention as a vehicle for distinguishing between cognitive and experiential deficits as basic causes of specific reading disability. Journal of Educational Psychology, 88, 601–638.

78


Vellutino, F. R., Scanlon, D. M., & Lyon, G. R. (2000). Differentiating between difficult-toremediate and readily remediated poor readers: More evidence against the IQachievement discrepancy definition of reading disability. Journal of Learning Disabilities, 33, 223–238. Vellutino, F. R., Scanlon, D. M., & Tanzman, M. S. (1998). The case for early intervention in diagnosing specific reading disability. Journal of School Psychology, 36, 367–397. Vellutino, F. R., Scanlon, D. M., & Zhang, H. (2007). Identifying reading disability based on response to intervention: Evidence from early intervention research. In: S. R. Jimerson, M. K. Burns & A. M. VanDerHeyden (Eds), Handbook of response to intervention: The science and practice of assessment and intervention (pp. 185–211). New York: Springer. Wickstrom, K. F., Jones, K. M., LaFleur, L. H., & Witt, J. C. (1998). An analysis of treatment integrity in school-based behavioral consultation. School Psychology Quarterly, 13, 141–154. Witt, J. C., Noell, G. H., LaFleur, L. H., & Mortenson, B. P. (1997). Teacher use of interventions in general education settings: Measurement and analysis of the independent variable. Journal of Applied Behavior Analysis, 30, 693–696. Wolf, M. M. (1978). Social validity: The case for subjective measurement or how applied behavior analysis is finding its heart. Journal of Applied Behavior Analysis, 11, 203–214. Yeh, S. S. (2007). The cost effectiveness of five policies for improving student achievement. Journal of Evaluation, 28, 416–436. Ysseldyke, J. E., & Christenson, S. L. (1987). Evaluating students’ instructional environments. Remedial and Special Education, 8, 17–24. Ysseldyke, J. E., O’Sullivan, P. J., Thurlow, M. L., & Christenson, S. L. (1989). Qualitative differences in reading and math instruction received by handicapped students. Remedial and Special Education, 10, 14–20. Ysseldyke, J. E., Thurlow, M. L., Christenson, S. L., & Weiss, J. (1987). Time allocated to instruction of mentally retarded, learning disabled, emotionally disturbed, and nonhandicapped elementary students. The Journal of Special Education, 21, 43–55. Ysseldyke, J. E., Vanderwood, M. L., & Shriner, J. (1997). Changes over the past decade in special education referral to placement probability: An incredibly reliable practice. Diagnostique, 23, 193–201. Zigmond, N. (1993). Learning disabilities from an educational perspective. In: G. R. Lyon, D. B. Gray, J. F. Kavanaugh & N. A. Krasnegor (Eds), Better understanding learning disabilities: New views from research and their implications for education and public policies (pp. 27–56). Baltimore: Brookes.

HOW TO USE PSYCHOLOGICAL TESTS FOR FUNCTIONAL DIAGNOSIS: THE CASE OF ASSESSMENT OF LEARNING DISABILITIES Giulia Balboni and Roberto Cubelli ABSTRACT Very often the diagnostic process provides only a descriptive label of the disorder. A useful diagnosis must reach the precise description and the functional interpretation of the deficits shown by a given individual. To this aim, four separate but intertwined steps are required: (1) clinical interview to collect personal and clinical anamnesis and to figure out the sociocultural context; (2) screening tests to describe the general cognitive picture and to detect the impaired functions; (3) standardized batteries to categorize the specific cognitive disorders according to the accredited taxonomies; (4) ad hoc investigation to identify the impaired cognitive components in the individual patient according to a sound theoretical model of the functional architecture of the cognitive processes. Only doing in this way, personalized educational and rehabilitative interventions may be planned and specific goals can be achieved.


79

80

GIULIA BALBONI AND ROBERTO CUBELLI

The term ‘‘diagnosis’’ means ‘‘discerning, distinguishing’’ and derives from the Greek word diagignoskein, which literally signifies ‘‘to know thoroughly’’ (from dia- ‘‘apart’’ and gignoskein ‘‘to learn’’). It follows that the diagnostic process is aimed to the recognition of a disease or condition and the interpretation of the underlying causes by its outward signs and symptoms. A useful diagnosis allows achieving the precise description and the functional interpretation of the behavioral or cognitive disorders of a given individual, i.e., it is intended to identify the impaired processes and mechanisms that may explain the observed pathological pattern of performance (e.g., Ellis & Young, 1988). In this way, when possible, personalized educational and rehabilitative interventions may be planned and specific goals for ameliorating everyday life can be achieved. Very often the diagnostic process consists of only the assignment of a descriptive label of the disorder, thus confirming what is already known without providing any useful information on the damaged components of the cognitive system. For example, in the case of a student with reading difficulties, it may happen that the diagnosis is limited to categorize the apparent disorder according to the available classification systems, like DSMIV-TR or ICD-10, based on the reduced levels of speed and accuracy. In this way, the existence of a problem in reading a written text is ascertained but no useful information is provided to teachers or parents who usually are aware of the reading problem and decide to consult professionals and experts to deal with this critical disorder. In clinical settings, the emphasis given to the objective property of psychological tests and the availability of normative scores has promoted the idea that tests are self-sufficient tools. In other words, arbitrary psychometric measures, like the number of the standard deviations below the mean obtained by the normative sample or the score assumed as pathological cut-off, are often considered sufficient to formulate a cognitive diagnosis and conclude a neuropsychological assessment. In some cases, in adjunct of the DSM classification, more sophisticated diagnosis allows identifying subtypes of a specific disorder. In the case of dyslexia, e.g., the pattern of performance in different tasks and the qualitative analysis of errors, based upon the dual route model (Coltheart & Rastle, 1994; Coltheart, Rastle, Perry, Langdon, & Ziegler, 2001) can lead to distinguishing between phonological and surface dyslexia. In the former, novel words are read less accurately than known words with mainly segmental errors; in the latter, regular words are read better than irregular words, with regularization errors and difficulties in segmenting letter strings. However, categorizing the behavioral pattern is not sufficient since the same clinical label can be applied to individuals showing impairments at different

Psychological Tests for Functional Diagnosis

81

cognitive levels (for different forms of developmental surface dyslexia see Friedmann & Lukov, 2008). The diagnostic process must go beyond any very broad clinical category and must lead, for each individual patient, to the identification of the specifically impaired cognitive processes. Actually, to obtain a useful diagnosis, and not just a label, a deeper assessment of the individual history and the overall pattern of abilities and disabilities is needed. Using tests without investigating the personal and clinical history impedes to evaluate all factors associated to the disorder (e.g., the social-cultural level or the medical interventions) that can have a role in the genesis of the disorder or can interact with its manifestation. Categorizing without attempting any interpretation of the origin of the difficulties does not allow the identification of the cognitive components specifically impaired and therefore prevents the possibility to plan an effective and personalized intervention. In order to achieve a functional interpretation of the disorder for all individual patients, four separate but intertwined steps are needed, each of them requiring different instruments (see Cubelli & Della Sala, in press): (1) clinical interview to collect personal and clinical anamnesis and to grasp the sociocultural context; (2) screening tests to describe the general cognitive picture and to detect the impaired functions; (3) standardized batteries to categorize the specific disorders according to the shared taxonomies; (4) ad hoc investigation to identify the impaired cognitive components according to a sound theoretical model of the processes investigated. Each stage is crucial to obtain a correct and useful functional diagnosis; the entire process must be always completed.

STEPS OF THE FUNCTIONAL DIAGNOSIS Clinical Interview The first step should provide information concerning the personal and clinical history of the person to be evaluated. The clinical interview is a qualitative method of evaluation based on oral questions regarding specific contents and aimed to isolate and contextualize the complained difficulties (e.g., Lang, 2003; Lis, Venuti, & De Zordo, 1995). It allows taking into consideration a given cognitive problem (e.g., reading difficulties), estimating how it interferes with daily living activities and how it is subjectively experienced. The clinical interview is useful to investigate the context within which the complaint arises and to ascertain whether other issues, either

82


personal (e.g., emotional and motivational factors), familiar (e.g., sociocultural level), or educational (e.g., poor teaching), may play a causal role. At the end of this first step, the professional should realize whether the difficulties are secondary to these factors or, alternatively, reflect a specific, primary cognitive disorder which is independent of the context. In the case of students with learning disabilities, the parents should be interviewed first, then the teachers, finally the students. Parents should be met both simultaneously and separately. When met individually, the information provided by the parents, either consistent or conflicting, is useful to draw a complete picture of the difficulties to be evaluated and their impact on the everyday activities. In contrast, interviewing both parents contribute to establish a collaborative relationship which is necessary to plan and conduct an efficient intervention. The clinical interview is aimed to describe: (1) the alleged learning deficits; (2) the medical history, with special attention paid to any physical or mental disorders which could be associated to the learning deficits, such as visual and auditory defects, language disorder, hyperactivity and attention deficits, emotional disorders like depression or anxiety; (3) all social and personal factors which can influence the learning attitude, i.e., the interaction with peers, classmates, and teachers, the motivation and personal interests; the school activities and methods of education; (4) the household environment, e.g., sociocultural level of parents and relatives, emotional interaction and conflicts, relevant facts and events. The interview with the teachers is aimed to address the same issues; in particular, it has to provide information on the educational experience and learning difficulties. As well as for the parents, also for the teachers, the clinical interview is critical to establish a cooperative relationship (Bigozzi, 2000). The interview with the student (e.g., Greenspan, 2003; Morrison & Anders, 2001) is crucial to assess whether they are aware of the difficulties and available to collaborate. The professional must develop a ‘‘diagnosing alliance’’ to achieve an adequate diagnosis and, if necessary, an effective intervention (Del Corno & Lang, 1996). Different areas should be investigated: cognitive level, inclination and preferences, social interaction, copying strategies, meta-cognitive judgments.

Screening Tests The second step should allow confirming the presence of the alleged difficulties. For this purpose, screening tests for assessing the different cognitive abilities must be used. Screening tests are brief and easy instruments


83

developed to be administered to large groups of people in order to detect signs and defective behavior (e.g., Crocker & Algina, 1986; Streiner, 2003). To be useful, the screening tests must have specific psychometric properties, known as diagnostic efficiency statistics (e.g., Hsu, 2002; Streiner, 2003). First, screening tests must be sensitive and specific (e.g., Glaros & Kline, 1988; Hasselblad & Hedges, 1995). The sensitivity of a test is defined as the accuracy in identifying persons who have the disorder and corresponds to the proportion of persons having the disorder who are detected by the given test. Given the score on a screening test obtained by two groups of people with and without a specific disorder, the sensitivity is equal to the number of impaired individuals having a pathological score (true positives) ratio the number of persons belonging to the clinical group (the true positives plus those who do not fail the test, i.e., the false negatives). The specificity of a test is the accuracy in detecting people who do not have a given disorder and corresponds to the proportion of persons without the disorder who are correctly classified by the test. The specificity is equal to the number of persons with a negative score on the test, indicating the absence of the disorder (true negatives), ratio the number of persons without the disorder (the true negatives plus those erroneously classified as showing the disorder, i.e., the false positives). Sensitivity and specificity can be combined into a single statistic called likelihood ratio (e.g., Kraemer et al., 1999). This statistics tells what the odds are that a positive result has come from a person with the disorder and is equal to: Likelihood ratio ¼

Sensitivity True Positive Rate ¼ 1 Specificity False Positive Rate

For any given test score, the likelihood ratio may be computed via the receiver operating characteristic analysis (ROC; e.g., Kraemer et al., 1999; McFall & Treat, 1999). The higher is the likelihood ratio the lower is the likelihood of false positives and false negatives that may be detected by means of the test. The number of false negatives must be low to avoid failures in identifying individuals showing the target disorder. However, information gained by means of the clinical interview is useful to prevent a person, who obtained a score within the normal range on a single screening test but shows difficulties in everyday activities and suffered with some illness, from being wrongly classified as unimpaired. Also the false positives should be few to prevent individuals to be further evaluated, thus inducing stress and lack of confidence in the examiner and the examinee, and ambiguous findings which are impossible to interpret (Streiner, 2003).

84


A screening test is an effective diagnostic tool when among people with a pathological score on a given test, the percentage of those who really show the deficit is higher than the incidence of that disorder in the general population (base rate; e.g., Hsu, 2002; Streiner, 2003). To have a measure of the increase in the predictive values over what would be expected by going with the base rate, the incremental positive predictive power (IPPP) must be computed (also called incremental validity; Gibertini, Brandenburg, & Retzlaff, 1986; Kraemer et al., 1999). IPPP ¼

PPP PðDxÞ 1 PðDxÞ

Positive predictive power (PPP) is equal to the number of persons with a disorder ratio the number of persons detected with the disorder by the test, and P(Dx) is the probability of having the disorder (base rate). The ratio is equal to zero when the test adds nothing to the base rate and is equal to 1 when there are no diagnostic errors. On screening tests, people of the control groups make few mistakes, most of them perform flawlessly. Therefore, the typical distribution of raw scores on screening tests does not approximate to the normal distribution curve, but is leptokurtic with some degrees of skewed (respectively positively or negatively based on the score being equal to the number of wrongly or correctly answered items). Given this kind of nonnormal distribution, the percentile rank (PR) is the most useful normative score (e.g., Allen & Yen, 2002; Crocker & Algina, 1986). An individual is classified as being affected by a given disorder if the score on the screening test is higher or lower (based on the raw score being equal to the number of wrongly or correctly answered items, respectively) than a conventional threshold. To identify this threshold, a criterion or a normative approach can be used. Based on the criterion approach, the threshold should be the PR corresponding to the raw score which allows discriminating two groups of persons with and without the disorder, the raw score for which the likelihood ratio is highest (e.g., Scott, Fletcher, & Martell, 2000). Based on the normative approach, given a test administered to a sample representative of the general population showing that the rate of the disorder is not different to the base rate in the general population, the threshold should be the PR that corresponds to the base rate. For example, if the base rate is 15%, the PR threshold should be equal to 15 (and people diagnosed as defective are those obtaining lower scores) when the raw score is the number of the correct answers, or equal to 85 (people diagnosed as defective are those obtaining


85

higher scores) when the raw score is the number of the incorrect answers. The final threshold should be close to both the thresholds identified with the criterion and normative approaches. The PR scores are less reliable for raw scores in the central part of the distribution than for those in the extremes; therefore, to identify the threshold as well as the normative score obtained by an individual a percentile band equal to the PR71 standard error of measurement should be considered (e.g., Crocker & Algina, 1986; see, for more details, Crawford, Garthwaite, & Slick, 2009). To detect learning disabilities, several screening tests are available. Professionals have to evaluate the likelihood ratio, IPPP, and all methods to compute the threshold of the different test and to identify the most valid tests for assessing each cognitive process. One of the most valid screening tests for reading difficulties available in Italy is the MT test, which measures performance speed and accuracy in children from one to ten grades (Cornoldi & Colpo, 1998, 1995; Cornoldi, Baldi, & Friso, 2010). The MT test includes short texts to be read aloud; the total time of reading and of the number and type of errors (e.g., omission or addition of syllables and words; pause longer than 5 seconds) are the dependent variables. Two different raw scores are computed: the number of seconds used for each one syllable and the numbers of errors. The normative scores are similar to the PRs (i.e., cumulative percentage); for each text four thresholds are available indicating four levels of performance: (a) very low with immediate intervention needed; (b) low with attention required; (c) just above the critical threshold; (d) well within the normal range suggesting a learned ability to read. For the same texts, the confidence interval computed using the standard error of measurement is available. The test can be administered at the beginning, at the middle, and at the end of the school year. A series of screening tests provide psychometric measures on reading and writing as well as on other cognitive functions, e.g., perception and memory, thus allowing the identification of the specific domains which need attention and further investigation.

Standardized Batteries The third step is aimed to classify an individual performance by assigning a clinical labeling according to standard diagnostic criteria, e.g., DSM-IV-TR. Using information gleaned from the preliminary steps, the cognitive processes alleged to be impaired should be addressed. The diagnosis is not based on a single test, but on the pattern of dissociation and association between tasks (see Ellis & Young, 1988).

86


To diagnose specific disorders, the assessment must be based on theoretical models describing the functional architecture of the mental processes and predicting the specific consequences of a selective impairment. For example, the Italian Battery for the Assessment of Developmental Dyslexia and Dysorthography-2 (Sartori, Job, & Tressoldi, 2007) is based on the dual route model of reading (Coltheart & Rastle, 1994) and aims to investigate the integrity of the lexical and sublexical systems. Diagnostic batteries must be reliable and valid. Based on the European Federation of Psychologists’ Associations Standing Committee on Tests and Testing (Mun˜iz, 2009), the internal consistency is adequate if higher than .7, good if between .8 and .9, or excellent if higher than .9. Several methods are available to investigate the validity of a diagnostic test. The differentiation between groups with and without the disorder to be detected is one of the most useful (diagnosis must be realized in agreement of a manual of diagnostic criteria and independently from the performance of the two samples). Moreover, also the incremental validity of a test should be investigated; it represents the amount of variance in correctly detecting the disorder that the test could explain over another similar test (e.g., Hunsley & Mayer, 2003; for an example, see Balboni, Naglieri, & Cubelli, 2010). This information is helpful when determining the relative value of each test available for the measurement of a specific disorder. The typical distribution of raw scores on diagnostic tests approximates to the normal distribution curve; also the normative scores like, e.g., the T score and the deviation IQ score have a normal distribution. With this kind of scores, a performance on a test may be labeled as indicating a disorder when the normative score of an individual is below the mean of the general population, i.e., proximately one or two SD below. Moreover, to avoid the effects of the error of measurement, the single normative score must be converted in an interval obtained by subtracting and adding to the score the standard error of measurement of the test multiplied for 1.96 (5% of errors) or 2.58 (1%; e.g., Crocker & Algina, 1986). Actually, to make a diagnosis, to take into account the score on a single test is not enough; the performances on two different tests (or groups of tests), selected in agreement with the theoretical model of the investigated cognitive process, must be compared (e.g., Coltheart, 2001; Sartori, 1988). Typically, a dissociation is defined as occurring when a person shows a ‘‘deficit’’ on test A but is ‘‘normal’’ or ‘‘within normal limits’’ on test B. To reveal whether a dissociation occurs, a fully operational definition of ‘‘deficit’’ and ‘‘normal’’ is necessary, and methods to make comparisons


87

based on inferential statistics and not just descriptive statistics must be used (Crawford, Garthwaite, & Gray, 2003). If the normative group of a test is representative of the general population, the most common method of inferring a deficit is to use the normal deviate z. For this purpose, the raw score on a test of the person being evaluated is converted to the normative score (e.g., T score) based on the normative tables of the test. Then, the corresponding z-score is computed using the following formula: z¼

NS NS Mean NSSD

where NS is the normative score obtained by the individual, e.g., 45 T; NS Mean and NS SD are the mean and the SD of the normative score used, respectively, e.g., 50 T and 10 T. Then, z refers to the table of areas under the normal curve. If the probability of a z-score at least as large as the one obtained is less than .05 (or some other selected alpha value), the performance is considered to be significantly lower than the normative sample and the person labeled with a deficit in that test. Otherwise, if the probability of a z-score at least as large as the one obtained is more than .05, it is inferred the absence of the presumed deficit. If a representative normative sample is not available and therefore the normative score are computed with a sample of modest size, the method proposed by Crawford and Howell (1998) based on a modified t-test (Sokal & Rohlf, 1995) can be used. Given a defective performance on test A and a good performance on test B, to ascertain whether the difference between the z-scores obtained in the two tests is large enough for inferring a dissociation, the difference in z-scores should be compared with the distribution of differences in z-scores obtained by the normative sample. For this purpose, test A and test B must be standardized with the same normative group and the correlation between the scores in the two tests has to be available. If the normative group is representative, the standard deviation of the difference AB (SDAB; Payne & Jones, 1957) must be computed using the following formula: SDAB ¼

pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 2 rAB

where rAB is the correlation between the two tests in the normative sample. Then, a z-score for the difference (zD) may be calculated dividing the difference in z-scores by SDAB . Using the table of areas under the normal curve, it is possible to test if zD exceeds the required critical value (i.e., 1.64

88


for one-tailed significance or 1.96 for a two-tailed test at the .05 level). If the normative group is not representative, to calculate zD Crawford, Howell, and Garthwaite (1998) proposed to divide the difference in z-scores by the standard error of the difference in the normative group and not by the standard deviation of the difference. In this way, a quantity is yielding that is distributed as t with n1 degrees of freedom (see Crawford et al., 1998, for more details). Also methods for obtaining confidence limits on the estimates of test score extreme as well as of test score differences had been developed (Crawford & Garthwaite, 2002). All these methods can be used to compare the scores obtained on different tasks with different scales of the test battery or to compare scores obtained in different tests (provided that the correlation between the scores in the two tests obtained by the control group is available). When assessing learning disabilities, critical dissociation has to be considered. For instance, the label ‘‘Surface Dyslexia’’ requires a normal ability to read aloud regular words and novel words coupled to an impaired ability to read aloud irregular words. The diagnosis should be confirmed by the analysis of errors. In languages with opaque orthography, e.g., English or French, the phonological information corresponding to the graphemes can be ambiguous: e.g., in English the letter sequence oEAW is pronounced differently in the words head, beat, heart, and great; similarly, in French the letter sequence oENW is read differently in the words chre´tiente´ [Christianity] and impatiente [impatient]. For a number of words, the correct pronunciation cannot be derived using the grapheme-to-phoneme conversion rules; therefore, to be read flawlessly ‘‘exception words’’ have to be learned by heart and stored in the lexicon with their particular orthography to phonology mappings. Typically, individual with Surface Dyslexia produce regularization errors, with irregular words pronounced with the typical phoneme to grapheme correspondences: e.g., disease - decease (see Folegatti, Pia, Berti, & Cubelli, submitted). Overall, the different error types depend on the specific properties of the languages: for instance, in Hebrew most errors are incorrect vowel pronunciations in reading unvoweled homographs (Birnboim & Share, 1995), whereas in Italian most errors are word stress mis-assignment (e.g., Job, Sartori, Masterton, & Coltheart, 1983; Toraldo, Cattani, Zonca, Saletta, & Luzzatti, 2006).

Ad Hoc Investigation The last step of the assessment is aimed to identify in each individual the functional locus of impairment underlying the assigned clinical label. For


89

this purpose, additional tests are needed. In the case of surface dyslexia, for instance, the damage can be located at several different loci in the lexical route. As suggested by Coltheart and Funnell (1987), it can be due to (1) an impairment at the level of the orthographic input lexicon; (2) an impaired connection between the orthographic input lexicon and both the phonological output lexicon and the semantic system; and (3) an impairment at the level of the phonological output lexicon. Different patterns of performances can be predicted. For example, the performance on the Orthographic Decision test (Pizzoli, Lami, Pignatti, D’Alessandro, & Cubelli, 2008), whereby patients are asked to discriminate between a word and a pseudohomophone distracter (e.g., Brain and Brane), is expected to be defective (at chance) if the deficit occurs at the level of the orthographic input lexicon, but within the normal range if the deficit is due to impaired postorthographic mechanisms. At this stage of assessment the tests are used to go beyond the clinical label. The investigation should be adapted for the single patient and in some cases requires the adoption of ad hoc prepared tests. The choice of the testing material should be theoretically founded, i.e., based on explicit models, and not behaviorally oriented, i.e., based on visible signs. Using this approach, it has been possible to identify different subtypes of developmental surface dyslexia (Friedmann & Lukov, 2008) and developmental surface dysgraphia (Gvion, Friedmann, Rubin-Zaxenberg, & Mor, 2010), thus getting a functional diagnosis for each individual patient.

CONCLUSION A useful diagnosis consists in the precise description of the defective behavior and error types and in the functional interpretation of the underlying impaired cognitive mechanisms. Comparing the overall pattern of performances (described in terms of dissociation between tasks and qualitative analysis of errors) with a theoretical model describing the functional architecture of the assessed cognitive process allows deriving the identification of the specific loci of impairment in each single patient. A descriptive label of the disorder only confirms what is already known without providing any explanation regarding the damaged functional components of the cognitive system. To obtain a functional diagnosis, four separate steps with different diagnostic instruments are required: (1) clinical interview; (2) screening tests; (3) standardized battery; (4) ad hoc

90


investigation. Only in this way it would be possible to gain all information needed to plan tailored educational and rehabilitative interventions.

ACKNOWLEDGMENT The authors are grateful to Margherita Lang for her comments and suggestions.

REFERENCES Allen, M. J., & Yen, W. M. (2002). Introduction to measurement theory. Long Grove, IL: Waveland Press, Inc. Balboni, G., Naglieri, J. A., & Cubelli, R. (2010). Concurrent and predictive validity of the Raven Progressive Matrices and the Naglieri Nonverbal Ability Test. Journal of Psychoeducational Assessment, 28, 222–235. Bigozzi, L. (2000). Apprendimento e riabilitazione a scuola. Aspetti psicologici. Roma, Italy: Carocci. Birnboim, S. L., & Share, D. L. (1995). Surface dyslexia in Hebrew: A case study. Cognitive Neuropsychology, 12, 825–846. Coltheart, M. (2001). Assumptions and methods in cognitive neuropsychology. In: B. Rapp (Ed.), The handbook of cognitive neuropsychology (pp. 3–21). Philadelphia: Psychology Press. Coltheart, M., & Funnell, E. (1987). Reading and writing: One lexicon or two?. In: A. Allport, D. Mackay, W. Prinz & E. Sheerer (Eds), Language perception and production (pp. 313–339). London: Academic Press. Coltheart, M., & Rastle, K. (1994). Serial processing in reading aloud: Evidence for dual-route models of reading. Journal of Experimental Psychology: Human Perception and Performance, 20, 1197–1211. Coltheart, M., Rastle, K., Perry, C., Langdon, R., & Ziegler, J. (2001). DRC: A dual route cascaded model of visual word recognition and reading aloud. Psychological Review, 108, 204–256. Cornoldi, C., Baldi, A. P., & Friso, G. (2010). Prove MT Avanzate di lettura e matematica-2 per il biennio della scuola superiore di II grado. Firenze, Italy: Giunti OS. Cornoldi, C., & Colpo, G. (1995). Nuove prove di lettura MT per la scuola media inferiore. Firenze: Giunti OS. Cornoldi, C., & Colpo, G. (1998). Prove di lettura MT per la scuola elementare-2. Firenze: Giunti OS. Crawford, J. R., & Garthwaite, P. H. (2002). Investigation of the single case in neuropsychology: Confidence limits on the abnormality of test scores and test score differences. Neuropsychologia, 40, 1196–1208. Crawford, J. R., Garthwaite, P. H., & Gray, C. D. (2003). Wanted: Full operational definitions of dissociations in single-case studies. Cortex, 39, 357–370.


91

Crawford, J. R., Garthwaite, P. H., & Slick, D. J. (2009). On percentile norms in neuropsychology: Proposed reporting standards and methods for quantifying the uncertainty over the percentile ranks of test scores. The Clinical Neuropsychologist, 23, 1173–1195. Crawford, J. R., & Howell, D. C. (1998). Comparing an individual’s test score against norms derived from small samples. The Clinical Neuropsychologist, 12, 482–486. Crawford, J. R., Howell, D. C., & Garthwaite, P. H. (1998). Payne and Jones revisited: Estimating the abnormality of test score differences using a modified paired samples t-test. Journal of Clinical and Experimental Neuropsychology, 20, 898–905. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Belmont, CA: Wadsworth, Thomson Learning. Cubelli, R., & Della Sala, S. (in press). The purposes of neuropsychological assessment and how to achieve them. Advances in Clinical Neuroscience and Rehabilitation. Del Corno, F., & Lang, M. (1996). La relazione con il paziente. Milano: Libreria Cortina. Ellis, A., & Young, A. (1988). Human cognitive neuropsychology. Hillsdale, NJ: Lawrence Erlbaum. Folegatti, A., Pia, L., Berti. A., & Cubelli, R. (submitted). Stress assignment errors in surface dyslexia: Evidence from two Italian patients with a selective deficit of the orthographic input lexicon. Friedmann, N., & Lukov, L. (2008). Developmental surface dyslexias. Cortex, 44, 1146–1160. Gibertini, M., Brandenburg, N., & Retzlaff, P. (1986). The operating characteristics of the Million Clinical Multiaxial Inventory. Journal of Personality Assessment, 50, 554–567. Glaros, A. G., & Kline, R. B. (1988). Understanding the accuracy of tests with cutting scores: The sensitivity, specificity, and predictive value model. Journal of Clinical Psychology, 44, 1013–1023. Greenspan, S. I. (2003). The clinical interview of the child (3rd ed.). Arlington, VA: American Psychiatric Publishing. Gvion, A., Friedmann, N., Rubin-Zaxenberg, I., & Mor, R. (2010). Subtypes of developmental surface dysgraphia. Procedia Social and Behavioral Sciences, 6, 145–147. Hasselblad, V., & Hedges, L. V. (1995). Meta-analysis of screening and diagnostic tests. Quantitative Methods in Psychology, 111, 167–178. Hsu, L. M. (2002). Diagnostic validity statistics and the MCMI-III. Psychological Assessment, 14, 410–422. Hunsley, J., & Mayer, G. J. (2003). The incremental validity of psychological testing and assessment: Conceptual, methodological, and statistical issue. Psychological Assessment, 15, 446–455. Job, R., Sartori, G., Masterton, J., & Coltheart, M. (1983). Developmental surface dyslexia in Italian. In: R. N. Malatesha & H. Whitaker (Eds), Dyslexia: A global issue (pp. 133–141). The Hague: Martinus Njihoff. Kraemer, H. C., Kazdin, A. E., Offord, D. R., Kessler, R. C., Jensen, P. S., & Kupfer, D. J. (1999). Measuring the potency of risk factors for clinical or policy significance. Psychological Methods, 4, 257–271. Lang, M. (2003). Il colloquio clinico. Milano, Italy: Libreria Cortina. Lis, A., Venuti, P., & De Zordo, M. R. (1995). Il colloquio come strumento psicologico (2nd ed.). Firenze, Italy: Giunti. McFall, R. M., & Treat, T. A. (1999). Quantifying the information value of clinical assessments with signal detection theory. Annual Review of Psychology, 50, 215–241.

92


Morrison, J., & Anders, T. F. (2001). Interviewing children and adolescents. New York: Guilford Press. Mun˜iz, J. (2009, July). The role of EFPA in setting standards for tests and test use. Paper presented at the 11th European Congress of Psychology, Oslo, Norway. Payne, R. W., & Jones, G. (1957). Statistics for the investigation of individual cases. Journal of Clinical Psychology, 13, 115–121. Pizzoli, C., Lami, L., Pignatti, B., D’Alessandro, D., & Cubelli, R. (2008). Il compito di decisione ortografica per la valutazione dell’acquisizione lessicale in eta` evolutiva. Psicologia Clinica dello Sviluppo, 12, 25–40. Sartori, G. (1988). From models to neuropsychological data and vice versa. In: G. Denes, P. Bisiacchi, C. Semenza & E. Andreewsky (Eds), Perspectives in cognitive neuropsychology (pp. 59–73). Hove, UK: Erlbaum. Sartori, G., Job, R., & Tressoldi, P. E. (2007). DDE-2 Batteria per la valutazione della dislessia e della disortografia evolutiva-2. Firenze, Italy: Giunti OS. Scott, M. S., Fletcher, K. L., & Martell, B. M. (2000). Selecting components for a screening test to identify three-year-olds at risk for mild learning problems. Education and Training in Mental Retardation and Developmental Disabilities, 35, 208–221. Sokal, R. R., & Rohlf, J. F. (1995). Biometry. San Francisco: W.H. Freeman. Streiner, D. L. (2003). Diagnostics tests: Using and misusing diagnostic and screening tests. Journal of Personality Assessment, 81, 209–219. Toraldo, A., Cattani, B., Zonca, G., Saletta, P., & Luzzatti, C. (2006). Reading disorders in a language with shallow orthography: A multiple single case study in Italian. Aphasiology, 20(9–11), 823–850.

A COMPARISON OF OBSERVATIONAL TECHNIQUES FOR ASSESSING STUDENTS’ SOCIAL BEHAVIOR Jugnu Agrawal, Dannette Allen-Bronaugh and Margo A. Mastropieri ABSTRACT This study compares two methods of data collection for students’ social behaviors. One method employed time sampling procedures, while the other method used handheld computerized devices and the Multi-Option Observation System for Experimental Studies (MOOSES) system. Both coding systems were used to assess social behaviors of students with emotional disabilities during writing instruction. The middle-school-aged students, all classified as having EBD, were enrolled in classes to improve their written expression. Students were assessed for on-task, off-task, and multitask behaviors. Results revealed some surprising differences. When students were relatively consistent with attendance and on-task behaviors, the methods yielded comparable results; however, when students were more disruptive and demonstrated more inconsistent behaviors, different patterns emerged. Implications and recommendations for future research and practice are discussed. Assessment and Intervention Advances in Learning and Behavioral Disabilities, Volume 24, 93–110 Copyright r 2011 by Emerald Group Publishing Limited All rights of reproduction in any form reserved ISSN: 0735-004X/doi:10.1108/S0735-004X(2011)0000024007

93

94

JUGNU AGRAWAL ET AL.

Assessing students’ behaviors is a critical component of evaluating both instruction and behavioral intervention for students. A variety of methods and procedures are available to researchers. These methods vary not only in the way the data are collected, but also how data may be analyzed. Time sampling and Multi-Option Observation System for Experimental Studies (MOOSES) are two assessment methods that have been extensively used to collect observational data in many studies. Time sampling is a procedure during which a time segment is divided into intervals and behavior is recorded at each prespecified time interval (Brulle & Repp, 1984). Observers record whether an occurrence of the behavior is present at the end of a predetermined interval, such as 30 seconds. This method is useful when an observer cannot watch the participant continuously in a particular setting or during a particular task. This method permits observations of a group of participants simultaneously, for example, a teacher can use this method to collect data on behaviors, such as student on-task engagement, for a larger group of individual students in her classroom. Moreover, this procedure is relatively cost effective. A stopwatch, a beep tape (a device with recorded signals such as clicks or beeps as prompts to observe and record behaviors), headphones, a clipboard, paper, and pencil are the only tools required. Time sampling has been used extensively to observe on-task behaviors of students with emotional and behavioral disorders (EBD) (e.g., Blood, 2010; Mastropieri et al., 2009). Time sampling can be completed in a relatively cost-effective way since the required materials are inexpensive. With advancing technologies, additional assessment and observational approaches are now available. A variety of systems use handheld computerized devices, which have the convenience of collecting and exporting data into files that are ready for data analysis. MOOSES is one program that allows for real-time collection and analysis of data obtained from observational processes and participants (Tapp & Wehby, 1993). MOOSES provides opportunities for more flexible data collection using researcherdesigned codes, including mutually exclusive events, interval coding, coding actual minutes and seconds of behavioral occurrences, and toggling between behaviors. Data collapsing and analytical procedures are also embedded within the program. MOOSES has been employed in multiple studies to code a variety of observational data for students of all ages and for assessing teacher behaviors (Reinke, Lewis-Palmer, & Merrell, 2008; WebsterStratton, Reid, & Stoolmiller, 2008). Several studies have used MOOSES to study student engagement, on-task behavior, teacher and student

Comparison of Observational Techniques

95

interactions, and their responses during instruction (e.g., Kamps & Greenwood, 2005; Tankersley, Kamps, Mancina, & Weidinger, 1996). Miller, Lane, and Wehby (2005) studied the effects of social skills training on the inappropriate behaviors of elementary school students with highincidence disabilities and used MOOSES to record the occurrence of inappropriate classroom behaviors and attending during baseline, intervention, and postintervention phases. MOOSES provides researchers with additional observational and recording functions that are not possible with time sampling procedures. However, MOOSES is more costly, requires the purchase of handheld computerized devices, and the software. This study was designed to determine whether differences existed between the two methods of assessment when used to measure students’ social behaviors during academic instruction.

METHOD Students Participants were seven middle school students, six males and one female, who were all classified as having emotional or behavioral disabilities. All students were participants in projects designed to improve their written expression as part of a larger project, but none of the presently analyzed data were included in those reports (Mastropieri et al., 2009, 2010, 2011). Four students attended a specialized middle school for students with EBD and three attended a traditional middle school. The racial and ethnic composition was diverse: three were African American, one Hispanic, two Caucasian, and one Asian. The average age was 13.3 years (range from 12.3 to 14.5 years). Table 1 presents the student characteristics. Students’ emotional and behavioral issues included social problems, academic engagement, self-regulation, following directions, and attendance. Students exhibited challenging behaviors, which is why they were selected for the current study. Setting The settings for this investigation were two middle schools in a large public school district on the east coast. One school with a population of approximately 100 was for students with EBD. There were more males

Male

Male

Male

Male

Male

Male

Female

Wilton

Liam

Edgar

Evan

George

Malcolm

Sally

African American

African American African American

Hispanic

Caucasian

Asian

Caucasian

Ethnicity

13.10

13.20

13.60

14.50

13.00

12.30

13.80

Age (Years, Months)

8

8

8

8

7

7

8

Grade Level

ED, OHI

ED, OHI

ED, LD, ESOL ED, LD

OHI, AUT, ED Serv AUT, ED Serv OHI, ED Serve

Special Education Categories

Work independently; decrease attention-seeking behaviors, following directions Self-awareness, coping skills

Coping strategies, attendance

Stress reduction strategies

Social problem solving, social awareness Verbal interpersonal interactions, on-task behavior Self-regulation

Behavioral Goals

Student Characteristics.

WISC IV; VCI 87; PRI 82; WMI 97; PSI 80; Full Scale 82; WJ BWL 80

WISC IV; VCI 100; PRI 106; WMI 80; PSI 75; Full Scale GAI 103; WJ BWL 80; TOWL III 96 WISC IV; VCI 83; PRI 94; WMI 62; PSI 73; Full Scale 75; WJ BWL 78 RIAS; VI 65; NI 98; CI 81; CMI 54: WJ BWL 86 WASI; VA 98; PA 104; Full Scale 101; WJ BWL 80

SBIS; VR 109; QR 108; AVR 92; STM 93; Composite 101; VABS 93/80 WJ BWL 92; UNIT 112

Test Scores

ED, emotional disabilities; ESOL, English for speakers of other languages; LD, learning disabilities; OHI, other health impairments; AUT, autism spectrum disorder; ED Serv, receiving emotional disabilities services; WISC IV, Wechsler Intelligence Scale for Children–fourth edition (Wechsler, 2003); VCI, Verbal Comprehension Index; PRI, Perceptual Reasoning Index; WMI, Working Memory Index; PSI, Processing Speed Index; Full Scale, Full Scale IQ; Full Scale GAI, Full Scale IQ for General Ability Index; WJ BWL, Woodcock Johnson Broad Written Language; WRM, Woodcock Reading Mastery Tests; TOWL-3, Test of Oral Written Language (Hammill & Larsen, 1996); UNIT, Universal Nonverbal Intelligence Test (Bracken & McCallum, 1998); SBIS, Stanford–Binet Intelligence Scale – fourth edition (Thorndike, Hagen, & Sattler, 1986); VR, verbal reasoning; QR, quantitative reasoning; STM, short-term memory; Composite, composite score; VABS, Vineland Adaptive Behavior Scale (Sparrow, Balla, Cicchetti, & Doll, 1984); RIAS, Reynolds Intellectual Assessment Scales Full Scale (Reynolds & Kamphaus, 2003); VI, Verbal Intelligence Index; NI, Nonverbal Intelligence Index; CI, Composite Intelligence Index; CMI, Composite Memory Index; WASI, Wechsler Abbreviated Scale of Intelligence (The Psychological Corporation, 1999); VA, verbal reasoning; PA, performance.

Gender

Name

Table 1.

96 JUGNU AGRAWAL ET AL.


97

(82%) than females (17%) enrolled, and 53% received free and reduced lunch. The sample was ethnically and racially diverse: 37% were Caucasian, 25% were African American, 23% were Hispanic, and 6% were Asian. The second school was a traditional middle school with a population of approximately 1,000 seventh and eighth grade students of whom 52% were males and 48% were females. Fifteen percent of the students received special education services and 11% of the students received free and reduced lunch. The racial and ethnic composition included 62% Caucasian, 19% Asian, 7% Hispanic, and 4% African Americans.

Project Staff The research team included two advanced doctoral students and a faculty member who had been participating in observing and recording behavior on similar projects. All had received extensive training in observing and recording social behaviors using videotapes, beep tapes, and MOOSES. Additionally, all had previous experience assessing, observing, and recording student behaviors.

MATERIALS Time Sampling A data-recording sheet was created to collect observational data using the time sampling method (see Fig. 1) with Microsoft Excel. The landscape orientation sheet was divided into 30-second segments to equal 30 scoring units for a 15-minute recording period. Qualitative notes about the classrooms, lessons, students, and behaviors could be written on a designated space on the sheet. Numeric codes were devised to record specific student behaviors. For example, one was for on-task, two for offtask, three for out of classroom, four for when students return to classroom. Table 2 provides all behavioral codes for both time sampling and MOOSES. A digitally recorded audio file containing clicks every 30 seconds for 15 minutes was created and included on a handheld computerized device; however, this file could also have been installed on an ipod or a comparable more inexpensive device. Earbud headphones were used with the handheld device so that only observers could hear the clicks.

Fig. 1.

Coding Sheet Used for Time Sampling.

98 JUGNU AGRAWAL ET AL.

99


Table 2.

Codes for Student Behavior.

Time Sampling

MOOSES

1 ¼ On-task behavior 2 ¼ Off-task behavior 3 ¼ Out of classroom 4 ¼ Student returns to classroom 5 ¼ Multitask

On-task (ot) behavior Off-task (ft) behavior Multitasking (mt) behaviors Amount of time in class (ic) Amount of time out of class (oc)

MOOSES An HP handheld computerized device was installed with the Minimooses program. Minimoose is the program designed to accompany the MOOSES program that is installed on PocketPC handheld devices to collect data. A PC with MOOSES software installed was used to record and analyze data. For the purpose of this study, the frequency and duration and sequential analysis tools of the MOOSES software were used. The Mininmooses and MOOSES were set up with the codes for the behavioral categories (see Table 2 for specific codes). Minimooses was set up on mixed mode for a total of 300-second (5 minutes) duration for observation, with 25 seconds for observation and 5 seconds for recording. It was decided to use 5-minute segments of coding for MOOSES and complete three MOOSES 5-minute segments during the identical 15-minute time sampling interval to have a direct comparison across observational techniques. Fiveminute samples per student have been used to provide samples of social behavior repeatedly in the literature. Frequently, 5-minute samples are added together for individuals across sessions. In addition, when observing larger numbers of students in a session, observers collect data on one student for 5 minutes, move to the next student for 5 minutes, and to subsequent students using MOOSES for 5 minutes each time. This procedure permits observers to collect continuous data on individual students, but also multiple students within a setting during a single class period. For example, Tankersley et al. (1996) reported using the 5-minute observational data collection sample successfully. Both Conditions Videotaped samples of student behaviors during instruction were selected for use in the current study. All selected segments contained the same

100


writing instructional sequences, and were edited to obtain video clips of the 7 targeted students over 11 instructional sessions. The videotapes were obtained as part of a larger project designed to teach written expression and had been used to assess fidelity of treatment. However, these tapes were never used to code on-task behaviors. Target students and segments were selected if segments included (a) unobstructed views of the target students for 15 minutes of a lesson; (b) sufficient sound clarity to hear teacher and student comments throughout the segment; and (c) sufficient sequential instructional days with specific students present. Microsoft Excel and Statistical Package for the Social Sciences (SPSS) were used for analyzing and coding the data obtained from both data collection procedures. All student social behaviors were operationally defined as described next. On-task Behaviors During whole group instruction, looking at the teacher, staying in the chair, listening to the teacher, and answering teacher’s questions were defined on-task behaviors. During individual work time, working actively on the writing process, including using the planning and organizing sheets, writing essays, and general compliant behavior such as keeping hands and feet to self and talking only when appropriate, were noted as on-task behaviors. Off-task Behaviors During whole group instruction and individual work time, putting heads down on desks, talking to peers, doodling, fidgeting with inappropriate materials, getting up and walking around in the classroom, not responding to teacher’s questions, and general noncompliant behaviors such as refusing to participate or speaking out inappropriately were regarded as off-task behaviors. Multitask Behaviors The code of multitask was included because it was noted that many students appeared to exhibit both on- and off-tasks behaviors simultaneously. For example, it was frequently noted that students would be participating actively during class discussions, but also be involved with fidgeting behaviors with inappropriate materials. It was decided that observers could reliably code multitask when the dominant behavior was actually on-task, but simultaneously the student was involved in some type of off-task behavior.


101

Procedures Researcher Training All researchers had prior experience in coding using the time sampling method. The researchers were trained by a faculty member to code and analyze the data using Minimooses and MOOSES. Multiple videotaped practice sessions were used to train the researchers in coding using both methods of data collection. Ten videos were used for training reliability. Once 100% reliability and comfort level using the handheld devices with Minimooses codes was obtained, coding began. Coding Procedures Once segments of student behavior over time were identified, 15-minute segments were coded using time sampling procedures. The same 15-minute segments were divided into three 5-minute segments and coded using the MOOSES procedure. This procedure was repeated across instructional days for all seven students to obtain a sufficient sample of observational data using each method. Reliability of Observation Interobserver agreement (IOA) was calculated for 30% of the observations for time sampling and MOOSES. The overall average IOA for time sampling was 92% (range 90.0–96.6%). The overall average IOA for MOOSES was 97% (range 89.6–100%). MOOSES was used to calculate IOA for frequency and duration measures. Percentages of agreement scores were calculated using the following formula: agreements/(agreements þ disagreements) 100 (MacLean, Tapp, & Johnson, 1985).

RESULTS Observational data from time sampling for each student was entered into Microsoft Excel. Those individual data points for on-task behavior were summed and added with the total multitask behaviors to obtain total ontask scores for each student for each video segment. It was decided to combine multitask with on-task behaviors because the findings separately paralleled one another. Scores were averaged and converted into mean percent of time on-task for each segment and for overall mean percentages. The data collected using Minimooses on the handheld devices was downloaded to a computer and analyzed using MOOSES software.

102


MOOSES calculated the percentage of on-task, off-task, and multitask behaviors. Mean percent of time on-task for each 5-minute segment was added to the mean percent of multitask for each 5-minute segment to yield a total time on-task for each 5-minute segment per student per observation. Descriptive data for each of the four segments (one 15-minute time sampling segment and three 5-minute MOOSES data segments) are displayed in Table 3 by student across all observation periods. Considerable variability was obtained in total on-task behaviors ranging from 10% to 100%. It was observed that overall the total on-task behavior by student appeared somewhat less than desirable for most students with the exception of Wilton. For example, for Wilton, time sampling obtained an overall 88% time on-task, while MOOSES-coded segments revealed 90%, 91%, and 92%, respectively, for segments 1, 2, and 3. In this case it appears that either observational technique provided percentages of time on-task that could be considered highly similar. This is also seen in Fig. 2, which displays Wilton’s on-task behavior by coding technique across instructional days. As can be seen, when examining daily data, some segments of MOOSES coding revealed more variability with substantially less on-task behavior. But overall Wilton’s on-task behavior was quite similar across instructional days and he had fairly consistent on-task behavior, indicating that either coding method revealed comparable results. Conversely, Liam’s results revealed more variability in on-task behavior across methods of observation. He obtained a mean of 81% on-task behavior with time sampling, but his MOOSES mean on-task scores revealed more variability with ranges from 75% (segment 1), to 84% (segment 2), to 92% (segment 3). Liam’s behavior ranged from 70% to 100% for on-task behavior with time sampling; however, his range was 29–100% for on-task using MOOSES. A different picture of Liam’s on-task behavior would be obtained depending on the observational technique selected and the amount of time observed. For example, if Liam were only observed for the first 5-minute segment, he was on-task only 75% of the time. In reviewing Liam’s daily mean data on Fig. 3, a different picture of his on-task behavior emerges because his behavior ranges from a low of 30% to 100% on-task during the first 5-minutes coding segment. However, when only examining the third 5-minute segment of his behavior, Liam was on-task 92% of the time. Liam’s on-task behavior ranges from a student who is mostly on-task to one who has challenges staying on-task when using time sampling versus the MOOSES. But perhaps even more interesting is how the 5-minute segments selected to code using MOOSES resulted in extreme variability for Liam’s on-task behaviors.

First 5 minutes MOOSES Second 5 minutes MOOSES Third 5 minutes MOOSES

Time Sampling

88 (0.11) 67–100 90 (0.11) 69–100 91 (0.10) 70–100 82 (0.16) 59–100

Wilton Mean (SD) Range

81(0.16) 70–100 75 (0.23) 29–100 84 (0.16) 60–100 92 (0.09) 60–100

85 (0.17) 52–100 75 (0.19) 50–100 74 (0.20) 50–100 90 (0.08) 40–100

Edgar Mean (SD) Range 67 (0.20) 37–93 68 (0.26) 39–100 78 (0.20) 50–100 73 (0.19) 28–100

Evan Mean (SD) Range 50 (0.27) 10–90 51 (0.26) 9–100 72 (0.15) 50–90 60 (0.30) 29–100

George Mean (SD) Range

Mean Percent Data for On-task Behavior.

Liam Mean (SD) Range

Table 3.

63(0.17) 27–87 65 (0.33) 30–100 81 (0.13) 49–100 58 (0.16) 23–80

Malcolm Mean (SD) Range

63 (0.17) 17–80 69 (0.32) 9–100 65 (0.23) 20–90 64 (0.25) 49–100

Sally Mean (SD) Range

Comparison of Observational Techniques 103

104

JUGNU AGRAWAL ET AL. 100

Mean Percentages

90

80 Time Sampling 1st 5 min MOOSES 2nd 5 min MOOSES 3rd 5min MOOSES

70

60

50

40

Fig. 2.

1

2

3

4

5

6 7 Days

8

9

10

11

Wilton’s Mean On-task Behavior by Day by Time Sampling versus 5-Minute MOOSES Samples.

100 90

Mean Percentages

80 70 Time Sampling 1st 5min MOOSES 2nd 5min MOOSES 3rd 5min MOOSES

60 50 40 30 20

Fig. 3.

1

2

3

4

5

6 7 Days

8

9

10

11

Liam’s Mean On-task Behavior by Day by Time Sampling versus 5-Minute MOOSES Samples.

105


Edgar’s patterns of on-task behavior also revealed significant variability. His mean 85% on-task with time sampling is somewhat the average of his three MOOSES segments of 75%, 74%, and 90%. However, this variability indicates that the selection of a method and timing sampling may obfuscate his actual on-task behavior. George and Malcolm’s data revealed generally low levels of on-task behavior, but the large ranges observed in on-task behavior observed indicate enormous variability in their on-task behavior over days. George obtained a mean of 50% on-task with time sampling (range 10–90%). When observing the same segments, but coding using the MOOSES his first 5-minute segment was 51 (range 9–100%) and highly similar to time sampling, but the second segment (M ¼ 72) and the third segment (M ¼ 60) represent more variability. Malcolm’s pattern of data is highly similar to George’s in that he obtained generally low levels of on-task behavior, but also demonstrated significant variability between the time sampling (M ¼ 63) and some MOOSES segments with means of 81–58. Fig. 4 displays graphically the extreme variability obtained in on-task behavior by days across the two methods and by the 5-minute segments. Evan and Sally both presented generally low levels of on-task behavior. But even with those low levels there were differences noted across the

100 90

Mean Percentages

80 70 60 Time Sampling 1st 5min MOOSES 2nd 5min MOOSES 3rd 5min MOOSES

50 40 30 20 10 0

Fig. 4.

1

2

3

4 Days

5

6

7

Malcolm’s Mean On-task Behavior by Day by Time Sampling versus 5-Minute MOOSES Samples.

106

JUGNU AGRAWAL ET AL. 100 90

Mean Percentages

80 70 60 Time Sampling 1st 5 min3 MOOSES 2nd 5 min MOOSES 3rd 5 min MOOSES

50 40 30 20 10 0

1

2

3

4

5

6

7

Days

Fig. 5.

Sally’s Mean On-task Behavior by Day by Time Sampling versus 5-Minute MOOSES Samples.

techniques employed. Sally’s time sampling mean of 63% ranged from 17% to 80%. Given such variability it is interesting to note that her means with each MOOSES segment appear more comparable to time sampling with means of 69, 65, and 64. However, Sally’s MOOSES ranges from 9% to 100%. Fig. 5 graphically shows Sally’s variability across days and highlights how the selection of a particular 5-minute segment might provide different on-task behaviors for her. Across these students with EBD, each observational technique does in fact present reliable and accurate indicators of time on-task. However, there is also considerable variability within student outcomes across measures when comparing the two techniques. Perhaps even more interesting, different 5-minute segments can reveal significant variability in on-task behavior.

DISCUSSION In the present study, different patterns of on-task behavior were obtained when different observational techniques were selected. Moreover, variability


107

in on-task behavior was also obtained when selecting different 5-minute samples of students’ behavior for some students. It must be recalled that this entire sample was identified as having EBD, so their behaviors might be considered more erratic than most. However, even within this present sample, students who were fairly consistent with their behavior during observation periods looked highly similar no matter which observation technique was employed. This was not only in the case of Wilton, who actually had relatively higher on-task behaviors, but also in the case of Sally, who had consistently lower on-task behavior. However, students who were behaviorally inconsistent more frequently appeared differently with respect to mean on-task behavior with the different observation techniques. These students also exhibited more variability across the three 5-minute segments in on-task when using the MOOSES. In the case of students with inconsistent on-task behaviors, MOOSES, if completed for substantially longer time periods on each student, may provide a more accurate picture of on-task behaviors, but the selection of a 5-minute sample may not yield a clear picture on on-task behavior. Many studies that employed MOOSES coded single students for 5 minutes and then moved to a different student for the next 5 minutes. When using such a procedure, researchers should be cautioned to look more closely at larger samples of individual behavior to improve accuracy of findings. In situation of coding multiple student behaviors in a class, time sampling may provide more information on larger groups of students simultaneously. However, time sampling will not provide the detailed duration and frequency data that MOOSES can provide on a single student. The selection of an observation technique may be based on multiple factors, but may also produce somewhat different findings. Both observational techniques have strengths and limitations. Time sampling is less expensive than MOOSES. In the present study, time sampling record sheets were created using Microsoft Excel, which is a commonly available program. Multiple students can be observed and coded simultaneously using time sampling, but it is more challenging to code multiple behaviors. The present time sampling record sheet contained space to document qualitative field notes on student behavior and setting. However, since time sampling captures a snapshot of the student behavior only at the end of the time segment, it is possible that students’ behaviors may be inaccurately reflected for the time prior to recording. Moreover, the device used to listen to the beeps can fail to start or cease operating during observations. MOOSES has several major advantages over time sampling alone. Multiple variables related to one student can be coded simultaneously.

108


In the current study, additional variables like praise academic, praise social, reprimand academic, and reprimand social were coded but not analyzed. MOOSES software calculates the total duration of the time on-task, offtask, and multitask for each student in percentages as well as total seconds. MOOSES software calculates the percentages and duration of a behavior in a simple way. Conversely, MOOSES software is expensive because it also requires the purchase of handheld computerized devices to house the Minimooses. Usually a single student’s behavior is coded using MOOSES. In this case, the duration codes on the Minimooses were adjusted to reflect the 5-minute observation duration with 25 seconds of observation interval and 5 seconds of recording interval. The codes in MOOSES are case sensitive and codes on the handheld devices must match the codes on the computer. As with any technology, users have to gain familiarity when using technology to make any required changes and adjustments to the MOOSES. In the present study, sometimes the Minimooses failed to start on time for the observation. In those instances the observation had to be conducted again, which was time consuming and cumbersome. In the present study, all that was required was to restart the movie; however, had this been live coding some behavioral coding would have been missed. Finally, the Minimooses screen does not provide the option of noting qualitative details. Both observational techniques have promise for providing an assessment of social behavior, such as time on-task. A careful consideration of the costs and benefits associated with each technique need to be clearly evaluated prior to selecting a technique. More importantly, however, this study provided initial evidence that different results are obtained with the different observational techniques. This was especially apparent when only 5-minute time segments were used to collect samples of student behavior. Careful consideration about the goals and objectives for assessing student behavior should be completed prior to selecting a technique, and those decisions should be weighed carefully with the costs and benefits associated with each potential observational technique. Future research replicating the study with a larger sample may provide additional evidence and guidelines for observers.

ACKNOWLEDGMENT Partial support for this research was provided by Grant No. H325D070008 from the U.S. Department of Education, Office of Special Education Programs, and Grant No. R324A070199-07 from the U.S. Department of


109

Education, Institute of Educational Sciences, awarded to Pennsylvania State University with a subcontract to George Mason University.

REFERENCES Blood, E. (2010). Effects of student response systems on participation and learning of students with emotional and behavioral disorders. Behavioral Disorders, 35, 214–228. Bracken, B. A., & McCallum, R. S. (1998). Universal nonverbal intelligence test. Itsaka, IL: Riverside. Brulle, A. R., & Repp, A. C. (1984). An investigation of momentary time sampling procedures with time series data. British Journal of Psychology, 75, 481–485. Hammill, D., & Larsen, S. (1996). Test of written language-3. Austin, TX: Pro-ED. Kamps, D. M., & Greenwood, C. R. (2005). Formulating secondary-level reading interventions. Journal of Learning Disabilities, 38, 500–509. doi: 10.1177/00222194050380060501 MacLean, W. E., Jr., Tapp, J. T., Sr., & Johnson, W. L. (1985). Alternate methods and software for calculating interobserver agreement for continuous observation data. Journal of Psychopathology and Behavioral Assessment, 7, 65–73. doi: 10.1007/BF00961847 Mastropieri, M. A., Scruggs, T. E., Cuenca-Sanchez, Y., Irby, N., Mills, S., & Mason, L. H. (2010). Persuading students with emotional disabilities to write: Findings from a design experiment. In: T. E. Scruggs & M. A. Mastropieri (Eds), Literacy and learning: Advances in learning and behavioral disabilities (Vol. 23, pp. 237–268). Bingley, UK: Emerald. doi: 10.1108/S0735-004X(2010)0000023011 Mastropieri, M. A., Scruggs, T. E., Irby, N., Allen-Bronaugh, D., Thompson, C., Guckert, M., et al. (2011). Fluent persuasive writing with counter reasons: Success with students with emotional disabilities. Paper to be presented at the annual meeting of the American Educational Research Association, New Orleans. Mastropieri, M. A., Scruggs, T. E., Mills, S., Irby, N., Cuenca-Sanchez, Y., & Bronaugh, D. A. (2009). Persuading students with emotional disabilities to write fluently. Behavioral Disorders, 35, 19–40. Miller, M., Lane, K. L., & Wehby, J. (2005). Social skills instruction for students with highincidence disabilities: A school-based intervention to address acquisition deficits. Preventing School Failure, 49, 27–40. doi: 10.3200/PSFL.49.2.27-39 Reynolds, C. R., & Kamphaus, R. W. (2003). Reynolds intellectual assessment scales. Lutz, FL: Psychological Assessment Resources Inc. Reinke, W. M., Lewis-Palmer, T., & Merrell, K. (2008). The classroom check-up: A classwide consultation model for increasing praise and decreasing disruptive behavior. School Psychology Review, 37, 315–332. Sparrow, S. S., Balla, D. A., Cicchetti, D. V., & Doll, E. A. (1984). Vineland adaptive behavior scale: Interview edition survey form manual. Circle Pines, MN: American Guidance Service. Tankersley, M., Kamps, D., Mancina, C., & Weidinger, D. (1996). Social interventions for head start children with behavioral risks: Implementation and outcomes. Journal of Emotional and Behavioral Disabilities, 4, 171–181. doi: 10.1177/106342669600400304 Tapp, J., & Wehby, J. (1993). MOOSES: Multiple option observation system for experimental studies. Available at http://www.eric.ed.gov.mutex.gmu.edu/PDFS/ED361372.pdf

110


The Psychological Corporation. (1999). Wechsler abbreviated scale of intelligence. San Antonio, TX: The Psychological Corporation. Thorndike, R. L., Hagen, E. P., & Sattler, J. M. (1986). Stanford–Binet intelligence scale (4th ed.). Rolling Meadows, IL: Riverside Publishing. Webster-Stratton, C., Reid, M. J., & Stoolmiller, M. (2008). Preventing conduct problems and improving school readiness: Evaluation of the incredible years teacher and child training programs in high-risk schools. The Journal of Child Psychology and Psychiatry, 49, 471–488. doi: 10.1111/j.1469-7610.2007.01861.x Wechsler, D. (2003). Wechsler intelligence scale for children (4th ed.). San Antonio, TX: The Psychological Corporation.

CURRICULUM-BASED MEASUREMENT FOR BEGINNING WRITERS: RECENT DEVELOPMENTS AND FUTURE DIRECTIONS Kristen L. McMaster, Kristen D. Ritchey and Erica Lembke ABSTRACT Many students with learning disabilities (LD) experience significant difficulties in developing writing proficiency. Early identification and intervention can prevent long-term writing problems. Early identification and intervention require reliable and valid writing assessments that can be used to identify students at risk and monitor their progress in response to intervention. One promising approach to assessing students’ performance and progress in writing is Curriculum-Based Measurement (CBM). In this chapter, we provide an overview of CBM. Next, we describe a theoretical framework for writing development, and discuss implications of this framework for developing writing assessments. We then describe current efforts to develop a seamless and flexible approach to monitoring student progress in writing in the early elementary grades, and highlight Assessment and Intervention Advances in Learning and Behavioral Disabilities, Volume 24, 111–148 Copyright r 2011 by Emerald Group Publishing Limited All rights of reproduction in any form reserved ISSN: 0735-004X/doi:10.1108/S0735-004X(2011)0000024008

111

112

KRISTEN L. MCMASTER ET AL.

important directions for future research. We end with a discussion of how teachers might eventually use CBM to make data-based decisions to provide effective individualized interventions for students who experience writing difficulties.

Writing is a critical skill for students to develop during their school years. Learning to write plays an important role in overall literacy development (Shanahan, 2006), and is an essential tool for students to communicate what they know (Graham & Perin, 2007) via written assignments, assessments, and essays or reports. Further, because writing is often used to synthesize and evaluate information, it may have an important impact on students’ development of content knowledge, including their ability to integrate knowledge and think critically (Bangert-Drowns, Hurley, & Wilkinson, 2004; Shanahan, 2004). In other words, not only must students learn to write, but writing also plays an important role in learning. Despite its importance, writing has not received a great deal of attention in educational research or practice (Berninger et al., 2006b), especially compared to reading and mathematics (Shanahan, 2006). National statistics appear to reflect this lack of attention: The latest Nation’s Report Card revealed that, in 2002, 72% of fourth graders, 69% of eighth graders, and 77% of twelfth graders were below proficient in writing (Persky, Daane, & Jin, 2003); and in 2007, 65% of 8th graders and 75% of 12th graders were performing below proficient (Salahu-Din, Persky, & Miller, 2008). Although these data suggest a small amount of improvement, proportions of students performing below proficient in writing remain unacceptable. Among those students not reaching proficient levels in writing are many students with learning disabilities (LD; Graham, Harris, & Larsen, 2001). For example, Salahu-Din et al. (2008) found that only 6% of eighth graders with LD reached the proficient level in writing; 46% were below the basic level. The writing of students with LD is often characterized by difficulties with handwriting, spelling, and mechanics, as well as by fewer ideas; poorer organization, planning, and revising; and lower quality than that of their peers without LD (Graham & Harris, 2002; Troia, 2006). These writing problems often go undetected until the late-elementary or middle-school grades, when they become increasingly difficult to remediate (Graham & Harris, 2005). Persistent writing difficulties can have long-term, negative consequences. They can affect students’ overall academic performance, diminish their chances of attending college, and limit employment opportunities (Graham & Perin, 2007). Early identification and intervention

113

CBM for Beginning Writers

with students at risk for serious writing difficulties or disabilities could prevent such consequences for many students (Berninger, Nielsen, Abbott, Wijsman, & Raskind, 2008; Berninger et al., 2006b). Early identification and intervention require reliable and valid assessment tools that can be used to (a) identify students in need of early writing intervention, (b) monitor students’ performance and progress during intervention, and (c) evaluate the effectiveness of interventions and make data-based decisions regarding the appropriateness of instruction. One assessment approach that shows promise as a tool for such data-based instructional decision making is Curriculum-Based Measurement (CBM; Deno, 1985). Over 30 years of research has been conducted on CBM in core academic areas (reading, mathematics, spelling, written expression, and content areas; see Foegen, Jiban, & Deno, 2007; McMaster & Espin, 2007; Wayman, Wallace, Wiley, Ticha, & Espin, 2007 for reviews). In recent years, more work has been conducted to develop and examine CBM for beginning writers, including research by the coauthors of this chapter. The purpose of this chapter is to describe recent advances in CBM for beginning writers, as well as to identify future directions for this work. First, we provide an overview of CBM. Then, we describe recent developments in CBM for beginning writers, starting with the theoretical framework in which these measures are grounded, followed by current efforts to develop tasks that yield reliable, valid, and instructionally useful progress-monitoring data. Throughout the chapter, we indicate areas in need of further research. We end with a case example of how teachers might use CBM to make data-based decisions to provide effective interventions for students who experience writing difficulties.

CBM OVERVIEW CBM was developed by Deno and colleagues (e.g., Deno, 1985; Deno, Mirkin, & Marston, 1982) as part of the Institute for Research on Learning Disabilities (IRLD) at the University of Minnesota to be a set of simple, efficient procedures that would provide general outcome indicators of student performance and progress. CBM was designed to reflect vital signs of academic health; that is, to indicate whether a student was making sufficient progress toward important academic goals, or whether progress was insufficient, indicating an underlying problem requiring further diagnosis and intervention (Deno, 1985). To provide general outcome

114


indicators of students’ performance and progress in a given academic area, Deno and colleagues sought measurement approaches that involved direct observation of academic behaviors. To be useful for ongoing instructional decision making, they sought to develop measures drawn directly from the curriculum in which a student’s progress was to be monitored, and that were reliable and valid, simple and efficient, easy to understand, and inexpensive (Deno). For example, in reading, the most common CBM approach involves listening to a student read aloud from grade-level text while the number of words read correctly in 1 min is recorded. Repeated (e.g., weekly) assessments using alternate passages of equivalent difficulty are administered to provide a picture of the student’s progress within the reading curriculum over time. Elementary students’ scores on CBM read-aloud measures have correlated strongly with standardized reading measures (see Wayman et al., 2007), suggesting that the CBM read-aloud approach provides the type of general outcome indicator that Deno and his colleagues were seeking. Further, the measures have been found to distinguish among students of different skill levels, and to be sensitive to growth made in brief time periods (Wayman et al., 2007). Similar approaches are used in mathematics (e.g., students complete basic math facts or math problems; see Foegen et al., 2007), spelling (students spell words that represent the corpus of words in the spelling curriculum; Fuchs, Fuchs, Hamlett, & Allinder, 1991a, 1991b), and written expression (students write in response to a story or picture prompt; see McMaster & Espin, 2007). CBM for writing (CBM-W) research initially involved story or picture prompts (e.g., Deno, Mirkin, & Marston, 1982); students provided a written response to the prompts for 3–5 min, and the prompts were then scored for number of words written (WW), words spelled correctly (WSC), and correct word sequences (CWS, which accounts for both spelling and grammar; Videen, Deno, & Marston, 1982). These measures served as reliable and valid indicators of student writing proficiency for upper-elementary students, and were relatively simple and efficient to administer and score; however, the measures yielded somewhat weak reliability (rs ¼ .20–.47; Deno et al., 1982), and weak to moderate criterion validity (rs ¼ .23–.63; Gansle et al., 2004; Jewell & Malecki, 2005; Parker, Tindal, & Hasbrouck, 1991) for early elementary students. Thus, our collective research teams have begun to explore CBM approaches that might serve as better indicators of young children’s developing writing skills. A search for general outcome indicators of early writing proficiency begs the question: what constitutes early writing? Theoretical models of writing development have provided guidance in

115


determining how we might gauge the vital signs of students’ early writing proficiency.

THEORETICAL FRAMEWORK FOR DEVELOPING CBM FOR BEGINNING WRITERS Skilled writers engage in multiple processes while writing, and for young writers, these processes typically begin to develop in early childhood, and continue to develop during the early elementary grades. Theoretical models of early writing developed by Bereiter and Scardamalia (1987) and by Berninger and colleagues (Berninger, 2000; Berninger, Fuller, & Whitaker, 1996; Berninger & Swanson, 1994) account for children’s growing writing proficiency. Berninger et al. (2002) organized these factors into a ‘‘simple view’’ of writing, in which transcription (handwriting or keyboarding, spelling) provides a foundation for early writing development, allowing the writer to translate ideas into text; and executive functions (conscious attention, planning, reviewing, revising, and self-regulation) serve to manage the writing process, all within the constraints of short-term, working, and long-term memory. The design of early writing assessments requires sensitivity and attention to the developmental challenges faced by young writers. For young writers, transcription and ideation are the most relevant factors influencing the text produced by children. The process our respective research teams have used in the development of CBM tasks and scoring procedures for beginning writers are informed by these aspects of writing development.

Transcription Transcription is the process of getting words onto paper, and handwriting (or keyboarding) and spelling are components of transcription. Young writers tend to focus on producing smaller units, such as forming individual letters and spelling individual words. As students’ transcription skills develop and become automatic, working memory capacity can then be devoted to higher-level processes such as composition and self-regulation (McCutchen, 2000), allowing students to engage more fully in planning and revising (Berninger & Swanson, 1994; Berninger et al., 2002). Students are then able to produce text that is longer and more complex and begin

116


transition from writing individual words to clauses, sentences, and longer texts. Transcription skills are important to writing quality. For example, Graham, Berninger, Abbott, Abbott, and Whitaker (1997) found that handwriting fluency and spelling explained a large proportion of the variance in both the length and quality of children’s writing; and Graham, Berninger, Weintraub, and Schafer (1998) found that copying tasks discriminate across a wide age range (grades 1–9) in terms of handwriting speed and legibility. Abbott, Berninger, and Fayol (2010) found that spelling is a strong predictor of paragraph writing in grades 1–7. Further, experimental and quasi-experimental studies have shown improvements in students’ writing quality in response to interventions targeting handwriting and spelling (Berninger et al., 1997; Berninger et al., 1998; Graham, Harris, & Fink, 2000; Graham, Harris, & Fink-Chorzempa, 2002; Jones & Christensen, 1999), suggesting a causal relation between development of transcription skills and overall writing proficiency. As described in more detail below, our respective research teams have developed CBM tasks for beginning writers that are intended to tap transcription skills, with the hypothesis that such measures would serve as general outcome indicators of children’s developing writing proficiency. Such measures have included dictated letter- and sound-writing tasks for very young children (Ritchey, 2004, 2006), dictated spelling tasks (e.g., Ritchey, Coker, & McCraw, 2010; Lembke, Deno, & Hall, 2003) and copying tasks (e.g., Lembke et al., 2003; McMaster, Du, & Petursdottir, 2009) for children as they progress through the early elementary grades.

Ideation Ideation is the process of generating ideas for written content, and draws on oral language sources such as vocabulary and domain knowledge. Juel, Griffith, and Gough (1986) reported that the number of generated ideas was a unique predictor of first and second graders’ writing quality. Coker (2006) found that receptive vocabulary proficiency predicted end-of-the-year writing quality for first-grade students. Written vocabulary also predicted writing quality for second- and fourth-grade students (Olinghouse & Leaird, 2009). Although there is limited research on the relation between ideation and early writing, assessments that do not attempt to capture ideation may miss a critical feature of written development.


117

Thus, our research teams have developed CBM tasks that require students to generate their own ideas. Given that young writers’ executive functions (e.g., conscious attention, planning, reviewing, revising, and self-regulation) are in development and are likely to be dependent on outside guidance and assistance (Berninger & Amtmann, 2003), we have added varying levels of scaffolding to these tasks, by creating more simple tasks (e.g., writing words or sentences rather than entire passages) and prompts (including letters, words, and pictures) intended to help young writers generate words and ideas to include in their writing. These measures have included Letter prompts to elicit word-level writing (McMaster et al., 2009), Picture-Word prompts (McMaster et al.) and Sentence-Writing prompts (Coker & Ritchey, 2010) to elicit sentence-level writing, and Picture prompts presented in a theme (McMaster et al., 2009) or a story sequence (Ritchey & Coker, in press) to elicit discourse-level (story) writing.

RECENT DEVELOPMENTS IN CBM FOR BEGINNING WRITERS Researchers (Fuchs, 2004; Wallace, Espin, McMaster, & Deno, 2007) have recently articulated important directions for programmatic CBM research. First, in a summary of the past, present, and future of CBM research, Fuchs outlined three stages needed to establish the utility of CBM for progress monitoring: Stage 1 involves examining technical features of the static score (i.e., a score obtained at a single time point), Stage 2 involves examining technical features of slope (i.e., rate of growth across time), and Stage 3 involves examining instructional utility. Fuchs indicated that, over the past 30 years, CBM researchers have addressed each of these three stages to varying degrees in reading, writing, spelling, and math, but that a disproportionate amount of this research has focused on Stage 1. Whereas Stage 1 is a critical first step in validating CBM, and is useful for identifying reliable and valid measures for screening purposes, Fuchs encouraged CBM researchers to ‘‘capitalize on CBM’s unique contribution for modeling progress and enhancing instruction’’ (p. 191) by focusing further attention on Stages 2 and 3. Second, in an introduction to a special issue of The Journal of Special Education featuring reviews of CBM research in reading, mathematics, and writing, Wallace et al. (2007) presented their vision of a ‘‘seamless and flexible system of progress monitoring’’ (p. 66) that could be used with

118


students across varying ages and skill levels, as well as across settings and curricula. Such a system of progress monitoring would be reliable and valid for indexing student performance, progress, and response to instruction both within and across grades. We use Fuchs’s (2004) three stages to summarize the work conducted thus far in CBM for beginning writers. We use Wallace et al.’s (2007) vision of a seamless and flexible system of progress monitoring to reflect upon what we currently know about CBM for beginning writers and to discuss future directions for a coherent program of research in this area. A summary of CBM tasks and scoring procedures, along with the stages and grade levels at which this research has been conducted, are presented in Table 1. For Stage 1 research, an ‘‘R’’ denotes measures with sufficient reliability (rW.70) and a ‘‘V’’ denotes measures with sufficient criterion-related validity (rW.50). Because there is not a consensus on criteria by which to judge reliability of measures, we based our judgment of sufficient reliability and validity on other standardized writing measures, for which reliability estimates have typically ranged from r ¼ .70 to above .90 and criterion-related validity coefficients have rarely exceeded r ¼ .60 (Taylor, 2003).

Stage 1 Research Much of our collective work thus far has focused on technical features of the static score, an important first step to establishing approaches that show promise as general outcome indicators of early writing proficiency (Fuchs, 2004). Ritchey and Coker have systematically examined measures developed at the subword, word, sentence, and discourse levels, in line with a perspective of how young children’s writing skills develop beginning in kindergarten and proceeding through early elementary grades (through grade 3). Lembke and McMaster and their colleagues have taken a more comparative approach, by administering tasks at word, sentence, and discourse levels and comparing their technical characteristics, primarily for students in grades 1 and 2. Subword-Level Measure Development Ritchey (2004, 2006) designed two CBM tasks to target writing skills at the earliest stages of writing development. The first task, Letter Writing, assesses students’ ability to write individual upper- and lower-case letters from dictation. Sixty kindergarten children were assessed in March

T&I

T&I

Picture-worde,f

T

T

Sentence writingg

Sentence level Sentence dictationc,d Sentence copyingc,d,e,f

T&I

3 min; WW, WSC, CWS; qualitative 3–5 min; CLS, WW, WSC, CWS

3 min; WW, WSC, CWS, C-IWS 3–5 min; CLS, WW, WSC, CWS

T

Word copying c,d,e

Letter prompt

N.T.; CLS 3 min; CLS, C-ILS, WW, WSC 2–3 min; CLS, C-ILS, WW, WSC 3–5 min; CLS, WW, WSC

T T

e

N.T.; CLS

N.T.; CL N.T.; CL

Duration and Scoring

T

T T

Domain

R

R

R&V

R&V R&V

K

R&V

R&V

R&V

R&V

R

R&V

R&V

Grade 1

V

V

V

V

Grade 2 Grade 3

Stage 1: Technical Features of the Static Score

Bi-monthly growthW0

K

Weekly growthW0 Reliable slopes; Weekly growthW0 Bi-monthly growthW0 Reliable slopes; Weekly growthW0

Weekly growthW0 Weekly growthW0

Grade 1

Grade 2

Stage 2: Technical Features of the Slope

Grade 3

Summary of CBM Tasks and Scoring Procedures for Beginning Writers by Stage of Research and Grade Level.

Word level Real word spellingb Nonsense wordb Word dictationc,d

Subword level Letter writinga Sound spellinga

CBM Task

Table 1.

CBM for Beginning Writers 119

T&I

3 min; WW, WSC, CWS 3–5 min; CLS, WW, WSC, CWS 5 min; WW, WSC, CWS, qualitative

3–5 min; WW, WSC, CWS

Duration and Scoring K

(Insufficient R & V)

R&V

R&V

Grade 1

R

R&V

R&V

R&V

Grade 2 Grade 3

Stage 1: Technical Features of the Static Score K

Reliable slopes; weekly growthW0

Grade 1

Bimonthly growthW0 for CWS and qualitative scoring only

Grade 2

Grade 3

Bimonthly growthW0 for all scores

Stage 2: Technical Features of the Slope

Note. T ¼ transcription, I ¼ ideation, CL ¼ correct letters, CLS ¼ correct letter sequences, C-ILS ¼ correct minus incorrect letter sequences, WW ¼ words written, WSC ¼ words spelled correctly, CWS ¼ correct word sequences, C-IWS ¼ correct minus incorrect word sequences, R ¼ sufficient reliability (rW0.70), V ¼ sufficient validity (rW.50). a Ritchey (2004, 2006). b Ritchey (2006), Ritchey et al. (2010). c Lembke et al. (2003). d Hampton et al. (2011). e McMaster et al. (2009). f McMaster et al. (2011). g Coker and Ritchey (2010). h McMaster and Campbell (2008). i Ritchey and Coker (in press).

Picture storyi

T&I

T&I

Photo prompte,h

Picture-theme

T&I

Discourse level Story prompte,f,h

e

Domain

CBM Task

Table 1. (Continued ) 120 KRISTEN L. MCMASTER ET AL.


121

and May. Reliability coefficients were r ¼ .90 for both test–retest and alternate-form reliability (at an eight-week interval), split-half reliability coefficients were r ¼ .90 and .92 (for March and May assessments, respectively), and internal consistency reliability was .94 (coefficient alpha for both March and May assessment points). Scores on Letter Writing were positively related to early reading and writing skills, including performance on norm-referenced and CBM-Reading assessments. Criterion-related validity coefficients with reading ranged from r ¼ .50 to .69. Criterionrelated validity coefficients with a second sample of kindergarten children (N ¼ 86) were investigated using the Test of Early Written Language (TEWL; Hresko, Herron, & Peak, 1996). The validity coefficient was r ¼ .67 with the Basic Writing subtest. The second subword-level task is Sound Spelling. This task requires children to spell individual letters after being presented with a sound (e.g., write the letter that makes the /t/ sound). In this way, basic encoding can be assessed at a stage when students have emerging, but perhaps incomplete, word-spelling skills. The technical features of Sound Spelling were investigated in the same two samples of children as Letter Writing. For Sound Spelling, the reliability coefficients were r ¼ .90 for test–retest and alternate-form reliability (at eight weeks), split-half reliability coefficients were r ¼ .92 and .91 (for March and May assessments, respectively), and internal consistency was .91 and .90 (for March and May assessments, respectively). Criterion-related validity coefficients with reading ranged from r ¼ .58 to .77. The criterion-related validity estimate with the TEWL Basic Writing subtest (Hresko et al., 1996) was r ¼ .67. The letter-writing tasks described above can be used to assess early writing development; however, the seamlessness of these measures within an ideal assessment framework may be limited due to ceiling effects once children learn to write all of their letters. Initial study of these measures did not include a timed component; yet, a timed component may help to eliminate ceiling effects and to be a more sensitive indicator of writing growth over time. Word-Level Measure Development Given the empirical relation between transcription skills and writing composition described above, it seems reasonable to expect that spelling measures might provide an indicator of children’s early writing development, particularly as they reach the ceiling on letter- and sound-writing tasks. Specifically, word-level spelling tasks may be used to assess the extent to which students are beginning to use letter-sound associations, combined

122


with phonological segmentation skills, to phonetically recode words. Word-level tasks may be helpful to use when assessing students during the developmental transition from writing individual letters to writing phrases and sentences. Traditional CBM spelling measures have assessed global outcomes of spelling, primarily across upper-elementary grade levels (e.g., Fuchs et al., 1991a, 1991b) with little work at first grade or below. A question remains regarding which particular words or word types might be used as indicators of writing skill. Two CBM spelling tasks have been developed for students in kindergarten to investigate early spelling skills limited to basic words following consonant–vowel–consonant patterns (c–v–c). One task, Real Word Spelling, requires spelling of real words (e.g., mad) while Nonsense Word Spelling requires spelling of phonetically regular pseudo words (e.g., sud). The spelling tasks have five items for children in kindergarten and include one word with each short vowel sound. Initial study focused on the relation between these tasks and students’ literacy development during the second half of kindergarten (Ritchey, 2006). Students’ responses were scored using correct letter sequences (CLS), an index of the number of correct adjacent letters (White & Haring, 1980). For Real Word Spelling, the reliability coefficients ranged from .84 to .90 (alternate forms and internal consistency). Nonsense Word Spelling yielded similar estimates with a range of .86–.89 for reliability and 92% for interrater agreement. The tasks were related to measures of reading with coefficients ranging from r ¼ .40 to. 50. The correlation between Real Word Spelling and Nonsense Word Spelling was r ¼ .92, suggesting that the tasks are capturing similar aspects of early writing skill and that one task, rather than both, may be sufficient to assess students’ spelling knowledge development. Ritchey et al. (2010) compared scoring procedures for the Real Word Spelling task with a different sample of kindergarten children. CLS was compared to a scoring procedure that involves counting the number of correct sounds, a phonological scoring system developed by Tangel and Blachman (1992), and a correct/incorrect scoring system. All scores were related to reading and writing outcomes with criterion-related validity coefficients around r ¼ .50 with the TEWL Basic Writing subtest (Hresko et al., 1996). All scores were also sensitive to bimonthly growth, with the largest slope estimates for the phonological scoring metric. These findings are tempered by applicability to a range of spelling words; different scoring metrics may be more appropriate for children at later stages of spelling and writing development.


123

Sentence-Level Measure Development Coker and Ritchey (2010) developed and evaluated a Sentence-Writing CBM task that could be used with children during the second half of kindergarten and first grade (see Fig. 1 for a sample). Students were given a prompt and asked to write about the topic (e.g., write about your favorite food for lunch). Two prompts were administered, and students were provided with 3 min to respond to each prompt. Kindergarten and firstgrade students (N ¼ 233) were administered Sentence Writing three times, at two-month intervals. Coker and Ritchey examined three types of scores: WW, WSC, and CWS. Alternate-form reliability was approximately r ¼ .80. Different criterion measures were used for kindergarten and first-grade students given differences in writing development at these levels. The TEWL (Hresko et al., 1996) was administered to kindergarten students, and the Woodcock–Johnson III (WJ-III) Spelling and Writing Samples subtests (Woodcock, McGrew, Schrank, & Mather, 2007) were administered to firstgrade students. For kindergarten scores, the criterion-related validity coefficients were strongest for WW (r ¼ .36), and for first grade, were strongest for CWS (r ¼ .55). Coker and Ritchey (2010) also investigated a qualitative scoring procedure to address limitations associated with scores that focus solely on production, to link the score more directly to instructional goals and

Fig. 1. Sample Response to Sentence Writing. In This Sentence, There are 16 WW, 14 Words Spelled Correctly, and 15 Correct Word Sequences. Using the Qualitative Scoring System, this Response Earns a Score of 11 (3/3 Points for Multiple Sentences, 2/3 Points because More than Half of the Words are Spelled Correctly, 3/3 Points for Complete Use of Capital Letters/Punctuation, 3/3 Points for Grammatical Structure, and 3/3 Points for a Response that is Linked to the Prompt and Includes an Elaboration. The Complete Rating Scale for Sentence Writing can be Found in Coker and Ritchey (2010).

124


objectives, and to address the inability of young students to produce large amounts of text. The qualitative score was a composite of ratings of five unique components of writing: response type (if response was a word or words, complete sentences, or multiple sentences), spelling, mechanics (capitalization and punctuation), grammatical structure, and relationship to prompt (the extent to which a student’s response was related to the topic presented in the prompt). The qualitative scoring rubric can be found in Coker and Ritchey’s (2010) article. The qualitative score had internal consistency of approximately .80, and concurrent criterion-related validity coefficients were approximately r ¼ .40 for kindergarten and r ¼ .50 for firstgrade students. Sentence-level writing has potential as an assessment that would provide a seamless transition from word-level skills (spelling) to composition of longer texts. Additional research is needed to determine at what point in children’s writing development this link should be made, as well as to determine the extent to which qualitative scores add instructionally useful information. Discourse-Level Measures As described earlier, traditional CBM-W tasks have involved discourse-level composition, whereby students respond to a writing prompt within a fixed time period (e.g., 3–5 min; Deno et al., 1982). In an attempt to examine the extent to which these traditional CBM-W tasks could be used seamlessly across students at different grade levels, McMaster and Campbell (2008) examined whether different types of discourse-level measures (e.g., Narrative, Expository, and Phot prompts) used with older students (fifth and seventh graders) could be extended downward to younger elementary students (third graders). They found that 3-min Photo prompts (which prompted students to write about a photograph of children doing schoolrelated activities) yielded sufficient reliability (rsW.70) and validity (rsW.50) coefficients within grade 3, and that 5-min narrative Story prompts yielded sufficient reliability and validity coefficients both within and across grade levels. A next step, then, is to determine whether such tasks can be extended to even earlier grade levels, or whether different tasks are needed to gauge young students’ writing at the discourse level. Ritchey and Coker (in press) investigated young students’ compositional skill at the discourse level by comparing the traditional CBM-W Story prompt to a new story-writing task based on a picture prompt (Picture Story; see Fig. 2). In the Picture Story task, students were given a three-step picture sequence (e.g., a boy brushing his teeth, a girl planting a garden) and given 5 min to write a story based on the pictures. The two story-writing


125

Fig. 2. Sample Response to Picture Story that Includes a Three-Picture Sequence of a Child Making a Peanut Butter and Jelly Sandwich. In this Response there are 31 WW, 27 Words Spelled Correctly, and 26 Correct Word Sequences. Using the Qualitative Scoring System, this Response Earns a Score of 14 (3/3 points for Including all Three Pictures, 3/3 Points Because the Response is More than an Elaborated Description and Includes a Character, 2/3 Points for Including More than One Transition Word, 0/3 Points for Including No Descriptive Words, 2/3 Points for Grammatical Structure because there are No Errors that Impact Understanding but the Response Includes Run-On Sentences, 2/3 Points for Including Capitalization/Punctuation, and 2/3 Points for Spelling Between 70 and 90% of the Word Correctly. The Complete Rating Scale for Picture Story can be Found in Ritchey and Coker (in press).

tasks were compared to determine to what degree, if any, providing scaffolding in the form of topical support could capture aspects of ideation. Second- and third-grade students wrote responses to both tasks (Story prompt and Picture Story) at three times points in the spring of the school year. Responses were scored using production scoring procedures (WW, WSC, CWS) and using a qualitative scoring procedure that captured aspects of written content (number of picture events included in the response), response quality (whether the response was a simple description, an elaborated description, or a story with narrative elements), use of descriptive words, inclusion of transition words, appropriate grammatical structure, mechanics, and spelling (percentage of WSC). The qualitative score was designed in the same manner as the qualitative score for Sentence Writing, and expanded to assess more developed writing abilities, including ideation. The internal consistency of the qualitative score ranged from .77 to .80. For both grades combined, the validity coefficients for Picture Story and production scores ranged from r ¼ .37 to .46 and from r ¼ .31 to .42 for the Story prompt using WJ-III Writing Samples (Woodcock et al., 2007) as the criterion. The validity coefficient for the qualitative score for Picture Story

126


was r ¼ .44. Whereas these coefficients are not as strong as those reported in other studies of CBM-W, the use of the Writing Samples subtest as the criterion may explain this differences because Writing Samples assesses limited aspects of narrative writing. Another extension of Stage 1 research efforts is to determine the accuracy with which students can be screened and identified as at risk (or not at risk) for writing problems. To do so, students are given a screening test and their performance is compared to performance on an outcome measure. For Story prompt and Picture Story, performance in the lowest quartile (below the 25th percentile) was compared to later performance at the lowest quartile (below the 25th percentile) on WJ-III (Woodcock et al., 2001, 2007) Writing Samples or a rating by their classroom teacher as being below grade level. The CWS score for both tasks provided the highest area under the curve (AUC1). AUC ranged from .573 to .852. Sensitivity and specificity indices were comparable for most scores, identifying a range of 25–100% of at-risk writers (as defined by WJ-III Writing Samples) and 59–78% of not-at-risk writers. Using the teacher rating to determine poor writer status, sensitivity was also variable (28–83% range) and specificity ranged from 62% to 85%. Of note, neither of the measures approached the levels of classification accuracy that are comparable to CBM in reading, in which sensitivity and specificity have been closer to 90% (Jenkins, Hudson, & Johnson, 2007).

Comparisons of Measures at Word, Sentence, and Discourse Levels Ritchey and Coker’s work sheds important light on a developmental progression of measures that can be used with children beginning in kindergarten and proceeding through the primary grades. Lembke, McMaster, and their colleagues have taken a somewhat different approach, by comparing different types of tasks at word, sentence, and discourse levels within individual studies. Because much of this work has been done simultaneous to Ritchey and Coker’s work described above, there is some variation in the tasks examined. Lembke et al.’s (2003) early work in identifying CBM tasks for beginning writers came about because of a need to find measures that could be used for students in early grades who could not yet brainstorm a topic and compose a story from a Story prompt. Lembke et al. administered Word and Sentence Copying and Word and Sentence Dictation tasks (see Figs. 3 and 4 for examples) to 15 at-risk second graders enrolled in a summer school program, along with other measures of writing to assess criterion validity. At the beginning and end of summer school students completed four writing


127

Fig. 3. Sample Response to the Word Copying Task (Lembke et al., 2003; Hampton et al., 2011), with Number of Letter Sequences Correct Out of Total Possible.

tasks that were administered for 2–3 min each: Word and Sentence Copying and Word and Sentence Dictation. The Word Copying task was administered for 2 min and consisted of students copying printed words onto lines below the words. Sentence Copying was similar to Word Copying, except that the students were

128


Fig. 4. Sample Student Response to the Sentence Dictation Task (Lembke et al., 2003; Hampton et al., 2011), With Number of Correct-Word Sequences Out of Total Possible. Example: Birds can fly. 1. The old boat sits in the sun. 2. The party was a surprise for daddy. 3. The black bear came to my house. 4. Father gave me a ball. 5. I look out my window.


129

prompted to copy sentences that were five to seven words in length on a line below each sentence for 3 min. For the Dictation tasks, administered for 3 min each, words or sentences were dictated twice; the student wrote the word or sentence after listening to the dictation. The student was provided the next word or sentence when he/she had paused for 5 s or had completed the previous item. Responses were scored for CLS, WW, WSC, and CWS (sentence-level measures only), and then correlated with ‘‘atomistic’’ (average WW, WSC, CWS, and correct minus incorrect word sequences (C-IWS) obtained from writing samples in response to Story prompts) and ‘‘holistic’’ (teachers’ global ratings of writing samples) variables. Lembke et al. (2003) determined that there was a wide dispersion of scores for each measure (as indicated by large standard deviations), and no floor or ceiling effects. Dictation measures yielded strong validity correlations with the atomistic criteria, particularly when scored using CLS, WSC, and CWS (rZ.80 for 22 out of 32 correlations across these scoring methods for the Dictation measures). Criterion validity coefficients were also strong for the Dictation measures scored using CLS, WSC, and CWS when compared to the more holistic criteria (teacher ratings and independent scoring of samples), although very few correlations were in the moderate to strong range (rZ.70 for 8 out of 32 correlations). In a second study completed by Lembke and her colleagues, Hampton, Lembke, and Summers (2011) examined the use of both Word and Sentence Copying and Dictation measures from the Lembke et al. (2003) study with 23 first graders. Each measure was administered for 3 min each, every week for eight weeks. Measures were scored using CLS and CWS, as well as correct minus incorrect letter sequences (C-ILS) and C-IWS. Alternateform reliability was strongest for Word Copying and Sentence Dictation (r ¼ .71–.98), and criterion validity with the TEWL (Hresko et al., 1996) was strongest for Word Dictation (r ¼ .52–.61). When comparing results between the Lembke et al. (2003) and the Hampton et al. (2011) studies, it is apparent that both Word and Sentence Dictation tasks function well for both first- and second-grade students. Moderate but significant criterion validity coefficients were also obtained for Sentence Copying with the first-grade students. Criterion validity coefficients were not as strong for first-grade students, with most correlations in the .50–.60 range, when compared to those for second-grade students, with most correlations in the .75–.85 range. These differences could be attributed to different criterion measures used for the two studies (TEWL vs. scoredwriting samples), although the administration and scoring for the TEWL does not differ substantially from the scoring of the writing samples.

130


Word-, Sentence-, and Discourse-Level Tasks In another set of studies, McMaster et al. (2009) compared traditional CBM-W prompts to word- and sentence-level copying tasks (designed to tap transcription skills), as well as to tasks that required word and sentence generation (designed to tap higher-level processes such as ideation; see Berninger et al., 1992). New tasks included Letter prompts (students were prompted to write words starting with letters provided), Picture-Word prompts (students were prompted to write sentences that included words provided along with a picture depicting the word), and Picture-Theme prompts (students were prompted to write stories that included a bank of four words paired with pictures related to a theme, such as birthday party). These tasks were also compared to CBM-W tasks typically used with older elementary students (Story and Photo prompts). See Figs. 5 and 6 for sample Picture-Word and Story prompts.

Fig. 5. Sample Student Response to the Picture-Word Task. In this Example, Underlined Words are Incorrectly Spelled, Upward-Pointing Carets (4) Denote Correct Word Sequences, and Downward-Pointing Carets (3) Denote Incorrect Word Sequences. There are 14 Words Written, 13 Words Spelled Correctly, and 15 Correct Word Sequences.


131

Fig. 6. Sample Student Response to a Story Prompt. In this Example, Underlined Words are Incorrectly Spelled, Upward-Pointing Carets (4) Denote Correct Word Sequences, and Downward-Pointing Carets (3) Denote Incorrect Word Sequences. There are 37 Words Written, 28 Words Spelled Correctly, and 19 Correct Word Sequences.

132


Fifty first graders responded to Word Copying, Sentence Copying, and Story prompts, and 50 responded to Letter, Picture-Word, Picture-Theme, and Photo prompts. Prompts were administered for 3- to 5-min in winter and spring and scored for CLS, WW, WSC, CWS, and C-IWS. Sentence Copying, Picture-Word, and Story prompts scored for CLS, WW, WSC, and CWS emerged as promising CBM tasks, with alternate-form reliability coefficients of rZ.70, and correlations with the Test of Written Language – 3 (TOWL-3; Hammill & Larsen, 1996), teacher ratings, and a district writing rubric of rZ.50. These findings were replicated for Sentence Copying, Picture-Word, and Story prompts in a second study with a new sample of 85 first graders (McMaster et al., 2011). Summary of Stage 1 Research and Future Directions The above-described research has yielded a range of CBM tasks and scoring procedures that appear promising for indexing beginning writers’ performance. Subword measures (Letter Writing and Sound Spelling) appear viable for assessing kindergartners’ early writing skills (Ritchey, 2004, 2006). Word-level measures (Ritchey’s spelling measures) appear viable for assessing kindergarten and first graders’ early writing skills. For kindergarten and first-grade students (at least in the second half of their respective grade levels), Sentence-Writing tasks appear viable (Coker & Ritchey, 2010). The technical adequacy of word- and sentence-level tasks of varying forms (those that require copying, dictation, or sentence generation) is also supported by Lembke et al. (2003), Hampton et al. (2011), and McMaster et al. (2009, 2011). Further, the technical adequacy of Story prompts traditionally used with older elementary students (e.g., McMaster & Campbell, 2008) appear to yield technically adequate scores for first graders (McMaster et al., 2009, 2011). This research provides an array of CBM options that may be used to gauge the writing proficiency of children across the early elementary grades. Yet, in the spirit of developing a seamless and flexible system of progress monitoring (Wallace et al., 2007), important ‘‘Stage 1’’ questions remain. Chief among these is: Of the array of available measures, is one (or more) of these options most appropriate as a general outcome indicator of children’s early writing proficiency within each grade level? Further, given that the measures examined thus far have yielded relatively modest criterion validity coefficients (which is typical of measures of writing; Taylor, 2003), presumably due to the difficulty in capturing this multidimensional process with just one measure (e.g., Parker et al., 1991), are there other approaches to assess writing that might yield scores with stronger validity? Some grade


133

levels (especially first grade) have received more focus than others, and it is important to continue to examine which measures are most reliable and valid for students at each grade level. Another important question is, can one (or more) of these measures be extended across students of different grade levels, such that a task could be administered at kindergarten and extended across first, second, and third grade to reflect students’ writing development across grade levels? Or is it possible to link measures that are more appropriate at the earliest stages of writing with measures that are appropriate at later stages (e.g., linking a subword-level measure in kindergarten to a word-level measure in early first grade to a discourselevel measure in later first grade and beyond)? Within the context of the questions raised above, other questions related to the measures’ technical characteristics need to be addressed with larger and more varied samples if they are to be used for data-based educational decisions. These include questions regarding the most appropriate scoring procedures and sample durations for students at different grade levels. The relation of measures to high-stakes outcomes in writing should also be examined. In some school districts, these high-stakes outcomes might be state tests while in other districts, outcomes might be district-rated writing samples or even just classroom-based writing expectations. Further, the classification accuracy of the various tasks must be determined if they are to be used to screen students and identify those who may need early writing intervention. Ritchey and Coker (in press) have begun to examine classification accuracy with the Story and Picture Story prompts; such examinations should be extended to other measures to determine which approaches are best for identifying students at risk for writing difficulties. In addition, normative data for students’ performance at different points in time (e.g., fall, winter, and spring) at each grade level would be useful for determining expected levels of performance and for setting benchmarks. For instructional decision making, further research is needed to expand the use of qualitative indicators of writing skills (such as those used by Coker & Ritchey, 2010; Ritchey & Coker, in press) in ways that are both efficient and effective in identifying students’ strengths and weaknesses that might be targeted in intervention. Stage 2 Research As Fuchs (2004) noted, establishing the technical adequacy of static scores is a necessary first step in determining the utility of CBM, but is insufficient if CBM is to be used to monitor student performance frequently over time. Thus, Fuchs indicated that another important focus for research is to examine

134


whether slopes derived from repeated CBM administration yield technically sound information. Thus far, only a few studies have been conducted to examine characteristics of slopes derived from CBM for beginning writers. Sensitivity to Growth in Short Time Periods. First, as part of the SentenceWriting study described above, Coker and Ritchey (2010) investigated whether WW, WSC, and CWS and qualitative scores from Sentence Writing were sensitive to growth made across three time points. Their analytic strategy was to determine if there was a significant rate of growth (i.e., slope) using growth curve modeling and if there were grade-level differences in end-of-year scores or slopes by grade (kindergarten and first grade). For all scores (WW, WSC, CWS, and the quality score) there were grade-level differences at the end of the school year: First-grade children wrote sentences that had significantly more words, WSC, CWS, and qualitative points than kindergarten children. As a group, students made significant rates of bimonthly growth (i.e., slopes were significantly greater than zero) on all scores, but writers in both grades had similar growth rates. The smallest slope estimate was a rate of increase of 1.10 CWS per two-month interval and the largest slope estimate was 2.03 quality points per twomonth time interval. A similar analysis was conducted with data from Ritchey and Coker’s (in press) study with second- and third-grade students using Picture Story and Story prompts. All scores were sensitive to grade-level differences, and third graders wrote stories with more words, more WSC, more CWS, and more qualitative points than second graders. All scores were also sensitive to growth at two-month intervals, and second- and third-grade students had the same mean rate of growth on all scores for Story prompts and CWS and the qualitative score for Picture Story. Only third-grade students made significant growth, and only on WW and WSC (second graders’ mean slopes were not significantly different from zero). One question raised by Coker and Ritchey’s (2010) Stage 2 research is whether CBM tasks administered on a weekly basis would be sensitive to growth in short time periods. Hampton et al. (2011) addressed this question by using Hierarchical Linear Growth Modeling (HLGM) to examine slopes from weekly CBM data generated from the word- and sentence-level dictation and copying tasks described earlier. They found that both wordand sentence-level dictation tasks and word- and sentence-level copying tasks had the capacity to show growth in short time periods, with significant growth characterized as slopes greater than 0. Weekly growth rates were greatest for the word-level measures (Copying and Dictation).


135

Reliability and Stability of Slopes, and Other Technical Questions. In another recent study of the capacity of CBM to assess weekly growth, McMaster et al. (2011) examined slopes from measures that emerged as promising in Stage 1 research described above (McMaster et al., 2009). The researchers compared sentence-level measures (Sentence Copying and Picture-Word) to discourse-level measures (Story prompts). Sentence-level prompts were administered weekly, and Story prompts every other week, to 85 first-grade students across 12 weeks. At the end of the 12 weeks, students completed the Story Construction subtest of the TOWL-3 (Hammill & Larsen, 1996). The researchers examined alternate-slope reliability (by comparing slopes from odd weeks with slopes from even weeks), incremental slope reliability (by comparing slopes from weeks 1 to 2, 1 to 3, 1 to 4 and so on with the overall 12-week slopes) and stability (by comparing standard errors across slopes of incremental lengths) of groupand individual-level slopes. They found that, for each task and scoring procedure, 8–9 weeks of data were needed to yield reliable and stable slopes. McMaster et al. (2011) also examined how many data points were needed to yield slopes significantly greater than zero (indicating which measures were sensitive to growth in brief time periods). The Picture-Word prompt appeared to be most sensitive to growth in brief periods, yielding slopes significantly greater than zero within four weeks of data collection. Parker, McMaster, Medhanie, and Silberglitt (2010) addressed other technical features of slopes from Picture-Word and Sentence Copying measures, including whether (a) early writing growth follows consistent trajectories across students, (b) growth is linear or nonlinear, (c) initial performance level has an impact on subsequent growth, and (d) growth differs by gender. Parker et al. applied linear-mixed modeling procedures to the dataset from McMaster et al. (2011). Results indicated that overall growth trajectories on both Picture-Word and Sentence Copying tasks required random-effect terms, suggesting that, whereas general trends in growth were observed across students, individual students had somewhat different trajectories. Moreover, growth trends for Sentence Copying tasks were linear, but trends for Picture-Word tasks were nonlinear, with a slight decelerating trend over time. Additional analyses indicated that neither initial performance level nor gender affected growth. Summary of Stage 2 Research and Future Directions. Although the studies described above provide insights regarding technical features of slopes derived from some CBM tasks for beginning writers, further Stage 2 research is needed. For example, preliminary evidence indicates that at least

136


8–9 data points are needed to obtain reliable and stable slopes on measures developed for first graders (McMaster et al., 2011), but more work is needed to determine which tasks and scoring procedures yield reliable and stable slopes both within and across early elementary grades, and under varying instructional conditions. In addition, we need to better understand how many – and how often – CBM writing data points should be collected to make accurate yet timely educational decisions (see Jenkins, Graff, & Miglioretti, 2009 for related work in the area of reading). Of particular concern in using progress-monitoring data for decisions within RTI is the amount of measurement error associated with estimating growth across brief intervals (e.g., Hintze, Shapiro, & Daly, 1998; Jenkins, Zumeta, Dupree, & Johnson, 2005); work is needed to determine whether and how this error can be minimized. Further research is also needed to better understand what early writing growth should look like. Preliminary evidence suggests we should expect individual differences in writing growth as measured by CBM, and that some tasks may show linear growth whereas others may show nonlinear growth (Parker et al., 2010). A critical question related to slope is: How much growth can we expect (cf. Fuchs, Fuchs, Hamlett, Walz, & Germann, 1993)? Of course, growth trajectories might vary at different points within and across school years and under different instructional conditions. A better understanding of what ‘‘average’’ writing growth looks like at the early grades will be important for making decisions about whether students are making sufficient writing progress in response to early writing instruction, as well as whether and when changes to that instruction should be made.

Stage 3 Research Clearly, continued research is needed at Stages 1 and 2; yet, to ‘‘capitalize on CBM’s unique contribution for y enhancing instruction’’ (Fuchs, 2004, p. 191) we must also begin to address Stage 3 questions. Specifically, when teachers use CBM to monitor the progress of beginning writers, does instructional decision making improve, and does improved instructional decision making lead to improved student writing outcomes? To our knowledge, no ‘‘Stage 3’’ studies have been conducted on CBM for beginning writers. We elaborate on the types of studies that could address instructional utility questions below.

137


FUTURE RESEARCH DIRECTIONS FOR CBM FOR BEGINNING WRITERS, AND IMPLICATIONS FOR PRACTICE As described in the opening of this chapter, writing is an important part of early literacy development, and early identification and intervention are critical for students at risk for writing difficulties and writing-related disabilities. Early identification and intervention require sound assessment procedures to identify students who might be at risk, as well as to track students’ progress in response to intervention. CBM tasks for beginning writers described in this chapter have been designed for teachers to screen and monitor the progress of students who might be at risk in early writing. We have highlighted important questions regarding technical features of scores and slopes derived from the various CBM tasks described under Stages 1 and 2 researches above. In addition to those questions, studying teachers’ use of these measures is an important next step if the measures are to be used for instructional decision making. Specifically, whereas it is important to develop and study measures in a controlled manner to establish their technical soundness, examining the feasibility and fidelity of teachers’ use of the measures in classroom contexts will help to bridge the ‘‘gap’’ between research and practice. Some important considerations regarding teacher implementation include: (1) Is scoring reliable? (2) Does time spent on administering and scoring the CBM tasks reduce the likelihood that teachers will use the measures consistently? (3) What impact would technological supports (e.g., computerized scoring, graphing, and data management tools) have on scoring reliability and feasibility of using the measures? (4) What types of decision-making rules will facilitate teachers’ consistent and effective use of CBM data to improve students’ early writing outcomes? (5) How can data collected during early writing interventions inform instruction? Each of these questions will be discussed in more detail, followed by a case example that integrates these issues in a reallife situation. Scoring Reliability Reliability of scoring for CBM-W measures is a concern, because CBM-W scoring procedures can be subjective and difficult to learn. One of the cornerstones of CBM is the ease and efficiency of administration and

138


scoring. In reading and mathematics, correct and incorrect answers are fairly easy to discern. In writing, scoring using the number of correct letter and words, or the number of correctly spelled words, is generally straightforward, although sometimes young children’s handwriting can be difficult to decipher. When scoring using CWS, discrimination of what is and what is not correct becomes more subjective. Future studies could include an examination of whether interrater reliability among teachers is similar to that obtained by researchers (in our research, we have generally aimed for interrater reliability of 90% or higher on WW and WSC, and at least 80% on more subjective measures like CWS). In practice, it is important to carefully examine and consistently follow scoring guidelines (see Videen et al., 1982, for example; scoring guidelines may also be obtained at www.progressmonitoring.org). Because it is possible that a scoring question may arise that is not addressed in published guidelines, we recommend that teachers document how that issue was handled and to consistently apply the solution each time that particular situation arises. In addition, if more than one person is scoring children’s writing samples within a school, it is important to assess reliability across scorers.

Time Spent Administering and Scoring CBM Tasks As mentioned, a key element of CBM is that the measures are efficient to administer and score (Deno, 1985). The CBM tasks for beginning writers described in this chapter are generally simple to administer, and many of them can be group-administered, which increases efficiency. Given that the duration for the measures ranges from 2 to 5 min, total administration time (including providing directions and practice) is typically around 10 min. Scoring can be time-consuming; yet, it is critical that prompts are scored in a timely manner if they are to be used for instructional decision making. Future research should include careful documentation of teachers’ time spent administering and scoring measures, including the extent to which they increase in their efficiency of completing these tasks over time. In addition, questions related to the frequency with which measures should be administered to obtain reliable data for instructional decision making may shed light on how teachers should allocate time to CBM administration and scoring. It is likely that teachers would administer the measures most frequently to students at risk or identified as having writing disabilities. In practice, teachers may wish to keep track of time spent administering and

139


scoring the CBM tasks, with the aim of improving their own efficiency (while at the same time maintaining accuracy).

Technological Supports Technology will likely play an increased role in teachers’ use of assessment tools and data (Fuchs, 2004). For example, it is possible that schools will allow (or require) children to use computers to produce writing samples. Research is needed to determine whether typewritten responses yield scores with similar technical characteristics as handwritten responses. In addition, technological tools such as computerized scoring programs are under development (e.g., Word Mark; Hooper, Miller, & Rose, 2010); word count and spell-checking functions in commonly available database software and can also be used to score for WW. In addition, web-based graphing and data-management programs that include writing measures (e.g., http:// www.aimsweb.com/; http://www.edcheckup.com/) are available. Future research could include an examination of the extent to which such tools increase the efficiency of teachers’ use of the measures. In practice, teachers will need to be aware of the range of options and capabilities of technological tools to determine which will be the most appropriate tool to meet their and their students’ needs.

Decision Rules Previous research has indicated that the availability of CBM data alone does not typically lead to changes in instruction or improved student outcomes (Stecker, Fuchs, & Fuchs, 2005). Rather, teachers require some level of guidance as to when an instructional change is needed. One form of such guidance is to provide teachers with decision rules that can be applied to the data they are collecting. Decision rules are typically applied to (a) setting a long-range goal for a student to meet in a given instructional period (e.g., the number of WSC that the student should be writing by the end of the school year) and (b) determining whether or not progress is sufficient to meet the goal. For example, once a student’s initial performance level is met (baseline), the teacher must decide on a reasonable but ambitious long-range goal, and draw a goal line from the baseline to the goal. The teacher then monitors the student’s progress; when a certain number of data points (such as the four-point rule described in the case example below), or the trend line

140


that represents the student’s progress, falls below the goal line, the teacher might decide to change instruction. When a certain number of data points or the trend line is above the goal line, the teacher might decide to raise the goal. Further research is needed to determine a range of reasonable expectations for beginning writers’ rate of growth to aid in decisions regarding appropriate goals to set. For example, if typical young writers grow at a rate of 1–2 WSC per week, educators could use this information to determine what might be an appropriate expectation for an individual student. Research is also needed to identify optimal decision rules, such that teachers make timely decisions without changing instruction prematurely (i.e., interventions may take several weeks to have an effect) but are also not prolonging ineffective interventions. In practice, teachers will need professional development regarding the available decision rules, including when and how to apply them appropriately and consistently.

Teachers’ Use of CBM Data to Improve Student Outcomes Importantly, research is also needed to determine if, when teachers use decision rules, they make effective instructional decisions, as evidenced by improved student writing outcomes. For example, researchers need to establish whether (and what types) of additional information might aid in the instructional decision-making process. Whereas CBM can indicate that an instructional change is needed, it may not be clear what type of instructional change would be most beneficial. Qualitative rubrics, such as that developed by Coker and Ritchey (2010), may aid in deciding what types of interventions to try. For example, qualitative examinations of a student’s writing might reveal that, whereas the student writes appropriate responses to given prompts, he or she struggles to spell the words correctly (or vice versa). The teacher might then select an intervention that addresses the specific area of weakness. Research is needed to determine the effect of using such information on teachers’ instructional decisions and student writing outcomes. Research should also examine whether additional supports are needed to ensure that teachers make effective instructional decisions (e.g., coaching from an expert, or sharing and discussing data with other teachers). In practice, when the data indicate that an intervention is needed, teachers should select from evidence-based early writing interventions in handwriting, spelling, and written expression. A variety of evidence-based early

141


writing interventions are available; for example, with respect to handwriting, intervention studies indicate that explicit instruction that incorporates modeling and extensive practice with feedback, goal-setting, and selfmonitoring leads to improved legibility and automaticity of young children’s handwriting, as well as improved fluency of composing text (e.g., Graham et al., 2000; Jones & Christensen, 1999). With respect to spelling, researchers have demonstrated that explicit instruction focusing on the alphabetic principle in isolation and in context can lead to significant improvement in composition length and spelling accuracy (Berninger et al., 2002; Berninger et al., 2006a; Graham et al., 2000). Evidence-based interventions also exist to support students’ written composition. One exemplary approach is Self-Regulated Strategy Development (SRSD), a written composition strategy developed by Graham, Harris, and colleagues. SRSD involves teaching students a strategy for planning, translating, and reviewing a written composition, along with a selfregulation process that enables students to monitor their own use of the strategy. SRSD was recently determined to meet all quality indicators and criteria to be considered an evidence-based practice (Baker, Chard, Ketterling-Geller, Apichatabutra, & Doabler, 2009).

CASE EXAMPLE Our discussion of current developments and future directions for CBM for beginning writers clearly illustrates that ongoing research is needed. Ultimately, the purpose for this research is to provide educators with reliable and valid tools to make instructional decisions that improve students’ early writing skills. To illustrate how these tools might eventually be used, we provide a case example of Ms. Taylor, a first-grade teacher, and her firstgrade team (which consists of four teachers total), who are preparing to implement CBM for beginning writers in their classrooms. It is important to note that the case example represents a vision of how CBM may be used, but that further research is needed to validate it for this purpose. Ms. Taylor and her first grade team attend a workshop to learn to administer, score, graph, and interpret results from CBM for beginning writers. They then meet in early fall of the school year to prepare the necessary CBM materials and to practice scoring. They each bring several samples of their students’ writing, copied for each teacher. Because they have learned that scoring using WSC has produced reliable and valid scores in research, they decide to begin with that scoring method. They each score

142


the same child’s writing sample using WSC and then compare their scores. They discuss any differences in scoring, referring to the scoring guidelines to resolve differences they encounter. They continue to score students’ samples and discuss scoring until they are within 90% agreement of each other, three times consecutively. They know that this takes time up-front, but understand that it is worthwhile to be consistent from the beginning, rather than having to backtrack and relearn scoring later in the year. As they score their second and third samples, the teachers observe that they are becoming more efficient in their scoring. They decide that they will keep a log of the amount of time it takes for administration and scoring of the measures, to keep track of how much instructional and preparation time CBM administration and scoring takes. Early in the school year, the first-grade teachers administer a Word Dictation (transcription) task and a sentence-writing (ideation) task to all students, given the reliability, validity, and capacity of these tasks to show growth. The teachers score the measures, and for 20% of the samples in each of their classes (about four students from each class), one other teacher scores as well, to make sure reliability of scoring is maintained. The teachers decide to monitor the progress of the lowest-scoring 25% of students from each class once per week. For this progress monitoring, they continue to use both the Word Dictation measure and Sentence Writing task. In addition to scoring for WSC, they use a qualitative scoring rubric to obtain additional information about students’ writing strengths and needs. The teachers decide to use a website from which they can access Word Dictation measures and which helps with data management. They enter their student scores each week and the data are graphed for them. They can also set goals for each student. Entering the data into the computerized system allows for quick creation of a picture of student progress. The teachers follow decision rules they have been using in reading for their CBM data. These include looking at graphed data after six weeks of instruction have occurred and six data points have been collected. For each student, the teachers examine the last four consecutive data points, comparing them to the goal line. If all four points are below the goal line, the teachers implement an instructional change. If all four are above the goal line, they raise the goal. If the four points vary, with some above and some below the goal line, they continue with the same instruction. See Fig. 7 for an example of individual student data and the corresponding decision rules that could be applied. As the 6-week mark hits, the teachers meet during their collaboration time and bring printed graphs for each student that they are progress monitoring.

143

CBM for Beginning Writers Individual Student’s Writing Progress 25.00

Words Spelled Correclty

4 points above line: Raise the goal!

20.00 Points above and below goal line: Keep as is.

15.00

10.00

4 points below goal line: Change instruction.

Decision 1 Decision 2 Decision 3 Goal Line

5.00

1/5 1/12 1/19 1/26 2/2 2/9 2/16 2/23 3/2 3/9 3/16 3/23 3/30 4/6 4/13 4/20 4/27 5/4

0.00

Fig. 7.

Sample Student Progress-Monitoring Data and Application of Decision Rules.

They divide the graphs into three piles based on the decision-making rules (pile 1–4 points below; pile 2–4 points above; pile 3 points above and below). For the students who need an intervention change, they examine qualitative scores, discuss the students’ particular needs, and determine evidence-based interventions that could be put into place to address those specific needs. Students with similar needs are placed in small intervention groups. For students in the ‘‘four points above’’ pile, they raise the long-range goals. As the school year continues, Ms. Taylor and her colleagues continue to monitor the progress of students at risk, and also add students for whom additional screening data (collected in winter of the school year) indicate that more assistance might be necessary. As the year progresses and the teachers become more confident in their procedures, they also begin to share the data with parents, who appreciate the ease with which the data can be interpreted, the focus on student progress, and the implementation of highquality interventions.

CONCLUSION Writing is a critical, yet somewhat neglected skill, and failure to acquire writing proficiency can have serious consequences for students as they progress through school and into postsecondary education and the workforce (Graham & Perin, 2007). For those at risk or identified as

144


having writing-related disabilities, early identification and intervention are essential to preventing these long-term negative consequences (Berninger, 2006; Berninger et al., 2008). CBM is a promising tool for early identification, intervention, and progress monitoring of early writing skills, yet it is clear that many questions must be addressed in research to establish its utility for screening, progress monitoring, and instructional decision making. Our intention for this chapter was to summarize recent developments in this area and to provide direction for future research, as well as eventual practical applications of CBM for data-based decision making aimed at improving outcomes for children who struggle with writing.

NOTE 1. The AUC statistic provides an index of the reliability with which groups can be distinguished (AUC ¼ .50 means probability equal to chance, AUC approaching 1 indicates perfect prediction). Optimal sensitivity and specificity rates are 100%.

REFERENCES Abbott, R. D., Berninger, V. W., & Fayol, M. (2010). Longitudinal relationships of levels of language in writing and between writing and reading in grades 1 to 7. Journal of Educational Psychology, 102, 281–298. Baker, S. K., Chard, D. J., Ketterling-Geller, L. R., Apichatabutra, C., & Doabler, C. (2009). Teaching writing to at-risk students: The quality of evidence for self-regulated strategy development. Bangert-Drowns, R. L., Hurley, M. M., & Wilkinson, B. (2004). The effects of school-based writing-to-learn interventions on academic achievement: A meta-analysis. Review of Educational Research, 74, 29–58. Bereiter, C., & Scardamalia, M. (1987). The psychology of written composition. Hillsdale, NJ: Lawrence Erlbaum. Berninger, V. W. (2000). Development of language by hand and its connections with language by ear, mouth and eye. Topics in Language Disorders, 20(4), 65–84. Berninger, V. W. (2006). Research-supported ideas for implementing reauthorized IDEA with intelligent professional psychological services. Psychology in the Schools, 43, 781–796. doi: 10.1002/pits.20188 Berninger, V. W., Abbott, R. D., Jones, J., Wolf, B. J., Gould, L., Anderson-Youngstrom, M., Shimada, S., & Apel, K. (2006a). Early development of language by hand: Composing, reading, listening, and speaking connections; three letter-writing modes; and fast mapping in spelling. Developmental Neuropsychology, 29, 61–92. Berninger, V. W., & Amtmann, D. (2003). Preventing written expression disabilities through early and continuing assessment and intervention for handwriting and/or spelling problems: Research into practice. In: H. L. Swanson, K. Harris & S. Graham (Eds), Handbook of research on learning disabilities (pp. 345–363). New York: Guilford.


145

Berninger, V. W., Fuller, F., & Whitaker, D. (1996). A process model of writing development across the life span. Educational Psychology Review, 8, 193–218. Berninger, V. W., Nielsen, K. H., Abbott, R. D., Wijsman, E., & Raskind, W. (2008). Writing problems in developmental dyslexia: Under-recognized and under-treated. Journal of School Psychology, 46, 1–21. Berninger, V. W., Rutberg, J. E., Abbott, R. D., Garcia, N., Anderson-Youngstrom, M., Brooks, A., & Fulton, C. (2006b). Tier 1 and tier 2 early intervention for handwriting and composing. Journal of School Psychology, 44, 3–30. Berninger, V. W., & Swanson, H. L. (1994). Modifying Hayes and Flower’s model of skilled writing to explain beginning and developing writers. In: E. Butterfield (Ed.), Children’s writing: Toward a process theory of development of skilled writing (pp. 57–81). Greenwich, CT: JAI Press. Berninger, V. W., Vaughan, K. B., Abbott, R. D., Abbott, S. P., Rogan, L. W., Brooks, A., et al. (1997). Treatment of handwriting problems in beginning writers: Transfer from handwriting to composition. Journal of Educational Psychology, 89, 652–666. Berninger, V. W., Vaughan, K., Abbott, R., Begay, K., Byrd, K., Curtin, G., et al. (2002). Teaching spelling and composition alone and together: Implications for the simple view of writing. Journal of Educational Psychology, 94, 291–304. Berninger, V. W., Vaughan, K., Abbott, R., Brooks, A., Abbott, S., Rogan, L., et al. (1998). Early intervention for spelling problems: Teaching functional spelling units of varying size with a multiple-connections framework. Journal of Educational Psychology, 90, 587–605. Berninger, V. W., Yates, C., Cartwright, A., Rutberg, J., Remy, E., & Abbott, R. (1992). Lower-level developmental skills in beginning writing. Reading and Writing: An Interdisciplinary Journal, 4, 257–280. Coker, D. L. (2006). Impact of first-grade factors on the growth and outcomes of urban schoolchildren’s primary-grade writing. Journal of Educational Psychology, 98, 471–488. Coker, D. L., & Ritchey, K. D. (2010). Curriculum based measurement of writing in kindergarten and first grade: An investigation of production and qualitative scores. Exceptional Children, 76, 175–193. Deno, S. L. (1985). Curriculum-based measurement: The emerging alternative. Exceptional Children, 52, 219–232. Deno, S. L., Mirkin, P., & Marston, D. (1982). Valid measurement procedures for continuous evaluation of written expression. Exceptional Children, 48, 368–371. Foegen, A., Jiban, C., & Deno, S. (2007). Progress monitoring measures in mathematics: A review of the literature. Journal of Special Education, 41, 121–139. Fuchs, L. S. (2004). The past, present, and future of curriculum-based measurement research. School Psychology Review, 33, 188–192. Fuchs, L. S., Fuchs, D., Hamlett, C. L., & Allinder, R. M. (1991a). The contribution of skills analysis to curriculum-based measurement in spelling. Exceptional Children, 57, 443–452. Fuchs, L. S., Fuchs, D., Hamlett, C. L., & Allinder, R. M. (1991b). Effects of expert system advice within curriculum-based measurement in teacher planning and student achievement in spelling. School Psychology Review, 20, 49–66. Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L., & Germann, G. (1993). Formative evaluation of academic progress: How much growth can we expect? School Psychology Review, 22, 27–48.

146


Gansle, K. A., Noell, G. H., VanDerHeyden, A. M., Slider, N. J., Hoffpauir, L. D., Whitmarsh, E. L., et al. (2004). An examination of the criterion validity and sensitivity to brief intervention of alternate curriculum-based measures of writing skill. Psychology in the Schools, 41, 291–300. Graham, S., Berninger, V. W., Abbott, R., Abbott, S., & Whitaker, D. (1997). The role of mechanics in composing of elementary school students: A new methodological approach. Journal of Educational Psychology, 89, 170–182. Graham, S., Berninger, V., Weintraub, N., & Schafer, W. (1998). Development of handwriting speed and legibility in grades 1–9. Journal of Educational Research, 92, 42–52. Graham, S., & Harris, K. R. (2002). Prevention and intervention for struggling writers. In: M. Shinn, G. Stoner & H. Walker (Eds), Interventions for academic and behavior problems: Vol. 2. Preventive and remedial techniques (pp. 589–610). Washington, DC: National Association of School Psychologists. Graham, S., & Harris, K. R. (2005). Improving the writing performance of young struggling writers: Theoretical and programmatic research from the center on accelerating student learning. Journal of Special Education, 39, 19–33. Graham, S., Harris, K. R., & Fink, B. (2000). Is handwriting causally related to learning to write? Treatment of handwriting problems in beginning writers. Journal of Educational Psychology, 92(4), 620–633. Graham, S., Harris, K. R., & Fink-Chorzempa, B. (2002). Contributions of spelling instruction to the spelling, writing, and reading of poor spellers. Journal of Educational Psychology, 94, 669–686. Graham, S., Harris, K. R., & Larsen, L. (2001). Prevention and intervention of writing difficulties for students with learning disabilities. Learning Disabilities Research & Practice, 16, 74–84. Graham, S., & Perin, D. (2007). A meta-analysis of writing instruction for adolescent students. Journal of Educational Psychology, 99, 445–476. Hammill, D. D., & Larsen, S. C. (1996). Test of Written Language – Third Edition. Austin, TX: PRO-ED, Inc. Hampton, D., Lembke, E., & Summers, J. (2011). The effects of progress monitoring students in grade 1 in early writing using curriculum-based measurement. Manuscript submitted for publication. Hintze, J. M., Shapiro, E. S., & Daly, E. J. III. (1998). An investigation of the effects of passage difficulty level on outcomes of oral reading fluency progress monitoring. School Psychology Review, 27, 433–445. Hooper, S., Miller, C., & Rose, S. (2010). e-Assessment progress monitoring environment. Minneapolis, MN: Available at http://avenuedhh.tumblr.com/ Hresko, W. P., Herron, S. R., & Peak, P. K. (1996). Test of Early Written Language (2nd ed). Austin, TX: PRO-ED, Inc. Jenkins, J. R., Graff, J. J., & Miglioretti, D. L. (2009). Estimating reading growth using intermittent CBM progress monitoring. Exceptional Children, 75, 151–163. Jenkins, J. R., Hudson, R. F., & Johnson, E. S. (2007). Screening for at-risk readers in a response to intervention framework. School Psychology Review, 36, 582–600. Jenkins, J. R., Zumeta, R., Dupree, O., & Johnson, K. (2005). Measuring gains in reading ability with passage reading fluency. Learning Disabilities Research & Practice, 20(4), 245–253. Jewell, J., & Malecki, C. K. (2005). The utility of CBM written language indices: An investigation of production-dependent, production-independent, and accurateproduction scores. School Psychology Review, 34, 27–44.


147

Jones, D., & Christensen, C. (1999). The relationship between automaticity in handwriting and students’ ability to generate written text. Journal of Educational Psychology, 91, 44–49. Juel, C., Griffith, P. L., & Gough, P. B. (1986). Acquisition of literacy: A longitudinal study of children in first and second grade. Journal of Educational Psychology, 78, 243–255. Lembke, E., Deno, S., & Hall, K. (2003). Identifying an indicator of growth in early writing proficiency for elementary school students. Assessment for Effective Intervention, 28, 23–35. McCutchen, D. (2000). Knowledge, processing, and working memory in writing and writing development. Educational Psychologist, 35, 13–24. McMaster, K. L., & Campbell, H. (2008). Technical features of new and existing measures of written expression: An examination within and across grade levels. School Psychology Review, 37, 550–566. McMaster, K. L., Du, X., & Petursdottir, A. (2009). Technical features of curriculum-based measures for beginning writers. Journal of Learning Disabilities, 42, 41–60. McMaster, K. L., Du, X., Yeo, S., Deno, S. L., Parker, D., & Ellis, T. (2011). Curriculum-based measures of beginning writing: Technical features of the slope. Exceptional Children, 77, 185–206. McMaster, K. L., & Espin, C. (2007). Curriculum-based measurement in writing: A literature review. Journal of Special Education, 41, 68–84. Olinghouse, N. G., & Leaird, J. T. (2009). The relationship between measures of vocabulary and narrative writing quality in second- and fourth-grade students. Reading and Writing: An Interdisciplinary Journal, 22, 545–565. Parker, D. C., McMaster, K. L., Medhanie, A., & Silberglitt, B. (2010). Modeling early writing growth with curriculum-based measures. Manuscript submitted for publication. Parker, R. I., Tindal, G., & Hasbrouck, J. (1991). Progress monitoring with objective measures of writing performance for students with mild disabilities. Exceptional Children, 58, 61–73. Persky, H. R., Daane, M. C., & Jin, Y. (2003). The Nation’s Report Card: Writing 2002. Available at http://nces.ed.gov/nationsreportcard/writing/. Retrieved on December 1, 2009. Ritchey, K. D. (2004, February). Writing skills in kindergarten: An examination of the psychometric properties of four measures. Paper presented at the Pacific Coast Research Conference, Coronado, CA. Ritchey, K. D. (2006). Learning to write: Progress monitoring tools for beginning and at-risk writers. Teaching Exceptional Children, 39(2), 22–26. Ritchey, K. D., & Coker, D. L. (in press). A comparison of the validity and utility of two curriculum based measurement writing tasks. Reading and Writing Quarterly: Overcoming Learning Difficulties. Ritchey, K. D., Coker, D. L., & McCraw, S. (2010). A comparison of scoring metrics for scoring beginning spelling. Assessment for Effective Intervention, 35, 78–88. Salahu-Din, D., Persky, H., & Miller, J. (2008). The Nation’s Report Card: Writing 2007. http:// nces.ed.gov/pubsearch/pubsinfo.asp?pubid¼2008468 Shanahan, T. (2004). Overcoming the dominance of communication: Writing to think and to learn. In: T. L. Jetton & D. A. Dole (Eds), Adolescent research and practice (pp. 59–73). New York: Guilford. Shanahan, T. (2006). Relations among oral language, reading, and writing development. In: C.A. MacArthur, S. Graham & J. Fitzgerald (Eds), Handbook of writing research (pp. 171–186). New York: Guilford.

148


Stecker, P. M., Fuchs, L. S., & Fuchs, D. (2005). Using curriculum-based measurement to improve student achievement: Review of research. Psychology in the Schools, 42, 795–819. Tangel, D. M., & Blachman, B. A. (1992). Effect of phoneme awareness instruction on kindergarten children’s invented spelling. Journal of Reading Behavior, 24, 233–261. Taylor, R. L. (2003). Assessment of exceptional students: Educational and psychological procedures (6th ed.). Boston, MA: Allyn and Bacon. Troia, G. A. (2006). Writing instruction for students with learning disabilities. In: C.A. MacArthur, S. Graham & J. Fitzgerald (Eds), Handbook of writing research (pp. 324–336). New York: Guilford. Videen, J., Deno, S. L., & Marston, D. (1982). Correct word sequences: A valid indicator of proficiency in written expression (Vol. IRLD-RR-84). Minneapolis: University of Minnesota, Institute for Research on Learning Disabilities. Wallace, T., Espin, C. A., McMaster, K., & Deno, S. L. (2007). CBM progress monitoring within a standards-based system. Journal of Special Education, 41, 66–67. Wayman, M. M., Wallace, T., Wiley, H. I., Ticha, R., & Espin, C. A. (2007). Literature synthesis on curriculum-based measurement in reading. Journal of Special Education, 41, 85–120. White, O. R., & Haring, N. G. (1980). Exceptional teaching. Columbus, OH: Merrill. Woodcock, R. W., McGrew, F. A., Schrank, F. A., & Mather, N. (2007). Woodcock Johnson III Tests of Achievement Normative Update. Rolling Meadows, IL: Riverside Publishing.

ISSUES IN ASSESSMENT FOR INTERVENTION IN IMPLEMENTATION OF RESPONSIVENESS TO INTERVENTION MODELS Rollanda E. O’Connor and Victoria Sanchez ABSTRACT Response to Intervention (RtI) models require valid assessments for decisions regarding whether a student should receive more intensive intervention, whether interventions improve performance, whether a student has improved sufficiently to no longer need intervention, or whether a student should be considered for a formal evaluation for special education. We describe assessment tools used currently in RtI models in reading in kindergarten through third grade, along with how these tools function in multiyear implementations of RtI. In addition to the measurement tools, we describe concerns regarding when RtI models are judged for their effects on reading improvement and the attrition that may inflate these results.

Response to Intervention (RtI) models were developed initially to enhance reading achievement and improve identification of students with reading Assessment and Intervention Advances in Learning and Behavioral Disabilities, Volume 24, 149–170 Copyright r 2011 by Emerald Group Publishing Limited All rights of reproduction in any form reserved ISSN: 0735-004X/doi:10.1108/S0735-004X(2011)0000024009

149

150

ROLLANDA E. O’CONNOR AND VICTORIA SANCHEZ

difficulties or disabilities. To accomplish this aim, decisions must be made – decisions regarding whether a student will receive RtI services, decisions whether a student is demonstrating adequate progress, decisions whether a student needs a different intervention, and decisions to determine if a student is in need of special education services. RtI models are becoming common in schools as educators grapple with issues surrounding early identification of reading risk and disability, assessment, and quality instruction. Although RtI models can be used across academic and behavioral areas, and for students beyond third grade, we limit our discussion in this chapter to RtI used as reading intervention in kindergarten through third grade to decrease the incidence and severity of reading difficulties. RtI uses universal screening and progress monitoring measures to identify students who may be at risk for future reading difficulties, along with just-in-time intervention, and so could play a pivotal role in the promotion of reading achievement. With the knowledge gained through assessments, RtI impacts instructional decisions for young students in the hope that progress will accelerate (Deno et al., 2009). A key foundation of RtI is assessment because assessments enable educators to identify students who may benefit from early intervention, monitor progress of students during intervention, and determine when students have gained sufficiently from intervention to need it no longer. Screenings are intended to accurately identify students who – without assistance – may develop reading disabilities (RD: true positives). Additionally, screenings are intended to limit the number of false positives in the sample – the students who appear to be at risk but catch up and succeed with typical good instruction (i.e., Tier 1). Students who develop reading difficulties but were not identified appropriately during screenings (false negatives) miss their opportunity for the early, intensive intervention that may improve their reading achievement. Universal screening assessments (i.e., administered to all students in a class or grade level) are designed to balance the number of false positives and negatives identified, while also coordinating a screening session that does not burden schools with lengthy testing times, which can decrease instructional opportunity. In this chapter, we describe available standardized and curriculum-based measures used in reading RtI frameworks, and how well they function in three- and four-year implementations of RtI in public schools. In order to do so, we consider emergent literacy skills of reading and reading comprehension and how they are tapped with screening and monitoring tools. In particular, how are assessments used to identify students for supplemental reading interventions and how are data gathered to inform

Assessments to Inform Interventions in RtI

151

intervention for identified students? As we move toward future models of RtI, it is important to consider the problems surrounding the identification of learning disabilities (LD) and how assessments might be modified to improve early identification of risk for reading difficulties.

UNIVERSAL SCREENING: SELECTING STUDENTS FOR INTERVENTION Current practice typically includes risk screenings three or four times per year to identify students who are not reaching grade-level expectations or benchmarks, and so are considered ‘‘at risk’’ for future failure. The indicators for students at risk for phonologically based reading difficulties in kindergarten through first grade have an established evidence base in the literature. Often measures collect how many letters students know, how many segments of words (i.e., phonemes or speech sounds) they produce, how many sight words they read, rate of nonsense word decoding, or rate of words they read correctly on a grade-level passage. The early research in models of RtI was designed to identify students who might benefit from early intervention in reading prior to students experiencing failure, which means identifying them before ‘‘real reading’’ is expected to occur, most often in kindergarten. Researchers who have studied good and poor readers in first or second grades have found significant differences predicting these groups as early as kindergarten (e.g., Liberman & Shankweiler, 1985; Share, Jorm, MacLean, & Matthews, 1984; Tunmer & Chapman, 1998), but using these differences to identify potentially poor readers in kindergarten reliably is difficult. Although the indicators are well established, the specific cut-points, or scores above or below which students are considered typical or at risk for difficulties, are variable across studies and samples of children. To illustrate the difficulty of identifying students prior to the development of measurable reading, O’Connor and Jenkins (1999) assessed students multiple times from the beginning of kindergarten through the end of first grade, using measures of pre-reading skills identified in earlier studies, including letter and letter–sound knowledge, phonemic blending and segmenting, nonsense word repetition, and vocabulary, as well as measures of word reading. Rather than seek significant differences between good and poor readers, they tried to establish cut-points for each measure so that

152


school personnel could identify students who might benefit from early reading intervention. In their first cohort, students who demonstrated reading difficulties at the end of first grade identified only 2.2 letters on average as they began kindergarten; however, some children identified as many as 10 letters and still developed reading problems over the next two years. Moreover, replication with a second cohort expanded the maximum scores of students who later developed reading problems to 12 letters. So, even though the average scores of poor readers are significantly lower than typical readers (e.g., 2.2 letters vs. 15.2), using these average differences will miss some children during the window of time that intervention may have improved their long-term outcomes. It is for these reasons that RtI developed: scores may indicate potential risk, but the students at highest risk are also likely to improve during intervention at slower rates than students who were falsely identified by low scores. By monitoring scores on key measures along with student growth on these measures, more accurate identification of risk may be possible. Our research team has explored the interface between assessment and intervention over several years of RtI implementation to determine the accuracy of assessments for predicting reading outcomes and the impact of intervention for decreasing the incidence and severity of RD. In addition to using available screening measures (e.g., AIMSweb: Shinn & Shinn, 2002; CBM: Deno, 1985; DIBELS: Good, Kaminski, Smith, Laimon, & Dill, 2001a), we have used variations of measures to explore whether we can improve the accuracy of prediction of risk across students with differing characteristics, including students from high-poverty communities (O’Connor, 2000; O’Connor, Fulmer, Harty, & Bell, 2005a; O’Connor, Harty, & Fulmer, 2005b), students who are English language learners (Linklater, O’Connor, & Palardy, 2009), or students with cognitive impairments (O’Connor, Bocian, Beebe-Frankenberger, & Linklater, 2010a). Two problems to balance are the ‘‘catch’’ and ‘‘release’’ criteria for intervention. Selecting students with multiple measures (e.g., poor letter knowledge alongside poor segmenting skills) decreases the proportion of students selected for intervention; however, improvement across measures during intervention is likely to occur at different rates, making the ongoing decisions about whether to continue an intervention problematic. For example, we used the combination of letter naming (fewer than nine) and first sound identification in spoken words (fewer than seven initial sounds) to ‘‘catch’’ kindergartners for intervention in the first year of a three-year study (O’Connor et al., 2010a). Half of the selected students were English


153

learners (ELs). These criteria missed only one student across eight schools who later developed reading difficulties, but it is likely that many of the selected students were false positives. Although the measures caught nearly all of the ‘‘right’’ students, scores at the first screening did not predict which students would respond well to our intervention. Responsiveness in kindergarten was also uneven: 10% of the students failed to learn to segment past the initial sounds in words, whereas 28% were still below average on letter naming. We found no difference between responsiveness of ELs and native English speakers to intervention; however, an untimed measure of segmenting that included corrective feedback was more sensitive than timed measures to growth in overall reading for the ELs (Linklater et al., 2009). Over the next two years, we also identified problems with ‘‘release’’ criteria: at what point do scores on key measures indicate that students are no longer at risk? Using scores from DIBELS or standardized tests that indicated successful responsiveness and outcomes in earlier studies (e.g., Coyne, Kame’enui, Simmons, & Harn, 2004; O’Connor, 2000), we released students back to Tier 1 (general class instruction) only to catch them falling behind again in the next academic year – a finding also noted in our 2005 studies. The choice of selection criteria affects which students enter the intervention pool, and if students across studies differ in relevant characteristics, they are also likely to differ in responsiveness to particular interventions. For example, Coyne et al. (2004) used naming fewer than six letters to select students for intervention, and verified these choices by teacher agreement that the student seemed to be at risk; O’Connor (2000) used fewer than 15 letters combined with naming fewer than four phoneme segments in 10 words and standardized scores below 86 on the Woodcock–Johnson subtest of letter–word identification. This latter criterion was also used by Scanlon, Vellutino, Small, Fanuele, and Sweeney (2005). For all of these studies, the rate of responsiveness in kindergarten was high (about threefourths of students were considered good responders); however, judgment of responsiveness in first grade varied across studies due to the choice and qualities of assessment tools used for making these determinations and the timing for judging responsiveness. Differences in timing of determining responsiveness are important because ‘‘what counts’’ as reading changes over time, so that students who appear strong on isolated or lower-level skills may still fail later in schooling as reading becomes more difficult. As a field, researchers in RtI seem more successful with ‘‘catch’’ than with ‘‘release’’ criteria regarding risk for poor reading outcomes.

154


Whereas earlier studies relied on nationally normed assessment tools as outcome measures, recently researchers have tried to predict successful reading as measured on state-specific outcomes. Roehrig, Petscher, Nettles, Hudson, and Torgesen (2008) used DIBELS oral reading fluency (DORF) as an indicator of risk for students in Florida’s Reading First schools, which is similar to other RtI models that have used oral reading fluency (ORF) as a screening tool. Roehrig and colleagues compared fluency and reading comprehension achievement on the Florida Comprehensive Assessment Test (FCAT) to determine whether risk classification based on fluency coincided with poor comprehension achievement – an important question considering the importance of high-stakes tests for students, teachers, and schools. Optimal benchmarks were recalibrated to allow for more accurate identification of at-risk students and reduction of false positives and negatives. Using DIBELS criteria, third grade fall benchmarks indicate that students who read fewer than 53 words correctly aloud in one minute are at risk for poor third-grade outcomes. The recalibration for the Florida sample suggested that using a criterion score of fewer than 45 words read correctly per minute (wcpm) during fall resulted in more accurate identification of atrisk students as predicted by the FCAT. Even though the recalibration was more accurate statistically, using this lower score for wcpm missed 17% of students who performed poorly on the FCAT. In an RtI model, one might expect that students reading so slowly (45 wcpm in third grade) would be receiving intensive intervention; however, intervention was not a focus of Roehrig et al.’s research. Perhaps more surprising is that some students reading 50–60 wcpm scored as proficient readers on the FCAT, which suggests that reading rate was not an optimal screening measure for predicting comprehension of slower readers. As the benchmarks changed for Florida Reading First schools, other states should also reconsider what ‘‘risk’’ truly means in terms of performance on their state testing. Recommended criteria for risk identification based on achievement in second grade or beyond may need revision as performance in relation to these state achievement tests is considered. Risk criteria, such as benchmarks for classifying students, are formulated most often with intact samples of students with and without risk for poor reading outcomes (e.g., DIBELS benchmarks based on Good, Simmons, & Kame’enui, 2001b; AIMSweb benchmarks based on Shinn & Shinn, 2002). When a large proportion of the students in the sample are at risk, such as in Reading First schools or when monitoring progress of students in Tier 2 interventions, these criteria may be set too low. If students have needed intensive intervention to reach these benchmarks, the likelihood that they


155

will retain ‘‘not at risk’’ standing over time without this support may be less than for students who attained benchmark scores with typical general class instruction. Although many studies have documented using measures such as oral reading rate to monitor progress of struggling readers, few have tried to establish cut-points among groups of slow readers for predicting performance on high-stakes tests. It may be that ‘‘safe’’ levels of reading rate differ when these students are considered separately. As the benchmark became more stringent in the Roehrig et al. (2008) sample, most ‘‘true’’ atrisk students were identified, which could maximize cost efficiency because students who would later demonstrate adequate performance on the FCAT would not be identified for services. In a climate of limited resources in education, minimizing service allocation to those who most need it maximizes efficiency. The trade-off for maximizing resources in this case was the group of failing students who would be deemed ineligible for Tier 2 intervention. In our research, we have analyzed the relationship between the California State Test, English Language Arts (CST ELA), and oral reading rate for second-grade students. The CST is California’s high-stakes assessment, and students who have a scaled score below 300 are considered below basic, while those whose scaled score is above 350 are considered proficient. The Grade 2 DIBELS benchmark criterion indicates that students who read fewer than 25 wcpm at the beginning of the year are at risk for reading difficulties, and students reading more than 44 wcpm are not at risk. Students in our sample included native English and EL students. Fig. 1 displays the relationship between Fall ORF scores and Spring CST ELA scores for the 260 students present for both measures. A moderately strong positive correlation is observed (r ¼ 0.675, p ¼ 0.00), indicating that students who read more words per minute in the fall also tend to score well in ELA by the end of the year. Results of a regression analysis indicate that ORF is a significant predictor of CST performance (b ¼ 0.675, p ¼ 0.000), accounting for 45% of the variance in ELA scores. We monitored each student’s risk classification across the school year to determine the rate of true and false positives and negatives based on fall and winter reading rate. On the Spring CST, 80 students scored below the basic scaled score of 300, indicating these were students in need of supplemental instruction. Of these students, 30 were correctly identified at fall screening using the DORF risk benchmark (true positives: 38% accurate identification). An additional 16 students were correctly identified by winter screening, increasing accurate identification to 58%. Lastly, 34 students were not identified in fall or winter using the recommended criterion, but

156

ROLLANDA E. O’CONNOR AND VICTORIA SANCHEZ R2 Linear = 0.456 CST ELA Standard Score 08-09

500

400

300

200 0

20

40

60

80

100

120

140

ORF (Fall 2008)

Fig. 1.

Relationship between Fall Oral Reading Fluency and Spring CST ELAscaled scores for Grade 2 Students.

still failed the state assessment (false negatives: 43% inaccurate identification), which means that for every 100 students who were facing academic failure, 43 students would not receive assistance if DORF scores were the selection criterion. If the recommended reading rate was adjusted to capture more of the below basic students – students who according to state standards are not meeting appropriate achievement levels – we would also open the gates of Tier 2 intervention to students who do not need assistance. Professionals are walking a tightrope, balancing the need to provide intervention to students who have the potential to develop RD with those who do not. These results demonstrate that sole reliance on one screening measure at one given point in time is insufficient to identify high-risk students adequately. RtI screenings in second grade and beyond should include multiple skills simultaneously for appropriate identification (see O’Connor et al., 2005a, 2010b; Simmons et al., 2008). As Compton, Fuchs, Fuchs, Bouton, Gilvert, and Barquero et al. (2010) suggest, multiple screening measures should be considered over a period of time to appropriately identify these students. Overreliance on fluency benchmarks alone could lead to failure of RtI models either because (a) they do not assist enough students in need of additional instruction or (b) they identify students who are already demonstrating sufficient achievement on high-stakes assessments.


157

These problems are not limited to reading rate; in addition to the problems of ORF in universal screening, we have found similar difficulties using standardized assessments. As an example, Coyne et al. (2004) reported strong success for their first graders who received early intervention because those who responded well in kindergarten continued to respond well in first grade as measured with the Woodcock Reading Mastery Tests (WRMT; Woodcock, 1998). While within the average band on the WRMT, students continued to score below average on rate of reading aloud, and scores of students who responded poorly in kindergarten were not reported; thus, it is difficult to evaluate the effect of the intervention and appropriateness of the measures used in the overall RtI model. Moreover, comprehension as assessed with the WRMT in first grade is a single-sentence word supply task, which may not emulate reading comprehension as most educators would define it. Standardized assessments, such as the Woodcock–Johnson Tests of Achievement (W–J) or WRMT, are used generally as outcome measures because students can be compared with national norms. These measures are not timed, although students must respond to each item within a specified number of seconds. In our research, they appear to underestimate risk for students who have received intervention in the early grades, particularly in reading comprehension. As examples, the average score for first graders identified with disabilities who participated in intervention in O’Connor’s (2000) study of RtI was 86.4; the average WRMT for the second grade poor responders in our current study is 92.6. Students in neither group would be judged as much outside the average band by these scores; nevertheless, teachers identified these students still as struggling readers despite intervention and their performance within an average range nationally. Listening to their effortful attempts to read remained painful. We see a strong need for continued efforts in development of measures that more accurately assess students’ reading ability relative to peers, and that capture school district expectations in the post-Reading First era that heightened the early emphasis on reading instruction.

THE PROBLEM OF UNIVERSAL SCREENING FOR READING COMPREHENSION A worry among educators is that the widely used assessments, such as ORF, are unable to identify students whose reading problems are seated in reading

158


comprehension. These students are sometimes called ‘‘late identified’’ RD because measures of early literacy such as letter–sound correspondence or segmenting pose few problems for these students. Rather, their reading problems stem from difficulties with vocabulary and comprehension of running text (Berninger, 2008; Catts, Fey, Tomblin, & Zhang, 1999; Shaywitz, Gruen, & Shaywitz, 2008). By characterizing children as at-risk readers in need of intervention based solely on fluency or decoding skills, educators may disregard the needs of students who no longer struggle with word reading but still struggle with comprehension. Berninger (2008) and Shaywitz et al. (2008) suggested that overreliance on phonologically based reading difficulties is leaving other groups of students (dysgraphic, specific language impairment, or nonphonologically based reading difficulties) behind. As an example, Catts, Adlof, and Weismer (2006) identified students with language-based LD who were identified for special education in fourth grade or later. Up until that time, their performance on screening measures was adequate. Students whose reading disability is seated in comprehension may be overlooked by common measures used in universal screening. It is possible that some of the false negatives in prediction studies are these very students; however, this question has received scant research attention. O’Connor, Bocian, and Sanchez’s (2010b) study of RtI compared students who were identified early versus those who were identified later with available reading assessments. Preliminary analyses suggest that an additional 66 students were identified in third grade as compared to the 93 students who were identified early on the basis of early literacy measures. That over a third of these students were missed in K-2 suggests that tapping low-level skills (e.g., phoneme segmenting, letter naming, decoding) and simple rate measures (e.g., ORF without also monitoring accuracy and comprehension) may miss more students than researchers generally report in studies that assess students in K-1. Deno (1985) has conducted decades of research on oral reading rate to screen for reading difficulties in elementary schools, and this work is foundational to models of RtI in reading. He also recognized that rate might not be sufficient for locating students with comprehension difficulties that were not also reflected in word reading, although these students are a minority among students with RD. To incorporate comprehension in screening and monitoring progress, Fuchs and Fuchs (1992) developed CBM maze, in which every seventh word in a passage was marked with a blank. Each blank provides students with three choices (one correct and two distracters) and students mark their choice as they read. Maze is scored as


159

the number of correct choices in two minutes of reading, and scoring that corrects for guessing has also been validated (Pierce, McMaster, & Deno, 2010). Although maze taps only surface comprehension, the addition of comprehension could make the measure more sensitive to poor growth among students whose reading problems are seated in the comprehension domain. Scores on maze also continue to be sensitive to growth after fourth grade, when oral reading rate gradually loses validity (Jenkins & Jewell, 1993; Ticha´, Espin, & Wayman, 2009). Few valid screening tools for comprehension-based reading difficulties exist currently. Deno et al. (2009) tested the use of CBM maze as a schoolwide monitoring system for use in a reading RtI model in Grades 2–6. Maze was used as a screening tool at fall, winter, and spring in order to identify students who were at risk. Maze performance across years predicted performance on the Metropolitan Achievement Test and the state reading assessment for students in grades 3 and 5. Also maze performance across the years improved, which suggests that the measure is sensitive to reading improvement over time. Deno et al. suggested that maze could be a valid system for identifying risk, and his team taught teachers how to identify the low-performing students and make instructional modifications based on data from assessments. This study adds to a growing body of research (e.g., see also Jenkins & Jewell, 1993) on the potential of maze assessments for universal screening. In addition to CBM-based measures, researchers have relied on normbased assessment such as the WRMT (Woodcock, 1998) or Gray Oral Reading Test (GORT-4; Wiederholt & Bryant, 2001) for identifying students’ degree of risk. An analysis of reading comprehension tests and the skills they assess by Keenan, Betjemann, and Olson (2008) found that passage comprehension on the WRMT relies heavily on decoding skills, even for students beyond second grade. The GORT-4 relies less on decoding for its comprehension assessment; however, the test is timed and slow readers who comprehend well may reach the ceiling for scoring the passage on rate and have the test terminated before tapping the upper limit of what they are able to understand. How these tools function with ELs who participate in a Tier 2 intervention in English is a further area that deserves research attention. Students who are typical readers tend to generate standardized scores on the WRMT and GORT-4 that are similar, but the scores can differ significantly across these measures for students in Tier 2 of an RtI model. For example, the WRMT tend to underidentify risk in kindergarten and first grade (O’Connor, 2000; O’Connor et al., 2010b; O’Connor et al., 2005a;

160


Simmons et al., 2008). Students who can barely read at the end of first grade can achieve standardized scores of 100 or more on this measure (O’Connor, Bocian, & Sanchez, 2009). In our current study, students selected for Tier 2 reading intervention scored an average of 100.8 in second and 96.9 in third grade on the WRMT. Standard scores from the GORT-4 for the same group of students more accurately depicted teachers’ views of their risk status as well as their ranking within the class (scores averaged 81.7 for second grade and 83.7 for third graders in Tier 2). Nevertheless, we are concerned that scores on the GORT-4 may underestimate students’ reading ability because the testing protocol does not allow students to look back at the text to answer questions, which may place undue burden on ELs in particular who often code-switch as they read. Although reading passages and answering questions may seem to more closely emulate classroom reading tasks, students in Tier 1 are taught to review and verify their answers based on the text, which is not allowed on the GORT-4. In all, the strengths and weaknesses of using particular skills, CBM probes, or standardized assessments as screening tools to identify students for intervention should be taken in account. A range of tools is available, and none address all contingencies.

PROGRESS MONITORING ASSESSMENT: PRACTICAL CONSIDERATIONS IN AN RTI MODEL Progress monitoring assessments are beneficial because student achievement is closely monitored during Tier 2 intervention and instructional modifications are made, if necessary. National and current focus rests on ORF as a key progress monitoring tool in general and special education settings because of its sensitivity, multiple probes for frequent use, quick administration, and instructional utility (Deno, 1985; Marcotte & Hintze, 2009). ORF is widely used in schools as a proxy for reading achievement in the early grades. Although three or four times per year may be sufficient to screen students who have not been identified previously (Good et al., 2001b; Roehrig et al., 2008), researchers have recommended monitoring as often as weekly (Fuchs, Fuchs, Hamlett, Walz, & Germann, 1993) for students at risk or receiving interventions. Putting frequency of monitoring to an experimental test, Jenkins, Graff, and Miglioretti (2009) compared the timing and conditions of monitoring using oral reading CBM for students with LD (mean age 10.6 years). They found that progress monitoring could


161

be completed every three weeks without affecting the validity of growth estimates, as long as multiple probes are administered during each testing session. Although these measures of reading rate reflect growth in students with phonologically based reading problems and evidence supports their reliability and accuracy, few alternatives are available to monitor progress in comprehension. When reading problems are centered on word reading, growth in reading rate is related to comprehension growth, but this relationship is less robust for students with vocabulary and comprehension difficulties unrelated to decoding. Paris and Hamilton (2009) suggest that because fluency is positively associated with comprehension, it is often assessed instead of assessing comprehension directly. This practice may have negative implications in practice due to either inappropriate identification for supplemental instruction, inappropriate match between instruction and assessment, or invalid conclusions regarding progress. One problem with ORF as a sole indicator of growth is that it becomes less sensitive as typical readers develop optimal oral reading rate – usually in fourth or fifth grade (Jenkins & Jewell, 1993). In contrast, maze (described earlier) is a silent reading test and could be more useful than ORF in third grade and beyond, when students need to read silently with comprehension to succeed in academic tasks. Using criterion scores to select students for risk from Deno et al. (2009), Sanchez (2010) collected maze data for at-risk third grade students. Administered monthly for students displaying some reading risk, students demonstrated increased scores in median correct over time (Med1 ¼ 9.34, Med2 ¼ 12.45, Med3 ¼ 13.03, Med4 ¼ 14.01). The median correct maze score correlated significantly with ORF, WRMT passage comprehension, and GORT-4 comprehension scores (correlations were 0.56, 0.39, and 0.37, respectively). Identifying accurate and functional tools to monitor progress is important even though, like most measures used in the research and in schools, each may have flaws. Shelton, Altwerger, and Jordan (2009) suspected that the kind of reading students do on packaged ORF passages, such as DIBELS or AIMSweb, may differ from the reading they do daily in class. They investigated the relationship between two types of passages for monitoring the reading progress of second graders: DORF passages and grade-level children’s literature. Although Shelton et al. (2009) only included scores from 14 students in one classroom, the scores demonstrated great variation between the DORF rate and the read-aloud rate with second grade literature. In a majority of the cases, students read more quickly with ORF passages than on the grade-appropriate text. Differences in rate suggested

162


that when children were reading ‘‘real’’ material, their rate slowed down to allow them to comprehend the passages. They suggested that children’s rate on the grade-appropriate literature may be a better reflection of rate during reading activities in the classroom when children are reading to learn rather than practicing building rate. The downside is that ‘‘real literature’’ is less quantifiable or generalizable than standardized passages. We replicated this study by comparing reading rate of second and third graders in Tier 2 intervention using DIBELS passages and passages from the students’ reading basal, Houghton–Mifflin (O’Connor & Bocian, 2009). To ensure that passages had not been read before, we selected basal passages from a few weeks ahead of when students would read them in class. Second grade students read, on average, 50 wcpm on the DORF and 33 wcpm on the Houghton–Mifflin passages. Third grade students read, on average, 68 wcpm on the DIBELS ORF and 63 wcpm on the Houghton–Mifflin passages. The differences between DIBELS passages and the Houghton– Mifflin passages were significant for both grade levels. As in Shelton et al. (2009), rate differed significantly in our sample of over 100 students per grade, with DORF passages read at consistently faster rates than the literature collection in their basal reader. It was unclear, however, whether students read the basal more slowly to understand the stories or because the text was more difficult, and so we could not validate Shelton et al.’s hypothesis. In addition to evaluating the rate of reading for the students, Shelton et al. (2009) also briefly explored the accuracy of reading. Results revealed that the faster ORF readers were by no means the more accurate readers. Likewise, the students who were the slowest ORF readers were not the more inaccurate readers. Untangling rate and accuracy may allow for better conceptualization of students’ reading and their reading difficulties. In universal screening and in monitoring progress during intervention, whether measures are timed may also influence whether students are viewed as responsive to intervention. We mentioned earlier how untimed tests such as the WRMT may inflate a student’s reading rank, but the answer is not simply to use timed tasks, even though students who read well tend also to read quickly. We illustrate this point with two hypothetical (but common) second grade students participating in RtI, Tier 2 intervention: (1) Mark, who reads poorly due to inaccurate identification of words, and (2) Angelina, who reads poorly in English (and perhaps in her native language also), and who is learning English alongside learning to read. At the beginning of intervention, our first student, Mark, missed 28% of the words he attempted to read on second grade passages. He participated in a


163

small group, Tier 2 intervention for nine weeks. His progress was monitored with his rate of reading aloud, such as with the measures of ORF in AIMSweb or DIBELS. The peers in his group have made steady gains of one or two words faster per week; however, Mark’s rate of growth is clearly flat. His intervention teacher believes his reading is improving, but teachers sometimes believe that students are more successful than they really are. We might conclude by the failure to see growth in reading rate that the intervention is not working well for Mark, and that his instruction should be changed or he should be formally evaluated for special education, but a careful look at his reading accuracy over time tells a different story. Across nine weeks, Mark’s accuracy on second grade text improved steadily from 72% to 86%. He still reads slowly, but he is beginning to understand what he reads and to search for answers to questions, where previously he sat passively during such assignments. Mark’s story is reflected across more than 10% of the students in our current RtI research, whose progress is revealed as steadily improving accuracy, rather than steadily improving rate. Although oral reading rate is a useful indicator of progress, using it as the sole criterion could misrepresent some students’ responsiveness to intervention. Angelina’s progress also shows scant improvement on the measure of oral reading rate. When selected for intervention, her rate of reading was slow for second grade (28 wcpm), and she was withdrawn in class and did not respond to questions or discussion about her reading. Her vocabulary score reflected the fact that she was learning English and had improved one point on a fivepoint scale in English language development since she began school as a kindergartner and EL beginner. Although her reading rate has not improved as much as other students in her group (from 28 to 34 wcpm in 10 weeks), the interventionist noted that she now both answers and asks questions about her reading, and specifically asks about the meanings of words she does not know. We lack valid progress monitoring tools for vocabulary improvement in an RtI model; however, her general education teacher has noted improvement on the basal reading tests administered every four weeks in class, which rely extensively on learning the vocabulary and comprehending the stories in the unit. Students in Tier 2 interventions are there because their learning is atypical. Some of these students may also require different types of monitoring to capture whether their reading is improving. In each case, we advocate considering additional aspects of reading beyond rate alone to monitor the progress of students in interventions. Measures of reading accuracy, of vocabulary, and of comprehension should also be taken into account. Across the country, teams of researchers are working on viable means to monitor progress in these important features of

164


reading (e.g., see Pierce et al. (2010) for variations on maze scoring; see Deno et al. (2009), Fuchs, Fuchs, and Hamlett (1989), Fuchs, Fuchs, and Maxwell (1988), Marcotte and Hintze (2009), McKeown, Beck, and Blake (2009), and Ticha´, Espin, and Wayman (2009) for comprehension measures; see Tuckwiller, Pullen, and Coyne (2010) for monitoring vocabulary). Until these measures are available commonly, we suggest that teams implementing RtI take a multifaceted approach to monitoring reading progress and determining responsiveness of students. In our RtI studies, we encounter various patterns of reading ability among our students. Some students are fast and accurate readers with good comprehension – established readers. Reading interventionists do not worry about these students and in fact this is what they strive to make the at-risk readers become. Conversely, students may be fast and accurate readers with poor comprehension, which is cause for concern. These students would not be identified in an RtI model that focuses on phonologically based reading difficulties, and for these students progress monitoring tools focused on reading comprehension may be essential. Consider also slow and inaccurate readers with good or poor comprehension. In either case, students may be identified for assistance given the emphasis on phonologically based RD because of their inaccuracy and slow rate. In the case of students with good comprehension, instructional practices could capitalize on comprehension to assist with word identification or focus on accuracy and fluency with minimal comprehension instruction for efficiency. The slow inaccurate reader with poor comprehension represents a poor reader across all dimensions and may need different instructional content that incorporates all aspects of reading development. These cases are interesting because they imply differing interventions or instructional practices for differing problems. They also exemplify how sole reliance on oral reading rate may interfere with educators’ attempt to disentangle and differentiate reading difficulties. When students are grouped together based on their reading rate scores and not by their strengths or weaknesses, implications for instruction can be difficult to draw, especially regarding comprehension instruction for students beyond first or second grade.

MONITORING COMPREHENSION Maze probes can be used in universal screening, as mentioned earlier, but they also have been used as a tool for monitoring progress during


165

interventions. Ticha´ et al. (2009) investigated the use of a read-aloud (fluency) procedure and a maze task (also referred to as a cloze procedure) as progress monitoring tools for reading comprehension of eighth grade students across a range of reading ability over a period of 10 weeks. A hierarchical linear model procedure was used to compare maze and readaloud scores over the 10 time points for students. The results indicated that the read-aloud measure did not produce a slope that was significantly different from 0 (i.e., did not demonstrate growth over time), while the slope for the maze did indicate growth. Growth on the maze was also related to change on the Woodcock–Johnson III Tests of Achievement, Broad Reading Cluster for higher- and lower-performing groups of students. Production tasks such as retell fluency and retelling protocols can assess comprehension of a passage directly, but they have shown lower reliability historically (e.g., Sudweeks, Glissmeyer, Morrison, Wilcox, & Tanner, 2004). Retelling procedures were used to assess comprehension of the students in Shelton et al.’s (2009) study. Floor effects were not found, which suggested that each of the students read with some level of understanding. Although ORF classified some readers as at risk, retell scores did not identify the same students. Comprehension scores did not differ between high-risk and low-risk readers, as identified by reading rate. Shelton et al. (2009) pointed out that rate scores cannot demonstrate how children use and process information in text. Because of the small sample and no information on reliability of retell scores, more research is needed. In a study of validity of various types of measures to predict outcomes on a state’s high-stakes test, Marcotte and Hintze (2009) analyzed the incremental and predictive utility of five types of reading comprehension assessments among fourth grade students: DORF, DIBELS Retell Fluency (RTF), sentence verification, written retell, and maze. They examined each measure’s relationship to reading proficiency on an English Language Arts performance assessment. Overall their results provided support for the use of three comprehension measures: maze, written retell, and sentence verification. After controlling for ORF in a standard multiple regression model, these three measures significantly contributed to the overall model of reading. These measures plus the ORF accounted for 66% of the variance in student performance on the statewide assessment. Marcotte and Hintze hypothesized that RTF did not add explainable variance in reading proficiency above and beyond ORF because of its low reliability. They suggested that a written retell may be more appropriate than retell fluency. However, for students at the lower range of reading

166


proficiency, a writing task could restrict student performance. An oral task allows students to provide information quickly and efficiently; however, writing may demonstrate their difficulties with spelling, forming sentences, and writing sufficient detail in the allotted time period to accurately represent comprehension. McKeown et al. (2009) introduced an oral retell procedure that has the potential to assess comprehension growth in addition to having instructional utility. Their instructional experiment contrasted two approaches to comprehension for fifth grade students: content comprehension versus strategy instructional approaches. They measured students’ comprehension with a sentence verification technique, similar to Marcotte and Hintze (2009), and an oral retell. Across their instructional approaches, the sentence verification technique failed to demonstrate differences in achievement, but the oral retell demonstrated differences between instructional approaches. Students in the content approach generated retells with greater length and quality. The authors suggested that the retell required a higher level of comprehension than the sentence verification technique. Due to the retells’ ability to distinguish progress between different instructional approaches, it may also be able to distinguish progress between students who struggle with reading and those who do not. Although McKeown et al. (2009) examined a range of ability within fifth grade, they did not analyze sensitivity of oral retells to discriminate between high and low achievers. McKeown et al. also suggested that the retell could be used to predict performance on a criterion measure of reading comprehension, such as a standardized reading assessment or a high-stakes state assessment; however, this possibility was not tested in their study. In summary, ORF, indicative of word reading skill, tends to improve through Grade 4, but may miss students whose main reading problem is vocabulary or comprehension. Maze taps surface comprehension, but may miss deeper levels of comprehension captured by retell quality. Retell quality, similar to the measure used by McKeown et al. (2009), may be beneficial for struggling students, but has been underresearched and may be less reliable than widely researched tools such as ORF and maze. Beyond RtI in first grade, educators should consider additional measures to determine student progress and make instructional changes. Additionally, because vocabulary is a key aspect of comprehension, future reading comprehension monitoring tools should be considered that incorporate vocabulary.


167

LOOKING TOWARD THE FUTURE OF RTI: LINGERING ISSUES As evaluations of RtI from Grade K-1 advance into the higher grades, stability of risk and whether positive effects of intervention persist take centerstage. Students whose skills improved with intervention and who caught up with their peers by the end of the school year may fail to grow during the summer months at a rate commensurate with average readers, or scores may decline on skills they once attained (O’Connor, 2000; O’Connor et al., 2010a; Vellutino, Scanlon, Zhang, & Schatschneider, 2008), and thereby fall into risk once again the following year. Additionally, students who appear strong on low-level skills may fall behind their peers as reading grows in complexity over time (O’Connor et al., 2005a); thus, the high responsiveness noted for kindergartners and first graders (Coyne et al., 2004) may be elusive for older poor readers (O’Connor et al., 2010b). Our research continues to note the instability of catching up for many struggling readers (O’Connor, 2000; O’Connor et al., 2005b, 2010a). This instability suggests that attributing success of intervention to end-of-year scores may be premature. It is likely that some students will need frequent boosters of intervention to remain strong readers once they achieve some narrowing of the reading gap with typical readers. We are also concerned with differential attrition between high- and lowrisk students in long-term studies of RtI. Most longitudinal studies lose participants as families move to other schools or districts or are lost to the database for other reasons. Recent studies of three and four years’ duration document a differential rate of attrition for average- and high-risk readers and for students who respond well or poorly. For example, Simmons et al. (2008) found that 46% of good responders, but only 20% of poor responders remained in their study at the end of third grade. O’Connor, Bocian, and Sanchez (2010b) found that 70% of good responders, but only 43% of poor responders, remained in the study through third grade. When the attrition of poor responders is twice that of good responders, we might be overestimating the effects of early intervention, because only half the poor responders remain in the study. The problem in the past with identification of students with LD was finding them past the years when early intervention might have greater success. Little ‘‘real reading’’ is expected of young children, and so diagnosing a discrepancy between ability (IQ) and reading achievement was tenuous prior to second or third grade. With an RtI model, students can

168


be identified for Tier 2 interventions with measures of skills (e.g., letter knowledge, phonemic awareness, or word decoding) known to contribute to reading development. Through earlier intervention, we may be able to reduce the severity of reading problems. While applauding strides in the last two decades on development of useful and nimble assessment systems, we argue that much more research on assessment tools is needed, along with their catch and release criteria for students participating in Tier 2 interventions.

REFERENCES Berninger, V. W. (2008). Defining and differentiating dysgraphia, dyslexia, and language learning disability within a working memory model. In: M. Mody & E. R. Silliman (Eds), Brain, behavior, and learning in language and reading disorders (pp. 103–134). New York, NY: Guilford Press. Catts, H., Adlof, S., & Weismer, S. (2006). Language deficits in poor comprehenders: A case for the simple view of reading. Journal of Speech, Language, and Hearing Research, 49, 278–293. Catts, H. W., Fey, M. E., Zhang, X., & Tomblin, J. B. (1999). Language basis of reading and reading disabilities: Evidence from a longitudinal investigation. Scientific Studies of Reading, 3, 331–362. Compton, D. L., Fuchs, D., Fuchs, L. S., Bouton, B., Gilvert, J. K., Barquero, L. A., Cho, E., & Crouch, R. C. (2010). Selecting at-risk first-grade readers for early intervention: Eliminating false positives and exploring the promise of a two-stage gated screening process. Journal of Educational Psychology, 102, 327–340. Coyne, M. D., Kame’enui, E. J., Simmons, D. C., & Harn, B. A. (2004). Beginning reading as inoculation or insulin: First-grade reading performance of strong responders to kindergarten intervention. Journal of Learning Disabilities, 37, 90–104. Deno, S. L. (1985). Curriculum-based measurement: The emerging alternative. Exceptional Children, 52(3), 219–232. Deno, S. L., Reschly, A. L., Lembke, E. S., Magnusson, D., Callender, S. A., Windram, H., & Stachel, N. (2009). Developing a school-wide progress-monitoring system. Psychology in the Schools, 46(1), 44–55. Fuchs, L. S., & Fuchs, D. (1992). Identifying a measure for monitoring student reading progress. School Psychology Review, 21, 45–58. Fuchs, L. S., Fuchs, D., & Hamlett, C. L. (1989). Monitoring reading growth using student recalls: Effects of two teacher feedback systems. Journal of Educational Research, 83, 103–110. Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L., & Germann, G. (1993). Formative evaluation of academic progress: How much growth can we expect? School Psychology Review, 22, 27–48. Fuchs, L. S., Fuchs, D., & Maxwell, L. (1988). The validity of informal reading comprehension measures. Remedial and Special Education, 9, 20–28.


169

Good, R. H., Kaminski, R. A., Smith, S., Laimon, D., & Dill, S. (2001a). Dynamic indicators of basic early literacy skills (6th ed.). Eugene, OR: University of Oregon. Good, R. H., III., Simmons, D. C., & Kame’enui, E. J. (2001b). The importance and decisionmaking utility of a continuum of fluency-based indicators of foundational reading skills for third-grade high-stakes outcomes. Scientific Studies of Reading, 5, 259–290. Jenkins, J. R., Graff, J. J., & Miglioretti, D. L. (2009). Estimating reading growth using intermittent CBM progress monitoring. Exceptional Children, 75(2), 151–163. Jenkins, J. R., & Jewell, M. (1993). Examining the validity of two measures for formative teaching: Reading aloud and maze. Exceptional Children, 59, 421–432. Keenan, J. M., Betjemann, R. S., & Olson, R. (2008). Reading comprehension tests vary in the skills they assess: Differential dependence on decoding and oral comprehension. Scientific Studies of Reading, 12, 281–300. Liberman, I., & Shankweiler, D. (1985). Phonology and the problems of learning to read and write. Remedial and Special Education, 6, 8–17. Linklater, D. L., O’Connor, R. E., & Palardy, G. J. (2009). Kindergarten literacy assessment of English only and English language learner students: An examination of the predictive validity of three phonemic awareness measures. Journal of School Psychology, 47, 369–394. Marcotte, A. M., & Hintze, J. M. (2009). Incremental and predictive utility of formative assessment methods of reading comprehension. Journal of School Psychology, 47, 315–335. McKeown, M. G., Beck, I., & Blake, R. G. K. (2009). Rethinking reading comprehension instruction: A comparison of instruction for strategies and content approaches. Reading Research Quarterly, 44(3), 218–253. O’Connor, R. E. (2000). Increasing the intensity of intervention in kindergarten and first grade. Learning Disabilities Research and Practice, 15, 43–54. O’Connor, R. E., & Bocian, K. (2009). Variations in responsiveness to intervention: Continuation Report. Washington, DC: IES. O’Connor, R. E., Bocian, K., Beebe-Frankenberger, M., & Linklater, D. (2010a). Responsiveness of students with language difficulties to early intervention in reading. Journal of Special Education, 43, 220–235. O’Connor, R. E., Bocian, K., & Sanchez, V. (2009). Response to intervention in Grades K-2 for students learning English as a second language. Paper presented at the annual conference of the Council for Exceptional Children, Seattle, WA, April. O’Connor, R. E., Bocian, K., & Sanchez, V. (2010b). Precision in responsiveness to intervention models. Paper presented at the annual conference of the Council for Exceptional Children, Nashville, TN, April. O’Connor, R. E., Fulmer, D., Harty, K., & Bell, K. (2005a). Layers of reading intervention in kindergarten through third grade: Changes in teaching and child outcomes. Journal of Learning Disabilities, 38, 440–455. O’Connor, R. E., Harty, K., & Fulmer, D. (2005b). Tiers of intervention in kindergarten through third grade. Journal of Learning Disabilities, 38, 532–538. O’Connor, R. E., & Jenkins, J. R. (1999). The prediction of reading disabilities in kindergarten and first grade. Scientific Studies of Reading, 3, 159–197. Paris, S. G., & Hamilton, E. E. (2009). The development of children’s reading comprehension. In: S. Israel & G. Duffy (Eds), Handbook of research on reading comprehension (pp. 32–53). New York: Routledge.

170


Pierce, R. L., McMaster, K. L., & Deno, S. L. (2010). The effects of using different procedures to score maze measures. Learning Disability Research and Practice, 25, 151–160. Roehrig, A. D., Petscher, Y., Nettles, S. M., Hudson, R. F., & Torgesen, J. K. (2008). Accuracy of the DIBELS oral reading fluency measure for predicting third grade reading comprehension outcomes. Journal of School Psychology, 46, 343–366. Sanchez, V. (2010). The reading comprehension puzzle: Reading comprehension and assessment in a response to intervention framework. Ph.D. dissertation, University of California, Riverside, CA. Retrieved on October 16, 2010, from Dissertations & Theses, University of California (Publication No. AAT 3412806). Scanlon, D. M., Vellutino, F. R., Small, S. G., Fanuele, D. P., & Sweeney, J. M. (2005). Severe reading difficulties – Can they be prevented? A comparison of prevention and intervention approaches. Exceptionality, 13, 209–227. Share, D., Jorm, A., MacLean, R., & Matthews, R. (1984). Sources of individual differences in reading acquisition. Journal of Educational Psychology, 76, 1309–1324. Shaywitz, S. E., Gruen, J. R., & Shaywitz, B. A. (2008). Dyslexia: A new look at neural substrates. In: M. Mody & E. R. Silliman (Eds), Brain, behavior, and learning in language and reading disorders (pp. 209–239). New York, NY: Guilford Press. Shelton, N. R., Altwerger, B., & Jordan, N. (2009). Does DIBELS put reading first? Literacy Research and Instruction, 48, 137–148. Shinn, M. R., & Shinn, M. M. (2002). AIMSweb training workbook: Administration and scoring of reading maze for using in general outcome measurement. Eden Prairie, MN: Edformation, Inc. Simmons, D. C., Coyne, M. D., Kwok, O., McDonagh, S., Harn, B., & Kame’enui, E. J. (2008). Indexing response to intervention: A longitudinal study of reading risk from kindergarten through third grade. Journal of Learning Disabilities, 41, 158–173. Sudweeks, R. R, Glissmeyer, C. B., Morrison, T. G., Wilcox, B. R.,. & Tanner, M. W. (2004). Establishing reliable procedures for rating ELL students’ reading comprehension using oral retellings. Reading Research and Instruction, 43, 65–86. Ticha´, R., Espin, C. A., & Wayman, M. M. (2009). Reading progress monitoring for secondary school students: Reliability, validity, and sensitivity to growth of reading-aloud and maze-selection measures. Learning Disabilities Research and Practice, 24(3), 132–142. Tuckwiller, E. D., Pullen, P. C., & Coyne, M. D. (2010). The use of the regression discontinuity design in tiered intervention research: A pilot study exploring vocabulary instruction for at-risk kindergartners. Learning Disability Research and Practice, 25, 137–150. Tunmer, W. E., & Chapman, J. W. (1998). Language prediction skill, phonological recoding ability, and beginning reading. In: C. Hulme & R. M. Joshi (Eds), Reading and spelling: Development and disorders (pp. 33–67). Mawah, NJ: Erlbaum. Vellutino, F. R., Scanlon, D. M., Zhang, H., & Schatschneider, C. (2008). Using response to kindergarten and first grade intervention to identify children at-risk for long-term reading difficulties. Reading and Writing: An Interdisciplinary Journal, 21, 437–480. Wiederholt, J. L., & Bryant, B. R. (2001). Examiner’s manual: Gray Oral Reading Tests-4. Austin, TX: Pro-Ed. Woodcock, R. W. (1998). Woodcock Reading May Tests – Revised Examiner’s Manual. Circle Pines, MN: American Guidance Services.

COMORBIDITY BETWEEN ATTENTION DEFICIT HYPERACTIVITY DISORDER AND READING DISABILITIES: IMPLICATIONS FOR ASSESSMENT AND TREATMENT Ana Miranda, Marı´ a Jesu´s Presentacioń, Rebeca Siegenthaler, Carla Colomer and Vicente Pinto ABSTRACT Comorbidity of attention deficit hyperactivity disorder (ADHD) and reading disabilities (RD) is greater than what would occur by chance. Considering the well-documented adverse impact of both ADHD and RD on development, the presence of both conditions may lead to particularly poor outcomes for affected people. This chapter, which reviews 43 research studies carried out in the last decade that have focused on the link between ADHD and RD, is divided into two broad nuclei of contents. First, studies are described that contribute information about characteristics of the comorbid phenotype. Second, studies related to procedures Assessment and Intervention Advances in Learning and Behavioral Disabilities, Volume 24, 171–211 Copyright r 2011 by Emerald Group Publishing Limited All rights of reproduction in any form reserved ISSN: 0735-004X/doi:10.1108/S0735-004X(2011)0000024010

171

172

MIRANDA ET AL.

directed toward evaluation and intervention in this problem are analyzed. The review carried out does not make it possible to extract definitive results on the exact nature of ADHD and RD comorbidity or, even less, reach conclusions about its causes. However, the literature-based evidence shows a cognitive profile of ADHD þ RD characterized by failure of various functions that can produce more severe functional deficits and worse neuropsychological, academic, and behavioral outcomes. Furthermore, the analysis of the set of results from the studies shows a limited efficacy of pharmacological and psychopedagogical treatments, and highlights the need for continued research on this topic. From a clinical and educational standpoint, the conclusions derived from this review underline the importance of performing an exhaustive evaluation of children and adolescents with symptoms of ADHD and/or RD, in order to be able to plan interventions with greater possibilities of success in each case.

Attention deficit hyperactivity disorder (ADHD) and reading disabilities (RD) are two of the most frequent childhood developmental disorders. Each of them affects between 6% and 9% of school-age children, although these percentages decline to 5% when the DSM-IV-TR (2000) criteria are applied. Subjects with RD are characterized by a reading performance, measured by individually administered normalized accuracy or comprehension tests, that is far below what would be expected given the child’s chronological age, IQ, and corresponding grade in school. Historically, experts have basically agreed in identifying a series of abilities or skills generally falling under the epigraph of phonological processing as responsible for RD in alphabetic languages. Various parameters can reflect adequate or inadequate phonological processing and, therefore, differentiate between good and poor readers: decoding skills (‘‘phonological decoding’’ or ‘‘word decoding’’), phonological awareness (‘‘phoneme awareness’’), and lexical access speed. In addition, although less important and less frequent, difficulties in performing naming speed (‘‘rapid-naming skills’’), verbal stimuli (letters, words, digits) and, even, nonverbal (colors, drawings) tasks (Pennington, Groisser, & Welsh, 1993; Purvis & Tannock, 1997) have been highlighted. For some authors, this double deficiency detected in people with dyslexia reflects the intervention of processes independent from reading, giving rise to interesting proposals like the double-deficit model by Wolf and Bowers (1999).

Comorbidity ADHD and RD

173

Furthermore, other studies suggest that in the RD groups there is an association between phonological processing dysfunctions and a marked inability to store and process verbal material in short-term memory (Rapala & Brady, 1990; McDougall, Hulme, Ellis, & Monk, 1994). The majority have specifically related reading problems to a deficit in the phonological loop that could damage, for example, temporal auditory processing. However, only a few have suggested that there could also be difficulties in the other essential component of working memory (WM) storage, the visuospatial, or even in some of the central executive (CE) functions. Generally, although not all the questions have been answered, it is possible to conclude that there is a large body of empirical evidence showing that phonological storage and processing skills would be more affected than executive functioning processes in RD, in contrast to the tendency usually found in ADHD. On the other hand, ADHD is a persistent pattern of inattention and/or hyperactivity/impulsivity, more frequent and serious than what is usually observed in subjects at a similar developmental level. Basing their work on the similarity between ADHD symptoms and the consequences of frontal lobe damage, many authors have pointed out that in ADHD the executive functions (EF), understood as the set of cognitive functions involved in the problemsolving processes when trying to achieve a certain objective or goal, might be especially affected. These executive functions encompass many domains: problem-solving strategies, inhibitory control at a cognitive and/or motor level, WM, self-regulation, cognitive flexibility, interference control, or planning. In general, the neuropsychological models of ADHD recognize the important role of the EF in this disorder (Barkley, 1997; Rapport, Chung, Shore, & Isaacs, 2001). However, the empirical findings in this regard are not as consistent as might be expected. The meta-analytic review by Willcutt, Doyle, Nigg, Faraone, and Pennington (2005) suggests that ADHD is associated with weaknesses in several key EF domains. The strongest and most consistent effects were obtained on measures of response inhibition, vigilance, spatial WM, and some measures of planning. This review also points to the existence of dysfunctions with very little executive charge in ADHD, dysfunctions that affect, for example, processing speed, naming speed, and fine and gross motor skills. In sum, there is sufficient evidence to defend the existence of a primary executive dysfunction in ADHD, but when the effect size of the EF in groups with and without ADHD is analyzed, the results only show a moderate effect, suggesting, therefore, that EF weaknesses are neither necessary nor sufficient to cause all cases of ADHD.

174

MIRANDA ET AL.

It has been demonstrated that ADHD and RD present a higher index of cooccurrence than what would be produced by mere chance. Thus, the estimations of RD among children diagnosed with ADHD range, according to different authors, from 10 to 40% (Del’Homme, Kim, Loo, Yang, & Smalley, 2007). On the other hand, among the population of children diagnosed with RD, the presence of ADHD has been estimated at between 15 and 40% (Shaywitz, Fletcher, & Shaywitz, 1995; Willcutt & Pennington, 2000). By themselves, these data are sufficiently noteworthy to justify researchers’ interest not only in studying the two disorders separately, but also in analyzing their high comorbidity. Considered in isolation, the basic symptomatology that defines RD and ADHD does not coincide, and the research has focused on identifying the combination of dysfunctions that could constitute the main deficit in the ADHD þ RD group. These dysfunctions must, logically, affect phonological processing (PP), WM, and executive functioning. In addition, the high coincidence of the two disorders (ADHD þ RD) poses additional questions for investigation: What are the specific psychocognitive characteristics of this group? Are these characteristics similar to or different from those exhibited by the pure ADHD and RD groups? Can one of the disorders be considered more ‘‘primary’’ than the other? How should this group be evaluated? Should the treatments for the ADHD þ RD group include phonological processing and EF, or should they focus on one specific aspect? In an attempt to respond to these and other similar questions, different hypotheses have been proposed. The phenocopy hypothesis defends the idea that the only neuropsychological functions affected in ADHD þ RD would be those characteristic of the RD group, related to phonological processing (Pennington et al., 1993). The cognitive subtype hypothesis considers the ADHD þ RD group as a specific disorder (Rucklidge & Tannock, 2002a), and predicts that the neuropsychological impairments follow an interactive pattern, different from those that characterize pure ADHD and RD. The hypothesis of a common genetic origin for both disorders (Willcutt et al., 2003) predicts the existence of some neuropsychological deficits common to RD, ADHD, and ADHD þ RD. De Jong, Oosterlaan, and Sergeant (2006) carried out an excellent review of the bibliography on the comorbidity of ADHD and RD, analyzing the repercussions of the results in identifying ADHD and RD endophenotypes and their comorbidity. These researchers concluded that executive functioning deficits (particularly in inhibition and WM) appear to be an endophenotypic candidate for the comorbidity of ADHD and RD. The

175


present study, along the lines of De Jong et al. (2006), is designed to examine the most recent studies that contribute to the knowledge base about the cognitive profile of the comorbid group (ADHD þ RD). Furthermore, completing the analysis of comorbidity, a second objective of this study consists of reviewing the studies on diagnosis and treatment focused on the specific problematic of the ADHD þ RD group, rating their usefulness in order to act with the greatest efficacy in the practical realm.

METHOD Systematic literature searches were conducted using PsycINFO, ERIC, and Medline databases from 2000 to July 2010. The terms ADHD, attention deficit disorder, and attention deficit/hyperactivity disorder were crossreferenced separately with reading disability, reading disorder, and dyslexia. The tables of contents of journals that commonly publish articles relevant to this topic were also reviewed (Exceptional Children, Journal of Abnormal Child Psychology, Journal of the American Academy of Child and Adolescent Psychiatry, Journal of Attention Disorders, Journal of Learning Disabilities and Learning Disability Quarterly). The broad search strategy resulted in a total of 514 articles. In an initial filtering, articles were eliminated that were duplicated (145), had only a theoretical nature (e.g., reviews that did not include an empirical study) (139), or were not strictly scientific articles, such as dissertation abstracts (29), erratum, introductions to a special issue or summaries (28). In the second filtering, among the empirical studies, articles were eliminated that focused on the analysis of causal biological factors (e.g., genetic, neurological, perinatal risk) (49), or did not include the ADHD þ RD comorbid group (81).

RESULTS Finally, a total of 43 publications met the conditions established in the present study. Of them, 17 analyzed specific components of the set of cognitive processes that seem to be involved in the relationship between ADHD and RD by comparing the performance of the pure groups and the combined group. Another 13 studies, with more complex proposals, combined measures of various cognitive domains in order to test the double dissociation hypothesis and further examine the profile of the group with both problems and the nature of the comorbidity. Finally, 13 studies

176

MIRANDA ET AL.

were designed to test the efficacy of evaluation tests and treatments in these clinical groups. The following sections review the most interesting results from the research in each of these categories.

ADHD AND RD: SINGLE DISSOCIATION STUDIES The majority of the set of more specific studies have focused on the comparative analysis of the groups on measures related to reading (phonological awareness and naming speed, above all) or actual reading performance (reading words and pseudowords, text reading, and reading comprehension). Other investigations have concentrated on the study of cognitive processes linked to executive functioning. Finally, a few studies have examined the repercussions of ADHD and RD and/or their association in the academic and socio-emotional spheres, providing information to further understand their transcendence in functional terms (Table 1).

Reading Processing One of the most complete studies among those that have evaluated reading in these groups was conducted by Ghelani, Sidhu, Jain, and Tannock (2004). These authors applied a battery of tests of reading component processes (lexical access, text reading, reading comprehension, and naming speed) to adolescents with ADHD, RD, RD þ ADHD, and controls. Regarding lexical access and text reading, Ghelani et al. (2004) found that, while the adolescents with RD (RD and ADHD þ RD groups) showed significant difficulties on all the word, pseudoword, and text-reading measures, the adolescents with ADHD showed adequate reading skills, although with subtle difficulties in text speed and accuracy and word speed. Reading comprehension was evaluated by means of a read-aloud task and another silent-reading task, which showed different deficits in the groups on these measures. Specifically, the adolescents with ADHD had slight difficulties in silent-reading comprehension. The adolescents with RD (problems in word identification and word attack) obtained scores on reading comprehension (oral and silent), which, although low compared to the control and ADHD groups, were within normal range. As expected, the comorbid group performed similarly to the RD group on the silent-reading tasks, but they had better scores on oral comprehension. One noteworthy finding is that, while both groups of adolescents with RD (alone and with ADHD) showed a generalized

N ¼ 142 7–14

N ¼ 150 7–14

N ¼ 476 7–12

N ¼ 92 X ¼ 8.2

Breier et al. (2002)


Capano et al. (2008)

Chan et al. (2008)

A, R þ A

A, A þ R, A þ M, AþRþM

A, R, R þ A, C

A, R, R þ A, C

A, R, R þ A, C

N ¼ 95 7–15


Groups

A, A (bright readers), A þ R, A þ R (low)

Sample/ age

Bonafina et al. (2000) N ¼ 54 7–11

Study

Table 1.

Spelling

Language receptive Language expressive Reading achievement Mathematical achievement Naming speed Phonologic processing Orthographic processing

CELF-3 CELF-3 WRAT-3, WRMT WIAT HKT-SpLD

Arithmetic Conductual functioning Auditory temporal functioning CTOPP Phonologic processing Discrimination/TOJ task (2/ Auditory temporal 3 elem.) functioning (verbal) CTOPP, RAN Auditory temp. functioning (no verbal) Phonologic processing Psychoacoustics tests Auditory temporal (BMLD, GDT, TOT) functioning Auditory no temporal functioning Auditory functioning temporal/no temporal WISC-R CI

WRAT-R CBCL, ICTQ VOT/TOT

WRAT-R

Measures

Single Dissociation Studies.

A þ M, A þ R, A þ R þ M oA; A þ R þ MoA þ R A þ R þ Mo A þ M, A þ RoA A þ R þ Mo A þ R oA; A þ R þ MoA þ M A þ R þ Mo A þ RoA þ M oA A þ R þ MoA þ MoA þ R oA R þ AoA n.d. R þ AoA

A, R þ AoC, R

n.d.

R, R þ AoC, A R, R þ AoC, A

n.d.

R, R þ AoC, A R, R þ AoC, A

A þ R groupsoA groups n.d. R, R þ AoC, A

A þ R groupsoA groups

Results

Comorbidity ADHD and RD 177

N ¼ 383 3–5

N ¼ 48 9–10

Pisecco et al. (2001)

Roodenrys et al. (2001)

Seidman et al. (2001) N ¼ 275 6–17

N ¼ 143 7–13

Martinussen and Tannock (2006)

A, A þ R, A þ M, A þ R þ M, C

R, A þ R, C

A, R, R þ A, C

A, R, A þ R, C

A, R, R þ A, C

N ¼ 119 18–55

Laasonen et al. (2009)

A,R, R þ A,C

A, R, A þ R, A þ O, A þ R þ O, C

N ¼ 113 6–15

Kibby and Cohen (2008)

A, R, A þ R, C

Groups

Kooistra et al. (2005) N ¼ 291 X ¼ 11.7

N ¼ 96 14–17

Sample/ age

Ghelani et al. (2004)

Study

Visual-motor integration

VMI

Digits Forward FW (WRALM) Digits Backward RDLS RDLS Lack of control scale Span tasks CHIPASAT Memory Updating Random Generation Broad battery of neuropsychological test

WAIS-III

BOTMP

CMS

A main effect R main effect n.d. R, A þ O, A þ R, A þ R þ OoA, C A, R, A þ O, A þ R, A þ R þ OoC R, R þ AoA, C

R, A þ RoA, C

R, A þ RoAoC n.d. A þ Ro AoC; RoC

R, A þ RoAoC

WRALM list learning Auditory CPT

A þ RoA, C A, CoA þ R

A, R, R þ AoC Verbal storage R, A þ RoA, C Spatial storage and CE A, R, A þ RoC Verbal CE A, R, A þ RoC Receptive language R, R þ AoC (3/5 years) Expressive language RoC (3 years) Lack of control R þ AoA, R, C Phonological loop R, A þ RoC Phonological Loop and CE A þ RoRoC Phonological Loop and CE A þ RoR, C Central Executive A þ RoR, C Stroop A þ RoA, C

Verbal WM: processing speed

Rapid word reading Reading comprehension Rapid naming speed (objects, color) Rapid naming speed (letters, digits) Visuospatial STM Verbal STM LTM and CE Motor skills (gross, fine)

TOWRE GORT-4, GSRT CTOPP CTOPP

Reading rate and accuracy

Results

GORT-4

Measures

Table 1. (Continued ).

178 MIRANDA ET AL.

Study 1: N ¼ 100 6–11 Study 2: N ¼ 98 13–16

N ¼ 306 8–18

Toplack et al. (2003)

Willcutt et al. (2007)

A, R, R þ A, C

A, A þ R, C

A, A þ R, C

A, R, A þ R, C

CBCL, YSR CBCL, YSR CBCL, YSR Anxiety/Depression

Psychophysical task

Finger tapping tasks

Estudio 1: A, A þ RoC Estudio 2: A þ RoA, C Estudio 1: A þ RoA, C Estudio 2: A þ RoA, C R þ AoAoC R þ AoAoR, C A, R þ AoRoC A, R þ AoRoC

Duration

Reproduction Academic outcomes Socials outcomes Externalizing comorbidity Internalizing comorbidity

n.d. n.d. n. d. R, A þ RoC, A n.d.

Free tapping Synchronized tapping Unpaced tapping Rhythm reproduction Frequency

A: ADHD only; C: Control; M: mathematical disorder; R: RD only; O: ODD; n.d.: no differences. BMLD: Binaural Masquing Level Differences; BOTMP: Bruininks–Oseretsky Test of Motor Proficiency; CBCL: The Child Behavior Checklist; CE: Central Executive; CELF-3: Clinical Evaluation of Language Function-3; CHIPASAT: Children’s Paced Auditory Serial Addition Task; CMS: Children’s Memory Scale; CTOPP: Comprehensive Test of Phonological Processing; FW: Finger Windows Task; GDT: Gap Detection Threshold; GORT-4: Gray Oral Reading Test – Fourth Edition; GSRT: Gray Silent Reading Test; HKT-SpLD: Hong Kong test of Specific Learning Difficulties in Reading and Writing; ICTQ: IOWA Conners Teacher’s Questionnaire; LTM : Long Term Memory; PIAT-R: Peabody Individual Achievement Tests Revised; RAN: The Rapid Automatized Naming Subtests; RAP: Rapid Auditory Processing; RDLS: Reynell Developmental Language Scales; STM: Short Term Memory; TOJ: Temporal Ordering Judgment Task; TOT: Tone Onset Time; TOWRE: Test of Word Reading Efficiency; VF: Visual Filtering Test; VMI: Visual–Motor Integration Test; VOT: Voice Onset Time; WIAT: Wechsler Individual Achievement Test; WISC-R: Wechsler Intelligence Scale for Children – Revised; WRALM: Wide Range Assessment of Memory and Learning; WRAT-3: Wide Range Achievement Test; WRAT-R: Wide Range Achievement Test – Revised; WRMT: Woodcock Reading Mastery Test Revised; YSR: Youth Self Report.

N ¼ 68 10–13

Tiffin et al. (2004)


180

MIRANDA ET AL.

impairment in naming speed for all types of stimuli (words, digits, colors, and figures), the group with ADHD showed a low response only on colors and figures. Ghelani et al. (2004) concluded that these results could provide evidence of the existence of a processing speed problem related to inattention. The importance of naming speed in RD þ ADHD in Chinese readers who use a nonalphabetic system (Cantonese) has also been highlighted by Chan, Hung, Liu, and Lee (2008). In Cantonese-speaking children with RD þ ADHD and pure ADHD, these researchers analyzed the prevalence of different impairments related to reading problems: phonological processing, naming speed, and orthographic processing. They found that the RD þ ADHD group had a significant impairment in naming speed (62.8% of the cases). The combined group also experienced deficits in orthographic knowledge (37.2%) and, to a lesser degree, phonological problems (11.6%). With regard to the number of concurrent reading deficiencies, 41.9% of the Chinese children with RD þ ADHD had only one cognitive impairment, 39.5% had two impairments, and 4.7% presented three impairments, with the most common being the combination of naming speed and orthographic processing deficits. Furthermore, the fact that the important naming speed problems of the RD þ ADHD children do not occur to the same degree in children with ADHD alone (18.4%) provides support for the hypothesis of a specific naming speed deficit in RD.

Time Perception One of the explanatory hypotheses for identification of ADHD is the existence of a deficit in time perception. Toplack, Rucklidge, Hetherington, John, and Tannock (2003) studied time perception in children and adolescents with ADHD (with and without RD), using two tasks related to the duration of tones. The first task consisted of determining which was the longer of two sounds (duration discrimination). The second consisted of reproducing the duration of tones (duration estimation). These authors also used a third control task consisting of frequency discrimination. Both groups (ADHD and ADHD þ RD) presented some alterations on the duration discrimination tasks and also on the accuracy with which they reproduced the intervals on the duration estimation task, although the comorbid group was affected more severely. However, there were no alterations on the control task of frequency discrimination. Likewise, both the participants with ADHD and those with ADHD þ RD showed a significant intraindividual variability on the duration estimation task. The authors concluded that deficits related to


181

time estimation are associated with ADHD, and more consistently with ADHD þ RD, and that these deficits, together with the great intraindividual variability, can have cascade effects in the time organization of the behavior in children and adolescents with these disorders. Nevertheless, the absence of an RD group in the study by Toplack et al. (2003) makes it impossible to draw solid conclusions about time perception in this clinical group and its differences with the ADHD groups. Somewhat different results on time perception were obtained by TiffinRichards, Hasselhorn, Richards, Banaschewsky, and Rothenberger (2004), who examined the performance of German children using a computermediated finger-tapping paradigm. The time perception of auditory stimuli was evaluated using discrimination tasks (which involve only time perception) and reproduction tasks (which also require motor responses). Tiffin-Richards et al. (2004) did not find differences among the groups (ADHD, RD, ADHD þ RD, and control) on any task, which would indicate that the capacity to maintain a certain response rate (free-tapping) or adjust the response to a certain stimulus pattern (synchronized tapping) was not affected in any of the clinical groups. Furthermore, the groups analyzed did not show differences in intraindividual variability when reproducing different intervals. However, without reaching significance, the RD and ADHD þ RD groups showed special difficulties in reproducing rhythmic patterns when the complexity of these patterns was increased.

Temporal Auditory Processing In order to analyze the hypothesis of the temporal auditory deficit in RD and its relationship with ADHD, different psychoacoustic tasks have been used by Breier and their collaborators: psychoacoustic tasks of continuous voice onset time, continuous tone onset time, judging temporal order and discrimination, with verbal auditory stimuli (phonemes), and nonverbal ones (tones). In contrast to their first results (Breier et al., 2001), the latest studies of children with RD and/or ADHD have not completely confirmed the existence of a temporal auditory processing deficit in any of the cases. Instead, they provide support for the existence of a specific impairment in language perception in children with RD. It was observed that the groups with RD did not present a specific sensitivity to the intervals between stimuli (ISI), but they performed worse on tasks with verbal stimuli than children without RD, and this occurred independent of the presence of ADHD (Breier , Gray, Fletcher, Foorman, & Klaas, 2002). When analyzing the

182

MIRANDA ET AL.

temporal and nontemporal auditory perception of nonverbal stimuli, a deficit was found only in the detection of tones with an asynchronous temporal onset in children with RD. However, the presence of ADHD was associated with a general reduction in performance across tasks. The pattern of results did not indicate a pervasive impairment in auditory temporal function in children with RD, but it did suggest a possible sensitivity to backward masking in this group. Results also indicated that the comorbid presence of ADHD is a significant factor in the performance of children with RD on psychoacoustic tasks (Breier, Fletcher, Foorman, Klaas, & Gray, 2003). An additional finding was that there was a significant correlation between phonological processing (phonological awareness and decoding) and auditory processing measures, although only in the groups without ADHD (controls and RD). Motor Skills Kooistra, Crawford, Dewey, Cantell, and Kaplan (2005) analyzed different motor measures (gross, fine, and integrated) in ADHD and their relationship to the presence of other disorders such as RD and ODD. The researchers compared the performance of children with ADHD, RD, ADHD þ RD, ADHD þ ODD, ADHD þ RD þ ODD, and controls on the Bruininks– Oseretsky Test of Motor Proficiency (BOTMP) and the Beery Test of Visual– Motor Integration (VMI). Kooistra et al. (2005) found that all the groups, except pure ADHD, scored lower than the control group on the gross and fine motor skills test. On the VMI, all the clinical groups performed significantly worse than the control. The results suggest that the motor problems of ADHD appear to be associated with the presence of comorbidity, and that the level of RD seems to be a key factor in predicting them. Working Memory The studies have been based on the Baddeley WM model, which proposes the subdivision of the WM into three components: a control system with limited attentional capacity called the CE, assisted by two subsidiary slave systems, the phonological loop, responsible for temporary storage and processing of verbal information; and the visuospatial sketchpad, specialized in short-term retention of visual information. Roodenrys, Koloski, and Grainger (2001) investigated the phonological loop and CE in children with RD and both RD and ADHD, showing that


183

the children with RD and ADHD þ RD were impaired in their use of the phonological loop compared to the control group. Likewise, both groups, RD and ADHD þ RD, performed worse than normal controls on tasks involving a combination of the phonological loop and CE, with the comorbid group showing poorer functioning than the RD group. Finally, ADHD þ RD performed worse than RD and control groups on tasks that only evaluated the CE. The authors concluded that the data demonstrate a deficit in phonological loop functioning associated with RD, and an independent deficit in the CE associated with ADHD. However, children with only ADHD did not participate in their investigation, so this study cannot directly demonstrate that CE deficits are specific to ADHD. Martinussen and Tannock (2006) found that verbal and spatial CE and spatial storage were affected in children with ADHD, RD, and ADHD þ RD, while only the two groups with RD showed an additional deficit in verbal storage. Specifically, symptoms of inattention, and not those of hyperactivity/impulsivity, were associated with memory impairments. Furthermore, these authors found that reading achievement and inattention symptoms contributed unique variance to verbal CE processing, and they concluded that verbal CE deficits (WM) may be a common neuropsychological weakness in children who exhibit inattention symptoms and/or language learning problems. Another study by Kibby and Cohen (2008) provided information confirming the consistency of verbal memory deficits affecting RD and RD þ ADHD groups. They applied the Children’s Memory Scale (CMS: Cohen, 1997) to children with ADHD, RD, RD þ ADHD, and controls. The results of between-groups comparison analyses showed that spatial memory was partially affected in the ADHD group, especially in subjects who were not taking medication. However, the RD group was affected on verbal memory tasks, specifically on tasks that require a phonetic coding of the material, as their coding by meaning was correct. Finally, the RD þ ADHD groups had impairments consistent with both disorders, but no additional deficits. Laasonen, Leppa¨ma¨ki, Tani, and Hokkanen (2009) administered the WAIS-III to a sample of adults from the Adult Dyslexia and Attention Deficit Disorder Project in Finland (Project DyAdd). The findings demonstrated that only the two groups with dyslexia, with and without ADHD, showed a poorer performance on WM, while all the clinical groups were poorer in processing speed than controls. In addition to WM, some studies have also evaluated an extended spectrum of executive functions. Seidman, Biederman, Monuteaux, Doyle,

184

MIRANDA ET AL.

and Faraone (2001) evaluated children with ADHD with and without learning disabilities (LDs), both in reading and in arithmetic, comparing them with a control group on a broad battery of neuropsychological tests. In their analysis, participants with ADHD þ RD performed significantly worse than ADHD and controls on the Stroop test and on a list-learning test (WRAML), although, surprisingly, they performed better on the auditory CPT (continuous performance test). The authors found that more severe neuropsychological dysfunctions were associated with comorbid LDs in ADHD children, especially when arithmetic difficulty was present.

Academic and Socio-Personal Adaptation The studies on this topic agree that the academic failure of children with ADHD seems to worsen when there is an associated RD. Bonafina, Newcorn, McKay, Koda, and Halperin (2000), for example, studied the academic and behavioral functioning of children with ADHD divided based on their full scale IQ (FSIQ) and reading ability. Generally, the academic performance in language and mathematics of children with ADHD and reading problems was significantly lower than that of children with ADHD but no RD. With regard to behavioral functioning, there were no significant differences in behavioral problems (ODD and conduct disorder, CD) among the four groups, although there was a tendency toward a greater problematicity in the two groups with more extreme reading levels (brilliant and severe RD). The relationship of mathematics disorders (MD) and RD in school children with ADHD has also been explored by Capano, Minden, Chen, Schachar, and Ickowicz (2008). For this purpose, they evaluated a large sample of children with ADHD, ADHD þ RD, ADHD þ MD, and ADHD þ RD þ MD on cognitive, language, mathematical achievement, and reading tests. Capano et al. (2008) found that MD was common in school-age children with ADHD (18.1% of MD þ ADHD) and frequently associated with RD (9.9% of ADHD þ RD þ MD). Their results also showed that the ADHD and RD association (just as with MD) is related to lower scores on IQ, language, and academic variables, than ADHD alone. However, the ADHD þ RD þ MD group was the most affected, which would indicate a unique biological cause. The longitudinal study by Willcutt, Betjemann, Pennington, Olson, DeFries, Wadsworth (2007) shows special interest in closely examining the academic, social, and behavioral functioning of groups with RD and/or


185

ADHD. Their sample is composed of children and adolescents identified in the Colorado Learning Disabilities Research Center (CLDRC) twin study and reevaluated after a period of five years, when they were adolescents or young adults. The findings confirmed that the comorbidity of RD and ADHD is frequently stable over time. Even when children with RD or ADHD do not meet the criteria for both disorders, often over time their scores for the other disorder rise. Regarding the groups analyzed, it was shown that the initial RD were associated with high scores on academic difficulties, depression, and behavioral problems. The initial ADHD group was related to academic (significant, although less than the RD group) and social difficulties. This group, together with the RD þ ADHD group, showed high scores on almost all the comorbid disorders (anxiety, depression, ODD, CD, alcohol, and marijuana abuse) five years later. Moreover, the group with RD þ ADHD also presented more stable deficits in reading than the RD, and was more affected than the pure RD and ADHD groups on measures of academic functioning and social difficulties. They also received more psychostimulant medication and special education. These results confirm that the cooccurrence of RD and ADHD increases the risk of negative results in adolescents and young adults, and they highlight the need for interventions for this group that take both disorders into account. Finally, the longitudinal study by Pisecco, Baker, Silva, and Brooke (2001) looked closely at the possible existence of initial temperamental and linguistic markers of RD and ADHD. Pisecco et al. (2001) investigated whether children who showed clear enough symptoms to receive a diagnosis of RD and/or ADHD at the age of 11 had presented differences in linguistic skills and temperamental traits (‘‘locus of control’’) when they were 3 and 5 years old. In their study, the participants were children with ADHD, RD þ ADHD, RD, and controls, all of them taken from the Dunedin Multidisciplinary Health and Development Study (New Zealand). The authors expected that the children who had RD at the age of 11 would have shown deficits in linguistic skills (reception and expression) at the ages of 3 and 5, and that the children in the groups with ADHD would exhibit more ‘‘low control’’ behaviors at the ages of 3 and 5. They found that boys from the RD groups consistently exhibited deficiencies in receptive language; specifically, the children with reading disabilities and ADHD experienced the most significant receptive language impairment. In regard to expressive language, children from the RD-only group exhibited significantly more language-related problems at age 3 than the children from the comparison group. For locus of control, at 3 and 5 years, only the group that would later

186

MIRANDA ET AL.

receive a diagnosis of RD þ ADHD showed a significantly greater number of ‘‘uncontrolled’’ behaviors. In contrast to what was expected, locus of control was not an early indication of greater risk of having ADHD than RD.

ADHD AND RD: DOUBLE DISSOCIATION STUDIES In double dissociation studies, processes are selected that are presumed to be primarily impaired in one disorder, but not in the other, so that a classic double dissociation occurs when two disorders are associated with opposite patterns of impairment in two different cognitive domains. The findings offer important information for increasing knowledge about the comorbid group and explaining the comorbidity between ADHD and RD (see Table 2). Several studies support a partial double dissociation between ADHD and RD, without revealing common deficits between the two disorders and comorbid group showing the additive combination of the deficits associated with each single disorder. Willcutt et al. (2001) compared individuals with ADHD, RD, RD þ ADHD, and controls on measures of executive functioning and phonetic awareness. They found that children with ADHD had difficulties in executive functioning, specifically in inhibition, but not in WM, set-shifting, or phonological processing, when intelligence was introduced as a covariable in the analyses (FSIQ was covaried), while the children with RD exhibited phonological deficits and verbal working memory (VWM) deficits. The comorbid group showed impairments on virtually all measures. Therefore, these authors found a partial double dissociation between RD and ADHD, with phonological processes and VWM deficits associated with RD but not with ADHD, and inhibitory deficits associated with ADHD but not with RD. Different naming speed tasks (colors and digits), together with balancing tests were applied to children with RD, ADHD, RD þ ADHD, and controls in a study by Raberger and Wimmer (2003). The most noteworthy findings show an association between poor performance on naming speed tasks and RD, and a relationship between poor balance and ADHD. Consistent with the central prediction of a double dissociation, Weiler, Bernstein, Bellinger, and Waber (2002) found that children with ADHD— predominantly inattentive type (ADHD/IA) differed from those without an attention disorder on the Serial Search task but not on the auditory processing task; the reverse was true for children with RD. Weiler et al. (2002)

Measures

Raberger & Wimmer (2003) N ¼ 40

Purvis & Tannock (2000) N ¼ 68

McGee et al. (2004) N ¼ 113

De Jong et al. (2009a) N ¼ 120

Time perception Time perception Inhibition Inhibition

CPT (dur. est.)

Time production

Stop task

Balancing tasks

Balancing

Impulsivity Spatial WM Vigilance

SSP (commission) CBTT CPT

CPT

Attention

Inhibition Attention Planning Set-shifting Verbal WM Verbal fluency Inhibition

A main effect

R main effect, A trend

A main effect

n.d.

A main effect

A and R main effect A main effect AoR, A þ R, C n.d.

R main effect

n.d. A, A þ RoR, C n.d. n.d. R þ AoA, C n.d.

Auditory analysis Phoneme segm. Continuous rapid naming

PST Naming task

Word attack

Auditory ph. awareness

Decoding process

Lexical decision

Ph. decoding Spelling accuracy R. comprehension Rapid naming Phonemic synthesis Phonemic deletion

TAA

WAS

Sound blending

Lexical decision (Pseudowords)

NDB THWQ RRP-ESAC RAN (digits) NDB NDB

Linguistic Skills Measures

Double Dissociation Studies.

Results

Executive Functions

SSP (omission)

SSP (RT)

Bental & Tirosh (2007) MFFT N ¼ 86 CRS Porteus Maze Test WCST Span tasks Word fluency

Study

Table 2.

R large effect, A small effect R, R þ AoAoC R main effect

R main effect

R main effect

R, A þ RoAoC

A and R main effect.

R, A þ RoA, C R, A þ RoA, C n.d. A þ RoA, C n.d. A, R, A þ RoC

Results


A þ RoC

Resp. execution

Weiler et al. (2000) N ¼ 82 Weiler et al. (2002) N ¼ 230

Proc. speed Visual proc. speed

VF

WM baseline WM

N-back (0) N-back (1)

Proc. speed tasks

Inhibition

Go/no-go

Set-shifting

WCST

Van de Voorde et al. (2010) N ¼ 76

Proc. speed WM

TT Digits backward

N ¼ 395 Tiffin-Richards et al. (2008) N ¼ 79

A main effect

A main effect A and R main effect A þ RoAoR

A þ RoA, R, C

A and R main effect R main effect A and R main effect A main effect

A, A þ RoC

Proc. speed Proc. speed

A þ RoA, C

WM

A þ RoA, C

A, R þ AoC

Results Inhibition

Motor output

Arithmetic, span task, digits task Symbol search/coding

Stop task

Measures

Executive Functions

Shanahan et al. (2006)

Rucklidge & Tannock (2002a) N ¼ 108

Study

Written language Oral language RAP

RAN

Ph. tasks

Morphosyntactic

RAN (numbers) Mottier Test

Verbal output

Linguistic Skills

Written language Oral language Auditory proc speed

Naming speed

Morphosyntactic processing Ph. processing

Proc. speed Phon. ST memory

Proc. speed

Naming speed

Naming speed

Naming speed

Naming speed

Naming speed

Measures

Stroop (color-word)

RAN (objects)

RAN (letters)

RAN (colors)

RAN (numbers)


R, A þ RoA n.d. R main effect

R main effect

R main effect

R main effect

R main effect R main effect

A þ RoA, R, C; A þ RoA A, A þ RoC; A þ RoR A, A þ RoC; A þ RoA A and R m. e.

A þ RoA, R, C

A þ RoA, C

Results

188 MIRANDA ET AL.

Inhibition Set-shifting WM Inhibition Set shifting Verbal WM Proc. speed

CPT, stop task WCST, CNT Span tasks, TT CPT, stop task

WCST, TT WM tasks Proc. speed task

PLT, PDT, LAC

PLT, PDT, LAC

n.d. ONR R, R þ AoAoC LD-O A, R, R þ AoC

A main effect n.d. R main effect A, R, R þ AoC Ph. decoding Orthographic cod.

Phoneme awareness

Phoneme awareness

R, R þ AoA, C R, R þ AoAoC

R, R þ AoA, C

R main effect

A: ADHD only; A þ R: ADHD þ RD; C: controls; R: RD only; R þ A: RD þ ADHD; n.d.: no differences. CBTT: Corsi Block Tapping Test; CNT: Contingency Naming Test; cod.: coding; CPT: Continuous Performance Test; CRS: Conners’ Rating Scales; dur. est.: duration estimation; LAC: Lindamood Auditory Conceptualization Test; LD-O: Lexical Decision by Orthography; MFFT: Matching Familiar Figures Test; NDB: Nitzan Diagnostic Battery; ONR: Oral Nonword Reading Task; PDT: Phoneme Deletion Task; Ph.: phonological; PLT: Pig Latin Test; Proc.: Processing; PST: Phoneme Segmentation Task; R.: Reading; RAN: Rapid Automatized Naming; RAP: Rapid Auditory Processing; RRP-ESAC: Report of Reading Performance of Elementary School Age Children; RT: Reaction Time; segm.: segmentation; SSP: Stop Signal Paradigm; ST: Short term; TAA: Test of Auditory Analysis; THWQ: Test of Hand Writing Quality; TT: Trailmaking Test; VF: Visual Filtering Test; WAS(WJMRT-R): Word Attack Subtest of the Woodcock–Johnson Reading Mastery; WCST: Wisconsin Card Sorting Test; WM: Working Memory.

Wilcutt, Pennington, Olson et al. (2005) N ¼ 437

Willcutt et al. (2001) N ¼ 314


190

MIRANDA ET AL.

applied a task of rapid auditory processing, which consisted of identifying whether pairs of tones of different durations, separated by different intervals between ISI stimuli, were equal or unequal. There was a main effect in the groups with RD, as the good readers made significantly fewer errors than the RD on the auditory processing task. However, the groups with ADHD/IA made significantly more errors on the Serial Search task. Decomposition of the visual search task into component operations demonstrated that children in the ADHD/IA group had a slow processing rate that was not attributable to inattention. In sum, the children with ADHD were not globally poor at information processing nor inattentive, but they demonstrated diminished visual processing speed. On the contrary, in a previous study Weiler, Bernstein, Bellinger, and Waber (2000) failed to support the anticipated double dissociation. Weiler et al. (2000) analyzed processing speed and its relationship with oral- and written-language measures, visuospatial skills, and motor speed, in a clinical sample of children who were candidates for learning difficulties, classified as ADHD, RD, ADHD þ RD, and controls. All the groups scored significantly below the norm on processing speed, although the deficit was more pronounced in the ADHD group than in the RD group. The most notable conclusion is that this deficit was shared by ADHD and RD, producing additive effects of the pure disorders in the comorbid group. The idea that ADHD and RD are linked with two opposite patterns of impairments in different processes has also been supported by McGee, Brodeur, Symons, Andrade, and Fahie (2004). These authors evaluated phonological awareness, together with time perception and attention, in a sample composed of four groups of children (ADHD, RD, ADHD þ RD, and controls). The results of the phonological task of sound blending showed that the children with RD, alone or combined with ADHD, scored significantly lower than the ADHD and control groups on this measure. Significant differences were also found between the groups on a duration estimation task. Specifically, the ADHD and ADHD þ RD groups made longer and more variable time estimations than the RD and controls, which would support the hypothesis that a time perception deficit is a specific marker of ADHD but not RD. Children with both ADHD and RD had an additive combination of the ADHD-only and RD-only groups without interaction effect. However, other studies have reported that neuropsychological deficits of the group ADHD þ RD are different from the additive combination of the deficits associated with ADHD pure and RD pure, suggesting that the whole is more than the sum of each part, as predicts the cognitive subtype hypothesis. Rucklidge and Tannock (2002a) suggest that ADHD and RD are


191

separable cognitive profiles. Their results show that ADHD groups (regardless of RD status) demonstrated slower processing speed, were slower at naming objects, and had inhibition deficits, as well as greater variability in responses. On the other hand, the RD adolescents (regardless of ADHD status) had poorer verbal WM and were much slower at rapid naming (RAN) of letters and words. The group of children with ADHD þ RD had a more severe automatization problem than the children with only ADHD or RD. This group was also more impaired in mental arithmetic and WM than the ADHD group, and produced additional cognitive deficits, including slower rates of naming numbers and colors and overall slower reaction times and less accuracy in responses on the Stop task (Go Reaction Time and Percent Correct). Based on these findings, Rucklidge and Tannock (2002a) suggest that automatization would be a good candidate for future endophenotypic studies of comorbid ADHD and RD. In agreement with these results, Bental and Tirosh (2007) found unique deficits in rapid naming and a more severe impairment in WM in the comorbid ADHD þ RD group. Moreover, the comorbid group shared basic characteristic impairments in attention and executive functions with the pure ADHD group, and in reading domain functions with the pure RD group. Forward stepwise regression pointed to associations between executive functions and word-reading accuracy in children with ADHD, in contrast to associations between linguistic functions and word accuracy in non-ADHD. Therefore, the combination of cognitive deficits in the subgroup of children with both ADHD and RD, and the relationship between accuracy in word decoding and executive functions, suggest a distinct clinical profile for the comorbid condition. Together, the two previous studies provide support for the cognitive subtype hypothesis. However, others, as discussed below, would at least partly support the common etiology hypothesis, as they have found at least one deficit common to ADHD, RD, and ADHD þ RD, with the profile of the comorbid group being similar to those of the other two groups separately. Purvis and Tannock (2000), using only inhibition and phonological processing tasks, detected that the two RD groups (pure RD and RD þ ADHD) were significantly impaired on the phonological processing measures, while the ADHD, RD, and ADHD þ RD were all found to be associated with inhibitory control. However, the inhibitory deficit of the group with ADHD was more generalized, while in the RD group it appeared to be more linked to slowness in performing the task applied (Stop task), which required choosing an answer. With regard to the comorbid group, they exhibited the deficits of both pure groups in an additive fashion.

192

MIRANDA ET AL.

Shanahan et al. (2006) provided support for the common etiology hypothesis. In the study a complete battery of tests (coding subtest WISC-R, Trailmaking Test, RAN, STROOP, Stop Signal task, GDS, CPS, ETS) was applied to children and adolescents with RD, ADHD, and RD þ ADHD. In the first place, Shanahan et al. (2006) found that all the processing speed measures were correlated, and that the factorial analysis revealed two taskdependent factors (verbal and motor outputs). They also observed that the RD and ADHD groups had similar patterns of deficits, although with different magnitudes. Both groups had a processing speed deficit that extended to all the tasks in the battery; however, in contrast to other results (Weiler et al., 2000, 2002), the relationship with RD was stronger. The comorbid group did not differ significantly from the RD group on any factor. Finally, on closer examination of the nature of the ADHD þ RD combination, they found that the interaction between the RD and ADHD groups was significant, suggesting a nonadditive relationship. Complementary analyses of underadditivity and correlations indicated that the processing speed deficits of the comorbid group were at least partially shared by RD and ADHD. Consequently, the conclusion was drawn that processing speed is a shared cognitive risk factor that can help to explain the nature of the comorbidity. Willcutt, Pennington, Olson, Chhabildas, and Hulslander (2005) evaluated a large sample of subjects from the CLDRC twin study on measures of component reading and language skills, executive functions, and processing speed. They found a dissociation between the groups with and without RD on measures of reading and language skills, but none of the neuropsychological tasks was specifically associated with ADHD. The children with ADHD, those with RD, and children with both RD and ADHD, all exhibited difficulties in processing speed (slower and more variable response), response inhibition, and verbal WM, while set-shifting deficits were not associated with either disorder when FSIQ was controlled. Since the three groups had overlapping deficits, the authors concluded that RD and ADHD could have a common genetic base, proposing processing speed as the main common deficit. The verbal WM and response inhibition tasks were less compelling candidates for the common deficit because the relations between ADHD and WM and RD and inhibition were relatively weak and inconsistent. VWM is another process identified as a deficit common to RD, ADHD, and their association. Therefore, it has been proposed as a factor that could explain the high degree of overlap between the two disorders. Thus, Tiffin-Richards, Hasselhorn, Woerner, Rothenberger, and Banaschewski (2008) evaluated the main deficits of RD and ADHD, finding that language processing functions were impaired in the dyslexic group, while the ADHD group was impaired on


193

cognitive flexibility. However, all of the groups, ADHD, RD, and ADHD þ RD, were affected on VWM, leading to the conclusion that VWM could be a cognitive deficit common to ADHD and dyslexia. In other cases, general memory problems have been found, measured by means of N-back tasks, only in the two groups with RD. Van De Voorde, Roeyers, Verte´, and Wiersema (2010) used linguistic and executive functioning tasks, measured in relation to a baseline measure of functioning (i.e., a low memory or inhibition load condition compared to a high load). These authors found no evidence of response inhibition problems in ADHD or RD, taking into account a baseline measure. Furthermore, although the performance of the subjects with ADHD in the memory condition (1-back) was worse than that of the RD, the deficits disappeared when comparing them with the control condition (0-back). The most interesting result for our purposes was that the children with ADHD and those with RD showed a highly inaccurate and variable response style. Above all, it should be pointed out that the percentage of errors of the children in the comorbid group was nearly equal to the sum of the means of the two single-disorder groups, suggesting that different factors underlie the high error rate in both disorders. Finally, the only double dissociation study that found the profile of the comorbid group to be similar to that of only one of the disorders is the study by De Jong et al. (2009a). These authors used a lexical decision task to measure lexical access skills, along with measures of inhibition and visuospatial WM. The results showed that the disorders evaluated were associated with impairments in inhibition and lexical decision, deficits that were more severe in RD than in ADHD. However, the visuospatial WM was affected only in the group with ADHD alone and not in the comorbid group or the RD group. Nevertheless, all main effects for ADHD were lost after covarying for both age and IQ, so that in ADHD the impairments were dependent on IQ, suggesting that the overlap in lexical decision and inhibition has a different origin for ADHD and RD.

ASSESSMENT AND INTERVENTION IN ADHD AND RD The few studies in this review that deal with evaluation have examined the validity of different instruments to discriminate the comorbid ADHD þ RD condition (see Table 3). Thus, Rucklidge and Tannock (2002b) set out to determine the discriminant validity of the BADDS (Brown ADD Scales). The

Sample/ Age

N ¼ 25 7–11

N¼3 17–18

N ¼ 102 8–12

N ¼ 159 8–16

N ¼ 291

Study

Bental and Tirosh (2008)

Crabtree et al. (2010)

De Jong et al. (2009b)

Dewey et al. (2003)

Dewey et al. (2001)

A, R, A þ R, C

A, R, A þ R

A, R, A þ R, P

L, L þ A

A þ R, A þ R (C)

Groups

ARFRT, LNST, LSST

WRAML predictive accuracy

PRECAA usefulness

Atomoxetine

CBCL, WRAML, WRAT

WRAML WJ-R VMI, BOTMP

SSP LDT PRECAA

CBTT

DSM-IV

WCST Self monitoring GAS, IRW, SMRS

MPH

Measurements

Word reading accuracy, nonword reading accuracy and rapid naming Strategy/set shift Immediate recall accuracy Quiz accuracy ADHD symptomatology Visuospatial working memory Inhibition Lexical decision Memory, cognition, coordination and learning Language Academic General memory Language skills Visual-motor skills, motor skills General memory

Evaluation and Intervention Studies.

Evaluation/ Intervention

Table 3.

The WRAML did not substantially improve predictive accuracy over more standard diagnostic tools

A þ RWA, R No improvements A, A þ RWR R, A þ RWA No improvements

A þ RWA, R No improvements A, A þ RWR

A þ RWA, R y P

Significant improvements A y A þ RWR y P

A þ RWA þ R (P) Significant improvements

Significant improvements

Results

194 MIRANDA ET AL.

A (C), R (C) A þ R (E)

A, R, A þ R, C

N ¼ 43 6–12

N ¼ 581 X ¼ 5.7

N ¼ 98 13–16

Keulers et al. (2007)

Rabiner and Malone (2004)

Rucklidge and Tannock (2002b)

A, R, A þ R, C

AþR

N¼6 9–10

Validity of BADDS

Tutoring

MPH

RTR, MPH

Combined

CRS-R, OCHSS

BADDS

DRS DSM-III-R

DMT, EMT BV

NDT CBA

OSST

Kurzweil 3000TM

Kastner et al. (2000)

GORT-3,

Computer

A, A þ R

CCR, CPT, RASS

MPH

A, A þ L (R, M)

N ¼ 95 6–12 N ¼ 20 19–57

Grizenko et al. (2006) Hecker et al. (2002)

BADDS

Reading scores Visual sustained attention Reading achievement Attention

Instructional level

Attention, stress, speed and fatigue Comprehension

Treatment response

Moderated correlations between Brown, CRS-R, and OCHSS

RWA, A þ R Attention problems predicted diminished reading achievement A y A þ RWR y C

All participants mastered more quickly during the optimal condition Most participants made fewer reading errors, had higher levels of correct words per minute, and read passages more quickly A þ RWA, R A, A þ RWR

No improvements

Significant improvements

A, A þ RWA þ L, A þ M


N ¼ 56 10–16

N ¼ 47 7–12

Sumner et al. (2009)

Tannock et al. (2000)

A, R, A þ R

A, A þ R

Groups

MPH

Atomoxetine

Evaluation/ Intervention

RAN

WMTB-C

K-TEA

AD/HD-RS

Measurements

ADHD symptomatology Reading decoding standard All other measures Central Executive Phonological loop Visuospatial sketchpad Color naming speed Letters naming speed

Significant improvements AWA þ R A þ RWA No improvements Significant improvements No improvements

A þ RWA

A¼AþR

Results

A: ADHD only; C: controls; E: experimental; L: learning disabilities; M: mathematical disabilities; P: placebo; R: RD only. Intervention. MPH: methylphenidate; PRECAA: Parent Ratings of Everyday Cognitive and Academic Abilities; RTR: Reading to Read. Measurements. AD/HD-RS: AD/HD Rating Scale; ARFRT: Animal Retrieval and Food Retrieval Tests; BADDS: Brown ADD Scale; BOTMP: Bruininks–Oseretsky Test of Motor Proficiency; BV: Bourdon–Vos Test; CBA: Curriculum Based Assessment; CBCL: Child Behavior Checklist; CBTT: Corsi Block Tapping Task; CCR: Consensus Clinical Response; CPT: Continuous Performance Test; CRS-R: Conners’ Rating Scales – Revised; DMT: Drie Minuten Test; DRS: Diagnostic Reading Scale; DSM-IV, DSM-III-R; Diagnostic and Statistical Manual of Mental Disorders; EMT: Een Minuut Test; GAS: Globe Anthology Series; GORT-3: Gray Oral Reading Test – 3; IRW: Immediate Recall Worksheet; K-TEA: Kaufman Test of Educational Achievement; LDT: Lexical Decision Task; LNST: Listening Number Span Test; LSST: Listening Sentence Span Test; NDT: Nelson–Denny Test; OCHSS: Ontario Child Health Study Scales; OSST: One Sample Sign Test; RAN: Rapid Automatized Naming Test; RASS: Restricted Academic Situation Scale; SMRS: Self-Monitoring Response Sheets; SSP: Stop Signal Paradigm; VMI: Visual–Motor Integration Developmental Test; WJ-R: Woodcock–Johnson Psychoeducational Battery – Revised; WMTB-C: Working Memory Test Battery for Children; WRAML: Wide Range Assessment of Memory and Learning; WRAT: Wide Range Achievement Test; WRMT: Woodcock Reading Mastery Test; WCST: Wisconsin Card Sorting Test.

Sample/ Age

Study


196 MIRANDA ET AL.


197

two groups with ADHD þ RD and ADHD who participated in the study obtained significantly higher scores than the RD and control groups on all the scales. The recommended cutoffs resulted in high rates of false negatives but few false positives. Therefore, these authors point out that the scale can be used for screening out ADHD, but not for its diagnosis, due to its low sensitivity. Dewey, Crawford, and Kaplan (2003) analyzed the discriminant and predictive power of another test designed for parents, the PRECAA (Parent Ratings of Everyday Cognitive and Academic Abilities). The authors found that this instrument correctly classified 81% of the children with ADHD, and was sensitive to group differences between children with RD and children with ADHD þ RD. Finally, Dewey Crawford, Kaplan, and Fisher (2001) examined the predictive power of a memory test (WRAML) in the diagnosis of ADHD, RD, or both. The WRAML only classified 58.5% of the participants correctly, while academic, intellectual, and behavioral measures together correctly classified 73.1%. These results suggest that the use of a measure of memory functioning, such as the WRAML, does not significantly improve the predictive accuracy of a diagnosis of ADHD, RD, or both, beyond what is provided by the more standard academic, intellectual, and behavioral diagnostic measures. The intervention studies deal with the two most common treatment modalities for ADHD, that is, pharmacological and psychopedagogical interventions. Several studies have shown the benefits of methylphenidate (MPH) in reading performance and other cognitive impairments frequently present in ADHD. The contribution by Tannock, Martinussen, and Frijters (2000) demonstrates that naming speed deficits are improved but not normalized by stimulant medication. These authors randomly assigned children with ADHD and ADHD þ RD to four intervention conditions lasting one week each: placebo, low dosage (10 mg), medium dosage (15 mg), and high dosage (20 mg). They found a lineal effect of dosage on colornaming speed, with the naming time declining from 55.8 s in the placebo condition to 51.1 seconds in the group with the highest dosage. In sum, MPH selectively improved color-naming speed but had no effect on the speed of naming letters or digits. Grizenko, Bhat, Schwartz, Ter-Stepanian, and Joober (2006) carried out a prospective, double-blind, placebo controlled, randomized, two-week crossover trial to study the response to MPH in children with ADHD and different types of LDs in reading and mathematics. These authors analyzed the degree of weekly improvement in each group of children, using the Consensus Clinical Response (CCR) scale, which evaluates basic ADHD symptomatology through observations of parents, teachers, and therapists.

198

MIRANDA ET AL.

One interesting initial finding was that the children with ADHD þ LDs responded significantly less to the methylphenidate than the children with ADHD without LDs (55% vs. 75%). But possibly the most surprising result was that the children with ADHD and LDs responded differentially to the MPH. Thus, while the group of children with RD presented a more positive response pattern, not unlike that of the group with ADHD alone, the group of children with MD responded very little to the medication. Keulers et al. (2007) analyzed the effects of MPH on reading performance in a unique 0.5 mg twice-daily dosing schedule for four weeks. Reading performance was compared with General Linear Model repeated measures among three groups: an experimental group of children with both ADHD and dyslexia; a control group of children with ADHD; and another control group of children with dyslexia. At the pretest evaluation, the group with ADHD obtained better scores than the two groups with dyslexia. The posttest evaluation showed that the medication improved the basic symptomatology of the groups with ADHD and greatly increased the number of correctly read words of the children in the ADHD and comorbid dyslexia group, compared with the children in the two control groups. Finally, Bental and Tirosh (2008) analyzed the immediate effect of MPH on decoding in the comorbid condition of ADHD þ RD in primary school boys with normal intelligence and linguistic skills. The boys were administered a single dose of the psychostimulant (0.3–0.4 mg/kg), with weekly intervals between testing sessions. Paired comparisons and first trial group comparisons of performance in the placebo condition and under methylphenidate were conducted. The basic finding was that the MPH selectively improved strategy/set shifting and facilitated improvement in both rapid-naming and word/nonword accuracy. Moreover, the authors found significant differences between the MPH and placebo conditions on strategy/set shifting but not on spelling. Two studies that fall into the general category of pharmacological interventions have studied the effects of atomoxetine on cognitive processing and reading performance. De Jong et al. (2009b) analyzed the possible benefits of administering atomoxetine (0.6 mg/kg for 7 days and 1.2 mg/kg for the following 21 days) for the EF (visuospatial memory, inhibition) and on lexical decision tasks in four groups of children: ADHD, ADHD þ RD, RD, and control. The analyses showed that the children in the two groups with ADHD significantly improved the basic symptomatology of the disorder with the medication. Furthermore, the children in the ADHD þ RD group improved inhibition with the atomoxetine treatment compared to placebo. No differential effects of atomoxetine were found for


199

lexical decision in comparison to placebo. In addition, no effects of atomoxetine were found in the ADHD and RD groups. The authors suggest the possibility that ADHD and ADHD þ RD are two different disorders at the neuropsychological and neurochemical levels. One interesting question addressed by Sumner et al. (2009), although tangentially, consists of finding out to what extent the effects of atomoxetine on certain component skills related to reading may correlate with changes in overall reading performance. For this purpose, they analyzed the effects of an intervention with atomoxetine, at doses ranging from 1.0 to 1.4 mg/kg once daily for 16 weeks, in adolescents distributed in two groups: ADHD and ADHD þ dyslexia. The atomoxetine significantly improved the basic symptomatology and reading comprehension standard scores in the two groups of adolescents with ADHD. Spelling subtest improvement was significant for the ADHD group, whereas the ADHD þ dyslexia group showed significant reading decoding improvements. The analysis of the change in neurocognitive processes helped to interpret the results. Thus, the ADHD group showed more marked improvement in the component scores related to CE functioning, and the ADHD and comorbid dyslexia group showed more marked improvement in component scores related to the phonological loop. These data suggest that the brain systems related to the therapeutic benefit of atomoxetine in reducing ADHD symptoms may be different in individuals with ADHD with dyslexia and ADHD without dyslexia. Other studies in this review have dealt with rating the efficacy of different types of psychoeducational interventions. The study conducted by Crabtree, Alber-Morgan, and Konrad (2010) was designed to examine the effects of self-monitoring and active responding on the reading comprehension of three high-school seniors with LDs and ADD (attention deficit disorder). According to the special education teacher, all three participants demonstrated deficits in reading comprehension. The intervention consisted of stories, self-monitoring response sheets, immediate recall worksheets, and quizzes. After the intervention was implemented, there was an immediate and substantial increase in the number of correct responses. During maintenance, all three participants continued to increase their correct responses. All three participants also demonstrated substantial and consistent increases in quiz accuracy when the intervention was introduced. Only the study by Hecker, Burns, Elkind, Elkind, and Katz (2002) was designed to determine the potential benefits of the use of modern technologies in the reading performance of postsecondary students with a diagnosis of ADHD. The majority of the participants were taking psychostimulant medication, and 25% suffered from RD. The dependent

200

MIRANDA ET AL.

variables used were reading speed and comprehension, along with distractibility and fatigue while reading, rated by means of self-assessment, independent reading logs, and results on reading tests, before and after applying a computer program. The software used provides a synchronized visual and auditory presentation of text, and incorporates study skill tools for highlighting and note taking. The comparison of the two evaluation times showed that performance clearly improved after the intervention. Specifically, the subjects improved their attentional capacity (with fewer distractions), increased time spent reading as well as reading speed, and reduced stress and fatigue. However, no differences were found on comprehension between the pretreatment and posttreatment evaluations, although the participants with very low comprehension improved their scores somewhat. The overall objective of Rabiner, Malone, and Conduct Problems Prevention Research Group (2004) was to examine whether attention problems moderate the impact of tutoring on the development of children’s early reading skills. As a group, participants in their study had greater attention problems than the general population. The program was designed for low-readiness children from disadvantaged backgrounds, and emphasized a phonics-based, mastery-oriented approach to the development of initial reading skills. Tutoring began with an individual assessment of the child’s skill development, and progressed according to each child’s ability to master the skills taught. Throughout first grade, tutors worked with the children individually three times a week during 30 min sessions. Two of these sessions took place during school hours, and one occurred during the parent-child group activities. The authors found that the children with RD alone improved their reading performance significantly, while the children with RD and associated attention problems did not benefit from the intervention. Finally, mention will be made of an intervention that combined psychostimulant medication (MPH) with the Reading to Read (RTR) program. With this combined treatment, Kastner, Tingstrom, and Edwards (2000) investigated the relationship among the scheduling time of a repeated reading intervention, MPH, and measures of oral-reading fluency, in six ADHD boys who had no comorbid diagnoses except RD. There were two conditions: RTR intervention conducted 45 min–1 h after ingestion of MPH (optimal), and RTR intervention conducted 3–4 h after ingestion of MPH (nonoptimal). The RTR intervention ended for each subject either when he reached grade level in the sequence or when the school year ended. Each student met individually with the interventionist 3–5 days per week. The

201


RTR materials consisted of 57 100-word passages from the beginning, middle, and ending thirds of each book in the series, with two alternative forms (A and B). All participants mastered passages more quickly during the optimal condition than during the nonoptimal condition. In addition, more participants made fewer reading errors, had higher levels of correct words per minute, and read passages more quickly during RTR administered during the optimal MPH period.

DISCUSSION One objective of this study was to identify the exact nature of reading disabilities in ADHD. The purpose was to find out whether children with the comorbid type showed the typical impairments of each of the two disorders separately, ADHD and RD, or whether they experienced a combination of the characteristics of each, with the same or greater severity. The majority of the findings show that, as occurs in the pure RD group, the ADHD þ RD group presents phonological awareness impairments (McGee et al., 2004; Willcutt et al., 2001; Willcutt, Pennington, Olson et al., 2005), makes more mistakes, and is slower than controls and pure ADHD on performing all types of phonological processing tasks, such as word and pseudoword reading, phonological suppression, segmentation, phonological synthesis, or spelling (McGee et al., 2004; Purvis & Tannock, 2000; TiffinRichards et al., 2008; Van de Voorde et al., 2010). Similar deficits seem to occur in adulthood (Laasonen, Lehtinen, Leppa¨ma¨ki, Tani, & Hokkanen, 2010). Naming speed problems in the RD and ADHD þ RD groups have been detected in various studies, arguing that this process is a specific marker for RD (Raberger & Wimmer, 2003; Seidman et al., 2001; Van de Voorde et al., 2010), even in nonalphabetic systems (Chan et al., 2008). Even so, the conclusion cannot be drawn that there is a complete dissociation between ADHD and RD on naming speed measures, as differentiated results have been found depending on the type of stimuli presented. Thus, the ADHD group obtains worse time scores than the control group on color and object naming, but no differences are found on letter and number naming (Rucklidge & Tannock, 2002a). One possible explanation is that color and object naming are more perceptually/semantically based, while letter and number naming require more phonological resources. However, the most significant finding is that the ADHD þ RD group is slower on all the rapidnaming tasks and has slower reaction times.

202

MIRANDA ET AL.

It can be said, then, that naming speed is a basic deficit in children and adolescents with RD with all types of stimuli (verbal and nonverbal), and in ADHD with nonverbal stimuli. The comorbid group shows a combination of the impairments of the pure groups, in agreement with the double dissociation hypothesis (Raberger & Wimmer, 2003; Seidman et al., 2001; Van de Voorde et al., 2010), although there are also data indicating that the ADHD þ RD group is more impaired, as proposed in the specific subtype hypothesis (Rucklidge & Tannock, 2002a). Related to rapid naming, in spite of being a more general construct, processing speed has shown its value in explaining the relationship between ADHD and RD in many studies, and it could represent an essential deficit shared by both disorders (an endophenotype) (De Jong et al., 2009a; Ghelani et al., 2004; Laasonen et al., 2009; Willcutt, Pennington, Olson et al., 2005; Shanahan et al., 2006; Weiler et al., 2000, 2002). More exhaustively, a response style characterized by great intraindividual variability in task performance could also be a cognitive impairment shared by subjects with ADHD, RD, and ADHD þ RD, as proposed in the common etiology hypothesis (Van de Voorde et al., 2009; Willcutt, Pennington, Olson et al., 2005). It still remains to be seen whether time perception is affected in individuals with ADHD and ADHD þ RD. Some studies (Toplack et al., 2003) have found that time estimation deficits are associated with ADHD, and especially with ADHD þ RD, and may be influencing the time organization of the behavior of people with these disorders. However, other studies have not found the same tendency (Tiffin-Richards et al., 2004). Greater agreement exists about the idea that groups with RD and ADHD þ RD are characterized by having less efficient temporal auditory functioning, especially with verbal stimuli. It also seems that these groups show a backward masking effect with nonverbal stimuli of an asymmetric temporal onset (Breier et al., 2001, 2002, 2003; Weiler et al., 2002). The possible perceptual and motor impairments in RD and ADHD, and the effects of other concurrent disorders like MD, CD, or DCD (developmental coordination disorder), have been studied very little. For now, it is known that the RD þ ADHD þ MD group is different from the ADHD þ MD and ADHD þ RD groups, as it presents significantly greater impairments in both expressive and receptive language (Capano et al., 2008). Similarly, the RD þ ADHD þ MD group shows a poorer performance on different neuropsychological tasks (Seidman et al., 2001), and,


203

more importantly, the perceptual-visual problems seem to be more related to the presence of ADHD and RD than to ADHD itself (Crawford & Dewey, 2008). The executive functions addressed in the studies reviewed were inhibition and WM. Although some studies find inhibition impairments only in the ADHD group and the comorbid group (Willcutt et al., 2001), others have also found RD to be related to behavioral inhibition (De Jong et al., 2009a; Purvis & Tannock, 2000; Willcutt, Pennington, Olson et al., 2005), thus questioning the role of inhibitory control as the only cognitive marker of ADHD, and suggesting a true comorbidity of the children with ADHD þ RD. In contrast, surprisingly, more recent studies have not found response inhibition problems in any group, that is, ADHD, RD, or ADHD þ RD (Bental &Tirosh, 2007; Van de Voorde et al., 2010). The conclusions that can be drawn about WM depend especially on the distinction between the different modalities. The few studies that have analyzed visuospatial WM have obtained completely disparate results. However, all the studies that have evaluated VWM have found impairments in both the ADHD þ RD group and the RD group (Kibby & Cohen, 2008; Roodenrys et al., 2001; Rucklidge & Tannock, 2002a; Van de Voorde et al., 2010; Willcutt et al., 2001). Moreover, some findings show that the impairments in the WM affect the three groups, ADHD, RD, and ADHD þ RD (Martinussen & Tannock, 2006; Tiffin-Richards et al., 2008; Willcutt, Pennington, Olson et al., 2005). The review carried out confirms that the cooccurrence of ADHD and RD increases the risk of negative results in the daily functioning of the affected children, adolescents, and young adults (Bonafina et al., 2000; Capano et al., 2008; Willcutt, Betjemann, Pennington, et al., 2007). Subjects in the comorbid group usually present more severe and more stable impairments than subjects with only RD and only ADHD on academic and social measures. They also show higher scores on anxiety, depression, and CD, which causes them to receive more psychotherapy, psychostimulant medication, and special education. Regarding ADHD and RD treatments, it can be said that the studies, to a greater or lesser extent, found positive effects of the interventions, pharmacological and nonpharmacological, on subjects’ reading performance. Likewise, as would be expected, the pharmacological interventions report significant improvements in the basic symptomatology of ADHD subjects. However, it is necessary to be cautious in drawing conclusions, due to the methodological limitations of the reduced number of studies analyzed,

204

MIRANDA ET AL.

such as variability in the diagnostic criteria utilized, different reading tests applied, limited duration of the intervention, a low number of participants, or different doses prescribed in the pharmacological treatments and whether or not they were maintained while the evaluations were performed.

FUTURE DIRECTIONS Although it provides interesting information, the review carried out does not make it possible to extract definitive results on the exact nature of ADHD and RD comorbidity or, even less, reach conclusions about its causes. The studies suggest that the ADHD þ RD group presents a profile characterized basically by a combination of symptoms of the two pure groups. These results could be consistent with the studies indicating that one or more genes increase the risk of ADHD and RD, with a stronger influence in the comorbid group (Pennington, Willcutt, & Rhee, 2005; Willcutt, Pennington, Olson, & DeFries, 2007). In sum, common genetic influences contribute to ADHD and RD comorbidity, but the mechanisms of these common genes are unknown. Both ADHD and RD can also be partly due to a cognitive impairment in the ability to process information rapidly and efficiently (an endophenotypic candidate for ADHD þ RD comorbidity). However, other studies seem to support both the cognitive subtype hypothesis and the common etiology hypothesis. It is possible, as stated by De Jong et al. (2006), that some subtypes of ADHD and RD have the same genetic origins, while other subtypes and their combinations are separate disorders. More research is necessary in order to discover the cause or causes of the high comorbidity between ADHD and RD, and to be able to decide among the different causal models proposed. A recent study by Willcutt et al. (2007) is an example of the line of research that should be developed. Willcutt et al. (2007) set out to examine the etiology of the association between ADHD and RD in preschool children who had not yet received reading instruction and, therefore, had not adverse experiences that could unleash the comorbidity (phenocopy hypothesis). Phenotypic analyses revealed small but significant correlations between high scores on inattention (DSM-IV) and low performance on the prereading measures (phonological awareness, rapid naming, verbal memory, vocabulary, grammar/morphology, and print knowledge), which did not occur in any case with hyperactivity/impulsivity. The results


205

confirm, therefore, that the RD and ADHD association is already present in the kindergarten period. The study by Willcutt et al. (2007) also included multivariate twin analyses, which indicated that practically all the phenotypic correlations between inattention and prereading are attributable to common genetic influences, which seem to be especially strong in phonological awareness, rapid naming, and verbal memory. For the authors, these results suggest that the association between prereading difficulties and ADHD in preschool children is at least partly due to common genetic influences. The difficulties in learning to read in students with ADHD are probably going to be more severe and more evident during adolescence and adulthood, when the WM and other executive functions are necessary for understanding texts with some complexity and progressing toward independent learning. Thus, longitudinal studies are needed to analyze the development of this association throughout the life cycle.

IMPLICATIONS FOR ASSESSMENT AND TREATMENT Given the high rate of ADHD and RD comorbidity, knowledge about the causes underlying their overlapping is essential, especially because the intervention procedures for these two disorders are very different. Traditionally, LDs are considered linguistic impairments, and ADHD is considered a behavioral impairment. Although it is important to continue to investigate the characteristic deficits affecting the ADHD and RD association and understand them better, the knowledge already available can help to provide better interventions for the problem. Individuals with ADHD þ RD have more severe cognitive impairments and worse results in academic, social, and emotional realms than the pure ADHD or RD, so that evaluations and interventions should be specifically designed for this comorbid group. Specifically, the evaluation of ADHD and RD should be psychoeducational and complex, contemplating all those aspects that have been shown to be affected to some degree. First, it is necessary for the evaluation of suspected ADHD subjects to include the analysis of reading skills and the different processes involved in them. Likewise, the evaluation of RD should include psychological tests that can detect ADHD. Second, in addition to screening and diagnosing both disorders, would be necessary that the evaluation includes a detailed history of cognitive, motor, perceptual, and

206

MIRANDA ET AL.

language development. Third, a study should also be conducted of academic performance, with special attention to mathematical skills. Finally, an evaluation of the behavioral and emotional problems and disorders that usually cooccur with ADHD þ RD should be included. These aspects should also be considered when designing individualized intervention programs directed toward all the deficiencies and needs detected. Furthermore, given the chronic nature of the comorbid syndrome, the entire life cycle should be contemplated, adapted to the different stages of development. The intervention should be, therefore, a psychological and educational one that addresses both the main deficiencies of ADHD and RD and the problems most frequently associated with their comorbidity. For some subjects, pharmacological treatment should also be included. Although the medication does not teach the student knowledge that he or she has not acquired, it can help to foster a favorable disposition toward learning. In the school context, it is necessary for special education teachers, but also for the entire school team, to receive training in how to interact with children with these problems. They should know about the characteristics and causes of the two disorders, their long-term effects, and the interventions that have proven most beneficial. Children with ADHD and RD will have to receive accommodations in school that will help them to control their deficiencies: modify the instruction method to make it more dynamic, divide the tasks into parts to avoid an excessive load on the WM, prolong the time of the activities and tests to adapt them to the slower processing style, modify the student’s environment and situation to facilitate the inhibition of stimuli not relevant to the tasks, take advantage of the possibilities of the new technologies to avoid the fatigue produced by the necessary reading practice. Adolescents and adults should also receive help to face and control their growing academic and workrelated responsibilities. For example, tutors can help them program and control the development and achievement of their daily objectives, break down long-term projects into components, train them in the use of facilitating materials, and teach them strategies of self-direction (self-observation and self-evaluation to monitor the performance of activities and studies). Especially interesting is the specific teaching of strategies to promote text comprehension, such as paraphrasing, summarizing, posing questions, prediction, or finding the main idea. Finally, it is important to keep in mind the feelings of helplessness and frustration that, unfortunately, are usually produced by ADHD and RD in students and their families. In these cases, it is imperative to complement educational and pharmacological interventions (if appropriate) with supportive counseling.

207


REFERENCES American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., rev.). DSM-IV-TR. Washington DC: Author. Barkley, R. A. (1997). Behavioral inhibition, sustained attention, and executive functions: Constructing a unifying theory of ADHD. Psychological Bulletin, 121, 65–94. Bental, B., & Tirosh, E. (2007). The relationship between attention, executive functions and reading domain abilities in attention deficit hyperactivity disorder and reading disorder: A comparative study. Journal of Child Psychology and Psychiatry, 48, 455–463. Bental, B., & Tirosh, E. (2008). The effects of methylphenidate on word decoding accuracy in boys with attention-deficit/hyperactivity disorder. Journal of Clinical Psychopharmacology, 28, 89–92. Bonafina, M. A., Newcorn, J. H., McKay, K. E., Koda, V. H., & Halperin, J. M. (2000). ADHD and reading disabilities. A cluster analytic approach for distinguishing subgroups. Journal of Learning Disabilities, 33, 297–307. Breier, J. I., Fletcher, J. M., Foorman, B. R., Klaas, P., & Gray, L. C. (2003). Auditory temporal processing in children with specific reading disability with and without attention deficit/hyperactivity disorder. Journal of Speech, Language, and Hearing Research, 46, 31–42. Breier, J. I., Gray, L. C., Fletcher, J. M., Diehl, R. L., Klaas, P., Foorman, B. R., & Molis, M. R. (2001). Perception of voice and tone onset time continua in children with dyslexia with and without attention deficit/hyperactivity disorder. Journal of Experimental Child Psychology, 80, 245–270. Breier, J. I., Gray, L. C., Fletcher, J. M., Foorman, B., & Klaas, P. (2002). Perception of speech and nonspeech stimuli by children with and without reading disability and attention deficit hyperactivity disorder. Journal of Experimental Child Psychology, 82, 226–250. Capano, L., Minden, D., Chen, S. X., Schachar, R. J., & Ickowicz, A. (2008). Mathematical learning disorder in school-age children with attention-deficit hyperactivity disorder. La Revue Canadienne de Psychiatrie, 53, 392–399. Chan, W. S. R., Hung, S. F., Liu, S. N., & Lee, C. K. K. (2008). Cognitive profiling in Chinese developmental dyslexia with attention-deficit/hyperactivity disorders. Reading and Writing: An Interdisciplinary Journal, 21, 661–674. Cohen, M. J. (1997). Examiner’s manual: Children’s memory scale. San Antonio, TX: Harcourt Brace & Company. Crabtree, T., Alber-Morgan, S. R., & Konrad, M. (2010). The effects of self-monitoring of story elements on the reading comprehension of high school seniors with learning disabilities. Education and Treatment of Children, 33, 187–203. Crawford, S. G., & Dewey, D. (2008). Co-occurring disorders: A possible key to visual perceptual deficits in children with developmental coordination disorder? Human Movement Science, 27, 154–169. De Jong, C. G. W., Oosterlaan, J., & Sergeant, J. A. (2006). The role of double dissociation studies in the search for candidate endophenotypes for the comorbidity of attention deficit hyperactivity disorder and reading disability. International Journal of Disability, Development and Education, 53, 177–193. De Jong, C. G. W., Van De Voorde, S., Roeyers, H., Raymaekers, R., Oosterlaan, J., & Sergeant, J. A. (2009a). How distinctive are ADHD and RD? Results of a double dissociation study. Journal of Abnormal Child Psychology, 37, 1007–1017.

208

MIRANDA ET AL.

De Jong, C. G. W., Van De Voorde, S., Roeyers, H., Raymaekers, R., Allen, A. J., et al. (2009b). Differential effects of atomoxetine on executive functioning and lexical decision in attention-deficit/hyperactivity disorder and reading disorder. Journal of Child and Adolescent Psychopharmacology, 19, 699–707. Del’Homme, M., Kim, T. S., Loo, S. K., Yang, M. H., & Smalley, S. L. (2007). Familial association and frequency of learning disabilities in ADHD sibling pair families. Journal of Abnormal Psychology, 35, 55–62. Dewey, D., Crawford, S. G., & Kaplan, B. J. (2003). Clinical importance of parent ratings of everyday cognitive abilities in children with learning and attention problems. Journal of Learning Disabilities, 36, 87–95. Dewey, D., Kaplan, B., Crawford, S. G., & Fisher, G. C. (2001). Predictive accuracy of the wide range assessment of memory and learning in children with attention deficit hyperactivity disorder and reading difficulties. Developmental Neuropsychology, 19, 173–189. Ghelani, K., Sidhu, R., Jain, U., & Tannock, R. (2004). Reading comprehension and reading related abilities in adolescents with reading disabilities and attention-deficit/hyperactivity disorder. Dyslexia, 10, 364–384. Grizenko, N., Bhat, M., Schwartz, G., Ter-Stepanian, M., & Joober, R. (2006). Efficacy of methylphenidate in children with attention-deficit/hyperactivity disorder and learning disabilities: A randomized crossover trial. Journal of Psychiatry Neuroscience, 31, 46–51. Hecker, L., Burns, L., Elkind, J., Elkind, K., & Katz, L. (2002). Benefits of assistive reading software for students with attention disorders. Annals of Dyslexia, 52, 243–272. Kastner, J. W., Tingstrom, D. H., & Edwards, R. P. (2000). The utility of reading to read with boys with ADHD-CT administered at two different intervals post methylphenidate ingestion. Psychology in the Schools, 37, 367–377. Keulers, E. H. H., Hendriksen, J. G. M., Feron, F. J. M., Wassenberg, R., Wuisman-Frerker, M. G. F., Jolles, J., & Vles, J. S. H. (2007). Methylphenidate improves reading performance in children with attention deficit hyperactivity disorder and comorbid dyslexia: An unblinded clinical trial. European Journal of Pediatric Neurology, 11, 21–28. Kibby, M. Y., & Cohen, M. J. (2008). Memory functioning in children with reading disabilities and/or attention deficit/hyperactivity disorder: A clinical investigation of their working memory and long-term memory functioning. Child Neuropsychology, 14, 525–546. Kooistra, L., Crawford, S., Dewey, D., Cantell, M., & Kaplan, B. J. (2005). Motor correlates of ADHD: Contribution of reading disability and oppositional defiant disorder. Journal of Learning Disabilities, 38, 195–206. Laasonen, M., Lehtinen, N., Leppa¨ma¨ki, S., Tani, P., & Hokkanen, L. (2010). Project DyAdd: Phonological processing, reading, spelling, and arithmetic in adults with dyslexia of ADHD. Journal of Learning Disabilities, 43, 3–14. Laasonen, M., Leppa¨ma¨ki, S., Tani, P., & Hokkanen, L. (2009). Adult dyslexia and attention deficit disorder in Finland project DyAdd: WAIS-III cognitive profiles. Journal of Learning Disabilities, 42, 511–527. Martinussen, R., & Tannock, R. (2006). Working memory impairments in children with attention-deficit hyperactivity disorder with and without comorbid language learning disorders. Journal of Clinical and Experimental Neuropsychology, 28, 1073–1094. McDougall, S., Hulme, C., Ellis, A., & Monk, A. (1994). Learning to read: The role of shortterm memory and phonological skills. Journal of Experimental Child Psychology, 58, 112–133.


209

McGee, R., Brodeur, D., Symons, D., Andrade, B., & Fahie, C. (2004). Time perception: Does it distinguish ADHD and RD children in a clinical sample? Journal of Abnormal Child Psychology, 32, 481–490. Pennington, B. F., Groisser, D., & Welsh, M. C. (1993). Contrasting cognitive deficits in attention deficit hyperactivity disorder versus reading disability. Developmental Psychology, 29, 511–523. Pennington, B. F., Willcutt, E. G., & Rhee, S. H. (2005). Analyzing comorbidity. In: R. V. Kail (Ed.), Advances in Child Development and Behavior (33, pp. 263–305). Oxford: Elsevier. Pisecco, S., Baker, D. B., Silva, P. A., & Brooke, M. (2001). Boys with reading disabilities and/or ADHD: Distinctions in early childhood. Journal of Learning Disabilities, 34, 98–106. Purvis, K. L., & Tannock, R. (1997). Language abilities in children with ADHD, reading disabilities and normal controls. Journal of Abnormal Child Psychology, 25, 133–144. Purvis, K. L., & Tannock, R. (2000). Phonological processing, not inhibitory control, differentiates ADHD and reading disability. Journal of the American Academy of Child & Adolescent Psychiatry, 39, 485–494. Raberger, T., & Wimmer, H. (2003). On the automaticity/cerebellar deficit hypothesis of dyslexia: Balancing and continuous rapid naming in dyslexic and ADHD children. Neuropsychologia, 41, 1493–1497. Rabiner, D. L., & Malone, P. S. Conduct Problems Prevention Research Group. (2004). The impact of tutoring on early reading achievement for children with and without attention problems. Journal of Abnormal Child Psychology, 32, 273–284. Rapala, M. M., & Brady, S. (1990). Reading ability and short term memory: The role of phonological processing. Reading and Writing, 2, 1–25. Rapport, M. D., Chung, K. M., Shore, G., & Isaacs, P. (2001). A conceptual model of child psychopathology: Implications for understanding attention deficit hyperactivity disorder and treatment efficacy. Journal of Clinical and Child Psychology, 30, 48–58. Roodenrys, S., Koloski, N., & Grainger, J. (2001). Working memory function in attention deficit hyperactivity disordered and reading disabled children. British Journal of Developmental Psychology, 19, 325–337. Rucklidge, J. J., & Tannock, R. (2002a). Neuropsychological profiles of adolescents with ADHD: Effects of reading difficulties and gender. Journal of Child Psychology and Psychiatry, 43, 988–1003. Rucklidge, J. J., & Tannock, R. (2002b). Validity of the brown ADD scales: An investigation in a predominantly inattentive ADHD adolescent sample with and without reading disabilities. Journal of Attention Disorders, 5, 155–164. Seidman, L. J., Biederman, J., Monuteaux, M. C., Doyle, A. E., & Faraone, S. V. (2001). Learning disabilities and executive dysfunction in boys with attention-deficit/hyperactivity disorder. Neuropsychology, 15, 544–556. Shanahan, M. A., Pennington, B. F., Yerys, B. E., Scott, A., Boada, R., Willcutt, E. G., Olson, R. K., & DeFries, J. C. (2006). Processing speed deficits in attention deficit/hyperactivity disorder and reading disability. Journal of Abnormal Child Psychology, 34, 585–602. Shaywitz, B. A., Fletcher, J. M., & Shaywitz, S. E. (1995). Defining and classifying learning disabilities and attention-deficit/hyperactivity disorder. Journal of Child Neurology, 10, S50–S57. Sumner, C. R., Gathercole, S., Greenbaum, M., Rubin, R., Williams, D., Hollandbeck, M., & Wietecha, L. (2009). Atomoxetine for the treatment of attention-deficit/hyperactivity

210

MIRANDA ET AL.

disorder (ADHD) in children with ADHD and dyslexia. Child and Adolescent Psychiatry and Mental Health, 3, 40. Tannock, R., Martinussen, R., & Frijters, J. (2000). Naming speed performance and stimulant effects indicate effortful, semantic processing deficits in attention-deficit/hyperactivity disorder. Journal of Abnormal Child Psychology, 28, 237–252. Tiffin-Richards, M. C., Hasselhorn, M., Richards, M. L., Banaschewsky, T., & Rothenberger, A. (2004). Time reproduction in finger tapping tasks by children with attention-deficit hyperactivity disorder and/or dyslexia. Dyslexia, 10, 299–315. Tiffin-Richards, M. C., Hasselhorn, M., Woerner, W., Rothenberger, A., & Banaschewski, T. (2008). Phonological short-term memory and central executive processing in attentiondeficit/hyperactivity disorder with/without dyslexia – Evidence of cognitive overlap. Journal of Neural Transmission, 11, 227–234. Toplack, M. E., Rucklidge, J. J., Hetherington, R., John, S. C. F., & Tannock, R. (2003). Time perception deficits in attention-deficit/hyperactivity disorder and comorbid reading difficulties in child and adolescent samples. Journal of Child Psychology and Psychiatry, 44, 888–903. Van De Voorde, S., Roeyers, H., Verte´, S., & Wiersema, J. R. (2010). Working memory, response inhibition, and within-subject variability in children with attention-deficit/ hyperactivity disorder or reading disorder. Journal of Clinical and Experimental Neuropsychology, 32, 366–379. Weiler, M. D., Bernstein, J. H., Bellinger, D., & Waber, D. P. (2002). Information processing deficits in children with attention-deficit/hyperactivity disorder, inattentive type, and children with reading disability. Journal of Learning Disabilities, 35, 449–462. Weiler, M. D., Bernstein, J. H., Bellinger, D. C., & Waber, D. P. (2000). Processing speed in children with attention-deficit/hyperactivity disorder, inattentive type. Child Neuropsychology, 6, 218–234. Willcutt, E. G., Betjemann, R. S., Pennington, B. F., Olson, B. F., DeFries, J. C., & Wadsworth, S. J. (2007). Longitudinal study of reading disability and attention-deficit/ hyperactivity disorder: Implications for education. Mind, Brain, and Education, 1, 181–192. Willcutt, E. G., Betjemann, R. S., Wadsworth, S. J., Samuelsson, S., Corley, R., DeFries, J. C., Byrne, B., Pennington, B. F., & Olson, R. K. (2007). Preschool twin study of the relation between attention-deficit/hyperactivity disorder and prereading skills. Reading and Writing, 20, 103–125. Willcutt, E. G., DeFries, J. C., Pennington, B. F., Olson, R. K., Smith, S. D., & Cardon, L. R. (2003). Genetic etiology of comorbid reading difficulties and ADHD. In: R. Plomin, J. C. DeFries, P. McGuffin & I. Craig (Eds), Behavioral genetics in a postgenomic era (pp. 227–246). Washington, DC: American Psychological Association. Willcutt, E. G., Doyle, A. E., Nigg, J. T., Faraone, S. V., & Pennington, B. F. (2005). Validity of the executive function theory of attention-deficit/hyperactivity disorder: A metaanalytic review. Biological Psychiatry, 57, 1336–1346. Willcutt, E. G., & Pennington, B. F. (2000). Psychiatric comorbidity in children and adolescents with reading disability. Journal of Child Psychology and Psychiatry, 41, 1039–1048. Willcutt, E. G., Pennington, B. F., Boada, R., Ogline, J. S., Tunick, R. A., Chhabildas, N. A., & Olson, R. K. (2001). A comparison of the cognitive deficits in reading disability and attention-deficit/hyperactivity disorder. Journal of Abnormal Psychology, 110, 157–172.


211

Willcutt, E. G., Pennington, B. F., Olson, R. K., Chhabildas, N., & Hulslander, J. (2005). Neuropsychological analyses of comorbidity between reading disability and attention deficit hyperactivity disorder: In search of the common deficit. Developmental Neuropsychology, 27, 35–78. Willcutt, E. G., Pennington, B. F., Olson, R. K., & DeFries, J. C. (2007). Understanding comorbidity: A twin study of reading disability and attention-deficit/hyperactivity disorder. American Journal of Medical Genetics (Neuropsychiatric Genetics), 8, 709–714. Wolf, M., & Bowers, P. G. (1999). The double-deficit hypothesis for the developmental dyslexias. Journal of Educational Psychology, 91, 387–407.

ASSESSMENT AND INTERVENTION IN SELF-DETERMINATION Michael L. Wehmeyer ABSTRACT Promoting the self-determination of students, particularly adolescents, with disabilities has become best practice in special education and transition services. Research documents that students who leave school as more self-determined young people achieve more positive employment and independent living outcomes and experience a higher quality of life. Further, promoting self-determination can provide an entry point to the general education curriculum for students with disabilities, and instruction to promote self-determination can enable students to better engage with and learn in the general education curriculum. This chapter defines the self-determination construct as it applies to the education of students with disabilities, examines the importance of such instruction, and provides information with regard to prevailing practices in assessment and instruction to promote this outcome.

Promoting the self-determination of students with disabilities has become a best practice in secondary education and transition services (Wehmeyer, Abery, Mithaug, & Stancliffe, 2003; Wehmeyer et al., 2007) for several reasons. First, self-determination status has been linked to the attainment of more positive academic (Konrad, Fowler, Walker, Test, & Wood, 2007; Assessment and Intervention Advances in Learning and Behavioral Disabilities, Volume 24, 213–249 Copyright r 2011 by Emerald Group Publishing Limited All rights of reproduction in any form reserved ISSN: 0735-004X/doi:10.1108/S0735-004X(2011)0000024011

213

214

MICHAEL L. WEHMEYER

Fowler, Konrad, Walker, Test, & Wood, 2007; Lee, Wehmeyer, Soukup, & Palmer, 2010) and transition outcomes, including more positive employment and independent living outcomes (Martorell, Gutierrez-Recacha, Perda, & Ayuso-Mateos, 2008; Wehmeyer & Palmer, 2003; Wehmeyer & Schwartz, 1997), and more positive quality of life and life satisfaction (Lachapelle et al., 2005; Nota, Ferrrari, Soresi, & Wehmeyer, 2007; Shogren, Lopez, Wehmeyer, Little, & Pressgrove, 2006; Wehmeyer & Schwartz, 1998). Second, research across special education disability categories has established the need for intervention to promote self-determination, documenting that students with intellectual disability (Wehmeyer, Agran et al., 2007), learning disabilities (Field, Sarver, & Shaw, 2003; Pierson, Carter, Lane, & Glaeser, 2008), emotional and behavioral disorders (Carter, Lane, Pierson, & Glaeser, 2006; Pierson et al., 2008) and autism (Wehmeyer & Shogren, 2008) are less self-determined than their nondisabled peers. Third, teachers believe that teaching students to become more self-determined is important (Carter, Lane, Pierson, & Stang, 2008; Thoma, Pannozzo, Fritton, & Bartholomew, 2008; Wehmeyer, Agran, & Hughes, 2000), and there are numerous curricular and instructional models identified to enable them to provide this instructional focus (Test, Karvonen, Wood, Browder, & Algozzine, 2000; Wehmeyer & Field, 2007). In a meta-analysis of single subject and group subject design studies, Algozzine, Browder, Karvonen, Test, and Wood (2001) found evidence for the efficacy of instruction to promote component elements of self-determined behavior, including selfadvocacy, goal setting and attainment, self-awareness, problem-solving skills, and decision-making skills. Cobb, Lehmann, Newman-Gonchar, and Morgen (2009) conducted a narrative metasynthesis – a narrative synthesis of multiple meta-analytic studies – covering seven existing meta-analyses examining self-determination and concluded that there is sufficient evidence to support the promotion of self-determination as effective. Also, research documents the positive impact of efforts to promote student involvement in educational and transition planning (Martin et al., 2006; Mason, Field, & Swailowsky, 2004; Test et al., 2004) on more positive transition and selfdetermination-related outcomes. Fourth, research has begun to document the effect of a variety of intraindividual or personal factors and environmental or ecological factors that serve as mediating or moderating variables in efforts to promote selfdetermination. Multiple studies have shown that individual and environmental factors impact a person’s relative self-determination (Nota, Ferrrari, Soresi, & Wehmeyer, 2007; Shogren et al., 2007; Stancliffe, Abery, & Smith, 2000; Wehmeyer & Bolding, 1999, 2001). Among these, research documents

Self-Determination

215

a consistent, significant positive relationship between self-determination and IQ scores (Stancliffe et al., 2000; Wehmeyer & Garner, 2003), though this relationship has, equally consistently, been of marginal practical significance, with correlations around r ¼ .15 to r ¼ .24. Research examining differences in self-determination by gender has found mixed results. Shogren et al. (2007) found that gender did predict self-determination status, while Wehmeyer and Garner (2003) found no differences on overall selfdetermination scores by gender. Nota et al. (2007) and Shogren et al. (2008) also found that gender significantly impacted self-determination, though Nota and colleagues, with an Italian sample, found that males had higher self-determination scores and Shogren and colleagues, with an American sample, identified females as having higher self-determination scores. Given its importance in educating students with disabilities, the purpose of this chapter is to define self-determination and examine its importance to successful education outcomes and, then, to identify methods, materials, and strategies to assess and promote self-determination.

WHAT IS SELF-DETERMINATION? The construct’s origins lie in 19th- and 20th-century discussions in philosophy and psychology about determinism and free will. Determinism is the philosophical doctrine positing that events, such as human behavior, are effects of preceding causes. Self-determination, as proposed by psychological theorists, is a general psychological construct within the organizing structure of theories of human agentic behavior. An agentic person is the ‘‘origin of his or her actions, has high aspirations, perseveres in the face of obstacles, sees more and varied options for action, learns from failures, and overall, has a greater sense of well being’’ (Wehmeyer & Little, 2009). Theories of human agentic behavior are organismic in that they view people as active contributors to, or authors of, their behavior, which is selfregulated and goal-directed action. Such actions are motivated by biological and psychological needs, directed toward self-regulated goals linked to these needs, precipitate self-governance of behavior, and require an explicit focus on the interface between the self and the context. Self-determination as a psychological construct, then, refers to self- (vs. other-) caused action – to people acting volitionally, based on their own will. Volition is the capability of conscious choice, decision, and intention. Self-determined behavior is volitional, intentional, and self-caused or self-initiated action.

216

MICHAEL L. WEHMEYER

Several theoretical models have emerged within the disability and special education literature that view self-determination within this organismic, agentic perspective as applied to the development of interventions in educational contexts. Wehmeyer et al. (2003) proposed a functional model of self-determination in which self-determination is conceptualized as a dispositional characteristic (enduring tendencies used to characterize and describe differences between people) based on the function a behavior serves for an individual. Self-determined behavior refers to ‘‘volitional actions that enable one to act as the primary causal agent in one’s life and to maintain or improve one’s quality of life’’ (Wehmeyer, 2005, p. 117). Broadly, causal agency implies that it is the individual who makes or causes things to happen in his or her life. Causal agency implies more than just causing action; it implies that the individual acts with an eye toward causing an effect to accomplish a specific end or to cause or create change. According to this theory, self-determined actions are identified by four essential characteristics: (1) the person acts autonomously; (2) the behavior is self-regulated; (3) the person initiates and responds to the event(s) in a psychologically empowered manner; and (4) the person acts in a self-realizing manner. These essential characteristics refer not to the behavior performed, but to the function the behavior serves for the individual; that is, whether the action enabled the person to act as a causal agent. These essential characteristics emerge across time as students develop and acquire skills and attitudes pertaining to a set of component elements of self-determined behavior (discussed in greater detail in the intervention section) that include, though are not necessarily limited to, choice, problem-solving, and decisionmaking skills; goal-setting and attainment skills; self-advocacy skills; selfmanagement, self-regulation, and self-directed learning skills; perceptions of control and efficacy; and self-knowledge and self-awareness skills. The theory’s use of behavioral autonomy draws from two sources: autonomy as synonymous with individuation and autonomy as roughly synonymous with independence. Developmental psychologists view the process of individuation, or the formation of the person’s individual identity, as a critical component of social and personality development. Individuation is, generally, the progression from being dependent on others for care and guidance to self-care and self-direction, the outcome of which is autonomous functioning or behavioral autonomy. Self-regulation is ‘‘a complex response system that enables individuals to examine their environments and their repertoires of responses for coping with those environments to make decisions about how to act, to act, to evaluate the desirability of the outcomes of the action, and to revise their

Self-Determination

217

plans as necessary’’ (Whitman, 1990, p. 373). Zimmerman and Rappaport (1988) forwarded the construct of psychological empowerment to account for the multidimensional nature of perceived control, which, according to these authors, had been previously treated as if it were a unidimensional construct. Through the process of learning and using problem-solving skills and achieving perceived or actual control in one’s life (e.g., learned hopefulness), individuals develop a perception of psychological empowerment that enables them to achieve desired outcomes. The term ‘‘self-realization’’ was used originally by Gestalt psychologists to refer to the intrinsic purpose in a person’s life. It also has more global meaning related to the ‘‘tendency to shape one’s life course into a meaningful whole’’ (Angyal, 1941, p. 355). People who are self-determined are self-realizing in that they use a comprehensive, and reasonably accurate, knowledge of themselves – their strengths and limitations – to act in such a manner as to capitalize on this knowledge. This self-knowledge and selfunderstanding forms through experience with and interpretation of one’s environment and is influenced by evaluations of significant others, reinforcement, and attributions of one’s own behavior. The primary research focus of the functional theory has been on people with intellectual disability, although the theory itself is not specific to people with disabilities. In another theoretical model derived from research in special education, Mithaug suggested that self-determination is an unusually effective form of self-regulation in which people who are self-determined regulate their choices and actions more successfully than others (Wehmeyer et al., 2003). Mithaug suggested that individuals are often in flux between existing states and goal or desired states. When a discrepancy between what one has and what one wants exists, an incentive for self-regulation and action becomes operative. With the realization that a discrepancy exists, the individual may set out to achieve the goal or desired state. Because of a previous history of failure, however, individuals may set expectations that are too low or too high. The ability to set appropriate expectations is based on the individual’s success in matching his or her capacity with present opportunity. Capacity is the individual’s assessment of existing resources (e.g., skills, interests, motivation), and opportunity refers to aspects of the situation that allow the individual to achieve the desired gain. Mithaug referred to optimal prospects as ‘‘just-right’’ matches in which individuals are able to correctly match their capacity (i.e., skills, interests) with existing opportunities (e.g., potential jobs). The experience generated during selfregulation is a function of repeated interaction between capacity and opportunity.

218

MICHAEL L. WEHMEYER

Mithaug (1998) noted that ‘‘self-determination always occurs in a social context’’ (p. 42) and suggests that ‘‘the social nature of the concept is worth reviewing because the distinction between self-determination and otherdetermination is nearly always in play when assessing an individual’s prospects for controlling their life in a particular situation’’ (p. 42).

Self-Determination, Disability, and Empowerment As noted, the self-determination focus in disability services, including special education and rehabilitation, emerged from deeply held convictions pertaining to the rights of people with disabilities to ‘‘control’’ their own lives. Within the context of the disability rights and advocacy movement, the self-determination construct has been imbued with an empowerment and ‘‘rights’’ orientation. Empowerment is a term usually associated with social movements and typically is used, as Rappaport (1981) stated, in reference to actions that ‘‘enhance the possibilities for people to control their lives’’ (p. 15). People have been unequivocal in their understanding of selfdetermination as a form of empowerment. The emphasis on promoting self-determination was introduced both in response to calls from people with disabilities for greater control in their lives, as well as to findings that transition and rehabilitation outcomes for people with disabilities (e.g., employment, independent living, community integration, etc.) were not as positive as desired. The proposal that self-determination is an important focus if people with disabilities are to achieve more positive adult outcomes is predicated on an assumption that self-determination and positive adult outcomes are causally linked; an assumption supported by a growing literature base. Wehmeyer and Schwartz (1997) measured the selfdetermination of students with cognitive disabilities, and then examined their adult outcomes one year after they left high school. Students in the high-selfdetermination group were more than twice as likely as young people in the low-self-determination group to be employed, and earned, on average, more per hour. Wehmeyer and Palmer (2003) conducted a second follow-up study, examining adult status of 94 students with cognitive disabilities one and three years postgraduation. One year after high school, students in the high-selfdetermination group were disproportionately likely to have moved from where they were living during high school, and by the third year they were still disproportionately likely to live somewhere other than their high school home and were significantly more likely to live independently. For employed students, those scoring higher in self-determination made statistically

Self-Determination

219

significant advances in obtaining job benefits, including vacation, sick leave, and health insurance, an outcome not shared by their peers in the low-selfdetermination group. Sowers and Powers (1995) showed that instruction on multiple components related to self-determination increased the participation and independence of students with severe disabilities in performing community activities. More recently, Wehmeyer et al. (in press) conducted a randomized-trial, placebo control group study of the effect of intervention, using the selfdetermined learning model of instruct (discussed subsequently) on outcomes including student self-determination, access to the general education curriculum, and transition and academic goal attainment, and determined that students involved in such instruction became more self-determined, had greater access to the general education curriculum, and had higher rates of goal attainment than did students in the control group.

ASSESSING INSTRUCTIONAL NEEDS IN SELF-DETERMINATION As is the case with instruction in any content area, assessment and instruction go hand-in-hand in efforts to promote self-determination. Determining instructional and curricular needs in the area of selfdetermination will involve a combination of standardized and informal procedures incorporating input from multiple sources, including the student, his or her family, professionals, and others. Informal procedures will be similar to those described by Clark (1996) with regard to transition assessment. Clark identified informal assessment from which transitionrelated decisions can be made as including: (a) situational or observational learning styles assessments, (b) curriculum-based assessment, (c) observational reports from teachers, employers, and family members, (d) situational assessments in home, community, and work settings, (e) environmental assessments, (f) personal future planning activities, (g) structured interviews with students, (h) structured interviews with parents, guardians, advocates, or peers, (i) adaptive, behavioral, or functional skill inventories, (j) social histories, (k) employability, independent living, and personal–social skills rating scales, and (l) technological or vocational education skills assessments. These types of assessment procedures enable planners to form a complete picture of student needs, interests, and abilities by gathering input from

220

MICHAEL L. WEHMEYER

multiple sources, and are important for determining the same things as they pertain to the need for instruction to promote self-determination.

Norm-Referenced Measures of Self-Determination The Arc’s Self-Determination Scale1 The Arc’s Self-Determination Scale (SDS, Wehmeyer & Kelchner, 1995) is a 72-item self-report measure based on the functional theory of selfdetermination. A total of 148 points are available on the scale, with higher scores indicating higher levels of self-determination. An overall selfdetermination score, as well as subscale scores for each of the four essential characteristics of self-determined behavior – autonomy, self-regulation, psychological empowerment, and self-realization – identified in the functional model of self-determination (Wehmeyer et al., 2003) can be calculated. The SDS was developed and normed with 500 adolescents with cognitive disabilities (Wehmeyer, 1996). Subsequent research (Shogren, et al., 2007, 2008) has verified the proposed theoretical structure of the SDS (i.e., four related, but distinct subscales (autonomy, self-regulation, psychological empowerment, and self-realization) that contribute to a higher-order selfdetermination construct). The SDS was demonstrated to have adequate reliability and validity in the measurement of self-determination for adolescents with cognitive disabilities (see Wehmeyer (1996) for details about reliability and validity). The scale has been used to conduct research into the relationship between self-determination and positive adult outcomes (Wehmeyer & Schwartz, 1997) and quality of life variables (Wehmeyer & Schwartz, 1998), the relationship between self-determination and environmental factors (Wehmeyer & Bolding, 1999, 2001), and to validate instructional strategies to promote selfdetermination (Wehmeyer, Palmer, Agran, Mithaug, & Martin, 2000) and materials to promote student-directed transition planning (Wehmeyer & Lawrence, 1995). One potential use of the SDS is to generate discussion about items the student finds interesting, problematic, or wants to discuss more broadly. A second use of the scale involves scoring it and comparing total, domain, and subdomain scores with scale norms and, more importantly, examining individual strengths and weaknesses across the domains. The normed data is provided only as a point of comparison, not so that students who perform below the mean should feel a sense of failure or otherwise use the information in a pejorative manner. Less than optimal performances in any

221

Self-Determination

area of the SDS should be followed by learning opportunities and experiences that enable the student to make progress in that particular area. Any use of the SDS with individual students should focus on potential educational goals and objectives. This discussion, in turn, can consider possible educational programs and activities to address and meet these goals and objectives. The AIR Self-Determination Scale2 The AIR Self-Determination Scale (AIR; Wolman, Campeau, Dubois, Mithaug, & Stolarski, 1994) assesses student capacity and opportunity for self-determination. The AIR has a Student, Educator, and Parent version and the Student (AIR-S) self-report version was utilized in the present study. The AIR-S version has 24 questions and also yields capacity and opportunity subscale scores. The capacity subscale consists of questions related to things students do related to self-determination (‘‘Things I Do’’ subscale) and how students feel about performing these self-determined behaviors (‘‘How I Feel’’ subscale). The opportunity subscale consists of questions regarding students’ perceptions of their opportunities to perform self-determined behaviors at home and at school. The AIR was developed and normed with 450 students with and without disabilities in California and New York (Wolman et al., 1994). The AIR was demonstrated to have adequate reliability and validity in the measurement of capacity and opportunity for self-determination (see Wolman et al., 1994 for details). Recent research (Shogren et al., 2008) has confirmed the theoretical structure of the AIR (i.e., two related subscales – capacity and opportunity – that contribute to a higher-order self-determination construct). This research also confirmed that while the SDS and the AIR-S are related (r ¼ .50), they measure distinct aspects of the self-determination construct. Self-Determination Assessment Battery3 A third assessment option involves the Self-Determination Assessment Battery developed by Field, Hoffman, and Sawilowsky (2004). The SelfDetermination Assessment Battery contains multiple tools that measure cognitive, affective, and behavioral factors related to self-determination. In addition, these factors are assessed from the perspectives of the student, the teacher, and the parent. This battery of instruments was developed to assess knowledge, behavior, and affective components of self-determination from

222

MICHAEL L. WEHMEYER

these varied perspectives and within the context of an intervention theory to promote self-determination forwarded by Field and Hoffman (1994). There are five instruments in the Self-Determination Assessment Battery (Field, et al., 2004), each of which is described below. 1. 2. 3. 4. 5.

Self-Determination Self-Determination Self-Determination Self-Determination Self-Determination

Knowledge Scale (SDKS) Pretest, forms A & B Parent Perception Scale (PPS) Teacher Perception Scale (TPS) Observation Checklist (SDOC) Student Scale (SDSS)

Self-Determination Knowledge Scale Pretest The SDKS-pre and SDKS-post are 37-item structured response instruments designed to assess the student’s cognitive knowledge of self-determination skills as taught in the Field and Hoffman (2005) Steps to Self-Determination curriculum. Approximately one-third of the items are in true–false format and the balance are multiple-choice questions (with three options). The reading level for these instruments is approximately fifth grade, enabling their use with students with mild–moderate cognitive disabilities. Self-Determination Parent Perception Scale and Teacher Perception Scale The PPS and TPS are 30-item questionnaires that are administered to parents and teachers, respectively. The items in these questionnaires were also derived from the Field and Hoffman (1994) model of self-determination intervention. The teacher or parent rates their student or child on a fivepoint Likert scale of ‘‘0’’ ¼ low to ‘‘4’’ ¼ high on a variety of behaviors, abilities, and skills associated with self-determination. Self-Determination Observation Checklist The SDOC is a 38-item behavioral observation checklist designed to be administered by classroom teachers or other appropriate personnel in the school environment. The student is observed for approximately five minutes during a class period. Behaviors that correlate to self-determination are checked. Self-Determination Student Scale The SDSS is a 92-item self-report instrument that measures both affective and cognitive aspects of the student’s self-determination. The items contain a brief stimulus, to which the student marks ‘‘That’s me’’ or ‘‘That’s not me.’’ The SDSS yields a variety of subscale scores, including General

Self-Determination

223

Positive, General Negative, Specific Positive, and Specific Negative. The general subscales relate to a student’s sense of global self-determination, while the specific subscales relate primarily to application in their education, home, and related environmental settings. The positive subscales indicate self-determination in areas of perceived strength, while the negative subscales indicate areas of perceived weakness in self-determination. The Self-Determination Assessment Battery instruments have many possible uses in education. First, they can be used to assist in educational planning. Because the instruments take into account the perspectives of the student, teacher, and parent, it is possible to identify areas of similarity and discrepancy among these three perspectives. This may provide insight to students regarding their functioning in different areas of their lives or how they are perceived in different situations. For example, a student may be rated more highly on some components by the parent than by the teacher. This provides an opportunity for discussion among the student, teacher, and parent to determine the reasons for this discrepancy. It may be that the student is exhibiting skills at home that he/she is not displaying at school or it may be that the teacher and the parent were using different criteria to evaluate the student’s performance. The discussion that can be generated from examining these differences can provide important feedback for the student and can lead to determining appropriate instructional interventions. Just as students are being rated from three different perspectives (i.e., the student, teacher, and parent), they are also being assessed in three different areas: cognition/knowledge, behavior, and affect. Examining the differences in the three different areas helps to determine appropriate interventions. The instruments clearly have varied uses for educational planning, both as a discussion tool in educational planning meetings that can help to promote greater self-awareness and as a tool that can help to identify appropriate educational interventions. In addition, the instruments can be used for program evaluation or research purposes. By using the instruments as preand posttests before and after an instructional intervention, data can be obtained that can help to assess the effectiveness of the intervention.

PROMOTING SELF-DETERMINATION Efforts to enhance the self-determination of youth with disabilities should involve multiple, parallel activities focused on teaching skills related to the component elements of self-determined behavior and promoting active involvement in educational planning and decision-making. As has already

224

MICHAEL L. WEHMEYER

been described, such interventions can positively impact transition-related outcomes for youth and young adults with disabilities.

Linking Assessment and Intervention As noted in the section on assessment, as is the case in any content area, instruction to promote self-determination should both be derived from and be informed by assessment. The following section describes intervention approaches grouped by areas identified in the functional model, discussed previously. It should be noted that the SDS was developed to operationalize that functional model, and assessment with that instrument provides data concerning instructional needs across the essential characteristics and, importantly, the component elements described in the model and articulated subsequently. As another example of means to link assessment and intervention, Field and Hoffman’s (1994) instructional model (depicted in Fig. 1) is directly linked to the Self-Determination Assessment Battery described previously, and assessment using those instruments can determine where in the instructional model intervention should begin.

Infusing Self-Determination Instruction into the General Education Curriculum As noted, infusing instruction on component elements of self-determined behavior into instruction across content areas provides the first focus for promoting self-determination. This section briefly identifies some key strategies to achieve this instructional outcome. It is worth noting that in the context of inclusive education and of school reform efforts that emphasize access to the general education curriculum, instruction on these component elements are relevant for all students, and not just students with disabilities, and that the content and student performance standards in most states include objectives pertaining to most of these component elements, and as such instruction to promote self-determination can be linked to the general education curriculum (Wehmeyer, Field, Doren, Jones, & Mason, 2004). Goal Setting and Attainment Goal setting and attainment skills are critical to students with disabilities becoming more self-determined. Goals act as regulators of human

225

Self-Determination

Know Yourself and Your Environment • • • •

Value Yourself • • •

Dream Know your strengths, weaknesses, needs and preferences Know options, supports and expectations Decide what is important to you

• •

Accept and value yourself Use weaknesses to find strengths Recognize and respect rights and responsibilities Take care of yourself Develop and nurture positive relationships

Plan • • • • •

Set goals Plan small steps to meet goals Anticipate results Be creative Visually rehearse

• • • • • •

Take risks Communicate Access resources and support Negotiate Deal with conflict and criticism Be focused and persistent

Act

Experience Outcomes and Learn • • • •

Fig. 1.

Compare outcome to expected outcome Compare performance to expected performance Realize success Make adjustments

Field and Hoffman (1994) Self-Determination Intervention Model. Reprinted with permission.

behavior. If a person sets a goal, it increases the probability that he or she will perform behaviors related to that goal (Latham & Locke, 1991). The process of promoting goal-setting and attainment skills involves teaching students to (1) identify and define a goal clearly and concretely, (2) develop a series of objectives or tasks to achieve the goal, and (3) specify the actions necessary to achieve the desired outcome. At each step, students must make choices and decisions about what goals they wish to pursue and

226

MICHAEL L. WEHMEYER

what actions they wish to take to achieve their goals. Goal-setting activities can be easily incorporated into a variety of transition-related activities and across multiple instructional areas, as well as in the educational planning process. Understanding the goal process is also important for social interactions, as understanding the goals and intentions behind the actions of others is critical to many such interactions (Hamilton, 2009) and instruction may be particularly important for certain populations of students. For example, research suggests that students with autism spectrum disorders tend to be more sequential in their goal-directed behavior, have difficulty engaging in multiple goal-directed activities concurrently, and tend to jump from activity to activity in the process of goal attainment (Ruble & Scott, 2002). Several strategies could be utilized to address this issue: Complex goals could be broken down into smaller subgoals, with fewer steps, that the student could complete in a shorter amount of time. Students could make a list of goals they are working toward, so they have a concrete, visual reminder of their goals. Further, strategies to promote self-regulated behavior (discussed below) could be utilized to enable students to self-monitor their progress toward their goals (Wehmeyer & Shogren, 2008). Research has suggested some strategies to make goals both meaningful and attainable for students with disabilities. First, goals should be challenging. They should not be so difficult, though, that the student cannot attain them, as this will lead to frustration and withdrawal from participation, but must provide enough motivation for the student to work to attain them. If goals are too easy, there is no motivation to engage in the work necessary to attain them, nor is there a feeling of accomplishment after achieving them. While it is preferable for students to participate in setting their own goals, at whatever level feasible, if this is not possible and goals need to be set by teachers, then the student’s preferences and interests should be incorporated into the goal to increase the student’s motivation to pursue the goal. Goals that have personal meaning are more likely to be attained (Locke & Latham, 1990). Choice Making Choice making (e.g., the expression of a preference between two or more options) has received considerable attention in the self-determination literature. One of the early impetuses for the educational emphasis on self-determination in the disability field was the increasing recognition of the lack of choice-making opportunities available to people with disabilities

Self-Determination

227

(Dunlap, Kern-Dunlap, Clarke, & Robbins, 1994; Wehmeyer & Bolding, 1999, 2001), and the negative impact this had both on behavior and development. Promoting choice making has several benefits. First, when students with disabilities are provided opportunities to make choices, reductions in problem behavior and increases in adaptive behaviors are observed (Shogren, Faggella-Luby, Bae, & Wehmeyer, 2004). Second, by making choices, students, particularly younger children, learn that they can exert control over their environment. For students to fully understand the process of choice making, including the various effects of making certain choices, choices need to be real and meaningful to the student. Some students may need to learn how to make choices, particularly if the student’s previous opportunities to do so have been restricted. To do this, picture cues can be used to teach students to choose between two or more depicted activities, with the selection of an activity followed immediately by performance of the activity (Bambara & Ager, 1992). A variety of adaptive equipment, ranging from picture communication systems to computer technology, can be used to support students with more severe disabilities to indicate their preferences. Choice opportunities can and should be infused through the school day. Students can be provided opportunities to choose within or between instructional activities. They can also choose with whom they engage in a task, where they engage in an activity, and if they complete an activity (Brown, Appel, Corsi, & Wenig, 1993). Problem Solving A problem is an activity or task for which a solution is not known or readily apparent. The process of solving a problem involves (a) identifying and defining the problem, (b) listing possible solutions, (c) identifying the impact of each solution, (d) making a judgment about a preferred solution, and (e) evaluating the efficacy of the judgment (D’Zurilla & Goldfried, 1971). A number of strategies to promote problem solving have been evaluated for students with disabilities. Bauminger (2002, 2007a, 2007b) developed a curriculum to teach students with autism, which focused on social and interpersonal problem-solving skills. Students were taught about social concepts, such as starting a conversation, and then presented a vignette of a student having difficulty implementing the skill. Students went through an eight-stage problem-solving process with their teacher where they (a) defined the problem, (b) discussed the emotions associated with the problem, (c) defined the alternative social actions, (d) considered the consequences of each alternative, (e) made a decision about the best alternative,

228

MICHAEL L. WEHMEYER

(f) role-played the solution with their teacher, (g) received homework to practice the social skill covered in the lesson at home with peers, and (h) received feedback from the teacher on the homework. After seven months, students generated more appropriate solutions to problems faced in social situations, and initiated more social interactions with their peers. Bernard-Opitz, Sriram, and Nakhoda-Sapuan (2001) developed a computer program to teach students with developmental disabilities social problem-solving skills. The program first presented pictures or videos of people experiencing social conflicts. The program guided students through an animated problem-solving process, in which they were asked to generate alternative solutions. After identifying an alternate solution, a video clip of the actors resolving the problem was presented. As students had repeated experience with the program, they generated more alternative solutions. The increase in the generation of solutions observed in both of these studies is important, as research suggests that generating more solutions to a problem often leads to a better resolution of the problem (D’Zurilla & Nezu, 1980). Decision Making A decision-making process involves coming to a judgment about which solution is best at a given time. Making effective decisions typically involves (a) identifying alternative courses of action, (b) identifying the possible consequences of each action, (c) assessing the probability of each consequence occurring, (d) choosing the best alternative, and (e) implementing the decision (Furby & Beyth Marom, 1992). Although the ability to engage in this process develops with age, research has shown that young children can engage in a systematic decision-making process, often by reducing and simplifying the steps in the decision-making process, although they are not as effective as older students (Crone, Vendel, & van der Molen, 2003). Thus, working to promote systematic decision-making skills is best addressed at the secondary level, while at the elementary level a focus on choice making and problem solving can support the development of effective decision-making skills later in life. Studies have shown repeatedly that adolescents with disabilities can effectively participate in the decision-making process (Taylor, Adelman, & Kaser Boyd, 1983, 1985; Wehmeyer & Lawrence, 1995). Additionally, research shows that students with disabilities want to be involved in decisions related to their life (Ruef & Turnbull, 2002). To support the acquisition of decision-making skills, a number of strategies can be implemented. Early on, students should be provided a wide array of choice opportunities, and receive instruction regarding how to make effective

Self-Determination

229

choices, as discussed previously. As students age, they should be provided overt instruction in the decision-making process. A number of curricular approaches have been developed to promote decision-making skills (see Baron & Brown, 1991), all of which can be individualized based on a student’s learning and support needs. When teaching decision-making skills, opportunities to make decisions should be imbedded in the curriculum. By supporting students to make decisions in ‘‘real-world’’ situations, they will better develop their ability to conceptualize and generalize the decisionmaking process. Self-Regulation and Student-Directed Learning Self-regulation is the process of setting goals, developing action plans to achieve those goals, implementing and following the action plans, evaluating the outcomes of the action plan, and changing actions plans, if the goal was not achieved (Mithaug, Mithaug, Agran, Martin, & Wehmeyer, 2003). The skills associated with self-regulation enable students to examine their environments, evaluate their repertoire of possible responses, and implement and evaluate a response (Whitman, 1990). Student-directed learning strategies involve teaching students strategies that enable them to modify and regulate their own behavior (Agran, KingSears, Wehmeyer, & Copeland, 2003). The emphasis in such strategies is shifted from teacher-mediated and -directed instruction to student-directed instruction. Research in education and rehabilitation has shown that studentdirected learning strategies are as successful, and often more so, as teacherdirected learning strategies, and that these strategies are effective means to increase independence and productivity. Among the most commonly used strategies are picture cues and antecedent cue-regulation strategies, selfinstruction, self-monitoring, self-evaluation, and self-reinforcement. These are briefly defined below (from Agran, King-Sears et al., 2003, see this for a comprehensive treatment of student-directed learning strategies). Picture cues and antecedent cue-regulation strategies involve the use of visual or audio cues that students use to guide their behavior. Visual cues typically involve photographs, illustrations, or line drawings of steps in a task that support students to complete an activity that consists of a sequence of tasks. Audio cues include prerecorded, taped directions or instructions that the students can listen to as they perform a task. Emerging technologies, like handheld computers, provide new and potentially powerful vehicles to deliver visual or auditory cues to learners. Selfinstruction involves teaching students to provide their own verbal cues prior to the execution of target behaviors. Self-monitoring involves teaching

230

MICHAEL L. WEHMEYER

students to observe whether they have performed a targeted behavior and whether the response met whatever existing criteria present. Self-evaluation involves teaching the student to compare his or her performance (as tracked through self-monitoring) and comparing that performance with a desired goal or outcome. Self-reinforcement involves teaching students to administer consequences to themselves (e.g., verbally telling themselves they did a good job). Self-reinforcement allows students to provide themselves with reinforcers that are accessible and immediate. Given access to selfadministered reinforcement, behavior change may be greatly facilitated. Self-Advocacy Students with disabilities need to learn the skills to advocate on their own behalf. To be an effective self-advocate, students have to learn both how to advocate and what to advocate for. There are ample opportunities for students to practice and learn self-advocacy skills within the context of the educational planning process. Too often, students’ perspectives have been lost, because students have not had the opportunities or the skills to express their perspective within the IEP, transition, or general educational planning meetings. A first step to enabling students to express their wants and needs during these meetings is educating students about their rights and responsibilities in these areas. They can be educated about their educational rights and responsibilities under the Individuals with Disabilities Education Act (IDEA), about their civil rights in under the Americans with Disabilities Act (ADA), or more generally about the rights available to citizens. When teaching students how to advocate for themselves, the focus should be on teaching students how to be assertive, how to effectively communicate their perspective (either verbally or in written or pictorial form), how to negotiate, how to compromise, and how to deal with systems and bureaucracies. Students need to be provided real-world opportunities to practice these skills. This can be done by embedding opportunities for selfadvocacy within the school day, by allowing students to set up a class schedule, work out their supports with a resource room teacher or other support provider, or participate in IEP and transition meetings. Perceptions of Efficacy and Control People who have positive perceptions of their efficacy believe they can perform the behavior required to achieve a desired outcome (Bandura, 1977). Further, individuals also have efficacy expectations, which are beliefs about the probability of the performance of a given behavior leading to a desired outcome. Research has shown that students with disabilities tend to

Self-Determination

231

have less adaptive perceptions of efficacy and outcome expectations than do students without disabilities (Wehmeyer, 1994). People who believe they have the ability to exert control over their lives and outcomes tend to be described as having an internal locus of control, whereas people who perceive others to largely control their lives and outcomes are described as having an external locus of control (Rotter, 1966). Research has shown that students’ perceptions of control and efficacy interact with academic, social, and behavioral outcomes, with students who have more adaptive perceptions of their abilities in each of these areas experiencing more positive outcomes (Wehmeyer, 1994). Students must be provided opportunities to develop adaptive perceptions of their efficacy in performing given behaviors and their ability to exert control over their lives. By enabling students to engage in problem solving, goal setting, and to make choices and decisions that are meaningful, they can learn that they have control over their outcomes and develop confidence in their ability to perform these behaviors and achieve their desired outcomes. Both teacher and classroom characteristics can influence students’ perceptions of efficacy and control. Overly controlling environments can diminish students’ perceptions of their ability to exert control and engage in actions that enable them to develop adaptive efficacy expectations. It is important for teachers to work to empower students to be active participants in their classrooms. Self-Awareness and Self-Knowledge For students to become more self-realizing, they must possess a reasonably accurate understanding of their strengths, abilities, unique learning and support needs, and limitations. Further, they must know how to utilize this understanding to maximize success and progress. However, like perceptions of efficacy and control, self-awareness and knowledge is not something that can simply be taught through direct instruction. Instead, students acquire this knowledge by interacting with their environment. Unfortunately, students with disabilities often learn to identify what they cannot do instead of what they can. This skews students’ perceptions of themselves, and influences how they interact with people and systems they encounter.

Self-Determined Learning Model of Instruction Like all educators, special education teachers use a variety of teaching models, defined as ‘‘a plan or pattern that can be used to shape curriculums

232

MICHAEL L. WEHMEYER

(long term courses of study), to design instructional materials, and to guide instruction in the classroom and other settings’’ (Joyce & Weil, 1980, p. 1). Such models are derived from theories about human behavior, learning, or cognition, and effective teachers employ multiple models of teaching, taking into account the unique characteristics of the learner and types of learning. The teaching model most frequently adopted by special educators is the contingency management model, drawing from operant psychology. The Self-Determined Learning Model of Instruction (Mithaug, Wehmeyer, Agran, Martin, & Palmer, 1998) is a model of teaching based on the component elements of self-determination, the process of self-regulated problem solving, and research on student-directed learning. It is appropriate for use with students with and without disabilities across a wide range of content areas, and enables teachers to engage students in the totality of their educational program by increasing opportunities to self-direct learning and, in the process, to enhance student self-determination. Implementation of the model consists of a three-phase instructional process depicted in Figs 2–4. Each instructional phase presents a problem to be solved by the student. The student solves each problem by posing and answering a series of four Student Questions per phase that students learn, modify to make their own, and apply to reach self-selected goals. Each question is linked to a set of Teacher Objectives. Each instructional phase includes a list of Educational Supports that teachers can use to enable students to self-direct learning. In each instructional phase, the student is the primary agent for choices, decisions, and actions, even when eventual actions are teacher-directed. The Student Questions in the model are constructed to direct the student through a problem-solving sequence in each instructional phase. The solutions to the problems in each phase lead to the problem-solving sequence in the next phase. Teachers implementing the model teach students to solve a sequence of problems to construct a means–ends chain – a causal sequence – that moves them from where they are (an actual state of not having their needs and interests satisfied) to where they want to be (a goal state of having those needs and interests satisfied). Its function is to reduce or eliminate the discrepancy between what students want or need and what students currently have or know. That means–ends sequence is constructed by having students answer the questions that connect their needs and interests to their actions and results via goals and plans. To answer the questions in this sequence, students must regulate their own problem solving by setting goals to meet needs, constructing plans to meet goals, and adjusting actions to complete plans. Thus, each instructional phase poses a problem the student must solve (What is my goal? What is my plan? What

233

Self-Determination

Phase 1: Set a Goal

Student Problem to Solve: What is my goal?

Educational Supports Student self-assessment of interests, abilities, and instructional needs. Awareness Training. Choice-Making Instruction. Problem-Solving Instruction. Decision-Making Instruction. Goal Setting Instruction

Teacher Objectives Enable students to identify specific strengths and instructional needs. Enable students to communicate preferences, interests, beliefs and values. Teach students to prioritize needs.

Student Question 1: What do I want to learn?

Teacher Objectives Enable students to identify their current status in relation to the instructional need. Assist students to gather information about opportunities and barriers in their environments.

Student Question 2: What do I know about it now?

Teacher Objectives Enable students to decide if action will be focused toward capacity building, modifying the environment, or both. Support students to choose a need to address from the prioritized list.

Student Question 3: What must change for me to learn what I don’t know?

Teacher Objectives Student Question 4: What can I do to make this happen?

Teach students to state a goal and identify criteria for achieving goal. Go to Phase 2

Fig. 2.

Instructional Phase 1 of Self-Determined Learning Model of Instruction.

234

MICHAEL L. WEHMEYER

Phase 2: Take Action

Student Problem to Solve: What is my plan?

Educational Supports Self-scheduling. Self-Instruction. Antecedent Cue Regulation. Choice-making instruction. Goal-Attainment strategies. Problem-solving instruction. Decision-making instruction. Self-Advocacy and assertiveness training. Communication skills training. Self-monitoring.

Teacher Objectives Student Question 5: What can I do to learn what I don't know?

Enable student to self-evaluate current status and self-identified goal status.

Teacher Objectives Enable student to determine plan of action to bridge gap between self-evaluated currrent status and self-identified goal status.

Student Question 6: What could keep me from taking action?

Teacher Objectives Collaborate with student to identify most appopriate instructional strategies. Teach student needed studentdirected learning strategies. Support student to implement student-directed learning strategies. Provide mutually agreed upon teacher-directed instruction.

Student Question 7: What can I do to remove these barriers?

Teacher Objectives Enable student to determine schedule for action plan. Enable student to implement action plan. Enable student to self-monitor progress.

Student Question 8: When will I take action?

Go to Phase 3

Fig. 3.


235

Self-Determination

Phase 3: Adjust Goal or Plan

Student Problem to Solve: What have I learned?

Educational Supports Self-evaluation strategies. Choice-making instruction. Goal-setting instruction. Problem-solving instruction. Decision-making instruction. Self-reinforcement strategies. Self-recording strategies. Self-monitoring.

Teacher Objectives Student Question 9: What actions have I taken?

Enable student to self-evaluate progress toward goal achievement.

Teacher Objectives Student Question10: What barriers have been removed?

Collaborate with student to compare progress with desired outcomes.

Teacher Objectives

Student Question 11: What has changed about what I don’t know?

Support student to re-evaluate goal if progress is insufficient. Assist student to decide if goal remains the same or changes. Collaborate with student to identify if action plan is adequate or inadequate given revised or retained goal. Assist student to change action plan if necessary.

Teacher Objectives Student Question 12: Do I know what I want to know?

Fig. 4.

Enable student to decide if progress is adequate, inadequate, or if goal has been achieved.


236

MICHAEL L. WEHMEYER

have I learned?) by, in turn, solving a series of problems posed by the questions in each phase. The four questions differ from phase to phase, but represent identical steps in the problem-solving sequence. That is, students answering the questions must (1) identify the problem, (2) identify potential solutions to the problem, (3) identify barriers to solving the problem, and (4) identify consequences of each solution. These steps are the fundamental steps in any problem-solving process and they form the means–end problem-solving sequence represented by the Student Questions in each phase and enable the student to solve the problem posed in each instructional phase. The Student Questions are written in first-person voice in a relatively simple format with the intention that they are the starting point for discussion between the teacher and the student. Some students will learn and use all 12 questions as they are written. Other students will need to have the questions rephrased to be more understandable. Still other students, due to the intensity of their instructional needs, may have the teacher paraphrase the questions for them. The first time a teacher uses the model with a student, the initial step in the implementation process is to read the question with or to the student, discuss what the question means, and then, if necessary, change the wording to enable the student to better understand the intent of the question. Such wording changes must, however, be made such that the problem-solving intent of the question remains intact. The Teacher Objectives within the model are just that – the objectives a teacher will be trying to accomplish by implementing the model. In each instructional phase, the objectives are linked directly to the Student Questions. These objectives can be met by utilizing strategies provided in the Educational Supports section of the model. The Teacher Objectives provide, in essence, a roadmap to assist the teacher to enable the student to solve the problem stated in the student question. The emphasis in the model on the use of instructional strategies and educational supports that are student-directed provides another means of teaching students to teach themselves. As important as this is, however, not every instructional strategy implemented will be student-directed. The purpose of any model of teaching is to promote student learning and growth. There are circumstances in which the most effective instructional method or strategy to achieve a particular educational outcome will be a teacher-directed strategy. Students who are considering what plan of action to implement to achieve a self-selected goal can recognize that teachers have expertise in instructional strategies and take full advantage of that expertise. The fundamental purpose of any model of instruction is to promote student learning. Teachers use models of instruction to drive curriculum and

Self-Determination

237

assessment development and to design instructional methods, materials, and strategies, all with the intent of improving the quality of the instructional experience and, presumably, enhancing student achievement. Thus, the first requirement of any model of instruction is that teachers can use the model to ‘‘teach’’ students educationally valued skills or concepts. We have proposed that the Self-Determined Learning Model of Instruction (SDLMI) has the added benefit of enhancing student self-determination. As such, evaluation activities have focused on ensuring that students with disabilities achieve educationally valued goals when provided instruction using the model and examining the impact of the model on student selfdetermination. Table 1 presents a summary of studies using the SDLMI. In general, these studies show that students provided instruction using the SDLMI achieve transition and academic achievement goals at levels greater than expected and gain skills related to self-determination. Most recently, Wehmeyer et al. (in press) conducted a randomized-trial, placebo control group study of the SDLMI, establishing a causal relationship between intervention with the model, increased self-determination, goal attainment, and access to the general education curriculum.

Student Involvement in Transition Planning Another important component of enhancing self-determination is promoting active involvement in transition planning. Test et al. (2004) conducted an extensive review of the literature pertaining to student involvement, and determined that students across disability categories can be successfully involved in transition planning, and that a number of programs, including those mentioned subsequently, are effective in increasing student involvement. Martin, Marshall, and Sale (2004) conducted a three-year study of middle, junior, and senior high-school IEP meetings and found that the presence of students at IEP meetings had considerable benefits, including increasing parental involvement and improving the probability that a student’s strengths, needs, and interests would be discussed. Involvement in education planning, decision-making, and instruction can take many forms, from students generating their own IEP goals and objectives, to tracking their progress on self-selected goals or objectives, to running their own IEP meeting. It is important to emphasize that it is not what the student does in the meeting that is critical, but, instead, the degree to which the student is an equal partner in and, to the greatest extent possible, in control of his or her planning. Students with severe disabilities can be involved in their educational program every bit as much as students

238

Table 1. Study

Wehmeyer, Palmer, Agran, Mithaug, and Martin (2000) Agran, Blanchard, and Wehmeyer (2000) McGlashingJohnson, Agran, Sitlington, Cavin, and Wehmeyer, (2003)

MICHAEL L. WEHMEYER

Empirical Validation of the Self-determined Learning Model of Instruction. Design

Subjects

Dependent Variables

Outcomes

Pre-post no control group

40 HS students: 13 w/MR, 17 w/LD, 10 w/EBD

SD, GAS, goal skills, locus of control

Progress on 80% of educational goals, enhanced SD and perceptions of control

Delayed multiple baseline across groups Multiple baseline across subjects

19 MS/HS students w/ severe MR

GAS, % correct on transition goals linked to functional needs

Students attained transition-related goals using model

4 HS students with severe MR

% Correct on job tasks, GAS

50 K-3rd grade students: 21w/ LD, 6 w/ MR, 5 w/Speech Imp, 2 w/OHI, 16w/ DD 22 MS students with MR

Goal attainment and skills

3 of the 4 participants achieved their selfselected goals; 1 student did not meet the mastery criterion, but performed at a higher level during the training condition than in baseline Students achieved goals at greater than expected level and acquired goals knowledge and skills

Palmer and Wehmeyer (2003)

Pre-post no control group

Palmer, Wehmeyer, Gipson, and Agran (2004) Agran, Cavin, Wehmeyer, and Palmer (2006) Agran, Wehmeyer, Cavin, and Palmer (2008)

Interrupted time series w/switching replication Multiple baseline across subjects Multiple baseline across subjects

Lee, Wehmeyer, Palmer, Soukup, and Little (2008)

Grouprandomized pre-post control group

SD, problemsolving skills, GAS

3 MS students with moderate MR

Academic goals linked to district standards

3 junior high students with cognitive disabilities

Goals linked to active participation in general education classroom Interval time sample data on access, selfdetermination, goal attainment

42 HS students (20 treatment/22 control) (6 w/ ADHD/ OHI, 3 w/ EBD, 32 w/ LD, 1 w/ Aut

Students able to achieve goals linked to general education curriculum using model Students mastered skills linked to standards in science and geography Students mastered classroom participation skills

Treatment group achieved significantly higher rate of goal attainment, selfregulation predicted access

Self-Determination

239

with less severe disabilities. Student involvement may look very different in these cases, and students with more severe disabilities may not be able to make independent decisions or solve problems, but that is not the criteria by which we should judge student involvement. It is, instead, the degree to which the student is actively engaged in his or her planning and education program. There are multiple advantages to student involvement. Research has shown that student involvement in goal setting and decision-making results in enhanced performance and motivation (Wehmeyer et al., 2003). Student involvement in the educational process is a good way to teach and allow students to practice skills important to self-determination (goal setting, decision-making, negotiation, etc.), self-advocacy, leadership, and teamwork. There are several programs designed to promote student involvement, and space restrictions allow only a brief description of several resources. A complete discussion of promoting student involvement is available from Wehmeyer and Field (2007). ChoiceMaker Self-Determination Transition Curriculum and Program The ChoiceMaker Self-Determination Transition Curriculum (Martin & Marshall, 1995) consists of three sections: (1) Choosing Goals, (2) Expressing Goals, and (3) Taking Action. Each section contains 2–4 teaching goals and numerous teaching objectives addressing six transition areas, including (a) an assessment tool, (b) Choosing Goals lessons, (c) the Self-Directed IEP, and (d) Taking Action lessons. The program includes a criterion-referenced self-determination transition assessment tool that matches the curricular sections. The Choosing Goals lessons enable students to learn the necessary skills and personal information needed to articulate their interests, skills, limits, and goals across one or more self-selected transition areas. The SelfDirected IEP lessons enable students to learn the leadership skills necessary to manage their IEP meeting and publicly disclose their interests, skills, limits, and goals identified through the Choosing Goals lessons. Rather than be passive participants at their IEP meetings, students learn to lead their meeting to the greatest extent of their ability. These lessons teach students 11 steps for leading their own planning meeting, including introducing attendees, reviewing goals, and stating goals for next year. The Taking Action materials enable students to learn how to break their long-range goals into specific goals that can be accomplished in a week. Students learn how they will attain their goals by deciding (1) a standard for goal performance, (2) a means to get performance feedback, (3) what motivates

240

MICHAEL L. WEHMEYER

them to do it, (4) the strategies they will use, (5) needed supports, and (6) schedules. There have been four studies examining the efficacy of the ChoiceMaker materials (Allen, Smith, Test, Flowers, & Wood, 2001; Cross, Cooke, Wood, & Test, 1999; Snyder, 2002; Snyder & Shapiro, 1997) documenting positive effects on student self-determination, skills in goal setting and leadership, and student involvement in educational planning. Whose Future is it Anyway (WFA)? A Student-Directed Transition Planning Program4 Whose Future is it Anyway? (Wehmeyer et al., 2004) consists of 36 sessions introducing students to the concept of transition and transition planning, and enabling students to self-direct instruction related to (a) self- and disability-awareness, (b) making decisions about transition-related outcomes, (c) identifying and securing community resources to support transition services, (d) writing and evaluating transition goals and objectives, (e) communicating effectively in small groups, and (f) developing skills to become an effective team member, leader, or self-advocate. The materials are student-directed in that they are written for students as end users. The level of support needed by students to complete activities varies a great deal. Some students with difficulty reading or writing need one-on-one support to progress through the materials, while others can complete the process independently. The materials make every effort to ensure that students retain this control while at the same time receiving the support they need to succeed. Students are encouraged to work on one session per week during the weeks between their previous transition planning meeting and the next scheduled meeting. The final two sessions review the previous sessions and provide a refresher for students as they head into their planning meeting. These materials have been field tested and validated for use with students with cognitive disabilities (Wehmeyer & Lawrence, 1995, 2004) and shown to impact student self-determination, self-efficacy for educational planning, and student involvement. Recently, Wehmeyer, Palmer, Lee, WilliamsDiehm, and Shogren (in press) conducted a randomized-trial evaluation of intervention with the WFA, finding a causal link between intervention with the WFA and greater self-determination and transition planning knowledge. Next S.T.E.P.: Student Transition and Educational Planning A third student-directed transition-planning program is the Next S.T.E.P. curriculum (Halpern, et al., 1997). The curriculum uses video and print materials developed for specific audiences (students, teachers, family

Self-Determination

241

members) to help students become motivated to engage in transition planning, self-evaluate transition needs, identify and select transition goals and activities, assume responsibility for conducting their own transition planning meeting, and monitor the implementation of their transition plans. The curriculum consists of 16 lessons, clustered into four instructional units, designed to be delivered in a 50-minute class period. These lessons include teacher and student materials, videos, guidelines for involving parents and family members, and a process for tracking student progress. Unit 1 (Getting Started) introduces and overviews transition planning, intended to enable students to understand the transition planning process and to motivate them to participate. Unit 2 (Self-Exploration and Self-Evaluation) includes six lessons that focus on student self-evaluation. Students work through activities that identify unique interests, strengths, and weaknesses in various adult-outcome oriented areas. At the end of this unit, students complete the student form of the Transition Skills Inventory, a 72-item rating instrument assessing how well the student is doing in four transition areas: (1) personal life, (2) jobs, (3) education and training, and (4) living on one’s own. Unit 3 (Developing Goals and Activities) includes five lessons regarding transition goal identification in the four areas comprising the Transition Skills Inventory. Students identify their hopes and dreams, then select from a range of potential goals in each area, narrowing the total set of transition goals to four or five goals that they prefer. In addition, students choose activities that will help them pursue the goals they have selected. Unit 4 (Putting a Plan into Place) includes three lessons preparing students for their transition-planning meeting. The lessons emphasize the implementation of their plan and work with students to ensure that they monitor their progress and, if necessary, make adjustments. Zhang (2001) examined the efficacy of the Next S.T.E.P. materials, and found implementation significantly impacted student self-determination. The Self-Advocacy Strategy for Education and Transition Planning Van Reusen, Bos, Schumaker, and Deshler (2002) developed a procedure that stresses the importance of self-advocacy to enhance student motivation and that is ‘‘designed to enable students to systematically gain a sense of control and influence over their own learning and development’’ (p. 1). Students progress through a series of lesson plans focusing on seven instructional stages. Stage 1, titled Orient and Make Commitments, broadly introduces education and transition planning meetings, the program itself, and how participation can increase student power and control in this process. Stage 2, titled Describe, defines and provides detailed information

242

MICHAEL L. WEHMEYER

about transition and education meetings and advantages students’ experience if they participate. In this stage the ‘‘I PLAN’’ steps of student participation are introduced. These steps provide a simple algorithm that students can use to chart their participation in planning meetings. In Stage 3, Model and Prepare, the teacher models the I PLAN steps so students can see the process in action. Students complete an Inventory, Step 1 in the I PLAN process, resulting in information they can use at their conference. Stage 4 is Verbal Practice, during which students are asked questions to make sure they know what to do during each step of the I PLAN strategy, and then verbally rehearse each of the steps. In Stage 5, Group Practice and Feedback, once students have demonstrated mastery of the steps in I PLAN, they participate in a simulated group conference. The student receives feedback from the teacher and other students and the group generates suggestions on where the student might improve. The simulated conference is audio- or videotaped for future reference. Stage 6, Individual Practice and Feedback, allows the student to meet independently with the teacher for practice, feedback and, eventually, mastery. The audio- or videotape from the previous stage is reviewed and students provide a self-evaluation of their performance. The student and instructor work together to improve areas of self-identified need and engage in another simulated conference that is also audio- or videotaped and used to document improvement and reevaluate performance. Stage 7, Generalization, is intended to generalize the I PLAN strategy to actual conferences. This stage has three phases: (a) preparing for and conducting the planning conference, (b) preparing for other uses of the strategy, and (c) preparing for subsequent conferences. Student-Led IEPs: A Guide for Student Involvement McGahee, Mason, Wallace, and Jones (2001) developed a guide to studentled IEPs (available online at http://www.nichcy.org/stuguid.asp) that introduces students to the IEP process, the purpose of an IEP, and suggestions for writing an IEP. Mason, McGahee-Kovac, Johnson, and Stillerman (2002) showed that students who used this process knew more about their IEP and showed enhanced self-confidence and self-advocacy.

SUMMARY Promoting self-determination and student involvement in educational planning has become best practice in the education of students with

243

Self-Determination

disabilities. Research confirms that students with disabilities who leave school as self-determined young people achieve more positive adult outcomes. Moreover, promoting student self-determination provides, as it were, a gateway to the general curriculum for students with disabilities, and can result in enhanced academic achievement. This chapter overviewed the methods, materials, and strategies to achieve the outcome that students with and without disabilities can become more self-determined. If educators are to achieve the outcomes envisioned by the transition mandates in IDEA, they will need to ensure that students with disabilities are provided sufficient opportunities to learn these skills and strategies, and to use them to play a meaningful role in their educational program, from planning to implementation.

NOTES 1. Available online at http://www.beachcenter.org/education_and_training/selfdetermination.aspx or http://www.ou.edu/content/education/centers-and-partnerships/zarrow/self-determination-assessment-tools.html. 2. Available online http://www.ou.edu/content/education/centers-and-partnerships/zarrow/self-determination-assessment-tools.html. 3. Information on ordering the Self-Determination Assessment Battery can be obtained from the Wayne State University Center for Self-Determination and Transition Business Office at 313/577-1638 or [email protected]. 4. Available online at http://www.ou.edu/content/education/centers-andpartnerships/zarrow/self-determination-education-materials/whos-future-is-it-anyway.html.

REFERENCES Agran, M., Blanchard, C., & Wehmeyer, M. L. (2000). Promoting transition goals and selfdetermination through student-directed learning: The Self-Determined Learning Model of Instruction. Education and Training in Mental Retardation and Developmental Disabilities, 35, 351–364. Agran, M., Cavin, M., Wehmeyer, M. L., & Palmer, S. (2006). Participation of students with severe disabilities in the general curriculum: The effects of the Self-Determined Learning Model of Instruction. Research and Practice for Persons with Severe Disabilities, 31, 230–241. Agran, M., King-Sears, M., Wehmeyer, M. L., & Copeland, S. R. (2003). Teachers’ guides to inclusive practices: Student-directed learning strategies. Baltimore, MD: Paul H. Brookes. Agran, M., Wehmeyer, M. L., Cavin, M., & Palmer, S. (2008). Promoting student active classroom participation skills through instruction to promote self-regulated learning and self-determination. Career Development for Exceptional Individuals, 31, 106–114.

244

MICHAEL L. WEHMEYER

Algozzine, B., Browder, D., Karvonen, M., Test, D. W., & Wood, W. M. (2001). Effects of interventions to promote self-determination for individuals with disabilities. Review of Educational Research, 71, 219–277. Allen, S. K., Smith, A. C., Test, D. W., Flowers, C., & Wood, W. M. (2001). The effects of ‘‘Self-Directed IEP’’ on student participation in IEP meetings. Career Development for Exceptional Individuals, 4, 107–120. Angyal, A. (1941). Foundations for a science of personality. Cambridge, MA: Harvard University Press. Bambara, L. M., & Ager, C. (1992). Using self-scheduling to promote self-directed leisure activity in home and community settings. Journal of the Association for Persons with Severe Handicaps, 17(2), 67–76. Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84(2), 191–215. Baron, J., & Brown, R. V. (Eds). (1991). Teaching decision making to adolescents. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. Bauminger, N. (2002). The facilitation of social-emotional understanding and social interaction in high-functioning children with autism: Intervention outcomes. Journal of Autism and Developmental Disorders, 32(4), 283–298. Bauminger, N. (2007a). Brief report: Individual social-multi-modal Intervention for HFASD. Journal of Autism and Developmental Disorders, 37, 1593–1604. Bauminger, N. (2007b). Brief report. Group social-multi-modal intervention for HFASD. Journal of Autism and Developmental Disorders, 37, 1605–1615. Bernard-Opitz, V., Sriram, N., & Nakhoda-Sapuan, S. (2001). Enhancing social problem solving in children with autism and normal children through computerassisted instruction. Journal of Autism and Developmental Disorders, 31(4), 377–398. Brown, F., Appel, C., Corsi, L., & Wenig, B. (1993). Choice diversity for people with severe disabilities. Education and Training in Mental Retardation, 28(4), 318–326. Carter, E. W., Lane, K. L., Pierson, M. R., & Glaeser, B. (2006). Self-determination skills and opportunities of transition-age youth with emotional disturbance and learning disabilities. Exceptional Children, 72(3), 333–346. Carter, E. W., Lane, K. L., Pierson, M. R., & Stang, K. K. (2008). Promoting selfdeterminatoin for transition-age youth: Views of high school general and special educators. Exceptional Children, 75(1), 55–70. Clark, G. M. (1996). Transition planning assessment for secondary-level students with learning disabilities. In: J. R. Patton & G. Blalock (Eds), Transition and students with learning disabilities: Facilitating the movement from school to adult life (pp. 131–156). Austin, TX: ProEd. Cobb, B., Lehmann, J., Newman-Gonchar, R., & Morgen, A. (2009). Self-determination for students with disabilities: A narrative metasynthesis. Career Development for Exceptional Individuals, 32(2), 108–114. Crone, E. A., Vendel, I., & van der Molen, M. W. (2003). Decision-making in disinhibited adolescents and adults: Insensitivity to future consequences or driven by immediate reward? Personality and Individual Differences, 35(7), 1625–1641. Cross, T., Cooke, N. L., Wood, W. M., & Test, D. W. (1999). Comparison of the effects of MAPS and ChoiceMaker on students’ self-determination skills. Education and Training in Mental Retardation and Developmental Disabilities, 34, 499–510.

Self-Determination

245

Dunlap, G., Kern-Dunlap, L., Clarke, S., & Robbins, F. R. (1994). Some characteristics of nonaversive intervention for severe behavior problems. In: E. Schopler & G. B. Mesibov (Eds), Behavioral issues in autism (pp. 227–245). New York: Plenum. D’Zurilla, T. J., & Goldfried, M. R. (1971). Problem solving and behavior modification. Journal of Abnormal Psychology, 78, 107–126. D’Zurilla, T. J., & Nezu, A. (1980). A study of the generation-of-alternatives process in social problem solving. Cognitive Therapy and Research, 4(1), 67–72. Field, S., & Hoffman, A. (1994). Development of a model for self-determination. Career Development for Exceptional Individuals, 17, 159–169. Field, S., & Hoffman, A. (2005). Steps to self-determination (2nd ed.). Austin, TX: ProEd. Field, S., Hoffman, A., & Sawilowsky, S. (2004). Self-Determination Assessment Battery. Detroit, MI: Wayne State University. Field, S., Sarver, M. D., & Shaw, S. F. (2003). Self-determination: A key to success in postsecondary education for studnets with learning disabilities. Remedial and Special Education, 24(6), 339–349. Fowler, C. H., Konrad, M., Walker, A. R., Test, D. W., & Wood, W. M. (2007). Selfdeterminatoin interventions’ effects on the academic perfromance of students with developmental disabilities. Education and Training in Developmental Disabilities, 42(3), 270–285. Furby, L., & Beyth Marom, R. (1992). Risk taking in adolescence: A decision-making perspective. Developmental Review, 12(1), 1–44. Halpern, A. S., Herr, C. M., Wolf, N. K., Doren, B., Johnson, M. D., & Lawson, J. D. (1997). Next S.T.E.P.: Student transition and educational planning. Austin, TX: Pro-Ed Inc. Hamilton, A. F. C. (2009). Research review: Goals, intentions and mental states: Challenges for theories of autism. The Journal of Child Psychology and Psychiatry, 50(8), 881–892. Joyce, B., & Weil, M. (1980). Models of teaching (2nd ed.). Englewood Cliffs, NJ: Prentice-Hall, Inc. Konrad, M., Fowler, C. H., Walker, A. R., Test, D. W., & Wood, W. M. (2007). Effects of selfdetermination interventions on the academic skills of students with learning disabilities. Learning Disabilities Quarterly, 30(2), 89–113. Lachapelle, Y., Wehmeyer, M. L., Haelewyck, M. C., Courbois, Y., Keith, K. D., Schalock, R., Verdugo, M. A., & Walsh, P. N. (2005). The relationship between quality of life and selfdetermination: An international study. Journal of Intellectual Disability Research, 49, 740–744. Latham, G. P., & Locke, E. A. (1991). Self-regulation through goal setting. Organizational Behavior and Human Decision Processes, 50(2), 212–247. Lee, S. H., Wehmeyer, M. L., Palmer, S. B., Soukup, J. H., & Little, T. D. (2008). Selfdetermination and access to the general education curriculum. The Journal of Special Education, 42, 91–107. Lee, S. H., Wehmeyer, M. L., Soukup, J. H., & Palmer, S. B. (2010). Impact of curriculum modifications on access to the general education curriculum for students with disabilities. Exceptional Children, 76(2), 213–233. Locke, E. A., & Latham, G. P. (1990). A theory of goal setting and task performance. Upper Saddle River, NJ: Prentice-Hall, Inc. Martin, J. E., & Marshall, L. H. (1995). ChoiceMaker: A comprehensive self-determination transition program. Intervention in School and Clinic, 30, 147–156.

246

MICHAEL L. WEHMEYER

Martin, J. E., Marshall, L. H., & Sale, P. (2004). A 3-year study of middle, junior high, and high school IEP meetings. Exceptional Children, 70, 285–297. Martin, J. E., Van Dycke, J. L., Christensen, W. R., Greene, B. A., Gardner, J. E., & Lovett, D. L. (2006). Increasing student participation in IEP meetings: Establishing the self-directed IEP as an evidenced-based practice. Exceptional Children, 72(3), 299–316. Martorell, A., Gutierrez-Recacha, P., Perda, A., & Ayuso-Mateos, J. L. (2008). Identification of personal factors that determine work outcome for adults with intellectual disability. Journal of Intellectual Disability Research, 52(12), 1091–1101. Mason, C., Field, S., & Swailowsky, S. (2004). Implementation of self-determination activities and student participation in IEPs. Exceptional Children, 70, 441–451. Mason, C., McGahee-Kovac, M., Johnson, L., & Stillerman, S. (2002). Implementing studentled IEPs: Student participation and student and teacher reactions. Career Development for Exceptional Individuals, 25, 171–192. McGahee, M., Mason, C., Wallace, T., & Jones, B. (2001). Student-led IEPs: A guide for student involvement. Arlington, VA: Council for Exceptional Children. McGlashing-Johnson, J., Agran, M., Sitlington, P., Cavin, M., & Wehmeyer, M. L. (2003). Enhancing the job performance of youth with moderate to severe cognitive disabilities using the Self-Determined Learning Model of Instruction. Research and Practice for Persons with Severe Disabilities, 28, 194–204. Mithaug, D. (1998). Your right, my obligation?. Journal of the Association for Persons with Severe Disabilities, 23, 41–43. Mithaug, D. E., Mithaug, D., Agran, M., Martin, J., & Wehmeyer, M. L. (Eds). (2003). Selfdetermined learning theory: Construction, verification, and evaluation. Mahwah, NJ: Lawrence Erlbaum Associates. Mithaug, D., Wehmeyer, M. L., Agran, M., Martin, J., & Palmer, S. (1998). The selfdetermined learning model of instruction: Engaging students to solve their learning problems. In: M. L. Wehmeyer & D. J. Sands (Eds), Making it happen: Student involvement in educational planning, decision-making and instruction (pp. 299–328). Baltimore: Paul H. Brookes. Nota, L., Ferrrari, L., Soresi, S., & Wehmeyer, M. L. (2007). Self-determination, social abilities, and the quality of life of people with intellectual disabilities. Journal of Intellectual Disability Research, 51, 850–865. Palmer, S., & Wehmeyer, M. L. (2003). Promoting self-determination in early elementary school: Teaching self-regulated problem-solving and goal setting skills. Remedial and Special Education, 24, 115–126. Palmer, S. B., Wehmeyer, M. L., Gipson, K., & Agran, M. (2004). Promoting access to the general curriculum by teaching self-determination skills. Exceptional Children, 70, 427–439. Pierson, M. R., Carter, E. W., Lane, K. L., & Glaeser, B. C. (2008). Factors influencing the selfdetermination of transition-age youth with high-incidence disabilities. Career Development for Exceptional Individuals, 31(2), 115–125. Rappaport, J. (1981). In praise of a paradox: A social policy of empowerment over prevention. American Journal of Community Psychology, 9, 1–25. Rotter, J. B. (1966). Generalized expectancies for internal versus external control of reinforcement. Psychological Monographs, 80(1), 244–248.

Self-Determination

247

Ruble, L. A., & Scott, M. M. (2002). Executive functions and the natural habitat behaviors of children with autism. Autism, 6(4), 365–381. Ruef, M. B., & Turnbull, A. P. (2002). The perspectives of individuals with cognitive disabilities and/or autism on their lives and their problem behavior. Research and Practice for Persons with Severe Disabilities, 27(2), 125–140. Shogren, K., Faggella-Luby, M., Bae, S. J., & Wehmeyer, M. L. (2004). The effect of choicemaking as an intervention for problem behavior: A meta-analysis. Journal of Positive Behavior Interventions, 6, 228–237. Shogren, K. A., Lopez, S. J., Wehmeyer, M. L., Little, T. D., & Pressgrove, C. L. (2006). The role of positive psychology constructs in predicting life satisfaction in adolescents with and without cognitive disabilities: An exploratory study. The Journal of Positive Psychology, 1, 37–52. Shogren, K. A., Wehmeyer, M. L., Palmer, S. B., Soukup, J. H., Little, T. D., Garner, N., & Lawrence, M. (2007). Examining individual and ecological predictors of the selfdetermination of students with disabilities. Exceptional Children, 73, 488–509. Shogren, K. A., Wehmeyer, M. L., Palmer, S. B., Soukup, J. H., Little, T., Garner, N., & Lawrence, M. (2008). Understanding the construct of self-determination: Examining the relationship between The Arc’s Self-Determination Scale and the American Institute for Research Self-Determination Scale. Assessment for Effective Instruction, 33, 94–107. Snyder, E. P. (2002). Teaching students with combined behavioral disorders and mental retardation to lead their own IEP meetings. Behavioral Disorders, 27, 340–357. Snyder, E. P., & Shapiro, E. S. (1997). Teaching students with emotional/behavioral disorders the skills to participate in the development of their own IEPs. Behavioral Disorders, 22, 246–259. Sowers, J., & Powers, L. (1995). Enhancing the participation and independence of students with severe physical and multiple disabilities in performing community activities. Mental Retardation, 33, 209–220. Stancliffe, R. J., Abery, B. H., & Smith, J. (2000). Personal control and the ecology of community living settings: Beyond living-unit size and type. Mental Retardation, 105, 431–454. Taylor, L., Adelman, H. S., & Kaser Boyd, N. (1983). Perspectives of children regarding their participation in psychoeducational decisions. Professional Psychology: Research and Practice, 14(6), 882–894. Taylor, L., Adelman, H. S., & Kaser Boyd, N. (1985). Minors’ attitudes and competence toward participation in psychoeducational decisions. Professional Psychology: Research and Practice, 16(2), 226–235. Test, D. W., Karvonen, M., Wood, W. M., Browder, D., & Algozzine, B. (2000). Choosing a self-determination curriculum: Plan for the future. Teaching Exceptional Children, 33, 48–54. Test, D. W., Mason, C., Hughes, C., Konrad, M., Neale, M., & Wood, W. M. (2004). Student involvement in individualized education program meetings. Exceptional Children, 70, 391–412. Thoma, C. A., Pannozzo, G. M., Fritton, S. C., & Bartholomew, C. C. (2008). A qualitative study of preservice techers’ understanding of self-determination for studetns with significant disabiltiies. Career Development for Exceptional Individuals, 31(2), 94–105. Van Reusen, A. K., Bos, C. S., Schumaker, J. B., & Deshler, D. D. (2002). The self-advocacy strategy for education and transition planning. Lawrence, KS: Edge Enterprises, Inc.

248

MICHAEL L. WEHMEYER

Wehmeyer, M. L. (1994). Employment status and perceptions of control of adults with cognitive and developmental disabilities. Research in Developmental Disabilities, 15, 119–131. Wehmeyer, M. L. (1996). A self-report measure of self-determination for adolescents with cognitive disabilities. Education and Training in Mental Retardation and Developmental Disabilities, 31, 282–293. Wehmeyer, M. L. (2005). Self-determination and individuals with severe disabilities: Reexamining meanings and misinterpretations. Research and Practice for Persons with Severe Disabilities, 30, 113–120. Wehmeyer, M. L., Abery, B., Mithaug, D. E., & Stancliffe, R. J. (2003). Theory in selfdetermination: Foundations for educational practice. Springfield, IL: Charles C. Thomas Publisher Ltd. Wehmeyer, M. L., Agran, M., & Hughes, C. (2000). A national survey of teachers’ promotion of self-determination and student-directed learning. Journal of Special Education, 34, 58–68. Wehmeyer, M. L., Agran, M., Hughes, C., Martin, J., Mithaug, D. E., & Palmer, S. (2007). Promoting self-determination in students with intellectual and developmental disabilities. New York: Guilford Press. Wehmeyer, M. L., & Bolding, N. (1999). Self-determination across living and working environments: A matched-samples study of adults with mental retardation. Mental Retardation, 37, 353–363. Wehmeyer, M. L., & Bolding, N. (2001). Enhanced self-determination of adults with intellectual disabilities as an outcome of moving to community-based work or living environments. Journal of Intellectual Disabilities Research, 45, 371–383. Wehmeyer, M. L., & Field, S. (2007). Self-determination: Instructional and assessment strategies. Thousand Oaks, CA: Corwin Press. Wehmeyer, M. L., Field, S., Doren, B., Jones, B., & Mason, C. (2004). Self-determination and student involvement in standards-based reform. Exceptional Children, 70, 413–425. Wehmeyer, M. L., & Garner, N. W. (2003). The impact of personal characteristics of people with intellectual and developmental disability on self-determination and autonomous functioning. Journal of Applied Research in Intellectual Disabilities, 16, 255–265. Wehmeyer, M. L., & Kelchner, K. (1995). The Arc’s Self-Determination Scale. Silver Springs, MD: The Arc of the United States. Wehmeyer, M. L., & Lawrence, M. (1995). Whose future is it anyway? Promoting student involvement in transition planning. Career Development for Exceptional Individuals, 18, 69–83. Wehmeyer, M. L., & Lawrence, M. (2004). A national replication of a student-directed transition planning process: Impact on student knowledge of and perceptions about transition planning. Lawrence, KS: Beach Center on Disability. Wehmeyer, M., Lawrence, M., Kelchner, K., Palmer, S., Garner, N., & Soukup, J. (2004). Whose future is it anyway? A student-directed transition planning process (2nd ed.). Lawrence, KS: Beach Center on Disability. Wehmeyer, M. L., & Little, T. D. (2009). Self-determination. In: S. Lopez (Ed.), The encyclopedia of positive psychology (Vol. 2, pp. 868–874). Boston, MA: Blackwell Publishing. Wehmeyer, M. L., & Palmer, S. B. (2003). Adult outcomes from students with cognitive disabilities three years after high school: The impact of self-determination. Education and Training in Developmental Disabilities, 38, 131–144.

Self-Determination

249

Wehmeyer, M. L., Palmer, S. B., Agran, M., Mithaug, D. E., & Martin, J. E. (2000). Promoting causal agency: The Self-Determined Learning Model of Instruction. Exceptional Children, 66(4), 439–453. Wehmeyer, M. L., Palmer, S. B., Lee, Y., Williams-Diehm, K., & Shogren, K. A. (in press). A randomized-trial evaluation of the effect of Whose Future is it Anyway? on selfdetermination. Career Development for Exceptional Individuals. Wehmeyer, M.L., Palmer, S.B., Shogren, K.A., Williams-Diehm, K., Obremski, S., ClavennaDeanne, B., Zuna, N. (in press). A randomized trial study of the Self-Determined Learning Model of Instruction. submitted for publication. Wehmeyer, M. L., & Schwartz, M. (1997). Self-determination and positive adult outcomes: A follow up study of youth with mental retardation or learning disabilities. Exceptional Children, 63, 245–255. Wehmeyer, M. L., & Schwartz, M. (1998). The relationship between self-determination and quality of life for adults with mental retardation. Education and Training in Mental Retardation and Developmental Disabilities, 33, 3–12. Wehmeyer, M. L., & Shogren, K. (2008). Self-determination and learners with autism spectrum disorders. In: R. Simpson & B. Myles (Eds), Educating children and youth with autism: Strategies for effective practice (2nd ed., pp. 433–476). Austin, TX: Pro-Ed, Inc. Whitman, T. L. (1990). Self-regulation and mental retardation. American Journal on Mental Retardation, 94(4), 347–362. Wolman, J., Campeau, P., Dubois, P., Mithaug, D., & Stolarski, V. (1994). AIR selfdetermination scale and user guide. Palo Alto, CA: American Institute for Research. Zhang, D. (2001). The effect of Next S.T.E.P. instruction on the self-determination skills of high school students with learning disabilities. Career Development for Exceptional Individuals, 24, 121–132. Zimmerman, M. A., & Rappaport, J. (1988). Citizen participation, perceived control, and psychological empowerment. American Journal of Community Psychology, 16, 725–750.

SUCCESS AND FAILURE WITH TIER-2 SRSD FOR TIMED-WRITING TESTS AMONG SECONDTHROUGH FIFTH-GRADE STUDENTS WITH WRITING AND BEHAVIORAL DIFFICULTIES: IMPLICATIONS FOR EVIDENCE-BASED PRACTICE Karin Sandmel, Kristen D. Wilson, Karen R. Harris, Kathleen Lynne Lane, Steve Graham, Wendy P. Oakes, Sharlene A. Kiuhara and Trish D. Steinbrecher ABSTRACT Relatively limited attention has been paid to the academic needs of students with emotional and behavioral difficulties. Effective interventions are needed to support these students academically, behaviorally, and Assessment and Intervention Advances in Learning and Behavioral Disabilities, Volume 24, 251–293 Copyright r 2011 by Emerald Group Publishing Limited All rights of reproduction in any form reserved ISSN: 0735-004X/doi:10.1108/S0735-004X(2011)0000024012

251

252

KARIN SANDMEL ET AL.

socially. The purpose of the concurrent studies reported here was to investigate the effectiveness of academic support in writing for fourth- and fifth-grade students (six boys, two girls) and second- and third-grade students (seven boys, one girl) with writing and behavioral difficulties. The Self-Regulated Strategy Development (SRSD) approach was implemented as a tier-2 intervention within a comprehensive, integrated three-tiered model of prevention including academic-, behavioral-, and social-skills components. Students learned an on-demand writing strategy for their state writing-competency test. Dependent measures included number of story writing elements, total number of words written, and writing quality. Fourth- and fifth-grade students who completed the intervention improved in total number of story elements. There were mixed results for the total number of words written and writing quality. Second- and third-grade students did not improve their total number of story elements, total words written, or writing quality. Students in both studies scored the intervention favorably, while there were mixed reactions from teachers. Findings, limitations, and suggestions for future research are discussed. Implications for the construct of evidence-based practice (EBP) are also explored, including concerns regarding frequent assessment of writing throughout intervention regardless of stage of instruction in the SRSD model.

Too many students with or at risk for emotional and behavioral disorders (EBD) are not succeeding academically in schools. Interventions for students with or at risk for EBD typically support students behaviorally or socially, not academically. In a meta-analysis, Reid, Gonzalez, Nordness, Trout, and Epstein (2004) reported that the academic performance of students with EBD when compared to students without disabilities was significantly lower in reading, math, spelling, and written expression. Nelson, Benner, Lane, and Smith (2004) conducted a cross-sectional study of 155 students with EBD and assessed students using the Woodcock Johnson – III (WJ-III; Woodcock, McGrew, & Mather, 2001). Based on the standard scores, the authors reported large academic-achievement gaps across all grade levels (K-12) for students with EBD across all content areas (e.g., reading, math, and writing). Academic interventions for students with EBD have not been sufficiently examined particularly in the area of writing (Lane, 2004). In this chapter we present two studies conducted concurrently to investigate the effectiveness of the Self-Regulated Strategy Development

Timed-Writing and SRSD Strategies

253

(SRSD) approach with a new on-demand writing strategy developed to address a state writing-competency test. We worked with students identified as having writing and behavior difficulties through data collected as part of a primary prevention program. The schools we worked with implemented a comprehensive, integrated, three-tiered model of prevention including academic (Response-to-Intervention; RtI), behavioral (Schoolwide Positive Behavior Support; SWPBS), and social skills (Character Under Construction; Forrest, 2000) components. The two studies extend the research on SRSD in two ways. First, in addition to working with second-grade students as in previous studies with students with or at-risk for EBD, we worked with fourth- and fifth-grade students with writing and behavioral difficulties. Second, students were taught a new writing strategy aligned with the state writing test. Passing such tests is critical for many students with EBD or other behavioral challenges, yet we were unable to locate any experimental research regarding intervention for writing-competency tests for any students (cf. Graham, Harris, & Hebert, in press). Finally, in counter point to the majority of the research on SRSD (Harris, Graham, Brindle, & Sandmel, 2009), SRSD instruction did not result in evidence of a functional relationship or meaningful effects on the writing of our students in second and third grade, although the instruction was promising in fourth and fifth grade. We discuss implications for the construct of evidence-based practice (EBP). In addition, in these two studies we collected writing probes during intervention far more frequently than in most SRSD research, in response to concerns raised regarding judgments of functional relationships when data is not collected with high frequency during intervention when investigating tiers-2 and -3 interventions (cf. Lieberman, Yoder, Reichow, & Wolery, 2010). Implications of such frequent assessment during intervention when studying complex academic learning are also discussed. First, we turn to research regarding the threetiered model of prevention and SRSD.

THREE-TIERED MODEL OF PREVENTION A comprehensive, integrated three-tiered model of prevention is a datadriven model designed to systematically identify and support students academically, behaviorally, and socially (Lane, Menzies, & Kalberg, in press). In this model, three tiers of prevention are developed: (a) primary prevention (tier 1), (b) secondary prevention (tier 2), and (c) tertiary prevention (tier 3). In tier 1, a schoolwide plan is designed and implemented

254


with a goal of preventing harm from occurring. All students and staff participate in the schoolwide plan with common expectations applied in all settings (e.g., hallway, cafeteria; Lane, Kalberg, & Menzies, 2009), and the use of EBPs in the general education classrooms (Bradley, Danielson, & Doolittle, 2007). All students are screened periodically in core academic areas (e.g., reading, mathematics) and are monitored on their academic progress (e.g., curriculum-based measures, grade-point average, course failures, high school credit attainment). Schoolwide expectations (e.g., be respectful, be responsible) are developed, explicitly taught to students in the beginning of the year, and revisited throughout the school year to support behavior. Students are provided opportunities to practice the behavioral expectations and are reinforced for demonstrating expected behaviors. Universal behavior screenings are conducted to identify students with or at risk for behavior difficulties, similar to the manner in which reading performance is assessed using commercially available academic screening tools (e.g., AIMSweb or Dynamic Indicators of Basic Early Literacy Skills). Finally, students receive instruction in social skills (e.g., Elliott & Gresham, 2008) or character education (Positive Action, 2008), with opportunities to practice and receive reinforcement to support social development. Tier 2 is designed to reverse harm by providing supplemental, evidencebased academic, behavior, and social support for students not responsive to tier-1 instruction (e.g., Lane et al., in press). Students’ academic and behavioral performance continues to be monitored and if students’ performance improves, students return to tier-1 instruction. Tier-2 interventions may simultaneously address students’ academic, behavior, or social difficulties (e.g., Oakes, Mathur, & Lane, 2010). Students not responsive to tier-2 supports may need more intensive, individualized academic, behavior, and social supports through additional, specialized tier-3 assistance. The use of EBPs is a key component to supporting students in the comprehensive, integrated three-tiered model of prevention. In the next section, research on the effectiveness of self-regulation and writing strategies, specifically with students identified with or at risk for EBD, is discussed.

SELF-REGULATED STRATEGY DEVELOPMENT AND STUDENTS WITH EBD Self-Regulated Strategy Development (SRSD) is an instructional approach for students with and without disabilities supported by 25 years of research


255

(Graham & Harris, 2003; Harris & Graham, 2008; Harris et al., 2009). SRSD addresses students’ cognitive, behavioral, and affective characteristics (Harris & Graham, 1996; Harris, Graham, Mason, & Friedlander, 2008). With SRSD instruction, students first learn about powerful academic strategies (e.g., writing strategies) and learn when and why they can be used. Next, students are introduced to self-regulation strategies (e.g., goal setting, self-instruction, self-monitoring, self-reinforcement). Students are supported as they learn to apply the academic strategies in tandem with the selfregulation strategies until they reach mastery. Self-efficacy and motivation to write are also targeted by components of SRSD instruction. There is promising evidence that the SRSD approach may be a suitable academic intervention for students with or at risk for EBD (Harris & Graham, 1999; Adkins, 2005; Mason & Shriner, 2008).

SRSD IN A THREE-TIERED MODEL OF PREVENTION Four studies have been conducted within a three-tiered model of prevention involving students who struggle with writing and are at risk for EBD (Lane, Harris, Graham, Weisenbach, & Brindle, 2008; Little, Lane, Harris, Graham, Story, Sandmel, 2010; Lane, Graham, Harris, Little, Sandmel, 2010; Lane, Harris, Graham, Driscoll, Sandmel, in press). Lane et al. (2008) conducted a single case design study with second-grade students (N ¼ 6) with writing and behavior difficulties in schools implementing a three-tiered model of prevention. Students learned a story writing strategy through SRSD instruction as a tier-2 intervention. Lane et al. reported students’ story elements, writing quality, and total words written improved from baseline to postinstruction and maintained over time. Little et al. (2010) conducted a single case design study with second-grade students (N ¼ 13) with writing and behavior difficulties in schools with developed three-tiered model of prevention plans. Seven students were identified by systematic school assessments as displaying externalizing (e.g., acting out, lying, verbal aggression) characteristics; six students displayed internalizing (e.g., shy, depressed, anxious) characteristics. Students were taught an opinion writing strategy through the SRSD approach as a tier-2 intervention. Little et al. concluded SRSD instruction had positive effects for students with externalizing or internalizing characteristics. Students increased the number of opinion elements in their essays and maintained

256


these gains over time. There were mixed effects for the writing quality and length of the opinion essays. In a third study, Lane et al. (2010) conducted a single case design study with second-grade students (N ¼ 13) with writing and behavior difficulties. Seven students displayed externalizing characteristics and six students displayed internalizing characteristics. Students were taught a story writing strategy through SRSD instruction. Students wrote a story about a black and white picture and all writing probes were scored for elements, quality, and length. Lane et al. concluded SRSD had a positive impact for improving the total number of story elements, but mixed outcomes for writing quality and the total number of words written. In a fourth study, Lane et al. (in press) conducted a randomized controlled trial to investigate the effectiveness of the SRSD approach and two writing strategies with 44 second-grade students with writing and behavior difficulties. The 21 students in the control group received writing instruction in the general education classroom, a combination of the process writing approach and traditional writing skills. The 23 students in the experimental group received writing instruction in the general education classroom and worked individually with trained research assistants (RAs) to learn two strategies for two genres. Opinion essay writing was taught first, followed by story writing. At posttest, students in the experimental group, when administered the writing prompts individually, significantly differed from students in the control group in opinion elements (po0.0001), length (p ¼ 0.0007), and quality (p ¼ 0.0003). At posttest, students in the experimental group, when administered writing prompts individually, differed significantly from students in the control group on story elements (p ¼ 0.0005) and story quality (p ¼ 0.0002), but not length. Lane et al. (in press) concluded students in the experimental group benefited from learning the self-regulation and writing strategies. Collectively, these studies offer preliminary evidence of the positive impact SRSD and writing strategies, used as a tier-2 intervention, has on improving the writing performance of young students with writing and behavior difficulties. In the previous studies, students improved the total number of story or persuasive writing elements, writing quality, and the length of the compositions. It is necessary to continue to examine the feasibility of the SRSD approach within a three-tiered model of prevention, particularly looking at the grade level of the students, inclusion criteria based on schoolwide data, and alternative writing strategies (e.g., on-demand writing).

257


THE CURRENT STUDIES In the two studies presented here, students were taught an overall planning strategy that helped them consider their audience and respond to the state writing prompt; they used a second strategy to plan and organize their writing. There were two primary research questions: (a) does SRSD instruction in on-demand writing strategies improve the writing performance of second- and third-grade students with writing and behavioral difficulties, and fourth- and fifth-grade students with writing and behavior difficulties and (b) do teachers and students view the SRSD approach to be socially valid?

STUDY 1: FOURTH- AND FIFTH-GRADE STUDENTS Method Participants and Setting The participants in this study were eight fourth- (N ¼ 4) and fifth- (N ¼ 4) grade students (six boys, two girls) who attended two rural inclusive elementary (grades K-5) schools in Tennessee. These schools employed a comprehensive, integrated, three-tiered model of prevention with academic, behavioral, and social-skills components. Students were identified as having writing and behavior difficulties according to data collected as part of regular school practices (description of screening procedures follows). All students were Caucasian and their ages ranged between 9.4 and 11.2 years (see Table 1). The populations of the two elementary schools were 450 and 650 students. Students attending the schools were primarily Caucasian (93–94%), and the student populations included students of Hispanic (2–5%), African American (1–2%), and Asian (1%) decent. The number of students receiving free/reduced lunch ranged from 14% to 40%. Both schools were in good standing under the No Child Left Behind Act. Primary Prevention Plan The participating schools regularly collected data on student performance. Teachers, staff, and parents developed schoolwide behavioral expectations (e.g., readiness, responsibility, respect, cooperation). Students were explicitly taught the behavioral expectations and the appropriate behaviors were demonstrated in all areas of the schools (e.g., hallway, cafeteria, classroom, playground). Students were reinforced using tickets paired with behavior-specific praise for appropriate social, academic, and

258


Table 1.

Characteristics of Fourth- and Fifth-Grade Students with Writing and Behavior Difficulties.

Student

Toby

Adam

Trey

Jess

Cora

Vera

Dan

Vance

1

1

2

2

2

3

3

3

9.8 Male C 114.5 97 111

9.8 Male C 97.1 95 106

11.2 Male C 91.3 96 96

10 Male C 123.2 103 93

9.4 Female C 85.5 94 115

10.1 Female C 105.8 113 112

10.3 Male C 105.8 108 105

9.8 Male C 108.7 102 114

2 5

1 6

1 13

3 7

3 6

2 9

3 6

2 8

SSRS-T (pre) Social skills (SS) Problem behavior (SS) Academic comp (SS)

80 106 70

96 100 82

74 115 80

84 123 82

83 128 79

66 130 85

85 112 84

85 127 84

SSRS-P (pre) Social skills (SS) Problem behavior (SS)

101 86

118 97

96 112

104 120

98 84

64 119

113 86

100 112

SSRS-T (post) Social skills (SS) Problem behavior (SS) Academic comp (SS)

89 106 83

99 93 89

82 110 72

92 123 94

100 103 78

89 106 91

98 108 86

87 137 83

a

a

a

a

a

a

1

1

11

86 112 2

112 84 0

68 126 0

96 89 0

101 100 3

Leg General demographics Age in years Gender Ethnicity IQ WJ-III (pre; SS) WJ-III (post; SS) Screening DWA SRSS

SSRS-P (post) Social skills (SS) Problem behavior (SS) Office discipline referrals

Note: C, Caucasian; IQ, intelligence quotient; Wechsler Intelligence Scale for Children – IV (WISC-IV; Wechsler, 2003); WJ-III, Woodcock Johnson – III Test of Achievement (Woodcock et al., 2001); Writing Sample Subtest; SS, standard score; SRSS, student risk screening scale; SSRS-T, Social Skills Rating System – teacher version (Gresham & Elliott, 1990); SSRS-P, Social Skills Rating System – parent version (Gresham & Elliott, 1990). a Information not available.

behavioral interactions with peers, faculty, staff, and parents. At designated times throughout the school year, students traded their tickets for rewards (e.g., lunch with the principal, homework pass). Teachers completed the Student Risk Screening Scale (SRSS; Drummond, 1994) to identify students with behavior difficulties.


259

Academically, students were assessed regularly in reading, mathematics, and writing. Students with difficulties in these academic areas received tier-2 assistance from a school specialist. Students who continued to experience difficulties in these academic areas were recommended for tier-3 support. The schools implemented the Character Under Construction (Forrest, 2000) curriculum to develop the students’ social skills. Each month, teachers and adults focused on a character trait (e.g., caring, courage) and students were taught behaviors to exhibit these traits in different settings (e.g., school, home). Writing Practices Prior to the beginning of the intervention, cooperating fourth- and fifthgrade teachers completed the Teacher Survey of Classroom Writing Practices (Cutler & Graham, 2008). The coefficient alphas for this measure range from 0.62 to 0.85. Based on survey results, teachers described the general classroom writing practices they implemented as a combination of traditional writing skills (e.g., grammar, punctuation) and the process writing approach (e.g., planning, drafting, and revising). The most frequent activities included: conferencing with students, teaching planning and revising strategies (including planning for personal narratives), using graphic organizers, parts of personal narratives, teaching sentence construction, spelling, grammar, punctuation, and capitalization skills, and practice responding to writing prompts. Less frequent writing activities included: handwriting skills, assigning homework, and dictation. Teachers did not teach students timed-writing strategies and/or self-regulated strategies. In addition to the teacher self-report survey, trained RAs completed two 15-minute observations of teachers’ writing practices during each phase of the study (e.g., baseline, intervention, postinstruction) using The Observation of Classroom Writing Practices measure (Agate, 2005; Graham, Harris, MacArthur, & Fink, 2003). The reliabilities for the process writing activities and skills instruction activities are 0.84 and 0.87, respectively (Agate, 2005; Graham et al., 2003). The activities observed most often included: conferencing with students, instructional goals stated, teaching of text organization and planning skills, reteaching skills, use of mini-lessons, strategies for taking the district-writing assessment, planning a personal narrative, and the parts to include in a personal narrative. The activities observed least included: assessment of student writing, assigned homework, use of dictation, writing centers, timed-writing strategies, and encouraging students to use self-statements. Based on the observations, the most and

260


least frequent activities observed were aligned with teachers’ self-reports of the activities they implemented in the classroom. Participant Selection Fourth- and fifth-grade students with behavior and writing difficulties were identified through the schools’ regular academic- and behavior-datacollection procedures for this study. Students met three inclusion criteria: (a) identified as at high or moderate risk for antisocial behavior as measured by the SRSS (Drummond, 1994), (b) low levels of writing proficiency according to district-writing prompt scores, and (c) not receiving special education services (these students were not included because they were already receiving additional support). Student Risk Screening Scale In the fall, teachers completed the SRSS (Drummond, 1994). The SRSS has seven items where teachers rate students on a 4-point Likert scale (0 ¼ never, 1 ¼ occasionally, 2 ¼ sometimes, 3 ¼ frequently) on different antisocial externalizing behaviors including: (a) stealing, (b) lying, cheating, sneaking, (c) problem behaviors, (d) peer rejection, (e) low academic achievement, (f) negative attitude, and (g) aggressive behavior. The scores were totaled and students who received a score between 9 and 21 (at high risk) or between 4 and 8 (at moderate risk) were considered for the study. The SRSS has been positively correlated with the Aggressive Behavior subscale of the Child Behavior Checklist (r ¼ 0.79; Walker, Ramsey, & Gresham, 2004) and has strong internal consistency with an overall alpha coefficients of 0.83 (Lane, Little, Casey, Lambert, & Wehby, 2009). District-Writing Assessment Fourth- and fifth-grade students completed district-writing prompts in the fall. Students were asked to respond to a Story About Me writing prompt (e.g., Pretend your friend called you and said, ‘‘You aren’t going to believe what I found!’’ You rushed over to your friend’s home. Before you begin to write, think about what your friend found and what happened. Now write a story about what happened when your friend found something.). Teachers and district leaders scored students writing prompts using a 7-point holistic Likert-type scale (0–6). Students must receive a 4 or higher to pass the district-writing assessment. Students who received a 1, 2, or 3 were identified as students with writing difficulties and were considered for participation.


261

Identified Students Once teachers consented to participate, parent/guardian permission forms were sent home to 17 students who met the inclusion criteria. Ten students returned signed permission forms allowing them to participate. When assented, nine students agreed to participate in the study and one student declined. During the study, one student moved during the intervention. Eight students completed the three phases (e.g., baseline, intervention, postinstruction) of the study and we report on their writing performance. Intervention Groups We intended to work with students in groups of three, however, due to scheduling we were not able to do this for all students. In Leg 1, a writing instructor (WI) worked with two fourth-grade students, Toby and Adam (see Table 2 for the number of instructional sessions each student received and the average length of sessions). In Leg 2, a WI worked with three students, Trey, Jess, and Cora. In Leg 3, a WI worked with two fifth-grade students, Vera and Dan and also worked individually with a fourth-grade student, Vance, at a different time due to class schedules. The WIs worked with the students in quiet, available spaces (e.g., conference room, empty classrooms, hallways). Writing Instructor Training Writing instructors, graduate students in education, attended a two-day, 12-hour training. During the training sessions, WIs watched a video, were provided with detailed lesson plans, activities related to the lessons, and support materials. The WIs practiced in partners implementing the lessons until they reached criterion: implementing lessons fluently and including all lesson components. WIs attended 4-hour booster sessions prior to the beginning of a new leg of instruction. Characteristics and Stages of SRSD Five characteristics of SRSD instruction address students’ individual needs (Harris et al., 2009; Sandmel et al., 2009). First, instruction is individualized to provide behavioral and academic supports students need to succeed as they move through the writing process. Second, students agree to commit to learning and applying the writing and self-regulation strategies. Teachers and students work collaboratively through the planning and writing process and students are supported until they apply the writing and self-regulation strategies independently. Third, the goals for writing are explicitly stated, challenging, and achievable. Teachers and students discuss the goals for

262


Table 2. Descriptive Characteristics of Instructional Sessions, Treatment Fidelity, and Social Validity for Study 1. Student

Toby

Phase

No. of Sessions

Baseline Intervention Follow-up

26

Adam Baseline Intervention Follow-up

26

Trey

Jess

Cora

Vera


24


23


21a


13


13

Vance Baseline Intervention Follow-up

15

Dan

Average Session Time

26 min

26 min

29 min

29 min

29 min

29 min

29 min

29 min

TF Teacher %

SD

TF Observer Collected

100.00

0.00

40.00%

100.00

99.00

99.00

99.00

100.00

100.00

99.00

0.00

0.03

0.03

0.03

0.00

0.00

0.02

36.00%

40.91%

39.13%

33.33%

38.46%

38.46%

33.33%

Observer TF %

SD

100.00

0.00

100.00

100.00

100.00

100.00

100.00

100.00

97.92

Social Validity IRP-15

CIRP

71

40

84

39

75

32

81

37

67

37

36

42

32

35

88

24

88

32

80

32

67

28

40

39

62

37

39

42

90

37

90

40

0.00

0.00

0.00

0.00

0.00

0.00

0.71

a

This student was taught with her peers for 16 sessions. She received individual instruction for the last four sessions.

writing (e.g., write a story that is fun to read, fun to write, makes sense, has several parts, captures readers’ attention). Individual goals may be developed to meet students’ writing abilities. Fourth, instruction is criterion-based, rather than time-based. Some students quickly adopt and apply the writing and self-regulation strategies, while some students need additional time before they write independently. Fifth, the stages of instruction can be repeated or skipped based on students’ writing needs and their motivation for writing.


263

Students were taught self-regulation strategies (e.g., goal setting, selfmonitoring, self-instructions, self-reinforcement) and two writing strategies through the SRSD approach (Graham & Harris, 2003). These writing strategies were developed to help students learn to write for their statewriting assessment. Students learned one planning strategy to respond to writing prompts (TACO: T – what is the topic of my writing prompt?; A – audience who will read my Story About Me; C – carefully read the writing prompt; O – organize my notes). Students also learned a planning strategy to organize their writing (TOWER: T – tell who, when, and where; O – order of events, beginning, middle, and end; W – what did the characters feel?; E – exciting million-dollar words; R – review my notes and begin to write). Students learned to apply the writing and self-regulation strategies through six stages of instruction: develop background knowledge, discuss it, model it, memorize it, support it, and independent performance.

Develop Background Knowledge. During the ‘‘develop background knowledge’’ stage, WIs and students discussed the purpose of strategies (tricks). Students were taught that the tricks would help them organize their thoughts, plan their stories, and improve their writing. WIs and students discussed writing prompts (e.g., state writing prompts) and the writing genre (Story About Me). Students were told that a Story About Me may be a true or fictional story, and that in many instances it will be a fictional story where they are the main character. Next, students were introduced to TACO and TOWER. Students learned the purpose of each letter, to write TACO on their writing prompt, and make notes for the parts. Students, through the use of TACO, identified the topic of the writing prompt and the audience. Students were told the audience was a teacher they did not know, but who knows writing. WIs and students discussed the importance of the audience and established goals for writing (e.g., makes sense, has all its parts, fun to read, uses exciting words, fun to write, captures the reader’s attention). Students identified parts present in the writing prompt and circled or underlined the parts and discussed the importance including those parts. Students learned to organize their notes using TOWER. Students were introduced to the TOWER mnemonic and discussed the different story components. Students discussed these parts in stories familiar to them and identified the story elements in model stories. While the stories were read, students were introduced to notes (e.g., enough words to

264


remember my ideas) and WIs modeled taking notes on a graphic organizer as the stories were read. Discuss It. During the ‘‘discuss it’’ stage, WIs and students continued examining models of good writing, taking notes, memorizing the tricks, and discussing writing prompts. WIs and students discussed where to use the tricks (e.g., during the weekly writing prompts). WIs and students examined a sample of poor writing and suggested new ideas to improve the story. Students were asked to make a commitment to learning and applying the tricks to their writing. Model It. During the ‘‘model it’’ stage, WIs modeled the writing process using TACO and TOWER. WIs modeled writing TACO and taking notes on the writing prompt sheet. Second, WIs modeled using the TOWER graphic organizer to make notes. WIs used the parts identified in the writing prompt on the graphic organizer and added more notes. Third, WIs modeled using the graphic organizer to draft a Story About Me. As the writing process was modeled, WIs stated out loud different types of self-statements: problem definition (What do I have to do here?), planning (I can use TACO and TOWER.), self-evaluation (Do I have all of my parts?), self-reinforcement (This is going to be my best story yet!), and coping (This is a boring topic. I can do this if I take my time.). After modeling the writing process, WIs discussed the self-statements and students developed and recorded statements to think of good ideas, for when they worked, and to check their work. Also during the ‘‘model-it’’ stage, students were introduced to graphing (self-monitoring) their stories. Students were presented a tower-graphing sheet: towers with eight windows and coins around the top of the tower. Students were shown how the windows corresponded to a story element (who, when, where, beginning, middle, end, feeling, and exciting milliondollar word). For each part included students colored a window and if more than one feeling or exciting million-dollar word was included, students colored the coins around the top of the tower. WIs told students one of their goals was to include all of their story parts (eight parts) to reach the treasure chest at the top of the tower. Memorize It. During the ‘‘memorize it’’ stage, students memorized the mnemonics (TACO and TOWER), the meanings of the mnemonics, and when to apply the mnemonics. Students were tested at the beginning of the instructional sessions, used flashcards, and played different games.


265

Memorization of the mnemonics promoted automaticity of using the writing strategies.

Support It. During the ‘‘support-it’’ stage, WIs and students worked collaboratively to write stories. WIs led the writing process as students assisted in identifying parts in the writing prompt, suggesting ideas for story parts and sentences to write. As students began to adopt the strategies, they wrote notes and stories themselves. WIs scaffolded students through the planning and composing processes by reminding them about the goals of writing, using self-statements, or including missing story parts. Students worked independently if they easily applied the writing strategies. During this stage, WIs and students looked at a baseline-writing sample and identified parts the students included and excluded. Students graphed their baseline-writing prompt and compared their performance to writing samples written during the intervention. Writing goals were reviewed and students were encouraged to commit to using the writing tricks. A second activity students in Legs 1 and 2 completed was looking at the writing quality-scoring rubric. WIs and students discussed the scoring rubric, what score was needed for passing the writing test (score of 4), what was reasonable for students to achieve (score of 4 or 5), and that students could achieve a passing score if they used the strategies. WIs and students looked at different writing samples (e.g., score of 3, score of 4, score of 6) and discussed the differences between the writing samples. WIs helped students develop personal goals (e.g., adding more feelings or exciting million-dollar words, making sure the story makes sense). Due to an insufficient amount of time, students in Leg 3 did not complete this activity. Second- and third-grade students did not complete this activity. Another activity fourth- and fifth-grade students in Leg 1 completed was discussing test-taking strategies. Students identified different components of the test directions (e.g., write on every line, use prompt sheet to plan). WIs and students discussed strategies for taking the test including how much time should be spent planning, writing, and reviewing the parts. Due to limited time, fourth- and fifth-grade students in Leg 2 and Leg 3 did not complete this activity. Second- and third-grade students did not complete this activity as it was not deemed appropriate for them. Finally, during the ‘‘support-it’’ stage, WIs showed students how to develop their own graphic organizer on the writing prompt and weaned them off the formal graphic organizer. WIs modeled writing TACO on the prompt sheet and TOWER on the back of the prompt sheet. WIs modeled

266


making notes before beginning to write and encouraged students to make notes on the writing prompt. Independent Performance. During the ‘‘independent performance’’ stage, students applied the writing and self-regulation strategies independently and to criterion (e.g., included eight parts). WIs and students reviewed the writing rubric, test-taking strategies, and practiced using the writing strategies. Fourth- and fifth-grade students and second-and third-grade students in Leg 3 did not complete this activity due to school closures because of severe flooding. Instruction ended when students independently wrote stories and met criterion or when time for instruction ran out due to school closings. Treatment Fidelity Treatment fidelity was collected in two forms: WI self-report and observer (see Table 2 for treatment fidelity). WIs were provided a form with the major components of each lesson and reported on the components completed during each session. Treatment fidelity, based on WI self-report, ranged from 99% to 100%. A trained observer collected treatment fidelity using the same form. Observer treatment fidelity was collected between 33.33% and 40.91% of the time. Treatment fidelity, based on observers’ reports, ranged between 97.92% (SD ¼ 0.71) and 100%. Descriptive Measures Five descriptive measures were collected to describe students’ behavioral characteristics and academic performance (see Table 1). WISC-IV. The Wechsler Intelligence Scale for Children, Fourth Edition (WISC-IV; Wechsler, 2003) was administered to students individually prior to the start of the intervention to describe students’ cognitive abilities. The reliability coefficient for the WISC-IV is 0.91. The short form was used and included two subtests: vocabulary and block design. WJ-III. The Woodcock Johnson III (Woodcock et al., 2001) Test of Achievement Writing Sample Subtest was individually administered to students prior to and upon completion of the intervention as a measure of academic achievement. The one-year test–retest correlation range is 0.76– 0.95 for 7 to 11-year-old students.


267

Social Skills Rating System – Teacher Version. The elementary version of the Social Skills Rating System – Teacher (SSRS-T; Gresham & Elliott, 1990) was completed by teachers prior to and upon completion of the intervention. The SSRS-T has three subsections: social skills, problem behavior, and academic competence. Teachers rated students’ social skills (e.g., cooperation, assertion, self-control) on 30 items using a 3-point Likerttype scale (0 ¼ never, 1 ¼ sometimes, 2 ¼ very often) and the importance of these skills (0 ¼ not important, 1 ¼ important, 2 ¼ critical). Teachers rated students problem behaviors (e.g., internalizing behaviors, externalizing behaviors, hyperactivity) on 18 items using a 3-point Likert-type scale (0 ¼ never, 1 ¼ sometimes, 2 ¼ very often). Teachers rated students’ academic competence by comparing the students’ academic performance to the whole class on nine items using a 5-point Likert-type scale (1 ¼ lowest 10%, 5 ¼ highest 10%). The SSRS-T has strong internal consistency (0.82– 0.94). Social Skills Rating System – Parent Version. The elementary version of the Social Skills Rating System – Parent (SSRS-P; Gresham & Elliott, 1990) was completed by parents/guardians prior to and upon completion of the intervention. The SSRS-P has two subsections: social skills and problem behavior. Parents/guardians rated students’ social skills (e.g., cooperation, assertion, self-control) on 38 items using a 3-point Likert scale (0 ¼ never, 1 ¼ sometimes, 2 ¼ very often) and the importance of these skills (0 ¼ not important, 1 ¼ important, 2 ¼ critical). Parents/guardians rated students’ problem behaviors (e.g., internalizing behaviors, externalizing behaviors, hyperactivity) on 17 items using a 3-point Likert scale (0 ¼ never, 1 ¼ sometimes, 2 ¼ very often). The SSRS-P has strong internal consistency (0.70–0.90). Writing Prompt Administration and Management Students were administered writing prompts similar to the state writing prompt for fifth-grade students, at baseline, during intervention (after every three instructional sessions, typically every Friday), and following instruction. The genre of writing was a Story About Me (e.g., fictional, personal narrative). Writing prompts were developed by the authors to match the style and genre of the state writing prompt (e.g., Suppose one day you found a time machine; see Table 3 for the number of writing prompts administered during each phase of instruction). Directions for administering the writing prompts were modeled on the state test directions. Students had 30 minutes to plan and write their story.

Baseline Intervention Postinstruction PND




Adam

Trey

Jess

4 6 3

5 7 3

3 7 3

3 7 3

Number of Writing Prompts

12.4 (2.30) 14.33 (1.03) 13.00 (1.73) 0%

8.67 (1.86) 11.29 (1.38) 11.67 (1.15) 11.11%

4.67 (1.53) 7.71 (4.11) 8.33 (2.08) 33.33%

4.00 (2.00) 9.00 (2.77) 10.33 (2.31) 100%

M (SD)

0.00 (2.66) 0.27 (0.95) 1.5 (1.22)

0.21(1.99) 0.11 (1.49) 1.00 (0.82)

0.50 (2.04) 1.54 (2.66) 1.50 (2.04)

2.00 (0.00) 0.71 (2.52) 2.00 (1.63)

Slope (Syx)

Elements

289.40 (139.14) 221.33 (56.02) 157 (30.61)

131.17 (14.22) 158.86 (44.90) 128 (13.08)

37 (6.08) 60.71 (11.84) 73.33 (14.57)

52 (23.07) 92.71 (23.12) 98.67 (28.94)

M (SD)

33.97 (112.84) 21.49 (35.08) 17.50 (35.52)

1.68 (15.13) 5.18 (47.64) 1.50 (18.37)

5.00 (4.90) 1.36 (12.57) 14.00 (5.72)

23.00 (2.45) 9.00 (13.71) 24.00 (22.86)

Slope (Syx)

Total Words

4.40 (1.14) 4.67 (0.82) 3.33 (0.58)

2.83 (0.41) 3.00 (0.58) 2.67 (0.58)

1.33 (0.58) 2.29 (1.11) 2.33 (0.58)

1.33 (0.58) 2.86 (0.90) 2.67 (1.15)

M (SD)

0.03 (1.31) 0.11 (0.87) 0.50 (0.41)

0.06 (0.42) 0.07 (0.61) 0.50 (0.41)

0.50 (0.41) 0.43 (0.68) 0.50 (0.41)

0.50 (0.41) 0.36 (0.51) 1.00 (0.82)

Slope (Syx)

Quality

Writing Outcomes for Fourth- and Fifth-Grade Student with Writing and Behavioral Difficulties.

Toby

Student

Table 3.

268 KARIN SANDMEL ET AL.




Veraa

Dana

Vancea

6 5 1

6 4 1

6 3 1

5 7 3

11.67 (2.79) 11.20 (1.30) (10.00) 0%

11.67 (0.82) 11.25 (0.05) (10.00) 0%

12.67 (2.07) 12.33 (1.53) (15.00) 0%

8.00 (2.00) 11.57 (1.99) 11.33 (1.15) 22.22%

0.47 (2.28) 0.10 (1.49)

0.08 (0.83) 0.09 (0.59)

0.39 (1.49) 0.5 (2.04)

0.03 (2.23) 0.61 (1.64) 1.00 (.82)

233.33 (65.77) 125.20 (44.34) (132.00)

190.83 (19.03) 151.75 (16.68) (128.00)

268.50 (38.35) 281.33 (21.13) (314.00)

147.67 (22.62) 146.86 (23.93) 137 (8.66)

8.84 (61.88) 12.90 (45.46)

1.82 (19.65) 5.06 (17.48)

0.20 (42.87) 1.50 (29.80)

5.44 (19.70) 2.39 (25.60) 7.50 (6.50)

4.00 (0.63) 3.00 (0.71) (3.00)

4.17 (0.41) 3.75 (0.50) (3.00)

4.83 (0.75) 5.00 (0.00) (6.00)

2.67 (0.52) 3.29 (0.76) 3.33 (0.58)

0.04 (0.69) 0.10 (0.80)

0.00 (0.46) 0.03 (0.61)

0.08 (0.75) 0.00 (0.00)

0.03 (0.57) 0.25 (0.58) 0.00 (0.82)

Note: PND, percentage of nonoverlapping data. This was calculated dividing the number of data points at postinstruction overlapping with the number of data points at baseline. a These students completed one postinstruction measure and mean values were not collected.


Cora

Timed-Writing and SRSD Strategies 269

270


Procedural fidelity for writing prompt administration was collected between 33.33% and 100% of the time across the three phases (e.g., baseline, intervention, postinstruction) and ranged between 97.5% (SD ¼ 5.00) and 100%. Trained RAs typed students’ writing prompts into the computer and corrected the writing for punctuation, spelling, and capitalization. Interrater reliability was calculated by the total number of agreements divided by agreements plus disagreements and for entry was 0.99. Writing Measures Writing prompts were scored for the number of elements present, writing quality, and total number of words written. Writing Elements. Writing prompts were scored for the number of elements present in each of the stories. The story elements were: who, when, where, beginning, middle, end, feelings, and exciting million-dollar words. Each element was scored on a 3-point Likert-type scale (0–2). An element was scored 0 if it was missing, 1 if the element was present, and 2 if the element was present and well developed. A total element score was calculated by adding up the scores for each element (range 0–16). Two trained RAs individually scored the writing prompts and interrater reliability was 0.89 during training and 0.96 during the study. Writing Quality. A holistic quality-scoring rubric, adapted from the state writing rubric, was used to score writing quality. The quality score, a 0–6 point Likert-type scale, measured students’ writing in the following categories: organization, idea development, sentence fluency, word choice, and audience awareness. Two RAs participated in a 2-hour training to score the writing prompts for quality. Interrater reliability was 0.91 during training and 0.86 during the study. Length. Length of students’ stories was calculated by counting the total number of words written. The Microsoft Word word-count tool was used to calculate the total words written. Social Validity Two social validity measures were collected prior to and upon completion of the study. Teachers completed the Intervention Rating Profile (IRP-15; Witt & Elliott, 1985). The purpose of the IRP-15 was to assess teachers’ perceptions and acceptability of the intervention and completion takes approximately 10 minutes. Teachers rated 15 items on a 6-point Likert-type


271

scale (1 ¼ strongly agree, 6 ¼ strongly disagree) at a time convenient to them. The internal consistency reliability coefficients range from 0.88 to 0.98. Students completed the Children’s Intervention Rating Profile (CIRP; Witt & Elliott, 1985) prior to and upon completion of the intervention. The CIRP was used to assess students’ perceptions and acceptability of the intervention. Students rated seven items on a 6-point Likert-type scale (1 ¼ I do not agree, 6 ¼ I agree) as the measure was read out loud. The internal consistency reliability coefficients range from 0.75 to 0.89. Experimental Design and Statistical Analysis A multiple probe across participants with multiple probes at baseline design (Kennedy, 2005) was used to evaluate the effects of the SRSD approach and writing strategies (TACO and TOWER) for improving the number of story elements, writing quality, and length of composition for students who struggle with writing and behavior difficulties. There were three experimental conditions: baseline, intervention, and postinstruction. It was anticipated that all procedures would be the same across conditions for students with the exception that students were in different legs of instruction and the implementation of the intervention was staggered to control for history and maturation. Students typically received three days of instruction weekly and then were administered a writing prompt every Friday. We intended that upon completion of the intervention, students would be immediately administered a writing prompt, and then administered postinstruction writing prompts over the next two to three weeks. As we explain shortly, this was not possible. After a minimum of three consecutive writing prompts administered during baseline, a visual inspection of the graphed number of writing elements present in the students’ stories was conducted and phase-change decisions were made. If the students’ scores on the writing elements were not stable after three administrations, additional writing prompts were administered until stability was established. Data collected during the intervention phase were used to make phase-change decisions about when to begin instruction for the next leg of students and when to begin administering postinstruction writing prompts. Incomplete Intervention and Data Collection for Leg 3 Students Unfortunately, due to catastrophic flooding and extended school closings, students in Leg 3 in both studies were unable to complete the intervention phase to criterion (a characteristic of SRSD instruction and research is that students remain in instruction until criterion for independent performance

272


are reached; Harris et al., 2009). Thus, intervention was terminated due to the imminent end of the school year, and a single postinstruction probe was given to students in Leg 3 to end the study. In both studies, students in Legs 1 and 2 received 21–26 sessions of instruction, while students in Leg 3 received only 12–15 sessions of instruction (see Tables 2 and 5). Students in Leg 1 were administered postinstruction writing prompts two successive weeks after instruction ended as intended. Students in Leg 2 were administered postinstruction writing prompts biweekly in order to complete postinstruction probes before the end of the school year. An unstandardized mean gain score effect size was calculated for the social validity measures (e.g., IRP-15 and CIRP; Lipsey & Wilson, 2001). The effect size calculation used was: preinstruction mean score subtracted from postinstruction mean divided by preinstruction standard deviation plus postinstruction standard deviation.

Results Writing Performance Data pertaining to the number of story elements were analyzed using traditional data-analytic techniques including examination of stability, level, and trend (see Fig. 1; Kennedy, 2005). In addition, total words written and writing quality were examined using mean and slope comparisons across phases (see Table 3). Baseline Writing Performance. The mean number of writing elements at baseline ranged from 4.00 (SD ¼ 2.00; Toby) to 12.67 (SD ¼ 2.07; Vera; writing elements score range 0–16). Toby and Adam wrote short, brief stories, but did not include all of the story elements in their baseline writing. Trey and Cora, averaged eight elements in their stories. Their stories included the basic story elements, but they did not elaborate the parts of their story. Jess, Vance, Vera, and Dan, averaged 11.67 (SD ¼ 0.82) to 12.67 (SD ¼ 2.07) story elements in their baseline-writing prompts. Jess received a 16 on a writing prompt and hit the ceiling of the writing measure (see Fig. 1). The mean number of words written at baseline writing performance ranged from 37 (SD ¼ 6.08; Adam) to 289.40 (SD ¼ 139.14; Jess). The mean writing quality scores at baseline ranged from 1.33 (SD ¼ 0.58; Toby and Adam) to 4.83 (SD ¼ 0.75; Vera). Overall, students’ performance patterns were stable during baseline, with the exception of Toby who demonstrated a counter therapeutic trend (slope ¼ 2.00).

273

Timed-Writing and SRSD Strategies Baseline

Post-Instruction

Intervention

16 14 12 10 8 6 4 2 0

Toby Adam

Narrative Elements

1 2

3 4

5 6

7

8 9 10 11 12 13 14 15 16 17 18

16 14 12 10 8 6 4 2 0

Trey Jess Cora

1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18

16 14 12 10 8 6 4 2 0

Vera Dan Vance 1

2 3

4 5 6

7 8

9 10 11 12 13 14 15 16 17 18 Sessions

Fig. 1.

Writing Elements for Fourth- and Fifth-Grade Students with Writing and Behavior Difficulties.

Intervention Writing Performance. During intervention, the mean number of writing elements across all subjects ranged from 7.71 (SD ¼ 4.11; Adam) to 14.33 (SD ¼ 1.03; Jess). Five of the eight students increased their mean number of writing elements, with both boys in Leg 1 and all three students in Leg 2 demonstrating an overall increasing trend. Vera, Dan, and Vance (Leg 3) were unable to complete intervention to criterion, as noted previously, and did not yet show any trend during intervention. The mean number of words written, during the intervention phase, ranged from 60.71 (SD ¼ 11.84; Adam) to 281.33 (SD ¼ 21.13; Vera). Toby, Adam, Trey, and Vera wrote longer stories during the intervention than at baseline. Jess, Dan, and Vance, on average, wrote shorter stories during the intervention than at

274


baseline. The mean writing quality scores during the intervention ranged from 2.29 (SD ¼ 1.11; Adam) to 5.00 (SD ¼ 0.00; Vera).

Post-instruction Writing Prompts. The average number of writing elements during post-instruction ranged from 8.33 (SD ¼ 2.08; Adam) to 13.00 (SD ¼ 1.73; Jess). Overall, in Legs 1 and 2, the trend was positive with the exception of Jess who demonstrated a counter therapeutic trend (slope ¼ 1.50). The average number of elements at post-instruction increased for Toby, Adam, and Trey. The average number of elements for Jess and Cora slightly decreased but were within one standard deviation of their average number of elements at intervention. Vera (Leg 3) completed only one probe as noted; her post-instruction score was higher than her baseline and intervention scores. The mean number of words written ranged from 73.33 (SD ¼ 14.57; Adam) to 157 (SD ¼ 30.62; Jess). Toby and Adam wrote the shortest stories, but their mean scores increased from their baseline and intervention writing prompts. The mean scores for total words written for Trey and Cora were approximately the same as their baseline and intervention performance. The mean score of total words written for Jess decreased at postinstruction. The mean writing quality scores ranged from 2.33 (SD ¼ 0.58; Adam) to 3.33 (SD ¼ 0.58; Jess and Cora). The SRSD approach and writing strategies had a positive impact on the total number of story elements included in the stories for students who completed the intervention to criterion. For Jess, who included all of the story elements once during baseline, the intervention had a positive impact by helping him to become a more consistent writer (e.g., including all of the parts all of the time). However, a functional relation was not established between the introduction of the intervention and changes in the number of elements evident in students’ writing probes because intervention could not be completed in Leg 3; thus, a sufficient number of replications could not be established. While SRSD instruction had a positive impact for number of story elements for students in Legs 1 and 2, further replications are needed. The SRSD instruction and writing strategies had mixed effects for improving the total number of words written by the students who completed the intervention. Two students, Toby and Adam, increased the total number of words written from baseline to intervention, and maintained an increased average at post-instruction. Trey and Cora averaged similar total number of words written across all phases. Jess averaged a high total number of


275

words written at baseline, but his average decreased as he moved to the intervention and post-instruction phases. There is insufficient evidence that the writing strategies and SRSD instruction had a positive impact on improving writing quality. From baseline to post-instruction, some students’ average writing quality increased (Cora and Adam) and some students’ average writing quality decreased (Trey, Jess, and Toby), but their average writing quality scores remained within one standard deviation of their performance throughout the study.

Social Validity Four teachers rated the intervention positively prior to the start of the intervention (see Table 2). Two teachers (one teacher had three students [Trey, Vera, Dan] in the study) did not rate the intervention positively. Upon completion of the post-instruction writing prompts, three students’ teachers scored the intervention positively and higher than at preintervention. For Vance, his teacher scored the intervention as favorable and the same as the preintervention score. For Cora, the teacher’s score decreased, but was still positive. For Trey, Vera, and Dan, the teacher scored the intervention negatively and the scores on the IRP-15 decreased postintervention. The effect size for the IRP-15 was nominal at 0.04.

CIRP. Prior to the start of the intervention, students rated the intervention positively (see Table 2). Upon completion of the postinstruction writing prompts, the CIRP scores for Adam, Trey, Vera, Dan, and Vance increased. The CIRP scores for Toby and Cora remained stable and positive. One student, Jess, rated the study less favorably upon completion. This student commented that students were asked to do too much writing and that they should not be tested so frequently in the future. The effect size for the CIRP was low at 0.21. In sum, the rating of the intervention by teachers and students were positive. Teachers scored the intervention favorably prior to the beginning of the intervention. Five of the teachers’ attitudes toward the intervention remained the same when the intervention ended, whereas one teacher did not score the intervention favorably for three of the students. The students rated the intervention favorably prior to and upon completion of the intervention with the exception of one student, Jess, who rated the intervention lower upon completion of the study.

276


STUDY 2: SECOND- AND THIRD-GRADE STUDENTS Method Participants and Setting The participants in this study were eight second- (N ¼ 5) and third- (N ¼ 3) grade students (seven boys, one girl). All of the students were Caucasian and their ages ranged between 7.12 and 8.79 (see Table 4). Participants were identified through the same procedures and met the same inclusion criteria as in Study 1. The second- and third-grade students in Legs 1 and 2 attended the same rural inclusive elementary schools as the students in Study 1. Students in Leg 3 attended a rural inclusive elementary school employing an integrated, three-tiered model of prevention with a population of approximately 125 students. The demographics of the school were 99% Caucasian and 1% other. Approximately 30% of the students qualified for free/reduced lunch. The school was in good standing under the No Child Left Behind Act. Writing Practices of Second- and Third-Grade Teachers Based on the Teacher Survey of Classroom Writing Practices (Cutler & Graham, 2008) results, teachers described the general classroom writing practices as a combination of traditional writing skills and the process writing approach. The most frequent activities included: encouraging invented spelling, teaching of spelling, grammar, punctuation, and capitalization, mini-lessons on writing skills and processes, and monitoring student progress to drive their instructional decisions. Less frequent writing activities included: assigning homework to students and teaching writing strategies. Trained RAs completed two 15-minute observations of second- and thirdgrade teachers’ writing practices during the three phases of the study. The observed activities most included: teaching students text organization, planning strategies, and grammar; reteaching skills and strategies; and stating the goals of instruction. The activities observed the least included: students and teachers assessing student writing, assigning homework, using dictation, teaching timed-writing strategies, using self-statements, and graphing story parts. Identified Students Parent/guardian permission forms were sent home to 13 students who met the inclusion criteria. Nine students returned signed permission forms allowing

277


Table 4.

Characteristics of Second- and Third-Grade Students with Writing and Behavior Difficulties.

Student

Finn

Wade

Andy

Carl

Anne

Phil

Chad

Drew

1

1

1

2

2

2

3

3

8.39 Male C 85.5 86 79

7.36 Male C 89 114

8.79 Male C 97.1 99 113

7.67 Male C 85.5 100 110

7.29 Female C 88.4 102 97

8.76 Male C 71.0 104 99

7.12 Male C 97.1 110 94

8.44 Male C 82.6 96 101

2 11

1 12

2 8

2 8

3 6

2 10

1 14

2 10

SSRS-T (pre) Social skills (SS) Problem behavior (SS) Academic comp (SS)

80 113 77

91 115 79

89 95 88

87 108 89

95 91 82

81 130 76

89 130 80

95 113 83

SSRS-P (pre) Social skills (SS) Problem behavior (SS)

109 105

78 108

104 108

84 100

114 110

101 102

113 89

111 102

SSRS-T (post) Social skills (SS) Problem behavior (SS) Academic comp (SS)

79 112 80

94 113 78

93 112 94

81 122 78

86 110 74

70 135 78

91 125 80

82 122 76

SSRS-P (post) Social skills (SS) Problem behavior (SS) Office discipline referrals

109 89 0

101 97 0

123 86 3

89 110 0

114 110 0

106 112 3

104 108 1

a

Leg General demographics Age in years Gender Ethnicity IQ WJ-III (pre; SS) WJ-III (post; SS) Screening DWA SRSS

a

a

0

Note: C, Caucasian; IQ, intelligence quotient; Wechsler Intelligence Scale for Children – IV (WISC-IV; Wechsler, 2003); WJ-III, Woodcock Johnson – III Test of Achievement (Woodcock et al., 2001); Writing Sample Subtest; SS, standard score; SRSS, student risk screening scale; SSRS-T, Social Skills Rating System – teacher version (Gresham & Elliott, 1990); SSRS-P, Social Skills Rating System – parent version (Gresham & Elliott, 1990). a Information not available.

them to participate. When assented, all students agreed to participate in the study. During the study, we learned one student did not meet the inclusion criteria as she had an individualized education plan. She continued instruction with her peers, but the data regarding her performance was excluded from this analysis and we report on the writing performance of eight students.

278


SRSD Intervention Procedures Second- and third-grade students received instruction in groups of two or three (see Table 5). The second- and third-grade WIs were trained with the fourth- and fifth-grade WIs. The same procedures were used for collecting WIs self-report fidelity and observer fidelity. Treatment fidelity, based on WI self-report, was 100%. Observer treatment fidelity was collected between 32% and 40.91% of the time and was 100%.

Table 5. Descriptive Characteristics of Instructional Sessions, Treatment Fidelity, and Social Validity for Study 2. Student

Phase

Number Average TF Teacher of Session Sessions Time %

Finn

Baseline Intervention Postinstruction

22

Wade Baseline Intervention Postinstruction

24

Andy Baseline Intervention Postinstruction

25

Carl


24


22


23

Chad Baseline Intervention Postinstruction

13

Drew Baseline Intervention Postinstruction

12

Anne

Phil

27 min

27 min

27 min

30 min

30 min

30 min

27 min

26 min

100.00

100.00

100.00

100.00

100.00

100.00

100.00

100.00

TF Observer Collected

SD

0.00 36.36%

0.00 33.33%

0.00 32.00%

0.00 37.50%

0.00 40.91%

0.00 39.13%

0.00 38.46%

0.00 33.33%

Observer TF Social Validity

%

SD

100.00

0.00

100.00

100.00

100.00

100.00

100.00

100.00

100.00

IRP-15 CIRP 83

34

76

40

75

32

73

37

82

35

75

42

75

32

54

42

75

37

76

30

82

31

24

42

78

32

87

32

90

37

75

33

0.00

0.00

0.00

0.00

0.00

0.00

0.00

279


Measures The same descriptive measures collected for the participants in Study 1 were collected for the second- and third-grade students (see Table 4). The writing prompts, administration procedures, and scoring procedures for the secondand third-grade students were the same as in Study 1. Procedural fidelity for administration of writing prompts was collected between 25% and 100% of the time across the three phases and ranged between 98.15% (SD ¼ 4.53) and 100%. The same writing and social validity measures collected in Study 1 were collected for the second- and third-grade students.

Results Writing Performance Data pertaining to the number of story elements were analyzed by the examination of stability, level, and trend (see Fig. 2; Kennedy, 2005). In addition, number of story elements, total number of words written, and writing quality were examined using mean and slope comparisons across phases (see Table 6). Baseline Writing Performance. The mean number of writing elements for students who participated in the second- and third-grade study at baseline ranged from 1.67 (SD ¼ 1.37; Chad) to 9.20 (SD ¼ 1.10; Phil; writing elements score range 0–16; see Table 6). Three students, Finn, Carl, and Chad, wrote short, brief stories and did not include all of the story elements. Three students, Andy, Anne, and Drew, included several basic story elements, but these elements were not elaborated. Two students, Wade and Phil, averaged above 8 on the story elements and included some elaborations to the story elements. Overall, students’ performance patterns were stable during baseline with the exception of Wade who demonstrated a positive trend (slope ¼ 1.30). The mean number of words written ranged from 10.67 (SD ¼ 7.34; Chad) to 262 (SD ¼ 69.93; Phil). The mean quality writing scores ranged from 1.00 (SD ¼ 0.00; Chad) to 3.40 (SD ¼ 0.55; Phil). Intervention Writing Performance. As noted previously, students in Leg 3 were unable to finish the intervention to criterion; this included Chad and Drew. During intervention, the average number of writing elements across all students ranged from 1.25 (SD ¼ 0.96; Chad) to 8.43 (SD ¼ 1.51; Wade). Finn and Andy increased their average number of writing elements in their

280

KARIN SANDMEL ET AL. Baseline

16 14 12 10 8 6 4 2 0

Post-Instruction

Finn Wade Andy

1

Narrative Elements

Intervention

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17

16 14 12 10 8 6 4 2 0

Carl Anne Phil

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17

16 14 12 10 8 6 4 2 0

Chad Drew 1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17

Sessions

Fig. 2.

Writing Elements for Second- and Third-Grade Students with Writing and Behavior Difficulties.

stories, but were within one standard deviation of their baseline performance. Wade, Anne, Andy, Chad, and Drew had similar number of writing elements on the intervention writing prompts as the baseline writing prompts. For Phil, the number of writing elements for the intervention prompts decreased and was not within one standard deviation of his baseline writing performance. Overall, students’ performance patterns were stable during the intervention phase. Carl showed a positive trend (slope ¼ 0.93) and Chad demonstrated a moderate positive trend (slope ¼ 0.50). The mean number of words written ranged from 4.50







Wade

Andy

Carl

Anne

Phil

5 6 3

5 6 3

5 6 3

4 7 3

4 7 3

4 7 3


9.20 (1.10) 6.28 (1.25) 7.67 (2.08) 0%

4.00 (2.12) 4.00 (2.27) 1.67 (0.58) 0%

2.40 (1.82) 4.43 (2.93) 4.67 (1.15) 14.29%

7.00 (1.83) 6.86 (2.79) 8.00 (1.00) 0%

8.25 (2.22) 8.43 (1.51) 9.00 (1.00) 0%

2.75 (0.96) 4.57 (1.90) 4.00 (1.00) 14.29%

M (SD)

0.28 (1.01) 0.04 (1.37) 0.50 (2.86)

0.17 (2.40) 0.11 (2.44) 0.50 (0.41)

0.12 (2.07) 0.93 (2.35) 0.00 (1.63)

0.40 (2.14) 0.11 (3.05) 1.00 (0.00)

1.30 (1.77) 0.50 (1.16) 1.00 (0.00)

0.30 (1.07) 0.07 (2.08) 0.50 (1.22)

Slope (Syx)

Elements

262.00 (69.93) 57.57 (19.59) 74.00 (52.20)

30.00 (17.97) 56.71 (54.26) 11.00 (2.65)

38.60 (9.29) 44.43 (27.29) 30.67 (13.61)

76.75 (27.90) 109.43 (33.81) 124.67 (30.92)

131.25 (41.48) 76.00 (47.85) 74.00 (18.19)

25.00 (6.16) 47.43 (23.99) 30.00 (5.00)

M (SD)

7.84 (77.75) 2.64 (20.52) 5.00 (73.48)

2.89 (19.14) 1.75 (59.29) 0.50 (3.67)

2.32 (8.57) 10.04 (18.15) 13.00 (5.72)

14.10 (25.89) 8.32 (31.37) 1.50 (43.68)

0.10 (50.81) 15.96 (36.34) 15.50 (13.47)

4.20 (3.59) 7.25 (19.90) 2.50 (6.12)

Slope (Syx)

Total Words

3.40 (0.55) 2.14 (0.69) 2.33 (0.58)

1.20 (0.45) 1.29 (0.49) 1.00 (0.00)

1.20 (0.45) 1.29 (0.49) 1.00 (0.00)

2.75 (0.96) 3.14 (0.69) 2.00 (1.73)

2.25 (0.50) 2.86 (0.69) 2.67 (0.58)

1.25 (0.05) 1.86 (0.38) 1.33 (0.58)

M (SD)

0.12 (0.54) 0.14 (0.68) 0.00 (0.82)

0.15 (0.32) 0.07 (0.51) 0.00 (0.00)

0.11 (0.41) 0.18 (0.33) 0.00 (0.00)

0.10 (1.16) 0.14 (0.68) 0.00 (2.45)

0.10 (0.59) 0.18 (0.63) 0.50 (0.41)

0.30 (0.39) 0.11 (0.33) 0.50 (0.41)

Slope (Syx)

Quality

Writing Outcomes for Second- and Third-Grade Students with Writing and Behavior Difficulties.

Finn

Student

Table 6.

Timed-Writing and SRSD Strategies 281

6 5 1



Chada

Drewa 5.67 (1.51) 5.00 (1.58) (6.00) 0%

1.67 (1.37) 1.25 (0.96) (4.00) 0%

M (SD)

0.05 (1.67) 0.10 (1.82)

0.08 (1.49) 0.50 (0.87)

Slope (Syx)

Elements

43.50 (9.35) 36.20 (7.26) (28.00)

10.67 (7.34) 4.50 (1.29) (19.00)

M (SD)

2.04 (5.07) 2.00 (7.55)

1.18 (6.26) 0.80 (0.95)

Slope (Syx)

Total Words

2.33 (0.52) 1.40 (0.55) (1.00)

1.00 (0.00) 1.00 (0.00) (1.00)

M (SD)

0.09 (0.41) 0.10 (0.61)

0.00 (0.00) 0.00 (0.00)

Slope (Syx)

Quality

Note: PND, percentage of nonoverlapping data. This was calculated dividing the number of data points at postinstruction overlapping with the number of data points at baseline. a These students completed one postinstruction measure and mean values were not collected.

6 4 1


Student

Table 6. (Continued )

282 KARIN SANDMEL ET AL.


283

(SD ¼ 1.29; Chad) to 109.43 (SD ¼ 33.81; Andy). The mean writing quality scores ranged from 1.0 (SD ¼ 0.00; Chad) to 2.86 (SD ¼ 0.69; Wade). Post-instruction Writing Prompts. The mean number of writing elements during post-instruction ranged from 1.67 (SD ¼ 0.58; Anne) to 9.00 (SD ¼ 1.00; Wade). The mean writing element scores for Wade, Andy, Carl, and Phil increased but were within one standard deviation of their intervention performance. The mean writing element scores for Finn and Anne decreased, but were within one standard deviation of their intervention writing performance. Students’ performance varied during the post-instruction phase. Andy demonstrated a positive trend (slope ¼ 1.00) and Phil and Finn demonstrated a moderately positive trend (slope ¼ 0.50). Carl’s performance was stable, Wade demonstrated a negative trend (slope ¼ 1.00), and Anne demonstrated a moderately negative trend (slope ¼ 0.50). The mean number of words written ranged from 11.00 (SD ¼ 2.65; Anne) to 124.67 (SD ¼ 30.92; Andy). The mean quality scores ranged from 1.00 (SD ¼ 0.00; Anne and Carl) to 2.67 (SD ¼ 0.58; Wade). The SRSD approach and writing strategies did not impact the total number of story elements included in the stories for second- and third-grade students who completed the intervention to criterion. A functional relation was not established between the introduction of the intervention and changes in the number of elements present in students’ writing prompts. There is insufficient evidence to state these writing strategies and SRSD instruction had a positive impact for improving the students’ total number of story elements, total words written, and writing quality. Social Validity Eight teachers rated the intervention positively prior to the start of intervention (see Table 5). Upon completion of the post-instruction writing prompt, Chad’s teacher scored the intervention positively and higher than at pretest. For Wade and Anne, their teachers scored the intervention favorably and approximately the same as at preintervention. For Finn, Andy, Carl, Phil, and Drew, their teachers scored the intervention lower at post-instruction. The effect size for the IRP-15 was 0.50. CIRP. Prior to the start of the intervention, all students rated the intervention positively (see Table 5). Upon completion of the postinstruction writing prompts, the CIRP scores for five students increased. The CIRP scores for Chad remained stable and positive. The CIRP scores

284


for Carl and Drew slightly decreased, but were positive. The effect size for the CIRP was 1.05. The rating of the intervention was mixed for the second- and third-grade teachers. Five teachers did not rate the intervention positively postinstruction and four teachers rated the intervention better or the same post-instruction. The students rated the intervention favorably prior to and upon completion of the intervention.

DISCUSSION The purpose of these two studies was to investigate the effectiveness of the SRSD approach with new on-demand writing strategies (TACO and TOWER) developed to address a state writing-competency test within a comprehensive, integrated three-tiered model of prevention for students with writing and behavior difficulties. Unfortunately, due to catastrophic flooding and extended school closings, students in Leg 3 in both studies were unable to complete the intervention phase to criterion. An important characteristic of the SRSD approach is that students remain in instruction until criterion for independent performance are reached (Harris et al., 2009). Thus, the Leg 3 data is of questionable utility, and an insufficient number of demonstrations and replications were obtained to demonstrate a functional relationship in each of the multiple baseline across participants studies. Fourth- and fifth-grade students who participated in intervention until criterion was reached (Legs 1 and 2) did, however, demonstrate increases in the total number of writing elements. There were mixed results for the total number of words written and no effects for the improvement of writing quality for the fourth- and fifth-grade students. In contrast, second- and third-grade students who participated in this study until criterion was reached (Legs 1 and 2) did not show improvements in the total number of writing elements, the total number of words written, or writing quality. Thus, there is no support for positive effects of SRSD with TACO TOWER at second- and third-grade levels at this time. In addition, before we discuss what we believe are important contributions from these studies and important implications for EBP, other limitations are noted. The writing measure used for participant selection in this study, previous district writing test scores, may not have identified students with writing difficulties as severe as we expected. The validity and reliability of the districtwriting prompt scores were not available to us; therefore, we cannot speak to


285

the sensitivity or specificity of this measure as a screening tool. Hollenbeck, Tindal, and Almond (1999) reported that the decision consistency between two raters assessing students’ writing performance on a state assessment averaged 55.5% when looking for an exact match on six writing traits, and 44.5% for off-by-1 (e.g., rater one’s score is 3 and rater two’s score is 4) match. Hollenbeck et al. argued the validity of decisions made about students’ writing performance are confounded by three factors including: (a) the agreement between raters, (b) the administration of one assessment, and (c) the arbitrary cut score. In the current studies, there were four students (Jess, Vera, Dan, and Vance, all in Study 1) who averaged 11 or more writing elements, out of 16, at baseline. One student, Jess, received a perfect score of 16 on one prompt. These students may not have needed this intervention or may not have been a good match to the instructional procedures. In all previous studies of students with emotional and behavioral difficulties, a standardized writing test, the Test of Written Language – III (TOWL-III; Hammill & Larsen, 1996), was used to identify students with writing difficulties (Lane, Harris et al., 2009; Little et al., in press). Future investigations could include a multiple screening process for identifying students with writing difficulties that begins with using the district writing assessment, with results confirmed by a psychometrically sound instrument, such as the TOWL-III. Due to scheduling conflicts, not all of the students received the intervention with the same teacher–student ratio (group size ranged from 2 to 3, with one student in Study 1 receiving instruction one-on-one). Effects of SRSD intervention individually, in pairs, or in a group of three may differ; some research has indicated that individual instruction may have stronger effects than group instruction (cf. Baker, Young, & Martin, 1990; Vaughn, Hughes, Moody, & Elbaum, 2001). The WIs in these studies reported that managing the behavioral and academic needs of these students in groups was difficult at times. Further investigation of teacher–student ratio in SRSD with students with emotional and behavioral difficulties is warranted.

CONTRIBUTIONS OF THESE STUDIES TO FUTURE RESEARCH Asking students to write on-demand has long been used as a means of assessment for various purposes. We were unable, however, to find any

286


experimental research on interventions for improving on-demand writing for state writing tests or other purposes (cf. Graham et al., in press). SRSD and the TACO TOWER strategies investigated in these studies appears promising at the fourth- and fifth-grade levels and should be further explored. We note that TACO TOWER is a complex set of strategies, and that the number of instructional sessions needed for students to reach independent performance was much higher for the fourth and fifth graders in this study than is typical in SRSD research. We cannot tell if this is solely related to the nature of the strategies, or may also be related to the characteristics of the students involved, and future research is needed to explore this. At the second- and third-grade levels, however, we did not see evidence of promising effects from SRSD and TACO TOWER. We believe it important to share our observations here. Although we initially thought that TACO TOWER would not be significantly more difficult to master than the story writing strategy (WWW, What ¼ 2, How ¼ 2) taught to similar students in second grade in earlier studies (cf. Lane et al., 2010, in press), our work with the secondthrough fifth-grade students in this study indicated that TACO TOWER is a more challenging strategy for students to master than is the WWW strategy. The WWW strategy requires students to create stories with all of the common story elements, typically in response to a picture prompt. Neither does it require students to write fictional, fantasy type stories that feature themselves as the main character based on a written prompt, nor does it require them to complete planning and writing within 35 minutes. Both of these factors appeared particularly difficult for the second- and third-grade students in Study 2. Although we modified the instruction in TACO TOWER to simplify it somewhat for second- and third-grade students, our modifications were clearly not adequate. The cognitive and metacognitive demands of TACO TOWER appeared overwhelming to the younger students. How to best work with second- and third-grade students to begin preparing them for state writing-competency tests remains an important research focus. Elements of TACO TOWER, such as evaluating a writing prompt and learning how to plan and write under time constraints, may prove fruitful with younger students if they are further broken down and instruction is spaced out over time. TOWER may be successfully taught first, and then instruction in TACO could be added. Further, states differ in the form of writing required across grade levels and even at the same grade level; thus, researchers will need to develop and investigate interventions aimed at different state competency writing tasks.


287

IMPLICATIONS FOR EVIDENCE-BASED PRACTICE Determining what practices are evidence-based and sustaining implementation of these practices in schools with high fidelity is critical to the future of special education (Odom, Brantlinger, Gersten, Thompson, & Harris, 2005). Cook, Smith, and Tankersley (in press) reviewed the burgeoning approaches and organizations devoted to evaluating and establishing EBPs. While great progress has been made, numerous issues exist, and these studies illustrate one of those issues. SRSD has been deemed an EBP by various groups (cf. Baker, Chard, Ketterlin-Geller, Apichatabutra, & Doabler, 2009; Graham & Perrin, 2007; National Center for Response to Intervention, http://www.rti4success.org/chart/instructionTools/). Yet, SRSD was not successful in this study with second- and third-grade students with a new strategy for a new writing task. To deem an intervention approach evidencebased may be premature in the face of the need for research across ages, grade levels, settings, types of subjects, and tasks. We are an experienced team, yet SRSD with TACO TOWER was not successful with second-grade students; it is likely that others who develop new writing strategies to use with SRSD may find their instruction ineffective despite the EBP label. We may need to further consider what is meant when we deem an approach evidence-based – for whom, when, where, and how? The two studies reported here indicate that far more research regarding SRSD is necessary to establish a set of writing strategies effective across grade levels, types of students, and writing tasks.

IMPLICATIONS FOR FREQUENT FORMAL ASSESSMENT DURING INSTRUCTION These studies also have implications for recent arguments for frequent (weekly or multiple times per week) formal assessment during instruction, even when the academic area is complex and progress is not anticipated weekly. Lieberman et al. (2010) argued that interventions at tiers 2 and 3 require that formal assessment data be collected more frequently and precisely than necessary for tier-1 interventions. They stated that for tier-2 interventions, data should be collected weekly, and at tier-3 data should be collected several times a week in order to monitor students’ response to intervention. In the current studies of SRSD at tier 2, based on their arguments, we gave writing tests to students weekly during intervention. Previous studies have included measures of performance during intervention, typically at particular points in the SRSD instructional stages

288


(cf. Danoff, Harris, & Graham, 1993; Kiuhara, O’Neill, Hawken, & Graham, 2010) or have assessed performance after students have met criterion in intervention (cf. Lane et al., 2008). Lieberman et al. (2010) based their arguments on their study, where experts were sent a survey composed of 16 carefully constructed, fictitious multiple baseline across participants graphs; data were analyzed to indicate whether changes in slope affected inference of a functional relationship. Further, data in the graphs were varied; important here was the manipulation meant to lead reviewers to expect the dependent variable would show a delayed change after the onset of treatment, as the authors noted is most likely to occur for development of academic variables and for learners who have more significant needs (such as those with severe disabilities). Their study involved reviewers for three journals (Journal of Behavioral Education, Education and Treatment of Children, and Journal of Applied Behavior Analysis). It is possible that reviewers for these journals are more experienced in, and accustomed to evaluating research with students with severe disabilities involving interventions often focused on either naturally occurring or discrete dependent variables, rather than research involving students with highincidence disabilities and measures of more complex learning that takes more time to occur. While 66 reviewers for these journals were asked to participate in their study, only 36 completed the survey (55%). While this return rate is low, this is further complicated by the fact that only 15 of these 36 subjects provided data that could be used to evaluate data graphs when change in the dependent variable could be expected to be delayed. The nature of the intervention described in Lieberman et al.’s (2010) study further complicates application of their (2010) findings to research involving SRSD and students with writing (and other) difficulties. In their study, to lead raters to expect the dependent variable would show a delayed change after the onset of treatment, the dependent variable was new words read, the population comprised of 6-year olds with severe intellectual disabilities, and the measurement context was a generalization setting. The raters were informed that the intervention, or independent variable, was a treatment using prompts and reinforcement that had been found to be effective and efficient in other populations and contexts. Thus, a further critical issue is the differences between Lieberman et al.’s (2010) independent variable and academic interventions such as SRSD for students who struggle academically but who do not have severe intellectual disabilities. Their described intervention was a relatively simple approach applied to a discrete task. In contrast, academic interventions with students with high-incidence disabilities or other struggling learners are often


289

complex, multicomponent, multifaceted approaches, and dependent measures are often not simple, naturally occurring, or observable, countable changes in behavior. Based on our experiences in the two current studies, we do not concur with Lieberman et al.’s (2010) recommendation that formal, graphical data on the target outcome variable be collected weekly in all tier-2 interventions, and believe that more research is needed to determine guidelines for researchers, schools, and teachers when evaluating tier-2 and -3 interventions among students with high-incidence disabilities or struggling learners who are not identified with disabilities. This does not mean, however, that we do not recommend collecting important assessment data during instruction. Harris and her colleagues (cf. Harris & Graham, 1996, 1999) have recommended, for example, that data be kept as students progress through SRSD instruction. Criteria for advancing through the stages of instruction in SRSD are embedded in the lessons. We tracked students’ progress in meeting these criteria, and this data is quite useful in monitoring performance and determining how to further adapt instruction to student needs (cf. Harris & Graham, 1996; Harris et al., 2008). We do concur with Lieberman et al. that when data is not collected and represented graphically during intervention, the graphs should be clearly marked to indicate this, rather than only explaining this in text. We were also concerned with performance issues that arose as we implemented weekly writing tests. Writing fatigue and reduced motivation may have influenced students’ performance on the writing prompts in this study. Students who reached criterion in intervention in these studies completed between 21 and 26 intervention sessions. This means that these students took 7–9 weekly writing tests, regardless of whether or not their WIs believed they had made sufficient progress in instruction to make such testing appropriate. Toward the end of the intervention phases and during the postinstruction writing prompt administrations, students appeared less motivated to write. One student, for example, after completing multiple writing prompts, asked how long she had written for and hoped that it was less than the week before. Another student explained that he had written less on a post-instruction prompt than he had on earlier prompts because he was just tired of writing. However, none of the students withdrew from the study, nor did they exhibit dissenting behaviors. Reduced motivation may in part be explained by the fact that in the first weeks of intervention, there is no reason to expect that students’ writing performance will improve weekly. As described earlier, the early stages of SRSD involve developing background knowledge, such as vocabulary and

290


genre understandings, that are needed to use the writing and self-regulation strategies. This typically takes some time, but while it is necessary, it does not represent sufficient intervention to change student performance (cf. Danoff et al., 1993; Kiuhara et al., 2010). The actual writing and selfregulation strategies are not taught until later in instruction. Previous research indicates that the ‘‘support it’’ stage, where performance while using these strategies is carefully scaffolded and teacher support slowly faded based on teacher observation and judgment of student performance, takes time and is critical (cf. Harris et al., 2009). Thus, requiring students to take weekly tests is equivalent to requiring them to fail for a period of time before they can begin to make progress; research has clearly shown that repeated failure is associated with reduced motivation, persistence, and performance (cf. Harris et al., 2009). Deliberately exposing students to such failure is not something we recommend based on our experiences and existing research. Further, the time spent in weekly assessments could have been more productively spent in instruction. In conclusion, effective test-taking, self-regulation, and writing strategies should be further investigated to identify strategies and skills students need to respond to on-demand writing tasks. Once effective interventions for ondemand writing are established, it will be important to investigate transfer of the use of the strategies to other tasks and settings. Writing has been and will continue to be a means for assessing student academic performance. As high stakes testing continues to play an important role in our schools and students are required to write on-demand, students with writing and behavior difficulties will need to learn writing, self-regulation, and testtaking strategies to succeed academically.

ACKNOWLEDGMENTS This research was supported by the Institute for Educational Sciences Grant (#R324B060018) from the Serious Behavior Disorders Competition (84.324B) to Vanderbilt University.

REFERENCES Adkins, M. H. (2005). Self-regulated strategy development and generalization instruction: Effects on story writing among second and third grade students with emotional and behavioral disorders. Doctoral dissertation, University of Maryland.


291

Agate, L. (2005). Investigation of primary grade writing instruction. Unpublished master’s thesis, University of Maryland. Baker, S. K., Chard, D. J., Ketterlin-Geller, L. R., Apichatabutra, C., & Doabler, C. (2009). Teaching writing to at-risk students: The quality of evidence for self-regulated strategy development. Exceptional Children, 75, 303–318. Baker, J., Young, M., & Martin, M. (1990). The effectiveness of small-group versus one-to-one remedial instruction for six students with learning difficulties. The Elementary School Journal, 91, 131–142. Bradley, R., Danielson, L., & Doolittle, J. (2007). Responsiveness to intervention: 1997 to 2007. Teaching Exceptional Children, 39, 8–12. Cook, B. G., Smith, G. J., & Tankersley, M. (in press). Evidence-based practices in education. In K. R. Harris, S. Graham, & T. Urdan (Eds), APA Educational Psychology Handbook (Vols. 1–3). Washington, DC: American Psychological Association. Cutler, L., & Graham, S. (2008). Primary grade writing instruction: A national survey. Journal of Educational Psychology, 100, 907–919. Danoff, B., Harris, K. R., & Graham, S. (1993). Incorporating strategy instruction within the writing process in the regular classroom: Effects on the writing of students with and without learning disabilities. Journal of Reading Behavior, 25, 295–322. Drummond, T. (1994). The Student Risk Screening Scale (SRSS). Grants Pass, OR: Josephine County Mental Health Program. Elliott, S. N., & Gresham, F. M. (2008). Social skills improvement system: Classwide intervention program. Bloomington, MN: Pearson Assessments. Forrest, D. B. (2000). Character under construction: Character building activities and reproducible for grades K-6. Chapin, SC: YouthLight. Graham, S., & Harris, K. R. (2003). Students with learning disabilities and the process of writing: A meta-analysis of SRSD studies. In: L. Swanson, K. Harris & Steve Graham (Eds), Handbook of learning disabilities (pp. 323–344). New York: Guildford Press. Graham, S., Harris, K.R., & Hebert, M. (in press). Assessing writing: Evidence-based procedures. Washington, DC: Alliance for Excellence in Education. Graham, S., Harris, K. R., MacArthur, C., & Fink, B. (2003). Primary grade teachers’ instructional adaptations for weaker writers: A national survey. Journal of Educational Psychology, 95, 279–293. Graham, S., & Perrin, D. (2007). Writing next: Effective strategies to improve writing of adolescent middle and high school. Washington, DC: Alliance for Excellence in Education. Gresham, F. M., & Elliott, S. N. (1990). Social skills rating system. Circle Pines, MN: American Guidance Service. Hammill, D., & Larsen, S. (1996). Test of Written Language – 3. Austin, TX: Pro-ED. Harris, K., & Graham, S. (1996). Making the writing process work: Strategies for composition and self-regulation (2nd ed.). Cambridge, MA: Brookline Books. Harris, K., & Graham, S. (1999). Programmatic intervention research: Illustrations from the evolution of self-regulated strategy development. Learning Disability Quarterly, 22, 251–262. Harris, K. R., & Graham, S. (2008). Self-regulated strategy development in writing: Premises, evolution, and the future. British Journal of Educational Psychology, Monograph Series II, 6, 113–136. Harris, K. R., Graham, S., Brindle, M., & Sandmel, K. (2009). Metacognition and children’s writing. In: D. Hacker, J. Dunlosky & A. Graesser (Eds), Handbook of metacognition in education (pp. 131–153). Mahwah, NJ: Erlbaum.

292


Harris, K. R., Graham, S., Mason, L. H., & Friedlander, B. (2008). Powerful writing strategies for all students. Baltimore: Brookes. Hollenbeck, K., Tindal, G., & Almond, P. (1999). Reliability and decision consistency: An analysis of writing mode at two times on a statewide test. Educational Assessment, 6, 23–40. Kennedy, C. H. (2005). Single-case designs for educational research. Boston: Allyn & Bacon. Kiuhara, S. A., O’Neill, R., Hawken, L.S., & Graham, S. (2010). The effects of Self-Regulated Strategies Development on the persuasive essay writing of 10th grade struggling writers. Manuscript in preparation. Lane, K. L. (2004). Academic instruction and tutoring interventions for students with emotional/behavioral disorders: 1990 to present. In: R. B. Rutherford, M. M. Quinn & S. R. Mathur (Eds), Handbook of research in emotional and behavioral disorders (pp. 462–486). New York: Guilford Press. Lane, K. L., Graham, S., Harris, K. R., Little, M. A., Sandmel, K., & Brindle, M. (2010). Story writing: The effects of self-regulated strategy development for second-grade students with writing and behavioral difficulties. Journal of Special Education, 44, 107–128. Lane, K. L., Harris, K., Graham, S., Weisenbach, J., Brindle, M., & Morphy, P. (2008). The effects of self-regulated strategy development on the writing performance of second grade students with behavioral and writing difficulties. Journal of Special Education, 41, 234–253. Lane, K. L., Harris, K., Graham, S., Driscoll, S. A., Sandmel, K., Morphy, P. (in press). The effects of self-regulated strategy development for second-grade students with writing and behavioral difficulties: A randomized control trial. Manuscript submitted for publication. Lane, K. L., Kalberg, J. R., & Menzies, H. M. (2009). Developing schoolwide programs to prevent and manage problem behaviors: A step-by-step approach. New York: Guilford Press. Lane, K. L., Little, M. A., Casey, A. M., Lambert, W., Wehby, J. H., & Weisenbach, J. L. (2009). A comparison of systematic screening tools for emotional and behavioral disorders: How do they compare?. Journal of Emotional and Behavioral Disorders, 17, 93–105. Lane, K. L., Menzies, M., & Kalberg, J. R. (in press). An integrated, comprehensive three-tier model to meet students’ academic, behavioral, and social needs. In K. Harris, T. Urdan, & S. Graham (Eds). American Psychological Association. Educational Psychology Handbook. Washington, DC: American Psychological Association. Lieberman, R. G., Yoder, P. J., Reichow, B., & Wolery, M. (2010). Visual analysis of multiple baseline across participants graphs when change is delayed. School Psychology Quarterly, 25, 28–44. Lipsey, M., & Wilson, D. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage. Little, M. A., Lane, K. L., Harris, K., Graham, S., Story, M., & Sandmel, K. (2010). Selfregulated strategies development for persuasive writing in tandem with schoolwide positive behavioral support: Effects for second grade students with behavioral and writing difficulties. Behavioral Disorders, 35, 157–179. Mason, L., & Shriner, J. (2008). Self-regulated strategy development instruction for writing an opinion essay: Effects for six students with emotional/behavior disorders. Reading and Writing: An Interdisciplinary Journal, 21, 71–94. Nelson, J., Benner, G., Lane, K., & Smith, B. (2004). Academic achievement of K-12 students with emotional and behavioral disorders. Exceptional Children, 71, 59–73.


293

Oakes, W. P., Mathur, S. R., & Lane, K. L. (2010). Reading interventions for students with challenging behavior: A focus on fluency. Behavioral Disorders, 35, 120–139. Odom, S. L., Brantlinger, E., Gersten, R., Thompson, B., & Harris, K. R. (2005). Research in special education: Scientific methods and evidence-based practices. Exceptional Children, 71, 137–148. Positive Action. (2008). Positive Action: Positive development for schools, families and communities. Twin Falls, ID: Positive Action. Reid, R., Gonzalez, J., Nordness, P., Trout, A., & Epstein, M. (2004). A meta-analysis of the academic status with emotional/behavior disturbances. The Journal of Special Education, 38, 130–143. Sandmel, K., Brindle, M., Harris, K., Lane, K., Graham, S., & Nackel, J. (2009). Making it work: Differentiating tier two self-regulated strategies development in writing in tandem with schoolwide positive behavioral support. Teaching Exceptional Children, 42, 22–33. Vaughn, S., Hughes, M. T., Moody, S. W., & Elbaum, B. (2001). Instructional grouping for reading for students with LD: Implications for practice. Intervention in School and Clinic, 36, 131–137. Walker, H. M., Ramsey, E., & Gresham, F. M. (2004). Antisocial behavior in school: Evidencebased practices (2nd ed.). Belmont, CA: Wadsworth. Wechsler, D. (2003). Wechsler Intelligence Scale for Children – Third Edition. San Antonio, TX: Harcourt Brace Jovanovich. Witt, J. C., & Elliott, S. N. (1985). Acceptability of classroom intervention strategies. In: T. R. Kratochwill (Ed.), Advances in school psychology (Vol. 4, pp. 251–288). Mahwah, NJ: Erlbaum. Woodcock, R. W., McGrew, K. S., & Mather, N. (2001). Woodcock-Johnson III Tests of Achievement. Itasca, IL: Riverside.

DEVELOPING WRITING FLUENCY FOR ADOLESCENTS WITH DISABILITIES Linda H. Mason and Richard M. Kubina ABSTRACT Adolescent students with disabilities often struggle with completing writing tasks efficiently. Until recently, most research regarding writing efficiency or fluency has examined production skills such as handwriting with young writers or examined how to use measures of fluency to assess student performance. In this chapter, 10 studies that directly address the impact of instruction on adolescents’ writing fluency will be reviewed. Findings indicated that when teacher modeling and structured practice was provided for writing within a time limit, students’ writing improved in the number of ideas or text parts written and in holistic quality. When measured, improvement generalized to a standardized writing fluency test. Implications for future research are noted.

In our modern, information-rich times, writing serves multiple purposes. Writing allows people to communicate, to learn, to express themselves artistically and creatively, to persuade others, and, in education, to demonstrate knowledge (Graham, 2006a). The purposes of writing would appear to place writing on level with other academics in regards to importance. Like Assessment and Intervention Advances in Learning and Behavioral Disabilities, Volume 24, 295–319 Copyright r 2011 by Emerald Group Publishing Limited All rights of reproduction in any form reserved ISSN: 0735-004X/doi:10.1108/S0735-004X(2011)0000024013

295

296

LINDA H. MASON AND RICHARD M. KUBINA

reading, mathematics, and science, writing becomes increasingly important for students’ success in school and for future outcomes. For instance, writing proficiency predicts a student’s academic achievement (Graham & Perin, 2007). Competence in writing has also become an imperative in both hiring and promotion, with the term ‘‘threshold skill’’ delineating important outcomes for obtaining employment and advancing a career (National Commission on Writing, 2004). With the stakes so high for attaining writing proficiency, instructional programs and methods for establishing writing competency and fluency continues to receive attention from teachers, school administrators, and business leaders (Katsiyannis, Zhang, Ryan, & Jones, 2007). While writing represents a highly complex activity, the basic definition of writing, or expressive writing, simply refers to the text a person composes (Kameenui & Simmons, 1990). Many effective programs and methods exist for establishing writing behavior, spanning areas from sentence construction (e.g., Saddler & Graham, 2007) to strategy instruction for complex informative writing tasks (e.g., De La Paz, 2001). The present data base surrounding writing allows one to conclude that many evidence-based practices exist for the teaching of writing (Graham & Harris, 2009). As with other curricular areas, however, the search for more effective instructional tactics and strategies continues.

CRITICAL SKILLS Two major reports in reading and mathematics demonstrate the current state of research and provide recommendations regarding critical academic skills. The National Reading Panel (2000) and the National Mathematics Advisory Panel (2008) summarized thousands of studies, highlighting topics for further investigation. Interestingly, ‘‘fluency’’ appeared in both reports as an essential component for practice and as a topic warranting further scrutiny. The National Reading Panel devoted an entire section to fluency, and reported instructional procedures that can lead to ‘‘meaningful improvements in reading expertise for students – for good readers as well as those experiencing difficulties’’ (National Institute of Child Health and Human Development, 2000, p. 3). Similarly, the National Mathematics Advisory Panel indicated that students need computational fluency, procedural fluency, and fluent recall of mathematical concepts in order to succeed and flourish in content areas (National Mathematics Advisory Panel, 2008). Practices that obstruct fluency evoked admonition: ‘‘The Panel

Developing Writing Fluency

297

cautions that to the degree that calculators impede the development of automaticity, fluency in computation will be adversely affected’’ (p. 50). A tremendous amount of potential exists for the study of fluency in both reading and mathematics. Writing, however, does not have an equivalent report. Hypothetically, what recommendation would be made by a ‘‘National Writing Panel’’ in regards to fluency? In some writing constructs, the relationship of fluency for a particular skill is well known. For instance, Graham, Berninger, Abbot, Abbott, and Whitaker (1997) found that handwriting and spelling accounted for 66% and 41% of the variance for compositional fluency for primary and intermediate grades. When examining compositional quality, handwriting accounted for 25% and 42% of the variance for primary and intermediate grades. Fluency for planning and composing across writing tasks, however, lacks analysis. Fortunately, the writing research literature that focuses on students with disabilities appears to concur that core fluency skills play a critical role in one’s developing competence in a complex skill. Within expressive writing, writers must master, or be fluent in, the mechanics of writing (e.g., handwriting, spelling, grammar), or more complex skills such as content generation will be negatively impacted (Graham et al., 1997). Writing fluency with core skills is only the beginning of understanding the relationship between pivotal skills competency. Berninger and Amtmann (2003) note that writing fluency can be examined within the context of the simple view of writing. Specific behaviors in text generation, transcription, and executive functions synergistically converge to produce an expressive writing repertoire. In some writing components, like transcription (i.e., handwriting, keyboarding, spelling), research shows how fluent behaviors affect more complex behaviors. For example, Kubina, Young, and Kilwein (2004) had students practice two component behaviors of spelling regular words: (a) hearing letter sounds and writing the corresponding grapheme and (b) the phonemic awareness skill of word segmentation. Once both component behaviors reached fluent levels, all students spelled words composed of the target letters with 100% accuracy even though spelling instruction was not provided.

WRITING FLUENCY Fluency in writing is not a new concept. With assessment, for example, fluent writing behaviors are often examined. In a systematic review of the

298


curriculum-based measurement literature for writing, McMaster and Espin (2007) concluded that simple measures of writing fluency, such as total amount of words written, are useful for screening purposes. The number of words written, however, does not serve as a technically adequate measure for more complex writing skills. Recent intervention study supports this claim that simple word counts are unlikely to provide comprehensive information on advanced writing skills. Mason, Kubina, Valasa, and Cramer (2010c), for example, implemented self-regulated strategy development (SRSD) instruction with five seventh- and eighth-grade students with severe emotional and behavioral disabilities (ED). After acquiring strategies for writing effectively, students engaged in quick writing or short writing tasks designed to help students rapidly apply a particular writing skill. The results demonstrated clear gains with improved quality measures and increases in the number of response parts. The total number of words written, however, decreased. As noted in previous intervention studies, increases in writing quantity do not necessarily indicate concomitant increases in writing quality (Graham, Harris, & Mason, 2005, 2006). As stated previously, although fluency has not been invisible, its reach into basic and advanced writing skills has not approached the threshold of scrutiny for fluency that has been developed in other content areas like reading and mathematics. Because fluency is universal to all human behavior regardless of age, gender, or ethnic status, a closer look at writing fluency can inform the field as to where it has been and indicate possible avenues of future exploration.

FLUENCY DEFINED Fluency can be defined generally, or can be applied to a specific level within a particular content area. One definition of fluency pertains to how quickly the behavior is performed. ‘‘Fluency refers to the appropriateness of the rate at which the student is accurately performing this newly acquired response’’ (Alberto & Troutman, 2009, p. 42). A student who can form the letters of their name legibly and with the proper sequence would not meet the definition of fluency if it took the student over a minute to write a short name such as ‘‘Sammy.’’ Quantity of writing can also apply to sentence length. Fluent writers will often write longer and more complex sentences than writers who have yet to attain proficiency. Fluency also can take a more specific definition. Mercer and Mercer (1998) defined fluency in written expression as the ‘‘quantity of verbal


299

output and y the number of words written’’ (p. 412). Generally, students who can write more would be judged as fluent compared to students who generate little content when they compose a story. As described previously, however, quantity does not correlate with quality. A functional definition of fluency should therefore specify the quantity and quality of the behavior. A functional definition of fluency, also called behavioral fluency, defines the quality and pace of the specific skill. Binder (1996) defines true mastery or fluency as ‘‘the fluid combination of accuracy plus speed that characterizes competent performance’’ (p. 164). Applied to a specific behavior, fluency would be defined as the specific accuracy level and speed. To illustrate the application, thinking about and writing uppercase cursive letters has a fluency aim of 60–80 letters with zero errors per minute (Freeman & Haughton, 1993). The accuracy is defined as 100%, and the speed is expressed as a range and occurs in a specified time unit of 1 min. As researchers and practitioners move forward with writing fluency as an instructional aim, a definition common to all writing behaviors is imperative. The clarity of defining and measuring fluency directly impacts replication and generalizability of research. In this chapter, we examine writing fluency studies conducted within two different pedagogical approaches, behavioral (i.e., precision teaching) and cognitive (i.e., strategy instruction). Fluency was measured by a variety of methods across the studies. In the first category, behavioral fluency studies, fluency was demonstrated by quality and pace or accuracy and speed in the measured production of the writing skills. These early studies (Albrecht, 1981; Spaulding, Haertel, Seevers, & Cooper, 1995) provide evidence that fluency instruction can impact not only the number of words written, but also other important writing elements such as ideas and text elements. In the second category, strategy instruction for fluency, persuasive writing instruction was implemented as a base for the intervention. Strategy instruction for fluency was evaluated through two approaches: (1) fluency taught explicitly within a quick writing intervention and (2) fluency as a generalization outcome measure for essay writing instruction.

BEHAVIORAL FLUENCY INTERVENTIONS Behavioral fluency interventions all share a common element: timed performance of behavior and repeated practice. One case study and one

300


single-case design study with young adolescents (4th–6th grade) demonstrated the effects of practicing writing behaviors to a fluency aim.

Fluency for Creative Writing An early case study conducted by Albrecht (1981) examined how the quantity of students’ creative writing could be improved. The participants were four 6th-grade students in a general education classroom. Each day, the teacher had the students select a topic for their writing. The students were then instructed to write as many short phrases or one-word ideas pertinent to the subject as possible. The students were timed for 1 min; their total number of ideas was then displayed graphically on a chart. Immediately following the 1-min timing and charting, students were given 10 min to write about their topic. The teacher instructed students to use ideas from the 1-min timing procedure and to not be concerned with spelling, mechanics, or sentence structure. After students wrote for 10 min, they counted and charted the number of words written. The written essay was considered a draft, and in an untimed period students revised their work focusing on spelling, mechanics, grammar, or any other aspect of the writing. The results of this procedure demonstrated a positive relationship between the short words or phrases generated in 1 min and the quantity of words written in the 10-min writing period. As students generated more ideas in their 1-min sample, they subsequently wrote more words in their essays. In other words, fluency with idea generation transferred to written output, even though the topics changed each writing period.

Fluency in Story Writing Spaulding et al. (1995), in a single-subject study, investigated the effects of teaching structure words and visual imagery on the correct and incorrect number of total words and descriptive words written in 10 min. The study used an alternating treatments design with four elementary students (one male and one female in 4th-grade, and two 5th-grade males) with learning disabilities (LD). During baseline, students were given a 10-min free writing period and a story starter. Across baseline all students received a different story starter. Stories were measured for total number of words written and descriptive


301

words. ‘‘Total number of words written’’ was defined as words in sentences that expressed a coherent thought. Spelling errors were not counted if the experimenters could not decipher the word. Descriptive words were defined as adjectives, adverbs, or metaphors. The three alternating interventions included: (a) different story starters, (b) the same story starter, and (c) the same story starters plus structure words and visual imagery. For different story starters, students heard the story starter and wrote it on their paper. They were given 3 min to think about what they wanted to write. For the second condition, students were given the same story starter; each session used the same story starter until they reached a fluency aim of 200 total words written in the 10-min period. In the third condition, the students followed the same procedures as the second condition, but the teacher randomly selected and introduced four structure words during the 10-min free write. The structure words described who, what, where, and when, and/or described shape, color, size, texture, movements, moods, or smells. The structure words were written on a piece of paper and taped to the students’ desk so they would have access to words during the free write. Results of the experiment showed that all students improved during free writing under the three alternating instructional conditions. Spaulding et al. (1995) concluded that the instructional effects practiced and learned with the third condition, using structure words and imagery, had the most positive impact in improving performance.

Behavioral Fluency Summary Albrecht (1981) and Spaulding et al. (1995) noted that when behaviors approached or met the fluency criteria, unlearning the behavior or not having it occur in other contexts was demonstrated to be difficult. A fluent behavior is one that is retained for long periods of time, endures or does not show fatigue, and can be applied to other skills (Binder, 1996; Kubina & Morrison, 2000). While there are few studies in the literature, it would appear that behavior fluency interventions have promise and should be considered for future research.

STRATEGY INSTRUCTION FOR FLUENCY In recent years, researchers examined the effects of an evidence-based intervention, SRSD, in improving writing fluency through persuasive

302


writing instruction with adolescents (middle and high school) with disabilities. Mason et al. (2009, 2010a, 2010b, 2010c) examined the effects of using SRSD for building fluency with instruction for a timed constructed persuasive paragraph (quick writing). SRSD for persuasive essay writing, and for persuasive essay writing in combination with quick writing, was examined by Mastropieri et al. (2009, 2010a, 2010b). The explicit instruction for strategy acquisition that is combined with procedures for self-regulated learning inherent in SRSD supports students’ skill development and mastery across the writing process (Mason, 2009). SRSD strengthens students’ understanding of planning, composing, and editing/revising, as well as helping them develop a positive attitude toward writing (Mason, Harris, & Graham, 2011). Assessment of students’ skills in writing fluently after SRSD instruction has been less understood. SRSD includes six recursive and flexible instructional stages for scaffolding student learning to meet individual student needs (Harris, Graham, Mason, & Friedlander, 2008). In the first stage, develop and activating background knowledge, the teacher introduces and reviews vocabulary related to the targeted strategy (e.g., POW (Pick my idea, Organize notes, Write and say more) þ TREE (Topic sentence, Reasons, Explanations, Ending sentence) for persuasive writing) and for writing to the specific task (e.g., essay or constructed paragraph response). Goal setting, self-monitoring, self-instruction, and self-reinforcement for selfregulating the writing process are introduced. In the next stage, discuss it, strategy steps and mnemonics and the purpose of the strategy are described. The students’ current level of persuasive writing performance is evaluated, and learning goals are established. The teacher models it in the third stage by applying the strategy while thinking aloud. All instructional materials and self-regulation procedures are utilized while modeling the processes for planning and composing. Students memorize strategy mnemonics and the steps of the strategy in stage four. In stage five, support it, the teacher scaffolds the students’ use of strategies for the specific task through collaborative and guided practice. In this stage, students learn to establish goals, monitor their strategy use, self-instruct, and positively reinforce themselves for achieving goals. In the final stage, independent performance, teacher and instructional material support (e.g., graphic organizers) is faded as the student takes ownership of the strategy and begins to apply what they have learned across settings and tasks. Results of a meta-analysis of writing studies indicated that SRSD has significant positive effects improving the quality, the number of persuasive parts or elements, and the length of students’ persuasive writing


303

(Graham, 2006b; Mason & Graham, 2008; Rogers & Graham, 2008; Taft & Mason, 2010). Effect sizes (ES) ranged from ES ¼ 0.82 to 5.18 for large group studies (ESW0.80 are considered large). Effect sizes for comparing postinstruction to baseline performance for single-subject studies ranged from 70 to 100 percent of nonoverlapping data (PND; PNDW0.90 are considered effective, PND between 0.70 and 0.90 are moderately effective) (Scruggs, Mastropieri, & Casto, 1987). Given the positive effects of SRSD for improving the quality, the number of parts written, and the length of persuasive writing, using SRSD as an intervention to support writing fluency seems most appropriate, and therefore was addressed in two lines of research established by Mason et al. (2009, 2010a, 2010b, 2010c) and by Mastropieri et al. (2009, 2010a, 2010b). The purpose of the research was to develop an intervention that improved students’ attention to writing within a defined time frame. In both intervention approaches, the six stages of strategy acquisition and four selfregulation procedures were included in criterion-based recursive SRSD instruction (see Appendix A for the SRSD for persuasive writing lesson structure implemented in the studies).1

BUILDING FLUENCY THROUGH QUICK WRITING Mason, Kubina, and colleagues investigated students’ writing fluency in five studies using SRSD for a persuasive quick write as the primary intervention (Mason, Kubina, & Taft, 2009; Mason et al., 2010c; Mason, Kubina, & Hoover, 2010a; Mason, Kubina, Kostewicz, Cramer, & Datchuk, 2010b). Quick writes are 10-min constructed writing responses (i.e., paragraph responses) to a question related to a specific topic and are often used in content classrooms to foster student reflection and elaboration about learning (Mitchell, 1996; Wood & Harmon, 2001). Quick writes support content learning by presenting a nonthreatening, informal, and brief writing activity for students, and are often used for assessing what students have learned in class activities and text reading (Fisher & Frey, 2004). Quick write instruction followed all procedures for strategy acquisition and self-regulation in SRSD. Writing prompts developed in prior research (e.g., Mason & Shriner, 2008) were modified and extended to be appropriate for small city and urban middle school students in the individual studies. In the first two studies, SRSD lessons for POW þ TREE focused writers to write, at minimum, an eight-part persuasive response: (1) Topic sentence (1 point earned), (2) Reasons – three or more (1 point earned for each reason),

304


(3) Explanations – write an explanation for each reason (1 point earned for each explanation), and (4) Ending sentence (1 point earned). Given schools’ criteria for including opposing views in persuasive writing, later studies expanded the TREE strategy steps by adding explicit focus to students’ attention to developing an effective counter reason and a counter argument refute for a total of 10 parts in the response. In addition to assessing students’ performance in writing TREE parts, the number of words written was counted for each response. Quality was also measured using a 0- to 7-point holistic scale based on anchor points (2 ¼ low, 4 ¼ medium, 6 ¼ high). Treatment acceptability data were collected from all study participants. Student disability categories and group mean (with standard deviations) performance results for the primary measures (i.e., parts written, number of words written, and quality written in a 10-min time frame) and for a generalization measure (i.e., standardized measure of writing fluency), when given prior to and after instruction, in each quick write study are reported in Table 1.

Quick Writing for Middle School Students with Disabilities The effects of quick writing instruction for students with disabilities were first tested in two single-case studies in an inclusive middle school (Mason et al., 2009). SRSD instruction for writing a paragraph was implemented in the first five to six 45-min instructional periods. Instruction varied only in the succeeding fluency intervention for writing the response in 10 min. In the first multiple-baseline across student group study, six 7th-grade students (four males, two females) were taught in pairs by a graduate research assistant. To improve generalizability of instruction, assessment data were collected by the special education teacher. After students had demonstrated independence in writing a response with all eight TREE parts, the instructor addressed improving fluency by providing practice in writing the timed response in one 45-min lesson. The practice lesson was repeated for one student. Results indicated that students improved performance with large effects: 94% PND for number of elements written at post instruction and 100% PND at maintenance. All students reported that the SRSD instruction had helped their writing performance and should be taught to other students. In the second study, a multiple-baseline study across three special education language arts classrooms was conducted with ten 7th- and 8thgrade students (four males, six females). After SRSD instruction for POW þ TREE, student fluency was supported in one 45-min lesson that

Students/Disability

5.05 (1.61) –

–

7.42 69.72 108.68 3.05 (1.35) (32.13) (31.28) (1.00) 9.78 – (2.20)

–

4.61 (2.03) 6.73 (2.23)

–

(14.62) (1.38)

(0.88)

5.15 (0.83) 6.60

(0.73) (26.42)

5.62 (1.24)

Post

(1.93)

Pre

7.13 76.02 111.06 3.27 (1.65) (31.12) (24.09) (1.25) 9.32 121.94 104.52 3.50

Post

3.63 (1.42) 8.40

Pre

Quality

7.25 87.43 103.75 3.39 (1.47) (18.03) (10.94) (1.47)

Post

Words

3.35 (1.12)

Pre

Parts

Timed Quick Write

– –

– –

–

–

–

Post

–

Pre

–

–

Post

Words Pre

–

–

–

–

–

Post

Quality

12.89 (2.90)

17.20 (9.09)

Pre

3.90 7.61 82.80 230.78 3.20 5.44 18.04 (1.52) (2.52) (67.75) (115.7) (1.4) (1.67) (5.04)

Pre

Parts

Untimed Essay

Study Means (Standard Deviations) for Persuasive Quick Writes, Essays, and Woodcock–Johnson-III Fluency Subtest.

Quick writing instruction Mason et al. (2009) 4 LD 1 LD/ADHD 1 ADHD 9 LD 1 ADHD Mason et al. 3 ED (2010c) 1 ED/ASD 1 ED/ADHD/SL Mason et al. 3 ED/ADHD (2010a) 26 at-risk Mason et al. 9 LD (2010b) 2 ADHD Essay Instruction Mastropieri et al. 4 ED (2010a) 2 ED/ADHD 1 ED/LD 1 ED/LD/ASD 1 EDLD/ADHD 1 ED/LD/SL/OHI

Study

Table 1.

21.78 (4.99)

18.2 (5.90)

21.00 (7.31)

Raw scores

–

–

Post

WJ-III

Developing Writing Fluency 305

Students/Disability

Pre

TPO 154.00 (26.71)

TPO 9.5 (0.41)

Post

TPO 93.47 (32.28)

Pre

Words

TPO 5.77 (1.58)

Post

Parts

Timed Quick Write

Pre

Pre

Post

Parts Pre

Post

Words Pre

Post

Quality Pre

Post

WJ-III

Standardized TPO 8.56 3.08 9.83 66.03 228.58 3.33 8.61 86.87 103.33 (0.76) (1.22) (0.41) (27.24) (61.92) (1.08) (0.71) (14.86) (14.87)

Standardized TPO 4.48 1.87 5.57 21.92 108.37 1.71 4.33 75.8 84.8 (1.25) (0.66) (2.13) (10.78) (50.39) (0.59) (1.76) (17.9) (4.2)

Post

Quality

Untimed Essay

WJ-III, Woodcock–Johnson – III fluency subtest; –, not collected or available at the time of publication; ED, emotional disability; LD, learning disability; ADHD, attention deficit hyperactivity disorder; SL, speech language disability; OHI, other health impaired; ASD, autism spectrum disorder; OHI, other health impaired; MD, multiple disability; ESL, English second language; TPO, 10-min timed assessment given to students after the additional quick write instruction (no preinstruction test available); preliminary results for manuscripts in preparation.

Essay þ quick writing instruction Mastropieri et al. 1 ED (2009) 3 ED/LD 3 ED/LD/ESOL 2 ED/LD/OHI 2 ED/OHI 1 ED/LD/SL/ESL Mastropieri 4 ED (2010b) 1 ED/LD 1 ED/ASD 2 ASD/ED 1 ED/ASD/OHI 2 OHI/ED 1 ED/MD

Study


306 LINDA H. MASON AND RICHARD M. KUBINA


307

included teacher-led modeling and student practice in writing within the 10min time limit. The lesson was not repeated. Two special education teachers delivered all assessments and instruction. Results indicated that students improved the number of elements written with medium effects: 77% PND at post instruction and 67% PND at maintenance. Although all students reported that POW þ TREE should be taught to other students, some students noted that instruction had occurred during the time usually allocated to homework time, that expressive writing was more fun, and that ‘‘four square [a writing method] is more straightforward and you do not have to think’’ (Mason et al., 2009, p. 12). Researchers observed that the results for quality of the response in both studies were disappointing: 56% PND at post instruction and 75% at maintenance in the first study, and 62% PND at post instruction and 50% PND at maintenance in the second study. In both studies, although students demonstrated improvement over baseline performance, the researchers speculated that insufficient scaffolding of support (e.g., no teacher modeling in study one, small group vs. paired practice in study two) and insufficient practice trials for writing in 10 min may have influenced post instruction variability in the quality of student responses.

Quick Writing for Middle School Students with ED Mason et al. (2010c) conducted a multiple-baseline study to evaluate the effectiveness of strategy instruction in persuasive writing with five 7th- and 8th-grade students (four males, one female) who attended a county alternative placement school for students with severe ED. A graduate assistant taught all students in one-to-one instruction to plan and write a 10min persuasive response with the 10-part TREE criterion. Instruction was conducted over five 30-min sessions plus five 10-min fluency practice sessions; assessment data were collected by the graduate assistant. Given the variability of performance in the prior two studies (Mason et al., 2009), additional prescriptive instructions for developing fluency and to support student practice in writing a 10-min timed response were used (see Appendix B for SRSD fluency lessons). During baseline, individual student performance varied significantly over time. For example, one student’s number of parts ranged from writing zero parts to writing five parts, while another student’s performance ranged from one part to seven parts. After SRSD for quick writing, positive effects were noted for all students in the quality of written responses, with 84% PND at

308


postinstruction and 60% at maintenance. The number of parts and words written also denoted improved performance. An important finding of the study was the performance stability observed across measures after instruction. An analysis of the response part type indicated that prior to instruction students could effectively write a topic sentence and a minimum of three reasons, but had difficulty in writing explanations, counter arguments, and ending sentences. After instruction, students were more consistent in writing all parts within the 10-min time frame. Performance gains also generalized to the Woodcock–Johnson III Writing Fluency (WJIII fluency; Woodcock, McGrew, & Mather, 2001) subtest2 with a small effect (ES ¼ 0.46). All students reported that the instruction benefited performance, ‘‘In my opinion teens my age should be taught the POW þ TREE strategy’’ (Mason et al., 2010c, pp. 152–153). Quick Writing for High School Students with ED A multiple-baseline across students study was conducted to evaluate the effectiveness of SRSD for quick writing with one 9th-grade and two 11thgrade male students with ED in an inclusive high school setting (Mason et al., 2010a). Although this study replicated procedures for teaching students to write a persuasive response with 10 parts and provided extra plan practice for writing in a 10-min time limit (Mason et al., 2010c), students’ schedules disrupted fluency practice – only one to two practices were provided. Instruction was conducted over five to seven 30-min sessions plus five 10-min sessions. Instruction was provided by the third author, an advanced graduate student who also worked in the school as a special education teacher. Baseline to postinstruction improvement was indicated with medium effects: 68% PND for the number of parts and the number of words written and 79% PND for quality. Students maintained improvement with 50% PND for the number of parts, 67% for the number of words written, and 83% for quality. The three students reported that the instruction had helped writing performance: ‘‘It has helped me become a better writer. It is easier for me to write. Before I would just think of something and write about it but now I have a strategy’’ (Mason et al., 2010a, p. 20). Quick Writing for At-Risk Middle School Students A regression discontinuity experimental study was conducted in four urban charter schools (Mason et al., 2010b). A total of 400 students in 7th and 8th

309


grade were given a group administered 10-min persuasive writing assessment and the WJ-III fluency subtest in their language arts classroom; 60 students, identified by assessment cut scores, qualified for instruction; 37 students consented to participate in the intervention. Consent to use classroom data for the purpose of research was obtained for a no-treatment sample of 242 students. Students with disabilities, and specifically students at risk for ED as noted in suspension and office report records, were included in the intervention and comparison student pools. The 37 qualifying students were taught to plan and write a 10-min persuasive response using SRSD for 10-part POW þ TREE. Individual graduate assistant delivered instruction was conducted over five 30-min sessions plus five 10-min fluency practice sessions. For students receiving instruction, individually administered writing performance data were collected prior to instruction (M ¼ 6.73 parts), immediately following instruction (M ¼ 9.78 parts), and at two maintenance time points (M ¼ 9.00, 9.76). Positive effects indicated that students had learned, and maintained performance, in writing a 10-min quick write response using the 10-part criterion. In addition, all students reported that the instruction benefited performance and should be taught to other students, ‘‘Because it helps with stating all the reasons why you believe something.’’ After completing maintenance testing for students in treatment, the 10-min timed persuasive writing assessment and the WJ-III fluency subtest were given in the language arts classroom to all 7th- and 8th-grade students. Preliminary analysis of data for the 242 no-treatment students and 37 treatment students indicated that the students receiving treatment, when retested in their classrooms, generalized learned performance to classroom (pretest to posttest ES ¼ 1.28 for number of persuasive elements written) and generalized learning to a standardized fluency measure (pretest to posttest ES ¼ 0.23). There was no difference in pre and posttest measures for the 242 students in the no-treatment condition.

Quick Write Summary SRSD for persuasive quick writing effectively improved the number of parts, the number of words, and the quality of students’ writing in a timed response. For adolescents with background knowledge of the writing process and some knowledge of the elements for effective persuasive writing, and when instruction is provided one-to-one or to student pairs, instruction is relatively brief consisting of five to six 45-min lessons and five 10-min

310


practice sessions. Researchers note that modeling timed writing plus providing students practice with feedback in writing a quick write in 10 min is critical for stabilizing performance with strong effects. In addition, it is important to note that all the reviewed studies implemented SRSD as recommended by including all six stages of strategy acquisition and four self-regulation procedures (Harris et al., 2008).

BUILDING FLUENCY THROUGH ESSAY WRITING Persuasive essay writing for fluency development has been tested in three experiments with middle school students with ED (Mastropieri et al., 2009, 2010a, 2010b). In each study, SRSD for the POW þ TREE persuasive writing strategy was implemented (Harris et al., 2008) for writing in an essay format. The standard, five paragraph minimum, adopted by the school system for an essay, was in alignment with state assessment expectations. Writing prompts developed in prior research (e.g., Mason et al., 2009; Mason & Shriner, 2008) were modified and extended to be appropriate for urban middle school students. Procedures for assessing writing performance by examining the number of parts, number of words, and holistic quality in the prior studies were also adopted. Student disability categories and results for the primary measures (i.e., number of parts written, number of words written, and holistic quality) and for a generalization measure (i.e., WJ-III fluency subtest), given prior to and after instruction, in each essay writing study are reported in Table 1. Essay Writing in a Day School In the first persuasive essay writing study, eight 8th-grade middle school students who attended a separate public day school for students with severe ED participated in a design experiment (Mastropieri et al., 2010a). Given the complexity of the students’ programming, the students (six males, two females) were often absent from the group instruction. In addition, students’ on task behavior and academic needs were variable. For example, on the WJ-III fluency subtest pretest scores ranged from beginning 3rd- to 11thgrade levels. Given the restrictions of the group setting, after five weeks of instruction (approximately twenty 29-min sessions given four days each week) the class was divided, based on ability, into two separate groups for instruction guided by graduate research assistants. The two student groups received, on average, an additional thirty 29-min sessions over the next


311

seven weeks. This study implemented eight-part TREE lesson plans used in prior research with elementary students with ED (e.g., Mason & Shriner, 2008). Modifications were made to materials (e.g., adding a separate column for explanations), and additional practice in identifying weaknesses of poorly written essays was provided to support the special learning needs of the students in this study. Although reference to a control or comparison group cannot be made, results of SRSD instruction for persuasive essay writing indicated that students made significant gains. Progress from pretest to postinstruction testing on the WJ-III fluency subtest indicated a moderate effect size (ES ¼ 0.67). This effect was somewhat less than what was noted for holistic quality, essay elements, transition words written, and length (number of words written and number of paragraphs written) at posttest and for maintenance testing (ES range 0.66–2.46). Given that fluency was not explicitly addressed during instruction, results indicated that SRSD instruction for persuasive essay writing had a generalized effect on fluency. Results of postinstruction interviews indicated that the students were positive about the intervention.

Essay Writing Plus Quick Writing in a Day School In the year following the design experiment, a multiple-baseline design study across four student groups was conducted to evaluate the effectiveness of SRSD for persuasive essay writing with 12 eighth-grade students (11 males, 1 female) with ED in the day school (Mastropieri et al., 2009). As in the prior study, students were taught to plan and write persuasive essays using SRSD for POW þ TREE. Following the lesson sequence used previously by Mason and Shriner (2008) and Mastropieri et al. (2010b), and following mastery in using the strategy to write an essay (fifty-five 30-min lessons), students were taught to apply the learned strategy to write fluently in 10 min. During fluency instruction, students were taught to use all previously learned TREE persuasive steps, but to write quickly in a paragraph format as described by Mason et al. (2009). In other words, students were encouraged to write responses that included a minimum of five sentences as opposed to five paragraphs. Timers and teacher modeling was used to support student practice in writing fluency. Essay writing performance during baseline was low across measures and improved after instruction, with 100% PND for the number of TREE parts written and the quality of the essay. Statistically significant gains were made

312


on the WJ-III fluency subtest with a strong effect size of 0.81. Students reported liking the instruction, and all stated that they use POW þ TREE in their other classrooms: ‘‘I am very glad that you guys came here to help me to write more, to learn how to write POW þ TREE’’ (Mastropieri et al., 2009, p. 36). In discussing results, Mastropieri and colleagues (2009) reflected that teaching essay writing to students with ED ‘‘exceeds the amount of instructional time typically devoted to teaching students how to write persuasive essays’’ (p. 37), and that optimal procedures for teaching fluent writing with this student population should be investigated.

Essay Writing Plus Quick Writing in an Inclusive School A multiple-baseline study across four student groups was conducted to evaluate the effectiveness of SRSD for POW þ TREE for essay and quick writing for 12 seventh- and eighth-grade male students with ED who attended an inclusive public school (Mastropieri et al., 2010b). Essay writing instruction ranged from 12 to 19 days, followed by 3–6 days of quick writing instruction. SRSD instruction and materials for the POW þ TREE persuasive writing strategy were based on those used in prior studies (e.g., Mastropieri et al., 2009, 2010b for persuasive writing; Mason et al., 2009 for quick writing). Quick writing instruction included modeling and providing practice for writing a 10-min response. Preliminary findings indicate that after instruction, significant gains from baseline were observed on all essay measures (PND ¼ 100% from baseline to postinstruction) and on the WJ-III fluency test. Intervention effects that were lower than postinstruction but higher than baseline were noted for maintenance and generalization essay probes administered 2–3 weeks after fluency testing. Researchers note that although instruction was intensive, instructional time/sessions were significantly less than in Mastropieri et al. (2009, 2010b). Students noted the positive benefits of the strategy instruction.

Essay Writing Summary SRSD instruction for essay writing and for essay writing plus quick writing resulted in students’ performance gains in the number of parts, the number of words, and the quality of a persuasive essay. Effects generalized to a standardized measure of writing fluency. For students with severe and multiple disabilities, intensive instruction spread over a considerable

313


amount of instructional time may be required. As in the quick write studies, it is critical to note that the recommended six stages of strategy acquisition and four self-regulation procedures inherent in SRSD instruction were implemented with a high degree of fidelity.

FUTURE DIRECTIONS The literature reviewed in this chapter offers a number of fluency interventions impacting a variety of writing behaviors such as idea generation, story writing, and persuasive writing. The effects of the behavioral fluency interventions show promise and offer a number of future directions for research. There was clear evidence to support the use of SRSD as an instructional approach in which to situate fluency instruction. Future investigations, for example, should closely examine discrete skills, such as idea generation, as in the Albrecht (1981) study, for a fluency threshold for complex writing skills. Fluency aims for idea generation and other basic skills (e.g., sentence writing as measured in the WJ-III fluency subtest) would serve as an important diagnostic and intervention marker. In addition, fluency in sentence production skills ranging from generating simple sentences to more syntactically mature sentences would also benefit from future research. More complex skills such as information writing, supported by evidence-based instructional approaches (e.g., SRSD), practiced to fluent levels, could potentially improve the fluency of writing in this genre of writing and foster generalization across similar writing tasks. Other processes, such as revision skills, all hold value in helping adolescents with disabilities become skillful writers, and should be considered for examination within the context of fluency.

NOTES 1. See Harris et al. (2008) for thorough description of SRSD and POW þ TREE, lesson plans, and materials. Mason, Benedek-Wood, and Valasa (2009) provide a description of quick writing instruction for adolescent writers. 2. During the WJ-III fluency subtest, students are asked to write complete sentences for picture and word cues in 7 min. 3. Note: Lesson structure was developed for adolescents who struggle with writing an effective persuasive paper. The students in the reviewed studies demonstrated the ability to write a few persuasive elements or parts prior to instruction. The lessons may need to be extended and broken down into multiple lessons for younger

314


students, for students with little knowledge of the writing process, or no knowledge of persuasive elements.

ACKNOWLEDGMENTS Support for this research was provided by grant no. R324A070199-07 of the U.S. Department of Education, Institute of Educational Sciences, awarded to The Pennsylvania State University.

REFERENCES Alberto, P. A., & Troutman, A. C. (2009). Applied behavior analysis for teachers (8th ed.). Upper Saddle River, NJ: Merrill. Albrecht, P. (1981). Using precision teaching techniques to encourage creative writing. Journal of Precision Teaching, 2, 18–21. Berninger, V., & Amtmann, D. (2003). Preventing written expression disabilities through early and continuing assessment and intervention for handwriting and/or spelling problems: Research into practice. In: H. L. Swanson, K. Harris & S. Graham (Eds), Handbook of research on learning disabilities (pp. 345–363). New York, NY: Guilford Press. Binder, C. (1996). Behavioral fluency: Evolution of a new paradigm. The Behavior Analyst, 19, 163–197. De La Paz, S. (2001). Teaching writing to students with attention deficit disorder and specific language impairment. The Journal of Educational Research, 95, 37–47. Fisher, D., & Frey, N. (2004). Improving adolescent literacy. Upper Saddle River, NJ: Pearson Prentice Hall. Freeman, G., & Haughton, E. (1993). Handwriting fluency. Journal of Precision Teaching, 10(2), 31–32. Graham, S. (2006a). Writing. In: P. Alexander & P. Winne (Eds), Handbook of educational psychology (pp. 457–477). Mahwah, NJ: Erlbaum. Graham, S. (2006b). Strategy instruction and the teaching of writing: A meta-analysis. In: C.A. MacArthur, S. Graham & J. Fitzgerald (Eds), Handbook of writing research (pp. 187–207). New York, NY: Guilford Press. Graham, S., Berninger, V. W., Abbot, R. D., Abbott, S. P., & Whitaker, D. (1997). Role of mechanics in composing of elementary school students: A new methodological approach. Journal of Educational Psychology, 89, 170–182. Graham, S., & Harris, K. R. (2009). Evidence-based writing practices: Drawing recommendations from multiple sources. British Journal of Educational Psychology, Vol. 6, 95–111. (Monograph series). Graham, S., Harris, K. R., & Mason, L. H. (2005). Improving the writing performance, knowledge, and self-efficacy of struggling young writers: The effects of self-regulated strategy development. Contemporary Educational Psychology, 30, 207–241.


315

Graham, S., & Perin, D. (2007). Writing next: Effective strategies to improve writing of adolescents in middle and high schools – A report to Carnegie Corporation of New York. Washington, DC: Alliance for Excellent Education. Harris, K. R., Graham, S., & Mason, L. H. (2006). Improving the writing performance, knowledge, and motivation of struggling writers in second grade: The effects of self-regulated strategy development. American Educational Research Journal, 42, 295–340. Harris, K. R., Graham, S., Mason, L. H., & Friedlander, B. (2008). Powerful writing strategies for all students. Baltimore, MD: Brooks Publishing Co. Inc. Kameenui, E. J., & Simmons, D. C. (1990). Designing instructional strategies: The prevention of academic learning problems. Englewood Cliffs, NJ: Macmillan. Katsiyannis, A., Zhang, D., Ryan, J. B., & Jones, J. (2007). High-stakes testing and students with disabilities: Challenges and promises. Journal of Disability Policy Studies, 18, 160–167. Kubina, R. M., & Morrison, R. (2000). Fluency in education. Behavior and Social Issues, 10, 83–99. Kubina, R. M., Young, A. E., & Kilwein, M. (2004). Examining an effect of fluency: Application of oral word segmentation and letters sounds for spelling. Learning Disabilities: A Multidisciplinary Journal, 13, 17–23. LaBerge, D., & Samuels, S. J. (1974). Toward a theory of automatic information processing in reading. Cognitive Psychology, 6, 293–323. Mason, L. H. (2009). Effective instruction for written expression. Perspectives on Language and Literacy, 35, 21–24. Mason, L. H., Benedek-Wood, E., & Valasa, L. (2009). Quick writing for students who struggle with writing. Journal of Adolescent and Adult Literacy, 53, 313–322. Mason, L. H., & Graham, S. (2008). Writing instruction for adolescents with learning disabilities: Programs of intervention research. Learning Disabilities Research and Practice, 23, 103–112. Mason, L. H., Harris, K. R., & Graham, S. (2011). Self-regulated strategy development for students with writing difficulties. Theory in Practice, 50, 20–27. Mason, L. H., Kubina, R., & Hoover, T. (2010a). Quick writing skills for low-achieving high school students. Manuscript submitted for publication. Mason, L. H., Kubina, R., Kostewicz, D., Cramer, A., & Datchuk, S. (2010b). Quick writing for struggling middle school students: A regression discontinuity study. Manuscript in preparation. Mason, L. H., Kubina, R., & Taft, R. (2009). Developing quick writing skills of middle school students with disabilities. Journal of Special Education. OnlineFirst, published on October 21, 2009; doi: 10.1177/0022466909350780 Mason, L. H., Kubina, R., Valasa, L., & Cramer, A. (2010c). Evaluating effective writing instruction of adolescent students in an emotional and/or behavioral support setting. Behavioral Disorders, 35, 140–156. Mason, L. H., & Shriner, J. (2008). Self-regulated strategy development instruction for six elementary students with emotional behavioral disorders. Reading and Writing: An Interdisciplinary Journal, 21, 71–93. Mastropieri, M. A., Scruggs, T. E., Cuenca-Sanchez, Y., Irby, N., Mills, S., Mason, L. H., & Kubina, R. (2010a). Persuading students with emotional disabilities to write: Findings from a design experiment. In: T. E. Scruggs & M. A. Mastropieri (Eds), Literacy and learning: Advances in learning and behavioral disabilities (Vol. 23). Bingley, UK: Emerald.

316


Mastropieri, M. A., Scruggs, T. E., Irby, N., Allen Bronaugh, D., Thompson, C., Guckert, M., Leins, P., & Cuenca-Sanchez, Y. (2010b). Writing instruction for diverse middle school learners with emotional and behavioral disabilities. Manuscript in preparation. Mastropieri, M. A., Scruggs, T. E., Mills, S., Irby, N., Cuenca-Sanchez, Y., Bronaugh, D. A., Thompson, C., Guckert, M., & Regan, K. (2009). Teaching students with emotional disabilities to write fluently. Behavioral Disorders, 35, 19–40. McMaster, K., & Espin, C. (2007). Technical features of curriculum-based measurement in writing. The Journal of Special Education, 41, 68–84. Mercer, C. D., & Mercer, A. R. (1998). Teaching students with learning problems (5th ed.). Upper Saddle River, NJ: Merrill. Mitchell, D. (1996). Writing to learn across the curriculum and the English teacher. English Journal, 85(September), 93–97. National Commission on Writing. (2004). Writing: A ticket to work y Or a ticket out: A survey of business leaders. National Institute of Child Health and Human Development. (2000). Report of the National Reading Panel. Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction (Publication No. 00-4769). Washington, DC: U.S. Government Printing Office. National Mathematics Advisory Panel. (2008). Foundations for success: The final report of the National Mathematics Advisory Panel. Washington, DC: U.S. Department of Education. Available at http://www.ed.gov/about/bdscomm/list/mathpanel/report/final-report.pdf. Retrieved on November 11, 2008. National Reading Panel. (2000). Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implication for reading instruction. Washington, DC: National Institute of Child Health and Human Development. Rogers, L., & Graham, S. (2008). A meta-analysis of single subject design writing intervention research. Journal of Educational Psychology, 100, 879–906. Saddler, B., & Graham, S. (2007). The effects of peer-assisted sentence-combining instruction on the writing performance of more and less skilled young writers. Journal of Educational Psychology, 97, 43–54. Scruggs, T. E., Mastropieri, M. A., & Casto, G. (1987). The quantitative synthesis of single subject research: Methodology and validation. Remedial and Special Education, 8, 24–33. Spaulding, J., Haertel, M. W., Seevers, R. L., & Cooper, J. O. (1995). Visual imagery and structure words: Accelerating number of words and number of descriptive words written during free writing. Journal of Precision Teaching and Celeration, 13, 13–24. Taft, R., & Mason, L. H. (2010). Examining effect of writing interventions: Spotlighting results for students with primary disabilities other than learning disabilities. Remedial and Special Education, RASW Online First, published on February 24, 2010; doi:10.1177/ 0741932510362242 Wood, K., & Harmon, M. (2001). Strategies for integrating reading and writing in middle and high school classrooms. Newark, DE: The International Reading Association. Woodcock, R. W., McGrew, K. S., & Mather, N. (2001). Woodcock-Johnson III. Itasca, IL: Riverside Publishing.


317

APPENDIX A. SRSD FOR POW þ TREE LESSON STRUCTURE3 It is important to note that SRSD instruction is recursive rather than linear. In other words, the six stages for strategy acquisition (develop and activate background knowledge, discuss it, model it, memorize it, support it, and independent practice) and four self-regulation procedures (goal setting, selfmonitoring, self-instructions, and self-regulation) were designed to be repeated and revisited based on individual student needs. The following support materials were used in the SRSD for POW (Pick my idea, Organize my notes, Write and say more) þ TREE (Topic sentence, Reasons: Three or more, Explanations: Explain each reason, Ending sentence) studies for quick writing and writing essays: (a) POW þ TREE mnemonic step chart, (b) learning contract, (c) graphic organizer for planning, (d) self-instruction recording sheet, and (e) graphing sheet for recording performance. Develop background knowledge, discuss it, memorize it, goal setting. The purpose of the first lesson was to introduce the POW þ TREE strategy steps and the writing format to be applied (e.g., paragraph or essay). To develop background knowledge, the instructor led a discussion of what good writers do when writing to persuade. Using the mnemonic chart, each POW þ TREE strategy step was described. Students then located the 8 (or 10 when teaching counter arguments) TREE parts in an example paper. In this lesson, the students also examined and identified TREE parts in one of their previously written papers collected during baseline or pretesting. The teacher and students discussed what could have been done to make their persuasive paper better. The teacher and students then established goals to learn POW þ TREE, write persuasive papers with all TREE parts, and signed the learning contract. In this first lesson, and all subsequent lessons as needed, students were provided time to practice memorizing the POW þ TREE steps. Memorize it, model it, goal setting, self-monitoring, self-instruction, selfreinforcement. Following the memorization practice, the instructor cognitively modeled (i.e., modeled while thinking out loud) using the POW þ TREE for writing a persuasive paper. The instructor used all support materials (e.g., graphic organizer, graphing chart) and the four selfregulation procedures while planning and writing. Following the instructor’s modeling, students developed and recorded personal self-instructions to be used while thinking of good ideas (‘‘Relax,’’ ‘‘Ask for help’’), while working (‘‘Use all TREE parts,’’ ‘‘I can write more than 3 reasons’’), and to check

318


work (‘‘Examine for all TREE,’’ ‘‘I got it!’’). The instructor and the students then revised the papers examined in the previous lesson. This was done collaboratively to ensure that the students wrote a revised paper with all TREE parts. Memorize it, support it, goal setting, self-monitoring, self-instruction, selfreinforcement. The instructor and students set goals and collaboratively wrote a persuasive paper in the next lesson. All supporting materials were used, self-instructions were reinforced, and self-monitoring was promoted by collaboratively examining and graphing the number of TREE parts written. The instructor and students self-reinforced for using all TREE parts in their paper. Memorize it, support it, goal setting, self-monitoring, self-instruction, selfreinforcement. By this time, the strategy was memorized. Students planned and wrote a persuasive paper using all support material with teacher guidance. Students were encouraged to set goals, self-monitor, and graph their performance in using all parts of TREE, self-instruct, and selfreinforce. The teacher returned to modeling or collaborative practice if any student was having difficulty. Memorize it, support it, goal setting, self-monitoring, self-instruction, selfreinforcement. Instruction was scaffolded until each student independently implemented all POW þ TREE strategy steps without the teacher, or instructional material support. Students continued to set goals, self-monitor, self-instruct, and self-reinforce for their efforts and success in writing a good persuasive paper. The teacher returned to modeling or collaborative practice if any student was having difficulty. Independent practice. Once a student had demonstrated independence in writing a persuasive paper with all TREE parts, the instructor introduced writing the response in a 10-min time frame.

APPENDIX B. SRSD FOR FLUENCY LESSONS Step one: Establish the students’ background knowledge about and prior experience in writing within a set time frame. Step two: Discuss ways to use time wisely for planning, writing, and revising a response in a set time frame. Revise self-instruction sheet to include things to say when writing, ‘‘I have one minute left, I need to finish this explanation and start on my ending sentence.’’ Step three: Cognitively model how to plan, write, and revise a response in a 10-min time frame. Use a timer and self-instruction prompts to guide the


319

writing process. After writing, model counting the TREE parts and graph performance. Step four: Support the students’ performance in writing within the 10-min time frame. Provide verbal and timer prompts. Encourage the students to use their own self-instructions as verbal prompts. The students should continue to self-monitor using all TREE parts and graph performance. Return to modeling, if needed, to reinforce process. Repeat as needed until the student no longer needs instructor verbal prompting. Step five: Provide the student a minimum of two independent practices in writing within the 10-min time frame with no instructor prompting. The student continues to self-monitor using all TREE parts and graph performance.

ENGAGING YOUTH WITH DISABILITIES IN SCHOOL: BUILDING AND SUSTAINING POSITIVE RELATIONSHIPS Elizabeth Talbott and Lisa S. Cushing ABSTRACT The development of social and emotional competence for children and adolescents occurs in the context of relationships they have with adults and peers. For students with disabilities and those who experience significant behavioral risk, building and sustaining positive relationships with teachers is vital to the development of social competence. In this chapter, we focus on relationships between teachers and students within a developmental framework, examining characteristics of these relationships for children and youth with and without disabilities, and showing how the quality of those relationships can be assessed and improved to foster student engagement in school. The chapter is comprised of three main sections. First, we provide a developmental framework for the development of positive, sustaining relationships with teachers for youth with and without disabilities. Second, we review methods for assessing the quality of those relationships; and third, we describe effective interventions to support sustaining relationships among students with disabilities and their teachers and peers. Assessment and Intervention Advances in Learning and Behavioral Disabilities, Volume 24, 321–339 Copyright r 2011 by Emerald Group Publishing Limited All rights of reproduction in any form reserved ISSN: 0735-004X/doi:10.1108/S0735-004X(2011)0000024014

321

322

ELIZABETH TALBOTT AND LISA S. CUSHING

The engagement of students with disabilities in school is influenced by a broad array of factors, ranging from the immediate social contexts of the home, school, and peer group; to larger social contexts that include interactions among those settings; to the effects of systems that impinge on the life of the child, such as federal and state education policies and how they play out (Bronfenbrenner, 1977, 1979, 1992; Talbott, Fleming, Karabatsos, & Dobria, in press). Furthermore, these factors are nested within each other, adding to the complexity associated with selecting any given context in which to intervene (National Research Council, 2004). Nevertheless, vital to the success of youth in school, and at the heart of the social context are relationships between teachers and students (Hamre & Pianta, 2001; Pianta, 1999; Harry & Klinger, 2006). Such relationships, beginning in the early grades, can have lasting influences (both positive and negative) on student outcomes (Saft & Pianta, 2001; Harry & Klinger, 2006). During the adolescent years, those relationships are no less vital, even as youth report a shift in the relative importance of adult and peer relationships in middle school, with peer relations surpassing adult relations with regard to importance (Lynch & Cicchetti, 1997). What does the school environment mean to the individual student? Relationships experienced by youth in school can act as a mediator between the context of the school and students’ school-related feelings, beliefs, and actions (Roeser & Eccles, 1998). In fact, what schools mean to students can matter more than the school environment itself (Ryan & Grolnick, 1986). Making meaning of the school environment, particularly during the adolescent years depends upon ‘‘a student’s involvement in a web of social relationships’’ that include teachers, family and friends (Cohen & Ball, 1999, as cited by the National Research Council, 2004, p. 17). Building and sustaining positive relationships with teachers for students with disabilities can provide unique challenges. Students with learning disabilities (LD), emotional and behavioral disabilities (EBD) and mild mental retardation (MMR) can be at increased risk of experiencing mental health problems that include both internalizing (i.e., depression, anxiety, and withdrawal) and externalizing problems (i.e., antisocial behavior and delinquency) (Al-Yagon & Mikulncer, 2004Al-Yagon & Mikulincer, 2004; Talbott & Fleming, 2003). Students with disabilities bring their own individual academic, social, and behavioral challenges to the classroom. Within the general education classroom, many students with disabilities, particularly those who display challenging behavior, can experience interactions and instructional contingencies that are significantly different from those of their peers without disabilities (Sutherland, Lewis-Palmer,

Building and Sustaining Positive Relationships

323

Stichter, & Morgan, 2008; Van Acker, Grant, & Henry, 1996). These differences can work against students to make academic and social success difficult, if not impossible, to attain (Sutherland et al., 2008). Both the challenges and the importance of supporting the mental health and successful adjustment of youth with high incidence disabilities cannot be over emphasized (Murray & Pianta, 2007). Building and sustaining positive teacher-student relations for youth with disabilities is vital to supporting their mental health (Murray & Pianta, 2007). In this chapter, we first provide a developmental framework for building and sustaining positive relationships between youth with and without disabilities and their teachers. Second, we review methods for assessing the quality of those relationships; and third, we describe effective interventions to support positive relationships among students with disabilities and their teachers and peers.

A DEVELOPMENTAL FRAMEWORK FOR TEACHER-STUDENT RELATIONSHIPS The development of social competence is derived from and embedded in the relationships children and adolescents have with adults (Pianta, Nimetz, & Bennett, 1997). The study of these relationships across the lifespan is grounded in attachment theory, in the early secure attachment relationships an infant shares with his or her caregivers (Bowlby, 1969/1982). Humans are biologically predisposed to form attachment relationships with primary caregivers (Beckwith, Rozga, & Sigman, 2002). Although these relationships are primarily affective in nature, they serve to shape the child’s thinking about relationships, called a cognitive working model (Bowlby, 1969/1982). The development of attachment relationships requires open, reflective, emotionally attuned and communicative behavior on the part of the caregiver; the adult’s ability to respond to the child depends in part on the child’s temperament and behavioral characteristics (Howe, 2006). That is, children with challenging behavior can tax parents’ ability to be flexible, responsive, and adaptive in their parenting (Beckwith et al., 2002). Yet, early secure attachments to caregivers influence children’s beliefs and expectations about themselves and others, shaping a ‘‘secure readiness to learn,’’ or a desire to deal competently with the environment and form successful relationships with adults (Lynch & Cicchetti, 1997, p. 82).

324


In the context of their relationships with caregivers, ‘‘children acquire the capacity to approach tasks in an organized, confident manner, to seek help when needed, and to use help appropriately’’ (Pianta et al., 1997, p. 265). Such skills do carry over into the classroom context (Pianta et al., 1997). Across development in childhood and adolescence, the ability to form and sustain positive relationships with adults is related to success in school. For example, school-aged children’s feelings of connection to others in school are positively correlated with their feelings of engagement and motivation in school (Connell & Wellborn, 1991). Relationships can have a protective quality as well: in kindergarten, children who were at risk for retention in kindergarten on the basis of standardized test scores were less likely to be retained if they had a positive relationship with a teacher (Pianta & Steinberg, 1992). These same relationships contributed to the development of social skills in the early grades: Pianta and Steinberg (1992) found that children with greater warmth and openness in relationships with teachers in kindergarten had better social skills with peers, tolerance for frustration, and work habits one year later in first grade (Pianta & Steinberg, 1992). Relationships with teachers retain their significance later in childhood as well. Murray and Greenberg (2001) found that students in fifth and sixth grades who had positive relationships with teachers and were bonded to school were more likely to report positive social and emotional adjustment; the opposite pattern of adjustment was the case for students who had poor relationships with teachers and low levels of bonding to school. The early years of adolescence are marked by significant changes in academic and psychological functioning, which influence the development and sustainability of positive relationships with teachers (Roeser & Eccles, 1998). Middle school youth are more likely than elementary aged children to have a disengaged pattern of relatedness with teachers and increasingly secure attachments to peers (Lynch & Cicchetti, 1997). Nevertheless, positive relationships with teachers at this age can serve a protective function. For example, Roeser and Eccles (1998) found that students’ positive relationships with teachers over the course of seventh and eighth grades in middle school predicted diminished anger and school truancy and increases in self esteem over time. Murray and Greenberg (2001) found that affiliation with teachers reported by students without disabilities in the middle grades was a significant, negative predictor of engagement in delinquency and conduct problems, just as self-reported bonds with school were significant, positive predictors of school competence (Murray & Greenberg, 2001). By high school, relationships between teachers and


325

students may change, but are no less vital to students’ development, and particularly to their coming to school, staying in class, staying in school, and graduating (National Research Council, 2004). High schools promote positive development by addressing needs for safety, belonging, respect, power, and accomplishment (National Research Council, 2004).

APPLICATION OF THE DEVELOPMENTAL FRAMEWORK FOR STUDENTS WITH DISABILITIES For students with high incidence disabilities, positive, sustaining relationships at school can be the difference between resilience and risk. The theoretical, developmental framework for the role of relationships across the lifespan is highly relevant for students, both prior and subsequent to their identification with high-incidence disabilities. Students with disabilities can clearly become securely attached to their caregivers, develop cognitive working models of that relationship, and thereby develop a secure readiness to learn (Beckwith et al., 2002). As is the case with all children, the development of attachment relationships depends upon the characteristics of both partners in the relationship (caregiver and child); in the case of the child, the nature of the disability can play a significant role (Howe, 2006). Parents must be sensitive, responsive, and emotionally in tune with their children; they must be able to communicate with their children and correctly interpret their behavior (Beckwith et al., 2002; Howe, 2006). Yet the clarity with which parents can communicate and interpret their children’s efforts to communicate is likely to be substantially influenced by the child’s temperament and behavioral characteristics (Howe, 2006). That said, it is the parent who plays a more important role than the child in shaping the quality of relationships (van IJzendoorn, Goldberg, Kroonenberg, & Frenkel, 1992). Such is the case with teachers, who take over from parents in relationship building when children enter school. In the early grades, if relationships with teachers are positive and close, with minimal levels of conflict, students at risk for special education referral can actually avoid that referral (Pianta, Steinberg, & Rollins, 1995). On the other hand, if teacher-student relationships are problematic in kindergarten, those relationships can contribute to persistent academic and behavior problems through eighth grade (Hamre & Pianta, 2001). Just as having a positive relationship with teachers can promote positive outcomes and

326


resilience, having a negative relationship with teachers can promote risk and exclusion from school (Decker, Dona, & Christenson, 2007). In their study of African-American students in kindergarten through sixth grade who were identified as being at risk for referral to special education for behavior problems, Decker et al. (2007) found that, as general education teachers described relationships of higher quality with their students, those students experienced increases in positive social and behavioral outcomes, as well as engagement in school. Likewise, as students reported higher quality relationships with teachers, they also experienced increases in positive behavioral, engagement, and academic outcomes (Decker et al., 2007). Once identified with high incidence disabilities, patterns of satisfaction and dissatisfaction with teachers begin to emerge. Murray and Greenberg (2001) found that, overall, fifth and sixth grade students with disabilities reported greater dissatisfaction with teachers, poorer bonds with school, and greater perceptions of danger and risk at school than did their peers without disabilities. Furthermore, students with LD, other health impairments, and no disabilities reported more positive affiliation with their teachers than did students with EBD and MR (Murray & Greenberg, 2001). Children and youth who were able to form bonds with their teachers experienced enhanced social and emotional adjustment, whereas those who were not able to form close bonds experienced the opposite (Murray & Greenberg, 2001). Adolescents with high incidence disabilities, particularly those with EBD, can have the greatest difficulty establishing and maintaining positive relations with teachers. The classroom can be a highly aversive environment for both students with EBD and their teachers (Sutherland et al., 2008). Although large, comprehensive high schools in the United States offer a wide range of courses and opportunities for youth, they can also lack a sense of community and the kind of unifying sense of purpose that is vital to promoting students’ positive relationships and sense of belonging (National Research Council, 2004). Yet, Ferguson (2002) found, in a survey of 100,000 high school students, that African-American students in particular responded well to teachers who (from the students’ perspective) cared about their learning. Just as during early childhood, having a supportive relationship with a teacher and active teacher involvement during adolescence can be the difference between students’ dropping out and staying in school – particularly for those students at greatest risk for leaving (Croninger & Lee, 2001). Students with EBD who developed strong caring relationships with teachers were less likely to engage in problem behaviors and displayed greater social competence and engagement in school (Mihalas, Morse, Allsopp, & McHatton, 2009; Murray & Greenberg, 2001).


327

Even in the most challenging of circumstances, teachers and students with disabilities have been able to form strong, positive bonds during the high school years. For example, Murray and Naranjo (2008) found, in interviews with African-American students with learning disabilities who attended high school in a poor neighborhood in Chicago, that relationships with teachers, particularly special educators, played a significant role in their graduation from high school. Students valued particular qualities in their teachers: among them, teachers who got to know and stayed in touch with students’ families; helped them throughout all of their difficult classes; regularly inquired about life at home and at school; and advocated for them with general education teachers (Murray & Naranjo, 2008). Finally, these special education teachers provided effective instruction, according to the students interviewed, and exhibited a powerful presence in the classroom (Murray & Naranjo, 2008). Interestingly, students reported that having a teacher of a different race did not preclude their development of a close relationship (Murray & Naranjo, 1998). Despite their experiences with the risks of living in a poor neighborhood and associated risk at school, all of the students with LD interviewed in this study graduated from high school.

ASSESSMENT OF RELATIONSHIPS Relationships are a shared phenomenon; they provide an interactive context in which children and adolescents learn (Hamre & Pianta, 2005). However, students must attend school in order to become engaged with others. Student attendance, then, represents a simple tool to determine whether or not students are available to begin building relationships. Student perspectives about their relationships with teachers can be as important as the environment of the school itself (Ryan & Grolnick, 1986). The most common means of assessing the quality of relationships is through rating scales, which have targeted key components of relationships from early childhood through adolescence. Pianta’s (2001) Student Teacher Relationship Scale (STRS) represents the only standardized and validated instrument for assessing relationships from the teacher’s perspective for children from early childhood through sixth grade (Decker et al., 2007). The instrument has also been used with children in early adolescence as well (Murray & Zvoch, 2010). This 28-item scale consists of teacher ratings of individual students, and measures relationship patterns on the following subscales: Conflict, Closeness, and Dependency. According to Hamre and Pianta (2001), the Conflict scale consists of 12 items, such as ‘‘This child

328


easily becomes angry at me’’ and ‘‘This child and I always seem to be struggling with each other.’’ The internal consistency of the Conflict scale is strong, with an alpha of .93 (Hamre & Pianta, 2001). The Closeness subscale consists of 12 items, such as ‘‘I share an affectionate, warm relationship with this child,’’ ‘‘If upset, this child will seek comfort from me,’’ and ‘‘This child spontaneously shares information about him/herself.’’ The internal consistency of the Closeness subscale is also strong, with an alpha of .86 (Hamre & Pianta, 2001; Pianta, 1996). Finally, the Dependency subscale consists of five items, such as ‘‘This child is overly dependent on me’’ and ‘‘This child reacts strongly to separation from me’’ (Hamre & Pianta, 2001; Pianta, 1996). The internal consistency of the Dependency subscale is the weakest of the three scales, with an alpha of .68 (Hamre & Pianta, 2001). When given to adolescents, items from the Dependency subscale are removed, with alphas of .90 for the Conflict and Closeness subscales (Murray & Zvoch, 2010). From the young student’s perspective, the Relatedness Scale (Wellborn & Connell, 1987) has strong psychometric characteristics. The Relatedness Scale consists of 17 items that determine students’ perceptions of the emotional quality of their relationships with teachers, and how close they feel to those teachers (Lynch & Cicchetti, 1997). The Relatedness Scale consists of two subscales: Emotional Quality and Psychological ProximitySeeking. The Emotional Quality scale consists of items that assess children’s feelings when they are in the company of teachers, with items such as ‘‘When I am with (name), I feel happy’’ (Lynch & Cicchetti, 1997). The Psychological Proximity Scale consists of items that assess the degree to which children wish they were closer to the relationship figure, with items such as ‘‘I wish (name) understood me better’’ (Lynch & Cicchetti, 1997). The two subscales demonstrate good internal consistency, with alphas ranging from .67 to .83 for Emotional Quality and .83 to .93 for Psychological Proximity-Seeking (Lynch & Cicchetti, 1997). From the adolescent’s perspective, the Inventory of Teacher-Student Relationships, IT-SR (Murray & Zvoch, 2010) was recently developed via adaptation of the Inventory of Parent and Peer Attachments (Armsden & Greenberg, 1987). Murray and Zvoch (2010) field tested the IT-SR with a sample of young adolescents from low income, African-American schools in Chicago. Three scales emerged in the factor analysis. The Communication scale consisted of eight items, such as ‘‘I tell my teacher about my problems and troubles,’’ and ‘‘My teacher understands me.’’ The internal consistency (alpha) of the Communication scale was .89. The Trust scale consisted of five items, such as ‘‘My teacher accepts me as I am,’’ and ‘‘I trust my


329

teacher.’’ The internal consistency (alpha) of the Trust scale was .84. The Alienation scale consisted of four items, such as ‘‘I get upset a lot more than my teacher knows about,’’ and I feel that no one understands me.’’ The internal consistency (alpha) of the Alienation scale was .72. Murray and Zvoch (2010) obtained strong concurrent and predictive validity and reliability for the IT-SR. Obtaining multiple perspectives of teacher-student relationships results in interesting findings. Decker et al. (2007) examined teacher-student relationships of African-American students with a history of behavior problems who were considered at risk for special education referral in kindergarten through sixth grades. They assessed teacher-student relationships from the perspective of both the teacher (using the STRS) and student (using the Relatedness Scale), and then examined the contributions of the different raters’ views to multiple outcomes, including teacher and child ratings of children’s social skills; teacher reports of disciplinary infractions; teacher, student, and observers’ reports of engagement in school; teacher ratings of academic performance; and curriculum-based measures of achievement. Emerging from this study was a complex picture of teacher-student relationships from different perspectives and their association with various student outcomes. From the teacher’s point of view, teacher-student relationships were a significant predictor of teacher ratings of social competence and student engagement for individual students, but not academic achievement. Further, teacher-student relationships were a significant predictor of behavioral referrals and suspensions (Decker et al., 2007). From the student’s point of view, the student-teacher relationship was a significant predictor of behavioral referrals, academic engaged time, and curriculum-based academic measures (Decker et al., 2007). Finally, from the student’s perspective, emotional quality of the relationship with teachers was the largest single predictor of behavioral referrals and academic engaged time (Decker et al., 2007). Assessment of relationships from both teacher and student perspectives is therefore critical, especially as concordance (or lack thereof) in teacher and student ratings can indicate the need for intervention. That is, teachers and students might both report a positive or negative relationship; or they might disagree, with one reporting a positive and the other reporting a negative relationship. Patterns of agreement (or lack thereof) between teachers and students regarding their relationship could have important implications for intervention. In analyses of these patterns of teacher and student perspectives on their relationships, Decker et al. (2007) found that patterns of relationships (concordance versus discordance) were significant

330


predictors of students’ self-reported social competence and engagement; they were significant predictors of teacher reports of students’ social skills and engagement. In particular, teacher and student perspectives about negative aspects of their relationship(s) appear to be important for intervention, with negative features such as conflict more strongly associated with emotional and behavioral adjustment than positive features (Decker et al., 2007; Hamre & Pianta, 2001; Murray & Zvoch, 2010). For students with disabilities, having poor relationships with teachers may be contributing to long-documented patterns of interactions between teachers and students, which frequently consist of low rates of positive teacher attention and praise, negative interactions around academic activities, and fewer opportunities to respond (Sutherland et al., 2008; Van Acker et al., 1996).

INTERVENTIONS TO BUILD AND SUSTAIN RELATIONSHIPS Successful outcomes for students in school are both academic and social. As it turns out, building and sustaining positive relationships between students with and without disabilities and their teachers has effects for a broad range of social and academic, task-oriented skills, such as work completion and accuracy, class preparation, eagerness to learn, and persistence (Anderson, Christenson, Sinclair, & Lehr, 2004; Decker et al., 2007; Hamre & Pianta, 2005). To be most effective, these interventions need to be implemented at multiple levels, including schoolwide and at the level of the individual teacher (Murray & Pianta, 2007). Here we review effective interventions at these two levels, with a focus on interventions that support the everyday classroom social interactions between teachers and students with disabilities. Teachers’ everyday social interactions with students can ameliorate the risk of school failure (Hamre & Pianta, 2005), and keep students coming to school, staying in class, staying in school, and graduating (Mihalas et al., 2009; Murray & Naranjo, 2008; National Research Council, 2004). Murray and Pianta (2007) provide a framework for building and sustaining relationships between teachers and students that accounts for multiple school contexts, including: (a) organizational structures and resources within the school, (b) classroom structures and practices, (c) teacher beliefs, behavior, and actions, and (d) individual, prosocial skills, for teachers as well as students (Murray & Pianta, 2007). Teacher interventions begin with the process of establishing a positive class climate, creating an ethic of


331

cooperation, and developing positive and effective classroom management procedures (Mihalas et al., 2009; O’Connor & Jenkins, 1996; Walker, Colvin, & Ramsey, 1995). From there, teachers are in a powerful position to build relationships with students and foster social and emotional development across the school years (Mihalas et al., 2009).

Schoolwide Interventions Creating a positive and welcoming school climate for students at risk for dropout and students with learning and emotional and behavioral disabilities requires intervention across the school years. One of the more promising and effective schoolwide interventions for building relationships is Check and Connect (Anderson et al., 2004; Lehr, Sinclair, & Christenson, 2004; Sinclair, Christenson, & Thurlow, 2005). Check and Connect is an intervention model designed to ‘‘promote student engagement in school through relationship building, problem solving, and persistence’’ (Anderson et al., 2004, p. 100). Though many social and emotional learning programs have been developed to promote social skills and prosocial behavior, Check and Connect is the only empirically based program with a focus on relationship building (Anderson et al., 2004). Originally designed as a dropout prevention program, Check and Connect was developed to keep middle school students with low attendance rates in school and on track to graduate (Sinclair, Christenson, Elevo, & Hurley, 1998). The procedure grew from collaborations between general education teachers, family members, and community outreach workers, with consultation from special educators, university personnel, and students themselves (Sinclair et al., 1998). Critical to the success of the program in elementary, middle, and high school has been to hire monitors to act as case managers and liaisons between school and home. Monitors are individuals with four year college degrees who are paid as paraprofessionals and hired by the local school district or university (Sinclair et al., 1998). Monitors help keep education salient for students and their families, and identify early signs that students are in trouble, by examining absences, suspensions, failing grades, and the like (Sinclair et al., 1998). Students have described their monitors as individuals who ‘‘stay on my back about coming to school’’ (Sinclair et al., 1998, p. 9). Monitors possess characteristics of persistence; they are willing to work with families and school staff; and they display strong communication and problem solving skills and belief in student success (Sinclair et al., 1998). In many cases and whenever possible,

332


monitors have worked with the same students across several years (Anderson et al., 2004). Check and Connect has proven to be effective at reducing absenteeism and increasing student engagement in school, at the elementary, middle, and high school levels (Anderson et al., 2004; Sinclair et al., 1998). Closeness and quality of relationships were assessed by Anderson et al. (2004) via a measure designed specifically for the Check and Connect project, rather than the STRS or similar measure; nevertheless, the intervention showed strong effects on their measure of relationships, as well as attendance, particularly for elementary students experiencing low levels of risk. For students with learning disabilities in seventh and eighth grades who were also receiving free and reduced lunches, Check and Connect was multifaceted, and consisted of four strategies: (a) the monitoring system described by Anderson et al. (2004); (b) regular feedback to the student about his or her educational progress; (c) ongoing discussions with monitors and teachers about the importance of staying in school, and (d) ongoing problem-solving with monitors and teachers (Sinclair et al., 1998). Furthermore, students experiencing significant risk (i.e., they had been suspended from school) received additional, intensive levels of monitoring as needed (Sinclair et al., 1998). Results revealed that by ninth grade, students who had participated in Check and Connect were significantly more likely than members of a control group to be engaged in school (Sinclair et al., 1998). Adult monitors working in Check and Connect are great resources for teachers. It is not always easy for teachers to be warm and supportive with students who can be angry, disruptive, or rude (Walker et al., 1995). By establishing connections with individual students and their families, monitors serve to re-engage children and adolescents in school, setting the stage for teachers to build on those relationships and focus on students’ academic progress (Anderson et al., 2004). Monitors’ perceptions of their relationships with students have predicted teacher ratings of academic engagement: work completion and accuracy, class preparation, eagerness to learn, and persistence (Anderson et al., 2004). Monitors can help make a teacher’s job easier and more successful. Attendance is a major focus of Check and Connect. Not only is it an easy outcome to measure, it is the first step in student engagement in school. Furthermore, school attendance rate is significantly correlated with students’ identification and placement in special education classes for MR, LD, and EBD (Talbott et al., in press). With increases in attendance for students at greatest risk, fewer students may require special education services.


333

Furthermore, Check and Connect also works for students in the highest risk groups: African American students attending urban high schools who also have emotional and behavioral disabilities (Christenson et al., 2005). Check and Connect with this group was implemented over the course of four and five years, and included an effort to follow students as they dropped in and out of different high schools (Christenson et al., 2005). Check and Connect was multi-faceted in this case, as it had been in the Sinclair et al. (1998) program, with links between case management of individual students via monitors, and teaching problem solving skills (Christenson et al., 2005). In a similar study (without the case management component) to promote teacher-student relationships for students with emotional and behavioral problems in a large urban high school in a low income neighborhood, Murray and Malmgren (2005) found that students who participated in the relationship-building intervention obtained higher grade point averages than members of a control group, but did not show changes in social and emotional adjustment.

Teacher Interventions to Promote Peer Relationships With students attending school with better regularity and building relationships with adults, teachers have the opportunity to focus on improving students’ interactions with their peers. Positive teacher-student relationships can and should support and influence effective peer collaboration. Peer collaboration has a long history of documented success, and is considered to be an effective, evidenced based practice (Institute for Education Sciences, 2007). Peer collaborative interventions promote access to rigorous, relevant learning experiences and afford students opportunities to establish new relationships with non-deviant peers, an especially critical component for youth with emotional and behavioral disabilities (Dodge, Dishion, & Lansford, 2006). Sustained contact with students who do not display patterns of disruptive behavior can serve as a catalyst to reduce frequency of problem behavior. Thus, students who display challenging behavior need multiple opportunities to establish relationships with students who do not have those patterns of behavior (Dodge et al., 2006). According to Panacek and Dunlap (2003), students with EBD are more likely to spend their time with adults in school and time with peers outside of school. Thus, it is imperative that they, and their peers with patterns of disruptive behavior, begin to feel a sense of belonging and membership within the school community with particular emphasis on peers without disabilities.

334


Peer interventions typically use student pairs to teach academic tasks to each other. This provides individual instructional assistance for the student with disabilities, while enhancing curricular contact for the student without disabilities (Kamps et al., 2008). Peer collaboration interventions have proven to be practical and effective strategies within general education settings (Carter, Cushing, Clark, & Kennedy, 2010; Cushing & Kennedy, 1997; Kamps et al., 2008; McMaster, Fuchs, & Fuchs, 2006). These procedures are relatively simple for teachers to implement, require less time and effort than alternative techniques, permit teachers to use their skills more efficiently, and are more feasible for teachers who serve a diverse population of students (Carter, Cushing, Clark, & Kennedy, 2005; Kamps et al., 2008; McMaster et al., 2006). Using peer interventions, academic, social, and behavioral improvements have been noted for students with learning and emotional and behavioral disabilities from elementary through high school (Bowman-Perrott, Greenwood, & Tapia, 2007; BowmanPerrott, 2009; Calhoon & Fuchs, 2003; Fuchs & Fuchs, 2005; Heron, Villareal, Yao, Christianson & Heron, 2006; Heron, Welsch, & Goddard, 2003; Maheady, Mallette & Harper, 2006; Ryan, Reid, & Epstein, 2004; Mastropieri, Spencer, Scruggs, & Talbott, 2000; Sutherland & Synder, 2007). Peer interventions have been used successfully with same age peers (Cushing & Kennedy, 1997; Kamps et al., 2008; McMaster et al., 2006; Sutherland et al., 2008) as well as cross age tutors (Mastropieri et al., 2000; Robinson, Schofield, & Steers-Wentzell, 2005). Assessment of peer interventions must include assessment of relationships, in addition to attendance and academic outcomes.

DISCUSSION For students at risk and students with disabilities, building and sustaining positive relationships with teachers is vital to their coming to school, staying in class, staying in school, and graduating. Students with patterns of disruptive behavior are especially vulnerable to experiencing poor relationships at school; yet these are the very students who can respond so positively to interventions with a relationship focus. In this chapter, we have provided a developmental framework for promoting positive, sustaining relationships with teachers for youth with and without disabilities; reviewed methods for assessing the quality of those relationships; and described effective interventions to support sustaining relationships among students with disabilities and their teachers and peers.


335

Student attendance turns out to be a powerful assessment tool, indicative of important components of a school’s functioning. First, student attendance has the potential to affect the learning environment of an entire school, because school funding is often directly tied to the number of students who regularly attend (Epstein & Sheldon, 2002). Students who attend regularly outperform their peers on standardized tests, thereby contributing to a school’s ranking and success (Lamdin, 1996; Sheldon, 2007). Student attendance can also be indicative of the quality of the classroom context, including the extent to which classes are chaotic or well organized. Likewise, student attendance can reflect the quality of studentteacher relationships (Epstein & Sheldon, 2002). Finally, as we have found in our review of studies that focus on relationship building by first promoting attendance, getting students to school is the first step in building those relationships. Following that, students can learn problem solving and social skills, as well as skills for studying and staying in school. Once students begin attending school on a regular basis, periodic assessment of teacher and student relationships from the perspective of students as well as teachers also appears to be vital. Students and teachers might not always agree about the quality of their relationships; that finding in itself should raise red flags for special education teachers who are focused on improving student engagement. Students’ perspectives on the quality of relationships must be assessed, and individual teachers identified who can build those relationships. Finally, support for teachers in relationship building, especially through the role of case managers or monitors, may help to reduce teacher burnout in an emotionally charged profession. ‘‘Emotional stress and poor emotion management consistently rank as the primary reasons teachers become dissatisfied and leave teaching’’ (Commons, 2010, p. 10). Helping teachers build and sustain positive relationships with students may ultimately support teachers’ mental health as well.

REFERENCES Al-Yagon, M., & Mikulncer, M. (2004). Socioemotional and academic adjustment among children with learning disorders: The mediational role of attachment-based factors. Journal of Special Education, 38, 111–123. Anderson, A. R., Christenson, S. L., Sinclair, M. F., & Lehr, C. A. (2004). Check & Connect: The importance of relationships for promoting engagement with school. Journal of School Psychology, 42, 95–113. Armsden, G. C., & Greenberg, M. T. (1987). The Inventory of Parent and Peer Attachment: Individual differences and their relationship to psychological well-being in adolescence. Journal of Youth and Adolescence, 16, 427–454.

336


Beckwith, L., Rozga, A., & Sigman, M. (2002). Maternal sensitivity and attachment in atypical groups. In: R. V. Kail (Ed.), Advances in child development and behavior (Vol. 30, pp. 231–263). Amsterdam, NL: Academic Press. Bowlby, J. (1969/1982). Attachment and loss, Vol. 1: Attachment. New York: Basic Books. Bowman-Perrott, L. (2009). Classwide peer tutoring: An effective strategy for students with emotional and behavioral disorders. Intervention in School and Clinic, 44, 259–267. Bowman-Perrott, L. J., Greenwood, C. R., & Tapia, Y. (2007). The efficacy of CWPT used in secondary alternative school classrooms with small teacher/pupil ratios and students with emotional and behavioral disorders. Education and Treatment of Children, 30(3), 65–87. Bronfenbrenner, U. (1977). Toward an experimental ecology of human development. American Psychologist, 32, 513–531. Bronfenbrenner, U. (1979). The experimental ecology of human development. Cambridge, MA: Harvard University Press. Bronfenbrenner, U. (1992). Ecological systems theory. In: R. Vasta (Ed.), Six theories of child development: Revised formulations and current issues (pp. 187–249). London: Jessica Kingsley. Calhoon, M. B., & Fuchs, L. (2003). The effects of peer-assisted learning strategies and curriculum-based measurement on the mathematics performance of secondary students with disabilities. Remedial and Special Education, 24(4), 235–245. Carter, E. W., Cushing, L. S., Clark, N. M., & Kennedy, C. H. (2005). Effects of peer support interventions on students’ access to the general curriculum and social interactions. Research and Practice for Persons with Severe Disabilities, 30, 15–25. Cohen, D. K., & Ball, D. L. (1999). Instruction, capacity, and improvement. Philadelphia, PA: Consortium for Policy Research and Education, University of Pennsylvania (CPRE-43). Commons, T. (2010). Effects of elementary teachers’ relationships with students with disabilities on retention decisions. Unpublished doctoral dissertation, University of Illinois at Chicago. Connell, J. P., & Wellborn, J. G. (1991). Competence, autonomy, and relatedness: A motivational analysis of self-system processes. In: M. R. Gunnar & L. A. Stroufe (Eds), Self-processes in development: Minnesota symposium on child psychology (Vol. 23, pp. 43–77). Hillside, N.J.: Erlbaum. Croninger, R. G., & Lee, V. E. (2001). Social capital and dropping out of high school: Benefits to at-risk students of teachers’ support and guidance. Teachers College Record, 103, 548–581. Cushing, L. S., & Kennedy, C. H. (1997). Academic effects on students without disabilities who serve as peer supports for students with disabilities in general education classrooms. Journal of Applied Behavior Analysis, 30, 139–152. Decker, D. M., Dona, D. P., & Christenson, S. L. (2007). Behaviorally at-risk African American students: The importance of student-teacher relationships for student outcomes. Journal of School Psychology, 45, 83–109. Dodge, K. A., Dishion, T. J., & Lansford, J. E. (2006). Deviant peer influences in programs for youth: Problems and solutions. New York: Guildford Press. Epstein, J. L., & Sheldon, S. B. (2002). Present and accounted for: Improving student attendance through family and community involvement. The Journal of Educational Research, 95, 308–318.


337

Ferguson, R. (2002). Who doesn’t meet the eye: Understanding and addressing racial disparities in high achieving suburban schools. Naperville, IL: North Central Regional Educational Laboratory. Fuchs, D., & Fuchs, L. (2005). Peer-assisted learning strategies: Promoting word recognition, fluency, and reading comprehension in young children. The Journal of Special Education, 39(1), 34–44. Hamre, B. K., & Pianta, R. C. (2001). Early teacher-child relationships and trajectory of children’s school outcomes through eighth grade. Child Development, 72, 625–638. Hamre, B. K., & Pianta, R. C. (2005). Can instructional and emotional support in the firstgrade classroom make a difference for children at risk of school failure? Child Development, 76, 949–967. Harry, B., & Klinger, J. (2006). Why are so many minority students in special education? New York, New York: Teachers College Press. Heron, T. E., Welsch, R. G., & Goddard, Y. L. (2003). Applications of tutoring systems in specialized subject areas: An analysis of skills, methodologies, and results. Remedial and Special Education, 22(5), 1–12. Howe, D. (2006). Disabled children, parent-child interaction and attachment. Child and Family Social Work, 11, 95–106. Institute for Education Sciences. (2007). Classwide peer tutoring. Retrieved from /ies.ed.gov/ ncee/wwc/publications/wwcallpublications.aspx/ Kamps, D. M., Greenwood, C., Arreaga-Mayer, C., Veerkamp, M. B., Utley, C., Tapia, Y., Bowman-Perrott, L., & Bannister, H. (2008). The efficacy of classwide peer tutoring in middle schools. Education and Treatment of Children, 31(2), 119–152. Lamdin, D. J. (1996). Evidence of student attendance as an independent variable in education production functions. The Journal of Educational Research, 89, 155–162. Lehr, C. A., Sinclair, M. F., & Christenson, S. L. (2004). Addressing student engagement and truancy prevention during the elementary school years: A replication study of the Check & Connect model. Journal of Education for Students Placed at Risk, 9, 279–301. Lynch, M., & Cicchetti, D. (1997). Children’s relationships with adults and peers: An examination of elementary and junior high students. Journal of School Psychology, 35, 81–99. Maheady, L., Mallette, B., & Harper, G. F. (2006). Four classwide peer tutoring models: Similarities, differences, and implications for research and practice. Reading & Writing Quarterly, 22, 65–89. Mastropieri, M. A., Spencer, V., Scruggs, T. E., & Talbott, E. (2000). Students with disabilities as tutors: An updated research synthesis. In: T. E. Scruggs & M. A. Mastropieri (Eds), Advances in learning and behavioral disabilities (vol. 14, pp. 247–279). Greenwich, CT: JAI Press. McMaster, K., Fuchs, D., & Fuchs, L. S. (2006). Research on peer-assited leaning strategies: The promise and limitations of peer-mediated instruction. Reading & Writing Quarterly, 22, 5–25. Mihalas, S., Morse, W. C., Allsopp, D. H., & McHatton, P. A. (2009). Cultivating caring relationships between teachers and secondary students with emotional and behavioral disorders. Remedial and Special Education, 30, 108–125. Murray, C., & Malmgren, K. (2005). Implementing a teacher-student relationship program in a high-poverty urban school: Effects on social, emotional, and academic adjustment and lessons learned. Journal of School Psychology, 43, 137–152.

338


Murray, C., & Naranjo, J. (2008). Poor, black, learning disabled, and graduating: Aninvestigation into factors and processes associated with school completion among high-risk urban youth. Remedial and Special Education, 29, 145–160. Murray, C., & Pianta, R. C. (2007). The importance of teacher-student relationships for adolescents with high incidence disabilities. Theory into Practice, 46, 105–112. Murray, C., & Zvoch, K. (2010). The Inventory of Teacher-Student Relationships: Factor structure, reliability, and validity among African-American Youth in low-income urban schools. Journal of Early Adolescence, 20, 1–33. National Research Council. (2004). Engaging schools: Fostering high school students’ motivation to learn. Washington, D.C.: National Academies Press. Retrieved from http:// www.nap.edu/catalog/10421.html. O’Connor, R. E., & Jenkins, J. R. (1996). Cooperative learning as an inclusion strategy: A closer look. Exceptionality, 6, 29–51. Pianta, R. C. (1996). Manual and scoring guide for the Student-Teacher Relationship Scale. Charlottesville: University of Virginia. Pianta, R. C. (1999). Enhancing relationships between children and teachers. Washington, D.C.: American Psychological Association. Pianta, R. C. (2001). Student-teacher relationship scale. Lutz, FL: Psychological Assessment Resources Inc. Pianta, R. C., Nimetz, S. L., & Bennett, E. (1997). Mother-child relationships, teacher-child relationships, and school outcomes in preschool and kindergarten. Early Childhood Research Quarterly, 12, 263–280. Pianta, R. C., Steinberg, M., & Rollins, K. (1995). The first two years of school: Teacher-child relationships and deflections in children’s classroom adjustment. Development and Psychopathology, 7, 295–312. Robinson, D. R., Schofield, J. W., & Steers-Wentzell, K. L. (2005). Peer and cross-age tutoring in math: Outcomes and their design implications. Education and Psychology Review, 17, 327–362. Roeser, R. W., & Eccles, J. S. (1998). Adolescents’ perceptions of middle school: Relation to longitudinal changes in academic and psychological adjustment. Journal of Research on Adolescence, 8, 123–158. Ryan, J. B., Reid, R., & Epstein, M. H. (2004). Peer-mediated interventions studies on academic achievement for students with EBD: A review. Remedial and Special Education, 25, 330–341. Ryan, R. M., & Grolnick, W. (1986). Origins and pawns in the classroom: Self-report and projective assessments of individual differences in children’s perceptions. Journal of Personality and Social Psychology, 50, 550–558. Saft, E. W., & Pianta, R. C. (2001). Teachers’ perceptions of their relationships with students: Effects of child age, gender, and ethnicity of teachers and children. School Psychology Quarterly, 16, 125–141. Sheldon, S. B. (2007). Improving student attendance with school, family, and community partnerships. Journal of Educational Research, 100, 267–275. Sinclair, M. F., Christenson, S. L., Elevo, D. L., & Hurley, C. M. (1998). Dropout prevention for youth with disabilities: Efficacy of a sustained school engagement procedure. Exceptional Children, 65, 7–21. Sinclair, M. F., Christenson, S. L., & Thurlow, M. L. (2005). Promoting school completion of urban secondary youth with emotional or behavioral disabilities. Exceptional Children, 71, 465–482.


339

Sutherland, K. S., Lewis-Palmer, T., Stichter, J., & Morgan, P. L. (2008). Examining the influence of teacher behavior and classroom context on the behavioral and academic outcomes for students with emotional or behavioral disorders. Journal of Special Education, 41, 223–233. Sutherland, K. S., & Synder, A. (2007). Effects of reciprocal peer tutoring and self-graphing on reading fluency and classroom behavior of middle school students with emotional or behavioral disorders. Journal of Emotional and Behavioral Disorders, 15(2), 103–118. Talbott, E., & Fleming, J. (2003). The role of social contexts and special education in the mental health problems of urban adolescents. Journal of Special Education, 37(2), 139–181. Talbott, E., Fleming, J., Karabatsos, G., & Dobria, L. (in press). Making sense of minority student classification into special education: School context matters. International Journal of Special Education. Retrieved from http://www.internationaljournalofspecialeducation.com/articles.cfm Van Acker, R., Grant, S. H., & Henry, D. (1996). Teacher and student behavior as a function of risk for aggression. Education and Treatment of Children, 19, 316–334. van IJzendoorn, M. H., Goldberg, S., Kroonenberg, P. M., & Frenkel, O. J. (1992). The relative effects of maternal and child problems on the quality of attachment: A meta-analysis of attachment in clinical samples. Child Development, 63, 840–858. Walker, H. M., Colvin, G., & Ramsey, E. (1995). Antisocial behavior in school: Strategies and best practices. Belmont, CA: Brooks Cole. Wellborn, J. G., & Connell, J. P. (1987). Rochester assessment package for children. Rochester, NY: University of Rochester.

IMPROVING INTERVENTION EFFECTIVENESS WITH UNIVERSITY–PUBLIC SCHOOL COHORT PARTNERSHIPS Margo A. Mastropieri, Thomas E. Scruggs, Nicole Conners, Mary Kealy, Nancy Morrison, Tina Diamond and Terry Werner ABSTRACT Findings are presented from a cohort program between a university and local school districts to prepare special-education personnel. Participants (N ¼ 164), who were from 15 different cohorts, responded to an online survey. Follow-up face-to-face interviews were conducted with a representative sample of 29 participants. Perceived benefits included improved social-emotional and academic support; improved pedagogical skills, including academic and behavioral strategies, collaboration, and feeling more acculturated to the school climate; and improved logistical and financial supports. Implications for use of cohorts in the preparation of special-education personnel are discussed.


341

342

MARGO A. MASTROPIERI ET AL.

Serious shortages of special-education teacher create challenges for both universities and school districts (Sindelar, McCray, Kiely, & Kamman, 2008). This was the impetus for the creation of a partnership between a university and local school districts. The goal of this particular partnership was to facilitate the timely preparation of conditionally licensed specialeducation personnel, while maintaining the integrity of the university-based programs. The development of cohorts within school districts was chosen as the optimal method for this university–district partnership program. A Holmes Group report, Tomorrow’s Schools of Education (The Holmes Group, 1995), suggested that teacher education programs should ‘‘organize its students into what we call ‘cohorts,’ the members of each cohort journeying together along a common path of professional learning and socialization that leads to lifelong personal and professional growth and development’’ (p. 50). In cohort models, students take classes together, and participate in formal and informal activities as a unit, although considerable variability has been noted (Mastropieri, Morrison, Scruggs, Bowdey, & Werner, 2008). Cohort programs can potentially provide students a more in-depth opportunity for studying and learning as part of a collaborative group of peers and mentoring with faculty, which purportedly facilitates professional learning and socialization in the teaching profession (The Holmes Group, 1995; see also Bullough, 2001). Since the 1995 recommendation, many variations of cohorts have been reported. A major characteristic of most cohort programs involves the enrollment of all students in common classes or programs. Most cohort programs require students to enroll in the same section of all classes in the major area of study, while some cohorts have students enrolled in some, but not all, of the same classes. Some cohorts are restricted in size with groups ranging from 10 to 25 individuals; however, larger sized cohorts have also been reported (e.g., Dinsmore & Wenger, 2006). One major assumption of cohorts is the development of a sense of community and common bond among students enrolled together (The Holmes Group, 1995). When this happens, cohorts provide students with a structure for developing a community of learners, in which learning is encouraged, internal supports exist, and lifelong professional learning and friendships begin. Some research has supported the use of cohorts in teacher preparation programs. For example, Graber (1996) reported that cohorts enhanced the preparation of teachers through the reliance on peers and positive effect on teacher beliefs during an undergraduate cohort preparation program. Additional evidence has suggested that collaboration and communication skills can be improved during cohort programs (e.g., Clarke, Erickson, Collins, & Phelan 2005). Collaboration skills are important to all

Partnership Cohorts

343

professional development but crucial to the development of special-education teachers. Recent research has also indicated that students gain additional academic and social-emotional support by participating in cohorts (Beck & Kosnik, 2001; Brownell, Yeager, Sindelar, vanHover, & Riley, 2004; Bullough, 2001). In a smaller number of studies, outcomes for students in cohorts have been compared to those of students in more traditional programs. In evaluations of both general education (Connor & Killmer, 2001) and special education (Scruggs & Mastropieri, 1993), students enrolled in cohort programs were evaluated as more effective teachers by cooperating teachers and university supervisors. Others have identified positive social emotional benefits associated with cohorts. Tietel (1997) described how students enrolled in a cohort program developed mutual trust with one another. This cohort promoted student comfort and a willingness to take risks not seen in traditional programs (Tietel, 1997). Additionally, Ross, Stafford, Church-Pupke, and Bondy (2006) described the development of a strong sense of belonging as a result of enrollment in cohort programs. Mastropieri et al. (2008) reviewed and summarized findings from 24 studies that examined the use of cohorts in the preparation of school personnel, of which only 7 examined special-education programs. Most studies employed small sample sizes (e.g., Ross et al., 2006, had a sample size of 25, while Brownell et al., 2004, had a sample size of 3) and reported on perceptions of participants and instructors on the value of their cohort program. Participants reported benefits including social emotional and peer supports, academic growth and support, and support in transitioning to a culture of teaching. For example, social-emotional support among peers was identified in the Tietel (1997) investigation, where participants in the cohort program described a ‘‘family’’ atmosphere, where information was shared as ‘‘friends and family’’ (p. 69). Academic support was reported by participants in a number of investigations, including those in the Ross et al. (2001) investigation, who reported that cohort peers helped each other stay motivated and keep-on task. In several studies (e.g., Beck & Kosnick, 2001; Dinsmore & Wenger, 2006), participants reported that they learned the importance of building a community for learning within the cohort. However, some studies (e.g., Maher, 2005), reported a number of challenges, including the development of cliques and interpersonal conflicts that negatively impacted the functioning of the cohort and conflicts between students and faculty (Sapon-Shevin & Chandler-Olcott, 2001). Although a number of investigations have been completed on cohort programs, several limitations in these studies suggest the need for additional

344


research. For example, in the area of special education, most identified studies either did not study cohorts as the primary purpose of the research (Edgar & Pair, 2005; Esposito & Lal, 2005; Scruggs & Mastropieri, 1993), included very small samples of participants (Jorissen, 2002; Ross et al., 2006), or reported unanticipated, negative outcomes (Sapon-Shevin & Chandler-Olcott, 2001). The present investigation was intended to contribute to existing literature by (a) focusing primarily on specialeducation cohort programs, (b) employing a larger sample, and (d) collecting both survey and individual interview data.

METHOD Participants Individuals (N ¼ 229) who had been enrolled in or were currently enrolled in one of 15 separate cohorts from two school districts were invited to participate. All participants were fully licensed as program completers or conditionally licensed in special education who all held full-time specialeducation teaching, or full-time special-education instructional assistants (IAs) positions in one of the two school districts and were all enrolled (or had been) in the university–district cohort program. Only 8.5% of the respondents were IAs, while the remaining 91.5% were special-education teachers. All participants who completed the online survey were asked to volunteer for a more in-depth interview and a representative sample (N ¼ 29) of volunteers was contacted for follow-up face-to-face interviews. Survey Instrument A larger survey instrument was developed by members of the research team from the university and the school districts. The items were designed based on critical components for personnel preparation in special education, including the elements related to the 10 Council for Exceptional Children Standards for teacher accreditation, for cohort programs in education, and for basic program evaluation connected with individual courses and the cohort program as a whole. Both open-ended and forced-choice items were included in six major sections. The first section contained 24 items, required completion of the human subjects consent and participant demographic data. The second section of 18 items addressed cohort-specific content. The third section of 22 items addressed required special-education licensure

345

Partnership Cohorts

coursework. The fourth section of 24 items covered supervised fieldwork and internships. The fifth section of 16 items addressed teaching competence. The final section addressed overall benefits, suggestions for changes, and requests for follow-up interviews and observations. For example, open items in the cohort-specific section included: ‘‘Please define what being part of a cohort meant for you,’’ ‘‘What types of supports were most valuable to you?,’’ and ‘‘How could we enhance courses and field experiences for you throughout the cohort.’’ A forced-choice item in the same section consisted of: ‘‘How valuable was the cohort experience to you’’ (not at all valuable, somewhat valuable, or very valuable). After pilot testing and revisions, the final survey was located on a secure web-based server and participants were provided a web-linked address for survey access. Reliability of the measure according to Cronbach’s alpha, was 0.974. Data presented and analyzed here include responses from parts 1, 2, and 6 (demographic data, cohort information, overall benefits, and suggestions for program changes). Data from parts 3, 4, and 5 did not address any cohort experiences, but more general program evaluation and were only used for program evaluation efforts.

Follow-up Interview Follow-up interviews were designed to provide an integrity check on survey responses and probe for additional information on benefits and challenges associated with the cohort program. Questions of relevance to the cohort experience included: ‘‘Why did you join the district university cohort?,’’ ‘‘What did you like about the cohort program?,’’ ‘‘What would you change about the cohort program?,’’ ‘‘Describe your greatest challenges in teaching,’’ ‘‘Has the cohort program been helpful in addressing these challenges?,’’ ‘‘Have you been able to take instructional strategies [learned in the cohort program] and apply them to the students you work with?,’’ ‘‘Provide some examples [of applying instructional strategies],’’ and ‘‘After completing the cohort program, do you feel more qualified to teach a variety of special-education classes and students?’’

Procedure Participants included all individuals from 15 cohort programs who were either currently enrolled in or had recently completed the cohort program

346


from two school districts. Once human subjects permission was approved by the university and each school district, all individuals were sent emails inviting them to respond to a web-based survey. Each individual was provided the survey link and a unique ID number to be used when accessing the survey. Email reminders were sent to those who had not responded at three five-week intervals. A representative sample of individuals who volunteered for the follow-up interview was contacted by email and asked to submit available times either before, during, or after school for the faceto-face interviews. Once the interview was scheduled, one of three trained interviewers met for approximately one hour with each individual. Meetings were typically scheduled in the participant’s school.

DATA ANALYSES Two types of data analytic procedures were employed in the present investigation. First, online survey data were downloaded from the web server, imported in excel and then into either SPSS or word-processing programs. Quantitative analyses were conducted with those survey items employing forced-choice responses and demographic data. Open-ended survey items were then imported in a word-processing program. Follow-up interviews were transcribed into word-processing programs and then qualitative data were imported into NVIVO for qualitative analyses. During the analyses, a team of researchers met to discuss findings and assisted with refining codes, categories, and ultimate themes. Some member checks occurred during which a small sample of cohort students was presented with findings and asked for input. All responses were read, re-read, coded, and grouped by themes, and confirmations and disconfirmations were examined (e.g., LeCompte & Preissle, 1993). An inductive approach was employed, using procedures described by Creswell (2008). During the open coding process, initial categories were generated across all data sources and included, for example, support; friendship; learning from peers; developing a sense of community; caring for peers; understanding shared demands of working; enrolling in college and having a family; academic support; financial support; camaraderie; scheduling; academic strategies in the area of reading and reading comprehension, math, study skills; classroom and behavior management; assessment; use of curriculum based measurement; knowledge of the laws and characteristics of individuals with disabilities; collaboration; improved experiences in classroom teaching; learning about IEPs; understanding the

347

Partnership Cohorts

paperwork demands of the district; working with coteachers and other related services personnel; working with families; and managing time. These coding categories were mutually agreed upon by all research team members. Coding decisions were discussed until all disagreements were resolved. During the coding of these categories, confirmations and disconfirmations were noted. Relationships among initial categories were examined during an axial coding phase. During this phase we systematically related the initially generated categories and concepts. We noted similarities and differences, and categories were refined and collapsed into more common themes of support, including social-emotional and academic; pedagogical-teaching, including academic and behavioral strategies, collaboration, and school and district acculturation; and logistical, including financial, scheduling, and tuition. Following this analysis, final themes of social-emotional and academic support, collaboration, including school acculturation, pedagogical skills, and logistical–financial support emerged. Confirmations and disconfirmations were accounted for, and all data were examined with reference to those respondents who were program completers versus those still enrolled in the program.

RESULTS Final Sample The final sample included 164 respondents out of 229 (72% response rate), who had been teaching for an average of 3.28 years (SD ¼ 2.1). Of that total number of respondents, 52.4% (n ¼ 86) had completed the cohort programs, while 47.5% (n ¼ 75) were still enrolled in the special-education cohort programs. Three respondents had withdrawn from the program due to financial reasons or relocation to another part of the country. The sample predominantly included females (n ¼ 132 females), who ranged in age from 24 to 67 with a mean age of 41 (SD ¼ 11.6). Most of the sample represented one large school district (81.1% from one district). The final samples’ racial and ethnic backgrounds represented 81.7% white, 7.9% African American, 3.7% Hispanic, 3.0% Asian American, and 3.7% mixed and other racial/ ethnic backgrounds. Twenty-nine individuals with a mean age of 38.8 years of whom 22 were females who were representatives of the larger sample were interviewed.

348


Educational Background Over 77.4% of the sample reported having bachelors degrees, 16% reported having bachelors and masters degrees, while 6.6% reported other combinations of degrees with bachelor degrees (e.g., Juris Doctor (JD), two MBAs). Undergraduate majors reported included psychology, sociology, child development (21.1%), history and political science (14.6%), English (9.1%), science (9%), business (8.4%), recreation, physical education (7.8%), art (6%), elementary education (5.5%), math or accounting (3.2%), secondary education (2.4%), or other areas including religion, prelaw, or foreign languages (12.9%).

Previous Employment Background Respondents reported a wide range of previous employment history, including business/retail/marketing, general education teachers, retired military, ministry, and other experiences. In addition, many respondents reported previous experiences with children such as teaching, substitute teaching, camp counselors, having siblings, and/or children with disabilities. Fifty-seven percent reported IA experiences in special-education settings, 17.1% in special and general education settings, with the remainder in general education settings.

Current Teaching Assignments The sample reported a range of current special-education teaching positions. For the 91.5% who were special-education teachers, grade levels for current positions included early childhood (7%), elementary (34%), middle school (24%), high school (31%), and vocational-career (3%). The classroom setting teaching assignments included self-contained (42.1%), full-time inclusion and coteaching (16%), resource rooms (3%), combinations of resource rooms, self-contained, inclusion and coteaching (34%), and itinerant and career education (5%). A range of student disability classifications was reported in these teaching assignments. Only 18% of the sample reported teaching combinations of students with learning disabilities and emotional disabilities, while over 70% reported teaching combinations of more than two disability classification areas, including students with learning disabilities, emotional disabilities, autism, developmental delays, mental retardation, other health impairments including visual impairments and hearing impairments.

349

Partnership Cohorts

Cohort Feedback Feedback on the cohort programs was overwhelmingly positive. Ninetyeight percent of the respondents reported positive attitudes toward participation in a cohort and this particular cohort experience. Overall, there was a remarkable similarity in respondents across 15 different cohorts over several years of implementation. Respondents reported that participation in the cohort provided additional support from faculty and peers over and above the level of support typically received in traditional university programs. Reports were highly consistent across both program completers and those still enrolled in the program, for enjoying the camaraderie and collaboration associated with being in a cohort for all of the classes within their program. Similarly, responses were highly consistent across younger and older participants. Some differences were obtained, however, for respondents from the initial cohorts versus all subsequent cohorts. The themes that emerged surrounding the benefits included social-emotional support, academic support, pedagogical skills, acculturation in the schools, and logistical–financial support. Each area is briefly described next.

Social-Emotional Support and Mature Learners Over 90% of the respondents indicated that they received social-emotional support as part of a cohort, and no disconfirmations were reported (e.g., respondents specifically mentioned a lack of social-emotional support). In the present example, since members of these cohorts were more mature learners, with a mean age of 41, and all were teaching with conditional licenses or working as IAs in challenging classroom situations, the socialemotional support appeared critical. One social-emotional support was identified as feeling part of a community of learners with both personal and academic benefits. The social-emotional supports identified were described as critical for sustaining enrollment in the program. Some respondents attributed the social-emotional support for assisting them with program completion. Many identified the network of peers as a major influence on their personal and professional lives. All respondents perceived their peers as colleagues who were enrolling in this journey together. Responses from more mature students suggested that their needs may have been more appropriately met in these cohort programs. These students repeatedly indicated they valued the social-emotional support available to them in the

350


cohort, and related the value of the support to their status as mature learners. For example, one comparatively more mature student wrote: As someone who’d completed my bachelor’s degree 18 years previously and had not been back to school since the cohort provided a supportive environment in which to delve back into coursework. Over the course of completing the coursework; the cohort provided me with life long friends and colleagues with whom I continue to share work and personal ties. I truly appreciated being able to complete the program with the same core group of people. I would not have been as successful taking courses on my own with random people. We all hit various hard spots between both work and personal lives and I relied on many of the members of my group to see me through.

The feeling of support from peers persisted not only throughout the cohort, but extending after program completion as written by one respondent: The cohort provided the support network to allow each of us to not only succeed in the program and earn our degree and teaching certificates; but is a support network that we continue to rely on as we build our careers in [district]. E-mails continue to fly between cohort members as we share concerns; ask advice and network for each other.

Another participant also highlighted in her written comments the enduring nature of the relationships established during the cohort program by stating, ‘‘I still get together with these people once-a-month over a drink and dinner; we have become a family of special educators.’’ Other respondents identified the caring aspect of cohorts as valuable. This common theme is represented in this respondent’s written comment: ‘‘First and foremost, it means being with colleagues who care.’’ Reports from program completers and those still enrolled in the program were highly consistent with respect to the finding of social-emotional support from the cohorts. In addition, although the cohorts seemed to be particularly helpful for more mature learners, similar responses were found across all cohorts and ages of participants. Reports from the face-to-face interviews also confirmed the finding of social emotional benefits. Improved Collaboration and Communication Seventy percent of the respondents reported feeling that the cohort became a community in which they all learned and grew together; no respondent disagreed with this perception. Collaboration skills were enhanced through the learning and growing within this community of learners. The majority of respondents revealed that the collaboration developed within each individual cohort. For example, one respondent reported on the survey:

Partnership Cohorts

351

‘‘Being a part of a cohort was a wonderful experience. It provided me with a support system of people that I could collaborate with.’’ Another respondent highlighted how this cohort experience assisted with not only the program and teaching issues, but also life issues and wrote: We have offered support for each other; bounced ideas off each other; supported each other when we tried to change job assignments; struggled through private and personal experiences together (marriages; babies born etc.)y For me, the comfort level and shared experiences will forever bond me to this COHORT group.

The common theme of camaraderie and collaboration was identified by many respondents as very important. One typical written response was, ‘‘Being a part of a cohort to me meant maintaining focus throughout y the program with the same group of people; and managing to work together collaboratively to help one another learn and accomplish a task at hand.’’ The collaborative support network provided by the cohort provided a network of consultants for participants to rely on. This was acknowledged by many respondents, for example, a representative written comment was: ‘‘Being part of a cohort y meant setting up a solid network of teachers/ colleagues that I could consult with questions regarding the cohort program as well as teaching scenarios.’’ Another individual added in the follow-up interview, ‘‘Keeping us together as a group/cohort was great because we could talk about our varied experiences. We were a tight-knit group and were able to develop helpful networks and friendships that extended beyond school.’’ Various aspects of collaboration were identified as important to members. For example, one respondent identified a valuable skill of learning ‘‘how to communicate with others – learning from others’ experiences.’’ Another typical response stated, ‘‘The cohort is an opportunity for teachers to learn from one another. Collaborative conversations allow one to feel there are others in the same struggling/triumphant position. It helps to create a safety net.’’ Another emphasized in the follow-up interview the importance of ‘‘y managing to work together collaboratively to help one another learn and accomplish a task at hand.’’ Reports across the various subgroups within this sample (e.g., ages, completers vs. those still enrolled and cohort number) confirmed the positive benefits of collaboration and increased communication from cohort participation. Similar findings were also confirmed during the follow-up faceto-face interviews with participants.

352


Academic Benefits The majority of respondents (61%) commented on the academic benefits associated with being part of the cohort program, with no disconfirmations. Respondents indicated that they had easier and better access to faculty who taught in the cohort programs. They reported that because they saw the same faculty over time during the cohort experience, they felt more connected with that faculty than when they had experiences being enrolled in a more traditional university. Further, they reported that this type of experience provided them with additional assistance on assignments and assistance with issues in their current teaching positions. It was also reported that peers within each cohort were frequently very supportive with assistance with one another on academic assignments. Similar findings were obtained across individual cohort membership, age of participants, and whether or not participants had completed the program. One respondent’s written response suggested an interaction of socialemotional benefits with academic and classroom-teaching benefits: Being part of a cohort means being able to take classes with a familiar group of people working within the same school system as myself. This has enabled me to use the knowledge learned in a more meaningful way and to get to know various different teachers and staff within my school system. This has been a big help to be able to share problems and solutions; ideas, etc., that are more relevant to my job within the system in which I work. Communication is easier when questions about course work arise and valuable friendships and professional relationships have developed from my participation in the cohort.

Another wrote favorably on the ‘‘academic support of day-to-day challenges as well as friendships on which to hang onto for support.’’ An individual who participated in the follow-up interviews added: ‘‘The coursework y provided me with the confidence to continue on with what I am doing.’’ Other individuals who were interviewed added that content from the courses assisted them with strategies for teaching. For example, one interviewee said: ‘‘In the elementary strategies class I took, we were given a ‘toolbox’ of strategies to use with elementary students.’’ Another identified content learned from the research class as beneficial in the following way: ‘‘I used lessons from my teacher research classy I did a project on helping a struggling 6th grade student learn math.’’ Many individuals combined their comments on academic and socialemotional supports provided by the cohort experiences. Apparently, many felt this type of combined support enable them to preserver through more challenging times. These findings were seen across all ages of participants as

Partnership Cohorts

353

well as program completers and those still enrolled in the program, but seen more frequently with the more mature older participants. Follow-up interviews confirmed such findings as well. Pedagogical Benefits The vast majority of respondents reported that their classroom teaching was greatly improved from participation in the cohort program. Moreover, they reported a sense of being more acculturated to the school climate, including rules and procedures, including both written and unwritten district policies. For example, respondents reported that they frequently shared instructional issues encountered during their classroom teaching with their peers and university cohort faculty. All participants reported that they learned about strategies and how to teach them during their cohort classes, but more importantly they translated that information into classroom applications with students. This may have been a major advantage of being part of these cohorts in which all participants worked full time in the schools, because all content presented during evening classes could be applied during the next day’s classes with students with disabilities. This ability to try-out and fieldtest immediately what university professors present may have had a greater impact on the ability to translate strategic instruction to practice. Moreover, being part of the cohort enabled students to take their field-test results back to the cohort class and discuss how and why strategic instruction either facilitated or did not facilitate learning on the part of their students. For example, all individuals interviewed reported learning and applying strategic instruction with students in their classes. In particular, many respondents reported learning about how to use graphic organizers to facilitate learning and then proceeded to provide follow-up examples of how they applied the use of these strategies during instruction. Many individuals reported directly using graphic organizers for teaching as a result of what was presented during their cohort classes. One representative’s comment was: I have used graphic organizers for my content and have attempted to incorporate the use of technology in to my lessons. What I developed in [my cohort] class last night, for example, [is] what I am using today in my class [and] was what I developed last night.

Another respondent stated, ‘‘I also use a lot of graphic organizers for writing when I teach English,’’ while a third participant reported integrating graphic organizers with technology during teaching, ‘‘I’m using graphic organizers more this year – they’re done on the computer.’’

354


Several individuals reported on the successful application of direct instruction and behavior management skills. For example, one participant stated she learned in her cohort classes that ‘‘All aspects of instruction – modeling and guided practice and proximity – has helped me to be successful.’’ Another representative comment referred to applying behavior management skills learned in the cohort classes: ‘‘I am also in the process of adjusting behaviors and tweaking my behavior management plans. I am working on several individual student behavior plans.’’ Many participants reported learning in their cohort classes about how to differentiate instruction for students with disabilities and apply those skills. For example, one stated: ‘‘I have learned about differentiation of instruction and how to address the different disabilities of students.’’ Another participant discussed how she applied skills about differentiation to her teaching in a more global way: I have learned a lot about differentiation and developing a toolbox [of strategies] to address individual needs. My ‘‘toolbox development’’ allows me to pull from numerous strategies for instructional purposes. My [university classroom] knowledge supports me when I adjust and monitor instruction from one day to the next. It provides me with a lot of flexibility when I teach.

Many respondents discussed instructional strategies they had learned for promoting literacy during cohort classes. For example, one participant stated, I learned about the cloze procedure, small group reading instruction, read aloud strategies, and activating prior knowledge of students. Now I understand more fully why teachers do these things in the learning process

Much of the learning in pedagogical skills reported by participants appear to be associated more with the program itself, including scope and sequence of coursework, rather than the impact of the cohort format for presenting this information, and to that extent, may have been similar to the on-campus program. However, the cohort format allowed participants to learn in school, and immediately apply newly acquired knowledge. Participants also reported general improvements in teaching. They reported faculty and peers were especially supportive in providing feedback and suggestions when sharing classroom-related issues. Respondents indicated sharing successful and unsuccessful classroom experiences, method, and materials. One respondent wrote, for example: ‘‘Being with other district teachers and IAs [instructional assistants] who are involved in Special Education gave me great insight into their classes and experiences.’’ Another respondent wrote, ‘‘I enjoyed working with teachers from [district] and taking all of the classes together. We formed a support network

Partnership Cohorts

355

that helped me get through my first years of teaching.’’ Another respondent described the classroom-teaching benefits as follows: We learned a lot from each others’ teaching assignments and what we shared about them. I feel that if I ever wanted to change my [classroom teaching] assignment that I have a network of friends and colleagues who would be able to steer me to openings in their schools; since they know how I work. And I would be able to do the same for them.

Another respondent acknowledged the importance of input of ‘‘other students who were able to show me the ins and outs of areas that I had no experience in teaching.’’ However, another respondent also underlined the importance of ‘‘meeting and talking to other people in my cohort that are teaching in similar settings and comparing notes.’’ One respondent summarized by stating her appreciation of learning ‘‘y realworld knowledge from interacting with teachers who were in the field. This was the most structured and professional ‘professional learning community’ provided to me.’’ Consistency was obtained in responses across variations of respondents (e.g., completers vs. still enrolled; age and cohort number). During follow-up interviews, participants were asked if after completing the cohort program, they felt more qualified to teach a variety of special-education classes and students. Respondents were overwhelmingly positive in their responses describing feeling more confident to teach, learning and applying more strategy instruction, and understanding the research and theory underlying the practice. Several representative responses include the following: ‘‘Yes, I am more confident about teaching students with disabilities. I feel more comfortable with the techniques and strategies to teach and help. I am able to provide more accommodations now than before. I would recommend the program to anyone needing initial special education licensure.’’ ‘‘Definitely, y now I can explain why I am doing the things that I do more articulately. I also like the research behind what we practice. The coursework so far has provided me with confidence to continue on with what I am doing [in the classroom].’’ Finally, many respondents in the follow-up interviews identified specific classroom-teaching challenges and were asked whether cohort experience assisted them. Challenges included classroom and behavior management, time management, using differentiated instruction, students’ lack of motivation, and handling so many types of students of differing disabilities. Follow-up interviews revealed that these individuals felt comfortable indicating that the cohort experiences did indeed help them address any

356


idiosyncratic challenges. For example, one participant indicated that classroom and behavior management was particularly challenging for her, but that the cohort experience, in fact, facilitated her abilities to better handle such situations in the future by stating: Yes, the classroom management classes made me think about how to manage my own classroom. I have to develop my own teaching style in terms of my management, like three easy rules to follow.

One participant summarized the cohort experiences as follows: The cohort program has helped me a lot. The collaboration class was a great class. It helped me answer questions of Why am I doing this?; working with ethics, values, working with families, paraprofessionals, etcy . Working and being in the cohort has provided an opportunity for me to apply things that I have just learned. It has been a real life opportunity to apply course content/theory to ‘‘real life’’ situations. We were also a [cohort] group that came with some background knowledge. We were older students and tended to be more cynical and critical than younger, [university] graduate students.

The majority of these perceptions represent a consensus that the cohort program did meet a need for these students in helping prepare them for fulltime special-education teaching positions. Moreover, it appeared that the value-added components of the program were seen as additional benefits for participants. Logistical and Financial Benefits Many respondents, including 86% of interviewees, reported logistical benefits of being in the cohort program. These benefits included the scheduling of classes to conform to the school-district schedule, rather than the university academic-year schedule. For example, fall-semester classes did not begin until after Labor Day when the public school academic-year started, rather than the week prior to Labor Day. In addition, vacation days and weeks were designed to coincide with the district and not the university schedule (e.g., spring break was scheduled during the school district spring break and not the university spring break). The sequence of courses for the entire program was available from the first day of classes, which enabled students to plan for program completion more efficiently. Finally, classes were held for four hours in an evening a week for a shorter number of weeks so that an ‘‘extra semester’’ was added to the spring-semester academic-year with classes starting at the beginning of January and again in March, compared with the traditional 15-week university spring semester. The vast majority of respondents indicated they appreciated the flexible scheduling that conformed to their teaching position’s academic-year schedule.

357

Partnership Cohorts

One respondent highlighted some of these logistical benefits by describing the program as ‘‘A fast track to fulfilling the HR [human resources and licensure] requirements of the position held within the county.’’ Another respondent captured the importance of the unique scheduling of the cohorts by stating, ‘‘The cohort was a simple way for educators to finish their masters and/or licensure on a set date and time without having to follow the schedule of normal college students.’’ The majority of the individuals, who participated in the survey and followup interviews, indicated they joined this cohort program because it was affordable, had a more flexible structure (semester), and was convenient. Respondents in the survey consistently reported that the costs associated with enrolling in these cohort programs were substantially lower than costs associated with enrolling independently as graduate students at the university. This was due to the negotiation between districts and the university for tuition costs. In many cases, the district assisted students by providing tuition reimbursements, and assisted with obtaining tuition waivers from the state. In addition, since all classes met at local schools, students did not have to drive to campus nor purchase a university parking permit. Individuals from the follow-up interviews overwhelmingly confirmed these responses. This theme is represented by the following respondent who stated, ‘‘The cohort also made it possible for me to take the classes and not take out student loans because of the discounted cost of the courses.’’ The current university–district partnership provided seemingly very important logistical and financial benefits for participants, as evidenced by the overwhelmingly positive responses concerning lower tuition and scheduling that aligned with school district’s calendar year rather than the university calendar.

Participant Recommendations Participants were asked to identify additional supports that would help the cohort experience and to make any other recommendations for the program. The vast majority of respondents were highly satisfied and made few suggestions. The following comment is representative of the majority: ‘‘I can’t think of anything additional; I felt very satisfied with the whole experience.’’ However, some respondents, predominantly members of the first cohorts, mentioned a few notable issues. One very important limitation identified by members of the early cohorts was the lack of sufficient face-to-face time with faculty advisors in off campus locations. Students wanted more assistance

358


with the program demands (e.g., especially the portfolio requirements), required testing (e.g., PRAXIS), and paperwork associated with applications for internships, graduation, reimbursements, registration, use of the university student web services, email, and licensure. In addition, respondents overwhelmingly wanted to avoid coming to campus for this advice. The cohort program addressed these concerns by hiring a point-ofcontact advisor who met initially and regularly with students in all cohorts at prespecified times either before or after cohort classes that were held in the local schools. In addition, a LISTSERV was established that emailed important information, such as registration deadlines, dates for applications for internships, and the like. The point-of-contact advisor also worked with students via email and phone on an as-needed basis, and was available for individual meetings. Another common suggestion included arranging contact with graduates of previous cohort members with current students. Respondents thought program graduates could serve as resources for current program participants. These respondents appeared to be requesting the assignment of group or individual mentors who had completed the program and who could serve as resources for questions concerning teaching positions and getting along within the school systems. Most of the supports requested surrounded classroom teaching and becoming acculturated in the school district. A similar suggestion included the establishment of more structured vehicles for maintaining contact with cohort members after program completion. These respondents appeared to be requesting follow-up alumni gatherings. Some respondents suggested that the districts should supply more leave time during program enrollment to provide individuals time to complete university assignments. Others suggested that the county and university should waive required courses for a variety of previous employment experiences. Some identified more individual concerns, such as having child care available at classes, more consistent use of blackboard, the web-based program used by the university across all cohort classes, and wireless internet access. A minority of respondents indicated feeling isolated from the university because everything was held on the district campuses. A few respondents requested a tour of the campus and its facilities, including the libraries, student unions, and computer labs. Many suggestions also were addressed within the program evaluation component not analyzed in this report. For example, respondents identified specific courses or course assignments and experiences that were considered helpful or unhelpful. Interview data corroborated the above findings and revealed requests for

359

Partnership Cohorts

changes to some specific classes and program requirements. The specific feedback on class issues is more relevant to internal program evaluation data for the programs, rather than the cohort experience, and is not addressed in this chapter.

DISCUSSION These findings revealed some very consistent and positive reports from 164 respondents who had been enrolled or were still enrolled in one of 15 cohort programs developed through a university–district partnership that was intended to meet state licensure and master’s degree requirements in special education. Over 98% of the responses reported the cohort experience was valuable to them, with the great majority indicating the experience was ‘‘very valuable.’’ These figures suggest that the cohort feature may have meant more to participants than a traditional university program. These findings were seen to be highly consistent across program completers and those still enrolled in the program, participant age, and cohort number (e.g., 1–15). Moreover, very specific positive benefits were obtained for supports in socialemotional and academic areas, improved pedagogical skills, and logistical and financial supports. These findings replicate those reported by previous researchers who reported on the value of cohorts in the preparation of teachers (e.g., Bullough, 2001; Graber, 1996). These findings also extend previous research in several ways. First, the present cohorts were a unique partnership between a university and school districts. Second, participants, who were, on average, a more mature group of learners (mean age 41), consisted of program completers and those still enrolled in the program, who reported highly consistent responses. Third, since many participants had been employed as IAs, this type of programming facilitated the education and movement of IAs into special-education teaching positions. Fourth, this analysis included 15 cohorts, which ranged in starting dates from fall 2001 through fall 2006, and which had provided opportunities to improve cohort programming based on annual student and faculty feedback.

Social-Emotional Support Participants in this study reported gaining social-emotional support from their peer cohort members. This finding also has been reported by other

360


researchers, including Ross et al., (2006), Seifert and Mandzuk (2006), and Dinsmore and Wenger (2006). All of these investigators reported on the social-emotional and psychological support reported by cohort members. For example, participants in the Ross et al. (2006) investigation reported obtaining comfort levels that enabled them to interact safely within their classrooms. In the present study, support was observed to be beneficial across several areas including building social friendships with cohort members that assisted them with their personal lives, outside of the university program and outside of their professional positions, as well as within the university program and their teaching positions. Many program completers indicated they still maintained contact with cohort members. Other recommendations included the structuring of more regular alumni gatherings of cohort members.

Academic Support Participants in this investigation also reported obtaining academic support from peers. Academic supports have also been reported by others. Mather and Hanley (1999) reported on the pedagogical content knowledge benefits associated with a cohort of elementary teacher education candidates. In the current sample, many participants indicated that the academic support combined with the social-emotional support enabled them to complete the program successfully. This particular sample was older than students in many teacher preparation programs, and some participants indicated they may have experienced more academic difficulties without the cohort members’ support. This sample was also working full time as conditionally licensed teachers or IAs. The challenges associated with working full time while earning teacher licensure and a master’s degree in special education can be demanding and even overwhelming at times to the best of students. The support provided by the cohort structure in this case may have been more crucial for these older students who were employed in the profession and enrolled in a university program simultaneously.

Pedagogical Skills Respondents in this investigation also reported receiving classroom-teaching supports from the cohort program. These findings also have been reported by Brownell et al. (2004). Brownell et al. studied beginning general

361

Partnership Cohorts

educators in a teacher learning cohort and reported that participants improved in their abilities to conceptualize instruction and manage classrooms for working with students with disabilities. In the present case, the vast majority of respondents were full-time special-education teachers with conditional licenses with a smaller percentage working as IAs within the same school districts. These individuals indicated that they gained knowledge about classroom teaching from participating in the cohorts. Frequent responses revealed that peers who were teaching in similar schooldistrict classes shared successful and unsuccessful experiences of teaching students with disabilities with one another. Information on school-district policies and procedures was also shared among cohort members. Since participants were full-time school-district employees, this type of communication facilitated a type of collaboration that appeared critical to the present job success of participants. Although the university classes were intended to maximize content and pedagogical learning (presumably in noncohort programs as well), the present type of collaboration and sharing of classroom experiences appears to extend beyond what might be expected from a traditional teacher preparation university program.

Logistical and Financial Supports In the present case, participants overwhelmingly reported a great deal of satisfaction with the logistical and financial arrangements of the cohorts. Important value-added components to the licensure and degree included the flexible scheduling, the holding of classes at school districts, the tuition assistance and financial assistance provided. These features may have been strong contributing factors to initial enrollment in the partnership cohort special-education program. Some reported they would have been unable to complete a degree program without the logistical and financial supports. This district–university partnership may have also helped reduce somewhat the special-education teacher shortage in the area. Although exact figures are unavailable, since districts encouraged full-time IAs to apply for the cohort program, they were attempting to groom their own future special-education teachers. This also indicates awareness on the part of districts that IAs may be a valuable resource as future fully licensed and highly qualified special-education teachers. In the current sample, a number of participants held IA positions prior to being hired as conditionally licensed special-education teachers.

362


Finally, since there were 15 cohorts in the present analysis, the district– university partnership was able to obtain annual feedback from program participants and modify and adjust the program to better meet student needs. For example, early cohort members indicated a need for more direct advisement on programmatic issues. The request for additional advisement was considered crucial for students. This was subsequently addressed with the hiring of a full-time point of contact advisor who provided information and answered questions in an open ongoing fashion. Later cohort members reported no such needs for additional advisement indicating that a need had been successfully met. In addition to academic advisement, early cohort members identified a need for individualized classroom-mentoring support. This resulted in the initiation of a classroom-based mentoring program, which provided many of these new teachers direct classroom support from veteran special-education teachers in the application of instructional strategies. After the first year, it was also determined that additional time from district staff was needed to manage the ongoing counseling and support needs of students. Thus, the continuous evaluation and feedback process allowed for ongoing program improvement and the development of effective cohort support structures. Other recommendations for future cohorts included arranging more contact with cohort alumni and the establishment of mentors from previous cohorts. These needs will be addressed for future cohorts.

Summary and Conclusions The overall findings from these 15 cohorts are surprisingly positive. The perceptions of participants indicate that this was a valuable experience extended beyond what might be found in a traditional preparation program. Previous research with cohorts has typically reported on a small sample of cohorts and individuals (e.g., Ross et al., 2006, reported on a sample of 25 students from a single-cohort group). This study extends that to a sample of 164 participants from 15 different cohort groups in a district–university partnership. The partnership in the present case may have facilitated the perceived supports since all individuals within a cohort were also full-time conditionally licensed special-education teachers or IAs within each respective district. Future research could examine the extent to which these findings are robust across variations of special-education personnel preparation programs, including those with undergraduate programs and those with more university-based traditional programs. It would be

363

Partnership Cohorts

interesting to determine whether district–university partnerships could be expanded to include other licensure areas. The cohort participation, with its value-added benefits, appeared to make a more meaningful experience for the vast majority of participants. This enhanced experience may be attributed to the enhanced social-emotional support perceived by the majority of participants. This social-emotional support appeared especially meaningful to more mature students who had been out of the university classroom for extended periods of time. In addition, however, the vast majority of participants mentioned academic benefits, including improved pedagogical skills such as learning and applying strategies with their students successfully. If cohort participants do exhibit improved pedagogical skills, then cohort experiences indeed may prove beneficial to personnel preparation programs. Perhaps this was due to the fact that these participants were all teaching everyday in the classroom and not the cohort experience. It may be that simply teaching in the classroom on a daily basis results in improved pedagogical skills. However, little research exists to document improved pedagogical skills acquired merely from teaching experience. Since challenges in teacher preparation exist with the translation of research to practice, perhaps cohort experiences of practicing teachers might provide a stronger bridge for addressing that gap in teaching.

Limitations and Future Research Finally, as with any investigation, there are limitations that need to be addressed with the present study. Survey research has inherent limitations, perhaps the greatest of which is its reliance on reported perceptions of participants rather than observable behaviors. One important issue is the difficulty separating effects of the cohort element from the academic program itself. Although many of the results presented here may also have been observed in noncohort programs, many do appear to be particularly relevant to the cohort-program structure. Some features were possible only in cohort programs (e.g., reduction of costs by working with the district in district schools). In many other instances, participants specifically attributed positive outcomes to the cohort structure of the program. Finally, participants did express several concerns about, for example, specific instructors or courses (not reported here) that bore no relation to their overall positive perceptions of the cohort feature. These findings suggest

364


strongly that the cohort itself served to promote many positive outcomes for participants.

REFERENCES Beck, C., & Kosnik, C. (2001). From cohort to community in a preservice teacher education program. Teaching and Teacher Education, 17, 925–948. Brownell, M. T., Yeager, E. A., Sindelar, P. T., vanHover, S., & Riley, T. (2004). Teacher learning cohorts: A vehicle for supporting beginning teachers. Teacher Education and Special Education, 27, 174–189. Bullough, R. V. (2001). Students, cohorts, school rhythms, and teacher education. Teacher Education Quarterly, 28, 97–110. Clarke, A., Erickson, G., Collins, S., & Phelan, A. (2005). Complexity science and cohorts in teacher education. Studying Teacher Education, 1, 159–177. Connor, Killmer, N. (2001). Cohorts, collaboration, and community: Does contextual teacher education really work? Action in Teacher Education, 23, 46–53. Creswell, J. W. (2008). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (3rd ed.). Upper Saddle River, NJ: Merrill/Prentice Hall. Dinsmore, J., & Wenger, K. (2006). Relationships in preservice teacher preparation: From cohorts to communities. Teacher Education Quarterly, 33, 57–74. Edgar, E., & Pair, A. (2005). Special education teacher attrition: It all depends on where you are standing. Teacher Education and Special Education, 28, 163–170. Esposito, M. C., & Lal, S. (2005). Responding to special education teacher shortages in diverse urban settings: An accelerated alternative credential program. Teacher Education and Special Education, 28, 100–103. Graber, K. (1996). Influencing student beliefs: The design of a high impact teacher education program. Teaching and Teacher Education, 12, 451–466. The Holmes Group. (1995). Third Holmes Group report: Tomorrow’s schools of education. East Lansing, MI: Author. Jorissen, K. (2002). Retaining alternate route teachers: The power of professional integration in teacher education. The High School Journal, 85, 45–56. LeCompte, M. D., & Preissle, J. (1993). Ethnography and qualitative design in educational research. San Diego, CA: Academic Press. Maher, M. A. (2005). The evolving meaning and influence of cohort membership. Innovative Higher Education, 30, 195–211. Mastropieri, M. A., Morrison, N., Scruggs, T. E., Bowdey, B., & Werner, T. (2008). The use of cohort programs in personnel preparation: Benefits and challenges. In: T. E. Scruggs & M. A. Mastropieri (Eds), Personnel preparation: Advances in learning and behavioral disabilities (Vol. 21, pp. 151–179). Bingley, UK: Emerald. Mather, D., & Hanley, B. (1999). Cohort grouping and preservice teacher education: Effects on pedagogical development. Canadian Journal of Education, 24, 235–250. Ross, D. D., Stafford, L., Church-Pupke, P., & Bondy, E. (2006). Practicing collaboration: What we learn from a cohort that functions well. Teacher Education and Special Education, 29, 32–43.

Partnership Cohorts

365

Sapon-Shevin, M., & Chandler-Olcott, K. (2001). Student cohorts: Communities of critique or dysfunctional families? Journal of Teacher Education, 52, 350–364. Scruggs, T. E., & Mastropieri, M. A. (1993). The effects of prior field experience on student teacher competence ratings. Teacher Education and Special Education, 16, 303–308. Seifert, K., & Mandzuk, D. (2006). Student cohorts in teacher education: Support groups or intellectual communities? Teachers College Record, 108, 1296–1320. Sindelar, P. T., McCray, E. D., Kiely, M. T., & Kamman, M. (2008). The impact of No Child Left Behind on teacher supply and the preparation of the workforce. In: T. E. Scruggs & M. A. Mastropieri (Eds), Personnel preparation: Advances in learning and behavioral disabilities (Vol. 21, pp. 89–123). Bingley, UK: Emerald. Tietel, L. (1997). Understanding and harnessing the power of a cohort model in preparing educational leaders. Peabody Journal of Education, 72, 66–85.

Assessment and Intervention (Advances in Learning and Behavioral Disabilities)

Policy and Practice (Advances in Learning and Behavioral Disabilities)

Technological Applications, Volume 15 (Advances in Learning and Behavioral Disabilities)

International Perspectives, Volume 20 (Advances in Learning and Behavioral Disabilities)

Identification and Assessment, Volume 16 (Advances in Learning and Behavioral Disabilities)

Cognition and Learning in Diverse Settings, Volume 18 (Advances in Learning and Behavioral Disabilities)

Diagnostic assessment of learning disabilities in childhood

Learning Disabilities: From Identification to Intervention

Research in Secondary Schools, Volume 17 (Advances in Learning and Behavioral Disabilities)

Working Memory and Academic Learning: Assessment and Intervention

Working Memory and Academic Learning: Assessment and Intervention

Geriatric Neuropsychology: Assessment and Intervention

School-Based Behavioral Assessment: Informing Intervention and Instruction (The Guilford Practical Intervention in Schools Series)

Assessment and Learning

Risk Assessment in People With Learning Disabilities (2nd ed)

Advances in behavioral finance,

Specific Learning Disabilities and Difficulties in Children and Adolescents: Psychological Assessment and Evaluation

Advances in behavioral economics

Advances in Behavioral Economics

Key Concepts in Learning Disabilities

Anxiety Disorders in Children and Adolescents: Research, Assessment and Intervention

Crisis Intervention Handbook: Assessment, Treatment, and Research (CRISIS INTERVENTION HANDBOOK)

Juvenile Delinquency: Prevention, Assessment, and Intervention

Adolescent Suicide: Assessment And Intervention 2nd Edition

WISC-IV Clinical Assessment and Intervention

Essentials of Dyslexia Assessment and Intervention

Learning Disabilities: New Research

Learners, Learning & Assessment (Learning, Curriculum and Assessment series)

Working with Children: Assessment, Representation and Intervention

Assessment and Learning in the Secondary School

Assessment and Intervention (Advances in Learning and Behavioral Disabilities)

Policy and Practice (Advances in Learning and Behavioral Disabilities)

Technological Applications, Volume 15 (Advances in Learning and Behavioral Disabilities)

International Perspectives, Volume 20 (Advances in Learning and Behavioral Disabilities)

Identification and Assessment, Volume 16 (Advances in Learning and Behavioral Disabilities)

Cognition and Learning in Diverse Settings, Volume 18 (Advances in Learning and Behavioral Disabilities)

Diagnostic assessment of learning disabilities in childhood

Learning Disabilities: From Identification to Intervention

Research in Secondary Schools, Volume 17 (Advances in Learning and Behavioral Disabilities)

Working Memory and Academic Learning: Assessment and Intervention

Working Memory and Academic Learning: Assessment and Intervention

Geriatric Neuropsychology: Assessment and Intervention

School-Based Behavioral Assessment: Informing Intervention and Instruction (The Guilford Practical Intervention in Schools Series)

Assessment and Learning

Risk Assessment in People With Learning Disabilities (2nd ed)

Advances in behavioral finance,

Specific Learning Disabilities and Difficulties in Children and Adolescents: Psychological Assessment and Evaluation

Advances in behavioral economics

Advances in Behavioral Economics

Key Concepts in Learning Disabilities

Anxiety Disorders in Children and Adolescents: Research, Assessment and Intervention

Crisis Intervention Handbook: Assessment, Treatment, and Research (CRISIS INTERVENTION HANDBOOK)

Juvenile Delinquency: Prevention, Assessment, and Intervention

Adolescent Suicide: Assessment And Intervention 2nd Edition

WISC-IV Clinical Assessment and Intervention

Essentials of Dyslexia Assessment and Intervention

Learning Disabilities: New Research

Learners, Learning & Assessment (Learning, Curriculum and Assessment series)

Working with Children: Assessment, Representation and Intervention

Assessment and Learning in the Secondary School

Recommend Documents