Statistics and Computing Series Editors: J. Chambers D. Hand W. Ha¨rdle
Statistics and Computing Brusco/Stahl: Branch-and-Bound Applications in Combinatorial Data Analysis. Dalgaard: Introductory Statistics with R. Gentle: Elements of Computational Statistics. Gentle: Numerical Linear Algebra for Applications in Statistics. Gentle: Random Number Generation and Monte Carlo Methods, 2nd Edition. Ha¨rdle/Klinke/Turlach: XploRe: An Interactive Statistical Computing Environment. Krause/Olson: The Basics of S-PLUS, 4th Edition. Lange: Numerical Analysis for Statisticians. Lemmon/Schafer: Developing Statistical Software in Fortran 95 Loader: Local Regression and Likelihood. ´ Ruanaidh/Fitzgerald: Numerical Bayesian Methods Applied to Signal Processing. O Pannatier: VARIOWIN: Software for Spatial Data Analysis in 2D. Pinheiro/Bates: Mixed-Effects Models in S and S-PLUS. Venables/Ripley: Modern Applied Statistics with S, 4th Edition. Venables/Ripley: S Programming. Wilkinson: The Grammar of Graphics.
Andreas Krause Melvin Olson
The Basics of S-PLUS Fourth Edition
Andreas Krause Novartis Pharma AG Basel 4002 Switzerland Series Editors: J. Chambers Bell Labs, Lucent Technologies 600 Mountain Ave. Murray Hill, NJ 07974 USA
Melvin Olson Novartis Pharma AG Basel 4002 Switzerland
D. Hand Department of Mathematics South Kensington Campus Imperial College, London London SW7 2AZ United Kingdom
W. Ha¨rdle Institut fu¨r Statistik und ¨ konometrie O Humboldt-Universita¨t zu Berlin Spandauer Str. 1 D-10178 Berlin, Germany
Library of Congress Control Number: 2005926768 ISBN-10: 0-387-26109-5 ISBN-13: 978-0387-26109-6
e-ISBN: 0-387-28390-0
Printed on acid-free paper.
© 2005 Springer Science+Business Media, Inc. All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science+Business Media, Inc., 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. Printed in the United States of America. 9 8 7 6 5 4 3 2 1 springeronline.com
(SBA)
Preface
This is now the fourth edition of “The Basics of S-Plus” since 1997. S-Plus saw a steady growth in popularity, and it established itself in many educational and business places as a major data analysis tool. S-Plus is valued for its modern, interactive data analysis environment, whether it is the primary system or a complement to other standards like SAS (the latter is in particular true for the industry we work in, pharmaceuticals). We have followed the various releases with new editions of our book, introducing over time major changes like the incorporation of S Version 4 (the underlying language), Trellis graphs, a graphical user interface, in particular for the Windows operating system, and a chapter on R and its differences to S-Plus (that are minor for the material covered in this book). This edition is an update from edition 3 to cover new functions and features of S-Plus Version 7.0 (working from the beta release for MS Windows and Linux), adding more practical tips and examples, and correcting a few mistakes. We are very grateful to all our readers, in particular those sending us suggestions, comments, and any other kind of feedback. You will see some of these reflected in the book. The number of new books appearing on S-Plus and its counterpart, R, continues to grow substantially each year. If you want to get an impression of the books available, Insightful maintains a list of S-Plus references on their web site, http://www.insightful.com/. The list currently comprises about 50 books and manuscripts. You can also search for the string “Splus” or “S-Plus” (without quotes) in an online bookstore, such as http://www.amazon.com/.
vi
Preface
Some of the books on S-Plus are “Analyzing Medical Data Using S and S-Plus,” by Everitt and Rabe-Hesketh; “Regression Modeling Strategies,” by Harrell; “Bootstrap Methods and Permutation Tests” by Hesterberg, Monaghan, et.al., “Applied Statistics in the Pharmaceutical Industry,” edited by Millard and Krause; “Mixed Effects Models in S and S-Plus,” by Pinheiro and Bates; “Modeling Survival Data: Extending the Cox Model,” by Therneau and Grambsch; and the fourth edition of “Modern Applied Statistics with S-Plus” as well as the second edition of ”S Programming,” both by Venables and Ripley. This book is intended to provide a head-start into the world of S-Plus (and serves as well as an introduction to R). We are going to introduce you to S-Plus by taking you through interactive sessions, entering commands and later analyzing data, pointing out tips and pitfalls, and complementing each session with exercises and detailed solutions. If you have never worked with S-Plus before and want to get a first impression, take a look at “A First Session,” “A Second Session,” and “Exploring Data” to get an idea of the philosophy of the programming language. If you intend to work with point-and-click graphical user interfaces, take a look at “Graphical User Interfaces.” We are convinced though that if you want to work permanently with S-Plus, you will need to use the command line and appreciate it soon. The book originates from lectures such that each chapter can serve as the basis for a 90-minute lecture. The exercises reinforce the material covered and sometimes point out a few more details. It should be interesting to work out a solution to an exercise before comparing it to the solution given here. There’s never a single solution. Finally, we would like to thank all those who have directly or indirectly contributed to this edition and previous ones. For this edition, Chyi-Hung Hsu, Michael Merz and Jos´e Pinheiro gave very helpful comments on the manuscript. John Kimmel at Springer–Verlag is simply a great person to work with, dedicated and professional. David Smith, Michael O’Connell and others at Insightful, Inc. (Seattle) continue to be supportive partners by providing us with alpha and beta releases of upcoming S-Plus versions. Following the discussion forum s-news provides an insight into frequently asked questions and problems that S-Plus users face. And last but not least, various people from all over the world supplied comments on earlier editions of the book, pointed out possible improvements and errors and provided a good motivation to revise and update the book with the current edition 4 – which you happen to be reading right now. We are always interested in any kind of comment that you might have about the book. You are welcome to contact us by sending an E-mail to
[email protected] and
[email protected]. Note though that we will not be able to provide general support, like answering programming questions. For these purposes please refer to the sections where we describe how to get help and support (pp. 413ff. and 422ff.). You
Preface
vii
might also want to check if your institution has a contract with Insightful that covers support. Finally, a Web page is set up to accompany this book: http://www.elmo.ch/doc/splus-book/ It provides information about the book, links to relevant Internet pages, and a list of errata (which we are confident will be fairly short).
Basel, Switzerland April 2005
Andreas Krause and Melvin Olson
Contents
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
v xvii xxi
1 Introduction 1.1 The History of S and S-Plus . . . . . . . . . . . . . . . 1.2 S-Plus on Different Operating Systems . . . . . . . . . 1.3 Notational Conventions . . . . . . . . . . . . . . . . . . .
1 2 4 6
2 Graphical User Interface 2.1 Introduction . . . . . . . . . . . . 2.2 System Overview . . . . . . . . . 2.2.1 Using a Mouse . . . . . . . 2.2.2 Object Explorer . . . . . . 2.2.3 Commands Window . . . 2.2.4 Toolbars . . . . . . . . . . 2.2.5 Graph Sheets . . . . . . . 2.2.6 Script Window . . . . . . 2.3 Getting Started with the Interface 2.3.1 Importing Data . . . . . . 2.3.2 Graphs . . . . . . . . . . . 2.3.3 Data and Statistics . . . . 2.3.4 Customizing the Toolbars 2.3.5 Chapters . . . . . . . . . . 2.4 Detailed Use of the GUI Interface
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
9 9 10 11 11 11 12 12 12 13 13 13 15 15 16 18
x
Contents
2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12 2.13 2.14 2.15 2.16 2.17 2.18 2.19 2.20
Object Explorer . . . . . . . Help . . . . . . . . . . . . . Data Export . . . . . . . . . Working Directory . . . . . Data Import . . . . . . . . . Data Summaries . . . . . . . Graphs . . . . . . . . . . . . Trellis Graphs . . . . . . . . Linear Regression . . . . . . PowerPoint (Windows Only) Excel (Windows Only) . . . Script Window . . . . . . . . UNIX/Linux GUI . . . . . . Summary . . . . . . . . . . . Exercises . . . . . . . . . . . Solutions . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
18 19 21 23 24 27 29 36 38 42 44 45 47 56 57 58
3 A First Session 3.1 General Information . . . . . . . . . . . 3.1.1 Starting and Quitting . . . . . . 3.1.2 The Help System . . . . . . . . 3.1.3 Before Beginning . . . . . . . . 3.2 Simple Structures . . . . . . . . . . . . 3.2.1 Arithmetic Operators . . . . . . 3.2.2 Assignments . . . . . . . . . . . 3.2.3 The Concatenate Command: c . 3.2.4 The Sequence Command: seq . 3.2.5 The Replicate Command: rep . 3.3 Mathematical Operations . . . . . . . 3.4 Use of Brackets . . . . . . . . . . . . . 3.5 Logical Values . . . . . . . . . . . . . . 3.6 Review . . . . . . . . . . . . . . . . . . 3.7 Exercises . . . . . . . . . . . . . . . . . 3.8 Solutions . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
73 73 74 75 75 76 76 77 79 80 81 82 84 85 88 91 92
4 A Second Session 4.1 Constructing and Manipulating Data 4.1.1 Matrices . . . . . . . . . . . . 4.1.2 Arrays . . . . . . . . . . . . . 4.1.3 Data Frames . . . . . . . . . . 4.1.4 Lists . . . . . . . . . . . . . . 4.2 Introduction to Functions . . . . . . 4.3 Introduction to Missing Values . . . . 4.4 Merging Data . . . . . . . . . . . . . 4.5 Putting It All Together . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
95 95 96 101 104 107 108 109 110 111
. . . . . . . . .
Contents
4.6 4.7
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . .
5 Graphics 5.1 Basic Graphics Commands . . . . . . . . . . . . 5.2 Graphics Devices . . . . . . . . . . . . . . . . . 5.2.1 Working with Multiple Graphics Devices 5.3 Plotting Data . . . . . . . . . . . . . . . . . . . 5.3.1 The plot Command . . . . . . . . . . . 5.3.2 Modifying the Data Display . . . . . . . 5.3.3 Modifying Figure Elements . . . . . . . . 5.4 Adding Elements to Existing Plots . . . . . . . 5.4.1 Functions to Add Elements to Graphs . 5.4.2 More About abline . . . . . . . . . . . 5.4.3 More on Adding Axes . . . . . . . . . . 5.4.4 Adding Text to Graphs . . . . . . . . . . 5.5 Setting Options . . . . . . . . . . . . . . . . . . 5.6 Figure Layouts . . . . . . . . . . . . . . . . . . 5.6.1 Layouts Using Trellis Graphs . . . . . . 5.6.2 Matrices of Graphs . . . . . . . . . . . . 5.6.3 Multiple-Screen Graphs . . . . . . . . . . 5.6.4 Figures of Specified Size . . . . . . . . . 5.7 Exercises . . . . . . . . . . . . . . . . . . . . . . 5.8 Solutions . . . . . . . . . . . . . . . . . . . . . .
xi
114 116
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
125 125 126 128 128 129 130 131 133 133 135 135 137 138 140 140 140 141 142 145 146
6 Trellis Graphics 6.1 An Example . . . . . . . . . . . . . . . . . . . . . . 6.2 Trellis Basics . . . . . . . . . . . . . . . . . . . . . . 6.2.1 Trellis Syntax . . . . . . . . . . . . . . . . . 6.2.2 Trellis Functions . . . . . . . . . . . . . . . 6.2.3 Displaying and Storing Graphs . . . . . . . 6.3 Output Devices . . . . . . . . . . . . . . . . . . . . 6.4 Customizing Trellis Graphs . . . . . . . . . . . . . . 6.4.1 Setting Options . . . . . . . . . . . . . . . . 6.4.2 Arranging the Layout of a Trellis Graph . . 6.4.3 Ordering of Graphs . . . . . . . . . . . . . . 6.4.4 Axis Customization . . . . . . . . . . . . . . 6.4.5 Modifying Panel Strips . . . . . . . . . . . . 6.4.6 Arranging Several Graphs on a Single Page 6.4.7 Updating Existing Trellis Graphs . . . . . . 6.4.8 Writing Panel Functions . . . . . . . . . . . 6.5 Further Trellis Hints . . . . . . . . . . . . . . . . . 6.5.1 Useful General Trellis Settings . . . . . . . . 6.5.2 Graphing Individual Profiles . . . . . . . . . 6.5.3 Preparing Data to Use for Trellis . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
153 154 156 156 157 157 158 160 160 161 163 164 165 165 167 168 171 172 173 174
. . . . . . . . . . . . . . . . . . . .
xii
Contents
. . . . .
175 175 177 181 183
7 Exploring Data 7.1 Descriptive Data Exploration . . . . . . . . . . . . . . . 7.2 Graphical Exploration . . . . . . . . . . . . . . . . . . . 7.2.1 Interactive Dynamic Graphics . . . . . . . . . . . 7.2.2 Old-Style Graphics . . . . . . . . . . . . . . . . . 7.3 Distributions and Related Functions . . . . . . . . . . . 7.4 Confirmatory Statistics and Hypothesis Testing . . . . . 7.5 Missing and Infinite Values . . . . . . . . . . . . . . . . . 7.5.1 Testing for Missing Values . . . . . . . . . . . . . 7.5.2 Supplying Data with Missing Values to Functions 7.5.3 Missing Values in Graphs . . . . . . . . . . . . . 7.5.4 Infinite Values . . . . . . . . . . . . . . . . . . . . 7.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.7 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . .
193 193 204 219 219 220 225 231 232 232 233 233 235 238
8 Statistical Modeling 8.1 Introductory Examples . . . . . . . . . 8.1.1 Regression . . . . . . . . . . . . 8.1.2 Regression Diagnostics . . . . . 8.2 Statistical Models . . . . . . . . . . . . 8.3 Model Syntax . . . . . . . . . . . . . . 8.4 Regression . . . . . . . . . . . . . . . . 8.4.1 Linear Regression and Modeling 8.4.2 ANOVA . . . . . . . . . . . . . 8.4.3 Logistic Regression . . . . . . . 8.4.4 Survival Data Analysis . . . . . 8.4.5 Endnote . . . . . . . . . . . . . 8.5 Exercises . . . . . . . . . . . . . . . . . 8.6 Solutions . . . . . . . . . . . . . . . . .
6.6 6.7
6.5.4 The subset Option . . . 6.5.5 Adding a Key . . . . . . 6.5.6 The subscripts Option in Exercises . . . . . . . . . . . . . Solutions . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . Panel Functions . . . . . . . . . . . . . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
251 251 251 253 255 256 257 258 261 263 265 267 268 271
9 Programming 9.1 Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1.1 Adding and Deleting List Elements . . . . . . 9.1.2 Naming List Elements . . . . . . . . . . . . . 9.1.3 Applying the Same Function to List Elements 9.1.4 Unlisting a List . . . . . . . . . . . . . . . . . 9.1.5 Generating a List by Using split . . . . . . . 9.2 Writing Functions . . . . . . . . . . . . . . . . . . . . 9.2.1 Documenting Functions . . . . . . . . . . . .
. . . . . . . .
. . . . . . . .
285 285 287 288 290 294 294 294 297
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9.2.2 9.2.3 9.2.4
9.3
9.4
9.5 9.6 9.7 9.8
Scope of Variables . . . . . . . . . . . . Parameters and Defaults . . . . . . . Passing an Unspecified Number of Parameters to a Function . . . . . . 9.2.5 Testing for Existence of an Argument . 9.2.6 Returning Warnings and Errors . . . . 9.2.7 Using Function Arguments in Graphics Iteration . . . . . . . . . . . . . . . . . . . . . 9.3.1 The for Loop . . . . . . . . . . . . . . 9.3.2 The while Loop . . . . . . . . . . . . . 9.3.3 The repeat Loop . . . . . . . . . . . . 9.3.4 Vectorizing a Loop . . . . . . . . . . . 9.3.5 Large Loops . . . . . . . . . . . . . . . Debugging: Searching for Errors . . . . . . . . 9.4.1 Syntax Errors . . . . . . . . . . . . . . 9.4.2 Invalid Arguments . . . . . . . . . . . 9.4.3 Execution or Run-Time Errors . . . . . 9.4.4 Logical Errors . . . . . . . . . . . . . . Output Using the cat Function . . . . . . . . The paste Function . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . Solutions . . . . . . . . . . . . . . . . . . . . .
10 Object-Oriented Programming 10.1 Creating Classes and Objects 10.2 Creating Methods . . . . . . . 10.3 Debugging . . . . . . . . . . . 10.4 Help . . . . . . . . . . . . . . 10.5 Summary and Overview . . . 10.6 Exercises . . . . . . . . . . . . 10.7 Solutions . . . . . . . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
11 Input and Output 11.1 Reading S-Plus Commands from a File: The source Function . . . . . . . . . . . 11.2 Data Import/Export: Easiest Method . . 11.3 Data Import/Export: General Method . 11.4 Data Import/Export: Basic Method . . . 11.5 Reading Data from the Terminal . . . . 11.6 Editing Data . . . . . . . . . . . . . . . . 11.7 Transferring Data . . . . . . . . . . . . . 11.8 Recording a Session . . . . . . . . . . . . 11.9 Exercises . . . . . . . . . . . . . . . . . . 11.10 Solutions . . . . . . . . . . . . . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
Contents
xiii
. . . . . . . . . . . .
297 298
. . . . . . . . . . . . Labels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
300 301 301 302 303 303 304 305 305 307 308 309 310 310 311 314 316 318 319
. . . . . . .
. . . . . . .
. . . . . . .
323 325 328 333 334 334 335 336
. . . . . . .
. . . . . . .
. . . . . . .
349 . . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
349 350 352 353 354 355 356 356 358 359
xiv
Contents
12 Tips and Tricks 363 12.1 Useful Techniques . . . . . . . . . . . . . . . . . . . . . . 363 12.1.1 Housekeeping: Cleaning Up Directories . . . . . . 363 12.1.2 Storing and Restoring Graphical Parameters . . . 364 12.1.3 Naming of Objects . . . . . . . . . . . . . . . . . 364 12.1.4 Repeating Commands . . . . . . . . . . . . . . . 365 12.2 Programming Environment and Techniques . . . . . . . 366 12.2.1 The Process of Developing a Function . . . . . . 366 12.2.2 Setting up an Editor and Running the Code in S-Plus . . . . . . . . . . . . . . . . . . . . . . 366 12.2.3 Treating Data Frames as Lists . . . . . . . . . . . 368 12.2.4 Working with Graph Sheets . . . . . . . . . . . . 369 12.2.5 Incorporating and Accessing C and Fortran Programs . . . . . . . . . . . . . . . . . . . . . . . . 371 12.2.6 Batch Jobs . . . . . . . . . . . . . . . . . . . . . . 374 12.2.7 Libraries . . . . . . . . . . . . . . . . . . . . . . . 376 12.3 Factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378 12.3.1 Creating Factors and Ordered Factors . . . . . . 378 12.3.2 Internal Representation of Factors . . . . . . . . . 380 12.3.3 Where Levels Play a Role . . . . . . . . . . . . . 381 12.3.4 Where Factors Can Lead Their Own Lives . . . . 382 12.3.5 How Factors Come Into Life . . . . . . . . . . . . 384 12.3.6 Adding and Dropping Factor Levels . . . . . . . . 385 12.4 Including Graphs in Text Processors . . . . . . . . . . . 386 12.4.1 Generating Graphs for Windows Applications . . 387 12.4.2 Generating PostScript Graphs . . . . . . . . . . . 388 12.4.3 PostScript Graphs in LATEX . . . . . . . . . . . . 389 12.4.4 If You Don’t Have a PostScript Printer . . . . . . 390 12.4.5 Greek Letters in Graphs . . . . . . . . . . . . . . 390 12.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 392 12.6 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . 394 13 S-Plus Internals 13.1 How S-Plus Works Under UNIX . . . . . . 13.1.1 The Working Chapter . . . . . . . . 13.1.2 Customization on Start-Up and Exit 13.2 How S-Plus Works Under Windows . . . . 13.2.1 Command Line Options . . . . . . . 13.2.2 Start-up and Exit Functions . . . . . 13.2.3 How the Script Window works . . . . 13.3 Storing Mechanism . . . . . . . . . . . . . . 13.4 Levels of Calls . . . . . . . . . . . . . . . . . 13.5 Exercises . . . . . . . . . . . . . . . . . . . . 13.6 Solutions . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
401 401 402 402 404 404 405 406 407 408 410 411
Contents
14 Information Sources on and Around S-Plus 14.1 Insightful . . . . . . . . . . . . . . . . . . . . 14.2 S-News: Exchanging Information with Other 14.3 The StatLib Server . . . . . . . . . . . . . . 14.4 What Next? . . . . . . . . . . . . . . . . . . 15 R 15.1 15.2 15.3
15.4
Development . . . . . . . . . . . . . . . . . Some Similarities Between R and S . . . . Some Differences Between R and S . . . . 15.3.1 Language . . . . . . . . . . . . . . 15.3.2 Libraries . . . . . . . . . . . . . . . 15.3.3 Trellis-Type Graphs . . . . . . . . . 15.3.4 Colors and Lines . . . . . . . . . . 15.3.5 Data Import and Export Formats . 15.3.6 Memory Handling . . . . . . . . . . 15.3.7 Mathematical Formulae in Graphs 15.3.8 Graphical User Interfaces . . . . . . 15.3.9 Start-Up Mechanism . . . . . . . . 15.3.10 Windows Integration . . . . . . . . 15.3.11 Support . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . .
16 Bibliography 16.1 Print Bibliography . . . . . . . 16.2 On-Line Bibliography . . . . . . 16.2.1 S-Plus Related Sources 16.2.2 TEX-Related Sources . . 16.2.3 Other Sources . . . . . . Index
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
xv
. . . . Users . . . . . . . .
. . . .
. . . .
. . . .
413 413 414 414 415
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
417 418 418 418 419 420 420 421 421 421 421 421 422 422 422 423
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
425 425 427 427 429 429 431
Figures
2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12 2.13 2.14 2.15 2.16 2.17 2.18 2.19 2.20 2.21 2.22 2.23 2.24
The S-Plus screen and its components The S-Plus main toolbar . . . . . . . . The 2D graph palette . . . . . . . . . . The Object Explorer toolbar . . . . . . The SearchPath . . . . . . . . . . . . . . Details of the Puromycin data set . . . . Data export . . . . . . . . . . . . . . . . Defining a working directory . . . . . . . Data import window . . . . . . . . . . . Sorting a data frame . . . . . . . . . . . Sorted Puromycin data . . . . . . . . . . Summary Statistics data dialog . . . . . Summary Statistics dialog . . . . . . . . Summary Statistics Report window . . . 2D Plot dialog (symbol) . . . . . . . . . Axis dialog box . . . . . . . . . . . . . . Axis font dialog box . . . . . . . . . . . Annotated scatterplot . . . . . . . . . . Saving a graph sheet . . . . . . . . . . . Trellis graphs – drag-and-drop . . . . . A Trellis graph conditioning on state . . Linear Regression dialog . . . . . . . . . Regression Formula dialog . . . . . . . . Regression Report window . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
10 12 14 19 20 20 22 24 25 26 26 28 28 29 31 32 32 33 35 36 37 39 39 40
xviii
2.25 2.26 2.27 2.28 2.29 2.30 2.31 2.32 2.33 2.34 2.35 2.36 2.37 2.38 2.39 2.40 2.41 2.42 2.43 2.44 2.45 2.46 2.47 2.48 2.49 2.50 2.51 2.52 2.53 2.54 2.55
Figures
Combining graphs onto one graph sheet . The Excel toolbar . . . . . . . . . . . . . The Script Window . . . . . . . . . . . . . Script Window toolbar . . . . . . . . . . . The UNIX/Linux Startup window . . . . The Objects Summary Window . . . . . . Data specification dialog . . . . . . . . . . Data viewer . . . . . . . . . . . . . . . . . Trellis scatterplot data specification . . . Trellis scatterplot symbol options . . . . . Trellis scatterplot . . . . . . . . . . . . . . Summary Statistics dialog . . . . . . . . . Summary Statistics Report window . . . . Linear Regression specification dialog . . . Linear Regression Report window . . . . . Linear regression residual plots . . . . . . Inserting a column of data . . . . . . . . . Loess smoother curves . . . . . . . . . . . Variable transformation for regression . . Regression output using log(conc) . . . . Residual plots using log(conc) . . . . . . . Regression formula using state . . . . . . Regression output using state . . . . . . . Regression formula using interaction term Regression output using interaction term . Object Explorer . . . . . . . . . . . . . . . Plot Symbol dialog . . . . . . . . . . . . . Geyser scatterplot . . . . . . . . . . . . . The Annotation palette . . . . . . . . . . Plot Line dialog . . . . . . . . . . . . . . . Geyser scatterplot with annotation . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
43 45 46 47 48 49 50 50 51 52 52 53 54 54 55 55 58 59 60 61 62 62 63 64 64 65 68 69 69 70 71
Different options for the type parameter of the plot function . . . . . . . . . . . . . . . . . . . . 5.2 An example of a graph layout . . . . . . . . . . . 5.3 A customized graph . . . . . . . . . . . . . . . . 5.4 The S-Plus definition of figure and plot regions 5.5 A customized graph layout . . . . . . . . . . . . 5.6 A customized display of the Geyser data set . . . 5.7 Trigonometric functions . . . . . . . . . . . . . . 5.8 Geometric figures (a): a rectangle. . . . . . . . . 5.9 Geometric figures (b): a circle . . . . . . . . . . . 5.10 Geometric figures (c): a spiral . . . . . . . . . . . 5.11 Lissajous figures . . . . . . . . . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
131 132 133 138 143 144 146 148 149 150 151
5.1
Figures
6.1 6.2 6.3 6.4 6.5 6.6 6.7 6.8 6.9 6.10 6.11 6.12
xix
An example of a Trellis graph layout . . . . . . . . . . . . Histograms of heights of singers for different voice parts . Customizing a Trellis graph arrangement . . . . . . . . . . Graph ordering in a Trellis arrangement . . . . . . . . . . A multigraph page of Trellis displays . . . . . . . . . . . . A multigraph page of Trellis displays with different sizes . Trellis display with customized panel function . . . . . . . Time profiles of Theophylline concentration . . . . . . . . Concentration-time profiles with dosing indicated . . . . . Trellis display of data spread, mean, and standard deviations Trellis histograms with added key of derived statistics. . . Trellis display of the Geyser data set (duration vs. waiting time) with regression lines added . . . . . . . . . . . . . . 6.13 Trellis display of the Geyser data set (waiting time vs. duration) with regression lines added . . . . . . . . . . . . 6.14 Trellis display of data spread, mean, and standard deviations. Group “g” consists only of missing values . . . . . . 6.15 Trellis display of data spread, mean, and standard deviations with non-default parameter settings . . . . . . . . . . . .
154 155 162 163 166 167 170 174 180 182 185
7.1 7.2 7.3 7.4 7.5
205 205 206 207
187 188 191 192
7.6 7.7 7.8 7.9 7.10 7.11 7.12 7.13 7.14 7.15 7.16 7.17 7.18 7.19
Histogram of the barley yield . . . . . . . . . . . . . . . . Histogram of the barley yield by site . . . . . . . . . . . . Histogram of the barley yield by site and year . . . . . . . Boxplot display of the barley yield split by site of harvest Boxplot display of the barley yield for all sites by year, employing the boxplot function . . . . . . . . . . . . . . Boxplot display of the barley yield for all years by site . . The Old Faithful Geyser data . . . . . . . . . . . . . . . . The Old Faithful Geyser data in subsets . . . . . . . . . . A perspective plot of the Geyser data using persp . . . . A perspective plot of the Geyser data using wireframe . . An image plot of the Geyser data using image . . . . . . . A contour plot of the Geyser data using contour . . . . . A contour plot of the Geyser data using contourplot . . A graphical comparison of three distributions . . . . . . . A graph of the binomial distribution . . . . . . . . . . . . A graphical display of the t-test . . . . . . . . . . . . . . . Different histogram displays of normally distributed data The Car data set . . . . . . . . . . . . . . . . . . . . . . . Estimating pi using a simulation technique . . . . . . . .
208 209 211 213 214 215 216 217 218 225 226 227 238 243 248
8.1 8.2 8.3 8.4
Regression diagnostic plots . . . . . . . Typical regression diagnostic plots . . . GAM regression diagnostic plots . . . . Design and interaction plots for ANOVA
254 278 279 283
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
xx
Figures
9.1
Two-dimensional functions drawn with persp . . . . . . .
320
10.1 Country data graph for a single country. . . . . . . . . . . 10.2 Country data graph for two countries . . . . . . . . . . . . 10.3 Country data graph for four countries . . . . . . . . . . .
331 332 336
11.1 Capital development with a given interest rate . . . . . .
362
12.1 The “play” button to run S-Plus scripts . . . . . . . . . 12.2 Renaming graph sheet pages . . . . . . . . . . . . . . . . . 12.3 Barley site factor levels reordered by maximum yield . . .
367 370 395
Tables
1.1 1.2
S-Plus system requirements . . . . . . . . . . . . . . . . Notational conventions . . . . . . . . . . . . . . . . . . . .
5 7
2.1
File types available with the Import From File facility . .
13
3.1 3.2 3.3 3.4 3.5
Arithmetic operations on vectors Different brackets in S-Plus . . Symbols for logical operators . . Truncation functions . . . . . . . Truncation results . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
83 84 86 94 94
The most common output devices . . . . . . . Commands for graphics devices . . . . . . . . . Options for type in the plot command . . . . Options for plotting functions . . . . . . . . . . Commands that add to existing graphs . . . . . Some examples of uses for abline . . . . . . . Optional parameters for plotting axes . . . . . Optional parameters for plotting axes using the axis function . . . . . . . . . . . . . . . . . . . 5.9 Optional parameters for adding text . . . . . . 5.10 Layout parameters . . . . . . . . . . . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
127 129 130 132 134 135 136
. . . . . . . . . . . . . . . . . .
136 137 139
Trellis functions . . . . . . . . . . . . . . . . . . . . . . . .
157
5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8
6.1
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
xxii
Tables
6.2 6.3 6.4
Trellis graphics devices . . . . . . . . . . . . . . . . . . . . Trellis plot options . . . . . . . . . . . . . . . . . . . . . . Trellis panel functions . . . . . . . . . . . . . . . . . . . .
7.1 7.2 7.3 7.4 7.5 7.6 7.7
Descriptive statistics functions . . . . . . . . . . Tabulation and split functions . . . . . . . . . . . Multivariate dynamic graphics functions . . . . . Old-style graphics functions . . . . . . . . . . . . Old-style multivariate graphics functions . . . . . Categorization of distribution-related functions in Distribution-related functions in S-Plus: Discrete distributions . . . . . . . . . . . . . . . . 7.8 Distribution-related functions in S-Plus: Continuous distributions . . . . . . . . . . . . . . 7.9 Statistical tests in S-Plus . . . . . . . . . . . . . 7.10 Estimating pi using a simulation technique . . .
. . . . . . . . . . . . . . . . . . . . . . . . . S-Plus
200 200 219 219 220 221
. . . . .
222
. . . . . . . . . . . . . . .
222 228 247
8.1 8.2 8.3 8.4 8.5
Swiss fertility data set . . Selected statistical models Model syntax . . . . . . . Kyphosis data frame . . . Leukemia data frame . . .
. . . . .
. . . . .
252 256 257 263 266
9.1 9.2
Some commands in browser, debugger, and recover . . Control sequences in S-Plus . . . . . . . . . . . . . . . .
315 315
11.1 File types and in UNIX . . . 11.2 File types and in Windows .
. . . . . . in S-Plus . . . . . . . . . . . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
extensions with the import/export . . . . . . . . . . . . . . . . . . . . extensions with the import/export . . . . . . . . . . . . . . . . . . . .
. . . . .
. . . . .
. . . . .
facility . . . . . facility . . . . .
12.1 S-Plus types and C/Fortran equivalents . . . . . . . . . .
160 161 168
351 351 373
1 Introduction
Over the years, the S language and S-Plus have undergone many changes. Since its development in the mid-1970s, the three main authors of S, Rick Becker, John Chambers, and Allan Wilks, have enhanced the entire language considerably. All their work was done at Bell Labs with the original goal of defining a language to make it easier to do repetitive tasks in data analysis, such as calculating a linear model. In the following years, many people contributed to the S project in one form or another. People outside Bell Labs also became aware of the interesting development and took part in it, and this is to a great extent the way S and S-Plus are still developed today. A very lively user community works with and on S/S-Plus, and they especially appreciate the S style of working in an integrated environment. Special strengths are the modern and flexible language and high quality graphics generation. It is noteworthy that the authors do not consider S as a primarily statistical system. The system was developed to be flexible and interactive, especially designed for doing data analysis of an exploratory nature, which began to boom in the late 1970s after the release of Tukey’s book (1977) on the subject. Most of the statistical functionality was added later, and many statistics routines, such as estimation, regression, and testing, were incorporated by the S-Plus team. This chapter describes the development of S and S-Plus over the years, clarifies the differences between the two, and points to some further references.
2
1. Introduction
1.1 The History of S and S-Plus The S Language, and some years later the New S, were developed at AT&T Bell Labs in the late 1970s and early 1980s, mainly by Rick Becker and John Chambers. Some years later, Allan Wilks joined the core team. Several other people have been involved in the project. Becker (1994) describes in great detail the foundation and development of S until 1994 (S Version 3). S Version 4 was developed by John Chambers at Bell Laboratories, Lucent Technologies. The year 1976 can perhaps be viewed as the founding year of S, the year in which the first concepts were developed and implemented. At first, the “system” consisted of a library of routines together with an interface, such that the kernel code itself could be kept unmodified. In 1981, the S team decided to rewrite the system in C and port it to the UNIX operating system. From 1981 to 1993, the source code was available to interested people outside Bell Labs. The next years revealed a strongly increasing interest among statisticians and data analysts in using the system, which was called S by then. It is remarkable that the important steps in the development of S were all marked by books, such that S users today talk about the days of the Brown Book, the Blue Book, and the White Book. The Green Book marks the most recent milestone. In 1984, as the interest in S began to rise, a real manual was necessary. The first book, today referred to as the Brown Book, was written by Becker and Chambers (1984). This version of S is now referred to as “Old S,” as no version numbers existed at the time. The QPE (Quantitative Programming Environment) developed by John Chambers set a milestone in the development of S. In 1988, it introduced the function concept (replacing the former macros), and new programming concepts were added. This work is described in the Blue Book (Becker, Chambers, and Wilks, 1988). During all these years, the user community added substantial functionality to S, and many sophisticated techniques such as tree regression, nonparametric smoothing, survival analysis, object-oriented programming, and new model formulation became a part of S. This step in the development was manifested and accompanied by the White Book (Chambers and Hastie, 1992). Version 4 of S came out in 1998 and is described in full detail in Chambers (1998), the Green Book. The object-oriented paradigm forms the basis of the entire language, documentation is integrated into an object, and the general paradigm “everything is an object” is followed throughout the language. In 1987, Douglas Martin at the University of Washington, Seattle, founded a small company to make S more popular. He realized that the major drawback of S was the need of professional support for end-users. Hence,
1.1. The History of S and S-Plus
3
he started the company Statistical Sciences, Inc. (StatSci). StatSci became the Data Analysis Products Division of MathSoft in 1994. When the company became independent again in 2001, it received a new name: Insightful Corporation. The company added more functionality to S, ported it to other hardware platforms, and provided the necessary support for technical and statistical problems. The enhanced version of S received a new name: S-Plus. S-Plus helped popularize S among nontechnical people. StatSci ported S-Plus to the only non-UNIX platform, releasing S-Plus for DOS in 1989 and S-Plus for Windows in 1993. Up to version 3, S-Plus for DOS/Windows and UNIX provided essentially the same functionality. S-Plus Version 4 was only released for the Windows platform. It was enhanced by a new graphical user interface (GUI) and a graphics system based on the Axum package. It adopts the Windows standard and gives a menu-based interface to S-Plus. Windows-specific functionality such as creation of PowerPoint slides or direct data exchange with other programs via DLLs was added. Much of the S-Plus functionality is available via menus and buttons, and the graphics are shown in an editable graph sheet. A major strength is that all functionality that is accessible via the menus can also be called from the command line. S-Plus comes with a script window where the corresponding command line input is shown. In fall 1998, S-Plus for Windows was released in version 4.5, which is split into a “Professional Edition” that comprises the full functionality and a menu-only version called “S-Plus Standard.” Version 5 of S-Plus was only released on UNIX systems in late 1998 and 1999. It is based on S Version 4 as described in Chambers (1998). For the first time, the Linux system for Intel PCs is supported. S-Plus 2000 for Windows systems (analogous to Office 2000 by Microsoft) was released in mid-1999, still based on S Version 3. Insightful released S-Plus Version 6 for UNIX in 2000 and for Windows in 2001. For the first time, the two systems are built from the same code base. The UNIX system has a graphical user interface based on Java and supports some Windows-specific elements such as reading Excel data files and writing graphs in WMF (Windows MetaFile) format. Version 6.2 introduced an important feature for automated batch runs, the so–called verbose logging. By calling S-Plus with the switch /SBATCH an exhaustive log file is created that facilitates scanning for possible problems in batch runs. Version 7.0 of S-Plus is about to be released by the time of writing this edition of the book (early 2005). We have an alpha and a beta version of S-Plus 7.0 for Windows and a beta version for Linux available. The new major version includes a library for handling big data sets without keeping it entirely in memory. The advertised limits are “no limits in the number of rows” and “tens of thousands of columns.” The library comprises functions for extracting simple statistics from columns and for calculating statistical
4
1. Introduction
models such as linear regression, generalized linear models, principal components, and clustering on big data sets. The GUI is revised and now based on a new toolkit, Eclipse. It comes with a workbench to develop S-Plus programs, track changes, and source control facilities. These new functionalities are only available for the newly introduced “Enterprise Developer” edition though and not part of the “Professional Developer” edition. S is still the heart of the system, and the core S team continues to work on the S system. The whole S functionality is incorporated in S-Plus and enhanced, and today the S system is no longer publicly available. In the remainder of the book, we will use S-Plus as the standard reference. In addition to S-Plus, the R project started in 1997. R is a software that is “not unlike S” and freely available. It has become increasingly popular in recent years since a core developer team continuously improved and developed the system further. A fairly large group of supporters around the world contributes libraries, in very much the way S started out a long time before. This group of people includes many of the contributors of S-Plus libraries. We will point out some of the similarities and differences between the two systems in a separate chapter.
1.2 S-Plus on Different Operating Systems Sometimes it is important to know about differences in software on various hardware systems or under different operating systems. This can be the case if you work on more than one computer system with S-Plus (and therefore need to exchange data files) or if you want to be informed about the differences before deciding in favor of a specific system. In this section, we discuss some details about different systems supported by S-Plus. In addition, the chapter provides some basic information about the general setup of files and structures in S-Plus. More information on the internal workings of S-Plus can be found in Sections 13.1 and 13.2. At present, S-Plus supports two major operating systems: UNIX (with most of its variants) and Windows (with most of its variants). Table 1.1 summarizes the currently supported hardware and operating systems. As the S source code is no longer available, machines not binary compatible to the ones supported are not able to run S or S-Plus. In those cases, the R system offers an alternative. We will get to the details in Chapter 15 (page 417). S-Plus has minimum requirement specifications regarding main memory and hard disk size. The requirements depend on the particular version and operating system. For a full installation, 300MB will be sufficient (including PDF manuals, etc.). For reasonable performance, 128MB of main memory (RAM) is a lower limit (per simultaneous user). Large data sets or intensive computations will require more memory.
1.2. S-Plus on Different Operating Systems
5
Table 1.1. S-Plus system requirements.
Operating systems (hardware)
System requirements (recommended minimum)
Windows 2000, Windows XP Home Pentium III, 512MB RAM, Edition, Windows XP Professional Edi- 450MB disk space tion, Windows 2003 Server (on Intel platforms) Solaris 2.8, 2.9 (SPARC processors) 160 MB RAM, 340 MB hard disk space Red Hat Enterprise Linux WS 3.0, ker- 160 MB RAM, nel 2.4.21, gcc version 3.2.3, glibc version 340 MB hard disk space 2.3.2 (Intel/AMD x86, 32 and 64 bit) AIX 5.1, 5.2 (POWER processors, IBM) 160 MB RAM, 340 MB hard disk space HP-UX 11.0, 11i (PA-RISC processor, 360 MB RAM, HP) 340 MB hard disk space The systems listed here are the officially supported systems for version 7.0. S-Plus may work other systems as well (for example, other Linux distributions). See the Insightful web page, http://www.insightful.com/, for further information. The minimum RAM requirements for Solaris, Linux, AIX, and HP-UX can be reduced to about 60 MB if S-Plus is used from the command line and the Java GUI is not used. As a side note, S-Plus consumes and releases memory dynamically during a session, depending on the needs. Therefore, it does not run out of memory until there is no more main memory and swap space available. If S-Plus runs out of main memory (RAM), the operating system assigns virtual memory (i.e., hard disk space) as a substitute. As this slows down the execution time dramatically, the machine should be equipped with a reasonable amount of memory. For improving performance, main memory is the first speed-up factor. If you are not satisfied with the performance, watch for permanent hard disk access while executing commands, or use a monitoring tool (like “top” under UNIX or the “Task Manager” under Windows) to track swapping activity. Differences Between Versions S-Plus has some differences in its implementation between the UNIX and the Windows versions. The most visible difference is the user interface. S-Plus for Windows has a typical Windows-like menu-based user interface. Many graphical functions can be started using the toolbar interface. Graphics are editable using point-and-click. Data and graphics transfer from and to other Windows applications (such as Word and Powerpoint) is smoothly
6
1. Introduction
integrated. Graphs can be put into Word or Powerpoint documents by using copy and paste, and if a large number of graphs is generated, the Powerpoint Wizard creates a Powerpoint presentation of all S-Plus graphs with a few mouse clicks. S-Plus for UNIX has a graphical user interface, too, but it differs slightly from the look and feel under Windows (due to its implementation in Java). It also lacks the toolbars, and graphs cannot be edited. In essence, it is straightforward to generate S-Plus graphs on Windows and include them in Windows document, and the same is true for S-Plus graphs on Unix and inclusion in documents that reside on the Unix machine. It requires a little extra effort if S-Plus runs on one system and the documents are edited on the other system. Unless you create a larger number of graphs it should not be too much extra work though. We will cover graph generation and inclusion in documents later on in more detail. Under S-Plus for Windows, the standard graphics device is a so-called graph sheet. The graph sheet can hold multiple pages. If some S-Plus code generates multiple graphs, they are shown on different pages and the user can switch between them by mouse clicks. S-Plus for UNIX does not have the graph sheet device. If a new graph page is created on the motif device, the page is cleared and the graph lost. The newly introduced Java device has that capability of holding multiple pages, too. S-Plus for UNIX allows easy integration of C and Fortran code by basically automatic compilation of a “shared object file” that is linked to the system at runtime. S-Plus for Windows allows one to dynamically link libraries (DLLs) with some more manual effort. S-Plus for Windows comes in two editions: “Professional” and “Standard.” The standard version does not provide command line access. S-Plus for UNIX has a special release called “Application Server.” This new implementation allows access to S-Plus via the Java interface (including the command line accessible from a Java interface that looks similar to the original command line). The application server allows users to access S-Plus using a Web browser. There are more differences, but most of them are not directly visible to the user. With very few exceptions, S-Plus code that runs under S-Plus for Windows will also run under S-Plus for UNIX, and vice versa.
1.3 Notational Conventions Before starting let us introduce a few notational conventions used throughout the book. To begin with, you must be aware that when running S-Plus, you will be asked for a new command with the greater than sign: >. If a single command extends over one line of input, S-Plus will determine that the command is incomplete and ask for completion. The prompt changes
1.3. Notational Conventions
7
to the plus sign (+) and you can enter the remainder of the command. A preview of this is shown below. > 3+(4*2 + ) We mostly omit the continuation prompt, as the code is easier to read and so that the + sign is not accidentally typed in as a “plus” operator. For longer program codes, we omit the prompt, too, as we do not assume interactive input but the use of an editor instead. S-Plus objects or expressions are written in a special font as with the example print. There are a few examples of commands to either the UNIX or DOS shells. For these examples, no prompt is used. When presenting commands, we sometimes include descriptive text. The descriptive text is written in S-Plus syntax for comments. Anything after the number sign # until the end of the line is treated as a comment and not interpreted as a command. A summary of these conventions is found in Table 1.2. Table 1.2. Notational conventions.
Convention Explanation S-Plus prompt Command has continued onto next line (displayed by S-Plus but omitted in this monograph) Commands S-Plus commands are shown in typewriter font No prompt For calls to the UNIX or DOS shell # Comment symbol indicating start of a comment placeholders Need to be replaced by an appropriate expression, such as filename, which needs to be replaced by a valid file name Ctrl-C Press and hold down the Ctrl key and press C Menu Menu entries and buttons are referred to in this font Note Notes point out something important, such as an example, an application, or an exception. The end of a note is marked by the symbol
> +
Finally, if S-Plus prints an object with dimensions – a vector or a matrix, for example – each row is preceded by the index of the following element, as in the following example where [1] indicates that the first element, 6, has the index 1. > 6:9 [1] 6 7 8 9 We mostly omit these indices to facilitate reading the output.
2 Graphical User Interface
2.1 Introduction S-Plus has an excellent graphical user interface (GUI) that allows the user to access its functionalities by pointing and clicking with the mouse. This convenient way of working with S-Plus was previously only available for the Windows platform but has now been added for the UNIX/Linux platform as well. The advantage of the GUI for the novice of S-Plus is that you don’t have to know the syntax of S-Plus to get started. All you need is a little familiarity with typical Windows software and a data set in some sort of standard format. The descriptions in this chapter are, for the most part, based on S-Plus for Windows. Users of UNIX/Linux will benefit from this chapter, despite it being aimed at the Windows user of S-Plus, as we will show typical applications and examples and point out major differences between the two operating systems. The chapter will end with a section dedicated solely to the GUI available with UNIX/Linux. The approach we will take in this chapter is to quickly introduce the S-Plus system design under Windows, show the briefest of explanations of how it functions, and finish by describing in detail the various tasks that will be needed to complete a data analysis, from data input to printing and saving the results. It is by no means intended to be an extensive or exhaustive exploration of the GUI but merely a way of familiarizing you
10
2. Graphical User Interface
with its structure, where to find things, and, most importantly, where and what to try for more detailed options.
2.2 System Overview When you open S-Plus under Windows by double-clicking on the S-Plus icon, you are greeted by a screen layout as shown in Figure 2.1. This may vary slightly according to the version of S-Plus you are using. The main elements that are visible include the Object Explorer, the Commands window, the menus, and the toolbar. Optionally, a graphics window can be opened. A short description of each of these components is given in the next several subsections.1
Figure 2.1. The S-Plus screen and its components: the Object Explorer, the Commands window, and the toolbars.
The general layout of the S-Plus system is similar to that of many popular Windows systems in that it has pull-down menus at the top and toolbars just below the menus. To use such a system it is useful to be a little familiar with basic point-and-click operations and how to use a mouse. 1 According to the version of S-Plus being used, some components might not be present or look slightly different.
2.2. System Overview
11
For those who are not that comfortable with window- and icon-based software, the following subsections provide a crash introduction to the essentials. The pull-down menus across the top are used to group categories of commands or options to be set. Under the File menu, we find actions relating to files (Open, Close, Import Data, Save) as well as to exiting the system. The toolbars below the menus contain buttons that are convenient shortcuts to commands found through the layers in the menus. Some of the toolbar buttons (e.g., 2D Plots) open a palette containing a myriad of options to complete your task. An online help facility is available through the main menu. The typical approach to handling the help system (i.e., search by index or topic) has been used. The big advantage of the help system in S-Plus is that it includes online manuals. We encourage liberal use of the help system.
2.2.1
Using a Mouse
Using a mouse efficiently is important to get the most out of the system. Clicking once on the left mouse button is usually used to highlight an item in a list (e.g., a file out of a list of files) or to select a menu heading or button from a toolbar. You will not always be able to guess at the function of a toolbar button merely by looking at the icon, but if you are at a loss, simply position the mouse over the button in question and a short text description will appear below it. Double-clicking on the left mouse button is used to select and execute. Examples include double-clicking on a file name to select it and start the function, or on a part of a graph to select and edit it. Clicking once with the right mouse button opens a context menu that changes depending on the item selected.
2.2.2
Object Explorer
The Object Explorer is used to get an overview of what is available on the system, including data, functions, and graphs. It operates in much the same way as the Windows Explorer in that there is a tree-like structure in the left pane and the details are displayed in the right pane. The Object Explorer can be opened either from the menu or from a toolbar button. Not only can it show what objects exist, but with data, for example, the Object Explorer is used to view (browse) it and even edit it. Double-click on the data you want to view or edit, and a spreadsheet will open containing the selected data. Once the spreadsheet is open, the data can be edited.
2.2.3
Commands Window
The Commands window is actually the heart of the S-Plus system. Every command that is performed via menus and buttons can be issued as a
12
2. Graphical User Interface
command directly from the prompt in the Commands window. In addition, there are many functions that can only be run in the Commands window. As examples, the functions of programming, personalized functions, data subsets, and logical operations are only available, or more extensively available, through the use of the Commands window.
2.2.4
Toolbars
The main toolbar contains many familiar commands, including Open, Save, Print, Copy, Paste, and Undo, and is shown in Figure 2.2. In S-Plus, however, one may also open the Commands window and Object Explorer, open a new graph sheet, open the History log, create 2D and 3D graphs, and so forth.
Figure 2.2. The S-Plus main toolbar.
An additional feature of the toolbars in S-Plus is that they are contextsensitive (smart). Open a graph sheet, for example, and extra buttons will appear in the toolbar area that are specific to graph sheets.
2.2.5
Graph Sheets
Graphs are drawn in windows referred to as graph sheets. Components of graphs can be edited and redefined by double-clicking on the component of interest. Labels can be changed, colors modified, axes redefined, and more, all through menus and dialog boxes available simply by clicking in the graph sheet. The Insert menu is useful for adding features and components to graphs that already exist in a graph sheet, and the Format menu is useful for changing the design of many components of an existing graph.
2.2.6
Script Window
The Script window also consists of two panes, one on top and one on the bottom. The top pane can be used as a development space in which to create or fine-tune a section of code. When the commands in the top pane are run, the output from them appears in the bottom pane. In function, the Script window is similar to the Commands window, the difference being that the former runs commands in segments, whereas the latter is completely interactive and executes commands one at a time as they are input. More details of the layout of the Script window and how it operates are provided in Section 2.16.
2.3. Getting Started with the Interface
13
2.3 Getting Started with the Interface If you want to use S-Plus having had no introduction, use the menus and toolbar buttons. For the most part, you should find them to be selfexplanatory. However, we have described a few key functions in a bit more detail to get you on your way.
2.3.1
Importing Data
You probably have your own data that you want to analyze, so the first thing you have to know is how to import it into S-Plus. There is an Import Data facility located in the File menu. In the From File dialog, you will be asked for the name you want for your data (which may be more than eight characters) and have the usual Windows boxes for specifying the name and location of your data file using the Browse button. Pay careful attention to the type of data (File Format pull-down menu) that you have and properly define it in the corresponding box. The data file types available are shown in Table 2.1. Table 2.1. File types available with the Import From File facility.
File ASCII Excel Lotus 1-2-3 MS SQL Server SAS Transport SPSS Portable
Format ASCII formatted dBase FoxPro Gauss Matlab Minitab Paradox QuattroPro SigmaPlot S-Plus Stata Systat
Epi Info Gauss UNIX MS Access SAS SPSS
When the spreadsheet containing newly imported data is closed, the new data automatically appear as a new entry in the Object Explorer.
2.3.2
Graphs
Graphs are created by connecting the data shown in the Object Explorer to a graphical function represented by a button in a palette. In the Object Explorer, the data are displayed with its full name, and after a click on the data object’s name, the elements of the selected data set are displayed in the right-hand pane of the window. By clicking on these elements, the set of variables to be plotted can be selected. To select a second and third variable, hold down the Ctrl key and click on the element. Before actually displaying the data graphically, the graph palettes need to be opened if they are not open yet. Open the 2D or 3D graph palette by
14
2. Graphical User Interface
using the appropriate icon in the toolbar. The 2D graph palette appears in Figure 2.3 and shows all the types of 2D graphs available.
Figure 2.3. The 2D graph palette.
Once the palette is open, you might want to move the mouse over the different icons slowly. If the mouse stops for a second, the name of the method represented by the button is displayed. Select the plot type by clicking on it. S-Plus executes the graphical method right away by using the selected data. Note For creating a graph, the order of selection of variables is important. The variable selected first becomes the x-variable, the second variable selected becomes the y-variable, and if a third variable is selected, it becomes the z-variable. Once data have been selected, another graphical method can be applied simply by clicking on a different graph button in the palette. Note If you click on a graphics button and a graph is set up but not drawn, this probably means that the data selected and the graph type chosen are not compatible. Selecting two variables and clicking on a 3D graphics method is an example of such a situation. Once a basic graph has been created, it can be edited. Double-click on the element to modify, for example, the axis. A window opens and offers the possibility of modifying all the components such as range, tickmarks, color, width, label, and much more.
2.3. Getting Started with the Interface
15
Using the Annotation palette that appears once a graph sheet is open, further elements, such as text, or graphics symbols, such as rectangles and arrows, can be inserted into the graph. A graph sheet may be printed using File - Print Graph Sheet... or saved using File - Save. All that is left is to specify the destination file name and directory.
2.3.3
Data and Statistics
Data can be summarized by using graphical display techniques but also by statistics such as the mean, median, minimum, and maximum values. There are Data and Statistics menus in the top menu bar that contain a large set of routines for manipulating and transforming a data set (Data) and routines to process the data (Statistics). First, select the statistical method to use and then specify the data set (or vice versa) and the options you want in the menu fields. The method is applied to the data by clicking on the OK button, and the results are displayed in a Report window. In this way, data can be summarized by looking at minimum and maximum values, at quantiles, or at the correlation between variables. Data can be processed by carrying out a t-test, a regression model, or any other method, and tick boxes can be ticked or not to choose options such as paired or unpaired or the degree of a polynomial to fit.
2.3.4
Customizing the Toolbars
The toolbars and palettes can be customized to some extent. It is possible to select which palettes show up on the screen. Select View from the top menu and click on Toolbars. A window opens that lets you choose which toolbars should be shown. Clicking on any of the selection fields immediately pops up or removes the corresponding palette on the desktop. These settings are stored and reused when S-Plus is started again. Be careful which ones are chosen, remember that the Graph toolbar will open when a graph is created, and so forth. The shapes of the palettes might not be satisfactory, depending on your screen resolution, personal preferences, and more. Clicking on one of the four edges allows you to change the size of the palettes, which is automatically set such that all icons fit. Furthermore, the palette buttons can be displayed in palette form or in toolbar form. Click on the palette and drag it to the top, where the primary toolbar is located. Drop it and the palette becomes a horizontal toolbar itself. Dragging the bar away from the top recreates a palette.
16
2. Graphical User Interface
Adding New Palettes and Buttons The palettes and menu buttons are open for extension. You can add menu buttons to an existing palette and create new palettes. You can generate new toolbars via the View menu by selecting the Toolbars entry and clicking on New. This produces a toolbar dialog box that can be used to create a new toolbar. You can add a new button by selecting the desired toolbar and opening the context menu with the right mouse button. The entry New Button opens up a window where you can specify the name of the button, the text to show when the mouse is over the button, and the S-Plus function to carry out. (You will learn more about functions as you proceed.) You can modify an existing button by selecting the button and opening its context menu (right mouse click). A property dialog displays the current settings.
2.3.5
Chapters
S-Plus gives you the ability to save files from different projects into different areas called chapters. Chapters provide an efficient mechanism for project management. All elements open during a session, such as Object Explorer and graph sheet, can be saved to a chapter such that on reopening the chapter, everything is as you left it. Since S-Plus keeps all data items ever created, we highly recommend the use of chapters as an easy way of organizing your work and switching between project files. The actual operation of chapters is quite straightforward. Under FileChapters you have the options of Attach/Create, Detach, and New Working. Create a new chapter by choosing the Attach/Create option and specifying the directory where it should be located. There is a space for defining a label for the new chapter that will appear in the search path. Be sure to choose position 1. S-Plus will create several subdirectories that it needs, including a .Data directory where data are stored. The new chapter is completely empty. We want to create some data in our new chapter so that when we switch between chapters we can verify that they actually perform the way we want them to and keep our projects separate.
Creating Data • Click on Data, then on Random Numbers... • For Data Set:, specify test • For Sample Size:, specify 50
2.3. Getting Started with the Interface
17
• Choose OK – The first entry should appear in the Object Explorer. • Click on Data, then on Transform • For Data Set:, specify test (chosen by default) • For Target Column:, specify Square • For Variable:, specify Sample • For Function:, specify Xˆ2 • Click on Add – Sampleˆ2 appears in Expression: • Choose OK – Can see that the test data sheet has been opened and contains the variables Sample (the original Gaussian distributed data) and Square (the square of the variable Sample).
The new data set appears in the right pane of the Object Explorer. We can switch back to the default chapter by choosing File-Chapters-New Working Chapter... and choosing the default user directory. “New” refers to the fact that you are defining which of the existing chapters to use, not that you are creating a new one. Does the newly created data set (test) appear in the Object Explorer of the default chapter? Switch back to the new chapter. Is test the only data set that appears in the Object Explorer here? Note The New Working Chapter option automatically detaches the old chapter as it attaches the new one. However, you do not have to detach the old chapter when accessing a new one. You could simply attach the new chapter and assign it to position 1 (File-Chapters-Attach/Create), moving the old chapter further down the search path. This action has two implications. Data sets from a nonactive chapter cannot be seen through the Object Explorer but can be accessed. If a data set in the active chapter has the same name as one in a nonactive chapter, the one in the active chapter will be accessed. In summary, the chapter feature is a convenient way to help keep yourself organized. We highly recommend that you begin using it right from the start. For practice with chapters, try creating one for each “chapter” of the book and keeping chapter-specific data sets in the respective S-Plus chapter.
18
2. Graphical User Interface
2.4 Detailed Use of the GUI Interface The previous section provided a quick introduction to some of the basic concepts involved with using the graphical user interface (GUI). Brief descriptions of how to perform several tasks were given, but without much in the way of specifics. The details of how to understand and work with the GUI are given throughout the following sections. We cannot emphasize enough that to really profit from this book, you need to be running S-Plus and following the examples while reading the book. Only by trying something on your own will you really learn it to the extent that you could do it on your own. We have left a lot of tasks to be done by the reader (you!) so as to build on the concepts immediately.
2.5 Object Explorer The Object Explorer plays the same role in S-Plus as the Explorer does in Windows in that the “contents” are displayed in a tree-like diagram on the left-hand side of the screen and the details on the right-hand side. Similar or related objects can be stored in folders, giving structure to your work. When an object is put into a folder, a link (shortcut) is created such that the actual location of the object is not changed, just its virtual location. Folders can be dragged and dropped between Object Explorers, between Explorer pages within an Object Explorer, or onto the toolbar as a button. Objects can be classified into three types: computational engine, interface, and document. Computational engine objects include data frames, matrices, lists, functions, and others, and are objects containing data or the functions that are used to process data. Interface objects relate to communication with the system and include search paths, menu items, toolbars, dialogs, and the like. Document objects refer to output from the system and include graph sheets, Reports, Scripts, and the like. All of the object types mentioned can be saved in folders in the Object Explorer. Folders can be organized according to object type, data set, project, model class, or in any way that seems to be helpful. Open the Object Explorer, if it is not already open, by clicking the Object Explorer button on the standard toolbar. It should look like the one in Figure 2.4. If you have not yet worked with S-Plus, the Object Explorer will be empty, and if you have, you will see different object types in the left pane and their details in the right pane. We want you to follow an example and to get practice exporting a data set, so we will take a built-in S-Plus data set, export it, and then import it before we begin analyzing it. First, we have to use the Object Explorer to find the data set.
2.6. Help
19
Figure 2.4. The Object Explorer toolbar.
S-Plus uses a system of “positions” that are searched sequentially until the desired object is found. A message that an object has not been found is only issued once all the positions have been searched. Position number 1 is the current working directory. By default, the Object Explorer is used to show the objects in the current working directory. Other positions are used to store general functions, statistical functions, other types of data sets, and the like. The data we want to use are a few positions down in the list because the built-in data sets are not meant to be worked on in a working directory but merely read from a safer location. The search path (written SearchPath in S-Plus) lists all the positions currently available to S-Plus. By examining the search path, we can find the built-in data sets, locate the one we want, and write it to the hard disk. The search path is conveniently located in the left pane of the Object Explorer. By default, the first directory shown in the SearchPath is the one that can be written to, the others are read only.
S-Plus Data Sets • Expand the search path list by clicking on the + next to the SearchPath icon in the left pane of the Object Explorer – The S-Plus data sets are located in the data directory (see Figure 2.5). • Expand the list of data sets by clicking on the + next to the data icon • Scroll down to find the Puromycin data set Double-clicking on Puromycin runs the application, which, in this case, opens a spreadsheet-like editor that can be used to view and edit the data. By single-clicking on Puromycin in the left-hand pane of the Object Explorer, a list of its variables is shown in the right-hand pane. The Puromycin data set contains the three variables conc=concentration, vel=velocity, and state, as shown in Figure 2.6.
2.6 Help At some point, you will want to use (or have to use) the help system in S-Plus, and we have reached a point where it is possible to explain this
20
2. Graphical User Interface
Figure 2.5. The SearchPath.
Figure 2.6. Details of the Puromycin data set.
simple procedure. Suppose we want to export the Puromycin data but don’t know how to start. Simply invoke the help system to find out how.
2.7. Data Export
21
Using the Help System – Exporting Data • Click on the Help main menu • Choose Available Help-S-Plus Help • Choose the Contents tab – Expand the section on Importing and Exporting Data – Choose Exporting Data Sets • Or, choose the Index tab – Type in “export” – Click on Exporting Data Sets – Choose Display at bottom • Or, choose the Search tab – Type in “export” – Click on List Topics – Click Export to File and choose Display
Using the Help System – Puromycin Data Details • Click on the Help main menu • Choose Available Help-Language Reference • Choose the Index tab • Type “pur” – Puromycin is highlighted • Choose Display at bottom – Window explaining the Puromycin data set appears The help system can be used in the same way to find information on (almost) any other topic. It is helpful to note that details on the built-in S-Plus data sets as well as on the Command window language are found in the Language Reference section of the help menu.
2.7 Data Export As promised, we’re going to export the Puromycin data, import it again, and then work with it. The entire process is menu-driven.
22
2. Graphical User Interface
Exporting Data • Click on Puromycin in the left-hand pane of the Object Explorer • Click on the File main menu • Choose Export Data-To File . . . • Use the Browse button to specify a location for the file, if desired • Specify the file type using the Files of type: pull-down menu – We chose SAS Version 7/8/9 - Windows (sas7bdat; sd7) and suggest that you do as well (you don’t need to have SAS installed on your computer). – Puromycin is already highlighted in From Data Frame: (see Figure 2.7).
Figure 2.7. Data export.
– The file extension and name are automatically updated to reflect the file type choice. • Choose OK The file should now be created with the name (Puromycin) and type (.sas7bdat) that we specified, residing in the directory of our choice. Use the Windows Explorer to verify that the desired data set has been created. Save the same data in different formats and open them with the appropriate program; for example, use Excel for *.xls files.
2.8. Working Directory
23
2.8 Working Directory When working on several different projects, it is often convenient to set up a different “directory” for each separate study on which you are working. Directories can be set up in two different ways: as shown in Section 2.3.5, or by defining multiple S-Plus icons on the desktop. The advantage of the former approach is that you never have to leave S-Plus to switch between project-specific directories, whereas the advantage of the latter approach is that the danger of inadvertently working in the wrong directory is minimized. The method of defining and switching between chapters is easy, flexible, and orderly (see p. 16), but, as people work in different manners, the method of defining a working directory and creating an S-Plus icon will be shown here.
Defining a Working Directory Create a copy of the S-Plus shortcut icon as follows: • Right-click on the S-Plus icon on the Windows desktop • Choose Copy • Right-click on open space on the Windows desktop • Choose Paste • Right-click on the new icon and choose Rename to give it a new name • Right-click on the new icon and choose Properties • Choose the Shortcut tab – Target: is highlighted and shows the command for running S-Plus. • After the existing text that is already in the box for Target:, add S PROJ=pathname – Use the path where you saved the Puromycin data. – The pathname must be enclosed with double quotes if there is a space in it. – The addition of /MULTIPLEINSTANCES will allow this session to run concurrently with other sessions, each using different working directories (chapters) • Start in: should be blank (see Figure 2.8) • Choose OK
24
2. Graphical User Interface
Figure 2.8. Defining a working directory.
You can now start S-Plus by double-clicking on the new icon, which will have the effect of working in the directory specified in the path name. Open the Object Explorer and verify that the new S-Plus icon is using the database (empty) defined by the new path name (we did this earlier).
2.9 Data Import After this preparation, we are finally ready to look at how to read a data set into S-Plus. The procedure is essentially the same as with exporting data and works in the following manner.
Importing Data • Choose File from the main menu • Choose Import Data • Choose From File . . . • Use the Browse button to locate the Puromycin data set (in SAS format) that we created earlier and highlight it by clicking on it (and choose OK if necessary)
2.9. Data Import
25
• Check that File Format appears as SAS Version 7/8/9 (sas7bdat) • Change the To Data Set name to puro to distinguish it from the source • Click on Update Preview to see a preview of the data that is about to be imported • Options can be chosen to change desired properties as in Figure 2.9 • Choose OK As soon as OK has been chosen, a Data window should open containing the data set displayed in typical spreadsheet fashion. It contains 23 rows and 3 columns (variables), just like the original data set.
Figure 2.9. Data import window.
As a reminder, you should always check the data to make sure that there are no unpleasant surprises. Notice that the puro data set is currently sorted by state and then by conc. It would make sense to have the data sorted by vel as well. The following steps can be used to sort a data set.
Sorting Data • Click on the Data sheet to be sorted
26
2. Graphical User Interface
• Choose Data from the main menu • Choose Restructure • Choose Sort . . . • Using the pull-down menu for Sort By Columns: – Click on state – Ctrl-click on conc – Ctrl-click on vel (see Figure 2.10) • Choose OK • Close Data window
Figure 2.10. Sorting a data frame.
See Figure 2.11 for the resulting sorted data set. Note that the first 2 rows both have state=treated and conc=0.02 but that now the first row has a smaller vel (=47.00) than the second row (vel=76.00). In fact, the original order of the rows appears in a column just to the left of the data, reflecting the fact that these two rows have switched positions.
Figure 2.11. Sorted Puromycin data.
2.10. Data Summaries
27
S-Plus refers to this data set as a “data frame.” Data frame is the term used by S-Plus to denote a rectangular collection of data, possibly including variables with different data types (e.g., character and integer). Many data sets typically encountered are actually data frames.
2.10 Data Summaries Now that we have a data frame in the Object Explorer, we want to “look” at the data in some sense. One of the easiest ways of getting a sense for your data is to generate simple summary statistics (mean, standard deviation, etc.). The following command box shows how this is done.
Generating Summary Statistics • Click on puro in the left-hand pane of Object Explorer to select it • Choose Statistics from the main menu • Choose Data Summaries – Options are Summary Statistics, Crosstabulations, and Correlations. • Choose Summary Statistics . . . – The Summary Statistics dialog opens to a data tab. – puro should be chosen as the default data frame (see Figure 2.12). • Choose All Variables (default) • Choose No Grouping Variables (default) – Can save results by supplying a name using Save As. • Choose Statistics tab at the top of dialog window. – Can choose specific statistics to be calculated (see Figure 2.13). • Choose OK A Report window is opened (Report1) containing the count of each level of the variable state and the summary statistics for the continuous variables conc and vel, as shown in Figure 2.14. We see, for example, that there are 12 treated and 11 untreated patients and that the mean conc is 0.3122. The contents of the Report window can be saved into a file by choosing File-Save As . . . and specifying a path and file name. The default file type is .rtf (rich text format), which can be opened by any standard editor or from within S-Plus.
28
2. Graphical User Interface
Figure 2.12. Summary Statistics data dialog.
Figure 2.13. Summary Statistics dialog.
2.11. Graphs
29
Figure 2.14. Summary Statistics Report window.
2.11 Graphs The problem with data summaries is that you don’t actually “see” the data itself. For a more visual summary of our data, we need to construct a graph. There are actually several different ways of constructing a graph in S-Plus using the GUI (menu buttons), but perhaps the easiest and most flexible method is to use the palettes. The palettes exist only with S-Plus for Windows, but the UNIX/Linux user can do the same things using the main menu options.
Creating a Graph Using a Palette • Open the Object Explorer if it is not already open • Expand the list in the left-hand pane of the Object Explorer such that the three variables of puro are visible in the right-hand pane • Click on conc to highlight it • Ctrl-click on vel so that both variables are highlighted – The order in which the variables are chosen (clicked) is important. The first will be used for the x-(horizontal) axis and the second for the y-(vertical) axis. • Click on the 2D palette button on the main toolbar
30
2. Graphical User Interface
• Click on the button for scatterplot (upper left) – A new graph sheet will open, containing a scatterplot of the two variables conc and vel. – If you don’t like the type of graph that you have chosen, simply click on a different one (e.g., line scatter, loess) and a new graph sheet will be opened with the same data but using a different plot type. Be adventurous and try several.
Although the default graph looks quite good, we’ll wait until after we explain some editing techniques before actually showing it. UNIX/Linux users don’t have the convenience of editing the graph after the fact. Instead, the details of the appearance of the graph have to be specified upfront. This leads us to the topic of how to edit a graph once it has been created. In particular, we are going to change the color and symbol for the points on the scatterplot so that they will be more visible.
Changing Color and Symbol of Points • Click on any point in the graph – A green knob will appear in the lower left of the graph. • Double-click on the green knob – A dialog window named Line/Scatter Plot [1] will appear, as in Figure 2.15. – The first tab for the window shows the Data to Plot panel. The data that are used in the plot could be redefined here. • Click on the Symbol tab • Using the pull-down menu, change the Style: to • Circle, Solid • Using the pull-down menu, change the Color: to Black (see Figure 2.15) • Choose OK – The symbol has changed both style and color.
Making the graph look “nicer” (depending on individual tastes) is usually of interest, particularly when it comes to labeling different parts of the graph. We will investigate how to change the axis labels, add a title, and add general text to the graph.
2.11. Graphs
31
Figure 2.15. 2D Plot dialog (symbol).
Changing Axis Labels • Click on the default x-axis label (conc) – It is now surrounded by green knobs. • Choose one of the following methods: – (A) One more click allows you to change the label by simply typing over the default (Default is automatically taken as the column name and is denoted ‘@Auto’). – (B1) Right-click on the default label (dialog appears). – (B2) Choose Edit In-place. . . (see Figure 2.16). • Change the text to “Concentration” • Click outside of the box to finish without hitting Return The label has now been changed and the text is better now; however, the font is rather small. We can easily change this.
Changing the Font • Right-click on axis label text • Choose Font. . .
32
2. Graphical User Interface
Figure 2.16. Axis dialog box.
• Define font size by varying Size: – Try 20 (see Figure 2.17). • Choose OK
Figure 2.17. Axis font dialog box.
We have now changed the label and font size for the x-axis and leave it up to you to repeat the procedure for the y-axis. Adding a title to a graph is most easily done using Insert from the main menu. The exact usage is explained below.
2.11. Graphs
33
Adding a Title • Click on Insert in the main menu • Click on Titles • Choose Main... • Type in the desired title (Use ‘Scatterplot’) • Click outside the title box to end • Change the font as desired (see Changing the Font) • Change the box as desired
Figure 2.18. Annotated scatterplot.
Graphs in S-Plus are designed to be interactive in more than one sense; that is, we created the graph to learn something about the data and want to get information out of it. One of the nicest features about S-Plus is that the graphs are not static; they are not merely nice pictures to look at and study. Try positioning the cursor over a point and holding it there for a second. What happens? A small box appears that contains three pieces of information: the index number and the x and y coordinates. The index
34
2. Graphical User Interface
refers to the number of the entry in the data sheet. Choose some points and look at their values and indices. There are two very useful buttons on the graph sheet toolbar that open the Graph Tools palette and the Annotation palette. By holding the cursor over the individual buttons in the palettes, you can see what their functions are. Explore the possibilities open to you with these and other buttons. We created the graph in Figure 2.18 with just a few clicks of the mouse. Now that the graph looks exactly the way we want, we’re ready to print it and save it for possible future use.
Printing a Graph • Click anywhere on the graph sheet to make it the active window • Choose File from the main menu • Choose Print Graph Sheet. . . – Make sure the settings for the printer are correct. • Choose OK
Saving a Graph (Sheet) • Click anywhere on the graph sheet to make it the active window • Choose File from the main menu • Choose Save As. . . • Specify the location and file name for the graph (see Figure 2.19) • Choose Save We have just saved the graph into a file, so we should be able to read the graph back into S-Plus. A file extension was not explicitly added to the file name, but the association was established with Windows so that the file is an S-Plus graph sheet. We leave it up to you to open the graph from the file that you just created. Another useful task that is often done is to create a graph in S-Plus for use in another application (e.g., MS Word). There are two ways of exporting a graph from S-Plus: the direct method and the indirect method. The quality is better with the indirect method but takes a couple of extra steps.
Exporting a Graph – Direct Method • Open a graph sheet and make sure it is the active window
2.11. Graphs
35
Figure 2.19. Saving a graph sheet.
• Press Ctrl-C (or Edit-Copy or Send Graph to Other App button from main toolbar) to Copy • Move to MS Word (or other Windows application) • Press Ctrl-V (or Edit-Paste) to Paste – The Graph is now in MS Word.
Exporting a Graph – Indirect Method • Open a graph sheet and make sure it is the active window • Choose File from the main menu • Choose Export Graph. . . – From this window, you can choose from many different file types. MS Word, for example, can easily read a Windows Metafile (*.WMF). • Specify the file type, file name, and location • Choose Export – A Windows Metafile can be inserted into MS Word using InsertPicture-From File and then specifying the file name and location. According to what software you have available on your system, you should try saving a graph using different file types and importing it into the appropriate package.
36
2. Graphical User Interface
2.12 Trellis Graphs Trellis graphs are a convenient way of exploring relationships in data. They work by graphing one or more variables while conditioning on one or more others. Panels are constructed for each level of the conditioning set, and a plot of the variables of interest is made in each panel. An example of their use might be to plot height and weight, but to have one panel for males and one for females. In this manner, it is then easy to see if the relationship between height and weight is the same between the sexes. We will now show how to make a Trellis graph, although we have devoted an entire chapter (Chapter 6) to it later.
Trellis Graphs – Drag-and-Drop (Windows only) • Create a scatterplot from the Puromycin data as we have done before • Left-click on state in the right-hand pane of the Object Explorer and keep the mouse key depressed • Drag the mouse to the upper portion of the graph sheet until a dashed rectangular box appears, as in Figure 2.20
Figure 2.20. Trellis graphs – drag-and-drop.
2.12. Trellis Graphs
37
• Release the mouse • The graph is automatically transformed into a Trellis graph, as in Figure 2.21
Figure 2.21. A Trellis graph conditioning on state.
There is a small trick to producing Trellis graphs using the menu; we will show that approach separately.
Trellis Graphs – Using the Menu • In the Object Explorer, click on the variable to be used on the xaxis (conc). In the Object Explorer, Ctrl-click on the variable to be used on the y-axis (vel). In the Object Explorer, Ctrl-click on the conditioning variable (state). • Click on Graph-2D-Plot in the main menu • An Insert Graph dialog appears • Change the axes type to Panel (the default is Linear) • Proceed as before to create a graph as in Figure 2.21
38
2. Graphical User Interface
The only difference in the final graphs between the two approaches is that the same symbol is used in all panels with the drag-and-drop method, whereas each panel gets a different color/symbol using the menu approach. The resulting message that we can draw from the Trellis graph is that the relationship between concentration and velocity is the same for state=treated as it is for state=untreated.
2.13 Linear Regression We have already looked at the summary statistics for the variables in our data set and at a scatterplot of concentration versus velocity. We are now ready to try to fit a linear regression model to our data.
Performing Linear Regression • In Object Explorer, click on data frame puro • Choose Statistics • Choose Regression • Choose Linear . . . – Puro is the default data frame. – Tabs for specifying Model, Results, Plot, and Predict (see Figure 2.22) • Note that a default formula for the regression has been chosen, Delete the formula as we will create a new one. • Choose Create Formula • In the Variables section, click on vel • In Add option, choose Response – Velocity will be the response (y) variable. • Click on conc • In Add option, choose Main Effect: (+) – Concentration will be the predictor (x) variable. – Can transform variables, add quadratics, and so forth. – In the Remove section, can tick box for removing y-intercept (see Figure 2.23) • Choose OK • Try the following options: • In the Save Model Object section, specify a name for storing the regression object (use the name my.reg).
2.13. Linear Regression
Figure 2.22. Linear Regression dialog.
Figure 2.23. Regression Formula dialog.
39
40
2. Graphical User Interface
• Click on the Results tab to move to that sheet • Tick ANOVA table (as an option) – Can also save fitted values and residuals • Click on Plot tab to move to that sheet – Can choose which plots to look at. None are ticked by default, but this doesn’t matter, as we’ll see later. • Click on Predict tab to move to that sheet – Can save predicted values and confidence intervals • Choose OK The Linear Regression dialog opens a new Report window, shown in Figure 2.24. Looking in the Report window at the ANOVA table, we see that conc has an F -value of 38.8 and a corresponding p-value of 3.5×10−6 . At first glance, it appears as though the model provides an excellent fit to the data. Recall from the scatterplot, however, that the relationship between the two variables does not look particularly linear. We should explore the fit of the linear regression model further by looking at certain graphical representations (i.e., the residual plots).
Figure 2.24. Regression Report window.
You will recall that we did not specify that any of the residual plots should be created, but we do not have to start from the beginning. Notice
2.13. Linear Regression
41
that the object my.reg now appears in the right-hand side of the Object Explorer. Right-click on my.reg and choose Plot.... The original menu for the residual plots appears, and we can now specify which plots we want to explore. Choose Residuals vs Fit, Response vs Fit, Residuals Normal QQ, and Cook’s Distance as a start by ticking the appropriate boxes and then choosing OK. A graph sheet is created with four pages, one for each plot (not shown). Note It should be mentioned at this point that graphs produced by statistical dialogs can be either editable or noneditable. The noneditable versions have the advantages of being quicker to produce and requiring less space for storage, but the disadvantages are that they cannot be edited to change labels, add annotations, and so forth. Since most computers have ample computing speed and space, we recommend the use of editable graphs. The desired type of graph can be defined using Options - Graphs Options... and ticking (or unticking) the desired default. The option Create Editable Graphics appears twice in the window and both should be ticked or unticked (as desired). The next section will not be possible to perform unless the residual plots were created as editable graphs (i.e., change the option and redo the graphs above). If we look at page 1 of the graph sheet, however, notice that the plot of residuals (y-axis) versus fitted values (x-axis) does not look like a cloud of points randomly distributed about the line y = 0. Furthermore, three points have been indicated as “suspect”: points 1, 10, and 23. A smoothed line has been added to make it all that much more apparent that there is a pattern to the points. Page 2 of the graph sheet shows the response variable (vel) versus the fitted values (x-axis) with an overlaid regression line (dotted) and a smoothed line (solid). Notice that the points, from left to right, are mostly below the line, mostly above the line, and then mostly below the line again. This trend indicates a poor fit of the model. Page 3 shows the residuals (y-axis) versus the quantiles of the standard normal (x-axis), which also indicates the points 1, 10, and 23. The fourth page contains Cook’s distance (y-axis) versus an index (count of the order of the points) on the x-axis. Here, we see that points 1, 13, and 23 have the largest values of Cook’s distance. The graphs suggest that the simple linear regression provides a poor fit to the data, but you will perform a better analysis in the exercises. This is a good place to mention that we can put multiple plots on a single graph page by copying and pasting. This feature is particularly relevant and helpful when trying to deal with a multipage graph sheet.
42
2. Graphical User Interface
Placing Multiple Graphs onto a Single Graph Sheet (Windows only) • This will not work if you have not followed the instructions in the Note above concerning editable graphs! • Click New on the toolbar • Click on graph sheet • Choose OK • Click on Window from the main menu and then Tile Vertical (just to simplify the copying process) • Click on page 1 of residual plots to highlight it • Choose Edit from the main menu and then Copy Graph Sheet Page • Click on the header of the target (empty) graph sheet to highlight it • Choose Edit from the main menu and then Paste – The graph is now copied in the target graph sheet. • Repeat the previous three steps to copy pages 2–4 onto the target graph sheet. – Notice how space is made at each step to accommodate each new graph. • Save the new graph sheet (calling it residuals). The result is shown in Figure 2.25.
2.14 PowerPoint (Windows Only) If you intend to make an MS PowerPoint presentation that includes graphs you have created in S-Plus, then your task has been made easy for you by an S-Plus wizard designed to set this up for you.
Putting Graphs into PowerPoint • Click on the PowerPoint Presentation button on the toolbar (upper right) – Creates PowerPoint presentation of selected, saved graph sheets – Opens the S-Plus PowerPoint Presentation wizard • Choose Next • Choose Add Graph. . .
2.14. PowerPoint (Windows Only)
43
Figure 2.25. Combining graphs onto one graph sheet.
• Choose scatter.sgr • Choose Add Graph. . . again • Choose residuals.sgr – Could add additional graphs using the same process • Click on Next • Choose Finish – PowerPoint is opened and creates the slides. • When complete, exit the wizard • It asks to save the lists of graphs – you can, but choose No • PowerPoint is open and is the active window • Choose File from the main menu (in PowerPoint) • Choose Save As. . . • Specify the file name and location for the PowerPoint file If several graph sheets have been saved, they can be put into the same PowerPoint presentation or one per presentation if you prefer. If a graph
44
2. Graphical User Interface
sheet containing more than one page has been saved, PowerPoint will create one slide per page.
2.15 Excel (Windows Only) The Windows-based version of S-Plus also installs add-ins for MS Excel. You should notice a floating toolbar in Excel with three buttons as well as a new entry for S-Plus in the main menu. These tools give you S-Plus graphics capabilities from within Excel. If you have a data set in Excel and want to produce S-Plus graphs, you don’t have to transfer your data to S-Plus but can stay in Excel and use the add-ins to produce your S-Plus graphs. The three buttons in the S-Plus toolbar (within Excel) are for creating a graph, modifying the layout of a graph, and modifying a plot. There are essentially three steps to create S-Plus graphs from within Excel. In Step 1, you are asked for the range of the data to be plotted. Keep in mind that you should specify the upper-left and lower-right corners of your block of data, as in A1:B10. If you highlight the data you want to plot before starting the process, the data range will be prespecified. In Step 2, you are asked to specify details for conditioning the data if you don’t want to plot certain data items. The last step is where you specify the graph and plot type, which uses the same dialog boxes as in S-Plus. The best way to test this is to open the Puromycin Excel file we created earlier and give it a try. There are now three options for working with Excel: importing an Excel data file (which converts the data to an S-Plus data frame), as explained earlier; using Excel directly to produce S-Plus graphics while in Excel, as explained above; or linking to an Excel data file while in S-Plus. The last option is discussed below.
Linking to Excel Files • Click on File and Open. . . • Choose the Puromycin Excel file – An Excel sheet is opened within S-Plus – Can see letters as column headers – Can enter Excel formulas, etc. • The following new buttons have been added to the toolbar (see Figure 2.26) – Left button starts the Excel to S-Plus Link Wizard – Middle button updates a link – Right button removes a link
2.16. Script Window
45
Figure 2.26. The Excel toolbar.
Running the Link Wizard creates a link so that the Excel data can be accessed in future S-Plus sessions. Be sure to tick the box Allow S-Plus to store link information in Excel. Selecting the Excel data name and then clicking on the Update Link button will allow S-Plus to update its version of the data to reflect any modifications that were made in Excel. Other S-Plus functions can now be run directly on the Excel data.
2.16 Script Window The Script window offers an easy way to build and run a program or to keep track of what has been done so far. The idea behind this type of window is that it is a place where commands can be entered, run, and saved, with the output appearing in a separate section of the window. It can also be used with GUI-style commands, but, as these can be quite cryptic, its greatest use is to be run in conjunction with line commands covered in the rest of the book.
Opening a Script Window • Choose New from the main toolbar • Choose Script File • Choose OK There are two panes in the window: the upper pane is used for entering and running commands, and the lower pane is for showing output. In the example to follow, you should open the Scatterplot graph sheet and choose Window (main menu)-Tile Vertical so that both the graph sheet (containing the scatterplot created earlier) and Script window can be accessed at the same time.
Viewing GUI Commands in the Script Window • Click and hold the mouse button on the scatterplot • Drag the mouse to the upper pane of the Script window • Release the mouse button – All graph options used for the scatterplot appear (see Figure 2.27)
46
2. Graphical User Interface
Figure 2.27. The Script Window.
Saving GUI Commands from the Script Window • Click in the upper pane of the Script window to make it active • Choose File from the main menu • Choose Save As. . . – Default file type is *.ssc • Specify the file name and location • Choose Save The GUI options and commands that have been shown here are sometimes difficult to interpret and are beyond the scope of this book. It can be helpful, however, simply to view what the current settings are at any given point in time. The real power and usefulness of the Script window in the context of this book is to edit and submit segments of code. Up until now all the “commands” we have entered have been in the form of mouse movements and interacting with the GUI. The commands to be entered in the Script window are more akin to programming language commands on which the S-Plus processor then acts. Suppose you have a data set x consisting of one vector and you wanted to calculate the mean of x. You would choose
2.17. UNIX/Linux GUI
47
Statistics, Data Summaries, and then Summary Statistics.... However, you could also ask for the mean to be calculated directly by specifying mean(x) in either the Commands window (executed with the Enter key) or in the Script window (executed with the Run button). You may have noticed when you opened the Script window that three buttons were added to the toolbar, as in Figure 2.28. The left button has a right arrow for running scripts, the middle button is to stop the processing of the commands in the script window, and the right button is for finding specified text.
Figure 2.28. Script Window toolbar.
Running Commands in a Script Window • Clear the existing text in the Script window – Choose Edit from main menu – Choose Select All – Delete • In the upper pane, enter mean(1:10) – Calculates the mean of the numbers from 1 to 10 • Click on the Run button – Command and output appear in lower pane
In this manner, you could work on groups of commands to fine-tune a certain process and submit them as a group. It is a good way to work when building a function or longer program, tasks that will appear towards the end of the book.
2.17 UNIX/Linux GUI The GUI available with S-Plus under UNIX/Linux works in more or less the same manner as under Windows. The biggest difference is that it doesn’t offer all the functionality as under Windows. Rather than point out all the similarities and differences between the two GUIs, we will simply
48
2. Graphical User Interface
provide an overview of how to use the GUI under UNIX/Linux, and the similarities and differences will become apparent.
Help System The help system on UNIX/Linux is slightly different than on Windows in that there is really only one menu choice. Click on Help from the main menu and the three choices are Contents, Index, and Search. It doesn’t matter which of the three choices is made as they are simply the three tabs of the window that is opened. The Index tab is used to look at functions and built-in S-Plus data sets, the Contents tab gives an overview, and the Search tab can be used to search for information on any topic.
Chapters Chapters must be specified manually. This means that if you are working on two different projects and want to create two chapters to keep them both separate, you have to create the chapters manually outside of S-Plus. Fortunately, the procedure is quite easy. Simply move to the desired directory and issue the UNIX/Linux command Splus CHAPTER. This will have the effect of creating a .Data subdirectory in which S-Plus will store data from that chapter. To access data from that chapter, simply invoke S-Plus from within that directory with the command Splus -g &. When S-Plus is started, it opens with only the Commands window open and should look like Figure 2.29.
Figure 2.29. The UNIX/Linux Startup window.
2.17. UNIX/Linux GUI
49
Objects Summary Window With the UNIX/Linux GUI, there is no Object Explorer; instead, there is an Objects Summary Window. To see the types of information it provides, we need some data. Go to the Commands window and type x 65) range from roughly 1 to 4.8. You could also group the data by looking at the distribution of waiting times for short durations as opposed to long durations. We chose to split the data according to a waiting time of 60 minutes such that all points to the left of this defining line have short waiting times and those to the right have long waiting times. The remaining task is to record all of this onto the scatterplot using the Annotation palette in Figure 2.53.
Adding Annotations to a Graph • Click on the Annotation palette in the toolbar • Using the Line Tool, draw a vertical line starting at the waiting time of 60 minutes. Click on Select Tool to turn off the Line Tool.
2.20. Solutions
Figure 2.52. Geyser scatterplot.
Figure 2.53. The Annotation palette.
69
70
2. Graphical User Interface
• Double-click on the line and change the Style, Color, and Weight, as in Figure 2.54, and then choose OK
Figure 2.54. Plot Line dialog.
• Use the Rectangle Tool to draw a box around the points on either side of the reference (vertical) line. What do you notice? Click on Select Tool to turn off Rectangle Tool. • Use the Comment Tool to label the rectangular boxes just drawn. Click on Select Tool and then change the font, text, and scale as desired. Experiment with the various fonts (choosing Apply, rather than OK, simplifies the experimentation process). Choose OK when you are satisfied with the results. • Use the Arrow Tool to connect a label to its corresponding box and click on Select Tool to end. Change the color of the arrow (and arrowhead) to black and choose OK. • Save the graph sheet
If you drew and labeled in a manner similar to us, you should now have a graph that looks something like the one we obtained in Figure 2.55. The take-home message here is that if the waiting time is short (< 60 minutes), then the eruption should be fairly long (4 minutes or more). If, however, the waiting time is longer (> 60 minutes), there is no saying how long the duration might be. We leave it to the reader to write the group means on the plot as well as a linear regression line. Saving the graph sheet into a PowerPoint presentation was covered earlier.
2.20. Solutions
Figure 2.55. Geyser scatterplot with annotation.
71
3 A First Session
The remainder of the book is devoted to the use of S-Plus via the command line. We will look at how to use S-Plus as a calculator for simple and complex calculations. We will look at different data structures and how to handle data flexibly. We will generate graphs displaying data and mathematical functions, explore data sets in detail, and use statistical models. Finally, we will import data in several ways, automatize tasks by writing functions, and investigate many details that can help a lot in practice. Our first session begins with some general information on using S-Plus, including how to start and quit and how to access the help system. This is followed by some useful commands and how to do basic arithmetic. Logical variables are covered, and the chapter ends with a review of the material from the chapter, followed by hands-on exercises. To greatly enhance your learning of S-Plus, you should be seated at your computer while completing the exercises and also while reading each chapter. The exercises offer a good opportunity to try out the commands yourself and get acquainted with the system.
3.1 General Information We begin with general information on how to start and quit the system and how to access the online help facility.
74
3. A First Session
3.1.1
Starting and Quitting
The very first thing you need to know is how to start S-Plus. If you have S-Plus on Windows, all you have to do is double-click on the S-Plus icon (or use the taskbar) and a new S-Plus window will be opened that should look approximately like the one in Figure 2.1 (p. 10). If you’re running S-Plus under UNIX, you need to create an S-Plus chapter. A chapter is basically a directory tree containing all objects you are going to create: data, functions, and more. S-Plus should always be started from a chapter. Log on to your UNIX system and create an S-Plus directory, mkdir S move to the new directory, cd S initialize it as a new S-Plus chapter, Splus CHAPTER and start up S-Plus from there. Splus Alternatively, enter Splus -g to start S-Plus with a graphical user interface. You don’t have to repeat the creation and initialization of the S-Plus chapter again. Just change to the directory and start up S-Plus next time. Note You can start S-Plus without defining chapters and simply work in the default chapter. The effect is that you will work in the same chapter for all of your projects, leading to a confusion of the objects that you created for different projects. We highly recommend having separate chapters for each of the projects you work on. The procedure for defining chapters is described above for UNIX and described on page 23 for Windows. In the Commands window, the S-Plus command prompt > (the “greater than” sign) appears and indicates that S-Plus is ready to accept your commands. Now that you have started S-Plus, you will, at some point, want to quit. You can quit S-Plus for Windows by clicking on the X in the upper righthand corner of the window, double-clicking on the upper left-hand corner of the window, or clicking once on File and once on Exit. The standard Windows combination Alt-F4 works as well. Another alternative is to type
3.1. General Information
75
> q() at the command prompt. The latter works under S-Plus for UNIX as well. A quicker exit is to use Ctrl-D (to quit the S-Plus shell).
3.1.2
The Help System
If you get stuck, don’t panic. S-Plus has an online help system. To start the help system, simply type help() at the S-Plus command prompt (or help.start() under UNIX). The help system is designed such that you do not have to know exactly what you’re looking for when you access it. It has a table of contents with commands categorized by function and a search system that operates by typing in the command or subject on which you want information. Based on the subject, rather than the specific command, you can get information on your topic of interest. If you prefer to use the mouse, the help system can like be started by clicking on Help in the main menu or on the help button on the main toolbar. Depending on the version of S-Plus being used, the help system might be set up differently. If you know the name of a specific S-Plus command, entering help(commandname), or alternatively ?commandname, will open a new help window with instructions on the functionality of commandname. If you are using Windows, you have the option of starting the help system by doubleclicking on the Help icon in the S-Plus folder. This enables one to use the help system without actually starting the entire system. The help(commandname) form of accessing the help system is especially useful when you know the name of a command but have forgotten the specific options it has, the input it requires, or the output it produces. The general help() command is useful if you don’t know an actual S-Plus command name but only a particular subject.
3.1.3
Before Beginning
We already covered notational conventions in the introductory chapter. Here, we present a few hints to keep in mind before you continue, a couple of which repeat the notational conventions. We present all S-Plus commands in typewriter font, and they may appear either within the body of the text, as in help(), or on a separate line with the S-Plus command prompt, as with > help() Most commands in S-Plus contain opening and closing parentheses because they are calls to functions. Options to the commands are quite often included between the parentheses to specify further how a command should be used. The parentheses delimit the command and signify that an execution is to be made. If the
76
3. A First Session
parentheses are omitted, the contents of the variable (if data) or the definition of the command will be printed. Try typing help with no parentheses, and the internal commands associated with it will be printed on the screen. It will sometimes be necessary to add a comment to either an S-Plus command or the output from one. The comment will be denoted with the number sign (#). Note This convention of writing notes is to catch your eye and draw your attention to something that is important – a helpful hint, or an exception to something just mentioned. The “” symbol is used to mark the end of a Note.
3.2 Simple Structures In this section, we show how variables can be assigned and simple data structures may be generated. We begin, however, by explaining how S-Plus handles simple arithmetic operations.
3.2.1
Arithmetic Operators
Arithmetic operators in S-Plus function in the same fashion as in most programming languages and like most calculators. For example, the plus sign (+) is used for addition, the minus sign (-) for subtraction, the asterisk (*) for multiplication, and the slash (/) for division. To raise values to a certain power, use the caret (ˆ) or, alternatively, two asterisks (**). For simple expressions, multiplication and division are evaluated before addition and subtraction. However, raising to a power takes precedence over all of them. You can experiment with the symbols and rules above by using S-Plus as a calculator. Suppose you want to multiply 7 by 3. Using the asterisk as mentioned above, enter > 7*3 [1] 21 and you are immediately shown the answer, 21. Note The answer in the example above includes the counter, [1]. For each line of output from S-Plus, a counter is included to number the first entry on that line of output. Thus, when variables with lots of entries are printed, it is easier to find a specific entry.
3.2. Simple Structures
77
Note Now that you understand the appearance of the counter in the S-Plus output, it will be omitted from (most of) the rest of the examples in the book for ease of presentation. Practicing a little more with arithmetic operators, you see that you must pay attention to the order of operators. Thus, the example > 7+2*3 13 yields an answer of 13 because multiplication is evaluated before addition. If you really wanted the 2 to be added to the 7 before multiplying by 3, parentheses must be included to demarcate which will be evaluated first. Thus, in the example > (7+2)*3 27 the parentheses separate the operation that has to be evaluated before multiplying by 3. We present additional simple examples below to illustrate the rules above. > 12/2+4 10 > 12/(2+4) 2 > 3ˆ2 9 > 2*3ˆ2 18 These examples already show how S-Plus works; that is, commands are entered and the response follows immediately. You do not have to write a program before you start (but you may).
3.2.2
Assignments
The act of assigning a value or a function to a variable name occurs all the time in S-Plus, and it is important that you be familiar with it. In mathematics, you would write x=3 to specify that a variable x equals 3. In S-Plus, there are several alternatives. The statement > x x3 Note For this reason, the underscore cannot be used as part of a variable name, as in my data. Use my.data for these cases. Throughout the book, we use “ x=3 is perfectly valid. Normally, you also need to see the value stored in a variable. This is done in one of two ways: either enter the name of the variable or explicitly print the variable with the print function. > x 2 > print(x) 2 The first method is easier and quicker, but the latter may have to be used in certain programming situations. Note All assignments in S-Plus remain until explicitly removed or overwritten. The rm command may be used to explicitly remove a variable. For example, rm(x) removes the variable x. Note The value of a variable may be changed at any time.
As an example of these concepts, we change the value of x and then remove it with the following: > x x 4 > rm(x) > x Error: Object "x" not found
# overwrite the previous value
# remove x
Note All assigned variables are written to disk and stored in the current working directory. This means that you can quit S-Plus at any time without having to remember to save your data, as it is done automatically.
3.2. Simple Structures
79
When you restart S-Plus, all data from previous sessions are still intact and ready to be used. How about trying it out now? The previous note brings up the point that, after a while, you may begin to forget the variable names you have already used. In this case, it is very easy to use the objects function to get a list of all the variable names currently being used in the data directory. If you used the command at this point, you would find the following: > x objects() "x" The function has a few optional parameters whereby you can search a different database (and look at all the S-Plus functions, for example) or search for only a subset of variable names. It would be good practice with the help system for you to find out about these options on your own.
3.2.3
The Concatenate Command: c
Up to this point, you have worked only with single numbers (called scalars). Although it is possible to work in this fashion, it is often advantageous to work with vectors, which are simply ordered collections of numbers. Suppose you have the numbers 1.5, 2, and 2.5, and you want to work with the square of each value. You could save each number into a variable and square each variable, or you could put all three numbers into a vector and square the vector. The latter is much quicker if you have an easy way to combine the numbers into a vector. The concatenate command (c) can be used to create vectors of any length. You simply have to list the elements to be combined, separating each element by a comma. To perform the simple task above, we create x and square it. > x x 1.5 2.0 2.5 > xˆ2 2.25 4.00 6.25 The c command can be used with variables. If we forgot a number, it could be added easily. > x x 1.5 2.0 2.5 3.0 Note The c command can be used with any type of data, like the character values below.
80
3. A First Session
> y y "This" "is" "an" "example" The c command works with a mixture of numeric and character values. On combining different types of data, S-Plus converts them all to the most general type. Therefore, the result is a character vector. > z z "1.5" "2" "2.5" "3" "x" The consequence of converting all elements to character strings is that numeric operators may no longer be applied to the vector. If you are running S-Plus under Windows and have had the Object Explorer open, you should have noticed that the list of variables in the data folder increased with each new variable.
3.2.4
The Sequence Command: seq
It is often useful to work with regular sequences of numbers. This could include the numbers from 1 to 10, the years 2001 to 2010, or the numbers from 0 to 1 using an increment of 0.1 (0.0, 0.1, 0.2, and so on). All of these examples are easily done with the seq command. The general syntax of this function is seq(lower, upper, increment). Our three examples are then > seq(1, 10, 1) 1 2 3 4 5 6 7 8 9 10 > seq(2001, 2010, 1) 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 > seq(0, 1, 0.1) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 It is helpful to know that the default increment used by the seq function is 1. Hence, we can shorten the first two examples slightly by > seq(1, 10) 1 2 3 4 5 6 7 8 9 10 > seq(2001, 2010) 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 There is a lower default value of 1 used by the seq function, so the first example can be further simplified to > a a 1 2 3 4 5 6 7 8 9 10
3.2. Simple Structures
81
The best shorthand for the seq function is to use the colon (:). Simply putting the lower value, a colon, and the upper value is equivalent to the seq function with an increment of 1. > b b 1 2 3 4 5 6 7 8 9 10 When using the colon operator, the lack of an increment is not a problem. > 0:10/10 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Compare the output obtained here to that from the third example above. Sequences can also be constructed in descending order as seen in the following two examples. > seq(5, 1, -1) 5 4 3 2 1 > 5:1 5 4 3 2 1 The seq function has an option that is sometimes useful. If you know how long the sequence should be but don’t know the exact upper value, use the length option as below. > seq(0, by=0.03, length=15) Thus, this sequence starts at 0 and goes up in increments of 0.03 until the vector contains 15 elements. The length option can be used to obtain equally spaced intervals. Consider our example above, but suppose that we want 15 values to be equally spaced between 0 and 1 (creating 16 intervals). What would you do?
3.2.5
The Replicate Command: rep
It is useful to generate data that follow a regular pattern. The replicate function takes a pattern and replicates it. The general form of the command is rep(pattern, number of times). > rep(1, 5) 1 1 1 1 1 > rep(c(0, 6), 3) 0 6 0 6 0 6 > rep(c(0, "x"), 3) "0" "x" "0" "x" "0" "x" > rep(1:3, 4) 1 2 3 1 2 3 1 2 3 1 2 3 The second option for the number of times to replicate the pattern need not be a scalar. If it is a vector, it should have the same number of elements as
82
3. A First Session
the pattern, and then each element is replicated the corresponding number of times. > rep(1:3, 1:3) 1 2 2 3 3 3 Another interesting example, left to the reader to try and interpret, can be constructed by invoking the rep function twice. > rep(1:3, rep(4, 3)) Note Sometimes, the length of the desired vector is known rather than the number of times a pattern should be replicated. In this case, the length option can be used in place of the number of times option. Suppose we want a vector with 10 elements using the pattern 1, 3, 2. To do this, we would use the length option as in the following example. > rep(c(1, 3, 2), length=10) 1 3 2 1 3 2 1 3 2 1
3.3 Mathematical Operations We have already seen how S-Plus can be used as an expensive calculator by simply entering the numbers and the desired operator(s). The example shown was to multiply 7 by 3 as in > 7*3 21 In Section 3.2.2, we learned how to save numbers of interest into variables for possible future use. The example above can be redone with this idea in mind. > > > >
a 2+a 2 7 12 17 22 Take a and b to be defined as in Table 3.1, and then see what happens with a third vector in the following example: > c a+b+c 3 9 15 21 Everything so far, although explanatory, has not been mathematically challenging. What if we had to evaluate the function 3x2 + 2y f (x, y) = (x + y)(x − y) for many values of x and y? You could take each pair of x and y and calculate f (x, y) for each pair, but that is rather tedious. S-Plus works in terms of vectors, and it is to your great advantage to do so as well. To
84
3. A First Session
approach the problem above, we need to build vectors x and y so that they contain our values of interest. > x y x 2 4 6 8 10 > y 1 2 3 4 5 We now create a vector z that evaluates the previous function f for each pair of elements in the vectors x and y. Thus, the vector z – which we create – will have the same number of elements as both x and y. > z z 2.160247 2.081666 2.054805 2.041241 2.033060
3.4 Use of Brackets There are three types of brackets in S-Plus: round (), square [ ], and curly {}. More conventionally, these are referred to as parentheses, brackets, and braces, respectively. We have already seen two uses of parentheses (for function calls and for grouping in arithmetic expressions). We will investigate the use of square brackets here, and braces are covered in Chapter 9 on programming. The different types of brackets used in S-Plus are summarized in Table 3.2. Table 3.2. Different brackets in S-Plus.
Bracket ( ) [ ] { }
Function For function calls as in f(x), and to set priorities as in 3*(4+2) Index brackets as in x[3] Block delimiter for grouping sequences of commands. Example: if { ... } else { ... }
Square brackets are used to index a particular vector or matrix entry. Suppose we have a vector with three elements and we want to look at or use just the first element. We simply put the number of the element of interest within square brackets, as in the following examples: > x x 0 10 20
3.5. Logical Values
> x[1] 0 > x[2] 10 > x[3] 20 > x[4] NA
85
# first element of x # second element of x # third element of x # doesn’t exist # indicates Not Available
The index reference that is enclosed within the square brackets can be a vector itself, indicating that more than one value is desired. > x[1:2] 0 10 > x[c(1, 3)] 0 20 This type of notation may also be used to exclude values. Suppose we want all values except the first. This is done in the same way as above, except that the minus sign is placed before the index. > x[-1] 10 20 > y x[-y] 20 > x[2] x 0 100 20
# use y as index
3.5 Logical Values There are two logical values (also called Boolean values), TRUE and FALSE, which are used extensively for logic. On computers, these variable types are used a great deal for making comparisons. To appreciate the usefulness of logical variables, consider the following examples. Does 3 equal 4? > 3==4 F Obviously, 3 does not equal 4, which gives us an answer of F (FALSE). Notice that the comparison was made with two equals signs. This and other comparison symbols are summarized in Table 3.3. Additional examples follow. > 3 = == !=
Function Less than Greater than Less than or equal to Greater than or equal to Equal to Not equal to
> 3!=4 T The two lines above show us that 3 is less than 4 and that 3 is not equal to 4. The exclamation point (!) is used to negate the equals sign, so ! = gives us “is not equal to.” Now, consider using logical variables with vectors. > x x < 3 T T F We see that 1 and 2 are less than 3, but that 3 itself is not. Note Logical values are coded 1 for TRUE and 0 for FALSE. As with most other languages, S-Plus stores logical variables as 1/0 values with an attached flag indicating that this variable is of type logical. With the flag and 1/0 coding, it is possible to perform both logical comparisons and numeric computation. > x x sum(x height[height weight[weight weight[height>65] 150 155 135 Conversely, we can look at the heights of people whose weight is less than 130.
88
3. A First Session
> height[weight x[x seq(1, 6, 2) 1 3 5 > rep(1, 6) 1 1 1 1 1 1 > rep(6, 1) 6 > rep(1:3, 2) 1 2 3 1 2 3 > rep(1:3, rep(2, 3)) 1 1 2 2 3 3
# 1 to 6, increment of 1 # 1 to 6, increment of 2 # repeat 1, 6 times # repeat 6, 1 time # repeat (1:3), twice # repeat 1 twice,...
Here are more examples of the principles from our first session using the Geyser data set of S-Plus. The Geyser data set comes with S-Plus as an example and contains data collected about the Old Faithful Geyser in Yellowstone National Park. There are two variables: the waiting times (in minutes) and the corresponding duration (also in minutes) of the eruption. We can use the attach function to enable use of the data. We simply attach a data set and then can directly access the components contained in it.
3.6. Review
89
> attach(geyser) We are now ready to do some work and start by creating a vector with the first 10 waiting times. > wait.first10 wait.first10 80 71 57 80 75 77 60 86 77 56 We follow that by creating another vector with the next five waiting times. > wait.next5 wait.next5 81 50 89 54 90 Concatenate the vector with the first 10 waiting times with the vector containing the next 5 waiting times to create a new vector with the first 15 waiting times. It might not be the most exciting example, but it is a nice example of vector concatenation. > wait.first15 wait.first15 80 71 57 80 75 77 60 86 77 56 81 50 89 54 90 To get some practice with data editing, insert the value 100 at the beginning of the wait.first10 vector. > c(100, wait.first10) 100 80 71 57 80 75 77 60 86 77 56 Do the same type of thing by putting the value 100 between the vectors wait.first10 and wait.next5. > c(wait.first10, 100, wait.next5) 80 71 57 80 75 77 60 86 77 56 100 81 50 89 54 90 For a little practice with subsetting, we will create vectors containing the waiting times less than 50 and the corresponding durations. > short.wait mean(short.wait) 47.8125 > mean(waiting) 72.31438 > mean(short.wait.duration) 4.432292 > mean(duration) 3.460814 > short.wait.duration 4.600000 4.466667 4.333333 4.000000 4.600000 4.916667 4.583333 4.000000 4.000000 4.466667 4.450000 4.416667 4.983333 4.000000 4.333333 4.766667 Logical operators can be used to create subsets as we saw above, but remember that a double equals sign (==) is used for the logical “equal to.” > short.wait.duration==4 F F F T F F F T T F F F F T F F Replacing one value with another is something that you most often do. Such replacements can easily be done in S-Plus by working on the lefthand side of the assignment statement ( adj.short.wait.dur adj.short.wait.dur[adj.short.wait.dur==4] adj.short.wait.dur 4.600000 4.466667 4.333333 4.010000 4.600000 4.916667 4.583333 4.010000 4.010000 4.466667 4.450000 4.416667 4.983333 4.010000 4.333333 4.766667 > detach(geyser) When you’re done with a data set, it is a good idea to detach it using the detach function. We have seen in this chapter how we can use just a few simple commands and ideas to calculate some rather sophisticated results.
3.7. Exercises
91
3.7 Exercises Exercise 3.1 Calculate the first 50 powers of 2 (i.e., 2, 2∗2, 2∗2∗2, and so on). Calculate the squares of the integer numbers from 1 to 50. Which pairs are equal (i.e., which integer numbers fulfill the condition 2n = n2 )? How many pairs are there? (Use S-Plus to solve all these questions!)
Exercise 3.2 Calculate the sine, cosine, and tangent for numbers ranging from 0 to 2∗π (with distance 0.1 between them). The constant π is available in S-Plus and is denoted by pi. Remember that tan(x) = sin(x)/cos(x). Now calculate the difference between tan(x) and sin(x)/cos(x) for the values above. Which values are exactly equal? What is the maximum difference? What is the cause of the differences?
Exercise 3.3 Use the S-Plus help routines (not the manuals) to find out how to use the functions round, trunc, floor, and ceiling and learn what they do. Predict what each of these functions will give as an answer for the numbers −3.7 and +3.8. Use S-Plus to test your predictions.
92
3. A First Session
3.8 Solutions Solution to Exercise 3.1 We first create a vector (ints) containing the integers from 1 to 50. > ints x x 2 4 8 16 32 64 128 256 512 1024 2048 ... > y y 1 4 9 16 25 36 49 64 81 100 121 ... We create a T/F vector (being the same length as both x and y), which contains a T when elements of x and y are equal and an F when they are not equal. > equal equal F T F T F ... F Now we print the values that are common in both x and y. > x[equal] 4 16 > ints[equal] 2 4 We can see which entries were the ones in common. (The second and the fourth entries of x were equal to the second and fourth entries of y.) The length function is used to determine how many elements are contained in a vector. > length(x[equal]) 2 We can do the same calculation using the sum function on the logical vector equal. > sum(equal) 2
3.8. Solutions
93
Solution to Exercise 3.2 It helps to notice that π is a built-in constant in S-Plus, denoted by pi. You need to generate the set of numbers using the seq function and then calculate the difference between the tan and sin/cos results. Note that the values below may differ on different machines. > > > > >
x matrix(data, nrow, ncol, byrow=F ) Note that there are four arguments. Only the first is needed. The others are optional. If neither nrow (the number of rows) nor ncol (the number of columns) is specified, as in > matrix(1:6) 1 2 3 4 5 6 the returned argument is a one-dimensional matrix (actually a column vector). If only one argument of nrow or ncol is given, as in > matrix(1:6, nrow=2) 1 3 5 2 4 6 the other value is determined (by simple division) and set. If both dimensions are supplied, the matrix is simply filled up. In all these situations, a dimension that does not fit the data can be specified. This is usually not what you want to do, but a matrix will be created nonetheless, and a warning will be issued. Let’s look at how this simple matrix was created and some of its characteristics. We wanted to put the integers 1 to 6 into the matrix using two rows. With six values and two rows, S-Plus does the simple division (6/2=3) to calculate that the resulting matrix will have three columns. S-Plus works in terms of column vectors; hence, it fills the first column first, then the second, and, finally, the third. In addition, the matrix can be filled rowwise instead of columnwise by setting the byrow argument to T. > matrix(1:6, nrow=2, byrow=T) 1 2 3 4 5 6 You will find it necessary to do this when reading in your own data.
4.1. Constructing and Manipulating Data
97
Try to guess what the following commands produce and then try them out. > > > > >
x data country.data country.data 208 8 2.4 1432 61 1.7 2112 82 2.0 Our matrix is not terribly useful to us in this state because we don’t know which column corresponds to which variable or which row pertains to which country. To make our lives easier, we can give names to the rows and/or columns by using the dimnames function. This is a useful way to remember what is stored in which fields. Dimnames are used to obtain a better display and can also be used to access field entries. Since a matrix has two dimensions, let’s say n rows and p columns, the two vectors containing the dimnames that need to be attached to the matrix should have n and p elements, respectively. Each element is a character string describing the corresponding contents. Example 4.1. Attaching dimnames to a matrix In the following code, we assign row and column headings, using dimnames, to a matrix. > dimnames(country.data) # NULL means empty NULL > dim(country.data) 3 3 > countries variables dimnames(country.data) country.data Austria 208 8 2.4 France 1432 61 1.7 Germany 2112 82 2.0 1 The
data are taken from The Economist Pocket World in Figures, 2002 Edition.
98
4. A Second Session
> dimnames(country.data) country.data GDP Pop Inflation 208 8 2.4 1432 61 1.7 2112 82 2.0 > dimnames(country.data) country.data GDP Pop Inflation Austria 208 8 2.4 France 1432 61 1.7 Germany 2112 82 2.0 > dimnames(country.data) [[1]]: "Austria" "France" "Germany" [[2]]: "GDP" "Pop" "Inflation" Notice that, with dimnames, the actual row and column names are entered using a list structure (covered later in this chapter). However, the important thing is that, for a matrix, there are two parts to the dimnames specification: the row names and then the column names, separated by a comma. The value NULL is used either to remove the dimnames of the rows or columns or simply to specify that none will be defined. Accessing Elements of Matrices To access a piece of information in a matrix, you need to know the corresponding coordinates from the rectangular layout. In other words, we need to know the row and column numbers of the cell of interest. We know that Austria is in the first row and that population is in the second column. To find the population of Austria, we specify the following code. > country.data[1, 2] 8 We discover that the population of Austria is 8 million. Note Data are accessed from a matrix by first specifying the row number(s) and then the column number(s). To access an entire row (column), put the number of the desired row (column) and all the corresponding column (row) numbers. For France, this becomes > country.data[2, 1:3] GDP Pop Inflation 1432 61 1.7
4.1. Constructing and Manipulating Data
99
A shorthand notation for this operation is to leave out the specification of all the column (row) numbers, as this is implied. > country.data[2, ] GDP Pop Inflation 1432 61 1.7 As you can see, this alternate notation leads to the same answer. You can also access data through the dimnames rather than the row and column numbers. The numbers need only be replaced by the appropriate names (surrounded by quotation marks). Some examples follow. > country.data["Germany", "Inflation"] 2 > country.data[, "Inflation"] Austria France Germany 2.4 1.7 2 Using these few simple principles, you should be able to access any part of your matrix. The same methods of accessing elements or sections of a matrix can also be used to assign or overwrite. Simply specify the elements you want to redefine and assign the new value(s). We could change the population of Austria to 10 million with the following command: > country.data["Austria", "Pop"] 8 > country.data["Austria", "Pop"] country.data["Austria", "Pop"] 10 We should set it back to its correct value though: > country.data["Austria", "Pop"] A B A+B 0 3
matrix(0:5, 2, 3) matrix(seq(0, 10, 2), 2, 3) 6 9
12 15
In S-Plus, there is one big exception to the rule of having matrices’ dimensions match. If you have a single number (scalar), it can be added to, subtracted from, multiplied by, or divided into a matrix of any dimension. Take the matrix A above, and define a scalar, d, to be any number. Try adding d+A. Is the result what you expected? Now, try A/d. Given what you saw with the addition, did the division give you what you expected? Matrix multiplication is more complicated and is not elementwise. The S-Plus syntax to multiply two matrices is as follows: > A %∗% B The trick to matrix multiplication is that the number of columns of the first matrix has to be equal to the number of rows of the second. Hence, in the example above, if matrix A has two rows and three columns, matrix B must have three rows but can have any number of columns. Note Matrix multiplication can be done elementwise as well. To do this, we simply use the normal multiplication sign (*). > A*B 0 8 2 18
32 50
For the resulting matrix, the first element of A is multiplied by the first element of B, and so on. In general, division is not possible with matrices. However, a common matrix computation is to take the inverse, which is possible in S-Plus using the solve function. More advanced work with matrices may be possible with the singular value decomposition. This function is appropriately named svd in S-Plus. Recall that column vectors may be multiplied (or divided), even if they both come from the same matrix. From our example with GDP, we can divide GDP by the population in the following way: > country.data[, "GDP"]/country.data[, "Pop"] Austria France Germany 26 23.47541 25.7561 Mathematical operations on vectors work in an elementwise fashion, unlike matrix multiplication.
4.1. Constructing and Manipulating Data
101
Merging Matrices Frequently, you may discover that you have forgotten a row or column (or several) and would like to add it at a later stage. To add rows or columns to matrices, simply create a new matrix containing the missing data and merge it to the existing matrix. To add extra rows, use the rbind function, and to add extra columns, use the cbind function. Suppose we forgot the variable, area (×1000 square kilometers), for the countries in our example. We could add this information to our matrix in the following way: > Area country.data country.data GDP Pop Inflation Area Austria 208 8 2.4 84 France 1432 61 1.7 544 Germany 2112 82 2.0 358 Notice that if you create a (column) vector, you can add it with no special manipulation. Also, if you give the vector the appropriate name, the dimnames are automatically updated as desired. You add a row in exactly the same manner as with column vectors, except that you use the rbind function instead of cbind. To demonstrate how this works, we will add a row of data for Switzerland. > Switzerland country.data country.data GDP Pop Inflation Area Austria 208 8 2.4 84 France 1432 61 1.7 544 Germany 2112 82 2.0 358 Switzerland 259 7 1.6 41 You can add multiple rows/columns in the same manner, but pay careful attention to the dimensions of both the existing matrix and the one to be added.
4.1.2
Arrays
In S-Plus, an array is a data construct that can be thought of as a multidimensional matrix (up to eight dimensions). It is difficult to conceive of anything beyond three dimensions, but arrays may sometimes be useful to hold multiple two- or three-dimensional matrices. One nice feature of this construct is that the rules and procedures introduced for matrices hold for arrays as well.
102
4. A Second Session
An array is defined as follows: > array(data, dim) The data must be a single expression such as c(1, 4, 11, 18, 12, 6) and the same for the dimensions. A simple array is created by the following command: > array(1:24, c(3, 4, 2)) This produces a three-dimensional array: the first dimension has three levels, the second has four levels, and the third has two levels. When printing an array, S-Plus starts with the highest dimension and works towards the lowest dimension, printing two-dimensional matrices at each stage. The easiest explanation of how this process works is to look at an example for yourself. Example 4.2. Handling arrays Here is a short example to show how arrays work. > x x , , 1 [,1] [,2] [,3] [,4] [1,] 1 4 7 10 [2,] 2 5 8 11 [3,] 3 6 9 12 , , 2 [,1] [,2] [,3] [,4] [1,] 13 16 19 22 [2,] 14 17 20 23 [3,] 15 18 21 24 If we fix the second index at the value 2, we get a 3×2 matrix. > x[, 2, ] [,1] [1,] 4 [2,] 5 [3,] 6
[,2] 16 17 18
Notice that S-Plus prints one matrix for each level of the third dimension (highest dimension) of the array. The three-dimensional array is shown in two-dimensional slices. The matrix above of dimension 3×4×2 is shown in two slices, each of dimension 3×4. Row and Column Functions Due to the matrix layout of data that one commonly encounters, it is often required to perform certain functions on each row or on each column
4.1. Constructing and Manipulating Data
103
of the data. The most commonly used functions on rows and/or columns have been made into functions in S-Plus. The operations of mean, sum, variance, and standard deviation can be calculated for each column of a data matrix or for each row (not the standard deviation). The names of the functions are self-explanatory and are denoted by; colMeans, colSums, colVars, colStdevs, rowMeans, rowSums, and rowVars. The simplest way to see what the functions do is to look at a simple example. We will use the country.data data set that we have already created and compute the mean of each of the variables (columns). > colMeans(country.data) GDP Pop Inflation Area 1002.75 39.5 1.925 256.75 We see that the mean population of our four countries is 38.5 million. Care must be taken about the interpretation of the output from these functions. If we had used the function rowMeans instead of colMeans, we would have obtained results, but what is the mean of “Austria?” In this example, it is the mean of the GDP, population, inflation, and area of Austria and has little meaning. In other situations, the row functions are appropriate. In the next subsection, we will explore a more general way of using any function and applying it to either the rows or columns. Apply The apply function offers a very elegant way of handling arrays and matrices. It works by successively applying the function of your choice to each row (first dimension), each column (second dimension), or each level of a higher dimension. The syntax is > apply(data, dim, function, ...) data is the name of your matrix or array, and function is the name of any S-Plus function that you want to apply to your data. For a matrix of two dimensions, the option dim can take the integer value of 1 or 2 to refer to the rows or columns, respectively. The option, ..., can be filled in with options to be passed on to the function being specified. We can use such a function to compute the maximum value of each variable from our European Countries example. Example 4.3. Using apply on arrays Here, we use the apply function to calculate the maximum of each column in our country data set. > apply(country.data, 2, max) GDP Pop Inflation Area 2112 82 2.4 544
104
4. A Second Session
The max function simply returns the maximum value of the values supplied to it. Remember that any function can be given to apply so that you can use it for many purposes. Note For simple functions like min and max, direct functions exist in S-Plus that can be used to replace apply. In the example above, the column maximums could have been calculated using the equivalent expression below. > colMaxs(country.data) Other common functions of this nature are colMins, colMaxs, colRanges, rowMins, rowMaxs, and rowRanges. These specialized functions are typically faster than using the more general method involving the apply function. However, apply is much more general as it can be used with any function including your own.
4.1.3
Data Frames
One of the most used data constructs in S-Plus is the data frame. Its popularity stems from its flexibility, as it allows you to bind data vectors of different types together, such that the data can be accessed like a matrix (similar to data sets in SAS). The syntax for creating a data frame is as follows: > data.frame(data1, data2, ...) Here, the ... notation simply shows the fact that you can specify as many data sets as you want. This functionality fits in nicely with the next step in our European Countries example. You may have noticed a major difference between the countries (other than language), which we will indicate with a new character variable - namely, all the countries belong to the European Union (EU) except Switzerland. We could try to use the principles from the previous section on merging matrices, but the different data types cause the following problem: > EU country.data1 country.data1 GDP Pop Inflation Area Austria "208" "8" "2.4" "84" France "1432" "61" "1.7" "544" Germany "2112" "82" "2.0" "358" Switzerland "259" "7" "1.6" "41" > apply(country.data1, 2, max)
EU "EU" "EU" "EU" "non-EU"
4.1. Constructing and Manipulating Data
105
Problem in (function(x, ..., na.rm=F): Numeric summary undefined for mode "character" Use traceback() to see the call stack The matrix was created, but all entries were converted to character values, and numeric calculations are no longer possible. This may be what you want to do on a rare occasion, but more often than not, you will want to preserve the numeric nature of your matrix by creating a data frame, as shown in the following example. Example 4.4. Creating data frames Here, we make a data frame out of our European Countries data set so that we can add a character variable. > country.frame country.frame GDP Pop Inflation Area EU Austria 208 8 2.4 84 EU France 1432 61 1.7 544 EU Germany 2112 82 2.0 358 EU Switzerland 259 7 1.6 41 non-EU For the most part, data frames can be used as if they were matrices. For instance, we can calculate the maximum values of all of the numeric components of our data frame. > apply(country.frame[, 1:4], 2, max) GDP Pop Inflation Area 2112 82 2.4 544 Note A data frame can consist of a mixture of different data types (numeric, character, factor, etc.). Numeric computations like the mean can of course only be applied to numeric variables. If not the entire data frame consists of numeric variables, the result is hard to predict because the data frame will be converted before running apply on it. A simple way out is to use the function sapply instead and test for each object if it is numeric or not and react according to the type of object. Here is an example: > sapply(barley, function(x) if(is.numeric(x)) mean(x) else NA) yield variety year site 34.42056 NA NA NA Remember that data frames are, besides lists, the only data structures that can combine different types of objects in one structure. The catch is that you can not tell what structure a data set has by looking at the output. You need to check:
106
4. A Second Session
> is.data.frame(barley) T > sapply(barley, class) yield variety year site "numeric" "ordered" "ordered" "ordered" The individual variables in the example above are then referenced by one of three different notations; country.frame$GDP, country.frame[, "GDP"] (as with matrices), or country.frame["GDP"]. These forms of referencing are rather long and tedious, especially if your typing is not up to par. However, if you know that you are going to be working with a particular data frame, there is some relief in the form of the attach command. If you “attach” a data frame, you can work with the simple variable names (without the country.frame$) until the data frame is detached with the detach function. Example 4.5. Attaching and detaching a data frame When attaching a data frame, you must be aware of variables already existing in your session that might conflict with the name of a variable from an attached data frame. We will see this problem below. > attach(country.frame) > Pop Austria France Germany Switzerland 8 61 82 7 > EU "EU" "EU" "EU" "non-EU" Note that the double quotes are included when this variable is printed. This is the local version of the variable that was originally created to add to the data frame. To see the data frame version, detach, remove the local version, and attach again. > > > >
detach() rm(EU) attach(country.frame) EU EU EU EU non-EU > detach() > EU Problem: Object "EU" not found Use traceback() to see the call stack Note The local version of a variable always has preference rather than a version from the attach function.
4.1. Constructing and Manipulating Data
107
In the example above, the local version of the variable EU was printed after the attach was made. We know this because double quotes were printed, explicitly showing us that the original character vector was used. After we deleted the local version and reattached, the double quotes no longer appear. The data frame version of the variable EU doesn’t contain double quotes because the data frame itself recognizes that the variable is of type character and dispenses with them. The attach function is much like the “set” command in a SAS data step.
4.1.4
Lists
Lists are a construct that allow you to tie together related data that do not share the same structure. Recall that with data frames, the data could be of different types but had to have the same structure (i.e., have the same length). Consider again the European Countries example. Suppose we want to record all the European countries in which German is an official or major language and then associate this information with the rest of our European data. First, you need to know that there are six European countries in which German is an official or major language: Austria, Belgium, Germany, Liechtenstein, Luxemburg, and Switzerland. We now continue the European Countries example by showing how to create and work with lists. Example 4.6. Creating lists We start by creating a vector to add to our European countries data to make a list. > Ger.Lang country.list country.list [[1]]: GDP Pop Inflation Area Austria " 208" " 8" "2.4" " 84" France "1432" "61" "1.7" "544" Germany "2112" "82" "2.0" "358" Switzerland " 259" " 7" "1.6" " 41" [[2]]: "Austria" "Liechtenstein"
"Belgium" "Luxemburg"
"Germany" "Switzerland"
We now try to access some specific information. > country.list$Ger.Lang NULL
EU " EU" " EU" " EU" "non-EU"
108
4. A Second Session
> country.list[[1]]["Austria", "Pop"] 8 > country.list[[1]]["Austria", "EU"] EU Levels: "EU" "non-EU" > apply(country.list[[1]][, 1:4], 2, max) GDP Pop Inflation Area 2112 82 2.4 544 The list function works by simply “pasting” together all the pieces supplied to it. It creates one item for each supplied argument, and these are sequentially referenced using double square brackets. Notice that the syntax used with data frames (country.list$Ger.Lang) does not work with lists, and a value of NULL is returned. With some modification, however, this syntax works as well. We must first assign labels as follows and then the syntax is recognized: > country.list country.list$Ger.Lang Individual elements in a list may be referenced by considering the double brackets as part of the name of the structure of interest. The command country.list[[1]] extracts the first list element. The referencing is then like any matrix. If you want to see the entry in the first row, second column, add [1, 2] (population of Austria). The entire reference is in the example. When this is used with a character variable, S-Plus gives you the additional information of the different levels for that variable (only two levels here). You can also give names to the various elements in a list, and then the referencing looks more like that from a data frame (e.g., country.list$Countries[1,1]). Use the names function to add names to a list. (A more thorough discussion of lists and how they work is covered in Section 9.1.)
4.2 Introduction to Functions One of the most valuable resources of S-Plus, and what distinguishes it from many other statistical software packages, is the ability to write your own functions. Functions that are built into S-Plus are called by stating the name of the function, the parentheses, and the parameter(s) being passed to the function. We have seen this often already, as in seq(1, 10, 1).
4.3. Introduction to Missing Values
109
The principle of defining your own functions is the same as above and can most easily be seen through a simple example of calculating the standard deviation, which is the square root of the variance. Example 4.7. Creating your own function > SD y SD(y) 3.02765 Here, we see that the function has one parameter (x) that is used to define the data for which we want to calculate the standard deviation. The actual calculations of the function are contained within curly brackets and can be extended over several lines with several commands. The result of the function that should be returned to the user should be contained within the return function. By default, the result of the calculation on the last line of the function will be returned (made visible) to the user. We could have defined the contents of the function as either return(sqrt(var(x))) or y attach(country.frame) > country.frame$GDP[1] country.frame$GDP NA 1432 2112 259 The whole process seems quite straightforward. However, the problem is that certain mathematical functions in S-Plus either don’t work or give unexpected values when applied to vectors containing missing values (NAs). Take, for example, the max function used several times in the European Countries example. When applied to the vector above, we don’t get the value of 2112 that you might have expected. > max(country.frame$GDP) NA S-Plus returns the value NA as the maximum of the GDP values. In a sense, it is being conservative because the value NA might arise as a product of other mathematical operations. In this case, the value of NA being
110
4. A Second Session
returned as the maximum of your vector would warn you that something strange happened in your calculations. In other cases though (like ours), you don’t care about the missing value because you put it there to represent unknown information. This problem can be circumvented by determining which values are not missing and keeping only the valid values to perform the desired mathematical function. We use the attach function here to avoid some typing. > > > >
> > > > >
rm(GDP) attach(country.frame) gdp.na > > > >
friends apply(geyser.matrix, 2, mean) waiting duration 72.31438 3.460814 > apply(geyser.matrix, 2, min) waiting duration 43 0.8333333 The minimum values above might be from corresponding readings, but more than likely, the minimum duration does not come from the minimum waiting time. You can check this by looking at the minimum waiting time and its corresponding duration. First, you need the index that corresponds to the minimum waiting time. > min.wait geyser.matrix[min.wait, ] waiting duration 43 4.333333 The conclusion is that the minimum waiting time (43 minutes) does not correspond to the minimum duration (0.83 minutes). In fact, it looks like the minimum waiting time might correspond to the longest duration. To look at this strange relationship between waiting times and durations, you will construct two small matrices: m.short.wait containing the shortest waiting times and corresponding durations, and m.long.wait containing the longest waiting times and corresponding durations. We create a logical vector of waiting times less than 48. > short.wait m.short.wait waiting duration 43 4.333333 45 4.416667 47 4.983333 47 4.766667 > m.long.wait 95, ] > dim(m.long.wait) 4 2 > m.long.wait waiting duration 108 1.950000 96 1.833333 98 1.816667 96 1.800000 You can place matrices on top of one another with the rbind function. > join join waiting duration 108 1.950000 96 1.833333 98 1.816667 96 1.800000 43 4.333333 45 4.416667 47 4.983333 47 4.766667
4.5. Putting It All Together
113
Notice that the items in the matrix join are not sorted. Values in a single vector may be sorted directly using the sort function, as in > sort(join[, 1]) 43 45 47 47 96 96 98 108 However, if you want to sort on more than one variable, it is best to use the order function to get a permutation of the indices that will give the desired ordering when used. The following is a short example of this. > order(join[, 1], join[, 2]) 5 6 8 7 4 2 3 1 > join[order(join[, 1], join[, 2]), ] waiting duration 43 4.333333 45 4.416667 47 4.766667 47 4.983333 96 1.800000 96 1.833333 98 1.816667 108 1.950000 We specified the waiting times first and then the durations to the order function so that our new list would be sorted first by waiting time and then, in case of a tie, by duration. The third and fourth entries both have a waiting time of 47 minutes. The row with the shorter duration comes first in the list. The variable ord contains the ways in which the rows should be ordered, and the join function puts them in this order.
114
4. A Second Session
4.6 Exercises Exercise 4.1 Work with the European Countries data used throughout most of this chapter. • Compute the GDP per capita and add it to the data frame. • Compute the maximum of each of the variables in the data frame. • Which country has the largest GDP per capita? • Assume that the GDP is not known for France (set it to “missing”). Recalculate the GDP per capita and the maximum value for each variable. • What has now happened to the maximum of the GDP variable? Suppress the row containing the missing value and again recalculate the maximum of each variable. • Look carefully at the maximum area. To which country does this area belong? Is this really the largest country in our small sample? What happened when we excluded the row with the missing GDP for France? Determine how to correctly adjust for the missing GDP and still get correct results for the maximum values. (Hint: Use the help facility.)
Exercise 4.2 This exercise is really a continuation of the previous one. Once you have added the variable of GDP per capita to the data frame from the European Countries example, sort the data frame according to GDP per capita. Notice that it puts the smallest GDP per capita first and the largest at the bottom. Use the help utility to change the sort so that the largest GDP per capita is first and the smallest is at the bottom (reverse sort).
Exercise 4.3 Create a list of European countries whose official or major language is French. Add this vector to the list from the European Countries example. Use S-Plus to produce a list of countries where both German and French are either an official or a major language. To do this, you will need to create a list where either German or French is spoken and find the countries that are duplicated in this list. (Hint 1: Use
4.6. Exercises
115
the help utility.) What are the physical characterics of these countries? (Hint 2: The French-speaking countries in Europe are Belgium, France, Luxemburg, and Switzerland.)
Exercise 4.4 Write a function with two parameters that offers a solution for the famous Y2K problem. The first parameter will be a two-digit year and the second will be a two-digit cutoff. The cutoff works such that if 30 is specified, then all years below (or equal to) 30 are considered to be in the 21st century (20xx) and years above 30 are considered to be in the 20th century (19xx). You will want to look at the syntax of the ifelse function.
Exercise 4.5 Continue the example on merging data shown in Section 4.4. In particular, merge the phone.numbers and addresses data frames such that: • All entries in both data frames appear • Only entries in the phone.numbers data frame appear • Only entries in the addresses data frame appear Not included in the solution is the extension that you should rename the names variable in the addresses data frame to friends and then perform a simple merge.
Exercise 4.6 For some extra practice with merging and data manipulation, use the S-Plus datasets car.test.frame and fuel.frame to create a new data frame where the merge is done by car name and the common variables are not repeated (except Mileage, but use the suffixes option). Summarize the mileage for each type of vehicle. Create a dataset that contains the vehicle with the highest mileage for each vehicle type.
116
4. A Second Session
4.7 Solutions Solution to Exercise 4.1 We begin by creating a new data frame by adding a new variable to an existing data frame. Notice that we have used the round function to specify that one place to the right of the decimal point will be printed. > country.frame1 country.frame1 GDP Pop Inflation Area EU GDPcap Austria 208 8 2.4 84 EU 26.0 France 1432 61 1.7 544 EU 23.5 Germany 2112 82 2.0 358 EU 25.8 Switzerland 259 7 1.6 41 non-EU 37.0 Use the apply and max functions to calculate the maximum of each of the variables (columns). Remember that the fifth column must be excluded from the calculations because it is not numeric. > apply(country.frame1[, -5], 2, max) GDP Pop Inflation Area GDPcap 2112 82 2.4 544 37 We set the GDP and GDPcap for France equal to NA and recalculate the maximum values. > country.frame1[2, c(1, > apply(country.frame1[, GDP Pop Inflation NA 82 2.4
6)] gdp.na apply(country.frame1[!gdp.na, -5], 2, max) GDP Pop Inflation Area GDPcap 2112 82 2.4 358 37 The problem we have encountered here is that, by searching for rows with missing values, we have excluded the entire row. In our example, this corresponds to all data on France. Hence, when we calculate the maximums, we find the largest area to be 358, which is that of Germany. What we really want to do is to exclude only that one missing value and not the whole row. The simple solution is found by looking at the help documentation for the max function. We find that the additional parameter na.rm excludes individual missing values from analyses.
4.7. Solutions
117
> help(max) > apply(country.frame1[, -5], 2, max, na.rm=T) GDP Pop Inflation Area GDPcap 2112 82 2.4 544 37
Solution to Exercise 4.2 The trick here is that we want to sort our rows according to the GDP per capita. Hence, we have to specify this variable to the order function. The order given is the order in which the rows must be printed to be ordered. (Remember to replace the NAs for France with their original values.) > ord.gdpcap ord.gdpcap 2 3 1 4 By simply putting the new ordering into the row indices of the data frame, we get the data frame with our desired sorting. > pr.sort pr.sort GDP Pop Inflation Area EU GDPcap France 1432 61 1.7 544 EU 23.5 Germany 2112 82 2.0 358 EU 25.8 Austria 208 8 2.4 84 EU 26.0 Switzerland 259 7 1.6 41 non-EU 37.0 We can’t stress enough that you must make use of S-Plus’ help facility. If you look up a related command, it often contains a cross-reference to the command you are actually interested in. Here, for example, we look at the help for the order function and see it has a reference to the rev function. Using a little imagination, we might think that this will give us the reverse (descending) ordering. If you follow the reference, you will see that rev is the reverse ordering, which works in the same way as the order function, except that it orders in descending order. > help(order) > pr.rev pr.rev GDP Pop Inflation Area EU GDPcap Switzerland 259 7 1.6 41 non-EU 37.0 Austria 208 8 2.4 84 EU 26.0 Germany 2112 82 2.0 358 EU 25.8 France 1432 61 1.7 544 EU 23.5 Sorting data frames can also be done using the Windows GUI. Try the same exercise using the GUI (Data - Restructure - Sort...).
118
4. A Second Session
Solution to Exercise 4.3 There are four European countries where French is an official or a major language. Using the list function, create the new vector and combine it with the original data frame and the vector of German-speaking countries. If you were to combine the vector of French-speaking countries with the country.list already created in the chapter, the numbering of the components would be rather strange, but it could be done. > Fr.Lang country.list1 country.list1 [[1]]: GDP Pop Inflation Area EU Austria " 208" " 8" "2.4" " 84" " EU" France "1432" "61" "1.7" "544" " EU" Germany "2112" "82" "2.0" "358" " EU" Switzerland " 259" " 7" "1.6" " 41" "non-EU" [[2]]: "Austria" "Liechtenstein" [[3]]: "Belgium"
"Belgium" "Luxemburg"
"France"
"Germany" "Switzerland"
"Luxemburg"
"Switzerland"
There are several ways of finding the countries where both French and German are spoken. The solution here is not unique. We proceed by putting the second and third components of the list into a new vector and using the duplicated function to find which country names have been duplicated. We found three countries where the two languages are spoken. > Both.frame dup dup "Belgium" "Luxemburg" "Switzerland" > country.list1[[1]][dup, ] GDP Pop Inflation Area EU NA NA NA NA NA NA NA NA NA NA Switzerland 259 7 1.6 41 non-EU Notice that the first two lines of our list containing the characteristics of the French- and German-speaking countries are filled with NAs. This is simply because we don’t have any country characteristics for Belgium and Luxemburg.
4.7. Solutions
119
Solution to Exercise 4.4 The idea is that if a date of 00 arises, chances are that it refers to a current date of the year 2000 and not to a date of birth of 1900. We set up a window of uncertainty by defining the cutoff year high enough that it captures all current years but low enough so that birth dates from the beginning of the 1900s are treated correctly. A good cutoff to use might be 10. Y2K merge(phone.numbers, addresses, all=T) friends phone address 1 a 1 Street 2 b 2 NA 3 c 3 Road 4 d NA Lane Note that all the entries from both data sets appear. A value of NA is assigned to variables that have no value as a result of the merge. To include only entries from the phone.numbers data frame, we need to use the all.x option in the same manner as we did above. > merge(phone.numbers, addresses, all.x=T) friends phone address 1 a 1 Street 2 b 2 NA 3 c 3 Road Here, we have an entry for a, b, and c but no entry for d because it doesn’t exist in the phone.numbers data frame.
120
4. A Second Session
In the same manner as above, if we want to include only the entries in the addresses data frame, we use the all.y option. > merge(phone.numbers, addresses, all.y=T) friends phone address 1 a 1 Street 2 c 3 Road 3 d NA Lane Here, we have no entry for b because it doesn’t exist in the addresses data frame.
Solution to Exercise 4.6 Begin by renaming the two data frames for easier use. > car fuel names(car) "Price" "Country" "Reliability" "Mileage" "Type" "Weight" "Disp." "HP" > names(fuel) "Weight" "Disp." "Mileage" "Fuel" "Type" Opening the data frames shows that the vehicle name has been saved as the row names. The variables that are common to the two data frames are Mileage, Type, and Disp. The simple use of the merge function clearly does not work. It looks as follows. > merge(car, fuel) The biggest problem with the simple approach is that the vehicle name is not a part of the output. To access the vehicle name, use the row names as in the following command to get us one step further. > merge(car, fuel, by.x="row.names", by.y="row.names") The next step is to use only unique variables, with the exception of Mileage which will be displayed from both data frames using the suffixes option. > car.fuel car.fuel.sort by(car.fuel.sort$MileageCar, car.fuel.sort$Type, summary) INDICES:Compact x Min.:21.00 1st Qu.:23.00 Median:24.00 Mean:24.13 3rd Qu.:25.50 Max.:27.00 ------------------------------------------------------INDICES:Large x Min.:18.00 1st Qu.:19.00 Median:20.00 Mean:20.33 3rd Qu.:21.50 Max.:23.00
122
4. A Second Session
In order to get a dataset with the highest mileage for each vehicle type, we need to sort the original dataset in descending order of mileage and keep the rows that are not duplicates of the vehicle type. > x high.mpg high.mpg Row.names Price Country Reliability 19 Ford Festiva 4 6319 Korea 4 28 Honda Civic CRX Si 4 9410 Japan 5 55 Toyota Camry 4 11588 Japan/USA 5 57 Toyota Cressida 6 21498 Japan 3 4 Buick Le Sabre V6 16145 USA 3 40 Nissan Axxess 4 13949 Japan NA
19 28 55 57 4 40
MileageCar Type Weight Disp. HP MileageFuel 37 Small 1845 81 63 37 33 Sporty 2170 97 108 33 27 Compact 2920 122 115 27 23 Medium 3480 180 190 23 23 Large 3325 231 165 23 20 Van 3185 146 138 20
19 28 55 57 4 40
Fuel 2.702703 3.030303 3.703704 4.347826 4.347826 5.000000
The commands above have simply looked at the sorted dataset and taken the first rows where a vehicle type appears. If there are ties of mileage for the same vehicle type, they will not be taken into account. To see these, we can either look at the original sort above, or we can use the split and lapply functions as shown below. > high.mpg.list do.call("rbind", high.mpg.list) The function do.call puts together a call to the function specified, rbind in this case, and adds all elements of the list as arguments to the function call.
5 Graphics
Graphs are one of S-Plus’ strongest capabilities and most attractive features. This section gives an overview of how to create graphs and how to use the different parameters to modify elements and layouts of graphs. There are many options and functions available for working with graphs and graphical elements. The most important commands and their functionality are treated, and many other possibilities are mentioned. If you are interested in more details, this chapter will enable you to quickly find and understand the relevant information in the manuals provided with S-Plus or from the online help. Windows users can switch back and forth between the graphical user interface and the command line. Even if the Windows GUI is used, all actions are mapped to commands, as has been explained earlier.
5.1 Basic Graphics Commands You can start to enter graphics commands right away. S-Plus determines if a graphics device (a window on the screen or a file) is already opened. If this is not the case, it opens a window on the screen and displays the graph in it. You can however open graphics devices manually, for example to maintain multiple graphics windows or screen windows and files simultaneously. Graphics windows are opened by using the command > graphsheet()
126
5. Graphics
on Windows systems. On UNIX systems, the corresponding command is > motif() or, if you use the GUI version, > java.graph() If we have a variable x and a variable y, we can plot the two variables against each other simply by entering > plot(x, y) If you don’t have two variables at hand, directly pass the data to plot. > plot(1:100, sin(1:100/10)) It’s as easy as that. The plot should now appear in the graph sheet. In the same way, a histogram of a variable x can be generated by entering > hist(x) Simple plotting using only the elementary plotting functions becomes insufficient very quickly. You might want to add lines, change the colors of the boxplot, have different symbols to plot the data, or change labels. Let us start systematically by specifying first where the output (the graph) should go. The next step is to create the graph.
5.2 Graphics Devices The graphics output depends on the device to which you send commands. Generating a line on the screen requires a different action internally than generating a line and storing it in a file. Therefore, you can either use the default graphics device (a window on the screen) or specify explicitly the device where the graph should be created. You do this by calling the appropriate S-Plus function (you can also call it the “device driver”). From now on, all commands producing graphical output send their result to the active device. The active device, if not specified otherwise, is typically the last device opened. In general, the standard graphics device on a Windows system is the graph sheet, opened with the graphsheet function. A graph sheet can be used to open a window on the screen or to create a graphics file of specified format. On UNIX systems, the standard graphics window is opened by using the motif function. For opening printer devices, the standard MS Windows printer device is started with > graphsheet(format="printer")
5.2. Graphics Devices
127
Under UNIX, you typically create a PostScript file and send it to a PostScript printer. The command to use is > postscript("filename") Of course, a standard way of printing graphs is to create the graph on the screen and, once it is finished, to click the Print button in the menu bar. The graph is automatically converted into the printer format and sent off to the printer. Knowing about the printer device is useful, especially if a series of graphs is to be created and sent to the printer. Instead of creating a graph and clicking the Print button, creating the next graph, printing it, and so forth, all graphs can be sent to the printer directly. We list the most common devices in Table 5.1. Table 5.1. The most common output devices.
System
Device function
UNIX
motif postscript java.graph pdf.graph wmf.graph hpgl hplj Windows graphsheet
Description Motif graphics window PostScript file Java graphics window PDF (Adobe Portable Document Format) WMF (Windows Metafile) file HPGL file HP LaserJet file Graphics window or creation of a file of specified format (JPEG, GIF, WMF, etc.) using graphsheet(file="filename", format="format")
Note: Printer functions require a file name in order to save the output to a printer file, as in postscript (file="graphics.ps"). The functions are called by adding parentheses to their names, as in graphsheet(). If you are interested in creating an interactive Web page using S-Plus graphics, look up how to create a “graphlet,” a Java graph that can be embedded in Web pages. After opening the graphics device, S-Plus is ready to receive graphics commands. The device is initialized, some starting values are set, and the user can create graphics elements for that device by calling the appropriate functions. An easy example is to plot the Geyser data we examined earlier. > plot(geyser$waiting, geyser$duration) If you want to close the last device opened, enter > dev.off() To close all open devices, enter > graphics.off()
128
5. Graphics
Note Do not forget to close the device. If you quit S-Plus, this is done for you. Closing the device is especially important if you are in the process of creating a graphics file or if you are sending graphics commands directly to a printer. Only if you close the device will the last page be ejected to the printer; otherwise, the eject page command will not be sent to the printer or added to the file, respectively. The reason for this is that S-Plus is still ready to receive graphics commands for the same page. Note Typically, there is a menu bar, which includes the Print button. If you click on this button, S-Plus generates a file from the graph on the screen and sends it to the system printer. Make sure that the correct printer is installed. On UNIX systems, S-Plus keeps a file with the name ps.out.xxxx.ps in the current directory (it can be configured to remove it after printing though). Options Device parameters can be supplied with the function call, such as specifying orientation (portrait or landscape) or the height and width of the plot. All the devices have several options that are set to a default value. You can override the default values by supplying the parameter to the function. For example, if the PostScript graphics should be in portrait mode (not horizontal but vertical) and stored in a file smallgraph.ps, you need to enter > postscript(file="smallgraph.ps", horizontal=F)
5.2.1
Working with Multiple Graphics Devices
S-Plus offers the possibility of working with more than one graphics device at a time. For interactive data analysis and presentations, it is very useful to have two or more graphics windows on the screen and to show interesting details by switching between them. Table 5.2 shows an overview of the most commonly used functions and provides a brief explanation of their functionality. The functions offer a great flexibility by having several graphics windows on the screen while analyzing a data set. They also offer a nice toolkit for creating an impressive demonstration of self-developed applications.
5.3 Plotting Data Having learned how to prepare S-Plus for receiving graphics commands, we proceed now to generating graphical output.
5.3. Plotting Data
129
Table 5.2. Commands for graphics devices.
Command
Effect
dev.list() dev.cur() dev.next()
Lists all currently open devices Displays name and number of the active device Returns the number and name of the next device on the list without activating it Returns the number and name of the previous device on the list without activating it Sets the active device to the device number n. Can be combined to dev.set(dev.next()) Copies a graph to a new device, as in dev.copy(motif) or dev.copy(graphsheet) Copies the current graph to device number n. Example: dev.copy(which=2) Closes the current device Sends the current graph to the printer device Closes all currently active devices Prompts the user to confirm erasing the contents of the currently active graphics device before displaying the next graph
dev.prev() dev.set(n) dev.copy(device) dev.copy(which=n) dev.off() dev.print() graphics.off() dev.ask(ask=T)
In the following section, we present the most important commands to produce S-Plus graphs. After acquiring this basic knowledge, you should be able to create graphs with more sophisticated functions.
5.3.1
The plot Command
The plot command is the most elementary graphics command. It is used to create a new figure. The plot function calculates the width and height of the figure and acts appropriately. The most elementary usage is, as shown earlier, > plot(x, y) to plot two variables x and y of equal length against each other. In this usage, the variable in the first position (x) is plotted along the horizontal axis and the one in the second position (y) is plotted along the vertical axis. You can also specify x only and omit y. If x is a vector, S-Plus plots an index vector against x (1, . . . , n, n being the length of x). If x is a matrix with two columns, S-Plus will plot the first column against the second. If x is a list containing the elements x and y, these elements are used for plotting. Every single point of the graph is displayed in the type option point, as this is the default. (We will come back to this in the next section.)
130
5. Graphics
You can change the character for the points by specifying the parameter pch. For example, > plot(x, y, pch="P") displays the data points with the character P.
5.3.2
Modifying the Data Display
If you want to display data points in a graph, there are many ways of doing it. You might display the data as dots, stars, or circles, simply connect them by a line, or both. The data might be displayed as height bars or made invisible to set up the axes only and add more elements later. For this purpose, the function plot has a parameter type. Figure 5.1 shows examples of the most commonly used type options. Options for the type mode in which to plot are summarized in Table 5.3. The statements used to generate the example graph are listed for reference. Table 5.3. Options for the parameter type in the plot command.
type option
Plot style
type="p" type="l" type="b" type="h" type="o" type="s" type="n"
Points (the default) Lines connecting the data Both (points and lines between points) Height bars (vertical) Overlaid points and connected lines Stairsteps Nothing
See Figure 5.1 for a display of the type options. These are the commands used to generate Figure 5.1. > > > > > > > > > >
x x y plot(x, y, axes=F, type="b", pch="x", xlab="", ylab="") We add axes with specified tick marks and mixed labels. The main horizontal axis is plotted at y = 0 by specifying pos=0. The labels for the vertical axes are right-adjusted by setting adj=1. Horizontal lines are added to the graph using abline. > axis(1, at=c(0, 1, 2, pi, 4, 5, 2*pi), labels=c(0, 1, 2, "Pi", 4, 5, "2 Pi"), pos=0) > axis(2, at=c(-1, -0.5, 0, 0.25, 0.5, 0.75, 1), adj=1) > abline(h=c(-1, -0.5, 0.5, 1), lty=3) Finally, we add text to the graph in a specified position and adjusted to the left. > text(pi, 0.1, " sin(pi)=0", adj=0) > title("The sine function\nfrom 0 to 2 Pi") The commands used above (and more) are summarized in Table 5.5. Table 5.5. Commands that add to existing graphs.
Function
Description
abline(a, b) arrows(x1, y1, x2, y2) axes() axis(n)
Line with intercept a and slope b Arrows from (x1, y1) to (x2, y2) x- and y-axes Axis at a specified side. 1=x-axis, 2=y-axis, etc. Outer box (frame) of a graph Points at the coordinates given by x and y Lines through the points given by x and y Shaded polygon figure Disconnected line segments from (x1, y1) to (x2, y2) text at the specified locations Title and/or subtitle
box() points(x, y) lines(x, y) polygon(x, y) segments(x1, y1, x2, y2) text(xpos, ypos, text) title("title", "subtitle")
Note Sometimes you will want to plot line segments. This is easy to do if you know that S-Plus interrupts a line where a missing value (NA) is encountered. The line does not connect from and to the missing value. We
5.4. Adding Elements to Existing Plots
135
will have a detailed look at missing values later. Alternatively, you can use the segments function.
5.4.2
More About abline
As abline is a very useful function for enhancing a graph with one or more lines, we give some application examples in Table 5.6 and have, in fact, already used it in Figure 5.3. The function abline is especially useful for drawing gridlines. If you want to have the gridlines more in the background of the picture, make them gray instead of black, such that they less disturb the impression of the whole picture. For this purpose, you need determine the color coding of the gray tones on your output device. Another approach is to add dotted lines instead of solid ones. As usual, there are many ways to achieve a result. You need to try out different possibilities to find the one most suitable for your application. Table 5.6. Some examples of uses for abline.
Function
Description
abline(a,b) abline(l)
Line with axis intercept a and slope b If l is a list and contains an element whose name is coefficients, it is used as argument to abline Parallel horizontal lines through all the points (0, x). Can be used to add grids, as in abline(h=0:5) The analogon for vertical lines. Parallel vertical lines through all points x on the x-axis
abline(h=x) abline(v=x)
Of course, horizontal and vertical parameters can be combined. Can you think of how to create a set of parallel diagonal lines with a single S-Plus command?
5.4.3
More on Adding Axes
Sometimes, it is important to have the tick marks sitting on the axes in the correct position. S-Plus tries to do this in a general and almost always satisfying way, but sometimes you might want to have nonequidistant tick marks or add a specific point on the axis. It is also possible to have the y-axis on the right-hand side and to change the labels to anything desired. Basically, there are two ways of achieving the desired graph. The first possibility is to supply the corresponding parameter setting to the plot function. The possibilities for setting axis styles in the call to a plot function (not just plot) are summarized in Table 5.7.
136
5. Graphics Table 5.7. Optional parameters for plotting axes.
Parameter
Description
lab=c(5,5,7) Number of ticks on x- and y-axis and label length las=0 Label style for axes: 0=parallel to axes, 1=horizontal, 2=perpendicular to axes mgp=c(3,1,0) Line of margin to place axis title, label, and axis line xaxt="s" x-axis type. xaxt="n" omits the x-axis yaxt="s" y-axis type. yaxt="n" omits the y-axis tck=-0.02 Tick mark length; positive numbers point into picture xaxp= The current tick parameter’s start, end, and intervals lty=1 Line type: 1=solid, 2=small breaks, etc. lwd=1 Line width: 1=default, 2=twice as thick, etc. Note: The values above refer to the preset values (defaults) used if nothing else is specified. The second possibility for obtaining specific axes is to create a graph without axes and add the axes later using the axis or axes function. In the latter case, one would enter > plot(x, y, axes=F) Now, you can go ahead and add axes in the way you want. The standard routine for doing this is axis to add a single axis, or axes to add all axes and the surrounding box. The box only can be added by using box. > box() > axis(4) The surrounding box and an axis on side number 4 is added, which is the right-hand side of the graph (see Table 5.9 for definition of side numbers). Some examples for adding specific axes are given in Table 5.8. Table 5.8. Optional parameters for plotting axes using the axis function.
Parameter
Description
side=n
Axis at side n, where n = 1 denotes the bottom, n = 2 the left, n = 3 the top, and n = 4 the right side Labels at position x Specification of what to put in place of the labels Shifted axis to go through the coordinate y las=0/1/2 adds labels parallel to the axis/horizontal/ rotated by 45◦
at=x labels=s pos=y las=m
Note: Of course, these parameters can be combined as in the following example: axis(3, at=(1:3)*pi, labels=c("pi", "2 pi", "3 pi")).
5.4. Adding Elements to Existing Plots
137
Note that an axis typically consists of lines. Therefore, all settings for lines, such as changing the thickness or the color, also apply to axes. For example, > axis(4, lwd=2, col=2) adds an axis on the right-hand side twice as thick as the standard style, and in color number 2.
5.4.4
Adding Text to Graphs
Adding some text to a graph is useful if specific details of a graph should be pointed out. The standard functions text and mtext do a good job. To obtain the desired result, supply additional parameters to these functions. Table 5.9. Optional parameters for adding text.
Parameter Description text and mtext: Text in a figure adj=0.5 Text adjustment. 0=left-justified, 0.5=centered, 1=right-justified cex=1 Character expansion relative to standard size cex=2 draws characters twice as big col=1 Color to plot in. 0 is the background color crt=0 Character rotation in degrees, counterclockwise srt=0 String rotation in degrees, counterclockwise font=1 Font for plotting characters, device-dependent mtext only: Text in the margin of the figure side=n Side to add text to. 1=bottom, 2=left, 3=top, 4=right outer=F Text not on the outer margin of the whole figure. If outer=T, text can be placed on top of a layout figure. For an example, see Figure 7.17 (page 238) line=n Places the text on line n towards the margin of the figure. Negative numbers lie inside the figure Note: Text is typically added by using the functions text or mtext. Some devices, such as text terminals, cannot provide all of the functionality, such as rotating text. A standard example for adding the text “extreme value” to a graph at the coordinates (1, 2), such that the text begins on the right-hand side of the point, is > text(1, 2, "extreme value", adj=0)
138
5. Graphics
Sometimes, the text overlaps or touches the point on the graph just slightly. In such a case, add an extra space to the beginning of the text. The optional parameters of text and mtext offer more possibilities (see Table 5.9).
5.5 Setting Options We have already seen that almost all graphics functions accept parameters for setting options such as colors, character size, or string adjustment. There are many more parameters, and almost all of the functions generating graphics output accept them as arguments. They can also be set generally such that they become the system default for all succeeding graphics commands. For doing this, we can use the par function. Check the different options that can be set by par. Enter the command > par() to see all options and how they are currently set. The help pages provide a very detailed explanation. We supply a brief overview of the most common parameters and their possibilities in Figure 5.4 and Table 5.10. If changed using the par function, these settings become global from then on. Nevertheless, if a new session is started, they are set back to the original system defaults. 63 margin ?
*
* *
margin 2
-
*
margin 4
-
Plot Figure
61 margin ?
Figure 5.4. The S-Plus definition of figure and plot regions.
The parameters referenced in Table 5.10 can be set by using the par function, as in the following example where the outer margin is set to be equal to 1 line on all sides, for all successive graphs. > par(mar=c(1, 1, 1, 1)) A setting can be retrieved by entering > par("mar")
5.5. Setting Options
139
Table 5.10. Layout parameters.
Parameter
Description
fin=c(m,n) pin=c(m,n) mar=c(5,4,4,2)+0.1 mai=c(1.41,1.13,1.13,0.58) oma=c(0,0,0,0) omi=c(0,0,0,0) plt=c(0.11,0.94,0.18,0.86)
Figure size in inches. m width, n height Picture in inches, as in fin All margins in lines All margins in inches Outer margin in lines Outer margin in inches Plot region coordinates as fraction of figure region x-axis and y-axis min. and max. Useful for querying a graph’s boundaries Multiple figure layout, rowwise plotting, to generate a (m, n) matrix of pictures Multiple figures, columnwise plotting Figure type. "m" for maximal size, "s" for square graphs
usr mfrow=c(m,n) mfcol=c(m,n) fty="style"
Note: In the left-hand column, the letters m and n are to be replaced by integer numbers. Settings given are the system default values. To query settings, use (for example) par("usr"). or > par()$mar If you run S-Plus under Windows, most of these settings can alternatively be changed by using the menu bars on top of the S-Plus window. Note You can add your own parameters to the system settings using an unused variable. The command > options() shows all settings not related to the graphics subsystem. A new variable my.personal.setting can be set to TRUE by entering > options(my.personal.setting=T) at the prompt. Your own functions can then use this parameter setting by asking for its setting as follows: > options("my.personal.setting") T
140
5. Graphics
5.6 Figure Layouts The graphical parameters we looked at in this chapter are all preset with meaningful values. Nevertheless, you might want to change the default values. This section deals with the most common cases to demonstrate how to achieve satisfying results quickly. If you use a multiple-figure layout, the system may not always choose the layout as you prefer (the size of the characters, the white space between figures, etc.). We now show, for the standard layouts, how parameters are set to generate the graphics wanted. Note Creating a multiple figure graph can sometimes be cumbersome. In some situations, you need to adjust the margins to be smaller or the font size to be larger. A graph’s exact layout is still to some extent device-dependent, so if you plan on creating a PostScript graph, don’t do the graph on the screen until the layout satisfies you. Adjust everything on the final output device. For PostScript graphs, specific options can be set using ps.options. For example, the overall font size can be set to 14 pt (1 mm is approximately 3 pt, or 1 inch is approximately 72 pt) with > ps.options(pointsize=14) Alternatively, you can set the character expansion. Enter > par(cex=1.5) to make the characters 50% larger.
5.6.1
Layouts Using Trellis Graphs
If you are planning to create an advanced figure layout, you should read the following sections and the next chapter on Trellis displays. If Trellis displays fulfill your needs, they can easily be composed into a multifigure graph of almost any layout. If you are already familiar with Trellis displays, jump to Section 6.4.6 (page 165). After having obtained an overview on what methods are available and how they can be used, you can take your pick. Select the method that is most convenient for getting the desired graph.
5.6.2
Matrices of Graphs
A very straightforward way of adding several figures to the same graph is to declare the graph as a matrix of figures. The graph can be a 2 × 2 matrix of figures, containing 4 figures. It can either be filled rowwise or columnwise. To illustrate, a 2 × 2 figure layout can be created with
5.6. Figure Layouts
> > > > > >
par(mfrow=c(2, 2)) x par(mar=c(2, 2, 1, 1)) such that there is a space of two lines on the lower and the left-hand side of the graph, and a single line on top and on the right-hand side of the graph. Note Once the graph page is filled with figures, a new page is opened (the screen is cleared) on creating another figure. One can jump to the next figure by using > frame() Entering frame() twice leaves a field blank. Restore the default settings by allowing for a single figure per page: > par(mfrow=c(1,1))
5.6.3
Multiple-Screen Graphs
Another way of creating a set of figures in the same graph is to use the split.screen function. In its simplest use, it works like the par(mfrow) command. One would enter > split.screen(c(2, 2)) to generate a 2 × 2 matrix of figures. In comparison to the par(mfrow) variant, splitting the screen provides the ability to hop around between
142
5. Graphics
areas. On the other hand, it does not automatically plot the next graph in the next plot area but waits for explicit specification of another plot area. > screen(2) will activate the second screen in the display. The matrix of screens can be filled either horizontally (which is the default) or vertically by setting byrow=F in the call to split.screen. It is possible to have a matrix of differently sized graphs or even any layout. If a new graph is started, its initial coordinates range from 0 to 1, similar to 0–100%). Each screen in the graph is defined by its lower left and top right corner. The following command defines a layout of a 2 × 1 matrix, where the first row stretches from 0 to 1 on the x-axis and from 0 to 3/4 on the y-axis. The second row stretches from 0 to 1 on the x-axis and from 3/4 to 1 on the y-axis. > split.screen(figs=matrix(c(0, 1, 0, 3/4, 0, 1, 3/4, 1), 2, 4, byrow=T)) Note that the matrix argument to figs must have four columns. Each row defines the corners of one of the screens. The screen layout is closed by using close.screen. > close.screen(all=T)
5.6.4
Figures of Specified Size
An alternative to what we have looked at so far is to fill an empty graph successively with figures and define the size of each figure as we go (i.e., right before the figure is created). We need to open a new graphics page, which is done by entering the command > frame()
# open next graph
After having opened a new graphics page, the coordinates of the plot are set to (0,0) for the lower left corner and (1,1) for the top right corner. Let us now plot the Geyser data set that comes with S-Plus. We want to set up a figure such that a scatterplot of the data appears as the main plot with the addition of a histogram of the x data on top of the scatterplot and a boxplot of the y data to the right-hand side of it. The layout we have in mind looks like the one in Figure 5.5. The plot itself should have the lower left corner coordinates (0, 0) and the top right corner coordinates (0.7, 0.7). Therefore, the figure on top of the data plot has the lower left corner (0.0, 0.7), and the upper right corner is (0.7, 1.0). For the vertical boxplot on the right side, we get the lower left corner (0.7,0) and the upper right corner (1.0, 0.7). By experimenting with the different sizes and the borders, we discover that it is better to have the figures overlapping a little bit.
5.6. Figure Layouts (0, 1)
(0.7, 1)
143
(1, 1)
Histogram of x (0, 0.7)
Graph of (x, y)
(0, 0)
(0.7, 0.7) (1, 0.7)
Boxplot of y
(0.7, 0)
(1, 0)
Figure 5.5. A customized graph layout.
> frame() > par(fig=c(0, 0.7, 0, 0.7)) > plot(geyser$waiting, geyser$duration, pch="*", xlab="Waiting Time", ylab="Duration of Eruption") > title("\nOld Faithful Geyser Data Set")1 > par(fig=c(0, 0.7, 0.65, 1)) > hist(geyser$waiting, xlab="") > par(fig=c(0.65, 1, 0, 0.7)) > boxplot(geyser$duration) Finally, we obtain Figure 5.6.
1 The control character \n is a “linefeed” character, advancing to the next line. In this way, the title gets closer to the graph.
5. Graphics
0 20 40 60
144
40
60
80
100
5 4 3
• 50
60
70 80 Waiting Time
2
•
1
• • • • • •• ••• •• •• • • •••• • • • •••• •• • ••• • ••• • • • • • •••• •• • • •• •••• ••• •• ••• • • • • • ••••• ••• ••••••••• • • •• •• ••••••• •••••• ••• •••• ••••••••••••••• ••••••• • • • • • • • • • • • • • • • • • • • • ••• •• • ••• •• • • • • ••••••••••••••••••••••••••••• ••• • •••• •••••• ••• • •
1
Duration of Eruption 2 3 4
5
Old Faithful Geyser Data Set
90
100
110
Figure 5.6. A customized display of the Geyser data set.
5.7. Exercises
145
5.7 Exercises Exercise 5.1 In a single figure, plot the functions sin, cos, and sin + cos with different colors and line styles. Use 1000 points in the interval [−10, 10] and label the figure with a title, subtitle, and axis labels.
Exercise 5.2 Construct two vectors, x and y, such that the S-Plus command > plot(x, y, type="l") creates the following figures: a) a rectangle or square b) a circle c) a spiral Hint: Do not think of y as being a function of x (in a mathematical sense), but think of how the trace of the figure can be created. For drawing circles you might use the property that, for any x, a point (sin(x), cos(x)) lies on the unit circle with origin (0, 0) and radius 1.
Exercise 5.3 We consider the so-called Lissajous figures. They are defined as z(x) =
sin(ax) z1 = z2 sin(bx)
or, in different notation, z1 (x) = sin(ax) z2 (x) = sin(bx) with a and b positive integers and x between 0 and 2π. Plot z1 against z2 , using lines to connect the points, and choose different pairs of values for a and b. Plot several figures on a single sheet. On what do the forms of the curves depend? Compare, for example, the figures for (a, b) = (3, 4), (3, 6), and (6, 8).
146
5. Graphics
5.8 Solutions Solution to Exercise 5.1 What we need to create this graph is a sequence of x values, for which we calculate the values sin(x) and cos(x). If several functions are to be plotted within one picture, the order of plotting the figures is important. The arguments to the plot function, usually the first curve, determine the boundaries of the figure. The following curves are then added to the existing picture within the existing limits. If you want to see more of the sin(x) + cos(x) curve than that which fits within the boundaries of sin(x), extend the y limits by using the option ylim of the plot function or – and this is easier – plot the curve sin(x) + cos(x) first.
−1.5
−1.0
−0.5
y values 0.0
0.5
1.0
1.5
Trigonometric Functions
−10
−5
0 x values sin(x), cos(x), and sin(x)+cos(x)
5
10
Figure 5.7. Trigonometric functions.
> > > > >
x > > > > >
x lines(x, y1) > lines(x, y2, lty=2, col=2) > lines(x, y3, lty=3, col=3) > title("Trigonometric Functions", "sin(x), cos(x), and sin(x)+cos(x)")
148
5. Graphics
Solution to Exercise 5.2 The idea of this exercise is to learn how to think in terms of geometric figures and not in terms of mathematical functions, where y is a function of x. (a) A rectangle is a simple figure consisting of four lines. In order to draw a rectangle, we need to create two vectors, x and y, containing the coordinates of the corners. These vectors must have five elements, as the line has to return to the first point to complete the figure.
A rectangle
Figure 5.8. Geometric figures (a): a rectangle
> > > >
x > > > >
z >
z > >
0.0 sin(3 * x)
0.5
1.0
−1.0
−0.5
0.0 sin(3 * x)
0.5
1.0
−1.0
−0.5
0.0 sin(3 * x)
0.5
1.0
−1.0
−0.5
0.0 sin(7 * x)
0.5
1.0
sin(8 * x) −1.0 −0.5 0.0 0.5 1.0
−0.5
sin(11 * x) −1.0 −0.5 0.0 0.5 1.0
−1.0
Figure 5.11. Lissajous figures for different values of a and b.
6 Trellis Graphics
Cleveland (1993) introduced the idea of Trellis displays as a graphical way of examining high-dimensional structure in data by means of conditional one-, two-, and three-dimensional graphs. The problem addressed is how observations of one or more variables depend on the observation of other variables. To answer this question, the data are subset (split up into groups) according to the observations of the conditioning variables – the ones that are supposed to have an influence on the others. Take an example. We want to determine how the relation between body weight and height of a set of people depends on their sex. To inspect this relationship graphically, we can split up the data into the groups male and female. Each of the two groups is displayed in a separate graph instead of blurring the sex effect in a single graph. If we had another variable available, say age group, observed as young, intermediate, and old, we could subset the data again. For each combination of observations for age and sex – six altogether – we generate a separate display of body weight and height. Arranging the displays in a reasonable way, there is a lot of information that can be derived. A Trellis graph layout is such an array of graphs, arranged as sketched in Figure 6.1. We can get the basic idea by looking at Figure 6.1. If the relationship between body height and weight for males is to be compared for different age groups, we need to compare the graphs in the first row of the graph matrix. How would we answer if there is – for young men and women – a different relationship between height and weight? Well, simply by comparing the two graphs in the leftmost column.
154
6. Trellis Graphics
sex=male age=young
sex=male age=intermediate
sex=male age=old
graph of weight, height
graph of weight, height
graph of weight, height
sex=female age=young
sex=female age=intermediate
sex=female age=old
graph of weight, height
graph of weight, height
graph of weight, height
Figure 6.1. An example of a Trellis graph layout.
If we switch to the middle column, we can graphically compare the relation between height and weight for the middle age group. Finally, the rightmost column gives us an idea of how different the height/weight relation is for old men and women. As an aside, Figure 6.1 displays the origin of the name Trellis, based on its layout of graphs. Altogether, a graph as sketched in Figure 6.1 displays the four-dimensional structure in the variables height, weight, sex, and age. Note Trellis graphs are implemented somewhat differently from the other graphics in S-Plus and this is why we treat them in a separate section. All Trellis graphs follow the same structure, have a consistent interface and parameter setting, and offer features such as saving graphs in variables. You might find Trellis graphs more appealing than the traditional graph routines just because of the layout, rendering, and default coloring. Therefore, you may want to use Trellis graphs as your default graph type.
6.1 An Example The data set singer contains measurements of the heights of singers in the New York Choral Society together with their assigned voice part. Entering > singer displays the entire data set. You see that it consists of two variables of name height and voice.part. We are interested in figuring out if the height of the singer differs between voice part groups. This is exactly the type of question for which Trellis has appropriate displays. We can get a set of histograms, one for each voice part’s height, by entering
6.1. An Example 60 Alto 2
65
70 Alto 1
75
60 Soprano 2
65 70 Soprano 1
155
75
40
30
Percent of Total
20
10
0 Bass 2
Bass 1
Tenor 2
Tenor 1
40
30
20
10
0 60
65
70
75
60
65
70
75
height
Figure 6.2. Heights of singers for the voice parts in the New York Choral Society.
> histogram(∼height | voice.part, data=singer) The histogram is shown in Figure 6.2.1 The graph easily reveals the basic structure of how the height of the singers differs for different voice parts. The height of the bass and tenor voice parts is typically larger than the height of the alto and soprano parts. Of course, this leads directly to the suspicion that the voice part is just an indicator for the sex of the singer. One would want to obtain the data for sex to condition on male and female and redo the graph. Unfortunately, the data are not available. Note some specificities of the call to the histogram and other Trellis graph functions. At first, you might have realized already that we called a function histogram to generate these histograms, although we have used the function hist before to generate a single histogram. histogram is 1 The strip background color is modified to make the labels more legible. This will be explained shortly. The commands used are the following.
> > > >
strip.bg xyplot(y∼x) is sufficient to display the graph on the graphics device. Note that the command xyplot(y∼x) on the command line calls the print function, when entered on the command line. The following note points out a pitfall when the graph is not created from the command line. Note If a Trellis graph function is not called from the top level (the interactive environment), a statement like histogram(∼x) calls the function histogram. The function histogram generates a histogram figure and returns it to the calling function without displaying it. In those cases, calling the print function explicitly is necessary to display the graph, as in print(histogram(∼x)) Otherwise, there will be no display of the graph calculated. This is important to keep in mind when writing functions and scripts. You might want to note that this is consistent with S-Plus’ behavior in other cases. Entering a statement like 3*5 on the command line returns the result; the same statement embedded in a block (like a function, as we will see later) forces S-Plus to calculate the result, but it will not be displayed unless you use print. Here is an example: > x xyplot(sin(x)∼x) > title("Sine Curve") Now, enter the following and observe closely what happens. > { xyplot(cos(x)∼x); title("Cosine Curve") } Try the same and omit the curly brackets.
6.3 Output Devices We have seen that Trellis graphs can be created without actually printing them. Mostly though, graphs are created to be displayed either on the screen or on a printout. Trellis uses the standard S-Plus output devices with slight modifications to their settings. This implies that it can make a
6.3. Output Devices
159
difference whether you display a Trellis graph on a standard graphics device or a Trellis graphics device customized for the needs of Trellis graphs. The following note can therefore be relevant to you. Note If no graphics window is opened yet and you create a graph to be displayed on the screen, S-Plus will open a graphics window. It will select either a “classical” or Trellis graphics window automatically. A Trellis device is nothing a classical device with settings that are specifically customized for Trellis displays. These include margins, colors, and other styles. One may also display Trellis graphs on the noncustomized devices that are opened (for example) by calling motif(), but the output will not be as nice as on a Trellis display. You can tell the difference by the background color: A Trellis-type graphics window has a gray background, whereas a standard graphics window has a white background (unless you changed the settings). The gray background on Trellis devices will not be exported to graphics files or Powerpoint slides, it is just a kind of virtual setting for the window. To a large extent, you do not need to worry about devices. However, the following will prove useful to you as soon as you start working with multiple devices or saving graphs to files. In order to print a Trellis graph, you need an open output device (a graphics window on the screen or a file on disk). Graphics commands are sent to that device. The general way of opening a Trellis device is to call trellis.device with the device to open as a parameter. For example, > trellis.device(graphsheet) opens a graph sheet window on Windows systems, > trellis.device(motif) opens a motif graphics window on a UNIX workstation, and > trellis.device(postscript, file="trellis1.ps", color=F) opens a PostScript file. An overview of some Trellis devices available is given in Table 6.2. After having started a graphics device, we can generate a new Trellis graph and display it right away; for example, by calling > histogram(∼height | voice.part, data=singer) We may also display a figure that has been created beforehand and saved in a variable. > graph.singers print(graph.singers) All comments for other devices hold for Trellis devices as well. You can open a set of devices, switch between them using dev.list() and
160
6. Trellis Graphics Table 6.2. Trellis graphics devices.
System
Trellis call
Windows trellis.device(graphsheet) trellis.device("printer", format="EPS") trellis.device("printer", format="clipboard") UNIX trellis.device(device)
Device Graph sheet window PostScript printer Windows clipboard device can be postscript, motif, pdf.graph, java.graph, etc.
Note: There are more graphics formats S-Plus can write, such as PDF and JPEG. See the documentation on Devices. To specify the file to write to, add file="filename" to the function call. dev.set(), and copy from one device to another using dev.copy. If you want to close the last device opened, enter > dev.off() To close all open devices, enter > graphics.off() If the current device is a graph sheet, it will be cleared before a new function call generates new graphs. Multiple graphs created during a single function call are all shown on separate pages of the graph sheet. When the next graphics-generating function is called, the graph sheet is reset to an empty state. If you want to keep the graphs, just open another graph sheet. The last device opened is automatically the current device. Note Our warning about closing devices before printing the graphics is valid for Trellis devices as well. See the note on page 128. In case you have entered graphics commands and do not get a full graph, check if you closed the device (enter dev.list()). If the device was closed, check if you put a print statement around the call to the graphics function.
6.4 Customizing Trellis Graphs 6.4.1
Setting Options
To see the current device settings in the form of a graphical display, use the function show.settings.
6.4. Customizing Trellis Graphs
> show.settings()
161
# show device settings
A graph is displayed where each plot shows the settings for one specific option list. The option names are shown below the graph. They are listed in Table 6.3. Retrieve the specific component you are interested in and look at its settings. In the following example, we retrieve the settings for the line width in Trellis graphs, change its value, and store the settings back to the system settings such that they become active for all lines in Trellis displays. > trellis.plot.line trellis.plot.line$lwd trellis.par.set("plot.line", trellis.plot.line) For all subsequent graphs of the current session, line widths will be “3.” Table 6.3. Trellis plot options.
Option name add.line box.dot dot.line plot.line strip.background wires
add.text bar.fill box.rectangle box.umbrella dot.symbol fill plot.symbol reference.line strip.shingle superpose.line
box.3d cloud.3d pie.fill regions superpose.symbol
Note: The settings above are retrieved and set using trellis.par.get("option") and trellis.par.set("option", ...). Setting options as described makes the settings permanent for all subsequent Trellis graphs. If you do not want to change the default settings permanently, you can as well supply arguments to the current Trellis plot function that perform the desired action. This approach does not call a specific function like trellis.par.set but extends the call to a Trellis function like xyplot. The call to the Trellis function will be somewhat lengthy, but the approach is more transparent in the sense that everything that is set different from the default is contained within the current function call. To see what’s possible look up the documentation on “trellis.args” and “trellis.3d.args.” The following sections will mostly employ this technique.
6.4.2
Arranging the Layout of a Trellis Graph
If the conditioning variable has eight categories, for example, one might not want to plot them on a 2 × 4 or 4 × 2 arrangement but rather in a 3 × 3 layout with one of the fields left blank. The layout is specified by using the layout parameter to define the overall layout, in this case by setting
162
6. Trellis Graphics
layout=c(3,3) in the call to the Trellis function. Where the graphs are positioned or if a field is left blank is indicated by a vector of logical values. A T (TRUE) in the skip argument indicates that the field is to be left blank. We use the singer data to generate the layout described. > histogram(∼height | voice.part, data=singer, layout=c(3,3), skip=c(rep(F,4),T,rep(F,4)))2
60 Alto 1
65
70 Soprano 2
75 Soprano 1
40 30 20 10 0
Alto 2
Percent of Total
Tenor 1
40 30 20 10 Bass 2
Bass 1
0
Tenor 2
40 30 20 10 0 60
65
70
75
60
65
70
75
height
Figure 6.3. Customizing a Trellis graph arrangement.
The result is shown in Figure 6.3. The first value of layout determines the number of columns in the graph, the second the number of rows (just in contrast to the specification of a matrix dimension). As a 3 × 3 matrix consists of 9 fields, the parameter skip is a logical vector of length 9 with a T in position 5 to indicate that field 5 is to be left blank. If layout is a vector of length 3, the third value defines the number of pages to generate. In the same way, one can change the aspect ratio of a graph. The aspect is the ratio of the height and width of a graph, or, to be precise, aspect ratio = y-axis length/x-axis length. 2 The
footnote about changing the strip color (page 155) holds here as well.
6.4. Customizing Trellis Graphs
163
The aspect ratio is important in the perception of a graph. Setting it to 1 creates a graph where the x-axis and y-axis have the same length. Setting it to "xy" generates a graph with a 45◦ banking. To make the x-axis 25% longer than the y-axis, set aspect=1/1.25. For 3D figures, you need an aspect argument of two values, defining height and depth relative to the width of the graph. Note that you don’t need to care about the size of the graph in inches or pixels; this is still determined by S-Plus to use the space on the page optimally.
6.4.3
Ordering of Graphs
You might have noticed that S-Plus has a standard way of arranging the graphs belonging to certain levels of the conditioning variables. If there is only a single conditioning variable, the graph that conditions on the first level of the conditioning variable shows up in the lower left corner. The second graph is to the right of it, until the bottom row is filled up. The second row from the bottom is filled next, and so it continues until all rows are filled. Check out the arrangement of levels in the original data by querying the levels of the conditioning variable. For the Barley data set, the levels of the variable site are obtained as follows: > levels(singer$voice.part) "Bass 2" "Bass 1" "Tenor 2" "Alto 2" "Alto 1" "Soprano 2"
"Tenor 1" "Soprano 1"
The ordering of the graphs – how the matrix of graphs is filled up – is shown in Figure 6.4 (left).
7
8
9
1
2
3
4
5
6
4
5
6
1
2
3
7
8
9
Figure 6.4. Default ordering of graphs in a 3 × 3 Trellis arrangement with a single conditioning variable (left) and ordering of graphs setting as.table=T (right).
164
6. Trellis Graphics
Note You might wonder why the ordering is not in the way one reads a book – from the top left to the lower right. You can arrange the graphs in this way by setting as.table=T. The default arrangement is as described because it creates the analogy to numeric x-axis and y-axis in putting the first (the smallest) value to the left on the x-axis and to the bottom on the y-axis. This is especially important if the categories are ordered. If the categories represent ordered factors such as low, medium, and high education or income, placing the “lowest” category lowest (physically) in a graph makes the reading of the graph much more intuitive. For example, the group with low education and low income is in the bottom left corner of the graph, whereas the group with high education and high income is in the top right corner, where the high x and y values are found on a numeric scale. The order of the graphs (panels) within one of the two arrangements is determined by the levels of the conditional factors. Changing the order of the levels is described in Section 12.3 (pp. 378 ff.).
6.4.4
Axis Customization
Suppose we want to have larger characters on the axis ticks in a scatterplot. The command > xyplot(duration∼waiting, data=geyser, scales=list(cex=1.5, tick.number=10)) generates the graph with the axis tick labels 50% larger than the standard size (1.5 times the size) and approximately 10 tick marks. S-Plus might alter the specification slightly if the number of ticks scales better. If we wanted to set different specifications for the x-axis and y-axis, we would supply to the scales argument a list that contains two lists of names x and y. For example, to set different character sizes for the axes and to have horizontal y–axis labels and vertical x– axis labels, we could enter > xyplot(duration∼waiting, data=geyser, scales=list( x=list(cex=1.5, las=2), y=list(cex=1.0, las=1) )) To set logarithmic axes, specify log=T either for a specific axis or for both axes. If you are interested in a specific base for the logarithm, for example if you would like to have powers of 2 as axis tickmarks, set log=2.o In some cases, for example if the range of the data varies widely between panels, one wants to allow for individual axis ranges for each panel. In this case, specify relation="free". Trellis will then leave space between panels to indicate the individual axes.
6.4. Customizing Trellis Graphs
165
There are more possibilities, for example specifying to what side of the panels to place the axes or how much space to leave between panels. For more details, see the documentation for trellis.args.
6.4.5
Modifying Panel Strips
The panel strips are the horizontal bars on top of each graph in a Trellis layout. They indicate the conditioning variable’s values for the graph. For the 2D Trellis functions, there is an argument strip: an optional specification of a function that draws the strips across the panels. It is preset with a call to the function strip.default. Take the example of calling dotplot. A call to dotplot such as > dotplot(year∼yield | site, data=barley) is equivalent to > dotplot(year∼yield | site, data=barley, strip=function(...) strip.default(...) ) If you are interested in all details, have a look at strip.default or check only the arguments of the function using > args(strip.default) You will see that strip.default is pretty long, but we can make our life simple. We can define a new strip function that simply calls strip.default with different settings. By looking at the documentation for strip.default, we find that the setting we are interested in for this example is the style setting. We want to set it to the value 1 to achieve that “the full strip label is colored in background color and the text string for the current factor level is centered in it” (documentation excerpt). > dotplot(year∼yield | site, data=barley, strip=function(...) strip.default(..., style=1) ) does the job for us.
6.4.6
Arranging Several Graphs on a Single Page
If you use Trellis graphs, you can experiment with the arrangement of the graphs by creating the graphs and storing them in variables. Once the graphs are generated, they can be moved around on the page. There are basically two ways of getting a multigraph page. First, a matrix of graphs with all graphs in rectangular equally sized areas can be generated. Second, we can give the separate graphs different sizes and any location on the page. Let us start with the first layout type. > graph1 graph2 graph3 waiting.rounded graph4 print(graph1, split=c(1,1,2,2), more=T) > print(graph2, split=c(1,2,2,2), more=T) > print(graph3, split=c(2,1,2,2), more=T) > print(graph4, split=c(2,2,2,2), more=F) Figure 6.5 was generated using the commands above. You see that a conditioned graph like graph4 can be part of a graph layout as well. Note also that we used the option as.table to change the ordering of the histograms. Look again at what we entered. The split argument allows the division of the graphics window into several areas and plots the graphs in them. The settings are as follows: print(trellisgraph, split=c(plot row, plot column, layout rows, layout columns), more=T/F)
6.4. Customizing Trellis Graphs
167
If the graphs should be of different size, note that the graphics pages’ lower left corner (LL) is denoted by the coordinates (0, 0), and the upper right corner (UR) by the coordinates (1, 1). We can position a graph on the page by specifying its lower left and upper right corner on the page in the form (xLL , yLL , xU R , yU R ). > print(graph1, position=c(0, 0.35, 1, 1), more=T) > print(graph2, position=c(0, 0, 1, 0.45)) Entering these commands generated Figure 6.6.
duration
5 4 3 2 1 50
60
70
80
90
100
110
Percent of Total
waiting
25 20 15 10 5 0 40
60
80
100
waiting
Figure 6.6. A multigraph page of Trellis displays with different sizes.
6.4.7
Updating Existing Trellis Graphs
If a Trellis graph is stored in a variable, it can surely be modified. A function to modify stored Trellis graphs is the update function. Suppose we generated a graph by entering > graph bettergraph > > >
x panel=function(x, y, ..., subscripts) { ... } the indices of the data used for the current panel will be passed to the panel function. The indices denote the row numbers in the overall data frame that make up the data set for the current panel. Suppose we plot the profiles of a drug concentration in patients over time, as in the Theophylline data set using xyplot(conc∼Time |Subject, data=Theoph). Each panel shows the concentration-time profile of a single subject. If we wanted to add more information to each panel and that information is stored in another variable, we need to know the rows of the current subject’s data in the data frame (in order to extract the additional information for this particular subject). Such additional information could be the gender of the subject or the dose the subject received. This example is very illustrative of how subscripts can be used, so let us look at some example code. We use the Theophylline data set stored as Theoph and add (artificial) treatment groups and dosing times to it. First, we copy the data set to a new object, x, and ensure it is a data frame by calling as.data.frame when copying it.4 > x x$PID trt.group x$treatment > > >
z par(mfrow=c(1,2)) > contour(hist2d(waiting, duration), xlab="waiting time", ylab="eruption time") > contour(hist2d(waiting, duration, xbreaks=seq(40, 110, by=5), ybreaks=seq(0, 6, by=1.5)),
5
5
7.2. Graphical Exploration
10
5
15
30 20 40
4
4
30 40 50
10
217
5
eruption time 3
eruption time 3
20 15
15 2520 10
2
2
20
10
1
1
30
50
60
70 80 90 waiting time
100
40
50
60
70 80 90 100 110 waiting time
Figure 7.12. A contour plot of the Old Faithful Geyser data using contour.
xlab="waiting time", ylab="eruption time") This excursion into data analysis using graphics for exploring structure reveals that the term “exploratory data analysis” describes a search for effects discovered by using many different displays and summaries of the same data set. A single number or display can never describe the whole complexity of a data set. The Trellis function contourplot generates an almost identical display. To refresh how to place multiple graphs on a single page, here are the commands to generate Figure 7.13. The calls to hist2d are as before for the functions wireframe and levelplot, we only change the breaks for the x- and y-axis. > z2 z2grid z2grid$hist z3 z3grid z3grid$hist print(contourplot(hist∼x*y, data=z2grid, xlab="waiting time", ylab="eruption length"), split=c(1,1,2,1), more=T) > print(contourplot(hist∼x*y, data=z3grid, xlab="waiting time", ylab="eruption length"), split=c(2,1,2,1), more=F) The function contourplot accepts more parameters than the ones we specified. Setting region=T underlies the contours with a level plot, for example. This concludes our tour of exploring the Barley and Geyser data sets. We have seen various displays of the data and gained some insight into what information it contains. You might now want to proceed with pursuing questions you might still have about the data or you could now go ahead and look at some of your own data. To round off, the following section references some more graph types: dynamic graphics and old-style graphics (non-Trellis).
7.2. Graphical Exploration
7.2.1
219
Interactive Dynamic Graphics
S-Plus offers some basic dynamic and interactive graphics functionality. Data can be rotated, subsets can be marked (“highlighted”), and individual observations can be identified using the mouse. The functions are listed in Table 7.3. Use the mouse to define a brush and move it over point clouds to mark (“highlight”) them. If you get stuck, consult the manuals. Just do it. Table 7.3. Multivariate dynamic graphics functions.
S-Plus function
Description
brush spin
Interactive marking of subsets on 2D scatterplots Rotating data, marking and highlighting of subsets
7.2.2
Old-Style Graphics
Occasionally one might want to use S-Plus’ old-style graphics. The Trellis concept has extended the functionality and offers better visualization (for example, by having axis tickmarks on all four pages of a graph). It can nevertheless come in handy to know about alternatives, and some of them are given in Table 7.4 and Table 7.5. The Trellis-type functions were given in Table 6.1 (p. 157). Most of the graphics functions have a long list of optional parameters. It is, of course, possible to use a function only in its elementary form (like pie(x)), but if you want to add colors, modify the labels, or explode one of the pies in the pie chart, you need to know what you can do. The boxplot function, for example, currently has 27 optional parameters. Table 7.4. Old-style graphics functions.
S-Plus function
Description
boxplot density
Boxplot display Density estimate for the distribution of the data. Use plot(density(x)) to see it graphically Histogram display Identification of the data point next to the point clicked on in the graphics window Pie chart Standard plot, depending on the data
hist identify pie plot
220
7. Exploring Data Table 7.5. Old-style multivariate graphics functions.
S-Plus function
Description
Barplot (bars with subbars stacked) Plot principal components and factor analysis results, show data and variables in a two-dimensional coordinate system boxplot Boxplot(s) of one or more variables in a single graph contour Contour lines of a two-dimensional distribution to access the common distribution function coplot Plot matrix of two variables conditioned on the values of a third variable dotchart Dotchart (values on parallel lines stacked) for a vector with optional variable grouping faces Chernoff faces illustrating by means of face elements a high-dimensional data set (up to 15 variables) hexbin Hexagonal binning, a display technique for spatial data using hexagonally shaped bins image Image plots, color scales for (geographical) heights or density values on a two-dimensional grid matplot Plotting columns of matrices against other matrix columns by using different symbols for each combination pairs Two-dimensional x − y scatterplots in a matrix of scatterplots for each combination of variables persp Perspective three-dimensional surface plots on a grid for density estimates stars Star plots of multivariate data in which each observation is displayed as a star. The larger the value the further away the point from the middle barplot biplot
7.3 Distributions and Related Functions This section examines a variety of distributions covered by built-in S-Plus functions. As we will see later, adding further distributions to the existing set is also very straightforward. We divide the distributions themselves into continuous and discrete distributions. Statistical distributions occur in many practical data analysis situations. Most models to describe data behavior are based on distributional assumptions and derive the estimates for unknown model parameters from the underlying (assumed) distributions. The most important functions related to a distribution are as follows. – The density function, which specifies a random variate distribution (with fixed parameters).
7.3. Distributions and Related Functions
221
– The probability function, which is the integral over the distribution function or, in the discrete case, the sum up to the specified point. In other words, the probability function at x gives the probability of the random variable X being less than or equal to x. – The quantile function, which is the inverse of the probability function. – The random number generating function, which generates numbers distributed according to the specified distribution function. For a large set of random numbers, the distribution of the random numbers (visualized, for example, by a histogram), looks approximately like the density function. S-Plus has a systematic naming of functions related to distributions. The functions related to the same distribution have the same name except for the first letter. The first letter indicates what type of function it is. Table 7.6 explains the system. Table 7.6. Categorization of distribution-related functions in S-Plus.
Type1 character
Function type
d p q r
Distribution function Probability function or cumulated density function Quantile function (inverse probability function) Random number generation
1
The significant letter for identifying the functionality of the distribution function is either d, p, q, or r. See also Table 7.7. To identify a specific function, we need to know the abbreviations for the distributions available to us, which are listed in Table 7.7 and Table 7.8. To determine the function’s name, combine the characteristic letter from Table 7.6 with the abbreviation in Table 7.7 or Table 7.8. For example, to generate random numbers from the normal distribution, the function to use is rnorm (“r” + “norm”). There are four functions related to the normal distribution dnorm for calculating the value of the distribution function pnorm for calculating the value of the probability function qnorm for calculating the inverse probability function rnorm for generating random numbers from the normal distribution These four functions all have the same parameters, as they refer to the same distribution, namely mean and sd, the mean and the standard deviation, respectively. Both arguments are optional. They are preset to define the standard normal distribution with mean 0 and standard deviation 1.
222
7. Exploring Data
Table 7.7. Distribution-related functions in S-Plus: Discrete distributions.
Distribution
S-Plus name1
binomial geometric hypergeometric negative binomial Poisson Wilcoxon (rank sum)
binom geom hyper nbinom pois wilcox
Parameters size prob m, n, k size lambda m
prob
prob n
1
The characteristic letter, one of d, p, q, r, plus the listed name, gives the name of the S-Plus function. Use the sample function to generate random numbers from a discrete distribution. Table 7.8. Distribution-related functions in S-Plus: Continuous distributions.
Distribution
S-Plus name1
Parameters2
beta Cauchy chi-square exponential F gamma logistic lognormal normal normal, multivariate3 stable Student t uniform Weibull
beta cauchy chisq exp f gamma logis lnorm norm mvnorm
shape1 location=0 df rate=1 df1 shape location=0 meanlog=0 mean=0 mean=rep(0,d)
shape2 shape=1
stab t unif weibull
index df min=0 shape
skewness=0
df2 scale=1 sdlog=1 sd=1 cov=diag(d)
max=1
1
The characteristic letter, one of d, p, q, r, plus the listed name, gives the name of the S-Plus function. 2 If parameters are preset with a default value, as in mean=0, they do not need to be specified unless a value other than the default is desired. 3 There is no quantile function for the multivariate normal distribution. Note The standard deviation, not the variance, is the argument used for dispersion by the functions for the normal distribution. The standard deviation is simply the square root of the variance. All these functions take numbers and vectors as input arguments. For a vector, the corresponding value for each element is computed. For example,
7.3. Distributions and Related Functions
223
> dnorm(-3:3, 0, 1:7) calculates dnorm(-3, 0, 1), dnorm(-2, 0, 2), ..., dnorm(3, 0, 7). You can check out to see the numerical accuracy of the distributional functions. Calculate the difference between some values x and p(q(x)) or q(p(x)), which should be 0 in theory (p stands for the probability and q for the quantile function of the distribution). Note As you might know, the random numbers generated by a computer are never really random. They consist of a sequence of numbers where each number depends somehow on the previous number generated. S-Plus uses the variable .Random.seed to store the state of the random number generator. You can use set.seed(n), where n is an integer number, to produce the same sequence of random numbers several times, maybe to test a self-written routine. If you want to switch again to a more uncontrolled sequence, you can safely delete the object .Random.seed from your working directory. When you call any random number generating function, S-Plus will regenerate it (but out of your explicit control).
Graphing Distributions You will often want to see what a distribution looks like, which usually means looking at the density function. Here, we need to make a distinction between continuous and discrete distributions. Recall that a continuous density has a continuous support; that is, all values of the density function in a given interval are greater than zero. This interval usually ranges from minus infinity to plus infinity. For this reason, we cannot graph the “whole” distribution from minus to plus infinity, but have to restrict ourselves to a part of it. A good idea is to plot, for example, 90% of the density support. We find by using qnorm(c(0.05, 0.95)), qt(c(0.05, 0.95), 5), and qcauchy(c(0.05, 0.95)) that the interval from −6 to +6 covers more than 90 percent of all the three densities. Similarly, we can determine that the densities for all distributions at x = 0 all lie below 0.4. We can therefore use the function xyplot and initialize it by plotting the vector of 0 and 0.4 on the y-axis against −6 and +6 on the x-axis. We now need to write a panel function that graphs our desired densities. Note that the values passed to xyplot are never plotted as the panel function ignores them. xyplot(c(0, 0.4)∼c(-6, 6), main="A graphical comparison of distributions", ylab="Density", xlab="", npoints=1000, panel=function(..., tdf=5, npoints=100)
224
7. Exploring Data
{ args > > >
n qnorm(c(alpha/2, 1-alpha/2), m, s)
Creation of a Graph Illustrating a Statistical Test The following describes how Figure 7.16 was created. It combines what we have learned about plotting distributions with graphical illustration of statistical testing. The combination of many functions shows how you can easily create even complex graphics such as Figure 7.16. We write a call to the function xyplot and define (in the call) a new panel function. The arguments passed to xyplot determine the x- and y-axis ranges that we can not influence any more from within the panel function. The panel function has the parameters alpha, the significance level, and sample1 and sample2, the data sets. The call to xyplot sets those parameters(!), everything needed is contained within the call to xyplot. xyplot(dt(seq(-3, 3, length=1000), df=19)˜ seq(-3, 3, length=1000), xlab="", ylab="t density", sample1=barley$yield[barley$site=="Crookston"], sample2=barley$yield[barley$site=="Morris"], alpha=0.05, panel=function(..., alpha, sample1, sample2) { args dimnames(car.all)[[1]]
242
7. Exploring Data
to get only the first component of the list, the names of the cars. Now, we save the names of the cars for later use. > car.names max.travel summary(Type) Compact Large 19 7
Medium 26
Small 22
Sporty 21
Van 10
Na’s 6
We see that there are only 7 cars classified as large in the data, 6 have no classification, and there are more than 20 cars in the groups Medium, Small, and Sporty. Determining the cars with the highest fuel consumption (or the lowest mileage) in the different categories is done by using min in combination with by as follows: > by(Mileage, Type, min, na.rm=T) Compact Large Medium Small Sporty 21 18 20 25 24
Van 18
Note that we need to supply the parameter na.rm=T to the min function, as there are missing values in the data. The minimum will be missing if any of the data are missing. In the same fashion, we obtain the maximum values. > by(Mileage, Type, max, na.rm=T) Compact Large Medium Small 27 23 23 37
Sporty 33
Van 20
You might have produced slightly different output, because we used the command unclass(by(...)) to obtain this shortened output. A first display of the data can be generated using splom. To analyze the four variables HP, Tank, Mileage, max.travel, we create a new data set containing only these data. > car.subset splom(∼car.subset) The scatterplot matrix (“splom”), a matrix of scatterplots for each combination of two variables, is shown in Figure 7.18. We immediately see strong relationships between many variables, such as between HP and tank size or between tank size and mileage in a negative way: the smaller the mileage, the larger the tank, and vice versa.
7.7. Solutions
400
450
243
450
400
max.travel 350
300 300 30
35
350
35
30
Mileage 25
20 20 20
25
25
25
20
Tank 15
10 200
15
10
250
250
200
HP
150
100 100
150
Figure 7.18. Some components of the Car data set (horsepower, tank size, mileage, and maximum travel distance).
We can confirm this by numbers by calculating the correlation matrix between the variables. If we just enter cor(car.subset), we get an error message telling us that S-Plus refuses to calculate the correlation for a data set with missing values. Looking up the cor function in the help system tells us that we need to set na.method="omit" to ignore the missing data for the correlation calculation. > cor(car.subset, na.method="omit") HP Mileage Tank max.travel HP 1.000000 -0.66615 0.67641 -0.020566 Mileage -0.666153 1.00000 -0.80134 0.372705 Tank 0.676410 -0.80134 1.00000 0.212167 max.travel -0.020566 0.37271 0.21217 1.000000 The results of our first guess can be immediately confirmed, as the correlation between HP and tank size is 0.68. There is also a strong negative correlation between tank size and mileage, about −0.8. Some functions do not have options to remove the missing values before doing the calculation. In this case, we would need to remove all data (all rows) that contain one or more missing values.
244
7. Exploring Data
The function is.na is useful for checking if any data are missing, and if you remember that TRUE values are coded with 1 and FALSE values with 0, then it becomes clear what to do. We sum up the TRUE and FALSE values obtained from is.na and see if the sum is 0 (indicating that none of the data are missing) or greater than 0 (indicating that one or more variables are missing). > problems problems problems car.without.missings > > >
n 1-pchisq(test.stat, 6) 0.2271776 This result helps give us confidence that the S-Plus random number generator is doing a good job.
Solution to Exercise 7.5 You can simulate a statistical test’s rejection percentage by generating data from the null hypothesis several times, and by calculating how often the test would reject the null hypothesis, although the data come from the distribution specified in the null hypothesis. In this exercise, we will only work on matrices and vectors, as this is an elegant and efficient way to program in S-Plus. We first generate 1000 samples of size 100 from the null hypothesis, the samples being normally distributed with mean 0 and standard deviation 1. The samples are stored in a matrix with each row representing one sample. > x m z qnorm(0.975, 0, 0.1)
246
7. Exploring Data
The result is a vector of TRUE/FALSE values, and summing them up yields the number of TRUE values, which in our case was 47. The mean value is the mean number of rejections over all 1000 samples, which should be around 0.05 (as we are testing on the 5% level). > mean(z) 0.047 You can also check what’s in the z variable by tabulating the data. > table(z) FALSE TRUE 953 47 To calculate the approximate power of the test when the data come from the alternative, or, in other words, the number of correct rejections of the null hypothesis, we generate samples from the alternative hypothesis and calculate how often the test rejects the null hypothesis. > > > >
x abline(h=0) The last plot is a QQ-plot with the expected line to tell us whether the residuals follow a normal distribution. > qqnorm(resid(fit)) > qqline(resid(fit)) The scatterplot of the data shows that there is indeed an approximately linear relationship between the two variables, which is accentuated by the estimated regression line that has been superimposed. A histogram of the residuals would ideally look like the bell-shaped curve of the normal distribution. The histogram here, with a little imagination, is not so far away from that ideal. The scatterplot of the residuals versus fitted values (values predicted from the regression equation) should show no discernable pattern. A lowess curve has been added to help detect whether or not hidden (or obvious) trends exist in the residuals. The flat curve indicates that perhaps the assumptions of the linear regression are met. A last check is to use a QQ-plot to assess whether the residuals follow an approximately normal distribution. The superimposed straight line is what we would expect if the residuals followed a normal distribution exactly. What we find is that our residuals, although not perfectly following a normal distribution, are not that far away from the line.
8.2 Statistical Models S-Plus offers many possible ways of modeling data according to the assumptions made on the distribution and type of outcome (see Table 8.2). Simple linear regression is done with the lm function, as shown in the examples in the previous two subsections, and analysis of variance (ANOVA) with the aov function. ANOVA is actually a regression model, and this can be run with the lm function, but the aov function makes it easier to define the classification variables used in these models. The glm function is based on exponential family theory and can be used to do logistic, Poisson, and other types of non-Gaussian regression. Such regression models are called generalized linear models (GLMs). The gam function extends the GLM methodology by allowing nonparametric functions to be estimated from the data using smoothing operations. A local regression model is done with the loess function, which is a very convenient way of gaining an understanding of the relationships in the data. Classification and regression tree models can be run with the tree function, and nonlinear models with the nls and ms functions. Classical multivariate models, which include multivariate ANOVA, factor analysis, and principal components analysis, can be fit with the respective functions of manova, factanal, and princomp. Loglinear modeling can be done
256
8. Statistical Modeling Table 8.2. Selected statistical models in S-Plus.
Function
Type of model
lm aov glm gam loess tree nls, ms manova factanal princomp crosstabs survfit coxph
Linear regression ANOVA Generalized linear models Generalized additive models Local regression smoother Classification and regression tree models Nonlinear models Multivariate ANOVA Factor analysis Principal components analysis Cross-classifications Kaplan–Meier survival curves Cox proportional hazards models
with the crosstabs function. Finally, for our short list, the most common survival analysis procedures can be done using the survfit and coxph functions.
8.3 Model Syntax A particularly nice feature of S-Plus is that it takes a unified approach to modeling, including the model syntax. All statistical models in S-Plus specify the response and explanatory variables in the same manner, namely Response∼Explanatory The tilde (∼) can be read as “is modeled by” or simply as “equals.” If we have the response variable, NumCars, and the explanatory variable, Income, and we want to model their relationship, we use the simple formula NumCars∼Income We discuss the slightly more complicated matter of how to use the formulas in the following sections. For the moment, we concern ourselves only with the syntax of the model formula. If you also have information on age and want to incorporate this into the model, simply add it with a plus sign (+) as in NumCars∼Income+Age Interactions are usually of interest and are specified using the colon (:) as shown in the following example: NumCars∼Income+Age+Income:Age
8.4. Regression
257
S-Plus has several more advanced ways of specifying formulas for models. They are listed, along with those already discussed, in Table 8.3. Table 8.3. Model syntax.
Terms included
Functionality
A+B A:B A*B A%in%B A/B A−B Aˆm
Main effects of both A and B Interaction between A and B Shorthand for A+B+A:B A nested in B Shorthand for (A+B) %in% A All of A except what is in B All terms in A and m-order interactions
8.4 Regression We now show, very briefly, how to use elementary regression functions. We include in the descriptions some commonly used modeling and modelbuilding techniques. The nonparametric regression functions, in particular, are one of S-Plus’ major strengths, and you should consult the manual for detailed explanations regarding their use. There are, in principle, two types of model fits in S-Plus, which are categorized by what is returned. Functions auch as linear regression return the parameters of the fit; that is, they return what is needed to determine the functional form. On the other hand, the nonparametric functions auch as ksmooth (kernel smoothing) return a list of points, x and y, where y contains the smoothed data at points x, as there is no explicit functional form for these types of smoothers. For the first type of model, it is helpful to do the fit with, for example, lm, > fit plot(x, y) and then add the fitted line with the command > abline(fit) These commands were shown in Section 8.1.2. In the second case, where the return argument consists of a list with x and y components (and others), you enter the following commands instead: > fit plot(x, y) > lines(fit)
8.4.1
Linear Regression and Modeling Techniques
As we already presented simple linear regression in Section 8.1.1, we use this section to review the unified modeling syntax presented in Section 8.3 and some of the model-building techniques available in S-Plus. The examples in this section use the Swiss fertility data used earlier. To begin, we simply repeat the model shown in Section 8.1.1. The model consisted of the response variable, Fertility, and the explanatory variable, Education, both contained in the data frame swiss. > fit1 summary(fit1) Call: lm(formula = Fertility ∼ Education, data = swiss) Residuals: Min 1Q Median 3Q Max -17.04 -6.711 -1.011 9.526 19.69 Coefficients: (Intercept) Education
Value Std. Error 79.6101 2.1041 -0.8624 0.1448
t value Pr(>|t|) 37.8357 0.0000 -5.9536 0.0000
Residual standard error: 9.446 on 45 degrees of freedom Multiple R-Squared: 0.4406 $F$-statistic: 35.45 on 1 and 45 degrees of freedom, the $p$-value is 3.659e-007 Correlation of Coefficients: (Intercept) Education -0.7558 Note that the output here is exactly the same as shown in the example in Section 8.1.1. We will now build on this simple model. With the unified approach to modeling, we can very easily make use of many sophisticated techniques. For example, suppose we decided that education was not adequate to describe fertility and we wanted to add more terms to the model. We want to update the previous model, and this is done, appropriately, with the update function as follows:
8.4. Regression
259
> fit2 summary(fit2) Call: lm(formula = Fertility ∼ Education + Catholic, data = swiss) Residuals: Min 1Q Median 3Q Max -15.04 -6.576 -1.426 6.126 14.32 Coefficients: (Intercept) Education Catholic
Value Std. Error 74.2319 2.3518 -0.7882 0.1293 0.1109 0.0298
t value Pr(>|t|) 31.5636 0.0000 -6.0968 0.0000 3.7224 0.0006
Residual standard error: 8.331 on 44 degrees of freedom Multiple R-Squared: 0.5746 $F$-statistic: 29.71 on 2 and 44 degrees of freedom, the $p$-value is 6.822e-009 Correlation of Coefficients: (Intercept) Education Education -0.6838 Catholic -0.6144 0.1540 Note that in the updated model, fit2, both Education and Catholic are significantly related to Fertility. Our choice of adding the variable Catholic was more or less random, but we could have used built-in functions to make the choice a little less whimsical. The add1 function takes a list of explanatory variables and adds them one at a time to the current model. The resulting output shows statistics for each explanatory variable had each been included together with the base model. > add1(fit1, .∼.+Agriculture+Examination+Catholic+ Infant.Mortality) Single term additions Model: Fertility ∼ Education Df Sum of Sq <none> Agriculture 1 61.9657 Examination 1 465.6258 Catholic 1 961.6083 Infant.Mortality 1 891.2462
RSS 4015.236 3953.270 3549.610 3053.627 3123.989
Cp 4372.145 4488.635 4084.975 3588.992 3659.354
260
8. Statistical Modeling
Applied to our data, we see that with the largest sums of squares and the lowest Cp value, the variable Catholic actually does provide the best fit when one variable is added to a model that includes Education. On the other hand, we could have chosen to use stepwise regression with the step function or to add a lot of variables and remove them using the drop1 (opposite of add1) function. The choice between using add1, drop1, or step is left entirely up to you. In general, they will yield the same model, but often in practice, they do not. So, have fun. S-Plus offers a host of residual checking techniques. Feel free to use the simple guidelines shown in Section 8.1.2, but you can do some more advanced work by simply plotting the fit from the regression. If you simply plot the fit of the regression model, you obtain six graphs, with one diagnostic plot per graph: residuals versus fitted values, square root of the absolute of the residuals versus fitted values, response versus fitted values, normal QQ-plot of residuals, r-f spread plot, and Cook’s distances. If, for convenience, you prefer to view them all at once on one page, enter the mfrow option as follows to set up the graphical area for a matrix layout of graphs: > par(mfrow=c(3, 2)) > plot(fit1) S-Plus can return the predicted values, standard errors, and confidence intervals using such functions as predict and pointwise. A variable’s polynomial effects are obtained with the poly function. To include linear, quadratic, and cubic terms of X in a model, use poly(X, 3), but beware that the resulting parameter estimates are in terms of orthogonal polynomial contrasts and are not on the same scale as X. To see the parameter estimates on the original scale, use > poly.transform(poly(X, 3), coef(fit)) Indicator Variables S-Plus can automatically generate indicator variables using the I function. Entering the term I(age > 70) into your model is a way of looking at the effect of “young” versus “old” rather than the effect of age on a continuous scale. Factors Factors are related to the issue of indicator variables. You may often have variables such as “political affiliation” or “treatment” that can assume one of several values. Giving numeric codes to these types of variables and entering them into your model as continuous variables makes little sense. To see the problem here, consider the following example. Suppose a questionnaire had been done collecting information on political affiliation,
8.4. Regression
261
where 1=Democratic, 2=Republican, 3=Independent, and 4=Other. If we entered the continuous variable “political affiliation” (values 1 to 4), we would be making assumptions such as Democratic is less than Republican, the difference between Democratic and Republican is the same as the difference between Republican and Independent, and Other is the highest. These assumptions, however, have no meaning and are of little interest. By defining these categorical variables as factors, it is possible to view the effect of each level of the variable separately or combined in certain ways using contrasts. Factor is a special data type in S-Plus and is defined with the factor function. The data set we have used so far doesn’t contain any categorical variables, but we can easily create one using the cut function as follows: > educ.lev educ.lev High Average Low Average ... The optional breaks specifies at which values of education to define the breaks for the levels, and labels defines the names for the new categories. We have created a new variable with three levels of education: low is from 0 to 5, average is from 5 to 10, and high is from 10 to 100 (i.e., over 10). The function is.factor can be used to determine if your variable is really a factor. As there is an inherent ordering to the levels of education, it makes more sense to think of the levels as being ordered. To define this data type, we use the function ordered. > educ.ord educ.ord 3 2 1 2 3 ... Notice that the values contained in educ.ord are numeric, with 1 representing low education, 2 average education, and 3 high education. Try fitting a regression model with educ.lev, and compare the results to the model with Education. What do you notice? Remember that Education is a continuous variable but that educ.lev is a categorical variable with three levels. Now, try fitting a model with educ.ord. How does this compare to the other two?
8.4.2
ANOVA
The analysis of variance (ANOVA) is a statistical method of analyzing data that assumes a continuous outcome variable and one or more classification variables. You might wonder how this differs from using a linear regression model with a factor variable. It doesn’t. Theoretically speaking, an
262
8. Statistical Modeling
ANOVA model is a regression model. The difference comes more with the presentation of the results, as we see in the following example. The aov function is used in exactly the same fashion as the lm function to fit the desired model. > swiss fit.ord summary(fit.ord) Df Sum of Sq Mean Sq F Value Pr(F) educ.ord 2 1350.891 675.4456 5.100272 0.01018307 Residuals 44 5827.064 132.4333 We see from the large F -value and very low p-value that education is related to fertility, but we already knew this from the regression model. Is there any information we can obtain from the ANOVA model that we haven’t already seen with the regression model? Of course! With a factor variable, you can split the degrees of freedom into contrasts of interest. Take the educ.ord variable from above, which has three levels and hence two degrees of freedom. We could use one degree of freedom for a linear test and one for a quadratic test. This breakdown of the degrees of freedom leads to results that are comparable to using polynomial regression, as mentioned in Section 8.4.1. The difference is that with contrasts you aren’t restricted to a polynomial transformation, but we show it here for simplicity. To obtain a polynomial breakdown of the degrees of freedom, use the split option together with lists: > summary(fit.ord, split=list(educ.ord=list(L=1, Df Sum of Sq Mean Sq F Value educ.ord 2 1350.891 675.446 5.100272 educ.ord: L 1 1173.055 1173.055 8.857705 educ.ord: Q 1 177.836 177.836 1.342838 Residuals 44 5827.064 132.433
Q=2))) Pr(F) 0.0101831 0.0047286 0.2527848
The linear and quadratic effects of education on fertility are displayed and tested, showing that it is actually the linear component of the relationship between education and fertility that is statistically significant. The main objective with many ANOVA models is merely to test for certain effects. Useful additional information includes the relative effect of each level of the term of interest. These are obtained using the dummy.coef function:
8.4. Regression
263
> dummy.coef(fit.ord) $"(Intercept)": (Intercept) 70.70725 $educ.ord: 4.932745
2.662745 -7.595490
The coefficients for educ.ord show the relative effect on fertility of the education levels of low, average, and high, respectively. Note that fertility drops as education increases. The choice between the lm and aov functions is often a matter of preference. If you are used to using one of the two, you naturally continue to do so.
8.4.3
Logistic Regression
So far, we have examined models for fitting data with a continuous outcome. Often though, the outcome is binary in nature, representing success/failure, 0/1, or presence/absence. In such cases, a logistic regression model is used to fit the data. Such models are fit in S-Plus using the glm (or gam) function. This function is a very general procedure, as it can be used to fit models in the exponential family, including the normal, binomial, Poisson, and gamma distributions. To explore this type of regression, we will use the built-in data frame, kyphosis, which contains a suitable variable with a binary outcome. The data frame contains the four variables described in Table 8.4. Table 8.4. Kyphosis data frame.
Variable
Description
Kyphosis Age Number Start
Presence/absence of the postoperative spinal deformity Age of child in months Number of vertebrae involved in surgery Beginning of the range of vertebrae involved
We want to see what relationship, if any, exists between the age, number, and start, and the subsequent presence of kyphosis. For example, are younger children more susceptible to kyphosis or more resilient? We use the glm function to fit our logistic model. All we need to specify is the model equation, the exponential family type, and the data frame as follows: > fit.kyph summary(fit.kyph) Call: glm(formula = Kyphosis ∼ Age + Number + Start, family = binomial, data = kyphosis) Deviance Residuals: Min 1Q Median 3Q Max -2.312363 -0.5484308 -0.3631876 -0.1658653 2.16133 Coefficients: Value Std. Error t value (Intercept) -2.03693225 1.44918287 -1.405573 Age 0.01093048 0.00644419 1.696175 Number 0.41060098 0.22478659 1.826626 Start -0.20651000 0.06768504 -3.051043 (Dispersion Parameter for Binomial family taken to be 1) Null Deviance: 83.23447 on 80 degrees of freedom Residual Deviance: 61.37993 on 77 degrees of freedom Number of Fisher Scoring Iterations: 5 Correlation of Coefficients: (Intercept) Age Age -0.4633715 Number -0.8480574 0.2321004 Start -0.3784028 -0.2849547
Number
0.1107516
The function call is repeated at the top, followed by some summary information on the residuals, the model coefficients, general fitting information, and the correlations between the model coefficients. The t-statistics given are partial t-tests, meaning that all other factors in the model have been adjusted for. Notice that p-values are not provided with the t-statistics, but this is an easy calculation to do in S-Plus. You learned how to do it in the previous chapter. The t-statistics indicate that age and the number of vertebrae do not seem to be related to kyphosis but that the starting vertebra of the surgery is related to it. We know that the degrees of freedom to be used for the ttests are the same as for the residual deviance (77). In this case, the number of degrees of freedom is large, so the cutoff for the t-distribution is similar to that of the normal (i.e., ± 1.96 for the usual significance level of 0.05).
8.4. Regression
265
Only the t-statistic for the starting vertebra of the surgery has a value that is more extreme than −1.96, implying that it is associated with kyphosis. To be a little more precise and a lot more helpful, we have calculated both the cutoff level of this distribution and the p-value. > qt(.975, 77) 1.991254 > qt(.025, 77) -1.991254 > 2*pt(-3.05, 77) 0.003137395 > 2*(1-pt(abs(-3.05), 77)) 0.003137395 Using an overall level of significance of 0.05 (each tail gets 0.025), we have calculated both the upper and lower cutoffs for the t-distribution with 77 degrees of freedom and the p-value using two slightly different methods. Note that this type of test is a form of Wald test and is asymptotically t-distributed (Venables and Ripley, 2002). The modeling techniques covered in the previous subsections can also be applied to logistic models. You may wish to try some of them now, or simply wait until the exercises.
8.4.4
Survival Data Analysis
What do we mean by “survival data”? Quite loosely, we refer to data that record the time until the occurrence of an event. In cancer clinical studies, for example, you measure the time (in days, months, etc.) until the patient’s death. In this case, the event is death. The problem is that, quite often, the patients either withdraw from the trial or are still alive at the end of it (rather good from the patients’ point of view). The problem raised is that the actual time of the event is not observed for these patients, so their survival time is “censored.” Special statistical techniques have been devised to analyze this type of data, including the famous Kaplan–Meier curves. Recent statistical research in this area has led to the topics of martingales and martingale residuals, and S-Plus is one of the few software packages to include this functionality. This book is not, however, a statistics book, so we will stick to the very basics, and it will be up to you to carry on with martingales should you so desire. With survival data analysis, the outcome “variable” consists of two pieces, the time to event and an indicator of whether the time refers to an event or is a censored time. It is then typical to have some sort of group variable such as a treatment indicator. One such data set included with S-Plus is the Leukemia data set, which contains three variables as described in Table 8.5. The data come from a trial to determine the efficacy of maintenance chemotherapy for acute myelogenous leukemia.
266
8. Statistical Modeling Table 8.5. Leukemia data frame.
Variable
Description
time status group
Time to remission after chemotherapy (in weeks) Indicator of status: 1=remission time; 0=censored time Treatment group: maintained or nonmaintained
The survfit function can be used to fit a Kaplan–Meier survival curve or other types of survival curves according to the options chosen. You must use the Surv option to define the composite outcome variable followed by the explanatory variables and the name of the data frame. > leuk.fit summary(leuk.fit) Call: survfit(formula = Surv(time, status) ∼ group, data = leukemia) group=Maintained lower upper time n.risk n.event survival std.err 95% CI 95% CI 9 11 1 0.909 0.0867 0.7541 1.000 13 10 1 0.818 0.1163 0.6192 1.000 18 8 1 0.716 0.1397 0.4884 1.000 23 7 1 0.614 0.1526 0.3769 0.999 31 5 1 0.491 0.1642 0.2549 0.946 34 4 1 0.368 0.1627 0.1549 0.875 48 2 1 0.184 0.1535 0.0359 0.944 group=Nonmaintained lower upper time n.risk n.event survival std.err 95% CI 95% CI 5 12 2 0.8333 0.1076 0.6470 1.000 8 10 2 0.6667 0.1361 0.4468 0.995 12 8 1 0.5833 0.1423 0.3616 0.941 23 6 1 0.4861 0.1481 0.2675 0.883 27 5 1 0.3889 0.1470 0.1854 0.816 30 4 1 0.2917 0.1387 0.1148 0.741 33 3 1 0.1944 0.1219 0.0569 0.664 43 2 1 0.0972 0.0919 0.0153 0.620 45 1 1 0.0000 NA NA NA The summary function applied to this class of data provides a radically different output than those from the previous regression models. Here, you get the estimated survival probability at many time points along with
8.4. Regression
267
standard deviations and confidence intervals for each of the two treatment groups. The estimated survival for the nonmaintained group at 30 weeks is 0.29, whereas it is 0.49 at 31 weeks for the maintained group. Printing the fit of the survival analysis directly yields certain summary statistics for each of the two groups, such as the estimated mean and median survival times. > leuk.fit Call: survfit(formula = Surv(time, status) ∼ group, data = leukemia) 0.95 0.95 n events mean se(mean) median LCL UCL group= 11 7 52.6 19.83 31 18 NA Maintained group= 12 11 22.7 4.18 23 8 NA Nonmaintained With this output, we see that the median survival time for the maintained group is 31 weeks but is only 23 weeks for the nonmaintained group. The next obvious question is whether or not the difference in survival between these two groups is statistically significant. Using the survdiff function, we can conduct a statistical test for the difference between two survival distributions: > survdiff(Surv(time, status)∼group, leukemia) Group N Observed Expected (O-E)ˆ2/E (O-E)ˆ2/V Maintained 11 7 10.69 1.27 3.4 Nonmaintained 12 11 7.31 1.86 3.4 Chisq= 3.4
on 1 degrees of freedom, p= 0.0653
This is only the beginning of S-Plus’ survival analysis capabilities. Use the help system to find out more options to be used with these models.
8.4.5
Endnote
The aim of this book is not to teach statistics to the reader, although much is explained in the text. The focus is primarily to show how to understand and use S-Plus, independent of one’s knowledge of statistics or computing. If your interest in using S-Plus to perform statistics has been sparked and you want to learn more, we refer you to other sources. In particular, you may want to look at Venables and Ripley (2002), Pinheiro and Bates (2000), Therneau and Grambsch (2000), or Harrell (2001).
268
8. Statistical Modeling
8.5 Exercises Exercise 8.1 In the regression model presented in this chapter, we assumed that education was a continuous variable. Often, however, we encounter variables that have discrete levels and must perform a regression model with them. Create four education groupings of roughly equal size and an indicator variable for each grouping. The first indicator variable should be equal to 1 for each observation in the first grouping and 0 for all other observations. The other indicator variables are similarly defined. Perform a multiple regression with these new indicator variables in place of the continuous version of education. What is the mean education level per grouping? Does the mean fertility differ between the groupings? What additional conclusions can you make from this analysis over the one presented in the chapter? (Hint: Use the quantile, cut, and cbind functions.)
Exercise 8.2 This exercise is designed to give you some practice with logistic regression, model building, and stepwise regression. Using the Kyphosis data set as in the section on logistic regression in this chapter, perform the following: • Fit a model to the kyphosis data containing only the variable Start as a predictor. • Use update to add a term for Age. • Use anova to compare the fits of the two models. Which is better? • If the model with both terms is better, use update to add the term Number; if not, use update to add Number directly to the model with Start. Compare these models. What do you conclude? You have now constructed a model using one approach. Compare this model with that obtained using stepwise regression. Are the two models the same? Which model do you think is the “best?” (Hint: Fit a model with all terms because the stepwise regression procedure (step) removes terms.) Plot the fit of the best-fitting model. What can be said from the diagnostic plots? Does the model fit well? Try using the plot.gam function, and then answer the same question.
8.5. Exercises
269
Exercise 8.3 In this exercise, we will lead you through the steps necessary to analyze the drug.mult data set. You will first need to familiarize yourself with the data set by reading its description in the help system. You will note that it has two factors: gender and time. The time factor is “hidden” in the layout of the data in that there is a response vector at each time point. An ANOVA model is the typical procedure to use for a continuous outcome variable and factor variables as the predictor variables. We have listed the tasks you will need to perform: • Rearrange the data into a new data frame such that there are variables for the response, sex, time, and subject. (Define sex and subject as class factor and time as class ordered.) • Perform an ANOVA model with response as the outcome and sex as the predictor. What do the results tell you about the effect of sex? • Perform an ANOVA model with response as the outcome and time as the predictor. What do the results tell you about the effect of time? • Perform an ANOVA model with response as the outcome and both sex and time as predictors. What do the results tell you about the effects of sex and time when both are in the model (simultaneous adjustment)? • Perform an ANOVA model with response as the outcome and predictors of sex, time, and the interaction of sex and time. What do the results tell you about the interaction of sex and time? The analysis to this point is fine as a first step, but it is not technically correct because it has assumed that the response is independent from one time point to the next. As the same subjects are used at each of the time points, their responses are surely correlated. Continue the analysis with some more advanced techniques. • Confirm that the observations are correlated across the time points. (Hint: The cor and var functions work on matrices.) • Perform a “split-plot” ANOVA. This type of model uses the aov function in conjunction with the Error function. What do the results tell you about the effects of sex and time? • Perform a multivariate analysis of variance (MANOVA) on the data. What do the results tell you about the effect of sex? (Hint: Use the manova function on the original data set.) Summarize the results using the univariate=T option. Does this shed any further light on the results, or does it confuse things further? What do all the results actually tell us about the effects of sex and time? Do all the results agree? What have we forgotten to do in this analysis?
270
8. Statistical Modeling
We have here a lot of results from fancy statistical techniques and, worst of all, they do not all agree with one another. What we have forgotten is the simple data summaries and graphics to show us what information is really in the data. Most statistical models rely on assumptions that may or may not be met. Simple summary statistics and graphical descriptions of the data cannot be overemphasized. Furthermore, they should be the first step of a data analysis, not something done at the end when the results from models are confusing. Nonetheless, the analysis for this exercise will conclude with a few graphical methods that are of tremendous help in trying to interpret effects in ANOVA models. • Perform summary statistics on the variables (not shown in solution). • Explore the main effects of the sex, time, and subject factors by using the graphical function plot.design. • Explore the interaction of sex and time by using the graphical function interaction.plot. Can you now make more informed conclusions about the effects of time and sex? Can you see why some people claim that statistics is an art rather than a science? For statisticians reading the book, the analysis is still not complete (although the solution is). The MANOVA test is probably too conservative to use for the final conclusion. The split-plot results rely on the assumption that the variance–covariance matrix is spherical. Looking at the variance– covariance matrix (and/or correlation matrix), it seems likely that it is not spherical. You could make a Huynh–Feldt adjustment to the degrees of freedom in the split-plot test and recompute the p-value, but this is only an approximation to departures from nonsphericity. The best option is to perform a longitudinal analysis of the data. With this modeling approach, it is possible to choose a model that is between the generality of the MANOVA model and the simplicity of the simple ANOVA (or split-plot) model. Longitudinal models can be fit with the lme function, lme standing for linear mixed effects. The statistics involved with longitudinal techniques are far beyond the scope of this book, but those with the desire may read the documentation (or Pinheiro and Bates, 2000) for the lme function and carry this analysis one step further.
8.6. Solutions
271
8.6 Solutions Solution to Exercise 8.1 To make our lives easier, we first extract the vectors of interest so that we don’t have to deal with matrices. > education fertility quants.educ quants.educ 0% 25% 50% 75% 100% 1 6 8 12 53 We will use the cut function to use the cutpoints defined by the quantiles to create a variable that contains a 1 for the first group, a 2 for the second group, and so on. Note that we have also used the include.lowest option to include the minimum value. Groups are normally defined starting with any value greater than the lower cutpoint and up to and including the upper cutpoint. This method’s flaw is that the minimum value gets left out. > cuts.educ tapply(education, cuts.educ, length) 1+ thru 6 6+ thru 8 8+ thru 12 14 11 11 > tapply(education, cuts.educ, mean) 1+ thru 6 6+ thru 8 8+ thru 12 3.785714 7.363636 10.72727
12+ thru 53 11 12+ thru 53 24
To make our indicator variables, we first make four vectors and fill them with zeros. We then put ones into the vectors in the appropriate elements.
272
8. Statistical Modeling
> educ1 |t|) 0.0000 0.0041 0.0016 0.1446
Residual standard error: 11.2 on 43 degrees of freedom Multiple R-Squared: 0.2479 $F$-statistic: 4.725 on 3 and 43 degrees of freedom, the $p$-value is 0.006164
8.6. Solutions
273
Correlation of Coefficients:
newxeduc1 newxeduc2 newxeduc3
(Intercept) -0.7483 -0.7071 -0.7071
newxeduc1
newxeduc2
0.5292 0.5292
0.5000
There is no indicator for education level 4, but we can estimate the fertility rate for that group based on the model above. The estimated coefficients above are for the intercept and the remaining three education levels. Since we left education level 4 out of the model, we have set its effect, not its value, to zero. To get the estimated fertility levels, take the coefficient from the intercept and add the education level coefficients for estimates of 60.6+13.7=74.3, 60.6+16.1=76.7, 60.6+7.1=67.7, and 60.6+0.0=60.6, respectively. The overall test of whether or not education is related to fertility rate is given by the F -test with 3 and 43 degrees of freedom. Note that the p-value for this test is well below the standard cutoff of 0.05, indicating that there is a significant relationship between the two measures. By construction, the mean education level increases in the four groupings, but the fertility decreases. Notice that there is little change in mean fertility level between the first two groups. This is reflected in the fact that the coefficients are nearly the same for the variables newxeduc1 and newxeduc2. This indicates that the true relationship between education and fertility is not exactly linear but is well-approximated by a linear model.
Solution to Exercise 8.2 We begin the solution by fitting a model with only the predictor Start and the outcome Kyphosis. > fit.a summary(fit.a) Call: glm(formula = Kyphosis ∼ Start, family = binomial, data = kyphosis) Deviance Residuals: Min 1Q Median 3Q Max -1.472875 -0.5176166 -0.4211055 -0.3412781 2.130472 Coefficients: Value Std. Error t value (Intercept) 0.8900613 0.62965699 1.413565 Start -0.2178862 0.06035673 -3.609973
274
8. Statistical Modeling
(Dispersion Parameter for Binomial family taken to be 1 ) Null Deviance: 83.23447 on 80 degrees of freedom Residual Deviance: 68.07218 on 79 degrees of freedom Number of Fisher Scoring Iterations: 4 Correlation of Coefficients: (Intercept) Start -0.8739978 With a “large” t-value of -3.61, there is clear evidence that the variable Start is related to the outcome of Kyphosis. Hence, it seems reasonable to use this model as a base and investigate the relevance of the other variables available to us. To add the variable Number, it is advantageous to use the update function (as opposed to add1) so that we can get nice summaries of our new model. The syntax is simple enough: we use .∼. to represent the previous model and can then add or remove terms from it. > fit.b summary(fit.b) Call: glm(formula = Kyphosis ∼ Start + Age, family = binomial, data = kyphosis) Deviance Residuals: Min 1Q Median 3Q Max -1.562354 -0.5976468 -0.3923343 -0.2353979 2.199761 Coefficients: Value Std. Error t value (Intercept) 0.225043496 0.74917916 0.3003868 Start -0.237923044 0.06422148 -3.7047271 Age 0.009507095 0.00587362 1.6186091 (Dispersion Parameter for Binomial family taken to be 1 ) Null Deviance: 83.23447 on 80 degrees of freedom Residual Deviance: 65.29907 on 78 degrees of freedom Number of Fisher Scoring Iterations: 4
8.6. Solutions
275
Correlation of Coefficients: (Intercept) Start Start -0.5614099 Age -0.5165854 -0.2967662 We have now fit a model with terms for both starting vertebra and age and can see that the former is associated with the outcome of kyphosis whereas the latter isn’t (t-value is less than 2). Sometimes, a term can appear to have added little to a model, but because of interdependence of the predictor terms, it is safer to perform a more structured test using the anova function. Note that we are not going to fit an ANOVA model (using the aov function) but merely compare the fit from two models. > anova(fit.b, test="Chi") Analysis of Deviance Table Binomial model Response: Kyphosis Terms added sequentially Df Deviance Resid. NULL Start 1 15.16229 Age 1 2.77311
(first to last) Df Resid. Dev Pr(Chi) 80 83.23447 79 68.07218 0.00009865 78 65.29907 0.09585957
The output from the anova function shows us that Start is highly significant (significantly reduced the residual deviance from a null model) but that the addition of Age is not. The addition of Age has resulted in a decrease in residual deviance of only 2.8 for a loss of 1 degree of freedom. Hence, at the typical cutoff of 0.05, Age does not significantly add to the overall model. We choose not to keep Age in the final model and next consider Number in the same fashion. > fit.c summary(fit.c) Call: glm(formula = Kyphosis ∼ Start + Number, family = binomial, data = kyphosis) Deviance Residuals: Min 1Q Median 3Q Max -1.903483 -0.5938547 -0.3885682 -0.2490264 2.214024 Coefficients: Value Std. Error t value (Intercept) -1.0288475 1.22345046 -0.8409392 Start -0.1849421 0.06397197 -2.8909871 Number 0.3574336 0.19963697 1.7904177
276
8. Statistical Modeling
(Dispersion Parameter for Binomial family taken to be 1 ) Null Deviance: 83.23447 on 80 degrees of freedom Residual Deviance: 64.53647 on 78 degrees of freedom Number of Fisher Scoring Iterations: 4 Correlation of Coefficients: (Intercept) Start Start -0.6285140 Number -0.8487089 0.2077516 The term Number is not statistically significant but note that the t-value of Start has decreased compared to its previous values, indicating that the two predictors are not completely independent of one another. This is all the more reason to perform the chi-square test from the anova function. > anova(fit.c, test="Chi") Analysis of Deviance Table Binomial model Response: Kyphosis Terms added sequentially (first to last) Df Deviance Resid. Df Resid. Dev Pr(Chi) NULL 80 83.23447 Start 1 15.16229 79 68.07218 0.00009865 Number 1 3.53571 78 64.53647 0.06006055 We see here that the residual deviance has been reduced by 3.5 with the addition of the term Number, which is nearly statistically significant (p=0.06). Since the p-value is so close to 0.05, we choose to keep the term Number in the final model, although this is perhaps not fully justified. The default for the stepwise regression function in S-Plus (step) is to start with a full model and drop terms that are deemed to be unnecessary. Therefore, we simply need to feed the full model to step and see what it does. > fit.all fit.all Start: AIC= 69.3799 Kyphosis ∼ Age + Number + Start
8.6. Solutions
277
Single term deletions Model: Kyphosis ∼ Age + Number + Start scale:
1
Df Sum of Sq RSS Cp <none> 70.26871 78.26871 Age 1 2.877010 73.14572 79.14572 Number 1 3.336562 73.60527 79.60527 Start 1 9.308866 79.57757 85.57757 Call: glm(formula = Kyphosis ∼ Age + Number + Start, family = binomial, data = kyphosis) Coefficients: (Intercept) Age Number Start -2.036932 0.01093048 0.410601 -0.20651 Degrees of Freedom: 81 Total; 77 Residual Residual Deviance: 61.37993 The stepwise deletion procedure hasn’t removed any of the terms. However, we just saw that a stepwise addition procedure (done manually) doesn’t arrive at the full model. This occurs frequently because of the criteria set for the automatic stepwise procedure. If they had been fine-tuned, we might have reached the same final model. In any case, we feel more confident in the model that we built manually (fit.c). We should now look at some diagnostic plots to see if the overall model fits the data well. A plot of fit.c is not very helpful in answering our question (see Figure 8.2). Because the response variable kyphosis is dichotomous and takes only the values 0 and 1, the standard diagnostic plots that work so well with Gaussian distributed data don’t convey much information. In this setting, it is much more useful to use one of the special plotting functions, plot.glm and plot.gam, which have been specially designed to plot more appropriate diagnostic graphs for these types of models. We have chosen to use plot.gam using the following code. > par(mfrow=c(1, 2)) > plot.gam(fit.c, resid=T, scale=6) The option resid specifies that partial residuals are to be plotted, and the scale option specifies the number of units that are to be plotted on the y-axis (important when trying to keep the same axis for all graphs). The
••••••• •• •• • •
Kyphosis 0.0 0.2 0.4 0.6 0.8 1.0
0.0
• • •
• •• •
• •• • •
•
• •
• • •
0.2 0.4 0.6 Fitted : Start + Number
•• • • • • • • • • • •
• 0.8
• •
•••••• •• ••••••• • • • •• • • • • • • 0.0 0.2 0.4 0.6 0.8 Fitted : Start + Number
Pearson Residuals −2 −1 0 1 2 3
•• • • •
sqrt(abs(Deviance Residuals)) 0.6 0.8 1.0 1.2 1.4
8. Statistical Modeling
Deviance Residuals −2 −1 0 1 2
278
1.0
••
••
•
• •
• •• •••
• •• •
•
• ••
• ••
•
•
••
• •• ••
• •
•
−3 −2 −1 0 1 Predicted : Start + Number
• • ••• • • • •• ••• • • • • • • • • • • • • • • • • • • • • • • • • • • • ••••••••••••••••• ••••• • • ••
••
• •
• −2 −1 0 1 2 Quantiles of Standard Normal
Figure 8.2. Typical regression diagnostic plots
resulting graph is shown in Figure 8.3. The partial residuals show that the linear relationship used with both predictor variables included in the model is approximately correct. Another question that could be asked at this point is how well the model could be used to predict outcomes. With this model, would we have predicted any false positives? False negatives? These questions, and lots of related ones, are left to the reader to explore.
Solution to Exercise 8.3 The drug.mult data set is in matrix format with one column for each time point and one row for each subject. There is also a column indicating the gender of each patient. With this type of a setup, the response is in several columns, one for each level of one of the factors of interest. For ANOVA models, S-Plus requires that all the responses be together in one column, so we need to transform the data to a more suitable arrangement. We create the factor variables using the rep function and then concatenate the responses using the c function. > subj sex time response newdrug nd.aov.sex summary(nd.aov.sex) Df Sum of Sq Mean Sq F Value Pr(F) sex 1 60.8017 60.80167 12.43195 0.001902097 Residuals 22 107.5967 4.89076 We see from this simple model that the factor sex is related to the response. In simpler terms, it means that the estimated mean response for males differs from that for females. This simple model has ignored time altogether.
280
8. Statistical Modeling
Our ANOVA model with time is run in the same fashion as the model for sex. > nd.aov.time summary(nd.aov.time) Df Sum of Sq Mean Sq F Value Pr(F) time 3 49.1083 16.36944 2.744479 0.07002713 Residuals 20 119.2900 5.96450 We see from this model that the variable time is not significantly related to the response at the 0.05 level (the default level of the test). The two models above have looked at the two predictor variables in isolation. If they are independent of one another, a joint model will come to the same conclusions. In practice, however, a joint model will yield slightly (or sometimes substantially) different results than the separate models. We show the results of the joint model as follows: > nd.aov.both summary(nd.aov.both) Df Sum of Sq Mean Sq F Value Pr(F) sex 1 60.80167 60.80167 19.75149 0.000278263 time 3 49.10833 16.36944 5.31763 0.007861779 Residuals 19 58.48833 3.07833 Notice that the variable time is highly significant in the joint model. This tells us that the mean response differs between males and females, and also between the time points. The next question to consider is the following. Is there a significant interaction between sex and time? One example of an interaction would be to have a large mean for males and a small mean for females at the early time points but the reverse at the late time points. In the model below, we add the interaction term. > nd.aov.inter summary(nd.aov.inter) Df Sum of Sq Mean Sq F Value Pr(F) sex 1 60.80167 60.80167 22.26827 0.0002317 time 3 49.10833 16.36944 5.99522 0.0061389 sex:time 3 14.80167 4.93389 1.80701 0.1864698 Residuals 16 43.68667 2.73042 We are now reasonably satisfied that the interaction between time and sex is not significant and stick to the conclusion that there are main effects of both variables. There is an obvious problem with the analyses we have performed up to this point that all boils down to correlation. Take any particular subject. Is his/her response at time 1 independent from his/her response at time 2? Although it is theoretically possible that these responses are independent, it
8.6. Solutions
281
is probable that they are correlated. The ANOVA model, however, assumes that all responses are independent from one another. The design of the study is somewhat typical and can be run in the ANOVA setting under certain other conditions. If we introduce a random effect for the subjects, we can make an adjustment for the correlation that exists. The type of model we run in this case is referred to as a split-plot ANOVA model and is fit in S-Plus with the simple addition of one extra term: > nd.split.plot summary(nd.split.plot) Error: subj Df Sum of Sq Mean Sq F Value Pr(F) sex 1 60.80167 60.80167 19.32256 0.01173 Residuals 4 12.58667 3.14667 Error: Within Df Sum of Sq Mean Sq F Value Pr(F) time 3 49.10833 16.36944 6.316184 0.0081378 sex:time 3 14.80167 4.93389 1.903751 0.1828514 Residuals 12 31.10000 2.59167 The output is slightly different in that it is split into two parts. In the “Error: subj” section, the “Residuals” term refers to the subjects nested within sex. This term has 4 degrees of freedom, n−1 for the females (=2) and n−1 for the males (=2). In the “Error: Within” section, the “Residuals” term stands for the interaction between sex, time, and subject nested within sex. These are perhaps statistical details that can be ignored. The main point is that we have new tests for sex, time, and their interaction. Notice that the interaction is still nonsignificant and that sex and time are both still significant. Notice that the p-value for time has hardly changed at all, whereas the p-value for sex, although still significant, has increased due to the correction for correlation. The split-plot ANOVA model assumes that the variance–covariance matrix across the time points is spherical. In practice, the definition of sphericity is met with the compound symmetry variance structure that assumes a common variance along the diagonal and a different but constant covariance in the off-diagonals. This seems to be a fairly restrictive assumption but one that fits the data quite well in many practical situations. Without going into a test to see whether or not the sphericity assumption of the split-plot model holds, it is also possible to run a multivariate analysis of variance (MANOVA) on the data. This type of model is designed for responses that are correlated, be it different response variables (pulse and blood pressure) or, as in our case, responses at different time
282
8. Statistical Modeling
points. One drawback is that we will lose the ability to test for an effect of time. The MANOVA assumes a general covariance structure such that the assumptions are always met (still must have Gaussian distributed data). This generality comes with a price called reduced power. Since it can test against any alternative hypothesis, it does not necessarily have power to test the particular hypothesis we have. This means that the MANOVA test might say that the effect of gender is not significant when it really is. If, however, this test says that gender is significant related to the response, then we can be sure of our result. > nd.manova summary(nd.manova) Df Pillai approx. F num den P-value Trace df df gender 1 0.928614 3.25209 4 1 0.391236 Residuals 4 > summary(nd.manova, univariate=T) Response: Y.1 Df Sum of Sq Mean Sq F Value Pr(F) gender 1 7.260000 7.260000 2.93531 0.161819 Residuals 4 9.893333 2.473333 Response: Y.2 Df Sum of Sq Mean Sq F Value Pr(F) gender 1 13.20167 13.20167 3.345017 0.1414016 Residuals 4 15.78667 3.94667 Response: Y.3 Df Sum of Sq Mean Sq F Value Pr(F) gender 1 4.681667 4.681667 35.55696 0.003971604 Residuals 4 0.526667 0.131667 Response: Y.4 Df Sum of Sq Mean Sq F Value Pr(F) gender 1 50.46 50.46 11.54691 0.02732576 Residuals 4 17.48 4.37 The p-value for gender shows that it is not significantly related to the response (p=0.39). The univariate ANOVAs show that the gender effect is significant at the last two time points. What do we do now? Which of the analyses do we believe? This is how many statisticians proceed with data analysis. The point is that much of the information contained in the data can be seen through the use of summary statistics and simple graphs. Furthermore, these are
8.6. Solutions
283
precisely the strengths of S-Plus and should have been the starting point of our analysis. So, we will now start at the beginning. You have been asked to perform the summary statistics on your own. There are two types of graphs that are of particular interest to ANOVA type models, namely plot.design and interaction.plot. We put the syntax below and show the resulting graphs in Figure 8.4. mean of response 78 79 80 81 82
S6 M
S4 S5
3 2
0 S3 S2
F 1
S1 sex Factors
time
84
subj
sex
76
mean of response 78 80 82
M F
0
1
2
3
time
Figure 8.4. Design and interaction plots for ANOVA.
> par(mfrow=c(2, 1)) > plot.design(newdrug) > interaction.plot(time, sex, response, data=newdrug) By default, these two functions plot the mean of the response at the different levels of the factors. As an option, you can also pass a different function to be used, such as the median. From the graphs, we quickly see that the mean response is higher for males than for females and that the mean response is higher at the later time points than at the earlier ones. In the interaction plot, the mean response is plotted for each gender across the time points. Note that, with the possible exception of the last time point, the curves are parallel, implying that there is no interaction between sex and time.
284
8. Statistical Modeling
One quick check of the sphericity assumption of the split-plot model is to look at the correlation and variance matrices of the data. > attach(drug.mult) > cor(cbind(Y.1, Y.2, Y.3, Y.4)) Y.1 Y.2 Y.3 Y.4 Y.1 1.0000000 0.7684940 0.7084921 0.3772931 Y.2 0.7684940 1.0000000 0.7282511 0.5883456 Y.3 0.7084921 0.7282511 1.0000000 0.8165430 Y.4 0.3772931 0.5883456 0.8165430 1.0000000 > var(cbind(Y.1, Y.2, Y.3, Y.4)) Y.1 Y.2 Y.3 Y.4 Y.1 3.430667 3.427333 1.339333 2.576 Y.2 3.427333 5.797667 1.789667 5.222 Y.3 1.339333 1.789667 1.041667 3.072 Y.4 2.576000 5.222000 3.072000 13.588 > detach() The variance changes from 3.4 to 5.8 to 1.0 to 13.6. This alone is evidence that a spherical variance structure is probably not sufficient to model our data. That leaves us with the alternative of the MANOVA model and the conclusion that the mean response does not differ between males and females. That is a conservative conclusion, and many more analyses could be done to “complete” this analysis. In particular, a longitudinal model could be fit using the lme function that fits a variance structure that is between the spherical assumption of the split-plot model and the general structure of the MANOVA model. There are also ways of arranging the data to get a MANOVA type of test of time (take differences from baseline for each patient and then test whether the mean vector containing only three responses differs from the zero vector). We started this exercise convinced that there is an effect of gender on the response (i.e., that the mean response is higher for males than for females). After some investigation involving summary statistics, graphical methods, statistical methods, and checking of assumptions, it is rather likely that there is no real difference in the mean responses between the sexes.
9 Programming
”Writing program code is a good way of debugging your thinking.” Bill Venables on S-News February 1997
Now that you have learned the elementary commands in S-Plus and many ways of applying them, it is time to discover its advanced programming functionalities. This chapter introduces list structures, loops, and writing functions, covers debugging matters, and gives some more details on how to use S-Plus as a programming language.
9.1 Lists A list is typically a very flexible structure in which to store data. Its main usage is in situations where some data belong together by their contents but the structure is very heterogeneous. Imagine a database of somebody’s floppy disks, each disk with a number, a short text description of the contents, and a list of files it contains. It would not be a good idea to store this kind of data in a matrix or data frame because the number of files is different for each disk. A single list can contain different element types; for example, a vector, a matrix, and another list, all at the same time. A list is constructed by calling the function list and enumeration of the objects to go into it. Here is an example for creating a list with three elements, each of them being a vector. > L > > >
x y z L
length(L) 3 You can add a new element by assigning the new element to an unused index: > L[[4]] L[[length(L)+1]] L[[2]] > > >
L L$comment names(L) # the names of the elements "number" "bool" "message" "comment" > names(L)[2] names(L) # the new names "number" "true and false" "message" "comment" To access the list element with the name “true and false,” put the name in double quotes (because of the spaces in its name). > L$"true and false" Note Using named list elements is a good idea. In this way, the order and position of the list elements are completely irrelevant and can always be changed. Note Named list elements can be addressed in several ways. The following commands all do the same thing: extracting the element “number” from the list L. > L$number > L$"number" > L[["number"]] Single brackets can be used as well, but the returned result differs slightly. The expressions above return a vector, whereas the expressions below return a list of length 1. > L["number"] > L[names(L)=="number"] Extracting a single list element is therefore typically done by using double brackets or the $ operator. Single brackets are useful if the operation might return more than one list element.
290
9. Programming
9.1.3
Applying the Same Function to List Elements
It is often very convenient to keep objects of the same type in a list, for example, the country data we used earlier. S-Plus provides some useful functions that enable the application of a function to all elements of the list, making the task easy and efficient.
lapply The tool lapply is designed for working on all elements of a list using the same function. We use the country data again and calculate some statistics of the list elements: > countries countries $GDP: 208 1432 2112 259 $Pop: 8 61 82 7 $Area: 84 544 358 41 Let us now figure out what the combined statistics of those countries are. > lapply(countries, sum) $GDP: 4011 $Pop: 158 $Area: 1027 Remember that all standard arithmetic is also a function within S-Plus, so we can also subtract 1 from each element of the list by entering > lapply(countries, "-", 1) You see that all additional parameters passed to lapply are passed unmodified to the function "-". The minus operator "-" is a function like any other. You could enter 5-2 or alternatively "-"(5, 2). In order to see its declaration, use get("-"). The index selection functions ¨[¨ and ¨[[¨ can be used in the same way. We extract the first element of each list component by supplying the parameter 1 to the function ¨[¨.
9.1. Lists
291
> lapply(countries, "[", 1) $GDP: 208 $Pop: 8 $Area: 84 However, to ensure that the class-specific function is used, wrap a function around it: > lapply(countries, function(x) return(x[1])) You might have noticed as well that we have lost the names of the countries as labels. We can add them to each list element as follows: > country.names countries lapply(as.list(1:length(x)), function(i, x, y) { plot(x[[i]], y[[i]]) }, x, y) Depending on the size of the arguments, the above can be much faster than an explicit loop.
292
9. Programming
sapply If we wanted to obtain a vector of sums instead of a list as above, we could enter > unlist(lapply(countries, sum)) GDP Pop Area 4011 158 1027 but we could have used sapply instead. The function sapply simplifies the structure of the results if possible. If the result is a list with elements of the same length, they are composed into a simpler structure. Using sapply to derive the sums of the list elements gives > sapply(countries, sum) GDP Pop Area 4011 158 1027 whereas using sapply to calculate summary statistics of all list elements gives > sapply(countries, summary) GDP Pop Area 208.0 7.00 41.00 246.2 7.75 73.25 845.5 34.50 221.00 1003.0 39.50 256.70 1602.0 66.25 404.50 2112.0 82.00 544.00
rapply If our data structure is a more complicated list, there is another function that might help: rapply descends down through all levels of a list if we have a list of lists, and it lets us define a function to apply only to objects of a certain class. One needs to specify the object to process and the function to apply. Optionally, one can set a restriction to use only objects of certain classes by specifying classes="vector of class names" and how to treat the result (the default is unlisting the list). Here is a simple example. Let us define a list of four components, the fourth being a list again. > x rapply(x, rev, how="replace") $four:
9.1. Lists
293
$four$B: "G" "F" "E" $four$A: 3 2 1 $three: 123 122 121 120 119 118 117 116 115 114 113 112 111 $two: "d" "c" "b" "a" $one: 5 4 3 2 1 You can see that the elements were reversed on all levels. The list element $four is now in position 1, and all its components are reversed as well. Let us calculate the mean of all numeric elements in the list: > rapply(x, mean, "numeric", how="replace") Problem in rapply(x, mean, "numeric", how="replace..: must supply a function of one argument Use traceback() to see the call stack S-Plus informs us that the function to apply must have exactly one argument. The function mean has one argument only (plus an option), such that we have to use a little trick: > f rapply(x, f, "numeric", how="replace") $one: 3 $two: "a" "b" "c" "d" $three: 117 $four: $four$A: 2 $four$B: "E" "F" "G"
294
9. Programming
9.1.4
Unlisting a List
Sometimes, it is necessary to extract the elements of a list to compose them again into a single structure. To do this, the function unlist takes all elements of a list and combines them in the most global data structure. For example, a list with vectors and matrices with numeric contents becomes one long vector containing all the numbers; a list with numeric and character contents becomes a character mode variable.
9.1.5
Generating a List by Using split
The split function splits up one variable according to another. The result is a list with as many elements as there are distinct values in the second variable. For example, > split(barley$yield, barley$site) generates a list of length 6, as there are six sites in the Barley data. Each list component contains the yield for one of the sites. The names of the list elements correspond to the levels of the site variable. The split data are now ready to apply a function to the yield data of each site separately.
9.2 Writing Functions Just like a macro in other languages, a function is simply a sequence of commands put together into a single new command, the function’s name. In contrast to macros, S-Plus functions offer the same flexibility and capability as other modern programming languages. Functions have arguments passed to them, and results are returned to the calling routine. All other variables defined in the function’s body are kept internally and disappear after the function is executed. In S-Plus, a function and its usage are very similar to their usage in mathematics. In mathematics, you write y = f (x), as you do in S-Plus: > y cube cube(3) 27 but you can also supply a vector, > x cube(x/2) 0.125 1.000 3.375 8.000 15.625 call it recursively, > cube(cube(x)) 1 512 19683 262144 1953125 or pass a structure such as a matrix. > x x 1 3 2 4 > cube(x) 1 27 8 64 A function can have more than one argument (or none at all). We will imitate the system’s division function. Yes, the operand "/" in S-Plus is itself a function, which can be accessed and modified exactly like any other function. Enter get("/") to see the declaration. The following function simply divides the first argument supplied by the second: > divide divide(7, 4) 1.75 # two numbers as arguments > a b divide(b, a) # two vectors as arguments 4 3.5 3.14286 2.875 2.6666 2.5 > divide(1:10, 3) # a vector and a number 0.3333 0.6666 1.0000 1.3333 1.6666 2.0000 2.3333 2.6666 3.0000 3.3333 The more general syntax of a function declaration is as follows:
296
9. Programming
function-name f ?f would give a description of our function and its arguments together with the little help text we entered. As it is very easy to document functions in this fashion, you should make it a custom to add a few lines of comments to each function declaration.
9.2.2
Scope of Variables
We will now cover the scope of variables declared within functions in more detail. To begin, note that all variables declared inside the body of a function are local and vanish after the function is executed. Their scope (the environment where the variables are known) does not extend beyond the function’s limits. To demonstrate this effect, we use the following example. Pay attention to the value of z before and after execution of the function f. > > > > >
z f x y z
f(1:10, (1:10)ˆ2, T) f(1:10, (1:10)ˆ2) f(1:10) f(y=(1:10)ˆ2) f(showgraph=T) f()
Moreover, the parameters can use other parameters in the default settings. See the next example and note that y is a function of x in the declaration of y and that error is a function of both x and y. Example 9.2. A function with cross-referencing default parameters The parameters of the function declared use other parameters to set default values. f f2(x=3) to obtain the same result (that x gets assigned the value 3 in the function). Sometimes, as in the boxplot function, it is useful to have the ... in the beginning, followed by optional arguments for special purposes. On
9.2. Writing Functions
301
the other hand, it is not useful to put required arguments behind the ... parameter. The following example shows how ... arguments are treated. Example 9.3. Usage of ... as a function argument The following function shows how to treat an unspecified number of parameters using the ... expression. We compare the distributions of the data passed to the function. To do this, we calculate the minimum and maximum over all data and set the x-axis to the same boundaries for all histograms: f f(sin(pi)) "sin(pi)"
# pass a number # pass a number assigned to x # pass a variable # pass an expression
In this way, you could add a title to a graph containing what was passed as x: title(deparse(substitute(x))) If a function has more than just one parameter, the x in the expression deparse(substitute(x)) can, of course, be replaced by any other parameter name. Another useful tool is the function sys.call(). It returns the complete call to the function in the form of a list. The first element is the name of the function, the second element is the first argument, and so forth.
9.3. Iteration
303
9.3 Iteration An iteration is, in principle, a loop or an repeatedly executed instruction cycle, with only a few changes in each cycle. In programming languages that are not matrix- or array-oriented, even simple matrix multiplication needs three nested loops (over rows, columns, and the indices). Since S-Plus is matrix-oriented, these operations are much more efficient and easy to formulate in mathematical terms. This means they are typically substantially faster than loops and the code is much easier to read and write. The following guideline is elementary but important to be kept in mind when writing S-Plus code. Note Whenever possible, try to avoid loops. Ninety-nine percent of the time, an operation on matrices or lists is much more elegant as well as much faster. Try to use vectorized statements and functions such as apply, lapply, rapply, sapply, tapply, by, split, and others. Here is an easy example. If we define a vector x with 100,000 elements and assign the square of x to y, without using a loop, the following two lines take a fraction of a second to execute: > x y x for(i in 1:10) { print(i) } If there is only a single statement after the for command, the brackets { } can be omitted, but for ease of reading the code and to make sure the
304
9. Programming
intended statements are included in the loop, we recommend using them at all times. The object ivalues can be of any type: a vector, a variable, a matrix, or anything else. S-Plus walks through all the elements of the argument specified, replacing the loop variable (here i) by the elements in ivalues successively. The looping variable can be of any mode, such as numeric, character, Boolean, or a mixture of these. Here are a few examples. Example 9.5. A numeric looping variable > for(i in c(3, 2, 9, 6)) { print(iˆ2) } 9 4 81 36 Example 9.6. A character looping variable > transport.media for(vehicle in transport.media) { print(vehicle) } "car" "bus" "train" "bike" Note There is no explicit return argument in loops. Thus, you must always use the print function to print values within a loop. Using a statement like > for(i in 1:4) { i } to print out the current value of i does not result in printouts of the values i takes.
9.3.2
The while Loop
In contrast to the for loop, the while statement is used if the total number of iteration cycles is not known before the loop starts. For example, if you want to iterate until a certain criterion is fulfilled, the while loop is the appropriate statement to use. Example 9.7. A simple while loop Calculate the sum over 1,2,3,..., until the sum is larger than 1000. n x[, !apply(is.na(x), 2, all)] Generating a vector containing the number of values less than zero for each column of a matrix is done by > colSums(x colSums(x colMeans(x x y n 1000][1] or > (1:100)[cumsum(1:100)>1000][1]
9.3.5
Large Loops
There are situations where a loop cannot be avoided. If these loops have many cycles or if the block of commands is long, S-Plus might need a lot of memory and maybe even stop execution of the command due to lack of memory. This is often the situation where one starts to write the program in C. It can still be used from within S-Plus, as we will see later. However, let us point out a function that might turn out to be useful. The problem with large loops is that S-Plus does not free memory until completion of an expression. The function For circumvents this problem by basically splitting up a for loop into a series of top-level expressions. The syntax is slightly different from the for function, but, most important, all variables used within a For loop must be global variables, not local ones defined within functions. Furthermore, all variables defined within the For loop will be global ones in the local directory, even after the loop completes. Here is an example of a function f with a simple loop: f x f(x) 6 7 8 NULL The function was run now, and the last expression of the For statement was printed at the end of each cycle. Let us check the value of x: > x 8 The global variable x is modified! This illustrates our statement that all assignments are done within the global environment. We have made another (possible) mistake. The argument to the function, x, is completely ignored by For. The global variable x is used, no matter what gets passed to f. This is demonstrated by calling f again: > f(99) 9 10 11 NULL > x 11 Obviously, one needs to experiment a little with the function and its side effects. If the for statement comprises a set of commands, enclose them in braces ({ and }) and separate the statements by semicolons (;). A larger loop is mostly run as a batch job, so you might want to save your commands to a file and run a batch job (described in Section 12.2.6, p. 374).
9.4 Debugging: Searching for Errors Now it is time to write your own functions. If you haven’t tried this yet, don’t worry. You’ll have plenty of opportunities in the exercise section. Of course, when we write functions, they don’t always do what we want. This is the trouble with computers in general. They do exactly what we tell them
9.4. Debugging: Searching for Errors
309
to do, but this is very often not what we want them to do. This section is about searching for sources of errors. We will deal with four types of errors: • Syntax errors While the system is reading the definition of a function, it encounters a problem and stops reading the definition. An example is to close more brackets than we have opened. We are given an error message, typically beginning with Syntax Error: ... • Invalid arguments The function received arguments that do not allow it to correctly execute the function. Calling a routine to do a calculation on numbers and passing character variables to it causes such errors. • Execution errors While executing the function, S-Plus encounters an error such as an undefined operation. One such example is plotting x against y with x and y having different lengths. S-Plus stops the execution and we need to track where the error source originates. Most of the time, S-Plus tells us Error in ... If the execution was aborted in a statement that is not the origin of the error, as when we try to plot only missing values, we need to find the source with tools referenced in the following item. • Logical errors The S-Plus system completes the operation without detecting an error. However, the result returned to us is wrong; for example, the mean value of a data set being smaller than the smallest value. We need to track the whole function for a logical error that caused a programming error. The above list of problems and errors is, in general, in order of difficulty of finding the source of the problem. We will now tackle these errors systematically, introducing the tools S-Plus has to handle them.
9.4.1
Syntax Errors
Syntax errors are detected most easily. We might have forgotten a closing bracket or confused the name of a function with something else, and the system tells us while reading in the declaration that it disagrees with what it has just read: Syntax error: Unbalanced parentheses, expected "}", before ")" at this point: Such a message is typically followed by the line of code in question, and having another look at the code should clarify what the problem was.
310
9. Programming
Note If S-Plus gives a syntax error when reading something in from a file, and you cannot see directly what the problem was (maybe because you cannot find the line of code), mark the code using the mouse and copy and paste it into the Command window. S-Plus will react on the input directly, and you can see the source of the problem. When executing a longer sequence of commands (e.g., if the source command is used to read in commands from a file), it helps to set > options(echo=T) To change S-Plus’ sensitivity to warnings, you can change the option warn to the values from −1 to 3. Setting > options(warn=3) lets S-Plus terminate the current execution whenever a warning occurs. By default (which is options(warn=0)), warnings are collected and returned at the end of a statement’s execution.
9.4.2
Invalid Arguments
When writing a function, some errors can be checked for automatically. For example, if you write a function that gets an input parameter x, and x is not allowed to be negative, you can check for it and exit the function in a controlled manner by issuing an error message: f traceback() 5: eval(action, sys.parent()) 4: doErrorAction("Problem in xˆ2: Non-numeric first operand", 1000) 3: xˆ2 2: f("aha") 1: Message: Problem in xˆ2: Non-numeric first operand We see the list of calls, and the problem becomes clear. The function f was called as f("aha"). Within f, an expression xˆ2 was executed, and an error action was started with the message "Problem in xˆ2: Non-numeric first operand." Now we know where the error occurred. If we see the problem directly, fine. If not, we might want to examine the variable’s contents during the function’s execution or get some more information on the other commands being processed. This is covered in the following section.
9.4.4
Logical Errors
If you are familiar with other programming languages like C, you might know that these languages typically come with a debugger. The purpose of a debugger is to trace the state of a program while it is being executed. To do this, the program is not executed all at once but interrupted in certain places or executed line by line. S-Plus has these facilities, too. If we know where a problem occurs but are unsure what the problem is, we might want to interrupt the program in that specific place and look at the contents of the variables. There are several ways of doing it. The most common and possibly easiest approach is to use the browser function.
browser The browser function called within another function interrupts the execution and lets the user query contents of variables and inspect levels of calls.
312
9. Programming
We can also overwrite any variables defined in the function by assigning something to them. The call can be inserted manually into a function, as in f trace(f, browser) to call browser on entering the function f, > trace(f, exit=browser) to enter browser on exiting the function f, and > trace(f, browser, exit=browser) to enter browser on entering and exiting f. Replace f by any function name. The function untrace ends the tracing process. Call untrace(f) to stop tracing the function f or untrace() to stop all tracing. Before calling untrace again, take a look at the function you are tracing.
debugger If you want to run a debugger after your function crashed, tell S-Plus to dump all current frames on exiting the function. S-Plus will write out all levels of calls and their variables on exiting. You can then run the debugger. As dumping out all frames might take some time, depending on the complexity of the function, the system default when an error occurs is to dump only the levels of function calls. By default, the settings are 3 This information was kindly provided by Insightful at the time of writing this book, based on beta releases of S-Plus Version 7.0. 4 Personal observation using S-Plus Version 7.0.0 beta for Linux 2.4.18. It works despite the fact that browser only calls browser.default.
314
9. Programming
> options(error=expression(dump.calls())) If we want to go through the levels of function calls and examine the variables’ contents, we need to change the setting to > options(error=expression(dump.frames())) and run the function again. The dump will then be done (if the function crashes again), and we can inspect the state of the system by using the debugger: > debugger() If your function works fine, you might want to reset the error action to just dumping calls.
recover S-Plus can automatically enter a debugging tool if an error occurs. The function recover has basically the same functionality as browser. We use it to examine the state of the system in case of an error. S-Plus is told to call recover each time an error occurs by setting an option as follows: > options(error=expression(if(interactive()) recover() else dump.calls())) If S-Plus is not run interactively but in batch mode, recover should not be called, as it expects interactive input via the keyboard. That is why we check if S-Plus is in the interactive mode by using interactive. If an error occurs, you are prompted with a message and asked if you want to debug the problem: Problem in xˆ2: Non-numeric first operand Debug ? ( y|n ): Enter a “y” to start the recover function. Table 9.1 lists the basic commands that are common to browser, recover, and debugger. Pick the debugging tool most appropriate for your current problem. Most likely you will have to experiment a little with the different tools and options. Finally, note that you can also debug S-Plus internal routines if necessary. Chapter 10 on object-oriented programming has some more details and an extended example of a debugging session in the exercises.
9.5 Output Using the cat Function In addition to the simple print function, which prints a variable’s value, there is a much more powerful tool: the cat function. If you know the cat
9.5. Output Using the cat Function
315
Table 9.1. Some commands in browser, debugger, and recover.
Command ? where up (down) q object any S-Plus expression
Effect Help. Show the names of all variables in the current evaluation frame1 Display current frame list Go up (down) in the list of evaluation frames Quit Show the contents of object Execute the expression and keep results or assignments in the current frame
1
Global variables defined in the interactive environment are also known within any frame. function from the C language, you will find many similarities. Here are some examples: > x cat(x) 1 2 3 > cat("Hello.") Hello. > x cat("x has a value of", x, ".") x has a value of 7 . S-Plus and C have similar control characters (see Table 9.2). Table 9.2. Control sequences in S-Plus.
Control sequence
Effect
\n \t \\ \" \b \r \octalcode
New line Tabulator Backslash (\) " Backspace Carriage return Number coding a character in octal format (not ASCII, see a PostScript manual)
These control characters can be used in any string, and the S-Plus functions will deal with them accordingly. For example, a graph title going over two lines can be created by inserting a new line control character somewhere in the title string and entering title("First\nand second line"). Note that there is no space around the \n, as this would add an extra space to the title.
316
9. Programming
9.6 The paste Function The paste function is one of S-Plus’ most universal functions. It takes any number of arguments and concatenates them into strings. Some examples are: • Title strings for graphics > s1 s2 title(s1, s2) • Names for matrix and array rows and columns: Assume that a matrix x with some medical data exists. > row.names col.names dimnames(x) paste(..., sep=" ", collapse=NULL) sep is the field separator, which is set to " " by default. A field separator is put in the space between two concatenated objects. The argument collapse is the object separator, empty by default, such that the objects to be pasted together do not get collapsed. If collapse is modified to "", the result is a long string instead of a vector of strings, if the arguments were vectors. Here are some more examples of how paste can be used: > str paste("String", 1:length(str), "(", str, ") has", nchar(str), "characters.") "String 1 ( A ) has 1 characters." "String 2 ( vector ) has 6 characters." "String 3 ( of ) has 2 characters." "String 4 ( strings ) has 7 characters." We can solve the problem of writing “character” and “characters,” respectively, very elegantly: > paste("String ", 1:length(str), " (", str, ") has ", nchar(str), " character", ifelse(nchar(str)==1, "", "s"), ".", sep="") "String 1 (A) has 1 character."
9.6. The paste Function
317
"String 2 (vector) has 6 characters." "String 3 (of) has 2 characters." "String 4 (strings) has 7 characters." The next example uses different settings of sep and collapse. > x names(x) paste(names(x), "=", 1:3, ".") "X = 1 ." "Y = 2 ." "Z = 3 ." > paste(names(x), "=", 1:3, ".", sep="") "X=1." "Y=2." "Z=3." > paste(names(x), "=", 1:3, ".", collapse="") "X = 1 .Y = 2 .Z = 3 ." > paste(names(x), "=", 1:3, ".", sep="", collapse="") "X=1.Y=2.Z=3." > paste(names(x), "=", 1:3, ".", sep="", collapse=" ") "X=1. Y=2. Z=3." Note The function paste has a counterpart to split character strings at specified separators. The function is called unpaste.
318
9. Programming
9.7 Exercises Exercise 9.1 Take a list L generated as follows. > L L sapply(L, mean) If we wanted to, we could generate the matrix containing mean, variance, and number of observations of each list element with a single command: > x dimnames(x) trace(show, browser) calls browser each time the function show is entered. > trace(show, browser, exit=browser) calls browser on entering and exiting the function show. > trace(show, exit=browser) calls browser on exiting the function show. To stop tracing of all functions, the function untrace can be used: > untrace() If there are too many calls to the function we want to trace, we can only trace it for calls with objects of certain classes such as country. The function traceMethod is called as follows to trace only calls to show with objects of class country: > traceMethod("show", "country", browser) The arguments are the same as for trace.
334
10. Object-Oriented Programming
10.4 Help Besides the standard online help, > help("helptopic") there are some ways to query object-oriented help pages. To query the definition of a class – for example, matrix – enter > class ? matrix If we are interested in the methods provided, we can ask for the generic function and its associated library of class-specific functions. > method ? show displays the list of class-specific show functions. We can also ask directly for the show function for objects of class country by entering > method ? show("country")
10.5 Summary and Overview Recall again what we have done in this chapter. We have created a new class country. We have defined a function to generate objects of the new class, and we have added class-specific methods to show and plot objects of class country. We have also added a new method " class(x) "character" > class(y) "factor" > class(z) "ordered" > is.factor(x) F > is.factor(y) T > is.factor(z) T The order of the categories – cheap, acceptable, expensive – might not be what we want. We might want to change the order of the categories from acceptable < cheap < expensive (alphabetically sorted) to cheap < acceptable < expensive (sorted by price category). There is a function levels to access the levels of a factor object. > levels(z) "acceptable" "cheap" "expensive" Using the function levels we can assign new levels to z, but we have to be careful. The following example illustrates what can go wrong. > levels(z) z cheap acceptable expensive cheap expensive
380
12. Tips and Tricks
expensive
expensive
acceptable cheap
cheap
cheap < acceptable < expensive The levels are now in the desired order, but look at the data. We have not reordered the categories but have changed the labels of the categories and therefore the data! Everything that used to be acceptable has become cheap, and everything that used to be cheap has become acceptable. Instead, we should have defined the order of the categories while constructing the object: > z z acceptable cheap expensive acceptable expensive expensive expensive cheap acceptable acceptable cheap < acceptable < expensive Alternatively, we could have redeclared z as an object of class ordered: > z print.default(z) 2 1 3 2 3 3 3 1 2 2 attr(, "levels"): "cheap" "acceptable" "expensive" attr(, "class"): "ordered" "factor" The factor object z is stored as a vector of integer numbers 1, 2, and 3. It has an attribute “levels” (containing the levels cheap, acceptable, and expensive, in that order) and an attribute “class” containing the classes to which the object z belongs (ordered and factor, in that order). Can you see now what we did before by assigning new values to the levels of z and why the data changed instead of the order of the levels? One might wonder if we can use those numeric representations for calculations. Trying it out on the three objects we already generated shows slightly different behavior: > 2*x
12.3. Factors
381
Problem in 2 * x: Non-numeric second operand Use traceback() to see the call stack > 2*y NA NA NA NA NA NA NA NA NA NA > 2*z Problem in Ops.ordered(e1, e2): "*" not meaningful for ordered factors Use traceback() to see the call stack Obviously, three different commands were executed for multiplying the three objects by two. Multiplying a factor object with a number results in missing values, whereas multiplying a character or an ordered object with a number results in an error message. Of course, if you know you want to use the numeric coding of the factors, you should convert the factor data using as.integer before carrying out any other operation.
12.3.3
Where Levels Play a Role
Factor levels are used by certain functions in S-Plus. If data are summarized in a table, the order of the table elements is determined by the order of the levels of the object. Therefore, the tabulation of y and z differs slightly. > table(y) acceptable cheap expensive 4 2 4 > table(z) cheap acceptable expensive 2 4 4 Trellis-type graphs use the order of the levels as well. For the barley data set, the levels of the year are assigned in reverse order. > levels(barley$year) "1932" "1931" This is why if you plot the barley data set with year as conditioning variable, 1932 is shown before 1931. Check the graph you get on entering > dotplot(site ∼ yield | year, data=barley) You should be able now to make a copy of the data set and reverse the factor levels such that 1931 is shown before 1932. Note Recall that you do not see the levels of a factor variable by printing out the raw data. There can be empty factor levels with no corresponding observation, and the order of levels is not visible either. The order of levels becomes relevant when the data is processed, for example in tabulating it or in displaying it graphically.
382
12. Tips and Tricks
It can be useful to arrange factor levels according to the inherent meaning (as in our example, cheap is less than acceptable is less than expensive). This kind of sorting is typically a manual process, the factor levels need to be specified in their order by typing them in. Sometimes it is useful to arrange factor levels according to another numeric variable, for example if the levels of the variable site in the Barley data set are arranged in order of average yield. The function reorder.factor serves that purpose. We will cover reorder.factor in an exercise.
12.3.4
Where Factors Can Lead Their Own Lives
The following illustrates some situations where factors might not behave the way you want them to behave. Using the paste Function The function paste is a frequently used tool. It can be used to generate row or column labels for the data or graph titles. We will use the paste function to illustrate the different results it generates for character, factor, and ordered objects. > paste("x is", x) "x is acceptable" "x is cheap" "x is acceptable" "x is expensive" "x is expensive" "x is cheap" "x is acceptable"
"x is expensive" "x is expensive" "x is acceptable"
This output is what we want, a combination of “x is” and the label. Now let us paste together “y is” and the factor y. > paste("y is", y) "y is 1" "y is 2" "y is 3" "y is 1" "y is 3" "y is 3" "y is 3" "y is 2" "y is 1" "y is 1" Obviously, this time S-Plus took the internal integer representation for pasting. Let’s try z: > paste("z is", z) "z is 2" "z is 1" "z is 3" "z is 2" "z is 3" "z is 3" "z is 3" "z is 1" "z is 2" "z is 2" We get numbers different from the ones for y. Can you explain it? The explanation is that the levels for y and z have a different order, and the numbers we get to see correspond to the index of the levels vector. Compare the numbers: 1 and 2 are swapped between y and z, corresponding to a swap of “acceptable” and “cheap” in the levels.
12.3. Factors
383
So what would we need to do to obtain the desired character vectors in the pasted result? As characters are what we want, converting the factor into a character object before pasting should produce the desired result. > paste("y is", as.character(y)) "y is acceptable" "y is cheap" "y is acceptable" "y is expensive" "y is expensive" "y is cheap" "y is acceptable"
"y is expensive" "y is expensive" "y is acceptable"
Comparisons with Factors What happens if we want to check if a certain element is present in the data and it is not present? Let us check for our three different objects. > x=="anything" F F F F F F F F F F > y=="anything" F F F F F F F F F F > z=="anything" NA NA NA NA NA NA NA NA NA NA For this reason, subsetting results in different output too. > x[x=="anything"] character(0) > y[y=="anything"] character(0) Levels: "acceptable" "cheap" "expensive" > z[z=="anything"] NA NA NA NA NA NA NA NA NA NA Levels: "cheap" "acceptable" "expensive" cheap < acceptable < expensive Using Factors for Subscripting If you recall Section 4.1.1 (page 99), the column names of a matrix can be used to reference the columns. In a similar manner, the elements of a vector can be referenced by their names, and arrays and data frames can be subscripted using their dimension names (dimnames). If factors are used for subscripting, it is important to realize that the numeric representation is used and not the character representation. Here is an example. > xdata xdata one two three 1 2 3 > xi xdata[, xi[1]] 1 > xdata[, xi[2]] 3 You can probably guess why xdata[, xi[2]] yields the value 3 and not 2. It is because xi[2] is internally represented as 3. The factor level 2 shows up in position 3 of the levels and to subscript the data frame, the numerical value 3 is used and not the string "two". > levels(xi) "one" "three" "two" So if you want to access data using the row and column names, make sure that you don’t use a factor variable for subscripting (or convert it to integer values beforehand).
12.3.5
How Factors Come Into Life
Factors are often generated when reading or manipulating data sets. For example, when reading data using the graphical user interface (with the default settings), S-Plus will recognize certain data as factor objects. If the functions importData, openData, or the older read.table are used, the defaults are set to convert string variables into factors. If you construct a new data frame using data.frame, character variables will become factors in the data frame. As this might happen without explicitly noticing, it might be preferable if the variables are not stored as factors but rather as characters. They can still be converted later if necessary. As a general rule of thumb, factors are preferable if Trellis graphs and statistical modeling functions are used. If the major aim is to manipulate the raw data, characters might be preferable. In general, manipulations using characters will be faster than those using factor objects. To prevent conversion to factors, use the settings stringsAsFactors=F for importData and as.is=T for read.table. Have a look at the documentation for the individual functions. The menu system of S-Plus for Windows allows you to change the default class for objects from factor to character. For creating a data frame with character data, use data.frame with I(x) as argument, where x needs to be replaced by the object to include in the data frame (example: data.frame(I(letters))). You should be aware that there are some side effects. As an example, the summary function treats character objects differently from factor objects.
12.3. Factors
385
We construct a data frame and use the function summary to summarize each element of the data frame. > xx xx x y z acceptable acceptable acceptable cheap cheap cheap expensive expensive expensive acceptable acceptable acceptable expensive expensive expensive expensive expensive expensive expensive expensive expensive cheap cheap cheap acceptable acceptable acceptable acceptable acceptable acceptable The display of the three elements of the data frame is identical, but using the function summary reveals that there are differences. > summary(xx) x Length: 10 Class: AsIs Mode:character
y acceptable:4 cheap:2 expensive:4
z cheap:2 acceptable:4 expensive:4
The summary function displays a table of values for the factor object and some information about the object’s internals, length and class, for the character vector – as if it does not really know what to do with it.
12.3.6
Adding and Dropping Factor Levels
The levels of a factor are an attribute of the object. They are not revised when changing the data of the object. For the objects as defined above, the valid levels are cheap, acceptable, and expensive. So, if we assign something else to an element of the object, it will not be accepted as a proper definition: > y[1] y y[1] y very expensive cheap acceptable expensive expensive cheap acceptable
expensive expensive acceptable
In a similar fashion, if all elements of a certain value are dropped, the level still lives on in the levels attribute. > y1 levels(y1) "acceptable" "cheap" "expensive" "very expensive" Although "cheap" no longer exists in the data set, the level "cheap" still exists (and will be used, for example, by table(y1)). This can make sense, of course, as we might have a sample that just by chance does not contain a certain observation, but we still might want to document a factor level that has not been observed. Sometimes, for example for generating graphs, it is better to drop an empty factor level unless you really want an empty graph (reflecting no data in that factor level). A straightforward way of doing this is to redeclare the object as a factor. > y1 levels(y1) "acceptable" "expensive" "very expensive" If you use Trellis graphs, you might also use the subset option to exclude the empty factor level (along the lines of subset=(y!="cheap")). You can experiment for yourself with the barley data set. Create a copy of the data set in your working directory by entering > barley graphsheet(file="figure##.wmf", format="WMF") the first graph is saved to the file figure01.wmf, the second to figure02.wmf, and so on. Just make sure to include multiple # signs if you intend to create many graphs. The same technique works for all other formats, including PostScript. Under some rare circumstances, we found that minor details such as axis tickmarks can be awkward using copy and paste. For those cases, we recommend you use the WMF format. Most Windows applications (including MS Word) can also include PostScript graphs. You just need to figure out how to insert the graph file into your Word or WordPerfect document, your PowerPoint slides, or any other document. If the PostScript graphs do not contain a “preview,” you will not see the graphs but a placeholder until you print them out (on a PostScript printer). If you don’t have a PostScript printer, refer to Section 12.4.4.
388
12. Tips and Tricks
12.4.2
Generating PostScript Graphs
PostScript graphs are of high quality. They are the standard for professional typesetters. Encapsulated PostScript (EPS) ensures that a graph can be integrated with other documents as an object of known size. EPS graphs can be used with almost any word processor. To generate an EPS file using S-Plus for Windows, use > graphsheet(file="graphfilename.eps", format="EPS") On UNIX systems, use > postscript(file="graphfilename.eps", horizontal=F, onefile=F, print.it=F) Now, enter a command to generate a graph and close the device, so that the file is completed: > dev.off()
# close the device
You can alternatively create the graph on a screen device and copy it to a PostScript file when it is finished. Use (for UNIX) > printgraph(method="postscript", file="graphfilename.eps", horizontal=F) or (for Windows) > dev.copy(graphsheet, format="EPS", file="graphfilename.eps") > dev.off() # close the device If you have a PostScript previewer such as GhostScript/GhostView (see the references), you can look at the PostScript graph on the screen. An Encapsulated PostScript graph is essential to ensure easy integration of the graph into a text document. Encapsulated means basically that the graph file contains information about the size of the graph. Word processors can derive all necessary information to integrate and resize the graph from the file. PostScript files are plain text files that you can look at. An Encapsulated PostScript graph contains header information such as the following: %!PS-Adobe-3.0 EPSF-3.0 %%Title: (S-PLUS Graphics) %%Creator: S-PLUS %%For: (Andreas Krause) %%CreationDate: Sat Apr 9 11:37:01 2005 %%BoundingBox: 20 170 592 622 The first line gives information about the EPS (Encapsulated PostScript) level, and a few lines later, the bounding box defines the size of the overall graph.
12.4. Including Graphs in Text Processors
389
Note A graph file should not contain more than one graph. A multiplegraph or even multiple-page document is difficult if not impossible to handle as a (single) object. The graph size should not be specified from within S-Plus. Creating a full page graph and resizing it within the text processing system gives better results.
12.4.3
PostScript Graphs in LATEX
We focus here on the LATEX 3 standard based on LATEX 2ε. If you want to consult standard sources to go into more detail, refer to “The LATEX Companion” or “The LATEX Graphics Companion” by Goossens et.al. (1993, 1997). If you use TEX or LATEX and don’t know exactly what system you are using, check the very first line of your document. There should be a line containing the word “documentclass.” If the document begins with “documentstyle,” you are using the old LATEX 2.09 and should consider upgrading. If you find neither of the two, you are most likely using plain TEX (your document will end on \bye). We assume that a PostScript file is already generated and ready to be included into a text file. LATEX offers a huge library of packages that can easily be downloaded from a CTAN (Comprehensive TeX Archive Network) site. Internet addresses are given in the references (pp. 425ff.). The “graphics” and “graphicx” packages are widely used (“graphicx” stands for enhanced “graphics”). Before attempting to download the packages, make sure they are not installed yet (just try what follows). Add \usepackage{graphicx} to the header of your document. A PostScript graph can now easily be included (as a floating object) with the statement \includegraphics{graphfilename.eps} If you want to automatically enumerate figures, dynamically reference them and generate a list of figures, use the figure environment. You can embed it in a new command called \Figure as follows. % Parameters: % figure legend, entry for "list of figures," filename \newcommand{\Figure}[3] { \begin{figure} \centerline{\includegraphics[width=\textwidth]{#3}} \caption[#2]{#1} \end{figure}
390
12. Tips and Tricks
} Note that the figure is automatically scaled to the document’s text width. The height is scaled proportionally to the width. The figure can now be included into the document. \Figure{Text underneath the figure\label{fig nicegraph}}% {The short title for the list of figures entry}% {graphfilename.eps} The figure is referenced by referencing the label: Figure∼\ref{fig nicegraph} (page∼\pageref{fig nicegraph}) You need to run LATEX twice, once for writing the new figure and label to files, and once for resolving the reference. Finally, there are many people working with LATEX as a text processor and S-Plus as statistics and graphics software. Therefore, there are many tools available interfacing the two. For example, on the StatLib server (see Section 14.3, p. 414), there is an S-Plus function for writing S-Plus data sets directly to a LATEX table environment.
12.4.4
If You Don’t Have a PostScript Printer
Even if no PostScript printer is available, you can insert the figures as PostScript format files into your document. Create a PostScript file of your document (for LATEX, use dvips; for Windows programs, install a PostScript printer such as the Apple Laserwriter and redirect the printer output to a file) and then use a translation program to convert the PostScript file to a format suitable for your printer. The most common tool for translating PostScript format to other formats is GhostScript, available on many electronic archives. The printout quality is very good, and you can also preview the printout on the screen. It supports far more than 100 differnet output devices.
12.4.5
Greek Letters in Graphs
If you need to add Greek letters or even formulas to your graphs, there are some ways of doing it. If you run S-Plus for Windows, you can create a graph and edit it. You might need to convert the graph to objects first by clicking with the right mouse button on the graph and selecting “Convert to objects.” Select the text to change and change the font to symbol. You can do so by using the context menu (right-click on the object) and selecting Font from the Format menu. A p will result in a lowercase pi (π) and a P in an uppercase pi (Π), an s will give a sigma (σ), and an S will generate an uppercase sigma (Σ).
12.4. Including Graphs in Text Processors
391
The Greek (symbol) font and other fonts cannot be combined within a single object such as a title, so that if you want to mix Greek and Latin letters, you need to create two text objects and align them against one another. If you run S-Plus on a UNIX system, you can use the symbol font as well. Use > text(xpos, ypos, text, font=13) to add Greek letters to your graph at a specific position. This method can be used for axis labels and other string positioning as well. As under Windows, you cannot mix Greek and Latin characters. If you intend to do so, check the StatLib library for contributed functions from other S-Plus users. If you use TEX/LATEX as the word processor, consider using the psfrag utility, available from any CTAN archive. Using this utility allows TEX formulas to be typeset within graphs.
392
12. Tips and Tricks
12.5 Exercises Exercise 12.1 Display the yield in the data set barley in histogram form. Use a Trellistype graph, use one panel per site, and arrange the sites such that the order corresponds to the maximum yield per site. Check your graph for the correct order of sites.
Exercise 12.2 We look at different behavior of objects of classes “character,” “factor,” and “ordered.” Take the vectors x, y, and z as generated on page 380: > set.seed(1234) > x y z x "acceptable" "cheap" "expensive" "expensive" "expensive" "expensive" "acceptable" "acceptable"
"acceptable" "cheap"
> y acceptable cheap expensive expensive
expensive cheap
acceptable expensive acceptable acceptable
acceptable cheap expensive expensive
expensive cheap
acceptable expensive acceptable acceptable
> z
cheap < acceptable < expensive Think about the result of using the < (“less than”) operator to compare the first and the second element of x, y, and z. What do you expect to get on entering > x[1] < x[2] > y[1] < y[2] > z[1] < z[2] taking into account the different classes of the objects? The contents of x, y, and z are identical, but the class attributes are not. So what would you guess to get on entering
12.5. Exercises
393
> x==y > x==z > y==z Note that if S-Plus attempts to compare objects of different class, it attempts to convert objects so that they belong to the same class before carrying out the comparison. Write down your answers before looking at the solution.
Exercise 12.3 Write a class-specific function summary for objects of class character. It should behave identically to the function summary when applied to a factor. For illustration, compare the difference between the output of > summary(barley) and > summary(barley2) where barley2 is defined by > barley2 barley2$site barley.extended barley.extended$site.by.yield histogram(∼yield | site.by.yield, data=barley.extended) In the graph created above, sites are sorted such that the first level, corresponding to the smallest maximum yield, gets shown in the lower left corner, and the last level in the upper right corner. This default ordering can be reversed by adding as.table=T to the call to histogram. > histogram(∼yield | site.by.yield, data=barley.extended, as.table=T) However, possibly the best way to graphically assess if the sites are actually ordered according to their maximum yield is to stack the histograms on top of one another. Since they have the same x-axis, a comparison of the maximum values is easy. For this purpose we specify explicitly the layout of the panels, 1 column and 6 rows. > histogram(∼yield | site.by.yield, data=barley.extended, layout=c(1,6)) The resulting graph is shown in Figure 12.3.
12.6. Solutions
Waseca
395
50 40 30 20 10 0
Crookston
50 40 30 20 10
Percent of Total
0
Morris
50 40 30 20 10 0
University Farm
50 40 30 20 10 0
Grand Rapids
50 40 30 20 10 0
Duluth
50 40 30 20 10 0 20
30
40
50
60
yield
Figure 12.3. Histograms of Barley yield by site. Sites (factor levels) sorted in ascending order of maximum yield.
Solution to Exercise 12.2 We apply the function < (“less than”) to objects of class character, factor, and ordered that contain the “same” data (except for the specification of class attributes). The data for x are > x "acceptable" "cheap" "expensive" "acceptable" "expensive" "expensive" "expensive" "cheap" "acceptable" "acceptable" First, let us test if x[1] is less than x[2]: > x[1] < x[2] T S-Plus tells us the statement is true, "acceptable" is less than "cheap", because it comes first when sorted alphabetically. > y[1] < y[2] NA The object y is of class factor, and the factor levels cannot be sorted. Therefore, the comparison “less than” is not permitted and the result is NA (not available).
396
12. Tips and Tricks
> z[1] < z[2] F The object z is of class ordered such that the order of the levels determines the result of the comparison. If z is printed, the order of the classes is shown as cheap < acceptable < expensive such that acceptable is not less than cheap for z as defined here. Let us find out what we get if we compare x to y, x to z, and y to z. If objects of different class are compared, S-Plus attempts to convert the objects such that they belong to the same class and picks the more “general” class for both objects. > x==y T T T T T T T T T T The objects x and y are compared by converting y to the class of x, character, before comparing them. This gives identical objects. > x==z T T T T T T T T T T The same conversion happens on comparing x to z. z is converted to character, and the result is the same as if we entered x==as.character(z). > y==z Warning messages: Incompatible methods ("Ops.factor", "Ops.ordered") for "==" in: y == z Problem in print.factor(object): cannot be interpreted as a factor: No levels Use traceback() to see the call stack The objects y and z cannot be compared to one another, as a factor cannot (meaningfully) be converted into an object of class ordered and vice versa. Therefore, we get this error message. Actually, the output above is a composition of two messages: a warning and an error message. Assign the result of the comparison to a variable and then print the variable. It shows that the warning is issued by the comparison, but the error is issued by the print function. Finally, let us convert y to class ordered and compare it to z: > ordered(y)==z Problem in Ops.ordered(e1, e2): "==": levels must be the same Use traceback() to see the call stack S-Plus tells us that if we want to compare objects of class ordered, they must have the same levels. Finally, we can get the comparison to work by entering
12.6. Solutions
397
> ordered(y, levels=levels(z)) == z T T T T T T T T T T or > as.character(y) == as.character(z) T T T T T T T T T T
Solution to Exercise 12.3 We are going to write a class-specific function summary that summarizes objects of class character in the form of a table, as it exists for objects of class factor. We use the Barley data set to test our routine. > barley2 barley2$site summary(barley) yield variety Min.:14.43333 Svansota,No. 462:24 1st Qu.:26.87500 Manchuria:12 Median:32.86667 No. 475:12 Mean:34.42056 Velvet,Peatland:24 3rd Qu.:41.40000 Glabron:12 Max.:65.76670 No. 457:12 Wisconsin No. 38,Trebi:24 year 1932:60 1931:60
site Grand Rapids:20 Duluth:20 University Farm:20 Morris:20 Crookston:20 Waseca:20 > summary(barley2) yield variety Min.:14.43333 Svansota,No. 462:24 1st Qu.:26.87500 Manchuria:12 Median:32.86667 No. 475:12 Mean:34.42056 Velvet,Peatland:24 3rd Qu.:41.40000 Glabron:12 Max.:65.76670 No. 457:12 Wisconsin No. 38,Trebi:24
398
12. Tips and Tricks
year 1932:60 1931:60
site Length: 120 Class: Mode:character
Converting an object of class character into class factor would give the desired result: > summary(as.factor(barley2$site)) Crookston Duluth Grand Rapids Morris 20 20 20 20 University Farm Waseca 20 20 We therefore set a new method that simply converts a character object into a factor object and calls the corresponding summary function. > setMethod("summary", signature(object="character"), function(object, ...) summary(as.factor(object))) Look up the chapter on object-oriented programming (chapter 10, pp. 323 ff.) to refresh how to set class-specific methods. Testing the revised routine to handle summaries of character objects gives the desired result. > summary(barley2) yield Min.:14.43333 1st Qu.:26.87500 Median:32.86667 Mean:34.42056 3rd Qu.:41.40000 Max.:65.76670
year 1932:60 1931:60
variety Svansota,No. 462:24 Manchuria:12 No. 475:12 Velvet,Peatland:24 Glabron:12 No. 457:12 Wisconsin No. 38,Trebi:24
site Crookston:20 Duluth:20 Grand Rapids:20 Morris:20 University Farm:20 Waseca:20
To remove the method again, use > removeMethod("summary", sig=c("character"))
12.6. Solutions
399
Solution to Exercise 12.4 When writing a batch job, we first need to write the batch job input. Use any editor you like to save the commands. x y y 4 This functionality is especially useful if a longer calculation was run and the result was not assigned to a variable.
13.4 Levels of Calls In S-Plus, as in most modern computer languages, there are levels of calls. Each level keeps its own variables and must pass them explicitly to other levels. Only global variables are known to all routines. In S-Plus, these are the variables of the interactive environment. Here is a short example designed to show the idea behind levels of calls. Try to figure out what value will be printed by the print statements if we enter f(3) after declaring the function as follows. Example 13.1. The effects of levels in expressions We declare a function g inside a function f and call the function with f(3). Think of what will be printed out with the statements print(x) that we inserted. f
