Smoothing Splines: Methods and Applications (Chapman & Hall CRC Monographs on Statistics & Applied Probability)

Statistics 121 The book provides unified frameworks for estimation, inference, and software implementation by using th...

Author: Yuedong Wang

51 downloads 1450 Views 5MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Statistics

121

The book provides unified frameworks for estimation, inference, and software implementation by using the general forms of nonparametric/ semiparametric, linear/nonlinear, and fixed/mixed smoothing spline models. The theory of reproducing kernel Hilbert space (RKHS) is used to present various smoothing spline models in a unified fashion. Although this approach can be technical and difficult, the author makes the advanced smoothing spline methodology based on RKHS accessible to practitioners and students. He offers a gentle introduction to RKHS, keeps theory at a minimum level, and explains how RKHS can be used to construct spline models. Smoothing Splines offers a balanced mix of methodology, computation, implementation, software, and applications. It uses R to perform all data analyses and includes a host of real data examples from astronomy, economics, medicine, and meteorology. The codes for all examples, along with related developments, can be found on the book’s web page.

Smoothing Splines

A general class of powerful and flexible modeling techniques, spline smoothing has attracted a great deal of research attention in recent years and has been widely used in many application areas, from medicine to economics. Smoothing Splines: Methods and Applications covers basic smoothing spline models, including polynomial, periodic, spherical, thin-plate, L-, and partial splines, as well as more advanced models, such as smoothing spline ANOVA, extended and generalized smoothing spline ANOVA, vector spline, nonparametric nonlinear regression, semiparametric regression, and semiparametric mixed-effects models. It also presents methods for model selection and inference.

Monographs on Statistics and Applied Probability 121

Smoothing Splines Methods and Applications

Wang

Yuedong Wang

C7755

C7755_Cover.indd 1

4/25/11 9:28 AM


C7755_FM.indd 1

4/29/11 2:13 PM

MONOGRAPHS ON STATISTICS AND APPLIED PROBABILITY General Editors F. Bunea, V. Isham, N. Keiding, T. Louis, R. L. Smith, and H. Tong 1 Stochastic Population Models in Ecology and Epidemiology M.S. Barlett (1960) 2 Queues D.R. Cox and W.L. Smith (1961) 3 Monte Carlo Methods J.M. Hammersley and D.C. Handscomb (1964) 4 The Statistical Analysis of Series of Events D.R. Cox and P.A.W. Lewis (1966) 5 Population Genetics W.J. Ewens (1969) 6 Probability, Statistics and Time M.S. Barlett (1975) 7 Statistical Inference S.D. Silvey (1975) 8 The Analysis of Contingency Tables B.S. Everitt (1977) 9 Multivariate Analysis in Behavioural Research A.E. Maxwell (1977) 10 Stochastic Abundance Models S. Engen (1978) 11 Some Basic Theory for Statistical Inference E.J.G. Pitman (1979) 12 Point Processes D.R. Cox and V. Isham (1980) 13 Identification of Outliers D.M. Hawkins (1980) 14 Optimal Design S.D. Silvey (1980) 15 Finite Mixture Distributions B.S. Everitt and D.J. Hand (1981) 16 Classification A.D. Gordon (1981) 17 Distribution-Free Statistical Methods, 2nd edition J.S. Maritz (1995) 18 Residuals and Influence in Regression R.D. Cook and S. Weisberg (1982) 19 Applications of Queueing Theory, 2nd edition G.F. Newell (1982) 20 Risk Theory, 3rd edition R.E. Beard, T. Pentikäinen and E. Pesonen (1984) 21 Analysis of Survival Data D.R. Cox and D. Oakes (1984) 22 An Introduction to Latent Variable Models B.S. Everitt (1984) 23 Bandit Problems D.A. Berry and B. Fristedt (1985) 24 Stochastic Modelling and Control M.H.A. Davis and R. Vinter (1985) 25 The Statistical Analysis of Composition Data J. Aitchison (1986) 26 Density Estimation for Statistics and Data Analysis B.W. Silverman (1986) 27 Regression Analysis with Applications G.B. Wetherill (1986) 28 Sequential Methods in Statistics, 3rd edition G.B. Wetherill and K.D. Glazebrook (1986) 29 Tensor Methods in Statistics P. McCullagh (1987) 30 Transformation and Weighting in Regression R.J. Carroll and D. Ruppert (1988) 31 Asymptotic Techniques for Use in Statistics O.E. Bandorff-Nielsen and D.R. Cox (1989) 32 Analysis of Binary Data, 2nd edition D.R. Cox and E.J. Snell (1989) 33 Analysis of Infectious Disease Data N.G. Becker (1989) 34 Design and Analysis of Cross-Over Trials B. Jones and M.G. Kenward (1989) 35 Empirical Bayes Methods, 2nd edition J.S. Maritz and T. Lwin (1989) 36 Symmetric Multivariate and Related Distributions K.T. Fang, S. Kotz and K.W. Ng (1990) 37 Generalized Linear Models, 2nd edition P. McCullagh and J.A. Nelder (1989) 38 Cyclic and Computer Generated Designs, 2nd edition J.A. John and E.R. Williams (1995) 39 Analog Estimation Methods in Econometrics C.F. Manski (1988) 40 Subset Selection in Regression A.J. Miller (1990) 41 Analysis of Repeated Measures M.J. Crowder and D.J. Hand (1990) 42 Statistical Reasoning with Imprecise Probabilities P. Walley (1991) 43 Generalized Additive Models T.J. Hastie and R.J. Tibshirani (1990) 44 Inspection Errors for Attributes in Quality Control N.L. Johnson, S. Kotz and X. Wu (1991) 45 The Analysis of Contingency Tables, 2nd edition B.S. Everitt (1992)

C7755_FM.indd 2

4/29/11 2:13 PM

46 The Analysis of Quantal Response Data B.J.T. Morgan (1992) 47 Longitudinal Data with Serial Correlation—A State-Space Approach R.H. Jones (1993) 48 Differential Geometry and Statistics M.K. Murray and J.W. Rice (1993) 49 Markov Models and Optimization M.H.A. Davis (1993) 50 Networks and Chaos—Statistical and Probabilistic Aspects O.E. Barndorff-Nielsen, J.L. Jensen and W.S. Kendall (1993) 51 Number-Theoretic Methods in Statistics K.-T. Fang and Y. Wang (1994) 52 Inference and Asymptotics O.E. Barndorff-Nielsen and D.R. Cox (1994) 53 Practical Risk Theory for Actuaries C.D. Daykin, T. Pentikäinen and M. Pesonen (1994) 54 Biplots J.C. Gower and D.J. Hand (1996) 55 Predictive Inference—An Introduction S. Geisser (1993) 56 Model-Free Curve Estimation M.E. Tarter and M.D. Lock (1993) 57 An Introduction to the Bootstrap B. Efron and R.J. Tibshirani (1993) 58 Nonparametric Regression and Generalized Linear Models P.J. Green and B.W. Silverman (1994) 59 Multidimensional Scaling T.F. Cox and M.A.A. Cox (1994) 60 Kernel Smoothing M.P. Wand and M.C. Jones (1995) 61 Statistics for Long Memory Processes J. Beran (1995) 62 Nonlinear Models for Repeated Measurement Data M. Davidian and D.M. Giltinan (1995) 63 Measurement Error in Nonlinear Models R.J. Carroll, D. Rupert and L.A. Stefanski (1995) 64 Analyzing and Modeling Rank Data J.J. Marden (1995) 65 Time Series Models—In Econometrics, Finance and Other Fields D.R. Cox, D.V. Hinkley and O.E. Barndorff-Nielsen (1996) 66 Local Polynomial Modeling and its Applications J. Fan and I. Gijbels (1996) 67 Multivariate Dependencies—Models, Analysis and Interpretation D.R. Cox and N. Wermuth (1996) 68 Statistical Inference—Based on the Likelihood A. Azzalini (1996) 69 Bayes and Empirical Bayes Methods for Data Analysis B.P. Carlin and T.A Louis (1996) 70 Hidden Markov and Other Models for Discrete-Valued Time Series I.L. MacDonald and W. Zucchini (1997) 71 Statistical Evidence—A Likelihood Paradigm R. Royall (1997) 72 Analysis of Incomplete Multivariate Data J.L. Schafer (1997) 73 Multivariate Models and Dependence Concepts H. Joe (1997) 74 Theory of Sample Surveys M.E. Thompson (1997) 75 Retrial Queues G. Falin and J.G.C. Templeton (1997) 76 Theory of Dispersion Models B. Jørgensen (1997) 77 Mixed Poisson Processes J. Grandell (1997) 78 Variance Components Estimation—Mixed Models, Methodologies and Applications P.S.R.S. Rao (1997) 79 Bayesian Methods for Finite Population Sampling G. Meeden and M. Ghosh (1997) 80 Stochastic Geometry—Likelihood and computation O.E. Barndorff-Nielsen, W.S. Kendall and M.N.M. van Lieshout (1998) 81 Computer-Assisted Analysis of Mixtures and Applications— Meta-analysis, Disease Mapping and Others D. Böhning (1999) 82 Classification, 2nd edition A.D. Gordon (1999) 83 Semimartingales and their Statistical Inference B.L.S. Prakasa Rao (1999) 84 Statistical Aspects of BSE and vCJD—Models for Epidemics C.A. Donnelly and N.M. Ferguson (1999) 85 Set-Indexed Martingales G. Ivanoff and E. Merzbach (2000) 86 The Theory of the Design of Experiments D.R. Cox and N. Reid (2000) 87 Complex Stochastic Systems O.E. Barndorff-Nielsen, D.R. Cox and C. Klüppelberg (2001) 88 Multidimensional Scaling, 2nd edition T.F. Cox and M.A.A. Cox (2001)

C7755_FM.indd 3

4/29/11 2:13 PM

89 Algebraic Statistics—Computational Commutative Algebra in Statistics G. Pistone, E. Riccomagno and H.P. Wynn (2001) 90 Analysis of Time Series Structure—SSA and Related Techniques N. Golyandina, V. Nekrutkin and A.A. Zhigljavsky (2001) 91 Subjective Probability Models for Lifetimes Fabio Spizzichino (2001) 92 Empirical Likelihood Art B. Owen (2001) 93 Statistics in the 21st Century Adrian E. Raftery, Martin A. Tanner, and Martin T. Wells (2001) 94 Accelerated Life Models: Modeling and Statistical Analysis Vilijandas Bagdonavicius and Mikhail Nikulin (2001) 95 Subset Selection in Regression, Second Edition Alan Miller (2002) 96 Topics in Modelling of Clustered Data Marc Aerts, Helena Geys, Geert Molenberghs, and Louise M. Ryan (2002) 97 Components of Variance D.R. Cox and P.J. Solomon (2002) 98 Design and Analysis of Cross-Over Trials, 2nd Edition Byron Jones and Michael G. Kenward (2003) 99 Extreme Values in Finance, Telecommunications, and the Environment Bärbel Finkenstädt and Holger Rootzén (2003) 100 Statistical Inference and Simulation for Spatial Point Processes Jesper Møller and Rasmus Plenge Waagepetersen (2004) 101 Hierarchical Modeling and Analysis for Spatial Data Sudipto Banerjee, Bradley P. Carlin, and Alan E. Gelfand (2004) 102 Diagnostic Checks in Time Series Wai Keung Li (2004) 103 Stereology for Statisticians Adrian Baddeley and Eva B. Vedel Jensen (2004) 104 Gaussian Markov Random Fields: Theory and Applications H˚avard Rue and Leonhard Held (2005) 105 Measurement Error in Nonlinear Models: A Modern Perspective, Second Edition Raymond J. Carroll, David Ruppert, Leonard A. Stefanski, and Ciprian M. Crainiceanu (2006) 106 Generalized Linear Models with Random Effects: Unified Analysis via H-likelihood Youngjo Lee, John A. Nelder, and Yudi Pawitan (2006) 107 Statistical Methods for Spatio-Temporal Systems Bärbel Finkenstädt, Leonhard Held, and Valerie Isham (2007) 108 Nonlinear Time Series: Semiparametric and Nonparametric Methods Jiti Gao (2007) 109 Missing Data in Longitudinal Studies: Strategies for Bayesian Modeling and Sensitivity Analysis Michael J. Daniels and Joseph W. Hogan (2008) 110 Hidden Markov Models for Time Series: An Introduction Using R Walter Zucchini and Iain L. MacDonald (2009) 111 ROC Curves for Continuous Data Wojtek J. Krzanowski and David J. Hand (2009) 112 Antedependence Models for Longitudinal Data Dale L. Zimmerman and Vicente A. Núñez-Antón (2009) 113 Mixed Effects Models for Complex Data Lang Wu (2010) 114 Intoduction to Time Series Modeling Genshiro Kitagawa (2010) 115 Expansions and Asymptotics for Statistics Christopher G. Small (2010) 116 Statistical Inference: An Integrated Bayesian/Likelihood Approach Murray Aitkin (2010) 117 Circular and Linear Regression: Fitting Circles and Lines by Least Squares Nikolai Chernov (2010) 118 Simultaneous Inference in Regression Wei Liu (2010) 119 Robust Nonparametric Statistical Methods, Second Edition Thomas P. Hettmansperger and Joseph W. McKean (2011) 120 Statistical Inference: The Minimum Distance Approach Ayanendranath Basu, Hiroyuki Shioya, and Chanseok Park (2011) 121 Smoothing Splines : Methods and Applications Yuedong Wang (2011)

Monographs on Statistics and Applied Probability 121


Yuedong Wang University of California Santa Barbara, California, USA

CRC Press Taylor & Francis Group 6000 Broken Sound Parkway NW, Suite 300 Boca Raton, FL 33487-2742 © 2011 by Taylor & Francis Group, LLC CRC Press is an imprint of Taylor & Francis Group, an Informa business No claim to original U.S. Government works Version Date: 20110429 International Standard Book Number-13: 978-1-4200-7756-8 (eBook - PDF) This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint. Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers. For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com and the CRC Press Web site at http://www.crcpress.com

TO YAN, CATHERINE, AND KEVIN

This page intentionally left blank

Contents

1 Introduction 1.1 Parametric and Nonparametric Regression . . 1.2 Polynomial Splines . . . . . . . . . . . . . . . . 1.3 Scope of This Book . . . . . . . . . . . . . . . 1.4 The assist Package . . . . . . . . . . . . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

1 1 4 7 9

2 Smoothing Spline Regression 2.1 Reproducing Kernel Hilbert Space . . . . . . 2.2 Model Space for Polynomial Splines . . . . . 2.3 General Smoothing Spline Regression Models 2.4 Penalized Least Squares Estimation . . . . . 2.5 The ssr Function . . . . . . . . . . . . . . . 2.6 Another Construction for Polynomial Splines 2.7 Periodic Splines . . . . . . . . . . . . . . . . 2.8 Thin-Plate Splines . . . . . . . . . . . . . . . 2.9 Spherical Splines . . . . . . . . . . . . . . . . 2.10 Partial Splines . . . . . . . . . . . . . . . . . 2.11 L-splines . . . . . . . . . . . . . . . . . . . . 2.11.1 Motivation . . . . . . . . . . . . . . . 2.11.2 Exponential Spline . . . . . . . . . . . 2.11.3 Logistic Spline . . . . . . . . . . . . . 2.11.4 Linear-Periodic Spline . . . . . . . . . 2.11.5 Trigonometric Spline . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

11 11 14 16 17 20 22 24 26 29 30 39 39 41 44 46 48

3 Smoothing Parameter Selection and Inference 3.1 Impact of the Smoothing Parameter . . . . . . . . 3.2 Trade-Offs . . . . . . . . . . . . . . . . . . . . . . 3.3 Unbiased Risk . . . . . . . . . . . . . . . . . . . . 3.4 Cross-Validation and Generalized Cross-Validation 3.5 Bayes and Linear Mixed-Effects Models . . . . . . 3.6 Generalized Maximum Likelihood . . . . . . . . . 3.7 Comparison and Implementation . . . . . . . . . . 3.8 Confidence Intervals . . . . . . . . . . . . . . . . . 3.8.1 Bayesian Confidence Intervals . . . . . . . . 3.8.2 Bootstrap Confidence Intervals . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

53 53 57 62 64 67 71 72 75 75 81

. . . . . . . . . . . . . . . .

ix

x 3.9

Hypothesis Tests . . . . . . . . . . . . . . . . 3.9.1 The Hypothesis . . . . . . . . . . . . . 3.9.2 Locally Most Powerful Test . . . . . . 3.9.3 Generalized Maximum Likelihood Test 3.9.4 Generalized Cross-Validation Test . . 3.9.5 Comparison and Implementation . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

84 84 85 86 87 87

4 Smoothing Spline ANOVA 4.1 Multiple Regression . . . . . . . . . . . . . . . . . . . 4.2 Tensor Product Reproducing Kernel Hilbert Spaces . 4.3 One-Way SS ANOVA Decomposition . . . . . . . . . 4.3.1 Decomposition of Ra : One-Way ANOVA . . . 4.3.2 Decomposition of W2m [a, b] . . . . . . . . . . . 4.3.3 Decomposition of W2m (per) . . . . . . . . . . . 4.3.4 Decomposition of W2m (Rd ) . . . . . . . . . . . 4.4 Two-Way SS ANOVA Decomposition . . . . . . . . . 4.4.1 Decomposition of Ra ⊗ Rb : Two-Way ANOVA 4.4.2 Decomposition of Ra ⊗ W2m [0, 1] . . . . . . . . 4.4.3 Decomposition of W2m1 [0, 1] ⊗ W2m2 [0, 1] . . . . 4.4.4 Decomposition of Ra ⊗ W2m (per) . . . . . . . . 4.4.5 Decomposition of W2m1 (per) ⊗ W2m2 [0, 1] . . . . 4.4.6 Decomposition of W22 (R2 ) ⊗ W2m (per) . . . . . 4.5 General SS ANOVA Decomposition . . . . . . . . . . 4.6 SS ANOVA Models and Estimation . . . . . . . . . . 4.7 Selection of Smoothing Parameters . . . . . . . . . . 4.8 Confidence Intervals . . . . . . . . . . . . . . . . . . . 4.9 Examples . . . . . . . . . . . . . . . . . . . . . . . . . 4.9.1 Tongue Shapes . . . . . . . . . . . . . . . . . . 4.9.2 Ozone in Arosa — Revisit . . . . . . . . . . . . 4.9.3 Canadian Weather — Revisit . . . . . . . . . . 4.9.4 Texas Weather . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

91 91 92 93 95 96 97 97 98 99 100 103 106 107 108 110 111 114 116 117 117 126 131 133

5 Spline Smoothing with Heteroscedastic and/or Correlated Errors 139 5.1 Problems with Heteroscedasticity and Correlation . . . 139 5.2 Extended SS ANOVA Models . . . . . . . . . . . . . . 142 5.2.1 Penalized Weighted Least Squares . . . . . . . . 142 5.2.2 UBR, GCV and GML Criteria . . . . . . . . . . 144 5.2.3 Known Covariance . . . . . . . . . . . . . . . . . 147 5.2.4 Unknown Covariance . . . . . . . . . . . . . . . . 148 5.2.5 Confidence Intervals . . . . . . . . . . . . . . . . 150 5.3 Variance and Correlation Structures . . . . . . . . . . . 150 5.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 153

xi 5.4.1 5.4.2 5.4.3 5.4.4

Simulated Motorcycle Accident — Revisit Ozone in Arosa — Revisit . . . . . . . . . Beveridge Wheat Price Index . . . . . . . Lake Acidity . . . . . . . . . . . . . . . .

. . . .

. . . .

. . . .

. . . .

153 154 157 158

6 Generalized Smoothing Spline ANOVA 163 6.1 Generalized SS ANOVA Models . . . . . . . . . . . . . 163 6.2 Estimation and Inference . . . . . . . . . . . . . . . . . 164 6.2.1 Penalized Likelihood Estimation . . . . . . . . . 164 6.2.2 Selection of Smoothing Parameters . . . . . . . . 167 6.2.3 Algorithm and Implementation . . . . . . . . . . 168 6.2.4 Bayes Model, Direct GML and Approximate Bayesian Confidence Intervals . . . . . . . . . . . 170 6.3 Wisconsin Epidemiological Study of Diabetic Retinopathy . . . . . . . . . . . . . . . . . . . . . . . . 172 6.4 Smoothing Spline Estimation of Variance Functions . . 176 6.5 Smoothing Spline Spectral Analysis . . . . . . . . . . . 182 6.5.1 Spectrum Estimation of a Stationary Process . . 182 6.5.2 Time-Varying Spectrum Estimation of a Locally Stationary Process . . . . . . . . . . . . . . . . . 183 6.5.3 Epileptic EEG . . . . . . . . . . . . . . . . . . . 185 7 Smoothing Spline Nonlinear Regression 195 7.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . 195 7.2 Nonparametric Nonlinear Regression Models . . . . . . 196 7.3 Estimation with a Single Function . . . . . . . . . . . . 197 7.3.1 Gauss–Newton and Newton–Raphson Methods . 197 7.3.2 Extended Gauss–Newton Method . . . . . . . . . 199 7.3.3 Smoothing Parameter Selection and Inference . . 201 7.4 Estimation with Multiple Functions . . . . . . . . . . . 204 7.5 The nnr Function . . . . . . . . . . . . . . . . . . . . . 205 7.6 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 206 7.6.1 Nonparametric Regression Subject to Positive Constraint . . . . . . . . . . . . . . . . . . . . . . 206 7.6.2 Nonparametric Regression Subject to Monotone Constraint . . . . . . . . . . . . . . . . . . . . . . 207 7.6.3 Term Structure of Interest Rates . . . . . . . . . 212 7.6.4 A Multiplicative Model for Chickenpox Epidemic 218 7.6.5 A Multiplicative Model for Texas Weather . . . . 223

xii 8 Semiparametric Regression 8.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . 8.2 Semiparametric Linear Regression Models . . . . . . 8.2.1 The Model . . . . . . . . . . . . . . . . . . . 8.2.2 Estimation and Inference . . . . . . . . . . . 8.2.3 Vector Spline . . . . . . . . . . . . . . . . . . 8.3 Semiparametric Nonlinear Regression Models . . . . 8.3.1 The Model . . . . . . . . . . . . . . . . . . . 8.3.2 SNR Models for Clustered Data . . . . . . . 8.3.3 Estimation and Inference . . . . . . . . . . . 8.3.4 The snr Function . . . . . . . . . . . . . . . 8.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . 8.4.1 Canadian Weather — Revisit . . . . . . . . . 8.4.2 Superconductivity Magnetization Modeling . 8.4.3 Oil-Bearing Rocks . . . . . . . . . . . . . . . 8.4.4 Air Quality . . . . . . . . . . . . . . . . . . . 8.4.5 The Evolution of the Mira Variable R Hydrae 8.4.6 Circadian Rhythm . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

227 227 228 228 229 233 240 240 241 242 245 247 247 254 257 259 262 267

9 Semiparametric Mixed-Effects Models 9.1 Linear Mixed-Effects Models . . . . . . . . . . . 9.2 Semiparametric Linear Mixed-Effects Models . . 9.2.1 The Model . . . . . . . . . . . . . . . . . 9.2.2 Estimation and Inference . . . . . . . . . 9.2.3 The slm Function . . . . . . . . . . . . . 9.2.4 SS ANOVA Decomposition . . . . . . . . 9.3 Semiparametric Nonlinear Mixed-Effects Models 9.3.1 The Model . . . . . . . . . . . . . . . . . 9.3.2 Estimation and Inference . . . . . . . . . 9.3.3 Implementation and the snm Function . . 9.4 Examples . . . . . . . . . . . . . . . . . . . . . . 9.4.1 Ozone in Arosa — Revisit . . . . . . . . . 9.4.2 Lake Acidity — Revisit . . . . . . . . . . 9.4.3 Coronary Sinus Potassium in Dogs . . . . 9.4.4 Carbon Dioxide Uptake . . . . . . . . . . 9.4.5 Circadian Rhythm — Revisit . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

273 273 274 274 275 279 280 283 283 284 286 288 288 291 294 305 310

A Data Sets A.1 Air Quality Data . . . . . . . . . . A.2 Arosa Ozone Data . . . . . . . . . A.3 Beveridge Wheat Price Index Data A.4 Bond Data . . . . . . . . . . . . . A.5 Canadian Weather Data . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

323 324 324 324 324 325

. . . . . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

xiii A.6 Carbon Dioxide Data . . . A.7 Chickenpox Data . . . . . A.8 Child Growth Data . . . . A.9 Dog Data . . . . . . . . . . A.10 Geyser Data . . . . . . . . A.11 Hormone Data . . . . . . . A.12 Lake Acidity Data . . . . . A.13 Melanoma Data . . . . . . A.14 Motorcycle Data . . . . . . A.15 Paramecium caudatum Data A.16 Rock Data . . . . . . . . . A.17 Seizure Data . . . . . . . . A.18 Star Data . . . . . . . . . . A.19 Stratford Weather Data . . A.20 Superconductivity Data . . A.21 Texas Weather Data . . . . A.22 Ultrasound Data . . . . . . A.23 USA Climate Data . . . . A.24 Weight Loss Data . . . . . A.25 WESDR Data . . . . . . . A.26 World Climate Data . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . .

325 325 326 326 326 327 327 327 328 328 328 328 329 329 329 330 330 331 331 331 332

B Codes for Fitting Strictly Increasing Functions 333 B.1 C and R Codes for Computing Integrals . . . . . . . . . 333 B.2 R Function inc . . . . . . . . . . . . . . . . . . . . . . 336 C Codes for Term Structure of Interest Rates 339 C.1 C and R Codes for Computing Integrals . . . . . . . . . 339 C.2 R Function for One Bond . . . . . . . . . . . . . . . . . 341 C.3 R Function for Two Bonds . . . . . . . . . . . . . . . . 342 References

347

Author Index

355

Subject Index

359


List of Tables

2.1 2.2 5.1 5.2

Bases of null spaces and RKs for linear and cubic splines under the construction in Section 2.2 with X = [0, b] . . Bases of null spaces and RKs for linear and cubic splines under the construction in Section 2.6 with X = [0, 1] . .

21 23

Standard varFunc classes . . . . . . . . . . . . . . . . . Standard corStruct classes for serial correlation structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Standard corStruct classes for spatial correlation structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

151

A.1 List of all data sets . . . . . . . . . . . . . . . . . . . . .

323

5.3

152 153

xv


List of Figures

1.1 1.2 1.3 1.4 1.5

Geyser data, observations, the straight line fit, and residuals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Motorcycle data, observations, and a polynomial fit . . . Geyser data, residuals, and the cubic spline fits . . . . . Motorcycle data, observations, and the cubic spline fit . Relationship between functions in the assist package and some of the existing R functions . . . . . . . . . . . . .

2.1 2.2 2.3 2.4 2.5

Motorcycle data, the linear, and cubic spline fits . . . . Arosa data, observations, and the periodic spline fits . . USA climate data, the thin-plate spline fit . . . . . . . . World climate data, the spherical spline fit . . . . . . . . Geyser data, the partial spline fit, residuals, and the AIC and GCV scores . . . . . . . . . . . . . . . . . . . . . . 2.6 Motorcycle data, the partial spline fit, and the AIC and GCV scores . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Arosa data, the partial spline estimates of the month and year effects . . . . . . . . . . . . . . . . . . . . . . . . . 2.8 Canadian weather data, estimate of the weight function, and confidence intervals . . . . . . . . . . . . . . . . . . 2.9 Weight loss data, observations and the nonlinear regression, cubic spline, and exponential spline fits . . . . . . 2.10 Paramecium caudatum data, observations and the nonlinear regression, cubic spline, and logistic spline fits . . 2.11 Melanoma data, observations, and the cubic spline and linear-periodic spline fits . . . . . . . . . . . . . . . . . . 2.12 Arosa data, the overall fits and their projections . . . . 3.1 3.2 3.3 3.4 3.5

Stratford weather data, observations, and the periodic spline fits with different smoothing parameters . . . . . Weights of the periodic spline filter . . . . . . . . . . . . Stratford data, degrees of freedom, and residual sum of squares . . . . . . . . . . . . . . . . . . . . . . . . . . . Squared bias, variance, and MSE from a simulation . . . PSE and UBR functions . . . . . . . . . . . . . . . . . .

2 2 3 4 10 22 25 28 31 33 34 36 38 43 45 48 51 54 57 59 61 65

xvii

xviii 3.6 3.7 3.8 3.9 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 4.10 4.11 4.12 4.13 4.14 4.15 4.16 5.1 5.2 5.3 5.4 5.5 5.6 5.7

PSE, CV, and GCV functions . . . . . . . . . . . . . . . Geyser data, estimates of the smooth components in the cubic and partial spline models . . . . . . . . . . . . . . Motorcycle data, partial spline fit, and t-statistics . . . . Pointwise coverages and across-the-function coverages . Ultrasound data, 3-d plots of observations . . . . . . . . Ultrasound data, observations, fits, confidence intervals, and the mean curves among three environments . . . . . Ultrasound data, the overall interaction . . . . . . . . . Ultrasound data, effects of environment . . . . . . . . . Ultrasound data, estimated tongue shapes as functions of length and time . . . . . . . . . . . . . . . . . . . . . . Ultrasound data, the estimated time effect . . . . . . . . Ultrasound data, estimated tongue shape as a function of environment, length and time . . . . . . . . . . . . . . Ultrasound data, the estimated environment effect . . . Arosa data, estimates of the interactions and smooth component . . . . . . . . . . . . . . . . . . . . . . . . . . . . Arosa data, estimates of the main effects . . . . . . . . . Canadian weather data, temperature profiles of stations in four regions and the estimated profiles . . . . . . . . Canadian weather data, the estimated region effects to temperature . . . . . . . . . . . . . . . . . . . . . . . . . Texas weather data, observations as curves . . . . . . . Texas weather data, observations as surfaces . . . . . . . Texas weather data, the location effects for four selected stations . . . . . . . . . . . . . . . . . . . . . . . . . . . Texas weather data, the month effects for January, April, July, and October . . . . . . . . . . . . . . . . . . . . . WMSEs and coverages of Bayesian confidence intervals with the presence of heteroscedasticity . . . . . . . . . . Cubic spline fits when data are correlated . . . . . . . . Cubic spline fits and estimated autocorrelation functions for two simulations . . . . . . . . . . . . . . . . . . . . . Motorcycle data, estimates of the mean and variance functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Arosa data, residuals variances and PWLS fit . . . . . . Beveridge data, time series and cubic spline fits . . . . . Lake acidity data, effects of calcium and geological location . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

68 79 80 83 93 118 119 120 122 123 125 126 128 129 132 134 135 135 137 138 140 141 149 154 155 157 160

xix 6.1 6.2 6.3 6.4 6.5 6.6 6.7

WESDR data, the estimated probability functions . . . Motorcycle data, estimates of the variance function based on three procedures . . . . . . . . . . . . . . . . . . . . Motorcycle data, DGML function, and estimate of the variance and mean functions . . . . . . . . . . . . . . . . Seizure data, the baseline and preseizure IEEG segments Seizure data, periodograms, estimates of the spectra based on the iterative UBR method and confidence intervals . Seizure data, estimates of the time-varying spectra based on the iterative UBR method . . . . . . . . . . . . . . . Seizure data, estimates of the time-varying spectra based on the DGML method . . . . . . . . . . . . . . . . . . .

7.1 7.2 7.3

Nonparametric regression under positivity constraint . . Nonparametric regression under monotonicity constraint Child growth data, cubic spline fit, fit under monotonicity constraint and estimate of the velocity . . . . . . . . . . 7.4 Bond data, unconstrained and constrained estimates of the discount functions, forward rates and credit spread . 7.5 Chickenpox data, time series plot and the fits by multiplicative and SS ANOVA models . . . . . . . . . . . . . 7.6 Chickenpox data, estimates of the mean and amplitude functions in the multiplicative model . . . . . . . . . . . 7.7 Chickenpox data, estimates of the shape function in the multiplicative model and its projections . . . . . . . . . 7.8 Texas weather data, estimates of the mean and amplitude functions in the multiplicative model . . . . . . . . . . . 7.9 Texas weather data, temperature profiles . . . . . . . . . 7.10 Texas weather data, the estimated interaction against the estimated main effect for two stations . . . . . . . . . . 8.1 8.2 8.3 8.4 8.5 8.6

Separate and joint fits from a simulation . . . . . . . . . Estimates of the differences . . . . . . . . . . . . . . . . Canadian weather data, estimated region effects to precipitation . . . . . . . . . . . . . . . . . . . . . . . . . . Canadian weather data, estimate of the coefficient function for the temperature effect . . . . . . . . . . . . . . Canadian weather data, estimates of the intercept and weight functions . . . . . . . . . . . . . . . . . . . . . . Superconductivity data, observations, and the fits by nonlinear regression, cubic spline, nonlinear partial spline, and L-spline . . . . . . . . . . . . . . . . . . . . . . . . .

176 178 182 186 187 189 192 207 210 210 214 219 222 222 224 225 226 236 238 249 249 253

254

xx 8.7 8.8 8.9 8.10 8.11 8.12 8.13 8.14 9.1 9.2 9.3 9.4 9.5 9.6 9.7 9.8 9.9 9.10 9.11 9.12

9.13 9.14

Superconductivity data, estimates of departures from the straight line model and the “interpolation formula” . . . 256 Rock data, estimates of functions in the projection pursuit regression model . . . . . . . . . . . . . . . . . . . . . . 259 Air quality data, estimates of functions in SNR models . 261 Star data, observations, and the overall fit . . . . . . . . 262 Star data, folded observations, estimates of the common shape function and its projection . . . . . . . . . . . . . 264 Star data, estimates of the amplitude and period functions 266 Hormone data, cortisol concentrations for normal subjects and the fits based on an SIM . . . . . . . . . . . . . . . 268 Hormone data, estimate of the common shape function in an SIM and its projection for normal subjects . . . . . . 270 Arosa data, the overall fit, seasonal trend, and long-term trend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Arosa data, the overall fit, seasonal trend, long-term trend and local stochastic trend . . . . . . . . . . . . . . . . . Lake acidity data, effects of calcium and geological location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dog data, coronary sinus potassium concentrations over time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dog data, estimates of the group mean response curves . Dog data, estimates of the group mean response curves under new penalty . . . . . . . . . . . . . . . . . . . . . Dog data, predictions for four dogs . . . . . . . . . . . . Carbon dioxide data, observations and fits by the NLME and SNM models . . . . . . . . . . . . . . . . . . . . . . Carbon dioxide data, overall estimate and projections of the nonparametric shape function . . . . . . . . . . . . . Hormone data, cortisol concentrations for normal subjects, and the fits based on a mixed-effects SIM . . . . . Hormone data, cortisol concentrations for depressed subjects, and the fits based on a mixed-effects SIM . . . . . Hormone data, cortisol concentrations for subjects with Cushing’s disease, and the fits based on a mixed-effects SIM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hormone data, estimates of the common shape functions in the mixed-effect SIM . . . . . . . . . . . . . . . . . . Hormone data, plot of the estimated 24-hour mean levels against amplitudes . . . . . . . . . . . . . . . . . . . . .

290 292 294 295 302 304 305 307 309 312 313

314 315 321

xxi

Symbol Description (x)+ x∧z x∨z det+ kr (x) (·, ·) k · k: X S H L N P A M R(x, z) Rd N S 2m (t1 , · · · , tk ) W2m [a, b] W2m (per) W2m (Rd ) W2m (S) ⊕ ⊗

max{x, 0} min{x, z} max{x, z} Product of the nonzero eigenvalues Scaled Bernoulli polynomials Inner product Norm Domain of a function Unit sphere Function space Linear functional Nonlinear functional Projection Averaging operator Model space Reproducing kernel Euclidean d-space Natural polynomial spline space Sobolev space on [a, b] Sobolev space on unit circle Thin-plate spline model space Sobolev space on unit sphere Direct sum of function spaces Tensor product of function spaces


Preface

Statistical analysis often involves building mathematical models that examine the relationship between dependent and independent variables. This book is about a general class of powerful and flexible modeling techniques, namely, spline smoothing. Research on smoothing spline models has attracted a great deal of attention in recent years, and the methodology has been widely used in many areas. This book provides an introduction to some basic smoothing spline models, including polynomial, periodic, spherical, thin-plate, L-, and partial splines, as well as an overview of more advanced models, including smoothing spline ANOVA, extended and generalized smoothing spline ANOVA, vector spline, nonparametric nonlinear regression, semiparametric regression, and semiparametric mixed-effects models. Methods for model selection and inference are also presented. The general forms of nonparametric/semiparametric linear/nonlinear fixed/mixed smoothing spline models in this book provide unified frameworks for estimation, inference, and software implementation. This book draws on the theory of reproducing kernel Hilbert space (RKHS) to present various smoothing spline models in a unified fashion. On the other hand, the subject of smoothing spline in the context of RKHS and regularization is often regarded as technical and difficult. One of my main goals is to make the advanced smoothing spline methodology based on RKHS more accessible to practitioners and students. With this in mind, the book focuses on methodology, computation, implementation, software, and application. It provides a gentle introduction to the RKHS, keeps theory at the minimum level, and provides details on how the RKHS can be used to construct spline models. User-friendly software is key to the routine use of any statistical method. The assist library in R implements methods presented in this book for fitting various nonparametric/semiparametric linear/nonlinear fixed/mixed smoothing spline models. The assist library can be obtained at http://www.r-project.org Much of the exposition is based on the analysis of real examples. Rather than formal analysis, these examples are intended to illustrate the power and versatility of the spline smoothing methodology. All data analyses are performed in R, and most of them use functions in the

xxiii

xxiv assist library. Codes for all examples and further developments related to this book will be posted on the web page http://www.pstat.ucsb.edu/faculty/yuedong/book.html This book is intended for those wanting to learn about smoothing splines. It can be a reference book for statisticians and scientists who need advanced and flexible modeling techniques. It can also serve as a text for an advanced-level graduate course on the subject. In fact, topics in Chapters 1–4 were covered in a quarter class at the University of California — Santa Barbara, and the University of Science and Technology of China. I was fortunate indeed to have learned the smoothing spline from Grace Wahba, whose pioneering work has paved the way for much ongoing research and made this book possible. I am grateful to Chunlei Ke, my former student and collaborator, for developing the assist package. Special thanks goes to Anna Liu for reading the draft carefully and correcting many mistakes. Several people have helped me over various phases of writing this book: Chong Gu, Wensheng Guo, David Hinkley, Ping Ma, and Wendy Meiring. I must thank my editor, David Grubbes, for his patience and encouragement. Finally, I would like to thank several researchers who kindly shared their data sets for inclusion in this book; they are cited where their data are introduced. Yuedong Wang Santa Barbara December 2010

Chapter 1 Introduction

1.1

Parametric and Nonparametric Regression

Regression analysis builds mathematical models that examine the relationship of a dependent variable to one or more independent variables. These models may be used to predict responses at unobserved and/or future values of the independent variables. In the simple case when both the dependent variable y and the independent variable x are scalar variables, given observations (xi , yi ) for i = 1, . . . , n, a regression model relates dependent and independent variables as follows: yi = f (xi ) + ǫi ,

i = 1, . . . , n,

(1.1)

where f is the regression function and ǫi are zero-mean independent random errors with a common variance σ 2 . The goal of regression analysis is to construct a model for f and estimate it based on noisy data. For example, for the Old Faithful geyser in Yellowstone National Park, consider the problem of predicting the waiting time to the next eruption using the length of the previous eruption. Figure 1.1(a) shows the scatter plot of waiting time to the next eruption (y = waiting) against duration of the previous eruption (x = duration) for 272 observations from the Old Faithful geyser. The goal is to build a mathematical model that relates the waiting time to the duration of the previous eruption. A first attempt might be to approximate the regression function f by a straight line f (x) = β0 + β1 x. (1.2) The least squares straight line fit is shown in Figure 1.1(a). There is no apparent sign of lack-of-fit. Furthermore, there is no clear visible trend in the plot of residuals in Figure 1.1(b). Often f is nonlinear in x. A common approach to dealing with nonlinear relationship is to approximate f by a polynomial of order m f (x) = β0 + β1 x + · · · + βm−1 xm−1 .

(1.3)

1

2

Smoothing Splines: Methods and Applications (b)

−10

50

waiting (min) 60 70 80

residuals (min) 0 5 10

90

15

(a)

1.5

1.5

2.5 3.5 4.5 duration (min)


FIGURE 1.1 Geyser data, plots of (a) observations and the least squares straight line fit, and (b) residuals.

Figure 1.2 shows the scatter plot of acceleration (y = acceleration) against time after impact (x = time) from a simulated motorcycle crash experiment on the efficacy of crash helmets. It is clear that a straight line cannot explain the relationship between acceleration and time. Polynomials with m = 1, . . . , 20 are fitted to the data, and Figure 1.2 shows the best fit selected by Akaike’s information criterion (AIC). There are waves in the fitted curve at both ends of the range. The fit is still not completely satisfactory even when polynomials up to order 20 are considered. Unlike the linear regression model (1.2), except for small m, coefficients in model (1.3) no longer have nice interpretations.

acceleration (g) −100 −50 0 50

o

ooooo

ooo ooo oooooo ooo oo o o o o o ooooo o

oo o oooo o o o o oo o o o o oo o o o o o o o ooo o o o o

0

10

o o

o oo o o o o o o o oo o o o o o o o o oo o o o o o oo o o oo o o o o o o o o o o oo o o o o o o o oo o o o o o

20

o

o

30 time (ms)

40

50

60

FIGURE 1.2 Motorcycle data, plot of observations, and a polynomial fit.

Introduction

3

In general, a parametric regression model assumes that the form of f is known except for finitely many unknown parameters. The specific form of f may come from scientific theories and/or approximations to mechanics under some simplified assumptions. The assumptions may be too restrictive and the approximations may be too crude for some applications. An inappropriate model can lead to systematic bias and misleading conclusions. In practice, one should always check the assumed form for the function f . It is often difficult, if not impossible, to obtain a specific functional form for f . A nonparametric regression model does not assume a predetermined form. Instead, it makes assumptions on qualitative properties of f . For example, one may be willing to assume that f is “smooth”, which does not reduce to a specific form with finite number of parameters. Rather, it usually leads to some infinite dimensional collections of functions. The basic idea of nonparametric regression is to let the data speak for themselves. That is to let the data decide which function fits the best without imposing any specific form on f . Consequently, nonparametric methods are in general more flexible. They can uncover structure in the data that might otherwise be missed.

(b)

−10

50

waiting (min) 60 70 80

residuals (min) 0 5 10

90

15

(a)

1.5


1.5


FIGURE 1.3 Geyser data, plots of (a) residuals from the straight line fit and the cubic spline fit to the residuals, and (b) the cubic spline fit to the original data.

For illustration, we fit cubic splines to the geyser data. The cubic spline is a special nonparametric regression model that will be introduced in Section 1.2. A cubic spline fit to residuals from the linear model (1.2) reveals a nonzero trend in Figure 1.3(a). This raises the question of

4

Smoothing Splines: Methods and Applications

whether a simple linear regression model is appropriate for the geyser data. A cubic spline fit to the original data is shown in Figure 1.3(b). It reveals that there are two clusters in the independent variable, and a different linear model may be required for each cluster. Sections 2.10, 3.8, and 3.9 contain more analysis of the geyser data. A cubic spline fit to the motorcycle data is shown in Figure 1.4. It fits data much better than the polynomial model. Sections 2.10, 3.8, 5.4.1, and 6.4 contain more analysis of the motorcycle data.


o

ooooo

ooo ooo oooooo ooo oo o o o o o ooooo o

oo o oooo o o o o oo o o o o oo o o o o o o o ooo o o o o

0

10

o o

o oo o o o o o o o oo o o o o o o o o oo o o o o o oo o o oo o o o o oo o o o o o o o o o o o o o oo o o o o o

20

o

o

30 time (ms)

40

50

60

FIGURE 1.4 Motorcycle data, plot of observations, and the cubic spline fit.

The above simple exposition indicates that the nonparametric regression technique can be applied to different steps in regression analysis: data exploration, model building, testing parametric models, and diagnosis. In fact, as illustrated throughout the book, spline smoothing is a powerful and versatile tool for building statistical models to exploit structures in data.

1.2

Polynomial Splines

The polynomial (1.3) is a global model which makes it less adaptive to local variations. Individual observations can have undue influence on the fit in remote regions. For example, in the motorcycle data, the behavior of the mean function varies drastically from one region to another.

Introduction

5

These local variations led to oscillations at both ends of the range in the polynomial fit. A natural solution to overcome this limitation is to use piecewise polynomials, the basic idea behind polynomial splines. Let a < t1 < · · · < tk < b be fixed points called knots. Let t0 = a and tk+1 = b. Roughly speaking, polynomial splines are piecewise polynomials joined together smoothly at knots. Formally, a polynomial spline of order r is a real-valued function on [a, b], f (t), such that (i) f is a piecewise polynomial of order r on [ti , ti+1 ), i = 0, 1, . . . , k; (ii) f has r − 2 continuous derivatives and the (r − 1)st derivative is a step function with jumps at knots. Now consider even orders represented as r = 2m. The function f is a natural polynomial spline of order 2m if, in addition to (i) and (ii), it satisfies the natural boundary conditions (iii) f (j) (a) = f (j) (b) = 0, j = m, . . . , 2m − 1. The natural boundary conditions imply that f is a polynomial of order m on the two outside subintervals [a, t1 ] and [tk , b]. Denote the function space of natural polynomial splines of order 2m with knots t1 , . . . , tk as N S 2m (t1 , . . . , tk ). One approach, known as regression spline, is to approximate f using a polynomial spline or natural polynomial spline. To get a good approximation, one needs to decide the number and locations of knots. This book covers a different approach known as smoothing spline. It starts with a well-defined model space for f and introduces a penalty to prevent overfitting. We now describe this approach for polynomial splines. Consider the regression model (1.1). Suppose f is “smooth”. Specifically, assume that f ∈ W2m [a, b] where the Sobolev space W2m [a, b] = f : f, f ′ , . . . , f (m−1) are absolutely continuous, Z b (f (m) )2 dx < ∞ . (1.4) a

For any a ≤ x ≤ b, Taylor’s theorem states that f (x) =

m−1 X ν=0

|

Z x f (ν) (a) (x − u)m−1 (m) (x − a)ν + f (u)du . ν! (m − 1)! a | {z } {z }

polynomial of order m

(1.5)

Rem(x)

It is clear that the polynomial regression model (1.3) ignores the remainder term Rem(x) in the hope that it is negligible. It is often difficult

6


to verify this assumption in practice. The idea behind the spline smoothing is to let data decide how large Rem(x) should be. Since W2m [a, b] is an infinite dimensional space, a direct fit to f by minimizing the least squares (LS) n 1X (yi − f (xi ))2 (1.6) n i=1

leads to interpolation. Therefore, certain control over Rem(x) is necessary. One natural approach is to control how far f is allowed to depart from the polynomial model. Under appropriate norms defined later in Sections 2.2 and 2.6, one measure of distance between f and polynomials Rb is a (f (m) )2 dx. It is then reasonable to estimate f by minimizing the LS (1.6) under the constraint Z b (f (m) )2 dx ≤ ρ (1.7) a

for a constant ρ. By introducing a Lagrange multiplier, the constrained minimization problem (1.6) and (1.7) is equivalent to minimizing the penalized least squares (PLS): Z b n 1X (yi − f (xi ))2 + λ (f (m) )2 dx. (1.8) n i=1 a In the remainder of this book, a polynomial spline refers to the solution of the PLS (1.8) in the model space W2m [a, b]. A cubic spline is a special case of the polynomial spline with m = 2. Since it measures Rb the roughness of the function f , a (f (m) )2 dx is often referred to as a roughness penalty. It is obvious that there is no penalty for polynomials of order less than or equal to m. The smoothing parameter λ balances the trade-off between goodness-of-fit measured by the LS and roughness Rb of the estimate measured by a (f (m) )2 dx. Suppose that n ≥ m and a ≤ x1 < x2 < · · · < xn ≤ b. Then, for fixed 0 < λ < ∞, (1.8) has a unique minimizer fˆ and fˆ ∈ N S 2m (x1 , . . . , xn ) (Eubank 1988). This result indicates that even though we started with the infinite dimensional space W2m [a, b] as the model space for f , the solution to the PLS (1.8) belongs to a finite dimensional space. Specifically, the solution is a natural polynomial spline with knots at distinct design points. One approach to computing the polynomial spline estimate is to represent fˆ as a linear combination of a basis of N S 2m (x1 , . . . , xn ). Several basis constructions were provided in Section 3.3.3 of Eubank (1988). In particular, the R function smooth.spline implements this approach for the cubic spline using the B-spline basis. For example, the cubic spline fit in Figure 1.4 is derived by the following statements:

Introduction

7

> library(MASS); attach(mcycle) > smooth.spline(times, accel, all.knots=T) This book presents a different approach. Instead of basis functions, representers of reproducing kernel Hilbert spaces will be used to represent the spline estimate. This approach allows us to deal with many different spline models in a unified fashion. Details of this approach for polynomial splines will be presented in Sections 2.2 and 2.6. When λ = 0, there is no penalty, and the natural spline that interpolates observations is the unique minimizer. When λ = ∞, the unique minimizer is the mth order polynomial. As λ varies from ∞ to 0, we have a family of models ranging from the parametric polynomial model to interpolation. The value of λ decides how far f is allowed to depart from the polynomial model. Thus the choice of λ holds the key to the success of a spline estimate. We discuss how to choose λ based on data in Chapter 3.

1.3

Scope of This Book

Driven by many sophisticated applications and fueled by modern computing power, many flexible nonparametric and semiparametric modeling techniques have been developed to relax parametric assumptions and to exploit possible hidden structure. There are many different nonparametric methods. This book concentrates on one of them, smoothing spline. Existing books on this topic include Eubank (1988), Wahba (1990), Green and Silverman (1994), Eubank (1999), Gu (2002), and Ruppert, Wand and Carroll (2003). The goals of this book are to (a) make the advanced smoothing spline methodology based on reproducing kernel Hilbert spaces more accessible to practitioners and students; (b) provide software and examples so that the spline smoothing methods can be routinely used in practice; and (c) provide a comprehensive coverage of recently developed smoothing spline nonparametric/semiparametric linear/nonlinear fixed/mixed models. We concentrate on the methodology, implementation, software, and application. Theoretical results are stated without proofs. All methods will be demonstrated using real data sets and R functions. The polynomial spline in Section 1.2 concerns the functions defined on the domain [a, b]. In many applications, the domain of the regression function is not a continuous interval. Furthermore, the regression function may only be observed indirectly. Chapter 2 introduces gen-

8


eral smoothing spline regression models with reproducing kernel Hilbert spaces on general domains as model spaces. Penalized LS estimation, Kimeldorf–Wahba representer theorem, computation, and the R function ssr will be covered. Explicit constructions of model spaces will be discussed in detail for some popular smoothing spline models including polynomial, periodic, thin-plate, spherical, and L-splines. Chapter 3 introduces methods for selecting the smoothing parameter and making inferences about the regression function. The impact of the smoothing parameter and basic concepts for model selection will be discussed and illustrated using an example. Connections between smoothing spline models and Bayes/mixed-effects models will be established. The unbiased risk, generalized cross-validation, and generalized maximum likelihood methods will be introduced for selecting the smoothing parameter. Bayesian and bootstrap confidence intervals will be introduced for the regression function and its components. The locally most powerful, generalized maximum likelihood and generalized cross-validation tests will also be introduced to test the hypothesis of a parametric model versus a nonparametric alternative. Analogous to multiple regression, Chapter 4 constructs models for multivariate regression functions based on smoothing spline analysis of variance (ANOVA) decompositions. The resulting models have hierarchical structures that facilitate model selection and interpretation. Smoothing spline ANOVA decompositions for tensor products of some commonly used smoothing spline models will be illustrated. Penalized LS estimation involving multiple smoothing parameters and componentwise Bayesian confidence intervals will be covered. Chapter 5 presents spline smoothing methods for heterogeneous and correlated observations. Presence of heterogeneity and correlation may lead to wrong choice of the smoothing parameters and erroneous inference. Penalized weighted LS will be used for estimation. Unbiased risk, generalized cross-validation, and generalized maximum likelihood methods will be extended for selecting the smoothing parameters. Variance and correlation structures will also be discussed. Analogous to generalized linear models, Chapter 6 introduces smoothing spline ANOVA models for observations generated from a particular distribution in the exponential family including binomial, Poisson, and gamma distributions. Penalized likelihood will be used for estimation, and methods for selecting the smoothing parameters will be discussed. Nonparametric estimation of variance and spectral density functions will be presented. Analogous to nonlinear regression, Chapter 7 introduces spline smoothing methods for nonparametric nonlinear regression models where some unknown functions are observed indirectly through nonlinear function-

Introduction

9

als. In addition to fitting theoretical and empirical nonlinear nonparametric regression models, methods in this chapter may also be used to deal with constraints on the nonparametric function such as positivity or monotonicity. Several algorithms based on Gauss–Newton, Newton– Raphson, extended Gauss–Newton and Gauss–Seidel methods will be presented for different situations. Computation and the R function nnr will be covered. Chapter 8 introduces semiparametric regression models that involve both parameters and nonparametric functions. The mean function may depend on the parameters and the nonparametric functions linearly or nonlinearly. The semiparametric regression models include many wellknown models such as the partial spline, varying coefficients, projection pursuit, single index, multiple index, functional linear, and shape invariant models as special cases. Estimation, inference, computation, and the R function snr will also be covered. Chapter 9 introduces semiparametric linear and nonlinear mixedeffects models. Smoothing spline ANOVA decompositions are extended for the construction of semiparametric mixed-effects models that parallel the classical mixed models. Estimation and inference methods, computation, and the R functions slm and snm will be covered as well.

1.4

The assist Package

The assist package was developed for fitting various smoothing spline models covered in this book. It contains five main functions, ssr, nnr, snr, slm, and snm for fitting various smoothing spline models. The function ssr fits smoothing spline regression models in Chapter 2, smoothing spline ANOVA models in Chapter 4, extended smoothing spline ANOVA models with heterogeneous and correlated observations in Chapter 5, generalized smoothing spline ANOVA models in Chapter 6, and semiparametric linear regression models in Chapter 8, Section 8.2. The function nnr fits nonparametric nonlinear regression models in Chapter 7. The function snr fits semiparametric nonlinear regression models in Chapter 8, Section 8.3. The functions slm and snm fit semiparametric linear and nonlinear mixed-effects models in Chapter 9. The assist package is available at http://cran.r-project.org Figure 1.5 shows how the functions in assist generalize some of the existing R functions for regression analysis.

10


glm XXXX XXX

z gam X

: HH

X smooth.spline H XXX

3 Hj z H

: nnr lm Q JQ Q nls XXXX J s XXX J X z : nlme XX J XX z X ^ J X lme 1 XX XXX X z slm X

ssr snr

snm

FIGURE 1.5 Functions in assist (dashed boxes) and some existing R functions (solid boxes). An arrow represents an extension to a more general model. lm: linear models. glm: generalized linear models. smooth.spline: cubic spline models. nls: nonlinear regression models. lme: linear mixed-effects models. gam: generalized additive models. nlme: nonlinear mixed-effects models. ssr: smoothing spline regression models. nnr: nonparametric nonlinear regression models. snr: semiparametric nonlinear regression models. slm: semiparametric linear mixed-effects models. snm: semiparametric nonlinear mixed-effects models.

Chapter 2 Smoothing Spline Regression

2.1

Reproducing Kernel Hilbert Space

Polynomial splines concern functions defined on a continuous interval. This is the most common situation in practice. Nevertheless, many applications require modeling functions defined on domains other than a continuous interval. For example, for spatial data with measurements on latitude and longitude, the domain of the function is the Euclidean space R2 . Specific spline models were developed for different applications. It is desirable to develop methodology and software on a general platform such that special cases are dealt with in a unified fashion. Reproducing Kernel Hilbert Space (RKHS) provides such a general platform. This section provides a very brief review of RKHS. Throughout this book, important theoretical results are presented in italic without proofs. Details and proofs related to RKHS can be found in Aronszajn (1950), Wahba (1990), Gu (2002), and Berlinet and Thomas-Agnan (2004). A nonempty set E of elements f, g, h, . . . forms a linear space if there are two operations: (1) addition: a mapping (f, g) → f + g from E × E into E; and (2) multiplication: a mapping (α, f ) → αf from R × E into E, such that for any α, β ∈ R, the following conditions are satisfied: (a) f + g = g + f ; (b) (f + g) + h = f + (g + h); (c) for every f, g ∈ E, there exists h ∈ E such that f + h = g; (d) α(βf ) = (αβ)f ; (e) (α + β)f = αf + βf ; (f) α(f + g) = αf + αg; and (g) 1f = f . Property (c) implies that there exists a zero element, denoted as 0, such that f + 0 = f for all f ∈ E. A finite collection of elements f1 , . . . , fk in E is called linearly independent if the relation α1 f1 + · · ·+ αk fk = 0 holds only in the trivial case with α1 = · · · = αk = 0. An arbitrary collection of elements A is called linearly independent if every finite subcollection is linearly independent. Let A be a subset of a linear space E. Define spanA , {α1 f1 + · · · + αk fk : f1 , . . . , fk ∈ A, α1 , . . . , αk ∈ R, k = 1, 2, . . . }.

11

12


A set B ⊂ E is called a basis of E if B is linearly independent and spanB = E. A nonnegative function || · || on a linear space E is called a norm if (a) ||f || = 0 if and only if f = 0; (b) ||αf || = |α|||f ||; and (c) ||f + g|| ≤ ||f || + ||g||. If the function || · || satisfies (b) and (c) only, then it is called a seminorm. A linear space with a norm is called a normed linear space. Let E be a linear space. A mapping (·, ·) : E × E → R is called an inner product in E if it satisfies (a) (f, g) = (g, f ); (b) (αf + βg, h) = α(f, h) + β(g, h); and (c) (f, f ) ≥ 0, and (f, pf ) = 0 if and only if f = 0. An inner product defines a norm: ||f || , (f, f ). A linear space with an inner product is called an inner product space. Let E be a normed linear space and fn be a sequence in E. The sequence fn is said to converge to f ∈ E if limn→∞ ||fn − f || = 0, and f is called the limit point. The sequence fn is called a Cauchy sequence if liml,n→∞ ||fl − fn || = 0. The space E is complete if every Cauchy sequence converges to an element in E. A complete inner product space is called a Hilbert space. A functional L on a Hilbert space H is a mapping from H to R. L is a linear functional if it satisfies L(αf + βg) = αLf + βLg. L is said to be continuous if limn→∞ Lfn = Lf when limn→∞ fn = f . L is said to be bounded if there exists a constant M such that |Lf | ≤ M ||f || for all f ∈ H. L is continuous if and only if L is bounded. For every fixed h ∈ H, Lh f , (h, f ) defines a continuous linear functional. Conversely, every continuous linear functional L can be represented as an inner product with a representer. Riesz representation theorem Let L be a continuous linear functional on a Hilbert space H. There exists a unique hL such that Lf = (hL , f ) for all f ∈ H. The element hL is called the representer of L. Let H be a Hilbert space of real-valued functions from X to R where X is an arbitrary set. For a fixed x ∈ X , the evaluational functional Lx : H → R is defined as Lx f , f (x). Note that the evaluational functional Lx maps a function to a real value while the function f maps a point x to a real value. Lx applies to all functions in H with a fixed x. Evaluational functionals are linear since Lx (αf + βg) = αf (x) + βg(x) = αLx f + βLx g. Definition A Hilbert space of real-valued functions H is an RKHS if every evaluational functional is continuous.

Smoothing Spline Regression

13

Let H be an RKHS. Then, for each x ∈ X , the evaluational functional Lx f = f (x) is continuous. By the Riesz representation theorem, there exists an element Rx in H such that Lx f = f (x) = (Rx , f ), where the dependence of the representer on x is expressed explicitly as Rx . Consider Rx (z) as a bivariate function of x and z and let R(x, z) , Rx (z). The bivariate function R(x, z) is called the reproducing kernel (RK) of an RKHS H. The term reproducing kernel comes from the fact that (Rx , Rz ) = R(x, z). It is easy to check that an RK is nonnegative definite. That is, R is symmetric R(x, z) = R(z, x), and for any α1 , . . . , αn ∈ R and x1 , . . . , xn ∈ X , n X

i,j=1

αi αj R(xi , xj ) ≥ 0.

Therefore, every RKHS has a unique RK that is nonnegative definite. Conversely, an RKHS can be constructed based on a nonnegative definite function. Moore–Aronszajn theorem For every nonnegative definite function R on X ×X , there exists a unique RKHS on X with R as its RK. The above results indicate that there exists an one-to-one correspondence between RKHS’s and nonnegative definite functions. For a finite dimensional space H with an orthonormal basis φ1 (x), . . . , φp (x), it is easy to see that p X R(x, z) , φi (x)φi (z) i=1

is the RK of H. The following definitions and results are useful for the construction and decomposition of model spaces. S is called a subspace of a Hilbert space H if S ⊂ H and αf + βg ∈ S for every α, β ∈ R and f, g ∈ S. A closed subspace S is a Hilbert space. The orthogonal complement of S is defined as S ⊥ , {f ∈ H : (f, g) = 0 for all g ∈ S}.

S ⊥ is a closed subspace of H. If S is a closed subspace of a Hilbert space H, then every element f ∈ H has a unique decomposition in the form f = g + h, where g ∈ S and h ∈ S ⊥ . Equivalently, H is decomposed

14


into two subspaces H = S ⊕ S ⊥ . This decomposition is called a tensor sum decomposition, and elements g and h are called projections onto S and S ⊥ , respectively. Sometimes the notation H ⊖ S will be used to denote the subspaces S ⊥ . Tensor sum decomposition with more than two subspaces can be defined recursively. All closed subspaces of an RKHS are RKHS’s. If H = H0 ⊕ H1 and R, R0 , and R1 are RKs of H, H0 , and H1 respectively, then R = R0 + R1 . Suppose H is a Hilbert space and H = H0 ⊕ H1 . If H0 and H1 are RKHS’s with RKs R0 and R1 , respectively, then H is an RKHS with RK R = R0 + R1 .

2.2

Model Space for Polynomial Splines

Before introducing the general smoothing spline models, it is instructive to see how the polynomial splines introduced in Section 1.2 can be derived under the RKHS setup. Again, consider the regression model yi = f (xi ) + ǫi ,

i = 1, . . . , n,

(2.1)

where the domain of the function f is X = [a, b] and the model space for f is the Sobolev space W2m [a, b] defined in (1.4). The smoothing spline estimate fˆ is the solution to the PLS (1.8). Model space construction and decomposition of W2m [a, b] The Sobolev space W2m [a, b] is an RKHS with the inner product (f, g) =

m−1 X

f

(ν)

(a)g

(ν)

ν=0

(a) +

Z

b

f (m) g (m) dx.

(2.2)

a

Furthermore, W2m [a, b] = H0 ⊕ H1 , where H0 = span 1, (x − a), . . . , (x − a)m−1 /(m − 1)! , Z b (2.3) (ν) H1 = f : f (a) = 0, ν = 0, . . . , m − 1, (f (m) )2 dx < ∞ , a

are RKHS’s with corresponding RKs

m X (x − a)ν−1 (z − a)ν−1 , (ν − 1)! (ν − 1)! ν=1 Z b (z − u)m−1 (x − u)m−1 + + du. R1 (x, z) = (m − 1)! (m − 1)! a

R0 (x, z) =

(2.4)


15

The function (x)+ = max{x, 0}. Details about the foregoing construction can be found in Schumaker (2007). It is clear that H0 contains the polynomial of order m in the Taylor expansion. Note that the basis listed in (2.3), φν (x) = (x − a)ν−1 /(ν − 1)! for ν = 1, . . . , m, is an orthonormal basis of H0 . For any f ∈ H1 , it is easy to check that Z x Z x Z x1 Z xm−1 ′ f (x) = f (u)du = · · · = dx1 dx2 · · · f (m) (u)du a Z ax Z x1 Z xam−2 a (m) = dx1 dx2 · · · (xm−2 − u)f (u)du = · · · a a a Z x (x − u)m−1 (m) = f (u)du. (m − 1)! a Thus the subspace H1 contains the remainder term in the Taylor expansion. Denote P1 as the orthogonal projection operator onto H1 . From the definition of the inner product, the roughness penalty Z

a

b

(f (m) )2 dx = ||P1 f ||2 .

(2.5)

Rb Therefore, a (f (m) )2 dx measures the distance between f and the parametric polynomial space H0 . There is no penalty to functions in H0 . The PLS (1.8) can be rewritten as n

1X (yi − f (xi ))2 + λ||P1 f ||2 . n i=1

(2.6)

The solution to (2.6) will be given for the general case in Section 2.4. The above setup for polynomial splines suggests the following ingredients for the construction of a general smoothing spline model: 1. An RKHS H as the model space for f 2. A decomposition of the model space into two subspaces, H = H0 ⊕ H1 , where H0 consists of functions that are not penalized 3. A penalty ||P1 f ||2 Based on prior knowledge and purpose of the study, different choices can be made on the model space, its decomposition, and the penalty. These options make the spline smoothing method flexible and versatile. Choices of these options will be illustrated throughout the book.

16

2.3


General Smoothing Spline Regression Models

A general smoothing spline regression (SSR) model assumes that yi = f (xi ) + ǫi ,

i = 1, . . . , n,

(2.7)

where yi are observations of the function f evaluated at design points xi , and ǫi are zero-mean independent random errors with a common variance σ 2 . To deal with different situations in a unified fashion, let the domain of the function f be an arbitrary set X , and the model space be an RKHS H on X with RK R(x, z). The choice of H depends on several factors including the domain X and prior knowledge about the function f . Suppose H can be decomposed into two subspaces, H = H0 ⊕ H1 ,

(2.8)

where H0 is a finite dimensional space with basis functions φ1 (x), . . . , φp (x), and H1 is an RKHS with RK R1 (x, z). H0 , often referred to as the null space, consists of functions that are not penalized. In addition to the construction for polynomial splines in Section 2.2, specific constructions of commonly used model spaces will be discussed in Sections 2.6–2.11. The decomposition (2.8) is equivalent to decomposing the function f = f0 + f1 ,

(2.9)

where f0 and f1 are projections onto H0 and H1 , respectively. The component f0 represents a linear regression model in space H0 , and the component f1 represents systematic variation not explained by f0 . Therefore, the magnitude of f1 can be used to check or test if the parametric model is appropriate. Projections f0 and f1 will be referred to as the “parametric” and “smooth” components, respectively. Sometimes observations of f are made indirectly through linear funcRb tionals. For example, f may be observed in the form a wi (x)f (x)dx where wi are known functions. Another example is that observations are taken on the derivatives f ′ (xi ). Therefore, it is useful to consider an even more general SSR model yi = Li f + ǫi ,

i = 1, . . . , n,

(2.10)

where Li are bounded linear functionals on H. Model (2.7) is a special case of (2.10) with Li being evaluational functionals at design points defined as Li f = f (xi ). By the definition of an RKHS, these evaluational functionals are bounded.


2.4

17

Penalized Least Squares Estimation

The estimation method will be presented for the general model (2.10). The smoothing spline estimate of f , fˆ, is the minimizer of the PLS n

1X (yi − Li f )2 + λ||P1 f ||2 , n i=1

(2.11)

where λ is a smoothing parameter controlling the balance between the goodness-of-fit measured by the least squares and departure from the null space H0 measured by ||P1 f ||2 . Functions in H0 are not penalized since ||P1 f ||2 = 0 when f ∈ H0 . Note that fˆ depends on λ even though the dependence is not expressed explicitly. Estimation procedures presented in this chapter assume that the λ has been fixed. The impact of the smoothing parameter and methods of selecting it will be discussed in Chapter 3. Since Li are bounded linear functionals, by the Riesz representation theorem, there exists a representer ηi ∈ H such that Li f = (ηi , f ). For a fixed x, consider Rx (z) , R(x, z) as a univariate function of z. Then, by properties of the reproducing kernel, we have ηi (x) = (ηi , Rx ) = Li Rx = Li(z) R(x, z),

(2.12)

where Li(z) indicates that Li is applied to what follows as a function of z. Equation (2.12) implies that the representer ηi can be obtained by applying the operator to the RK R. Let ξi = P1 ηi be the projection of ηi onto H1 . Since R(x, z) = R0 (x, z) + R1 (x, z), where R0 and R1 are RKs of H0 and H1 , respectively, and P1 is self-adjoint such that (P1 g, h) = (g, P1 h) for any g, h ∈ H, we have ξi (x) = (ξi , Rx ) = (P1 ηi , Rx ) = (ηi , P1 Rx ) = Li(z) R1 (x, z).

(2.13)

Equation (2.13) implies that the representer ξi can be obtained by applying the operator to the RK R1 . Furthermore, (ξi , ξj ) = Li(x) ξj (x) = Li(x) Lj(z) R1 (x, z). Denote p

T = {Li φν }ni=1 ν=1 , Σ = {Li(x) Lj(z) R1 (x, z)}ni,j=1 ,

(2.14)

where T is an n × p matrix, and Σ is an n × n matrix. For the special case of evaluational functionals Li f = f (xi ), we have ξi (x) = R1 (x, xi ), p T = {φν (xi )}ni=1 ν=1 , and Σ = {R1 (xi , xj )}ni,j=1 .

18


Write the estimate fˆ as fˆ(x) =

p X

dν φν (x) +

ν=1

n X

ci ξi (x) + ρ,

i=1

where ρ ∈ H1 and (ρ, ξi ) = 0 for i = 1, . . . , n. Since ξi = P1 ηi , then ηi can be written as ηi = ζi + ξi , where ζi ∈ H0 . Therefore, Li ρ = (ηi , ρ) = (ζi , ρ) + (ξi , ρ) = 0.

(2.15)

Let y = (y1 , . . . , yn )T and fˆ = (L1 fˆ, . . . , Ln fˆ)T be the vectors of observations and fitted values, respectively. Let d = (d1 , . . . , dp )T and c = (c1 , . . . , cn )T . From (2.15), we have fˆ = T d + Σc. (2.16) P Furthermore, ||P1 f ||2 = || ni=1 ci ξi + ρ||2 = cT Σc + ||ρ||2 . Then the PLS (2.11) becomes 1 ||y − T d − Σc||2 + λcT Σc + ||ρ||2 . n

(2.17)

It is obvious that (2.17) is minimized when ρ = 0, which leads to the following result in Kimeldorf and Wahba (1971). Kimeldorf–Wahba representer theorem Suppose T is of full column rank. Then the PLS (2.11) has a unique minimizer given by fˆ(x) =

p X

ν=1

dν φν (x) +

n X

ci ξi (x).

(2.18)

i=1

The above theorem indicates that the smoothing spline estimate fˆ falls in a finite dimensional space. Equation (2.18) represents the smoothing spline estimate fˆ as a linear combination of basis of H0 and representers in H1 . Coefficients c and d need to be estimated from data. Based on (2.18), the PLS (2.17) reduces to 1 ||y − T d − Σc||2 + λcT Σc. n

(2.19)

Taking the first derivatives leads to the following equations for c and d: (Σ + nλI)Σc + ΣT d = Σy, T T Σc + T T T d = T T y,

(2.20)


19

where I is the identity matrix. Equations in (2.20) are equivalent to Σ + nλI ΣT Σc Σy = . TT TTT d TTy There may be multiple sets of solutions for c when Σ is singular. Nevertheless, all sets of solutions lead to the same estimate of the function fˆ (Gu 2002). Therefore, it is only necessary to derive one set of solutions. Consider the following equations (Σ + nλI)c + T d = y, T T c = 0.

(2.21)

It is easy to see that a set of solutions to (2.21) is also a set of solutions to (2.20). The solutions to (2.21) are d = (T T M −1 T )−1 T T M −1 y, c = M −1 {I − T (T T M −1 T )−1 T T M −1 }y,

(2.22)

where M = Σ + nλI. To compute the coefficients c and d, consider the QR decomposition of T , R T = (Q1 Q2 ) , 0 where Q1 , Q2 , and R are n × p, n × (n − p), and p × p matrices; Q = (Q1 Q2 ) is an orthogonal matrix; and R is upper triangular and invertible. Since T T c = RT QT1 c = 0, we have QT1 c = 0 and c = QQT c = (Q1 QT1 + Q2 QT2 )c = Q2 QT2 c. Multiplying the first equation in (2.21) by QT2 and using the fact that QT2 T = 0, we have QT2 M Q2 QT2 c = QT2 y. Therefore, c = Q2 (QT2 M Q2 )−1 QT2 y. (2.23) Multiplying the first equation in (2.21) by QT1 , we have Rd = QT1 (y − M c). Thus, d = R−1 QT1 (y − M c). (2.24) Equations (2.23) and (2.24) will be used to compute coefficients c and d. Based on (2.16), the first equation in (2.21) and equation (2.23), the fitted values fˆ = T d + Σc = y − nλc = H(λ)y, (2.25) where H(λ) , I − nλQ2 (QT2 M Q2 )−1 QT2

(2.26)

20


is the so-called hat (influence, smoothing) matrix. The dependence of the hat matrix on the smoothing parameter λ is expressed explicitly. Note that equation (2.25) provides the fitted values while equation (2.18) can be used to compute estimates at any values of x.

2.5

The ssr Function

The R function ssr in the assist package is designed to fit SSR models. After deciding the model space and the penalty, the estimate fˆ is completely decided by y, T , and Σ. Therefore, these terms need to be specified in the ssr function. A typical call is ssr(formula, rk) where formula and rk are required arguments. Together they specify y, T , and Σ. Suppose the vector y and matrices T and Σ have been created in R. Then, formula lists y on the left-hand side, and T matrix on the right-hand side of an operator ~. The argument rk specifies the matrix Σ. In the most common situation where Li are evaluational functionals, the fitting can be greatly simplified since Li are decided by design points xi . There is no need to compute T and Σ matrices before calling the ssr function. Instead, they can be computed internally. Specifically, a direct approach to fit the standard SSR model (2.7) is to list y on the left-hand side and φ1 (x), . . . , φp (x) on the right-hand side of an operator ~ in the formula, and to specify a function for computing R1 in the rk argument. Functions for computing the RKs of some commonly used RKHS’s are available in the assist package. Users can easily write their own functions for computing RKs. There are several optional arguments for the ssr function, some of which will be discussed in the following chapters. In particular, methods for selecting the smoothing parameter λ will be discussed in Chapter 3. For simplicity, unless explicitly specified, all examples in this chapter use the default method that selects λ using the generalized cross-validation criterion. Bayesian and bootstrap confidence intervals for fitted functions are constructed based on the methods in Section 3.8. We now show how to fit polynomial splines to the motorcycle data. Consider the construction of polynomial splines in Section 2.2. For simplicity, we first consider the special cases of polynomial splines with m = 1 and m = 2, which are called linear and cubic splines, respectively. Denote x ∧ z = min{x, z} and x ∨ z = max{x, z}. Based on (2.3)


21

and (2.4), Table 2.1 lists bases for null spaces and RKs of linear and cubic splines for the special domain X = [0, b].

TABLE 2.1

Bases of null spaces and RKs of linear and cubic splines under the construction in Section 2.2 with X = [0, b] m Spline φν R0 R1 1 Linear 1 1 x∧z 2 Cubic 1, x 1 + xz (x ∧ z)2 {3(x ∨ z) − x ∧ z}/6

Functions linear2 and cubic2 in the assist package compute evaluations of R1 in Table 2.1 for linear and cubic splines, respectively. Functions for higher-order polynomial splines are also available. Note that the domain for functions linear2 and cubic2 is X = [0, b] for any fixed b > 0. The RK on the general domain X = [a, b] can be calculated by a translation, for example, cubic2(x-a). To fit a cubic spline to the motorcycle data, one may create matrices T and Σ first and then call the ssr function: > T Sigma ssr(accel~T-1, rk=Sigma) The intercept is automatically included in the formula statement. Therefore, T-1 is used to exclude the intercept since it is already included in the T matrix. Since Li are evaluational functionals for the motorcycle example, the ssr function can be called directly: > ssr(accel~times, rk=cubic2(times)) The inputs for formula and rk can be modified for fitting polynomial splines of different orders. For example, the following statements fit linear, quintic (m = 3), and septic (m = 4) splines: > ssr(accel~1, rk=linear2(times)) > ssr(accel~times+I(times^2), rk=quintic2(times)) > ssr(accel~times+I(times^2)+I(times^3), rk=septic2(times)) The linear and cubic spline fits are shown in Figure 2.1.

22

Smoothing Splines: Methods and Applications Cubic spline fit

Linear spline fit


o o oo

o ooo

oo o o oooo oooooooooo oooo oo o o o o o o o o o o ooooo oo o oo o oo ooo o oo oo oo o o o oo o oo o o oo o oo o o o o o ooo oo o o

0

10

o

o o

o

o oo

o o o o

oo

o oo o o o o o o o o o oo oo o o o oo o o o o o

oo o o ooooo oooooooooo oooo o oo o o o o o o o o o oooooo oo o oo o oo ooo o oo oo oo o o o oo o oo o o oo o oo o o o o o ooo oo o o

o

20 30 40 time (ms)

50

60

0

10

o o

o ooo

o o o o

o

oo o

o o

o oo o o o o o o o oo oo o o o oo o o o

o o

20 30 40 time (ms)

50

60

FIGURE 2.1 Motorcycle data, plots of observations (circles), the linear spline fit (left), and the cubic spline fit (right) as solid lines, and 95% Bayesian confidence intervals (shaded regions).

2.6

Another Construction for Polynomial Splines

One construction of RKHS for the polynomial spline was presented in Section 2.2. This section presents an alternative construction for W2m [0, 1] on the domain X = [0, 1]. In practice, without loss of generality, a continuous interval [a, b] can always be transformed into [0, 1]. Let kr (x) = Br (x)/r! be scaled Bernoulli polynomials where Br are R1 defined recursively by B0 (x) = 1, Br′ (x) = rBr−1 (x) and 0 Br (x)dx = 0 for r = 1, 2, . . . (Abramowitz and Stegun 1964). The first four scaled Bernoulli polynomials are k0 (x) = 1, k1 (x) = x − 0.5, 1 1 k12 (x) − , k2 (x) = 2 12 1 1 7 k4 (x) = k14 (x) − k12 (x) + . 24 2 240

(2.27)

Alternative model space construction and decomposition of W2m [0, 1] The Sobolev space W2m [0, 1] is an RKHS with the inner product Z 1 Z 1 m−1 X Z 1 (f, g) = f (ν) dx g (ν) dx + f (m) g (m) dx. (2.28) ν=0

0

0

0


23

Furthermore, W2m [0, 1] = H0 ⊕ H1 , where H0 = span k0 (x), k1 (x), . . . , km−1 (x) , Z 1 H1 = f : f (ν) dx = 0, ν = 0, . . . , m − 1, Z

0 1

0

(2.29)

(f (m) )2 dx < ∞ ,

are RKHS’s with corresponding RKs R0 (x, z) =

m−1 X

kν (x)kν (z),

ν=0

(2.30)

R1 (x, z) = km (x)km (z) + (−1)m−1 k2m (|x − z|).

The foregoing alternative construction was derived by Craven and Wahba (1979). Note that the inner product (2.28) is different from (2.2). Again, H0 contains polynomials, and the basis listed in (2.29), φν (x) = kν−1 (x) for ν = 1, . . . , m, is an orthonormal basis of H0 . Denote P1 as the orthogonal projection operator onto . From the definition R 1 H1(m) of the inner product, the roughness penalty 0 (f )2 dx = ||P1 f ||2 . Based on (2.29) and (2.30), Table 2.2 lists bases for the null spaces and RKs of linear and cubic splines under the alternative construction in this section.

TABLE 2.2 cubic m 1 2

splines Spline Linear Cubic

Bases of the null spaces and RKs for linear and under the construction in Section 2.6 with X = [0, 1] φν R0 R1 1 1 k1 (x)k1 (z) + k2 (|x − z|) 1, k1 (x) 1 + k1 (x)k1 (z) k2 (x)k2 (z) − k4 (|x − z|)

Functions linear and cubic in the assist package compute evaluations of R1 in Table 2.2 for linear and cubic splines respectively. Functions for higher-order polynomial splines are also available. Note that the domain under construction in this section is restricted to [0, 1]. Thus the scale option is needed when the domain is not [0, 1]. For example, the following statements fit linear and cubic splines to the motorcycle data:

24


> ssr(accel~1, rk=linear(times), scale=T) > ssr(accel~times, rk=cubic(times), scale=T) The scale option scales the independent variable times into the interval [0, 1]. It is a good practice to scale a variable first before fitting. For example, the following statements lead to the same cubic spline fit: > x ssr(accel~x, rk=cubic(x))

2.7

Periodic Splines

Many natural phenomena follow a cyclic pattern. For example, many biochemical, physiological, or behavioral processes in living beings follow a daily cycle called circadian rhythm, and many Earth processes follow an annual cycle. In these cases the mean function f is known to be a smooth periodic function. Without loss of generality, assume that the domain of the function X = [0, 1] and f is a periodic function on [0, 1]. Since periodic functions can be regarded as functions defined on the unit circle, periodic splines are often referred to as splines on the circle. The model space for periodic spline of order m is W2m (per) = f : f (j) are absolutely continuous, f (j) (0) = f (j) (1), Z 1 j = 0, . . . , m − 1, (f (m) )2 dx < ∞ . (2.31) 0

Craven and Wahba (1979) derived the following construction. Model space construction and decomposition of W2m (per) The space W2m (per) is an RKHS with inner product (f, g) =

Z

0

1

Z f dx

0

1

Z gdx +

1

f (m) g (m) dx.

0

Furthermore, W2m (per) = H0 ⊕ H1 , where H0 = span 1 , Z 1 H1 = f ∈ W2m (per) : f dx = 0 , 0

(2.32)


25

are RKHS’s with corresponding RKs R0 (x, z) = 1, R1 (x, z) = (−1)m−1 k2m (|x − z|).

(2.33)

R1 Again, the roughness penalty 0 (f (m) )2 dx = ||P1 f ||2 . The function periodic in the assist library calculates R1 in (2.33). The order m is specified by the argument order. The default is a cubic periodic spline with order=2. We now illustrate how to fit a periodic spline using the Arosa data, which contain monthly mean ozone thickness (Dobson units) in Arosa, Switzerland, from 1926 to 1971. Suppose we want to investigate how ozone thickness changes over months in a year. It is reasonable to assume that the mean ozone thickness is a periodic function of month. Let thick be the dependent variable and x be the independent variable month scaled into the interval [0, 1]. The following statements fit a cubic periodic spline: > data(Arosa); Arosa$x ssr(thick~1, rk=periodic(x), data=Arosa)

300

thickness 350

400

The fit of the periodic spline is shown in Figure 2.2.

1

2

3

4

5

6 7 month

8

9

10 11 12

FIGURE 2.2 Arosa data, plot of observations (points), and the periodic spline fits (solid line). The shaded region represents 95% Bayesian confidence intervals.

26

2.8


Thin-Plate Splines

Suppose f is a function of a multivariate independent variable x = (x1 , . . . , xd ) ∈ Rd , where Rd is the Euclidean d-space. Assume the regression model yi = f (xi ) + ǫi , i = 1, . . . , n, (2.34) where xi = (xi1 , . . . , xid ) and ǫi are zero-mean independent random errors with a common variance σ 2 . Define the model space for a thin-plate spline as d W2m (Rd ) = f : Jm (f ) < ∞ , (2.35) where

d (f ) = Jm

X

α1 +···+αd

m! α ! . . . αd ! =m 1

Z

∞

−∞

···

Z

∞

−∞

∂mf α1 d ∂x1 . . . ∂xα d

2 Y d

dxj .

j=1

(2.36) d Since Jm (f ) is invariant under a rotation of the coordinates, the thinplate spline is especially well suited for spatial data (Wahba 1990, Gu 2002). Define an inner product as Z ∞ Z ∞ X m! ··· (f, g) = α ! . . . αd ! −∞ −∞ α1 +···+αd =m 1 Y d ∂ mg ∂ mf dxj . (2.37) αd αd 1 1 ∂xα ∂xα 1 . . . ∂xd 1 . . . ∂xd j=1 Model space construction of W2m (Rd ) With the inner product (2.37), W2m (Rd ) is an RKHS if and only if 2m − d > 0. Details can be found in Duchon (1977) and Meinguet (1979). A thinplate spline estimate is the minimizer to the PLS n

1X d (yi − f (xi ))2 + λJm (f ) n i=1

(2.38)

d in W2m (Rd ). The null space H0 of the penalty functional Jm (f ) is the space spanned by polynomials in d variables of total degree up to m − 1.

Smoothing Spline Regression 27 d+m−1 Thus the dimension of the null space p = . For example, d when d = 2 and m = 2, 2 2 2 2 ) Z ∞ Z ∞ ( 2 2 ∂ f ∂ f ∂ f J22 (f ) = +2 + dx1 dx2 , 2 ∂x ∂x ∂x ∂x22 1 2 −∞ −∞ 1 and the null space is spanned by φ1 (x) = 1, φ2 (x) = x1 , and φ3 (x) = x2 . In general, denote φ1 , . . . , φp as the p polynomials of total degree up to m−1 that span H0 . Denote Em as the Green function for the m-iterated Laplacian Em (x, z) = E(||x − z||), where ||x − z|| is the Euclidean distance and ( d (−1) 2 +1+m |u|2m−d log |u|, d even, E(u) = |u|2m−d , d odd. p Let T = {φν (xi )}ni=1 ν=1 and K = {Em (xi , xj )}ni,j=1 . The bivariate function Em is not the RK of W2m (Rd ) since it is not nonnegative definite. Nevertheless, it is conditionally nonnegative definite in the sense that T T c = 0 implies that cT Kc ≥ 0. Referred to as a semi-kernel, the function Em is sufficient for the purpose of estimation. Assume that T is of full column rank. It can be shown that the unique minimizer of the PLS (2.38) is given by (Wahba 1990, Gu 2002)

fˆ(x) =

p X

ν=1

dν φν (x) +

n X

ci ξi (x),

(2.39)

i=1

where ξi (x) = Em (xi , x). Therefore, Em plays the same role as the RK R1 . The coefficients c and d are solutions to (K + nλI)c + T d = y, T T c = 0.

(2.40)

The above equations have the same form as those in (2.21). Therefore, computations in Section 2.4 carry over with Σ being replaced by K. The semi-kernel Em is calculated by the function tp.pseudo in the assist package. The order m is specified by the order argument with default as order=2. The USA climate data contain average winter temperatures in 1981 from 1214 stations in USA. To investigate how average winter temperature (temp) depends on geological locations (long and lat), we fit a thin-plate spline as follows: > attach(USAtemp) > ssr(temp~long+lat, rk=tp.pseudo(list(long,lat)))

28

Smoothing Splines: Methods and Applications 10 15 20 25 30 35 40 45 50 55 60 65 70

FIGURE 2.3 USA climate data, contour plot of the thin-plate spline fit. The contour plot of the fit is shown in Figure 2.3. A genuine RK for W2m (Rd ) is needed later in the computation of posterior variances in Chapter 3 and the construction of tensor product splines in Chapter 4. We now discuss briefly how to derive the genuine RK. Define inner product (f, g)0 =

J X

wj f (uj )g(uj ),

(2.41)

j=1

where uj are fixed points in Rd , and wj are fixed positive weights such P that Jj=1 wj = 1. Points uj and weights wj are selected in such a way that the matrix {(φν , φµ )0 }pν,µ=1 is nonsingular. Let φ˜ν , ν = 1, . . . , p, be an orthonormal basis derived from φν with P φ˜1 (x) = 1. Let P0 be the p projection operator onto H0 defined as P0 f = ν=1 (f, φν )0 φν . Then it can be shown that (Gu 2002) R0 (x, z) =

p X

φ˜ν (x)φ˜ν (z),

ν=1

R1 (x, z) = (I − P0(x) )(I − P0(z ) )E(||x − z||)

(2.42)


29

are RKs of H0 and H1 , W2m (Rd ) ⊖ H0 , where P0(x) and P0(z ) are projections applied to the arguments x and z, respectively. p Assume that T = {φν (xi )}ni=1 ν=1 is of full column rank. Let Σ = {R1 (xi , xj )}ni,j=1 . One relatively simple approach to compute φ˜ν and Σ is to let J = n, uj = xj , and wj = n−1 . It is easy to see that {(φν , φµ )0 }pν,µ=1 = n−1 T T T , which is nonsingular. Let R T = (Q1 Q2 ) 0 be the QR decomposition of T . Then √ (φ˜1 (x), . . . , φ˜p (x)) = n(φ1 (x), . . . , φp (x))R−1 and Σ = Q2 QT2 KQ2 QT2 . The function tp computes evaluations of R1 in (2.42) with J = n, uj = xj and wj = n−1 .

2.9

Spherical Splines

Spherical spline, also called spline on the sphere, is an extension of both the periodic spline defined on the unit circle and the thin-plate spline defined on R2 . Let the domain be X = S, where S is the unit sphere. Any point x on S can be represented as x = (θ, φ), where θ (0 ≤ θ ≤ 2π) is the longitude and φ (−π/2 ≤ φ ≤ π/2) is the latitude. Define R 2π R π m 2 2 f )2 cos φdφdθ, m even,   0 − π2 (∆ ( m−1 ) J(f ) = R 2π R π m−1 (∆ 2 f )2θ 2  + (∆ 2 f )2φ cos φdφdθ, m odd,  2 0 −π 2 cos φ

where the notation (g)z represents the partial derivative of g with respect to z, ∆f represents the surface Laplacian on the unit sphere defined as ∆f =

1 1 fθθ + (cos φfφ )φ . 2 cos φ cos φ

Consider the model space m W2 (S) = f :

Z f dx < ∞, J(f ) < ∞ . S

30


Model space construction and decomposition of W2m (S) W2m (S) is an RKHS when m > 1. Furthermore, W2m (S) = H0 ⊕ H1 , where H0 = span{1}, Z m H1 = f ∈ W2 (S) : f dx = 0 , S

are RKHS’s with corresponding RKs

R0 (x, z) = 1, ∞ X 2i + 1 1 R1 (x, z) = Gi (cos γ(x, z)), 4π {i(i + 1)}m i=1 where γ(x, z) is the angle between x and z, and Gi are the Legendre polynomials. Details of the above construction can be found in Wahba (1981). The penalty ||P1 f ||2 = J(f ). The RK R1 is in the form of an infinite series, which is inconvenient to compute. Closed-form expressions are available only when m = 2 and m = 3. Wahba (1981) proposed replacing J by a topologically equivalent seminorm Q under which closed-form RKs can be derived. The function sphere in the assist package calculates R1 under the seminorm Q for 2 ≤ m ≤ 6. The argument order specifies m with default as order=2. The world climate data contain average winter temperatures in 1981 from 725 stations around the globe. To investigate how average winter temperature (temp) depends on geological locations (long and lat), we fit a spline on the sphere: > data(climate) > ssr(temp~1, rk=sphere(cbind(long,lat)), data=climate) The contour plot of the spherical spline fit is shown in Figure 2.4.

2.10

Partial Splines

A partial spline model assumes that yi = sTi β + Li f + ǫi ,

i = 1, . . . , n,

(2.43)


31

−30

−25

−35

−20 −5

−40

50

−15

0

5

−10

15

20

10

latitude 0

25

25

25 20

−50

15

10

5

−150

−100

−50

0 longitude

50

100

150

FIGURE 2.4 World climate data, contour plot of the spherical spline fit. where s is a q-dimensional vector of independent variables, β is a vector of parameters, Li are bounded linear functionals, and ǫi are zero-mean independent random errors with a common variance σ 2 . We assume that f ∈ H, where H is an RKHS on an arbitrary domain X . Model (2.43) contains two components: a parametric linear model and a nonparametric function f . The partial spline model is a special case of the semiparametric linear regression model discussed in Chapter 8. Suppose H = H0 ⊕ H1 , where H0 = span{φ1 , . . . , φp } and H1 is an RKHS with RK R1 . Denote P1 as the projection onto H1 . The function f and parameters β are estimated as minimizers to the following PLS: n

1X (yi − sTi β − Li f )2 + λ||P1 f ||2 . n i=1

(2.44)

p Let S = (s1 , . . . , sn )T , T = {Li φν }ni=1 ν=1 , X = (S T ), and Σ = n {Li(x) Lj(z) R1 (x, z)}i,j=1 . Assume that X is of full column rank. Following similar arguments as in Section 2.4, it can be shown that the PLS (2.44) has a unique minimizer, and the solution of f is given in (2.18).

32


Therefore, the PLS (2.44) reduces to 1 ||y − Xα − Σc||2 + λcT Σc, n where α = (β T , dT )T . As in Section 2.4, we can solve α and c from the following equations: (Σ + nλI)c + Xα = y, X T c = 0.

(2.45)

The above equations have the same form as those in (2.21). Thus, computations in Section 2.4 carry over with T and d being replaced by X and α, respectively. The ssr function can be used to fit partial splines. When Li are evaluational functionals, the partial spline model (2.43) can be fitted by adding s variables at the right-hand side of the formula argument. When Li are not evaluational functionals, matrices X and Σ need to be created and supplied in the formula and rk arguments. One interesting application of the partial spline model is to fit a nonparametric regression model with potential change-points. A changepoint is defined as a discontinuity in the mean function or one of its derivatives. Note that the function g(x) = (x − t)k+ has a jump in its kth derivative at location t. Therefore, it can be used to model changepoints. Specifically, consider the following model yi =

J X j=1

k

βj (xi − tj )+j + f (xi ) + ǫi ,

i = 1, . . . , n,

(2.46)

where xi ∈ [a, b] are design points, tj ∈ [a, b] are change-points, f is a smooth function, and ǫi are zero-mean independent random errors with a common variance σ 2 . The mean function in model (2.46) has a jump at tj in its kj th derivative with magnitude βj . The choice of model space for f depends on the application. For example, the polynomial or periodic spline space may be used. When tj and kj are known, model k (2.46) is a special case of partial spline with q = J, sij = (xi − tj )+j , and T si = (si1 , . . . , siJ ) . We now use the geyser data and motorcycle data to illustrate changepoints detection using partial splines. For the geyser data, Figure 1.3(b) indicates that there may be a jump in the mean function between 2.5 and 3.5 minutes. Therefore, we consider the model yi = β(xi − t)0+ + f (xi ) + ǫi ,

i = 1, . . . , n,

(2.47)

where xi are the duration variable scaled into [0, 1], t is a change-point, and (x − t)0+ = 0 when x ≤ t and 1 otherwise. We assume that f ∈


33

W22 [0, 1]. For a fixed t, say t = 0.397, we can fit the partial spline as follows: > attach(faithful) > x ssr(waiting~x+(x>.397), rk=cubic(x)) The partial spline fit is shown in Figure 2.5(a). No trend is shown in the residual plot in Figure 2.5(b). The change-point is fixed at t = 0.397 in the above fit. Often it is unknown in practice. To search for the location of the change-point t, we compute AIC and GCV (generalized cross-validation) criteria on a grid points between 0.25 and 0.55: > aic for (t fit > > > > >

library(fda); attach(CanadianWeather) y 0 and γ > 0. It is easy to see that H0 is the kernel of the differential operator L=D−

δγ exp(−γx) . 1 + δ exp(−γx)

The Wronskian is an 1 × 1 matrix W (x) = {1 + δ exp(−γx)}−1 . Since {W T (a)W (a)}−1 = {1 + δ exp(−γa)}2 , then the RK of H0 R0 (x, z) =

{1 + δ exp(−γa)}2 . {1 + δ exp(−γx)}{1 + δ exp(−γz)}

(2.60)

The Green function   1 + δ exp(−γs) , s ≤ x, G(x, s) = 1 + δ exp(−γx) 0, s > x.

Thus the RK of H1

R1 (x, z) = {1 + δ exp(−γx)}−1 {1 + δ exp(−γz)}−1 x ∧ z − a + 2δγ −1 [exp(−γa) − exp{−γ(x ∧ z)}] +δ 2 (2γ)−1 [exp(−2γa) − exp{−2γ(x ∧ z)}] . (2.61)

The paramecium caudatum data consist of growth of paramecium caudatum population in the medium of Osterhout. We now illustrate how to fit a logistic spline to the paramecium caudatum data. Observations are shown in Figure 2.10. Let y = density and x = days. We first fit the following logistic growth model yi =

β1 + ǫi , 1 + β2 exp(−β3 x)

i = 1, . . . , 25,

(2.62)


45

using the statements > data(paramecium); attach(paramecium) > para.nls logit.rk tmp1

library(fda); attach(melanoma) x ssr(y~1, rk=periodic(x), limnla=log10(73*.001)) where the argument limnla specifies a search range for log10 (nλ). To see how a spline fit is affected by the choice of λ, periodic spline fits with six different values of λ are shown in Figure 3.1. It is obvious that the fit with λ = ∞ is a constant, that is, f∞ ∈ H0 . The fit with λ = 0 interpolates data. A larger λ leads to a smoother fit. Both λ = 0.0001 and λ = 0.00001 lead to visually reasonable fits. In practice it is desirable to select the smoothing parameter using an objective method rather than visual inspection. In a sense, a data-driven choice of λ allows data to speak for themselves. Thus, it is not exaggerating to say that the choice of λ is the spirit and soul of nonparametric regression. We now inspect how λ controls the fit. Again, consider model (1.1) for Stratford weather data. Let us first consider a parametric approach that approximates f using a trigonometric polynomial up to a certain

53

54


0.0

λ=∞

0.2

0.4

0.6

0.8

1.0

λ = 0.001

λ = 1e−04 100 80

temperature (Fahrenheit)

60 40 20

λ = 1e−05

λ = 1e−06

λ=0

100 80 60 40 20

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

time

FIGURE 3.1 Stratford weather data, plot of observations, and the periodic spline fits with different smoothing parameters. degree, say k, where 0 ≤ k ≤ K and K = (n − 1)/2 = 36. Denote the corresponding parametric model space for f as √ √ (3.1) Mk = span{1, 2 sin 2πνx, 2 cos 2πνx, ν = 1, . . . , k}, where M0 = span{1}. For a fixed k, write the regression model based on Mk in a matrix form as y = Xk βk + ǫ, where √ √ 1 √2 sin 2πx1 √2 cos 2πx1  1 2 sin 2πx2 2 cos 2πx2  Xk =  . .. ..  .. . . √ √ 1 2 sin 2πxn 2 cos 2πxn 

√ √  · · · √2 sin 2πkx1 √2 cos 2πkx1 · · · 2 sin 2πkx2 2 cos 2πkx2    .. ..  . . ··· √ √ · · · 2 sin 2πkxn 2 cos 2πkxn

is the design matrix, xi = i/n, β k = (β1 , . . . , β2k+1 )T , and ǫ = (ǫ1 , . . . , ǫn )T . Since design points are equally spaced, we have the following

Smoothing Parameter Selection and Inference

55

orthogonality relations: n

2X cos 2πνxi cos 2πµxi = δν,µ , 1 ≤ ν, µ ≤ K, n i=1 n

2X sin 2πνxi sin 2πµxi = δν,µ , 1 ≤ ν, µ ≤ K, n i=1 n

2X cos 2πνxi sin 2πµxi = 0, n i=1

1 ≤ ν, µ ≤ K,

where δν,µ is the Kronecker delta. Therefore, XkT Xk = √ nI2k+1 , where I2k+1 is an identity matrix of size 2k+1. Note that XK / n is an orthogT ˜ = XK onal matrix. Define the discrete Fourier transformation y y/n. −1 T T T ˆ ˜k, Then the LS estimate of β k is βk = (Xk Xk ) Xk y = Xk y/n = y ˜ k consists of the first 2k + 1 elements of the discrete Fourier where y ˜ . More explicitly, transformation y n

1X βˆ1 = yi = y˜1 , n i=1 √ n 2X yi sin 2πνxi = y˜2ν , 1 ≤ ν ≤ k, βˆ2ν = n i=1 √ n 2X yi cos 2πνxi = y˜2ν+1 , 1 ≤ ν ≤ k. βˆ2ν+1 = n i=1

(3.2)

Now consider modeling f using the cubic periodic spline space W22 (per). The exact solution was given in Chapter 2. To simplify the argument, let us consider the following PLS min

f ∈MK

(

n

1X (yi − f (xi ))2 + λ n i=1

Z

0

1

′′ 2

)

(f ) dx ,

(3.3)

where the model space W22 (per) is approximated by MK . The following discussion holds true for the exact solution in W22 (per) (Gu 2002). However, the approximation makes the following argument simpler and transparent. Let fˆ(x) = α ˆ1 +

K X

ν=1

√ √ ˆ 2ν+1 2 cos 2πνx) (ˆ α2ν 2 sin 2πνx + α

56


ˆ where be the solution to (3.3). Then fˆ , (fˆ(x1 ), . . . , fˆ(xn ))T = XK α, ˆ = (ˆ α α1 , . . . , α ˆ 2K+1 )T . The LS 1 1 1 T ||y − fˆ ||2 = || √ XK (y − fˆ )||2 n n n 1 T 1 T ˆ 2 = || XK y − XK XK α|| n n ˆ 2. = ||˜ y − α|| Thus (3.3) reduces to the following ridge regression problem (ˆ α1 − y˜1 )2 + +λ

K X

K X (ˆ α2ν − y˜2ν )2 + (ˆ α2ν+1 − y˜2ν+1 )2 ν=1

(2πν)4 (ˆ α22ν + α ˆ 22ν+1 ).

(3.4)

ν=1

The solutions to (3.4) are α ˆ 1 = y˜1 , y˜2ν , 1 + λ(2πν)4 y˜2ν+1 , = 1 + λ(2πν)4

α ˆ2ν = α ˆ2ν+1

ν = 1, . . . , K,

(3.5)

ν = 1, . . . , K.

Thus the periodic spline is essentially a low-pass filter: components at frequency ν are downweighted by a factor of 1 + λ(2πν)4 . Figure 3.2 shows how λ controls the nature of the filter: more high frequencies are filtered out as λ increases. Comparing (3.2) and (3.5), it is clear that selecting an order k for the trigonometric polynomial model may be viewed as hard thresholding, and selecting the smoothing parameter λ for the periodic spline may be viewed as soft thresholding. Now consider the general spline model (2.10). From (2.26), the hat matrix H(λ) = I − nλQ2 (QT2 M Q2 )−1 QT2 . Let U EU T be the eigendecomposition of QT2 ΣQ2 , where U(n−p)×(n−p) is an orthogonal matrix and E = diag(e1 , . . . , en−p ). The projection onto the space spanned by T PT , T (T T T )−1 T T = Q1 R(RT R)−1 RT QT1 = Q1 QT1 . Then H(λ) = I − nλQ2 U (E + nλI)−1 U T QT2 =

Q1 QT1

= PT +

+ Q2 QT2 − nλQ2 U (E Q2 U V U T QT2 ,

(3.6) −1

+ nλI)

U

T

QT2 (3.7)

57

0.0

0.2

weights 0.4 0.6

0.8

1.0


0

5

10

15 20 frequency

25

30

35

FIGURE 3.2 Weights of the periodic spline filter, 1/(1 + λ(2πν)4 ), plotted as a function of frequency ν. Six curves from top down correspond to six different λ: 0, 10−6 , 10−5 , 10−4 , 10−3 , and ∞. where V = diag(e1 /(e1 + nλ), . . . , en−p /(en−p + nλ)). The hat matrix is divided into two mutually orthogonal matrices: one is the projection onto the space spanned by T , and the other is responsible for shrinking part of the signal that is orthogonal to T . The smoothing parameter shrinks eigenvalues in the form eν /(eν + nλ). The choices λ = ∞ and λ = 0 lead to the parametric model H0 and interpolation, respectively. Equation (3.7) also indicates that the hat matrix H(λ) is nonnegative definite. However, unlike the projection matrix for a parametric model, H(λ) is usually not idempotent. H(λ) has p eigenvalues equal to one and the remaining eigenvalues less than one when λ > 0.

3.2

Trade-Offs

Before introducing methods for selecting the smoothing parameter, it is helpful to discuss some basic concepts and principles for model selection. In general, model selection boils down to compromises between different aspects of a model. Occam’s razor has been the guiding principle for the compromises: the model that fits observations sufficiently well in the least complex way should be preferred. To be precise on fits observations sufficiently well, one needs a quantity that measures how well a model fits the data. One such measure is the LS in (1.6). To be precise on the least complex way, one needs a quantity that measures the complexity of a

58


model. For a parametric model, a common measure of model complexity is the number of parameters in the model, often called the degrees of freedom (df). For example, the df of model Mk in (3.1) equals 2k + 1. What would be a good measure of model complexity for a nonparametric regression procedure? Consider the general nonparametric regression model (2.10). Let fi = Li f , and f = (f1 , . . . , fn ). Let fˆ be an estimate of f based on a modeling procedure M, and fî = Li fˆ. Ye (1998) defined generalized degrees of freedom (gdf) of M as gdf(M) ,

n X ∂Ef (fî ) i=1

∂fi

.

(3.8)

The gdf is an extension of the standard degrees of freedom for general modeling procedures. It can be viewed as the sum of the average sensitivities of the fitted values fî to a small change in the response. It is easy to check that (Efron 2004) n 1 X gdf(M) = 2 Cov(fî , yi ), σ i=1

Pn ˆ where i=1 Cov(fi , yi ) is the so-called covariance penalty (Tibshirani and Knight 1999). For spline estimate with a fixed λ, we have fˆ = H(λ)y based on (2.25). Denote the modeling procedure leading to fˆ as Mλ and H(λ) = {hij }ni,j=1 . Then Pn Pn n n X ∂ j=1 hij fj ∂Ef ( j=1 hij yj ) X = = trH(λ), gdf(Mλ ) = ∂fi ∂fi i=1 i=1

where tr represents the trace of a matrix. Even though λ does not have a physical interpretation as k, trH(λ) is a useful measure of model complexity and will be simply referred to as the degrees of freedom. For Stratford weather data, Figure 3.3(a) depicts how trH(λ) for the cubic periodic spline depends on the smoothing parameter λ. It is clear that the degrees of freedom decrease as λ increases. To illustrate the interplay between the LS and model complexity, we fit trigonometric polynomial models from the smallest model with k = 0 to the largest model with k = K. The square root of residual sum of squares (RSS) are plotted against the degrees of freedom (2k + 1) as circles in Figure 3.3(b). Similarly, we fit the periodic spline with a wide range of values for the smoothing parameter λ. Again, we plot the square root of RSS against the degrees of freedom (trH(λ)) as the solid line in Figure 3.3(b). Obviously, RSS decreases to zero (interpolation) as the degrees of freedom increase to n. The square root of RSS keeps

Smoothing Parameter Selection and Inference (b) o

square root of RSS 5 10 15

degrees of freedom 10 20 30 40 50 60 70

20

(a)

ooo oooo ooo oo ooo ooo o ooooo oo oo oo oooo o o

0

0 −12

−10

−8

−6 −4 log10(λ)

59

−2

0

0

10

20 30 40 50 60 degrees of freedom

70

FIGURE 3.3 Stratford data, plots of (a) degrees of freedom of the periodic spline against the smoothing parameter on the logarithm base 10 scale, and (b) square root of RSS from the trigonometric polynomial model (circles) and periodic spline (line) against the degrees of freedom. declining almost linearly after the initial big drop. It is quite clear that the constant model does not fit data well. However, it is unclear which model fits observations sufficiently well. Figure 3.3(b) shows that the LS and model complexity are two opposite aspects of a model: the approximation error decreases as the model complexity increases. Our goal is to find the “best” model that strikes a balance between these two conflicting aspects. To make the term “best” meaningful, we need a target criterion that quantifies a model’s performance. It is clear that the LS cannot be used as the target because it will lead to the most complex model. Even though there is no universally accepted measure, some criteria are widely accepted and used in practice. We now introduce a criterion that is commonly used for regression models. Consider the loss function L(λ) =

1 ˆ ||f − f ||2 . n

(3.9)

Define the risk function, also called mean squared error (MSE), as 1 ˆ ||f − f ||2 . (3.10) MSE(λ) , EL(λ) = E n We want the estimate fˆ to be as close to the true function f as possible. Obviously, MSE is the expectation of the Euclidean distance between the estimates and the true values. It can be decomposed into two com-

60


ponents:

= = = = ,

MSE(λ) 1 E||(Efˆ − f ) + (fˆ − Efˆ )||2 n 1 2 1 E||Efˆ − f ||2 + E(Efˆ − f )T (fˆ − Efˆ ) + E||fˆ − Efˆ ||2 n n n 1 ˆ 1 ||Ef − f ||2 + E||fˆ − Efˆ ||2 n n 1 σ2 ||(I − H(λ))f ||2 + trH 2 (λ) n n b2 (λ) + v(λ), (3.11)

where b2 and v represent squared bias and variance, respectively. Note that bias depends on the true function, while the variance does not. Based on notations introduced in Section 3.2, let h = (h1 , . . . , hn−p )T , U T QT2 f . From (3.6), we have 1 ||(I − H(λ))f ||2 n ( 2 2 ) 1 T nλ nλ = f Q2 U diag ,..., U T QT2 f n e1 + nλ en−p + nλ 2 n−p 1X nλhν = . n ν=1 eν + nλ

b2 (λ) =

From (3.7), we have σ2 σ2 trH 2 (λ) = tr(PT + Q2 U V 2 U T QT2 ) n n ( ) n−p X eν 2 σ2 = p+ . n eν + nλ ν=1

v(λ) =

The squared bias measures how well fˆ approximates the true function f , and the variance measures how well the function can P be estimated. n−p As λ increases from 0 to ∞, b2 (λ) increases from 0 to ν=1 h2ν /n = T 2 2 2 ||Q2 f || /n, while v(λ) decreases from σ to pσ /n. Therefore, the MSE represents a trade-off between bias and variance. Note that QT2 f represents the signal that is orthogonal to T . It is easy to check that db2 (λ)/dλ|λ=0 = 0 and dv(λ)/dλ|λ=0 < 0. Thus, dMSE(λ)/dλ|λ=0 < 0, and MSE(λ) has at least one minimizer λ∗ > 0. Therefore, when MSE(0) ≤ MSE(∞), there exists at least one λ∗ such that the corresponding PLS estimate performs better in


61

terms of MSE than the LS estimate in H0 and the interpolation. When MSE(0) > MSE(∞), considering the MSE as a function of δ = 1/λ, we have db2 (δ)/dδ|δ=0 < 0 and dv(δ)/dδ|δ=0 = 0. Then, again, there exists at least one δ ∗ such that the corresponding PLS estimate performs better in terms of MSE than the LS estimate in H0 and the interpolation. To calculate MSE, one needs to know the true function f . The following simulation illustrates the bias-variance trade-off. Observations are generated from model (1.1) with f (x) = sin(4πx2 ) and σ = 0.5. The same design points as in the Stratford weather data are used: xi = i/n for i = 1, . . . , n and n = 73. The true function and one realization of observations are shown in Figure 3.4(a). For a fixed λ, the bias, variance, and MSE can be calculated since the true function is known in the simulation. For the cubic periodic spline, Figure 3.4(b) shows b2 (λ), v(λ), and MSE(λ) as functions of log10 (nλ). Obviously, as λ increases, the squared bias increases and the variance decreases. The MSE represents a compromise between bias and variance.

1.5 0.5

y −0.5 −1.5 0.0

0.2

0.4

0.6

x

(b)

squared bias,variance, and MSE 0.00 0.10 0.20 0.30

(a)

0.8

1.0

−8

−7

−6 −5 −4 log10(nλ)

−3

−2

FIGURE 3.4 Plots of (a) true function (line) and observations (circles), and (b) squared bias b2 (λ) (dashed), variance v(λ) (dotted line), and MSE (solid line) for the cubic periodic spline.

Another closely related target criterion is the average predictive squared error (PSE) 1 + ˆ 2 (3.12) ||y − f || , PSE(λ) = E n + T where y + = f + ǫ+ are new observations of f , ǫ+ = (ǫ+ 1 , . . . , ǫn ) are + independent of ǫ, and ǫi are independent and identically distributed

62


with mean zero and variance σ 2 . PSE measures the performance of a model’s prediction for new observations. We have 1 + 2 ˆ PSE(λ) = E ||(y − f ) + (f − f )|| = σ 2 + MSE(λ). n Thus PSE differs from MSE only by a constant σ 2 . Ideally, one would want to select λ that minimizes the MSE (PSE). This is, however, not practical because MSE (PSE) depends on the unknown true function f that one wants to estimate in the first place. Instead, one may estimate MSE (PSE) from the data and then minimize the estimated criterion. We discuss unbiased and cross-validation estimates of PSE (MSE) in Sections 3.3 and 3.4, respectively.

3.3

Unbiased Risk

First consider the case when the error variance σ 2 is known. Since 1 2 ˆ ||y − f || E n 1 2 1 2 T 2 ˆ ˆ =E ||y − f || + (y − f ) (f − f ) + ||f − f || n n n 2 2σ trH(λ) + MSE(λ), (3.13) = σ2 − n then, UBR(λ) ,

1 2σ 2 ||(I − H(λ))y||2 + trH(λ) n n

(3.14)

is an unbiased estimate of PSE(λ). Since PSE differs from MSE only by a constant σ 2 , one may expect the minimizer of UBR(λ) to be close to the minimizer of the risk function MSE(λ). In fact, a stronger result holds: under certain regularity conditions, UBR(λ) is a consistent estimate of the relative loss function L(λ) + n−1 ǫT ǫ (Gu 2002). The function UBR(λ) is referred to as the unbiased risk (UBR) criterion, and the minimizer of UBR(λ) is referred to as the UBR estimate of λ. It is obvious that UBR(λ) is an extension of the Mallow’s Cp criterion. The error variance σ 2 is usually unknown in practice. In general, there are two classes of estimators for σ 2 : residual-based and differencebased estimators. The first class of estimators is based on residuals from an estimate of f . For example, analogous to parametric regression, an


63

estimator of σ 2 based on fit fˆ = H(λ)y, is σ ˆ2 ,

||(I − H(λ))y||2 . n − trH(λ)

(3.15)

The estimator σ ˆ 2 is consistent under certain regularity conditions (Gu 2002). However, it depends critically on the smoothing parameter λ. Thus, it cannot be used in the UBR criterion since the purpose of this criterion is to select λ. For choosing the amount of smoothing, it is desirable to have an estimator of σ 2 without needing to fit the function f first. The difference-based estimators of σ 2 do not require an estimate of the mean function f . The basic idea is to remove the mean function f by taking differences based on some well-chosen subsets of data. Consider the general SSR model (2.10). Let Ij = {i(j, 1), . . . , i(j, Kj )} ⊂ {1, . . . , n} be a subset of index and d(j, k) be some fixed coefficients such that Kj X

d2 (j, k) = 1,

k=1

Since

E then

k=1

 Kj X 

Kj X

d(j, k)yi(j,k)

k=1

d(j, k)Li(j,k) f ≈ 0,

2  

≈E

 Kj X 

j = 1, . . . , J.

d(j, k)ǫi(j,k)

k=1

 2 Kj J X  X 1 d(j, k)yi(j,k) σ ˜2 ≈  J j=1 

2 

= σ2 ,



(3.16)

k=1

provides an approximately unbiased estimator of σ 2 . The estimator σ ˜2 is referred to as a differenced-based estimator since d(j, k) are usually PKj chosen to be contrasts such that k=1 d(j, k) = 0. The specific choices of subsets and coefficients depend on factors including prior knowledge about f and the domain X . Several methods have been proposed for the common situation when x is a univariate continuous variable, f is a smooth function, and Li are evaluational functionals. Suppose design points are ordered such that x1 ≤ x2 ≤ · · · ≤ xn . Since f is smooth, then f (xj+1 ) − f (xj ) ≈ 0 when neighboring design points are √ close to each other. Setting Ij = {j, j + 1} and d(j, 1) = −d(j, 2) = 1/ 2 for j = 1, . . . , n−1, we have the first-order difference-based estimator proposed by Rice (1984): n

2 σ ˜R =

X 1 (yi − yi−1 )2 . 2(n − 1) i=2

(3.17)

64


Hall, Kay and Titterington (1990) proposed the mth order differencebased estimator (m )2 n−m X X 1 2 σ ˜HKT = δk yj+k , (3.18) n − m j=1 k=1

Pm Pm where coefficients δk satisfy k=1 δk = 0, k=1 δk2 = 1, and δ1 δm 6= 0. Optimal choices of δk are studied in Hall et al. (1990). It is easy to 2 see that σ ˜HKT corresponds to Ij = {j, . . . , j + m} and d(j, k) = δk for j = 1, . . . , n − m. 2 2 Both σ ˜R and σ ˜HKT require an ordering of design points that could be problematic for multivariate independent variables. Tong and Wang (2005) proposed a different method for a general domain X . Suppose X is equipped with a norm. Collect squared distances, dij = ||xi − xj ||2 , for all pairs {xi , xj }, and half squared differences, sij = (yi − yj )2 /2, for all pairs {yi , yj }. Then E(sij ) = {f (xi ) − f (xj )}2 /2 + σ 2 . Suppose {f (xi )−f (xj )}2 /2 can be approximated by βdij when dij is small. Then the LS estimate of the intercept in the simple linear model sij = α + βdij + ǫij ,

dij ≤ D,

(3.19)

2 provides an estimate of σ 2 . Denote such an estimator as σ ˜TW . Theoretical properties and the choice of bandwidth D were studied in Tong and Wang (2005). To illustrate the UBR criterion as an estimate of PSE, we generate responses from model (1.1) with f (x) = sin(4πx2 ), σ = 0.5, xi = i/n for i = 1, . . . , n, and n = 73. For the cubic periodic spline, the UBR functions based on 50 replications of simulation data are shown in Figure 3.5 where the true variance is used in (a) and the Rice estimator is used in (b).

3.4

Cross-Validation and Generalized Cross-Validation

Equation (3.13) shows that the RSS underestimates the PSE by the amount of 2σ 2 trH(λ)/n. The second term in the UBR criterion corrects this bias. The bias in RSS is a consequence of using the same data for model fitting and model evaluation. Ideally, these two tasks should be separated using independent samples. This can be achieved by splitting the whole data into two subsamples: a training (calibration) sample for

Smoothing Parameter Selection and Inference UBR with Rice estimator

UBR with true variance

0.1

PSE and UBR 0.2 0.3 0.4 0.5

0.6

65

−8

−7

−6

−5 −4 log10(nλ)

−3

−2

−8

−7

−6

−5 −4 log10(nλ)

−3

−2

FIGURE 3.5 Plots of the PSE function as solid lines, the UBR functions with true σ 2 as dashed lines (left), and the UBR functions with 2 Rice estimator σ ˜R . The minimum point of the PSE is marked as long bars at the bottom. The UBR estimates of log10 (nλ) are marked as short bars. model fitting, and a test (validation) sample for model evaluation. This approach, however, is not efficient unless the sample size is large. The idea behind cross-validation is to recycle data by switching the roles of training and test samples. For simplicity, we present leaving-out-one cross-validation only. That is, each time one observation will be left out as the test sample, and the remaining n − 1 samples will be used as the training sample. Let fˆ[i] be the minimizer of the PLS based on all observations except yi : 1X (yj − Lj f )2 + λ||P1 f ||2 . (3.20) n j6=i

The cross-validation estimate of PSE is n

CV(λ) ,

2 1 X ˆ[i] Li f − yi . n i=1

(3.21)

CV(λ) is referred to as the cross-validation criterion, and the minimizer of CV(λ) is called the cross-validation estimate of the smoothing parameter. Computation of fˆ[i] based on (3.21) for each i = 1, . . . , n would be costly. Fortunately, this is unnecessary due to the following lemma.

66


Leaving-out-one Lemma For any fixed i, fˆ[i] is the minimizer of 2 1 X 1 ˆ[i] Li f − Li f + (yj − Lj f )2 + λ||P1 f ||2 . n n

(3.22)

j6=i

[Proof] For any function f , we have 2 1 X 1 ˆ[i] Li f − Li f + (yj − Li f )2 + λ||P1 f ||2 n n j6=i

1X (yj − Li f )2 + λ||P1 f ||2 ≥ n j6=i 2 1 X ≥ yj − Lj fˆ[i] + λ||P1 fˆ[i] ||2 n j6=i 2 1 X 2 1 ˆ[i] = Li f − Li fˆ[i] + yj − Lj fˆ[i] + λ||P1 fˆ[i] ||2 , n n j6=i

where the second inequality holds since fˆ[i] is the minimizer of (3.20). The above lemma indicates that the solution to the PLS (3.20) without the ith observation, fˆ[i] , is also the solution to the PLS (2.11) with the ith observation yi being replaced by the fitted value Li fˆ[i] . Note that the hat matrix H(λ) depends on the model space and operators Li only. It does not depend on observations of the dependent variable. Therefore, both fits based on (2.11) and (3.22) have the same hat matrix. That is, [i] [i] fˆ = H(λ)y and fˆ = H(λ)y [i] , where fˆ = (L1 fˆ[i] , . . . , Ln fˆ[i] )T and y [i] is the same as y except that the ith element is replaced by Li fˆ[i] . Denote H(λ) = {hij }ni,j=1 . Then Li fˆ = Li fˆ[i] =

n X

hij yj ,

j=1

X j6=i

hij yj + hii Li fˆ[i] .

Solving for Li fˆ[i] , we have Li fˆ − hii yi . Li fˆ[i] = 1 − hii Then

Li fˆ − hii yi Li fˆ − yi Li fˆ[i] − yi = − yi = . 1 − hii 1 − hii


67

Plugging into (3.21), we have CV(λ) =

n 1 X (Li fˆ − yi )2 . n i=1 (1 − hii )2

(3.23)

Therefore, the cross-validation criterion can be calculated using the fit based on the whole sample and the diagonal elements of the hat matrix. Replacing hii by the average of diagonal elements, trH(λ), we have the generalized cross-validation (GCV) criterion Pn 1 (Li fˆ − yi )2 GCV(λ) , n i=1 (3.24) 2 . 1 tr(I − H(λ)) n

The GCV estimate of λ is the minimizer of GCV(λ). Since trH(λ)/n is usually small in the neighborhood of the optimal λ, we have σ2 2σ 2 1 2 2 2 ||(I − H(λ))f || + trH (λ) + σ − trH(λ) E{GCV(λ)} ≈ n n n 2 1 + trH(λ) n = PSE(λ){1 + o(1)}.

The above approximation provides a very crude argument supporting the GCV criterion as a proxy for the PSE. More formally, under certain regularity conditions, GCV(λ) is a consistent estimate of the relative loss function. Furthermore, GCV(λ) is invariant to an orthogonal transformation of y. See Wahba (1990) and Gu (2002) for details. One distinctive advantage of the GCV criterion over the UBR criterion is that the former does not require an estimate of σ 2 . To illustrate the CV(λ) and GCV(λ) criteria as estimates of the PSE, we generate responses from model (1.1) with f (x) = sin(4πx2 ), σ = 0.5, xi = i/n for i = 1, . . . , n, and n = 73. For cubic periodic spline, the CV and GCV scores for 50 replications of simulation data are shown in Figure 3.6.

3.5

Bayes and Linear Mixed-Effects Models

Assume a prior for f as F (x) =

p X

ν=1

1

ζν φν (x) + δ 2 U (x),

(3.25)

68

Smoothing Splines: Methods and Applications 0.6

GCV

0.1

0.1

0.2

PSE and CV 0.3 0.4 0.5

PSE and GCV 0.2 0.3 0.4 0.5

0.6

CV

−8

−7

−6 −5 −4 log10(nλ)

−3

−2

−8

−7

−6 −5 −4 log10(nλ)

−3

−2

FIGURE 3.6 Plots of the PSE function as solid lines, the CV functions as dashed lines (left), and the GCV functions as dashed lines (right). The minimum point of the PSE is marked as long bars at the bottom. CV and GCV estimates of log10 (nλ) are marked as short bars. iid

where ζ1 , . . . , ζp ∼ N(0, κ), U (x) is a zero-mean Gaussian stochastic process with covariance function R1 (x, z), ζν and U (x) are independent, and κ and δ are positive constants. Note that the bounded linear functionals Li are defined for elements in H. Its application to the random process F (x) is yet to be defined. For simplicity, the subscript i in Li is ignored in the following definition. Define L(ζν φν ) = ζν Lφν . The definition of LU requires the duality between the Hilbert space spanned by a family of random variables and its associated RKHS. Consider the linear space n o X U= W : W = αj U (xj ), xj ∈ X , αj ∈ R

with inner product (W1 , W2 ) = E(W1 W2 ). Let L2 (U ) be the Hilbert space that is the completion of U. Note that the RK R1 of H1 coincides with the covariance function of U (x). Consider a linear map Ψ : H1 → L2 (U ) such that Ψ{R1 (xj , ·)} = U (xj ). Since (R1 (x, ·), R1 (z, ·)) = R1 (x, z) = E{U (x)U (z)} = (U (x), U (z)),

the map Ψ is inner product preserving. In fact, H1 is isometrically isomorphic to L2 (U ). See Parzen (1961) for details. Since L is a bounded linear functional in H1 , by the Riesz representation theorem, there exists a representer h such that Lf = (h, f ). Finally we define LU , Ψh.


69

Note that LU is a random variable in L2 (U ). The application of L to F Pp 1 is defined as LF = ν=1 ζν Lφν + δ 2 LU . When L is an evaluational functional, say Lf = f (x0 ) for a fixed x0 , we have h(·) = R1 (x0 , ·). Consequently, LU = Ψh = Ψ{R1 (x0 , ·)} = U (x0 ), the evaluation of U at x0 . Therefore, as expected, LF = F (x0 ) when L is an evaluational functional. Suppose observations are generated by yi = Li F + ǫi ,

i = 1, . . . , n,

(3.26)

iid

where the prior F is defined in (3.25) and ǫi ∼ N(0, σ 2 ). Note that the normality assumption has been made for random errors. We now compute the posterior mean E(L0 F |y) for a bounded linear functional L0 on H. Note that L0 is arbitrary, which could be quite different from Li . For example, suppose f ∈ W2m [a, b] and Li are evaluational functionals. Setting L0 f = f ′ (x0 ) leads to an estimate of f ′ . Using the correspondence between H and L2 (U ), we have E(Li U Lj U ) = (Li U, Lj U ) = (Li(x) R1 (x, ·), Lj(z) R1 (z, ·)) = Li(x) Lj(z) R1 (x, z),

E(L0 U Li U ) = (L0 U, Lj U ) = (L0(x) R1 (x, ·), Lj(z) R1 (z, ·)) = L0(x) Lj(z) R1 (x, z). Let ζ = (ζ1 , . . . , ζp )T , φ = (φ1 , . . . , φp )T and L0 φ = (L0 φ1 , . . . , L0 φp )T . 1 Then F (x) = φT (x)ζ + δ 2 U (x). It is easy to check that 1

y = T ζ + δ 2 (L1 U, . . . , Ln U )T + ǫ ∼ N(0, κT T T + δΣ + σ 2 I), (3.27) and 1

L0 F = (L0 φ)T ζ + δ 2 L0 U

∼ N(0, κ(L0 φ)T L0 φ + δL0(x) L0(z) R1 (x, z)).

(3.28)

E{(L0 F )y} = κT L0 φ + δL0 ξ,

(3.29)

Furthermore, where ξ(x) = (L1(z) R1 (x, z), . . . , Ln(z) R1 (x, z))T ,

L0 ξ = (L0(x) L1(z) R1 (x, z), . . . , L0(x) Ln(z) R1 (x, z))T .

(3.30)

70


Let λ = σ 2 /nδ and η = κ/δ. Using properties of multivariate normal random variables and equations (3.27), (3.28), and (3.29), we have E(L0 F |y) = (L0 φ)T ηT T (ηT T T + M )−1 y + (L0 ξ)T (ηT T T + M )−1 y. (3.31) It can be shown (Wahba 1990, Gu 2002) that for any full-column rank matrix T and symmetric and nonsingular matrix M , lim (ηT T T + M )−1 = M −1 − M −1 T (T T M −1 T )−1 T T M −1 ,

η→∞

lim ηT T (ηT T T + M )−1 = T (T T M −1 T )−1 T T M −1 .

η→∞

(3.32)

Combining results in (3.31), (3.32), and (2.22), we have lim E(L0 F |y) = (L0 φ)T T (T T M −1 T )−1 T T M −1 y

κ→∞

+ (L0 ξ)T {M −1 − M −1 T (T T M −1 T )−1 T T M −1 }y

= (L0 φ)T d + (L0 ξ)T c = L0 fˆ.

The above result indicates that the smoothing spline estimate fˆ is a Bayes estimator with a diffuse prior for ζ. From a frequentist perspective, the smoothing spline estimate may be regarded as the best linear unbiased prediction (BLUP) estimate of a linear mixed-effects (LME) model. We now present three corresponding LME models. The first LME model assumes that y = T ζ + u + ǫ,

(3.33)

where ζ = (ζ1 , . . . , ζp )T are deterministic parameters, u = (u1 , . . . , un )T are random effects with distribution u ∼ N(0, σ 2 Σ/nλ), ǫ = (ǫ1 , . . . , ǫn )T are random errors with distribution ǫ ∼ N(0, σ 2 I), and u and ǫ are independent. The second LME model assumes that y = T ζ + Σu + ǫ,

(3.34)

where ζ are deterministic parameters, u are random effects with distribution u ∼ N(0, σ 2 Σ+ /nλ), Σ+ is the Moore–Penrose inverse of Σ, ǫ are random errors with distribution ǫ ∼ N(0, σ 2 I), and u and ǫ are independent. It is inconvenient to use the above two LME models for computation since Σ may be singular. Write Σ = ZZ T , where Z is a n × m matrix with m = rank(Σ). The third LME model assumes that y = T ζ + Zu + ǫ,

(3.35)


71

where ζ are deterministic parameters, u are random effects with distribution u ∼ N(0, σ 2 I/nλ), ǫ are random errors with distribution ǫ ∼ N(0, σ 2 I), and u and ǫ are independent. It can be shown that the BLUP estimates for each of the three LME models (3.33), (3.34), and (3.35) are the same as the smoothing spline estimate. See Wang (1998b) and Chapter 9 for more details.

3.6

Generalized Maximum Likelihood

The connection between smoothing spline models and Bayes models can be exploited to develop a likelihood-based estimate for the smoothing parameter. From (3.27), the marginal distribution of y is N(0, δ(ηT T T + M )). Consider the following transformation QT2 w1 (3.36) = √1 T T y. w2 η It is easy to check that w1 = QT2 y ∼ N(0, δQT2 M Q2 ), δ Cov(w1 , w2 ) = √ QT2 (ηT T T + M )T → 0, η → ∞, η δ T Var(w 2 ) = T (ηT T T + M )T → δ(T T T )(T T T ), η → ∞. η Note that the distribution of w2 is independent of λ. Therefore, we consider the negative marginal log-likelihood of w 1 l(λ, δ|w1 ) =

1 1 log |δQT2 M Q2 | + w T1 (QT2 M Q2 )−1 w 1 + C1 , 2 2δ

(3.37)

where C1 is a constant. Minimizing l(λ, δ|w 1 ) with respect to δ, we have wT (QT M Q2 )−1 w 1 . δˆ = 1 2 n−p

(3.38)

The profile negative log-likelihood ˆ 1 ) = 1 log |QT M Q2 | + n − p log δˆ + C2 lp (λ, δ|w 2 2 2 n−p w T1 (QT2 M Q2 )−1 w1 = + C2 , log 1 2 {det(QT M Q2 )−1 } n−p 2

(3.39)

72


where C2 is another constant. The foregoing profile negative log-likelihood is equivalent to GML(λ) ,

w T1 (QT2 M Q2 )−1 w 1 {det(QT2 M Q2 )−1 }

1 n−p

=

y T (I − H(λ))y

+

1

[det {(I − H(λ))}] n−p

, (3.40)

where the second equality is based on (2.26), and det+ represents the product of the nonzero eigenvalues. The function GML(λ) is referred to as the generalized maximum likelihood (GML) criterion, and the minimizer of GML(λ) is called the GML estimate of the smoothing parameter. From (3.38), a likelihood-based estimate of σ 2 is σ ˆ2 ,

ˆ T (QT M Q2 )−1 w1 nλw y T (I − H(λ))y 1 2 = . n−p n−p

(3.41)

The GML criterion may also be derived from the connection between smoothing spline models and LME models. Consider any one of the three corresponding LME models (3.33), (3.34), and (3.35). The smoothing parameter λ is part of the variance component for the random effects. It is common practice in the mixed-effects literature to estimate the variance components using restricted likelihood based on an orthogonal contrast of original observations where the orthogonal contrast is used to eliminate the fixed effects. Note that w 1 is one such orthogonal contrast since Q2 is orthogonal to T . Therefore, l(λ, δ|w 1 ) in (3.37) is the negative log restricted likelihood, and the GML estimate of the smoothing parameter is the restricted maximum likelihood (REML) estimate. Furthermore, the estimate of error variance in (3.41) is the REML estimate of σ 2 . The connection between a smoothing spline estimate with GML estimate of the smoothing parameter and a BLUP estimate with REML estimate of the variance component in a corresponding LME model may be utilized to fit a smoothing spline model using software for LME models. This approach will be adopted in Chapters 5, 8, and 9 to fit smoothing spline models for correlated observations.

3.7

Comparison and Implementation

Theoretical properties of the UBR, GCV, and GML criteria can be found in Wahba (1990) and Gu (2002). The UBR criterion requires an estimate of the variance σ 2 . No distributional assumptions are required for the UBR and GCV criteria, while the normality assumption is required in


73

the derivation of the GML criterion. Nevertheless, limited simulations suggest that the GML method is quite robust to the departure from the normality assumption. Theoretical comparisons between UBR, GCV, and GML criteria have been studied using large-sample asymptotics (Wahba 1985, Li 1986, Stein 1990) and finite sample arguments (Efron 2001). Conclusions based on different perspectives do not always agree with each other. In practice, all three criteria usually perform well and lead to similar estimates. Each method has its own strengths and weaknesses. The UBR and GCV criteria occasionally lead to gross undersmooth (interpolation) when sample size is small. Fortunately, this problem diminishes quickly when sample size increases (Wahba and Wang 1995). The argument spar in the ssr function specifies which method should be used for selecting the smoothing parameter λ. The options spar=‘‘v’’, spar=‘‘m’’, and spar=‘‘u’’ correspond to the GCV, GML, and UBR methods, respectively. The default choice is the GCV method. We now use the motorcycle data to illustrate how to specify the spar option. For simplicity, the variable times is first scaled into [0, 1]. We first use the Rice method to estimate error variance and use the estimated variance in the UBR criterion: > x vrice mcycle.ubr.1 summary(mcycle.ubr.1) Smoothing spline regression fit by UBR method ... UBR estimate(s) of smoothing parameter(s) : 8.60384e-07 Equivalent Degrees of Freedom (DF): 12.1624 Estimate of sigma: 23.09297 The option varht specifies the parameter σ 2 required for the UBR method. The summary function provides a synopsis including the estimate of the smoothing parameter, the degrees of freedom trH(λ), and the estimate of standard deviation σ ˆ. Instead of the Rice estimator, we can estimate the error variance using 2 Tong and Wang’s estimator σ ˜TW . Note that there are multiple observations at some time points. We use these replicates to estimate the error variance. That is, we select all pairs with zero distances: > d mcycle.gml summary(mcycle.gml) Smoothing spline regression fit by GML method ... GML estimate(s) of smoothing parameter(s) : 4.729876e-07 Equivalent Degrees of Freedom (DF): 13.92711 Estimate of sigma: 22.57701 For the motorcycle data, all three methods lead to similar estimates of the smoothing parameter and the function f .


3.8 3.8.1

75

Confidence Intervals Bayesian Confidence Intervals

Consider the Bayes model (3.25) and (3.26). The computation in Section 3.5 can be carried out one step further to derive posterior distributions. In the following arguments, as in Section 3.5, a diffuse prior is assumed for ζ with κ → ∞. For simplicity of notation, the limit is not expressed explicitly. 1 Let F0ν = ζν φν for ν = 1, . . . , p, and F1 = δ 2 U . Let L0 , L01 , and L02 be bounded linear functionals. Since F0ν , F1 , and ǫi are all normal random variables, then the posterior distributions of L0 F0ν and L0 F1 are normal with the following mean and covariances. Posterior means and covariances For ν, µ = 1, . . . , p, the posterior means are E(L0 F0ν |y) = (L0 φν )eTν d,

(3.42)

E(L0 F1 |y) = (L0 ξ)T c,

and the posterior covariances are δ −1 Cov(L01 F0ν , L02 F0µ |y) = (L01 φν )(L02 φµ )eTν Aeµ , δ −1 Cov(L01 F0ν , L02 F1 |y) = −(L01 φν )eTν B(L02 ξ), δ

−1

(3.43)

T

Cov(L01 F1 , L02 F1 |y) = L01 L02 R1 − (L01 ξ) C(L02 ξ),

where eν is a vector of dimension p with the νth element being one and all other elements being zero, the vectors c and d are given in (2.22), and the matrices A = (T T M −1 T )−1 , B = AT T M −1 , and C = M −1 (I − B). The vectors ξ and Lξ are defined in (3.30). Proofs can be found in Wahba (1990) and Gu (2002). Note that H = span{φ1 , . . . , φp } ⊕ H1 . Then any f ∈ H can be represented as f = f01 + · · · + f0p + f1 ,

(3.44)

where f0ν ∈ span{φν } for ν = 1, . . . , p, and f1 ∈ H1 . The estimate fˆ can also be decomposed similarly fˆ = fˆ01 + · · · + fˆ0p + fˆ1 , where fˆ0ν = φν dν for ν = 1, . . . , p, and fˆ1 = ξ T c.

(3.45)

76


The functionals L0 , L01 , and L02 are arbitrary as long as they are well defined. Equations in (3.42) indicate that the posterior means of components in F equal their corresponding components in the spline estimate fˆ. Equations (3.42) and (3.43) can be used to compute posterior means and variances for any combinations of components of F . Specifically, consider the linear combination Fγ (x) =

p X

γν F0ν (x) + γp+1 F1 (x),

(3.46)

ν=1

where γν equals 1 when the corresponding component in F is to be included and 0 otherwise, and γ = (γ1 , . . . , γp+1 )T . Then, for any linear functional L0 , E(L0 Fγ |y) = Var(L0 Fγ |y) =

p X

γν (L0 φν )dν + γp+1 (L0 ξ)T c,

ν=1 p X p X

ν=1 µ=1 p X

+

ν=1

γν γµ Cov(L0 F0ν , L0 F0µ |y)

(3.47)

γν γp+1 Cov(L0 F0ν , L0 F1 |y)

+ γp+1 Cov(L0 F1 , L0 F1 |y). For various reasons it is often desirable to have interpretable confidence intervals for the function f and its components. For example, one may want to decide whether a nonparametric model is more suitable than a particular parametric model. A parametric regression model may be considered not suitable if a larger portion of its estimate is outside the confidence intervals of a smoothing spline estimate. Consider a collection of points x0j ∈ X , j = 1, . . . , J. For each j, posterior mean E{Fγ (x0j )|y} and variance Var{Fγ (x0j )|y} can be calculated using equations in (3.47) by setting L0 F = F (x0j ). Then, 100(1 − α)% Bayesian confidence intervals for fγ (x0j ) =

p X

γν f0ν (x0j ) + γp+1 f1 (x0j ),

j = 1, . . . , J

(3.48)

j = 1, . . . , J,

(3.49)

ν=1

are E{Fγ (x0j )|y} ± z α2

q Var{Fγ (x0j )|y},

where z α2 is the 1 − α/2 percentile of a standard normal distribution.


77

In particular, let F = (L1 F, . . . , Ln F )T . Applying (3.42) and (3.43), we have E(F |y) = H(λ)y,

Cov(F |y) = σ 2 H(λ).

(3.50)

Therefore, the posterior variances of the fitted values Var(Li F |y) = σ 2 hii , where hii are diagonal elements of the matrix H(λ). When Li are evaluational functionals Li f = f (xi ), Wahba (1983) proposed the following 100(1 − α)% confidence intervals p fˆ(xi ) ± z α2 σ (3.51) ˆ hii , where σ ˆ is an estimates of σ. Note that confidence intervals for a linear combination of components of f can be constructed similarly. Though based on the Bayesian argument, it has been found that the Bayesian confidence intervals have good frequentist properties provided that the smoothing parameter has been estimated properly. They must be interpreted as “across-the-function” rather than pointwise. More precisely, define the average coverage probability (ACP) as n

ACP =

1X P {f (xi ) ∈ C(α, xi )} n i=1

for some (1 − α)100% confidence intervals {C(α, xi ), i = 1, . . . , n}. Rather than considering a confidence interval for f (τ ), where f (·) is the realization of a stochastic process and τ is fixed, one may consider confidence intervals for f (τn ), where f is now a fixed function and τn is a point randomly selected from {xi , i = 1, . . . , n}. Then ACP = P {f (τn ) ∈ C(α, τn )}. Note that the ACP coverage property is weaker than the pointwise coverage property. For polynomial splines and C(α, xi ) being the Bayesian confidence intervals defined in (3.51), under certain regularity conditions, Nychka (1988) showed that ACP ≈ 1 − α. The predict function in the assist package computes the posterior mean and standard deviation of Fγ (x) in (3.46). The option terms specifies the coefficients γ, and the option newdata specifies a data frame consisting of the values at which predictions are required. We now use the geyser, motorcycle, and Arosa data to illustrate how to use the predict function to compute the posterior means and standard deviations. For the geyser data, we have fitted a cubic spline in Chapter 1, Section 1.1 and a partial spline in Chapter 2, Section 2.10. In the following we fit a cubic spline using the ssr function, compute posterior means and standard deviations for the estimate of the smooth component P1 f using the predict function, and plot the estimate of the smooth components and 95% Bayesian confidence intervals:

78


> geyser.cub.fit grid geyser.cub.pred grid1 plot(eruptions, waiting, xlab=‘‘duration (mins)’’, ylim=c(-6,6), ylab=‘‘smooth component (mins)’’, type=‘‘n’’) > polygon(c(grid1,rev(grid1)), c(geyser.cub.pred$fit-1.96*geyser.cub.pred$pstd, rev(geyser.cub.pred$fit+1.96*geyser.cub.pred$pstd)), col=gray(0:8/8)[8], border=NA) > lines(grid1, geyser.cub.pred$fit) > abline(0,0,lty=2) where the option pstd specifies whether the posterior standard deviations should be calculated. Note that the option pstd=T can be dropped in the above statement since it is the default. There are in total three components in the cubic spline fit: two basis functions φ1 (constant) and φ2 (linear) for the null space, and the smooth component in the space H1 . In the order in which they appear in the ssr function, these three components correspond to the intercept (~1, which is automatically included), the linear basis specified by ~x, and the smooth component specified by rk=cubic(x). Therefore, the option terms=c(0,0,1) was used to compute the posterior means and standard deviations for the smooth component f1 in the space H1 . The estimate of the smooth components and 95% Bayesian confidence intervals are shown in Figure 3.7(a). A large portion of the zero constant line is outside the confidence intervals, indicating the lack-of-fit of a linear model (the null space of the cubic spline). For the partial spline model (2.43), consider the following Bayes model yi = sTi β + Li F + ǫi ,

i = 1, . . . , n,

(3.52)

where the prior for β is assumed to be N(0, κIq ), the prior F is defined iid

in (3.25), and ǫi ∼ N(0, σ 2 ). Again, it can be shown that the PLS estimates of components in β and f based on (2.44) equal the posterior means of their corresponding components in the Bayes model as κ → ∞. Posterior covariances and Bayesian confidence intervals for β and fγ can be calculated similarly. We now refit the partial spline model (2.47) and compute posterior means and standard deviations for the estimate of the smooth component P1 f :

Smoothing Parameter Selection and Inference (b)

smooth component (min) −0.04 0.00 0.04

smooth component (min) −6 −4 −2 0 2 4 6

(a)

79

1.5


1.5


FIGURE 3.7 Geyser data, plots of estimates of the smooth components, and 95% Bayesian confidence intervals for (a) the cubic spline and (b) the partial spline models. The constant zero is marked as the dotted line in each plot. > t geyser.ps.fit geyser.ps.pred t))) Since there are in total four components in the partial spline fit, the option terms=c(0,0,0,1) was used to compute the posterior means and standard deviations for the smooth component f1 . Figure 3.7(b) shows the estimate of the smooth components and 95% Bayesian confidence intervals for the partial spline. The zero constant line is well inside the confidence intervals, indicating that this smooth component may be dropped in the partial spline model. That is, a simple linear change-point model may be appropriate for this data. For the motorcycle data, based on visual inspection, we have searched a potential change-point t to the first derivative in the interval [0.2, 0.25] for the variable x in Section 2.10. To search for all possible change-points to the first derivative, we fit the partial spline model (2.48) repeatedly with t taking values on a grid point in the interval [0.1, 0.9]. We then calculate the posterior mean and p standard deviation for β. Define a t-statistic at point t as E(β|y)/ Var(β|y). The t-statistics were calculated as follows: > tgrid > >

t1 geyser.ps.fit.m anova(geyser.ps.fit.m, simu.size=500) Testing H_0: f in the NULL space

LMP GML

test.value simu.size simu.p-value approximate.p-value 0.000352 500 0.634 1.000001 500 0.634 0.5

where the approximate.p-value was computed using the mixture of two Chi-square distributions. For the Arosa data, consider the hypothesis H0 : f ∈ P,

H1 : f ∈ W22 (per) and f ∈ / P,

where P = span{1, sin 2πx, cos 2πx} is the model space for the sinusoidal model. Two approaches can be used to test the above hypothesis: fit a partial spline and fit an L-spline:


89

> arosa.ps.fit anova(arosa.ps.fit,simu.size=500) Testing H_0: f in the NULL space test.value simu.size simu.p-value LMP 0.001262064 500 0 GCV 0.001832394 500 0 > arosa.ls.fit anova(arosa.ls.fit,simu.size=500) Testing H_0: f in the NULL space test.value simu.size simu.p-value LMP 2.539163e-06 500 0 GCV 0.001828071 500 0 The test based on the L-spline is usually more powerful since the parametric and the smooth components are orthogonal.


Chapter 4 Smoothing Spline ANOVA

4.1

Multiple Regression

Consider the problem of building regression models that examine the relationship between a dependent variable y and multiple independent variables x1 , . . . , xd . For generality, let the domain of each xk be an arbitrary set Xk . Denote x = (x1 , . . . , xd ). Given observations (xi , yi ) for i = 1, . . . , n, where xi = (xi1 , . . . , xid ), a multiple regression model relates the dependent variable and independent variables as follows: yi = f (xi ) + ǫi ,

i = 1, . . . , n,

(4.1)

where f is a multivariate regression function, and ǫi are zero-mean independent random errors with a common variance σ 2 . The goal is to construct a model for f and estimate it based on noisy data. There exist many different methods to construct a model space for f , parametrically, semiparametrically, or nonparametrically. For example, a thin-plate spline model may be used when all xk are univariate continuous variables, and partial spline models may be used if a linear parametric model can be assumed for all but one variable. This chapter introduces a nonparametric approach called smoothing spline analysis of variance (smoothing spline ANOVA or SS ANOVA) decomposition for constructing model spaces for the multivariate function f . The multivariate function f is defined on the product domain X = X1 ×X2 ×· · ·×Xd . Note that each Xk is arbitrary: it may be a continuous interval, a discrete set, a unit circle, a unit sphere, or Rd . Construction of model spaces for a single variable was introduced in Chapter 2. Let H(k) be an RKHS on Xk . The choice of the marginal space H(k) depends on the domain Xk and prior knowledge about f as a function of xk . To model the joint function f , we start with the tensor product of these marginal spaces defined in the following section.

91

92

4.2


Tensor Product Reproducing Kernel Hilbert Spaces

First consider the simple case when d = 2. Denote RKs for H(1) and H(2) as R(1) and R(2) , respectively. It is known that the product of nonnegative definite functions is nonnegative definite (Gu 2002). As RKs, both R(1) and R(2) are nonnegative definite. Therefore, the bivariate function on X = X1 × X2 R((x1 , x2 ), (z1 , z2 )) , R(1) (x1 , z1 )R(2) (x2 , z2 ) is nonnegative definite. By Moore–Aronszajn theorem, there exists a unique RKHS H on X = X1 × X2 such that R is its RK. The resulting RKHS H is called the tensor product RKHS and is denoted as H(1) ⊗H(2) . For d > 2, the tensor product RKHS of H(1) , . . . , H(d) on the product domain X = X1 × X2 × · · · × Xd , H(1) ⊗ H(2) ⊗ · · · ⊗ H(d) , is defined recursively. Note that the RK for a tensor product space equals the product of RKs of marginal spaces. That is, the RK of H(1) ⊗ H(2) ⊗ · · · ⊗ H(d) equals R(x, z) = R(1) (x1 , z1 )R(2) (x2 , z2 ) · · · R(d) (xd , zd ), where x ∈ X , z = (z1 , . . . , zd ) ∈ X , X = X1 × X2 × · · · × Xd , and R(k) is the RK of H(k) for k = 1, . . . , d. For illustration, consider the ultrasound data consisting of tongue shape measurements over time from ultrasound imaging. The data set contains observations on the response variable height (y) and three independent variables: environment (x1 ), length (x2 ), and time (x3 ). The variable x1 is a factor with three levels: x1 = 1, 2, 3 corresponding to 2words, cluster, and schwa, respectively. Both continuous variables x2 and x3 are scaled into [0, 1]. Interpolations of the raw data are shown in Figure 4.1. In linguistic studies, researchers want to determine (1) how tongue shapes for an articulation differ under different environments, (2) how the tongue shape changes as a function of time, and (3) how changes over time differ under different environments. To address the first question at a fixed time point, we need to model a bivariate regression function f (x1 , x2 ). Assume marginal spaces R3 and W2m [0, 1] for variables x1 and x2 , respectively. Then we may consider the tensor product space R3 ⊗ W2m [0, 1] for the bivariate function f . To address the second question, for a fixed environment, we need to model a bivariate regression function f (x2 , x3 ). Assume marginal spaces W2m1 [0, 1] and W2m2 [0, 1] for variables

Smoothing Spline ANOVA 70

65 60 55 50 45 40

60 55 50 45 40

2words 120

140

50 45

0

80

100

120

140

(mm) length

150 100 50

s) (m

le

m) ngth (m

55

e tim

100

60

schwa200

150 100 50

s) (m

s) (m

80

e tim

e tim

0

65

40

cluster 200

200 150 100 50

93

70

65

height (mm)

height (mm)

height (mm)

70

0

80

100

FIGURE 4.1 Ultrasound data, 3-d plots of observations.

x2 and x3 , respectively. Then we may consider the tensor product space W2m1 [0, 1] ⊗ W2m2 [0, 1] for the bivariate function. To address the third question, we need to model a trivariate regression function f (x1 , x2 , x3 ). We may consider the tensor product space R3 ⊗ W2m1 [0, 1] ⊗ W2m2 [0, 1] for the trivariate function. Analysis of the ultrasound data are given in Section 4.9.1. The SS ANOVA decomposition decomposes a tensor product space into subspaces with a hierarchical structure similar to the main effects and interactions in the classical ANOVA. The resulting hierarchical structure facilitates model selection and interpretation. Sections 4.3, 4.4, and 4.5 present SS ANOVA decompositions for a single space, tensor product of two spaces, and tensor product of d spaces. More SS ANOVA decompositions can be found in Sections 4.9, 5.4.4, 6.3, and 9.2.4.

4.3

120

140

)

(mm length

One-Way SS ANOVA Decomposition

SS ANOVA decompositions of tensor product RKHS’s are based on decompositions of marginal spaces for each independent variable. Therefore, decompositions for a single space are introduced first in this section. Spline models for a single independent variable have been introduced in Chapter 2. Denote the independent variable as x and the regression

94


function as f . It was assumed that f belongs to an RKHS H. The function f was decomposed into a parametric and a smooth component, f = f0 + f1 , or in terms of model space, H = H0 ⊕ H1 . This can be regarded as one form of the SS ANOVA decomposition. We now introduce general decompositions based on averaging operators. Consider a function space H on the domain X . An operator A is called an averaging operator if A = A2 . Instead of the more appropriate term as an idempotent operator, the term averaging operator is used since it is motivated by averaging in the classical ANOVA decomposition. Note that an averaging operator does not necessarily involve averaging. As we will see in the following subsections, the commonly used averaging operators are projection operators. Thus they are idempotent. Suppose model space H = H0 ⊕ H1 , where H0 is a finite dimensional space with orthogonal basis φ1 (x), . . . , φp (x). Let Aν be the projection operator onto the subspace {φν (x)} for ν = 1, . . . , p, and Ap+1 be the projection operator onto H1 . Then the function can be decomposed as f = (A1 + · · · + Ap + Ap+1 )f , f01 + · · · + f0p + f1 .

(4.2)

Correspondingly, the model space is decomposed into H = {φ1 (x)} ⊕ · · · ⊕ {φp (x)} ⊕ H1 . For simplicity, {·} represents the space spanned by the basis functions inside the bracket. Some averaging operators Aν (subspaces) can be combined. For example, combining A1 , . . . , Ap leads to the decomposition f = f0 + f1 , a parametric component plus a smooth component. When φ1 (x) = 1, combining A2 , . . . , Ap+1 leads to the decomposition f = f01 + f˜1 , where f01 is a constant independent of x, and f˜1 = f02 + · · · + f0p + f1 collects all components that depend on x. Therefore, f = f01 + f˜1 decomposes the function into a constant plus a nonconstant function. For the same model space, different SS ANOVA decompositions may be constructed for different purposes. In general, we denote the one-way SS ANOVA decomposition as f = A1 f + · · · + Ar f,

(4.3)

where A1 + · · · + Ar = I, and I is the identity operator. The above equality always holds since f = If . Equivalently, in terms of the model space, the one-way SS ANOVA decomposition is denoted as H = H(1) ⊕ · · · ⊕ H(r) . The following subsections provide one-way SS ANOVA decompositions for some special model spaces.

Smoothing Spline ANOVA

4.3.1

95

Decomposition of Ra : One-Way ANOVA

Suppose x is a discrete variable with a levels. The classical one-way mean model assumes that yik = µi + ǫik ,

i = 1, . . . , a; k = 1, . . . , ni ,

(4.4)

where yik represents the observation of the kth replication at level i of x, µi represents the mean at level i, and ǫik represent random errors. Regarding µi as a function of i and expressing explicitly as f (i) , µi , then f is a function defined on the discrete domain X = {1, . . . , a}. It is easy to see that the model space for f is the Euclidean a-space Ra . Model space construction and decomposition of Ra The space Ra is an RKHS with the inner product (f, g) = f T g. Furthermore, Ra = H0 ⊕ H1 , where H0 = {f : f (1) = · · · = f (a)}, a X H1 = {f : f (i) = 0},

(4.5)

i=1

are RKHS’s with corresponding RKs R0 (i, j) =

1 , a

1 R1 (i, j) = δi,j − , a

(4.6)

and δi,j is the Kronecker delta. Details about the above construction can be found in Gu (2002). Define an averaging operator A1 : Ra → Ra such that a

1X A1 f = f (i). a i=1 The operator A1 maps f to the constant function that equals the average over all indices. Let A2 = I − A1 . The one-way ANOVA effect model is based on the following decomposition of the function f : f = A1 f + A2 f , µ + αi ,

(4.7)

where µ is the overall mean, and αi is the effect at level i. From the definition Pa of the averaging operator, αi satisfy the sum-to-zero side condition i=1 αi = 0. It is clear that A1 and A2 are the projection operators

96


from Ra onto H0 and H1 defined in (4.5). Thus, they divide Ra into H0 and H1 . Pa Under the foregoing construction, ||P1 f ||2 = i=1 {f (i) − f¯}2 , where P a f¯ = i=1 f (i)/a. For balanced designs with ni = n, the solution to the PLS (2.11) is aλ fˆ(i) = y¯i· − (¯ yi· − y¯·· ), (4.8) 1 + aλ Pn Pa where y¯i· = ¯·· = ¯i· /a. k=1 yik /n, and y i=1 y PaIt is easy to check that when n = 1, σ 2 = 1 and λ = (a − 3)/a( i=1 y¯i·2 − a + 3), the spline estimate fˆ is the James–Stein estimator (shrink toward mean). Therefore, in a sense, the Stein’s shrinkage estimator can be regarded as spline estimate on a discrete domain. There exist other ways to decompose Ra . For example, the averaging operator A1 f = f (1) leads to the same decomposition as (4.7) with the set-to-zero side condition: α1 = 0 (Gu 2002).

4.3.2

Decomposition of W2m [a, b]

Under the construction in Section 2.2, let Aν f (x) = f (ν−1) (a)

(x − a)ν−1 , (ν − 1)!

ν = 1, . . . , m.

(4.9)

It is easy to see that A2v = Av . Thus, they are averaging operators. In fact, Aν is the projection operator onto {(x − a)ν−1 /(ν − 1)!}. Let Am+1 = I − A1 − · · · − Am . The decomposition f = A1 f + · · · + Am f + Am+1 f corresponds to the Taylor expansion (1.5). It decomposes the model space W2m [a, b] into W2m [a, b] = {1} ⊕ {x − a} ⊕ · · · ⊕ {(x − a)m−1 /(m − 1)!} ⊕ H1 , where H1 is given in (2.3). For f ∈ H1 , conditions f (ν) (a) = 0 for ν = 0, . . . , m−1 are analogous to the set-to-zero condition in the classical one-way ANOVA model. Under the construction in Section 2.6 for W2m [0, 1], let Z 1 (ν−1) Aν f (x) = f (u)du kν (x), ν = 1, . . . , m. 0

Again, Aν is an averaging (projection) operator extracting the polynoR1 mial of order ν. In particular, A1 f = 0 f (u)du is a natural extension


97

of the averaging in the discrete domain. Let Am+1 = I − A1 − · · · − Am . The decomposition f = A1 f + · · · + Am f + Am+1 f decomposes the model space W2m [0, 1] into W2m [0, 1] = {1} ⊕ {k1 (x)} ⊕ · · · ⊕ {km−1 (x)} ⊕ H1 , R1 where H1 is given in (2.29). For f ∈ H1 , conditions 0 f (ν) dx = 0 for ν = 0, . . . , m − 1 are analogous to the sum-to-zero condition in the classical one-way ANOVA model.

4.3.3

Decomposition of W2m (per)

Under the construction in Section 2.7, let Z 1 A1 f = f du 0

be an averaging (projection) operator. Let A2 = I − A1 . The decomposition f = A1 f + A2 f decomposes the model space W2m (per) = 1 ⊕ f ∈ W2m (per) :

4.3.4

Z

0

1

f du = 0 .

Decomposition of W2m (Rd )

For simplicity, consider the special case with d = 2 and m = 2. Decompositions for general d and m can be derived similarly. Let φ1 (x) = 1, φ2 (x) = x1 , and φ3 (x) = x2 be polynomials of total degree less than m = 2. Let φ˜1 = 1, and φ˜2 and φ˜3 be an orthonormal basis such that (φ˜ν , φ˜µ )0 = δν,µ based on the norm (2.41). Define two averaging operators A1 f (x) = A2 f (x) =

J X

wj f (uj ),

j=1

J X j=1

(4.10) wj f (uj ) φ˜2 (uj )φ˜2 (x) + φ˜3 (uj )φ˜3 (x) ,

98


where uj are fixed points in R2 , and wj are fixed positive weights such PJ that j=1 wj = 1. It is clear that A1 and A2 are projection operators onto spaces {φ˜1 } and {φ˜2 , φ˜3 }, respectively. To see how they generalize averaging operators, define a probability measure µ on X = R2 by assigning probability wj to the point uj , j = 1, . . . , J. Then A1 f (x) = R R R f φ˜1 dµ and A2 f (x) = ( R2 f φ˜2 dµ)φ˜2 (x) + ( R2 f φ˜3 dµ)φ˜3 (x). ThereR2 fore, A1 and A2 take averages with respect to the discrete probability measure µ. In particular, µ puts mass 1/n on design points xj when J = n, wj = 1/n and uj = xj . A continuous density on R2 may be used. However, the resulting integrals usually do not have closed forms, and approximations such as a quadrature formula would have to be used. It is then essentially equivalent to using an approximate discrete probability measure. Let A3 = I − A1 − A2 . The decomposition f = A1 f + A2 f + A3 f divides the model space W2m (R2 ) = 1 ⊕ φ˜2 , φ˜3 ⊕ f ∈ W2m (R2 ) : J22 (f ) = 0 , d where Jm (f ) is defined in (2.36).

4.4

Two-Way SS ANOVA Decomposition

Suppose there are two independent variables, x1 ∈ X1 and x2 ∈ X2 . Consider the tensor product space H(1) ⊗ H(2) on the product domain X1 × X2 . For f as a marginal function of xk , assume the following one-way decomposition based on Section 4.3, (k)

f = A1 f + · · · + A(k) rk f, (k)

k = 1, 2,

where Aj are averaging operators on H(k) and for the joint function, we have

P rk

j=1

(4.11) (k)

Aj

= I. Then

r1 X r2 X (1) (2) (1) (2) (2) f = A1 +· · ·+A(1) A +· · ·+A f = Aj1 Aj2 f. (4.12) r1 r2 1 j1 =1 j2 =1

The above decomposition of the bivariate function f is referred to as the two-way SS ANOVA decomposition.


99

Denote (k)

(k)

H(k) = H(1) ⊕ · · · ⊕ H(rk ) ,

k = 1, 2,

as the one-way decomposition to H(k) associated with (4.11). Then, (4.12) decomposes the tensor product space n o n o (1) (1) (2) (2) H(1) ⊗ H(2) = H(1) ⊕ · · · ⊕ H(r1 ) ⊗ H(1) ⊕ · · · ⊕ H(r2 ) =

r1 X r2 X

j1 =1 j2 =1

(1)

(2)

H(j1 ) ⊗ H(j2 ) .

Consider the special case when rk = 2 for k = 1, 2. Assume that (k) is independent of xk , or equivalently, H0 = {1}. Then the decomposition (4.12) can be written as (k) A1 f

(1)

(2)

(1)

(2)

(1)

(2)

(1)

(2)

f = A1 A1 f + A2 A1 f + A1 A2 f + A2 A2 f , µ + f1 (x1 ) + f2 (x2 ) + f12 (x1 , x2 ),

(4.13)

where µ represents the grand mean, f1 (x1 ) and f2 (x2 ) represent the main effects of x1 and x2 , respectively, and f12 (x1 , x2 ) represents the interaction between x1 and x2 . (k) For general rk , assuming A1 f is independent of xk , the decomposi(k) (k) tion (4.13) can be derived by combining operators A2 , . . . , Ark into (k) (k) (k) one averaging operator A˜2 = A2 + · · · + Ark . Therefore, decomposition (4.13) combines components in (4.12) and reorganizes them into the overall main effects and interactions. The following subsections provide two-way SS ANOVA decompositions for combinations of some special model spaces.

4.4.1

Decomposition of Ra ⊗ Rb : Two-Way ANOVA

Suppose both x1 and x2 are discrete variables with a and b levels, respectively. The classical two-way mean model assumes that yijk = µij + ǫijk ,

i = 1, . . . , a; j = 1, . . . , b; k = 1, . . . , nij ,

where yijk represents the observation of the kth replication at level i of x1 , and level j of x2 , µij represents the mean at level i of x1 and level j of x2 , and ǫijk represent random errors. Regarding µij as a bivariate function of (i, j) and letting f (i, j) , µij , then f is a bivariate function defined on the product domain X = {1, . . . , a}×{1, . . . , b}. The model space for f is the tensor product space

100

Smoothing Splines: Methods and Applications (1)

Ra ⊗ Rb . Define two averaging operators A1

(2)

and A2

such that

a

(1)

A1 f =

1X f (i, j), a i=1 b

(2)

A1 f = (1)

1X f (i, j). b j=1

(2)

A1 and A1 map f to univariate functions by averaging over all levels (1) (1) (2) (2) of x1 and x2 , respectively. Let A2 = I −A1 and A2 = I −A1 . Then the classical two-way ANOVA effect model is based on the following SS ANOVA decomposition of the function f : (1)

(1)

(2)

(2)

f = (A1 + A2 )(A1 + A2 )f (1)

(2)

(1)

(2)

(1)

(2)

(1)

(2)

= A1 A1 f + A2 A1 f + A1 A2 f + A2 A2 f , µ + αi + βj + (αβ)ij ,

(4.14)

where µ represents the overall mean, αi represents the main effect of x1 , βj represents the main effect of x2 , and (αβ)ij represents the interaction between x1 and x2 . The sum-to-zero side conditions are satisfied from the definition of the averaging operators. Based on one-way ANOVA decomposition in Section 4.3.1, we have (1) (1) (2) (2) (1) (2) Ra = H0 ⊕ H1 and Rb = H0 ⊕ H1 , where H0 and H0 are (1) (2) subspaces containing constant functions, and H1 and H1 are orthog(1) (2) onal complements of H0 and H0 , respectively. The classical two-way ANOVA model decomposes the tensor product space n o n o (1) (1) (2) (2) Ra ⊗ Rb = H0 ⊕ H1 ⊗ H0 ⊕ H1 n o n o (1) (2) (1) (2) = H0 ⊗ H0 ⊕ H1 ⊗ H0 n o n o (1) (2) (1) (2) ⊕ H0 ⊗ H1 ⊕ H1 ⊗ H1 . (4.15)

The four subspaces in (4.15) contain components µ, αi , βj , and (αβ)ij in (4.14), respectively.

4.4.2

Decomposition of Ra ⊗ W2m [0, 1]

Suppose x1 is a discrete variable with a levels, and x2 is a continuous variable in [0, 1]. A natural model space for x1 is Ra and a natural model space for x2 is W2m [0, 1]. Therefore, we consider the tensor product space Ra ⊗ W2m [0, 1] for the bivariate regression function f (x1 , x2 ). For


101

simplicity, we derive SS ANOVA decompositions for m = 1 and m = 2 only. SS ANOVA decompositions for higher-order m can be derived similarly. In this and the remaining sections, the construction in Section 2.6 for the marginal space W2m [0, 1] will be used. Similar SS ANOVA decompositions can be derived under the construction in Section 2.2. Consider the tensor product space Ra ⊗ W21 [0, 1] first. Define two (1) (2) averaging operators A1 and A1 as a 1 X f, a x =1 1 Z 1 (2) A1 f = f dx2 , (1)

A1 f =

0

(1)

(2)

where A1 and A1 extract the constant term out of all possible func(1) (1) (2) (2) tions for each variable. Let A2 = I − A1 and A2 = I − A1 . Then (1) (1) (2) (2) f = A1 + A2 A1 + A2 f (1)

(2)

(1)

(2)

(1)

(2)

(1)

(2)

= A1 A1 f + A2 A1 f + A1 A2 f + A2 A2 f , µ + f1 (x1 ) + f2 (x2 ) + f12 (x1 , x2 ).

(4.16)

Obviously, (4.16) is a natural extension of the classical two-way ANOVA decomposition (4.14) from the product of two discrete domains to the product of one discrete and one continuous domain. As in the classical ANOVA model, components in (4.16) have nice interpretations: µ represents the overall mean, f1 (x1 ) represents the main effect of x1 , f2 (x2 ) represents the main effect of x2 , and f12 (x1 , x2 ) represents the interaction. Components also have nice interpretation collectively: µ + f2 (x2 ) represents the mean curve among all levels of x1 , and f1 (x1 )+f12 (x1 , x2 ) represents the departure from the mean curve at level x1 . Write Ra = (1) (1) (2) (2) (1) (1) H0 ⊕ H1 and W21 [0, 1] = H0 ⊕ H1 , where H0 and H1 are given R 1 (2) (2) in (4.5), H0 = {1} and H1 = {f ∈ W21 [0, 1] : 0 f du = 0}. Then, in terms of the model space, (4.16) decomposes Ra ⊗ W21 [0, 1] n o n o (1) (1) (2) (2) = H0 ⊕ H1 ⊗ H0 ⊕ H1 n o n o n o n o (1) (2) (1) (2) (1) (2) (1) (2) = H0 ⊗ H0 ⊕ H1 ⊗ H0 ⊕ H0 ⊗ H1 ⊕ H1 ⊗ H1

, H0 ⊕ H1 ⊕ H2 ⊕ H3 .

(4.17)

To fit model (4.16), we need to find basis for H0 and RKs for H1 , H2 , and H3 . It is clear that H0 contains all constant functions. Thus, H0

102


is an one-dimensional space with the basis φ(x) = 1. The RKs of H0 (1) (2) (2) and H1 are given in (4.6), and the RKs of H0 and H1 are given in 1 2 3 Table 2.2. The RKs of H , H , and H can be calculated using the fact that the RK of a tensor product space equals the product of RKs of the (1) (2) involved marginal spaces. For example, the RK of H3 = H1 ⊗ H1 equals (δx1 ,z1 − 1/a){k1 (x2 )k1 (z2 ) + k2 (|x2 − z2 |)}. Now suppose we want to model the effect of x2 using the cubic spline space W22 [0, 1]. Consider the tensor product space Ra ⊗ W22 [0, 1]. Define (1) (2) (2) three averaging operators A1 , A1 , and A2 as a 1 X f, a x =1 1 Z 1 (2) A1 f = f dx2 , 0 Z 1 (2) ′ A2 f = f dx2 (x2 − 0.5), (1)

A1 f =

0

(1)

(2)

where A1 and A1 extract the constant function out of all possible (2) functions for each variable, and A2 extracts the linear function for x2 . (1) (1) (2) (2) (2) Let A2 = I − A1 and A3 = I − A1 − A2 . Then (1) (1) (2) (2) (2) f = A1 + A2 A1 + A2 + A3 f (1)

(2)

(1)

(2)

(1)

(2)

= A1 A1 f + A1 A2 f + A1 A3 f (1)

(2)

(1)

(2)

(1)

(2)

+ A2 A1 f + A2 A2 f + A2 A3 f

, µ + β × (x2 − 0.5) + f2s (x2 )

ss + f1 (x1 ) + γx1 × (x2 − 0.5) + f12 (x1 , x2 ),

(4.18)

where µ represents the overall mean, f1 (x1 ) represents the main effect of x1 , β ×(x2 −0.5) represents the linear main effect of x2 , f2s (x2 ) represents the smooth main effect of x2 , γx1 ×(x2 −0.5) represents the smooth–linear ss interaction, and f12 (x1 , x2 ) represents the smooth–smooth interaction. The overall main effect of x2 f2 (x2 ) = β × (x2 − 0.5) + f2s (x2 ), and the overall interaction between x1 and x2 ss f12 (x1 , x2 ) = γx1 × (x2 − 0.5) + f12 (x1 , x2 ).

It is obvious that f2 and f12 are the results of combining averaging (2) (3) operators A2 and A2 . One may look at the components in the overall


103

main effects and interactions to decide whether to include them in the model. The first three terms in (4.18) represent the mean curve among all levels of x1 , and the last three terms represent the departure from the mean curve. The simple ANCOVA (analysis of covariance) model with x2 being modeled by a straight line is a special case of (4.18) with ss ss f2s = f12 = 0. Thus, checking whether f2s and f12 are negligible provides (1) (1) a diagnostic tool for the ANCOVA model. Write Ra = H0 ⊕ H1 and (2) (2) (2) (1) (1) W22 [0, 1] = H0 ⊕ H1 ⊕ H2 , where H0 and H1 are given in (4.5), R1 (2) (2) (2) H0 = {1}, H1 = {x2 − 0.5}, and H2 = {f ∈ W22 [0, 1], 0 f du = R1 ′ f du = 0}. Then, in terms of the model space, (4.18) decomposes 0 Ra ⊗ W22 [0, 1] o n o n (2) (2) (2) (1) (1) ⊗ H0 ⊕ H1 ⊕ H2 = H0 ⊕ H1 n o n o n o (1) (2) (1) (2) (1) (2) = H0 ⊗ H0 ⊕ H0 ⊗ H1 ⊕ H0 ⊗ H2 n o n o n o (1) (2) (1) (2) (1) (2) ⊕ H1 ⊗ H0 ⊕ H1 ⊗ H1 ⊕ H1 ⊗ H2 , H0 ⊕ H1 ⊕ H2 ⊕ H3 ⊕ H4 , (1)

(2)

(1)

(2)

(1)

(4.19)

(2)

where H0 = {H0 ⊗ H0 } ⊕ {H0 ⊗ H1 }, H1 = H0 ⊗ H2 , H2 = (1) (2) (1) (2) (1) (2) H1 ⊗ H0 , H3 = H1 ⊗ H1 , and H4 = H1 ⊗ H2 . It is easy to see 0 that H is a two-dimensional space with basis functions φ1 (x) = 1, and φ2 (x) = x2 − 0.5. RKs of H1 , H2 , H3 , and H4 can be calculated from (1) (1) (2) (2) (2) RKs of H0 and H1 given in (4.6) and the RKs of H0 , H1 , and H2 given in Table 2.2.

4.4.3

Decomposition of W2m1 [0, 1] ⊗ W2m2 [0, 1]

Suppose both x1 and x2 are continuous variables in [0, 1]. W2m [0, 1] is a natural model space for both effects of x1 and x2 . Therefore, we consider the tensor product space W2m1 [0, 1] ⊗ W2m2 [0, 1]. For simplicity, we will derive SS ANOVA decompositions for combinations m1 = m2 = 1 and m1 = m2 = 2 only. SS ANOVA decompositions for other combinations of m1 and m2 can be derived similarly. Consider the tensor product space W21 [0, 1] ⊗ W21 [0, 1] first. Define two averaging operators as (k)

A1 f = (k)

where A1

Z

1

f dxk ,

k = 1, 2,

0

extracts the constant term out of all possible functions of

104

Smoothing Splines: Methods and Applications (k)

(k)

xk . Let A2 = I − A1

for k = 1, 2. Then

(1) (1) (2) (2) f = A1 + A2 A1 + A2 f (1)

(2)

(1)

(2)

(1)

(2)

(1)

(2)

= A1 A1 f + A2 A1 f + A1 A2 f + A2 A2 f , µ + f1 (x1 ) + f2 (x2 ) + f12 (x1 , x2 ).

(4.20)

Obviously, (4.20) is a natural extension of the classical two-way ANOVA decomposition (4.14) from the product of two discrete domains to the product of two continuous domains. Components µ, f1 (x1 ), f2 (x2 ), and f12 (x1 , x2 ) represent the overall mean, the main effect of x1 , the main effect of x2 , and the interaction between x1 and x2 , respectively. In terms of the model space, (4.20) decomposes W21 [0, 1] ⊗ W21 [0, 1] o n o n (1) (1) (2) (2) = H0 ⊕ H1 ⊗ H0 ⊕ H1 n o n o n o n o (1) (2) (1) (2) (1) (2) (1) (2) = H0 ⊗ H0 ⊕ H1 ⊗ H0 ⊕ H0 ⊗ H1 ⊕ H1 ⊗ H1

, H0 ⊕ H1 ⊕ H2 ⊕ H3 ,

R1 (k) (k) where H0 = {1} and H1 = {f ∈ W21 [0, 1] : 0 f dxk = 0} for (1) (2) (1) (2) (1) (2) 0 1 2 k = 1, 2, H = H0 ⊗ H0 , H = H1 ⊗ H0 , H = H0 ⊗ H1 , and (1) (2) H3 = H1 ⊗ H1 . H0 is an one-dimensional space with basis φ(x) = 1. (k) The RKs of H1 , H2 , and H3 can be calculated from RKs of H0 and (k) H1 given in Table 2.2. Now suppose we want to model both x1 and x2 using cubic splines. That is, we consider the tensor product space W22 [0, 1]⊗W22 [0, 1]. Define four averaging operators

(k)

A1 f = (k)

A2 f = (1)

(2)

Z

1

f dxk , 0

Z

0

1

f ′ dxk (xk − 0.5),

k = 1, 2,

where A1 and A1 extract the constant function out of all possible (1) (2) functions for each variable, and A2 and A2 extract the linear function

Smoothing Spline ANOVA (k)

(k)

(k)

for each variable. Let A3 = I − A1 − A2

105

for k = 1, 2. Then

(1) (1) (1) (2) (2) (2) f = A1 + A2 + A3 A1 + A2 + A3 f (1)

(2)

(1)

(2)

(1)

(2)

= A1 A1 f + A1 A2 f + A1 A3 f (1)

(2)

(1)

(2)

(1)

(2)

(1)

(2)

(1)

(2)

(1)

(2)

+ A2 A1 f + A2 A2 f + A2 A3 f + A3 A1 f + A3 A2 f + A3 A3 f

, µ + β2 × (x2 − 0.5) + f2s (x2 )

ls + β1 × (x1 − 0.5) + β3 × (x1 − 0.5) × (x2 − 0.5) + f12 (x1 , x2 )

sl ss + f1s (x1 ) + f12 (x1 , x2 ) + f12 (x1 , x2 ),

(4.21)

where µ represents the overall mean; β1 × (x1 − 0.5) and β2 × (x2 − 0.5) represent the linear main effects of x1 and x2 ; f1s (x1 ) and f2s (x2 ) represent the smooth main effect of x1 and x2 ; β3 × (x1 − 0.5) × (x2 − ls sl ss 0.5), f12 (x1 , x2 ), f12 (x1 , x2 ) and f12 (x1 , x2 ) represent the linear–linear, linear–smooth, smooth–linear, and smooth–smooth interactions between x1 and x2 . The overall main effect of xk fk (xk ) = βk × (xk − 0.5) + fks (xk ),

k = 1, 2,

and the overall interaction between x1 and x2 ls sl f12 (x1 , x2 ) = β3 × (x1 − 0.5) × (x2 − 0.5) + f12 (x1 , x2 ) + f12 (x1 , x2 ) ss + f12 (x1 , x2 ).

The simple regression model with both x1 and x2 being modeled by ls sl straight lines is a special case of (4.21) with f1s = f2s = f12 = f12 = ss f12 = 0. In terms of the model space, (4.21) decomposes W22 [0, 1] ⊗ W22 [0, 1] n o n o (1) (1) (1) (2) (2) (2) = H0 ⊕ H1 ⊕ H2 ⊗ H0 ⊕ H1 ⊕ H2 n o n o n o (1) (2) (1) (2) (1) (2) = H0 ⊗ H0 ⊕ H1 ⊗ H0 ⊕ H2 ⊗ H0 n o n o n o (1) (2) (1) (2) (1) (2) ⊕ H0 ⊗ H1 ⊕ H1 ⊗ H1 ⊕ H2 ⊗ H1 n o n o n o (1) (2) (1) (2) (1) (2) ⊕ H0 ⊗ H2 ⊕ H1 ⊗ H2 ⊕ H2 ⊗ H2 , H0 ⊕ H1 ⊕ H2 ⊕ H3 ⊕ H4 ⊕ H5 ,

R1 (k) (k) (k) where H0 = {1}, H1 = {xk − 0.5}, and H2 = {f ∈ W22 [0, 1] : 0 f R1 (1) (2) (1) (2) dxk = 0 f ′ dxk = 0} for k = 1, 2, H0 = {H0 ⊗ H0 } ⊕ {H1 ⊗ H0 } ⊕

106


(1)

(2)

(1)

(2)

(1)

(2)

(0)

(2)

{H0 ⊗ H1 } ⊕ {H1 ⊗ H1 }, H1 = H2 ⊗ H0 , H2 = H2 ⊗ H1 , (1) (2) (1) (2) (1) (2) H3 = H0 ⊗ H2 , H4 = H1 ⊗ H2 , and H5 = H2 ⊗ H2 . H0 is a four-dimensional space with basis functions φ1 (x) = 1, φ2 (x) = x1 − 0.5, φ3 (x) = x2 − 0.5, and φ4 (x) = (x1 − 0.5) × (x2 − 0.5). The RKs of H1 , (k) (k) (k) H2 , H3 , H4 and H5 can be calculated from RKs of H0 , H1 , and H2 given in Table 2.2.

4.4.4

Decomposition of Ra ⊗ W2m (per)

Suppose x1 is a discrete variable with a levels, and x2 is a continuous variable in [0, 1]. In addition, suppose that f is a periodic function of x2 . A natural model space for x1 is Ra , and a natural model space for x2 is W2m (per). Therefore, we consider the tensor product space Ra ⊗ W2m (per). (1) (2) Define two averaging operators A1 and A1 as a 1 X f, a x =1 1 Z 1 (2) f dx2 . A1 f = (1)

A1 f =

0

(1)

(1)

Let A2 = I − A1

(2)

(2)

and A2 = I − A1 . Then

(1) (1) (2) (2) f = A1 + A2 A1 + A2 f (1)

(2)

(1)

(2)

(1)

(2)

(1)

(2)

= A1 A1 f + A2 A1 f + A1 A2 f + A2 A2 f , µ + f1 (x1 ) + f2 (x2 ) + f12 (x1 , x2 ),

(4.22)

where µ represents the overall mean, f1 (x1 ) represents the main effect of x1 , f2 (x2 ) represents the main effect of x2 , and f12 (x1 , x2 ) represents the (1) (1) interaction between x1 and x2 . Write Ra = H0 ⊕ H1 and W2m (per) = (2) (2) (1) (1) (2) H0 ⊕ H1 , where H0 and H1 are given in (4.5), H0 = {1}, and R 1 (2) H1 = {f ∈ W2m (per) : 0 f du = 0}. Then, in terms of the model space, (4.22) decomposes Ra ⊗ W2m (per) n o n o (1) (1) (2) (2) = H0 ⊕ H1 ⊗ H0 ⊕ H1 n o n o n o n o (1) (2) (1) (2) (1) (2) (1) (2) = H0 ⊗ H0 ⊕ H1 ⊗ H0 ⊕ H0 ⊗ H1 ⊕ H1 ⊗ H1

, H0 ⊕ H1 ⊕ H2 ⊕ H3 ,

Smoothing Spline ANOVA (1)

(2)

(1)

107

(2)

(1)

(2)

where H0 = H0 ⊗ H0 , H1 = H1 ⊗ H0 , H2 = H0 ⊗ H1 , and (1) (2) H3 = H1 ⊗ H1 . H0 is an one-dimensional space with basis function φ(x) = 1. The RKs of H1 , H2 , and H3 can be calculated from RKs of (1) (1) (2) (2) H0 and H1 given in (4.6) and RKs of H0 and H1 given in (2.33).

4.4.5

Decomposition of W2m1 (per) ⊗ W2m2 [0, 1]

Suppose both x1 and x2 are continuous variables in [0, 1]. In addition, suppose f is a periodic function of x1 . A natural model space for x1 is W2m1 (per), and a natural model space for x2 is W2m2 [0, 1]. Therefore, we consider the tensor product space W2m1 (per)⊗ W2m2 [0, 1]. For simplicity, we derive the SS ANOVA decomposition for m2 = 2 only. Define three averaging operators (1)

A1 f = (2)

A1 f = (2) A2 f (1)

(1)

Let A2 = I − A1

=

Z Z

1

f dx1 , 0 1

f dx2 , 0

Z

0

1

′

f dx2 (x2 − 0.5).

(2)

(2)

(2)

and A3 = I − A1 − A2 . Then

n on o (1) (1) (2) (2) (2) f = A1 + A2 A1 + A2 + A3 f (1)

(2)

(1)

(2)

(1)

(2)

= A1 A1 f + A1 A2 f + A1 A3 f (1)

(2)

(1)

(2)

(1)

(2)

+ A2 A1 f + A2 A2 f + A2 A3 f

, µ + β × (x2 − 0.5) + f2s (x2 )

sl ss + f1 (x1 ) + f12 (x1 , x2 ) + f12 (x1 , x2 ),

(4.23)

where µ represents the overall mean, f1 (x1 ) represents the main effect of x1 , β × (x2 − 0.5) and f2s (x2 ) represent the linear and smooth main sl ss effects of x2 , f12 (x1 , x2 ) and f12 (x1 , x2 ) represent the smooth–linear and smooth–smooth interactions. The overall main effect of x2 f2 (x2 ) = β × (x2 − 0.5) + f2s (x2 ), and the overall interaction between x1 and x2 sl ss f12 (x1 , x2 ) = f12 (x1 , x2 ) + f12 (x1 , x2 ).

108


(1)

(2)

(2)

(2)

Write W2m1 (per) = H0 ⊕ H1 and W22 [0, 1] = H0 ⊕ H1 ⊕ H2 , R1 (1) (1) (2) where H0 = {1}, H1 = {f ∈ W2m1 (per) : 0 f du = 0}, H0 = {1}, R R 1 1 (2) (2) H1 = {x2 − 0.5}, and H2 = {f ∈ W22 [0, 1] : 0 f du = 0 f ′ du = 0}. Then, in terms of the model space, (4.23) decomposes W2m1 (per) ⊗ W22 [0, 1] n o n o (1) (1) (2) (2) (2) = H0 ⊕ H1 ⊗ H0 ⊕ H1 ⊕ H2 n o n o n o (1) (2) (1) (2) (1) (2) = H0 ⊗ H0 ⊕ H0 ⊗ H1 ⊕ H0 ⊗ H2 n o n o n o (1) (2) (1) (2) (1) (2) ⊕ H1 ⊗ H0 ⊕ H1 ⊗ H1 ⊕ H1 ⊗ H2 , H0 ⊕ H1 ⊕ H2 ⊕ H3 ⊕ H4 , (1)

(2)

(1)

(2)

(1)

(2)

where H0 = {H0 ⊗ H0 } ⊕ {H0 ⊗ H1 }, H1 = H0 ⊗ H2 , H2 = (1) (2) (1) (2) (1) (2) H1 ⊗ H0 , H3 = H1 ⊗ H1 , and H4 = H1 ⊗ H2 . H0 is a twodimensional space with basis functions φ(x) = 1 and φ(x) = x2 − 0.5. (1) The RKs of H1 , H2 , H3 , and H4 can be calculated from the RKs H0 (1) (2) (2) (2) and H1 given in (2.33) and the RKs of H0 , H1 , and H2 given in Table 2.2.

4.4.6

Decomposition of W22 (R2 ) ⊗ W2m (per)

Suppose x1 = (x11 , x12 ) is a bivariate continuous variable in R2 , and x2 is a continuous variable in [0, 1]. In addition, suppose that f is a periodic function of x2 . We consider the tensor product space W22 (R2 )⊗ W22 (per) for the joint regression function f (x1 , x2 ). Let φ1 (x1 ) = 1, φ2 (x1 ) = x11 , and φ3 (x1 ) = x12 be polynomials of total degree less than 2. Define three averaging operators (1)

A1 f = (1)

A2 f = (2)

A1 f =

J X

wj f (uj ),

j=1

J X j=1

Z

wj f (uj ){φ˜2 (uj )φ˜2 + φ˜3 (uj )φ˜3 },

1

f dx2 ,

0

where uj are fixed points in R2 , wj are fixed positive weights such that PJ ˜ ˜ ˜ j=1 wj = 1, φ1 = 1, and φ2 and φ3 are orthonormal bases based on

Smoothing Spline ANOVA (1)

(1)

(1)

the norm (2.41). Let A3 = I − A1 − A2

109 (2)

(2)

and A2 = I − A1 . Then

n on o (1) (1) (1) (2) (2) f = A1 + A2 + A3 A1 + A2 f (1)

(2)

(1)

(2)

(1)

(2)

= A1 A1 f + A2 A1 f + A3 A1 f (1)

(2)

(1)

(2)

(1)

(2)

+ A1 A2 f + A2 A2 f + A3 A2 f = µ + β1 φ˜2 (x1 ) + β2 φ˜3 (x1 ) ls ss + f1s (x1 ) + f2 (x2 ) + f12 (x1 , x2 ) + f12 (x1 , x2 ),

(4.24)

where µ is the overall mean, β1 φ˜2 (x1 ) + β2 φ˜3 (x1 ) is the linear main effect of x1 , f1s (x1 ) is the smooth main effect of x1 , f2 (x2 ) is the main ls ss effect of x2 , f12 (x1 , x2 ) is the linear–smooth interaction, and f12 (x1 , x2 ) is smooth–smooth interaction. The overall main effect of x1 f1 (x1 ) = β1 φ˜2 (x1 ) + β2 φ˜3 (x1 ) + f1s (x1 ), and the overall interaction ls ss f12 (x1 , x2 ) = f12 (x1 , x2 ) + f12 (x1 , x2 ). (1)

(1)

(1)

(2)

(2)

Write W22 (R2 ) = H0 ⊕ H1 ⊕ H2 and W2m (per) = H0 ⊕ H1 , where (1) (1) (1) H0 = {1}, H1 = {φ˜2 , φ˜3 }, H2 = {f ∈ W22 (R2 ) : J22 (f ) = 0}, R1 (2) (2) m1 H0 = {1}, and H1 = {f ∈ W2 (per) : 0 f du = 0}. Then, in terms of the model space, (4.24) decomposes W22 (R2 ) ⊗ W2m (per) n o n o (1) (1) (1) (2) (2) = H0 ⊕ H1 ⊕ H2 ⊗ H0 ⊕ H1 n o n o n o (1) (2) (1) (2) (1) (2) = H0 ⊗ H0 ⊕ H1 ⊗ H0 ⊕ H2 ⊗ H0 n o n o n o (1) (2) (1) (2) (1) (2) ⊕ H0 ⊗ H1 ⊕ H1 ⊗ H1 ⊕ H2 ⊗ H1 , H0 ⊕ H1 ⊕ H2 ⊕ H3 ⊕ H4 , (1)

(2)

(1)

(2)

(1)

(2)

(4.25)

where H0 = {H0 ⊗ H0 } ⊕ {H1 ⊗ H0 }, H1 = {H2 ⊗ H0 }, H2 = (1) (2) (1) (2) (1) (2) {H0 ⊗ H1 }, H3 = {H1 ⊗ H1 }, and H4 = {H2 ⊗ H1 }. The basis (1) (1) functions of H0 are 1, φ˜2 , and φ˜3 . The RKs of H0 and H1 are 1 and (1) φ˜2 (x1 )φ˜2 (z 1 ) + φ˜3 (x1 )φ˜3 (z 1 ), respectively. The RK of H2 is given in (2) (2) (2.42). The RKs of H0 and H1 are given in (2.33). The RKs of H1 , 2 3 4 H , H , and H can be calculated as products of the RKs of the involved marginal spaces.

110


4.5

General SS ANOVA Decomposition

Consider the general case with d independent variables x1 ∈ X1 , x2 ∈ X2 , . . . , xd ∈ Xd , and the tensor product space H(1) ⊗H(2) ⊗· · ·⊗H(d) on X = X1 × X2 × · · · × Xd . For f as a function of xk , assume the following one-way decomposition as in (4.3), (k)

f = A1 f + · · · + A(k) rk f, (k)

1 ≤ k ≤ d,

(4.26)

(k)

where A1 + · · · + Ark = I. Then, for the joint function, (1) (d) f = A1 + · · · + A(1) . . . A1 + · · · + A(d) r1 rd f rd r1 X X (1) (d) = ... Aj1 . . . Ajd f. j1 =1

(4.27)

jd =1

The above decomposition of the function f is referred to as the SS ANOVA decomposition. Denote (k) (k) H(k) = H(1) ⊕ · · · ⊕ H(rk ) , k = 1, 2, . . . , d as the one-way decomposition to H(k) associated with (4.26). Then, (4.27) decomposes the tensor product space H(1) ⊗ H(2) ⊗ · · · ⊗ H(d) n o n o (1) (1) (d) (d) = H(1) ⊕ · · · ⊕ H(r1 ) ⊗ · · · ⊗ H(1) ⊕ · · · ⊕ H(rd ) =

r1 X

j1 =1

...

rd X

jd =1

(1)

(1)

(d)

H(j1 ) ⊗ . . . ⊗ H(jd ) . (d)

The RK of H(j1 ) ⊗ · · · ⊗ H(jd ) equals (k)

Qd

k=1

(4.28) (k)

(k)

R(jk ) , where R(jk ) is the RK

of H(jk ) for k = 1, . . . , d. Consider the special case when rk = 2 for all k = 1, . . . , d. Assume (k) (k) that A1 f is independent of xk , or equivalently, H(1) = {1}. Then the decomposition (4.27) can be written as ( ) X Y (k) Y (k) f= A2 A1 f B⊆{1,...,d}

=µ+

d X

k=1

k∈B c

k∈B

fk (xk ) +

X k > > > >

data(ultrasound) ultrasound$y predict(ultra.el.c.fit, terms=c(1,1,1,1,1,1), newdata=expand.grid(x2=grid,x1=as.factor(1:3))) The default for the option terms is a vector of all 1’s. Therefore, this option can be dropped in the above statement. The fits and 95% Bayesian confidence intervals are shown in Figure 4.2.

120


Note that in model (4.18) the first three terms represent the mean curve among three environments and the last three terms represent the departure of a particular environment from the mean curve. For comparison, we compute the estimate of the mean curve among three environments: > predict(ultra.el.c.fit, terms=c(1,1,0,1,0,0), newdata=expand.grid(x2=grid,x1=as.factor(1))) The estimate of the mean curve is also displayed in Figure 4.2. The difference between the tongue shape under a particular environment and the average tongue shape can be made by comparing two lines in each plot. To look at the effect of each environment more closely, we calculate the estimate of the departure from the mean curve for each environment: > predict(ultra.el.c.fit, terms=c(0,0,1,0,1,1), newdata=expand.grid(x2=grid,x1=as.factor(1:3))) The estimates of environment effects are shown in Figure 4.4. We can see that, comparing to the average shape, the tongue shape for 2words is front-raising, and the tongue shape for cluster is back-raising. The tongue shape for schwa is close to the average shape.

2words

schwa

−5

height (mm) 0 5

cluster

80

100 120 140 length (mm)

80

100 120 140 length (mm)

80

100 120 140 length (mm)

FIGURE 4.4 Ultrasound data, plots of effects of environment, and 95% Bayesian confidence intervals. The dashed line in each plot represents the constant function zero. The model space of the SS ANOVA model (4.18) is M = H0 ⊕ H1 ⊕ H ⊕ H3 ⊕ H4 , where Hj for j = 0, . . . , 4 are defined in (4.19). In 2


121

particular, the spaces H3 and H4 contain smooth–linear and smooth– smooth interactions between x1 and x2 . For illustration, suppose now that we want to fit model (4.18) with the same smoothing parameter for penalties to functions in H3 and H4 . That is to set λ3 = λ4 in the PLS. As discussed in Section 4.6, this can be achieved by combining H3 and H4 into one space. The following statements fit the SS ANOVA model (4.18) with λ3 = λ4 : > ultra.el.c.fit1 summary(ultra.el.c.fit1) ... GCV estimate(s) of smoothing parameter(s) : 5.648863e-02 2.739598e-05 3.766634e-05 Equivalent Degrees of Freedom (DF): 13.52603 Estimate of sigma: 2.65806 Next we investigate how the tongue shapes change over time for each environment. Figure 4.1 shows 3-d plots of observations. For a fixed environment, consider a bivariate regression function f (x2 , x3 ) where both x2 and x3 are continuous variables. Therefore, we model the joint function using the tensor product space W2m1 [0, 1] ⊗ W2m2 [0, 1]. The SS ANOVA decompositions of W2m1 [0, 1] ⊗ W2m2 [0, 1] were presented in Section 4.4.3. Note that variables x2 and x3 in this section correspond to x1 and x2 in Section 4.4.3. The following statements fit the tensor product of linear splines with m1 = m2 = 1, that is, the SS ANOVA model (4.20), under environment 2words (x1 = 1): > ultrasound$x3 ssr(height~1, data=ultrasound, subset=ultrasound$env==1, rk=list(linear(x2), linear(x3), rk.prod(linear(x2),linear(x3)))) The following statements fit the tensor product of cubic splines with m1 = m2 = 2, that is, the SS ANOVA model (4.21), under environment 2words and calculate estimates at grid points: > ultra.lt.c.fit[[1]] grid ultra.lt.c.pred predict(ultra.lt.c.fit[[1]], term=c(0,0,1,1,0,1,1,1,1), newdata=expand.grid(x2=grid,x3=grid))

100

120

length

140

(mm)


123

Figure 4.6 shows the contour plots of the estimated time effect for three environments. Regions where the lower bounds of the 95% Bayesian confidence intervals are positive are shaded in dark grey, while regions where the upper bounds of the 95% Bayesian confidence intervals are negative are shaded in light grey.

2words

cluster

200

10

4

2

schwa

5

0

6

0

−5

6

2

4

time (ms) 100 150

0 −4

−2

−3 4

−2

0

0

5

0

50

6

2

10

2

2

6

4

0

0

4

80

100 120 140 length (mm)

80

100 120 140 length (mm)

80

100 120 140 length (mm)

FIGURE 4.6 Ultrasound data, contour plots of the estimated time effect for three environments based on the SS ANOVA model (4.21). Regions where the lower bounds of the 95% Bayesian confidence intervals are positive are shaded in dark grey. Regions where the upper bounds of the 95% Bayesian confidence intervals are negative are shaded in light grey.

Finally we investigate how the changes of tongue shapes over time differ among different environments. Consider a trivariate regression function f (x1 , x2 , x3 ) in tensor product space R3 ⊗W2m1 [0, 1]⊗W2m2 [0, 1]. For simplicity, we derive the SS ANOVA decomposition for m1 = m2 = 2 only. Define averaging operators: 3 1 X f, 3 x =1 1 Z 1 (k) A1 f = f dxk , 0 Z 1 (k) A2 f = f ′ dxk (xk − 0.5), (1)

A1 f =

k = 2, 3,

0

(1)

(2)

(3)

where A1 , A1 , and A1

extract the constant function out of all pos-

124


(3)

sible functions for each variable, and A2 and A2 extract the linear (1) (1) (k) (k) (k) function for x2 and x3 . Let A2 = I − A1 and A3 = I − A1 − A2 for k = 2, 3. Then (1) (1) (2) (2) (2) (3) (3) (3) f = A1 + A2 A1 + A2 + A3 A1 + A2 + A3 f (1)

(2)

(3)

(1)

(2)

(1)

(1)

(2)

(3)

(3)

(1)

(2)

(2)

(3)

(1)

(1)

(2)

(3)

(1)

(2)

(1)

(2)

(1)

(2)

(3)

(3)

(1)

(2)

(3)

(2)

(3)

(1)

(2)

(3)

(1)

(2)

(3)

(1)

(2)

(3)

(3)

(1)

(2)

(3)

(1)

(2)

(3)

(3)

(1)

(2)

(3)

(1)

(2)

(3)

= A1 A1 A1 f + A1 A1 A2 f + A1 A1 A3 f

+ A1 A2 A1 f + A1 A2 A2 f + A1 A2 A3 f + A1 A3 A1 f + A1 A3 A2 f + A1 A3 A3 f + A2 A1 A1 f + A2 A1 A2 f + A2 A1 A3 f + A2 A2 A1 f + A2 A2 A2 f + A2 A2 A3 f + A2 A3 A1 f + A2 A3 A2 f + A2 A3 A3 f

, µ + β2 × (x3 − 0.5) + f3s (x3 )

ls + β1 × (x2 − 0.5) + β3 × (x2 − 0.5)(x3 − 0.5) + f23 (x2 , x3 )

ss sl (x2 , x3 ) + f23 (x2 , x3 ) + f2s (x2 ) + f23 ss sl + f1 (x1 ) + f13 (x1 , x3 ) + f13 (x1 , x3 )

sl sll sls + f12 (x1 , x2 ) + f123 (x1 , x2 , x3 ) + f123 (x1 , x2 , x3 ) ss ssl sss + f12 (x1 , x2 ) + f123 (x1 , x2 , x3 ) + f123 (x1 , x2 , x3 ),

(4.50)

where µ represents the overall mean; f1 (x1 ) represents the main effect of x1 ; β1 × (x2 − 0.5) and β2 × (x3 − 0.5) represent the linear main effects of x2 and x3 ; f2s (x2 ) and f3s (x3 ) represent the smooth main effects of x2 and sl sl x3 ; f12 (x1 , x2 ) (f13 (x1 , x3 )) represents the smooth–linear interaction bels sl tween x1 and x2 (x3 ); β3 × (x2 − 0.5) × (x3 − 0.5), f23 (x2 , x3 ), f23 (x2 , x3 ) ss and f23 (x2 , x3 ) represent linear–linear, linear–smooth, smooth–linear sll and smooth–smooth interactions between x2 and x3 ; and f123 (x1 , x2 , x3 ), sls ssl sss f123 (x1 , x2 , x3 ), f123 (x1 , x2 , x3 ), and f123 (x1 , x2 , x3 ) represent three-way interactions between x1 , x2 , and x3 . The overall main effect of xk , fk (xk ), equals βk−1 × (xk − 0.5) + fks (xk ) for k = 2, 3. The overall intersl ss action between x1 and xk , f1k (x1 , xk ), equals f12 (x1 , xk )+f1k (x1 , xk ) for k = 2, 3. The overall interaction between x2 and x3 , f23 (x2 , x3 ), equals ls sl ss β3 × (x2 − 0.5) × (x3 − 0.5) + f23 (x2 , x3 ) + f23 (x2 , x3 ) + f23 (x2 , x3 ). The sll overall three-way interaction, f123 (x1 , x2 , x3 ), equals f123 (x1 , x2 , x3 ) + sls ssl sss f123 (x1 , x2 , x3 ) + f123 (x1 , x2 , x3 ) + f123 (x1 , x2 , x3 ). We fit model (4.50) as follows: > ssr(height~I(x2-.5)+I(x3-.5)+I((x2-.5)*(x3-.5)), data=ultrasound, rk=list(shrink1(x1),

80

height (mm)

height (mm)

height (mm)

80


70 60 50

70

60 50 40

1.0 0.8 0.6 0.4 0.2 0.0 0.0

1.0 0.8 0.6 0.4 0.2 0.0 0.0

le

m) ngth (m

1.0

50

cluster

0.2

0.4

0.6

0.8

len

) gth (mm

schwa

1.0

1.0 0.8 0.6 0.4 0.2 0.0 0.0

s) (m

0.4

0.8

s) (m

s) (m

0.2

0.6

60

e tim

2words

125

70

40

e tim

e tim

40

80

0.2

FIGURE 4.7 Ultrasound data, 3-d plots of the estimated tongue shape as a function of environment, length and time based on the SS ANOVA model (4.50). cubic(x2), cubic(x3), rk.prod(shrink1(x1),kron(x2-.5)), rk.prod(shrink1(x1),cubic(x2)), rk.prod(shrink1(x1),kron(x3-.5)), rk.prod(shrink1(x1),cubic(x3)), rk.prod(cubic(x2),kron(x3-.5)), rk.prod(kron(x2-.5),cubic(x3)), rk.prod(cubic(x2),cubic(x3)), rk.prod(shrink1(x1),kron(x2-.5),kron(x3-.5)), rk.prod(shrink1(x1),kron(x2-.5),cubic(x3)), rk.prod(shrink1(x1),cubic(x2),kron(x3-.5)), rk.prod(shrink1(x1),cubic(x2),cubic(x3)))) The estimates of all three environments are shown in Figure 4.7. Note that the first nine terms in (4.50) represent the mean tongue shape surface over time, and the last nine terms in (4.50) represent the departure of an environment from this mean surface. To look at the environment effect on the tongue shape surface over time, we calculate the posterior mean and standard deviation of the departure for each environment: > pred Arosa$x1 Arosa$x2 arosa.ssanova.fit1 summary(arosa.ssanova.fit1) ... GCV estimate(s) of smoothing parameter(s) : 5.442106e-06 2.154531e-09 3.387917e-06 2.961559e-02 Equivalent Degrees of Freedom (DF): 50.88469 Estimate of sigma: 14.7569 The mean function f (x) in model (4.23) evaluated at design points f = (f (x1 ), . . . , f (xn ))T can be represented as f = µ1 + f 1 + f 2 + f 12 , where 1 is a vector of all ones, f 1 = (f1 (x1 ), . . . , f1 (xn ))T , f 2 = (f2 (x1 ), . . . , f2 (xn ))T , f 12 = (f12 (x1 ), . . . , f12 (xn ))T , and f1 (x), f2 (x) and f12 (x1 ) are the main effect of x1 , the main effect of x2 , and the interaction between x1 and x2 . Eliminating the constant term, we have f ∗ = f ∗1 + f ∗2 + f ∗12 , Pn ∗ ∗ ∗ ∗ where a∗ = a − a ¯1, and a ¯ = i=1 ai /n. Let fˆ , fˆ 1 , fˆ 2 , and fˆ 12 be the ∗ ∗ ∗ ∗ estimates of f , f 1 , f 2 , and f 12 , respectively. To check the contributions of the main effects and interaction, we compute the quantities πk = ∗ ∗ ∗ ∗ ∗ ∗ (fˆ k )T fˆ /||fˆ ||2 for k = 1, 2, 12, and the Euclidean norms of fˆ , fˆ 1 , fˆ 2 , ∗ and fˆ 12 : > > > > > > > > >

f1 r predict(mcycle.ps.fit3, newdata=data.frame(x=grid, s1=(grid-t1)*(grid>t1), s2=(grid-t2)*(grid>t2), s3=(grid-t3)*(grid>t3)))

(a)

(b)

o

0

10

acceleration (g) −50 0 50

o

20 30 40 time (ms)

50

60

o o

o o ooo o oo o o oo o o o oo o o o o oo o o o o oooo ooooooooo oooo o oo o ooo oo o o o o o ooo oo o o o o o o o o ooooo oo o o o oo o oo ooo o oo o oo oo o o o oo o oo o o oo o oo o o o o o ooo oo o o

−150

log(squared residual) −5 0 5

o o ooo oo o oo o o o oo o o oooo ooo ooo o o oo o o o o o ooooooo oo o oo o ooo oo o o oo o oo ooo o o o ooooo o o o o o o o o o oo o o o o o o o o o o o o o o o oo o o o oo o oo o oo o o o o o o o o oo o o o o o o o o

0

10

20 30 40 time (ms)

50

60

FIGURE 5.4 Motorcycle data, plots of (a) logarithm of the squared residuals (circles) based on model (3.53), and the cubic spline fit (line) to logarithm of the squared residuals; and (b) observations (circles), new PWLS fit (line), and 95% Bayesian confidence intervals (shaded region).

The PWLS fit and 95% Bayesian confidence intervals are shown in 5.4(b). The impact of the unequal variances is reflected in the widths of confidence intervals. The two-step approach adapted here is crude, and the variation in the estimation of the variance function has been ignored in the construction of the confidence intervals. Additional methods for estimating the mean and variance functions will be discussed in Section 6.4.

5.4.2

Ozone in Arosa — Revisit

Figure 2.2 suggests that the variances may not be a constant. Based on fit to the trigonometric spline model (2.75), we calculate residual variances for each month and plot them on the logarithm scale in Figure 5.5(a). It is obvious that variations depend on the time of the year. It seems that a simple sinusoidal function can be used to model the variance function.

Spline Smoothing with Heteroscedastic and/or Correlated Errors 155 > v a b summary(b) ... Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 5.43715 0.05763 94.341 8.57e-15 *** sin(2 * pi * a) 0.71786 0.08151 8.807 1.02e-05 *** cos(2 * pi * a) 0.49854 0.08151 6.117 0.000176 *** --Residual standard error: 0.1996 on 9 degrees of freedom Multiple R-squared: 0.9274, Adjusted R-squared: 0.9113 F-statistic: 57.49 on 2 and 9 DF, p-value: 7.48e-06

log(variance) 5.5 6.5

(a)

(b) o

o

o

o o

o o o

o

o

thickness 300 350 400

o

4.5

o

1

3

5

7 month

9

11

1

3

5

7 month

9

11

FIGURE 5.5 Arosa data, plots of (a) logarithm of residual variances (circles) based on the periodic spline fit in Section 2.7, the sinusoidal fit to logarithm of squared residuals (dashed line), and the fit from model (5.31) (solid line); and (b) observations (dots), the new PWLS fit (solid line), and 95% Bayesian confidence intervals (shaded region).

The fit of the simple sinusoidal model to the log variance is shown in Figure 5.5(a). We now assume the following variance function for the trigonometric spline model (2.75): v(x) = exp(ζ1 sin 2πx + ζ2 cos 2πx), and fit the model as follows:

(5.31)

156


> arosa.ls.fit1 summary(arosa.ls.fit1) ... GML estimate(s) of smoothing parameter(s) : 3.675780e-09 Equivalent Degrees of Freedom (DF): 6.84728 Estimate of sigma: 15.22466 Combination of: Variance function structure of class varExp representing expon 0.3555942 Variance function structure of class varExp representing expon 0.2497364 The estimated variance parameters, 0.3556 and 0.2497, are very close to (up to a scale of 2 by definition) those in the sinusoidal model based on residual variances — 0.7179 and 0.4985. The fitted variance function is plotted in Figure 5.5(a) which is almost identical to the fit based on residual variances. Figure 5.5(b) plots the trigonometric spline fit to the mean function with 95% Bayesian confidence intervals. Note that these confidence intervals are conditional on the estimated variance parameters. Thus they may have smaller coverage than the nominal value since variation in the estimation of variance parameters is not counted. Nevertheless, we can see that unequal variances are reflected in the widths of these confidence intervals. Observations close in time may be correlated. We now consider a first-order autoregressive structure for random errors. Since some observations are missing, we use the continuous AR(1) correlation structure h(s, ρ) = ρs , where s represents distance in terms of calendar time between observations. We refit model (2.75) using the variance structure (5.31) and continuous AR(1) correlation structure as follows: > Arosa$time arosa.ls.fit2 summary(arosa.ls.fit2) ... GML estimate(s) of smoothing parameter(s) : 3.575283e-09 Equivalent Degrees of Freedom (DF): 7.327834 Estimate of sigma: 15.31094 Correlation structure of class corCAR1 representing

Spline Smoothing with Heteroscedastic and/or Correlated Errors 157 Phi 0.3411414 Combination of: Variance function structure of class varExp representing expon 0.3602905 Variance function structure of class varExp representing expon 0.3009282

where the variable time represents the continuous calendar time in months.

5.4.3

Beveridge Wheat Price Index

6.0

The Beveridge data contain the time series of annual wheat price index from 1500 to 1869. The time series of price index on the logarithmic scale is shown in Figure 5.6.

o o ooo o o o o o o o o o oo o o o o oo o o o o o o o oo o ooo oo o o o o o o oo o ooo o o oo oo o o o o o oo oo o o o o oo o o o o o oo oo oo o oo oo ooo o o o o ooo ooo o o o o o o oo o o o oo oo o o o o oo o ooo oo oo oo o o oo o o o o o o o o o o oo ooo ooo o o o o o oo oo o o o o o oo o o o o ooo ooo oo o o o o o oo o o ooo ooo o o oo o o o o o o o oooooo o o o o oo o oo o o o o o ooo o o o o o oo o o o o o o o o o oo o o o o o o o oo o o ooo o o o o o o o o oo oo o o o ooo o oo o o o o o o ooo o oo o o o o o o o oo o o o o o o o o o o oo o oo oo o o o

3.0

log of price index 3.5 4.0 4.5 5.0

5.5

o

o o

2.5

oooo

oo oo oo o o oo o o o oo oo o o oo oo o o o o o oooo o

o o o

o o

o

o

1500

1600

1700 year

1800

FIGURE 5.6 Beveridge data, observations (circles), the cubic spline fit under the assumption of independent random errors (dashed line), and the cubic spline fit under AR(1) correlation structure (solid line) with 95% Bayesian confidence intervals (shaded region).

158


Let x be the year scaled into the interval [0, 1] and y be the logarithm of price index. Consider the following nonparametric regression model yi = f (xi ) + ǫi ,

i = 1, . . . , n,

(5.32)

where f ∈ W22 [0, 1] and ǫi are random errors. Under the assumption that ǫi are independent with a common variance, model (5.32) can be fitted as follows: > library(tseries); data(bev) > y bev.fit2 summary(bev.fit2) GML estimate(s) of smoothing parameter(s) : 1.249805e-06 Equivalent Degrees of Freedom (DF): 8.091024 Estimate of sigma: 0.2243519 Correlation structure of class corAR1 representing Phi 0.6936947 Estimate of f and 95% Bayesian confidence intervals under AR(1) correlation structure are shown in Figure 5.6 as the solid line and the shaded region.

5.4.4

Lake Acidity

The lake acidity data contain measurements of 112 lakes in the southern Blue Ridge mountains area. It is of interest to investigate the dependence of the water pH level on calcium concentration and geological location. To match notations in Chapter 4, we relabel calcium concentration t1

Spline Smoothing with Heteroscedastic and/or Correlated Errors 159 as variable x1 , and geological location x1 (latitude) and x2 (longitude) as variables x21 and x22 , respectively. Let x2 = (x21 , x22 ). First, we fit an one-dimensional thin-plate spline to the response variable ph using one independent variable x1 (calcium): ph(xi1 ) = f (xi1 ) + ǫi ,

(5.33)

where f ∈ W22 (R), and ǫi are zero-mean independent random errors with a common variance. > data(acid) > acid$x21 anova(acid.tp.fit1) Testing H_0: f in the NULL space test.value simu.size simu.p-value LMP 0.003250714 100 0.08 GCV 0.008239078 100 0.01 Both p-values from the LMP and GCV tests suggest that the departure from a straight line model is borderline significant. The estimate of the function f in model (5.33) is shown in the left panel of Figure 5.7. Observations of the pH level close in geological locations are often correlated. Suppose that we want to model potential spatial correlation among random errors in model (5.33) using the exponential spatial correlation structure with nugget effect for the location variable x2 = (x21 , x22 ). That is, we assume that (1 − c0 ) exp(−s/ρ), s > 0, hnugg (s, c0 , ρ) = 1, s = 0, where c0 is the nugget effect and s is the Euclidean distance between two geological locations. Model (5.33) with the above correlation structure can be fitted as follows:

160


8.0

pH

7.5

7.0 0.2

6.5

pH

6.0

0.0 −0.2

1.0

1.5

−0.4 −0.02

0.02 0.00 −0.02

e

0.5

0.00

0.02

latitude

git ud

0.0

calcium(log10 mg/L)

lon

−0.5

FIGURE 5.7 Lake acidity data, the left panel includes observations (circles), the fit from model (5.33) (solid line), the fit from model (5.33) with the exponential spatial correlation structure (dashed line), and estimate of the constant plus main effect of x1 from model (5.36) (dotted line); the right panel includes estimate of the main effect of x2 from model (5.36). > acid.tp.fit2 summary(acid.tp.fit2) ... GML estimate(s) of smoothing parameter(s) : 53310.63 Equivalent Degrees of Freedom (DF): 2 Estimate of sigma: 0.3131702 Correlation structure of class corExp representing range nugget 0.02454532 0.62321744 The GML estimate of the smoothing parameter is large, and the spline estimate is essentially a straight line (left panel of Figure 5.7). The smaller smoothing parameter in the first fit with independence assumption might be caused by the spatial correlation. Equivalent degrees of freedom for f have been reduced from 8.2 to 2. We can also model the effect of geological location directly in the mean function. That is to consider the mean pH level as a bivariate function of x1 (calcium) and x2 (geological location): ph(xi1 , xi2 ) = f (xi1 , xi2 ) + ǫi ,

(5.34)

where ǫi are zero-mean independent random errors with a common vari-

Spline Smoothing with Heteroscedastic and/or Correlated Errors 161 ance. One possible model space for x1 is W22 (R), and one possible model space for x2 is W22 (R2 ). Therefore, we consider the tensor product space W22 (R) ⊗ W22 (R2 ). Define three averaging operators (1)

A1 f = (1)

A2 f = (2)

A1 f = (2) A2 f

=

J1 X

wj1 f (uj1 ),

j=1

J1 X

wj1 f (uj1 )φ˜12 (uj1 )φ˜12 ,

j=1

J2 X

wj2 f (uj2 ),

j=1

J2 X j=1

wj2 f (uj2 ){φ˜22 (uj2 )φ˜22 + φ˜23 (uj2 )φ˜23 },

where uj1 and uj2 are fixed points in R and R2 , and wj1 and wj2 are P 1 PJ2 ˜ fixed positive weights such that Jj=1 wj1 = j=1 wj2 = 1; φ11 = 1 and φ˜12 are orthonormal bases for the null space in W22 (R) based on the norm (2.41); φ˜21 = 1, and φ˜22 and φ˜23 are orthonormal bases for the null (1)

(1)

(2)

space in W22 (R2 ) based on the norm (2.41). Let A3 = I − A1 − A2 (2) (2) (2) and A3 = I − A1 − A2 . Then we have the following SS ANOVA decomposition: n on o (1) (1) (1) (2) (2) (2) f = A1 + A2 + A3 A1 + A2 + A3 f (1)

(2)

(1)

(2)

(1)

(2)

= A1 A1 f + A2 A1 f + A3 A1 f (1)

(2)

(1)

(2)

(1)

(2)

(1)

(2)

(1)

(2)

(1)

(2)

+ A1 A2 f + A2 A2 f + A3 A2 f + A1 A3 f + A2 A3 f + A3 A3 f , µ + β1 φ˜12 (x1 ) + β2 φ˜22 (x2 ) + β3 φ˜23 (x2 ) + β4 φ˜12 (x1 )φ˜22 (x2 ) ls + β5 φ˜12 (x1 )φ˜23 (x2 ) + f1s (x1 ) + f2s (x2 ) + f12 (x1 , x2 ) sl ss + f12 (x1 , x2 ) + f12 (x1 , x2 ).

(5.35)

Due to small sample size, we ignore all interactions and consider the following additive model y1 = µ + β1 φ˜12 (xi1 ) + β2 φ˜22 (xi2 ) + β3 φ˜23 (xi2 ) + f1s (xi1 ) + f2s (xi2 ) + ǫi ,

(5.36)

where ǫi are zero-mean independent random errors with a common variance. We fit model (5.36) as follows:

162


> acid.ssanova.fit summary(acid.ssanova.fit) ... GML estimate(s) of smoothing parameter(s) : 4.819636e-01 2.870235e-07 Equivalent Degrees of Freedom (DF): 8.768487 Estimate of sigma: 0.2560684 The estimate of the main effect of x1 plus the constant, µ + β1 φ˜12 (x1 ) + f1s (x1 ), is shown in Figure 5.7(a). This estimate is almost identical to that from model (5.33) with the exponential spatial correlation structure. The estimate of the main effect of x2 is shown in the right panel of Figure 5.7. An alternative approach to modeling the effect of geological location using mixed-effects will be discussed in Section 9.4.2.

Chapter 6 Generalized Smoothing Spline ANOVA

6.1

Generalized SS ANOVA Models

Generalized linear models (GLM) (McCullagh and Nelder 1989) provide a unified framework for analysis of data from exponential families. Denote (xi , yi ) for i = 1, . . . , n as independent observations on independent variables x = (x1 , . . . , xd ) and dependent variable y. Assume that yi are generated from a distribution in the exponential family with the density function yi h(fi ) − b(fi ) + c(yi , φ) , (6.1) g(yi ; fi , φ) = exp ai (φ) where fi = f (xi ), h(fi ) is a monotone transformation of fi known as the canonical parameter, and φ is a dispersion parameter. The function f models the effect of independent variables x. Denote µi = E(yi ), Gc as the canonical link such that Gc (µi ) = h(fi ), and G as the link function such that G(µi ) = fi . Then h = Gc ◦ G−1 , and it reduces to the identity function when the canonical link is chosen for G. The last term c(yi , φ) in (6.1) is independent of f . A GLM assumes that f (x) = xT β. Similar to the linear models for Gaussian data, the parametric GLM may be too restrictive for some applications. We consider the nonparametric extension of the GLM in this chapter. In addition to providing more flexible models, the nonparametric extension also provides model building and diagnostic methods for GLMs. Let the domain of each covariate xk be an arbitrary set Xk . Consider f as a multivariate function on the product domain X = X1 ×X2 ×· · ·×Xd . The SS ANOVA decomposition introduced in Chapter 4 may be applied to construct candidate model spaces for f . In particular, we will assume that f ∈ M, where M = H0 ⊕ H1 ⊕ · · · ⊕ Hq 163

164


is an SS ANOVA model space defined in (4.30), H0 = span{φ1 , . . . , φp } is a finite dimensional space collecting all functions that are not penalized, and Hj for j = 1, . . . , q are orthogonal RKHS’s with RKs Rj . The same notations in Chapter 4 will be used in this Chapter. We assume the same density function (6.1) for yi . However, for generality, we assume that f is observed through some linear functionals. Specifically, fi = Li f , where Li are known bounded linear functionals.

6.2 6.2.1

Estimation and Inference Penalized Likelihood Estimation

Assume that ai (φ) = a(φ)/̟i for i = 1, . . . , n where ̟i are known constants. Denote σ 2 = a(φ), y = (y1 , . . . , yn )T and f = (f1 , . . . , fn )T . Let li (fi ) = ̟i {b(fi ) − yi h(fi )}, i = 1, . . . , n, (6.2) Pn and l(f ) = i=1 li (fi ). Then the log-likelihood n X i=1

log g(yi ; fi , φ) =

n X yi h(fi ) − b(fi )

ai (φ)

i=1

=−

1 l(f ) + C, σ2

+ c(yi , φ)

(6.3)

where C is independent of f . Therefore, up to an additive and a multiplying constant, l(f ) is the negative log-likelihood. Note that l(f ) is independent of the dispersion parameter. For a GLM, the MLE of parameters β are the maximizers of the loglikelihood. For a generalized SS ANOVA model, as in the Gaussian case, a penalty term is necessary to avoid overfitting. We will use the same form of penalty as in Chapter 4. Specifically, we estimate f as the solution to the following penalized likelihood (PL)   q   X n λj kPj f k2 , (6.4) min l(f ) +  f ∈M  2 j=1

where Pj is the orthogonal projector in M onto Hj , and λj are smoothing parameters. The multiplying term 1/σ 2 is absorbed into smoothing parameters, and the constant C is dropped since it is independent of f . The multiplying constant n/2 is added such that the PL reduces to the

Generalized Smoothing Spline ANOVA

165

PLS (4.32) for Gaussian data. Under the new inner product defined in (4.33), the PL can be rewritten as n o n (6.5) min l(f ) + λkP1∗ f k2∗ , f ∈M 2 Pq where λj = λ/θj , and P1∗ = j=1 Pj is the orthogonal projection in M onto H1∗ = ⊕qj=1 Hj . Note that the RK of H1∗ under the new inner Pq product is R1∗ = j=1 θj Rj . It is easy to check that l(f ) is convex in f under the canonical link. In general, we assume that l(f ) is convex in f and has a unique minimizer in H0 . Then the PL (6.5) has a unique minimizer (Theorem 2.9 in Gu (2002)). We now show the Kimeldorf–Wahba representer theorem holds. Let R0 be the RK of H0 and R = R0 + R1∗ . Let ηi be the representer associated with Li . Then, from (2.12), ηi (x) = Li(z) R(x, z) = Li(z) R0 (x, z) + Li(z) R1∗ (x, z) , δi (x) + ξi (x). That is, the representers ηi for Li belong to the finite dimensional subspace S = H0 ⊕ span{ξ1 , . . . , ξn }. Let S c be the orthogonal complement of S. Any f ∈ H can be decomposed into f = ς1 + ς2 , where ς1 ∈ S and ς2 ∈ S c . Then we have Li f = (ηi , f ) = (ηi , ς1 ) + (ηi , ς2 ) = (ηi , ς1 ) = Li ς1 . Consequently, for any f ∈ H, the PL (6.5) satisfies n X

li (Li f ) +

n λkP1∗ f k2∗ 2

li (Li ς1 ) +

n λ(kP1∗ ς1 k2∗ + kP1∗ ς2 k2∗ ) 2

li (Li ς1 ) +

n λkP1∗ ς1 k2∗ , 2

i=1

=

n X i=1

≥

n X i=1

where equality holds iff ||P1∗ ς2 ||∗ = ||ς2 ||∗ = 0. Thus, the minimizer of the PL falls in the finite dimensional space S, which can be represented as fˆ(x) = =

p X

ν=1 p X ν=1

dν φν (x) + dν φν (x) +

n X

i=1 n X i=1

ci ξi ci

q X j=1

θj Li(z ) Rj (x, z).

(6.6)

166


For simplicity of notation, the dependence of fˆ on the smoothing parameters λ and θ = (θ1 , . . . , θq ) is not expressed explicitly. Let T = p {Li φν }ni=1 ν=1 , Σk = {Li Lj Rk }ni,j=1 for k = 1, . . . , q and Σθ = θ1 Σ1 + · · · + θq Σq . Let d = (d1 , . . . , dp )T and c = (c1 , . . . , cn )T . Denote fˆ = (L1 fˆ, . . . , Ln fˆ)T . Then fˆ = T d + Σθ c. Note that kP1∗ fˆk2∗ = cT Σθ c. Substituting (6.6) into (6.5), we need to solve c and d by minimizing n I(c, d) = l(T d + Σθ c) + λcT Σθ c. (6.7) 2 Except for the Gaussian distribution, the function l(f ) is not quadratic and (6.7) cannot be solved directly. For fixed λ and θ, we will apply the Newton–Raphson procedure to compute c and d. Let ui = dli /dfi and wi = d2 li /dfi2 , where fi = Li f . Let uT = (u1 , . . . , un )T and W = diag(w1 , . . . , wn ). Then ∂I = Σθ u + nλΣθ c, ∂c ∂I = T T u, ∂d ∂2I = Σθ W Σθ + nλΣθ , ∂c∂cT ∂2I = Σθ W T, ∂c∂dT ∂2I = T T W T. ∂d∂dT The Newton–Raphson procedure iteratively solves the linear system c−c Σθ W Σθ + nλΣθ Σθ W T T T W Σθ TTW T d−d −Σθ u − nλΣθ c , (6.8) = −T T u where the subscript minus indicates quantities evaluated at the previous Newton–Raphson iteration. Equations in (6.8) can be rewritten as (Σθ W Σθ + nλΣθ )c + Σθ W T d = Σθ W fˆ − Σθ u , T T W Σθ c + T T W T d = T T W fˆ − T T u ,

(6.9)

where fˆ = T d + Σθ c . As discussed in Section 2.4, it is only necessary ˇ = fˆ − W −1 u . It is easy to see to derive one set of solutions. Let y that a solution to ˇ, (Σθ + nλW −1 )c + T d = y T T c = 0,

(6.10)


167

is also a solution to (6.9). Note that W is known at the current Newton– Raphson iteration. Since equations in (6.10) have the same form as those in (5.6), then methods in Section 5.2.1 can be used to solve (6.10). Furthermore, (6.10) corresponds to the minimizer of the following PWLS problem: q n X 1X wi− (ˇ yi − fi )2 + λj kPj f k2 , n i=1 j=1

(6.11)

ˇ . Therefore, at each iteration, the where yˇi is the ith element of y Newton–Raphson procedure solves the PWLS criterion with working variables yˇi and working weights wi− . Consequently, the procedure can be regarded as iteratively reweighted PLS.

6.2.2

Selection of Smoothing Parameters

With canonical link such that h(f ) = f , we have E(yi ) = b′ (fi ), Var(yi ) = b′′ (fi )ai (φ), ui = ̟i {b′ (fi )−yi }, and wi = ̟i b′′ (fi ). Therefore, E(ui /wi ) = 0 and Var(ui /wi ) = σ 2 wi−1 . Consequently, when fˆ− is close to f and under some regularity conditions, it can be shown (Wang 1994, Gu 2002) that the working variables approximately have the same structure as in (5.1) yˇi = Li f + ǫi + op (1), (6.12) where ǫi has mean 0 and variance σ 2 wi−1 . The Newton–Raphson procedure essentially reformulates the problem to model f on working variables at each iteration. From the above discussion and note that W is known at the current Newton–Raphson iteration, we can use the UBR, GCV, and GML methods discussed in Section 5.2.3 to select smoothing parameters at each step of the Newton–Raphson procedure. Since working data are reformulated at each iteration, the target criteria of the above iterative smoothing parameter selection methods change throughout the iteration. Therefore, the overall target criteria of these iterative methods are not explicitly defined. A justification for the UBR criterion can be found in Gu (2002). One practical problem with the iterative methods for selecting smoothing parameters is that they are not guaranteed to converge. Nevertheless, extensive simulations indicate that, in general, the above algorithm works reasonably well in practice (Wang 1994, Wang, Wahba, Chappell and Gu 1995). Nonconvergence may become a serious problem for certain applications (Liu, Tong and Wang 2007). Some direct noniterative methods

168


for selecting smoothing parameters have been proposed. For Poisson and gamma distributions, it is possible to derive unbiased estimates of the symmetrized Kullback–Leibler discrepancy (Wong 2006, Liu et al. 2007). Xiang and Wahba (1996) proposed a direct GCV method. Details about the direct GCV method can be found in Gu (2002). A direct GML method will be discussed in Section 6.2.4. These direct methods are usually more computationally intensive. They have not been implemented in the current version of the assist package. It is not difficult to write R functions to implement these direct methods. A simple implementation of the direct GML method for gamma distribution will be given in Sections 6.4 and 6.5.3.

6.2.3

Algorithm and Implementation

The whole procedure discussed in Sections 6.2.1 and 6.2.2 is summarized in the following algorithm. Algorithm for generalized SS ANOVA models 1. Compute matrices T and Σk for k = 1, . . . , q, and set an initial value for f . ˜ k = W 1/2 Σk W 1/2 for k = 1, . . . , q, 2. Compute u , W , T˜ = W 1/2 T , Σ 1/2 ˜ = W y ˇ , and fit the transformed data with smoothing and y parameters selected by the UBR, GCV, or GML method. 3. Iterate step 2 until convergence. The above algorithm is easy to implement. All we need to do is to compute quantities ui and wi . We now compute these quantities for some special distributions. First consider logistic regression with logit link function. Assume that y ∼ Binomial(m, p) with density function m g(y) = py (1 − p)m−y , y = 0, . . . , m. y Then σ 2 = 1 and li = −yi fi +mi log(1+exp(fi )), where fi = log(pi /(1− pi )). Consequently, ui = −yi + mi pi and wi = mi pi (1 − pi ). Note that binary data is a special case with mi = 1. Next consider Poisson regression with log-link function. Assume that y ∼ Poisson(µ) with density function g(y) =

1 y µ exp(−µ), y!

y = 0, 1, . . . .


169

Then σ 2 = 1 and li = −yi fi + exp(fi ), where fi = log µi . Consequently, ui = −yi + µi and wi = µi . Last consider gamma regression with log-link function. Assume that y ∼ Gamma(α, β) with density function g(y) =

1 y α−1 y exp − , Γ(α)β α β

y > 0,

where α > 0 and β > 0 are shape and scale parameters. We are interested in modeling the mean µ , E(y) as a function of independent variables. We assume that the shape parameter does not depend on independent variables. Note that µ = αβ. The density function may be reparametrized as αy αα α−1 y exp − , y > 0. g(y) = Γ(α)µα µ The canonical parameter −µ−1 is negative. To avoid this constraint, we adopt the log-link. Then σ 2 = α−1 and li = yi exp(−fi ) + fi , where fi = log µi . Consequently, ui = −yi /µi + 1 and wi = yi /µi . Since wi are nonnegative, then l(f ) is a convex function of f . For the binomial and the Poisson distributions, the dispersion parameter σ 2 is fixed as σ 2 = 1. For the gamma distribution, the dispersion parameter σ 2 = α−1 . Since this constant has been separated from the definition of li , then we can set σ 2 = 1 in the UBR criterion. Therefore, for binomial, Poisson, and gamma distributions, the UBR criterion reduces to 1 ˜ y k2 + 2 trH, ˜ UBR(λ, θ) = k(I − H)˜ (6.13) n n ˜ is the hat matrix associated with the transformed data. where H In general, the weighted average of residuals n

2 σ ˆ−

1X = wi− n i=1

ui− wi−

2

n

=

1 X u2i− n i=1 wi−

(6.14)

provides an estimate of σ 2 at the current iteration when it is unknown. Then the UBR criterion reduces to UBR(λ, θ) =

2 σ− 1 ˜ y k2 + 2ˆ ˜ k(I − H)˜ trH. n n

(6.15)

There are two versions of the UBR criterion given in equations (6.13) and (6.15). The first version is favorable when σ 2 is known.

170


The above algorithm is implemented in the ssr function for binomial, Poisson, and gamma distributions based on a collection of Fortran subroutines called GRKPACK (Wang 1997). The specific distribution is specified by the family argument. The method for selecting smoothing parameters is specified by the argument spar with “u˜”, “v”, and “m” representing UBR, GCV, and GML criteria defined in (6.15), (5.25) and (5.26), respectively. The UBR method with fixed dispersion parameter (6.13) is specified as spar=‘‘u’’ together with the option varht for specifying the fixed dispersion parameter. Specifically, for binomial, Poisson and gamma distributions with σ 2 = 1, the combination spar=‘‘u’’ and varht=1 is used.

6.2.4

Bayes Model, Direct GML and Approximate Bayesian Confidence Intervals

Suppose observations yi are generated from (6.1) with fi = Li f . Assume the same prior for f as in (4.42): F (x) =

p X

ν=1

q X p ζν φν (x) + δ θj Uj (x), 1 2

(6.16)

j=1

iid

where ζν ∼ N(0, κ); Uj (x) are independent, zero-mean Gaussian stochastic processes with covariance function Rj (x, z); ζν and Uj (x) are mutually independent; and κ and δ are positive constants. Conditional on ζ = (ζ1 , . . . , ζp )T , f |ζ ∼ N(T ζ, δΣθ ). Letting κ → ∞ and integrating out ζ, Gu (1992) showed that the marginal density of f 1 p(f ) ∝ exp − f T Σ+ − Σ+ T (T T Σ+ T )−1 T T Σ+ f , θ θ θ θ 2δ where Σ+ is the Moore–Penrose inverse of Σθ . The marginal density of θ y, Z p(y) = p(y|f )p(f )df , (6.17)

usually does not have a closed form since, except for the Gaussian distribution, the log-likelihood log p(y|f ) is not quadratic in f . Note that log p(y|f ) =

n X i=1

log g(yi ; fi , φ) = −

1 l(f ) + C. σ2

We now approximate l(f ) by a quadratic function. Let uic and wic be ui and wi evaluated at fˆ. Let uc = (u1c , . . . , unc )T , Wc = diag(w1c , . . . , wnc ), and y c = fˆ −Wc−1 uc . Note that ∂l(f )/∂f | ˆ = f


171

uc , and ∂ 2 l(f )/∂f ∂f T | ˆ = Wc . Expanding l(f ) as a function of f f around the fitted values fˆ to the second order leads to l(f ) ≈

1 1 (f − y c )T Wc (f − y c ) + l(fˆ ) − uTc Wc−1 uc . 2 2

(6.18)

Note that log p(f ) is quadratic in f . Then it can be shown that, applying approximation (6.18), the marginal density of y in (6.17) is approximately proportional to (Liu, Meiring and Wang 2005) 1 T −1 − 12 − 21 T −1 − 12 ˆ p(y) ∝ |Wc | |V | |T V T | p(y|f ) exp u W uc 2σ 2 c c 1 T −1 −1 T −1 −1 T −1 − V T (T V T ) T V y c , (6.19) × exp − y c V 2

where V = δΣθ + σ 2 Wc−1 . When Σθ is nonsingular, fˆ is the maximizer of the integrand p(y|f )p(f ) in (6.17) (Gu 2002). In this case the foregoing approximation is simply the Laplace approximation. 1/2 ˜ = Wc1/2 Σ Wc1/2 , T˜ = Wc1/2 T , and the QR de˜ c = Wc y c , Σ Let y θ θ ˜1 Q ˜ 2 )(R ˜ T 0)T . Let U EU T be the eigendecompocomposition of T˜ be (Q ˜ T2 Σ ˜ Q ˜ sition of Q θ 2 , where E = diag(e1 , . . . , en−p ), e1 ≥ e2 ≥ . . . ≥ en−p ˜ T2 y ˜ . Then it can be are eigenvalues. Let z = (z1 , . . . , zn−p )T , U T Q shown (Liu et al. 2005) that (6.19) is equivalent to 1 T −1 −1 ˆ ˜ p(y) ∝ |R| p(y|f ) exp u W uc 2σ 2 c c ) ( n−p n−p 2 X Y 1 z 1 ν . × (δeν + σ 2 )− 2 exp − 2 ν=1 δeν + σ 2 ν=1

Let δ = σ 2 /nλ. Then an approximation of the negative log-marginal likelihood is ˜ + 1 l(fˆ ) − 1 uT W −1 uc DGML(λ, θ) = log |R| σ2 2σ 2 c c n−p 1X zv2 + log(eν /nλ + 1) + 2 . (6.20) 2 i=1 σ (eν /nλ + 1) ˜ eν , and zν all depend on λ and θ even though Notice that fˆ , uc , Wc , R, the dependencies are not expressed explicitly. The function DGML(λ, θ) is referred to as the direct generalized maximum likelihood (DGML) criterion, and the minimizers of DGML(λ, θ) are called the DGML estimate of the smoothing parameter. Section 6.4 shows a simple implementation of the DGML criterion for the gamma distribution.

172


Let y c = (y1c , . . . , ync )T . Based on (6.12), consider the approximation model at convergence yic = Li f + ǫi ,

i = 1, . . . , n,

(6.21)

−1 where ǫi has mean 0 and variance σ 2 wic . Assume prior (6.16) for f . Then, as in Section 5.2.5, Bayesian confidence intervals can be constructed for f in the approximation model (6.21). They provide approximate confidence intervals for f in the generalized SS ANOVA model. The bootstrap approach may also be used to construct confidence intervals, and the extension is straightforward. Connections between smoothing spline models and LME models are presented in Sections 3.5, 4.7, and 5.2.4. We now extend this connection to data from exponential families. Consider the following generalized linear mixed-effects model (GLMM) (Breslow and Clayton 1993)

G{E(y|u)} = T d + Zu,

(6.22)

where G is the link function, d are fixed effects, u = (uT1 , . . . , uTq )T are random effects, Z = (In , . . . , In ), uk ∼ N(0, σ 2 θk Σk /nλ) for k = 1, . . . , q, and uk are mutually independent. Then u ∼ N(0, σ 2 D/nλ), where D = diag(θ1 Σ1 , . . . , θq Σq ). Setting uk = θk Σk c as in the Gaussian case (Opsomer et al. 2001) and noting that ZDZ T = Σθ , we have u = DZ T c and uT {Cov(u)}+ u = nλcT ZDD+ DZ T c/σ 2 = nλcT ZDZ T c/σ 2 = nλcT Σθ c/σ 2 . Note that Zu = Σθ c. Therefore the PL (6.7) is the same as the penalized quasi-likelihood (PQL) of the GLMM (6.22) (equation (6) in Breslow and Clayton (1993)).

6.3

Wisconsin Epidemiological Study of Diabetic Retinopathy

We use the Wisconsin Epidemiological Study of Diabetic Retinopathy (WESDR) data to illustrated how to fit an SS ANOVA model to binary responses. Based on Wahba, Wang, Gu, Klein and Klein (1995), we investigate how probability of progression to diabetic retinopathy at the first follow-up (prg) depends on the following covariates at baseline: duration of diabetes (dur), glycosylated hemoglobin (gly), and body mass index (bmi). Let y be the response variable prg where y = 1 represents progression of retinopathy and y = 0 otherwise. Let x1 , x2 , and x3 be the covariates dur, gly, and bmi transformed into [0, 1]. Let x = (x1 , x2 , x3 ) and


173

f (x) = logitP (y = 1|x). We model f using the tensor product space W22 [0, 1]⊗W22 [0, 1]⊗W22 [0, 1]. The three-way SS ANOVA decomposition can be derived similarly using the method in Chapter 4. For simplicity, we will ignore three-way interactions and start with the following SS ANOVA model with all two-way interactions: f (x) = µ + β1 × (x1 − 0.5) + β2 × (x2 − 0.5) + β3 × (x3 − 0.5)

+ β4 × (x1 − 0.5)(x2 − 0.5) + β5 × (x1 − 0.5)(x3 − 0.5)

+ β6 × (x2 − 0.5)(x3 − 0.5) + f1s (x1 ) + f2s (x2 ) + f3s (x3 ) sl ss ls (x1 , x2 ) + f12 (x1 , x2 ) + f12 (x1 , x2 ) + f12

ls sl ss (x1 , x3 ) + f13 (x1 , x3 ) + f13 (x1 , x3 ) + f13 ls sl ss + f23 (x2 , x3 ) + f23 (x2 , x3 ) + f23 (x2 , x3 ).

(6.23)

The following statements fit model (6.23) with smoothing parameter selected by the UBR method: > > > > > >

data(wesdr); attach(wesdr) y 0)| (int.dur.bmi$fit+1.96*int.dur.bmi$pstd int.gly.bmi mean((int.gly.bmi$fit-1.96*int.gly.bmi$pstd>0)| (int.gly.bmi$fit+1.96*int.gly.bmi$pstd wesdr.fit3 summary(wesdr.fit3) UBR estimate(s) of smoothing parameter(s) : 4.902745e-06 5.474122e+00 1.108322e-05 Equivalent Degrees of Freedom (DF): 11.78733 > comp.est3 comp.norm3 print(round(comp.norm3)) 7.22 33.71 12.32 4.02 8.99 11.95 10.15 0.00 6.79 Based on the estimates of smoothing parameters and the norms, the nonparametric main effect of x2 , f2s , can be dropped. Therefore, we fit the final model: > wesdr.fit4 summary(wesdr.fit4) ... UBR estimate(s) of smoothing parameter(s) : 4.902693e-06 1.108310e-05 Equivalent Degrees of Freedom (DF): 11.78733 Estimate of sigma: 1 To look at the effect of dur, with gly and bmi being fixed at the medians of their observed values, we compute estimates of the probabilities and posterior standard deviations at a grid point of dur. The estimated probability function and the approximate 95% Bayesian confidence intervals are shown in Figure 6.1(a). The risk of progression of retinopathy increases up to a duration of about 10 years and then decreases, possibly caused by censoring due to death in patients with longer durations. Similar plots for the effects of gly and bmi are shown in Figures 6.1(b) and

1.0 probability 0.4 0.6

0.0

0.2

probability 0.4 0.6

0.0

0.2

probability 0.4 0.6

0.2 0.0 0 10 30 50 duration (yr)

(c)

0.8

(b)

0.8

(a)

1.0


0.8

1.0

176

10 15 20 gly. hemoglobin

20 30 40 50 body mass index (kg/m2)

FIGURE 6.1 WESDR data, plots of (a) the estimated probability as a function of dur with gly and bmi being fixed at the medians of their observed values, (b) the estimated probability as a function of gly with dur and bmi being fixed at the medians of their observed values, and (c) the estimated probability as a function of bmi with dur and gly being fixed at the medians of their observed values. Shaded regions are approximate 95% Bayesian confidence intervals. Rugs on the bottom and the top of each plot are observations of prg. 6.1(c). The risk of progression of retinopathy increases with increasing glycosylated hemoglobin, and the risk increases with increasing body mass index until a value of about 25 kg/m2 , after which the trend is uncertain due to wide confidence intervals. As expected, the confidence intervals are wider in areas where observations are sparse.

6.4

Smoothing Spline Estimation of Variance Functions

Consider the following heteroscedastic SS ANOVA model yi = L1i f1 + exp{L2i f2 /2}ǫi ,

i = 1, . . . , n,

(6.24)

where fk is a function on Xk = Xk1 × Xk2 × · · · × Xkdk with model space Mk = Hk0 ⊕Hk1 ⊕· · ·⊕Hkqk for k = 1, 2; L1i and L2i are bounded linear iid functionals; and ǫi ∼ N(0, 1). The goal is to estimate both the mean function f1 and the variance function f2 . Note that both the mean and variance functions are modeled nonparametrically in this section. This


177

is in contrast to the parametric model (5.28) for variance functions in Chapter 5. One simple approach to estimating functions f1 and f2 is to use the following two-step procedure: 1. Estimate the mean function f1 as if random errors are homoscedastic. 2. Estimate the variance function f2 based on squared residuals from the first step. 3. Estimate the mean function again using the estimated variance function. The PLS estimation method in Chapter 4 can be used in the first step. Denote the estimate at the first step as f˜1 and ri = yi −L1i f˜1 as residuals. Let zi = ri2 . Under suitable conditions, zi ≈ exp{L2i f2 }χ2i,1 , where χ2i,1 are iid Chi-square random variables with degree of freedom 1. Regarding Chi-square distribution as a special case of the gamma distribution, the PL method described in this chapter can be used to estimate the variance function f2 at the second step. Denote the estimate at the second step as f˜2 . Then the PWLS method in Chapter 5 can be used in the third step with known covariance W −1 = diag(L21 f˜2 , . . . , L2n f˜2 ). We now use the motorcycle data to illustrate the foregoing two-step procedure. We have fitted a cubic spline to the logarithm of squared residuals in Section 5.4.1. Based on model (3.53), consider the heteroscedastic partial spline model yi = f1 (xi ) + exp{f2 (xi )/2}ǫi , i = 1, . . . , n, (6.25) P3 where f1 (x) = j=1 βj (x − tj )+ + g1 (x), t1 = 0.2128, t2 = 0.3666 iid

and t3 = 0.5113 are the change-points in the first derivative, and ǫi ∼ N(0, 1). We model both g1 and f2 using the cubic spline model space W22 [0, 1]. The following statements implement the two-step procedure for model (6.25): # > > > >

step 1 t1

Smoothing Splines: Methods and Applications (Chapman & Hall CRC Monographs on Statistics & Applied Probability)

CRC Monographs on Statistics & Applied Probability)

CRC Monographs on Statistics & Applied Probability)

CRC Monographs on Statistics & Applied Probability)

CRC Monographs on Statistics & Applied Probability)

Expansions and Asymptotics for Statistics (Chapman & Hall CRC Monographs on Statistics & Applied Probability)

Statistics in the 21st Century (Chapman & Hall CRC Monographs on Statistics & Applied Probability)

Residuals and Influence in Regression (Chapman & Hall CRC Monographs on Statistics & Applied Probability 18)

Multivariate Models and Multivariate Dependence Concepts (Chapman & Hall CRC Monographs on Statistics & Applied Probability)

Antedependence Models for Longitudinal Data (Chapman & Hall CRC Monographs on Statistics & Applied Probability)

Introduction to Time Series Modeling (Chapman & Hall CRC Monographs on Statistics & Applied Probability)

Generalized Linear Models, Second Edition (Chapman & Hall CRC Monographs on Statistics & Applied Probability)

An Introduction to the Bootstrap (Chapman & Hall CRC Monographs on Statistics & Applied Probability)

Mixed Effects Models for Complex Data (Chapman & Hall CRC Monographs on Statistics & Applied Probability)

Nonlinear Time Series: Semiparametric and Nonparametric Methods (Chapman & Hall CRC Monographs on Statistics & Applied Probability)

Density Estimation for Statistics and Data Analysis (Chapman & Hall CRC Monographs on Statistics & Applied Probability)

Group Sequential Methods (Chapman & Hall CRC Interdisciplinary Statistics)

Design and Analysis of Cross-Over Trials, Second Edition (Chapman & Hall CRC Monographs on Statistics & Applied Probability)

Analysis of Time Series Structure: SSA and Related Techniques (Chapman & Hall CRC Monographs on Statistics & Applied Probability)

Nonparametric Regression and Generalized Linear Models: A roughness penalty approach (Chapman & Hall CRC Monographs on Statistics & Applied Probability)