Institute of Mathematical Statistics LECTURE NOTES-MONOGRAPH SERIES
Model Selection
Edited by P. Lahiri
Institute of Mathematical Statistics LECTURE NOTES-MONOGRAPH SERIES Volume 38
Model Selection
Edited by P. Lahiri
Institute of Mathematical Statistics Beachwood, Ohio
Institute of Mathematical Statistics Lecture Notes-Monograph Series
Series Editor: Joel Greenhouse
The production of the IMS Lecture Notes-Monograph Series is managed by the IMS Business Office: Julia A. Norton, IMS Treasurer, and Elyse Gustafson, IMS Executive Director.
Library of Congress Control Number: 2001 135412 International Standard Book Number 0-940600-52-8 Copyright © 2001 Institute of Mathematical Statistics All rights reserved Printed in the United States of America
TABLE OF CONTENTS
PREFACE On Model Selection
1
a R. Rao and Y. Wu Discussion
Sadanori Konishi
58
Discussion
Rahul Mukerjee
60
Rejoinder
C R. Rao and Y. Yu 64
The Practical Implementation of Bayesian Model Selection
65
Hugh Chipman, Edward I. George and Robert E. McCulloch Discussion
M. Clyde 117
Discussion
Dean P. Foster and Robert A. Stine 124
Rejoinder
Hugh Chipman, Edward I. George and Robert E. McCulloch 131
Objective Bayesian Methods for Model Selection: Introduction and Comparison
135
James O. Berger and Luis R. Pericchi Discussion
J. K. Ghosh and Tapas Samanta 194
Discussion
Fulυio De Santis 197
Rejoinder
James O. Berger and Luis R. Pericchi 203
Scales of Evidence for Model Selection: Fisher versus Jeffreys
208
Bradley Efron and Alan Gous Discussion
R. E. Kass 246
Discussion
G. S. Datta and P. Lahiri 249
Rejoinder
Bradley Efron and Alan Gous 254
PREFACE
This volume originates from a conference that I organized in Lincoln, Nebraska, U. S. A., during March 24-26, 1999. One of the main themes of the conference was the growing field of model selection. The conference was supported by the United States Postal Service, the National Center for Health Statistics, the Gallup Organization and the University of Nebraska-Lincoln. About eighty researchers including some of the contributors of this volume attended the conference. Shortly after the conference, I discussed the possibility of publishing a volume on model selection with David Ruppert, the former editor of the IMS Lecture NotesMonograph Series. We agreed that a volume containing a few long review papers on model selection would be of great interest to researchers. The intent was not to publish a volume of conference proceedings but to provide an overview of model selection from the perspective of a few experts in the field. This volume contains four review papers with discussions and rejoinders. Since the papers are long, a table of contents at the beginning of each paper is provided. We start the volume with the paper by C. R. Rao and Y. Wu, who survey various model selection procedures with reference to regression, categorical data and time series analysis. The paper also discusses the impact of carrying out statistical analysis after a model is selected. The second paper by H. Chipman, E. I. George and R. E. McCulloch reviews the basics of Bayesian model selection and discusses at length two important practical issues associated with the implementation of Bayesian model selection: (i) specification of the prior distributions and (ii) the calculation of posterior distributions. These practical issues are then illustrated in variable selection for the linear models and CART model selection problems. In the third paper, J. O. Berger and L. R. Pericchi motivate the Bayesian model selection and point out the need for objective Bayesian methods in model selection. The authors provide a comprehensive review of four objective Bayesian model selection approaches and extensively discuss criteria for evaluating different model selection methods. The objective Bayesian model selection methods are then critically examined and compared. We conclude this volume with the paper by B. Efron and A. Gous who compare different model selection methods using hypothesis testing, a simple case of model selection. In one-dimensional case, the authors demonstrate how two apparently different approaches (Fisher's and Jeffreys') to hypothesis testing can be possibly reconciled by
giving an interpretation of Fisher's scale in terms of Bayes factors. Similar arguments, however, fail in higher dimensions. I would like to gratefully acknowledge the University of Nebraska-Lincoln, the Gallup Organization, the National Science Foundation and the University of Maryland at College Park for supporting the project. I would like to take this opportunity to thank Joel Greenhouse, editor of the IMS Lecture Notes-Monograph Series, and David Ruppert for their encouragement and a number of constructive suggestions. I also thank Shijie Chen, Kristen Olson and Barbara Rolfes for their help in typing and editorial work and the Institute of Mathematical Statistics for publishing this volume. I believe that the four long review papers along with their discussions and rejoinders provide an excellent overview of model selection. This volume will certainly be a valuable resource for researchers. I thank all the authors and discussants for their outstanding contributions in this volume. P. Lahiri Milton Mohr Distinguished Professor of Statistics University of Nebraska-Lincoln Adjunct Professor, Joint Program in Survey Methodology University of Maryland at College Park Senior Scientist, The Gallup Organization, Princeton
Model Selection IMS Lecture Notes - Monograph Series (2001) Volume 38
On Model Selection C. R. Rao and Y. Wu Pennsylvania State University and York University
Abstract The task of statistical model selection is to choose a family of distributions among a possible set of families, which is the best approximation of reality manifested in the observed data. In this paper, we survey the model selection criteria discussed in statistical literature. We are mainly concerned with those used in regression analysis and time series.
C.R. Rao is Eberly Professor, Department of Statistics,Pennsylvania State University, University Park, PA 16802, U.S.A; email:
[email protected]. Y.Wu is Professor, Department of Mathematics and Statistics, York University, 4700 Keele Street, Toronto, Ontario, Canada M3J 1P3; email:
[email protected]. Rao was partially supported by Army Research Office: DAAD 19-01-1-0563 and Wu was partially supported by the Natural Sciences and Engineering Research Council of Canada.
Contents 1 Introduction
3
2 Selection of a model based on hypothesis testing
4
3 Model selection based on prediction errors
6
4 Information theoretic criteria
11
5 Cross-validation, bootstrap and related model selection methods
19
6 Baysian approach to model selection
24
7 Robust model selection
28
8 Order selection in time series
31
9 Model selection in categorical data analysis
36
10 Model selection in nonparametric regression
37
11 Data-oriented penalty
41
12 Statistical analysis after model selection
43
13 Conclusions
45
On Model Selection
1
3
Introduction
Let {M7, 7 E Γ} be candidate models for the observations, where Γ is an index set. It is possible that the true model is not included in {M 7 }. Based on the data, we need to select a model from {M7, 7 E Γ} through a suitable model selection criterion. Many model selection procedures have been proposed in the literature. Each one is designed for a particular application. Model selection problems are encountered almost everywhere. In linear regression analysis, it is of interest to select the right number of nonzero regression parameters. With the smallest true model, statistical inferences can be carried out more efficiently. In the analysis of time series, it is essential to know the true orders of an ARMA model. In problems of clustering, it is important to find out the number of clusters. In the signal detection, it is necessary to determine the number of true signals, and so on. In this paper, we survey the model selection criteria discussed in statistical literature. Almost all statistical problems can be considered as model selection problems, but in this paper, we will be mainly concerned with those used in regression analysis and time series. Some interesting examples can be found in Burnham and Anderson (1998), among others. The paper is arranged as follows: In Section 2, model selection based on hypothesis testing is examined. In Section 3, selection of a model based on the prediction errors is surveyed. In Section 4, the information theoretic criteria are discussed. In Section 5, the role of cross-validation and bootstrap methods in model selection is covered. In Section 6, Baysian approaches to model selection are described. In Section 7, studies on robust model selection are examined. In Section 8, the results on order selection in time series are presented. In Section 9, model selection criteria in categorical data analysis are explored. In Section 10, the investigation on model selection in nonparametric regression is reviewed. In Section 11, data-oriented penalties are discussed. In Section 12, the effect of prior model selection based on data on the inferential statistical analysis of the same data is examined. In the sequel, for an index set κ;, \κ\ denotes the size of «:, c(κ) denotes the sub-vector containing the components of the vector c that are indexed by the integers in n and A(κ) denotes the sub-matrix containing the columns of a matrix A that are indexed by the integers in «, PA is used to denote the projection operator on the linear space generated by the column vectors of A, and Y ~ (μ, Σ) is used to represent the random vector Y distributed according to a multivariate distribution with mean vector μ and covariance matrix Σ. For convenience, ί : m denotes the set of {i, ί + 1,..., m} in this paper, where £ < m are positive integers.
4
2
C. R. Rao and Y. Wu
Selection of a model based on hypothesis testing
A model selection procedure can be constructed based on a sequence of hypothesis tests. Assume that there is an order in the set of candidate models {Mi, i = 1,2,...} such that Mi is a preferable to Mi+i. A sequence of hypotheses, Hi0 : Mi holds true versus Hiα : Mi+ι holds true, i = 1,2,..., can be tested sequentially. Once Hi0 is accepted, the test procedure stops and the model Mi is selected. Another such procedure is to replace Hiα by H'iα : One of {Mj : j > i) holds true. It can be seen that if all the tests in two test procedures have the same thresholds, the acceptance of Mi in the second test procedure will imply that of Mi in the first test procedure. Now assume that there is a partial order in a finite index set. Using the partial order "-
