Computational Methods in Statistics and Econometrics (Statistics, a Series of Textbooks and Monographs)

Computational Methods in Statistics and Econometrics Hisashi Tanizaki Kobe University Kobe, Japan

MARCEL

MARCEL DEKKER, INC. D E K K E R

NEW YORK • BASEL

Marcel Dekker, Inc., and the author(s) make no warranty with regard to the accompanying software, its accuracy, or its suitability for any purpose other than as described in the book. This software is licensed solely on an "as is" basis. The only warranty made is that the medium on which the software is recorded is free of defects. Marcel Dekker, Inc., will replace a diskette or CD-ROM found to be defective if such defect is not attributable to misuse by the purchaser or his agent. The defective diskette or CD-ROM must be returned within 10 days to: Customer Service, Marcel Dekker, Inc., Cimarron Road, Monticello, NY 12701. Although great care has been taken to provide accurate and current information, neither the author(s) nor the publisher, nor anyone else associated with this publication, shall be liable for any loss, damage, or liability directly or indirectly caused or alleged to be caused by this book. The material contained herein is not intended to provide specific advice or recommendations for any specific situation. Trademark notice: Product or corporate names may be trademarks or registered trademarks and are used only for identification and explanation without intent to infringe. Library of Congress Cataloging-in-Publication Data A catalog record for this book is available from the Library of Congress. ISBN: 0-8247-4804-2 This book is printed on acid-free paper. Headquarters Marcel Dekker, Inc., 270 Madison Avenue, New York, NY 10016, U.S.A. tel: 212-696-9000; fax: 212-685-4540 Distribution and Customer Service Marcel Dekker, Inc., Cimarron Road, Monticello, New York 12701, U.S.A. tel: 800-228-1160; fax: 845-796-1772 World Wide Web http://www.dekker.com The publisher offers discounts on this book when ordered in bulk quantities. For more information, write to Special Sales/Professional Marketing at the headquarters address above. Copyright © 2004 by Marcel Dekker, Inc. All Rights Reserved. Neither this book nor any part may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, microfilming, and recording, or by any information storage and retrieval system, without permission in writing from the publisher. Current printing (last digit): 10 9 8 7 6 5 4 3 2 1 PRINTED IN THE UNITED STATES OF AMERICA

STATISTICS: Textbooks and Monographs D. B. Owen Founding Editor, 1972-1991 Associate Editors Statistical Computing/ Nonparametric Statistics Professor William R. Schucany Southern Methodist University

Multivariate Analysis Professor Anant M. Kshirsagar University of Michigan

Probability Professor Marcel F. Neuts University of Arizona

Quality Control/Reliability Professor Edward G. Schilling Rochester Institute of Technology

Editorial Board Applied Probability Dr. Paul R. Garvey The MITRE Corporation

Statistical Distributions Professor N. Balakrishnan McMaster University

Economic Statistics Professor David E. A. Giles University of Victoria

Statistical Process Improvement Professor G. Geoffrey Vining Virginia Polytechnic Institute

Experimental Designs Mr. Thomas B. Barker Rochester Institute of Technology

Stochastic Processes Professor V. Lakshmikantham Florida Institute of Technology

Multivariate Analysis Professor Subir Ghosh University of California-Riverside

Survey Sampling Professor Lynne Stokes Southern Methodist University

Time Series Sastry G. Pantula North Carolina State University

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43.

The Generalized Jackknife Statistic, H. L Gray and W. R. Schucany Multivariate Analysis, Anant M. Kshirsagar Statistics and Society, Walter T. Federer Multivariate Analysis: A Selected and Abstracted Bibliography, 1957-1972, Kocherlakota Subrahmaniam and Kathleen Subrahmaniam Design of Experiments: A Realistic Approach, Virgil L. Anderson and Robert A. McLean Statistical and Mathematical Aspects of Pollution Problems, John W. Pratt Introduction to Probability and Statistics (in two parts), Part I: Probability; Part II: Statistics, Narayan C. Giri Statistical Theory of the Analysis of Experimental Designs, J. Ogawa Statistical Techniques in Simulation (in two parts), Jack P. C. Kleijnen Data Quality Control and Editing, Joseph I. Naus Cost of Living Index Numbers: Practice, Precision, and Theory, Kali S. Banerjee Weighing Designs: For Chemistry, Medicine, Economics, Operations Research, Statistics, Kali S. Banerjee The Search for Oil: Some Statistical Methods and Techniques, edited by D. B. Owen Sample Size Choice: Charts for Experiments with Linear Models, Robert E. Odeh and Martin Fox Statistical Methods for Engineers and Scientists, Robert M. Bethea, Benjamin S. Duran, and Thomas L Boullion Statistical Quality Control Methods, Irving W. Burr On the History of Statistics and Probability, edited by D. B. Owen Econometrics, Peter Schmidt Sufficient Statistics: Selected Contributions, Vasanf S. Huzurbazar (edited by Anant M. Kshirsagar) Handbook of Statistical Distributions, Jagdish K. Patel, C. H. Kapadia, and D. B. Owen Case Studies in Sample Design, A. C. Rosander Pocket Book of Statistical Tables, compiled by R. E. Odeh, D. B. Owen, Z. W. Bimbaum, and L. Fisher The Information in Contingency Tables, D. V. Gokhale and Solomon Kullback Statistical Analysis of Reliability and Life-Testing Models: Theory and Methods, Lee J. Bain Elementary Statistical Quality Control, Irving W. Burr An Introduction to Probability and Statistics Using BASIC, Richard A. Groeneveld Basic Applied Statistics, 8. L. Raktoe andJ. J. Hubert A Primer in Probability, Kathleen Subrahmaniam Random Processes: A First Look, R. Syski Regression Methods: A Tool for Data Analysis, Rudolf J. Freund and Paul D. Minton Randomization Tests, Eugene S. Edgington Tables for Normal Tolerance Limits, Sampling Plans and Screening, Robert E. Odeh and D. B. Owen Statistical Computing, William J. Kennedy, Jr., and James E. Gentle Regression Analysis and Its Application: A Data-Oriented Approach, Richard F. Gunst and Robert L. Mason Scientific Strategies to Save Your Life, /. D. J. Brass Statistics in the Pharmaceutical Industry, edited by C. Ralph Buncher and Jia-Yeong Tsay Sampling from a Finite Population, J. Hajek Statistical Modeling Techniques, S. S. Shapiro and A. J. Gross Statistical Theory and Inference in Research, T. A. Bancroft and C.-P. Han Handbook of the Normal Distribution, Jagdish K. Patel and Campbell B. Read Recent Advances in Regression Methods, Hrishikesh D. Vinod and Aman Ullah Acceptance Sampling in Quality Control, Edward G. Schilling The Randomized Clinical Trial and Therapeutic Decisions, edited by Niels Tygstrup, John M Lachin, and Erik Juhl

44. Regression Analysis of Survival Data in Cancer Chemotherapy, Walter H. Carter, Jr., Galen L Wampler, and Donald M. Stablein 45. A Course in Linear Models, Anant M. Kshirsagar 46. Clinical Trials: Issues and Approaches, edited by Stanley H. Shapiro and Thomas H. Louis 47. Statistical Analysis of DMA Sequence Data, edited by B. S. Weir 48. Nonlinear Regression Modeling: A Unified Practical Approach, David A. Ratkowsky 49. Attribute Sampling Plans, Tables of Tests and Confidence Limits for Proportions, Robert E. Odeh and D. B. Owen 50. Experimental Design, Statistical Models, and Genetic Statistics, edited by Klaus Hinkelmann 51. Statistical Methods for Cancer Studies, edited by Richard G. Come/I 52. Practical Statistical Sampling for Auditors, Arthur J. Wilburn 53. Statistical Methods for Cancer Studies, edited by Edward J. Wegman and James G. Smith 54. Self-Organizing Methods in Modeling: GMDH Type Algorithms, edited by Stanley J. Far/ow 55. Applied Factorial and Fractional Designs, Robert A. McLean and Virgil L Anderson 56. Design of Experiments: Ranking and Selection, edited by Thomas J. Santner and Ajit C. Tamhane 57. Statistical Methods for Engineers and Scientists: Second Edition, Revised and Expanded, Robert M. Bethea, Benjamin S. Duran, and Thomas L. Bouillon 58. Ensemble Modeling: Inference from Small-Scale Properties to Large-Scale Systems, Alan E. Gelfand and Crayton C. Walker 59. Computer Modeling for Business and Industry, Bruce L. Bowerman and Richard T. O'Connell 60. Bayesian Analysis of Linear Models, Lyle D. Broemeling 61. Methodological Issues for Health Care Surveys, Brenda Cox and Steven Cohen 62. Applied Regression Analysis and Experimental Design, Richard J. Brook and Gregory C. Arnold 63. Statpal: A Statistical Package for Microcomputers—PC-DOS Version for the IBM PC and Compatibles, Bruce J. Chalmer and David G. Whitmore 64. Statpal: A Statistical Package for Microcomputers—Apple Version for the II, II+, and lie, David G. Whitmore and Bruce J. Chalmer 65. Nonparametric Statistical Inference: Second Edition, Revised and Expanded, Jean Dickinson Gibbons 66. Design and Analysis of Experiments, Roger G. Petersen 67. Statistical Methods for Pharmaceutical Research Planning, Sten W. Bergman and John C. Gittins 68. Goodness-of-Fit Techniques, edited by Ralph B. D'Agostino and Michael A. Stephens 69. Statistical Methods in Discrimination Litigation, edited by D. H. Kaye and Mikel Aickin 70. Truncated and Censored Samples from Normal Populations, Helmut Schneider 71. Robust Inference, M. L. Tiku, W. Y. Tan, and N. Balakrishnan 72. Statistical Image Processing and Graphics, edited by Edward J. Wegman and Douglas J. DePriest 73. Assignment Methods in Combinatorial Data Analysis, Lawrence J. Hubert 74. Econometrics and Structural Change, Lyle D. Broemeling and Hiroki Tsurumi 75. Multivariate Interpretation of Clinical Laboratory Data, Adelin Albert and Eugene K. Ham's 76. Statistical Tools for Simulation Practitioners, Jack P. C. Kleijnen 77. Randomization Tests: Second Edition, Eugene S. Edgington 78. A Folio of Distributions: A Collection of Theoretical Quantile-Quantile Plots, Edward B. Fowlkes 79. Applied Categorical Data Analysis, Daniel H. Freeman, Jr. 80. Seemingly Unrelated Regression Equations Models: Estimation and Inference, Virendra K. Srivastava and David E. A. Giles

81. Response Surfaces: Designs and Analyses, Andre I. Khuri and John A. Cornell 82. Nonlinear Parameter Estimation: An Integrated System in BASIC, John C. Nash and Mary Walker-Smith 83. Cancer Modeling, edited by James R. Thompson and Barry W. Brown 84. Mixture Models: Inference and Applications to Clustering, Geoffrey J. McLachlan and Kaye E. Basford 85. Randomized Response: Theory and Techniques, Arijit Chaudhuri and Rahul Mukerjee 86. Biopharmaceutical Statistics for Drug Development, edited by Karl E. Peace 87. Parts per Million Values for Estimating Quality Levels, Robert E. Odeh and D. B. Owen 88. Lognormal Distributions: Theory and Applications, edited by Edwin L Crow and Kunio Shimizu 89. Properties of Estimators for the Gamma Distribution, K. O. Bowman and L R. Shenton 90. Spline Smoothing and Nonparametric Regression, Randall L. Eubank 91. Linear Least Squares Computations, R. W. Farebrother 92. Exploring Statistics, Damaraju Raghavarao 93. Applied Time Series Analysis for Business and Economic Forecasting, Sufi M. Nazem 94. Bayesian Analysis of Time Series and Dynamic Models, edited by James C. Spall 95. The Inverse Gaussian Distribution: Theory, Methodology, and Applications, Raj S. Chhikara and J. Leroy Folks 96. Parameter Estimation in Reliability and Life Span Models, A. Clifford Cohen and Betty Jones Whitten 97. Pooled Cross-Sectional and Time Series Data Analysis, Terry E. Dielman 98. Random Processes: A First Look, Second Edition, Revised and Expanded, R. Syski 99. Generalized Poisson Distributions: Properties and Applications, P. C. Consul 100. Nonlinear Lp-Norm Estimation, Rene Gonin and Arthur H. Money 101. Model Discrimination for Nonlinear Regression Models, Dale S. Borowiak 102. Applied Regression Analysis in Econometrics, Howard E. Doran 103. Continued Fractions in Statistical Applications, K. O. Bowman and L. R. Shenton 104. Statistical Methodology in the Pharmaceutical Sciences, Donald A. Berry 105. Experimental Design in Biotechnology, Perry D. Haaland 106. Statistical Issues in Drug Research and Development, edited by Karl E. Peace 107. Handbook of Nonlinear Regression Models, David A. Ratkowsky 108. Robust Regression: Analysis and Applications, edited by Kenneth D. Lawrence and Jeffrey L. Arthur 109. Statistical Design and Analysis of Industrial Experiments, edited by Subir Ghosh 110. ^/-Statistics: Theory and Practice, A. J. Lee 111. A Primer in Probability: Second Edition, Revised and Expanded, Kathleen Subrahmaniam 112. Data Quality Control: Theory and Pragmatics, edited by GunarE. Liepins and V. R. R. Uppuluri 113. Engineering Quality by Design: Interpreting the Taguchi Approach, Thomas B. Barker 114. Survivorship Analysis for Clinical Studies, Eugene K. Harris andAdelin Albert 115. Statistical Analysis of Reliability and Life-Testing Models: Second Edition, Lee J. Bain and Max Engelhardt 116. Stochastic Models of Carcinogenesis, Wai-Yuan Tan 117. Statistics and Society: Data Collection and Interpretation, Second Edition, Revised and Expanded, Walter T. Federer 118. Handbook of Sequential Analysis, B. K. Ghosh and P. K. Sen 119. Truncated and Censored Samples: Theory and Applications, A. Clifford Cohen 120. Survey Sampling Principles, £. K. Foreman 121. Applied Engineering Statistics, Robert M. Bethea and R. Russell Rhinehart 122. Sample Size Choice: Charts for Experiments with Linear Models: Second Edition, Robert E. Odeh and Martin Fox 123. Handbook of the Logistic Distribution, edited by N. Balakrishnan 124. Fundamentals of Biostatistical Inference, Chap T. Le 125. Correspondence Analysis Handbook, J.-P. Benzecri

126. Quadratic Forms in Random Variables: Theory and Applications, A. M. Mathai and Serge B. Provost 127. Confidence Intervals on Variance Components, Richard K. Burdick and Franklin A. Graybill 128. Biopharmaceutical Sequential Statistical Applications, edited by Karl E. Peace 129. Item Response Theory: Parameter Estimation Techniques, Frank B. Baker 130. Survey Sampling: Theory and Methods, Arijit Chaudhuri and Horst Stenger 131. Nonparametric Statistical Inference: Third Edition, Revised and Expanded, Jean Dickinson Gibbons and Subhabrata Chakraborti 132. Bivariate Discrete Distribution, Subrahmaniam Kocherlakota and Kathleen Kocherlakota 133. Design and Analysis of Bioavailability and Bioequivalence Studies, Shein-Chung Chow and Jen-pei Liu 134. Multiple Comparisons, Selection, and Applications in Biometry, edited by Fred M. Hoppe 135. Cross-Over Experiments: Design, Analysis, and Application, David A. Ratkowsky, Marc A. Evans, and J. Richard Alldredge 136. Introduction to Probability and Statistics: Second Edition, Revised and Expanded, Narayan C. Giri 137. Applied Analysis of Variance in Behavioral Science, edited by Lynne K. Edwards 138. Drug Safety Assessment in Clinical Trials, edited by Gene S. Gilbert 139. Design of Experiments: A No-Name Approach, Thomas J. Lorenzen and Virgil L Anderson 140. Statistics in the Pharmaceutical Industry: Second Edition, Revised and Expanded, edited by C. Ralph Buncherand Jia-Yeong Tsay 141. Advanced Linear Models: Theory and Applications, Song-Gui Wang and Shein-Chung Chow 142. Multistage Selection and Ranking Procedures: Second-Order Asymptotics, Nitis Mukhopadhyay and Tumulesh K. S. Solanky 143. Statistical Design and Analysis in Pharmaceutical Science: Validation, Process Controls, and Stability, Shein-Chung Chow and Jen-pei Liu 144. Statistical Methods for Engineers and Scientists: Third Edition, Revised and Expanded, Robert M. Bethea, Benjamin S. Duran, and Thomas L. Boullion 145. Growth Curves, Anant M. Kshirsagar and William Boyce Smith 146. Statistical Bases of Reference Values in Laboratory Medicine, Eugene K. Harris and James C. Boyd 147. Randomization Tests: Third Edition, Revised and Expanded, Eugene S. Edgington 148. Practical Sampling Techniques: Second Edition, Revised and Expanded, Ran/an K. Som 149. Multivariate Statistical Analysis, Narayan C. Giri 150. Handbook of the Normal Distribution: Second Edition, Revised and Expanded, Jagdish K. Pate/ and Campbell B. Read 151. Bayesian Biostatistics, edited by Donald A. Berry and Dalene K. Stangl 152. Response Surfaces: Designs and Analyses, Second Edition, Revised and Expanded, Andre I. Khuri and John A. Cornell 153. Statistics of Quality, edited by Subir Ghosh, William R. Schucany, and William B. Smith 154. Linear and Nonlinear Models for the Analysis of Repeated Measurements, Edward F. Vonesh and Vemon M. Chinchilli 155. Handbook of Applied Economic Statistics, Aman Ullah and David E. A. Giles 156. Improving Efficiency by Shrinkage: The James-Stein and Ridge Regression Estimators, Marvin H. J. Gruber 157. Nonparametric Regression and Spline Smoothing: Second Edition, Randall L. Eubank 158. Asymptotics, Nonparametrics, and Time Series, edited by Subir Ghosh 159. Multivariate Analysis, Design of Experiments, and Survey Sampling, edited by Subir Ghosh

160. Statistical Process Monitoring and Control, edited by Sung H. Park and G. Geoffrey Vining 161. Statistics for the 21st Century: Methodologies for Applications of the Future, edited by C. R. Rao and Gabor J. Szekely 162. Probability and Statistical Inference, Nitis Mukhopadhyay 163. Handbook of Stochastic Analysis and Applications, edited by D. Kannan and V. Lakshmikantham 164. Testing for Normality, Henry C. Thode, Jr. 165. Handbook of Applied Econometrics and Statistical Inference, edited by Aman Ullah, Alan T. K. Wan, and Anoop Chaturvedi 166. Visualizing Statistical Models and Concepts, R. W. Farebrother 167. Financial and Actuarial Statistics: An Introduction, Dale S. Borowiak 168. Nonparametric Statistical Inference: Fourth Edition, Revised and Expanded, Jean Dickinson Gibbons and Subhabrata Chakraborti 169. Computer-Aided Econometrics, edited by David E. A. Giles 170. The EM Algorithm and Related Statistical Models, edited by Michiko Watanabe and Kazunori Yamaguchi 171. Multivariate Statistical Analysis: Second Edition, Revised and Expanded, Narayan C. Gin 172. Computational Methods in Statistics and Econometrics, Hisashi Tanizaki 173. Applied Sequential Methodologies, edited by Nitis Mukhopadhyay, Sujay Datta, and Saibal Chattopadhyay Additional Volumes in Preparation

To My Family, Miyuki, Toshifumi and Yasuyuki

Preface As the personal computer progresses day by day, computer-intensive procedures have been developed in the field of statistics and econometrics. The computational procedures in statistics and econometrics include both Monte Carlo methods and nonparametric methods (or distribution-free methods). In the first half of this book, the Monte Carlo methods are discussed. That is, some representative random number generation methods and their applications are shown. The second half of this book is related to computer-intensive statistical techniques other than Monte Carlo methods and simulations, where the nonparametric methods are introduced. Chapter 1 is an introduction to statistics and econometrics, which corresponds to my lecture notes in mathematical statistics course (about 15 lectures, each 90 minutes long) for first-year graduate students. Based on Chapter 1, the Monte Carlo and nonparametric methods are discussed in Chapters 2-8. In the Monte Carlo methods, we discuss how to generate various random draws. Almost all the random draws are based on the uniform random draws. Therefore, it is one of the most important tasks to investigate uniform random number generation. Transforming the variable from the uniform random variable, various random draws are generated in Chapter 2, e.g., Bernoulli random draws, binomial random draws, normal random draws, %2 random draws, t random draws, F random draws, exponential random draws, gamma random draws, beta random draws, Cauchy random draws, logistic random draws and others. Importance resampling, rejection sampling and the Metropolis-Hastings algorithm are the methods to generate random draws from any distribution, which are useful tools for random number generation even when it is not easy to generate the random draws. Three sampling methods are discussed in Chapter 3. Thus, in the Monte Carlo methods, random number generation is very important. Once we have the random draws, simply the arithmetic average of the random draws indicates the estimate of mean. The arithmetic average is approximately distributed with a normal random variable by the central limit theorem. Therefore, the statistical inference also becomes quite easy, using the random draws. As some applications in the Monte Carlo methods, Baysian inference (Chapter 4), bias correction of the ordinary least squares (OLS) estimator in the autoregressive models (Chapter 5) and nonlinear non-Gaussian state space modeling (Chapter 6) are shown.

vi

PREFACE

In the nonparametric methods, nonparametric statistical tests are discussed in Chapters 7 and 8. Nonparametric tests of difference between two sample means include score tests (e.g., Wilcoxon rank sum test, normal score test, logistic score test, Cauchy score test and so on) and Fisher's randomization test (or Fisher's permutation test). The nonparametric tests of difference between two sample means are discussed in Chapter 7. One of the features of nonparametric tests is that we do not have to impose any assumption on the underlying distribution. From no restriction on the distribution, it could be expected that nonparametric tests would be less powerful than the conventional parametric tests such as the t test. However, it is shown that the Wilcoxon rank sum test is as powerful as the t test under the location-shift alternatives and moreover that the Wilcoxon test is sometimes much more powerful than the / test. Especially, the remarkable fact about the Wilcoxon test is that it is about 95 per cent as powerful as the t test for normal data. It is known that Pitman's asymptotic relative efficiency of the normal score test relative to the t test is greater than one under the location-shift alternatives. This implies that the power of the normal score test is always larger than that of the t test. It is known that the normal score test is less powerful than the Wilcoxon test if the tails of the underlying distributions are diffuse. Since in general the nonparametric tests require a large computational burden, however, there are few studies on small sample properties although asymptotic properties from various aspects were studied in the past. In addition to testing difference between two sample means, in Chapter 8 we also consider testing independence between two samples, which corresponds to testing correlation coefficient and regression coefficient. Small sample properties are discussed in the nonparametric statistical tests part of the book. Thus, some selected representative computer-intensive methods are treated in this book, where the source codes are shown by Fortran 77 and sometimes C languages for the purpose of practical understanding. For this book, I used seven personal computers, i.e., • Athlon 1.4 GHz CPU, Windows 2000 Operating System • Pentium III 1.4 GHz Dual CPU, and Windows 2000 Operating System • Pentium III 1.0 GHz Dual CPU, and Windows 2000 Operating System • Athlon 2000+ Dual CPU, and Windows 2000 Operating System • Athlon 2000+ Dual CPU, and Linux (Slackware 8.0) Operating System (see http: //www. slackware. com for Slackware) • Pentium III 1.4 GHz Dual CPU, and Linux (Plamo Linux 2.2.5, which is equivalent to Slackware+Japanese) Operating System (see http: //www. linet. jp/Plamo or http: //plamo-linux. jp for Plamo Linux, which is a Japanese site) • Pentium III 1.0 GHz Dual CPU, and Linux (Plamo Linux 2.2.5) Operating System

PREFACE

vii

For almost two years, my personal computers have been running all the time to prepare this book. Under the Windows 2000 Operating System, the following Fortran and C compilers are used for computation. • Open WATCOM C/C++ and Fortran 1.0 (http: //www. openwatcom. org) • Cygwin (http: //www. cygwin. com) • DJGPP(http://www.delorie.com/dj gpp) In addition to the free compilers shown above, in Section 7.5, Tables 7.4 and 7.5, the IMSL library (http: //www. vni . com/products/imsl) with Microsoft Fortran PowerStation Version 4.00 is used to obtain the percent points of the normal, t and F distributions. I am indebted to many people for assistance with this book. All cannot be mentioned in this short space, but I would like to acknowledge the Acquisitions Editor Taisuke Soda (Marcel Dekker, Inc.), who suggested that I write a book on statistical computing. I presented some chapters at the Econometrics workshop, Graduate School of Economics, Kobe University. I would like to thank the participants at the workshop for valuable comments. Furthermore, as mentioned above, Chapter 1 is based on my lecture notes in mathematical statistics course. Graduate students found some errors in the lecture notes. I am grateful to them for that. This research was partially supported by Japan Society for the Promotion of Science, Grants-in-Aid for Scientific Research (C)(2) #14530033 and Grants-in-Aid for the 21st Century COE Program #15COE500. I used MIpX to write this book. I did all of the typing, programming and so on. Therefore, all mistakes are mine. In programming, I use Fortran 77 in the first half of the book (Chapters 2-6) and C in the last half (Chapters 7 and 8). The pointer, which is often used in C language, is not familiar to Fortran users. Therefore, in Chapters 7 and 8, I did not use the pointer. Instead, I declared the external variables in the source codes, which are similar to the common sentence in Fortran 77. Finally, all the source codes used in this book are available in the CD-ROM attached with this book (therefore, readers do not have to type the source codes from the beginning).

Hisashi Tanizaki

Contents Preface 1

Elements of Statistics 1.1 Event and Probability 1.1.1 Event 1.1.2 Probability 1.2 Random Variable and Distribution 1.2.1 Univariate Random Variable and Distribution 1.2.2 Multivariate Random Variable and Distribution 1.2.3 Conditional Distribution 1.3 Mathematical Expectation 1.3.1 Univariate Random Variable 1.3.2 Bivariate Random Variable 1.4 Transformation of Variables 1.4.1 Univariate Case 1.4.2 Multivariate Cases 1.5 Moment-Generating Function 1.5.1 Univariate Case 1.5.2 Multivariate Cases 1.6 Law of Large Numbers and Central Limit Theorem 1.6.1 Chebyshev's Inequality 1.6.2 Law of Large Numbers (Convergence in probability) 1.6.3 Central Limit Theorem 1.7 Statistical Inference 1.7.1 Point Estimation 1.7.2 Statistic, Estimate and Estimator 1.7.3 Estimation of Mean and Variance 1.7.4 Point Estimation: Optimality 1.7.5 Maximum Likelihood Estimator 1.7.6 Interval Estimation 1.8 Testing Hypothesis 1.8.1 Basic Concepts in Testing Hypothesis 1.8.2 Power Function

v 1 1 1 2 4 4 8 10 11 11 16 22 22 24 24 24 27 29 29 32 33 35 35 36 36 38 43 46 49 49 50

CONTENTS 1.8.3 Testing Hypothesis on Population Mean 1.8.4 WaldTest 1.8.5 Likelihood Ratio Test 1.9 Regression Analysis 1.9.1 Setup of the Model 1.9.2 Ordinary Least Squares Estimation 1.9.3 Properties of Least Squares Estimator 1.9.4 Multiple Regression Model Appendix 1.1: Integration by Substitution Appendix 1.2: Integration by Parts Appendix 1.3: Taylor Series Expansion Appendix 1.4: Cramer-Rao Inequality Appendix 1.5: Some Formulas of Matrix Algebra References

1 Monte Carlo Statistical Methods 2 Random Number Generation I 2.1 Uniform Distribution: 17(0,1) 2.1.1 Properties of Uniform Distribution 2.1.2 Uniform Random Number Generators 2.2 Transforming t7(0,1): Continuous Type 2.2.1 Normal Distribution: MO, 1) 2.2.2 Normal Distribution: Nfacr1) 2.2.3 Log-Normal Distribution 2.2.4 Exponential Distribution 2.2.5 Gamma Distribution: G(a,/3) 2.2.6 Inverse Gamma Distribution: IG(a,fl) 2.2.7 Beta Distribution 2.2.8 Chi-Square Distribution: x\k) 2.2.9 F Distribution: F(m,ri) 2.2.10 ;Distribution: t(k) 2.2.11 Double Exponential Distribution (LaPlace Distribution) . . . 2.2.12 Noncentral Chi-Square Distribution:^2^; a) 2.2.13 Noncentral F Distribution: F(m,n; a) 2.2.14 Noncentral t Distribution: t(k;a) 2.3 Inverse Transform Method 2.3.1 Uniform Distribution: U(a,b) 2.3.2 Normal Distribution: N(Q,\) 2.3.3 Exponential Distribution 2.3.4 Double Exponential Distribution (LaPlace Distribution) . . . 2.3.5 Cauchy Distribution

51 52 54 58 58 59 60 66 68 69 70 70 74 75

77 79 79 79 80 84 84 91 92 93 95 97 98 101 108 Ill 116 118 120 121 122 123 125 126 127 128

CONTENTS

xi

2.3.6 Logistic Distribution 130 2.3.7 Extreme-Value Distribution (Gumbel Distribution) 131 2.3.8 Pareto Distribution 132 2.3.9 Weibull Distribution 134 2.4 Using U(Q, 1): Discrete Type 135 2.4.1 Rectangular Distribution (Discrete Uniform Distribution) . . . 136 2.4.2 Bernoulli Distribution 137 2.4.3 Geometric Distribution (Pascal Distribution) 138 2.4.4 Poisson Distribution 141 2.4.5 Binomial Distribution: B(n,p) 143 2.4.6 Negative Binomial Distribution 147 2.4.7 Hypergeometric Distribution 149 2.5 Multivariate Distribution 152 2.5.1 Multivariate Normal Distribution: 7V(//, Z) 152 2.5.2 Multivariate t Distribution 157 2.5.3 WishartDistribution: W(n,I) 159 2.5.4 Dirichlet Distribution 162 2.5.5 Multinomial Distribution 165 References 167 3 Random Number Generation II 3.1 Composition Method 3.1.1 Composition of Uniform Distributions 3.1.2 Normal Distribution: 7V(0,1) 3.1.3 Binomial Distribution: B(n,p) 3.1.4 Bimodal Distribution with Two Normal Densities 3.2 Rejection Sampling 3.2.1 Normal Distribution: N(Q, 1) 3.2.2 Gamma Distribution: G(a, 1) for 0 < a < 1 and 1 < a 3.3 Importance Resampling 3.3.1 Normal Distribution: N(0,\) 3.3.2 Gamma Distribution: G(a, l ) f o r O < a < 1 3.3.3 Beta Distribution 3.4 Metropolis-Hastings Algorithm 3.4.1 Normal Distribution: #(0,1) 3.4.2 Gamma Distribution: G(or, 1) forO < a < 1 3.4.3 Beta Distribution 3.5 Ratio-of-Uniforms Method 3.5.1 Normal Distribution: N(0,1) 3.5.2 Gamma Distribution: G(a,p) 3.6 Gibbs Sampling 3.7 Comparison of Sampling Methods 3.7.1 Standard Normal Random Number Generators

171 171 172 172 176 177 178 182 . . . . 183 187 188 191 194 195 202 204 206 208 209 210 215 217 217

xii

CONTENTS 3.7.2 3.7.3 3.7.4 3.7.5 References

II

Chi-Square Random Number Generators Binomial Random Number Generators Rejection Sampling, Importance Resampling and the Metropolis-Hastings Algorithm Sampling Density in the Metropolis-Hastings Algorithm . . .

219 221 222 227 244

Selected Applications of Monte Carlo Methods

247

4 Bayesian Estimation 4.1 Elements of Bayesian Inference 4.1.1 Bayesian Point Estimate 4.1.2 Bayesian Interval for Parameter 4.1.3 Prior Probability Density Function 4.2 Heteroscedasticity Model 4.2.1 Introduction 4.2.2 Multiplicative Heteroscedasticity Regression Model 4.2.3 Bayesian Estimation 4.2.4 Monte Carlo Study 4.2.5 Summary 4.2.6 Appendix: Are M- 5000 and N= 104 Sufficient? 4.3 Autocorrelation Model 4.3.1 Introduction 4.3.2 Setup of the Model 4.3.3 Monte Carlo Experiments 4.3.4 Summary References

249 249 250 250 251 253 253 254 257 259 267 268 269 270 270 274 281 282

5 Bias Correction of OLSE in AR Models 5.1 Introduction 5.2 OLSE Bias 5.3 Bias Correction Method 5.3.1 Optimization Procedure 5.3.2 Standard Error, Confidence Interval and Etc 5.3.3 Standard Error of Regression 5.4 Monte Carlo Experiments 5.4.1 AR(1) Models 5.4.2 AR(p) Models 5.5 Empirical Example 5.6 Summary 5.7 Appendix: Source Code References

285 285 286 290 292 293 293 293 294 299 302 307 307 311

CONTENTS

xiii

6 State Space Modeling 6.1 Introduction 6.2 State Space Models 6.2.1 Definition 6.2.2 Applications 6.3 Recursive Algorithm 6.3.1 Filtering 6.3.2 Smoothing 6.3.3 Discussion 6.3.4 Estimation of Parameter 6.4 Non-Recursive Algorithm 6.4.1 Smoothing 6.4.2 Estimation of Parameter 6.4.3 Discussion 6.5 Monte Carlo Studies 6.5.1 Simulation Procedure 6.5.2 Results and Discussion 6.6 Empirical Example 6.6.1 Introduction 6.6.2 Setup of the Model 6.6.3 Results and Discussion 6.6.4 Further Models and Discussion 6.6.5 Concluding Remarks 6.7 Summary Appendix 6.1: Density-Based Recursive Algorithm Appendix 6.2: Recursive and Non-Recursive Algorithms Appendix 6.3: Linear and Normal System Appendix 6.4: Two-Filter Formula References

313 313 315 315 316 324 326 328 330 332 335 337 338 339 341 341 342 351 351 352 359 364 368 371 373 374 376 382 383

III

Nonparametric Statistical Methods

7 Difference between Two-Sample Means 7.1 Introduction 7.2 Overview of Nonparametric Tests 7.2.1 Score Tests 7.2.2 Fisher's Randomization Test 7.3 Asymptotic Relative Efficiency 7.3.1 Score Test 7.3.2 fTest 7.3.3 Comparison between Two Tests 7.4 Power Comparison (Small Sample Properties)

391 393 393 394 395 398 399 399 405 406 409

xiv

8

CONTENTS 7.4.1 Setup of the Monte Carlo Studies 7.4.2 Results and Discussion 7.5 Empirical Example: Testing Structural Changes 7.6 Summary Appendix 7.1: On Generation of Combinations Appendix 7.2: Equivalence between Fisher and t Tests Appendix 7.3: Random Combination Appendix 7.4: Testing Structural Change References

411 412 425 431 432 438 439 440 442

Independence between Two Samples 8.1 Introduction 8.2 Nonparametric Tests on Independence 8.2.1 On Testing the Correlation Coefficient 8.2.2 On Testing the Regression Coefficient 8.3 Monte Carlo Experiments 8.3.1 On Testing the Correlation Coefficient 8.3.2 On Testing the Regression Coefficient 8.4 Empirical Example 8.5 Summary Appendix 8.1: Permutation Appendix 8.2: Distribution of p References

445 445 446 446 450 453 453 464 474 477 478 479 484

Source Code Index

485

Index

489

List of Tables 1.1

Type I and Type II Errors

3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 3.12 3.13 3.14 3.15

Precision of the Estimates and Computational Time Precision of the Estimates and Computational Time Precision of the Estimates and Computational Time Comparison of Three Sampling Methods Comparison of Three Sampling Methods: CPU Time (Seconds) Sampling Density III: Classification into the Four Cases Estimates of the Moments Standard Errors of the Estimated Moments Sampling Density III (Estimated Moments and Standard Errors) /(5) Distribution (Sampling Density I) Logistic Distribution (Sampling Density I) LaPlace Distribution (Sampling Density I) Gumbel Distribution (Sampling Density I) Sampling Density II Sampling Density III

49 218 220 221 223 . . . 224 229 231 232 . . . 232 240 240 241 241 242 242

4.1 The Exogenous Variables x\j and x^, 4.2 The AVE, RMSE and Quartiles: n = 20 4.3 BMLE: n = 15, c = 2.0, M = 5000 and N = 104 4.4 BMLE: n = 20 and c = 2.0 4.5 MLE:« = 20andp = 0.9 4.6 BE with M = 5000 and W = 104: « = 20andp = 0.9 4.7 BE with M = 5000 and N = 5000: n = 20 and p = 0.9 4.8 BE with M = 1000 and TV = 104: n = 20andp = 0.9

260 263 267 268 277 277 277 278

5.1 Estimates of a{ (Case: fa = Q) — N(0,1) Error 5.2 Estimates of a-, (Case:/32 = 0) — C r 2 ( l ) - I ) / V 2 Error 5.3 Esti mates of av (Case: & = 0) — £/(-V3, V5) Error 5.4 AR(2) Model: N(0,1) and U(- A/3, A/3) Errors for n = 20,40,60 ... 5.5 AR(3) Models: N(0,1) Error for n = 20,40 5.6 U.S. Consumption Function: 1961 - 1998 5.7 Japanese Consumption Function: 1961 - 1998

295 296 297 300 301 304 304

xvi

LIST OF TABLES 6.1 6.2 6.3 6.4 6.5 6.6 6.7 6.8 6.9 6.10 6.11 6.11

Revision Process of U.S. National Accounts (Nominal GDP) /(-) and q(-) for Densities (6.19) -(6.21), (6.23) and (6.24) Number of Generated Random Draws at Time t Filtering (6.19) Smoothing (6.21) with Filtering (6.19) Computation Time (minutes) Filtering and Smoothing Using Sampling Density S(6.21) withF(6.19): N' = 1000,2000,5000 IRS(6.71)BasedonIRF(6.19) Basic Statistics in Models 1-3 Basic Statistics in Models 2a - 2c Basic Statistics in Models 2a - 2c—< Continued >—

320 330 331 343 344 345 348 349 350 359 366 367

7.1 7.2 7.2 7.2 7.2 7.2 7.3 7.4 7.5 7.6 7.7

ARE of Score Tests Relative to / Test Empirical Sizes and Sample Powers Empirical Sizes and Sample Powers—< Continued >— Empirical Sizes and Sample Powers—< Continued >— Empirical Sizes and Sample Powers—< Continued >— Empirical Sizes and Sample Powers—< Continued >— Number of Combinations: (N)andnl = « 2 = 15 Testing Structural Change by Nonparametric Tests: p-Values Testing Structural Change by Nonparametric Tests: p-Values Output of Programs I & II CPU Time (Seconds)

409 413 414 415 416 417 424 428 429 433 437

8.1 8.1 8.1 8.1 8.1 8.2 8.2 8.2 8.2 8.2 8.3 8.4 8.5

Empirical Sizes and Sample Powers (H0 : p = 0 and H\ : p > 0) . . . 454 Empirical Sizes and Sample Powers—< Continued >— 455 Empirical Sizes and Sample Powers—< Continued >— 456 Empirical Sizes and Sample Powers—< Continued >— 457 Empirical Sizes and Sample Powers—< Continued >— 458 Empirical Sizes and Sample Powers (H0 : ft = 0 for i = 2,3,4). . . . 465 Empirical Sizes and Sample Powers —< Continued >— 466 Empirical Sizes and Sample Powers —< Continued >— 467 Empirical Sizes and Sample Powers —< Continued >— 468 Empirical Sizes and Sample Powers —< Continued >— 469 CPU Time (minutes) 474 t(n - k) versus Permutation 475 Output by permutation(n): Caseofn=4 479

List of Figures 1.1 1.2 1.3 1.4

Probability Function f(x) and Distribution Function F(x) Density Function f ( x ) and Distribution Function F(x) Type I Error (a) and Type II Error OS) True and Estimated Regression Lines

2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12 2.13 2.14 2.15 2.16 2.17 2.18 2.19 2.20 2.21 2.22 2.23 2.24 2.25

Uniform Distribution: [7(0,1) 79 Standard Normal Distribution: N(0,1) 84 Normal Distribution: N(^,cr2) 91 Log-Normal Distribution 92 Exponential Distribution 94 Gamma Distribution: a= 1,2,3 and/? = 1 95 Beta Distribution 99 Chi-Square Distribution:^2^) 101 F Distribution 109 tDistribution: t(k) Ill Double Exponential Distribution (Laplace Distribution) 116 Inverse Transformation Method: Continuous Type 123 Uniform Distribution: U(a, b) 124 Cauchy Distribution: a = 0 and 0 - 1 128 Logistic Distribution: a - 0 and/3 = 1 130 Extreme-Value Distribution (Gumbel Distribution): a = 0 and /? = 1 . 131 Pareto Distribution 133 Weibull Distribution: a= l , 2 , 3 a n d / 2 = 1 134 Inverse Transformation Method: Discrete Type 136 Rectangular Distribution (Discrete Uniform Distribution) 136 Bernoulli Distribution 138 Geometric Distribution (Pascal Distribution) 139 Poisson Distribution 141 Binomial Distribution 144 Negative Binomial Distribution 147

3.1 3.2

Approximation of Standard Normal Density Rejection Sampling xvii

7 8 51 59

173 181

xviii

LIST OF FIGURES

3.3 3.4 3.5 3.6 3.7

Acceptance Probability: Equation (3.7) Bimodal Normal Distribution: f ( x ) Bimodal Normal Distribution: p'(x) and p"(x), where p(x) = log/(AT) Acceptance Probabilities in Sampling Density I: Contour Lines . . . . Acceptance Probabilities in Sampling Density II

212 228 228 233 234

4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 4.10 4.11 4.12 4.13

Acceptance Rates in Average: M = 5000 and N = 104 Empirical Distributions of $\ Empirical Distributions of /J2 Empirical Distributions of/83 Empirical Distributions of 71 Empirical Distributions of 72 The Arithmetic Average from the 104MLE's of AR(l)Coeff. The Arithmetic Average from the KOBE'S of AR(l)Coeff. Empirical Distributions of J3i Empirical Distributions of j82 Empirical Distributions of j83 Empirical Distributions of p Empirical Distributions of a-2f

261 264 264 265 265 266 276 276 279 279 279 280 280

5.1 5.2 5.3 5.4 5.5

The Arithmetic Average from the 104 OLSEs of AR(l)Coeff. The Arithmetic Average from the 104 OLSEs of AR(l)Coeff. The Arithmetic Average from the 104 OLSEs of AR(l)CoefT. U.S. Consumption Function Japanese Consumption Function

. . . . 288 . . . . 288 . . . . 289 305 306

6.1 6.2 6.3 6.4 6.5 6.6 6.7

Nikkei Stock Average (Japanese Yen) Percent Changes of Nikkei Stock Average (%) Movement of Trend Component a, (%) Diffusion and Composit Indices Volatility in Models 2 and 3 Movement of Trend Component a, (%): Models 2a-2c Volatility in Models 2a - 2c

351 352 361 362 363 369 370

7.1 7.1 7.2 7.3 7.4

Sample Powers: n = 12 and a = 0.10 Sample Powers: n = 12 and a = 0.10—< Continued >— p-Values — Import Function (7.19): Table 7.3 p-Values — Import Function (7.20): Table 7.5 Tree Diagram

421 422 430 430 436

8.1 Sample Powers: n= 8.1 Sample Powers: « = 8.2 Sample Powers: n = 8.2 Sample Powers: n-

10 and a- 0.10 461 10 and a = 0.10—< Continued >— 462 10, k = 4, a = 0.10and#• = Ofor i = 1,2,3 . . . 471 10,fc= 4, a = O.lOand/?, = 0 f o r i = 1,2,3 . . . 472

LIST OF FIGURES 8.3

Empirical Distribution for £3 in Equation (7.20)

xix 476

Computational Methods in Statistics and Econometrics

Chapter 1 Elements of Statistics In this chapter, the statistical methods used in the proceeding chapters are summarized. Mood, Graybill and Bose (1974), Hogg and Craig (1995) and Stuart and Ord (1991, 1994) are good references in Sections 1.1 - 1.8, while Judge, Hill, Griffiths and Lee (1980) and Greene (1993, 1997, 2000) are representative textbooks in Section 1.9.

1.1 Event and Probability 1.1.1

Event

We consider an experiment whose outcome is not known in advance but an event occurs with probability, which is sometimes called a random experiment. The sample space of an experiment is the set of all possible outcomes. Each element of a sample space is called an element of the sample space or a sample point, which represents each outcome obtained by the experiment. An event is any collection of outcomes contained in the sample space, or equivalently a subset of the sample space. A simple event consists of exactly one element and a compound event consists of more than one element. Sample space is denoted by D and sample point is given by a>. Suppose that event A is a subset of sample space D. Let a> be a sample point in event A. Then, we say that a sample point w is contained in a sample space A, which is denoted by w e A. A set of the sample points which does not belong to event A is called the complementary event of A, which is denoted by Ac. An event that does not have any sample point is called the empty event, denoted by 0. Conversely, an event which includes all possible sample points is called the whole event, represented by fl. Next, consider two events A and B. A set consisting of the whole sample points which belong to either event A or event B is called the sum event, which is denoted by A n B. A set consisting of the whole sample points which belong to both event A and event B is called the product event, denoted by A n B. When A n B - 0, we say that events A and B are mutually exclusive.

2

CHAPTER 1. ELEMENTS OF STATISTICS

Example 1.1: Consider an experiment of casting a die. We have six sample points, which are denoted by W] = {1 }, a>2 = (2), 013 = (3), 5 = {5} and , represents the sample point that we have i. In this experiment, the sample space is given by Q = {toi,to2,toi,u>4,u>s,<j}6}. Let A be the event that we have even numbers and B be the event that we have multiples of three. Then, we can write as A = {to2,to4,tot} and B = {to-j,tof}. The complementary event of A is given by Ac = {0)1,0)3,0)5}, which is the event that we have odd numbers. The sum event of A and B is written as A U B = [0)2, 0*3, a>4, o>g}, while the product event is A n B = {o^}. Since A n Ac = 0, we have the fact that A and Ac are mutually exclusive. Example 1.2: Cast a coin three times. In this case, we have the following eight sample points: tol = (H,H,H), o>5 = (T,H,H),

o>2 = (H,H,T), o>6 = (T,H,T),

0)3 = (H,T,H), o>7 = (T,T,H),

o>4 = (H,T,T), o>8 = (T,T,T),

where H represents head while T indicates tail. For example, (H,T,H) means that the first flip lands head, the second flip is tail and the third one is head. Therefore, the sample space of this experiment can be written as: il = {0)1,012,0)3,014,0)5,0)6,0)7,018}.

Let A be an event that we have two heads, B be an event that we obtain at least one tail, C be an event that we have head in the second flip, and D be an event that we obtain tail in the third flip. Then, the events A, B, C and D are give by: A = {0)2,0)3,0)5},

B= {0>2,O)3,OI 4 ,O>5,O)6, 0>7,0>8),

C - {o>i,O)2, 0)5,0)6),

D = {o>2, W4,O)6,O)8}.

Since A is a subset of B, denoted by A c B, a sum event is A U B = B, while a product event is A n B = A. Moreover, we obtain C n D - {o>2, o)6) and C U D = {0),,OI 2 ,0> 4 ,0)5,0> 6 ,OI 8 }.

1.1.2 Probability Let n(A) be the number of sample points in A. We have n(A) < n(B) when A c B. Each sample point is equally likely to occur. In the case of Example 1.1 (Section 1.1.1), each of the six possible outcomes has probability 1/6 and in Example 1.2 (Section 1.1.1), each of the eight possible outcomes has probability 1/8. Thus, the probability which the event A occurs is defined as:

In Example 1.1, P(A) = 3/6 and P(A n B) = 1/6 are obtained, because n(fi) = 6, n(A) ~ 3 and n(A n B) = 1. Similarly, in Example 1.2, we have P(C) = 4/8,

1.1. EVENT AND PROBABILITY

3

P(A n B) = P(A) = 3/8 and so on. Note that we obtain P(A) < P(B) because of A CB. It is known that we have the following three properties on probability: 0 2 , - - - , respectively. The probability which event {a>;X((jS) - x, and Y(OJ) = yj) occurs is given by:

1.2. RANDOM VARIABLE AND DISTRIBUTION

9

where /^(x,, yt) represents the joint probability function of X and Y. In order for fxy(Xi,yj) to be a joint probability function, fv(xi,yj) has to satisfies the following properties: /„.(**, ?;)> 0, i

i,j=l,2,-..

j

Define ./*(.*,•) and .^Cvj) as:

Then, /*(*,-) and fyfyj) are called the marginal probability functions of X and 7. /,(*;) and ./j,(yj) also have the properties of the probability functions, i.e., fx(xt) > 0 and £,/,U-) = 1, and fy(y}) > 0 and £, /,(?;) = 1Continuous Random Variables: Consider two continuous random variables X and y. For a domain D, the probability which event {a»; (X(cj), y(w)) 6 D} occurs is given by: P((X, Y) € D) = JT /„(*,>•) d;c d>, where /^U.j) is called the joint probability density function of X and Y or the joint density function of X and Y. fv(x,y) has to satisfy the following properties:

fxy(x,y)dxdy=]. U — IX)

U-&3

Define fx(x) and fy(y) as: -00

/vW = I

/Aj^.y)^,

for all x and j,

*J -OQ

fy(y)= f /^y)^, t/ — OO

where /^(;c) and _^,(y) are called the marginal probability density functions of X and Y or the marginal density functions of X and y. For example, consider the event [at; a < X(w) < b, c < Y(CJ) < d], which is a specific case of the domain D. Then, the probability that we have the event {u>;a < X(w) < b, c < Y(CJ) < d} is written as:

P(a<X 0,

for all y and / = 1,2, • • -,

fxy(xi,y)Ay - 1. i

The marginal probability function of X is given by: fCO

fx(xi)=

I

fxy(Xi,y) Ay,

for ;' = 1,2, • • •. The marginal probability density function of Y is:

1.2.3

Conditional Distribution

Discrete Random Variable: y'i is represented as:

The conditional probability function of X given Y =

The second equality indicates the definition of the conditional probability, which is shown in Section 1.1.2. The features of the conditional probability function /^.fety/) are: yj) = 1 ,

for any j.

Continuous Random Variable: The conditional probability density function of X given Y = y (or the conditional density function of X given Y = y) is:

The properties of the conditional probability density function fx\y(x\y) are given by:

f*iy(x\y) > o, f°° I

fx\y(x\y) Ax=\,

for any Y = y.

1.3. MATHEMATICAL

EXPECTATION

11

Independence of Random Variables: For discrete random variables X and Y, we say that X is independent (or stochastically independent) of Y if and only if fxy(Xj, >'/) = fx(Xi)fy(yj). Similarly, for continuous random variables X and 7, we say that X is independent of Y if and only if fly(x, y) = fx(x)fy(y). When X and Y are stochastically independent, g(X) and h(Y) are also stochastically independent, where g(X) and h(Y) are functions of X and Y.

1.3 Mathematical Expectation 1.3.1

Univariate Random Variable

Definition of Mathematical Expectation: Let g(X) be a function of random variable X. The mathematical expectation of g(X), denoted by E(g(X)), is defined as follows: g(Xj)pi = y g(xt)f(xi), i g(x)f(x) Ax,

(Discrete Random Variable), (Continuous Random Variable).

The following three functional forms of g(X) are important.

1. g(X) = X. The expectation of X, E(X), is known as mean of random variable X. V Xif(xi),

(Discrete Random Variable),

r-»CO

xf(x) dx,

(Continuous Random Variable),

When a distribution of X is symmetric, mean indicates the center of the distribution. 2.

The expectation of (X - /j)2 is known as variance of random variable X, which is denoted by V(X). / t ( x j - fi)2f(Xi), (jc — fj.) f(x) dx, 0

= crl,

(or cr2).

(Discrete Random Variable), (Continuous Random Variable),

12

CHAPTER 1. ELEMENTS OF STATISTICS If X is broadly distributed, cr2 = V(X) becomes large. Conversely, if the distribution is concentrated on the center, 0

(Discrete Case),

^»00

I

I (x - Hx)(y - Hy)f(x,y) dx dy, (Continuous Case).

*7 —00 ^/ —00

Thus, covariance is defined in the case of bivariate random variables.

4. g(X, Y) = The mathematical expectation of ee'y+*y is called the moment-generating function, which is denoted by 4>(6\, #2) and written as:

ea^i+lhyif(xt,yj), svx

I

(Discrete Case),

/^co

I

eBt*+ ?;) + Z Z y.i = E(X) + E(F). For continuous random variables X and Y, we can show: yTCO

E(X+Y)=

f^Ca

I

(x +

y)fxy.(X,y)dxdy

*J— co »7— co

.y) djrdy -ca U-ca

2. Theorem: E(XY) = E(X)E(Y), when X is independent of Y. Proof: For discrete random variables X and Y,

If X is independent of F, the second equality holds, i.e., /,.(*,-, y.,-) =

fx(xi)fy(yj).

For continuous random variables X and F, E(XF)= \J-CO

iJ-OO

dj: dy

xfx(x) Ax}( ]

yfy(y) dy) = E(X)E(F). J-CQ

When A" is independent of F, we have f,y(x,y) = fx(x)fy(y) ity. 3. Theorem: Cov(X, F) = E(XY) -

E(X)E(Y).

in the second equal-

1.3. MATHEMATICAL EXPECTATION

19

Proof: For both discrete and continuous random variables, we can rewrite as follows: Cov(X, Y) = E((X - fitKY - Mv)) = E(XY - ^ - (iy = E(XY) - E&XY) - EfayX) + Wy = E(XY) - ^E(T) - nyE(X) + wy = E(XY) - nxHy - HyHx + fixtiy = E(XY) = E(XY) In the fourth equality, the theorem in Section 1 .3. 1 is used, i.e., E(jixY) = fixE(Y) and EfayX) = fiyE(X). 4. Theorem: Cov(X, Y) = 0, when X is independent of Y. Proof: From the above two theorems, we have E(XF) = E(X)E(Y) when X is independent of Y and Cov(X, Y) = E(XY) - E(X)E(Y). Therefore, Cov(X, Y) = 0 is obtained when X is independent of Y. 5. Definition: The correlation coefficient between X and Y, denoted by p^,, is defined as: ~

Cov(X, Y)

—

Cov(X, Y)

When pxy > 0, we say that there is a positive correlation between X and Y. As pxy approaches 1 , we say that there is a strong positive correlation between X and Y. When pxy < 0, we say that there is a negative correlation between X and Y. As p^. approaches - 1 , we say that there is a strong negative correlation between X and Y. 6. Theorem: pi7 = 0, when X is independent of Y. Proof: When X is independent of Y, we have Cov(X, Y) = 0. Therefore, we can obtain the result p^, =

= 0. However, note thatp^ = 0 does not mean yV(T) VV(T) the independence between X and Y.

1. Theorem: V(X ±Y) = V(X) ± 2Cov(X, Y) + V(Y). Proof: For both discrete and continuous random variables, V(X ± Y) is rewritten as follows: V(X ±Y) = E(((X ±Y)- E(X ± 7))2) = E(((X - /j,) ± (7 - ^,))2)

20

CHAPTER 1 . ELEMENTS OF STATISTICS = E((X - /O2 ± 2(X - fix)(Y - fiy) + (Y- ^,)2) = E((X - p,)2) ± 2E((X - |O(y - Hy)) + E((K - ^,)2) ± 2Cov(X, Y) + V(T). 8. Theorem: - 1 < p^ < 1 . Proof: Consider the following function of t: f(t) = V(Xt - Y), which is always greater than or equal to zero because of the definition of variance. Therefore, for all t, we have f(t) > 0. f(t) is rewritten as follows: f(t) = V(Xt -Y) = V(Xt) - 2Cov(Xt, Y) + V(7) = t2V(X) - 2tCov(X, Y) + V(K) (Cov(X,Y))2 V(X) ' In order to have f(t) > 0 for all /, we need the following condition:

, V(X)

~

because the first term in the last equality is nonnegative, which implies: (Cov(X. F))2 Therefore, we have:

From the definition of correlation coefficient, i.e., p~, =

'

we

obtain the result: -1 < pAy < 1 . 9. Theorem: V(X + F) = V(X) + V(7), when X is independent of Y. Proof: From the theorem above, V(X± Y) = V(X) + 2Cov(X, Y) + V(F) generally holds. When random variables X and Y are independent, we have Cov(X, Y) = 0. Therefore, V(X + Y) = V(X) + V(Y) holds, when X is independent of Y. 10. Theorem: For « random variables X\, X^, • • -, Xn,

V(

1.3. MATHEMATICAL EXPECTATION

21

where E(X;) = ^ and at is a constant value. Especially, when X\, X2, • • •, Xn are mutually independent, we have the following:

Proof: For mean of £,• a,-X,-, the following representation is obtained.

The first and second equalities come from the previous theorems on mean. For variance of £,- fliX;, we can rewrite as follows:

When Xi, Xj, • • -, Xn are mutually independent, we obtain Cov(X,-,X,-) = 0 for all i 2 j from the previous theorem. Therefore, we obtain:

Note that Cov(X,.,X,.) = E((X; - //)2) = V(X;). 11. Theorem: n random variables X], X2, • • • , Xn are mutually independently and identically distributed with mean n and variance a2. That is, for all i = 1 , 2, • • • , n, E(X,-) = JJL and V(X,-) = a2- are assumed. Consider arithmetic average X = (1/n) 2"=i ^i- Then, mean and variance of X are given by:

E(X) = n,

V(X) = — . «

Proof: The mathematical expectation of X is given by:

E(aX) = aE(X) in the second equality and E(X + Y) = E(X) + E(F) in the third equality are utilized, where X and Y are random variables and a is a constant value. For these formulas, see p. 1 5 in Section 1.3.1 and p. 1 7 in this section.

22

CHAPTER 1. ELEMENTS OF STATISTICS The variance of X is computed as follows: i " i " i « i " i V(X) = V(- V Xi) = -V( Y X,) = - V V(X,) = - V (T2 = -rtcr2 n 4V « 4V n 4V » 4V «

We use V(flX) = a2V(X) in the second equality and V(X + Y) = V(X) + V(Y) for X independent of Y in the third equality, where X and F denote random variables and a is a constant value. For these formulas, see p. 1 5 in Section 1 .3. 1 and p.20 in this section.

1.4 Transformation of Variables Transformation of variables is used in the case of continuous random variables. Based on a distribution of a random variable, a distribution of the transformed random variable is derived. In other words, when a distribution of X is known, we can find a distribution of Y using the transformation of variables, where Y is a function of X.

1.4.1

Univariate Case

Distribution of Y = i/t~l(X): Let fx(x) be the probability density function of continuous random variable X and X = t//(Y) be a one-to-one transformation. Then, the probability density function of F, i.e., /y(y), is given by:

We can derive the above transformation of variables from X to F as follows. Let ft(x) and Fx(x) be the probability density function and the distribution function of X, respectively. Note that Fx(x) = P(X < x) and fx(x) = F'x(x). When X = t//(Y), we want to obtain the probability density function of F. Let /y(y) and Fy(y) be the probability density function and the distribution function of F, respectively. In the case of ^'(X) > 0, the distribution function of F, Fy(y), is rewritten as follows: Fv(y) = P(F 0 and i/t'(X) < 0, equations (1 .4) and (1 .5) indicate the following result:

/>>(>') = I^ which is called the transformation of variables. Example 1.9: When X has a standard normal density function, i.e., when X N(0, 1 ), we derive the probability density function of Y, where Y = p + crX . Since we have: i]j'(y) = 1 1 a- is obtained. Therefore, the density function of F, fy(y), is given by:

/,GO = l0' which indicates the normal distribution with mean // and variance cr2, denoted by

On Distribution of Y = X2: As an example, when we know the distribution function of X as Fx(x), we want to obtain the distribution function of Y, Fy(y), where Y = X2. Using Fx(x), Fy(y) is rewritten as follows:

Therefore, when we have fx(x) and Y = X2, the probability density function of Y is obtained as follows:

fy(y) = Fy(y) =

CHAPTER 1. ELEMENTS OF STATISTICS

24

1.4.2

Multivariate Cases

Bivariate Case: Let f^(x, y) be a joint probability density function of X and Y. Let X = if/^U, V) and Y = t//2(U, V) be a one-to-one transformation from (X, Y) to (U, V). Then, we obtain a joint probability density function of U and V, denoted by fuv(u, v), as follows: where J is called the Jacobian of the transformation, which is defined as: dx dx du dv J= dy dy du dv Multivariate Case: Let fz(x\, X2, • • • , xn) be a joint probability density function of X i , X2, ••• X,,. Suppose that a one-to-one transformation from ( X i , X 2 , - - - , X n ) to (y,,y 2 ,-..,y,,)isgivenby:

Then, we obtain a joint probability density function of YI, Y2, • • - , Yn, denoted by fy(y\,yi, •••,}'„), as follows:

fy(y\,yi, •••,yn) where 7 is called the Jacobian of the transformation, which is defined as:

J=

dx\

dx\

dx\

y']

dy2

dy,,

dy}

dy2

dyn

dxn

dxn

dx,, dyn

1.5 Moment-Generating Function 1.5.1 Univariate Case As discussed in Section 1.3.1, the moment-generating function is defined as tf>(6) = E(eox). In this section, several important theorems and remarks of the momentgenerating function are summarized.

1.5. MOMENT-GENERATING

FUNCTION

25

For a random variable X, ^'n = E(X") is called the nth moment of X. Then, we have the following first theorem. 1. Theorem: (ff) is written as: ee*f(x) Ax. The nth derivative of>(0), denoted by 0(n)(0), is:

Evaluating ^("'(0) at 6 - 0, we obtain: 0 (n) (0)= f *»/(*) dx = E(X") = /4 J -00

where the second equality comes from the definition of the mathematical expectation. 2. Remark: Let X and Y be two random variables. When the moment-generating function of X is equivalent to that of Y, we have the fact that X has the same distribution as Y. 3. Theorem: Let (&) be the moment-generating function of X. Then, the moment-generating function of Y, where Y - aX + b, is given by eb6(aff). Proof: Let 4>y(ff) be the moment-generating function of Y. Then, \(6), $2(6), • • •, n(ff) bg the moment-generating functions of X, , X 2 , - - •, Xn, which are mutually independently distributed random variables. Define Y = X^ + X2 + • • • + Xn. Then, the moment-generating function of Y is given by ) = M f| E(e^') =1 f] J 0& „ = \ (0(^))". n) Example 1.10: For the binomial random variable, the moment-generating function 0(0) is known as: 0(0) = (pe° + 1 - p f ,

which is discussed in Example 1.5 (Section 1.3.1). Using the moment-generating function, we check whether E(X) = np and V(X) = np(\ - p) are obtained when X is a binomial random variable. The first- and the second-derivatives with respect to 9 are given by: "(0) = npe\pea + 1 - p)"'1 + n(n - \)P1e1\pee + 1 - p)""2.

1.5.

MOMENT-GENERATING

FUNCTION

27

Evaluating at 9 = 0, we have:

0'(0) = np,

E(X2) = 0"(0) = np + n(n-\ )pi.

Therefore, V(X) = E(X2) - (E(X))2 = np(l - p) can be derived. Thus, we can make sure that E(X) and V(X) are obtained from (6).

1.5.2 Multivariate Cases Bivariate Case: As discussed in Section 1.3.2, for two random variables X and F, the moment- generating function is defined as 0(01; 62) = E(e8lX+S2Y). Some useful and important theorems and remarks are shown as follows. 1. Theorem: Consider two random variables X and Y. Let 0(0i,02) be the moment-generating function of X and Y. Then, we have the following result:

Proof: Let fxy(x,y) be the probability density function of X and Y. From the definition, 0(0i,02) is written as: 0(0,,02) = E(eeiX+thY) -

I

I

ee":+ 2 4-* ;=i ' which is equivalent to the moment-generating function of the normal distribution with mean // £"=i ai ar>d variance cr2 2;Li «?> where n and cr2 in (f>x(ff) is simply replaced by // 2"=1 a/ and cr2 £"=] a2 in 0^(0), respectively. Moreover, note as follows. When at = 1 /« is taken for all z' = 1,2, - • - , « , i.e., when ft = X is taken, ft = X is normally distributed as: X ~ N(JJ, a2In). The readers should check difference between Theorem 11 on p.21 and this theorem.

1.6 1.6.1

Law of Large Numbers and Central Limit Theorem Chebyshev's Inequality

In this section, we introduce Chebyshev's inequality, which enables us to find upper and lower bounds given a certain probability.

30

CHAPTER 1. ELEMENTS OF STATISTICS

Theorem: Let g(X) be a nonnegative function of the random variable X, i.e., g(X) > 0. If E(g(X)) exists, then we have:

(1.6) for a positive constant value k. Proof: We define the discrete random variable U as follows:

Thus, the discrete random variable U takes 0 or 1 . Suppose that the probability function of U is given by: /(«) = P(U = «), where P(U = u) is represented as:

P((7 = 0) = P(g(X) < k). Then, in spite of the value which U takes, the following equation always holds:

which implies that we have g(X) > k when U = 1 and g(X) > 0 when U = 0, where k is a positive constant value. Therefore, taking the expectation on both sides, we obtain: E(g(X)) > kE(U), (1.7) where E(£7) is given by:

i E(£7) = ^ UP(U = u) = \xP(U=\) + GxP(U = ff)= P(U = 1) u=6

= P(g(X)>k).

(1.8)

Accordingly, substituting equation (1.8) into equation (1.7), we have the following inequality:

1.6. LAW OF LARGE NUMBERS AND CENTRAL LIMIT THEOREM

31

Chebyshev's Inequality: Assume that E(X) = //, V(X) - cr2, and A is a positive constant value. Then, we have the following inequality: P(\X-n\>Acr)Ao) ACT) + P(\X-/j] < ACT) = 1 , we can derive the following inequality: v\}-.

(1.9)

An Interpretation of Chebyshev's inequality: 1 /A2 is an upper bound for the probability P(\X - n\ > ACT). Equation (1.9) is rewritten as:

P(jj - Aa- < X < n + ACT) > 1 -- -. That is, the probability that X falls within ACT units of jj is greater than or equal to 1 - 1 1 A2. Taking an example of A - 2, the probability that X falls within two standard deviations of its mean is at least 0.75. Furthermore, note as follows. Taking e = ACT, we obtain as follows:

i.e.,

V(X) P(\X - E(X)\ >e)< ——, e1 which inequality is used in the next section.

(1 10)

32

CHAPTER 1. ELEMENTS OF STATISTICS

Remark: Equation (1.10) can be derived when we take g(X) = (X - //)2, // = E(X) and k = e2 in equation (1.6). Even when we have ji ± E(X), the following inequality still hold:

Note that E((X - yu)2) represents the mean square error (MSB). When // = E(X), the mean square error reduces to the variance.

1.6.2

Law of Large Numbers (Convergence in probability)

Law of Large Numbers: Assume that X\, X2, • • - , Xn are mutually independently and identically distributed with mean E(A";) = p. and variance V(X,) = a2 < oo for all /. Then, for any positive value e, as n —> oo, we have the following result: ti\>£)

— > 0,

where Xn = (1 /«) £"=1 Xt. We say that Xn converges to // in probability. Proof: Using (1.10), Chebyshev's inequality is represented as follows: P(\Xn - E(X,,)\ > e) where X in (1.10) is replaced by Xn. As in Section 1.3.2 (p.21), we have E^,,) = /j and V(Xn) - a2 In, which are substituted into the above inequality. Then, we obtain:

P(\Xn-H\>e)< —2 . tie Accordingly, when n —> oo, the following equation holds: 2

P(\Xn-IJ.\>e) n is obtained as n —> oo, which is written as: plimX n = //. This theorem is called the law of large numbers. The condition P(\Xn -[i\ > e) —> 0 or equivalently P(\Xn -fi\ < e) —> 1 is used as the definition of convergence in probability. In this case, we say that Xn converges to // in probability.

1.6. LAW OF LARGE NUMBERS AND CENTRAL LIMIT THEOREM

33

Theorem: In the case where X\, Xi, • • •, Xn are not identically distributed and they are not mutually independently distributed, we assume that

V n —" —> 0,

as n

Then, we obtain the following result:

That is, Xn converges to lim —- in probability. This theorem is also called the law of n-too n large numbers.

1.6.3

Central Limit Theorem

Central Limit Theorem: X, , X2, • • -, Xn are mutually independently and identically distributed with E(X,) = n and V(X,) = cr2 for all i. Both /j and cr2 are finite. Under the above assumptions, when n —> oo, we have:

which is called the central limit theorem. Proof: Define F/ = —- . We can rewrite as follows: 0 as n —> oo. Since ,(9) = e^ is the moment-generating function of the standard normal distribution (see p. 14 in Section 1.3.1 for the moment-generating function of the standard normal probability density), we have:

or equivalently,

o-/V« The following expression is also possible:

1.7. STATISTICAL INFERENCE

35

Corollary 1: When E(X;) = p, V(X;) = a2 and I,, = (!/«) £?=i X;, note that

Therefore, we can rewrite the above theorem as:

Corollary 2: Consider the case where Xi, ^2, • • •, X,, are not identically distributed and they are not mutually independently distributed. Assume that lim nV(XJ = cr1 < oo, M—»co

where Xn = (1 /n) 2/Li ^- Then, when n —> oo, we have:

1.7

Statistical Inference

1.7.1

Point Estimation

Suppose that the functional form of the underlying distribution on population is known but the parameter 0 included in the distribution is not known. The distribution function of population is given by f ( x ; 0). Let x\,x2,---,xn be the n observed data drawn from the population distribution. Consider estimating the parameter 6 using the n observed data. Let 9n(x\, x2, • • •, xn) be a function of the observed data x\, x2, • • •, xn. Suppose that 6,,(xi, X2, • • •, xn) is constructed from the purpose of estimating the parameter 0. dn(xi, x^, • • -, xn) takes a certain value given the n observed data. Then, On(x\, x^, • • •, xn) is called the point estimate of 0, or simply the estimate of 0. Example 1.11: Consider the case of 0 = (ji,cr), where the unknown parameters contained in population is given by mean and variance. A point estimate of population mean IJL is given by: 1 "

//„(•*!> *2, • • - , * « ) = x = - V x,. A point estimate of population variance cr2 is:

1 aj,(xi, x-i, • • • , xn) = s2 = —

"^

"

(xt - x)2.

36

CHAPTER 1. ELEMENTS OF STATISTICS

An alternative point estimate of population variance cr2 is: xi - I)2.

1.7.2

Statistic, Estimate and Estimator

The underlying distribution of population is assumed to be known, but the parameter 9, which characterizes the underlying distribution, is unknown. The probability density function of population is given by f ( x ; 6). Let X i; X2, • • -, Xn be a subset of population, which are regarded as the random variables and are assumed to be mutually independent. x\, x-i, •• •, xn are taken as the experimental values of the random variables X], X2, • • -, Xn. In statistics, we consider that n-variate random variables X\, X2, • • -, Xn takes the experimental values x\, x2, • • •, xn by chance. There, the experimental values and the actually observed data series are used in the same meaning. As discussed in Section 1.7.1, 9n(X], x2, • • •, x,,) denotes the point estimate of 0. In the case where the observed data x\, x2, • • • , xa are replaced by the corresponding random variables Xi, X2, • • -, Xn, a function of X}, X2, • • -, Xn, i.e., 6(X\, X2, • • •, Xn), is called the estimator of 9, which should be distinguished from the estimate of 9, i.e., G(Xi,X2,--;Xn).

Example 1.12: Let X] , X2, • • -, X,, denote a random sample of n from a given distribution f ( x ; 9). Consider the case of 9 = (//, a2). The estimator of /j is given by X = (l/n)£"_, ^i> while the estimate of ju is x = ( I /«) Et i XL The estimator of cr2 is S 2 = £"=1 (Xi - X) 2 /(n - 1 ) and the estimate of There are numerous estimators and estimates of 9. All of (1 In) £"=) X;, (X\ +Xn)/2, median of (X\, X2, • • -, Xn) and so on are taken as the estimators of//. Of course, they are called the estimates of 9 when X, is replaced by ;c; for all ;'. Similarly, both S2 = E"=,(X; - X)2/(« - 1) and S'2 = Z2=1(X; - X)2/n are the estimators of tr2. We need to choose one out of the numerous estimators of 6. The problem of choosing an optimal estimator out of the numerous estimators is discussed in Sections 1 .7.4 and 1.7.5. In addition, note as follows. A function of random variables is called a statistic. The statistic for estimation of the parameter is called an estimator. Therefore, an estimator is a family of a statistic.

1.7.3

Estimation of Mean and Variance

Suppose that the population distribution is given by f(x;9). The random sample X\, X2, • • •, X,, are assumed to be drawn from the population distribution f(x\6\ where 9 = (n, cr2). Therefore, we can assume that X! , X2, • • •, Xn are mutually independently

1.7. STATISTICAL INFERENCE

37

and identically distributed, where "identically" implies E(X,) = // and V(X,) = a2 for all i. Consider the estimators of 6 - (/u, e)

42

CHAPTER!. ELEMENTS OF STATISTICS

for a random variable X. Here, replacing X by X, we obtain E(X) and V(X) as follows: E(X)=//,

V(X) = ^-,

because E(X/) = ju and V(X,) = cr2 < co are assumed for all i. Then, when « —> oo, we obtain the following result:

which implies that X —> ^ as n —> oo. Therefore, we can conclude that X is a consistent estimator of ju. Summarizing the results up to now, X is an unbiased, minimum variance and consistent estimator of population mean p.. When the distribution of X, is assumed to be normal for all i, X leads to an unbiased, efficient and consistent estimator of p. Example 1.16:

Suppose that X \ , X 2 , - - - , X t t are mutually independently, identically 1 " andnormally distributed with mean u and variance cr2. Consider 5"*2 = - > (X-X)2, n & which is an estimate of a2. In Remark on p. 32, X and p are replaced by S"2 and cr2. Then, we obtain the following inequality:

We compute E((S"2 - cr2)2). Since (« - l)5 2 /cr 2 ~ ^2(n - 1), we obtain E((« 1 " — l)5 2 /cr 2 ) = n- 1 and V((n- l)5 2 /cr 2 ) = 2(n- 1), where S2 = - V(X ; -X) 2 . See i=\ Section 2.2.8 (p. 106) for the chi-square distribution^-2^ - 1). Therefore, E(52) = a2 and V(5 2 ) = 2cr4/(« - 1) can be derived. Using S**2 = S2(n - \)/n, we have the following: E(( oo, we obtain: e 2

2

2

Because S** —> cr , S** is a consistent estimator of cr2. Thus, S**2 is not unbiased (see Section 1 .7.3, p.38), but is is consistent.

1.7. STATISTICAL INFERENCE

1.7.5

43

Maximum Likelihood Estimator

In Section 1.7.4, the properties of the estimators X and S2 are discussed. It is shown that X is an unbiased, efficient and consistent estimator of// under normality assumption and that S2 is an unbiased estimator of cr2. Note that S2 is not efficient but consistent (we do not check these features of S2 in this book). The population parameter 6 depends on a functional form of the population distribution f(x\ &). It corresponds to (//, cr2) in the case of the normal distribution and/? in the case of the exponential distribution (Section 2.2.4). Now, in more general cases, we want to consider how to estimate 6. The maximum likelihood estimator gives us one of the solutions. Let A"i, X2, • • •, Xn be mutually independently and identically distributed random samples. X, has the probability density function f(x; ff). Under these assumptions, the joint density function o f X \ , X2, • • •, X,, is given by:

where 9 denotes the unknown parameter. Given the actually observed data x^ , X 2 , - - •, xn, the joint density f(x\ ,x2,-- •, xn; 6) is regarded as a function of 8, i.e.,

1(6) = 1(9; x) = 1(9; *,, Jt2, • • • , *„) 1(9) is called the likelihood function. Let 9n be the 9 which maximizes the likelihood function. Replacing x\, x2, • • •, xn by X,, X2, • • •, Xn, 9n = 6n(X\, X2, • • -, X,,) is called the maximum likelihood estimator, while 9n(x\, x2, • • -, xn) is called the maximum likelihood estimate. That is, solving the following equation: 99

,

the maximum likelihood estimator 9n = 9n(X\ , X 2 , - - - , X,,) is obtained. Example 1.17: Suppose that X 1? X2, • • -, Xn are mutually independently, identically and normally distributed with mean // and variance 0". Note that 0,, depends on X\, X2, • • -, X,, as well as d, i.e., 9n = 0n(X], X2, • • -, Xn;0). Now we assume that we can solve (1.19) with respect to 9, which is rewritten as follows: P(0L(X} , X2, • • • , X,,; 0*) < 0 < 8u(Xi , X2, • • • , X,,; 0")) = 1 - «-

(1-20)

( 1 .20) implies that 0 lies on the interval (0L(X, , X2, • • •, XB; 0*), 0£/(X, , X2, • • -, X«; 0**)) with probability 1 - a. Depending on a functional form of 0n(X], X2, • • •, Xn;0), we possibly have the situation that 0* and 0" are switched with each other. Now, we replace the random variables Xi, X2, • • •, X,, by the experimental values x\ , x2, • • -, xn. Then, we say that the interval: (6\(*, , x-i, • • • , xn; 0*), BU(XI ,x2,---,xn; 0*'))

is called the 100 x (1 - a)% confidence interval of 0. Thus, estimating the interval is known as the interval estimation, which is distinguished from the point estimation. In the interval, OL(X\ , x2, • • -, xn; 0*) is known as the lower bound of the confidence interval, while BU(X\, x2, • • •, xn; 0**) is the upper bound of the confidence interval. Given probability a, the 0,.(X, , X2, • • •, X,,; 0*) and BV(X\ , X 2 , - - - , X n ; 9") which satisfies equation (1.20) are not unique. For estimation of the unknown parameter 0, it is more optimal to minimize the width of the confidence interval. Therefore, we should choose 0* and 0** which minimizes the width 0j/(Xi, X2, • • -, Xn; 0**) - 0/,(Xi, X2, •••,X, 1 ;0*). Interval Estimation of X: Let X l5 X2, • • •, Xn be mutually independently and identically distributed random variables. X, has a distribution with mean p. and variance o"1. From the central limit theorem,

Replacing a2 by its estimator S2 (or S*

Therefore, when n is large enough,

where z* and z" (zf < z") are percent points from the standard normal density function. Solving the inequality above with respect to fi, the following expression is obtained.

48

CHAPTER]. ELEMENTS OF STATISTICS

where BL and Qv correspond to X - z"S/ V« ar>d X - z*S/ V«, respectively. The length of the confidence interval is given by:

which should be minimized subject to:

f

f(x) dx=\-a,

i.e.,

F(z")-F(z')=\-a, where F(-) denotes the standard normal cumulative distribution function. Solving the minimization problem above, we can obtain the conditions that f(z") = /(z**) for z* < z" and that f(x) is symmetric. Therefore, we have:

where za/i denotes the 100 X a/2 percent point from the standard normal density function. _ Accordingly, replacing the estimators X and S2 by their estimates "x and s2, the 100 x (1 - a)% confidence interval of// is approximately represented as: /_

.v

(X — Zall——, ^ V«

_

s \

X+Za/2—-p), Vn

for large n. For now, we do not impose any assumptions on the distribution of Xt. If we assume that Xi is normal, ^(X - //)/S has a t distribution with n - 1 degrees of freedom for any n. Therefore, 100 x (1 - a)% confidence interval of// is given by: (x - ta,i(n - 1)—:, x + ta/2(n - 1)—), where ta/2(n - 1) denotes the 100 X a/2 percent point of the t distribution with n - 1 degrees of freedom. See Section 2.2.10, p. 115 for the t distribution. Interval Estimation of f)n: Let X\, X2, • • •, X,, be mutually independently and identically distributed random variables. X, has the probability density function f(xc,6). Suppose that 9,, represents the maximum likelihood estimator of 0. From (1.17), we can approximate the 100 x (1 - a)% confidence interval of 0 as follows:

1.8. TESTING HYPOTHESIS

49

Table 1.1: Type I and Type II Errors

Acceptance of HO Rejection of H0

HQ is true. Correct judgment Type I Error (Probability a = Significance Level)

//o is false. Type II Error (Probability/?) Correct judgment (1 -£ = Power)

1.8 Testing Hypothesis 1.8.1

Basic Concepts in Testing Hypothesis

Given the population distribution f ( x \ 0), we want to judge from the observed values x\, x2, • • •, xn whether the hypothesis on the parameter 0, e.g. 9 = 60, is correct or not. The hypothesis that we want to test is called the null hypothesis, which is denoted by HO : 6 = 60. The hypothesis against the null hypothesis, e.g. 0 2 00, is called the alternative hypothesis, which is denoted by HI : 62 BQ. Type I and Type II Errors: When we test the null hypothesis HQ, as shown in Table 1.1 we have four cases, i.e., (i) we accept HQ when HO is true, (ii) we reject HO when H0 is true, (iii) we accept HO when HO is false, and (iv) we reject H0 when HO is false, (i) and (iv) are correct judgments, while (ii) and (iii) are not correct, (ii) is called a type I error and (iii) is called a type II error. The probability which a type I error occurs is called the significance level, which is denoted by a, and the probability of committing a type II error is denoted by /3. Probability of (iv) is called the power or the power function, because it is a function of the parameter 0. Testing Procedures: The testing procedure is summarized as follows. 1. Construct the null hypothesis (H0) on the parameter. 2. Consider an appropriate statistic, which is called a test statistic. Derive a distribution function of the test statistic when H0 is true. 3. From the observed data, compute the observed value of the test statistic. 4. Compare the distribution and the observed value of the test statistic. When the observed value of the test statistic is in the tails of the distribution, we consider that H0 is not likely to occur and we reject H0. The region that HO is unlikely to occur and accordingly HQ is rejected is called the rejection region or the critical region, denoted by R. Conversely, the region that

50

CHAPTER 1. ELEMENTS OF STATISTICS

//o is likely to occur and accordingly H0 is accepted is called the acceptance region, denoted by A. Using the rejection region R and the acceptance region A, the type I and II errors and the power are formulated as follows. Suppose that the test statistic is give by T = T(X-[,Xi, • • • , X n ) . The probability of committing a type I error, i.e., the significance level a, is given by: P(T(Xi , X 2 , - - - , X n ) £ R\H0 is true) = a, which is the probability that rejects H0 when HO is true. Conventionally, the significance level a = 0.1, 0.05, 0.01 is chosen in practice. The probability of committing a type II error, i.e., /?, is represented as: P(T(X} , X2, • • • , X,,) e A\H0 is not true) = ft, which corresponds to the probability that accepts H0 when HO is not true. The power is defined as 1 —ft, i.e., P(7XX, , X 2 , - - - , X n ) e R\H0 is not true) = \-ft, which is the probability that rejects H0 when H0 is not true.

1.8.2

Power Function

Let X-[, X2, • • •, X,, be mutually independently, identically and normally distributed with mean fj. and variance a2. Assume that a2 is known. In Figure 1.3, we consider the hypothesis on the population mean /j, i.e., the null hypothesis H0 : n = jufl against the alternative hypothesis H\ : p. = ni, where //i > //0 is taken. The dark shadow area corresponds to the probability of committing a type I error, i.e., the significance level, while the light shadow area indicates the probability of committing a type II error. The probability of the right-hand side of /* in the distribution under H-\ represents the power of the test, i.e., 1 -ft. In the case of normal population, the distribution of sample mean X is given by:

For the distribution of X, see the moment-generating function of X in Theorem on p.29. By normalization, we have:

a-

«

Therefore, under the null hypothesis H0 : jj. = ju fl . we obtain:

—9^ ~ MO, i), a-1 y«

1.8. TESTING

51

HYPOTHESIS Figure 1.3: Type I Error (a) and Type II Error (J3) Asa is small, i.e., as /* goes to right, fj becomes large.

The distribution under the null hypothesis (Ho)

The distribution under the alternative hypothesis (Hi)

Acceptance region

Rejection region

where n is replaced by po- Since the significance level a is the probability which rejects //o when H0 is true, it is given by:

a=

-p, V" where Za denotes 1 00 x a percent point of the standard normal density function. Therefore, the rejection region is given by: X > //0 + zacr/ V«Since the power 1 - ft is the probability which rejects H0 when H} is true, it is given by: ^a~l -yn

-vs -4

where F(-) represents the standard normal cumulative distribution function, which is given by F(x) = J^(27rr' /2 exp(-jf 2 ) dr. The power function is a function of//], given /(o and a.

1.8.3

Testing Hypothesis on Population Mean

Let Xi, X?, • • •, Xn be mutually independently, identically and normally distributed with mean // and variance o~2. Assume that o~2 is known. Consider testing the null hypothesis H0 : p. = JIQ. When the null hypothesis H0 is true, the distribution of X is given by:

52

CHAPTER!. ELEMENTS OF STATISTICS

Therefore, the test statistic is given by: -*Jn(X -/J.Q)/ xlO\ we reject the null hypothesis Hn : p. = ^0 at the significance level a. Note that^(l) denotes the 100 x a percent point from the chi-square distribution with one degree of freedom. Example 1.20: Suppose that X], X2, • • •, Xn are mutually independently, identically and normally distributed with mean zero and variance cr2. The normal probability density function with mean p and variance a1 is given by: f(x;n,cr2) =

2;ro-2

By the likelihood ratio test, we want to test the null hypothesis H0 : n= HQ against the alternative hypothesis H\ : /j ± }i0. The likelihood ratio is given by:

1.8.

TESTING HYPOTHESIS

57

where