Probability for Statisticians

Preface There is a thin self-contained textbook within this larger presentation. To be sure that this is well understoo...

Author: Galen R. Shorack

114 downloads 1473 Views 3MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Preface There is a thin self-contained textbook within this larger presentation. To be sure that this is well understood, I will describe later how I have used this text in the classroom at the University of Washington in Seattle. Let me first indicate what is different about this book. As implied by the title, there is a difference. Not all the difference is based on inclusion of statistical material. (To begin, Chapters 1–6, provide the mathematical foundation for the rest of the text. Then Chapters 7–8 hone some tools geared to probability theory, while Chapter 9 provides a brief introduction to elementary probability theory right before the main emphasis of the presentation begins.) The classical weak law of large numbers (WLLN) and strong law of large numbers (SLLN) as presented in Sections 10.2–10.4 are particularly complete, and they also emphasize the important role played by the behavior of the maximal summand. Presentation of good inequalities is emphasized in the entire text, and this chapter is a good example. Also, there is an (optional) extension of the WLLN in Section 10.6 that focuses on the behavior of the sample variance, even in very general situations. Both the classical central limit theorem (CLT) and its Lindeberg and Liapunov generalizations are presented in two different chapters. They are first presented in Chapter 11 via Stein’s method (with a new twist), and they are again presented in Chapter 14 using the characteristic function (chf) methods introduced in Chapter 13. The CLT proofs given in Chapter 11 are highly efficient. Conditions for both the weak bootstrap and the strong bootstrap are developed in Chapter 11, as is a universal bootstrap CLT based on light trimming of the sample. The approach emphasizes a statistical perspective. Much of Section 11.1 and most of Sections 11.2–11.5 are quite unusual. I particularly like this chapter. Stein’s method is also used in the treatment of U-statistics and Hoeffding’s combinatorial CLT (which applies to sampling from a finite population) in the optional Chapter 17. Also, the chf proofs in Section 14.2 have a slightly unusual starting point, and the approach to gamma approximations in the CLT in Section 14.4 is new. Both distribution functions (dfs F (·)) and quantile functions (qfs K(·) ≡ F−1 (·)) are emphasized throughout (quantile functions are important to statisticians). In Chapter 7 much general information about both dfs and qfs and the Winsorized variance is developed. The text includes presentations showing how to exploit the inverse transformation X ≡ K(ξ) with ξ ∼ = Uniform(0, 1). In particular, Chapter 7 inequalities relating the qf and the Winsorized variance to some empirical process results of Chapter 12 are used in Chapter 16 to treat trimmed means and L-statistics, rank and permutation tests, sampling from finite populations, and bootstrapping. (Though I am very fond of Sections 7.6–7.11, their prominence is minimized in the subsequent parts of the text.) At various points in the text choices can be voluntarily made that will offer the opportunity for a statistical example or foray. (Even if the instructor does not exercise a particular choice, a student can do so individually.) After the elementary introduction to probability theory in Section 9.1, many of the classical distributions

viii

PROBABILITY FOR STATISTICIANS

of statististics are introduced in Section 9.2, while useful linear algebra and the multivariate normal distribution are the subjects of Section 9.3 and Section 9.4. Following the CLT via Stein’s method in Section 11.1, extensions in Section 11.2– 11.3, and application of these CLTs to the bootstrap in Sections 11.4–11.5, there is a large collection of statistical examples in Section 11.6. During presentation of the CLT via chfs in Chapter 14, statistical examples appear in Sections 14.1, 14.2, and 14.4. Statistical applications based on the empirical df appear in Sections 12.10 and 12.12. The highly statistical optional Chapters 16 and 17 were discussed briefly above. Also, the conditional probability Sections 8.5 and 8.6 emphasize statistics. Maximum likelihood ideas are presented in Section A.2 of Appendix A. Many useful statistical distributions contain parameters as an argument of the gamma function. For this reason, the gamma and digamma functions are first developed in Section A.1. Section A.3 develops cumulants, Fisher information, and other useful facts for a number of these distributions. Maximum likelihood proofs are in Section A.4. It is my hope that even those well versed in probability theory will find some new things of interest. I have learned much through my association with David Mason, and I would like to acknowledge that here. Especially (in the context of this text), Theorem 12.4.3 is a beautiful improvement on Theorem 12.4.2, in that it still has the potential for necessary and sufficient results. I really admire the work of Mason and his colleagues. It was while working with David that some of my present interests developed. In particular, a useful companion to Theorem 12.4.3 is knowledge of quantile functions. Sections 7.6–7.11 present what I have compiled and produced on that topic while working on various applications, partially with David. Jon Wellner has taught from several versions of this text. In particular, he typed an earlier version and thus gave me a major critical boost. That head start is what turned my thoughts to writing a text for publication. Sections 8.6, 19.2, and the Hoffman–Jorgensen inequalities came from him. He has also formulated a number of exercises, suggested various improvements, offered good suggestions and references regarding predictable processes, and pointed out some difficulties. My thanks to Jon for all of these contributions. (Obviously, whatever problems may remain lie with me.) My thanks go to John Kimmel for his interest in this text, and for his help and guidance through the various steps and decisions. Thanks also to Lesley Poliner, David Kramer, and the rest at Springer-Verlag. It was a very pleasant experience. This is intended as a textbook, not as a research manuscript. Accordingly, the main body is lightly referenced. There is a section at the end that contains some discussion of the literature.

Contents Use of this Text xiii Definition of Symbols

xviii

Chapter 1. Measures 1. Basic Properties of Measures 1 2. Construction and Extension of Measures 3. Lebesgue–Stieltjes Measures 18

12

Chapter 2. Measurable Functions and Convergence 1. 2. 3. 4. 5.

Mappings and σ-Fields 21 Measurable Functions 24 Convergence 29 Probability, RVs, and Convergence in Law Discussion of Sub σ-Fields 35

33

Chapter 3. Integration 1. 2. 3. 4. 5.

The Lebesgue Integral 37 Fundamental Properties of Integrals 40 Evaluating and Differentiating Integrals 44 Inequalities 46 Modes of Convergence 51

Chapter 4. Derivatives via Signed Measures 1. 2. 3. 4.

Decomposition of Signed Measures 61 The Radon–Nikodym Theorem 66 Lebesgue’s Theorem 70 The Fundamental Theorem of Calculus 74

Chapter 5. Measures and Processes on Products 1. 2. 3. 4.

Finite-Dimensional Product Spaces 79 Random Vectors on (Ω, A, P ) 84 Countably Infinite Product Probability Spaces Random Elements and Processes on (Ω, A, P )

86 90

Chapter 6. General Topology and Hilbert Space 1. General Topology 95 2. Metric Spaces 101 3. Hilbert Space 104

x


Chapter 7. Distribution and Quantile Functions 1. Character of Distribution Functions 107 2. Properties of Distribution Functions 110 3. The Quantile Transformation 111 4. Integration by Parts Applied to Moments 115 5. Important Statistical Quantities 119 6. Infinite Variance 123 7. Slowly Varying Partial Variance 127 8. Specific Tail Relationships 134 9. Regularly Varying Functions 137 10. Some Winsorized Variance Comparisons 140 11. Inequalities for Winsorized Quantile Functions

147

Chapter 8. Independence and Conditional Distributions 1. 2. 3. 4. 5. 6.

Independence 151 The Tail σ-Field 155 Uncorrelated Random Variables 157 Basic Properties of Conditional Expectation 158 Regular Conditional Probability 168 Conditional Expectations as Projections 174

Chapter 9. Special Distributions 1. 2. 3. 4.

Elementary Probability 179 Distribution Theory for Statistics 187 Linear Algebra Applications 191 The Multivariate Normal Distribution 199

Chapter 10. WLLN, SLLN, LIL, and Series 0. Introduction 203 1. Borel–Cantelli and Kronecker Lemmas 204 2. Truncation, WLLN, and Review of Inequalities 206 3. Maximal Inequalities and Symmetrization 210 4. The Classical Laws of Large Numbers, LLNs 215 5. Applications of the Laws of Large Numbers 223 6. General Moment Estimation 226 7. Law of the Iterated Logarithm 235 8. Strong Markov Property for Sums of IID RVs 239 9. Convergence of Series of Independent RVs 241 10. Martingales 246 11. Maximal Inequalities, Some with Boundaries 247 12. A Uniform SLLN 252

CONTENTS

xi

Chapter 11. Convergence in Distribution 1. 2. 3. 4. 5. 6. 7. 8.

Stein’s Method for CLTs 255 ˜ ˇ Winsorization and Truncation 264 Identically Distributed RVs 269 Bootstrapping 274 Bootstrapping with Slowly Increasing Trimming Examples of Limiting Distributions 279 Classical Convergence in Distribution 288 Limit Determining Classes of Functions 292

276

Chapter 12. Brownian Motion and Empirical Processes 1. Special Spaces 295 2. Existence of Processes on (C, C) and (D, D) 298 3. Brownian Motion and Brownian Bridge 302 4. Stopping Times 305 5. Strong Markov Property 308 6. Embedding a RV in Brownian Motion 311 7. Barrier Crossing Probabilities 314 8. Embedding the Partial Sum Process 318 9. Other Properties of Brownian Motion 323 10. Various Empirical Processes 325 11. Inequalities for the Various Empirical Processes 333 12. Applications 338

Chapter 13. Characteristic Functions 1. 2. 3. 4. 5. 6. 7.

Basic Results, with Derivation of Common Chfs 341 Uniqueness and Inversion 346 The Continuity Theorem 350 Elementary Complex and Fourier Analysis 352 Esseen’s Lemma 358 Distributions on Grids 361 Conditions for φ to Be a Characteristic Function 363

Chapter 14. CLTs via Characteristic Functions 0. 1. 2. 3. 4. 5. 6.

Introduction 365 Basic Limit Theorems 366 Variations on the Classical CLT 371 Local Limit Theorems 380 Gamma Approximation 383 Edgeworth Expansions 390 ¯n ) Approximating the Distribution of h(X

396

xii


Chapter 15. Infinitely Divisible and Stable Distributions 1. 2. 3. 4.

Infinitely Divisible Distributions 399 Stable Distributions 407 Characterizing Stable Laws 410 The Domain of Attraction of a Stable Law

412

Chapter 16. Asymptotics via Empirical Proceses 0. 1. 2. 3. 4.

Introduction 415 Trimmed and Winsorized Means 416 Linear Rank Statistics and Finite Sampling The Bootstrap 432 L-Statistics 437

426

Chapter 17. Asymptotics via Stein’s Approach 1. U-Statistics 449 2. Hoeffding’s Combinatorial CLT

458

Chapter 18. Martingales 1. Basic Technicalities for Martingales 467 2. Simple Optional Sampling Theorem 472 3. The Submartingale Convergence Theorem 473 4. Applications of the S-mg Convergence Theorem 481 5. Decomposition of a Submartingale Sequence 487 6. Optional Sampling 492 7. Applications of Optional Sampling 499 8. Introduction to Counting Process Martingales 501 9. Doob–Meyer SubmartingaleDecomposition 511 · 10. Predictable Processes and 0 H dM Martingales 516 11. The Basic Censored Data Martingale 522 12. CLTs for Dependent RVs 529

Chapter 19. Convergence in Law on Metric Spaces 1. Convergence in Distribution on Metric Spaces 2. Metrics for Convergence in Distribution 540

531

Appendix A. Distribution Summaries 1. 2. 3. 4.

The Gamma and Digamma Functions 546 Maximum Likelihood Estimators and Moments Examples of Statistical Models 555 Asymptotics of Maximum Likelihood Estimation

References Index 575

568

551 563

Use of this Text The University of Washington is on the quarter system, so my description will reflect that. My thoughts are offered as a potential help, not as an essential recipe. The reader will note that the problems are interspersed with the text. It is important to read them as they are encountered. Chapters 1–5 provide the measure-theoretic background that is necessary for the rest of the text. Many of our students have had at least some kind of an undergraduate exposure to part of this subject. Still, it is important that I present the key parts of this material rather carefully. I feel it is useful for all of them. Chapter 1 (measures; 5 lectures) Emphasized in my presentation are generators, the monotone property of measures, the Carathéodory extension theorem, completions, the approximation lemma, and the correspondence theorem. Presenting the correspondence theorem carefully is important, as this allows one the luxury of merely highlighting some proofs in Chapter 5. [The minimal monotone class theorem of Section 1.1, claim 8 of the Carathédory extension theorem proof, and most of what follows the approximation lemma in Section 1.2 would never be presented in my lectures.] {I always assign Exercises 1.1.1 (generators), 1.2.1 (completions), and 1.2.3 (the approximation lemma). Other exercises are assigned, but they vary each time.} Chapter 2 (measurable functions and convergence; 4 lectures) I present most of Sections 2.1, 2.2, and 2.3. Highlights are preservation of σ-fields, measurability of both common functions and limits of simple functions, induced measures, convergence and divergence sets (especially), and relating →µ to →a.s (especially, reducing the first to the second by going to subsequences). I then assign Section 2.4 as outside reading and Section 2.5 for exploring. [I never lecture on either Section 2.4 or 2.5.] {I always assign Exercises 2.2.1 (specific σ-fields), 2.3.1 (concerning →a.e. ), 2.3.3 (a substantial proof), and 2.4.1 (Slutsky’s theorem).} Chapter 3 (integration; 7 lectures) This is an important chapter. I present all of Sections 3.1 and 3.2 carefully, but older, Section 3.3 is left as reading, and some of the Section 3.4 inequalities (Cr , H¨ Liapunov, Markov, and Jensen) are done carefully. I do Section 3.5 carefully as far as Vitali’s theorem, and then assign the rest as outside reading. {I always assign Exercises 3.2.1–3.2.2 (only the zero function), 3.3.3 (differentiating under the integral sign), 3.5.1 (substantial theory), and 3.5.7 (the Scheffé theorem).} Chapter 4 (Radon–Nikodym; 2 lectures) I present ideas from Section 4.1, sketch the Jordan–Hahn decomposition proof, and then give the proofs of the Lebesgue decomposition, the Radon–Nikodym theorem, and the change of variable theorem. These final two topics are highlighted. The fundamental theorem of calculus of Section 4.4 is briefly discussed. [I would never present any of Section 4.3.] {I always assign Exercises 4.2.1 (manipulating Radon– Nikodym derivatives), 4.2.7 (mathematically substantial), and 4.4.1, 4.4.2, and 4.4.4 (so that the students must do some outside reading in Section 4.4 on their own).}

xiv


Chapter 5 (Fubini; 2 lectures) The first lecture covers Sections 5.1 and 5.2. Proving Proposition 5.2.1 is a must, and I discuss/prove Theorems 5.1.2 (product measure) and 5.1.3 (Fubini). The remaining time is spent on Section 5.3. [I rarely lecture from Section 5.4, but I do assign it as outside reading.] {I always assign Exercises 5.3.1 (measurability in a countable number of dimensions) and 5.4.1 (the finite-dimensional field).} Chapter 6 (topology and Hilbert space, 0 lectures) [This chapter is presented only for reference. I do not lecture from it.] The mathematical tools have now been developed. In the next three chapters we learn about some specialized probabilistic tools and then get a brief review of elementary probability. The presentation on the classic topics of probability theory then commences in Chapter 10. Chapter 7 (distribution functions (dfs) and quantile functions (qfs); 4 lectures) This chapter is quite important to this text. Skorokhod’s theorem in Section 7.3 must be done carefully, and the rest of Sections 7.1–7.4 should be covered. Section 7.5 should be left as outside reading. [Lecturing from Sections 7.6–7.11 is purely optional, and I would not exceed one lecture.] {I always assign Exercises 7.1.1 (on continuity of dfs), 7.3.3 (F −1 (·) is left continuous), 7.3.3 (change of variable), and 7.4.2 (for practice working with X ≡ K(ξ)). Consider lecturing on Theorem 7.6.1 (the infinite variance case).} Chapter 8 (conditional expectation; 2 lectures) The first lecture covers Sections 8.1 and 8.2. It highlights Proposition 8.1.1 (on the preservation of independence), Theorem 8.1.2 (extending independence from π-systems), and Kolmogorov’s 0-1 law. The other provides some discussion of the definition of conditional probability in Section 8.4, includes proofs of several parts of Theorem 8.4.1 (properties of conditional expectation), and discusses Definition 8.5.1 of regular conditional probability. [I never lecture on Sections 8.3, 8.5, or 8.6.] {I always assign Exercises 8.1.2 and 8.1.3 (they provide routine practice with the concepts), Exercise 8.4.1 (discrete conditional probability), Exercise 8.4.3 (repeated stepwise smoothing in a particular example), and part of Exercise 8.4.4 (proving additional parts of Theorem 8.4.1).} Chapter 9 (elementary probability; 0 lectures) Sections 9.1 and 9.2 were written to provide background reading for those graduate students in mathematics who lack an elementary probability background. Sections 9.3 and 9.4 allow graduate students in statistics to read some of the basic multivariate results in appropriate matrix notation. [I do not lecture from this chapter.] {But I do assign Exercises 9.1.8 (the Poisson process exists) and 9.2.1(ii) (so that the convolution formula is refreshed).} Chapter 10 (laws of large numbers (LLNs) and inequalities; 3 lectures for now) Since we are on the quarter system at the University of Washington, this leaves me 3 lectures to spend on the law of large numbers in Chapter 10 before the Christmas break at the end of the autumn quarter. In the first 3 lectures I do Sections 10.1 and 10.2 with Khinchine’s weak law of large numbers (WLLN), Kolmogorov’s inequality only from Section 10.3, and at this time I present Kolmogorov’s strong law of large numbers (SLLN) only from Section 10.4. {I always assign Exercises 10.1.1 (Cesàro summability), 10.2.1 (it generates good ideas related to the proofs), 10.2.3 (as it

USE OF THIS BOOK

xv

practices the important Op (·) and op (·) notation), 10.4.4 (the substantial result of Marcinkiewicz and Zygmund), 10.4.7 (random sample size), and at least one of the alternative SLLN proofs contained in 10.4.8, 10.4.9, and 10.4.10.} At this point at the beginning of the winter quarter the instructor will have his/her own opinions about what to cover. I devote the winter quarter to the weak law of large numbers (WLLN), an introduction to the law of the iterated logarithm (LIL), and various central limit theorems (CLTs). That is, the second term treats the material of Chapters 10–11 and 13–17. I will outline my choices for which parts to cover. Chapter 10 (LLNs, inequalities, LIL, and series; 6 lectures) My lectures cover Section 10.3 (symmetrization inequalities and Lévy’s inequality for the WLLN, and the Ottovani–Skorokhod inequality for series), Feller’s WLLN from Section 10.4, the Glivenko–Cantelli theorem from Section 10.5, the LIL for normal rvs in Proposition 10.7.1, the strong Markov property of Theorem 10.8.1, and the two series Theorem 10.9.2. [I do not lecture from any of Sections 10.6, 10.10, 10.11, or 10.12 at this time.] {I always assign Exercise 10.7.1 (Mills’ ratio).} Chapter 11 (CLTs via Stein’s method; 3 lectures) From section 11.1 one can prove Stein’s first lemma and discuss his second lemma, prove the Berry–Esseen theorem, and prove Lindeberg’s CLT. Note that we have not yet introduced characteristic functions. Chapter 13 (characteristic functions (chfs); 6 lectures) I do sections 13.1–13.5. {I always assign Exercises 13.1.1 and 13.1.3(a) (deriving specific chfs) and 13.4.1 (Taylor series expansions of the chf).} Chapter 14 (CLTs via chfs; 6 lectures) The classical CLT, the Poisson limit theorem, and the multivariate CLT make a nice lecture. The chisquare goodness of fit example and/or the median example (of Section 11.6) make a lecture of illustrations. Chf proofs of the usual CLTs are given in Section 14.2 (Section 13.5 could have been left until now). If Lindeberg’s theorem was proved in Chapter 11, one might do only Feller’s converse now via chfs. Other examples from either Section 14.2 or 11.6 could now be chosen, and Example 11.6.4 (weighted sums of iid rvs) is my first choice. [The chisquare goodness of fit example could motivate a student to read from Sections 9.3 and 9.4.] At this stage I still have at least 7 optional lectures at the end of the winter quarter and about 12 more at the start of the spring quarter. In my final 16 lectures of the spring quarter I feel it appropriate to consider Brownian motion in Chapter 12 and then martingales in Chapter 18 (in a fashion to be described below). Let me first describe some possibilities for the optional lectures, assuming that the above core was covered. Chapter 17 (U-statistics and Hoeffding’s combinatorial CLT) Sections 17.1 and 17.2 are independent of each other. The Berry–Esseen potential of Lemma 11.1.1 is required for Section 17.1. Either one or two lectures could then be presented on U-statistics from Section 17.1. The alternative Stein formulation of Motivation 11.1.1 is required for section 17.2. Two additional lectures would give the Hoeffding combinatorial CLT and its corollary regarding sampling from finite populations.

xvi


Chapter 11 (statistical examples) Sections 11.6, 14.2, and 14.6 contain appropriate examples and exercises. Chapter 11 (bootstrap) Both Sections 11.4 and 11.5 on the bootstrap require only Theorem 11.2.1. Chapters 11 and 19 (convergence in distribution) Convergence in distribution on the line is presented in Sections 11.7 and 11.8. [This is extended to metric spaces in Chapter 19, but I do not lecture from it.] Chapter 11 (domain of normal attraction of the normal df) The converse of the CLT in Theorem 11.3.2 requires the Giné–Zinn symmetrization inequality and the Khinchine inequality of Section 13.3 and the Paley–Zygmund inequality of Section 3.4. Chapters 7, 10 and 11 (domain of attraction of the normal df) Combining Sections 7.6–7.8, the Section 10.3 subsection on maximal inequalities of another ilk, Section 10.6, and Sections 11.2–11.3 makes a nice unit. Lévy’s asymptotic normality condition (ANC) of (7.7.14) for a rv X has some prominence. In Chapter 7 purely geometric methods plus Cauchy–Schwarz are used to derive a multitude of equivalent conditions. In the process, quantile functions are carefully studied. In Section 10.6 the ANC is seen to be equivalent to a result akin to a WLLN for the rv X 2 , and so in this context many additional equivalent conditions are again derived. Thus when one comes to the CLT in Sections 11.2 and 11.3, one already knows a great deal about the ANC. Chapter 15 (infinitely divisible and stable laws) First, Section 15.1 (infinitely divisible laws) is independent of the rest, including Section 15.2 (stable laws). The theorem stated in Section 15.4 (domain of attraction of stable laws) would require methods of Section 7.9 to prove, but the interesting exercises are accessible without this. Chapter 14 (higher-order approximations) The local limit theorem in Section 14.3 can be done immediately for continuous dfs, but it also requires Section 13.6 for discrete dfs. The expansions given in Sections 14.4 (gamma approximation) and 14.5 (Edgeworth approximation) also require Exercise 13.4.6. Assorted topics suitable for individual reading These include Section 8.6 (on alternating conditional expectations), Section 10.12 (a uniform SLLN), Section 16.4 (L-statistics), Sections 18.8–18.11 (counting process martingales), and Section 18.12 (martingale CLTs). The primary topics for the spring quarter are Chapter 12 (Brownian motion and elementary empirical processes) and Chapter 18 (martingales). I have never covered Chapter 12 until the spring, but I placed it rather early in the text to make clear that it doesn’t depend on any of the later material. Chapter 12 (Brownian motion; 6 lectures) I discuss Section 12.1, sketch the proof of Section 12.2 and carefully apply that result in Section 12.3, and treat Section 12.4 carefully (as I believe that at some point a lecture should be devoted to a few of the more subtle difficulties regarding measurability). I am a bit cavalier regarding Section 12.5 (strong Markov property), but I apply it carefully in Sections 12.6, 12.7, and 12.8. I assign Section 12.9 as outside reading. [I do not lecture on Theorem 12.8.2.] {I always assign Exercises

USE OF THIS BOOK

xvii

12.1.2 (on (C, C)), 12.3.1 (various transforms of Brownian motion), 12.3.3 (integrals of normal processes), 12.4.1 (properties of stopping times), 12.7.3(a) (related to embedding a rv in Brownian motion), and 12.8.2 (the LIL via embedding).} At this point let me describe three additional optional topics that could now be pursued, based on the previous lectures from Chapter 12. Chapter 12 (elementary empirical processes) Uniform empirical and quantile processes are considered in Section 12.10. Straightforward applications to linear rank statistics and two-sample test of fit are included. One could either lecture from Section 12.12 (directly) or 12.11 (with a preliminary lecture from Sections 10.10–10.11, or leave these for assigned reading.) Chapter 16 (trimmed means and/or simple linear rank statistics) Both possibilities listed here require Section 12.10 as well as the quantile inequality of (7.11.3). Asymptotic normality of linear rank statistics and a finite sampling CLT are derived in Section 16.2, and the bootstrap is presented in Section 16.3. The general CLT (Theorem 16.1.1) and asymptotic normality of trimmed means (Theorem 16.1.2, but only present the β = 0 case) are derived in Section 16.1; this will also require stating/proving the equivalence of (16.1.3) and (7.6.4), which is shown in Theorem 7.1.1. Chapter 18 (martingales; 10 lectures) I cover most of the first seven sections. {I always assign Exercises 18.1.4 (a counting process martingale), 18.3.2 (a proof for continuous time mgs), 18.3.7, and 18.3.9 (on Lr -convergence).} Appendix A (maximum likelihood estimation) I see this as being of considerable interest in conjunction with statistical pursuits, rather than as a direct part of a course on probability theory.

Definition of Symbols ∼ = means “is distributed as” ≡ means “is defined to be” a = b ⊕ c means that |a − b| ≤ c Un =a Vn means “asymptotically equal” in the sense that Un − Vn →p 0 X∼ = (µ, σ2 ) means that X has mean µ and variance σ2 X∼ = F (µ, σ2 ) means that X has df F with mean µ and variance σ2 ¯ n is the “sample mean” and X ¨ n is the “sample median” X (Ω, A, µ) and (Ω, A, P ) denote a measure space and a probability space σ[C] denotes the σ-field generated by the class of sets C ¯ for the Borel sets B and B¯ ≡ σ[B, {+∞}, {−∞}] F(X) denotes X −1 (B), ξ will always refer to a Uniform(0, 1) rv means “nondecreasing” and ↑ means “strictly increasing” 1A (·) denotes the indicator function of the set A “df” refers to a distribution function F (·) “qf” refers to a quantile function K(·) ≡ F −1 (·) ˜ The “tilde” symbol denotes Winsorization ˇ The “h´ aˇcek” symbol denotes Truncation λ(·) and λn (·) will refer to Lebesgue measure on the line R and on Rn See page 119 for “dom(a, a )” Brownian motion S, Brownian bridge U, and the Poisson process N The empirical df Fn and the empirical df Gn of Uniform(0, 1) rvs → is associated with convergence in the LIL (see page 235) “mg” refers to a martingale “smg” refers to a submartingale > = means “≥” for a submartingale and “=” for a martingale > The symbol “=” is paired with “s-mg” in this context. Prominence Important equations are labeled with numbers to give them prominence. Thus, equations within a proof that are also important outside the context of that proof are numbered. Though the typical equation in a proof is unworthy of a number, it may be labeled with a letter to help with the “bookkeeping.” Likewise, digressions or examples in the main body of the text may contain equations labeled with letters that decrease the prominence given to them. Integral signs and summation signs in important equations (or sufficiently complicated equations) are large, while those in less important equations are small. It is a matter of assigned prominence. The most important theorems, definitions, and examples have been given titles in boldface type to assign prominence to them. The titles of somewhat less important results are not in boldface type. Routine references to theorem 10.4.1 or definition 7.3.1 do not contain capitalized initial letters. The author very specifically wishes to downgrade the prominence given to this routine use of these words. Starting new sections on new pages allowed me to carefully control the field of vision as the most important results were presented.

Chapter 1

Measures 1

Basic Properties of Measures

Motivation 1.1 (The Lebesgue integral) The Riemann integral of a continuous function f , we will restrict attention to f (x) ≥ 0 for a ≤ x ≤ b for convenience, is formed by subdividing the domain of f , forming approximating sums, and passing b to the limit. Thus the mth Riemann sum for a f (x) dx is defined as RSm ≡

(1)

m

f (x∗mi ) [xmi − xm,i−1 ],

i=1

where a ≡ xm0 < xm1 < · · · < xmm ≡ b (with xm,i−1 ≤ x∗mi ≤ xmi for all i) satisfy meshm ≡ max[xmi −xm,i−1 ] → 0. Note that xmi −xm,i−1 is the measure (or length) of the interval [xm,i−1 , xmi ], while f (x∗mi ) approximates the values of f (x) for all xm,i−1 ≤ x ≤ xmi (at least it does if f is continuous on [a, b]). Within the class C+ of all nonnegative continuous functions, this definition works reasonably well. But b b it has one major shortcoming. The conclusion a fn (x) dx → a f (x) dx is one we often wish to make. However, even when all fn are in C + and f (x) ≡ lim fn (x) b actually exists, it need not be the case that f is in C+ ; and thus a f (x) dx may not even be well-defined. A different approach is needed. (Note figure 1.1.) The Lebesgue integral of a nonnegative function is formed by subdividing the b range. Thus the mth Lebesgue sum for a f (x) dx is defined as LSm ≡

(2)

m m2

i=1

k−1 × measure 2m

k−1 k x : m ≤ f (x) < m 2 2

b

,

and a f (x) dx is defined to be the limit of the LSm sums as m → ∞. For what class M of functions f can this approach succeed? The members f of the class M will need to be such that the measure (or length) of all sets of the form k−1 k (3) x : m ≤ f (x) < m 2 2 1

2

CHAPTER 1. MEASURES

can be specified. This approach leads to the concept of a σ-field A of subsets of [a, b] that are measurable (that is, we must be able to assign to these intervals a number called their “length”), and this leads to the concept of the class M of measurable functions. This class M of measurable functions will be seen to be closed under passage to the limit and all the other operations that we are accustomed to perb b forming on functions. Moreover, the desirable property a fn (x) dx → a f (x) dx for functions fn “converging” to f will be broadly true. 2

xm0

xm1

…

xm,i-1

xmi

…

Riemann sums The domain of f(·) is equally divided.

1 2m

Lebesgue sums The range of f(·) is equally divided. Figure 1.1 Riemann sums and Lebesgue sums.

xmm

1.

BASIC PROPERTIES OF MEASURES

3

Definition 1.1 (Set theory) Consider a nonvoid class A of subsets A of a nonvoid set Ω. (For us, Ω will be the sample space of an experiment.) (a) Let Ac denote the complement of A, let A ∪ B denote the union of A and B, let A ∩ B and AB both denote the intersection, let A \ B ≡ ABc denote the set difference, let A B ≡ (Ac B ∪ ABc ) denote the symmetric difference, and let ∅ denote the empty set. The class of all subsets of Ω will be denoted by 2Ω . Sets A and B are called disjoint if AB = ∅, and sequences of sets An or classes of sets At ∞ are called disjoint if all pairs are disjoint. Writing A + B or 1 An will also denote a union, but will imply the disjointness of the sets in the union. As usual, A ⊂ B denotes that A is a subset of B. We call a sequence An increasing (and we will nearly always denote this fact by writing An ) when An ⊂ An+1 for all n ≥ 1. We call the sequence decreasing (denoted by An ) when An ⊃ An+1 for all n ≥ 1. We call the sequence monotone if it is either increasing or decreasing. Let ω denote a generic element of Ω. We will use 1A (·) to denote the indicator function of A, which equals 1 or 0 at ω according as ω ∈ A or ω ∈ A. (b) A will be called a field if it is closed under complements and unions. (That is, A and B in A requires that Ac and A ∪ B be in A.) [Note that both Ω and ∅ are necessarily in A, as A was assumed to be nonvoid, with Ω = A ∪ Ac and ∅ = Ωc .] (c) A will be called a σ-field if it is closed under complements and countable unions. (That is, A, A1 , A2 , . . . in A requires that Ac and ∪∞ 1 An be in A.) (d) A will be called a monotone class provided it contains ∪∞ 1 An for all increasing sequences An in A and contains ∩∞ 1 An for all decreasing sequences An in A. (e) (Ω, A) will be called a measurable space provided A is a σ-field of subsets of Ω. (f) A will be called a π-system provided AB is in A for all A and B in A; and A will be called a π ¯ -system when Ω in A is also guaranteed. If A is a field (or a σ-field), then it is closed under intersections (under countable ∞ c c intersections); since AB = (Ac ∪ B c )c (since ∩∞ 1 An = (∪1 An ) ). Likewise, we could have used “intersection” instead of “union” in our definitions by making use ∞ c c of A ∪ B = (Ac ∩ B c )c and ∪∞ 1 An = (∩1 An ) . Proposition 1.1 (Closure under intersections) (a) Arbitrary intersections of fields, σ-fields, or monotone classes are fields, σ-fields, or monotone classes, respectively. [For example, ∩{Fα : Fα is a field under consideration} is a field.] (b) There exists a minimal field, σ-field, or monotone class generated by (or, containing) any specified class C of subsets of Ω. We call C the generators. For example, (4) σ[C] ≡ {Fα : Fα is a σ-field of subsets of Ω for which C ⊂ Fα } is the minimal σ-field generated by C (that is, containing C) . (c) A collection A of subsets of Ω is a σ-field if and only if it is both a field and a monotone class. ∞ n ∞ Proof. (c) (⇐) ∪∞ 1 An = ∪1 (∪1 Ak )) ≡ ∪1 Bn ∈ A since the Bn are in A and are . Everything else is even more trivial. 2

4

CHAPTER 1. MEASURES

Exercise 1.1 (Generators) Let C1 and C2 denote two collections of subsets of the set Ω. If C2 ⊂ σ[C1 ] and C1 ⊂ σ[C2 ], then σ[C1 ] = σ[C2 ]. Prove this fact. Definition 1.2 (Measures) Consider a set function µ : A → [0, ∞] (that is, µ(A) ≥ 0 for each A ∈ A) having µ(∅) = 0. (a) If A is a σ-field and µ is countably additive (abbreviated c.a.) in that (5)

∞ ∞

An = µ(An ) µ n=1

for all disjoint sequences An in A,

n=1

then µ is called a measure (or, equivalently, a countably additive measure) on (Ω, A). The triple (Ω, A, µ) is then called a measure space. We call µ finite if µ(Ω) ∞. < ∞ We call µ σ-finite if there exists a measurable decomposition of Ω as Ω = 1 Ωn with Ωn ∈ A and µ(Ωn ) < ∞ for all n. [If A is not a σ-field, we will still call µ a measure on (Ω, A), provided that (5) holds for all sequences An for which ∪∞ 1 An is in A. We will not, however, use the term “measure space” to describe such a triple. We will consider below measures on fields, on certain π ¯ -systems, and on some other collections of sets. A useful property of a collection of sets is that along with any 1 , . . . , Ak it also include all sets sets A of the type Bk ≡ Ak Ack−1 · · · Ac2 Ac1 ; then n1 An = n1 Bk is also included.] (b) We call µ a finitely additive measure (abbreviated f.a.) on (Ω, A) if n n (6) µ( 1 An ) = 1 µ(An ) for all disjoint sequences An in A for which n1 An is also in A. Definition 1.3 (Outer measures) Consider a set function µ∗ : 2Ω → [0, ∞]. (a) Suppose that µ∗ satisfies the following three properties. Null: µ∗ (∅) = 0. Monotone: µ∗ (A) ≤ µ∗ (B) forall A ⊂ B. ∞ ∞ Countable subadditivity: µ∗ ( 1 An ) ≤ 1 µ∗ (An ) for all An . Then µ∗ is called an outer measure. (b) An arbitrary subset A of Ω is called µ∗ -measurable if (7)

µ∗ (T ) = µ∗ (T A) + µ(T Ac )

for all subsets T ⊂ Ω.

Sets T used in this capacity are called test sets. (c) We let A∗ denote the class of all µ∗ -measurable sets, that is, (8)

A∗ ≡ {A ∈ 2Ω : A is µ∗ -measurable}.

[Note that A ∈ A∗ if and only if µ∗ (T ) ≥ µ∗ (T A) + µ∗ (T Ac ) for all T ⊂ Ω, since the other inequality is trivial by the subadditivity of µ∗ .] Motivation 1.2 (Measure) In this paragraph we will consider only one possible measure µ, namely the Lebesgue-measure generalization of length. Let CI denote the set of all intervals of the types (a, b], (−∞, b], and (a, +∞) on the real line R, and for each of these intervals I we assign a measure value µ(I) equal to its length, thus ∞, b − a, ∞ in the three special cases. All is well until we manipulate the sets

1.


5

in CI , as even the union of two elements in CI need not be in CI . Thus, CI is not a very rich collection of sets. A natural extension is to let CF denote the collection of all finite disjoint unions of sets in CI , where the measure µ(A) we assign to each such set A is just the sum of the measures (lengths) of all its disjoint pieces. Now CF is a field, and is thus closed under the elementary operations of union, intersection, and complementation. Much can be done using only CF and letting “measure” be the “exact length”. But CF is not closed under passage to the limit, and it is thus insufficient for many of our needs. For this reason the concept of the smallest σ-field containing CF , labeled B ≡ σ[CF ], is introduced. We call B the Borel sets. But let us work backwards. Let us assign an outer measure value µ∗ (A) to every subset A in the class 2R of all subsets of the real line R. In particular, to any ∞subset A we assign the value µ∗ (A) that is the infimum of all possible numbers n=1 µ(An ), in which each An is in the field CF (so that we know its measure) in which the An ’s form and ∞ a cover of A (in that A ⊂ ∪∞ µ(A n ) is a natural upper 1 An ). Thus each number 1 bound to the measure (or generalized length) of the set A, and we will specify the infimum of such upper bounds to be the outer measure of A. Thus to each subset A of the real line we assign a value µ∗ (A) of generalized length. This value seems “reasonable”, but does it “perform correctly”? Let us say that a particular set A is µ∗ -measurable (that is, it “performs correctly”) if µ∗ (T ) = µ∗ (T A) + µ∗ (T Ac ) for all subsets of the real line R, that is, if the A versus Ac division of the line divides every subset T of the line into two pieces in a fashion that is µ∗ -additive. This is undoubtedly a combination of reasonableness and fine technicality that took some time to evolve in the mind of its creator, Carathéodory, while he searched for a condition that “worked”. In what sense does it “work”? The collection A∗ of all µ∗ -measurable sets turns out to be a σ-field. Thus the collection A∗ is closed under all operations that we are likely to perform; and it is big enough, in that it is a σ-field that contains CF . Thus we will work with the restriction µ∗ |A∗ of µ∗ to the sets of A∗ (here, the vertical line means “restricted to”). This is enough to meet our needs. There are many measures other than length. For an and right-continuous function F on the real line (called a generalized df) we define the Stieltjes measure of an arbitrary interval (a, b] (with −∞ ≤ a < b ≤ ∞) in CI by µF ((a, b]) = F (b) − F (a), and we extend it to sets in CF by adding up the measure of the pieces. Reapplying the previous paragraph, we can extend µF to the µ∗F -measurable sets. It is the important Carathéodory extension theorem that will establish that all Stieltjes measures (including the case of ordinary length, where F (x) = x as considered in the first paragraph) can be extended from CF to the Borel sets B. That is, all Borel sets are µ∗ -measurable for every Stieltjes measure. One further extension is possible, in that every measure can be “completed” (see the end of section 1.2). We note here only that when the Stieltjes measure µF associated with the generalized df F is “completed”, its domain of definition is extended from the Borel sets B (which all Stieltjes measures have in common) to a larger collection BˆµF that depends on the particular F . It is left to section 1.2 to simply state that this is as far as we can go. That is, except in trivial special cases, we find that BˆµF is a proper subset of 2R . (That is, it is typically impossible to try to define the measure of all subsets of Ω in a suitable fashion.) 2

6

CHAPTER 1. MEASURES

Example 1.1 (Some examples of measures, informally) (a) Lebesgue measure: Let λ(A) denote the length of A. (b) Counting measure: Let #(A) denote the number of “points” in A (or the cardinality of A). (c) Unit point mass: Let δω0 (A) ≡ 1{ω0 } (A), assigning measure 1 or 0 as ω0 ∈ A or not. 2 Example 1.2 (Borel sets) (a) Let Ω = R and let C consist of all finite disjoint unions of intervals of the types (a, b], (−∞, b], and (a, +∞). Clearly, C is a field. Then B ≡ σ[C] will be called the Borel sets (or the Borel subsets of R). Let µ(A) be defined to be the sum of the lengths of the intervals composing A, for each A ∈ C. Then µ is a c.a. measure on the field C, as will be seen in the proof of theorem 1.3.1 below. (b) If (Ω, d) is a metric space and U ≡ {all d-open subsets of Ω}, then B ≡ σ[U] will be called the Borel sets or the Borel σ-field . (c) If (Ω, d) is (R, | · |) for absolute value | · |, then σ[C] = σ[U] even though C = U. [This claim is true, since C ⊂ σ[U] and U ⊂ σ[C] are clear. Then, just make a trivial appeal to exercise 1.1.] ¯ ≡ [−∞, = ∞] denote the extended real line and let B¯ ≡ σ[B, {−∞}, {+∞}]. (d) Let R 2 Proposition 1.2 (Monotone property of measures) measure space. Let (A1 , A2 , . . .) be in A. (a) If An ⊂ An+1 for all n, then ∞ (9) µ( 1 An ) = lim µ(An ).

Let (Ω, A, µ) denote a

n→∞

(b) If µ(An0 ) < ∞ for some n0 , and An ⊃ An+1 for all n, then µ( ∞ (10) n=1 An ) = lim µ(An ) . n→∞

[Letting Ω denote the real line R, letting An = [n, ∞), and letting µ denote either Lebesgue measure or counting measure, we see the need for some requirement.] (c) (Countable subadditivity) Whenever (A1 , A2 , . . .) and ∪∞ 1 An are all in A, then ∞ ∞ µ( 1 Ak ) ≤ 1 µ(Ak ) ; and this also holds true for a measure on a field or on a π ¯ -system. Proof.

(p)

(a) Now,

∞ µ( ∪∞ with A0 ≡ ∅ 1 An ) = µ( 1 (An \ An−1 )) ∞ = 1 µ(An \ An−1 ) by c.a. n = lim k=1 µ(Ak \ Ak−1 ) n n = lim µ( k=1 (Ak \ Ak−1 )) by f.a. n

(q)

= lim µ(An ). n

1.


7

(b) Without loss of generality, redefine A1 = A2 = · · · = An0 . Let Bn ≡ A1 \An , so that Bn . Thus, on the one hand, lim µ(Bn ) = µ( ∪∞ 1 Bn ) n

by (a)

c = µ( ∪∞ 1 (A1 ∩ An )) c = µ(A1 ∩ ∪∞ 1 An ) c = µ(A1 ∩ (∩∞ 1 An ) )

= µ(A1 ) − µ( ∩∞ 1 An ).

(r)

On the other hand, lim µ(Bn ) = lim{ µ(A1 ) − µ(An )} n

(s)

n

by f.a.

= µ(A1 ) − lim µ(An ). n

Equate (r) and (s); since µ(A1 ) < ∞, we can cancel it to obtain the equality µ(∩∞ 1 An ) = limn µ(An ). (c) Let B1 ≡ A1 , B2 ≡ A2 Ac1 , . . . , Bk ≡Ak Ack−1 · · · Ac1 . Then these newly n defined sets Bk are disjoint, and ∪nk=1 Ak = k=1 Bk . Hence [a technique worth remembering] n n n µ( k=1 Ak ) = µ( k=1 Bk ) = k=1 µ(Bk ) (11) n n where k=1 Ak = k=1 Bk for Bk ≡ Ak Ack−1 · · · Ac1 ≤ (t)

≤

n

1 µ(Ak )

∞ 1

µ(Ak )

by monotonicity.

Let n → ∞, and use part (a) to get the result.

2

Definition 1.4 (Continuity of measures) A measure µ is continuous from below (above) if µ(lim An ) = lim µ(An ) for all An (for all An , with at least one µ(An ) finite). We call µ continuous in case it is continuous both from below and from above. The next result is often used in conjunction with the Carathéodory extension theorem of the next section. View it as a converse to the previous proposition. Proposition 1.3 (Continuity of measures) If a finitely additive measure µ on either a field or σ-field is either continuous from below or has µ(Ω) < ∞ and is continuous from above at ∅, then it is a countably additive measure.

8 Proof.

(a) (b) (c)

CHAPTER 1. MEASURES Suppose first that µ is continuous from below. Then ∞ n µ( 1 An ) = µ( lim 1 Ak ) n by continuity from below = lim µ( 1 Ak ) n = lim 1 µ(Ak ) by f.a. (where we used only that A is a field) ∞ = 1 µ(An ),

giving the required countable additivity. Suppose next that µ is finite and is also continuous from above at ∅. Then f.a. (even if A is only a field) gives n ∞ n ∞ µ( ∞ 1 An ) = µ( 1 Ak ) + µ( n+1 Ak ) = 1 µ(Ak ) + µ( n+1 Ak ) ∞ (d) → 1 µ(Ak ) + 0, ∞ ∞ where µ( n+1 Ak ) → µ(∅) = 0 by continuity from above at ∅, since n+1 Ak ∅ and µ is finite. 2 Definition 1.5 (liminf and limsup of sets) Let ∞ ∞ lim An ≡ n=1 k=n Ak = {ω : ω is in all but finitely many An ’s} (12) ≡ {ω : ω ∈ An a.b.f.}, where we use a.b.f. to abbreviate in all but finitely many cases. Let ∞ limAn ≡ ∞ n=1 k=n Ak = {ω : ω is in an infinite number of An ’s} (13) ≡ {ω : ω ∈ An i.o.}, where we use i.o. to abbreviate infinitely often. [It is important to learn to read these two mathematical equations in a way that makes it clear that the verbal description is correct.] Note that we always have lim An ⊂ lim An . Define (14)

lim An ≡ lim An

whenever lim An = lim An .

We also let lim inf An ≡ lim An and lim sup An ≡ lim An , giving us alternative notations. Definition 1.6 (lim inf and lim sup of numbers) Recall that for real number sequences an one defines lim an ≡ lim inf an and lim an ≡ lim sup an by lim inf n→∞ an ≡ limn→∞ (inf k≥n ak ) = supn≥1 (inf k≥n ak ) (15)

and

lim supn→∞ an ≡ limn→∞ supk≥n ak = inf n≥1 supk≥n ak ,

and these yield the smallest limit point and the largest limit point, respectively, of the sequence an .

1.


9

Definition 1.7 (“Little oh”, “big oh”, and “at most” ⊕) an ≡ o(rn )

if

an /rn → 0 ,

an ≡ O(rn )

if

lim |an /rn | ≤ (some M ) < ∞.

if

|an − bn | ≤ cn .

We write:

(16) We write (17)

an = bn ⊕ cn

This last notation allows us to string inequalities together linearly, instead of having to start a new inequality on a new line. (I use it often.) Proposition 1.4 Clearly, lim An equals ∪∞ 1 An when An is an sequence, and lim An equals ∩∞ 1 An when An is a sequence. Exercise 1.2 We always have µ(lim inf An ) ≤ lim inf µ(An ), while the inequality lim sup µ(An ) ≤ µ(lim sup An ) holds if µ(Ω) < ∞. Exercise 1.3 (π-systems and λ-systems) A class D of subsets is called a λ-system if it contains the space Ω and all proper differences (A \ B, for A ⊂ B with both A, B ∈ D) and if it is closed under monotone increasing limits. [Recall that a class is called a π-system if it is closed under finite intersections, while π ¯ -systems are also required to contain Ω.] (a) The minimal λ-system generated by the class D is denoted by λ[D]. Show that λ[D] is equal to the intersection of all λ-systems containing D. (b) A collection A of subsets of Ω is a σ-field if and only if it is both a π-system and a λ-system. (c) Let C be a π-system and let D be a λ-system. Then C ⊂ D implies that σ[C] ⊂ D. Proposition 1.5 (Dynkin’s π-λ theorem) Let µ and µ be two measures on the measurable space (Ω, A). Let C be a π ¯ -system, where C ⊂ A. Then (18) Proof. (19)

¯ -system C µ = µ on the π

implies

µ = µ on σ[C].

We first show that σ[C] = λ[C]

when C is a π-system.

Let D ≡ λ[C]. By the easy exercise 1.3(a)(b) it suffices to show that D is a π-system (that is, that A, B ∈ D implies A ∩ B ∈ D). We first go just halfway; let (a)

E ≡ {A ∈ D : AC ∈ D for all C ∈ C}.

Then C ⊂ E. Also, for A, B ∈ E with B ⊂ A and for C ∈ C we have (since both AC and BC are in D) that (A \ B)C = (AC \ BC) ∈ D, so that A \ B ∈ E. Thus E = D, since D was the smallest such class. We have thus learned of D that (b)

AC ∈ D for all C ∈ C, for each A ∈ D.

10

CHAPTER 1. MEASURES

To go the rest of the way, we define (c)

F ≡ {D ∈ D : AD ∈ D for all A ∈ D}.

Then C ⊂ F, by (b). Also, for A, B ∈ F with B ⊂ A and for D ∈ D we have (since both AD and BD are in D) that (A \ B)D = (AD \ BD) ∈ D, so that A \ B ∈ F. Thus F = D, since D was the smallest such class. We have thus learned of D that (d)

AD ∈ D for all A ∈ D, for each D ∈ D.

That is, D is closed under intersections; and thus D is a π-system. We will now demonstrate that G ≡ {A ∈ A : µ(A) = µ (A)} is a λ-system on Ω. First, Ω ∈ G, since Ω is in the π ¯ -system C. Second, when A ⊂ B are both in G we have the equality (e)

µ(B \ A) = µ(B) − µ(A) = µ (B) − µ (A) = µ (B \ A),

giving B \ A ∈ G. Finally, let An A with all An ’s in G. Then proposition 1.2(i) yields the result (f)

µ(A) = lim µ(An ) = lim µ (An ) = µ (A),

so that A ∈ G. Thus G is a λ-system. Thus the set G on which µ = µ is a λ-system that contains the π ¯ -system C. Applying (19) shows that σ[C] ⊂ G. 2 The previous proposition is very useful in extending independence from small classes of sets to large ones. The next proposition is used in proving the Carathéodory extension theorem, Fubini’s theorem, and the existence of a regular conditional probability distribution. Proposition 1.6 (Minimal monotone class; Halmos) The minimal monotone class M ≡ m[C] containing the field C and the minimal σ-field σ[C] generated by the same field C satisfy (20)

m[C] = σ[C]

when C is a field.

Proof. Since σ-fields are monotone classes, we have that σ[C] ⊃ M. If we now show that M is a field, then proposition 1.1.1(c) will imply that σ[C] ⊂ M. To show that M is a field, it suffices to show that (a)

A, B in M implies AB, Ac B, AB c are in M.

Suppose that (a) has been established. We will now show that (a) implies that M is a field. Complements: Let A ∈ M, and note that Ω ∈ M, since C ⊂ M. Then A, Ω ∈ M implies that Ac = Ac Ω ∈ M by (a).

1.


11

Unions: Let A, B ∈ M. Then A ∪ B = (Ac ∩ B c )c ∈ M. Thus M is indeed a field, provided that (a) is true. It thus suffices to prove (a). For each A ∈ M, let MA ≡ {B ∈ M : AB, Ac B, AB c ∈ M}. Note that it suffices to prove that (b)

MA = M

for each fixed A ∈ M.

We first show that (c)

MA is a monotone class.

Let Bn be monotone in MA , with limit set B. Since Bn is monotone in MA , it is also monotone in M, and thus B ≡ limn Bn ∈ M. Since Bn ∈ MA , we have ABn ∈ M, and since ABn is monotone in M, we have AB = limn ABn ∈ M. In like fashion, Ac B and AB c are in M. Therefore, B ∈ MA , by definition of MA . That is, (c) holds. We next show that (d)

MA = M

for each fixed A ∈ C.

Let A ∈ C and let C ∈ C. Then A ∈ MC , since C is a field. But A ∈ MC if and only if C ∈ MA , by the symmetry of the definition of MA . Thus C ⊂ MA . That is, C ⊂ MA ⊂ M, and MA is a monotone class by (c). But M is the minimal monotone class containing C, by the definition of M. Thus (d) holds. But in fact, we shall now strengthen (d) to (e)

MB = M

for each fixed B ∈ M.

The conditions for membership in M imposed on pairs A, B are symmetric. Thus for A ∈ C, the statement established above in (d) that B ∈ M(= MA ) is true if and only if A ∈ MB . Thus C ⊂ MB , where MB is a monotone class. Thus MB = M, since (as was earlier noted) M is the smallest such monotone class. Thus (e) (and hence (a)) is established. 2

12

2

CHAPTER 1. MEASURES

Construction and Extension of Measures

Definition 2.1 (Outer extension) Let Ω be arbitrary. Let µ be a measure on a field C of subsets Ω. For each A ∈ 2Ω define ∞ ∞ ∗ (1) µ(An ) : A ⊂ An with all An ∈ C . µ (A) ≡ inf n=1

n=1

∗

Now, µ is called the outer extension of µ. The sequences A1 , A2 , . . . are called Carathéodory coverings. [There is always at least one covering, since Ω ∈ C.] Theorem 2.1 (Carath´ eodory extension theorem) A measure µ on a field C can be extended to a measure on the σ-field σ[C] generated by C, by defining (2)

µ(A) ≡ µ∗ (A)

for each A in A ≡ σ[C].

If µ is σ-finite on C, then the extension is unique on A and is also σ-finite. Proof. The proof proceeds by a series of claims. Claim 1: µ∗ is an outer measure on (Ω, 2Ω ). Null: Now, µ∗ (∅) = 0, since ∅, ∅, . . . is a covering of ∅. Monotone: Let A ⊂ B. Then every covering of B is also a covering of A. Thus µ∗ (A) ≤ µ∗ (B). Countably subadditive: Let all An ⊂ Ω be arbitrary. Let > 0. For each An there is a covering {Ank : k ≥ 1} such that ∞ ∗ n (3) since µ∗ (An ) is an infimum. 1 µ(Ank ) ≤ µ (An ) + /2 , [The choice of a convergent series (like /2n ) that adds to is an important technique for the reader to learn.] Now ∪n An ⊂ ∪n ∪k Ank ). Thus since µ∗ is monotone µ∗ ( n An ) ≤ µ∗ ( n k Ank ) ≤ n k µ (Ank ) since the Ank ’s form a covering of the set n k Ank ≤ n [ µ∗ (An ) + /2n ] = n µ∗ (An ) + . But > 0 was arbitrary, and thus µ∗ ( n An ) ≤ n µ∗ (An ). Claim 2: µ∗ |C = µ (that is, µ∗ (C) = µ(C) for all C ∈ C) and C ⊂ A∗ . Let C ∈ C. Then µ∗ (C) ≤ µ(C), since C, ∅, ∅, . . . is a covering of C. For the other direction, we let A1 , A2 , . . . be any covering of C. Since µ is c.a. on C, and since ∪∞ 1 (An ∩ C) = C ∈ C, we have from proposition 1.1.2(c) that ∞ ∞ µ(C) = µ( ∞ 1 (An ∩ C)) = 1 µ(An ∩ C) ≤ 1 µ(An ), and thus µ(C) ≤ µ∗ (C). Thus µ(C) = µ∗ (C). We next show that any C ∈ C is also in A∗ . Let C ∈ C. Let > 0, and let T be given. There exists a covering {An }∞ 1 ⊂ C of T such that

2. (a)

CONSTRUCTION AND EXTENSION OF MEASURES

13

∞ µ∗ (T ) + ≥ 1 µ(An ) since µ∗ (T ) is an infimum ∞ c = ∞ 1 µ(CAn ) + 1 µ(C An ) since µ is c.a. on C with C and An in C

(b)

≥ µ∗ (CT ) + µ∗ (C c T )

since CAn covers CT and C c An covers C c T.

But > 0 is arbitrary. Thus C ∈ A∗ . Thus C ⊂ A∗ . Claim 3: The class A∗ of µ∗ -measurable subsets of Ω is a field that contains C. Now, A ∈ A∗ implies that Ac ∈ A∗ : The definition of µ∗ -measurable is symmetric in A and Ac . And A, B ∈ A∗ implies that AB ∈ A∗ : For any test set T ⊂ Ω we have the required inequality µ∗ (T ) = µ∗ (T A) + µ∗ (T Ac )

since A ∈ A

= µ∗ (T AB) + µ∗ (T AB c ) + µ∗ (T Ac B) + µ∗ (T Ac B c ) since B ∈ A∗ with test set T A and with test set T Ac ≥ µ∗ (T AB) + µ∗ (T AB c + T Ac B + T Ac B c ) since µ∗ is countably subadditive (c)

= µ∗ (T AB) + µ∗ (T (AB)c ).

Thus A∗ is a field. Claim 4: µ∗ is a f.a. measure on A∗ . Let A, B ∈ A∗ be disjoint. Finite additivity follows from µ∗ (A + B) = µ∗ ((A + B)A) + µ∗ ((A + B)Ac ) since A ∈ A∗ with test set A + B (d)

= µ∗ (A) + µ∗ (B).

Trivially, µ∗ (A) ≥ 0 for all sets A. And µ∗ (∅) = 0, since φ, φ, . . . is a covering of φ. Claim 5: A∗ is a σ-field, ∞and it contains σ[C]. n We will show that A ≡ 1 An ∈ A∗ whenever all An ∈ A∗ . Now, 1 Ak ∈ A∗ , since A∗ is a field. We thus have n n n µ∗ (T ) = µ∗ (T ( 1 An )) + µ∗ (T ( 1 An )c ) since 1 An ∈ A∗ n ≥ µ∗ (T ( 1 An )) + µ∗ (T Ac ) since µ∗ is monotone and ( n1 An )c ⊃ Ac n (e) ≥ 1 µ∗ (T Ak ) + µ∗ (T Ac ) by claims 3 and 4. Letting n → ∞ gives ∞ (f) µ∗ (T ) ≥ 1 µ∗ (T An ) + µ∗ (T Ac ) (g)

≥ µ∗ (T A) + µ∗ (T Ac )

Thus A ∈ A∗ .

since µ∗ is countably subadditive.

14

CHAPTER 1. MEASURES

Claim 6: µ∗ is c.a. on A∗ . ∗ Replace T by A in (f) to get µ∗ (A) ≥ ∞ 1 µ (An ), and then countable subadditivity gives the reverse inequality. It remains only to establish the uniqueness, for which we will appeal to the minimal monotone class result of proposition 1.1.6. This proposition is interesting in its own right. Claim 7: Uniqueness holds when µ is a finite measure. Let µ1 and µ2 denote any two extensions of µ. Let M ≡ {A ∈ σ[C] : µ1 (A) = µ2 (A)} denote the class where they are equal. We will first show that (h)

M is a monotone class.

Let An be monotone in M. Then (i)

µ1 ( lim An ) = lim µ1 (An )

by propositions 1.1.4 and 1.1.2

(j)

= lim µ2 (An )

since µ1 = µ2 on M

(k)

= µ2 ( lim An )

by propositions 1.1.4 and 1.1.2.

Thus (h) holds. Since C ⊂ M, the minimal monotone class result of proposition 1.1.6 implies that σ[C] ⊂ M. Thus µ1 = µ2 on σ[C] (and possibly on even more sets than this). Thus the claimed uniqueness holds. [Appeal to proposition 1.1.6 could be replaced by appeal to Dynkin’s π-λ theorem of proposition 1.1.5.] Claim 8: Uniqueness holds when µ is a σ-finite measure (label the sets of the measurable partition as Ωn ). We must again demonstrate the uniqueness. Fix n. We will consider µ, µ1 , µ2 on C, on σ[C] ∩ Ωn , and on σ[C ∩ Ωn ]. We first show that (l)

σ[C] ∩ Ωn ≡ {A ∩ Ωn : A ∈ σ[C]} = σ[C ∩ Ωn ] ≡ σ[{C ∩ Ωn : C ∈ C}].

Now, C is a subset of the σ-field D ≡ {A ∈ σ[C] : A ∩ Ωn ∈ σ[C ∩ Ωn ]}, and hence σ[C] ⊂ D, so that σ[C] ∩ Ωn ⊂ σ[C ∩ Ωn ]. The other direction is even more obvious. Now, µ1 |σ[C ∩ Ωn ] = µ2 |σ[C ∩ Ωn ], since µ|σ[C ∩ Ωn ] is a finite measure on the field C ∩ Ωn . Thus for A ∈ σ[C] we have µ1 (A) = ∞ by c.a. 1 µ1 (AΩn ) ∞ = 1 µ2 (AΩn ) since µ1 = µ2 on σ[C ∩ Ωn ] = σ[C] ∩ Ωn , by claim 7 (m)

= µ2 (A)

completing the proof.

by c.a., 2

Question We extended our measure µ from the field C to a collection A∗ that is at least as big as the σ-field σ[C]. Have we actually gone beyond σ[C]? Can we go further?

2.


15

Definition 2.2 (Complete measures) Let (Ω, A, µ) denote a measure space. If µ(A) = 0, then A is called a null set. We call (Ω, A, µ) complete if whenever we have B ⊂ (some A) ∈ A with µ(A) = 0, we necessarily also have B ∈ A. [That is, all subsets of sets of measure 0 are required to be measurable.] Exercise 2.1 (Completion) (4)

Let (Ω, A, µ) denote a measure space. Show that

Aˆµ ≡ {A : A1 ⊂ A ⊂ A2 with A1 , A2 ∈ A and µ(A2 \ A1 ) = 0}

(5)

= {A ∪ N : A ∈ A, and N ⊂ (some B) ∈ A having µ(B) = 0}

(6)

= {A N : A ∈ A, and N ⊂ ( some B) ∈ A having µ(B) = 0},

and that Aˆµ is a σ-field. Define µ ˆ on Aˆµ by (7)

µ ˆ(A ∪ N ) = µ(A)

for all A ∈ A and for all N ⊂ (some B) ∈ A having µ(B) = 0. Show that (Ω, Aˆµ , µ ˆ) is a complete measure space for which µ ˆ |A = µ. [Note: A proof must include a demonstration that definition (7) leads to a well-defined µ ˆ. That is, whenever A1 ∪ N1 = A2 ∪ N2 we must have µ(A1 ) = µ(A2 ), so that µ ˆ(A1 ∪ N1 ) = µ ˆ(A2 ∪ N2 ).] Definition 2.3 (Lebesgue sets) The completion of Legesgue measure on (R, B, λ) is still called Lebesgue measure. The resulting completed σ-field Bˆλ of the Borel sets B is called the Lebesgue sets. Corollary 1 When we complete a measure µ on a σ-field A, this completed measure µ ˆ is the unique extension of µ to Aˆµ . [It is typical to denote the extension by µ also (rather than µ ˆ).] Corollary 2 Thus when we begin with a σ-finite measure µ on a field C, both ˆ [C]µ are unique. the extension to A ≡ σ[C] and the further extension to Aˆµ ≡ σ ∗ ∗ ˆ Here, we note that all sets in Aµ = σ ˆ [C]µ are in the class A of µ -measurable sets. Proof. Consider corollary 1 first. Let ν denote any extension to Aˆµ . We will demonstrate that (a)

ν(A ∪ N ) = µ(A)

for all A ∈ A, and all null sets N

(that is, ν = µ ˆ). Assume not. Then there exist sets A ∈ A and N ⊂ (some B) in A with µ(B) = 0 such that ν(A ∪ N ) > µ(A) [necessarily, ν(A ∪ N ) ≥ ν(A) = µ(A)]. For this A and N we have µ(A) < ν(A ∪ N ) = ν(A ∪ (Ac N )) (b)

where Ac N ⊂ Ac B = (a null set)

= ν(A) + ν(Ac N ) ≤ ν(A) + ν(B) since ν is a measure on the completion

(c)

= µ(A) + µ(B)

since ν is an extension of µ.

Hence µ(B) > 0, which is a contradiction. Thus the extension is unique.

16

CHAPTER 1. MEASURES

We now turn to corollary 2. Only the final claim needs demonstrating. Suppose A is in σ ˆ [C]µ . Then A = A ∪ N for some A ∈ A and some N satisfying N ⊂ B with µ(B) = 0. Since A∗ is a σ-field, it suffices to show that any such N is in A∗ . Since µ∗ is subadditive and monotone, we have (d)

µ∗ (T ) ≤ µ∗ (T N ) + µ∗ (T N c ) = µ∗ (T N c ) ≤ µ∗ (T ),

because µ∗ (T N ) = 0 follows from using B, ∅ ∅, . . . to cover T N . Thus equality holds in this last equation, showing that N is µ∗ -measurable. 2 Exercise 2.2 Let µ and ν be finite measures on (Ω, A). (a) Show by example that Aˆµ and Aˆν need not be equal. (b) Prove or disprove: Aˆµ = Aˆν if and only if µ and ν have exactly the same sets of measure zero. (c) Give an example of an LS-measure µ on R for which Bˆµ = 2R . Exercise 2.3 (Approximation lemma; Halmos) Let the σ-finite measure µ on the field C be extended to A = σ[C], and also refer to the extension as µ. Then for each A ∈ A (or in Aˆµ ) such that µ(A) < ∞, and for each > 0, we have (8)

µ(A C) <

for some set C ∈ C.

[Hint. Truncate the sum in (1.2.1) to define C, when A ∈ A.] Definition 2.4 (Regular measures on metric spaces) Let d denote a metric on Ω, let A denote the Borel sets, and let µ be a measure on (Ω, A). Suppose that for each set A in Aˆµ , and for every > 0, one can find an open set O and a closed set C for which both C ⊂ A ⊂ O and µ(O \ C ) < . Suppose also that if µ(A) < ∞, one then requires that the set C be compact. Then µ is called a regular measure. [Note exercise 1.3.1 below.] Exercise 2.4 (Nonmeasurable sets) Let Ω consist of the sixteen values 1, . . . , 16. (Think of them arranged in four rows of four values.) Let C1 = {1, 2, 3, 4, 5, 6, 7, 8}, C3 = {1, 2, 5, 6, 9, 10, 13, 14},

C2 = {9, 10, 11, 12, 13, 14, 15, 16}, C 4 = {3, 4, 7, 8, 11, 12, 15, 16}.

Let C = {C1 , C2 , C3 , C4 }, and let A = σ[C]. (a) Show that A ≡ σ[C] = 2Ω . (b) Let µ(Ci ) = 14 for 1 ≤ i ≤ 4. Show that Aˆµ = A, and that it thus contains 24 = 16 sets. (c) Let µ(Ci ) = 13 for 1 ≤ i ≤ 3, with µ(C4 ) = 0. Then show that Aˆµ contains 27 = 128 sets. (d) Illustrate proposition 2.2 below in the context of this exercise. Proposition 2.1 (Not all sets are measurable) Let µ be a measure on A ≡ σ[C], with C a field. If B ∈ Aˆµ , then there are infinitely many measures on σ[Aˆµ ∪ {B}] that agree with µ on C. [Thus the σ-field Aˆµ is as far as we can go with the unique extension process.] (We merely state this observation for reference, without proof.) [To exhibit a subset of R not in B requires the axiom of choice.]

2.


Proposition 2.2 (Not all subsets are Lebesgue sets) that is not in Bˆλ .

17

There is a subset D of R

Proof. Define the equivalence relation ∼ on elements of [0, 1) by x ∼ y if x−y is a rational number. Use the axiom of choice to specify a set D that contains exactly one element from each equivalence class. Now define Dz ≡ {z + x (modulo 1) : x ∈ D} for each rational z in [0, 1), so that [0, 1) = z Dz represents [0, 1) as a countable outer measure; union of disjoint sets. Moreover, all Dz must have the same call it a. Assume D = D is measurable. But then 1 = λ([0, 1)) = λ(D ) = 0 z z z a gives only z a = 0 (when a = 0) and z a = ∞ (when a > 0) as possibilities. This is a contradiction. Thus D ∈ Bˆλ . 2

Proposition 2.3 (Not all Lebesgue sets are Borel sets) There necessarily exists a set A ∈ Bˆλ \ B that is a Lebesgue set but not a Borel set. Proof.

This proof follows exercise 7.3.3 below.

Exercise 2.5

2

Every subset A of Ω having µ∗ (A) = 0 is a µ∗ -measurable set.

Coverings Earlier in this section we encountered Carathéodory coverings. Exercise 2.6 (Vitali cover) (a) We say that a family V of intervals I is a Vitali cover of a set D if for each x ∈ D and each > 0 there exists an interval I ∈ V for which x ∈ I and λ(I) < . (b) (Vitali covering theorem) Let D ⊂ R be arbitrary. Let V be a Vitali cover of D. Then there exists a finite number of pairwise disjoint intervals (I1 , . . . , Im ) in V for which Lebesgue outer measure λ∗ satisfies m (9) λ∗ (D \ j=1 Ij ) < . [Lebesgue measure λ will be formally shown to exist in the next section, and λ∗ will be discussed more fully.] [Result (9) will be useful in establishing the Lebesgue result that increasing functions on R necessarily have a derivative, except perhaps on a set having Lebesgue measure zero.] Exercise 2.7 (Heine–Borel) If {Ut : t ∈ T } is an arbitrary collection of open sets that covers a compact subset D of R, then there exists a finite number of them U1 , . . . , Um that also covers D. [We are merely stating this well-known and frequently used result in the disguise of an exercise so that the reader can easily contrast it with the two other ideas of Carathéodory covering and Vitali covering introduced in this chapter.]

18

3

CHAPTER 1. MEASURES

Lebesgue–Stieltjes Measures

At the moment we know only a few measures informally. We now construct the large class of measures that lies at the heart of probability theory. Definition 3.1 (Lebesgue–Stieltjes measure) A measure µ on the real line R assigning finite values to finite intervals is called a Lebesgue–Stieltjes measure. [The measure µ on (R, 2R ) whose value µ(A) for any set A equals the number of rationals in A is not a Lebesgue–Stieltjes measure.] Definition 3.2 (gdf) A finite function F on R that is right-continuous is called a generalized df (to be abbreviated gdf). Then F− (·) ≡ limy· F (y) denotes the left-continuous version of F . The mass function of F is defined by ∆F (·) ≡ F (·) − F− (·),

while

F (a, b] ≡ F (b) − F (a) for all a < b

is called the increment function of F . We identify gdfs having the same increment function. Only one member F of each equivalence class obtained by such identification satisfies F− (0) = 0, and this F can (and occasionally will) be used as the representative member of the class (also to be called the representative gdf). Example 3.1 We earlier defined three measures on (R, B) informally. (a) For Lebesgue measure λ, a gdf is the identity function F (x) = x. (b) For counting measure, a gdf is the greatest integer function F (x) = [x]. (c) For unit point mass at x0 , a gdf is F (x) = 1[x0 ,∞) (x). 2 Theorem 3.1 (Correspondence theorem; Lo` eve) (1)

µ((a, b]) = F (a, b]

The relationship

for all − ∞ ≤ a < b ≤ +∞

establishes a 1-to-1 correspondence between Lebesgue–Stieltjes measures µ on B and the representative members of the equivalence classes of generalized dfs. [Each such µ extends uniquely to Bˆµ .] Notation 3.1 We formally establish some notation that will be used throughout. Important classes of sets include: (2)

CI ≡ {all intervals (a, b], (−∞, b], or (a, +∞) : −∞ < a < b < +∞}.

(3)

CF ≡ {all finite disjoint unions of intervals in CI } = (a field) .

(4)

B ≡ σ[CF ] ≡ (the σ-field of Borel sets).

(5)

Bˆµ ≡ (the σ-field B completed for the measure µ).

(6)

B¯ ≡ σ[B, {−∞}, {+∞}].

2

3.

LEBESGUE–STIELTJES MEASURES

19

Proof. Given an LS-measure µ, define the increment function F (a, b] via (1). We clearly have 0 ≤ F (a, b] < ∞ for all finite a, b, and F (a, b] → 0 as b a, by proposition 1.1.2. Now specify F− (0) ≡ 0, F (0) ≡ µ({0}), F (b) ≡ F (0) + F (0, b] for b > 0, and F (a) = F (0) − F (a, 0] for a < 0. This F (·) is the representative gdf. Given a representative gdf, we define µ on the collection I of all finite intervals (a, b] via (1). We will now show that µ is a well-defined and c.a. measure on this collection I. Nonnegative: µ ≥ 0 for any (a, b], since F is . Null: µ(∅) = 0, since ∅ = (a, a] and F (a, a] = 0. ∞ Countably additive andwell-defined: Suppose I ≡ (a, b] = ∞ 1 In ≡ 1 (an , bn ]. ∞ ∞ We must show that µ( I ) = µ(I ). n n 1 1 n ∞ First, we will show that 1 µ(In ) ≤ µ(I). Fix n. Then 1 Ik ⊂ I, so that (relabel if necessary, so that I1 , . . . , In is a left-to-right ordering of these intervals) n n (a) 1 µ(Ik ) = 1 F (ak , bk ] ≤ F (a, b] = µ(I). Letting n → ∞ in (a) gives the first claim. ∞ Next, we will show that µ(I) ≤ 1 µ(In ). Suppose b−a > > 0 (the case b−a = 0 is trivial, as µ(∅) = 0). For each n ≥ 1, use the right continuity of F to choose an

n > 0 so small that (b)

F (bn , bn + n ] < /2n ,

and define Jn ≡ (an , cn ) ≡ (an , bn + n ).

These Jn form an open cover of the compact interval [a + , b], so that some finite number of them are known to cover [a + , b], by the Heine–Borel theorem. Sorting through these intervals one at a time, choose (a1 , c1 ) to contain b, choose (a2 , c2 ) to contain a1 , choose (a3 , c3 ) to contain a2 , . . . ; finally (for some K), choose (aK , cK ) to contain a + . Then (relabeling the subscripts, if necessary) K K K F (a + , b] ≤ F (aK , c1 ] ≤ 1 F (ak , ck ] ≤ 1 F (ak , bk ] + 1 /2k ∞ (c) ≤ 1 µ(Ik ) + . Let 0 in (c) to obtain the second claim as ∞ (d) µ(I) = F (a, b] ≤ ∞ 1 F (ak , bk ] = 1 µ(Ik ). We will now show that µ is a well-defined c.a. measure on the given field CF . If A = n In ∈ CF with each In of type (a, b], then we define µ(A) ≡ n µ(In ). If we also have another representation A = m Im of this set, then we must show (where the subscripts m and n could take on either a finite or a countably infinite number of values) that (e) n µ(In ) = m µ(Im ). Now, Im = A ∩ Im = n In Im and In = AIn = m Im In , so that (f) m µ(Im ) = m n µ(In Im ) = n m µ(In Im ) = n µ(In ) (g)

= µ(A).

Finally, a measure µ on CF determines a unique measure on B, as is guaranteed by the Carathéodory extension of theorem 1.2.1. 2

20 Exercise 3.1 measures.

CHAPTER 1. MEASURES Show that all Lebesgue–Stieltjes measures on (R, B) are regular

Probability Measures, Probability Spaces, and DFs Definition 3.3 (Probability distributions P (·) and dfs F (·)) (a) In probability theory we think of Ω as the set of all possible outcomes of some experiment, and we refer to it as the sample space. The individual points ω in Ω are referred to as the elementary outcomes. The measurable subsets A in the collection A are referred to as events. A measure of interest is now denoted by P ; it is called a probability measure, and must satisfy P (Ω) = 1. We refer to P (A) as the probability of A, for each event A in AˆP . The triple (Ω, A, P ) (or (Ω, AˆP , Pˆ ), if this is different) is referred to as a probability space. (b) An right-continuous function F on R having F (−∞) ≡ limx→−∞ F (x) = 0 and F (+∞) ≡ limx→+∞ F (x) = 1 is called a distribution function (which we will abbreviate as df). [For probability measures, setting F (−∞) = 0 is used to specify the representative df.] Corollary 1 (The correspondence theorem for dfs) Defining P (·) on all intervals (a, b] via P ((a, b] ) ≡ F (b) − F (a) for all −∞ ≤ a < b ≤ +∞ establishes a 1-to-1 correspondence between all probability distributions P (·) on (R, B) and all dfs F (·) on R. Exercise 3.2

Prove the corollary.

Chapter 2

Measurable Functions and Convergence 1

Mappings and σ-Fields

Notation 1.1 (Inverse images) Suppose X denotes a function mapping some ¯ ≡ R ∪ {±∞}; we denote this by X : Ω → R. ¯ set Ω into the extended real line R Let X + and X − denote the positive part and the negative part of X, respectively: X(ω) if X(ω) ≥ 0, (1) X + (ω) ≡ 0 else, (2)

X − (ω) ≡

−X(ω) if X(ω) ≤ 0, 0 else.

Note that (3)

X = X+ − X−

and |X| = X + + X − = X + 2 X − = 2 X + − X .

We also use the following notation: (4)

[ X = r ] ≡ X −1 (r) ≡ { ω : X(ω) = r }

for all real r,

(5)

[ X ∈ B ] ≡ X −1 (B) ≡ { ω : X(ω) ∈ B }

for all Borel sets B,

(6)

X −1 (B) ≡ { X −1 (B) : B ∈ B } .

We call these the inverse images of r, B, and B, respectively. We let (7)

B¯ ≡ σ[ B, {+∞}, {−∞}] .

Inverse images are also well-defined when X : Ω → Ω∗ for arbitrary sets Ω and Ω∗ . 2 21

22

CHAPTER 2. MEASURABLE FUNCTIONS AND CONVERGENCE

For A, B ∈ Ω we define A B ≡ ABc ∪ Ac B and A \ B ≡ AB c . There is use for the notation (8)

X ≡ sup |X(ω)| , ω∈Ω

and we will also reintroduce this sup norm in other contexts below.

Proposition 1.1 Let X : Ω → Ω and Y : Ω → Ω . Let T denote an arbitrary index set. Then for all A, B, At ⊂ Ω we have (9) (10)

X −1 (B c ) = [ X −1 (B) ]c , X −1 ( t∈T At ) = t∈T X −1 (At ),

X −1 (A \ B) = X −1 (A) \ X −1 (B), X −1 ( t∈T At ) = t∈T X −1 (At ).

For all sets A ⊂ Ω , the composition Y ◦ X satisfies (11) Proof.

(Y ◦ X)−1 (A) = X −1 (Y −1 (A)) = X −1 ◦ Y −1 (A) . 2

Trivial.

Proposition 1.2 (Preservation of σ-fields)

Let X : Ω → Ω . Then:

(12)

X −1 (a σ-field A of subsets of Ω ) = (a σ-field A of subsets of Ω ) .

(13)

X −1 (σ[C ] ) = σ[X −1 (C )]

for any collection C of subsets of Ω .

{A : X −1 (A ) ∈ (a specific σ-field A of subsets of Ω)} (14) Proof.

= (a σ-field A of subsets of Ω ) . Now, (12) is trivial from proposition 1.1. Consider (14). Now: A ∈ A

(a)

implies X −1 (A ) ∈ A

A n ’s ∈ A (b)

implies X −1 (A c ) = [X −1 (A )]c ∈ A implies A c ∈ A , implies X −1 (A n )’s ∈ A

implies X −1 ( n An ) = n X −1 (An ) ∈ A

implies

n An

∈ A .

This gives (14). Consider (13). Now, (c)

X −1 (σ[C ] ) = (a σ-field containing X −1 (C )) ⊃ σ[X −1 (C )] .

Then (14) shows that (d)

A ≡ { A : X −1 (A ) ∈ σ[X −1 (C )] } = (a σ-field containing C ) ⊃ σ[C ] ,

so that (e)

X −1 (σ[C ] ) ⊂ X −1 (A ) ⊂ σ[X −1 (C )] .

Combining (c) and (e) gives (13). [We apply (13) below to obtain (2.2.6).]

2

1.

MAPPINGS AND σ-FIELDS

23

Roughly, using (12) we will restrict X so that X −1 (B¯ ) ⊂ A for our original ¯ B¯ ), while (14) tells us (Ω, A, P ), so that we can then “induce” a distribution on (R, that the collection A is such that we can always induce a distribution on (Ω , A ). We do this in the next section. First we generalize our definition of Borel sets to n dimensions. Example 1.1 (Euclidean space)

Let

Rn ≡ R × · · · × R ≡ {(r1 , . . . , rn ) : each ri is in R} . Let Un denote all open subsets of Rn , in the usual Euclidean metric. Then (15)

Bn ≡ σ[Un ] is called the class of Borel sets of Rn .

Following the usual notation, B1 ×· · ·×Bn ≡ {(b1 , . . . , bn ) : b1 ∈ B1 , . . . , bn ∈ Bn }. Now let n (16) i=1 B ≡ B × · · · × B ≡ σ[{B1 × · · · × Bn : all Bi are in B}]. Now consider (17)

σ[ {(−∞, r1 ] × · · · × (−∞, rn ] : all ri are in R} ].

Note that these three σ-fields are equal. Just observe that each of these three classes generates the generators of the other two classes, and apply exercise 1.1.1. [Surely, we can define a generalization of area λ2 on (R2 , B2 ) by beginning with λ2 (B1 × B2 ) = λ(B1 ) × λ(B2 ) for all B1 and B2 in B, and then extending to all sets in B2 . We will do this in theorem 5.1.1, and call it Lebesgue measure.] 2

24

2


Measurable Functions

We seek a large usable class of functions that is closed under passage to the limit. This is the fundamental property of the class of measurable functions. Propositions 2.2 and 2.3 below will show that the class of measurable functions is also closed under all of the standard mathematical operations. Thus, this class is sufficient for our needs. Definition 2.1 (Simple functions, etc.) Let the measure space (Ω, A, µ) be given and fixed throughout all of our discussion. The indicator function 1A (·) of the set A ⊂ Ω is defined by 1 if ω ∈ A, (1) 1A (ω) ≡ 0 else. A simple function is of the form (2)

X(ω) ≡

n

xi 1Ai (ω)

for

n

Ai = Ω with all Ai ∈ A, and xi ∈ R.

1

i=1

An elementary function is of the form (3)

X(ω) ≡

∞

xi 1Ai (ω)

for

i=1

∞

¯ Ai = Ω with all Ai ∈ A, and xi ∈ R.

i=1

Definition 2.2 (Measurability) Suppose that X : Ω → Ω , where (Ω , A ) is a measurable space. We say that X is A -A-measurable if X −1 (A ) ⊂ A. We also denote this by writing (4)

X : (Ω, A) → (Ω , A )

or

X : (Ω, A, µ) → (Ω , A )

(or even X : (Ω, A, µ) → (Ω , A , µ ) for the measure µ “induced” on (Ω , A ) by ¯ B¯ ), the mapping X, as will soon be defined). In the special case X : (Ω, A) → (R, we simply call X measurable; and in this special case we let F(X) ≡ X−1 (B¯ ) denote the sub σ-field of A generated by X. ¯ Suppose σ[C ] = B. ¯ Proposition 2.1 (Measurability criteria) Let X : Ω → R. Then measurability can be characterized by: (5)

X is measurable

if and only if X −1 (C) ⊂ A,

(6)

X is measurable

if and only if X −1 ( [−∞, x] ) ∈ A for all x ∈ R ;

and note that we could replace [−∞, x] by any one of [−∞, x), [x, +∞], or (x, +∞].

Proof. (a) (b)

Consider (5). Now, X −1 (B¯ ) = X −1 (σ[C ] ) = σ[X −1 (C)] ⊂A

by proposition 2.1.2

since X −1 (C) ⊂ A, and A is a σ-field.

2.

MEASURABLE FUNCTIONS

25

The other direction is trivial. Thus (5) holds. To demonstrate (6), we need to show that B¯ satisfies (c)

¯ σ[ {[−∞, x] : x ∈ R } ] = B.

Since B = σ[CI ] for CI as in (1.3.2) and (d)

(a, b] = [−∞, b] ∩ [−∞, a]c ,

(e)

{−∞} = ∩n [−∞, −n],

[−∞, b) = ∪∞ 1 [−∞, b − 1/n],

{+∞} = ∩n [−∞, n]c ,

the equality (c) is obvious. The rest is trivial.

etc.,

2

Proposition 2.2 (Measurability of common functions) Let X, Y , and Xn ’s be measurable functions. Consider cX with c > 0, −X, inf Xn , sup Xn , lim inf Xn , lim sup Xn , lim Xn if it exists, X 2 , X ± Y if it is well-defined, XY where 0 · ∞ ≡ 0, X/Y if it is well-defined, X + , X − , |X|, g(X) for continuous g, and the composite function g(X) for all measurable functions g. All of these are measurable functions. Proposition 2.3 (Measurability via simple functions) (7)

Simple and elementary functions are measurable.

(8)

¯ is measurable X :Ω→R if and only if X is the limit of a sequence of simple functions.

Moreover: (9)

If X ≥ 0 is measurable, then X is the limit of a sequence of simple functions that are ≥ 0 and .

[The Xn ’s and Zn ’s that are defined in the proof below are important.] Proof.

The functions in proposition 2.2 are measurable, since: [−X < x] = [X > −x].

(a)

[cX < x] = [X < x/c],

(b)

[inf Xn < x] = ∪[Xn < x],

sup Xn = − inf(−Xn ).

(c)

lim inf Xn = sup( inf Xk ),

lim sup Xn = − lim inf(−Xn ).

n

(d)

k≥n

(e)

lim Xn = lim inf Xn , provided that lim Xn (ω) exists for all ω. √ √ √ √ [X 2 < x] = [− x < X < x ] = [X < x ] ∩ [X ≤ − x ]c .

(f)

[X + Y > x] = ∪r {[X > r] ∩ [Y > x − r] : r is rational}.

(g)

X − Y = X + (−Y ),

(h)

X/Y = X × (1/Y ),

XY = [(X + Y )2 − (X − Y )2 ]/4 .

26


where [1/Y < x] = [Y > 1/x] for x > 0, and [1/Y < x] = [Y < 1/x] for x < 0 and [1/Y < 0] = [Y < 0]. X+ = X ∨ 0

(i)

and

X − = (−X) ∨ 0 .

For g measurable, (g ◦ X)−1 (B¯ ) = X −1 (g −1 (B¯ )) ⊂ X −1 (B¯ ) ⊂ A. Then continuous g are measurable, since g −1 (B) = g −1 (σ[ open sets ] ) = σ[g−1 ( open sets )] ⊂ σ[ open sets ] ⊂ B¯ ,

(j)

¯ where we now apply the and both g−1 ({+∞}) = ∅ ∈ B¯ and g −1 ({−∞}) = ∅ ∈ B, result for measurable g. We now prove proposition 2.3. Claim (7) is easy. Consider (8). Define simple functions Xn by n2n k−1 Xn ≡ × 1 k−1 − 1 k−1 (10) [ 2n ≤ X < 2kn ] [ 2n ≤ −X < 2kn ] 2n k=1

+ n × {1[ X≥n ] − 1[ −X≥n ] } . Since |Xn (ω) − X(ω)| ≤ 2−n for |X(ω)| < n, we have Xn (ω) → X(ω)

(k)

as n → ∞

for each ω ∈ Ω.

Also, the nested subdivisions k/2n cause Xn to satisfy Xn

(l)

when X ≥ 0.

We extend proposition 2.3 slightly by further observing that (11)

Xn − X → 0

as n → ∞,

if X is bounded.

Also, the elementary functions ∞ k−1 (12) Zn ≡ × 1 k−1 − 1 k−1 k k [ 2n ≤X< 2n ] [ 2n ≤−X< 2n ] 2n k=1

+ ∞ × {1[ X=∞ ] − 1[ X=−∞ ] } are always such that (13)

(Zn − X) × 1[−∞<X 0, = 0, < 0. The exercise is to give geometric descriptions of the σ-fields F(Z1 ), F(Z2 ), and F(Z1 , Z2 ). Proposition 2.5 (The form of an F(X)-measurable function) Suppose that Y is a measurable function on (Ω, A) that is F(X)-measurable. Then there must exist a measurable function g on (R, B) such Y = g(X). Proof. [The approach of this proof is to consider indicator functions, simple functions, nonnegative functions, general functions. This approach will be used again and again. Learn it!] Suppose that Y = 1D for some set D ∈ F(X), so that Y is an indicator function that is F(X)-measurable. Then we can rewrite Y as Y = 1D = 1X −1(B) = 1B (X) = g(X), for some B ∈ B. Thus the proposition holds for indicator functions. It holds for simple functions, since when all Bi ∈ B, m m m Y = 1 ci 1Di = 1 ci 1X −1 (Bi ) = 1 ci 1Bi (X) ≡ g(X). Let Y ≥ 0 be F(X)-measurable. Then there do exist simple F(X)-measurable functions Yn such that Y ≡ limn Yn = limn gn (X) for the simple B-measurable functions gn . Now let g = lim gn , which is B-measurable, and note that Y = g(X). For general Y = Y + − Y − , use g = g + − g − . 2 Exercise 2.2 (Measurability criterion) Let C denote a π ¯ -system of subsets of Ω. Let V denote a vector space of functions (that is, X + Y ∈ V and αX ∈ V for all X, Y ∈ V and all α ∈ R). (a) Suppose that: (17)

1C ∈ V

for all C ∈ C,

(18)

If An A

with 1An ∈ V,

then 1A ∈ V.

Show that 1A ∈ V for every A ∈ σ[C]. (b) It then follows trivially that every simple function m (19) Xn ≡ 1 αi 1Ai is in V ; m here m ≥ 1, all αi ∈ R, and 1 Ai = Ω with all Ai ∈ σ[C ]. (c) Now suppose further that Xn X for Xn ’s as in (19) implies that X ∈ V. Show that V contains all σ[C]-measurable functions.

3.

CONVERGENCE

29

3

Convergence

Convergence Almost Everywhere Definition 3.1 (→a.e. ) Let X1 , X2 , . . . denote measurable functions on (Ω, A, µ) ¯ B¯ ). Say that the sequence Xn converges almost everywhere to X (denoted to (R, by Xn →a.e. X as n → ∞) if for some N ∈ A for which µ(N ) = 0 we have Xn (ω) → X(ω) as n → ∞ for all ω ∈ N . If for all ω ∈ N the sequence Xn (ω) is a Cauchy sequence, then we say that the sequence Xn mutually converges a.e. and denote this by writing Xn − Xm →a.e. 0 as m ∧ n → ∞. ¯ B¯ ). Exercise 3.1 Let X1 , X2 , . . . be measurable functions from (Ω, A, µ) to (R, ˜ ˜ (a) If Xn →a.e. X, then X = X a.e. for some measurable X. (b) If Xn →a.e. X and µ is complete, then X itself is measurable. Proposition 3.1 A sequence of measurable functions Xn that are a.e. finite converge a.e. to a measurable function X that is a.e. finite if and only if these functions Xn converges mutually a.e. [Thus we can redefine such functions on null sets and make them everywhere finite and convergent.] Proof. The union of the countable number of null sets on which finiteness or convergence fails is again a null set N . On N c , the claim is just a property of the real numbers. 2 Proposition 3.2 (The convergence and divergence sets are measurable) Consider the finite measurable functions X, X1 , X2 , . . . (perhaps they have been redefined on null sets to achieve this); thus, they are B-A-measurable. Then the convergence and mutual convergence sets are measurable. In fact, the convergence set is given by ∞ ∞ ∞ 1 |Xm − X| < (1) [Xn → X] ≡ ∈ A, k n=1 m=n k=1

and the mutual convergence set is given by ∞ ∞ ∞ 1 (2) [Xn − Xm → 0] ≡ |Xm − Xn | < ∈ A. k n=1 m=n k=1

Proof. Just read the right-hand side of (1) as, for all ≡ 1/k > 0 there exists an n such that for all m ≥ n we have |Xm (ω) − X(ω)| < 1/k. (Practice saying this until it makes sense.) 2 Taking complements in (1) allows the divergence set to be expressed via ∞ ∞ ∞ ∞ 1 c (3) [ Xn → X] = |Xm − X| ≥ ≡ Ak with Ak in k, k n=1 m=n k=1

where (4)

Ak =

∞

n=1 Dkn ,

k=1

where the Dkn ≡

∞

m=n [ |Xm − X|

≥ 1/k ] are in n.

30


Proposition 3.3 (i) We have

(5)

Consider finite measurable Xn ’s and a finite measurable X.

if and only if Xn →a.e. (such an X) ∞ for all > 0. µ( ∞ n=1 m=n [ |Xm − Xn | ≥ ] ) = 0 ,

[A finite limit X exists if and only if the Cauchy criterion holds; and we want to be able to check for the existence of a finite limit X without knowing its value.] (ii) (Most useful criterion for →a.e. ) When µ(Ω) < ∞, we have

(6)

if and only if Xn →a.e. (such an X) ∞ µ( m=n [ |Xm − Xn | ≥ ] ) → 0, for all > 0,

(7)

µ( [ max |Xm − Xn | ≥ ] ) ≤ for all N ≥ n ≥ (some n ), for all > 0.

Proof.

if and only if

n≤m≤N

This is immediate from (3), (4), and proposition 1.1.2.

2

Remark 3.1 (Additional measurability for convergence and divergence) Suppose we still assume that X1 , X2 , . . . are finite measurable functions. Then the following sets are seen to be measurable:

(8) (9)

¯ ]c = [ lim inf Xn < lim sup Xn ] [ω : Xn (ω) → X(ω) ∈ R = rational r [ lim inf Xn < r < lim sup Xn ] ∈ A, [ lim sup Xn = +∞ ] = ∞ m=1 [ lim sup Xn > m ] ∈ A.

These comments reflect the following fact: If Xn (ω) does not converge to a finite number, then there are several different possibilities; but these interesting events are all measurable. 2

Convergence in Measure Definition 3.2 (→µ ) A given sequence of a.e. finite and measurable functions X1 , X2 , . . . is said to converge in measure to the measurable function X taking ¯ (to be denoted by Xn →µ X as n → ∞) if values in R (10)

µ( [ |Xn − X| ≥ ] ) → 0

as n → ∞,

for all > 0.

[This implies that X must be finite a.s., as ∞ [ |X| = ∞] ⊂ { k=1 [ |Xk | = ∞]} ∪ [ |Xn − X| ≥ ] shows.] We say that these Xn converge mutually in measure, which we denote by writing Xm − Xn →µ 0 as m ∧ n → ∞, if µ( [ |Xm − Xn | ≥ ] ) → 0 as m ∧ n → ∞, for each > 0.

3.

CONVERGENCE

Proposition 3.4 Proof. (a)

31

˜ then X = X ˜ a.e. If Xn →µ X and Xn →µ X,

For all > 0 ˜ ≥ 2 ] ) ≤ µ( [ |Xn − X| ≥ ] ) + µ( [ |Xn − X| ˜ ≥ ] ) → 0. µ( [ |X − X|

Thus (b)

˜ ) = µ( [ |X − X| ˜ ≥ 1/k ] ) ≤ ∞ µ( |X − X| ˜ ≥ 1/k ) = ∞ 0 , µ( [ X = X] 1 1 k

as claimed.

2

Exercise 3.2 (a) Show that in general →µ does not imply →a.e. . (b) Give an example with µ(Ω) = ∞ where →a.e. does not imply →µ . Theorem 3.1 (Relating →µ to →a.e. ) finite a.e.

Let X, X1 , X2 , . . . be measurable and

(11)

Xn →a.e. (such an X)

if and only if

Xn − Xm →a.e. 0.

(12)

Xn →µ (such an X)

if and only if

Xn − Xm →µ 0.

(13)

Let µ(Ω) < ∞. Then Xn →a.e. (such an X) implies Xn →µ X.

(14)

(Riesz) If Xn →µ X, then for some subsequence nk we have Xnk →a.e. X.

(Reducing →µ to →a.e. by going to subsequences) Suppose µ(Ω) < ∞. Then

(15)

Xn →µ X if and only if each subsequence n has a further n on which Xn →a.e. (such an X).

Proof. Now, (11) is proposition 3.1, and (12) is exercise 3.3. Result (13) comes from the elementary observation that ∞ (a) µ( [ |Xn − X| ≥ ] ) ≤ µ( m=n [ |Xm − X| ≥ ] ) → 0, by (6). To prove (14), choose nk ↑ such that (b)

µ(Ak ) ≡ µ( [ |Xnk − X| > 1/2k ] ) < 1/2k ,

with µ( [ |Xn − X| > 1/2k ] ) < 1/2k for all n ≥ nk . Now let ∞ ∞ (c) Bm ≡ k=m Ak , so that µ(Bm ) ≤ k=m 2−k ≤ 1/2m−1. ∞ c On Bm = m Ack we have |Xnk − X| ≤ 1/2k for all k ≥ m, so that (16)

c |Xnk (ω) − X(ω)| ≤ 1/2k → 0 as k → ∞ for each ω ∈ Bm ,

with µ(Bm ) < 1/2m−1 . Thus (d)

Xnk (ω) → X(ω) as k → ∞ for each ω ∈ C ≡

∞

c m=1 Bm

,

32


where (e)

µ(C c ) = µ(

∞

m=1 Bm )

≤ lim sup µ(Bm ) ≤ lim 1/2m−1 = 0

as required. (For exercise 3.3 below, replace X above by Xnk+1 , and claim a.s. convergence to some X on this subsequence. Then show that the whole sequence converges in measure to this X.) Consider the unproven half of (15) by assuming that every n contains a further n as claimed. Assume that Xn →µ fails. Then for some > 0 and some n (f)

limn µ( [ |Xn − X| > ] ) = (some a) > 0.

But we are given that some further subsequence n has Xn →a.e. X, and thus Xn →µ X by (13). Thus (g)

limn µ( [ |Xn − X| > ] ) = 0;

but this is a contradiction of (f).

2

Exercise 3.3 As in (12), show that Xn →µ X if and only if Xm − Xn →µ 0. [Hint. Adapt the proof of (16).] Exercise 3.4 Suppose that µ(Ω) < ∞ and g is continuous a.e. µX (that is, g is continuous except perhaps on a set of µX measure 0). Then Xn →µ X implies g(Xn ) →µ g(X).

4.

PROBABILITY, RVS, AND CONVERGENCE IN LAW

4

Probability, RVs, and Convergence in Law

33

Definition 4.1 (Random variable and df ) A probability space (Ω, A, P ) is just a measure space for which P (Ω) = 1. Now, X : (Ω, A, P ) → (R, B) will be called a random variable (to be abbreviated rv); thus it is a B-A-measurable ¯ B¯ ), then we will call X an extended rv. The function. If X : (Ω, A, P ) → (R, distribution function (to be abbreviated df) of a rv is defined by (1)

FX (x) ≡ P (X ≤ x)

for all − ∞ < x < ∞.

We recall that F ≡ FX satisfies (2)

F is and right continuous, with F (−∞) = 0 and F (+∞) = 1.

We let CF denote the continuity set of F , that contains all points at which F is continuous. [That F is trivial, and the other three properties all follow from the monotone property of measure, since (∞, x] = ∩∞ n=1 (−∞, x + an ] for every sequence ∞ an 0, ∩∞ n=1 (−∞, −n] = ∅, and ∪n=1 (−∞, n] = R.] (If F is and right continuous with F (−∞) ≥ 0 and F (+∞) ≤ 1, then F will be ¯ B¯ )) will be denoted by PX , called a sub df.) The induced measure on (R, B) (or (R, and it satisfies (3)

PX (B) = P (X −1 (B)) = P (X ∈ B)

for all B ∈ B

(for all B ∈ B¯ if X is an extended rv). We call this the induced distribution of X. We use the notation X ∼ = F to denote that the induced distribution PX (·) of the rv X has df F . We say that rvs Xn (with dfs Fn ) converge in distribution or converge in law to a rv X0 (with df F0 ) if (4)

Fn (x) = P (Xn ≤ x) → F0 (x) = P (X0 ≤ x)

at each x ∈ CF0 .

We abbreviate this by writing either Xn →d X0 , Fn →d F0 , or L(Xn ) → L(X0 ). Notation 4.1 Suppose now that {Xn : n ≥ 0}, are rvs on (Ω, A, P ). Then it is customary to write Xn →p X0 (in place of Xn →µ X0 ) and Xn →a.s. X0 (as well as Xn →a.e. X0 ). The “p” is an abbreviation for in probability, and the “a.s.” is an abbreviation for almost surely. Anticipating the next chapter, we let Eg(X) denote g(X) dµ, or g(X) dP when µ is a probability measure P . We say that Xn converges to X0 in rth mean if E|Xn − X0 |r → 0. We denote this by writing Xn →r X0 or Xn →Lr X0 . 2 Proposition 4.1 Then Xn →d X.

Suppose that the rvs X ∼ = F and Xn ∼ = Fn satisfy Xn →p X.

Proof. (This result has limited importance. But the technique introduced here is useful; see exercise 4.1 below.) Now, (a) (b)

Fn (t) = P (Xn ≤ t) ≤ P (X ≤ t + ) + P ( |Xn − X| ≥ ) ≤ F (t + ) +

for all n ≥ some n .

34


Also, Fn (t) = P (Xn ≤ t) ≥ P (X ≤ t − and |Xn − X| ≤ ) ≡ P (AB) ≥ P (A) − P (Bc ) = F (t − ) − P ( |Xn − X| ≥ ) ≥ F (t − ) −

for n ≥ (some n ).

Thus for n ≥ (n ∨ n ) we have (c)

F (t − ) − ≤ lim Fn (t) ≤ lim Fn (t) ≤ F (t + ) + .

If t is a continuity point of F , then letting → 0 in (c) gives Fn (t) → F (t). Thus Fn →d F . 2 The following elementary result is extremely useful. Often, one knows that Xn →d X, but what one is really interested in is a slight variant of Xn , rather than Xn itself. The next result was designed for just such situations. Theorem 4.1 (Slutsky) Suppose that Xn →d X, while the rvs Yn →p a and Zn →p b as n → ∞ (here Xn , Yn , and Zn are defined on a common probability space, but X need not be). Then (5)

Yn × Xn + Zn →d aX + b

as n → ∞.

Exercise 4.1

Prove Slutsky’s theorem. [Hint. Recall proposition 4.1.]

Exercise 4.2

Let c be a constant. Show that Xn →d c if and only if Xn →p c.

Remark 4.1 Suppose X1 , X2 , . . . are independent rvs with a common df F . Then Xn →d X0 for any rv X0 having df F . However, there is no rv X for which Xn converges to X in the sense of →a.s. , →p , or →r . 2

5.

DISCUSSION OF SUB σ-FIELDS

5

Discussion of Sub σ-Fields

35

Consider again a sequence of rvs X1 , X2 , . . . where each quantity Xn is a measurable transformation Xn : (Ω, A, P ) → (R, B, Pn ), with Pn denoting the induced measure PXn . Each of the rvs Xn is B-F(Xn )-measurable, with F(Xn ) a sub σ-field of A. Even though the intersection of any number of σ-fields is a σ-field, the union of even two σ-fields need not be a σ-field. We thus define the sub σ-field generated by X1 , . . . , Xn as n −1 (Bn ) ≡ (X1 , . . . , Xn ) , (1) F(X1 , . . . , Xn ) ≡ σ[ k=1 F(Xk ) ] = X for X where the equality will be shown in proposition 5.2.1 below. Note that F(X1 , . . . , Xn ) ⊂ F(X1 , . . . , Xn , Xn+1 ), so that these necessarily form an increasing sequence of σ-fields of A. Also, define ∞ (2) F(X1 , X2 , . . .) ≡ σ[ k=1 F(Xk ) ]. It is natural to say that such Xn ’s are adapted to the F(X1 , . . . , Xn ). In fact, if F1 ⊂ F2 ⊂ · · · is any sequence of σ-fields for which F(X1 , . . . , Xn ) ⊂ Fn for all n, then we say that the Xn ’s are adapted to the Fn ’s. Think of F(X1 , . . . , Xn ) as the amount of information available at time n from X1 , . . . , Xn ; that is, you have available for inspection all of the probabilities (3)

P ((X1 , . . . , Xn ) ∈ Bn ) = P ((X1 , . . . , Xn )−1 (Bn )) = P(X1 ,...,Xn ) (Bn ),

for all Borel sets Bn ∈ Bn . Rephrasing, you have available for inspection all of the probabilities (4)

P (A), for all A ∈ F(X1 , . . . , Xn ).

At stage n+1 you have available P (A) for all A ∈ F(X1 , . . . , Xn , Xn+1 ); that is, you have more information available. [Think of Fn \ F(X1 , . . . , Xn ) as the amount of information available to you at time n that goes beyond the information available from X1 , . . . , Xn ; perhaps it comes from other rvs not yet mentioned, but it is available nonetheless.] Suppose we are not given rvs, but rather (speaking informally now, based on your general feel for probability) we are given joint dfs Fn (x1 , . . . , xn ) that we think ought to suffice to construct probability measures on (Rn , Bn ). In (2.2.16) we saw that for n = 1 we could just let (Ω, A, µ) = (R, B, µF ) and use X(ω) = ω to define a rv that carried the information in the df F . How do we define probability measures Pn on (Rn , Bn ) so that the coordinate rvs (5)

Xk (ω1 , . . . , ωn ) = ωk

for all (ω1 , . . . , ωn ) ∈ Rn

satisfy (6)

Pn (X1 ≤ x1 , . . . , Xn ≤ Xn ) = F (x1 , . . . , xn )

for all (x1 , . . . , xn ) ∈ Rn ,

and thus carry all the information in Fn ? Chapter 5 will deal with this construction. But even now it is clear that for this to be possible, the Fn ’s will have to satisfy some kind of consistency condition as we go from step n to n + 1. Moreover, the consistency problem should disappear if the resulting Xn ’s are “independent”.

36


But we need more. We will let R∞ denote all infinite sequences ω1 , ω2 , . . . for which each ωi ∈ R. Now, the construction of (5) and (6) will determine probabilities ∞ on the collection Bn × k=n+1 R of all subsets of R∞ of the form ∞ Bn × k=n+1 R (7) ≡ {(ω1 , . . . , ωn , ωn+1, . . .) : (ω1 , . . . , ωn ) ∈ Bn , ωk ∈ R for k ≥ n + 1}, with Bn ∈ Bn . Each of these collections is a σ-field (which within this special probability space can be denoted by F(X1 , . . . , Xn )) in this overall probability space (R∞ , B∞ , P∞ ), for some appropriate B∞ . But what is an appropriate σ-field B∞ for such a probability measure P∞ ? At a minimum, B∞ must contain ∞ ∞ ∞ (8) σ[ n=1 {Bn × k=n+1 R}] = σ[ n=1 F(X1 , . . . , Xn )], and indeed, this is what we will use for B∞ . Of course, we also want to construct the measure P∞ on (R∞ , B∞ ) in such a way that n ∞ (9) P∞ (−∞, x ] × R = Fn (x1 , . . . , xn ) for all n ≥ 1 k k=1 k=n+1 and for all x1 , . . . , xn . The details are given in chapter 5. Until chapter 5 we will assume that we are given the rvs X1 , X2 , . . . on some (Ω, A, P ), and we will need to deal only with the known quantities F(X1 , . . . , Xn ) and F(X1 , X2 , . . .) defined in (1) and (2). This is probability theory: Given (Ω, A, P ), we study the behavior of rvs X1 , X2 , . . . that are defined on this space. Now contrast this with statistics: Given a physical situation producing measureθ ments X1 , X2 , . . ., we construct models {(R∞ , B∞ , P∞ ) : θ ∈ Θ} based on various θ plausible models for Fn (x1 , . . . , xn ), θ ∈ Θ, and we then use the data X1 , X2 , . . . and the laws of probability theory to decide which model θ0 ∈ Θ was most likely to have been correct and what action to take. In particular, the statistician must know that the models to be used are well-defined. We also need to extend all this to uncountably many rvs {Xt : t ∈ T }, for some interval T such as [a, b], or [a, ∞), or [a, ∞], or (−∞, ∞), . . . . We say that rvs Xt : (Ω, A, P ) → (R, B) for t ∈ T are adapted to an sequence of σ-fields Ft if Fs ⊂ Ft for all s ≤ t with boths, t ∈ T and if each Xt is Ft -measurable. In this situation we typically let RT ≡ t∈T Rt and then let (10) Ft ≡ F(Xs : s ≤ t) ≡ σ[ s Xs−1 (B) : s ≤ t and s ∈ T ] for all t ∈ T. This is also done in chapter 5 (where more general sets T are, in fact, considered). The purpose in presenting this section here is to let the reader start now to become familiar and comfortable with these ideas before we meet them again in chapter 5 in a more substantial and rigorous presentation. (The author assigns this as reading at this point and presents only a very limited amount of chapter 5 in his lectures.)

Chapter 3

Integration 1

The Lebesgue Integral

Let (Ω, A, µ) be a fixed measure space and let X, Y, Xn , . . . denote measurable n ¯ B¯ ). If Ω = functions from (Ω, A, µ) to (R, 1 Ai where A1 , . . . , An are in A, then A1 , . . . , An is called a partition (or measurable partition) of Ω. n Definition 1.1 (Lebesgue integral X dµ or X) If X = i=1 xi 1Ai is a simple function (in that all real xi ≥ 0 and A1 , . . . , An is a partition of Ω), then (1)

X dµ ≡

n

xi µ(Ai ).

i=1

If X ≥ 0, then (2) X dµ ≡ sup Y dµ : 0 ≤ Y ≤ X and Y is a simple function . [Of course, we must show that the value of representation of X that is specified.] For general measurable X, + X dµ ≡ X dµ − X − dµ, (3)

X dµ in (1) is independent of the

X + dµ and X − dµ is finite. We let L1 ≡ L1 (Ω, A, µ) ≡ {X : |X| dµ < ∞}, + L+ 1 ≡ L1 (Ω, A, µ) ≡ {X ∈ L1 : X ≥ 0}, Lr ≡ Lr (Ω, A, µ) ≡ Lr (µ) ≡ {X : |X|r dµ < ∞}, for each r > 0.

provided that at least one of

(4)

If X (which is not equals a measurable function Y on a set A having measurable) µ(Ac ) = 0, then X dµ ≡ Y dµ. [It is trivial that the value X dµ is independent 37

38

CHAPTER 3. INTEGRATION

of the choice of Y and A.] If X is measurable and X dµ is finite, then X is called integrable. For any A ∈ A, (5) X dµ ≡ X 1A dµ. A

We also use the notation (especially in proofs, to save space) (6) X ≡ X dµ ≡ (the integral of X) ≡ EX ≡ (the expectation of X) . For ordinary Lebesgue measure µ on R, we often write

X dµ =

X(r) dr .

It needs to be demonstrated that the above definition makes sense and that X dµ satisfies the following elementary properties.

Proposition 1.1 (Elementary properties of the integral) It holds that definition 1.1 of the integral is unambiguous. Now suppose that the functions X and Y are measurable, that X dµ and Y dµ are well-defined, and that their sum (the number X dµ + Y dµ) is a well-defined number in [−∞, +∞]. Then (7) (X + Y ) dµ = X dµ + Y dµ and cX dµ = c X dµ, (8)

X ≥0

implies

X dµ ≥ 0.

Proof. Consider case of simple functions. first the m m xi µ(Ai ) for simple functions X = 1 xi 1Ai makes Step 1: Defining X dµ = 1 X dµ well-defined for such simple functions. Suppose that we also have Y = n1 yj 1Bj . Then m n n m (a) i=1 xi j=1 1Ai Bj = X = j=1 yj i=1 1Ai Bj , so that xi = yj if Ai Bj = ∅. Thus m m n m n i=1 xi µ(Ai ) = i=1 xi j=1 µ(Ai Bj ) = i=1 j=1 xi µ(Ai Bj ) = =

m n i=1

n

j=1 yj

j=1

since xi = yj if Ai Bj = ∅

yj µ(Ai Bj )

m

i=1 µ(Ai Bj )

=

n j=1

yj µ(Bj ) ;

and since the two extreme terms represent the two different definitions of X dµ, we see that X dµ is well-defined. Step 2: The integral m behaves linearly n for simple functions. m n Suppose X = 1 xi 1Ai and Y = 1 yj 1Bj . Then X +Y = 1 1 (xi +yj ) 1Ai Bj . We thus have m n (X + Y ) dµ = 1 1 (xi + yj ) µ(Ai Bj ) m n m n = 1 1 xi µ(Ai Bj ) + 1 1 yj µ(Ai Bj )

1.

THE LEBESGUE INTEGRAL n m µ(Ai Bj ) + 1 yj 1 µ(Ai Bj ) n X dµ + Y dµ, = m 1 xi µ(Ai ) + 1 yj µ(Bj ) = =

m

39

1

xi

n 1

which establishes the additivity. Step 3: Even in general, the monotonicity is trivial; and so is cX = c X. The proof for general X ≥ 0 and Y ≥ 0 is included in the proof of the monotone convergence theorem (MCT) (that is, the first theorem of the next section). That is, we will prove the MCT using linearity just for simple functions, and then we will use the MCT to obtain the current linearity for any functions X ≥ 0 and Y ≥ 0. (The final linearity step is then trivial. Just write X = X + −X − and Y = Y + −Y − and do algebra.) 2

40

2


Fundamental Properties of Integrals

Theorem 2.1 (MCT, the monotone convergence theorem) Xn X a.e. for measurable functions Xn ≥ 0. Then (1) 0 ≤ Xn dµ X dµ .

Suppose that

Proof. By redefining on null sets if necessary, we may assume that Xn X for all ω. Thus X is measurable, by proposition 2.2.2. Also, X n is , and so a ≡ lim Xn exists in [0, ∞]. Moreover, Xn ≤ X implies Xn ≤ X; and so we conclude that a ≤ X. Let Y be an arbitrary simple function satisfying 0 ≤ Y ≤ X. Fix 0 < θ < 1. Then note that An ≡ [Xn ≥ θ Y ] Ω (since 0 ≤ θ Y ≤ X on [X = 0] and 0 ≤ θ Y < X on [X > 0] are both trivial). We thus have (a) θ Y × 1An = θ Y × 1An ≤ Xn × 1An ≤ Xn ≤ a; and passing to thelimit as n → ∞ gives θ Y ≤ a (by proposition 3.1.1) for each 0 < θ < 1, so that Y ≤ a. Since 0 ≤ Y ≤ X is arbitrary, this gives X ≤ a. This completes the proof of the MCT. 2 Proof. We now return to the linearity of the integral for general measurable functions X ≥ 0 and Y ≥ 0. Let Xn X and Yn Y be the measurable simple functions of (2.2.10). Then Xn +Yn X +Y . Thus the MCT twice, the linearity of the integral for simple functions, and then the MCT again give the general linearity of the integral X + Y = lim Xn + lim Yn = lim ( Xn + Yn ) (a) = lim (Xn + Yn ) by simple function linearity (b) = (X + Y ) by the MCT. 2

Theorem 2.2 (Fatou’s lemma) For Xn ’s measurable, (2) lim Xn dµ ≤ lim Xn dµ, provided that Xn ≥ 0 a.e. for all n. Proof. (3) so that (a) (b)

Now, a.e. we have Xn ≥ Yn ≡ inf Xk lim Xn , k≥n

or

lim Xn = lim Yn with Yn ,

lim Xn = lim Yn = lim Yn by the MCT = lim Yn ≤ lim Xn since Yn ≤ Xn . 2

2.

FUNDAMENTAL PROPERTIES OF INTEGRALS

41

Theorem 2.3 (DCT, the dominated convergence theorem) Suppose now that |Xn | ≤ Y a.e. for all n, for some dominating function Y ∈ L1 ; and suppose either (i) Xn →a.e. X or (ii) Xn →µ X. Then (4) (that is, Xn →L1 X). |Xn − X| dµ → 0 as n → ∞ [Note that (supn≥1 |Xn | ) would make a suitable dominating function.] Corollary 1 Note that (4) implies Xn dµ → X dµ (5) (that is, (6)

EXn → EX),

Xn dµ − X dµ → 0. sup

A∈A

A

A

Proof. (i) Suppose that Xn →a.e. X. Then Zn ≡ |Xn − X| →a.e. 0, with 0 ≤ Zn ≤ 2Y a.s., of the functions 0 and where both 2Y are in L1 . Thus Fatou’s lemma gives 0 = 0 dµ = lim inf Zn dµ ≤ lim inf Zn . Now apply Fatou’s lemma to the rvs 2Y − Zn , and conclude that (a) (2Y − 0) ≤ lim (2Y − Zn ) ≤ lim (2Y − Zn ) by Fatou (7) = lim ( 2Y − Zn ) = 2Y − lim Zn . Hence, lim sup Zn ≤ 0 = 0. Combining the two results gives (b) 0 ≤ lim inf Zn ≤ lim sup Zn ≤ 0 ; so lim Zn = 0, as claimed. (ii) Suppose Xn →µ X. Let a ≡ lim sup Zn ≥ 0. Let n be a subsequence such that Zn → a. But Zn →µ 0, so theorem 2.3.1 gives a further subsequence n such that Zn →a.e. 0, while we still have Zn → a. But Zn → 0 by case (i). Thus a = 0. Thus (c) 0 ≤ lim sup Zn = a = 0, or Zn → 0 . (iii) Consider the corollary. We have | Xn − X| ≤ |Xn − X|, and thus (d) | A Xn − A X| ≤ A |Xn − X| ≤ |Xn − X| → 0 uniformly in all A ∈ A. Theorem 2.4

∞ 1

2 Xn dµ =

∞

Proof. Note that 0 ≤ Zn ≡ MCT to the Zn ’s. 2

1

n 1

Xn dµ

Xk Z ≡

if Xn ≥ 0 a.e., for all n. ∞ 1

Xk a.e., and now apply the

42


Theorem 2.5 (Absolute continuity of the integral) (8) |X| dµ → 0 as µ(A) → 0.

Fix X ∈ L1 . Then

A

That is,

A

Proof. (a)

|X| dµ < , provided only that µ(A) < (an appropriate δ ).

Now,

|X| 1[ |X|≤n]

|X| 1[ |X|>n] ≤ /2

|X| by the MCT, so we may claim that

for n ≥ N ≡ (some N ) .

Thus (b)

A

|X| ≤

A

|X| 1[ |X|≤N ] +

provided that µ(A) ≤ /(2N ) .

|X| 1[ |X|>N ] ≤ N × µ(A) + /2 ≤ ,

2

Definition 2.1 (Induced measure) Suppose that X : (Ω, A, µ) → (Ω , A ) is a measurable function. Recall from (2.2.15) that (9)

µ (A ) ≡ µX (A ) = µ(X −1 (A ))

for all A ∈ A ,

and µ is a measure on (Ω , A ), called the induced measure of X. Theorem 2.6 (Theorem of the unconscious statistician) First, the induced measure µX (·) of the rv X determines the induced measure µg(X) for all measurable ¯ B¯ ). Second, functions g : (Ω , A ) → (R, (10) g(X(ω)) dµ(ω) = g(x) dµX (x) for all A ∈ A , X −1 (A )

A

in the sense that if either side exists, then so does the other and they are equal. Proof. (a)

Now, g(X) is measurable. By (2.1.11) and (2.2.5) we see that µg(X) (B) = µ( [g(X) ∈ B] ) = µ(X −1 ◦ g −1 (B)) = µX (g −1 (B))

is well-defined, since g−1 (B) ∈ A . Thus the first claim holds. We now turn to the second claim. It suffices to prove this when A = Ω and X −1 (Ω ) = Ω, since we need only replace g by g × 1A , noting that 1A (X(ω)) = 1X −1 (A ) (ω). Case 1. g = 1A : Then 1A (X) dµ = 1X −1 (A ) dµ = µ(X −1 (A )) = µX (A ) = 1A dµX . n n Case 2. g = i=1 ci 1Ai , where i=1 Ai = Ω with Ai ∈ A : Then n n g(X) dµ = i=1 ci 1Ai (X) dµ = i=1 ci 1Ai (X) dµ (b)

(c)

=

n

i=1 ci

1Ai dµX =

g dµX .

2.

FUNDAMENTAL PROPERTIES OF INTEGRALS

43

Case 3. g ≥ 0: Let gn ≥ 0 be simple with gn g : Then g(X) dµ = lim gn (X) dµ by the MCT, since gn (X) g(X) = lim gn dµX by case 2 (d) by the MCT. = g dµX Case 4. g is measurable, and either g(X)+ dµ or g(X)− dµ is finite: Using g = g + − g − , we note that g(X)+ = g + (X) and g(X)− = g − (X). Then g(X) dµ = g(X)+ dµ − g(X)− dµ = g + (X) dµ − g − (X) dµ = g + dµX − g − dµX by case 3 (e) = g dµX . In the arguments (b), (c), (d), (e) one should start from the end that is assumed to exist, in order to make a logically tight argument. 2

Exercise 2.1 (Only the zero function) Show that (11) X ≥ 0 and X dµ = 0 implies µ( [X > 0] ) = 0 . Exercise 2.2 (Only the zero function) = 0, (12) X dµ = for all A ∈ A ≥0 A

Show that implies

X=

= 0 a.e., ≥ 0 a.e.

Exercise 2.3 Consider a measure space (Ω, A, µ). Let µ0 ≡ µ|A0 for a sub σ-field A0 of A. Starting with indicator functions, show that X dµ = X dµ0 for any A0 -measurable function X.

44

3


Evaluating and Differentiating Integrals

Let (R, Bˆµ , µ) denote a Lebesgue–Stieltjes measure space that has been completed. If g is Bˆµ -measurable, then g dµ is called the Lebesgue–Stieltjes integral ofg; and if F is the generalized df corresponding to µ, then we also use the notation g dF . b Also, a g dF ≡ (a,b ] g dF = 1(a,b ] g dF . Theorem 3.1 (Equality of LS and RS integrals) Let g be continuous on b [a, b]. Then the Lebesgue–Stieltjes integral a g dF equals the Riemann–Stieltjes integral. [And since the LS-integral equals the RS-integral, we can continue to evaluate most LS-integrals using the methods learned in elementary calculus.] Proof. We first recall the classical setup associated with the definition of the RS-integral. Consider any sequence of partitions a ≡ xn0 < · · · < xnn ≡ b such that the partition Xn ≡ {xn0 , xn1 , . . . , xnn } is a refinement of Xn−1 in the sense that Xn−1 ⊂ Xn . Then if meshn ≡ max1≤k≤n (xnk − xn,k−1 ) → 0, and if x∗nk ’s are such that xn,k−1 < x∗nk ≤ xnk , we have (letting gn (a) be defined by right continuity) n (a) gn ≡ k=1 g(x∗nk ) 1(xn,k−1 ,xnk ] → g uniformly on [a, b ], since g is uniformly continuous. Thus for all such sequences the LS-integral of section 3.1 satisfies b b g dF = lim a gn dµ by the DCT, bounded by a constant a n n = lim 1 g(x∗nk ) µ(xn,k−1 , xnk ] = lim 1 g(x∗nk ) F (xn,k−1 , xnk ] (b)

= lim {a Riemann–Stieltjes sum for the integral of g}

(c)

= {the Riemann–Stieltjes integral of g},

and this holds for all partitions and x∗nk ’s as above, provided only that meshn → 0. b Thus the LS-integral a g dF also equals the RS-integral. 2 Exercise 3.1

We state a few additional facts here, just for completeness:

(1)

g is RS-integrable with respect to F g is continuous a.e. µF (·).

if and only if

(2)

If g is RS-integrable with respect to F, b then the RS and LS-integrals a g dF are equal.

Let D(F ) and D(g) denote the discontinuity sets of F and g. Then (3)

g is not RS-integrable when D(F ) ∩ D(g) = ∅.

Exercise 3.2 Suppose that the improper RS-integral of a continuous function b g on R, defined by RS( g dF ) ≡ lima→−∞,b→∞ (RS a g dF ) exists finitely. Then b lima→−∞,b→∞ (LS a |g| dF ) need not be finite. Thus the fact that an improper RS-integral exists does not imply that the function is LS-integrable. Construct an example. 2

3.

EVALUATING AND DIFFERENTIATING INTEGRALS

45

Exercise 3.3 (Differentiation under the integral sign) Suppose that the function X(t, ·) is a measurable function, for each t ∈ [a, b ]. Suppose also that for ∂ a.e. ω the partial derivative ∂t X(t, ω) exists for all t in the nondegenerate interval [a, b], and that ∂ for all t ∈ [a, b ], where Y ∈ L1 . X(t, ω) ≤ Y (ω) ∂t Then (using one-sided derivatives at the end points) d ∂ (4) X(t, ω) dµ(ω) = X(t, ω) dµ(ω) dt Ω Ω ∂t

for all t ∈ [a, b ].

46

4


Inequalities

Convexity We begin by briefly reviewing convexity. A real-valued function f defined on some interval I of real numbers is convex if (1)

f (αx + (1 − α)y) ≤ α f (x) + (1 − α) f (y) for all x, y in I and all 0 ≤ α ≤ 1.

If f is convex on an interval, then f is continuous on the interior of the interval. Also, the left and right derivatives exist and satisfy D− (x) ≤ D+ (x) at each point in the interior of the interval. The following is useful. Convexity on the interval I holds if and only if (2)

f ((x + y)/2) ≤ [f (x) + f (y)]/2 for all x, y in I, provided that f is also assumed to be bounded on I.

[There exist functions satisfying the inequality in (2) that are not continuous, but they are unbounded in every finite interval. Thus requiring (1) for all 0 ≤ α ≤ 1 is strictly stronger then requiring it to hold only for α = 12 .] We need a simple test for convexity, and so note that f is convex if (3)

f (x) is for all x

or

f (x) ≥ 0 for all x .

We call f strictly convex if strict inequality holds in any of the above. If f is convex, then there exists a linear function l such that f (x) ≥ l(x) with equality at any prespecified x0 in the interior of the domain I of f ; this function is called the supporting hyperplane. 2 Definition 4.1 (Moments) [The following definitions make sense on a general measure space (Ω, A, µ), and are standard notation on aprobability space (Ω, A, P ).] Recall from (3.1.6) that Eh(X) = h(X(ω)) dµ(ω) = h(X) dµ = h(X). Let (4)

µ ≡ EX ≡ (the mean of X),

(Note two different uses of µ)

σ2 ≡ Var[X] ≡ E(X − µ)2 = (the variance of X), (5) σ ≡ StDev[X] ≡ (the standard deviation of X) . We will write X ∼ = (µ, σ2 ) if EX = µ and Var[X] = σ2 < ∞. We will also write 2 X∼ = F (µ, σ ) if X also has df F (·). Now let (6)

EX k ≡ (kth moment of X),

for k ≥ 1 an integer,

(7)

E|X|r ≡ (rth absolute moment of X),

for r > 0,

(8)

Xr ≡ {E|X|r }1/r ≡ (rth norm of X),

for r > 0,

(9)

µk ≡ E(X − µ)k ≡ (kth central moment of X),

for k ≥ 1,

(10)

Cov[X, Y ] ≡ E[(X − µX )(Y − µY )] = (the covariance of X and Y ).

Note that Cov[X, X] = Var[X]. (We only need the case when µ(Ω) = 1.)

4.

INEQUALITIES

47

Proposition 4.1 (Ls ⊂ Lr ) We require µ(Ω) < ∞. Then Ls ⊂ Lr whenever 0 < r < s. [So if E|X|r < ∞, then E|X|r and EX k are finite for all 0 ≤ r , k ≤ r.]

Proof. Now, |x|r ≤ 1 + |x|r ; and integrability is equivalent to absolute integrability. Note that µ(Ω) < ∞ was used to claim 1 ∈ L1 . 2 Proposition 4.2 Let µ(Ω) < ∞. Then σ2 < ∞ holds if and only if EX 2 < ∞. In this case σ2 = EX 2 − µ2 . Proof. Suppose EX 2 < ∞. Then EX 2 − µ2 = E(X 2 ) − E(2µX) − E(µ2 ) = 2 E(X − µ) = Var[X]. Note that proposition 4.1 was used for EX. Thus µ(Ω) < ∞ was used. Suppose that σ2 < ∞. Then E{(X − µ)2 + 2µ(X − µ) + µ2 } = EX 2 . 2

Inequality 4.1 (Cr -inequality) E|X + Y |r ≤ Cr E|X|r + Cr E|Y |r , where Cr = 2r−1 for r ≥ 1 and Cr = 1 for 0 < r ≤ 1. Proof. There are no restrictions on µ. Case 1. r > 1: Then |x|r is a convex function of x, since its derivative is ↑ . Thus |(x + y)/2|r ≤ [|x|r + |y|r ]/2; and now take expectations. Case 2. 0 < r ≤ 1: Now, |x|r is concave and for x ≥ 0; just examine derivatives. Thus |x + y|r − |x|r ≤ |y|r , and now take expectations. 2 Inequality 4.2 (H¨ older’s inequality) E|X Y | ≤ E1/r |X|r E1/s |Y |s for r > 1, where s is defined by 1/r + 1/s = 1. Alternatively, X Y 1 ≤ Xr × Y s . When expectations are finite we have equality if and only if there exist constants A and B (not both 0) for which A |X|r = B |Y |s a.e. Proof. The result is trivial if E|X|r = 0 or ∞. Likewise for E|Y |s . So suppose that both expectations are in (0, ∞). Since f (x) = ex is convex, it satisfies (1) with α ≡ 1/r and 1 − α = 1/s, x ≡ r log |a| and y ≡ s log |b| for some a and b; thus (1) becomes (11)

exp( 1r x +

1 s

y) ≤

1 r

ex +

1 s

ey ,

or

Young’s inequality (12)

|ab| ≤

|a|r |b|s + r s

for all a, b

with equality only when |b| = |a|1/(s−1) .

Now let a = |X|/Xr and b = |Y |/Y s , and take expectations. Equality holds if and only if |Y |/Y s = (|X|/Xr )1/(1−s) a.e. (that is, all mass is located at equality in (a)) if and only if s/s−1 |Y |s |X| |X|r (a) = = a.e. E|Y |s Xr E|X|r

48


Equality extends to the initially excluded cases also if and only if there exist A and B (not both 0) such that A |X|r = B |Y |s a.e. This also gives the next inequality as an immediate corollary. 2 Exercise 4.1 (Convexity inequality) (13)

α

u v

1−α

≤ α u + (1 − α)

Show that

for all 0 < α < 1 and all u, v ≥ 0.

Use this to reprove H¨ older’s inequality. Inequality 4.3 (Cauchy–Schwarz inequality) ( E|XY | )2 ≤ EX 2 EY 2 . 2 2 If both √ EX and EY √ take values in (0, ∞), then equality holds if and only if |X| / EX 2 = |Y | / EY 2 a.e. Example 4.1 (Correlation inequality) For rvs X and Y having positive and finite variances, it holds that (14)

−1 ≤ ρ ≤ 1,

for the correlation ρ of X and Y defined by (15)

Cov[X, Y ] ρ ≡ ρX,Y ≡ Corr [X, Y ] ≡ . Var[X] Var[Y ]

2

Exercise 4.2 Consider rvs X and Y . Show that ρ = +1 if and only if X − µX = a (Y − µY ) for some a > 0, ρ = −1 if and only if X − µX = a (Y − µY ) for some a < 0. Thus ρ measures linear dependence, not general dependence. Inequality 4.4 (Liapunov’s inequality) (16)

E1/r |X|r

Let µ(Ω) < ∞. Then

is in r for all r ≥ 0.

Proof. Without loss of generality, set µ(Ω) = 1. Suppose E|X|b < ∞. We use µ(Ω) < ∞ via proposition 4.1 for existence of the [0, b] moments. Let 0 ≤ a ≤ b. Apply H¨ older’s inequality with 1/r, 1/s, X, Y replaced by a/b, (b − a)/b, |X|a , 1 to obtain E( |X|a · 1) ≤ ( E|X|b )a/b ( E 1)(b−a)/b . This gives ( E|X|a )1/a ≤ ( E|X|b )1/b for 0 ≤ a ≤ b.

2

Exercise 4.3 (Littlewood’s inequality) Let mr ≡ E|X|r denote the rth absolute moment. Then for r ≥ s ≥ t ≥ 0 we have ms−t mr−s ≥ msr−t . In particular, t r (17)

m4 m21 ≥ m32 .

4.

INEQUALITIES

49

Inequality 4.5 (Minkowski’s inequality) for all r ≥ 1.

E1/r |X + Y |r ≤ E1/r |X|r + E1/r |Y |r

Proof. This is trivial for r = 1. Suppose r > 1, and note that s = r/(r − 1). Then for any measure µ we have E{|X + Y |r } ≤ E{|X| |X + Y |r−1 } + E{|Y | |X + Y |r−1 }

(a)

≤ ( Xr + Y r ) |X + Y |r−1 s

by H¨ older’s inequality

= (Xr + Y r ) E1/s |X + Y |(r−1)s = (Xr + Y r ) E1/s |X + Y |r . If E|X + Y |r = 0, the result is trivial. If not, we divide to get the result. Inequality 4.6 (Basic inequality) all measurable X we have (18) Proof. (a) (b)

2

Let g ≥ 0 be on [0, ∞) and even. Then for

µ( |X| ≥ λ ) ≤ E g(X)/g(λ)

for all λ > 0 .

Now,

E g(X) = [ |X|≥λ] g(X) dµ + [ |X| 0.

Inequality 4.8 (Chebyshev’s inequality) (19)

µ( |X − µ| ≥ λ) ≤ Var[X]/λ2

for all λ > 0 . Any rv X ≥ 0 with finite mean satisfies

Inequality 4.9 (Paley–Zygmund) (20) Proof.

P (X > λ) ≥ [(EX − λ) ] /EX + 2

2

for each λ > 0 .

Now, EX = E(X 1[X≤λ]) + E(X 1[X>λ]) ≤ λ +

E(X 2 ) P (X > λ)

by Cauchy–Schwarz. Rearranging gives the inequality.

2

Inequality 4.10 (Jensen’s inequality) Suppose g is convex on (a, b), where −∞ ≤ a < b ≤ ∞. Then if µ(X ∈ (a, b)) = µ(Ω) = 1 and if EX is finite (and hence a < EX < b), then (21)

g( EX ) ≤ Eg(X) .

For strictly convex g we have equality if and only if X = EX a.e.

50 Proof. (a)

CHAPTER 3. INTEGRATION Let l(·) be a supporting hyperplane to g(·) at EX. Then Eg(X) ≥ El(X)

(b)

= l(EX)

since l(·) is linear and µ(Ω) = 1

(c)

= g(EX)

since g(·) = l(·) at EX .

Now g(X) − l(X) ≥ 0. Thus Eg(X) = El(X) if and only if g(X) = l(X) a.e. µ if and only if X = EX a.e. µ . 2

Inequality 4.11 (Bonferroni) n n n i =j P (Ai Aj ) i=1 P (Ai ) ≥ P (∪i=1 Ai ) ≥ i=1 P (Ai ) − for any collection of events Ak on a probability space (Ω, A, P ). ˜ equal Exercise 4.4 Suppose E|X| < ∞, and let µ ≡ EX. Fix c < d. Let X ˜ Show that c, X, d according as [X < c], [c ≤ X ≤ d], [d < X], and set µ ˜ ≡ EX. ˜ −µ E|X ˜|r ≤ E|X − µ|r for all r ≥ 1. Exercise 4.5 (Hardy) Suppose h ∈ L2 (0, 1) and define T h(u) = u−1 for 0 < u ≤ 1. Use the Cauchy–Schwarz inequality to show that 1 1 {T h(u)}2 du ≤ 4 0 h2 (u) du. 0 u [Hint: Write T h(u) = u−1 0 h(s)sα s−α ds for some α.]

u 0

h(s) ds

Exercise 4.6 (Wellner) Suppose that T ∼ = Binomial(n, p). [Thus P (T = k) = n k n−k p (1 − p) for 0 ≤ k ≤ n, and the measure associated with T has mean np k and variance np(1 − p).] Then use inequality 4.6 with g(x) = exp(rx) and r > 0, to show that (22)

P (T /n ≥ p ) ≤ exp(−np h( )),

where h( ) ≡ (log( ) − 1) + 1. Exercise 4.7 (Geometric mean) whenever all xk ≥ 0.

Show that (x1 ×· · ·×xn )1/n ≤ (x1 +· · ·+xn )/n

5.

MODES OF CONVERGENCE

51

5

Modes of Convergence

Definition 5.1 (Modes of convergence) Let X and Xn ’s be measurable and a.e. finite from the measure space (Ω, A, µ) to (R, B). (a) Recall that Xn converges a.e. to X (denoted by Xn →a.e. X) if (1)

Xn (ω) → X(ω) for all ω ∈ A,

where µ(Ac ) = 0.

(b) Also, recall that Xn converges in measure to X (denoted by Xn →µ X) if (2)

µ( [ ω : |Xn (ω) − X(ω) | ≥ ] ) → 0

for each > 0.

(c) Now (rigorously for the first time), Xn converges in rth mean to X (denoted by Xn →r X or Xn →Lr X) if (3)

E|Xn − X|r → 0

for Xn ’s and X in Lr ;

here, r > 0 is fixed. [Note from the Cr -inequality that if Xn − X and one of X or Xn is in Lr , then the other of Xn or X is also in Lr .] Recall from chapter 2 that Xn →a.e. (some a.e. finite X) holds if and only if Xn − Xm →a.e. 0 as m ∧ n → ∞ . Likewise, in chapter 2 we had Xn →µ (some X) if and only if µ( |Xn − Xm | ≥ ) → 0 as m ∧ n → ∞ for all > 0. Exercise 5.1 (Completeness of Lr ) (a)

(I) Let Xn ’s be in any Lr , for r > 0.

(Riesz–Fischer) Xn →r (some X ∈ Lr ) if and only if Xn − Xm →r 0.

That is, Lr is complete with respect to →r . Prove (a), using (2.3.16). Then show that (Lr , · r ) is a complete and separable metric space (provided that we identify X and X whenever X = X a.e.). (II) Let µ(Ω) < ∞. Then: (b)

If Xn →r X, then Xn →r X for all 0 < r ≤ r.

(c)

Show by example that Xn →r X does not imply that Xn →a.e. X.

(d)

Show by example that Xn →a.e. X does not imply that Xn →L1 X.

[Hint: Use Fatou’s lemma in (a) and H¨ older’s inequality in (b).] Summary Let X and Xn ’s be as in definition 5.1. Then

(4)

Xn converges a.e., in measure, or in Lr to some X if and only if Xn is Cauchy a.e., in measure, or in Lr .

52


Consequences of Convergence in Distribution on (Ω, A, P ) Notation 5.1 Suppose now that µ really denotes a probability measure, and so we will label it P . Recall that Xn converges in distribution to X (denoted by Xn →d X, Fn →d F or L(Xn ) → L(X) with L(·) referring to “law”) when the dfs F and Fn of X and Xn satisfy (5)

Fn (x) → F (x)

as n → ∞

for each continuity point x ∈ CF of F (·) .

[Note that Fn ≡ 1[1/n,∞) →d F ≡ 1[0,∞) , even though Fn (0) = 0 → 1 = F (0).] The statement →d will carry with it the implication that F corresponds to a probability measure P , which can be viewed as the PX = µX of an appropriate rv X. Theorem 5.1 (Helly–Bray) Consider some (Ω, A, P ). Suppose Fn →d F , and suppose that g is bounded and is continuous a.s. F . Then (6) E g(Xn ) = g dFn → g dF = E g(X) . Conversely, Eg(Xn ) → Eg(X) for all bounded, continuous g implies that Fn →d F . Theorem 5.2 (Mann–Wald) Consider some (Ω, A, P ). Suppose Xn →d X, and let g be continuous a.s. F . Then g(Xn ) →d g(X). Proof. We really leave this proof to the next exercise, but we also give a “look-ahead” proof here. (See theorem 7.3.2 below.) Skorokhod theorem If Xn →d X, there are Y and Yn on some (Ω, A, P ) having (7)

Yn ∼ = Xn and Y ∼ = X and especially

Yn → Y a.s. PX (·) .

Note that A1 ≡ {ω : Yn (ω) → Y (ω)} has P (A1 ) = 1. Also, (a)

P (A2 ) ≡ P ({ω : g is continuous at Y (ω)})

(b)

= PY ({y : g is continuous at y} ) = 1.

Thus A ≡ A1 ∩ A2 has P (A) = 1. Especially, (c)

g(Yn (ω)) → g(Y (ω))

for all ω ∈ A,

with P (A) = 1 .

Since g is bounded, applying the DCT to (7) gives the Helly–Bray claim that (d) g dFn = g(Yn ) dP → g(Y ) dP = g dF. We note additionally that since (7) implies g(Yn ) →a.s. g(Y ), it also implies g(Yn ) →d g(Y ). Since g(Xn ) ∼ = g(Yn ) and g(X) ∼ = g(Y ), we can conclude that g(Xn ) →d g(X). This argument did not use the boundedness of g, and so proves the Mann–Wald theorem. [The Helly–Bray theorem will be used later in this section (in proving Vitali’s theorem). Proving it as indicated in the next exercise would have been possible now, but the proof based on Skorokhod’s theorem is more in keeping with the spirit of this book.]

5.


53

Consider the converse. Let g (·) equal 1, be linear, equal 0 on (−∞, x − ], on [x − , x], on [x, ∞); and let h (·) equal 1, be linear, equal 0 on (−∞, x], on [x, x + ], on [x + , ∞). Then F (x − ) ≤ E g (X) = lim E g (Xn ) ≤ lim Fn (x) ≤ lim Fn (x) ≤ lim E h (Xn ) = lim E h (X) ≤ F (x + ),

(e)

so that Fn (x) → F (x) at all continuity points of F . Definition 5.2 (Limit-determining class) continuous functions g on the line. If Eg(Xn ) → Eg(X) for all g ∈ G

2

Let G denote a class of bounded Xn →d X ,

implies

then call G a limit-determining class. [The proof of the converse half of Helly–Bray exhibited one such class of particularly simple functions. (See section 11.8.)] Exercise 5.2 Prove the Mann–Wald and Helly–Bray theorems for all bounded continuous g, without appeal to theorem 7.3.2 of Skorokhod. [Truncate the real line at large continuity points ±M of F , and then use the uniform continuity of g on the interval [−M, M ] to obtain a simple proof in this special case of bounded and continuous functions g.]

General Moment Convergence on (Ω, A, µ) Theorem 5.3 (Moment convergence under →r )

E|Xn |r → E|X|r

(8) Proof.

and EXnk → EX k

for all 0 ≤ k, r ≤ r.

Let (Ω, A, µ) be arbitrary and 0 < r < 1. The Cr -inequality gives E|Xn |r ≤ E|Xn − X|r + E|X|r

(a)

If Xn →r X, then

and E|X|r ≤ E|X − Xn |r + E|Xn |r ,

so that | E|Xn |r − E|X|r | ≤ E|Xn − X|r → 0

(9)

when 0 < r ≤ 1.

Suppose r ≥ 1. Then using Minkowski’s inequality twice (as in (a)) gives 1/r (10) when r ≥ 1. E |Xn |r − E1/r |X|r ≤ E1/r |Xn − X|r → 0, Combining (9) and (10) shows that E|Xn |r → E|X|r . Apply exercise 5.1(b) for r . Now, |Xn+ −X + | equals |Xn −X|, |Xn −0|, |0−X|, |0−0| just as [Xn ≥ 0, X ≥ 0] [Xn ≥ 0, X < 0], [Xn < 0, X ≥ 0], [Xn < 0, X < 0]. Thus (11)

|Xn+ − X + | ≤ |Xn − X|,

and

|Xn− − X − | ≤ |Xn − X|

also. Hence Xn+ →r X + and Xn− →r X − , so that E(Xn+ )r → E(X + )r and E(Xn− )r → E(X − )r . Cross-product terms are 0, since X + (ω)X − (ω) = 0, so (b)

EXnk = E(Xn+ )k + (−1)k E(Xn− )k → E(X + )k + (−1)k E(X − )k = E(X k ) . 2

54


Uniform Integrability and Vitali’s Theorem Definition 5.3 (Uniformly integrable) A collection of measurable Xt ’s is called integrable if sup t E|Xt | < ∞. Further, a collection of rvs {Xt : t ∈ T } is said to be uniformly integrable (and is abbreviated u.i.) if (12)

sup E[ |Xt | × 1[ |Xt |≥λ ] ] → 0

as λ → ∞ .

t∈T

Remark 5.1 Suppose that |Xt | ≤ Y a.s. for some Y ∈ L1 . Then for some null set Nt , we have [|Xt | ≥ λ] ⊂ [|Y | ≥ λ] ∪ Nt , and µ(|Xt | ≥ λ) ≤ µ(|Y | ≥ λ) → 0 uniformly in t as λ → ∞ (use Markov’s inequality). Also, supt∈T E|Xt | ≤ E|Y |. Finally, for each fixed t, |Xt | dµ ≤ [ |Y |≥λ ] Y dµ → 0 uniformly in t as λ → ∞ [ |Xt |≥λ ] by the absolute continuity of the integral of Y in theorem 3.2.5 and by Markov’s inequality. Thus, (13)

if |Xt | ≤ Y for some Y ∈ L1 ,

then the Xt ’s are uniformly integrable .

Exercise probability measure P . (a) Let X ≥ 0 have df F . Then ∞ 5.3 Consider a∞ EX = 0 P (X ≥ y) dy = 0 [1 − F (y)] dy will be shown to hold in (7.4.11) below. For now, we shall just use this fact. (The claimed formula can also be established for simple functions by summing by parts; and then the full claim follows from the MCT. This cumbersome proof is possible now. It constitutes this exercise.) (b) Use the result of (a) to show that for X ≥ 0 and λ ≥ 0 we have ∞ X dP = λ P (X ≥ λ) + λ P (X ≥ y) dy. [X≥λ] (c) Suppose there is a Y ∈ L1 such that P ( |Xn | ≥ y) ≤ P ( Y ≥ y) for all y > 0 and all n ≥ 1. Then use (b) to show that {Xn : n ≥ 1} is uniformly integrable. Exercise 5.4 (Uniform integrability criterion) If supt E|Xt |r < ∞ for some r > 1, then the Xt ’s are uniformly integrable. (Compare this to theorem 5.6 of de la Vallée Poussin below, by letting G(x) = xr .) Theorem 5.4 (Uniform absolute continuity of integrals) Let µ(Ω) < ∞. A family of measurable Xt ’s is uniformly integrable if and only if both (14)

sup E|Xt | < ∞

(the collection is integrable)

t

(15)

µ(A) < δ

implies

(a)

|Xt | dµ <

sup t

Proof.

and

(uniform absolute continuity).

A

Suppose (12) holds. Then for λ large enough, (12) gives E|Xt | ≤ λ µ(Ω) + [ |Xt |≥λ ] |Xt | dµ ≤ λ µ(Ω) + 1

for all t;

5.


55

thus the collection is integrable. Suppose µ(A) < δ. Then A |Xt | dµ = A |Xt | × 1[ |Xt | 0. Suppose that Xn ’s ∈ Lr satisfy Xn →µ X. The following are equivalent when µ(Ω) < ∞: (16)

{|Xn |r : n ≥ 1} are uniformly integrable rvs.

(17)

Xn →r X.

(18)

E|Xn |r → E|X|r .

(19)

lim E|Xn |r ≤ E|X|r < ∞.

Exercise 5.5 (a) We need only omit “(16) implies (17)” in Vitali’s theorem when we instead have µ(Ω) = ∞. (b) Give an example to demonstrate the previous claim. (c) Show that t ∈ [0, ∞) may replace n ∈ {0, 1, 2, . . .} in Vitali’s theorem. Remark 5.2 (Vitali’s theorem in L1 ) Let r = 1, µ(Ω) < ∞, and Xn →µ X. Then (16) – (19) are equivalent by Vitali’s theorem. As a preliminary (from Fatou’s lemma and Xn →a.e. X on some subsequence n ), (20)

E|X| = E| lim Xn | = E lim |Xn | ≤ lim E|Xn | ≤ lim E|Xn | ≤ lim E|Xn |

always holds. Thus lim sup E|Xn | ≤ E|X| < ∞ implies that E|Xn | → E|X| < ∞. Hence E|Xn − X| → 0

if and only if

E|Xn | → E|X|

if and only if

(21) {|Xn | : n ≥ 1} are uniformly integrable rvs.

2

Corollary 1 (L1 -convergence) Let X and Xn ’s be measurable and a.e. finite. Let µ(Ω) < ∞. Then Xn →1 X if and only if both Xn →µ X and {|Xn | : n ≥ 1} are uniformly integrable rvs.

56


Proof. Suppose (16) holds. Now, Xn →a.s. X for some subsequence by theorem 2.3.1. Thus E|X|r = E(lim inf |Xn |r ) ≤ lim inf E|Xn |r < ∞ using Fatou’s lemma and then (14). Thus X ∈ Lr . Now, the Cr -inequality gives |Xn − X|r ≤ Cr {|Xn |r + |X|r }, so that the rvs |Xn − X|r are clearly uniformly integrable. Thus E|Xn − X|r = E{|Xn − X|r × 1[ |Xn −X|> ] } + E{|Xn − X|r × 1[ |Xn −X|≤ ] } ≤ + r × µ(Ω)

(a)

for n large enough;

the in (a) is from (15), since µ(|Xn − X| ≥ ) → 0 by hypothesis. Thus (17) holds. Now, (17) implies (18) by theorem 5.3. Also (18) trivially implies (19). Suppose (19) holds. Define fλ to be a continuous function on [0, ∞) that equals |x|r , 0, or is linear, according as |x|r ≤ λ, |x|r ≥ λ + 1, or λ ≤ |x|r ≤ λ + 1. Then (graphing fλ (x) and xr on [0, λ + 1]) limn [ |Xn |r >λ+1] |Xn |r dµ = limn { |Xn |r dµ − [ |Xn |r ≤λ+1] |Xn |r dµ} (b) ≤ E|X|r − limn [ |Xn |r ≤λ+1] |Xn |r dµ by (19) ≤ E|X|r − lim E fλ (Xn )

(d)

= E|X|r − E fλ (X) ≤ [ |X|≥λ ] |X|r dµ

(e)

→0

(c)

implying (16).

by the Helly–Bray theorem

as λ → ∞ since X ∈ Lr , 2

We now generalize exercise 5.4. In fact, we are able to obtain a characterization of uniform integrability. Theorem 5.6 (de la Vallée Poussin) Let µ(Ω) < ∞. A family of L1 -integrable functions Xt is uniformly integrable if and only if there exists a convex function G on [0, ∞) for which G(0) = 0, G(x)/x → ∞ as x → ∞ and (22) Proof. (a)

supt E G(|Xt |) < ∞ .

For λ so large that G(x)/x ≥ c for all x ≥ λ we have |Xt | dµ ≤ 1c [ |Xt |≥λ ] G(|Xt | ) dµ ≤ 1c supt EG(|Xt | ) <

[ |Xt |≥λ ]

for c sufficiently large. Thus (22) implies {Xt : t ∈ T } is uniformly integrable. Now we show that {Xt : t ∈ T } u.i. implies (22) for some G. We define x G(x) = 0 g(y) dy where (with a sequence bn having b0 = 0, to be specified below) we set g(x) = bn for all n ≤ x < n + 1, n ≥ 0. Define an (t) ≡ µ( |Xt | ≥ n). We note that

(b)

E G( |Xt | ) ≤ b1 µ(1 ≤ |Xt | < 2) + (b1 + b2 ) µ(2 ≤ |Xt | < 3) + · · · = ∞ n=1 bn an (t).

5.


57

∞ It thus suffices to choose bn ∞ such that supt 1 bn an (t) < ∞. By the definition of uniform integrability, we can choose integers cn ↑ ∞ such that (c) supt [ |Xt |≥cn ] |Xt | dµ ≤ 1/2n . Thus for all t we have ∞ 1/2n ≥ [ |Xt |≥cn ] |Xt | dµ ≥ i=cn i µ(i ≤ |Xt | < i + 1) = ≥ (d)

=

∞ i=cn

∞ j=cn

∞ j=cn

i j=1

∞ i=j

µ(i ≤ |Xt | < i + 1 ) µ(i ≤ |Xt | < i + 1) =

∞ j=cn

µ( |Xt | ≥ j )

aj (t).

Thus, interchanging the order of summation, ∞ ∞ −n (e) 1= ∞ ≥ supt ∞ n=1 2 n=1 j=cn aj (t) = supt j=1 bj aj (t) for bj ≡ (the number of integers n such that cn is ≤ j). As was noted above, the proof is given by (e). 2 Exercise 5.6 (a) Let ξ ∼ = Uniform(0, 1), and let Xn ≡ (n/ log n)1[0,1/n] × (ξ) for n ≥ 3. Show that these Xn are uniformly integrable and Xn dP → 0, even though these rvs are not dominated by any fixed integrable rv Y . (b) Let Yn ≡ n1[0,1/n] (ξ) − n1[1/n,2/n](ξ). Show that these Yn are not uniformly integrable, but that Yn dP → 0.

Summary of Modes of Convergence Results Theorem 5.7 (Convergence implications) be measurable and a.e. finite. (Note figure 5.1.)

Let 0 < r ≤ r. Let X and Xn ’s

(i) If Xn →a.e. X and µ(Ω) < ∞, then Xn →µ X. (ii) If Xn →µ X, then Xn →a.e. X on some subsequence n . (iii) If Xn →r X, then Xn →µ X and {|Xn |r : n ≥ 1} are uniformly integrable. (iv) If Xn →µ X and {|Xn |r : n ≥ 1} are uniformly integrable rvs, then Xn →r X. If Xn →µ X and lim E|Xn |r ≤ E|X|r < ∞, then Xn →r X. (v) If Xn →r X and µ(Ω) < ∞, then Xn →r X. (vi) If Xn →p X, then Xn →d X. (vii) If µ(Ω) < ∞, then Xn →µ X if and only if every subsequence {n } contains a further subsequence {n } for which Xn →a.e. X. (viii) If Xn →d X, then Yn →a.e. Y for Skorokhod rvs with Yn ∼ = Xn and Y ∼ = X.

58


5

Xn →r X

|Xn|r u.i. 4

Xn →a.e. X

1 if µ(Ω) < ∞

6

if µ(Ω) < ∞

Xn →r′ X for all r′ ≤ r

3

2

X n →µ X

if µ = P

7

Xn →d X

Some Xn′ →a.e. X

if µ(Ω) < ∞

Every n′ contains an n˝ for which Xn˝ → a.e. X

Figure 5.1 Convergence implications.

Proof.

See theorem 2.3.1 for (i) and (ii). Markov’s inequality gives µ( |Xn − X| ≥ ) ≤ E|Xn − X|r / r → 0,

and then Vitali’s theorem gives (iii). Vitali’s theorem also gives (iv). Liapunov’s inequality gives (a)

E|Xn − X|r ≤ {E|Xn − X|r (r/r ) }r /r = {E|Xn − X|r }r /r ,

as claimed in exercise 5.1(b); so (v) holds. Proposition 2.4.1 gives (vi). And then theorem 2.3.1 gives (vii). The Skorokhod construction of theorem 7.3.2 was stated below in (7); and (7) gives (viii). 2

5.


59

Approximation of Functions in Lr by Continuous Functions Let Cc denote the class of continuous functions on R that vanish outside a compact (∞) set, and then let Cc denote the subclass that has an infinite number of continuous derivatives. Let Sc denote m the class of all step functions on R, where such a step function is of the form 1 yj 1Ij for disjoint finite intervals Ij . Further, let F denote a generalized df, and let µ(·) ≡ µF (·) denote the associated Lebesgue–Stieltjes measure. Let X denote a rv on (Ω, A) = (R, B). Theorem 5.8 (The continuous functions are dense in Lr , r ≥ 1) Suppose throughout that X ∈ Lr , for some r ≥ 1. (a) (Continuous functions) Then there exists a bounded and continuous function Y in Cc for which |X − Y |r dµF < , for each > 0. Thus the class Cc is -dense within the class of integrable functions Lr . (∞) (b) We may even insist that the function Y belong to Cc . (c) (Step functions) Such a close approximation may also be found within the step functions Sc , making them -dense also. (d) All this extends to rvs on (Ω, A) = (Rn , Bn ). + Proof. Let r = 1 first. We need only consider X ; and we approximate it by a n simple function X = 1 xi 1Ai so closely that |X −X | dµF < /3. For each such xi > 0 we must have µ(Ai ) < ∞, since X dµF < ∞. Now, the approximation lemma of exercise 1.2.3 guarantees sets B1 , . . . , Bn made up of a finite disjoint union of intervals of the form (a, b] for which

µ(Ai Bi ) < /(3n |xi |), (p)

X ≡

n 1

xi 1Bi satisfies

and so

|X − X | dµF < /3.

m Rewrite X = 1 yj 1Cj with C j = (aj , bj ]. Then X is the step function called m for in part (c). Define δ ≡ /(3 1 yj ). Now approximate 1Cj by the continuous function Yj that equals 1, is linear, equals 0 according as x ∈ [aj , bj ], as x ∈ [aj − δ, aj ]∪ [bj , bj + δ], as x ∈ (−∞, aj − δ] ∪ [bj + δ, ∞). Then let Y ≡ m 1 Yj , which has |X − Y | dµF < /3. Thus |X − Y | dµF < , as called for in part (a). For (b) we consider the function ψ(x/δ) [where 1 (23)

ψ(x) ≡ x1 0

exp(−1/((s(1 − s))) ds exp(−1/((s(1 − s))) ds

for 0 ≤ x ≤ 1,

with ψ(x) equal to 1 or 0 according as x ≤ 0 or x ≥ 1] is easily seen to have an infinite number of continuous derivatives on R. Use it on [aj − δ, aj ] and [bj , bj + δ] (∞) to connect 0 to 1, instead of linear connections. The result is a function in Cc . For general r ≥ 1 we need only know the trivial fact that |a − b|r ≤ |ar − br | for all a, b ≥ 0 and r ≥ 1. Just note example 5.1.1(d) below for part (d). 2

60


Miscellaneous Results Exercise e’s theorem) (a) If f0 , f1 , f2 , . . . are ≥ 0 on (Ω, A, µ) and 5.7 (Scheff´ satisfy Ω fn dµ = 1, and fn →a.e. f0 with respect to µ, then (24) f0 dµ → 0 as n → ∞. sup fn dµ − A∈A

A

A

[The name derives from pointing out its usefulness in probability theory.] (We can think of this conclusion as the uniform convergence of measures.) (b) In fact, we require only that Ω fn dµ → Ω f0 dµ = 1 and fn →µ f0 . [Hint. Note Vitali’s theorem.] Exercise 5.8 (→a.u. ) (i) We define Xn →a.u. X (which is an abbreviation for almost uniform convergence) to mean that for all > 0 there exists an A with µ(A ) < such that Xn →uniformly X on Ac . (ii) (Egorov) If µ(Ω) < ∞ and Xn →a.s. X, then Xn →a.u. X. If |Xn | ≤ Y a.s. for all n where Y ∈ Lr with r > 1, then we may omit the hypothesis µ(Ω) < ∞. (iii) (a) If Xn →a.u. X, then both Xn →a.e. X and Xn →µ X. (b) If Xn →µ X, then Xn →a.u. X on some subsequence n . Exercise 5.9 (Lusin) Let X be measurable on (Ω, A, µ), where Ω is a locally compact Hausdorff space [every point has a neighborhood whose closure is compact, such as the real line R with the usual Euclidean metric] and A = σ[{open sets}]. Suppose X(ω) = 0 for all ω ∈ A, where µ(A) < ∞. Let > 0. Then there exists Y , where Y (ω) = 0 for all ω ∈ B, with the set B compact, and where Y is continuous, sup |Y | ≤ sup |X|, and µ({ω : X(ω) = Y (ω)}) < . [That is, a measurable function is “almost equal” to a continuous function.] Exercise 5.10 (Loève) Suppose X1 , X2 , . . . are integrable on (Ω, A, µ). Define φn (A) ≡ A Xn for all A ∈ A, and suppose φn (A) converges to a finite number for all A ∈ A. Define |φ|n (A) ≡ A |Xn |. Then supn |φ|n (Ω) < ∞. Moreover, sup |φ|n (A) → 0 n

as either µ(A) → 0 or A 0.

Finally, there exists an integrable function X (that is unique a.e. µ) for which φn (A) → φ(A) for all A ∈ A, with φ(A) ≡ A X. (Relate this to the absolute continuity of measures introduced below.)

Chapter 4

Derivatives via Signed Measures 1

Decomposition of Signed Measures

Definition 1.1 (Signed measure) A signed measure on a σ-field field) A (or a is a set function φ : A → (−∞, +∞] for which φ(∅) = 0 and φ( An ) = φ(An ) for all countable disjoint sequences of An ’s in A (that have An in A ). When additivity is required only on finite unions, then φ is called a finitely additive (f.a.) signed measure. (If φ≥ 0, it is a measure or f.a. measure.) If |φ(Ω)| < ∞, then φ is ∞ called finite. If Ω = 1 Ωn with all Ωn ∈ A and all |φ(Ωn )| < ∞, then φ is called σ-finite. Proposition 1.1 (Elementary properties) (i) If φ(A) is finite and B ⊂ A, then φ(B) is finite. ∞ Thus φ(Ω) finiteis∞equivalent to φ(A) being finite for all A ∈ A. (ii) If |φ( 1 An )| < ∞, then 1 φ(An ) is absolutely convergent. Proof. (a)

(i) Now, (a finite number ) = φ(A) = φ(B) + φ(A \ B)

implies that φ(B) and φ(A \ B) are both finite numbers. φ(An ) is≥ 0 or < 0. Let A− An or (ii) Let A+ n equal An or n equal ∅ as + + ∅ as + φ(An ) is ≤0 or > 0. Then φ(A ) = φ( A ) < ∞ by (i), since A ⊂ An . n n − n Likewise, φ(A− ) = φ( A ). Now, convergent series of numbers in [0, ∞) may n n be rearranged at will. Thus |φ(An )| = φ(A+ φ(A− 2 n)− n ) is finite. Example 1.1 (The prototypical example) Let X be measurable. Then (1) φ(A) ≡ X dµ is a signed measure if X − ∈ L1 . A

Note that φ is finite if X ∈ L1 . Also, φ is σ-finite if X is a.e. finite and µ is σ-finite. 61

62 Proof. (a)

CHAPTER 4. DERIVATIVES VIA SIGNED MEASURES Now, φ(∅) = 1∅ dµ = 0 dµ = 0. Also, φ(A) = A X + dµ − A X − dµ ≥ − A X − dµ > −∞

for all A ∈ A. Finally, ∞ φ( 1 An ) = Σ An X = Σ An X + − Σ An X − + − = by the MCT, twice An X − An X ∞ (b) = ( An X + − An X − ) = 1 φ(An ). An X = Thus φ is a signed measure. Note that|φ(A)| = | A X| ≤ A |X| ≤ |X| < ∞ for all A, if X ∈ L1 . Let Ω ≡ n Ωn be a measurable decomposition for the σ-finite µ. Then the sets Ωnm ≡ Ωn ∩ [m ≤ X < m + 1] and Ωn,±∞ ≡ Ωn ∩ [x = ±∞], for n ≥ 1 and for all integers m, is a decomposition showing φ to be σ-finite. 2 Definition 1.2 (Continuous signed measure) A signed measure φ is continuous from below (above) if φ( lim An ) = lim φ(An ) for all An (for all An , with at least one φ(An ) finite). We call φ continuous in case it is continuous both from below and from above. Proposition 1.2 (Monotone property and continuity of signed measures) A signed measure on either a field or a σ-field is finitely additive and continuous. Conversely, if a finitely additive signed measure on either a field or σ-field is either continuous from below or is finite and continuous from above at ∅, then it is a countably additive signed measure. Proof. This result has nearly the same proof as does the corresponding result for measures; see proposition 1.1.2 and proposition 1.1.3. 2 Theorem 1.1 (Jordan–Hahn) Let φ be a signed measure on (Ω, A). Then Ω can be decomposed as Ω = Ω+ + Ω− , where (2)

Ω+ is a positive set for φ, in that φ(A) ≥ 0 for all A ⊂ Ω+ ,

(3)

Ω− is a negative set for φ, in that φ(A) ≤ 0 for all A ⊂ Ω− .

Moreover, we define measures on (Ω, A) via (4)

φ+ (A) ≡ φ(A ∩ Ω+ )

and

φ− (A) ≡ −φ(A ∩ Ω− ) ,

with φ+ a measure and φ− a finite measure on (Ω, A). Of course, φ+ (Ω− ) = 0 and φ− (Ω+ ) = 0. We will call φ+ , φ− , and |φ|(·)| ≡ φ+ + φ− the positive part, the negative part, and the total variation measure associated with φ; thus (5)

|φ(·)| ≡ φ+ (·) + φ− (·)

is the total variation measure on (Ω, A),

Moreover, the following relationships hold: (6)

φ+ (A) ≡ sup{φ(B) : B ⊂ A, B ∈ A} , φ− (A) ≡ − inf{φ(B) : B ⊂ A, B ∈ A}.

1.

DECOMPOSITION OF SIGNED MEASURES

Exercise 1.1

63

If φ and ψ are signed measures, then so is φ + ψ.

Exercise 1.2 Identify φ+ , φ− , |φ|, and |φ(Ω)| in the context of the prototypical situation of example 1.1. Proof. Let B denote some set having φ(B) < 0. [That φ(B) > −∞ is crucial; this proof will not work on the positive side.] (If no such set exists, let Ω+ ≡ Ω, giving |φ| = φ+ = φ and φ− ≡ 0.) We now show that (a)

B contains a negative set C.

If B is a negative set, use it for C. If not, then we will keep removing sets Ak with φ(Ak ) > 0 from B until only a negative set C is left. We will choose sets Ak with φ(Ak ) ≥ 1 as many times as we can, then φ(Ak ) ≥ 12 as many times as we can, . . . . To this end, let n1 ≡ min{i : φ(A1 ) ≥ 1/i for some A1 ⊂ B, with A1 ∈ A}, ··· (b) nk ≡ min{i : φ(Ak ) ≥ 1/i for some Ak ⊂ B \ ∪k−1 i=1 Ai , with Ak ∈ A} ··· . Let C ≡ B \ k Ak , where the union is infinite (unless the process of choosing nk ’s terminates) and where only finitely many sets Ak exist for each 1/i [else proposition 1.1(i) would be violated]. The c.a. of φ then gives (c) 0 > φ(B) = φ(C) + k φ(Ak ) ≥ φ(C) > −∞. Moreover, C is a negative set, since no subset can have measure exceeding 1/i for any i. Now we know that we have at least one negative set. So we let (d)

d ≡ inf{φ(C) : C is a negative set } < 0,

and define Ω− ≡ ∪k Ck ,

where Ck denotes a sequence of negative sets for which φ(Ck ) d. Now, Ω− is also a negative set (else one of the Ck ’s would not be), and thus φ(Ω− ) ≥ d, because it must exceed the infimum of such values. But φ(Ω− ) ≤ d also holds, since φ(Ω− ) = φ(Ck ) + φ(Ω− \ Ck ) ≤ φ(Ck ) for all k gives φ(Ω− ) ≤ d. Thus φ(Ω− ) = d. Then Ω+ is a positive set, since if φ(A) < 0 for some A ⊂ Ω+ , then the set Ω− ∪ A would have φ(A ∪ Ω− ) < d (which is a contradiction). 2 + Exercise 1.3 The set Ω+ is essentially unique, in that if Ω+ 1 and Ω2 both satisfy + + the theorem, then |φ|(Ω1 Ω2 ) = 0.

Lebesgue Decomposition Definition 1.3 (Absolute continuity of measures) Let µ and φ denote a measure and a signed measure on a σ-field A. Call φ absolutely continuous with respect to µ, denoted by φ µ, if φ(A) = 0 for each A ∈ A having µ(A) = 0. We say φ is singular with respect to µ, denoted by φ ⊥ µ, if there exists a set N ∈ A for which µ(N ) = 0 while |φ|(N c ) = 0.

64

CHAPTER 4. DERIVATIVES VIA SIGNED MEASURES

Theorem 1.2 (Lebesgue decomposition) Suppose the measure µ and the signed measure φ are σ-finite on the space (Ω, A). Then there exists a unique decomposition of φ as (7)

φ = φac + φs

where φac µ and φs ⊥ µ,

with φac and φs σ-finite measures. Moreover, φac (A) = (8) Z0 dµ for all A ∈ A A

for some finite A-measurable function Z0 , which is unique a.e. µ. Proof. By σ-finiteness and the Jordan–Hahn decomposition, we need only − give the proof if µ and φ are finite measures; just consider φ+ Ωn and φΩn separately ∞ (n = 1, 2, . . .) of the σ-finite decomposition Ω = 1 Ωn . (To give the details would be pedantic.) We now establish the existence of the decomposition in the reduced problem when φ and µ are finite measures. Let (a) Z ≡ {Z : Z ≥ 0, Z ∈ L1 and A Z dµ ≤ φ(A) for all A ∈ A}. Now, Z = ∅, since Z ≡ 0 is in Z. Case 1: φ µ. The first step is to observe that (b)

Z1 , Z2 ∈ Z implies Z1 ∨ Z2 ∈ Z.

With A1 ≡ {ω ∈ A : Z1 (ω) > Z2 (ω)} and A2 ≡ AAc1 , we have (c) (Z1 ∨ Z2 ) dµ = A1 Z1 dµ + A2 Z2 dµ ≤ φ(A1 ) + φ(A2 ) = φ(A). A Thus (b) holds. Now choose a sequence Zn ∈ Z such that (d) Z dµ → C ≡ supZ∈Z Ω Z dµ ≤ φ(Ω) < ∞. Ω n Replace Zn by Z1 ∨ · · · ∨ Zn via (b), and thus take the functions Zn in (d) to be an sequence of functions; letZ0 ≡ lim Zn . The MCT then gives (for any A ∈ A, and also for Ω) (e) so that Z0 ∈ Z , A Z0 dµ = lim A Zn dµ ≤ lim φ(A) ≤ φ(A), (f) Z dµ = lim Ω Zn dµ = C, allowing also that Z0 ≥ 0 is finite. Ω 0 We now define (g) φac (A) ≡ A Z0 dµ

and φs (A) ≡ φ(A) − φac (A)

for all A ∈ A.

Then φac is a finite measure, which can be seen by applying example 1.1 with C finite; and φac µ. Moreover, φs ≡ φ − φac ≥ 0 (since Z0 ∈ Z), so that φs is a finite measure by exercise 1.1. If φs (Ω) = 0, then φ = φac and we are done, with φs ≡ 0.

1.

DECOMPOSITION OF SIGNED MEASURES

65

Suppose φs (Ω) > 0. Then (since µ(Ω) is finite) there is some θ > 0 for which (h)

φs (Ω) > θ µ(Ω).

Let Ω+ and Ω− denote the Jordan–Hahn decomposition for φ∗ ≡ φs − θ µ. Then (i)

µ(Ω+ ) > 0;

µ(Ω+ ) = 0 would imply φs (Ω+ ) = 0 [since φs = φ − φac with φ µ assumed for Case 1 and with φac µ obvious from example 4.1.1], which would then contradict (h) via (j)

φs (Ω) − θ µ(Ω) = φ∗ (Ω) = φ∗ (Ω− ) ≤ 0.

Now, φs (AΩ+ ) ≥ θ µ(AΩ+ ) (since φ∗ (Ω+ ) ≥ 0). Thus φ(A) = φac (A) + φs (A) ≥ A Z0 dµ + φs (AΩ+ ) ≥ A Z0 dµ + θ µ(AΩ+ ) (k) ≥ A (Z0 + θ 1Ω+ ) dµ for all A ∈ A. This implies both Zθ ≡ Z0 + θ 1Ω+ ∈ Z and Ω Zθ dµ = C + θ µ(Ω+ ) > C. But this is a contradiction. Thus φs (Ω) = 0. Thus φ equals φac and satisfies (8), and the theorem holds in Case 1. The a.s. µ uniqueness of Z0 follows from exercise 3.2.2. (This also establishes the Radon–Nikodym theorem below.) Case 2: General φ. Let ν ≡ φ + µ, and note that both φ ν andµ ν. Then by Case 1 (l) φ(A) = A X dν and µ(A) = A Y dν for all A ∈ A for finite ν-integrable functions X ≥ 0 and Y ≥ 0 that are unique a.e. ν. Let D ≡ {ω : Y (ω) = 0}, and then Dc = {ω : Y (ω) > 0}. Define φs (A) ≡ φ(AD) and φac (A) = φ(ADc ). Since µ(D) = D Y dν = D 0 dν = 0 and φs (Dc ) = φ(DDc ) = φ(∅) = 0, we have φs ⊥ µ. If µ(A) = 0, then A Y dν = 0; and thus Y = 0 a.e. ν in A by exercise 1.2.1. But Y > 0 on ADc , and so ν(ADc ) = 0 follows. Then φac (A) = φ(ADc ) = 0, since φ ν. Thus φac µ. Consider the uniqueness of the decomposition. If φ = φac + φs = φ¯ac + φ¯s , then ψ ≡ φac − φ¯ac = φ¯s − φs satisfies both ψ ⊥ µ and ψ µ. Thus ψ ≡ 0. 2 (m)

66

2


The Radon–Nikodym Theorem

Recall that the absolute continuity φ µ means that (1)

φ(A) = 0

µ(A) = 0 with A ∈ A.

whenever

Theorem 2.1 (Radon–Nikodym) Suppose both the signed measure φ and the measure µ are σ-finite on a measure space (Ω, A, µ). Then φ µ if and only if there exists uniquely a.e. µ a finite-valued A-measurable function Z0 on Ω for which (2) φ(A) = Z0 dµ for all A ∈ A . A

Moreover, φ is finite if and only if Z0 is integrable. The function Z0 of (1) is often denoted by very suggestive notation: dφ dµ for all A ∈ A. φ(A) = A dµ

dφ dµ ,

so that we also have the following

The function Z0 is called the Radon–Nikodym derivative of φ with respect to µ. Proof. The Lebesgue decomposition theorem shows that such a Z0 necessarily exists. The sufficiency is just the trivial example 4.1.1. The “moreover” part is also a trivial result. 2 Theorem 2.2 (Change of variable theorem) Let µ ν with µ a σ-finite measure on (Ω, A). If X dµ is well-defined, then dµ (3) X dµ = X dν for all A ∈ A , dν A A One useful special case results from b b · (4) f dG = f g dH when G ≡ g dH for a generalized df H, a

a

where we agree that

b a

≡

a

(a,b]

.

Case 1. X = 1B , for B ∈ A: Then the Radon–Nikodym theorem gives dµ dµ (a) A 1B dµ = µ(AB) = AB dν dν = A 1B dν dν . n Case 2. X = 1 ci 1Bi , for a partition Bi : Case 1 and linearity of the integral give Proof.

dµ dµ dν = A X dν . dν dν Case 3. X ≥ 0: Let Xn ≥ 0 be simple functions that to X. Then the MCT twice gives dµ dµ (c) X dµ = lim A Xn dµ = lim A Xn dν = A X dν . A dν dν (b)

A

X dµ =

n

i=1 ci

A 1Bi

dµ =

n 1

ci

A

1Bi

2.

THE RADON–NIKODYM THEOREM

67

Case 4. X measurable and at least one of X + , X − in L1 : Then X dµ = A X + dµ − A X − dµ A dµ dµ dµ dν − A X − dν = A X dν , dν dν dν so long as one of A X + dµ and A X − dµ is finite. 2 (d)

=

A

X+

Exercise 2.1 Let µ and ν be σ-finite measures on (Ω, A). Let φ and ψ be σ-finite signed measures on (Ω, A). Then (5)

dφ dψ d(φ + ψ) = + a.e. µ dµ dµ dµ

if φ µ and ψ µ ,

(6)

dφ dφ dµ = · a.e. ν dν dµ dν

if φ µ and µ ν.

Note that theorem 3.2.6 (of the unconscious statistician) is another change of ¯ A¯ ) and g : (Ω, ¯ A¯ ) → (R, ¯ B¯ ), then variable theorem. That is, if X : (Ω, A) → (Ω, (7) g(X) dµ = g dµX = y dµg(X) (y) for all B ∈ B, (g◦X)−1 (B)

g−1 (B)

B

when one of the these integrals is well-defined. (See also exercise 7.3.3 below.) Exercise 2.2 Let Pµ,σ2 denote the N (µ, σ2 ) distribution. Let P be Cauchy. (a) Show that Pµ,1 P0,1 and compute dPµ,1 /dP0,1 . (b) Show that P0,σ2 P0,1 and compute dP0,σ2 /dP0,1 . (c) Compute dP/dP0,1 and dP0,1 /dP . Exercise 2.3 Flip a coin. If heads results, then X is a Uniform(0, 1) outcome; but if tails results, then X is a Poisson(λ) outcome. The resulting distribution on R is labeled P . (a) Let µ denote Lebesgue measure on R. Find the Lebesgue decomposition of P with respect to this µ; that is, write P = Pac + Ps . (b) Let µ denote counting measure on {0, 1, 2, . . .}. Determine the Lebesgue decomposition of P with respect to this µ. [If need be, see the definitions of various distributions in chapter 9.] Exercise 2.4 If µ1 , µ2 , . . . are finite measures on some (Ω, A), then there exists a finite measure µ on (Ω, A) such that µk µ for each k ≥ 1. Exercise 2.5 Let µ and ν be signed measures on (Ω, A). Show that the following are equivalent: (a) µ ν. (b) µ+ ν and µ− ν. (c) |µ| |ν|. Exercise 2.6 Let µ and ν be finite measures on (Ω, A). Then µ ν if and only if for every > 0 there exists δ > 0 such that ν(A) < δ implies µ(A) < .

68


Exercise 2.7 (Halmos) Suppose µ, µ1 , µ2 , . . . and ν, ν1 , ν2 , . . . are finite measures on some (Ω, A) for which µk νk for each k ≥ 1. Suppose also that n n µ(A) = lim and ν(A) = lim k=1 µk (A) k=1 νk (A) n→∞

n→∞

for all A ∈ A. Show that the following hold a.e. ν : {d nk=1 νk } /{dν} 1 and {d nk=1 µk } /{dν} {dµ /dν} , (8) n n {d k=1 µk } /{d k=1 νk } → {dµ /dν} . These can be thought of as theorems about Radon–Nikodym derivatives, about absolute continuity of measures, or about change of variables. Exercise 2.8 Let A denote the collection of all subsets A of an uncountable set Ω for which either A or Ac is countable. Let µ(A) denote the cardinality of A. Define φ(A) to equal 0 or ∞ according as A is countable or uncountable. Show that φ µ. Then show that the Radon–Nikodym theorem fails. Exercise 2.9

For a σ-finite measure µ and a finite measure ν on (Ω, A), let

φ(A) ≡ µ(A) − ν(A)

for all A ∈ A .

(a) Show that φ is a signed measure. φ(A) = A (f − g) d(µ + ν),

(b) Show that

for measurable functions f and g with g ∈ L+ 1 (µ + ν). (Note example 4.1.1.) (c) Determine φ+ , φ− , and |φ|; and determine |φ|(Ω) in case µ is also a finite measure. Exercise 2.10 (Total variation distance between probability measures) Define P and Q to be probability measures on (Ω, A). (a) Show that the total variation distance dT V (P, Q) between P and Q satisfies (9) dT V (P, Q) ≡ {supA∈A |P (A) − Q(A)|} = 12 | p − q | dµ for any measure µ dominating both P and Q (that is, P µ and Q µ). (b) Note specifically that the choice of dominating measure µ does not affect the value of dT V (P, Q). (c) Use the previous exercise to show that dT V (P, Q) = |P − Q| (Ω) /2 . Exercise 2.11 (Hellinger distance between probability measures) Let P and Q denote probability measures on (Ω, A). Define the Hellinger distance H(P, Q) by √ √ (10) H 2 (P, Q) ≡ 12 [ p − q ]2 dµ for any measure µ dominating both P and Q. Show that the choice of dominating measure µ does not affect the value of H(P, Q).

2.

THE RADON–NIKODYM THEOREM

69

Exercise 2.12 Let φ be a σ-finite signed measure. Define X dφ = X dφ+ − X dφ− when this is finite. Show that | X dφ| ≤ |X| d|φ| . Exercise 2.13 Let µ be a σ-finite measure on (Ω, A), and define φ(A) ≡ for all A ∈ A for some µ-integrable function X. Show that for all A ∈ A. |φ|(A) = A |X| dµ

A

X dµ

Exercise 2.14 Let (Ω, A) be a measurable space, and let M denote the collection of all finite signed measures µ on (Ω, A). Let µ ≡ |µ|(Ω). Thus µ1 − µ2 = |µ1 − µ2 |(Ω). Show that (M · ) is a complete metric space.

70

3


Lebesgue’s Theorem

Theorem 3.1 (Lebesgue) Suppose F is an function on [a, b]. Then F has a measurable derivative F that exists a.e. λ on [a, b]. Proof.

Consider the Dini derivates D F (x) ≡ lim suph→0+ [F (x + h) − F (x)]/h , D− F (x) ≡ lim suph→0+ [F (x) − F (x − h)]/h , D+ F (x) ≡ lim inf h→0+ [F (x + h) − F (x)]/h , D− F (x) ≡ lim inf h→0+ [F (x) − F (x − h)]/h . +

Trivially, D+ F (x) ≥ D+ F (x) and D− F (x) ≥ D− F (x). All four derivates having the same finite value is (of course) the definition of F being differentiable at x, with the common value of the derivates being called the derivative of F at x and being denoted by F (x). Let

(a)

A ≡ {x : D+ F (x) > D− F (x)} ≡ r,s Ars ≡ r,s {D+ F (x) > s > r > D− F (x)},

where the union is over all rational r and s. It suffices to show that all Ars have outer Lebesgue measure zero, in that λ∗ (Ars ) = 0. To this end, let U be an open set for which Ars ⊂ U with λ(U ) < λ∗ (Ars ) + . For each x ∈ Ars we can specify infinitely many and arbitrarily small h for which [x − h, x] ⊂ U and [F (x) − F (x − h)]/h < r. This collection of closed intervals covers Ars in the sense of Vitali (see exercise 1.2.6). Thus some finite disjoint collection of them m has interiors I1 ≡ (x1 −h1 , x1 ), . . . , Im ≡ (xm −hm , xm ) for which Brs ≡ Ars ∩( i=1 Ii ) has λ∗ (Brs ) > λ∗ (Ars ) − . Then m m ∗ (b) i=1 [F (xi ) − F (xi − hi )] < r i=1 hi ≤ r λ(U ) < r [λ (Ars ) + ] . For each y ∈ Brs we can specify infinitely many and arbitrarily small h for which [y, y + h] ⊂ (some Ii ) and [F (y + h) − F (y)]/h > s. This collection covers Brs in the sense of Vitali. Thus some finite disjoint collection of them has interiors J1 ≡ (y1 , y1 + h1 ), . . . , Jn ≡ (yn , yn + hn ) for which Crs ≡ Brs ∩ ( nj=1 Jj ) has λ∗ (Crs ) > λ∗ (Brs ) − . Then n n ∗ ∗ (c) j=1 [F (yj + hj ) − F (yj )] > s j=1 hj ≥ s [λ (Brs ) − ] > s [λ (Ars ) − 2 ]. Moreover, since the disjoint union of the Jj ’s is a subset of the disjoint union of the Ii ’s, results (b) and (c) yield m r [λ∗ (Ars ) + ] > i=1 [F (xi ) − F (xi − hi )] (d) ≥ nj=1 [F (yj + hj ) − F (yj )] > s [λ∗ (Ars ) − 2 ] for every > 0. That is, r λ∗ (Ars ) ≥ s λ∗ (Ars ). But r < s. Thus λ∗ (Ars ) = 0 for all rational r and s. Thus λ∗ (A) = 0. The sets where any two other Dini derivates take on unequal values also have measure zero, by analogous arguments. That is, F has all four derivates equal to some value F (x) for a.e. x in [a, b].

3.

LEBESGUE’S THEOREM

71

Now, the measurable function difference quotients (e)

Dn F (x) ≡ n[F ((x + 1/n) ∧ b) − F (x)]

on [a, b] converge a.e. λ to F (x) on [a, b], so that F (x) is measurable. Applying Fatou’s lemma to the Dn F (which are ≥ 0, since F is ) gives b b b (f) F (x) dx = a [ lim Dn F (x)] dx ≤ lim a Dn F (x) dx a = lim

b a

[F ((x + 1/n) ∧ b) − F (x)] dx

b+1/n

n F (b) dx −

(g)

= lim [

(h)

≤ F (b) − F (a),

b

a+1/n a

n F (x) dx ]

where we use monotonicity of F in the last step. Thus F is integrable, and is also finite a.e. λ, since we just showed that b (1) whenever F is on [a, b]. a F (x) dx ≤ F (b) − F (a) That is, F is differentiable a.e. λ, and its derivative F satisfies (1). [The Lebesgue singular df in example 7.1.1 below will show that equality need not hold in (1). An example in Hewitt and Stromberg (1965, p. 278) shows that F (x) = 0 is possible for all x, even with a ↑ F. ] 2 Theorem 3.2 (Term-by-term differentiation n of series) Let gk be on [a, b] for each k ≥ 1, and suppose that Sn (x) ≡ k=1 gk (x) converges at x = a and x = b. Then Sn (x) → S(x) for all x in [a, b], for some finite-valued function S(x). Mainly, S (·) exists a.s. λ and is given by (2)

S (x) =

∞

gk (x).

k=1

n Corollary 1 If the power series S(x) ≡ ∞ n=1 an (x − a) converges absolutely for x = a + R, then for all |x − a| < R we may differentiate S(x) term by term. Moreover, this is true for any number of derivatives of S. Proof.

Note that Sn (a) is a convergent sum. Now write n Sn (x) = Sn (a) + [Sn (x) − Sn (a)] = Sn (a) + k=1 [gk (x) − gk (a)] .

Since sequences bounded above converge, the convergence at x = a and x = b gives convergence at all x in the interval. We may assume all gk ≥ 0 on [a, b] with g(a) = 0; else replace gk by gk (·) − g(a). Since S and all Sn are , the derivatives S and all Sn exist a.e. λ by theorem 3.1 (of Lebesgue). Now, (a)

Sn (x) ≤ Sn+1 (x) ≤ S (x)

a.e. λ ;

72


both essentially follow from ∞

(b)

Sn (x + h) − Sn (x) gk (x + h) − gk (x) S(x + h) − S(x) = + h h h n+1 ≥

Sn (x + h) − Sn (x) . h

From (a) we see (without having made use of gk (a) = 0) that (c)

Sn (·) converges a.e. λ with lim Sn ≤ S a.e. λ.

Because Sn , it suffices to show that Sn i →a.e. S for some subsequence ni . Since Sn (b) S(b), we may specify ni so large that 0 ≤ S(b) − Sni (b) < 2−i , and then ∞ ∞ (d) 0 ≤ S(x) − Sni (x) = ni +1 gk (x) ≤ ni +1 gk (b) = S(b) − Sni (b) < 2−i , for all x ∈ [a, b]. Thus ∞ ∞ (e) 0 ≤ i=1 [S(x) − Sni (x)] ≤ i=1 2−i = 1

for all x ∈ [a, b],

where the series in (e) has summands (f)

hi (x) ≡ S(x) − Sni (x)

that are in x.

Thus conclusion (c) also applies to these hi ’s (not just the gk ’s), and we thus conclude from (c) that the series n (g) Tn ≡ i=1 hi converges a.e. λ . But a series of real numbers can converge only if its nth term goes to 0; that is, (h)

S (x) − Sn i (x) = hi (x) → 0

a.e. λ.

As noted above, this suffices for the theorem.

Exercise 3.1

2

Prove the corollary.

Example 3.1 (Taylor’s expansion) Suppose g(·) is defined in a neighborhood of a. Let x∗ denote a generic point between x and a. Let (3)

P1 (x) ≡ g(a) + g (a)(x − a) ,

(4)

P2 (x) ≡ P1 (x) + g (a)(x − a)2 /2! ,

(5)

P3 (x) ≡ P2 (x) + (x − a)3 /3! , . . . ,

(6)

R1 (x) ≡ [g(x) − g(a)]/(x − a)

or g (a),

as

x = a or x = a,

(7)

R2 (x) ≡ 2! [g(x) − P1 (x)]/(x − a)2

or g (a),

as

x = a or x = a,

(8)

R3 (x) ≡ 3! [g(x)−P2 (x)]/(x−a)3

or g (a),

as

x = a or x = a .

3.

LEBESGUE’S THEOREM

73

Then l’Hospital’s rule gives (provided g (a), g (a), g (a), . . . exist, respectively) (9) (10)

lim R1 (x) = g (a) = R1 (a),

x→a

g (x) − P1 (x) g (x) − g (a) = lim = g (a) = R2 (a), x→a x→a x−a x−a

lim R2 (x) = lim

x→a

2! [g (x) − P2 (x)] g (x) − P2 (x) = lim 2 x→a x→a (x − a) x−a

lim R3 (x) = lim

x→a

(11)

g (x) − g (a) = g (a). x→a x−a

= lim

Thus we find it useful to use the representations (with g(k) (a) abbreviating that g (k) (·) exists at a, and with g(k) (·) abbreviating that g(k) (x) exists for all x in a neighborhood of a) P1 (x) + [R1 (x) − g (a)](x − a) if g (a), g(x) = (12) ∗ P1 (x) + [g (x ) − g (a)](x − a) if g (·) ,   if g (a),  P2 (x) + [R2 (x) − g (a)](x − a)/2! (13) g(x) = P2 (x) + [g (x∗ ) − g (a)](x − a)2 /2!   = P (x) + g (x∗ )(x − a)2 /2! if g (·) , 1

(14)

 3   P3 (x) + [R2 (x) − g (a)](x − a) /3! g(x) = P3 (x) + [g (x∗ ) − g (a)](x − a)3 /3!   = P (x) + g (x∗ )(x − a)3 /3! 2

if g (a), if g (·) .

∞ k k Exercise 3.2 Let f (x) = ∞ 0 ak x / 1 bk X in some interval. Suppose that all ak , bk > 0 and ak /bk ↑. Then f (x) > 0 for all x in that interval. (This result is useful in conjunction with the monotone likelihood ratio principle.)

74

4


The Fundamental Theorem of Calculus

Definition 4.1 (Bounded variation) Let F denote a real-valued function on [a, b]. The total variation of F over [a, b] is defined by Vab F ≡ V[a,b] F (1)

≡ sup

n

|F (xk ) − F (xk−1 )| : a ≡ x0 < x1 < · · · < xn ≡ b, n ≥ 1 .

k=1

We say that F is of bounded variation (BV) on [a, b] if Vab F < ∞. It is clear that (2)

Vab F = Vac + Vcb

for a ≤ c ≤ b and F of BV.

Definition 4.2 (Absolutely continuous functions) A real-valued function F on any subinterval I of the line R is said to be absolutely continuous if for all > 0 there exists a δ > 0 such that (3)

n

|F (dk ) − F (ck )| <

whenever

k=1

n (dk − ck ) < δ k=1

and the (ck , dk ) are disjoint subintervals of I. Definition 4.3 (Lipschitz condition) A real-valued function F on any subinterval I of R is said to be Lipschitz if for some finite constant M we have (4)

|F (y) − F (x)| ≤ M |y − x|

for all x and y in I.

We first establish some elementary relationships among the Lipschitz condition, absolute continuity, bounded variation, and the familiar property of being . These concepts have proven to be important in the study of differentiation. We will soon proceed further in this direction, and we will also consider the relationship between ordinary derivatives and Radon–Nikodym derivatives. We first recall from theorem 1.3.1 (the correspondence theorem) that every generalized df F can be associated with a Lebesgue–Stieltjes measure µF via µF ((a, b]) ≡ F (b) − F (a). Proposition 4.1 (The basics) (i) If F is of BV on [a, b], then

We establish some relationships.

(5)

with F1 (x) ≡ Vax F

F (x) = F1 (x)−F2 (x)

and F2 (x) ≡ Vax F −F (x)

both being on [a, b]. (ii) If F is absolutely continuous, then it is of BV. The F1 and F2 in (i) are both absolutely continuous and , and F = F1 − F2 a.s. λ ≡ Lebesgue measure. (iii) Lipschitz functions are absolutely continuous.

4.

THE FUNDAMENTAL THEOREM OF CALCULUS

75

Proof. Consider (i). Now, F1 (x) = Vax F is obviously ; use (2). Then F2 is also , since for x ≤ y we have F2 (y) − F2 (x) = [Vay F − F (y)] − [Vax F − F (x)] (a)

= Vxy F − [F (y) − F (x)] ≥ 0.

Consider (ii). Suppose that the df F (·) is absolutely continuous. Letting = 1 and choosing δ1 so small that equally spaced a ≡ x0 < x1 < · · · < xn ≡ b have mesh ≡ (b − a)/n < δ1 , we have from (2) that n n (b) Vab F = k=1 V[xk−1 ,xk ] F ≤ k=1 1 = n; and thus F is of BV. Since F1 and F2 are , their derivatives exist a.e. by theorem 4.3.1 (Lebesgue’s theorem). But we must still show that F1 is absolutely continuous if F is. So we n suppose that F is absolutely continuous, and specify that 1 (dk − ck ) < δ/2 for some choice of n, ck ’s, and dk ’s. Then for each fixed k with 1 ≤ k ≤ n and the tiny number /(2n), the definition of the BV of F gives mk (c) V[ck ,dk ] F < j=1 |F (ankj ) − F (ank,j−1 )| + ( /2n) for some choice of ck ≡ ank0 < · · · < ankmk ≡ dk . These add to give n n n k=1 |F1 (dk ) − F1 (ck )| = k=1 |V[a,dk ] − V[a,ck ] | = k=1 V[ck ,dk ] F n

mk

(d)

≤

(e)

≤ ( /2) + ( /2) =

k=1

j=1

|F (ankj ) − F (ank,j−1 )| +

n k=1

( /2n)

by absolute continuity of F , since n mk n (f) k=1 k=1 (dk − ck ) < δ/2 j=1 (ankj − ank,j−1 ) = was specified. Consider (iii). Being Lipschitz implies absolute continuity with δ = /M .

Exercise 4.1 Vab F ±

2

If F is of BV on [a, b], define F1 ≡ Vab F + and F2 ≡ Vab F − , where n ≡ sup{ k=1 [F (xk ) − F (xk−1 )]± : a ≡ x0 < · · · < xn ≡ b, n ≥ 1}.

Verify that F = F1 − F2 with F1 and F2 both (providing an alternative to (5)). x Exercise 4.2 Let f be continuous on [a, b], and define F (x) = a f (y) dy for each a ≤ x ≤ b. Then F is differentiable at each x ∈ (a, b) and F = f on (a, b). [Since f is continuous, we need only the Riemann integral. Can we extend this to the Lebesgue integral? Can we reverse the order, and first differentiate and then integrate? The next theorem answers these questions.]

76


Theorem 4.1 (Fundamental theorem of calculus) (i) Let F be absolutely continuous on [a, b], and let λ denote Lebesgue measure. Then the derivative F (·) of F (·) exists a.e. λ on [a, b], and is integrable with respect to λ (using only that F is of BV). Moreover, x (6) F (x) − F (a) = F (y) dy for all x ∈ [a, b]. a

x (ii) Conversely, if F (x) − F (a) = a f dλ for some f that is integrable with respect to λ on [a, b], then F is of BV and F = f a.e. λ on [a, b]. Moreover, F is absolutely continuous on [a, b]. Remark 4.1 The fundamental theorem of calculus can be summarized by saying that F is absolutely continuous if and only if it is the integral of its derivative. The ordinary derivative F is, in fact, also a Radon–Nikodym derivative of the signed measure naturally associated with F ; proposition 4.2 below makes this even clearer. [The Lipschitz condition represents “niceness with a vengeance,” as it guarantees that all difference quotients are uniformly bounded.] 2 b Proof. Consider the converse. If F (x) ≡ F (a)+ a f (y)dy for a ≤ y ≤ b, then F is absolutely continuous by the absolute continuity of the integral theorem. Then F is of bounded variation on [a, b] and F exists a.e. λ in [a, b], by proposition 4.1(ii). But does F = f a.e. λ? Case 1: Suppose |f | is bounded by some finite M on [a, b]. We could consider f + and f − separately, but we will simply assume without loss of generality that f ≥ 0. Then x+1/n the difference quotient Dn F (x) ≡ n x f (y)dy of F also satisfies |Dn F | ≤ M on [a, b], and Dn F (x) → F (x) a.e. Applying the DCT (with dominating function identically equal to M ) once for each fixed x ∈ [a, b] gives x x x a F (y) dy = a lim Dn F (y) dy = lim a n [F (y + 1/n) − F (y)] dy x+1/n a+1/n = lim [n x F (y) dy − n a F (y) dy ] = F (x) − F (a) (a)

=

x a

by continuity of F

f (y)dy.

Thus F (y) = f (y) a.e. on [a, b] by exercise 3.2.2 (refer also to the prototypical example 4.1.1). Case 2: Suppose f is integrable. Again, f ≥ 0 may be assumed. Let fn (·) ≡ n∧f (·), x with f − fn ≥ 0. Now, a fn has derivative fn a.e. on [a, b], by case 1. Thus d x d x d x (b) F (x) = dx a f (y)dy = dx a fn (y)dy + dx a [f (y) − fn (y)]dy ≥ fn (x) + 0 for all n, and hence F (x) ≥ f (x) a.e. on [a, b]. Thus (c)

b a

F (x)dx ≥

b a

f (x)dx = F (b) − F (a), which is ≥

b a

F (x)dx

4.

THE FUNDAMENTAL THEOREM OF CALCULUS

by (4.3.1). Thus b (d) a [F (x) − f (x)] dx = 0

77

with F (x) − f (x) ≥ 0 a.e.,

so that F = f a.e. on [a, b] by exercise 3.2.1. Consider the direct half. Now, F exists a.e. on [a, b] by proposition 4.1(ii), and it is integrable by (4.3.1). Also, x F (x) − F (a) = a f dλ (e) where f ≡ dµF /dλ by proposition 4.2 below, and it is integrable by (4.3.1). Now apply the converse half of the present theorem to conclude that F = f a.e. on [a, b]. 2 Suppose F is and absolutely continuous on R. Use µF ((a, b]) ≡ F (b) − F (a) and the correspondence theorem to associate a Lebesgue–Stieltjes measure µF with F (which is a generalized df). Proposition 4.2 Let F be and absolutely continuous on R. Then µF λ with Radon–Nikodym derivative dµF /dλ, and x (7) F (x) − F (a) = a f dλ where f ≡ dµF /dλ. Proof. Let µ ≡ µF and fix an interval [a, b]. Given > 0, let δ > 0 be as in the definition (3) of absolute continuity. Let A ∈ B be a subset of [a, b] having λ(A) < δ /2. Recalling our definition (1.2.1) of λ via Carathéodory coverings, we can claim that Lebesgue measure satisfies ∞ (a) λ(A) = inf{ ∞ n=1 λ(An ) : A ⊂ n=1 An for An ’s in the field BF }. Thus for some fixed choice of sets in the field BF , we can write ∞ ∞ (b) A ⊂ 1 (cn , dn ] , where 1 (dn − cn ) < λ(A) + δ /2 < δ (recall that each An in the field BF is a finite disjoint union of intervals (c, d] with c and d finite, with finiteness following from A ⊂ [a, b]). Thus ∞ ∞ µF (A) ≤ µF ( 1 (cn , dn ]) ≤ 1 µF ((cn , dn ]) ≤

∞ 1

[F (dn ) − F (cn )] ≤ lim m

m 1

[F (dn ) − F (cn )]

≤ lim = , m

m using F absolutely continuous with 1 (dn − cn ) < δ in the last line. Thus µF (A) < whenever λ(A) < δ , so that µF (A) = 0 whenever λ(A) = 0. Apply this with A = [k, k + 1] for every −∞ < k < ∞. Adding up on k gives µF (A) = 0 whenever λ(A) = 0, for any A ∈ B. Thus µF λ. Now apply the Radon–Nikodym theorem to obtain f as in (7). 2

78


Densities of Transformed Random Variables Example 4.1 (Change of variable) Suppose that X is a rv on (Ω, A, P ) with df FX λ ≡ (Lebesgue measure), with density fX . Let (8)

Y ≡ g(X)

where g−1 is ↑ and absolutely continuous.

Then FY (y) ≡ P (Y ≤ y) = P (g(X) ≤ y) = P (X ≤ g−1 (y)) = FX (g −1 (y)) , where the composition FY = FX (g −1 ) of these absolutely continuous functions is absolutely continuous. So the fundamental theorem of calculus tells us that FY is the integral of its derivative. We can then compute this derivative from the ordinary chain rule. Thus b b −1 d FY (b) − FY (a) = a FY (r) dλ(r) = a [FX (g (r)) dr g −1 (r)] dλ(r) for all a ≤ b. Thus FY λ with density (9)

fY (y) = fX (g −1 (y))

d dy

g −1 (y)

on the line. Call (d/dy) g−1 (y) the Jacobian of the transformation.

2

Exercise 4.3 (a) Show that the (well-defined) composition of two absolutely continuous functions is absolutely continuous. (b) Suppose that h : [a, b] → (0, ∞) is absolutely continuous on [a, b]. Show that log h is also absolutely continuous on [a, b]. Exercise 4.4 Let F be right continuous and of BV on R, with F (−∞) = 0. Let µ denote the associated LS-measure. Then µ λ if and only if F is an absolutely continuous function on R. Exercise 4.5 (Specific step functions that are dense in L2 ) Let h ∈ L2 ([0, 1], B, λ). Consider the following two approximations to h(·). Let ¯ m (t) ≡ m i/m ˇ m (t) ≡ h(i/(m + 1)) h and h (i−1)/m h(s) ds for (i − 1)/m < t ≤ i/m and m ≥ 1. Show that: (10)

¯ m → h a.s. and L2 . h

ˇ m → h a.s. and L2 h provided that h is . 1 ¯ m − h)2 dt = 1 (h2 − h ¯ 2 ) dt, and then [Hint. Show that 0 ≤ 0 (h m 0 i/m ¯ m (t) = m(i/m − t) { h h ds/(i/m − t)} t t + m(t − (i − 1)/m) { (i−1)/m h ds/(t − (i − 1)/m)} → h(t) a.s. (11)

leads easily to a.s. convergence. Alternatively, use the fact that the continuous functions are dense in L2 .]

Chapter 5

Measures and Processes on Products 1

Finite-Dimensional Product Measures Suppose (Ω, A) and (Ω , A ) are measurable

Definition 1.1 (Product spaces) spaces. Define (1)

A × A ≡ σ[F] where F ≡

m (Ai × Ai ) : m ≥ 1 Ai ∈ A and Ai ∈ A , i=1

(2)

F0 ≡ {A × A : A ∈ A and A ∈ A }.

Here A × A ≡ {(ω, ω ) : ω ∈ A, ω ∈ A } is called a measurable rectangle. The σ-field A × A ≡ σ[F] is called the product σ-field. (Ω × Ω , A × A ) is called the product measurable space. The sets A × Ω and Ω × A are called cylinder sets. Proposition 1.1

F is a field (as figure 5.1 below shows).

A1′ A2′

A1 A2 Figure 1.1 The field F.

Theorem 1.1 (Existence of the product measure) Let (Ω, A, µ) and (Ω , A , ν) be σ-finite measure spaces. Define φ on the field F via m m

(3) (Ai × Ai ) = µ(Ai ) × ν(Ai ). φ i=1

i=1

79

80

CHAPTER 5. MEASURES AND PROCESSES ON PRODUCTS

Then φ is a well-defined and σ-finite measure on the field F. Moreover, φ extends uniquely to a σ-finite measure, called the product measure and also denoted by φ, on (Ω × Ω , A × A ). Even when completed, this measure is still unique and is still referred to as the product measure φ. Proof. (See the following exercise. Although the proof is “obvious”, it still requires some tedious detail.) But we will give a better proof herein very soon. 2

Exercise 1.1 Verify that φ is well-defined on F0 , and that φ is countably additive on F0 . Then verify that φ is well-defined on F, and that φ is countably additive on F. Thus φ is a σ-finite measure on F, so that the conclusion of theorem 1.1 follows from the Carathéodory extension of theorem 1.2.1 and its corollary. Exercise 1.2

Use induction to show that theorem 1.1 extends to n-fold products.

Example 1.1 (Lebesgue measure in n dimensions, etc.) (a) We define n n ¯ n , B¯n ) ≡ ¯ ¯ (Rn , Bn ) = i=1 (R, B) and (R i=1 (R, B ) to be the n-fold products of the real line R with the Borel sets B and of the extended ¯ with the σ-field B¯ ≡ σ[B, {+∞}, {−∞}], respectively. Recall from real line R example 2.1.1 that Bn = σ[Un ], where Un denotes all open subsets of Rn . We will refer to both Bn and B¯n as the Borel sets. ¯ B¯ ) by (b) Let λ denote Lebesgue measure on (R, B), as usual. We extend λ to (R, the convention that λ({+∞}) = 0 and λ({−∞}) = 0. Then n ¯n , B¯n , λn ) ≡ n (R, ¯ B, ¯ λ) (4) and (R (Rn , Bn , λn ) ≡ i=1 (R, B, λ) i=1 provides us with a definition of n-dimensional Lebesgue measure λn as the natural generalization of the concept of volume. It is clear that (5)

(Rm × Rn , Bm × Bn , λm × λn ) = (Rm+n , Bm+n , λm+n ),

and that this holds on the extended Euclidean spaces as well. (It is usual not to add the symbol in dealing with the completions of these particular measures.) (c) Now, λ is just a particular Lebesgue–Stieltjes measure on (R, B). Any Lebesgue– ¯ B¯ ) yields an obvious n-fold product on either Stieltjes measure µF on (R, B) or (R, ¯n , B¯n ), which could appropriately be denoted by µF × · · · × µF . (Rn , Bn ) or (R Further, we will let Fn denote the field consisting of all finite disjoint unions of sets of the form I1 × · · · × In where each Ik is of the form (a, b],(−∞, b] or (a, +∞) when considering (Rn , Bn ) (or of the form (a, b], [−∞, b], or (a, +∞] when considering ¯ n , B¯n )). (That is, in the case of (Rn , Bn ) there is the alternative field Fn that (R also generates the σ-field Bn ; and this Fn is made up of simpler sets than is the field B × · · · × B used in definition 1.1.) (d) The approximation lemma of (1.2.8) now shows that if (µF × · · · × µF )(A) < ∞ and if > 0 is given, then (µF × · · · × µF )(A C) < for some C in (the simpler field) Fn . That is, the simpler field gives us a nicer conclusion in this example, because its sets C are simpler. 2

1.

FINITE-DIMENSIONAL PRODUCT MEASURES

81

Exercise 1.3 The measures on (Rn , Bn ) introduced above in example 1.1 are regular measures. Show this. Definition 1.2 (Sections) (a) Let X denote a function on Ω × Ω . For each ω in Ω, the function Xω (·) on Ω defined by Xω (ω ) ≡ X(ω, ω ), for each ω in Ω , is called an ω-section of X(·, ·) . An ω -section of X(·, ·) is defined analogously. (b) Let C be a subset of Ω×Ω . For each ω in Ω, the set Cω = {ω : (ω, ω ) is in C } is called the ω-section of C. An ω -section of C is defined analogously. Theorem 1.2 (Product measure) Let (Ω, A, µ) and (Ω, A , ν) denote finite measure spaces. Let C ∈ A × A . Then: (6) (7)

Every Cω ∈ A and every Cω ∈ A whenever C ∈ A × A , φ(C) ≡ µ(Cω ) dν(ω ) = ν(Cω ) dµ(ω) for every C ∈ A × A , Ω

Ω

and this φ is exactly the product measure φ = µ × ν of theorem 1.1 . Proof. We first show (6). This result is trivial for any C in F0 , or any C in F. Now let S denote the class of all sets C in A × A for which (6) is true. Then S is trivially seen to be a σ-field, using (a)

(n)

(∪n C (n) )ω = ∪n Cω

and (C c )ω = (Cω )c .

But since F ⊂ S, we have that A × A = σ[F] equals S. Let M denote the collection of all sets C in A × A for which (7) holds. Clearly, M contains F0 and F. Note that if Cn converges monotonically to some C, then 1Cn converges monotonically to 1C and (b)

every section of 1Cn converges monotonically to the corresponding section of 1C .

We will now use this fact to show that the class M of all subsets of A × A for which (7) holds is a monotone class; it will then follow by proposition 1.1.6 that M = σ[F] = A × A . Let Cn denote a sequence of sets in the class M that converge monotonically (we will consider only the case, since we only need to take complements in the case), and we give the name C to the limiting set. Since 1Cn 1C , the function 1C is (A × A )-measurable, and thus every section of 1C is measurable by (6). Now, for fixed ω the number h(ω ) ≡ µ(Cω ) = 1 (ω)dµ(ω) is (by the MCT and (b)) the limit of the sequence of numbers Ω C,ω hn (ω ) ≡ µ(Cn,ω ) = Ω 1Cn ,ω (ω)dµ(ω), for each ω in Ω . Thus the function h on Ω is the limit of the functions hn on Ω ; and since Cn is in M, the functions hn are A -measurable; thus h is A -measurable by proposition 2.2.2. Moreover, the finite numbers φ(Cn ) are bounded above by µ(Ω) ν(Ω ), and thus converge to some number; call it φ(C). That is, (c) φ(C) ≡ lim φ(Cn ) ≡ lim Ω { Ω 1Cn ,ω (ω)dµ(ω)} dν(ω ) when Cn is in M n

n

82


(d)

= lim n

= (e)

=

Ω

hn dν =

hn } dν Ω {lim n

(ω)dµ(ω)} dν(ω )

Ω

{

Ω

µ(Cω ) dν(ω ) .

Ω 1C,ω

=

Ω

by the MCT and hn h

hdν

by the definition of h

(Since φ(C) is finite, we see that h is ν-integrable. Thus h(ω ) is finite for a.e. [ν] ω .) The argument for each fixed ω is symmetric, and it gives the second equality in (7). Thus C is in M, making M the monotone class A × A ; and (b) holds. [Thus the result (7) holds for the set function φ. But is φ a measure?] In this paragraph we will show that the product measure φ of theorem 1.1 exists, and is defined by (e). To this end, let D1 , D2 , . . . be pairwise disjoint sets in A × A , n ∞ and let Cn ≡ 1 Dk C ≡ 1 Dk . Then linearity of both single integrals shows (in the second equality) that n ∞ n (f) 1 φ(Dk ) = lim 1 φ(Dk ) = lim φ( 1 Dk ) = lim φ(Cn ) = φ(C) n

n

n

by (c) through (e) ∞ = φ( 1 Dk ),

(g)

so that φ is c.a., and a measure on A × A . We have just verified that the product measure of (3) exists on A × A , and is given by (7). That is, we have just proven theorem 1.1 and given the representation (7) for φ(C). Note that the product measure φ also satisfies φ(C) = Ω×Ω 1C (ω, ω ) dφ(ω, ω ). 2 Theorem 1.3 (Fubini) Let (Ω, A, µ) and (Ω , A , ν) be σ-finite measure spaces. Let φ = µ × ν on (Ω × Ω , A × A ). Suppose that X(ω, ω ) is φ-integrable (i.e., X −1 (B¯ ) ⊂ A × A and Ω×Ω X dφ is finite). Then: (8)

All ω -sections Xω (·) of X are A-measurable functions on Ω.

(9)

For a.e. [ν] fixed ω , the function Xω (·) ≡ X(·, ω ) is µ-integrable. The function h(ω ) ≡ Xω (ω) dµ(ω) is a ν-integrable function of ω .

(10)

(11) Ω×Ω

Ω

X(ω, ω )dφ(ω, ω ) = [ X(ω, ω )dµ(ω)]dν(ω ) =

Ω

Ω

h(ω )dν(ω ).

Ω

[Setting X equal to 1C in (11) for C ∈ A × A shows how the value φ(C) of the product measure φ at C was defined as an iterated integral; recall (7).] Corollary 1 (Tonelli) Let X be A × A -measurable and suppose either (12) |X| dµ dν < ∞ or |X| dν dµ < ∞ or X ≥ 0. Then the claims of Fubini’s theorem are true, including (13) X dφ = X dµ dν = X dν dµ.

1.

FINITE-DIMENSIONAL PRODUCT MEASURES

83

Corollary 2 (µ × ν null sets) A set C in A × A is (µ × ν)-null if and only if almost every ω-section of C is a ν-null set. That is, for C ∈ A × A we have (14)

µ × ν(C) = 0

if and only if

ν(Cω ) = 0 for a.e. [µ] ω in Ω.

Proof. By using the σ-finiteness of the two measures to decompose both Ω and Ω , we may assume in this proof that both µ and ν are finite measures. We begin by discussing only measurability questions. We will first show that (a)

all ω -sections of an (A × A )-measurable function X are A-measurable.

The previous theorem shows that (b)

all ω -sections Xω of X are A-measurable whenever X = 1C for some C ∈ A × A .

¯ Now let X denote any (A × A )-measurable function. Then for any B in B, (c)

−1 Xω−1 (B)} (B) = {ω : X(ω, ω ) ∈ B} = {ω : (ω, ω ) ∈ X

is the ω -section of the indicator function of the set C = X −1 (B); so (b) shows that any arbitrary ω -section of this X is A-measurable, and so establishes (a) and (8). We now turn to all the other claims of the Fubini and Tonelli theorems. By theorem 1.2 they hold for all (A × A )-measurable indicator functions. Linearity of the various integrals shows that the theorems also hold for all simple functions. Applying the MCT to the various integrals shows that the theorems also hold for all (A × A )-measurable X ≥ 0. Then linearity of the integral shows that the theorems also hold for all X for whichever of the three integrals exists finitely (the double integral or either iterated integral). Corollary 2 follows immediately by applying (13) and exercise 3.2.2 (only the zero function) to the integral of the function 1C . 2

84

2


Random Vectors on (Ω, A, P )

We will now treat measurable functions from a probability space (Ω, A, P ) to a Euclidean space (Rn , Bn ), with n ≥ 1. Let x ≡ (x1 , . . . , xn ) denote a generic vector in the Euclidean space Rn . ≡ (X1 , . . . , Xn ) is such that Definition 2.1 (Random vectors) Suppose X X : Ω → Rn is Bn -A-measurable. Then X is called a random vector (which is also by abbreviated rv ). Define the joint distribution function (or just df) of X n F (x) ≡ FX1 ,...,Xn (x1 , . . . , xn ) = P ( i=1 [Xi ≤ xi ] ) . Write x ≤ y to denote that xi ≤ yi for all 1 ≤ i ≤ n; and now define the basic rectangles (x, y ] ≡ ×ni=1 (xi , yi ] whenever x ≤ y. Let n F (x, y ] ≡ P ( i=1 [xi < Xi ≤ yi ] ) (1) for all x ≤ y . ≡ (X1 , . . . , Xn ) : Ω → Rn or R ¯n Proposition 2.1 (Measurability) Now, X is such that Bn -A-measurable B-A-measurable X is if and only if each X is i ¯ B¯n -A-measurable B-A-measurable. Thus, a random vector is measurable if and only if each coordinate rv is measurable. Proof.

We give the details for finite-valued functions. (⇒) Now, [Xi ≤ xi ] = Xi−1 ((−∞, xi ] ) = X −1 (R × · · · × R × (−∞, xi ] × R × · · · × R) ∈ A.

≤ x ] = ∩n [Xi ≤ xi ] ∈ A, since each Xi is measurable, where (⇐) Also, [X i=1 σ[ {all (−∞, x1 ] × · · · × (−∞, xn ]} ] = Bn . Moreover, −1 (A∗ ) ∈ A} = (a σ-field) A∗ ≡ {A∗ : X by proposition 2.1.2. Thus X −1 (Bn ) ⊂ A.

Exercise 2.1 (Joint df) (2)

2

A joint df F is and right continuous and satisfies

All F (x1 , . . . , xi−1 , −∞, xi+1 , . . . , xn ) = 0

and

F (x1 , . . . , xi−1 , +∞, xi+1 , . . . , xn ) (3)

= FX1 ,...,Xi−1 ,Xi+1 ,...,Xn (x1 , . . . , xi−1 , xi+1 , . . . , xn )

for all j = 1, . . . , n and x1 , . . . , xn .

F (∞, . . . , ∞) = 1,

2.

RANDOM VECTORS ON (Ω, A, P )

85

Exercise 2.2 Suppose F : Rn → R is and right continuous and satisfies (2) and (3). Then there exists a unique probability measure P ≡ PF on Bn that satisfies (4)

P ((x, y ] ) = F (x, y ]

for all x ≤ y .

This is a generalization of the correspondence theorem to n > 1. Now note that the identity function X(ω) ≡ ω, for each ω ∈ Rn , is a random vector on (Rn , Bn ) that has as its joint df the function F above. Thus, given any joint df F , there is a having F as its joint df. This is in the spirit of example 2.2.1. random vector X Exercise 2.3 (Hoeffding–Fréchet bounds) (i) Show that Ha (u, v) ≡ u ∧ v and Hb (u, v) ≡ (u + v − 1)+ are dfs on [0, 1]2 with Uniform(0, 1) marginal dfs. Determine the sets on which the densities of Ha and Hb are positive. (ii) Suppose that F (x, y) is a df on R2 with marginal dfs G(x) ≡ F (x, ∞) and H(y) ≡ F (∞, y), respectively. Show that Hb (G(x), H(y)) ≤ F (x, y) ≤ Ha (G(x), H(y)) for all (x, y) ∈ R2 .

86

3


Countably Infinite Product Probability Spaces

We now begin to carry out the program discussed in section 5. That is, we will extend the notion of rvs and product probability measures to a countably infinite number of dimensions. Notation 3.1 (R∞ and B∞ ) Let ∞ (1) R∞ ≡ n=1 R ≡ {(x1 , x2 , . . .) : xn ∈ R for all n ≥ 1}. Let I denote an interval of the type (c, d ], (−∞, d], or (c, +∞). By an n-dimensional rectangle we will mean any set of the form I1 × · · · × In × R × R × · · ·, where each interval In is of the type above. A finite-dimensional rectangle is an n-dimensional rectangle, for some n ≥ 1. A cylinder set is a set of the form Bn × R × R × · · · with Bn in Bn for some n ≥ 1. Thus: (2)

CI ≡ {all finite-dimensional rectangles} = {I1 × · · · × In × R × R × · · · : n ≥ 1, all In as above},

(3)

CF ≡ {all finite disjoint unions of finite-dimensional rectangles},

(4)

C∞ ≡ {all cylinder sets} ≡ {Bn × R × R × · · · : n ≥ 1, Bn ∈ Bn }.

Both CF and C∞ are fields, and a trivial application of exercise 1.1.1 shows that (5)

B∞ ≡ σ[CI ] = σ[CF ] = σ[C∞ ].

Thus, extending a measure from CI to B∞ will be of prime interest to us. We first extend the criterion for measurability from n dimensions to a countably infinite number of dimensions. 2 ≡ (X1 , X2 , . . .) : Ω → R∞ Proposition 3.1 (Measurability on B∞ ) (a) Now, X is B∞ -A-measurable if and only if each Xn is B-A-measurable. is B∞ -A-measurable and if (i1 , i2 , . . .) is an arbitrary sequence of integers, (b) If X then Y ≡ (Xi1 , Xi2 , . . .) is B∞ -A-measurable. Exercise 3.1

Prove proposition 3.1.

Notation 3.2

We will use the notation

(6)

F(Xi ) ≡ Xi−1 (B)

and

−1 F(Xi1 , Xi2 , . . .) ≡ Y −1 (B∞ ) = σ[ ∞ n=1 Xin (B)]

to denote the minimal sub σ-fields of A relative to which the quantities Xi and ≡ (Xi1 , Xi2 , . . .) are measurable. Y 2 Now suppose that Pn is a probability measure on (Rn , Bn ), for each n ≥ 1. The question is, When can we extend the collection {Pn : n ≥ 1} to a measure on (R∞ , B∞ )? Reasoning backwards to see what conditions the family of finitedimensional distributions should satisfy leads to the following definition.

3.

COUNTABLY INFINITE PRODUCT PROBABILITY SPACES

87

Definition 3.1 (Consistency) Finite-dimensional distributions {(Rn , Bn , Pn )}∞ n=1 are consistent if for every n ≥ 1, every B1 , . . . , Bn ∈ B, and every 1 ≤ i ≤ n, Pn−1 ((X1 , . . . , Xi−1 , Xi+1 , . . . , Xn ) ∈ B1 × · · · × Bi−1 × Bi+1 · · · × Bn ) (7)

= Pn ((X1 , . . . , Xi−1 , Xi , Xi+1 , . . . , Xn ) ∈ B1 ×· · ·×Bi−1 ×R×Bi+1 ×· · ·×Bn ).

Theorem 3.1 (Kolmogorov’s extension theorem) An extension of any to a probability P (·) consistent family of probability measures {(Rn , Bn , Pn )}∞ n=1 on (R∞ , B∞ ) necessarily exists, and it is unique. We will first summarize the main part of this proof as a separately stated result that seems of interest in its own right. Theorem 3.2 ((R∞ , B∞ ) extension theorem; Breiman) Let P on CI satisfy: (a) P ≥ 0 and P (R∞ ) = 1. m (b) If D = m j=1 Dj for n-fold rectangles D and Dj , then P (D) = 1 P (Dj ). (c) If D denotes any fixed n-dimensional rectangle, then there exists a sequence of compact n-dimensional rectangles Dj for which Dj D and P (Dj ) P (D). [That is, P is well-defined and additive on n-dimensional rectangles and satisfies something like continuity from below.] Then there exists a unique extension of P to B∞ . Proof.

(Recall the continuity result of proposition 1.1.3.) Now, CF ≡ {all finite disjoint unions of finite-dimensional rectangles} = {a field generating B∞ }.

(p) For A =

m 1

Dj ∈ B0 , define P (A) ≡

m j=1

P (Dj ).

m m First, we will show that P is well-defined on CF . Let A = 1 Dj = 1 Dk . m m Now, Dk = Dk A = 1 Dk Dj and Dj = Dj A = 1 Dj Dk . Thus (q) (r)

P (A) = =

m 1

P (Dj ) =

m m 1

1

m 1

P(

P (Dk ∩ Dj ) =

m 1

m 1

Dj Dk ) =

P(

m 1

m m 1

1

Dk ∩ Dj ) =

P (Dj Dk ) m 1

P (Dk ) = P (A).

Next, A1 , . . . , Am ∈ CF be such that we will show that P is f.a. on CF . So we mlet i A≡ m 1 Ai ∈ CF also. Then, by writing Ai = 1 Dij , m mi m mi m (s) P (A) = P ( m 1 Ai ) = P ( 1 1 1 P (Ai ), 1 Dij ) = 1 P (Dij ) = (using condition (b) in each of the last two equalities), since P is well-defined. We will now show that P is continuous from above at ∅. Let An ’s in CF be such that An ∅. We must show that P (An ) 0. Assume not. Then ∞ P (An ) > 0; and by going to subsequences, we may assume that An = A∗n × n+1 R, where each A∗n is a finite union of disjoint rectangles (repeat some members of the sequence if

88


necessary in order to have A∗n ⊂ Rn ). By condition (c), choose Bn∗ ⊂ A∗n such that Bn∗ is a finite union of compact disjoint rectangles in Rn with ∞ (t) where Bn ≡ Bn∗ × n+1 R. P (An \ Bn ) < /2n+1, n ∞ Let Cn = 1 Bk ≡ Cn∗ × n+1 R, with Cn∗ compact in Rn (the Bn ’s need not be , but the Cn ’s are). Then we observe that Cn ∅, since Cn ⊂ Bn ⊂ An with An ∅; but we also have P (Cn ) ≥ /2, since n n n P (An \ Cn ) ≤ k=1 P (An \ Bk ) ≤ k=1 P (Ak \ Bk ) ≤ 1 /2k+1 ≤ /2. (u) But Cn with P (Cn ) ≥ /2 for all n is not compatible with the conclusion that (n) (n) Cn ∅: Let x(1) ∈ C1 , . . . , x(n) ∈ Cn , . . . , where x(n) ≡ (x1 , x2 , . . .). Choose an (N1 ) → (some x1 ) ∈ C1∗ ; then choose a further initial subsequence N1 such that x1 (N2 ) (N2 ) subsequence N2 such that (x1 , x2 ) → (some (x1 , x2 )) ∈ C2∗ ; . . . . Along the (N ) diagonal subsequence, say N , we have xj → xj , for all j. Now, x = (x1 , x2 , . . .) ∈ Cn for all n. Hence Cn ∅. But this is a contradiction, and thus allows us to claim that P (An ) 0 for any An ’s in CF that satisfy An ∅. Now apply the continuity of measures in proposition 1.1.3, and then apply the Carathéodory extension of theorem 1.2.1 to complete the proof. 2 Proof. (v)

We now turn to the Kolmogorov extension theorem. The P defined by P (B1 ×· · ·×Bn ×· · ·) ≡ Pn (B1 ×· · ·×Bn ) = Pm+n (B1 ×· · ·×Bn ×R×· · ·×R)

is a well-defined f.a. probability on CI = {all finite-dimensional rectangles}; this follows from the consistency condition (7). Thus (a) and (b) of theorem 3.2 hold. We will now verify (c). Fix n. Let Dn be an arbitrary but fixed n-dimensional rectangle. It is clearly possible to specify compact n-dimensional rectanglesDnj for ∞ which Dnj Dn as j → ∞. Write Dj = Dnj × ∞ n+1 R and D = Dn × n+1 R, so that Dj D. Thus, by the continuity of signed measures in proposition 1.1.3, (w)

P (Dj ) = Pn (Dnj ) Pn (Dn ) = P (D),

since Pn is a measure on (Rn , Bn ). Thus (c) holds. The conclusion follows from theorem 3.2. 2 Example 3.1 (Coordinate rvs) Once consistent probability measures Pn (·) on (Rn , Bn ) have been extended to a probability measure P (·) on (R∞ , B∞ ), it is appropriate then to define Xn (x1 , x2 , . . .) = xn , for each n ≥ 1. These are rvs on the probability space (Ω, B, P ) ≡ (R∞ , B∞ , P ). Moreover, −1 (Bn × ∞ R)) P ((X1 , . . . , Xn ) ∈ Bn ) = P ((X1 , . . . , Xn )−1 (Bn )) = P (X n+1 ∞ (8) = P (Bn × n+1 R) = Pn (Bn ) ≡ (X1 , X2 , . . .) : Ω → R∞ for all Bn ∈ Bn . We thus have a realization of X that is B∞ -A-measurable, and each (X1 , . . . , Xn ) induces the distribution Pn on (Rn , Bn ). This is the natural generalization of example 2.2.1 and the comment below exercise 5.2.2. 2

3.

COUNTABLY INFINITE PRODUCT PROBABILITY SPACES

89

Theorem 3.3 (The finite dfs define probability theory) = (X1 , X2 , . . .) denote any random element on (R∞ , B∞ ). Then P itself Let X and whether or not there exists a finite rv X such that Xn converges to X in the sense of →a.s. , →p , →r , or →d can be determined solely by examination of the finite-dimensional distributions of X. Proof. Let C denote the π ¯ -system of all sets n1 (−∞, xi ], for some n ≥ 1 and all xi ∈ R. The finite-dimensional distributions (even the finite-dimensional dfs) determine P∞ on C, and hence on B∞ = σ[C] (appeal to Dynkin’s π-λ theorem of proposition 1.1.5). To emphasize the fact further, we now consider each convergence mode separately. →d : Obvious. →r : E|Xn − X|r → 0 if and only if E|Xn − Xm |r < for all n, m ≥ some N . →p : Xn →p X if and only if P (|Xn − Xm | > ) < for all n, m ≥ some N . →a.s. : Xn →a.s. X if and only if ∞ 1 = P (∪∞ n=1 ∩m=n [|Xm − Xn | ≤ ])

for all > 0

= lim lim P (∩N m=n [|Xm − Xn | ≤ ]) = lim lim{a function of FXn ,...,XN }. n

N

n

N

The proof is complete 2

Example 3.2 (Equivalent experiments) Perhaps I roll an ordinary die n times with the appearance of an even number called “success.” Perhaps I draw a card at random n times, each time from a freshly shuffled deck of standard playing cards, with “red” called “success.” Perhaps I flip a fair coin n times with “heads” called “success.” Note that (X1 , . . . , Xn ) has the same distribution in all three cases. Thus, if I report only the data from one of these experiments, you can not hope to determine which of the three experiments was actually performed. These are called equivalent experiments. 2

90

4


Random Elements and Processes on (Ω, A, P )

Definition 4.1 (Projections and finite-dimensional subsets) Let MT denote a collection of functions that associate with each t of some set T a real number denoted by either xt or x(t). [T is usually a Euclidean set such as [0, 1], R, or [0, 1] × R. The collection MT is often a collection of “nice” functions, such as the continuous functions on T .] For each integer k and all (t1 , . . . , tk ) in T we let πt1 ,...,tk denote the projection mapping of MT into k-dimensional space Rk defined by (1)

πt1 ,...,tk (x) ≡ (x(t1 ), . . . , x(tk )).

Then for any B in the set of all k-dimensional Borel subsets Bk of Rk , the set πt−1 (B) is called a finite-dimensional subset of M . 1 ,...,tk Exercise 4.1 Show that the collection M0T of all finite-dimensional subsets of MT is necessarily a field. (This is true no matter what collection MT is used.) Definition 4.2 (Measurable function spaces, finite-dimensional distributions, random elements, and normal processes) We let MT denote the σ-field generated by the field M0T . We call M0T and MT the finite-dimensional field and the finitedimensional σ-field, respectively. Call the measurable space (MT , MT ) a measurable function space over T . Given any probability space (Ω, A, P ) and any measurable space (Ω∗ , A∗ ), an ∗ A -A-measurable mapping X : Ω → Ω∗ will be called a random element. We denote this by X : (Ω, A) → (Ω∗ , A∗ ) or by X : (Ω, A, P ) → (Ω∗ , A∗ ), or even by X : (Ω, A, P ) → (Ω∗ , A∗ , P ∗ ), where P ∗ denotes the induced probability on the image space. A random element X : (Ω, A, P ) → (MT , MT , P ∗ ) in which the image space is a measurable function space will be called a process. The finite-dimensional distributions of a process are the distributions induced on (Rk , Bk ) by the projection mappings πt1 ,...,tk : (MT , MT , P ∗ ) → (Rk , Bk ). If all of the finite-dimensional distributions of a process X are multivariate normal (see section 9.3 below), then we call X a normal process. Definition 4.3 (Realizations and versions) If two random elements X and Y (possibly from different probability spaces to different measurable function spaces) have identical induced finite-dimensional distributions, then we refer to X and Y as different realizations of the same random element and we call them equivalent random elements. We denote this by agreeing that X∼ means that X and Y are equivalent random elements. =Y [We will see in chapter 12 that a process called Brownian motion can be realized on both (R[0,1] , B[0,1] ) and (C, C), where C ≡ C[0,1] denotes the space of all continuous functions on [0, 1] and C ≡ C[0,1] denotes its finite-dimensional σ-field.] If X and Y are defined on the same probability space and P (Xt = Yt ) = 1 for all t ∈ T , then X and Y are called versions of each other. (In chapter 12 we will see versions X and Y of Brownian motion where X : (Ω, A, P ) → (R[0,1] , B[0,1] ) and Y : (Ω, A, P ) → (C[0,1] , C[0,1] ). Of course, this X and Y are also different realizations of Brownian motion.)

4.

RANDOM ELEMENTS AND PROCESSES ON (Ω, A, P )

91

Definition 4.4 (Finite-dimensional convergence, →f d ) Suppose X, X1 , X2 , . . . denote processes with image space (MT , MT ). If the convergence in distribution (2)

πt1 ,...,tk (Xn ) = (Xn (t1 ), . . . , Xn (tk )) →d (X(t1 ), . . . , X(tk )) = πt1 ,...,tk (X)

holds for all k ≥ 1 and all t1 , . . . , tk in T , then we write Xn →f d X as n → ∞, and we say that the finite-dimensional distributions of Xn converge to those of X.

The General Stochastic Process Notation 4.1 (RT , BT )) (3)

We now adopt the convention that

(RT , BT ) denotes the measurable function space with RT ≡

t∈T Rt ,

where each Rt is a copy of the real line. Thus RT consists of all possible realvalued functions on T , and BT is the smallest σ-field with respect to which all πt are measurable. We call a process X : (Ω, A, P ) → (RT , BT ) a general stochastic process. We note that a general stochastic process is also a process. But we do not yet know what BT looks like. A set BT ∈ BT is said to have countable base t1 , t2 , . . . if (4)

BT = πt−1 (B∞ ) 1 ,t2, ...

for some B∞ ∈ B∞ ;

here B∞ is the countably infinite-dimensional σ-field of section 5.3. Let BC denote the class of countable base sets defined by (5)

BC ≡ {BT ∈ BT : BT has a countable base}.

[Recall F(X1 , X2 , . . .) and F(Xs : s ≤ t) measurability from section 2.5.]

2

Proposition 4.1 (Measurability in (RT , BT )) Now, BC is a σ-field. In fact, BC is the smallest σ-field relative to which all πt are measurable; that is, (6)

BT = B C .

Also (generalizing proposition 5.2.1), (7)

X is BT -A-measurable if and only if Xt is B-A-measurable for each t ∈ T.

Proof. Clearly, BT is the smallest σ-field containing BC ; so (6) will follow from showing that BC is a σ-field. Now, C is closed under complements, since c πt−1 (B∞ )c = πt−1 (B∞ ). Suppose that B1 , B2 , . . . in BC have countable bases 1 ,t2 ,... 1 ,t2 ,... ∞ T1 , T2 , . . ., and let T0 = ∪m=1 Tm . Then using the countable set of distinct coor∞ ∞ dinates in T0 , reexpress each Bm as Bm = πT−1 (Bm ) for some Bm ∈ B∞ . Then 0 −1 ∞ ∞ ∞ ∪m=1 Bm = πT (∪m=1 Bm ) is in BC . Thus BC is closed under countable unions. Thus BC is a σ-field. Now to establish (7): Suppose X is BT -A-measurable. Thus (a)

Xt−1 (B) = X −1 (πt−1 (B)) ∈ A for B ∈ B,

so that each Xt is B-A-measurable. Suppose that each Xt is B-A-measurable. Then exercise 5.3.1 shows that (Xt1 , Xt2 , . . .) is B∞ -A-measurable for all sequences t1 , t2 , . . . of elements of T . That is, X −1 (BC ) ⊂ A. Since BT = BC , we thus have X −1 (BT ) ⊂ A, and hence X is BT -A-measurable. 2

92


Remark 4.1 (Consistency of induced distributions in (RT , BT )) Any general stochastic process X : (Ω, A, P ) → (RT , BT ) has a family of induced distributions (8)

(Bk )) Pt∗1 ,...,tk (Bk ) = P (X −1 ◦ πt−1 1 ,...,tk

for Bk ∈ Bk

for all k ≥ 1 and t1 , . . . , tk ∈ T . These distributions are necessarily consistent in the sense that (9)

Pt∗1 ,...,tk (B1 × · · · × Bi−1 × R × Bi+1 × · · · × Bk ) = Pt∗1 ,...,ti−1 ,ti+1 ,...,tk (B1 × · · · × Bi−1 × Bi+1 × · · · × Bk )

for all k ≥ 1, all B1 , . . . , Bk ∈ B, all 1 ≤ i ≤ k, and all t1 , . . . , tk ∈ T . [The next result gives a converse. It is our fundamental result on the existence of stochastic processes with specified distributions.] 2 Theorem 4.1 (Kolmogorov’s consistency theorem) Given a consistent set of distributions as in (9), there exists a distribution P on (RT , BT ) such that the identity map X(ω) = ω, for all ω ∈ RT , is a general stochastic process X : (RT , BT , P ) → (RT , BT ) whose family of induced distributions is the Pt∗1 ,...,tk of (9). Exercise 4.2 Prove theorem 4.1. [Define P ∗ (B) = P (πT−1 (B)) for B ∈ BC and i each countable subset Ti of T . Use notational ideas from the proof of proposition 4.1 to show easily that P ∗ (·) is well-defined and countably additive.] Example 4.1 (Comment on (R[0,1] , B[0,1] )) The typical function x in RT has no smoothness properties. Let T = [0, 1] and let C denote the subset of R[0,1] that consists of all functions that are continuous on [0, 1]. We now show that (10)

C ∈ B[0,1] .

Let (Ω, A, P ) denote Lebesgue measure on the Borel subsets of [0, 1]. Let ξ(ω) = ω. Now let X : (Ω, A, P ) → (R[0,1] , B[0,1] ) via Xt (ω) = 0 for all ω ∈ Ω and for all t ∈ T . Let Y : (Ω, A, P ) → (R[0,1] , B[0,1] ) via Yt (ω) = 1t (ω). Now, all finite-dimensional distributions of X and Y are identical. Note, however, that [ω : X(ω) ∈ C] = Ω, while [ω : Y (ω) ∈ C] = ∅. Thus C cannot be in B[0,1] . 2

Smoother Realizations of General Stochastic Processes Suppose now that X is a process of the type X : (Ω, A, P ) → (RT , BT , P ∗ ). As the previous example shows, X is not the unique process from (Ω, A, P ) that induces the distribution P ∗ on (RT , BT ). We now let MT denote a proper subset of RT and agree that MT denotes the σ-field generated by the finite-dimensional subsets of MT . Suppose now that X(ω) ∈ MT for all ω ∈ Ω. Can X be viewed as a process X : (Ω, A, P ) → (MT , MT , P˜ ) such that (MT , MT , P˜ ) has the same finitedimensional distributions as does (RT , BT , P ∗ )? We now show that the answer is necessarily yes. Interesting cases arise when the functions of the MT above have smoothness properties such as continuity. The next result is very important and useful.

4.

RANDOM ELEMENTS AND PROCESSES ON (Ω, A, P )

93

Theorem 4.2 (Smoother realizations of processes) Consider an arbitrary measurable mapping X : (Ω, A, P ) → (RT , BT , P ∗ ). (i) Let MT ⊂ RT . Then we can view X as a process X : (Ω, A) → (MT , MT ) if and only if every sample path X· (ω) = X(·, ω) is in MT and every Xt (·) ≡ X(t, ·) is a random variable. (ii) Let X(Ω) ⊂ MT ⊂ RT . Then X : (Ω, A, P ) → (MT , MT , P˜ ), where the finitedimensional distributions of (MT , MT , P˜ ) are the same as those of (RT , BT , P ∗ ). Proof. (11)

Note first that AT ≡ X −1 (MT ) ≡ X −1 (MT ) ∩ X −1 (BT ) = X −1 (MT ∩ BT )

is a σ-field

(though none of the subsets of AT may be in A). Then (Ω, AT , PT ) is a probability space where (12)

PT (X −1 (MT ) ∩ X −1 (BT )) ≡ P (X −1 (BT ))

for all BT ∈ BT .

(i) (⇐) For any t1 , . . . , tk in T and any Bt1 , . . . , Btk in B we have (Bt1 × · · · × Btk )) = X −1 (πt−1 (Bt1 ) ∩ · · · ∩ πt−1 (Btk )) X −1 (πt−1 1 ,...,tk 1 k = ∩ki=1 X −1 (πt−1 (Bti )) = ∩ki=1 Xt−1 (Bti ) i i (a)

∈A

since each Xti is a random variable.

Thus X −1 (MT ) = X −1 (σ[ all πt−1 (Bt1 × · · · × Btk ) ∩ MT ] 1 ,...,tk = σ[ all X −1 (πt−1 (Bt1 × · · · × Btk ) ∩ MT )] 1 ,...,tk = σ[a collection of subsets of AT ] (b)


by (a)

⊂ AT ,

so that X is MT -A-measurable. (⇒) This is now trivial, since X −1 (MT ) ∩ Xt−1 (B) = X −1 (MT ∩ πt−1 (B)) ∈ A for each B ∈ B. (ii) This is now clear, and it summarizes the most useful part of this theorem. 2

Remark 4.2 It is interesting to consider the case where MT is a countable or finite set. The resulting (MT , MT , P˜ ) is the natural probability space. 2

Chapter 6

General Topology and Hilbert Space 1

General Topology

Only the definitions and the statements of the major results are presented. Definition 1.1 (Topology, open set, neighborhood, and boundary) (a) A topological space (or a space) is a pair (M, U) where M is a set and U is a family of subsets that satisfies (1)

∪V ≡ ∪V ∈V V ∈ U, for any V ⊂ U , ∩V ∈ U for any finite V ⊂ U, and

∅ and M are in U.

U is called a topology, and the sets in U are called open sets. The complements of all open sets are called closed sets. An open set U containing a point x ∈ M is called a neighborhood of x. (b) The interior A◦ of A is int(A)≡ A◦ ≡ ∪{U : U ⊂ A with U open}. The closure A¯ of A is defined by cl(A)≡ A¯ ≡ ∩{F : F ⊃ A with F closed}. The boundary ∂A of A is ∂A ≡ A¯ \ A◦ . (c) Call x an accumulation point of A if A \ {x} contains x. Exercise 1.1 (a) Show that A¯ = {x : every neighborhood U of x has U ∩ A = ∅}. ¯ while A ∩ B ⊂ A¯ ∩ B. ¯ (b) Show that A ∪ B = A¯ ∪ B, (c) Show that A◦ = M \Ac , and that (A∩B)◦ = A◦ ∩B ◦ , while (A∪B)◦ ⊃ A◦ ∪B ◦ . (d) Show that ∂A = A¯ ∩ Ac , M \ ∂A = A◦ ∪ (Ac )◦ , A¯ = A ∪ ∂A, and A◦ = A \ ∂A. Definition 1.2 (Bases and subbases) (a) Call B a base for the topology U if B ⊂ U and if each nonvoid U ∈ U satisfies U = ∪V for some V ⊂ B. We then say that B generates U. (b) A family of sets S is a subbase for U if ∪S = M and the family of all finite intersections of members of S is a base for U. 95

96

CHAPTER 6. GENERAL TOPOLOGY AND HILBERT SPACE

Exercise 1.2 (Bases and subbases) (a) B is a base for a topology U on M ≡ ∪B if and only if for each x ∈ M and each neighborhood U ∈ U of x there exists a B ∈ B such that x ∈ B ⊂ U . (b) B is a base for some topology on M ≡ ∪B if and only if whenever B1 and B2 are members of B and x ∈ B1 ∩ B2 , there is a B3 ∈ B such that x ∈ B3 ⊂ B1 ∩ B2 . (c) Any collection S of subsets of M having ∪S = M is a subbase for a unique topology on M . Exercise 1.3 (Relative topology) Let (M, U) be a topological space. First let N ⊂ M , and then let V ≡ {U ∩ N : U ∈ U}. (a) Then V is a topology (the relative topology) for N . (b) Moreover, B ⊂ N is open (closed) with respect to V if and only if there is an A ⊂ M that is open (closed) with respect to U for which B = A ∩ N . ¯ ∩ N , while the interior (c) The closure cl(B|N ) of an arbitrary B in (N, V) equals B c c int(B|N ) of an arbitrary B in (N, V) equals ( B ) ∩ N .

Convergence Definition 1.3 (Sequences) A sequence is a function x whose domain is the natural numbers {1, 2, · · ·}, and we let xn ≡ x(n). Suppose x takes values xn in some space (M, U). Then the sequence is said to converge to some x0 ∈ M if for each neighborhood U of x0 there exists a natural number nU such that xn ∈ U for all n ≥ nU . We write xn → x0 or limn xn = x0 to denote that the sequence xn converges to x0 . Definition 1.4 (Nets) A partial ordering is a relationship R that is reflexive (xRx), antisymmetric (xRy and yRx implies x = y), and transitive (xRy and yRz implies xRz). A directed set is a set I under a relationship for which, for any i, j ∈ I there is a k ∈ I satisfying i k and j k. A net is any function whose domain is such a directed set I (with xi ≡ x(i)) taking values in some topological space (M, U). The net {xi }i∈I is said to converge to a value x0 in M if for each neighborhood U of x0 there is an iU ∈ I for which xi ∈ U for all i iU . [The set of all neighborhoods of a point constitutes a directed set. So does the set of all partitions of an interval.]

Compactness Definition 1.5 (Compactness) (a) A collection A of subsets of M covers the subset B of M if B ⊂ ∪A; and A is called an open cover of B if all sets A in A are open. A set A is called compact if every open cover A of A has a finite subcover V (that is, A ⊂ ∪V, where V ⊂ A contains only finitely many subsets of A). [This is called the Heine–Borel property.] (b) A ⊂ M is compact (relatively compact) if it (its closure) is compact in the relative topology. (c) The collection A has the finite intersection property if ∩F = ∅ for each finite subcollection F of subsets of A.

1.

GENERAL TOPOLOGY

97

Exercise 1.4 (a) A closed subset of a compact space (M, U) is compact. (b) Let B be a base for (M, U), and let A ⊂ M . Then A is compact if and only if every cover of A by members of the base B admits a finite subcover. (c) (Alexander’s lemma) Let S be a subbase for the topological space (M, U). Then M is a compact space if and only if each cover of M by members of the subbase S admits a finite subcover. (d) A space (M, U) is compact if and only if each family of closed sets with the finite intersection property has a nonempty intersection. (e) (M, U) is compact if and only if every net on M has at least one accumulation point in M .

Separation Definition 1.6 (Separation) (a) (M, U) is called a Hausdorff space if for each x = y in M there are disjoint open sets U and V such that x ∈ U and y ∈ V . (b) (M, U) is called a normal space if for each pair of disjoint closed sets A and B there are disjoint open sets U and V having A ⊂ U and B ⊂ V . (c) (M, U) is called separable if it contains a countable subset D that is dense in M ; ¯ = M. here D is called dense if D (d) (M, U) is called first countable if for each x ∈ M there is a family Nx of neighborhoods Nx of x such that for each neighborhood U of x we have x ∈ Nx ⊂ U for some neighborhood Nx in Nx . (e) (M, U) is called second countable or perfectly separable if the topology U admits a countable base B. Exercise 1.5 (a) A sequence with values in a Hausdorff space converges to at most one point. ¯ (b) If all values xn are in A and xn → x0 , then x0 ∈ A. Exercise 1.6 (a) A perfectly separable (M, U) is separable. (b) (Lindel¨ of) An open cover of a perfectly separable space necessarily admits a countable subcover. Exercise 1.7 (a) A compact subset of a Hausdorff space (M, U) is closed. (b) A compact Hausdorff space is normal. (c) A space (M, U) is normal if and only if for each closed C and open U with C ⊂ U there exists an open V with C ⊂ V ⊂ V¯ ⊂ U . (d) A subset of a Hausdorff (normal) space (M, U) is Hausdorff (normal) in its relative topology.

Continuity Definition 1.7 (Continuous functions and homeomorphisms) (a) Let f : M → N for topological spaces (M, U) and (N, V). Call the function (or mapping) f continuous at x if for each neighborhood V of f (x) there is a neighborhood U of x for which f (U ) ⊂ V .

98


(b) A 1-to-1 function that maps onto the image space is called a bijection. (c) A 1-to-1 bicontinuous function f (that is, both f and f −1 are continuous and onto) is called a homeomorphism. Exercise 1.8 (Conditions for continuity)

The following are equivalent:

(a)

f is a continuous function.

(b)

Inverse images of all open sets are open.

(c)

Inverse images of all closed sets are closed.

(d)

Inverse images of all subbasic open sets are open.

(e)

For each x ∈ M and for each net {xi }i∈I converging to this x, {f (xi )}i∈I → f (x) also converges.

(f)

f (A¯ ) ⊂ f (A)

(g)

¯ for all subsets B of N . f −1 (B) ⊂ f −1 (B)

(h)

f −1 (B ◦ ) ⊂ [f −1 (B)]◦

for all subsets A of M.

for all subsets B of N .

Exercise 1.9 (a) (Dini) If fn are continuous real-valued functions on a compact (M, U) for which fn (x) f (x) for each x ∈ M , then fn converges uniformly to f . (b) A uniform limit of bounded and continuous functions fn is also bounded and continuous. Exercise 1.10 (Continuity and compactness) (a) If f : M → N is continuous and M is compact, then f (M ) is compact. [Thus [0, 1] cannot be mapped continuously onto R.] (b) The composition of continuous functions is continuous. (c) If f : M → N is 1-to-1 and continuous (that is, it is a bijection) where M is compact and N is Hausdorff, then f is a homeomorphism. Exercise 1.11 (Urysohn’s lemma) (M, U) is normal if and only if for each pair of disjoint closed sets A and B there is a continuous function f : M → [0, 1] having f (x) = 0 for all x ∈ A and f (x) = 1 for all x ∈ B. Exercise 1.12 (Tietze) Let (M, U) be a normal space, and let A be a closed subset. If the function f : A → [−1, 1] is continuous, then f admits a continuous extension from M to [−1, 1].

1.

GENERAL TOPOLOGY

99

Product Spaces Definition 1.8 (Product topology) Let P {Ma } ≡ a∈A Ma denote the product set associated with the individual topological spaces (Ma , Ua ) over the index set A; this product set is defined to be {x : x is a function on A with xa ≡ x(a) ∈ Ma }. The a0 th projection function πa0 : P {Ma } → Ma0 is defined by πa0 (x) = xa0 . Let S ≡ {πa−1 (U ) : U ∈ Ua and a ∈ A}. Then S is the subbase for a topology called the product topology. Exercise 1.13 (a) Projection functions are continuous in the product topology. (Recall definition 5.4.1 for the projection maps πt1 ,...,tk (·).) (b) (Tychonoff) P {Ma } with the product topology is compact if and only if each (Ma , Ua ) is compact. (c) The product of countably many separable topological spaces is a separable space in the product topology. ¯a } for all choices Ba ⊂ Ma , and thus the product of closed sets (d) P {Ba } = P {B is closed in the product topology. (e) (P {Ba })◦ = P {Ba◦ } for all choices Ba ⊂ Ma , provided that A is finite. (f) If each (Ma , Ua ) is Hausdorff, then the product topology is Hausdorff. (g) Let (M, U) be a topological space and suppose fa : M → Ma is a continuous function for each a ∈ A. Define f : M → P {Ma } by πa f (x) = fa (x) for each x ∈ M . Then f is continuous in the product topology.

Local Compactness Definition 1.9 (Locally compact) Let (M, U) be a topological space. Let x ∈ M . The space is locally compact at x if x has a neighborhood with compact closure (that is, for each U ∈ U having x ∈ U there exists a V ∈ U with x ∈ V ⊂ U with V¯ compact). The space is locally compact if it is locally compact at each point. [Rn and (0, 1) are locally compact.] Exercise 1.14 (Locally compact Hausdorff spaces are nice) Let M be a locally compact Hausdorff space. Let C ⊂ U ⊂ M , where C is compact and U is open. (a) Then there is an open set V whose closure is compact having C ⊂ V ⊂ V¯ ⊂ U . (b) (Urysohn’s lemma) There is a continuous function f : M → [0, 1] for which f (x) equals 1 for x ∈ C and 0 for x ∈ U c . (c) There is an open set V and a compact set D where C ⊂ V ⊂ D ⊂ U with V a countable union of compact sets and D a countable intersection of open sets. Sets of the type V form a base for the topology. Exercise 1.15 (One-point compactification) The one-point compactification of any locally compact Hausdorff space (M, U) makes it a compact Hausdorff space. ¯ ≡ M ∪ {p}, and designate the open sets U ¯ to be all open sets U That is, let M c ¯ , U) ¯ is plus all sets {p} ∪ C with C compact and {p} the one new point. Then (M a compact Hausdorff space.

100


Category Definition 1.10 (Category) A subset A of (M, U) is nowhere dense if (A¯ )◦ = ∅. If A can be written as a countable union of nowhere dense sets, then it is said to be of the first category; otherwise, it is said to be of the second category. Exercise 1.16 (Baire category theorem) second category.

A locally compact Hausdorff space is

Topological Properties Definition 1.11 (Topologically equivalent) If there exists a homeomorphism between the two topological spaces (M, U) and (N, V) (that is, there exists a 1-to-1 bicontinuous mapping between them), then there exist 1-to-1 correspondences both between points and between open sets. We then say that these two topological spaces are topologically equivalent. Thus any properties that are defined solely in terms of the open sets either hold or fail simultaneously in the two spaces. Such concepts are called topological concepts. [These include closed set, closure, interior, boundary, accumulation point, compactness, separability, perfect separability, local compactness, category, continuity, and being either Hausdorff or normal.] We are about to introduce metric spaces. We will soon see that two different metrics can induce the same topology. Thus distance in a metric space is not a topological property. A homeomorphism f that leaves distance unchanged (thus, d2 (f (x), f (y)) = d1 (x, y)) is called an isometry.

2.

METRIC SPACES

2

Metric Spaces

101

Definition 2.1 (Metric space) (a) A semimetric space (M, d) consists of a set M together with a semimetric d that satisfies (1)

d(x, y) = d(y, x) ≥ 0,

d(x, z) ≤ d(x, y) + d(y, z),

and d(x, x) = 0

for all x, y, z ∈ M . If d(x, y) = 0 implies x = y, then d is called a metric and (M, d) is called a metric space. [If all d(x, y) ≤ (some finite c), then d is called a bounded metric.] Call the set (2)

Sr (x) ≡ {y ∈ M : d(x, y) < r} an open sphere or ball about x of radius r.

(b) The collection of all such spheres is the base for a topology, called the metric topology, and is denoted by Ud . The abbreviation (M, d, Ud ) refers to the metric space with its metric topology, and we call it a topological metric space. (c) Let (M, U) be a topological space. Suppose there exists a metric d on M whose open balls form a base for the topology U. Then the topology U is said to be metrizable. Exercise 2.1 (Equivalent metrics) (a) Two metrics d and ρ on a set M lead to equivalent topologies if and only if for each x ∈ M and each > 0 there is a δ > 0 such that for all y ∈ M , d(x, y) < δ implies ρ(x, y) <

and

ρ(x, y) < δ implies d(x, y) < .

[If only the first holds, then Ud ⊂ Uρ , meaning that the ρ topology is finer.] Such equivalent metrics lead to the same continuous functions. If both sets of δ values do not depend on x, then the metrics are called uniformly equivalent metrics and lead to the same uniformly continuous functions. (b) Define four equivalent metrics on Rn . (c) ρ ≡ d/(1 + d) defines a metric equivalent to the d metric on (M, d); and ρ is a bounded metric. Exercise 2.2 (a) The closed sphere {y : d(x, y) ≤ r} equals Sr (x) with respect to the metric topology Ud . (b) Any metric space is homeomorphic to a metric space with a bounded metric. (c) A metric space is perfectly separable if and only if it is separable. (d) A compact metric space is separable and perfectly separable. (e) Any metric space is Hausdorff, normal, and first countable. (f) The product of countably many metrizable in the product spaces is metrizable n topology, with the bounded metric d(x, y ) ≡ ∞ 1 ψ(dn (xn , yn ))/2 , where we define ψ(t) ≡ t/(1 + t). If each is separable, then so is the product. (g) The relative topology on a subset A of the metric space (M, d, Ud ) is the same as the metric topology for (A, d). (h) Every subspace of a separable metric space is separable in its relative topology. (i) A separable metric space is locally compact if and only if it is the union of U1 ⊂ U2 ⊂ · · · with each U¯n a compact subset of Un+1 .

102


Completeness Definition 2.2 (Complete) (a) We will call a sequence x in a metric space (M, d) a Cauchy sequence if d(xm , xn ) → 0 as m ∧ n → ∞. (b) The topological metric space (M, d, Ud ) is called complete when every Cauchy sequence x having values xn in M converges to a member of M . Exercise 2.3 (a) A sequence x taking values in a metric space (M, d) converges to a member x0 of M if and only if d(xn , x0 ) → 0. (b) Every compact metric space is complete, separable, and perfectly separable. (c) Every closed subspace of a complete metric space is complete. (d) A metric space in which every sequence has a convergent subsequence is perfectly separable.

Compactness Definition 2.3 (a) A metric space (M, d) is totally bounded if for every > 0 there is a finite subset F of M such that for each x ∈ M there exists a y ∈ F sufficiently close to x to have d(x, y) < . (b) Let diam(A) ≡ sup{d(x, y) : x, y ∈ A} denote the diameter of A. The metric space is called bounded if diam(M ) is finite. Exercise 2.4 (Compactness) The following conditions are equivalent to the compactness of a subset K of the metric space (M, d, Ud ). (a)

(Heine–Borel property = compactness, by definition): Every open cover of K has a finite subcover.

(b)

(Bolzano–Weierstrass property): Every infinite subset of K has a limit point in K.

(c)

(Sequential compactness (or relative compactness)): Every sequence in K has a subsequence converging to a point in K.

(d)

(Countable compactness): Every countable open cover of K has a finite subcover.

(e)

K is totally bounded and complete.

(f)

K is totally bounded, and any sequence of closed spheres Sn in K whose diameters → 0 has ∞ 1 Sn = {some singleton x} ∈ K.

Exercise 2.5

A subset of Rn is compact if and only if it is closed and bounded.

Exercise 2.6 If the closed subset A and the compact subset B are disjoint, then we have d(A, B) ≡ inf{d(x, y) : x ∈ A, y ∈ B} > 0.

2.

METRIC SPACES

103

Definition 2.4 (Covering numbers) Let (M, d) be a totally bounded metric space. Let > 0. Let N ( , M ) denote the minimum n for which M = ∪n1 Ak for sets Ak all having diam(Ak ) ≤ 2 . Let D( , M ) be the largest number m of points xk in M having d(xi , xj ) > for all i = j. These are called covering numbers. Exercise 2.7

Show that N ( , M ) ≤ D( , M ) ≤ N ( /2, M ).

Spaces of Functions Exercise 2.8 (Uniform norm) Let Cb (M ) denote the collection of all bounded and continuous functions from the topological space (M, U) to R, and define the uniform norm · on the functions of Cb (M ) by f ≡ sup{|f (x)| : x ∈ M }. (a) A uniform limit of bounded, continuous functions is bounded and continuous. (b) Moreover, (Cb (M ), · ) defines a complete metric space. (c) Let (M, U) be a compact Hausdorff space. Then (Cb (M ), · ) is separable if and only if (M, U) is metrizable. Definition 2.5 (Uniform continuity) A mapping f from a metric space (M1 , d1 ) to a metric space (M2 , d2 ) is uniformly continuous if for all > 0 there exists δ > 0 for which d2 (f (x), f (y)) < whenever d1 (x, y) < δ . Definition 2.6 (Equicontinuity) (a) A collection F of bounded and continuous functions from a topological space (M, U) to a metric space (N, d) is equicontinuous if for every > 0 and each x ∈ M there is a neighborhood U of x for which d(f (x), f (y)) < for all y ∈ U and all f ∈ F. (b) In fact, if (M, ρ, U = Uρ ) is a metric space for which d(f (x), f (y)) < for all f ∈ F whenever ρ(x, y) < δ , then the functions of F are called uniformly equicontinuous. Exercise 2.9 A continuous function (an equicontinuous family of functions) from a compact metric space into another metric space is uniformly continuous (is uniformly equicontinuous). Exercise 2.10 (Ascoli’s theorem) Let F be an equicontinuous class of functions from a separable metric space (M, d) to a metric space (N, ρ). Let fn denote a sequence of such functions for which cl({fn (x) : n ≥ 1}) is compact for each x ∈ M . Then there is a subsequence fn that converges pointwise to an f that is continuous on M , and this convergence is uniform on any compact subset K of M . Exercise 2.11 (Arzel` a’s theorem) (a) A collection F of real-valued, bounded, and continuous functions f on a compact topological space (M, U) is a totally bounded subset of (Cb (M ), ·) if and only if it is a collection of uniformly bounded and uniformly equicontinuous functions. (b) A subset F of (Cb ([a, b]), · ) is compact if and only if F is closed, bounded, and equicontinuous with respect to · .

104

3


Hilbert Space

Definition 3.1 (i) (Vector space) Let V be a real (or a complex) vector space; that is, x + y ∈ V and c x ∈ V for all x, y in the commutative group V and for all real scalars c ∈ R (or all complex scalars c ∈ C). Moreover, c(x + y) = cx + cy, (c + d)x = cx + dx, c(dx) = (cd)x and 1x = x must be satisfied. (ii) (Inner product space) A vector space H is an inner product space if for each pair x, y ∈ H there is a real (or a complex) number x, y ≡ (the inner product of x and y) satisfying: (a)

c x, y = cx, y

for all x, y ∈ H, and all c ∈ R

(b)

y, x = x, y

( or satisfying, y, x = x, y ),

(c)

x + y, z = x, z + y, z

(d)

x, x ≥ 0

( or all c ∈ C ),

for all x, y, z ∈ H,

for all x ∈ H, with equality only if x = 0.

(iii) (Norm) We use (d) to define the norm (denoted by · ) of x; thus (e) so that x = x, x. x2 ≡ x, x , (iv) (Orthogonal, and orthogonal complement) We agree that (f)

x⊥y

means that x, y = 0,

and we then say that x is orthogonal to y. We also let (g)

M ⊥ ≡ {x : x ⊥ y for all y ∈ M },

and we call it the orthogonal complement of M . (v) (Hilbert space) If H is an inner product space that is complete (with respect to the metric defined by the norm x − y ), then H is a Hilbert space. Let µ denote a fixed positive measure. Example 3.1 (L2 is a Hilbert space) Then let L2 (µ) ≡ {f : f 2 dµ < ∞}, which is a Hilbert space with inner product f, g = f g dµ. Thus for any probability measure P on a measurable space (Ω, A), the collection of rvs L2 (P ) is a Hilbert space with X, Y ≡ E(XY ). This follows from the completeness that was presented in exercise 3.5.1. 2 Proposition 3.1 (Elementary properties) Let x, y ∈ H. (a)

(Cauchy–Schwarz)

|x, y| ≤ x y.

(b)

(Triangle inequality)

x + y ≤ x + y.

(c)

(Pythagoras theorem)

x + y2 = x2 + y2

(d)

(Parallelogram law)

x + y2 + x − y2 ≤ 2 [ x2 + y2 ].

(e)

(Polarization identity)

4x, y = x + y2 − x − y2

(f)

(Linear functionals) x → x, y, x → y, x, are uniformly continuous linear functionals on H.

(g)

The only continuous linear functionals on H are those in (f).

if x ⊥ y.

when c’s are real.

and x → x

3.

HILBERT SPACE

105

Definition 3.2 (Subspaces) We say that M ⊂ H is a subspace of the vector space H if M is also a vector space. [It is enough if x + y and c x are in M for all x, y ∈ M and all scalars c.] We say that M ⊂ H is a closed subspace if it is a subspace that contains all of its own limit points. Exercise 3.1 Let M be a subspace of the Hilbert space H. (a) Then M ⊥ is a closed subspace of M . (b) (M ⊥ )⊥ = M . The following theorem gives fundamental properties of a general Hilbert space. Theorem 3.1 (Orthogonal projections) Let M be a closed subspace of H. (i) Unique mappings P and Q on H exist such that (1)

x = P x + Qx

for all x ∈ H

with P x ∈ M and Qx ∈ M⊥ .

This P and Q are called the orthogonal projections of H onto M and M⊥ . (ii) Specifically and additionally, P and Q satisfy: (2)

x2 = P x2 + Qx2 ,

(3)

x − P x = inf{ x − y : y ∈ M}

(4)

P and Q are linear maps.

for each x ∈ H,

(iii) If M is a proper subset of H, then there exists a y = 0 in H such that y ⊥ M . Moreover, the space spanned by M and y is closed. Exercise 3.2

Prove theorem 3.1.

Example 3.2 (HD is a closed subspace of H) Let (5)

HD ≡ {X ∈ L2 (P ) : F(X) ⊂ D},

where D denotes a fixed sub σ-field of the basic probability space (Ω, A, P ). Then exercise 3.5.1 shows that HD is a closed subspace of H ≡ L2 (P ). 2 Exercise 3.3

Present the details for the assertion of example 3.2.

Definition 3.3 (Independent, span, orthonormal, Fourier coefficient, and basis) (i) Call vectors {x1 , . . . , xn } in V linearly independent if c1 x1 + · · · + cn xn = 0 implies c1 = · · · = cn = 0. Call the set Z linearly independent if all of its finite subsets are linearly independent. Let the span of Z (denoted by S[Z]) denote the set of all finite linear combinations of elements of the set Z. (ii) If Z ≡ {za : a ∈ A} is a subset of H indexed by A, then the elements of Z are called orthonormal if za , zb = 0 for all a, b ∈ A, while za = 1 for all a ∈ A. Moreover, the values (6)

xA (a) ≡ x, za , for all a ∈ A, are called the Fourier coefficients of x

relative to Z . A maximal orthonormal set is called an orthonormal basis.

106


Theorem 3.2 (Coordinates) Let vectors {z1 , . . . , zn } be orthonormal in the Hilbert space H. n (i) If x = 1 ci zi , then for each 1 ≤ k ≤ n, (7)

ck = x, zk , the z1 , . . . , zn are linearly independent, and x2 =

n

2 1 |ck |

.

(ii) For each x ∈ H and scalars c1 , . . . , cn we have n n x − 1 x, zk zk ≤ x − 1 ck zk , (8) with equality if and only if ck = x, zk . Moreover, n (9) 1 x, zk zk is the orthogonal projection of x onto the span S[z1 , . . . , zn ]. n Call d(x, S[z1 , . . . , zn ]) ≡ inf{x − 1 ck zk : ck ’s are scalars} the distance from x to the span S[z1 , . . . , zn ]; then (10) d2 (x, S[z1 , . . . , zn ]) = x2 − n1 |x, zk |2 . (iii) (Bessel’s inequality) For any orthonormal set {z1 , . . . , zn }, 2 2 2 (11) a∈A |xA (a)| = a∈A |x, za | ≤ x . Thus each x ∈ H has at most a countable number of nonzero Fourier coefficients. (iv) Let 2 (A) denote L2 (A, 2A , counting measure). We rewrite Bessel’s inequality as xA ∈ 2 (A) with xA 2 ≤ x. (Then, of course, xA − yA 2 ≤ x − y.) Theorem 3.3 (Orthonormal bases) The following are equivalent:

Let {za : a ∈ A} be an orthonormal set.

(a)

{za : a ∈ A} is an orthonormal basis,

(b)

The span of these za ’s is · -dense in H, x2 = a∈A |xA (a)|2 for all x ∈ H, (Parseval’s identity) x, y = a∈A xA (a) yA (a)

(c) (d)

for all x, y ∈ H.

Theorem 3.4 (H is isomorphic to 2 ) (a) Suppose the Hilbert space H has an orthonormal basis {za : a ∈ A}. Then the mapping x → xA maps H onto 2 (A). And it is an isomorphism in that it preserves inner products via the correspondence (12) x, y = xA , yA 2 ≡ xA (a)yA (a) for all x, y ∈ H. a∈A

(b) Every Hilbert space possesses an orthonormal basis. Thus every Hilbert space H is isomorphic to some 2 (A). (c) Every orthonormal basis for the Hilbert space H has the same cardinality. (d) H is separable if and only if H contains an orthonormal basis that is finite or countably infinite. Exercise 3.4 (Gram–Schmidt) From any n linearly independent vectors one can define n orthonormal vectors that have the same span.

Chapter 7

Distribution and Quantile Functions 1

Character of Distribution Functions

Let X : (Ω, A, P ) → (R, B) denote a rv with distribution function (df ) FX , where (1)

FX (x) ≡ P (X ≤ x)

for − ∞ < x < ∞.

Then F ≡ FX was seen earlier to satisfy (2)

F is and right continuous, with F (−∞) = 0 and F (+∞) = 1.

Because of the following proposition, any function F satisfying (2) will be called a df. [If F is , right continuous, 0 ≤ F (−∞), and F (+∞) ≤ 1, we earlier agreed to call F a sub-df. As usual, F (a, b] ≡ F (b) − F (a) denotes the increments of F , and ∆F (x) ≡ F (x) − F− (x) is the mass of F at x.] Proposition 1.1 (There exists an X with df F ) If F satisfies (2), then there exists a probability space (Ω, A, P ) and a rv X : (Ω, A, P ) → (R, B) for which the df of X is F . We write X ∼ = F. Proof. The corollary to the correspondence theorem (theorem 1.3.1) shows that there exists a unique probability distribution P on (Ω, A) ≡ (R, B) for which P ((a, b]) = F (b) − F (a) for all a ≤ b. Now define X(ω) = ω for all ω ∈ R to be the identity function on R. 2 Theorem 1.1 (Decomposition of a df) (3)

Any df F can be decomposed as

F = Fd + Fc = Fd + Fs + Fac = (Fd + Fs ) + Fac ,

where Fd , Fc , Fs , and Fac are the unique sub-dfs of the following types (unique among those sub-dfs equal to 0 at −∞): (4) Fd is a step function of the form j bj 1[aj ,∞) (with all bj > 0). 107

108

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

(5)

Fc is continuous.

(6)

Fs is singular, with its measure orthogonal to Lebesgue measure. · Fac = −∞ fac (y) dy for some fac ≥ 0 that is finite, measurable, and unique a.e. λ.

(7)

Proof. Let {aj } denote the set of all discontinuities of F , which can only be jumps; and let bj ≡ F (aj ) − F− (aj ). There can be only a countable number of jumps, since the number of jumps of size exceeding size 1/n is certainly bounded by n. Now define Fd ≡ j bj 1[aj ,∞) , which is obviously and right continuous, since Fd (x, y] ≤ F (x, y] 0 as y x (the inequality holds, since the sum of jump sizes over every finite number of jumps between a and b is clearly bounded by F (x, y], and then just pass to the limit). Define Fc = F − Fd . Now, Fc is , since for x ≤ y we have Fc (x, y] = F (x, y] − Fd (x, y] ≥ 0. Now, Fc is the difference of right-continuous functions, and hence is right continuous; it is left continuous, since for x y we have (a) Fc (x, y] = F (x, y] − x z is a continuity point of F . Thus lim sup Fn−1 (t) ≤ z, since there are continuity points x that z. Summary Fn−1 (t) → F −1 (t) for all but at most a countably infinite number of t’s (namely, for all but those t’s that have multiply realizable t-quantiles; these correspond to the heights of flat spots of F , and these flat spot heights t are exactly the discontinuity points of K). That is, Kn →d K. The proof of the converse is virtually identical. 2

Exercise 3.1 (Left continuity of K) Show that K(t) = F −1 (t) is left continuous on (0, 1). [Note that K is discontinuous at t ∈ (0, 1) if and only if the corresponding Ut is nonvoid (see theorem 7.2.1). Likewise, the jump points cj and the jump sizes dj of K(·) are equal to the t values and the λ(Ut ) values of the multiply realizable t-quantiles.] [We earlier agreed to use ai and bi for the jump points and jump sizes of the associated df F .]

3.

THE QUANTILE TRANSFORMATION

Exercise 3.2 (Properties of dfs) F ◦ F −1 (t) ≥ t

113

(i) For any df F we have

for all 0 ≤ t ≤ 1,

and equality fails if and only if t ∈ (0, 1) is not in the range of F on [−∞, ∞]. (ii) (The probability integral transformation) If X has a continuous df F , then F (X) ∼ = Uniform(0, 1). In fact, for any df F , P (F (X) ≤ t) ≤ t

for all 0 ≤ t ≤ 1 ,

with equality failing if and only if t is not in the closure of the range of F . (iii) For any df F we have F −1 ◦ F (x) ≤ x

for all − ∞ < x < ∞ ,

and equality fails if and only if F (y) = F (x) for some y < x. Thus P (F −1 ◦ F (X) = X) = 0

whenever X ∼ = F.

(iv) If F is a continuous df and F (X) ∼ = Uniform(0, 1), then X ∼ = F. Proposition 3.2 (The randomized probability integral transformation) Let X denote an arbitrary rv. Let F denote its df, and let (aj , bj )’s denote an enumeration of whatever pairs (jump point, jump size) the df F possesses. Let η1 , η2 , . . . denote iid Uniform(0, 1) rvs (that are also independent of X). Then (8) ξ˙ ≡ F (X) − j bj ηj 1[X=aj ] ∼ = Uniform(0, 1), (9)

˙ = K(ξ). X = F −1 (ξ)

[We have reproduced X from a Uniform(0, 1) rv that was defined using both X and some independent extraneous variation. Note figure 3.1.] Proof. We have merely smoothed out the mass bj that F (X) placed at F (aj ) by subtracting the random fractional amount ηj bj of the mass bj . 2 Exercise 3.3 (Change of variable) Suppose that rvs X ∼ = F and Y ∼ = G are −1 related by G(H) = F and X = H (Y ), where H is right continuous on the real line with left-continuous inverse H −1 . (a) Then set g, X, µ, µX , A in the theorem of the unconscious statistician equal to g, H −1 , G, F, (−∞, x] to conclude that (10) g(H −1 ) dG = g dF, (−∞,H(x)]

(−∞,x]

since (H −1 )−1 ((−∞, x]) = {y : H −1 (y) ≤ x} = (−∞, H(x)]. (b) Making the identifications G = I, H = F , and Y = ξ ∼ = Uniform(0, 1) gives especially (via part (a), or via (2) and (3)) −1 (11) g(F (t)) dt = g dF [0,F (x)]

(−∞,x]

for arbitrary df F and any measurable g.

114


Proof. We now prove proposition 1.2.3. Let D be a subset of [0, 1] that is not Lebesgue measurable; its existence is guaranteed by proposition 1.2.2. Let B ≡ F −1 (D). Then B is a subset of the Cantor set C of example 7.1.1. Since λ(C) = 0 and B ⊂ C, it follows that B is a Lebesgue set with λ(B) = 0. We now assume that B is Borel set (and look for a contradiction). We note that F (B) is also a Borel set, since F (being ) is Borel measurable. However, F is 1-to-1 on C, and so F (B) = D. That is, D is a Borel set, and hence D is a Lebesgue set. This is the contradiction we sought. 2

The Elementary Skorokhod Construction Theorem Let X0 , X1 , X2 , . . . be iid F . Then Xn →d X0 , but the Xn do not converge to X0 in the sense of →a.s. , →p , or →r . However, when general Xn →d X0 , it is possible to replace the Xn ’s by rvs Yn having the same (marginal) dfs, for which the stronger result Yn →a.s. Y0 holds. Theorem 3.2 (Skorokhod) Suppose that Xn →d X0 . Define ξ(ω) = ω for each ω ∈ [0, 1] so that ξ ∼ = Uniform(0, 1) on (Ω, A, P ) ≡ ([0, 1], B ∩ [0, 1], λ), for Lebesgue measure λ. Let Fn denote the df of Xn , and define Yn ≡ Fn−1 (ξ) for all n ≥ 0. Let DK0 denote the at most countable discontinuity set of K0 . Then both Yn ≡ Kn (ξ) ≡ Fn−1 (ξ) ∼ = Xn ∼ = Fn

for all n ≥ 0

and

(12) Yn (ω) → Y0 (ω) Proof.

for all ω ∈ DK0 .

This follows trivially from proposition 3.1.

2

Exercise 3.4 (Wasserstein distance) For k = 1 or 2, define Fk ≡ {F : F is a df, and |x|k dF (x) < ∞} , dk (F1 , F2 ) ≡

1 0

|F1−1 (t) − F2−1 (t)|k dt

for all F1 , F2 ∈ Fk .

Show that both (Fk , dk ) spaces are complete metric spaces, and that (13)

dk (Fn , F0 ) → 0 Fn →d F0

and

(with all {Fn }∞ 0 ∈ Fk ) k |x| dFn (x) → |x|k dF0 (x) .

if and only if

4.

INTEGRATION BY PARTS APPLIED TO MOMENTS

4

Integration by Parts Applied to Moments

115

Integration by Fubini’s theorem or “integration by parts” formulas are useful in many contexts. Here we record a few of the most useful ones.

Integration by Parts Proposition 4.1 (Integration by parts formulas) Suppose that both the leftcontinuous function U and the right-continuous function V are monotone functions. Then for any a ≤ b we have both U+ (b)V (b) − U (a)V− (a) = (1) U dV + V dU and [a,b]

[a,b]

U (b)V (b) − U (a)V (a) =

(2)

U dV + (a,b]

V dU, [a,b)

written as where U+ (x) ≡ limy x U (y) and V− (x) ≡ lim yx V (y). [Symbolically, d(U V ) = U− dV + V+ dU , it means also that h d(U V ) = h [U− dV + V+ dU ] for any measurable h ≥ 0.] 1[x′ < y′]

b 1[x′ ≥ y′] a a

b

Figure 4.1 Integration by parts.

Proof.

We can apply Fubini’s theorem at steps (a) and (b) to obtain [U+ (b) − U (a)] [V (b) − V− (a)] = [a,b] { [a,b] dU } dV

(a)

=

(b)

= =

[a,b]

[a,b]

[1[x k). Exercise 4.2

Prove formulas (11)–(13).

Exercise 4.3

Prove formulas (14)–(16).

Exercise 4.4

Prove formula (17).

Exercise 4.5 value, say k.

Give an extension of (13) to arbitrary rvs in case r takes an integral

118


Exercise 4.6 (Integration by parts formulas) We showed in proposition 4.1 earlier that d(U V ) = U− dV + V+ dU . (i) Now show (noting that dU− = dU+ ) that (18)

dU 2 = U− dU + U+ dU = (2U + ∆U )dU

for ∆U ≡ U − U− .

(ii) Apply proposition 4.1 to 1 = U · (1/U ) to obtain (19)

d(1/U ) = −{1/(U+U− )} dU = −{1/(U (U + ∆U ))} dU.

(iii) Show by induction that for k = 1, 2, . . . we have k−1 i k−i−1 (20) dU k = ( i=0 U+ U− ) dU. Exercise 4.7 (21)

Show that for an arbitrary df F and any measurable h ≥ 0,

d(F/(1 − F )) = {1/((1 − F )(1 − F− ))} dF.

Exercise 4.8 For any df F we have [F (x + θ) − F (x)] dx = θ for each θ ≥ 0. ∼ (0, σ2 ) with df F . Then g(x) ≡ Exercise 4.9 (Stein) Suppose X = x is a density. (And g(x) = − −∞ y dF (y)/σ2 is also true.)

∞ x

y dF (y)/σ2

5.

IMPORTANT STATISTICAL QUANTITIES

5

Important Statistical Quantities

119

˘ ˜ ˇ Notation 5.1 (Trimming, Winsorizing, and Truncating, and dom(a, a)) Let dom(a, a ) denote [0, 1 − a ), (a, 1], or (a, 1 − a ) according as X ≥ 0, X ≤ 0, ˜ a,a (·) denote K(·) Winsorized outside dom(a, a ). Thus when or otherwise. Let K K+ (a) < 0 < K(1 − a ), (1)

˜ a,a (t) equals K+ (a), K(t), K(1 − a ) K as 0 < t ≤ a, a < t < 1 − a , 1 − a ≤ t < 1

˜ (while a ≡ 0 and K(a) ≡ K(0) if X ≥ 0, etc). Let ξ denote a Uniform(0, 1) rv. Let ˜ a,a (ξ) ≡ 1 K ˜ a,a (t) dt, µ ˜(a, a ) ≡ EK (2) 0 which is the (a, a )-Winsorized mean of the rv K(ξ), and let 2 ˜ a,a (ξ)] = 1 K ˜ a,a σ ˜ 2 (a, a ) ≡ Var[K ˜(a, a )2 (t) dt − µ 0 (3)

=

1 1 0

0

˜ a,a (s) dK ˜ a,a (t) [s ∧ t − st] dK

denote the (a, a )-Winsorized variance. For general X, let (4)

µ ˜(a) ≡ µ ˜(a, a) ,

σ ˜2 (a) ≡ σ ˜ 2 (a, a) ,

and

˜ a (·) ≡ K ˜ a,a (·) ; K

but µ ˜(a) ≡ µ ˜0,a if X ≥ 0, etc. We now let 0 ≤ kn < n − kn ≤ n denote integers, and then let an ≡ kn /n and an ≡ kn /n ,

so that 0 ≤ an < 1 − an ≤ 1 .

˜ n (·) denote K(·) Winsorized outside dom(an , an ). Let Let K 1−a µ ˇn ≡ µ ˇK (an , an ) ≡ an n K(t) dt, µ ˘n ≡ µ ˇK (an , an ) ≡ µn /(1 − an − an ), (5) ˜ n (ξ) ≡ 1 K ˜ n (t) dt, ˜K (an , an ) ≡ EK µ ñ ≡ µ 0

so that µ ˘n is the (an , an )-trimmed mean, µ ˇn is herein called the (an , an )-truncated (6)

µ ñ is the (an , an )-Winsorized mean, and mean of the rv K(ξ). Then let 2 ˜ n (ξ)] = 1 1 [s ∧ t − st] dK ˜ n (s) dK ˜ n (t) σ ñ2 ≡ σ ˜K (an , an ) ≡ Var[K 0 0

denote the (an , an )-Winsorized variance. When they are finite, the mean µ and variance σ2 satisfy 1 µ ≡ µK = 0 K(t) = x dF (x) = E(X) = EK(ξ), 1 1 2 σ2 ≡ σK = 0 0 [s ∧ t − st] dK(s) dK(t) = EX 2 − µ2 = EK 2 (ξ) − µ2 . Let a· ≡ inf{t : K(t) ≥ 0}, and let a◦ ≡ a· ∧ (1 − a· ). (But a· ≡ 0 if X ≥ 0, and a· ≡ 1 if X ≤ 0.) Now, (K − µ ñ )+ and (K − µ ñ )− denote the positive and negative parts of K − µ ñ , and let (7)

¯ n ≡ [K − µ ¯ n2 ≡ −[(K − µ K ˜ n ] and K ñ )− ]2 + [(K − µ ñ )+ ]2

on (0, 1).

120


In this context, we may wish to assume that both (8)

(kn ∧ kn ) → ∞

(an ∨ an ) → 0;

and

and perhaps we will also assume an /an → 1 and/or (kn − kn )/(kn ∧ kn ) → 0 . We will refer to kn , kn as the trimming/Winsorizing numbers and an , an as the trimming/Winsorizing fractions. Describe the case of (8) as slowly growing to ∞. Now suppose that Xn1 , . . . , Xnn is an iid sample with df F and qf K. Let Xn:1 ≤ · · · ≤ Xn:n denote the order statistics (that is, they are the ordered values of Xn1 , . . . , Xnn ). Let Kn (·) on [0, 1] denote the empirical qf that equals Xn:i on ((i − 1)/n, i/n], for 1 ≤ i ≤ n, and that is right continuous at zero. Now let ¯n ≡ 1 X Xnk = µKn (0, 0) n

1 ¯ n )2 = σ 2 (0, 0) (Xnk − X Kn n

n

(9)

n

Sn2 ≡

and

k=1

k=1

denote the sample mean and the “sample variance.” We also let

(10)

n−k 1 n ˇ Xn:i = µ ˇKn (an , an ) , Xn ≡ n

˘n ≡ X

i=kn +1

1 n − kn − kn

n−kn

Xn:i ,

i=kn +1

(11)

n−kn 1 ˜ Xn ≡ Xn:i + kn Xn−kn = µ ˜Kn (an , an ) kn Xn:kn +1 + n i=kn +1

denote the sample (an , an )-truncated mean, the sample (an , an )-trimmed mean, ˜ n:1 , . . . , X ˜ n:n denote the (an , an )and the sample (an , an )-Winsorized mean. Let X ˜ n . Now note that Winsorized order statistics, whose empirical qf is K 2 ˜ n:i = µ ˜ (an , a ); let S˜2 ≡ 1 ˜ n:i − Xñ )2 = σ ñ = 1 X X (X ˜K ñ (an , an ) n n Kn n i=1 n i=1 n

(12)

n

denote the sample (an , an )-Winsorized variance. Let σ ˘n2 ≡ σ ñ2 /(1 − an − an )2

and

S˘n2 ≡ Sñ2 /(1 − an − an )2 .

¯ n , Sn , X ˘ n , Sñ estimate µ, σ, µ Of course, X ˘n , σ ñ . We also define the standardized estimators √ ¯ √ ˘ √ ˇ n (Xn − µ) n (Xn − µ ˘n ) n (Xn − µ ˇn ) (13) Zn ≡ and Zˇn ≡ = , σ σ ˘n σ ñ and the studentized estimators √ ¯ n (Xn − µ) (14) Tn ≡ and Sn

√ T˘n ≡

˘n − µ n (X ˘n ) = S˘n

√

ˇn − µ n (X ˇn ) . Sñ

[The first formula for T˘n is for statistical application, while the second formula is for probabilistic theory.] We will very often assume that these independent rvs

5.

IMPORTANT STATISTICAL QUANTITIES

121

Xn1 , . . . , Xnn having df F and qf K are defined in terms of independent Uniform(0, 1) rvs ξn1 , . . . , ξnn via (see above (7.4.3)) (15)

Xnk ≡ K(ξnk )

for 1 ≤ k ≤ n.

(If we started with iid K rvs X1 , . . . , Xn and then defined Xnk ’s via (15) for the ξ˙k ’s of (7.3.8), then the Xnk ’s of (15) are just the original Xk ’s.) Thus the device of (15) is broadly useful. ¨ n to equal Xn:(n+1)/2 or (Xn:n/2 + Xn:n/2+1 )/2, We define the sample median X according as n is odd or even. In the previous context, let ξn:1 < · · · < ξn:n denote the order statistics of iid n ≡ (Rn1 , . . . , Rnn ) denote the ranks of these ξn1 , . . . , ξnn , Uniform(0, 1) rvs. Let R n is a and let Dn ≡ (Dn1 , . . . , Dnn ) denote their antiranks. Thus the rank vector R random permutation of the vector (1, 2, . . . , n) , while Dn is the inverse permutation; and these satisfy (16)

ξnDnk = ξn:k

and

ξnk = ξn:Rnk .

We will learn later that (17)

(ξn:1 , . . . , ξn:n ) and (Rn1 , . . . , Rnn ) are independent random vectors.

Such notation is used throughout the remainder of this book.

2

The Empirical DF Notation 5.2 (Empirical dfs and processes) Let X1 , X2 , . . . be iid with df F and qf K. The empirical df Fn of (X1 , . . . , Xn ) is defined by (18)

Fn (x) ≡

1 1 1(−∞,x] (Xk ) = 1[Xk ≤x] n n n

n

k=1

k=1

for − ∞ < x < ∞.

This is a step function on the real line R that starts at height 0 and jumps by height 1/n each time the argument reaches another observation as it moves from left to right along the line. We can think of Fn as an estimate of F . The important study of the empirical process √ (19) En (x) ≡ n [Fn (x) − F (x)] for x ∈ R will allow us to determine how this estimator Fn of F performs. We also let ξ1 , ξ2 , . . . be iid Uniform(0, 1), with true df the identity function I on [0, 1] and with empirical df (20)

Gn (t) ≡

1 1 1[0,t] (ξk ) = 1[ξk ≤t] n n n

n

k=1

k=1

for 0 ≤ t ≤ 1.

The corresponding uniform empirical process is given by √ (21) Un (t) ≡ n [Gn (t) − t] for t ∈ [0, 1].

122


If we now define an iid F sequence X1 , X2 , . . . via Xk ≡ F −1 (ξk ) = K(ξk ), then the empirical df and empirical process of these (X1 , . . . , Xn ) satisfy (22)

(Fn − F ) = [Gn (F ) − I(F )] on R valid for every ω,

and

En = Un (F )

on R,

as follows by (7.3.3). (If we use the ξ˙k ’s of (7.3.8), then the Fn on the left in (22) is everywhere equal to the Fn of the original Xk ’s.) Thus our study of properties of En can proceed via a study of the simpler Un , which is then evaluated at a deterministic F . (Recall also in this regard theorem 5.3.3 about probability being determined by the finite dimensional distributions.) 2

6.

INFINITE VARIANCE

6

Infinite Variance

123

Whenever the variance is infinite, the Winsorized variance σ ˜2 of the previous section ˜ a,a denote K completely dominates the square µ ˜2 of the Winsorized mean. Let K Winsorized outside (a, 1 − a ). Theorem 6.1 (Gnedenko–Kolmogorov) Every nondegenerate qf K satisfies 1−a 1−a 2 (1) |K(t)| dt}2 / a K (t) dt = 0 whenever EK 2 (ξ) = ∞, lim sup{ a a∨a →0

(2)

˜ 2 (ξ) → 1 as (a∨a ) → 0 ˜ a,a (ξ)]/EK Var[K a,a

whenever

EK 2 (ξ) = ∞ .

Proof. Let h be positive, continuous, symmetric about t = 12 , ↑ to ∞ on 1 1 [ 2 , 1), and suppose it satisfies Ch ≡ 0 h2 (t) dt < ∞. Let b ≡ 1 − a . Then Cauchy–Schwarz provides the bound b b b b { a |K(t)| dt}2 = { a h(t)|K(t)/h(t)| dt}2 ≤ a h2 (t) dt a [K 2 (t)/h2 (t)] dt. (a) Fix c ≡ c so close to zero that Ch /h2 (c) < . Fix c , and let a ∨ a → 0. Then b b b b { a |K(t)| dt}2 / a K 2 (t) dt ≤ Ch a [K 2 (t)/h2 (t)] dt/ a K 2 (t) dt

(c)

c b K 2 (t) dt ( a + 1−c ) K 2 (t) dt b 2 K (t) dt ≤ Ch + a h2 (1/2) h2 (c) 1−c b ≤ Ch { c K 2 (t) dt/h2 (1/2)}/ a K 2 (t) dt +

(d)

< 2

1−c

(b)

c

for a and b near enough to 0 and 1, since EK 2 (ξ) = ∞.

2 Then (2) follows from [aK+ (a) + a K(1 − a )]2 / [aK+ (a) + a K 2 (1 − a )] → 0.

2

Exercise 6.1 (Comparing contributions to the variance) Let K(·) be arbitrary. Establish the following elementary facts: =0 if EK 2 (ξ) < ∞, 2 2 2 (a) lim sup [aK+ (a) + a K (1 − a )]/˜ σ ≤1 if EK 2 (ξ) = ∞. a∨a →0 1−a 0.

And condition (7) holds if and only if σ (t) does not grow too fast, that is, if it is a slowly varying function of t as t → 0 (in the sense of the definition below). Thus it is appropriate to examine slow variation in the infinite-variance case. Should we

6.

INFINITE VARIANCE

125

examine it in the df domain or the qf domain? We shall do both, in the next section. And since the problem is so “central”, we shall do it very carefully. Often this problem is made rather difficult, by examining this problem in the context of the CLT and in the context of the general theory of slowly varying functions. But the equivalence of all the conditions in the next section follows in an elementary fashion from simple pictures and a dash of Cauchy–Schwarz. In fact, we will not even mention the CLT in the next section. The situation also becomes simpler if one ignores the general theory of slowly varying functions and uses the fact that F and K are monotone, and so no properties of slowly varying functions will be used in the next section either. Somewhere we want to connect with probability theory by showing that each of the equivalent conditions in the next section is also equivalent to (6) and to n ¯ n )2 / n = Xn2 − (X ¯ n )2 . Sn2 / σ (8) ˜ (1/n)2 →p 1, where Sn2 ≡ (Xk − X 1

The reader is also referred to section 11.3, where a major CLT presentation is made. (Actually, quite general necessary and sufficient conditions for normality are given, but all the proofs given are quite simple.) And the reader is referred to section 10.6, where the consistency of sample moments is studied in the infinitemoment case. By the time all this is put together, a huge list of equivalencies has been developed that includes (6), (7), (8), and all the conditions of section 7.7, section 10.6, and sections 11.2 – 11.3. Many are very interesting and very useful. Roughly, it is easier to develop new conditions in the context of the weak law of 2 large numbers (WLLN) [for the rvs Xnk ] than in the context of the CLT. Many of the conditions are also new. Finally, we can use the list of equivalencies developed in the next section to make simpler proofs in sections 10.6 and 11.3. The next theorem is just a teaser; it is established in section 11.3. Theorem 6.2 (A studentized CLT) Let Xn1 , . . . , Xnn ’s be row independent, with nondegenerate df F . Then the following are equivalent: n 2 (9) /n ]/νn →p 1 for any one specific νn > 0 for which n νn . [ 1 Xnk (10) (11)

Sn2 /˜ σn2 →p 1 . √ ñ − µ n [X ˜(1/n)]/˜ σ(1/n) →d N (0, 1) .

When any/all of these hold, then both √ ¯n − µ (12) n [X ˜(1/n)]/˜ σ(1/n) →d N (0, 1) √ ¯ (13) n [Xn − µ ˜(1/n)]/Sn →d N (0, 1) .

and

Definition 6.1 (Slowly varying functions) (a) Call L(·) > 0 slowly varying at 0 (written L ∈ R0 or L ∈ L) if it satisfies L(ct)/L(t) → 1 as t → 0, for each positive number c > 0. (b) The function l(·) > 0 on (0, ∞) is called slowly varying at ∞ (written as l ∈ U0 ) if it satisfies l(cx)/l(x) → 1 as x → ∞, for each positive number c > 0. Note that L(t) = log(1/t) and l(x) = log(x) are slowly varying. They are the prototypes. Note also that when σ2 is finite, the Winsorized variance function σ ˜2 (t) 2 2 2 2 is always slowly varying, since σ ˜ (ct)/˜ σ (t) → σ /σ = 1 for all c > 0.

126

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS Warning

Let us discuss the rest of this chapter. Strong forms of the classical WLLN and the CLT require some understanding of contributions to the mean and variance from the various parts of the distribution, and sections 7.6 – 7.11 are specifically geared to this. The author recommends reading only the current section 7.6 now, and that is just to have some rough idea as to what is here. Look quickly at the statement of theorem 7.7.1, but no more for now. Sections 7.6 and 7.7 are used “lightly” in section 10.6, on the consistent estimation of “moments.” But no other section in chapter 10 appeals to these results at all. Thus you are encouraged to wait until section 10.6 to see whether you wish to read them, even if you do eventually read them carefully. Even when you get to section 10.6, read that section first. The appeal to the present sections is purposefully minimized. When X has a finite variance, the classical CLT will be seen to hold. Even when X has infinite variance, we may still be able to claim a CLT. One approach is to see for which infinite-variance distributions the CLT still holds with only the minor modification of changing the centering and scaling constants. This is done in chapter 11 using the direct approach methods of analysis that we have already learned; we first establish the Berry–Esseen theorem, the Lindeberg–Feller theorem, and the “asymptotic normality condition” without any appeal whatsoever to results of sections 7.6 – 7.11. [In chapter 14 we shall again derive some of the CLT results, now using solely the characteristic function approach (that is, an approach based on the Fourier-analytic method to be studied in chapter 13).] Another approach to the CLT is to “trim” the sample observations Xn1 , . . . , Xnn . ˇ n is the average of the entire trimmed sample, and µ Roughly, if X ˇn and σ ñ are the truncated moment and the Winsorized standard deviation defined above, then √ ˇ n [Xn − µ ˇ n ]/˜ σn →d N (0, 1) for a huge class of dfs, provided only that we trim a slowly growing fraction of the observations. This is contained in chapter 16, and is based on the empirical process methods of chapter 12; it appeals to the companion inequalities of section 7.11 and the first part of section 7.10 also. In chapter 16, uniformity in the CLT conclusion is emphasized. The reader is again encouraged to wait. That is, read in chapter 16 until you feel the need to come back here. Section 7.8 is devoted to one result. It connects the section 7.7 qf domain t-results with its df domain x-results. Section 7.9 is a self-contained unit, used in chapter 15 on classic stable law results. Roughly, with a stable law the contributions from the two extreme tails must either balance or exhibit domination by one tail. If you trim the extremes away (in chapter 16), then you need to worry only about the moderately extreme observations. And this type of thing is done carefully at the end of section 7.10. I have been very conscious of downplaying the material of this chapter so that most results presented do not depend on it. In the end only sections 10.6, 11.2, and 11.3 (depending only on sections 7.6 – 7.7) and chapters 15 and 16 do. But if the reader wants to understand all the jots and tittles of the very best results in these cited locations, then this material does enter in. In fact, I hope that many of you will find this material very interesting and useful. The text is organized so that all general qf material is in just the present chapter. 2

7.

SLOWLY VARYING PARTIAL VARIANCE

7

Slowly Varying Partial Variance

127

Many facts about slowly varying functions can be learned from simple pictures. We concentrate here on just such facts. Lévy’s condition (14) has emerged as the necessary and sufficient condition of choice for the CLT; and we will now derive many equivalent ways to demonstrate it. [Necessary and sufficient conditions for consistent estimation of the variance parameters V (1/n) and σ ˜2 (1/n) as defined below are also equivalent to the conditions on the current list.] (Note (20) below.) Notation 7.1 Let Y denote an arbitrary rv (with df F and qf K); let X ≡ |Y |. Let FX and KX denote the df and qf of X. For 0 < t < 1, let xt ≡ KX (1 − t). Let Yt denote Y Winsorized outside [−xt , xt ]. Define 2 (1) v(t) ≡ KX (1 − t) , V (t) ≡ (t,1] v(s) ds , σ ˜2 (t) ≡ Var[Yt ] , (2)

q(t) ≡ KX (1 − t) ,

(3)

U (x) ≡

[0,x]

m(t) ≡

y 2 dFX (y) , M (x) ≡

(t,1]

q(s) ds ,

[0,x]

y dFX (y) ,

µ ˜(t) ≡ EYt , V˜ (t) ≡ EYt2 .

Theorem 7.1 (Partial variance, with symmetric Winsorizing) (A): The following [also referred to as (4)(A)–(12)(A)] are equivalent (as t → 0): (4)

[˜ σ 2 (ct) − σ ˜ 2 (t)]/˜ σ2 (t) → 0

for all 0 < c < 1.

(5)

[V˜ (ct) − V˜ (t)]/V˜ (t) → 0

for all 0 < c < 1.

(6)

r(t) ≡ t v(t)/V (t) → 0 .

(7)

[V (ct) − V (t)]/V (t) → 0

for all 0 < c < 1.

(8)

t [v(ct) − v(t)]/V (t) → 0 √ d(t) ≡ t [q(ct) − q(t)]/ V (t) → 0

for all 0 < c < 1.

(9)

(11)

t v(ct)/V (t) → 0 [m(ct) − m(t)]/ t V (t) → 0

(12)

r˜(t) ≡ t v(t)/˜ σ2 (t) → 0 .

(10)

for all 0 < c < 1. for all 0 < c < 1. for all 0 < c < 1.

(B): Specify a sequence an 0 as n → ∞ that satisfies lim sup an /an+1 < ∞. Conditions (4)(B)–(12)(B) are obtained by replacing t by an in (4)(A)–(12)(A). These conditions (4)(B)–(12)(B) are also equivalent to condition (4)(A). (C): Conditions (5)(C)–(11)(C) are obtained by replacing V (an ) or V˜ (an ) by σ ˜ 2 (an ) in the denominators of (5)(B)–(11)(B). Then (5)(C)–(11)(C) are also equivalent to the condition (4)(A). (D): The most useful choices are an ≡ /n (equivalently, an ≡ 1/n with c ≡ ), or the alternative an ≡ n /n with n 0 subject to lim sup n / n+1 < ∞.

128


(E): The following are equivalent (as x → ∞) to the previous conditions. [Any sequence xn to be specified below is assumed to satisfy lim(xn+1 /xn ) < ∞) .] (13)

U ∈ U0

(that is, U is slowly varying at ∞).

(14)

R(x) ≡ x2 P (X > x)/U (x) → 0

(equivalently, R(xn ) → 0).

(15)

x[M (cx) − M (x)]/U (x) → 0

for any one (or, all) fixed c > 1.

(16)

˜ (x) ≡ U (x) + x2 P (X > x) U

defines a function in U0 .

Theorem 7.2 (Partial variance, equal fractions) Consider an arbitrary rv ˜ a,a (·) denote K(·) Winsorized outside dom(a, a), and Y with df F and qf K. Let K now redefine − v(t) ≡ {[K+ (17) (t)]2 + [K + (1 − t)]2 } and V (t) ≡ (t,1] v(s) ds ,

(18)

− q(t) ≡ [ K+ (t) + K + (1 − t) ]

and m(t) ≡

(19)

˜ t,t (ξ)] µ ˜(t) ≡ E[K

˜ t,t (ξ)] . and σ ˜2 (t) ≡ Var[K

(t,1]

q(s) ds ,

All equivalences stated in parts (A), (B), and (C) of the previous theorem are still 2 ˜ t,t valid for the new definitions of q, v, σ ˜2 , and V˜ (t) ≡ E[K (ξ)]. [In the previous theorem we had q2 (t) = v(t). In the present theorem we have v(t) ≤ q2 (t) ≤ 2v(t), which is still convenient and meets all our needs. What is crucial is that m(·), q(·), v(·), and V (·) have an appropriate meaning. Note that this V (t) is the truncated (with respect to (t, 1 − t)) second moment.] Note Theorem 7.1 Winsorizes symmetrically about zero, while the companion theorem 7.2 Winsorizes equal fractions from each tail. Remark 7.1 (Why all these equivalencies?) In section 10.6 we have a suitable elementary bridge between these two sets of equivalencies that will allow us to combine the lists. Here is how we will do it. For Yn1 , . . . , Ynn iid with arbitrary 2 df F and qf K, the quantity Tn ≡ [maxk Ynk ]/[nYn2 ] can be equivalently seen (with respect to →p ) either as T1n based on Xnk ≡ K|Y | (ξnk )’s or as T2n based on Ynk ≡ KY (ξnk )’s. Now, the list L1 of conditions in theorem 7.1 contains (14), while the list L2 of conditions in theorem 7.2 contains (6)(B) (based on (17)). In theorem 10.6.1 we show that T1n →p 0 is equivalent to a list L3 that contains condition (14) of L1 . In theorem 10.6.2 we show that Tn2 →p 0 is equivalent to a list L4 that contains condition (6)(B) (based on (17)) of L2 . Thus Tn →p 0 is on both L3 and L4 . Thus all conditions on the lists L1 , L2 , L3 , and L4 are equivalent. In theorem 10.6.4 we add to the combined list the condition that Sn2 /˜ σ 2 (1/n) →p 1; and we will also add In theorem 11.3.1 we will add to the list the condition √ (7.6.9). ˜ n) − µ (7.6.6) that Zñ ≡ n [X(a ˜(an )] / σ ˜ (an ) →d N (0, 1) (for an = 1/n, and other appropriate an ). Knowing these last two facts, it is indeed very interesting to have a large list. 2

7.


129

Proof. Clearly, all of (4)(A)–(12)(C) (in the context of both theorem 7.1 and theorem 7.2), hold when σ2 < ∞. So from here on we always assume that σ2 = ∞. (A): Now, (5) implies (6), since for each d > 1 figure 7.1(a) demonstrates that (a)

t v(d t) + t [v(t) − v(d t)] t v(t) ≤ V (t) V (d t) + (d − 1) t v(d t)

(b)

≤

1 d t v(d t) + d t [v(t) − v(d t)] d−1 V˜ (d t)

(c)

≤

1 d V˜ (dt/d) − V˜ (d t) + d−1 d−1 V˜ (d t)

(d)

→ 1/(d − 1) + 0

by (5).

Since this holds for any d > 1, it gives (6).

v(·)

t

v(·)

dt (a)

ct

v(·)

t

ct ct 2

(b)

t (c)

Figure 7.1 Comparison of areas.

Next, (6) implies (7), since for each fixed 0 < c < 1 figure 7.1(b) shows that (e)

[V (ct) − V (t)]/V (ct) ≤ [(1 − c)/c][ctv(ct)/V (ct)] → 0 .

Suppose (7), that V ∈ R0 . This implies [V (ct/2) − V (t)]/V (t) → 0, and figure 7.1(c) then demonstrates that (f)

(ct/2) [v(ct) − v(t)]/V (t) ≤ [V (ct/2) − V (t)]/V (t) → 0 ;

and so (8) holds. Supposing (8) about v(·), we will establish (9) about d(·). Now, √ (g) t[v(ct) − v(t)]/V (t) = d(t) × [d(t) + 2 t q(t)] / V (t) ] ≥ d2 (t) ; and so d(t) → 0, as required for (9), is implied.

130

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS Then (9) implies (5), since figure 7.1(b) also shows that V˜ (ct) − V˜ (t) t[v(ct) − v(t)] ≤ ˜ V (t) + tv(t) V (t) √ √ √ t [q(ct) − q(t)] t [q(ct) − q(t)] 2 tq(t) = × + V˜ (t) V˜ (t) V˜ (t) √

(h)

≤

t [q(ct) − q(t)] × V (t)

√

√ t [q(ct) − q(t)] 2 tq(t) + V (t) tv(t)

= d(t){d(t) + 2} → 0 . We have closed the (5)–(9) circle. It is now trivial that (6) and (8) imply (10), while (10) even more trivially implies (6). We obtain (11) implies (6) via t √ 1/2 tv(t) t q(t) m(ct) − m(t) ct q(s) ds (i) (1 − c) ≤ = → 0. = (1 − c) V (t) V (t) t V (t) t V (t) Then (7) implies (11) via Cauchy–Schwarz in t t (1 − c)t ct q 2 (s) ds 1/2 m(ct) − m(t) ct q(s) ds = ≤ t V (t) t V (t) t V (t) (j)

≤

V (ct) − V (t) V (t)

1/2 → 0.

We next show that (4) is equivalent to the simpler (5). Suppose (4) holds, so that σ ˜ 2 (·) ∈ R0 . We use Gnedenko and Kolmogorov’s theorem 7.6.1 to write 2 σ ˜ (ct) − σ V˜ (ct) − V˜ (t) ˜ 2 (t) µ ˜2 (ct) − µ ˜2 (t) σ ˜ 2 (ct) σ ˜ 2 (t) = + × 2 × 2 2 σ ˜ (ct) σ ˜ (ct) σ ˜ (t) V˜ (t) V˜ (t) (k)

=

σ ˜ 2 (ct) − σ ˜ 2 (t) σ ˜ 2 (ct) + o(1) + o(1) × 2 × [1 + o(1)] = o(1) ; 2 σ ˜ (ct) σ ˜ (t)

and this implies V˜ (·) ∈ R0 , which is (5). Now suppose that (5) holds, so that V˜ (·) ∈ R0 . The same Gnedenko–Kolmogorov theorem now gives " 2 ! σ ˜ 2 (ct) − σ ˜ 2 (t) V˜ (ct) − V˜ (t) µ ˜ (t) − µ ˜2 (ct) V˜ (ct) = + σ ˜ 2 (t) V˜ (t)[1 + o(1)] V˜ 2 (ct) V˜ (t)[1 + o(1)] (l)

=

V˜ (ct) − V˜ (t) [1 + o(1)] + [o(1) + o(1)][1 + o(1)] = o(1) ; V˜ (t)

and this implies σ ˜2 (·) ∈ R0 , which is (4).

7.


131

That (6) and (12) are equivalent is immediate from (7.6.2), which gives (m)

r˜(t) =

t v(t) t v(t) t v(t) r(t) ∼ = . = σ ˜ 2 (t) 1 + r(t) V˜ (t) − µ ˜2 (t) V˜ (t)

(B): We next show that (6)(B) implies (6)(A). Suppose that (6)(B) holds for even one sequence an 0 having lim an /an+1 < ∞. We are thus given that rn ≡ an v(an )/V (an ) → 0. So (6)(A) holds via (n)

sup

an+1 ≤t≤an

≤

t v(t) V (an+1 ) an an+1 v(an+1 ) V (an+1 ) an rn+1 ≤ = V (t) an+1 V (an+1 ) V (an ) an+1 V (an )

an rn+1 an+1

#

1−

since (o)

an − 1 rn+1 → 0 , an+1

V (an ) 1≥ = 1− V (an+1 )

(an+1 ,an ]

v(t) dt

V (an+1 )

≥ 1−

an+1 v(an+1 ) an −1 → 1. V (an+1 ) an+1

Since(6)(A) trivially implies (6)(B), the condition (6)(B) is now on the list. That (5)(B) implies (5)(A) follows from (p)

1≤

sup

an+1 ≤t≤an

V˜ (ct) V˜ (c an+1 ) V˜ ((c/M )an ) ≤ → 1 × 1 = 1, ≤ V˜ (t) V˜ (an ) V˜ (an )

since lim(an /an+1 ) < M < ∞. We next show that (6)(B) implies (7)(B). Now, (6)(B) (since it implies (6)(A)) allows use of (e) with t = an to conclude (7)(B). The arguments of (f)–(h) can be repeated with t = an to show that (7)(B) implies (8)(B), which implies (9)(B), which implies (5)(B). Now, (5)(B) (which was shown in (p) to imply (5)(A), and thus it also implies (6)(A)) implies (6)(B) by using the argument of (a)–(d) (which is valid by using (6)(A)). Also, (10)(B) can be added by the same argument used for (10)(A). The arguments of (i) and (j) with t = an allow us to add (11)(B). Finally, the equivalence of (4)(B) and (5)(B) still follows from (k) and (l) with t = an . Next observe that (12)(B) is equivalent to (6)(B), since (m) gives (q)

r˜(an ) =

an v(an ) r(an ) an v(an ) ∼ = . σ ˜ 2 (an ) V (an ) + an v(an ) 1 + r(an )

Thus (12)(B) is on our list. (C): Note that (6)(C) is exactly (12)(B), and so it is on our list, and thus it implies (12)(A). We may reread (e) (with the denominator V (an ) replaced by σ ˜ 2 (an ) throughout) to see that (6)(C) (that is, its (12)(A) consequences) implies (7)(C). Then rereading (f)–(h) (with the same denominator replacement) shows that (7)(C) implies (8)(C), which implies (9)(C), which implies (5)(C). We now close the circle on (5)(C)–(9)(C) by noting that (5)(C) implies (5)(B), again using the Gnedenko and Kolmogorov result (7.6.2). We can add (10)(C) by the same trivial argument as before. Rereading (i)–(j) (with the new denominator) then allows us to add (11)(C).

132


The proof of theorem 7.2 is nearly identical. All but lines (h) and (j) are identical; line (i) is identical because m is still the integral of q. But (h) and (j) are not identical because we no longer have q2 = v. But we do have v2 = [q + ]2 +[q − ]2 , where q = q + +q − , and that is enough. Just factor the two pieces separately in (h) and (j), and apply the trivial inequalities (a + b)2 ≤ 2(a2 + b2 ) and a ∨ b ≤ a + b ≤ 2(a ∨ b). 2 Proof. (E): We prove this part separately. We first show that (14) implies (13). So suppose that (14) holds, so that R(x) ≡ x2 P (X > x)/U (x) → 0 as x → ∞. If c < 1, then [U (x) − U (cx)]/U (cx) = (cx,x] y 2 dFX (y)/U (cx) (a)

≤ c−2 [(cx)2 P (X > cx)/U (cx)] → 0;

and for c > 1 it is analogous that [U (cx) − U (x)]/U (x) → 0. Thus, U is slowly varying, as in (13). Suppose (13) holds. Then for all x ≥ (some x ), 2x 2 y dFX (y) x2 P (x < X ≤ 2x) U (2x) − U (x) (b) ≤ x ≤ < , U (x) U (x) U (x) (c)

x2 P (2x < X ≤ 4x)

U (2x) < ≤ [(1 + )/4] , U (x) 4 U (x)

(d)

x2 P (4x < X ≤ 8x)

U (4x) U (2x) < 2 ≤ [(1 + )/4]2 , . . . . U (x) 4 U (2x) U (x)

So for x ≥ x we add these to get (e)

R(x) = x2 P (X > x)/U (x) ≤

∞

k=0 [(1

+ )/4]k ≤ ≡ /(1 − (1 + )/4) .

Thus R(x) → 0 as x → ∞. That is, (13) implies (14). We will obtain (15) implies (14) analogously to (b)–(d). (To keep notation nearly identical to (b)–(d), we shall pretend that c = 2; but really, 4 = c2 , 8 = c3 , . . . .) Now, for all x ≥ (some x ), (15) gives 2x x x dFX (y) x2 P (x < X ≤ 2x) x[M (2x) − M (x)] (f) ≤ ≤ < , U (x) U (x) U (x) (g)

x2 P (2x < X ≤ 4x)

U (2x)

2x[M (2x) − M (x)]

≤ ≤ {1 + } ≤ (1 + 2 ) , U (x) 4 U (x) 4 U (x) 4 x2 P (4x < X ≤ 8x)

U (4x) U (2x) ≤ 2 U (x) 4 U (2x) U (x)

(h)

≤

4x[M (4x) − M (2x)]

(1 + 2 ) 1 + ≤ 2 (1 + 2 )2 . 42 U (2x) 4

Add these to get R(x) → 0 as x → ∞, as in (e). Thus (15) implies (14).

7.


133

We obtain (13) implies (15) via cx 2 y dFX (y) [U (cx) − U (x)] x[M (cx) − M (x)] (i) ≤ x ≤ → 0. U (x) U (x) U (x) Now, (13) implies (16) follows from (j)

˜ (cx) − U ˜ (x) U U (cx) − U (x) (c2 + 1)x2 P (X > x) + → 0, ≤ ˜ (x) U (x) U (x) U

since (13) now brings (14) with it. Then (16) implies (14), since ˜ (cx) − U ˜ (x)]/U ˜ (x) [U cx ˜ (x) = [ x y 2 dFX (y) + (cx)2 P (X > cx) − x2 P (X > c)]/U ˜ ≥ {[x2 [P (X > x) − P (X > cx)] + c2 x2 P (X > cx) − x2 P (X > x)}/U(x) (k)

= [(c2 − 1)/c2 ]/ {1 + 1/ [(cx)2 P (X > cx)/U (cx)]} ,

and the extreme left term going to 0 forces (cx)2 P (X > cx)/U (cx) → 0. The second condition in (14) suffices, since 2 x2 xn P (X > xn ) x2 P (X > x) (l) sup ≤ lim n+1 ≤ O(1)×R(xn ) → 0 . 2 U (x) xn U (xn ) xn ≤x≤xn+1 Thus (13)–(16) are equivalent. That (14) and (6)(A) are equivalent is the subject of the next section. At that point (13)–(16) will be on our list. 2 Though already established, we will still give a simple proof that V (·) ∈ L implies V˜ (·) ∈ L. If V ∈ L, a simple graph shows that 0 ≤ (1 − c)av(a)/V (a) ≤ [V (ca) − V (a)]/V (a) → 0 , implying that tv(t)/V (t) → 0 as t → 0. Then the equation # V˜ (ct) V (ct) tv(t) (0 + 1) × 1 ctv(ct) = +1 × +1 → =1 ˜ V (ct) V (t) V (t) 0+1 V (t) shows that V˜ ∈ R0 . We also note that when V ∈ L, then [V˜ (ct) − V˜ (t)]/V˜ (t) ≤ t[v(ct) − v(t)]/V (t) → 0 , showing that V˜ (·) ∈ L follows. Exercise 7.1 (20)

Verify that an equivalent condition in theorem 6.2 is

2 t [K+ (ct) ∨ K 2 (1 − ct)] / σ ˜ 2(t)

→0

as t → 0,

for each fixed c > 0.

(This condition is singled out for heavy use in chapter 16. It is purposely left to the exercises to encourage chapter 16 readers to put some effort into these proofs.)

134

8


Specific Tail Relationships

We list two relationships that will prove important, the first concerning the CLT and variance estimation and the second the WLLN. They compare the height of the qf with the magnitude of a partial moment. Theorem 8.1 (The (x, t)-tail equivalence; CLT) Consider any X ≥ 0, with qf K. Then (1) lim sup tK r (1−t)/ [t,1] K r (1−s) ds = lim sup xr P (X > x)/ [0,x] y r dF (y) , x→∞

t→0

for each r > 0. The same is true for the lim inf, and for the lim (if it exists). Proof. (2)

By the change of variable of (7.3.11), r r for all x. [0,F (x)] K (s) ds = [0,x] y dF (y)

Define t = 1 − F (x). Then (a)

r(t) ≡

t K r (1−t) [0,1−t]

if

K r (s) ds

= R(x) ≡ x

r

P (X>x)

[0,x]

y r dF (y)

t K r (1 − t) = xr P (X > x).

But examination of figure 8.1 shows that (from an x point of view) (3)

t K r (1 − t) = xr P (X > x)

fails

when and only when (4)

x is any point in the domain of some flat spot of F, other than the left endpoint.

Also, R is continuously across any such flat spot, and R(·) approaches a limit at both ends of each flat spot. So far, we have considered all values of x, but we have omitted values of t not in the range of the F (x)’s. But equality fails in (3) (from a t point of view) when and only when (5)

t is any point in the domain of some flat spot of K, other than the left endpoint.

Also, r is continuously across any such flat spot as t ; and r(·) approaches a limit at both ends of each a flat spot. Finally, note that all right and left limits achieved by R(·) across flat spots are achieved as left and right limits of r(·), and vice versa. This suffices for the claim. [Note also figure 10.4.1 below. And bare in mind that (a ∨ b) ≤ a + b ≤ 2(a ∨ b) for all a, b > 0.] 2

Definition 8.1 (Order-r qfs) (6)

A qf K is of order r (with r > 0) if

t{|K+ (t)|r + |K(1 − t)|r } → 0

as t → 0.

)

)( )( )

0

t-values

x-values

1

0

where

(

( (

is equal to or xP(X > x) is not equal to ∫ K(s) ds [0,x] y dF (y)

)(

tK(1 – t)

)(

[0,1 – t]

∫

(

(

Figure 8.1 (x, t)-tail equivalence comparison, when r = 1.

1

)

0

t

SPECIFIC TAIL RELATIONSHIPS

(

)

Not Equal

8.

K(1 – t)

F (x)

135

Not

Equal

x

136


Theorem 8.2 (The (x, t)-order equivalence; WLLN) Consider any X ≥ 0, with qf K. Then (7)

lim sup t|K(1 − t)|r = lim sup xr P (X > x) , t→0

x→∞

for each r > 0. The same is true for lim inf, and lim (if it exists). Proof. As figure 8.1 shows, the values (or limit of the values) of t|K(1 − t)|r r and x P (X > x) are equal at the associated pairs of endpoints related to any flat spot of F or any flat spot of K; and they are monotone across the flat spots. At all nonflat-spot pairs of associated points, they are equal. That is, at all of the key values (i.e., local extremes) the quantities in question are equal. 2 Remark 8.1 Clearly, if E|X|r < ∞ for X ∼ = K, then K is of order r, since for some finite M , (8)

g(x) ≡ |x|r F (x)[1 − F (x)] ≤ M on R, while g(x) → 0 as |x| → ∞,

(9)

h(t) ≡ t[|K(t)|r + |K(1 − t)|r ] ≤ M on (0, 1), while h(t) → 0 as t → 0. 2

9.

REGULARLY VARYING FUNCTIONS

137

9

Regularly Varying Functions

Definition 9.1 (Regularly varying functions, at 0) Call V (·) > 0 regularly varying at 0 with characteristic exponent r (written V ∈ Rr ) if L(t) ≡ t−r V (t) satisfies L(ct)/L(t) → 1 for each c > 0 (such a function L was called slowly varying, and we agreed to write L ∈ L ≡ R0 ). [Clearly, shifting a qf up or down has absolutely no effect on whether or not it is varies regularly.] 1 Theorem 9.1 Let V (t) ≡ t v(s)ds, where v ≥ 0 and v on some 0 < t < a◦ . Consider also V˜ (t) ≡ V (t) + t v(t). Let β ≥ 0. (Monotone density theorem) (1)

V ∈ R−β

iff

tv(t)/V (t) → β as t → 0.

(2)

V ∈ R−β

implies

v ∈ R−(β+1) ,

when β > 0.

(Karamata theorem) (3)

v ∈ R−(β+1)

implies

V ∈ R−β ,

(4)

V ∈ R−β

implies

V˜ (t) ∈ R−β

Proof.

Consider the “if” half of (1). For 0 < b < a < 1 we note that at (a − b) t v(at) ≤ bt v(s)ds = V (bt) − V (at) ≤ (a − b) t v(bt).

(a)

when V (0) = ∞. and t v(t)/V˜ (t) → β/(1 + β).

Also, V (bt) − V (at) V (bt) b−β L(bt) V (at) a−β L(at) = × − × −β −β −β t L(t) (bt) L(bt) L(t) (at) L(at) L(t) (b)

→ (b−β − a−β ).

The left side of (a) thus gives (c)

lim sup t 0

(b−β − a−β ) d −β t v(at) ≤ − , and is also ≤ − b =β t−β L(t) b−a db b=1

by setting a = 1 and letting b 1. The right side of (a) analogously gives (d)

lim inf t 0

t v(bt) (b−β − a−β ) d ≥− , and is also ≥ − b−β =β −β t L(t) b−a db b=1

by setting b = 1 and letting a 1. Combining (c) and (d) gives t v(t)/V (t) → β, so the “if” half of (1) holds. Then (2) and (4) are immediate. Consider the converse part of (1). Let r(t) ≡ t v(t)/V (t), so that the ratio r(t) ∈ [β − , β + ] for all 0 < t ≤ (some t ). Now, V is absolutely continuous on every closed subinterval of (0, t ] by the fundamental theorem of calculus. And thus

138


log V is absolutely continuous on the same closed subintervals (see (4.4.3)(b)), and so we may claim that (e)

(log V ) (s) = −r(s)/s

a.e. on (0, t ].

Thus for any 0 < t ≤ t , we can integrate to get t (f) log V (t) − log V (t ) = t [ r(s)/s ] ds, and this gives t [(ct)β V (ct)]/[tβ V (t)] = c β e t

ct

[r(s)/s] ds

t = e β log c e

ct

[r(s)/s] ds

[(r(s)−β)/s] ds

(g)

=e

(h)

= (some value between c− and c ) for all 0 < t ≤ t .

ct

But > 0 is arbitrary. Thus V ∈ R−β . Moreover, (f) shows that V may be written in the form d (5) V (t) = c t−β exp( t [r(s) − β]/s ds) for some constants c, d and for r(t) ≡ t v(t)/V (t) → β as t → 0. Consider (3). Fix 0 < r < 1. Let L(·) ∈ L. Then for all w ≤ (some t0 ≡ tr ), (i)

(1 − )L(w) ≤ L(rw) ≤ (1 + )L(w).

(Recall that a = b ⊕ c means that |a − b| ≤ c.) We write rt t V (rt) − V (rt0 ) = rt0 v(u) du = r t0 v(rw) dw = r−β

t0 t

with u ≡ r w

w−(β+1) L(rw) dw

(j)

= r−β [1 ⊕ ]

(k)

= r−β [1 ⊕ ]

t0 t

t0 t

w−(β+1) L(w) dw

using (i)

v(w) dw = r−β [1 ⊕ ] [V (t) − V (t0 )].

Thus (l)

(1 − ) r−β ≤

V (rt) − V (rt0 ) ≤ (1 + ) r−β . V (t) − V (t0 )

Since V (t) → ∞ as t → 0, we have from (l) that V ∈ R−β .

2

Definition 9.2 (Regularly varying functions, at ∞) A function U > 0 on (0, ∞) is called regularly varying with exponent ρ (written U ∈ Uρ ) when l(x) ≡ x−ρ U (x) satisfies l(cx)/l(x) → 1 for each c > 0 (such a function l(·) was called slowly varying at ∞).

9.

REGULARLY VARYING FUNCTIONS

139

Theorem 9.2 (Partial variance, β > 0) [Let α ≡ 2/(β + 1) and β = (2 − α)/α.] (i) Based on the definitions in (7.7.1) and (7.7.3), the following are equivalent: (6)

U ∈ U2 β/(1+β) = U2−α .

(7)

x2 P (X > x)/U (x) → β

as x → ∞.

(8)

t v(t)/V (t) → β

as t → 0.

(9)

V ∈ R−β .

(10)

v ∈ R−(β+1) .

(ii) Based on the definition in (7.7.17), the conditions (6)–(10) are equivalent. Proof. The equivalence of (8), (9), and (10) follows from theorem 9.1. Then add (7) using theorem 8.1. That (6) is also equivalent is the subject of the next exercise. 2 Exercise 9.1 Establish the initial equivalence (6) in theorem 9.2. [Hint. Use the integration by parts result P (X > x) = (x,∞) x−2 dU (x).] − Exercise 9.2 (De Haan) Let v− (t) ≡ [K+ (t)]2 and v+ (t) ≡ [K + (1 − t)]2 . Let t t V − ≡ 0 v − (s) ds and V + ≡ 0 v + (t) ds. If v− (t)/v+ (t) → c ∈ [0, ∞] as t → 0, then V − (t)/V + (t) → c as t → 0, provided that both integrals are infinite on (0, 1).

The following exercises summarize some standard general regular variation results. [That is, h is not assumed to be monotone. Karamata’s result is now harder.] Exercise 9.3 (Regular variation holds uniformly) (a) Let h denote any function that is regularly varying at 0, of order r ∈ R. Then for 0 < a < b < 1 we have the uniform convergence of both: (11)

sup |h(ct)/h(t) − cr | → 0 a≤c≤b

(12)

sup a≤u≤v≤b

as t → 0,

v v | u h(ct) dc/h(vt) − u cr dc/v r | → 0

as t → 0.

(b) Prove (11) assuming h . [Hint. Use (5) for (b). That (12) follows from (11) is easy, even in general.] Exercise 9.4 (Karamata) Let h be regularly varying at 0, of order r ∈ R. (i) Suppose r ≤ −1. Then 1 (13) h(s) ds/[th(t)] → −1/(r + 1) as t → 0. t 1/2 (ii) Suppose r > −1 or both r = −1 and 0 h(s) ds < ∞. Then t (14) as t → 0. 0 h(s) ds/[th(t)] → 1/(r + 1) Question Theorem 9.1 shows that when β > 0, the collection of quantile functions {K : V (·) ∈ R−β } form a subset of the collection {K : V˜ (·) ∈ R−β }. Are these two collections actually the same collection?

140


10

Some Winsorized Variance Comparisons

Notation 10.1

(1)

We agreed in notation 7.5.1 that for 0 ≤ a ≤ 1 − a ≤ 1,

˜ ≡K ˜ a,a (·) denotes K(·) Winsorized outside dom(a, a ), K(·) ˜ ˜ ˜ 2 (ξ), with ξ ∼ µ ˜ ≡ EK(ξ), σ ˜ 2 ≡ Var[K(ξ)], V˜ ≡ EK = Uniform(0, 1), 2 ˜ ˜ q˜ ≡ K(1 − a ) − K(a) and V ≡ dom(a,a ) K (t) dt .

˜ ˜ a,a (t) equals K+ (a), =K Thus whenever K+ (a) < 0 < K(1 − a ) we note that K(t) K(t), K(1 − a ) according as t is in (0, a], (a, 1 − a ), [1 − a , 1), while only the right tail of K(·) is modified if X ≥ 0, and only the left tail of K(·) is modified if X ≤ 0. (Recall also that ao ≡ a. ∧ (1 − a. ) for a. ≡ inf{t : K(t) ≥ 0}, from just above (7.5.7).) In any case, 1 1 ˜ a,a (s) dK ˜ a,a (t) (2) σ ˜2 = σ ˜ 2 (a, a ) ≡ 0 0 (s ∧ t − st) dK for 0 ≤ a < 1 − a ≤ 1, which shows that σ ˜2 (a, a ) always Var[K(ξ)] as a 0 and a 0. Setting a = a = 0 gives a valid representation of the variance in [0, ∞] without mention of µ.

K0(·)

0

a0 1 – a0

1

Figure 10.1 If K ∈ K, then the graph of (K – µK)/σK lies entirely in the shaded region.

Fix a qf K0 having mean 0 and finite variance σ02 , and (for some 0 < a0 < fixed) define the class of qfs (3)

1 2

K ≡ {K(·) : |K − µK |/σK ≤ |K0 | on (0, a0 ] ∪ [1 − a0 , 1)}

to be all qfs whose standardized form is bounded in the tails by the fixed qf K0 . (See figure 10.1.) Let µ ≡ µK and σ ≡ σK for each K ∈ K. 2

10.

SOME WINSORIZED VARIANCE COMPARISONS

Inequality 10.1 (Uniform Winsorized variance comparisons) as (a ∨ a ) → 0,

141 Note that

σ 2 − σ 2 |/˜ σ2 → 0 , supK∈K |˜ √ supK∈K |˜ (5) µ − µ|/(˜ σ a ∨ a ) → 0 (and µ ˇ may replace µ ˜), √ √ (6) ˜| + a |K(1 − c a ) − µ ˜|}/˜ σ→0 and supK∈K { a |K+ (ca) − µ √ for every fixed 0 < c, c < 1. [Thus supK∈K n |ˇ µ−µ|/˜ σn → 0 when an = an = 1/n.] (4)

Proof. Let K ∼ = (0, 1) with K ∈ K. The following bounds are uniform over K ∈ K and all 0 ≤ (a ∨ a ) ≤ (some a ). For the first moment comparison we use 1 a (7) |µ − µ ˜|/σ ≤ 0 |K0 (t)| dt + 1−a |K0 (t)| dt + a |K0+ (a)| + a |K0 (1 − a )| (a) since

≤

√

a ∨ a × ,

√ a √ |K0 (t)| dt ≤ || t K0 (t)||a0 × 0 t−1/2 dt = o( a ) √ follows from t K0 (t) → 0 as t → 0 whenever σ2 < ∞. Comparing second moments shows that for all 0 ≤ (a ∨ a ) ≤ (some a ) we have 1 2 1 2 ˜ (t) dt /σ 2 0 K (t) dt − 0 K (8)

(9) Finally,

(10)

a 0

≤

a 0

+

1 1−a

2 (a) + a K02 (1 − a ) ≤ . K02 (t) dt + aK0+

√ √ { a |K+ (ca) − µ ˜| + a |K(1 − c a ) − µ ˜|}/σ √ √ √ √ ≤ 2 ca |K0+ (ca)|/ c + 2 c a |K0 (1 − c a )|/ c ≤ .

Simple algebra completes the proof.

2

Inequality 10.2 (Basic Winsorized variance inequality) We suppose that 0 ≤ a ≤ 1 − a ≤ 1 (with a = 0 allowed only if X ≥ 0, and a = 0 allowed only if ˜ c (·) ≡ K ˜ ca,c a (·), and let µ X ≤ 0). Fix 0 < c, c < 1. Let K ˜c , σ ˜c2 , V˜c , and q˜c denote the same quantities for this new qf. It is immediate from figure 10.2 (using |˜ µ| ≤ q˜) that for a ∨ a sufficiently small, (11)

0≤σ ˜c2 − σ ˜ 2 = (V˜c − V˜ ) − (˜ µ2c − µ ˜2 ) = {V˜ ⊕ (a + a ) q˜c2 − V˜ } + {(˜ µ ⊕ (a + a ) q˜c )2 − µ ˜2 }

(12)

≤ 4(a + a ) q˜c2 .

142


Inequalities Aimed at the Infinite-Variance Case Suppose the qf K(·) satisfies K+ (a) < 0 < K(1 − a ). Fix 0 < c, c < 1. Define and

d ≡

√ a σ ˜

[K(1 − a) − µ ˜] ,

− K+ (ca)|

and

D ≡

√ a σ ˜

[K(1 − c a ) − K(1 − a )] ,

2 2 [K+ (ca) − K+ (a)]

and

˜ ≡ D

a σ ˜2

(13)

d ≡

√ a σ ˜

(14)

D≡

√ a σ ˜ [K+ (a)

(15)

˜≡ D

a σ ˜2

[µ ˜ − K+ (a)]

[K 2 (1 − ca ) − K 2(1 − a )] .

All of these are then ≥ 0. (Note figure 10.2.)

K(·)

˜ K c

˜ K

0 ˜→ µ

ca

a

˜ K

˜ K c

˜ and K˜ (·). Figure 10.2 The qfs K(·) c

a′

c′a′

1

10.


Inequality 10.3 (Tail relationships) (16)

143

Suppose K+ (a) < 0 < K(1 − a ). Then

0 ≤ d ∨ d ≤ 1 ,

(19)

√ √ 2 (˜ µc − µ ˜)2 /˜ σ 2 ≤ ( a D + a D )2 ≤ 2 (a D2 + a D ), ˜ + c D ˜ ≤ (V˜c − V˜ )/˜ ˜ +D ˜ , cD σ2 ≤ D √ √ ˜ = (D )2 + 2D a K(1 − a )/˜ ˜ = D2 + 2D a |K+ (a)|/˜ σ and D σ, D

(20)

˜ 2 )/˜ σ 2 ≤ D (D + 2d) + D (D + 2d ) ≤ D (D + 2) + D (D + 2) , (˜ σc2 − σ

(21) (22)

q˜ ≤ |K+ (a) − µ ˜| + |K(1 − a ) − µ ˜| ≤ 2 q˜ , √ √ √ a |K+ (ca)|/˜ σ ≤ D + a |K+ (a)|/˜ σ ≤ D + d + aµ ˜/˜ σ,

(23)

D + D ≤ 2 [(a ∨ a )/(a ∧ a )]1/2 × (˜ qc − q˜) / q˜ .

(17) (18)

Proof. Figure 10.2 immediately gives all but (20) and (23). For (23) observe 1 ˜ that σ ˜2 ≥ 0 (K(t) −µ ˜)2 dt ≥ (a ∧ a ) (˜ q /2)2 . Since ν = µY minimizes E[(Y − ν)2 ], 1 1 ˜ c (t) − µ ˜ (a) (˜ σc2 − σ ˜ 2 )/˜ σ 2 ≤ ( 0 (K ˜)2 dt − 0 (K(t) −µ ˜)2 dt)/˜ σ2 a 1 (b) ≤ ( 0 + 1−a )(same) dt/˜ σ2 ≤ D2 + 2Dd + (D )2 + 2D d . 2 There are many equivalent ways of expressing that a df or qf is in the domain of attraction of a normal or stable distribution. The next inequality enables us to go back and forth between various of these equivalent conditions and to establish new ones. We now prove that (24)–(28) below are equivalent. [We write (24)u –(28)u to denote uniformity of the inequalities in the qfs over some collection Ku of qfs K(·).] It matters not here whether the variance of K(·) is finite or infinite. Inequality 10.4 (Tail equivalencies) (A) Let the qf K be arbitrary. With fixed 0 < c, c < ∞ (bounded away from 0 and ∞), as (a ∨ a ) → 0 the following are equivalent: (24)

D ∨ D → 0 ,

(25)

[˜ σc2 − σ ˜ 2 ]/˜ σ2 → 0 , ˜ ∨D ˜ → 0 , D

(26) (27) (28)

[V˜c − V˜ ]/˜ σ2 → 0 , [V˜c − V˜ ]/V˜ → 0 .

(B) These are also equivalent for a specific c ≡ ca → 1 and c ≡ ca → 1 as the maximum (a ∨ a ) → 0. (C) The condition a ∨ a → 0 may be replaced by a specific an ∨ an → 0 as n → ∞. (D) If any one of (24)–(27) holds uniformly over a class Ku of qfs K, then all of them hold uniformly over the same class Ku . (E) Everywhere in (24)–(27) that a σ ˜ 2 appears in a denominator it may be replaced ˜ everywhere simultaneously by V . (F) Suppose X ≥ 0. We may let a ≡ 0, and claim everything above with respect only to a .

144


Proof. [The proofs are written assuming 0 < c, c < 1, with only minor adjustments needed otherwise.] Now, (27) is equivalent to (26) by (18), and (26) implies (24) by (19) (all implications holding with the claimed uniformity). Then (24) implies (25) by (20) (also with the claimed uniformity). We will show in the next paragraph that (25) implies (27) and (24) (also with the claimed uniformity). ˜ From the definition of K(·) we have (for rectangular regions R1 , R2 , R3 , R4 and corner regions R5 , R6 , R7 , R8 ) (a)

(b)

˜ 2 ]/˜ σ2 [˜ σc2 − σ 1−c a 1−c a 1−a 1−a = ( ca − a ) (s ∧ t − st) dK(s) dK(t)/˜ σ2 ca a = Σ81 Ri (s ∧ t − st) dK(s) dK(t)/˜ σ2 ≥{

(c)

a

1−a σ }{ a (1 ca sdK(s)/˜

+ (d)

a a

ca ca (s

− t)dK(t)/˜ σ}

∧ t − st)dK(s)dK(t)/˜ σ2 + {(six terms}

√ ˜ + (ξ)/˜ ≥ {c aD}{EK σ } + {(c/2)D2 } + {(six analogous terms)}

˜ + (ξ) term by parts). for a ∨ a small enough (integrate the EK Suppose (25) holds. From (d) (along with the other six analogous conclusions) we learn that the (25) conclusion [˜ σc2 − σ ˜ 2 ]/˜ σ 2 → 0 (uniformly) implies both √ ˜ (29) D a E|K(ξ)|/˜ σ → 0 (uniformly) and D → 0 (uniformly). Thus when [˜ σc2 − σ ˜ 2 ]/˜ σ 2 → 0 (uniformly) this gives (recall (20), too) σ ˜2 V˜c − V˜ |˜ µ2c − µ ˜2 | |˜ µc − µ ˜| |˜ µc − µ ˜| |˜ µ| ˜c2 − σ (e) = + 2 − ≤ σ ˜2 σ ˜2 σ ˜2 σ ˜ σ ˜ σ ˜ (30)

√ √ √ √ ˜ ≤ [ aD + a D ]{[ aD + a D ] + 2E|K(ξ)|/˜ σ} → 0

(uniformly), and so assuming [˜ σc2 −˜ σ 2 ]/˜ σ 2 → 0) implies |V˜c − V˜ |/˜ σ2 → 0 (uniformly). That is, (25)u implies (27)u . Thus (24)u –(27)u are equivalent. Since |(V˜c − V˜ )/V˜ | ≤ |V˜c − V˜ |/˜ σ2 , we have (27)u implies (28)u . Finally, V˜ − V˜ V˜c − V˜ |V˜c − V˜ | µ ˜2 c (f) − × ; = σ ˜2 σ ˜2 V˜ V˜ but µ ˜2 /˜ σ 2 is bounded as a ∨ a → 0 whenever σ2 ∈ (0, ∞), and it likewise goes to zero whenever σ 2 = ∞ by (7.6.2). Thus (f) shows that (28) implies (27). Note also that (28)u implies (27)u in any class Ku in which (31)

lima∨a →0 supK∈Ku µ ˜2 /˜ σ2 < ∞ .

Claim (E) also follows from (7.6.2) when EK 2 (ξ)2 = ∞, and is trivial when this is finite. Claims (B), (C), and (G) are now trivial. 2

10.


145

The Statistical Domain of Attraction Definition 10.1 (Domains of attraction) Let K be a qf, and define V˜ (t) ≡ − ˜ t,t (ξ)] as in (1). Let v− (t) ≡ [K+ V˜ (t, t) = Var[K (t)]2 and v+ (t) ≡ [K + (1 − t)]2 . − + Call K(·) balanced if v (t)/v (t) → (1 − p)/p for some p ∈ [0, 1]. Now define (32)

D ≡ ∪β≥0 {all K : V (·) ∈ some R−β , and K(·) is balanced if β > 0},

and call D the classical total domain of attraction. [It is customary to focus on the relationship α ≡ 2/(β + 1) instead.] Let (33)

˜ ≡ ∪{all qf’s K : V˜ ∈ R−β for some β ≥ 0}, D

˜ the statistical domain of attraction. [In trimming observations the and we call D two extreme tails do not have to be in balance, as they will be thrown away. That is, the natural CLT for the applied statistician can apply more generally than the probabilist’s natural CLT. We are preparing for this.] Theorem 10.1 (Tail comparisons for regularly varying qfs) Suppose the qf ˜ ≡ {all qfs K : V˜ (·) ∈ R−β for some β ≥ 0}. Suppose further K is in the class D that the followimnmg two conditions hold: (i): Fix c, c (34)

(if β = 0)

or

(ii): c ≡ ca → 1 and c ≡ ca → 1 as (a ∨ a ) → 0 (if β > 0), (i): 0 < lim a /a ≤ lim a /a < ∞ (if β = 0)

(35)

or

(ii): a /a → 1 (if β > 0).

Then all of the conclusions (24)–(28) hold. Proof. Because of inequality 10.4, we need only establish (28). We suppose that 0 < a, a < a◦ ≡ (a· ∧ (1 − a· )). We assume a < a in deriving an inequality (the other case of a < a is symmetric). Let V˜ (a) ≡ V˜ (a, a), etc. Now, [V˜ (ca, c a ) − V˜ (a, a )] V˜ (a, a ) (a)

[V˜ (ca) − V˜ (a)] [V˜ (c a ) − V˜ (a )] ≤ + × V˜ (a) V˜ (a )

$

[V˜ (a ) − V˜ (a)] 1+ V˜ (a)

% .

Suppose V˜ ∈ R−β , with β ≥ 0. Then L(a) ≡ aβ V˜ (a) ∈ L by definition of R−β . For β > 0 we have

(b)

[V˜ (ca) − V˜ (a)] (ca)−β L(ca) − a−β L(a) = a−β L(a) V˜ (a) L(ca) −β =c − 1 + (c−β − 1) → 0 , L(a)

146


since L ∈ L, a → 0, and c → 1 as (a ∨ a ) → 0, while [V˜ (ca) − V˜ (a)]/V˜ (a) → 0 directly when β = 0 and V˜ ∈ R0 . Of course, [V˜ (c a ) − V˜ (a )]/V˜ (a ) is identical to the previous claim, since c → 1. For β > 0 (with 1 − θ ≡ a /a → 1 as (a ∨ a ) → 0) the other term in (a) satisfies (c)

(d)

[a(1 − θ)]−β L(a(1 − θ)) − a−β L(a)] [V˜ (a ) − V˜ (a)] = a−β L(a) V˜ (a) L(a(1 − θ)) −β = (1 − θ) − 1 + [(1 − θ)−β − 1] → 0, L(a)

while [V˜ (a ) − V˜ (a)]/V˜ (a) → 0 directly for β = 0. Note that if β = 0, we needed only 0 < a/M < a < M a < ∞ to conclude that V˜ (a )/V˜ (a) → 1. 2 Definition 10.2 (Uniformity class Ku ) qfs defined by

Consider the uniformity classes Ku of

(36)

Ku ≡ {K : K(1 − t) = −K(t) = t−β for 0 < t < 1/2, for all 0 < β ≤ β0 } ,

(37)

Ku ≡ {K(t) ≡ [−t−β · 1[ta· ] ] Q(t) : 0 < β ≤ β0 , Q ∈ Q} ,

where Q is any class of qfs Q contained in L = R0 that are uniformly slowly varying at 0 and at 1. Theorem 10.2 (Uniformity class Ku ) classes Ku above that (38)

As a ∨ a → 0, we have for either of the

supK∈Ku D ∨ D → 0 ,

provided that either (39)

0 < lim inf(c ∧ c ) ≤ lim sup(c ∧ c ) < ∞ (if β = 0)

(40)

c ≡ ca → 1 and c ≡ ca → 1 as a ∨ a → 0 (if β > 0).

or

Thus all of (24)–(28) hold uniformly over such classes Ku . Proof. We now prove this for the class of qfs (37), with the result for the class of qfs (36) as a corollary. If we bound 4(a ∨ a )/(a ∧ a ) by M 2 , then (23) gives (a) (b)

D ≤ M |K+ (ct) − K+ (t)|/|K+ (t)| ≤ 2M |(ct)−β Q(ct) − t−β Q(t)|/t−β |Q(t)| = c−β |Q(ct) − Q(t)|/|Q(t)| + (c−β − 1) → 0 .

2

11.

INEQUALITIES FOR WINSORIZED QUANTILE FUNCTIONS

147

11

Inequalities for Winsorized Quantile Functions

The key to the smoothly functioning power of the following inequalities is the formulation of the tail terms in (3) and (7) below. The inequalities look clumsy, but they work extremely efficiently. Note that the upper bounds that appear in equations (1) and (6) do not depend at all on the unknown qf K. Roughly, parts (i), (ii), and (iii) prepare for the central limit theorem (CLT), the law of large numbers (LLN) (whether for second moments or for first moments), and general integral variance estimators. [If σ ñ = 0, then just multiply through by this symbol. We agree that b a ≡ (a,b) throughout the course of this section.] Use the notation of section 7.5. All qfs are assumed to be nondegenerate. Inequality 11.1 (Winsorized variance inequalities) Let K(·) be arbitrary. Let 0 < c ≤ an < 1 − an ≤ 1 − c < 1. Let 1/n ≤ r/n ≤ 12 ≤ 1 − r /n ≤ 1 − 1/n. (i) For the CLT Fix ν ∈ (0, 12 ). The following statements hold: 1−an −ν 9/ν 1/2−ν (1) n [t(1 − t)] dK(t)/˜ σn ≤ . an (kn ∧ kn )ν an −ν √ (2) n [t(1 − t)]1/2−ν dK(t)/˜ σn ≤ an |K+ (c) − K+ (an )| /˜ σn . c

1−an

(3)

−ν

n an

where

√ γn ≡ √ γn ≡

[t(1 − t)]

1/2−ν

9/ν dK(t)/˜ σn ≤ + γn + γn , [ (kn ∨ r) ∧ (kn ∨ r )]ν

r |K+ (an ) − K+ (r/n)| √ × 1[an 2 subspaces H1 , . . . , Hk . Let us suppose that X1 , . . . , Xk are rvs on (Ω, A, P ) and let (14)

Hi ≡ HF (Xi ) ,

Pi Y ≡ E(Y | Xi )

for i = 1, . . . , k

for Y ∈ H ≡ L2 (P ). Define (15)

H+ ≡ H1 + · · · + Hk .

As when k = 2, let P+ denote the projection onto H+ , Q+ ≡ I−P+ , and Qi ≡ I−Pi , i = 1, . . . , k. Exercise 6.3 (Halperin) (16)

Define

Q = Q1 , . . . , Qk .

Then (17)

Qn Y →L2 Q+ Y = PH⊥ ⊥ Y. 1 ∩···∩Hk

Other generalizations and extensions of theorem 6.1 are known, including rates of convergence. We are simply stating some known results in this section as an “exercise”. Exercise 6.4 (Rota–Burkholder) Y ∈ Lr (P ) for some r > 1, then (18)

(Q1 Q2 )n Y →Lr

and a.s

Let Q1 and Q2 be as in theorem 6.1. (i) If

(some rv in Lr (P )).

(ii) If E{|Y | log+ |Y |} < ∞, then (19)

(Q1 Q2 )n Y →L1

and a.s

(some rv in L1 (P )).

Conversely, if convergence holds in (19) for all choices of the projections Q1 and Q2 , then we have E{|Y | log+ |Y |} < ∞. The alternating methods developed here have many applications in statistics. One important application is to the additive regression models.

Chapter 9

Special Distributions 1

Elementary Probability

Independent Bernoulli Trials If P (X = 1) = p = 1 − P (X = 0), then X is said to be a Bernoulli(p) rv. We refer to the event [X = 1] as “success”, and [X = 0] as “failure”. Let X1 , . . . , Xn be iid Bernoulli(p), and let Tn ≡ X1 + · · · + Xn denote the number of successes in n independent Bernoulli(p) trials. Now, P (Xi = xi for 1 ≤ i ≤ n) = p

n 1

xi

n

(1 − p)n−

1

xi

if all xi equal 0 or 1;

this formula gives the joint distribution of X1 , . . . , Xn . From this we obtain n k (1) P (Tn = k) = p (1 − p)n−k for 0 ≤ k ≤ n, k since each of the nk different possibilities that place k of the 1’s in specific positions in an n-vector containing k outcomes 1 and n − k outcomes 0 has probability pk (1−p)n−k , from the earlier display. We denote this by writing Tn ∼ = Binomial(n, p) when (1) holds. Note that Binomial(1, p) is the same as Bernoulli(p). Let X1 , X2 , . . . be iid Bernoulli(p). Let Y1 ≡ W1 ≡ min{n : Tn = 1}. Since we can rewrite the event [Y1 = k] = [X1 = · · · = Xk−1 = 0, Xk = 1], we have (2)

P (Y1 = k) = (1 − p)k−1 p

for k = 1, 2, . . . .

We write Y1 ∼ = GeometricT(p). Now let Wm ≡ min{n : Tn = m}. We call Wm the waiting time to the mth success; Wm counts the number of turns until the mth success. We let Ym ≡ Wm − Wm−1 for m ≥ 1, with W0 ≡ 0, and we call the Ym ’s the interarrival times. Note that [Wm = k] = [Tk−1 = m − 1 and Xk = 1]. Hence k−1 m (3) P (Wm = k) = p (1 − p)k−m for k = m, m + 1, . . . . m−1 We write Wm ∼ = Negative Binomial Turns(m, p) ≡ NegBiT(m, p). [We agree that NegBiF(m, p) denotes the distribution of Wm − m, and that this “F” connotes “failures”; the rv Wm − m counts the number of failures prior to the mth success.] 179

180

CHAPTER 9. SPECIAL DISTRIBUTIONS

Exercise 1.1

Explain why Y1 , Y2 , . . . are iid GeometricT(p).

Since the number of successes in the first n1 +n2 trials is the same as the number of successes in the first n1 trials plus the number of successes in the next n2 trials, it is clear that (4)

T1 +T2 ∼ = Binomial(n1 +n2 , p)

for independent rvs Ti ∼ = Binomial(ni , p).

Likewise, waiting for m1 successes and then waiting for m2 more successes is the same as waiting for m1 + m2 successes in the first place. Hence, (5)

W1 +W2 ∼ = NegBiT(m1 +m2 , p) for independent rvs Wi ∼ = NegBiT(mi , p).

Urn Models Suppose an urn contains N balls that are identical, except that M bear the number 1 and N − M bear the number 0. Thoroughly mix the balls in the urn. Draw one ball at random. Let X1 denote the number on the ball drawn. Then X1 ∼ = Bernoulli(p) with p ≡ M/N . Now replace the ball in the urn, thoroughly mix, and draw at random a second ball with number X2 . Continue the process. This is the sampling with replacement scheme. Then Tn ≡ X1 + · · · + Xn ∼ = Binomial(n, p), where p = M/N represents the probability of success in n independent Bernoulli(p) trials. Suppose now that the same scheme is repeated, except that the balls are not replaced. In this sampling without replacement scheme X1 , . . . , Xn are dependent Bernoulli(p) rvs with p = M/N . Also, M N −M (6)

P (Tn = k) =

k

Nn−k

,

provided that the value k is possible.

n

We write Tn ∼ = Hypergeometric(M, N − M ; n). Suppose now that sampling is done without replacement, but the N balls in the urn bear the numbers a1 , . . . , aN . Let X1 , . . . , Xn denote the numbers on the first n balls drawn, and let the general finite Tn ≡ X1 + · · · + Xn . We call this N 2 ¯)2 /N sampling model. Call a¯ ≡ N 1 ai /N the population mean and σa ≡ 1 (ai − a 2 the population variance. Note that Xi ∼ ) for all 1 ≤ i ≤ n, since we now (¯ a , σ = a assume n ≤ N . From (7.3.4), we have N (7) 0 = Var[ 1 Xi ] = N Var[X1 ] + N (N − 1)Cov[X1 , X2 ], N with the 0 valid, since 1 Xi is a constant. Solving (7) yields (8)

Cov[X1 , X2 ] = −σa2 /(N − 1).

As in (7), and using (8), Var[Tn ] = nσa2 − n(n − 1)σa2 /(N − 1). Thus 1 2 n−1 (9) Var[Tn /n] = σa 1 − , n N −1 where [1 − (n − 1)/(N − 1)] is called the correction factor for finite population sampling.

1.

ELEMENTARY PROBABILITY

Exercise 1.2

181

Verify (8) and (9).

Exercise 1.3 Suppose that T1 ∼ = Binomial(m, p) and T2 ∼ = Binomial(n, p) are independent. Then the conditional distribution of T1 given that T1 + T2 = k is Hypergeometric(k, m + n − k; m).

The Poisson Process Suppose now that Xn1 , . . . , Xnn are iid Bernoulli(pn ), where npn → λ as n → ∞. Let Tn ≡ Xn1 + · · · + Xnn , so that Tn ∼ = Binomial(n, pn ). An easy calculation shows that (10)

P (Tn = k) → λk e−λ /k!

for k = 0, 1, . . . .

When P (T = k) = λk e−λ /k! for k = 0, 1, . . . , we write T ∼ = Poisson(λ). The above can be used to model the following Geiger counter experiment. A radioactive source with large half-life is placed near a Geiger counter. Let N(t) denote the number of particles registered by time t. We will say that {N(t) : t ≥ 0} is a Poisson process. [Do note that our treatment is purely informal.] Physical considerations lead us to believe that the increments N(t1 ), N(t1 , t2 ], . . . , N(tk−1 , tk ] should be independent rvs; here, the increment N(ti−1 , ti ] ≡ N(ti ) − N(ti−1 ) is the number of particle counts across the interval (ti−1 , ti ]. We say that N has independent increments. We now define (11)

ν ≡ E N(1) ≡ [the intensity of the process].

Let M denote the number of radioactive particles in our source, and let Xi equal 1 or 0 depending on whether or not this ith particle registers by time t = 1. It seems possible to assume that X1 , . . . , XM are iid Bernoulli. Since N(1) = X1 + · · · + XM has mean ν = E N(1) = M EX1 , this leads to N(1) ∼ = Binomial(M, ν/M ). By the first paragraph of this section, N(1) is thus approximately a Poisson(ν) rv. We now alter our point of view slightly, and agree that we will use this approximation as our model. Thus N(1) is a Poisson(ν) rv. Because of the stationary and independent increments we thus have (12)

N(s, t] ≡ N(t) − N(s)∼ = Poisson(ν(t − s))

(13)

N has independent increments.

for all 0 ≤ s ≤ t,

Agree also that N(0) ≡ 0. (This is actually enough to rigorously specify a Poisson process.) Let Y1 ≡ W1 ≡ inf{t : N(t) = 1}. Since (14)

[Y1 > t] = [N(t) < 1] = [N(t) = 0],

we see that 1 − FY1 (t) = P (Y1 > t) = P (N(t) = 0) = e−νt by (12). Thus Y1 has df 1 − exp(−νt) for t ≥ 0 and density (15)

fY1 (t) = νe−νt

for t ≥ 0;

we write Y1 ∼ = Exponential(ν). Now let Wm ≡ inf{t : N (t) = m}; we call Wm the mth waiting time. We call Ym ≡ Wm − Wm − 1, m ≥ 1, the mth interarrival time.

182


In light of the physical properties of our Geiger counter model, and using (13), it seems reasonable that (16)

Y1 , Y2 , . . . are iid Exponential(ν) rvs.

Our assumption of the previous sentence could be expressed as follows: (17)

Y1 and N1 (t) ≡ N(Y1 , Y1 + t] = N(Y1 + t) − N(Y1 ) are independent, N1 is again a Poisson process, with intensity ν.

We will call this the strong Markov property of the Poisson process. Additionally, (18)

[Wm > t] = [N(t) < m],

m−1 k −νt so that 1 − FWm (t) = P (Wm > t) = /k! ; the derivate of this k=0 (νt) e expression telescopes, and shows that Wm has density (19)

fWm (t) = ν m tm−1 e−νt /Γ(m)

for t ≥ 0.

We write Wm ∼ = Gamma(m, ν). Since waiting for m1 counts and then waiting for m2 more counts is the same as waiting for m1 + m2 counts in the first place, (20)

Z1 + Z 2 ∼ = Gamma(m1 + m2 , ν)

for independent Zi ∼ = Gamma(mi , ν).

It is true that (19) is a density for any real number m > 0, and the property (20) still holds for all positive mi ’s. Exercise 1.4

Verify (10).

Exercise 1.5

Verify (19)

Exercise 1.6

Verify that (20) holds for arbitrary real mi > 0.

Exercise 1.7 If X ∼ = Poisson(ν1) and Y ∼ =Poisson(ν2), then the conditional distribution of X given that X + Y = n is Binomial(n, ν1 /(ν1 + ν2 )). Exercise 1.8 Use Kolmogorov’s extension theorem to show that a Poisson process N exists on (R[0,∞) , B[0,∞) ). Then apply the smoother realizations theorem 5.4.2.

Location and Scale If a > 0, then FaZ+b (x) = P (aZ + b ≤ x) = P (Z ≤ (x − b)/a) = FZ ((x − b)/a) holds for any FZ (·). Thus for any density fZ (·), the rv aZ + b has density (21)

faZ+b (x) = a1 fZ ( x−b a )

for − ∞ < x < ∞.

1.


183

Normal Distributions Suppose the rv Z has density (22)

fZ (x) =

√1 2π

2

exp(− x2 )

for − ∞ < x < ∞;

then Z is said to be a standard normal rv. So the rv X ≡ µ + σZ ∼ = (µ, σ2 ) has density (23)

√1 2π σ

2 exp(− 12 ( x−µ σ ) )

for − ∞ < x < ∞,

and we write X ∼ = Normal(µ, σ2 ), or just X ∼ = N (µ, σ2 ). Exercise 1.9 Show that the formula fZ (·) of (22) is a density. Then show that this density has mean 0 and variance 1. [Transform to polar coordinates to compute ( fZ (x) dx)2 = 1.] The importance of the normal distribution√derives from the following theorem. ¯ − µ)/σ ∼ Recall that if X1 , . . . , Xn are iid (µ, σ2 ), then n (X = (0, 1) for the sample ¯ average Xn ≡ (X1 + · · · + Xn )/n. This is only a statement about moments. But much more is true. The result we now state will be proved in chapter 11. We will use it in the meantime for motivational purposes. Theorem 1.1 (Classical CLT) √ ¯ n − µ) →d N (0, σ2 ) (24) n (X

If X1 , . . . , Xn are iid (µ, σ2 ), then as n → ∞.

Let σ > 0. Then the Zn below is asymptotically normal, in that √ ¯ n − µ)/σ →d N (0, 1) (25) Zn ≡ n (X as n → ∞. Suppose that Z is N (0, 1). Then (26)

√ √ √ √ FZ 2 (x) = P (Z 2 ≤ x) = P (− x ≤ Z ≤ x ) = FX ( x ) − FZ (− x );

thus Z 2 has density (27)

√ √ 1 fZ 2 (x) = √ [fZ ( x ) − fZ (− x )] 2 x

for x ≥ 0.

[Note that formula (27) is true for any density fZ (·).] Plugging into (27) for this Z shows that (28)

fZ 2 (x) = (2πx)−1/2 exp(−x/2)

for x ≥ 0;

this is called the Chisquare(1) distribution. Note that Chisquare(1) is the same as Gamma( 12 , 12 ). Thus (20) establishes that (29)

if X1 , . . . , Xm are iid N (0, 1), then

m i=1

1 where Chisquare(m) ≡ Gamma( m 2 , 2 ).

Xi2 ∼ = Chisquare(m),

184


Uniform and Related Distributions We write X ∼ = Uniform(a, b) if (30)

fX (x) =

1 (b−a) 1[a,b] (x)

=

1 (b−a)

on [a, b].

By far the most important special case is Uniform(0, 1). A generalization of this is the Beta(c, d) family. We write X ∼ = Beta(c, d) if (31)

fX (x) =

1 β(c,d)

xc−1 (1 − x)d−1 1[0,1] (x) =

1 β(c,d)

xc−1 (1 − x)d−1

on [0, 1],

where β(c, d) ≡ Γ(c)Γ(d)/Γ(c + d). Suppose that ξ1 , . . . , ξn are iid Uniform(0, 1). Let 0 ≤ ξn:1 ≤ · · · ≤ ξn:n ≤ 1 denote the ordered values of the ξi ’s; we call the ξn:i ’s the uniform order statistics. It seems intuitive that ξn:i equals x if (i − 1) of the ξi ’s fall in [0, x), 1 of the ξi ’s is equal to x, and n − i of the ξi ’s fall in (x, 1]. There are n!/[(i − 1)!(n − i)!] such designations of the ξni ’s, and for each such designation the chance of the rv’s falling in the correct parts of [0, 1] is xi−1 (1 · dx)(1 − x)n−i . Thus (32)

fξn:i (x) =

n! xi−1 (1−x)n−i 1[0,1] (x), or ξn:i ∼ = Beta(i, n−i+1). (i − 1)!(n − 1)!

Exercise 1.10 Give a rigorous derivation of (32) by computing 1 − Fξn:i (x) and then differentiating it. Exercise 1.11 Chose a point at random on the surface of the unit sphere (with probability proportional to area). Let Θ denote the longitude and Φ denote the latitude (relative to some fixed axes) of the point so chosen. Determine the joint density of Θ and Φ.

The Cauchy Distribution We say X ∼ = Cauchy(b, a) if (33)

fX (x) = 1/{aπ[1 + (x − b)2 /a2 ]}

on (−∞, ∞).

By far the most important special case is Cauchy(0, 1); we then say simply that X ∼ = Cauchy, and its density is given by 1/[π(1 + x2 )] on (−∞, ∞). Verify that E|X| = ∞. We will see below that if X1 , . . . , Xn are iid Cauchy, then the sample ¯ n ≡ (X1 + · · · + Xn )/n ∼ average X = Cauchy. These two facts make the Cauchy ideal for many counterexamples.

Double Exponential and Logistic Distributions We say X ∼ = Double Exponential(b, a) when (X − b)/a has density 12 exp(−|x|) on the line. We say X ∼ = Logistic(b, a) when (X − b)/a has density ex /(1 + ex )2 = −x/2 x/2 2 1/(e + e ) on the line. Exercise 1.12 Now, X ≡ F −1 (ξ) has df F by the inverse transformation. So, compute F −1 for the Logistic(0, 1) and the Double Exponential(0, 1) distributions.

1.


185

Rademacher Random Variables and Symmetrization Many problems become simpler if the problem is symmetrized. One way of accomplishing this is by the appropriate introduction of Rademacher rvs. We say that

is a Rademacher rv if P ( = 1) = P ( = −1) = 12 . Thus ∼ = 2 Bernoulli( 12 ) − 1. ∼ We say that X is a symmetric rv if X = −X. If X and X are iid, then s X ≡ (X − X ) ∼ = (X − X) = −(X − X ) = −X s ; hence X s is a symmetric rv. Exercise 1.13 If X is a symmetric rv independent of the Rademacher rv , then X∼ = X always holds.

The Multinomial Distribution Suppose that B1 + · · · + Bk = R for Borel sets Bi ∈ B; recall that we call this i ≡ (Xi1 , . . . , Xik ) a partition of R. Let Y1 , . . . , Yn be iid rvs on (Ω, A, P ). Let X ≡ (1B1 (Yi ), . . . , 1Bk (Yi )) for 1 ≤ i ≤ n, and set (34)

n n n n T ≡ (T1 , . . . , Tk ) = ( Xi1 , . . . , Xik ) = ( 1B1 (Yi ), . . . , 1Bk (Yi )). i=1

i=1

i=1

i=1

Note that X1j , . . . , Xnj are iid Bernoulli(pj ) with pj ≡ P (Yi ∈ Bj ), and thus Tj ∼ = Binomial(n, pj ) (marginally). Note that T1 , . . . , Tn are dependent rvs. The joint distribution of (T1 , . . . , Tn ) is called the Multinomial(n, p ) distribution. We now derive it. The number of ways to designate n1 of the Yi ’s to fall in B1 , . . . , Bk , and nk of the Yi ’s to fall in Bk is the multinomial coefficient n! n (35) ≡ , where n1 + · · · + nk = n. n1 . . . nk n1 ! · · · nk ! k Each such designation occurs with probability 1 pi ni . Hence for each possible n, n (36) P (T = n) ≡ P (T1 = n1 , . . . , Tk = nk ) = pn1 1 · · · pnk k . n1 . . . nk It is now a trivial calculation that (37)

Cov[Xij , Xil ] = E1Bj (Yi )1Bl (Yi ) − E1Bj (Yi )E1Bl (Yi ) = −pj pl

if j = l.

Thus (38)

Cov[Tj , Tl ] = −n pj pl

for all j = l.

Thus (with Dp a diagonal matrix having each dii = pi )  (39)

     T1 p1 p1 (1 − p1 ) −p1 pk . . .  .   .      ∼     . . , n . . .   =     = ( p , n [Dp − p p ] ).       . . . . . Tk pk −pk p1 pk (1 − pk )

186


Assorted Facts Stirling’s Formula For all n > 1 we have √ n! = ean nn+1/2 e−n 2π, (40) where 1/(12n + 1) < an < 1/(12n). Eulers’s Constant n (41) i=1 1/i − log n ↑ γ ≡ 0.577215664901533 . . . . Exercise 1.14

If

∞ 1

an < ∞, there exists a cn ↑ ∞ such that

∞ 1

cn an < ∞.

Elementary Conditional Probability One defines the conditional probability of the event A given that the event B has occurred via P (A|B) ≡ P (AB)/P (B) when P (B) = 0. One then calls A and B independent if P (A|B) = P (A), because the probability of A is then unaffected by whether or not B occurred. Thus both of the following statements hold: (42)

Definition: P (A|B) ≡ P (AB)/P (B) leads to Theorem: P (AB) = P (B)P (A|B).

(43)

Definition: Independence means P (A|B) = P (A) leads to Theorem: P (AB) = P (A)P (B) if A and B are independent.

The big advantage of computation of P (A|B) via the theorem of (42) is that one can often revisualize P (A|B) in the context of a much simpler problem. Thus the probability of drawing two Reds when drawing at random without replacement from an urn containing 6 Reds and 4 Whites is P (R1 R2 ) = P (R1 )P (R2 |R1 ) = (6/10) × (5/9), where we revisualized to an urn containing 5 Reds and 4 Whites to compute P (R2 |R1 ) = 5/9. [Had we used sampling with replacement, our answer would have been (6/10) × (6/10) via (43).] [In the next exercise, revisualization works superbly to trivialize the problem.] Exercise 1.15 Determine the probability that the “shooter” wins in the game of craps. [When trying to convert a “point” of “ten” (say), we can revisualize and say that on the turn on which the game ends the dice will be showing either one of the 3 tens or one of the 6 sevens, and the probability of this conversion is clearly 3/(3 + 6).]

2.

DISTRIBUTION THEORY FOR STATISTICS

187

2

Distribution Theory for Statistics

Convolution If X and Y are independent rvs on (Ω, A, P ), then FX+Y (z) = P (X + Y ≤ z) = x+y≤z dFX (x) dFY (y) =

∞ z−x −∞

(1)

∞

= −∞

−∞

dFY (y) dFX (x)

FY (z − x) dFX (x) ≡ FX ∗ FY (z)

is a formula, called the convolution formula, for FX+Y in terms of FX and FY (the symbol ∗ defined here stands for “convolution”). In case Y has density fY with respect to Lebesgue measure, then so does X + Y . In fact, since z ∞ ∞ z −∞ −∞ fY (y − x) dFX (x) dy = −∞ [ −∞ fY (y − z) dy] dFX (x) =

∞

F −∞ Y

(z − x) dFX (x) = FX+Y (z),

we see that X + Y has a density given by ∞ (2) fX+Y (z) = fY (z − x) dFX (x). −∞

In case both X and Y have densities, we further note that ∞ (3) fX+Y (z) = fY (z − x)fX (x) dx ≡ fY ∗ fX (z). −∞

Exercise 2.1 Use (2) to show that for X and Y independent: (i) X ∼ = N (µ1 , σ12 ) and Y ∼ = N (µ2 , σ22 ) implies X + Y ∼ = N (µ1 + µ2 , σ12 + σ22 ). ∼ ∼ (ii) X = Cauchy(0, a1 ) and Y = Cauchy(0, a2 ) has X + Y ∼ = Cauchy(0, a1 + a2 ). (iii) X ∼ = Gamma(r1 , θ) and Y ∼ = Gamma(r2 , θ) has X + Y ∼ = Gamma(r1 + r2 , θ). Exercise 2.2 (i) If X1 , . . . , Xn are√iid N (0, 1), then the normed sample average necessarily satisfies (X1 + · · · + Xn )/ n ∼ = N (0, 1). (ii) If X1 , . . . , Xn are iid Cauchy(0, 1), then (X1 + · · · + Xn )/n ∼ = Cauchy(0, 1). If X and Y are independent rvs taking values in 0, 1, 2, . . ., then clearly (4)

P (X + Y = k) =

k

P (X = i)P (Y = k − i)

for k = 0, 1, 2, . . . .

i=0

Exercise 2.3

Use (3) to show that for X and Y independent:

X∼ = Poisson(λ1 ) and Y ∼ = Poisson(λ2 ) has X + Y ∼ = Poisson(λ1 + λ2 ) .

188


A fundamental problem in probability theory is to determine constants bn and an > 0 for which iid rvs X1 , . . . , Xn , . . . satisfy (5)

(X1 + · · · + Xn − bn )/an →d G,

as n → ∞

for some nondegenerate df G. Exercise 2.2 gives us two examples of such convergence; each was derived via the convolution formula. Except in certain special cases, such as exercises 2.1 – 2.3, the various convolution formulas are too difficult to deal with directly. For this reason we need to develop a more oblique, but ultimately more convenient, approach if we are to solve problems of the form (5). This is taken up in chapters 11, 13, 14, and 15.

Other Formulas Exercise 2.4 Suppose that X and Y are independent with P (Y ≥ 0) = 1. Show that products and quotients of these rvs satisfy ∞ FXY (z) ≡ P (XY ≤ z) = 0 FX (z/y) dFY (y) (6) for all z, (7)

FX/Y (z) ≡ P (X/Y ≤ z) =

∞ 0

FX (zy) dFY (y)

for all z.

If FX has a density fX , then changing the order of integration above shows that FXY and FX/Y have densities given by ∞ (8) fXY (z) = 0 y −1 fX (z/y) dFY (y) for all z, (9)

fX/Y (z) =

∞ 0

y fX (yz) dFY (y)

for all z.

∼ N (0, 1), U ∼ Exercise 2.5 Let Z = = χ2m , and V ∼ = χ2n be independent. (a) Establish these classically important results: (10)

Z ∼ = Student’s tm . U/m

(11)

U/m ∼ = Snedecor’s Fm,n . V /n

(12)

U ∼ = Beta(m/2, n/2) . U +V

Here (13)

Γ((m + n)/2) 1 ftm (x) ≡ √ 2 π m Γ(m/2) (1 + x /m)(m+n)/2

(14)

fFm,n (x) ≡

Γ((m + n)/2) (m/n)m/2 xm/2−1 Γ(m/2)Γ(n/2) (1 + mx/n)(m+n)/2

for − ∞ < x < ∞, for 0 < x < ∞.

(b) Compute the kth moment of each of these three distributions.

2.

DISTRIBUTION THEORY FOR STATISTICS

189

Exercise 2.6 (15)

If Y1 , . . . , Yn+1 are iid Exponential(θ), then (Y1 + · · · + Yi )/(Y1 + · · · + Yn+1 ) ∼ = Beta(i, n − i + 1).

Exercise 2.7 Let X1 , . . . , Xn be iid N(µ, σ2 ). √ ¯ ∼ (a) Show that Wn ≡ n (X n − µ)/ σ = N(0, 1). n 2 2 ¯ n )2 / σ 2 ∼ (b) Show that (n − 1)Sn / σ ≡ 1 (Xk − X = χ2n−1 . (c) Show that Wn and Sn2 are independent rvs. √ ¯ ∼ (d) Show that Tn ≡ n (X n − µ)/ Sn = Student’s tn−1 . √ [Hint. Let Γ ≡ |[ γij ]| be an orthogonal matrix with all γ1j = 1/ n. Now let ≡ Γ (X − µ 1 )/ σ. This yields iid N(0, 1) rvs Z1 , . . . , Zn , with Wn = Z1 ∼ Z = N(0, 1) n 2 ∼ 2 and (n − 1)S 2 = Z =χ . Apply exercise 2.5.] n

2

k

n−1

Statistical Confidence Intervals Example 2.1 Suppose we model the performances of n independent repetitions X1 , . .√. , Xn of an experiment as iid N (µ, σ2 ) rvs. The previous exercise shows ¯ n − µ)/σ is a N (0, 1) rv independent of the sample variance estimator n (X that n 2 ¯ n )2 /(n − 1) of σ2 , and that Sn /σ ∼ Sn ≡ 1 (Xk − X = {χ2n−1 /(n − 1)}1/2. Thus √ ¯ n − µ]/Sn ∼ (16) Tn ≡ n [X = Tn−1 ≡ Student tn−1 . Specify tp/2 such that P (−tp/2 ≤ Tn−1 ≤ tp/2 ) = 1 − p; perhaps p = .05. Then with the “large” probability of 1 − p = .95 we have √ ¯ n − µ]/Sn ≤ tp/2 ) (17) 1 − p = P (−tp/2 ≤ Tn ≤ tp/2 ) = P (−tp/2 ≤ n [X √ √ ¯ n ≤ µ + tp/2 Sn / n ≤ X ¯n) (18) = P (µ − tp/2 Sn / n ≤ X √ √ ¯ n − tp/2 Sn / n ≤ µ ≤ X ¯ n + tp/2 Sn / n) . (19) = P (X That is: (20)

¯ n ± tp/2 Sn /√n The random interval X will contain the unknown value of µ an average of (1 − p) × 100% of the time.

So when we apply this to the data values√x1 , . . . , xn , we can have (1 − p) × 100% confidence that the interval x ¯n ± tp/2 sn / n did enclose the true (but unknown) value of µ. We say that √ ¯ n ± tp/2 Sn / n provides a (1 − p) × 100% confidence interval (21) X for the unknown mean µ. Or we say that √ (22) x¯n ± tp/2 sn / n provides a (1 − p) × 100% numerical confidence interval for the unknown mean µ. There is a probability of 1−p (or a (1−p)×100% chance) that the former will contain the unknown value of µ when the X-experiment is repeated n times. There is a (1 − p) × 100% confidence (or degree of belief) that the latter did contain the unknown value of µ after the X-experiment √ was repeated n times giving the actual data values x1 , . . . , xn . We call tp/2 sn / n the numerical margin for error exhibited by our experiment. 2

190


Transformations of Random Variables Exercise 2.8 Suppose X has density fX (·) with respect to Lebesgue measure λn (·) on n-dimensional Euclidean space Rn . (a) Let Y ≡ AX denote a linear transformation with A a nonsingular matrix. The Jacobian of this linear transformation is ∂(old) ∂x i with |J|+ = |A−1 | = 1/|A|. J ≡ ≡ = A−1 , ∂(new) ∂yj Verify that the rv Y has a density fY (·) with respect to Lebesgue measure that is given by fY (y) = fX (A−1 y)/ |A| on Rn . (b) Suppose now that X has density fX (·) with respect to Lebesgue measure on a region RX in Rn . Suppose the 1-to-1 transformation Y ≡ g(X) from RX to the region RY ≡ g(RX ) has a nonsingular Jacobian with continuous elements at each point of the region. Show that Y has a density given by fY (y) = fX (g −1 (y)) × |[∂(old)/∂(new)]|. (Any “nice” transformation is locally linear.) Exercise 2.9 Suppose that U ≡ XY and V ≡ X/Y for rvs having joint density fXY (·, where x > 0 and y > 0. The inverse transformation is √·) on the region X = U V and Y = U/V with a “nice” Jacobian that is equal to 2v. Thus the joint density of U, V is √ 1 fU V (u, v) = 2v fXY ( uv, u/v )) on the appropriate (u, v)-region, provided that the transformation is 1-to-1. Now evaluate fU V (·, ·) and fV (·) in the following cases. (a) X and Y are independent Exponential(1). (b) X and Y are independent with density 1/(xy)2 on x, y ≥ 1. Evaluate fU (·). (c) X and Y are independent N(0, 1). [Note that this transformation is not 1-1.] (d) X ∼ = N(0, 1) and Y ∼ = Uniform(0, 1) are independent. [This exercise demonstrates vividly the important role played by the regions RX and RY .]

3.

LINEAR ALGEBRA APPLICATIONS

191

3

Linear Algebra Applications

Notation 3.1 (Mean vector and covariance matrix) Let X ≡ (X1 , . . . , Xn ) be a rv. Then E(X) ≡ µ ≡ (µ1 , . . . , µn ) , where µi ≡ E(Xi ) is called the mean vector. And Σ ≡ |[ σij ]| ≡ |[ Cov[Xi , Xj ] ]| is called the covariance matrix. (By the Cauchy–Schwarz inequality, both of µ and Σ are well-defined provided that each of 2 σii ≡ Var[Xi ] ≡ Cov[Xi , Xi ] is finite.) Definition 3.1 (Linear algebra) We will operate on n-dimensional space Rn with n × n matrices and n × 1 vectors. (i) A matrix Γ with column vectors γi (that is, Γ = [γ1 , . . . , γn ] ) is called orthogonal if Γ Γ = I. [Thus γi γj equals 1 or 0 according as i = j or i = j; when γi γj = 0 we say that these vectors are orthogonal, and we write γi ⊥ γj .] Under the orthogonal transformation of Rn onto itself defined by y = Γx, the image of each γi is the standardized basis vector ei ≡ (0, . . . , 0, 1, 0, . . . , 0) with the 1 in the ith slot. (ii) Call a symmetric matrix A positive definite (written A > 0) if x Ax > 0 for all vectors x = 0. Call it nonnegative definite (written A ≥ 0) if x Ax ≥ 0 for all vectors x = 0. (iii) If A is symmetric and idempotent (that is, if AA = A), then A is called a projection matrix (the symbol P is often used for a projection matrix). (iv) Let Da be the diagonal matrix with dii = ai (and dij = 0 for all i = j). (v) Let R[A] denote the column space of A; that is, it is the set of all vectors that can be written as linear combinations of the column vectors of A. n n (vi) Call x Ax = 1 1 xi aij xj a quadratic form in the vector x. What follows is the statistician’s main result from linear algebra. We simply state it, then interpret it geometrically in discussion 3.1, and then put it into a very useful format in discussion 3.2. Theorem 3.1 (Principal axes theorem) Let A denote an arbitrary real and symmetric matrix of rank r. (a) There exists an orthogonal matrix Γ ≡ [γ1 , . . . , γn ] and a diagonal matrix D for which we have the representation (1)

A = Γ D Γ

and/or

Γ A Γ = D

with rank(D) = r.

The γi are called eigenvectors, while the corresponding dii are called eigenvalues. (See (39) below for further comments.) (b) If A > 0 (A ≥ 0), then all dii > 0 (dii ≥ 0). We can specify Γ such that d11 ≥ · · · ≥ drr > 0 = dr+1,r+1 = · · · = dnn . (c) If P is a projection matrix, then all dii = 1 or 0. Moreover, we must have n r ≡ rank(A) = tr(D) = tr(A) = 1 aii . Discussion 3.1 (Spectral decomposition) Consider a projection matrix P of rank r. Then the transformation y = P x can be broken down as n (2) P x = Γ D Γ x = [γ1 , . . . , γn ] D [γ1 , . . . , γn ] x = 1 dii (γi x) γi ,

192


where (γi x) γi is the projection of x onto γi in the direction of γi , and where this term is present when dii = 1 and is absent when dii = 0. Also, P x ⊥ (I − P )x, where n (3) (I − P )x = i=1 [1 − dii ] (γi x) γi projects onto R[γr+1 , . . . , γn ] = R⊥ [γ1 , . . . , γr ]. Finally, P x = ri=1 dii (γi γi ) x = [ ri=1 Pi ] x (4) with Pi ≡ γi γi . This is called the spectral decomposition of the transformation y = P x. 2 Exercise 3.1 (5)

(a) Show that for compatible matrices B and C,

tr(BC) = tr(CB)

and rank(BC) ≤ rank(B) ∧ rank(C),

giving rank(A Γ) = rank(A) above. (b) Prove theorem 3.1(b)(c) using theorem 3.1(a). (c) Show that R[A] = R[AA ] and R[A ] = R[A A]. Proposition 3.1 (Properties of E(·)) (6)

E(A X B + C) = A E(X) B + C

(a) It holds that and Cov[A X, B Y ] = A Cov[X, Y ] B .

(b) Any covariance matrix ΣX ≡ |[ Cov[Xi , Xj ] ]| satisfies ΣX ≥ 0. Exercise 3.2


Discussion 3.2 (Versions of Σ− and Σ−1/2 ) Let X ∼ = (µ, Σ). According to the principal axes theorem, we can make the decomposition (for any orthogonal matrix ∆ whatsoever) 1/2 1/2 D 0 D 0 Σ = Γ D Γ = Γ ∆ ∆ Γ 0 0 0 0 (7) 1/2 1/2 D 0 D 0 = Γ Γ = ( Γ D1/2 )( D1/2 Γ ) ≡ AA , 0 0 0 0 1/2

where D1/2 has the numbers dii on its diagonal and where A is n×k. The presence of ∆ ∆ (which equals I ) shows that this decomposition is not unique. Continuing on gives (8)

Σ = Γ D Γ = (Γ D Γ ) (Γ D Γ ) ≡ Σ1/2 Σ1/2 ,

(9)

Σ−1/2 ≡ Γ D−1/2 Γ ,

where

dii

(10)

Σ− ≡ Γ D− Γ ,

where

D− ≡ D−1/2 D−1/2 .

−1/2

≡

0 √1 dii

if dii = 0, if dii > 0,

3.


193

Note that (11)

Σ Σ− Σ = Σ

(12)

Σ1/2 Σ−1/2 =

and

Σ1/2 Σ−1/2 Σ1/2 = Σ1/2 ,

Ik 0 = Σ−1/2 Σ1/2 0 0

and Σ Σ− =

Ik 0 = Σ− Σ . 0 0

These last two results are in keeping with the definition of generalized inverses. Recall that the generalized inverse B− of the matrix B is defined to be any matrix B − that satisfies B B − B = B. A generalized inverse always exists. It has the following interpretation. Fix the matrix B and the vector c. Then (13)

B β = c (with any c ∈ R[B] ) has the solution βˆ = B − c .

Stated this way, it is clear that the generalized inverse does always exist.

2

Theorem 3.2 (Properties of covariance matrices) (a) The following results are equivalent for real matrices: (14)

Σ is the covariance matrix of some rv Y.

(15)

Σ is symmetric and nonnegative definite.

(16)

There exists an n × n matrix A such that Σ = AA .

(Recall (7) for A.)

(b) The matrix ΣX is positive definite (that is, ΣX > 0) if and only a vector c = 0 and a constant b do not exist for which c X = b a.s. Proof. Now, (14) implies (15): Σ is symmetric, since EYi Yj = EYj Yi . Also, a Σa = Var[a Y ] ≥ 0 for all vectors a, so that Σ ≥ 0. Also, (15) implies (16): Just recall (7). Also, (16) implies (14): Let X ≡ (X1 , . . . , Xn ) , where X1 , . . . , Xn are independent N (0, 1). Let Y ≡ AX. Then Y has covariance matrix Σ = AA by (6). 2

Exercise 3.3

Prove theorem 3.2(b).

Exercise 3.4 Let X ∼ = (θ, Σ) and let B be symmetric. (a) E{(X − b) B (X − b)} = tr(B Σ) + (θ − b) B (θ − b). n (b) If Σ = σ2 I, then tr(B Σ) = σ2 tr(B) = σ2 i=1 bii . (c) If Σ = σ2 I and B is idempotent, then tr(B Σ) = σ2 tr(B) = σ2 rank(B). Exercise 3.5 For symmetric A there exists an upper (or lower) triangular matrix H for which A = H H . If A > 0 (or A ≥ 0), we may suppose that all hii > 0 (or hii ≥ 0).

194


Discussion 3.3 (Best linear predictor and multiple correlation) the partitioned random vector Y0 ∼ 0 σ00 σ0 (17) , with |Σ| = 0 . = Y 0 σ0 Σ

Consider

The best linear predictor of Y0 based on Y is (18)

α0 Y ≡ σ0 Σ−1 Y

(or α0 ≡ Σ−1 σ0 ) ,

where “best” is in the sense that (19)

Var[ Y0 − β Y ] ≥ Var[ Y0 − α0 Y ] = σ00 − σ0 Σ σ0

for all β.

In parallel with this, (20)

Corr[ Y0 , β Y ] ≥ Corr[ Y0 , α0 Y ]

for all β.

The maximized value of the correlation (that is, the multiple correlation coefficient) is given by , σ0 Σ−1 σ0 (21) , ρ0·1,...,n ≡ Corr[ Y0 , α0 Y ] = σ00 and the variance of the best linear predictor is also easily seen to equal (22)

Var[ Y0 − α0 Y ] = σ00 − σ0 Σ−1 σ0 = σ00 (1 − ρ20·1,...,n ) .

[Proof. The first holds, since Var[ Y0 − β Y ] = Var[ (Y0 − α0 Y ) + (α0 − β)Y ] = Var[ Y0 − α0 Y ] + 2 · 0 + (α0 − β) Σ (α0 − β) ≥ Var[ Y − α0 Y ] . The second holds, since Corr2 [ Y0 , β Y ] =

σ Σ−1 σ0 (β σ0 )2 ≤ 0 , σ00 β Σ β σ00

with equality only at β = c Σ−1 σ0 (as follows from application of Cauchy–Schwarz).] Simple linear regression model We now want the best linear predictor of Y based on X. The conditional distribution of Y given that X = x is given by 2 σXY σXY 2 (23) Y X = x ∼ µ + (x − µ ), σ − = (α + β x, σ2 ) , = Y X Y 2 2 σX σX expressing the moments in terms of σXY σY σXY (24) ρ≡ , β≡ρ , α ≡ µY − µX 2 , σ2 ≡ σY2 (1 − ρ2 ) . σX σY σX σX This leads directly to the simple linear regression model that conditionally on X = x the observations Yi satisfy (25) Yi = α + β xi + i where i ∼ = (0, σ2 ) are iid

with (26)

β=ρ

σY , σX

σ2 ≡ σY2 (1 − ρ2 ) ,

α ≡ µY − µX β .

2

3.


195

Discussion 3.4 (Conditional moments and projections) (1) (1) µ Σ11 Σ12 Y ∼ (27) , . Y = = Y (2) µ(2) Σ21 Σ22

Suppose that

Then the moments of the conditional distribution of Y (1) given that Y (2) = y (2) are summarized in (2) (28) Y (1) Y (2) = y (2) ∼ − µ(2) ), Σ11 − Σ12 Σ−1 = (µ(1) + Σ12 Σ−1 22 (y 22 Σ21 ). To see this, just define (1) (1) (2) − µ(1) ) − Σ12 Σ−1 − µ(2) ) Z (Y 22 (Y Z≡ (29) ≡ . Y (2) − µ(2) Z (2) It is a minor calculation that 0 Σ11 − Σ12 Σ−1 22 Σ21 0 (30) Z∼ , . = 0 0 Σ22 The exercises will show that (31)

|Σ| = |Σ22 | |Σ11 − Σ12 Σ−1 22 Σ21 | .

[Proof. It is easy to compute ΣZ,12 = Σ12 − Σ12 Σ−1 22 Σ22 = 0. Trivially, ΣZ,22 = Σ22 . Then (a) (b)

−1 −1 ΣZ,11 = Σ12 − 2Σ12 Σ−1 22 Σ22 + Σ12 Σ22 Σ22 Σ22 Σ21

= Σ12 − Σ12 Σ−1 22 Σ21 .

(2) Since Y (1) = µ(1) + Z (1) + Σ12 Σ−1 with Y (2) = Z (2) , we see that conditionally 22 Z (2) (c) Y (1) Y (2) = y (2) ∼ = µ(1) + Z (1) + Σ12 Σ−1 22 z (2) ∼ , ΣZ,11 ) = (µ(1) + Σ12 Σ−1 22 z (2) = (µ(1) + Σ12 Σ−1 − µ(2) ), ΣZ,11 ) , 22 (y

as required.] [See exercise 3.7 below for (31).]

2

Exercise 3.6 Consider the rvs Z (1) and Z (2) in (29). Suppose µ(1) = 0 and µ(2) = 0. Let H1 and H2 denote the Hilbert spaces generated by the rv subsets Y (1) ≡ (Y1 , . . . , Yk ) and Y (2) ≡ (Yk+1 , . . . , Yn ), respectively. Show that Z (1) is the projection of Y (1) into the Hilbert space H2⊥ . (See (6.3.1).) Discussion 3.5 (Partitioned matrices) Let 11 12 A11 A12 A A (32) A≡ , and write A−1 ≡ A21 A22 A21 A22 when the inverse exists. We agree that A11 is k × k.

196


Exercise 3.7 (a) If |A22 | = 0, show that |A| = |A22 | |A11 − A12 A−1 22 A21 |. (b) If |A| = 0, show that |A + x y | = |A| (1 + y A−1 x) for all vectors x, y. [Hint. Appeal to 0 C 0 I = |C| |D| and work with B≡ ED −A12 A−1 22 I for appropriate choices.] Exercise 3.8

(a) Show that for a symmetric A having |A11 | = 0 and |A22 | = 0 :

(33)

−1 A11 = (A11 − A12 A−1 22 A21 )

22 and A12 = −A−1 11 A12 A .

(34)

−1 A22 = (A22 − A21 A−1 11 A12 )

11 and A21 = −A−1 22 A21 A .

[Hint. Start multiplying the partioned form of A A−1 = I.] (b) Obtain analogous formulas from A−1 A = I. (c) Show that (35)

A11 A11 + A12 A21 = I

Exercise 3.9 (36)

and A11 A12 + A12 A22 = 0 .

2

Show that for symmetric A,

∂ [ β A β] = 2 A β . ∂β

Discussion 3.6 (Simultaneous decomposition) For a real symmetric matrix A that is nonnegative definite (that is, A ≥ 0) we wrote (37)

A = Γ D Γ

and

Γ A Γ = D

with d11 ≥ · · · ≥ drr > 0. (A) We note that (38)

|A − λ I| = |Γ| |D − λ I| |Γ | = |D − λ I| = 0

all have the same solutions d11 , . . . , drr , 0, and thus d11 , . . . , drr are indeed the nonzero eigenvalues of A. Moreover, (37) gives (39)

AΓ = ΓD

or A γi = dii γi

for 1 ≤ i ≤ r,

so that γ1 , . . . , γr are the corresponding eigenvectors. (B) Suppose A > 0 and B ≥ 0. Then (40)

|B − λ A| = 0 , |A−1/2 B A−1/2 − λ I| = 0 , have the same solutions λ.

|A−1 B − λ I| = 0

[Just note that |B − λ A| = |A1/2 | |A−1/2 B A−1/2 − λ I| |A1/2 | and |A−1 | |B − λ A| = |A−1 B − λ I| .] Writing A−1/2 B A−1/2 = ∆ Dθ ∆ with ∆ orthogonal gives (41)

B = (A1/2 ∆) Dθ (∆ A1/2 ) and A = (A1/2 ∆) (∆ A1/2 ) .

This last formula is called the simultaneous decomposition of A and B.

2

3.


Discussion 3.7 (42)

197

(a) (Cauchy–Schwarz) For all vectors x, y:

(x y)2 ≤ x2 y2 ,

with equality (for y = 0) if and only if x = cy for some constant c. (b) For any real symmetric matrix A > 0 (43)

a A a = d11 (A) a a

max a =0

(as follows immediately from (1), with d11 (A) the largest eigenvalue of A). (c) Let A > 0, and fix C ≥ 0 and b = 0. Then (44)

(x y)2 ≤ (x A x) (y A−1 y),

with equality (when y = 0) if and only if x = c A−1 y for some c. Also, (45)

min {a A a} = 1/(1 A−1 1),

a 1=1

with equality only at a0 ≡ A−1 1/(1 A−1 1). Also, (46)

max a =0

(a b)2 = b A−1 b, a A a

with equality only when a = (some c) A−1 b. Also, max a =0

(47)

a C a (a A1/2 ) (A−1/2 C A−1/2 ) (A1/2 a) = max a =0 a A a (a A1/2 ) (A1/2 a)

= d11 (A−1/2 CA−1/2 ) = d11 (C A−1 ) ,

with (a C a) = (a b)2 an important special case (also approachable via (46)). (d) Let A > 0, let Bk×n have rank(B) = k, and let bk×1 = 0. Then (48)

min {a A a} = b [B A−1 B ]−1 b,

B a=b

is achieved at a0 ≡ B [B A−1 B ]−1 b .

Exercise 3.10 Prove (42)–(47) (the equality in (42) needs some attention). [The harder (48) is proven below.] Proof.

Consider (48). Now, a A a ≥

(a y)2 y A−1 y

for all y = 0

(a)

=

[a B [B A−1 B ]−1 b ]2 b [B A−1 B ]−1 (B A−1 B )[B A−1 B ]−1 b

(b)

=

[b [B A−1 B ]−1 b ]2 b [B A−1 B ]−1 b

if y ≡ B [B A−1 B ]−1 b

for all a, using B a = b

= b [B A−1 B ]−1 b yielding a bound not depending on a, which proves (48).

2

198


Exercise 3.11 (Minimum variance unbiased linear estimators) (a) Let X1 , . . . , Xn 2 be uncorrelated with common mean n µ and common finite variance σ . All linear n estimators T ≡ 1 ai Xi having 1 ai = 1 are unbiased estimators of µ (that is, ET = µ). Show that the choice with all ai = 1/n has minimum variance within this class of linear unbiased estimators. (b) Determine the minimum variance unbiased linear estimator of the common mean µ when the variances are σ2 /c1 , . . . , σn2 /cn , with the ck being known constants.

4.

THE MULTIVARIATE NORMAL DISTRIBUTION

4

The Multivariate Normal Distribution

199

Definition 4.1 (Jointly normal) Call Y = (Y1 , . . . , Yn ) jointly normal with 0 means if there exist iid N (0, 1) rvs X1 , . . . , Xk and an n × k matrix A of known , constants for which Y = AX. [We again write Y in this section, rather than Y when the context seems clear.] Note that the n × n covariance matrix ΣY ≡ Σ of the random vector Y is (1)

Σ ≡ ΣY = E Y Y = EAXX A = AA .

The covariance matrix of X is the k × k identity matrix Ik . We will write X ∼ = N (0, Ik ), and we will write Y ∼ = N (0, Σ). Then write Y ∼ = N (µ, Σ) if Y − µ ∼ = N (0, Σ). Call Y multivariate normal with mean vector µ and covariances matrix Σ. Call Y nondegenerate when |Σ| = 0 (that is, the determinant of Σ is not equal to 0). Say that Y1 , . . . , Yn are linearly independent if (rank Σ) = n. Of course, this means that (2)

Y is nondegenerate if and only if rank(A) = n.

Now, Σ is symmetric. Also aΣa = Var[aY ] ≥ 0 for all vectors a. When aΣa ≥ 0 for all vectors a, the symmetric matrix Σ is called nonnegative definite, and one writes Σ ≥ 0. Theorem 4.1 (Densities) If Y ∼ = N (0, Σ) is nondegenerate, then Y has density (with respect to Lebesgue measure on Rn ) given by 1 fY (y) = (3) exp(−y Σ−1 y/2) for all y ∈ Rn . (2π)n/2 |Σ|1/2 [Note that each possible normal distribution is completely determined by µ and Σ.] ∼ N (0, Ik ). Proof. Now, Y = XA, where AA = Σ, (rankA) = n, |A| = 0, X = It is trivial that (a) with fX (x) ≡ (2π)−n/2 exp(−x x/2). P (X ∈ Bn ) = 1Bn fX (x)dx Thus X = A−1 Y gives

P (Y ∈ Bn ) = P (AX ∈ Bn ) = P (X ∈ A−1 Bn ) = 1A−1 Bn (x)fX (x) dx ∂x = 1A−1 Bn (A−1 y)fX (A−1 y)| | dy ∂y ∂x = 1Bn (y)(2π)−n/2 exp(−(A−1 y) (A−1 y)/2)| | dy ∂y = (2π)−n/2 |Σ|−1/2 exp(−y Σ−1 y/2) dy,

(b)

Bn −1

) (A−1 ) = (AA )−1 = Σ−1 and ∂x (c) | | = |A−1 | = |A−1 ||A−1 | = |Σ−1 | = 1/ |Σ|. ∂y This is the required statement. 2

since (A

200


Theorem 4.2 (Characteristic functions and representations) (a) If we are given a random vector Y = An×k X k×1 where X ∼ = N (0, Ik ), we have (4)

φY (t) ≡ E eit Y = exp(−t Σt/2)

with Σ ≡ AA and rank(Σ) = rank(A). (b) If Y has characteristic function φY (t) ≡ EeitY = exp(−t Σt/2) with Σ ≥ 0 of rank k, then Y ∼ (5) with (rank A) = k and X ∼ = An×k X k×1 = N (0, I). (Thus the number of independent rvs Xi ’s needed is equal to the rank of A.) Proof. Our proof will use the fact that the characteristic function φY of any rv Y is unique (as will be shown below in chapter 13.) [When a density function does not exist, one can use this characteristic function for many of the same purposes.] We observe that φY (t) = E exp(it AX) = E exp(i(A t) X) (a)

= exp(−(A t) (A t)/2)

since E eitX j = exp(−t2 /2)

by example 13.1.2 below (b)

= exp(−t (AA )t/2).

The converse follows from (9.3.7).

2

Even when a multivariate normal rv Y does not have a density, the characteristic function can often be manipulated to establish a desired result. Theorem 4.3 (Marginals, independence, and linear combinations) Suppose that Σ11 Σ12 Y = (Y1 , . . . , Yk , Yk+1 , . . . , Yn ) ∼ . with Σ≡ = N (0, Σ) Σ21 Σ22 (i) The marginal covariance matrix of (Y1 , . . . , Yk ) is the k × k matrix Σ11 , and (Y1 , . . . , Yk ) ∼ (6) = N (0, Σ11 ). (ii) If Σ12 = 0, then (Y1 , . . . , Yk ) and (Yk+1 , . . . , Yn ) are independent. (iii) If (Y1 , Y2 ) is a jointly normal rv, then Y1 and Y2 are independent if and only if they have the zero covariance Cov[Y1 , Y2 ] = 0. (iv) Linear combinations of normals are normal. Proof. (i) Use the first k coordinates of the representation Y = AX. (ii) Use the fact that one can factor 1 Σ11 0 φY (t) = exp(− t t) . 0 Σ22 2 (iii) Just apply (ii), as the other direction is trivial. (iv) Z m×1 ≡ B m×n Y n×1 = B(AX) = (BA)X. 2

4.

THE MULTIVARIATE NORMAL DISTRIBUTION

201

Theorem 4.4 (Conditional distributions) If (1) (1) Y µ Σ11 Σ12 ∼ Y = (7) N , , = Y (2) µ(2) Σ21 Σ22 then (8)

(2) − µ(2) ), Σ11 − Σ12 Σ−1 Y (1) Y (2) = y (2) ∼ = N (µ(1) + Σ12 Σ−1 22 (y 22 Σ21 ).

Note that (9)

|Σ| = |Σ22 ||Σ11 − Σ12 Σ−1 22 Σ21 |.

Proof.

The vector (1) (1) (2) Z − µ(1) ) − Σ12 Σ−1 − µ(2) ) (Y 22 (Y Z≡ ≡ Z (2) Y (2) − µ(2)

(10)

is just a linear combination of the Yi ’s, and so it is normal. We need only verify the means and variances. But we did this in discussion 9.3.4. 2 Exercise 4.1 Show that (Y1 , Y2 ) can have normal marginals without being jointly normal. [Hint. Consider starting with a joint N (0, I) density on R2 and move mass in a symmetric fashion to make the joint distribution nonnormal, but still keeping the marginals normal.]

Quadratic Forms Exercise 4.2 Let Y n×1 ∼ = N (0, I), and suppose that A is symmetric and of rank r. Then Y AY ∼ = χ2r if and only if A is a projection matrix (that is, A2 = A). Exercise 4.3 Let Y n×1 ∼ = N (0, I). Suppose that A and B are symmetric and both Y AY and Y BY have chisquare distributions. Show that Y AY and Y BY are independent if and only if AB = 0. Exercise 4.4 Suppose A and B are n × n projection matrices with ranks rA and rB , and suppose A B = 0 and I − A − B ≥ 0. Then: (a) I − A is a projection matrix of rank n − rA . (b) I − A − B is a projection matrix of rank n − rA − rB . Exercise 4.5 Suppose Y n×1 ∼ = N (0, Σ), and let A be an arbitrary symmetric matrix of rank r. Show that Y AY ∼ = χ2r if and only if AΣA = A.

The Multivariate CLT The following result is theorem 14.1.3, but we also list it here for convenient referral. Theorem 4.5 Suppose that the random vectors X1 , . . . , Xn are iid (µ, Σ). Then √ ¯ n − µ) →d N (0, Σ) (11) n (X as n → ∞.

202


Normal Processes To specify a normal process, we must specify consistent distributions (in the sense of Kolmogorov’s consistency theorem). But µ and Σ completely specify N (µ, Σ), while the marginals of N (µ, Σ) are N (µ(1) , Σ11 ). Thus a normal process exists, provided only that the mean value function µ(·) on I and the covariance function Cov(·, ·) on I × I are well-defined and are such that Cov(·, ·) is nonnegative definite (meaning that every n-dimensional covariance matrix formed from it is nonnegative definite). We call {S(t) : 0 ≤ t < ∞} a Brownian motion if S is a normal process having (12)

ES(t) = 0

and

Cov[S(s), S(t)] = s ∧ t

for all s, t ≥ 0.

Since this covariance function is nonnegative definite, a version of the process S exists on (R[0,∞) , B[0,∞) ) by the Kolmogorov consistency condition. Then (13)

U(t) ≡ −[S(t) − t S(1)] for all 0 ≤ t ≤ 1 is called Brownian bridge.

It is a normal process on (R[0,1] , B[0,1] ) for which (14)

EU(t) = 0

and

Cov[U(s), U(t)] = s ∧ t − st

for all 0 ≤ s, t ≤ 1.

Chapter 10

WLLN, SLLN, LIL, and Series 0

Introduction

This is one of the classically important chapters of this text. The first three sections of it are devoted to developing the specific tools we will need. In the second section we also present Khinchin’s weak law of large numbers (WLLN), which can be viewed as anticipating both of the classical laws of large numbers (LLNs). Both the classical weak law of large numbers (Feller’s WLLN) and classical strong law of large numbers (Kolmogorov’s SLLN) are presented in section 10.4, where appropriate negligibility of the summands is also emphasized. This section is the main focus of the chapter. Some applications of these LLNs are given in the following section 10.5. Then we branch out. Suppose a sample moment is formed when the underlying true moment is infinite. Can a parameter be defined that captures the behavior of this estimator? This is the subject of section 10.6, and it ties in rather closely with section 7.7. The law of the iterated logarithm (LIL), the strong Markov property, and convergence of infinite series are treated in sections 10.7 – 10.9. The choice was made to be rather specific in section 10.4, with easy generalizations in section 10.9. The usual choice is to begin more generally, and then specialize. Martingales (mgs) are introduced briefly in section 10.10, both for limited use in chapter 12 and so that the inequalities in the following section 10.11 can be presented in appropriate generality. The final section 10.12 contains a uniform version of the earlier SLLN.

203

204

1

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

Borel–Cantelli and Kronecker lemmas

The first three sections will develop the required tools, while applications will begin with the LLNs (the first of which appears in section 10.3). We use the notation ∞ ∞ [An i.o.] = [ω : ω ∈ An infinitely often] = n=1 m=n Am = limn An . (1) This concept is important in dealing with convergence of various random elements. The following lemmas exhibit a nice dichotomy relative to sequences of independent events. Lemma 1.1 (Borel–Cantelli lemma) ∞

(2)

P (An ) < ∞

implies

For any events A n ,

P (An i.o.) = 0.

n=1

Lemma 1.2 (Second Borel–Cantelli lemma) events A1 , A2 , . . . , we have the converse ∞

(3)

P (An ) = ∞

implies

For a sequence of independent

P (An i.o.) = 1.

n=1

Thus ∞ independent events A1 , A2 , . . . have P (An i.o.) equal to 0 or 1 according as 1 P (An ) is finite or infinite. Proof.

We use proposition 1.1.2 freely. Now, ∞ ∞ ∞ (a) P (An i.o.) = P ( limn n Am ) = limn P ( n Am ) ≤ limn n P (Am ) = 0 ∞ whenever 1 P (An ) < ∞. Also, ∞ ∞ ∞ P ([ lim An ]c ) = P ( n=1 m=n Acm ) = limn P ( m=n Acm ) = limn limN P (

N

c m=n Am )

N

= limn limN

(c)

N ≤ limn limN exp(− m=n P (Am ))

(d)

∞ = limn exp(− m=n P (Am )) = limn exp(−∞) = limn 0 = 0,

using

∞ 1

P (An ) = ∞.

m=n [1

− P (Am )]

(b)

by independence since 1 − x ≤ exp(−x)

2

Remark 1.1 (Kolmogorov’s 0-1 law) In theorem 8.2.1 we considered the tail σ-field T ≡ ∩∞ n=1 F(Xn , Xn+1 , . . .) of an arbitrary sequence of independent rvs X1 , X2 , . . . . We learned that P (D) = 0 or 1 for all D ∈ T . 2

1.

BOREL–CANTELLI AND KRONECKER LEMMAS

205

Lemmas About Real Numbers There is an important bridge going from the convergence of series to the convergence of averages. [An alternative bridge is provided by the monotone inequality 10.11.1 (see exercise 10.4.10).] Lemma 1.3 (Kronecker’s lemma) Let x1 , x2 , . . . be arbitrary real numbers. If bn ≥ 0 and ∞, then (4)

n

xk → r

implies

1

Proof. (a)

1 bn

k Let sk ≡ 1 xk with s0 ≡ 0 and b0 ≡ 0. Summing by parts gives n 1 n 1 n−1 (bk − bk+1 ) sk + b1n bn sn 1 bk xk = bn 1 bk (sk − sk−1 ) = bn 0 =−

(c)

=−

(d)

n 1

n 1

ak sk−1 + sn

where

ak ≡

n bk − bk−1 ≥ 0 with 1 ak = 1 bn

ak (sk−1 − r) + (sn − r).

n 1 n N bn 1 bk xk ≤ 1 |ak (sk−1 − r)| + N +1 |ak (sk−1 − r)| + |sn − r| N

(e)

bk xk /bn → 0.

1

(b)

Thus

n

≤

1

≤ 3

(bk − bk−1 )|sk−1 − r| + +

bn

for N sufficiently large, but fixed

for n sufficiently larger than N ,

so that thelemma holds. n n [Since 1 xk → r, we must have xk → 0. Note that 1 bk xk /bn puts large weight only on the later terms.] 2 Exercise 1.1 (Ces` aro summability) If Sn ≡

n k=1

xk → r, then

n k=1

Sk /n → r.

Exercise 1.2 (Toeplitz) Let ank (for 1 ≤ k ≤ kn , with kn → ∞) be such that: kn (i) For every fixed k, we have ank → 0. (ii) k=1 |ank | ≤ c < ∞, for every n. kn Let xn ≡ k=1 ank xk . Then (a) xn → 0 implies kn If k=1 ank → 1, then

xn → 0.

implies xn → x. n In particular, if bn ≡ k=1 ak ∞, then n (c) entails xn → x finite k=1 ak xk /bn → x. (b)

xn → x

[This exercise will not be employed anywhere in this text.]

206

2


Truncation, WLLN, and Review of Inequalities

Truncated rvs necessarily have moments, and this makes them easier to work with. But it is crucial not to lose anything in the truncation. Definition 2.1 (Khinchin equivalence) Two sequences of rvs X1 , X2 , . . . and ∞ Y1 , Y2 , . . . for which n=1 P (Xn = Yn ) < ∞ are called Khinchin equivalent. Proposition 2.1 Let X1 , X2 , . . . and Y1 , Y2 , . . . be Khinchin equivalent rvs. (a) If Xn →a.s. or p , then Yn →a.s. or p , and the limits are equal. n n (b) If Sn ≡ 1 Xk →a.s. or p , then Tn ≡ 1 Yk →a.s. or p . (c) If Sn /bn →a.s. or p and bn → ∞, then Tn /bn →a.s. or p , with equal limits. Proof. (p)

The Borel–Cantelli lemma gives P (Xn = Yn i.o.) = 0.

Thus the a.s. statements for Xn and Sn are trivial. Moreover, since Xn (ω) = Yn (ω) for all n ≥ some n(ω), we have (q) (r)

Sn(ω) + Sn − Sn(ω) Sn(ω) + Tn − Tn(ω) Sn(ω) − Tn(ω) Tn Sn = = = + bn bn bn bn bn = o(1) + Tn /bn

using bn → ∞. Since a sequence (such as Xn , Sn or Sn /bn ) converges in probability if and only if each subsequence n contains a further subsequence n on which the convergence is a.s., the in probability statements follow directly from the a.s. statements. 2 Inequality 2.1 (Sandwiching E|X|) (1)

∞

P (|X| ≥ n) ≤ E|X| ≤

n=1

∞

For any rv X we have (see figure 2.1)

P (|X| ≥ n).

n=0

If X is a rv with values 0, 1, 2, . . . , then (2)

E(X) =

∞

P (X ≥ n).

n=1

∞ Proof. If X ≥ 0, then EX = 0 [1 − F (x)] dF (x) by (7.4.11). If X ≥ 0 is integer valued, then ∞ k E(X) = ∞ k=0 k P (X = k) = k=1 n=1 P (X = k) ∞ ∞ = n=1 k=n P (X = k) (a) = ∞ n=1 P (X ≥ n). For the greatest integer function [·], an arbitrary rv satisfies (b)

[|X|] ≤ |X| ≤ [|X|] + 1.

2.

TRUNCATION, WLLN, AND REVIEW OF INEQUALITIES

207

Moreover, (a) shows that ∞ ∞ (c) E[|X|] = n=1 P ([|X|] ≥ n) = n=1 P (|X| ≥ n) , with (d)

E{ [|X|] + 1 } =

∞

n=1 P (|X|

≥ n) + 1 =

ˇ ˜ Example 2.1 (Truncating and Winsorizing) truncate and Winsorize the Xn by defining (3)

∞

n=1 P (|X|

≥ n) + P (|X| ≥ 0).2

Let X1 , X2 , . . . be iid as X. Let us

ˇ n = Xn ×1[|X | 0, Chebyshev’s inequality gives √ 2 P ( |Y¯n − µn | > ) ≤ n1 2 n Var[Yn1 ] ≤ 1n E(Yn1 (a) )

2 (b) ≤ 1n E Yn1 1[|X1 |≤ √n ] + 1[|X1 |> √n ] 2 n n

+

(d)

≤ +

(e)

≤ 2

(c)

≤

1

n n

[

[|x|>

√ n n) → 0 n 1 (c) 1 E {|Xnk | × 1[|Xnk |≤n ] } → 0 n

for all > 0 .

¯ n − µn →p 0 still holds even if (c) is weakened appropriately. (ii) Show that X Remark 2.1 There are two natural ways to proceed to improve Khinchin’s WLLN ¯ n →a.s. µ ; and this is in the iid case. One way is to obtain the conclusion X done in Kolmogorov’s SLLN (theorem 10.4.2 below). Another way is to relax the assumption of a finite mean and center differently; and this is done in Feller’s WLLN (theorem 10.4.1 below). [Other possibilities and other approaches will be outlined in the exercises of section 10.4.] In section 10.3 we will develop a number of inequalities (so called “maximal inequalities”) to help us to the stated goal. (At the end of this section the reader could go directly to section 10.4, and then go to section 10.3 for the inequalities as they are needed.) 2

2.

TRUNCATION, WLLN, AND REVIEW OF INEQUALITIES

209

Review of General Inequalities from Measure Theory As we have completed the transition from measure theory to probability theory, we take this opportunity to restate without comment a few of the most important inequalities presented earlier. Inequality 2.2 (Review) Then:

Let X and Y be rvs on a probability space (Ω, A, P ).

(9)

Cr -inequality: E|X + Y |r ≤ Cr {E|X|r + E|Y |r } for r > 0, Cr ≡ 2(r∨1)−1 .

(10)

H¨ older:

E|XY | ≤ (E|X|r )1/r (E|Y |s )1/s

for r > 1, 1r +

(11)

Liapunov:

(E|X|r )1/r is in r,

for r ≥ 0.

(12)

Markov:

P (|X| ≥ λ) ≤ E|X|r /λr

(13)

Dispersion :

E|X| ≤ E|X + Y |

(14)

Jensen:

g(EX) ≤ Eg(X)

for all λ > 0;

1 s

= 1.

requires r > 0.

for independent rvs with 0 means.

if g is convex on some (a, b) ⊂ R having P (X ∈ (a, b)) = 1, and if EX is finite. Definition 2.2 (“Big ohp ”, and “little ohp ”, =a , and “at most” ⊕) (a) We say that Zn is bounded in probability [and write Zn = Op (1)] if for all > 0 there exists a constant M for which P (|Zn | ≥ M ) < . For a sequence an , we write Zn = Op (an ) if Zn /an = Op (1); and we say that Zn is of order an , in probability. (b) If Zn →p 0, we write Zn = op (1). We write Zn = op (an ) if Zn /an →p 0. (c) This notation (without subscript p) was also used for sequences of real numbers zn and an . For example, zn = o(an ) if zn /an → 0. (Note that o(an ) = op (an ).) (d) Write Un =a Vn if Un −Vn →p 0; and call Un and Vn asymptotically equal. (This is effectively a passage to the limit that still allows n to appear on the right-side.) (e) We write a = b ⊕ c if |a − b| ≤ c. (This can be used in the same fashion as op (·), but it allows one to keep track of an absolute bound on the difference. Especially, it allows inequalities to be strung together more effectively.) Exercise 2.3 For independent rvs X and Y and r ≥ 1, we have E|X + Y |r is finite if and only if E|X|r and E|Y |r are finite. Exercise 2.4 γn Vn →p 0.

If Vn = Op (1) and γn = op (1) are rvs on the same (Ω, A, P ), then

210

3


Maximal Inequalities and Symmetrization

Sums of independent random variables play an important role in probability and statistics. Our goal initially in this section is to develop probability bounds for the maximum of the first n partial sums. Such inequalities are called maximal inequalities. The most famous of these is Kolmogorov’s inequality. For symmetric rvs, Lévy’s inequality is an extremely clean and powerful version of such a maximal inequality; it does not require the underlying rvs to have any moments. Neither does the Ottavani–Skorokhod inequality, which is true for arbitrary rvs, though it is not nearly as clean. Inequality 3.1 (Kolmogorov) Let X1 , X2 , . . . be independent, with Xk ∼ = (0, σk2 ). Let Sk ≡ X1 + · · · + Xk . Then n (1) P max |Sk | ≥ λ ≤ Var[Sn ]/λ2 = σk2 /λ2 for all λ > 0. 1≤k≤n

k=1

[This contains Chebyshev’s inequality that P (|Sn | ≥ λ) ≤ Var[Sn ]/λ2 for all λ > 0.] n Proof. Let Ak ≡ [max1≤j 0. We may replace ≥ by > in the three events in (2).

3.

MAXIMAL INEQUALITIES AND SYMMETRIZATION

211

Let m ≡ median(X). Now, the first inequality comes from

Proof.

P (X s ≥ λ) = P [(X − m) − (X − m) ≥ λ] ≥ P (X − m ≥ λ) P (X − m ≤ 0) ≥ P (X − m ≥ λ)/2.

(a)

The second inequality holds, since for any real a, P (|X s | ≥ λ) = P (|(X − a) − (X − a)| ≥ λ) ≤ P (|X − a| ≥ λ/2) + P (|X − a| ≥ λ/2) = 2P (|X − a| ≥ λ/2).

(b)

2

Inequality 3.3 (L´ evy) Let X1 , . . . , Xn be independent and symmetric rv’s. Let Sn ≡ X1 + · · · + Xn . Then (3) for all λ > 0, P max |Sk | ≥ λ ≤ 2 P (|Sn | ≥ λ) 1≤k≤n

(4)

P

max |Xk | ≥ λ ≤ 2P (|Sn | ≥ λ)

1≤k≤n

for all λ > 0.

Proof. Let Ak ≡ [max1≤j 0, exp(−λ2 /2) ≤ πλ both Lévy’s inequality and the Ottavani–Skorokhod inequality to follow offer the hope of a much better bound. 2 Inequality 3.4 Let Sk ≡ X1 + · · · + Xk for independent rvs Xk . (Ottavani–Skorokhod) For all 0 < c < 1 we have P (|Sn | ≥ cλ) (7) P max |Sk | ≥ λ ≤ for λ > 0. 1≤k≤n [1 − max1≤k≤n P (|Sn − Sk | > (1 − c)λ)] √ (8) ≤ 2 P (|Sn | ≥ cλ) for all λ ≥ 2 StDev[Sn ]/(1 − c). (Etemadi) Alternatively, P (max1≤k≤n |Sk | ≥ 4λ) ≤ 4 max1≤k≤n P (|Sk | ≥ λ)

for all λ > 0.

n Proof. Let Ak ≡ [S1 < λ, . . . , Sk−1 < λ, Sk ≥ λ], so that k=1 Ak = [max1≤k≤n Sk ≥ λ]. Thus k is the smallest index for which Sk exceeds λ. (This is now the third time we have used this same trick.) Note that (a) (b)

a ≡ min1≤k≤n P (Sn − Sk ≤ (1 − c)λ) = 1 − max1≤k≤n P (Sn − Sk > (1 − c)λ) ≥ 1 − max1≤k≤n Var[Sn − Sk ]/[(1 − c)λ]2

(c)

by Chebyshev’s inequality

≥ 1 − Var[Sn ]/[(1 − c)λ]2 √ ≥ 12 if λ ≥ 2 StDev[Sn ]/(1 − c)

allows us to “improve” (7) to (8). Meanwhile, (7) comes from n (d) a × P (max1≤k≤n Sk ≥ λ) ≤ k=1 P (Sn − Sk ≤ (1 − c)λ)P (Ak ) (e) = nk=1 P (Ak ∩ [Sn − Sk ≤ (1 − c)λ]) by independence (f)

≤ P (Sn ≥ cλ).

Combining (f) and (b) with the symmetric result completes the proof.

Exercise 3.1

Prove Etemadi’s inequality.

2

3.

MAXIMAL INEQUALITIES AND SYMMETRIZATION

213

Exercise 3.2 Consider the independent symmetrized rvs Xks ≡ Xk − Xk , for 1 ≤ k ≤ n, where each Xk is an independent copy of Xk . Let mk denote a median of Xk , and let a denote an arbitrary number. Show that for all λ > 0 we have (9)

2−1 P (max |Xk − mk | ≥ λ) ≤ P (max |Xks | ≥ λ) ≤ 2P (max |Xk − a| ≥ λ/2).

[Then P (max |Xks | ≥ λ) ≤ 2P (|Sns | ≥ λ) ≤ 4P (|Sn − a| ≥ λ/2) (for any real a), by inequality 3.3 and inequality 3.2.]

Inequalities for Rademacher RVs Inequality 3.5 (Symmetrization; Gin´ e–Zinn) Let X1 , . . . , Xn be iid rvs, and let 1 , . . . , n denote an independent sample of iid Radamacher rvs (that satisfy P ( k = ±1) = 12 ). Then n . / . / n 1 1 P √ (10) for all λ > 0 .

i Xi > 2λ ≤ sup 2 P √ Xk > λ n n n≥1 k=1

Proof. (a)

By conditioning on the Rademacher rvs we obtain n P (n−1/2 | 1 k Xk | > 2 λ) ≤ P (n−1/2 | k:k =1 k Xk | > λ) + P (n−1/2 | k:k =−1 k Xk | > λ) ≤ E P (n−1/2 |

(b)

k=1

k:k =1

≤ 2 supk≤n P (n−1/2 | ≤ 2 supk≤n P (k−1/2 |

(c)

k 1

k

≤ 2 supk≤∞ P (k−1/2 |

as required.

Xk | > λ) + E P (n−1/2 |

1

k:k =−1

Xk | > λ)

Xk | > λ) Xk | > λ)

k 1

Xk | > λ) ,

2

Exercise 3.3 (Khinchin inequality) Suppose 1 , . . . , n are iid Rademacher rvs. Let a1 , . . . , an be real constants. For p ≥ 1 it holds that n 2 1/2 n n 2 1/2 p 1/p (11) Ap ≤ E | 1 ak k | ≤ Bp 1 ak 1 ak √ for some constants Ar and Br . Establish this for p = 1, with A1 = 1/ 3 and B1 = 1. [Hint. Use Littlewood’s inequality with r, s, t equal to 4, 2, 1.] Exercise 3.4 Let X1 , . . . , Xn be independent with 0 means, and independent of the iid Rademacher rvs 1 , . . . , n . Let φ be and convex on R. Then n n n (12) Eφ(| 1 k Xk |/2) ≤ Eφ(| 1 Xk |) ≤ Eφ(2| 1 k Xk |) . [Hint. The left side is an average of terms like Eφ(| n1 ek (Xk − EXk )|/2), for independent Xk ∼ = Xk and with each ek equal to ±1.]

214


Maximal Inequalities of Another Ilk Discussion 3.1 (Weak negligibility) Let Yn1 , . . . , Ynn be independent with dfs Fn1 , . . . , Fnn . Let θ > 0 be given. Define xθn by requiring [−xθn , xθn ] to ¯n ≡ n Fnk /n assigns be the smallest closed and symmetric interval to which F 1 n probability at least 1 − θ/n. Let P¯n (x) ≡ n1 1 P (|Ynk | > x) denote the average tail probability, and then let Kn denote the qf of the df 1 − P¯n (·). Note the quantile relationship xθn = K n (1 − θ/n). For any > 0, let pnk ≡ P (|Ynk | > ). Now, the maximum Mn ≡ [max1≤k≤n |Ynk | ] satisfies n n n (13) 1 − exp(− 1 pnk ) ≤ P (Mn > ) = 1 − 1 (1 − pnk ) ≤ 1 pnk . [The equality is simple, and the bounds follow from it using first 1 − x ≤ exp(−x) and then ∩n1 Ack = [∪n1 Ak ]c .] This gives the standard result that n (14) if and only if nP¯n ( ) = 1 pnk → 0 for all > 0. Mn →p 0 Since K n (1 − θ/n) = inf{x : 1 − P¯n (x) ≥ 1 − θ/n} = inf{x : P¯n (x) ≤ θ/n}, we have (15)

nP¯n ( ) ≤ θ

K n (1 − θ/n) ≤ .

if and only if

Suppose 0 < ≤ 1 and θ > 0 are fixed, and that we are considering all n exceeding some n,θ . Thus conclusions (14) and (15) give (the seemingly new emphasis) (16)

Mn →p 0

xθn = K n (1−θ/n) → 0 for all θ > 0. 2

if and only if

Discussion 3.2 (Weak negligibility in the LLN context) Let νn > 0 be constants. Applying the previous paragraph to the rvs |Ynk |/n νn , whose θ/nth quantile is xθn /n νn , gives the equivalencies (17)

Mn ≡ maxk |Ynk |/n νn →p 0 ,

(18)

xθn /n νn → 0 n 1 P (|Ynk |/n νn > ) → 0

(19)

for all θ > 0 , for all 0 < ≤ 1 .

Useful choices for νn might be the truncated absolute moment [ |y|≤x1n ] |y| dF¯n (y) and the Winsorized absolute moment ν1n + x1n P¯n (x1n ). (Here x1n equals xθn with θ = 1.) 2 Inequality 3.6 (Daniels’ equality) With high probability there is a linear bound on the uniform empirical df Gn . Thus, for each 0 < λ < 1, (20)

Proof. (a) (b)

(c)

P (Gn (t) ≤ t/λ for 0 ≤ t ≤ 1) = P (ξn:k ≥ λk/n for 1 ≤ k ≤ n) = 1 − λ . Now, (ξn:1 , . . . ξn:n ) has joint density n! on its domain. Thus P (Gn (t) ≤ t/λ for 0 ≤ t ≤ 1) = P (ξn:k ≥ λk/n for 1 ≤ k ≤ n) 1 tn t3 t2 = λ λ(n−1)/n · · · λ2/n λ/n n! dt1 · · · dtn = ··· = n = n! tn! −

λtn−1 n!

1 = 1−λ. λ

2

4.

THE CLASSICAL LAWS OF LARGE NUMBERS, LLNS

215

4

The Classical Laws of Large Numbers, LLNs

It is now time to present versions of the laws of large numbers under minimal hypotheses. The weak law of large numbers (WLLN) will establish →p , while the ¯n. strong law of large numbers (SLLN) will establish →a.e. of a sample average X Theorem 4.1 (WLLN; Feller) Let Xn1 , . . . , Xnn be iid with df F and qf K, ¯ n ≡ (Xn1 + · · · + Xnn )/n. The following are equivalent: for each n. Let X (1)

¯ n − µn →p 0 X

for some constants µn .

(2)

τ (x) ≡ xP (|X| > x) → 0

as x → ∞ .

(3)

t{|F+−1 (t)| + |F −1 (1 − t)|} = t{|K+ (t)| + |K(1 − t)|} → 0

(4)

Mn ≡ [ n1 max |Xnk | ] →p 0.

as t → 0 .

1≤k≤n

1−1/n K(t) dt When (1) holds, possible choices include µn ≡ [−n,n] x dF (x), νn ≡ 1/n ¯ and median(Xn ). If E|X| < ∞, then (1) holds and µn = µ ≡ EX works. Theorem 4.2 (SLLN; Kolmogorov)

Suppose X, X1 , X2 , . . . are iid. Then:

(5)

E|X| < ∞

implies

¯ n →a.s. µ ≡ EX. X

¯ n →L1 EX.] [Moreover, X

(6)

E|X| = ∞

implies

¯ n | =a.s. ∞. limn→∞ |X

(7)

E|X| < ∞

if and only if

¯ n | < ∞ a.s. lim |X

(8)

E|X| < ∞

if and only if

Mn ≡ [ n1 max |Xk | ] →a.s. 0. 1≤k≤n

Conditions (4) and (8) show the sense in which these LLNs are tied to the size of the maximal summand. This is an important theme, and the reader should should not lose sight of it. We now give a symmetric version of condition (4). Theorem 4.3 (The maximal summand) Let Xn1 , . . . , Xnn , n ≥ 1, be iid row independent rvs with df F . We then let Xnk ≡ Xnk −Xnk denote their symmetrized versions. Fix r > 0. [Most important is r = 1.] Then (9) (10)

τ (x) ≡ xr P (|X| > x) → 0 iff τ s (x) ≡ xr P (|X s | > x) → 0 iff max |Xnk − a|/n1/r →p 0 for each real a iff

1≤k≤n

s max |Xnk |/n1/r →p 0.

1≤k≤n

Proof. We first consider the SLLN. Let Yn ≡ Xn × 1[ |Xn | 0 we have (e)

pnN ≡ P (maxn≤m≤N |Zm − Zn | ≥ ) m = P (maxn≤m≤N | i=n+1 [(Yi − µi )/i] | ≥ ) → 0.

Then Kolmogorov’s inequality yields (e) via N N pnN ≤ −2 n+1 Var[(Yi − µi )/i] ≤ −2 n+1 σi2 /i2 2 2 (f) ≤ −2 ∞ for all N n+1 σi /i →0

as n → ∞,

provided that ∞ ∞ 2 2 2 (g) n=1 σn /n = n=1 Var[Yn ]/n < ∞. Now, this last is seen to be true via the Kolmogorov type argument ∞ ∞ ∞ 2 2 2 2 2 1 Var[Yn − µn ]/n ≤ 1 EYn /n = 1 [|x| 0.

That is, lim |Sn |/n ≥ C/2 a.s. for all C. That is, lim |Sn |/n = ∞ a.s.

218

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES ¯ n →L1 µ in (5).] Thus (5)–(7) hold. [Apply Vitali with exercise 4.16 below for X

Consider (8). Suppose Mn →a.s. 0. Then a.s. for all n ≥ (some nω ) we have (o)

[ max1≤k≤n |Xk | ] /n < ,

and hence

|Xn |/n < .

We merely repeat this last statement, writing (p)

An ≡ [ |Xn |/n ≥ ]

satisfies

P (An i.o.) = P (|Xn |/n ≥ i.o.) = 0.

Thus inequality 10.2.1 (by applying iid, and then the second Borel–Cantelli) gives ∞ ∞ (q) E|X|/ = E|X/ | ≤ n=0 P (|X/ | ≥ n) = n=0 P (|Xn |/n ≥ ) < ∞. Conversely, suppose E|X| < ∞. Then Xn /n →a.s. µ by the SLLN. Since Xn Sn − nµ n − 1 Sn−1 − (n − 1)µ µ (r) = − + →a.s. 0 − 1 · 0 + 0 = 0, n n n n−1 n we have a.s. that (s)

|Xn |/n ≤

for all n ≥ (some nω ).

Thus for all n exceeding some even larger nω we have 1 |Xk | k |Xk | (11) max ≤ max · ≤ max |Xk | ∨ max 1≤k≤n n 1≤k≤n n k≥nω k n 1≤k≤nω (t)

≤ n−1 [a fixed number depending on ω] + ≤ 2

|Xk | k

using (s),

where we will have to increase the specification on nω for (t). Thus Mn →a.s. 0. From (7.4.13) we see (note figure 4.1) that 1 −1 ∞ (12) P (|X| > x) dx < ∞ iff |F (t)| dt < ∞ . 2 E|X| < ∞ iff 0 0 Suppose X1 , . . . , Xn are independent, with Xi ∼ = (0, σi2 ). Then n ∼ (0, n σ2 ), while X ¯ n ≡ Sn /n ∼ Sn ≡ X1 + · · · + Xn = = (0, 1 σi2 /n2 ). 1 i

Remark 4.1

Chebyshev’s inequality and Kolmogorov’s inequality give n (a) P (|Sn | ≥ λ) ≤ Var[Sn ]/λ2 = i=1 σi2 /λ2 for all λ > 0, (13) (b) P (max1≤k≤n |Sk | ≥ λ) ≤ Var[Sn ]/λ2 for all λ > 0. ¯ n →p µ, by Chebyshev’s For X1 , X2 , . . . iid (µ, σ2 ), the inequality (13)(a) gives X ¯ inequality. But the WLLN conclusion Xn →p µ should not require the variance σ2 to be finite, as this cheap proof based on (13)(a) requires. Indeed, Khintchine’s WLLN of theorem 10.2.1 didn’t. Exercise 4.8 below outlines one very cheap proof of the SLLN using “only” the Borel–Cantelli lemma, and exercise 4.9 outlines a slightly improved version that also uses Kolmogorov’s inequality. Kolmogorov’s proof of the full SLLN made the key step of incorporating truncation. Exercise 4.10 describes an elementary way to avoid use of Kronecker’s lemma. 2

4.


219

Proof. Consider the WLLN. Suppose (2) holds. Define Ynk ≡ Xnk × 1[|Xnk |≤n] and µn ≡ EYnk = [−n,n] x dF (x). Now, ¯ n − µn | ≥ P (|X

√

¯ n − EY¯n | ≥

) = P (|X

(14)

≤ P (|Y¯n − EY¯n | ≥

(15)

≤

(a)

1 = − n

1

√

) +

n

1 P (Ynk

Var[Y¯n ]+nP (|X| > n) ≤ [0,n]

1 n

√

)

= Xk )

2 E Yn1 + τ (n) (Truncation inequality)

x2 dP (|X| > x) + τ (n)

n+ = −1 −n−1 x2 P (|X| > x)0− + n−1 [0,n] P (|X| > x) 2x dx + τ (n) using integration by parts = 1 [−τ (n) + 0 + 2 n−1 [0,n] τ (x) dx] + τ (n) (16)

≤

2 n

τ (x) dx [0,n]

for 0 < ≤ 1. Note that τ (x) ≤ x, and choose M > 0 so large that τ (x) < 2 /4 for x > M . Applying these to (16) gives n M n 2 2 2 M2 (b) n 0 τ (x) dx ≤ n { 0 τ (x) dx + M 4 dx} ≤ n + 2 ≤

for all n ≥ 2M 2 / 2 . Combining (16) and (b), it follows that √ ¯ n − µn | ≥ ) ≤

(c) P (|X for n ≥ (some N ) . ¯ n −µn →p 0. We also have X ¯ n −median(X ¯n ) →p 0, since the symmetrization Thus X inequality 10.3.2 gives (d)

¯ n )| ≥ ) ≤ 4P (|X ¯ n − µn | ≥ /2) → 0. ¯ n − median(X P (|X

[The acceptability of the third exhibited choice for the centering constant is left to exercise 4.1 below.] In any case, we have shown that (2) implies (1). The equivalence of (2) and (3) follows as in the proof of theorem 7.8.1. Note also figure 4.1, bearing in mind that (a ∨ b) ≤ a + b ≤ 2(a ∨ b). Consider the equivalence of (4) and (2). We know from (10.3.14) that Mn →p 0 if and only if nP (|X| > n) → 0, that is, if and only if τ ( n) = nP (|X| > n) → 0 for all > 0. Thus Mn →p 0 if and only if τ (x) → 0 as x → ∞. We still neeed to show that (1) implies (2), but we’ll wait a paragraph for this. Consider next theorem 4.3. We need only provide a proof with r = 1, as we may just replace |X| by |Y | ≡ |X|r , after raising |X| to the power r. Now, τX (x) → 0 implies τ s (x) → 0 by the right-hand side of inequality 10.3.2 with a = 0, while the left-hand side then gives τX−med (x) → 0. The equivalence of (4) and (2) then gives max |Xnk − med|/n →p 0, which trivially gives max |Xnk | →p 0, which gives τX (x) → 0 by the equivalence of (4) and (2). We are done with theorem 4.3.

220


Finally, we prove that (1) implies (2). Suppose that there exists some constant ¯ n − µn →p 0. Let S s = Sn − S , where S ≡ µn such that Sn /n − µn = X n n n ∼ Xn1 + · · · + Xnn with Xnk = Xnk and with Xnk ’s and Xnk ’s independent. Thus s Sns /n →p 0. Then Mn ≡ max1≤k≤n |Xnk |/n →p 0 by the (10.3.4) Lévy inequality. Thus τ (x) → 0 by theorem 4.3. 2 Exercise 4.1 Verify that the choice νn ≡ WLLN as a centering constant in (1).

1−1/n 1/n

K(t) dt also works in the

We have just seen that good inequalities lead to good theorems! In sections 10.9 and 12.11 we will add to our growing collection of good inequalities. Some will be used in this text, and some will not. But the author thinks it important to illustrate these possibilities. ¯n| = ∞ Exercise 4.2 When E|X| = ∞, the SLLN above showed that limn |X a.s. Show the following stronger result. If X1 , X2 , . . . are iid with E|X| = ∞, then ¯ n − cn | =a.s. ∞ limn→∞ |X for every sequence of constants cn . Exercise 4.3 (Erickson)

If EX − < ∞ but EX + = ∞, then lim Sn /n =a.s. +∞.

Exercise 4.4 (Marcinkiewicz–Zygmund) Let X1 , X2 , . . . be iid. Let 0 < r < 2. Establish the equivalence (17)

E|X|r < ∞

if and only if

n

n 1 (Xk − c) →a.s. 0 for some c. 1/r k=1

If so, then c = EX when 1 ≤ r < 2, while c is arbitrary (so c = 0 works) when 0 < r < 1. [Hint. Truncate via Yn ≡ Xn × 1[|Xn |≤n1/r ] .] Let X1 , X2 , . . . be iid with E|X| = ∞. If an /n ↑, then ∞ = 0 a.s., < ∞, (18) lim |Sn |/an = according as P (|Xn | ≥ an ) = n=1 = ∞ a.s., = ∞. ∞ [Note that P (|Xn | ≥ an i.o.) equals 0 or 1 according as 1 P (|Xn | ≥ an ) is finite or infinite.] Exercise 4.5 (Feller)

Exercise 4.6

Clarify the overlap between (17) and (18).

Exercise 4.7 (Random sample size) (a) Suppose X1 , X2 , . . . are iid and (0, σ2 ). Let Nn ≥ 0 be any positive integer-valued rv satisfying Nn /n →p c ∈ (0, ∞). Then (19) SNn /Nn − µn →p 0, for µn ≡ [−n,n] x dF (x). (b) Suppose X1 , X2 , . . . are iid and µ ≡ EX is finite. Let Nn ≥ 0 be any positive integer-valued rv satisfying Nn /n →a.s. c ∈ (0, ∞). Then (20)

SNn /Nn →a.s. µ.

4.


221

Exercise 4.8 Let X1 , X2 , . . . be independent rvs with means 0 and suppose that all EXk4 ≤ (some M ) < ∞. ¯ n ≡ Sn /n ≡ (X1 + · · · + Xn )/n satisfies (a) (Cantelli’s inequality) Verify that X P (|Sn | ≥ λ) ≤ 3M n2 /λ4

for all λ > 0.

¯ n →a.s. 0. (b) (A very weak SLLN) Show that under these circumstances X Exercise 4.9 (A weak SLLN) If X1 , X2 , . . . are independent and Xk ∼ = (0, σk2 ) 2 ¯ with σk ≤ (some M ) < ∞ for all k, then Xn →a.s. 0 as n → ∞. Prove this by ¯ n2 | = |Sn2 /n2 | > 0 i.o.) = 0, and showing P (|Sn | ≥ n ) ≤ M/(n 2 ) so that P (|X then showing that the “block maximum” ∆n ≡ maxn2 i.o.) = 0. Exercise 4.10 (Alternative proof of the SLLN) Apply either the H´ ajek–Rényi inequality (inequality 10.11.3) or the monotone inequality (inequality 10.11.1) to replace use of the Kronecker lemma in the SLLN proof. Exercise 4.11 (St. Petersburg paradox) Let X1 , X2 , . . . be iid rvs for which P (X = 2m ) = 1/2m for m ≥ 1. Show that (Sn − bn )/an →p 0 for bn ≡ n log2 n and an ≡ n log2 n (where log2 · is used here to denote logarithm to the base 2). Exercise 4.12 (Spitzer) (21)

EX = 0

Show that ∞ if and only if n=1 P (|Sn | ≥ n )/n < ∞ for all > 0.

Exercise 4.13 If X1 , X2 , . . . are iid Exponential(1), then lim Xn / log n = 1 a.s. and Xn:n / log n → 1 a.s. Exercise 4.14

√ If X1 , X2 , . . . are iid N(0, 1), then Xn:n / 2 log n →p 1.

Exercise 4.15 (a) Does the WLLN hold for the Cauchy distribution? (b) Does the WLLN hold when P (|X| > x) = 2/[x log x] for x ≥ e, X symmetric? (c) Make up one more example of each of these two types. Exercise 4.16 (Uniform integrability of sample averages) Let X1 , X2 , . . . be iid, ¯ n ≡ (X1 +· · ·+Xn )/n. Then the rvs {Xn : n ≥ 1} are uniformly integrable and let X ¯ n : n ≥ 1} are uniformly integrable. if and only if the rvs {X Exercise 4.17 (a) Let row independent rvs Xn1 , . . . , Xnn be iid with the df F (·) (having mean µ) for all n ≥ 1. If Mn ≡ [max1≤k≤n |Xnk |/n] →p 0 (and using the fact that EMn ≤ E|X|), show that (22)

EMn = E[max1≤k≤n |Xnk |/n] → 0.

(b) For iid X1 , X2 , . . . show that E|X| < ∞ if and only if Mn →L1 0; see theorem 4.2.

222


Generalizations of the LLNs Our results allow simple generalizations of both the WLLN and SLLN. Theorem 4.4 (General WLLN and SLLN) Let X1 , X2 , . . . be independent. Then n 2 2 n (23) implies 1 σk /bn → 0 1 (Xk − µk )/bn →p 0, ∞ 2 2 n (24) implies 1 σk /bk < ∞ with bn ∞ 1 (Xk − µk )/bn →a.s. 0. Proof. The first claim is immediate from Chebyshev’s inequality. Also, (f) in the SLLN proof shows that ∞ 2 2 n (a) implies 1 σk /bk < ∞ 1 (Xk − µk )/bk →a.s. (some rv S). n Then Kronecker’s lemma gives 1 (Xk − µk )/bn →a.s. 0. 2

5.

APPLICATIONS OF THE LAWS OF LARGE NUMBERS

223

5

Applications of the Laws of Large Numbers

Let X1 , X2 , . . . be iid F . Let Fn denote the empirical df of X1 , . . . , Xn , given by (1)

Fn (x) ≡ Fn (x, ω) ≡

1 1 1(−∞,x] (Xk (ω)) = 1[Xk ≤x] n n n

n

k=1

k=1

Theorem 5.1 (Glivenko–Cantelli) (2)

Fn − F ≡

sup

−∞<x 0 is arbitrary, we have shown that sup0≤t≤1 |Gn (t) − t| →a.s. 0. That is, (4)

Fn − F = Gn (F ) − F ≤ Gn − I →a.s. 0 ,

as claimed.

2

Example 5.1 (Weierstrass approximation theorem) If f is continuous on [0, 1], then there exist polynomials Bn such that Bn − f = sup0≤t≤1 |Bn (t) − f (t)| → 0 as n → ∞.

224 Proof. (5)

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES (Bernstein) Define the Bernoulli polynomials n k n k Bn (t) = f t (1 − t)n−k for 0 ≤ t ≤ 1 n k k=0

(a)

= Ef (T /n)

where T ∼ = Binomial(n, t).

Since f is continuous, f is bounded by some M , and f is uniformly continuous on [0, 1] having |f (x) − f (y)| < whenever |x − y| < δ . Then |f (t) − Bn (t)| = nk=0 [f (t) − f (k/n)] nk tk (1 − t)n−k ≤ | {k:|k/n−t| 1. We are able to conclude that a.e. number in [0, 1] is normal with respect to Lebesgue measure λ. [Comment: 13 = 0.010101 . . . in base 2 is normal in base 2, but 13 = 0.1000 . . . in base 3 is not normal in base 3.] [This was a historically important example, which spurred much of the original development.] Proof. (a)

Let (Ω, A, P ) = ([0, 1], B ∩ [0, 1], λ). Let ∞ ω = n=1 βn (ω)/dn define rvs β1 , β2 , . . . .

Note that the βn ’s are iid discrete uniform on 0, 1, . . . , d − 1. Thus, letting ηnk = 0 or 1 according as βn = k or βn = k, we have n (b) λ(Ad,k ) ≡ λ({ω : n−1 j=1 ηjk → 1/d}) = 1 by the SLLN. Thus Ad ≡ ∩d−1 k=0 Ad,k has λ(Ad ) = 1; that is, a.e. ω in [0, 1] is normal to base d. Then trivially, A ≡ ∩∞ d=1 Ad has λ(A) = 1. And so, a.e. ω in [0, 1] is normal. 2

5.

APPLICATIONS OF THE LAWS OF LARGE NUMBERS

225

Example 5.3 (SLLN for random sample size) Let Nn be positive-integer valued rvs for which Nn /n →a.s. c ∈ (0, ∞), and let X1 , X2 , . . . be iid with mean µ. (a) Then (6)

SNn /n →a.e. µc

as n → ∞.

(b) If X1 , X2 , . . . are iid Bernoulli(p) and Nn (ω) ≡ min{k : Sk = n}, then the waiting times Nn satisfy Nn /n →a.s. 1/p. Proof. (a) Now, Sn /n →a.s µ by the SLLN, and thus Nn →a.s. ∞ implies SNn /Nn →a.s. µ. Thus SNn /n = (SNn /Nn ) (Nn /n) →a.s. µc, using Nn →a.s. ∞ by c > 0. (b) We also have 1 = SNn /n = (SNn /Nn ) (Nn /n),

so

Nn /n = 1/(SNn /Nn ) →a.s. 1/p,

completing the proof. Note that we could also view Nn as the sum of n iid Geometric(p) rvs, and then apply the SLLN. 2 Exercise 5.1 (Monte Carlo estimation) Let h : [0, 1] → [0, 1] be continuous. (i) Let Xk ≡ 1[h(ξk )≥Θk ] , where ξ1 , ξ2 , . . . Θ1 , Θ2 , . . . are iid Uniform(0, 1) rvs. Show that this sample average is a strongly consistent estimator of the integral; that is, ¯ n →a.s. 1 h(t) dt. show that X 0 1 (ii) Let Yk ≡ h(ξk ). Show that Y¯n →a.s. 0 h(t) dt. ¯ n ] and Var[Y¯n ]. (iii) Evaluate Var[X

226

6


General Moment Estimation

Notation 6.1 (General case on [0, ∞)) Let Yn1 , . . . , Ynn denote row indepenn dent rvs having dfs Fn1 , . . . , Fnn . Let P¯n (x) ≡ 1 P (|Ynk | > x)/n. Let the qf K n (·) denote the inverse of the df 1 − P¯n (x). Let xθn denote the θ/nth quantile of the df 1 − P¯n . [Thus x1n ≡ K n (1 − 1/n) is the upper (1/n)-quantile of 1 − P¯n , while [−x1n , x1n ] is the shortest interval containing at least 1 − 1/n of the probability.] r r + · · · + Xnn )/n. Let the Let Xnk ≡ |Ynk | for 1 ≤ k ≤ n. Fix r > 0. Consider (Xn1 rth partial absolute moment, or truncated absolute moment, Un (·) be defined by (1) |y|r dF¯n (y) for each x > 0 . Un (x) ≡ [ |y|≤x]

Let uθn ≡ Un (xθn ). The heaviness of the tails of F¯n (·) is measured with Rn (·) defined by (2) Rn (x) ≡ xr P¯n (x)/ [ |y|≤x] |y|r dF¯n (y) for each x > 0. 2 Theorem 6.1 (Consistency of the absolute moments) (A) Conditions (4)–(7) are equivalent, and imply (3), where (3)

Xnr / u1n →p 1,

where u1n = Un (x1n ) ,

(4)

xrθn /n uθn → 0

for each 0 < θ ≤ 1 ,

(5)

for each 0 < θ ≤ 1 ,

(6)

xrθn /n u1n → 0 n r 1 P (Xnk /n u1n > ) → 0

(7)

[ max1≤k≤n

1 n

for each 0 < ≤ 1 .

r Xnk ] / u1n →p 0 .

(B) Fix θ > 0. Then Rn (xθn −) → 0 implies xrθn /n uθn → 0, as in (4). (C) The Winsorized moment u ˜1n ≡ u1n + xr1n P¯n (x1n ) satisfies u ˜1n /u1n → 1 when condition (5) holds. Corollary 1 Suppose Ynk ≡ Xnk × 1[−n,n] (Xnk ) satisfies ¯ n →p 1. and EY¯n → 1 and Var[Y¯n ] → 0. Then X Exercise 6.1

n 1

P (Ynk = Xnk ) → 0,

Prove the previous corollary. Can you prove a converse?

Proof. It suffices to prove this theorem for r = 1 (reduce the problem by transforming from X to X r ). Fix θ > 0. Let µθn ≡ [ |y|≤xθn ] |y| dF¯n (y) = Un (xθn ). Also, (5)–(7) were shown equivalent in (10.3.17)–(10.3.19). Now, (5) trivially implies (4) (since uθn ≥ u1n ), and the converse holds, since (a)

xθn xθn (xθn /n µ1n ) ≥ = . n µθn n [µ1n + xθn /n] 1 + (xθn /n µ1n )

6.

GENERAL MOMENT ESTIMATION

227

Consider (5) implies (3) next. Define Znk to equal Xnk if Xnk ≤ xθn and to equal 0 otherwise. Note that EZ¯n = µθn . Fix > 0. Then for some fixed tiny θ ≡ θ , we have 1≥ (b)

µ1n + n1 xθn − µ1n µθn − µ1n xθn µ1n ≥1− ≥1− =1− µθn µθn µθn n µθn

≥1−

xθn ≥ 1− . n µ1n

Also, the truncation inequality (10.4.15) (that is, the steps (c) and (d) below) gives ¯ n − µθn | > µθn ) ≤ P (|Z¯n − µθn | > µθn ) + n P (Xnk = Znk ) P (|X (c) 1 n 2 ≤ 1 E(Znk (d) )/( 2 µ2θn n2 ) + nP¯n (xθn ) = [ |y|≤xθn ] y 2 dF¯n (y)/( 2 n µ2θn ) + nP¯n (xθn )

|y| dF¯ (y)/( 2 n µ2θn ) + θ

(e)

≤ xθn

(f)

= −2 {xθn /nµθn } + θ ≤ −2 {xθn /nµ1n } + θ ≤ 2

[ |y|≤xθn ]

for all n exceeding some suffficiently large n (by (5)). Thus for n large enough, | Xnr − µ1n | µθn Xnr − µθn µ1n 1 (g) = · +1− ≤ [ + ] µ1n µ1n µθn µθn 1 −

with probability at least 1 − 2 . Thus (3) holds. Now (C) is trivial, since nP¯n (x1n ) ≤ 1. Observe that (h)

P¯n (x1n ) ≤ 1/n = θ −1 (θ/n) ≤ θ−1 P¯n (xθn −) ,

and hence (B) holds via (i)

xθn 1 xθn P¯n (xθn −) 1 ≤ = Rn (xθn −) → 0 . n µθn θ [ |y|≤xθn ] |y| dF¯n (y) θ

2

Notation 6.2 (Absolute moments in the iid case) Let Y and all rvs Ynk now have common df F (·) ≡ FY (·) and qf K(·) ≡ KY (·). [Let X ≡ |Y |. The rvs Xnk ≡ |Ynk | may be represented as KX (ξnk ) for row-independent Uniform(0, 1) rvs r ξn1 , . . . , ξnn .] Then P (·) ≡ 1 − FX (·), and define m(·) ¯ ≡ KX (1 − ·). The rth partial ¯ absolute moment, or truncated absolute moment, M(·) and U (·) are defined by r ¯ (t) ≡ M KX (s) ds on [0, 1], U (x) ≡ [0,x] y r dFX (y) on [0, ∞). [0,1−t) ¯ (θ/n) and uθn ≡ U (xθn ), where xθn ≡ KX (1 − θ/n) is defined to be Let m ¯ θn ≡ M the (1 − θ/n)th quantile. The Winsorized absolute moment ˜¯ 1n ≡ m ¯ 1n + n1 m(1/n) ¯ equals u˜1n ≡ u1n + x1n P (x1n ) , m even though the lead terms may not be equal.

2

228


Theorem 6.2 (Consistency of absolute moments in the iid case) rvs the conditions (3)–(15) are equivalent, where

For iid

(8)

Xnr / νn →p 1

for some constants νn > 0 with n νn ,

(9)

(a) Xnr / m ¯ 1n →p 1 ,

(b) Xnr / u ˜ 1n →p 1 ,

(10)

R(x) ≡ xr P (x)/U (x) = xr P (X > x)/ [0,x] y r dFX (y) → 0 or R(x−) → 0,

(11)

¯ (t) = t K r (1 − t)/ r¯(t) ≡ t m(t)/ ¯ M X

(12)

r¯(an ) → 0

for any one an 0 with lim(an+1 /an ) < ∞ ,

(13)

n P (X r > n m ¯ 1n ) → 0

for all > 0 ,

(14)

[ max1≤k≤n

1 n

r Xnk ]/m ¯ 1n →p 0 ,

(15)

[ max1≤k≤n

1 n

r Xnk ] / Xnr →p 0 .

[0,1−t)

(c) Xnr / u1n →p 1 ,

r KX (s) ds → 0

as t → 0 ,

˜¯ 1n /νn → 1 under (8). Of course, m ¯ 1n /νn → 1, u1n /νn → 1, and u ˜1n /νn = m Proof. Without loss of generality, set r = 1. Suppose (8). Let c > 1. Note ¯ n /νn ≥ c) → 0 first that P (maxk Xnk < cnνn )= P (all Xnk < cnνn ) → 1; else P (X is true, and (8) fails. But the conclusion P (maxk Xnk < cnνn ) → 1 implies by (10.3.19) that we have (the general background result) that for all c > 1, n (a) 1 P (Xnk ≥ c nνn ) → 0 {and P ([maxk Xnk /n νn ] ≥ c) → 0} . Consider (8) implies (10). Let > 0 be tiny. Let c > 1. Truncate the Xnk ’s via (b) Znk ≡ Xnk 1[Xnk νn ) ≤ Var[Z¯n ]/( νn )2 + nP (c n νn −) P (|X (c)

≤ (c/ 2 ) µn /νn + nP (c n νn −) ,

where nP (c n νn −) → 0 by (a). We next show that nP (c n νn −) → 0 implies that (d)

lim µn /νn > 0

(equivalently, lim νn /µn < ∞).

Suppose µn /νn → 0 were true on some subsequence n . Then (c) would give (e)

¯ n /νn − µn /νn = X ¯ n /νn − o(1) →p 0 X

but this is a contradiction. Thus (d) holds. Thus

(or

¯ n /νn →p 0) ; X

6. (f)


229

c n νn P (c n νn −) lim ≤ c [ lim (νn /µn )] [ lim nP (c n νn −)] [0, c n νn ) y dFX (y) = O(1) o(1) → 0 .

So, (10) holds on the subsequence xn = cnνn . Thus [as n P (Xn,n+1 /n νn ≥ c) → 0 follows from equation (a)], (g)

¯ ¯ n+1 X n νn Xn Xn,n+1 νn = + {1 + 2 + op (1)} ≤ 1 + op (1) = νn+1 n + 1 νn+1 νn n νn νn+1

shows that lim νn+1 /νn ≤ 3. Thus (since nνn ) sup

(h)

[c n νn ≤x≤c (n+1) νn+1 ]

xP (x−) ≤ y dFX (y) [0, x)

n+1 n

c νn+1 c n νn P (c n νn −) →0 c νn y dFX (y) [0, c n νn )

by (f), so that (10) holds. In fact, the stronger R(x−) → 0 was just verified. We appeal also to theorem 6.1(A)(B). It has now been shown that (8) implies (10) with R(·−), which implies (4), which implies (3), which implies (9)(c), which implies (8). Purely geometric arguments in section 7.7 show that (10), (11), and (12) are equivalent. Thus (3), (8), (9)(c), (10) with R(·−), (11), and (12) have been added to the list (4)–(7). [And (9)(b) then follows from (11).] We next show that (11) implies (9)(a); and since (9)(a) trivially implies (8), this will add (9)(a) to the list. Define (i)

Znk ≡ KX (ξnk ) 1[ξnk 0. Then for a fixed tiny θ ≡ θ , the truncation inequality (10.4.15) gives (j)

¯n − m P (|X ¯ θn | > m ¯ θn ) ≤

n 1

2 EZnk /( n m ¯ θn)2 + n(θ/n)

(k)

≤

≤

2 KX (s) ds [0,1−θ/n) 2 2

nm ¯ θn

+θ

(θ/n) KX (1 − θ/n) 1 (θ/n) m(θ/n) ¯ +θ = 2 + θ < 2

2 ¯ θ m ¯ θn θ

M (θ/n)

for n sufficiently large, by (11). In like fashion (l)

m ¯ 1n (1 − θ) (θ/n) KX (1 − θ/n) [0,1−1/n) KX (s) ds 1≥ = ≥1− ≥1−

m ¯ θn θ [0,1−θ/n) KX (s) ds [0,1−θ/n) KX (s) ds

for all n sufficiently large, by (11). Thus, for n ≥ (some n ) we have

230 (m)

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES ¯n − m m ¯ θn |X ¯ 1n | = m ¯ 1n m ¯ 1n

¯ X m ¯ 1n 1 ¯ θn 2

n−m + 1 − ≤ [ + ] ≤ m ¯ θn m ¯ θn 1 −

1−

¯ n /m ¯ 1n →p 1, giving (9)(a). with probability at least 1 − 2 . Thus X Next, (11) implies (13). Let > 0. Observe that n n n (n) ¯ θn) ≤ 1 P (Znk > 2 nm ¯ θn)+ 1 P (Xnk = Znk ) < 2

1 P (Xnk > 2 nm for all large n, as in bounding ¯ 1n to replace the right side of (j). Then (l) allows m m ¯ θn in (n), which implies n1 P (Xnk > 3 nm ¯ 1n ) < 2 for all large n. This gives condition (13). Consider (13) implies (12) with t = an = 1/n. Fix 0 < θ ≤ 1. State (13) as (o)

P (KX (1 − ξ)/(n m ¯ 1n ) > ) < θ/n

This implies KX (1 − θ/n)/(n m ¯ 1n ) ≤ for all all n ≥ nθ ]; that is, (12) holds with t = an Then (10.3.17) and (10.3.19) show that (13) Finally, we add (15) to the list. Now (14) (p)

for all n ≥ (some nθ ) . n ≥ nθ [entailing r¯(1 − 1/n) ≤ for = 1/n. Thus (13) has been added. is equivalent to (14). implies (15), via

¯ n = {[max Xnk /n ]/ m ¯n} [max Xnk /n ]/ X ¯ 1n } × {m ¯ 1n /X = op (1) [1/(1 + op (1))] →p 0 ;

using (14) implies (9)(a). Now, suppose (15) holds. We will verify (12). Daniels’ inequality 10.3.6 shows that the event An ≡ [ ξn:k ≥ (k/n)/λ for all 1 ≤ k ≤ n] has P (An ) ≥ 1 − for all n ≥ (some n ), provided λ is specified large enough (with order statistics ξn:1 ≤ · · · ≤ ξn:n ). Thus (letting 1n denote the indicator of An ) n ¯ n × 1n = 1n × n m(ξ X (q) ¯ n] k=1 ¯ n:k )/n ≤ λ [ k=1 m((k/n)/λ)/λ ≤ λ{[m(1/nλ)/nλ] ¯ + (1/nλ,1/λ] m(t) ¯ dt} ¯ ≤ 3λ (1/2nλ,1/λ] m(t) ¯ dt ≤ 3λM(1/2nλ) , with probability exceeding 1 − . This gives (r)

¯ (1/2nλ)] ≤ 3λ [maxk KX (1 − ξnk )]/[n X ¯n] 1n × [maxk KX (1 − ξnk )]/[nM →p 0,

and since P ([min ξnk ] ≥ 1/2nλ) = (1 − 1/2nλ)n → exp(−1/2λ) > 0, this gives (s)

¯ m(1/2nλ)/[n ¯ M(1/2nλ)] → 0,

which is (12) with an = 1/2nλ. We showed that (8) implies the stronger R(x−) → 0. And theorem 7.7.1 shows that the weaker R(x) → 0 implies (11); and this was seen above to imply (9)(a), and hence (8). Thus R(x) → 0 and R(x−) → 0 are equivalent. 2

6.


231

Notation 6.3 (Moments in the iid case) Let Yn1 , . . . , Ynn be a triangular array of row independent rvs, all with common df F and qf K. Now define the rvs + − Ynk ≡ Ynk × 1[Ynk ≥0] and Ynk ≡ −Ynk × 1[Ynk 0. Let − m− (t) ≡ |K+ (t)|r ,

M (t) ≡ m− 1n ≡

(t,1]

m(s) ds

(1/n,1]

m− (s) ds ,

m+ (t) ≡ |K + (1 − t)|r , m ≡ m− + m+ ,

where m+ 1n ≡

(1/n,1]

m+ (s) ds ,

(16) m± 1n

+ + − simultaneously denote any one of m− 1n , m1n , or m1n ± m1n ,

− Y¯rn ≡

1 n

± Y¯rn

n k=1

− r [Ynk ] ,

+ Y¯rn ≡

1 n

n k=1

+ r [Ynk ] ,

− + Y¯rn ≡ Y¯rn + Y¯rn ,

− ¯+ + − simultaneously denote any one of Y¯rn , Yrn or Y¯rn ± Y¯rn .

¯ ≡ KX (·) denote the qf of the rv X ≡ |Y |, and then let Let K(·) ¯ ¯ (t) ≡ ¯ r (1 − t) , K(s) ds = (t,1] m(s) (17) M ¯ ds, where m(t) ¯ ≡K [0,1−t) as was considered in Theorem 6.2. 2 Theorem 6.3 (Consistency of moments in the iid case) Fix an r > 0. Just some of the possible equivalent conditions are as follows: n (18) [ n1 k=1 |Ynk |r ]/m1n →p 1, where m1n ≡ M (1/n) . (19) as t → 0 . r(t) ≡ t m(t)/M (t) = t m(t)/ (t,1] m(s) ds → 0 (20)

r(an ) → 0

(21)

[max1≤k≤n

(22)

[max1≤k≤n

for any one specific an 0 having lim (an /an+1 ) < ∞ . |K(ξnk )|r ]/ m1n →p 0 n r r 1 1 k=1 |K(ξnk )| ] →p 0 . n |K(ξnk )| ] / [ n 1 n

Moreover, these are also equivalent to the list of theorem 6.2, which includes n ¯ (1/n) . (23) [ n1 k=1 |Ynk |r ]/m ¯ 1n →p 1, where m ¯ 1n ≡ M When any one of (3)–(15) and (18)–(23) holds, we also have both (24)

± [Y¯rn − m± 1n ]/m1n →p 0

(25)

m1n / m ¯ 1n → 1 .

and

Proof. Let r = 1. That (19) implies (18) is virtually the same as (i)–(m) in the proof of theorem 6.2, but now with Znk ≡ |K(ξnk )| 1(θ/n x1n ) . 2 where v¯1n ≡ [0,1−1/n) K|Y | (s) ds = V|Y | (1/n) .

(36)

Yn2 / v1n →p 1,

where v1n ≡ V (1/n) = dom(1/n,1/n) K 2 (s) ds .

(37)

Yn2 /νn →p 1 for any one specific νn > 0 for which n νn .

(38)

[maxk

1 n

2 Ynk ] / ν1n →p 0 where ν1n is any one of v1n , v¯1n , u ˜1n or u1n .

(39)

[maxk

1 n

2 Ynk ] / Yn2 →p 0 .

(40)

Dn ≡ [maxk

(41)

ñ2 →p 1. Sn2 / σ

1 n

[0,1−t)

2 K|Y | (s) ds .

for any one an 0 having lim (an /an+1) < ∞ .

|Ynk − Y¯n |2 ] / Sn2 →p 0

where Sn2 ≡ Yn2 − (Y¯n )2 .

Of course, (29) implies u1n /v1n → 1, v¯1n /v1n → 1, and u˜1n /v1n = v˜¯1n /v1n → 1. [Theorems 7.7.1 and 7.7.2 present well more than 48 other equivalencies similar to (29)–(32), all of which follow from using just geometrical considerations and Cauchy–Schwarz.]

234


Proof. We note that the equivalence of (29) through (39) (and many other equivalencies) have already been established. We were able to accomplish this by combining the lists of theorems 6.2 and 6.3, which was possible since (38) appeared on both lists. Suppose (39). Suppose E(Y 2 ) = ∞ (since the other case is trivial). Then ¯ (Yn )2 / Yn2 →p 0 by (26), which makes (40) trivial. Suppose (40). Now, 2 2 Y ¯n | ¯n max |Ynk −Y Y nk √ max √ 2 − √ 2 n Yn2 nYn nYn (a) = ; Dn = 1 − (Y¯n )2 / Yn2 1 − (Y¯n )2 / Yn2 here 0 ≤ (Y¯n )2 / Yn2 ≤ 1 always holds, and |Y¯n |/ n Yn2 →a.s. 0 always holds. Thus Dn (ω) → 0 for a.e. fixed (Yn1 (ω), Yn2 (ω), . . .) implies that the numerator of (a) goes to zero for a.e. fixed ω. Thus [max |Ynk | ]/ nYn2 converges in probability to zero, as required for (39) (by going to subsequences). That (41) is equivalent to (34) follows using proposition 6.1. 2 2 Remark 6.1 Let σ ˜|Y | (x) denote the variance of the rv Y Winsorized outside [−x, x]. 2 Let σ ˜ (t) denote the variance of the rv KY (ξ) Winsorized outside dom(t, t). Note 2 2 that σ ˜|Y ˜ 2 (1/n). Thus σ ˜|Y ˜ 2 (1/n)/v1n → 1 when (29) holds | (x1n ) = σ | (x1n )/v1n = σ with Var[Y ] = ∞; we just apply both (32) and (27). 2

7.

LAW OF THE ITERATED LOGARITHM

235

7

Law of the Iterated Logarithm

Theorem 7.1 (LIL; Hartman–Wintner; Strassen) Let X1 , X2 , . . . be iid rvs. Consider the partial sums Sn ≡ X1 + · · · + Xn . (a) If EX = 0 and σ2 ≡ Var[X] < ∞, then (1)

lim sup √ n→∞

Sn = σ a.s., 2n log log n

while

lim inf √ n→∞

Sn = −σ a.s. 2n log log n

(b) In fact, (2)

√

Sn →a.s. [−σ, σ]. 2n log log n

√ [That is, for a.e. ω the limit set of Sn / 2n log log n is exactly [−σ, σ] ]. (c) Conversely, if (3)

lim sup √ n→∞

|Sn | < ∞ a.s., 2n log log n

Theorem 7.2 (The other LIL; Chung) (4)

lim inf max

n→∞ 1≤k≤n

then

EX = 0 and σ2 < ∞.

If X1 , X2 , . . . are iid (0, σ2 ), then

|Sk | 2 log log n √ = π/2 a.s. nσ

[We state this for fun only, as it has seen little application.] Versions of both theorems are also known for cases other than iid. The classical proof of theorem 7.1 in full generality begins with truncation, and then carefully uses exponential bounds for bounded rvs. A more modern proof relies upon Skorokhod embedding of the partial sum process in Brownian motion. This general proof is outlined in the straightforward exercise 12.8.2, after embedding is introduced. But the proof below for the special case of normal rvs contains several of the techniques used in the classical proof of the general case (and in other related problems). And it is also a crucial component of the general case in exercise 12.8.2. Proposition 7.1 Let Z1 , Z2 , . . . be iid√N (0, 1) rvs. Let Sn ≡ Z1 + · · · + Zn and √ bn ≡ 2 log log n. Then lim supn→∞ Sn / n bn = 1 a.s. Proof. (5)

Let > 0. We will use the exponential bound √ exp[−(1 + )λ2 /2] ≤ P (Sn / n ≥ λ) ≤ exp[−(1 − )λ2 /2]

for all λ > λ

(for some λ ) [see Mills’ ratio exercise 7.1 below], and the Lévy maximal inequality (a)

P ( max Sk ≥ λ) ≤ 2P (Sn ≥ λ) 1≤k≤n

for all λ > 0.

Let nk ≡ [ak ] for a > 1; a sufficiently small a will be specified below. Now,

236

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES Ak ≡

nk−1 ≤m≤nk

[Sm ≥

√

m(1 + 2 )bm ]

⊂

(b) since (c)

√

max

nk−1 ≤m≤nk

Sm ≥ (1 + 2 )

√ nk−1 bnk−1 nk , nk

n is and bn is ; so that for k sufficiently large, √ nk−1 by (a) P (Ak ) ≤ 2P Snk / nk ≥ (1 + 2 ) bnk−1 nk 1 2 1−

≤ 2 exp − (1 − )(1 + 2 ) 2 log k by (5) 2 a ≤ 2 exp(−(1 + ) log k) = 2/k 1+

(d)

for a sufficiently close to 1

= (a convergent series).

Thus P (Ak i.o.) = 0 by Borel–Cantelli. Since > 0 is arbitrary, we thus have (e)

Sn lim sup √ ≤ 1 a.s. n bn n→∞

Also, (e) yields an extra result that will be useful below. Thus (f)

P (Ak i.o.) = 0

for any large positive a.

We must now show that the lim in (e) is also ≥ 1 a.s. We will still use nk ≡ [ak ], but a will be specified sufficiently large below. We write Snk = Snk−1 +(Snk −Snk−1 ), so that Snk−1 Snk − Snk−1 nk−1 bnk−1 Snk (g) = + √ √ √ nk bnk nk bnk nk−1 bnk−1 nk bnk (h)

Snk−1 Snk − Snk−1 1 ∼ √ ·1· √ + √ . a nk−1 bnk−1 nk bnk

Now, the independent events (i) have (j)

Bk ≡ [Snk −Snk−1

√ Snk − Snk−1 (1 − 2 ) nk bnk √ ≥ (1−2 ) nk bnk ] = √ ≥ √ nk − nk−1 nk − nk−1

1 nk P (Bk ) ≥ exp − (1 + )(1 − 2 )2 b2nk 2 nk − nk−1 1 2 (1 + )a ≥ exp − (1 + )(1 − 2 ) 2 log k 2 a−1 ≥ exp (−(1 − ) log k)

(k)

for a sufficiently large

= 1/k 1− = (a series with infinite sum),

by (5)

7.

LAW OF THE ITERATED LOGARITHM

237

so that P (Bk i.o.) = 1 by the second Borel–Cantelli lemma. But P (Ak i.o.) = 0 and P (Bk i.o.) = 1 means that (l)

P (Ack ∩ Bk i.o.) = 1.

Moreover, on Ack ∩ Bk we have, using (h), (i), and the symmetric version of (b), (m)

√

Snk (1 + 2 )(1 + ) √ ≥− + (1 − 2 ) ≥ (1 − 3 ) nk bnk a

for the constant a specified sufficiently large. Thus, even on the subsequence nk , since > 0 was arbitrary, (n)

lim sup √ k→∞

Snk ≥ 1 a.s. nk bnk

Combining (e) and (n) gives the proposition.

Exercise 7.1 (Mills’ ratio) (6)

λ2

2

(a) For all λ > 0

λ 1 1 1 √ exp(−λ2 /2) < P (N (0, 1) > λ) < √ exp(−λ2 /2) , + 1 2π λ 2π

which can be rewritten as (7)

1 λ φ(λ) < 1 − Φ(λ) < φ(λ) , λ2 + 1 λ

where φ and Φ denote the standard normal N (0, 1) density and df, respectively. Thus if λn ∞, we have (with δn 0) (8)

P (N (0, 1) > λn ) = exp(−λ2n (1 − δn )/2)

as n → ∞.

[This clean exponential bound for normal rvs is the key to a simple LIL proof in proposition 7.1. The classic Hartman–Wintner proof uses truncation to achieve a reasonable facsimile.] (b) (Ito–McKean) Show that (9)

√

2 2 φ(λ) < 1 − Φ(λ) < √ φ(λ) . λ2 + 4 + λ λ2 + 2 + λ

Exercise 7.2 In place of (c) in the LIL proof of proposition 7.1, use Mills’ ratio √ √ to bound P (An ) ≡ P (Sn / n ≥ (1 + 2 ) 2 log n). Use that bound directly to show √ √ that lim sup |Sn |/( n 2 log n) ≤ 1 a.s. [This “poor” result will show the value of using the “block of indices” in the definition of Ak in the proof we gave.] Exercise 7.3 Suppose arbitrary events An and Bn satisfy P (An i.o.) = 1 and P (Bnc i.o.) = 0. Show that P (An ∩ Bn i.o.) = 1.

238


Summary Suppose X, X1 , X2 , . . . are iid (µ, 1). Then: n k=1 (Xk − µ) →a.s. 0 (10) by the SLLN. n n (11)

k=1 (Xk n1/r

− µ)

→a.s. 0

for all 1 ≤ r < 2, by Marcinkiewicz–Zygmund.

n (Xk − µ) √k=1 by the LIL. →a.s. [−1, 1] 2n log log n √ Suppose we go all the way to n in the denominator. Then the classical CLT gives (12)

n (13)

k=1 (Xk

√

− µ)

n

→d N (0, 1)

by the CLT,

even though we have divergence to ±∞ for a.e. ω (by the LIL).

2

Exercise 7.4 (rth mean convergence theorem) Let X, X1 , X2 , . . . be iid, and consider the partial sums Sn ≡ X1 + · · · + Xn . Let 0 < r < 2 (and suppose EX = 0 in case 1 ≤ r < 2). The following are equivalent: (a)

E|X|r < ∞.

(b)

Sn /n1/r →a.s. 0.

(c)

E|Sn |r = o(n).

(d)

E(max1≤k≤n |Sk |r ) = o(n).

8.

STRONG MARKOV PROPERTY FOR SUMS OF IID RVS

239

8

Strong Markov Property for Sums of IID RVs

≡ (S1 , S2 , . . .). Let X1 , X2 , . . . be iid and let Sn ≡ X1 + · · · + Xn . Let S Definition 8.1 The rv N is a stopping time for the sequence of rvs S1 , S2 , . . . if [N = k] ∈ F(S1 , . . . , Sk ) for all k ≥ 1. It is elementary that (1) (2)

FN ≡ F(Sk : k ≤ N ) : A ∩ [N = k] ∈ F(S1 , . . . , Sk ) for all k ≥ 1} = (a σ-field), ≡ {A ∈ F(S)

since it is clearly closed under complements and countable intersections. Proposition 8.1

Both N and SN are FN -measurable.

Proof. Now, to show that [N ≤ m] ∈ FN we consider [N ≤ m] ∩ [N = k] equals this implies [N ≤ m] ∈ FN . Likewise, [N = k] or ∅, both of which are in F(S); (a)

[SN ≤ x] ∩ [N = k] = [Sk ≤ x] ∩ [N = k] ∈ F(S1 , . . . , Sk ),

implying that [SN ≤ x] ∈ FN .

2

Theorem 8.1 (The strong Markov property) the increments continuing from the random time (3)

S˜k ≡ SN +k − SN ,

If N is a stopping time, then

k ≥ 1,

have the same distribution on (R∞ , B∞ ), as does Sk , k ≥ 1. Moreover, defining S˜ ≡ (S˜1 , S˜2 , . . .) , (4)

˜ ≡ F(S˜1 , S˜2 , . . .) is independent of FN (hence of N and SN ). F(S)

Proof.

Let B ∈ B∞ and A ∈ FN . Now, ∞ P ([S˜ ∈ B] ∩ A) = n=1 P ([S˜ ∈ B] ∩ A ∩ [N = n]) ∞ = n=1 P ([(Sn+1 − Sn , Sn+2 − Sn , . . .) ∈ B] ∩ (A ∩ [N = n]))

(a)

with A ∩ [N = n] ∈ F(S1 , . . . , Sn ) ∞ = n=1 P ([(Sn+1 − Sn , Sn+2 − Sn , . . .) ∈ B]) P (A ∩ [N = n]) ∈ B)∞ P (A ∩ [N = n]) = P (S n=1 (b)

∈ B) P (A). = P (S

Then use P (S˜ ∈ B) = P (S = B) to Set A = Ω in (b) to conclude that S˜ ∼ = S. rewrite (b) as (c)

P ([S˜ ∈ B] ∩ A) = P (S˜ ∈ B) P (A),

which is the statement of independence.

2

240


Definition 8.2

(5)

Define waiting times for return to the origin by

W1 ≡ min{n : Sn = 0} · · · Wk ≡ min{n > Wk−1 : Sn = 0}

with W1 = +∞ · · · with Wk = +∞

if the set is empty,

if the set is empty.

Then define Tk ≡ Wk − Wk−1 , with W0 ≡ 0, to be the interarrival times for return to the origin. Proposition 8.2 are, in fact, iid.

If P (Sn = 0 i.o.) = 1, then T1 , T2 , . . . are well-defined rvs and

Proof. Clearly, each Wk is always an extended-valued rv, and the condition P (Sn i.o.) = 1 guarantees that Wk (ω) is well-defined for all k ≥ 1 for a.e. ω. Now, T1 = W1 is clearly a stopping time. Thus, by the strong Markov property, (1) T1 is independent of the rv S˜(1) ≡ S˜ with kth coordinate S˜k ≡ S˜k ≡ ST1 +k − ST1 Thus T2 is independent of the rv S˜(2) with kth coordinate and S˜(1) ≡ S˜ ∼ = S. (2) (1) (1) Continue with S˜(3) , S˜k ≡ S˜T2 +k − S˜T2 = ST1 +T2 +k − ST1 +T2 and S˜(2) ∼ = S˜(1) ∼ = S. etc. [Note the relationship to interarrival times of a Poisson process.] 2

Exercise 8.1 (Wald’s identity) (a) Suppose X1 , X2 , . . . are iid with mean µ, and N is a stopping time with finite mean. Show that Sn ≡ X1 + · · · + Xn satisfies (6)

ESN = µEN.

(b) Suppose each Xk equals 1 or −1 with probability p or 1 − p for some 0 < p < 1. Then define the rv N ≡ min{n : Sn equals −a or b}, where a and b are strictly positive integers. Show that N is a stopping time that is a.s. finite. Then evaluate EN . [Hint. [N ≥ k] ∈ F(S1 , . . . , Sk−1 ), and is thus independent of Xk , while SN = ∞ X k 1[N ≥k] .] k=1

9.

CONVERGENCE OF SERIES OF INDEPENDENT RVS

241

9

Convergence of Series of Independent RVs

In section 10.4 we proved the SLLN after recasting it (via Kronecker’s lemma) as a theorem about a.s. convergence of infinite series. In this section we consider the convergence of infinite series directly. Since the convergence set of a series is a tail event (recall remark 10.1.1), convergence can happen only with probability 0 or 1. Moreover, the first theorem below seems to both limit the possibilities and broaden the possible approaches to them. All proofs are given at the end of this section. Let X1 , X2 , . . . be independent. Then, for some rv S, we have

Theorem 9.1 (1)

Sn ≡

n

Xk →a.s. S

k=1

iff

n

Xk →p S

iff

k=1

n

Xk →d S.

k=1

[We will show the first equivalence now, and the second in chapter 14.] Theorem 9.2 (The 2-series theorem) Xk ∼ = (µk , σk2 ). We then have (2)

n

µk → µ and

k=1

∞

Let X1 , X2 , . . . be independent, with

σk2 < ∞ implies Sn ≡

k=1

n

Xk →a.s. (some rv S).

k=1

If further |Xk | ≤ c for all k, then (including a converse) (3)

S0,n ≡

n

(Xk − µk ) →a.s. (some rv S0 )

∞

if and only if

k=1

(4)

Sn ≡

n

σk2 < ∞,

k=1

Xk →a.s. (some rv S) iff

k=1

n

∞

µk → µ and

k=1

σk2 < ∞.

k=1

Moreover, in all cases, S = S0 + µ and ∞ ∞ (5) Var[S] = σ2 ≡ k=1 σk2 , ES = µ ≡ k=1 µk

and

Sn →L2 S.

If a series is to converge, the size of its individual terms must be approaching zero. Thus the rvs must be effectively bounded. Thus truncation should be particularly effective for series. Theorem 9.3 (The 3-series theorem) Let X1 , X2 , . . . be independent rvs. (c) (a) Define Xk to be the trimmed Xk that equals Xk or 0 as |Xk | ≤ c or as |Xk | > c. Then n (6) Sn ≡ Xk →a.s. (some rv S) k=1

if and only if for some c > 0 the following three series all converge: (7)

Ic ≡

∞ k=1

P (|Xk | > c),

IIc ≡

∞ k=1

(c)

Var[Xk ],

IIIc ≡

∞

(c)

EXk .

k=1

(b) The condition (7) holds for some c > 0 if and only if it holds for all c > 0. n (c) If either Ic , IIc , or IIIc diverges for any c > 0, then k=1 Xk diverges a.s.

242


Example 9.1 Suppose X1 , X2 , . . . are independent and are uniformly bounded. They are assumed to be independent of the iid Rademacher rvs 1 , . . . , n . Then ∞ 2 n (8) if and only if k=1 k Xk →a.s. (some rv S) k=1 σk < ∞. ∞ Moreover, S ∼ σ2 ). [This is immediate from the 2-series theorem.] 2 = (0, k=1

k

Exercise 9.1 X1 , X2 , . . . are iid with P (Xk = 1) = P (Xk = −1) = 12 . ∞ Suppose n Show that n=1 Xn /3 →a.s. (some S), and determine the mean, variance, and the name of the df FS of S. Also determine the characteristic function of S (at some point after chapter 13). n Exercise 9.2 (a) Show that k=1 ak Xk →a.s. (some S) when X1 , X2 , . . . are independent with Xk ∼ = Uniform(−k, k) for k ≥ 1, and where 0 < a < 1. (b) Evaluate the mean and variance of S. Exercise 9.3 LetX1 , X2 , . . . be arbitrary rvs withall Xk ≥ 0 a.s. Let c > 0 n be arbitrary. Then ∞ k=1 E(Xk ∧ c) < ∞ implies that k=1 Xk →a.s. (some rv S), and conversely. Exercise 9.4 ∞

(a) Let Z1 , Z2 , . . . be iid N (0, 1) rvs. Show that

2 2 k k=1 [Z2k−1 + Z2k ]/2 1 ∞ 0n n+k n=1 k=1 Zk /2

→a.s. (some rv), →a.s. (some rv),

and determine (if possible) the mean, variance, and distribution of the limiting rvs. ∞ (b) Let Y1 , Y2 , . . . be iid Cauchy(0, 1) rvs. Does k=1 Yk /2k →a.s. (some rv)? If so, what is the distribution of the limit?

Proofs Proof. Consider theorem 9.1. Now, →a.s. always implies →p . So we turn to the converse. Suppose Sn →p S (which is equivalent to Sm − Sn →p 0). To establish Sn →a.s. , it is equivalent to verify of the convergence set (2.3.7) that (9)

lim lim P ( max |Sm − Sn | ≥ ) = 0 n

N

n≤m≤N

for all > 0.

But Ottavani–Skorokhod’s inequality 10.3.4 gives (a) (b)

m P (maxn≤m≤N |Sm − Sn | ≥ ) = P (maxn≤m≤N | n Xk | ≥ ) N N ≤ P (| n Xk | ≥ /2)/[1 − maxn≤m≤N P (| m Xk | > /2)]

(c)

≤ P (|SN − Sn | ≥ /2)/[1 − maxn≤m≤N P (|SN − Sm+1 | > /2)]

(d)

= o(1)/[1 − o(1)] < θ

for all n, N ≥ (some n,θ ),

using SN − Sm →p 0 for (d). Thus (9) holds, and Sn →a.s. (some rv S ). The a.s. limit S equals S a.s. by proposition 2.3.4. 2 The proof of the 2-series theorem above will require a converse of Kolmogorov’s inequality that is valid for bounded rvs.

9.


243

Inequality 9.1 (Kolmogorov’s other inequality) Consider independent zero-mean rvs Xk , and set Sk ≡ X1 + · · · + Xk for 1 ≤ k ≤ n. Suppose |Xk | ≤ (some M ) < ∞ for all k. Then (10) P ( max |Sk | ≤ λ) ≤ (λ + M )2 / nk=1 σk2 for all λ > 0. 1≤k≤n

Proof. Let Ak ≡ [max1≤j λ),

=

2 1 {E(Sk

by independence

where in step (b) we take advantage of |Sk | ≤ |Sk−1 | + |Xk | ≤ λ + M on Ak . We also note that E Sn2 1n A = ESn2 − E{Sn2 1[Mn ≤λ] } 1

(d)

k

≥ Var[Sn ] − λ2 P (Mn ≤ λ) = Var[Sn ] − λ2 [1 − P (Mn > λ)],

using |Sn | ≤ λ on the event [Mn ≤ λ] to obtain (d). Combining (c) and (d) and doing algebra gives (e) (f)

P (Mn > λ) ≥ ≥1−

Var[Sn ] − λ2 (λ + M )2 = 1 − (λ + M )2 + Var[Sn ] − λ2 (λ + M )2 + Var[Sn ] − λ2

(λ + M )2 . Var[Sn ]

2

Proof. Consider theorem 9.2. We first verify (2), and without loss of generality we assume that all µk = 0. By theorem 9.1, we need only show that Sm − Sn →p 0. But this follows immediately from Chebyshev’s inequality, since ∞ P (|Sm − Sn | ≥ ) ≤ Var[Sm − Sn ]/(n 2 ) ≤ n+1 σk2 /(n 2 ) <

for all sufficiently large n. Thus (2) holds. Consider the forward half of (3), again taking all µk = 0. Since Sn →a.s. , for some sufficiently large λ we have (a)

0 < P (supn |Sn | ≤ λ)

(b)

= limN P (max1≤n≤N |Sn | ≤ λ) by the monotone property of a measure 2 ≤ limN (λ + c)2 / N by Kolmogorov’s other inequality 9.1 1 σk

(c)

= (λ + c)2 /

∞ 1

Note that (c) implies that

σk2 . ∞ 1

σk2 < ∞, giving (3).

244


Consider the forward half of (4). Suppose that Sn →a.s. S. The plan is first to symmetrize, so that we can use (3) to prove (4). Let Xn ’s be independent, and independent of the Xn ’s with Xn ∼ = Xn ; then Xns ≡ Xn − Xn denotes the symmetrized rv. Since →a.s. depends only on the finite-dimensional distributions, n the given fact that Sn →a.s. S implies that Sn ≡ 1 Xks →a.s. (some rv S ) ∼ = S. We can thus claim that n (d) Sns ≡ 1 Xks →a.s. S s ≡ S − S . ∞ s Now, |Xns | ≤ 2c; thus (d) and (3) imply that 1 Var[Xn ] < ∞. Thus ∞ 2 ∞ s (e) 1 σn = 1 Var[Xn ]/2 < ∞. n Now, (e) and (2) imply that 1 (Xk − µk ) →a.s. (some rv S0 ) with mean 0. Thus n n n (f) 1 µk = ( 1 Xk ) − [ 1 (Xk − µk )] →a.s. S − S0 . Thus (4) holds, and S = S0 + µ. We next verify (5) that Var[S] = σ2 and ES = µ. Now, (g)

Var[S] = ≤

(h)

1 2

1 2

E(S − S )2 =

lim E(Sn − Sn )2

= lim

n 1

σk2 =

∞ 1

1 2

E[ lim(Sn − Sn )2 ] =

1 2

E[( lim (Sn − Sn )2 )]

by Fatou’s lemma σk2 = σ 2 .

And Var[S] ≥ σ2 , since for all n Var[S] = 12 E(S − S )2 = 12 Var[ (i)

≥

n 1

n 1

Xns ] + 12 Var[

∞

s n+1 Xn ]

σk2 + 0.

Thus Var[S] = σ2 , and hence ES = µ by theorem 3.5.3 on moment convergence. 2 Proof. Consider the 3-series theorem. Consider (a) and (b) in its statement: Suppose that the 3 series converge for at least one value of c. Then II and III imply (c) that n1 Xk →a.s. by (2). Thus n1 Xk →a.s. by proposition 10.2.1, since I < ∞ (c) (c) implies that X1 , X 2 , . . . and X1 , X2 , . . . are Khinchin equivalent sequences. n Suppose that 1 Xk →a.s. . Then for all c > 0 we have P (|Xn | > c i.o.) = 0, so that I < ∞ holds for all c > 0 by the second Borel–Cantelli lemma. Thus n (c) (c) (c) 1 Xk →a.s. for all c, since I < ∞ implies that X1 , X2 , . . . and X1 , X2 , . . . are Khinchine equivalent sequences. Thus II < ∞ and III < ∞ for all c by the 2-series theorem result (4). Consider (c). Kolmogorov’s 0-1 law shows that Sn either converges a.s. or else diverges a.s.; and it is not convergent if one of the three series fails to converge. 2

9.


245

Other Generalizations of the LLNs Exercise 9.5 The following (with r = 1) can be compared to theorem 10.4.4. If X1 , X2 , . . . are independent with 0 means, then ∞ 2r r+1 (11) < ∞ for some r ≥ 1 implies Sn /n →a.s. 0. 1 E|Xn | /n Exercise 9.6 (Chung) We give another variation of theorem 10.4.4. Suppose φ > 0 is even and continuous and either φ(x)/x but φ(x)/x2 or else φ(x) but φ(x)/x . Let bn ∞. Suppose X1 , X2 , . . . are independent with 0 means. Then ∞ Eφ(Xn )/φ(bn ) < ∞ implies both n=1 n (12) ∞ X /b → and X a.s. n=1 n n 1 k /bn →a.s. 0. The WLLN is taken up again after the characteristic function tool has been introduced.

246

10


Martingales

Definition 10.1 (Martingales) (a) A sequence of rvs S1 , S2 , . . . on a probability space (Ω, A, P ) that is adapted to an sequence of σ-fields A1 ⊂ A2 ⊂ · · · is called a martingale (abbreviated mg) if E|Sk | < ∞ for all k, and (1)

E(Sk |Ai ) =a.s. Si

for all i ≤ k in the index set.

If (Sk , Ak ), k ≥ 1 is a mg, then the increments Xk ≡ Sk − Sk−1 are called the martingale differences. ¯ A collection {St : t ∈ I} (b) Let I denote a subinterval of the extended real line R. of rvs on some (Ω, A, P ) that is adapted to an family of σ-fields {At : t ∈ I} is called a martingale if E|St | < ∞ for all t ∈ I, and (2)

E(St |Ar ) =a.s. Sr

for all r ≤ t in I.

(c) If “=” is replaced by “ ≥” in either of (1) or (2), then either of {Sk : k ≥ 1} or {St : t ∈ I} is called a submartingale (or submg). Example 10.1 (The prototypical example) Let X1 , . . . , Xn denote independent rvs with 0 means, and set Sk ≡ X1 +· · ·+Xk and Ak ≡ σ[X1 , . . . , Xk ] for 1 ≤ k ≤ n. Then the sequence of partial sums satisfies (3)

(Sk , Ak ), 1 ≤ k ≤ n, is a mg,

while (4)

(Sk2 , Ak ), 1 ≤ k ≤ n, is a submg.

The first claim is trivial, and the second holds, since E(Sk2 |Ai ) = E{Si2 + 2Si (Sk − Si ) + (Sk − Si )2 |Ai } (a)

≥ Si2 + 2Si E{Sk − Si |Ai } + 0 = Si2 + 0 + 0 = Si2 ,

using (8.4.20) and (8.4.16).

2

Exercise 10.1 (Equivalence) (a) Show that (St , At ), t ∈ I, is a martingale if and only if (5) St dP = Sr dP for all Ar ∈ Ar and all r ≤ t with r, t ∈ I. Ar

Ar

(b) For a submartingale, just replace “=” by “≥” in (5). Notation 10.1 We will use  and =  mg submg and ≥ (6)  > s-mg and =

the following notational system: for a martingale. for a submartingale. for a s-mg (mg or submg, as the case may be).

Thus exercise 10.1 states that (St , At ), t ∈ I, is a s-mg if and only if > (7) St dP = Sr dP for all Ar ∈ Ar , and for all r ≤ t with r, t ∈ I.2 Ar

Ar

11.

MAXIMAL INEQUALITIES, SOME WITH BOUNDARIES

11

Maximal Inequalities, Some with Boundaries

247

Inequality 11.1 (Monotone inequality) For arbitrary rvs X1 , . . . , Xn and for constants 0 < b1 ≤ · · · ≤ bn we let Sk ≡ X1 + · · · + Xk and obtain k X |Sk | i ≤ 2 max (1) max . 1≤k≤n bk 1≤k≤n bi i=1

If all Xi ≥ 0, then we may replace 2 by 1. [This also holds in higher dimensions, when properly formulated. See Shorack and Smythe(1976).] Proof. (a)

Define b0 = 0, X0 = 0, Yj = Xj /bj , and Tk = Sk =

k

j=1 bj ∆Tj

=

k j=1

∆Tj

j i=1

∆bi =

k

k i=1

j=1 Yj .

Then

Tik ∆bi ,

k k where ∆bj ≡ bj −bj−1 , ∆Tj ≡ Tj −Tj−1 , and Tik ≡ j=i Yj . As i=1 (∆bi /bk ) = 1 with each ∆bi /bk ≥ 0, we have k (b) max |Sk |/bk ≤ max { i=1 |Tik | (∆bi /bk )} 1≤k≤n

1≤k≤n

(c)

≤ max

(d)

≤2

1≤k≤n

max |Tik |

1≤i≤k

since an average does not exceed the maximum

max |Tk | .

1≤k≤n

Note that 1 can replace 2 in step (d) if all Xi ≥ 0.

2

Martingale Maximal Inequalities Inequality 11.2 (Doob) Let (Sk , Ak ), 1 ≤ k ≤ n, be a submg and define the maximum Mn ≡ max1≤k≤n Sk . Then (2) λP (Mn ≥ λ) ≤ Sn dP ≤ ESn+ ≤ E|Sn | for all λ > 0, [Mn ≥λ]

(3)

P (Mn ≥ λ) ≤ inf E(erSn )/erλ r>0

for all λ > 0.

If (Sk , Ak ), 1 ≤ k ≤ n is a zero-mean mg, then (Sk2 , Ak ), 1 ≤ k ≤ n is a submg. This allows the maximum to be bounded by (4)

P (Mn ≥ λ) ≤ Var[Sn ]/λ2

for all λ > 0.

[This last is Kolmogorov’s inequality, valid for zero-mean mgs.]

248


Proof. (a)

Since E(Sn |Ak ) ≥ Sk a.s. by the definition of a submg, we have for all Ak ∈ Ak Ak Sn dP = Ak E(Sn |Ak ) dP ≥ Ak Sk dP

by (8.4.1) in the definition of conditional expectation. Now let (b)

Ak ≡ [max1≤j 0 follows from Jensen’s inequality for conditional expectation. Applying Doob’s first inequality (2) above thus gives (3). [This in often sharper than (2), though it requires the existence of the moment generating function E exp(rSn ).] When (Sk , Ak ) is a mg, then (Sk2 , Ak ) is also a submg (by another application of the same Jensen’s inequality), so that (2) applied to the latter submg gives (4). 2 Inequality 11.3 (H´ ajek–R´ enyi) Let (Sk , Ak ), 1 ≤ k ≤ n, be a mg with all ESk = 0. Let 0 < b1 ≤ · · · ≤ bn . Then $ n % N 4 2 2 2 2 (5) P max |Sk |/bk ≥ λ ≤ 2 σk /bn + σk /bk for all λ > 0. n≤k≤N λ k=1

k=n+1

Proof. (We give the proof for independent rvs.) The monotone inequality bounds the maximum partial sum via / . k S Xi n + (6) max |Sk |/bk ≤ 2 max . n≤k≤N n≤k≤N bn bi i=n+1

Applying Kolmogorov’s inequality (4) to (6) gives N (a) P max |Sk |/bk ≥ λ ≤ (λ/2)−2 Var[Sn /bn ] + k=n+1 Var[Xk ]/b2k n≤k≤N

(b)

= (4/λ2 )

n 1

σk2 /b2n +

2 2 σ /b n+1 k k .

N

(A different proof can eliminate the factor 4.)

2

11.


249

Exercise 11.1 To complete ajek–Rényi inequality for mgs, one the proof of the H´ can show that Tk ≡ Sn /bn + kn+1 Xi /bi is such that (Tk , Ak ), n ≤ k ≤ N , is also a mg, and that Var[SN ] is equal to the right-hand side of (b). Do it. Inequality 11.4 (Birnbaum–Marshall) Let (S(t), A(t)), 0 ≤ t ≤ θ, be a mg having S(0) = 0, ES(t) = 0, and ν(t) = ES 2 (t) finite and continuous on [0, θ]. Suppose that paths of S are right (or left) continuous. Let q(·) > 0 on (0, θ] be and right (or left) continuous. Then θ θ −2 (7) P (S/q0 > λ) ≤ 4λ [q(t)]−2 dν(t) for all λ > 0. 0

Proof. (a)

Because of right (or left) continuity and S(0) = 0, we have θ n n P (S/q0 ≤ λ) = P maxn |S(θi/2 )|/q(θi/2 ) ≤ λ for all n ≥ 1 0≤i≤2

= lim P

maxn |S(θi/2 )|/q(θi/2 ) ≤ λ n

n

0≤i≤2

≥ lim{1 − λ−2

2n 1


E[S 2 (θi/2n ) − S 2 (θ(i − 1)/2n )]/q 2 (θi/2n )}

by H´ ajek–Rényi = 1 − 4λ−2 lim (b)

→ 1 − 4λ−2

θ 0

2n 1

q −2 (θi/2n ) [ν(θi/2n ) − ν(θ(i − 1)/2n )]

[q(t)]−2 dν(t)

using the MCT.

2

Inequality 11.5 (Doob’s Lr -inequality) (i) Let (Sk , Ak ), for 1 ≤ k ≤ n, be a submg. Consider Mn ≡ max1≤k≤n Sk+ . Let r > 1. Then (8)

r r E Mnr ≤ ( r−1 ) E{(Sn+ )r } .

(ii) Let (Sk , Ak ), for 1 ≤ k ≤ n, be a mg. Consider Mn ≡ max1≤k≤n |Sk |. Let r > 1. Then (9)

r r E Mnr ≤ ( r−1 ) E{|Sn |r } .

Proof. Now, (Sk+ , Ak ), for 1 ≤ k ≤ n, is also a submg, by the conditional version of Jensen’s inequality. (Or refer to (18.1.7).) [And refer to (18.1.6) for case (ii).] Thus in case (i) we have ∞ (a) E Mnr = 0 rλr−1 P (Mn > λ) dλ by (7.4.13) ∞ (b) ≤ 0 rλr−1 λ−1 E{Sn+ 1[Mn ≥λ] } dλ by Doob’s inequality 11.2 (c)

= E{Sn+

Mn 0

rλr−2 dλ}

by Fubini

250

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES r = E{ Sn+ ( r−1 ) Mnr−1 }

(d)

r ) (E{(Sn+ )r })1/r (E{Mnr })(r−1)/r ≤ ( r−1

by H¨ older’s inequality,

where r−1 + s−1 = 1 implies that s = r/(r − 1). So (e)

r ) (E(Sn+ )r )1/r , (E Mnr )1−(r−1)/r ≤ ( r−1

which gives the results. (Just change Sn+ to |Sn | for case (ii).)

2

Hoffman–Jorgensen Inequalities The following inequalities show that “in probability” control of the overall sum and of the maximal summand actually gives control of moments of sums of independent rvs. Inequality 11.6 (Hoffmann–Jorgensen, probability form). Let X1 , . . . , Xn be independent rvs, and let Sk ≡ X1 + · · · + Xk for 1 ≤ k ≤ n. Let λ, η > 0. Then (10)

P ( max |Sk | > 3λ + η) ≤ {P ( max |Sk | > λ)}2 + P ( max |Xi | > η) . 1≤k≤n

1≤i≤n

1≤k≤n

If the Xi ’s are also symmetric, then both (11)

P ( max |Sk | > 3λ + η) ≤ {2P (|Sn | > λ)}2 + P ( max |Xi | > η)

(12)

P (|Sn | > 2λ + η) ≤ {2P (|Sn | > λ)}2 + P ( max |Xi | > η) .

1≤k≤n

1≤i≤n

and

1≤i≤n

Inequality 11.7 (Hoffmann–Jorgensen, moment form). Let the rvs X1 , . . . , Xn be independent, and let Sk ≡ X1 + · · · + Xk for 1 ≤ k ≤ n. Suppose that each Xi ∈ Lr (P ) for some r > 0. Then r r 4 r (13) E max |Sk | ≤ 2 (4t0 ) + 2 · 4 E max |Xi | , 1≤k≤n

1≤i≤n

where t0 ≡ inf{t > 0 : P (max1≤k≤n |Sk | > t) ≤ 1/(2 · 4r )}. If the Xi ’s are also symmetric, then r r r r (14) E|Sn | ≤ 2 (3t0 ) + 2 · 3 E max |Xi | , 1≤i≤n

where t0 ≡ inf{t > 0 : P (|Sn | > t) ≤ 1/(8 · 3r )}. Proof. Consider inequality 11.6. Let τ ≡ inf{k ≤ n : |Sk | > λ}. Then [τ = k] n depends only on X1 , . . . , Xk , and [maxk≤n |Sk | > λ] = k=1 [τ = k]. On [τ = k], |Sj | ≤ λ] if j < k, and for j ≥ k, (a)

|Sj | = |Sj − Sk + Xk + Sk−1 | ≤ λ + |Xk | + |Sj − Sk | ;

11.


251

hence (b)

max1≤j≤n |Sj | ≤ λ + max1≤i≤n |Xi | + maxk<j≤n |Sj − Sk | .

Therefore, by independence, P (τ = k, max1≤k≤n |Sk | > 3λ + η) (c)

≤ P (τ = k, max1≤i≤n |Xi | > η) + P (τ = k) P (maxk<j≤n |Sj − Sk | > 2λ) .

But maxk<j≤n |Sj − Sk | ≤ 2 max1≤k≤n |Sk |, and hence summing over k on both sides yields (d)

P (max |Sk | > 3λ + η) ≤ P (max |Xi | > η) + {P (max |Sk | > λ)2 } . k≤n

i≤n

k≤n

The second inequality follows from the first by Lévy’s inequality 10.3.3. For the symmetric case, first note that (e)

|Sn | ≤ |Sk−1 | + |Xk | + |Sn − Sk | ,

so that P (τ = k, |Sn | > 2λ + η) (f)

≤ P (τ = k, max1≤i≤n |Xi | > η) + P (τ = k) P (|Sn − Sk | > λ) ;

and hence summing over k then yields P (|Sn | > 2λ + η) (g)

≤ P (max |Xi | > η) + P (max |Sk | > λ) P (max |Sn − Sk | > λ) . i≤n

k≤n

k≤n

The third inequality again follows from Lévy’s inequality.

2

Proof. Consider inequality 11.7. Here is the proof of (14); the proof of (13) is similar. Let u > t0 . Then, using (12) for (i), u ∞ (h) E|Sn |r = 3r ( 0 + u ) P ( |Sn | > 3t) d(tr ) by (7.4.11). ∞ ∞ (i) ≤ (3u)r + 4 · 3r u P ( |Sn | > t)2 d(tr ) + 3r u P (max1≤i≤n |Xi | > t)d(tr ) ∞ (j) ≤ (3u)r +4·3r P ( |Sn | > u) u P ( |Sn | > t) d(tr )+ 3r E(max1≤i≤n |Xi |r ) . Since 4 · 3r P (|Sn | > u) ≤ (k)

1 2

by our choice of u, applying (7.4.11) again (to (j)) gives

E|Sn |r ≤ 2 · (3u)r +

1 2

E|Sn |r + 2 · 3r E(max1≤i≤n |Xi |r ) .

Simple algebra now gives (14).

Exercise 11.2

2

Provide the details in the case of (13)

252


12

A Uniform SLLN

There are many other extensions and generalizations of the SLLN. A generalization that handles dependent but stationary sequences of rvs is the ergodic theorem, which is presented in many texts. Another direction of generalization is to allow the (realvalued) rvs to be replaced by random elements in a Banach space such as C[0, 1] or D[0, 1]. In fact, the Glivenko–Cantelli theorem can be treated alternatively by theorems of this kind. Another extension involves the uniformity in F (or P ) of the convergence. Theorem 12.1 (Chung) Suppose that X1 , X2 , . . . are iid P ∈ P, where the collection P satisfies the uniform integrability condition sup EP {|X| × 1[|X|≥λ]} = sup [|x|≥λ] |x| dP (x) → 0 (1) as λ → ∞. P ∈P

P ∈P

Then, for every > 0, ¯ m − EX| ≥ → 0 (2) sup P sup |X P ∈P

Proof.

as n → ∞.

m≥n

It follows easily from (1) that supP ∈P EP |X| ≤ (some M ) < ∞;

(a)

the constant M will appear repeatedly in the following calculations. Let > 0. We will prove (2) by producing an N0 ≡ N0 ( ) such that the left side in (2) is less than

for n ≥ N0 . Here is the choice that works. First, by (1), we can choose N ≡ N ( ) so large that 16 2 √ (b) + 6 sup |x| dP (x) + sup [|x|≥N ] |x| dP (x) < /4. 1/4 ] [|x|≥N 2

N P ∈P P ∈P For this N = N ( ) we can then choose N0 = N0 ( ) ≥ N ( ) so large that both 4N 2 (c) N0 supP ∈P [|x|≥N0/4N ] |x| dP (x) < /2 and (d)

NM N0

+ supP ∈P

[|x|≥N ]

|x| dP (x) < /2.

To show that this choice of N and N0 works, we set (e)

Xk ≡ Xk 1[|Xk | for some n ≥ N0

X | >

/4 for some n ≥ N ≤ P | n1 N k 0 k=1 1 n

+P | n k=N +1 (Xk − EXk )| > /4 for some n ≥ N0

(g)

≡ I + II ,

(h) since | n1

(i)

253

n

k=N +1 EXk

− n1 ESn |

n N ≤ | n1 k=N +1 EXk − EXk | + | n1 k=1 EXk | ≤ [|x|≥N ] |x| dP (x) + NnM ≤ /2

for all n ≥ N0 ,

uniformly in P, by (d).

Now we bound the two terms in (h). The first term is easy, since (j) I ≤ P (| N10 N k=1 Xk | > /4) 4N ≤ N P (|X| > N0 /4N ) ≤ N N [|x|>N0 /4N ] |x| dP (x) 0 (k)

< /2

uniformly in P, by (c).

To handle II, first note that it follows from the proof of Kronecker’s lemma that uniformly in P, P (| n1 nk=N +1 (Xk − EXk )| > /4 for some n ≥ N0 ) → 0 , provided uniformly in P that n P (| k=N +1 (Xk − EXk )/k | > /4 for some n ≥ N0 ) → 0 . Since N0 ≥ N , it suffices to show that uniformly in P we have n P (AN, ) ≡ P (| k=N (Xk − EXk )/k | > /4 for some n ≥ N ) → 0. (l) Now, (m)

P (AN, ) = P (AN, ∩ [Xk = Xk for all N ≤ k < ∞])

(n)

+P (AN, ∩ [Xk = Xk for some N ≤ k < ∞]) n ∞ ≤ P (| k=N (Xk − EXk )/k| > /4 for some n ≥ N ) + k=N P (|Xk | ≥ k) ∞ 2 1 ≤ 16 k=N k2 [|x| 0. It would also suffice to show such convergence for other classes of functions. Definitions of other appropriate classes G will be given in section 11.7 and section 11.8. The study of characteristic functions in chapters 13 and 14 is merely an appeal to yet a different G class. Immediately after sections 11.7 and 11.8 the reader could turn to chapter 19, where such topics are treated on general metric spaces for various G classes. 2

1.

STEINS METHOD FOR CLTS

259

2 Theorem 1.1 (Berry–Esseen CLT) Suppose the rvs Xnk ∼ ), for = (µnk , σnk n 2 1/2 1 ≤ k ≤ n, are independent. Let Yk ≡ (Xnk − µnk )/( 1 σnk ) . Now define n 2 3/2 n n ∼ (0, 1) (12) ) . W ≡ k=1 Yk = and γ ≡ ( 1 E|Xnk − µnk |3 )/( 1 σnk

There is uniform convergence to normality when γ ≡ γn → 0, in that (13)

FW − Φ ≤ 9 γ .

Proof. Let Wk ≡ W − Yk = j =k Yj . Further, let Y1 , . . . , Yn be independent copies of the rvs Y1 , . . . , Yn . Specifying that h ≡ hz, is linear on [z, z + ], for the associated f we have E[ W f (W )] = n1 E[ Yk f (W )] n (a) = 1 E{Yk [f (Wk )+Yk f (Wk +Yk∗ )]} for some Yk∗ between 0 and Yk (b) = n1 [0×Ef (Wk )]+ n1 E{Yk2 f (Wk +Yk∗ )} as Yk ,Wk are independent n = 1 E{Yk2 f (Wk + Yk∗ )} (14) for Yk∗ between 0 and Yk . We can also write (this is cute, crucial, and trivial—it uses W ∼ = Wk + Yk ) n n (15) Ef (W ) = ( 1 EYk2 )E[f (W )] = 1 E[ Yk2 f (Wk + Yk )] . We thus obtain from (4) that (c)

(16)

FW − Φ ≤ supz |Eh(W ) − Φh| + √2π ≤ supz |E[f (W ) − W f (W )]| + √2π n ≤ supz 1 E Yk2 [f (Wk + Yk ) − f (Wk )] + [f (Wk ) − f (Wk + Yk∗ )] +

(17)

≤ + supz

n 1

E Yk2 (|Yk | + |Yk |)(1.3 + 2|Wk |)

+ E[hz, (Wk +

(18) (19)

(d)

Yk )

−

hz, (Wk )|Yk

] + E[hz, (Wk +

Yk∗ )

− hz, (Wk )|Yk ]

by applying (7) twice, and its d(·) bound twice in the next step n = 1 E{Yk2 {(|Yk | + |Yk |) (1.3 + 2|Wk |) + |Yk | + + |Yk | + }} +

n n ≤ 1 {E|Yk |3 + E(Yk2 ) E|Yk |}(1.3 + 2E|Wk | + 1) + 2 ( 1 EYk2 ) +

by the independence of Yk and Wk n ≤ 1 {E|Yk |3 (1.3 + 2 + 1) + (E|Yk |3 )2/3 (E|Yk |3 )1/3 (1.3 + 2 + 1)} + 2 +

n ≤ 8.6 1 E|Yk |3 + 3 ≤ 9 γ + 3

with > 0 arbitrary. [It is my “especially” remark following Stein’s lemma that makes this proof so nice (and the Lindeberg–Feller proof below).] 2

260

CHAPTER 11. CONVERGENCE IN DISTRIBUTION

2 Theorem 1.2 (Lindeberg–Feller CLT) Suppose Xnk ∼ ) are = Fnk (µnk , σnk n 2 n 2 independent, for 1 ≤ k ≤ n. Let sdn ≡ 1 σnk . Define Zn ≡ 1 (Xnk − µnk )/sdn . The following are equivalent:

(20)

FZn − Φ → 0

(21)

LFn

≡

n k=1

[max1≤k≤n P (|Xnk − µnk |/sdn > )] → 0 .

and

[|x−µnk |/sdn ≥]

x − µnk sdn

2 dFnk (x) → 0

for all > 0 .

[Rvs that satisfy the second condition in (20) will be called uniformly asymptotically negligible (and abbreviated uan).] [Condition (21) is Lindeberg’s condition; Feller established its necessity.] Moreover, Lindeberg’s condition implies both (22)

Mn →p 0

and

2 [ max1≤k≤n σnk /sdn ] → 0 ,

for the maximum Mn ≡ [max1≤k≤n |Xnk − µnk |/sdn ]. Proof. Let the n Yk ≡ (Xnk − µnk )/sdn , and let Fk denote n df of the rv Yk . Then W ≡ Zn = k=1 Yk ∼ = (0, 1). Define the average df F¯n ≡ 1 Fk /n. Lindeberg’s (21) gives 2 (a) σnk /sd2n ≤ [ [|y|≤] y 2 dFk (y) + [|y|≥] y 2 dFk (y) (b)

≤ 2 + LFn ≤ ,

for n ≥ (some n ) .

2 Thus [maxk σnk /sd2n ] → 0, giving the uan condition. Since we easily calculate that (c) LFn ≥ 2 n1 P (|Xnk − µnk |/sdn ≥ ) ,

Lindeberg’s condition also implies Mn →p 0 via (10.3.14). (·) associated with linear hz, (·) are uniformly Lipschitz Fix > 0. Since the fz, on R (by (8)), the term in (16) is bounded by 5 , provided that all |Yk | ∨ |Yk | never exceeds the appropriate δ = 2 (note (8) again). Thus (16) and f ≤ 2 give the upper bound (23) FW − Φ ≤ 4 n1 [|y|≥2 ] y 2 dFk (y) + 4 n1 EYk2 P ( |Yk | ≥ 2 ) + 5

(d)

≤4

n 1

[|y|≥2 ]

y 2 dFk (y) + 4 [maxk P ( |Yk | ≥ 2 )] + 5 .

Thus Lindeberg’s LFn -condition (21) (which implies uan) implies FW − Φ ≤ 6

for n ≥ (some n ), for > 0 arbitrary. Thus FW − Φ → 0. Note the nice format n (e) LF = y 2 dF¯n (y) when all µnk = 0 . [|y|≥]

The necessity will be established in chapter 14, using characteristic functions. However, we will use the necessity in this chapter. 2

1.


261

Many interesting statistics can be put in the form W + ∆, where W is a “nice” rv. For us, nice will mean that we have already established a useful bound on FW − Φ. Can we then claim a useful bound on FW +∆ − Φ as well? The previous methods are also well suited for establishing such bounds, as the next lemma shows. This applies beautifully to U -statistics; see section 17.1. [The reader will have sufficient background after this section to read all of chapter 17 except for the short subsections on the SLLN and the functional CLT for U -statistics.] Lemma 1.3 (Berry–Esseen potential)

For any rvs W and ∆ we have

FW +∆ − Φ ≤ FW − Φ + 4 E|W ∆| + 4 E|∆| . √ √ [The ∆-terms are bounded by 4 EW 2 E∆2 + 4 E∆2 ; and often EW 2 ≤ 1 will provide an easy upper bound.] (24)

Proof.

Recall that a = b ⊕ c means that |a − b| ≤ c. Now, FW +∆ − Φ ≤ FW +∆ − FW + FW − Φ ,

where we take the difference again for FW +∆ − FW = [FW +∆ − Φ] − [FW − Φ] (a)

= E[f (W + ∆) − (W + ∆)f (W + ∆)] − E[f (W ) − W f (W )] ⊕

by (4), twice = E[f (W + ∆) − f (W )] − E(W [f (W + ∆) − f (W )] − E[∆f (W + ∆)] ⊕ .

Using Stein’s lemma on the first term gives FW +∆ − FW (b)

≤ E{|∆| (1.3 + 2|W |) + |∆| + } + f E|W ∆| + f E|∆| +

≤ (2 + f ) E|W ∆| + (2.3 + f ) E|∆| + 2

(c)

≤ 4 E|W ∆| + 4 E|∆| + 2 ,

with > 0 arbitrary. (Note also a related (13.5.7).)

Exercise 1.1 (25)

2

Let rvs X and ∆ be arbitrary. Suppose G (·) = g(·). Then

FX+∆ (·) − G(·) ≤ FX (·) − G(·) + g + P (|∆| > ) for > 0 .

[To avoid loss, a strong bound like P (|∆| > ) ≤ K is needed, but not moments.]

262


and Motivation 1.2 (Alternative Stein formulation) Consider both W ≡ W (X) Y ), and define W ≡ W + V . Suppose that W and W are coupled via V ≡ V (X, (26)

(W, W ) ∼ = (W , W )

(a) :

= −λW (b) : E(V |X)

(cc) : EV 2 = 2λ

for some 0 < λ < 1. Thus (W )] = − 1 E[V f (W )] E[W f (W )] = − λ1 E[E(V |X)f λ =

1 2λ

E{V [f (W ) − f (W )]}

by the symmetry of the exchangeability (26)(a) (27)

=

1 2λ

E{V 2 [f (W + V ∗ )}

for some V ∗ between 0 and V . Then for any linear h ≡ hz, , plugging the associated f ≡ fz, into (5) gives |Eh(W ) − Φh| = |E{f (W ) − W f (W )}|

(28)

(29)

=

1 2λ

|E{E(V 2 ) f (W ) − V 2 f (W + V ∗ )}|

≤

1 2λ

− EV 2 }f (W )] | + | E [{E(V 2 |X)

1 2λ

| E [V 2 {f (W + V ∗ ) − f (W )}] |

≤ f

1 2λ

2 E |E{V 2 |X}−EV |+ 21λ E {V 2 [|V |(1.3+2|W |)+|V |+ ]} by (7)

≤ f

1 2λ

− EV 2 | + E |E{V 2 |X}

≤

1 λ

− EV 2 | + E |E{V 2 |X}

1 λ

2 2λ

{1.15 E |V |3 + E |V 3 W |} +

2λ

EV2

{1.15 E |V |3 + { E V 4 }3/4 {E W 4 }1/4 } +

2λ

.

And since > 0 is arbitrary, the recipe of (4) leads to (30)

− EV 2 | + 1.15 E |V |3 + { E V 4 }3/4 {E W 4 }1/4 } . FW − Φ ≤ λ1 { E |E{V 2 |X}

This bound (30) is useful in additional cases. [In the context of the Berry–Esseen theorem, and exploiting the independence still available at line (28), the author can obtain the bound 14γ via this approach.] One can replace the second term in (28) by 21λ f E |V |3 , by (8). This yields FW − Φ ≤ (31)

≤

1 λ

1 λ

− EV 2 | + E |E{V 2 |X}

− EV 2 | + E |E{V 2 |X}

1 λ×

1 (2π)1/4 λ1/2

E |V |3 +

√ 2π

{E |V |3 }1/2 ,

√ by specifying 2 ≡ 2π E|V |3 /λ. [We replaced h by 1/ , since these h’s are linear on [z, z + ].] Recall my “especially” remark at the end of lemma 1.2(i), noting how the

terms in the numerator and denominator had to be “balanced” in (31). The result was an inferior compromise. (Read exercise 1.2 below.) 2

1.


263

Exercise 1.2 (Comparing different Stein-type strategies) (a) Establish a bound on (28) in the context of the Berry–Esseen theorem. (b) Establish a bound on (30) in the context of the Berry–Esseen theorem. [This costs a fourth-moment assumption, as opposed to a third-moment assumption.] (c) Establish a bound on (31) in the context of the Berry–Esseen theorem. [Now, you will get convergence only at the rate 1/n1/4. This is your penalty for ignoring my “especially” remark in the last sentence of lemma 11.1.1. (Note also the difference between (17.2.6) and (17.2.7) in Hoeffding’s combinatorial CLT below.)] (Read the paragraph below (31).) Exercise 1.3 Suppose Zn →d Z0 , and let Pn (B) ≡ P (Zn ∈ B) for all Borel sets B and all n ≥ 0. Let C denote an open subset of the real line. Show that lim inf n Pn (C) ≥ P0 (C). In this entire section we have used only the self-contained discussion 10.3.1 with (10.3.14) from chapter 10.

264

2


˜ ˇ Winsorization and Truncation

Definition 2.1

(a) Call the rvs Xn1 , . . . , Xnn weakly negligible if

Mn ≡ [ max1≤k≤n |Xnk | ] →p 0 . (b) Call them uniformly asymptotically negligible (or uan) if max1≤k≤n P ( |Xnk | ≥ ) → 0

for all > 0.

Notation 2.1 (Weak negligibility in the CLT context) Let Xn1 , . . . , Xnn be independent rvs having dfs Fn1 , . . . , Fnn . Let θ > 0 be given. Define xθn by requiring [−xθn , xθn ] to be the smallest closed and symmetric interval to which F¯n ≡ 1 n 1 n ¯ 1 Fnk assigns probability at least 1 − θ/n. Let Pn (x) ≡ n 1 P (|Xnk | > x) n denote the average tail probability, and then let K n denote the qf of the df 1− P¯n (·). Note the quantile relationship xθn = K n (1 − θ/n). ˜ nk denote Xnk Winsorized outside [−xθ0 n , xθ0 n ] (often, θ0 ≡ 1). Let µ ñk Let X 2 and σ ñk denote the resulting means and variances, and then set both µ ñ ≡ n1 n1 µ ñk n 2 √ and σ ñ2 ≡ n1 1 σ ñk . Applying discussion 10.3.1 to the rvs |Xnk |/ n σ ñ , whose √ θ/nth quantile is xθn / n σ ñ , shows that the following are equivalent conditions for weak negligibility: √ (1) ñ →p 0 . Mn ≡ [ max |Xnk | ]/ n σ 1≤k≤n

(2) (3)

√ xθn / n σ ñ → 0 for all 0 < θ ≤ θθ0 . √ n ñ ) → 0 for all 0 < ≤ 1 . 1 P (|Xnk | > n σ

2

Theorem 2.1 (CLT using Winsorization) Let Xn1 , . . . , Xnn be independent rvs having dfs Fn1 , . . . , Fnn . If any/all of (1), (2), and (3) holds, then both √ ¯n − µ (4) Z¯n ≡ n [X ñ ]/˜ σn →d N (0, 1) and √ ˜ n ≡ [max |Xnk − µ (5) M ñk | ]/ n σ ñ →p 0 . k

˜ θ denote Xnk Winsorized outside [−xθn , xθn ]; Notation 2.2 For 0 < θ ≤ θ0 let X nk θ θ let µ ñk and σ ñk denote the resulting means and standard deviations, and define n θ n θ 2 2 ˜ θn ≡ 1 n X ˜ θ , and let both µ ˜θn ≡ n1 1 µ ñk and σ ˜θn ≡ n1 1 (˜ σnk ) . Let X nk 1 n √ θ ˜ θn − µ ˜θ − µ ˜ θn ]/˜ σθn . Define third central moments γñk ≡ E |X ˜ θnk |3 , Z˜θn ≡ n [X nk n θ ˜ θn ≡ [ maxk |Y˜ θ | ] denote the maximal summand. and let γ˜θn ≡ 1 γñk . Let M nk √ θ θ θ ˜ ˜ Call the rvs Y ≡ (X − µ ˜ )/ n σ ˜θn the associated rvs. 2 nk

nk

nk

˜ n →p 0 by the Proof. Now, all µ ñk ∈ [−xθ0 n , xθ0 n ]. Thus Mn →p 0 implies M triangle inequality. n n θ ˜ )), we have Observe that with δn ≡ P ( 1 Xnk = 1 X nk √ ¯ ˜ (a) P (Zn ≤ z) = P (Zθn × (˜ σθn /˜ σn ) + n(˜ µθn − µ ˜ n )/˜ σn ≤ z) ⊕ δn .

2.

˜ ˇ WINSORIZATION AND TRUNCATION

265

n n θ 3 ˜ ) ≤ θ is trivial, and since F ˜ − Φ ≤ 9 γ˜θn /√n σ Since P ( 1 Xnk = 1 X ˜θn nk Zθn by the Berry–Esseen theorem, (4) will follow immediately from showing that (6) below holds. [Recall that a = b ⊕ c means |a − b| ≤ c.] 2 Inequality 2.1 (6)

γ˜θn (i) √ 3 nσ ˜θn

For each 0 < θ ≤ θ0 , the inequalities (7), (8), and (9) lead to √ n|µ ˜θn − µ ñ | σ ˜2 → 0, (j) θn → 1 , (k) → 0. 2 σ ñ σ ñ

ñ = σ ˜1n . Bounding Proof. For convenience, set θ0 = 1. We then note that σ θ 3 θ ˜θ − µ one power of |X ˜ | by 2 x in the integrand of each γ ˜ gives θn nk nk nk (7)

x x γ˜ √ θn3 ≤ 2 √ θn ≤ 2 √ θn → 0, nσ ˜θn nσ ˜θn nσ ñ

using (2). Since the probability outside [−x1n , x1n ] is at most 1/n, √ xθn n|µ ˜θn − µ ñ | (8) ≤√ →0 σ ñ nσ ñ using (2). We need some notation before turning to (6)(j). Let V˜θn denote the ˜ θ , and set Vñ ≡ V˜1n . Now, F¯n assigns at most 1/n probability second moment of X nk to the complement of the interval [−x1n , x1n ], and on this complement the integrand of V˜θn never exceeds 4x2θn . And so (note also inequality 7.10.2) (a) (b)

2 2 2 2 0 ≤ [˜ σθn −σ ñ2 ]/˜ σθn = {V˜θn − Vñ }/˜ σθn − {˜ µ2θn − µ ˜ 2n }/˜ σθn 2 2 = {Vñ ⊕ (2 xθn )2 /n − Vñ }/˜ σθn − {[ µ ˜ n ⊕ xθn /n]2 − µ ˜2n }/˜ σθn 2 2 ≤ 4x2θn /[n˜ σθn ] + 2 [ |˜ µn | xθn ]/[n˜ σθn ]+0

(9)

2 2 ≤ 6 {x2θn /n˜ σθn } ≤ 6 {x2θn /n˜ σ1n }→0

¯ by (2). Thus (6) holds, and it gives the normality √ of Zn in (4). [We have actually established (4) under the slightly weaker xθn / n σ ˜θn → 0 for all 0 < θ < θ0 .] 2 Exercise 2.1

Show that Sn /˜ σn →p 1 holds in the context of theorem 2.1.

Theorem 2.2 (Asymptotic normality) Let Xn1 , . . . , Xnn be independent rvs. ˇ nk be the truncated rv that equals Xnk or 0 according Let σ > 0. Fix c > 0. Let X as |Xnk | < c or as |Xnk | ≥ c. Let Mn ≡ [max1≤k≤n |Xnk | ]. The following are equivalent: n 2 (10) for uan rvs Xn1 , . . . , Xnn . k=1 Xnk →d N (µ, σ ) n for all > 0 (equivalently, Mn →p 0) and k=1 P (|Xnk | ≥ ) → 0 (11) n n 2 ˇ nk → µ ˇ nk ] → σ2 . µ ˇn ≡ EX and σ ˇn ≡ Var[X k=1

k=1

If (11) holds for one c > 0, then it holds for all c > 0. [These results also hold with ˜ nk , µ ˇ nk , µ Winsorized quantities X ñ , σ ñ2 replacing the truncated quantities X ˇn , σ ˇn2 .]

266


n n n ˇ Proof. Suppose (11). Then P ( 1 Xnk = 1 Xnk ) ≤ 1 P (|Xnk | ≥ c) → 0. ˇ nk − µ Thus we need only show that Zˇn ≡ ( n1 X ˇn )/ˇ σn →d N (0, 1). It suffices to ˇ n of (11.1.21) satisfies LF ˇ n → 0. We will do so presently. verify that Lindeberg’s LF ˇ nk | →p 0. Thus First observe that Mn →p 0, by (10.3.14). Thus max |X (a)

ˇ nk | → 0, m ˇ n ≡ max |ˇ µnk | ≤ E max |X

by the DCT with dominating function c. Thus for n ≥ (some n ) we have (b)

m ˇ n ≤ σ/8,

and also

σ ˇn ≥ σ/2 .

ˇ n of (11.1.21) satisfies Then the Lindeberg–Feller quantity LF ˇ ≡ 12 n (c) LF [x − µ ˇnk ]2 dFnk (x) n k=1 [|x−ˇ σ ˇ µnk |≥ σ ˇn ]∩[|x|≤c] n

c+ σ/8 σ/2

(d)

≤

(e)

≤

(f)

→0

c+ σ/8 σ/2

2 n

k=1 [|x−ˇ µnk |≥ σ ˇn ]∩[|x|≤c]

2 n k=1

P (|Xnk | ≥ σ/4)

dFnk (x)

for n ≥ n , by (b)

by (b) again

by (11).

˜ nk and Thus (10) holds. [For the Winsorized version, just recopy this proof using X dFñk .] We will not prove the converse. 2

Analogous Results for QFs This subsection primarily contains technical details to be used below. Notation 2.3 Let Xn1 , . . . , Xnn be row-independent rvs having dfs Fnk and qfs ˜ nk denote Knk Winsorized outside Knk . Let an ≡ θn /n where θn 0, and let K (an , 1 − an ); so it equals Knk+ (an ), Knk (t), Knk (1 − an ) according as t ≤ an , an < ˜ nk ≡ K ˜ nk (ξnk ), for row-independent Uniform(0, 1) t < 1 − an, 1 − an ≤ t. Define X 2 ∼ ˜ rvs ξnk . Writing Xnk = (˜ µnk , σ ñk ) with third absolute central moment γñk and first absolute central moment m ˜ nk defines our notation for these various moments. Let n 2 1/2 n n ˜sdn ≡ [ 1 σ ñk ] and γñ ≡ 1 γñk . We next define µ ñ ≡ n1 k=1 µ ñk and then n 2 1/2 2 1 1 2 let σ ñ ≡ n ˜sdn = n [ 1 σ ñk ] . We further let n n ˜ nk − µ (12) Z¯n ≡ k=1 (Xnk − µ ˜ nk )/ ˜sdn and Zñ ≡ k=1 (X ñk )/ ˜sdn . To measure the potential outliers of each Xnk , we define (13)

qnk ≡ Knk (1 − an ) − Knk+ (an )

and

qn ≡ [max1≤k≤n qnk ] .

2

Now, µ ñk is necessarily contained in the obvious interval of length qnk . It is ˜ nk − µ important and trivial (since sup |X ñk |/˜sdn ≤ qnk /˜sdn ) that (14)

γñ / ˜sd3n ≤ [max1≤k≤n qnk ]/ ˜sdn = qn / ˜sdn .

2.

˜ ˇ WINSORIZATION AND TRUNCATION

267

The simple inequality (12) gives useful results. [The basic idea we employ is the ˜ nk = Xnk for all 1 ≤ k ≤ n) ≥ (1 − 2θn/n)n → 1. Thus trivial observation that P (X Zñ will have the same limiting distribution as Z¯n . We will study Zñ instead of Z¯n , because it is normed and because (14) is applicable.] The normalized summands ˜ nk − µ ñk )/˜sdn Yñk ≡ (X of Zñ will be called the equivalent summands. These are particularly appealing to statisticians because of their natural normalization, and because the worst potential outliers have been eliminated. Applying Berry–Esseen’s theorem 11.1.1 via (14) immediately gives part (a) of the following theorem. We will establish part (b) in the remarks that follow the theorem. Part (b) is a useful result, and in the iid case it is a good result. [Our goal in this subsection is remark 2.3 below.] (The case where all θn = θ is also covered by the results below; and since (1 − θ/n)n → exp(−2θ) > 1 − 2θ, this case can also be made to work.) 2 Theorem 2.3 (a) Let Xnk denote any array of row-independent rvs, and appeal to notation 2.2. Whenever the equivalent summands Yñk above satisfy γñ /˜sd3n → 0 (which is implied by qn /˜sdn → 0, as in (14)), it follows that (15) Z¯n ≡ nk=1 (Xnk − µ ˜ nk )/ ˜sdn →d N (0, 1) and Zñ →d N (0, 1) . (b) If Zñ →d N (0, 1) for the uan equivalent summands Yñk above, then n 2 (16) an k=1 qnk /˜sd2n → 0 . Remark 2.1 (Necessity) Fix 0 < < 1, and fix 0 < θ < θ0 . Line (i) of the general necessity proof of the Lindeberg–Feller theorem in section 14.2 will establish from the LFn condition that appears in (11.1.21) that for all n ≥ (some n,θ ) we have the bound n n 2 1 [|y|≥] y dFnk (y) ≤ θ + θ 1 [|y|≥] dP (Ynk ≤ y) (17) n = θ{1 + 1 P (|Ynk | ≥ )} . [Thus bounding lim n1 P (|Ynk | ≥ ) will imply that the Lindeberg–Feller term on the left-hand side has limit zero.] When that same necessity argument is read in qf notation (that is, replace x, dFnk (x), and the old interval In in line (e) of that ˜ nk (t), dt, and the new interval In ≡ (an , 1 − an )), that same necessity proof by K argument (for arbitrary 0 < an < 12 ) leads (via lines (j) and (l) of that proof) to n n ñk )/˜sdn ]2 dt ≤ + 1 (an ,1−an )c dt 1 (an ,1−an )c [(Knk (t) − µ (18) = {1 + 2 θn } . [A suitable choice for an will prove to be θ/n, for any 0 < θ < θ0 , or θn /n with θn 0 (and perhaps requiring lim sup(θn /θn+1 ) < ∞ also).] We will appeal to these inequalities in the present section. (A reader familiar with characteristic functions can read the straightforward proof of (15) and (16) in theorem 14.2.2 now.) 2

268


Remark 2.2 (Technical comment) We wish to observe the truth of (22) below. Define the tail contributions 2 (19) d2n ≡ an n1 [ Knk+ (an )−˜ µnk ]2 /˜sd2n , d n ≡ an n1 [ Knk (1−an )−˜ µnk ]2 /˜sd2n . Now note that n 2 ˜ (20) ñk ]2 dt/˜sd2n = d2n + d n . 1 (an ,1−an )c [Knk (t) − µ Applying the inequality (a2 + b2 ) ≤ (|a| + |b|)2 ≤ 2 (a2 + b2 ) termwise allows us to n 2 2 sandwich an 1 [Knk (1−an )−Knk+(an )]2 /˜sd2n between (d2n +d n ) and 2 (d2n +d n ). Thus n 2 2 (21) d2n + d n → 0 if and only if an 1 qnk /˜sd2n → 0 . We now turn to theorem 2.3. Suppose the Zñ of (12) and (15) satisfies Zñ →d N (0, 1) for a particular sequence an ≡ θn /n having θn → 0. Appealing to (18) with ˜ nk ’s gives K 2 2 d2n ∨ d n → 0 (equivalently an n1 qnk /˜sd2n → 0 ) whenever (22) ˜ ˜ Zn →d N (0, 1) and the equivalent summands Ynk are uan, since the right side of (18) equals (1 + 2 θn ) → , for all arbitrarily small . Thus theorem 2.3(b) holds. This argument is also valid when we set all θ = 1. That is, if we set all θ = 1 and ˜ verify that both the resulting Y˜ nk are uan and that the resulting Zn →d N (0, 1), n 2 2 then we can conclude that an 1 qnk /˜sdn → 0 for the value resulting from the choice an = 1/n. That is, (22) also holds when an = 1/n for all n. 2 Remark 2.3 (Summary for iid rvs) When the Ynk are iid and √ θn 0, the condition in theorem 2.3(a) is implied by the requirement that qn / n σ ñ → 0 ; and this is equivalent to [an qn2 /˜ σn2 ]/θn → 0. Also, the condition in theorem 2.3(b) reduces to the requirement that [an qn2 /˜ σn2 ] → 0 . We will apply this formally in the next section as theorem 11.3.1(e). (All we need for these to be equivalent is “an added touch” to close the gap. We close it in the next section.) Obtaining this remark was the main goal of this subsection. However, what has already been established is sufficient for our treatment of the bootstrap in section 11.4. 2

3.

IDENTICALLY DISTRIBUTED RVS

269

3

Identically Distributed RVs

Notation 3.1 (Slowly varying σ ˜2 (·)) Let K be a fixed qf. Let a > 0 be tiny. We agree that dom(a, a) denotes [0, 1 − a), (a, 1], or (a, 1 − a) according as X ≥ 0, ˜ a,a (·) denotes K Winsorized outside dom(a, a). X ≤ 0, or general X, and that K ˜ a,a (·) [For example, when X takes on both positive and negative values, then K equals K+ (a), K(t), K(1 − a) according as t ≤ a, a < t < 1 − a, 1 − a ≤ t.] − − (t). Then define v(t) ≡ [K+ (1 − t)]2 + [K+ (t)]2 . Define q(t) ≡ K + (1 − t) + K+ 2 [Note that v(t) ≤ q (t) ≤ 2 v(t).] Let ξ be a Uniform(0, 1) rv. Let X ≡ K(ξ), and ˜ ˜ a,a (ξ). We also define notation for various moments by agreeing that X(a) ≡K (1)

˜ ˜ a,a (ξ) ∼ X(a) ≡K µ(a), σ ˜2 (a)) = (˜

with

˜ γ˜ (a) ≡ E|X(a) −µ ˜(a)|3 .

Now define the rvs (Xn1 , . . . , Xnn ) ≡ (K(ξn1 ), . . . , K(ξnn )) for row-independent Uniform(0, 1) rvs ξn1 , . . . , ξnn ; thus they are row-independent with df F and qf K. Our interest is in √ ¯ ñ ]/˜ σn , where µ ñ ≡ µ ˜(1/n) and σ ñ ≡ σ ˜ (1/n) , Z¯n ≡ n [X n−µ (2)

˜ n ) ≡ √n [X ñ − µ Z(a ˜(an )]/˜ σ (an ),

ñ ≡ where X

1 n

n ˜ 1 Xnk (an ),

¯n − µ ¯ n ) ≡ √n [X ˜(an )]/˜ σ (an ) . Z(a This is enough notation for the following theorem, but we will also prepare now for its aftermath. Let σ ñ ≡ σ ˜1n ≡ σ ˜ (1/n). Let v1n ≡ V (1/n), where V (t) ≡ dom(t,t) K 2 (s) ds on [0, 12 ] and (3) 1 m(t) ≡ dom(t,t) |K(s)| ds on [0, 2 ] , which for all t sufficiently small equal dom(t,t) v(s) ds and dom(t,t) q(s) ds. The Winsorized second moment [when Winsorized outside dom(1/n, 1/n)] is given by ˜2 v˜1n ≡ EK 1/n,1/n (ξ). Let |X| have df F|X| (·) and qf K|X| (·). Define partial moments or truncated moments in terms of both dfs and qfs via 2 V|X| (t) ≡ [0,1−t) K|X| (s) ds on [0, 1] , (4) U (x) ≡ [|y|≤x] y 2 dF (y) on [0, ∞) . Let v¯1n ≡ V|X| (1/n) and u1n ≡ U (x1n ), where we define xθn ≡ K|X| (1 − θ/n) to be the (1 − θ/n)-quantile of the rv |X|. Thus [−xθn , xθn ] is the shortest interval symmetric about the origin that contains at least proportion θ/n of the X probability. The Winsorized second moment in the present context is u ˜1n , where 2 u˜1n ≡ u1n + x21n P ( |X| > 1/n) equals v˜¯1n ≡ v¯1n + K|X| (1 − 1/n)/n, 2 (x) denote the variance of even though the lead terms may not be equal. Let σ ˜|X| 2 an X Winsorized outside [−x, x], and note that σ ˜|X| (x1n ) = σ ñ2 = σ ˜ 2 (1/n).

2

270


Theorem 3.1 (Studentized CLT for σ ˜2 ∈ L) Suppose r(t) ≡ tq2 (t)/˜ σ2 (t) → 0 as t → 0. [Any n 0 below satisfies lim sup( n / n+1) → 0.] Then:

(b)

Z¯n →d N (0, 1) whenever ñ in (2) whenever µ ˇn ≡ (1/n,1−1/n) K(t) dt can replace µ

r(t) → 0.

(c)

˜ Z(1/n) →d N (0, 1)

if and only if

r(t) → 0.

(d)

¯ n /n) →d N (0, 1) Z(

if and only if r(t) → 0.

(e)

˜ n /n) →d N (0, 1) Z(


(f)

Xn2 /νn →p 1


(a)

for one νn > 0 with n νn

r(t) → 0.

Conditions (6)–(29) below are all equivalent. (Conditions (6), (9), (15), (19) (21), (22), (26), (27), (28), and (29) are especially interesting.) Proof. Consider (a). We are Winsorizing the qf on [t, 1 − t], not the df on [−xθn , xθn ]. Nevertheless, we essentially repeat the argument used for (10.2.4). Let √ n ˜ nk ( /n)] − µ (m) µ ñ ≡ µ ˜( /n) , σ ñ ≡ σ ˜ ( /n) , Zñ ≡ n { n1 [ 1 X ñ }/˜ σn . (Now, a = b ⊕ c means that |a − b| ≤ c.) First observe, the probability that n with n ˜ nk ( /n)), that the two rvs differ bounded by n ≡ P ( k=1 Xk = k=1 X √ P (Z¯n ≤ z) = P (Zñ × (˜ (n) σn /˜ σn ) + n(˜ µn − µ ñ )/˜ σn ≤ z) ⊕ n . n n ˜ nk ( /n)) ≤ is trivial, and since the Berry–Esseen Since n = P ( 1 Xk = 1 X √ 3 theorem gives FZñ − Φ ≤ 9 γñ / n σ ñ , the current theorem will follow from showing that for each > 0 we have √ σ ˜ ( /n) n |µ ˜( /n) − µ ˜(1/n) | γñ (5) → 1 , (k) → 0. (i) √ 3 → 0 , (j) nσ ñ σ ˜ (1/n) σ ˜ (1/n) − Recall from above that q(a) ≡ K + (1 − a) + K+ (a). Then when r(t) → 0, we have √

/n q( /n) γ˜ 1 √ n3 ≤ √ × (o) = r( /n)/ → 0 using r(t) → 0, nσ ñ

σ ˜ ( /n)

(p)

(q)

σ ˜ 2 ( /n)/˜ σ 2 (1/n) → 1 in analogy with (11.2.9), using r(t) → 0, √ √ n|µ ˜( /n) − µ ˜(1/n) |/˜ σ(1/n) ≤ {q( /n) − q(1/n)}/ n σ ˜ (1/n) √ ≤ q( /n)/ n σ ˜ (1/n) √ ≤ {r( /n)/ } {˜ σ ( /n)/˜ σ(1/n)} → 0 by (o) and (p).

Thus (5), and thus (a). Just (5)(i)( = 1) gives the normality in (c). [Temporarily set aside normality in (e).] Consider the implications of normality in (c) and (e). Remark 2.2 shows that the asymptotic normality implies (11.2.16) (with either

3.


271

an = 1/n for (c) or an = n /n for (e)). Also, r(t) → 0 gives (b). See theorems 7.7.1 and 7.7.2 and theorems 10.6.1 – 10.6.4 for proposition 3.1. Then (f) is (23). Then (d) and (e) are equivalent, since P (Zñ ( /n) = Z¯n ( /n)) ≥ (1 − 2 /n)n → 1. ˜ n /n) →d N (0, 1). For the normality in (e), we will use (10) to establish that Z(

And we will do so by showing that (5)(i) (in the present circumstances √ we need √ an n , rather than an ) satisfies γñn ≤ r( n /n)/ n → 0 for r(t) ≡ t q(t)/˜ σ (t) and for some choice of n 0. (Recall again remark 11.2.3.) Well, in the worst situation r(t) → 0 rather slowly. So we define r¨(t) ≡ sup{r(s) : 0 < s ≤ t}, so that r¨(t) 0. This allows (r) r(t r¨(t))/ r¨(t) ≤ r(t)/ r¨(t) ≤ r¨(t)/ r¨(t) ≤ r¨(t) 0 , and thus the choice n ≡ r¨(1/n) works! Finally, it is trivial that (s)

lim ( n / n+1) = lim{¨ r (1/n)/¨ r(1/(n + 1))} ≤ 1.

2

Proposition 3.1 (Conditions equivalent to the slow variation of σ ˜2 (·)) The following contains just the “best” items from our list of equivalencies. When any one (hence all) of these equivalencies hold, we will write either K ∈ D(Normal) or F ∈ D(Normal). We will then say that K or F is in the domain of attraction of the normal distribution. [We require one specific an 0 having lim (an /an+1 ) < ∞) in (10) and in (15); and the choices an = /n or an = n /n turn out to be the most useful.] The following are equivalent: (6)

σ ˜ 2 (·)

is slowly varying at zero.

(7)

V|X| (·)


(8)

V (·)


(9) (10)

r2 (t) ≡ t q2 (t)/˜ σ2 (t) → 0 √ an q(an )/˜ σ (an ) → 0

(11)

2 t K|X| (1 − t)/V|X| (t) → 0

for any one specific an 0 . 2 where V|X| (t) ≡ [0,1−t) K|X| (s) ds .

(12)

2 t K|X| (1 − ct)/V|X| (t) → 0

for all 0 < c ≤ 1 .

(13)

2 t [K+ (t) ∨ K 2 (1 − t)]/V (t) → 0

where V (t) ≡

(14)

for all 0 < c ≤ 1 .

(17)

2 t [K+ (ct) ∨ K 2 (1 − ct)]/˜ σ 2 (t) → 0 √ q( /n) / [ n σ ˜ (1/n)] → 0 √ [q( /n) − q(1/n)] / [ n σ ˜ (1/n)] → 0 √ n [m( /n) − m(1/n)] / σ ˜ (1/n) → 0

(18)

[v( /n) − v(1/n)] / [n σ ñ2 (1/n)] → 0

(15) (16)

as t → 0 .

for all > 0. for all > 0. for all > 0. for all > 0 .

dom(t,t) K

2

(s) ds .

272 (19)

CHAPTER 11. CONVERGENCE IN DISTRIBUTION U (·)

is slowly varying at infinity,

where U (x) ≡

2

dF (y).

|y| dF (y).

(20)

x [M (cx) − M (x)] /U (x) → 0

(21)

R(x) ≡ x2 P (|X| > x)/ U (x) → 0

or x2 P (|X| ≥ x)/ U (x−) → 0 .

(22)

Xn2 /νn →p 1

for a specific νn > 0 with nνn .

(23)

Xn2 / νn →p 1,

(24)

[maxk

(25)

n P (X 2 > c n V|X| (1/n)) → 0 √ xθn / [ n σ ñ ] → 0

(26)

1 n

[|y|≤x]

for νn one of v1n , v˜1n , u1n , v¯1n or u ˜1n = v˜¯1n .

2 Xnk ] / νn →p 0

for νn one of v1n , v˜1n , u1n , v¯1n or u ˜1n = v˜¯1n .

(28)

2 [maxk Xnk ] / n Xn2 →p 0. √ ¯ n | ]/ n Sn →p 0 . [maxk |Xnk − X

(29)

ñ2 →p 1 . Sn2 / σ

(27)

for all c > 0; M (x) ≡

[|y|≤x] y

for all c > 0. for all 0 < θ ≤ 1.

Proof. The equivalence of conditions (6)–(29) was specifically developed earlier (as cited in the previous theorem), where much longer lists of equivalences are given. And theorem 3.1 needed only to tie the CLT into the combined list. As the introduction points out, many of these equivalences are found in the literature. (Be aware that having a different denominator in a condition, or being required to verify it only on some sequence of values, can be very valuable.) 2 √ ¯ Remark 3.1 (Natural parameters) The conclusion n [X n − νn ]/τn →d N (0, 1) for any νn and τn > 0 could be shown (with heavy-duty mathematics) to imply that σ ˜2 (·) ∈ L in the context of theorem 3.1.] Granted this result, the theorem of types (theorem 13.7.1) then gives √ σ (1/n) → 1 and n | νn − µ ˜(1/n) |/˜ σ(1/n) → 0 τn /˜ √ ¯ (30) whenever n[Xn − νn ]/τn →d N (0, 1). √ ¯ Then µ ˜(1/n) and σ ˜ (1/n) could be used, giving n [X σ(1/n) →d N (0, 1). n −µ(1/n)]/˜ ˜ n ) used in (2) allowed us to avoid all that in a natural way. Introducing the Z(a Moreover, statisticians want to use means and standard deviations. 2

A Converse of the CLT Theorem 3.2 (Domain of normal of the normal df ) n attraction √ iid rvs X1 , . . . , Xn , and set Zn = 1 Xk / n. Then: implies

Consider

EX1 = 0 and E(X12 ) < ∞.

(31)

Zn = Op (1)

(32)

EX1 = 0 and Var[X1 ] < ∞

imply

Zn →d N (0, Var[X1 ] ).

3.


273

Proof. (Giné and Zinn) Now,√(32) was established previously. Consider (31). Fix t > 0. Let Zn ≡ n1 k Xk / n. By Giné–Zinn symmetrization of (10.3.10), we have (a)

P (Zn > 2λ) ≤ 2 P (Zn > λ) ;

and thus P (Zn > λ) = Op (1) by our hypotheses. Also, Khinchin’s inequality in exercise 10.3.3 (regarding the Xk ’s as fixed constants) gives (b)

E |Zn | ≥ A1 Xn2

1/2

= c Sn

with Sn2 ≡ Xn2 .

Applying Paley–Zygmund’s inequality 3.4.9 to Zn (conditioned on fixed values of the Xk ’s) for the first inequality and (b) for the second yields 2 2 2 (EZn − t)+ (cSn − t)+ t 2 (c) P (|Zn | > t) ≥ ≥ = c 1 − Sn cSn (E{(Zn )2 })1/2 (d)

≥ (c2 /4) 1[Sn>2t/c] .

Taking expectations across the extremes of this inequality with respect to the Xk ’s gives the bound (e)

P (|Zn | > t) ≥ (c2 /4) P (Sn > 2t/c) .

Thus Sn = Op (1), by combining (e), (a), and the hypothesis. Fix M > 0. The SLLN gives n 2 2 1 (f) 1 Xk 1[Xk2 ≤M ] →a.s. E{X1 1[X12 ≤M ] } . n But →a.s. implies →d . Thus, applying exercise 11.1.3 to the open set (t, ∞) gives n 2 1 1(0,E(X 2 1 2 (g) 1 Xk 1[Xk2 ≤M ] > t) )) (t) ≤ lim n P ( n 1

(h)

[X1 ≤M ]

≤ sup P ( n1 n

n 1

Xk2 1[Xk2 ≤M ] > t),

for each t > 0.

It follows that (i)

sup 1(0,E(X 2 1 2 (t) ≤ sup sup P ( n1 1 [X ≤M ] )) M >0 1 M >0 n n 2 1 ≤ sup P ( n 1 Xk > t)

n 1

Xk2 1[Xk2 ≤M ] > t) .

n

(j)

≤ sup P (Sn2 > t) . n

Since Sn =Op (1), we have Sn2 =Op (1); and this implies that we can specify a t value of t0 in (j) so large that the right-hand side of (j) at t0 is less than 12 . But this implies that for this t0 the indicator function in (i) must equal zero uniformly in M . This means that (k)

sup E(X12 1[X12 ≤M ] ) ≤ t0 .

M >0

But this last supremum equals E(X12 ), and hence we must have E(X12 ) ≤ t0 < ∞. To complete the proof, we must now show that E(X1 ) = 0. Since EX12 < ∞, ¯ n →p EX1 . But the hypothesis that Zn = Op (1) implies that the WLLN√gives X ¯ n = Zn / n →p 0. Combining these gives EX1 = 0. 2 X

274

4


Bootstrapping

Suppose X1 , . . . , Xn are an iid sample from F . Denote the empirical df of the n ≡ (X1 , . . . , Xn ) by Fn (·). This empirical df Fn has mean X ¯ n and sample X ∗ ≡ (X ∗ , . . . , X ∗ ) denote an iid sample from Fn , called the variance Sn2 . Let X n n1 nn ¯ n∗ and Sn∗ denote the mean and the standard deviation of bootstrap sample. Let X the bootstrap sample. Since the moments of Fn exist, we will work with normed ∗ ¯ n | ]/√n Sn ]. summands. The maximum normed summand is Mn∗ ≡ [maxk |Xnk −X (Note that the normed summands always constitute a uan array.) Now consider √ n ) ≡ [ maxk |Xk − X ¯ n | ]/ n Sn . (1) Dn ≡ D(X √ (For all 0 < θ ≤ θ0 ≡ 12 the quantity xθn / n σ ñ in (11.2.2) has value Dn . Note that Dn is formed from the original sample, while Mn∗ is formed from the bootstrap sample. The following theorem is in the spirit of (11.2.4).) (Note, moreover, that 0 ≤ Mn∗ ≤ Dn , while P (Mn∗ = Dn ) ≥ 1 − (1 − 1/n)n → 1 − 1/e > 0 also holds.) The “standardized” rv of theorem 1.2.1(i) is now equal to √ √ ¯ n∗ − X ¯ n ]/Sn , ¯ n∗ − X ¯ n ]/Sn∗ . Z¯n∗ ≡ n [X (2) and also define Tn∗ ≡ n [X Agree that the weak bootstrap holds if Z¯n∗ →d N (0, 1) for the joint probability on Ω × Ω∗n . Agree that the strong bootstrap holds if for a.e. given sequence of values of X1 , X2 , . . . it is true that the conditional distribution of Z¯n∗ on Ω∗n satisfies Z¯n∗ →d∗ N (0, 1). Theorem 4.1 (Bootstrapping) Consider Z¯n∗ in the iid case. (a) The weak bootstrap for Z¯n∗ is equivalent to both Dn →p 0

and/or

σ ˜ 2 (·) ∈ L .

(b) The strong bootstrap for Z¯n∗ is equivalent to both Dn →a.s. 0

and/or

Var[X1 ] = σF2 ∈ (0, ∞) .

Corollary 1 (a) The weak bootstrap holds for Tn∗ whenever Dn →p 0. (b) The strong bootstrap holds for Tn∗ whenever Dn →a.s. 0. Proof. Consider the normality. (b) The actual (11.2.2) requirement [i.e. that n ] is implied by requiring Dn → 0 Dn → 0 for a.e. fixed sequence of values of X a.s. (a) The proof given for (11.2.4) established √ a uniform rate of convergence to normality. Thus whenever θ < /2 and {xθn / n σ ñ } < δ for an appropriate value of δ , we could conclude that FZ¯n − Φ ≤ . Now consider the present √ situation. For each ω ∈ Ω it is true for all 0 < θ ≤ θ0 that the quantity {xθn / n σ ñ } is bounded by Dn , and thus we need only show that for the appropriate value δ we have P (Dn ≤ δ ) > 1 − /2 for all n ≥ (some n ). That is, it suffices to show that Dn →p 0.

4.

BOOTSTRAPPING

275

Consider the converse. Suppose Z¯n∗ →d N (0, 1) for a fixed array (xn1 , . . . , xnn ). The summands are necessarily uan. Thus (11.2.22) (with all θ = 1) gives Dn → 0 for this same fixed array. Thus Dn →a.s. 0 is implied by the strong bootstrap, and Dn →p 0 is implied (by going to subsequences) by the weak bootstrap. Now (11.3.28) and (11.3.6) show that Dn →p 0 is equivalent to σ ˜ 2 (·) ∈ L. And it is known from Kolmogorov’s SLLN in theorem 10.4.1 that Dn →a.s. 0 is equivalent to Var[X1 ] < ∞. Consider the corollary. Use that (10.6.5) implies (10.6.3) (with both r = 2 and r = 1), and use (10.6.26) (with r = 1 and s = 2). Conclude that we analogously have both (3)

Sn∗ /Sn →p×p∗ 1 if Dn →p 0

(4)

Sn∗ /Sn →p∗ 1 for a.e. value of (X1 , X2 , . . .) if

Exercise 4.1

and Dn →a.s. 0.

Establish all the details of (3) from theorem 10.6.1.

2

276

5


Bootstrapping with Slowly Increasing Trimming

Let kn and kn denote trimming numbers, with kn ∧ kn → ∞. But suppose the trimming fractions (an ∨ an ) ≡ (kn ∨ kn )/n → 0. Bootstrapping such a trimmed mean from an iid sample always works! Notation 5.1 Let Kn ≡ Fn−1 denote the qf associated with some fixed nondegenerate df Fn . We can always specify that dom(an , an ) ≡ (an , 1 − an ) for any n(an ∧ an ) → ∞. However, if Fn− (0) = 0, then specifying that dom(0, an ) ≡ [0, 1 − an ) is preferable; and if Fn (0) = 1, then specifying that dom(an , 0) ≡ (an , 1] is preferable. So we agree to let an ∧an denote an ∧an , or an , or an according to the ˜ n (·) denote Kn (·) Winsorized outside of dom(an , a ), and let µ ñ scheme used. Let K n 2 ˜ n (ξ) . Then X ˜ nk ≡ K ˜ n (ξnk ) are (unoband σ ñ denote the mean and variance of K ˜ n when ξnk are row-independent Uniform(0, 1) servable) row-independent with qf K n ˜ ˜ ˜ ˜ ˜ 2 rvs. Let Xn ≡ (Xn1 + · · · + Xnn )/n and Sñ2 ≡ 1 (X nk − Xn ) /n. Let √ √ ñ − µ ñ − µ (1) ñ ]/˜ σn and Zˆn ≡ n [X ñ ]/Sñ . Zñ ≡ n [X ˜ n1 − µ ˜ n |3 . Let Fn denote the collection of all dfs Fn for which σ ñ > 0. Let γñ ≡ E|X 2 Theorem 5.1 (Universal studentized CLT) Suppose the trimming fractions satisfy n(an ∧ an ) = (kn ∧ kn ) → ∞. Then uniformly in Fn : √ 3 (2) FZñ − Φ ≤ 9 γñ / n σ ñ ≤ 9/ n(an ∧ an ) → 0 . (3)

P (|Sñ /˜ σn − 1| ≥ ) → 0.

(4)

FZˆn − Φ → 0.

Requiring (an ∨ an ) → 0 guarantees that every nondegenerate df F is eventually in all further Fn . Example 5.1 Let all Fn be Bernoulli(10−10 ). Then n must be huge before σ ñ (an ) > 0. 2 √ 3 ñ is immediate from the Berry–Esseen Proof. That FZñ − Φ ≤ 9 γñ / n σ ˜ n (t)− µ ˜ n (t)− µ theorem. Maximizing |K ñ |1 in the integrand of γñ (but leaving |K ñ |2 to integrate) gives √ 3 √ − γñ / n σ ñ ≤ [ |K + (1 − an ) − µ ñ | ∨ | K+ (an ) − µ ñ | ]/ n σ ñ (5) ≤ 1/ n (an ∧ a n ) → 0 , as claimed. Let qn ≡ K(1 − an ) − K+ (an ), as usual. Moreover, n ñk − µ ñ − µ (a) Sñ2 /˜ σn2 = { [ 1 (X ñ )2 /n]/˜ σn2 } − {(X ñ )/˜ σn }2 ≡ {I2n } − {I1n }2 , where Chebyshev’s inequality gives both P (|I1n | ≥ ) ≤ 1/( 2 n) → 0 and

5.

BOOTSTRAPPING WITH SLOWLY INCREASING TRIMMING n

(b) (c)

P (|I2n − 1| ≥ ) ≤ ≤

1

277

˜ nk − µ E[(X ñ )4 ] (an ∧ an ) qn2 1 ≤

2 n2 σ ñ4

2 n(an ∧ an ) σ ñ2

2 → 0.

2 n(an ∧ an )

Thus (3) holds. Writing √ ñ − µ Zˆn = n (X (d) ñ )/Sñ = Zñ + Zñ (˜ σn /Sñ − 1) , we obtain (4); note that P (An ) ≡ P ([|Zñ | ≤ M ]) can be made uniformly small, even though this set depends on F , while P (|˜ σn /Sñ − 1| ≥ ) → 0 uniformly. 2 n ≡ (Xn1 , . . . , Xnn ) denote an iid sample from the qf Kn , Notation 5.2 Let X 2 ¯ n , Sn , Gn , and Kn (·) denote its sample mean, sample variance, sample and let X ˜ ≡ (X ˜ n1 , . . . , X ˜ nn ) denote third absolute central moment, and sample qf. Let X n the (kn , kn )-Winsorized sample, for integers kn and kn . Let an ≡ kn /n and an ≡ kn /n for kn ∧ kn going to ∞ (here kn ∧ kn will denote either kn ∧ kn , or kn , or kn as ˜ n denote the sample mean, ˜ n , Sñ , G ˜ n , and K in the scheme of notation 5.1). Let X sample variance, sample third central moment, and sample qf of the population ˜ . Let X ∗ ≡ (X ∗ , . . . , X ∗ ) denote the iid bootstrap sample from K ˜ n (·), and X n n1 nn n ∗ 3 ∗2 2n∗ and S ∗ let X n be the sample mean and sample variance of Xn . Let Pn denote the bootstrap probability distribution. Now define the statistics of interest √ √ 2∗ ≡ n [ X 2∗ − X ˜ n ]/Sñ 4∗ ≡ n [ X 2∗ − X ˜ n ]/S 2∗ . (6) Z and Z n n n n n [We saw in the previous section that the sample mean and sample variance Xn∗ and √ ¯ n ]/Sn Sn∗2 of an iid bootstrap sample from Kn are such that Z¯n ≡ n [ Xn∗ − X satisfies the strong (or the weak) bootstrap if and only if Var[X] ∈ (0, ∞) (or F ∈ D(Normal)). But next we see the glories of Winsorizing! Winsorizing does do what Winsorizing was supposed to do. The bootstrap always works, provided that we just trim a bit.] 2 Theorem 5.2 (Universal bootstrap CLT) Suppose the trimming fractions are n such that n(an ∧ an ) → ∞ in the context of notation 5.2. Then uniformly in all X n , conditional on X n , for which Sñ > 0 we have that for a.e. X √ 3 ˜ n / n Sñ ≤ 9/ n(an ∧ an ) = 9/ kn ∧ kn → 0 , (7) FZ2∗ − Φ ≤ 9 G n 2∗ − Sñ |/Sñ ≥ X n ) → 0 , (8) P∗n (|S n (9)

FZ4∗ − Φ → 0,

(10)

lim Sñ > 0 if we also specify that (an ∨ an ) → 0 , with F nondegenerate .

Proof.

n

This is immediate from the previous theorem.

2

278


Remark 5.1 If we knew how to Winsorize correctly in theorem 5.1, it would be a useful theorem. The point is, we always know how to Winsorize correctly in the bootstrap of theorem 5.2. But should we instead do bootstrap sampling from the empirical qf Kn itself, ˜ n , and then Winsorize this sample? No! Sampling from K ˜ n gives rather than K us the analog of theorem 5.1, while sampling from Kn (it can be shown) does not. ˜ of (7.10.33) provided Sampling from Kn does work for any qf K in in the class D that (kn ∧ kn )/ log2 n → 0 with an ∨ an → 0 (as can be shown by the methods of chapter 16). 2 Let ξ1 , ξ2 , . . . be iid Uniform(0, 1) rvs. Let Xnk ≡ Fˇn−1 (ξk ) with −1 −1 1(0,1/2)(t) + (1 − t) ∨ (1/n) 1[1/2,1) (t). Fˇn−1 (t) ≡ − t ∨ (1/n) ¯ n ≡ n Xnk /n. Compute Vn , as well as the higher Let Vn ≡ Var[Xnk ], and let X k=1 4 moments E|Xnk |3 and EXnk . √ ¯ √ (a) Show that Zn ≡ n X n / Vn →d N (0, 1) by verifying the Lindeberg condition. (b) What conclusion does the Berry–Esseen theorem imply for Zn ? n 2 /n satisfies Xn2 /Vn →p 1. (c) Show that Xn2 ≡ k=1 Xnk √ ¯ 2 1/2 → N (0, 1). (d) Of course, this immediately implies that Tn ≡ n X n /( Xn ) d 2 (e) Show that (E|Xnk |) /Vn → 0. Exercise 5.1

Exercise 5.2

Formulate and solve another example in the spirit of exercise 5.1.

n ¯ n )2 /(n − 1) satisfies Sn2 /˜ Exercise 5.3 Verify that Sn2 ≡ k=1 (Xnk − X σn2 →p 1, and do it by verifying a Lindeberg type condition in the context of theorem 11.2.1.

6.

EXAMPLES OF LIMITING DISTRIBUTIONS

279

6

Examples of Limiting Distributions

Example 6.1 (Delta method) (a) Suppose cn [Wn − a] →d V where cn → ∞, and suppose g(·) is differentiable at a (recall (4.3.7)). Then (as in the chain rule proof of calculus) immediately (1)

cn [ g(Wn ) − g(a)] =a {g (a)} · cn [ Wn − a] →d {g (a)} · V .

[Recall that Un =a Vn means that Un − Vn →p 0.] (b) The obvious vector version of this has the conclusion (2)

n ) − g(a )] =a { ∇g(a)} · cn [ W n − a ] . cn [ g(W

2

Example 6.2 (Asymptotic normality of the sample variance) Suppose the rvs X1 , . . . , Xn are iid (µ, σ2 ) with µ4 ≡ EX 4 < ∞ and σ2 > 0. Then (3)

Sn2

n 1 ¯ 2 = (the sample variance). ≡ (Xk − X) n−1 k=1

For a useful phrasing of conclusions, define Zk ≡ (Xk − µ)/σ ∼ = (0, 1) , (4)

Yk ≡ Zk2 = [(Xk − µ)/σ]2 ∼ = (1, µσ44 − 1) = (1, 2 (1 + γ2 /2)) ,

where γ2 ≡ (µ4 − 3σ 4 )/σ4 ≡ (the kurtosis) measures the tail heaviness of the distribution of X. We will show that as n → ∞ both √ √ (5) n [Sn2 − σ 2 ] √21σ2 =a √12 n [ Y¯n − 1] →d N (0, 1 + γ2 /2) and (6)

√

n [Sn − σ] σ2 =a

Proof.

n [ Y¯n − 1] →d

√

2

2 N (0, 1 + γ2 /2) .

Now, 2 Sn σ2

(a)

√

=

=

1 n−1

n

n 1 n−1 n

k=1

¯ n )2 (Xk −X σ2

n k=1

=

n 1 n−1 n

n k=1

¯

[( Xkσ−µ ) − ( Xnσ−µ )]2

[Zk − Z¯n ]2 .

Then note from (a) that √ √ 6 √ n (Sn2 − σ 2 ) n 1 5√ ¯ n √ √ (b) = n ( Yn − 1) − n Z¯n2 − √ 2 n−1 2 2σ 2 (n − 1) (c)

=a

√1 2

√

n ( Y¯n − 1) →d N (0, Var[ Y ]/2).

2

280


Exercise 6.1 (a) Determine the joint limiting distribution (of appropriately nor¯ n and Sn ≡ [ n (Xk − X ¯ n )2 /(n − 1)]1/2 in the iid case. (Consider the malized) X 1 representation of Sn in (6) as a normed sum of the rvs Yk .) What condition on the moments is required for the result? (b) Determine the asymptotic distribution of the (appropriately normalized) rv ¯ n (the coefficient of variation) in the iid case. Obtain a similar representaSn /X tion. (You may suppose that the Xk ’s now take values in (0, ∞).) (c) Note that (6) provides a stronger conclusion than just asymptotic normality, in that it forms a superb starting point for the further asymptotic work in (a) and (b). Note also (13) below. ¯ n and Sn2 ) Let X1 , . . . , Xn be iid. Note/show Exercise 6.2 (Moments of X 2 ¯n ∼ that (provided that µ or σ is well-defined) X = (µ, σ2 ) and ESn2 = σ 2 . Show that (provided that µ3 or µ4 is well-defined): (7)

¯ n − µ)3 = E(X

µ3 n2

(8)

¯ n − µ)4 = E(X

3 σ4 n2

(9)

Var[Sn2 ] =

1 n

.

{ µ4 −

+

µ4 −3 σ4 n3

n−3 n−1

σ4 }

. and

¯ n , Sn2 ] = Cov[X

1 n

µ3 .

√ ¯2 2 2 Exercise 6.3 If X1 , . . . , Xn are iid (µ, σ2 ), then n [X n − µ ] →d 2 µ × N (0, σ ) 2 ¯ (by the delta method). But what is the asymptotic distribution of nXn when µ = 0? √ Exercise 6.4 (Two sample problems) If m (Sm − θ) →d N (0, 1) as m → ∞ √ and n (Tn − θ) →d N (0, 1) as n → ∞ for independent rvs Sm and Tn , then mn m+n (Sm − Tn ) →d N (0, 1) as m ∧ n → ∞. [Hint: Consider what happens under the additional hypothesis that λmn ≡ m/(m + n) → λ ∈ [0, 1]. Then show that this hypothesis can be dropped. This is useful for the two-sample t-test and F -test.] Exercise 6.5 (Simple linear rank statistics) Let TN ≡ √1N N 1 ci aπ(i) , where (π(1), . . . , π(N )) achieves each of the N ! permutations of (1, . . . , N ) with probability 1/N ! . Here, the ci and ai are constants. Show that: N Eaπ(i) = a ¯N , Var[aπ(i) ] = σa2 ≡ N1 1 (ai − a ¯N )2 , (10) Cov[ aπ(i) , aπ(j) ] = − N1−1 σa2 for all i = j . (11)

√1 N

[Hint. Var[

E TN = c¯N · a ¯N ,

N 1

Var[TN ] =

N N −1

σc2 σa2 .

aπ(i) ] = 0, as in (9.1.8).]

¨n) Example 6.3 (The median X Let X1 , . . . , Xn be iid with df F (· − θ), for 1 some θ ∈ R, where F (0) = 2 and F (0) > 0 exists and exceeds zero. [Thus ¨ n denote Xi ∼ = θ + i , for i ’s that are iid F (·) with a unique median at 0.] Let X any sample median. (a) Then √ ¨ n − θ] →d Z1 ∼ (12) Z1n ≡ n [X = N (0, 4[F 1(0)]2 ) .

6.


281

√ ¯ 2 ∼ (b) If F (·) ∼ = (µ, σ2 ) also, then Z2n ≡ n [X n − (µ + θ)] →d Z2 = N (0, σ ). In fact, the joint distribution of (Z1n , Z2n ) →d (Z1 , Z2 ), where the covariance of the limiting distribution is given by (13)

E{[ − µ] × [1[>0] − 1/2]/F (0)} .

2

Proof. The “mid-value”, or (population) median, of the distribution of the Xk ’s is any value θ satisfying P (X ≤ θ) ≥ 12 and P (X ≥ θ) ≥ 12 ; in the present case its value is θ. The ordered values of the Xk ’s are denoted by Xn:1 ≤ · · · ≤ Xn:n , and ¨ n is defined to be Xn:m or any are called the order statistics. The sample median X point in the interval [(Xn:m , Xn:m+1)] according as n equals 2m + 1 or 2m is odd or even. Let Xnl and Xnr denote the left and right endpoints of the interval of possible sample medians (of course, Xnl = Xnr =√Xn:m+1 if n = 2m + 1 is odd). n By the event equality [Xnr − θ ≤ y/ n ] = [ 1 1[Xi −θ≤y/√n ] > n/2], we have √ √ n P ( n [Xnr − θ] ≤ y) = P (Xnr − θ ≤ y/ n ) = P ( 1 1[Xi −θ≤y/√n ] > n/2) (14)

= P ( n1

n

= P ( √1n (a)

1[i ≤y/√n ] > 1/2)

1

n 1

since i ≡ Xi − θ ∼ = F (·)

{1[i >y/√n ] − 12 } < 0)

= P (Wn + An < 0) .

Here (15)

Wn ≡

√1 n

n 1

{1[i>0] − P ( i > 0)} →d Z1 ∼ = N (0, p(1 − p))

with p ≡ P ( i > 0) = 12 , and (as we will now show) (16) An ≡ √1n n1 {1[i >y/√n ] − 1[i >0] } →p −y F (0) . Note that all terms in the summation in An are of the same sign. Then (b)

An →p −y F (0±)

according as y > 0 or y < 0,

since EAn = (c)

√

√ √ √ n [ P ( > y/ n ) − P ( > 0)] = −y [F (y/ n ) − F (0)]/ [y/ n ]

→ −y F (0±)

[provided only that both F (0±) exist]

and (d)

√ √ Var[An ] = [F (y/ n ) − F (0)]{1 − [F (y/ n ) − F (0)]} → 0 .

Thus Wn + An →d Z1 − yF (0) via (15), (16), and Slutsky. By →d we then have √ P ( n [Xnr − θ] ≤ y) = P (Wn + An < 0) (e)

→ P (Z1 − yF (0) ≤ 0) = P (Z1 /F (0) ≤ y) for each y.

282


That is, √

∼ N (0, p(1 − p)/ [F (0)]2 ) . n [Xnr − θ] →d Z1 /F (0) = √ In like fashion, [ n [Xnl − θ] ≤ y] = [ n1 1[i ≤y/√n ] ≥ n/2], so that (f)

(g)

√ P ( n [Xnl − θ] ≤ y) = P ( n1 1[i ≤y/√n ] ≥ n/2) = P (Wn + An ≤ 0) .

Thus the same argument as before gives √ (h) n [Xnl − θ] →d Z1 /F (0) . ¨ n in between, via Now we squeeze the general X √ √ √ ¨ n − θ] ≤ y) ≤ P ( n [Xnl − θ] ≤ y) , (i) P ( n [Xnr − θ] ≤ y) ≤ P ( n [X where both ends converge to P (Z1 /F (0) ≤ y). This completes the proof. In the course of this proof we have demonstrated the events √ ¨ [ ω : n (X n (ω) − θ) ≤ y ] and [ ω : Wn (ω) ≤ y F (0) ] (17) differ by a probabilistically negligible amount. For the joint result, apply (17) and the multivariate CLT to (Wn , Z2n ).

2

Exercise 6.6 (Joint asymptotic normality of quantiles) For 0 < p < 1, the pth quantile xp of F is now defined as xp ≡ F −1 (p). (a) Show that if F has a derivative F (xp ) > 0 at xp , then √ (18) n [ Xn:[np] − xp ] →d N (0, p(1 − p)/ [F (xp )]2 ) as n → ∞. (b) Establish joint normality for pi and pj quantiles, where the covariance matrix of the asymptotic distribution has (i, j)th entry σij ≡ [(pi ∧ pj ) − pi pj ]/ [F (xpi )F (xpj )] . Write out the analogue of (17), and use it. Exercise 6.7

What happens when you try to apply (12) to:

(a) F (x) = exp(−|x|)/2 ? or (b) F (x) = Show that

√

1 2

1[−1,0) (x) +

1 4

1[0,2] (x) ?

¨ n − θ] →d (a rv) in both cases. (In case (b) it is not normal.) n [X

Exercise 6.8 Use (11.1.13) to verify (12) both for n = 2m + 1 odd and for ¨ n ≤ Xn:m+1 , n = 2m even. Since Xn:m ≤ X √ n ¨ n − θ) ≤ z) ≤ P ( n 1[ ≤y/√n ] ≥ n ]) . P ( 1 1[i ≤y/√n ] > n2 ]) ≤ P ( n (X 1 i 2 (The right side is an equality when n is odd.)

6.


283

Exercise 6.9 Consider (with hypothesis as weak as possible) the asymptotic distribution of (appropriately normalized forms of) both n n 1 1 ¯ ¨ (19) and 1 |Xk − Xn | 1 |Xk − Xn | n n for iid samples X1 , . . . , Xn from a df F (µ, σ2 ) having median ν. Exercise 6.10 Let X1 , X2 , . . . be independent with Xk ∼ = Uniform(−k, k). Then establish that Sn /σn →d N (0, 1). Exercise 6.11 Determine the limiting distribution of n 2 n 3/2 k=1 (Xk − X2n+1−k ) k=1 (Xk − X2n+1−k )/n where X1 , X2 , . . . are iid (µ, σ2 ) rvs. Exercise 6.12 Determine the .95-quantile of the limiting distribution of √ n −Xk / n , k=1 Uk for independent rvs with Xk ∼ = Double Exponential(0, 1) and Uk ∼ = Uniform(0, 1). Example 6.4 (Weighted sums of iid rvs) Suppose that rvs Xn1 , . . . , Xnn are row independent and iid (µ, σ2 ). Let cn ≡ (cn1 , . . . , cnn ) for n ≥ 1, and set n n 2 c¯n ≡ k=1 cnk /n and σcn ≡ k=1 (cnk − c¯n )2 /n ≡ SScc /n . Suppose we have the uan condition (20)

0 1 max1≤k≤n (cnk − c¯n )2 /n max1≤k≤n (cnk − c¯n )2 Dc ≡ D(cn ) ≡ = →0 n 2 σcn ¯n )2 k=1 (cnk − c

as n → ∞. Then (21)

n (cnk − c¯n ) Xnk − µ √ →d N (0, 1). n σcn σ

k=1

[We need not center the cnk ’s if the Xnk ’s have mean 0.] Proof. Without loss of generality, set µ = 0. Now, Lindeberg’s condition holds, as we demonstrate via 2 n (cnk − c¯n ) 2 √ x dF (x) √ σ n σcn [|cnk −¯ cn | |x|≥ σ n σcn ] k=1

(a)

≤ σ −2 · 1 ·

since Dc → 0 and

√ [|x|≥σ/ Dc ]

x2 dF (x) → 0,

x2 dF (x) < ∞.

2

The preceding example is useful in regression situations, and in connection with the projection techniques of section 9.3. See section 14.2.

284


Exercise 6.13 (Monte Carlo estimation) Let h : [0, 1] → [0, 1] be a measurable 1 function, and let θ = 0 h(t) dt. Let X1 , Y1 , X2 , Y2 , . . . be iid Uniform(0, 1) rvs. Define two different estimators of θ by T1n ≡

n k=1

h(Xk )/n

and

T2n ≡

n

k=1 1[Xk ≤h(Yk )] /n.

(a) Show that both T1n and T2n are unbiased estimators of θ, and determine which estimator has the smaller variance. Indicate how the variance of each estimator could be estimated. (b) Determine the joint asymptotic distribution of appropriately normalized forms of T1n and T2n . Exercise 6.14 (An analogue of the student-t statistic based on quartiles) Let X1 , . . . , Xn be iid with df F (·). Let m ≡ [n/4], for the greatest integer [·]. Let Un ≡ Xn:m ,

¨ n ≡ (the median), Vn ≡ X

Wn ≡ Xn:n+1−m

denote the quartiles and median of the sample. Make appropriate assumptions regarding F (·). (a) Determine the joint asymptotic distribution of √ √ ( n [Vn − ν], n [Wn − Un − µ] ) for appropriately defined µ and ν. (b) Simplify this if the Xi are symmetrically distributed about 0. (c) Determine the asymptotic distribution under symmetry of the (student-t like) statistic (formed from three sample quantiles) Tn ≡

√

n [Vn − ν]/ [Wn − Un ].

Exercise 6.15 Let the Xk ’s be iid Cauchy(0, 1) in the previous exercise. (d) Evaluate F (x) = P (X ≤ x) for x ∈ R. (e) Solve p = F (x) for xp ≡ F −1 (p), when 0 < p < 1. (f) Express your answers to (b) and (c) of the previous exercise in the present context. Exercise 6.16 (Poisson estimation) Let X1 , . . . , Xn be iid Poisson(θ). ¯ (a) Reasonable estimators of θ include the nsample mean T1n ≡ Xn , the sample n 2 variance T2n ≡ Sn , and T3n ≡ 1 kXk / 1 k (which puts more emphasis on the more recent observations). Evaluate lim Var[Tin ] for i = 1, 2, 3. ¯ n2 − X ¯ n /n and T5n ≡ Xn2 − X ¯ n are both unbiased estimators (b) Verify that T4n ≡ X of θ2 . Evaluate lim Var[Tin ] for i = 4, 5. √ ¯ 2 ¯ (c) Determine the asymptotic distribution of Dn ≡ n [ X n − Sn ]/ Xn when the observations really do follow a Poisson distribution. (d) What is the asymptotic distribution of Dn when the observations Xk actually follow a NegBiT(r, p) distribution?

6.


285

Theorem 6.1 (Doeblin’s CLT for a random number of rvs) Consider iid (0, σ2 ) rvs X1 , X2 , . . . . Let {νn }∞ n=1 be integer-valued rvs such that the proportion νn /n →p c ∈ (0, ∞) as n → ∞. Let Tn ≡ X1 + · · · + Xn denote the total. Then √ (22) Tνn / νn →d N (0, σ2 ). [Note that νn and X1 , X2 , . . . need not be independent.] Proof.

Now, ,

% $ Tνn − T[cn] T[cn] [cn] + (23) . νn [cn] [cn] Note that T[cn] / [cn] →d N (0, 1) and [cn]/νn = paragraph we will show that (a) (Tνn − T[cn] )/ [cn] →p 0. Tν √n = νn

[cn] c cn νn /n

→p 1. In the next

The theorem then follows from Slutsky’s theorem. We now let An ≡ [|Tνn − T[cn] |/ [cn] > ], and note that (b) P (An ) = ∞ k=1 P (An ∩ [νn = k]) = P (An ∩ [νn = k]) + {k:|k−[cn]|≤3 cn}

(c)

≡

+

1

P (An ∩ [νn = k])

{k:|k−[cn]|>3 cn}

2.

Since νn /[cn] →p 1, for n sufficiently large we have νn 3 3 cn − 1| >

2 ≤ P (|νn − [cn]| > cn) ≤ P | [cn] [cn] νn (d) ≤P | − 1| > 3 < . [cn] Also, applying Kolmogorov’s inequality twice, ≤ P max |T − T | >

[cn] k [cn] 1 3 |k−[cn]|≤ cn

≤P

max 3

[cn]≤k≤ cn

|Tk − T[cn] | > [cn]

+P

max 3

[cn]− cn≤k≤[cn]

|Tk − T[cn] | >

[cn]

[cn]+3 cn

≤2

Var[Xk ]/ 2 [cn] ≤ 2 3 cnσ2 / 2 [cn]

k=[cn]+1

(e)

≤ 4 σ2

for all n sufficiently large.

Combining (d) and (e) into (c) shows P (An ) → 0, as required.

2

286


Exercise 6.17 (Sample correlation coefficient Rn ; Cramér and Anderson) Let us suppose that Xi 0 1ρ are iid , for 1 ≤ i ≤ n, Yi 0 ρ1 √ and that the Σ below has finite entries. Consider n [Rn − ρ], where Rn is the sample correlation coefficient. Thus Rn ≡ SSXY /{ SSXX SSY Y }1/2 for the sums n ¯ n )(Yi − Y¯n ), etc. of squares SSXY ≡ 1 (Xi − X (a) Reduce the case of general means, variances and covariances to this case. (b) Note that  1 n    √ 1 (Xi Yi − ρ) n Z1  √1 n (X 2 − 1)  (24) = N (0, Σ)  n 1  →d  Z2  ∼ i n 2 Z √1 3 (Y − 1) i 1 n

with



 E(X 2 Y 2 ) − ρ2 E(X 3 Y ) − ρ E(XY 3 ) − ρ (25) Σ ≡  E(X 3 Y ) − ρ EX 4 − 1 E(X 2 Y 2 ) − 1  . 3 2 2 E(XY ) − ρ E(X Y ) − 1 EY 4 − 1 √ ∼ (c) Then show that n [Rn − ρ] →d Z1 − ρ2 Z2 − ρ2 Z3 √ = N (0, τ 2 ), and evaluate τ 2 . (d) Show that when X and Y are independent, then n [Rn − ρ] →d N (0, 1). (e) If the (Xi , Yi ) are jointly normal, show that   1 + ρ2 2 ρ 2 ρ (26) Σ =  2ρ 2 2 ρ2  . 2 ρ 2 ρ2 2 Then simplify the expression for τ 2 and obtain √ (27) n [Rn − ρ] →d N (0, (1 − ρ2 )2 ) . √ (f) Show that n [g(Rn ) − g(ρ)] →d N (0, 1) for g(t) ≡ 12 log( 1+t 1−t ). √ ρ (g) Approximating n − 3 [g(Rn ) − g(ρ) − 2(n−1) ] by N (0, 1) is excellent. (See example 14.6.1.) ¯ n , Yi − Y¯n ] = (1 − 1 ) Cov[Xi , Yi ]. (h) Show that Cov[Xi − X n Exercise 6.18 (Extreme value quantiles) Let X and X1 , . . . , Xn be iid with the Weibull(α, β) density f (x) = (βxβ−1 / αβ ) exp(−(x/α)β ) on x ≥ 0. Now, (X/α)β ∼ = Exponential(1), and thus Y ≡ log X satisfies Y ∼ =ν+τ W

where ν ≡ log α and τ ≡ 1/β

and W has the Extreme Value density for minima given by exp(w−ew ) on (−∞, ∞). Let Yn:1 ≤ · · · ≤ Yn:n denote the order statistics of the rvs Yk ≡ log Xk . First, let 0 < p1 < p2 < 1, and then define Un ≡ Yn:[np1 ] and Vn ≡ Yn:[np2 ] . We seek values p1 and p2 such that √ n (V − ν) n √ →d N (0, Σ). n (Vn − Un − τ )

6.


287

(a) Let 0 < p < 1. Evaluate yp ≡ FY−1 (p), fY (yp ) and p(1 − p)/fY2 (yp ). (b) Determine values of p1 and p2 that achieve the objective. (c) Establish the claimed asymptotic normality, and evaluate Σ both symbolically and numerically. Exercise 6.19 (Estimating a common normal mean) Consider independent rvs X1 , . . . , Xm and Y1 , . . . , Yn from N (θ, σ2 ) and N (θ, τ 2 ). When γ ≡ σ2 /τ 2 is known, the unbiased estimator of θ that has minimum variance (for all possible values of the parameters within this model) is known to be θˆ ≡ θˆmn ≡

¯ m +γ·nY ¯n mX m+γ·n

.

2 ˆ ≡ α ˆmn (SX , SY2 ) depend only on the Define α ≡ αmn ≡ m/(m +γ · n). Let α m 2 2 ¯ two sample n variances SX ≡ 1 (Xi − Xm ) /(m − 1) = SSXX /(m − 1) and also SY2 ≡ 1 (Yj − Y¯n )2 /(n − 1) = SSY Y /(n − 1), and suppose that α ˆ is a rv with values in [0, 1]. We hypothesize that

α ˆ2 / α2 →p 1

as m ∧ n → ∞.

(All limits below are to be taken as m ∧ n → ∞.) Then define ¯ n + (1 − α) θˆ ≡ θˆmn = α ˆX ˆ Y¯n , 2 ≡ vo2 ≡ vomn

1 m

α2 σ2 +

2 Vˆ 2 ≡ Vˆmn ≡

1 m

2 α2 SX +

α ˜≡α ˜ mn ≡

m 2 /S 2 m+nSX Y

1 n

(1 − α)2 τ 2 ,

1 n

(1 − α)2 SY2 ,

2 = mSY2 /(mSX + nSY2 ) .

2 Note that γ˜ ≡ γ˜mn ≡ SX /SY2 →p 1, and α ˜ 2 / α2 →p 1 is indeed true. ˆ (a) Show that Eθ = θ. (b) Show that (θˆ − θ)/ vo →d N (0, 1). (c) Show that |Vˆ 2 − vo2 |/ vo2 →p 0. (d) Thus (θˆ − θ)/ Vˆ →d N (0, 1). 2 ˆ in terms of Eˆ (e) Evaluate v2 ≡ vmn ≡ Var[θ] α2 and E(1 − α) ˆ 2. 2 2 2 (f) Attempt to show that |Vˆ − v |/ v →p 0 (view it as a final step to claim the normality in (d), if one first shows that (θˆ− θ)/ v →d N (0, 1)). [The author believes that you will be required to assume that α/ ˆ α →L2 1 in order to achieve this end.] (There is a lesson here! Note (g).) (g) Does α/ ˜ α →L2 1?

Exercise 6.20 (Exponential estimation) Let X1 , . . . , Xn be iid Exponential(θ). ¯ n . Another unbiased esThe minimum variance estimator of θ is known to be X ¯ n / Γn (1 + 1/n), where G ¯ n ≡ ( n Xk )1/n denotes the getimator of θ is Tn ≡ G 1 ometric mean of the observations. Evaluate the limiting ratio of the variances ¯ n ]/ Var[Tn ]. lim Var[X Exercise 6.21 Let X1 , . . . , Xn be iid Poisson(λ). Show the moment convergence ¯ n − λ|3 → E|N (0, 1)|3 . E|X

288

7


Classical Convergence in Distribution

Definition 7.1 (Sub-dfs) (a) Suppose we have rvs Xn ∼ = Fn and X. We now wish to allow the possibility that X is an extended rv. In this case, we assume that H is a sub-df (we will not use the notation F in this context), and we will write X ∼ = H. The interpretation in the case of an extended rv X is that H(−∞) = P (X = −∞), H(x) = P (−∞ ≤ X ≤ x) for all −∞ < x < ∞, and 1 − H(+∞) = P (X = +∞). The set CH of all points at which H is continuous is called the continuity set of H. (b) If Fn (x) → H(x) as n → ∞ at each x ∈ CH of a sub-df H, then we say that Xn (or Fn ) converges in sub-df to X (or H), and we write Xn →sd X (or Fn →sd H) as n → ∞. [What has happened in the case of sub-df convergence is that amounts H(−∞) and 1 − H(+∞) of mass have escaped to −∞ and +∞, respectively.] (c) We have agreed that Fn , F , etc. denote a bona fide df, while Hn , H, etc. may denote a sub-df. Thus Fn →d F (with letter F rather than letter H) will still imply that the limit is necessarily a bona fide df. [The next definition provides a condition that guarantees (in a totally obvious way, on R at least) that any possible limit is a bona fide df.] Definition 7.2 (Tightness) A family P of distributions P on R is called tight if for each > 0 there is a compact set (which for one-dimensional rvs is just a closed and bounded set) K with (1)

P (K ) = P (X ∈ K ) ≥ 1 −

for all dfs P ∈ P.

Theorem 7.1 (Helly–Bray) If Fn →d F and g is bounded and continuous a.s. F , then the expectations satisfy (2) Eg(Xn ) = g dFn → g dF = Eg(X). Conversely, if (2) holds for allbounded continuous g, then Fn →d F . [Thus Fn →d F if and only if g dFn → g dF for all bounded and continuous g.] Theorem 7.2 (Continuous mapping theorem; Mann–Wald) Suppose that Xn → X and suppose that g is continuous a.s. F . Then g(Xn ) →d g(X). How do we establish that Fn →d F ? We already have the necessary and sufficient condition of Helly–Bray’s theorem 7.1 (or 3.5.1). (We should now recall our definition of the determining class used in the context of the proof of theorem 3.5.1.) We can also show convergence in distribution of more complicated functions of rvs via Mann–Wald’s continuous mapping theorem 7.2 (or 3.5.2) [for example, Zn →d Z implies that Zn2 →d Z 2 ∼ = χ21 ]. We introduced the concept of tightness above to trivially guarantee that any possible limit is necessarily a bona fide df. This becomes more important in light of the next theorem. Theorem 7.3 (Helly’s selection theorem) Let F1 , F2 , . . . be any sequence of dfs. There necessarily exists a subsequence Fn and a sub-df H for which Fn →sd H as n → ∞. If the subsequence of dfs is tight, then the limit is necessarily a bona fide df.

7.

CLASSICAL CONVERGENCE IN DISTRIBUTION

289

Corollary 1 Let F1 , F2 , . . . be any sequence of dfs. Suppose there is one fixed sub-df H such that for every sd-convergent subsequence {Fn }, we have Fn →sd (this same H). Then the whole sequence satisfies Fn →sd H. [In particular, let F1 , F2 , . . . be any sequence of dfs. Suppose there is a fixed df F˜ such that for every subsequence Fn that converges in distribution, we have Fn →d F˜ for this same F˜ . Then Fn →d F˜ .] Proof. Let r1 , r2 , . . . denote the rationals. Using Bolzano–Weierstrass, choose a subsequence n1j such that Fn1j (r1 ) → (some a1 ). A further subsequence n2j also satisfies Fn2j (r2 ) → (some a2 ). Continue in this fashion. The diagonal subsequence njj converges to ai at ri for all i ≥ 1. [This Cantor diagonalization technique is important. Learn it!] Define H on the rationals via H(ri ) = ai . Now define H on the reals via H(x) = inf{H(ri ) : ri > x}; this H is a sub-df, since it clearly is , takes values in [0, 1], and is right-continuous. Then the diagonal subsequence, we will now refer to it as n , is such that Fn →sd H. Consider the corollary. Fact: Any bounded sequence of real numbers contains a convergent subsequence; and the whole original sequence converges if and only if all subsequential limit points are the same. We showed that every subsequence of Fn contains a further subsequence Fn for which Fn (x) → H(x) for each fixed x ∈ CH . Thus the whole sequence has Fn (x) → H(x) for each x ∈ CH . Thus Fn →sd H. 2 Exercise 7.1 (Convergence of expectations and moments) (a) Suppose Fn →sd H and Fn− (a) → H− (a) and that both Fn (b) → H(b) for some −∞ < a < b < ∞ in CH . Then (3) g dFn → [a,b] g dH for all g ∈ C[a,b] ≡ {g : g is continuous on [a, b]}, [a,b] (4) g dFn → g dH for all g ∈ C0 , where C0 ≡ {g : g is continuous on R and g(x) → 0 as |x| → ∞}. [Actually, g continuous a.s. H suffices in (3) and (4).] (b) Let g be continuous a.s. H. If Fn →sd H, then lim inf |g| dFn ≥ |g| dH. (c) Suppose Fn →d F . If g and h are continuous functions such that |g(x)| → ∞ as |x| → ∞ and |h(x)/g(x)| → 0 as |x| → ∞, then lim sup |g| dFn < ∞ implies h dFn → h dF as n → ∞. (d) If E|Xn |r0 < (some M ) < ∞ for all n, then Fn →d F implies (5)

E|Xn |r → E|X|r and EXnk → EX

Exercise 7.2 (P´ olya’s lemma) (6)

for all 0 < r < r0 and all 0 < k < r0 .

If Fn →d F for a continuous df F , then

Fn − F → 0.

Thus if Fn →d F with F continuous and xn → x, then Fn (xn ) → F (x). Exercise 7.3 (Verifying tightness) Suppose Xn ∼ = Fn . Show that {Fn : n ≥ 1} r is tight if either (a) lim E|Xn | < ∞ for some r > 0, or (b) Fn →d F .

290


Equivalent Definitions of Convergence in Distribution The condition Fn (x) → F (x) can be rewritten as Pn ((−∞, x]) → P ((−∞, x]), and as E1(−∞,x](Xn ) → E1(−∞,x] (X). Thus →d is reduced to computing expectations of the particularly simple function 1(−∞,x] ; but these simple functions have the disadvantage of being discontinuous. Definition 7.3 (Closure, interior, and boundary) The closure of B is defined to ¯ ≡ ∩{C : B ⊂ C and C is closed}, while B0 ≡ ∪{U : U ⊂ B and U is open} is be B ¯ is the smallest closed set called the interior of B. These have the property that B containing B, while B0 is the largest open set contained within B. The boundary of ¯ \ B 0 . A set B is called a P -continuity set if P (∂B) = 0. B is defined to be ∂B ≡ B (These definitions are valid very generally, not just on R.) Theorem 7.4 (→d equivalencies) Let F, F1 , F2 , . . . be the dfs associated with the probability distributions P1 , P2 , . . . . Let Cb denote all bounded, continuous functions g on R, and then let Cbu denote all bounded and uniformly continuous functions g on R. The following are equivalent: (7)

Fn →d F.

(8)

Fn (x) → F (x)

(9)

Eg(Xn ) = g dFn → g dF = Eg(X) for all g in Cb .

(10)

for all x in a dense set.

g dFn → g dF

for all g in Cbu .

(11)

lim Pn (B) ≤ P (B)

for all closed sets B.

(12)

lim Pn (B) ≥ P (B)

for all open sets B.

(13)

lim Pn (B) = P (B)

for all P -continuity sets B.

(14)

lim Pn (I) = P (I)

(15)

L(Fn , F ) → 0 for the Lévy metric L (see below).

for all (including unbounded) P -continuity intervals I.

Exercise 7.4 That (7)–(10) are equivalent is either trivial, or done previously. Cite the various reasons. Then show that (11)–(15) are also equivalent to →d . Exercise 7.5 (Lévy’s metric) between F and G) (16)

For any dfs F and G define (the 45◦ distance

L(F, G) ≡ inf{ > 0 : F (x − ) − ≤ G(x) ≤ F (x + ) + for all x}.

Show that L is a metric and that the set of all dfs under L forms a complete and separable metric space. Also show that Fn →d F is equivalent to L(Fn , F ) → 0.

7.

CLASSICAL CONVERGENCE IN DISTRIBUTION

291

Convergence of Types Definition 7.4 (Type) When Y ∼ = (X − b)/a for some a = 0, we say that X and Y are of the same type. [Suppose that Xn →d X where X is not degenerate. Then if an → a = 0 and bn → b, we know from Slutsky’s theorem that (Xn − bn )/an →d Y ∼ = (X − b)/a.] Theorem 7.5 (Convergence of types) Suppose that both Xn →d X and Yn ≡ (Xn − bn )/an →d Y , where an > 0 and X and Y are nondegenerate. Then there exists a > 0 and a real b such that an → a, bn → b and Y ∼ = (X − b)/a. Remark 7.1 The classical CLT implies that if X1 , X2 , . . . are iid (0, σ2 ), then √ Sn / n →d N (0, 1). The above theorem tells us that no matter how we normalize Sn , the only possible nondegenerate limits in distribution are normal distributions. Moreover, √ if Sn /an →d (some rv), the limiting distribution can be nondegenerate only if an / n → (some constant) ∈ (0, ∞). Exercise 7.6 Prove theorem 7.5 on the convergence of types. [Hint. Just consider continuity points x, x of the df FX (·) and continuity points y, y of the df FY (·) for which 0 < FX (x) < FY (y) ≤ FY (y ) < FX (x ) < 1. Then for all n large enough one will have x ≤ an y + bn ≤ an y + bn ≤ x .]

Higher Dimensions If X, X1 , X2 , . . . are k-dimensional random vectors with dfs F, F1 , F2 , . . ., then we say that Xn converges in distribution to X if (17)

Fn (x) → F (x)

for all x ∈ CF ,

just as in one dimension. The Helly–Bray theorem, the Mann–Wald theorem, Helly’s selection theorem, and Polya’s lemma all hold in k dimensions; generalizations of the other results also hold. Moreover, if Xn denotes the first j coordinates of Xn , with 1 ≤ j < k, then Xn →d X implies Xn →d X . Exercise 7.7 Prove the k-dimensional Helly–Bray theorem (along the lines of exercise 3.5.2) using Helly’s selection theorem and P´ olya’s lemma. Prove that Xn →d X implies Xn →d X . After reading section 11.8, prove the k-dimensional version of the Mann–Wald theorem. Exercise 7.8

Prove that theorem 7.4 holds in k dimensions.

See also theorem 13.3.2 and theorem 14.1.3 below.

292

8


Limit Determining Classes of Functions

We can approximate the functions 1(−∞,z] (·) to an arbitrary degree of accuracy within various classes of particularly smooth functions. Within these classes of functions we do not have to worry about the continuity of the limiting measure at z, and this will make these classes more convenient. Indeed, section 11.1 considered the highly specialized class H0 of this type. Definition 8.1 (Limit determining class) A set G of bounded, continuous functions g is called a determining class if for any dfs F˜ and F , the requirement ˜ that gdF = g dF for all g ∈ G implies F˜ = F . (b) If Eg(Xn ) → Eg(X) for all g ∈ G implies that Xn →d X, then G is called a limit determining class. Definition 8.2 (Various classes of smooth functions) (i) Let C (let Cb ) [let Cbu ] denote the class of continuous (bounded and continuous) [bounded and also (k) (∞) uniformly continuous] functions on R. Let Cb (let Cb ) denote the subclasses with k (with all) derivatives bounded and continuous. (ii) An extra c on these classes will indicate that all functions vanish outside some compact subset of R. (iii) Let C0 denote the subclass of C that converge to 0 as |x| → ∞. (iv) Let H0 denote the class of all hz, with z real and > 0; here hz, (x) equals 1, is linear, equals 0 according as x is in (−∞, z], is in [z, z + ], is in [z + , ∞) (this class was introduced in the proofs of Helly–Bray’s theorem 3.5.1 and Stein’s lemma 11.1.1). (v) Let G 0 denote the class of all functions ga,b, with a < b and > 0; here ga,b, (x) equals 0, is linear, equals 1 according as x is in (−∞, a − ] ∪ [b + , ∞), is in [a − , a] ∪ [b, b + ], is in [a, b]. Theorem 8.1 (Criteria for →d ; a kinder and gentler Helly–Bray) (i) Let F1 , F2 , . . . be tight. Let G be a limit determining class. (a) If g dFn → (some #g ) for each g ∈ G, then Fn →d F . Moreover, it must be that #g = g dF . (b) Conversely: If Fn →d F , then g dFn → g dF for each g ∈ G. (k) (∞) (ii) Each class C0 , Cb , Cbu , Cb with k ≥ 1, Cb , H0 , and G 0 is both a determining class and a limit determining class. (iii) So, too, if we add an extra subscript c to the various C-classes in (ii). [For some proofs in the literature, functions g with sharp corners are unhandy.] Exercise 8.1

Prove the previous theorem.

Exercise 8.2 (Higher dimensions) Show that the natural extension of each of the results of this section to Rk is valid. Exercise 8.3 Exhibit at least one more limiting determing class.

8.

LIMIT DETERMINING CLASSES OF FUNCTIONS

293

Moments as a Determining Class for a Moment Unique Limit Theorem 8.2 (CLT via moments; Fréchet–Shohat) (a) Suppose F is the unique df having the specific moments µk = xk dF (x), for all integers k ≥ 1. Then Fn →d F whenever µnk ≡ xk dFn (x) → µk ≡ xk dF (x) (1) for all k ≥ 1. (b) Any normal df is determined by its moments. Proof. By the Helly selection theorem we have Fn →sd H for some subsequence n and some sub-df H. However, lim E|Xn |2 < ∞, so that {Fn : n ≥ 1} is tight by Markov’s inequality. Thus H is a bona fide df, and Fn →d H. Also, for all k ≥ 1 k x dF (x) = lim xk dFn (x) by hypothesis k = x dH(x) (a) by exercise 11.7.1(d). Thus xk dH(x) = xk dF (x) for all k ≥ 1; and since only F has these moments, we conclude that H = F . Thus Fn →d F . Moreover, Fn →d (this same F ) on any convergent subsequence n . Thus Fn →d F , by the corollary to theorem 11.7.3. See exercise 7.5 for part (b) of the theorem. 2 In general, moments do not determine a distribution uniquely; thus {xk : k ≥ 1} is not a determining class. This is shown by the following exercise. Exercise 8.4 (Moments need not determine the df; Heyde) log X ∼ = N (0, 1); thus √ 2 fX (x) = x−1 e−(log x) /2 / 2π for x > 0.

Suppose that the rv

For each −1 ≤ a ≤ 1, let Ya have the density function fa (y) = fX (y)[1 + a sin(2π log y)] for y > 0. Show that X and each Ya have exactly the same moments. [Knowing that these particular distributions have this property is not worth much; it is knowing that some dfs have this property that matters.] Though we have just seen that moments do not necessarily determine a df, it is often true that a given df F is the unique df having its particular moments (name them {µk : k ≥ 1}). Here is an “exercise” giving various sufficient conditions. Exercise 8.5 (When moments do determine a df) Suppose any one of the following hold: ∞ −1/(2k) (a) =∞ (Carleman’s condition). 1 µ2k ∞ k (b) in some interval. 1 µ2k t /(2k)! < ∞ (c)

lim |µk |1/k /k < ∞.

294


Then at most one distribution function F can possess this particular set of moments µk = xk dF (x). [Note Feller(1966, pp. 227–228) and/or Breiman(1968, p. 182). Carleman’s condition is actually necessary and sufficient.] The exercise is to show that (c) suffices. Exercise 8.6 Show that the N (0, 1) distribution is uniquely determined by its moments (by appealing to the previous proposition). Summary The methods of this section that establish →d by verifying the moment condition that Eg(Xn ) → Eg(X) for all functions g in some given determining class G can be extended from the present setting of the real line to more general settings. Also, Chapter 13 carefully considers all the results associated with the particular determining class G ≡ {gt (·) ≡ eit· : t ∈ R}. We can let φ(t) ≡ EeitX = Egt (X) for t ∈ R denote the expectations of all of these gt (X)’s; we call this φ(·) the characteristic function of the rv X. Chapter 13 is a specialized study of the characteristic function. Chapter 14 will apply this characteristic function tool to the CLT. 2

Chapter 12

Brownian Motion and Empirical Processes 1

Special Spaces

General Metric Spaces Let (M, d) denote an arbitrary metric space and let Md denote its Borel σ-field (that is, the σ-field generated by the collection of all d-open subsets of M ). Let MB d denote the σ-field generated by the collection of all open balls, where a ball is a subset of M of the form {y : d(y, x) < r} for some x ∈ M and some r > 0; call this the Baire σ-field. [The important concept of weak convergence is best described in the context of metric spaces.] Exercise 1.1 (1)

Now, MB d ⊂ Md , while

MB d = Md

if (M, d) is a separable metric space.

The Special Spaces (C, C) and (D, D) For functions x, y on [0, 1], define the uniform metric (or supremum metric) by (2)

x − y ≡ sup |x(t) − y(t)| . 0≤t≤1

Let C denote the set of all continuous functions on [0, 1]. Then (3)

(C, ) is a complete and separable metric space.

Here C will denote the σ-field of Borel subsets of C; then CB will denote the σ-field of subsets of C generated by the open balls, and C will denote the σ-field generated by the finite-dimensional subsets of C (that is, all πt−1 (Bk ) for which 0 ≤ t1 ≤ · · · ≤ tk ≤ 1 and Bk ∈ Bk ). It can be shown that (4)

C

= CB = C . 295

296

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES

Let D denote the set of all functions on [0, 1] that are right continuous and possess left-hand limits at each point. (In some applications below it will be noted that D is also used to denote the set of all left-continuous functions on [0, 1] that have right-hand limits at each point. This point will receive no further mention. In some cases we will admit to D, and/or to C, only functions X having X(0) = 0, etc. This, too, will receive little, if any, further mention.) Then (5)

(D, ) is a complete metric space that is not separable.

Here D will denote the Borel σ-field of subsets of D, then DB will denote the σ-field of subsets of D generated by the open balls, and D will denote the σ-field generated by the finite-dimensional subsets of D. It can be shown that (6)

D = DB ,

and both are proper subsets of D ,

and moreover, (7)

C ∈D

and

C = C ∩D.

We now digress briefly. The proper set inclusion of (6) caused difficulties in the historical development of √ the theory of empirical processes (note that the uniform empirical process Un = n(Gn − I) takes values in D). To circumvent these difficulties, various authors showed that it is possible to define a metric d on D that has nice properties (see exercise 1.4 below); thus there is a d(·, ·) for which (8)

(D, d) is a complete and separable metric space

whose Borel σ-field Dd satisfies (9)

Dd = D.

Moreover, for all x, xn in D the metric d satisfies (10)

xn − x → 0

implies

d(xn , x) → 0,

while (11)

d(xn , x) → 0 with x ∈ C

implies

xn − x → 0.

The metric d will not be important to us. We are able to replace d by in our theorems; however, we include some information on d as an aid to the reader who wishes to consult the original literature. Exercise 1.2

Verify (3) and (4).

Exercise 1.3 (i) Verify (5). [Hint. For each 0 ≤ t ≤ 1 define a function xt in D by letting xt (s) equal 0 or 1 according as 0 ≤ s ≤ t or t ≤ s ≤ 1.] (ii) Verify (6). [Hint. Consider ∪{Ot : 0 ≤ t ≤ 1} where Ot is the open ball of radius 13 centered at xt .] (iii) Verify (7).

1.

SPECIAL SPACES

Exercise 1.4 (12)

297

Consult Billingsley (1968, chapter 3) to verify (8)–(11) for

d(x, y) ≡ inf{x − y ◦ λ ∧ λ − I : λ ∈ Λ},

where Λ consists of all ↑ continuous maps of [0, 1] onto itself. [Roughly, this metric measures how closely x and a slightly perturbed (via λ) y line up, where too much perturbation is penalized.] Exercise 1.5 (13)

Verify that

C is both -separable and d-separable, viewed as a subset of D.

[We will require the -separability below.] Let q ≥ 0 be positive on (0, 1). For functions x, y on [0, 1] we agree that (14)

(x − y)/q is the · /q-distance between x and y,

when this is well-defined (that is, when x/q and y/q are finite). Exercise 1.6 It is useful to be able to view C∞ ≡ C[0,∞) as a metric space; of course, this denotes the class of all continuous functions on [0, ∞). (We may sometimes require a subclass, such as the one consisting of functions that equal zero at zero; and we will make no further mention of this.) Let C∞ ≡ C[0,∞) denote the finite-dimensional σ-field. Consider (C∞ , C∞ ) = (C[0,∞) , C[0,∞) ). (a) For functions x and y on [0, ∞), define (15)

ρ∞ (x, y) ≡

∞ k=1

2−k

ρk (x, y) , 1 + ρk (x, y)

where ρk (x, y) ≡ sup0≤t≤k |x(t) − y(t)|. Show that (C[0,∞) , ρ∞ ) is a metric space. (b) Show that ρ∞ (x, y) → 0 if and only if ρk (x, y) → 0 for each 0 < k < ∞. (c) Show that (C[0,∞) , ρ∞ ) is a complete and separable metric space. Moreover, the σ-field Cρ∞ of Borel subsets is the same as the σ-field C[0,∞) of finite-dimensional B subsets, as is C∞ . (d) Verify that (D[0,∞) , ρ∞ ) is a complete metric space, and that the Borel σ-field Dρ∞ satisfies C[0,∞) ∈ Dρ∞ and Cρ∞ = Dρ∞ ∩ C[0,∞) . Also, D[0,∞) = DρB∞ is a proper subset of Dρ∞ . (e) Other spaces of continuous and right-continuous functions are analogously treated. They will receive no specific mention.

Independent Increments and Stationarity If T is an interval in (−∞, ∞), then we will write (16)

X(s, t] ≡ X(t) − X(s)

for any s, t ∈ T,

and we will refer to this as an increment of X. If X(t0 ), X(t0 , t1 ], . . . , X(tk−1 , tk ] are independent rvs for all k ≥ 1 and all t0 < · · · < tk in T , then we say that X has independent increments. If X(s, t] ∼ = X(s + h, t + h] for all s, t, s + h, t + h in T with h ≥ 0, then X is said to have stationary increments. If (X(t1 + h), . . . , X(tk + h)) ∼ = (X(t1 ), . . . , X(tk )) for all k ≥ 1, h ≥ 0, and all time points in T , then X is said to be a stationary process.

298

2


Existence of Processes on (C, C) and (D, D)

When dealing with processes, we would like to work with the smoothest version possible. This is the version that best models physical reality. It is important at this point to recall theorem 5.4.2 on the existence of smoother versions of processes. Roughly, if all of the sample paths of a process are shown to lie in a (useful) subset of the current image space, then we can restrict ourselves to that subset. Theorem 2.1 (Existence of processes on (C, C)) Begin with a process X : (Ω, A, P ) → (R[0,1] , B[0,1] , PX ) . Suppose that for some a, b > 0 the increments of X satisfy (1)

E|X(s, t]|b ≤ K · F (s, t]1+a

for all 0 ≤ s, t ≤ 1,

where F is a continuous df concentrated on [0, 1] and F (s, t] ≡ F (t) − F (s). Then there exists an equivalent version Z : (Ω, A, P ) → (R[0,1] , B[0,1] , PZ ) for which (2)

Z : (Ω, A, P ) → (C, C, PZ ),

with Z(t) = X(t) a.s., for each t in [0, 1].

Corollary 1 (Sample path properties) For any 0 < δ < a/b and any > 0, there exists a constant K ≡ K,δ,a,b for which the process Z of (2) satisfies (3)

P ( |Z(s, t] | ≤ K · F (s, t]δ for all 0 ≤ s ≤ t ≤ 1 ) ≥ 1 − .

Proof. Case 1. Suppose that the df F of (1) is F (t) = t on [0, 1]. Let 0 < δ < a/b be fixed. Let λ ≡ (a/b − δ)/2. Define tni ≡ i/2n for 0 ≤ i ≤ 2n and n ≥ 1. For n ≥ 0 define processes Z n : (Ω, A, P ) → (C, C) by letting (a)

Z n(t) ≡ X(tni ) + 2n (t − tni ) [X(tn,i+1 ) − X(tni )]

for tni ≤ t ≤ tn,i+1 ,

for each 0 ≤ i ≤ 2n − 1; thus Z n(·) equals X(·) at each tni and Z n(·) is linear on the intervals between these points. Define (b)

Uni ≡ | X(tn,i−1 , tni ] |

for 1 ≤ i ≤ 2n .

If we define (c)

∆n (t) ≡ Z n(t) − Z n−1(t)

for 0 ≤ t ≤ 1,

then for tn−1,i ≤ t ≤ tn−1,i+1 we have |∆n (t)| ≤ | [X(tn,2i ) + X(tn,2i+2 )]/2 − X(tn,2i+1 ) | (d)

= |X(tn,2i , tn,2i+1 ] − X(tn,2i+1 , tn,2i+2 ]|/2 ≤ [Un,2i+1 + Un,2i+2 ]/2

(e)

≤ [Un,2i+1 ∨ Un,2i+2 ]

for all n ≥ 1. Thus for all n ≥ 1 we have (f)

∆n ≤ Vn ≡ [max1≤i≤2n Uni ] .

2.

EXISTENCE OF PROCESSES ON (C, C) AND (D, D)

299

Let θ > 0 be arbitrary but fixed, and define (g)

pn ≡ P (∆n δ > 2 θ 2−nλ ) ≡ P (|∆n (s, t]|/(t − s)δ > 2 θ 2−nλ for some 0 ≤ s ≤ t ≤ 1).

Recalling (f) shows that (h)

|∆n (s, t]| ≤ 2 Vn

for all 0 ≤ s ≤ t ≤ 1.

Thus (i)

for 2−n ≤ t − s,

|∆n (s, t]|/(t − s)δ ≤ 2 Vn 2nδ

while |∆n (s, t]|/(t − s)δ ≤ [|∆n (s, t]|/(t − s)] (t − s)1−δ ≤ [Vn 2n ] 2−n(1−δ) (j)

for 0 ≤ t − s ≤ 2−n

= Vn 2nδ

(to see this, consider |∆n (s, t]/(t − s)| when s and t are both points in some [tn,i−1 , tni ]). Thus for all n ≥ 1, we have pn ≤ P (2 Vn 2nδ > 2 θ 2−nλ ) ≤ P (Vn > θ 2−n(δ+λ) ) by (g) 2n by (f) ≤ i=1 P (Uni > θ 2−n(δ+λ) ) 2n b ≤ i=1 E Uni /[θ 2−n(δ+λ) ]b by Markov’s inequality ≤ 2n [K2−n(1+a)]/[θ 2−n(δ+λ) ]b

by (1)

= K θ−b 2−n(a−b(δ+λ)) = K θ−b 2−nb(a/b−δ−λ) (k)

= K θ−b 2−nλb

since a/b − δ = 2λ > 0.

Since 0 ≤ t − s ≤ 1, we also have p0 ≡ P ( |Z 0(s, t]| > 2 θ (t − s)δ for some 0 ≤ s ≤ t ≤ 1) ≤ P ( |X(0, 1]| > 2 θ) ≤ E|X(0, 1]|b /(2θ)b (l)

≤ K θ−b = K θ−b 2−0·λ b .

Now, λb = (a − δb)/2 > 0, and so 2−λ b < 1; hence ∞ n=0 pn < ∞. Thus for arbitrarily small θ, we have for m sufficiently large (recall (g) for · δ ) that P (maxm≤k θ) = lim P (maxm≤k≤n Z k − Z mδ > θ) n→∞

≤ lim P (maxm≤k≤n Z k − Z mδ > 2 θ

n

n→∞

for

∞ m+1

2−kλ < 1

≤ lim P (maxm≤k≤n n→∞

≤ lim

n→∞

(m)

≤

n i=m+1

∞

i=m+1

2−kλ )

m+1

pi ≤

k i=m+1

∆i δ > 2 θ

n i=m+1

2−iλ )

P (∆i δ > 2 θ 2−iλ ) ∞ i=m+1

Kθ−b 2−iλ b → 0

as m → ∞,

300


so that (n)

Z n(t) ≡ Z 0(t) +

n k=1

∆k (t)

converges uniformly on [0, 1] for a.e. ω; call the limit function Z(t). Since the uniform limit of continuous functions is continuous, ∞ (o) Z = n=0 ∆n = lim Z n is a continuous function on [0, 1] for a.e. ω. Now, Z = lim Z n, and since Z n equals X at each tni , we have (4)

Z(t ni) = X(tni )

at each tni = i/2n with 0 ≤ i ≤ 2n and n ≥ 0.

Thus all finite-dimensional distributions with diadic rational coordinates are equal. For other t, we pick diadic rationals t1 , t2 , . . . such that tm → t. Then X(tm ) →p X(t) as m → ∞ by (1) and Chebyshev, while Z(t m) →a.s. Z(t) as m → ∞, since Z has continuous sample paths. Thus Z(t) = X(t) a.s. by proposition 2.3.4. By redefining Z ≡ 0 on the null set of (n), we may assume (p)

Z : (Ω, A) → (C, C)

by theorem 5.4.2. So finite-dimensional distributions agree: In particular, we have PZ ([x ∈ C : xt ∈ B]) = PX ([x ∈ R[0,1] : xt ∈ B]) for all sets B ∈ Bk and for all t ∈ [0, 1]k for any k ≥ 1. Case 2. General F . Define (q)

Y(t) ≡ X(F −1 (t))

for 0 ≤ t ≤ 1,

where F −1 (t) ≡ inf{x ∈ [0, 1] : F (x) ≥ t}. Then for 0 ≤ s ≤ t ≤ 1, E|Y(s, t]|b = E|X(F −1 (s), F −1 (t)] |b ≤ K [ F ◦ F −1 (t) − F ◦ F −1 (s)]1+a (r)

= K (t − s)1+a ,

since F ◦ F −1 = I for continuous F by exercise 7.3.2. Now use case 1 to replace Y ¯ : (Ω, A) → (C, C). Then define by an equivalent process Y (s)

¯ )∼ Z = Y(F = X(F −1 ◦ F )

by (q).

Now, F −1 ◦ F (t) = t, unless F (t − ) = F (t) for some > 0; see exercise 7.3.2. But in this case equation (1) shows that ∆X is 0 across that same interval. Thus X(F −1 ◦ F ) ∼ = X. For the corollary, in case 1 we have (using (o) in line 2, (g) and (k) in line 3) P (| Z(s, t] |/(t − s) δ > 2 θ/(1 − 2−λ ) for some 0 ≤ s ≤ t ≤ 1) ∞ ∞ = P (| k=0 ∆k (s, t] |/(t − s)δ > 2 θ n=0 2−nλ for some 0 ≤ s ≤ t ≤ 1) ∞ ∞ ≤ n=0 pn ≤ n=0 K θ−b /2n λ b = K θ−b /(1 − 2−λ b ) (t)

→0

as θ → ∞.

Take K to be an appropriately large value of θ. Use the transformation F −1 again in case 2. 2

2.

EXISTENCE OF PROCESSES ON (C, C) AND (D, D)

301

Exercise 2.1 Prove (2), by simplifying the proof of theorem 2.1 as much as possible with this simpler goal in mind. We merely state an analogous result for the existence of processes on (D, D). Theorem 2.2 (Existence of processes on (D, D))

Consider a process

X : (Ω, A, P ) → (R[0,1] , B[0,1] , PX ) . Suppose that for some K > 0, b > 0, and a > (5)

1 2

E|X(r, s] X(s, t] |b ≤ K · F (r, s]a F (s, t]a

we have for all 0 ≤ r ≤ s ≤ t ≤ 1,

where F is any df concentrated on [0, 1]. Then there exists an equivalent process Z : (Ω, A, P ) → (R [0,1] , B[0,1] , PZ ), which in fact satisfies (6)

Z : (Ω, A, P ) → (D, D, PZ ),

[See Billingsley (1968, pp. 130, 134).]

with Z(t) = X(t) a.s., for each t ∈ [0, 1] .

302

3


Brownian Motion and Brownian Bridge

We define {S(t) : 0 ≤ t ≤ 1} to be a Brownian motion on [0, 1] if S is a normal process having the moment functions (1)

ES(t) = 0

and

Cov[S(s), S(t)] = s ∧ t

for all 0 ≤ s, t ≤ 1.

This covariance function is nonnegative definite (in the sense of section 9.3), and these distributions are consistent; thus Kolmogorov’s consistency theorem shows that the process S exists as a random element on (R [0,1] , B[0,1] ). Modifying this S on a set of measure zero (as in theorem 12.2.1), we may may create a version of S that satisfies (2)

all sample paths of S are continuous functions on [0, 1] that equal 0 at 0.

Thus (as with the smoother realizations of theorem 5.4.2) there is a nice realization of S having smoother paths; that is, (3)

S exists as a process on (C, C).

So, Brownian motion exists as the coordinate map St (ω) ≡ ωt for some distribution P on (Ω, A) = (C, C). This is a more convenient realization of S (than is the one guaranteed by Kolmogorov’s consistency theorem). For either realization (4)

S has stationary and independent increments.

In fact, its sample paths satisfy (5)

P (|S(s, t]| ≤ K (t − s)δ for all 0 ≤ s ≤ t ≤ 1) ≥ 1 −

for some K,δ , for any fixed > 0 and for any fixed 0 < δ < 12 . This follows from theorem 12.2.1 and its corollary, since for any k ≥ 1, (6)

E S(s, t] 2k = [1 · 3 · · · (2k − 1)] (t − s)k

for all 0 ≤ s ≤ t ≤ 1,

and since a/b = (k − 1)/(2k) 12 as k → ∞. (Note that (5) would allow a further application of the smoother realizations theorem using just this smaller subset of such functions in C.) [No appeal has been made to section 1.] Let us now define (7)

V(t) ≡ S(t) − t S(1)

and

U(t) ≡ −V(t)

for 0 ≤ t ≤ 1.

Then both U and V are obviously normal processes on (C, C) and satisfy (5); just observe that V(t) is a simple linear combination of two normal rvs. Moreover, trivial calculations give (8)

EU(t) = 0

and

Cov[U(s), U(t)] = s ∧ t − st

for all 0 ≤ s, t ≤ 1.

Call U a Brownian bridge. And V is also a Brownian bridge. In a similar fashion we establish the existence of Brownian motion on (C∞ , C∞ ). In particular, a Brownian motion on (C∞ , C∞ ) is given by (9)

S(t) = (1 + t) U(t/(1 + t)),

0 ≤ t < ∞.

3.

BROWNIAN MOTION AND BROWNIAN BRIDGE

303

Recall the proposition 10.7.1 LIL result. In section 12.8 we will establish the companion LIL result for Brownian motion that √ lim sup |S(t)|/[ t b(t)] = 1 a.s. (10) (the LIL for S at infinity), t→∞

where b(t) ≡ 2 (1 ∨ log log t). (We will use it in very minor ways in the meantime.) The next exercise similarly defines some additional normal processes. These may provide a useful revisualization device that enables calculation. Exercise 3.1 (Transformations of Brownian motion) Let Z ∼ = N(0, 1) and the Brownian bridges V, U(1) , and U(2) be independent. Fix a > 0. Show that: (11) (12) (13) (14)

S(t) = V(t) + tZ, 0 ≤ t ≤ 1, √ S(at)/ a, 0 ≤ t < ∞,

is a Brownian motion.

S(a + t) − S(a), t ≥ 0, √ √ 1 − a U(1) ± a U(2)



is a Brownian bridge.

(15)

√ 1 Z(t) ≡ [U(1) (t) + U(2) (1 − t)]/ 2, 0 ≤ t ≤ , 2


(16)

U(t) = (1 − t) S(t/(1 − t)),

is a Brownian bridge;

0 ≤ t ≤ 1,

use the LIL at infinity of (10) to show that this U(·) converges to 0 at t = 1. (17)

t S(1/t),

0 ≤ t < ∞,

is a Brownian motion;

apply the LIL of (10) to verify that these sample paths converge to 0 at t = 0. Exercise 3.2 (LIL for S and U at 0) Use (10), (17), and then (7) to show that √ √ lim sup |S(t)|/[ t b(1/t)] = 1 a.s.; and lim sup |U(t)|/[ t b(1/t)] = 1 a.s. (18) t→0

t→0

Exercise 3.3 (Integrals of normal processes are normal rvs) (a) Suppose X is a normal process on (C, C) with mean function m(·) and covariance function Cov(·, ·) that are both continuous on [0, 1]. Let the weight function g(·) also be continuous. Suppose X/q10 = Op (1) for a function q ≥ 0 that is on (0, 12 ] and on [ 12 , 1). Let K(·) be an and left continuous function. Suppose 1 also that 0 q |g| dK < ∞. Show that the integrated process 1 X(t) g(t) dK(t) 0 (19) 1 1 1 ∼ Cov(s, t) g(s)g(t) dK(s) dK(t)) , = N ( m(t) g(t) dt, 0

0

0

provided that both its mean and variance are well-defined. 1 (b) Determine the distribution of 0 U(t) dt. 1 (c) Develop results for 0 S g dK, for appropriate functions g and K. [Hint. (a) The Riemann sums are normally distributed. (b) Perhaps, use (18).]

304


√ Exercise 3.4 Let Z0 , Z1 , Z2 , . . . be iid N (0, 1). Let fj (t) ≡ 2 sin(jπt), for j ≥ 1; these are orthogonal functions. Verify that U(t) ≡ ∞ (20) 0 ≤ t ≤ 1, is a Brownian bridge. j=1 Zj fj (t) / jπ, Thus the process S(t) ≡ U(t) + t Z 0 is a Brownian motion on [0, 1]. Moreover, 1 ∞ W 2 ≡ 0 U2 (t) dt ∼ (21) = j=1 Zj2 /(j 2 π 2 ). This rv has a distribution that is well tabled (the asymptotic null distribution of the Cramér–von Mises statistic). Show that Z is a Brownian motion on [0, 1], where t for 0 ≤ t ≤ 1. Z(t) ≡ U(t) + 0 [U(s)/(1 − s)] ds

Exercise 3.5 (22)

Exercise 3.6 (White noise) (a) Suppose that h and ˜h on [0, 1] are in L2 . View white noise as an operator dS that takes the function h into a rv [0,1] h(t) dS(t) in the sense of →L2 . Define this integral first for step functions, and then use exercise 4.4.5 to define it in general. Then show that [0,1] h(t) dS(t) exists as such an →L2 limit for all h in L2 . (b) In case h has a bounded continuous derivative h on [0, 1], show that 1+ (23) [0,1] h(t) dS(t) ≡ hS| 0− − [0,1] S(t) h (t) dt . ˜ dS(t). (c) Determine the joint distribution of [0,1] h(t) dS(t) and [0,1] h(t) (d) Define [0,1] h(t) dU(t), and obtain the marginal and joint distributions of all three of the rvs in (c) and (d). Exercise 3.7 (Conditional Brownian motion) Let 0 ≤ r < s < t. Determine the conditional distribution of S(s) given that S(r) = y and S(t) = z. Put your answer in a format that allows some insight to be offered. Exercise 3.8 Find the solution V (t) of the stochastic differential equation with V (t) = −kV (t) + σS (t). Determine its covariance function. (Think of a tiny particle suspended in a liquid whose velocity is impeded by the viscosity of the liquid and is additionally subjected to random changes from collisions with particles in the medium.) [Hint. Rewrite the equation first as ekt [V (t) + kV (t)] = σ ekt S (t), then transform it to t V (t) = V (0)e−t + σ 0 e−σ(t−s) dS(s) , and then use integration by parts to give meaning to dS(·).]

4.

STOPPING TIMES

4

Stopping Times

305

We first paraphrase the main result of this section. If we observe a right-continuous process at a random time that depends on the process only through its past, then the result is a rv (that is, it is measurable). Notation 4.1 Let (Ω, A, P ) denote our basic probability space. We suppose that our time set is a linearly ordered set such as [0, 1], [0, ∞), [0, ∞], {0, 1, 2, . . .}, {0, 1, 2, . . . , ∞}. Let X denote a process with such an index set, defined on (Ω, A, P ). We now suppose that the At ’s are an collection of sub σ-fields of A, in that As ⊂ At whenever s < t. If it further holds that each Xt is an At -measurable rv, then we say that the process is adapted to the At ’s. The minimal such collection of σ-fields is the histories σt ≡ σ[Xs−1 (B) : s ≤ t]. Roughly, σt denotes all events for the process up to time t. We let At+ ≡ ∩∞ n=1 At+1/n ; and if At+ = At for all t ≥ 0, then we call the σ-fields At right continuous. Let At− ≡ σ[As : s < t]. Then 2 let A∞ ≡ σ[∪t 0. For some t ≥ 0 we define X(τ + t) − X(τ ) on [τ < ∞] , (1) Y (t) ≡ 0 on [τ = ∞] .

Then Y : [τ < ∞] ∩ Ω, [τ < ∞] ∩ A, P ( · [τ < ∞]) → (D[0,∞) , D[0,∞) ) and (2)

P Y ∈ F [τ < ∞] = P (X ∈ F )

for all F ∈ D[0,∞) .

Moreover, for all F ∈ D[0,∞) and for all A ∈ Aτ , we have

(3) P [Y ∈ F ] ∩ A[τ < ∞] = P ( [X ∈ F ] ) × P A[τ < ∞] . Thus if P (τ < ∞) = 1, then X and Y are equivalent processes and the process Y is independent of the σ-field Aτ . Proof. That Y : (Ω ∩ [τ < ∞], A ∩ [τ < ∞], P (·|[τ < ∞]) → (D[0,∞) , D[0,∞) ) follows from proposition 12.4.3. This proposition and exercise 12.4.1 show that (4)

At ≡ Aτ +t are and right continuous, with Y adapted to the At ’s.

Case 1. Suppose the finite part of the range of τ is a countable subset {s1 , s2 , . . .} of [0, ∞). Let t1 , . . . , tm ≥ 0, let B1 , . . . , Bm be Borel subsets of the real line, and let A ∈ Aτ . Then P ([Y (t1 ) ∈ B1 , . . . , Y (tm ) ∈ Bm ] ∩ A ∩ [τ < ∞]) = = =

k

P ([Y (t1 ) ∈ B1 , . . .] ∩ A ∩ [τ = sk ])

k

P ([X(t1 + sk ) − X(sk ) ∈ B1 , . . .] ∩ A ∩ [τ = sk ])

k

P (X(t1 + sk ) − X(sk ) ∈ B1 , . . .) P (A ∩ [τ = sk ])

= P (X(t1 ) ∈ B1 , . . .) (a)

k

P (A ∩ [τ = sk ])

= P (X(t1 ) ∈ B1 , . . . , X(tm ) ∈ Bm ) P (A ∩ [τ < ∞]),

where the third equality holds as A ∩ [τ = sk ] = (A ∩ [τ ≤ sk ]) ∩ [τ = sk ] is in Ask , and is thus independent of the other event by the independent increments of X.

5.

STRONG MARKOV PROPERTY

309

Putting A = [τ < ∞] in (a) yields P (Y (t1 ) ∈ B1 , . . . , Y (tm ) ∈ Bm |[τ < ∞]) (b)

= P (X(t1 ) ∈ B1 , . . . , X(tm ) ∈ Bm );

substituting (b) into (a) and dividing by P (τ < ∞) yields P ([Y (t1 ) ∈ B1 , . . . , Y (tm ) ∈ Bm ] ∩ A|[τ < ∞]) (c)

= P (Y (t1 ) ∈ B1 , . . . , Y (tm ) ∈ Bm |[τ < ∞]) P (A|[τ < ∞]).

Thus (b) and (c) hold for the class G of sets of the form [Y (t1 ) ∈ B1 , . . . , Y (tm ) ∈ Bn ] and for all sets A in Aτ . Since G generates Y −1 (D[0,∞) ), equation (b) implies (2). Since G is also closed under finite intersections (that is, it is a π ¯ -system), (c) and proposition 8.1.1 imply (3). Case 2. Now consider a general stopping time τ . For n ≥ 1, define $ k/n for (k − 1)/n < τ ≤ k/n and k ≥ 1, τn ≡ 1/n for τ = 0, (d) ∞ for τ = ∞ . Note that τn (ω) τ (ω) for ω ∈ [τ < ∞]. For k/n ≤ t < (k + 1)/n we have [τn ≤ t] = [τ ≤ k/n] ∈ Ak/n ⊂ At (so that τn is a stopping time), and also for A in Aτ that A ∩ [τn ≤ t] = A ∩ [τ ≤ k/n] ∈ Ak/n ⊂ At (so that Aτ ⊂ Aτn ). Define (e)

Yn (t) = X(τn + t) − X(τn )

on [τn < ∞] = [τ < ∞],

and let it equal 0 elsewhere. By case 1 results (b) and (c), both (f)

P (Yn ∈ F |[τ < ∞]) = P (X ∈ F )

(g)

P ([Yn ∈ F ] ∩ A |[τ < ∞]) = P (Yn ∈ F |[τ < ∞]) P (A |[τ < ∞])

and

hold for all F in D[0,∞) and all A in Aτ (recall that Aτ ⊂ Aτn as above, and [τ < ∞] = [τn < ∞]). Let (r1 , . . . , rm ) be any continuity point of the joint df of (Y (t1 ), . . . , Y (tm )), and define

(h)

Gn G G∗ H

≡ ≡ ≡ ≡

[Yn (t1 ) < r1 , . . . , Yn (tm ) < rm , τ < ∞], [Y (t1 ) < r1 , . . . , Y (tm ) < rm , τ < ∞], [Y (t1 ) ≤ r1 , . . . , Y (tm ) ≤ rm , τ < ∞], [X(t1 ) < r1 , . . . , X(tm ) < rm ].

By the right continuity of the sample paths, Yn (t) → Y (t) for every t and every ω in [τ < ∞]; thus (i)

G ⊂ lim Gn ⊂ lim Gn ⊂ G∗ .

310


Thus P (G|τ < ∞) ≤ P (lim Gn |[τ < ∞]) ≤ lim P (Gn |τ < ∞) by (i), then DCT = P (H) = lim P (Gn |τ < ∞)

by using (f) twice

by the DCT and (i) ≤ P ( lim Gn |τ < ∞) ≤ P (G∗ |τ < ∞) m ≤ P (G|τ < ∞) + i=1 P (Y (ti ) = ri |τ < ∞) (j)

= P (G|τ < ∞),

since (r1 , . . . , rm ) is a continuity point. Thus (j) implies (k)

P (G|τ < ∞) = P (H),

and this is sufficient to imply (2). Likewise, for A ∈ Aτ ⊂ Aτn , P (G ∩ A|[τ < ∞]) ≤ P ( lim Gn ∩ A|τ < ∞)

by (i)

≤ lim P (Gn ∩ A|τ < ∞) by the DCT = lim P (Gn |τ < ∞) P (A|τ < ∞) = P (G|τ < ∞) P (A|τ < ∞)

by (j)

= lim P (Gn |τ < ∞) P (A|τ < ∞) = lim P (Gn ∩ A|τ < ∞)

by (c), with [τ < ∞] = [τn < ∞]

by (j)

by (c), with [τ < ∞] = [τn < ∞]

≤ P ( lim Gn ∩ A|τ < ∞) ≤ P (G∗ ∩ A|τ < ∞) by the DCT, then (i) m ≤ P (G ∩ A|τ < ∞) + i=1 P (Y (ti ) = ri |τ < ∞) (l)

= P (G ∩ A|τ < ∞),

since (r1 , . . . , rm ) is a continuity point. Thus (l) implies (m)

P (G ∩ A|τ < ∞) = P (G|τ < ∞) P (A|τ < ∞);

and using proposition 8.1.1 again, we see that this is sufficient to imply (3). The final statement is immediate, since when P (τ < ∞) = 1 we necessarily have P (A|τ < ∞) = P (A) for all A ∈ A. 2

Exercise 5.1 Formulate (9.1.17) rigorously.

6.

EMBEDDING A RV IN BROWNIAN MOTION

311

6

Embedding a RV in Brownian Motion

Let a, b > 0. For a Brownian motion S on (C ∞ , C∞ ), we define τ ≡ τab ≡ inf{t : S(t) ∈ (−a, b) c }

(1)

to be the first time S hits either −a or b. Call τ a hitting time. [Show that τ is a stopping time.] Note figure 6.1.

b

(·)

0

τab

–a

Figure 6.1 The stopping time τab .

Theorem 6.1 (Embedding via τab )

Let τ ≡ τab . Then:

(2)

ES(τ ) = 0 .

(3)

P (S(τ ) = −a) = b/(a + b)

(4)

Eτ = a b = ES 2(τ ) .

(5)

Eτ r ≤ r Γ(r) 22r ES 2r(τ ) ≤ r Γ(r) 22r ab (a + b)2r−2

and

P (S(τ ) = b) = a/(a + b) .

for all r ≥ 1 .

Definition 6.1 (Martingale) A process {M (t) : t ≥ 0} is a continuous parameter martingale (mg) if E|M (t)| < ∞ for all t, M is adapted to the At ’s, and (6)

E{M (t) |As } =a.s. M (s)

for all 0 ≤ s ≤ t.

Definition 6.2 (Stopping time) If τ is a random time (just a rv that is ≥ 0) for which the event [τ ≤ t] ∈ At for all t, then we call τ a stopping time.

312


Future theorem Let τ be a stopping time. With appropriate regularity conditions on a mg M , we can claim that (7)

EM (τ ) = EM (0) .

Our present applications are simple special cases of a result called the optional sampling theorem for mgs. The general version will be proven in chapter 18. We will use it for such simple special cases now. 2 Proof. The independent increments of S lead to satisfaction of the mg property stated in (6). Also, S is suitably integrable (we will see later) for (7) to hold. Thus, with p ≡ P (S(τ ) = b), we have (a)

0 = ES(τ ) = bp − a(1 − p),

or

p = a/(a + b) .

Also, the process (8)

{S 2(t) − t : t ≥ 0}

is a mg adapted to the σ-fields At ≡ σt ,

since E{S 2(t) − t|As } = E{[S(t) − S(s) + S(s)] 2 − t|As } = E{[S(t) − S(s)] 2 + 2 S(s)[S(t) − S(s)] + S 2(s) − t|As } = E{[S(t) − S(s)] 2} + 2 S(s)E{S(t) − S(s)} + S 2(s) − t = t − s + 2 S(s) · 0 + S 2(s) − t = S 2(s) − s .

(b)

This process is also suitably integrable, so that optional sampling can be used to imply E[S(τ ) 2 − τ ] = 0. Thus (c)

Eτ = ES 2(τ ) = (−a)2 · b/(a + b) + b2 · a/(a + b) = ab .

We leave (5) to exercise 12.7.3 below.

2

Theorem 6.2 (Skorokhod embedding of a zero-mean rv) Suppose X is a rv with df F having mean 0 and variance 0 ≤ σ2 ≤ ∞. Then there is a stopping time τ such that the stopped rv S(τ ) is distributed as X; that is, (9)

S(τ ) ∼ = X.

Moreover, (10)

Eτ = Var[X],

and for any r ≥ 1 we have (11)

Eτ r ≤ Kr E|X|2r

with Kr ≡ r Γ(r) 22r .

6.

EMBEDDING A RV IN BROWNIAN MOTION

313

Proof. For degenerate F , just let τ ≡ 0. Thus suppose F is nondegenerate. Let (A, B) be independent of S, with joint df H having (12)

for a ≥ 0, b > 0 .

dH(a, b) = (a + b) dF (−a) dF (b)/EX+

The procedure is to observe (A, B) = (a, b) according to H, and then to observe τab , calling the result τ . (Clearly, τab = 0 if a = 0 is chosen.) Note that [τ ≤ t] can be determined by (A, B) and {S(s) : 0 ≤ s ≤ t}, and hence is an event in At ≡ σ[A, B, S(s) : 0 ≤ s ≤ t}. For t ≥ 0,

(a)

P (S(τ ) > t) = E(P {S(τ ) > t|A = a, B = b}) = [0,∞) (0,t] 0 · dH(a, b) + [0,∞) (t,∞) (a/(a + b)) dH(a, b)

(b)

=

(c)

= 1 − F (t) ,

(t,∞)

[0,∞)

a dF (−a) dF (b) / EX + =

(t,∞)

by (3)

dF (b) EX − / EX +

since EX = 0 with X nondegenerate implies EX + = EX − . Likewise, for t ≥ 0, (d) P (S(τ ) ≤ −t) = [0,t) (0,∞) 0 · dH(a, b) + [t,∞) (0,∞) (b/(a + b)) dH(a, b)

(e)

=

(f)

= F (−t) .

[t,∞)

(0,∞) b dF (b) dF (−a) / EX

+

=

[t,∞)

dF (−a)

Thus S(τ ) ∼ = X. Moreover, Eτ = E(E{τ |A = a, B = b}) = E(E{S 2(τ )|A = a, B = b}) = ES 2(τ ) (g)

= EX 2 = Var[X] .

Note that (a + b)2r−1 ≤ 22r−2 [a2r−1 + b2r−1 ] by the Cr -inequality. Thus Eτ r = E(E{τ r |A = a, B = b}) (h)

(i) (j)

≤ 4 r Γ(r) E(AB(A + B)2r−2 )

by (5)

≤ 4 r Γ(r) E(AB(A + B)2r−1 /(A + B)) B A A2r + B 2r ≤ 4 r Γ(r) 22r−2 E A+B A+B = Kr E(E{S 2r(τ )|A = a, B = b}) = Kr E(S 2r(τ )) = Kr EX 2r ,

as claimed.

2

314

7


Barrier Crossing Probabilities

For −a < 0 < b we defined the hitting time (1)

τab ≡ inf{t : S(t) ∈ (−a, b) c },

where S denotes Brownian motion on (C ∞ , C∞ ). We also considered the rv S(τ ab), which is called Brownian motion stopped at τab . We saw that it took on the two values b and −a with the probabilities p ≡ a/(a + b) and q ≡ 1 − p = b/(a + b). For a > 0 we define the stopping time (the hitting time of a) (2)

τa ≡ inf{t : S(t) ≥ a} .

[Now, [τa ≤ c] = ∩q 0 and ∞ P (S 10 > a) = 4 k=1 P ((4k − 3) a < N (0, 1) < (4k − 1) a) ∞ 4 (−1)k (2k + 1)2 π 2 =1− exp − for a > 0. π 2k + 1 8 a2 k=0

Proof. Define the stopping time τa ≡ τa ∧ c, and note that τa = τa on the event [S(c) > a]. Now, [τ a ≤ c] ∈ Aτ is independent of the Brownian motion {Y(t) ≡ S(τ + t) − S(τ ) : t ≥ 0}, by strong Markov. In (b) below we will use that S(τ a ) = a on [S(c) > a]. We have P (τa ≤ c) = P (τa ≤ c) + 0 (a)

= P ([τa ≤ c] ∩ [S(c) > a]) + P ([τ a ≤ c] ∩ [S(c) < a]) + 0

(b)

= P ([τa ≤ c] ∩ [S(c) − S(τ a ) > 0]) + P ([τa ≤ c] ∩ [S(c) − S(τ a ) < 0])

(c)

= 2 P ([τa ≤ c] ∩ [S(t) − S(τ a) > 0])

(d)

= 2 P (S(t) > a),

using the strong Markov property

since the events in (c) and (d) are identical. The two-sided boundary of formula (4) follows from a more complicated reflection principle. Let A+ ≡ [S + > a] = [S exceeds a somewhere on [0, 1]] and A − ≡ [S − > a] = [S falls below −a somewhere on [0, 1]]. Though [S > a] = A + ∪A− , we have P (S > a) < P (A + ) + P (A− ), since we included paths that go above a and then below −a (or vice versa) twice. By making the first reflection in figure 7.1, we see that the probability of the former event equals that of A+− = [S + > 3a], while that of the latter equals that of A−+ = [S − > 3a]. But subtracting out these probabilities from P (A+ ) + P (A− ) subtracts out too much, since the path may then have recrossed the other boundary; we compensate for this by adding back in the probabilities of A+−+ ≡ [S + > 5a] and A−+− ≡ [S − > 5a], which a second reflection shows to be equal to the appropriate probability. But we must continue this process ad infinitum. Thus

7.

BARRIER CROSSING PROBABILITIES P (S 10 > a) =

(e)

315

P (A+ ) − P (A+− ) + P (A+−+ ) − · · · + P (A− ) − P (A−+ ) + P (A−+− ) − · · ·

by symmetry = 2 [P (A+ ) − P (A+− ) + P (A+−+ ) − · · ·] ∞ by (3) = 2 k=1 (−1)k+1 2 P (N (0, 1) > (2k − 1) a) ∞ = 4 k=1 P ((4k − 3) a < N (0, 1) < (4k − 1) a)

(f)

(g)

as claimed. The final expression (5) is left for the reader; it is reputed to converge more quickly. 2 Exercise 7.1

Prove (5). (See Chung (1974, p. 223).)

The second reflection 3a The first reflection 2a a The path 0 (·) –a

Figure 7.1 The reflection principle for Brownian motion.

Theorem 7.2 (The reflection principle for linear boundaries; Doob) the line c t + d with c ≥ 0, d > 0. Then: (6) (7) Proof. (8)

Consider

P (S(t) ≥ ct + d for some t ≥ 0) = exp(−2 c d) . k+1 P (|S(t)| ≥ c t + d for some t ≥ 0) = 2 ∞ exp(−2 k2 c d) . k=1 (−1) Now, for any θ = 0 the process {V (t) ≡ exp([θ S(t) − θ 2 t/2]) : t ≥ 0}

is a mg (with V (0) ≡ 1).

This holds with σt ≡ σ[S(s) : s ≤ t] (using the mgf of a normal rv), since E{V (t)|σs } = exp([θS(s) − θ 2 s/2] + θ [S(s, t] − θ(t − s)/2]) (a)

= V (s) exp(−θ2 (t − s)/2) E{exp(θ N (0, t − s))}

(b)

= V (s).

316


Thus if we now redefine τab ≡ inf{t : X(t) ≡ S(t) − θt/2 ∈ (−a, b) c } where we have a > 0, b > 0, then V (t) = eθ X(t) . Hence (c)

1 = EV (τab ) = P (X(τab ) = −a) e−θa + P (X(τab ) = b) eθb ,

so that (9)

P (X(τab ) = b) = (1 − e−θa )/(eθb − e−θa )

(d)

→ e−θb

if θ > 0 and a → ∞

(e)

= e−2cd

if θ = 2c and b = d.

But this same quantity also satisfies (f)

P (X(τab ) = b) → P (X(t) ≥ b for some t)

as a → ∞

= P (S(t) − θt/2 ≥ b for some t) = P (S(t) ≥ θt/2 + b for some t) = P (S(t) ≥ ct + d for some t)

(g)

if c = θ/2 and d = b.

Equating (g) to (e) (via (f) and (9)) gives (6).

Exercise 7.2

2

Prove (7). (See Doob (1949).)

Theorem 7.3 (Kolmogorov–Smirnov distributions)

Both

P (U± > b) = exp(−2b2 ) for all b > 0 k+1 exp(−2 k2 b2 ) P (U > b) = 2 ∞ k=1 (−1)

and

(10) (11)

for all b > 0 .

Now, U− ∼ = U+ and

Proof.

P (U+ > b) = P (U(t) > b for some 0 < t < 1) = P ((1 − t) S(t/(1 − t)) > b)

(12)

for some 0 ≤ t ≤ 1,

= P (S(r) > b + r b for some r ≥ 0) = exp(−2 b2 )

(a)

letting r = t/(1 − t)

by theorem 7.2.

Likewise, (b) (c)

P (U > b) = P (|S(s)| > b + s b for some s ≥ 0) ∞ = 2 k=1 (−1)k+1 exp(−2 k2 b2 )

by theorem 7.2.

2

by (12.3.16)

7.

BARRIER CROSSING PROBABILITIES

317

Exercise 7.3 (a) Prove (12.6.5) for r = 2. (b) Prove (12.6.5) for integral r. [Hint. The Vθ of (8) are martingales on [0, ∞). Differentiate under the integral sign in the martingale equality (13) for all A ∈ As . A Vθ (t) dP = A Vθ (s) dP Then conclude that [∂k /∂θk Vθ (t)]θ=0 is a martingale for each k ≥ 1. For k = 4 √ this leads to S 4(t) − 6tS 2(t) + 3t2 = t2 H4 (S(t)/ t ) being a martingale on [0, ∞); here H4 (·) is the fourth Hermite polynomial (defined in (14.5.15)). We need to work only with this single specific martingale in part (a); the rest of this hint is simply an intuitive explanation of how this martingale arises.]

318

8


Embedding the Partial Sum Process

The Partial Sum Process Let Xn1 , . . . , Xnn be row-independent rvs having a common F (0, 1) distribution, and let Xn0 ≡ 0. We define the partial sum process Sn on (D, D) by 1 1 S n(t) ≡ √ Xni = √ Xni n i=0 n i=0 [nt]

(1)

k

for

k k+1 ≤t< , 0≤k≤n n n

(or for all k ≥ 0, in case the nth row is Xn1 , Xn2 , . . .). Note that (2)

[ns] [nt] Cov[S n(s), S n(t)] = i=1 j=1 Cov[Xni , Xnj ]/n = [n (s ∧ t)]/n for 0 ≤ s, t ≤ 1

for the greatest integer function [·]. We suspect that Sn “converges” to S. We will establish this shortly.

Embedding the Partial Sum Process Let {S(t) : t ≥ 0} denote a Brownian motion on (C ∞ , C∞ ). Then √ Z n(t) ≡ n S(t/n) for t ≥ 0 is also such a Brownian motion.

Notation 8.1 (3)

By using the Skorokhod embedding technique of the previous section repeatedly on the Brownian motion Z n, we may guarantee that for appropriate stopping times τn1 , . . . , τnn (with all τn0 ≡ 0) we obtain that (4)

Xnk ≡ Z n(τn,k−1 , τnk ],

for 1 ≤ k ≤ n,

are iid F (0, 1) rvs.

Let S n denote the partial sum process of these Xnk ’s. Then, for t ≥ 0 we have

[nt] τ S n(t) = √1n k=1 Xnk = √1n Z n(τn,[nt] ) = S n,[nt] n (5)

[nt] = S n1 k=1 Tnk = S(I n(t)) with Tnk ≡ (τnk − τn,k−1 ) and In (t) ≡ n1 τn,[nt] . Observe that: (6)

Xn1 , . . . , Xnn

are iid F (0, 1), in each row.

(7)

Tn1 , . . . , Tnn

(8)

r ETnk ≤ Kr · E|Xnk |2r ,

are iid with means = 1 = Var[X], in each row. with

Kr ≡ r Γ(r) 22r .

2

Theorem 8.1 (Skorokhod’s embedding theorem) The partial sum process S n on (D, D) of row-independent F (0, 1) rvs formed as above satisfies (9)

S n − S → p 0

as n → ∞.

8.

EMBEDDING THE PARTIAL SUM PROCESS

319

Notice: The joint distributions of any Sm, S n in theorem 8.1 are not the same as they would be if formed from a single sequence of iid rvs. In fact, we have no idea of what these joint distributions may be. However, the partial sums of an iid sequence do not generally converge to their limit in the sense of →p , so we have gained a great deal via the embedding. Theorem 8.2 (Embedding at a rate) Suppose that EX 4 < ∞. Let I denote the identity function. Then for each 0 ≤ ν < 14 , the process S n of (5) satisfies (10)

nν (S n − S)/I 1/2−ν 11/n = Op (1).

Proof. Consider theorem 8.1. Let I denote the identity function. Suppose we now show that (a)

In − I10 = sup0≤t≤1 |τn,[nt] /n − t| →p 0.

Then on any subsequence n where →p 0 in (a) may be replaced by →a.s. 0, the continuity of the paths of S will yield (b)

S n (·) − S(·) = S(I n ) − S → a.s. 0,

and thus (9) will follow. This is a useful argument; learn it. It therefore remains to prove (a). The WLLN gives (c)

In (t) = τn,[nt] /n →p t

for any fixed t.

Using the diagonalization technique, we can extract from any subsequence a further subsequence n on which (d)

In (t) →a.s. t

for all rational t.

But since all functions involved are monotone, and since the limit function is continuous, this implies that a.s. (e)

In (t) → t uniformly on [0, 1] .

Thus (a) follows from (e), since every n has a further n with the same limit. Thus the conclusion (9) holds. In the proof just given, the conclusion (9) can trivially be replaced by (f)

sup0≤t≤m |S n(t) − S(t)| → p 0.

Appealing to exercise 12.1.6(b) for the definition of · ∞ , we thus obtain (11)

ρ∞ (S n, S) → p 0

on (C∞ , C∞ ) ,

provided that the rvs Xn1 , Xn2 , . . . are appropriately iid (0, σ2 ). [We consider the proof of theorem 8.2 at the end of this section.] 2 Let g : (D, D) → (R, B) and let ∆g denote the set of all x ∈ D for which g is not · -continuous at x. If there exists a set ∆ ∈ D having ∆g ⊂ ∆ and P (S ∈ ∆) = 0, then we say that g is a.s. · -continuous with respect to the process S.

320


Theorem 8.3 (Donsker) Let g : (D, D) → (R, B) denote an a.s. · -continuous mapping that is D-measurable. Then g(S n) : (Ω, A, P ) → (R, B), and both (12)

g(S n) →p g(S)

as n → ∞

for the S n of (5)

(13)

g(S n) →d g(S)

as n → ∞

for any S n having the same distribution.

and

(D-measurability is typically trivial, and hypothesizing it avoids the measurability difficulties discussed in section 12.1.) [Theorem 8.2 allows (13) for D-measurable functionals g that are continuous in other · /q-metrics.] Proof. Now, S n −S is a D-measurable rv, and S n −S → p 0 for the S n of (5). Thus any subsequence n has a further subsequence n for which S n − S → 0 for all ω ∈ A , where P (A ) = 0. Moreover, (a)

P (A ∪ [S ∈ ∆]) ≤ P (A ) + P (S ∈ ∆) = 0 ,

and if ω ∈ A ∪[S ∈ ∆], then g(S n(ω)) → g(S(ω)) holds, since S n(ω)−S(ω) → 0 and g is -continuous at S(ω). Thus g(S n) →p g(S) as n → ∞ for the S n of (5). Thus g(S n) →d g(S) for the S n of (5), and hence of (13) also. [Note that we are dealing only with functionals for which the compositions g(Sn ) and g(S) are (Ω, A, P ) → (R, B) measurable.] 2

Example 8.1 (14)

Since the functionals · and ·+ are a.s. · -continuous,

+ S + n →d S

and

S n →d S .

The limiting distributions are those given in theorem 7.1.

2

Exercise 8.1 Let X0 ≡ 0 and X1 , X2 , . . . be iid (0, σ2 ). Define Sk ≡ X1 +· · ·+Xk for each integer k ≥ 0. (a) Find the asymptotic distribution of (S1 + · · · + Sn )/cn for an appropriate cn . (b) Determine a representation for the asymptotic distribution of the “absolute area” under the partial sum process, as given by (|S1 | + · · · + |Sn |)/cn .

The LIL Recall the (10.7.1) LIL for a single sequence of iid F (0, 1) rvs X1 , X2 , . . . with partial sums Sn ≡ X1 + · · · + Xn ; that is (15) limn→∞ |Sn |/ 2 n log log n = 1 a.s. The two LILs for Brownian motion (recall (12.3.7) and (12.3.18)) are (16) lim t→∞ |S(t)|/ 2 t log log t = 1 a.s., (17)

lim t→0 |S(t)|/

2 t log log(1/t) = 1 a.s.

8.

EMBEDDING THE PARTIAL SUM PROCESS

Notation 8.2 which the rvs (18)

321

Define stopping times T1 , T2 , . . . (with T0 = 0) having mean 1 for

Xk ≡ S(τ k−1, τk ] are iid as F .

Let τk ≡ T0 + T1 + · · · + Tk for k ≥ 0, and define the partial sums n (19) Sn ≡ k=1 Xk = S(τ n) = S(n) + [S(τ n) − S(n)]. [Note that this embedding differs from that in notation 8.1. This one is based on a single sequence of rvs X1 , X2 , . . . .] 2 Exercise 8.2 (The LIL) (a) First prove (15), while assuming that (16) is true. [Hint. By proposition 10.7.1, we want to show (roughly) that √ |S(τ n) − S(n)|/ 2 n log log n →a.s. 0 or that √ (20) |S(τ [t]) − S(t)|/ 2 t log log t →a.s. 0. We will now make rigorous this approach to the problem. First apply the SLLN to τ [t]/t as t → ∞. Then define ∆k ≡ sup{|S(t) − S(t k)| : tk ≤ t ≤ tk+1 }, with tk ≡ (1 + a)k for some suitably tiny a > 0. Use a reflection principle and Mills’ ratio to show that P (∆k ≥ (an appropriate ck )) < ∞. Complete the proof using Borel–Cantelli.] (b) Now that you know how to deal with the “blocks” ∆k , model a proof of (16) on the proof of proposition 10.7.1.

Proof for Embedding at a Rate Proof. Consider theorem 8.2. Let d2 ≡ Var[T ]. Let Log k ≡ 1 ∨ (log k). Let M ≡ M be specified below, and define √ k (a) Acn ≡ [max1≤k≤n | i=1 (Tni − 1)|/(d k Log k) ≥ 2M/d]. Then the monotone inequality gives √ k Acn ⊂ [max1≤k≤n | i=1 {(Tni − 1)/(d i Log i)}| ≥ M/d] (b) ≡ [max1≤k≤n | ki=1 Yni | ≥ M/d], where the Yni ’s are independent with mean 0 and variance (i Log2 i)−1 . Thus the Kolmogorov inequality gives n n P (Acn ) ≤ (d/M )2 Var[ 1 Yni ] = (d/M )2 1 (i Log2 i)−1 ∞ (21) ≤ (d/M )2 1 (i Log2 i)−1 ≡ (d/M )2 v 2 < 2 if M > dv/

(c)

< .

Thus (d)

. P (Bn ) ≡ P

max

1≤k≤n

nν | S(

k

1 Tni /n) − (k/n)1/2−ν

S(k/n)|

2M ≥√ dv

/

322

(e)

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES ≤ P (Bn ∩ An ) + P (Acn ) k n | S( 1 Tni /n)−S(k/n)| √ ≤ k=1 P [2M k ( Log k)/n]1/2 ≥ n

2M k1/2−ν 1 √ √ √ n dv [2M k ( Log k)/n]1/2

∩ An +

√ | S(r + k/n) − S(k/n)|/ a ≥ b) +

(f)

≤

(g) (h)

and with ≥ b as on the right in (e) √ n using (k) below ≤ 3 k=1 P (sup 0≤r≤a | S(t, t + r]|/ a ≥ b/3) +

n ≤ 12 k=1 P (N (0, 1) ≥ b/3) +

by the reflection principle

(i)

≤ 12

(22)

k=1 P (sup 0≤|r|≤a

√ with a ≡ 2M k (Log k)/n (as in An in (a)),

≤ 12

n k=1 n k=1

(j)

exp(−(b/3)2 /2) +

by Mills’ ratio

M k 1/2−2ν exp − +

9dv Log k

< 2 ,

if M ≡ M is large enough and if 0 ≤ ν < 14 (this final step holds, since we have ∞ δ 0 exp(−c x ) dx → 0 as c → ∞). The inequality (g) used sup 0≤|r|≤a |S(r + k/n) − S(k/n)| (k)

≤ sup 0≤r≤a | S(r+k/n)−S(k/n)|+2 sup 0≤r≤a | S(r+k/n−a)−S(k/n−a)|

[with t in (g) equal to k/n or k/n − a, and with a as above (see (f))]. Now, P (Bn ) ≤ 2 shows that (9) is true, provided that the sup over all of [1/n, 1] is replaced by the max over the points k/n with 1 ≤ k ≤ n. We now “fill in the gaps”. Thus (even a crude argument works here) √ P ( n max1≤k≤n−1 sup 0≤t≤1/n | S(t + k/n) − S(k/n)| / k 1/2−ν ≥ M ) √ n−1 1/n ≤ k=1 P ( S 0 ≥ M k 1/2−ν / n ) n−1 ≤ 4 k=1 P (N (0, 1) ≥ M k1/2−ν ) by the reflection principle n−1 ≤ 4 k=1 exp(−M 2 k 1−2ν /2) (23) by Mills’ ratio (l)

< ,

if M ≡ M is large enough and if 0 ≤ ν < Exercise 8.3 (24)

1 2

(even).

2

Suppose EX 4 < ∞. Show that the process S n of (5) satisfies

(n1/4 / log n) S n − S = O p (1) .

[Hint. Replace nν /(k/n)1/2−ν by n1/4 / log n in the definition of Bn in (d). Now determine the new form of the bounds in (20) and (21).] [While interesting and often quoted in the literature, this formulation has little value for us.]

9.

OTHER PROPERTIES OF BROWNIAN MOTION

9

Other Properties of Brownian Motion

323

Here we collect some selected sample path properties of Brownian motion, just to illustrate a sample of what is known. Some proofs are outlined in the exercises. Definition 9.1 (Variation)

For a sequence of partitions

Pn ≡ {(tn,k−1 , tnk ] : k = 1, . . . , n} of [0, 1] (with 0 ≡ tn0 < · · · < tnn ≡ 1), define the rth variation of S corresponding to P n by n (1) Vn (r) ≡ k=1 |S(t nk) − S(t n,k−1)|r . We call these partitions nested if Pn ⊂ Pn+1 for all n ≥ 1. We further define the mesh of the partitions to be Pn ≡ sup1≤k≤n |tnk − tn,k−1 |. Theorem 9.1 (Nondifferentiability) (a) Almost every Brownian path is nowhere differentiable. (b) In fact, Vn (1) → ∞ a.s. if Pn → 0. (c) (Finite squared variation) Vn (2) →L2 1 if Pn → 0. (d) (Finite squared variation) Vn (2) →a.s. 1 ∞ if either (i) n=1 Pn < ∞ or (ii) The Pn are nested with mesh approaching 0. (e) (Dudley) Vn (2) →a.s. 1 if and only if (log n) Pn → 0. Theorem 9.2 (Lévy) (2)

The H¨ older condition is true:

|S(t) − S(s)| = 1 a.s. lim sup 2 a log(1/a) {0≤s 0. Show that: √ Cr ≡ E|Z|r = 2r/2 Γ( r+1 2 )/ π.

(b)

|S(t n,k−1, tnk ] |r ∼ = ( Cr |tnk − tn,k−1 |r/2 , (C2r − Cr2 ) (tnk − tn,k−1 )r ) .

Exercise 9.2

(β) Now show that E Vn (2) = 1 and Var[Vn (2)] ≤ (C2r − Cr2 ) Pn , giving ∞ 2 −2 ∞ (c) 1 P ( |Vn (2) − 1| ≥ ) ≤ (C2r − Cr )

1 Pn < ∞. (γ) Finally, demonstrate the truth of theorem 9.1(d), case (i). Exercise 9.3 Prove theorem 9.1(b) when all tnk = k/2n . [Hint. Let 0 < λ < 1. The Paley–Zygmund inequality gives P (Vn (1) > λE Vn (1)) ≥ (1 − λ)2 E2 Vn (1)/E(Vn2 (1)) → (1 − λ)2 , where E Vn (1) → ∞.]

10.

VARIOUS EMPIRICAL PROCESS

325

10

Various Empirical Process

Suppose that ξn1 , . . . , ξnn are iid Uniform(0, 1). Their empirical df Gn is defined by 1 1[0,t] (ξnk ) n n

(1)

Gn (t) ≡

for 0 ≤ t ≤ 1

k=1

for ξn:k ≤ t < ξn:k+1

= k/n

(2)

and 0 ≤ k ≤ n ,

where 0 ≡ ξn:0 ≤ ξn:1 ≤ · · · ≤ ξn:n ≤ ξn:n+1 ≡ 1 are the order statistics; see figure 10.1. Note that nGn (t) ∼ = Binomial(n, t) ∼ = (t, t(1 − t)). The Glivenko–Cantelli theorem shows that Gn converges uniformly to the true df I; that is, (3)

Gn − I →a.s. 0

(even for the present triangular array of ξnk ’s).

(The Cantelli proof of the SLLN based on fourth moments shows that Gn (t) →a.s. t for each fixed t; even for triangular arrays. The rest of the proof is identical.) The uniform empirical process Un is defined by (4)

Un (t) ≡

√

1 n [Gn (t) − t] = √ [1[ξnk ≤t] − t] n n

for 0 ≤ t ≤ 1.

k=1

This process is also pictured in figure 10.1. The means and covariances of Un are the same as those of Brownian bridge U, in that (5)

EUn (t) = 0

and

Cov[Un (s), Un (t)] = s∧t−st

for all 0 ≤ s, t ≤ 1;

this follows easily from (6)

Cov[1[0,s] (ξnk ), 1[0,t] (ξnk )] = s ∧ t − st

for 0 ≤ s, t ≤ 1.

{Moreover, for any dk ’s and ek ’s we have immediately from this that n n n (7) Cov[ 1 dk 1[0,s] (ξnk ), 1 ek 1[0,t] (ξnk ))] = ( 1 dk ek ) × [s ∧ t − st] ; we would have n1 E[dk , ek ] instead, if these were rvs independent of the ξnk ’s.} We note that G−1 n (t) ≡ inf{x ∈ [0, 1] : Gn (x) ≥ t} is left continuous, with (8)

G−1 n (t) = ξn:k for (k − 1)/n < t ≤ k/n

and G−1 n (0) = 0, as in figure 10.1. The uniform quantile process Vn is defined by √ (9) Vn (t) ≡ n [G−1 for 0 ≤ t ≤ 1. n (t) − t] The key identities relating Un and Vn are (with I the identity function) the trivial √ (10) Un = −Vn (Gn ) + n [G−1 on [0, 1] , n ◦ Gn − I] √ −1 (11) Vn = −Un (G−1 on [0, 1] . n ) + n [Gn ◦ Gn − I] Note that (12)

Gn ◦ G−1 n − I = 1/n

and

G−1 n ◦ Gn − I = [max1≤k≤n+1 δnk ] ;

here δnk ≡ (ξn:k − ξn:k−1 ), for 1 ≤ k ≤ n + 1, denotes the kth of the n + 1 uniform spacings.

326


¨ n and G ¨ −1 defined by It is sometimes convenient to use the smoothed versions G n ¨ n (ξn:k ) = k/(n + 1) and G ¨ −1 (k/(n + 1)) = ξn:k (13) G for 0 ≤ k ≤ n + 1, n connected linearly between points. Sometimes the smoothed uniform quantile pro¨ −1 (t) − t] is a useful variation on Vn . The Glivenko–Cantelli ¨ n (t) ≡ √n [G cess V n theorem implies that ¨ n −I →a.s. 0, G ¨ −1 −I →a.s. 0 ; Gn −I →a.s. 0, G−1 −I →a.s. 0, G (14) n

n

see figure 10.1. Coupling these with the identities (10) and (11) shows that (15)

Un − U →p 0

Vn − V →p 0 .

if and only if

Let cn ≡ (cn1 , . . . , cnn ) denote a vector of known constants normalized so that n n 2 cn· ≡ n1 k=1 cnk = 0 and σc,n ≡ n1 k=1 (cnk − cn· )2 = 1 , and let (16) n c4n· ≡ n1 k=1 [cnk − cn· ]4 . We suppose that these constants also satisfy the uan condition √ √ (17) max |cnk − c¯n |/[ n σc,n ] = max |cnk |/ n → 0 1≤k≤n

1≤k≤n

as n → ∞.

The weighted uniform empirical process is defined by n 1 (18) Wn (t) ≡ √ cnk [1[ξnk ≤t] − t] for 0 ≤ t ≤ 1. n k=1

The Wn process is pictured in figure 10.1. It is trivial from (7) that (19)

Cov[Wn (s), Wn (t)] = s ∧ t − st

for 0 ≤ s, t ≤ 1.

It is easy to show that Wn →f d W , where W denotes another Brownian bridge, one that is independent of U. n ≡ (Rn1 , . . . Rnn ) denote the ranks of ξn1 , . . . , ξnn ; and then denote the Let R n ≡ (Dn1 , . . . , Dnn ) . Then R n is a random permutation of the antiranks by D vector (1, . . . , n) , while Dn is the inverse permutation. These satisfy (20)

ξnDnk = ξn:k

and

ξnk = ξn:Rnk .

As observed in example 8.5.3, (21)

(ξn:1 , . . . , ξn:n ) and (Rn1 , . . . , Rnn )

are independent rvs.

The empirical finite sampling process Rn is defined by (22)

[(n+1)t] 1 Rn (t) ≡ √ cnDnk n

for 0 ≤ t ≤ 1.

k=1

The Rn process is also pictured in figure 10.1. The key identities are ¨ −1 ) (23) Wn = Rn (Gn ) or Rn = Wn (G on [0, 1]. n These identities give (24)

Wn − W →p 0

if and only if

Rn − R →p 0,

as with (15). Because of (21), we see that (25)

Rn

and Vn

are independent processes.

10.


1

1 n

n–1 n

–1 n

ξn:n

…

… 1 n 0

327

ξn:1

…

0 ξn:1

ξn:n 1

0

…

1 n

n–1 1 n

n

n

n

CnDnn

ξn:1 CnDn1

ξn:n 1 CnDn2

Figure 10.1

n,

–1 n ,

1 n+1

CnDnn

CnDn1

n 1 n+1 CnDn2

n,

n

and

n.

328

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES We reiterate that

(26)

W

and V = −U

are independent Brownian bridges; n n this is further corroborated, since (7) with 1 dk ek = 1 cnk /n = 0 imply that the cross covariance (27)

Cov[Un (s), Wn (t)] = 0

for all 0 ≤ s, t ≤ 1 .

We will prove only part of theorems 10.1 and 10.3 (namely, that (28) holds). For the believability of the rest, we will rely on (28), our earlier proof that S n can be embedded at a rate, and the proof of theorem 10.2. (Shorack(1991) contains these proofs, written in the current style and notation.) See section 12.11 for proofs of theorems 10.2 and 10.4. Theorem 10.1 (Convergence of the uniform processes) We can define independent Brownian bridges U = −V and W and row-independent Uniform(0, 1) rvs ξn1 , . . . , ξnn on a common probability space (Ω, A, P ) in such a way that (28)

Un − U →p 0

and

Vn − V →p 0 ,

(29)

Wn − W →p 0

and

Rn − R →p 0 ,

provided that the cnk ’s are uan with c¯n = 0, σc2 = 1, and c4n < ∞. Theorem 10.2 (Pyke–Shorack) Let q > 0 on (0, 1) be on [0, 12 ], on [ 12 , 1], 1 and have 0 [q(t)]−2 dt < ∞. Then: (30)

(Un − U)/q →p 0

and

(Vn − V)/q →p 0 .

(31)

(Wn − W )/q →p 0

and

(Rn − W )/q →p 0 .

Corollary 1 (Cs¨ org˝ o–Révész) We may replace 1/q in the previous theorem by K, for any qf K having Var[K(ξ)] < ∞. Theorem 10.3 (Weighted approximation of the uniform processes) The embeddings of the previous theorem are such that for any 0 ≤ ν < 14 we have (a) (M. Cs¨ org˝ o, S. Cs¨ org˝ o, Horv´ ath, Mason) 1−1/n

(32)

∆νn ≡ nν (Un − U)/[I ∧ (1 − I)]1/2−ν 1/n

(33)

¯ νn ≡ nν (Vn − V)/[I ∧ (1 − I)]1/2−ν 1−1/2n = Op (1) . ∆ 1/2n

= Op (1) ,

(b) (Shorack) Suppose lim c4n < ∞. Then (34)

˙ νn ≡ nν (Wn − W )/[I ∧ (1 − I)]1/2−ν 1−1/n = Op (1) , ∆ 1/n

(35)

¨ νn ≡ nν (Rn − W )/[I ∧ (1 − I)]1/2−ν 1−1/2n = Op (1) . ∆ 1/2n

[The supremum limits in (32) and (34) may be changed to c/n and 1 − c/n for any constant c > 0. This relates to exercise 10.3 below.]

10.


329

Theorem 10.4 (Weighted approximation of Gn ; Mason) Gn , any n ≥ 1, any 0 < ν < 12 , and all λ > 0 we have = = = nν (Gn − I) = 0 = = Op (1) . = (36) ∆νn ≡ = [I ∧ (1 − I)]1−ν =

For any realization of

¨ −1 in (36). We may replace Gn by G n Example 10.1 (R-statistics) Consider the simple linear rank statistics n n 1 1 Rnk i (37) cnk K =√ K cnDnk Tn ≡ √ n+1 n+1 n n k=1

(38)

=

1 0

K dRn = −

k=1

1 0

Rn dK ,

where the last step holds if K = K1 − K2 with each Ki and left continuous on (0, 1). As in (12.3.19), this suggests that 1 (39) Tn →p 0 W dK ∼ = N (0, Var[K(ξ)]), provided that the uan condition holds and provided that Var[Ki (ξ)] < ∞ for i = 1, 2. Indeed, this can be shown to be true. (Writing 1 1 Tn = − 0 W dK ⊕ (Rn − W )/q 0 [q(t)]−2 d|K| provides a simple proof in case this integral is finite for some square integrable function q and for total variation measure d|K|.) We will return to this in chapter 16 below. 2 Proof. Consider Vn . We will represent our uniforms rvs as a normed sum of Exponential(1) rvs. Thus we really begin with a Skorokhod embedding of iid Exponential(1) rvs. Let F (x) = 1 − exp(−(x + 1)) for x ≥ −1, so that F is a (0, 1) df; and if X ∼ = F, then X + 1 ∼ = Exponential(1). According to Skorokhod’s embedding theorem, there exist row-independent rvs Xn1 , . . . , Xnn with df F such that the partial sum process S n of the nth row satisfies S n − S → p 0 for some Brownian motion S. We now define (40)

ηnk ≡ k + Xn1 + · · · + Xnk

and ξn:k ≡ ηnk /ηn,n+1

for 1 ≤ k ≤ n + 1.

It is an elementary exercise below to show that these ξnk ’s are distributed as n row-independent Uniform(0, 1) order statistics. Let Gn denote their empirical df and Un their uniform empirical process. The key identity relating Vn to S n is √ k ηk k (a) Vn−1 = n−1 − n−1 ηn n−1 √ n n − 1 ηk − k k ηn − n k k √ √ = − − n−1 − ηn n n n−1 n n n

330

(41)

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES n = ηn

-

1 k n−1 [S n(k/n) − (k/n) S n(1) ] − √ , n n−1 n

so that for 0 ≤ t ≤ 1, (42)

n Vn−1 (t) = ηn

-

n−1 1 [ S n(In (t)) − In (t) S n(1) ] − √ In (t), n n−1

where In (t) ≡ k/n for (k − 1)/(n − 1) < t ≤ k/(n − 1) and 1 ≤ k ≤ n − 1 with In (0) ≡ 0 satisfies In − I → 0. Note that ηn /n →p 1 by the WLLN and that Gn − I →p 0. Thus (43)

Vn − V →p 0

for

V ≡ S − I S(1)

follows from the identity (42), In − I → 0, and the fact that (b)

S(I n) − S ≤ S(I n) − S(I n) + S(I n) − S ≤ S n − S + S(I) − S →

p

0,

by continuity of all sample paths of the S process. All sample paths of√V are continuous, and the maximum jump size of |Vn − V| is bounded above by [ n max1≤i≤n+1 δni ]; so Vn − V →p 0 and (12) imply √ √ (44) nG−1 as n → ∞. n ◦ Gn − I = [ n max1≤k≤n+1 δni ] →p 0 Thus

√

(c)

n [Gn ◦ G−1 n − I ] + V √ ≤ Vn (Gn ) − V(Gn ) + V(Gn ) − V + n Gn ◦ G−1 n − I

(d)

≤ Vn − V + V(Gn ) − V + op (1)

Un − U = − Vn (Gn ) +

= V(Gn ) − V + op (1) (e)

= op (1),

using Gn − I|| →p 0 and uniform continuity of the sample paths of V. We will prove Mason’s theorem in the next section. 2 Exercise 10.1

Establish the claim made just below (40).

Example 10.2 (The supremum functionals) · -continuous. Then (45)

g(Un ) →p g(U)

and

Suppose g : (D, D) → (R, B) is a.s.

g(Vn ) →p g(V)

for the special constructions of theorem 10.1. Moreover, convergence in distribution holds for any versions of these processes. Letting # denote +, −, or | · |, we can thus claim the convergence in distribution 1 2 1 # U# (46) and U (t) dt →d 0 U2 (t) dt n →d U 0 n for any versions of these processes. These limiting distributions were given in section 12.7. 2

10.


331

Exercise 10.2 (The two-sample uniform process) (i) Let Gm and Hn be the empirical dfs of two independent Uniform(0, 1) special constructions. Let √ √ Um = m(Gm − I) and Vn ≡ n(Hn − I) denote the corresponding empirical process, and let λmn ≡ n/(m + n). Then mn (Gm − Hn ) = ( λmn Um − 1 − λmn Vn ) Wmn ≡ m+n has (47)

Wmn − W0mn →p 0 as m ∧ n → ∞, where √ √ W0mn ≡ ( λmn U − 1 − λmn V) is a Brownian bridge.

# We thus have W# mn →d W , for Brownian bridge W . Write out the details. (ii) Now use a discrete reflection principle to compute the exact distribution of P (W+ nn ≥ a), and pass to the limit in the resulting expression to obtain (12.7.9). (This provides an alternative to the earlier method.) [Hint. Go through the order statistics of the combined sample from smallest to largest. If it is from sample 1, step up one unit as you go to the right one unit. If it is from sample 2, step down one unit as you go to the right one unit. In this way, perform a random walk from (0, 0) to (2n, 0). What is the chance you ever cross a barrier of height a?]

Example 10.3 (The Kolmogorov–Smirnov and Cramér–von Mises statistics) Let ξn1 , . . . , ξnn be the iid Uniform(0, 1) rvs of the special construction. Let F denote an arbitrary df. Then Xnk ≡ F −1 (ξnk ), 1 ≤ k ≤ n, are iid F . Let Fn denote the empirical √ df of Xn1 , . . . , Xnn and let En denote the empirical process defined by En (x) ≡ n [Fn (x) − F (x)]. Now, En = Un (F ). Thus (28) implies En − U(F ) ≤ Un − U →p 0, where equality holds if F is continuous. Thus √ # √ # (48) nDn ≡ n (Fn − F )# = U# if F is continuous. n →d U Likewise, a change of variable allows elimination of F , and gives 1 1 (49) Wn2 ≡ n(Fn −F )2 dF = 0 U2n (t) dt →d 0 U2 (t) dt if F is continuous. √ These statistics are used to test whether or not F is really the true df, and nDn# and Wn2 all measure how far the estimate Fn of the true df differs from the √ hypothesized df F . [The percentage points of the asymptotic distributions of nDn# and Wn2 , under the null hypothesis when F is really the true df, are available.] (i) (i) Consider now the two-sample problem in which the rvs Xnj ≡ F −1 (ξnj ), for i = 1, 2 and 1 ≤ j ≤ ni , of independent special constructions have empirical dfs (1) (2) Fn1 and Fn2 . Note that for independent uniform empirical processes (2) [F(1) n1 − Fn2 ] =

(50)

n1 n2 n1 +n2

(51)

=a Wn1 ,n2 (F )

n2 n1 +n2

U(1) n1 (F ) −

n1 n1 +n2

U(2) n2 (F ) ≡ Wn1 ,n2 (F )

if F is continuous,

where (52)

Wn1 ,n2 ≡

n1 n1 +n2

U(1) −

n2 n1 +n2

U(2) ∼ =U

for all n1 and n2 .

332


This gives the asymptotic null distribution for the various supremum and integral functionals with which we have dealt, no matter which version of these processes is considered. 2 Exercise 10.3

Show that n ξn:n →d Exponential(1).

1 Exercise 10.4 ( 0 g dUn ) Suppose Var[g(ξ)] and Var[h(ξ)] are finite. 1 1 (a) Show that for some rvs (to be labeled 0 g dU and 0 h dW ) we have 1 1 1 1 (53) g dUn →p 0 g dU and h dWn →p 0 h dW . 0 0 (b) Show also that 1 1 (54) 0 g dVn →p − 0 g dU Exercise 10.5 (Mason) (55)

and

1 0

h dRn →p

1 0

h dW .

Consider the ∆nν of (32). For some a > 0,

supn≥2 E exp(a ∆nν ) < ∞.

[Hint. This is too hard to be an “exercise”, but it is a very nice bound.]

11.

INEQUALITIES FOR VARIOUS EMPIRICAL PROCESSES

11

Inequalities for Various Empirical Processes

333

We wish to apply the Birnbaum–Marshall and H´ ajek–Rényi inequalities to various martingales (mgs) associated with the processes of the previous section. Proposition 11.1 (Various martingales) (1)

{Un (t)/(1 − t) : 0 ≤ t < 1}

is a mg.

(2)

{Wn (t)/(1 − t) : 0 ≤ t < 1}

is a mg.

(3)

{U(t)/(1 − t) : 0 ≤ t < 1}

is a mg.

(4)

{Vn (k/(n + 1)) : (1 − k/(n + 1)) : 0 ≤ k ≤ n}

is a mg.

(5)

{Rn (k/(n + 1))/(1 − k/n) : 0 ≤ k ≤ n − 1}

is a mg.

Proof. (a)

Let At ≡ σ[1[ξ≤s] : 0 ≤ s ≤ t]. Then E{1[ξ≤t] − t|As } = 1[ξ≤s] + = 1[ξ≤s] +

(b)

=

t−s 1−s {1

1−t 1−s {1[ξ≤s]

t−s 1−s 1[ξ>s]

−t

− 1[ξ≤s]} − t

− s} ,

so that (6)

[1[ξ≤t] − t]/(1 − t),

0 ≤ t ≤ 1,

is a mg.

Noting (12.10.7), summing (6) shows that (1) and (2) hold. Let At ≡ σ[U(s) : 0 ≤ s ≤ t]. Then (9.3.8) gives (c)

E(U(t)|As ) = {[s(1 − t)]/[s(1 − s)]}U(s),

since U(t)|U(s) is normal with mean µt + Σts Σ−1 ss [U(s) − µs ]. Thus (3) holds. Let Znk ≡ Rn (k/(n + 1))/(1 − k/n), and set ∆Znk ≡ Znk − Zn,k−1 for integers 1 ≤ k ≤ n − 1. Then

n k n k−1 n−k ∆Znk = n−k − n−k R n+1 R n−k+1 n+1 (d)

=

n √1 n n−k

cnDnk +

1 n−k+1

k−1 j=1

cnDnj .

Let Ak ≡ σ[Dn1 , . . . , Dnk ]. Then E(∆Znk |Ak−1 ) = (e)

=

n √1 n n−k

= 0,

n √1 n n−k

1 n−k+1

[E(cnDnk |Ak−1 ) +

n j=k

since c¯n = 0.

cnDnj +

1 n−k+1

1 n−k+1

k−1 j=1

k−1 j=1

cnDnj

cnDnj ]

334


Apply the finite sampling results (9.1.8) and (9.1.9) to (d) to conclude that

2 σ2 (k−1)−1 k−1 n 2 k−1 Var[∆Znk ] = c,n (f) 1 − n−k+1 n n−k n−1 + (n−k+1)2 1 − n−1 (7)

=

2 σc,n n2 . n − 1 (n − k)(n − k + 1)

Let Ak ≡ σ[ξn:1 , . . . ξn:k ]. Then (g)

E(ξn:k |Ai ) −

k n+1

= ξn:i +

k−i n−i+1

[1 − ξn:i ] −

k n+1

=

n−k+1 n−i+1

ξn:i −

i n+1

,

since the conditional distribution of ξn:k given ξn:i is that of the (k − i)th order statistic in a sample of size n − i from Uniform(ξn:i , 1), and (9.1.32) can be applied. Thus (4) holds. 2 ¨n, Inequality 11.1 (Pyke–Shorack) Let X denote one of the processes Un , V Wn , Rn , or U. Let q > 0 on [0, θ] be and right continuous. Then for all λ > 0 we have the probability bound θ (8) P (X/qθ0 ≥ λ) ≤ (16/λ2 ) 0 [q(t)]−2 dt . Proof. Let X denote any one of Un , Wn , or U. Then X(t)/(1 − t) is a mg with mean 0 and variance ν(t) ≡ t/(1 − t). Thus the Birnbaum–Marshall inequality gives P ( X(t)/q(t) θ0 ≥ λ) = P ( [X(t)/(1 − t)]/[q(t)/(1 − t)] θ0 ≥ λ) (a)

≤ (4/λ)2

θ 0

[(1 − t)/q(t)]2 d [t/(1 − t)] = (4/λ)2

θ 0

[q(t)]2 dt .

Let X denote Rn . Then, with bk ≡ q(k/(n + 1)) and m ≡ [(n + 1)θ], P ( Rn (t)/q(t) θ0 ≥ λ) = P (max1≤k≤m | Rn (k/(n + 1)) |/bk ≥ λ)

≤ P max1≤k≤m |R n(k/(n+1))/(1−k/n)| ≥λ bk /(1−k/n) (b)

(c)

m

Var[∆Znk ] k=1 [bk /(1−k/n)]2

≤

4 λ2

≤

4 λ2

≤

4 1 λ2 n

by (7) and H´ ajek–Rényi

m

(n−k)2 n2 1 k=1 n−1 (n−k)(n−k+1) n2 b2k

m k=1

b−2 k ≤

16 λ2

θ 0

by (7)

[q(t)]−2 dt .

(We can improve (a) and (c) by a factor of 4, as stated in the H´ ajek–Rényi inequality, but there is no real point to this.) 2

Exercise 11.1

¨n. Verify (8) for V

11.


Inequality 11.2 (In probability linear bounds on Gn and G−1 n )

> 0 there exists λ ≡ λ so small that the event An on which

335 For all

(9)

Gn (t) ≤ t/λ on [0, 1],

Gn (t) ≥ λt on [ξn:1 , 1],

(10)

Gn (1−t) ≤ 1−λ(1−t) on [0, ξn:n ), √ |Gn (t) − t| ≤ 1/(λ n ) on [0, 1]

Gn (1−t) ≥ 1−(1−t)/λ on [0, 1] ,

(11)

has P (An ) ≥ 1 − for all n ≥ 1. Let 1n denote the indicator function of An . (These conclusions hold for any realization of Gn and G−1 n .) (Note that linear bounds on G−1 n are also established by this result.) Proof.

Now, (ξn:1 , . . . ξn:n ) has joint density n! on its domain. Thus P (Gn (t) ≤ t/λ for 0 ≤ t ≤ 1) = P (ξn:k ≥ λk/n for 1 ≤ k ≤ n) 1 tn t3 t2 = λ λ(n−1)/n · · · λ2/n λ/n n! dt1 · · · dtn = · · ·

(a)

= n!

(b)

tn n!

−

λtn−1 n!

1 =1−λ λ

≥ 1 − /3

(c)

for all λ ≤ λ ≡ /3,

and for all n. This gives the upper bound of (9). And (10.3.20) gives Daniels’ equality P (Gn /I ≤ λ) = P (Gn (t) ≤ t/λ for 0 ≤ t ≤ 1) (12)

= P (ξn:k ≥ λk/n for 1 ≤ k ≤ n) = P (G−1 n (t) ≥ λt for all 0 ≤ t ≤ 1) =1−λ

for all 0 ≤ λ ≤ 1 .

We now turn to the lower bound of (9). Now, (13)

Sk ≡ n ξn:k+1 /k,

1 ≤ k ≤ n − 1,

is a reversed mg,

as a rearrangement of E(ξn:k |ξn:k+1 ) = [k/(k + 1)] ξn:k+1 shows. So, immediately, (d)

S˜k ≡ S(n−1)−k+1 = Sn−k = n ξn:n−k+1 /(n − k)

is a mg for 1 ≤ k ≤ n − 1.

Now calculate 1 − P (Gn (t) ≥ λt everywhere on [ξn:1 , 1]) (e)

[or = P (I/Gn 1ξn:1 > 1/λ)]

= P (ξn:k+1 > (k/n)/λ for some 1 ≤ k ≤ n − 1) = P (max1≤k≤n−1 Sk > 1/λ) = P (max1≤k≤n−1 S˜k > 1/λ) = P (max1≤k≤n−1 exp(rS˜k ) > exp(r/λ))

(f)

˜

≤ inf r>0 [e−r/λ EerSn−1 ] = inf r>0 [e−r/λ Eern ξn:2 ] by Doob’s (10.10.3)

336

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES 1

(g)

e−r/λ ernt n(n − 1) t(1 − t)n−2 dt n = inf r>0 e−r/λ 0 ers n(n − 1) (s/n) (1 − s/n)n−2 ds/n n ≤ inf r>0 e−r/λ 0 sers (1 − s/n)n−2 ds n since 1 − s/n ≤ e−s/n ≤ inf r>0 e−r/λ 0 s ers e−s e2 ds ∞ ≤ e2 inf r>0 e−r/λ 0 s exp(−(1 − r)s) ds

(h)

= e2 inf r>0 e−r/λ /(1 − r)2

(i)

= (e2 /4λ2 ) exp(−1/λ)

(j)

< /3

= inf r>0

0

from the mean of an exponential density since differentiation gives r = 1 − 2λ

for λ ≡ λ small enough. Thus the lower bound in (9) holds. Then (10) follows from (9) by symmetry. Finally, (11) holds since Un = Op (1). In fact, we have Chang’s inequality 1 P (I/G−1 n ξn:1 ≤ x) = P (Gn (t) ≥ t/x on all of [ξn:1 , 1])

(14)

≥ 1 − 2 x2 e−x

for all x > 1.

2

Proof. Consider Mason’s theorem 12.10.4. Apply the Pyke–Shorack inequality with divisor q(t) ≡ (a ∨ t)1−ν to obtain

(a)

(b)

P (nν (Gn (t) − t)/t1−ν ba ≥ λ) = P ( Un /t1−ν ba ≥ λ n(1/2)−ν ) b ≤ P (Un /qb0 ≥ λn1/2−ν ) ≤ 4 0 (a ∨ t)−(2−2ν) dt/(λ2 n1−2ν ) =

4 λ2 (an)1−2ν

+

4 λ2 n1−2ν

=

4 λ2 (an)1−2ν

−

4 λ2 n1−2ν

b a

·

t−(2−2ν) dt

1 b (1−2ν)t1−2ν |a

≤ 8(1 − 2ν)−1 /[λ2 (an)1−2ν ] .

Using a = 1/n, b = 12 and the symmetry about 12 gives .= / = = nν [Gn (t) − t] =1−1/n 16 1 = = (15) P = ≥λ ≤ . = 1−ν [t ∧ (1 − t)] (1 − 2ν) λ2 1/n But [0, 1/n] is easy (and [1 − 1/n, 1] is symmetric), since on [0, 1/n] we have (c)

nν |Gn (t) − t|/t1−ν ≤ (nt)ν [1 + Gn (t)/t] ≤ 1 + Gn (t)/t ;

and thus (9) gives .= / = = nν [Gn (t) − t] =1/n 1 = (16) P = ≥λ ≤ = [t ∧ (1 − t)]1−ν = λ−1 0

for λ > 1.

11.


337

¨ −1 and V ¨ n replacing Gn and We can repeat this same proof up to (13) with G n ¨ −1 , as the Un , because of the Pyke–Shorack inequality. Then (0, 1/n] is trivial for G n values on this whole interval are deterministically related to the value at 1/n. 2

Exercise 11.2 Prove the Pyke–Shorack theorem 12.10.2. [Hint. Model your proof on (a) of the previous proof, with a = 0 and b sufficiently small, and with theorem 12.10.1 sufficient on [b, 1 − b].]

338


12

Applications

Theorem 12.1 (Donsker) Let g : (D, D) → (R, B) denote an a.s. · -continuous mapping that is D-measurable. Then g(Un ) : (Ω, A, P ) → (R, B), and (1)

g(Un ) →p g(U)

as n → ∞

for the Un of (12.10.28),

(2)

g(Un ) →d g(U)

as n → ∞

for an arbitrary Un .

[These conclusions hold for D-measurable functionals g that are continuous in other · /q-metrics as well.] Exercise 12.1

Write out the easy details to prove this Donsker theorem.

Example 12.1 (Tests of fit) (i) Call F stochastically larger than F0 whenever PF (X > x) ≥ PF0 (X > x) for all x (with strict inequality for at least one x), and write F ≥s F0 . To test the null hypothesis H0 that F = F0 is true against the alternative hypothesis Ha that F ≥s F0 it is reasonable to reject the H0 claim that ∞ √ for large values of Birnbaum’s statistic Zn ≡ −∞ n [ Fn (x) − F0 (x) ] dF0 (x). Now suppose that H0 is true, with a continuous df F0 . Then ∞ 1 1 1 (3) Zn ∼ ). = −∞ Un (F0 ) dF0 = 0 Un (t) dt →d Z ≡ 0 U(t) dt ∼ = N (0, 12 (ii) Alternatively, one could form the Cramér–von Mises statistic ∞ √ (4) Wn ≡ −∞ { n [ Fn (x) − F0 (x) ]}2 dF0 (x) ∼ = = (5)

∞

2 −∞ Un (F0 ) dF0

1 0

→d =

U2n (t) dt

1 0

by (7.5.22)

when F0 is continuous, by (7.3.10)

U2 (t) dt

1 ∞ ∞ 1 1 k=1 φk (t) πk Zk }{ j=1 φj (t) πj Zj } dt 0{ for the orthonormal functions φk (t) ≡

√

(see below)

2 sin(πkt) on [0, 1]

and iid N(0, 1) rvs Zk

(6)

=

∞ ∞

=

∞

k=1

1 j=1 π 2 jk

1 k=1 π 2 k2

Zj Zk

1 0

φk (t)φj (t) dt

Zk2 .

This shows that Wn is asymptotically distributed as an infinite weighted sum of independent χ21 rvs. This representation of the limiting distribution has been used to provide tables. If Wn ≡ Wn (F0 ) is computed but a different df F is true, then ∞ ∞ 2 2 1 (7) n Wn = −∞ [ Fn − F0 ] dF0 →a.s. −∞ [ F − F0 ] dF0 > 0 .

12.

APPLICATIONS

339

[In statistical parlance, this shows that the Wn -test is consistent against any alternative F = F0 .] (iii) A third possibility is the Anderson–Darling statistic ∞ √ 1 2 { n [ Fn − F0 ]}2 Un (t) (8) dF0 = dt An ≡ F (1 − F ) t(1 − t) 0 0 −∞ 0 for F0 continuous 1 2 ∞ U (t) 2 1 →d dt ∼ = k(k+1) Zk . 0 t(1 − t)

(9)

2

k=1

Proof. (i) Consider Birnbaum’s Zn . Method 1: By (7.5.22) and then the change of variable theorem of (7.3.10) (with identity function H) one obtains the first two steps of (3). Apply Donsker for the third step. Appeal to (12.3.19) for the →d to a normal rv Z. Finally, appeal to Fubini’s theorem for both 1 1 1 (a) EZ = E 0 U(t) dt = 0 E(U(t)) dt = 0 0 dt = 0 and 11 E(Z 2 ) = E{ 0 0 U(s) U(t) ds dt} 11 11 = 0 0 E{U(s) U(t)} ds dt = 0 0 [s ∧ t − s t] ds dt 1t (b) = 0 0 s(1 − t) ds dt = 1/12 . Method 2: Apply (12.10.28) for 1 1 (c) |Un (t) − U(t)| dt ≤ 0 1 dt × Un − U →p 0 0

to replace step three in the above. Thus Zn →d Z. The rest of the justification of example 12.1 is outlined in exercise 12.2 and exercise 12.3. 2 Exercise 12.2 Consider the Cramér–von Mises statistic Wn . (I) Verify step (5). Use (12.10.28). (II) We now seek to justify the step representing U as an infinite series. To this end formally write ∞ U(t) = 1 φk (t) π1k Zk (p) for iid N(0, 1) rvs Zk and the orthonormal functions φk (·). First recall the group of trigonometric identities

(q)

sin(A + B) = sin A cos B + cos A sin B , 2 sin A cos B = sin(A + B) + sin(A − B) , cos(A + B) = cos A cos B − sin A sin B , 2 cos A sin B = sin(A + B) − sin(A + B) .

340


1 Use these to verify that 0 φj (t)φk (t) dt equals 0 or 1 according as j = k or j = k. [Think of this formal U(t) as an odd function on [−1, 1], and thus only these φk (·) are needed.] Then note that the Fourier coefficients and the Fourier series are 1 (r) U, φk = 0 U(t) φk (t) dt = π1k Zk , (s)

U=

∞

k=1 U, φk φk

=

∞

1 k=1 φk π k

Zk .

So, verify that the series in (p) converges a.s. and then everything so far for the formal U is rigorous. Then Parseval’s identity (note theorem 6.3.3) gives 1 2 ∞ ∞ 1 2 2 2 (t) 1 |U, φk | = 1 π 2 k 2 Zk . 0 U (t) dt = U = Finally, one needs to verify the step (u) below in the identity ∞ ∞ ∞ E{ j=1 φj (s) π1j Zj × k=1 φk (t) π1k Zk } = 1 π21k2 φk (s) φk (t) = (u)

2 π2

∞ 1

1 k2

sin(π k s) sin(π k t)

= s∧ t −st,

and thus the (originally formal) process U is in fact a Brownian bridge. Where did this idea come from? Verifying that 1 1 (v) on [0, 1] 0 Cov[s, t] φk (s) ds = π k φk (t) shows that Cov[s, t] ≡ Cov[U(s), U(t)] = s∧t−st has eigenvalues π1k with associated eigenfunctions φk (·) for k = 1, 2, . . . . [Recall the spectral decomposition of matrices in (9.3.2)–(9.3.4).] Exercise 12.3 Verify the results claimed for the Anderson–Darling statistic An . [Verifying →d will be a little trickier this time, since (12.10.30) will now be needed in place of (12.10.28).] The rest is roughly similar in spirit, but the details are now a geat deal more complicated. Fundamentally, one must now represent the covariance function Cov[s, t] = (s ∧ t − s t)/ s(1 − s) t(1 − t) as a convergent infinite series of orthonormal functions. (Hopefully, at least the approach is now clear. Providing the details is hard work.)

Chapter 13

Characteristic Functions 1

Basic Results, with Derivation of Common Chfs

Elementary Facts Definition 1.1 (Characteristic function) Let X be an arbitrary rv, and let F denote its df. The characteristic function of X (abbreviated chf ) is defined (for all t ∈ R) by ∞ φ(t) ≡ φX (t) ≡ EeitX = −∞ eitx dF (x) (1) ∞ ∞ ≡ −∞ cos(tx) dF (x) + i −∞ sin(tx) dF (x). With dF replaced by h dµ, we call this the Fourier transform of the signed measure h dµ. [We note that φX (t) exists for −∞ < t < ∞ for all rvs X, since |eitX(ω) | ≤ 1 for all t and all ω.] Proposition 1.1 (Elementary properties) (a)

φ(0) = 1 and |φ(t)| ≤ 1

(b)

φ(·) is uniformly continuous on R.

(c)

φaX+b (t) = eitb φX (at)

(d)

¯ φ(−t) = φ(t)

(e)

φ is real-valued if and only if X ∼ = −X.

(f) (g)

Let φ denote an arbitrary chf.

for all t ∈ R.

for all t ∈ R.

for all t ∈ R.

|φ(·)|2 is a chf [of the rv X s ≡ X−X , with X and X iid with chf φ]. n φn Xi (·) = i=1 φXi (·) when X1 , . . . , Xn are independent. 1

Proof. Now, (a), (c), (d), and (g) are trivial. (f) If X and X are independent with characteristic function φ, then φX−X = φX φ−X = φφ¯ = |φ|2 . (e) If X ∼ = −X, 341

342

CHAPTER 13. CHARACTERISTIC FUNCTIONS

then φX = φ¯X ; so φX is real. If φ is real, then φX = φ¯X = φ−X ; so X ∼ = −X by the uniqueness theorem below. For (b), we note that for all t |φ(t + h) − φ(t)| = | [expi(t+h)x −eitx ] dF (x)| ≤ |eitx | |eihx − 1| dF (x) ≤ |eihx − 1| dF (x) → 0 (a) as h → 0, by the DCT with dominating function 2. The converse of (g) is false. Let X1 ≡ X2 and X3 be two iid Cauchy(0, 1) rvs. We will see below that φCauchy (t) = exp(−|t|), giving φ2X1 (t) = φX1 +X2 (t) = φX1 +X3 (t) for all t. 2 Motivation 1.1 (Proving the CLT via chfs) In this chapter we present an alternative method for establishing Fn →d F . It is based on the fact (to be demonstrated below) that the complex exponential functions eit· on R, indexed by t ∈ R, form a limit determining class. Saying this another way, the chf φ determines the distribution P , or the df F (or the density f , if there is one). Thus (as is shown in the continuity theorem below) we can establish that Fn →d F by showing that φn (·) → φ(·) on R. Indeed, using just the elementary properties listed above, it is trivial to give an informal “proof” of the classical CLT. Thus, we begin by expanding the chf of one rv X as √ √ (a) φ(X−µ)/√n (t) = φX−µ (t/ n ) = Eeit(X−µ)/ n (b)

it (it)2 = E{1 + √ (X − µ) + (X − µ)2 /2 + o(t2 /n)} n n it (it)2 = 1 + √ E(X − µ) + E(X − µ)2 /2 + o(t2 /n) n n

(c)

= 1 + 0 − t2 σ2 /2n + o(t2 /n) = 1 − t2 [σ2 + o(1)]/2n.

(In section 13.4 we will make such expansions rigorous, and in section 13.5 we will estimate more carefully the size of the errors that were made.) Then the standardized sum of the iid rvs X1 , . . . , Xn is √ √ ¯ n − µ) = n (Xk − µ)/ n , (d) Zn ≡ n(X 1 and it has chf (e) (f) (g)

n φZn (t) = k=1 φ(Xk −µ)/√n (t) = [φ(X−µ)/√n (t)]n n 2 2 t2 [σ2 + o(1)] = 1− → e−t σ /2 2n = φN (0,σ2 ) (t)

as will be shown below.

Since φZn (·) → φZ (·) on R, where Z ∼ = N (0, 1), the uniqueness theorem and the continuity theorem combine to guarantee that Zn →d Z. In principle, this is a rather elementary way to prove the CLT.

1.

BASIC RESULTS, WITH DERIVATION OF COMMON CHFS

343

Think of it this way. To have all the information on the distribution of X, we must know P (X ∈ B) for all B ∈ B. We have seen that the df F also contains all this information, but it is presented in a different format; a statistician may well regard this F format as the “tabular probability calculating format.” When a density f exists, it also contains all the information about P ; but it is again presented in a different format, which the statistician may regard as the “distribution visualization format.” We will see that the chf presents all the information about P too. It is just one more format, which we may well come to regard as the “theorem proving format.” 2

Some Important Characteristic Functions Distribution Binomial(n, p) Poisson(λ) GeometricF(p) Normal(µ, σ2 ) Exponential(θ) Chisquare(n) Gamma(r, θ) Uniform(0, 1) Double Exp(θ) Cauchy(0,1) de la Vallée Poussin Triangular

Density

n k n−k ;0 ≤ k ≤ n k p (1 − p) −λ k e λ /k!; k ≥ 0 pqk ; k ≥ 0 2 2 √ e−(x−µ) /2σ / 2πσ e−x/θ /θ on R+ x(n/2)−1 e−x/2 /[2n/2 Γ(n/2)] xr−1 e−x/θ /[θr Γ(r)] on R+ 1[0,1] e−|x|/θ /2θ 1/[π(1 + x2 )] (1 − cos x)/(πx2 ) on R [1 − |t|] × 1[−1,1] (t)

Chf [1 + p(eit − 1)]n exp(λ(eit − 1)) p(1 − qeit )−1 exp(itµ − σ2 t2 /2) (1 − itθ)−1 (1 − 2it)−n/2 (1 − itθ)−r [exp(it) − 1]/it 1/(1 + θ 2 t2 ) e−|t| [1 − |t|]1[−1,1] (t) 2(1 − cos x)/(x2 ) on R

Table 1.1

Review of Some Useful Complex Analysis A function f is called analytic on a region (a connected open subset of the complex plane) if it has a derivative at each point of the region; if it does, then it necessarily has derivatives of all orders inthe region. If z0 is an at each point m n −n , then isolated singularity of f and f (z) = ∞ n=0 an (z − z0 ) + n=1 bn (z − z0 ) k ≡ (the residue of f at z0 ) = b1 . Thus if f has a pole of order m at z0 (that is, bn = 0 for n > m in the expansion above), then g(z) ≡ (z − z0 )m f (z) is such that b1 = g (m−1) (z0 )/(m − 1)! . Thus b1 = k = (residue of f at z0 ) = lim (z − z0 )f (z) for a simple pole at z0 . z→z0

We also note that a smooth arc is described via equations x = φ(t) and y = ψ(t) for a ≤ t ≤ b when φ and ψ are continuous and not simultaneously zero. A contour is a continuous chain of a finite number of smooth arcs that do not cross the same point twice. Closed means that the starting and ending points are identical. (See Ahlfors (1953, pp. 102, 123) for what follows.)

344


Lemma 1.1 (Residue theorem) If f is analytic on a region containing a closed contour C, except for a finite number of singularities z1 , . . . , zn interior to C at which f has residues k1 , . . . , kn , then (for counterclockwise integration over C) n g(z) (2) f (z) dz = 2πi kj = dz = 2πi g(z0 ) for one simple pole . C C z − z0 j=1 Lemma 1.2 Let f and g be functions analytic in a region Ω. Suppose that f (z) = g(z) for all z on a set that has an accumulation point in Ω. We then have the equality f (z) = g(z) for all z ∈ Ω.

Evaluating Various Characteristic Functions Example 1.1 (Derivation of the Cauchy(0, 1) chf) Let t > 0. Let C denote the upper semicircle centered at the origin with radius R parametrized counterclockwise; and let A (for arc) denote C without its base. Then the Cauchy chf satisfies 1 eitz 1 1 (a) 2πi × Residue at i = eitz dz π 1 + z2 π 1 + z2 C (b)

(c)

1 = π →

1 π

R

1 1 eitx dx + 1 + x2 π

−R

∞

−∞

A

1 eitz dz 1 + z2

1 eitx dx + 0 = φ(t) 1 + x2

as R → ∞,

since the second integral in (b) is bounded in absolute value by 1 1 1 1 dz = πR → 0 as R → ∞. π A R2 − 1 π R2 − 1 Moreover, we know from (2) that the left-hand side of (a) equals (d)

2πi(z − i)

eitz |z=i = e−t π(1 + iz)(1 − iz)

for t > 0.

Since the Cauchy is symmetric, φ(−t) = φ(t) = exp(−|t|); or integrate the contour clockwise with t < 0. 2 Example 1.2 (Derivation of the N (0, 1) chf) √ ∞ 2 φ(t) = −∞ eitx (e−x /2 / 2π ) dx.

Let X be N (0, 1). Then

Let us instead think of φ as a function of a complex variable z. That is, √ ∞ 2 φ(z) = −∞ eizx e−x /2 dx/ 2π. Let us define a second function ψ on the complex plane by ψ(z) ≡ e−z

2

/2

.

1.

BASIC RESULTS, WITH DERIVATION OF COMMON CHFS

345

Clearly, φ and ψ are analytic on the whole complex plane. Let us now consider the purely imaginary line z = iy. On this line it is clear that ψ(iy) = ey

2

/2

,

and since elementary calculations show that √ √ ∞ ∞ 2 2 2 2 φ(iy) = −∞ e−yx e−x /2 dx/ 2π = −∞ e−(x+y) /2 dx ey /2 / 2π = ey /2 , we have ψ = φ on the line z = iy. Thus lemma 1.2 implies that ψ(z) = φ(z) for all z in the plane. Thus φ(t) = ψ(t) for all real z = t. That is, φ(t) = exp(−t2 /2)

for all real t

as claimed. (A similar approach works for the gamma distribution in exercise 1.3 below.) 2 Exercise 1.1 Derive the N (0, 1) chf via the residue theorem. Then extend to N (µ, σ2 ). [Hint. Let C denote a closed rectangle of height t with base [−R, R] on the x-axis.] Exercise 1.2 (a) Derive the Poisson(λ) chf (by summing power series). (b) Derive the Geometric(p) chf. (c) Derive the Bernoulli(p), Binomial(n, p), and NegBiT(m, p) chfs. Exercise 1.3 (a) Derive the Gamma(r, θ) chf. [Hint. Note example 1.2.] (b) Determine the Exponential(θ), Chisquare(n), and Double Exponential(θ) chfs. (c) Determine the Logistic(0, 1) chf. Exercise 1.4

Show that the real part of a chf (or Re φ(·) ) is itself a chf.

346

2


Uniqueness and Inversion

For the chf to be a useful tool, there must be a 1-to-1 correspondence between dfs and chfs. The fact that this is so is called the uniqueness theorem. We give a simple proof of the uniqueness theorem at the end of this subsection. But the simple proof does not establish an inversion formula that expresses the df as a function of the chf. In order to establish an inversion formula, we will need some notation, and an inversion formula useful for other purposes will require a hypothesis on the chf that is strong enough to allow some useful simplification. Let U denote a rv with continuous density fU (·), and let W denote a rv with a bounded and continuous density fW (·) and with chf φW (·); and suppose we are lucky enough to determine a complementary pair that (for some constant c) satisfy the relationship (1)

fU (t) = c φW (−t)

for all real t.

(Complementary pair)

We give three examples of such pairs. Let Z ∼ = N (0, 1), T ∼ = Triangular(0, 1), and let D have the de la Vallée Poussin density. Then examples of (1) are √ (2) U =Z and W = Z, with c = 1/ 2π, (3)

U =T

and

W = D,

with c = 1,

(4)

U =D

and

W = T,

with c = 1/2π.

(The Cauchy(0, 1) and the Double Exponential(0, 1) then lead to two additional complementary pairs.) (The beauty of this is that we can nearly eliminate the use of complex analysis.) An arbitrary rv X, having df FX (·) and chf φX (·), may not have a density. Let us recall from the convolution formula (9.2.2) that (if U has a density) a slightly perturbed version Xa of X is smoother than X, in that (5)

Xa ≡ X + aU always has a density fa (·) ; and Xa →d X as a → 0

by Slutsky’s theorem, since a U →p 0 as a → 0. Thus F (·) = lim Fa (·) at each point in the continuity set CF of F . This is the key to the approach we will follow to establish an inversion formula. Theorem 2.1 (Uniqueness theorem)

Every df on the line has a unique chf.

Theorem 2.2 (Inversion formula) If an arbitrary rv X has df FX (·) and chf φX (·), we can always write r2 (6) FX (r2 ) − FX (r1 ) = lim fa (t) dt for all r1 < r2 in CFX , a→0

r1

where the density fa (·) of the rv Xa ≡ X + aU of (5) [with U as in (1)] is given by ∞ (7) e−itv φX (v)cfW (av) dv for all t ∈ R. fa (t) = −∞

2.

UNIQUENESS AND INVERSION

347

Theorem 2.3 (Inversion formula for densities) that satisfies the integrability condition ∞ (8) |φX (t)| dt < ∞,

If a rv X has a chf φX (·)

−∞

then X has a uniformly continuous density fX (·) given by ∞ 1 (9) fX (x) = e−itx φX (t) dt. 2π −∞ Remark 2.1 The uniqueness theorem can be restated as follows: The set of complex exponentials G ≡ {eitx for x ∈ R : t ∈ R} is a limit determining class. This is so because knowing all values of φX (t) = EeitX allows the df F to be determined, via the inversion formula. 2 Proof.

(a)

From the convolution formula (9.2.2) we have ∞ fa (t) = −∞ a1 fU ( t−x a ) dFX (x) =c

∞

1 −∞ a

φW ( x−t a ) dFX (x)

by (1)

∞ ∞ = (c/a) −∞ −∞ ei(x−t)w/a fW (w) dw dFX (x) ∞ ∞ = (c/a) −∞ e−itw/a fW (w) −∞ ei(w/a)x dFX (x) dw (b)

by Fubini

∞ ∞ = (c/a) −∞ e−itw/a φX (w/a) fW (w) dw = c −∞ e−itv φX (v) fW (av) dv.

Since Xa →d X, at continuity points r1 < r2 of F we have (with Xa ∼ = Fa (·)) r2 (c) FX (r2 ) − FX (r1 ) = lim {Fa (r2 ) − Fa (r1 )} = lim r1 fa (t) dt. a→0

a→0

This establishes theorems 2.1 and 2.2. The particular formula given in (c) is essentially useless, but the mere fact that one can recover FX from φX via some formula is enough to establish the important property of uniqueness. We now turn to theorem 2.3, in which we have added a hypothesis that allows the previous formula to be manipulated into a simple and useful form. Suppose that (8) holds, so that applying the DCT to (b) (using a constant times |φX (·)| as a dominating function) gives [recall the hypothesis on the fW (·) of (1)] as a → 0 that ∞ (d) fa (t) → f (t) ≡ [cfW (0)] −∞ e−itv φX (v) dv, since fW is bounded and is continuous at 0. Note that uniform continuity of f follows from the bound ∞ |f (t) − f (s)| = [cfW (0)] | −∞ [e−itv − e−isv ]φX (v) dv| (e)

≤ [cfW (0)]

∞

−∞ |e

−i(t−s)v

− 1| |φX (v)| dv → 0

as t − s → 0,

348


by applying the DCT (with dominating function [cfW (0)]|φX (·)|. The uniform convergence of fa to f on any finite interval involves only an |fW (0) − fW (av)| term under the integral sign. That f really is the density of FX follows from applying this uniform convergence in (c) to obtain r (f) FX (r2 ) − FX (r1 ) = r21 f (t) dt . Specifying U = W = Z gives [cfW (0)] = 1/(2π) (as it always must), so that the conclusion (9) holds. 2 Esseen’s inequality 13.5.1 below provides an important extension of theorem 2.2 by showing that if two chfs are sufficiently close over most of their domain, then the corresponding dfs will be uniformly close over their entire domain. Exercise 2.1 Show that setting W = Z in line (c) of the previous proof leads, for any rv X, to the alternative inversion formula ∞ −itr2 2 2 1 e − e−itr1 (10) FX (r2 ) − FX (r1 ) = lim φX (t) e−a t /2 dt a→0 2π −∞ −it at all continuity points r1 < r2 of FX (·). [This is one possible realization of (6).] Exercise 2.2 Derive the chf of the Triangular(0, 1) density via an elementary integration. Then use theorem 2.3 to derive the chf of de la Vallée Poussin density, while simultaneously verifying that the nonnegative and real integrable function (1 − cos x)/(πx2 ) really is a density. Following section 13.4, determine E|X| when X has the de la Vallée Poussin density. Exercise 2.3 (Kernel density estimator) Since the rv X having df FX (·) and chf φX (·) may not have a density, we choose instead to estimate the density fa (·) of (5) and (7) using ∞ fâ (x) ≡ c (11) e−itx φˆX (t)fW (at) dt −∞

[where fU (·) = cφW (−·), and where we now insist on EU = 0] with the empirical chf φˆX (·) defined by ∞ n 1 itXj ˆ φX (t) ≡ (12) eitx d Fn (x) = e for − ∞ < t < ∞. n j=1 −∞ (a) Verify that fâ (·) is actually a kernel density estimator, meaning that it is expressible as n 1 ∞ x−y 1 1 x − Xj ˆ (13) fa (x) = fU d Fn (y) = fU . a −∞ a n j=1 a a [This has statistical meaning, since we are averaging densities centered at each of the observations.]

2.

UNIQUENESS AND INVERSION

349

(b) Show that fâ (x) is always unbiased (in that it has mean fa (x)) and has a variance we can calculate; thus for all x ∈ R we have (14) (15)

Efâ (x) = fa (x) , ∞ x−y 1 1 2 2 ˆ dF (y) − [fa (x)] . Var[fa (x)] = f n a2 −∞ U a (2)

(c) Supposing that FX (·) has a density f (·) ∈ Cb , determine the order of the mean squared error (16)

MSE{fâ (x)} ≡ Bias{fâ (x)} + Var[fâ (x)] ≡ {E(fâ (x)) − f (x)}2 + Var[fâ (x)]

of fâ (x), viewed as an estimator of f (x). Show that this MSE is of order n−4/5 when a is of order n−1/5 , and that this is the minimal attainable order. (d) Note that the choice U = Z (or U = T ) leads to an fâ (·) that is the sum of n normal (or triangular) densities that are centered at the n data points and that have a scale parameter directly proportional to a. (e) Obtain an expression for lima→0 a4/5 MSE {fâ (x)} in terms of f (x), f (x), and f (x) when a = n−4/5 (and obtain it for both of the choices U = Z and U = T ). (f) We could also motivate this idea of a kernel density estimator based on (13) alone. How much of what we have done still carries over for a general kernel? What properties should a good kernel exhibit? What can you prove in this more general setting? Exercise 2.4 Use the table of chfs above to show in what sense the sums of independent Binomial, Poisson, NegBiT, Normal, Cauchy, Chisquare and Gamma rvs have distributions that again belong to the same family. (Recall section 9.2, noting that chfs have allowed the complicated operation of convolution of dfs or densities to be replaced by the simple operation of multiplication of chfs.)

350

3


The Continuity Theorem

Theorem 3.1 (Continuity theorem for chfs; Cram´ er–L´ evy) (i) If φn → φ where φ is continuous at 0, then φ is the chf of a bona fide df F and Fn →d F . (ii) Fn →d F implies φn → φ uniformly on any finite interval |t| ≤ T . Inequality 3.1 (Chf bound on the tails of a df) 1/λ (1) P (|X| ≥ λ) ≤ 7λ 0 [1 − Real φ(t)] dt Proof. (a)

For any df F we have for all λ > 0.

Now, 1/λ 1/λ ∞ λ 0 [1 − Real φ(t)] dt = λ 0 −∞ [1 − cos(tx)] dF (x) dt ∞ 1/λ = −∞ λ 0 [1 − cos(tx)] dt dF (x) 1/λ ∞ = −∞ (λ/x)[xt − sin(xt)]0 dF (x) ∞ = −∞ [1 − sin(x/λ)/(x/λ)] dF (x) ≥ [|x|/λ≥1] [1 − sin(x/λ)/(x/λ)] dF (x)

(b)

= inf [|y|≥1] [1 − sin(y)/y] P (|X| ≥ λ) = [1 − sin(1)] P (|X| ≥ 1/ )

(c)

= (.1585 . . . )P (|X| ≥ λ) ≥ P (|X| ≥ λ)/7,

as claimed. [It may be instructive to compare this to Chebyshev’s inequality.]

2

Proof. Consider theorem 3.1. (i) The uniqueness theorem for chfs shows that the collection G of complex exponential functions form a determining class, and the expectations of these are hypothesized to converge. It thus suffices (by the kinder and gentler Helly–Bray theorem (theorem 11.8.1(i)(a))) to show that {Fn : n ≥ 1} is tight. Now, 1/λ (a) limn→∞ P (|Xn | ≥ λ) ≤ limn→∞ 7λ 0 [1 − Real φn (t)] dt 1/λ = 7λ 0 [1 − Real φ(t)] dt by the DCT, with dominating function 2 (b)

→0

as λ → ∞ ,

so that {Fn : n ≥ 1} is tight. (ii) Now replacing Xn →d X by versions Yn →a.s. Y (and using Skorokhod’s construction) gives for |t| ≤ T that (c) |φn (t) − φ(t)| ≤ |eitYn − eitY | dP ≤ |eit(Yn −Y ) − 1| dP ≤ sup|t|≤T |eit(Yn −Y ) − 1| dP (d)

→0

by the DCT, with dominating function 2.

2

3.

THE CONTINUITY THEOREM

351

Higher Dimensions ≡ (X1 , . . . , Xk ) from If X1 , . . . , Xk are rvs on (Ω, A, P ), then the Bk -A-mapping X is Ω to Rk induces a measure PX on (Rk , Bk ). The characteristic function of X (2)

φX ( t ) ≡ E eit

X

= E ei [t1 X1 +···+tk Xk ]

for t ≡ (t1 , . . . , tk ) ∈ Rk .

Without further explanation, we state simply that the uniqueness theorem and the Cramér–Lévy continuity theorem still hold, based on minor modifications of the previous proof. We also remark that all equivalences of →d in theorem 11.7.1 are still valid. But we now take up an extremely useful approach to showing convergence in distribution in higher dimensions. is The characteristic function of the one-dimensional linear combination λ X (3)

φλ X (t) = E ei [tλ1 X1 +···+tλk Xk ]

for t ∈ R.

Comparison of this with (2) shows that knowing φX ( t ) for all t ∈ Rk is equivalent to knowing φλ X (t) for all t ∈ R and λ ∈ Rk with |λ| = 1. This immediately yields the following useful result. Theorem 3.2 (Cram´ er–Wold device) (4)

φλ X n (t) → φλ X (t)

for all t ∈ R

If Xn ≡ (Xn1 , . . . , Xnk ) satisfy and for each λ ∈ Rk ,

n →d X. (It suffices to show (4) for all unit vectors λ in Rk .) [In fact, what then X n →d λ X for all such λ (no matter what method we use to we require is that λ X show it).] Theorem 3.3 The rvs kX1 , . . . , Xk are independent if and only if the joint chfs satisfy φX (t1 , . . . , tk ) = 1 φXi (ti ). Exercise 3.1

Prove theorem 3.3.

352

4


Elementary Complex and Fourier Analysis

Lemma 4.1 (Taylor expansions of log(1 + z) and eiz ) [Note that log z is a many-valued function of a complex z = reiθ ; any of (log r) + i[θ + 2πm] for m = 0, ±1, ±2, . . . will work for log z. However, when we write log z = log r + iθ, we will always suppose that −π < θ ≤ π. Moreover, we denote this unique determination by Log z; this is the principal branch.] The Taylor series expansion of Log (1 + z) gives ∞ m−1 |Log (1 + z) − k=1 z k /k| = | k=m z k /k| (1) m |z|m ≤ |z|m (1 + |z| + |z|2 + · · ·) ≤ m(1−|z|) for |z| < 1. Thus (2)

|Log (1 + z) − z| ≤ |z|2 /(2(1 − θ))

for |z| ≤ θ < 1.

From another Taylor series expansion we have for all z that (3)

|ez −

m−1 k=0

∞ |z|j ∞ j! |z|m e|z| z k /k! | = | k=m z k /k! | ≤ |z|m j=0 ≤ . j! (j + m)! m!

Lemma 4.2 (Taylor expansion of eit ) Let m ≥ 1 and 0 ≤ δ ≤ 1, and require that m + δ > 0. Then for all real t we have m δ21−δ it (it)k (4) |t|m+δ ≡ Km,δ |t|m+δ . e − ≤ k! (m + δ) · · · (2 + δ)(1 + δ)(0 + δ) k=0

Proof. The proof is by induction. For m = 0 we have both |eit −1| ≤ 2 ≤ 2 |t/2|δ for |t/2| ≥ 1, and t |t| (a) |eit − 1| ≤ | 0 ieis ds| ≤ 0 ds = |t| ≤ 2 |t/2|δ for |t/2| ≤ 1; so that (4) holds for m = 0. We now assume that (4) holds for m − 1, and we will t verify that it thus holds for m. Since the formula eit − 1 = i 0 eis ds holds and we m−1 t k k also have m 1 (it) /k! = i k=0 0 [(is) /k!] ds, then it m t is m−1 k k e − = i (it) /k! e − (is) /k! ds k=0 k=0 0 |t|

(b)

≤ Km−1,δ

(c)

= Km,δ |t|m+δ .

0

sm−1+δ ds

by the induction step

[The next inequality is immediate. See Chow and Teicher (1997).]

2

Inequality 4.1 (Moment expansion inequality) Suppose E|X|m+δ < ∞ for some m > 0 and 0 ≤ δ ≤ 1. Then m (it)k (5) EX k ≤ Km,δ |t|m+δ E|X|m+δ for all t. φ(t) − k! k=1

4.

ELEMENTARY COMPLEX AND FOURIER ANALYSIS

353

Some Alternative Tools Lemma 4.3 (The first product lemma) satisfy the following conditions: n (a) as n → ∞. θn ≡ 1 θnk → θ (b) (c)

δn ≡ [ max1≤k≤n |θnk | ] → 0. Mn ≡ nk=1 |θnk | satisfies

For all n ≥ 1, let complex θn1 , . . . , θnn

δn Mn → 0.

Then (6)

n >

(1 + θnk ) → eθ

as n → ∞.

k=1

Proof. As soon as n is so large that 0 < δn ≤ 12 (and we are on the principal branch), (2) gives n n n (p) | k=1 Log (1 + θnk ) − k=1 θnk | ≤ k=1 |θnk |2 ≤ δn Mn → 0. Thus (q)

n k=1

Log (1 + θnk ) → θ

as n → ∞.

Moreover, (p) shows that n n (r) Log ( k=1 (1 + θnk )) = k=1 Log (1 + θnk ). Because of (r), (q) implies (6). [See Chung (1974).]

2

Lemma 4.4 (The second product lemma) If z1 , . . . , zn and w1 , . . . , wn denote complex numbers with modulus at most 1, then n n n > > (7) zk − wk ≤ |zk − wk | . k=1

Proof. (a) (b)

k=1

k=1

This is trivial for n = 1. We will use induction. Now, n−1 n−1 | nk=1 zk − nk=1 wk | ≤ zn n−1 k=1 zk − k=1 wk + |zn − wn | k=1 wk n−1 n−1 n−1 n−1 ≤ k=1 zk − k=1 wk +|zn − wn |· k=1 1 ≤ k=1 |zk − wk |+|zn − wn |

by the induction step. [See most newer texts.]

2

Inequality 4.2 (Moment expansions of chfs) Suppose E|X|m < ∞ for some m ≥ 0. Then its chf satisfies m (it)k (8) EX k |t|m → 0 as t → 0. φ(t) − k! k=0

354 Proof. (a) (b)

CHAPTER 13. CHARACTERISTIC FUNCTIONS Use the real expansions for sin and cos to obtain eitx = cos(tx) + i sin(tx) = =

m k=0

(itx)k k!

+

(itx)m m!

m−1 k=0

(itx)k k!

+

(itx)m m!

[cos(θ1 tx) + i sin(θ2 tx)]

[cos(θ1 tx) + i sin(θ2 tx) − 1].

Here, we have some θ1 , θ2 with 0 ≤ |θ1 | ∨ |θ2 | ≤ 1. Then (8) follows from (b) via (9)

limt→0 E|X m [cos(θ1 tX) − 1 + i sin(θ2 tX)]| = 0,

by the DCT with dominating function 3|X|m . [See Breiman (1968).]

Inequality 4.3 (Summary of useful facts) gives the highly useful φ(t) − [1 − σ2 t2 /2] = o(t2 )

2

Let X ∼ = (0, σ2 ). Result (8) then

as t → 0,

while applying (5) twice gives (10)

φ(t) − [1 − σ2 t2 /2] ≤ σ 2 t2 ∧ E|X|3 |t|3 /6

for all t ∈ R.

Exercise 4.1 (Distributions determined by their moments) (a) Suppose that E|X|n < ∞. Then the nth derivative φ(n) (·) is a continuous that EX n = i−n φ(n) (0). function given by φ(n) (t) = in E(X n eitX ), so ∞ (b) If all moments are finite, then φ(t) = n=0 (it)n EX n /n! within the radius of convergence. (c) If the radius of convergence in (b) is strictly positive, then the distribution having the stated moments is uniquely determined by its moments. (d) The normal distribution is uniquely determined by its moments. (e) The series in (b) converges for |t| < r if and only if E exp(t |X|) < ∞ for |t| < r. Exercise 4.2 (a) If φ (0) is finite, then σ2 is finite. Prove this. (b) In fact, if φ(2k) (0) is finite, then EX 2k < ∞. Prove this. Exercise 4.3 (Bounds on (1 − x/n)n ) 0 ≤ t < 1 at t = x/n to show that 0 ≤ e−x − (1 − nx )n ≤

x2 n

e−x

(i) Use (1 + t) ≤ et ≤ 1/(1 − t) for for 0 ≤ x < n .

(ii) (Hall and Wellner) Show that 2 e−2 ≤ n supx≥0 e−x − (1 − x/n)n 1[0,n] (x) ≤ (2 + n−1 )e−2 for all n ≥ 1.

4.


355

Results from Fourier Analysis On some occasions we will need to know the behavior of φ(t) for |t| large. Lemma 4.5 (Riemann–Lebesgue lemma)

∞

(11) −∞

eitx g(x)dx → 0

If

∞ −∞

|g(x)|dx < ∞, then

as t → ∞.

m Proof. Now, Ψ ≡ {ψ ≡ 1 ci 1(ai ,bi ] : ai , bi , ci ∈ R and m ≥ 1} is dense in L1 ∞ by theorem 3.5.8; that is, if −∞ |g(x)|dx < ∞, then there exists ψ ∈ Ψ such that ∞ ∞ |g − ψ| dx < . Thus γ(t) ≡ | −∞ eitx g(x)dx| satisfies −∞ γ(t) ≤

∞

≤ +

(a)

−∞ |e

m 1

itx

∞ | |g(x) − ψ(x)| dx + | −∞ eitx ψ(x) dx|

b |ci | | aii eitx dx|.

It thus suffices to show that for any a, b in R we have

b

eitx dx → 0

(12)

as |t| → ∞.

a

A quick picture of sines and cosines oscillating very fast (and canceling out over the interval) shows that (12) is trivial. (Or write eitx = cos(tx) + i sin(tx) and compute the integrals.) 2

Lemma 4.6 (Tail behavior of chfs) If F has density f with respect to Lebesgue measure, then φ(t) → 0 as |t| → ∞. If F has n+1 integrable derivatives f, f , . . . , f (n) on R, then (13)

tn φ(t) → 0

as |t| → ∞.

Proof.

The first statement is immediate. Then use φ(t) = eitx f (x) dx = f (x) d(eitx /it) ∞ = (eitx /it) f (x)−∞ − eitx f (x) dx/(it)

(a)

= − eitx f (x) dx/(it)

with

f (·) ∈ L1 .

Applying the Riemann–Lebesgue lemma to (a) gives tφ(t) → 0 as |t| → ∞. Keep on integrating by parts and applying the Riemann–Lebesgue lemma. 2

356


Other Alternative Tools Then chf always exists, so it can always be used. However, if X ≥ 0 or if X is integer valued, then Laplace transforms or probability generating functions offer more elementary tools. Exercise 4.4 (Laplace transform) Let F + denote the class of all dfs F having + F− (0) = 0. For any df F ∈ F we define the Laplace transform L of F by ∞ L(λ) = E e−λX = 0 e−λx dF (x) (14) for λ ≥ 0. (a) Establish an analogue of proposition 1.1(a), (b), (c), and (g). (b) (Uniqueness) Show that each df in F+ has a unique Laplace transform. (c) (Continuity) Let Xn ∼ = Fn ∈ F + . If Ln (λ) → (some L(λ)) for all λ ≥ 0 with L(·) right continuous at 0, then L is the Laplace transform of a df F ∈ F + for which the convergence in distribution Fn →d F holds. (d) Establish analogues of inequality 4.1 on moment expansions. Exercise 4.5 (Probability generating function) Let F I denote the class of all dfs F assigning mass 1 to the integers 0, 1, 2, . . . . For any df F ∈ F I we define the probability generating function g of F by ∞ (15) g(z) = E z X = k=0 pk z k for all complex z having |z| ≤ 1. (a) Establish an analogue of proposition 1.1. (b) (Uniqueness) Show that each df F in FI has a unique generating function. (c) (Continuity) Let Xn ∼ = Fn ∈ F I . If gn (z) → (some g(z)) for all |z| ≤ 1 with g(·) continuous at 1, then g is the generating function of a df F in FI for which Fn →d F . Exercise 4.6 (Cumulant generating function) ψX (·) of a rv X is defined by (16)

The cumulant generating function

ψX (t) ≡ Log φX (t) = Log E(eitX ),

and is necessarily finite for t-values in some neighborhood of the origin. Temporarily suppose that all moments of X are finite. Let µk ≡ E(X − µ)k , for k ≥ 1. Then when µ = EX = 0 and with σ2 ≡ µ2 , we have φX (t) = 1 − t2 σ2 /2 + (it)3 µ3 /3! + (it)4 µ4 /4! + · · · ≡ 1 + z . We would thus have ψX (t) = Log φX (t) = Log (1 + z) = z − z 2 /2 + z 3 /3 + · · · = (it)2 µ2 /2! + (it)3 µ3 /3! + (it)4 (µ4 − 3µ22 )/4! + · · · (17) (18)

= (it)2 σ2 /2! + (it)3 µ3 /3! + t4 (µ4 − 3σ 4 )/4! + · · · ∞ ≡ j=2 (it)j κj /j! ,

4.


357

where κj is called the jth cumulant of X. Note that for independent rvs X1 , . . . , Xn , n n (the jth cumulant of k=1 Xk ) = k=1 (the jth cumulant of Xk ), (19) which is √ nice. In the iid case, the third and fourth cumulants of the standardized ¯ n − µ)/σ are rv Zn ≡ n(X √ √ γ1 / n ≡ (µ3 /σ 3 )/ n (20) and γ2 /n ≡ (µ4 /σ 4 − 3)/n, where γ1 measures skewness and γ2 measures tail heaviness. [This is particularly ¯ n ) of skewness disappears at nice; it√ shows that the effect (on the distribution of X rate 1/ n, while the effect of tail heaviness disappears at rate 1/n.] Finally, if only E|X|m < ∞ for some m ≥ 1, then in a sufficiently small neighborhood of the origin j m m (21) |ψ(t) − m j=2 κj (it) /j! | ≤ cm |t| E|X| δm (t), where δm (t) 0 as t 0, and m−1 (22) |ψ(t) − j=2 κj (it)j /j! | ≤ c¯m |t|m E|X|m for some universal constant c¯m . The exercise is to establish carefully that all of this is true.

358

5


Esseen’s Lemma

Let G denote a fixed function having G(−∞) = 0, G(+∞) = 1, having derivative bounded by some constant M , having ∞ g on the real line for which |g(·)| is ∞ xg(x) dx = 0, and then let ψ(t) ≡ eitx g(x) dx. Let F denote a general df −∞ −∞ having mean 0, and let φ denote its characteristic function. We wish to estimate F − G = sup−∞<x 0 we have the uniform bound 1 a φ(t) − ψ(t) 24 g (1) F − G ≤ dt + . π −a t πa Proof. The key to the technique is to smooth by convolving F and G with the df Ha whose density ha and characteristic function γa are given by 1 − cos(ax) 1 − |t|/a if |t| ≤ a, ha (x) = (2) on R and γ (t) = a 0 if |t| > a. πax2 Let Fa and Ga denote the convolutions of F and G with Ha . We will now show that (3)

F − G ≤ 2Fa − Ga + 24 g/(πa).

Let ∆ ≡ F − G. Now, ∆(x) = ∆+ (x) and ∆− (x) exist for all x; thus there exists x0 such that either D ≡ F − G = |∆(x0 )| or D = |∆− (x0 )|. Without loss of generality, we suppose that D = |∆(x0 )| (just replace X, Y by −X, −Y if not). Note figure 5.1. Without loss of generality, we act below as though ∆(x0 ) > 0 (if ∆(x0 ) < 0, then z < x0 rather than z > x0 ). Now, since F is and g is bounded by M , we have (a)

∆(z − x) ≥ D/2 + M x

for |x| ≤ ,

where ≡ D/2M and z ≡ x0 + . Trivially (since D was the supremum), (b)

∆(z − x) ≥ −D

for |x| > .

Thus, with ∆a ≡ Fa − Ga , using (a) and (b) gives ∞ (c) Fa −Ga ≥ ∆a (z) = −∞ ∆(z−x)ha (x) dx (d)

≥

−

by the convolution formula

[D/2 + M x] ha (x) dx − D [|x|>] ha (x) dx

= (D/2)[1 −

[|x|>] ha (x) dx]

since xha (x) is odd

+M ·0−D

[|x|>] ha (x) dx

5.

ESSEEN’S LEMMA

359

= (D/2) − (3D/2) (e)

[|x|>] ha (x) dx

≥ (D/2) − (12M/πa)

= F − G/2 − (12M/πa),

(which is (3)), since ∞ 2 (f) [|x|>] ha (x) dx ≤ 2 (2/πax ) dx = 4/(πa ) = 8M/(πaD). F y = D – Mx

D

G D/2

ε x0

ε z

x

Figure 5.1 Bounds for Esseen’s lemma.

We now bound Fa − Ga . By the Fourier inversion formula, Fa and Ga have bounded continuous “densities” that satisfy a (4) fa (x) − ga (x) = −a e−itx [φ(t) − ψ(t)] γa (t) dt/(2π). From this we suspect that a 1 φ(t) − ψ(t) ∆a (x) = γa (t) dt . (5) e−itx 2π −a −it That the integrand is a continuous function that equals 0 at t = 0 (since F and G have 0 “means”, inequality 13.4.1 gives this) makes the right-hand side well-defined, and we may differentiate under the integral sign by the DCT [with dominating function γa (·)] to get the previous equation (4). Thus ∆a (x) can differ from the right-hand side of (5) by at most a constant; but this constant is 0, since obviously ∆a (x) → 0 as |x| → ∞, while the right-hand side does the same by the Riemann– Lebesgue lemma. Equation (5) gives a φ(t) − ψ(t) 1 dt (6) |∆a (x)| ≤ for all x. 2π −a t Combining (3) and (6) gives (1).

2

360


Corollary 1 (Stein) Suppose that instead of convolving F and G with the Ha of (2), we convolve with an arbitrary df H instead. In this situation we obtain (7) Proof. (d)

F − G ≤ 2 F ∗ H − G ∗ H + 8 g E|H −1 (ξ)| . Picking up at line (d) of the previous proof (with Y ∼ = H), we obtain F ∗ H − G ∗ H ≥ [−,] [D/2 + M y] dH(y) − D P (|Y | > ) ≥ (D/2) [1 − P (|Y | > )] − M E|Y | − D P (|Y | > ) ≥ (D/2) − (3D/2) P (|Y | > ) − M E|Y |

(e)

≥ (D/2) − 4M E|Y |

using Markov’s inequality and ≡ D/2M in the last step.

2

6.

DISTRIBUTIONS ON GRIDS

6

Distributions on Grids

361

Definition 6.1 We say that a rv X is distributed on a grid if there exist real numbers a, d such that the probabilities pn ≡ P (X = a + nd) satisfy ∞ −∞ pn = 1. We call d the span of the grid. The maximal span is sup{|d| : |d| is a span}. Proposition 6.1 If t0 = 0, the following are equivalent: (a) |φ(t0 )| = 1. (b) |φ| has period t0 ; that is, |φ(t + nt0 )| = |φ(t)| for all n and t. (c) The rv X is distributed on a grid of span d = 2π/t0 . Proof. Suppose that (a) holds. Then φ(t0 ) = eiα for some real α. That is, it x 0 e dF (x) = eiα , or ei(t0 x−α) dF (x) = 1. Taking real parts gives ∞ (p) [1 − cos(t0 x − α)] dF (x) = 0. −∞ Since the integrand is nonnegative for all X, this means that 1 − cos(t0 x − α) = 0

(q)

a.s. F ;

that is, (r)

t0 X − α ∈ {2πm : m = 0, ±1, ±2, . . .}

a.s.

That is, X ∈ {α/t0 + (2π/t0 )m : m = 0, ±1, ±2, . . .} a.s.; so (c) holds. Suppose (c) holds. Then (b) holds, since ∞ |φ(t + nt0 )| = | m=−∞ pm ei(t+nt0 )(a+dm) | ∞ = |ei(t+nt0 )a | | m=−∞ pm ei(t+nt0 )dm | =| (s)

=|

∞ m=−∞

∞ m=−∞

pm ei(t+2πn/d) dm| pm eitdm | |ei2πnm | = |φ(t)|.

Suppose that (b) holds. Then (t)

1 = |φ(0)| = |φ(0 + t0 n)| = |φ(0 + t0 1)| = |φ(t0 )|,

so that (a) holds.

2

Corollary 2 If a = 0 in (c), then we may replace |φ| by φ in (a) and (b), and proposition 6.1 will still hold. Proposition 6.2 One of the following possibilities must hold: (d) |φ(t)| < 1 for all t = 0. (e) |φ(t)| < 1 for 0 < t < 2π/d and |φ(2π/d)| = 1. Thus, X has maximal span d. (f) |φ(t)| = 1 for all t. And so φ(t) = eiat for all t and P (X = a) = 1, for some a.

362


Proof. Clearly, either (d), (e), or (f) holds, or else |φ(tn )| = 1 for some sequence tn → 0. In this latter case, |φ(mtn )| = 1 for all m, for each n by proposition 6.1. Since {mtn : n ≥ 1, m = 0, ±1, ±2, . . .} is dense in R and since φ, and thus |φ|, is continuous, we must have case (f) again. It remains to establish the consequences of (e) and (f). Consider (e). Proposition 6.1 shows that (e) holds if and only if both d is a span and no number exceeding d is a span. In the case of (f), we have |φ(t1 )| = 1 = |φ(t2 )| for some t1 and t2 having t1 /t2 = (an irrational number). But |φ(t1 )| = 1 and |φ(t2 )| = 1 imply that both 2π/t1 and 2π/t2 are spans. Thus if at least two points have positive mass, then the distance between them must equal m1 2π/t1 for some integer m1 and it must equal m2 2π/t2 for some integer m2 . That is, 2πm1 /t1 = 2πm2 /t2 , or t1 /t2 = m1 /m2 = (a rational number). This contradiction shows that there can be at most one mass point a. 2

Exercise 6.1 (Inversion formula for distributions on Let X be distributed a grid) it(a+dn) on a grid with pn = P (X = a + dn). Then φ(t) = ∞ p e . Show that n −∞ (1)

pm =

d 2π

π/d −π/d

φ(t) e−it(a+dm) dt.

7.

CONDITIONS FOR φ TO BE A CHARACTERISTIC FUNCTION

7

Conditions for φ to Be a Characteristic Function

Example 7.1

363

Now, for a > 0,

Fa (x) = [1 − cos(ax)]/(πax2 )

for x ∈ R

is a de la Vallée Poussin density function with chf φa (t) = (1 − |t|/a) 1[−a,a] (t)

for t ∈ R.

Let Fa denote the df. Then F ≡

n

pi Fai

with

pi ≥ 0,

n 1

pi = 1,

and 0 < a1 < · · · < an

1

is a df with characteristic function n φ = 1 pi φai . Thus any even function φ ≥ 0 with φ(0) = 1 whose graph on [0, ∞) is a convex polygon is a chf. 2 Proposition 7.1 (P´ olya) Let φ ≥ 0 be an even function with φ(0) = 1 whose graph on [0, ∞) is convex and ↓. Then φ is a chf. Proof. Pass to the limit in the obvious picture, using the continuity theorem to complete the proof. 2 Bochner’s theorem below gives necessary and sufficient conditions for a function to be a chf. We merely state it, as a background fact. Its proof can be found in a number of the standard texts. Definition 7.1 A complex-valued function φ(·) on R is nonnegative definite if for any finite set T and any complex-valued function h(·) we have ¯ (1) s,t∈T φ(s − t)h(s)h(t) ≥ 0. Theorem 7.1 (Bochner) A complex-valued function φ(·) is a chf if and only if it is nonnegative definite and continuous.

Chapter 14

CLTs via Characteristic Functions 0

Introduction

The classical CLT states that if X1 , X2 , . . . are iid (µ, σ2 ), then √ ¯ n − µ) →d N (0, σ2 ) n (X as n → ∞. Chapters 11 and 14 consider the following generalizations: (i) Nonidentically distributed rvs. (ii) Triangular arrays of row-independent rvs Xn1 , . . . , Xnn for n ≥ 1. (iii) The speed of convergence of the dfs to the limiting df. (iv) Convergence of the density functions as well. (v) Random sample sizes. (vi) Certain dependent rvs. (vii) The multidimensional case. (viii) Nonnormal limits (with both the degenerate WLLN and the Poisson). In chapter 15 we will consider situations that lead to stable and infinitely divisible rvs as limits. Chapters 16, 17, and 18 all contain additional extensions leading to asymptotic normality.

365

366

1

CHAPTER 14. CLTS VIA CHARACTERISTIC FUNCTIONS

Basic Limit Theorems

The goal of this section is to use a chf approach to present the classical central limit theorems for sums of iid random variables in R and in Rk . We also compare and contrast the central limit theorem with the Poisson limit theorem.

The Classical CLT Theorem 1.1 (Classical CLT) For each n ≥ 1, let Xn1 , . . . , Xnn be iid F (µ, σ2 ); this denotes that the df F (·) of the Xnk ’s has mean µ and finite variance σ2 . Define ¯ n ≡ Tn /n. Then as n → ∞, the total Tn ≡ Xn1 + · · · + Xnn and the average X √ n 2 ¯ n − µ) = √1 (Tn − nµ) = √1 (1) n (X k=1 (Xnk − µ) →d N (0, σ ) . n n Proof. (a)

(b)

Now, for fixed t we have [with r(t) → 0 as t → 0] √ n φ√n (X¯ n −µ) (t) = k=1 φ(Xnk −µ)/√n (t) = [φXnk −µ (t/ n )]n ! 2 2 "n σ2 t t t √ + √ r √ by inequality 13.4.2 = 1− 2 n n n n σ2 t2 t2 t = 1− . + ×r √ 2n n n

The first product lemma (lemma 13.4.3) with θ = −σ2 t2 /2 trivially applies. Thus φ√n (X¯ n −µ) (t) → e−σ t /2 = φN (0,σ2 ) (t) , √ ¯ 2 using table 13.1.1. Thus n (X er–Lévy continuity n − µ) →d N (0, σ ) by the Cram´ theorem (theorem 13.3.1) and the uniqueness theorem (theorem 13.2.1). Had we chosen to appeal to the second product lemma (lemma 13.4.4) instead, we would have instead claimed that φ√n (X¯ −µ) (t) − (1 − σ2 t2 /2n)n n n n (d) = k=1 φ(Xnk −µ)/√n (t) − k=1 (1 − σ 2 t2 /2n) n φ(X −µ)/√n (t) − (1 − σ2 t2 /2n) (e) ≤ 2 2

(c)

k=1

(f)

≤

n

nk

2

k=1 (t

√ √ /n)r(t/ n ) = t2 r(t/ n ) → 0.

But (1 − σ2 t2 /2n)n → exp(−σ2 t2 /2) = φN (0,σ2 ) (t), so the continuity theorem and the uniqueness theorem complete the proof. 2

Degenerate Limits Exercise 1.1 (WLLN, or classical degenerate convergence theorem) Use ¯ n →p µ as n → ∞. chfs to show that if X1 , X2 , . . . are iid with mean µ, then X Equivalently, (2)

¯ n →d (the degenerate distribution with mass 1 at µ). X

1.

BASIC LIMIT THEOREMS

367

The Classical PLT Theorem 1.2 (Classical Poisson limit theorem; the PLT) For each n ≥ 1, suppose that Xn1 , . . . , Xnn are independent Bernoulli(λ ) rvs for which the values n nk n of the parameters satisfy λn ≡ k=1 λnk → λ while k=1 λ2nk → 0. (This is true if λn1 = · · · = λnn = λn /n for all n, with λn → λ). Then (3)

Tn ≡ Xn1 + · · · + Xnn →d Poisson(λ)

Proof.

From table 13.1.1 we have φXnk (t) = 1 + λnk (eit − 1). Thus n n φTn (t) = k=1 φXnk (t) = k=1 [1 + λnk (eit − 1)]

(a) (b)

→ exp(λ(eit − 1))

(c)

= φPoisson(λ) (t)

as n → ∞.

by the product lemma by table 13.1.1.

Now apply the Cramér–Lévy continuity theorem and the uniqueness theorem. Exercise 1.2 (Poisson local limit theorem) (4)

P (Tn = k) → P ( Poisson(λ) = k)

2

Show that as n → ∞, for k = 0, 1, . . . ,

when λn1 = · · · = λnn in the PLT. Show that this implies dT V (Pn , P ) ≡ sup{|Pn (A) − P (A)| : A ∈ B} → 0, where Tn ∼ = Pn and Poisson(λ) ∼ = P . [This is improved in exercise 14.4.4 below.] √ Exercise 1.3 Show that if Tλ ∼ = Poisson(λ), then (Tλ − λ)/ λ →d N (0, 1) as the parameter λ → ∞. (5)

A Comparison of Normal and Poisson Convergence Exercise 1.4 (6)

Mn ≡

(a) Suppose the hypotheses of the classical CLT hold. Show that √ max |Xnk − µ|/ n →p 0 .

1≤k≤n

(b) Suppose the hypotheses of the classical PLT hold. Show that (7) Mn ≡ max |Xnk | →d Bernoulli(1 − e−λ ) . 1≤k≤n

(c) There is something fundamentally different regarding the negligibility of the corresponding terms in these two cases! The CLT involves summing many tiny pieces, but the PLT arises from very occasionally having a “large” piece. Remark 1.1 Let Xn1 , . . . , Xnn be independent. Let pnk ≡ P (|Xnk | ≥ ). Recall equation (10.3.14) for n Mn ≡ max |Xnk | →p 0 if and only if (8) pnk → 0 for all > 0, 1≤k≤n

(9)

k=1

n n 1 − exp(− k=1 pnk ) ≤ P (Mn ≥ ) ≤ k=1 pnk .

2

368


The Multivariate CLT n ≡ (Xn1 , . . . , Xnk ) , n ≥ 1, Theorem 1.3 (Classical multivariate CLT) Let X be a sequence of iid ( µ, Σ) random vectors. Then (10)

Proof. (a)

n 1 (Xj − µ ) →d Nk (0, Σ) 1/2

n

as n → ∞.

j=1

For any λ ∈ Rk the rvs j − µ Yj ≡ λ (X ) ∼ = (0, λ Σλ)

are iid for j = 1, . . . , n.

Thus the classic CLT gives √ (b) n Y¯n →d N (0, λ Σλ). n ≡ n−1/2 n (X j − µ That is, Z ) satisfies 1 (c)

φλ Z n (t) = φ√n Y¯n (t) → exp(−λ Σλ t2 /2).

∼ ∼ Now, if Z = Nk (0, Σ), then λ Z = N (0, λ Σλ ); and hence (d)

φλ Z (t) = exp(−λ Σλ t2 /2).

Thus (c) and (d) give φλ Zn (t) → φλ Z (t) for all t ∈ R, for each λ ∈ Rk . Thus the n →d Z. Cramér–Wold theorem (theorem 13.3.2) shows that Z 2 √ Exercise 1.5 (Empirical process; Doob) Let Un ≡ n [Gn − I] be the uniform empirical process of sections 7.5 and 12.10, and let U denote the Brownian bridge of section 9.4. Show that Un →f d U as n → ∞; that is, show that for any set of points 0 < t1 < · · · < tk < 1 we have (Un (t1 ), . . . , Un (tk )) →d (U(t1 ), . . . , U(tk ))

as n → ∞.

(Essentially, all results in chapter 12 derive from this example—via a suggestion of Doob(1949).) Exercise 1.6 (Partial sum process of iid rvs) Let S n denote the partial sum process of iid (0, 1) rvs (see (11) below) and let S denote Brownian motion. Show that S n →f d S as n → ∞. [Hint. Set things up cumulating from the left, and then transform. Or note that the random element you must consider can be written in a form equivalent to something simpler. Or use the Cramér–Wold device. One of these methods is much simpler then the others.] Exercise 1.7 (Partial sum process) Suppose that Xn1 , . . . , Xnn are independent and satisfy Lindeberg’s condition. Define S n on [0, 1] by letting (11)

S n(t) =

with s2nk ≡ motion.

k

k

i=1 Xni /snn

2 i=1 σni

for

s2n,k+1 s2nk ≤ t < , s2nn s2nn

0 ≤ k ≤ n,

and s2n0 ≡ 0. Show that S n →f d S, where S denotes Brownian

1.

BASIC LIMIT THEOREMS

369

Example 1.1 (Chisquare goodness of fit statistic) Suppose Ω = Now let X1 , . . . , Xn be iid on (Ω, A) with all pi ≡ P (X ∈ Ai ) > 0. Let (12)

Nni ≡

n

1Ai (Xj ) ≡ ( the number of Xj ’s that fall in Ai )

k i=1

Ai .

for 1 ≤ i ≤ k.

j=1

√ (a) Now, (Z1j , . . . , Zkj ) , with Zij ≡ (1Ai (Xj ) − pi )/ pi , has mean vector 0 and covariance matrix Σ = |[σii ]| with σii = 1 − pi and σii = − pi pi for i = i . √ n ≡ n Zj / n →d W ∼ (b) Thus W = Nk (0, Σ) as n → ∞, by theorem 1.3. 1 (c) The usual chisquare goodness of fit statistic is (13)

Qn ( p) ≡

k (Nni − npi )2 i=1

npi

W =W n n →d W W

=

k (Observedi − Expectedi )2 Expectedi i=1

by the Mann–Wald theorem

)∼ ) (ΓW = (ΓW = Chisquare(k − 1) ; √ √ √ here Γ is k × k and orthogonal with first row p , so that Γ Σ Γ = Γ[I − p p ]Γ = I − (1, 0, . . . , 0) (1, 0, . . . , 0). This has diagonal elements (0, 1, 1, . . . , 1) with all ∼ off-diagonal elements 0, and then ΓW = N (0, ΓΣΓ ) (by (8.3.5) and (9.3.6)). We also use (9.1.29) for (16). [If a value of Expected is unknown, it should be replaced ˆ by an appropriate estimator Expected.] (See exercise 14.2.13 below.) (This statistic is just a quadratic form.) 2 (14)

Exercise 1.8 (Independence in an I × J table) Suppose both Ω = J and Ω = j=1 Bj represent partitions of Ω. (a) Let pij ≡ P (Ai Bj ) = pi· p·j , where pi· ≡ P (Ai ) and p·j ≡ P (Bj ). Let

I

i=1 Ai

Nij ≡ (the number of iid observations X1 , . . . , Xn that fall in Ai Bj ). J I Let pî· ≡ j=1 Nij /n and pˆ·j ≡ i=1 Nij /n. Show that (15) Qan ≡ Ii=1 Jj=1 (Nij − nˆ pi· pˆ·j )2 /(nˆ pi·pˆ·j ) →d Chisquare((I − 1)(J − 1)). (b) Let pi|j ≡ P (Ai |Bj ). Let n ≡ n·1 + · · · + n·J . For each 1 ≤ j ≤ J, let (j) (j) Nij ≡ (the number of iid P (·|Bj ) observations X1 , . . . , Xnj that fall in Ai Bj ). J Let pî|j ≡ j=1 Nij /n·j . Show that when σ[A1 , . . . , AI ] and σ[B1 , . . . , BJ ] are independent, the chisquare statistic satisfies (16) Qbn ≡ Ii=1 Jj=1 (Nij − nj pî|j )2 /(nj pî|j ) →d Chisquare((I − 1)(J − 1)) as n1 ∧ · · · ∧ nJ → ∞. (c) Suppose that both sets of marginal totals n1· , . . . , nI· and n·1 , . . . , n·J are fixed, and that both sum to n. Suppose that n balls are assigned to the IJ cells at random without replacement, subject to the side conditions on the marginal totals stated above. Let Nij denote the number assigned to the (i, j)-th cell. Then I J (17) Qcn ≡ i=1 j=1 (Nij −ni· n·j /n)2 /(ni· nj· /n) →d Chisquare((I − 1)(J − 1))

370


as (n1· ∧ · · · ∧ nI· ) ∧ (n·1 ∧ · · · ∧ n·J ) → ∞. [Suppose I = 5 different social groups are at work in broadcasting, where the sum of the I group sizes ni· of our data is n = 250. The number whose salaries fall in each decile (thus J = 10) of the observed salaries is necessarily n·j = n/J = 25. The statistic in (17) can be used to test for independence of group and salary level.] 2 Exercise 1.9 Let Vn2 now denote the sample variance. Show, in the context of Doeblin’s CLT, that Tνn /Vνn →d N (0, 1) as n → ∞. Exercise 1.10 Prove a version of Doeblin’s theorem for Xnk ’s independent but not iid; assume the Lindeberg condition and νn /n →p c ∈ (0, ∞). [Revert to the Liapunov condition, if necessary.]

Limiting Distributions of Extremes Exercise 1.11 (a) Let ξn1 , . . . , ξnn be iid Uniform(0, 1) rvs. Then the sample minimum ξn:n satisfies nξn:n → Exp(1). (b) Now, ξn:n is the sample maximum. Determine the joint asymptotic distribution of nξn:1 and n(1 − ξn:n ). Exercise 1.12 (Special cases of Gnedenko’s theorem) of an iid sample X1 , . . . , Xn from F (·). Then:

Let Xn:n be the maximum

(a)

P (Xn:n − log n ≤ y) → exp(−e−y ) when 1 − F (x) = e−x for x ≥ 0.

(b)

P (n1/b Xn:n ≤ y) → exp(−|y|b ) for all y < 0, when 1 − F (x) = |x|b for − 1 ≤ x ≤ 0, with b > 0.

(c)

P (Xn:n /n1/a ≤ y) → exp(−y−a ) for all y > 0, a when 1 − F (x) = 1/x for x ≥ 1, with a > 0.

for all y ∈ R,

[Distributions that are “suitably similar” to these prototypes yield the same limiting results, and these limits are the only possible limits.]

2.

VARIATIONS ON THE CLASSICAL CLT

371

2

Variations on the Classical CLT

Notation 2.1 Let Xnk , 1 ≤ k ≤ n for each n ≥ 1, be row-independent rvs having 2 means µnk and variances σnk , and let γnk ≡ E|Xnk − µnk |3 < ∞ denote the third absolute central moments . Let n 2 (1) and γn ≡ nk=1 γnk , and sdn ≡ k=1 σnk (2)

Zn ≡

n

1

sdn

[Xnk − µnk ] .

k=1

Let φnk (·) denote the chf of (Xnk − µnk )/sdn .

2

Theorem 2.1 (Rate of convergence in the CLT) Consider the rvs above. The df FZn of the standardized Zn is uniformly close to the N (0, 1) df Φ, in that (3)

FZn − Φ ≤ 13 γn /sd3n .

Corollary 1 (Liapunov CLT) (4)

Zn →d N (0, 1)

whenever

γn /sd3n → 0.

Corollary 2 (Berry–Esseen for iid rvs) F (µ, σ2 ) with γ ≡ E|X − µ|3 < ∞. Then (5)

FZn − Φ ≤

Let Xn1 , . . . , Xnn be iid rvs with df

8γ √ . σ3 n

Proof. [There is considerable advantage to treating theorem 2.1 by giving a careful proof of the rate of convergence to normality based on Esseen’s lemma and lemma 13.4.1.] Without loss of generality, we assume that all µnk = 0. We now note that (recall that a = b ⊕ c means |a − b| ≤ c) 2 2 n φZn (t) − e−t /2 = k=1 φnk (t) − e−t /2 ≤ e−t

2

(6)

≡ e−t

2

n Log k=1 e

/2

/2

φnk (t) +t2 /2

|ez − 1| ≤ e−t

2

/2

|z| e|z|

− 1 by (13.4.3)

for all z, where n n 2 i2 t2 σnk 2 (a) |z| = Log φnk (t) + t /2 = Log (1 + [φnk (t) − 1]) − 2 2 sd n k=1 k=1 n 2 i2 t2 σnk 2 (7) ≤ [φnk (t) − 1 + ] ⊕ |φnk (t) − 1| 2 sd2n k=1

provided that |φnk (t) − 1| ≤ 12 , using (13.4.2)

372

CHAPTER 14. CLTS VIA CHARACTERISTIC FUNCTIONS |t|3 γn ≤ + 3 6 sdn k=1 n

(b)

.

|t|3/2 E|Xnk |3/2 sd3/2 n

/2 2 where K1,1/2 =

K1,1/2

8 9

using (13.4.4) [with m = 2 and δ = 1, then with m = 1 and δ = 12 ] (c)

≤

1 2a

|t|3

19

a ≡ sd3n /γn

with

9

using Liapunov’s inequality for (E|Xnk |3/2 )2 ≤ γnk (d)

≤

1 2 4 t

on |t| ≤

9 38

a.

But validity of (7) required that all |φnk (t) − 1| ≤ 12 . However, (13.4.4) with m = 1 and δ = 1 reexpresses this requirement as (e)

2 |φnk (t) − 1| ≤ (t2 σnk /2 sd2n ), and must be ≤

1 2

.

For any fixed t, the bound on |z| in (c) goes to 0 whenever 1/a = γn /sd3n → 0. More2 over, (e) always holds when γn /sd3n → 0, since (max σnk /sd2n )3/2 ≤ (max γnk /sd3n ) ≤ (γn /sd3n ) → 0. Thus φn (t) → exp(−t2 /2) and Zn →d N (0, 1) by (6), whenever we have γn /sd3n → 0. That is, corollary 1 holds. [This is already a good CLT!] Consider the requirement of (e) again. If |t| ≤ a1/3 , then (f) (g)

2 |φnk (t) − 1| ≤ (t2 /2) · σnk /sd2n ≤ (1/2) a2/3 [ γn /sd3n ]2/3

≤

1 2

on 0 ≤ |t| ≤ a1/3 .

9 a we have not yet justified the requirement made But on the interval a1/3 < |t| ≤ 38 in (e), which gave (d); so we will now show that (d) is still true, even for the larger range a1/3 ≤ |t| ≤ 38 a. Now, |φn (t)|2 is the chf of the symmetrized rv Zns ≡ Zn − Zn (and this rv has mean 0, variance 2, and third absolute moment bounded above by 8γn /sd3n (via the Cr -inequality)). Thus

|φZn (t)| ≤ [ |φn (t)|2 ]1/2 ≤ [1 + 0 − (h)

≤ exp(−t2 [ 12 −

(i)

≤ exp(−t2 /4)

2|t| 3

γn / sd3n ])

2t2 2!

+

|t|3 3!

8γn / sd3n ]1/2

using 1 − x ≤ e−x

for |t| ≤ (3/8) (sd3n /γn ),

as was desired. This leads to (j)

|φZn (t) − e−t

2

/2

| ≤ 2 e−t

2

/4

≤ 2( |t|3 /a) e−t

2

/4

for a1/3 ≤ |t| ≤ 38 a.

Key chf inequality Combining (6), (d), (c), and (g) with (j) gives 2 2 (8) for 0 ≤ |t| ≤ 38 sd3n /γn . φZn (t) − e−t /2 ≤ (2 |t|3 γn /sd3n ) e−t /4

2.


373

We apply (8) to Esseen’s lemma and get (since we know the variance of a normal distribution) FZn − Φ ≤

(k)

(3/8)a

1 2|t|3 −t2 /4 3.04769 dt + e π|t| a (3/8) a

−(3/8)a

3.04769 1 2 ∞ 2 −t2 /4 t e dt + ≤ a π −∞ 3/8 √ . = [8/ π + (8/3) 3.04769]/a = 12.641/a ≤ 13/a.

(l)

In the iid case use K1,1 = |t|3 β |z| ≤ √ + n 6 n (m)

≤

t2 σ2 2nσ 2

5 β √ |t|3 12 n

1 2

and β ≡ E|X|3 /σ 3 ≥ 1 in (b), and obtain

2

|t|3 β t4 β 2 ≤ √ + 6 n 4n

for all |t| ≤

√

n/β,

with (d) necessarily valid. Thus (8) can be replaced in the iid case by (9)

|φZn (t) − e−t

2

/2

|≤

2 5 γ √ |t|3 e−t /12 3 12 σ n

on 0 ≤ |t| ≤

√ this yields 8γ/ n σ3 when the steps leading to (l) are repeated.

Theorem 2.2 (Lindeberg–Feller) (10)

Zn →d N (0, 1)

(11)

LFn

≡

nσ3 /γ;

2

The following are equivalent: [max1≤k≤n P (|Xnk − µnk |/sdn > )] → 0 .

and

n k=1

√

[|(x−µnk )|≥ sdn ]

x − µnk sdn

2 dFnk (x) → 0

for all > 0 .

[Condition (11) implies that Mn ≡ [max1≤k≤n |Xnk −µnk |/sdn ] →p 0, via (14.1.9).] Proof. (Lindeberg) We prove the sufficiency here, with the necessity considered in the following separate proof. We note that the moment expansion inequality (inequality 13.4.1) gives bounds on βnk (t), where (12)

φnk (t) ≡ 1 + θnk (t) ≡ 1 −

2 σnk t2 + βnk (t) sd2n 2

defines θnk (t) and βnk (t). Moreover (in preparation for the product lemma) n > n n σ2 t2 (13) φZn (t) = k=1 φnk (t) = k=1 [1 + θnk (t)] = 1 − nk2 + βnk (t) sdn 2 k=1

374


where (14)

θn (t) ≡

n

θnk (t) = −

k=1

n n n 2 t2 σnk 2 + β (t) = −t /2 + βnk (t) . nk 2 sd2n k=1 k=1 k=1

The moment expansion inequality of (13.4.10) (compare this with (7)) gives n n 2 σnk t2 (a) βnk (t) = φnk (t) − 1 − 0 + 2 sdn 2 k=1

(15)

≤

(b)

≤

k=1

n it(x − µnk ) [it(x − µnk )]2 it (x−µnk )/sdn dFnk (x) − 1+ + e 2 sdn 2 sd n k=1 n k=1

(c)

+

[|(x−µnk )| 0. Note that [max1≤k≤n |θnk (t)| ] → 0 as required by the product lemma, since we can actually claim that 2 (d) σnk /sd2n ≤ [ [|x−µnk |≤ sdn ] (x − µnk )2 dFnk (x)/sd2n + (e)

[|x−µnk |> sdn ]

≤ 2 + o(1) ≤ ,

(x − µnk )2 dFnk (x)/sd2n

for n ≥ (some n ) .

2

Proof. (Feller) We proved sufficiency in the previous proof; we now turn to necessity. Suppose that condition (10) holds. Applying (13.4.2) [since the easy exercise 14.2.9 below applied to our uan rvs shows that the terms znk = φnk (t) − 1 converge uniformly to 0 on any finite interval] gives n n n (17) | k=1 Log φnk (t) − k=1 [φnk (t) − 1] | ≤ k=1 |φnk (t) − 1|2 n 2 (18) ≤ [max1≤k≤n |φnk (t) − 1|] × (t2 /2) × [ k=1 σnk /sd2n ] by (13.4.5) (a)

≤ o(1) × (t2 /2) × 1 → 0,

using (10).

We thus have n n (b) Log 1 φnk (t) = 1 [φnk (t) − 1] + o(1).

2.


375

But we also know that (c)

Log

n 1

φnk (t) → −t2 /2,

since we have assumed asymptotic normality. [Recall that a = b ⊕ c means that |a − b| ≤ c.] Combining (b) and (c) shows that for every tiny > 0 and every huge M > 0 we have (d)

−t2 /2 = Real(−t2 /2) = Real{

n 1

[φnk (t) − 1]} ⊕ for |t| ≤ M

for all large n; that is, for n ≥ (some nM ) we have (e)

t2 /2 =

n 1

[1 − cos(t (x − µnk )/sdn )] dFnk (x) ⊕ on |t| ≤ M.

Define yk ≡ (x − µnk ). We further define In ≡ [|x − µnk | < sdn ]. Note that (f)

0 ≤ 1 − cos(ty/sdn ) ≤ (t2 y 2 /2 sd2n ).

Thus for all |t| ≤ M we have for all n ≥ nM that (g)

(t2 /2)

n

= (t2 /2) −

k=1

(19)

=

(i)

≤2

(j)

≤ (2/ 2 )

(k)

≤ 2/ 2 + .

k=1

c [1 In

n

c In

k=1

In

n k=1

In

(yk2 /sd2n ) dFnk (x)]

(t2 yk2 /2 sd2n ) dFnk (x)

In

n

≤ (t2 /2) − k=1

n

(h)

n

(yk2 /sd2n ) dFnk (x) = (t2 /2) [1−

c In

k=1

[1 − cos(tyk /sdn )] dFnk (x)

− cos(tyk /sdn )] dFnk (x) ⊕

by (f)

by (e)

[the key step]

dFnk (x) +

n k=1

2 c (yk In

sd2n ) dFnk (x) +

Specifying t2 = M 2 = 4/( 2 × θ) in (g) (for some 0 < θ < 1) shows that for all n ≥ nθ we have (l)

sd−2 n

n 1

c In

yk2 dFnk (x) ≤ 2 θ,

where θ > 0 is arbitrary. Thus, the Lindeberg condition (11) holds.

2

376


Remark 2.1 (Lindeberg’s condition) still be true that (a)

Sn /σn →d N (0, a2 )

with

(i) If Lindeberg’s condition fails, it may a2 < 1 and [max1≤k≤n σk2 /σn2 ] → 0.

Let the rv’s Un be independent (0, 1) with Un equal to −cn, ∞ 0, cn with probabilities ∞ 1/(2n2 ), 1 − 1/n2 , 1/(2n2 ). Since 1 P (|Un | ≥ ) = 1 n−2 < ∞, the Borel– Cantelli lemma shows√that for a.e. ω the rv sequence Un satisfies Un = 0 only ¯n →p 0 follows. Now let Y1 , Y2 , . . . be iid (0, 1) rvs, finitely often. Thus n U √ so that n Y¯n →d N (0, 1) by the CLT. For n ≥ 1 set Xn ≡ Yn + Un , and let Sn ≡ X1 + · · · + Xn . Note that σn2 ≡ Var[Sn ] = (1 + c2 )n. So, by Slutsky’s theorem, √ √ ¯n )/ 1 + c2 Sn /σn = ( n Y¯n )/ 1 + c2 + ( n U →d N (0, 1)/ 1 + c2 + 0 ∼ = N (0, 1/(1 + c2 )) (b)

= N (0, a2 )

with a2 = 1/(1 + c2 ) < 1.

√ 2 ˇnk /ˇ σn2 ] → 0. But, even so, Lindeberg’s (One could also let c = n.) Note that [max σ condition fails, since a n √ LFn = x2 dFXk (x) n k=1 [|x|≥ n/a ] (c)

∼

a n

√

k:ck≥

c2 (kc)2 + o(1) → > 0; k2 (1 + c2 ) n/a

the nonzero contribution shown in the last step is due to Uk , whereas we do already know that the contribution due to the Yk ’s is o(1). This example shows that it is possible to have Xn →d X without having Var[Xn ] → Var[X]. Note that Var[N (0, 1/(1 + c2 )] = 1/(1 + c2 ) < 1 = lim 1 = limVar[Sn /σn ] (via the Fatou lemma and Skorokhod’s theorem). (ii) Note that if Xn1 ∼ = N (0, pn), for some 0 < p < 1, Xnk ≡ 0 for 2 ≤ k ≤ [pn], and Xnk ∼ N (0, 1) for pn < k ≤ n for independent rvs Xnk , then Sn /σn →d N (0, 1), = 2 while Lindeberg’s condition fails and [max1≤k≤n σnk /σn2 ] → p > 0. 2 Remark 2.2 It is known that the constant 8 in (5) can be replaced by 0.7975. 3 It is also known in the iid case √ with E|X| < ∞ that the “limiting distribution measure” d(F, Φ) ≡ limn→∞ nFZn − Φ exists, and moreover that it takes on the √ √ value supF (σ3 /γ) d(F, Φ) = ( 10 + 3)/(6 2π ) = 0.409. This sup is achieved by √ √ c [Bernoulli(a) − a], where c = ( 10− 3)/2 and a = (4 − 10 )/2. Thus the constant 0.7975 cannot be greatly improved. Many other improvements and refinements of the Berry–Esseen theorem are possible. The books by Bhattacharya and Rao (1976, pp. 110, 240) and Petrov(1977) both give many. We list three as “exercises” in exercise 2.11 below. 2 Exercise 2.1 Beginning with (15), obtain the Berry–Esseen bound (but with a different constant) by appeal to the second product lemma.

2.


377

Exercise 2.2 (Liapunov’s (2 + δ)-CLT) Define γnk = E|Xnk |2+δ for every value 0 < δ ≤ 1. Suppose we have Liapunov’s (2 + δ)-condition that n 2+δ /σn2+δ → 0 for some 0 < δ ≤ 1. k=1 E|Xnk | Show that Sn /σn →d N (0, 1). Exercise 2.3 Construct an example with iid X1 , X2 , . . . for which the Lindeberg condition holds, but for which Liapunov’s (2 + δ)-condition fails for each 0 < δ ≤ 1. Exercise 2.4 Establish the (2 + δ)-analogue of theorem 2.1. [Hint. Use both 2 + δ and 1 + δ moments in line (b) of the theorem 2.1 proof, via lemma 13.4.2.] Exercise 2.5 (Liapunov-type WLLN) Let Xn1 , . . . , Xnn , n ≥ 1, be a triangular array of row-independent rvs with 0 means. Then n 1+δ /n1+δ → 0 for some 0 < δ ≤ 1 k=1 E|Xnk | ¯ n →p 0 as n → ∞. implies that X Exercise 2.6

Formulate a WLLN in the spirit of the Lindeberg–Feller theorem.

Exercise 2.7 (Pitman) Exercise 2.8

¯ n →p a holds if and only if φ (0) = ai. For iid rvs, X

Complete the proof of theorem 10.9.1.

Exercise 2.9 (Some uan equivalences)

The following are equivalent:

(a)

|Xnk |’s are uan [meaning that max1≤k≤n P (|Xnk | ≥ ) → 0 for all > 0].

(b)

[max1≤k≤n |φnk (t) − 1|] → 0 uniformly on every finite interval of t’s. max1≤k≤n α(x) dFnk (x) → 0 for α(x) ≡ x2 ∧ 1.

(c)

¯ n for a single iid Exercise 2.10 Let Zn denote an appropriately normalized X sequence of rvs. Prove the following statement: Z¯n →d N (0, 1) if and only if the symmetrized rvs Z¯ns →d N (0, 2) (provided it’s true). Exercise 2.11 (a) (Petrov) Suppose Xn1 , . . . Xnk are row independent rvs for n 2 2 which Xnk ∼ ), and set σn2 ≡ k=1 σnk and Fn (x) ≡ P (Sn /σn ≤ x). Then = (0, σnk for some universal constant C we have 0 1 3 2 (20) Fn −Φ ≤ C σn−3 nk=1 EXnk 1[|Xnk | 0, and x/g(x) is for x > 0,

then for some absolute constant C we have the very nice result n 2 (22) Fn − Φ ≤ C 1 E[Xnk g(Xnk )]/σn2 g(σn ).

378


(c) (Nagaev) Bounds on |Fn (x) − Φ(x)| that decrease as |x| → ∞ are given (in the iid case) in the expression √ (23) for all real x. |Fn (x) − Φ(x)| ≤ C(E|X|3 /σ 3 )/( n(1 + |x|3 ) (d) (Bernstein) Let r > 2. Consider row-independent rvs Xn1 , . . . , Xnn for which n 2 we have Xnk ∼ ). Let Zn ≡ k=1 (Xnk − µnk )/σn . = (0, σnk n r/2 (α) Suppose k=1 |Xnk |r /σn → 0. Then we have Zn →d N (0, 1) as well as the conclusion that E|Zn |r → E|N (0, 1)|r . 2 /σn2 ] → 0. (β) The converse holds if we also have [max1≤k≤n σnk (e) See Petrov (1975, pp. 118, 113, 125, 103) for (a), (b), (c), and (d).

Statistical Applications Exercise 2.12 (Simple linear regression) Consider the simple linear regression model of (9.3.25); thus we are assuming that (24)

Ynk = α + β xnk + k

for iid rvs k ∼ = (0, σ2 )

and for known constants cnk for 1 ≤ k ≤ n. The least squares estimators (LSEs) α ˆ n and βˆn of α and β are defined to be those values of a and b that minimize the sum of squares n1 [ Ynk − (a + b xnk ) ]2 . (a) Show that the LSEs are given by n n ¯n ) Ynk 1 (xnk − x ˆ ¯ (25) α ˆn = Yn and βn = ≡ k=1 dnk Ynk . n 2 (x − x ¯ ) nk n 1 n (b) Let SSxx ≡ 1 (xnk − x ¯n )2 and dn = (dn1 , . . . , dnn ) , (26) ¯n |2 D(xn ) ≡ max |xnk − x SSxx = max dnk = D(dn ) . 1≤k≤n

1≤k≤n

Use the Cramér–Wold device and the weighted sums of example 11.6.4 to show that √ n [ˆ αn − α] (27) √ →d N (0, σ 2 I), provided that D(dn ) = D(xn ) → 0 SSxx [βˆn − β] (recall (11.6.20)). [Note also that the LSE βˆn of β is given by βˆn = SSxY /SSxx .] Definition 2.1 (Noncentral distributions) (a) Let X1 , . . . , Xm be independent, m and suppose that Xi ∼ = N (θi , σ2 ). Let θ2 ≡ 1 θi2 , and define δ via δ2 ≡ θ2 / σ 2 . Show that the quadratic form m m U ≡ 1 Xi2 / σ 2 ∼ (28) = (Z1 + θ)2 + i=2 Zi2 , where Z1 , . . . , Zm are iid N(0, 1) rvs. Denote this distribution by (29)

U∼ = χ2m (δ 2 /2),

and say that U is distributed as noncentral chisquare with m degrees of freedom and noncentrality parameter δ.

2.


379

(b) Let Y ∼ = N (θ, 1), U ∼ = χ2m (δ 2 /2) and V ∼ = χ2n be independent rvs. We define the noncentral Student-tn (θ) distribution and the noncentral Snedecor-Fm,n (δ 2 /2) distribution by writing (30)

Tn (θ) ≡

Y ∼ n U U/m ∼ = = Student-tn (θ) and = Snedecor-Fm,n (δ 2 /2) . V /n m V V /n

(Note that Tn2 (θ) ∼ = F1,n (θ2 /2).) Proposition 2.1 (Form of the noncentral distributions) Consider the rvs U , V , and Y of the previous definition. Let y > 0. (a) The rv U of (28) satisfies ∞ (31) P (χ2m (δ 2 /2) > y) = k=0 P (Poisson(δ2 /2) = j) × P (χ2m+2j > y) . Here, Poisson(λ) denotes a Poisson rv with mean λ, and χ2r denotes an ordinary chisquare rv with r degrees of freedom. (b) It is thus trivial that 2 (32) P (Fm,n (δ 2 /2) > y) = ∞ k=0 P (Poisson(δ /2) = j) × P (Fm+2j,n > y) . √ (c) For Cn ≡ 2(n+1)/2 Γ(n/2) π n we have

(33)

P (Tn (δ) > y) ∞∞ = C1n y 0 u(n−1)/2 e−u/2 exp(− 12 (v( un )1/2 − δ)2 ) du dv .

Exercise 2.13


Exercise 2.14 (Chisquare goodness of fit, again) (a) (Local alternatives) We suppose that the statistic Qn ≡ Qn ( p0 ) of (14.1.13) is computed, but that in √ reality the true parameter vector is now pn ≡ p0 + a/ n (with k1 ai = 0, so that the coordinates pni add to 1). Let pˆni ≡ Nni /n estimate pni for 1 ≤ i ≤ k. Show that the vector √ √ nk×1 ≡ |[ n (ˆ + d , (34) W pni − p0i )/ p0i ]| →d W √ √ ∼ where W = N (0, I − p p ) and di ≡ ai / p0i for 1 ≤ i ≤ k. Thus (35)

. W n →d Q ≡ (W + d ) (W + d ) ∼ Qn = W = χ2k−1 (d d/2) n

(b) (Fixed alternatives) Suppose that Qn ≡ Qn ( p0 ) is computed, but a fixed p is true. Show that k 2 1 (36) i=1 (pi − p0i ) / p0i . n Qn →a.s. Exercise 2.15 (37) (38)

∼ Suppose X = N (θ, Σ), with rank(Σ) = r. Show that

Σ− X = Y Y ∼ X , = χ2r (θ Σ− θ/2)

where Ir 0 −1/2 Y ≡ Σ−1/2 X = (ΓD−1/2 Γ )X ∼ N Σ θ, . = 0 0

380

3


Local Limit Theorems

Recall from Scheffé’s theorem that if fn and f are densities with respect to some dominating measure µ, then (1)

fn (x) → f (x)

a.e. µ

implies that (2)

dT V (Pn,P ) ≡ sup |P (Xn ∈ B) − P (X ∈ B)| → 0

as n → ∞.

B∈B

Thus convergence of densities implies convergence in total variation distance, which is stronger than convergence in distribution. We will now establish (1) in a CLT context, for summands that are either suitably continuous or else are distributed on a grid. Theorem 3.1 (Local limit theorem, continuous case) Let X, X1 , X2 , . . . be ∞ √ iid (0, σ2 ) with −∞ |φX (t)| dt < ∞. Then Sn / n has a density fn (·) for which (3)

sup

−∞<xδ√n ] e−t /2 dt =

√

n θn−1

(6)

= o(n−r )

(g)

<

∞

−∞ |φX (t)| dt

√ 2 + 2(δ n )−1 e−nδ /2

for any r > 0 whenever lim|t|→0 |φX (t)| < 1.

for n ≥ (some n2 ).

Combining (e), (f), and (g) into (d) establishes the claim made in the theorem.

2

Proof. Consider theorem 3.2. By the inversion formula (13.6.1) given for distributions on grids, √ √ πσ√n/d √ σ n σ n d √ (h) pn (x) = φnX (t/ n )e−itx dt. √ d d σ n 2π −πσ n/d By the inversion formula (13.2.9) given for densities, ∞ 2 (i) fZ (x) = (1/2π) −∞ e−t /2 e−itx dt.

382


Thus (j)

√ |(σ n/d) pn (x) − fZ (x)| ≤ (1/2π)

πσ√n/d

+(1/2π) (k)

=

√ −πσ n/d

√ 2 |φnX (t/ n ) − e−t /2 | dt

2 √ e−t /2 [|t|>πσ n/d ]

+ [|t|≤a]

√ [ad√n ] |φX (t/ n )|n dt ≤ n [|s|>d ] |φX (s)|n ds (o)

≤ θn−1

√ ∞ n −∞ |φX (s)| ds = o(n−r ),

for any r > 0,

since the θn term goes to 0 geometrically. Likewise, I3n = o(n−r ), for any r > 0, since |φG |k satisfies (4)(b), for some k. Combine (n) and (o) into (m) to get (5)(b). Consider (5)(a). We will apply Esseen’s lemma. Thus

(p) (q)

FZn (x) − FGr (x) ≡ I1n + I2n + I3n 1 ≤ { [|t|≤d√n ] + [d√n 0 and set u ≡ p/c and v ≡ q/d, and then define r = c2 n and s = d2 n and set √ √ ∼ (0, 1; √1 (− p3/2 + q3/2 ), 1 ( p22 + q22 )) Gr,s ≡ − p Gc2 n + q Gd2 n = c d n c d n (9) √ √ = (0, 1; √1n (− p u + q v), n1 (u2 + v 2 )) . [This approach can (multiply) match all (µ3 , µ4 ) pairs for which µ23 ≤ (µ4 − 3).] Theorem 4.2 (Poisson approximation) Consider a rv X on the integers and let Zn be as in (3). Let r ≡ n/µ23 , so that √ √ Gr ≡ [Poisson(r) − r]/ r ∼ = (0, 1; µ3 / n, µ23 /n). (i) Then for some constants CF and CF,n → 0 (that may depend on the df of the standardized rv (X − µ)/σ): (10)

pZn − pGr ≤ CF /n3/2

when µ4 < ∞ .

(11)

pZn − pGr ≤ CF,n /n

when only µ3 ∈ (0, ∞) .

√ [Most probabilities that one computes involve summing over the appropriate M n number of terms that are each of the type pZn (·).] √ (ii) Suppose r and s can be specified so that Gr,s ∼ = (0, 1; µ3 / n, (µ4 − 3)/n). Then n2 can replace n3/2 in (10), provided that µ5 < ∞. And n3/2 can replace n in (11), provided that µ4 < ∞. Proof. The appropriate inversion formula now (for a distribution on the grid am + b) is given by a (12) pm ≡ P (X = m) = 2π [ |t|≤π/a ] exp (−it(am + b)) φX (t) dt .

4.

GAMMA APPROXIMATION

387

By the previous proof (including the previous step (a) normal component, but now appearing in step (u)) yields √ (u) |φZn (t) − φGr (t)| ≤ (c2 t4 /n) exp (−(5/36) t2) for |t| ≤ d n , for c and d that may depend on the df of (X −µ)/σ. Applying the inversion formula in (12) now gives √ √ √ (v) n |P (Zn = m/ n ) − P (Gr = m/ n ) | √ √ √ √ = 2 π n√n | [ |t|≤π√n ] e−itm/ n [φZn (t/ n ) − φGr (t/ n )] dt| √ √ (w) ≤ [ |t|≤d√n ] |φZn (t/ n ) − φGr (t/ n )| dt √ √ + [d√n 0 and all x > 0 we have ∞ ∞ 2 |dn (x)| ≤ −∞ |φn (t)| dt = −∞ e−t /2 |ez+ − (1 + z)| dt (e)

=

√ [|t|≤θ n/E|X|3 ]

e−t

2

/2

|ez+ − (1 + z)| dt + o(n−r )

for any r > 0, as in (14.3.6). Now, (f)

|φn (t)| = e−t

2

√ n )−(−t2 /2)

|enψ(t/

≡ e−t

/2

|ez+ − (1 + z)| = e−t

≤ e−t

/2

[| | e|| e|z| + z 2 e|z| /2]

2

(g)

/2

2

2

√ − [1 + (γ1 /3!)(it)3 / n ]|

/2

|ez (e − 1) + (ez − (1 + z))|

√ using (13.4.3). Note that for all |t| ≤ θ n/E|X|3 we have √ (h) |z| ≤ (|t|3 /6) E|X|3 / n ≤ θt2 /6 ≤ t2 /8 if θ ≤ 34 , √ √ | | = |nψ(t/ n ) − [−t2 /2 + (γ1 /6) (it)3 / n ]| (i)

√ ≤ c3 |t|3 E|X|3 δ(θ)/ n ≤ c3 t2 θ δ(θ) ≤ t2 /8

if θ is small enough,

where δ(·) denotes the function δ3 (·) function of (13.4.21) associated with the rv X/σ. Using (h) and (i), the bound in (e) becomes (for some θ small enough) 2 |dn (x)| ≤ [|t|≤θ √n/E|X|3 ] e−t /2 {| | e|| e|z| + z 2 e|z| /2} dt + o(n−r ) (j)

√ ∞ 2 ≤ c3 δ(θ) [E|X|3 / n ] −∞ |t|3 e−t /4 dt +[(E|X|3 )2 /(72n)]

(k)

= o(n−1/2 )

∞ −∞

|t|6 e−3t

uniformly in x,

2

/8

dt + o(n−r )

394


since a tiny δ(θ) results from a sufficiently tiny θ. Thus (11) holds. For (12), we replace the bound in line (i) above by √ (l) | | ≤ c¯4 t4 EX 4 /n ≤ t2 /8, which is valid for |t| ≤ n/ 8 c¯4 E|X|4 , as (13.4.22) guarantees. We then use (l) instead of (i) during (j) (now integrated √ over the interval [|t| ≤ n/ 8/¯ c4 EX 4 ]). We now turn to (13), and then (14). We first redefine 1 γ2 γ12 1 γ1 H (m) H3 + H4 + . dn ≡ fn − f0 1 + √ 6 n 4! 2 · (3!)2 n 3! Taking the inverse of its Fourier transform φn (·) gives (as in (e)) that for any fixed value of θ > 0 and all x, ∞ |dn (x)| ≤ −∞ |φn (t)| dt √ 2 2 (n) = [|t|≤θ √n/EX 4 ] e−t /2 |et /2+nψ(t/ n ) − {1 + z + z 2 /2}| dt + o(n−r ) 2 = [|t|≤θ √n/EX 4 ] e−t /2 |ez+ − (1 + z + z 2 /2)| dt + o(n−r ) (o) for each fixed r > 0, with 1 γ2 1 γ1 (p) z≡√ (it)3 + (it)4 . n 3! n 4! The final details are nearly the same as before. Exercise 5.3 Proof. (q)

2

Finish the details of the previous proof of theorem 5.1(b).

Consider theorem 5.2(a). We note that 1 γ 1 γ1 1 Dn ≡ Fn − F0 + f0 √ H2 H3 , has Dn = dn ≡ fn − f0 1 + √ n 3! n 3!

where dn is as in line (a) of the previous proof (just use x H2 (x) − 2x = H3 (x) to verify this). Esseen’s lemma then gives √ 1 |φn (t)| 24 f0 [1 + (γ1 /3!)H3 / n ] √ (r) Dn ≤ dt + , π [|t|≤a√n/E|X|3 ] |t| π a n/E|X|3 where φn is the same φn appearing in (21). Since the norm in the√second term on the right of (r) is bounded, the second term in (r) is less that / n whenever a ≡ a( , FX/σ ) chosen large enough. Fix this a in the limits of integration √ of (r), and then break this integral into two pieces: the integral over [|t| ≤ θ n/E|X|3 ] √ 3 with θ as in√(i), and the integral over [|t| > θ n/E|X| ]. The integral over the set [|t| > θ n/E|X|3 ] is o(n−r ), for any r > 0 (` a la (14.4.6), as before at line √ (e)). Finally, the value of the integral over the set [|t| ≤ θ n/E|X|3 ] is bounded by a term like the right-hand side of (j) (in which |t|3 and t6 are replaced in those integrals by t2 and |t|5 , to account for division by |t| in the integrand of (r)). This completes the proof of (17) when X is not distributed on a grid. For (18), the initial region of integration in (r) must be [|t| ≤ an/EX 4 ], and then an/EX 4 will also appear below the norm term. Moreover, we will now use θ for a, since only O(1/n) is required.

5.

EDGEWORTH EXPANSIONS

395

Consider theorem 14.5.2(b). We note that 1 γ1 1 γ2 γ12 √ H2 (·) + H3 (·) + Dn (·) ≡ Fn (·)−F0 (·)+f0 (·) (s) H5 (·) n 3! n 4! 2 · (3!)2 has derivative Dn (x) = dn (t)

(u)

= fn (x) − f0 (x) + f0 (x)

1 γ1 √ [2x − xH2 (x)] n 3!

1 γ2 γ12 2 4 2 + f0 (x) [3(x − 1) − xH3 (x)] + [5(x − 6x − xH5 (x)] n 4! 2(3!)2 1 γ1 1 γ2 γ12 √ = fn (x)−f0 (x)+f0 (x) H3 (x) + H4 (x) + H6 (x) , n 4! 2(3!)2 n 3!

and this is the same dn as in (m) of the previous proof. Thus the final details are nearly the same as before. 2

Exercise 5.4

Complete the details in the previous proof of theorem 5.2(b).

Exercise 5.5 Consider a non-iid case in which all dfs Fnk have third and/or fourth moments that are of the same order. Then all of the previous results still obtain. Exercise 5.6 (Large deviations) Suppose the moment generating function (or mgf ) MX (t) ≡ EetX of the rv X √is finite for 0 ≤ |t| < . Let X1 , X2 , . . . be iid ¯ n − µ) and let F0 (·) denote the N (0, 1) df. (0, σ2 ). Let Fn (·) denote the df of n(X Show that (22)

[1 − Fn (xn )]/[1 − F0 (xn )] → 1,

provided that

xn = o(n1/6 ).

396

6


¯n) Approximating the Distribution of h(X

Example 6.1 Suppose the basic rv X satisfies X ∼ = (µ, σ2 ), with third and fourth central moments µ3 and µ4 . Let X1 , X2 , . . . denote iid repetitions. We consider the ¯ n ), for some h(·) satisfying distribution of h(X (1)

h(x) = h(µ) + h (µ) (x − µ) + 12 h (µ) (x − µ)2 +

1 6

h (µ) (x − µ)3

1 + 24 h (µ) (x − µ)4 + · · ·

(2)

≡ c0 + c1 (x − µ) + c2 (x − µ)2 + c3 (x − µ)3 + c4 (x − µ)4 + · · · .

Then we have the following approximations: ¯ n ) = c0 + 1 1 c2 σ2 + 12 [ 1 c3 µ3 + Eh(X (3) n 2

(4)

n

6

¯ n ) − Eh(X ¯ n ) = c1 (X ¯ n − µ) + h(X

1 2

1 8

c4 σ4 ] + · · · .

¯ n − µ)2 − c2 [(X

¯ n − µ)3 − + 16 c3 [(X

1 n2

1 n

σ2 ]

µ3 ] + · · · .

(5)

¯ n )] = Var[h(X

c21 σ2 +

1 n2

[c1 c2 µ3 + ( 12 c22 + c1 c3 ) σ4 ] + · · · .

(6)

¯ n ) − Eh(X ¯ n )]3 = E[h(X

1 n2

[c31 µ3 + 3 c21 c2 σ4 ] + · · · .

(7)

¯ n ) − Eh(X ¯ n )]4 − 3 Var[h(X ¯ n )] = E[h(X

1 n

0 n2

+··· .

We call h(·) a normalizing transformation if the lead term on the right of (6) equals 0. We call h(·)√a variance stabilizing transformation if the variance of the limiting ¯ n ) − h(µ)] does not depend on the parameter in some family distribution of n [h(X being used to model the distribution of X. (See Bickel and Doksum (1977) and Kendall and Stuart(1977).) 2 Proof. (8)

The first two claims are immediate from exercise 11.6.2: ¯ n − µ)3 = 12 µ3 . E(X

(9)

¯ n − µ)4 = E(X

n

1 n3

[E(X − µ)4 + 3(n − 1) σ 4 ] =

1 n2

3 σ4 +

1 n3 (µ4

− 3σ 4 ) .

Squaring the equation in (4) gives ¯ n − µ)2 + c1 c2 [E(X ¯ n − µ)3 − 0] ¯ n )] = c2 E(X Var[h(X 1 (a) ¯ n − µ)2 − 1 σ2 ]2 + 1 c1 c3 [E(X ¯ n − µ)4 − 0] + · · · + 41 c22 E[(X n 3 (b)

=

1 n

c21 σ2 + c1 c2

(c)

=

1 n

c21 σ2 +

1 n2

1 n2

µ3 +

[c1 c2 µ3 +

1 4 1 2

c22 [ n12 3 σ 4 −

(e)

= c31

1 n2

µ3 + 3 c21

1 2

σ4 ] +

1 3

c1 c3 [ n12 3 σ 4 ] + · · ·

c22 σ4 + c1 c3 σ4 ] + · · · .

Cubing the equation in the second claim gives ¯ n )−Eh(X ¯ n )]3 = c3 E(X ¯n −µ)3 +3 c2 (d) E[h(X 1

1 n2

1 1 2

¯ n −µ)4 − 1 σ2 1 σ2 ]+· · · c2 [E(X n n

c2 [ n12 3 σ 4 − n12 σ4 ] + · · · =

1 n2

[c31 µ3 + 3 c21 c2 σ4 ] + · · · .

[Clearly, rigorous bounds via Taylor series and (13.4.8) would allow the error terms above to be declared to be o(·) of a bound on the last indicated term.] 2

6.

¯N ) APPROXIMATING THE DISTRIBUTION OF H(X

Example 6.2 (Wilson–Hilferty approximation) (10)

{Vn1/3 − (1 −

2 9n )}/

2 9n

397

Let Vn ∼ = Chisquaren /n. Then

is very nearly N (0, 1) .

This follows from the example above with X ≡ Chisquare1 , µ = 1, σ2 = 2, µ3 = 8, h(x) = x1/3 , c0 = 1, c1 = 1/3, c2 = −2/9, c3 = 10/27, c4 = −80/81. Then ¯n) = 1 − Eh(X

2 9n

+

0 n2

+···,

¯ n )] = Var[h(X

2 9n

+

0 n2

+ ···,

(11) ¯ n ) − Eh(X ¯ n )]3 = E[h(X

0 n2

+··· .

This approximation is known to work very well, and the calculations of (11) should give some insight into this. There is another way to look at this. Consider the family of transformations h(x) = xγ , for some γ. We wish to choose γ so that ¯ n ) is 0 up to terms of order 1/n2 . Now, h (x) = γ xγ−1 and the skewness of h(X γ−2 h (x) = γ(γ − 1) x , while µ = 1, σ2 = 2, and µ3 = 8. Thus the lead term is h (µ)3 µ3 + 3 [h (µ)]2 h (µ) σ4 = γ 3 · 8 + 3 γ 2 · γ(γ − 1) · 4 = 4 γ 3 [2 + 3 (γ − 1)] , which equals 0 if γ = 13 .

2

Example 6.3 (Normalizing transformation for the Poisson) Suppose that the rv ¯n X ∼ = Poisson(λ). Let X1 , X2 , . . . be iid Poisson(λ). What transformation h of X 2 has skewness 0 for the term of order 1/n ? To answer this, suppose that h(x) = xγ , so that h (x) = γ xγ−1 and h (x) = γ(γ − 1) xγ−2 . Since µ = λ, σ2 = λ, and µ3 = λ, the coefficient of 1/n2 in the expression for the third central moment of the ¯ n ) under consideraton is rv h(X h (µ)3 µ3 + 3 [h (µ)]2 h (µ) σ4 = [γ λγ−1 ]3 · λ + 3 [γ λγ−1 ]2 · γ(γ − 1) λγ−2 · λ2 = γ 3 λ3 γ−2 [1 + 3(γ − 1)] , and this equals 0 if γ = 23 . Note that the resulting transformation h(x) = x2/3 differs from the variance stabilizing transformation h(x) = x1/2 . 2 Exercise 6.1 Powers of x do not always suffice. (a) Consider the binomial, which √ is a sum of iid Bernoulli(p) rvs. Show that the transformation h(x) = arcsin( x ) is variance stabilizing, while the transformation x h(x) = 0 [t(1 − t)]−2/3 dt is a normalizing transformation. (b) For the sample correlation coefficient based on bivariate normal observations, show that Fisher’s transformation h(x) = 12 log((1 + x)/(1 − x)) is both variance stabilizing and normalizing.

Chapter 15

Infinitely Divisible and Stable Distributions 1

Infinitely Divisible Distributions

Definition 1.1 (Triangular arrays, and the uan condition) A triangular array is just a collection of rvs Xn1 , . . . , Xnn , n ≥ 1, such that the rvs in the nth row are independent. Call it a uan array if the uniform asymptotic negligibility condition holds, that is (1)

max P ( |Xnk | ≥ ) → 0

1≤k≤n

as n → ∞ for all > 0.

The uan condition is a natural one for preventing one term from dominating the whole sum. The Problem: Let Sn ≡ Xn1 + · · · + Xnn denote the nth row sum of a uan array. (i) Find the family of all possible limit laws of Sn . (ii) Find conditions for convergence to a specified law of this form. Find specialized results for further restrictions on the uan array. (a) Suppose variances exist. (b) Suppose the limit law is normal or Poisson. (c) Consider Sn = [(X1 +· · ·+Xn )−Bn ]/An for a singly subscripted sequence of iid rvs X1 , . . . , Xn , . . . . Some of the results in this chapter are stated with only indications of the proofs. The goal in this chapter is simply to develop some rough understanding of the subject. We will see in this section that the set of all possible limit laws of row sums Sn ≡ Xn1 + · · · + Xnn of a uan array of Xnk ’s is exactly the class of infinitely divisible laws, which we now define. 399

400 CHAPTER 15. INFINITELY DIVISIBLE AND STABLE DISTRIBUTIONS Definition 1.2 (Infinitely divisible) Call both the rv Y and its distribution infinitely divisible (id ) if for every value of n it is possible to decompose Y into n iid components as (2)

Y ∼ = Yn1 + · · · + Ynn

for some iid rvs Yn1 , · · · , Ynn .

We denote the class of all id distributions by I; the subclass with finite variance is denoted by I2 . (We remark that the Yni ’s of this definition form a uan array, but this needs to be shown.) Exercise 1.1 (Chf expansions for the uan array Xn1 + · · · + Xnn ) uan array of rvs Xnk , as in (1). (a) Let Fnk and φnk denote the df and the chf of Xnk . Show that (3)

[max1≤k≤n |φnk (t) − 1|] → 0

Consider a

uniformly on every finite interval.

[Hint. Integrate over |x| < and |x| ≥ separately to obtain |φnk (t) − 1| ≤ δ + 2P (|Xnk | ≥ ), from which point the result is minor.] We then define (as will be useful regarding an expansion of Log (1 + (φnk (·) − 1)) below) n (4)

n (t) ≡ k=1 |φnk (t) − 1|2 . n 2 2 ) , and σn2 ≡ 1 σnk ≤M 0. Γ(z) = 0

Thus, the digamma function ψ(·) trivially satisfies ∞ z−1 −t d 1 d (12) log Γ(z) = Γ (z)/Γ(z) = Γ(z) e dt ψ(z) ≡ dz dz 0 t (13)

=

1 Γ(z)

∞ 0

(log t) tz−1 e−t dt .

∞ 1 Thus, −γ = 0 (log t) e−t dt. Also, Γ(z) = 0 (− log t)z−1 dt. Repeat and then differentiate the formula Γ (z) = Γ(z) ψ(z) to obtain (14)

Γ (z) = Γ(z) ψ(z) and Γ (z) = Γ(z) ψ (z) + Γ (z) ψ(z) = Γ(z) {ψ (z) + [ψ(z)]2 } .

Continued differentiation will produce additional formulas. There is also an easy duplication formula (15)

Γ(2z) =

22z−1/2 Γ(z) Γ(z + 12 ) .

√1 2π

(Just apply √ (7) repeatedly to the left-hand side of (15), and then cancel terms. Use Γ( 12 ) = π, as will be shown to be true below.) Differentiate (15) twice to obtain (16)

ψ(2 z) = 12 [ψ(z)+ψ(z+ 21 )]+log 2

Now, plug z = (17)

1 2

ψ (2 z) = 14 [ψ (z)+ψ (z+ 12 )] .

and

into (16) to obtain (using (10))

ψ( 12 ) = −γ − log 4

ψ ( 12 ) = 3 ψ (1) =

and

π2 2

.

Applying (8) to ψ(·) over and over yields the useful evaluation formulas 1 (18) ψ(n + z) = ψ(1 + z) + n−1 for z = −1, −2, −3, . . . , k=1 k+z ψ(n) = ψ(1) +

n−1

= −γ +

1 k=1 k

ψ(n + 12 ) = ψ( 12 ) +

n−1

1 k=0 k+1/2

n−1

1 k=1 k

,

= ψ( 12 ) + 2

n−1

1 k=0 2k+1

.

Now, Γ(·), log Γ(·), ψ(·), ψ (·), ψ (·), and ψ (·) are all tabled in Abramowitz and Stegun (1972, p. 267) for 1 ≤ z ≤ 2; these can be used in conjunction with the formulas (18), (19), etc. From (5) we have ∞ 1 (19) ψ (z) = k=0 (k+z) for z = −1, −2, −3, . . . , 2 ψ (n) =

π2 6

−

ψ (n + 12 ) =

n−1

π2 2

1 k=1 k2

−4

,

n

∞ ∞ 1 using k=1 (2k−1) 2 = k=1 simple algebra).

1 k=1 (2k−1)2

1 k2

−

1 4

∞

,

1 k=1 k2

=

π2 6 (1

− 14 ) =

π2 8

(from (10) and

548

APPENDIX A. DISTRIBUTION SUMMARIES

Remark 1.1 We state the following asymptotic formulas from Abramowitz and Stegun. As |z| → ∞ with | arg z| < π, both (20)

ψ(z) ∼ log z −

(21)

ψ (z) ∼

1 z

+

1 2z

1 2z 2

−

+

1 12z2

1 6z 3

−

+

1 120z4

1 30z5

+

−

1 252z6

1 42z7

−

+

1 30z9

1 240z8

−···

and

+··· .

Those authors also include the integrated formulas (22)

ψ(z) = −γ +

∞ 0

e−t −e−zt 1−e−t

and ψ (z) =

∞ 0

te−zt (1−e−t )

dt = z

∞ 0

tz−1 (1−t)

dt .

All of the asymptotic and integrated formulas remain valid when differentiated repeatedly. 2 Let Γ(z) ≡

Remark 1.2 (Summary of values of some constants) d and ψ(z) ≡ dz log Γ(z) = Γ (z)/Γ(z). Then n

1 k=1 k

− log n] = −

∞

γ = 0.5772156649 . . . = lim [

(24)

Γ(1) = 1,

Γ( 12 ) =

(25)

ψ(1) = −γ,

ψ( 12 ) = −γ − log 4,

ψ(z + 1) = ψ(z) +

(26)

ψ (1) =

ψ ( 12 ) =

ψ (z + 1) = ψ (z) −

(27)

ψ (k) (z) = (−1)k+1 k!

(28)

π2 6 ,

 2  π /6 ∞ = −2 1  4 π /15 Function Γ(z) Γ (z) ψ(z) ψ (z) ψ (z) ψ (z)

1 j3

√

π,

tz−1 e−t dt

Γ(z + 1) = z Γ(z) ,

π2 2 ,

∞

0

0

(log t) e−t dt ,

(23)

n→∞

∞

1 j=0 (z+j)k+1

1 z

, 1 z2

,

for k ≥ 1 and z = 0, −1, −2, . . . .

= 1.6449340668.. . = −2.4041138063.. . = 6.4939394023.. .

$ if

k=1 k=2 k=3

and z = 1, and z = 1, and z = 1.

z = 1/2

z=1

z=2

1.7724538509 −3.4802309069 −1.9635100260 4.9348022005 −16.8287966441 97.4090910340

1.0000000000 −0.5772156649 −0.5772156649 1.6449340668 −2.4041138063 6.4939394023

1.0000000000 0.4227843351 0.4227843351 0.6449340668 −0.4041138063 0.4939394023

Exercise 1.1 Verify the values in the above table. (The last digit of ψ ( 12 ) is suspect, to within four units.)

1.

THE GAMMA AND DIGAMMA FUNCTIONS

549

Remark 1.3 (Digression on Z ∼ (0, 1) really having mean 0 and variance 1.) = N√ 2 Consider the density f (x) ≡ e−x /2 / 2 π. This is indeed a density, since ∞ 2π ∞ 2 2 2 ( −∞ 21π e−(x +y )/2 dx dy)2 = 0 0 21π e−r r dr dθ = 1 . √ Thus, Γ( 12 ) = π, since ∞ ∞ 2 1 = 2 0 √12 π e−x /2 dx = √22 π 0 e−t √12 t dt = √1π Γ(1/2) . Clearly, EN (0, 1) = 0, and E[N (0, 1)]2 =

√2 π

Γ( 32 ) =

√2 1 π 2

Γ( 12 ) = 1. 2

Remark 1.4 (Digression on Beta(r, s)) Now, 1 (29) B(r, s) ≡ 0 tr−1 (1 − t)s−1 dt = Γ(r) Γ(s)/Γ(r + s)

for all r, s > 0 .

This holds, since

∞∞ Γ(r) Γ(s) = 0 0 xr−1 e−x y s−1 e−y dx dy 1 ∞ uy r−1 s−1 −y/(1−u) y 1 = 0 0 ( 1−u ) y e 1−u 1−u dy du letting u ≡ x/(x + y)

=

with 1∞ 0

0

x = uy/(1 − u) and dx = (1 − u)−2 y du (uv)r−1 [(1 − u)v]s−1 e−v v dv du

letting v ≡ y/(1 − u) with y = (1 − u) dv and dy = (1 − u) dv ∞ r+s−1 −v 1 = 0 v e dv · 0 ur−1 (1 − u)s−1 du = Γ(r + s) B(r, s) . With t ≡ x/(1 + x) (with (1 − t) = 1/(1 + x) and dt = ∞ xr−1 (30) B(r, s) = 0 (1+x) 2 r+s dx Proof.

1 (1+x2 )

dx), obtain

Consider (2). Now, | log(1 + nz ) − nz | = | −

1 z2 2 n2

+

1 z3 3 n3

−···|

for |z| ≤ N/2 and n > N ≤ (a)

≤

|z|2 |z| n2 {1 + n 2

1N 2 n2

.

+

|z|2 n2

+ · · ·} ≤

1 N2 1 4 n2 {1 + 2

+

1 22

+ · · ·}

∞ Thus, for |z| ≤ N/2 the function SN (z) ≡ n=N +1 {log(1+ nz )− nz } is an absolutely and uniformly convergent series of analytic functions, and hence SN (·) is analytic on |z| ≤ N/2, too. Consequently, exp(SN (·)) is, too, and hence also exp(S1 (·)). And since N is arbitrary, S1 (·) is analytic for all complex z; hence zeγz S1 (z) also. Thus, expression (2) is valid. Consider (1). Letting ak ≡ k1 − log k+1 k , we have n 1 n n+1 γn ≡ k=1 n − log n = k=1 ak + log n n n+1 = k=1 ( k1 − log k+1 k ) + log n n 1 t (b) = k=1 0 k(k+t) dt + log n+1 n .

550


So for n > N we have 1 γn − γN ≤ nk=N +1 0 k12 dt + log(1 + n1 ) − log(1 + n (c) = k=N +1 k12 + log(1 + n1 ) − log(1 + N1 ) ,

1 N)

showing that γ ≡ lim γn exists. Perhaps, the following figure 1.1 is better yet.

2

1 f(x) = x

1/1 1

Figure 1.1

Σkn = 1 k1

1/2 2

1/3 3

1/4 4

1/5 5

6

– log n = (shaded area) → γ = .57721566. . . .

(a) Derive Euler’s formula that for z = −1, −2, . . . , ∞ z Γ(z) = n=1 {(1 + 1/n)z /(1 + z/n)} .

Exercise 1.2 (31)

Begin with the expression 1 lim e(1 + Γ(z) = z

1 2

+··· +

1 m −log m)z

m→∞

lim

m

m→∞

n=1 {(1

+ nz ) e−z/n } .

(b) Use this last expression for 1/z Γ(z) and use (2) for Γ(z + 1), and then show that the ratio converges to 1, and so obtain (7). Exercise 1.3 (Bounds on (1 − x/n)n ) Note that (32)

(1 + y) ≤ ey ≤ 1/(1 − y)

for 0 ≤ y ≤ 1 ,

which yields (33)

1 ≥ ey (1 − y) ≥ (1 − y 2 )

for 0 ≤ y ≤ 1.

Use this (with y = t/n) to show that (34)

0 ≤ e−x − (1 − nx )n ≤

x2 −x n e

for all 0 ≤ x < n.

Exercise 1.4 Establish (11). [Hint. Start by considering ∞ n | 0 e−t tz−1 dt − 0 (1 − nt )n tz−1 dt | . Evaluate the second integral repeatedly using integration by parts, and then apply n formula (31) to establish that 0 (1 − nt )n tz−1 dt → Γ(z) for Re(z) > 0.] Exercise 1.5

Verify the term-by-term differentiation leading to (4) and (5).

2.

MAXIMUM LIKELIHOOD ESTIMATORS AND MOMENTS

551

2

Maximum Likelihood Estimators and Moments

Motivation 2.1 (Maximum likelihood estimator, MLE) Suppose the outcome X of some experiment is modeled as a random element on the probability space (X , A, Pθ ), where the particular parameter value θ is an unknown point in the fixed set Θ of all possible parameter values. We suppose that each Pθ µ with ≡ (X1 , . . . , Xn ) , density pθ (x) for x ∈ X , for some fixed σ-finite measure µ. Let X where the Xk ’s denote n independent repetitions of the basic X-experiment. Let us agree to call the joint density function n Lx (θ) ≡ pθ (x ) = 1 pθ (xk ) the observed likelihood function of the observed data value x when we think of the joint density expression as a function of the parameter value θ while holding the experimental outcome x fixed. Then n n (1) θ ≡ (θ) ≡ (θ|x ) ≡ log pθ (x ) = 1 log pθ (xk ) ≡ 1 (θ|xk ) will be called the observed log likelihood function. The maximum likelihood estimate ˆ x ) of the parameter that maximizes the observed likelihood of θ is that value θˆn ≡ θ( This highly intuitive procedure for for the data value x of the experiment X. generating an estimate of the unknown parameter is due to R.A. Fisher. Meanwhile ) is called the theoretical log likelihood, and a formula/algorithm for the θ ≡ (θ|X ˆX ) that maximizes this expression produces the maximum likelihood value θˆn ≡ θ( estimator. Since the value that maximizes the likelihood is the same value that maximizes the log likelihood, and since sums are typically easier to maximize than products, it is usual to think of the MLE θˆn of θ as that value of the parameter that maximizes the log likelihood θ . It is often possible to determine the MLE θˆn by setting the derivative of the log likelihood with respect to θ equal to zero and solving the resulting likelihood equation for an expression for the MLE θˆn . Suppose now that the parameter of the model is actually an m dimensional ) is vector θ ≡ (θ1 , . . . , θm ) . Associated with the theoretical log likelihood (θ|X ˙ ˙ X ) whose rth coordinate is given by the m × 1 score vector ˙θ ≡ (θ) ≡ (θ| n ˙ ∂ ≡ ∂ log pθ (X ) = n (2) ˙r (θ) ≡ ˙r (θ|X) k=1 ∂θr log pθ (Xk ) = k=1 r (θ|Xk ) . ∂θr ˆX ) of the vector θ is now typically determined The formula for the MLE θˆn ≡ θ( by stating that the likelihood equation (it’s an m × 1 vector equation) (3)

0 = ˙θ

has as its solution the MLE θˆn .

˙ X ) = 0. Suppose our model is “sufficiently regular” for the following Usually, Eθ (θ| to exist. Define the m × m Fisher information matrix for one observation X as (4)

T ˙ ˙ Iθ ≡ Eθ {(θ|X) (θ|X) } = |[ Eθ {˙r (θ|X) ˙s (θ|X)} ]| = |[ Eθ {−¨rs (θ|X)} ]|

where ¨rs (θ|X) ≡

∂ ∂ ∂θr ∂θs

(θ|X) (and where the last equality is typically true). The ˆX ) (with its corresponding numerical fundamental result about the MLE θˆn ≡ θ( ˆ x )) is that in a “sufficiently regular” model, θˆn satisfies data values θˆn = θ( √ (5) n [θˆn − θ] →d N (0, Iθ−1 ) . 2

552


Location and Scale Models Suppose we model an experimental outcome Y via a location-scale model as (6)

Y =ν+τ W

where W has a fixed density f (·) on the line, with −∞ < ν < ∞ and τ > 0. Let (7)

g(·) ≡ − log f (·)

g (·) = −f (·)/f (·) .

with

Let wp ≡ F −1 (p) denote the pth quantile of W , so that Y has pth quantile (8)

yp ≡ ν + τ wp

for each 0 < p < 1.

It is assumed throughout this section that the df F (·) satisfies sufficient regularity conditions for everything that follows to make sense. We will loosely refer to these cases as regular models. Let θ ≡ (ν, τ ) and Θ ≡ R × R+ . Then one observation Y has (9)

y−ν y−ν θ = log pθ (y) = log τ1 f ( y−ν τ ) = − log τ + log f ( τ ) = − log τ − g( τ ) .

The first partials (that is, the coordinates of the score vector) are given by (10)

y−ν ˙τ = − τ1 + y−ν τ2 g ( τ ) = = ˙ν = τ1 g ( y−ν τ )

1 τ 1 τ

[w g (w) − 1]

=

[g (w)]

=

1 τ 1 τ

[−1 − w f (w)/f (w)] , [−f (w)/f (w)] .

The two quantities in square brackets in (10) have 0 means, both here and in (12) below; they are called the optimal score functions for scale and for location. The second partials are given by 2

(11)

(y−ν) y−ν −¨τ τ = − τ12 + 2 y−ν g ( y−ν τ3 g ( τ ) + τ4 τ ), 1 y−ν ¨ −νν = τ 2 g ( τ ) , y−ν −¨τ ν = 12 g ( y−ν ) + y−ν ), 3 g ( τ

τ

τ

τ

or (in the more revealingly w-format) are given by

(12)

−¨τ τ = −¨νν = −¨τ ν =

1 τ2 1 τ2 1 τ2

{w2 g (w) + 2[w g (w) − 1] + 1} , {g (w)} , {w g (w) + [g (w)]} .

˙ X) = 0) that We observe from (10) (and also from Eθ (θ; (13)

E {g (W )} = 0

and

E {W g (W )} = 1 .

Then the Fisher information matrix is given from (12) and (13) by 1 −Eθ ¨τ τ −Eθ ¨τ ν E{W 2 g (W ) + 1} EW g (W )} (14) Iτ ν ≡ Iθ = = −Eθ ¨τ ν −Eθ ¨νν E{W g (W )} E{g (W )} τ2 ! " f (W ) 2 f (W ) 2 1 E{[−1 − W f (W ) ] } E{W [ f (W ) ] } 1 1 c00 c01 (15) = 2 ≡ C ≡ . τ τ2 τ 2 c01 c11 E{W [ f (W ) ]2 } E{[ f (W ) ]2 } f (W )

f (W )

Note that E{W g (W )} = 0 when W is symmetrically distributed. Moreover, the matrix C does not depend on θ.

2.

MAXIMUM LIKELIHOOD ESTIMATORS AND MOMENTS

553

Suppose now that an iid sample Y1 , . . . , Yn is observed. Then the MLEs τˆ ≡ τˆn and νˆ ≡ νˆn are typically obtained by jointly solving 0 = ˙τ = nτ { n1 ni=1 Wi g (Wi ) − 1} = nτ { W g (W ) − 1} , n (16) 0 = ˙ν = n { 1 g (Wi ) − 1} = n { g (W ) − 1} . τ

n

i=1

τ

That is, the solutions τˆ and νˆ are such that when the Wi themselves are replaced by the residuals Wi ≡ (Yi − νˆ)/ˆ τ in these likelihood equations in (16), then these expressions do indeed yield the intended value of 0. We estimate I(τ, ν) by the observed information matrix Iˆn which is now defined by ! " ˆ 2 g (W ˆ g (W ˆ )+1 W ˆ) 1 W (17) Iˆn ≡ 2 ˆ g (W ˆ) ˆ) τˆ W g (W 1 cˆ00 cˆ01 1 (18) ≡ 2 ≡ 2 Cˆn . cˆ01 cˆ11 τˆ τˆ The fundamental theorem about the solutions of the likelihood equations τˆn and νˆn 1/2 in location-scale models is that under sufficient regularity (recall (9.3.9) for Cˆn ) √ n (ˆ τn − τ )/ˆ τn 1/2 ˆ √ (19) Cn →d N (0, I2 ) n (ˆ νn − ν)/ˆ τn (for the 2 × 2 identity I2 ), and also √ τn − τ )/ˆ τn √ n (ˆ (20) →d N (0, C −1 ) . n (ˆ νn − ν)/ˆ τn ˆ n ≡ |[dîj ]| ≡ Cˆ −1 . We note the conventional We define D ≡ |[dij ]| ≡ C −1 and D n wisdom that the approximation in (19) is typically somewhat more accurate than the approximation in (20). Note also that the solutions τˆ and νˆ of the likelihood equations are invariant, in that for all a > 0 and all real b we have (21)

+ b 1 ) = a νˆ(Y ) τˆ(aY

and

+ b 1 ) = a νˆ(Y ) +b. νˆ(aY

In case the true density of W is modeled to belong to some family of densities fγ (·) indexed by the parameter γ, then we need to expand the likelihood equations to include the appropriate terms. Thus, we let (22)

˙γ = −g( ˙ y−ν τ ),

ˆ ) = 0 at the MLEs. and note that g( ˙ W

The MLEs will still be invariant, now satisfying (for a > 0, b real, and all γ) (23)

+ b 1 ) = γˆ (Y ) , τˆ(aY + b 1 ) = a τˆ(Y ) , νˆ(aY + b 1 ) = a νˆ(Y ) +b. γˆ (aY

The Fisher information matrix now becomes  − τ1 E{W g˙ (W )} E{¨ g(W )} 1 (24) Iγ,τ,ν ≡  − τ E{W g˙ (W )} τ12 E{W 2 g (W ) + 1} 1 − τ1 E{g˙ (W )} τ 2 E{W g (W )}

 − τ1 E{g˙ (W )} 1 . τ 2 E{W g (W )} 1 E{g (W )} τ2

(Note how τ appears in this matrix. The expectations do not depend on τ or ν.)

554


Moments Definition 2.1

For a rv W , let µ ≡ EW and σ2 ≡ Var[W ]. Then let 1/2

(25)

µ3 ≡ E(W − µ)3

and

γ1 ≡ β1

(26)

µ4 ≡ E(W − µ)4 ,

β2 ≡ µ4 /σ 4 ,

and

≡ µ3 /σ 3 ,

γ2 ≡ β2 − 3 .

Recall that for a rv W we agreed that φ(t) ≡ EeitW denotes the characteristic function (or chf) and that M (t) ≡ EetW denotes the moment generating function (or mgf). We also agreed that ψ(t) ≡ Log φ(t) denotes the cumulant generating function (or cgf). The statement of exercise 13.4.6 contains all the useful facts about this cgf. However, we will also call C(t) ≡ log M (t) the cgf (for those values t for which it is well defined). The so called method of moments estimator (or MME) θ¯ ≡ θ¯n of θ is computed ¯ n and also ¯ n for θ¯n (or by jointly solving µ(θ) = X by solving the equation µ(θ) = X 2 µ2 (θ) = Xn for θ ≡ (θ1 , θ2 ) in the two dimensional case), etc.

3.

EXAMPLES OF STATISTICAL MODELS

555

3

Examples of Statistical Models

Discrete Distributions Exercise 3.1 (Binomial) (1)

µ = np,

(2)

γ1 =

Show that for the Binomial(n, p) distribution, µ3 = npq(q − p),

σ2 = npq,

q−p √ npq

and

γ2 =

1−6pq npq

µ4 = 3(npq)2 + npq(1 − 6pq) ,

.

This distribution has mgf M (t) = [1 + p(et − 1)]n for all t. Exercise 3.2 (Poisson) (3)

Show that the cgf of the Poisson(λ) distribution is ∞ C(t) = λ(et − 1) = 0 λtk /k! ,

and so the mth cumulant κm equals λ for all m. Show that (4)

µ = σ2 = µ3 = λ,

µ4 = λ(1 + 3λ),

Exercise 3.3 (Negative binomial)

γ1 =

√1 , λ

γ2 =

1 λ

.

Show that the NegBiT(r, p) has

(5)

µ = r/p, σ2 = rq/p2 , µ3 = rq(1+q)/p3 , µ4 = [3(rq)2 +rq(1+4q +q 2 )]/p4 ,

(6)

γ1 =

1+q √ rq

and

γ2 =

1+4q+q2 rq

.

The only change for NegBiF(r, p) is that the mean is now rq/p = r/p − 1. Exercise 3.4 (Discrete uniform distribution of ranks) Show that the Discrete Uniform(1, N ) distribution that takes on each value 1, . . . , N with probability 1/N has moments given by N +1 2 ,

(7)

µ=

(8)

γ1 = 0,

− 1),

σ2 =

2 1 12 (N

γ2 =

3 3N 2 −7 5 N 2 −1 ,

µ4 =

2 1 240 (3N

− 7)(N 2 − 1),

µ6 =

4 1 112 (3N

− 18N 2 + 31)(N 2 − 1) .

All odd central moments equal 0. We state for convenient reference that: n 2 1 n 1 1 k = 2 n(n + 1) , 1 k = 6 n(n + 1)(2n + 1) , (9) n 3 1 2 n 4 1 2 2 1 k = 4 n (n + 1) , 1 k = 30 n(n + 1)(2n + 1)(3n + 3n − 1) . Exercise 3.5 (Hypergeometric) The Hypergeometric(pN, qN ; n) distribution has: µ = np, γ1 =

q−p √ npq

σ2 = npq [1 − ×

1−

n−1 N −1

(N −1)(N +6) β2 = 3 (N −2)(N −3) +

n−1 N −1

×

1 N pq

],

N −2n N −2

µ3 = npq(q − p) NN−2n −2 , ,

[1 − 6 NN+1 (pq +

n(N −n) )] . N2

556


Classical Normal Theory Distributions Exercise 3.6 (Normal) Let Z ∼ = N (0, 1). Then γ1 = 0 and γ2 = 0. Thus, γ1 and γ2 take the standardized values 0 and 0 for any normal distribution. Also, √ (10) E|Z|r = 2r/2 Γ( r+1 for all r > 0. 2 )/ π , Thus, E|Z| = 2/π and EZ 4 = 3. The mgf of N(µ, σ2 ) is M (t) = exp(µt + σ2 t2 /2). Exercise 3.7 (Gamma) Verify what follows. (a) If Y ∼ = Gamma(r, 1), then √ (11) µ = r, σ2 = r, γ1 = 2/ r , γ2 = 6/r . Also, E(1/Y ) = 1/(r − 1) for r > 1. The mode is located at (α − 1)β. (b) The Gamma(α, β) distribution has cgf C(t) = −α log(1 − βt) for |t| < 1/β, and so the mth cumulant is given by κm = (m − 1)! αβ m . Thus √ (12) µ = αβ, σ2 = αβ 2 , γ1 = 2/ α, γ2 = 6/α , (13)

µ2 = (α + α2 )β 2 ,

µ3 = 2αβ 3 ,

µ4 = (3α2 + 6α)β 4 .

Let θ ≡ (α, β) . The log likelihood θ ≡ log pθ (x) and its partial derivatives are log pθ (x) = − log Γ(α) − α log β + (α − 1) log x − x/β , (14)

˙α = −ψ(α) − log β + log x and ˙β = −¨αα = ψ (α) ,

−¨ββ =

1 β3

α β

+

[2x − αβ] ,

x β2

=

1 β2

[x − βα] ,

−¨αβ =

1 β

.

˙ X) = 0), we learn that From this (and Eθ (θ; (15) (16)

Eθ (log X) = ψ(α) + log β and Eθ X = αβ , ψ (α) β1 1 α −β −1 and Iθ = Iα,β = Iθ = . α 1 α ψ (α) − 1 −β β 2 ψ (α) β β2

Two other natural parameterizations of the Gamma involve the choice α, ν ≡ 1/β or the choice α, µ ≡ αβ. These yield α ψ (α)−1 0 ψ (α) − ν1 α (17) Iα,ν = or I = . α,µ α α − ν1 0 µ2 ν2 Suppose tht V ∼ = χ2α /χ2β , where α, β > 0. Verify that

Exercise 3.8

β−2r α β EV r = B( α+2r 2 , 2 )/B( 2 , 2 )

whenever α > 0 and β > 2r with r > 0. Verify that r

V α+2r β−2r+2s E{ (1+V )/B( α2 , β2 ) )s } = B( 2 , 2

whenever α > 0, β > 0, α + 2r > 0 and β − 2r + 2s > 0.

3.


557

Exercise 3.9 (Student-t) Verify what follows. (a) If T ∼ = Student-tβ , then √ β Γ((β − 1)/2) √ (18) E|T | = for β > 1 , π Γ(β/2) β β−2

(19)

Var[T ] =

for β > 2 ,

(20)

γ1 = 0 for β > 3

and

γ2 =

6 β−4

for β > 4 .

(b) Suppose that pθ (x) = fβ ((x−θ1 )/θ2 ), where fβ (·) denote the Student-tβ density subject to θ2 > 0 with θ1 real. Verify that: (21) (22) (23)

−

f (x) (β + 1)x = f (x) β + x2

∞ −∞

∞

−∞ {−[1 +

Iθβ =

xf (x)/f (x)]}2 dx = 2β/(β + 3) ,

2β 0 β+3 β+1 0 β+3

=

iscale 0

n n−2

σ2 =

γ2 =

8(n − 4) m(m + n − 2)

γ1 =

.

for n > 4 ,

1/2

2m + n − 2 n−6

for n > 6 ,

12{(n − 2)2 (n − 4) + m(m + n − 2)(5n − 22)} m(n − 6)(n − 8)(m + n − 2)}

r r+s 2(s − r) r+s+2

β2 = 3

for n > 8 .

(a) If X ∼ = Beta(r, s) with r, s > 0, then

Exercise 3.11 (Beta) µ=

ilocation

for n > 2 ,

2 n2 (m + n − 2) m(n − 2)2 (n − 4)

γ1 =

0

If X ∼ = Snedecor-Fm,n , then

Exercise 3.10 (Snedecor-F ) µ=

xf (x) β(x2 − 1) = , f (x) x2 + β

{−f (x)/f (x)}2 dx = (β + 1)/(β + 3) ,

(24)

−1−

and

and

σ2 =

r+s+1 rs

1 r s , r+s+1 r+s r+s

1/2 ,

r + s + 1 (r + s)rs + 2(r2 − rs + s2 ) . r+s+3 (r + s + 2)rs

558


(b) Let θ ≡ (α, β) . The log likelihood θ ≡ log pθ (x) and its partial derivatives are log pθ (x) = log Γ(α + β) − log Γ(α) − log Γ(β) + (α − 1) log x + (β − 1) log(1 − x) , (25)

˙α = −ψ(α + β) − ψ(α) + log x and ˙β = −ψ(α + β) − ψ(β) + log(1 − x) , −¨αα = ψ (α + β) + ψ (α), −¨ββ = ψ (α + β) + ψ (β), −¨αβ = ψ (α + β).

˙ X) = 0) we learn that From this (and Eθ (θ; (26) (27)

Eθ (log X) = ψ(α)+ψ(α+β) and Eθ (log(1−X)) = ψ(β)+ψ(α+β) , ψ (α) + ψ (α + β) ψ (α + β) . I(α,β) ≡ Iθ = ψ (α + β) ψ (β) + ψ (α + β)

Other Distributions Exercise 3.12 (Double Exponential) (a) Show that the Double Exponential(0, 1) distribution with density f (w) = 12 e−|w| on the line has mean 0, variance 2, skewness γ1 = 0, tail heaviness γ2 = 3, β2 ≡ µ4 /σ 4 = 4!/22 = 6, and 4th cumulant κ4 ≡ µ4 − 3σ 4 = 12. Note that all odd moments of W equal 0 and that |W | ∼ = Exponential(1). (b) Consider a location-scale model Y = ν+τ W with W ∼ = Double Exponential(0, 1). Determine the MLEs τˆ and νˆ. Show next that log f (w) = −|w| − log 2, with score functions −f (w)/f (w) = sign(w) for w = 0, and −[1 + wf (w)/f (w)] = |w| − 1 for w = 0. Show that the Fisher information matrix equals τ12 I2 . Exercise 3.13 (Logistic) Verify the following facts. (a) The Logistic(0, 1) distribution with density (28)

f (w) = e−w /(1 + e−w ) = 1/(e−w/2 + ew/2 )2

on (−∞, ∞)

has mean 0, variance π 2 /3, skewness γ1 = 0, tail heaviness γ2 = 1.2 and 4th cumulant κ4 = 2π 4 /15. All of its odd moments equal 0. In fact, the moment generating function M (·) and the cumulant generating function C(·) are given by (29)

M (t) = Γ(1 − t) Γ(1 + t)

and

C(t) = log Γ(1 − t) + log Γ(1 + t)

for |t| < 1. Thus, the mth cumulant equals 0 for m odd and it equals 2ψ(m−1) (1) for m even. (b) Consider a location-scale model Y = ν + τ W with W ∼ = Logistic(0, 1). Show how to determine the MLEs τˆ and νˆ. Verify that −f (w)/f (w) = 2F (w) − 1. Then show that the Fisher information matrix is diagonal, where c11 = 1/3 is easy and where c00 = π12 + 13 requires substantial effort. Exercise 3.14 (Uniform) Show that the Uniform(−1, 1) distribution has mean 0, variance 13 , γ1 = 0 and γ2 = −1.2.

3.


559

Exercise 3.15 (Log Gamma(α) distribution) Suppose X ∼ = Gamma(α, 1) with density fX (x) = xα−1 e−x /Γ(α) for x ≥ 0, for some α > 0. Then W ≡ log X has the Log Gamma distribution with density function (30)

1 Γ(α)

f (w) =

exp(α y − ey )

on (−∞, ∞).

Verify the following facts. The mode is located at log α. (a) The cumulant generating function of W is (31)

C(t) = log{Γ(t + α)/Γ(α)} = log M (t)

for |t| < α .

Thus, the mth cumulant of W is given by dm (32) = ψ (m−1) (α) κm = dt m C(t) t=0

for the digamma function ψ(·). From this one obtains (33)

σ2 = ψ (α)

µ = ψ(α),

γ2 = ψ (α)/[ψ (α)]2 .

3/2

γ1 = ψ (α)/[ψ (α)]

(b) Recalling (14), we obtain E{eY } = M (t) t=1 d Y E{Y e } = dt M (t) t=1 (34) d2 2 Y E{Y e } = dt2 M (t)

= Γ(α + 1)/Γ(α) = α , = Γ (α + 1)/Γ(α) = α ψ(α + 1) , = Γ (α + 1)/Γ(α)

t=1

= α{ψ (α + 1) + [ψ(α + 1)]2 } . Then note that (35)

g(w) ≡ log Γ(α) − α w + ew ,

g (w) = −α + ew

and g (w) = ew .

Thus the location-scale model Y = ν + τ W has Fisher information given by (A.1.14), which simplifies to E{Y 2 eY + 1} E{Y eY } 1 (36) ≡ τ12 C (α) Iτ,ν ≡ τ 2 E{Y eY } E{eY } (37)

=

1 τ2

1 + α{ψ (α + 1) + [ψ(α + 1)]2 } α ψ(α + 1)

α ψ(α + 1) α

.

Specialize equations (A.2.16) and (A.2.17) to the present model. Exercise 3.16 (Extreme Value) The so called Extreme Value(0, 1) distribution for minimums corresponds to the special case of the previous exercise with α = 1. Thus, W ≡ log X has the Extreme Value distribtuion with density (38)

f (w) = exp(y − ey )

on (−∞, ∞).

560


Verify the following facts. (a) The cumulant generating function is (39)

C(t) = log Γ(t + 1) = log M (t)

for |t| < 1 .

From (32) one obtains µ = ψ(1) = −γ (40)

2 = π6 3/2

2

σ = ψ (1) γ1 = ψ (1)/σ

γ2 = ψ (1)/σ4

= −0.5772157 . . . = 1.644934 . . . ∞ = −2 1 i13 /(π2 /6)3/2 = −1.139547 . . . =

π 4 /15 (π 2 /6)2

= 2.4000 . . .

∞ using 1 i−3 = 1.2020569032 . . . . This distribution trivially has its mode at 0. The df equation p = F (y) = 1 − exp(−ey ) then yields a pth quantile given by (41)

1 1 yp = log2 ( 1−p ) = log(log 1−p )

for 0 < p < 1.

Note that two particularly useful values are y.5 = log2 2 = −0.3665129 . . . and yp = 0 for p = 1 − 1/e = .3621206 . . . . (b) Recalling (34), we obtain E{eY } = M (t) = Γ(2) = 1.0000000 , t=1 d E{Y eY } = dt M (t) = Γ (2) = 1 ψ(21) = 1 − γ = 0.422784 . . . , (42) t=1 d2 E{Y 2 eY } = dt = Γ (2) = {ψ (2) + [ψ(2)]2 } = 0.823681 . . . . 2 M (t) t=1

Then note that (43)

g(w) ≡ − log f (w) = −w + ew ,

g (w) = −1 + ew ,

and g (w) = ew .

Thus, the location-scale model Y = ν + τ W has Fisher information given by (37), and this simplifies to 2 π 1−γ + (1 − γ)2 (44) Iτ,ν = τ12 6 ≡ τ12 C (α) 1 1−γ (45)

=

1.823681 . . . 0.422784 . . .

1 τ2

with (46)

−1 Iτ,ν = τ2

(47)

= τ2

6 π2

1 −(1 − γ)

0.607927 . . . −.473999 . . .

0.422784 . . . 1.000000 . . .

,

−(1 − γ) π2 2 6 + (1 − γ) −.473999 . . . 0.977502 . . .

Specialize equations (16) and (17) to the present model.

3.


Exercise 3.17 (Weibull) (48)

By X ∼ = Weibull(α, β) we mean that

β xβ−1 exp(−(x/α)β ) αβ

fX (x) =

561

for x ≥ 0 .

This distribution is often used to model lifetimes. The survival function or reliability function R(x) ≡ P (X > x) is given by (49)

R(x) = 1 − F (x) = P (X > x) = exp(−(x/α)β )

for x ≥ 0 .

Verify the following claims. (a) It is easy to compute that (50)

µ = α Γ(1 + 1/β)

σ2 = α2 {Γ(1 + 2/β) − Γ2 (1 + 1/β)} .

and

Note that (51)

V ≡ (X/α)β ∼ = Exponential(1) and W ≡ log V ∼ = Extreme Value(0, 1).

It thus holds that Y ≡ log X satisfies the location-scale model (52)

Y =ν+τ W

with ν ≡ log α and τ = 1/β ,

where W has the Extreme Value(0, 1) density f (w) = exp(w − ew ) on the line. Note 1 }1/β of X and yp of Y satisfy that the quantiles xp = α{log 1−p (53)

1 β(yp − log α) = β(xp − log α) = log2 ( 1−p ) = log(log

1 1−p )

for 0 < p < 1.

n ) “should” be a straight Thus a plot of the ordered values log Xn:i vs log2 ( n−i+1/2 line of slope 1/β and horizontal intercept log α at pα ≡ 1 − 1/e = 0.6321206 . . . . (b) Show that the log likelihood for one observation satisfies

(54)

˙α =

β α

[V − 1]

and

˙β =

1 α

[1 + V − V log V ] ,

and also satisfies (55)

−¨αα = −¨ββ = −¨αβ =

β α2 {(β + 1)V − 1} , 2 1 β 2 {1 + V log V } , 1 α {1 − V − V log V } .

Thus, the Fisher information is given by β 2 /α2 −(1 − γ)/α Iαβ = −(1 − γ)/α [(π2 /6) + (1 − γ)2 ]/β 2 (56) β 2 /α2 −0.4227843 . . . /α = −0.4227843 . . . /α 1.823681 . . . /β 2

.

The determinant of Iαβ equals (π2 /6)/α2 = 1.644934 . . . /α2 . Note also that 0.607927 . . . β 2 −0.257022 . . . α −1 (57) Iαβ = . −0.257022 . . . α 1.108665 . . . α2 /β 2

562


Then note that (58)

E{V log V } = ψ(2) = 1 − γ , E{V log2 V } = ψ (2) + [ψ(2)]2 = −1 + (π 2 /6) + (1 − γ)2 .

(c) Show that the MLEs α ˆ and βˆ satisfy both (59)

α ˆ=

X βˆ

1/βˆ and

ˆ , 0 = H(β)

where (60)

H(β) ≡

X β log X Xβ

− log X −

1 β

.

Use Cauchy-Schwarz to verify that H (β) > 0 for all β > 0. Show that the NewtonRaphson iteration (61)

βˆm = βˆm−1 − H(βˆm−1 )/H (βˆm−1 )

for m = 1, 2, . . .

will necessarily converge quickly to a solution, starting from an initial value of n (62) βˆ0 ≡ (π2 /6)/ n1 1 (log Xi − log X )2 . Develop a formula for the observed information Iˆn . Exercise 3.18 (Pareto) Develop results similar to exercise 3.15(a) for the Pareto density f (w) = β w−(β+1) on w > 1, for β > 0.

4.

ASYMPTOTICS OF MAXIMUM LIKELIHOOD ESTIMATION

563

4

Asymptotics of Maximum Likelihood Estimation

The Model (M0) We suppose that the basic experiment X under consideration is modeled as having distribution Pθ for some value of θ in the set Θ ⊂ Rm . The distributions Pθ are known, but the true value of θ is not. We thus let (X , A, Pθ ) denote a probability space, for each θ ∈ Θ ⊂ Rm , and we suppose that the true distribution of the experimental outcomes X is one of the members Pθ of this family. Suppose also that each Pθ (·) has a density pθ (·) with respect to some fixed σ-finite measure µ on the measurable space (X , A). These Pθ ’s are assumed to be identifiable (that is, θ = θ implies Pθ = Pθ ). We now suppose that X1 , . . . , Xn are ≡ (X1 , . . . , Xn ) . iid repetitions of the basic X-experiment distributed as Pθ . Let X Frequently, it will be convenient to let θ0 denote the true value of the parameter θ. Notation 4.1 We agree that (for one repetition X of the basic experiment) ≡ n L(θ|Xk ), (1) L(θ|x) ≡ Lx (θ) ≡ pθ (x), L(θ) ≡ L(θ|X) k=1 n ≡ (2) (θ|X) ≡ log pθ (X), (θ) ≡ (θ|X) k=1 (θ|Xk ), (3)

(B|X) ≡ supθ∈B (θ|X),

≡ supθ∈B (θ|X), (B) ≡ (B|X)

(4)

˙r (θ|X) ≡

˙ ˙ X) ≡ ˙r (θ|X) m×1 , (θ) ≡ (θ|

(5)

¨rs (θ|X) ≡

∂ ∂θr

(θ|X),

∂ ∂ ∂θr ∂θs

¨ ≡ (θ| ¨ X) ≡ ¨rs (θ|X) m×m . (θ)

(θ|X),

We further agree that (6)

˙ (θ|X) ≡ (˙1 (θ|X), . . . , ˙m (θ|X)) ≡ (the score vector) ,

(7)

I(θ) ≡ Eθ {˙r (θ|X) ˙s (θ|X) } ≡ (Fisher’s information matrix),

(8)

¨ ¯ I(θ) ≡ Eθ {−¨rs (θ|X)} ≡ E{−(θ|X)},

(9)

= L(Θ|X). θˆn ≡ θˆn (X) is a MLE if L(θˆn |X)

(The MLE θˆn may not exist, may not be unique, or may be difficult to determine.) The likelihood ratio statistic (LR-statistic) for testing H0 : θ = θ0 vs H1 : θ = θ0 is defined as a comparison of the likelihoods via (10)

≡ L(Θ)/L(θ0 ); Λn ≡ Λn (X)

and thus

2 log Λn = 2[(Θ) − (θ0 )]. 2

The natural approach to finding the MLE θˆn is to solve the likelihood equations (11)

0 = ˙r (θ|X)

for 1 ≤ r ≤ m.

If we are unsure the solution determines a maximum of the likelihood, we will denote it by θñ and call it a solution of the likelihood equations. Let θ0 be true. Let √ ˙ 0 |X) (12) Zn ≡ Zn (θ0 ) ≡ √1n (θ and Dn ≡ n(θñ − θ0 ), (13)

Iˆn (θ) ≡ −

1 n

m×m ¨rs (θ|X)

and

˜ n ≡ L(θñ )/L(θ0 ). Λ

Call Iˆn (θñ ) the observed information matrix. Typically, Zn →d Z ∼ = N (0, I(θ0 )).

564


Basic Assumptions (M1) The parameter set Θ in Rm contains an open neighborhood of θ0 in which: (i) For a.e. x the log likelihood (θ|x) is twice continuously differentiable in θ. ... (ii) For a.e. x the third order partials rsγ are continuous in θ and satisfy ...

| rsγ (θ|x)| ≤ (someM (x)) having Eθ0 M (X) < ∞ (for all 1 ≤ r, s, γ ≤ m). (M2) (i) Eθ0 {˙r (θ0 |X)} = 0 for each 1 ≤ r ≤ m. (ii) Eθ0 {[˙r (θ0 |X)]2 } < ∞ for each 1 ≤ r ≤ m. ¯ 0 ) ≡ − Eθ0 {¨rs (θ0 |X)} > 0 (iii) I(θ0 ) ≡ Eθ0 {˙r (θ0 |X) ˙s (θ0 |X)} = I(θ ¯ 0 ) and I(θ0 ) is positive definite). (that is, both I(θ0 ) = I(θ (M3) (i) For a.e. ω and for all n ≥ (some nω ), a solution θñ of the likelihood equations (11) exists having θñ →a.s. θ0

(with respect to Pθ0 );

that is, a strongly consistent solution θñ of (11) exists. (ii) A solution θñ of (11) exists for which θñ →p θ0

(with respect to Pθ0 );

that is, a weakly consistent solution θñ of (11) exists. (M4) The support {x : pθ (x) > 0} does not depend on θ. ˜ n) Theorem 4.1 (Asymptotic distributions of θñ and Λ (A) Suppose that conditions (M0), (M1)(i)(ii) and (M2)(i)(ii)(iii) hold, and that a strongly consistent estimator θñ exists as in (M3)(i). Suppose θ0 is true. Then both Iˆn (θñ ) ≡ − n1 nk=1 ¨rs (θñ |Xk ) =a I(θ0 ), and (14) ˜ I(θn ) =a I(θ0 ) if I(·) is assumed continuous at θ0 ; the same is true for the inverses, since · −1 is a continuous function. Also, √ Dn ≡ n (θñ − θ0 ) =a I −1 (θ0 ) Zn →d D ≡ I −1 (θ0 ) Z ∼ = N (0, I −1 (θ0 )) (15) with Z ∼ = N (0, I(θ0 )), √ √ Wn ≡ [ n (θñ − θ0 )] Iˆn (θñ ) [ n(θñ − θ0 )] =a Dn I(θ0 ) Dn (16) =a Rn ≡ Zn I −1 (θ0 ) Zn →d Z I −1 (θ0 ) Z ∼ = χ2m , (17) (18)

˜ ˜ n = 2 log L(θn ) =a Dn I(θ0 ) Dn →d Z I −1 (θ0 ) Z ∼ 2 log Λ and = χ2m , L(θ0 ) √ √ [ n (θñ −θ0 )] I(θñ ) [ n (θñ −θ0 )] =a Dn I(θ0 ) Dn →d Z I −1 (θ0 ) Z ∼ = χ2m ,

since I(θ) is continuous at θ0 . (B) When θ = θ0 is in reality true (and for Kullback-Leibler information K(Pθ , Pθ0 )) 2 ˜ n →p 2 K(Pθ , Pθ0 ) ≡ 2 Eθ log pθ (X) > 0, with respect to Pθ . (19) log Λ n pθ0 (X)

4.


565

(C) We can replace (M3)(i) by (M3)(ii) in this theorem; just go to subsequences. [By Efron and Hinkley (1978), (16) and (17) are more accurate than (15) and (18).] Exercise 4.1 Show that (M1)(i)(ii), (M2)(i)(ii)(iii), and (M4) imply (M3)(ii). Exercise 4.2 When θ = θ0 in theorem 4.1, we have (in addition to (19)) (20)

1 n

Wn →p (θ − θ0 ) I(θ) (θ − θ0 ) > 0

for Wald’s Wn

(21)

1 n

˙ 0 |X)} I −1 (θ) Eθ {(θ ˙ 0 |X)} Rn →p Eθ {(θ

for Rao’s Rn .

and

Corollary 1 Suppose that q(θ) is also differentiable at θ0 . Then, when θ0 is true and with Q(θ) ≡ ∂θ∂r qi (θ)K×m for q(θ) = (q1 (θ), . . . , qK (θ)) , we have √

n [ q(θñ ) − q(θ0 )] =a Q(θ0 ) Dn =a Q(θ0 ) I −1 (θ0 ) Zn

(22) →d N (0, Q(θ0 ) I −1 (θ0 ) Q (θ0 )). Proof. For a.e. ω and for all n ≥ nω , we use (M1)(i) and (M3)(i) with 0 ≤ t∗ ≤ 1 for the mean value expansion of the continuously differentiable function of a real and obtain variable f (t) ≡ ˙r (θ0 + t(θñ − θ0 )|X), √ = √1 ˙r (θ0 |X) − n m (θñs − θ0s ) [− 1 ¨rs (θ∗ |X)], (a) 0 = √1n ˙r (θñ |X) n s=1 n n for 1 ≤ r ≤ m, where θn∗ = θ0 + t∗ (θñ − θ0 ). Or, 0 = Zn − Iˆn (θn∗ ) Dn . From (M2)(i)(ii) and the multivariate CLT we have n ˙ ∼ = √1 (23) Zn ≡ √1n ˙r (θ0 |X) k=1 r (θ0 |Xk ) →d Z = N (0, I(θ0 )). n From (M2)(ii)(iii) and the SLLN, Iˆn (θ0 ) = − (24)

1 n

=− ¨rs (θ0 |X)

1 n

n

¨

k=1 rs (θ0 |Xk )

→a.s. Eθ0 {−¨rs(θ0 |X)} = Eθ0 {˙r (θ0 |X) ˙s (θ0 |X)} = I(θ0 ).

Use (M1)(ii) with 0 ≤ t∗∗ ≤ 1 for the mean value expansion of the continuously to obtain differentiable function f (t) ≡ ¨rs (θ0 + t(θn∗ − θ0 )|X) + 1 ¨rs (θ0 |X)| = | 1 n [¨rs (θ0 |Xk ) − ¨rs (θ∗ |Xk )]| | − n1 ¨rs (θn∗ |X) n k=1 n n = | n1

n

m

k=1

m

γ=1 (θ0γ

n |θ0γ − θñγ |} × { n1 k=1 M (Xk )}

(b)

≤{

(c)

→a.s. {0} × {Eθ0 M (X)} = 0

γ=1

...

∗ − θnγ ) rsγ (θ0 + t∗∗ (θn∗ − θ0 )|Xk )|

using (M1)(ii)

566


by (M3)(i), and by (M1)(ii) and the SLLN. Combining (24) and (c) (on each of the m × m coordinates) gives (d)

−

1 n

→a.s. I(θ0 ) ¨rs (θn∗ |X)

[that is, Iˆn (θn∗ ) →a.s. I(θ0 )] ;

and since · −1 is a continuous function of its argument, for all n ≥ (some nω ) the −1 exists and satisfies inverse − n1 ¨rs (θn∗ |X) (e)

−

1 n

−1 →a.s. I −1 (θ0 ) ¨rs (θn∗ |X)

[that is, Iˆn−1 (θn∗ ) →a.s. I −1 (θ0 )] .

We can now apply (e) and (23) to (a) (for all n ≥ (some nω )) to conclude that √ −1 Zn =a I −1 (θ0 ) Zn →d I −1 (θ0 ) Z, (f) n (θñ − θ0 ) = − n1 ¨rs (θn∗ |X) which gives (15). One permissible choice for θn∗ in (d) and (e) is θñ ; thus the conclusions (d) and (e) give (g)

Iˆn (θñ ) →a.s. I(θ0 )

Iˆn−1 (θñ ) →a.s. I −1 (θ0 ) ,

and

as claimed in (14). Thus (f) and (g) give (16). And (18) follows likewise, since I(θñ ) →a.s. I(θ0 ) in that case using the continuity of I(·) in (14). Of course, the final claim of (14) is trivial. We now turn to (17). For a.e. ω and for n ≥ nω we now Taylor expand as = (θñ |X) + m (θ0r − θ˜0r ) ˙r (θñ |X) (θ0 |X) (h) r=1 + 12

m m r=1

˜

s=1 (θnr

− θ0r )(θñs − θ0s ) ¨rs (θn∗ |X)

for some θn∗ = θñ + t∗ (θ0 − θñ ) with 0 ≤ t∗ ≤ 1. Thus ˜ n = 2 [(θñ |X) − (θ0 |X)] 2 log Λ = (i)

n

r=1 0

+2×

1 2

√

n (θñ − θ0 ) × −

1 n

× ¨rs (θn∗ |X)

√

= Dn Iˆn (θn∗ ) Dn

for Dn and Iˆn (·) as in (12) and (13)

=a Dn I(θ0 ) Dn

using (d)

→d D I(θ0 ) D

where D ∼ = N (0, I −1 (θ0 )) by (15)

∼ = χ2m .

(j)

Finally, we consider (19). When θ = θ0 is really true, we have 2 n

(k)

n (θñ − θ0 )

ñ = log Λ =

2 n

=

2 n

2 n

− (θ0 |X)] [(θñ |X)

− (θ0 |X)] + [(θ|X) n k=1

k) log ppθθ (X (Xk ) + 0

2 n 2 n

− (θ|X)] [(θñ |X) − (θ|X)] . [(θñ |X)

4.


567

The first term in (k) converges a.s. Pθ to 2 K(Pθ , Pθ0 ) by the SLLN. The second − (θ|X)] =a 1 χ2 →p 0 using (j); the hypotheses still term in (k) has n2 [(θñ |X) n m must hold in a neighborhood of the truth, which is now θ. Thus (19) holds. Corollary 1 follows immediately by the delta method, since q(·) is differentiable. We now turn to corollary 2 below. As in (h), for t ≡ (t1 , . . . , tn ) we have √ ˙ 0 |X) − (θ0 |X) = √t (θ + 1 √t ¨rs (θ0 + √t∗ |X) √t (l) (θ0 + t/ n |X) 2

n

= t Zn −

1 2t

(m)

=a t Zn −

(n)

→d t Z −

Iˆn (θ0 +

1 2 t 1 2 t

n

n

n

t∗ √ )t n

ˆ 0) t I(θ

I(θ0 ) t ∼ = N (− 21 σ02 , σ02 )

with σ02 ≡ t I(θ0 ) t .

2

Corollary 2 Suppose (M0),(M1)(i)(ii) and (M2)(i)(ii)(iii) hold. Then √ − (θ0 |X) =a t Zn − 1 t I(θ0 ) t →d N (− 1 σ02 , σ02 ), (25) (θ0 + t/ n |X) 2

2

with = t I(θ0 ) t. (These local alternatives are said to be contiguous to the null hypothesis θ0 .) σ02

Notation 4.2 If the likelihood equations are difficult to solve or have multiple roots, then a one-step approximation is possible. We thus suppose that θ¯n is a preliminary estimator of θ that satisfies n1/4 (θ¯n − θ0 ) →p 0 √ (and this does hold if n (θ¯n − θ0 ) = Op (1)). We now iterate once to obtain n (27) θˇn ≡ θ¯n + √1n Iˆn−1 (θ¯n ) Zn (θ¯n ) where Zn (θ) ≡ √1n k=1 ˙r (θ|Xk ) .

(26)

(Newton-Raphson suggests 0 = 2

√1 n

. ˙ θñ |X) = (

√1 n

˙ θ¯n |X) + √n (θñ − θ¯n ) Iˆn (θ¯n ).) (

Theorem 4.2 (One-step approximation) Suppose (M1)(i)(ii), (M2)(i)(ii)(iii) and condition (26) hold. Then (28)

θˇn may replace θñ throughout theorem 4.1 and corollary 1.

(Note that one can (in fact) iterate any fixed finite number of times.) Proposition 4.1 (Kullback–Leibler information number) Let P and Q denote a probability measure and a subprobability measure (that is, Q(X ) ≤ 1) on (X , A) having densities p(·) and q(·) with respect to a σ-finite measure µ on (X , A). Then p(X) a number in [0, ∞] always, (29) K(P, Q) ≡ EP log = 0 if and only if P=Q . q(X) Exercise 4.3 Prove the previous one-step approximation theorem. Exercise 4.4 Prove the previous Kullback-Leibler proposition.

568

References I would like to use this discussion of the literature to say a very heartfelt “Thank you!” to a number of people who have figured prominently in my professional life. Especially, I want to thank my professors Fred Andrews (University of Oregon) and Lincoln Moses (Stanford University), whose volunteered efforts on my behalf had far-reaching consequences on most aspects of my life. I shall offer some thoughts on my own personal history as well as the subject matter of this book. My view is strongly affected by how I came to learn about these things. Others have undoubtedly had different experiences. Measure theory This text begins with five chapters devoted to measure theory. Halmos (1950) has had a major influence on what future books on measure theory and real analysis would contain and how they would present the subject. Other books on measure theory and real analysis that I have found to be especially useful include Royden (1963), Hewitt and Stromberg (1965), Rudin (1966), and the nicely simplified presentation of Bartle (1966). Many theorems in this introductory part are to some degree recognizable from several of these sources (and/or from the other sources listed in the probability section below). Certainly, Halmos’s book was a popular one while I was getting my M.S. degree in mathematics at the University of Oregon, 1960–1962. My own introduction to “real and abstract analysis” came from a beautiful course taught by Karl Stromberg. Later, Edwin Hewitt was a colleague at the University of Washington. So it is a real pleasure for me to cite their work at various points. Lou Ward taught the topology course that I took at Oregon. He gave us a list of theorems, and we had to come up with proofs and present them. That was the most fun I ever had in the classroom. A good deal of chapter 6 reflects what I learned in his course. Kelly (1955), Copson (1968), and Housain (1977) are useful published sources. Watching over the Oregon graduate students in mathematics was Andrew Moursand, chairman. He really cared about all of us, and I owe him my thanks. Probability Loéve’s (1977-78, originally 1955) presentation has been a very influential work on probability, certainly from the pedagogical point of view. To me, it refines and specializes much general analysis to probability theory, and then treats a broad part of this subject. Clearly, many learned probability from his text. Also, many seem to follow notational conventions used in his book. But I was rather late in learning from it. My original training was at Stanford from lectures that became Chung (1974), and those lectures also reflected Chung’s efforts regarding translation of Gnedenko and Kolmogorov (1954). I truly enjoyed Chung’s course, and his book. Breiman’s (1968) style coincided most closely with my own. I particularly liked his treatment of partial sum and empirical processes, as one would suspect from my own research. I have sometimes used his text as a “permanent reference” to stand beside my own notes in my courses on probability theory. My choice of notation has been most influenced by Loéve and Breiman. Feller (1966) has a different flavor from most probability texts, and it includes various interesting approaches not found

REFERENCES

569

elsewhere. And it is informative on rates of approximation. Billingsley (1968) created some excitement and spawned much interesting work, and a bit of that is included here. Doob’s (1954) work on martingales has had a huge influence on the subject. I find Meyer (1966) and Hall and Heyde (1980) particularly significant. Lectures by Tom Fleming that led to Fleming and Harrington (1991) sparked part of my martingale presentation here. Whittaker and Watson (1963) is still a superb source for the gamma function. Lehmann (1959) has greatly influenced my view of conditional probability and expectation. This brings me back to the University of Oregon, and to Fred Andrews. Fred “recruited me to statistics” and then taught a year-long course out of Lehmann’s book (even though I was the only student), and he was one of those who lined me up for a National Science Foundation fellowship that made it possible for me to go to Stanford University. Don Truax also figured heavily in this. He cared about me, and I learned a lot from him. Thank you both! The scene shifts southward. My years at Stanford were very fruitful, and I met some fine people. Ingram Olkin is fun, a good teacher, and he went out of his way to be helpful to me. The multivariate topics in chapter 9 represent what I learned from him. Lincoln Moses was my thesis advisor. This relationship grew out of a course in nonparametric statistics that I took from him. One of the topics in his course was Charles Stein’s approach to the central limit theorem. Lin spoke on it for three days, even though he had to leave a couple of well-acknowledged gaps in his presentation—because he believed it was good work. That gave me a profound respect for him as a teacher. The topic caught my imagination, and chapters 11 and 17 reflect this. Lin was also my assigned advisor when I arrived at Stanford. His second sentence to me was, “OK, Shorack, what’s important to you in life?” My answer had a lot to do with the geography of the Pacific Northwest. Two months before I graduated he responded on my behalf to a University of Washington opening. Wow! At Washington I had a chance to teach courses in probability and statistics. And I learned a lot from my association with Ron Pyke, and later with Jon Wellner. The presentations in parts of chapters 12, 16, and 19 reflect this to varying degrees. Fritz Scholz got me started on gamma approximations in the central limit theorem. Likewise, work with David Mason on quantile functions, embedding, and trimmed means is reflected in parts of chapters 7, 10, and 16. I offer them all my thanks. Obviously, I also owe a huge debt to “the literature” in regard to all these topics, and I will list some of those sources below. However, this is a textbook. It is not a research monograph. My emphasis is on presentation, not attribution. Often, my citation concerns where I learned something rather than who did it originally. And in some areas (especially, chapters 7, 12, and 16) I have offered only a sampling of the citations that I could have given. Moreover, I have often chosen to cite a book instead of an original paper. My own work is cited “too heavily” because it is written in the same style and notation as this book. The bibliography contains quite a number of other books on probability theory, and many are very good books. But it is the ones listed above that have had the most influence on me. I hope that the reader will find that my book also has a somewhat different flavor—a statistical flavor.

570


Special thanks to Chari Boehnke, Roger and Heather Shorack, the Michael Boehnke family, the Barbara Aalund family, Kathleen Shorack, David Mason, Michael Perlman, Fritz and Roberta Scholz, Jon Wellner, the Jan Beirlant family, Piet Groeneboom, Frits Ruymgaart, Derek Dohn, and Pauline Reed for enabling me to write this book. CHAPTERS 1-6 Ash, R. (1972) Real Analysis and Probability. Academic Press, New York. Bartle, R. (1966) The Elements of Integration. John Wiley & Sons, New York. Breiman, L. (1968) Probability. Addision-Wellsey, Reading, MA. Chow, Y. and H. Teicher (1997) Probability Theory: Independence, Interchangeability, Martingales 3rd ed. Springer-Verlag, New York. Cohn, D. (1980) Measure Theory. Birkh¨ auser, Boston. Copson, E. (1968) Metric Spaces. Cambridge University Press, Cambridge. Halmos, P. (1950) Measure Theory. Van Nostrand, Princeton, NJ. Hewitt, E. and K. Stromberg (1965) Real and Abstract Analysis. Springer-Verlag, New York. Husain, T. (1977) Topology and Maps. Plenum Press, New York. Lo` eve, M. (1977–78) Probability Theory. Springer-Verlag, New York. Meyer, P. (1966) Probability and Potentials. Blaisdell, Waltham, MA. Royden, H. (1967) Real Analysis. Macmillan, New York. Rudin, W. (1966) Real and Complex Analysis. McGraw-Hill, New York.

CHAPTER 8 Breiman, L. (1968) Probability. Addision-Wellsey, Reading, MA. Breiman, L. and J. Friedman (1985) Estimating optimal transformations for multiple regression and correlation. J. Amer. Statist. Assoc. 80 580-619. Lehmann, E. (1959) Testing Statistical Hypotheses. John Wiley & Sons, New York.

CHAPTER 10 Breiman, L. (1968) Probability. Addision-Wellsey, Reading, MA. Chow, Y. and H. Teicher (1997) Probability Theory. Independence, Interchangeability, Martingales 3rd ed. Springer-Verlag, New York. Chung, K. (1974) A Course in Probability Theory 2nd ed. Academic Press, New York. Feller, W. (1966) An Introduction to Probability Theory and Its Applications. Vol. 2. John Wiley & Sons, New York. Gin´ e, E., F. G´ otze, and D. Mason (1998) When is the student t-statistic asymptotically standard normal? Ann. Probab. 25 1514-1531. Gnedenko, B. and A. Kolmogorov (1954) Limit Distributions for Sums of Independent Random Variables. Addison-Wesley, Cambridge, Mass. Hoffman–Jorgensen, J. (1974) Sums of independent Banach space valued random variables. Studia Math. 52 159-186.

REFERENCES

571

Kallenberg, O. (1997) Foundations of Modern Probability. Springer-Verlag, New York. Lo` eve, M. (1977-78) Probability Theory. Springer-Verlag, New York. Resnick, S. (1999) A Probability Path. Birkh¨ auser, Boston. Strassen, V. (1967). Almost sure behavior of sums of independent random variables and martingales. Proceeding of the Fifth Berkeley Symposium on Mathematical Statistics and Probability 2 315-343, University of California Press, Berkeley CA.

CHAPTER 11 Anderson, T. (1984) Introduction to Multivariate Statistical Analysis. John Wiley & Sons., New york. Billingsley, P. (1968) Convergence of Probability Measures. John Wiley & Sons, New York. Cs¨ org˝ o, S. and D. Mason (1989) Bootstrapping empirical functions. Ann. Statist. 17 1447–1471. Gin´ e, E. and J. Zinn (1990) Necessary conditions for the bootstrap of the mean. Ann. Statist. 17 684–691. Hall, P. (1990) Asymptotic properties of the bootstrap for heavy tailed distributions. Ann. Statist. 18 1342-1360. Lehmann, E. and G. Casella (1998) Theory of Point Estimation 3rd ed. SpringerVerlag, New York. Lo` eve, M. (1977-78) Probability Theory. Springer-Verlag, New York. Reinhert, G. (1977) Couplings for normal approximations with Stein’s method. To appear in Conference Proceedings, Microsurveys in Discrete Probability. Shorack, G. (1998) On Stein’s appraoch to the central limit theorem. Univ. Washington Dept. of Statistics Technical Report 339. Stein, C. (1986) Approximate Computation of Expectations. IMS, Hayward, CA.

CHAPTERS 7, 12, and 16 Billingsley, P. (1968) Convergence of Probability Measures. John Wiley & Sons, New York. Breiman, L. (1968) Probability. Addision-Wellsey, Reading, MA. Chung, K. (1974) A Course in Probability Theory 2nd ed. Academic Press, New York. Cram´ er, H. and M. Leadbetter (1967) Stationary and Related Stochastic Processes. John Wiley & Sons, New York. Cs¨ org˝ o, M., S. Cs¨ org˝ o, L. Horv´ ath and D. Mason (1986) What portion of the sample makes a partial sum asymptotically stable or normal? Probab. Theory Related Fields 72 1–16. Cs¨ org˝ o, M., S. Cs¨ org˝ o, L. Horv´ ath and D. Mason (1986) Weighted empirical and quantile processes. Ann. Probab. 14 31–85. Cs¨ org˝ o, S., E. Haeusler, and D. Mason (1988) The asymptotic distribution of trimmed sums. Ann. Probab. 16 672-699.

572


Cs¨ org˝ o, S., E. Haeusler, and D. Mason (1989) A probabilistic approach to the asymptotic distribution of sums of independent identically distributed random variables. Adv. in Appl. Math. 9 259-333. Doob, J. (1949) Heuristic approach to the Kolmogorov–Smirnov theorems. Ann. Math. Statist. 20 393-403. Freedman, D. (1971) Brownian Motion and Diffusion. Holden-Day, San Francisco. Haeusler, E. and D. Mason (1989) A law of the iterated logarithm for modulus trimming. Colloquia Math. Soc. J. Bolyai, 57 Limit Theorems in Probability and Statistics. Mason, D. (1999) An exponential inequality for weighted approximation to the uniform empirical process with applications. University of Deleware Technical Report. Mason, D. and G. Shorack (1990) Necessary and sufficient conditions for asymptotic normality of trimmed L-statistics. J. Statist. Plan. Inf. 25 111-139. Meyer, P. (1966) Probability and Potentials. Blaisdell, Waltham, MA. Ross, S. (1997) An Introduction to Probability Models, 6th ed., Academic Press, San Diego. Shorack, G. (1998) The asymptotic normality condition. Univ. Washington Dept. of Statistics Technical Reports 323, 336, 337, and 338. Shorack, G. (1991) Embedding the finite sampling process at a rate. Ann. Probability 19 826-842. Shorack, G. (1996) Linear rank statistics, finite sampling, permutation tests and Winsorizing. Ann. Statist. 24 1371-1385. Shorack, G. (1997a) Inequalities for quantile functions with a uniform studentized CLT that includes trimming. Nonparametric Statist. 8 307-335. Shorack, G. (1997b) Uniform CLT, WLLN, LIL and bootstrapping in a data analytic approach to trimmed L-statistics. J. Statist. Plan. Inf. 60 1–44. Shorack, G. (1998) Applications of weighted approximations via quantile inequalities. Asymptotic Methods in Probability and Statistics, A Volume in Honor of Mikl´ os Cs¨ org˝ o. B. Szyszkowicz, Ed., Elsevier, Amsterdam. Shorack, G. and J. Wellner (1986) Empirical Processes with Applications to Statistics. John Wiley & Sons, New York. CHAPTERS 13-15 Alfors, L. (1979) Complex Analysis McGraw-Hill, Inc., New York. Bickel, P. and K. Doksum (1977) Mathematical Statistics: Basic Ideas and Special Topics. Holden-Day, San Francisco. Bhattacharya, R. and R. Rao (1976) Normal Approximation and Asymptotic Expansions. John Wiley & Sons., New York. Billingsley, P. (1986) Probability and Measure. John Wiley & Sons, New York. Breiman, L. (1968) Probability. Addision-Wellsey, Reading, MA. Durrett, R. (1996). Probability: Theory and Examples. Wadsworth, Belmont, CA. Feller, W. (1966) An Introduction to Probability Theory and Its Applications. Vol. 2. John Wiley & Sons, New York.

REFERENCES

573

Galambos, J. (1995) Advanced Probability Theory 2nd ed. Marcel Dekker, New York. Gnedenko, B. and A. Kolmogorov (1954) Limit Distributions for Sums of Independent Random Variables. Addison-Wesley, Cambridge, Mass. Hall, P. (1983) Chi squared approximations to the distribution of a sum of independent random variables. Ann. Statist. 11 1028-1036. Kendall, M. and A. Stuart (1977) The Advanced Theory of Statistics Vol. 2. Macmillan, New York. L´ evy, P. (1937) Théorie de l’Addition des Variables Aléatoires. Bautier-Villars, Paris. Lo` eve, M. (1977-78) Probability Theory. Springer-Verlag, New York. Petrov, V. (1975) Sums of Independent Random Variables. Springer-Verlag, New York. Shiryayev, A. (1984) Probability. Springer-Verlag, New York.

CHAPTER 17 Barbor, A. and P. Hall (1984) Stein’s method and the Berry–Esseen theorem. Austral. J. Statist. 26 8-15. Bolthausen, E. (1984) An estimate of the remainder in a combinatorial central limit theorem. Z. Wahrsch. Verw. Geb. 66 379-386. ˇ H´ ajek, J. and V. Sidak (1967) Theory of Rank Tests. Academic Press, New York. Hoeffding, W. (1948) A class of statistics with asymptotically normal distribution. Ann. Math. Statist. 19 293–325. Hoeffding, W. (1951) A combinatorial central limit theorem. Ann. Math. Statist. 22 558-566. Reinhert, G. (1977) Couplings for normal approximations with Stein’s method. To appear in Conference Proceedings, Microsurveys in Discrete Probability. Serfling, R. (1980) Approximation Theorems of Mathematical Statistics. John Wiley & Sons, New York. Stein, C. (1986) Approximate Computation of Expectations. IMS, Hayward, CA.

CHAPTER 18 Andersen, P., O. Borgan, R. Gill and N. Keiding (1993) Statistical Models Based on Counting Processes. Springer-Verlag, New York. Breiman, L. (1968) Probability. Addision–Wellsey, Reading, MA. Doob, J. (1953) Stochastic Processes. John Wiley & Sons, New York. Fleming, T. and D. Harrington (1991) Counting Processes and Survival Analysis. John Wiley & Sons, New York. Hall, P. and C. Heyde (1980) Martingale Limit Theory and Its Application. Academic Press, New York. Liptser, R. and A. Shiryaev (1977) Statistics of Random Processes I. SpringerVerlag, New York. Meyer, P. (1966) Probability and Potentials. Blaisdell, Waltham, MA. Williams, D. (1991) Probability with Martingales. Cambridge University Press, Cambridge.

574


Chapter 19 Billingsley, P. (1968) Convergence of Probability Measures. John Wiley & Sons, New York. Dudley, R. (1976) Convergence of laws on metric spaces, with a view to statistical testing. Matematisk Institut Aarhus Universitet. Dudley, R. (1989) Real Analysis and Probability. Wadsworth & Brooks/Cole, Pacific Grove, CA. Skorokhod, A. (1956) Limit theorems for stochastic processes. Theor. Probab. Appls. 1 261-290.

APPENDIX A Abramowitz, M. and I. Stegun (1964) Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables. National Bureau of Standards Applied Mathematics Series, 55 U.S. Government Printing Office, Washington, DC. Efron, B., and D. Hinkley (1978) Assessing the accuracy of the maximum likelihood estimator: Observed versus expected Fisher information. Biometrika 65 457-482. Johnson, N., A. Kotz, and N. Balakrishnan (1994) Continuous Univariate Distributions Volumes 1 and 2. John Wiley & Sons, New York. Rao, C. (1965) Linear Statistical Inference and Its Applications 2nd ed. John Wiley & Sons, New York. Whittaker, E. and G. Watson (1963) A Course of Modern Analysis. Cambridge University Press, Cambridge.

Index Brownian motion, 202, 235, 302, 311, 318, 469, 500, 536 integrals of, 303 LIL, 303, 320 path properties, 323 reflection principle, 314, 315, 321, 322, 331, 409 transformations, 303, 318

a.b.f., 8 absolute continuity, 255, 258 fundamental theorem of calculus, 76, 78, 137 of functions, 74, 76, 78, 323 of log f , 78, 138 of measures, 60, 63, 64, 66–68, 78, 108 of the integral, 42, 54, 66, 76, 480 Radon–Nikodym, 66 adapted, 35, 36, 305 added touch, 124, 268 ancillary statistic, 173 approximation by continuous functions, 59, 60 by elementary functions, 26 by simple functions, 26 by step functions, 59, 78 ¯ n )), 396 approximation for L(h(X approximation lemma, for sets, 16, 80, 155, 156 associated rvs, 264

Cantor set, 108, 114 Carathéodory covering, 12, 17, 77 extension theorem, 5, 10, 12, 80, 88, 480 censored data, 523, 527 chain rule, 67, 78 change of variable, 78, 113, 134 Radon–Nikodym, 66–68 unconscious statistician, 42, 67, 113 chf, 242, 294, 341, 351 complementary pair, 346 continuity theorem, 342, 350, 351, 366, 381 distributions on grids, 361 empirical, 348 Esseen’s lemma, 358, 371, 373, 385, 391, 394 examples, 343–345, 349 expansion for uan array, 400 Fourier, Laplace, etc., 341, 356 inversion formula, 346–348, 359, 362, 380, 381, 385, 386, 393 moment expansions, 352–354, 371, 374 multivariate normal, 200 tail behavior, 355

best linear predictor, 194 bootstrap, 274, 277, 388, 432 Bayesian, 434 Borel sets, 5, 6, 17, 18, 23 ¯ 18 B,Bˆµ,B, generators of B, 24 generators of Bn , 23, 35, 80 generators of B∞ , 36, 86 countable base BC of BT , 91 Borel–Cantelli, 204, 217, 218, 236, 237, 244, 321, 376 bounded variation, 74–76 total variation measure, 62 branching processes, 484 Brownian bridge, 202, 302, 326 575

576 uniqueness theorem, 342, 346, 347, 350, 351, 366 chisquare tests, 369, 379, 389 CLT, 125, 126, 147 asymptotic normality condition, 265, 406, 529 basic dependent, 529 Berry–Esseen, 259, 261, 263, 267, 270, 276, 278, 371, 376, 450 Berry–Esseen for U-statistics, 454 bootstrap, 274, 434 classical, 183, 238, 342, 366 connection with variance estimation, 125, 233, 270, 271 creating normality, 428 delta method, 279 Doeblin, 285, 370 Edgeworth (see also), 391 examples, 279, 280, 282, 283, 286, 287, 368, 369 finite sampling, 388, 430, 464, 465 for general iid uan arrays, 406 Fréchet–Shohat via moments, 293 gamma approximation, 383, 388, 389, 459 Hoeffding’s combinatorial, 458 L-statistics, 441, 442 Liapunov, 371, 377 Lindeberg–Feller, 260, 266, 267, 373, 404, 529 local limit (see also), 380 martingale, 530 multivariate, 201, 368 negligibility, 215, 233, 234, 367 Poisson approximation, 386 R-statistics, 427 random sample size, 285 Rebolledo, 503, 505 slowly varying σ ˜ 2 (·), 233, 268, 270– 272 Stein’s approach, 255, 262, 263, 459 Studentized CLT for σ ˜2 (·) ∈ L, 270, 272 trimmed means, 416, 417, 420 U-statistics, 450, 453

PROBABILITY FOR STATISTICIANS universal bootstrap CLT, 277 universal studentized CLT, 276 Winsorized, 264, 270 compensator, 487, 488, 501, 511 complete measure, 15, 18, 29 space Lr , 51 completeness assumption, 469 conditional expectation, 158, 159, 168, 171, 246, 248 properties of, 163 conditional probability, 159, 160, 186, 201 regular, 10, 168–170 contiguity, 567 convergence a.e., 29–31, 51 a.s., 33, 78, 89, 114, 241 almost uniform, 60 Ces` aro summability, 205, 217 DCT, 41 fd convergence →f d , 91 in distribution, 33, 52, 89, 112, 114, 241, 288, 290, 291, 293, 342, 531 in L2 , 78 in measure, 30, 31, 51 in probability, 33, 51, 89, 241 in quantile, 112 in rth mean, 33, 51, 89 in sub-df, 288 MCT, 40 modes, 57 mutual, 29–31, 51 set, 29, 217, 242, 252 uniform, 60 convex, 46, 48, 49, 56, 468 convolution formula, 187, 347–349 correlation coefficient, 286, 463 multiple, 194 Spearman, 462 counting process, 470, 501 coupling, 262, 389, 542 Cramér Cramér–Lévy continuity theorem, 350

INDEX Cramér–von Mises, 304, 331, 338, 339 Cramér–Wold device, 351, 368, 378 cumulant gf, 356, 371, 384, 387, 388, 392 cumulative hazard function, 470, 506, 522, 527 decomposition of Normals, 405 decomposition of Poissons, 405 dense Bernoulli polynomials, 223 continuous functions, 59, 78 in Hilbert space, 106 step functions, 59, 78, 482 density estimation, 348 derivative differentiable, 70, 75, 76 Dini derivates, 70 Lebesgue theorem, 70, 71, 78 Radon–Nikodym, 66–68, 74, 76– 78, 158 series, 71 Taylor’s theorem, 72, 352 under integral sign, 45, 317 determining class, 347 (k) (∞) C,C0 ,Cb ,Cbu ,Cb ,Cb , 292 chf, 294 G,G0 ,G1 ,G2 , 292 limit, 53 moments, 293, 294, 354 df, 20, 33, 107 absolutely continuous, 108 decomposition of, 107 generalized, 18, 19, 27 joint, 84, 85, 154 jumps and flat spots, 110, 134 Lebesgue singular, 71, 108 properties, 113 singular, 108 sub, 33, 107, 288 support, 110 diagonalization, 289 discontinuity set, 26 distribution Bernoulli, 179

577 Beta, 184, 549 Binomial, 179, 555 Cauchy, 184, 187, 242, 284, 342– 344, 412 Chisquare, 183 Compound Poisson, 402 de la Vallée Poussin, 343, 346, 348, 363 Discrete Uniform, 555 Double Exponential, 184, 283, 345, 558 Exponential, 181, 287 Extreme Value, 286, 559 Gamma, 182, 383, 392, 406, 556 Geometric, 179 hitting time, 409 Hypergeometric, 180, 555 Log Gamma, 559 Logistic, 184, 345, 558 Multinomial, 185, 369, 402 Multivariate Normal, 199, 286 NegBiT, 179, 387, 555 Noncentral, 378, 379, 389 Normal, 183, 287, 405, 549, 556 Pareto, 562 Poisson, 181, 284, 287, 386, 405, 555 sampling without replacement, 180 Snedecor’s Fm,n , 188, 557 stable laws (see also), 407 Student’s tm , 188, 557 Triangular, 343, 346, 348 Uniform, 184, 558 Weibull, 561 divergence set, 29 domain of attraction, 145 D(Normal), 267, 268, 271, 272, 277 DN (Normal), 413 D(G) of the stable law G, 407, 412 DN (G) of the stable law G, 407, 412 ˜ 145, 417, 442 statistical D, total D, 145 Edgeworth expansions, 386, 391, 392

578 eigenvalues, 191, 196, 340 embedding Cs¨ org˝ o, Cs¨ org˝ o, Horv´ ath, Mason, 328 Shorack, 328 Skorokhod, 235, 312, 318, 329 empirical chf, 348 df Fn , 121, 223, 325 df Gn , 121, 223, 325, 417 process En , 121 process Un , 121, 325, 368, 417 qf Kn , 120 quantile process Vn , 325 two-sample process Wm,n , 331 weighted process Wn , 326 finite sampling process Rn , 326 equivalent summands, 267 Euclidean space, 23 Euler’s constant, 186, 546, 550 expectation, 38 of products, 152 extremes, 370 filtration, 469, 489, 493, 511 finite-dimensional convergence →f d , 91 distributions, 90 projection mappings πt1 ,...,tk , 90 rectangle, 86, 89 σ-field, 90 subsets, 90 function elementary, 24–26 indicator, 3, 24 measurable, 24 ψ-function, 59 simple, 24–26, 28, 37, 166 step, 59 gambler’s ruin, 499, 500 gamma approximation; the GLT, 383 gamma function, 546 digamma, 546 generalized inverse, 193 Σ− and Σ−1/2 , 192, 379 generators, 4

PROBABILITY FOR STATISTICIANS ¯ of the induced F(X) ≡ X −1 (B), 24 of the induced F(X1 , . . . , Xn ), 35 of the induced F(X1 , X2 , . . .), 35 regarding X −1 , 22 regarding independence, 9 σ-field, 3, 10, 24 various for Borel sets B, 24 various for Borel sets Bn , 23, 35, 80 various for Borel sets B∞ , 36, 86 Glivenko–Cantelli, 223, 325, 326 Hermite polynomials, 317, 390, 391, 470 Hewitt–Savage zero-one law, 156 Hilbert space, 104, 174 2 -isomorphism, 106 alternating projections (ACE), 175 Fourier coefficients, 105, 106, 340 Gram–Schmidt, 106 idempotent operator, 174 inequalities, 104, 106 L2 is a Hilbert space, 104 orthogonal projections, 105, 174 orthonormal basis, 105, 106 projection operator, 176 self-adjoint operator, 174 subspaces, 105 hitting time, 311, 314, 409 Hodges–Lehmann estimator, 455, 456 i.o., 8, 204 increment function, 18 independence tests, 369 independent, 163 rvs, 151, 153, 154, 157, 189, 200, 201, 351 σ-fields, 151, 153 indicator function proof, 28, 38, 42, 66, 81, 83, 164, 166 induced distribution, 23, 24, 27, 33, 42, 52, 92 distribution consistency, 87 inequality basic, 49

INDEX Birnbaum–Marshall, 249, 333 Bonferroni, 50 bounds on (1 − x/n)n , 354, 550 Cantelli, 221 Cauchy–Schwarz, 48, 197 Chang, 336 Chebyshev, 49, 218 chf bound on tails, 350 chf key inequality, 372 convexity, 48 correlation, 48 Cr -inequality, 47, 163, 209, 313, 372 Daniels’ equality, 214, 230, 335 dispersion, 167, 209 Doob, 247, 249, 335 Etemadi, 212 geometric mean, 50 Giné–Zinn symmetrization, 213, 273 H´ ajek–Rényi, 221, 248, 333, 452, 453 Hardy, 50 Hilbert space inequalities, 104, 106 Hoeffding–Fréchet, 85 Hoffman–Jorgensen, 250 H¨ older, 47, 48, 163, 209 Jensen, 49, 163, 166, 167, 209, 248, 468 Khinchin, 213, 273 Kolmogorov, 210, 217, 218, 247, 248, 253, 285, 321 Kolmogorov’s other, 243 Lévy, 211, 220, 235 Liapunov, 48, 163, 209, 371 Littlewood, 48 Markov, 49, 209 Mills’ ratio, 235, 237, 321, 322 Minkowski, 49, 53, 163 moment expansions of chfs, 352, 354, 371, 374 monotone, 205, 221, 247, 248, 321 Ottavani–Skorokhod, 212, 242 Paley–Zygmund, 49, 273, 324 Pyke–Shorack, 328, 334, 336

579 sandwiching the mean, 206, 207, 215, 217, 218 Shorack, 141, 143, 147, 214, 256, 458 Shorack–Smythe, 247 symmetrization, 210, 219 truncation, 219, 227–229, 232 upcrossing, 473 Wellner, 50 Winsorized variance, 141, 143, 147 Young, 47 infinitely divisible, 400 limits of, 400 log chf is never zero, 400 subclass I2 , 400 information, 35 integrable, 38 collection, 54 L1 , 37 L2 , 78 Lr , 37, 47 L+ r , 37 product, 82 uniformly, 54–56, 221, 252, 473, 477, 478, 488, 492, 511, 516 integral, 38 improper, 44 Lebesgue, 37 Lebesgue–Stieltjes, 44 linearity, 38, 163 Riemann, 1, 75 Riemann–Stieltjes, 44 integration by parts formulas, 115, 118 inverse image, 21, 22 of σ-fields, 22 inverse transformation, 111 Jacobian, 78, 190 Khinchin equivalent rvs, 206, 207, 215, 244 Kolmogorov consistency theorem, 92, 302 extension theorem, 87 Gnedenko–Kolmogorov theorem, 123, 232 inequality (see also), 210

580 Kolmogorov–Smirnov, 316, 331 representation theorem for I, 402 SLLN (see also), 215, 275 zero-one law, 155, 204, 482 Kullback-Leibler information, 564, 567 large deviations, 395 Lebesgue decomposition theorem, 64, 66, 108 integral, 37 Lebesgue–Stieltjes measure, 4, 18, 20, 77, 78, 80 measure, 6, 23, 38, 67 measure λn , 80 sets, 15, 17 singular df, 71, 108 sums, 1 theorem re derivatives, 70, 71, 78 likelihood ratios, 470 LIL, 235, 238, 321, 323 see also Brownian motion, 303 U-statistics, 452 limit determining class, 292 limit theorem general uan terms, 405 uan terms with negligible variances, 403 Lindeberg’s LFn , 260, 266, 267, 278, 283, 373, 375, 376, 436 linear algebra, 191 Lipschitz condition, 74, 540 LLN, 126, 148, 218, 222, 224–226 Glivenko–Cantelli (see also), 223 negligibility, 215, 226 random sample size, 220 SLLN of Kolmogorov, 208, 215, 220, 275, 481 strong, 221, 224, 238, 245, 325 U-statistics, 452, 453, 481 uniform SLLN of Chung, 252 weak, 208, 224, 226, 228, 245, 366, 377, 406, 425 WLLN of Feller, 208, 215, 220 WLLN of Khinchin, 208 local limit theorem, 380

PROBABILITY FOR STATISTICIANS Mann–Whitney statistic, 456 martingale, 246, 311, 317, 467 closes, 473, 477, 478, 493 CLT, 530 convergence theorem, 473, 477, 478 counting process, 470 decomposition, 487, 488, 502, 511, 512 equivalence, 246, 467 examples re empiricals, 333 exponential, 470, 499, 500 integrable, 468 Kakutani, 470, 482 local, 511 > = notation, 246 optional sampling theorem, 312, 472, 492, 493 reversed, 335, 453, 478, 481, 498 s-mg, 246, 467 square-integrable, 468 sub mg, 246, 247, 467 transform, 489, 490, 502, 513, 516 U-statistic, 452 Wald, 469 Mason theorem, 329, 332, 336 maximum likelihood, 551, 563 measurability criterion, 24, 25 measurable, 24 as a limit, 25, 29 A -A-measurable, 24 Bn -measurable, 84 B∞ -measurable, 86 common functions are, 25 C-measurable, 298 D-measurable, 301, 320 Aτ -measurable, 306 F(X)-measurable, 24, 28, 86, 158 F(X1 , . . . , Xn )-measurable, 35 F(X1 , X2 , . . .)-measurable, 35, 86, 91 Ft -measurable, 36 F(Xs : s ≤ t)-measurable, 36, 91 ˜ F(S)-measurable, 239 measurability criterion, 28 non, 16, 92

INDEX partition, 37 progressively, 306 set, 26, 29 σ[C]-measurable, 24, 28 space, 3 measurable function spaces, 90 (C[0,∞) , C[0,∞) ), 297 (M, d) with Md or MB d , 295 (C, C), 90, 295, 298 (D, D), 295, 298, 469 general space (MT , MT ), 90 (R[0,1] , B[0,1] ), 90 (RT , BT ), 91 measure, 4 absolute continuity, 63, 64, 66– 68, 78, 108 Borel, 531 complete, 15, 18, 29 continuous, 7, 62, 88 counting, 6, 67 finite, 61 induced, 23, 24, 27, 33, 42, 52, 92 Lebesgue, 4, 6, 23, 38, 67 Lebesgue λn , 80 Lebesgue–Stieltjes, 4, 18, 20, 77, 78, 80 monotone property, 6, 62 motivation, 4 outer, 4 outer extension, 12 positive part, 62, 64 positive set, 62 probability, 20 product, 80, 81 regular, 16, 20 σ-finite, 12, 14, 61, 64, 66, 80 signed, 61, 64, 66, 76 singular, 63, 64, 71, 108 space, 4 total variation, 62, 513 uniform convergence, 41, 54, 60 mesh, 44 metric space, 101, 295 Arzel` a theorem, 103, 539, 541 Ascoli theorem, 103 compact, 102

581 compactness equivalents, 102 complete, 102, 537 covering numbers, 103 discontinuity set, 26 equicontinuity, 103 equivalent metrics, 101 properties, 101 regular measure, 16 relatively compact, 537 separable, 295, 537 sup norm, 103 totally bounded, 102 uniform continuity, 103 metrics d on (D, D), 297 Dudley, 540 Hellinger, 68, 543 Kolmogorov, 543 Lévy, 290 Prohorov, 540 total variation, 68, 380, 543, 544 modulus of continuity, 539 moment, 46, 47, 117 conditional, 195 consistent estimation, 226, 228, 231, 270, 272, 275 convergence of, 53, 244, 289, 293, 376 correlation, 48, 157, 463 covariance, 46, 117, 157, 199 cumulant, 357, 388 generating function, 395 mean, 46, 116, 117, 119 partial, 127, 128, 134, 226–228, 231, 269 partial variance, 145 skewness γ1 and tail heaviness γ2 , 279, 357, 383, 386–388, 391, 392, 554 standard deviation, 46 variance, 46, 116, 117, 119, 233 moments determine the normal, 293 moments of stable laws, 409 Monte Carlo, 225 natural parameters, 272

582 negligibility, 213–215, 217–221, 226, 228, 231, 233, 264, 274, 275, 367, 427, 430 uan, 260, 264, 283, 326, 377, 399 nonnegative definite, 193, 202, 363 norm L2 -norm, 104 q-norm, 329 rth mean, 46 sup, 22, 103, 295, 330 null set, 15, 83 oh big Op , 209 big O, 9 little op , 209 little o, 9 o-plus ⊕, or “at most”, 9 optional sampling, 472, 492, 499 order statistics, 120, 172, 173, 184, 281, 325 orthogonal, 104, 191, 338, 339, 369, 379, 512 partial sum process S n, 318, 319, 368, 536, 539 partition, 37 P∞ , 36 PLT, 367, 404 negligibility, 367 Poisson limit theorem, 367, 404 Poisson compound, 402 generalized, 402 Poisson approximation, 386, 389 Poisson process, 181, 470, 500 positive definite, 191 positive part, 21, 119, 468 predictable, 487, 489, 502, 511, 513, 514, 516 covariation, 512, 517 σ-field, 513, 514 variation, 489, 490, 501, 502, 512, 516 probability integral transformation, 113 inverse, 111 process, 90

PROBABILITY FOR STATISTICIANS convergence on (D, D), 538 counting, 470 D-class, 511 empirical process (see also), 121 existence of on (C, C), 298 existence of on (D, D), 301 general, 91 increasing, 511 independent increments, 297 normal, 90, 202 predicatable (see also), 487 realizations (equivalent), 90 realizations (smoother), 93 stationary, 297 stationary increments, 297 versions, 90 product F0 , F, A × A = σ[F], µ × ν, 79 countable, 86 cylinder set, 86 Fubini theorem, 82 integrable, 82 measurable rectangle, 79 measure, 80, 81 measure existence, 80, 82 null sets, 83 sections, 81, 82 σ-field, 79 space, 79 topology, 99 product lemma, 353, 366, 373 product limit estimator, 507 projection, 105, 195, 201, 449 conditional expectation, 174 mappings πt1 ,...,tk , 90 U-statistic, 450 qf, 111, 123, 145, 147, 231 order-r, 134, 136 standardized class K bounded by K0 , 140 ˜ 119, 123, 140, Winsorized qf K, 141, 143, 145, 147 quadratic form, 191, 369 quantile, 110, 112, 282, 286 median, 210, 280

INDEX random censorship, 505 random sample size, 225, 285 random variable (see also rv), 33 ranks and antiranks, 121, 172, 280, 326 rationals, 169, 289, 300, 362, 410, 476, 514 regression, 194, 378, 463 regularly varying, 145, 419 qf (see also), 137 De Haan, 139 definition: at infinity, 138 definition: at zero, 137 Karamata theorem, 137, 139 monotone density theorem, 137 uniformly, 139 representation theorem Kolmogorov for I2 , 402 Lévy–Khinchin, 405, 412 revisualization, 186, 303 rv, 33 existence of, 27, 85, 88, 92, 107 extended, 33 Rademacher, 185, 213, 242 random vector, 84 random element, 90 random process, 90 sample covariance, 455 mean, 120, 280 median, 121, 280 quantiles, 282, 284 space, 20 trimmed mean, 120 truncated mean, 120 variance, 120, 189, 270, 272, 278– 280, 370, 451 Winsorized mean, 120 Winsorized variance, 120 series, 71 three-series theorem, 241, 244 two-series theorem, 241 set theory, 3 λ-system, 9, 153 limsup (liminf), 8, 155, 204 monotone class, 3, 10, 81, 170

583 π ¯ -system, 3, 6, 9, 28, 153, 165, 309 set difference A \ B, 3 σ-field, 3 symmetric difference A∆B, 3 σ-field, 3 µ∗ -measurable sets A∗ , 4 A = σ[C], 12 Aˆµ , the completed σ-field, 15 histories, 305, 469 ¯ 24 induced F(X) ≡ X −1 (B), predictable, 513 preservation of, 22 symmetric, 156 tail, 155 skewness γ1 and tail heaviness γ2 , 383, 454 Skorokhod elementary theorem, 52, 114, 404 embedding (see also), 318 theorem, 376, 534, 542 slowly varying, 124, 125, 127, 137, 233, 268, 269, 271, 272, 419 space Borel, 169 Hilbert, 104, 174 inner product, 104 measurable, 3 measure, 4 probability, 20, 33 sample, 20 vector, 28, 104 spectral decomposition, 191, 340 St. Petersburg paradox, 221 stable laws, 407, 409, 410 strictly stable, 409, 411 symmetric, 410 statistics L-statistics, 437, 441 permutation, 428 R-statistics, 280, 329, 426, 462 U-statistics, 449, 456, 481 V-statistics, 457 Stein’s approach to CLTs, 255, 262, 263, 458, 459 Stirling’s formula, 186

584 stopping time, 239, 305, 311, 318, 321, 469, 490, 492 localizing, 511 strong Markov property, 182, 239, 240, 308, 409 subsequences, 55, 60, 288, 403, 533, 538 criterion relating →a.e. to →µ , 31 sufficient statistic, 172, 173 symmetrized rv, 210, 213, 219, 244, 372 tail equivalence, 143, 145 CLT, 134 WLLN, 136, 219 theorem absolute continuity of the integral, 42, 54 Arzel` a, 103, 539, 541 Ascoli, 103 Bochner, 363 Carathéodory extension (see also), 10, 12 Carleman, 293 Chentsov, 538 chf (see also this topic), 346 continuous mapping, 288, 534 convergence implications, 57 convergence of types, 272, 291, 408, 409, 412 correspondence, 18, 20, 27, 77, 85, 107 DCT, 41, 163 de la Vallée Poussin, 54, 56 Donsker, 320, 338, 539 Doob, 368, 487, 488, 539 Dynkin’s π-λ, 9 Egorov, 60 Esseen’s lemma (see also chf), 358 Fatou’s lemma, 40, 55, 71, 163, 244, 376, 404 Fubini, 10, 82, 115, 116, 170 fundamental of calculus, 76, 137, 258 Gnedenko–Kolmogorov, 123, 131, 232 Heine–Borel, 17, 19, 96, 110

PROBABILITY FOR STATISTICIANS Helly selection, 288, 291, 293, 539 Helly–Bray, 52, 56, 288, 291, 403, 404 kinder and gentler, 292 Hilbert space, 105, 106 Jordan–Hahn, 62 Karamata, 137, 139 Kolmogorov (see also), 87 Kronecker’s lemma, 205, 217, 221, 253 Lebesgue (see also), 64 Loève, 60 local limit (see also), 380 Lusin, 60 Mann–Wald, 52, 288, 291, 369 Marcinkiewicz–Zygmund, 220, 238 Mason, 329, 332, 336 MCT, 40, 163 mg convergence (see also), 473 moment convergence, 53, 244, 289 monotone class of Halmos, 10, 81, 170 monotone density, 137 only the zero function, 31, 43, 83, 164 P´ olya’s lemma, 289, 291 portmanteau, 290, 532 principal axes, 191 Prohorov, 537, 539 Radon–Nikodym, 66, 77, 158 residue, 344, 345 Riemann–Lebesgue lemma, 355, 359, 381, 392 Riesz, 31 Riesz–Fischer, 51 rth mean convergence, 238 Scheffé, 60, 380 Skorokhod (see also), 114 Slutsky, 33, 34, 285, 346, 376, 533 smoother realization of a process, 93 Stein’s lemma, 256 Strassen, 235, 323, 542 supporting hyperplane, 46, 50 Taylor, 72, 352 Tonelli, 82

INDEX Ulam, 541 unconscious statistician, 42, 67, 113, 158 uniform absolute continuity, 54 Vitali, 218 Vitali (see also), 55 von Neumann, 176 Weierstrass approximation, 223 tightness, 288, 289, 350, 537, 539 topology, 95 base and subbase, 95 boundary, 95, 100, 290 category, 100 compact, 96, 100, 102 continuity, 98, 100 homeomorphism, 97, 100 isometry, 100 locally compact, 99, 100 net, 96 one-point compactification, 99 product, 99 relative, 96, 323, 508 separation, 97, 100 triangular array, 365, 399 trigonometric identities, 340 trimmed, 119 fraction, 120, 417 mean, 119, 416 truncated, 207, 218, 226, 228, 235, 265 mean, 119, 226, 228, 416 type, 109, 291 convergence of types, 291 theorem of types, 272 uan, 283, 326, 377, 399 uncorrelated, 157 uniformity class, 146, 419 upcrossing, 473, 475, 478 Vitali covering, 17, 70 theorem, 55, 60, 475, 479, 498 waiting time, 179, 240 Wald’s identity, 240 Wasserstein distance, 114

585 Wilcoxon statistic, 454 Wilson–Hilferty approximation, 397 Winsorized, 119, 207, 277 fraction, 120 mean, 119, 226–228 moment, 269 outside, 119, 123, 127, 128, 140, 141, 143, 264, 266, 269, 276, 416 variance, 119, 123, 140, 141, 143, 145, 147, 232, 269–272, 276

Probability for Statisticians

Probability for Statisticians

Probability for Statisticians

Numerical analysis for statisticians

Numerical analysis for statisticians

Stereology for Statisticians

Numerical Analysis For Statisticians

Statistics for Non-Statisticians

Numerical Analysis for Statisticians

Stereology for Statisticians

A matrix handbook for statisticians

Numerical Analysis for Statisticians, Second Edition

Twelve British statisticians

A Matrix Handbook for Statisticians (Wiley Series in Probability and Statistics)

Probability For Dummies

Elementary Probability for Applications

Quantum probability for probabilists

Probability For Dummies

Probability for Risk Management

Statistical Thinking for Non-Statisticians in Drug Regulation

Nonparametric Statistics for Non-Statisticians: A Step-by-Step Approach

Statistical Thinking for Non-Statisticians in Drug Regulation

Statistical Thinking for Non-Statisticians in Drug Regulation

Asymptotic Approximations for Probability Integrals

Probability theory for statistical methods

Asymptotic approximations for probability integrals

Subjective Probability Models for Lifetimes

Probability

Probability

Probability

Probability

Probability for Statisticians