An Introduction to Measure-theoretic Probability, 2nd edition

An Introduction to Measure-Theoretic Probability Second edition An Introduction to Measure-Theoretic Probability Secon...

Author: George G. Roussas | 2014

184 downloads 1643 Views 5MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

An Introduction to Measure-Theoretic Probability Second edition

An Introduction to Measure-Theoretic Probability Second edition

by GEORGE G. ROUSSAS Department of Statistics University of California, Davis

AMSTERDAM • BOSTON • HEIDELBERG • LONDON NEW YORK • OXFORD • PARIS • SAN DIEGO SAN FRANCISCO • SINGAPORE • SYDNEY • TOKYO

Academic Press is an imprint of Elsevier

Academic Press is an imprint of Elsevier The Boulevard, Langford Lane, Kidlington, Oxford OX5 1GB, UK Radarweg 29, PO Box 211, 1000 AE Amsterdam, The Netherlands 225 Wyman Street, Waltham, MA 02451, USA 525 B Street, Suite 1800, San Diego, CA 92101-4495, USA Second edition 2014 Copyright © 2014, 2005 Elsevier Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means electronic, mechanical, photocopying, recording or otherwise without the prior written permission of the publisher. Permissions may be sought directly from Elsevier’s Science & Technology Rights Department in Oxford, UK: phone (+44) (0) 1865 843830; fax (+44) (0) 1865 853333; email: [email protected]. Alternatively you can submit your request online by visiting the Elsevier web site at http://elsevier.com/locate/ permissions, and selecting Obtaining permission to use Elsevier material. Notice No responsibility is assumed by the publisher for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions or ideas contained in the material herein. Because of rapid advances in the medical sciences, in particular, independent verification of diagnoses and drug dosages should be made. Library of Congress Cataloging-in-Publication Data Application submitted British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library For information on all Academic Press publications visit our web site at store.elsevier.com Printed and bound in USA 14 15 16 17 18 10 9 8 7 6 5 4 3 2 1 ISBN: 978-0-12-800042-7

This book is dedicated to the memory of Edward W. Barankin, the probabilist, mathematical statistician, classical scholar, and philosopher, for his role in stimulating my interest in probability with emphasis on detail and rigor. Also, to my dearest sisters, who provided material means in my needy student years, and unrelenting moral support throughout my career.

Pictured on the Cover Carathéodory, Constantine (1873–1950) He was born in Berlin to Greek parents and grew up in Brussels, Belgium. In high school, he twice won a prize as the best Mathematics student in Belgium. He studied Military Engineering in Belgium, and pursued graduate studies in Göttingen under the supervision of Hermann Minkowski. He is known for his contributions to the theory of functions, the calculus of variations, and measure theory. His name is identified with the theory of outer measure, an application of which in measure theory is the so-called Carathéodory Extension Theorem. Also, he did work on the foundations of thermodynamics, and in 1909, he published the “first axiomatic rigid foundation of thermodynamics,” which was acclaimed by Max Planck and Max Born. From correspondence between Albert Einstein and Constantine Carathéodory, it may be deduced that Carathéodory’s work helped Einstein in shaping some of his theories. In 1924, he was appointed professor of Mathematics at the University of Munich, where he stayed until his death in 1950.

xiii

Preface to First Edition This book in measure-theoretic probability has resulted from classroom lecture notes that this author has developed over a number of years, by teaching such a course at both the University of Wisconsin, Madison, and the University of California, Davis. The audience consisted of graduate students primarily in statistics and mathematics. There were always some students from engineering departments, and a handful of students from other disciplines such as economics. The book is not a comprehensive treatment of probability, nor is it meant to be one. Rather, it is an excursion in measure-theoretic probability with the objective of introducing the student to the basic tools in measure theory and probability as they are commonly used in statistics, mathematics, and other areas employing this kind of moderately advanced mathematical machinery. Furthermore, it must be emphasized that the approach adopted here is entirely classical. Thus, characteristic functions are a tool employed extensively; no use of martingale or empirical process techniques is made anywhere. The book does not commence with probabilistic concepts, and there is a good reason for it. As many of those engaged in teaching advanced probability and statistical theory know, very few students, if any, have been exposed to a measure theory course prior to attempting a course in advanced probability. This has been invariably the experience of this author throughout the years. This very fact necessitates the study of the basic measure-theoretic concepts and results—in particular, the study of those concepts and results that apply immediately to probability, and also in the form and shape they are used in probability. On the basis of such considerations, the framework of the material to be dealt with is therefore determined. It consists of a brief introduction to measure theory, and then the discussion of those probability results that constitute the backbone of the subject matter. There is minimal flexibility allowed, and that is exploited in the form of the final chapter of the book. From many interesting and important candidate topics, this author has chosen to present a brief discussion of some basic concepts and results of ergodic theory. From the very outset, there is one point that must be abundantly clarified, and that is the fact that everything is discussed in great detail with all proofs included; no room is allowed for summary unproven statements. This approach has at least two side benefits, as this author sees them. One is that students have at their disposal a comprehensive and detailed proof of what are often deep theorems. Second, the instructor may skip the reproduction of such proofs by assigning their study to students. In the experience of this author, there are no topics in this book which can be omitted, except perhaps for the final chapter. With this in mind, the material can be taught in two quarters, and perhaps even in one semester with appropriate calibration of the rate of presentation, and the omission of proofs of judiciously selected

xv

xvi

Preface to First Edition

theorems. With all details presented, one can also cover an entire year of instruction, perhaps with some supplementation. Most chapters are supplied with examples, and all chapters are concluded with a varying number of exercises. An unusual feature here is that an Answers Manual of all exercises will be made available to those instructors who adopt the book as the textbook in their course. Furthermore, an overview of each one of the 15 chapters is included in an appendix to the main body of the book. It is believed that the reader will benefit significantly by reviewing the overview of a chapter before the material in the chapter itself is discussed. The remainder of this preface is devoted to a brief presentation of the material discussed in the 15 chapters of the book, chapter-by-chapter. Chapter 1 commences with the introduction of the important classes of sets in an abstract space, which are those of a field, a σ-field, including the Borel σ-field, and a monotone class. They are illustrated by concrete examples, and their relationships are studied. Product spaces are also introduced, and some basic results are established. The discussion proceeds with the introduction of the concept of measurable functions, and in particular of random vectors and random variables. Some related results are also presented. This chapter is concluded with a fundamental theorem, Theorem 17, which provides for pointwise approximation of any random variable by a sequence of so-called simple random variables. Chapter 2 is devoted to the introduction of the concept of a measure, and the study of the most basic results associated with it. Although a field is the class over which a measure can be defined in an intuitively satisfying manner, it is a σ-field—the one generated by an underlying field—on which a measure must be defined. One way of carrying out the construction of a measure on a σ-field is to use as a tool the so-called outer measure. The concept of an outer measure is then introduced, and some of its properties are studied in the second section of the chapter. Thus, starting with a measure on a field, utilizing the associated outer measure and the powerful Carathéodory extension theorem, one ensures the definition of a measure over the σ-field generated by the underlying field. The chapter is concluded with a study of the relationship between a measure over the Borel σ-field in the real line and certain point functions. A measure always determines a class of point functions, which are nondecreasing and right-continuous. The important thing, however, is that each such point function uniquely determines a measure on the Borel σ-field. In Chapter 3, sequences of random variables are considered, and two basic kinds of convergences are introduced. One of them is the almost everywhere convergence, and the other is convergence in measure. The former convergence is essentially the familiar pointwise convergence, whereas convergence in measure is a mode of convergence not occurring in a calculus course. A precise expression of the set of pointwise convergence is established, which is used for formulating necessary and sufficient conditions for almost everywhere convergence. Convergence in measure is weaker than almost everywhere convergence, and the latter implies the former for finite measures. Almost everywhere convergence and mutual almost everywhere


convergence are equivalent, as is easily seen. Although the same is true when convergence in measure is involved, its justification is fairly complicated and requires the introduction of the concept of almost uniform convergence. Actually, a substantial part of the chapter is devoted in proving the equivalence just stated. In closing, it is to be mentioned that, in the presence of a probability measure, almost everywhere convergence and convergence in measure become, respectively, almost sure convergence and convergence in probability. Chapter 4 is devoted to the introduction of the concept of the integral of a random variable with respect to a measure, and the proof of some fundamental properties of the integral. When the underlying measure is a probability measure, the integral of a random variable becomes its expectation. The procedure of defining the concept of the integral follows three steps. The integral is first defined for a simple random variable, then for a nonnegative random variable, and finally for any random variable, provided the last step produces a meaningful quantity. This chapter is concluded with a result, Theorem 13, which transforms integration of a function of a random variable on an abstract probability space into integration of a real-valued function defined on the real line with respect to a probability measure on the Borel σ-field, which is the probability distribution of the random variable involved. Chapter 5 is the first chapter where much of what was derived in the previous chapters is put to work. This chapter provides results that in a real sense constitute the workhorse whenever convergence of integrals is concerned, or differentiability under an integral sign is called for, or interchange of the order of integration is required. Some of the relevant theorems here are known by names such as the Lebesgue Monotone Convergence Theorem, the Fatou–Lebesgue Theorem, the Dominated Convergence Theorem, and the Fubini Theorem. Suitable modifications of the basic theorems in the chapter cover many important cases of both theoretical and applied interest. This is also the appropriate point to mention that many properties involving integrals are established by following a standard methodology; namely, the property in question is first proved for indicator functions, then for nonnegative simple random variables, next for nonnegative random variables, and finally for any random variables. Each step in this process relies heavily on the previous step, and the Lebesgue Monotone Convergence Theorem plays a central role. Chapter 6 is the next chapter in which results of great utilitarian value are established. These results include the standard inequalities (Hölder (Cauchy–Schwarz), Minkowski, cr, Jensen), and a combination of a probability/moment inequality, which produces the Markov and Tchebichev inequalities. A third kind of convergence—convergence in the rth mean—is also introduced and studied to a considerable extent. It is shown that convergence in the rth mean is equivalent to mutual convergence in the rth mean. Also, necessary and sufficient conditions for convergence in the rth mean are given. These conditions typically involve the concepts of uniform continuity and uniform integrability, which are important in their own right. It is an easy consequence of the Markov inequality that convergence in the rth mean

xvii

xviii


implies convergence in probability. No direct relation may be established between convergence in the rth mean and almost sure convergence. In Chapter 7, the concept of absolute continuity of a measure relative to another measure is introduced, and the most important result from utilitarian viewpoint is derived; this is the Radon–Nikodym Theorem, Theorem 3. This theorem provides the representation of a dominated measure as the indefinite integral of a nonnegative random variable with respect to the dominating measure. Its corollary provides the justification for what is done routinely in statistics; namely, employing a probability density function in integration. The Radon–Nikodym Theorem follows easily from the Lebesgue Decomposition Theorem, which is a deep result, and this in turn is based on the Hahn–Jordan Decomposition Theorem. Although all these results are proved in great detail, this is an instance where an instructor may choose to give the outlines of the first two theorems, and assign to students the study of the details. Chapter 8 revolves around the concept of distribution functions and their basic properties. These properties include the fact that a distribution function is uniquely determined by its values on a set that is dense in the real line, that the discontinuities, being jumps only, are countably many, and that every distribution function is uniquely decomposed into two distribution functions, one of which is a step function and the other a continuous function. Next, the concepts of weak and complete convergence of a sequence of distribution functions are introduced, and it is shown that a sequence of distribution functions is weakly compact. In the final section of the chapter, the so-called Helly–Bray type results are established. This means that sufficient conditions are given under which weak or complete convergence of a sequence of distribution functions implies convergence of the integrals of a function with respect to the underlying distribution functions. The purpose of Chapter 9 is to introduce the concept of conditional expectation of a random variable in an abstract setting; the concept of conditional probability then follows as a special case. A first installment of basic properties of conditional expectations is presented, and then the discussion proceeds with the derivation of the conditional versions of the standard inequalities dealt with in Chapter 6. Conditional versions of some of the standard convergence theorems of Chapter 5 are also derived, and the chapter is concluded with the discussion of further properties of conditional expectations, and an application linking the abstract definition of conditional probability with its elementary definition. In Chapter 10, the concept of independence is considered first for events and then for σ-fields and random variables. A number of interesting results are discussed, including the fact that real-valued (measurable) functions of independent random variables are independent random variables, and that the expectation of the product of independent random variables is the product of the individual expectations. However, the most substantial result in this chapter is the fact that factorization of the joint distribution function of a finite number of random variables implies independence of the random variables involved. This result is essentially based on the fact that σ-fields generated by independent fields are themselves independent.


Chapter 11 is devoted to characteristic functions, their basic properties, and their usage for probabilistic purposes. Once the concept of a characteristic function is defined, the fundamental result, referred to in the literature as the inversion formula, is established in a detailed manner, and several special cases are considered; also, the applicability of the formula is illustrated by means of two concrete examples. One of the main objectives in this chapter is that of establishing the Paul Lévy Continuity Theorem, thereby reducing the proof of weak convergence of a sequence of distribution functions to that of a sequence of characteristic functions, a problem much easier to deal with. This is done in Section 3, after a number of auxiliary results are first derived. The multidimensional version of the continuity theorem is essentially reduced to the one-dimensional case through the so-called Cramér–Wold device; this is done in Section 4. Convolution of two distribution functions and several related results are discussed in Section 5, whereas in the following section additional properties of characteristic functions are established. These properties include the expansion of a characteristic function in a Taylor-like formula around zero with a remainder given in three different forms. A direct application of this expansion produces the Weak Law of Large Numbers and the Central Limit Theorem. In Section 8, the significance of the moments of a random variable is dramatized by showing that, under certain conditions, these moments completely determine the distribution of the random variable through its characteristic function. The rigorous proof of the relevant theorem makes use of a number of results from complex analysis, which for convenient reference are cited in the final section of the chapter. In the next two chapters—Chapters 12 and 13—what may be considered as the backbone of classical probability is taken up: namely, the study of the central limit problem is considered under two settings, one for centered random variables and one for noncentered random variables. In both cases, a triangular array of row-wise independent random variables is considered, and, under some general and weak conditions, the totality of limiting laws—in the sense of weak convergence—is obtained for the row sums. As a very special case, necessary and sufficient conditions are given for convergence to the normal law for both the centered and the noncentered case. In the former case, sets of simpler sufficient conditions are also given for convergence to the normal law, whereas in the latter case, necessary and sufficient conditions are given for convergence to the Poisson law. The Central Limit Theorem in its usual simple form and the convergence of binomial probabilities to Poisson probabilities are also derived as very special cases of general results. The main objective of Chapter 14 is to present a complete discussion of the Kolmogorov Strong Law of Large Numbers. Before this can be attempted, a long series of other results must be established, the first of which is the Kolmogorov inequalities. The discussion proceeds with the presentation of sufficient conditions for a series of centered random variables to convergence almost surely, the Borel– Cantelli Lemma, the Borel Zero–One Criterion, and two analytical results known as the Toeplitz Lemma and the Kronecker Lemma. Still the discussion of another two results is needed—one being a weak partial version of the Kolmogorov Strong Law

xix

xx


of Large Numbers, and the other providing estimates of the expectation of a random variable in terms of sums of probabilities—before the Kolmogorov Strong Law of Large Numbers, Theorem 7, is stated and proved. In Section 4, it is seen that, if the expectation of the underlying random variable is not finite, as is the case in Theorem 7, a version of Theorem 7 is still true. However, if said expectation does not exist, then the averages are unbounded with probability 1. The chapter is concluded with a brief discussion of the tail σ-field of a sequence of random variables and pertinent results, including the Kolmogorov Zero–One Law for independent random variables, and the so-called Three Series Criterion. Chapter 15 is not an entirely integral part of the body of basic and fundamental results of measure-theoretic probability. Rather, it is one of the many possible choices of topics that this author could have covered. It serves as a very brief introduction to an important class of discrete parameter stochastic processes—stationary and ergodic or nonergodic processes—with a view toward proving the fundamental result, the Ergodic Theorem. In this framework, the concept of a stationary stochastic process is introduced, and some characterizations of stationarity are presented. The convenient and useful coordinate process is also introduced at this point. Next, the concepts of a transformation as well as a measure-preserving transformation are discussed, and it is shown that a measure-preserving transformation along with an arbitrary random variable define a stationary process. Associated with each transformation is a class of invariant sets and a class of almost sure invariant sets, both of which are σ-fields. A special class of transformations is the class of ergodic transformations, which are defined at this point. Invariance with respect to a transformation can also be defined for a stationary sequence of random variables, and it is so done. At this point, all the required machinery is available for the formulation of the Ergodic Theorem; also, its proof is presented, after some additional preliminary results are established. In the final section of the chapter, invariance of sets and of random variables is defined relative to a stationary process. Also, an alternative form of the Ergodic Theorem is given for nonergodic as well as ergodic processes. In closing, it is to be pointed out that one direction of the Kolmogorov Strong Law of Large Numbers is a special case of the Ergodic Theorem, as a sequence of independent identically distributed random variables forms a stationary and ergodic process. Throughout the years, this author has drawn upon a number of sources in organizing his lectures. Some of those sources are among the books listed in the Selected References Section. However, the style and spirit of the discussions in this book lie closest to those of Loève’s book. At this point, I would like to mention a recent worthy addition to the literature in measure theory—the book by Eric Vestrup, not least because Eric was one of our Ph.D. students at the University of California, Davis. The lecture notes that eventually resulted in this book were revised, modified, and supplemented several times throughout the years; comments made by several of my students were very helpful in this respect. Unfortunately, they will have to remain anonymous, as I have not kept a complete record of them, and I do not want to provide an incomplete list. However, I do wish to thank my colleague and friend Costas Drossos for supplying a substantial number of exercises, mostly accompanied by


answers. I would like to thank the following reviewers: Ibrahim Ahmad, University of Central Florida; Richard Johnson, University of Wisconsin; Madan Puri, Indiana University; Doraiswamy Ramachandran, California State University at Sacramento; and Zongwu Cai, University of North Carolina at Charlotte. Finally, thanks are due to my Project Assistant Newton Wai, who very skillfully turned my manuscript into an excellent typed text. George G. Roussas Davis, California November 2003

xxi

Preface to Second Edition This is a revised version of the first edition of the book with copyright year 2005. The basic character of the book remains the same, although its style is slightly different. Whatever changes were effected were made to correct misprints and oversights; add some clarifying points, as well as insert more references to previous parts of the book in support of arguments made; make minor modifications in the formulation, and in particular, the proof of some results; and supply additional exercises. Specifically, the formulation of Theorem 8 in Chapter 3 has been rearranged. The proof of Theorem 3, case 3, in Chapter 4, has been simplified. The proof of Theorem 12 in Chapter 5 has been modified. Proposition 1, replaces Remark 6(ii) in Chapter 7. The proof of Theorem 3(iii) in Chapter 8 has been modified, and so has the concluding part of the proof of Theorem 5 in the same chapter. Likewise for the proofs of Theorems 7 and 8 in the same chapter. Remark 2 was inserted in Chapter 9 in order to further illustrate the abstract definition and the significance of the conditional expectation. Section 3 of Chapter 11 has been restructured. Theorem 3 has been split into two parts, Theorem 3 and Theorem 3*. Part (i) is the same in both of these theorems, as well as in the original theorem. There is a difference, however, in the formulation of part (ii) in the new versions of the theorems. Theorem 3 here is formulated along the familiar lines involving distribution functions and characteristic functions of random variables. Its formulation is followed by two lemmas, which facilitate its proof. The formulation of the second part of Theorem 3* is more general, and along the same lines as the converse of the original Theorem 3. Theorem 3* is also followed by two lemmas and one proposition, which lead to its justification. This section is concluded with two propositions, Propositions 2 and 3, where some restrictions imposed in the formulation of Lemmas 1–4 and Proposition 1 are lifted. In the same chapter, Chapter 11, the proof of Theorem 8 is essentially split into two parts, with the insertion of a “Statement” (just a few lines below relation (11.28)), in order to emphasize the asserted uniformity in the convergence. In Chapter 12, Example 3 was added, first to illustrate the process of checking the Lindeberg-Feller condition, and second to provide some insight into this condition. In Chapter 14, the second part of Lemma 7 has been modified, and so has its proof . Finally, a new chapter, Chapter 16, has been added. This Chapter discusses some material on statistical inference, and it was added for the benefit of statistically oriented users of the book and upon the suggestion of a reviewer of the revised edition of the book. Its main purpose, however, is to demonstrate as to how some of the theorems, corollaries, etc. discussed in the book apply in establishing statistically inference results. For a chapter-by-chapter brief description of the material discussed in the book, and also advice us to how the book can be used, the reader should go over the preface of its first edition.

xxiii

xxiv

Preface to Second Edition

The Answers Manual has been revised along the same lines as the text of the book. Thus, misprints and oversights have been corrected, and a handful of solutions have been modified. Of course, solutions to all new exercises are supplied. Again, the Answers Manual, in its revised version, will be made available to all those instructors who adopt the book as the textbook in their course. Misprints and oversights were located in the usual way; that is, by teaching from the book. Many of the misprints and oversights were pointed out by attentive students. In this respect, special mention should be made of my students Qiuyan Xu and Gabriel Becker. Clarifications, modifications, and rearrangement of material, as described earlier, were also stimulated, to a large extent, by observations made and questions posed by students. Warm thanks are extended to all those who took my two-quarter course the last two offerings. Also, I am grateful to Stacy Hill and Paul Ressel for constructive comments. In particular, I am indebted to Michael McAssey, a formerly graduate student in the Department of Statistics, for the significant role he played toward the revision of the book and the Answers Manual. The accuracy and efficiency by which he handled the material was absolutely exemplary. Thanks are also due to Chu Shing (Randy) Lai for most efficiently implementing some corrections and inserting additional material into the book and the Answer Manual. In closing, I consider it imperative to mention the following facts. Each chapter is introduced by a brief summary, describing the content of the chapter. In addition, there is an appendix in the book, Appendix A, where a much more extensive description is provided, chapter-by-chapter. It is my opinion that the reader would benefit greatly by reading this appendix before embarking on the study of the chapters. In this revision, a new appendix, Appendix B, has been added, providing a brief review of the Riemann–Stieltjes integral, and its relationship to the Riemann integral and the Lebesgue integral on the real line. The Riemann–Stieltjes integral is used explicitly in parts of Chapter 8, and implicitly in part of Chapters 11 through 13. Finally, it is mentioned here that some notation and abbreviations have been added to refresh readers’ memory and ensure uniformity in notation. George G. Roussas Davis, California September 2013

CHAPTER

Certain Classes of Sets, Measurability, and Pointwise Approximation

1

In this introductory chapter, the concepts of a field and of a σ -field are introduced, they are illustrated by means of examples, and some relevant basic results are derived. Also, the concept of a monotone class is defined and its relationship to certain fields and σ -fields is investigated. Given a collection of measurable spaces, their product space is defined, and some basic properties are established. The concept of a measurable mapping is introduced, and its relation to certain σ -fields is studied. Finally, it is shown that any random variable is the pointwise limit of a sequence of simple random variables.

1.1 Measurable Spaces Let be an abstract set (or space) and let C be a class of subsets of ; i.e., C ⊆ P(), the class of all subsets of . Definition 1.

C is said to be a field, usually denoted by F, if

(i) C is nonempty. (ii) If A ∈ C, then Ac ∈ C. (iii) If A1 , A2 ∈ C, then A1 ∪ A2 ∈ C.

Remark 1. In view of (ii) and (iii), the union A1 ∪ A2 may be replaced by the intersection A1 ∩ A2 . Examples. (Recall that a set is countable if it is either finite or it has the same cardinality as the set of integers. In the latter case it is countably infinite. A set is uncountable if it has the same cardinality as the real numbers.) (1) C = {, } is a field called the trivial field. (It is the smallest possible field.) (2) C = {all subsets of } = P() is a field called the discrete field. (It is the largest possible field.) (3) C = {, A, Ac , } for some A with ⊂ A ⊂ . (4) Let be infinite (countably or not) and let C = {A ⊆ ; A is finite or Ac is finite}. Then C is a field. (5) Let C be the class of all (finite) sums (unions of pairwise disjoint sets) of the partitioning sets of a finite partition of an arbitrary set (see Definition 12 below). Then C is a field (induced or generated by the underlying partition). An Introduction to Measure-Theoretic Probability, Second Edition. http://dx.doi.org/10.1016/B978-0-12-800042-7.00001-3 Copyright © 2014 Elsevier Inc. All rights reserved.

1

2

CHAPTER 1 Certain Classes of Sets, Measurability

Remark 2. In Example 4, it is to be observed that if is finite rather than infinite, then C = P(). Consequences of Definition 1. (1) , ∈ F for every F. (2) If A j ∈ F, j = 1, . . . , n, then nj=1 A j ∈ F. (3) If A j ∈ F, j = 1, . . . , n, then nj=1 A j ∈ F. Remark 3. It is shown by examples that A j ∈ F, j ≥ 1, need not imply ∞ ∞ A ∈ F, and similarly for A (see Remark 5 below). j j j=1 j=1 Definition 2. C is said to be a σ -field , usually denoted by A, if it is a field and (iii) in Definition 1 is strengthened to (iii ) If A j ∈ C, j = 1, 2, . . ., then ∞ j=1 A j ∈ C. Remark 4. In view of (ii) and (iii), the union in (iii ) may be replaced by the intersection ∞ j=1 A j . Examples. (6) C = {, } is a σ -field called the trivial σ -field. (7) C = P() is a σ -field called the discrete σ -field. (8) Let be uncountable and let C = {A ⊆ ; A is countable or Ac is countable}. Then C is a σ -field. (Of course, if is countably infinite, then C = P()). (9) Let C be the class of all countable sums of the partitioning sets of a countable partition of an arbitrary set . Then C is a σ -field (induced or generated by the underlying partition). Remark 5. A σ -field is always a field, but a field need not be a σ -field. In fact, in Example 4 take = (real line), and let A j = {k integer; − j ≤ k ≤ j}, j = 0, 1, . . . Then A j ∈ C, nj=0 A j ∈ C for any n = 0, 1, . . . but ∞ j=0 A j (= set of all integers) ∈ / C. Let I be any index set. Then Theorem 1. (i) If F j , j ∈ I are fields, so is j∈I F j = {A ⊆ ; A ∈ F j , j ∈ I }. (ii) If A j , j ∈ I are σ -fields, so is j∈I A j = {A ⊆ ; A ∈ A j , j ∈ I }.

Proof.

Immediate.

Let C be any class of subsets of . Then Theorem 2. (i) There is a unique minimal field containing C. This is denoted by F(C) and is called the field generated by C.

1.1 Measurable Spaces

(ii) There is a unique minimal σ -field containing C. This is denoted by σ (C) and is called the σ -field generated by C. Proof. (i) F(C) = j∈I F j , where {F j , j ∈ I } is the nonempty class of all fields containing C. (ii) σ (C) = j∈I A j , where {A j , j ∈ I } is the nonempty class of all σ -fields containing C. Remark 6. Clearly, σ (F(C)) = σ (C). Indeed, C ⊆ F(C), which implies σ (C) ⊆ σ (F(C)). Also, for every σ -field Ai ⊇ C it holds Ai ⊇ F(C), since A i is a field (being a σ -field), and F(C) is the minimal field (over C). Hence σ (C) = i Ai ⊇ F(C). Since σ (C) is a σ -field, it contains the minimal σ -field over F(C), σ (F(C)); i.e., σ (C) ⊇ σ (F(C)). Hence σ (C) = σ (F(C)). Application 1. Let = and C0 = {all intervals in } = {(x, y), (x, y], [x, y), [x, y], (−∞, a), (−∞, a], (b, ∞), [b, ∞); x, y ∈ , x < y, a, b ∈ }. Then σ (C0 ) is denoted by B and is called the Borel σ -field over the real line. The sets in B are ¯ = ∪ {−∞, ∞}. ¯ is called the extended real line and the called Borel sets. Let _ σ -field B generated by B ∪ {−∞} ∪ {∞} the extended Borel σ -field. x, x + n1 with Remark 7. {x} ∈ B for every x ∈ . Indeed, {x} = ∞ n=1 1 x, x + n1 ∈ B. Hence ∞ n=1 x, x + n ∈ B, or {x} ∈ B. Alternatively, with a < x < b, we have {x} = (a, x] ∩ [x, b) ∈ B. Definition 3. The pair (, A) is called a measurable space and the sets in A _ ¯ B ) the measurable sets. In particular, (, B) is called the Borel real line, and (, extended Borel real line. Let C again be a class of subsets of . Then C is called a monotone class if A j ∈ C, j = 1, 2, . . . and A j ↑ (i.e., de f A1 ⊆ A2 ⊆ · · · ) or A j ↓ (i.e., A1 ⊇ A2 ⊇ · · · ), then lim j→∞ A j = ∞ j=1 A j ∈ C de f ∞ and lim j→∞ A j = j=1 A j ∈ C, respectively. Definition 4.

Theorem 3. A σ -field A is a monotone field (i.e., a field that is also a monotone class) and conversely. Proof. One direction is immediate. As for the other, let F be a monotone ∞field and A ∈ F. We have: let any A j ∈ F, j = 1, 2, . . .. To show that ∞ j j=1 j=1 A j = n B , where B = A1 ∪ (A1 ∪ A2 ) ∪ · · · ∪ (A1 ∪ · · · ∪ An ) ∪ · · · = ∞ n j=1 A j , ∞n=1 n and hence Bn ∈ F, n = 1, 2, . . . and Bn ↑. Thus n=1 Bn ∈ F. Theorem 4. If M j , j ∈ I , are monotone classes, so is j∈I M j = {A ⊆ ; A ∈ M j , j ∈ I }. Proof.

Immediate.

3

4


There is a unique minimal monotone class M containing C. Proof. M = j∈I M j , where {M j , j ∈ I } is the nonempty class of all monotone classes containing C. Theorem 5.

Remark 8.

{M j , j ∈ I } is nonempty since σ (C) or P() belong in it.

Remark 9. It may be seen by means of examples (see Exercise 12) that a monotone class need not be a field. However, see the next lemma, as well as Theorem 6. Lemma 1. Let C be a field and M be the minimal monotone class containing C. Then M is a field. Proof. where

In order to prove that M is a field, it suffices to prove that relations (*) hold, ⎫ (i) A ∩ B ∈ M ⎬ (∗) for every A, B ∈ M, we have: (ii) Ac ∩ B ∈ M . ⎭ ⎩ (iii) A ∩ B c ∈ M ⎧ ⎨

(That is, for every A, B ∈ M, their intersection is in M, and so is the intersection of any one of them by the complement of the other.) In fact, M ⊇ C, implies ∈ M. Taking B = , we get that for every A ∈ M, Ac ∩ = Ac ∈ M (by (ii)). Since also A ∩ B ∈ M (by (i)) for all A, B ∈ M, the proof would be completed. In order to establish (*), we follow the following three steps: Step 1. For any A ∈ M, define M A = {B ∈ M; (*) holds}, so that M A ⊆ M. Obviously A ∈ M A , since ∈ M. It is asserted that M A is a monotone class. Let de f B j = B ∈ M A ; i.e., to show B j ∈ M A , j = 1, 2, . . . , B j ↑. To showthat ∞ j=1 that (*) holds. We have: A ∩ B = A ∩ ( j B j ) = j (A ∩ B j ) ∈ M, since M is monotone and A ∩ B j ↑. Next, Ac ∩ B = Ac ∩ ( j B j ) = j (Ac ∩B j ) ∈ M M, by (*)(ii), and Ac ∩ B j ↑. Finally, A ∩ B c = A ∩ ( j B j )c = since Ac ∩ B j ∈ c A ∩ ( j B j ) = j (A ∩ B cj ) with A ∩ B cj ∈ M by (*)(iii) and A ∩ B cj ↓, so that c j (A ∩ B j ) ∈ M since M is monotone. The case that B j ↓ is treated similarly, and the proof that M A is a monotone class is complete. Step 2. If A ∈ C, then M A = M. As already mentioned, M A ⊆ M. So it suffices to prove that M ⊆ M A . Let B ∈ C. Then (*) holds and hence B ∈ M A . Therefore C ⊆ M A . By step 1, M A is a monotone class and M is the minimal monotone class containing C. Thus M ⊆ M A and hence M A = M. Step 3. If A is any set in M, then M A = M. We show that C ⊆ M A , which implies M ⊆ M A since M A is a monotone class containing C and M is the minimal monotone class over C. Since also M A ⊆ M, the result M A = M would follow. To show C ⊆ M A , take B ∈ C and consider M B . Then M B = M by step 2. Since A ∈ M, we have A ∈ M B , which implies that B ∩ A, B c ∩ A, and B ∩ Ac all belong in M; or A ∩ B, Ac ∩ B, and A ∩ B c belong in M, which means that B ∈ M A . Theorem 6. Let C be a field and M be the minimal monotone class containing C. Then M = σ (C).

1.2 Product Measurable Spaces

Proof. Evidently, M ⊆ σ (C) since every σ -field is a monotone class. By Lemma 1, M is a field, and hence a σ -field, by Theorem 3. Thus, M ⊇ σ (C) and hence M = σ (C). Remark 10. Lemma 1 and Theorem 6 just discussed provide an illustration of the intricate relation of fields, monotone classes, and σ -fields in a certain setting. As will also be seen in several places in this book, monotone classes are often used as tools in arguments meant to establish results about σ -fields. In this kind of arguments, the roles of a field and of a monotone class may be substituted by the so-called π -systems and λ-systems, respectively. The definition of these concepts may be found, for example, in page 41 in Billingsley (1995). A result analogous to Theorem 6 is then Theorem 1.3 in page 5 of the reference just cited, which states that: If P is a π -system and G is a λ-system, then P ⊂ G implies σ (P) ⊂ G.

1.2 Product Measurable Spaces Consider the measurable spaces (1 , A1 ), (2 , A2 ). Then Definition 5. The product space of 1 , 2 , denoted by 1 × 2 , is defined as follows: 1 × 2 = {ω = (ω1 , ω2 ); ω1 ∈ 1 , ω2 ∈ 2 }. In particular, for A ∈ A1 , B ∈ A2 the product of A, B, denoted by A× B, is defined by: A× B = {ω = (ω1 , ω2 ); ω1 ∈ A, ω2 ∈ B}, and the subsets A × B of 1 × 2 for A ∈ A1 , B ∈ A2 are called (measurable) rectangles. A, B are called the sides of the rectangle. From Definition 5, one easily verifies the following lemma. Lemma 2. Consider the rectangle E = A × B. Then, with “+” denoting union of disjoint events, (i) E c = (A × B c ) + (Ac × 2 ) = (Ac × B) + (1 × B c ). Consider the rectangles E 1 = A1 × B1 , E 2 = A2 × B2 . Then (ii) E 1 ∩ E 2 = (A1 ∩ A2 ) × (B1 ∩ B2 ). Hence E 1 ∩ E 2 = if and only if at least one of the sets A1 ∩ A2 , B1 ∩ B2 is . Consider the rectangles E 1 , E 2 as above, and the rectangles F1 = A 1 × B1 , F2 = A 2 × B2 . Then

(iii) (E 1 ∩ F1 ) ∩ (E 2 ∩ F2 ) = [(A1 ∩ A1 ) × (B1 ∩ B1 )] ∩ [(A2 ∩ A2 ) ×(B2 ∩ B2 )] (by (ii)) = [(A1 ∩ A 1 ) ∩ (A2 ∩ A 2 )] ×[(B1 ∩ B1 ) ∩ (B2 ∩ B2 )]

(by (ii))

A2 ) ∩ (A 1

= [(A1 ∩ ×[(B1 ∩

∩ A 2 )] B2 ) ∩ (B1 ∩ B2 )].

Hence, the left-hand side is if and only if at least one of (A1 ∩ A2 ) ∩ (A 1 ∩ A 2 ), (B1 ∩ B2 ) ∩ (B1 ∩ B2 ) is . Theorem 7. Let C be the class of all finite sums (i.e., unions of pairwise disjoint) of rectangles A × B with A ∈ A1 , B ∈ A2 . Then C is a field (of subsets of 1 × 2 ).

5

6


Proof. Clearly, C = . Next, let we show that E ∩ F ∈ C. In mE, F ∈ C. Then E i , F = nj=1 F j with E i = Ai × Bi , i = fact, E, F ∈ C implies that E = i=1 m n 1, . . . , m, F j = A j × B j , j = 1, . . . , n. Thus E ∩ F = i=1 j=1 (E i ∩ F j ) and

E i ∩ F j , E i ∩ F j are disjoint for (i, j) = (i , j ) by Lemma 2 (ii), (iii). Indeed, in Lemma 2 (iii), make the identification: A1 = Ai , B1 = Bi , A2 = Ai , B2 = Bi , A 1 = A j , B1 = B j , A 2 = A j , B2 = B j to get (E i ∩ F j ) ∩ (E i ∩ F j ) = [(Ai ∩ Ai ) ∩ (A j ∩ A j )] × [(Bi ∩ Bi ) ∩ (B j ∩ B j )] by the third line on the right-hand side in Lemma 2(iii), and at least one of (Ai ∩ Ai ) ∩ (A j ∩ A j ), (Bi ∩ Bi ) ∩ (B j ∩ B j ) is equal to . Then, by Lemma 2(iii) again, (E i ∩ F j ) ∩ (E i ∩ F j ) = , and therefore m n

E ∩ F = i=1 j=1 (E i ∩ F j ). However, E i ∩ F j = (Ai ∩ A j ) × (Bi ∩ B j ) (by Lemma 2(ii)), and Ai ∩ A j ∈ A1 , Bi ∩ B j ∈ A2 , i = 1, . . . , m, j = 1, . . . , n. Thus E ∩ F is the sum of finitely many rectangles and hence E ∩ F ∈ C. (By ∈ C, k = 1, . . . , , then induction it is also true that if E k k=1 E k ∈ C.) Finally, m m m E i )c = i=1 E ic = i=1 [(Ai × Bic ) + (Aic × 2 )] (by Lemma 2(i)), E c = ( i=1 and Ai × Bic , Aic × 2 are disjoint rectangles so that their sum is in C. But then so is their intersection over i = 1, . . . , m by the induction just mentioned. The proof is completed. Remark 11. Clearly, the theorem also holds true if we start out with fields F1 and F2 rather than σ -fields A1 and A2 . Definition 6. The σ -field generated by the field C is called the product σ -field of A1 , A2 and is denoted by A1 ×A2 . The pair (1 ×2 , A1 ×A2 ) is called the product measurable space of the (measurable) spaces (1 , A1 ), (2 , A2 ). If we have n ≥ 2 measurable spaces (i , Ai ), i = 1, . . . , n, the product measurable space (1 ×· · ·×n , A1 ×· · ·×An ) is defined in an analogous way. In particular, if 1 = · · · = n = and A1 = · · · = An = B, then the product space (n , B n ) is the n-dimensional Borel space, where n = × · · · × , B n = B × · · · × B (n factors), and B n is called the n-dimensional Borel σ -field. The members of B n are called the n-dimensional Borel sets. Now we consider the case of infinitely (countably or not) many measurable spaces (t , At ), t ∈ T , where the (= ) index set T will usually be the real line or the positive half of it or the unit interval (0, 1) or [0,1].

Definition 7. The product space of t , t ∈ T , denoted by t∈T t or T , is defined by T = t∈T t = {ω = (ωt , t ∈ T ); ωt ∈ t , t ∈ T }. By forming the point ω = (ωt , t ∈ T ) with ωt ∈ t , t ∈ T , we tacitly assume, by invoking the axiom of choice, that there exists a function on T into t∈T t with t = , t ∈ T , whose value at t, ωt , belongs in t . Now for T = {1, 2}, 1 × 2 = {ω = (ω1 , ω2 ); ω1 ∈ 1 , ω2 ∈ 2 }. Also, let f : T → 1 ∪ 2 such that f (1) ∈ 1 , f (2) ∈ 2 . Then ( f (1), f (2)) ∈ 1 × 2 . Conversely, any (ω1 , ω2 ) ∈ 1 × 2 is the (ordered) pair of values of a function f on T into 1 ∪ 2 with f (1) ∈ 1 , f (2) ∈ 2 ; namely, the function for which f (1) = ω1 , f (2) = ω2 . Thus, 1 × 2 may be looked upon as the collection of all functions f on T into 1 ∪ 2 with f (1) ∈ 1 , f (2) ∈ 2 . Similar interpretation

1.3 Measurable Functions and Random Variables

holds for any finite collection of (= ) i , i = 1, . . . , n, as well as

any collection of (= ) t , t ∈ T (= ) (by the axiom of choice). Thus, T = t∈T t = { f : T → t∈T t ; f (t) ∈ t , t ∈ T }. In particular, if T = and t = , t ∈ T , then T = t∈T t is the set of all real-valued functions defined on . Remark 12. In many

applications, we take T = [0, 1], t = , t ∈ T , and we consider subsets of t∈T t , such as the set of all continuous functions, denoted by C([0, 1]), or the set of all bounded and right-continuous functions, denoted by D([0, 1]). Next, for any

positive integer N , let TN = {t1 , . . . , t N } with ti ∈ T , i = 1, . . . , N , and let A TN = t∈TN At . Then A TN is a rectangle in t1 × · · · × t N . Furthermore,

Definition 8. The subset A TN × t∈T c t = t∈TN At × t∈T c t of t∈T t N N

is called a product cylinder in T = t∈T t with basis A TN and sides At ∈ At , t ∈ TN . Theorem 8. Let C be the class of all finite sums of all product cylinders. Then C is a field (of subsets of t∈T t ). The proof of this theorem is based on the same ideas as those used in proving Theorem 7. Definition 9. The σ -field generated by C is called the product σ -field of At , t ∈ T , and is denoted by AT = t∈T At . The pair (T = t∈T t , AT = t∈T At ) is called the product measurable space of the (measurable) spaces (t , At ), t ∈ T . The space (∞ , B ∞ ), the (countably) infinite dimensional Borel space, where ∞ = × × · · ·, and B ∞ = B × B × · · ·, is often of special interest. B ∞ is the (countably) infinite-dimensional Borel σ -field. The members of B ∞ are called (countably) infinite dimensional Borel sets. For more information, see also page 62 of Loève (1963).

1.3 Measurable Functions and Random Variables Let , be two spaces and let X be a mapping such that X : → . Then the set operator X −1 associated with the mapping X is defined as follows: Definition 10. X −1 : P( ) → P() and X −1 (A ) = A, where A = {ω ∈ ; X (ω) ∈ A }; X −1 (A ) is the inverse image of A under X . From Definition 10 it follows that Theorem 9. (i) (ii) (iii) (iv)

If A ∩ B = , then [X −1 (A )] ∩ [X −1 (B )] = . X −1 (A c ) = [X −1 (A )]c . X −1 ( j∈I A j ) = j∈I X −1 (A j ) and X −1 ( j∈I A j ) = j∈I X −1 (A j ). X −1 ( j∈I A j ) = j∈I X −1 (A j ).

7

8


(v) X −1 (A − B ) = X −1 (A ) − X −1 (B ) (equivalently, X −1 (A ∩ B c ) = X −1 (A ) ∩ [X −1 (B )]c ). (vi) If A ⊆ B , then X −1 (A ) ⊆ X −1 (B ). (vii) C ⊆ C

, then X −1 (C ) ⊆ X −1 (C

), where X −1 (C ) = {A ⊆ ; A = X −1 (A ) for some A ∈ C }; and similarly for C

. Now let us assume that is supplied with a σ -field A . Then we have Theorem 10. Define the class C of subsets of as follows: C = X −1 (A ). Then C is a σ -field (i.e., the inverse image of a σ -field is a σ -field). Remark 13.

This σ -field is called the σ -field induced (in ) by X .

Proof of Theorem 10.

This is immediate from (ii) and (iii) of Theorem 9.

Next assume that is supplied with a σ -field A. Then Theorem 11. Define the class C of subsets of as follows: C = {A ⊆ ; X −1 (A ) ∈ A}. Then C is a σ -field. Proof.

Immediate from (ii) and (iii) of Theorem 9.

de f

Theorem 12. Let C be a class of subsets of and let A = σ (C ). Then A = σ [X −1 (C )] = X −1 (A ). Proof. We have: C ⊆ A implies X −1 (C ) ⊆ X −1 (A ), and this implies A ⊆ X −1 (A ) because X −1 (A ) is a σ -field by Theorem 10. Thus, to show X −1 (A ) ⊆ A. Define C ∗ as follows: C ∗ = {A ⊆ ; X −1 (A ) ∈ A}. Then, clearly, C ⊆ C ∗ , and C ∗ is a σ -field by Theorem 11, and the fact that X −1 (C ∗ ) ⊆ A. Hence A ⊆ C ∗ and therefore X −1 (A ) ⊆ X −1 (C ∗ ) ⊆ A. Thus X −1 (A ) ⊆ A and X −1 (A ) = A. Now assume that both and are supplied with σ -fields A and A , respectively. Then Definition 11. If X −1 (A ) ⊆ A we say that X is measurable with respect to A and A , or just measurable if no confusion is possible. In particular, if ( , A ) = (n , B n ) and X is measurable, we say that X is an n-dimensional random vector and if n = 1, a random variable (r.v.). If (, A) = (n , B n ), ( , A ) = (m , B m ) and f : →

is measurable, then f is called a Borel function, and for m = 1 a Baire function. The meaning and significance of Theorem 12 are this: if we want to check measurability of X , it suffices only to check that X −1 (C ) ⊆ A, where C is a class generating A . Indeed, if X : (, A) → ( , A ), then (A, A )− measurability of X means X −1 (A ) ⊆ A. Let X −1 (C ) ⊆ A and let A = σ (C ). Then σ [X −1 (C )] ⊆ A. But σ [X −1 (C )] = X −1 (A ). Thus X −1 (A ) ⊆ A. In particular, in the Borel real line, X is a r.v. if only X −1 (C0 ) or X −1 (C j ) or X −1 (C j ) ⊆ A, j = 1, . . . , 8, where C0 is as in Application 1, the classes C j , j = 1, . . . , 8, are the classes of intervals each consisting of intervals from C0 of one type, and C j is the class taken from C j when the endpoints of the intervals are restricted to be rational numbers j = 1, . . . , 8. Theorem 13. Let X : (, A) → ( , A ) be measurable and let f : ( , A ) → (

, A

) be measurable. Define f (X ) : →

as follows: f (X )(ω) = f [X (ω)].


Then the mapping f (X ) is measurable. That is, a measurable mapping of a measurable mapping is a measurable mapping. Proof. For A

∈ A

, [ f (X )]−1 (A

) = X −1 [ f −1 (A

)] = X −1 (A ) with A ∈ A . Thus X −1 (A ) = A ∈ A. Corollary 1. Proof.

Borel functions of random vectors are random vectors.

Take ( , A ) = (n , B n ), (

, A

) = (m , B m ).

(, A), ( , A )

and assume that and We now consider the measurable spaces are also provided with topologies T and T , respectively. (Recall that T is a topology for if T is a class of subsets of with the following properties: (i) , ∈ T , (ii) T is closed under finite intersections of members of T , and (iii) T is closed under arbitrary unions of members of T .) The pair (, T ) is called a topological space, and the members of T are called open sets. Also, f : (, T ) → ( , T ) is said to be continuous (with respect to the topologies T and T ), if f −1 (T ) ∈ T for every T ∈ T . Theorem 14. Let f : → be continuous and let that T ⊆ A, A = σ (T ). Then f is measurable. Proof. Continuity of f implies f −1 (T ) ∈ T , T ∈ T . Hence f −1 (T ) ⊆ A. Since T generates A , we have f −1 (A ) = σ [ f −1 (T )] ⊆ A by Theorem 12. Application 2. Recall that a class of sets in T is a base for T if every T in T is the union of members of this class. A topology T and the corresponding topological space are called separable if there exists a countable base for T . In the spaces (k , B k ), k ≥ 1, the “usual” topology Tk is the one with base the class of all finite open intervals (rectangles) or only the class of all open intervals (rectangles) with rational endpoints. This second base is countable and the topology Tk and the space (k , Tk ) are separable. Then, clearly, B k is generated by Tk (see Theorem 7, Definition 6, and the paragraph following it). Thus we have Corollary 2. Let X : (, A) → (n , B n ) be measurable and let f : (n , B n ) → (m , B m ) be continuous. Then f (X ) : → m is measurable (i.e., continuous functions of a random vector are random vectors). Proof.

Follows by the fact that Tm and Tn generate B m and B n , respectively.

This corollary implies that the usual operations applied on r.v.s, such as forming sums, products, or quotients, will give r.v.s. Now if X : → n , then X can be written as X = (X 1 , . . . , X n ). In connection with this we have Theorem 15. Let X = (X 1 , . . . , X n ) : (, A) → (n , B n ). Then X is a random vector (measurable function) if and only if X j , j = 1, . . . , n are r.v.s. Proof. Let Bi ∈ B, i = 1, . . . , n. Then X −1 (B1 × · · · × Bn ) = (X 1 , . . . , X n )−1 (B1 × · · · × Bn ) = (X 1 ∈ B1 ) ∩ · · · ∩ (X n ∈ Bn ) = [X 1−1 (B1 )] ∩ · · · ∩ [X n−1 (Bn )]. Thus, if X j , j = 1, . . . , n are r.v.s, then X −1 j (B j ) ∈ A for every j and hence so is

9

10


n

X −1 (B j ). So, if X j , j = 1, . . . , n, are measurable, so is X (by the definition of the product σ -field B n ). Next, consider the projection functions f j : n → such that f j (x1 , . . . , xn ) = x j , j = 1, . . . , n. It is known that f j , j = 1, . . . , n, are continuous, hence measurable. Then X j = f j (X ), j = 1, . . . , n, and the measurability of X implies the measurability of X j , j = 1, . . . , n. j=1

Let X be a r.v. Then the positive part of X , denoted by X + , and the negative part of X , denoted by X − , are defined as follows: X if X ≥ 0 0 if X ≥ 0, + − X = ; X = 0 if X < 0 −X if X < 0 . Then, clearly, X = X + − X − and |X | = X + + X − . Now as a simple application of the Corollary to Theorem 14, we show that both X + and X − are r.v.s and then, of course, so is |X |. To this end, take n = m = 1 and define f by: f (x) = x + , which is a continuous function of x, and similarly for f (x) = x − . Directly, the measurability of X + is established as follows. In order to prove the measurability of X + , it suffices to show that (X + )−1 ((−∞, x]) ∈ A for x ∈ . For x < 0, (X + ≤ x) = . For x = 0, (X + ≤ 0) = (X ≤ 0) ∈ A. For x > 0, (X + ≤ x) = (X + = 0) ∪ (0 < X + ≤ x) = (X ≤ 0) ∪ (0 < X ≤ x) = (X ≤ x) ∈ A. (Recall that for a sequence {xn } of real numbers, and as n → ∞: xn = x¯ if for every ε > 0 there exists n(ε) > 0 (1) lim sup xn or lim n n

integer such that n ≥ n(ε) implies xn ≤ x¯ + ε, and xn > x¯ − ε for at least one n ≥ n(ε). (2) limn inf xn or lim xn = x if for every ε > 0 there exists n(ε) > 0 n

integer such that n ≥ n(ε) implies xn ≥ x − ε and xn < x + ε for at least one n ≥ n(ε). Also, f (3) lim xn = infn supi≥n xi = infn yn , yn de = supi≥n xi , n

so that yn ↓ and set infn yn de f

= limn yn = x. ¯ de f

(4)

lim xn = supn infi≥n xi = supn z n , z n = infi≥n xi , n

so that z n ↑ and set supn z n de f

= limn z n = x. ¯ If x ≥ x, ¯ For every n ≥ 1, z n ≤ yn so that supn z n ≤ infn yn or equivalently x ≤ x. then the common value x = x¯ = x is the limn of xn .)


Next let X n , n ≥ 1, be r.v.s. Then define the following mappings (which are assumed to be finite). The sup and inf are taken over n ≥ 1 and all limits are taken as n → ∞. ⎫ : supn X n ω = supn X n (ω) supn X n ⎪ ⎪ ⎪ ⎪ ⎪ infn X n : infn X n ω = infn X n (ω) ⎪ ⎪ ⎬ lim sup X n or lim X n : lim sup X n (ω) = lim sup X n (ω) ⎪ ω ∈ . ⎪ n n n ⎪ ⎪ n ⎪ ⎪ lim inf X n or lim X n : lim inf X n (ω) = lim inf X n (ω) ⎪ ⎭ n

n

n

n

Then lim infn X n ≤ lim supn X n and if lim infn X n = lim supn X n , this defines the mapping limn X n . Then we have the following: If X n , n ≥ 1, are r.v.s, then the mappings just defined are also ∞ Proof. We have (supn X n ≤ x) = (X n ≤ x, n ≥ 1) = n=1 (X n ≤ x) ∈ A. Thus supn X n is a r.v. Now infn X n = −sup(−X n ) and then the measurability of supn X n implies the measurability of infn X n . Next, lim supn X n = infn (sup j≥n X j ). Thus, if Yn = sup j≥n X j , then Yn , n ≥ 1, are r.v.s and then so is the infn Yn . Finally, lim inf n X n = −lim supn (−X n ), and then the previous result implies the measurability of lim infn X n . The measurability of limn X n , if the limit exists, is an immediate consequence of the last two results. _ ¯ B ), the extended Borel real line, is an A measurable mapping X on (, A) into (, extended r.v. Then Theorem 16 still holds true if the operations applied on X n , n ≥ 1, produce extended r.v.s. Theorem 16. r.v.s.

j ∈ I } be a Definition 12. Consider the measurable space (, A) and let {A j , collection of sets in A such that Ai ∩ A j = , i, j ∈ I , i = j, and j A j = . Then this collection is called a (measurable) partition of . The partition is finite if I is a finite set and infinite otherwise. partition of , Definition 13. Let {A j , j = 1, . . . , n} be a (finite, measurable) n α and define the mapping X : → as follows: X = j=1 j I A j , such that α j ∈ , j = 1, . . . , n (which may be assumed to be distinct). Then X is called a infinite measurable) partition of , simple r.v. If {A j , j = 1, 2, . . .} is a (countably then the mapping X : → such that X = ∞ j=1 α j I A j , α j ∈ , j = 1, 2, . . . (which may be assumed to be distinct) is called an elementary r.v. Remark 14.

By I A we denote the indicator of the set A; i.e., 1 if ω ∈ A, I A (ω) = 0 if ω ∈ Ac .

It is evident that simple and elementary r.v.s are indeed r.v.s. What is more important, however, is that some kind of an inverse of this statement also holds true. More precisely,

11

12


Theorem 17. Proof.

Every r.v. is the pointwise limit of a sequence of simple r.v.s.

Consider the r.v. X , the interval [−n, n), and define the sets: j −1 j , j = −n2n + 1, −n2n + 2, . . . , n2n , An j = ≤ X < 2n 2n A n = (X < −n), A

n = (X ≥ n), n = 1, 2, . . .

Then, clearly, {An j , j = −n2n + 1, . . . , n2n , A n , A

n } is a (measurable) partition n j−1 of . Thus, if we define X n by X n = n2 j=−n2n +1 2n I An j + (−n)I A n + n I A

n , then X n is a simple r.v. We are going to show next that X n (ω) → X (ω) for every ω ∈ . n→∞

Let ω ∈ . Then there exists n o = n o (ω) such that |X (ω)| < n o . It is asserted that ω ∈ An j for n ≥ n o , some j = −n2n + 1, . . . , n2n . This is so because for j n n n ≥ n o , [−n, n) ⊇ [−n o , n o ) and the intervals [ j−1 2n , 2n ), j = −n2 + 1, . . . , n2 form a partition of [−n, n). Let that ω ∈ An j(n) . Then j(n)−1 ≤ X (ω) < j(n) 2n 2n . But j(n)−1 then X n (ω) = 2n so that |X n (ω) − X (ω)|
0. For each j, it follows from the definition of μ∗ (A j ) that there exists a covering ∞

ε > μ(A jk ). (2.2) 2j k=1

Now, from A j ⊆ ∞ A jk , j = 1, 2 . . ., it follows that ∞ Aj ⊆ ∞ k=1 j=1 j=1

∞

∞ k=1 A jk ; i.e., {A jk , j, k = 1, 2, . . .} is a covering of j=1 A j . Hence ⎛ ⎞ ∞ ∞ ∞

μ∗ ⎝ Aj⎠ ≤ μ(A jk ). (2.3) A jk ∈ F, k = 1, 2, . . . , such that μ∗ (A j ) +

j=1

From (2.2), we have ∞ ∞ j=1 k=1

μ(A jk ) ≤

∞ j=1

j=1 k=1

⎛

⎞ ∞ 1 μ∗ (A j ) + ε ⎝since = 1⎠ . 2j

(2.4)

j=1

From (2.3) and (2.4), we get ⎛ ⎞ ∞ ∞

Aj⎠ ≤ μ∗ (A j ) + ε. μ∗ ⎝ j=1

j=1

Letting ε → 0, we get the desired result. (iii) Since μ is σ -finite on F, there exists a partition {A j , j = 1, 2, . . .} of with A j ∈ F, j = 1, 2, . . ., such that μ(A j ) < ∞, j = 1, 2, . . . But A j ∈ F implies μ∗ (A j ) = μ(A j ), j = 1, 2, . . ., by (i). Thus μ∗ is also σ -finite. (iv) Finally, μ∗ () = μ() < ∞, by (i), since ∈ F. Theorem 3 exhibits the existence (and provides the construction) of an outer measure, namely μ∗ . Then we may denote μ∗ by μo . This outer measure μo is said to be induced on P() by μ defined on F. Definition 6. Let μo be an outer measure. Then a set A ⊆ is said to be μo measurable, if for every D ⊆ , we have: μo (D) = μo (A ∩ D) + μo (Ac ∩ D) (i.e., μo is additive for A ∩ D and Ac ∩ D).

Ω

A

Ac D

2.2 Outer Measures

Remark 6. (i) Since D = (A ∩ D) ∪ (Ac ∩ D) implies μo (D) ≤ μo (A ∩ D) + μo (Ac ∩ D), in order to check μo -measurability for A, it suffices to check that μo (D) ≥ μo (A ∩ D) + μo (Ac ∩ D)

for every D ⊆ .

(ii) There are μo -measurable sets. In fact and are such sets, because D = ( ∩ D) ∪ (c ∩ D) , and μo ( ∩ D) = μo (D), μo (c ∩ D) = μo () = 0, so that μo (D) = μo ( ∩ D) + μo (c ∩ D), and similarly for the set. Theorem 4.

Let μo be an outer measure. Then

(i) The class Ao of μo -measurable sets is a σ -field. (ii) μo , restricted on Ao , is a measure.

Proof. (i) We first prove that Ao is a field. Let A ∈ Ao . Then μo (D) = μo (A ∩ D) + μo (Ac ∩ D) = μo (Ac ∩ D) + μo [(Ac )c ∩ D] for every D ⊆ , and this shows that Ac ∈ Ao . Next, let A, B ∈ Ao . To show that (A ∩ B) ∈ Ao. Since B ∈ Ao , we get by writing μo (D) = μo (B ∩ D) + μo (B c ∩ D) and taking D to be A ∩ D and Ac ∩ D, successively, o μ (A ∩ D) = μo [B ∩ (A ∩ D)] + μo [B c ∩ (A ∩ D)], μo (Ac ∩ D) = μo [B ∩ (Ac ∩ D)] + μo [B c ∩ (Ac ∩ D)], so that μo (A ∩ D) + μo (Ac ∩ D) = μo [B ∩ (A ∩ D)] + μo [B c ∩ (A ∩ D)] + μo [B ∩ (Ac ∩ D)] + μo [B c ∩ (Ac ∩ D)].

(2.5)

But A ∈ Ao implies μo (D) = μo (A∩ D)+μo (Ac ∩ D). Taking this into consideration and the fact that μo is finitely subadditive, (2.5) becomes μo (D) ≥ μo (A ∩ B ∩ D) + μo [(A ∩ B c ∩ D) + (Ac ∩ B ∩ D) +(Ac ∩ B c ∩ D)]. Now (A ∩ B c ∩ D) + (Ac ∩ B ∩ D) + (Ac ∩ B c ∩ D) = D ∩ [(A ∩ B c ) + (Ac ∩ B) + (Ac ∩ B c )], = D ∩ [(A B) + (A ∪ B)c ] = D ∩ (A ∩ B)c . Therefore (2.6) becomes μo (D) ≥ μo [(A ∩ B) ∩ D] + μo [(A ∩ B)c ∩ D] so that (A ∩ B) ∈ Ao . So Ao is a field.

(2.6)

25

26

CHAPTER 2 Definition and Construction

o o Finally, 2, . . ., and set

∞A j ∈ A , j = 1,

∞ we prove that A is a σo -field. Let A = j=1 A j . To show that A ∈ A . Since j=1 A j = A1 + (Ac1 ∩ A2 ) + · · · + (Ac1 ∩ · · · ∩ Acn−1 ∩ An ) + · · · and (Ac1 ∩ · · · ∩ Acn−1 ∩ An ) ∈ Ao , n = 2, 3, . . ., by the factthat Ao is a field, it suffices to assume that the A j s are pairwise disjoint. Set Bn = nj=1 A j , B0 = . Then Bn ∈ Ao , n = 1, 2, . . ., and therefore (2.7) μo (D) = μo (Bn ∩ D) + μo Bnc ∩ D for every D ⊆ .

Next, An ∈ Ao . Thus, by writing μo (D) = μo (An ∩ D) + μo (Acn ∩ D) and taking D to be Bn ∩ D, we have μo (Bn ∩ D) = μo [An ∩ (Bn ∩ D)] + μo Acn ∩ (Bn ∩ D) , = μo (An ∩ D) + μo (Bn−1 ∩ D), since An ⊆ Bn and Acn ∩ Bn = Acn ∩ (A1 + · · · + An ) = A1 + · · · + An−1 = Bn−1 . the same way with That is, μo (Bn ∩ D) = μo (An ∩ D)+μo (Bn−1 ∩ D). Working in o μ (Bn−1 ∩ D), etc. (or by using induction), we get μo (Bn ∩ D) = nj=1 μo (A j ∩ D). Then (2.7) becomes as follows: n

μo (D) =

μo (A j ∩ D) + μo Bnc ∩ D .

j=1

But μo (Bnc ∩ D) ≥ μo (Ac ∩ D), since A ⊇ Bn or equivalently Ac ⊆ Bnc , and μo is nondecreasing. Thus μ (D) ≥ o

n

μo (A j ∩ D) + μo (Ac ∩ D).

j=1

Letting n → ∞, we get μo (D) ≥

∞

μo (A j ∩ D) + μo (Ac ∩ D),

j=1

⎤ ∞ ≥ μo ⎣ (A j ∩ D)⎦ + μo (Ac ∩ D), ⎡

j=1

= μ (A ∩ D) + μo (Ac ∩ D). o

Then A ∈

Ao ,

(2.8)

and this completes the proof of part (i).

(ii) Consider the first line on the right-hand side in inequality (2.8): μo (D) ≥ ∞ o o c j=1 μ (A j ∩ D) + μ (A ∩ D), D ⊆ , and set A instead of D. Then μo (A) ≥

∞

μo (A j ∩ A) + μo () =

j=1

since A j ∩ A = A j for all j.

∞ j=1

μo (A j ),

2.3 The Carathéodory Extension Theorem

Since the opposite inequality is always true, by sub-σ -additivity of μo , the proof is completed. (Observe that A0 may be contained strictly in P(); see Exercise 28(ii) in this chapter.) In the following section, an outer measure will be instrumental in extending a given measure from a field to the σ -field generated by it.

2.3 The Carathéodory Extension Theorem This section is devoted to the discussion of the Carathéodory extension theorem, which provides the basis for the construction and extension of measures. Theorem 5 (Carathéodory Extension Theorem). Let μ be a measure on a field F. Then (i) μ can be extended to the σ -field A generated by F. (ii) If μ is finite on F, then the extension is unique and finite. (iii) If μ is σ -finite on F, then the extension is unique and σ -finite.

Definition 7. The unique finite (σ -finite) extension of a finite (σ -finite) measure μ on F to A = σ (F) is called the Carathéodory extension. Proof of Theorem 5. (i) Let μ∗ be the set function defined just prior to Theorem 3. Then we saw that μ∗ is an outer measure on P(), is an extension of μ on F, and is σ -finite or finite, if μ is so, respectively. Also, μ∗ is a measure on the σ -field A∗ of μ∗ -measurable sets, by Theorem 4. Then, all we have to prove is that A ⊆ A∗ , or just that F ⊆ A∗ . Let A ∈ F and D ⊆ . To show μ∗ (D) ≥ μ∗ (A ∩ D) + μ∗ (Ac ∩ D). From the definition of μ∗ , for ε > 0, there exists a covering {A j , j = 1, 2, . . .} of D in F such that ∞ ∗ μ (D) + ε > μ(A j ). (2.9) j=1

Now μ(A j ) = μ[(A ∩ A j ) + (Ac ∩ A j )] = μ(A ∩ A j ) + μ(Ac ∩ A j ) = μ∗ (A ∩ A j ) + μ∗ (Ac ∩ A j ), since μ and μ∗ agree on F. Thus, (2.9) becomes μ∗ (D) + ε >

∞

μ∗ (A ∩ A j ) +

j=1

∞

μ∗ (Ac ∩ A j ).

j=1

(2.10)

∞ Next, ∞ D so that ( ∞ j=1 A j ⊇ j=1 A j ) ∩ A = j=1 (A ∩ A j ) ⊇ A ∩ D and

∞ c ∩ A ) ⊇ Ac ∩ D. Thus, (A ( j=1 A j ) ∩ Ac = ∞ j j=1 ∞ j=1

μ∗ (A ∩ A j ) ≥ μ∗

∞

(A ∩ A j ) ≥ μ∗ (A ∩ D),

j=1

27

28


∞ j=1

μ∗ (Ac ∩ A j ) ≥ μ∗

∞

(Ac ∩ A j ) ≥ μ∗ (Ac ∩ D).

j=1

Adding up these relationships and using also (2.10), we get μ∗ (D) + ε ≥ μ∗ (A ∩ D) + μ∗ (Ac ∩ D). Since this is true for every ε > 0, we get then μ∗ (D) ≥ μ∗ (A ∩ D) + μ∗ (Ac ∩ D). This proves that A is μ∗ -measurable and hence A ∈ A∗ . (ii) That the extension is finite has been seen in Theorem 3. So, all we have to do is to prove uniqueness. Let μ1 be the above seen extension and let μ2 be any other extension. Define M ⊆ A as follows: M = {A ∈ A; μ1 (A) = μ2 (A)}. We shall show that M is a monotone class and equals A. First F ⊆ M, since μ1 = μ2 = μ on F. Let now {An } be a monotone sequence of sets in M. Then μ1 (limn→∞ An ) = limn→∞ μ1 (An ) = limn→∞ μ2 (An ) = μ2 (limn→∞ An ) (by the finiteness of μ1 and μ2 ); i.e., μ1 (limn→∞ An ) = μ2 (limn→∞ An ) and hence limn→∞ An ∈ M. Thus, M is a monotone class. Then M contains the minimal monotone class over F that coincides with A (by Theorem 6 in Chapter 1). Thus, μ1 , μ2 coincide on A. (iii) Again, the σ -finiteness of the extension follows from Theorem 3, and we only have to establish uniqueness. The σ -finiteness of μ implies the existence of a partition {A j , j = 1, 2, . . .} of in F such that μ(A j ) < ∞, j = 1, 2, . . .. For each A j , consider the classes F A j = {A j ∩ B; B ∈ F}, A A j = {A j ∩ B; B ∈ A}. Then F A j is a field and A A j is a σ -field. Furthermore, A A j is the σ -field generated by F A j (see Exercises 8 and 9 in Chapter 1). Let μ1 , μ2 be as in (ii). Then μ1 = μ2 , and finite on A A j by (ii). Next, let A ∈ A. Then A = ∞ j=1 (A ∩ A j ), while A ∩ A j ∈ A A j , j = 1, 2, . . ., so that μ1 (A ∩ A j ) = μ2 (A ∩ A j ), j = 1, 2, . . . Thus μ1 (A) = ∞ ∞ j=1 μ1 (A ∩ A j ) = j=1 μ2 (A ∩ A j ) = μ2 (A). Therefore μ1 = μ2 on A. Special cases. (1) Let = and let C be the class of all finite sums of intervals in . Then C is a field (by Exercise 7(ii) in Chapter1), and B = σ (C). Let μ(I ) = length of I , where I is an interval, and let μ(A) = nj=1 μ(I j ), if A ∈ C and hence A = nj=1 I j , I j , j = 1, 2, . . . , n, intervals. The set function μ is σ -finite, since, for ∞ example, = ∞ n=0 (−n −1, −n]+(0, 1)+ n=1 [n, n +1) and μ((−n −1, −n]) = μ([n, n + 1)) = 1 (finite). Then, provided that μ is well defined and a measure on C—which we will show later on (Theorem 7)—the unique extension of μ on B is called the Lebesgue measure. Let us denote it by λ. (2) For n ≥ 2, let C be the class of all finite sums of rectangles in n . Then C is a field and B n = σ (C) (by Theorem 7 in Chapter 1 and its extension). If

2.3 The Carathéodory Extension Theorem

B = A1 × · · · × An , A j Borel setsin , j = 1, . . . , n, define μ(B) as follows: μ(B) = nj=1 λ(A j ), and if E = mj=1 B j , B j , j = 1, . . . , m, rectangles in n , define μ(E) by μ(E) = mj=1 μ(B j ). It is easily seen that μ is σ -finite on C. Actually, there exists a denumerable partition of n by rectangles in n that are cartesian products of intervals in (recall that a set is denumerable if it has the same cardinality as the set of integers). Then, provided μ is well defined and a measure on C (that this is so is seen as a straightforward generalization of Theorem 7), the unique extension of μ on B n is called the n-dimension Lebesgue measure. Let us denote it by λn . Remark 7. (i) Special case 1 indicates why the natural original class C on which a measure μ is defined is a field. There is also another reason, and that is that, if the class C is not a field, then the extension of μ to the minimal σ -field over C need not be unique although μ may be finite (σ -finite) on C. (ii) If μ on C is not finite (σ -finite), then the Carathéodory extension on A = σ (C) need not be unique. (iii) An extension of μ to A = σ (C) may be σ -finite while μ is not so on C. In connection with these remarks, see also Exercises 12 and 13. Example 2 demonstrates the futility of starting out with an arbitrary class of sets in defining a measure on a field or a σ -field. Example 1 is a prelude to Example 2. Example 1. Let Fi , i = 1, 2, be fields of subsets of a set and define the classes Ci , i = 1, 2, 3, by: C1 = F1 ∪ F2 = {A ⊆ ;

A ∈ F1 or A ∈ F2 },

so that C1 is closed under complementation; C2 = {all finite intersections of members of C1 = {A ⊆ ; A = A1 ∩ · · · ∩ Am , Ai ∈ C1 , i = 1, . . . , m, m ≥ 1}; C3 = {all finite unions of members of C2 } n

= {A ⊆ ; A = Ai with Ai ∈ C2 , i = 1, . . . , n, n ≥ 1} = {A ⊆ ;

A=

i=1 n

Ai with Ai = Ai1 ∩ · · · ∩ Aim i , Ai1 , . . . , Aim i ∈ C1 ,

i=1

m i ≥ 1 integer, i = 1, . . . , n, n ≥ 1}. Set C3 = F. Then F is a field. This is so by Exercise 41(i) in Chapter 1, where the role of C and F1 is played by C1 , the role of F2 is played by C2 , and the role of F3 (= F) is played by C3 (= F).

29

30


Example 2. In reference to Example 1, take F1 = {, A, Ac , } and F2 = {, B, B c , }(A = B, A ∩ B = ), so that C = C1 = F1 ∪ F2 = {, A, Ac , B, B c , }. Then, as is easily seen, C2 = {, A, Ac , B, B c , A ∩ B, A ∩ B c , Ac ∩ B, Ac ∩ B c , }; also, C3 = {, A, Ac , B, B c , A ∩ B, A ∩ B c , Ac ∩ B, Ac ∩ B c , A ∪ B, A ∪ B c , Ac ∪ B, Ac ∪ B c , (A ∩ B) ∪ (Ac ∩ B c ), (A ∩ B c ) ∪ (Ac ∩ B), }; as it can be verified. Set C3 = F. Then F is a field on account of Example 1. Alternatively, the assertion is checked directly, since F is closed under complementation and under the union of any two of its members (see Exercise 35). Clearly, F is the field generated by C(= C1 = F1 ∪ F2 ). This is so by Exercise 41(ii) cited above or by direct considerations. On the class C, define μi , i = 1, 2, as follows: μi () = 0, μi (A) = 0.40, μi (B) = 0.35, μi (Ac ) = 0.60, μi (B c ) = 0.65, μi () = 1, so that μi , i = 1, 2, are (probability) measures on C. Next, extend these measures from C to the field F generated by C in the following manner: μ2 (A ∪ B) = 0.60, μ1 (A ∪ B) = 0.50 μ1 (A ∪ B c ) = 0.90 μ1 (Ac ∪ B) = 0.85 μ1 (Ac ∪ B c ) = 0.75

μ2 (A ∪ B c ) = 0.80, μ2 (Ac ∪ B) = 0.75, μ2 (Ac ∪ B c ) = 0.85,

μ1 (A ∩ B) = 0.25 μ1 (A ∩ B c ) = 0.15

μ2 (A ∩ B) = 0.15, μ2 (A ∩ B c ) = 0.25,

μ1 (Ac ∩ B) = 0.10 μ1 (Ac ∩ B c ) = 0.50

μ2 (Ac ∩ B) = 0.20, μ2 (Ac ∩ B c ) = 0.40

.

The assigned values are legitimate, because they satisfy the ordering property (C ⊂ D implies μi (C) ≤ μi (D), i = 1, 2) and the additivity property. Furthermore, whereas μ1 and μ2 coincide on C, they do not coincide on F. That is, a measure defined on a class C (which is not a field) may assume more than one extension to a measure on the field generated by C. The same holds if F is replaced by a σ -field A. The interested reader may find a rather extensive treatment of measure theory in the reference Vestrup (2003).

2.4 Measures and (Point) Functions Let μ be a measure on B, the Borel σ -field in such that μ(finite interval) < ∞ (the Lebesgue measure, for example, does this). Then for a constant c, define a function

2.4 Measures and (Point) Functions

Fc = F : → as follows: F(x) =

c + μ((0, x]) if x ≥ 0, c − μ((x, 0]) if x < 0.

(Then F(0) = c, since (0, 0] = and μ() = 0.) Then we have the following easy theorem. Theorem 6.

Let F be defined as above. Then F is

(i) Nondecreasing. (ii) Continuous from the right.

Proof. (i) Let 0 ≤ x1 < x2 . Then F(x1 ) = c + μ((0, x1 ]) ≤ c + μ((0, x2 ]) = F(x2 ). Next, let x1 < 0 ≤ x2 . Then F(x1 ) = c − μ((x1 , 0]) ≤ c + μ((0, x2 ]) = F(x2 ). Finally, let x1 < x2 < 0. Then F(x1 ) = c−μ((x1 , 0]) ≤ c−μ((x2 , 0]) = F(x2 ). (ii) Let x ≥ 0 and choose xn ↓ x as n → ∞ here and in the sequel. Then (0, xn ] ↓ (0, x] so that μ((0, xn ]) ↓ μ((0, x]), or c + μ((0, xn ]) ↓ c + μ((0, x]), or equivalently, F(xn ) ↓ F(x). Next, let x < 0, and pick xn such that xn ↓ x. Then (xn , 0] ↑ (x, 0] so that μ((xn , 0]) ↑ μ((x, 0]), or equivalently, −μ((xn , 0]) ↓ −μ((x, 0]), or c − μ((xn , 0]) ↓ c − μ((x, 0]), or equivalently, F(xn ) ↓ F(x). Thus, we proved that a measure μ on B with the property that μ(finite interval) < ∞ defines a class of (point) functions on → that are nondecreasing and continuous from the right. Each such function is called a distribution function (d.f.). If μ is finite, then each F is bounded. In particular, if μ is a probability measure and, if we take c = μ(−∞, 0], then F is a d.f. of a r.v. X (i.e., in addition to (i) and (ii), F(−∞) = lim x→−∞ F(x) = 0, F(∞) = lim x→∞ F(x) = 1). Now we will work the other way around. Namely, we will start with any function F that is nondecreasing and continuous from the right, and we will show that such a function induces a measure on B. To this end, define the class C ⊂ B as follows: C = ∪ {(α, β]; α, β ∈ , α < β}, and on this class, we define a function as follows: de f ((α, β]) = (α, β) = F(β) − F(α), () = 0. Then we have the following easy lemma. Lemma 1.

Let C and be defined as above. Then

(i) ≥ 0. (ii) (α, β) ↓ 0 as β ↓ α. (iii) If α1 ≤ α2 ≤ · · · ≤ αn , then n−1 j=1 (α j , α j+1 ) = (α1 , αn ) = F(αn ) − F(α1 ). (iv) is nondecreasing.

31

32


Proof. (i) Obvious from the nondecreasing property of F. (ii) Obvious by the continuity from the right of F. n−1 n−1 (iii) j=1 (α j , α j+1 ) = j=1 [F(α j+1 ) − F(α j )] = F(αn ) − F(α1 ) = (α1 , αn ). (iv) If (α1 , α2 ] ⊃ (α3 , α4 ], then by (iii) and (i), (α1 , α2 ) = (α1 , α3 ) + (α3 , α4 ) + (α4 , a2 ) ≥ (α3 , α4 ). Next, we have the following less obvious lemma. Lemma 2. The function on C is a measure. That is, () = 0, ((α, β]) ≥ 0, ∞ (α , β ], it holds ((α, β]) = and for (α, β] = ∞ j j=1 j j=1 ((α j , β j ]). Proof. Since () = 0, and ≥ 0 (and is nondecreasing), all that remains ∞ (α , β ], then (α, β) = to prove is that is σ -additive; i.e., if (α, β] = j j j=1 j (α j , β j ). Consider the n intervals (α j , β j ], j = 1, . . . , n. These intervals are nonoverlapping and we may rearrange them (α j1 , β j1 ], (α j2 , β j2 ], . . . , (α jn , β jn ] so that n

α j1 < β j1 ≤ α j2 < β j2 ≤ · · · ≤ α jn < β jn .

Then i=1 (α ji , β ji ) ≤ (α j1 , β j1 ) + (β j1 , α j2 ) + · · · + (α jn , β jn ) = is ≤ (α, β), since (α j1 , β jn ] ⊆ (α, β]. So, (α j1 , β jn ), by Lemma n 1 (iii), and this (α ji , β ji ) = nj=1 (α j , β j ) ≤ (α, β), which implies for every finite n, i=1 ∞

(α j , β j ) ≤ (α, β).

(2.11)

j=1

We then have to establish the reverse inequality; i.e., ∞

(α j , β j ) ≥ (α, β).

(2.12)

j=1

Consider (α, β], choose 0 < ε < β − α, and look at the interval [α + ε, β]. By Lemma 1 (ii), (β j , β j + δ) ↓ 0 as δ ↓ 0. Thus, there exists δ j > 0 such that (β j , β j + δ j )
β,

α j,i−1 < α ji < β j,i−1 + δ j,i−1 < β ji + δ ji , i = 2, . . . , m. Now look at the following intervals: (α j1 , β j1 + δ j1 ], (β j1 + δ j1 , β j2 + δ j2 ], . . . , (β j,m−1 + δ j,m−1 , β jm + δ jm ] (see the following picture). αj1 (| ( | α α+ε

αj2 | ]( βj1 +δj1

αji | ( ] ] βj2 +δj2 . . . βj,i−1 +δj,i−1 βj,i +δj,i . . .

αjm ] ] | ( βj,m−1 +δj,m−1 β βj,m+δj,m

These intervals are nonoverlapping and their sum is (α j1 , β jm +δ jm ] ⊃ (α+ε, β]. Then (α + ε, β) ≤ (α j1 , β jm + δ jm ) = (α j1 , β j1 + δ j1 ) +

m

(β j,i−1 + δ j,i−1 , β ji + δ ji )

i=2

≤

m

(α ji , β ji + δ ji ),

i=1

because (β j,i−1 +δ j,i−1 , β ji +δ ji ] ⊂ (α ji , β ji +δ ji ] and is nondecreasing. That is, (α + ε, β) ≤

m

(α ji , β ji + δ ji ).

i=1

Next,

m

(α ji , β ji + δ ji ) ≤

j=1

∞

(α j , β j + δ j ),

j=1

(since (α ji , β ji + δ ji ], i = 1, . . . , m are only m of the intervals (α j , β j + δ j ], j = 1, 2, . . .) = ≤ =

∞

[(α j , β j ) + (β j , β j + δ j )]

j=1 ∞

(α j , β j ) +

j=1 ∞

ε (by (2.13)), 2j

(α j , β j ) + ε;

j=1

(α + ε, β) ≤

∞ j=1

(α j , β j ) + ε.

i.e.,

33

34


But (α+ε, β) = F(β)− F(α+ε) and hence F(β)− F(α+ε) ≤ ∞ j=1 (α j , β j )+ ε. Letting ε ↓ 0 and using Theorem 6(ii), we get then F(β) − F(α) = (α, β) ≤ ∞ (α , β ) which is (2.12). Therefore (2.11) and (2.12) show that is a measure j j j=1 on C. Theorem 7. Let F : → be a d.f. Then F uniquely determines a measure ν on B such that ν((α, β]) = F(β) − F(α), called the measure induced by F. If F is bounded, the measure is finite, and, in particular, if F is a d.f. of a r.v., then the measure is a probability measure. Proof. In terms of F, define as was done just prior to Lemma 1. Then consider the class C = C ∪ {(−∞, α]; α ∈ } ∪ {(β, ∞); β ∈ } and let C be the class of all finite sums in C . Then the class C is a field generating B (by Exercise 7(i) in Chapter 1), and, clearly, the elements of C are sums of countably many elements of C; i.e., if A ∈ C , then A = j I j with I j ∈ C, j = 1, 2, . . . On C , we define a function ν as follows: ν(A) = (I j ), if A = I j , I j ∈ C, j = 1, 2, . . . j

j

We will first show that ν is well defined; i.e., if A = i Ii and A = j I j , then (Ii ) = Clearly, Ii ∩ I j ∈ C, i, j = 1, 2, . . . Next, Ii = j (I j )(= ν(A)). i j Ii ∩ I j , since Ii ⊆ A = j I j so that (Ii ) = ( j Ii ∩ I j ) = j (Ii ∩ I j ), since is a measure on C (by Lemma 2). In a similar fashion, (I j ) = i (Ii ∩ I j ). Hence I j = I j ∩ Ii = Ii ∩ I j = (Ii ), j

j

i

i

j

i

as was asserted. Clearly, ν coincides with on C, and we will next show that ν is a measure on Let A j ∈ C , j = 1, 2, . . ., such C . It suffices to prove σ -additivity. that Ai ∩ A j = . To show that ν(A) = A ∈ C , i = j, and let A = j j j ν(A j ). Since A j ∈ C , j = 1, 2, . . ., we have that A j = i I ji with I ji ∈ C, i = 1, 2, . . . , j, i = 1, 2 . . .} form a partition of A. Thus ν(A) = Clearly, {I ji j,i (I ji ) = j i (I ji ) = j ν(A j ). The remaining assertions are obvious; e.g., uniqueness follows from Theorem 5 since ν is at least σ -finite. Remark 8. The preceding derivations also justify the questions left open in the two special cases discussed earlier (special cases right after the proof of Theorem 5). The d.f. F which induces the Lebesgue measure λ is defined by F : → , F(x) = x. Remark 9. If X is a r.v., its probability distribution (or just distribution) is usually denoted by PX and is defined by: PX (B) = P(X ∈ B), B ∈ B, so that PX is a probability measure on B. Next, if FX is the d.f. of X , then FX (x) = PX ((−∞, x]), x ∈ , so that FX is determined by PX . Theorem 7 shows that the converse is also true; that is, FX uniquely determines PX .


Exercises. 1. If is countable and μ is defined on P() by: μ(A) = number of points of A, show that μ is a measure. Furthermore, μ is finite or σ -finite, depending on whether is finite or denumerable, respectively. (This measure is called the counting measure.) 2. Refer to the field C of Example 4 in Chapter 1 and on C, define the set function P as follows: P(A) = 0 if A is finite, and P(A) = 1 if Ac is finite. Then show that (i) P is finitely additive. (ii) If is denumerable, P is not σ -additive. (iii) If is uncountable, P is σ -additive and a probability measure. 3. Refer to the σ -field C of Example 8 in Chapter 1 and on C, define the function P as follows: P(A) = 0 if A is countable, and P(A) = 1 if Ac is countable. Then show that P is a probability measure. 4. Let An , n = 1, 2, . . . be events in the probability space (, A, P) such that P(An ) = 1 for all n. Then show that P(∩∞ n=1 An ) = 1. 5. Let {Ai , i ∈ I } be an uncountable collection of pairwise disjoint events in the probability space (, A, P). Then show that P(Ai ) > 0 for countably many Ai s only. Hint: If In = {i ∈ I ; P(Ai ) > n1 }, then the cardinality of In is ≤ n − 1, n ≥ 2, and I0 = {i ∈ I ; P(Ai ) > 0} = ∪n≥2 In . 6. Let be an infinite set (countable or not) and let A be the discrete σ -field. Let ω2 , . . .} ⊂ , and with each ωn , associate a nonnegative number {ω1 , pn (such that ∞ ωn ∈A pn . n=1 pn ≤ ∞). On A, define the set function μ by: μ(A) = Then show that μ is a measure on A. 7. In the measure space (, A, μ) a set A ∈ A is called an atom, if μ(A) > 0 and for any B ⊆ A with B ∈ A, it follows that μ(B) = 0, or μ(B) = μ(A). In reference to Exercise 6, identify the atoms of A. 8. In any measure space (, A, μ) and with any An ∈ A, n = 1, 2, . . ., show that μ lim inf An ≤ lim inf μ(An ); also, μ lim sup An ≥ lim sup μ(An ), n→∞

provided μ

n→∞

∞ j=n

n→∞

n→∞

A j < ∞ for some n.

In Exercises 9 and 10, show that the set function μ◦ defined on P() is, indeed, an outer measure. 9. is an arbitrary set, ω0 is a fixed point of , and μ◦ is defined by: μ◦ (A) = I A (ω0 ). 10. is the set of 100 points arranged in a square array of 10 columns, each with 10 points, and μ◦ is defined by: μ◦ (A) = number of columns that contain at least one point of A.

35

36


11. For an arbitrary set containing at least two points, consider the trivial field F = {, }, and let (the finite measure) μ be defined on F by: μ() = 0 and μ() = 1. (i) Determine the set function μ∗ on P(), as is given in the definition just prior to Theorem 3 (which μ∗ is, actually, an outer measure, by Theorem 3(ii)). (ii) Show that the σ -field of μ∗ -measurable sets, A∗ , say, is the trivial σ -field, so that A∗ is strictly contained in P(). 12. Let = {ω1 , ω2 , ω3 , ω4 }, let C = {, {ω1 , ω2 }, {ω1 , ω3 }, {ω2 , ω4 }, {ω3 , ω4 }, }, and define μ on C as follows: μ({ω1 , ω2 }) = μ({ω1 , ω3 }) = μ({ω2 , ω4 }) = μ({ω3 , ω4 }) = 3, μ() = 6, μ() = 0. Next, on P(), define the measures μ1 and μ2 by taking μ1 ({ω1 }) = μ1 ({ω4 }) = μ2 ({ω2 }) = μ2 ({ω3 }) = 1, μ1 ({ω2 }) = μ1 ({ω3 }) = μ2 ({ω1 }) = μ2 ({ω4 }) = 2. Then show that C is not a field. μ is a measure on C. Both μ1 and μ2 are extensions of μ (from C to P()). Construct the outer measure μ∗ (as is defined in Definition 5) by means of μ defined on C. (v) Conclude that μ∗ = μ1 = μ2 (so that, if the class C is not a field, the extension of (even a finite measure μ on C) need not be unique).

(i) (ii) (iii) (iv)

13. Let = {0, 1, 2, . . .}, let A = {1, 3, 5, . . .}, and let C = {, A, Ac , }. Let μ be the counting measure on C, and let μ1 , μ2 be defined on P() by μ1 (B) = the number of points of B, μ2 (B) = 2μ1 (B). Then show that C is a field. μ is not σ -finite on C. Both μ1 and μ2 are extensions of μ and are also σ -finite. Determine the outer measure μ∗ (as is defined in Definition 5) by showing that μ∗ (B) = ∞ whenever B = . (v) Show that the σ -field of μ∗ -measurable sets, A∗ say, is equal to P().

(i) (ii) (iii) (iv)

(From this example, we conclude that if μ is not σ -finite on the field C, then there need not be a unique extension. Also, there may be σ -finite extensions, such as μ1 and μ2 here, when the original measure on C is not σ -finite.) 14. Construct additional examples to illustrate the points made in Exercises 11(ii), 12, and 13.


15. Consider the measure space (, A, μ), and define the classes A∗ and A¯ as follows: A∗ = {A M; A ∈ A, M ⊆ N with N ∈ A and μ(N ) = 0}, A¯ = {A ∪ M; A ∈ A, M ⊆ N with N ∈ A and μ(N ) = 0}. Then show that (i) A M = (A − N ) ∪ [N ∩ (A M)]. (ii) A ∪ M = (A − N ) [N ∩ (A ∪ M)]. From parts (i) and (ii), conclude that ¯ (iii) A∗ = A. ¯ defined in Exercise 15, and show that A¯ (and 16. Refer to the classes A∗ and A, hence A∗ ) is a σ -field. 17. Refer to Exercise 15 and on A∗ , define μ∗ by: μ∗ (A M) = μ(A). (i) Show that μ(A − N ) = μ(A). By Exercise 15(ii), μ∗ (A ∪ M) = μ(A − N ). Therefore, by part (i), we may define μ∗ on A¯ by: μ∗ (A ∪ M) = μ(A). (ii) Show that μ∗ so defined is well defined; that is, if A1 ∪ M1 = A2 ∪ M2 , where Ai and Mi , i = 1, 2 are as in Exercise 15, then μ(A1 ) = μ(A2 ). (iii) Show that μ∗ is a measure on A¯ (and hence on A∗ , by Exercise 15(iii)). ¯ μ∗ ) ( and (, A∗ , μ∗ )) is called completion Remark: The measure space (, A, of (, A, μ). ¯ defined in Exercise 15 contains the class Cˆ 18. (i) Show that the class A∗ (= A) defined by Cˆ = {B ⊆ ; either B = A for some A ∈ A, or B ⊆ N for some N ∈ A with μ(N ) = 0}. (That is, Cˆ is taken from A by supplementing A with all subsets of all null sets in A; that is, sets of measure 0.) ˆ def ¯ (ii) Also, show that σ (C) = Aˆ = A∗ (= A). 19. If (, A, μ) is a measure space, by means of an example, show that there may be subsets M ⊂ N , where N are null sets in A, with M ∈ / A. (If it so happens, however, that for every M ⊆ N for every null N ∈ A, then M ∈ A, we say that A is complete with respect to μ.) 20. If μ◦ is an outer measure and A◦ is the σ -field of μ◦ -measurable sets, then show that A◦ is complete (with respect to μ◦ ). 21. Consider the measure space (, B, μ) where μ(finite interval) < ∞, and for c ∈ , let Fc : → be defined by c + μ((0, x]) x ≥ 0, Fc (x) = c − μ((x, 0]) x < 0. Then, by means of an example, show that Fc need not be left-continuous.

37

38


22. Consider the measurable space (, A), and let μ be a set function defined on A and taking nonnegative values, and such that μ() = c with 0 < c < ∞. Then show that μ cannot be additive. 23. Let μ1 , . . . , μn be σ -finite n measures on (, A) and set μ = μ1 + · · · + μn (in μi (A), A ∈ A). Then show that μ is also σ -finite. the sense that μ(A) = i=1 24. In the probability space (, A, P), show that B)] = P(A) + P(B) − 2P(A ∩ B). (i) P[(A ∩ B c ) ∪ (Ac ∩ n P(Ai ) − (n − 1). (ii) P(A1 ∩ · · · ∩ An ) ≥ i=1 25. Consider the probability space (, A, P), where = {ω1 , ω2 , ω3 , ω4 }, A is the discrete σ -field, and P is defined by P({ω1 }) =

1 1 1 3 , P({ω2 }) = , P({ω3 }) = , P({ω4 }) = . 6 3 5 10

For n = 1, 2, . . ., define An by: A2n−1 = {ω1 , ω2 }, A2n = {ω2 , ω3 }. Determine the: limn→∞ An , limn→∞ An and compute the probabilities: P( lim An ), P( lim An ), lim P(An ), lim P(An ). n→∞

n→∞

n→∞

n→∞

26. Let = {ω1 , ω2 , . . .} and let A be a σ -field of subsets of . Then show that (i) A = P() if and only if {ωi } ∈ A for all ωi . (ii) A measure μ on A(= P()) is determined by defining μ({ωi }), i ≥ 1. 27. Suppose that = {ω1 , ω2 , . . .}, and let A be the discrete σ -field. On A, define the set function μ by 0 if A is finite, μ(A) = ∞ if A is infinite. Then show that (i) μ is nonnegative, μ() = 0 and is finitely additive, but it is not σ -additive. (ii) = limn→∞ An for a sequence {An }, with An ⊆ An+1 , n ≥ 1, and μ(An ) = 0 (and therefore μ(Acn ) = ∞), n ≥ 1. 28. Let = {1, 2, . . .}, and for any A ⊆ , let a = sup A. On the discrete σ -field, define μ0 by ⎧ a ⎨ a+1 if A is finite, μ0 (A) = 0 if A = , ⎩ 1 if A is infinite. Then (i) Show that μ0 is an outer measure. (ii) Determine the class A◦ of μ◦ -measurable sets. 29. Recall that m is a median of a r.v. X if P(X ≤ m) ≥ 21 and P(X ≥ m) ≥ 21 . Then show that, if m is a median of the r.v. X , then −m is a median of the r.v. −X .


30. Recall that a r.v. X is said to be symmetric about 0, if X and −X have the same distribution. If X is symmetric (about 0), then show that 0 is a median of X ; i.e., P(X ≤ 0) ≥ 21 and P(X ≥ 0) ≥ 21 . 31. Let μ0 be an outer measure, and suppose that μ0 (A) = 0 for some A ⊂ . Then show that μ0 (A ∪ B) = μ0 (B) for every B ⊆ . 32. Consider the measure space (, A, μ), and suppose that A is complete with respect to μ. Let f , g : → be such that μ( f = g) = 0. Then show that, if one of f or g is measurable, then so is the other. 33. Consider the measure space (, A, μ), and let A be complete with respect to μ. Let A ∈ A with μ(A) = 0, and let f be an arbitrary function f : A → . Then f is measurable. 34. (i) If μ1 and μ2 are two measures on (, A), then show that μ = μ1 + μ2 is also a measure, where μ(A) = μ1 (A) + μ2 (A), A ∈ A. (ii) If at least one of the measures μ1 and μ2 is complete, then so is μ. 35. Refer to Example 2 and: (i) Show that the class C3 consists of the unions of any two members of the class C2 . (ii) Also, show that C3 is closed under complementation and the formation of the union of any two of its members. (iii) Conclude that C3 is a field, and, indeed, the field generated by C.

39

CHAPTER

3

Some Modes of Convergence of Sequences of Random Variables and their Relationships

In this short chapter, we introduce two basic kinds of convergence, almost everywhere convergence and convergence in measure, and we then investigate their relationships. The mutual versions of these convergences are also introduced, and they are related to the respective convergences themselves. Furthermore, conditions for almost sure convergence are established. Finally, the concept of almost uniform convergence is defined and suitably exploited.

3.1 Almost Everywhere Convergence and Convergence in Measure Consider the measure space (, A, μ) and let {X n }, n = 1, 2, . . . , X be r.v.s. Then Definition 1. We say that {X n } converges almost everywhere (a.e.) to X and write a.e.

X n → X , if X n (ω) → X (ω) except on a set N (A) such that μ(N ) = 0; i.e., n→∞

n→∞

a μ-null set. We also write μ(X n X ) = 0. In particular, if μ is a probability n→∞

measure, this convergence is called almost sure (a.s.) convergence. We say that {X n } converges mutually a.e., if μ(X m − X n 0) = 0, or equivalently, X n+ν − m,n→∞

a.e.

X n → 0 uniformly in ν ≥ 1 (see also Exercise 20). Then, by the Cauchy criterion n→∞ for sequences of numbers, {X n } converges a.e. to X , if and only if it converges mutually a.e. If we modify X n to X n on a null set Nn and X to X on null set N0 , de f a.e. then, since ∞ n=1 Nn ∪ N0 = N ∈ A and μ(N ) = 0, we have that: X n → X , if n→∞

a.e.

X n → X , {X n } n→∞

converges mutually a.e., if {X n } converges mutually a.e., and the preceding statement about equivalence of a.e. convergence and mutual convergence of {X n } is still valid. Of course, μ(X n = X n ) = μ(X = X ) = 0, n = 1, 2, . . . . μ

We say that {X n } converges in measure to X and write X n → X , if, n→∞ for every ε > 0, μ(|X n − X | ≥ ε) → 0. In particular, if μ is a probability measure, Definition 2.

n→∞

this convergence is called convergence in probability. We say that {X n } converges mutually in measure, if for every ε > 0, μ(|X m − X n | ≥ ε) → 0, or equivam,n→∞

lently, μ(|X n+ν − X n | ≥ ε) → 0 uniformly in ν ≥ 1 (see also Exercise 21). The n→∞

An Introduction to Measure-Theoretic Probability, Second Edition. http://dx.doi.org/10.1016/B978-0-12-800042-7.00003-7 Copyright © 2014 Elsevier Inc. All rights reserved.

41

42

CHAPTER 3 Some Modes of Convergence of Sequences

convergence is mutual convergence in probability, if μ is a probability measure. Obviously, these convergences remain intact when the X n s and X are modified as μ before. We will show later on (see Theorems 2 and 6) that X n → X , if and only if n→∞ {X n } converges mutually in measure. a.e. a.e. Clearly, if X n → X and X n → X , then μ(X = X ) = 0. This is also true for n→∞ n→∞ convergence in measure but is less obvious. So μ

Proof. so that

μ

Let X n → X and X n → X . Then μ(X = X ) = 0. n→∞ n→∞ We have X − X = (X − X n ) + (X n − X ) ≤ |X n − X | + X n − X ,

Theorem 1.

X − X ≥ ε ⊆ |X n − X | ≥ ε ∪ X n − X ≥ ε , and hence 2 2 ε ε μ X − X ≥ ε ≤ μ |X n − X | ≥ + μ X n − X ≥ → 0. 2 2 n→∞ Thus, μ( X − X ≥ ε) = 0, for every ε > 0 and hence μ X − X ≥ k1 = 0, k = 1, 2, . . .. But (X = X ) = ∞ X − X ≥ 1 . Thus μ(X = X ) ≤

k

k=1

1 μ X − X ≥ k

k

= 0; i.e., μ(X = X ) = 0.

Convergence in measure implies mutual convergence in measure. That is, Theorem 2. Proof.

μ

Let X n → X . Then {X n } converges mutually in measure.

n→∞

We have: |X m − X n | ≤ |X m − X | + |X n − X | .

Thus,

ε μ([|X m − X n | ≥ ε]) ≤ μ |X m − X | ≥ 2 ε → 0. +μ |X n − X | ≥ 2 m,n→∞

Remark 1. It is obvious that any one of the modes of convergences introduced so far is true for any subsequence of a given sequence. What we are going to do in the remaining part of this section is to find an expression for the set X n → X , then use this result to formulate a criterion for a.e. convergence n→∞ of X n to X . The set of convergence X n → X consists of all those ω ∈ for which: For n→∞

every ε > 0, there is an n = n(ε, ω) ≥ 1 (integer) such that, for all ν ≥ 1, it holds |X n+ν (ω) − X (ω)| < ε.

(3.1)

3.1 Almost Everywhere Convergence and Convergence in Measure

That (3.1) holds for all ν ≥ 1 means ω∈

∞

(|X n+ν − X | < ε) for some n ≥ 1, and every ε > 0.

(3.2)

ν=1

That (3.2) holds for some n ≥ 1 means ω∈

∞ ∞

(|X n+ν − X | < ε) for every ε > 0.

(3.3)

n=1 ν=1

That (3.3) holds for every ε > 0 means ω∈

∞ ∞

(|X n+ν − X | < ε).

ε>0 n=1 ν=1

∞ ∞ Thus: X n → X = n=1 ν=1 (|X n+ν − X | < ε). Clearly, ε > 0 may n→∞

ε>0

be replaced by εk ↓ 0 or k1 (↓ 0). Then

∞ ∞ ∞

1 |X | Xn → X = . n+ν − X < n→∞ k

k=1 n=1 ν=1

In a similar way we find that the set for which {X n } converges mutually is

∞ ∞ ∞

1 |X n+ν − X n | < . (X n+ν − X n n→∞ → 0) = k ν≥1 k=1 n=1 ν=1

Both of these sets are measurable, since they are taken by countable operations on measurable sets. Thus we have Theorem 3.

We have

∞ ∞ ∞

1 |X n+ν − X | < (X n → X ) = ∈ A, n→∞ k k=1 n=1 ν=1

∞ ∞ ∞

1 |X | (X n+ν − X n n→∞ ∈ A, → 0) = n+ν − X n < k ν≥1 k=1 n=1 ν=1

∞ ∞ ∞ 1 |X n+ν − X | ≥ and hence, (X n X ) = ∈ A, n→∞ k k=1 n=1 ν=1

∞ ∞ ∞ 1 |X n+ν − X n | ≥ (X n+ν − X n n→∞ ∈ A. 0) = k ν≥1

k=1 n=1 ν=1

In the following, it is understood that k, n = 1, 2, . . . and ν = 1, 2, . . . (or ν = 0, 1, . . .).

43

44


a.e.

Now X n → X means μ(X n X ) = 0, or equivalently, n→∞

n→∞

1 |X n+ν − X | ≥ μ = 0. k n ν k

1 = 0 implies μ − X| ≥ But μ k n ν |X n+ν − X | ≥ k n ν |X n+ν 1 1 |X | = 0, k = 1, 2, . . . , since ↑ − X ≥ n+ν n ν k n ν k k |X n+ν − X | ≥ k1 as k → ∞. In the other way around, if for all k = 1, 2, . . . , these sets have measure zero, then the measure of (X n X ) is zero. Thus n→∞ |X n+ν − X | ≥ k1 = 0, k = 1, 2, . . . , μ(X n X ) = 0 if and only if μ n ν n→∞ and in a similar fashion μ(X n+ν − X n n→∞ 0) = 0 if and only if μ n ν ν≥1 |X n+ν − X n | ≥ k1 = 0, k = 1, 2, . . .. These statements are true for any μ. Assume now that μ is finite. We have then, for a fixed k,

1 1 |X n+ν − X | ≥ |X n+ν − X | ≥ ↓ , as n → ∞, so that k k ν n ν

1 1 |X n+ν − X | ≥ |X n+ν − X | ≥ μ ↓μ as n → ∞. k k ν n ν Therefore, in this case,

1 |X n+ν − X | ≥ μ(X n X ) = 0 if and only if μ → 0 n→∞ n→∞ k ν

for each k = 1, 2, . . . , and in a similar way, μ(X n+ν

1 |X n+ν − X n | ≥ X n ) = 0 if and only if μ → 0 n→∞ n→∞ k ν≥1 ν

for each k = 1, 2, . . .. Thus we have the following theorem. Theorem 4.

1 |X n+ν − X | ≥ μ(X n X ) = 0 if and only if μ n→∞ k n ν = 0, k = 1, 2, . . . ,

and

μ(X n+ν

1 |X n+ν − X n | ≥ − X n n→∞ 0) = 0 if and only if μ k ν≥1 n ν

= 0, k = 1, 2, . . . .

(3.4)

3.2 Convergence in Measure is Equivalent to Mutual Convergence

In particular, if μ is finite, then

1 |X n+ν − X | ≥ μ(X n X ) = 0 if and only if μ → 0, n→∞ n→∞ k ν k = 1, 2, . . . ,

and μ(X n+ν − X n n→∞ 0) = 0 if and only if μ

ν≥1

|X n+ν

ν

1 − Xn| ≥ k

→ 0, k = 1, 2, . . .

n→∞

a.e.

μ

n→∞

n→∞

Corollary. If μ is finite, then X n → X implies X n → X . Proof.

a.e.

In fact, let X n → X , or μ(X n X ) = 0. However, n→∞

n→∞

1 |X n+ν − X | ≥ μ(X n X ) = 0 is equivalent to μ → 0, n→∞ n→∞ k ν k = 1, 2, . . . .

But

1 μ |X n − X | ≥ k Thus

1 μ |X n − X | ≥ k

⎛

⎞

1 ⎠. |X n+ν − X | ≥ ≤ μ⎝ k ν≥0

→ 0,

n→∞

k = 1, 2, . . . , or equivalently,

μ

Xn → X . n→∞

Remark 2. This need not be true if μ is not finite. Also, the inverse need not be true even if μ is finite. These points can be illustrated by examples (see Exercises 2(i) and 2(ii)).

3.2 Convergence in Measure is Equivalent to Mutual Convergence in Measure It is first shown that convergence in measure ensures the existence of a subsequence that converges almost everywhere. Then, by means of this result and almost uniform convergence, it is shown that convergence in measure and mutual convergence in measure are equivalent, as is the case in almost everywhere convergence.

45

46


Theorem 5. (i)

If {X n } converges mutually in measure, there is a subsequence {X n k } of {X n } a.e.

and a r.v. X such that X n k → X as n → ∞ (which implies n k → ∞). μ a.e. (ii) If X n → X , there is a subsequence {X n k } of {X n } such that X n k → X , a r.v. n→∞

n→∞

X , and μ(X = X ) = 0. Proof.

(i) {X n } converges mutually in measure implies that

1 1 μ |X m − X n | ≥ k < k for m, n ≥ n(k), k = 1, 2, . . . . 2 2

(3.5)

Define n 1 = n(1) n 2 = max{n 1 + 1, n(2)} n 3 = max{n 2 + 1, n(3)}. .. . Then n 1 < n 2 < n 3 < · · · → ∞ since each term increases at least by 1. For k = 1, 2, . . ., we set X k = X n k and define

∞ 1 Ak = X k+1 − X k ≥ k , Bn = Ak . 2 k=n Then μ(Ak ) = μ X n k+1 − X n k ≥ 21k < 21k , by (3.5), since n k , n k+1 ≥ n(k) from 1 their own definition, and hence μ(Bn ) ≤ ∞ k=n μ(Ak ) ≤ 2n−1 ; i.e., μ(Bn ) ≤

1 2n−1

.

(3.6)

We are aiming at showing that {X k } converges mutually a.e. To this end, for ε > 0, 1 choose n 0 = n(ε) such that 2n01−1 < ε. Then 2n−1 < ε, n ≥ n 0 . Now for k ≥ n ≥ n 0 and ν ≥ 1, one has X k+ν − X k = X k+ν − X k+ν−1 + X k+ν−1 − X k+ν−2 + X k+ν−2 − X k+ν−3 + · · · + X k+1 − X k + X ≤ X k+ν − X k+ν−1 − X k+ν−2 k+ν−1 + · · · + X − X + X k+ν−2 − X k+ν−3 k+1 k k+ν−1 k+ν−1 = − X X j+1 X j+1 − X j j ≤ ≤

j=k ∞

X j+1 − X j .

j=n

j=n


Therefore, if ω ∈ Bnc =

implies

∞

Ack , or equivalently, ω ∈ Acj , j ≥ n, then 1 X j+1 (ω) − X j (ω) < j 2 k=n

∞ X j+1 (ω) − X j (ω) ≤ j=n

1 < ε, 2n−1

and this implies in turn that X k+ν (ω) − X k (ω) < ε, k ≥ n(≥ n 0 ), ν ≥ 1.

(3.7)

This, in turn, gives that X k+ν − X k ≥ ε ⊆ Bn , k ≥ n(≥ n 0 ), ν = 1, 2, . . . . Hence

∞ X

k+ν

− X k ≥ ε ⊆ Bn , k ≥ n(≥ n 0 )

ν=1

implies

∞ ∞

X

k+ν

− X k ≥ ε ⊆ Bn (n ≥ n 0 ),

k=n ν=1

and therefore, by (3.6), ⎛

X μ⎝

k+ν

k≥n ν≥1

⎞ − X ≥ ε ⎠ ≤ μ(Bn ) ≤ k

1 2n−1

, (n ≥ n 0 ),

1 so that μ k≥1 ν≥1 X k+ν − X k ≥ ε ≤ 2n−1 (n ≥ n 0 ). Letting n → ∞, we get ⎛ ⎞

X − X ≥ ε ⎠ = 0 for every ε > 0, or equivalently, μ⎝ k+ν k ⎛

k≥1 ν≥1

X ⎝ μ

k+ν

k≥1 ν≥1

⎞

1 ⎠ = 0, m = 1, 2, . . . (see (3.4)). − X k ≥ m

Then Theorem 4 applies and gives that {X k } converges mutually a.e. and therefore there exists a r.v. X such that {X k } = {X n k } (subsequence of {X n }) converges to X a.e. as k → ∞. μ (ii) Let X n → X . Then, by Theorem 2, {X n } converges mutually in measure, and n→∞

a.e.

hence, by part (i), there exists {X k } ⊆ {X n } such that X k → X . It remains for us k→∞

to show that μ(X = X ) = 0. This is done in the Proof of Theorem 5(ii) (continued later).

47

48


Remark 3. It is to be pointed out here that the subsequence, constructed in Theorem 5, that converges a.e. to a r.v. will be used extensively in many instances and for various purposes. Theorem 6. If {X n } converges mutually in measure, then {X n } converges in measure to a r.v. X as n → ∞. Proof (For the case that μ is finite). By Theorem 5 (i), there exists a subsequence a.e.

μ

k→∞

k→∞

{X nk } of {X n } and a r.v. X such that X n k → X . Also X n k → X by the Corollary μ

to Theorem 4 (since μ is finite). To show that X n → X . Indeed, n→∞

|X n − X | ≤ X n − X n k + X n k − X implies ε ε (|X n − X | ≥ ε) ⊆ X n − X n k ≥ ∪ X nk − X ≥ , 2 2 so that

ε ε μ(|X n − X | ≥ ε) ≤ μ X n − X n k ≥ + μ X nk − X ≥ . 2 2 Letting n → ∞ (which implies that n k → ∞), we get μ X n − X n k ≥ 2ε → 0 from mutual convergence in measure of {X n }, and μ X n k − X ≥ 2ε → 0. Hence, μ as n → ∞, μ |X n − X | ≥ ε → 0, or equivalently, X n → X . n→∞

Theorem 6 is valid even if μ is not finite, but for its proof we need some preliminary results. (See Proof of Theorem 6 (continued) later.) a.u.

Definition 3. We say that {X n } converges almost uniformly to X and write X n → X , n→∞

if for every ε > 0, there exists Aε ∈ A such that μ(Aε ) < ε and X n → X unin→∞

formly on Acε . We say that {X n } converges mutually almost uniformly if for every ε > 0 there exists Aε ∈ A such that μ(Aε ) < ε and X m − X n → 0 uniformly on m,n→∞

a.u.

Acε . Of course, X n → X if and only if {X n } converges mutually almost uniformly. n→∞ (Observe that there exist such sequences; e.g., the subsequence {X n k } constructed in Theorem 5 is such a sequence. See also Theorem 8 below.) Theorem 7.

a.e.

μ

n→∞

n→∞

a.u.

We have that X n → X implies that for k1 there exists Ak such that μ(Ak ) < n→∞ and X n → X uniformly on Ack , k = 1, 2, . . .. Set A = ∞ k=1 Ak . Then A ∈ A

Proof. 1 k

a.u.

n→∞

If X n → X , then X n → X and X n → X .

n→∞

1 and μ(A) k ) < k . Letting k → ∞, we then get μ(A) = 0. On the other hand, ∞≤ μ(A c c A = k=1 Ak and therefore for every ω ∈ Ac , it follows that ω ∈ Ack for some a.e.

k = 1, 2, . . . , so that X n (ω) → X (ω). That is, X n → X on Ac or X n → X . n→∞

n→∞

n→∞


a.u.

Again, X n → X implies that for every ε > 0 there exists Aε ∈ A such that μ(Aε ) < ε

n→∞ and X n

→ X uniformly on Acε . This second part means that for every

n→∞

δ > 0 there exists N (δ, ε) > 0 (independent of ω ∈ Acε ) such that for n ≥ N (δ, ε), |X n (ω) − X (ω)| < δ, ω ∈ Acε . Thus, for n ≥ N (δ, ε), (|X n − X | ≥ δ) ⊆ Aε , so that μ μ(|X n − X | ≥ δ) ≤ μ(Aε ) < ε; i.e., X n → X . n→∞

a.e.

Compare with Remark 2 where X n → X does not necessarily imply

Remark 4.

n→∞

μ

that X n → X . n→∞

Theorem 8. (i) If {X n } converges mutually in measure, then there exists {X k } ⊆ {X n } and a r.v. a.u.

X such that X k → X . μ

k→∞

a.u.

(ii) If X n → X , then there exists {X k } ⊆ {X n } and a r.v. X such that X k → X and

n→∞ μ(X =

k→∞

X ) = 0.

Proof. (i) Consider the subsequence {X k } constructed in Theorem 5(i). By the arguments which led to relation (3.7), for every ε > 0, there is n 0 = n(ε) > 0 integer and a set Bnc with μ(Bn ) < ε, n ≥ n 0 , such that, on Bnc , X k+ν (ω) − X k (ω) < ε, k ≥ n 0 , ν = 1, 2, . . . . Applying the above for n = n 0 , we get: for every ε > 0, there exists n 0 = n(ε) positive integer and a set Bn 0 with μ(Bn 0 ) < ε, such that, on Bnc0 , X (3.8) k+ν (ω) − X k (ω) < ε, k ≥ n 0 , ν = 1, 2, . . . . But (3.8) is the definition of mutual a.u. convergence. Then there exists a r.v. X a.u. such that X k → X . μ

k→∞

(ii) If X n → X , then {X n } converges mutually in measure (by Theorem 2). Then, n→∞

a.u.

μ

μ

k→∞

k→∞

k→∞

by part (i), X k → X . However, X k → X by Theorem 7. So X k → X and μ

X k → X (since {X k } ⊆ {X n }). Therefore μ(X = X ) = 0 by Theorem 1. k→∞

Remark 5. See Exercise 11 for an implication of a.e. convergence with regards to a.u. convergence. We are now in a position to complete the parts of various proofs left incomplete so far. (ii)

49

50


Proof of Theorem 5(ii) (continued). Recall that in Theorem 5 (ii) we assumed μ

a.e.

n→∞ r.v. X .

k→∞

X n → X and we proved the existence of a subsequence {X k } such that X k → X , What remained to be verified was that μ(X = X ) = 0. This is done in a Theorem 8 (ii). Proof of Theorem 6 (continued). We can now give the proof of Theorem 6 for not necessarily a finite μ. Consider the sequence {X n k } of Theorem 5 or Theorem μ a.u. a.e. 8. Then X n k → X implies (X n k → X and) X n k → X by Theorem 7. Hence k→∞ k→∞ k→∞ μ(|X n − X | ≥ ε) ≤ μ X n − X n k ≥ 2ε + μ X n k − X ≥ 2ε → 0 + 0 = 0, the μ

k→∞

first by assumption and the second by the fact that X n k → X .

k→∞

μ

Corollary to Theorem 2 and 6. X n → X if and only if {X n } converges mutually n→∞ in measure. A brief summary of the results obtained in this chapter is as follows. The sequence of r.v.s we refer to is {X n }, and a limiting r.v. is X . 1. Convergence in measure implies (μ−) uniqueness of the limit (Theorem 1). 2. Convergence in measure is equivalent to mutual convergence in measure (Theorems 2 and 6). 3. Expressions of the set of pointwise convergence (and hence of nonconvergence), as well as the set of pointwise mutual convergence (and hence of mutual nonconvergence) (Theorem 3). 4. Necessary and sufficient conditions for a.e. (and mutual a.e.) convergence for any μ, and in particular, for finite μ (Theorem 4). 5. Almost everywhere convergence and finiteness of μ imply convergence in measure. The converse need not be true even for finite μ (Corollary to Theorem 4, Exercise 2(i), (ii)). 6. Convergence in measure (or mutual convergence in measure) implies the existence of a subsequence that converges a.e. (or converges mutually a.e.) (Theorem 5). 7. Almost uniform convergence implies a.e. convergence and convergence in measure (Theorem 7). 8. Convergence in measure (or mutual convergence in measure) implies the existence of a subsequence that converges a.u. to a r.v. (or converges mutually a.u.) (Theorem 8). Exercises. μ

1. If X n → X , then show directly (that is, without reference to other results) that: (X n

n→∞ − X )+

μ

μ

μ

μ

→ 0, (X n − X )− → 0, X n+ → X + and X n− → X − .

n→∞

Hint: For the convergence μ

n→∞ n→∞ μ X n+ → X + , show that |X n+ − n→∞

n→∞

X + | ≤ |X n − X |, and

likewise for X n− → X − . To this end, use Exercise 28 in Chapter 1. n→∞


2. By means of examples, show that: a.e.

μ

n→∞ μ

n→∞ a.e.

n→∞

n→∞

(i)

X n → X need not imply X n → X if μ is not finite.

(ii)

X n → X need not imply X n → X even if μ is finite.

Hint: In (i), take = , A = B, μ = λ, the Lebesgue measure, and choose the r.v.s suitably. In part (ii), take = (0, 1], A = B , μ = λ, the Lebesgue measure, and choose the r.v.s suitably. ∞ 3. For any sequence of events {An }, n ≥ 1, show that P(An ) < ∞ implies n=1

P(lim sup An ) = 0. n→∞

4. The sequence {X n }, n ≥ 1, of r.v.s is said to converge completely to 0, if, for ∞ P(|X n | ≥ ε) < ∞. every ε > 0, n=1 a.e.

(i) Show that, if {X n }, n ≥ 1, converges completely to 0, then X n → 0. n→∞

(ii) By means of an example, show that complete convergence is not necessary for a.s. convergence. Hint: For part (i), use Exercise 3 here and Exercise 4 in Chapter 2. For part (ii), take = (0, 1], A = B , P = λ, the Lebesgue measure, and choose the r.v.s suitably. a.e. Note: The most common way of establishing that X n → X is to show that n→∞ {X n − X }, n ≥ 1, converges completely to 0. P

5. Show that X n → X if and only if, for every {n } ⊆ {n}, there exists a further subsequence

n→∞ {n } ⊆

a.s.

{n } such that X n → X and any two limiting r.v.s are n →∞

a.s. equal. a.s. 6. By means of an example, show that X n → c = 0 need not imply that P(X n = n→∞

0) = 1 for all n. 7. If for some εn > 0 with a.s.

∞

εn < ∞, it holds that

n=1

∞

P(|X n+1 − X n | ≥ εn ) < ∞,

n=1

then show that X n → to a r.v. n→∞

Hint: It suffices to show that {X n } converges mutually a.s. To this end, set An = (|X n+1 − X n | ≥ εn ), n ≥ 1, and use Exercise 3 in this chapter in order to c conclude that P( lim An ) = 0. Then, by setting A = lim An and N c = A , it n→∞

n→∞

follows that the event N c (with P(N c ) = 1) is the set over which {X n } converges mutually. 8. For n = 1, 2, . . . , let X n and X be (real-valued) r.v.s and let g : → be conP

P

tinuous. Then show that X n → X implies g(X n ) → g(X ) (so that continuity n→∞ n→∞ preserves convergence in probability).

51

52


Hint: One way of approaching this problem is to use Exercise 5. 9. For n = 1, 2, . . . , let X n , Yn , X , and Y be (real-valued) r.v.s and let g : 2 → P

P

P

n→∞

n→∞

n→∞

be continuous. Then show that X n → X and Yn → Y implies g(X n , Yn ) → g(X , Y ) (so that continuity preserves convergence in probability). a.s. 10. Show that X n → X if and only if there is a sequence 0 < εn → 0 such that n→∞

n→∞

⎡

⎤ P⎣ |X k − X | ≥ εk ⎦ → 0. n→∞

k≥n

Hint: For the part εn → 0, show that, for every ε > 0, there exists N = N (ε) n→∞ (|X k − X | ≥ ε) ⊆ (|X k − X | ≥ εk ) such that k ≥ N and n ≥ N imply k≥n

k≥n a.s.

and then use Theorem 4 suitably. For the part X n → X , use Theorem 4 in order n→∞ to conclude that P |X k − X | ≥ ε → 0. Applying this conclusion for n→∞

k≥n

m ≥ 1, show that there exists a sequence n m ↑ ∞ as m → ∞ such that ⎡ ⎤

1 ⎦< 1 . |X k − X | ≥ P⎣ m 2m k≥n m

Finally, for n m ≤ k < n m+1 , set εk =

1 m

and show that

⎡

⎤ ⎡ ⎤ P⎣ |X k − X | ≥ εk ⎦ ≤ P ⎣ |X k − X | ≥ εk ⎦ ≤ k≥n

k≥n m

1 2n−1

.

a.e.

11. (Egorov’s Theorem). Show that, if μ is finite, then X n → X implies that n→∞

a.u.

Xn → X . n→∞

Hint: For an arbitrary ε > 0 to be kept fixed throughout and k ≥ 1 integer, use Theorem 4 in order to conclude that there (ε, k) > 0 such exists Nk = N |X n − X | ≥ k1 . Thus, if Aε = that μ(Aε,k ) < 2εk , k ≥ 1, where Aε,k = n≥Nk Aε,k , then μ(Aε ) ≤ ε. Finally, show that X n (ω) → X (ω) uniformly in n→∞

k≥1

ω ∈ Aε . 12. Show that the complement Ac of the set A of (pointwise) convergence of a sequence of r.v.s {X n }, n ≥ 1, is expressed as follows: Ac = ∪ {ω ∈ ; lim inf r ,s

n→∞

X n (ω) ≤ r < s ≤ lim sup X n (ω)}, where the union is taken over all rationals r and s with r < s.

n→∞


13. For n = 1, 2, . . . , let X n , X be r.v.s defined on the measurable space (, A), and suppose that X n −→ X (pointwise). Then show that n→∞

(i) X is σ (X 1 , X 2 , . . .)-measurable. (ii) σ (X 1 , X 2 , . . .) = σ (X 1 , X 2 , . . . , X ). 14. For a sequence of r.v.s {X n }, n ≥ 1, show that the set A = {ω ∈ ;

∞

X n (ω)

n=1

converges} is in the σ (X m , X m+1 , . . .) for every m ≥ 1. 15. Let X n , n ≥ 1, be r.v.s defined on the measure space (Ω, A, P), and suppose ∞ that P(|X n | > n) < ∞. Then show that lim |Xnn | ≤ 1 a.s. n→∞

n=1

Hint: Refer to Exercise 3 in this chapter. 16. For n = 1, 2, . . . , let Zn = (Z n1 , . . . , Z nk ) and Z = (Z 1 , . . . , Z k ) be kP

P

dimensional r.v.s. Then we say that Zn → Z if ||Zn − Z|| → 0, or n→∞ n→∞ 1/2 k P P (Z n j − Z j )2 → 0. Then show that Zn → Z if and only if n→∞

j=1

n→∞

P

Z n j → Z j , j = 1, . . . , k. n→∞ P Xn → n→∞

P

17. If X and g : → is continuous, then g(X n ) → g(X ) (see n→∞ Exercise 8). Show that this need not be true, if g is not continuous. Hint: Consider the function δc (x) = 0 for x < c, and δc (x) = 1 for x ≥ c, for some constant c ∈ . μ 18. If X n −→ X and |X n | ≤ Y a.e. n ≥ 1, then show that |X | ≤ Y a.e. with respect n→∞ to the measure μ. 19. Refer to Exercise 13 and continue as follows: For n = 1, 2, . . . , let X n , X be r.v.s a.e.

defined on the measure space (, A, μ), and suppose that X n → X . Then, by n→∞ means of concrete examples, show that: (i) X is σ (X 1 , X 2 , . . .)-measurable. (ii) X is not σ (X 1 , X 2 , . . .)-measurable. a.e.

(iii) If X n → X , show that the X n s and X can be modified into X n s and n→∞ X , so that X n → X pointwise, X is σ X 1 , X 2 , . . . -measurable, and n→∞ ∞ n=1 μ X n = X n = 0 = μ(X = X ). (As a consequence, instead of the X n s and X one could use the X n s and X , without loss of generality, and also ensure that X is σ (X 1 , X 2 , . . .)measurable.) (iv) Consider the measurable space (, A, μ), and suppose that, for some ω0 ∈ , {ω0 } ∈ A and μ({ω0 }) = 0. Define X n (ω) = 0 on {ω0 }, and X 2n−1 (ω0 ) = 2, X 2n (ω0 ) = 3, n ≥ 1; and X (ω) = 0 on {ω0 }c , X (ω0 ) = 1.

53

54


a.e.

Then verify that X n → X . Furthermore, modify the X n s and X as indin→∞ cated in part (iii), so that the conclusions of that part hold. a.e.

20. Show that the convergence X m − X n −→ 0 as m, n → ∞ is equivalent to the a.e.

convergence X n+ν − X n −→ 0 uniformly in ν ≥ 1. n→∞

21. Show that, for every ε > 0 : μ(|X m − X n | ≥ ε) → 0 as m, n → ∞ if and only if μ(|X n+ν − X n | ≥ ε) −→ 0 uniformly in ν ≥ 1. n→0

CHAPTER

The Integral of a Random Variable and its Basic Properties

4

In this chapter, the concept of the integral of a r.v. X with respect to a measure μ is defined, and the basic properties of integrals are established. The definition of the integral is given first for simple r.v.s, then for nonnegative r.v.s, and finally for any r.v.s. These things are done in Section 4.1, whereas the basic properties of the integral are discussed in Section 4.2. A brief summary of the results in these sections is also presented at the end of the second section. In the next short and final section of the chapter, the measure space (, A, μ) is replaced by the probability space (, A, P), and the probability distribution of a r.v. X , PX , is introduced. There is only one result here, Theorem 13 , whose significance lies in that integration over an abstract space is transformed to integration over the real line .

4.1 Definition of the Integral Consider nthe measure space (, A, μ) and let X be nonnegative simple r.v.; i.e., X = j=1 α j I A j , where α j ≥ 0, j = 1, . . . , n, and {A j , j = 1, . . . , n} is a partition of . To such anX , we assign a possibly extended number, denoted by I (X ), as follows: I (X ) = nj=1 α j μ(A j ). We also make throughout the convention that ±∞ × 0 = 0. Then, Theorem 1. The function I : on the class of ≥ 0 simple r.v.s into [0, ∞] as defined above is well defined. m Proof. If also X = i=1 βi I Bi is another representation of X , we have to prove that n m j=1 α j μ(A j ) = i=1 βi μ(Bi ). Clearly, {A j ∩ Bi , j = 1, . . . , n, i = 1, . . . , m} is a partition of . For j = 1, . . . , n, i = 1, . . . , m, define ci j as follows: α j = βi if A j ∩ Bi = . c ji = whatever (e.g., = 0 for definiteness) if A j ∩ Bi = Then j,i

c ji μ(A j ∩ Bi ) =

j

i


c ji μ(A j ∩ Bi )

j i A j ∩ Bi =


55

56

CHAPTER 4 The Integral of a Random Variable and its Basic Properties

=

α j μ(A j ∩ Bi ) =

j

j i A j ∩ Bi =

i.e.,


j,i


j,i

=

αj

μ(A j ∩ Bi ) =

i A j ∩Bi =

α j μ(A j );

j

α j μ(A j ), and in a similar fashion

j

βi μ(Bi ). Thus

i

α j μ(A j )

j

βi μ(Bi ) (see also Exercise 15).

i

Theorem 2. Let X , Y be nonnegative simple r.v.s such that X ≥ Y . Then I (X ) ≥ I (Y ); furthermore, if the measure μ is finite, then I (X ) > I (Y ) if and only if X > Y on a set of positive measure. In particular, X ≥ 0 implies I (X ) ≥ 0; furthermore, if the measure μ is finite, then I (X ) > 0 if and only if X > 0 on a set of positive measure. m n Proof. Let X= j,i α j × j=1 α j I A j , Y = i=1 βi I Bi . Then, clearly, X = α μ(A I A j ∩Bi , Y = i, j βi I A j ∩Bi since I = 0, and hence I (X ) = j j ∩ j,i Bi ), I (Y ) = i, j βi μ(A j ∩ Bi ). Now X ≥ Y implies α j ≥ βi on A j ∩ Bi (which are = ) and hence I (X ) ≥ I (Y ). Next, let μ be finite. Then if X > Y on a set E with μ(E) > 0, one has that α j ≥ βi on all A j ∩ Bi (which are = ) and α j > βi on E ∩ (A j ∩ Bi ) for some A j ∩ Bi with μ[E ∩ (A j ∩ Bi )] > 0. Since α j μ(A j ∩ Bi ) = α j μ[E ∩ (A j ∩ Bi )] + α j μ[E c ∩ (A j ∩ Bi )] > βi μ[E ∩ (A j ∩ Bi )] + βi μ[E c ∩ (A j ∩ Bi )] = βi μ(A j ∩ Bi ), we have that I (X ) > I (Y ). Finally if I (X ) > I (Y ), there must exist (by contradiction) a set E such that μ[E ∩ (A j ∩ Bi )] > 0 for some A j ∩ Bi for which α j > βi (whereas always α j ≥ βi ). Indeed, if α j = βi on all A j ∩ Bi = , then I (X ) = I (Y ), a contradiction. If α j > βi on A j ∩ Bi with μ(A j ∩ Bi ) = 0 and α j = βi on all A j ∩ Bi with μ(A j ∩ Bi ) > 0, then again I (X ) = I (Y ), a contradiction. Thus, there must exist A j ∩ Bi = with μ(A j ∩ Bi ) > 0 on which α j > βi ; i.e., X > Y on a set of positive measure; namely, E = A j ∩ Bi for some j, i. The special case follows by taking Y = 0. Bi

Aj E

4.1 Definition of the Integral

Next, let X be a nonnegative r.v. (not necessarily a simple r.v.). Then, by the Corollary to Theorem 17, Chapter 1, there exist X n nonnegative, simple r.v.s such that X n ↑ X . To such an X we assign the quantity I (X ) as follows: I (X ) = limn→∞ I (X n ). This limit exists and, clearly, I (X ) ≥ 0 (but it may happen that I (X ) = ∞). Then we have the following theorem. Theorem 3. The (possibly extended) function I : on the class of ≥ 0 r.v.s into [0, ∞] as defined earlier is well defined. Remark 1. Before we proceed with the proof of this theorem we observe that the I just defined coincides with the I defined before on the class of ≥ 0 simple r.v.s since in such a case we can take X n = X , n = 1, 2, . . .. Thus the I is an extension from the class of ≥ 0 simple r.v.s to the class of ≥ 0 r.v.s. Proof of Theorem 3.

In order to prove the theorem it suffices to prove that

If Y is a ≥ 0 simple r.v. with Y ≤ X , then I (Y ) ≤ lim I (Z n ), with 0 ≤ Z n simple r.v.s ↑ X ,

(4.1)

where here and in the sequel, all limits are taken as n → ∞, unless otherwise specified. In fact, if (4.1) is true, then for 0 ≤ X n simple ↑ X , 0 ≤ Yn simple ↑ X , we have Yn ≤ X implies I (Yn ) ≤ lim I (X n ) and lim I (Yn ) ≤ lim I (X n ). Also X n ≤ X implies I (X n ) ≤ lim I (Yn ) and lim I (X n ) ≤ lim I (Yn ). Thus lim I (X n ) = lim I (Yn ). In order to establish (4.1) we distinguish three cases. de f 0. Case 1. μ() rn We have X n = i=1 αni I Ani , Y = sj=1 β j I B j . Choose ε > 0 such that ε < m and define Cn as follows: Cn = (X n > Y − ε). Then Cn ↑ , since X n ↑ X and X ≥ Y imply X > Y − ε. Indeed, from the definition of Cn and the fact that X n ↑ X , de f we have Cn ↑, so that lim Cn = C = n≥1 Cn . Then C = . If not, there exists ω ∈ but ω ∈ / C if and only if ω ∈ n≥1 Cnc , which implies X n (ω) ≤ Y (ω) − ε for every n, and hence X (ω) ≤ Y (ω) − ε, a contradiction. Next, rn rn I (X n ) = αni μ(Ani ) ≥ αni μ(Ani ∩ Cn ), i=1

i=1

since Ani ⊇ Ani ∩ Cn . n s Also μ(Ani ∩ Cn ) = sj=1 μ(Ani ∩ Cn ∩ B j ). Hence I (X n ) ≥ ri=1 j=1 αni × μ(Ani ∩ Cn ∩ B j ).Now,since ω ∈ (Ani ∩ Cn ∩ B j ) implies αni > β j − ε, rn s we get I (X n ) ≥ j=1 (β j − ε)μ(Ani ∩ C n ∩ B j ), and this is equal to: i=1 s (β − ε)μ(C ∩ B ); i.e., we have n j j=1 j I (X n ) ≥

s

(β j − ε)μ(Cn ∩ B j ).

j=1

(Notice that up to now no use of the finiteness of μ has been made.)

(4.2)

57

58


Now, Cn ∩ B j = B j − B j ∩ Cnc . Hence μ(Cn ∩ B j ) = μ(B j ) − μ(Cnc ∩ B j ), since Cnc ∩ B j ⊆ B j and μ is finite. Thus I (X n ) ≥

s

(β j − ε)[μ(B j ) − μ(Cnc ∩ B j )]

j=1

=

s

β j μ(B j ) − ε

j=1

s

μ(B j ) −

j=1

= I (Y ) − εμ() −

s

s (β j − ε)μ(Cnc ∩ B j ) j=1

(β j − ε)μ(Cnc ∩ B j )

j=1

≥ I (Y ) − εμ() − (max Y )

s

μ(Cnc ∩ B j )

j=1

= I (Y ) − εμ() − (max Y )μ(Cnc ); i.e., I (X n ) ≥ I (Y ) − εμ() − (max Y )μ(Cnc ).

(4.3)

Now, Cn ↑ if and only if Cnc ↓ , which implies (since μ is finite) μ(Cnc ) ↓ μ() = 0. Therefore, letting n → ∞ in (4.3), we get lim I (X n ) ≥ I (Y ) − εμ(). Now letting ε → 0, we get I (Y ) ≤ lim I (X n ), which is (4.1). de f Case 2. μ() =∞, min Y = m > 0. s Then I (X n ) ≥ j=1 (β j − ε)μ(Cn ∩ B j ) according to (4.2), where no use of the finiteness of μ was made. Now, s s (β j − ε)μ(Cn ∩ B j ) ≥ (m − ε)μ(Cn ∩ B j ) j=1

j=1

= (m − ε)

s

μ(Cn ∩ B j )

j=1

= (m − ε)μ(Cn ); i.e., I (X n ) ≥ (m − ε)μ(Cn ).

(4.4)

Since μ(Cn ) ↑ μ() = ∞ and m − ε > 0, we get, by taking the limits in (4.4) as (4.1). n → ∞ : lim I (X n ) = ∞ and hence I (Y ) ≤ lim I (X n ), which is Case 3. μ() ≤ ∞, min Y = m = 0. Recall that 0 ≤ Y = sj=1 β j I B j ≤ X , n and we wish to show that for any 0 ≤ ri=1 αni I Ani = X n ↑ X , we have I (Y ) ≤ lim I (X n ). de f B jk where the If A = (Y > 0), then min A Y = m A > 0, whereas A = summation is over those jk s for which the corresponding β jk s are > 0, and of course,

4.1 Definition of the Integral

Ac = B jl , where the summation is over those jl s for which the corresponding β jl s are = 0. Let X (A) be the restriction of X to A, and set Y (A) =

s

β j I B j ∩A ,

j=1

X n(A) =

rn

αni I Ani ∩A .

i=1

Then, clearly, 0 ≤ X n(A) simple r.v.s ↑ X (A) , since X n(A) = X n and X (A) = X on A, and X n ↑ X . Also, Y (A) ≤ X (A) because Y (A) = Y and X (A) = X on A, and Y ≤ X . Since m A = min Y (A) > 0, case 1 or case 2 applies (depending on whether (A) μ() < ∞ or μ() = ∞, respectively) and gives that I (Y (A) ) ≤ lim I (X n ). However, Y = β j IB j = β jk I B jk + β jl I B jl = β jk I B jk , j

so that I (Y ) = ≤

β jk μ(B jk ) =

β jk μ(B jk ∩ A), since B jk ∩ A = B jk

β j μ(B j ∩ A) = I (Y (A) ); i.e., I (Y ) ≤ I (Y (A) ).

j

Also, I (X n(A) ) =

αni μ(Ani ∩ A)

i

≤

αni μ(Ani ) = I (X n ),

i (A)

so that lim I (X n ) ≤ lim I (X n ). Combining the results obtained, we have then I (Y ) ≤ I (Y (A) ) ≤ lim I (X n(A) ) ≤ lim I (X n ), as was to be seen.

So if X is a ≥ 0 r.v. then I (X ) is well defined (but it may be = ∞). Now let X be any r.v. Then X = X + − X − , where X + , X − ≥ 0. Thus I (X + ), I (X − ) are well defined. If not both I (X + ), I (X − ) are = ∞, we define I (X ) by: I (X ) = I (X + ) − I (X − ). Clearly, if X ≥ 0 then X = X + and I (X ) = I (X + ), since I (0) = 0; and if X is a nonnegative simple r.v., then I (X ) coincides with the I (X ) as defined at the beginning of this section, since we can take X n = X , n ≥ 1. Thus this I on the class of all r.v.s for which I (X + ) − I (X − ) is defined is an extension of the I on the class of ≥ 0 r.v.s. Definition 1. The possibly extended function I defined on the class of r.v.s X for which I (X + ) − I (X − ) exists is calledthe integralof X over the space with respect is to the measure μ and is denoted by X dμ or X dμ, or X if no confusion possible. In particular, if μ is a probability measure P, then X dμ = X d P is denoted by E X and is called the (mathematical) expectation of X .

59

60


n n Thus if X = i=1 αi I Ai ≥ 0 then X dμ = i=1 αi μ(A i ); if the r.v. X is ≥ 0 Xn dμ for any 0 ≤ X n but not necessarily a simple r.v., then X dμ = lim n→∞ simple r.v. ↑ X as n → ∞; and for any r.v. X , X dμ = X + dμ − X − dμ, − + provided that at least one of X dμ, X dμ is < ∞. If now A ∈ A, then I A is measurable, and hence so is X I A for any r.v. Then, Definition 2. The integral of the r.v. X over A ∈ A, denoted by A X dμ, is defined by (X I A )dμ = (X I A )dμ, provided this latter integral exists. + − Definition 3. We say that X is integrable if both X , X are < ∞; i.e., X exists and is finite.

4.2 Basic Properties of the Integral In this section, we discuss the basic properties of the integral stated in the form theorems. Theorem 4. If X and Y are ≥ 0 r.v.s, then (X + Y ) = X + Y . m Proof. i=1 αi I Ai mis, X = n That m n First, let X and Y be≥n 0 simple r.v.s. α I and Y = β I = β I , i A ∩B j B j A ∩B i j j i j αi ≥ 0, i i=1 j=1 j=1 j=1 i=1 = 1, . . . , n. 1, . . . , m, β j ≥ 0, j m m n (αi + β j )I Ai ∩B j , and X = i=1 αi μ(Ai ), Y Then X + Y = i=1 j=1 n j=1 β j μ(B j ), and (X + Y ) = + +

β j μ(Ai i=1 j=1 m n βj

j=1

∩ Bj) =

μ(Ai ∩ B j ) =

i=1

= = =

m m n n (αi + β j )μ(Ai ∩ B j ) = αi μ(Ai ∩ B j ) i=1 j=1

m n

of

i=1 j=1 m i=1 m i=1

αi

n

μ(Ai ∩ B j )

j=1

αi μ(Ai ) +

n

β j μ(B j ) = X + Y .

j=1

Then, clearly, Next, let X n , Yn be ≥ 0 simple r.v.s such that X n ↑ X , Yn ↑ Y . n αni I Ani , X n +Yn are ≥ 0 simple r.v.s such that (X n +Yn ) ↑ (X +Y ). Let X n = rj=1 sn rn sn X n + Yn = Y = j=1 βnj I Bn j . Then i=1 j=1 (αni + βn j )I Ani ∩Bn j , and n just seen for the nonnegative simple r.v.s X (X n + Yn ) = X n + Yn , as was and Y ; i.e., (X n + Yn ) = X n + Yn . Taking the limits, as n → ∞, we then get (X + Y ) = X + Y . Remark 2. The theorem generalizes in an obvious manner to any finite number of nonnegative r.v.s. Theorem 5. Let A, B ∈ A with A ∩ B = . Then, (i) If A+B X exists, then A X , B X also exist and A+B X = A X + B X .

4.2 Basic Properties of the Integral

(ii) If A X , B X exist, exist (i.e., it is not of the form and A X + B X also ± ∞ ∓ ∞), then A+B X also exists and A+B X = A X + B X . Proof.

(i) Clearly

+ X I A+B = X + I A+B = X + I A + X + I B = (X I A )+ + (X I B )+ ,

since A ∩ B = . So

+ X I A+B = (X I A )+ + (X I B )+ and similarly, (4.5)

− X I A+B = (X I A )− + (X I B )− .

+ Now that A+B X = X I A+B exists implies that at least one of X I A+B ,

−

+ XI is < ∞. Let X I A+B < ∞. By (4.5) and Theorem 4, we have A+B + X I A+B = (X I A )+ + (X I B )+ and hence both (X I A )+ , (X I B )+ are < ∞.

− Thus X I A+B = (X I A )− + A X , B X exist. Again by (4.5) and Theorem 4, (X I B )− . Thus

+

− X= X I A+B = X I A+B − X I A+B A+B = (X I A )− + (X I B )− (X I A )+ + (X I B )+ − = (X I B )+ − (X I B )− (X I A )+ − (X I A )− + X+ X ; i.e., X= X+ X. = X IA + X IB = A

B

A+B

−

A

B

The same result follows if weassume that X I A+B < ∞. (ii) Now, let A X , B X , A X + B X exist. Then either (X I A )+ or (X I A )− is < ∞, or both. + − Also, either (X I B ) or (X I B ) is < ∞, or both. From the existence of A X + B X we then have X+ X = (X I A )+ − (X I A )− + (X I B )+ − (X I B )− . A

B

Set

(X I A )+ = a + , +

+

(X I B ) = b , Then

(X I A )− = a − , (X I B )− = b− .

X = a + − a − + b+ − b− ,

X+ A

B

61

62


and its existence means only the following cases may occur: ⎧ ± ± ⎨ a , b are all < ∞ a + and/or b+ = ∞ but a − and/or b− finite . ⎩ − a and/or b− = ∞ but a + and/or b+ finite − For any of these cases, the (a+ + b+ ) − (a − grouping + b ) is legitimate, and this + − is equal to A+B X − A+B X = A+B X . Thus A+B X exists and A+B X = A X + B X.

Remark 3. The theorem generalizes in an obvious manner to any finite number of pairwise disjoint events. (see also Exercise 5.) Corollary. If X exists, then for any A ∈ A, A X exists and X = A X + Ac X . Furthermore, if X is integrable over , then X is integrable over any A ∈ A. Proof. The first part of the corollary follows from Theorem 5(i) with A = A, c . As for the second part, we have: the fact that X is that B = A finite implies + − X , X < ∞. Since X + = A+Ac X + = A X + + Ac X + , X − = − = − − + − < ∞, and hence X A+Ac X A X + Ac X , we then get A X , A X is integrable over A. Theorem 6. If X exists and c ∈ , then cX exists and cX = c X . Proof. We distinguish the following cases: Case 1. Let c = −1. Then cX = −X . But:(−X )+ = X − , (−X )− = X + , so that X − = (−X )− − (−X )+ X = X+ − −X = − X . =− (−X )+ − (−X )− = − (−X ) or Case 2. Let c ≥ 0. If c = 0, then c × X = 0implies cX = 0 = 0 = 0 X , since 0 × (±∞) is also 0 (for the case that X = ±∞). We assume now that + + − − + − c > 0. Then (cX ) = cX , (cX ) = cX , so that cX = (cX ) − (cX ) = cX + − cX − ; i.e., cX =

cX + −

cX − ,

(4.6)

provided it exists. + hence0 ≤ cX n simple Since X + ≥ 0, there exist 0 ≤ X n simple r.v.s ↑ X and + r.v.s ↑ cX as n → ∞, which implies cX n → cX + . But cX n = c X n n→∞ + X , we then get cX + = for simple r.v.s, as is readily seen. Since also X n → n→∞ c X + . In a similar way, cX − = c X − , and then (4.6) exists and gives the desired result. Case 3. Let c < 0. Then cX = (−1)[(−c)X] = −1 (−c)X , by case 1, and this is = −(−c) X , by case 2, since −c ≥ 0; i.e., cX = c X .


Theorem 7.

If X ≥ 0, then

X ≥ 0.

r.v.s ↑ X as Proof. Let n → ∞ whereas 0 ≤ X n simple X , we get then X ≥ 0. Since also X n → n→∞

X n ≥ 0 by Theorem 2.

Theorem 8. (i) If X ≤ Y and X , Y exist, then X ≤ Y . (ii) If X ≤ Y and X − < ∞ (so that X exists), then Y exists and X ≤ Y by part (i). (iii) If X ≤ Y and Y + < ∞ (so that Y exists), then X exists and X ≤ Y bypart (i). (iv) If X exists, then X ≤ |X |. Proof. (i) Let 0 ≤ X ≤ Y . ThenY = X + (Y − X ) = X + Z , where Z = Y − X ≥ 0. Hence Z ≥ 0 by Theorem 7. Thus Y = X + Z by Theorem 4. Also, + ≤ Y + , X − ≥ Y − , clearly. Then X ≤ Y . In general, X ≤ Y implies X + ≤ − Y − , by what we have just X ≤ Y + , X − ≥ Y − or − X − + − − ≤ + − Y − or X Y X ≤ Y , since established. Therefore X X , Y exist. − − we (ii) From X − ≥ Y − above, get that X < ∞, which implies Y < ∞, so that Y exists and X ≤ Y by part (i). (iii) From X + ≤ Y + above, we get that Y + < ∞ so that X + < ∞, and hence X exists and X ≤ Y by part (i). X ≤ |X | |X | |X | |X | (iv) Indeed, − ≤ X ≤ implies − ≤ X ≤ or |X |. Theorem 9. If X = Y a.e. and X exists, then Y exists and X = Y . Remark 4. We first observe that if μ(B) = 0, for some B ∈ A, then B Z = 0, for + exist 0 ≤ Z n I B simple r.v.s ↑ Z + I B as any r.v. Z . In fact, Z ≥ 0 implies thatthere + + Z I B = B Z . But B Z n = 0, because, n → ∞; hence B Z n = Z n I B → n→∞ rn rn if Z n = αn j An j , then Z n I B = αn j I B∩An j and B Z n = Z n I B = j=1 j=1 rn α μ(B ∩ An j ) = 0. This implies that B Z + = 0. Similarly, B Z − = 0, so j=1 nj that B Z = 0. Proof of Theorem 9. Let A = (X = Y ). Then A = (X − Y = 0) and A is −1 measurable, since A = (X − Y ) ({0}) and X − Y is a r.v. Now, the existence of X implies the existence of A X , Ac X and X = A X + Ac X , by the Corollary to Theorem 5. But X+ X = X IA + X = Y IA + 0 c A Ac A = Y = Y +0= Y+ Y A

A

A

Ac

63

64


(by 4), and this is equal to Remark Y = X.

Y by Theorem 5(ii). So

Y exists and

X is integrable if and only if |X | is integrable. + − + − Proof. Now X being integrable implies X , X < ∞. Since |X | = X + X , we get |X | = X + + X − by Theorem 4, and then |X | Y ≥ −∞. So if X = Y = ±∞, then X + Y = ±∞, and, by means of simple r.v.s, it is seen (see also Exercise 6) that (X + Y ) = ±∞. So (X + Y ) = X + Y in this case. Next assume that at least one of X , Y is finite and, in order to fix ideas, let Y be finite. This implies that A Y is finite for any A ∈ A by the Corollary to Theorem 5. Now, we consider the following partition of by the sets A j , j = 1, . . . , 6, defined as follows: Theorem 10.

A1 = X ≥ 0, Y ≥ 0, X + Y ≥ 0 A4 = X < 0, Y ≥ 0, X + Y ≥ 0 ,

A2 = X ≥ 0, Y < 0, X + Y ≥ 0 ; A5 = X < 0, Y ≥ 0, X + Y < 0 ,

A3 = X ≥ 0, Y < 0, X + Y < 0 A6 = X < 0, Y < 0, X + Y < 0 . The existence of X , Y implies the existence of A j X , A j Y , j = 1, . . . , 6, by the Corollary to Theorem 5. We will prove that A j (X + Y ) = A j X + A j Y , j = 1, . . . , 6, which will imply that

(X + Y ) =

(X + Y )

=

Aj

j

j

=

j

=

(X + Y ) Aj

j

X+ Aj

X+ Aj

Y Aj

j

Y Aj


=

X+

=

j

Aj

X+

Y

j

Aj

Y

by Remark 3 and Exercise 5. We have A1 : (X + Y )I A1 = X I A1 + Y I A1 with X I A1 , Y I A1 ≥ 0. Then A1 (X + Y ) = (X I A1 +Y I A1 ) = X I A1 + Y I A1 , by Theorem 4, and this is = A1 X + A1 Y . A2 : X I A2 = (X + Y )I A2+ (−Y )I A2 with (X + Y )I A2 , (−Y )I A2 ≥ 0. Then, by Theorem 4, A2 X = X I A2 = [(X + Y )I A2 + (−Y )I A2 ] = (X + Y )I A2 + −Y I A2 = A2 (X + Y ) − A2 Y . Since A2 Y is finite (by the Corollary to Theorem 5), we get A2 (X + Y ) = A2 X + A2 Y . A3 : −Y I A3 = −(X by + Y )I A3+ X I A3 with −(X + Y )I A3 , XI A3 ≥ 0. Then, Theorem 6, − A3 Y = − A3 (X + Y )+ A3 X or A3 Y = A3 (X +Y )− A3 X , and since A3 Y is finite, so is A3 X . Then we get A3 (X + Y ) = A3 X + A3 Y . A4 : Y I A4 = (X +Y )I A4 +(−X )I A4 and A4 Y = A4 (X +Y )+ A4 − X = A4 (X + Y ) − A4 X . Since A4 Y is finite, so is A4 X and A4 (X + Y ) = A4 X + A4 Y . A5 : −X I A5 = Y I A5 +(−X −Y )I A5 and − A5 X = A5 Y − A5 X +Y or − A5 X − X + Y. A5 Y = − A5 X + Y , or A5 X + A5 Y = A5 A6 : −(X + Y )I A6 = −X I A6 − Y I A6 and − A6 X + Y = − A6 X − A6 Y or A6 X + Y = A6 X + A6 Y . The interested reader may find a rather extensive treatment of integration in the reference Vestrup (2003). Here is a brief description of the results obtained in Sections 4.1 and 4.2. For a simple r.v. X , the quantity I (X ) was defined, to be called eventually the integral of X . For two nonnegative simple r.v.s X and Y with X ≥ Y , it follows that I (X ) ≥ I (Y ); and I (X ) > I (Y ) if and only if X > Y on a set of positive μ-measure in the case that μ is finite. For a nonnegative r.v. X , the quantity I (X ) is defined, and it is shown that it is well defined. and set I (X ) = X dμ or just For any r.v. X , define I (X), provided it exists, X . Also, for A ∈ A, define A X dμ or just A X , and also define integrability of a r.v. X . For two nonnegative r.v.s X , Y , it holds (X + Y ) = X + Y . For A, B in A with A ∩ B = ∅, and any r.v. X for which A+B X exists, it follows that A X , B X also exist and A+B X = A X + B X . Furthermore, if the assumption is that all three A X , B X , and A X + B X exist, then A+B X also exists and A+B X = A X + B X .

65

66


X exists, then so does A X for every A ∈ A. Furthermore, if X is finite, so is A X for every A ∈ A. If X exists, then cX exists for every c ∈ , and cX = c X . X ≥ 0 implies X ≥ 0. If X ≤ Y and X and Y exist, then X ≤ Y ; if X ≤ Y and X −< ∞, then + X exists and X ≤ Y ; if X ≤ Y and Y < ∞, then X exists and X ≤ Y . If X = Y a.s. and X exists, then Y also exists and X = Y. X < ∞ if and only if |X | < ∞. If |X | ≤ Y and Y < ∞, then X is integrable. If all three X , Y , and X + Y exist, then (X + Y ) exists and (X + Y ) = X + Y. If

4.3 Probability Distributions In this short section, we consider a r.v. X defined on the probability space (, A, P), and on B, define the set function PX as follows: PX (B) = P(X −1 (B)) = P(X ∈ B). Then it is easy to see that PX is a probability measure on B. Definition 4. The set function PX on B associated with the r.v. X is called the (probability) distribution of X . The point function FX : → [0, 1] defined by FX (x) = PX ((−∞, x]) = P(X ≤ x) is called the distribution function (d.f.) of X . (See Theorem 6 and the comments following it in Chapter 2.) It is easily seen that FX satisfies the following properties: (1) is nondecreasing; (2) de f

de f

is continuous from the right; (3) FX (+∞) = lim x→∞ FX (x) = 1; (4) FX (−∞) = lim x→−∞ FX (x) = 0. (See Theorem 6 and the comments following it in Chapter 2.) Conversely, a function on into [0, 1] that satisfies properties (1)–(4) uniquely defines a probability measure Q on B (by Theorem 7 in Chapter 2), and it is also true that there exists a r.v. X whose PX is this distribution Q. The simplest example would be that of taking (, A, P) = (, B, Q) and let X (ω) = ω , ω ∈ . Then, clearly, Q is the distribution of X , PX . Also, if Y ∼ U (0, 1) and X = F −1 (Y ), then X ∼ F, where F −1 (y) = inf{x ∈ ; F(x) ≥ y}. (See, e.g., Theorem 7, Chapter 2 in Roussas, 1997.) The following theorem is an important one in statistics, since it allows integration over the real line rather than over the abstract space . Theorem 13.

Let g : → be measurable. Then we have g(X )d P = g(x)d PX , E g(X ) =

also denoted by g(x)d FX , in the sense that if one side exists, so does the other, and they are equal. (See also the Appendix regrading the notation g(x)d FX .) Proof. We use the familiar method of proving the theorem in several steps starting with indicator functions. Let g = I B some B ∈ B. Then g(X ) = I A (X ), where

4.3 Probability Distributions

A = X −1 (B). Then, g(X )d P = I A (X )d P = P(A) = P(X −1 (B)) = PX (B) = I B (x)d PX = g(x)d PX .

n Next, let g be a nonnegative simple function; i.e., g(x) = i=1 αi I Bi (x) with αi > a (measurable) partition of , and let Ai = 0, i = 1, . . . , n, where {B1 , . . . , Bn } is n αi I Ai (X ), and by linearity of the integral X −1 (Bi ), i = 1, . . . , n. Then g(X ) = i=1 (see also Exercise 8) and the previous step, n n g(X )d P = αi I Ai (X )d P = αi I Bi (x)d PX

i=1

=

n

i=1

αi I Bi (x) d PX =

i=1

g(x)d PX .

Now, let rgn be nonnegative. Then there exist 0 ≤ gn (x) simple ↑ g(x); i.e., α I (x) (αni ≥ 0, i = 1, . . . , rn ), which implies that 0 ≤ gn (x) = rni=1 ni Bni −1 where Ani = X (Bni ). Then ↑ g(X ) as n → ∞ gn (X ) = i=1 αniI Ani (X ) simple → g(X )d P, gn (x)d PX → g(x)d PX , whereas gn (X ) gn (X )d P n→∞ n→∞ d P = gn (x)d PX for all n, by the previous step. Hence g(X )d P = g(x)d PX . + − + Finally, for any g, write g(x) = g (x) − g (x), which implies g(X ) = g (X ) − − g (X ). Now, if g(X )d P exists, it then follows that either g + (X )d P < ∞ or g − (X )d P < ∞

or both. Since g + (X )d P = g + (x)d PX and g − (X )d P = g − (x)d PX , by the previous step, it follows that either + g (x)d PX < ∞ or g − (x)d PX < ∞

or both, respectively. Thus, g(x)d PX exists and + g(X )d P = g (X )d P − g − (X )d P = g + (x)d PX − g − (x)d PX = g(x)d PX . Likewise, the existence of equality.

g(x)d PX implies the existence of

g(X )d P and their

67

68


Remark 5. The proof is exactly the same for k-dimensional random vectors. However, we do not intend to present details. Exercises. 1. Construct an example where r.v.s X n and X , defined on a probability space a.s.

P

n→∞

n→∞

(, A, P), are such that X n → X pointwise (hence X n → X and X n → X ), n→∞

but E X n E X .

n→∞ 2. If the r.v. X ≥ 0 and X dμ = 0, then show that μ(X = 0) = 0. 3. If the r.v. X takes on the values n, n = 1, 2, . . ., with probability 1, so that ∞ ∞ n=1 P(X = n) = 1, show that E X = n=1 P(X ≥ n). 4. (i) For a r.v. X and any two disjoint events A and B, show that

(X I A+B )+ = (X I A )+ + (X I B )+ , (X I A+B )− = (X I A )− + (X I B )− . (ii) More generally, for any finite collection of pairwise disjoint events Ai , i = 1, . . . , n, show that

n X Ii=1 Ai

+

=

n i=1

X I Ai

n −

+

− n X I Ai . , X I i=1 = Ai i=1

5. If for the pairwise disjoint events Ai , i = 1, . . . , n, the integral n Ai X exists, ni=1 then the integrals Ai X , i = 1, . . . , n, exist, and n Ai X = i=1 X. i=1 Ai 6. Let X and Y be two X = Y= ∞ or simple r.v.s such that X = Y = −∞. Then show that (X + Y ) exists and (X + Y ) = X + Y (= ∞ or −∞). be two measures on (, A) and suppose that X d(μ1 + μ2 ) 7. Let μ1 and μ 2 exists. Then X dμi exist, i = 1, 2, and X d(μ1 + μ2 ) = X dμ 1 + X dμ2 . n , i = 1, . . . , n, are integrable, then so is the r.v. i=1 X i and 8. If the r.v.s X i n n X X = . i i i=1 i=1 9. Let X and Y be r.v.s with finite second moments, and set E X = μ1 , EY = μ2 , 0 < V ar (X ) = σ12 , 0 < V ar (Y ) = σ22 . Then the covariance and the correlation coefficient of X and Y , denoted respectively, by Cov(X , Y ) and ρ(X , Y ), are defined by: Cov(X , Y ) = E[(X −μ1 )(Y −μ2 )] = E(X Y )−μ1 μ2 and ρ(X , Y ) = Cov(X , Y )/σ1 σ2 . (i) Then show that −σ1 σ2 ≤ Cov(X , Y ) ≤ σ1 σ2 , and Cov(X , Y ) = σ1 σ2 if and only if P Y = μ2 + σσ21 (X − μ1 ) = 1, and Cov(X , Y ) = −σ1 σ2 if and only if P Y = μ2 − σσ21 (X − μ1 ) = 1. (ii) Also, −1 ≤ ρ(X , Y ) ≤ 1, and ρ(X , Y ) = 1 if and only if σ2 P Y = μ2 + (X − μ1 ) = 1, σ1

4.3 Probability Distributions

and ρ(X , Y ) = −1 if and only if P Y = μ2 −

σ2 σ1 (X

− μ1 ) = 1.

Hint: Use Exercise 2. 10. Let = {−2, −1, 3, 7}, and let the measure μ be defined by: μ({−2}) = 2, μ({−1}) = 1, μ({3}) = 3, μ({7}) = 7. Let X be a r.v. defined by: X (−2) = X (−1) = −1, and X (3) = X (7) = 1. Then, for A = {−2, 3, 7}, compute the integral A X dμ. 11. Let (, A, μ) = ((−5, 5), B(−5,5) , λ) where λ is the Lebesgue measure, and let the r.v. X be defined as follows: ⎧1 , ω ∈ (−5, 2) ⎪ ⎪ ⎪ 21 ⎨ ,ω=2 X (ω) = 3 ⎪ 1, ω ∈ (2, 3] ⎪ ⎪ ⎩ 0, ω ∈ (3, 5). Then, for A = [−1, 4], compute the integral A X dμ. 12. Let = {0, 1, 2, . . .}, let A be the discrete σ -field of subsets of , and on A, define the function μ by: μ(A) = number of nonnegative integers in A. Then show that μ is a measure, and indeed, a σ -finite measure. 13. Let g : → (0, ∞) be nondecreasing in (0, ∞) and symmetric about 0 (g(−x) = g(x), x ∈ ), and let X be a r.v. such that E g(X ) < ∞. Then show that: P(|X | ≥ c) ≤ E g(X )/g(c) for every c > 0. 14. For a r.v. X , show that |X | 1 E for every c > 0. P(|X | ≥ c) ≤ 1 + c 1 + |X | Hint: Use Exercise 13 above. 15. Let X be a simple r.v. defined on (, A), so that (see Definition 13 in Chapter 1) X = nj=1 α j I A j , where {A j , j = 1, . . . , n} is a (measurable) partition of , m and the α j , j = 1, . . . , n, are assumed to be distinct. Next, let X = i=1 βi I Bi be any other representation of X . Then show that each A j , j = 1, . . . , n, is the sum of some Bi ’s. Remark 6. Under the assumption that the α j , j = 1, . . . , n, are distinct, the partition {A j , j = 1, . . . , n} may be called irreducible. For such a partition, there is a unique representation of the respective simple r.v. and consequently, Theorem 1 is rendered superfluous. 16. If X and Y are two identically distributed integrable r.v.s, then E X I(|X |≤c) = E Y I(|Y |≤c) for any constant c.

69

CHAPTER

5

Standard Convergence Theorems, The Fubini Theorem

This chapter consists of two sections. In the first section, we discuss the standard convergence theorems, such as the Lebesgue Monotone Convergence Theorem, the Fatou–Lebesgue Theorem, and the Dominated Convergence Theorem. The Lebesgue Monotone Convergence Theorem and the Dominated Convergence Theorem provide conditions under which, from the limit of an integral, one can pass to the integral of the limit. The Fatou–Lebesgue Theorem deals with inequalities, involving lim inf and lim sup, rather than equalities as was the case in the former two theorems. As an application, two theorems are discussed, on the basis of which the interchange of the operations of differentiation and integration is valid. This section is concluded with the introduction of the concept of convergence in distribution. Its relation to convergence in probability is mentioned, and a result, particularly useful in Statistics, is stated, involving convergence in distribution of two sequences of r.v.s. The purpose of the second section is to establish the Fubini Theorem. This result gives conditions under which the interchange of the order of integration, involving a double integral, is valid. A number of auxiliary results have to be established first, and these include the so-called Product Measure Theorem.

5.1 Standard Convergence Theorems and Some of Their Ramifications Theorem 1 (Lebesgue Monotone Convergence Theorem; interchange of the order of lim and for ≥ 0 nondecreasing sequences If 0 ≤ X n ↑ X as n→ ∞, of r.v.s). where X may be an extended r.v., then X n ↑ X as n → ∞ (i.e., lim X n = n→∞ lim X n ). n→∞

Proof. In the sequel all limits are specified. taken as n → ∞ unlesss otherwise 0 ≤ X n ↑ X implies that X n , X exist, and X n ↑ implies that X n ↑. So, to show X n ↑ X , let 0 ≤ X nk simple r.v.s ↑ X n , k → ∞, and define the r.v.s Yk by Yk = max X nk , k = 1, 2, . . .: 1≤n≤k

X 11 X 12 · · ·

X 1k

X 21 X 22 · · · X 2k · · · · · · ·

···

↑ X1

· · · ↑ X2 · · · ·


71

72

CHAPTER 5 Standard Convergence Theorems, The Fubini Theorem

X k1 X k2 · · ·

X kk

···

↑ Xk

· · · · · · · · · · · X n1 X n2 · · · X nk · · · ↑ X n ·

·

·

·

·

·

·

·

·

·

·

Then, clearly, 0 ≤ Yk simple ↑, X nk ≤ Yk , n = 1, . . . , k. Next, Yk ≤ X k . In fact, since Yk = max X nk , we have that for ω ∈ there exists 1 ≤ n o (ω) ≤ k such 1≤n≤k

that Yk (ω) = X n o (ω),k . But X n o (ω),k (ω) ≤ X n o (ω) (ω) since X n o,k ↑ X n o as k → ∞, and X n o (ω) (ω) ≤ X k (ω), since n o (ω) ≤ k and X n ↑. Thus Yk ≤ X k and therefore X nk ≤ Yk ≤ X k , n = 1, . . . , k. Keeping n fixed and letting k → ∞, we get X n ≤ lim Yk ≤ X . k→∞

Now letting n → ∞, we get X ≤ lim Yk ≤ X , so that lim Yk = X . So 0 ≤ Yk simpler r.v.s ↑ X k→∞ k→∞ implies that Yk ↑ X as k → ∞. From Yk ≤ X k ≤ X , we get Yk ≤ X k ≤ X so that lim Yk = X ≤ lim sup X k ≤ X . k→∞

k→∞

So X k → X as k → ∞, and of course, X k ↑ X . In other words, lim X n = lim X n . Remark 1. Clearly, the convergence X n ↑ X may be only a.e., and X n ≥ 0 may also be true only a.e. Corollary 1 (Interchanging the and in a series of ≥ 0 r.v.s). Xn. (i) If X n ≥ 0, n = 1, 2, . . . , then n Xn = n (ii) If the r.v. X ≥ 0 a.e. and {A j , j ≥ 1} are pairwise disjoint, then A j X = j j Aj X. Proof. ∞ (i) Let Yn = nj=1 X j . Then 0 ≤ Yn ↑ ∞ X j and hence Yn ↑ X . j=1 n n n ∞ j=1 j X X X X = . Thus lim = But Yn = j j j = j=1 j j=1 j=1 j=1 ∞ j=1 X j . (ii) = X = X IA j X = X I I A j j j X IA j = j j Aj j Aj (by part (i)) and this equals to j Aj X.

5.1 Standard Convergence Theorems and Some of Their Ramifications

a.e. Corollary 2. If Y ≤ X n ↑ X (a possibly extended r.v.) and Y < ∞, then a.e. X n ↑ X (i.e., here 0 is replaced by the integrable r.v. Y ). Proof. From Y ≤ X n a.e., we have Y + ≤ X n+ a.e. and X n− ≤ Y − a.e., so that X n exists. Next, Y ≤ X n ↑ X implies 0 ≤ X n − Y ↑ X − Y and this, by the theorem, a.e. a.e. a.e. a.e. implies (X n − Y ) ↑ (X − Y ) or (by Theorem 12 in Chapter 4) Xn ↑ X X n − Y ↑ X − Y or since Y < ∞. Theorem 2 (Fatou–Lebesgue Theorem; interchange of

and lim inf, lim sup, lim).

a.e.

(i) If the r.v.s Y , X n , n = 1, 2, . . . are such that Y ≤ ≤X n , n = 1, 2, . . . , and Y is integrable, then lim inf X n ≤ lim inf X n . n→∞ n→∞ Xn ≤ (ii) If X n ≤ Z , n = 1, 2, . . . , and Z is integrable, then lim sup a.e. n→∞ lim sup X n . n→∞ a.e.

a.e.

a.e.

(iii) If Y ≤ X n ≤ Z , n = 1, 2, . . . , where Y and Z are as above and X n → X , a n→∞ X and |X n | → |X | . possibly extended r.v., then X n → n→∞

Proof.

n→∞

In what follows, all limits are taken as n → ∞.

(i) Assume first that 0 ≤ X n , n = 1, 2, . . . , and define Yn by Yn = inf X k . Then k≥n

a.e.

lim Yn = lim inf X k = lim inf X n and this convergence is, clearly, ↑; i.e., k≥n

a.e.

0 ≤ Yn ↑ lim inf X n . Then the Lebesgue Monotone Convergence Theorem n applies and gives: Yn ↑ lim inf X n . On the other hand, Yn ≤ X n . Thus X n and lim Yn ≤ lim inf X n , so that Yn ≤ lim inf X n ≤ lim inf X n . a.e.

In the general case, we consider X n −Y that are ≥ 0, n = 1, 2, . . . , and apply the previous results. Specifically, Y ≤ lim inf X n a.e. implies (lim inf X n )− ≤ Y − a.e., so that lim inf X n exists. We then get lim inf(X n − Y ) ≤ lim inf (X n − Y ) or lim inf X n − Y ≤ lim inf X n − Y

73

74


or

lim inf X n ≤ lim inf

X n , since

Y is finite.

Then, by part (i), (ii) X n ≤ Z a.e. or −Z ≤ −X n a.e. and −Z is integrable. lim inf(−X n ) ≤ lim inf −X n = lim inf(− X n ). But lim inf(−X n ) = − lim sup X n . Then

− lim sup X n ≤ lim inf − X n = − lim sup X n or

lim sup X n ≤ − lim sup X n or lim sup X n ≤ lim sup X n .

−

a.e. (iii) Y ≤ X n , n ≥ 1, Y < ∞, imply, by (i), lim inf X n ≤ lim inf X n . a.e. Also, X n ≤ Z , n ≥ 1, | Z | < ∞; thus, by (ii), lim sup Xn ≤ lim sup X n . Hence: lim inf X n = X ≤ lim inf X n ≤ lim sup X n ≤ lim sup X n = X . Thus X n → X . Next, Y ≤ X n ≤ Z or − Z ≤ −X n ≤ −Y so that 0 ≤ |X n | ≤ |Y | + |Z | a.e. and (|Y | + |Z |) < ∞. Since |X n | → |X |, we get |X n | → |X |.

We proceed with a definition and a lemma. Definition 1. Let μ be a measure on A and let ϕ be a set function on A. We say that ϕ is absolutely continuous with respect to μ, and write ϕ μ, if for every A ∈ A for which μ(A) = 0, we have ϕ(A) = 0. Lemma 1. Let ϕ be a nonnegative, σ -additive, and finite set function. Then ϕ μ if and only if, for every ε > 0, there exists δ = δ(ε) > 0 such that μ(A) < δ imply ϕ(A) < ε. For its proof, refer to Exercise 6. Also, see Theorem 27.1 on page 191 in Munroe (1953). Corollary. Let X be integrable and define the finite and ≥0 set function ϕ (actually, finite measure) as follows: ϕ(A) = A |X | dμ, A ∈ A. Then ϕ μ. Remark 2. ϕ as defined above is called the indefinite integral of |X |, and is σ additive on account of Corollary 1(ii) to Theorem 1. Proof of Corollary.

See Remark 4 in Chapter 4.


Alternatively, let the r.v.s X n be such that ⎧ ⎨ X (ω) if |X (ω)| < n if X (ω) ≥ n X n (ω) = n ⎩ −n if X (ω) ≤ −n, n = 1, 2, . . . (so that |X n | ≤ n). Then, clearly, 0 ≤ |Xn | ↑ |X | as n → ∞εand hence, as n → ∞ |X n |ε↑ |X | < ∞. Choose n o such that |X | < |X n | + 2 , n ≥ n o . Set δ = δ(ε) = 2n 0 and let A be such that μ(A) < δ. Then X n ≤ n o μ(A) + ε , |X | = |X | − X no + ϕ(A) = o 2 A A A A ε by the fact that X n o ≤ n o and the choice of n o . This is < n o × 2n o + 2ε ; i.e., ϕ(A) < ε. Theorem 3 (Dominated Convergence Theorem; interchange of and lim). If |X n | ≤ a.e.

μ

Y a.e., n = 1, 2, . . . with Y integrable and either (a) X n → X or (b) X n → X , n→∞ n→∞ then (i) A X n → A X uniformly in A ∈ A. n→∞ (ii) Xn → X. n→∞ |X n − X | → 0. (iii) n→∞

(iv) (i) and (iii) are equivalent. Remark 3.

(i) In the proof we will use thefollowing property: If Z exists, then Z ≤ |Z |, as was seen in Theorem 8 (iv) of Chapter 4. (ii) Part (iv) is true under an integrability assumption only of the r.v.s involved. This integrability is ensured here by the assumptions made (see parts (a) and (b) in the proof of the theorem.) Proof of Theorem 3. All limits that follow are taken as n → ∞ unless otherwise specified. We first establish (iv) under either mode of convergence of X n to X . (iv) Let that |X n − X | → 0. Then Xn − |X n − X | ≤ |X n − X | → 0 X = (X n − X ) ≤ A

A

A

A

independently of A. So, A X n → A X uniformly in A ∈ A. Next, |X n − X | = (X n − X )+ + (X n − X )− = (X n − X ) − (X n − X ) (X −X ≥0) (X n −X 1/r Yr (ω) = Z (ω) if Z (ω) ≤ 1/r .

(5.1)

Then, clearly, 0 ≤ Yr ≤ Z and Yr → 0 (since 0 ≤ Yr ≤ r1 everywhere), and r →∞ this implies Yr → 0 by Theorem 2 (iii). Now, let Z n (ω) < r1 . Then, if Z (ω) > r1 , r →∞

it follows from (5.1) that Yr (ω) = 1/r and hence Z n (ω) ≤ Yr (ω). If Z (ω) ≤ r1 , then Yr (ω) = Z (ω) by (5.1), and hence Z n (ω) ≤ Yr (ω) since always Z n ≤ Z . To summarize, Z n < r1 implies Z n ≤ Yr and so Zn ≤ Yr ≤ Yr → 0. (5.2) r →∞

(Z n < r1 )

(Z n < r1 )

μ

Now, Z n → 0 implies μ(Z n ≥ 1/r ) → 0. So, if we define ϕ(A) =

n→∞

A, then ϕ μ and ϕ is finite because Z is integrable. Therefore Z → 0 (by Lemma 1). n→∞

A

Z dμ, A ∈

(5.3)

(Z n ≥ r1 )

By Z n ≤ Z and (5.2) and (5.3), we get then Zn = Zn + Z n ≤ Yr + (Z n < r1 )

(Z n ≥ r1 )

by letting n → ∞ first and then by letting r → ∞.

Z →0

(Z n ≥ r1 )

Remark 4. Theorems 1–3 remain true if the set {1, 2, . . .} is replaced by T ⊆

and n → ∞ is replaced by t → to , t, to ∈ T . The following two theorems provide sufficient conditions that allow the interchange of the order of executing the operations of integration and differentiation.


d Theorem 4 (Interchange of dt and ). Let T ⊆ and let to be an interior point of T . Let X = X (ω, t) be a real-valued function on × T that is an integrable r.v. X (·,t)−X (·,to ) a.e. ∂X for each t ∈ T and such that ( ∂t )to exists for a.e. ω ∈ and ≤Y t−to ∂ X (·,t) d integrable for all t in some neighborhood of to . Then the dt X (·, t) and ∂t exist and are finite for t = t0 , and they are equal; i.e.,

∂ X (·, t) d . X (·, t) = dt ∂t t0 to (·,to ) 1 X (·, t) − X (·, to ) = X (·,t)−X . Proof. We have t−t t−to 0 X (·,t)−X (·,to ) a.e. ∂ X (·,t) → ( ∂t )to since ( ∂∂tX )to exists a.e., while for ts in a neight−to t→to ∂ X (·,t) (·,to ) a.e. ≤ Y a.e. and hence borhood of to , X (·,t)−X ≤ Y integrable, so that ∂t t−to t

Now

∂ X (·,t) ∂t to

o

is integrable. Then Remark 4 applies and gives

lim

t→to

∂ X (·, t) X (·, t) − X (·, to ) X (·, t) − X (·, to ) = lim = . t→to t − to t − to ∂t to

Since the left-hand side is equal to lim

t→to

X (·,t)− X (·,to ) d , it follows that dt t−to

X (·, t) t

o

is finite, and the asserted equality holds. d Theorem 5 (Interchange of dt and ). Let T = [α, β] ⊂ and let X = X (ω, t) be defined on × T into and be such that: X is an integrable r.v. for each t ∈ T , ∂ X∂t(·,t) a.e. exists for a.e. ω and all t ∈ T , ∂ X∂t(·,t) ≤ Y integrable, for all t ∈ T . Then, for each to ∈ [α, β],

∂ X (·, t) d . X (·, t) = dt ∂t to to Remark 5. respectively.

For t = α or t = β, we mean derivative from the right or left,

Proof of Theorem 5. Calculus, a.e.:

We have, by the Mean Value Theorem of Differential

∂ X (ω, ·) X (ω, t) − X (ω, to ) = (t − to ) ∂t

t ∗ (ω)

,

where t ∗ (ω) t, to . lies between ∂ X (·,t) a.e. (·,to ) Since ∂t ≤ Y integrable, for all t ∈ T , we get X (·,t)−X ≤ Y a.e. Then t−to Theorem 4 applies and gives the result. Application 1. Results such as Theorems 4 and 5 have wide applicability in statistics. They are employed, e.g., when establishing the Cramér–Rao inequality, asymptotic normality of the Maximum Likelihood Estimate, and in many other instances.

77

78


This section is concluded with the concept of convergence in distribution, and two results involving convergence in distribution. To this end, let X n , n ≥ 1, and X be r.v.s defined on the probability space (, A, P), and let Fn and F be their d.f.s, respectively. Also, let C(F) be the continuity set of F (the set on which F is continuous). Then d

We say that {X n } converges in distribution to X and write X n → X ,

Definition 2.

n→∞

if Fn (x) → F(x) for all x ∈ C(F). We also denote this convergence by writing n→∞

Fn ⇒ F and call it weak convergence of {Fn } to F.

n→∞

The following theorem relates convergence in probability and convergence in distribution. P

d

n→∞

n→∞

If X n → X , then X n → X . The inverse is not true, in general. It

Theorem 6.

is true, however, if P(X = c) = 1.

For its proof, see, e.g., page 168 in Loève (1963), or page 183 in Roussas (1997). At this point, it should be mentioned that, although the book Loève (1963) is used as a standard reference here, there are of, course, other books which present a thorough treatment of probability; e.g., Shiryaev (1995) is such a book. In the following example convergence in distribution does not imply convergence in probability. Example 1. Let = {1, 2, 3, 4}, A = P(), P({1}) = P({2}) = P({3}) = P({4}) = 41 . Define X n , n ≥ 1, and X as follows:

X n (1) = X n (2) = 1 X n (3) = X n (4) = 0,

n≥1

;

X (1) = X (2) = 0 X (3) = X (4) = 1.

P

Then |X n − X | = 1 for all ω ∈ . Hence X n X , clearly. Next n→∞ ⎧ ⎧ ⎨ 0, x < 0 ⎨ 0, x < 0 FX n (x) = 21 , 0 ≤ x < 1 ; FX (x) = 21 , 0 ≤ x < 1 ; ⎩ ⎩ 1, x ≥ 1 1, x ≥ 1 P

i.e., FX n (x) = FX (x), x ∈ and hence FX n ⇒ FX , while X n X . n→∞

n→∞

The following theorem is very useful in statistics. Let X n , Yn , n ≥ 1, and X be r.v.s such that FX n ⇒ FX (or

Theorem 7 (Slutsky).

n→∞

d

P

n→∞

n→∞

X n → X ) and Yn → c. Then d

(i ) X n ± Yn → X ± c,

(i) FX n ±Yn ⇒ FX ±c ,

n→∞

n→∞

(ii) FX n Yn ⇒ FcX , n→∞

(iii) F X n ⇒ F X , c = 0. Yn

Remark 6.

Xn Yn

n→∞

c

; or

d

(ii ) X n Yn → cX , (iii )

n→∞ d Xn → Xc . Yn n→∞

P

is well defined with probability → 1, since Yn → c = 0. n→∞

n→∞


For the proof of the theorem, see, e.g., pages 102–103 in Rao (1965), or page 201 in Roussas (1997). Example 2. In this example, we use some of the results obtained so far to show that, under certain conditions and in a certain sense, convergence in distribution is preserved. To this end, let X n , n = 1, 2, . . . , be r.v.s, let g : → be differentiable, and let its derivative g (x) be continuous at a point d. Also, let cn be constants such that d

d

n→∞

n→∞

0 = cn → ∞, and let cn (X n −d) → X , a r.v. Then cn [g(X n )−g(d)] → g (d)X . n→∞

All of the following limits are taken as n → ∞. In the first place, by assumption, d

d

P

cn (X n −d) → X and cn−1 → 0, so that X n −d → 0, by Theorem 7 (ii), or X n −d → 0, P

by Theorem 6. Hence |X n − d| → 0, by Exercise 8 in Chapter 3. Next, expand g(X n ) around d according to Taylor’s formula in order to obtain g(X n ) = g(d) + (X n − d)g (X n∗ ), where X n∗ is a r.v. lying between d and X n . Hence cn [g(X n ) − g(d)] = cn (X n − d)g (X n∗ ). P

P

P

However, |X n∗ − d| ≤ |X n − d| → 0, so that X n∗ → d, and hence g (X n∗ ) → g (d), by the continuity of g (x) at d and the exercise cited above. Then, by Theorem 7 (ii), d

d

cn (X n − d)g (X n∗ ) → g (d)X , and therefore cn [g(X n ) − g(d)] → g (d)X .

An application of the result discussed in the previous example is given below. Example 3. (i) Let X 1 , . . . , X n be i.i.d. r.v.s with mean μ ∈ and variance σ 2 ∈ (0, ∞), and let g : → be differentiable with derivative continuous at μ. Then √

n[g( X¯ n ) − g(μ)] → N (0, [σ g (μ)]2 ), d

n→∞

where X¯ n is the sample mean of the X j s. (ii) In particular, if the X j s are distributed as B(1, p), then √ d n [ X¯ n (1 − X¯ n ) − pq] → N (0, pq(1 − 2 p)2 ), n→∞

where q = 1 − p. √

n( X¯ n − μ) → X ∼ √ n→∞ N (0, σ 2 ). Then the assumptions of Example 2 are fulfilled, so that n[g( X¯ n ) −

(i) Indeed, the Central Limit Theorem (CLT) gives that

d

d

g(μ)] → g (μ)X ∼ N (0, [σ g (μ)]2 ). n→∞

(ii) Here μ = p and σ 2 = pq, and take g(x) = x(1 − x), 0 < x < 1. Then g( X¯ n ) = X¯ n (1 − X¯ n ), and g (x) = 1 − 2x, so that g ( p) = 1 − 2 p. The result then follows from part (i).

79

80


The result obtained in Example 2 and its applications in Example 3 are often referred to as the delta method. See also Exercise 26 in Chapter 11. Below, a simple application of an instance of Theorem 7 (iii) is given in a testing hypothesis problem. Application 2. Let X 1 , . . . , X n be i.i.d. N (μ, σ 2 ) r.v.s. For testing H : μ = μo against A : μ = μo , say, at level of significance α (where σ is unknown), one uses the t-test and determines the cutoff point c by the requirement that √ n( X¯ n − μo ) P(|tn | > c) = α, where t = 1 ¯ 2 j (X j − X n ) n−1 is tn−1 distributed under H . Now, whether the X s are normal or not, we set √ ( X¯ n −μo ) √ n σ n( X¯ n − μo ) d and have as n → ∞, → N (0, 1), tn = (H ) σ 1 2 /σ 2 ¯ (X − X ) j n j n−1 1 1 P (X j − X¯ n )2 → 1, σ2 n − 1 j

d

provided, of course, μ, σ 2 are finite (in the nonnormal case). Then as n → ∞, tn →

(H )

Z ∼N (0, 1), and hence the size α of the test will be intact for large n no matter whether the normality assumption is valid or not. This is known as the robustness property of the t-test. In a probability framework, convergence almost everywhere (a.e.) becomes almost sure (a.s.) convergence, and convergence in measure becomes convergence in probability. These modes of convergences and convergence in distribution introduced earlier are related as follows, on the basis of the Corollary to Theorem 4 in Chapter 3, and Theorem 3 here: a.s.

P

d

X n → X implies X n → X implies X n → X , as n → ∞. Also, d

P

X n → X implies X n → X if P(X = c) = 1, but not otherwise, and P

a.s.

X n → X need not imply X n → X , as n → ∞. (For the last statement, see Exercise 2 (ii) in Chapter 3.)

5.2 Sections, Product Measure Theorem, the Fubini Theorem The content of this section is highly technical, and a brief outline of the basic concepts and results is as follows. Consider the σ -finite measure spaces (i , Ai , μi ), i = 1, 2, and the product measurable space (1 × 2 , A1 × A2 ).

5.2 Sections, Product Measure Theorem, the Fubini Theorem

First, for any E ∈ A1 × A2 , define 2 -sections of E at ω1 , to be denoted by E ω1 , and 1 -sections of E at ω2 , to be denoted by E ω2 , and show that they are measurable. Second, for a measurable function f : 1 × 2 → , define its 2 -section at ω1 , f ω1 (·) : E ω1 → , and the 1 -section of f at ω2 , f ω2 (·) : E ω2 → , and show that all these sections are measurable. Third, define the functions f : 1 → and g : 2 → by: f (ω1 ) = μ2 (E ω1 ) and g(ω2) = μ1 (E ω2 ). Then show that f and g are (nonnegative and) measurable, and that f dμ1 = gdμ2 . Fourth, on A1 × A2 , define λ by λ(E) = f dμ1 = gdμ2 . Then show that λ is a σ -finite measure, and that λ(A1 × A2 ) = μ1 (A1 )μ2 (A2 ), A1 ∈ A1 , A2 ∈ A2 . Because of this, λ is referred to as the product measure (of μ1 and μ2 ) and is denoted by μ1 × μ2 . Finally, consider the r.v. X : (1 × 2 , A1 × A2 , μ1 × μ2 ) → , and look at the following integrals (whose existence is assumed here): X (ω1 , ω2 )d(μ1 × μ2 ), X (ω1 , ω2 )dλ = X (ω1 , ω2 )dμ1 dμ2 = X (ω1 , ω2 )dμ1 dμ2 , X (ω1 , ω2 )dμ2 dμ1 . X (ω1 , ω2 )dμ2 dμ1 = Then conditions are given under which the above three integrals exist, and they are all equal. Definition 3.

For E ∈ A = A1 × A2 and for ω1 ∈ 1 , ω2 ∈ 2 , we define

E ω1 = {ω2 ∈ 2 ; (ω1 , ω2 ) ∈ E},

E ω2 = {ω1 ∈ 1 ; (ω1 , ω2 ) ∈ E}.

Clearly, E ω1 ⊆ 2 , E ω2 ⊆ 1 ; E ω1 is called an 2 -section of E at ω1 , and E ω2 is called an 1 -section of E at ω2 . Ω2 Eω1

E

ω2

Eω2

ω1

Ω1

Remark 7. Clearly, if E = A × B, then E ω1 = B or depending on whether ω1 ∈ A or ω1 ∈ / A, and similarly E ω2 = A or depending on whether ω2 ∈ B or / B. ω2 ∈

81

82


Theorem 8.

Every section of a measurable set is measurable.

Proof. Let C = {E ∈ A; every section of E is measurable}. Then, by the previous remark, C contains all rectangles, and furthermore it is a σ -field. In fact, ∞ let E n ∈ E , and let ω ∈ . Then E = C, n = 1, 2, . . . , let E = ∞ n 1 1 ω 1 n=1 n=1 E n , ω1 , as is easily seen (see Exercise 7(iii)). Since E n,ω1 ∈ A2 , n = 1, 2, . . . , we have that E ω1 ∈ A2 . Similarly, E ω2 ∈ A1 , ω2 ∈ 2 . Thus E ∈ C. Now if E ∈ C, then E c ∈ C. In fact, for ω1 ∈ 1 we have (E c )ω1 = (E ω1 )c , as is easily seen (see Exercise 7(v)). Since E ω1 ∈ A2 , we have E ωc 1 ∈ A2 . Similarly, E ωc 2 ∈ A1 , ω2 ∈ 2 . Thus E c ∈ C. As mentioned already, C contains the class of all rectangles in 1 ×2 . Hence A ⊆ C. Since also C ⊆ A by its definition, we have C = A. Let f : E ⊆ 1 × 2 → . Definition 4. For ω1 ∈ 1 , the function f ω1 , defined on E ω1 into by f ω1 (ω2 ) = f (ω1 , ω2 ), (ω1 , ω2 ) ∈ E, is called an 2 -section of f at ω1 . Similarly, for ω2 ∈ 2 , the function f ω2 , defined on E ω2 into by f ω2 (ω1 ) = f (ω1 , ω2 ), (ω1 , ω2 ) ∈ E, is called an 1 -section of f at ω2 . Theorem 9. If f : (1 ×2 , A1 ×A2 ) → ( , B) is measurable, then every section of it is also measurable. Proof.

Let B ∈ B. Then f ω−1 (B) = {ω2 ∈ 2 ; f ω1 (ω2 ) ∈ B} = {ω2 ∈ 2 ; f (ω1 , ω2 ) ∈ B} 1 = {ω2 ∈ 2 ; (ω1 , ω2 ) ∈ f −1 (B)} = ( f −1 (B))ω1 .

(B) is simply an Now f −1 (B) ∈ A1 × A2 by the measurability of f , and f ω−1 1 2 -section of f −1 (B) at ω1 , thus an A2 measurable set. So, f ω1 is A2 -measurable and in a similar fashion f ω2 is A1 -measurable. Consider now the σ -finite measure spaces (i , Ai , μi ), i = 1, 2, and let (1 × 2 , A1 × A2 ) be the product measurable space. For E ∈ A1 × A2 , define the functions f and g on 1 and 2 , respectively, as follows: f (ω1 ) = μ2 (E ω1 ), g(ω2 ) = μ1 (E ω2 ).

(5.4)

These functions can be defined since E ω1 and E ω2 are measurable (Theorem 8). With this notation, we have the following theorem. Theorem 10. For every E ∈ A = A1 × A2 ,the functions f and g as defined above are nonnegative, measurable, and f dμ1 = gdμ2 . Proof. Let M = {E ∈ A1 × A2 ; for the respective functions f and g defined by (5.4), the theorem is true}. Then (i) M = since clearly, 1 × 2 ∈ M. (ii) M is closed under countable sums. In fact,let E n ∈ M, n = 1, 2, . . . , with E i ∩ E j =, i = j, and n E n . To show that E ∈ M. set E = E (see also Exercise 7(iii)), we have Since E ω1 = ( n E n )ω1 = n n,ω1


f (ω1 ) = μ2 (E ω1 ) = μ2 ( n E n,ω1 ) = n μ2 (E n,ω1 ) = n f n (ω1 ), where f n (ω1 ) = μ2 (E n,ω1 ). That is, f (ω 1) = n f n (ω1 ). Now E n ∈ M implies that f n is ≥ 0, measurable. Thus f = n f n ≥ 0 and since nk=1 f k → (↑) f and n→∞ n f is measurable, it follows that f is measurable. Also, f = k n fn = k=1 f by Corollary 1 to Theorem 1. In a similar way, g ≥ 0 and measure n n f n dμ1 = and g= n gn = n gn , where gn (ω2 ) =μ1 (E n,ω2 ).But gn dμ2 , n = 1, 2, . . . , because E n ∈ M. Hence f dμ1 = gdμ2 . (iii) If E = A × B, then E ∈ M. In fact, f (ω1 ) = μ2 (E ω1 ) = μ2 (B)I A (ω1 ), B (ω2 ). Thus f , g are ≥ 0, measurable. Next, g(ω2 ) = μ1 (E ω2 ) = μ1 (A)I f dμ1 = μ2 (B)μ1 (A) = gμ2 = μ1 (A)μ2 (B). Hence f dμ1 = gdμ2 . (iv) If C is the field (by Theorem 7 in Chapter 1) of all finite sums of measurable rectangles in 1 × 2 , then C ⊆ M. This follows from (ii) and (iii). (v) M is a monotone class. Let first E n ∈ M, n = 1, 2, . . . . with E n ↑. Then to de f show that E = lim E n = n E n ∈ M. First, we notice that E n ↑, implies n→∞ E n,ω1 ↑, ω1 ∈ 1 (see also Exercise 7(i)), and E ω1 = n E n ω1 = n E n,ω1 (see also Exercise 7(iii)) or E ω1 = ( lim E n )ω1 = lim E n,ω1 . Next, with the n→∞ n→∞ limits taken everywhere as n → ∞, f (ω1 ) = μ2 (E ω1 ) = μ2 lim E n,ω1 = lim μ2 E n,ω1 = lim f n (ω1 ), where f n (ω1 ) = μ2 (E n,ω1 ); i.e., f (ω1 ) = lim f n (ω1 ), ω1 ∈ 1 .

Since f n ≥ 0, measurable, so is f . But fn (ω1 ) = μ2(E n,ω1 ) ≤ μ2 E n+1,ω1 = f n+1 (ω1 ); i.e., 0 ≤ f n ↑ f , and this implies f n dμ1 ↑ f dμ1 . In a similar fashion, 0 ≤ gn ↑ g, and this implies gn dμ2 ↑ gdμ2 , where g(ω2 ) = μ1 (E ω2 ), gn (ω2 ) = μ1 (E n,ω2 ). But f n dμ1 = gn dμ2 , since E n ∈ M. Hence f dμ1 = gdμ2 implies E ∈ M. de f Let now E n ∈ M, n = 1, 2, . . . , E n ↓. To show that E = lim E n = n E n ∈ M. First, thatμ1 and μ2 are finite. Again, E n ↓ E implies E n,ω1 ↓, and assume E = n E n,ω1 or E ω1 = (limE n )ω1 = limE n,ω1 , ω1 ∈ 1 (see also E ω1 = n n ω1 Exercice 7(iv)). Next, f (ω1 ) = μ2 (E ω1 ) = μ2 (limE n )ω1 = μ2 limE n,ω1 = lim μ2 E n,ω1 (by finiteness of μ2 ), = lim f n (ω1 ); f (ω1 ) = lim f n (ω1 ), ω1 ∈ 1 . Since f n ≥ 0, measurable, so is f . But f n (ω1 ) = μ2 (E n,ω1 ) ≥ μ2 E n+1,ω1 = f n+1 (ω1 ), and f 1 (ω1 ) = μ2 (E 1,ω1 ) ≤ μ2 (2 ) < ∞, ω1 ∈ 1 . Thus 0 ≤ fn ≤ μ2 (2 ) < ∞ with f n → f . Then Theorem 2 (iii) implies f n dμ1 → f dμ1 . Similarly, 0 ≤ gn measurable ≤ μ1 (1 ) < ∞ and gn → g, where g(ω2 ) = μ1 (E ω2 ), gn (ω2 ) = μ1 (E n,ω2 ), ω2 ∈ 2 , and hence gn dμ2 →

83

84


gdμ2 . But f n dμ1 = gn dμ2 , since E n ∈ M. Hence f dμ1 = gdμ2 , which implies that E ∈ M. Now consider the case where μ1 and μ2 are σ -finite. Their σ -finiteness implies the existence of partitions {Ai , i = 1, 2, . . .} and {B j , j = 1, 2, . . .} of 1 and 2 , respectively, for which μ1 (Ai ) < ∞ and μ2 (B j ) < ∞ for all i and j. For each i and j, define on A1 and A2 , respectively, the finite measures μ1i (A) = μ1 (A ∩ Ai ) and μ2 j (B) = μ2 (B ∩ B j ) (see Exercise 8). For any E ∈ A1 × A2 , set f (ω1 ) = μ2 (E ω1 ),

f j (ω1 ) = μ2 j (E ω1 ) = μ2 E ∩ (Ai × B j ) ω with ω1 ∈ Ai 1

= μ2 (E ω1 ∩ B j ), g(ω2 ) = μ1 (E ω2 ),

gi (ω2 ) = μ1i (E ω2 ) = μ1 E ∩ (Ai × B j ) ω with ω2 ∈ B j 2

= μ1 (E ω2 ∩ Ai ). Then observe that f (ω1 ) =

∞

f j (ω1 ) and g(ω2 ) =

j=1

∞

gi (ω2 ).

(5.5)

i=1

It follows that f j , gi are ≥ 0, f j is A1 -measurable, gi is A2 -measurable, and on account of (5.5), so are f and g, respectively. Also, f j dμ1i = gi dμ2 j for all iand j. It follows that ∞ ∞ ∞ ∞ ∞ ∞ (5.6) f j dμ1i = gi dμ2 j = gi dμ2 j . i=1 j=1

i=1 j=1

j=1 i=1

However,

f j dμ1i =

⎛ ⎞ ⎝ f j ⎠ dμ1i (by Corollary 1 to Theorem 1)

j

=

j

(by the definition of f )

f dμ1i

=

(by Exercise 9),

f dμ1 Ai

and i

j

f j dμ1i =

i

f dμ1 = Ai

i

( f I Ai )dμ1 =

( f I Ai )dμ1 i


= f I Ai dμ1 i

= =

i Ai

gi dμ2 j =

i

and

gi dμ2 j =

i

gi dμ2 j =

gdμ2 j =

gdμ2 , Bj

j

(5.7)

i

j

(since {Ai ; i ≥ 1}is a partition of 1 ).

f dμ1

Likewise,

f dμ1 (by Corollary 1 to Theorem 1)

gdμ2 = Bj

j Bj

gdμ2 =

gdμ2 .

(5.8)

Relations (5.5)–(5.8) then yield f dμ1 = gdμ2 . Since, as already mentioned, and A2 -measurable, respectively, the the f and g are nonnegative and A1 -measurable proof is completed. Thus, in all cases, f dμ1 = gdμ2 and this implies E ∈ M. So we have that M is a nonempty monotone class containing C; hence, M contains the minimal monotone class over C, which is A1 × A2 (by Theorem 6 in Chapter 1). Since also M ⊆ A1 × A2 , it follows that M = A1 × A2 . Thus the theorem is true for every E ∈ A1 × A2 . Remark 8. The theorem need not be true if μ1 , μ2 are not σ -finite, as the following example shows. This fact also has repercussions to Theorems 11 and 12 later. Example 4. Let 1 = 2 = [0, 1], let A1 = A2 = B[0,1] , let μ1 be the Lebesgue measure on A1 , and define the set function μ2 on A2 by: μ2 (A) = number of points in A. Then 2 is an infinite μ ∞ measure since μ2 is nondecreasing, μ2 () = 0, and ∞ Ai ) = i=1 μ2 (Ai ) = ∞ for whatever Ai s in A2 . ∞ = μ2 ( i=1 a partition of [0, 1] with However, μ2 is not σ -finite. Indeed, if {A1 , A2 , . . .} is ∞ A s would have to be finite, and μ2 (Ai ) < ∞, then the i i=1 Ai = [0, 1], which ∞ is a contradiction as i=1 Ai is countable. Next, define the functions f ∗ and g ∗ as follows: f ∗ , g ∗ : [0, 1] × [0, 1] → [0, 1], f ∗ (x, y) = x, and g ∗ (x, y) = y. Then f ∗ and g ∗ are measurable, since, e.g., for any 0 ≤ x1 < x2 ≤ 1, ( f ∗ )−1 ([x1 , x2 ]) = [x1 , x2 ] × [0, 1], which is (A1 , A2 )-measurable, and likewise for g ∗ . It follows that h = f ∗ − g ∗ is also measurable. Now, let D be the main diagonal of the square [0, 1] × [0, 1]; i.e., D = {(x, y) ∈ [0, 1] × [0, 1]; x = y}. Then D is measurable, because h −1 ({0}) = D. Next, for each x ∈ [0, 1], the [0, 1]section of D at x, Dx , is Dx = {y ∈ [0, 1]; (x, y) ∈ D} = {y} (with y = x), and likewise D y = {x} (with x = y). Therefore the functions f and g defined in relation (5.4) are here f (x) = μ2 (Dx ) = μ2 ({y}) = 1 and g(y) = μ1 (D y ) = μ1 ({x}) = 0.

85

86


So, f (x) = 1 for all x ∈ [0, 1] and g(y) = 0 for all y ∈ [0, 1]. It follows that f dμ1 = 1 = 0 = gdμ2 [0,1]

[0,1]

(by following the convention that 0 × ∞ = 0).

Theorem 11 (Product Measure Theorem). Let (i , Ai , μi ), i = 1, 2, be two σ finite measure spaces. Define λ on A1 × A2 as follows: for E ∈ A1 × A2 , λ(E) = μ2 (E ω1 )dμ1 = μ1 (E ω2 )dμ2 . Then (i) (ii) (iii) (iv)

λ is a measure. If E = A × B, A ∈ A1 , B ∈ A2 , then λ(A × B) = μ1 (A)μ2 (B). λ is σ -finite. If μ is defined on the rectangles A × B, A ∈ A1 and B ∈ A2 , by μ(A × B) = the field C of finite sums of (measurable) μ1 (A)μ2 (B), and is extended to rectangles in 1 ×2 by μ(E) = ri=1 μ1 (Ai )μ2 (Bi ), where E = ri=1 Ai × Bi , Ai ∈ A1 , Bi ∈ A2 , i = 1, . . . , r , then μ is well defined on C, is a σ -finite measure on C, and λ is the unique extension of μ from C to A1 × A2 .

Remark 9. It should be mentioned at this point that the measure λ is instrumental in establishing the properties of the set function μ asserted in part (iv) of Theorem 11. Proof of Theorem 11. (i) From Theorem 10, if f(ω1 ) = μ2 (E ω1 ) and g(ω2 ) = μ1 (E ω2 ), then f , g are ≥ 0, measurable, and f dμ1 = gdμ2 . Thus, for E ∈ A1 × A2 , λ(E) is well defined. Next, λ is a measure. It suffices to prove that λ has the following properties: (a) λ() = 0, (b) λ is σ -additive, (c) λ is nondecreasing or nonnegative. That λ() = 0 is obvious. ∞ Let now E n ∈ A1 ×A 2 , n ≥ 1, E i ∩E j = , i = j. To show that n=1 λ(E n ) = ∞ λ(E), where E = n=1 E n . Let f , f n be defined by f (ω1 ) = μ2 (E ω1 ), f n (ω1 ) = μ2 (E n,ω1 ). But, as in Theorem 10 (ii), ⎛ ⎞ ∞ ∞ ∞ ⎠ = μ2 μ2 (E ω1 ) = μ2 ⎝ En E n,ω1 = μ2 (E n,ω1 ); n=1

ω1

i.e., f (ω1 ) =

n=1 ∞

n=1

f n (ω1 ), ω1 ∈ 1 .

n=1

Then, ∞ ∞ f n dμ1 = f n dμ1 (by Corollary 1 to Theorem 1), f dμ1 = n=1

or λ(E) =

∞ n=1

λ(E n ).

n=1


Finally, let E, F ∈ A1 × A2 with E ⊂ F. To show that λ(E) ≤ λ(F). Indeed, E ⊂ F implies E ω1 ⊆ Fω1 (see Exercise 7(i)), so that μ2 (E ω1 ) ≤ μ2 (Fω1 ).

Hence,

μ2 (E ω1 )dμ1 ≤

μ2 (E ω1 )dμ1 or λ(E) ≤ λ(F).

(Alternatively, λ(E) ≥ 0, trivially.) (ii) Let E = A × B, A ∈ A1 , B ∈ A2 . Then f (ω1 ) = μ2 (B)I A (ω1 ), and hence λ(E) = f dμ1 = μ1 (A)μ2 (B). (iii) Let {Ai , i = 1, 2, . . .}, {B j , j = 1, 2, . . .} be partitions of 1 , 2 , respectively, such that μ1 (Ai ) < ∞, μ2 (B j ) < ∞, i, j = 1, 2, . . . . Then {Ai × B j , i, j = 1, 2, . . .} is a partition of 1 × 2 and λ(Ai × B j ) = μ1 (Ai ) μ2 (B j ) < ∞, i, j = 1, 2, . . . . (iv) By part (ii), μ(A× B) = λ(A× B)(= μ1 (A)μ2 (B)). Next, if E = ri=1 Ai × Bi with Ai ∈ A1 and Bi ∈ A2 , i = 1, . . . , r , then r

μ(Ai × Bi ) =

i=1

r

μ1 (Ai )μ2 (Bi ) =

i=1

=λ

r

r

λ(Ai × Bi )

i=1

Ai × Bi

i=1

(since λis a measure on A1 × A2 ⊃ C) = λ(E). Likewise, if E =

s j=1 s

Aj × B j , then

μ(Aj × B j ) =

j=1

=

s j=1 s

μ1 (Aj )μ2 (B j ) λ(Aj × B j )

j=1 s

= λ(

Aj × B j )

j=1

= λ(E) so that μ is well defined on C. Furthermore, μ is a measure on C because μ = λ on C and λ is a measure on A1 × A2 ⊃ C. Finally, μ is σ -finite (by the proof of part (iii)). Hence by the Carathéodory Extension Theorem (Theorem 5 in Chapter 2), λ is the unique extension of μ from C to A1 × A2 . Definition 5. The measure λ as defined previously is the product measure of μ1 and μ2 denoted by μ1 × μ2 .

87

88


Corollary. A set E ∈ A1 × A2 is λ-null (λ(E) = 0) if and only if almost every section of it is null. Proof.

We assume μ1 (1 ), μ2 (2 ) > 0, since otherwise λ = 0. We have

λ(E) =

μ2 (E ω1 )dμ1 =

μ1 (E ω2 )dμ2 .

Hence λ(E) = 0 if and only if

μ2 (E ω1 )dμ1 =

μ1 (E ω2 )dμ2 = 0,

and since the integrants are ≥ 0, these relations imply μ2 (E ω1 ) = 0 a.e. [μ1 ], μ1 (E ω2 ) = 0 a.e. [μ2 ]. (See Exercise 2 in Chapter 4.)

5.2.1 Preliminaries for the Fubini Theorem Let (i , Ai , μi ), i = 1, 2, be two σ -finite measure spaces and consider the product measure space (1× 2 , A1 × A2 , μ1 × μ2 = λ). Let X be a r.v. defined on 1 × 2 and let that X dλ = X d( μ1 × μ2 ) exist. This integral is also called the double integral of X . Next, for fixed ω1 ∈ 1 , set X ω1 (ω2 ) = X (ω1 , ω2 ), and for fixed ω2 ∈ 2 , set X ω2 (ω1 ) = X (ω1 , ω2 ) (apply Definition 4 with E = 9, X ω1 (·), X ω2 (·) are A2 , A1 -measurable, respectively. We 1 × 2 ). By Theorem (·)dμ , X (·)dμ , exists and set f (ω ) = X (·)dμ = assume now that X ω 2 ω 1 1 ω 2 1 2 1 X (ω1 , ·)dμ2 , g(ω2 ) = X ω2 (·)dμ1 = X (·, ω2 )dμ1 . Then f and g are meaas is seen in the proof of Theorem 12 later, and assume that f dμ1 = surable, [ X (ω1 , ω2 )dμ2 ]dμ1 and gdμ2 = [ X (ω1 , ω2 )dμ1 ]dμ2 , exist. These integrals are also called iterated integrals. The question then arises: under what conditions de f X (ω1 , ω2 )dμ1 dμ2 X (ω1 , ω2 )dμ1 dμ2 = = X (ω1 , ω2 )dμ2 dμ1 de f = X (ω1 , ω2 )dμ2 dμ1 ? The answer to this question is given by the Fubini Theorem. Theorem 12 (The Fubini Theorem). Consider the product σ -finite measure space (1 × 2 , A1 × A2 , μ1 × μ2 = λ) and let on 1 × 2 that X be a r.v. defined dμ = X dμ2 dμ1 and their is either nonnegative or λ-integrable. Then X dμ 1 2 common value is X dλ.


Proof. Assume first that X is nonnegative (Halmos). Then all X dλ, X dμ1 dμ2 , X dμ2 dμ1 exist (since X ≥ 0, X ω1 (·) ≥ 0, X ω2 (·) ≥ 0, f (ω1 ) ≥ 0, g(ω2 ) ≥ 0, and assuming that f and g are appropriately measurable). We will present the proof in three steps. Assume first that X = I E for some E ∈ A1 × A2 . Then X (ω , ω ) = I E (ω1 , ω2 ) = I E ω2 (ω1 ), measurable for each fixed ω2 , and hence 1 2 X (ω1 , ω2 )dμ1 = I E ω2 (ω1 )dμ1 = μ1 (E ω2 ) (measurable, by Theorem 10), and μ1 (E ω2 )dμ2 . Similarly, X (ω1 , ω2 )dμ2 dμ1 = X (ω1 , ω2 )dμ1 dμ2 = μ = X dλ by μ2 (E ω1 )dμ1 . But μ2 (E ω1 )dμ1 = 1 (E ω2 )dμ2 = λ(E) Theorems 10 and 11. Thus X dμ1 dμ2 = X dμ2 dμ1 = X dλ. By linearity of the integral, the theorem is then true for nonnegative simple r.v.s. Next, let X be any nonnegative r.v., and in the sequel, take all limits as n → ∞. Then there exist 0 ≤ X n simple r.v.s ↑ X , and by the Monotone Convergence Theorem, X dλ. (5.9) X n dλ → For each fixed ω2 ∈ 2 , X n (·, ω2 ) and X (·, ω2 ) are (A1 -)measurable and 0 ≤ X n (·, ω2 ) ↑ X (·, ω2 ), so that, by the Monotone Convergence Theorem, (0 ≤) X n (·, ω2 )dμ1 ↑ X (·, ω2 )dμ1 . (5.10) But X n (·, ω2 )dμ1 is (A2 -)measurable (as a finite linear combination of integrals of indicators) by the previous step; also, X (·, ω2 )dμ1 is (A2 -)measurable as a limit of (A2 -)measurable r.v.s. Then from (5.10) and the Monotone Convergence Theorem, it follows that X (ω1 , ω2 )dμ1 dμ2 ↑ X (ω1 , ω2 )dμ1 dμ2 , or

X n dμ1 dμ2 →

Likewise,

X dμ1 dμ2 .

(5.11)

X dμ2 dμ1 .

(5.12)

X n dμ2 dμ1 →

However, for each n ≥ 1, X n dμ1 dμ2 = X n dμ2 dμ1 = X n dλ. Then, from (5.9)–(5.13), we get X dμ1 dμ2 = X dμ2 dμ1 = X n dλ, as was to be seen.

(5.13)

89

90


Now suppose that X is λ-integrable (Loève). Since X = X + − X − and the theorem is true for X + and X − , it follows that it is true for X . In more detail, apply the result just obtained to X + and X − to get X + dμ1 dμ2 = X + dμ2 dμ1 = X + dλ, X − dμ1 dμ2 = X − dμ2 dμ1 = X − dλ, so that

(X + − X − )dμ1 dμ2 =

or

−

X dμ1 dμ2 − X dμ1 dμ2 = X + dμ2 dμ1 − X − dμ2 dμ1 = X + dλ − X − dλ,

or

+

(X + − X − )dμ2 dμ1 =

X dμ1 dμ2 =

(X + − X − )dλ,

X dμ2 dμ1 =

X dλ.

Remark 10. From the preceding elaboration, it also follows that theorem is the + dλ = ∞ or true even if both sides are either ∞ or −∞, depending on whether X − X dλ = ∞. Remark 11. The Fubini Theorem also holds (and is established in a similar fashion) for any finite number of product σ -finite measure spaces. Exercises. 1. If the r.v. X is integrable, then show that: n P(|X | ≥ n) → 0. By a counterexn→∞ ample, show that the converse need not be true. Hint: For the converse part, let X take on the values n = 3, 4, . . . , with probabilities pn proportional to (log n + 1)/(n log n)2 . 2. Let X be an integrable r.v. Then, for every ε > 0, there is a simple r.v. X ε such that |X − X ε | < ε. Hint: Write X = X + − X − , consider nonnegative sequences of simple r.v.s {X n } and {Yn } converging to X + and X − , appropriately employ the Dominated Convergence Theorem, and finally, define X ε in terms of X n s and Yn s. 3. Establish the following generalized version of part (iii) of Theorem 2 (Fatou– Lebesgue Theorem). Namely, for n = 1, 2, . . ., let X n , Un , Vn and X , U , V be r.v.s such that a.e. a.e. a.e. U n ≤ Xn ≤ Vn a.e., n ≥ 1, and as n → ∞, X n → X ,Un → U , Vn → V , X finite. Un → U finite and Vn → V finite. Then X n → n→∞


4. Establish the following generalized version of Theorem 3 (Dominated Convergence Theorem). Namely, for n = 1, 2, . . . , let X n , Un , X , U be r.v.s such that a.e. a.e. |X n | ≤ Un a.e., n ≥ 1, and as n → ∞, X n → X , Un → U , and Un → U finite. Then, as n → ∞: (i) X n → X finite; (ii) |X n − X | → 0; (iii) A X n → A X uniformly in A ∈ A. 5. In reference to Exercise 4, leave the setup and the assumptions intact except that a.e.

μ

a.e.

μ

the assumptions X n → X and Un → U are replaced by X n → X and Un → U as n → ∞. Then conclusions (i)–(iii) hold. Hint: For part (i), use the following fact: A sequence of real numbers {xn } converges to a real number x as n → ∞, if and only if for any subsequence {m} ⊆ {n} there exists a further subsequence {r } ⊆ {m} such that xr → x. r →∞

6. Let μ be a σ -finite measure and let ϕ be a nonnegative σ -additive and finite function, both defined on the measurable space (, A). Then show that ϕ μ is equivalent to the following: for every ε > 0, there exists δ = δ(ε) > 0 such that μ(A) < δ implies ϕ(A) < . Hint: If the assertion were not true when ϕ μ, there would exist an ε > 0 such that for every δ > 0 and some A with μ(A) < δ, we would have ϕ(A) ≥ ε. Apply this argument for δn = 1/2n , μ(An ) < 1/2n , and ϕ(An ) ≥ ε for some An , and set A = lim sup An in order to get μ(A) = 0 and ϕ(A) ≥ ε, a contradiction. n→∞

7. All sets figuring below are subsets of the product space 1 × 2 . Then show that (i) E ⊆ F implies E ω1 ⊆ Fω1 and E ω2 ⊆ Fω2 , ω1 ∈ 1 , ω2 ∈ 2 . (ii) E ∩ F = implies E ω1 ∩ Fω1 = E ω2 ∩ Fω2 = , ω1 ∈ 1 , ω2 ∈ 2 . (iii) For n = 1, 2, . . . , ∪ E n = ∪ E n,ω1 , ∪ E n = ∪ E n,ω2 , and in n n ω1 ω2 n n

En = E n,ω1 , En = E n,ω2 , ω1 ∈ 1 , particular, n

ω1

ω2 ∈ 2 . (iv) For n = 1, 2, . . . , ∩ E n n

n

ω1

ω2

n

= ∩ E n,ω1 , ∩ E n n

n

n

ω2

= ∩ E n,ω2 , ω1 ∈ 1 , n

ω2 ∈ 2 . (v) (E c )ω1 = (E ω1 )c , (E c )ω2 = (E ω2 )c , ω1 ∈ 1 , ω2 ∈ 2 . 8.

(i) Consider the measure space (, A, μ) and let C be a fixed set in A. On A, define the function μ◦ by: μ◦ (A) = μ(A ∩ C). Then show that μ◦ is a measure. (ii) Let X be ar.v. defined on (, A, μ) and X dμ exists. Then suppose that show that X dμ◦ also exists and that X dμ◦ = C X dμ.

9. Consider the (σ -finite) measure space (, A, μ), and let {Ai , i = 1, 2, . . .} be a (measurable) partition of . For each i, define the measure (see Exercise 8) on (, A, μ) for which μi by: μi (A) = μ(A ∩ Ai ). Then, if X is a r.v. defined the integral X dμ exists, show that the integrals X dμ i , i ≥ 1, also exist and ∞ X dμ = X dμ. i i=1

91

92


10. If X is a simple r.v. (not necessarily nonnegative), defined on the (product) space (1 × 2 , A1 × A2 , λ = μ1 × μ2 ) with μ1 and μ2 σ -finite, for which X dλ exists, then appealing to Theorem 12 and Remark show directly (i.e., without , ω )dμ dμ = X (ω , ω )dμ dμ = X (ω1 , ω2 )dλ = 10) that X (ω 1 2 1 2 1 2 2 1 X dλ. 11. Let X be a r.v. defined on the product space (1 × 2 , A1 × A2 , λ = μ1 × μ2 ) and suppose that X dλ exists. Then show that the Fubini theorem holds true. 12. If the r.v.s X 1 , . . . , X n are i.i.d. with E X 1 = μ ∈ and V ar (X 1 ) = σ 2 ∈ (0, ∞), √ ¯ d then by the CLT, n( X n −μ) → Z ∼ N (0, 1), where X¯ n = 1 nj=1 X j is the σ

n

n→∞

sample mean of the X j s. Show that the CLT implies the Weak Law of Large P Numbers (WLLN); i.e., X¯ n → μ. n→∞

13. Let (, A, μ) = ((0, 1], B(0,1] , λ) where λ is the Lebesgue measure, and consider the function f : (0, 1] → defined by 1 1 < x ≤ , n = 1, 2, . . . . n+1 n Then investigate whether or not the integral (0,1] f dλ exists. Also, compute it if it exists, as well as the (0,1] | f |dλ. 14. For n = 1, 2, . . . , consider the r.v.s X 1 , X 2 , . . . and X , and show that

|X n − X | P → 0. X n −→ X if and only if E n→∞ 1 + |X n − X | n→∞ f (x) = (−1)n n,

Hint: Refer to Exercise 14 in Chapter 4. 15. Take (, A, P) = ((0, 1), B(0,1) , λ), λ being the Lebesgue measure, and, for n = 0, 1, . . ., define the r.v.s X n by: X 2n+1 = I(0, 1 ) , 2

X 2n+2 = I[ 1 ,1) . 2

Then, as n → ∞, show that: lim inf X n ≤ lim inf X n , lim sup X n ≤ lim sup X n (as it should be, by Theorem 2, since 0 ≤ X n ≤ 1 for all n, and the bounds are integrable). 16. In reference to Exercise 12 in Chapter 4, define the r.v. X by X (ω) = 21ω and then show that X dμ = 2. 17. For n ≥ 1, let h n , gn , G n , and h, g, G be real-valued measurable functions defined on ( k , B k , λk ), where λk is the Lebesgue measure on B k , be such that: (i) As n → ∞, h n (x) → h(x), gn (x) → g(x), G n (x) → G(x) a.e. [λk ].


(ii) For all n ≥ 1, gn (x) ≤ h n (x) ≤ G n (x) a.e. [λk ]. (iii)

gn dλ → g dλ , G n dλ → k

k

g dλk , G dλk are finite. and k

k

k

k

k

k

G dλk , as n → ∞,

k

Then show that k h n dλk → k h dλk and k h dλk is finite. 18. For n ≥ 1, let X n , Yn , and X be r.v.s defined on the probability space (, A, P), d

and suppose that X n → X as n → ∞. Then, as n → ∞: P

d

(i) Yn − X n → c, a constant, implies Yn → X + c. P

d

(ii) Yn → c implies X n Yn → cX . 19. As a variant of Exercise 6, consider the following situation. Let (, A, P) be a probability space and let X be an integrable r.v. On A, define the (finite) measure ν by ν(A) = A X d P. Then show that ν 0 there exists δ = δ(ε)(> 0) such that P(A) < δ implies ν(A) < ε. Hint: Use appropriately the Dominated Convergence Theorem. 20. Let X be a nonnegative integrable r.v. Then: (i) Use the Fubini Theorem to show that EX =

∞

P(X ≥ t)dt.

0

(ii) Apply this result in case the d.f. of X is F(x) = 1 − e−λx for x ≥ 0 and 0 for x < 0 (λ > 0); or F(x) = 0 for x < 0, F(x) = x for 0 ≤ x ≤ 1, and F(x) = 1 for x ≥ 1.

93

CHAPTER

Standard Moment and Probability Inequalities, Convergence in the r th Mean and its Implications

6

This chapter consists of two sections. The first section is devoted to the standard moment and probability inequalities. They include the Hölder (Cauchy–Schwarz) inequality, referring to the expectation of the product of r.v.s; the Minkowski and the cr -inequality, referring to the expectation of the sum of r.v.s; and the Jensen inequality, concerning convex functions. The probability inequality in Theorem 6 provides both an upper and a lower bound, and the upper bound gives the Markov inequality and the Tchebichev inequality, as special cases. In the second section, the concepts of convergence in the r th mean as well as mutual convergence in the r th mean are introduced, and it is shown that they are equivalent. Most of the remainder of this section is devoted to establishing various implications of convergence in the r th mean, and also to giving sufficient conditions that imply convergence in the r th mean. The concepts of uniform integrability and uniform continuity are instrumental in these derivations. These facts are summarized in the form of a table for easy reference.

6.1 Moment and Probability Inequalities From now on the measure spaces to be considered will be probability spaces (, A, P). Theorem 1. Let X be a r.v. whose r th absolute moment is finite; i.e., E |X |r < ∞. Then E |X |r < ∞ for all 0 ≤ r ≤ r .

Proof. For each ω ∈ , we have |X (ω)|r ≤ 1 + |X (ω)|r for 0 ≤ r ≤ r . In fact, this is clearly true if |X (ω)| ≤ 1, while if |X (ω)| > 1, this is also true because then |X (ω)|r < |X (ω)|r ; this inequality implies E |X |r ≤ 1 + E |X |r < ∞. Theorem 2 (The Hölder inequality). Let X , Y be two r.v.s and let r > 1. Then E |X Y | ≤ E 1/r |X |r ×E 1/s |Y |s for s > 0 such that r1 + 1s = 1 (where it is assumed that E |X |r, E |Y |r < ∞, because otherwise the inequality is trivially true). In particular, for r = s = 2, we have E |X Y | ≤ E 1/2 |X |2 × E 1/2 |Y |2 or E 2 |X Y | ≤ E |X |2 E |Y |2 (which is known as the Cauchy–Schwarz inequality).

In proving Theorem 2, we need the following lemma. An Introduction to Measure-Theoretic Probability, Second Edition. http://dx.doi.org/10.1016/B978-0-12-800042-7.00006-2 Copyright © 2014 Elsevier Inc. All rights reserved.

95

96

CHAPTER 6 Standard Moment and Probability Inequalities, Convergence

Lemma 1. For x, y ∈ and r , s > 0 such that r1 + 1s = 1, we have |x y| ≤

|x|r r

+ |y|s . s

Proof. Since |x y| = |x| |y|, it suffices to prove the lemma for x, y > 0; i.e., to s r show x y ≤ xr + ys . For fixed x, consider the following function of y, f (y) = y s−1 (s−1)y s−2 xr . Thus f (y) = 0 yields s , y > 0. Hence f (y) = − r y 2 + s = r (s−1) y s ; from r1 + 1s = 1, it follows that r (s−1) = 1. So x r = y s . Next, s s s−3 r r f (y) = 2x + (s−1)(s−2)y , which for x r = y s becomes 2y + (s−1)(s−2) y s−3 > 0. s s r y3 r y3 Thus f (y) is minimized for y s = x r , and this minimum is xr ry xr

+

min f (y) = x. So, f (y) ≥ x or y>0

x r ys + ≥ x y. r s

Proof of Theorem 2. In the first place, if E |X |r = 0 then |X |r = 0 a.s. so that X = 0 a.s. and X Y = 0 a.s. or |X Y | = 0 a.s. Then E |X Y | = 0; thus the inequality is true. Similarly, if E |Y |s = 0. So, we assume that E |X |r, E |Y |r > 0, and set s r X (ω) Y (ω) x = E 1/r , y = E 1/s . Then |x y| ≤ |x|r + |y|s becomes |X |r |Y |s |X (ω)|r |Y (ω)|s |X (ω)Y (ω)| ≤ + for all ω ∈ ; hence r s r E 1/r |X | E 1/s |Y | r E |X | s E |Y |s E |X |r E |Y |s 1 1 E |X Y | ≤ + = + = 1 or r s r s 1/r 1/s E |X | E |Y | r E |X | sE |Y | r s E |X Y | ≤ E 1/r |X |r E 1/s |Y |s . Corollary. Proof.

E 1/r |X |r is nondecreasing in r (> 0).

Consider the Cauchy–Schwarz inequality E 2 |X Y | ≤ E |X |2 × E |Y |2 and

for 0 < r < r , replace X by |X |

r −r 2

and Y by |X |

r +r 2

. Then we get

E 2 |X |r ≤ E |X |r −r E |X |r +r , hence 2 log E |X |r ≤ log E |X |r −r 1 + log E |X |r +r or log E |X |r ≤ (log E |X |r −r + log E |X |r +r ). 2

Thus, if we consider log E |X |r as a function of r , g(r ) say, then g is convex, since the last relationship above is equivalent to g(r ) ≤ 21 [g(r − r ) + g(r + r )], where

+r ) r = (r −r )+(r and g is continuous. (To see that g is continuous, let r ↑ ro ; 2 r then |X | ≤ 1 + |X |ro (by Theorem 1), so that the Dominated Convergence Theorem applies and gives E |X |r → E |X |ro or g(r ) → g(ro ). Next, let r ↓ ro . Then |X |r ≤ r →r0

r →r0

1 + |X |r1 for some r1 > ro and all ro ≤ r ≤ r1 , so that the previous argument applies.) Now g(0) = 0 and the slope of the line through the points (0, 0) and ) 1 r (r , g(r )) is increasing in r . But this slope is g(r r (r > 0). Thus r log E |X | ↑ in r or 1

1

log E r |X |r ↑ in r or E r |X |r ↑ in r .

6.1 Moment and Probability Inequalities

g(r)

(r,g(r))

0

r

r

(For the facts on convex functions stated earlier, see, e.g., page 73 in Hardy et al. (1967)) For r ≥ 1, we have

Theorem 3 (The Minkowski inequality). 1

1

1

E r |X + Y |r ≤ E r |X |r + E r |Y |r (where it is assumed that E |X |r , E |Y |r < ∞, because otherwise the inequality is trivially true). Proof.

For r = 1, we have |X + Y | ≤ |X | + |Y | so that E |X + Y | ≤ E |X | + E |Y | .

So the inequality is true. Now let r > 1. Then E |X + Y |r = E(|X + Y | |X + Y |r −1 ) ≤ E(|X | |X + Y |r −1 + |Y | |X + Y |r −1 ) = E(|X | |X + Y |r −1 ) + E(|Y | |X + Y |r −1 ). At this point, applying the Hölder inequality for the given r (> 1) and s such that 1 1 r + s = 1 (from which it follows that s = r /(r − 1)), we get 1

1

1

1

E |X + Y |r ≤ E r |X |r E s |X + Y |(r −1)s + E r |Y |r E s |X + Y |(r −1)s 1

1

1

= E s |X + Y |(r −1)s (E r |X |r + E r |Y |r ). Now, from

1 r

+

1 s

= 1 we get (r − 1)s = r . Thus 1

1

1

E |X + Y |r ≤ E s |X + Y |r (E r |X |r + E r |Y |r ). 1

Hence, if E |X + Y |r > 0, then divide both sides by E r |X + Y |r to get 1

1

1

E r |X + Y |r ≤ E r |X |r + E r |Y |r . If E |X + Y |r = 0, the inequality is trivially satisfied.

97

98


For r ≥ 0, we have

Theorem 4 (The cr -inequality).

E |X + Y |r ≤ cr (E |X |r + E |Y |r ), where 1 if r ≤ 1 cr = r −1 2 if r > 1.

For the proof of this theorem we need the following lemma. Lemma 2. For x, y ∈ and r ≥ 0, we have |x + y|r ≤ cr (|x|r + |y|r ), where cr is as in the theorem. Proof. Since |x + y| ≤ |x| + |y| implies |x + y|r ≤ (|x| + |y|)r , it suffices to show that (|x| + |y|)r ≤ cr (|x|r + |y|r ). From this it also follows that it suffices to prove the lemma for x, y > 0. Case 1: 0 ≤ r ≤ 1. We have y x , < 1 imply x+y x+y and

x x+y

r

x , > x+y

y x+y

r >

y x+y

x r + yr >1 (x + y)r

or (x + y)r ≤ x r + y r , as was to be seen. y x , 1 − p = x+y . Then p, 1 − p < 1. Set also f ( p) = Case 2: r > 1. Set p = x+y r r p + (1 − p) and minimize it. From f ( p) = r pr −1 − r (1 − p)r −1 = 0, we get pr −1 = (1 − p)r −1 ; hence p = 1 − p and p =

1 . 2

Next, f ( p) = r (r − 1)[ pr −2 + (1 − p)r −2 ] is > 0 for all p(> 0) and hence for p = 21 . Since min f ( p) = f ( 21 ) = 2r1−1 , we have then: 0< p0). Replacing X by |X |r1 , we then get r2 1 1 r2 r2 E |X |r1 r1 ≤ E |X |r1 r1 or E r1 |X |r1 ≤ E |X |r2 or E r1 |X |r1 ≤ E r2 |X |r2 ; 1

i.e., E r |X |r ↑ in r (r > 0), as was seen before. Remark 1.

This was also established in the Corollary to Theorem 2.

Definition 1. α is said to be the almost sure sup of the r.v. X (α = a.s. sup X ) if P(X > α) = 0 and for any β < α, P(X > β) > 0. Theorem 6.

Let X be a r.v. and let g : → [0, ∞), Borel function. Then

(i) If g is even, and nondecreasing on [0, ∞), we have that E g(X ) − g(c) E g(X ) ≤ P(|X | ≥ c) ≤ , c > 0 constant, α g(c) and α = a.s. sup g(X ).

99

100


(ii) If g is nondecreasing on , then we have E g(X ) E g(X ) − g(c) ≤ P(X ≥ c) ≤ , c ∈ . α g(c) Remark 2.

∞ ∞

is interpreted as 0.

Proof. (i) Let A = (|X | ≥ c). Then for ω ∈ A we have |X (ω)| ≥ c; equivalently, X (ω) ≥ c or −X (ω) ≥ c. Hence g(−X (ω)) = g(X (ω)) ≥ g(c); i.e.,g(X ) ≥ g(c) on A whether X ≥ c or X ≤ −c. Similarly, g(X ) ≤ g(c) on Ac . Next, E g(X ) = g(X )dP = g(X )dP + g(X )dP Ac

A

and

g(c)P(A) ≤ A g(X )dP ≤ α P(A) . 0 ≤ Ac g(X )dP ≤ g(c)

Thus g(c)P(A) ≤ Eg(X ) ≤ α P(A) + g(c) and hence P(A) = P(|X | ≥ c) ≤

E g(X ) E g(X ) − g(c) and P(|X | ≥ c) ≥ . g(c) α

(ii) Let B = (X ≥ c). Then g(X ) ≥ g(c) on B and g(X ) ≤ g(c) on B c . Since E g(X ) = g(X )dP + g(X )dP, Bc

B

we get

g(c)P(B) ≤ B g(X )dP ≤ α P(B) , 0 ≤ B c g(X )dP ≤ g(c)

which leads to g(c)P(B) ≤ E g(X ) ≤ α P(B) + g(c), g(X ) and hence P(B) = P(X ≥ c) ≤ Eg(c) , P(X ≥ c) ≥ E g(X α)−g(c) .

Special Cases: By taking g(x) = |x|r , r > 0, we get from the right-hand side of |r the inequality in (i): P(|X | ≥ c) ≤ E |X cr , which is the Markov inequality.

Also, P(|X − E X | ≥ c) ≤ E |X −crE X | , by replacing X by X − E X (by assuming that E X is finite); in particular, for r = 2 we have the Tchebichev inequality: r

P(|X − E X | ≥ c) ≤

σ 2 (X ) . c2

This section is concluded with a simple example regarding the Tchebichev inequality.

6.2 Convergence in the r th Mean, Uniform Continuity

Example 1. When the distribution of X is not known, which is most often the case in statistics, the Tchebichev inequality lends itself handily to determining the smallest sample size n, so that the so-called sample mean will lie within a prescribed multiple of standard deviations σ from the population mean μ with probability no smaller than a preassigned value p. Thus, if X 1 , . . . , X n are i.i.d. r.v.s with expectation μ ∈ n X j with and variance σ 2 ∈ (0, ∞), then the sample mean of X j s is X¯ n = n1 j=1

E X¯ n = μ and σ 2 ( X¯ n ) = σ 2 /n. Then P(| X¯ n − μ| < kσ ) ≥ 1 − nk1 2 , and if we set 1 − nk1 2 ≥ p, then the required sample size is the smallest value of n that is greater 1 than or equal to k 2 (1− . The Markov inequality provides an upper bound for the p)

probability P(|X | ≥ c) when E|X |2 = ∞ but E|X |r < ∞ for some (0 0. We say that X n converges in the r th mean to X , and write (r ) X n → X , if E |X n − X |r → 0. n→∞

n→∞

(2)

For r = 2, the convergence is referred to as convergence in quadratic mean, X n → X , n→∞

q.m.

or X n → X .

n→∞

Remark 3. Since E |X n − X |r ≤ cr (E |X n |r +E |X |r ), we have that E |X n − X |r < ∞, n = 1, 2, . . . . At this point, it should be mentioned that the limit in the r th mean is a.s. uniquely defined. That is, we have the following Proposition 1.

(r )

(r )

n→∞

n→∞

Let X n → X and X n → Y or E |X n − X |r → 0 and

E |X n − Y |r → 0. Then X = Y a.s.

n→∞

n→∞

Proof.

Indeed, E |X − Y |r = E |(X n − Y ) − (X n − X )|r ≤ cr (E|X n − X |r + E|Yn − Y |r ) → 0, n→∞

so that E |X − Y |r = 0 and hence |X − Y |r = 0 a.s. or X = Y a.s.

The following theorem will prove useful in many cases. Theorem 7.

(r )

P

n→∞

n→∞

Let E |X n |r < ∞ for all n. Then X n → X implies X n → X and

E |X n |r → E |X |r finite. (However, see also Theorem 14.) n→∞

101

102


Proof. The first conclusion is immediate by the Markov inequality. As for the second, we have Case 1: 0 < r ≤ 1. By the cr -inequality we get E |X n |r = E |(X n − X ) + X |r ≤ E |X n − X |r + E |X |r E |X |r = E |(X n − X ) + X n |r ≤ E |X n − X |r + E |X n |r , so that E |X n |r − E |X |r ≤ E |X n − X |r E |X |r − E |X n |r ≤ E |X n − X |r , or − E|X n |r − E|X |r ≤ E|X n − X |r

and hence E |X n |r − E |X |r ≤ E |X n − X |r → 0 so that E |X n |r → E |X |r . n→∞

n→∞

Case 2: r > 1. By the Minkowski inequality we get 1 1 1 1 E r |X n |r = E r |(X n − X ) + X |r ≤ E r |X n − X |r + E r |X |r 1

1

1

1

E r |X |r = E r |(X n − X ) + X n |r ≤ E r |X n − X |r + E r |X n |r , so that ⎧ 1 ⎨ E r |X n |r − E r1 |X |r ≤ E r1 |X n − X |r

⎩ E r1 |X |r − E r1 |X n |r ≤ E r1 |X n − X |r , or − E r1 |X n |r − E r1 |X |r ≤ E r1 |X n − X |r

1 1 1 1 1

and hence E r |X n |r − E r |X |r ≤ E r |X n − X |r → 0; thus E r |X n |r → E r |X |r n→∞

or E |X n |r → E |X |r .

n→∞

Finiteness of E |X |r follows from E |X n |r − E |X |r ≤ E |X n − X |r (for 1 1

1 0 < r ≤ 1) or from E r |X n |r − E r |X |r ≤ E r |X n − X |r (for r > 1). n→∞

Theorem 7 is supplemented now by the following result. (r )

P

n→∞

n→∞

If X n → X , then X n → X (as was seen in Theorem 7). Conversely,

Theorem 8. P

(r )

n→∞

n→∞

if X n → X and P(|X n | ≤ M < ∞) = 1, then X n → X , for every r > 0. Proof.

P

Assume that X n → X and P(|X n | ≤ M) = 1. We have then P

a.s.

n→∞

n→∞

X n → X implies that there exists {m} ⊆ {n} such that X m → X (by Theorem 5 (ii) in Chapter 3). Hence P(|X n | ≤ M) = 1 implies P(|X | ≤ M) = 1 (by Exercise 18 in Chapter 3). Then, by Lemma 2, |X n − X |r ≤ cr (|X n |r + |X |r ) and P(|X n − X |r ≤ 2cr M r ) = 1,


and therefore

|X n − X |r dP +

E |X n − X |r =

(|X n −X |≥ε) ≤ 2cr M r P(|X n

|X n − X |r dP

(|X n −X | 0, there exists δ(ε) independent of n such that |X n | < ε for every n ≥ 1. P(A) < δ(ε) implies A

In this definition, the index n may be replaced by t ∈ T ⊆ . (See also Exercise 6 in Chapter 5.) Theorem 11. The r.v.s X n , n ≥ 1, are uniformly integrable if and only if the integrals of their absolute values are bounded and uniformly continuous. Proof. Assume uniform integrability. Then for ε > 0, there exists c = c(ε) > 0 |X n | < ε for all n. Now large enough such that (|X n |≥c)

|X n | =

|X n | + (|X n | 0 such that

Take δ = δ(ε) =

ε 2c

(|X n |≥c)

|X n |
0, there exists δo (ε) > 0 such that if P(A) < δo (ε) then A |X |r < 2cεr . From |X n − X |r → 0 we have that there exists n o such that |X n − X |r < 2cεr for n→∞

n > n o . Therefore for n > n o and A such that P(A) < δo (ε), we have ε ε |X n |r ≤ cr + cr = ε. 2cr 2cr A Next consider A |X n |r , n = 1, . . . , n o . Then for

ε > 0 there exists δn (ε), n = 1, . . . , n o , such that P(A) |X n |r < ε, < δn (ε) implies A

because |X n |r is absolutely continuous. Set δ(ε) = min{δo (ε), δ1 (ε), . . . , δn (ε)}. Then for ε > 0 and A such that P(A) < δ(ε) we have A |X n | < ε for all n. Theorem 13 (Necessary and sufficient conditions for convergence in the r th mean). Let E|X n |r < ∞ for all n. Then (r )

(i) X n → X if and only if, either n→∞ P (ii) X n → X and |X n |r are uniformly continuous, or n→∞ P (ii ) X n → X and |X n − X |r are uniformly continuous. n→∞

Proof. The theorem is established by showing that: (i) ⇒ (ii), (ii ) ⇒ (i) (which implies that (ii ) ⇒ (ii)) and (ii) ⇒ (ii ) (which implies that (ii) ⇒ (i)). Then (i) and (ii) are equivalent, and (i) and (ii ) are also equivalent. In the form of a diagram, we have (i) (ii)

(r )

(ii ) P

Indeed, X n → X implies X n → X (by Theorem 8) and that |X n |r , n ≥ 1, are n→∞ n→∞ uniformly integrable (by Theorem 12), which, in turn, implies that |X n |r , n ≥ 1,


are uniformly continuous (and |X n | ≤ M(< ∞), n ≥ 1) (by Theorem 11). So (i) ⇒ (ii). Next, (ii ) ⇒ (i) because, with An = (|X n − X | ≥ ε), for any ε > 0, |X n − X |r = |X n − X |r + |X n − X |r ≤ ε + εr , Acn

An

by (ii ) for all n > n 0 = n 0 (ε), so that |X n − X |r → 0; i.e., (ii ) ⇒ (i). Finally, (ii) ⇒ (ii ) because, by the cr -inequality, |(X n − X )I A |r ≤ cr |X n I A |r + cr |X I A |r , A ∈ A,

or

|X n − X |r ≤ cr A

|X n |r + cr A

|X |r .

(6.1)

A

By (ii), |X n |r < A

ε for all n, provided P(A) < δ(ε), some suitable 2cr δ(ε) > 0.

Next, we show that |X |r < A

ε , 2cr

for the same A as that in (6.2).

(6.2)

(6.3)

(This would be true if E|X |r < ∞, which we do not know.) That (6.3) is true is seen P

a.s.

as follows: By (ii), X n → X . Hence there exists {m} ⊆ {n} such that X m → X or n→∞ m→∞ a.s. r |X m |r I A → X I A , A ∈ A, with P(X = X ) = 0. Henceforth we treat X as if m→∞ it were X . So, a.s. 0 ≤ |X m |r I A → |X |r I A . Then by part (iii) of the Fatou–Lebesgue Theorem m→∞

(Theorem 2 in Chapter 5), lim inf |X m |r I A =

lim |X m |r I A = m→∞ ≤ lim inf |X m |r I A

m→∞

|X |r I A

m→∞

or

|X |r ≤ lim inf A

m→∞

|X m |r .

(6.4)

A

Again, by (ii) (see also (6.2)), ε |X m |r ≤ for all m, provided P(A) < δ(ε). 2c r A

(6.5)

107

108


From (6.4) and (6.5), it follows that ε |X |r ≤ for the same A as in (6.5). 2c r A Then (6.2) and (6.3) hold simultaneously, and hence (6.1) yields ε ε |X n − X |r ≤ cr + cr = ε for all n, 2c 2c r r A

as was to be seen. Corollary 1. Proof.

(r )

(r )

n→∞

n→∞

If X n → X then X n → X for all (0 0 and A with P(A) as small as required but < r 2 , we have ε r |X n − X | implied by A |X n − X | < 2 for all n by uniform continuity of (r )

X n → X . The proof is completed.

n→∞

Corollary 2.

P

(r )

n→∞

n→∞

Let X n → X and E |X n |r ≤ M(< ∞), n ≥ 1. Then X n → X for

all (0 0, let An = (|X n | ≥ c). Then, for |X n | > 0, we get

|X n |r = |X n |r On An ,

Hence

|X n |r 1 r = |X n | . r |X n | |X n |r −r

1 1 1 1 ≤ , or ≤ r −r . r −r |X n | c c |X n |

|X n |r = A

|X n |r + A∩An

A∩Acn

|X n |r


= A∩An

≤

1 1

cr −r

|X n

|r −r

|X n | + r

A∩Acn

|X n |r 1

|X n |r + cr P(A) ≤ A∩An

cr −r

M + cr P(A).

Then for ε > 0, take c sufficiently large so that r 1−r M < 2ε . Also choose A such that c P(A) < εr = δ(ε). Thus, for ε > 0, there exists δ(ε) > 0 such that P(A) < δ(ε) 2c implies A |X n |r < ε. Corollary 3 (Compare with the converse part of Theorem 8).

P

Let X n → X and n→∞

(r )

a.s.

|X n | ≤ Y , for all n with EY r < ∞. Then X n → X . n→∞

r a.s. r r r Proof. |X n | ≤ Y implies the r |X n | ≤ Y a.s. and hence |X n | ≤ Y . Then absolute continuity of Y (by Exercise 11) implies the uniform continuity of |X n |r , hence the result. (1)

Let 0 ≤ X n , and E X n < ∞ for all n. Then X n → X if and only if

Lemma 3.

n→∞

P

X n → X and E X n → E X finite. n→∞

n→∞

Throughout the proof all limits are taken as n → ∞ unless otherwise

Proof.

(1)

P

specified. Let X n → X . Then X n → X and E X n → E X finite by Theorem 7 (applied P

P

with r = 1). Hence it suffices to prove the converse. Now X n → X implies X n −X → 0 P

P

P

or X − X n → 0. Next, it is easily seen that Yn → Y implies g(Yn ) → g(Y ) for every continuous function g (see also Exercise 8 in Chapter 3). The function g(x) = x + P

P

is continuous. Hence X − X n → 0 implies (X − X n )+ → 0. Now 0 ≤ X n , n ≥ 1, P

and X n → X implies X ≥ 0 a.s. (passing to a subsequence {m} ⊆ {n} such that a.s. X m → X , so that X ≥ 0 a.s. since X n ≥ 0 a.s. for all n), and (X − X n )+ ≤ X , a.s. n→∞ So we have P

(0 ≤)(X − X n )+ ≤ X with E X < ∞ and (X − X n )+ → 0. Then, the Dominated Convergence Theorem gives E(X − X n )+ → 0.

(6.6)

It is also given that E X n → E X finite, which implies E(X − X n ) → 0.

(6.7)

From (6.6) and (6.7) and the relation X − X n = (X − X n )+ − (X − X n )− we get E(X − X n )− → 0.

(6.8)

109

110


Adding (6.6) and (6.8), we get (1)

E |X − X n | → 0 or X n → X . Theorem 14 (Vitali’s Theorem).

(r )

Let E |X n |r < ∞ for all n. Then X n → X if and n→∞

P

only if X n → X and E |X n |r → E |X |r finite. n→∞

Proof.

n→∞

Throughout the proof all limits are taken as n → ∞. (r )

P

X n → X implies X n → X and E |X n |r → E |X |r finite ( by Theorem 7). So it suffices to show the converse. P P Now X n → X implies |X n |r → |X |r and we also have E |X n |r → E |X |r finite. Set Yn = |X n |r , Y = |X |r . Then we have 0 ≤ Yn , and EYn < ∞ for all sufficiently (1)

P

large n and Yn → Y , EYn → EY finite. Then the lemma applies and gives Yn → Y (1) or |X n |r → |X |r . Hence |X n |r are uniformly continuous by Theorem 13. So we (r ) P have: X n → X and |X n |r are uniformly continuous. Then X n → X , by Theorem 13 again. Remark 6. Lemma 3 is true if P is replaced by a σ -finite measure μ and X n , X by probability densities functions f n , f . Then the condition E X n → E X is trivially (1) true, since these quantities will be equal to 1. Then f n → f (i.e., | f n − f | dμ → 0) μ

if and only if f n → f . In this form the lemma is known as Scheffé’s Theorem. (See Scheffé (1947).) More precisely, we have Theorem 15 (Scheffé’s Theorem). If for n = 1, 2, . . . , f n and f : k → are p.d.f.s with respect to some σ -finite measure μ (e.g., the Lebesgue measure) in k , then as n → ∞, (1)

fn → f (μ)

μ

if and only if f n → f .

Proof. Markov’s inequality is still true if P is replaced by μ, as is easily seen directly (see Exercise 12). μ (1) Thus, with all limits taken as n → ∞, we have f n → f implies f n → f since (μ) μ μ(| f n − f | ≥ ε) ≤ ε−1 | f n − f | dμ. Next, it is easily seen that f n → f or f − μ

μ

f n → 0 implies ( f − f n )+ → 0 (see Exercise 1 in Chapter 3). As before, 0 ≤ ( f − μ f n )+ ≤ f with ( f − f n )+ → 0 and f dμ = 1 implies ( f − f n )+ dμ → 0, f dμ by the Dominated Convergence Theorem, whereas f ndμ = 1 → 1 = or ( f − f n )dμ = 0. Thus ( f − f n )− dμ → 0. So ( f − f n )+ dμ → 0 and (1) ( f − f n )− dμ → 0 imply | f n − f | dμ → 0 or f n → f . (μ)


(r )

(1)

n→∞

n→∞ (r )

Notice that X n → X is equivalent to |X n |r → |X |r . Indeed, by The-

Remark 7.

P

orem 14, and with all limits taken as n → ∞, we have X n → X implies X n → X and P

E|X n |r → E|X |r finite. Hence |X n |r → |X |r and E|X n |r → E|X |r finite, and there(1)

(1)

P

fore, by Theorem 14 again, |X n |r → |X |r . Also, |X n |r → |X |r implies |X n |r → |X |r P

and E|X n |r → E|X |r finite, and hence X n → X and E|X n |r → E|X |r finite. There(r )

fore X n → X .

(r ) Another consequence of the convergence X n → X is that A |X n |r → A |X |r uniformly in A ∈ A. Indeed,

(|X n |r − |X |r ) ≤

|X n |r − |X |r ≤ E |X n |r − |X |r ,

A

A

(1)

independent of A and converging to 0, since |X n |r → |X |r . The following table presents in summary form the main implications of convergence in the r th mean, as well as conditions under which convergence in the r th mean holds. All limits are taken as n → ∞. (r ) X n → X if and only if {X n } , n ≥ 1, converges mutually in the r th mean. ⎧ P ⎪ Xn → X ⎪ ⎪ ⎪ ⎪ E |X n |r → E |X |r finite ⎪ ⎪ ⎨ |X n |r are uniformly integrable (r ) X n → X implies r ⎪ |X n | arer uniformly continuous ⎪ ⎪ ⎪ |X n − X | are uniformly continuous ⎪ ⎪ ⎪ ⎩ (r ) Xn → X , 0 < r ≤ r . ⎧ P(|X n | ≤ M < ∞) = 1, n ≥ 1 ⎪ ⎪ ⎪ ⎪ E ⎨ |X n |r → E |X |r finite P X n → X and any one of |X n |r are uniformly continuous ⎪ ⎪ |X n − X |r are uniformly continuous ⎪ ⎪ ⎩ |X n | ≤ Y a.s., n ≥ 1, and EY r < ∞ (r )

imply X n → X . (r )

P

X n → X and E |X n |r ≤ M < ∞, n ≥ 1, imply X n → X , 0 < r < r . Finally, combining the converging implications at the end of Section 5.2 in Chapter 5 with Theorem 8 here, we have that the four modes of convergence: a.s., in probability, in distribution, and in the r th mean, are related as follows: a.s.

P

d

Xn → X ⇒ Xn → X ⇒ Xn → X ⇑ r Xn → X .

111

112


Exercises. 1. Establish the following generalized version of Theorem 2 (the Hölder inequality). Namely, show that for any n (≥ 2) r.v.s X 1 , . . . , X n and any positive numbers r1 , . . . , rn with r11 + · · · + r1n = 1, it holds that 1

1

E|X 1 . . .X n | ≤ E r1 |X 1 |r1 . . .E rn |X n |rn . 2. Let g : I open interval in → be convex; i.e., for every x, x ∈ I and every α ∈ [0, 1], it holds g(αx + (1 − α)x ) ≤ αg(x) + (1 − α)g(x ). Then show that (i) g is continuous. (ii) For every x0 ∈ I , there exists λ(x0 ) ∈ such that g(x) − g(x0 ) ≥ λ(x0 ) × (x − x0 ), x ∈ I . Hint: For part (i), choose x1 < x0 < x2 , set α = (x2 − x0 )/(x2 − x1 ), β = (x0 − x1 )/(x2 − x1 ), use convexity of g to get g(x0 ) ≤ αg(x1 ) + βg(x2 ), and take the lim inf by letting x2 ↓ x0 . Next, let x1 < x2 < x0 , take α = (x0 − x2 )/ (x0 − x1 ), β = (x2 − x1 )/(x0 − x1 ), use convexity of g to get g(x2 ) ≤ αg(x1 ) + βg(x0 ), and take the lim sup as x2 ↑ x0 . For part (ii), let x1 < x0 < x, and with α = (x − x0 )/(x − x1 ) and β = (x0 − x1 )/(x − x1 ), use convexity of g in order to get g(x) − g(x0 ) ≥ [g(x0 ) − g(x1 )](x − x0 )/(x0 − x1 ). The result then follows by taking x1 = cx0 for some c > 0 so that cx0 ∈ I . 3. Establish the following generalized version of Theorem 3 (the Minkowski inequality). Namely, show that for any n (≥ 2) r.v.s X 1 , . . . , X n and any r ≥ 1, it holds that 1 1 1 E r |X 1 + ··· + X n |r ≤ E r |X 1 |r + ··· + E r |X n |r . 4. If

∞

a.s.

E|X n − X |r < ∞ for some r > 0, then X n → X . n→∞

n=1

Hint: Use Exercise 4 in Chapter 3. a.s.

(r )

n→∞

n→∞

5. If X n → X , and X n → Y for some r > 0, it follows that P(X = Y ) = 0. 6. Construct r.v.s X n , n ≥ 1, and X on some probability space (, A, P) such that P

(r )

n→∞

n→∞

X n → X , but X n X for any r ≥ 1. 7. Construct r.v.s X n , n ≥ 1, and X on some probability space (, A, P) such that a.s.

(r )

n→∞

n→∞

X n X , but X n → X for any r > 0. 8. Let X n , n = 1, 2, . . ., be r.v.s such that E|X n |β ≤ M (< ∞), n ≥ 1, for some β > 0. Then show that |X n |α , n ≥ 1, are uniformly integrable for all α with 0 < α < β. 9. For n = 1, 2, . . ., let X n be r.v.s such that P(X n = cn) = n1 and P(X n = 0) = 1 − n1 , for some c > 0. Investigate whether or not these r.v.s are uniformly integrable.


10. If E|X | < ∞, show that |X | is absolutely continuous; that is, A |X |→0 when P(A)→0. 11. For n = 1, 2, . . ., let X n , Yn and X , Y be r.v.s such that P(X n ≥ Yn ≥ 0) = 1, P

P

n→∞

n→∞

X n → X , Yn → Y , and E X n → E X finite. Then show that E|Yn − Y | → 0. n→∞

n→∞

12. For any r.v. X defined on the measure space (, A, μ), show directly that the Markov inequality holds; namely, μ(|X | ≥ c) ≤ c−r |X |r dμ for any r > 0 and any c > 0. 13. For a r.v. X and some r > 0, show that E|X |r < ∞ if and only if E|X − c|r < ∞ for every (finite) constant c. 14. For a r.v. X with E X = μ ∈ and V ar (X ) = σ 2 = 0, show that P(X = μ) = 1. 15. For n = 1, 2, . . ., consider the r.v.s X n and X and show that P

(i) |X n | → 0 if and only if n→∞ P

(ii) |X n | → 0 if and only if n→∞

16.

P |X n | → 0. 1+|X n | n→∞ |X n | → 0. E 1+|X n | n→∞

Hint: For part (i), see Exercise 8 in Chapter 3. (i) For r > 1 and x j ∈ , j = 1, . . . , n, show that

r

n n

−1

−1

n

xj ≤ n |x j |r .

j=1 j=1 (ii) From part (i), deduce that

r

n

1 n

1 E

X j

≤ E|X j |r . n n

j=1

j=1

(On the other hand, by the Minkowski inequality,

r

n

n

1 1 1/r E 1/r

X j

≤ E |X j |r .) n n

j=1

j=1 17. For n = 1, 2, . . ., suppose that X n and X are r.v.s defined on the probability space P

(, A, P), and suppose that X n → X and {X n } are uniformly integrable. Then n→∞ show that A X n dP → A X dP uniformly in A ∈ A. n→∞

18. If for n = 1, 2, . . ., the r.v.s {X n } and {Yn } are uniformly integrable, show that {X n + Yn } are also uniformly integrable. 19. For n = 1, 2, . . ., let X n be identically distributed with E X n ∈ . Then show n that { X¯ n } are uniformly integrable, where X¯ n = n1 X j is the sample mean of X 1, . . . , X n .

j=1

113

114


a.s.

20. Let Y = sup |X n | with EY r < ∞ (for some r > 0) and X n → X . Then show n→∞

n≥1

that

E|X |r

(r )

< ∞ and X n → X . n→∞

21. For n = 1, 2, . . ., let X n be r.v.s such that E X n = μn ∈ and Var(X n ) = P

σn2 → 0. Then show that X n − μn → 0. n→∞

n→∞

22. Let the r.v.s X n , n ≥ 1, be defined as follows:X n (ω) = 2n if ω ∈ (0, n1 ), and X n (ω) = 0 otherwise, where (, A, P) = ((0, 1], B(0,1] , λ) and λ is the Lebesgue measure. Then show that X n → 0 pointwise, but E|X n |r to any n→∞

finite number for any r > 0; in fact, E|X n |r → ∞ for any r > 0.

n→∞

n→∞

23. For n = 1, 2, . . ., let the r.v. X n be defined by

⎧ 1 ⎪ ⎪ n c with probability ⎪ ⎪ n ⎨ 2 Xn = 0 with probability 1 − ⎪ n ⎪ ⎪ ⎪ ⎩ n −c with probability 1 , where c is a positive constant. n Then show that P

(i) X n → 0. (ii) (iii)

n→∞ E|X n |r → 0, 0 < n→∞ (r ) X n → 0, cr < 1. n→∞

cr < 1; E|X n |r → ∞, cr > 1. n→∞

24. Let E X n2 < ∞ for all n. Then, if E|X n − X |2 → 0, show that E|X n2 − X 2 | → 0. n→∞

n→∞

25. For n = 1, 2, . . ., let X n , Yn and X , Y be r.v.s defined on the probability space (r )

(s)

n→∞

n→∞

(, A, P), and suppose that X n → X , Yn → Y where r , s > 1 with 1 s

(1)

1 r

+

= 1. Then show that X n Yn → X Y . n→∞

26. Let (, A, P) = ((0, 1), B(0,1) , λ), where λ is the Lebesgue measure, and let X n be a r.v. defined by ⎧ 1 ⎪ ⎪ ⎨ 1, ω ∈ 0, n , n = 1, 2, . . . . Xn = 1 ⎪ ⎪ ,1 ⎩ 0, ω ∈ n (r )

a.s.

Then show that X n → 0 for any r > 0, and X n −→ 0; indeed, X n −→ 0 n→∞ n→∞ n→∞ pointwise. (Compare it with Exercise 7.)


(r )

27. If X n → X , show that there is a subsequence {X n k } ⊆ {X n } such that n→∞ a.s.

X nk → X . n→∞

28. For n = 1, 2, . . ., let the r.v. X n be defined by Xn =

en with probability n −2 0 with probability 1 − n −2 .

Then show that P

a.s.

n→∞ (r )

n→∞

(i) X n → 0 (indeed, X n → 0). (ii) X n 0 for any r > 0 (in fact, E|X n |r −→ ∞ for all r > 0). n→∞ n→∞ (Compare it with Exercises 6 and Theorem 8.) 29. For n = 1, 2, . . ., let the r.v. X n be defined by ⎧ cn with probability 2−n ⎨2 Xn = 0 with probability 1 − 2−n+1 ⎩ cn −2 with probability 2−n , for some positive constant c. a.s.

(i) Show that X n → 0. n→∞

(r )

(ii) Determine the condition c and r (> 0) must satisfy, so that X n → 0. n→∞ (Compare it with Exercises 6 and 8.) 30.

(i) For any r.v. X , show that P(|X | ≥ c) ≤ E|X |/c (c > 0), by using the obvious inequality |X | ≥ cI (|X | ≥ c). In particular, if X ≥ 0, then P(X ≥ c) ≤ E X /c. (ii) For any r.v. X and any c > 0, show that P(X ≥ c) ≤ e−tc Eet X (t > 0), P(X ≤ c) ≤ e−tc Eet X (t < 0). (iii) If E X 2 < ∞, then show that P(|X | > 0) ≥ (E|X |)2 /E X 2 . In particular, if X ≥ 0, then P(X > 0) ≥ (E X )2 /E X 2 .

31. For n ≥ 1, let X n be r.v.s with E X n = μn , σ 2 (X n ) = σn2 , and suppose that, as n → ∞, μn → μ ∈ , and σn2 → 0. Then show that E(X n − μ)2 → 0 and P

X n → μ. 32. For n ≥ 1, let X n and Yn be r.v.s defined on the probability space (, A, P), and P

suppose that |X n | and |Yn |, n ≥ 1, are uniformly integrable and X n − Yn → 0 as n → ∞. Then show that: (i) |X n − Yn |, n ≥ 1, are uniformly integrable. (ii) E|X n − Yn | −→ 0. n→∞

115

116


33. Consider the probability space (, A, P), and let be an open subset of . Let g(·; ·) : × → be (A × B )-measurable, where B is the σ -field of Borel subsets of . We say that g(·; θ ) is differentiable in q.m. at θ , if there exists a ˙ θ ) (the quadratic mean derivative of g(·; θ ) (A × B )-measurable function g(·; at θ ) such that q.m. ˙ θ ). h −1 [g(·; θ + h) − g(·; θ )] −→ g(·; n→∞

1 −|x−θ| ,x 2e

Now let p(x; θ ) = ∈ (θ ∈ ) be the double exponential probability density function, and for θ, θ ∗ ∈ , set 1 1 g(x; θ, θ ∗ ) = [ p(x; θ ∗ )/ p(x; θ )]1/2 = exp |x − θ | − |x − θ ∗ | . 2 2 (i) Show that g(x; θ, θ ∗ ) is not pointwise differentiable with respect to θ ∗ at (θ, θ ) for any θ = x ∈ . (ii) If the r.v. X is distributed according to p(·; θ ), show that g(X ; θ, θ ∗ ) is differentiable in q.m. with respect to θ ∗ at (θ, θ ), θ ∈ , with q.m. derivative g(X ˙ ; θ ) given by: ⎧ −1/2 if X < θ ⎨ g(X ˙ ; θ ) = 0 (for example) if X = θ . ⎩ 1/2 if X > θ 34. All r.v.s appearing below are defined on the probability space (, A, P). (i) If X ≥ 0, then E X ≥ 0 and E X = 0 only if P(X = 0) = 1. (ii) Let X ≥ Y with finite expectations. Then E X ≥ EY and E X = EY only if P(X = Y ) = 1. (iii) Let X > Y with finite expectations. Then E X > EY . (iv) Let g : I open subset of be strictly convex (i.e., g[αx + (1 − α)x ] < αg(x) + (1 − α)g(x ), x, x ∈ I , 0 < α < 1), let Z be a r.v. taking values in I , let E Z ∈ I , and let E g(Z ) exist. Then E g(Z ) > g(E Z ) unless P(Z = constant) = 1.

CHAPTER

The Hahn–Jordan Decomposition Theorem, The Lebesgue Decomposition Theorem, and the Radon–Nikodym Theorem

7

This chapter revolves around three classical theorems. In Section 7.1, it is shown that any σ -additive set function ϕ may be decomposed into the difference of two measures (i.e., ϕ is a signed measure). This is a brief description of the Hahn–Jordan Decomposition Theorem. In the following section, this decomposition is used in order to obtain the Lebesgue Decomposition Theorem. This theorem states that, if μ and ν are any two σ -finite measures, then ν can be decomposed uniquely into two components νc and νs . The component νc is absolutely continuous with respect to μ, and νs is singular with respect to μ. In the third and final section of the chapter, the Lebesgue Decomposition Theorem is specialized to the case where ν μ in order to establish the Radon–Nikodym Theorem. Namely, to show that ν is the indefinite integral, with respect to μ, of a r.v. that is nonnegative, a.e. [μ] finite, and a.e. [μ] unique. A corollary to this theorem justifies the replacement in integrals of a probability distribution by its probability density function (p.d.f.) with respect to a dominating measure, as is routinely done in Statistics.

7.1 The Hahn–Jordan Decomposition Theorem In all that follows, ϕ is a σ -additive set function defined on (, A) and taking values in (−∞, ∞]. The value −∞ is excluded in order to avoid expressions of the form ∞ − ∞. It will also be assumed that ϕ is finite for at least one set A. Then from A = A + + + · · · and the σ -additivity of ϕ it follows that ϕ() = 0. From this fact it also follows that ϕ is finitely additive. Definition 1. The continuity of ϕ is defined the same way the continuity of a measure has been defined; i.e., ϕ is continuous from below if An ↑ A as n → ∞ implies ϕ(An ) → ϕ(A); it is continuous from above if An ↓ A as n → ∞ with |ϕ(An 0 )| < ∞ for some n 0 implies ϕ(An ) → ϕ(A); and it is continuous if it is n→∞ continuous from both below and above. An Introduction to Measure-Theoretic Probability, Second Edition. http://dx.doi.org/10.1016/B978-0-12-800042-7.00007-4 Copyright © 2014 Elsevier Inc. All rights reserved.

117

118

CHAPTER 7 The Hahn–Jordan Decomposition Theorem

Lemma 1. Proof.

Every σ -additive set function ϕ is continuous.

Let An ↑ A as n → ∞ and suppose first that |ϕ(An )| < ∞, for all n. Then A=

∞

A j = A1 +

j=1

∞ (A j − A j−1 ) j=2

and hence ϕ(A) = ϕ(A1 ) +

∞

ϕ(A j − A j−1 ).

j=2

But A j−1 ⊆ A j implies that A j = A j−1 + (A j − A j−1 ), so that ϕ(A j ) = ϕ(A j−1 ) + ϕ(A j − A j−1 ), and since |ϕ(An )| < ∞, for all n, we get ϕ(A j − A j−1 ) = ϕ(A j ) − ϕ(A j−1 ). Therefore ϕ(A) = ϕ(A1 ) + lim

n→∞

n

ϕ(A j − A j−1 )

j=2

= ϕ(A1 ) + lim ϕ(An ) − ϕ(A1 ) = lim ϕ(An ). n→∞

n→∞

Now suppose that ϕ(An 0 ) = ∞ for some n 0 . Then for all n ≥ n 0 , An = An 0 + (An − An 0 ), so that ϕ(An ) = ϕ(An 0 )+ϕ(An − An 0 ) = ∞. Also A = An 0 +(A− An 0 ) implies ϕ(A) = ϕ(An 0 ) + ϕ(A − An 0 ) = ∞, so that ϕ(An ) → ϕ(A). n→∞

Next, let An ↓ A asn → ∞ and |ϕ(An 0 )| < ∞ for some n o . Then An 0 = A+ ∞ j=n 0 A j − A j+1 , so that ϕ(An 0 ) = ϕ(A) + + lim

n→∞

∞ j=n 0 n

ϕ(A j − A j+1 ) = ϕ(A) ϕ(A j − A j+1 ).

j=n 0

For j ≥ n 0 , A j = A j+1 + (A j − A j+1 ), so that ϕ(A j ) = ϕ(A j+1 ) + ϕ(A j − A j+1 ), and since ϕ(An 0 ) is finite, so are ϕ(A j+1 ) for all j ≥ n 0 (as follows by induction). Hence ϕ(An 0 ) = ϕ(A) + lim ϕ(An 0 ) − ϕ(An+1 ) = ϕ(A) + ϕ(An 0 ) − lim ϕ(An ). n→∞

Thus lim ϕ(An ) = ϕ(A) and ϕ is continuous. n→∞

n→∞

7.1 The Hahn–Jordan Decomposition Theorem

Definition 2. For a set function ϕ on (, A), define the set functions ϕ + and ϕ − on (, A), as follows: ϕ + (A) = sup{ϕ(B); B ⊆ A, B ∈ A}, B ⊆ A,

ϕ − (A) = − inf{ϕ(B);

B ∈ A}.

Remark 1. Both ϕ + (A) and ϕ − (A) are ≥ 0 for all A. This is so, because ϕ(∅) = 0 implies ϕ + (A) ≥ 0 for all A. Also, inf{ϕ(B); B ⊆ A} = − sup{−ϕ(B); B ⊆ A}, or − inf{ϕ(B); B ⊆ A} = sup{−ϕ(B); B ⊆ A}, and since −ϕ() = 0, we have sup{−ϕ(B); B ⊆ A} ≥ 0, and hence ϕ − (A) ≥ 0 for all A. Then we may formulate and prove the following result. Theorem 1 (Hahn–Jordan Decomposition Theorem). Let ϕ be a σ -additive function defined on (, A), let ϕ + , ϕ − be as in Definition 2, and let −∞ < m = inf{ϕ(A); A ∈ A} ≤ sup{ϕ(A); A ∈ A} = M ≤ ∞. Then (i) There exists at least one set D ∈ A such that ϕ(A) ≥ 0 for every A ⊆ D and ϕ(A) ≤ 0 (A ∈ A). for every A ⊆ D c (ii) ϕ + (A) = ϕ(A ∩ D), ϕ − (A) = −ϕ(A ∩ D c ). (iii) ϕ + , ϕ − , |ϕ| = ϕ + + ϕ − are measures and ϕ − is finite. (iv) ϕ = ϕ + − ϕ − (in the sense that ϕ(A) = ϕ + (A) − ϕ − (A), A ∈ A). Proof.

(i) For j = 1, 2, . . ., let ε j > 0 and j ε j < ∞ e.g., ε j = A j ∈ A be such that m ≤ ϕ(A j ) ≤ m + ε j . Next, from A2 = (A1 ∩ A2 ) + (A2 − A1 ),

1 j2

. For each j, let

A1 ∪ A2 = A1 + (A2 − A1 ),

it follows that ϕ(A1 ∩ A2 ) = ϕ(A2 ) + ϕ(A1 ) − ϕ(A1 ∪ A2 ). Therefore m ≤ ϕ(A1 ∩ A2 ) = ϕ(A2 ) + ϕ(A1 ) − ϕ(A1 ∪ A2 ) ≤ 2m + ε1 + ε2 − m = m + ε1 + ε2 . Thus, m ≤ ϕ(A1 ∩ A2 ) ≤ m + (ε1 + ε2 ) and, clearly, this is true for any two sets An , An+1 . Furthermore, it is easily seen by induction that m ≤ ϕ(

k j=n

Aj) ≤ m +

k j=n

εj.

119

120


Letting k → ∞ and utilizing the continuity of ϕ, we obtain m ≤ ϕ(

∞

Aj) ≤ m +

j=n

Now set D c = lim inf An = n→∞

n→∞

n=1

∞ j=n

ϕ(D c ) = ϕ(lim

∞

εj.

(7.1)

j=n

∞ ∞

lim sup Acn . Then, as n → ∞,

∞

j=n

A j , so that D =

∞ ∞ n=1

j=n

Acj =

A j ↑ D c and by continuity of ϕ,

A j ) = lim ϕ(

j=n

∞

Aj) = m

j=n

on account of (7.1). That is, ϕ(D c ) = m. Next, let A ⊆ D c . Then D c = A + (D c − A) and hence m = ϕ(D c ) = ϕ(A) + ϕ(D c − A) ≥ ϕ(A) + m, so that ϕ(A) ≤ 0. Finally, if A ⊆ D, then m ≤ ϕ(A + D c ) = ϕ(A) + ϕ(D c ) = ϕ(A) + m, so that ϕ(A) ≥ 0. This completes the proof of (i). (ii) Let B ⊆ A. Then ϕ(B) = ϕ[(B ∩ D) + (B ∩ D c )] = ϕ(B ∩ D) + ϕ(B ∩ D c ) ≤ ϕ(B ∩ D) (since ϕ(B ∩ D c ) ≤ 0) ≤ ϕ(B ∩ D) + ϕ[(A − B) ∩ D] (since ϕ[(A − B) ∩ D] ≥ 0) = ϕ(A ∩ D)

(since (B ∩ D) + (A − B) ∩ D = A ∩ D).

That is, for every B ⊆ A, ϕ(B) ≤ ϕ(A∩ D) and this is, in particular, true for B ⊆ (A ∩ D). Since in forming sup ϕ(B) we may restrict ourselves to B ⊆ A B⊆A

with B ⊆ A ∩ D (because ϕ(C ∩ D c ) ≤ 0), we have that ϕ(B) ≤ϕ(A ∩ D) for every B ⊆ A ∩ D and hence ϕ(A ∩ D) = sup ϕ(B) = ϕ + (A). B⊆A

Next, for B ⊆ A,

ϕ(B) = ϕ(B ∩ D) + ϕ(B ∩ D c ) ≥ ϕ(B ∩ D c ) (since ϕ(B ∩ D) ≥ 0) ≥ ϕ(B ∩ D c ) + ϕ[(A − B) ∩ D c ] (since ϕ[(A − B) ∩ D c ] ≤ 0) = ϕ(A ∩ D c ) (since (B ∩ D c ) + (A − B) ∩ D c = A ∩ D c ). That is, for every B ⊆ A, ϕ(B) ≥ ϕ(A ∩ D c ) and this is, in particular, true for B ⊆ A ∩ D c . Since in forming sup [−ϕ(B)] we may restrict ourselves to B⊆A

B ⊆ A with B ⊆ A ∩ D c (because ϕ(C ∩ D) ≥ 0), we have that −ϕ(B) ≤ −ϕ(A ∩ D c ) for every B ⊆ A ∩ D c ,

7.1 The Hahn–Jordan Decomposition Theorem

and hence −ϕ(A ∩ D c ) = sup [−ϕ(B)] = − inf ϕ(B) = ϕ − (A). B⊆A

B⊆A

(iii) That ϕ + is a measure follows from the fact that ϕ + (A) = ϕ(A ∩ D), ϕ() = 0, ϕ(B) ≥ 0 for every B ⊆ D and the σ -additivity of ϕ; similarly for ϕ − . Finally, from (−∞ −∞, but the proof is somewhat more complicated (see, e.g., pages 86–87 in Loève (1963), or pages 104–106 in Neveu (1965)). (ii) If μ1 , μ2 are two measures such that μ1 (A)−μ2 (A) is defined for every A ∈ A, then μ1 − μ2 is called a signed measure. Thus the theorem shows that every σ -additive function is a signed measure. (iii) For the set D, one has ϕ(D) = M and ϕ(D c ) = m. In fact, there exist Bn ∈ A such that ϕ(Bn ) → M. Then n→∞

ϕ(D) = ϕ(D ∩ Bn ) + ϕ(D ∩ Bnc ) ≥ ϕ(D ∩ Bn ) (since ϕ(D ∩ Bnc ) ≥ 0), ≥ ϕ(D ∩ Bn ) + ϕ(D c ∩ Bn ) (since ϕ(D c ∩ Bn ) ≤ 0) = ϕ(Bn ) (by finite additivity of ϕ). That is, ϕ(D) ≥ ϕ(Bn ) and hence, as n → ∞, ϕ(D) ≥ M. Since also ϕ(D) ≤ M, it follows that ϕ(D) = M. Next, recall that m = inf{ϕ(A); A ∈ A}. Then there exists Cn ∈ A such that ϕ(Cn ) → m as n → ∞. Furthermore, ϕ(D c ) = ϕ(D c ∩ Cn ) + ϕ(D c ∩ Cnc ) ≤ ϕ(D c ∩ Cn ) (since ϕ(D c ∩ Cnc ) ≤ 0), ≤ ϕ(D c ∩ Cn ) + ϕ(D ∩ Cn ) (since ϕ(D ∩ Cn ) ≥ 0), = ϕ(Cn ) (by finite additivity of ϕ). That is, ϕ(D c ) ≤ ϕ(Cn ) and hence, as n → ∞, ϕ(D c ) ≤ m. Since also ϕ(D c ) ≥ m, it follows that ϕ(D c ) = m. Corollary. Under the assumptions of the theorem, |ϕ| is bounded if and only if |ϕ|() < ∞, or if and only if ϕ + () < ∞.

121

122


Proof. The first assertion follows by the fact that |ϕ| is a measure, so that |ϕ|(A) ≤ |ϕ|() for all A ∈ A. The second assertion follows by the fact that ϕ − () ≤ −m < ∞, and the expression |ϕ|() = ϕ + () + ϕ − (). Definition 3. If μ, ν are two measures on A, we recall that ν is said to be μcontinuous (or absolutely continuous with respect to μ), denoted by ν μ, if μ(A) = 0 implies that ν(A) = 0. We also say that ν is dominated by μ or μ dominates ν. If ν μ and μ ν, then μ and ν are said to be mutually absolutely continuous and write μ ≈ ν. ν is said to be μ-singular (or singular with respect to μ) if there exists N ∈ A with μ(N ) = 0 such that ν(A) = ν(A ∩ N ) for every A ∈ A. We also say that ν and μ are orthogonal and write ν ⊥ μ.

7.2 The Lebesgue Decomposition Theorem Theorem 2 (Lebesgue Decomposition Theorem). Let μ, ν be two σ -finite measures on A. Then (i) There exists a decomposition of ν into a μ-continuous measure νc and a μsingular measure νs such that ν = νc + νs (in the sense that ν(A) = νc (A) + νs (A), A ∈ A). (ii) The decomposition in (i) is unique. r.v. X determined (iii) νc is the indefinite integral of a nonnegative, a.e. [μ] finite (A) = X dμ and if νc (A) = up to μ-equivalence (i.e., if for every A ∈ A, ν c A dμ for another r.v. X , then μ(X = X ) = 0). X A Proof. Case 1: μ, ν finite. (i) Let X = {X ≥ 0, a.e. [μ] finite r.v.s; A X dμ ≤ ν(A), A ∈ A}. Then X = 0 ∈ X , so that X = , and for every X ∈ X , (0 ≤) X dμ ≤ ν() < ∞. Thus de f X dμ = α < ∞ and there exists {X n } ⊆ X such that sup X ∈X X n dμ → α. n→∞

Define Yn by: Yn = max X k . Then Yn ↑ as n → ∞ and let X = lim Yn . Thus n→∞ 1≤k≤n 0 ≤ Yn ↑ X as n → ∞ and hence Y dμ ↑ X dμ as n → ∞. On the other n hand, X n ≤ Yn and hence X n dμ ≤ Yn dμ. Letting n → ∞, we get then α ≤ X dμ. Thus X dμ ≥ α. (7.2) Now define A1 = (Yn = X 1 ) A2 = (Yn = X 2 ) − A1

7.2 The Lebesgue Decomposition Theorem

A3 = (Yn = X 3 ) − (A1 + A2 ) . . . . . . . . . . An = (Yn = X n ) − (A1 + A2 + · · · + An−1 ). Then, clearly, {A1 , . . . , An } is a partition of and therefore for any A ∈ A, one has n n Yn dμ = Yn dμ = Yn dμ = X j dμ nj=1 A∩A j

A

≤

n j=1

j=1

A∩A j

j=1

A∩A j

ν(A ∩ A j ) = ν(A); Yn dμ ≤ ν(A) for every A ∈ A, and hence Yn ∈ X .

i.e., A

Thus Yn dμ ≤ α and since Yndμ ↑ X dμ as n → ∞, one has X dμ ≤ α. This result, together with (7.2), gives X dμ = α. Furthermore, Yn dμ ≤ ν(A) for every A ∈ A and Yn dμ ↑ X dμ as n → ∞ A

A

A

(as follows from 0 ≤ Yn ↑ X as n → ∞) imply that A X dμ ≤ ν(A) for every A ∈ A . Since (0 ≤) X dμ = α < ∞ implies that X is a.e. [μ] finite, we have that X ∈ X and X dμ = α. Now define νc on A as follows: νc (A) = A X dμ. Then νc is μ-continuous. Next, define νs by: νs (A) = ν(A) − νc (A), A ∈ A. Since X dμ ≥ 0, ν(A) − νc (A) = ν(A) − A

by the fact that X ∈ X , we have that νs (A) ≥ 0, A ∈ A. Since both ν and νc are measures, it follows that νs is itself a measure. The proof of (i) will be completed by showing that νs is μ-singular. To this end, for each n define the set function ϕn as follows: 1 ϕn (A) = νs (A) − μ(A), A ∈ A. n Then, clearly, ϕn is a finite σ -additive set function. Hence, by Theorem 1, there exists Dn ∈ A such that

Set D =

∞

ϕn (A ∩ Dn ) ≥ 0, ϕn (A ∩ Dnc ) ≤ 0,

c j=1 D j .

A∩

r j=1

A ∈ A.

Then, as r → ∞,

D cj ↓ A ∩ D and hence ϕn (A ∩

r j=1 j =n

D cj ) → ϕn (A ∩ D) r →∞

(7.3)

123

124


by Lemma 1. On the other hand, for r ≥ n, ϕn (A ∩

r

D cj ) = ϕn [(A ∩

j=1

r

D cj ) ∩ Dnc ] ≤ 0 by (7.3)

j=1 j =n

(where the role of A in (7.3) is played by A ∩ rj=1 D cj here) for all n ≤ r ; i.e., j =n

ϕn (A ∩ rj=1 D cj ) ≤ 0, n ≤ r , and hence, as r → ∞ ϕn (A ∩ D) ≤ 0 , for all n. Equivalently, νs (A ∩ D) −

1 1 μ(A ∩ D) ≤ 0 for all n, or νs (A ∩ D) ≤ μ(A ∩ D) n n

for all n. Letting n → ∞, we get νs (A ∩ D) = 0, A ∈ A. Since νs (A) = νs (A ∩ D) + νs (A ∩ D c ), we have that νs (A) = νs (A ∩ D c ). Thus νs will be μ-singular if we show that μ(D c ) = 0. To this end, we have νc (A) = ν(A) − νs (A) = ν(A) − νs (A ∩ D c ) ≤ ν(A) − νs (A ∩ Dn ) (since Dn ⊆ D c ). Thus

A

1 1 X + I Dn dμ = νc (A) + μ(A ∩ Dn ) n n (from the definition of νc ) 1 ≤ ν(A) − νs (A ∩ Dn ) + μ(A ∩ Dn ) n (by the previous inequality) 1 = ν(A) − [νs (A ∩ Dn ) − μ(A ∩ Dn )] n = ν(A) − ϕn (A ∩ Dn ) (from the definition of ϕn ),

≤ ν(A) (since ϕn (A ∩ Dn ) ≥ 0). So, for every A ∈ A, one has A X + n1 I Dn dμ ≤ ν(A), so that (X + n1 I Dn ) ∈ X . Hence (X + n1 I Dn )dμ ≤α. But A (X + n1 I Dn )dμ ≤ α + n1 μ(Dn ), since X dμ = α;

c therefore α+ n1 μ(Dn ) ≤ α, so that μ(Dn ) = 0, for all n. Since D = ∞ j=1 D j implies n c c j=1 D j ↑ D as n → ∞, we obtain that μ(D ) = 0. The proof of (i) is complete. (ii) In (i) we proved that ν = νc + νs , where X dμ = X dμ + X dμ νc (A) = A∩D A∩D c A = X dμ = νc (A ∩ D), A∩D


since μ(A ∩ D c ) ≤ μ(D c ) = 0. Also let ν = νc + νs , where νc (A) = νc (A ∩ D0 ), νs (A) = νs (A ∩ D0c ) with μ(D0c ) = 0. We have

νc (A) + νs (A) = νc (A) + νs (A) (= ν(A)) or νc (A) − νc (A) = νs (A) − νs (A) for every A ∈ A.

(7.4)

Set N = D c ∩ D0c . Then μ(N ) = 0 and hence νc (N ) = νc (N ) = 0, since they are both μ-continuous. For A ∈ A, write A = (A ∩ N ) + (A ∩ N c ). Then νc (A) − νc (A) = [νc (A ∩ N ) − νc (A ∩ N )] + [νc (A ∩ N c ) − νc (A ∩ N c )]

= [νc (A ∩ N ) − νc (A ∩ N )] + [νs (A ∩ N c ) − νs (A ∩ N c )] (by (7.4)).

But νc (A ∩ N ) = νc (A ∩ N ) = 0 since A ∩ N ⊆ N and μ(N ) = 0. Next, νs (A ∩ N c ) = νs (A ∩ N c ) = 0 because A ∩ N c = A ∩ D ∩ D0 ⊆ A ∩ D, A ∩ D0 and νs (A ∩ D) = νs (A ∩ D0 ) = 0. So νc = νc and therefore νs = νs . (iii) In (i) it was seen that νc (A) = A X dμ for every A ∈ A. Let also νc (A) = A X dμ for every A ∈ A. To show that X = X a.e.[μ]. In fact, if μ(X −X > 0) > 0, then there exists ε > 0 such that μ(X − X > ε) > 0. Hence (X −X >ε) (X − X )dμ ≥ εμ(X − X > ε) > 0, which is a contradic tion (to the assumption that A X dμ = A X dμ = νc (A) for every A A). Thus μ(X − X > 0) = 0 and similarly μ(X − X < 0) = 0, so that μ(X = X ) = 0. Case 2: μ, ν σ -finite. (i) From the σ -finiteness of μ, ν, there exist two countable partitions of {A j , j = 1, 2, . . .}, {Aj , j = 1, 2, . . .}, say, such that μ(A j ), ν(Aj ) < ∞, j ≥ 1. Consider the intersection of these two partitions, which is another partition of ; call it {B j , j ≥ 1}. Then μ(B j ), ν(B j ) < ∞, j ≥ 1. Consider the restrictions μn , νn of μ, ν on Bn ; i.e., μn (A) = μ(A ∩ Bn ) μ(A) = n μn (A) Then , A ∈ A. (7.5) νn (A) = μ(A ∩ Bn ) ν(A) = n νn (A). On each Bn , the theorem is true. Therefore νn = νnc +νns , where νnc is μn -continuous and νns is μn -singular. We assert that νnc is, actually, μ-continuous. In fact, let μ(A) = 0. Then μn (A) = 0 for all n and hence νnc (A) = 0 for all n. Next, from the μn -singularity of νns we have that there exists Nn ∈ A such that μn (Nn ) = 0 and νns (A) = νns (A ∩ Nn ), A ∈ A, so that νns (A ∩ Nnc ) = 0, A ∈ A. Look at Bnc . Then νn (A ∩ Bnc ) = μ(A ∩ Bnc ∩ Bn ) = μ() = 0, and similarly μn (A ∩ Bnc ) = 0. But νnc is μn -continuous. Hence νnc (A ∩ Bnc ) = 0 and this, together with νn (A ∩ Bnc ) = 0, implies that νns (A ∩ Bnc ) = 0. To summarize, νns (A ∩ Nnc ) = 0, νns (A ∩ Bnc ) = 0, A ∈ A.

(7.6)

125

126


We assert that νns is μ-singular. In fact, νns [A ∩ (Bn ∩ Nn )c ] = νns [A ∩ (Bnc ∪ Nnc )] = νns [(A ∩ Bnc ) ∪ (A ∩ Nnc )] = νns (A ∩ Bnc ) + νns (A ∩ Nnc ) −νns (A ∩ Bnc ∩ Nnc ) = 0 by (7.6). So for the set Mn = Bn ∩ Nn , we have that μn (Mn ) = 0 (since μn (Nn ) = 0) and νns (A ∩ Mnc ) = 0. But μn (Mn ) = μ(Mn ∩ Bn ) = μ(Mn ) from (7.5) and the fact that Mn ⊆ Bn . Thus μ(Mn ) = 0 and νns (A ∩ Mnc ) = 0, which is equivalent to saying that νns is μ-singular. Up to this point we have shown that νnc is μ-continuous and νns is μ-singular (on Bn ). The μ-singularity of νns implies the existence of a set Nn ∈ A such that μ(Nn ) = 0 and νns (A ∩ Nnc ) = 0. Set N = ∞ j=1 N j . Then μ(N ) = 0 and νns (A ∩ N ) = νns [A ∩ ( c

∞

N j )c ]

j=1

= νns [A ∩ (

∞

N jc )]

j=1

c

≤ νns (A ∩ Nn ) = 0; i.e., μ(N ) = 0, and hence νnc (N ) = 0, and νns (A ∩ N c ) = 0. Next, νnc (A) = νnc (A ∩ N c ) + νnc (A ∩ N ) = νnc (A ∩ N c ) = νnc (A ∩ N c ) + νns (A ∩ N c ) = νn (A ∩ N c ). That is, νnc (A) = νn (A ∩ N c ) and therefore de f

νc (A) =

νnc (A) =

n

νn (A ∩ N c ) = ν(A ∩ N c ) (by (7.5)).

(7.7)

n

Next, νns (A) = νns (A ∩ N ) + νns (A ∩ N c ) = νns (A ∩ N ) = νns (A ∩ N ) + νnc (A ∩ N ) = νn (A ∩ N ). That is, νns (A) = νn (A ∩ N ) and therefore de f

νs (A) =

n

νns (A) =

νn (A ∩ N ) = ν(A ∩ N ) (by (7.5)).

(7.8)

n

From (7.7) and (7.8), we have that ν(A) = νc (A) + νs (A), A ∈ A, whereas νc is μ-continuous since every νnc is μ-continuous. Furthermore, νs is μ-singular since νs (A ∩ N c ) = ν(A ∩ N c ∩ N ) = ν() = 0. This completes the proof of (i).


(ii) By (i), ν = νc + νs , where νc (A) = ν(A ∩ N c ), νs (A) = ν(A ∩ N ), A ∈ A, and μ(N ) = 0. Let ν = νc + νs be another decomposition of ν into a μ-continuous measure νc and a μ-singular measure νs . For each n, consider the and ν of ν, ν , ν , ν , and ν , respectively, to B . restrictions νn , νnc , νns , νnc c s n ns c s + ν and hence ν = ν , ν = ν by part Then we have νn = νnc + νns = νnc nc ns ns nc ns (ii) in Case 1. Since νc (A) =

νnc (A ∩ Bn ), νs (A) =

n

and νc (A) =

νns (A ∩ Bn )

n

νnc (A ∩ Bn ), νs (A) =

n

νns (A ∩ Bn )

n

we have νc = νc , νs = νs , as was to be seen. (iii) From part (i) of Case 1, we have X n dμn , νnc (A) =

A ∈ A,

A

where X n is ≥ 0 and an a.e. [μn ] finite r.v. Actually, since μn assigns measure 0 outside Bn , we may assume that X n is 0 on Bnc . In the course of the proof of (i) in the present case, it was seen that νnc (A ∩ Bnc ) = 0. Therefore νnc (A) = νnc (A ∩ Bn ) and hence X n dμn = X n dμn . A

A∩Bn

On , define X as follows: X = n X n . Then, clearly, X (ω) = X n (ω) for ω ∈ Bn , and X n dμn = X n dμn = X n dμ = X dμ. νnc (A) = A

Therefore νc (A) =

A∩Bn

A∩Bn

n

X dμ = A∩Bn

A∩Bn

X dμ. A

That X is a.e. [μ] well defined follows as in part (iii) of Case 1.

Remark 3. The theorem is true if ν is only a σ -additive, σ -finite set function (see page 132 in Loève (1963)), but then the r.v. X need not be ≥ 0. Remark 4. The theorem is still true if ν is not even a σ -finite set function; i.e., if ν is only a σ -additive set function the theorem is true, but then X need be neither ≥ 0 nor a.e. [μ] finite. Remark 5. measure.

The theorem need not be true if μ is not σ -finite even if ν is a finite

127

128


7.3 The Radon–Nikodym Theorem Theorem 3 (Radon–Nikodym Theorem). Let μ, ν be two σ -finite measures on A such that ν is μ-continuous (ν μ). Then ν is the indefinite integral of a nonnegative, a.e. [μ] finite r.v. X that is unique up to μ-equivalence. Proof. By Theorem 2, ν = νc + νs , where νc μ and νs is μ-singular. So there exists N ∈ A with μ(N ) = 0 such that νs (A) = νs (A ∩ N ), A ∈ A. Then, for A ∈ A, (7.9) ν(A) = νc (A) + νs (A). In (7.9), take A = N and use the assumption that ν μ and the fact that νc μ to obtain: 0 = ν(N ) = νc (N ) + νs (N ) = 0 + νs (N ) = νs (N ). That is, νs (N ) = 0 and hence νs (A) = νs (A ∩ N ) ≤ νs (N ) = 0. In other words, νs (A) = 0 for all A ∈ A, orto put it differently, νs = 0. It follows that ν = νc , and therefore ν(A) = νc (A) = A X dμ, A ∈ A, for a nonnegative r.v. X which is a.e. [μ] finite and a.e. [μ] uniquely determined. Remark 6. It is to be emphasized that a.e. [μ] uniqueness of X means that, if Y is another nonnegative, a.e. [μ] finite r.v. such that A X dμ = A Y dμ, A ∈ A, then X = Y a.e. [μ]. Actually, this is true for any two r.v.s X and Y (that or if extended are finite everywhere, r.v.s., are a.e. [μ] finite) for which the integrals X dμ and Y dμ exist. That is, the following result holds. Proposition 1. Let X and Y be finite, (real-valued) r.v.s, or extended r.v.s but a.e. [μ] and assume that the integrals X dμ and Y dμ exist. Then A X dμ = A Y dμ for all A ∈ A implies that X = Y a.e. [μ]. Proof. Indeed, the existence of X dμ and Y dμ implies the existence of A X dμ and A Y dμ, A ∈ A (by the Corollary to Theorem 5 in Chapter 4). Assume first since | A X dμ| = | X I A dμ| ≤ that X dμ is finite. Then A X dμ is also finite, + dμ and − dμ, because ∞ > X X |X |dμ = |X |dμ < ∞, and so are A A − + X dμ. X dμ + + From dμ is finite and then so are A +Y dμ − X dμ = Y dμ, we have that Y A, asbefore. Next, from A X dμ = A Y dμ, we get A X dμ− and A Y dμ, A∈ − dμ = + dμ− − + − X Y A A A Y dμ, or because offiniteness, A X dμ+ A Y dμ = + − + − + − A Y dμ + A X dμ, or A (X + Y )dμ = A (Y + X )dμ. Since this is true for all A ∈ A and the integrands are ≥ 0 (and a.e. [μ] finite), we obtain X + + Y − = Y + + X − a.e. [μ], or X + − X − = Y + − Y − a.e. [μ], or X = Y a.e. [μ]. Similarly if Y dμ is finite. Now, that X dμ = ∞. Then from ∞ = X dμ = X + dμ− X − dμ, suppose + X − dμ X + dμ = ∞ and < ∞. Then,for any A ∈ A, AX dμ ≤ ∞ and we get − dμ < ∞. From X dμ = Y dμ, we get Y + dμ = ∞ and Y − dμ < ∞, so X A − that A Y + Therefore the relation A X dμ = A Y dμ A−Y dμ < ∞. dμ +≤ ∞ and becomes A X dμ − A X dμ = A Y + dμ − A Y − dμ, or, because of finiteness of

7.3 The Radon–Nikodym Theorem

− − + A X dμ and A Y dμ, A (X + − + − X + Y = Y + X a.e. [μ],

+ Y − )dμ = A (Y + + X − )dμ. Hence, as before, or X + − X − = Y + − Y − a.e. [μ], or X = Y a.e.

[μ]. Likewise if Y dμ = ∞. from −∞ = X dμ = X + dμ− X − dμ, it follows that If+ X dμ = −∞, then X dμ < ∞ and X − dμ = ∞. Therefore A X + dμ < ∞ and A X − dμ ≤ ∞ for A ∈ A. Once again −∞ = X dμ = Y dμ = Y + dμ − Y − dμ implies all + − + − < ∞ Y dμ < ∞ and A Y dμ ≤ ∞. and Y dμ = ∞, so that AY dμ + − the relation A X dμ = A Y dμbecomes A X dμ− A X dμ = A Y + dμ− Hence − + dμ and + − + becauseof finiteness of A Y dμ, A Y dμ, − A X dμ − A Y or, − AX + − + − + dμ = − A Y dμ − A X dμ, or A (X + Y )dμ = A (Y + X )dμ. It follows that X − + Y + = Y − + X + a.e. [μ], or X + − X − = Y + − Y − a.e. [μ], or X = Y a.e. [μ]. Similarly if Y dμ = −∞. So, in all cases as described above, A X dμ = A Y dμ for all A ∈ A implies X = Y a.e. [μ]. Remark 7. The r.v. X of the theorem is called a Radon–Nikodym derivative of dν ν with respect to μ and is denoted by X = dμ . Let now f : → and suppose x b that −∞ f (t)dt exists. Then for all x ∈ (−∞, b], −∞ f (t)dt = F(x) exists and d F(x) d x = f (x) at all continuity points x ∈ (−∞, b] of f . The point function F may also be thought of as a set function over the intervals (−∞, x], x ∈ (−∞, b]. Thus, dν generalizes the notation d F(x) the notation dμ dx . Remark 8. In most applications, (, A) = (n , B n ), ν = P and μ = λn , the n-dimension Lebesgue measure. This is the case, e.g., with all of the common distributions such as uniform, normal, gamma, beta, or Cauchy. In the discrete case, such as that of binomial, Poisson, negative binomial, or hypergeometric, the measure μ is the counting measure; i.e., the measure that assigns mass one to singletons. Corollary. Let μ and λ be σ -finite measures on A such that μ λ, and let X be a r.v. for which X dμ exists. Then

X dμ = A

X A

dμ dλ for every A ∈ A. dλ

Proof. First, let X = I B for some B ∈ A. Then

X dμ = A

I B dμ = A

dμ = μ(A ∩ B).

(7.10)

A∩B

Also, μ(A ∩ B) = A∩B

dμ dλ = dλ

(I B A

dμ )dλ = dλ

X A

dμ dλ. dλ

(7.11)

129

130


From (7.10) and (7.11), we have then A X dμ = let X = ri=1 αi I Ai , αi ≥ 0, i = 1, . . . , r . Then X dμ =

r

A

αi

I Ai dμ =

A

i=1

r

A

dμ X dλ, A ∈ A. Now, dλ

αi

i=1

A

I Ai

dμ dλ dλ

(by the previous step) r dμ dμ = X dλ, αi I Ai dλ = dλ dλ A A i=1

so, again, the conclusion holds true. Next, let X ≥ 0. Then there exist 0 ≤ X n simple r.v.s ↑ X as n → ∞, which implies 0 ≤ X n I A ↑ X I A as n → ∞. Therefore, by the Lebesgue Monotone Convergence Theorem, X n dμ → X dμ. (7.12) X I A dμ or X n I A dμ → n→∞

n→∞

A

A

However,

dμ dλ (by the previous step) dλ A A dμ dμ dμ ↑X as n → ∞ . (7.13) → X dλ since 0 ≤ X n n→∞ A dλ dλ dλ dμ dλ, A ∈ A. From (7.12) and (7.13), it follows that A X dμ = A X dλ Finally, for any r.v. X for which the X dμ exists, we have

X n dμ =

Xn

+

−

+ dμ

X dλ − X dμ − X dμ = dλ A A A dμ dμ (X + − X − ) X dλ = dλ. = dλ dλ A A Thus, in all cases, A X dμ = A (X dμ dλ )dλ, A ∈ A. X dμ =

A

X A

− dμ

dλ

dλ

Remark 9. The result stated in the preceding corollary is what lies behind the routine replacement in integrals of probability distributions by their p.d.f.s. Thus, let X be a r.v. defined on the probability space (, A, P) with probability distribution function PX , let B ∈ B and A = X −1 (B), and let g : → be (measurable and) such that E g(X ) exists. Then (by Theorem 13 in Chapter 4), d PX g(x) (x) dλ, g(X ) d P = g(x) d PX = dλ A B B


if PX 0 and f 1 = 0, then K ( f 0 , f 1 ) = ∞. = 0 and f 1 > 0, then K ( f 0 , f 1 ) = 0. = f 1 = 0, then K ( f 0 , f 1 ) = 0. > 0 and f 1 > 0, then the integrand is finite.

Thus, it suffices to focus on this last case alone. By setting S0 = ( f 0 > 0) and S1 = ( f 1 > 0), we may assume that S0 = S1 , since the cases S0 − S1 = and S1 − S0 = are covered by (a) and (b) above. So, let S = ( f 0 > 0) = ( f 1 > 0), A = X −1 (S), so that Pi (A) = 1, i = 0, 1, where Pi is the probability measure induced by f i , i = 0, 1. The function g(z) = − log z (z > 0) is strictly convex (since g (z) = z −2 > 0). Then with Z = f 1 (X )/ f 0 (X ), g(Z ) is strictly convex on A (with Pi (A) = 1, i = 0, 1). Thus, g(E0 Z ) ≤ E0 g(Z ) and equality occurs if and only if P0 (Z = c, a constant) = 1. This is so by Exercise 35 in Chapter 6. Equivalently, f 1 (X ) f 1 (X ) f 1 (X ) ≤ E0 − log = −E0 log = K ( f 0 , f 1 ), − log E0 f 0 (X ) f 0 (X ) f 0 (X ) or −K ( f 0 , f 1 ) ≤ log E0

f 1 (X ) = log f 0 (X )

S

f 1 (x) f 0 (x)dν = log f 0 (x)

and −K ( f 0 , f 1 ) = 0 only if P0 From

f 1 (X ) f 0 (X )

f 1 (x)dν = log 1 = 0, S

f 1 (X ) = c = 1. f 0 (X )

= c a.s.[P0 ], we get f 1 (X ) c = E0 = f 0 (X )

S

f 1 (x) f 0 (x)dν = 1, f 0 (x)

so that, first, −K ( f 0 , f 1 ) ≤ 0, equivalently, K ( f 0 , f 1 ) ≥ 0, and secondly, K ( f 0 , f 1 ) = 0 only if P0 [ f 1 (X ) = f 0 (X )] = 1; also, P1 [ f 1 (X ) = f 0 (X )] = 1. The proof is completed. Exercises. 1. Verify the relation An 0 = A +

∞

− A j+1 ) used in the proof of Lemma 1; ∞

An , then for namely, if {An }, n ≥ 1, form a nonincreasing sequence and A = ∞ n=1 any n 0 ≥ 1, the given relation holds. In particular, A1 = n=1 (An − An+1 ) if ∞

An = .

n=1

j=n 0 (A j

2. Given the finite measures μn , n = 1, 2, . . ., on the measurable space (, A), define a probability measure μ such that μn μ for all n (i.e., μ(A) = 0 for A ∈ A implies μn (A) = 0, n ≥ 1).


3. Let C be the class of all probability measures defined on the measurable space (, A), and let d(P, Q) = P − Q = 2 sup{|P(A) − Q(A)|; A ∈ A}. (i) Then show that d is a distance in C; i.e., show that d(P, Q) ≥ 0 and d(P, Q) = 0 only if P = Q, d(P, Q) = d(Q, P), and d(P, Q) ≤ d(P, R) + d(R, Q) where R ∈ C. Next, let μ be a σ -finite measure in A such that P 0) and C = (g − f > 0). 4. Let U (α, β) be the uniform distribution over the interval [α, β] (α < β), and for n ≥ 1, let Pn and Q n be the (probability) measures corresponding to U (− n1 , 1) and U (0, 1 + n1 ). Then show that Pn − Q n → 0 as n → ∞, by using part (ii) of Exercise 3. 5. Let P and Q be two probability measures defined on the measurable space (, A), and suppose that P ≈ Q (i.e., P 0. Hint: For B = ( f − g > 0), show that P − Q = 2[P(B) − Q(B)], and then work with P(B) and Q(B) by introducing C = (|Z | > ε). 6. Let P, Q and f , g be as in Exercise 3 and, without loss of generality, we suppose that the dominating measure μ is a probability measure (e.g., μ = (P + Q)/2). Define ρ by ρ = ( f g)1/2 dμ, and show that: (i) ρ ≤ 1. 2 1/2 (ii) 2(1 − ρ) ≤ d(P, Q) ≤ 2(1 − ρ ) , where (by Exercise 3), d(P, Q) = | f − g|dμ = P − Q. Replacing P, Q and f , g, ρ by Pn , Q n and f n , gn , ρn , n ≥ 1, part (ii) becomes: 2(1 − ρn ) ≤ d(Pn , Q n ) ≤ 2(1 − ρn2 )1/2 .

133

134


(iii) From this last relation, conclude that, as n → ∞, d(Pn , Q n ) → 0 if and only if ρn → 0. 7. For n ≥ 1, let f n be real-valued measurable functions defined on an open set E ⊂ containing 0, and suppose that | f n (x)| ≤ M(< ∞) for all n and all x ∈ E, and that f n (x) −→ 0 for x ∈ E. Then show that, for any sequence xn −→ 0, n→∞ n→∞ there exists a subsequence {xm } ⊆ {xn } and a λ-null subset N of E (which may depend on {xn }) such that f m (x + xm ) −→ 0, x ∈ E − N ; here λ is the Lebesgue m→∞ measure. Hint: Consider the integral | f n (x + xn )|d(x), where is the d.f. of the N (0, 1) distribution; refer to Exercise 17 in Chapter 5, and use the fact that the measure induced by and λ are mutually absolutely continuous. 8. For n ≥ 1, let f n be real-valued measurable functions defined on , and suppose that | f n (x)| ≤ M(< ∞) for all n and x ∈ , and that lim f n− (x) = 0 for x ∈ . n→∞

Then for any sequence {xn } with xn −→ 0 it holds lim sup f n (x + xn ) ≥ 0 a.e. n→∞

[λ], where λ is the Lebesgue measure.

n→∞

CHAPTER

Distribution Functions and Their Basic Properties, Helly–Bray Type Results

8

This chapter deals with some basic properties of d.f.s, the concept of weak and complete convergence of sequences of d.f.s, and some Helly–Bray type theorems. More precisely, in Section 8.1, it is shown that a d.f. is, actually, determined by its values on a dense set in ; that the set of its discontinuity points is countable; and that it is uniquely decomposed into three d.f.s, of which one is a step function, the second is a continuous d.f. whose induced measure is absolutely continuous with respect to the Lebesgue measure, and the third is a continuous d.f. whose induced measure is singular with respect to the Lebesgue measure. In the following section, the concepts of weak and complete convergence of a sequence of d.f.s is introduced, and their relationships are discussed. It is also shown that for any sequence of d.f.s, there is always a subsequence that converges weakly to a d.f. In the final section, the integral of a bounded and continuous real-valued function defined on with respect to a d.f. is considered; such integrals are to be interpreted in the Riemann–Stieltjes sense. Then conditions are given under which weak or complete convergence of a sequence of d.f.s implies convergence of associated integrals. Problems of this type arise often in research and applications.

8.1 Basic Properties of Distribution Functions Definition 1.

We say that F is a d.f. (not necessarily of a r.v.) if F : → and

(1) 0 ≤ F(x) ≤ 1, x ∈ . (2) F ↑. (3) F is continuous from the right. def

def

As usual, F(+∞) = lim F(x), F(−∞) = lim F(x). Then 0 ≤ F(−∞), x→∞

x→−∞

F(∞) ≤ 1, but F(−∞) need not be = 0 and F(+∞) need not be = 1. Proposition 1. Defining property (3) is not essential in defining a d.f. in the following sense: if F ∗ satisfies (1) and (2), then one can construct a d.f. F (i.e., F satisfies properties (1)–(3)) which coincides with F ∗ whenever F ∗ is continuous or only right continuous. An Introduction to Measure-Theoretic Probability, Second Edition. http://dx.doi.org/10.1016/B978-0-12-800042-7.00008-6 Copyright © 2014 Elsevier Inc. All rights reserved.

135

136

CHAPTER 8 Distribution Functions and Their Basic Properties

Proof. In the first place, F ∗ can have only jumps. Now if x ∈ C(F ∗ ), we set / C(F ∗ ), we set F(x) = F ∗ (x + 0) which exists F(x) = F ∗ (x), whereas if x ∈ because of (2). Then, clearly, F satisfies properties (1)–(3). Actually more is true, namely, Proposition 2. Let D be a dense subset of (such as the set of rational numbers), and let FD : D → satisfy (1) and (2). Then FD determines uniquely a d.f. F in . Proof. In the first place, we may assume that FD is also continuous from the right in D, by Proposition 1. Next, for x ∈ , let xn ∈ D such that xn ↓ x as n → ∞. Then define F : → [0, 1] as follows: F(x) = FD (x), x ∈ D, and F(x) = lim FD (xn ), x ∈ D c . Clearly, F satisfies (1)–(3). n→∞

If F1 , F2 are two d.f.s such that F1 (x) = F2 (x), x ∈ D, then F1 ≡ F2 .

Corollary.

Proof. Let x ∈ D c . Then there exists xn ∈ D such that xn ↓ x as n → ∞. It follows that, as n → ∞, F1 (xn ) ↓ F1 (x), F2 (xn ) ↓ F2 (x). But F1 (xn ) = F2 (xn ), n ≥ 1. Hence F1 (x) = F2 (x). We mentioned previously that a d.f. can have only discontinuities that are jumps. The following theorem refers to the number of these jumps. Theorem 1. Any d.f. F has a countable number of discontinuous points (which, of course, can be equal to 0). Proof. If F is continuous everywhere there is no problem. Let then F be discontinuous and let (α, β] be a finite interval in . Let x1 , . . . , xn be n points such that α < x1 < · · · < xn ≤ β at which F is discontinuous. ( α

]( x1

]( x2

] x3

...

...

]( xn

] β

Then we have F(α) ≤ F(x1− ) < F(x1 ) ≤ F(x2− ) < F(x2 ) ≤ · · · ≤ F(xn − ) < F(xn ) ≤ F(β). The lengths of the jumps are F(x j ) − F(x j − ),

j = 1, . . . , n.

Summing them up, we get n [F(x j ) − F(x j − )] = [F(x1 ) − F(x1− )] + [F(x2 ) − F(x2− )] j=1

+ · · · + [F(xn−1 ) − F(xn−1− )] + [F(xn ) − F(xn − )] ≤ [F(x1 ) − F(x1− )] + [F(x2− ) − F(x1 )] + [F(x2 ) − F(x2− )] + · · · + [F(xn − ) − F(xn−1 )] + [F(xn ) − F(xn − )]

8.1 Basic Properties of Distribution Functions

(by adding some nonnegative numbers), = F(xn ) − F(x1− ) ≤ F(β) − F(α); i.e., n [F(x j ) − F(x j − )] ≤ F(β) − F(α).

(8.1)

j=1

Relation (8.1) implies the following: for ε > 0, the number of jumps in (α, β] with length of jump > ε cannot be greater than 1ε [F(β) − F(α)]. In fact, if this number is K , and K > 1ε [F(β) − F(α)], then the sum of lengths of the K jumps is ≥ K ε > F(β) − F(α), which contradicts (8.1). Applying this for m = 1, 2, . . . , we get the number of jumps in (α, β] with length of jump > m1 is ≤ m[F(b) − F(a)], thus finite. Hence the total number of jumps in (α, β] is countable. Since can be written as the sum of denumerably many intervals, we have that the total number of jumps in is countable. Definition 2. Consider the numbers xn , n = 1,2, . . . , and let the positive numbers p(xn ) be associated with xn ,n ≥ 1, such that n p(xn ) < ∞. Define F ∗ : → [0, ∞) as follows: F ∗ (x) = xn ≤x p(xn ). Then F ∗ is said to be a step function. Remark 1. F ∗ as just defined is not necessarily a step function in the usual sense; i.e., constant over intervals, since the xn s can form a set dense in ; e.g., the rational numbers. However, if {xn , n ≥ 1} is not dense in , these two concepts coincide. Theorem 2 (Decomposition Theorem). Any d.f. F is uniquely decomposed into two d.f.s Fc , Fd such that Fc is continuous, Fd is a step function, and F = Fc + Fd . Proof. Let xn , n ≥ 1, be the discontinuity points of F and set p(xn ) = F(xn ) − F(xn −) and p(x) = 0 for x = xn , n ≥ 1. Then, clearly, n p(xn ) < ∞(actually, ≤ 1, by (1) in Definition 1). Define: p(xn ), Fc (x) = F(x) − Fd (x). Fd (x) = xn ≤x

We show first that Fd is a d.f. In the first place, 0 ≤ Fd (x) ≤ 1, x ∈ (apply (8.1) for β = x and let α → −∞). Next, for y > x, we get p(xn ) ≥ 0, Fd (y) − Fd (x) = x<xn ≤y

so that Fd ↑. So it remains to show continuity from the right. We have, by (8.1), p(xn ) ≤ F(y) − F(x). (8.2) (0 ≤)Fd (y) − Fd (x) = x<xn ≤y

Letting y ↓ x and utilizing the right continuity of F, we get Fd (y) − Fd (x) → 0 or Fd (y) → Fd (x).

137

138


We next prove that Fc is a continuous d.f. Since F, Fd are both d.f.s and Fd (x) ≤ F(x) (by (8.1)), x ∈ , we have that 0 ≤ Fc (x) ≤ 1 (by (8.1)), x ∈ . Next, Fc is ↑. In fact, for x < y, we have, by (8.2), p(xn ) ≤ F(y) − F(x), by (8.1). Fd (y) − Fd (x) = x<xn ≤y

Hence Thus

F(y) − F(x) − Fd (y) − Fd (x) ≥ 0.

Fc (y) − Fc (x) = F(y) − Fd (y) − F(x) − Fd (x) = F(y) − F(x) − Fd (y) − Fd (x) ≥ 0,

so that Fc ↑. Now Fc is right continuous since both F, Fd are so. Thus it remains to prove that Fc is continuous from the left. Again for x < y, we have Fc (y) − Fc (x) = F(y) − Fd (y) − F(x) − Fd (x) = F(y) − F(x) − p(xn ) x<xn ≤y

= F(y) − F(x) − p(y) −

p(xn )

x<xn 0 and all sufficiently large n, −ε < Fn (−∞) − Fn (∞) + F(∞) − F(−∞) , or Fn (−∞) − F(−∞) + F(∞) + ε > Fn (∞). Hence as n → ∞, lim sup Fn (−∞) − F(−∞) + F(∞) + ε ≥ lim sup Fn (∞) ≥ lim inf Fn (∞), or F(∞) + ε ≥ lim sup Fn (∞) ≥ lim inf Fn (∞) ≥ F(∞), since by part (i), lim sup Fn (−∞) − F(−∞) ≤ 0 and lim inf Fn (∞) ≥ F(∞). Letting ε → 0, we obtain lim inf Fn (∞) = lim sup Fn (∞) = F(∞). The proof of the other direction is immediate. Theorem 4.

Fn ⇒ F if and only if Fn (x) → F(x) on a set D dense in . n→∞

n→∞

Proof. In all that follows, n → ∞ unless otherwise noted. If Fn ⇒ F, then Fn (x) → F(x), x ∈ C(F) which is a set dense in . Now let Fn (x) → F(x), x ∈ D. We show that this convergence is true for x ∈ C(F). For x ∈ C(F), there exists x , x

∈ D such that x < x < x

. Hence Fn (x ) ≤ Fn (x) ≤ Fn (x

). Letting n → ∞, we get F(x ) ≤ lim inf Fn (x) ≤ lim sup Fn (x) ≤ F(x

). Now let x ↑ x, x

↓ x through values of D. Then we get F(x−) = F(x) ≤ lim inf Fn (x) ≤ lim sup Fn (x) ≤ F(x+) = F(x). Thus Fn (x) → F(x).

Remark 3. It is to be emphasized that in the proof of Theorem 4 only continuity points of F are relevant; discontinuity points and right-continuity points play no role in the proof.

143

144


Now let {Fn } be any sequence of d.f.s. Of course, this sequence may or may not converge weakly. However, there is always a subsequence which converges weakly; i.e., Theorem 5 (Weak Compactness Theorem). If {Fn } is a sequence of d.f.s, there exists a subsequence that converges weakly to a d.f. Proof. Let D = {x1 , x2 , . . .} be a set dense in , and look at the bounded sequence {Fn (x1 )}, n ≥ 1. Then there exists a convergent subsequence; call it {Fn1 (x1 )}. Here, and in the remainder of this proof, all limits are taken as n → ∞. Next, look at the subsequence {Fn1 } (of {Fn }) and evaluate it at x2 to get the bounded sequence {Fn1 (x2 )}. Then there exists a convergent subsequence; call it {Fn2 (x2 )}. Since {Fn2 } ⊆ {Fn1 }, it follows that {Fn2 (x1 )} is also convergent. Proceeding like this, at the nth step we consider the subsequence {F1,n−1 , F2,n−1 , F3,n−1 , . . .} and evaluate it at xn to get the bounded sequence {F1,n−1 (xn ), F2,n−1 (xn ), F3,n−1 (xn ), . . .}. Then there exists a convergent subsequence; call it {F1n (xn ), F2n (xn ), F3n (xn ), . . .}. Since {F1n , F2n , F3n , . . .} ⊆ {F1,n−1 , F2,n−1 , F3,n−1 , . . .}, it follows that {F1n (x j ), F2n (x j ), F3n (x j ), . . .} is also convergent for j = 1, . . . , n − 1. The process continues in this manner indefinitely. In terms of an array, we have that F11 F12 . . F1n . .

F21 F22 . . F2n . .

F31 F32 . . F3n . .

··· ··· . . ··· . .

converges at converges at . . . . . converges at . . . . .

x1 x1 , x2 x1 , x2 , . . . , xn . .

At this point, consider the diagonal sequence {Fnn } and argue that {Fnn } converges at every point of D. To this end, let xn ∈ D and look at {Fnn (xn )}. Clearly, {Fnn } ⊆ {F1n , F2n , F3n , . . .} except for the first n − 1 terms F11 , F22 , . . . , Fn−1,n−1 . Hence {Fnn (xn )} converges, since so does {F1n (xn ), F2n (xn ), F3n (xn ), . . .}. Thus, {Fnn } converges on D and defines the function FD (x) = lim Fnn (x), x ∈ D. Clearly, FD satisfies the properties of being between 0 and 1 and nondecreasing, since 0 ≤ Fnn (x) ≤ 1, x ∈ , and Fnn is nondecreasing. Next, extend FD to F ∗ from D to by defining F ∗ thus: F ∗ (x) = FD (x), x ∈ D, and F ∗ (x) = limn→∞ FD (xn ) with xn ∈ D, xn ↓ x. Then, clearly, 0 ≤ F ∗ (x) ≤ 1, x ∈ , and F ∗ is nondecreasing. Furthermore, Fnn (x) −→ F ∗ (x), x ∈ C(F ∗ ). Indeed, for x ∈ C(F ∗ ), there exist x n→∞

and x

in D such that x < x < x

, so that Fnn (x ) ≤ Fnn (x) ≤ Fnn (x

), and this implies as n → ∞, lim Fnn (x ) = FD (x ) = F ∗ (x ) ≤ lim inf Fnn (x) ≤ lim sup Fnn (x) ≤ lim Fnn (x

) = FD (x

) = F ∗ (x

) (since F ∗ = FD on D). Thus F ∗ (x ) ≤ lim inf Fnn (x) ≤ lim sup Fnn (x) ≤ F ∗ (x

).

8.3 Helly–Bray Type Theorems for Distribution Functions

Next, let x ↑ x and x

↓ x whereas always x , x

∈ D. Then lim F ∗ (x ) = F ∗ (x−) = F ∗ (x) ≤ lim inf Fnn (x) ≤ lim sup Fnn (x) ≤ lim F ∗ (x

) = F ∗ (x+) = F ∗ (x) (since x ∈ C(F ∗ )). Therefore, as n → ∞, lim inf Fnn (x) = lim sup Fnn (x) = lim Fnn (x) = F ∗ (x); i.e., Fnn (x) −→ F ∗ (x) for x ∈ C(F ∗ ), as asserted. Finally, modify F ∗ into F as n→∞

follows: F(x) = F ∗ (x), x ∈ C(F ∗ ), and for x ∈ C(F ∗ ), F(x) = lim F ∗ (xn ) where xn ∈ C(F ∗ ) with xn ↓ x as n → ∞, so that F inherits the set of continuity points of F ∗ ; i.e., C(F) = C(F ∗ ), and is also right-continuous. Therefore F is a d.f. and Fnn ⇒ F, as was to be seen. Remark 4. The method employed in the proof for constructing the subsequence {Fnn } is known as the Cantor diagonal method.

8.3 Helly–Bray Type Theorems for Distribution Functions Remark 5. The following integrals are to be understood in the Riemann–Stieltjes sense. For some comments, regrading such integrals and related results, the reader is referred to Appendix B. As a prelude to this section, let us revisit briefly Chapter 5. In that chapter, it was seen that, if a sequence of r.v.s X n , n ≥ 1, converges to a r.v. X (as n → ∞), either a.e. (as is the case in Theorem 1 and its corollaries, Theorem 2(iii) and Theorem 3 under (a)), or in measure (as is thecase in Theorem 3 under (b)), then the integrals X n dμ converge to the integral X dμ, provided some additional requirements are met. Now, let Fn and F be the d.f.s of X n and X , respectively, and suppose that Fn ⇒ F c or Fn −→ F as n → ∞. Also, let g : → be a Borel function, sothat g(X n ) and g(X ) are r.v.s. Then, by Theorem 13 of Chapter 4, g(X n )dP = g(x)dP X n = g(x)d F , and g(X )dP = g(x)dP = n X g(x)d F (see also Remarks 6 of this chapter). A question which may then arise is this: under what further conditions is it true that g(x)d Fn −→ g(x)dF, over a set B ⊆ ? B

n→∞ B

It is this kind of questions for which the Helly–Bray type results provide an answer (see Theorems 6,7, and 8). Theorem 6 (Helly–Bray Lemma).

Let Fn ⇒ F. Then for every α, β, n→∞

−∞ < α < β < ∞, such that Fn (α) → F(α), n→∞

Fn (β) → F(β) n→∞

145

146


and every g : [α, β] → continuous (and hence bounded), we have gdFn → gdF. (α,β]

n→∞ (α,β]

Proof. Consider the partition α = x1 < x2 < · · · < xm+1 = β of [α, β] and on (α, β] define the function gm (x) =

m

g(ξ j )I(x j ,x j+1 ] (x),

j=1

where ξ j ∈ (x j , x j+1 ], j = 1, . . . , m. Then we assert that, as m → ∞, so that max x j+1 − x j → 0, we have gm (x) → g(x), x ∈ (α, β]. In fact, let x ∈ (α, β].

1≤ j≤m

Then x ∈ (xk , xk+1 ] for some k, while also ξk ∈ (xk , xk+1 ]. Now, as m → ∞,

max x j+1 − x j → 0 implies xk+1 − xk → 0 so that ξk → x 1≤ j≤m

and hence g(ξk ) → g(x) by continuity of g. But g(ξk ) = gm (x). Thus gm (x) → g(x).

m→∞

Next,

g dFn − g dF ≤ (g − gm )dFn

(α,β] (α,β] (α,β]

+ (gm − g)dF + gm dF − gm dFn

, (α,β]

(α,β]

(α,β]

clearly, while all these integrals exist either because the integrand is continuous or because it is a step (in the usual sense) function. Then the right-hand side of the relation above is bounded by

2 sup |gm (x) − g(x)| + gm dF − gm dFn

, (α,β]

x∈(α,β]

(α,β]

since F(β) − F(α) ≤ 1,

Fn (β) − Fn (α) ≤ 1.

Next, (α,β]

gm dF =

m j=1 (x j ,x j+1 ]

and similarly, m gm dFn = (α,β]

j=1 (x j ,x j+1 ]

gm dF =

m

g(ξ j ) F(x j+1 ) − F(x j ) ,

j=1

gm dFn =

m j=1

g(ξ j ) Fn (x j+1 ) − Fn (x j ) .


Hence

(α,β]

g dFn −

(α,β]

m

g(ξ j ) g dF

≤ 2 sup |gm (x) − g(x)| + x∈(α,β]

j=1

× Fn (x j+1 ) − F(x j+1 ) + Fn (x j ) − F(x j ) .

Taking the partitioning points x2 . . . , xm in C(F), using weak convergence and the assumption that Fn (α) → F(α) and Fn (β) → F(β), we have Fn (x j ) → F(x j ), n→∞ j = 1, . . . , m. Thus, the second summand at the right-hand side of the last inequality → 0. Next, n→∞ sup |gm (x) − g(x)| = max sup |gm (x) − g(x)| ; x ∈ (x j , x j+1 ] 1≤ j≤m

x∈(α,β]

sup g(ξ j ) − g(x) ; x ∈ (x j , x j+1 ] 1≤ j≤m

≤ max sup g(ξ j ) − g(x) ; x ∈ [x j , x j+1 ] 1≤ j≤m = max |g(ξ j ) − g(y j )|; some y j ∈ [x j , x j+1 ]

= max

1≤ j≤m

by continuity of g, and this is equal to |g(ξk ) − g(yk )| for some k = 1, . . . , m. As m → ∞, this last expression converges to

zero by the uniform continuity of g in [α, β] and the fact that max x j+1 − x j → 0 implies |ξk − yk | → 0. Thus, as 1≤ j≤m

n → ∞,

(α,β]

g dFn →

(α,β]

g dF.

Let Fn ⇒ F. Then, for every g :

Theorem 7 (Helly–Bray Extended Lemma).

n→∞

→ continuous and such that g(±∞) = 0 (in the sense that lim g(x) = 0), x→±∞ we have g dFn → g dF. n→∞ Proof. Since g is bounded and continuous on , g dFn , g dF exist (finite) and are taken as follows: g dFn = lim g dFn , g dF = lim g dF.

α→−∞ (α,β] β→∞

α→−∞ (α,β] β→∞

Next, for any −∞ < α < β < ∞, we have

g dFn −

≤

g dF g dF − g dF n

(α,β] (α,β]

+

g dF − g dF

(α,β]

+ g dFn − g dFn

.

(α,β]

(8.3)

147

148


In the sequel, α and β will be continuity points of F, and α and β will be taken as small as needed and as large as needed, respectively. Then Fn (α) → F(α), Fn (β) → n→∞

n→∞

F(β), and hence, by Theorem 6,

ε

g dFn − g dF

< for all sufficiently large n.

3 (α,β] (α,β] Also,

ε g dF

< . 3 (α,β]

g dFn

=

g dFn

≤

g dF −

Finally,

g dFn −

(α,β]

−(α,β]

(8.4)

(8.5)

sup |g(x)|
0. Remark 7. In closing, it should be mentioned that the results discussed in this chapter, pertaining to weak convergence of d.f.s, are very special cases of the subject matter of weak convergence of probability measures. For an exposition of such material, see, e.g., Billingsley (1999). Exercises. 1. Let F : → [0, 1] be nondecreasing, right continuous, F(−∞) = 0 and F(∞) = 1 (i.e., a d.f. of a r.v.), and let F −1 be defined by F −1 (y) = inf{x ∈ ; F(x) ≥ y}, y ∈ [0, 1]. Next, consider the probability space (, A, P) = ([0, 1], B[0,1] , P), where P = λ is the Lebesgue measure, and on , define the function X by X (ω) = F −1 (ω). Then show that X is a r.v. and that its d.f. is F. Hint: Show that F −1 (y) ≤ t if and only if y ≤ F(t), t ∈ .

149

150


P

2. (i) If X n → X , then show that FX n ⇒ FX . n→∞

(ii)

3. 4. 5. 6.

n→∞

By an example, show that the converse statement in part (i) need not be true.

Hint: For part (i), show that, for every ε > 0 and any x ∈ , FX n (x) ≤ P(|X n − X | ≥ ε) + FX (x + ε), and FX (x − ε) ≤ P(|X n − X | ≥ ε) + FX n (x). Next, let x ∈ C(FX ) and take the limits first as n → ∞ and then as ε → 0 to obtain the desired result. Let Fn , n ≥ 1, be d.f.s of r.v.s and suppose that Fn ⇒ F where F is a d.f. with n→∞ V ar F = 1. Then show that F is the d.f. of a r.v. Show that the Weak Compactness Theorem (Theorem 5) still holds, if condition 1 in Definition 1 is replaced by: (0 ≤)F(x) ≤ B, x ∈ , n ≥ 1, for some B < ∞. Show that Theorems 6–8 hold true if, for all x ∈ , 0 ≤ F(x) ≤ B and 0 ≤ Fn (x) ≤ B, n = 1, 2, . . ., for some B > 0, as indicated in Remark 6. For n = 1, 2, . . ., consider the r.v.s X n , X and Y such that |X n | ≤ Y , EY r < ∞ d

(for some r > 0), and X n −→ X . Then show that E|X n |r −→ E|X |r . n→∞

n→∞

7. Let X be a r.v. with d.f. F; i.e., F(x) = P(X ≤ x), x ∈ . Then show that (i) P(X < x) = F(x−) (the left-hand side limit). (ii) F(x) is continuous at x if and only if P(X = x) = 0. 8. In the following expression, determine the constants α and β, so that the function F defined is the d.f. of a r.v. F(x) = 0, x ≤ 0, and F(x) = α + βe−x

2 /2

, x > 0. c

9. For n = 1, 2, . . ., let Fn and F be d.f.s such that Fn −→ F, and let F be n→∞

continuous. Then show that Fn (x) −→ F(x) uniformly in x ∈ . n→∞ Hint: For ε > 0, choose a and b sufficiently small and sufficiently large, respectively, so that F(a) < 3ε , F(∞) − F(b) < 3ε . Next, partition [a, b] by a = x0 < x1 < · · · < xk−1 < xk = b, so that F(x j ) − F(x j−1 ) < 3ε , j = 1, . . . , k. Finally, by taking x ∈ [a, b], or x < a, or x > b, show that |Fn (x) − F(x)| < ε for all x, provided n ≥ N some integer independent of x ∈ . 10. Let X be a r.v. with continuous d.f. FX , and define the r.v. Y as follows: Y = X if X ≤ C, and Y = C if X > C, where C is a constant. Then (i) Determine the d.f. of the r.v. Y , FY . (ii) Show that FY = F1 + F2 , where F1 is a continuous d.f. and F2 is a step function. 11. By a simple example demonstrate that it is possible that Fn , n ≥ 1, are d.f.s of r.v.s with Fn ⇒ F, a d.f., but not that of a r.v. n→∞

12. For n ≥ 1, let X n and X be r.v.s defined on the probability space (, A, P), and d

suppose that, as n → ∞, E|X n | → E|X | < ∞ and X n → X . Then show that |X n |, n ≥ 1, are uniformly integrable.


13. For n ≥ 1, let Fn and F be d.f.s of r.v.s, let f n be real-valued measurable functions defined on , let f : → be continuous, and let g : → [0, ∞) be continuous. Assume that: (i) Fn → F as n → ∞. (ii) | f n (x)| ≤ g(x), x ∈ , n ≥ 1. (iii) f n (x) −→ f (x) uniformly on finite intervals. n→∞ (iv) g dFn −→ g dF and g dF < ∞. n→∞ Then show that f n dFn −→ f dF and f dF is finite. n→∞ Hint: Use the Helly–Bray Lemma (Theorem 6 in this chapter).

151

CHAPTER

Conditional Expectation and Conditional Probability, and Related Properties and Results

9

This chapter is primarily about conditional expectations and conditional probabilities defined in terms of σ -fields. The definitions are based on Theorems 2 and 3 of Chapter 7. In Section 9.1, the concept of the conditional expectation, given a σ -field, is defined, and then the conditional probability is derived as a special case. In the following section, some basic properties of conditional expectations and of conditional probabilities are discussed. In Section 9.3, versions for conditional expectations of standard convergence theorems proved in Chapter 5 are established. The same is done for the moment inequalities discussed in Chapter 6. Some additional properties of conditional expectations and of conditional probabilities are presented in the fourth section of the chapter. The section is concluded with an application, illustrating that the elementary definition of a conditional probability density function coincides with the more general definition used here. In the course of derivations in Section 9.4, the concept of independence of r.v.s is needed, as well as a result regarding the expectations of the product of two independent r.v.s (see Lemma 1). An elaboration on independence and the proof of Lemma 1 are found in Chapter 10.

9.1 Definition of Conditional Expectation and Conditional Probability Consider the probability space (, A, P) and let X be a r.v. defined on this space with E |X | < ∞. Let B be a σ -field with B ⊆ A and denote by PB the restriction of P to B. On A, define the set functions ϕ + and ϕ − as follows: + + − ϕ (A) = X dP, ϕ (A) = X − dP, A ∈ A. A +

A

−

Denoting by ϕB and ϕB the restrictions of ϕ + , ϕ − to B, we clearly have + − (B) = X + dP, ϕB (B) = X − dP, B ∈ B. ϕB B

B


153

154

CHAPTER 9 Conditional Expectation and Conditional Probability

+ − Then, clearly, ϕB , ϕB are PB -continuous (finite) measures (see also Exercise 1) and therefore, by Theorem 3 in Chapter 7, there exists a.s. [PB ] well defined, ≥ 0, B-measurable (a.s. [PB ] finite) r.v.s Y + , Y − such that + − ϕB (B) = Y + dPB , ϕB (B) = Y − dPB , B ∈ B. B

B

Then, by setting Y = Y + − Y − , we have that Y is a.s. [PB ] well defined, B-measurable, a.s. [PB ] finite, and Y dPB = X dP, B ∈ B. B

B

(However, see also Proposition 1 in Chapter 7.) Definition 1. Given an integrable r.v. X defined on the probability space (, A, P), and a σ -field B ⊆ A, the conditional expectation of X , given B, is any B-measurable r.v. Y , denoted by E B X (or E(X | B)) such that B E X dPB = X dP, B ∈ B. (9.1) B

B

EB I

By taking X = I A , A ∈ A, then A is called conditional probability of A, given B, is denoted by P B A, and we have B E I A dPB = P B A dPB B B = I A dP = P(A ∩ B), B ∈ B. (9.2) B

Special Cases: For B ∈ A with P(B) > 0 and a r.v. X with E |X | < ∞, define the r.v. E B X as follows: 1 (E B X )(ω) = X dP, ω ∈ . P(B) B Now let {B j , j = 1, 2, . . .} be a (measurable) partition of with P(B j ) > 0, all j, and let B be the σ -field generated by this partition. Define the function Y on as follows: ∞ 1 Y = (E B j X )I B j , E B j X = X dP, j ≥ 1. P(B j ) B j j=1 Then, clearly, Y is B-measurable and for any B ∈ B, one has B = i∈I Bi for some I ⊆ {1, 2, . . .}. Therefore, by Exercise 1, ⎡ ⎤ ⎣ (E B j X )I B j ⎦dPB Y dPB = Y dPB = Y dPB = i∈I Bi

B

=

i∈I

i∈I

Bi

i∈I

Bi

j

P(Bi ) (E Bi X )PB (Bi ) = (E Bi X )P(Bi ) = X dP P(Bi ) Bi i∈I

i∈I

9.1 Definition of Conditional Expectation and Conditional Probability

=

X dP = Bi

i∈I

i∈I Bi

i.e., B

Y dPB =

B

X dP =

X dP = B

E B X dPB ,

B

E B X dPB ;

B ∈ B,

and therefore Y = E B X a.s. (by Proposition 1 in Chapter 7). That is, in the case that B is generated by a countable (measurable) partition {B j , j ≥ 1} with P(B j ) > 0 1 for all j, then for any integrable r.v. X , one has that (E B X )(ω) = P(B X dP, for j) Bj a.a. (almost all) ω ∈ B j , j ≥ 1. In particular, if X = I A , A ∈ A, then

B P(A ∩ B j ) P A (ω) = , a.a. ω ∈ B j , P(B j )

j ≥ 1.

(9.3)

If the partition of consists of two sets B, B c ∈ A with 0 < P(B) < 1, then the σ -field generated by {B, B c } is B = {, B, B c , } and P(A∩B) a.a. ω ∈ B

B P(B) P A (ω) = P(A∩B c) a.a. ω ∈ B c . P(B c ) Thus for a.a. ω ∈ B, (P B A)(ω) = P(A | B), and for a.a. ω ∈ B c , (P B A)(ω) = P(A | B c ). Now suppose that the σ -field B ⊆ A is generated, as before, by a countable partition {B j , j ≥ 1} with P(B j ) > 0 for all j, and that for A ∈ A, the conditional probability of A, given B, is defined by (9.3). In this context, it will be convenient to write PωB A rather than (P B A)(ω). Thus, we have PωB A =

P(A ∩ B j ) , a.a. ω ∈ B j , P(B j )

j ≥ 1.

(9.4)

From (9.4), it is obvious that for every fixed ω ∈ (lying outside a null set), PωB as a function on A is a probability measure. Then, if X is a r.v., we may talk about the integral of X with respect to PωB , provided, of course, this integral exists. To this end, let X be a P-integrable r.v. Then for an arbitrary but fixed ω ∈ (lying outside a null set), we have that ω ∈ B j for exactly one j. Therefore, by Exercise 2, 1 B X dP = E B j X , ω ∈ B j , j ≥ 1, X (ω )dPω = P(B j ) B j and hence = (E B j X )I B j = EωB X , ω ∈ (lying outside a null set). (9.5) X (ω )dPB ω j

This result then shows that if B is generated as before, one may define conditional probability first and then get the conditional expectation as the integral of a P-integrable r.v. X with respect to the conditional probability.

155

156


Remark 1. The result just stated need not be true if B is not of the form just mentioned. (See, e.g., Loève (1963), pages 353–354.) This section is concluded with two examples illustrating some concepts discussed here. Remark 2. Here is a possible explanation of the meaning/usefulness of the conditional probability and conditional expectation in their abstract setting. From the definition of the conditional probability, we have, for A ∈ A, P B A dPB = I A dP = P(A ∩ B), B ∈ B, B

so that

B

1 P(B)

B

P(A ∩ B) = P(A|B) P(B)

P B A dPB =

Now, on the right-hand side in the last relation above, we have the conditional probability of A, given B, as is given by the familiar elementary definition. The left-hand side in the sane relation tells us that the actual calculation of P(A|B) is obtained through the integration of the (B-measurable) r.v. P B A over B, and norming it by the P(B). Next, from the definition of the conditional expectation of a r.v. X (E X ∈ ), we have, E B X dPB = X dP, B ∈ B, B

so that

1 P(B)

B

E B X dPB =

B

1 P(B)

X dP = B

X dP(·|B),

where P(·|B) is the elementary definition of the conditional probability, given B (see also Exercise 2). So, E B X has the property that it is the (B-measurable) r.v. whose weighted (by P(B)) integral over B gives the expectation of X with respect to the conditional probability measure P(·|B). Example 1. Let X 1 , . . . , X n be i.i.d. r.v.s defined on the probability space (, A, Pθ ), where the parameter θ takes values in the parameter space , an open subset of k , k ≥ 1, and set X = (X 1 , . . . , X n ). Also, let T = (T1 , . . . , Tm ) be an m-dimensional statistic of X; i.e., a measurable function of X not involving any unknown quantities. We say that T is sufficient for θ , if the conditional distribution of X, given the σ -field induced by T, BT , does not depend on θ . That is, if AX is the σ -field induced by X (so that AX ⊆ A and BT ⊆ AX since T = T(X)), then EθBT I A = PθBT A is independent of θ for all A ∈ AX . As an application of this concept,let X 1 , . . . , X n be i.i.d. r.v.s distributed as B(1, θ ), θ ∈ = (0, 1), and set T = nj=1 X j . Then T is sufficient for θ . To this end, set X = (X 1 , . . . , X n ) and observe that X takes on 2n values in n , call them xi = (xi1 , . . . , xin ), i = 1, . . . , 2n . Thus, if Ai = X−1 ({xi }), then AX is generated by {Ai , i = 1, . . . , 2n }. Next, T ∼ B(n, θ ) andBT is generated m by {Bt , t = 0, 1, . . . , n} where Bt = T −1 ({t}) = X−1 (Ct ) = j=1 Ai j , where Ct = {xi j = (xi j 1 , . . . , xi j n ) ∈ n , each coordinate being either 0 or 1 and their

9.1 Definition of Conditional Expectation and Conditional Probability

sum being equal to t, j = 1, . . . , m = nt }, since there are nt ways of choosing t positions out of n to place t 1s and 0s in the remaining n − t positions; if t = 0, then

n = 1, and this is the number of ways of placing 0s in all n places (and 1s nowhere). 0 With A ∈ AX , apply the special case following Definition 1 to get n Pθ (A ∩ Bt ) , t = 0, 1, . . . , n. (E Bt,θ I A )I Bt , E Bt,θ I A = Pθ (Bt ) t=0 p Clearly, for each A ∈ AX , A = k=1 A jk , for some p, so that ⎞

p ⎛ m q ⎠ ⎝ A ∩ Bt = A jk ∩ Ai j = Akl , for some q,

PθBT A = EθBT I A =

k=1

j=1

l=1

and Pθ (Akl ) = Pθ (X 1 = xkl 1 , . . . , X n = xkl n ) = Pθ (X 1 = xkl 1 ) . . . Pθ (X 1 = xkl n ) = θ xkl 1 (1 − θ )1−xkl 1 . . . θ xkl n (1 − θ )1−xkl n = θ t (1 − θ )n−t .

It follows that Pθ (A ∩ Bt ) = qθ t (1 − θ )n−t . Since Pθ (T = t) = nt θ t (1 − θ )n−t , it follows that qθ t (1 − θ )n−t Pθ (A ∩ Bt ) q = n t = n E Bt,θ I A = n−t Pθ (Bt ) θ (1 − θ ) t t independent of θ ; that is, E Bt,θ I A is independent of θ , and then so is PθBT I A . It follows that T is sufficient for θ . Example 2. Refer to the previous example, and suppose that the independent r.v.s θ ; i.e., X j ∼ P(θ ), θ ∈ = X 1 , . . . , X n are distributed as Poisson with parameter (0, ∞), j = 1, . . . , n. Then the statistic T = nj=1 X j is sufficient for θ . As in Example 1, X = (X 1 , . . . , X n ) takes values xi = (xi1 , . . . , xin ) in n where each xi j = 0, 1, . . . , j = 1, . . . , n. Thus, if Ai = X−1 ({xi }), then AX is and BT is generated by {B generated by {Ai , i = 1, 2, . . .}. Next, T ∼ P(nθ ) t, t = m A , where C = xi j = 0, 1, . . .} where Bt = T −1 ({t}) = X −1 (Ct ) = i t j j=1 (xi j 1 , . . . , xi j n ) ∈ n ; each coordinate being ≥ 0 integer and their sum being equal

to t, j = 1, . . . , m = n+t−1 . That m = n+t−1 follows from the fact that there are t t

n+t−1 ways of selecting t positions out of n to place xi j 1 , . . . , xi j n ≥ 0 integers with t xi j 1 + · · · + xi j n = t (see also Exercise 19). For A ∈ AX , we have, as in Example 1, PθBT A = EθBT I A =

∞

(E Bt ,θ I A )I Bt , E Bt ,θ I A =

t=0

Furthermore, again as in Example 1, A ∩ Bt =

q l=1

Pθ (A ∩ Bt ) , t = 0, 1, . . . . Pθ (Bt ) Akl for some q, and

Pθ (Akl ) = Pθ (X 1 = xkl 1 , . . . , X n = xkl n )

157

158


θ xkl 1 θ xkl n . . . e−θ x kl 1 ! x kl n ! t θ = e−nθ , x kl 1 ! . . . x kl n !

= e−θ

so that Pθ (A ∩ Bt ) = qe−nθ θ t /xkl 1 ! . . . xkl n !. Since Pθ (T = t) = e−nθ (nθ )t /t!, it follows that Pθ (A ∩ Bt ) qe−nθ θ t /xkl 1 ! . . . xkl n ! = Pθ (Bt ) e−nθ (nθ )t /t! = qt!/n t xkl 1 ! . . . xkl n !

E Bt,θ I Ai =

independent of θ . Then so is PθBT I A , and T is sufficient for θ .

9.2 Some Basic Theorems About Conditional Expectations and Conditional Probabilities In all that follows the σ -field B ⊆ A. Also, Proposition 1 in Chapter 7 will be used throughout without explicit reference to it. Theorem 1.

Let E |X | < ∞. Then

(i) E(E B X ) = E X . (ii) E A X = X a.s. (iii) If X is B-measurable then E B X = X a.s.

Proof. (i) From the definition of E B X , one has E B X dPB = X dP, B

B ∈ B.

B

Then for B = , we get B B E X dPB = E X dP = X dP or E(E B X ) = E X . (ii) Since both X and E A X are A-measurable and E A X dPA = X dPA , A

we have that E A X = X a.s.

A

A ∈ A,

9.2 Some Basic Theorems About Conditional Expectations

(iii) As in (ii), both E B X and X are B-measurable and B E X dPB = X dPB , B ∈ B, B

so that

EB X

B

= X a.s.

Theorem 2. (i) If X = c a.s., then E B X = c a.s. (ii) If X ≥ Y a.s. and E |X | , E |Y | < ∞, then E B X ≥ E B Y a.s. (iii) If for j = 1, . . . , n, X j are integrable and c j are constants, one has ⎛ ⎞ n n cj X j⎠ = c j E B X j a.s. EB ⎝ j=1

j=1

(iv) P B = 1 a.s., P B () = 0 a.s., P B A ≥ 0 a.s., E B ( n B j=1 c j P A j a.s.

n

j=1 c j I A j )

=

Proof. (i) For every B ∈ B, we have B E X dPB = X dP = c dP = c dPB (since X = c a.s.). B

B

That is,

B

B

B

E X dPB =

B

B

c dPB for every B ∈ B, and both E B X

and c are B-measurable. Hence E B X = c a.s. (ii) We have B E X dPB = X dP B B ≥ Y dP (since X ≥ Y a.s.) B

EBY

and this equals B dPB , B ∈ B. Since both E B X and E B Y are B-measurable, we obtain that E B X ≥ E B Y a.s. (see also Exercise 4). (iii) We have EB c j X j dPB = c j X j dP = cj X j dP B

=

j

B

j

cj

B

E B X j dPB =

j

B

j

j

c j E B X j dPB ,

B

B ∈ B.

159

160


B Since both E B ( j c j X j ) and j c j E X j are B-measurable, it follows that E B ( j c j X j ) = j c j E B X j a.s. (iv) We have B

P B d PB =

B

I dP = P(B) = PB (B) =

B

1 d PB ,

B ∈ B,

while both P B and 1 are B-measurable. Hence P B = 1 a.s.; B P dPB = I dP = 0 = 0 dPB , B ∈ B, B

B

B

and both P B and 0 are B-measurable. Thus P B = 0 a.s.; P B A dPB = I A dP = P(A ∩ B) ≥ 0 = 0 dPB , B

B

B ∈ B,

B

and both P B A and 0 are B-measurable. Hence P B A ≥ 0 a.s. (by Exercise 4). Finally, EB(

c j IA j ) =

j

c j E B I A j a.s. (by (iii)) and this equals

j

B

cj P Aj;

j

i.e., E B (

j

c j IA j ) =

j

c j P B A j a.s.

9.3 Convergence Theorems and Inequalities for Conditional Expectations Theorem 3 (Lebesgue Monotone Convergence Theorem for Conditional Expectaa.s.

tions). If, as n → ∞, 0 ≤ X n ↑ X a.s. and E X < ∞, then 0 ≤ E B X n ↑ E B X a.s. ∞ B In particular, P B ( ∞ j=1 A j ) = j=1 P A j a.s. Proof.

With all limits taken as n → ∞, we have 0 ≤ X n ≤ X n+1 , so that 0 ≤ E B X n ≤ E B X n+1 a.s.

Therefore E B X n ↑ Y (some Y ) a.s. and Y is B-measurable since it is the a.s. limit of B-measurable r.v.s. (See also Exercise 19 in Chapter 3.) Then E B X n dPB ↑ Y dPB , B ∈ B, B

B

9.3 Convergence Theorems and Inequalities for Conditional Expectations

by the classical Monotone Convergence Theorem. But B E X n dPB = X n dP and X n dP ↑ X dP = E B X dPB . B

B

B

Thus

B

Y dPB =

B

B

E B X dPB ,

B

B ∈ B,

a.s.

so that Y = E B X a.s. Therefore 0 ≤ E B X n ↑ E B X a.s. For the second part, we have ⎛ ⎞ n n n a.s. B A j ) = E B Inj=1 A j = E B ⎝ IA j ⎠ = E IA j PB( j=1

j=1

↑

a.s.

∞

EB IA j =

∞

j=1

j=1

PB A j .

j=1

On the other hand, n

Aj ↑

j=1

∞

A j implies

In

j=1

Aj

=

n

j=1

IA j ↑

j=1

, and E B Inj=1 A j ↑ E B I∞ I∞ j=1 A j j=1 A j

∞ j=1

IA j = ⎛

a.s., or P B ⎝

n

⎞ Aj⎠↑

j=1

⎛ ⎞ ∞ PB ⎝ A j ⎠ a.s. j=1

Comparing this with the last result obtained, we get ⎛ ⎞ ∞ ∞ Aj⎠ = P B (A j ) a.s. PB ⎝ j=1

Theorem 4.

Suppose that E |Y | , E |X Y | < ∞ and let X be B-measurable. Then E B (X Y ) = X E B Y a.s.

Proof.

j=1

E B (X Y )

E B (I

First let X = I A , A ∈ B. Then = A Y ) and hence E B (I A Y )dPB = I A Y dP = Y dP = E B Y dPB B B A∩B A∩B = (I A E B Y )dPB , B ∈ B, B

E B (I

EBY

a.s., since both E B (I A Y ) and I A E B Y are B-measurable, so that AY ) = IA B B or E (X Y ) = X E Y a.s.

161

162


Next, let X=

n

α j IA j ,

A j ∈ B,

j = 1, . . . , n.

j=1

Then E B (X Y ) = E B [(

α j I A j )Y ] = E B (

j a.s.

=

a.s.

α j Y IA j ) =

j

α j E B (Y I A j )

j

(by Theorem 2(iii)) α j I A j E B Y (by the previous step)

j

= X E B Y ; i.e., E B (X Y ) = X E B Y a.s. Next, suppose that X , Y ≥ 0 and let 0 ≤ X n simple r.v.s ↑ X as n → ∞. Then, as n → ∞, 0 ≤ X n Y ↑ X Y implies E B (X n Y ) ↑ E B (X Y ) a.s. But

E B (X n Y ) = X n E B Y a.s. and X n E B Y ↑ X E B Y a.s.

Thus E B (X Y ) = X E B Y a.s. Finally, for any r.v.s X and Y , X Y = (X + − X − )(Y + − Y − ) = X + Y + − X + Y − − X − Y + + X − Y − , and the previous results complete the proof of the theorem. Theorem 5 (Conditional Inequalities).

For r , s > 0, suppose that

E |X | , E |Y | , E |Y |s < ∞. r

r

Then (i) E B |X Y | ≤ (E B |X |r )1/r (E B |Y |s )1/s a.s., provided r1 + 1s = 1. (ii) (E B |X + Y |r )1/r ≤ (E B |X |r )1/r + (E B |Y |r )1/r a.s., provided r ≥ 1. (iii) E B |X + Y |r ≤ cr (E B |X |r + E B |Y |r ) a.s., where cr = 1 if (0 1. r (iv) E B X ≤ E B |X | a.s. and, more generally, E B X ≤ E B |X |r a.s., r ≥ 1. Proof. (i) Since E B |X |r and E B |Y |s are a.s. finite, the inequality is trivially true on a possibly exceptional null set; so we may assume that they are finite. More precisely, let A = {ω ∈ ; E B |X |r = 0}, B = {ω ∈ ; E B |Y |s = 0}. First, focus on the set A. If P(A) = 0, then there is no issue regarding ω ∈ A. If P(A) > 0, then since A ∈ B, we have E B |X |r dPB = |X |r dP, 0= A

A

which implies that |X |r = 0 a.s. on A (by Exercise 5(i)), or X = 0 a.s. on A, and this implies that X Y = 0 a.s. on A, of |X Y | = 0 a.s. on A. Hence E B |X Y | = 0


a.s. on A (by Exercise 5(ii)), and therefore the inequality is true on A. Likewise, the inequality is true on B, so that it is true on A ∪ B. Next, consider (A ∪ B)c = Ac ∩ B c on which both E B |X |r and E B |Y |s are positive. Then in the inequality |αβ| ≤

|β|s |α|r + , r s

and on Ac ∩ B c , set:

1/r 1/s

, β = Y / E B |Y |s . α = X / E B |X |r

Then, on Ac ∩ B c , |X Y | |X |r |Y |s

1/r

1/s ≤ B r + B s r E |X | sE |Y | E B |X |r E B |Y |s and hence, by taking the conditional expectations, given B, and using B-measurability of E B |X |r and E B |Y |s , we get, a.s. on Ac ∩ B c , E B |X Y | E B |X |r E B |Y |s 1 1

1/r

1/s ≤ B r + B s = + = 1. B r B s r s r E |X | sE |Y | E |X | E |Y | Therefore

E B |X Y | ≤ (E B |X |r )1/r (E B |Y |s )1/s

a.s. on Ac ∩ B c . (ii) For r = 1, we have |X + Y | ≤ |X | + |Y |, so that a.s.

E B |X + Y | ≤ E B (|X | + |Y |) = E B |X | + E B |Y | . a.s.

For r > 1, |X + Y |r = |X + Y | |X + Y |r −1 ≤ (|X | + |Y |) |X + Y |r −1 = |X | |X + Y |r −1 + |Y | |X + Y |r −1 . Hence, with s > 1 such that r1 + 1s = 1, we have a.s. E B |X + Y |r ≤ E B |X | |X + Y |r −1 + E B |Y | |X + Y |r −1 1/r 1/s a.s. B ≤ E |X |r × E B |X + Y |(r −1)s 1/r 1/s + E B |Y |r × E B |X + Y |(r −1)s 1/s 1/r B (r −1)s E B |X |r = E |X + Y | 1/r B r + E |Y | 1/s 1/r 1/r B r B r B r E |X | , = E |X + Y | + E |Y |

163

164


since (r − 1)s = r as follows from

1 r

+

1 s

= 1. That is,

E B |X + Y |r ≤ (E B |X + Y |r )1/s [(E B |X |r )1/r + (E B |Y |r )1/r ] a.s. Now, if (E B |X + Y |r )1/s = 0 on a set A, then E B |X + Y |r = 0 on A, whereas on Ac , (E B |X + Y |r )1/s > 0. Then on A, the desired inequality is true and so is on Ac by dividing through by (E B |X + Y |r )1/s . Thus we get the result. (iii) We have seen that |X + Y |r ≤ cr (|X |r + |Y |r ). Hence by Theorem 2 (ii), (iii), we get E B |X + Y |r ≤ cr (E B |X |r + E B |Y |r ) a.s. (iv) Let X = X + − X − . Since X + , X − ≥ 0, we get E B X + , E B X − ≥ 0 a.s. and a.s. a.s. a.s. E B X = E B (X + − X − ) = E B X + −E B X − , so that E B X ≤ E B X + +E B X − = E B (X + + X − ) = E B |X |. (For the case that r > 1, see Exercise 6.) a.s.

Theorem 6 (Convergence in the rth Mean for Conditional Expectations). For n ≥ 1, (r )

let X n , X be r.v.s such that E |X n | , E |X | < ∞. Then if X n → X , r ≥ 1, it follows n→∞

(r )

that E B X n → E B X .

n→∞

Proof.

We have r r E E B X n − E B X = E E B (X n − X ) ≤ E(E B |X n − X |r )

r since for r ≥ 1, E B Z ≤ E B |Z |r a.s., by Theorem 5 (iv), and the last expression equals E |X n − X |r → 0. n→∞

Theorem 7 (Fatou–Lebesgue Theorem, Dominated Convergence Theorem for Conditional Expectations). For n ≥ 1, let X n , X , Y , Z be integrable r.v.s. Then, with n → ∞ as appropriate: (i) Y ≤ X n a.s., n ≥ 1, implies E B (lim inf X n ) ≤ lim inf E B X n a.s., provided lim inf X n is integrable. (ii) X n ≤ Z a.s., n ≥ 1, implies lim sup E B X n ≤ E B lim sup X n a.s., provided lim sup X n is integrable. In particular, (i ) Y ≤ X n ↑ X a.s. implies E B X n ↑ E B X a.s. a.s.

a.s.

(ii ) Y ≤ X n ≤ Z a.s., n ≥ 1, and X n → X , imply E B X n → E B X . a.s.

a.s. (iii ) |X n | ≤ U a.s., n ≥ 1, U integrable and X n → X imply E B X n → E B X . (See also Exercise 11.)

Proof.

In all that follows, n → ∞ as appropriate.

(i) We have X n − Y ≥ 0 a.s., n ≥ 1, and a.s.

0 ≤ inf (X j − Y ) ↑ lim inf (X j − Y ) = lim inf(X n − Y ) = lim inf X n − Y . j≥n

j≥n


Hence, by Theorem 3,

B inf (X j − Y ) ↑ E B lim inf(X n − Y ) = E B lim inf X n E j≥n

−E B Y a.s. Next, inf j≥n (X j − Y )

≤

X n − Y , so that

EB

a.s.

inf (X j − Y ) ≤ j≥n

E B (X n − Y ) = E B X n − E B Y . Hence lim E B inf (X j − Y ) ≤ lim inf E B X n − E B Y a.s. or a.s.

j≥n

E B lim inf X n − E B Y ≤ lim inf E B X n − E B Y a.s. or

E B lim inf X n ≤ lim inf E B X n , since E |Y | < ∞, so that E B Y < ∞ a.s. (See also Exercise 7.) (ii) It is proved either similarly, or by utilizing the relations according to which X n ≤ Z a.s. is equivalent to −Z ≤ −X n a.s. and this implies that E B [lim inf(−X n )] ≤ lim inf E B (−X n ) a.s. But E B [lim inf(−X n )] = a.s. a.s. E B (− lim sup X n ) = −E B (lim sup X n ) and lim inf E B (−X n ) = lim inf(−E B X n ) = − lim sup E B X n . Thus, −E B (lim sup X n ) ≤ − lim sup E B X n a.s., or lim sup E B X n ≤ E B (lim sup X n ) a.s. (i ) We have Y ≤ X n ↑ X a.s. or 0 ≤ X n − Y ↑ X − Y a.s., so that E B (X n − Y ) ↑ E B (X − Y ) a.s. (by Theorem 3) or E B X n − E B Y ↑ E B X − E B Y a.s. and hence E B X n ↑ E B X , since E B Y < ∞ a.s.

(ii ) By (i) and (ii) we have

a.s. E B lim inf X n ≤ lim inf E B X n ≤ lim sup E B X n a.s. B

≤ E lim sup X n . But lim inf X n = lim sup X n = X a.s., a.s. a.s. so that E B X ≤ lim inf E B X n ≤ lim sup E B X n ≤ E B X . Thus,

EB Xn → EB X . a.s.

(iii ) Apply (ii ) with Y = −U and Z = U .

165

166


Theorem 8 (Jensen Inequality for Conditional Expectations). Let X be a r.v. taking values in I , an open interval in , with E X ∈ I , and let g : I → be convex nd such that E |g(X )| < ∞. Then g(E B X ) ≤ E B g(X ) a.s. In particular, if g is also nondecreasing, then Y ≤ E B X a.s. implies g(Y ) ≤ E B g(X ) a.s.

Proof. In the proof, treat I as if it were the entire for convenient writing. It is well known that g is continuous and either monotone in or there exists x0 ∈ such that g is nonincreasing in (−∞, x0 ] and nondecreasing in [x0 , ∞). (See Exercise 2 in Chapter 6, and the book by Hardy et al., (1967).) Consider this latter case first. Clearly, without loss of generality, we may assume that x0 = 0 and that g(x0 ) = 0. Thus g(x) ≥ 0 for all x ∈ . Next, let Z be a simple r.v.; i.e., Z = kj=1 z j I A j , where {A j , j = 1, . . . , k} is a (measurable) partition of . Then, clearly, ⎞ ⎛ z j IA j ⎠ = g(z j )I A j , g(Z ) = g ⎝ j

so that

⎡

E B g(Z ) = E B ⎣

j

⎤

j a.s.

=

a.s.

g(z j )I A j ⎦ =

⎛ a.s.

g(z j )E B I A j ≥ g ⎝

j

E B g(z j )I A j

j

⎞ z j E B I A j ⎠ (see Exercise 8)

j

(by the convexity of g and the fact that j E B I A j = j P B A j = 1 a.s.) ⎡ ⎛ ⎞⎤ a.s. ⎣ B ⎝ =g E z j I A j ⎠⎦ = g E B Z . j

That is,

E B g(Z ) ≥ g E B Z a.s.

(9.6)

Now for the given r.v. X , let {X n }, n ≥ 1, be a sequence of simple r.v.s such that, as n → ∞ here and in the remainder of the proof, X n → X , |X n | ≤ |X | , (|X n | ≤ n), and (0 ≤)g(X n ) ≤ g(X ). (In Theorem 17, Chapter 1, take X n = 2jn for j = −n2n + 1, . . . , 0, so that X ≤ X n , n and take X n = j−1 2n for j = 1, . . . , n2 , so that X n ≤ X .) We have then, by (9.6),


with Z replaced by X n , By the Dominated (Theorem 7 (iii)),

E B g(X n ) ≥ g E B X n a.s. Convergence

Theorem

for

(9.7) conditional

expectations

EB Xn → EB X . a.s.

g (x )

[ j −1

) j

2n

2n

0

Hence

[ j −1

) j

2n

2n

x

g(E B X n ) → g(E B X ). a.s.

Also

(9.8) a.s.

0 ≤ g(X n ) ≤ g(X ) and g(X n ) → g(X ). Thus the Fatou–Lebesgue Theorem for conditional expectations again (Theorem 7(ii ) or (iii )) gives a.s. (9.9) E B g(X n ) → E B g(X ). Relations (9.7)–(9.9) give that E B g(X ) ≥ g E B X a.s. Next, let g be monotone. In this case, {X n } may be chosen so that X n → X , |X n | ≤ |X | , (|X n | ≤ n), and |g(X n )| ≤ |g(X )| . Indeed, if g is either increasing or decreasing in , then it will look like either one of the accompanying first four figures. In either case, we may assume (by switching the curve) that it looks like one of the last two pictures. Consider the figure at the left-hand side, as the other figure is treated similarly. For j ≤ 0, take X n = j/2n .

167

168


Then X ≤ X n , g(X ) ≤ g(X n ), |X n | ≤ |X |, and |g(X n )| ≤ |g(X )|. For j ≥ 1, take X n = ( j − 1)/2n . Then |X n | = X n ≤ X = |X |, and |g(X n )| = g(X n ) ≤ g(X ) = |g(X )|. So, in all cases, |X n | ≤ |X | and |g(X n )| ≤ |g(X )|. g(x)

g(x)

x

0

x

0

g (x )

g (x )

x

0

x

0

g (x )

g(x)

[− j 2− 1 , 2j ) n

n

0

x

[

j−1, j 2n 2n

)

[

)

j −1 2n

j 2n

0

j −1 2n

j 2n

[

)

x

9.4 Further Properties of Conditional Expectations

As before, the convexity of g implies (9.7), and thus we reach the same conclusion as before. Likewise if g is nonincreasing in . For the second conclusion, one has Y ≤ E B X a.s. implies g(Y ) ≤ g E B X a.s.

Since g E B X ≤ E B g(X ) a.s., we obtain the result.

9.4 Further Properties of Conditional Expectations and Conditional Probabilities Theorem 9.

If B ⊆ B (⊆ A) and E |X | < ∞, then

E B (E B X ) = E B X = E B (E B X ) a.s.

Proof. The first equality follows by the fact that E B X is B -measurable, since it is B-measurable and B ⊆ B . As for the second equality, we have for all B ∈ B, B B E (E X )dPB = E B X dPB B B = X dP = E B X dPB ; B

i.e.,

B

E B (E B X )dPB =

B

B

E B X dPB ,

B ∈ B,

and hence E B (E B X ) = E B X a.s. since they are both B-measurable.

Now, let X , Y be two r.v.s defined on the probability space (, A, P), and set B X = X −1 (Borel σ -field in ), BY = Y −1 (Borel σ -field in ). Then the elementary definition of independence of the r.v.s. X , Y is equivalent to independence of B X and BY ; i.e., P(A ∩ B) = P(A)P(B), for all A ∈ B X and B ∈ BY . A σ -field B ⊆ A and Y are said to be independent if B and BY are independent. A formalization of these concepts and further elaboration on independence may be found in Section 1 of Chapter 10. The following result will be needed in the sequel; its proof is deferred to Chapter 10 (see Section 10.3). Lemma 1. If the r.v.s X and Y are independent and E |X | , E |Y | < ∞, then E |X Y | < ∞ and E(X Y ) = (E X )(EY ). Remark 3.

By induction, the lemma is true for any n independent integrable r.v.s.

169

170


Theorem 10. Let X be an integrable r.v. such that X and B are independent. Then E B X = E X a.s. Proof. For any B ∈ B, we have that I B and X are independent since {∅, B, B c , } ⊆ B. Next, B E X dPB = X dP = (X I B )dP = E(X I B ) = (E X )(E I B ) B

B

(by independence of X and I B ) E X dP = E X dPB . = (E X )P(B) = B

That is,

B

E B X dPB =

B

B

E X dPB ,

B ∈ B,

whereas both E B X and E X are B-measurable. Thus E B X = E X a.s.

and set Now let Y be a function defined on (, A, P) into BY = A ⊆ ; Y −1 (A ) = A for some A ∈ A . Then, BY is a σ -field of subsets of (by Theorem 11 in Chapter 1). Let also BY = Y −1 (BY ), so that BY is a σ -field ⊆ A. On BY and BY define the probability measures PY and PY , respectively, as follows: PY (B) = P(B), B ∈ BY ; PY (B ) = P[Y −1 (B )] = PY (B), B ∈ BY , B = Y −1 (B ). Definition 2. If X is an integrable r.v. and Y is as above, the conditional expectation of X , given Y , is denoted by E Y X or E(X | Y ) and is defined to be E BY X . The reason for this notation will be explained hereafter. For this purpose, we need the following result, which is, actually, closely related to Theorem 13 in Chapter 4. Lemma 2. Let (, BY , PY ), ( , BY , PY ), be as before and let g : ( , BY ) → ( , Borel σ -field) be measurable, so that g(Y ) : (, BY ) → ( , Borel σ -field) is measurable (a r.v.). Then g(y)dPY = g(Y ) dPY , B ∈ BY , B = Y −1 (B ), B

B

in the sense that, if one of these integrals exists, then so does the other, and both are equal. Proof. The proof follows the familiar line. First, let g(y) = I A (y), A ∈ BY . Then, clearly, g(Y ) = I A (Y ), A = Y −1 (A ). Therefore g(y)dPY = I A (y)dPY = dPY = PY (A ∩ B ) = P[Y −1 (A ∩ B )] B B A ∩B = P(A ∩ B) = PY (A ∩ B) = dPY = I A (Y ) dPY = g(Y ) dPY . A∩B

B

B


That is, the theorem is true for indicators. Then it is true for the case that g is a nonnegative simple r.v. Next, it is true for the case that g is a ≥ 0 r.v., by the Lebesgue Monotone Convergence Theorem, and finally for any g, as described earlier, by writing g as g = g + −g − . (For the details, see Exercise 9; also, Theorem 13 in Chapter 4). Theorem 11. Let (, BY , PY ), ( , BY , PY ) be as before and let X be an integrable r.v. Then E Y X (= E BY X ) is a.s. ([P]) a function of Y ; i.e., it depends on ω ∈ only through Y for almost all (a.a.) ω ∈ . More precisely, there exists a (measurable) function g : ( , BY ) → Borel real line, such that (E Y X )(ω) = g(Y (ω)) a.s. [P]. Proof. On BY , define the finite measures ϕ + , ϕ − and the finite set function ϕ as follows: ϕ + (A) = X + dP, ϕ − (A) = X − dP, A A + − ϕ(A) = ϕ (A) − ϕ (A) = X dP. A

Then ϕ + , ϕ − , ϕ

are all PY -continuous finite measures and ϕ is a signed measure. On BY , define the measures ϕ + , ϕ − and the set function ϕ as follows: ϕ + (A ) = ϕ + (A), ϕ − (A ) = ϕ − (A), ϕ (A ) = ϕ(A),

A = Y −1 (A ).

Then ϕ + , ϕ − are finite measures, ϕ is a signed measure, and all three are PY -continuous. Then, by Theorem 3 in Chapter 7, there exist BY -measurable, nonnegative (a.s. [PY ] finite), and a.s. [PY ] well-defined r.v.s g+ , g− such that ϕ + (A ) = g+ dPY , ϕ − (A ) = g− dPY , A ∈ BY . A

A

By setting g = g+ − g− , we have that g is BY -measurable, (a.s. [PY ] finite), and a.s. [PY ] well defined, and such that ϕ (A ) = gdPY , A ∈ BY . A

But

A

g dPY = ϕ (A )[= ϕ + (A ) − ϕ − (A ) = ϕ + (A) − ϕ − (A)] = ϕ(A) = X dP = E Y X dPY . A

Thus,

A

On the other hand,

g dPY = A

A

A

g(y)dPY =

g(y)dPY =

E Y X dPY . A

g(y) dPY , A

171

172


by Lemma 2. Thus,

E X dPY =

g(Y ) dPY ,

Y

A

A ∈ BY .

A

Since both E Y X and g(Y ) are BY -measurable, we have that E Y X = g(Y ) a.s., as was to be seen. Remark 4. In the last theorem, we have shown that there exists a measurable function g on into , (a.s. [PY ] finite), and a.s. [PY ] well defined such that E Y X = E(X | Y ) = g(Y ) a.s. Now, let y be in the range of Y , let {y} ∈ BY , and set A y = Y −1 ({y}) (so that A y ∈ BY ). Then for ω ∈ A y , (E Y X )(ω) = g(y) a.s. This is what we mean by writing E(X | Y = y) = g(y). If A ∈ X −1 (Borel σ − field in ), it follows that A = X −1 (B) for some B ∈ (Borel σ -field in ). Then E Y I A = P Y A = P Y (X ∈ B), (since A = (X ∈ B)) to be denoted by P(X ∈ B | Y ) and the notation P(X ∈ B | Y = y) is used to denote the value of P(X ∈ B | Y ) at ω ∈ A y . Following is an example where Theorem 11 is employed along with other results obtained in this chapter. Example 3. Let X 1 , . . . , X n be i.i.d. r.v.s defined on the probability space (, A, Pθ ), where the parameter θ lies in the parameter space , an open subset of , and set X = (X 1 , . . . , X n ). Let U = U (X) be a statistic (a measurable function of X not involving any unknown quantities), and let T be a sufficient statistic for θ (as defined in Example 1 of this chapter). The statistic U , as an estimator of θ , is called unbiased, if Eθ U = θ for all θ ∈ . Consider the Eθ (U |T ). Then this conditional expectation has the following two properties: It is independent of θ (by sufficiency), and it depends on T only a.s. (by Theorem 11). Therefore, we may set Eθ (U |T ) = φ(T ). Then also, Eθ φ(T ) = Eθ [Eθ (U |T )] = Eθ U = θ for all θ , so that φ(T ) is also an unbiased estimator of θ . Furthermore, σθ2 [φ(T )] ≤ σθ2 (U ) for all θ , so that φ(T ) is at least as good as U in terms of variances (which are assumed to be finite). The variances inequality is seen as follows. On , consider the function g(x) = (x − θ )2 , which is convex (because g (x) = 2 > 0), and apply the Jensen inequality for conditional expectations (Theorem 8 in this chapter) to obtain [φ(T ) − θ ]2 = [Eθ (U |T ) − θ ]2 a.s.

= [Eθ (U − θ |T )]2 ≤ Eθ [(U − θ )2 |T ] a.s.

Taking the Eθ -expectations of both sides, we get σθ2 [φ(T )] = Eθ [Eθ (U |T ) − θ ]2 ≤ Eθ (U − θ )2 = σθ2 (U ) for all θ .


Application 1. Let X and Y be discrete (real-valued) r.v.s defined on (, A, P), y with Y taking on the values y j with P(Y = y j ) > 0 for all j ≥ 1. In the sequel, will stand for such a value. Let BY be the σ -field generated by {y j }, j ≥ 1 , and let BY = Y −1 (BY ). Then, for every A ∈ BY , A = Y −1 (B ) for some B ∈ BY . Let PY be the restriction of P to BY , and let PY be the probability distribution of Y ; i.e., PY (B ) = PY (A) for every B ∈ BY , where A = Y −1 (B ). Let x be any one of the values x1 , x2 , . . . of X , set B = {x}, and let C = X −1 (B) = X −1 ({x}) = (X = x). Define ϕ as follows: ϕ(A) =

A ∈ BY .

IC dP,

(9.10)

A

Then,

IC dP(= P(A ∩ C)) = (E Y IC )dPY A A = E(IC |Y )dPY = P(C|Y )dPY .

ϕ(A) =

A

Define ϕ by

(9.11)

A

ϕ (B ) = ϕ(A),

B ∈ BY ,

A = Y −1 (B ).

(9.12)

Then ϕ is a finite measure on BY and PY -continuous (since PY (B ) = 0 implies PY (A) = 0 or P(A) = 0, hence ϕ(A) = 0, so that ϕ (B ) = 0). Then, by Theorem 3 in Chapter there exists a BY -measurable function g, nonnegative (a.s. [PY ] finite), 7, and a.s. PY well defined, such that ϕ (B ) = By Lemma 2,

B

g(y)dPY =

B

g(y)dPY ,

B ∈ BY .

g(Y )dPY (A = Y −1 (B )),

(9.14)

A

and therefore, by means of (9.10)–(9.14), g(Y )dPY = P(C|Y )dPY , A

(9.13)

A ∈ BY .

(9.15)

A

Since g(Y ) and P(C|Y ) are BY -measurable, relation (9.15) implies that P(C|Y ) = P(X = x|Y ) = g(Y ) a.s., and, in particular, P(C|Y )(ω) = P(X = x|Y )(ω) = g(y) a.s. for ω ∈ A y = Y −1 ({y}).

(9.16)

Next, for a fixed x denoting any one of the values x1 , x2 , . . . of X , P(X = x|Y = y) =

P(X = x, Y = y) is BY -measurable (see also Exercise 10) P(Y = y)

173

174


and by (9.11), B

=

P(X = x|Y = y)dPY =

P(X = x, Y = y) P(Y = y) P(Y = y)

y∈B

P(X = x, Y = y) = P(X = x, Y ∈ B )

y∈B

= P(A ∩ C) = ϕ(A) g(y)dPY (by (9.12) and (9.13)). = ϕ (B ) =

(9.17)

B

The BY -measurability of P(X = x|Y = y) and of g(y), and relation (9.17) imply that (9.18) P(X = x|Y = y) = g(y) a.s. PY . Therefore, by (9.16) and (9.18), P(X = x|Y = y) = P(X = x|Y )(ω) a.s. for ω ∈ A y = Y −1 ({y}).

(9.19)

Relation (9.19) states that P(X = x|Y = y) is a.s. equal to P(C|Y ) evaluated at each ω in A y . This result demonstrates that, in the present setup, the general definition and the elementary definition of a conditional p.d.f. coincide (a.s.). Exercises. 1. If X is an integrable r.v. and {An , n = 1, 2, . . .} are pairwise disjoint events, show that ∞ X dP = X dP. ∞ Ai

i=1

i=1

Ai

Hint: Split X into X + and X − and use Corollary 1 (ii) to the Lebesgue Monotone Convergence Theorem (Theorem 1 in Chapter 5). 2. Let X be an integrable r.v. and, for B ∈ A with P(B) > 0, consider the conditional probability on A, P(·|B). Then show that 1 X dP(·|B) = X dP. P(B) B Hint: Go through the familiar four steps that X is an indicator function, a simple r.v., a nonnegative r.v., any r.v. 3. Consider the probability space (, A, P) = ([0, 1], B[0,1] , λ), where λ is the Lebesgue measure, and let F be the σ -field generated by the class {[0, 41 ], ( 41 , 23 ], ( 23 , 1]}. Also, let X be the r.v. defined by: X (ω) = ω2 , ω ∈ . Then show that E(X |F) = α1 I[0, 1 ] + α2 I( 1 , 2 ] + α3 I( 2 ,1] , 4

4 3

3

and compute α1 , α2 , α3 . Hint: Refer to the special cases discussed right after Definition 1.


4. Let X and Y be B-measurable and integrable r.v.s. We further assume that B X dP ≥ B Y dP for every B ∈ B(⊆ A). Then show that X ≥ Y a.s. Hint: By setting Z = Y − X , we have B Z dP ≤ 0 for all B ∈ B, and we wish to conclude that Z ≤ 0 a.s. Set C = (Z ≤ 0) and D = (Z > 0)(= C c ). Then it suffices to show D Z dP ≤ 0 and that P(D) = 0. By taking B = D, we have Z dP = (Z I )dP ≥ 0 since Z > 0 on D, so that Z dP D D D = 0. Thus, it suffices to show that, if for a r.v. Z with D = (Z > 0) it holds that D Z dP = 0, then P(D) = 0. This can be done through the four familiar steps. 5. (i) Let X ≥ 0 a.s. on a set A with P(A) > 0, and suppose that A X dP = 0. Then show that X = 0 a.s. on A. (ii) Let X ≥ 0, integrable, and X = 0 a.s. on a set A ∈ B(⊆ A), with P(A) > 0. Then show that E B X = 0 a.s. on A. Hint: (i) With C = (X > 0), we have 0 = A X dP = D X dP, D = A ∩ C. So, D X dP = 0 and X > 0 on D. Show that P(D) = 0 which would be equivalent to saying that X = 0 a.s. on A. Do it by going through the four familiar steps. (iii) Use the fact that B X dP = B E B X dPB for all B ∈ B, replace B by B ∩ A, and conclude that I A E B X = 0 a.s. This would imply that E B X = 0 a.s. on A. 6. If E|X |r < ∞, then show that |E B X |r ≤ E B |X |r a.s., r ≥ 1. Hint: One way of establishing this inequality is to use the Jensen inequality (Theorem 8). For this purpose, take g(x) = |x|r , r ≥ 1, and observe that it is convex. (It is convex for x ≥ 0 and symmetric about the y-axis, hence convex in .). 7. If the r.v. X is integrable, then E B X is finite a.s. 8. Recall that a function g : I (open interval) ⊆ → is said to be convex if g(α1 x1 + α2 x2 ) ≤ α1 g(x1 ) + α2 g(x2 ) for all α1 , α2 ≥ 0 with α1 + α2 = 1, and all x1 , x2 ∈ I . Prove the following generalization: if g is as above, then: g(α1 x1 + · · · + αn xn ) ≤ α1 g(x1 ) + · · · + αn g(xn )

(*)

for any n ≥ 2, any α1 , . . . , αn ≥ 0 with α1 +· · ·+αn = 1, and all x1 , . . . , xn ∈ I . Hint: Use the induction method. Inequality (*) is true for n = 2, assume it to be true for n = k and establish it for n = k + 1. In the expression g(α1 x1 + · · · + αk+1 xk+1 ) group the terms in two parts, one containing the first k terms and one containing the last term. In the first group, multiply and divide by 1 − αk+1 (assuming, without loss of generality, that αk+1 < 1), and use the induction hypothesis. 9. Fill in the details in proving Lemma 2. Hint: Refer to the proof of Theorem 13 in Chapter 4. 10. Let X and Y be discrete r.v.s and recall (from the application following Theorem 11) that BY is the σ -field of subsets of defined by BY = {B ⊆ ; Y −1 (B ) = A for some A ∈ A}. For x, y ∈ with P(Y = y) > 0, consider the

175

176


conditional probability P(X = x|Y = y) = P(X = x, Y = y)/P(Y = y) and show that, for each fixed x, the function P(X = x|Y = ·) is BY -measurable. 11. The Dominated Convergence Theorem in its classical form states: If |X n | ≤ Y a.s.

P

n→∞

n→∞

a.s., n ≥ 1, EY < ∞, and either X n → X or X n → X , then E X n → E X finite. n→∞

In the framework of conditional expectations, we have shown that: If |X n | ≤ Y a.s. a.s. a.s., n ≥ 1, EY < ∞, and X n → X with E X finite, then E B X n → E B X for n→∞

n→∞

any σ -field B ⊆ A (see Theorem 7 (iii )). a.s. By means of an example, show that the convergence X n → X cannot be n→∞

P

a.s.

n→∞

n→∞

replaced by X n → X and still conclude that E B X n → E B X . 12. Let the r.v.s X and Y have the Bivariate Normal distribution with parameters μ1 , μ2 in , 0 < σ1 , σ2 < ∞, and ρ ∈ [−1, 1], so that their joint probability density function (p.d.f.) is given by p X ,Y (x, y) = where 1 q= 1 − ρ2

+

1

2π σ1 σ2 1 − ρ 2

x − μ1 σ1

!2

y − μ2 σ2

− 2ρ !2 "

e−q/2 ,

x − μ1 σ1

!

y − μ2 σ2

!

, x, y ∈ .

(i) Show that the exponent may be written thus: ! ! ! x − μ1 2 y − μ2 x − μ1 2 2 2 −ρ (1 − ρ )q = + (1 − ρ ) σ2 σ1 σ1 !2 !2 y−b x − μ1 = + (1 − ρ 2 ) , σ2 σ1 ρσ2 where b = μ2 + (x − μ1 ). σ1 (ii) From part (i), it follows that:

" (x − μ1 )2 p X ,Y (x, y) = √ exp − 2σ12 2π σ1 " (y − b)2 1 exp − ×√ . 2π (σ2 1 − ρ 2 ) 2(σ2 1 − ρ 2 )2 1


From this expression, and without any actual integration, conclude that the r.v. X is distributed as N (μ1 , σ12 ); i.e., X ∼ N (μ1 , σ12 ), and by symmetry, Y ∼ N (μ2 , σ22 ). 13. In reference to Exercise 12 (ii), and without any actual operations, conclude that the conditional distribution of the r.v. Y , give X = x, is N (b, σ22 (1 − ρ 2 )), where 2 b = μ2 + ρσ σ1 (x − μ1 ); and by symmetry, the conditional distribution of X , given 1 Y = y, is N (c, σ12 (1 − ρ 2 )), where c = μ1 + ρσ σ2 (y − μ2 ). 14. (i) In reference to Exercises 12 and 13, and by writing E(X Y ) = E[E(X Y )|X ], show that E(X Y ) = μ1 μ2 + ρσ1 σ2 . (ii) Use the definition of the covariance of two r.v.s X and Y (with finite second moments), Cov(X , Y ) = E[(X − E X )(Y − EY )] = E(X Y ) − (E X )(EY ), in order to conclude that, in the present case, Cov(X , Y ) = ρσ1 σ2 . (iii) From part (ii), conclude that, in the present case, the correlation coefficient of the r.v.s X and Y , ρ(X , Y ), is equal to ρ. 15. If the r.v.s X and Y have the Bivariate Normal distribution with parameters μ1 , μ2 in , 0 < σ1 , σ2 < ∞ and ρ ∈ [−1, 1], set U=

Y − μ2 X − μ1 and V = σ1 σ2

and show that the r.v.s U and V have the Bivariate Normal distribution with parameters 0, 0, 1, 1 and ρ, by transforming the joint p.d.f. of X and Y into the joint p.d.f. of U and V . 16. Let X and Y be r.v.s defined on the probability space (, A, P), and suppose that E X 2 < ∞. Then show that (i) The conditional variance of X , given Y , is given by the formula Var (X |Y ) = E{[X − E(X |Y )]2 |Y } = E(X 2 |Y ) − [E(X |Y )]2 a.s. (ii) Var(X ) = E[Var (X |Y )] + Var [E(X |Y )]. 17. (Wald) (i) Let X 1 , X 2 , . . . be r.v.s and let N be a r.v. taking the values 1, 2, . . ., all defined on the probability space (, A, P). Define the function X as X (ω) = X 1 (ω) + · · · + X N (ω) (ω), and show that X is a r.v. (ii) Now suppose that the X i s are independent and identically distributed with E X 1 = μ ∈ , that N is independent of the X i s, and that E N < ∞. Then show that E(X |N ) = μN , and therefore E X = μ(E N ).

177

178


(iii) If in addition to the assumptions made in part (ii), it also holds that Var(X 1 ) = σ 2 < ∞ and Var(N ) < ∞, then show that Var(X |N ) = σ 2 N . (iv) Use parts (ii) and (iii) here and part (ii) of Exercise 16 in order to conclude that Var(X ) = σ 2 (E N ) + μ2 Var(N ). Hint: For parts (ii) and (iii), use the special case right after Definition 1, and part (i) of Exercise 16. 18. Let B be a sub-σ -field in (, A, P) which is equivalent to the trivial σ -field {, } (in the sense that, for every B ∈ B, either P(B) = 0 or P(B) = 1), and let X be a B-measurable (integrable) r.v. Then show that X = E X a.s. 19. In reference to Example 2, show that, for each t = 0, 1, . . ., there are, indeed,

n+t−1 xi = (xi1 , . . . , xin ) where each xi1 , . . . , xin ranges from 0 to t, and t

xi1 + · · · + xin = t, i = 1, . . . , n+t−1 . t

CHAPTER

Independence

10

The concept of independence of two σ -fields and two r.v.s was introduced in Section 4 of the previous chapter, because it was needed in Lemma 1 there. What we do in this chapter is to elaborate to a considerable extent on the concept of independence and some of its consequences. In Section 10.1, the relevant definitions are given, and a result regarding independence of functions of independent r.v.s is stated and proved. The highlight of the chapter is Theorem 1, which states that the factorization of the joint d.f. of n r.v.s to the individual d.f.s implies independence of the r.v.s involved. Section 10.2 is devoted to establishing those auxiliary results, which are needed for the proof of Theorem 1. In the final section of the chapter, the proof of Theorem 1 is given, as well as the proof of Lemma 1 in Chapter 9.

10.1 Independence of Events, σ -Fields, and Random Variables Here, we recall the definition of independence of events, and then extend it to independence of classes of events (and in particular, fields or σ -fields), and independence of r.v.s. In all that follows, (, A, P) is the underlying probability space. Definition 1. Two events A1 , A2 are said to be independent (stochastically, statistically, or in the probability sense), if P(A1 ∩ A2 ) = P(A1 )P(A2 ).

(10.1)

For n ≥ 2, the events A1 , . . . , An are said to be independent, if for any k ≥ 2 and any (integers) n 1 , . . . , n k with 1 ≤ n 1 < n 2 < · · · < n k ≤ n, it holds that P(An 1 ∩ An 2 ∩ · · · ∩ An k ) = P(An 1 )P(An 2 ) · · · P(An k ).

(10.2)

Any collection of events, {Ai , i ∈ I }, is said to be independent, if any finite subcollection is a set of independent events. Definition 2. Two classes of events C1 and C2 with C j ⊆ A, j = 1, 2, are said to be independent, if for all choices of events A j ∈ C j , j = 1, 2, relation (10.1) holds. For An Introduction to Measure-Theoretic Probability, Second Edition. http://dx.doi.org/10.1016/B978-0-12-800042-7.00010-4 Copyright © 2014 Elsevier Inc. All rights reserved.

179

180

CHAPTER 10 Independence

n ≥ 2, the classes of events C j ⊆ A, j = 1, . . . , n, are said to be independent, if for all choices of events A j ∈ C j , j = 1, . . . , n, the events {A1 , . . . , An } are independent; i.e., relation (10.2) holds. Any collection of classes of events C j ⊆ A, j ∈ I , are said to be independent, if any finite subcollection out of these classes is a collection of independent classes. In particular, if the classes C j are fields F j or σ -fields A j , then we talk about independence of fields or σ -fields, respectively. Definition 3. Consider the r.v.s X 1 and X 2 , and let A j = X −1 j (B), j = 1, 2, be the σ -fields induced by them in (B being the Borel σ -field in ). Then we say that the r.v.s X 1 and X 2 are independent, if the σ -fields A1 and A2 are independent. The σ field A1 and the r.v. X are said to be independent if the σ -fields A1 and A2 = X −1 (B) are independent. For n ≥ 2, the r.v.s X j , j = 1, . . . , n, are said to be independent, if the σ -fields A j = X −1 j (B), j = 1, . . . , n, induced by them are independent. Finally, restricting ourselves to denumerably many r.v.s, we say that the r.v.s {X n , n ≥ 1} are independent, if any finite collection of them is a set of independent r.v.s. Remark 1. In order to establish independence of n fields F j or σ -fields A j , j = 1, . . . , n, it suffices to show that, for all choices of events A j ∈ F j or A j ∈ A j , j = 1, . . . , n, it holds that P(A1 ∩ · · · ∩ An ) = P(A1 ) · · · P(An ).

(10.3)

In other words, it is not necessary to check (10.2) for all subcollections of k events out of the n events. This is so, because by taking A j = for j = n 1 , . . . , n k , relation (10.3) reduces to (10.2). Remark 2. From Definition 2, it is immediate that subclasses of independent classes are also independent. We record below a simple but very useful result. Proposition 1. Borel functions of independent r.v.s are independent r.v.s. That is, if the r.v.s {X n , n ≥ 1} are independent and Yn = gn (X n ), where gn : → measurable, then the r.v.s {Yn , n ≥ 1} are independent. Proof. It follows immediately by Remark 2, because, if A X n are the σ -fields induced by the r.v.s X n , and AYn are the σ -fields induced by the r.v.s Yn , then AYn ⊆ A X n , n ≥ 1. This is so because Yn−1 (B) = [gn (X n )]−1 (B) = X n−1 [gn−1 (B)] ∈ A X n since gn−1 (B) ∈ B, B ∈ B. Now, consider the r.v.s X j , j = 1, . . . , n, and let A j , j = 1, . . . , n be the σ fields induced by them. Then by Definition 3 and Remark 1, independence of the r.v.s X 1 , . . . , X n amounts to the validity of relation (10.3) for all choices of the events A j ∈ A j , j = 1, . . . , n. But A j ∈ A j means that A j = X −1 j (B j ), some B j ∈ B. Therefore relation (10.3) becomes, equivalently, P(X 1 ∈ B1 , . . . , X n ∈ Bn ) = P(X 1 ∈ B1 ) . . . P(X n ∈ Bn )

(10.4)

10.2 Some Auxiliary Results

for all B j ∈ B, j = 1, . . . , n. Thus, establishing independence for the r.v.s X 1 , . . . , X n amounts to checking relation (10.4). However, in lower level textbooks, it is claimed that the r.v.s X 1 , . . . , X n are independent, if P X 1 ≤ x1 , . . . , X n ≤ xn = P(X 1 ≤ xn ) . . . P(X n ≤ xn ), or P X 1 ∈ (−∞, x1 ], . . . , X n ∈ (−∞, xn ] = P X 1 ∈ (−∞, x1 ] . . . P X n ∈ (−∞, xn ] (10.5) for all x1 , . . . , xn in . Relation (10.5) is a very special case of (10.4), taken from it for B j = (−∞, x j ], j = 1, . . . , n. The question then arises as to whether this claim is valid, as it should be. The justification of this claim is the content of the next result. Theorem 1. The r.v.s X 1 , . . . , X n are independent, if and only if relation (10.5) holds for all x1 , . . . , xn in . As already stated, the validity of (10.5) is a special case of (10.4), taken from it for B j = (−∞, x j ], x j ∈ ; or to put it differently, by taking B j s from the class C = {(−∞, x]; x ∈ }. We show below that (10.5) also holds when this class is enlarged to include , , and any interval in finite or not, and of any form. The proof of this is facilitated by the results established in the following section.

10.2 Some Auxiliary Results In this section, three lemmas and one proposition are established, on which the proof of Theorem 1 is based. Lemma 1. If relation (10.5) holds when the B j s in (10.4) are chosen from the class C, then it holds if the B j s are chosen from the class C0 : C0 = {(−∞, x], (−∞, x), (x, ∞), [x, ∞), (x, y], [x, y), (x, y), [x, y], , ; x, y ∈ } = {, , (x, ∞), (x, y], (−∞, x], [x, y], (x, y), [x, y); x, y ∈ } (listed in a convenient order). Proof. In (10.5), if at least one of the intervals (−∞, x j ], j = 1, . . . , n, is replaced by , then it becomes identity, 0 = 0. Thus, (10.5) holds with intervals (−∞, x j ], j = 1, . . . , n, replaced by . Next, P(X 1 ∈ , X j ≤ x j , j = 2, . . . , n) = P X1 ∈ (−∞, ym ], X j ≤ x j , j = 2, . . . , n where ym ↑ ∞, m → ∞ m

181

182


=P (X 1 ≤ ym ), X j ≤ x j , j = 2, . . . , n m

(X 1 ≤ ym , X j ≤ x j , j = 2, . . . , n) =P

m

= P lim(X 1 ≤ ym , X j ≤ x j , j = 2, . . . , n) m

= lim P(X 1 ≤ ym , X j ≤ x j , j = 2, . . . , n) m

= lim P(X 1 ≤ ym )P(X 2 ≤ x2 ) · · · P(X n ≤ xn ) m

= P(X 1 ∈ )P(X 2 ≤ x2 ) · · · P(X n ≤ xn ). So, the factorization in (10.5) holds if one of the intervals (−∞, x j ], j = 1, . . . , n (which without loss of generality may be taken to be the interval (−∞, x1 ]), is replaced by . Assuming the factorization to be true when k of the preceding intervals (2 ≤ k < n) are replaced by , we show as before that it is also true if k + 1 intervals are replaced by . So, by the induction hypothesis, any number of intervals may be replaced by and the factorization in (10.5) holds. Next, P(X 1 > x, X j ≤ x j , j = 2, . . . , n) = P(X 1 ∈ ( − (−∞, x]), X j ≤ x j , j = 2, . . . , n) = P((X 1 ∈ , X j ≤ x j , j = 2, . . . , n) −(X 1 ≤ x, X j ≤ x j , j = 2, . . . , n)) = P(X 1 ∈ , X j ≤ x j , j = 2, . . . , n) −P(X 1 ≤ x, X j ≤ x j , j = 2, . . . , n) = P(X 1 ∈ )P(X 2 ≤ x2 ) · · · P(X n ≤ xn ) −P(X 1 ≤ x)P(X 2 ≤ x2 ) · · · P(X n ≤ xn ) = [P(X 1 ∈ ) − P(X 1 ≤ x)]P(X 2 ≤ x2 ) · · · P(X n ≤ xn ) = P(X 1 > x)P(X 2 ≤ x2 ) · · · P(X n ≤ xn ). As in the previous step, the factorization holds if any number of the intervals (−∞, x j ], j = 1, . . . , n, is replaced by intervals of the form (x, ∞). Next, P(x < X 1 ≤ y, X j ≤ x j , j = 2, . . . , n) = P(((X 1 ≤ y) − (X 1 ≤ x)), X j ≤ x j , j = 2, . . . , n) = P(X 1 ≤ y, X j ≤ x j , j = 2, . . . , n) −P(X 1 ≤ x, X j ≤ x j , j = 2, . . . , n) = P(X 1 ≤ y)P(X 2 ≤ x2 ) · · · P(X n ≤ xn ) −P(X 1 ≤ x)P(X 2 ≤ x2 ) · · · P(X n ≤ xn )


= [P(X 1 ≤ y) − P(X 1 ≤ x)]P(X 2 ≤ x2 ) · · · P(X n ≤ xn ) = P(x < X 1 ≤ y)P(X 2 ≤ x2 ) · · · P(X n ≤ xn ), and the same is true, if any number of the (−∞, x j ], j = 1, . . . , n, is replaced by intervals of the form (x, y]. Next, P(X 1 < x, X j ≤ x j , j = 2, . . . , n) (−∞, ym ], X j ≤ x j , j = 2, . . . , n) where ym ↑ x, m → ∞ = P(X 1 ∈ m

(X 1 ≤ ym ), X j ≤ x j , j = 2, . . . , n =P m

(X 1 ≤ ym , X j ≤ x j , j = 2, . . . , n) =P

m

= P lim(X 1 ≤ ym , X j ≤ x j , j = 2, . . . , n) m

= lim P(X 1 ≤ ym , X j ≤ x j , j = 2, . . . , n) m

= lim P(X 1 ≤ ym )P(X 2 ≤ x2 ) · · · P(X n ≤ xn ) m

= P(X 1 < x)P(X 2 ≤ x2 ) · · · P(X n ≤ xn ). Then, arguing as in the previous steps, we conclude that (10.5) holds if any number of the (−∞, x j ], j = 1, . . . , n, is replaced by intervals of the form (−∞, x). Next, P(x ≤ X 1 ≤ y, X j ≤ x j , j = 2, . . . , n) = P(((X 1 ≤ y) − (X 1 < x)), X j ≤ x j , j = 2, . . . , n) = P(X 1 ≤ y, X j ≤ x j , j = 2, . . . , n) −P(X 1 < x, X j ≤ x j , j = 2, . . . , n) = P(X 1 ≤ y)P(X 2 ≤ x2 ) · · · P(X n ≤ xn ) −P(X 1 < x)P(X 2 ≤ x2 ) · · · P(X n ≤ xn ) = [P(X 1 ≤ y) − P(X 1 < x)]P(X 2 ≤ x2 ) · · · P(X n ≤ xn ) = P(x ≤ X 1 ≤ y)P(X 2 ≤ x2 ) · · · P(X n ≤ xn ), and arguing as before, we conclude that (10.5) holds if any number of the (−∞, x j ], j = 1, . . . , n, is replaced by intervals of the form [x, y]. Next, P(X 1 ≥ x, X j ≤ x j , j = 2, . . . , n) (ym , ∞), X j ≤ x j , j = 2, . . . , n where ym ↑ x, m → ∞ = P X1 ∈ m

183

184


=P (X 1 > ym ), X j ≤ x j , j = 2, . . . , n m

(X 1 > ym , X j ≤ x j , j = 2, . . . , n) =P

m

= P lim(X 1 > ym , X j ≤ x j , j = 2, . . . , n) m

= lim P(X 1 > ym , X j ≤ x j , j = 2, . . . , n) m

= lim P(X 1 > ym )P(X 2 ≤ x2 ) · · · P(X n ≤ xn ) m

= P(X 1 ≥ x)P(X 2 ≤ x2 ) · · · P(X n ≤ xn ), and arguing as in previous steps, we conclude that (10.5) holds if any number of the (−∞, x j ], j = 1, . . . , n, is replaced by intervals of the form [x, ∞). Next, P(x < X 1 < y, X j ≤ x j , j = 2, . . . , n) (x, ym ], X j ≤ x j , j = 2, . . . , n where ym ↑ y, m → ∞ = P X1 ∈ =P

m

(x < X 1 ≤ ym ), X j ≤ x j , j = 2, . . . , n

m

=P

(x < X 1 ≤ ym , X j ≤ x j , j = 2, . . . , n)

m

= P lim(x < X 1 ≤ ym , X j ≤ x j , j = 2, . . . , n) m

= lim P(x < X 1 ≤ ym , X j ≤ x j , j = 2, . . . , n) m

= lim P(x < X 1 ≤ ym )P(X 2 ≤ x2 ) · · · P(X n ≤ xn ) m

= P(x < X 1 < y)P(X 2 ≤ x2 ) · · · P(X n ≤ xn ), and as in previous steps, it follows that (10.5) holds when any number of the (−∞, x j ], j = 1, . . . , n, is replaced by intervals of the form (x, y). Finally, P(x ≤ X 1 < y, X j ≤ x j , j = 2, . . . , n) = P X1 ∈ (ym , y), X j ≤ x j , j = 2, . . . , n where ym ↑ x, m → ∞ m

(ym < X 1 < y), X j ≤ x j , j = 2, . . . , n =P m

(ym < X 1 < y, X j ≤ x j , j = 2, . . . , n) =P m


= P lim(ym < X 1 < y, X j ≤ x j , j = 2, . . . , n) m

= lim P(ym < X 1 < y, X j ≤ x j , j = 2, . . . , n) m

= lim P(ym < X 1 < y)P(X 2 ≤ x2 ) · · · P(X n ≤ xn ) m

= P(x ≤ X 1 < y)P(X 2 ≤ x2 ) · · · P(X n ≤ xn ), and as before, we conclude that (10.5) holds when any number of the intervals (−∞, x j ], j = 1, . . . , n, is replaced by intervals of the form [x, y). Finally, combining the 10 conclusions just reached, we have that the factorization (10.5) holds when the B j s in (10.4) are chosen in any way from the class C0 . This completes the proof of the lemma. Lemma 2. Let F be the field (by Exercise 7(ii) in Chapter 1) of all finite sums of members of C0 . Then (10.5) holds, if any number of intervals (−∞, x j ], j = 1, . . . , n, are replaced by elements of F.

m Ii with Ii ∈ C0 , and Proof. Let I ∈ F. Then I = i=1 P(X 1 ∈ I , X j ≤ x j , j = 2, . . . , n) = P X1 ∈ Ii , X j ≤ x j , j = 2, . . . , n i

=P (X 1 ∈ Ii ), X j ≤ x j , j = 2, . . . , n i

=P (X 1 ∈ Ii , X j ≤ x j , j = 2, . . . , n) . i

However,

(X 1 ∈ Ii , X j ≤ x j , j = 2, . . . , n) P

i

=

P X 1 ∈ Ii , X j ≤ x j , j = 2, . . . , n .

i

Therefore P(X 1 ∈ I , X j ≤ x j , j = 2, . . . , n) P(X 1 ∈ Ii )P(X 2 ≤ x2 ) · · · P(X n ≤ xn ) (by Lemma 1) = i

= P(X 2 ≤ x2 ) · · · P(X n ≤ xn )

i

= P(X 2 ≤ x2 ) · · · P(X n ≤ xn )P

P(X 1 ∈ Ii )

X1 ∈

i

Ii

185

186


= P(X 2 ≤ x2 ) · · · P(X n ≤ xn )P(X 1 ∈ I ) = P(X 1 ∈ I )P(X 2 ≤ x2 ) · · · P(X n ≤ xn ). Thus, the factorization in (10.5) holds if one of the intervals (−∞, x j ], j = 1, . . . , n (which without loss of generality may be taken to be the interval (−∞, x1 ]), is replaced by a member I of F. Assuming it to be true when k of the intervals (2 ≤ k < n) are replaced by members of F, we show as before that the factorization also holds if k + 1 intervals are replaced by I s. So, by the induction hypothesis, any number of intervals may be replaced by I s and the factorization holds. This completes the proof. With F as in Lemma 2, we know that B = σ (F); i.e., B is generated by F. Let −1 F j = X −1 j (F) and A j = X j (B), j = 1, . . . , n. Then by Theorem 12 in Chapter 1, A j = σ (F j ), j = 1, . . . , n. A reformulation of Lemma 2 in terms of elements in the F j s is as follows: for all F j ∈ F j , j = 1, . . . , n, P(F1 ∩ · · · ∩ Fn ) = P(F1 ) · · · P(Fn ).

(10.6)

It is our intention to show that (10.6) holds when the F j s are arbitrary members of the respective A j s. To this effect, let M1 be defined as follows: M1 = {B ∈ A1 ; P(B ∩ F2 ∩ · · · ∩ Fn ) = P(B)P(F2 ) · · · P(Fn ) with F j ∈ F j , j = 2, . . . , n}.

(10.7)

Then, by (10.6) and (10.7), F1 ⊆ M1 ⊆ A1 , and we shall show that M1 is a monotone class. To this end, let {Am }, m ≥ 1, be a monotone sequence of elements of M1 with limit A. We shall show that A ∈ M1 , which amounts to showing that P(A ∩ F2 ∩ · · · ∩ Fn ) = P(A)P(F2 ) · · · P(Fn )

(10.8)

for all F j ∈ F j , j = 2, . . . , n. Indeed, as m → ∞,

P(A ∩ F2 ∩ · · · ∩ Fn ) = P lim Am ∩ F2 ∩ · · · ∩ Fn m

= P ∪ Am ∩ F2 ∩ · · · ∩ Fn , if Am ↑ m

= P ∪(Am ∩ F2 ∩ · · · ∩ Fn ) m

= P lim(Am ∩ F2 ∩ · · · ∩ Fn ) m

= lim P(Am ∩ F2 ∩ · · · ∩ Fn ) m

= lim P(Am )P(F2 ) · · · P(Fn ) m

= P(A)P(F2 ) · · · P(Fn ), and the same is true, if Am ↓, by replacing union by intersection. This justifies the assertion made. Next, define M2 by M2 = {B ∈ A2 ; P(A1 ∩ B ∩ F3 ∩ · · · ∩ Fn ) = P(A1 )P(B)P(F3 ) · · · P(Fn ) with A1 ∈ M1 and F j ∈ F j , j = 3, . . . , n}.

(10.9)

10.3 Proof of Theorem 1 and of Lemma 1 in Chapter 9

Then, by (10.8) and (10.9), F2 ⊆ M2 ⊆ A2 , and we shall show that M2 is a monotone class. This is done as in the previous step by letting {Am }, m ≥ 1, be a monotone sequence of elements of M2 with limit A, and showing that P(A1 ∩ A ∩ F3 ∩ · · · ∩ Fn ) = P(A1 )P(A)P(F3 ) · · · P(Fn )

(10.10)

for all A1 ∈ M1 and all F j ∈ F j , j = 3, . . . , n. Continuing on like this, let Mn be the class defined by Mn = {B ∈ An ; P(A1 ∩ · · · ∩ An−1 ∩ B) = P(A1 ) · · · P(An−1 )P(B) with A j ∈ M j , j = 1, . . . , n − 1}.

(10.11)

Then as before, Fn ⊆ Mn ⊆ An , and Mn is a monotone class, because if {Am }, m ≥ 1, is a monotone sequence of elements of Mn with limit A, then it is shown as in the previous steps that P(A1 ∩ · · · ∩ An−1 ∩ A) = P(A1 ) · · · P(An−1 )P(A)

(10.12)

for all A j ∈ M j , j = 1, . . . , n − 1. Gathering together the results just obtained, we then have the following Lemma 3. For j = 1, . . . , n, the classes M j defined by (10.7), (10.9), and (10.11) are monotone classes with the property that F j ⊆ M j ⊆ A j , and by (10.12), n n P(A j ), for all A j ∈ M j , j = 1, . . . , n. (10.13) P ∩ Aj = j=1

j=1

The following proposition is an immediate consequence of what has just been discussed. Proposition 2. If the fields F j , j = 1, . . . , n are independent, then so are the σ -fields generated by them, A j = σ (F j ), j = 1, . . . , n. Proof. In Lemma 3, it was proved that relation (10.6) implies relation (10.13) for all A j ∈ M j , where M j are monotone classes with F j ⊆ M j ⊆ A j , j = 1, . . . , n. But, by Theorem 6 in Chapter 1, M j = A j , j = 1, . . . , n. This completes the proof.

10.3 Proof of Theorem 1 and of Lemma 1 in Chapter 9 We are now ready to prove Theorem 1 here and Lemma 1 in the previous chapter. Proof of Theorem 1. If the r.v.s are independent, then n n P(A j ), for all A j ∈ A j , j = 1, . . . , n, P ∩ Aj = j=1

j=1

187

188


and in particular, this is true for A j = X −1 j ((−∞, x j ]), j = 1, . . . , n, which gives (10.5). In the other way around, if (10.5) holds, then so does (10.13) by way of Lemma 3 and Proposition 2, for all A j ∈ A j , j = 1, . . . , n, which establishes independence of X j , j = 1, . . . , n. Remark 3.

Relation (10.5) can also be written as follows in terms of d.f.s: FX 1 ,...,X n (x1 , . . . , xn ) = FX 1 (x1 ) · · · FX n (xn )

for all x1 , . . . , xn in . This section is concluded with the proof of Lemma 1 in Chapter 9. Proof of Lemma 1 in Chapter 9. The proof follows the four familiar steps. It is established successively for indicators, simple r.v.s, nonnegative r.v.s, and any r.v.s. So, let X = I A , Y = I B , A, B ∈ A. Then the σ -fields B X = {∅, A, Ac , } and BY = {∅, B, B c , } are independent because of the independence of the r.v.s = X and Y . Next, X Y = I A∩B , so that E(X

Ym) = P(A ∩ B) = P(A)P(B) n (E X )(EY ). Now let X = (E I A )(E I B ) =

i=1 αi I Ai and Y = j=1 β j I B j , so that X Y = i j γi j I Ai ∩B j , where γi j = αi β j , if Ai ∩ B j = , and whatever; e.g., 0, if Ai ∩ B j = , and E(X Y ) = γi j P(Ai ∩ B j ) = αi β j P(Ai ∩ B j ) i

j

= =

i

j

i

j

αi β j P(Ai )P(B j )

αi P(Ai ) × β j P(B j ) = (E X )(EY ). i

j

Next, let X and Y be ≥ 0 r.v.s and let n → ∞ in the remainder of this proof. Then there exist simple r.v.s X n and Yn such that 0 ≤ X n ↑ X and 0 ≤ Yn ↑ Y , so that 0 ≤ X n Yn ↑ X Y . By the Lebesgue Monotone Convergence Theorem, we have then: E X n → E X , EYn → EY , so that (E X n )(EYn ) → (E X )(EY ), and E(X n Yn ) → E(X Y ). However, E(X n Yn ) = (E X n )(EYn ) by the previous step. Thus, E(X Y ) = (E X )(EY ) < ∞. Finally, for any r.v.s X and Y , we have X Y = (X + − X − )(Y + − Y − ) = X + Y + − X + Y − − X − Y + + X − Y − , and all expectations E X + , E X − , EY + , EY − are finite by the assumption that E |X | < ∞ and E |Y | < ∞. Then, by the previous step, all expectations E(X + Y + ), E(X + Y − ), E(X − Y + ), E(X − Y − ) are finite, and therefore E(X Y ) = E(X + Y + ) − E(X + Y − ) − E(X − Y + ) + E(X − Y − ) = (E X + )(EY + ) − (E X + )(EY − ) −(E X − )(EY + ) + (E X − )(EY − ) = (E X + )(EY + − EY − ) − (E X − )(EY + − EY − ) = (EY + − EY − )(E X + − E X − ) = (E X )(EY ) finite.


Exercises. 1. If the events A1 , . . . , An are independent, then so are the events A1 , . . . , An where Ai is either Ai or Aic , i = 1, . . . , n. Hint: The proof is done by double induction. See also Theorem 6 in Chapter 2 in Roussas (1997). 2. Consider the measurable space (, A), let Fi , i = 1, 2, be fields with Fi ⊆ A, i = 1, 2, and define F by F = {all finite unions of As with A ∈ A; A = A1 ∪ A2 or A = A1 ∩ A2 , Ai ∈ Fi , i = 1, 2}. Then show that F is a field. 3. For any real numbers p1 , . . . , pn such that 0 ≤ pi ≤ 1, i = 1, . . . , n, show that n n n 1 − exp − pi ≤ 1 − (1 − pi ) ≤ pi , m = 1, . . . , n. i=m

i=m

i=m

Hint: For the left-hand side, use the inequality e x ≥ 1 + x, x ∈ , and for the right-hand side employ the induction method. 4. Let An , n = 1, 2, . . ., be independent events in the probability space (, A, P). Then show that

(i) ∞ n=1 P(An ) = ∞ if and only if P(lim supn→∞ An ) = 1. ∞ (ii) n=1 P(An ) < ∞ if and only if P(lim supn→∞ An ) = 0. Hint: For part (i), use the left-hand side inequality in Exercise 3 here, and Exercise 3 in Chapter 3. For part (ii), use part (i) and Exercise 3 in Chapter 3 again. Remark: This result is referred to as the Borel Zero-One Criterion. 5. Let An , n = 1, 2, . . ., be independent events in the probability space (, A, P), and suppose that limn→∞ An exists, call it A. Then show that P(A) = 0 or P(A) = 1. Hint: Use Exercise 4. 6. Let X n , n ≥ 1, be independent r.v.s distributed as B(1, p), and set X¯ n =

n P a.s. n −1 i=1 X i . Then show that X¯ n → p and X¯ k 2 → p. n→∞

k→∞

Hint: For the second conclusion, use Exercise 4(i) here and Theorem 4 in Chapter 3. 7. (i) If X n , n = 1, 2, . . ., are independent r.v.s defined on the probability space

a.s. (, A, P), show that X n → 0 if and only if ∞ n=1 P(|X n | ≥ ε) < ∞ for n→∞ every ε > 0. (ii) Reconcile this result with the result in Exercise 4 in Chapter 3. Hint: Use Exercise 4(ii) here and Theorem 4 in Chapter 3. 8. Refer to Exercise 11 in Chapter 9 and construct a concrete example, by means of independent r.v.s (e.g., Binomially distributed r.v.s), to demonstrate the correctness of the assertion made there.

189

190


9. The r.v.s X 1 , . . . , X k are independent if and only if in relation (10.4) all B j s are replaced by intervals (a j , b j ] with a j , b j in and a j < b j , j = 1, . . . , k. Hint: Work as in Lemma 1 in order to show that P(X ≤ x1 , a j < X j ≤ b j , j = 2, . . . , k) = P(X ≤ x1 )

k

P(a j < X j ≤ b j ),

j=2

and complete the factorization by replacing one of the remaining X j , j = 2, . . . , k, at a time. 10. Let X and Y be independent r.v.s and suppose that E X exists. For every B ∈ B, let A = Y −1 (B) and show that A X d P = (E X )P(A). 11. Show that a r.v. X is independent of itself if and only if P(X = c) = 1 for some (finite) constant c. 12. (i) For two r.v.s with finite second moments, it follows that, if X and Y are independent, then they are uncorrelated (i.e., ρ(X , Y ) = 0, or equivalently, Cov(X , Y ) = 0). Justify this statement. (ii) For the case that the r.v.s X and Y have the Bivariate Normal distribution, use Exercise 14(iii) in Chapter 9 in order to show that, if X and Y are uncorrelated, then they are independent. 13. If the r.v.s X 1 and X 2 are independent, then show that Eeit(X 1 +X 2 ) = Eeit X 1 × Eeit X 2 ,

t ∈ ,

and by induction, Eeit(X 1 +···+X k ) = Eeit X 1 × · · · × Eeit X k ,

t ∈ ,

for the independent r.v.s X 1 , . . . , X k . Hint: Write eit(X 1 +X 2 ) = eit X 1 × eit X 2 = [cos(t X 1 ) + i sin(t X 1 )] × [cos(t X 2 ) + i sin(t X 2 )] and use Proposition 1. 14. (i) If the r.v.s X and Y are independent, distributed as N (0, σ 2 ), and U = X + Y , V = X − Y , then show that U and V are independent, distributed as N (0, 2σ 2 ), by transforming the joint p.d.f. of X and Y into the joint p.d.f. of U and V . (ii) If the r.v.s X and Y are independent, distributed as N (μ1 , σ 2 ) and N (μ2 , σ 2 ), respectively, use part (i) in order to show that U and V are independent, distributed as N (μ1 + μ2 , 2σ 2 ) and N (μ1 − μ2 , 2σ 2 ), respectively. 15. Consider the probability space (, A, P), and let A1 , . . . , An be independent events with P(Ak ) = p, k = 1, . . . , n. Next, define the function X : → as follows: X (ω) = the number of A1 , . . . , An containing ω. Then show that (i) X is a r.v. (ii) The distribution of X is B(n, p) (i.e., Binomial with parameters n and p).


16. Consider the probability space (, A, P), let {An }, n ≥ 1, be a sequence of events, and set X n = I An . Then show that the events {A1 , A2 , . . .} are independent if and only if the r.v.s X 1 , X 2 , . . . are independent. 17. If X ∼ B(n, p), compute the probability that P(number of Hs = number of Ts + r ), r = 0, 1, . . . , n. 18. Let X 1 , . . . , X n be independent identically distributed (i.i.d.) r.v.s defined on the probability space (, A, P) and having d.f. F. Let Fn be the empirical d.f. defined in terms of the X i s; i.e., 1 Fn (x, ω) = [number of X 1 (ω), . . . , X n (ω) ≤ x]. n de f

Then show that sup{|Fn (x, ·) − F(x)|; x ∈ } = Dn (·) is a r.v. That is, although Dn (·) is arrived at through noncountable operations, it is still a r.v. de f Hint: Define Dn+ and Dn− by: Dn+ = Dn+ (·) = supx∈ [Fn (x, ·) − F(x)], Dn− = de f

Dn− (·) = supx∈ [F(x) − Fn (x, ·)], so that Dn = max{Dn+ , Dn− }. Next, show that Dn+ = max{max1≤i≤n [ ni − F(yi )], 0}, and Dn− = max{max1≤i≤n [F(yi − 0) − i−1 n ], 0}, where yi = x (i) , i = 1, . . . , n, and x (1) ≤ x (2) ≤ · · · ≤ x (n) are the ordered xi s, xi = X i (ω), and ω is an arbitrary but fixed ω ∈ . 19. (Glivenko–Cantelli). Refer to Exercise 18 and show that a.s.

sup{|Fn (x, ω) − F(x)|; x ∈ } −→ 0. n→∞

Hint: For 0 < p < 1, define x p by: x p = inf{x ∈ ; F(x) ≥ p}, so that F(x) ≥ p for x ≥ x p , and F(x) < p for x < x p , which implies F(x p − 0) ≤ p. Next, replace p by i/k (k ≥ 2 integer), i = 0, 1, . . . , k, to get the points xki , with −∞ ≤ xk0 < xk1 , and xk,k−1 < xkk ≤ ∞. Then, for x ∈ [ ki , i+1 k ), i = 0, 1, . . . , k − 1, it holds that i i +1 ≤ F(xki ) ≤ F(x) ≤ F(xk,i+1 − 0) ≤ , k k so that F(xk,i+1 −0)− F(xki ) ≤ k1 . Use this result and the nondecreasing property of F and Fn to obtain, for x ∈ and i = 0, 1, . . . , k: 1 Fn (x) − F(x) ≤ [Fn (xk,i+1 − 0) − F(xk,i+1 − 0)] + , k 1 Fn (x) − F(x) ≥ [Fn (xki ) − F(xki )] − , k so that |Fn (x) − F(x)| ≤ max{|Fn (xk,i+1 − 0) − F(xk,i+1 − 0)|, 1 |Fn (xki ) − F(xki )|; i = 0, 1, . . . , k − 1} + . k Finally, take the sup over x ∈ (which leaves the right-hand side intact), and use the SLLN to each one of the (finitely many) terms on the right-hand side to arrive at the asserted conclusion.

191

CHAPTER

Topics from the Theory of Characteristic Functions

11

This chapter is a rather extensive one consisting of nine sections. The main theme of the chapter is the introduction of the concept of a characteristic function (ch.f.) and the study of some of its properties, as well as some of the ways it is used for probabilistic purposes. It is to be emphasized that ch.f.s are not studied, to the extent they are, for their own sake; rather, they are looked upon as a powerful tool for the purpose of obtaining certain probability results. A brief description of the sections is as follows. In the first section, the concept of the ch.f. of a d.f. is defined and some of its basic properties are established. In the following section, the so-called inversion formula is proven in several forms. The significance of this formula is that it allows recovery of the distribution by means of its ch.f. The application of this inversion formula is illustrated by way of two simple examples. One of the basic convergences in probability is convergence in distribution, which, in practice, is not easy to check directly. The so-called Paul Lévy Continuity Theorem, which is the main result in Section 11.3, replaces convergence in distribution by convergence of ch.f.s; this latter convergence is much easier to handle. Convergence in distribution in higher than one-dimensional spaces is in essence replaced by convergence in distribution in the real line. This is done by way of the Cramér-Wold device, which is discussed in Section 11.4. The convolution of two d.f.s, its interpretation, and several related results are discussed in Section 11.5, whereas some technical properties of ch.f.s are studied in the following section. An application of some of these results yields the Weak Law of Large Numbers and the Central Limit Theorem; this is done in Section 11.7. The basic result discussed in the next section is that, under certain regularity conditions, the moments of a distribution uniquely determine the distribution. For its rigorous justification certain concepts and results from complex analysis are required, which are dealt with in the final section of the chapter.

11.1 Definition of the Characteristic Function of a Distribution and Basic Properties In all that follows, d.f.s are nonnegative, nondecreasing, right-continuous functions with finite variations; it is not assumed that the variations are necessarily bounded by 1 unless otherwise stated (see also Exercises 4 and 5 in Chapter 8). An Introduction to Measure-Theoretic Probability, Second Edition. http://dx.doi.org/10.1016/B978-0-12-800042-7.00011-6 Copyright © 2014 Elsevier Inc. All rights reserved.

193

194

CHAPTER 11 Topics from the Theory of Characteristic Functions

Definition 1. The characteristic function f of a d.f. F (in the sense of Definition 1 in Chapter 8; see also Remark 5 there) is, in general, a complex-valued function defined on by it x e d F(x) = cos t xd F(x) + i sin t xd F(x). (11.1) f (t) =

The integration in (11.1) is to be understood either in the sense of Riemann–Stieltjes, or as integration with respect to the measure induced by F (see also Appendix B). The integral is well defined for all t ∈ , since cos t x and sin t x are F-integrable. If F is the d.f. of a r.v. X , then (11.1) may be rewritten as f X (t) = Eeit X = E cos t X + iE sin t X .

(11.2)

Some basic properties of a ch.f. are gathered next in the form of a theorem. Theorem 1. (i) | f (t)| ≤ Var F, t ∈ , and f (0) = Var F. In particular, if f (0) = 1 and 0 ≤ F(x) ≤ 1, then f is the ch.f. of a r.v. (ii) f is uniformly continuous in . (iii) If f is the ch.f. of a r.v. X , then f α X +β (t) = eiβt f X (αt), t ∈ , where α and β are constants. (iv) If f is the ch.f. of a r.v. X , then f −X (t) = f X (t), t ∈ , where, for z = x + i y(x, y ∈ ), z¯ = x − i y. dn (v) If for some positive integer n the nth moment E X n is finite, then dt n f X (t)|t=0 = inE Xn. Remark 1. In the proof of the theorems, as well as in other cases, the following property is used: [g(x) + i h(x)]dμ ≤ |g(x) + i h(x)| dμ = [g 2 (x) + h 2 (x)]1/2 dμ,

where g and h are real-valued functions, and [g(x) + i h(x)]dμ = i h(x)dμ. Its justification is left as an exercise (see Exercise 1). Proof of Theorem 1.

g(x)dμ +

For convenience omit in the integration. Then

(i) | f (t)| = | eit x d F(x)| ≤ |eit x |d F(x) = Var F, and f (0) = d F(x) = Var F. If f (0) = 1, then Var F = 1, which together with 0 ≤ F(x) ≤ 1, x ∈ , implies F(−∞) = 0, F(∞) = 1, so that F is the d.f. of a r.v. (ii) | f (t + h) − f (t)| = | ei(t+h)x d F(x) − eit x d F(x)| = | [ei(t+h)x − eit x ] d F(x)| = | [eit x (ei hx − 1)]d F(x)| ≤ |eit x (ei hx − 1)|d F(x) = |ei hx − 1|d F(x). Now |ei hx − 1| ≤ 2, which is independent of h and F-integrable. Furthermore, ei hx − 1 → 0 as h → 0. Therefore the Dominated Convergence

11.2 The Inversion Formula

Theorem applies and gives |ei hx − 1|d F(x) → 0 as h → 0. So | f (t + h) − f (t)| is bounded by a quantity that is independent of t and → 0 as h → 0. This establishes uniform continuity for f . (iii) f α X +β (t) = Eeit(α X +β) = E[eiβt ei(αt)X ] = eiβt E[ei(αt)X ] = eiβt f X (αt). (iv) f −X (t) = Eeit(−X ) = Eei(−t)X = E[cos(−t X ) + i sin(−t X )] = E[cos(t X ) − i sin(t X )] = E cos t X − iE sin t X = E cos t X + iEsin t X = f X (t). (v) Consider, e.g., the interval [−r , r ] for some r > 0. Then, for t ∈ [−r , r ], ∂ it X = i X eit X exists, and | ∂t∂ eit X | ≤ |X |, independent of t and integrable. ∂t e Then, by Theorem 5 in Chapter 5, ∂ it X d d d F(x) f (t) = eit X d F(x) = e dt dt ∂t = i (X eit X )d F(x), d f (t)|t=0 = i (X eit X )|t=0 d F(x) = iE X . and, in particular, dt d k it X = i k X k eit X exists, and The same applies for any k, 1 ≤ k ≤ n, since dt ke k

d it X | dt e | ≤ |X k |, independent of t and integrable. In particular, k k k i X d F(x) = i k E X k .

dk dt k

f (t)|t=0 =

11.2 The Inversion Formula By means of relation (11.1), the d.f. F defines its ch.f. f . The converse is also true; i.e., if it is given that f is the ch.f. of a d.f. F, then F can be recovered by means of the so-called inversion formula. More precisely, we have the following result. Theorem 2.

Let F be a d.f. (not necessarily of a r.v.) and let f be its ch.f. Then

(i) For any a, b ∈ (a < b) and T > 0: F(a) + F(a−) F(b) + F(b−) − 2 2 T −ita 1 e − e−itb f (t)dt. = lim T →∞ 2π −T it

(11.3)

(ii) If a, b ∈ C(F) and T > 0: 1 T →∞ 2π

F(b) − F(a) = lim

T

−T

e−ita − e−itb f (t)dt. it

(11.4)

195

196


(iii) If X is a discrete r.v. taking on the value x j with probability P(X = x j ) = p(x j ), j ≥ 1, then T 1 p(x j ) = lim e−it x j f (t)dt, j ≥ 1, T > 0. T →∞ 2T −T In the course of the proof of the theorem, the following two facts will be needed, which are recorded here for easy reference. b Fact 1. The Dirichlet integrals a sinx x d x are bounded uniformly in a, b ∈ . 0 ∞ Fact 2. −∞ sinx x d x = 0 sinx x d x = π2 . (See, e.g., integral 417 in Tallarida (1999).) Also, the following remarks are in order. Remark 2. (i) As it will be seen in the proof of the theorem, the integrals on the right-hand sides of relations (11.3) and (11.4) are real. (ii) It is to be noticed that in (11.3) and (11.4), we consider the Cauchy principal values of the integrals (i.e., integrals over intervals symmetric with respect to the origin); the integrals taken over arbitrary limits may fail to exist. (iii) It is to be pointed out here that, whereas a d.f. F determines the corresponding ch.f. uniquely, the converse is not true. That is, a ch.f. determines a class of d.f.s through the difference F(b) − F(a), a, b ∈ C(F) (Theorem 2(ii)) rather than a unique d.f. Any two such d.f.s differ by a constant. Indeed, let F and G be two such d.f.s. Then, for x and a in C(F) ∩ C(G) with a < x, we have F(x) − F(a) = G(x) − G(a)(= the right-hand side in (11.4)). Letting a ↓ −∞ through C(F) ∩ C(G), we get F(x) − F(−∞) = G(x) − G(−∞) or F(x) − G(x) = F(−∞) − G(−∞), a constant for x ∈ . Nevertheless, all these d.f.s define the same measure (Theorem 7 in Chapter 4). In particular, if f is the ch.f. of a r.v., then the corresponding measure is a probability measure (Theorem 1(i) here). Proof of Theorem 2. (i) Set g(t) =

e−ita − e−itb , it

J (T ) =

1 2π

T

−T

g(t) f (t)dt.

(11.5)

Expanding e−ita and e−itb around 0, up to terms of second order, dividing by t( = 0), and taking the limit as t → 0, we obtain limt→0 g(t) = b − a. Then define by continuity, g(0) = b − a. (See also Exercise 2.) Since b e−ita − e−itb = e−it y dy (both for t = 0 and t = 0), and it a ∞ f (t) = eit x d F(x), −∞


we have 1 J (T ) = 2π =

1 2π

=

1 2π

T

−T T

b

a ∞

e

−it y

b

−T −∞ a ∞ b T −∞ a

−T

dy

∞

−∞

e

it x

d F(x) dt

eit(x−y) dyd F(x)dt eit(x−y) dtdyd F(x),

(11.6)

where the change of the order of the integration is allowed by the Fubini Theorem (Theorem 12 in Chapter 5), since |eit(x−y) | = 1 is integrable with respect to the d.f. F (over (−∞, ∞)), the Lebesgue measure (over (a, b)), and the Lebesgue measure again (over (−T , T )). Next, for x = y: T T 1 eit(x−y) dt = deit(x−y) i(x − y) −T −T 1 = [ei T (x−y) − e−i T (x−y) ]. i(x − y) On the other hand, from eiv = cos v + i sin v, e−iv = cos v − i sin v, we get: eiv − e−iv = 2i sin v or 2i1 (eiv − e−iv ) = sin v, so that 1 sin[T (x − y)] [ei T (x−y) − e−i T (x−y) ] = 2T . i(x − y) T (x − y) Since sinx x → 1 as x → 0, sinx x is defined to be 1 at x = 0 by continuity. But T T for x = y : −T eit(x−y) dt = −T dt = 2T . Thus, for all x, y, relation (11.6) becomes T ∞ b sin[T (x − y)] dyd F(x). (11.7) J (T ) = π −∞ a T (x − y) b (x−y)] Next, consider the integral πT a sin[T T (x−y) dy and set T (x − y) = −u, so that dy =

du T ,u

: T (a − x), T (b − x), to get 1 T (b−x) sin u de f T b sin[T (x − y)] dy = du = K T (x), π a T (x − y) π T (a−x) u

so that by (11.7) and (11.8), we have ∞ K T (x)d F(x). J (T ) = −∞

(11.8)

(11.9)

By Fact 1, K T (x) is bounded uniformly in T , a, b, and x, and its bound (a finite constant) is F-integrable. Next, by Fact 2, ⎧ ⎫ ⎨ 0 if x < a or x > b ⎬ de f lim K T (x) = 1 if a < x < b = K (x). (11.10) ⎩1 ⎭ T →∞ if x = a or x = b 2

197

198


By the boundedness of K T (x) by a constant (independent of T ), F-integrable, and relation (11.10), the Dominated Convergence Theorem applies, and therefore (11.9) gives ∞ ∞ K T (x)d F(x) = K (x)d F(x) lim J (T ) = lim T →∞

= +

T →∞ −∞

a−

−∞ b

K (x)d F(x) +

K (x)d F(x) +

a

K (x)d F(x)

a+

∞

K (x)d F(x) +

b

−∞ b−

a

K (x)d F(x)

b+

1 = 0 + [F(a) − F(a−)] + 1 × [F(b−) − F(a)] 2 1 + [F(b) − F(b−)] + 0 2 1 1 = [F(b) + F(b−)] − [F(a) + F(a−)]. 2 2

(11.11)

Then relations (11.5) and (11.11) complete the proof of part (i). (ii) It follows from part (i), since F(a) = F(a−) and F(b) = F(b−). (iii) We have T T e−it x j f (t)dt = e−it x j eit xk p(xk ) dt −T

−T

= =

T

−T

k

p(xk )

k

k

e

it(xk −x j )

T

−T

p(xk ) dt

eit(xk −x j ) dt,

by the Fubini Theorem, because the integrand is integrable with respect to the counting measure (or the probability measure induced by { p(xk )}; k ≥ 1), and the Lebesgue measure over [−T , T ], and this is = p(x j ) × 2T + ∞ it(xk −x j ) dt. But for xk = x j , k = j −∞ e T T T it(xk −x j ) e dt = cos t(xk − x j )dt + i sin t(xk − x j )dt −T

−T T

−T

=

−T

cos t(xk − x j )dt

T 1 d sin t(xk − x j ) (xk − x j ) −T sin T (xk − x j ) − sin[−T (xk − x j )] 2 sin T (xk − x j ) = = . xk − x j xk − x j =


Therefore, the original integral becomes T 1 1 p(x j ) × 2T e−it x j f (t)dt = 2T −T 2T 2 sin T (xk − x j ) 1 + p(xk ) 2T xk − x j xk =x j

= p(x j ) +

xk =x j

p(xk )

sin T (xk − x j ) . T (xk − x j )

Set T (xk − x j ) = x and expand sin x around 0 up to terms of order one to obtain sin x = sin 0 + (x − 0) cos x|x=x ∗ (for some x ∗ = θ x, |θ | ≤ 1) = x cos x ∗ , so that T (xk − x j ) cos x ∗ sin T (xk − x j ) = = cos x ∗ , and T (xk − x j ) T (xk − x j ) sin T (xk − x j ) ∗ T (x − x ) = | cos x | ≤ 1 k

j

independent of T and integrable (with respect to the measure { p(xk ), xk = x j }). Therefore, by the Dominated Convergence Theorem, lim

T →∞

p(xk )

xk =x j

sin T (xk − x j ) T (xk − x j )

sin T (xk − x j ) =0 T (xk − x j ) xk =x j sin T (xk − x j ) 1 ≤ since −→ 0 T (xk − x j ) |T (xk − x j )| T →∞ T −it x 1 j f (t)dt = p(x ). and hence lim T →∞ 2T j −T e =

p(xk ) lim

T →∞

To this theorem, there are the following three corollaries. Let a = x − h and b = x + h (h > 0) be continuity points of F. Then 1 T sin(th) −it x e f (t)dt. F(x + h) − F(x − h) = lim T →∞ π −T t

Corollary 1.

Proof.

For this choice of a and b, we have e−ita − e−itb = e−it(x−h) − e−it(x+h) = e−it x (eith − e−ith ) = e−it x × 2i sin(th),

199

200


so that 1 2π

T −T

1 e−ita − e−itb f (t)dt = it π

T

−T

sin(th) −it x e f (t)dt. t

The result follows from part (ii). Corollary 2. given by

The d.f. F is differentiable and its derivative at a, 1 h→0 T →∞ 2π

p(a) = lim lim

T −T

F (a)

1 − e−ith −ita e f (t)dt ith

= p(a), is (11.12)

if and only if the right-hand side in (11.12) exists. Proof. In part (ii) of the theorem, set b = a + h (with both a and a + h continuity points of F, h > 0). Then e−ita − e−itb = e−ita − e−it(a+h) = e−ita (1 − e−ith ), and hence 1 1 [F(a + h) − F(a)] = lim T →∞ 2π h

T −T

1 − e−ith −ita e f (t)dt. ith

(11.13)

Suppose first that p(a) exists. Then, taking the limits in (11.13), as h → 0, we have the desired result. Next, if the limit (as h → 0) on the right-hand side of (11.13) exists, then p(a) exists and is taken as stated. Similarly for h < 0. ∞ Corollary 3. If −∞ | f (t)|dt < ∞, then the derivative F (x) = p(x) exists, is bounded and continuous in , and is given by ∞ 1 p(x) = e−it x f (t)dt. (11.14) 2π −∞ From the expansion e−ith = 1 − ithe z , with z = −ithθ, θ real, |θ | ≤ 1,

Proof.

1−e−ith z we have ith = e , (t −ith Also, 1−eith ≤ |e z | =

= 0), and

1−e−ith ith

is defined to be 1 for t = 0 by continuity.

1. Thus

∞ 1 − e−ith −ita 1 − e−ith −ita e e f (t)dt = f (t)I[−T ,T ] (t)dt ith ith −T −∞ −ith with 1−eith e−ita f (t)I[−T ,T ] (t) ≤ | f (t)| independent of T and (Lebesgue-) integrable, and 1 − e−ith −ita 1 − e−ith −ita lim e e f (t)I[−T ,T ] (t) = f (t). T →∞ ith ith T

Then the Dominated Convergence Theorem applies and gives ∞ T 1 − e−ith −ita 1 − e−ith −ita lim e e f (t)dt = f (t)dt. T →∞ −T ith ith −∞

(11.15)


In the integrand on the right-hand side of (11.15), look at h as an index, and observe that −ith | 1−eith e−ita f (t)| ≤ | f (t)| independent of h, (Lebesgue-) integrable, and 1−e−ith −ita ith e

f (t) → e−ita f (t). Therefore, the Dominated Convergence Theorem h→0

again gives

lim

∞

h→0 −∞

1 − e−ith −ita e f (t)dt = ith

∞ −∞

e−ita f (t)dt.

Thus, from (11.15) and (11.16), T ∞ 1 1 − e−ith −ita 1 lim lim e f (t)dt = e−ita f (t)dt. h→0 T →∞ 2π −T ith 2π −∞

(11.16)

(11.17)

Since left-hand side of (11.17) is equal to p(a) by (11.12), have p(a) = ∞the−ita ∞ we 1 1 −it x f (t)dt. The e f (t)dt. Replacing a by x, we have p(x) = e 2π −∞ ∞ −it x 2π −∞ ∞ 1 1 f (t)dt| ≤ 2π boundedness of p(x) follows by p(x) = | 2π −∞ e −∞ | f (t)| (t)| 1 −it x dt < ∞, whereas continuity follows thus: | 2π e f (t)| ≤ | f2π independent of 1 −it x 1 −it x0 x, (Lebesgue-) integrable, and 2π e f (t) → 2π e f (t). Then the Dominated x→x0

Convergence Theorem completes the proof. Here are two examples that illustrate how the inversion formula applies. Example 1.

Let the r.v. X be distributed as B(n, p), so that its ch.f. is given by n n n itk n k n−k p q ( peit )k q n−k f (t) = e = k k k=0

k=0

= ( peit + q)n , t ∈ (q = 1 − p). Apply Theorem 2(iii) to recover p(x), x = 0, . . . , n. We have T T 1 1 −it x e f (t)dt = e−it x ( peit + q)n dt 2T −T 2T −T T n n 1 −it x it k n−k ( pe ) q = e dt k 2T −T k=0 T n n k n−k i(k−x)t 1 p q e = dt k 2T −T k=0 n 1 n k n−k T i(k−x)t p q e dt = k 2T −T k=0 n n k n−k T i(k−x)t 1 p q = e dt k 2T −T k=0,k =x 1 n x n−x p q + × 2T 2T x

201

202


n n k n−k ei(k−x)T − e−i(k−x)T p q k 2i T (k − x)

= p(x) + = p(x) +

k=0,k =x n k=0,k =x

n k n−k sin(k − x)T p q , k (k − x)T

and by taking the limit, as T → ∞, the second term on the right-hand side above T −it x 1 1 → 0 as |x| → ∞). It follows that lim T →∞ 2T tends to 0 (since | sinx x | ≤ |x| −T e f (t)dt = p(x) as in Theorem 2(iii). Example 2. Let the r.v. X be distributed as N (0, 1), so that its ch.f. is f (t) = e−t ∞ 2 Since | f (t)|dt = −∞ e−t /2 dt < ∞, Corollary 3 applies. We have then ∞ ∞ 1 1 2 −it x e f (t)dt = e−it x e−t /2 dt 2π −∞ 2π −∞ ∞ 1 2 = e−(t +2it x)/2 dt 2π −∞ ∞ 1 2 2 2 = e−[t +2it x+(i x) ]/2 e(i x) /2 dt 2π −∞ 2 e−x /2 ∞ −(t+i x)2 /2 e dt = 2π −∞ 2 e−x /2 ∞ 1 −u 2 /2 du = √ √ e 2π −∞ 2π e−x /2 1 2 = √ × 1 = √ e−x /2 = p(x). 2π 2π

2 /2

.

2

11.3 Convergence in Distribution and Convergence of Characteristic Functions—The Paul Lévy Continuity Theorem In this section, two versions of the Paul Lévy continuity theorem are stated and proved, after a number of auxiliary results have been established. The significance of this theorem is that convergence in distribution of a sequence of r.v.s to a r.v. is reduced to convergence of ch.f.s. Convergence in distribution is not easy to deal with, whereas convergence of ch.f.s is amenable to a large body of analytical facts and techniques. Below, we define the so-called integral ch.f. of a d.f. F, which is needed as a tool in the sequel. To this end, let F be a d.f. (not necessarily of a r.v.) with a ch.f. f . In terms of f , we define the function fˆ as follows: t f (v)dv, t ∈ . (11.18) fˆ(t) = 0

11.3 Convergence in Distribution and Convergence

Then fˆ(t) =

t 0

e

ivx

d F(x) dv =

t 0

e

ivx

d F(x)dv =

t

0

eivx dv d F(x)

(by the Fubini Theorem, since the integrand is integrable). Now, for x = 0, we have

t

eivx dv =

0

while for x = 0, we have

t 0

eit x = 1 + it x +

1 ix

t

0

deivx =

1 ivx t eit x − 1 e |0 = , ix ix

dv = t. Expanding e(i x)t around t = 0, we get (it x)2 ∗ × eit x , for some t ∗ with |t ∗ | ≤ 1, 2

so that eit x − 1 t2 ∗ = t + (i x) eit x . ix 2 eit x −1 ix

is defined by continuity at x = 0 (which is what we usually do), then t = t, and this is what we get as a value of 0 eivx dv for x = 0. So, for any x, we have then it x e −1 ˆ d F(x). (11.19) f (t) = ix

Thus, if

eit x −1 i x |x=0

Definition 2. The function fˆ as defined in (11.18) or (11.19) is called the integral ch.f. of the d.f. F. Remark 3. There is a one-to-one correspondence between f and fˆ. In fact, f uniquely determines fˆ by means of (11.18). On the other hand (i.e., if we are told that fˆ is the integral ch.f. of a ch.f. f and we want to recover f ), since f is continuous, we have fˆ (t) = f (t), so that fˆ uniquely determines f . The simplest (and most commonly used) version of the continuity theorem is the following. Theorem 3 (Paul Lévy Continuity Theorem). For n ≥ 1, let Fn and F be d.f.s of r.v.s with respective ch.f.s f n and f . Then, as n → ∞, c

(i) Fn ⇒ F (or, equivalently, Fn →F) implies f n → f on . c

(ii) f n → f on implies Fn ⇒ F (or, equivalently, Fn →F).

For the proof of part (ii), we need the following two auxiliary results. Lemma 1. Let f n and f be as in Theorem 3, and let fˆn and fˆ be the respective integral ch.f.s. Then, as n → ∞, f n → f on implies fˆn → fˆ on .

203

204


Proof. For t > 0 (and similarly for t < 0), | f n (v)I[0,t] (v)| ≤ I[0,t] (v) independent of n, (Lebesgue-) integrable, and f n (v)I[0,t] (v) → f (v)I[0,t] (v). Then, by the n→∞ Dominated Convergence Theorem, t t f n (v)dv = f n (v)I[0,t] (v)dv → f (v)I[0,t] (v)dv = f (v)dv, n→∞

0

0

or fˆn → fˆ on .

Lemma 2. Let Fn , F and f n , f be as in Theorem 3, and let fˆn and fˆ be the respective integral ch.f.s of f n and f . Then, as n → ∞, fˆn → fˆ on implies Fn ⇒ F (or, c equivalently, Fn →F). In the course of the proof of this lemma, as well as elsewhere, the following elementary fact is employed, which is stated here as a remark. Remark 4. If {z n }, n ≥ 1, is a sequence of (real or complex) numbers, then z n → z 0 as n → ∞, if and only if for any subsequence {m} ⊆ {n} there exists a further subsequence {r } ⊆ {m} such that zr → z 0 as r → ∞. Proof of Lemma 2. In this proof, all limits are taken as {n} or subsequencies thereof converge to ∞. To show that Fn (x) → F(x), x ∈ C(F). By Remark 4, it suffices to show that for every {m} ⊆ {n} there exists {r } ⊆ {m} such that Fr (x) → F(x), x ∈ C(F). Since {Fm } (evaluated at x ∈ C(F)) is bounded (by 1), there exists {r } ⊆ {m} such that Fr ⇒ F0 , some d.f. on . (This is so by Theorem 5 in Chapter 8.) ˆ Clearly, 0 ≤ F0 (x) ≤ 1, x ∈ , and let f0 and f 0 be the ch.f. and the integral ch.f. of F0 , respectively. We have fˆr (t) = (eit x − 1)/i x d Fr (x) with (eit x − 1)/i x continuous (in x) over (for each arbitrary but fixed t ∈ ) and |(eit x − 1)/i x| ≤ 2/|x| → 0 as |x| → ∞. Also, Fr ⇒ F0 . Then (by Theorem 7 in Chapter 8), it x it x e −1 e −1 d Fr (x) → d F0 (x), i x ix or fˆr → fˆ0 on . However, fˆn → fˆ on , so that fˆr → fˆ and fˆ0 = fˆ on . It follows that f 0 = f on , and therefore F0 − F = c, some constant c (see also Remark 2(iii)). We shall show that c = 0, which will establish the assertion that Fn ⇒ F. Indeed, from f 0 = f on , we have f 0 (0) = f (0), or V ar F0 = V ar F = 1. So, 0 ≤ F0 (x) ≤ 1, x ∈ , and V ar F0 = 1. Then F0 (−∞) = 0 and F0 (∞) = 1. Finally, from F0 (x) − F(x) = c, we get, as x → −∞, F0 (−∞) − F(−∞) = c, or 0 − 0 = c; i.e., c = 0 and F0 = F. Proof of Theorem 3. (i) For each arbitrary t ∈ , eit x is continuous (in x) and bounded (by 1). Since c

Fn →F, Theorem 8 in Chapter 8 applies and gives that eit x d Fn (x) → eit x d F(x) = f (t), f n (t) =

or f n → f on .


(ii) It follows from Lemmas 1 and 2. (Or, in more detail, from f n → f or , we have fˆn (t) =

t 0

f n (v)dv =

f n (v)I[0,t] (v) dv

with | f n (v)I[0,t](v) | ≤ I[0,t] (v) (since | f n (v)| ≤ V ar Fn = 1) independent of n and (Lebesgue-) integrable. Also, f n (v)I[0,t] (v) → f (v)I[0,t] (v). Hence f n (v)I[0,t] (v) dv → f (v)I[0,t] (v) dv = fˆ(t); fˆn (t) =

i.e.; fˆn → fˆ on . Next, in order to show that Fn ⇒ F, suffices to show that for every {Fm } ⊆ {Fn }, there is {Fr } ⊆ {Fm } such that Fr ⇒ F. To this end, let {Fm } ⊆ {Fn }. Then there is {Fr } ⊆ {Fm } such that Fr → F0 , some d.f. This is so by Theorem 6 in Chapter 8. Let f 0 and fˆ0 be the ch.f. and the integral ch.f., respectively, of F0 . Then Fr ⇒ F0 implies fˆr → f 0 on , by part (i). However, fˆn → fˆ, by what it was just proved. Hence fˆr → fˆ, and therefore fˆ = fˆ0 , which implies f = f 0 . But then F0 − F = c, a constant. To show that c = 0. Indeed, 0 ≤ F0 ≤ 1 (since 0 ≤ Fn ≤ 1 for all n), and 1 = f (0) = f 0 (0) implies that V ar F0 = 1. Then F0 (−∞) = 0. Since F0 (x) − F(x) = c, x ∈ , letting x ↓ −∞, we get 0 − 0 = c, so that F0 = F, and therefore Fn ⇒ F.) A version of Theorem 3, whose part (ii) is seemingly less restrictive than part (ii) of Theorem 3, is the following. Theorem 3* (Paul Lévy Continuity Theorem). For n ≥ 1, let Fn be d.f.s of r.v.s with respective ch.f.s f n . Then (i) If Fn ⇒ F, a d.f. of a r.v. with ch.f. f , it follows that f n → f on , as n → ∞. (ii) Let f n → g, some function on continuous at the origin. Then Fn =⇒ F, n→∞ n→∞ where F is a (uniquely determined) d.f. of a r.v. Remark 5. Clearly, part (i) is the same in both Theorems 3 and 3*. In part (ii) of Theorem 3*, it is not required that the limit g of { f n } be a ch.f. at all (even less the ch.f. of a r.v.), but it turns out that g is, indeed, the ch.f. of a r.v. and that Fn =⇒ F, n→∞ the uniquely determined d.f. corresponding to f . For the proof of part (ii) of Theorem 3*, we need the following auxiliary results. Lemma 3. Let f n and g be as in Theorem 3*, and let fˆn be the integral ch.f. ˆ corresponding to t f n . Then, as n → ∞, f n → g on implies that f n → gˆ on , where g(t) ˆ = 0 g(v)dv (where the integral is to be understood in the Lebesgue sense). Proof. It is the same as that of Lemma 1, where f and fˆ are replaced by g and g, ˆ respectively.

205

206


Before the second auxiliary result is formulated, we have to introduce the concept of weak (and complete) convergence of a sequence of d.f.s to a d.f. up to an additive constant. At this point, recall that, if Fn , n ≥ 1, and F are d.f.s (not necessarily of r.v.s), then weak convergence Fn ⇒ F (as n → ∞) means Fn (x) → F(x), x ∈ C(F). In this section, the need for a modification of this convergence arises. This is due to the fact that there is no one-to-one correspondence between d.f.s and the corresponding ch.f.s (although there is such one-to-one correspondence between the measures induced by the d.f.s and the respective ch.f.s). From relation (11.1), it follows that it is, actually, an entire class of d.f.s that induce the same ch.f.; namely, a d.f. F and any d.f. F + c, c ∈ , constant, for which F(x) + c ≥ 0, and any two members of this class differ by a constant. On the other hand, by means of the inversion formula, a ch.f. defines a class of d.f.s any two of which differ by a constant. So, for a d.f. F, all d.f.s F + c, c ∈ such that F(x) + c ≥ 0, x ∈ , have the same ch.f., and a given ch.f. f determines a class of d.f.s of the form just described. These observations lead to the following definition. Definition 3. For n ≥ 1, let Fn and F be d.f.s (not necessarily of r.v.s). We say that {Fn } converges weakly to F up to an additive constant, and we write Fn =⇒ F n→∞ uac, if for every {n 1 } ⊆ {n} and {n 2 } ⊆ {n} with Fn 1 =⇒ F1 and Fn 2 =⇒ F2 , d.f.s, it n→∞

n→∞

holds that F1 − F2 = c, some constant. Also, we say that {Fn } converges completely c to F uac, and write Fn → F uac, if Fn =⇒ F uac and V ar Fn −→ V ar F. (The c

n→∞

n→∞ c

n→∞

convergence Fn −→ F, trivially, implies Fn −→ F uac.) n→∞

n→∞

Remark 6. (i) If F0 is the limiting d.f. of a (weakly) converging subsequence of {Fn }, n ≥ 1, then any other limiting d.f. F is of the form F = F0 + c, some c ∈ . (ii) Let F0 and F be as in part (i). Then, if F(−∞) = F0 (−∞) = 0, it follows that F(x) = F0 (x), x ∈ . Consequently, if F0 is the d.f. of a r.v., then so is F. Indeed, F(x) = F0 (x) + c, and as x → −∞, F(−∞) = F0 (−∞) + c, or 0 = 0 + c, and c = 0. (iii) Clearly, all the limiting d.f.s F determine the same ch.f. f . c

c

n→∞

n→∞

(iv) If Fn =⇒ F, then, trivially, Fn =⇒ F uac, and if Fn −→ F, then Fn −→ F uac. n→∞

n→∞

Lemma 4. For n ≥ 1, let Fn be d.f.s of r.v.s with respective ch.f.s and integral ch.f.s t f n and fˆn . Let gˆ be as in Lemma 3 (i.e., g(t) ˆ = 0 g(v)dv, t ∈ , with g defined on and being continuous at the origin), so that fˆn → gˆ in . Then Fn =⇒ F uac, some d.f. F with ch.f. f , and fˆ = gˆ on .

n→∞

Proof. In the proof, all limits are taken as {n} or subsequences thereof tend to ∞. Let {n 1 } and {n 2 } be any subsequences of {n} such that Fn 1 ⇒ F1 and Fn 2 ⇒ F2 , some d.f.s. (Such subsequences exist, by Theorem 5 in Chapter 8, since the Fn s are bounded (by 1).) Let f 1 , f 2 , and fˆ1 , fˆ2 be the respective ch.f.s and integral ch.f.s of F1 and F2 . As in the proof of Lemma 2, fˆn i → fî on , i = 1, 2, and since


ˆ on , and hence f 1 = f 2 on . fˆn → gˆ on , we conclude that fˆ1 = fˆ2 (= g) Thus, all (weakly) convergent subsequences of {Fn } determine the same ch.f., call it f . Therefore F1 − F2 = c, so that Fn ⇒ F uac, where the d.f. F is in the class of d.f.s determined by f . Finally, fˆ = fˆ1 = fˆ2 = gˆ from above. Proposition 1. For n ≥ 1, let Fn be d.f.s of r.v.s with respective ch.f.s and integral f n and fˆn , let g be a function on continuous at the origin, and let g(t) ˆ = ch.f.s t g(v)dv, t ∈ (where the integral is to be understood in the Lebesgue sense). 0 Suppose that f n −→ g on . Then Fn =⇒ F, a (uniquely determined) d.f. of a r.v. n→∞

n→∞

Proof. With all limits taken as n → ∞, we have that f n → g on implies fˆn → gˆ on (by Lemma 3), and this, in turn, implies (by Lemma 4) that Fn ⇒ F uac, some d.f. with ch.f. f and fˆ = g. ˆ From fˆ = gˆ on , we have t t 1 t 1 t f (v)dv = g(v)dv, t ∈ , or f (v)dv = g(v)dv, t = 0. t 0 t 0 0 0 Taking the limits, as t → 0, we get f (0) = g(0) (by continuity at 0; see also Exercise 3). From f n → g on , we have 1 = f n (0) → g(0), so that g(0) = f (0) = 1. The d.f.s F1 and F2 in Lemma 4 take values in [0, 1], as they are limits of such sequences. So, 0 ≤ Fi (x) ≤ 1, x ∈ , and V ar Fi = f (0) = 1, i = 1, 2. It follows Fi (−∞) = 0, so that F1 (x) − F2 (x) = c yields, as x → −∞, 0 − 0 = c. Then the limiting d.f. F in Lemma 4 is uniquely determined, and is the d.f. of a r.v. Proof of Theorem 3*. All we have to do is to justify part (ii). However, this is the conclusion of Proposition 1. Convergence in distribution is preserved under continuity, as the following example shows. Example 3.

d

Let X 1 , X 2 , . . ., and X be r.v.s such that X n −→ X , and let g : → n→∞

d

be continuous. Then g(X n ) −→ g(X ). n→∞

Indeed, by Theorem 3, it suffices to show that f g(X n ) (t) −→ f g(X ) (t). However, n→∞

f g(X n ) (t) = Eeitg(X n ) = E cos[tg(X n )] + iE sin[tg(X n )] = cos[tg(x)]d FX n (x) + i sin[tg(x)]d FX n (x) cos[tg(x)]d FX (x) + i sin[tg(x)]d FX (x) −→ n→∞

(by Theorem 8 in Chapter 8, since the integrands are bounded and continuous on ) = Eeitg(X ) = f g(X ) (t). c

The foregoing convergence is valid, because FX n −→ FX , and cos[tg(x)] and n→∞

sin[tg(x)] are bounded and continuous on , so that Theorem 8 in Chapter 8 applies.

207

208


d

As a simple application, we have that X n −→ Z

∼

n→∞

N (0, 1) implies

d

X n2 −→ Z 2 whose distribution is the so-called chi-square with one degree of free-

n→∞ dom, Z 2

∼ χ12 . It is of some importance to observe that suitable versions of the results incorporated in the Lemmas 1, 2, 3, 4, and Proposition 1 also hold under less restrictive conditions. This is the content of the following proposition.

Proposition 2. With n ≥ 1, let Fn be (uniformly) bounded d.f.s (not necessarily of r.v.s) with ch.f.s f n and integral ch.f.s fˆn . Then (i) If Fn =⇒ F uac, some d.f. F with ch.f. f and integral ch.f. fˆ, it follows that n→∞

fˆn → fˆ on . n→∞

ˆ some function on , it follows that there exists a d.f. F (not nec(ii) If fˆn → g, n→∞

essarily of a r.v.) with ch.f. f and integral ch.f. fˆ such that Fn =⇒ Fuac and n→∞

fˆ = gˆ on .

With all limits taken as {n} or subsequences thereof tend to ∞, we have

Proof.

(i) In order to show that fˆn → fˆ on , it suffices to show (by Remark 4) that for every {m} ⊆ {n} there exists {r } ⊆ {m} such that fˆr → fˆ on . Indeed, by looking at {Fm }, there exists a subsequence {Fr } ⊆ {Fm } (by the uniform boundedness of the Fn s and Theorem 5 in Chapter 8) such that Fr ⇒ F, some d.f. with ch.f. f and integral ch.f. fˆ. Then it x it x e −1 e −1 ˆ d Fr (x) → d F(x) = fˆ(r ), fr (t) = ix ix as in the proof of Lemma 2, or fˆr → fˆ on . It follows that fˆn → fˆ on . (ii) For any {n 1 } ⊆ {n} and {n 2 } ⊆ {n} with Fn 1 ⇒ F1 and Fn 2 ⇒ F2 , some d.f.s F1 and F2 , one has (by part (i)) that fˆn 1 → fˆ1 and fˆn 2 → fˆ2 on , where fˆ1 and fˆ2 are the integral ch.f.s corresponding to F1 and F2 . Since also fˆn 1 → gˆ ˆ and hence f 1 = f 2 for the and fˆn 2 → gˆ on , it follows that fˆ1 = fˆ2 (= g), respective ch.f.s. Then F1 − F2 = c, some constant c, and hence Fn ⇒ F uac, some d.f. F with ch.f. f (= f 1 = f 2 ). Corollary. For n ≥ 1, let Fn , f n and fˆn be as in the proposition, and suppose that f n → g a.e., (with respect to Lebesgue measure) on . Then Fn =⇒ Fuac, some n→∞ n→∞ d.f. F (not necessarily of the r.v.) with ch.f. f , and f = g a.e., Proof. For n ≥ 1 and t ∈ , | f n (t)| are uniformly bounded by a constant, and f n → g a.e., where here and in the sequel all limits are taken as n → ∞. Then t t t def f n (v)dv → g(v)dv, or fˆn (t) → g(t) ˆ = g(v)dv, t ∈ . 0

0

0


Then, by part (ii) of the proposition, Fn ⇒ F uac, some d.f. F with ch.f. f and integral ch.f. fˆ, and fˆ = gˆ on . From t t ˆ f (v)dv and g(t) ˆ = g(v)dv, f (t) = 0

0

it follows that fˆ = f on , and gˆ = g a.e., (see, e.g., Theorem 10 on page 107 of Royden (1988)). However, fˆ = gˆ on . Hence f = g a.e. Proposition 3. With n ≥ 1, let Fn be (uniformly) bounded d.f.s (not necessarily of r.v.s) with ch.f.s f n and integral ch.f.s fˆn . Then c

(i) If Fn → F uac, some d.f. F with ch.f. f , it follows that f n → f on . n→∞

n→∞

(ii) If f n → g, some function on continuous function at the origin, it follows that n→∞

c

there exists a d.f. F (not necessarily of a r.v.) with ch.f. f such that Fn → F n→∞ uac and f = g on . Proof.

With all limits taken as {n} or subsequences thereof tending to ∞, we have

(i) It suffices to prove that, for every {m} ⊆ {n}, there exists {r } ⊆ {m} such that fr → f on . Looking at {Fm }, there exists {Fr } ⊆ {Fm } such that Fr ⇒ F, some d.f. F with ch.f. f . Since for each arbitrary and fixed t ∈ , eit x is bounded and continuous in (as a function of x), it follows (by Theorem 8 in Chapter 8) that it x e d Fr (x) → eit x d F(x) = f (t), t ∈ , fr (t) =

so that f n → f on . (ii) For n ≥ 1 and t ∈ , | f n (t)| are uniformly bounded by a constant, and f n → g on . Then t t def f n (v)dv → g(v)dv = g(t), ˆ t ∈ . fˆn (t) = 0

0

Therefore, by part (ii) of Proposition 2, it follows that there exists a d.f. F (not necessarily of a r.v.) with ch.f. f and integral ch.f. fˆ such that Fn ⇒ F uac and fˆ = gˆ on . That is, t t 1 t 1 t f (v)dv = g(v)dv, t ∈ , or f (v)dv = g(v)dv, t = 0. t 0 t 0 0 0 By taking the limits as t → 0, we have then (see also Exercise 3) f (0) = g(0). Since f n → g on , we have f n (0) → g(0), or V ar Fn = f n (0) → g(0) = c f (0) = V ar F. Therefore Fn →F uac.

209

210


11.4 Convergence in Distribution in the Multidimensional Case—The Cramér–Wold Device For k ≥ 2, one may define a k-dimensional d.f. F and establish properties similar to those stated and proved in Chapter 8 for a one-dimensional d.f. Also, one may define its ch.f. and establish properties and results analogous to the ones shown in the first two sections of this chapter. However, we will only restrict ourselves to the definition of the d.f. of a k-dimensional random vector and its ch.f., and state two theorems analogous to Theorems 2 and 3. Definition 4. (i) The d.f. of a k-dimensional random vector X = (X 1 , . . . , X k ), or the joint d.f. of the r.v.s X 1 , . . . , X k , is defined by FX (x) = FX 1 ,...,X k (x1 , . . . , xk ) = P(X 1 ≤ x1 , . . . , X k ≤ xk ), x1 , . . . , xk ∈ . (ii) The ch.f. of the random vector X, or the joint ch.f. of the r.v.s X 1 , . . . , X k , is defined by f X (t) = EetX = Eet1 X 1 +···+tk X k , t1 , . . . , tk ∈ , where “ ” denotes transpose. (iii) For n ≥ 1, let Fn be the d.f. of the k-dimensional random vector Xn = (X 1n , . . . , X kn ), and let F be the d.f. of the random vector X = (X 1 , . . . , X k ). d

Then Xn → X or Fn =⇒ F, if FXn (x) → FX (x) for all continuity points x of n→∞ n→∞ n→∞ FX . A version of Theorem 2 for the k-dimensional case reads as follows. Theorem 2 . Let X be a k-dimensional random vector, X = (X 1 , . . . , X k ), with d.f. F and ch.f. f . Then, for continuity points a = (a1 , . . . , ak ) and b = (b1 , . . . , bk ) of F, it holds that P(a j < X j ≤ b j , j = 1, . . . , k) k Tk T1 k e−it j a j − e−it j b j 1 ··· f (t1 , . . . , tk ) × = lim 2π it j −Tk −T1 j=1

dt1 · · · dtk , as (0 x2 , J (x1 ) − J (x2 ) = [F1 (x1 − y) − F1 (x2 − y)]d F2 (y) ≥ 0,

as x1 > x2 implies F1 (x1 − y) ≥ F1 (x2 − y). Hence J (x1 ) ≥ J (x2 ). Next, J is continuous from the right. In fact, let x0 ∈ and let x ↓ x0 . Then F1 (x − y) ↓ F1 (x0 − y) while F1 (x − y) ≤ C1 , independent of x, and integrable. Hence, the Dominated Convergence Theorem gives F1 (x − y)d F2 (y) → F1 (x0 − y)d F2 (y) = J (x0 ). J (x) =

Finally, if F1 and F2 are d.f.s of r.v.s, then J (x) ≤ 1, by (11.21), and J (−∞) = 0, J (∞) = 1. In fact, as x → −∞, then F1 (x − y) → 0, y ∈ . Since again F1 (x − y) ≤ 1, integrable, we get, by the Dominated Convergence Theorem, F1 (x − y)d F2 (y) → 0; i.e., J (x) → 0, or J (−∞) = 0. x→−∞

Next, x → ∞ implies F1 (x − y) → 1, y ∈ . As before,

F1 (x − y)d F2 (y) →

or J (∞) = 1.

1d F2 (y) = 1; i.e., J (x) → 1, x→∞

Remark 7. If J ∗ (x) = F2 (x − y)d F1 (y), then J ∗ (x) is also a d.f., denoted by F2 ∗ F1 . Now let f , f 1 , f 2 be the ch.f.s corresponding to F, F1 , F2 . Then the following theorem is true.

Theorem 6. If F = F1 ∗ F2 , then f = f 1 × f 2 . Conversely, if f 1 and f 2 are the ch.f.s of the d.f.s F1 and F2 , respectively, and if we set f = f 1 × f 2 , then f is the ch.f. of the d.f. F, where F = F ∗ + c, for some constant c, and F ∗ = F1 ∗ F2 .

11.5 Convolution of Distribution Functions and Related Results

Let F = F1 ∗ F2 . Then f (t) = eit x d F(x) =

Proof.

lim

α→−∞,β→∞ (α,β]

eit x d F(x).

For α < β, look at (α, β] and consider the following partition for each n: α

(

α = x n1

x n2

x n3

]

···

x nk n

β

xn,k n +1 = β

where the partitioning points are chosen so that max (xn, j+1 − xn j ) → 0. n→∞

j=1,...,kn

Then (α,β]

e

it x

kn

d F(x) = lim

n→∞

= lim

j=1 kn

n→∞

eit xn j [F(xn, j+1 ) − F(xn j )] eit xn j

j=1

− = lim

F1 (xn, j+1 − y)d F2 (y)

F1 (xn j − y)d F2 (y)

kn

n→∞ j=1

eit xn j [F1 (xn, j+1 − y)

−F1 (xn j − y)]d F2 (y) ⎧ ⎨ kn eit(xn j −y) [F1 (xn, j+1 − y) = lim n→∞ ⎩ j=1 ⎫ ⎬ − F1 (xn j − y)]eit y d F2 (y). ⎭

(11.22)

n it(x −y) But kj=1 e n j [F1 (xn, j+1 − y) − F1 (xn j − y)] are partial sums tending to the integral of eit x over the interval (α − y, β − y] with respect to F1 . Furthermore, these partial sums are bounded in absolute value by kn [F1 (xn, j+1 − y) − F1 (xn j − y)] = F1 (xn,kn +1 − y) j=1

−F1 (xn1 − y) ≤ V1 ,

213

214


the variation of F1 , independent of n, and F2 -integrable, whereas lim

kn

n→∞

eit(xn j −y) [F1 (xn, j+1 − y) − F1 (xn j − y)]eit y

j=1

=

(α−y,β−y]

eit x d F1 (x) eit y .

Then the Dominated Convergence Theorem gives ⎧ ⎨ kn lim eit(xn j −y) [F1 (xn, j+1 − y) n→∞ ⎩ j=1 −F1 (xn j − y)]eit y d F2 (y) ⎧ ⎨ kn = eit(xn j −y) [F1 (xn, j+1 − y) lim ⎩n→∞ j=1 it y −F1 (xn j − y)]e d F2 (y) it x = e d F1 (x) eit y d F2 (y),

(α−y,β−y]

or, by (11.22), (α,β]

Next,

e

it x

d F(x) =

(α−y,β−y]

(α−y,β−y]

e

it x

d F1 (x) e

e

it x

d F1 (x) eit y d F2 (y).

≤ V1 , independent of α, β,

it y

and F2 -integrable, whereas

lim e d F1 (x) eit y (α−y,β−y] = eit x d F1 (x) eit y = f 1 (t)eit y . it x

α→−∞,β→∞

Therefore, by the Dominated Convergence Theorem again, and (11.23), lim eit x d F(x) α→−∞,β→∞ (α,β]

(11.23)

11.5 Convolution of Distribution Functions and Related Results

= =

lim

α→−∞,β→∞

=

lim

α→−∞,β→∞

(α−y,β−y]

e

(α−y,β−y]

f 1 (t)eit y d F2 (y) = f 1 (t)

d F1 (x) eit y d F2 (y) eit x d F1 (x) eit y d F2 (y)

it x

eit y d F2 (y) = f 1 (t) f 2 (t).

(11.24)

Since the left-hand side of (11.24) is eit x d F(x) = f (t), we have then f (t) = f 1 (t) f 2 (t). Thus, if F = F1 ∗ F2 , then f = f 1 × f 2 . From F ∗ = F1 ∗ F2 and the direct part, we have f ∗ = f 1 × f 2 , where f ∗ is the ch.f. of F ∗ . Also, f = f 1 × f 2 . Thus, f = f ∗ . Hence f is a ch.f. and the corresponding d.f. is F = F ∗ + c, for some constant c. To this theorem, there are the following three corollaries. Corollary 1.

The product of two ch.f.s is a ch.f.

Proof. Let f 1 and f 2 be two ch.f.s. For j = 1, 2, the ch.f. f j determines a class C j of d.f.s, any two of which differ by a constant. Let F j ∈ C j , j = 1, 2, be any two d.f.s, and let J = F1 ∗ F2 . Then, by Theorem 5, J is a d.f., and let f be its ch.f. Then, by Theorem 6, f = f 1 × f 2 , so that the product f 1 × f 2 is a ch.f. Corollary 2. For any two d.f.s F1 and F2 , we have F1 ∗ F2 = F2 ∗ F1 uac, and F1 ∗ F2 = F2 ∗ F1 , if F1 and F2 are d.f.s of r.v.s. Proof. Let F1 ∗ F2 = J and F2 ∗ F1 = J ∗ with respective ch.f.s f and f ∗ . Then, by Theorem 6, f = f 1 × f 2 and f ∗ = f 2 × f 1 . Since f 1 × f 2 = f 2 × f 1 , we have f = f ∗ , so that J − J ∗ = c, for some constant c, which proves the first assertion. For the second assertion, we have that if both F1 and F2 are d.f.s of r.v.s, then J and J ∗ are d.f.s of r.v.s by Theorem 5. Since f = f ∗ , by the first part here, it follows that J = J ∗ (see Remark 6(ii)). Corollary 3. If F1 and F2 are, respectively, the d.f.s of the independent r.v.s X 1 and X 2 , then F = F1 ∗ F2 (= F2 ∗ F1 ) is the d.f. of the r.v. X 1 + X 2 . Proof.

By Lemma 1 in Chapter 10, f X 1 +X 2 (t) = Eeit(X 1 +X 2 ) = E(eit X 1 × eit X 2 ) = (Eeit X 1 )(Eeit X 2 ) = f X 1 (t) f X 2 (t) = f (t),

where f is the ch.f. of F1 ∗ F2 (by Corollary 1). Since F1 ∗ F2 is a d.f. of a r.v., it follows that it is the d.f. of X 1 + X 2 . This section is concluded with the definition of symmetry of a r.v., and some results related to it. Definition 6. The r.v. X is said to be symmetric about zero, if the r.v.s X and −X have the same distribution; i.e., P(X ≤ x) = P(−X ≤ x) = P(X ≥ −x), x ∈ .

215

216


Theorem 7.

We have

(i) For any ch.f. f , f (−t) = f (t), t ∈ . (ii) If f X is the ch.f. of the r.v. X , then f X is the ch.f. of the r.v. −X . (iii) X is symmetric about zero if and only if its ch.f. f X is real.

Proof. (i) Let F be any d.f. corresponding to f . Then f (−t) = e−it x d F(x) [cos(−t x) + i sin(−t x)]d F(x) = [cos(t x) − i sin(t x)]d F(x) = = =

[cos(t x) + i sin(t x)]d F(x) eit x d F(x) = f (t). (See also Theorem 1(iv).)

(ii) By Theorem 1(iii), f α X +β (t) = eiβt f X (αt). For α = −1, β = 0, this becomes f −X (t) = f X (−t). But f X (−t) = f X (t). Thus f X (t) is the ch.f. of −X . (Or, by part (i), f −X (t) = Eeit(−X ) = Eei(−t)X = f X (−t) = f X (t).) (iii) Let X be symmetric about zero with d.f. FX . Then it x f X (t) = e d FX (x) = eit x d F−X (x) = f −X (t) = f X (t)

by part (ii); i.e., f X = f X , so that f X is real. Next, let f X be real. Then f X = f X . But f X = f −X by part (ii). Thus f X = f −X or FX and F−X are the same; hence, X is symmetric about zero.

11.6 Some Further Properties of Characteristic Functions In this section, two main results (Theorems 8 and 9) are established pertaining to ch.f.s of r.v.s. Theorem 8 (and its corollary) may also be established for certain ch.f.s that are not necessarily ch.f.s of r.v.s. Theorem 8. For n = 1, 2, . . ., let f n , f be ch.f.s of r.v.s Then, if f n → f on , it n→∞ follows that the convergence is uniform in closed intervals in .

11.6 Some Further Properties of Characteristic Functions

Proof. It suffices to prove that f n (t) → f (t) uniform in t ∈ [−T , T ], T > 0. Let Fn , F be the d.f.s corresponding to f n , f . Then we have it x it x | f n (t) − f (t)| = e d Fn (x) − e d F(x) it x it x ≤ e d Fn (x) − e d F(x) (α,β] (α,β] + eit x d Fn (x) + eit x d F(x) . (11.25) −(α,β]

But

−(α,β]

eit x d F(x) ≤ d F(x) = 1 − [F(β) − F(α)], −(α,β] −(α,β] eit x d Fn (x) ≤ d Fn (x) = 1 − [Fn (β) − Fn (α)].

−(α,β]

−(α,β]

Thus, (11.25) becomes

| f n (t) − f (t)| ≤

(α,β]

e

it x

d Fn (x) −

(α,β]

e

it x

d F(x)

+{1 − [F(β) − F(α)]} + {1 − [Fn (β) − Fn (α)]}.

(11.26)

Pick α, β to be continuity points F and such that 1 − [F(β) − F(α)]
n 1 = n 1 (ε), and the last expression is < (11.26) becomes it x e d Fn (x) − | f n (t) − f (t)| ≤ n ≥ n1.

(α,β]

2ε 7

(α,β]

e

it x

3ε d F(x) + , 7 (11.28)

The proof of the theorem would be completed (by means of (11.28)) if we knew that eit x d Fn (x) → eit x d F(x) uni f or mly in t ∈ [−T , T ]. (α,β]

n→∞ (α,β]

This convergence is true for each t (by the Helly–Bray Lemma, Theorem 6 in Chapter 8), and the uniformity in t ∈ [−T , T ] is the content of the following result.

217

218


Statement. Under the assumptions of Theorem 8 and with α and β being continuity points of F, it holds it x e d Fn (x) → eit x d F(x) uni f or mly in t ∈ [−T , T ]. n→∞ (α,β]

(α,β]

Proof.

Pick points α = x1 < x2 < · · · < x N < x N +1 = β

to be continuity points of F and such that max (xk+1 − xk ) ≤

k=1,...,N

ε , 7T

and on (α, β], define the function gt as follows: gt (x) = eit xk if x ∈ (xk , xk+1 ], k = 1, . . . , N . Pick n ≥ n 2 = n 2 (ε, N ), so that |Fn (xk ) − F(xk )|
0. Then f n → f uniformly in [−T , T ] and f is continuous. Hence, by Proposition 4, the convergence is continuous in [−T , T ]. In particular, f n (tn ) → f (t). Lemma 5. Let X be a r.v. with ch.f. f . Then if E|X |n < ∞ for some positive dn integer n, it follows that f (n) (t) = dt n f (t) is continuous in . Proof.

As was seen in the proof of Theorem 1(v), f (n) (t) = i n eit X X n d F(x).

Next,

i n eit X X n → i n eit0 X X n and i n eit X X n ≤ |X |n , t→t0

independent of t and integrable. Hence the Dominated Convergence Theorem completes the proof. Theorem 9. Let X be a r.v. such that E|X |n < ∞ for some positive integer n, and let f be its ch.f. Set m (k) = E X k , k = 0, 1, . . . , n. Then one has f (t) =

n−1 (k) m k=0

k!

(it)k + ρn (t), t ∈ ,

where 1 n−1 (n) (t x)d x (where the integral exists because of (i) ρn (t) = t n 0 (1−x) (n−1)! f Lemma 5), or (n) (ii) ρn (t) = mn! (it)n + o(t n ), or (n)

(iii) ρn (t) = θ μn! |t|n , where θ = θ (n, t) is such that |θ | ≤ 1, and μ(n) = E|X |n . Proof. (i) In the first place, f (n) (u) exists and is continuous (by Theorem 1(v) and Lemma 5), so that the integral indicated exists in the Riemann sense. Next, the result is, clearly, true for t = 0 (00 = 1). Thus we may assume in the sequel that t = 0, if need be. Now 1 (1 − x)n−1 (n) f (t x)d x ρn (t) = t n (n − 1)! 0 1 1 = (t − t x)n−1 f (n) (t x)d(t x) (n − 1)! 0 t 1 = (t − u)n−1 f (n) (u)du (n − 1)! 0 (by setting t x = u)

11.6 Some Further Properties of Characteristic Functions

t 1 (t − u)n−1 d f (n−1) (u) (n − 1)! 0 1 (n − 1) = (t − u)n−1 f (n−1) (u)|t0 + (n − 1)! (n − 1)! t × (t − u)n−2 f (n−1) (u)du

=

0

t n−1 f (n−1) (0) (n − 1)! t 1 (t − u)n−2 f (n−1) (u)du; + (n − 2)! 0 i.e., integrating by parts, we get t 1 n−2 f (n−1) (u)du. f (n−1) (0) + (n−2)! 0 (t − u) =−

n−1

t ρn (t) = − (n−1)! Also, t 1 n−2 f (n−1) (u)du = (n−2)! 0 (t − u)

=

t 1 n−2 d f (n−2) (u) (n−2)! 0 (t − u) 1 n−2 f (n−2) (u)|t 0 (n−2)! (t − u) (n−2) t n−3 (n−2) + (n−2)! 0 (t − u) f (u)du t n−2 (n−2)

=−

(n − 2)! t 1

+ (n−3)!

f

0 (t

(0)

− u)n−3 f (n−2) (u)du,

so that t n−1 t n−2 f (n−1) (0) − f (n−2) (0) (n − 1)! (n − 2)! t 1 (t − u)n−3 f (n−2) (u)du. + (n − 3)! 0

ρn (t) = −

Proceeding in this manner, the (n − 1)th integration by parts yields t t (1) t − f (0) + f (1) (u)du = − f (1) (0) + f (t) − 1, 1! 1! 0 so that ρn (t) = −1 −

t (1) t n−2 f (0) − · · · − f (n−2) (0) 1! (n − 2)!

t n−1 f (n−1) (0) + f (t) (n − 1)! n−1 k t (k) f (0) + f (t) =− k! −

k=0

221

222


=−

n−1 k t k=0

k!

i k m (k) + f (t)

(since f (k) (0) = i k m (k) ) =−

n−1 (k) m

k!

k=0

f (t) =

(it)k + f (t), so that

n−1 (k) m k=0

(ii) By part (i), ρn (t) =

1 tn (n−1)! 0 (1 −

(n − 1)! ρn (t) = tn

k!

(it)k + ρn (t), t ∈ .

x)n−1 f (n) (t x)d x, so that, for t = 0,

1

(1 − x)n−1 f (n) (t x)d x 0 1 = (1 − x)n−1 (iu)n eit xu d F(u) d x 0 dn since n f (λ) dλ n n ∂ iλu d iλu e d F(u) = e = d F(u) n dλn ∂λ = (iu)n eiλu d F(u) 1

(1 − x)n−1 u n eit xu d F(u)d x 1 n n n−1 it xu u (1 − x) e d x d F(u) =i = in

0

(11.33)

0

(by the Fubini Theorem, which applies since |(1 − x)n−1 u n eit xu | ≤ |u|n is Lebesgue × F-integrable over [0, 1] × ). Now, |(1 − x)n−1 eit xu | = |1 − x|n−1 ≤ 1 (over [0, 1]) independent of t and Lebesgue-integrable over [0, 1]. Furthermore, (1 − x)n−1 eit xu → (1 − x)n−1 , t→0

so that the Dominated Convergence Theorem yields 1 1 1 (1 − x)n−1 eit xu d x → (1 − x)n−1 d x = . t→0 0 n 0 Next, 1 n n−1 it xu u ≤ |u|n independent of t, and F (1 − x) e d x 0

−integrable.

11.7 Applications to the Weak Law of Large Numbers

Furthermore, u n

1 0

un . Therefore, by the Dominated Cont→0 n

(1−x)n−1 eit xu d x →

vergence Theorem, 1 n u d F(u) un (1 − x)n−1 eit xu d x d F(u) → t→0 n 0 1 = m (n) . n n 1 It follows that u [ 0 (1 − x)n−1 eit xu d x]d F(u) = n1 m (n) + o(1) (where o(1) → 0 as t → 0), and then by (11.33), (n − 1)! i n (n) m + o(1) or ρn (t) ρ (t) = n tn n m (n) m (n) (it)n + t n o(1) = (it)n + o(t n ), = n! n! as was to be seen. (iii) Again, as in part (ii) (see first and last lines on the right-hand side of relation (11.33)), 1 1 n n−1 it xu (1 − x)n−1 f (n) (t x)d x = i n u (1 − x) e d x d F(u) 0

≤

0 1

|u| (1 − x) d x d F(u) 0 1 1 = |u|n d F(u) = μ(n) , n n so that

|ρn (t)| =

tn (n − 1)!

0

n

n−1

1

|t|n (n) μ . (1 − x)n−1 f n (t x)d x ≤ n!

Then there exists θ = θ (n, t) with |θ | ≤ 1 such that ρn (t) = θ |t|n! × μ(n) = n

(n)

θ μn! |t|n .

11.7 Applications to the Weak Law of Large Numbers and the Central Limit Theorem The following two results are applications of two theorems in the previous sections, Theorems 3 and 9. Application 1 (Weak Law of Large Numbers, WLLN). Let X 1 , . . . , X n be i.i.d. r.v.s with E X 1 = m (1) finite. Then X1 + · · · + Xn P → m (1) . X¯ n = n→∞ n

223

224


Discussion.

By Theorem 9 (ii), applied for n = 1, we get m (1) (it) + o(t) (where f is the ch.f. of the X i s). 1!

f (t) = 1 + Hence

n t t (1) t f X¯ n (t) = f = 1+m i +o n n n n n (1) t + to(1) t it im = 1 + m (1) + o(1) = 1 + , n n n n

where for a fixed t, o(1) → 0 as n → ∞, so that im (1) t + to(1) → im (1) t. This n→∞ implies that n im (1) t + to(1) (1) 1+ → eim t , (11.34) n→∞ n (1)

so that f X¯ n (t) → eim t which is the ch.f. of the r.v. X that is equal to m (1) with n→∞ probability one. Hence, by Theorem 3, d P X¯ n → m (1) or, equivalently X¯ n → m (1) . n→∞

n→∞

Remark 8. In relation (11.34), we use the familiar result (1 + cn → c as n → ∞ (see also Exercise 5).

cn n n )

→ ec when

Application 2 (Central Limit Theorem, CLT). Let X 1 , . . . , X n be i.i.d. r.v.s with E X 1 = m (1) finite and σ 2 (X 1 ) = σ 2 ∈ (0, ∞). Then √ Sn − E Sn n[ X¯ n − m (1) ] d = → Z ∼ N (0, 1), n→∞ σ (Sn ) σ where n n 1 Sn = X j and X¯ n = X j. n j=1

j=1

X −m (1)

Discussion. Set Y j = j σ . Then the r.v.s Y1 , . . . , Yn are i.i.d. with EY1 = 0 and σ 2 (Y1 ) = EY12 = 1. Let f 1 be the ch.f. of the Yi s. Then by Theorem 9 (ii), applied for n = 2, we get EY12 EY1 (it) + (it)2 + o(t 2 ) 1! 2! t2 t2 = 1 − + o(t 2 ) = 1 − + t 2 o(1). 2 2

f 1 (t) = 1 +

Hence f Sn −E Sn (t) = f n

n X j −m (1) (t) = f

j=1

j=1

σ (Sn )

√ σ n

t n (t) = f √ √ 1 Yj/ n n

11.8 The Moments of a Random Variable Determine its Distribution

n t2 t2 = 1− + o(1) 2n n n 2 − t2 + t 2 o(1) t2 = 1+ → e− 2 , n→∞ n since −

t2 t2 + t 2 o(1) → − as o(1) → 0 for a fixed t. n→∞ n→∞ 2 2

t2

The fact that e− 2 is the ch.f. of a r.v. Z ∼ N (0, 1) completes the proof.

11.8 The Moments of a Random Variable Determine its Distribution This section consists of two main results, Theorems 10 and 11. In the first of these results, a condition is given under which a ch.f. expands into an infinite series, and in the second, conditions are stated under which the moments of a r.v. completely determine its distribution. In the proof of Theorem 11, one needs a number of concepts and results from complex analysis, which are presented in the next section. Let X be a r.v. such that E|X |n < ∞ for n = 1, 2, . . ., and let f be m (n) n its ch.f. Then for any t ∈ for which the series ∞ n=0 n! (it) converges, one has ∞ m (n) f (t) = n=0 n! (it)n .

Theorem 10.

Proof.

Set Sn (t) =

n m (k) k=0

k!

(it)k ,

and observe that the assertion is true for t = 0 (00 = 1). So, let t0 = 0 be a point for (k) (2r ) which Sn (t0 ) converges. Then mk! (it0 )k → 0. In particular, m(2r )! (it0 )2r → 0, and k→∞

this is equivalent to

r →∞

μ(2r ) 2r |t0 | → 0, r →∞ (2r )!

since μ(2r ) = m (2r ) . From Theorem 9 (iii), applied for n = 2r , we have f (t0 ) =

2r −1 k=0

so that

m (k) μ(2r ) 2r μ(2r ) 2r (it0 )k + θ |t0 | = S2r −1 (t0 ) + θ |t0 | , k! (2r )! (2r )!

μ(2r ) μ(2r ) |t0 |2r ≤ |t |2r → 0 | f (t0 ) − S2r −1 (t0 )| = θ (2r )! 0 r →∞ (2r )!

(11.35)

225

226


on account of (11.35). Thus S2r −1 (t0 ) → f (t0 ).

(11.36)

r →∞

Next,

μ(2r ) m (2r ) |S2r (t0 ) − S2r −1 (t0 )| = (it0 )2r = |t |2r → 0 (2r )! 0 r →∞ (2r )!

by (11.35). Hence S2r (t0 ) → f (t0 ) and this together with (11.36) gives that r →∞ Sn (t0 ) → f (t0 ). n→∞ ∞ Remark 9. Recall that if n=0 cn z n is a power series where z is, in general, a complex variable, then the radius ρ of convergence of the series is given by ρ = 1/ lim sup |cn |1/n , n→∞

so that ρ −1 = lim supn→∞ |cn |1/n , and the series converges for |z| < ρ. Lemma 6. Let X be a r.v. such that μ(n) = E|X |n < ∞ for all n = 1, 2, . . . (so that m (n) = E X n are also finite). Then the series S(t) =

∞ m (n) n=0

n!

∗

(it) and S (t) = n

∞ μ(n) n=0

n!

|t|n

have the same radii of convergence. Let ρ1 and ρ2 be the radii of convergence of the series S(t) and S ∗ (t), (n) 1/n (n) 1/n respectively. Then, since mn! ≤ μn! (by the fact that |E X n | ≤ E|X |n ),

Proof.

it follows that equivalently,

1 ρ2

or ρ1 ≥ ρ2 . Thus, it suffices to show that ρ1 ≤ ρ2 , or (2n) 1/2n (2n) 1/2n ≤ ρ11 . To this end, we have μ(2n)! = m(2n)! , so that

1 ρ1

≤

1 ρ2 ,

μ(2n) lim sup (2n)! n→∞

1/2n

m (2n) = lim sup (2n)! n→∞

1/2n ≤

1 . ρ1

(11.37)

Next, [μ(2n−1) ]1/(2n−1) ≤ [μ(2n) ]1/2n (by the fact that E 1/r |X |r ↑ in r > 0), so that 1/(2n−1) 1/(2n) μ(2n) μ(2n−1) [(2n)!]1/2n ≤ × . (11.38) (2n − 1)! (2n)! [(2n − 1)!]1/(2n−1) At this point, assume for a moment that [(2n)!]1/2n = 1. n→∞ [(2n − 1)!]1/(2n−1) lim

(11.39)

11.8 The Moments of a Random Variable Determine its Distribution

Then (11.38) gives by means of (11.37)

μ(2n−1) lim sup (2n − 1)! n→∞

1/(2n−1)

Since

⎧ ⎨

1/n

1 . ρ1

μ(2n) = max lim sup ⎩ n→∞ (2n)! n! 1/(2n−1) ⎫ ⎬ μ(2n−1) lim sup , ⎭ (2n − 1)! n→∞

1 = lim sup ρ2 n→∞

μ(n)

≤

we have, from (11.37) and (11.40), that Remark 10.

1 ρ2

≤

1 ρ1 ,

(11.40)

1/(2n) ,

so that ρ1 = ρ2 .

The proof of (11.39) is left as an exercise (see Exercise 7).

Corollary to Lemma 6. A sufficient condition for the series S(t) to converge for some t0 = 0 is that m (n) 1/n [μ(n) ]1/n < ∞. lim sup < ∞ or lim sup n n→∞ n! n→∞ Proof. The first assertion is immediate and the second follows from the first by Stirling’s formula (see Exercise 8). Theorem 11. Let X be r.v. with d.f. F and ch.f. f , and suppose that E X n = m (n) ∈ , n ≥ 1, and let μ(n) = E|X |n . Then m (n) n (i) If the series S(t) = ∞ n=0 n! (it) converges for some t0 = 0, it follows that the distribution of X is uniquely determined (by the moments of X ). (ii) A sufficient condition for S(t) to converge for some t0 = 0 is that (n) 1/n lim supn→∞ μn! < ∞. Proof. (i) Let F0 be a d.f. of a r.v., potentially different from F, with corresponding ch.f. f 0 , such that

x n d F0 (x) = m (n) , n ≥ 0.

m (n) n Then the series S0 (t) = ∞ n=0 n! (it) (= S(t)) converges for t = t0 = 0, and hence for all t with |t| ≤ t0 , supposing without loss of generality that t0 > 0. For each such t, S(t) represents f (t), and S0 (t) represents f 0 (t) on account of

227

228


Theorem 10. Then, by Fact 4 in the next section, f (z) and f 0 (z) are defined and are analytic for |I m(z)| < t0 . Furthermore, f

(n)

∞ i k m (k) k−n (n) (z) = = f 0 (z), |z| < t0 , z (k − n)! k=n

and in particular, f (z) =

∞ k (k) i m k=0

k!

z k = f 0 (z), |z| < t0 .

(11.41)

Thus, f (z) and f 0 (z) are analytic for |I m(z)| < t0 , and f (z) = f 0 (z) for |z| < t0 , by means of (11.41). Then, by Fact 5, f (z) = f 0 (z) for |I m(z)| < t0 (by continuous extension). In particular, f (t) = f 0 (t) for t ∈ , so that F = F0 . (ii) Immediate by the Corollary to Lemma 6. The last theorem is illustrated by the following example. Let Z ∼ N (0, 1), so that m (2k) =

Example 4. Then f (t) =

∞ m (2k) k=0

(2k)!

(it)

2k

(2k)! , m (2k+1) 2k k!

= 0, k = 0, 1, . . ..

2 k ∞ ∞ t2 t 1 1 2k (it) = − = = e− 2 , k 2 k! k! 2 k=0

k=0

as was expected. This section is concluded with some comments and a result involving the expansion of a logarithmic function. Recall that if z is a complex number, then the log z is any complex number w such that ew = z. This relation defines a many-valued function, since if w is a solution of ew = z, then so is w + 2nπi, n = 0, ±1, . . ., because e2nπi = 1. Now, every complex number z may be written as z = |z|eiθ for some θ with −π < θ ≤ π . Then w = log |z| + iθ (|z| = 0), −π < θ ≤ π , is a solution of ew = z, since ew = elog |z|+iθ = |z|eiθ = z. This solution is called the principal branch of the logarithm of z and is usually denoted by log p z. In all that follows, we shall work with log p although we shall not indicate the p. The following result will prove useful in many situations. Lemma 7.

For any complex number z, one has log(1 + z) = z[1 + ε(z)] = z(1 + θ z), if |z| ≤

where |ε(z)| ≤ |z| and |θ | = |θ (z)| ≤ 1. Proof.

For |z| < 1, it is known that log(1 + z) = z −

z3 z4 z2 + − + ··· 2 3 4

1 , 2

11.9 Some Basic Concepts and Results from Complex Analysis

∞ zn (−1)n+1 n n=1 z z2 z3 = z 1− + − + ··· 2 3 4 = z[1 + ε(z)], (z = 0),

=

where

(11.42)

z2 z3 z − + ··· ε(z) = − + 2 3 4

Now z z2 z3 |ε(z)| = − + − + · · · 2 3 4 z 2 2 2 = −1 + z − z + · · · 2 3 4 |z| 2 2 2 ≤ 1 + |z| + |z| + · · · . 2 3 4 |z| 1 + |z| + |z|2 + · · · ≤ 2 |z| 1 ≤ |z|, = 2 1 − |z|

(11.43)

provided |z| ≤ 21 . Thus, we have |ε(z)| ≤ |z| for |z| ≤ 21 by (11.43) and log(1 + z) = z[1 + ε(z)] by (11.42). Of course, the fact that |z| ≤ 21 implies |ε(z)| ≤ 21 ; it also implies that we can write ε(z) = θ z for some θ = θ (z) with |θ | ≤ 1; (i.e., θ (z) = ε(z) z ). Thus, log(1 + z) = z(1 + θ z).

11.9 Some Basic Concepts and Results from Complex Analysis Employed in the Proof of Theorem 11 In what follows, C stands for the complex plane, and I m(z) stands for the imaginary part of the complex number z = x + i y; i.e., I m(z) = y, x, y ∈ . Definition 8. A function g : S ⊆ C → C is said to be differentiable at z 0 ∈ S with derivative g (z 0 ), if g(z) − g(z 0 ) → g (z 0 ) z − z0 as z tends to z 0 in all possible ways; g is differentiable in S, if it is differentiable at each z ∈ S. Definition 9. The function g is called analytic in S, if it is differentiable in S. If S = C and g is analytic, then it is called entire.

229

230


Fact 1. If g is analytic in S, then the derivatives of all orders g (n) , n ≥ 1, exist (and are given by a certain formula involving the Cauchy integral). Fact 2.

A function g represented by a power series g(z) =

∞

cn z n , |z| ≤ r (some r > 0)

n=0

is analytic in |z| < r . Fact 3. If g is analytic for |z| ≤ r (some r > 0) (or analytic on and inside a simple closed contour C), then, for every z with |z| < r (or every z inside C), g(z) can be represented by a power series. More specifically, for every a with |a| < r (or every a inside C), ∞ g(z) = g (n) (a)(z − a)n , |z − a| < δ, n=0

where δ is the distance of a from the nearest point of the circumference |z|(n)= r (or n the distance of a from the nearest point of C). In particular, g(z) = ∞ n=0 g (0)z , |z| < r . Facts 2 and 3 justify the following definition. Definition 10.

The ch.f. f is said to be r -analytic, if f (t) =

∞ an n=0

n!

t n , |t| < r (r > 0), an ∈ C,

and is called entire, if r = ∞. Let f be a ch.f. with corresponding d.f. F, and for z = x + i y, x, y, ∈ , define f (z) by i zu e d F(u) = ei xu × e−yu d F(u) f (z) = −yu e cos xud F(u) + i e−yu sin xud F(u), =

provided (see also Proposition 5 below), e−yu d F(u) < ∞.

(11.44)

Fact 4. Let f be a ch.f. of a r.v. with corresponding d.f. F, and assume that the moments m (n) = x n d F(x) are finite for all n = 1, 2, . . ., and that S(t0 ) given in Lemma 6 converges for some t0 = 0. Then f (t) is r -analytic for |t| < r (some r > 0). Also, f (z) is defined and is analytic for z in the strip defined by |I m(z)| = |y| < r .


This is so by Proposition 5, which follows. Furthermore, for z with |z| < r , all derivatives f (n) (z), n ≥ 1, exist and f (n) (z) =

∞ i k m (k) k−n z , (k − n)! k=n

and, in particular, f (z) =

∞ m (k) k=0

k!

(i z)k .

Remark 11. That f (z) is well defined for z with |I m(z)| = |y| < r is due to the fact that relation (11.44) is satisfied here, as Proposition 5 below shows. Fact 5. Let g, g0 : S → C (where {z ∈ C; |I m(z)| < r } ⊆ S ⊆ C), and suppose that g(z) and g0 (z) are analytic for z with |I m(z)| < r (r > 0). Furthermore, assume that g(z) = g0 (z) for |z| < r . Then g(z) = g0 (z) for z with |I m(z)| < r . m (n) n Proposition 5. Assume that the series S(t) = ∞ n=0 n! (it) converges for some t0 = 0 (as we do in the formulation of Theorem 11), and assume without loss of generality that t0 > 0. Then relation (11.44) is satisfied, so that f (z) is well defined for z with |I m(z)| = |y| < t0 . In the first place, convergence of the series S(t) for t with |t| < t0 implies μ(n) n convergence of the series S ∗ (t) = ∞ n=0 n! |t| ; this is so, by Lemma 6. Next, for 0 < t < t0 , ∞ ntn |x| et|x| d F(x) = d F(x) n=0 n! ∞ |x|n t n d F(x) = n!

Proof.

n=0

(by Corollary 1(ii) to Theorem 1 in Chapter 5) ∞ n ∞ t μ(n) n t < ∞, = |x|n d F(x) = n! n! n=0

n=0

as already pointed out. But et|x| d F(x) =

= so that

0 −∞

0

−∞ 0 −∞

∞

et|x| d F(x) +

et|x| d F(x)

0

e

−t x

e−t x d F(x) < ∞,

∞

d F(x) +

et x d F(x),

0

0

∞

et x d F(x) < ∞.

(11.45)

231

232


Next, for 0 < y < t0 , −yu e d F(u) =

≤

−∞ 0 −∞

whereas, for −t0 < y < 0, e−yu d F(u) =

=

0

0 −∞ 0 −∞

e

−yu

∞

d F(u) +

e−yu d F(u)

0

e−yu d F(u) + V ar F < ∞ (by (11.45)),

e−yu d F(u) +

∞

e−yu d F(u)

0

e

(−y)u

≤ V ar F +

∞

d F(u) + ∞

e(−y)u d F(u)

0

e(−y)u d F(u) < ∞ (by (11.45)).

0

Thus,

e

−yu d F(u)

< ∞ for |y| < t0 , which is (11.44).

Remark 12. Material pertaining to this section may be found in the classical reference Titchmarsch (1939). Exercises. 1. Let g and h be real-valued functions defined on for which g(x)dμ and h(x)dμ are finite, where μ is a (σ -finite) measure in . Then show that: [g(x) + i h(x)]dμ ≤ |g(x) + i h(x)| dμ.

In particular, |E Z | ≤ E|Z |, where Z is a complex-valued r.v.; i.e., Z = X + iY with X and Y real-valued r.v.s. Hint: Use polar coordinates. 2. In reference to the proof of Theorem 2, show that limt→0 g(t) = b −a as claimed there. 3. In reference to the proof of Proposition 1 (see also proof of Proposition 3), provide the details of the convergence 1 t g(v)dv → g(0) as t ↓ 0. t 0 4. Show that |ei x − 1| ≤ |x| for all x ∈ . Hint: Write ei x = cos x + i sin x, express cos x in terms of sin2 2x , and use the inequality | sin t| ≤ |t|, t ∈ (which you also must prove). 5. Show that (1 + cnn )n → ec , when cn → c, where cn ∈ C, the complex plane, n→∞ n→∞ n ≥ 1.


6. Let X be a r.v. having the Cauchy distribution with parameters μ = 0 and σ = 1 1 (i.e., the p.d.f. of X is given by p(x) = π1 × 1+x 2 , x ∈ ). Then show that: (i) The E X does not exist. (ii) The ch.f. f X (t) = e−|t| , t ∈ . Next, let X 1 , . . . , X n be independent r.v.s distributed as X and set Sn = X 1 + · · · + X n . Then (iii) Identify the ch.f. f Sn /n (t). (iv) Show that

7.

8. 9.

10.

Sn P 0 n n→∞

by showing that

Sn d 0. n n→∞

(Although, by intuition, one would expect such a convergence, because of symmetry about 0 of the Cauchy distribution!) ∞ x) d x = π2 e−|t| (see, e.g., integral 403 Hint: For part (ii), use the result 0 cos(t 1+x 2 in Tallarida (1999); also see integral 635 in the same reference). [(2n)!]1/2n Show that: [(2n−1)!] 1/(2n−1) → 1. n→∞ √ 1 Hint: Use the Stirling formula, which states that n!/ 2π × n n+ 2 × e−n tends to 1 as n → ∞. Establish the validity of the relations claimed in the Corollary to Lemma 6. Hint: Use the Stirling formula cited in the hint of Exercise 7. If X 1 , . . . , X n are i.i.d. r.v.s with E X 1 = μ ∈ and σ 2 (X 1 ) = σ 2 ∈ (0, ∞), then √ ⇒ Z ∼ (by Application 2 to Theorem 9 in this chapter) it follows that Sσn −nμ n n→∞ n N (0, 1), where Sn = j=1 X j . √ does not converge in probability as n → ∞. Show that Sσn −nμ n √ Hint: Set Yn = (Sn − nμ)/σ n and show that, as n → ∞, {Yn } does not converge mutually in probability by showing that {Y2n − Yn } does not converge in probability to 0. According to the WLLN (Application 1 to Theorem 9), if the r.v.s X 1 , . . . , X n are i.i.d. with finite E X 1 , then Sn P −→ E X 1 , where Sn = X j. X¯ n = n n→∞ n

j=1

Sn !

The following example shows that it is possible for n to converge in probability to a finite constant, as n → ∞, even if the E X 1 does not exist. To this effect, for j = 1, 2, . . ., let X j be i.i.d. r.v.s such that P(X j = −n) = P(X j = n) = 1 c/n 2 log n, n ≥ 3, where c = 21 ∞ n=3 n 2 log n). Then show that E X 1 does not P exist, but Snn −→ 0, where Sn = nj=1 X j . n→∞

Hint: Show that E X 1 does not exist by showing that E X 1+ = E X 1− = ∞. Next, set X n j = X j if |X j | < n, and X n j = 0 otherwise, j = 1, 2, . . . , n ≥ 3,

233

234


n Sn∗ P Sn and let Sn∗ = −→ 0 by showing that j=1 X n j . Then show that (i) n − n n→∞ ∗ ∗ ∗ S S S P Snn = nn −→ 0; (ii) E nn = 0; (iii) V ar nn −→ 0; (iv) from (ii) and n→∞

(iii), conclude that

Sn∗ P −→ 0; n n→∞

n→∞

then (i) and (iv) complete the proof.

In all Exercises 11–16, i is to be treated as a real number, subject, of course, to the requirement that i 2 = −1. 11. If X ∼ B(n, p), show that f X (t) = ( peit + q)n , q = 1 − p. 12. If X ∼ P(λ), show that f X (t) = eλe 13.

it −λ

.

2 e−t /2 .

(i) If Z ∼ N (0, 1), show that f Z (t) = (ii) If X ∼ N (μ, σ 2 ), use the fact that Z = order to show that f X (t) = eiμt−

X −μ σ

σ 2t2 2

∼ N (0, 1) and part (i) in

.

14. If X has the Gamma distribution with parameters α and β; i.e., if its p.d.f. is given by 1 x α−1 e−x/β , x > 0 (and 0 for x ≤ 0), Γ (α)β α ∞ where the Gamma function Γ (α) is given by Γ (α) = 0 y α−1 e−y dy (α, β > 0), then show that 1 . f X (t) = (1 − iβt)α In particular, for α = 1 and β = 1/λ, we get the ch.f. of the Negative Exponential distribution with parameter λ; i.e., f X (t) = 1/(1 − itλ ); and for α = r2 (r > 0 integer) and β = 2, we get the ch.f. of the chi-square distribution with r degrees of freedom; i.e., f X (t) = 1/(1 − 2it)r /2 . 15. If the r.v.s X and Y have the Bivariate Normal distribution with parameters μ1 , μ2 in , 0 < σ1 , σ2 < ∞, and ρ ∈ [−1, 1], show that their joint ch.f. is given by f X ,Y (t1 , t2 ) = exp iμ1 t1 + iμ2 t2 1 − σ12 t12 + 2ρσ1 σ2 t1 t2 + σ22 t22 . 2 p(x; α, β) =

For this purpose, do the following: (i) Assume first that μ1 = μ2 = 0 and σ1 = σ2 = 1, and use Exercises 12 (ii) in Chapter 9 and 13 (ii) in this chapter to show that: 12 2 f X ,Y (t1 , t2 ) = exp − t1 + 2ρt1 t2 + t2 . 2


(ii) For the general case, use the transformations U = (X − μ1 )/σ1 , V = (Y − μ2 )/σ2 and verify that EU = E V = 0, V ar (U ) = V ar (V ) = 1, ρ(U , V ) = ρ(X , Y ) = ρ. Then use Exercise 15 in Chapter 9 and part (i) here to arrive at the desired expression for the ch.f. f X ,Y . 16. Let the r.v.s X and Y have the Bivariate Normal distribution with parameters μ1 , μ2 ∈ , 0 < σ1 , σ2 < ∞, and ρ ∈ [−1, 1], and set U = X +Y , V = X −Y . (i) Verify that EU = μ1 + μ2 , V ar (U ) = σ12 + σ22 + 2ρσ1 σ2 , E V = μ1 − μ2 , V ar (V ) = σ12 + σ22 − 2ρσ1 σ2 , and Cov(U , V ) = σ12 − σ22 (by using Exercises 12 (ii) and 14 (ii) in Chapter 9). (ii) Since fU ,V (t1 , t2 ) = Eeit1 U +it2 V = Eeit1 (X +Y )+it2 (X −Y ) = Eei(t1 +t2 )X +i(t1 −t2 )Y = f X ,Y (t1 + t2 , t1 − t2 ), use Exercise 15 in order to conclude that fU ,V (t1 , t2 ) = exp {i(μ1 + μ2 )t1 + i(μ1 − μ2 )t2 1 − (σ12 + σ22 + 2ρσ1 σ2 )t12 + 2(σ12 − σ22 )t1 t2 2 +(σ12 + σ22 − 2ρσ1 σ2 )t22 ]}. (iii) From part (ii) and Exercise 15, conclude that the r.v.s U and V have the Bivariate Normal distribution with parameters μ1 +μ2 , μ1 −μ2 , σ12 +σ22 + 2ρσ1 σ2 = τ12 , σ12 + σ22 − 2ρσ1 σ2 = τ22 , and ρ(U , V ) = (σ12 − σ22 )/τ1 τ2 . (iv) From part (iii) and Exercise 12 (ii) in Chapter 10, conclude that U and V are independent if and only if σ1 = σ2 . 17. In this exercise, the r.v.s X 1 , . . . , X k are independent with distributions as indicated, and X = X 1 + · · · + X k . Then use Exercise 13 in Chapter 10, the inversion formula (Theorem 2), and the appropriate ch.f.s in order to show that: (i) If X j ∼ B(n j , p), j = 1, . . . , k, then X ∼ B(n, p), where n = n 1 + · · · + nk . (ii) If X j ∼ P(λ j ), j = 1, . . . , k, then X ∼ P(λ), where λ = λ1 + · · · + λk . (iii) If X j ∼ N (μ j , σ j2 ), j = 1, . . . , k, then X ∼ N (μ, σ 2 ), where μ = μ1 + · · · + μk and σ 2 = σ12 + · · · + σk2 . Also, c1 X 1 + · · · + ck X k ∼ N (c1 μ1 + · · · + ck μk , c12 σ12 + · · · + ck2 σk2 ), where c1 , . . . , ck are constants. (iv) If X j ∼ Gamma with parameters α j and β, j = 1, . . . , k, then X ∼ Gamma with parameters α = α1 + · · · + αk and β. In particular, if X j has the Negative Exponential distribution with parameter λ, j = 1, . . . , k, then X ∼ Gamma with parameters α = k and β = 1/λ, whereas, if X j ∼ χr2j , j = 1, . . . , k, then X ∼ χr2 , where r = r1 + · · · + rk .

235

236


18. Show that the r.v.s X 1 , . . . , X k are independent if and only if, for all t1 , . . . , tk in , f X 1 ,...,X k (t1 , . . . , tk ) = f X 1 (t1 ) × · · · × f X k (tk ). Hint: Use Proposition 1 in Chapter 10 (and work as in Exercise 13 in the same chapter), and Theorem 2 . 19. If f is a ch.f., then show that f is positive definite; i.e., for all tk , tl reals and all complex numbers z k , zl , we have n n

f (tk − tl )z k z¯l ≥ 0

k=1 l=1

for every integer n ≥ 1. 20. For n = 1, 2, . . ., let X n be a r.v. distributed as Poisson with parameter n, X n ∼ √ P(n) (so that E X n = V ar (X n ) = n), and set Yn = (X n − n)/ n. Then show d

that Yn −→ Z ∼ N (0, 1). n→∞

2

3

∗

Hint: Use the expansion: ei z = 1+i z − z2 − i z6 ei z for some complex number z ∗ . 21. Let F1 and F2 be d.f.s, and let G be their convolution, G = F1 ∗ F2 . Then: (i) If F1 is absolutely continuous with respect to Lebesgue measure with p.d.f. p1 , it follows that G is absolutely continuous with respect to Lebesgue measure with p.d.f. p given by ∞ p1 (u − y)d F2 (y). p(u) = −∞

(ii) Furthermore, if F2 is also absolutely continuous with respect to Lebesgue measure with p.d.f. p2 , then ∞ p(u) = p1 (u − y) p2 (y)dy. −∞

22. If the r.v.s X and Y are i.i.d. with ch.f. f , then the ch.f. of the r.v. X − Y is | f (t)|2 . 23. Let X 1 , . . . , X n be independent r.v.s, each of which is symmetric about 0. Then the r.v. X 1 + · · · + X n is also symmetric about 0. 24. For n ≥ 1, let gn and g be functions defined on E ⊆ k into , and recall that {gn } is said to converge continuously to g on E, if for every x ∈ E, gn (xn ) → g(x) whenever xn → x, as n → ∞. Then show that if {gn } converges continuously to g on E, it follows that g is continuous on E. 25. For n ≥ 1, let gn , g and E be as in Exercise 24. Then: (i) If {gn } converges to g uniformly on E and g is continuous on E, it follows that {gn } converges continuously to g on E. (ii) If E is compact and {gn } converges continuously to g on E, then the convergence is uniform. 26. For n ≥ 1, let X n , Yn and X , Y be r.v.s defined on the probability space (, A, P), let d1 , d2 and cn be constants with 0 = cn → ∞ as n → ∞, and suppose that


d

cn (X n − d1 , Yn − d2 )→(X , Y ) as n → ∞. Let g : 2 → be differentiable, and suppose its (first-order) partial derivatives g˙ x , g˙ y are continuous at (d1 , d2 ). Then show that, as n → ∞, d

cn [g(X n , Yn ) − g(d1 , d2 )] −→ [g˙ x (d1 , d2 ) g˙ y (d1 , d2 )](X Y ) = g˙ x (d1 , d2 )X + g˙ y (d1 , d2 )Y .

237

CHAPTER

The Central Limit Problem: The Centered Case

12

In this chapter, we discuss what is perhaps the most important problem in probability theory. Namely, conditions are sought under which partial sums of independent r.v.s converge in distribution to some limiting law. Also, the determination of the totality of such laws is sought. A special but extremely important case is that where the limiting law is the Normal distribution, and a very special case of it is the Central Limit Theorem (CLT) in its simplest and popular form. In order to cover as many cases of practical importance as feasible, the approach used here is that of employing a triangular array of r.v.s. The r.v.s within each row are assumed to be independent, but no such assumption is made for those in distinct rows. Of course, it is assumed that the number of r.v.s in the rows tends to infinity as the rank of rows increases. A brief description of what is done in this chapter is as follows. After basic notation and assumptions are taken care of, necessary and sufficient conditions are given for the sequence of partial sums to converge in distribution to a N (0, 1) distributed r.v. This is the extremely useful Normal Convergence Criterion due to Lindeberg and Feller, and stated as Theorem 1. Its proof is derived as a special case of Theorem 2 stated and proved in Section 12.1, and is deferred to Section 12.3. In Section 12.2, the problem is cast in broad generality. Instead of seeking conditions under which the limiting law is normal, we investigate under what conditions there is convergence in distribution. Also, the totality of such limiting laws is identified. The culmination of these discussions is Theorem 2, which follows after a number of auxiliary results are established. In Section 12.3, first the proof of Theorem 1 is derived as a special case of that of Theorem 2. Then several sets of sufficient conditions are given for convergence in distribution to the N (0, 1) distribution. One such condition is expressed in terms of absolute moments and is the basis of the so-called Liapounov Theorem (Theorem 3). If the sum of variances of the r.v.s in each row is not equal to 1, which is part of the basic assumption made, then Theorem 1 assumes an easy modification, which is Theorem 4. The next modification of Theorem 1 is that where one is dealing with a single sequence of r.v.s, and leads to Theorem 5. Should the underlying r.v.s be also identically distributed, then one has the Corollary to Theorem 5, which is the CLT in its simplest and popular form. An Introduction to Measure-Theoretic Probability, Second Edition. http://dx.doi.org/10.1016/B978-0-12-800042-7.00012-8 Copyright © 2014 Elsevier Inc. All rights reserved.

239

240

CHAPTER 12 The Central Limit Problem: The Centered Case

This chapter is concluded with a final section, Section 12.4, where technical results stated in Section 12.2 are proved. In conclusion, it is to be pointed out that the term “centered” used in the title of this chapter simply indicates that all r.v.s involved have expectations zero. This is always achieved by subtracting off the expectations from the r.v.s, or by “centering” the r.v.s.

12.1 Convergence to the Normal Law (Central Limit Theorem, CLT) In the present section, we shall concern ourselves with an array of r.v.s X n j , j = 1, . . . , kn , where kn → ∞, and for each n, the r.v.s X n1 , . . . , X nkn are independent, n→∞ but not necessarily identically distributed. ⎧ ⎪ ⎪ X 11 , X 12 , . . . , X 1k1 independent ⎪ ⎪ ⎨ X 21 , X 22 , . . . , X 2k2 independent .................. ............ ⎪ ⎪ X n1 , X n2 , . . . , X nkn independent ⎪ ⎪ ⎩ .................. ............ However, r.v.s in different rows need not be independent. The probability spaces on which rows are defined may even be different. We shall assume throughout that E X n j , j = 1, . . . , kn , are finite for all n. Then we may consider the r.v.s Yn j = X n j − E X n j with EYn j = 0. Thus, without loss of generality, we shall assume that E X n j = 0, j = 1, . . . , kn , n = 1, 2, . . . . To summarize: For each n, X n j , j = 1, . . . , kn , are independent r.v.s. (within each row), kn → ∞, and E X n j = 0, j = 1, . . . , kn , n = 1, 2, . . ..

n→∞

In the present section, we shall also assume that σn2j = σ 2 (X n j ) = E X n2 j < ∞, j = 1, . . . , kn , n = 1, 2, . . . , and we shall set sn2

=

kn

σn2j and sn = + sn2 .

j=1

Then the problem to concern ourselves with is this: under what condition is it true that kn Sn ⇒ N (0, 1), where Sn = L Xnj ? sn n→∞ j=1

Here and in the sequel, it is convenient to denote the distribution of a r.v. X by L(X ), and also read it as the (distribution) law of X . Then by L(X n ) ⇒ L we n→∞

12.1 Convergence to the Normal Law (Central Limit Theorem, CLT)

understand the usual weak convergence of d.f.s, or convergence of r.v.s in distribution. d

That is, FX n ⇒ FX , or X n → X . n→∞

n→∞

Before we formulate the problem precisely and solve it, we notice that for kn = n, the array of the r.v.s in question becomes a triangular array; i.e., ⎧ X 11 ⎪ ⎪ ⎪ ⎪ ⎨ X 21 , X 22 ········· ⎪ ⎪ X n1 , X n2 , . . . , X nn ⎪ ⎪ ⎩ ·················· Furthermore, if X n j = X j , j = 1, . . . , n, and they are also identically distributed, then we have E X j = 0, σ 2 (X j ) = σ 2 , sn2 = nσ 2 , and ⎧ X1 ⎪ ⎪ ⎪ ⎪ ⎨ X 1, X 2 ······· ⎪ ⎪ X 1, X 2, . . . , X n ⎪ ⎪ ⎩ ···············

with Sn = nj=1 X n j = nj=1 X j . √ Since sn = σ n, then the problem of finding conditions under which n Sn j=1 X j =L L ⇒ N (0, 1) √ n→∞ sn σ n becomes the classical CLT. Going back to the problem we started out with, we have the r.v.s X n j , j = 1, . . . , kn → ∞, independent in each row, n→∞

E X n j = 0, σn2j = σ 2 (X n j ) = E X n2 j < ∞, sn2 =

kn

σn2j ,

j=1

Sn =

kn j=1

Then

E

Xnj sn

Xnj ,

Sn = sn

= 0 and

kn j=1 Xnj sn

kn j=1

Xnj . sn

σ

2

Xnj sn

kn 1 = 2 σn2j = 1. sn j=1

n

2 , we have Thus, replacing X n j by Z n j = j=1 σ (Z n j ) = 1 and therefore, without loss of generality, we may assume that sn2 = 1.

241

242


In order to summarize, we have then: For each n, the r.v.s X n j , j = 1, . . . , kn → ∞ are row-wise independent, n→∞

n E X n j = 0, σn2j = σ 2 (X n j ) < ∞, sn2 = kj=1 σn2j = 1.

n Set Sn = kj=1 X n j and find conditions under which L(Sn ) ⇒ N (0, 1). n→∞

The condition to be imposed upon the r.v.s X n j in order for L(Sn ) ⇒ N (0, 1) to n→∞ hold true is of the following nature: it is assumed that the contribution of each one of the summands X n j in the sum Sn is asymptotic negligible in the following sense. Definition 1.

The r.v.s X n j , j = 1, . . . , kn → ∞, are said to be uniformly asympn→∞

totic negligible (u.a.n.) if, for every ε > 0,

max P(|X n j | > ε) → 0.

n→∞

1≤ j≤kn

Actually, we shall make an assumption that will imply u.a.n. Namely, max σn2j → 0.

1≤ j≤kn

n→∞

That this assumption implies u.a.n. is seen thus max P(|X n j | > ε) ≤ max

1≤ j≤kn

E X n2 j

1≤ j≤kn

ε2

=

1 max σ 2 → 0. ε2 1≤ j≤kn n j n→∞

We are going to show the following result, which is the general form of the CLT, and from which other variants follow. Theorem 1 (Normal Convergence Criterion or Lindeberg–Feller Theorem). For each n, let the r.v.s X n j , j = 1, . . . , kn → ∞, be row-wise independent, assume that n→∞

n E X n j = 0 and σn2j = σ 2 (X n j ) < ∞, and set sn2 = kj=1 σn2j = 1 (by assumption). Then ⎛ ⎞ kn L⎝ X n j ⎠ ⇒ N (0, 1) and max σn2j → 0, j=1

n→∞

1≤ j≤kn

n→∞

if and only if, for every ε > 0, gn (ε) =

kn j=1 (|x|≥ε)

where Fn j is the d.f. of the r.v. X n j .

x 2 dFn j → 0, n→∞

This theorem is a special case of a more general result to be obtained in the next section. The proof of Theorem 1 and of some variations of it are given in Section 12.3.

12.1 Convergence to the Normal Law (Central Limit Theorem, CLT)

This section is concluded with three examples. The first illustrates the checking of the condition gn (ε) → 0; the second presents a case where the condition n→∞

gn (ε) → 0 is not fulfilled; and the third is meant to shed some light on what lies n→∞

behind the condition gn (ε) → 0. n→∞

Example 1.

Let Y1 , . . . , Yn be i.i.d. r.v.s with EY1 = μ ∈ and σ 2 (Y1 ) = σ 2 ∈

(0, ∞), and set X n j =

Y j −μ √ , σ n

j = 1, . . . , n. Then, for each n, X n1 , . . . , X nn are i.i.d.

n 2 = σ 2 = 1 , so that s 2 = σ 2 = 1 (and with E X n1 = 0, σ 2 (X n1 ) = σn1 nj n j=1 n j n n ⇒ N (0, 1) if and max1≤ j≤n σn2j = n1 → 0). Then by Theorem 1, L j=1 X n j n→∞

n→∞ n 2 only if gn (ε) = → 0 for every ε > 0, where Fn j = Fn j=1 (|x|≥ε) x dFn j (x) n→∞ is the d.f. of the X n j s. Actually, the asymptotic normality has been established in Application 2 of Chapter 11. So, it must hold that gn (ε) → 0 for every ε > 0. That n→∞ this is, indeed, the case is2 seen as follows: √ gn (ε) = n (|x|≥ε) x dFn (x) = n (|x|≥ε) x 2 dF(μ + xσ n), since Fn (x) = √ √ μ + xσ n) = F(μ + xσ n), where F is the d.f. of P(X n1 ≤ x) = P(Y1 ≤ √ the Y j s. By setting μ + xσ n = y, we get gn (ε) =

n

(y − μ)2 dF(y) σ 2 n (|y−μ|≥εσ √n) 1 1 = 2 (y − μ)2 dF(y) − 2 (y − μ)2 dF(y) σ σ (|y−μ| 0, as asserted.

It is to be observed, however, that Sn = an Tn + bn , where Tn = nj=1 Yn j ∼ √ √ B(n, pn ), an = 1/ npn qn , bn = −npn / npn qn , and that P(Tn = t) →

n→∞

P(T = t), where T ∼ P(λ), t = 0, 1, . . . (see, e.g., Theorem 1, Chapter 3, in Roussas √ def d (1997)). It follows (by Theorem 7 in Chapter 5) that an Tn + bn → √1 T − λ = λ n→∞ √ x X , where X takes on the values √x − λ with respective probabilities e−λ λx! , λ x = 0, 1, . . .. (Refer also to the Example 4 here, as well as Application 1 in Chapter 13.) Example 3. For j = 1, . . . , n, let X n j be independent r.v.s distributed as N (0, 1/n), so that E X n j = 0, σ 2 (X n j ) = σn2j = n1 , sn2 = 1, and max1≤ j≤n σn2j = n1 → 0. n→∞ , j = 1, . . . , n, to their sum Sn = Thus, the contribution of each one of the r.v.s X n j

n j=1 X n j is negligible. More precisely, the r.v.s X n j , j = 1, . . . , n, are u.a.n. Since Sn ∼ N (0, 1), it follows that gn (ε) → 0 for every ε > 0 (by Theorem 1). n→∞

In order to gain some insight as to why gn (ε) should converge to 0, and what this fact reflects, let us proceed with the computation of gn (ε). Here gn (ε) = n x 2 dFn (x), where Fn is the d.f. of X n1 (|x|≥ε)

12.2 Limiting Laws of L(Sn ) Under Conditions (C)

=n But

√ nx 2 n x 2 pn (x)d x, where pn (x) = √ e− 2 . 2π (|x|≥ε)

√ nx 2 n x pn (x)d x = √ x 2 e− 2 d x 2π (|x|≥ε) (|x|≥ε) √ nx 2 2 n x 2 e− 2 d x, = √ 2π (x≥ε)

2

and

(x≥ε)

x 2 e−

nx 2 2

dx =

ε − nε2 1 e 2 + n n

(x≥ε)

e−

nx 2 2

dx

(by integration by parts). Therefore,

However,

√

√ nx 2 2ε √ − nε2 2 n ne 2 + √ e− 2 d x. gn (ε) = √ 2π 2π (x≥ε)

ne−nε

2 /2

→ 0, and

n→∞

√ √ 2 2 n 1 − nx2 −x 2 /2(1/ n)2 e dx = 2 dx √ √ √ e 2π (x≥ε) 2π × (1/ n) (x≥ε) = 2P(X n1 ≥ ε) = P(|X n1 | ≥ ε) = max P(|X n j | ≥ ε) 1≤ j≤n

1 → 0. nε2 n→∞ So, gn (ε) consists essentially of the maximum probability of the individual X n j s being outside (−ε, ε), and this ought to be small, as the X n j s are concentrated around 0. Another way of looking at gn (ε) is to intepret it as the sum of the segments of the truncated variances of the X n j s taken outside (−ε, ε). Again, this quantity must be small, as the variances of the X n j s are close to 0. ≤

12.2 Limiting Laws of L(Sn ) Under Conditions (C) For reference, we gather together the basic assumptions made so far. ⎧ For each n, the r.v.s X n j , j = 1, . . . , kn → ∞, ⎪ ⎪ n→∞ ⎪ ⎨ are row-wise independent,

n (C) E X n j = 0, σn2j = σ 2 (X n j ) < ∞ and sn2 = kj=1 σn2j = 1; ⎪ ⎪ ⎪ ⎩ also, max 2 1≤ j≤kn σn j → 0. n→∞

(12.1)

245

246


The purpose of this section is the identification of the limiting laws of L(Sn ) under conditions (C). In all that follows, maxima, summations, and products are over 1 ≤ j ≤ kn , Fn j

n is the d.f. of X n j and f n j is the ch.f. of X n j . Also, Sn = kj=1 X n j , and by log z, z complex, we mean log p z (the principal branch of the log z). Limits and limit suprema are taken as n → ∞, unless otherwise specified. Our problem is that of finding the limiting laws (in the weak sense) of L(Sn ). The following lemma is established first. Lemma 1.

With the foregoing notation and under (C) in (12.1), one has {log f n j (t) − [ f n j (t) − 1]} → 0, t ∈ . j

Proof.

By Theorem 9 (iii) in Chapter 11, applied with n = 2, one has E Xnj 2 E Xnj (it) + θn j t 1! 2! σn2j 2 t , |θn j | = |θn j (t)| ≤ 1. = 1 + θn j 2 2

f n j (t) = 1 +

σ2 σ2 n j 2 nj 2 t ≤ t , | f n j (t) − 1| = θn j 2 2

Hence

so that max | f n j (t) − 1| ≤ j

(12.2)

t2 max σn2j → 0 (for each t). 2 j

Therefore, for each t ∈ and for all n ≥ n(t) (independent of j), one has | f n j (t) − 1| ≤

1 uniformly in j = 1, . . . , kn . 2

(12.3)

Next, for t as above and n ≥ n(t), we have, by means of (12.3) and Lemma 7 in Chapter 11, log f n j (t) = log{1 + [ f n j (t) − 1]} = [ f n j (t) − 1] + θn∗j [ f n j (t) − 1]2 for some θn∗j = θn∗j (t) with |θn∗j | ≤ 1. Thus, log f n j (t) − [ f n j (t) − 1] = θn∗j [ f n j (t) − 1]2 , and | log f n j (t) − [ f n j (t) − 1]| ≤ | f n j (t) − 1|2 ,

j = 1, . . . , kn ,

since |z 2 | = |z|2 for a complex number z (see also Exercise 10).

(12.4)


Hence, by employing (12.4) and (12.2), one has {log f n j (t) − [ f n j (t) − 1]} ≤ | log f n j (t) − [ f n j (t) − 1]| j j ≤ | f n j (t) − 1|2 j

≤ [max| f n j (t)−1|]

j

≤ [max | f n j (t) − 1|] j

t2

=

| f n j (t)−1|

j

t2 2 σn j 2 j

[max | f n j (t) − 1|].

2

(12.5)

j

On the other hand, as was seen earlier, max | f n j (t) − 1| ≤ j

t2 max σn2j → 0. 2 j

(12.6)

By means of (12.6), relation (12.5) then gives {log f n j (t) − [ f n j (t) − 1]} → 0, as was to be seen.

j

If sn2 ≤ c rather than equal 1, then the right-hand side in (12.5) is

Remark 1. bounded by Corollary.

t2 2

c max j | f n j (t) − 1| ≤

t2 2 2 c max j σn j

→ 0, so that the result is still true.

Under (C) and with sn2 = 1 replaced by sn2 ≤ c, one has log f n (t) − ψn (t) → 0, t ∈ ,

where f n (t) =

f n j (t) and ψn (t) =

j

Proof.

j

(eit x − 1)dFn j (x).

By independence, f n is the ch.f. of Sn and by the definition of f n , j

f n j (t) − 1 = =

log f n j (t) = log f n (t),

e

it x

dFn j (x) − 1 =

(eit x − 1)dFn j (x),

(12.7)

e

it x

dFn j (x) −

dFn j (x)

247

248


so that

[ f n j (t) − 1] =

j

j

(eit x − 1)dFn j (x) = ψn (t).

(12.8)

Then the result follows from the lemma (along with Remark 1) and (12.7) and (12.8). From the corollary, it follows that, if ψn (t) → ψ(t), where eψ(t) , t ∈ , is a ch.f. of a r.v. (see Proposition 2(i) below), then f n (t) → eψ(t) , t ∈ , and therefore L(Sn ) ⇒ to the law corresponding to eψ(t) . The converse is also true by Theorem 3 in Chapter 11 and continuity of log p z. Therefore, in order to find the limiting laws of L(Sn ) (in the weak convergence sense), it suffices to find the (pointwise) limits of ψn (t), t ∈ . This is the subject matter of this section and the content of Theorem 2 below. To this end, we have (eit x − 1)dFn j (x) = (eit x − 1 − it x)dFn j (x), ψn (t) =

j

since

Next,

j

it x dFn j (x) = it

2 eit x −1−it x → − t2 , x2 x→0

x dFn j (x) = itE X n j = 0.

by expanding eit x around t = 0 (see, e.g., discussion

following relation (11.18) in Chapter 11). Then by defining 2 − t2

eit x −1−it x x2

to be equal to

for x = 0 (i.e., by continuity), we get eit x − 1 − it x x 2 dFn j (x) ψn (t) = x2 j eit x − 1 − it x x 2 d y dFn j (y) = x2 −∞ j it x e − 1 − it x x 2 = d y dFn j (y) x2 −∞ j it x e − 1 − it x d K n (x), = x2

or

ψn (t) =

eit x − 1 − it x d K n (x), x2

K n (x) =

j

x −∞

y 2 dFn j (y).

(12.9)

In connection with the function K n defined in (12.9), we have Proposition 1.

The function K n defined in (12.9) is the d.f. of a r.v.

Proof. In fact, 0 ≤ K n ↑, clearly, K n (−∞) = 0 also clearly, and K n (∞) =

2

∞ 2 j −∞ y dFn j (y) = j σn j = 1. K n is also right-continuous. For this, it suffices


x to show that −∞ y 2 dFn j (y) is right-continuous. To this end, let x ↓ x0 . Then x0 x 2 2 y dFn j (y) − y dFn j (y) = y 2 dFn j (y) −∞ (x0 ,x] −∞ = y 2 I(x0 ,x] (y) dFn j (y),

and y 2 I(x0 ,x] → y 2 I (y) = 0, whereas y 2 I(x0 ,x] (y) ≤ y 2 x↓x0

independent of x and Fn j − integrable. Thus, the Dominated Convergence Theorem implies that y 2 I(x0 ,x] (y) dFn j (y) → 0, x↓x0

or equivalently,

x

−∞

y 2 dFn j (y) →

x0

x↓x0 −∞

y 2 dFn j (y).

Next, |eit x − 1 − it x| |eit x − 1| + |t x| 2|t x| 2|t| → 0 ≤ ≤ = x2 |x|2 |x|2 |x| 0 0. Then, by Lemma 4 (i), max j σn2j → 0, whereas by Lemma 4 (iii), L(Sn ) ⇒ N (0, 1). On the other hand, under conditions (C), if L(Sn ) ⇒ N (0, 1), then gn (ε) → 0 for every ε > 0; this is so by Lemma 5. The proof is complete. In reference to Example 2, we show that Theorem 1 does not provide the asymptotic distribution of L(Sn ). However, the asymptotic distribution of L(Sn ) may be obtained directly, as is done in the following example. As usual, all limits are taken as n → ∞. Example 4. In reference to Example 2, show directly, by means of ch.f.s, that √ L(Sn ) ⇒ L(cX √ + d), where the r.v. X is distributed as P(λ), X ∼ P(λ), c = 1/ λ, and d = − λ.

√ Indeed, recalling that Sn = nj=1 (Yn j − pn )/ npn qn , we have √ f Sn (t) = f nj=1 (Yn j − pn )/√npn qn (t) = f nj=1 (Yn j − pn ) (t/ npn qn ) = =

n j=1 n j=1

√ f Yn j − pn (t/ npn qn ) √ npn qn

e−i pn t/ √

= e−inpn t/ =e

npn qn

√ −inpn t/ npn qn

√ f Yn j (t/ npn qn )

√ [ f Yn1 (t/ npn qn )]n √

( pn eit/

npn qn

+ q n )n

(since f Yn1 (t) = ( pn eit + qn )n ) n √ √ npn eit/ npn qn /qn n −inpn t/ npn qn . 1+ = qn e n

255

256


However, qnn

−npn n = (1 − pn ) = 1 + → e−λ , n n

√

√ npn qn

e−inpn t/

and

→ e−iλt/

n→∞ √

1+

npn eit/

√ √ npn eit/ npn qn it/ λ . → λe qn √ √ it/ λ −λ e−it λ × eλe , which is it −λ

= e−it

n → eλe

n

n

√

√ it/ λ

λ

,

,

It follows that f Sn (t) → e−λ × e−it

since

X ∼ P(λ) with ch.f. eλe

npn qn /q

λ

the ch.f. of the r.v. cX + d =

√1 λ

√

λ

√ it/ λ

× eλe = √ X − λ, where

.

Observe that the result obtained here is consistent

with that to be arrived at in n Application 1 of Chapter 13, where it is shown that L j=1 Yn j ⇒ P(λ). As a consequence of Theorem 1, we have the following results. Theorem 3 (Liapounov Theorem). For each n ≥ 1, and with n → ∞ throughout, let the r.v.s X n j , j = 1, . . . , kn → ∞, be

row-wise independent and such that E X n j = 0 and σn2j = σ 2 (X n j ) < ∞, with sn2 = j σn2j = 1. Then, if for some δ > 0, E|X n j |2+δ → 0, (12.21) j

it follows that L(Sn ) ⇒ N (0, 1) and max σn2j → 0, where Sn = Proof.

j

j

(12.22)

Xnj .

By Theorem 1, it suffices to show that, for every ε > 0, gn (ε) = x 2 dFn j (x) → 0, j

(|x|≥ε)

where Fn j is the d.f. of X n j . We have E|X n j |2+δ = |x|2+δ dFn j (x) ≥ |x|2+δ dFn j (x) (−∞,∞) (|x|≥ε) 2 δ δ |x| |x| dFn j (x) ≥ ε x 2 dFn j (x), = (|x|≥ε)

so that

(|x|≥ε)

E|X n j |2+δ ≥ εδ gn (ε).

j

Therefore, relation (12.21) implies gn (ε) → 0, which ensures (12.22).

12.3 Conditions for the Central Limit Theorem to Hold

The theorem is true, in particular, if δ = 1; i.e., if

Remark 2.

E|X n j |3 → 0.

j

Corollary. replaced by

The conclusion of the theorem holds true if condition (12.21) is |X n j | ≤ Mn j (< ∞) a.s., max Mn j → 0. j

Proof.

We have

|X n j | = 3

j

X n2 j |X n j |

a.s. ≤ max Mn j X n2 j , j

j

j

so that, by taking expectation of both sides, we obtain E|X n j |3 ≤ max Mn j E X n2 j = max Mn j → 0, j

j

j

j

2

2 2 since j E Xnj = j σ (X n j ) = sn = 1. Then Remark 2 applies and gives the result. Now Theorem 1 can be generalized as follows. Theorem 4. For each n ≥ 1, let the r.v.s X n j , j = 1, . . . , kn → ∞, be row-wise

n σn2j . independent and such that E X n j = 0 and σn2j = σ 2 (X n j ) < ∞. Set sn2 = kj=1 Then σn2j Sn ⇒ N (0, 1) and max 2 → 0 , L sn sn if and only if, for every ε > 0, gn (ε) =

1 x 2 dFn j (x) → 0, sn2 (|x|≥εsn ) j

where Sn = Proof.

j

X n j and Fn j is the d.f. of X n j .

Set Yn j =

Xnj , sn

j = 1, . . . , kn .

Then for each n, the r.v.s Yn j , j = 1, . . . , kn , are independent (within each row) and such that EYn j = 0, τn2j = σ 2 (Yn j ) =

σn2j sn2

< ∞, and set τn2 =

kn j=1

τn2j = 1.

257

258


That is, the basic assumptions of Theorem 1 are satisfied for the r.v.s Yn j , j = 1, . . . , kn . Thus, by setting Sn Yn j = and gn (ε) = x 2 dG n j (x), Tn = sn (|x|≥ε) j

j

where G n j is the d.f. of Yn j , we obtain L(Tn ) ⇒ N (0, 1) and max j τn2j → 0, if and only if gn (ε) → 0 for every ε > 0. However, Xnj ≤ x = P(X n j ≤ xsn ) = Fn j (xsn ), G n j (x) = P(Yn j ≤ x) = P sn so that gn (ε) =

(|x|≥ε)

j

x 2 dG n j (x) =

j

(|x|≥ε)

x 2 dFn j (xsn )

1 = 2 y 2 dFn j (y) (by setting xsn = y) sn (|y|≥εsn ) j 1 = 2 x 2 dFn j (x). sn (|x|≥εsn ) j

So, L(Tn ) ⇒ N (0, 1) and max j τn2j → 0, if and only if 1 x 2 dFn j (x) → 0, sn2 (|x|≥εsn ) j

or

σn2j ⇒ N (0, 1) and max 2 → 0, if and only if sn 1 x 2 dFn j (x) → 0 for every ε > 0. gn (ε) = 2 sn (|x|≥εsn ) L

Sn sn

j

Specializing the result obtained to a single sequence of r.v.s, we have the following theorem. Theorem 5. Let the r.v.s X j , j = 1, . . . , kn , be independent and such that E X j = 0

n σ j2 . Then and σ j2 = σ 2 (X j ) < ∞. Set sn2 = kj=1 σ j2 Sn ⇒ N (0, 1) and max 2 → 0 , L (12.23) j sn sn if and only if, for every ε > 0, gn (ε) = where Sn =

1 x 2 dF j (x) → 0 , sn2 (|x|≥εsn )

(12.24)

j

j

X j and F j is the d.f. of X j .

12.3 Conditions for the Central Limit Theorem to Hold

Proof. For each n, set Yn j = independent,

Xj sn

, j = 1, . . . , kn . Then Yn j , j = 1, . . . , kn , are

EYn j = 0, τn2j = σ 2 (Yn j ) =

σ j2 sn2

< ∞, and τn2 =

kn

τn2j = 1.

j=1

Sn With Tn = j Yn j = sn and G n j = F j being the d.f. of Yn j , j = 1, . . . , kn , Theorem 1 applies to the present Yn j s and gives the result, since gn (ε) = x 2 dG n j (x) = x 2 dF j (xsn ) j

=

1 sn2

(|x|≥ε)

j

(|y|≥εsn )

j

(|x|≥ε)

y 2 dF j (y) (by setting xsn = y)

1 = 2 x 2 dF j (x). sn (|x|≥εsn )

j

Corollary. Let the r.v.s X j , j = 1, . . . , n, be i.i.d. with E X j = 0 and σ 2 = σ 2 (X j ) < ∞. Then relation (12.24) is satisfied, and so is (12.23), which here becomes σ2 Sn Sn 1 ⇒ N (0, 1) (and =L L = → 0). √ 2 sn nσ n σ n We have sn2 = nσ 2 , so that 1 1 2 n x dF(x) = 2 x 2 dF(x), gn (ε) = nσ 2 (|x|≥εσ √n) σ (|x|≥εσ √n) where F is the common d.f. of the X j s. Now, by the fact that x 2 dF(x) < ∞, and the relations Proof.

x 2 I(|x|≥εσ √n) (x) ≤ x 2 , n ≥ 1, x 2 I(|x|≥εσ √n) (x) → 0, the Dominated Convergence Theorem applies and gives (|x|≥εσ √n) x 2 dF(x) → 0. Hence the result. Remark 3. form.

The corollary just established provides the CLT in its most common

This section is concluded with an example providing a couple of useful results. Example 5. Let X 1 , . . . , X n be i.i.d. r.v.s with E X 1 = μ ∈ and Var(X 1 ) = σ 2 ∈ (0, ∞), and let X¯ n be the sample mean of the X j s (which is used for esti

2 mating μ), X¯ n = n1 nj=1 X j . Then E X¯ n = μ, Var( X¯ n ) = σn , and by the CLT, √ √ d n( X¯ n −μ) d −→ Z ∼ N (0, 1), or n( X¯ n − μ) → Y ∼ N (0, σ 2 ). σ n→∞

n→∞

259

260


When μ is known, the sample variance Sn2 of the X j s (which is used for estimating √

¯ d σ 2 ) is Sn2 = n1 nj=1 (X j − μ)2 , and n( XSnn −μ) −→ Y ∼ N (0, 1). Setting Y j = n→∞

(X j − μ)2 , the r.v.s Y1 , . . . , Yn are i.i.d. with EY1 = Var(X 1 ) = σ 2 and Var(Y1 ) = EY12 −(EY1 )2 = E(X 1 −μ)4 −σ 4 = μ4 −σ 4 , where μ4 is the fourth central moment of X 1 , assuming, of course, that E X 14 < ∞. So, E Sn2 = σ 2 and Var(Sn2 ) = (μ4 − σ 4 )/n. By the CLT again, √ √ n(Y¯n − σ 2 ) n(S 2 − σ 2 ) d = n → Z ∼ N (0, 1), μ4 − σ 4 μ4 − σ 4 n→∞ or

√ d n(Sn2 − σ 2 ) −→ Y ∼ N (0, μ4 − σ 4 ).

n→∞

If μ is unknown, the sample variance used for estimating σ 2 is S¯n2 = nj=1 (X j −

X¯ n )2 /(n −1). Then, as is easily seen, nj=1 (X j − X¯ n )2 = nj=1 (X j −μ)2 −n( X¯ n − 2 μ)2 , and hence E S¯n2 = 1 (nσ 2 − n × σ ) = σ 2 . Furthermore, n−1

S¯n2 = so that √

n( S¯n2 − σ 2 ) =

As n → ∞,

n

n n S2 − ( X¯ n − μ)2 , n−1 n n−1

√ √ n √ ¯ n √ σ2 n − [ n( X n − μ)]2 . n(Sn2 − σ 2 ) + n−1 n−1 n−1

n √ 2 n−1 n(Sn

d

− σ 2 ) → Y ∼ N (0, μ4 − σ 4 ), by the result obtained n→∞ √ √ 2√ n earlier and Theorem 7 (ii) in Chapter 5, σn−1n −→ 0, and n−1 [ n( X¯ n − μ)]2 −→ 0, √ d by the fact that n( X¯ n − μ) −→ Y ∼ N (0, σ 2 ), Example 3 in Chapter 11, and Theorem 7 (ii) in Chapter 5 again. Then, by the theorem just cited, it follows that √ d n( S¯n2 − σ 2 ) −→ Y ∼ N (0, μ4 − σ 4 ).

12.4 Proof of Results in Section 12.2 In this section, we provide the proofs of Proposition 2 and its three corollaries stated in Section 12.2, as well as the proof of Proposition 3, also stated in Section 12.2. For the proof of Proposition 2, the following lemma is needed which is also of independent interest. To this effect, recall that from Theorem 1 and Lemma 5 of Chapter 11, we have that if E|X n | < ∞, then dn f (t)|t=0 = i n E X n dt n and f n (t) is continuous on . The following result is also true (and is used in the proof of Proposition 2).

12.4 Proof of Results in Section 12.2

Lemma 6. Let X be a r.v. with d.f. F and ch.f. f . Then, if f (2n) (0) exists and is finite (in norm), it follows that E|X k | < ∞ for all k ≤ 2n and E X k = i1k f (k) (0). Proof. For a complex-valued function g defined on , set g(u) = g(u + h) − g(u −h), h ∈ , and define (n) g(u) recursively. Then it can be seen (see Exercise 7) that 2n (2n) r 2n f [(2n − 2r )h], (12.25) f (0) = (−1)

r r =0

and

(2n) f (0) 1 o(h 2n ) 1 (2n) = f (0) + = f (2n) (0) + 2n o(1) 2n 2n 2n (2h) 2 h 2 with o(1) −→ 0. h→0

(12.26)

By replacing f by what it is equal to, (12.25) becomes as follows: 2n (2n) r 2n

f (0) = (−1) ei[(2n−2r )h]x dF(x) r r =0 2n r 2n −i hx r i hx 2n−r (e = (−1) ) (e ) dF(x) r r =0 2n e−i hx − ei hx dF(x) = 2n ei hx − e−i hx dF(x) = 2n 2n 2n [2i sin(hx)] dF(x) = i 2 [sin(hx)]2n dF(x) = = (−1)n 22n [sin(hx)]2n dF(x).

Thus

sin(hx) 2n 2n

(2n) f (0) n = (−1) x dF(x). (12.27) (2h)2n hx Letting h → 0 and utilizing (12.26), (12.27) becomes sin(hx) 2n 2n (2n) n f (0) = (−1) lim x dF(x). (12.28) h→0 hx 2n Now 0 ≤ sin(hx) x 2n (here is where the even-order derivative is employed!). hx Thus, by the Fatou–Lebesgue Theorem (Theorem 2 in Chapter 5), sin(hx) 2n 2n sin(hx) 2n 2n x dF(x) = lim inf x dF(x) lim h→0 h→0 hx hx

261

262


sin(hx) 2n 2n x dF(x) hx h→0 sin(hx) 2n 2n = lim x dF(x) h→0 hx = x 2n dF(x) = E X 2n = (−1)n f (2n) (0),

≥

lim inf

since

sin t → 1. t t→0

From this result and (12.28), it follows that E X 2n < ∞ and hence

E|X |k < ∞ for all k ≤ 2n. That E X k = Chapter 11. Proof of Proposition 2.

1 ik

f (k) (0) follows from Theorem 1(v) in

Recall that it x e − 1 − it x d K (x), ψ(t) = x2

(12.29)

where K is a d.f. of bounded variation. (i) As was pointed out just prior to the formulation of the proposition, for each it x x t ∈ , the function e −1−it is well defined for all x ∈ and also continuous x2 and bounded. Thus, the integral in (12.29) is well defined (and finite in norm) as a Riemann–Stieltjes integral and is taken as the limit of Riemann–Stieltjes sums corresponding to any arbitrarily chosen collection of division points and any points in the corresponding intervals. More precisely, for an < 0 < bn with an → −∞ and bn → ∞ where, which here and in the sequel all limits are taken as n → ∞, consider the interval (an , bn ] and divide it into subintervals by the points an = xn0 < xn1 < · · · < xn,rn −1 < xnrn = bn (rn → ∞), which are chosen to be = 0 and such that max (xnk − xn,k−1 ) → 0.

1≤k≤rn

For each subdivision, we have the following corresponding Riemann–Stieltjes sum de f

Tn (t) =

rn eit xnk −1−it x k=1

2 xnk

nk

K (xn,k−1 , xnk ]

−K (x n,k−1 ,x nk ] i = t+ xnk k

K (xn,k−1 ,xnk ] it xnk (e 2 xnk

− 1) ,

(12.30)

where K (xn,k−1 , xnk ] stands for the variation of K over the interval (xn,k−1 , xnk ]. By setting K (xn,k−1 , xnk ] , βnk = xnk , and xnk K (xn,k−1 , xnk ] = (≥ 0), 2 xnk

αnk = − λnk


relation (12.30) becomes as follows: iαnk t + λnk (eiβnk t − 1) . Tn (t) =

(12.31)

k

Now, for each n, let Ynk , k = 1, . . . , rn , be independent (within each row) r.v.s such that Ynk is P(λnk ), the Poisson distribution with parameter λnk , so that its ch.f. is given by it f Ynk (t) = eλnk (e −1) . Set Z nk = αnk + βnk Ynk . Then the r.v.s Z nk , k = 1, . . . , rn , are independent (within each row) and the ch.f. of the r.v. Z nk is given by f Z nk (t) = eiαnk t eλnk (e Thus

iβnk t −1)

= eiαnk t+λnk (e

iβnk t −1)

!

iαnk t + λnk (eiβnk t − 1) exp Tn (t) = exp = is a ch.f. (that of the r.v.

.

"

k

e

iαnk t+λnk (eiβnk t −1)

k

Z nk ). But it x e − 1 − it x Tn (t) → d K (x) = ψ(t), x2

so that

k

de f

de f

f n (t) = e Tn (t) → eψ(t) = f (t). eit x −1−it x x2

(12.32) t2

Define g(t; x) by: g(t; x) = for x = 0, and g(t; x) = − 2 for x = 0. Restrict |t| ≤ 1 and look at g(t; x) as a function of x. Then, for x = 0, |g(t; x)| = it x e −1−it x |eit x −1|+|t x| 2 ≤ 2|tx 2x| ≤ |x| → 0 as |x| → ∞, and, for x = 0, ≤ x2 x2 |g(x; t)| ≤ 1, so that |g(· ; t)| is bounded by a bound independent of t (|t| ≤ 1) and K -integrable. Also, the integrand tends to 0, as t → 0, for all x ∈ . Then, as t → 0, the Dominated Convergence Theorem yields ψ(t) → ψ(0) = 0, so that ψ is continuous at 0 and then so is f (t). Thus, we have that f n are ch.f.s of r.v.s and, by (12.32), f n → f in with f being continuous at the origin. Then the converse part of the Paul Lévy Continuity Theorem (Theorem 3 in Chapter 11) implies that f is the ch.f. of a uniquely determined d.f. of a r.v., call it X . This establishes part (i). Restrict attention to |t| ≤ 1, and observe that, for x = 0, | ∂t∂ g(t; x)| = it(ii) e x −1 |t x| i x ≤ |x| = |t| ≤ 1 independent of t and K -integrable. Since

ψ(t) =

g(t; x)d K (x) =

{0}

g(t; x)d K (x) +

−{0}

g(t; x)d K (x)

263

264


eit x − 1 − it x v0 2 d K (x), t + 2 x2 −{0} v0 = K (0) − K (0−), =−

Theorem 5 in Chapter 5 applies and yields eit x − 1 − it x d d K (x) ψ (t) = −v0 t + dt −{0} x2 ∂ eit x − 1 − it x d K (x) = −v0 t + x2 −{0} ∂t it x e −1 i d K (x). = −v0 t + x −{0}

(12.33)

In particular,

ψ (0) = 0. (12.34) 2 it x Next, for x = 0, | ∂t∂ 2 g(t; x)| = ∂t∂ i e x−1 = | − eit x | = 1, again independent of t and K -integrable. Then by the theorem just cited and (12.33), it x e −1 d i d K (x) ψ (t) = −v0 + dt −{0} x it x e −1 ∂ i d K (x) = −v0 + x −{0} ∂t it x = −v0 − e d K (x) = − eit x d K (x). (12.35) −{0}

In particular,

ψ (0) = −Var K .

(12.36)

Now, eψ(t) = f (t) is the ch.f. of the r.v. X , whose second-order derivative at 0 is eψ(t) |t=0 = ψ (t)eψ(t) |t=0 = ψ (t)eψ(t) + [ψ (t)]2 eψ(t) |t=0 = ψ (0) = −Var K finite, by (12.34) and (12.36) and the fact that ψ(0) = 0. Then Lemma 6 applies and gives 1 ψ(t) e |t=0 = ψ (t)eψ(t) |t=0 = 0, EX = i and

1 ψ(t) e |t=0 = −ψ (0) = Var K (by (12.36)). i2 The proof of the proposition is completed. E X2 =


Proof of Corollary 1.

From (12.29), we get

ψ(t) = n where, for each n,

eit x − 1 − it x K (x) , d x2 n ψ

K n

is a d.f. of bounded variation. Then, by Proposition 2(i), e n ψ

is the ch.f. of a r.v. Let X n j , j = 1, . . . , n, be row-wise i.i.d. r.v.s with ch.f. e n . ψ

Then the ch.f., f n , of Sn = nj=1 X n j is f n = nj=1 e n = eψ , so that L(Sn ) = L(corresponding to eψ ). It remains for us to check conditions (C). By Proposition ψ d ψ(t)/n d 2 ψ(t)/n e |t=0 = 0, dt |t=0 = 2(ii), applied with e n rather than eψ , we have dt 2e −Var Kn , by (12.34) and (12.36), so that the first and second moments of the distribution ψ

corresponding to the ch.f. e n are 0 and Var Kn , respectively. Thus, E X n j = 0, and σn2j = σ 2 (X n j ) = Var Kn = n1 Var K = n1 , j = 1, . . . , n, so that max σn2j = n1 → 0, j

and sn2j = j σn2j = 1. Proof of Corollary 2. Actually, the only point so far where the condition sn2 = 1 was used was in the proof of Lemma 1, which, however, also holds true under the K is condition sn2 ≤ c (see Remark 1). So, if K is a d.f. with Var K = v, then nv ψ

K also a d.f. with Var nv = n1 . Hence e n is the ch.f. of row-wise i.i.d. r.v.s X n j , j =

1, . . . , n, with E X n j = 0 and σ 2 (X n j ) = Var Kn = nv . It follows that j σn2j = v and max j σn2j = nv → 0.

Proof of Corollary 3. (i) For an arbitrary t ∈ , consider, e.g., the interval [t −1, t +1] and apply Theorem 5 in Chapter 5 to obtain, as in relation (12.35), ψ (t) = − eit x d K (x).

(ii) Let K ∗ be another d.f. as described in part (ii), and such that

eit x − 1 − it x d K ∗ (x) = ψ(t), t ∈ . x2

Then, as in part (i), ψ (t) = −

or − ψ (t) =

eit x d K (x) = −

eit x d K (x) =

eit x d K ∗ (x),

eit x d K ∗ (x).

(12.37)

265

266


In particular, de f

− ψ (0) = Var K = Var K ∗ = v0 ,

(12.38)

so that, by (12.37), − Thus, the d.f.s

K v0

and

ψ (t) = v0 K∗ v0

eit x d

K (x) = v0

eit x d

K ∗ (x) . v0

(12.39)

K K∗ v0 (−∞) = v0 (−∞) = 0. Since, ∗ that vK0 = Kv0 in , or K = K ∗ in .

have variation 1 and

(12.39), they have the same ch.f., it follows

by

Proof of Proposition 3. (i) As in the proof of Proposition 2(i), let g(t; x) = (eit x − 1 − it x)/x 2 for x = 0, and (see discussion following relation (12.32)) g(t; x) = −t 2 /2 for x = 0. Then for x = 0, |g(t; x)| ≤ 2|t| |x| → 0 as x → ∞, so that, for each t ∈ , g(t; x) is bounded and continuous in as a function of x. Thus, if K n ⇒ K , a d.f. (with Var K ≤ c), then the Helly–Bray Extended Lemma (Theorem 7 in Chapter 8; see also Remark 4 in the same chapter) applies and yields ψn (t) = g(t; x) d K n (x) → g(t; x) d K (x) = ψ(t), t ∈ . (12.40)

(ii) Consider the sequence {K n } and apply the Weak Compactness Theorem (Theorem 5 in Chapter 8) to obtain {K m } ⊆ {K n } such that K m ⇒ K ∗ , some d.f. m→∞

(with Var K ∗ necessarily ≤ c). Define ψ ∗ by it x e − 1 − it x ψ ∗ (t) = d K ∗ (x), t ∈ . 2 x

Then, by part (i), ψm → ψ ∗ in . Since also ψm → ψ in , it follows that m→∞

m→∞

ψ ∗ = ψ in . Next, let {K r } ⊆ {K n } distinct from {K m } such that K r ⇒ K ∗∗ , some d.f. (of variation necessarily ≤ c), and define ψ ∗∗ by it x e − 1 − it x ψ ∗∗ (t) = d K ∗∗ (x), t ∈ . x2

r →∞

Then, as before, ψr → ψ ∗∗ in , whereas ψr → ψ in also, so that ψ ∗∗ = ψ r →∞

r →∞

in . It follows that ψ ∗∗ = ψ ∗ (= ψ) in . On the other hand, by relation (12.37), −ψ (t) = eit x d K ∗ (x) = eit x d K ∗∗ (x),

so that the d.f.s K ∗ and K ∗∗ have the same ch.f. Then, by the inversion formula (Theorem 2 in Chapter 11) it follows that K ∗ = K ∗∗ +C, for a constant C. Hence (by Definition 3 in Chapter 11) K n ⇒ K uac, some d.f. K (with Var K ≤ c), and all of them determine the same ψ.


Exercises. 1. If the independent r.v.s X j , j ≥ 1, are distributed as U (− j, j), then show that the Lindeberg condition (see relation (12.24)) holds, so that L( Ssnn ) ⇒ N (0, 1), n→∞

where Sn = nj=1 X j and sn2 = nj=1 σ j2 , σ j2 = Var(X j ). Hint: Recall that X ∼ U (α, β) means that the r.v. X has the uniform distribution with parameters α and β (α < β), its probability density function is p(x) =

n α+β (α−β)2 1 2 j=1 j = β−α I[α,β] (x), E X = 2 and Var(X ) = 12 . Finally, recall that n(n + 1)(2n + 1)/6. 2. If the independent r.v.s X j , j ≥ 2, are distributed as follows: P(X j = − j α ) = P(X j = j α ) = = 1−

2 jβ

1 , jβ

P(X j = 0)

(α, β > 0),

show that the restriction β < 1 ensures that the Lindeberg condition (relation (12.24)) holds. Hint: Show that the restriction β < 1 ensures that the set of js with j = 1, . . . , n, and | ± j α | ≥ εsn , is empty for all ε > 0, for large n. For an arbitrary, but fixed β(< 1), it is to be understood that j ≥ j0 , where j0 = 21/β , if 21/β is an integer, or j0 = [21/β ] + 1 otherwise. This ensures that 1 − j2β is nonnegative.

3. Let the r.v.s X j , j ≥ 1, be distributed as follows: P(X j = ± j α ) =

1 , 6 j 2(α−1)

P(X j = 0) = 1 −

1 , 3 j 2(α−1)

α > 1. Show that the Lindeberg condition (relation (12.24)) holds, if and only if α < 23 .

Conclude that L( Ssnn ) ⇒ N (0, 1), where Sn = nj=1 X j and sn2 = nj=1 σ j2 , n→∞

σ j2 = Var(X j ). Hint: For α < 23 , show that j 2α < ε2 sn2 , j = 1, . . . , n, which is implied by n 2α
0 and large n: [(εsn )1/2 ], in order to conclude that gn (ε) 0. n→∞

5. Let X n , n ≥ 1, be independent r.v.s such that |X n | ≤ Mn a.s. with Mn = o(sn )

where sn2 = nj=1 σ j2 → ∞ and σ j2 = Var(X j ). n→∞

Set Sn = nj=1 X j and show that L

Sn − E Sn sn

⇒ N (0, 1).

n→∞

Hint: From the assumption Mn = o(sn ), it follows that εsn , n > n 0 (= n(ε)), ε > 0. Write

Mn → 0, so that sn n→∞

Mn
n 0 ),

j=n 0 +1

and since the first term tends to 0, as n → ∞, work with the second term only. To this end, set Yn j = (X j − E X j )/τn , where τn2 = sn2 − sn20 , and show that the r.v.s Yn j , j = n 0 + 1, . . . , n, satisfy the Liapounov condition (for δ = 1) (see Theorem 3). 6. Let X n j , j = 1, . . . , kn → ∞ be row-wise independent r.v.s with E X n j = 0,

n σn2j = 1, and max1≤ j≤kn σn2j → 0. Then, σ 2 (X n j ) = σn2j < ∞, sn2 = kj=1 n→∞

n with Sn = kj=1 X n j , and under the assumption that L(Sn ) ⇒ N (0, 1), show

n P X n2 j → 1. that kj=1 n→∞

Hint: By Theorem 1, it follows that, for every ε > 0, gn (ε) = j (|x|≥εsn ) x 2 dFn j → 0, where Fn j is the d.f. of X n j . One way of proceeding is to show that n→∞ $

2# → 0, j P(|X n j | ≥ ε) → 0, j σ 2 X n2 j I(X 2 0). Then, the corresponding ψ¯ is given by it x e − 1 − it x ¯ ψ(t) = d K (x) = (eit − 1 − it)λ. x2 In (13.21), take α = λ. Then the ψ that corresponds to (λ, K ) is ψ(t) = itλ + (eit − 1 − it)λ = λ(eit − 1),

(13.34)

where we observe that eψ(t) is the ch.f. of P(λ), the Poisson distribution with parameter λ. Then, we have the following theorem. Theorem 2. For each n ≥ 1, let the r.v.s X n j , j = 1, . . . , kn → ∞, be rowwise independent with E X n j = αn j and σ 2 (X n j ) = σn2j finite, and such that αn = k n k n k n 2 2 j=1 αn j → λ, sn = j=1 σn j → λ (0 < λ < ∞). Set Sn = j=1 X n j . Then L(Sn ) ⇒ P(λ), and max σn2j → 0,

(13.35)

1≤ j≤kn

if and only if, for every ε > 0, h n (ε) =

kn j=1 (|x−1|≥ε)

x 2 dFn j (x + αn j ) → 0.

(13.36)

The proof of Theorem 2 is facilitated by the following lemmas. Lemma 3. For each n ≥ 1, let X n j , j = 1, . . . , kn → ∞, be row-wise independent r.v.s with E X n j = αn j and σ 2 (X n j ) = σn2j finite, and such that αn =

kn

αn j → λ ∈ , sn2 =

j=1

kn

σn2j → λ.

j=1

n Let Fn j be the d.f. of X n j , and set Sn = kj=1 X n j . Finally, suppose that, for every ε > 0, kn h n (ε) = x 2 dFn j (x + αn j ) → 0. (13.37) j=1 (|x−1|≥ε)

Then, (i) max1≤ j≤kn σn2j → 0, (so that conditions (C ) in (13.2) are satisfied).

13.3 Two Special Cases of the Limiting Laws of L(Sn )

(ii) With K¯ n and K defined by (13.32) and (13.33), respectively, it holds K¯ n ⇒ K . (iii) L(Sn ) ⇒ L, where L is the Poisson distribution with parameter λ. Proof. (i) Clearly,

σn2j = = ≤

(x − αn j )2 dFn j (x) =

y 2 dFn j (y + αn j ) (by setting x − αn j = y) x 2 dFn j (x + αn j ) + x 2 dFn j (x + αn j )

(|x−1|≥ε) h n (ε) + ε2 .

(|x−1| 0, K¯ n (1 − ε) → 0 and K¯ n (1 + ε) → λ. Indeed, ¯ x 2 dFn j (x + αn j ) ≤ h n (ε) → 0, K n (1 − ε) = (−∞,1−ε]

j

whereas 0 ≤ sn2 − =

j

≤

(−∞,1+ε]

j

(1+ε,∞)

j

[1+ε,∞)

x 2 dFn j (x + αn j )

x 2 dFn j (x + αn j ) x 2 dFn j (x + αn j ) ≤ h n (ε) → 0.

Therefore, since sn2 → λ, it follows that K¯ n (1 + ε) = j

(−∞,1+ε]

x 2 dFn j (x + αn j ) → λ.

(iii) By part (i), K¯ n ⇒ K , and by assumption αn → λ. Then Theorem 2 (ii) applies and gives that L(Sn ) ⇒ L, where the d.f. L has ch.f. eψ(t) where ψ(t) = ¯ iλt + ψ(t) = iλt + (eit − 1 − it)λ = λ(eit − 1), which is the ch.f. of the Poisson distribution with parameter λ. Lemma 4. For each n ≥ 1, let X n j , j = 1, . . . , kn → ∞, be row-wise independent r.v.s with E X n j = αn j and σ 2 (X n j ) = σn2j finite, and such that αn =

kn j=1

αn j → λ ∈ , sn2 =

kn j=1

σn2j → λ.

283

284

CHAPTER 13 The Central Limit Problem: The Noncentered Case

Set Sn =

k n

j=1 αn j

and suppose that L(Sn ) ⇒ P(λ), and max σn2j → 0. 1≤ j≤kn

Thus the convergence in (13.36) holds for every ε > 0. Proof. Since αn → λ and L(Sn ) ⇒ P(λ), Theorem 2 (i) implies that K¯ n ⇒ K uac. Next, x 2 dFn j (x + αn j ) h n (ε) = (|x−1|≥ε)

j

=

(−∞,1−ε]

j

+

j

≤


[1+ε,∞)

(−∞,1−ε]

j

+



j

(1+ 2ε ,∞)


ε . = K¯ n (1 − ε) + sn2 − K¯ n 1 + 2 Thus, as in the proof of Lemma 2, passing to a subsequence {r } ⊆ {n} for which K¯ r ⇒ K ∗ a d.f. with K ∗ = K + c, for some constant c, we have r →∞

ε ε → K ∗ (1 − ε) + λ − K ∗ 1 + K¯ r (1 − ε) + sr2 − K¯ r 1 + 2 r →∞ 2 = K (1 − ε) + c + λ ε −K 1 + −c 2 = 0 + λ − λ = 0,

so that h r (ε) → 0, and then h n (ε)→0.

Proof of Theorem 2.

r →∞

It follows from Lemmas 3 and 4.

As an application to Theorem 2, we present the familiar result, according to which Binomial probabilities are approximated by Poisson probabilities. Application 1 (Approximation of a Binomial distribution by a Poisson distribution). For n ≥ 1, let X n j , j = 1, . . . , n, be row-wise independent r.v.s distributed as with parameter pn ), such that pn → 0 and npn → λ ∈ (0, ∞). B(1, pn ) (Bernoulli Set Sn = nj=1 X n j (so that Sn is distributed as B(n, pn ), Binomial with parameters n and pn ). Then L(Sn ) ⇒ L(X ), where X is distributed as P(λ) (Poisson with parameter λ).


First, we verify nthat the conditions of Theorem 2 are satisfied here. Indeed, E X n j = with λ ∈ (0, ∞). Next, pn and αn = j=1 αn j = npn → λ, by assumption, σn2j = σ 2 (X n j ) = pn qn , where qn = 1 − pn . Also, sn2 = nj=1 σn2j = npn qn → λ, since npn → λ and qn → 1. Then, in order to show that L(Sn ) ⇒ L(X ) (and max j σn2j → 0), it suffices to show that, for every ε > 0, h n (ε) =

j

(|x−1|≥ε)

x 2 dFn j (x + pn ) → 0,

where Fn j = Fn is the d.f. of X n j . Since Fn (x + αn j ) = Fn (x + pn ), and Fn is given by ⎧ ⎨ 0 for x < 0, Fn (x) = qn for 0 ≤ x < 1, ⎩ 1 for x ≥ 1, we have

⎧ ⎨ 0 for x < − pn , Fn (x + pn ) = qn for − pn ≤ x < 1 − pn = qn , ⎩ 1 for x ≥ 1 − pn = qn , Fn (x + pn )

1

}p

n

qn

x = −pn

0

|

1 − ε qn

|

1

|

1+ε

x

285

286

CHAPTER 13 The Central Limit Problem: The Noncentered Case

(see figure). We have h n (ε) =

(|x−1|≥ε)

j

=n =n

(|x−1|≥ε)


x 2 dFn (x + pn )

(−∞,1−ε]

x 2 dFn (x + pn )

+n =n

[1+ε,∞)

(−∞,1−ε]

x 2 dFn (x + pn ) x 2 dFn (x + pn )

(since Fn (x + pn ) = 1 for x ≥ 1 + ε) = npn2 qn (since qn > 1 − ε for sufficiently large n, so that the only value taken by x with positive probability is − pn ) = npn × pn qn ≤ npn × pn → λ × 0 = 0. Thus, h n (ε) → 0, and therefore L(Sn ) ⇒ P(λ). (Also, max1≤ j≤n σn2j = pn qn ≤ pn → 0.) As a consequence of it, we have n x n−x λx pn qn → e−λ , x = 0, 1, . . . x x! In fact, for x = 0, 1, . . ., take x1 = x − 21 and x2 = x + 21 , so that x1 and x2 are continuity points of the d.f. F of X ∼ P(λ). Thus, FSn (xi ) → FX (xi ), i = 1, 2. On the other hand, P(Sn = x) = P(Sn ≤ x2 ) − P(Sn ≤ x1 ), and likewise P(X = x) = P(X ≤ x2 ) − P(X ≤ x1 ). Thus, P(Sn = x) = P(Sn ≤ x + 0.5) − P(Sn ≤ x − 0.5) → P(X ≤ x + 0.5) − P(X ≤ x − 0.5) = P(X ≤ x) − P(X ≤ x − 1) = P(X = x) = e−λ or

n x

λx , x!

pnx qnn−x → e−λ λx! . x

Exercises. 1. Under the assumption of Theorem 1, the convergence in (13.29) holds if and only if (13.30) holds for every ε > 0. With the notation X¯ n j = X n j − αn j and X¯ n j having d.f. F¯n j , and S¯n = nj=1 X¯ n j , show that the convergence in (13.29) follows from the condition kn x 2 d F¯n j (x) → 0 for every ε > 0. j=1 (|x|≥εsn )

n→∞


2. In Theorem 1 of this chapter, as well as Theorem 1 of Chapter 12, under some basic assumptions on the r.v.s X n j , j = 1, . . . , kn , given in the theorems cited, a necessary and sufficient condition is given (namely, gn (ε) −→ 0 for every ε > 0) n→∞ so that L(Sn ) ⇒ N (μ, σ 2 ), μ ∈ , σ 2 ∈ (0, ∞), n→∞ k n where Sn = j=1 X n j . Thus, P(Sn ≤ x) = FSn (x) −→ (x; μ, σ 2 ), x ∈ , n→∞

where (x; μ, σ 2 ) is the d.f. of the

N (μ, σ 2 ) distribution. Actually, the preceding convergence is uniform in x ∈ . That is, for every ε > 0 there exists N (ε) > 0, independent of x ∈ , such that n ≥ N (ε) implies |FSn (x) − (x; μ, σ 2 )| < ε for all x ∈ .

(∗ )

This result is a consequence of the following fact (known as Pólya’s Lemma). Namely, if F and {Fn } are d.f.s of r.v.s such that Fn (x) −→ F(x), x ∈ , and F n→∞ is continuous, then the convergence is uniform in x ∈ . Establish this fact. Hint: Refer to Exercise 9 in Chapter 8. See also Lemma 1, page 206, in Roussas (1997). Remark: The uniformity asserted in (*) is what legitimizes as simple an approximation as that in the Binomial case. Thus, if X 1 , . . . , X n are independent, distributed as B(1, p), and if a and b are with for sufficiently large

0≤

a < b ≤ n, then, integers n b−np a−np √ − , where S = n, P(a < Sn ≤ b) √ n j=1 X j , q = 1− p, and npq npq

b−np n √ (x) = (x; 0, 1). This is so, because P(Sn ≤ b) = P Z ≤ j j=1 npq , where

X − p b−np b−np n √ √ Z j = √1n √j pq , j = 1, . . . , n, and P j=1 Z j ≤ npq − npq < ε for n ≥ N (ε) independently of

b−np √ npq ;

and likewise for P(Sn ≤ a).

287

CHAPTER

Topics from Sequences of Independent Random Variables

14

This chapter consists of a selection of topics from sequences of r.v.s. The r.v.s considered are most often independent, although not necessarily identically distributed. The central result is the statement and proof of the Strong Law of Large Numbers (SLLN), Theorem 7. It is important to notice that all results prior to Theorem 7, consisting of six theorems and two lemmas, are used directly or indirectly in the proof of the SLLN. This may serve as a tribute to its depth and complexity. The chapter opens with the Kolmogorov inequalities, the detailed discussion of which takes up the entire first section. Kolmogorov inequalities provide upper and lower bounds for the probability of the maximum of the absolute value of partial sums of independent r.v.s. They are instrumental, albeit indirectly, in the proof of the SLLN. Section 14.2 is devoted to discussing four theorems. Theorem 2 gives a suf ficient condition, in terms of variances, for the series n≥1 (X n − E X n ) to converge a.s. Theorems 3 and 4 taken together state that, if n≥1 P(An ) < ∞, then P(lim supn→∞ An ) = 0, and if the former sum is ∞, then the latter probability is 1. The first conclusion (Borel–Cantelli Lemma) holds for any events, whereas the second conclusion (Borel Zero–One Criterion) requires independence. Theorem 5 discusses some technical consequences of the assumption that n≥1 P(X n = X n ) < ∞ for two sequences of r.v.s. The following two lemmas, Lemma 3 (Toeplitz) and Lemma 4 (Kronecker), are analytic results and lend themselves here and in many other situations. In Section 14.3, a precursor to the SLLN, Theorem 6, is discussed, as well as a lemma providing bounds for the E|X | of a r.v. X in terms of sums of certain probabilities, and, finally, the SLLN itself is proved. In the following section, it is shown that the SLLN essentially holds, even if the expectation of the underlying r.v.s is infinite. If, however, this expectation does not exist, then the averages Sn /n are unbounded with probability one. In the final section, two main results are discussed. One is the Kolmogorov Zero– One Law, which states that tail events defined on a sequence of independent r.v.s have probability either 0 or 1. This conclusion has important ramifications regarding the limits of sequences, series, etc., of independent r.v.s. The other result gives necessary and sufficient conditions for the a.s. convergence of a series of independent r.v.s, and because of its form, it is referred to as the three series criterion. These main results An Introduction to Measure-Theoretic Probability, Second Edition. http://dx.doi.org/10.1016/B978-0-12-800042-7.00014-1 Copyright © 2014 Elsevier Inc. All rights reserved.

289

290

CHAPTER 14 Topics from Sequences of Independent Random Variables

are obtained after the concept of the tail σ -field is introduced and a related result is established.

14.1 Kolmogorov Inequalities In this chapter, we consider certain topics related to sequences of independent r.v.s with a view of establishing the Strong Law of Large Numbers (SLLN) (Theorem 7). We start with Kolmogorov inequalities, which provide upper and lower bounds for probabilities of the maximum of partial sums of r.v.s centered at their expectations. Theorem 1 (Kolmogorov inequalities). Let X j , j = 1, . . . , n, be independent r.v.s (not necessarily i.d.), and let E X j be finite and |X j | ≤ c, where c is finite or ∞. Then, for every ε > 0, one has n 2 (ε + 2c)2 j=1 σ j 1 − n ≤ P max |S − E S | ≥ ε ≤ , (14.1) k k 2 1≤k≤n ε2 j=1 σ j where Sn =

n j=1

X j and σ j2 = σ 2 (X j ).

We observe that σ j2

∞ if and only if E X 2j

∞. Thus, if σ j2

Remark 1. = = = ∞ for some j, then c = ∞. Then the right-hand side of (14.1) becomes ∞ and the left-hand side becomes 0, interpreting ∞ ∞ as 1. Thus (14.1) is trivially true, and we may therefore 2 assume that σ j < ∞ for all j. Next, if c = ∞, the left-hand side of (14.1), where c appears, is −∞ and therefore we may assume that c < ∞, since the proof of the right-hand side of the inequality does not depend on the finiteness or not of c, as will be seen (see Lemma 1). Under this assumption, we have |X j | ≤ c, so that |E X j | ≤ c and therefore |X j − E X j | ≤ 2c. Thus, if we set Y j = X j − E X j , j = 1, . . . , n, then the Y j s are independent, EY j = 0, and |Y j | ≤ c∗ (= 2c) < ∞. Therefore, in carrying out the proof of the theorem, we may assume that E X j = 0 and |X j | ≤ 2c < ∞ (or work with the Y j s). The proof of the theorem is rather long, in particular the proof of the left-hand-side inequality. It might then be appropriate to split it in two parts in the form of lemmas. Lemma 1. Under the assumptions of Theorem 1 and the additional nonrestrictive assumption that E X j = 0 (but without assuming finiteness of the constant c), the right-hand side of (14.1) holds. Proof.

For ε > 0, we set Ak = max |S j | ≥ ε , k = 1, . . . , n, 1≤ j≤k

and

Bk =

A0 = ,

(14.2)

max |S j | < ε and |Sk | ≥ ε , k = 2, . . . , n,

1≤ j≤k−1

B1 = (|S1 | ≥ ε) = A1 , S0 = 0.

(14.3)

14.1 Kolmogorov Inequalities

Then it is clear that Bk ∩ Bl = , for k = l, and Aj =

j

Bk ,

j = 1, . . . , n.

(14.4)

k=1

Indeed, if C j = (|S j | ≥ ε), j = 1, . . . , n, then n

c An = ∪ C j = C1 + (C1c ∩ C2 ) + · · · + (C1c ∩ · · · Cn−1 ∩ Cn ) j=1

= B1 + B2 + · · · + Bn . Next,

Bk

Sn2 d P

=

Sn2 I Bk d P = E(Sn2 I Bk ) = E I Bk [Sk + (Sn − Sk )]2

= E[(I Bk Sk ) + I Bk (Sn − Sk )]2 = E(I Bk Sk2 ) + 2E[I Bk Sk (Sn − Sk )] + E[I Bk (Sn − Sk )2 ].

(14.5)

Clearly, Bk ∈ σ (S1 , . . . , Sk ), so that I Bk is σ (S1 , . . . , Sk )-measurable. It follows that I Bk Sk is σ (S1 , . . . , Sk )-measurable. Also, I Bk Sk and Sn − Sk are independent, since the former is defined in terms of the r.v.s X j , j = 1, . . . , k, whereas the latter is defined in terms of the r.v.s X j , j = k + 1, . . . , n. It follows that E[I Bk Sk (Sn − Sk )] = E(I Bk Sk )E(Sn − Sk ) = 0, so that (14.5) becomes Sn2 d P = E(I Bk Sk2 ) + E[I Bk (Sn − Sk )2 ] ≥ E(I Bk Sk2 ) Bk = Sk2 d P ≥ ε2 P(Bk ) by means of (14.3); i.e., B k Sn2 d P ≥ ε2 P(Bk ). Bk

Adding over k = 1, . . . , n, and taking into consideration relation (14.4), we get then Sn2 d P ≥ ε2 P(An ). (14.6) An

But

An

Therefore

Sn2 d P ≤

Sn2 d P = E Sn2 =

σ j2 by independence.

j

σ j2

≥ ε P(An ) and hence P(An ) ≤ 2

j

which is the right-hand side of the required inequality.

j

σ j2

ε2

,

291

292


Lemma 2. Under the assumptions of Theorem 1 (including boundedness, |X j | ≤ c(< ∞), j = 1, . . . , n), the left-hand side of (14.1) holds. Proof.

From (14.2) and (14.3), it follows that Bk ⊆ Ak , Ak−1 ∩ Bk = and, by (14.4), Ak = (B1 + · · · + Bk−1 ) + Bk

(14.7)

= Ak−1 + Bk , so that Ak−1 = Ak − Bk , and I Ak−1 = I Ak − I Bk . Thus Sk−1 I Ak−1 + X k I Ak−1 = Sk I Ak−1 = Sk I Ak − Sk I Bk , by means of (14.7). Therefore, squaring out and taking the expectations, we obtain 2 I Ak−1 ) + E(X k2 I Ak−1 ) = E(Sk2 I Ak ) − E(Sk2 I Bk ), E(Sk−1

(14.8)

because E(Sk−1 I Ak−1 X k I Ak−1 ) = E[(Sk−1 I Ak−1 )X k ] = E(Sk−1 I Ak−1 )E X k = 0, and E(Sk I Ak Sk I Bk ) = E(Sk2 I Ak I Bk ) = E(Sk2 I Bk ) since Bk ⊆ Ak . Now I Ak−1 is defined in terms of X 1 , . . . , X k−1 and hence it is independent of X k . Therefore E(X k2 I Ak−1 ) = E X k2 E I Ak−1 = E X k2 P(Ak−1 ) = σk2 P(Ak−1 ). Then (14.8) becomes 2 I Ak−1 ) + σk2 P(Ak−1 ) = E(Sk2 I Ak ) − E(Sk2 I Bk ). E(Sk−1

(14.9)

But Sk I Bk = Sk−1 I Bk + X k I Bk , and hence |Sk I Bk | ≤ |Sk−1 I Bk | + |X k I Bk | < ε I Bk + 2cI Bk = (ε + 2c)I Bk ,

(14.10)

since |Sk−1 | < ε on Bk and |X k | ≤ 2c. Squaring out both sides of (14.10) and taking expectations, we get E(Sk2 I Bk ) ≤ (ε + 2c)2 P(Bk ). (14.11) Next, by means of (14.11), relation (14.9) becomes 2 I Ak−1 ) σk2 P(Ak−1 ) = E(Sk2 I Ak ) − E(Sk2 I Bk ) − E(Sk−1 2 ≥ E(Sk2 I Ak ) − E(Sk−1 I Ak−1 ) − (ε + 2c)2 P(Bk ).

(14.12)

14.1 Kolmogorov Inequalities

But Ak−1 ⊆ An , by virtue of (14.4), and therefore (14.12) becomes 2 σk2 P(An ) ≥ E(Sk2 I Ak ) − E(Sk−1 I Ak−1 ) − (ε + 2c)2 P(Bk ).

(14.13)

Summing over k = 1, . . . , n, on both sides of (14.13), we obtain, by means of (14.4),

σk2

P(An ) ≥ [E(S12 I A1 ) − E(S02 I A0 ) + E(S22 I A2 ) − E(S12 I A1 )

k

+ E(S32 I A3 ) − E(S22 I A2 ) + · · · + E(Sn2 I An ) 2 − E(Sn−1 I An−1 )] − (ε + 2c)2 P(An )

= [E(Sn2 I An ) − E(S02 I A0 )] − (ε + 2c)2 P(An ) = E(Sn2 I An ) − (ε + 2c)2 P(An ) (since S0 = 0) = E Sn2 (1 − I Acn ) − (ε + 2c)2 P(An ) = E Sn2 − E(Sn2 I Acn ) − (ε + 2c)2 P(An )

2 σk − E(Sn2 I Acn ) − (ε + 2c)2 P(An ); i.e., =

k

σk2

P(An ) ≥

k

σk2

− E(Sn2 I Acn ) − (ε + 2c)2 P(An ).

(14.14)

k

But |Sn | < ε on Acn , since Acn = ∩nj=1 C cj = ∩nj=1 (|S j | < ε). Therefore (14.14) becomes

2 2 σk P(An ) ≥ σk − ε2 P(Acn ) − (ε + 2c)2 P(An ) k

k

=

σk2

k

=

− ε2 P(Acn ) − (ε + 2c)2 [1 − P(Acn )]

σk2 − (ε + 2c)2

k

+[(ε + 2c)2 − ε2 ]P(Acn )

2 σk − (ε + 2c)2 ; i.e., ≥

k

k

σk2

P(An ) ≥

k

σk2

− (ε + 2c)2 .

293

294


Hence (ε + 2c)2 P(An ) ≥ 1 − 2 , k σk

as was to be shown. Proof of Theorem 1.

It is the combination of Lemmas 1 and 2.

Remark 2. In Kolmogorov inequalities, it is clear that, in forming partial sums, we can sum between any two positive integers m + 1 and m + r with m + r ≤ n. Then, the inequalities become ⎞ ⎛ m+k m+k 2 (ε + 2c) ≤ P ⎝ max Xj −E X j ≥ ε⎠ 1 − m+r 1≤k≤r 2 j=m+1 j=m+1 σj j=m+1

⎞ m+k = P ⎝ max (X j − E X j ) ≥ ε⎠ ≤ 1≤k≤r j=m+1 ⎛

m+r j=m+1 ε2

σ j2 .

This actually amounts to reindexing the r.v.s. Indeed, by setting Y j = X j+m and τ 2j = σ 2 (Y j ), j = 1, . . . , r , we have that the Y j s are independent and |Y j | ≤ c. Then inequalities (14.1) hold; i.e., ⎞ ⎛ r k k 2 2 (ε + 2c) j=1 τ j ⎠ ⎝ 1 − r ≤ P max Yj − E Yj ≥ ε ≤ . 2 1≤k≤r ε2 j=1 τ j j=1

j=1

k m+k m+k r k 2 But j=1 Y j − E j=1 Y j = j=1 τ j = j=m+1 X j − E j=m+1 X j and m+r 2 j=m+1 σ j , and then the remark is valid.

14.2 More Important Results Toward Proving the Strong Law of Large Numbers In this section, a number of results are established, which will lead to the proof of the SLLN. These results (six in number) are important in their own right and are used in many other situations in probability and mathematical statistics. The next theorem states, in effect, that, if the variances of r.v.s do not vary wildly, then the series consisting of the r.v.s converges a.s. More precisely, we have Theorem 2. For n ≥ 1, let X n be independent r.v.s such that E X n is finite and ∞ 2 σn2 = σ 2 (X n ) < ∞. Then, if ∞ n=1 σn < ∞, it follows that n=1 (X n − E X n ) converges a.s.

14.2 More Important Results Toward Proving the Strong

Proof. Set Tk = kj=1 (X j − E X j ) and apply the right-hand side of the relation in Remark 2 in order to obtain m+r

max |Tm+k − Tm | ≥ ε ≤

P

1≤k≤r

σ j2

j=m+1 ε2

.

(14.15)

However,

max |Tm+k − Tm | ≥ ε

1≤k≤r

= |Tm+k − Tm | ≥ ε for at least one k with 1 ≤ k ≤ r =

r |Tm+k − Tm | ≥ ε . k=1

Thus, relation (14.15) becomes P

r

|Tm+k − Tm | ≥ ε

m+r

≤

k=1

σ j2

j=m+1 ε2

.

(14.16)

In (14.16), let r → ∞ and use continuity from below of the probability measure P to obtain ∞ σ j2

r j=m+1 |Tm+k − Tm | ≥ ε ≤ . (14.17) P ε2 k=1

By letting m → ∞ in (14.17), and using the fact that

2 n≥1 σn

< ∞, we get

∞ lim P |Tm+k − Tm | ≥ ε = 0 for every ε > 0.

m→∞

(14.18)

k=1

However, (14.18) is a necessary and sufficient condition for mutual a.s. convergence of {Tm }, by Theorem 4 in Chapter 3, and hence for a.s. convergence of Tm = mj=1 (X j − E X j ), as was to be seen. The following result states that, if the probabilities of a sequence of events are small, then the lim supn→∞ of the sequence has probability zero. Theorem 3 (Borel–Cantelli Lemma). Let {An }, n ≥ 1, be an arbitrary sequence of events. Then, if n P(An ) < ∞, it follows that P(lim supn→∞ An ) = 0.

295

296


Proof.

With n and m tending to ∞, we have ⎛ ⎞ ⎛ ⎞ ∞ ∞ ∞ A j ⎠ = P ⎝lim Aj⎠ P(lim sup An ) = P ⎝ n

⎛

= lim P ⎝ n

∞

= lim lim P ⎝ n

However,

⎛ lim P ⎝ m

m

A j ⎠ ≤ lim m

lim lim P ⎝ n

j=1

m

m j=n

j=n

A j ⎠ = lim P ⎝lim n

m

m

⎞

m

⎞ Aj⎠

j=n

Aj⎠ .

(14.19)

j=n

⎞

⎛

∞

m

j=n

so that

⎛

⎞

j=n

⎛

since

n

n=1 j=n

m

P(A j ) =

j=n

∞

P(A j ),

j=n

⎞ A j ⎠ ≤ lim

∞

n

P(A j ) = 0,

(14.20)

j=n

P(A j ) < ∞. Relations (14.19) and (14.20) complete the proof.

The Borel Zero–One Criterion discussed later restates in its part (i) the Borel– Cantelli theorem for independent events. The important conclusion is part (ii), where it is stated what happens to the probability of the lim supn→∞ of the sequence when the individual probabilities are large. Theorem 4 (Borel Zero–One Criterion). Let the events An , n ≥ 1, be independent. Then (i) If n P(An ) < ∞, it follows that P(lim supn→∞ An ) = 0, and (ii) If n P(An ) = ∞, it follows that P(lim supn→∞ An ) = 1. Proof. (i) This is a special case of Theorem 3, which holds regardless of the independence or not of the events involved. (ii) For the proof of (ii), we need the following inequality n

1 − e−

j=1 x j

≤1−

n

(1 − x j ) for 0 ≤ x j ≤ 1, j = 1, . . . , n.

(14.21)

j=1

(For its justification, see Exercise 3 in Chapter 10.) Next, with n and m tending to ∞ we have, by (14.19), P(lim sup An ) = lim lim P(∪mj=n A j ) n

n

m


= lim lim[1 − P(∪mj=n A j )c ] n

m

= lim lim[1 − P(∩mj=n Acj )] n m ⎤ ⎡ m P(Acj )⎦ (by independence) = lim lim ⎣1 − n

m

j=n

⎧ ⎨

⎫ ⎬

m

= lim lim 1 − [1 − P(A j )] ; i.e., n m ⎩ ⎭ j=n ⎫ ⎧ m ⎬ ⎨ P(lim sup An ) = lim lim 1 − [1 − P(A j )] . n m ⎩ ⎭ n

(14.22)

j=n

Applying (14.21) with x1 = P(An ), x2 = P(An+1 ), . . . , xm−n+1 = P(Am ), we obtain 1−

m

[1 − P(A j )] ≥ 1 − e−

m j=n

P(A j )

j=n

→ 1 − e−

∞

j=n

P(A j )

m→∞

= 1 − e−∞ = 1. Therefore

⎫ ⎧ m ⎬ ⎨ [1 − P(A j )] ≥ 1, so that, by (14.22), lim lim 1 − n m ⎩ ⎭ j=n

P(lim sup An ) ≥ 1, hence P(lim sup An ) = 1. n

n

Remark 3. The significance of the Borel Zero–One Criterion is that, when dealing with independent events, there is no room for the value of the probability of their lim supn→∞ other than either 0 or 1; no values strictly between 0 and 1 can occur. The theorem just proved has the following three corollaries. Corollary 1. Suppose that the events An , n ≥ 1, are independent and An → A n→∞ (in the sense that lim inf An = n→∞

∞ ∞ n=1 j=n

Aj =

∞ ∞ n=1 j=n

A j = lim sup An = A). n→∞

Then P(A) is either 0 or 1. Proof. We and by the theorem, P(A) is either 0 or 1 according have A = limn An to whether n P(An ) < ∞ or n P(An ) = ∞.

297

298


Corollary 2.

Proof.

For independent events An , n ≥ 1, we have ! 1 if n P(Acn ) < ∞, P(lim inf An ) = n→∞ 0 if n P(Acn ) = ∞.

In fact, P(lim inf An ) = P( n→∞

∞ ∞

⎛ Aj) = 1 − P ⎝

n=1 j=n

⎛

= 1− P⎝

∞ ∞

⎞

∞ ∞

⎞c Aj⎠

n=1 j=n

Acj ⎠

n=1 j=n

= 1 − P(lim sup Acn ) n→∞ " 1 − 0 = 1 if n P(Acn ) < ∞, = 1 − 1 = 0 if n P(Acn ) = ∞.

a.s.

Corollary 3. Consider the independent r.v.s X n , n ≥ 1, and suppose that X n → c, n→∞ constant. Then for every δ > 0, one has ∞

P(|X n − c| ≥ δ) < ∞.

n=1

Proof. Set An (δ) = An = (|X n − c| ≥ δ). Then the events An , n ≥ 1, are independent. Next, let N c be the set of points for which X n (ω) → c. Then P(N c ) = 1, n→∞

and for every ω ∈ N c there are finitely many ns for which ω ∈ An , because otherwise X n (ω) c. Since lim supn→∞ An consists of points belonging to infinitely many n→∞ An s (by Exercise 30(ii) in Chapter 1), it follows that lim supn→∞ An ⊆ N , so that ∞ P(A ) < ∞, because otherwise P(lim supn→∞ An ) = 0. Then n=1 n ∞ P(lim supn→∞ An ) = 1. Hence n=1 P(|X n − c| ≥ δ) < ∞ for every δ > 0. In the result discussed next, we consider two sequences of r.v.s, and the probabilities of the events over which respective r.v.s differ from each other. If these probabilities are small, then one can reach a number of conclusions, which is the content of the next theorem. Theorem 5.

For n ≥ 1, consider the r.v.s X n and X n , and set An = (X n = X n ),

A = (X n = X n i.o.) = lim sup An n→∞

(where i.o. is read “infinitely often”; see also Exercise 30 in Chapter 1). Then, if ∞ P(A ) < ∞, it follows that n=1 n (i) P(A) = 0. ∞ (ii) Let B and B be the sets of convergence of ∞ n=1 X n and n=1 X n , respectively. c c Then B ∩ A = B ∩ A and P(B ∩ A) = P(B ∩ A) = P(B ∩ B ∩ A) = 0


(i.e., the set of convergence of the two series is essentially the same, although the limits may be distinct). S (iii) The sets of convergence of the sequences { bSnn } and { bnn }, as n → ∞, differ only by a null set and the limits of the sequences are the same, where Sn =

n

X j , Sn =

j=1

n

X j and 0 = bn ↑ ∞ as n → ∞.

j=1

Proof. (i) Throughout the proof all limits are taken as n → ∞. Then, with An = (X n = X n ), n ≥ 1, and A = (X n = X n i.o.) = lim supn An , the result follows from Theorem 3. (The last equality in$Chapter 2.) $∞ from Exercise 30(ii) # follows ∞ #∞ c c = A , we have A (ii) From A = lim supn An = ∞ n=1 j=n j n=1 j=n A j = c c lim inf n An (see Exercise 30(i) in Chapter #2). Then,c for ω ∈ A or, equivalently, ω ∈ lim inf n Acn , we have ω ∈ ∞ j=n 0 +1 A j for some n 0 = n 0 (ω) c and An for all n > n 0 , hence X n (ω) n > n 0 . Since = X n (ω), then ω ∈ n0 n0 X n (ω) = X n (ω) + n>n 0 X n (ω), n X n (ω) = n n=1 n=1 X n (ω) + n 0 that n X n (ω) → n=1 X n (ω) + X (ω), if and only n>n 0 X n (ω), it follows n 0 if n X n (ω) → X n>n 0 X n (ω) and n=1 n (ω) + X (ω), where X (ω) = X (ω) = n>n 0 X n (ω). Thus, B ∩ Ac ⊆ B ∩ Ac and B ∩ Ac ⊆ B ∩ Ac , so that B ∩ Ac = B ∩ Ac , and P(B ∩ B ∩ A) ≤ P(B ∩ A) = P(B ∩ A) ≤ P(A) = 0. Ac

A

B∩B ∩A BΔB

B

B B ∩ Ac = B ∩ Ac

(iii) For each ω ∈ Ac , there exists n(ω) = n 0 such that X n (ω) = X n (ω) for all n > n 0 . For such ns and 0 = bn ↑ ∞, we write n 0 n n Sn (ω) j=1 X j (ω) j=n 0 +1 X j (ω) j=1 X j (ω) = = + bn bn bn bn and

Sn (ω) = bn

n j=1

X j (ω)

bn

n 0 =

j=1

X j (ω)

bn

n +

j=n 0 +1

bn

X j (ω)

,

299

300


so that

n0 S (ω) 1 Sn (ω) X j (ω) − X j (ω) . − n = bn bn bn j=1 n 0 1 As j=1 X j (ω) − X j (ω) is independent of n and bn → 0, it follows that

Sn (ω) Sn Sn (ω) Sn c bn − bn → 0. Therefore { bn } and { bn } converge on the same subset of A , where c P(A ) = 1, and to the same limit. That is, if C and C are the sets of convergence of S

{ bSnn } and { bnn }, respectively, then C ∩ Ac = C ∩ Ac as in part (ii), and P(C ∩ A) = P(C ∩ A) = P(C ∩ C ∩ A) ≤ P(A) = 0. This section is closed with two lemmas, the Toeplitz lemma and the Kronecker lemma, which are analytic rather than probabilistic. They are used decisively toward the proof of the SLLN, as well as in many other instances in probability and mathematical statistics. Let αn j , j = 1, . . . , kn → ∞, n ≥ 1, be (real)

Lemma 3 (Toeplitz Lemma). numbers such that

n→∞

αn j → 0 for each j,

(14.23)

n→∞

and

kn

|αn j | ≤ c(< ∞) for all n.

(14.24)

j=1

Let {xn } be a sequence of (real) numbers and define the sequence {yn } as follows: yn =

kn

αn j x j , n ≥ 1.

j=1

Then one has (i) If xn → 0, then yn → 0. n→∞ n→∞ n (ii) If kj=1 αn j → 1 and xn → x finite, then yn → x. n→∞ n→∞ n→∞ (iii) For λn > 0, n ≥ 1, set bn = nj=1 λ j and suppose that bn ↑ ∞ as n → ∞. n λ j x j → x. Then, if xn → x finite, it follows that b1n kj=1 n→∞

n→∞

Proof. Throughout the proof all limits are taken as n → ∞. (i) Since xn → 0, it follows that for every ε > 0, there exists an integer n(ε) = n 0 > 0 such that for n > n 0 , |xn | < εc . Thus for all sufficiently large n (so that kn > n 0 ), kn n0 kn |yn | = αn j x j = αn j x j + αn j x j j=1 j=1 j=n 0 +1


n0 ≤ αn j x j j=1 kn ε |αn j | c j=n 0 +1 n0 αn j x j + ε (by (14.24)), and this is ≤ j=1

+

≤ ( max |x j |) 1≤ j≤n 0

n0

|αn j | + ε.

j=1

By taking now the limits and utilizing (14.23), we obtain lim sup |yn | ≤ ε for every ε > 0, so that yn → 0. n

(ii) We have yn =

kn

⎛ αn j x j = ⎝

j=1

Now

⎛ ⎝

⎞ αn j ⎠ x +

j

⎞ αn j ⎠ x → x since

j

αn j (x j − x).

j

αn j → 1, and

j

αn j (x j − x) → 0

j

by (i) with xn replaced by xn − x, so that xn − x → 0. Thus yn → x. (iii) Set λj , j = 1, . . . , n. αn j = bn Then j αn j = 1 and αn j → 0 for each fixed j, so that the assumptions in n→∞ (ii) are satisfied. Therefore λj 1 αn j x j = xj = λ j x j → x by (ii). bn bn j

j

j

Lemma 4 (Kronecker Lemma). Consider the sequence {xn }, n ≥ 1, of (real) numbers, and suppose that nj=1 x j → s finite. Let also 0 = bn ↑ ∞ as n → ∞. Then n→∞ 1 n b x → 0. j j j=1 bn n→∞ Proof. Let b0 = 0, and set sn+1 = nj=1 x j and s1 = 0. Then one has n n 1 1 bjxj = b j (s j+1 − s j ) bn bn j=1

j=1

301

302


⎛ = But

n

1 ⎝ bn

b j s j+1 =

j=1

n

b j s j+1 −

j=1

n−1

n

⎞ bjsj⎠ .

(14.25)

j=1

b j s j+1 + bn sn+1 ,

j=1

and if we set j + 1 = r , so that r = 2, . . . , n, we obtain n

b j s j+1 =

n

br −1 sr + bn sn+1 =

r =2

j=1

n

br −1 sr + bn sn+1 ,

r =1

since b0 = 0. Therefore (14.25) becomes ⎛ ⎞ n n n 1 1 ⎝ bjxj = b j−1 s j + bn sn+1 − bjsj⎠ bn bn j=1 j=1 j=1 ⎡ ⎤ n 1 ⎣ = (b j − b j−1 )s j ⎦ bn sn+1 − bn j=1

n 1 = sn+1 − (b j − b j−1 )s j . bn

(14.26)

j=1

Thus, by setting λ j = b j − b j−1 > 0, j ≥ 1, we have bn = nj=1 λ j with bn ↑ ∞ as n → ∞. Since sn → s finite, part (iii) of Lemma 3 applies and gives that n→∞

n n 1 1 λjsj = (b j − b j−1 )s j → s. n→∞ bn bn j=1

j=1

This result, together with (14.26), implies that

1 bn

n

→ s j=1 b j x j n→∞

− s = 0.

14.3 Statement and Proof of the Strong Law of Large Numbers In this section, the SLLN is stated and proved. Before this is done, another theorem, Theorem 6, is established that is some kind of precursor to the SLLN. Also, an additional result is required for the proof of the SLLN, which is of independent interest; it is stated as a lemma. Theorem 6 (Kolmogorov).

For n ≥ 1, consider the independent r.v.s X n and σ2

∞ n < ∞, for suppose that E X n is finite and σn2 = σ 2 (X n ) < ∞. Then, if n=1 b2 n

14.3 Statement and Proof of the Strong Law of Large Numbers

0 < bn ↑ ∞ as n → ∞, it follows that Sn − E Sn a.s. → 0, where Sn = X j. n→∞ bn n

j=1

Proof.

We have

σ2 n

n bn2

Xn

Therefore

n

bn

=

Xn bn

Xn − E Xn bn n

n

−E

Xn bn

σ2

=

< ∞.

converges a.s., by Theorem 2. Thus, for ω in the set of convergence of this series, we E Xn have that n X n (ω)− converges. For an arbitrary, but fixed such an ω, set bn xn = We have then that and gives that

n j=1

X n (ω) − E X n , n ≥ 1. bn

x j → s(= s(ω)) finite. Then the Kronecker lemma applies n→∞

n 1 b j x j → 0, or equivalently, n→∞ bn

1 bn

j=1 n j=1

= Thus

Sn −E Sn → 0 bn n→∞

Remark 4.

bj

n X j (ω) − E X j 1 = [X j (ω) − E X j ] bj bn j=1

Sn (ω) − E Sn → 0. n→∞ bn

a.s.

For bn = n, we get

Sn −E Sn a.s. → 0, n n→∞

provided

σn2 n n2

< ∞ (which

happens, e.g., if σn2 = σ 2 for all n), and if E X n = μ ∈ for all n, then However, this result is true without assuming finiteness of σn2 .

Sn a.s. → μ. n n→∞

The following lemma will facilitate the proof of the SLLN in addition to being interesting in its own right. Lemma 5.

Let X be a r.v. and set An = (|X | ≥ n), n ≥ 1.

Then one has

∞

P(A j ) ≤ E|X | ≤ 1 +

j=1

(so that E|X | < ∞ if and only if

∞

j=1

∞ j=1

P(A j ) < ∞).

P(A j )

(14.27)

303

304


Proof.

Let An , n ≥ 1, be defined by (14.27) and set A0 = . Then as n → ∞, de f An ↓ , Bn = An−1 − An = n − 1 ≤ |X | < n , Bi ∩ B j = , i = j,

∞

(14.28)

B j = .

j=1

Next,

(n − 1)[P(An−1 ) − P(An )] = (n − 1)P(Bn ) ≤

|X |d P Bn

= E(|X |I Bn ) ≤ n P(Bn ) = n[P(An−1 ) − P(An )],

(14.29)

and n P(An ) ≤

An

Now, for each r ≥ 1, we have r

|X |d P = E(|X |I An ).

(14.30)

P(A j ) = 0 × [P(A0 ) − P(A1 )] +

j=1

1 × [P(A1 ) − P(A2 )] + 2 × [P(A2 ) − P(A3 )] + ····················· + (r − 1) × [P(Ar −1 ) − P(Ar )] + r × P(Ar ) r ( j − 1)[P(A j−1 ) − P(A j )] + r P(Ar ), = j=1

which, by means of (14.29) and (14.30) suitably applied, becomes ≤

r

E(|X |I B j ) + E(|X |I Ar )

j=1

=

rj=1 B j

|X |d P +

|X |d P + (|X |≥r ) |X |d P = E|X |, =

|X |d P = Ar

(0≤|X | 0, all three series (i)–(iii) converge, then the series n≥1 X n converges a.s. (b) If n≥1 X n converges a.s., then all three series (i)–(iii) converge for every c > 0. Proof.

P(X n = X nc ) < ∞. There(a) Clearly, (|X n | ≥ c) = (X n = X nc ), and by (i), n≥1 fore, by Theorem 5 (ii), the series n≥1 X n and n≥1 X nc converge (essentially) on the same set. On account of (ii), Theorem and gives that 2 applies c − E X c ) converges a.s. Then, by (iii), c converges a.s., and (X X n n≥1 n≥1 n n hence so does n≥1 X n . a.s. (b) Now, a.s. convergence of n≥1 X n implies X n → 0. Then, by Corollary 3 to n→∞ Theorem 4, it follows that, for every c > 0, n≥1 P(|X n | ≥ c)< ∞. Thus the the series n≥1 X n and seriesc in (i) converges. Next, as noted in part (a), X converge (essentially) on the same set. Since n n≥1 n≥1 X n converges a.s., so does n≥1 X nc . However, |X nc | < c, n ≥ 1. Then Lemma 8 applies and yields that n≥1 σ 2 (X nc ) < ∞ and n≥1 E X nc converges. So, the series in (ii) and (iii) converge, and the proof is completed. Exercises 1. P

1. In reference to Exercise 6 in Chapter 11, explain why Snn 0. n→∞ 2. For a r.v. X , show that E|X | < ∞ if and only if ∞ n=1 P(|X | ≥ nc) < ∞ for some fixed constant c > 0. 3. Let X 1 , X 2 , . . . be independent r.v.s, and let the r.v. X be An = σ (X n , X n+1 , . . .)measurable for every n ≥ 1. Then show that P(X = c) = 1 for some (finite) constant c. ∞ 4. For n = 1, 2, . . ., suppose ∞ that the r.v.s X n are independent and that n=1 E|X n | < ∞. Then show that n=1 |X n | converges a.s. Hint: Use the Markov inequality, the special case of Theorem 6 in Chapter 6, and Theorem 11 here. 5. For n ≥ 1, let X n and X be r.v.s defined on the measure space ( , A, μ). Then, by ∞ Theorem 4 in Chapter 3, X n → X a.e. if and only if μ(∩∞ n=1 ∪ν=0 (|X n+ν − X | ≥ n→∞

1 k)

= 0 for each arbitrary but fixed k = 1, 2, . . . Replace μ by a probability measure P, and show that X n → 0 a.s. if and only if P(lim supn→∞ An ) = 0, n→∞

where An = (|X n | ≥ k1 ) for each arbitrary but fixed k = 1, 2, . . . 6. Consider any events (independent or not) An , n ≥ 1, and suppose that ∞ n=1 P(An ) < ∞. Then P(lim supn→∞ An ) = 0. This is so by Theorem 3 in this chapter (see also Theorem 4(i)). Regarding the converse of this statement, we have that, if P(lim supn→∞ An ) = 0 and the events An , n ≥ 1, are independent, then ∞ n=1 P(An ) < ∞. Justify this assertion.

317

318


7. Regarding the converse stated in Exercise 6, if the events An , n ≥ 1, are not independent, then P(lim supn→∞ An ) = 0 need not imply that ∞ n=1 P(An ) < ∞. Give one or two concrete examples to demonstrate this assertion. Hint: Take ( , A, P) = ((0, 1), B(0,1) , λ), λ being the Lebesgue measure. Then (a) Take X n = I(0, 1 ) , n ≥ 1, and show that X n → 0, so that X n → 0 a.s. n

n→∞

n→∞

Then, by Exercise 5, P(lim supn→∞ An ) = 0, where A n = (|X n | ≥ 1/k) for any arbitrary but fixed k = 1, 2, . . . Also, show that ∞ n=1 P(An ) = ∞. (b) Take X n = I(0, 1 ) , n ≥ 1, and show that X n → 0, so that X n → 0 a.s. n→∞ n→∞ n2 ∞ Again, P(lim supn→∞ An ) = 0, as in (a). Also, show that n=1 P(An ) < ∞.

CHAPTER

Topics from Ergodic Theory

15

The ultimate purpose of this chapter is the formulation and proof of the Ergodic Theorem (see Theorem 1 and its Corollaries 1 and 4, as well as Theorem 3 and its Corollaries 1 and 2). To this effect, several concepts must be introduced first, and a substantial number of results must also be established; most of them are of independent interest. The chapter is organized in six sections. In Section 15.1, the basic concept of a discrete parameter (stochastic) process is introduced as well as the special case of the coordinate process. Also, the concept of (strict) stationarity is introduced, and some characterizations of it are discussed. In Section 15.2, the concept of a measurable measure-preserving transformation is introduced, as well as the special case of the shift transformation. Stationary processes are then defined by means of a measurable measure-preserving transformation and a r.v. It is shown that the coordinate process is defined by way of the shift transformation, and that the coordinate process is stationary if and only if the shift transformation is measure-preserving. Furthermore, the interplay between stationary processes and the coordinate process is studied. The concepts of invariant and of almost sure invariant sets under a transformation are taken up in Section 15.3. It is shown, among other things, that invariant and almost invariant sets form σ -fields and one is the completion of the other. Also, the concept of ergodicity of a transformation is introduced here. In Section 15.4, the concept of invariance, relative to a transformation, is extended to a r.v., and it is shown that a r.v. is invariant if and only if it is measurable with respect to the σ -field of invariant sets. Also, it is shown that a transformation is ergodic if and only if every r.v., invariant relative to the underlying transformation, is equal to a constant with probability 1. In the subsequent section, the Ergodic Theorem as well as the so-called Maximal Ergodic Theorem are formulated and proved, and their forms are also noted under ergodicity of the underlying transformation. In Section 15.6, a process X is considered, without being stipulated that it is defined by means of a measurable measure-preserving transformation, and the concepts of invariance of a set and of a r.v., relative to X, are defined. The invariant sets form a σ -field, and a r.v. is invariant if and only if it is measurable with respect to the σ -field of invariant sets. Ergodicity of X is also defined here. The Ergodic Theorem An Introduction to Measure-Theoretic Probability, Second Edition. http://dx.doi.org/10.1016/B978-0-12-800042-7.00015-3 Copyright © 2014 Elsevier Inc. All rights reserved.

319

320

CHAPTER 15 Topics from Ergodic Theory

is reformulated and proved, and its form is noted under ergodicity of X. The section is concluded with the derivation of processes by means of a given process X, and the study of properties inherited from X, such as stationarity, invariance, and ergodicity.

15.1 Stochastic Process, the Coordinate Process, Stationary Process, and Related Results As has already been mentioned, the ultimate purpose of this chapter is to present a proof of the Ergodic Theorem. To this end, one will have first to build up the necessary mathematical machinery, and prove a series of auxiliary results. To start with, let X = X 1 , X 2 , . . . and X = X 1 , X 2 , . . . be two (discrete time parameter) stochastic processes or just processes (i.e., two infinite sequences of r.v.s) defined on the probability spaces (, A, P) and ( , A , P ), respectively, and taking values in (∞ , B ∞ ), the infinite cartesian product of Borel real lines (, B). Definition 1. The processes X and X just described have the same distribution, if for every B ∈ B ∞ , one has P X ∈ B = P X ∈ B . Clearly, two processes with the same distribution are indistinguishable from the probabilistic point of view. Thus, what counts is the distribution of a process rather than the probability space on which it is defined. It would then be desirable to replace the probability space of a given process by another one that would be easier to deal with. This can actually be done, as will be shown in the sequel. Definition 2. Consider the measurable space (∞ , B ∞ ), where the points of ∞ are denoted by x = (x1 , x2 , . . .). Then, for each n ≥ 1, define on ∞ the real-valued function Xˆ n (x) = xn . ˆ = ( Xˆ 1 , Xˆ 2 , . . .) is called the coordinate Then Xˆ n is a r.v. and the process X process. −1 That Xˆ n is a r.v. is immediate, since for every set A in B, Xˆ n (A) is the cylinder × · · · × ×A × × · · · which is in B ∞ . We then have the following result. n−1

Proposition 1. Let X = (X 1 , X 2 , . . .) be a process defined on the probability space (, A, P) and taking values in (∞ , B ∞ ). Then there is a probability measure Pˆ on ˆ = ( Xˆ 1 , Xˆ 2 , . . .) have B ∞ such that the given process and the coordinate process X ˆ the same distribution, under P and P, respectively. Proof. The probability measure Pˆ in question is simply the distribution of X ˆ = P(X ∈ B). Since, clearly, for every B ∈ B ∞ , under P; i.e., for B ∈B ∞ , P(B) ˆ ∈ B) = B, we have P(X ∈ B) = P( ˆ ∈ B). ˆ X (X

15.1 Stochastic Process, the Coordinate Process, Stationary Process

In view of this proposition, all definitions and results involving the distribution of a process may be given in terms of the coordinate process by employing the appropriate probability measure. Thus, the process X = (X 1 , X 2 , . . .), defined on (, A, P) into (∞ , B ∞ ), may be thought of as being the coordinate process and ˆ then (, A, P) = (∞ , B ∞ , P). We now give the following definition. Definition 3. The process X = (X 1 , X 2 , . . .) is said to be (strictly) stationary, if for every m ≥ 1, every 1 ≤ n 1 < n 2 < · · · < n m , with n 1 , n 2 , . . . , n m integers and every k ≥ 1, one has P

X n1 , X n2 , . . . , X nm ∈ B = P X n 1 +k , X n 2 +k , . . . , X n m +k ∈ B(k) ,

(15.1)

where B is any cylinder with base a Borel set in mj=1 Bn j and all of whose sides are equal to , and B(k) is the cylinder with the same base as that of B but located in

m j=1 Bn j +k and all of whose sides are equal to . In the sequel, we will write B, rather than B(k) , but the preceding explanation should be kept in mind. We shall also write B for the cylinder with base the Borel set B in the σ -field mj=1 Bn j . Since the distribution of the random vector (X n 1 , X n 2 , . . . , X n m ) is determined by the joint d.f.of X n 1 , X n 2 , . . . , X n m , relation (15.1) is equivalent to the following one, FX n1 ,X n2 ,...,X nm (xn 1 , xn 2 , . . . , xn m ) = FX n1 +k ,X n2 +k ,...,X nm +k (xn 1 , xn 2 , . . . , xn m ),

(15.2)

for all xn 1 , xn 2 , . . . , xn m ∈ . From (15.1) (or (15.2)), it follows, in particular, that the r.v.s X n , n ≥ 1, are identically distributed. Interpreting n as time, the concept of stationarity is then clear: No matter what times they are associated with, the joint distribution of any finite number of r.v.s from a stationary process is the same, provided their relative distance (in time) remains the same. In particular, no matter at what time one starts making observations, the distribution of the outcomes is the same. From (15.1), it also follows that, for every k ≥ 1, and any B ∈B ∞ , P

X 1 , X 2 , . . . ∈ B = P X k+1 , X k+2 , . . . ∈ B .

(15.3)

This is so because all finite dimensional distributions of the processes X = (X 1 , X 2 , . . .) and X(k) = (X k+1 , X k+2 , . . .) are the same (by (15.1)), and then so are the induced infinite dimensional measures in (15.3) (by the Carathéodory Extension Theorem and the Kolmogorov Consistency Theorem; for the latter, see, e.g., Loève (1963), page 93). Actually, stationarity is characterized by the joint distributions of all finitely many consecutive r.v.s. More precisely, one has

321

322


Proposition 2. The process X = (X 1 , X 2 , . . .) is stationary, if and only if, for every m, k ≥ 1, and every B ∈ B m , one has (15.4) P X 1 , X 2 , . . . , X m ∈ B = P X k+1 , X k+2 , . . . , X k+m ∈ B . Proof. One direction is obvious from (15.1) by taking n j = j, j = 1, . . . , m. So, it suffices to show the other direction. To this end, let 1 ≤ n 1 < n 2 < · · · < n m . One has, by (15.4), FX 1 ,X 2 ,...,X nm (x1 , x2 , . . . , xn m ) = FX k+1 ,X k+2 ,...,X k+nm (x1 , x2 , . . . , xn m )

(15.5)

for all x1 , x2 , . . . , xn m ∈ . In (15.5), replace the xs that are = from xn j , j = 1, . . . , m, by ∞(in the sense of letting them → ∞). Then one has FX n1 ,X n2 ,...,X nm (xn 1 , xn 2 , . . . , xn m )

= FX n1 +k ,X n2 +k ,...,X nm +k (xn 1 , xn 2 , . . . , xn m ),

which is (15.2). But (15.2) implies (15.1).

We also have the following result. Proposition 3. For n ≥ 1, let X n be independent r.v.s. Then X = (X 1 , X 2 , . . .) is stationary, if and only if the X n s are identically distributed. Proof. If X is stationary, the X n s are always identically distributed. For the converse, we have FX 1 ,...,X n (x1 , . . . , xn ) = FX 1 (x1 ) · · · FX n (xn ) = FX k+1 (x1 ) · · · FX k+n (xn ) = FX k+1 ,...,X k+n (x1 , . . . , xn ) for every k ≥ 1, and then Proposition 2 completes the proof.

Given a stationary process, one may derive any number of other stationary processes. This is so because of the following result. Proposition 4. Let X = (X 1 , X 2 , . . .) be a stationary process defined on (, A, P) and taking values in (∞ , B ∞ ), and let ϕm be defined as follows: ϕm : (m , B m ) → (, B), measurable, 1 ≤ m ≤ ∞. Set

Yn = ϕm X n , X n+1 , . . . , X n+m−1 , n ≥ 1.

(15.6)

Then the process Y = (Y1 , Y2 , . . .) is stationary. Proof. By Proposition 2, it suffices to prove that, for every n, k ≥ 1, and every B ∈ B n , one has (15.7) P (Y1 , Y2 , . . . , Yn ) ∈ B = P (Yk+1 , Yk+2 , . . . , Yk+n ) ∈ B .

15.2 Measure-Preserving Transformations, the Shift Transformation

Set y j = ϕm (x j , x j+1 , . . . , x j+m−1 ), j ≥ 1, and let A = n+m−1 ; y1 , y2 , . . . , yn ∈ B . Then one has

x1 , x2 , . . . , xn+m−1 ∈

P (Y1 , Y2 , . . . , Yn ) ∈ B = P (X 1 , X 2 , . . . , X n+m−1 ) ∈ A = P (X k+1 , X k+2 , . . . , X k+n+m−1 ) ∈ A (by stationarity of X) = P (Yk+1 , Yk+2 , . . . , Yk+n ) ∈ B which is (15.7).

Corollary. If the r.v.s X n , n ≥ 1, are i.i.d., then the process Y, as defined by (15.6), is stationary. Proof.

It follows from Propositions 3 and 4.

15.2 Measure-Preserving Transformations, the Shift Transformation, and Related Results Consider the probability space (, A, P), and let T : (, A) → (, A), T −1 (A) ⊆ A; i.e., T is a measurable transformation on into itself. T will be always assumed to be measurable, whether explicitly stated or not. Definition 4.

The transformation T is said to be measure-preserving, if

P T −1 A = P A for every A ∈ A. −1 Remark 1. Actually, it suffices to show that P T A = P(A) for A belonging to a measure-determining class, such as the field F generating A. More precisely, Proposition 5. Let F be a field such that σ (F) = A, and assume that, for every A ∈ F, P(T −1 A) = P(A). Then, for every A ∈ A, P(T −1 A) = P(A). Proof. Let M = {A ∈ A; P(T −1 A) = P(A)}. Then M is a monotone class. Indeed, M ⊆ F by assumptions. Thus, M = . Next, let An ∈ M, n ≥ 1, be monotone. To show that lim An ∈ M, or equivalently, P(T −1 (lim An )) = P(lim An ) where here and in the sequel the limits are taken as n → ∞. To this end, we have, for An ↑: −1 −1 −1 ∪ An = P ∪ T An P(T (lim An )) = P T n≥1

n≥1

= P(lim T −1 An ) = lim P(T −1 An ) = lim P(An ) = P(lim An ).

323

324


For An ↓:

= P ∩ T −1 An P(T −1 (lim An )) = P T −1 ∩ An = P(lim T

−1

n≥1

n≥1

−1

An ) = lim P(An ) = P(lim An ).

An ) = lim P(T

So, M is monotone, contains F with σ (F) = A, and is contained in A. Hence M = A (by Theorem 6 in Chapter 1). −n Remark 2. If T is measure-preserving, then P T A = P(A), n ≥ 1, where T −n A = T −1 (T −(n−1) A), n ≥ 1, and T −0 = I , the identity in . In fact, for n = 1 the statement is true. Let us assume it to be true for some n > 1. Then P[T −(n+1) A] = P[T −1 (T −n A)] = P(T −n A) (by the measure-preserving property of T ) = P(A) (by the induction hypothesis). In terms of a measure-preserving transformation T , one may define any number of stationary processes. In fact, let X be a r.v. defined on (, A, P), and define the r.v.s X n , n ≥ 1, as follows:

(15.8) X n (ω) = X T n−1 ω , ω ∈ , where T 2 ω = T (T ω), T k ω = T (T k−1 ω), and T 0 = I , the identity in .

(15.9)

Then one has the following result. Proposition 6. Let T be a (measurable) measure-preserving transformation defined on (, A, P) into itself, and let X be a r.v. defined on this probability space. Let X n , n ≥ 1, be defined by (15.8) and (15.9). Then the process X = (X 1 , X 2 , . . .) is a stationary process. Proof. In the first place, the measurability of T implies the measurability of T 2 , since for A ∈ A, (T 2 )−1 (A) = T −1 [T −1 (A)], and by induction, T n is measurable for all n ≥ 1. Thus, X n , n ≥ 1, are r.v.s. It remains to prove stationarity. For B ∈ B n , we set A = ω ∈ ; X 1 (ω), . . . , X n (ω) ∈ B and A = ω ∈ ; X k+1 (ω), . . . , X k+n (ω) ∈ B and we wish to show that P(A) = P(A ). This will follow, by the measure-preserving property of T , by showing that Lemma 1. Proof.

With A and A as just defined, we have A = T −k A. de f

First, T −k A ⊆ A . Indeed, ω ∈ T −k A implies T k ω = ω ∈ A. Then B (X 1 (ω ), . . . , X n (ω )) = (X (T 0 ω ), . . . , X (T n−1 ω )) = (X (T 0 T k ω), . . . , X (T n−1 T k ω)) = (X (T k ω), . . . , X (T k+n−1 ω)) = (X k+1 (ω), . . . , X k+n (ω)), so that ω ∈ A ; i.e., T −k A ⊆ A .

15.2 Measure-Preserving Transformations, the Shift Transformation

de f

Next, A ⊆ T −k A. That is, for ω ∈ A , to show ω ∈ T −k A or T k ω = ω ∈ A. We have (X 1 (ω ), . . . , X n (ω )) = (X (T 0 ω ), . . . , X (T n−1 ω )) = (X (T 0 T k ω), . . . , X (T n−1 T k ω)) = (X (T k ω), . . . , X (T k+n−1 ω)) = (X k+1 (ω), . . . , X k+n (ω)) and this does belong in B since ω ∈ A . Thus, (X 1 (ω ), . . . , X n (ω )) ∈ B, which implies that ω ∈ A. Hence A ⊆ T −k A, and therefore A = T −k A. A certain transformation to be introduced next is of special interest. Definition 5.

The transformation S, defined as S : ∞ → ∞ , so that S(x1 , x2 , . . .) = (x2 , x3 , . . .),

is called the shift transformation. We then have the following result. The shift transformation S defined on (∞ , B ∞ ) is measurable.

Proposition 7. Proof.

In order to prove measurability for S, it suffices to show that S −1 I1 × · · · × In × × · · · ∈ B ∞ for any n ≥ 1,

and any intervals I j , j = 1, . . . , n, in . We have S −1 I1 × · · · × In × × · · · = x = (x1 , x2 , . . .) ∈ ∞ ; S x = (x2 , x3 , . . .) ∈ I1 × · · · × In × × · · · } = x = (x1 , x2 , . . .) ∈ ∞ ; x2 ∈ I1 , . . . , xn+1 ∈ In , x j ∈ , j ≥ n + 2 = × I1 × · · · × In × × · · · , which is in B ∞ .

(15.10)

ˆ = ( Xˆ 1 , Xˆ 2 , . . .) In terms of the shift transformation S, the coordinate process X can be expressed as

(15.11) S 0 is the identity in ∞ . Xˆ n (x) = Xˆ 1 S n−1 x , n ≥ 1 We now have the following result.

ˆ = Xˆ 1 , Xˆ 2 , . . . , defined on Proposition 8. If the coordinate process X ˆ is stationary, then the shift transformation S is measure-preserving, (∞ , B ∞ , P), and vice versa. Proof. Since the class of cylinders I1 × · · · × In × × · · · ; I j , j = 1, . . . , n, intervals in determines the measure Pˆ on B ∞ , by Remark 1 and Proposition 5, it

325

326


suffices to show that S is measure-preserving over this class of sets alone. We have Pˆ I1 × · · · × In × × · · · = Pˆ Xˆ 1 , . . . , Xˆ n ∈ I1 × · · · × In

= Pˆ Xˆ 1 ∈ I1 , . . . , Xˆ n ∈ In = Pˆ Xˆ 2 ∈ I1 , . . . , Xˆ n+1 ∈ In ˆ (by stationarity of X), = Pˆ × I1 × · · · × In × × · · · which, by (15.10), completes one direction of the proof. The converse is true by Proposition 6. Now Propositions 7 and 8 state, in effect, that every stationary process is essentially generated by a measure-preserving transformation. More precisely, we have Proposition 9. Let X = X 1 , X 2 , . . . be a stationary process defined on the probability space (, A, P), and let Pˆ be the distribution of X, under P. Then the ˆ is defined in terms of the shift transformation S, has the same coordinate process X distribution X, and is stationary; S is measure-preserving. It follows from (15.11), Propositions 8, and 1. Thus, from a probability point of view, if X = X 1 , X 2 , . . . is a stationary process defined on the probability space (, A, P), we may assume that there is a (measurable) measure-preserving transformation T on (, A, P) into itself such that X n = X (T n−1 ), n ≥ 1, for some r.v. X , in the sense that X n (ω) = X [T n−1 (ω)], ω ∈ , n ≥ 1.

Proof.

15.3 Invariant and Almost Sure Invariant Sets Relative to a Transformation, and Related Results The basic concepts in this section are those of an invariant and of an almost invariant set given below. Definition 6. Let T be a (measurable) transformation on (, A, P) into itself. A set A ∈ A is said to be invariant (under T ), if T −1 A = A. A set A ∈ A is said to be a.s. invariant, if P(A T −1 A) = 0, or equivalently, P(A ∩ T −1 A) = P(A) = P(T −1 A). Remark 3. It is easily seen that, if A is invariant, then P(T −n A) = P(A), and if T is a.s. invariant, then P(A T −n A) = 0, n ≥ 0. That is, Proposition 10. for n ≥ 0.

a.s.

If A is a.s. invariant, then P(A T −n A) = 0, so that A = T −n A a.s.

a.s.

a.s.

Proof. In the first place, we observe that A = B and B = C imply A = C. Indeed, a.s. a.s. A = B means that P(A B) = 0 or P(A ∩ B c ) = P(Ac ∩ B) = 0, and B = C

15.3 Invariant and Almost Sure Invariant Sets Relative to a Transformation

means that P(B C) = 0 or P(B ∩ C c ) = P(B c ∩ C) = 0. Then P(A ∩ C c ) = P[(A ∩ C c ) ∩ B] + P[(A ∩ C c ) ∩ B c ] = P[A ∩ (B ∩ C c )] + P[(A ∩ B c ) ∩ C c ] ≤ P(B ∩ C c ) + P(A ∩ B c ) = 0, P(Ac ∩ C) = P[(Ac ∩ C) ∩ B] + P[(Ac ∩ C) ∩ B c ] = P[(Ac ∩ B) ∩ C] + P[(B c ∩ C) ∩ Ac ] ≤ P(Ac ∩ B) + P(B c ∩ C) = 0. a.s.

Thus, P(A ∩ C c ) = P(Ac ∩ C) = 0 and hence P(A C) = 0 or A = C. Next, by the a.s. invariance of A, we have, P(A) = P(T −1 A) = P(A ∩ T −1 A), and by the measure-preserving property of T , P(T −1 A) = P(T −2 A) = P[T −1 (A ∩ T −1 A)] = P[T −1 A ∩ T −2 A]. Thus P(T −1 A) = P(T −2 A) = P[T −1 A ∩ T −2 A], hence P(T −1 A T −2 A) = 0 a.s. or T −1 A = T −2 A. a.s. a.s. Since also A = T −1 A, we have, by the introductory observation, A = T −2 A. a.s. −3 a.s. a.s. −n a.s. −n −2 Likewise, T A = T A = · · · = T A, so that A = T A, n ≥ 0. Proposition 11. Let T be a (measurable) measure-preserving transformation on (, A, P) into itself. Then one has (i) If A = B a.s. and B is a.s. invariant, then so is A. (ii) If B is a.s. invariant, then P(B c ∩T −n B) = 0 and P(B∩T −n B) = P(B), n ≥ 0. Proof.

a.s.

a.s.

First, show that A = B implies T −1 A = T −1 B. Indeed, P(T −1 A) = P(A)(by the measure-preserving property of T) a.s.

= P(B)(since A = B) = P(T −1 B)(by the measure-preserving property of T), and a.s.

P(A) = P(B) = P(A ∩ B)(since A = B) = P[T −1 (A ∩ B)] (by the measure-preserving property of T) = P(T −1 A ∩ T −1 B). So, P(T −1 A) = P(T −1 B) = P(T −1 A ∩ T −1 B), or P(T −1 A T −1 B) = 0, so that a.s. T −1 A = T −1 B.

327

328


We now proceed with the proof of the proposition. a.s.

a.s.

a.s.

(i) A = B implies T −1 A = T −1 B, by the introductory observation. But B = T −1 B a.s. a.s. a.s. a.s. by Proposition 10. Thus A = B = T −1 B = T −1 A, so that A = T −1 A, and A is a.s. invariant. a.s. (ii) By Proposition 10, we have B = T −n B, since B is a.s. invariant. Hence −n c P(B ∩ T B) = P(B) and P(B ∩ T −n B) = 0. We also have Proposition 12. The class J of invariant sets in A under a (measurable) transformation T is a σ -field. Proof.

We have that T −1 = , so that ∈ J and so J is not empty. Next,

c T −1 Ac = T −1 A = Ac , if A ∈ J , so that Ac ∈ J .

Finally,

⎛ ⎞ T −1 ⎝ A j ⎠ = T −1 A j = A j , if A j ∈ J , j

so that

j

j

Aj ∈ J.

j

Proposition 13. Let J be the class of all a.s. invariant sets in (, A, P) under the (measurable) measure-preserving transformation T . Then, for every A ∈ J , there exists a set A ∈ J such that A = A a.s.; i.e., P(A A ) = 0. −n C. Then Proof. Let C ∈ A and set B = ∞ n=0 T T −n B ⊆ B, n ≥ 0. In fact,

⎛ T −n B = T −n ⎝

⎞

∞

T − j C⎠ =

j=0

⊆

∞

∞

(15.12)

T −n− j C =

j=0

∞

T−jC

j=n

T − j C = B.

j=0

Next, set D=

∞

T −n B.

(15.13)

n=0

Then D is invariant. In fact, T

−1

D=T

−1

∞

n=0

T

−n

B

=

∞ n=0

T −n−1 B =

∞ n=1

T −n B

15.3 Invariant and Almost Sure Invariant Sets Relative to a Transformation

=

∞

T −n B

n=1 ∞

=

T −n B

B+

∞

T −n B

Bc

n=1

B

n=1

=

∞

T −n B = D,

n=0

because B c ∩ T −1 B = , since T −1 B ⊆ B, by (15.12). (Actually, this may be considered as a way of constructing invariant sets.) Suppose now that C ∈ J . Then we will show that D = C a.s. by showing that B = C a.s. and D = B a.s. In fact, since C ⊆ B, it suffices to show that P(B ∩ C c ) = 0. We have ∞ ∞ c −n c c −n P(B ∩ C ) = P C ∩T C T C ∩C = P n=0

≤

∞

n=0

P C c ∩ T −n C .

n=0

But

P C c ∩ T −n C = 0, by Proposition 11(ii).

Thus, B = C a.s., and hence B is a.s. invariant by Proposition 11(i). Now from (15.13), we have that D ⊆ B, so that ∞ c D B = B ∩ Dc = B ∩ T −n B = B∩ = B∩

n=0 ∞

n=0 ∞

T

−n

B

c

T

−n

B

c

n=0

=

∞ n=0

∞ B ∩ T −n B c = B ∩ T −n B c . n=1

But B ∩ T −n B c = B ∩ T −n ( − B) = B ∩ (T −n − T −n B) = B ∩ ( − T −n B) = B − B ∩ T −n B and P(B ∩ T −n B) = P(B),

329

330


by the fact that B is a.s.invariant, and Proposition 11 (ii). Thus, P(B ∩ T −n B c ) = 0, and P(D B) = 0, so that D = B a.s. and hence D = C a.s. To summarize, C is a.s. invariant, D is invariant, and C = D a.s. Hence, if we set A = C, then A = D. Proposition 14.

Let J be as in Proposition 12, and define the class J as follows:

J = {A ∈ A;

P(A B) = 0 for some B ∈ J } .

(15.14)

Then (i) Every A ∈ J is a.s. invariant. (ii) J is a σ -field. Proof. First, J = since, clearly, J ⊆ J by Proposition 13. Next, for A ∈ J , one has that there exists B ∈ J such that P(A B) = 0. Thus, A = B a.s. Since B is trivially a.s. invariant, it follows, by Proposition 11(i), that A is a.s. invariant. (ii) If A ∈ J , then for some B ∈ J , 0 = P(A B) = P(A ∩ B c ) + P(Ac ∩ B) = P[Ac ∩ (B c )c ] + P[(Ac )c ∩ B c ] = P(Ac B c ) and B c ∈ J , so that Ac ∈ J . Next, let A j ∈ J , j = 1, 2, . . . Then (i)

0 = P(A j B j ) = P(A j ∩ B cj ) + P(Acj ∩ B j ) for some B j ∈ J . We then set B = Bi and get i

⎛ ⎞ ⎛ ⎤ ⎞ ⎡⎛ ⎞c P ⎝ A j B ⎠ = P ⎝ A j ∩ B c ⎠ + P ⎣⎝ A j ⎠ ∩ B ⎦ j

j

j

⎛ ⎞ ⎛ ⎞ = P ⎝ Aj ∩ Bic ⎠ + P ⎝ Acj ∩ Bi ⎠ ⎡

j

i

j

i

⎤ c c ≤ P⎣ Aj ∩ Bj ⎦ + P Ai ∩ Bi ≤ Thus,

j

Aj ∈

J .

j

P A j ∩ B cj +

i

P Aic ∩ Bi = 0.

i

j

Definition 7. The σ -field J , as defined by (15.14), is called the completion of J with respect to A and P. (See also Exercise 17 in Chapter 2.) Proposition 15. Let J be the class of all a.s. invariant sets in A under the (measurable) measure-preserving transformation T . Then J is a σ -field and, indeed, J = J .

15.4 Measure-Preserving Ergodic Transformations, Invariant Random

Proof. By Proposition 13, J ⊆J , whereas, by Proposition 14(i), J ⊆J . Thus, J =J . Definition 8. Let T be a (measurable) transformation on (, A, P) into itself. Then T is said to be ergodic, if for every A ∈ J , one has P(A) = 0 or 1.

15.4 Measure-Preserving Ergodic Transformations, Invariant Random Variables Relative to a Transformation, and Related Results The concept of invariance is also defined for r.v.s. Definition 9. Let T be a (measurable) transformation on (, A, P) into itself, and let X be a r.v. on the same space. We say that X is invariant (relative to T ), if X (ω) = X (T ω), ω ∈ . Remark 4. For an invariant r.v. X , it is immediate that X (ω) = X (T n ω), n ≥ 0 (and for n = −1, −2, . . ., if T is one-to-one onto). Indeed, let ω = T −1 ω. Then X (T −1 ω) = X (ω ) = X (T ω ) = X (T (T −1 ω)) = X (ω), and likewise, if T −n ω = ω , then X (T −n ω) = X (ω ) = X (T ω ) = X (T (T −n ω)) = X (T −(n−1) ω) = X (ω), by the induction hypothesis. The fact that X (ω) = X (T n ω), ω ∈ , n = 0, ±1, . . ., means that X (ω) remains constant on the orbit T n ω, n = 0, ±1, . . . Now the question arises as to what r.v.s are invariant. The answer to this is given by the following result. Proposition 16. Let T be a (measurable) transformation on (, A, P) into itself, and let X be a r.v. on the same space. Then X is invariant (relative to T ), if and only if X is J -measurable, where J is the σ -field of (T -) invariant sets in A. Proof. Let X be invariant, and set A(x) = {ω ∈ ; X (ω) ≤ x} , x ∈ . Then T −1 A(x) = {ω ∈ ; T ω ∈ A(x)} = {ω ∈ ; X (T ω) ≤ x} = {ω ∈ ; X (ω) ≤ x} = A(x), since X (T ω) = X (ω). Thus, T −1 A(x) = A(x), x ∈ , and this establishes J measurability for X . Next, let X be J -measurable. We shall show that X is invariant. Since every X is the pointwise limit of simple r.v.s, it suffices to show the result for the case that X is

331

332


only an indicator function. So let X = I A with A ∈ J . Then X (T ω) = I A (T ω) = IT −1 A (ω) = I A (ω) = X (ω), since T −1 A = A. (See also Exercise 4.)

Ergodicity and invariant r.v.s are related as follows. Proposition 17. Let T be a (measurable) transformation on (, A, P) into itself. Then T is ergodic, if and only if every real-valued r.v. invariant relative to T , defined on the same probability space, is a.s. equal to a finite constant. Proof. Suppose that every invariant r.v. is a.s. equal to a constant, and let A be an arbitrary set in J . If we set X = I A , then X is J -measurable, and therefore invariant, by Proposition 16. Since X is equal to 1 or 0 with probability P(A) or P(Ac ), respectively, we have that P(A) = 1 or P(Ac ) = 1. So P(A) = 0 or P(A) = 1. Next, suppose that T is ergodic, so that P(A) = 0 or 1 for every A ∈ J . We shall prove that every invariant r.v. X is a.s. equal to a constant. Since X is invariant, it is J -measurable and hence P(X < x) = 0 or 1 for every x ∈ . On the other hand, P(X < x) → 1 as x → ∞. Hence P(X < x) = 1 for all x ≥ x1 , some sufficiently large x1 . Set x0 = inf {x ∈ ; P(X < x) = 1} ; x0 is finite; i.e., x0 > −∞, because otherwise P(X < x) = 1 for all x would imply 1 = P(X < x) ↓ P(X = −∞) as x ↓ −∞, a contradiction. Then P x0 − ε < X < x0 + ε = P X < x0 + ε −P X ≤ x0 − ε = 1 − 0 = 1 for every ε > 0. Letting ε ↓ 0 and observing that x0 − ε < X < x0 + ε ↓ X = x0 , we get P(X = x0 ) = 1.

15.5 The Ergodic Theorem, Preliminary Results We are now in a position to formulate and prove the Ergodic Theorem. Theorem 1 (Ergodic Theorem). Let T be a (measurable) measure-preserving transformation on (, A, P) into itself, and let X be a r.v. on the same space such that E |X | < ∞. Define X n , n ≥ 1, as X n (ω) = X (T n−1 ω), ω ∈ ,

15.5 The Ergodic Theorem, Preliminary Results

and let J be the invariant σ -field (relative to T ). Then n n 1 1 a.s. X j (ω) = X (T j−1 ω) → E X |J . n→∞ n n j=1

(15.15)

j=1

For the proof of Theorem 1, we need the following results. Lemma 2. Let T be a (measurable) measure-preserving transformation on (, A, P) into itself, and let X be a r.v. on the same space for which E X exists. Then E X (ω) = E X (T ω). Proof. Since any r.v. is split into two nonnegative r.v.s, which are nondecreasing pointwise limits of nonnegative simple r.v.s, it suffices to show the result for the case that X is an indicator r.v. So, let X = I A , A ∈ A. Then E X (ω) = P(A) and E X (T ω) = E I A (T ω) = E IT −1 A (ω) = P(T −1 A) = P(A) by the measurepreserving property of T ; i.e., E X (ω) = E X (T ω). (See also Exercise 5.) Alternative Simpler Proof. Let A = X −1 (B), B ∈ B. Then (X T )−1 (B) = T −1 A, and P(A) = P(T −1 A). Thus, X and X T have the same distribution under P. But # # # # Xd P = xd Q X and (X T )d P = xd Q X T .

Since FX ≡ FX T , we obtain

$

Xd P =

$

Theorem 2 (Maximal Ergodic Theorem). rem 1, and, for ω ∈ , set Sk (ω) =

k

X j (ω) =

j=1

and

(X T )d P.

Let T , X and X n , n ≥ 1, be as in Theo-

k

X (T j−1 ω), k ≥ 1,

(15.16)

j=1

Mn (ω) = max 0, S1 (ω), . . . , Sn (ω) .

(15.17)

#

Then

(Mn >0)

Proof.

X d P ≥ 0.

(15.18)

For 1 ≤ k ≤ n, we have, by replacing in (15.16) and (15.17) ω by T ω, Mn (T ω) ≥ Sk (T ω),

so that, by (15.16), X (ω) + Mn (T ω) ≥ X (ω) + Sk (T ω) = Sk+1 (ω), since Sk (T ω) = · · · + X k+1 (ω). Thus,

k % j=1

X (T j−1 T ω) = Sk+1 (ω) − X (ω) =

k %

X (T j ω) = X 2 (ω) +

j=1

X (ω) ≥ Sk+1 (ω) − Mn (T ω), k = 1, . . . , n.

(15.19)

333

334


But S1 (ω) = X (ω) and Mn (T ω) ≥ 0, so that X (ω) ≥ S1 (ω) − Mn (T ω).

(15.20)

From (15.19) and (15.20), it follows that X (ω) ≥ max S1 (ω), . . . , Sn (ω) − Mn (T ω). Therefore #

#

(Mn >0)

X (ω)d P ≥

(Mn >0)

max S1 (ω), . . . , Sn (ω)

−Mn (T ω)} d P # = Mn (ω) − Mn (T ω) d P, by (15.17). (Mn >0)

But

#

# (Mn >0)

Mn (ω)d P =

#

= Therefore

#

(Mn >0)

(Mn ≥0)

Mn (ω)d P

Mn (ω)d P, since Mn ≥ 0.

Mn (ω) − Mn (T ω) d P =

by Lemma 2. It follows that

#

Mn (ω)d P # − Mn (T ω)d P (Mn >0) # ≥ Mn (ω)d P # − Mn (T ω)d P = 0,

$ (Mn >0)

X d P ≥ 0, which is (15.18).

Lemma 3. Let T , X , X n , n ≥ 1, and J be as in Theorem 1, and suppose that E X |J = 0 a.s. Then relation (15.15) holds true. Proof. Let Sn , n ≥ 1, be defined by (15.16), and let X¯ = lim sup and in the sequel all limits are taken as n → ∞. Then

Sn n ,

where here

Sn+1 (ω) − X (ω) Sn (T ω) = lim sup X¯ (T ω) = lim sup n ' n & Sn+1 (ω) Sn+1 (ω) X (ω) n + 1 + × = lim sup = X¯ (ω); = lim sup − n n n+1 n+1


i.e., X¯ (T ω) = X¯ (ω), which means that X¯ is invariant. Then X¯ is measurable with respect to J , and therefore, if for ε > 0 we set D = ( X¯ > ε), it follows that D is invariant. Now define the r.v. X ∗ as follows: (15.21) X ∗ (ω) = X (ω) − ε I D (ω). Let X n∗ = X ∗ (T n−1 ), n ≥ 1, and let Sk∗ and Mn∗ be the quantities defined by (15.16) and (15.17), respectively, when X is replaced by X ∗ . Then Theorem 2 implies that # X ∗ d P ≥ 0. (15.22) (Mn∗ >0)

Define the sets An , n ≥ 1, by Then, clearly,

An = Mn∗ > 0 .

An = Mn∗ > 0 =

Also, let the set A be defined by

max Sk∗ > 0 .

1≤k≤n

A=

sup Sk∗ k≥1

(15.23)

>0 .

(15.24)

Then, from (15.23) and (15.24), it clearly follows that An ↑ A. From the fact that D is invariant, we have that T − j D = D, j ≥ 0, so that X ∗ (T j ω) = X (T j ω) − ε I D (T j ω) = X (T j ω) − ε IT − j D (ω) = X (T j ω) − ε I D (ω); i.e., X ∗ (T j ω) = X (T j ω) − ε I D (ω). & ' Sk∗ (ω) Sk (ω) = − ε I D (ω), k k

Therefore

and

∗ S A = sup Sk∗ > 0 = sup k > 0 k≥1 k≥1 k Sk = sup − ε ID > 0 k k≥1 Sk > ε ∩ D, so that A ⊆ D. = sup k≥1 k

(15.25)

335

336


Now, from the definition of lim sup, it follows that lim sup

Sk Sk Sk ≤ sup ; i.e., X¯ ≤ sup . k k k≥1 k≥1 k

Since on D, X¯ > ε, it follows that on D, sup k≥1

Sk k

> ε, so that

Sk∗ Sk D ⊆ sup > ε ∩ D = sup >0 k≥1 k k≥1 k = sup Sk∗ > 0 = A. k≥1

Thus A = D. ( ∗ ( ∗ ∗ ∗ ( ( Therefore (15.25) gives An ↑ D. So we have X I An → X I D , X I An ≤ |X | ≤ |X | + ε, independent of n, and E |X | $+ ε < ∞. Hence the Dominated Conver$ $ gence Theorem applies and gives that An X ∗ d P → D X ∗ d P. But An X ∗ d P = $ $ ∗ ∗ (Mn∗ >0) X d P ≥ 0,by (15.23), so that D X d P ≥ 0. Next, one has, by means of (15.21), # # ∗ X − ε IDd P X dP = D #D # = X − ε dP = X d P − ε P(D) D #D = E X |J d P − ε P(D) D

= 0 − ε P(D) = −ε P(D). Therefore −ε P(D) ≥ 0, and hence P(D) = 0. That is to say, P( X¯ > ε) = 0, for every ε > 0, so that X¯ ≤ 0 a.s. So the conclusion so far has been that lim sup

Sn ≤ 0 a.s. n

(15.26)

Next, replacing the r.v. X by −X , the corresponding Sn is −Sn , and the same arguments above yield Sn Sn Sn = − lim inf or lim inf ≥ 0 a.s. a.s. 0 ≥ lim sup − n n n This result, together with (15.26), then gives that Sn a.s. → 0. n


Proof of Theorem 1. Let Y = X −E X |J . Then E Y |J = 0 a.s., and of course, E X |J is J -measurable, hence invariant, by Proposition 16. Then, by Remark 4, E X |J (T n ω) = E X |J (ω), n ≥ 0, ω ∈ . It follows that

1 n

%n

j=1 Y

T j−1 ω =

1 n

%n j=1

X T j−1 ω − E X |J , whereas

n 1 j−1 a.s. Y T ω → 0, n j=1

by Lemma 3. Thus

n 1 j−1 a.s. X T ω → E X |J . n j=1

Corollary 1. If T is also ergodic, then the right-hand side of (15.15) is equal to E X a.s. Proof. E X |J is J -measurable, and since T is ergodic, it follows that E X |J = c a.s. by Proposition 17. Then E E X |J = E X = c. Thus E X |J = E X a.s. Corollary 2.

If T is also ergodic, then for every A ∈ A, one has n 1 j−1 a.s. IA T ω → P(A). n→∞ n

(15.27)

j=1

Proof. P(A).

It follows by Corollary 1, and the fact that a.s. E I A |J = E I A =

Remark 5. Relation (15.27) says, in effect, that, if T is ergodic, then, for almost all ω ∈ , the proportion of the points ω, T 1 ω, T 2 ω, . . ., as n → ∞, which lie in any set A ∈ A, is equal to P(A). Definition 10. Consider the measurable space (, A), and let P1 , P2 be two probability measures on A. We say that P1 and P2 are orthogonal, and we write P1 ⊥ P2 , if there exists a set A ∈ A, such that P1 (A) = 1 and P2 (A) = 0 (so that P2 (Ac ) = 1). Corollary 3. Let T : (, A) → (, A) be a (measurable) measure-preserving transformation with respect to two probability measures P1 and P2 on A, and suppose that T is also ergodic with respect to both P1 and P2 . Then either P1 ≡ P2 or P1 ⊥ P2 . Proof. It suffices to prove that, if P1 = P2 , then P1 ⊥ P2 . To this end, let A ∈ A be such that P1 (A) = P2 (A). Then, by Corollary 1, n 1 j−1 IA T ω → P1 (A) n→∞ n j=1

337

338


except on a set B1 ∈ A such that P1 (B1 ) = 0. Also, n 1 j−1 IA T ω → P2 (A) n→∞ n j=1

= 0. Since P1 (A) = P2 (A), we have except on a set B2 ∈ A such that P2 (B2 ) % that B1c ∩ B2c = , since, for ω ∈ B1c , n1 nj=1 I A (T j−1 ω) → P1 (A) and, for n→∞ % ω ∈ B2c , n1 nj=1 I A (T j−1 ω) → P2 (A), and P1 (A) = P2 (A), so that B1 ⊇ B2c , and n→∞

therefore P2 (B1 ) ≥ P2 (B2c ) = 1; i.e., P1 (B1 ) = 0 or P1 (B1c ) = 1 and P2 (B1 ) = 1, so that P1 ⊥ P2 . Corollary 4.

Under the same assumptions as those of Theorem 1, one has n n 1 1 j−1 (1) X j (ω) = X T ω → E X |J . n→∞ n n j=1

j=1

Proof. As it follows from the proof of Theorem 1, we may assume, without loss of generality, that E X |J = 0 a.s. Then we set Yn (ω) =

n n 1 1 j−1 X j (ω) = X T ω , ω ∈ , n ≥ 1, n n j=1

j=1

and we shall show that, as n → ∞ (here as well as throughout the proof), E |Yn | → 0.

(15.28)

a.s.

By Theorem 1, we have that Yn → 0. Then Egorov’s Theorem (see Exercise 11 in Chapter 3) implies that, for every ε > 0 there exists A = A(ε) ∈ A such that P(A) ≤ ε and Yn →0 uniformly on Ac . Thus, for all sufficiently large n independent of ω ∈ Ac , one has #

#

E |Yn | =

|Yn | d P + A

# Ac

|Yn | d P ≤

|Yn | d P + ε P(Ac ) A

# n # 1 (( (( c X j d P + ε P(A ) = |X |d P + ε P(Ac ), ≤ n A A j=1

and therefore

# lim sup E |Yn | ≤

|X | d P + ε P(Ac ). A

(15.29)


But

#

# |X | d P = A

#

#

A∩(|X |>k)

#

A∩(|X |>k)

≤ ≤

(|X |>k)

and therefore (15.29) becomes lim sup E |Yn | ≤

|X | d P +

A∩(|X |≤k)

|X | d P

|X | d P + k P(A)

|X | d P + εk,

# (|X |>k)

Letting ε → 0, we obtain lim sup E |Yn | ≤

|X | d P + εk + ε P(Ac ). # (|X |>k)

|X | d P.

Now letting k → ∞ and utilizing the fact that E |X | < ∞, we get lim sup E |Yn | = 0. Thus, E |Yn | → 0, which is (15.28). Now, let X = X 1 , X 2 , . . . be a stationary process defined on (, A, P) into ∞ ( , B ∞ ), and let Pˆ be the distribution of X under P. Then the coordinate process ˆ = ( Xˆ 1 , Xˆ 2 , . . .) defined on (∞ , B ∞ , P) ˆ is also stationary (by Proposition 1); it X assumes the representation

Xˆ n (x) = Xˆ 1 S n−1 (x) (by (15.11)), where S is the shift transformation on ∞ (see Definition 5); and the shift transformation S is measurable and measure-preserving (by Proposition 8). The processes X and ˆ are equivalent probabilistically, in the sense that they have the same distribution X ˆ respectively). And, whereas X may not have a representation in (under P and P, ˆ does. It follows, terms of a measure-preserving transformation T on into itself, X in particular, that # # X 1d P = Xˆ 1 d Pˆ = E Xˆ 1 , E X1 = provided these expectations exist. Thus, if E |X 1 | < ∞, and the shift transformation is also ergodic, then we have that n 1 ˆ X j →E Xˆ 1 = E X 1 a.s. and in the first mean n j=1

(by Corollaries 1 and 3). Hence n 1 X j →E X 1 a.s. and in the first mean. n j=1

339

340


15.6 Invariant Sets and Random Variables Relative to a Process, Formulation of the Ergodic Theorem in Terms of Stationary Processes, Ergodic Processes Now, it would be desirable to have a definition of invariance and ergodicity, and a formulation of the Ergodic Theorem in terms of the original process without passing to the equivalent coordinate process and without the assumption that X is generated by a transformation T . This is actually done here. The appropriate definition of invariance of a set in A is the following one. Definition 11. A set A ∈ A is said to be invariant (relative to X), if there exists a set B ∈ B ∞ such that A = X−1 B and such that −1 A = X n , X n+1 , . . . B for all n ≥ 1. The class J of invariant sets in A is a σ -field.

Proposition 18.

Proof. J is nonempty since ∈ J , and closure of J under countable unions and complementation is immediate. (See also Exercise 6.) ˆ Remark 6. Consider the coordinate process X and the shift transformation S. Then the σ -field of invariant sets under S, J S , say, is defined by ) * J S = B ∈ B ∞ ; S −1 B = B . Because then S −n B = B, and S −n B = × · · · × ×B, n ≥ 0, n

we have that JS =

⎧ ⎨ ⎩

B ∈ B∞ ;

⎫ ⎬

B = × · · · × ×B, n ≥ 0 . ⎭ n

ˆ i.e., Next, let Jˆ be the σ -field of invariant sets relative to X; 1 2

−1 ∞ ˆ ˆ ˆ J = B∈B ; X n , X n+1 , . . . B = B, n ≥ 0 . But, clearly,

Xˆ n , Xˆ n+1 , . . .

−1

B = × · · · × ×B, n ≥ 1. n−1

Thus, Jˆ = Also,

⎧ ⎨ ⎩

B ∈ B∞ ;

⎫ ⎬

B = × · · · × ×B, n ≥ 0 , so that J S = Jˆ . ⎭ n

15.6 Invariant Sets and Random Variables Relative to a Process

Definition 12. A r.v.Y on (, A) is said to be invariant (relative to X), if there exists a measurable mapping ϕ on (∞ , B ∞ ) into (, B) such that Y = ϕ X n , X n+1 , . . . for all n ≥ 1. The following definition and result will be needed later. Let Z 1 : (, A) → (1 , A 1 ), measurable, Z 2 : (, A) → (2 , A 2 ), measurable. Then Definition 13. We say that Z 2 is a function of Z 1 , if for all ω, ω ∈ , ω = ω for which Z 1 (ω) = Z 1 (ω ), it follows that Z 2 (ω) = Z 2 (ω ). Lemma 4. Let Z 1 , Z 2 be defined as before, and suppose that the σ -field induced by Z 2 , A2 ⊆ A1 , the σ -field induced by Z 1 . Then, if A2 contains the singletons of points in the range of Z 2 , it follows that (i) Z 2 is a function of Z 1 . (ii) There exists a unique function Z on (Z 1 (), A 1 ∩ Z 1 () to (2 , A 2 )) defined by Z (ω1 ) = Z 2 (ω; Z 1 (ω) = ω1 ), so that Z 2 = Z (Z 1 ). (iii) Z is A 1 ∩ Z 1 ()-measurable. The proof of this lemma is left as an exercise (Exercise 7). Proposition 19. Let Y be a r.v. defined on (, A). Then Y is invariant (relative to X), if and only if Y is J -measurable, where J is the σ -field of invariant sets (relative to X). Proof. Let Y be invariant. Then there exists a measurable mapping ϕ on (∞ , B ∞ ) into (, B) such that Y = ϕ X n , X n+1 , . . . for all n ≥ 1. −1 −1 −1 (ϕ C) = X n , X n+1 , . . . B, Let C ∈ B. Then A = Y −1 C = X n , X n+1 , . . . where B ∈ B ∞ and A ∈ A. Thus, for every A ∈ Y −1 B , there exists B ∈B ∞ such that −1 B for all n ≥ 1, so that A ∈ J ; A = X n , X n+1 , . . . i.e., Y is J -measurable. Next, let Y be J -measurable. Then Y −1 B ⊆ J . Make the following identification: (1 , A1 ) = (∞ , B ∞ ), (2 , A2 ) = (, B), = X 1, X 2, . . . , Z2 = Y .

Z1

From the definition of J , it follows that −1 ∞ B for all n ≥ 1. J ⊆ X n , X n+1 , . . .

341

342


Thus,

−1 ∞ Y −1 B ⊆ X 1 , X 2 , . . . B .

Since B contains all singletons in , it follows that the Definition 13 applies andgives the existence of a measurable mapping ϕ defined on the range of X 1 , X 2 , . . . into (, B) such that Y = ϕ X 1, X 2, . . . . By of X 1 , X2 , . . . is the same as that of stationarity of X, it follows that the range X n , X n+1 , . . . for all n ≥ 1. Thus, ϕ X n , X n+1 , . . . is well defined. In order to complete the proof, it suffices to show that ϕ X 1 , X 2 , . . . = ϕ X n , X n+1 , . . . for all n ≥ 1. To see this, let −1 Bx = ϕ −1 {x}, x ∈ , and let A x = X 1 , X 2 , . . . Bx . −1 Then A x = Y −1 {x}, so that A x ∈ J . But then A x = X n , X n+1 , . . . Bx for all n ≥ 1, so that for ω ∈ A x , X n (ω), X n+1 (ω), . . . ∈ Bx for all n ≥ 1, and hence ϕ X n (ω), X n+1 (ω), . . . = x for all n ≥ 1. This completes the proof. Now the Ergodic Theorem becomes as follows. Theorem 3 (Ergodic Theorem). Let X = X 1 , X 2 , . . . be a stationary process defined on (, A, P), and let J be the σ -field of invariant events (relative to X). Then, if |E X 1 | < ∞, one has n 1 a.s. X j → E X 1 |J . n→∞ n

(15.30)

j=1

ˆ be the coordinate process. Proof. Let Pˆ be the distribution of X under P, and let X ˆ and gives that Then Theorem 1 applies to the process X n

1 ˆ X j → Xˆ = E Xˆ 1 |JS a.s. Pˆ . n→∞ n j=1

ˆ 0 ) = 0, Let B0 ∈ B ∞ be the exceptional set, and let A0 = X−1 B0 . Then P(A0 ) = P(B 1 %n c and on A0 , n j=1 X j converges to X , say. We shall show that X = E X 1 |J a.s. We show that X is invariant, and hence J -measurable, by Proposition 19. To this end, set X n + · · · + X n+k−1 , n ≥ 1. Then X = lim ϕ1,k k→∞ k c on A0 .

ϕn,k =


But

X 1 + · · · + X n+k−1 n+k−1 X 1 + · · · + X n−1 × − k→∞ k→∞ n+k−1 k k X 1 + · · · + X n+k−1 = lim = X on Ac0 . k→∞ n+k−1 Thus, if we set limk→∞ ϕ1,k = ϕ X 1 , X 2 , . . . on Ac0 , then lim k→∞ ϕn,k = ϕ Xn , X n+1 , . . . on Ac0 , and both are equal to X ; i.e., (X =)ϕ X n , X n+1 , . . . = ϕ X 1 , X 2 , . . . , n ≥ 1 on Ac0 , and hence X is invariant. From the fact that X = ϕ X n , X n+1 , . . . , n ≥ 1, and the convergences lim ϕn,k = lim

n 1 X j → X on Ac0 , n→∞ n j=1

where

n 1 ˆ X j → Xˆ on B0c , n→∞ n j=1

Ac0 = X−1 B0c with P Ac0 = Pˆ B0 = 0,

we get that

Xˆ = ϕ Xˆ 1 , Xˆ 2 , . . . a.s. Pˆ .

(15.31)

Thus, for every A = X−1 B, B ∈ B ∞ , we get # # # #

Xd P = ϕ X 1, X 2, . . . d P = ϕ Xˆ 1 , Xˆ 2 , . . . d Pˆ = Xˆ d Pˆ A

A

B

B

is finite and ⎛ ⎞ ⎛ ⎞ # # # # n n 1 1 ⎝ ⎝ X j⎠ dP = X d P, Xˆ j ⎠ d Pˆ → Xˆ d Pˆ = n→∞ B n n A B A j=1

j=1

by Corollary 4 (in Section 15.5), because ( ( ( ⎡⎛ ( ⎞ ⎤ ( ( # ( (# n ( ( ( (1 n 1 ( ( ⎣⎝ ( ˆ ˆ ˆ ˆ ⎠ ⎦ X j − X d P( ≤ X j − X (( d P ( ( n A ( n j=1 ( ( ( A j=1 ( ( ( ( (1 n ( ( ˆ ˆ ≤E( X j − X (( → 0, ( n j=1 ( n→∞ so that

#

⎛

⎞ # n n # 1 1 ⎝ ⎠ Xj dP = X jdP → X d P. n→∞ A n n A A j=1

j=1

(15.32)

343

344


This is true, in particular, for A ∈ J . But then there exists B ∈ B ∞ such that −1 A = X k , X k+1 , . . . B, k ≥ 1. By stationarity, we then obtain # # # # Xk d P = Xk d P = X 1d P = X 1 d P. A

(X k ,X k+1 ,...)−1 B

Therefore implies that (15.32) X = E X 1 |J a.s.

$ A

X 1d P =

(X 1 ,X 2 ,...)−1 B

$ A

A

X d P for every A ∈ J . It follows that

Corollary 1.

The convergence in (15.30) holds true in the first mean. Proof. We have X = ϕ X 1 , X 2 , . . . , and X = E X 1 |J a.s., so that E X 1 |J = ϕ X 1 , X 2 , . . . a.s. Thus, ( ( ( ( ( ( ( ( (1 n (1 n ( ( ( ( ( X j − E X 1 |J ( = E ( X j − ϕ X 1 , X 2 , . . . (( E( ( n j=1 ( ( ( n j=1 ( ( ( ( ( # ( # (

(( (1 n (1 n ( = (( X j − ϕ X 1 , X 2 , . . . (( d P = (( Xˆ j − ϕ Xˆ 1 , Xˆ 2 , . . . (( d Pˆ ( n j=1 ( n j=1 ( ( ( ( ( # ( (1 n ( ( ˆ ˆ = ( X j − X (( d Pˆ ( n j=1 (

(by (15.31)), and this tends to 0 as n → ∞ by Corollary 4 (in Section 15.5). Definition 14. The stationary process X = X 1 , X 2 , . . . is said to be ergodic, if P(A) = 0 or 1 for every A ∈ J . We then have the following corollary to Theorem 3. Corollary 2. If X = X 1 , X 2 , . . . of Theorem 3 is also ergodic, then (15.30) holds true with the right-hand side being replaced by E X 1 . $ $ $ $ A E(X 1 |J )d P = $Proof. For A ∈ J , A E(X 1 |J )d P = A X 1 d $P. For P(A) = 1, E(X 1 |J )d $ E X 1 = E X 1 d P = A E X 1 d P; and for $ P = E[E(X 1 |J )] = P(A) = 0, A E(X 1 |J )d P = 0 = A E X 1 d P. Hence E(X 1 |J ) = E X 1 a.s. In Proposition 4, it was seen that functions defined on a stationary process also produce stationary processes. The same is true with regards to ergodicity. More precisely, one has Proposition 20. Let X = X 1 , X 2 , . . . be a stationary process defined on (, A, P) and ϕ be defined as follows: ϕ : (m , B m ) → (, B), measurable, 1 ≤ m ≤ ∞. Set Yn = ϕ X n , X n+1 , . . . , X n+m−1 , n ≥ 1, and Y = Y1 , Y2 , . . . .

.


Then (i) The process Y is stationary. (ii) If J X and JY are the invariant σ -fields associated with the processes X and Y, respectively, it follows that JY ⊆ J X . (iii) If X is ergodic, so is Y. Proof. (i) It follows from Proposition 4. (ii)

−1 B, and Let A ∈JY . Then there exists B ∈ B ∞ such that A = Y1 , Y2 , . . . actually, −1 A = Yn , Yn+1 , . . . B, n ≥ 1. Set ϕ j = ϕ(x j , x j+1 , . . . , x j+m−1 ),

j ≥ 1, and let C =

xn , xn+1 , . . . ;

(ϕn , ϕn+1 , . . .) ∈ B} ∈ B ∞ . For n ≥ 1,

−1 B = ϕ(X n , . . . , X n+m−1 ), A = Yn , Yn+1 , . . . −1 B ϕ(X n+1 , . . . , X (n+1)+m−1 ), . . . = ϕ(X n , . . . , X n+m−1 ), ϕ(X n+1 , . . . , X (n+1)+m−1 ), . . . ∈ B −1 C. = X n , X n+1 , . . . ∈ C = X n , X n+1 , . . .

Thus, A is invariant with respect to X, and hence A ∈ JX . (iii) It is immediate from (ii) and ergodicity of X.

Exercises. 1. Let (, A, P) = ([0, 1), B[0,1) , λ) where λ is the Lebesgue measure, and let the transformation T be defined by & & 1 1 1 1 , T (x) = x − , x ∈ ,1 . T (x) = x + , x ∈ 0, 2 2 2 2 Then show that T is measurable and measure-preserving. 2. Let (, A) = ([0, 1), B[0,1) ), and let the transformation T be defined by T (x) = cx, x ∈ [0, 1), where c is a constant in (0, 1). Then show that there is no probability measure P on B[0,1) such that P({x}) = 0, x ∈ [0, 1), and for which the transformation T is measure-preserving. 3. Refer to Exercise 1 and examine the transformation T from the ergodicity viewpoint. 4. Complete the proof of the converse part of Proposition 16.

345

346


5. Complete the proof of Lemma 2. 6. Work out the details of the proof of Proposition 18. 7. Complete the proof of Lemma 4. Hint: (i) For ω, ω ∈ with ω = ω and Z 1 (ω) = Z 1 (ω ), suppose that Z 2 (ω) = Z 2 (ω ). Use the assumptions that A2 contains the singletons of points in Z 2 (), and that A2 ⊆ A1 , in order to arrive at a contradiction. (ii) By part (i), Z is well defined, and Z 2 = Z (Z 1 ). If also Z 2 = Z (Z 1 ), then show that Z (ω1 ) = Z (ω1 ) for all ω1 ∈ Z 1 (). (iii) For D ∈ A2 , we have A1 ⊆ A2 A = Z 2−1 (D) = Z 1−1 (B) with B = Z −1 (D) ⊆ Z 1 (), and A = Z 1−1 (C) for some C ∈ A1 . Conclude that B = C ∩ Z 1 (). 8. If X n , n ≥ 1, are i.i.d. r.v.s defined on the probability space (, A, P), then (i) Show that the process X = (X 1 , X 2 , . . .) is ergodic. (ii) Derive the Strong Law of Large Numbers under the assumption that E X 1 is finite. Hint: (i) By Proposition 3, it follows that X = (X 1 , X 2 , . . .) is stationary. Let J be the σ -field of invariant sets relative to X (see Definition 11), and let T be the tail σ -field defined on X = (X 1 , X 2 , . . .) (see Definition 1 and preceding discussion in Chapter 14). Then J ⊆ T . Conclude the discussion by using Theorem 10 in Chapter 14. (ii) Refer to Corollary 2 of Theorem 3.

CHAPTER

Two Cases of Statistical Inference: Estimation of a Real-Valued Parameter, Nonparametric Estimation of a Probability Density Function

16

The objective of this chapter is to present some cases, where probabilistic results obtained in previous chapters are used for statistical inference purposes. This should enhance the interest of statisticians in the probability part of the book. There are two instances of statistical inference considered here. One is estimation of a (real-valued) parameter by means of the so-called Maximum Likelihood Estimator (or Estimate) (MLE), and the other is the nonparametric estimation of an unknown probability density function (p.d.f.) by means of the kernel method of estimation. The former problem is discussed in Sections 16.1,16.2,16.3, and the latter in Sections 16.5 and 16.6.

16.1 Construction of an Estimate of a Real-Valued Parameter As a brief introduction, let X 1 , . . . , X n be independent identically distributed (i.i.d.) observations (real-valued r.v.s) defined on the probability space (, A, Pθ ), where here θ is a real-valued parameter taking values in the parameter space , an open subset of the real line . The problem is to construct on estimator (or estimate) of θ ; i.e., a (measurable) function of the X i s taking values in . There are several ways of going about it, and one of the most popular is that of using a MLE (should such an estimate exist). That is, form the so-called likelihood function of the X i s—which in the present set-up is their joint p.d.f.—and then attempt to maximize it with respect to the parameter θ over the parameter space . Should a unique maximizer exist, to be denoted by θˆn = θˆ (X 1 , . . . , X n ), then θˆn is proclaimed to be the MLE of θ . There are several reasons as to why such an estimate is desirable, but we are not going to elaborate on it here. Instead, we are going to give a set of conditions under which one may construct a sequence of roots, θñ = θ˜ (X 1 , . . . , X n ), of the likelihood functions, which is a strongly consistent estimate of θ ; i.e., θñ → θ a.s. [Pθ ], θ ∈ . n→∞


347

348

CHAPTER 16 Two Cases of Statistical Inference

Under an enhanced set of assumptions, such an estimate is also asymptotically normal, when suitably normalized. These results also hold true for the MLE, should it exist.

16.2 Construction of a Strongly Consistent Estimate of a Real-Valued Parameter Let X 1 , . . . , X n be i.i.d. r.v.s defined on the probability space (, A, Pθ ), where the probability measure Pθ depends on a real-valued parameter θ lying in the parameter space , an open subset of . Let X be a r.v. distributed as the X i s, and suppose it has a p.d.f. f (·; θ ) (a Radon-Nikodym derivative with respect to Lebesgue measure; see Remark 1, Chapter 7). We intend to give sufficient conditions under which a strongly consistent estimate of θ may be constructed, based on the likelihood function. Below, we list perhaps the most economical conditions under which this happens. Assumptions. (A1) Assume to be an open interval in . (A2) Suppose that the positivity set of f (·; θ ) is independent of θ ; i.e., S = {x ∈ ; f (x; θ ) > 0} is independent of θ . (A3) Suppose that the p.d.f. f (·; θ ) is identifiable; i.e., for every θ1 and θ2 in with θ1 = θ2 , there exists a Borel set B(⊆S) with Pθ (X ∈ B) > 0 for all θ ∈ such that f (x; θ1 ) = f (x; θ2 ) for x ∈ B. ∂ f (x; θ ) exists for all θ ∈ and all x ∈ S. (A4) Suppose that the derivative ∂θ (A5) Let θ0 be the unknown but true value of the parameter θ , and for any θ ∈ , set K (θ0 , θ ) = Eθ0 log

f (X ; θ0 ) f (X ; θ )

(16.1)

(where log stands throughout for the natural logarithm). Then assume that K (θ0 , θ ) < ∞. In the proof of the consistency theorem stated below, the following result is used which is stated here as a lemma. Lemma 1. Let K (θ0 , θ ) be defined as in (16.1). Then, under assumptions (A2) and (A3), it follows that K (θ0 , θ ) > 0. 2

Proof. (Outline). Set y = − log x(x > 0). Then dd x y2 = x −2 > 0, so that this function is convex. Then apply Jensen’s inequality (see Theorem 15, Chapter 6) with g(X ) = − log [ f (X ; θ1 )/ f (X ; θ0 )] to obtain − log Eθ0 [ f (X ; θ1 )/ f (X ; θ0 )] ≤ Eθ0 {− log[ f (X ; θ1 )/ f (X ; θ0 )]} = K (θ0 , θ1 ). However, Eθ0 [ f (X ; θ1 )/ f (X ; θ0 )] =

S

f (x; θ1 ) f (x; θ0 ) d x f (x; θ0 )

16.2 Construction of a Strongly Consistent Estimate

=

f (x; θ1 )d x = S

f (x; θ1 )d x = 1,

so that − log 1 = 0 ≤ K (θ0 , θ1 ). By (A3), it is seen that K (θ0 , θ1 ) > 0.

In all that follows, we work on the set S (see Assumption (A2)), θ0 stands for the true (but unknown) value of the parameter θ, Xn stands for the random vector (X 1 , . . . , X n ),and δ is an arbitrary positive number. The likelihood funcn f (X i ; θ ), and the log-likelihood function is n (θ |Xn ) = tion is L n (θ |Xn ) = i=1 log L n (θ |Xn ). We may now state the consistency result alluded to earlier. Theorem 1. Under assumptions (A1)–(A5), we can construct a sequence { θn }, n ≥ 1, of estimates of θ0 , where θn is a root of the log-likelihood equation ∂ n ( θn |Xn ) = ∂θ

a.s. 0 on a set of Pθ0 -probability 1, such that θn −→θ0 as n→∞ (i.e., θn −→ θ0 on a set (Pθ0 )

n→∞

of Pθ0 -probability 1). Proof. The proof consists of two parts. First, it is shown that, for δ > 0 sufficiently small, there exists a positive integer n δ such that, for n ≥ n δ and on a set of Pθ0 probability 1, ∂ n (θδ |Xn ) = 0, |θδ − θ0 | < δ, ∂θ where θδ is a (local) maximum of the log-likelihood function n (θ |Xn ). Next, suitably exploiting this result, we proceed with the construction of the θn ’s as described. To this end, set Iδ = (θ0 − δ, θ0 + δ), I¯δ = [θ0 − δ, θ0 + δ], and take δ small enough, so that I¯δ ⊂ . Also, set f (x; θ0 + δ) f (X ; θ0 + δ) log = f (x; θ0 ) d x, J+δ = −K (θ0 , θ0 +δ) = Eθ0 log f (X ; θ0 ) f (x; θ0 ) S (16.2) so that J+δ ≤ 0 (by Lemma 1), and indeed, J+δ < 0 on account of (A3). Then n 1 f (X i ; θ0 + δ) 1 n (θ0 + δ|Xn ) − n (θ0 |Xn ) = log n n f (X i ; θ0 ) i=1

n 1

f (X i ; θ0 + δ) c = −→ J+δ on a set N+δ log , n f (X i ; θ0 ) n→∞ i=1

c ) = 1 (by the SLLN, see Theorem 4 in Chapter 14). Thus, for every say, with Pθ0 (N+δ c ω ∈ N+δ , there exists n +δ = n +δ (ω) positive integer such that 1 n (θ0 + δ|Xn ) − n (θ0 |Xn ) − J+δ < − 1 J+δ for n ≥ n +δ . (16.3) n 2 c and n ≥ n . The inequality in (16.3) From this point on, we work with ω ∈ N+δ +δ implies 1 1 n (θ0 + δ|Xn ) − n (θ0 |Xn ) < J+δ , n 2

349

350


or n (θ0 + δ|Xn ) < n (θ0 |Xn ) +

n J+δ , 2

and hence n (θ0 + δ|Xn ) < n (θ0 |Xn ) (since J+δ < 0).

(16.4)

At this point, refer to relation (16.2), set J−δ = −K (θ0 , θ0 − δ), and work as above in order to conclude that n (θ0 − δ|Xn ) < n (θ0 |Xn )

(16.5)

c with P (N c ) = 1 and n ≥ n for ω ∈ N−δ θ0 −δ = n −δ (ω) positive integer. Set −δ c c c (so that P (N c ) = 1), and n = max(n , n ). Then, for ω ∈ N c Nδ = N+δ ∩ N−δ θ0 δ +δ −δ δ δ and n ≥ n δ , both relations (16.4) and (16.5) hold. Now, look at n (θ |Xn ) as a function of θ ∈ I¯δ (always with ω ∈ Nδc and n ≥ n δ ). By (A4), n (θ |Xn ) is continuous in θ (by differentiability). Then n (θ |Xn ) attains (at least) one (local) maximum; i.e., there is θδ = θδ (Xn ) with θδ ∈ Iδ (open interval, not closed I¯δ ), because of inequalities (16.4) and (16.5). So, n (θδ |Xn ) is a (local) maximum of n (θ |Xn ), θ ∈ Iδ , and of course, |θδ − θ0 | < δ. On the other hand, any local maxima are roots of the log-likelihood ∂ n (θ |Xn ) = 0 (because of the differentiability of n (θ |Xn )). Thus we equation ∂θ have ∂ n (θδ |Xn ) = 0, |θδ − θ | < δ (16.6) ∂θ (on the set Nδc with Pθ0 (Nδc ) = 1, and for n ≥ n δ ). We now embark on the construction of the desired sequence { θn }, n ≥ 1. To this end, for each k = 1, 2, . . ., select (0 0, set ψ(x, t, ρ) = sup|t −t| 0 sufficiently small, such that E ψ(X , t, ρt ) < μ(t) + ε. The collection of (balls) {St = {t ; |t − t| < ρt }, t ∈ T }, clearly, cover T . Then, by assumption (i), there is a finite number (of these balls) Sti , i = 1, . . . , m whose union is T . (This is so by the Heine–Borel covering theorem.) Now, it is clear that for each t ∈ T , there is at least one i = 1, . . . , m, such that t ∈ Sti . Then, from the definition of ψ(x, t, ρ), it follows that ϕ(x, t) ≤ ψ(x, ti , ρti ), t ∈ Sti . Replacing x by X j , we get ϕ(X j , t) ≤ ψ(X j , ti , ρti ), t ∈ Sti , j = 1, . . . , n, and hence

n n 1

1

ϕ(X j , t) ≤ ψ(X j , ti , ρti ), t ∈ Sti . n n j=1

j=1

It follows that ⎤ ⎤ ⎡ n n

1 1 sup ⎣ ϕ(X j , t)⎦ ≤ sup ⎣ ψ(X j , ti , ρti )⎦ n n t∈T t∈T j=1 j=1 ⎤ ⎡ n

1 = max ⎣ ψ(X j , ti , ρti )⎦ . 1≤i≤m n ⎡

j=1

However, by the SLLN, n 1

a.s. ψ(X j , ti , ρti ) −→ Eψ(X , ti , ρti ), i = 1, . . . , m, n→∞ n j=1

and hence ⎤ n

1 a.s. max ⎣ ψ(X j , ti , ρti )⎦ −→ max Eψ(X , ti , ρti ). n→∞ 1≤i≤m 1≤i≤m n ⎡

j=1

On the other hand, for sufficiently large n, E ψ(X , ti , ρti ) < μ(ti ) + ε, i = 1, . . . , m,

16.3 Some Preliminary Results

so that max Eψ(X , ti , ρti ) ≤ max [μ(ti ) + ε]

1≤i≤m

1≤i≤m

= max μ(ti ) + ε ≤ sup μ(t) + ε. 1≤i≤m

t∈T

Hence, for sufficiently large n, ⎤ ⎡ n

1 ψ(X j , ti , ρti )⎦ ≤ sup μ(t) + ε a.s., max ⎣ 1≤i≤m n t∈T j=1

and then

⎤ n

1 ϕ(X j , t)⎦ ≤ sup μ(t) + ε a.s. sup ⎣ n t∈T t∈T ⎡

j=1

Taking the limsup as n→∞, we get ⎧ ⎤⎫ ⎡ n ⎨ ⎬

1 limsup sup ⎣ ϕ(X j , t)⎦ ≤ sup μ(t) + ε a.s. ⎩ t∈T n ⎭ t∈T j=1

Finally, letting ε→0, we obtain ⎧ ⎤⎫ ⎡ n ⎨ ⎬

1 limsup sup ⎣ ϕ(X j , t)⎦ ≤ sup μ(t) a.s. ⎩ t∈T n ⎭ t∈T

j=1

The result below is a uniform version of the SLLN, which, of course, is interesting on its own right; it will also be employed in establishing asymptotic normality of the strongly consistent estimates of Theorem 1 and of the MLE, should it exist. Proposition 1. there, it holds

In the notation of Lemma 2 and under the assumptions (i)–(iii) made ⎤ n 1

⎣ lim ϕ(X j , t) − μ(t)⎦ = 0 a.s. sup n→∞ t∈T n j=1

(Thus, as n→∞, n1

⎡

n

j=1 ϕ(X j , t)

a.s.

−→ μ(t) uniformly in t ∈ T .)

Proof. In the first place, the function μ(t) = Eϕ(X , t) is continuous. Indeed, ϕ(X , t ∗ ) → ϕ(X , t ∗ ) as t ∗ → t (by assumption (ii)), and |ϕ(X , t ∗ )| ≤ h(X ), independent of t ∗ , with Eh(X ) < ∞ (by assumption (iii)). It follows that μ(t ∗ ) = Eϕ(X , t ∗ ) −→ Eϕ(X , t) = μ(t). ∗ t →t

353

354


(This is so by the Dominated Convergence Theorem, Theorem 3, Chapter 5.) Next, suppose that the proposition is true when μ(t) = 0; i.e., we suppose that ⎤ ⎡ n 1

⎣ ϕ(X j , t)⎦ = 0 a.s., sup lim n→∞ t∈T n j=1 de f

and replace ϕ(x, t) by ψ(x, t) = ϕ(x, t) − μ(t). Clearly, ψ(x, t) satisfies assumption (ii), and |ψ(x, t)| = |ϕ(x, t) − μ(t)| ≤ |ϕ(x, t)| + |μ(t)| ≤ h(x) + Eh(X ), since |ϕ(X , t)| ≤ h(X ) is equivalent to −h(X ) ≤ ϕ(X , t) ≤ h(X ) and hence −Eh(X ) ≤ Eϕ(X , t) ≤ Eh(X ), or |Eϕ(X , t)| ≤ Eh(X ), equivalently, |μ(t)| ≤ Eh(X ). So ψ(x, t) also satisfies assumption (iii) (with h(x) replaced by h(x) + Eh(X )). Then ⎤ ⎡

1 n ⎣ ψ(X j , t)⎦ = 0 a.s., sup lim n→∞ t∈T n j=1

or

⎫ n 1 ⎬ lim ϕ(X j , t) − μ(t) sup n→∞ ⎩ t∈T n ⎭ j=1 ⎤ ⎡ n 1

⎣ = lim ϕ(X j , t) − μ(t)⎦ = 0 a.s. sup n→∞ n t∈T ⎧ ⎨

(16.9)

j=1

In the sequel, we work with ψ(x, t) and apply Lemma 2 to get ⎧ ⎤⎫ ⎡ n ⎬ ⎨

1 limsup sup ⎣ ψ(X j , t)⎦ ≤ 0 a.s. ⎭ n n→∞ ⎩ t∈T

(16.10)

j=1

Clearly, the function −ψ(x, t) also satisfies assumptions (ii)–(iii) (and E[−ψ(X , t)] = 0). Then Lemma 2 yields ⎧ ⎤⎫ ⎡ n ⎬ ⎨

1 ψ(X j , t)⎦ ≤ 0 a.s. (16.11) limsup sup ⎣− ⎭ n n→∞ ⎩ t∈T j=1

On the basis of the last two conclusions and Remark 2 below, we have ⎫ ⎧ ⎬ n ⎨ 1

ψ(X j , t) 0 ≤ limsup sup ⎩ n→∞ t∈T n j=1 ⎭

16.4 Asymptotic Normality of the Strongly Consistent Estimate

⎧ ⎤⎫ ⎤⎫ ⎡ n n ⎬ ⎬ ⎨

1 1 ≤ limsup sup ⎣ ψ(X j , t)⎦ + limsup sup ⎣− ψ(X j , t)⎦ ⎭ ⎭ n n n→∞ ⎩ t∈T n→∞ ⎩ t∈T ⎧ ⎨

⎡

j=1

j=1

≤ 0 a.s.

(16.12) Therefore the limn→∞ supt∈T n1 nj=1 ψ(X j , t) exists with probability 1 and equals 0. Reverting to the function ϕ(x, t), we obtain the desired result, as is explained in relation (16.9). The completion of the proof requires justification of the second inequality on the right-hand side in relation (16.12). This is done below.

Remark 2.

For any functions gn (t), t ∈ T , n ≥ 1, the following are true: sup |gn (t)| = max sup gn (t), sup[−gn (t)] , t∈T

t∈T

t∈T

and limsup sup |gn (t)| ≤ limsup sup gn (t) + limsup sup[−gn (t)] . n→∞

t∈T

n→∞

n→∞

t∈T

t∈T

The first assertion is immediate. As for the second, we have: Set αn = sup gn (t), βn = sup[−gn (t)], t∈T

t∈T

and α = limsup αn , β = limsup βn . Then, from the definition of the limsup, we n→∞

n→∞

n→∞

have that, for every ε > 0 and all sufficiently large n, αn ≤ α + ε, βn ≤ β + ε. Hence max{αn , βn } ≤ (α + β) + ε, and therefore limsup [max{αn , βn }] ≤ (α + β) + ε n→∞ = limsup αn + limsup βn + ε. n→∞

n→∞

Letting ε→0, we get the desired result.

16.4 Asymptotic Normality of the Strongly Consistent Estimate The objective here is to establish asymptotic normality for the strongly consistent estimate constructed in Theorem 1 and of the MLE, should it exist, under suitable conditions. For this purpose, we review the relevant notation and list conditions under which this result may be established. To this end, let X be a r.v. defined on the probability space (, A, Pθ ), where the parameter θ belongs in the parameter space .

355

356


Assumptions. (B1) The parameter space is an open subset of . (B2) The set S = {x ∈ ; f (x; θ ) > 0} is independent of θ . (B3) The p.d.f. f (·; θ ) is identifiable; i.e., for every θ1 and θ2 in with θ1 = θ2 , there exists a set B(⊆S) with Pθ (X ∈ B) > 0 for all θ ∈ such that f (x; θ1 ) = f (x; θ2 ) for x ∈ B. ∂ f (x; θ ) exists for all θ and all x ∈ S and (B4) The derivative ∂θ ∂ ∂ f (x; θ )d x. f (x; θ )d x = ∂θ S ∂θ S (B5) Let θ0 be the (unknown) true value of the parameter θ , and set f (X ; θ0 ) , θ ∈ . K (θ0 , θ ) = Eθ0 log f (X ; θ ) Then it is assumed that 0 < K (θ0 , θ ) < ∞. ∂2 (B6) The derivative ∂θ 2 f (x; θ ) exists for all θ and all x ∈ S, is continuous in θ , and ∂2 ∂θ 2

f (x; θ )d x = S

S

(B7) Set I (θ0 ) = Eθ0

∂2 f (x; θ )d x. ∂θ 2

!2 ∂ log f (X ; θ ) ∂θ θ=θ0

for the Fisher information number. Then it is assumed that 0 < I (θ0 ) < ∞. (B8) For each θ ∈ , there is a compact 2 neighborhood N (θ ) with θ belonging in the ∂ interior of N (θ ), such that ∂θ 2 log f (x; θ ) ≤ H (x) for all θ in N (θ ) and all x ∈ S, for some bounding (measurable) function H with Eθ H (X ) < ∞. Notice that (B1)–(B3) are the same as (A1)–(A3), and (B4) is a strengthening of (A4). It has already been seen (see Theorem 1) that, under assumptions (B1)–(B5), there is at least one root θn of the likelihood equation, with probability 1, so that a.s. θn −→ θ0 as n→∞. Here it will be shown that this sequence is also asymptotically (Pθ0 )

normal. Namely, θn } be Theorem 2. Let θ0 be the (unknown) true value of the parameter θ , and let { a sequence as in Theorem 1. Then, under assumptions (B1)–(B8),

d √ n θn − θ0 −→ N (0, 1/I (θ0 )), as n→∞. (Pθ0 )

The following results, stated as lemmas, are used on many occasions, and either directly or indirectly in the proof of Theorem 2.


Lemma 3. Let X be a r.v. defined on the probability space (, A, Pθ ), θ ∈ , with p.d.f. f (·; θ ). Then: (i) Under assumptions (B1), (B2), and (B4), ∂ Eθ log f (X ; θ ) = 0. ∂θ (ii) Under assumptions (B1), (B2), and (B6),

2 2 ∂ ∂ log f (x; θ ) f (x; θ ) d x = Eθ log f (X ; θ ) I (θ ) = ∂θ ∂θ ∂ log f (X ; θ ) (by part (i)) = Varθ ∂θ 2 ∂ = −Eθ log f (X ; θ ) . ∂θ 2

(iii) Under assumptions (B1), (B2), (B4), and (B6), n 1 ∂ d log f (X j ; θ )−→ N (0, I (θ )), as n→∞, √ (Pθ0 ) ∂θ n j=1

where X 1 , . . . , X n are independent r.v.s distributed as the r.v. X . Proof. (i) We have f (x; θ ) d x, so that by differentiation, 1= S d ∂ f (x; θ )d x (by Theorem 4, Chapter 5) 0= f (x; θ ) d x = dθ S S ∂θ " ∂ f (x; θ ) = f (x; θ ) f (x; θ )d x ∂θ S ∂ ∂ log f (x; θ ) f (x; θ ) d x = Eθ log f (X ; θ ) ; i.e., = ∂θ S ∂θ ∂ Eθ ∂θ log f (X ; θ ) = 0, as was to be seen. (ii) Once again, f (x; θ ) d x, and hence 1= S

d2 0= dθ 2

f (x; θ ) d x = S

S

∂2 f (x; θ )d x (by Theorem 4, Chapter 5) ∂θ 2

357

358


2 ∂ ∂2 log f (x; θ ) f (x; θ )d x − f (x; θ ) d x 2 S ∂θ S ∂θ 2 ∂ + log f (x; θ ) f (x; θ ) d x S ∂θ ⎧ ∂ 2 ⎫ ⎨ ∂2 f (x; θ ) f (x; θ ) ⎬ ∂θ 2 − ∂θ 2 = f (x; θ ) d x + I (θ ) f (x; θ ) ⎭ S ⎩ f (x; θ ) 2 ∂ 2 ∂ f (x; θ ) f (x; θ ) − ∂θ f (x; θ ) ∂θ 2 f (x; θ ) d x + I (θ ) = f 2 (x; θ ) S # !$ ∂ f (x; θ ) ∂ ∂θ = f (x; θ ) d x + I (θ ) ∂θ f (x; θ) S ∂ ∂ log f (x; θ ) f (x; θ ) d x + I (θ ) = S ∂θ ∂θ 2 ∂ = log f (x; θ ) f (x; θ ) d x + I (θ ) 2 S ∂θ 2 ∂ = Eθ log f (X ; θ ) + I (θ ), so that ∂θ 2 2 ∂ I (θ ) = −Eθ log f (X ; θ ) , as was to be seen. ∂θ 2

=

(iii) By parts (i) and (ii), ∂ ∂ log f (X ; θ ) = 0, Varθ log f (X ; θ ) = I (θ ). Eθ ∂θ ∂θ Then the CLT (Corollary to Theorem 2, Chapter 12) yields

∂ 1 d log f (X j ; θ ) −→ N (0, 1), n→∞ n I (θ ) j=1 ∂θ n

√

and hence (by Theorem 4 (ii), Chapter 5), n 1 ∂ d log f (X j ; θ ) −→ N (0, I (θ )). √ n→∞ ∂θ n j=1

Lemma 4.

Set

ψ(X ; θ ) =

∂ ∂2 ˙ ; θ) = log f (X ; θ ), and ψ(X log f (X ; θ ). ∂θ ∂θ 2


Also, let

1

Bn = − 0

n 1 ψ˙ X j ; θ0 + λ( θn − θ0 ) dλ. n

(16.13)

j=1

a.s.

Then, under assumptions (B1)–(B3), (B4), (B6)–(B8), it holds Bn −→ I (θ0 ), as n→∞. (Pθ0 )

Proof.

By assumption (B8), ψ(X ˙ ; θ ) ≤ H (X ), θ ∈ N (θ0 ), Eθ0 H (X ) < ∞,

˙ ˙ ; θ )→ψ(X ˙ ; θ0 ) as whereas by assumption (B6) (continuity of ψ(x; θ ) in θ ), ψ(X θ →θ0 . Then ˙ ; θ ) −→ Eθ0 ψ(X ˙ ; θ0 ) (= −I (θ0 ), by Lemma 3(ii)). Eθ0 ψ(X θ→θ0

(16.14)

(This is so by the Dominated Convergence Theorem, Theorem 3, Chapter 5.) For some ρ > 0, set Sρ = {θ ∈ ; |θ − θ0 | ≤ ρ}, and in Lemma 2 identify T with ˙ ; θ ). Then assumptions(i)–(iii) in the ˙ θ ), and μ(t) with Eθ0 ψ(X Sρ , ϕ(x, t) with ψ(x; lemma just cited are satisfied. Under these assumptions, Proposition 1 applies and yields n a.s. 1

˙ ˙ ψ(X j ; θ ) − E0 ψ(X ; θ ) −→0 as n→∞. sup θ∈Sρ n j=1 (Pθ0 ) This means that for every ω in an event E 1 with Pθ0 (E 1 ) = 1 the last convergence holds (pointwise) when the r.v.s X j are evaluated at ω; i.e.,

1 n ˙ ˙ ψ[X j (ω); θ ] − Eθ0 ψ(X ; θ ) −→0 as n→∞. sup θ∈Sρ n j=1 Hence, for every ε > 0 there exists N1 = N1 (ε, ω) > 0 integer, such that

1 n ε ˙ j (ω); θ ] − Eθ0 ψ(X ˙ ; θ ) < , n ≥ N 1 . sup ψ[X 2 θ∈Sρ n j=1

(16.15)

a.s. On the other hand, by Theorem 1, θn −→ θ0 as n→∞, which implies that, for every (Pθ0 )

ω in an event E 2 with Pθ0 (E 2 ) = 1, θn −→ θ0 . Thus, for ω ∈ E 2 and ρ > 0 as n→∞

above there exists N2 = N2 (ρ, ω) > 0 integer such that | θn (ω) − θ0 | < ρ, n ≥ N2 .

(16.16)

359

360


Set N0 = N0 (ε, ρ, ω) = max{N1 , N2 } and restrict ω in E 1 ∩ E 2 , call it E 0 (with Pθ0 (E 0 ) = 1). Then both inequalities (16.15) and (16.16) hold, provided n ≥ N0 ; i.e.,

ε 1 n ˙ ˙ ψ[X j (ω); θ ] − Eθ0 ψ(X ; θ ) < and | θn (ω) − θ0 | < ρ, n ≥ N0 . sup θ∈Sρ n j=1 2 (16.17) From this point on, we work with ω in E 0 (with Pθ0 (E 0 ) = 1) and n ≥ N0 . Also, for notational convenience, omit the evaluation of r.v.s at ω. We have then 1

n 1 ˙ ψ X j ; θ0 + λ(θn − θ0 ) dλ − I (θ0 ) |Bn (θ0 ) − I (θ0 )| = − 0 n j=1 1

n 1 = θn − θ0 ) dλ + I (θ0 ) ψ˙ X j ; θ0 + λ( 0 n j=1 1

n 1 ˙ ; θ0 ) = θn − θ0 ) dλ − Eθ0 ψ(X ψ˙ X j ; θ0 + λ( 0 n j=1 1 n 1

˙ ; θ ) ≤ ψ˙ X j ; θ0 + λ( θn − θ0 ) dλ − Eθ0 ψ(X 0 n j=1 ˙ ; θ ) − Eθ0 ψ(X ˙ ; θ0 ) + Eθ0 ψ(X ⎫ ⎧ 1 ⎨ n ⎬

1 ˙ ; θ ) dλ = ψ˙ X j ; θ0 + λ( θn − θ0 ) − Eθ0 ψ(X ⎭ 0 ⎩ n j=1 ˙ ; θ ) − Eθ0 ψ(X ˙ ; θ0 ) . + Eθ0 ψ(X (16.18) But on E 0 and for n ≥ N0 , [θ0 + λ( θn − θ0 )| θn − θ0 )] − θ0 = |λ( ≤ |θn − θ0 | (since 0 ≤ λ ≤ 1) < ρ (by (16.17)), so that θ0 + λ( θn − θ0 ) (= θ (ω)), call it θ ∗ , lies in Sρ . Also, on the right-hand side of (16.18), replace the arbitrary θ by θ ∗ . We have then ⎤ ⎡ 1 n

⎣1 ˙ j ; θ ∗ ) − Eθ0 ψ(X ˙ ; θ ∗ )⎦ dλ ψ(X |Bn (θ0 ) − I (θ0 )| ≤ n 0 j=1 ∗ ˙ ; θ ) − Eθ0 ψ(X ˙ ; θ0 ) . + Eθ0 ψ(X (16.19)


However, by (16.17),

n ε 1

˙ j ; θ ∗ ) − Eθ0 ψ(X ˙ ; θ ∗ ) < ψ(X 2 n j=1

as long as θ ∗ stays in Sρ (which it does) and for n ≥ N0 , which implies that ⎤ ⎡ 1 n

1 ∗ ∗ ⎣ ˙ j ; θ ) − Eθ0 ψ(X ˙ ; θ )⎦ dλ ψ(X n 0 j=1 ⎡ ⎤ 1 n ε 1

∗ ∗ ⎣ ˙ j ; θ ) − Eθ0 ψ(X ˙ ; θ )⎦ dλ < . ≤ ψ(X (16.20) 2 n 0 j=1

Furthermore, by (16.14), for ε > 0 there exists δ(ε) > 0 such that ε Eθ ψ(X ˙ ; θ ∗ ) − Eθ0 ψ(X ˙ ; θ0 ) < , (16.21) 0 2 provided |θ ∗ −θ0 | < δ(ε). This will be, indeed, the case if ρ is chosen to be < δ(ε). So, for every ε > 0, choose ρ < δ(ε), so that (16.21) is satisfied. Inequality (16.20) is also satisfied for ω in E 0 (with Pθ0 (E 0 ) = 1) and n ≥ N0 . Combining relations (16.18), (16.20), and (16.21), we obtain |Bn (θ0 )− I (θ0 )| < ε on the event E 0 (with Pθ0 (E 0 ) = 1), provided n ≥ N0 . This is equivalent to saying that Bn (θ0 ) −→ I (θ0 ) as n→∞. (Pθ0 )

The remark below will be used in the proof of Theorem 2. Remark 3. When expanding a function according to Taylor’s formula, the following form of a remainder often proves convenient. To this effect, let g be a real-valued function defined on , and assume it has a continuous derivative to be denoted by g. ˙ Then 1

g(x + t) = g(x) + t

g(x ˙ + λt)dλ.

0

A similar expression holds when we assume the existence of higher-order derivatives, as well as when g is defined on k (k ≥ 2). Such formulas are exhibited in relation (8.14.3), page 186, in Dieudonné (1960). Proof of Theorem 2. For the independent r.v.s X 1 , . . . , X n with p.d.f. f (·; θ ), consider the likelihood function L n (θ |Xn ) = nj=1 f (X j ; θ ), where Xn = (X 1 , . . . , X n ), and set n

log f (X j ; θ ). n (θ )(= n (θ |Xn )) = log L n (θ |Xn ) = j=1

Then, by the notation introduced in Lemma 4,

∂ ψ(X j ; θ ), ˙n (θ ) = n (θ ) = ∂θ n

j=1

361

362


and hence

∂ ˙ ˙ j ; θ ). ψ(X ¨n (θ ) = n (θ ) = ∂θ n

j=1

Consider ˙n (θ ) and, by using Remark 3 with g(x) replaced by ˙n (θ ), expand it to obtain 1 ¨n [θ0 + λ(θ − θ0 )] dλ ˙n (θ ) = ˙n (θ0 ) + (θ − θ0 ) 0 ⎫ ⎧ 1 ⎨

n ⎬ ˙ j ; θ0 + λ(θ − θ0 )] dλ. (16.22) ψ[X = ˙n (θ0 ) + (θ − θ0 ) ⎭ 0 ⎩ j=1

˙ In (16.22), replace √ θ by θn , recall that n (θn ) = 0 (with Pθ0 -probability 1), and divide both sides by n to get ⎫ ⎧ 1⎨ n ⎬ √ 1 ˙ 1

˙ j ; θ0 + λ( θn − θ0 ) θn − θ0 )] dλ − ψ[X √ n (θ0 ) = n( ⎭ n 0 ⎩ n j=1 √ = Bn (θ0 ) × n( θn − θ0 ) (by the definition of Bn (θ0 ) in (16.13)). a.s.

By Lemma 4, Bn (θ0 )−→ I (θ0 ) as n→∞. Therefore, with Pθ0 -probability 1 and suffi(Pθ0 )

a.s.

ciently large n, Bn−1 (θ0 ) exists and, clearly, Bn−1 (θ0 )−→ I −1 (θ0 ). Also, by Lemma 3 (Pθ0 )

(iii),

d √1 ˙n (θ0 ) −→ n (Pθ0 )

N (0, I (θ0 )). Then Slutsky’s theorem (Theorem 8(ii), Chapter 8)

implies that 1 1 d × N (0, I (θ0 )) = N (0, I −1 (θ0 )). Bn−1 (θ0 ) × √ ˙n (θ0 ) −→ (Pθ0 ) I (θ0 ) n √ √ From √1n ˙n (θ0 ) = Bn (θ0 )× n( θn −θ0 ), we get n( θn −θ0 ) = Bn−1 (θ0 )× √1n ˙n (θ0 ) (with Pθ0 -probability 1), so that, as n→∞, √

d n( θn − θ0 )−→ N (0, I −1 (θ0 )), (Pθ0 )

as was to be seen. To this theorem, there is the following corollary.

Corollary. Suppose that for all sufficiently large n and with Pθ0 -probability 1 there exists a unique MLE θn of θ0 . Then, under assumptions (B1)–(B8) and as n→∞, we have √ d n( θn − θ0 ) −→ N (0, I −1 (θ0 )). (Pθ0 )


Proof. It is immediate, since all arguments used in the proofs apply when θn is replaced by θn (see also Corollary to Theorem 1). Remark 4. When the parameter space is r -dimensional, assumptions (B1), (B4), (B6)–(B8) have got to be modified suitably. The proofs of Lemma 2 and Proposition 1 are independent of the dimensionality of T . However, the formulations of Theorem 2 and of Lemmas 3 and 4 must be modified, as well as their proofs. The conclusion of the theorem, nevertheless, remains valid, properly interpreted. Most of the relevant derivations for the multidimensional parameter case, both for Theorems 1 and 2, can be found in Theorem 17, page 114, and Theorem 18, page 121, in Ferguson (1996). In reference to Theorem 1, it should be pointed out that the assumptions made there are sufficient but not necessary—albeit economical—for the theorem to hold. The following example illustrates the point. Example 1. Let X ∼ U (0, θ ). Then assumptions (A2) and (A4) are not satisfied. On θn is given by θn = X (n) , the basis of a random sample of size n, X 1 , . . . , X n , the MLE θn − θ ). Then its and its p.d.f. is given by gn (t; θ ) = θnn t n−1 , 0 < t < θ . Set Yn = n(

n−1 p.d.f. is f Yn (y; θ ) = 1n θ + y , −nθ < y < 0, and the p.d.f. of Z n = θn − θ is f Z n (t; θ ) =

n θ n (θ

θ

n

+ t)n−1 , −θ < t < 0. It follows that, for every ε > 0,

Pθ (| θn − θ | > ε) = Pθ ( θn − θ > ε or θn − θ < −ε) = Pθ (θn − θ > ε) + Pθ ( θn − θ < −ε) = Pθ ( θn − θ < −ε) −ε −ε n n 1 n−1 n = n (θ + t) dt = n × (θ + t) θ −θ θ n −θ &n % 1 ε = n (θ − ε)n = 1 − . θ θ

∞ ε n By choosing (0

An Introduction to Measure-theoretic Probability, 2nd edition

Introduction to Probability, 2nd Edition

Introduction to Probability, 2nd Edition

An Introduction to Optimization, 2nd Edition

An Introduction to Mechanics, 2nd edition

An Introduction to Fire Dynamics, 2nd Edition

An Introduction to Numerical Analysis (2nd Edition)

An Introduction to Health Psychology, 2nd Edition

An Introduction to Probability Theory

An introduction to mathematical probability

An introduction to probability theory

An Introduction to Probability Theory

Introduction to Vygotsky, 2nd Edition

Introduction to Algebra, 2nd edition

Introduction to metallurgy, 2nd Edition

Probability. An introduction

An introduction to probability and statistics

An Introduction to Probability and Inductive Logic

An Introduction to Measure and Probability

An introduction to measure and probability

An introduction to probability and statistics

Art Therapy: An Introduction, 2nd edition

Spectroscopy in Catalysis: An Introduction, 2nd Edition

Optics and Photonics: An Introduction, 2nd Edition

Clinical Audiology: An Introduction, 2nd Edition

Fundamentals of Probability (2nd Edition)

Probability and Statistics (2nd Edition)

Probability and Statistics (2nd Edition)

Nanotribology and Nanomechanics: An Introduction - 2nd Edition

An Introduction to Measure-theoretic Probability, 2nd edition

Introduction to Probability, 2nd Edition

Introduction to Probability, 2nd Edition

An Introduction to Optimization, 2nd Edition

An Introduction to Mechanics, 2nd edition

An Introduction to Fire Dynamics, 2nd Edition

An Introduction to Numerical Analysis (2nd Edition)

An Introduction to Health Psychology, 2nd Edition

An Introduction to Probability Theory

An introduction to mathematical probability

An introduction to probability theory

An Introduction to Probability Theory

Introduction to Vygotsky, 2nd Edition

Introduction to Algebra, 2nd edition

Introduction to metallurgy, 2nd Edition

Probability. An introduction

An introduction to probability and statistics

An Introduction to Probability and Inductive Logic

An Introduction to Measure and Probability

An introduction to measure and probability

An introduction to probability and statistics

Art Therapy: An Introduction, 2nd edition

Spectroscopy in Catalysis: An Introduction, 2nd Edition

Optics and Photonics: An Introduction, 2nd Edition

Clinical Audiology: An Introduction, 2nd Edition

Fundamentals of Probability (2nd Edition)

Probability and Statistics (2nd Edition)

Probability and Statistics (2nd Edition)

Nanotribology and Nanomechanics: An Introduction - 2nd Edition

Recommend Documents