An Introduction to Probability Theory and Its Applications, Vol. 2

4A v

~ 7~

-r371~ I

..'"'- ..,I 'I . ~T

An Introduction

to .Probability Theory and Its Applications WILLIAM FELLER (1906-'1970) Eugene Higgins Professor of Mathematics Princeton University

VQLUME II

Preface to the First Edition 1941 and 1948) the intere~t in prQbability was nQt yet widespread. Teaching was Qn a very limited scale and tQpics such as :MarkQv chains, which are nQW extensively used in several disciplines, were highly specialized chapters Qf pure mathematics. The first vQlume may therefQre be likened to. an allpurpQse travel guide to. a strange cQuntry. To. describe the nature Qf prQbability it had to. stress the mathematical CQntent Qf the theQry as well as the surprising variety Qf PQtential applicatiQns. It was predicted that the ensuing fluctuatiQns in the level Qf difficulty WQuld limit the usefulness Qf the bQQk. In reality it is widely used even tQday, when its nQvelty has WQrn Qff and its attitude and material are available in newer bQQks written fQr special purpQses. The bQQk seems even to. acquire new friends. The fact that laymen are nQt deterred. by passages which prQved difficult to. students Qf mathematics shQWS that the level Qf difficulty cannQt be measured Qbjectively; it depends Qn the type Qf infQrmatiQn Qne seeks and the details Qne is prepared to. skip. The traveler Qften has the chQice between climbing' a peak Qr using a cable car. In view .Qf thi~ success the secQnd vQlume is written in the same style. It invQlve$ harder mathematics, but mQst Qf the text can be read Qn different levels. The handling Qf measure theQry may illustrate this PQint. Chapter IV cQntains an infQrmal intrQductiQn to. the basic ideas Qf measure theQry and the cQnceptual fQundatiQns Qf prQbability. The same chapter lists the fe~ facts Qf measure theQry used in the subsequent chapters to. fQrmulaie . analytical theQrems in their simplest fQrm and to. aVQid futile discussiQns Qf regularity conditiQns. The main functiQn Qf measure theQry in this cQnnectiQn is to. justify fQrmal QperatiQns and passages to the limit that WQuld never be . questiQned by a nQn-mathematician. Readers interested primarily in practical results will therefQre nQt feel any need fQr measure theQry. To. facilitate access to. the individual tQpics the chapters are rendered as self-cQntained as PQssible, and sQmetimes special cases are treated separately ahead Qf the general theQry. VariQus tQpics (such as stable distributiQns and renewal theQry) are discussed at several places frQm different angles. To. aVQid repetitiQns, the definitiQns and' illustrative examples are cQllected in AT THE TIME THE FIRST VOLUME OF THIS BOOK WAS WRITTEN (BETWEEN

vii

Vlll

PREFACE

chapter VI, which may be described as a collection of introductions to the subsequent chapters. The skeleton of the book consists of chapters V, VIII, and XV. The reader will decide for himself how much of the preparatory chapters to read and which excursions to take. Experts will find new results and proofs, but more important is the attempt to consolidate and unify the general methodology. Indeed, certain parts of probability suffer from a lack of coherence because the usual grouping and treatment of problems depend largely on accidents of the historical development. In the resulting confusion closely related problems are not recognized as such and simple things are obscured by complicated methods. Considerable simplifications were obtained by a systematic exploitation and development of the best available techniques. This is true in particular for the proverbially messy field of limit theorems (chapters XVI-XVII). At other places simplifications. were achieved by treating problems in their natural context. For example, an elementary consideration of a particular random walk led to a generalization of an asymptotic estimate which had been derived by hard and laborious methods in risk theory (and under more restrictive conditions independently in queuing). I have tried to achieve mathematical rigor without pedantry in style. For example, the statement that 1/(1 + e) is the characteristic function of ie- Ixl scem~ to me a desirable and legitimate abbreviation for the logically correct version that the function which at the point ~ assumes the value 1/(1 + ~2) is the characteristic function of the function which at the point x assumes the value ie- Ixl • I fear that the brief historical remarks and citations do not render justice to the many authors who contributed t6 probability, but I have tried to give credit wherever possible. The original work is now in many cases superseded by newer research, and as a rule full references are given only to papers to which the reader may want to turn for additional information. For example, no reference is given to my own work on limit theorems, whereas a paper describing observations or theories underlying an example is cited even if it contains no mathematics. 1 Under these circumstances the index of authors gives no indication of their importance for probability theory. Another difficulty is to do justice to the pioneer work to which we owe new directions of research, new approaches, and new methods. Some theorems which were considered strikingly origi naI and deep now appear with simple proofs among more refined results. It is difficult to view such a' theorem in its historical perspective and to realize that here as elsewhere it is the first step that counts. This system was used also in the first volume but was misunderstood by some subsequent writers; they now attribute the methods used in the book to earlier scientists who could not have known them. 1

ACKNOWLEDGMENTS

Thanks to the support by the U.S. Army Research Office of work in probability at Princeton University I enjoyed the help of J. Goldman, L. ~itt, M. Silverstein, and, in particular, of M. M. Rao. They eliminated many inaccuracies and obscurities. All chapters were· rewritten many times and preliminary versions of the early chapters were circulated among friends. In this way I benefited from comments by J. Elliott, ~. S. Pinkham, and L. J. Savage. My special thanks are due to J. L. Doob and J. Wolfowitz for advice. and criticism. The graph of the Cauchy random walk was supplied by H. Trotter. The printing was supervised by Mrs. H. MCDou.gal, and the appearance of the book owes much to her. WILLIAM FELLER

October 1965

.-

IX

THE MANUSCRIPT HAD BEEN FINISHED AT THE TIME OF THE AUTHOR'S DEATH

but no proofs had been received. I am grateful to the publisher for providing a. proofreader to compare the print against the manuscript and for compiling the index. 1. Goldman, A. Grunbaum, H. McKean, L. Pitt, and A. Pittenger divided the book among themselves to check on the mathematics. Every mathematician knows what an incredible amount of work that entails. I express my deep gratitude to these men and extend my heartfelt thanks for their labor of love.

May 1970

CLARA

Xl

N.

FELLER

In trod tiction

11iE CHARACTER AND ORGANIZATION OF THE BOOK REMAIN UNCHANGED, BUT

the entire text has undergone a thorough revision. Many parts (Chapter XVII, in particular) have been completely rewritten and a few new sections have been added. At a number of places the exposition was simplified by streamlined (and sometimes new) arguments. Some new material has been incorporated into the text. While writing the first edition I was haunted by the fear of an excessively long volume.. Unfortunately, this led me to spend futile months in shortening the original text and economizing on displays. This damage has now been repaired, and a great effort has been spent to make the reading easier. Occasional repetitions will also facilitate a direct access to the individual chapters "and make it possible to read certain parts of this book in conjunction with Volume I. Concerning the organization of the material, see the introduction to the first edition (repeated here), starting with the second paragraph. I am grateful to many readers for pointing out errors or omissions. I especially thank D. A. Hejhal, of Chicago, for an exhaustive and penetrating list of errata and for suggestions covering the entire book. January 1970 Princeton, N.J.

WILLIAM FELLER

xiii

Abbreviations and Conventions Iff Epoch.

is an abbreviation for

if and only if.

This term is used for points on the time axis, while time is reserved for intervals and durations. (In discussions of stochastic processes the word "times" carries too heavy a burden. The systematic use of "epoch," introduced by J. Riordan, seems preferable to varying substitutes such as moment, instant, or point.)

Intervals are denoted' by bars:

0,

I-I

b is an open, a, b a closed interval; -I

1-

half-open intervals are denoted by a, band 0, b. This notation is used also in higher dimensions. The pertinent conventions for vector notations and order relations are found in V,l (and also in IV,2). The symbol (a, b) is reserved for pairs and for points. r 2 :JV, :R ,:R stand for the line, the plane, and the r-dimensional Cartesian space. 1 refers to volume one, Roman numerals to chapters. Thus 1; XI,(3.6) refers to section 3 of chapter XI of volume 1. ~ indicates the end of a proof or of a collection of examples. nand 91 denote, respectively, the normal density and distribution function with zero expectation and unit variance. 0, 0, and Let u "and v depend on a parameter x which tends, say, to a. Assuming that v is positive we write I"J.

u u

= O(v»)

=

o(v)

Ul"Jv

u

J

if v

{remains bounded ~0 ~1.

I(x) U{dx}. For this abbreviation see V,3. Regarding Borel sets and Baire functions, see the introduction to chapter V.

x.v

Contents CHAPTER'

I

1

THE EXPONENTIAL AND THE UNIFORM DENSITIES •

1. Introduction 2. Densities_ Convolutions . 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13.

1

3

The Exponential Density Waiting Time Paradoxes. The Poisson Process The Persistence of Bad Luck. ' Waiting Times and Order Statistics. The Uniform Distribution Ra.ndom Splittings. Convolutions and Covering Theorems . Random Directions The Use of Lebesgue Measure Empirical Distributions Problems for Solution.

8 11 15 17

21 25 26

. '

29 33 36 39

CHAPTER

II

SPECIAL DENSITIES. RANDOMIZATION.

1. Notations and Conventions .

2. *3. 4. 5. 6.

Gamma Distributions . Related Distributions of Statistics Some Common Densities. Randomization and Mixtures Discrete Distributions

.

,

45 45 47

48 49

53 55

• Starred sections are not required for the understanding of the sequel and should be omitted at first reading. xvii

xviii

CONTENTS

7. Bessel Functions and Random Walks

58

8. Distributions on a Circle 9. Problems for Solution.

61 64

CHAPTER

In

DENSITIES IN HIGHER DIMENSIONS.

NORMAL DENSITIES AND

66

PROCESSES

1. Densities 2. Conditional Distributions.

66 71

3. Return to the Exponential and the Uniform Distributions *4. A Characterization of the Normal Distribution 5. Matrix Notation. The Covariance Matrix. 6. Normal Densities and Distributions. *7. Stationary Normal Processes. 8. Markovian Normal Densities. 9. Problems for Solution.

74 . 77 80 83 87 94 99

CHAPTER

IV

103

PROBABILITY MEASURES AND SPACES.

1. Baire Functions 2. Interval Functions and Integrals in at r

.

104 106

3. a-Algebras. Measurability

112

4. Probability Spaces. Random Variables.

115

5. The Extension Theorem .

118

6. Product Spaces. Sequences of Independent Variables. 7. Null Sets. Completion

121 125

CHAPTER

V

at r

127

1. Distributions and Expectations

]28 136 138 143

PROBABILITY DISTRIBUTIONS IN

2. Preliminaries 3. Densities 4. Convolutions

CONTENTS

XIX

5. Symmetrization.

148

6. Integration by Parts.' Existence of Moments 7. Chebyshev's -Inequality

.

150 151

152 156

8. Further Inequalities. Convex Functions

9. Simple Conditional Distributions. Mixtures *10. Conditional Distributions.

160

*11.

162 165

Conditional Expectations .

12. Problems for Solution CHAPTER

VI A

SURVEY OF SOME IMPORTANT DISTRIBUTIONS AND.·PROCESSES

169

1. Stable Distributions in 9t1

169

2. Examples

7. Examples and Problems .

173 176 179 182 184 187

8. Raridonl Walks.

190

3. Infinitely Divisible Distributions in

~1 •

4. Processes with Independent Increments,. *5. Ruin Problems in Compound Poisson Processes

6. Renewal Processes.

- 194

9. The Queuing Process . 10.' Persistent and Transient Random Walks

200

11. General Markov Chains

205 209 215

*12. Martingal~s.

13. Problems for Solution. CHAPTER

vn

LAWS OF LARGE NUMBERS.

ApPLICATIONS IN ANALYSIS.

219

1. Main Lemma and Notations. 2. Bernstein Polynomials. Absolutely Monotone Functions

219

3. Moment Problems.

224

*4. Application to Exchangeable Variables . *5. Generalized Taylor Formula and Semi-Groups

6. Inversion Formulas for Laplace Transforms

222

. 228 230 . 232

xx

CONTENTS

*7. Laws of Large Numbers for Identically Distributed Variables 234 *8. Strong Laws 237 *9. Generalization to Martingales 241 10. Problems for Solution. 244

CHAPTER

VIII

THE BASIC LIMIT THEOREMS

247

1. 2. 3. 4. *5. '6. *7. 8. *9. 10.

247 252 254 258 265 267 270 275 279 284

Convergence of Measures. Special Properties . Distributions as Operators The Central Limit Theorem Infinite Convolutions . Selection Theorems Ergodic Theorems for Markov Chains. Regular Variation . Asymptotic Properties of Regularly Varying Functions Problems for Solution.

CHAPTER

IX

INFINITELY DIVISIBLE. DISTRIBUTIONS AND SEMI-GROUPS.

1. 2. 3. 4. 5. 6. 7. 8. 9. 10.

Orientation . Convolution Semi':'Groups Preparatory Lemmas Finite Variances The Main Theorems Example: Stable Semi-Groups Triangular Arrays- with Identical Distributions. Domains of Attrac.tion Variable Distributions. The Three-Series Theorem Problems for Solution.

290 290 293 296 298 300 305 308 312 JI6 318

CONTENTS

XXI

CHAPTER

X

MARKOV PROCESSES AND SEMI-GROUPS .

1. 2. 3. 4. 5. 6. 7. 8. 9. 10.

The Pseudo-Poisson 1'ype. A Variant: Linear Increments Jump Processes. Diffusion Processes in 5t1 • The Forward Equation. Boundary Conditions Diffusion in Higher Dimensions . Subordinated Processes Markov Processes and Semi-Groups The "Exponential Formula" of Semi-Group Theory Generators. The Backward Equation .

321 322 324 326 332 337 344 . 345 349 353 356

CHAPTER

XI

RENEWAL THEORY

358

1. 2. *3. 4. 5. 6. 7. 8. *9. 10.

358 364 366 368 372 374 377 379 380 385

The Renewal Theorem Proof of the Renewal Theorem ... Refinements Persistent Renewal Processes The Number Nt of Renewal Epochs Terminating (Transient) Processes . Diverse Applications . Existence of Limits in Stochastic Processes. Renewal Theory on the Whole Line. Problems for Solution .

CHAPTER

xn

RANDOM WALKS IN .

5t1

.

389

390 1. Basic Concepts and Notations . 394 2. Duality. Types of Random Walks 3. Distribution of Ladder Heights. Wiener-Hopf Factor398 ization . 402 3a. The Wiener-Hopf Integral Equation.

.,

CONTENTS

XXll

4. 5. 6. 7. 8. 9. 10.

Examples Applications A Combinatorial Lemma. Distribution of Ladder Epochs The Arc Sine Laws Miscellaneous Complements Problems for Solution.

404 408 412

413 417 423 425

CHAPTER

XIII

LAPLACE TRANSFORMS. TAUBERIAN THEOREMS. RESOLVENTS

1. Definitions. The Continuity Theorem 2. Elementary Properties. 3. Examples 4. 5. *6. *7. *8. 9. 10. 11.

Completely Monotone Functions. Inversion Formulas Tauberian Theorems . Stable Distributions Infinitely Divisible Distributions. Higher Dimensions Laplace Transforms for Semi-Groups The Hille-Yosida Theorem Problems for Solution.

429 429 434 436 439 442 448 449 452 454 458 463

CHAPTER

XIV

ApPLICA TIONS OF LAPLACE TRANSFORMS

1. 2. 3. 4. 5. 6. 7. 8. 9. 10.

T~e

466

Renewal Equation: Theory. 466 468 Renewal-Type Equations: Examples Limit Theorems Involving Arc Sine Distributions.. 470 Busy Periods and Related Branching Processes 473 Diffusion Processes 475 Birth-and-Death Processes and Random Walks 479 The Kolmogorov Differential Equations 483 Example: The Pure Birth Process . 488 Calculation. of Ergodic Limits and of First-Passage Times 491 Problems for Solution. 495

xxiii

CONTENTS

CHAPTER

. XV

CHARACTERISTIC FUNCTIONS

1. 2. 2a. 3. 4. 5. 6. 7. *8. 9.

498

.

Definition. Basic Properties

498

Special Distributions. Mixtures

502

Some Unexpected Phenomena

505

Uniqueness. Inversion Formulas

507

Regularity Properties .

511

The Central Limit Theorem for Equal Components.

515

The Lindeberg Conditions

518

Characteristic Functions in Higher Dimensions Two Characterizations' of the Normal Distribution

521 525

Problems for Solution.

526

CHAPTER

XVI·

EXPANSIONS RELATED TO THE CENTRAL LIMIT THEOREM.'

I. 2. 3. 4.

•

531

Notations

532

Expansions for Densities .

533

Sm.oothing . Expansions for Distributions. 5. The Berry-Esseen Theorems .

536 538 542

6. Expansions in the Case of Varying Components' .

546

7. Large Deviations .

548

CHAPTER

XVII

INFINITELY DIVISIBLE DISTRIBUTIONS.

554

I. 'Infinitely Divisible Distributions.

554

2. Canonical Forms. The Main Limit Theorem .

558

2a. Derivatives of Characteristic Func,tions . 3. Examples and Special Properties. 4. Special Properties . 5. Stable Distributions and Their Domains of Attraction *6. Stable Densities 7. Triangular Arrays .

565 566 570 574 581 583

CONTENTS

XXIV

*8. *9. *10. 11. 12.

The Class L. Partial Attraction. "Universal Laws" Infinite Convolutions . Higher Dimensions Problems for Solution.

588 590 592 593 595

CHAPTER XVllI

598

APPLICATIONS OF FOURIER METHODS TO RANDOM WALKS

598

1. The Basic Identity.

*2. 3. 4. 5. 6. 7.

Finite Intervals. Wald's Approximation The Wiener-Hopf Factorization. Implications and Applications Two Deeper Theorems Criteria for Persistency Problems for Solution.

601 604 609 612 .

614

616

CHAPTER XIX

HARMONIC ANALYSIS

619

1. 2. 3. 4. *5. 6. 7.

619

The Parseval Relation. Positive Definite Functions Stationary Processes . Fourier Series . The Poisson Summation Formula Positive Definite Sequences . L2. Theory .

8~

Stochastic Processes and Integrals 9. Problems for Solution.

620 623 626 629 633

635 641 647

ANSWERS TO PROBLEMS .

651

SOME BOOKS ON COGNATE SUBJECTS

655

INDEX .

657

An Introduction to Probability Theory and Its Applications·

CHAPTER I

The Exponential and the Unifortn Densities

1. INTRODUCfION In the course of volume 1 we had repeatedly to deal with probabilities defined by sums of many" small terms,. and we used approximations of the form . (1.1)

P{a

< X·~ b} ""

f

fez) tIx.

The prime example is the normal appr