Introduction to mathematical probability

INTRODUCTION TO

MATHEMATICAL PROBABILITY

BY

J. V. USPENSKY Professor of lllathematica, Stanford University

FIRST EDITION SECOND IMPRESSION

McGRAW-HILL BOOK COMPANY, INc. NEW YORK AND LONDON 1937

CoPYRIGHT, 1937, BY THE McGRAW-HILL BooK CoMPANY, INc. �HINTED

IN THE UNITED

STATES OF AMERICA

All rights reserved. This book, or parts thereof, may not be reproduced in any form without permission of the publishe1·s.

THE MAPLE PRESS COMPANY, YORK, PA.

PREFACE This book is an outgrowth of lectures on the theory of probability which the author has given at Stanford University for a number of years.

At first a short mimeographed text covering only the elementary

parts of the subject was used for the guidance of students.

As time

went on and the scope of the course was gradually enlarged, the necessity arose of putting into the hands of students a more elaborate exposition of the most important parts of the theory of probability.

Accordingly

a rather large manuscript was prepared for this purpose.

The author

did not plan at fir!'t to publish it, but students and other persons who had opportunity to peruse the manuscript were so persuasive that publication was finally arranged. The book is arranged in such a way that the first part of it, consisting of Chapters I to XII inclusive, is accessible to a person without advanced mathematical knowledge. tions.

Chapters VII and VIII are, perhaps, excep­

The analysis in Chapter VII is rather involved and a better way

to arrive at the same results would be very desirable.

At any rate, a

reader who does not have time or inclination to go through all the intricacies of this analysis may skip it and retain only the final results, found in Section 11.

Chapter VIII, though dealing with interesting

and historically important problems, is not important in itself and may without loss be omitted by readers.

Chapters XIII to XVI incorporate

the results of modern investigations.

Naturally they are more complex

and require more mature mathematical preparation. Three appendices are added to the book. far the most important.

Of these the second is by

It gives an outline of the famous Tshebysheff­

Markoff method of moments applied to the proof of the fundamental theorem previously established by anoth�r method in Chapter XIV. No one will dispute Newton's assertion: "In scientiis addiscendis exempla magis prosunt quam praecepta." theory of probability.

But especially is it so in the

Accordingly, not only are a large number of

illustrative problems discussed in the text, but at the end of each chapter a selection of problems is added for the benefit of students. them are mere examples.

Some of

Others are more difficult problems, or even

important theorems which did not find a place in the main text.

In all

such cases sufficiently explicit indications of solution (or proofs) are given. v

vi

PREFACE

The book does not go into applications of probability to other sciences. To present these applications adequately another volume of perhaps larger size would be required.

.

No one is more aware than the author of the many imperfections in the plan of this book and its execution.

To present an entirely satis­

factory book on ·probability is, indeed, a difficult task.

But even with

all these imperfections we hope that the book will prove useful, especially since it contains much material not to be found in other books on the same subject in the English language.

J. V. UsPENSKY. STANFORD UNIVERSITY, September, 1937.

CONTENTS PAGE PREFACE.

•

V

•

1

INTRODUCTION

Scope of Probability-Necessity and Chance-Genesis of Idea. of Prob­ ability-Classical Definition of Mathematical Probability-Its Limitations -Short Sketch of Historical Development of Theory of Probability. CHAPTER I COMPUTATION OF PROBABILITY BY DIRECT ENUMERATION OF C ASES

•

.

14

•

Simple Problems Solved by Direct Application of Definition-Permutations, Arrangements and Combinations-Some Illustrative Problems-Problems for Solution. CHAPTER II THEORE�IS OF TOTAL AND COMPOUND PROBABILITY

.

.

.

.

•

.

.

.

•

•

.

•

.

27

Mutually Exclusive or Incompatible Events-Theorem of Total Prob­ ability in Restricted and Generalized Form-Conditional Probability­ Theorem of Compound Probability-Independent Events-Compound Probability of Independent Events-Problems Illustrating the Combined Use of Theorem of Total and Compound Probabilities-Problems for Solution. CHAPTER III REPEATED TRIALS.

•

.

.

.

.

.

.

.

.

.

.

.

•

.

.

•

•

.

.

.

•

•

•

•

.

.

.

44

Independent and Dependent Trials-Bernoullian Trials-Most Probable Number of Successes-Approximate Formulas in Case of a. Large Number of Trials-Markoff's Method of Continued Fractions-Problems for Solution. CHAPTER IV PROBABILITIES OF HYPOTHESES AND BAYES' THEOREM.

Bayes' Formula for Probability a Estimation

of

Probabilities

•

•

•

•

•

.

•

.

•

•

•

60

Posteriori-Probability of Future Events­

from

Observed

Frequencies-Criticism

of

Underlying Assumptions-Problems for Solution. CHAPTER V UsE OF DIFFERENCE EQUATIONS IN SOLVING PROBLEMS OF PROBABILITY

74

Simple Chains of Trials-Problem of Runs-Examples Requiring Use of Partial Difference Equations-Problems for Solution. CHAPTER VI BERNOULLI'S THEORE�I

•

•

•

.

•

.

•

•

•

.

•

.

•

•

•

.

•

•

•

•

Statement and Proof of Bernoulli's Theorem-Theorem of

•

•

•

.

•

Cantelli­

Significance of Bernoulli's Theorem-Application to Games of Chancevii

96

viii

CONTENTS Experimental

Verification

of

PAGE Buffon's

Bernoulli's Theorem-Appendix:

Needle Problem-Problems for Solution. CHAPTER VII APPROXIMATE EvALUATION OF PROBABILITIES IN BERNOULLIAN CASE Statement

of .Problem-Analytical

ability-Analysis

Leading

to

Representation

Final

Approximate

of

•

.

Required

.

.

.

119

Prob­

Formula-Numerical

Examples-Problems for Solution. CHAPTER VIII FuRTHER CoNSIDERATIONs oN GAMES OF CHANCE .

.

.

.

.

.

.

.

.

. .

. . . 139

Problems Concerning Ruin of Gamblers When Number of Games is Un­ limited-Stakes Equal-Case of Unequal Stakes-Probability of Ruin in Case of a Limited Number of Games-Problems for Solution. CHAPTER IX MATHEMATICAL ExPECTATION. Stochastic

. ,

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. 161

Variables-Definition of Mathematical Expectation-Mathe­

matical Expectation of Sums-Examples-Mathematical Expectation of Products-Dispersion

and

Standard Deviation-Dispersion

of

Sums­

Examples-Problems for Solution. CHAPTER X THE LAw OF LARGE NuMBERs .

182

Tshebysheff's Lemma-Law of Large Numbers-Sufficient Conditions for Its Validity-Some Experimental Results-Case of Infinite Dispersion: Theorem of Markoff-Law of Large Numbers for Identical Variables: Theorem of Khintchine-Problems for Solution. CHAPTER XI APPLICATION OF THE LAW OF LARGE NUMBERS .

.

208

Poisson's Theorem-Divergence Coefficient-Lexis' and Poisson's Cases­ Tschuprow's

Theorem-Examples

of

Dependent

Trials-Experimental

Results for Simple Chains-Problems for Solution. CHAPTER XII PROBABILITIES IN CONTINUUM .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. 235

Notion of Probability for Continuous Variables-Case of One Variable­ Density of Probability-Distribution Function, Moments and Character­ istic Function-Examples-Case of Two or More Variables-Geometrical Problems-Problems for Solution. CHAPTER XIII THE GENERAL CoNCEPT oF DISTRIBUTION .

.

.

.

.

Increasing Functions and Stieltjes' Integrals-Representation of Distribu­ tion Function by Stieltjes' Integral-Moments and Characteristic Func­ tion-Inequalities for Moments-Composition of Distribution Functions­ Examples-Determination of Distribution When Its Characteristic Function Is Known-Composition of Characteristic Functions-Examples-Problems for Solution.

260

CONTENTS

IX

CHAPTER XIV FUNDAMENTAL LIMIT THEOREMS

Introductory

.

.

.

.

.

.

.

PAGE

.

•

.

Remarks-Laplace-Liapounoff's

Lemma-Proof

of

Laplace-Liapounoff's

•

.

.

.

.

.

.

.

.

.

. 283

Theorem-Fundamental

Theorem-Particular

Cases­

Estimation of Error Term-Hypothesis of Elementary Errors-Examples Concerning Dependent Variables-Problems for Solution. CHAPTER XV NORIIIAL DISTRIBUTION IN Two DIMENSIONS. INDEPENDENT VECTORS.

LIMIT THEOREM FOR SUMS OF

ORIGIN OF NORMAL CORRELATION

.

.

.

.

.

.

308

Properties of Two-dimensional Normal Distribution-Limit Theorem for Sums of Independent Vectors-Application to Trials with Three or More Alternatives-Pearson's "x2-Test"-Hypothetical Explanation of Normal Correlation-Problems for Solution. CHAPTER XVI DISTRIBUTION OF CERTAIN FUNCTIONS OF NORMALLY DISTRIBUTED VARIABLES

.

331

Distribution of Sums of Squares-Distribution of Sums of Squares of Deviations from the Mean-" Student's" Distribution-Distribution of the Correlation Coefficient of a Sample-Problems for Solution. APPENDIX I EuLER's SUIIIIIIATION FORMULA-STIRLING's FOR!IIULA-SOME DEFINITE INTEGRALS

.

.

•

.

.

.

.

.

.

.

•

•

•

.

.

.

.

.

.

•

•

.

.

.

.

•

.

.

.

.

.

.

.

.

•

347

.

.

•

356

APPENDIX II METHOD OF Mo!IIENTS AND ITS APPLICATIONS.

Introductory

Remarks-Continued

Series-Continued

Fractions

.

.

.

Fractions

Associated

with

Associated

rb d"' )a z -x

.

with

(x)

Power

Properties of

·

Denominators of Their Convergents-Equivalent Point Distributions­ Tshebysheff's Inequalities-Application to Normal Distribution-Tsheby­ sheff-Markoff's Fundamental Theorem-Application

to Sums of Inde­

pendent Variables. APPENDIX III ON A GAUSSIAN PROBLEM

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

396

Statement of the Probl�m-Analytical Expression for P,.(x) and Its Deriv­ ative-More General Recurrence Relation-Main Inequalities-Solution of the Gaussian Problem. TABLE OF PROBABILITY INTEGRAL.

407

INDEX

409

.

.

.

.

.

.

.

•

•

.

•

•

•

INTRODUCTION TO MATHEMATICAL PROBABILITY INTRODUCTION . Quanto enim minus rationis terminis comprehendi posse videbatur, quae fortuita aunt atque incerta, tanto admira­ bilior ars censebitur, cui ista quoque subjacent.-

CHR. HUYGENS, De ratiociniis in ludo aleae.

1. It is always difficult to describe with adequate conciseness and clarity the object of any particular science; its methods, problems, and results are revealed only gradually.

But if one must define the scope

of the theory of probability the answer may be this: The theory of probability is a branch of applied mathematics dealing with the effects of chance.

Here we encounter the word

11

chance," which is often used in

everyday language but with rather indefinite meaning.

To make clearer

the idea conveyed by this word, we shall try first to clarify the opposite idea expressed in the word "necessity." physical.

Necessity may be logical or

The statement "The sum of the angles in a triangle is equal

to two right angles" is a logical necessity, provided we assume the axioms of Euclidean geometry; for in denying the conclusion of the admitted premises, we violate the logical law of contradiction. The following statements serve to illustrate the idea of physical necessity:

A piece of iron falls down if not supported. Water boils if heated to a sufficiently high temperature. A die thrown on a board never stands on its edge. The logical structure of all these statements is the same: When certain conditions which may be termed occurs of necessity.

11

causes" are fulfilled, a definite effect

But the nature of this kind of necessity is different

from that of logical necessity.

The latter, with our organization of

mind, appears absolute, while physical necessity is only a result of extensive induction.

We have never known an instance in which water,

heated to a high temperature, did not boil; or a piece of iron did not fall down; or a die stood on i ts edge.

For that reason we are led to believe

that in the preceding examples (and in innumerable similar instances) the effect follows from its

11

cause" of necessity. 1

2

INTRODUCTION TO MATHEMATICAL PROBABILITY

Instead of the term "physical necessity" we may introduce the abstract idea of "natural law."

Thus, it is a "natural law" that the

piece of iron left without support will fall down.

Natural laws derived

from extensive experiments or observations may be called "empirical laws" to distinguish them from theoretical laws.

In all exact sciences

which have reached a high degree of development, such as astronomy, physics, and chemistry, scientists endeavor to build up an abstract and simplified image of the infinitely complex physical world-an image which can be described in mathematical terms.

With the help of

hypotheses and some artificial concepts, it becomes possible to deri"e mathematically certain laws which, when applied to the world of reality, represent many natural phenomena with an amazing degree of accuracy. It is true that in the development of the sciences it sometimes becomes necessary to recast the previously accepted image of the physical world, but it is remarkable that the fundamental theoretical laws even then undergo but slight modification in substance or interpretation. The chief endeavor of the exact sciences is the discovery of natural laws, and their formulation is of the greatest importance to the promotion of human knowledge in general and to the extension of our powers over natural phenomena. Are the events caused by natural laws absolutely certain? but for all practical purposes they may be considered as certain.

No, It is

possible that one or another of the natural laws may fail, but such However, granted that the

failure would constitute a real "miracle."

possibility of miracles is consistent with the nature of scientific knowledge, actually this possibility may be disregarded.

2. If the preceding explanations throw a faint light upon the concept of necessity, it now remains to illuminate by comparison some charac­ teristic features inherent in the concept of"chance."

To say that chance

is a denial of necessity is too vague a statement, but examples may help us to understand it better. If a die is thrown upon a board we are certain that one of the six faces will turn up.

But whether a particulm· face will show depends on what

we call chance and cannot be predicted.

Now, in the act of tossing a

die there are some conditions known to us: first, that it is nearly cubic in shape; further, if it is a good die, its material is as nearly as possible homogeneous.

Besides these known conditions, there are other factors

influencing the motion of the die which are completely inaccessible to our knowledge.

First among them are the initial position and the impulse

imparted by the player's hand.

These depend on an "act of will"-an

agent which may act without any recognizable motivation-and therefore they are outside the domain of rational knowledge.

Second, supposing

the initial conditions known, the complexity of the resulting motion defies any possibility of foreseeing the final result.

SEc.

3)

3

INTRODUCTION

Another example: If equal numbers of white and black balls, which do not differ in any respect except in color, are concealed in an urn, and we draw one of them blindly, it is certain that its color will be either white or black, but whether it will be black or white we cannot predict: that depends on chance.

In this example we again have a set of known

conditions: namely, that balls in equal numbers are white and black, and that they are not distinguishable except in color.

But the final result

depends on other c'onditions completely outside our knowledge.

First,

we know nothing about the respective positions of the white and black balls; second, the choice of one or the other depends on an act of will. It is an observed fact that the numbers of marriages, divorces, births, deaths, suicides, etc., per 1,000 of population, in a country with nearly settled living conditions and during not too long a period of time, do not remain constant, but oscillate within comparatively narrow limits.

For

a given year it is impossible to predict what will be their numbers: that depends on chance.

For, besides some known conditions, such as the

level of prosperity, sanitation, and many other things, there are unnum­ bered factors completely outside our knowledge. Many other examples of a similar kind can be cited to illustrate the notion of chance.

They all possess a common logical structure which

can be described as follows: an event A may materialize under certain known or "fixed" conditions, but not necessarily; for under the same fixed conditions other events B, 0, D, . . . are also possible.

The mate­

rialization of A depends also upon other factors completely outside our control and knowledge.

Consequently, whether A will materialize or

not under such circumstances cannot be foreseen; the materialization of

A is due to chance, or, to express it concisely, A is a contingent event. 3. The idea of necessity is closely related to that of certainty.

Thus

it is "certain" that everybody will die in the due course of time.

In

the same way the idea of chance is related to that of probability or likeli­

hood.

In everyday language, the words "probability" and "probable"

are used with different shades of meaning.

By saying, "Probably it will

rain tomorrow," we mean that there are more signs indicating rainy weather than fair for tomorrow.

On the other hand, in the statement,

"There is little probability in the story he told us," the word "proba­ bility" is used in the sense of credibility.

But henceforth we shall use

the word as equivalent to the degree of credence which we may place in the possibility that some contingent event may materialize.

The

"degree of credence" implies an almost instinctive desire to compare probabilities of different events or facts.

That such comparison is

possible one can gather from the following examples:

I live on the second floor and can reach the ground either by using the stairway or by jumping from the window. injured, though not necessarily.

Either way I might be

How do the probabilities of being

4

INTRODUCTION TO MATHEMATICAL PROBABILITY

Everyone, no doubt, will say that

injured compare in the two cases?

the probability of being injured by jumping from the window is "greater" than the probability of being injured while walking down the stairway. Such universal agreement might be due either to personal experience or merely to hearsay about similar experiences of other persons.

An urn contains an equal number of white and black balls that are similar in all respects except color. black or white.

One ball is drawn.

It may be either

How do the probabilities of these two cases compare?

One almost instinctively answers: "They are equal." Now, if there are 10 white balls and 1 black ball in the urn, what about the probabilities of drawing a white or a black ball?

Again one

would say without hesitation that the probability of drawing a white ball is greater than that of drawing a black ball. Thus, probability appears to be something which admits of compari­ sons in magnitude, but so far only in the same way as in the intensity of pain produced by piercing the skin with needles. But it is a noteworthy observation that men instinctively try to characterize probabilities numerically in a naive and unscientific manner. We read regularly in the sporting sections of newspapers, predictions that in a coming race a certain horse has two chances against one to win over another horse, or that the chances of two football teams are as

10 to 7, etc.

No doubt experts do know much about the respective

horses and their riders, or the comparative strengths of two competing football teams, but their numerical estimates of chances have no other merit than to show the human tendency to assign numerical values to probabilities which most likely cannot be expressed in numbers. It is possible that a man endowed with good common sense and ripe judgment can weigh all available evidence in order to compare the probabilities of the various possible outcomes and to direct his actions accordingly so as to secure profit for himself or for society.

But precise

conclusions can never be attained unless we find a satisfactory way to represent or to measure probabilities by numbers, at least in some cases.

4. As in other fields of knowledge, in attempting to measure proba­ bilities by numbers, we encounter difficulties that cannot be avoided except

by

making

certain

ideal

assumptions

and

agreements.

In

geometry (we speak of applied and not of abstract geometry), before explaining how lengths of rectilinear segments can be measured, we must first agree on criteria of equality of two segments.

Similarly, in dealing

with probability, the �rst step is to answer the question: When may two contingent events be considered as equally probable or, to use a more common expression, equally likely?

From the statements of Jacob

Bernoulli, one of the founders of the mathematical theory of probability, one can infer the following criterion of equal probability:

5

INTRODUCTION

SEc. 4)

Two contingent events m·e considered as equally p1·obable if, after taking into consideration all relevant- evidence, one of them cannot be expected in preference to the other.

Certainly there is some obscurity in this criterion, but it is hardly possible to substitute any better one.

To be perfectly honest, we must

admit that there is an unavoidable obscurity in the principles of all the sciences in which mathematical analysis is applied to reality. The application of Bernoulli's criterion to particular cases is beset with difficulties and requires good common sense and keen judgment. There is much truth in Laplace's statement: "La theorie des probabilites n'est au fond que le bon sens reduit au calcul." To elucidate the nature of these difficulties, let us consider an urn filled with white and black balls, but in unknown proportion.

The only

evidence we have, namely, that there are both white and black balls in the urn, in this case appears insufficient for any conclusion about the respective probabilities of drawing a white or a black ball.

We instinc­

tively think of the numbers of the two kinds of balls, and, being in ignorance on this point, we are inclined to suspend judgment.

But if we

know that white and black balls are equal in number and distributed without any sort of regularity, this knowledge appears sufficient to assume the equality of the probabilities of drawing a white or a black ball.

It is possible that, perhaps unconsciously, we are influenced by the

commonly known fact that if we repeatedly draw a ball out of the urn many times, returning the ball each time before drawing again, the white and the black balls appear in nearly equal numbers. If an urn contains a certain number of identical balls distinguished from one another by some characteristic signs, for example, by the numbers 1, 2, 3, . . . , the knowledge that the balls are identical and are distributed without regularity suffices in this case to cause us to conclude that the probabilities for drawing any of the balls should be considered as equal.

Again, in so readily assuming this conclusion we

may be influenced by the fact empirically observed (by ourselves or by others) that in a long series of drawings, with balls being restored to the urn after each withdrawal, the balls appear with nearly the same frequency. An ordinary die is tossed. Should we consider the possible numbers of points 1, 2, 3, 4, 5, 6 as equally probable? To pronounce any judg­ ment, we must know something about the die.

If it is known that the

die has a regular cubic shape and that its material is homogeneous, we readily agree on the equal probabilities of all the numbers of points 1, 2, 3, 4, 5, 6. And this a priori conclusion, based on Bernoulli's cri­ terion, agrees with the observed fact that each number of points does appear nearly an equal number of times in a long series of throws, if the

6

INTRODUCTION TO MATHEMATICAL PROBABILITY

die is a good one.

However, if we only know that the die has a regular

shape, but not whether or not it is loaded, it is only sensible to suspend judgment. These examples show that before trying to apply Bernoulli's criterion, we must have at our disposal some evidence the amount of which cannot be determined by any general rules. It may be also that the reason a priori must be supplemented by some empirical evidence.

In some

cases, lacking sufficient grounds to assert equal probabilities for two events, we may assume them as a hypothesis, to be kept until for some reason we are forced to abandon it.

6. Besides the ticklish question: When are we entitled to consider events as equally probable? there is another fundamental assumption required to make possible the measurement of probabilities by numbers. Events a 1, a2, . . . an form an exhaustive set of :possibilities under certain fixed conditions S, if at least one of them must necessarily mate­ rialize. They are mutually exclusive if any two of them cannot material­ ize simultaneously. The fundamental assumption referred to consists in the possibility of subdividing results consistent with the conditions S into a number of exhaustive, mutually exclusive, and equally likely events, or cases (as they are commonly called):

This being granted, the probability of any one of these cases is assumed to be 1 / n. An event A may materialize in several mutually exclusive particular forms:

a,

events

a,

"X; that is, if A occurs, then one and only one of the "X occurs also, and conversely the occurrence of one of these events necessitates the occurrence of A. Thus, if A consists in {3, {3, . •

.

•

•

•

drawing an ace from a deck of cards, A may materialize in four mutually exclusive forms: as an ace of hearts, diamonds, clubs, or spades. Let an event A be represented by its particular forms a1, a2, . . . am, which together with other events am+1, am+2, . . . an constitute an exhaustive set of mutually exclusive and equally likely cases consistent with the conditions S.

Events a1, a2, .. . am are called" cases favorable toA."

Definition of Mathematical Probability.

If, consistent with conditions

S, there m·e n exhaustive, mutually exclusive, and equally likely cases, and m of them are favorable to an event A, then the mathematical probability of

A is defined as the mtio m/n.

. In drawing a card from a full deck there are 52 and no more mutually

exclusive and equally likely cases; 4 of them are favorable for drawing an ace; hence the probability of drawing an ace is %2

=

Us.

From an urn containing 10 white, 20 black, and 5 red balls, one ball is drawn.

Here, distinguishing individual balls, we have 35 equally likely

7

INTRODUCTION

SEC. 6)

cases.

Among them there are 10, 20, and 5 cases, favorable respectively

to a white, a black, or a red ball.

Hence the probabilities of drawing a

white, a black, or a red ball are, respectively, 3-f, '?7, and�. In the first example, instead of 52 cases, we may consider only 13 cases according to the denominations of the cards.

These cases being

regarded as equally likely, there is only one of them favorable to an ace.

The probability of drawing an ace is 7f'a.

This observation makes

it clear that the subdivision of all possible results into equally likely cases can be done in various ways.

To avoid contradictory estimations

of the same probability we must always observe the following rules: Two events are equally likely if each of them can be represented by equal numbers of equally likely forms. Two events are not equally likely if they are represented by unequal numbers of equally likely forms. Thus, if two equally likely events are each represented by different numbers of their respective forms, then the latter cannot be considered as equally likely. Each card is characterized by its denomination and the suit to which it belongs.

Noting denominations, we distinguish 13 cases, but each

of these is represented by 4 new cases according to the suit to which the card belongs. Altogether we have, then, 52 cases recognized as equally likely; hence, the above-mentioned 13 cases should be considered as equally likely. In connection with

the

definition of

mathematical

probability,

mention should be made of an important principle not always explicitly stated.

If

are all mutually exclusive and equally likely cases consistent with certain conditions, and the indication of the occurrence of an event B makes cases b1, b2, . . . bp impossible, cases a1, a2, considered as equally likely.

.

.

urn with six tickets bearing numbers 1, 2, . . . 6. drawn in succession.

.

am

still should be

To illustrate this principle, consider an Two tickets are

If nothing is known about the number of the first

ticket, we still have six possibilities for the number of the second ticket, which we agree to consider as equally likely.

But as soon as the number

of the first ticket becomes known, then there are only five cases left concerning the number of the second ticket.

According to the above

principle we must consider these five cases as equally likely. G. Probability as defined above is represented by a number contained

between 0 and 1.

In the extreme case in which the probability is 0, it

indicates the impossibility of an event.

On the contrary, in the other

extreme case in which the probability is 1, the event is certain.

When

8

INTRODUCTION TO MATHEMATICAL PROBABILITY

the probability is expressed by a number very near to

1,

it means that

the overwhelming majority of cases are favorable to the event. contrary, a probability near to

0

On the

shows that the proportion of favorable

cases is small. From our experience we know that events with a small probabil­ ity seldom happen.

1/1,000,000,

For instance, if the probability of an event is

the situation may be likened to the drawing of a white ball

from an urn containing

999,999

black balls and a single white one.

This white ball is practically lost among the majority of black balls, and for all practical purposes we may consider its extraction impossible. Similarly, the probability

999,999/1,000,000

may be considered, from a

practical standpoint, as an indication of certainty.

What limit for

smallness of probability is to be set as an indication of practical impos­ sibility?

Evidently there is no general answer to this question.

Every­

thing depends on the risk we can face if, contrary to expectation, an event with a small probability should occur.

Hence, the main problem

of the theory of probability consists in finding cases in which the proba­ bility is very small or very near to bility is very near to

1,"

1.

Instead of saying, "The proba­

we shall say, "great probability," although,

of course, the probability can never exceed

1.

7. The definition of mathematical probability in Sec. 5 is essentially

the classical definition proposed by Jacob Bernoulli and adopted by Laplace and almost all the important contributors to the theory of probability.

But, since the middle of the nineteenth century (Cournot,

John Stuart Mill, Venn), and especially in our days, the classical definition has been severely criticized.

Several attempts have been made to rear

up the edifice of the mathematical theory of probability on quite a different definition of mathematical probability.

It does not enter into

our plan to criticize these new definitions, but, in the opinion of the author, many of them are self-contradictory.

Modern attempts to build

up the theory of probability as an axiomatic science may be interesting in themselves as mental exercises; but from the standpoint of applica­ tions the purely axiomatic science of probability would have no more value than, for example, would the axiomatic theory of elasticity. The most serious objection to the classical definition is that it can be used only in very simple and comparatively unimportant cases like games of chance. not a new one.

This objection, stressed by von Mises, is in reality

It is one of the objections Leibnitz made against Jacob

Bernoulli's views concerning the possibility of applications of the theory of probability to various important fields of human endeavor and not merely to games of chance. It is certainly true that the classical definition cannot be directly applied in many important cases.

But is it the fault of the definition

9

IN7'RODUCTION

SEC. 8)

or is it rather due to our ignorance of the innermost mechanisms which, apart from chance, contribute to the materialization or nonmaterializa­ tion of contingent events?

It seems that this is what Jacob Bernoulli

meant in his reply to Leibnitz: Objiciunt prima, aliam esse rationem calculorum, aliam morborum aut muta­ tionum aeris; illorum numerum determinatum esse, horum indeterminatum et

vagum.

Ad quod respondeo, utrumque respectu cognitionis nostrae aequi poni

incertum et indeterminatum; sed quicquam in se et sua natura tale esse, non magis a nobis posse concipi, quam concipi potest, idem simul ab Auctore naturae creatum esse et non creatum: quaecumque enim Deus fecit, eo ipso dum fecit, etiam determinavit. 1

8. A brilliant example of how the profound study of a subject finally makes it possible to apply the classical definition of mathematical probability is afforded in the fundamental laws of genetics (a science of comparatively recent origin, whose importance no one can deny), dis­ covered by the Augustinian monk, Gregor Mendel (1822--1884). During 2 eight years Mendel conducted experimental work in crossing different varieties of the common pea plant with the purpose of investigating how pairs of contrasting characters were inherited.

For the pea plant there

are several pairs of such contrasting characters: round or wrinkled seeds, tallness or dwarfness, yellow or green pod color, etc. Let·us concentrate our attention on a definite pair of contrasting characters, yellow or green pod color.

Peas with green pod color always breed true.

Also some

peas with yellow color always breed true, while still others produce both varieties.

True breeding pea plants constitute two pure races: A with

yellow pod color and B with green pod color, while plants with yellow pods not breeding true constitute a hybrid race, C.

Crossing plants of the race A with those of the race B and planting the seeds, Mendel

obtained a first generation F 1 of hybrids.

Letting plants of the first

generation self-fertilize and again planting their seeds to produce the second generation F2, Mendel found that in this generation there were

428 yellow pod plants and 152 green pod plants in the ratio 2.82: I. In regard to other contrasting characters the ratio of approximately 3: 1 was

observed in all cases.

Mendel's results.

Later experimental work only confirmed

Thus, combined experiments of Correns, Tschermak,

and others gave among plants and

195,477 individuals of F2, 146,802 yellow pod 48,675 green pod plants, in the ratio 3.016: 1.

1 To understand the beginning of this statement see the translation from "Ars conjectandi" in Chap. VI, p.

105.

1865, but passed completely unnoticed until the same facts were rediscovered by DeVries, Correns, and Tschermak. Modern genetics dates from about this time. 2

Mendel's results were published in

in about

1900

INTRODUCTION TO MATHEMATICAL PROBABILITY

10

Mendel not only discovered such remarkable regularities, but also suggested a rational explanation of the observed ratio 3: 1, which with some modifications is accepted even today.

Bodies of plants and

animals are built up of enormous numbers of cells, among which the reproductive cells, or gametes, differ from the remaining "somatic" cells in some important qualities. possess a definite structure.

Cells are not homogeneous, but

In somatic cells there are found bodies,

called chromosomes, whose number is even and the same for the same species.

Exactly half of this number of chromosomes is found in repro­

ductive cells.

Chromosomes are supposed to be seats of hypothetical

"genes," which are considered as bearers of various heritable characters. A chromosome of one pure race A bearing a character

a

differs from the

homologous chromosome of another pure race B bearing a contrasting character b in that they contain genes of different kinds. a

Since characters

and b are borne by definite chromosomes, the situation in regard to the

two characters

a

and b is exactly the same as if gametes of both races

contained just one chromosome. 0 and ®.

Let us represent them symbolically by

In the act of fertilization a pair of paternal and maternal

gametes conjugate and form a zygote, which by division and growth produces all cells of the filial generation.

Certain of these cells become

the germ cells and are set apart for the formation, by a complicated process, of gametes, one half of which contain the chromosome of the paternal type and the other half that of the maternal type. According to this theory, in crossing two individuals belonging to races A and B, zygotes of the first generation F1 will be of the type 0 -®, and will produce gametes, in equal numbers, of the types 0, ®.

Now if two individuals of F1 (hybrids) are crossed (or one individual self-fertilized as in the cases of some plants), one paternal gamete con­ jugates with one maternal, and for the resulting zygote there are four possibilities: 0- 0

0- ®

®- 0

®-®

These possibilities may be considered as equally probable, whence the probabilities for an individual of the generation F2 to belong respec­ tively to the races A, B, Care.%:,.%:,�.

Similarly, one easily finds that

in crossing an individual of the race A with one of the hybrid race C, the probabilities of the offspring belonging to A or Care both equal to�. It is easy now to offer a rational explanation of the Mendelian ratio 3: 1.

In the case of pea plants, individuals of the race A and hybrids

are not distinguishable in regard to the color of their pods.

Hence the

probability of the offspring of a hybrid plant having yellow pods is %;, while for the offspring to have green pods the probability is �. When the generation F 2 consists of a great many individuals, the theory

SEC.

11

INTRODUCTION

10)

of probability shows that the ratio of the number of yellow pod plants to the number of green pod plants is not likely to differ much from the ratio

3: 1.

In crossing plants of the race A: with hybrids, the offspring, if

numerous, will contain plants of race A or C, respectively, in a proportion which is not likely to differ much from 1: 1.

And this conclusion was

experimentally verified by Mendel himself.

9. If in the case of the Mendelian laws the profound study of the mechanism of heredity together with hypothetical assumptions of the kind used in physics, chemistry, etc., paved the way for a rational explanation of observed phenomena on the basis of the theory of proba­ bility, in many other important instances we are still unable to reach the same degree of scientific understanding.

Stability of statistical ratios

observed in many cases suggests the idea that they should be explained on the basis of probability.

For instance, it has been observed that

the ratio of human male and female births is nearly 51: 50 for large samples, and this is largely independent of climatic conditions, racial differences, living conditions in different countries, etc.

Although the

factors determining sex are known, yet some complications not suffi­ ciently cleared up prevent estimation of probabilities of male and female births. In all instances of the pronounced stability of statistical ratios we may believe that some day a way will be found to estimate probabilities in such cases.

Therefore many applications of the theory of probability

to important problems of other sciences are based on belief in the existence of the probabilities with which we are concerned.

In other cases in

which the theory of probability is used, we may have grave doubts as to whether this science is applied legitimately.

The fact that many

applications of probability are based on belief or faith should not dis­ courage us; for it is better to do something, though it may be not quite reliable, than nothing.

Only we must not be overconfident about the

conclusions reached under such circumstances. After all, is not faith at the bottom of all scientific knowledge? Physicists speak of electrons, which never have been seen and are known only through their visible manifestations.

Electrons are postulated

just to coordinate into a coherent whole a large variety of observed phenomena.

Is not this faith?

It must be, for according to Paul

(Hebrews, 11: 1), "Faith is the substance of things hoped for, the evidence of things not seen."

10. In concluding this introduction it remains to give a short account Although ancient philoso­

of the history of the theory of probability.

phers discussed at length the necessity and contingency of things, it seems that mathematical treatment of probability was not known to the ancients.

Apart from casual remarks of Galileo concerning the correct

12

INTRODUCTION TO MATHEMATICAL PROBABILITY

evaluation of chances in a game of dice, we find the true origin of the science of probability in the correspondence between two great men of the seventeenth century, Pascal

(1623-1662)

and Fermat

(1601-1665).

A French nobleman, Chevalier de Mere, a man of ability and great experience in gambling, asked Pascal to explain some seeming contradic­ tions between his theoretical reasoning and the observations gathered from gambling.

Pascal solved this difficulty and attacked another

problem proposed to him by de Mere.

On hearing from Pascal about

these problems, Fermat became interested in them, and in their private correspondence these two great men laid the first foundations of the science of probability.

Bertrand's statement, "Les grands noms de

Pascal et de Fermat decorent le berceau de cette science" cannot be disputed. Huygens

a great Dutch scientist, became acquainted

(1629-1695),

with the contents of this correspondence and, spurred on by the new ideas, published in

1654

a first book on probability, "De ratiociniis in

ludo aleae," in which many interesting and rather difficult problems on probabilities in games of chance were solved.

To him we owe the

concept of "mathematical expectation" so important in the modern theory of probability. Jacob Bernoulli

(1654-1705)

meditated on the subject of probability

for about twenty years and prepared his great book, "Ars conjectandi," which, however, was not published until eight years after his death in

1713,

by his nephew, Nicholas Bernoulli.

Bernoulli envisaged the

subject from the most general point of view, and clearly foresaw a whole field of applications of the theory of probability outside of the narrow circle of problems relating to games of chance.

To him is due the

discovery of one of the most important theorems known as "Bernoulli's theorem." The next great successor to Bernoulli is Abraham de Moivre

1754),

Chances," was first published in in

{1667-

whose most important work on probability, "The Doctrine of

1756.

1718

and twice reprinted in

1738

and

De Moivre does not contribute much to the principles, but this

work is justly renowned for new and powerful methods for the solution of more difficult problems.

Many important results, ordinarily attrib­

uted to Laplace and Poisson, can be found in de Moivre's book. Laplace

(1749-1827),

whose contributions to celestial mechanics

assured him everlasting fame in the history of astronomy, was very much interested in the theory of probability from the very beginning of his scientific career.

After writing several important memoirs on the

subject, he finally published, in

1812,

his great work "Theorie analytique

des probabilites," accompanied by a no less known popular exposition, "Essai philosophique sur les probabilites," destined for the general educated public.

Laplace's work, on account of the multitude of new

13

INTRODUCTION

SEC. 10)

ideas, new analytic methods, and new results, in all fairness should be regarded as one of the most outstanding contributions to mathematical literature.

It exercised a great influence on later writers on probability

in Europe, whose work chiefly consisted in elucidation and development of topics contained in Laplace'!:! book. Thus in European countries further development of the theory of probability was somewhat retarded.

But the subject took on important

developments in the works of Russian mathematicians: Tshebysheff (1821-1894) and his former students, A. Markoff (1856-1922) and A. Liapounoff (1858-1918).

Castelnuovo in his fine book "Calcolo delle

probabilita" rightly regards the contributions to the theory of probability due to Russian mathematicians as the most important since the time of Laplace. At the present time interest in the theory of probability is revived everywhere, but again the most outstanding recent contributions have been made in Russia, chiefly by three prominent mathematicians: S. Bernstein, A. Khintchine, and A. Kolmogoroff. In closing this introduction it seems proper to quote the closing words of the "Essai philosophique sur les probabilites": On voit par cet Essai, que la theorie des probabilites n' est au fond, que le bon sens reduit au calcul: elle fait apprecier avec exactitude, ce que les esprits justes sentent par une sorte d'instinct, sans qu'ils puissent souvent s'en rendre compte. Elle ne laisse rien d'arbitraire dans le choix des opinions et des partis a prendre, toutes les fois que l'on peut, a son moyen, determiner le choix le plus avantageux. Par la, elle devient le supplement le plus heureux, a !'ignorance et a la faiblesse de l'esprit humain.

Si l'on considere les methodes analytiques auxquelles cette

theorie a donne naissance, la verite des principes qui lui servent de base, la logique fine et delicate qu' exige leur emploi dans la solution des problemes, les etablissements d'utilite publique qui s'appuient sur elle, et !'extension qu'elle a rec;me et qu'elle peut re�evoir encore, par son application aux questions les plus importantes de la Philosophie naturelle et des sciences morales; si l'on observe ensuite, que dans les choses m�mes qui ne peuvent �tre soumise au calcul, elle donne les aperQUS les plus s11rs qui puissent nous guider dans nos jugements, et qu'elle apprend a se garantir des illusions qui souvent nous egarent; on verra qu'il n'est point de science plus digne de nos meditations. References JACOB BERNOULLI: "Ars conjectandi," 1713. LAPLACE: "Essai philosophique sur les probabilit6s," Oeuvres VII, pp. v-cliii. BERTRAND: "Calcul des probabilites," pp. vi-1, Paris, 1889. POINCARE: "Calcul des probabilit6s," 2d ed., pp. 1-23, Paris, 1912. KRIEs: "Die Principien der Wahrscheinlichkeitsrechnung.

Eine logische Unter-

.suchung," Freiburg, 1896. J. M. KEYNEs:"A Treatise on Probability," London, 1921. VON MISEs: "Wahrscheinlichkeitsrechnung," Leipzig, 1931.

A. KoLMOGOROFF: "Grundbegriffe der Wahrscheinlichkeitsrechnung," Berlin, 1933.

CHAPTER I

COMPUTATION OF PROBABILITIES BY DIRECT ENUMERATION OF CASES 1. The probability of an event can be found by direct application of the definition when it is possible to make a complete enumeration of all equally likely cases.. as well as of those favorable to that event.

Here

we shall consider a few problems, beginning with the simplest, to illustrate this direct method of evaluating probabilities.

Problem 1.

Two dice are thrown.

What is the probability of

obtaining a total of 7 or 8 points?

Solution.

Suppose we distinguish the dice by the numbers 1 and 2.

There are 6 possible cases as to the number of points on the first die; and each of these cases can be accompanied by any of the 6 possible numbers of points on the second die. gether 6 X 6

=

36 different cases.

Hence, we can distinguish alto­

Provided the dice are ideally regular

in shape and perfectly homogeneous, we have good reason to consider these 36 cases as equally likely, and we shall so consider them. Next, let us find out how many cases are favorable to the total of

7 points.

This may happP.n only in the following ways: First Die

Second Die

1

6

2

5

3

4

4

3

5

2

6•

1

Likewise, for 8 points: First Die

Second Die

2

6

3

5

4

4

5

3

6

2

That is, out of the total number of 36 cases there are 6 cases favorable to 7 points and 5 cases favorable to 8 points; hence, the probability of obtaining 7 points is

2. Problem 2.

%a

and the probability of obtaining 8 points is

A coin is tossed three times in succession.

is the probability of obtaining 2 heads? obtaining tails at least once? 14

%a.

What

What is the probability of

15

COMPUTATION OF PROBABILITIES

SEC. 3)

Solution. In the first throw there are two possible cases: heads or tails. And if the coin is unbiased (which we assume is true) these two cases may be considered as equally likely. In two throws there are 2 X 2 4 cases; namely, both of the two possible cases in the first toss =

can combine with both of the possible cases in the second. in three throws the number of cases will be 2 X 2 X 2

=

Similarly, To find

8.

the number of cases favorable to obtaining 2 heads, we must consider that this can happen only in three ways: Heads Heads Tails

Heads Tails Heads

Tails Heads Heads

The number of favorable cases being 3, the probability of obtaining two heads is %. To answer the second part of the question, we observe that there is only one case when tails does not turn up. Therefore, the number of 1 7, so that cases favorable to obtaining tails at least once is 8 -

the required probability is

3. Problem 3. cards.

=

:%.

Two cards are drawn from a deck of well-shuffled

What is the probability that both the extracted cards are

aces? Solution.

Since there are 52 cards in the deck, there are 52 ways

of extracting the first card.

Mter the first card has been withdrawn,

the second extracted card may be one of the remaining 51 cards. fore, the total number of ways to draw two cards is 52 X51.

There­ All these

cases may be considered as equally likely. To find the number of cases favorable to drawing aces, we observe that there are 4 aces; therefore, there are 4 ways to get the first ace. Mter it has been extracted, there are 3 ways to get a second ace.

Hence,

the total number of ways to draw 2 aces, is 4 X 3, and the required probability is:

Problem 4.

4X3

1

52 X51

13 X 17

1 =

22(

Two cards are drawn from a full pack, the first card

being returned to the pack before the second is taken.

What is the

probability that both the extracted cards belong to a specified suit?

Solution.

There are 52 ways of getting the first card.

For the

second drawing, there are also 52 ways, because by returning the first extracted card to the pack, the original number was restored.

Under

such circumstances, the total number of ways to extract two cards is 52 X 52.

Now, because there are 13 cards in a suit, the number of

cases favorable to obtaining two cards of a specified suit is 13 X13.

16

INTRODUCTION TO MATHEMATICAL PROBABILITY

[CHAP. I

Therefore, the required probability is given by: 1 X 1

13 X 13 52 X 52

4 X 4

=

1 . 16

An urn contains 3 white and 5 black balls.

4. Problem 6. ball is drawn.

=

One

What is the probability that it is black? The total number of balls is 8.

Solution.

may imagine that they are numbered.

To distinguish them, we

As to the number on the ball

drawn, there are 8 possible cases that may reasonably be considered as equally likely.

Obviously, there are 5 cases favorable to the black color

of the ball drawn.

Therefore, the required probability is%.

By a slight modification of the last problem, we come to the following interesting situation: The contents of the urn are the same as in the foregoing

Problem 6. problem.

But this time we suppose that one ball is drawn, and, its color

unnoted, laid aside.

Then another ball is drawn, and we are required to

find the probability that it is black or white. Suppose again that the balls are numbered, so that the

. Solution.

white balls bear numbers 1, 2, and 3; and the black balls bear numbers 4, 5, 6, 7, 8.

Obviously, there are 8 ways to get the first ball, and what­

ever it is, there remain only 7 ways to get the second ball. number of equally likely cases is 8 X 7

=

The total

56.

It is a little more difficult to find the number of cases favorable to extracting a white or black ball in the second drawing. interested in the white color of the second ball.

Suppose we are

If the first ball drawn is

a white one, it may bear one of the numbers 1 to 3.

Whatever this

number is, the second ball, if it is white, can bear only the two remaining numbers.

Therefore, under the assumption that the first ball is a white

one, the number of favorable cases is 3 X 2

=

6.

Again, supposing that

the first ball drawn is black, we have 5 possibilities as to its number, and, corresponding to any one of these possibilities, there are 3 possibilities as to the number of the white ball to be taken in the second drawing, so that the number of favorable cases now is 5 X 3 of all favorable cases is 6 + 15 the

white

ball

is

2�6

=

%.

=

21.

=

15.

The number

The required probability for

In the same way,

that the probability for the black ball is

%.

we should find

It is remarkable that

these two probabilities are the same as if only a single ball had been drawn. The situation is quite different if we know the color of the first ball. Suppose, for instance, that it is white. likely cases will then be 3 X 7

=

to getting another white ball is 3 X 2 this case is 3"7.

The total number of equally

21; and the number of cases favorable =

6, so that the probability in

17

COMPUTATION OF PROBABILITIES

Smc. 6]

This last example shows clearly how much probability depends upon a given or known set of conditions.

5. Problem 7. drawers.

Three boxes, identical in appearance, each have two

The first box contains a gold coin in each drawer; the second

contains a silver coin in each drawer; but the third contains a gold coin in one drawer and a silver coin in the other. dom.

(a) A box is chosen at ran­

What is the probability that it contains coins of different metals?

(b) A box is chosen, one of its drawers opened, and a gold coin found. What is the probability that the other drawer contains a silver coin?

Solution.

(a) Since nothing outwardly distinguishes one box from

the other, we may recognize three equally likely cases, and among them is only one case of a box with coins of different metals. Therefore, we estimate the required probability as�. (b) As to the second question, one is tempted to reason as follows: The fact that a gold coin was found in one drawer leaves only two possibilities as to the content of the other drawer; namely, that the coin in it is either gold or silver.

Hence, the probability of a silver coin in

the second drawer seems to be Y2.

But this reasoning is fallacious.

It is true that, when the gold coin is found in one drawer, there are only two possibilities left as to the content of the other drawer; but these possibilities cannot be considered as equally likely.

To see this point

clearly, let us distinguish the drawers of the first box by the numbers 1 and 2; those of the second box by the numbers 3 and 4; finally, in the third box, 5 will distinguish the drawer containing the silver coin, while

6 will represent the drawer with the gold coin. Instead of three equally likely cases: box 1,

box 2,

box 3

we now have six cases: drawers 1, 2;

drawers 3, 4;

drawers 5, 6,

which, with reference to the fundamental assumptions, must be con­ sidered as equally likely.

If nothing were known about the contents

of the drawer which has been opened, the number of this drawer might be either 1, 2, 3, 4, 5, or 6. But as soon as the gold coin is discovered in it, cases 3, 4, and 5 become impossible, and there remain three equally likely assumptions as to the number of the opened drawer: it may be either 1 or

2 or 6.

That leaves three cases, and in only one of them, namely, in

case 6, will the other drawer contain a silver coin.

Thus the answer

to the second question (b) is�.

6. In the preceding problems the enumeration of cases did not present any difficulty.

We are now going to discuss a few problems in

which this enumeration is not so obvious but can be greatly simplified

18

INTRODUCTION TO MATHEMATICAL PROBABILITY

[CHAP. I

by the use of well-known formulas for the number of permutations, arrangements, and combinations. Let m distinct objects be represented by the letters a, b, c, . . . l. Using all these objects, we can place them in different orders and form "permutations." For instance, if there are only three letters, a, b, and c, all the possible permutations are: abc, acb, bac, bca, cab, cba,-6 different permutations out of 3 letters. In general, the number of permutations Pm of m objects is expressed by

Pm = 1

·

2 3 ·

·

·

m= m!

·

If n objects are taken out of the total number of m objects to form groups, attention being paid to the order of objects in each group, then these groups are called "arrangements." For instance, by taking two letters out of the four letters a, b, c, d, we can form the following 12 arrangements: ab ae ad

ba be bd

da db de

ca

eb cd

Denoting by the symbol A;:. the number of arrangements of m objects taken nat a time, the following formula holds: A;:.

=

m(m- 1)(m- 2)

·

·

(m - n + 1).

·

Again, if we form groups of n objects taken out of the total number of m objects, this time paying no attention to the order of objects in the group,

we

form

"combinations."

For instance,

following

are

the

different combinations out of 5 objects taken 3 at a time: abc ade

abd bed

abe bee

acd bde

ace cde

In general, the number of combinations out of m objects taken at a time, which is usually denoted by the symbol C;:., is given by

(Jn m

m(m- 1)(m- 2) · (m- n + 1) . 1·2·3 .. ·n ·

=

n,

·

It is useful to recall that the same expression may also be exhibited as follows:

(Jn

'"

=

ml , nl(m- n)l

whence, by substituting m- n instead of n, the useful formula C;:. can be derived.

=

0;::-"

SEc.

COMPUTATION OF PROBABILITIES

71

19

7. After these preliminary remarks, we can turn to the problems in which the foregoing formulas will often be used.

a

An urn contains

Problem 8.

a+ fJ

white balls and

b

black balls.

a white and fJ black balls.

them there will be exactly

If we do not distinguish the order in which the balls come

Solution.

out of the urn, the total number of ways to get

a + b

total number

a+ fJ

balls out of the

C:tf

balls is obviously expressed by

and this is

the number of all possible and equally likely cases in this problem. number of ways to draw

c:;

balls in the urn is of drawing

fJ

a white balls

and similarly

out of the total number

CC

a white

b

of black balls.

Now

balls combines with every possible group of

a white

black balls to form the total of

balls and

fJ

C;: CC.

fJ

black balls, so that

a white

the number of ways to form all the groups containing black balls is

The

a of white

represents the number of ways

black balls out of the total number

every group of

fJ

If

balls are drawn from this urn, find the probability that among

balls and

This is also the number of favorable cases;

·

hence, the required probability is

c:. ce C::#'

_

p-

or, in a more explicit form,

(1) P - 1 2 _

1 2

(a + fJ) 1 2 fJ a(a - 1) (a - a + 1) b(b - 1) (a + b (a + b)(a + b - 1) ·

·

·

·

·

·

·

a

·

·

·

·

·

·

·

·

·

An urn contains

Problem 9. and

m tickets

·

·

·

·

·

a

·

·

(b - fJ + 1) 1)

- fJ +

tickets bearing numbers from

n

are drawn at a time.

1

to

n,

What is the probability that i of

the tickets removed have numbers previously specified?

Solution. one.

This problem does not essentially differ from the preceding

In fact, i tickets with preassigned numbers can be likened to i

white balls, while the remaining tickets correspond to the black balls. The required probability, therefore, can be obtained from the expression (1) by taking

a

=

i,

b

n

=

- i,

a

=

i,

fJ

=

m - i and,

all simplifications

performed, will be given by (2)

p=

m(m - 1) n ( n - 1)

·

·

·

·

·

·

(m - i + 1) (n - i + 1)

·

The conditions of this problem were realized in the French lottery, which was operated by the French royal government for a long time but ·

discontinued soon

after

the

Revolution

of

1789.

Similar

lotteries

continued to exist in other European countries throughout the nineteenth century.

In the French lottery, tickets bearing numbers from

1

to

90

20

INTRODUCTION TO MATHEMATICAL PROBABILITY

[CHAP.

I

were sold to the people, and at regular intervals drawings for winning numbers were held in different French cities. numbers were drawn. he received

At each drawing,

5

If a holder of tickets won on a single number,

15 times its cost to him.

If he won on two, three, four, or

five tickets, he could claim respectively

270, 5,500, 75,000, and, finally,

1,000,000 times their cost to him. The numerical values of the probabilities corresponding to these different cases are worked out as follows: we must taken and i

=

1, 2, 3, 4, or 5 in the expression (2).

= 90,

m

= 5,

The results are

5 1 90 I8' 5. 4 2 Two tickets = 90·89 sol' 5·4·3 1 Three tickets ' 90. 89. 88 11748 5·4·3·2 1 Four tickets ' 90. 89. 88. 87 511038 5·4·3·2·1 1 . . ' Five tickets 90·89·88·87 ·86 43949268 Single ticket

=

=

=

a white balls and b black

From an urn containing

8. Problem 10.

ones, a certain number of balls, k, is drawn, and they are laid aside, their color unnoted.

Then one more ball is drawn; and it is required to find

the probability that it is a white or a black ball.

Solution.

Suppose the k balls removed at first and the last ball

drawn are laid on k +

1 different places, so that the last ball occupies

the position at the extreme right. of k +

The number of ways to form groups

1 balls out of the total number of a + b balls, attention being

paid to the order, is

(a+ b)(a+ b - 1) ··· (a+ b - k). Such is the total number of cases in this problem, and they may all be considered as equally likely.

To find the number of cases favorable to

a white ball, we observe that the last place should be occupied by one of the

a white balls.

Whatever this white ball is, the preceding k balls

form one of the possible arrangements out of taken k at a time.

a+ b

-

1 remaining balls

Hence, it is obvious that the number of cases favorable

to a white ball is

a(a + b

-

1) ··

(a + b

-

k),

and therefore the required probability is given by

COMPUTATION OF PROBABILITIES

SEC. 9]

21

for a white ball. In a similar way we find the probability bj(a + b) of drawing a black ball. These results show that the probability of getting white or black balls in this problem is the same as if no balls at all were removed at first. Here we have proof that the peculiar circumstances observed in Prob. 6 are general. 9. Problem 11. Two dice are thrownn times in succession.

What is

the probability of obtaining double six at least once? Solution. As there are 36 cases in every throw and each case of the throw can combine with each case of the second throw, and so on, st fir

the total number of cases inn throws will be 36". Instead of trying to find the number of favorable cases directly, it is easier to find the number

of unfavorable cases; that is, the number of cases in which double sixes would be excluded. In one throw there are 35 such cases, and inn throws

Now, excluding these cases, we obtain 36" - 35" there will be 35". favorable cases; hence, the required probability is p

=

(it)".

1-

If one die were thrownn times in succession, the probability to obtain

6 points at least once would be p

=

1-

(!)".

Now, suppose we want to find the number of throws sufficient to assure a probability > .72 of obtaining double six at least once.

To this

end we must solve the inequality

CH)"




log 2 log 36

-

log 35

=

24 6 ·

·

·

·

It means that in 25 throws there is more likelihood to obtain double six at least once than not to obtain it at all.

On the other hand, in

24 throws, we have less chance to succeed than to fail. Now, if we dealt with a single die, we should find that in 4 throws there are more chances to obtain 6 points at least once than there are chances to fail. This problem is interesting in a historical respect, for it was the first problem on probability solved by Pascal, who, together with his great contemporary Fermat, had laid the first foundations of the theory of probability.

This problem was suggested to Pascal by a certain French

- nobleman, Chevalier de Mere, a man of great experience in gambling. He had observed the advantage of betting for double six in 25 throws and for one six (with a single die) in 4 throws.

He found it difficult to

22

INTRODUCTION TO MATHEMATICAL PROBABILITY

[CHAP.

I

understand because, he said, there were 36 cases for two dice and 6 cases for one die in each throw, and yet it is not true that 25:4

=

Of

36:6.

course, there is no reason for such an arbitrary conclusion, and the cor­ rect solution as given by Pascal not only removed any apparent paradoxes in this case, but it led to the same number, 25, observed by gamblers in their daily experience.

A certain number

10. Problem 12.

among N compartments.

of identical balls is distributed

n

What is the probability that a certain speci­

fied compartment will contain h balls?

Solution.

To find the number of all possible cases in this problem,

suppose that we distinguish the balls by numbering them from 1 to

n.

The ball with the number 1 may fall into any of theN compartments, which givesN cases.

The ball with the number 2 may also fall into any

one of theN compartments; so that the number of cases for 2 balls will beN·N

=

N2•

Likewise, for 3 balls the number of cases will be N2

and for any number

n

•

N

=

N3,

of balls the number of cases will be N".

To find

the number of favorable cases, first suppose that a group of h specified balls falls into a designated compartment.

The remaining n - h balls may

be distributed in any way amongN - 1 remaining compartments. the number of ways to distribute n

-

But

h balls among N - 1 compart­

ments is (N - 1)"-" and this becomes the number of all favorable cases in which a specified group of h balls occupies the designated compartment. Now, it is possible to form C! such groups; therefore, the total number of favorable cases is given by C! · (N - 1)"-", and the required probability will be

p,. In case

n,

- C! _

·

N ( - 1)"-" . N"

N and h are large numbers, the direct application of this

formula becomes difficult, and it is advisable to seek an approximate expression for p,..

To this end we write the preceding expression thus:

where

p

=

�:r

i·- M._.P. ( �)( �) . ( � }

p.� 1·2

1 -

1 -

.

. .

1 -

h

1

SEC. 10)

Now, supposing 1 �

( - �)( 1

(a)

23

COMPUTATION OF PROBABILITIES

1 -

k � h

h- 1, we have

�

)

k

= 1-

�

k

+

(h

k) n-;

> 1

�

- ·

On the other hand, k(hand so

(

(b)

1-

(a)

The inequalities

k\ n}

(

k) � 4' h2

) (

h- k 1- n-

�

1-

h

2n

)

2 •

and (b) give simple lower and upper limits for P.

For we can write P2 thus:

(a )

and then apply

or (b), which leads to these inequalities

P


(

)

h-1

1- -n -r . h

Correspondingly, w e have

Ph




Problem 13. points with

n

Solution.

(NY ( (NY (

1· 2 · 3

) (

h h-1 1-2n

) (

h h-1 1-- 2 ,

1 1-. h N

·

1· 2 3 · ·

·

"

1 1-h N

"

"-"

)

n

)

What is the probability of obtaining a given sum 8 of

dice? The number of all cases for

n

dice is evidently 6".

The

number of favorable cases is the same as the total number of solutions of the equation

(1)

«t

+

«2

+

·

·

·

+

a,.

= 8

where a1, a2, a,. are integers from 1 to 6. This number can be determined by means of the following device: Multiplying the polynomial •

(2) by itself, the product will consist of terms

24

INTRODUCTION TO MATHEMATICAL PROBABILITY

where

«1

and

a2

[0uAP.

I

independently assume all integral values from 1 to 6.

Collecting terms with the same exponent 8, the coefficient of X' will give the number of solutions of the equation

«1, «2

being subject to the above mentioned limitations.

Similarly, multiplying the same polynomial

(2)

three times in itself

and collecting terms with the same exponent 81 the coefficient of give the number of solutions of equation

(1)

for n

=

3.

x• will

In general, the

number of solutions of equation (1) for any n·is the coefficient of

x• in

the expanded polynomial

(x + x2 + xa + x• + z& + z&)". Now we have identically

and by the binomial theorem

x"(1- x6)" {1

-

x)-n

n =

�(-1)1C�xn+&l l =0 ..

=

� C::+l-1x' T2 > · · · > T,. so that

To

If (n+1)p < 1 the greatest integer (n + 1)p is 0, and there is only one greatest term T0• If, 1, there are two terms To however, (n + 1)p T1 greater than is the greatest term.

contained in

=

=

others. If

(n+1)q � 1, all the terms in series (a) are >1 with the exception 1; namely, when (n+1)q 1.

of the last term, which may be equal to

=

Hence,

To< T1 < · · ·


np.

We shall denote this probability by

8)

SEC.

53

REPEATED TRIALS

A similar notation Q(l) will be used to denote the probability

P(l).

that the number of failures is > l where again l > nq. The probability P(k, l) of the inequalities k � m � l can be expressed as follows:

P(k, l) if

>

l




l

np; P(k, l) l are < np.

k and P(l) we

- P(l) - Q(n - k)

=

P(k - 1) - P(l)

and finally =

Q (n - l

- 1) - Q(n - k)

have the expression

' pl+ 1+n 1 (l+1) ( -z- 1)l lqn-z(n

:

+

{ ��; �+ - (� + ;��7; i)- 2) (�Y+ · · · } 1

The first factor

nl l+l n-l-1 (l+1) !(n - l - 1)1P q can be approximately evaluated by the method of the preceding section. The whole difficulty resides in the evaluation of the sum

s

- 1+n -l l -2 1 qp

_

+

+

(n - l

(l

- 1)(n - l - 2) (pq)2+

+ 2)(l + 3)

which is a particular case of the hypergeometric series R F(a,tJ,'Y,X )

=

In fact

a{3 a(a + 1)(3{(3 + 1) 2 x+ 1+ . x+ 1·2'Y('Y+1) 1 'Y

(

F -n+ l + 1,1,l+ 2,

- �)

=

S.

Now, owing to this connection between S and hypergeometric series, S can be represented in the form of a continued fraction.

First, it is

easy to establish the following relations:

'Y

F(a,{3 + 1,

+ 1,x)

F(a+1,{3,'Y+ 1, x)

=

F(a, {3,'Y,x) + a('Y - {3) +x (a+1,{3+ 1, 'Y + 2, x) + 'Y('Y 1 F(a,{3, 'Y,x) +. f3('Y - a) (a+1,{3+ 1, 'Y+ 2, x). + x 'Y('Y +1

l

=

{

54

INTRODUCTION TO MATHEMATICAL PROBABILITY

Substituting

[CHAP.

III

a+n,{3 + n,'Y+2n and a+n,{3+n + 1,'Y + 2n + 1, a,{3,'Y in these relations and setting

respectively, for

X2n =F(a+n,{3+n,-y+2n,x); X2n+l =F(a+n,{3+n+ 1,'Y+2n+ 1,x) a2"

�+��-a+� (-y+2n)('Y+2n-1);

=

a2n+l =

�+��-{3+� (-y+2n)(-y+2n + 1)

for brevity, we have

Xo = Xt - a1xX2 X1 = X2- a2xXa

whence

1 In our particular case X1 =F( -n+ l+ 1, 1, l+2,x), and a2n-2z-1

Xo

=

1

= 0.

Hence, taking

x = _'/! and introducing new notations, we have a q

finite continued fraction (5)

Cn-!-1 dn-1-1

1 +

1

where

Ck

(6)

_ -

(n k - l)(l + k)p . (l+2k-1)(l+2k)q' -

Every one of the numbers

Ct.

k(n+k)p (l + 2k)(l + 2k + 1)q.

will be positive and < 1 if this is true for

Now

Ct = if

Ck

dk =

l

>

(n-l 1)p (l+2)q -


)"-m

i=l

Supposing again

Pt

1

=

a1

P2

Tc' =

a2

2

=

Tc'

·

·

•

a�c

= ·

·

·

=

P�c

=

1 =

k

1

SEc.

7)

PROBABILITIES OF HYPOTHESES AND BAYES' THEOREM

and letting k

-+ oo

formula

Q

{8)

=

• Cm Rl

(7)

69

in the limit becomes

J /xm+m•{1 - x)n+n.-m-m•dx J:lxm{1 - x)n-md:J: 0

•

This formula leads to the following conclusion: When the unknoum probability p of an event E may have any value between limits 0 and and the a priori probability of its be'ing contained between

{j

-

a

a

1

and {j is

(so that equal probabilities correspond to intervals of equal length),

the probability that the event E will happen m1 times in n1 trials following n trials which produced Em times is given by formula {8).

In particular, for n1

=

m1

=

1 (evaluating integrals by the known

formula), we have

Q

=

m+ 1 n+2

.

This is the much disputed "law of succession" established by Laplace.

7. Bayes' formula, and other conclusions derived from it, are neces­ sary consequences of fundamental concepts and theorems of the theory of probability.

Once we admit these fundamentals, we must admit Bayes'

formula and all that follows from it. But the question arises: When may the various results established in this chapter be legitimately applied?

In general, they may be applied

whenever all the conditions of their validity are fulfilled; and in some artificial theoretical problems like those considered in this chapter, they unquestionably are legitimately applied.

But in the case of practical

applications it is not easy to make sure that all the conditions of validity are fulfilled, though there are some practical problems in which the use of Bayes' formula is perfectly legitimate.1

In the history of probability

it has happened that even the most illustrious men, like Laplace and Poisson, went farther than they were entitled to go and made free use principally of formulas lems.

{6)

and

{8)

in various impQrtant practical prob­

Against the indiscrimina;te use of these formulas sharp objections

have been raised by a number of authors, especially in modern times. The first objection is of a general nature and hits the very existence of a priori probabilities.

If an urn is given to us and we know only that

it contains white and black balls, it is evident that no means are available to estimate a priori probabilities of various hypotheses as to the propor­ tion of white balls.

Hence, critics say, a priori probabilities do not exist

at all, and it is futile to attempt to apply Bayes' formula to an urn with an unknown proportion of balls. 1

At first this objection may appear

One such problem can be found in an excellent book by Thornton C. Fry, "Prob­

ability and Its Engineering Uses," New York, 1928.

70

INTRODUCTION TO MATHEMATICAL PROBABILITY

[CHAP. IV

very convincing, but its force is somewhat lessened by considering the peculiar mode of existence of matfiematical objects. Some property of integers, unknown to me, is not present in my mind, but it is hardly permissible to say that it does not exist; for it does exist in the minds of those who discover this property and know how to prove it. Similarly, our urn might have been filled by some person, or selected from among urns with known contents.

To this person the a priori

probabilities of various proportions of white and black balls might have been known.

To us they are unknown, but this should not prevent

us from attributing to them some potential mode of existence at least as a sort of belief. To admit a belief in the existence of certain unknown numbers is common to all sciences where mathematical analysis is applied to the world of reality.

If we are allowed to introduce the element of belief

into such "exact" sciences as astronomy and physics, it would be only fair to admit it in practical applications of probability. The second and very serious objection is directed against the use of formula (6), and for similar reasons against formula

(8).

Imagine,

again, that we are provided with an urn containing an enormous number of white and black balls in completely unknown proportion.

Our aim

is to find the probability that the proportion of white balls to the total number of balls is contained between two given limits.

To that end, we

make a long series of trials as described in Prob. 5 and find that actually in n trials, white balls appeared m times.

The probability we seek would

result from Bayes' formula, provided numerical values of a priori proba­ bilities, assumed on belief to be existent, were known.

Lacking such

knowledge, an arbitrary assumption is made, namely, that all the a priori probabilities have the same value.

Then, on account of the

enormous number of balls in our urn, formula (6) can be used as an approximate expression of P. positive number

e ,

It can be shown that, given an arbitrary

however small, the probability of the inequalities m --

n

e

< p
.

It is important to find means to determine the

probability Pn, the a priori probability of E in the nth trial when the results of the preceding trials remain unknown; as well as to determine the probability pc�> of E in the nth trial when we possess the positive information that E has materialized in the kth(k < n - 1) trial.

2. Thus we are led to the following problem concerning simple chains of dependent trials: The initial probability

Problem 1.

P1 of the event E in a simple

chain of trials being known, find the probability Pn of E in the nth trial when the results of the preceding trials remain completely unknown. Also, find the probability

pC�> of E in the nth trial when it is known that E has happened in the kth trial where k < n - 1. Solution. In the nth trial the event E can happen either preceded by E in the (n - 1)st trial, the probability of which is Pn-1, or preceded by F in the (n - 1)st trial, the probability of which is 1 - p,._1• By the theorem of compound probability, the probability of the succession

EE is Pn-1a, while the probability of the succession FE is (1 Hence, the total probability (1)

Pn

=

-

Pn--1){3.

Pn is

aPn-1 + {3(1 - Pn-1)

=

(a - {3)pn-1 + {3.

This is an ordinary equation in finite differences.

It has a particular

solution

Pn where

c

=

c

const.

=

is determined by the equation

c

=

(a - {3)c + {3,

whence 8

c provided 1

If 1 + {J

means that never

1 + {3 E

-

a

� 0.1

=

On

-{3 -::- - a ::-1--:+-the

other

the

corresponding

a = 0 or a {J = 1, we necessarily have a = 1, {J 0, which must occur in all the trials if it actually occurs in the first trial, and -

-

=

occurs if it does not actually occur at the outset.

extreme case-in which interest.

hand,

a

-

{J

=

-

This case, as well as the other

1 can therefore be excluded as not possessing real

INTRODUCTION TO MATHEMATICAL PROBABILITY

76

[CHAP. V

homogeneous equation

Yn

=

(a - f3)Yn-1

has a general solution

Yn

=

involving an arbitrary constant

C(a-{3)n-1 Adding to it the previously found

C.

particular solution, we obtain the general solution of (1) in the form Pn

The arbitrary constant

C(a -{3)n-1 + 1+ -a'

=

;

is determined by the initial condition

C

{3 C+ + 1 {3

_

a

=

so that finally Pn

=

�

1+

If

a

_

+

Pl

we see that

p,.

1+

1 +

does not depend on

cause we may exclude the cases

a -{3 is

� a) (a- {3)n-1,

(Pl-

=

n

P1 _

{3 {3- a

and is constantly equal to

a -{3

=

1 or

a -{3

=

-1,

p 1•

Be­

so that

contained between -1 and 1, we may conclude from the above

expression that

p,.,

if not a constant, at any rate tends to the limit

{3 1+{3-a as

n

increases indefinitely.

As to

p�k>

we find in a similar way that it satisfies the equation

(2) of the same form as equation (1).

But the initial condition in this

a because the probability of E happening in the (k + 1)st case is pJok_/.1 trial is a when it is known that E occurred in the preceding trial. The =

solution of

(2)

satisfying this initial condition is

p �>

{3 =

1 +{3

1 -a + (a -{3)n-k -a 1 +{3 -a

·

As the second term in the right-hand member decreases with increas­ ing

n

and finally becomes less than any given number, we see that the

positive information concerning the result of the kth trial has less and less

SEc. 3)

USE OF DIFFERENCE EQUATIONS IN SOLVING PROBLEMS 77

influence on the probability of E in the following trials, and in remote trials this influence becomes quite insignificant. An urn contains a white and b black balls, and a series of drawings of

Example.

one ball at a time is made, the ball removed being returned to the urn immediately after the taking of the next following ball. What is the probability that the nth ball drawn is white when: (a) nothing is known about the preceding drawings; (b) the kth ball drawn is white? In this particular problem we have and

a- 1 a a a = = P1 = a+b - 1, f3 a+b-1 a+ b •

--

a {3 =pl. = 1+{3-a a+b --

Thus

Pn

=

a a +b

P1 =

--·

That is, the probability for any ball drawn to be white is the same as that for the first ball, nothing being known about the results of the previous drawings. expression for

p�k\

p�k) So, for instance, if

The

is, in this example,

=

a b + (-1)"-k( a+b)(a + b-1)"-• a+b

a = 1, b = 2,

n

p�l)

= 5, k = 3, 1 1 2 = + . 3 3 22 =2;

the information that the third ball was white raises to

7'2

the probability that the fifth

ball will be white; it would be �without such information.

3. The next problem chosen to illustrate the use of difference equa­ tions is interesting in several respects.

It was first propounded and

solved by de Moivre.

Problem 2.

In a series of independent trials, an event E has the If, in this series, E occurs at least r times in

constant probability p.

succession, we say that there is a run of r successes.

What is the proba­

bility of having a run of r successes inn trials, where naturallyn > r?

Solution.

Let us denote by y,. the unknown probability of a run of

r in n trials.

In n + 1 trials the probability of a run of r will then be

Yn+l·

Now, a run of

1·

in n + 1 trials can happen in two mutually

exclusive ways: first, if there is a run of r in the firstn trials, and second,

if such a run can be obtained only in n + 1 trials. the first hypothesis is y,..

The probability of

To find the probability of the second hypothe­

sis, we observe that it requires the simultaneous realization of the follow­ ing conditions: (a) There is no run of which is 1

-

Yn-r·

(b)

1·

in the first

In the (n

-

n

-

r

trials, the probability of

r + 1)st trial, E does not occur,

INTRODUCTION TO MATHEMATICAL PROBABILITY

78

q

the probability of which is remaining As

=

1 - p.

(c)

trials, the probability of which is

r

(a) , (b), (c)

[CHAP. V

Finally, E occurs in the

p•.

are independent events, their simultaneous mate­

rialization has the probability

(1 - y,._,)qp•. At the same time, this is the probability of the second hypothesis. Adding it to

y,.,

we must obtain the total probability Yn+1•

Thus

(3) and this is an ordinary linear difference equation of the order

r

+ 1.

Together with the obvious initial conditions Yo

Y1

=

it serves to determine instance, taking n

=

y,.

= r,

·

·

·

=

0,

completely for n

=

Yr

=

+ 1,

= r

(3)

r

P'

+ 2, . . . .

For

p• + p•q.

+ 1, we obtain

= r

YrH

and so forth.

Yr-1

we derive from Yr+1

Again, taking n

=

=

P" +

2p"q

Although, proceeding thus, step by step, we can find the

required probability

y,. for any

given n, this method becomes very labori­

ous for large n and does not supply us with information as to the behavior of

y,. for

large n.

It is preferable, therefore, to apply known methods of

solution to equation by introducing z,. =

z,.

(3). First we can obtain a homogeneous equation 1 - y,. instead of y,.. The resulting equation in

is

(4)

- z,. + qp•z,._r

Zn+l

=

0

and the corresponding initial conditions are: Zo

=

Z1

=

'

'

'

=

Zr-1

=

1j

Zr

1 - p".

=

We could use the method of particular solutions as in the preceding problem, but it is more convenient to use the method of generating functions.

The power series in � rp{E)

=

Zo + Z1� + Z2�2 +

·

·

·

is the so-called generating function of the sequence Zo, z1, z2, . . . . If we succeed in finding its sum as a definite function of �.the development of this function into power series will have precisely of �".

z,.

as the coefficient

To obtain rp(�) let us multiply both members of the preceding

series by the polynomial

SEc. 4]

USE OF DIFFERENCE EQUATIONS IN SOLVING PROBLEMS 79

The multiplication performed, we have

-

� + qp•�•-H)!J'W

+ (z1- zon 1 = Zo + + (Zr-1 - Zr-2)�•- + + (z - Zr-1)�· + (Zr 1 - z,. + qp•zo)�r+1 + ,. + have In the right-hand member the terms involving �·H, �r+2, vanishing coefficients by virtue of equation (4); also z�c - Z1c-1 = 0 for r - 1, while k 1, 2, 3, (1

·

·

·

... ·

=

Zo =

and

1

Zr - Zr-1

so that

=

·

·

.

-p•

and 1 - p·�· IJ'(�) =:= 1 - � + qp·�r+1'

The generating function IJ'(�) thus is a rational function and can be The

developed into a power series of � according to the known rules. coefficient of ��� gives the general expression for z,..

Without any dif­

ficulty, we find the following expression for z,.:

z,.

(5) where

=

fJn,r - p•fJn-r,r

n r+l fJ,.,,.

=

� ( -1)1C�_1,.(qp•)1

1=0

and fJ,._,.,,. is obtained by substituting n- r instead of n.

If n is not very

large compared with r, formula (5) can be used to compute z,. and

Yn For instance, if n

z20

=

=

=

1-

z,..

20, r = 5, and p = q = ,%, we easily find

(

)

45 15 10 1 10 10 1 - 64 + 642 - 643 - 32 1 - 64 + 642

and hence

Z20

= 0.75013

correct to five decimals; Y20 = 0.24987 is the probability of a run of 5 heads in 20 tossings of a coin.

4. But if

n

is large in comparison with r, formula (5) would require

so much labor that it is preferable to seek for an approximate expression for z,. which will be useful for large values of n.

It often happens, and

in many branches of mathematics, but especially so in the theory of probability, that exact solutions of problems in certain cases are not of any use.

That raises the question of how to supplant them by con-

80

INTRODUCTION TO MATHEMATICAL PROBABILITY

[CHAP. V

venient approximate formulas that readily yield the required numbers. Therefore, it is an important problem to find approximate formulas where exact ones cease to work.

Owing to the general importance of approxi­

mations, it will not be out of order to enter into a somewhat long and complicated investigation to obtain a workable approximate solution of our prob . lem in the interesting case of a large n. Since cp(�) is a rational function, the natural way to get an appropriate expression of

Zn

would be to resolve cp(�) into simple fractions, correspond­

ing to various roots of the denominator, and expand those fractions in power series of �. However, to attain definite conclusions following this method, we must first seek information concerning roots of the equation 1

_

5. Let

� + qpr�rH

where =

0:

=

0.

pr (1 - p).

When

p varies from 0 to 1, the maximum of pr(1

p

� 1 and is

=

r

rj (r+ 1)rH so that

r

a

�

-

p) is attained for

rj (r+ 1)r+1 in all cases.

r

To deal with the most interesting case, we shall assume

r p �o. 6. The equation

�

0.1

- - a�r+l = 1

i6

1 Hence,

1

>

+ a�'= I pe

r+

1

can be exhibited in the form

=

·

The statement is thus completely proved.

�

�

1

-

8 may be

'

Substituting

_

sin

0

1.

here, and again equating imaginary parts, we get

ap'+l

sin

r8 =

sin

8

and, combining this with (7), P

1

=

(r

sin

+

1)8

sin 1·8

a=

;

m88 . -. -(m

. sin The extreme values of the rat1o

mteger > 1) correspond to certain

SID

roots of the equation

sin

cos

8 m8 � �7n�8�

m

=

sin

s

=

Vl

The equality sign is excluded if sin

8

+

r8)' sin 8 (1· + 1)8]r+1•

(sin [sin

cos

but for every root of this equation

m8 8, t : (m _ 8 m 1) sin2

differs from 0.

�

82

INTRODUCTION TO MATHEMATICAL PROBABILITY

[CHAP. V

If the imaginary part of � is positive, the argument 8 is contained between 0 and than

r

-

r

r.

�f

In this case, it cannot be less than For, if 0 < 8 < r8

�1

or greater

�1

r

sinr8

r

>

or

sin r( + 1)8 r( + 1)8

sinr8 r > __ . sin (r + 1)8 r + 1 At the same time

1 sin8 > __ sin (r + 1)8 r + 1

and hence a

=

{

sin r8 sin8 rr > sin r( + 1)8 sin r( + 1)8 r( + 1)r+1'

which is impossible.

}r

That 8 cannot be greater than

r-

r

� 1 follows

simply, because in this case, sin r( + 1)8 and sinr8 would be of opposite signs andpwould be negative. As

11'

r + 1

� 8 ;;;;!

r-

11'

1, we have

r +

psin8>psin r On the other hand, sin

x > 2x/r

if 0
•

x

�f


1/p.

Hence,

2 (r + 1)p'

Thus, imaginary parts of all complex roots have the same lower bound

2 (r + 1)p

of their absolute values. 7. Denoting the roots of the equation/� ( ) �,.; we have

(k

=

=

0 by

1, 2, . . . r + 1)

r+l

V'W

=

1- P�r. � 1- 1 1· � (1- p)�r.r( + 1- r�r.) �"

(

k=l

)

SEc.

USE OF DIFFERENCE EQUATIONS IN SOLVING PROBLEMS

7]

Hence, expanding each term into power series of coefficients of

�", we find

r+l

z

� 1- P�k

,.

=

since

�;;"

·

k=l

I

+ 1 "H (1 - p�k)�" < r p + r� r(1 - p) 1 - k) (1 - p)�k(r

I!�k - PI < 2p;

1�;;11 < p; If

� and collecting

� ( 1 - p)�k r + 1- rE,;

For every imaginary root, we have

I

83

1

< lr + 1 - r�kl

(r + 1)p, 2r

r is odd, there are r - 1 imaginary roots and the part in the expression

of z,. due to them in absolute value is less than

(r + 1)(r- 1) n+2 p r(1 - p) The term corresponding to the root z

"

IBI

where




Ya+l,r

=

(n -a)(n -a -1) 1 2 ·

a+ 2 ' n

_

2

a+ 1 ' + n n

and so on. 12. If, in the problem of runs, pis supposed to be >

ity of a run of

r in

ntrials is greater than 1

_

(

p - P1

r -

(r + 1)p ,

+

r

r+l

=

r+1

)

r(p + p,) p�+l 2 1 - p,

where p1 < __ is a root of the equation

PHI - p,)

1- •prove

p' ( 1 - p).

that the probabil­

USE OF DIFFERENCE EQUATIONS IN ·SOLVING PROBLEMS 95 13. To find an asymptotic expression of probability for a run of r in n independent trials, if p �

___!:_ , the following proposition is of importance: Imaginary and nega­

r+ 1

tive roots of the equation

(1 - 8):1:"

are

.

- X

+8

n 0 1 of the equation

2'11' (1 - 8)R" - R + s cos n

-

Prove the truth of this statement.

14. Given

0.

n of black and white balls in known

urns containing the same number

8

=

proportions, drawings are made in the following manner: first, a single ball is drawn out of every urn; second, the ball drawn from the first urn is placed into the second; that drawn from the second is placed in the third, and so on; finally, the ball drawn from the last urn is placed in the first, so that again every urn contains n balls. Sup­ posing that this operation is repeated t times, find the probability of drawing a white

ball from the xth urn. Solution. Let be the required probability.

First, it can be shown that it

y.,,,

satisfies the equation

Ys,l

1

+

( ;)y.,,l-1 ;Yz-1,1-1•

=

-

Y2,o, y.,o are known; and, moreover, the function The initial probabilities must satisfy a boundary condition of the periodic type, o Hence,

Yt,o, y.,,1 y.,, ( ) [

•

•

•

Y ,1 y.,1• =

applying Lagrange's method, the following solution is found

=

1

-

1

;

'

f(x) +

t

1 . (n

t(t - 1) 1) + f 1 . 2(n 1)• (x

- 1/(:c

-

y.,,O

when

-

-

2) +

where

j(ie)

=

and the definition is extended to

x

x>O

� 0 by setting

f( -x)

=

f(s

-

:c).

If, to begin with, all urns contain the same number of white and black balls, so that f(x)

=

const.

=

p, we shall have, no matter what t is,

References DEMOJVRE: "Doctrine of Chances," 3d ed., 1756. LAPLACE: "Theorie analytique des probabilites," Oeuv1·es VII, pp. 49 if. LAGRANGE: "M6moire sur les suites r6currentes dont les termes varient de plusieurs manieres diff6rentes, etc.," Oeuvres IV, pp. 151 JJ.

CHAPTER VI BERNOULLI'S THEOREM 1. This chapter will be devoted to one of the most important and

beautiful theorems in the theory of probability, discovered by Jacob Bernoulli and published with a proof remarkably rigorous (save for some irrelevant limitations assumed in the proof) in his admirable posthumous book"Ars conjectandi" {1713).

This book is the first attempt at scien­

tific exposition of the theory of probability

as

a separate branch of

mathematical science.

If, inn trials, an event E occurs m times, the number mis called the

"frequency" of E in n trials, and the ratio m/n receives the name of "relative frequency."

Bernoulli's theorem reveals an important proba­

bility relation between the relative frequency of E and its probability p. Bernoulli's Theorem. With the probability approaching 1 or certainty lL8 near lL8 we please, we may expect that the relative frequency of an event E in a series of independent trials with constant probability p will differ from that probability by less than any given number E > 0, provided the number

of trials is taken sufficiently large. In other words, given two positive numbers

E

and .,, the probability

Pof the inequality

will be greater than 1 limit depending upon Proof.

E

-

11

and

if the number of trials is above a certain fl.

Several proofs of this important theorem are known which

are shorter and simpler but less natural than Bernoulli's original proof. It is his remarkable proof that we shall reproduce here in modernized form. a. Denoting by T m,

as

usual, the probability of m successes in n trials.

we shall show first that {1) if b > a and k > 0.

Since the ratio

Tz.r1 T.,

n- x p = x+l q 96

SEc. 1)

97

BERNOULLI'S THEOREM

decreases as x increases we have forb >

a

or Changingb, a, respectively, intob + 1, a + 1;b + 2, a + 2; + k, it follows from the last inequality that

·

·

·

b + k,

a

that is, Tb+" Tb

Ta+".


o.

C=

T,.

+ +

the first of which contains p.

TA+l T,. +t

-

+ . . . + + + ·

·

·

X= g terms.

T,._l T,.

Combining terms of the

second sum into groups of g terms (the last group may consist of less than

g terms) and setting for brevity

+

At=

T,.

A2 =

T��+u

Aa =

T,.Hu

T ,.+l

+

+

+ + T,.Ho+l +

T,.+cl-1

+

·

·

·

T��+u+t

·

·

·

·

+ T,.Hu-t + T,.+Bu-t

we shall have

a=

At

+

Az

Aa

+

+

and at the same time

A2

(2)

At

T,. , T>o.


o.+u T>o.

+ +

+ +

T>o.+u+l T>o.+t

is less than the greatest of numbers

·

•

·

•

·

+ T>o.+au-t + T>o.+u-t

98

INTRODUCTION TO MATHEMATICAL PROBABILITY

[CHAP. VI

But by inequality (1) T>.+o > T>.+o+l > T>.+1 T>.

hence A1 < T"

T>..

A

Similarly, A2 < Tl'+o, T" A1

and again by inequality (1) TP+o < T>.+u, T>. T"

Consequently A2 < T" , T).

Aa

A2

A1

and inequalities

(2)


.+l=n-"X'f!..

As "X� np

which shows that

T�:

1

"X+ 1 q

.

npq n - "X l! ::5: < 1 "X+ 1 q - npq + q < 1.

The inequality just established shows that in the following expression: T" T>.

=

Tp--a+1 Tp--a T>.+1 T" Tp-1 Tl'-1 . Tl'-2 . . . Tp--a . Tp-a-1 . . . T).

all the factors are

< 1.

Consequently, if we retain

only, replacing the others by 1, we get

Moreover,

a

� g first factors

SEC. 1)

99

BERNOULLI'S THEOREM

whence the following important inequality results:

)

(n- p.+al!. a· \P -a+1 q

T,. < Tx

(3)

Here a is an arbitrary positive integer �g. Now, let

e

be an arbitrary positive number.

Then we can show that

for n

(4) we have both (i)

� a(1 +E)

-

-q

E(p +E)

-

n P.+ a'!!.. :S: _ P_ p.-'a+1q -p+E

and

(ii)

a � g.

Since p. � np+nE, it suffices to show that (i) is satisfied for p. If

p. = np +

= np+nE.

nE inequality (i) is equivalent to

-

nq

nE+a

np+ne

- a+

:S:-q-

1-p+ e

or, after obvious simplifications, nE(p

a(1+E)

+ e) �

But this inequality follows from

- q.

To establish (ii), since

(4).

are integers, it suffices to show that

a

< g+ 1.

'X < np+1 and consequently g+ 1 > nE. lished if we can show that nE �

a

But

p+ E

a and g

np+nE,

Hence (ii) will be estab­

which by virtue of

a(1+E)

p. �

- q =a

(4)

will be true if

>

that is, if a(1+E)- q �

ap+ae

or aq

- q � 0 which is obviously true, a being a positive integer. d. The auxiliary integer a is still at our disposal. Given an arbitrary positive number 11 < 1 we shall determine a as the least integer satisfying

the inequality

or

At the same time

1

log-

11 '---7" - " >

-

( �)

-----.

log

1+

a

-

1

100

INTRODUCTION TO MATHEMATICAL PROBABILITY

and since log

(1 �) +

>

p

[CHAP. VI

� /we shall have

1 p+ E a< 1 + --log1J

E

and 1 + E a(1 + E) - q < log ! + !. E(p + E) E2 1J E Consequently, if (5)

n�

then by virtue of (i) and

(3)

1+

E

--

E2

1 1 log- + 1J

E

T,. < T>. 1/, and by virtue of

(2)

A 1 < A 11, A 2 < A 111 < A 112, A a < A 211 < A113, whence (6) This inequality holds if n satisfies (5).

No trace of the auxiliary

integer a is left. e. Let us now consider the inequalities m

-E

1

or if only

n �

1 + E --

2 E

1

1

'lj

E

log- + -·

This completes the proof of Bernoulli's theorem. For example, if p = q = � and

E =

0.01, '11 = 0.001 we get from (5)

n � 69,869 which shows that in 69,869 trials or more there are at least 999 chances against 1 that the relative frequency will differ from� by less than

7l00.

The number 69,869 found as a lower limit of the number of trials is much too large.

A much smaller number of trials would suffice to fulfill

all the requirements.

From a practical standpoint, it is important to

find as low a limit as possible for the necessary number of trials (given and '11).

E

With this problem we shall deal in the next chapter.

2. Bernoulli's theorem states that for arbitrarily given exists a number no(E, '11) such that for any single value

n

E

and '11 there

> no(E, '11) the

probability of the inequality

will be greater than 1 given

E

-

11·

The question naturally arises, whether for

and 11 a number N(E, 11) depending upon

E and '11 can be found

such

that the probability of simultaneous inequalities

'11· The following theo­ N(E, '11) will still be greater than 1 rem due to Cantelli shows that this question can be answered positively.

for all n >

Cantelli's Theorem. satisfying the inequality

-

For given

E

< 1, 11 < 1 let N be an integer

102

INTRODUCTION TO MATHEMATICAL PROBABILITY

[CHAP. VI

The probability that the 1·elative f1·equencies of anevent E will differ f1·om p by less thane inthe Nth and all the following t1·ials is r g eater than 1 - TJ.

Proof.

We shall prove first that the probability Q,. of the inequality

.I�- PI�

will always be less than 2e-��n•'. preceding section for any rJ > 0

According to results proved in the

Q,. < if n >

This inequality, if we take TJ

E --

1+

e2

=

E

1J

log

1

-

rJ

1 +e

·

2e-��n•' becomes

1+e

1

1+e

+ ; - --2 log 2 n > --n 2

e

and in this form it is evident, since for e < 1 1

Hence, as stated,

-

1 +e

-E

log 2 < 1 - 2 log 2 < 0.

Q,. < 2e-ln•'.

(8)

The event A, in which we are interested, consists in simultaneous fulfillment of all the inequalities

for n

=

N, N + 1, N + 2, . . . .

The opposite event B consists in

the fulfillment of at least one of the inequalities

where ncan coincide either with N, or with N + 1, or with N + 2, The probability of B, which we shall denote by R, certainly does not exceed the sum of the probabilities of all the inequalities

for n N, N + 1, N + 2, . Consequently, referring to (8), =

..

SEc. 3)

103

BERNOULLI'S THEOREM

To satisfy the inequality

it suffices to take

N >

2

-2 E

log

2

-

f/

+

2

-2 E

log

1 1 -6

;1• 1

Now 2

log 1 2 E

1 -e

2

2

log 2 + 2. t•• < 2 E . E

Consequently, if

we shall haveR < f/ and at the same time the probability of A will be greater than 1

-

f/, which proves Cantelli's theorem.

SIGNIFICANCE OF BERNOULLI'S THEOREM 3. As was indicated in the Introduction, one of the most important problems in the theory of probability consists in the discovery of cases

where the probability is very near to 0 or, on the contrary, very near to 1,

because cases with very small or very "great" probability may have real practical interest.

In Bernoulli's theorem we have a case of this kind;

the theorem shows that with the probability approaching as near to 1 or certainty as we please, we may expect that in a sufficiently long series of independent trials with constant probability, the relative fre­ quency of an event will differ from that probability by less than any specified number, no matter how small.

But it lies in the nature of the

idea of mathematical probability, that when it is near 1, or, on the con­ trary, very small, we may consider an event with such probability as practically certain in the first case, and almost impossible in the second. The reason is purely empirical. To illustrate what we mean, let us consider an indefinite series of independent trials, in which the probability of a certain event remains constantly equal to �-

It can be shown that if the number of trials

is, for instance, 4.0,000 or more, we may expect with a probability > 0.999 that the relative frequency of the event will differ from � by less than 0.01.

In other words, we are entitled to bet at least 999 against 1 that

the actual number of occurrences will lie between the limits 0.49n and 0.51n if n � 40,000.

If we could make a positive statement of this

kind without any mention of probability, we should be offering an ideal scientific prediction.

However, our knowledge in this case is incomplete

INTRODUCTION TO MATHEMATICAL PROBABILITY

104

[CHAP. VI

and all we are entitled to state is this: we are more sure to be right in predicting the above limits for the number of occurrences than in expect­ ing to draw a white ball from an urn containing 999 white and only 1 black ball. In practical matters, where our actions almost never can be directed with perfect confidence, even incomplete knowledge may be taken as a

sure guide. Whoever has tried to win on a single ticket out of 10,000 knows from experience that it is virtually impossible. Now the convic­ tion of impossibility would be still greater if one tried to win on a single ticket out of 1,000,000. In the light of such examples, we understand what value may be attached to statements derived from Bernoulli's theorem: Although the fact we expect is not bound to happen, the probability of its happening is so great that it may really be considered as certain.

Once in a great

while facts may happen contrary to our expectations, but such rare excep­ tions cannot outweigh the advantages in everyday life of following the indications of Bernoulli's theorem.

And herein lies its immense practical

value and the justification of a science like the theory of probability. It should, however, be borne in mind that little, if any, value can be attached to practical applications of Bernoulli's theorem, unless the conditions presupposed in this theorem are at least approximately ful­ filled: independence of trials and constant probability of an event for every trial. And in questions of application it is not easy to be sure whether one is entitled to make use of Bernoulli's theorem; consequently, it is too often used illegitimately.

It is easy to understand how essential it is to discover propositions

of the same character under more general conditions, paying especial attention to the possible dependence of trials. achievements in this direction.

There have been valuable

In the proper place, we shall discuss the

more important generalizations of Bernoulli's theorem.

4. When the probability of an event in a single experiment is known, Bernoulli's theorem may serve as a guide to indicate approximately how

often this event can be expected to occur if the same experiments are

repeated a considerable number of times under nearly the same condi­ tions.

When, on the contrary, the probability of an event is unknown

and the number of experiments is very large, the relative frequency of that event may be taken as an approximate value of its probability. Bernoulli himself, in establishing his theorem, had in mind the approxi­ mate evaluation of unknown probabilities from repeated experiments.

That is evident from his explanations preceding the statement of the theorem itself and its proof.

Inasmuch as these explanations are interest­

ing in themselves, and present the original thoughts of the great discov­ erer, we deem it advisable here to give a free translation from Bernoulli's _

Smc. 4)

book.

BERNOULLI'S THEOREM

105

After calling attention to the fact that only in a few cases can

probabilities be found a priori, Bernoulli proceeds as follows: So, for example, the number of cases for dice is known.

Evidently there are

as many cases for each die as there are faces, and all these cases have an equal chance to materialize. For, by virtue of the similitude of faces and the uniform

distribution of weight in a die, there is no reason why one face should show up more readily than another, as there would be if the faces had a different shape or if one part of a die were made of heavier material than another.

So one knows

the number of cases when a white or a black ticket can be drawn from an urn, and besides, it is known that all these cases are equally possible, because the num­ bers of tickets of both kinds are determined and known, and there is no apparent reason why one of these tickets could be drawn more readily than any other. But, I ask you, who among mortals will ever be able to define as so many cases, the number, e.g., of the diseases which invade innumerable parts of the human body at any age and can cause our death?

And who can say how much more

easily one disease than another-plague than dropsy, dropsy than fever- can kill a man, to enable us to make conjectures about the future state of life or death?

Who, again, can register the innumerable cases of changes to which the

air is subject daily, to derive therefrom conjectures as to what will be its state after a month or even after a year?

Again, who has sufficient knowledge of the

nature of the human mind or of the admirable structure of our body to be able, in games depending on acuteness of mind or agility of body, to enumerate cases in which one or another of the participants will win?

Since such and similar

things depend upon completely hidden causes, which, besides, by reason of the innumerable variety of combinations will forever escape our efforts to detect them, it would plainly be an insane attempt to get any knowledge in this fashion. However, there is another way to obtain what we want.

And what is impossi­

ble to get a priori, at least can be found a posteriori; that is, by registering the results of observations performed a great many times.

Because it must be pre­

sumed that something may occur or not occur as many times as it had previously been observed to occur or not occur under similar conditions.

For instance, if,

in the past, 300 men of the same age and physical build as Titus is now, were investigated, and it were found that 200 of them had died within a decade, the others continuing to enjoy life past this term, one could pretty safely conclude that there are twice as many cases for Titus to pay his debt to nature within the next decade than to survive beyond this term.

So it is, if somebody for many

preceding years had observed the weather and noticed how many times it was fair or rainy; or if somebody attended games played by two persons a great many times and noticed how often one or the other won; by these very observations he would be able to discover the ratio of cases which in the future might favor the occurrence or failure of the same event under similar circumstances. And this empirical way of determining the number of cases by experiments is neither new nor unusual.

For the author of the book "Ars cogitandi," a man

of great acumen and ingenuity, in Chap. 12 recommends a similar procedure, and everybody does the same in daily practice.

Moreover, it cannot be con­

cealed that for reasoning in this fashion about some event, it is not sufficient to

106

INTRODUCTION TO MATHEMAT !CAL PROBABILITY

[CHAP. VI

make a few experiments, but a great quantity of experiments is required; because even the most stupid ones by some natural instinct and without any previous instruction (which is rather remarkable) know that the more experiments are made, the less is the danger to miss the scope. Although this is naturally known to anyone, the proof based on scientific principles is by no means trivial, and it is our duty now to explain it.

However,

I would consider it a small achievement if I could only prove what everybody knows anyway. There remains something else to be considered, which perhaps nobody has even thought of. Namely, it remains to inquire, whether by thus augmenting the number of experiments the probability of getting a genuine ratio between numbers of cases, in which some event may occur or fail, also augments itself in such a manner as finally to surpass any given degree of certitude; or whether the probleni, so to speak, has its own asymptote; that is, there exists a degree of certitude which never can be surpassed no matter how the observations are multiplied; for instance, that it never is possible to have a probability greater than �. %, or% that the real ratio has been attained.

To illustrate this by an

example, suppose that, without your knowledge, 3,000 white stones and 2,000 black stones are concealed in a certain urn, and you try to discover their numbers by drawing one stone after another (each time putting back the stone drawn before taking the next one, in order not to change the number of stones in the urn) and notice how often a white or a black stone appears. The question is, can you make so many drawings as to make it 10, or 100, or 1,000, etc., times more probable (that is, morally certain) that the ratio of frequencies of white and black stones will be 3 to 2, as is the case with the number of stones in the urn, than any other ratio different from that? If this were not true, I confess nothing would be left of our attempt to explore the number of cases by experiments. But if this can be attained and moral certitude can finally be acquired (how that can be done I shall show in the next chapter), we shall have cases enumerated a posteriori with almost the same confidence as if they were known a priori. that, for practical purposes, where

11

morally certain" is taken for

11

And

absolutely

certain" by Axiom 9, Chap. II, is abundantly sufficient to direct our conjectures in any contingent matter not less scientifically than in games of chance. For if instead of an urn we take the air or the human body, that contain in themselves sources of various changes or diseases as the urn contains stones, we shall be able in the same manner to determine by observations how much more likely one event is to happen than another in these subjects. To avoid misunderstanding, one must bear in mind that the ratio of cases which we want to determine by experiments should not be taken in the sense of a precise and indivisible ratio (for then just the contrary would happen, and the probability of attaining a true ratio would diminish with the increasing number of observations) but as an approximate one; that is, within two limits, which, however, can be taken as near as we wish to each other. For instance, if, in the case of the stones, we take pairs of ratios so�00 and 29%00 or soo�000 and 299%000, etc., it can be shown that it will be more probable than any degree of probability that the ratio found in experiments will fall within these limits than outside of them.

Such, therefore, is the problem which we have decided to

publish here, now that we have struggled with it for about twenty years.

The

SEc. 5]

BERNOULLI'S THEOREM

107

novelty of this problem as well as its great utility, combined with equal difficulty, may add to the weight and value of other parts of this doctrine.-" Ars Conjec­ tandi," pars quarta, Cap. IV, pp. 224-227.

APPLICATION TO GAMES OF CHANCE 5. One of the cases in which the conditions for application of Ber­ noulli's theorem are fulfilled is that of games of chance. It is not out of place to discuss the question of the commercial values of games from the standpoint of Bernoulli's theorem. "Game of chance" is the term we apply to any enterprise which may give us profit or may cause us loss, depending on chance, the probabilities of gain or loss being lmown. The following considerations can be applied, therefore, to more serious questions and not only to games played for pastime or for the sake of gaining money, as in gambling. Suppose that, by the conditions of the game, a player can win a

a of money, with the probability p; or can lose another b with the probability q = 1 -p.

certain sum sum

If this game can be repeated any number of times under the same conditions, the question arises as to the probability for a player to gain or lose a sum of money not below a given limit. the total number of games, and by wins.

Let us denote by

n

m the number of times the player

Considering a loss as a negative gain, his total gain will be

K =ma-(n - m)b. It.is convenient to introduce instead of

m another number a defined by

a=m- np and called "discrepancy."

Expressed in terms of

a the preceding expres­

sion for the gain becomes

K =n(pa-qb) + (a + b)a. The expression

E=pa-qb entering as the coefficient of

n has, as we shall see, an important bearing

on the conclusion as to the commercial value of the game. "mathematical expectation" of the player. expectation is

positive.

discrepancy less than

It is called the

Suppose at first that this

By Bernoulli's theorem the probability for a -nE,

E being an arbitrary positive number, is

smaller than any given number, provided, of course, the number of games is sufficiently large.

At the same time, with the probability approaching

1 as near as we please, we may expect the discrepancy to be � -nE. However, if this is the case, the total gain will surpass the number

n[E- E(a + b)]

INTRODUCTION TO MATHE1l-IA'l'ICAL PROBABILITY

108

[CHAP. VI

which, for sufficiently large n, itself is greater than any specified positive number. It is supposed, of course, that E is small enough to make the difference

E-

(

E a

+b)

positive. And that means that the player whose mathematical expecta­ tion is positive may expect with a probability approaching certainty as near as we please to gain an arbitrarily large amount of money if nothing prevents him from playing a sufficient number of games. On the contrary, by a similar argument, we can see that in case of a negative mathematical expectation, the player has an arbitrarily small probability to escape a loss of an arbitrarily large amount of money, again under the condition that he plays a sufficiently large number of games. Finally, if the mathematical expectation is 0, it is impossible to make any definite statement concerning the gain or loss by the player, except that it is ve�y unlikely that the amount of gain or loss will be considerable compared with the number of games. It follows from this discussion that the game is certainly favorable for the player if his mathematical expectation is positive, and unfavorable if it is negative.

_In case the mathematical expectation is 0, neither

of the parties participating in the game has a decided advantage and then the game is called equitable. equitable.

Usually, games serving as amusements are

On the contrary, all of the games operated for commercial

purposes by individuals or corporations are expressly made to be profita­ ble for the administration; that is, the mathematical expectation of the administration of a game operated for lucrative purposes is positive at each single turn of the game and, correspondingly, the expectation of any gambler is negative.

This confirms the common observation that those

gamblers who extend their gambling over large numbers of games are almost inevitably ruined.

At the same time, the theory agrees with

the fact that great profits are derived by the administrations of gaming places. A good illustration is afforded by the French lottery mentioned on page

19,

which, as is well known, was a very profitable enterprise operated

by the French government.

Now, if we consider the mathematical

expectation of ticket holders in that lottery, we find that it was negative in all cases; namely, denoting by following expectations: On 1 ticket On 2 tickets

M

(U (ttl

On 3 tickets (-f.N!h and so forth.

the sum paid for tickets, we find the

-

-

-

1)M 1)M 1)M

=

=

=

iM, -H-M, -HHM,

-

SEc. 6]

109

BERNOULLI'S THEOREM

On the other hand, the expectation of the administration was always positive, and because of the great number of persons taking part in this lottery, the number of games played by the administration was enormous, and it was assured of a steady and considerable income. This was an enterprise avowedly operated for the purpose of gambling, but the same principles underlie the operations of institutions having great public value, such as insurance companies, which, to secure their income, always reserve certain advantages for themselves. EXPERIMENTAL VERIFICATION OF BERNOULLI'S THEOREM 6. Bernoulli's theorem, like any other mathematical proposition, is a deduction from ideal premises. To what extent these premises may be considered as a good approximation to reality can be decided only by experiments. Several experiments established for the purpose of testing various theoretical statements derived from general propositions of the theory of probability, are reported by different authors. Here we shall discuss those purporting to test Bernoulli's theorem. I. Buffon, the French naturalist of the eighteenth century, tossed a coin 4,040 times and obtained 2,048 heads and 1,992 tails. Assuming that his coin was ideal, we have a probability of .% for either heads or tails. Now, the relative frequencies obtained by his experiments are:

H-Hmt

=

=

0.507 for heads 0.493 for tails

and they differ very little from the corresponding probabilities, 0.500. In this case, the conclusions one might derive from Bernoulli's theorem are verified in a very satisfactory manner. II. De Morgan, in his book "Budget of Paradoxes" (1872), reports the results of four similar experiments. In each of them a coin was tossed 2,048 times and the observed frequencies of heads were, respec­ tively, 1,061, 1,048, 1,017, 1,03.9. The relative frequencies corresponding to these numbers are

HU

=

0.518;

tlli

=

0.512;

tffi

=

0.497;

UH

=

0.507.

The agreement with the theory again is satisfactory. III. Charlier, in his book "Grundziige der mathematischen Statistik," reports the results of 10,000 drawings of one playing card out of a full deck. Each card drawn was returned to the deck before the next draw­ ing. The actual result of these experiments was that black cards appeared 4,933 times, and consequently the frequency of red cards was 5,067. The relative frequencies in this instance are:

NM -frr�

=

=

0.4933 for a black card 0.5067 for a red card

110

INTRODUCTION TO MATHEMATICAL PROBABILITY

and they differ but slightly from the probability, drawn will be black or white.

[CHAP. VI

0.5000, that the. card

The agreement between theory and experi­

ment in this case, too, is satisfactory. IV. The author of this book made the following experiment with playing cards: After excluding the

12 face cards from the pack, 4 cards 40, and the number of trials

were drawn at a time from the remaining was carried to

7,000.

The number of times in each thousand that the

four cards belonged to different suits, was: I

II

III

IV

V

VI

VII

113

113

103

105

105

118

108

Altogether the frequency of such cases was

765 in 7,000 trials, whence

we find for the relative frequency

...['-lh = 0.1093 while the probability for taking

4 cards belonging to different suits is

HH = 0.1094. V. In J. L. Coolidge's "Introduction to Mathematical Probability," one finds a reference to an experiment made by Lieutenant R. S. Hoar, U.S.A., but the reported results are incomplete.

The author of this book

repeated the same experiment which consisted in 1,000 drawings of 5 cards

52 cards. The results were: 503 times the 5 cards were each of diff erent denominations; 436 times 2 were of the same denomination with 3 scattered; 45 times there were 2 pairs of 2 different denominations and 1 odd card; 14 times 3 were of the same denomination with 2 scattered; 2 times there were 2 of one denomination and 3 of another. The remaining possible combination, 4 cards of the same denomination with 1 odd, never appeared. The probabilities of these at a time, from a full pack of

different cases are, respectively,

ttH = 0.507; rlh = 0.021;

HH = 0.423; � = 0.001;

Nh = 0.048; -rrn- = 0.000.

507, 423, 48, 21, 1, 0, 503, 436, 45, 14, 2, 0. The dis­ crepancies are generally small and the greatest of them, 13, is still within The corresponding theoretical frequencies are

while the observed frequencies were reasonable limits.

Deeper investigation shows that the probability that

13 is about �; hence, the observed deviation 13 units cannot be considered abnormal.

a discrepancy will not exceed of

VI. Bancroft H. Brown published, in the American Mathematical Monthly, vol.

26, page 351, the results of a series of 9,900 games of craps.

This game is played with two dice, and the caster wins unconditionally if he produces

7 or 11 points, which are called "naturals"; he loses the

6]

Smc.

111

BERNOULLI'S THEOREM

But if he produces

game in case of 2, 3, or 12 points, called "craps."

4, 5, 6, 8, 9, or 10 "points," he does not win, but has the right to cast the dice an unlimited number of times until he throws the same number of points that he had before, or until he throws a 7.

If he throws 7 before

obtaining his point, he loses the game; ot�erwise he wins. It is a good exercise to find the probability of winning this game. It is

tH that is, a little less than�.

=

0.493

Multiplying the number of games, in our

case 9,900, by this probability, we find that the theoretical number of successes is 4,880 and of failures, 5,020.

Now, according to Bancroft H.

Brown, the actual numbers of successes and losses are, respectively,

4,871 and 5,029.

The discrepancy

4871 - 4880

=

-9

is extremely small, even smaller than could reasonably be expected. The same article gives the number of times "craps" were produced; namely, 2 appeared 259 times, 3 appeared 508 times, and 12 appeared

293 times, making the total number of craps 1,060.

The probability

of obtaining craps is

hence, the theoretical number of craps should be 1,100.

1060 - 1100

=

The discrepancy,

-40, is more considerable this time. but still lies within

reasonable limits. VII. E. Czuber made a complete investigation of lotteries operated on the same plan as the French lottery, in Prague between 1754 and 1886, and in Briinn between 1771 and 1886.

2,854 in Prague and 2,703 in BrUnn.

The number of drawings was

The probability that in each draw­

ing the sequence of numbers is either increasing or decreasing, is

-h

=

0.01667

while the observed relative frequency of such cases was Prague: 0.01612; BrUnn: 0.01739 and in both places combined

0.01674. The probabilities that among five numbers in each drawing there is none or only one of the numbers 1, 2, 3, . . . 9, are, respectively,

0.58298 and 0.34070.

112

INTRODUCTION TO MATHEMATICAL PROBABILITY

[CHAP. VI

The corresponding relative frequencies were Prague: 0.58655 and 0.32656 Brunn: 0.57899 and 0.34591 and in both places combined 0.58183 and 0.33587, respectively.

The probability of drawing a determined number is 7f8.

Now, according

to Czuber, for the lottery in Prague the actual number of occurrences for single tickets varied from 138 (for No. 6) to 189 (for No. 83), so that for all tickets the discrepancy varied from -20 to 31.

Besides, there were

only 16 numbers with a discrepancy greater than 15 in absolute value. All these results stand in good accord with the theory. VIII. One of the most striking experimental tests of Bernoulli's theorem was made in connection with a problem considered for the first time by Buffon.

A board is ruled with a series of equidistant parallel

lines, and a very fine needle, which is shorter than the distance between lines, is thrown at random on the board.

Denoting by l the length of

the needle and by h the distance between lines, the probability that the needle will intersect one of the lines (the. other possibility is that the

needle will be completely contained within the strip between two lines) is found to be

p

=

2l 1T'h

-·

The remarkable thing about this expression is that it contains the number 1T'

=

3.14159

·

circle to its diameter.

·

·

expressing the ratio of the circumference of a

In the appendix we shall indicate how this expres­

sion can be obtained, because in this problem we deal with a different concept of probability. Suppose we throw the needle a great many times and count the number of times it cuts the lines.

By Bernoulli's theorem we may expect

that the relative frequency of intersections will not differ greatly from the theoretical probability, so that, equating them, we have the means of finding an approximate value of

'IT'.

One series of experiments of this kind was performed by R. Wolf,

astronomer in Zurich, between 1849 and 1853.

In his experiments the

width of the strips was 45 mm., and the length of the needle was 36 mm. Thus the theoretical probability of intersections is 72

451T'

=

0.5093.

SEc. 6]

113

BERNOULLI'S THEOREM

The needle was thrown 5,000 times and it cut the lines 2,532 times; whence, the relative frequency

·

un

=

o.5o64

.

The agreement between the two numbers is very satisfactory. If, relying on Bernoulli's theorem, we set the approximate equation 72 457r

=

0.5064,

we should find the number 3.1596 for r, which differs from the known value of

1r

by less than 0.02.

In another experiment of the same kind reported by De Morgan in the aforementioned book, Ambrose Smith in 1855 made 3,204 trials with a needle the length of which was % of the distance between lines.

There

were 1,213 clear intersections, and 11 contacts on which it was difficult to decide.

If on this ground, we should consider half of them as inter­

sections, we should obtain about 1,218 intersections in 3,204 trials, which would give the number 3.155 for 1r.

If all of the contacts had been treated

as intersections the result would have been 3.1412--very close to the real value of r. In an excellent book "Calcolo delle Probabilita," vol. 1, page 183, 1925, by G. Castelnuovo, reference is made to experiments performed by Professor Reina under whose direction a needle of 3 em. in length was thrown 2,520 times, the distance between lines being 6 em.

Taking into

account the thickness of the needle, the probability of intersection was found to be 0.345, while actual experiments gave the relative frequency of intersections as 0.341.

APPENDIX Bufton's Needle Problem.

Let h be the width of the strip between

two lines and l < h the length of the needle.

The position of the needle

can be determined by the distance x of its middle point from the nearest line and the acute angle rp formed by the needle and a perpendicular dropped from the middle point to the line. It is apparent that

x

may

vary from 0 to h/2 and rp varies within the limits 0 and r/2. We cannot define in the usual way the probability of the needle cutting the line, for there are infinitely many cases with respect to the position of the needle. However, it is possible to treat this problem as the limiting case of another problem with a finite number of possible cases, where the usual definition of probability can be applied. Suppose that h/2 is divided into an arbitrary number m of equal parts 8

=

h/2m and the right angle

1r/2

into

n

equal parts

w = 1r/2n.

Suppose, further, that the distance x may have only the values

o, a, 28,_ .. . m8

114

INTRODUCTION TO MATHEMATICAL PROBABILITY

[CHAP. VI

and the angle If' the values

0, w, 2w, . . . nw. This gives

N

(m + 1){n + 1)

=

cases as to the position of the needle, and it is reasonable to assume that these cases are equally likely.

To find the number of favorable cases, we

notice that the needle cuts one of the lines if

x

and If' satisfy the inequality

l X< 2 COS I(J. The number of favorable cases therefore, is equal to the number of systems of integers i,

j

satisfying the inequality

. l . �a < 2 cos JCiJ

'

(A)

supposing that i may assume only the values 0, 1, 2, . . . m and the values 0,

j

only

Because we suppose l < h the greatest

1, 2, . . . n.

value of i satisfying condition

(A) is less than m and we can disregard �m. Now for given j there are k + 1

the requirement that i should be

values of i satisfying (A) if k denotes the greatest integer which is less than

l

6 2

.

COS JW.

In other words, k is an integer determined by the conditions

k
u

�T m=O

twice with respect to

u and set u = 0.

2. If Q is the probability of the inequality l m - npl !5;; nE prove that

Indication of the Proof.

In the identity

n (m

� Tm

- np)2 = npq

m=O drop all the terms in which l m

- npl < nE and in the remaining terms replace (m- np)2

by

n2E2•

The resulting inequality

lm-npl�n• is equivalent to the statement. 3. Prove that p > 1 -"

if

n

-

> pq/'IE2.

Q, Q < pq/nE2 and pq/nf2 < 11 if n > pq/11E2, Indication of the Proof. P = 1 The following two problems show how probability considerations can be used in proving purely analytical propositions. 4. S. Bernstein's Proof of Weierstra.ss' Theorem.

The famous theorem due to Weier­

strass states that for any continuous functionf(x) in a closed interval a � x � b there exists a polynomial P(x) such that lf(x) - P (x)l < for

a

�

x

� b where

u

u

is a:1 arbitrary positive number.

By a proper linear trans­

formation the interval (a, b) can be transformed into the interval

to 8. Bernstein, the polynomial n

P (x) =

� C;:'xm(1 m=O

-m (;)

x)n

f

(0, 1).

According

117

BERNOULLI'S THEOREM for sufficiently large n satisfies the inequality

lf(x) uniformly in the interval 0 � x � Indication of the Proof. For x

P(x) l < ,

1. 0 and

=

/( 1 )

=

It suffices to prove the statement for 0 < n

independent trials.

z =

1 we have /tO)

x

< 1.

Let

x. be

a constant probability in

We have

f(x) - P(x)

=

�

C:'xm(1 -

[

x)"-m J(x) -

m=O

1(;)}

By the property of continuous functions, there is a number positive number

tr

P(O) and

P(1).

..

(a)

=

E

corresponding to any

such that

1/(x') - f(x)l




_pqtp2 - !(4pq)2 2 4

sin' I£!. >

2

_pqiP2 _ p2q21P' 2 4

�}

2

>0

SEC. 5]

1"frp

APPROXIMATE EVALUATION 0/t' PROBABILITIES

11" < 2" =

123

Hence,

(6) � 11"/2. 4. Let r be defined by

and this is valid forrp

r2 = 3B;t.

Assuming B,. $1; 25 from now on, we shall have,

�!

T2

and a fortiori r
;

Let

.

r1

r �'

=

i,..R-d

.)

( IT 1 +

�

lc=O

[ :>.) :>.-lc � n-:>.

-k n - X

) <e�c-o

x

(:>.+!>"

� e2(n-:>.>.

Consequently

m X"' Pm < 1- � ei����; 1 + �1 + � 1·2 + ·· · + 1·2·3···m

( )

[

..

n

But

},. "

whence

Pm <eXQmi

Qm

=

e

<e

1

}. -:>.-2n,

( ) - [ 1+�1+ 1x•· 2 + 1-;

:>.

•

.

On the other hand, ..

1

- Pm

whence and

n(n =

,.-m+l

- 1)

•

•

•

"'

(n - 1£

+ 1)

]·

q"-"P"

.

xm + · ·3 12 · · ·m

]·

=

1 Pm <eX(1 - Qm) eX. P m > eXQm + 1

-

-

The final statement follows immediately from both inequalities obtained for P ...

137

APPROXIMATE EVALUATION OF PROBABILITIES 10. With the usual notation, show that

where �

Q

6 n

=

[ (

(n-m)A•_m(m-1) 2n• 2n 1

Indication of the Proof.

_

(J

m)).l E MoiVRE: "Miscellanea analytica," Lib. V, 1730. DE MoiVRE: "Doctrine of Chances," 3d ed., 1756.

LAPLAcE: "Theorie analytique des probabilit�," Oeuvres VII, pp. 28G-285. Ca. DE

LA

VALLEE-PoussiN: Demonstration nouvelle du theoreme de Bernoulli,

Ann. Soc. Sci: Bruxelles, 31. D. MIRIIIIANOFF: Le jeu de pile ou face et les formules de Laplace et de J. Eggenberger, Oommentarii Mathematici Helvetici 2, pp. 133-168.

CHAPTER VIII . FURTHER CONSIDERATIONS ON GAMES OF CHANCE 1. When a person undertakes to play a very large number of games " under theoretically identical conditions, the inference to be drawn from Bernoulli's theorem is that that person will almost certainly be ruined if the mathematical expectation of his gain in a single game is negative. In case of a positive expectation, on the other hand, he is very likely to win as large a sum as he likes in a sufficiently long series of games. Finally, in an equitable game when the mathematical expectation of a gain is zero, the only inference to be drawn from Bernoulli's theorem is that his gain or loss will likely be small in comparison with the number of games played. These conclusions are appropriate however, only if it is possible to continue the series of games indefinitely, with an agreement to postpone the final settling of accounts until the end of the series.

But if the

settlement, as in ordinary gambling, is made at the end of each game, it may happen that even playing a profitable game one will lose all his money and will have to discontinue playing long before the number of games becomes large enough to enable him to realize the advantages which continuation of the games would bring to him. A whole series of new problems arises in this connection, known as problems on the duration of play or ruin of gamblers.

Since the science

of probability had its humble origin in computing chances of players in different games, the important question of the ruin of gamblers was discussed at a very early stage in the historical development of the theory of probability.

The simplest problem of this kind was solved by

·Huygens, who in this field had such great successors as de Moivre, Lagrange, and Laplace. 2. It is natural to attack the problem first in its simplest aspect, and then to proceed to more involved and difficult questions. Problem 1.

Two players A and B play a series of games, the proba­

bility of winning a single game being p for A and q for B, and each game ends with a loss for one of them.

If the loser after each game gives his

adversary an amount representing a unit of money and the fortunes of A and B are measured by the whole numbers a and b, what is the proba­ bility that A (or B) will be ruined if no limit is set for the number of games? 139

140

INTRODUCTION TO MATHEMATICAL PROBABILITY

Solution.

[CHAP. VIII

It is necessary first to show how we can attach a definite

numerical value to the probability of the ruin of A if no limit is set for the number of games. As in many similar cases (see, for instance, Prob. 15, page 41) we start by supposing that a limit is set. Let n be this limit.

There is only a finite number of mutually exclusive ways in which

A can be ruined in n games; either he can be ruined just after the first game, or just after the second, and so on. Denoting by p1, p2, . . . p,. the probabilities for A to be ruined just after the first, second, . . . nth game, the probability of his ruin before or at the nth game is P1+ P2 + · ·

·

+ p,..

Now, this sum being a probability, must remain , Ya+1>-1,

is negative, then Proof.

Let

that the player

a+ b

:..__

•

•

•

Ya+l>-fJ+1

!5;:; 0 for x = O, 1, 2, ... a + b. u�"'> (k = 0, 1, 2, . . . a - 1) represent the probability A whose actual fortune is x (and that of his adversary

y,.

x) will be forced to quit when his fortune becomes exactly k. u�"'> is a solution of equation (4) satisfying the conditions

Evidently

=

SEc. 3)

FURTHER CONSIDERATIONS ON GAMES OF CHANCE

n�k> = 0 for a+ b- 1,

x = 0, 1, ... k- 1, k+ 1, uf•> a+ b - fJ+ 1; 1.

. . . a- 1;

145

a+ b,

==

Similarly, if v�0(l = 0, 1, 2, . . . fJ

1) represents the probability that

-

the player B will be forced to quit when the fortune of A becomes exactly =

a+ b - l, v�1> will be a solution of (4) satisfying the conditions

x = 0, 1, 2, . . . a for v�!> = 0 a+ b - l- 1, ... a+ b- fJ + 1;

-

1; a+ b, ... a+ b - l+ 1, v��-� = 1.

Thus we get a+ fJparticular solutions of (4), and it is almost evident that these solutions are independent.

Moreover, since they represent

probabilities, u�"> � 0, v�n � 0 for x = 0, 1, 2, ... a + b.

Now, any

solution y., of (4) with given values of

Ya-1 Ya+b-/3+1

Yo, Y1, . Ya+b> Ya+b-1,

•

can be represented thus

y.,

=

/3-1 a -1 �y�r,u�k> + �Ya+b-!V�!>. k=O

l=O

Hence, y., � 0 for x = 0, 1, 2, . ..a+ b if none of the numbers

Yo, Y1, . . . Ya-1 Ya+b> Ya+b-1, Ya+b-/3+1 •

is negative.

•

•

This interesting property of the solutions of equation (4)

derived almost intuitively from the consideration of probabilities can be established directly. (See Prob. 9, page 160.) The lemma just proved yields almost immediately the following proposition: If for any two solutions y� and y�' of equation (4) the inequality holds for X = 0, 1, 2,

a - 1;

a+ b, a+ b- 1, ... a+ b - fJ + 1,

the same inequality will be true for all x = 0, 1, 2, ... a+ b.

It suffices to notice that y., = y�' - y� is a solution of the linear equation (4) and, by hypothesis, y., � 0 for a+ b - 1, ... a+ b - fJ+ 1.

x = 0,

Now we can come back to our problem. expectation of A

pfJ-

qa

1, 2, . . . a- 1; a+ b, First, if the mathematical

1 46

INTRODUCTION TO MATHEMATICAL PROBABILITY

is different from 0, equation (7) has two positive roots:

1

[CHAP. VIII

and 8.

With

arbitrary constants C and D

y�

=

C + D8"

is a solution of (4). Whatever C and D may be, y� as a function of x varies monotonically. Therefore, if C and D are determined by the conditions

y�

=

1,

Y�-tb-11+1

0

=

we shall have

y� � 1 y� � 0

0, 1, 2, . . . a - 1 + b-{3+ 1, . . . a + b

if

x =

if

x = a

and by the above established lemma, taking into account conditions (5) and

(6), we shall have for the required probability the following inequality

or, substituting the explicit expression for

y�,

8a+b-JI+l - 8"' . Y .., = > 8a+b-J1+1 1 _

If, on the contrary, C and Dare determined by Y�-1 ,;

1,

we shall have

y� G 1 y� G 0

if

X

if

x =

and

Finally, taking

x = a,

0, 1, 2, . . . a - 1 a + b-{3+ 1, . a + b

8a+b-a+1 - 8z-a+1




1,

we

+ tp(fJ)a'"']fJ1dfJ.

c

Here the integration is made along a circle of sufficiently large radius and

f(B)

and

tp(B)

are two unknown functions which can be developed into

series of descending powers of {J. in

x

and t.

For

x

=

Obviously z,.,, satisfies

(13)

identically

0 and t � 0 we have the condition

�i [f(B)

+ tp(B)]B1dfJ

=

1;

t

o,

=

1, 2, .

which is satisfied if

J(fJ) + tp(B)

(17) Condition

(15) will

=

1 B-1

--·

be satisfied if

aa+bJ(B) + a'a+biP(B)

(18)

=

and it remains to show that at the same time

(17)

and

{18),

0

(16)

we have

/(fJ) tp(fJ)

=

=

is satisfied.

Solving

a'a+b 1 a+b - ()(.a-rb . fJ - 1 a' -aa+b . 1 a+b - aa+b fJ - 1 a'

and

(19) /(fJ)a"' + tp({J)a'"'

=

a'a+ba"' - aa+ba'"' aa+b) (fJ 1)(a'a+b _

_

=

Now let

a

(q)"'

fJ.

Evidently x

=

_

aa+b-:�:

a' the other root whose fJ starts with the term the development of (19) for 1, 2, 3, . . . a + b - 1

be the root vanishing for

fJ

=

oo

and

development in series of descending powers of containing

a'a+b-:o

P (fJ - 1)(a'a+b - aa+b)'

INTRODUCTION TO MATHEMATICAL PROBABILITY

156

[CHAP. VIII

... a+

does not contain terms involving the first power of 1/fJ, and hence b -1 as it should be. The solution 1, 2, 3, o if x z.,,o being (16) (15), (14), ying unique, its analytical expression is tisf sa 13) of ( =

=

therefore

whence for x =

a

Zs,t

- (�ria'IJa'+b'*-s" - atJ+b-s 2-n-i

and t

- all+l>

c_

fJ'dfJ fJ- 1'

n

=

To find an explicit expression for z,.,,. it remains to find the coefficient of 1/fJ in the development of

P

(p) a',.+ b - a11+1> a' q

=

,.

a'"

in series of descending powers of fl. ways. First we can substitute for

and present P in the form

P=

q

1or developing into series

P=

[a�� (�)"a��+�"+ (�Y+��aa..

H"

_

But the coefficient of 1/fJ in

a:

This can be done in two different its expression in

(!!.)" a,.+2b (qp)IJ+b a211+2b

all-

fJ" fJ-1

a"

_

fJ" fJ- 1

__,

(�)'*2"aa..

H" +

amfJn

fJ-1 by the second solution of Prob. 3 is the probability a fortune games. z,.,,.

=

m

fJ-1

for a player with

to be ruined by an infinitely rich player in the course of

Hence, the final expression for

y,.,,.

Ym,n

... ]1-.

" q Y��-t2b,n - (p)

+

is

(p)a+b Y3<J-t2b,n (qp)"+2bYa��-t4b,n q Za,,.

-

+

•

•

•

1

n

SEc.

7)

FURTHER CONSIDERATIONS ON GAMES OF CHANCE

157

the terms of this series being alternately of the form

() � () -

and

p

ka+kb Y =

L

Sill 'P Sill

r

a by

p > q. an even number, then

i sin a

0; ak; � 0 if k ;o!E i; au + a1; + (2) One of these sums is positive.

·

·

·

+ a,.; � 0.

If these forms assume nonnegative values, then every Proof by induction: Express x..

x,.

through

x,,

f,. - a,,.x, - a2nX1 =

.

. x,._,

� O(i

-

·

·

·

=

1, 2,

•

.

.

n).

- a,._1,,.:r;,._1

a,.,.

and substitute into the remaining forms. .

x;

x1, ... x,._,, thus:

satisfy the same conditions

Show that the resulting forms in x,, x2, Hence, it remains to prove the

(1) and (2).

proposition for two forms, which can easily be done.

References

CHR. HuYGENs: "De ratiociniis in ludo aleae," 1654. DE MoiVRE: "Doctrine of Chances," 3d ed., 1756. LAGRANGE: M6moire sur les suites r6currentes dont les termes varient de plusieurs manieres diff6rentes, etc., Oeuvres IV, pp. 151ff. LAPLAcE: "Thoorie analytique des probabilit6s," Oeuvres VII, pp. 228-242. BERTRAND: "Calcul des probabilit6s," pp. 104-141, Paris, 1889. MARKOFF: "Wahrscheinlichkeitsrechnung," pp. 142--146, Leipzig, 1912. 1

The author is indebted to Professor Besikovitch of Cambridge, England, for the

communication of this direct proof.

CHAPTER IX MATHEMATICAL EXPECTATION 1. Bernoulli's theorem, important though it is, is but the first link

in a chain of theorems of the same character, all contained in an extremely general proposition with which we shall deal in the next chapter.

But

before· proceeding to this task, it is necessary to extend the definition of "mathematical

expectation"-an

important

concept

originating

in

connection with games of chance. If, according to the conditions of the game, the player can win a sum

a with

probability

p,

and lose a sum b with probability q

=

1

-

p,

the mathematical expectation of his gain is by definition

pa-

qb.

Considering the loss as a negative gain, we may say that the gain of the player may have only two values,

a

and -b, with the corresponding

probabilities p and q, so that the expectation of his gain is the sum of the products of two possible values of the gain by their probabilities.

In this

case, the gain appears as a variable quantity possessing two values. Variable quantities with a definite range of values each one of which, depending on chance, can be attained with a definite probability, are called "chance variables," or, using a Greek term, "stochastic:' variables. They play an important part in the theory of probability. variable is defined

(a)

A stochastic

if the set of its possible values is given, and (b) if

the probability to attain each particular value is also given. It is easy to give examples of stochastic variables. game of chance is a stochastic variable with two values.

The gain in a The number of

points on a die that is tossed, is a stochastic variable with six values, 1, 2, . . . 6, each of which has the same probability 76. A number on a ticket drawn from an urn containing 20 tickets numbered from 1 to 20, is a stochastic variable with 20 values, and the probability to attain any one of them is balls.

72o.

Each of two urns contains 2 white and 2 black

Simultaneously, one ball is transferred from the first urn into the

second, while one ball from the latter is transferred into the first.

·

After

this exchange, the number of white balls in one of the urns may be regarded as a stochastic variable with three values, 1, 2, 3, whose corresponding probabilities are, respectively, �.

72,

�.

It is natural to extend the

concept of mathematical expectation to stochastic variables in general. 161

162

INTRODUCTION TO MATHEMATICAL PROBABILITY

x possesses n values:

Suppose that a stochastic variable

and

[CHAP. IX

P1, P2, . .Pn ·

denote the respective probabilities for x to assume values x1, x2, By definition the mathematical expectation of x is

E(x)

=

P1X1 + P2X2 +

·

·

·

.

.

•

Xn.

+ PnXn.

It is understood in this definition that the possible values of the variable

x are nume1·ically different. For instance, if the variable is a

number of points on a die, its numerically different values are 1, 2, 3, 4,

5,

6, each having the same probability,.%'. By definition, the mathematical expectation of the number of points on a die is H1 + 2

+ 3 + 4 + 5 + 6)

=

3.5.

If the variable is the number on a ticket drawn from an urn containing 20 tickets numbered from 1 to 20, its numerically different values are represented by numbers from 1 to 20, and the probability of each of these values is �o, so that the mathematical expectation of the number on a ticket is io-(1

+ 2 +

.

. . + 20)

=

10.5.

2. It is obvious that the computation of mathematical expectation requires only the knowledge of the numerically different values of the variables with their respective probabilities. But in some cases this computation is greatly simplified by extending the definition of mathe­ matical expectation. Suppose that, corresponding to mutually exclusive

A1, A2, . . . Am, the variable x assumes the values x1, x2, ...Xm, with the corresponding probabilities p1, p2, ... Pmi we can define the mathematical expectation of x by

and exhaustive cases

E(x)

=

P1X1 + P2X2 +

·

'

' + PmXm.

What distinguishes this extended definition from the original one is that in the second definition the values

x1, x2, . . . x,. need not be numerically

different; the only condition is that they are determined by mutually exclusive and exhaustive cases. To make this distinction clear, suppose that the variable

x is the

number of points on two dice. Numerically different values of this variable are

2, 3, 4,

, 5 6, 7, 8, 9, 10, 11, 12

and their respective probabilities

n.�.�.n.�.�.�.n.�.�.n·

SEC.

2]

163

MATHEMATICAL EXPECTATION

Therefore, by original definition, the expectation of xis

-a\ + /u- + H + i* + H + H + H + « + H + H + H

=

W = 7.

But we can distinguish 36 exhaustive and mutually exclusive cases accord­ ing to the number of points on each die and, correspondingly,

36 values

of the variable x, as shown in the following table:

First die

Second die

1 1 1. 1 1 1

1 2

1 2 3 4 5 6

1 2

3 3 3 3 3 3

Second die

3 4 5 6 7

4 4 4 4 4 4

1 2

3 4 5 6 7 8

5 5 5 5 5 5

4 5 6 7 8 9

6 6 6 6 6 6

2

3 4 5 6

2 2 2 2 2 2

First die

X

3 4 5 6

The probability of each of these

X

5 6 7 8 9

3 4 5 6

10

1 2

6 7 8 9

3 4 5 6

10 11

1 2

7 8 9

3 4 5 6

10 11 12

36 cases being .73s, by the extended

definition the mathematical expectation of x is

2 + 2. 3 + 4 . 3 + 5 . 4 + 6 5 + 7 . 6 + 8 . 5 + 9 4 + 10 . 3 + 11 . 2 + 12 36 =7 .

.

as it should be. It is important to show that both definitions always give the same value for the mathematical expectation. Let x1, x2,

•

•

•

Xm be the values of the variable

x corresponding

to mutually exclusive and exhaustive cases A1, A2, Am, and, p1, p2, . . . Pm, their respective probabiltties. By the extended defini­ •

tion of mathematical expectation, we have

(1)

•

•

164

[CHAP. IX

INTRODUCTION TO MATHEMATICAL PROBABILITY

The values x1, x2, . . . Xm are not necessarily numerically different, the numerically different values being

�. 71, s, . . . X. We can suppose that the notation is chosen in such a way that Xt, x2, ... Xa are equal to �; . . Xb are equal to 71; Xa+l , Xa+2, Xb+1, Xb+2,

. Xc are equal to

•

. . x,, are equal to X.

xl+l, X1+2,

Hence, the right-hand member of

(p1 +

P2

+

·

·

·

+pa)� +

s;

(Pa+l

(1)

+

can be represented thus:

Pa+2

+ + Pb)71 + + (Pl+t + Pl+2 + ·

·

·

·

·

·

·

·

·

+ + Pm)X.

But by the theorem of total probabilities, the sum Pt

represents the probability

+

P

P2

+

·

·

·

+

Pa

for the variable x to assume a determined

value �' because this can happen in a mutually exclusive ways; namely, when x = x1, or x = x2,

.

. . or x = Xa.

By a similar argument we see

that the sums Pa+l Pb+l Pl+l

represent the probabilities values 71, s, . . . X.

+ Pa+2 + + Pb+2 +

+ Pb + P•

+ Pl+2 + Q, R, . . T

+

P•n

for the variable x to assume

.

Therefore, the right-hand member of

(1)

reduces

to the sum

P� + Q71 + Rs +

·

·

·

+ T'A.

which, by the original definition, is the mathematical expectation of x. If,

corresponding to mutually exclusive and exhaustive cases, a

variable x assumes the same value a-in other words, remains constant­ it is almost evident that its mathematical expectation is a, because the sum of the probabilities of mutually exclusive and exhaustive cases is 1. It is also evident that the expectation of ax where a is a constant, is equal to a times the expectation of x. NOTE: Very often the mathematical expectation of a stochastic variable is called its "mean value."

MATHEMATICAL ExPECTATION OF A SuM 3. In many cases the computation of mathematical expectation is greatly facilitated by means of the following very general theorem:

SEc. 3]

MATHEMATICAL EXPECTATION

165

The mathematical expectation of the sum of several variables

Theorem.

is equal to the sum of their expectations; or, in symbols,

E(x +

y

+z+

·

·

·

+ w)

=

E(x ) + E(y ) + E(z) +

·

·

·

+ E(w).

We shall prove this theorem first in the case of a sum of two Let x assume numerically different values x1, x2, Xm, while numerically different values of y are Y1, Y2, . . . Yn· In regard to

Proof.

variables.

•

•

•

x + y we can distinguish mn mutually exclusive cases; namely, when x assumes a definite value x, and y another definite value y1, while i andj range respectively over numbers 1, 2, 3, ... m and 1, 2, 3, . . . n.

the sum

If

Pii denotes

the probability of coexistence of the equalities

Y

=

Yi

we have by the extended definition of mathematical expectation n

m

E(x +

y)

=

� �Pii(x, + Yi), i=li-l

or y) =

m

n

m

E(x +

(2)

n

� � PifXi + � � Pi!Yi·

i=li=l

i=li-l

As the variable x assumes a definite value x, in n mutually exclusive ways (namely, when the value

Yh Ys, . . . Yn

Xi

of x is accompanied by the values

of y) it is obvious that the sum n

�Poi i=l

represents the probability

Pi of the equality x

=

x ,.

In a similar manner

we see that 'the sum

represents the probability qi of the equality y m

n

� �PiiXi

m =

i=1j=1

n

�X; �Pii

=

Yi·

Therefore

m =

i=1 i=l

.�PiXi

=

E(x),

i=l

and similarly m

n

� �PiiYi i-li=1

n. =

m

� �P•iYi i-1i-1

n. =

�qiYi i-1

=

E(y ) ;

INTRODUCT ION TO MATHEMATICAL PROBABILITY

166

that is, by

[CHAP.

IX

(2) E(x + y)

E(x) + E(y)

=

.

which proves the theorem for the sum of two variables.

x + y + z, we may consider

If we deal with the sum of three variables it at first as the sum of

x +y

and

z

and, applying the foregoing result,

we get

E(x + y + z)

=

E(x + y) + E(z) ;

E(x) + E(y)

and again, by substituting

E(x + y + z)

=

for

E(x + y),

E(x) + E(y) + E(z).

In a similar way we may proceed farther and prove the theorem for the sum of any num her of variables.

4. The theorem concerning mathematical expectation of sums,

simple though it is, is of fundamental importance on account of its very general nature and will be used frequently.

At present, we shall use it

in the solution of a few selected problems.

Problem 1.

What is the mathematical expectation of the sum of

points onn dice?

Solution.

Denoting by

Xi

the number of points on the ith die, the

sum of the points onn dice will be 8

X1 + X2 +

=

·

+ Xn,

·

and by the preceding theorem

E(s)

=

E(x1) + E(x2) +

·

+ E(xn).

·

·

But for every single die

E(xi)

=

i

t;

=

therefore

E(s) Problem 2.

=

1, 2, ... n; 7n

2.

What is the mathematical expectation of the number of

successes inn trials with constant probability p?

Solution.

Suppose that we attach to every trial a variable which

has the value 1 in case of a success and the value 0 in case of failure.

the variables attached to trials 1, Xn,

their sum

m

=

2, 3, .. .

X1 + X2 +

·

n

·

are denoted by

+ Xn

·

obviously gives the number of successes in n trials. required expectation is

E(m)

=

E(x1) + E(x2) +

·

If

x1, x2,

·

·

+ E(xn).

Therefore, the

Smc.

4]

167

MATHEMATWAL EXPECTATION

But for every i

=

1,

2, 3,

E(x;) because

x,

. . n

.

p 1 + (1 - p)

=

·

·

may have values

1

0

=

p,

and 0 with the probabilities

p

and

1

-

p

which are the same as the probabilities of a success or a failure in the ith trial.

Hence,

E(m)

np

=

or

E(m- np)

=

0,

which may also be written in the form n

� Tm(m- np)

0.

=

m=O

116

This result was obtained on page complicated way.

in a totally different and more

The new deduction is preferable in that it is more

elementary and can easily be extended to more complicated cases, as we shall see in the next problem.

Problem 3.

P• in

known about the results of other trials. expectation of the number of successes

Solution.

Xi

=

independent or

the ith trial when nothing is What is the mathematical

m in n

trials?

Xi connected with 1 when the trial results in a success

Again let us introduce the variable

the ith trial in such a way that and

n trials

Suppose that we have a series of

not, the probability of an event being

x,

=

0 when it results in failure.

m

=

Obviously,

X1 + X2 +

'

•

+ Xn

•

and

E(m)

=

E(xt) + E(x2) +

·

·

·

+

E(x,.).

But

E(xi)

=

1

·

Pi

+ 0

·

(1 - Pi)

=

Pi

and therefore

E(m)

=

Pt + P2 +

·

·

·

+

p,..

For instance, if we have 5 urns containing 1 white, 9 black; 2 white, 8 black; 3 white, 7 black; 4 white, 6 black; 5 white, 5 black balls, and we draw one ball out of every urn, the mathematical expectation of the number of white balls taken will be:

E(m) Problem 4. drawn.

=

io- + 1� + -h + -h + -h

=

·1.5.

An urn contains a white and b black balls, and c balls are

What is the mathematical expectation of the number of the

white balls drawn?

[CHAP.

INTRODUCTION TO MATHEMATICAL PROBABILITY

168

IX

To every ball taken we attach a variable which has the if the extracted ball is white, and the value 0 otherwise. The number of white balls drawn will then be

Solution.

value

1

8 = X1 + x2 +

·

·

·

+ x•.

But the probability that the ith ball removed will be white when nothing is known of the other balls is a

� b; therefore

a b 1 +a E(x,) = a 0 = +b +b ·

·

a a

+ b

for every i, and the required expectation is ca

E(8) =a ' +b Problem 5.

An urn contains

m tickets are drawn at a time.

n

tickets numbered from

1

to

n,

and

What is the mathematical expectation

of the sum of numbers on the tickets dr�wn?

Solution.

Suppose that m tickets drawn from the urn are disposed

in a certain order, and a variable is attached to every ticket expressing its number.

Denoting the variable attached to the ith ticket by

x,,

the sum of the numbers on all m tickets apparently is

8 = X1 +X2 + ' ' ' + Xm. But when taken singly, the variable

1, 2, 3, . . . n, the numbers being 1/n.

x, may represent any of

the numbers

probability of its being equal to any one of these By the definition of mathematical expectation, we

have

E(x,) =

1+2+3+ n

·

·

·

+n

=

n+1 --

2

,

and therefore

E(8) =

m (n+1).

2

For example, taking the French lottery where

n = 90

and m

""

5, we

find for the mathematical expectation of the sum of numbers on all

5

tickets

5. 91 E(8) = -- = 227.5. 2 Problem 6.

An urn contains n tickets numbered from 1 to n.

These

tickets are drawn one by one, so that a certain number appears in the first place, another number in the second place, and so on.

We shall say

SEc. 4]

MATHEMATICAL EXPECTATION

I69

that there is a "coincidence" when the number on a ticket corresponds to the place it occupies. For instance, there is a coincidence when the first ticket has number I or the second ticket has number 2, etc. Find . the mathematical expectation of the number of coincidences. Also, find the probability that there will be none, or one, or two, etc., coincidences.

Solution.

Let

x,

denote a variable which has the value I if there is

coincidence in the ith place, otherwise 8 =

x,

0.

=

The sum

X1 + X2 + . . ' + Xn

gives the total number of coincidences and

But

E(x ') ·

=

I

-

I

I

·

n

=

-

n

because the probability of drawing a ticket with the number i in the ith place without any regard to other tickets obviously is I/n; therefore,

E(s)

n

=

!

·

n

1.

=

On the other hand, denoting the probability of exactly i coincidences by

p,, we have by definition

E(s)

=

P1 + 2p2 +

·

·

·

+ npn,

and, comparing with the preceding result, we obtain

(3)

P1 + 2p2 +

·

·

+ npn

·

=

1.

Let us denote by cp(n) the probability that in drawing n tickets, we shall It is easy to express P• by means of cp(n - i).

have no coincidences.

In fact, we have exactly i coincidences in

Cin

(n - i + I) n(n - I) - ---'-----=I..:..·-;2-· " -- 3 -.-'--. -.-'1..,-. ---'-----'_

·

·

·

mutually exclusive cases; namely, when the tickets of one of the c� specified groups of i tickets have numbers corresponding to their places while the remaining n - i tickets do not present coincidences at all.

By the theorem of compound probability, the probability of i coincidences in i specified places is I

I

1 n-i+I

INTRODUCTION TO MATHEMATICAL PROBABILITY

170

[CHAP. IX

and the probability of the absence of coincidences in the remaining n - i places is lfJ(n - i). The probability of exactly i coincidences in i specified

places is therefore

lfJ(n- i) n(n- 1)· · · (n - i + 1)' and the total probability P• of exactly of places is P•

=

or

i coincidences without specification

n(n - 1)· · · (n - i + 1) lfJ(n - i) · n(n- 1)· · · (n - i + 1)' 1 ·2 3 ··· i ·

(4)

P•

=

lfJ(n - i) . - -=2�. -=a- -___,i =-1· . ....!. .

lfJ(O) has no meaning, but the preceding formula holds i = n if we assume lfJ(O) 1. Substituting expression (4) for P• into (3), we reach the relation

The symbol good even for

lfJ(n

_

or changing

=

1) + n

lfJ(n - 3) + lfJ(n - 2) + 21 1!

.. +

(O) np

the sum being extended over all integers m which are >np.

Denoting by F(x, y) the

sum

F(x, y)=

�

cr;:xmyn-m

m>np

we have

�

aF(p, q) Tm(m- np)=p---- npF(p, q). ap

m>np

On the other hand, by Euler's theorem on homogeneous functions

aF aF nF(p, q)=p-;:- + q-• aq ap whence

m>np

Here 1-' represents an integer determined by

'-' ;;; np + 1 < '-' + 1.

The answer is therefore given by the simple formula

M

=

2npqC�=IP"-1q"-"·

2. By applying Stirling's formula (Appendix 1, page 347) prove the following

result:

M where

c

and n is so large HINT:

log log

as

( �7) ( � n; ) M:

M:

2

2

q

q

�7( ( 2

=

to make

c

=

q

1 +

IBI < 1

1 max. -- ' _1_ np- 1 nq- 1

)

;;; X o.

/

< 2(n -iJ) + 2(nq >

�Be}

1

� iJ') �

"

-

1

2

ma . x

( � /�

4(n 12(nq- t'J') 12(np - iJ) 0 ;;; " ;;; 1 j t'J' + = 1 "

np

iJ)2

' {} nq

�

,,

t'J 4(nq

)

: iJ')2

178

INTRODUCTION TO MATHEMATICAL PROBABILITY

[CHAP. IX

3. What is the expectation of the number of failures preceding the first success jn

an indefinite series of independent trials with the probability p?

qp+2q •p+3q3p+ .

Ans.

pq .

.

=

---

(1 - q)•

=

q

-·

p

4. Balls are taken one by one out of an urn containing a white and b black balls

until the first white ball is drawn.

What is the expectation of the number of black

balls preceding the first white ball? Ans. 1. By direct application of definition the following first expression for the required expectation

[

M is obtained:

a b b(b - 1) +2 + M=-(a+b- 1)(a+b- 2) a+b a+b - 1 +3

b(b - 1)(b - 2) + ... (a+b- 1)(a+b - 2) (a+b - 3)

Ans. 2. However, it is possible to find a simpler expression for M. number of black balls preceding the first white ball, by

Denote by

Xt

]

.

the

x. the number of black balls

between the first and second white ball, and so on; finally, by Xa+t the number of black We have

balls following the last white ball. Xt

+X2 +

•

•

+Xa +l = b

•

and

E(xt)+E(xo)+ · · · +E(x a+t)

b.

=

But as the probability of every sequence of balls (that is, of every system of numbers

t, x., . ..

X

Xa+t

) is the same, namely, alb! (a+b)!

it is easy to see that

E(:�;.)

=

That is,

E(x.) = · · · = E(xa+t)

M.

=

(a+ 1)M= b

or

b

M= -- · a+1

Equ9:ting this to the preceding expression for

M, an interesting identity can be

obtained, whose direct proof is left to the student. 6. In Prob.

6, page 168, to determine the probability

no.

Consequently, we shall have

p > 1-., for all

n

>

no.

This conclusion leads to the following important theorem

due, in the main, to Tshebysheff:

SEc. 3]

185

THE LAW OF LARGE NUMBERS

With the probability approaching 1 m·

The Law of Large Numbers.

certainty as near as we please, we may expect that the arithmetic mean of values actuclly assumed by n stochastic variables will differ from the arithmetic mean of their expectations by less than any given number, however small, provided the number of variables can be taken sufficiently large and provided the condition as

n�

oo

is fulfilled. If, instead of variables x., we consider new variables with their means

=

z,

=

x,

0, the same theorem can be stated as follows:

-

a,

For a fixed e > 0, however small, the probability of the inequality

I

Z1 + Z2 +

.

·

+ z,.

·

n

,

� e

tends to 1 as a limit when n increases indefinitely, provided

B;�o. n

This theorem is very general.

It holds for independent or dependent

variables indifferently if the sufficient condition for its validity, namely, that as

n�

oo

is fulfilled.

3. This condition, which is recognized as sufficient, is at the same time necessary, if the variables

z1, z2, ... z,.

are uniformly bounded;

..

that is, if a constant number (one independent of n), C, can be found so that all particular values of than C.

z, (i

=

1, 2, .

n) are numerically less

Let P, as before, denote the probability of the inequality

jz1 + Z2 +

·

·

·

+ z,.j

� ne.

Then the probability of the opposite inequality

jz1 + Z2 +

·

·

·

+ z,.j

will be 1- P. Now, by definition,

B,.

=

E(z1 + z2 +

whence one can easily derive the inequality

> ne

[CHAP. X

INTRODUCTION TO MATHEMATICAL PROBABILITY

186

from which it follows that

B ; n


i

i-1

and it is easy to show that as provided a>�.

n--+ co

But the law of large numbers no longer holds if a ;::i!

proof of this is more difficult.

�.

The

8. The following extension of Tshebysheff's lemma was indicated by Kolmogoroft'. 0, E(:x:n b;, Let :x:1, :x: • :X:n be independent variables; E(:x:i) •

•

•

=

B,.

=

b1+ bt+

•

•

•

+ bn

=

INTRODUCTION TO MATHEMATICAL PROBABILITY

202 and

8k

Denoting by

P

Xt

=

+Xz +

·

•

•

+Xk;

k

=

[CHAP. X

1, 2, ... n.

the probability of the inequality

(A)

max.

P < 1/t2• Indication of the Proof.

(8�, 8�, ... 8�) > B.tz,

we shall have

The inequality

(A)

can materialize if and only if one of

the following mutually exclusive events occurs: event event event

If

e1: 8� > B.t2; ez; 8� ;;:i! B.t2; e3; 8� ;;:i! B.tz;

8� > B.t2; 8� ;;:i! B.t2;

(e;) represents the

probability of

P

=

e;(i

=

(et) + (ez) +

1, 2, ... n) ·

·

·

then

+ (e.).

Now consider the conditional mathematical expectation E(8�1ek) of

ek has occurred.

Since the indication of

these variables and

8k

are independent.

E(s�lek)

=

ek does not affect variables Xk+t, Hence

E(8:lek) +bk+t +

·

·

·

8�

given that

Xk+z,

..

. x.,

+b. > B.t2•

On the other hand

B.

=

n

� (ek)E(8!lek) > B.t21 (et) + (ez) +

E(s�)

whence P < 1/tz. 9. The Strong Law as

·

of Large Numbers (Kolmogoroff).

•

+ (e.))

Using the same notations

in the preceding problem, show that the probability of the simultaneous inequalities

will be greater than 1 of

·

k=l

E

-

8 + =:;;; n+1 _E,

1 nII "' provided

n exceeds a

1 1

8n+2 ;;:i! , n+2 E

.

.

•

certain limit depending on the choice

and "' and granted the convergence of the series

��. �nz

Indication of the Proof. T;

=

max.

(

)

8m- 8 n m

---

1

Consider variables

and denote by q, the probability of the inequality lemma

i

for T;

=

> �E.

1, 2, 3, . . . By Kolmogoro:ff's

203

THE LAW OF LARGE NUMBERS and

oo ql + qz + qa +

·

•

1=2•n-1

< 4E-2!2 ,� n 2 2 2 !

·

. i=l

1=21-•n

or

oo

1=2'n-1

< 16E-2! ! i=l 1=2•-•n

(),

�

.. q1 + q2 + qa +

·

·

< 16E-2 �

·

!!._,._

�kB

k=n Hence, the probability of fulfillment of all the inequalities

..,

is greater than

� 3-2E; i

1, 2, 3,

=

..

1- 16c2!b kB,.

-·

k=n

ls,./kl

The inequalities

�

e; k

=

taneously

n, n + 1, n + 2, i

=

and

. . . are satisfied when simul­

1, 2, a,

-

The probability of the last inequality ·being greater than 1

4B,.

-, the probability

nte2

of simultaneous inequalities

Ik�� -

S

k

E'

,

=

n, n + 1, n + 2,

.

•

•

a fortiori will be greater than

B,./n1 tends to 0 when

This inequality suffices to complete the proof if we notice that the series

is convergent.

10. Let x1, x2,

by

P,.(E)

and

•

x,.

be identical stochastic variables and E (x 1)

•

•

•

+ Xn

n

>E

and

X1 + X2 +

+X.. ����----��
,.(E), respectively, the probabilities of the inequalities

X1 +X2 +

. according

=

E (xn > or 1

(Tshebysheff's inequality)

1, 2, ... n, which can be assumed without loss of generality. provided E(x1) = 0, i In case variables are totaUy independent and are subject to certain limitations of com­ paratively mild character, S. Bernstein has shown that Tshebysheff's inequality can be considerably improved. 12. Let x1, x2, x,. be totally independent variables. We suppose E(x;) = 0, E(x�) = b; and =

for i

=

1, 2,

•

•

. n and h > 2,

c

being a certain constant.

Show that

BnEII:

A= E{e•l < e2(1- .. ) where EC

�

IT

IT.

is an arbitrary positive number < 1 and

Indication of the Proof.

E

is a positive number so small that

We have

.. e"fJI � 1

+

EX;

+

�

n=2

whence

E(e""'•) � 1 +

00

::'

E"

"

b;e2

b ie• � (Ec)" < e2(1- ..). n=O

THE LAW OF LARGE NUMBERS

Q

13. If

denotes the probability of the inequality

+

X1

show that

X2

+

If

Q is

of the Proof.

(J

14. S. Bernstein's Inequality.

a

>

Bne

tt

+E

2(1 - a)

the probability of the inequality

2

< e-1 and

Q

Denoting by

lx1 + X2 +

·

·

·

Q




and M is an upper bound

n

P

•

·

Pn ·

·

for an + p.

,

of the inequality

[CHAP. X

INTRODUCTION TO MATHEMATICAL PROBABILITY

206

In the Bernoullian case P•

=

P2

=

·

·

P > 1

·

=

-

Pn, }..

pq and consequently

=

n•' 2e -2Pq+f•.

has the 17. An indefinite series of totally independent variables Xt, x2, xa, property that the mathematical expectations of any odd power of these variables is •

rigorously

2k) (�)\2k)l. k! '

\x,

=

1, 2, 3,

Xt + X2 + where Bn

=

•

0 while

=

E' for i

•

·

s

b;

2

-

=

E (xD

Prove that the probability of either one of the inequalities ·

·

+ Xn >

bt + b2 +

·

·

·

Xt + X2 +

or

tv'2B,;

+ bn is less than

e-12 (S.

·

·

•

+ Xn < -

Bernstein).

ty'2B;.

Prove first that

e'bi

E (e•x•)

� e2.

18. Positive and negative proper decimal fractions limited to, say, five decimals, are obtained in the following manner: From an urn containing tickets with numbers

0, 1, 2,

.

•

. 9

in equal proportion, five tickets are drawn in succession (the ticket

drawn in a previous trial being returned before the next) and their respective numbers

This fraction, if not equal to 0, is preceded by the sign +or -,according as a coin tossed at the same time

are written in succession as five decimals of a proper fraction. shows heads or tails.

Thus, repeating this process several times, we may obtain as

many positive or negative proper fractions with five decimals as we desire. can be said about the probability that the sum of between prescribed limits - "' and oo?

Ans.

n

These

What

such fractions will be contained

n

fractions may be considered as

so many identical stochastic variables for each of which

{3

=

E(x2)

(1

-

lQ-6)(2 - lQ-6)

=

6




1 - 2.,.

hold simultaneously, we have

yspq 1 -

(3)

- 1

1

+

< � < yspq 1 +fT.

f1

M.

B

f1

-

I

(f

Therefore the probability of these inequalities is again > let us take

1

- 271•

Now

T

q where

-r

= 2+-r

is another positive number arbitrarily chosen.

l 1

+

-

f1 (f

1

= 1 + T', 1

+ :

>

1 -

Then

T,

"

Hence, the inequalities

yspq l - -r) 1

- 2.,.

It suffices to take T

M. = ----=E

yspq

to arrive at the following proposition: The probability of the inequality

':! ySpql < B

for a fixed

E

-

M.

E

and sufficiently large number of series can be made as near to

1 as we please. If spq is somewhat large, the quotient

vspq M. differs but little from

V'lf'/2

(see Chap. IX, Prob.

2,

page 177).

Hence,

when the number of series is large and the series themselves sufficiently long, we may expect with great probability that the quotient A

B will not differ much from

v'i/2.

212

[CHAP. XI

INTRODUCTION TO MATHEMATICAL PROBABILITY

DIVERGENCE CoEFFICIENT 3. The considerations of the preceding section can be generalized. n series containing s trials each, and let

Let us consider again

represent the numbers of successes in each of these series.

Without

specifying the nature of the trials (which can be independent or depend­ ent) we shall denote by by

q=

p

the mean probability in all

1 - pits complement.

N = ns

trials and

Again considering the quotient n

! <m•- sp)2

=-i =--=1----,..-----

Q=

Np q

we seek its mathematical expectation E(Q) =D. When all the

N trials

are of the Bernoullian type, D

possible to imagine cases when "coefficient of dispersion." divergence coefficient." quencies in

n

If

D

> 1 or

D

We shall call

m1, m2, ... m,.

D

But it is also

1.

=

Lexis calls

< 1.

VD

the

itself the "theoretical

are actually observed fre­

series, the quotient

n

D

'

! <m•- sp)2

=

'-i =--=1'-o-: ----c :-- -

Npq

may be called "empirical divergence coefficient."

Then, if the law of

large numbers can be applied to variables Xi=

(m, - sp)2 Npq ;

i = 1, 2, 3, .

.

.

'

we can expect with probability, approaching certainty as near as we please, that the inequality

ID'- Dl


+ 8(p,- p))2

E(m,-

=

8p,)2 + 82(p, - p)2

since E(m;-

8p,)

0.

=

On the other hand, 8

8

E(m;-

sp,)2

=

i-1

and

8

!Pi•- !P'•

=

i=1

i=1 8

8

!

(p,-

Pii)2

=

-8pf

+!Pi�, i -1

i -1

whence

8p,- !7'1•

8

E(m,

- 8p,)2

=

sp,- 8p f

-

! (p, - Pi•)2• i-1

. n

Now, letting i take values 1, 2, results, we get

and taking the sum of the

n

n

! E(m,- 8p,)2

=

i-1

8

n

n8p - 8! p' -

! !(p, - Pii)2. i-1i-1

i-1

But

n

n

8!(P- p;)2

=

-n8p2

+ 8!pf

i-1

whence finally

i:al n

n

D

=

1

+

8- 1 npq

--

� (p- P•)2 i-1

�� (p,- Pii)2• 1

-

Npq

i-1 i-1

Two particular cases deserve special attention.

214

INTRODUCTION TO MATHEMATICAL PROBABILITY

Lexis' Case.

[CHAP.

XI

Probabilities remain the same within each series,

but vary from series to series.

In this case

p ii

=

D becomes:

Pi and the expression of

n

D =1 +

s-1�

� (p - Pi)2 . npq i=l

The theoretical divergence coefficient in this case is always greater than 1 and may be arbitrarily large. Poisson's Case. The probabilities of. the corresponding trials in all series are the same, so that

Pii and

P = Pi =

= 1ri

7rl + 7r2 +

+

8

'Trs

��

------------

In this case the divergence coefficient 8

� (p - 7ri)2

D= 1 -

1 = --= .._ =------·•:.

spq

is always less than 1. Since the law of large numbers evidently is applicable to variables

X·• =

(m,- sp)2 ' Npq

we may expect that the empirical divergence coefficient D' will not differ much from D if the number of series is sufficiently large. ·For numerical illustration let us consider 100 series each containing 100 trials, such that in 50 series the probability is % and in the remaining 50 series it is ,%.

Here we evidently have Lexis' case.

The mean

probability in all trials is

p=! and 100

� (!

- p;)2 = 50 'Th +

50

'

Th

=

1.

i=l

Finally,

D =1 + U = 4.96. Now, suppose that we combine in pairs series of 100 trials with probability % and series of 100 trials with probability %, to form 50

4)

SEc.

APPLICATIONS OF THE LAW OF LARGE NUMBERS

series each of 200 trials.

215

Evidently we have here Poisson's case.

mean probability in each series again is

p

=

The

� and

200

� (! Finally,

-

'11'>)2

100

=

·

·dnr + 100 Th ·

=

2.

i-1

D

=

1

-

-h

=

0.96.

The consideration of the divergence coefficient may be useful in testing the assumed independence of trials and values of probabilities attached to these trials.

In the simplest case of Bernoullian trials with

a constant and known probability, the theoretical divergence coefficient is 1.

Now, if the number of series is sufficiently large and the empirical

divergence coefficient turns out to be considerably different from 1, we must admit with great probability that the trials we deal with are not of the supposed type.

If, however, the empirical divergence coefficient

turns out to be near 1, that does not conclusively prove the hypothesis concerning the independence of trials and the assumed value of the probability.

It only makes this hypothesis plausible.

There are cases of dependent trials (complex chains considered by Markoff) in which the theoretical divergence coefficient is exactly

i.

and

the probability of an event has the same constant value in each trial, insofar as the results of other trials remain unknown.

Cases like that

may easily be mistaken for Bernoullian trials without further detailed study of the entire course of trials.

4. When there is good reason to believe that the trials are independent with a constant but unknown probability, we cannot in all rigor find the value of the empirical divergence coefficient n

�(m;- sp)2 D'

=

..'--'- 1�==

--

Npq

to compare it with the theoretical divergence coefficient D

=

1, since

p

remains unknown. But, relying on Bernoulli's theorem, we can take the quotient

M N where

M

=

m1 + m2 +

as an approximate value of

p.

·

·

By taking

expression for D' we get another number

·

+

p

m,.

=

M/N in the preceding

216

INTRODUCTION TO MATHEMATICAL PROBABILITY

[CHAP.

XI

However, considering m1, m2, ... mn

which in general is close to D'.

not as observed but as eventual numbers of successes in n series, the mathematical expectation of D" is different from

1.

To avoid this

difficulty, it is better to consider a slightly different quotient n

-

� (mi- s;y n(N -1) Q

(n

=

- 1)j}(N - M)

For this quotient there exists a theorem discovered and proved for the first time by the eminent Russian statistician Tschuprow.

Theorem. The mathematical expectation of Q is 1·igorously equal to Proof. Here we shall develop the proof given by Markoff. The

1.1

above given expression of Q presents itself in the form% and therefore has no meaning in two cases: M cases we set Q

=

=

0 or

1 by definition.

M N. For these exceptional M 0 nor M N, we =

If neither

=

=

can present Q in the form

(4)

Q

Considering m1, m2,

.

=

� mi2- M2 n (N- 1) n n- 1 M(N- M) .

. . mn as stochastic variables assuming integral

values from 0 to s, the probability of a definite system of values

IS

To get the expectation of Q we must multiply it by P and take the sum E(Q)

'l;PQ

=

extended over all non-negative integers m1, m2, . . . mr., each of them not exceeding s. To perform this multiple summation we first collect all terms with a given sum m1 + m2 + 1

·

·

·

+ mn

=

M.

The theorem itself and its proof given by Markoff can be extended to the case of

series of unequal length.

SEc.

4)

217

APPLICATIONS OF THE LAW OF LARGE NUMBERS

Let the result of this summation be S,,t. sum

Then it remains to take the

N

� SM

M=O

to have the desired expression E(Q). To this end we first separate two terms corresponding toM 0 and M N. In the former case =

=

and the probability of such an event is qN while Q case the probability of which is pN, while again Q

E(Q)

=

1.

=

=

1.

In the latter

Thus

N -1

� S.u.

pN + qN +

M=l

To find SM we observe that the denominator of Q has a constant value when summation is performed over variable integers m1, m2, . . . m,. connected by the relation m1 + m2 +

·

·

·

+ m,.

=

M.

Hence, it suffices to find two sums '];Pm;

and

'];P

extended over integers m1, m2, . . . mn varying within limits 0 and and having the sumM. To this end consider the function

V

=

(pte�• + q)•(pteh + q)•

·

involving n + 1 arbitrary variables t, ��. b, V consists of terms of the form

·

·

.

(pte�· + q)• . . �...

Evidently we obtain the sum '];P by setting�� and taking the coefficient of tM in the expansion

(Vh,=h=

.

.

.

�.=o

=

=

�2

(pt + q)N,

Thus

(5)

�p

�

=

Nl M N-M MI(N -M) 1P q '

To find '];Pm; take the second derivative o2V a�;

s

When developed,

=

·

·

·

=

���

=

0

2 18

and after setting

�1 �2 = ·

=

·

·

·

(�2�r)

and take the coefficient of

2-[

tM.

�n

=

=

O, expand

�·-Eo- ... -�·=0

Thus we find

]

(N-1)! (N-2)! 1) + �p M N� m;- 8 (M-1)l(N-M)I 88( - (M-2)!(N-M)! p q M.

(6)

(4), (5),

Referring to

SM

[CHAP. XI

INTRODUCTION TO MATHEMATICAL PROBABILITY

=

and (6), we easily get

(N- 2)!N n(N- 1) nN- n + (n- 1)M (N- M) n(M - 1)!(N- M)l[ + (N - n)(M - 1) - M (N- 1)]pMqN-M;

or, after obvious simplifications,

S

-

'11-

Nl pMqN-M. M l(N - M)!

Hence N-1

! SM

=

(p + q)N - p N - qN

=

M=1

1 - pN - qN,

and finally E(Q)

=

1.

Markoff, using the same method, succeeded in finding the explicit expression of the expectation E(Q-

2

1) •

Since there is no difficulty in finding this expression except for some­ what tedious calculations, we give it here without entering into details of the proof:

E(Q

-

2

1)

N-1

=

�M- 1 N- M- 1 2 N(N - n) N (n-1)(N- 2)(N- 3)��. N - M c::pMq -M, M=1

whence the following inequality immediately follows: E(Q-

1)2




1,

1,

However, X;

as we can easily see.

Since

E(x,) = E(xD = p2 1 + (1 - p2) o = p2 ·

·

XI + X2 + + Xn will + Xn) = np2• E(xi + X2 +

the expectation of the sum

·

·

·

·

·

be

·

As to the dispersion of this sum, it can be expressed as follows:

n

Bn =

� E(x, - p2)2 + 2� E(x, - p2)(xi - p2).

i-1

i>i

Now

(8)

E(x, - p2)2 = E(xD - 2p2E(x,) + p4 = p2(1 - p2)

and

(9)

E(x, - p2)(xi - p2) = E(x, - p2) E(xi - p2) = 0 ·

for j > i + 1 because then

(10)

x;

and

E(x; - p2)(xi+I - p2)

xi

=

are independent.

E(x;x;+I) - p4 = pa - p4

since the probability of simultaneous events

X;= 1,

But

7)

SEc.

is p3•

223

APPLICATIONS OF THE LAW OF LARGE NUMBERS

Taking into account Bn

(8), (9), and (10), we find

=

np2q(3p +

1) - 2p 3q

and the condition as is satisfied. x2,

•

•

•

x,.

n---?

oo

Hence, the law of large numbers holds for variables x1, To express it in the simplest form, it suffices to notice that

the sum XI+ X2 + ' ' ' +X,.

represents the number of pairs EE occurring in consecutive trials of the Bernoullian series of n + pairs by m.

1 trials.

Let us denote the frequency of such

Then, referring to the law of large numbers, we get the

following proposition: If in n consecutive pairs of Bernoullian t1ials the frequency of double successes EE ism, then the probability of the inequality

1 as near as we please, when n becomes sufficiently large.

will approach

7. Simple chains of trials, described in Chap. V, Sec. 1, offer a good example of dependent trials to which the law of large numbers can be Let PI be the given probability of an event E in the first trial. According to the ·definition of a simple chain, the probability of E in

applied.

any subsequent trial is a or {3 according as E occurred or failed to occur in the preceding trial.

By p,. we ·denote the probability for E to occur

in the nth trial when the results of other trials are unknown.

{j

=a-

{3,

p= 1

{3

Let

{j·

--

-

Then, according to the developments in Chap. V, Sec. 2,

) -I Pn = P + (Pt - p {jn , whence PI+ P2 + ' ' ' + Pn �__:_.!...__:_ .::_ n __:_�

=

p+

__

barring the trivial cases {)

=

1 or {j

=

-1.

{j n -1---{j'

Pt- P 1 n

-

It follows that p represents

the limit of the mean probability inn trials when n increases indefinitely, and for that reason p may be called the mean probability in an infinite chain of trials.

When it is known that E has occurred in the ith trial, its

224

[CHAP. XI

INTRODUCTION TO MATHEMATICAL PROBABILITY

probability of occurring in some subsequent jth trial is given by

p}')

p + q{Ji-i,

=

q = 1 - p.

In the usual way we associate with trials 1, so that in general

Xt, x2, x3,

•

•

2, 3, . . . variables

•

x, x,

=

=

1 when E occurs in the ith trial 0 when E fails to occur in the ith trial.

Evidently

E(xi)

=

E(xr)

=

Pi·

In order to prove that the law of large numbers can be applied to variables for large

x1, x2, xa, . . , we must have an idea of the behavior of B,. n. By definition .

n

+ Xn - Pn)2 =

� E(xi- p;)2 +

i=l

+ 2 � E[(xi- Pi)(xi - Pi)]. i>i

The first sum can easily be found.

E(x, - Pi)2

=

We have

Pi- p� =pq + (q - p)(pt - p)fli-1 - (Pt - p)2fJ2i-2

whence n

A

=

� E(x;- Pi)2"' npq

i=l

neglecting terms which remain bounded.

As to the second sum, we

observe first that

E(x, - p;)(xi - Pi)

E(x;xf) - PiPi·

=

Again, since the probability of

XiXf is evidently

=

1

p;p<jl we have E(xix;) =PiP}»,

and

E(xi - pi)(x1 - p;)

=

Pi.P}i> - p;) pqfJi-i + 1 + (p - p)(q - p){Ji-1 - (p1 =

_

p)2fJHi-2.

1, 2, ... n - 1, we must take the sum of these expr�ssions letting j run over i + 1, i + 2, ...n. The result of this Now, for a fixed i

=

summation is

pq

O

_

0n-Hl

1- o

[Ji

0n

'

2 i + (pt - p)(q - p) � - (Pt - p) o _

0i-l

0n-l . 1- fJ _

SEc. 8)

APPLICATIONS OF THE LAW OF LARGE NUMBERS

Taking i

225

1, 2, 3, . . . n - 1 and neglecting in the sum the terms

=

which remain bounded, we get B

�E(x,- p,)(x;- p;)

=

,_

i>i

6 npq-_ 1- 6

whence B,.

=

1+6

A +2B,_np� ·

This asymptotic equality suffices to show that B ,..-+0 n2

as

n-+oo.

Therefore the law of large numbers can be applied to variables x2,

xa,

. . . .

x 1,

Since the sum X1 + X2 +

represents the frequency of

E

'

' ' +X,.

=

m

in n trials, the law of large numbers in

this particular case can be stated as follows: For a fixed

E

> 0, no matter

how small, the probability of the inequality

l tends to 1 as n -+

+P2 + · m - P1

oo

·

·

+p,.

n

n

,

<e

•

The arithmetic mean P1 +P2 +

·

·

·

+p,.

n

itself approaches the limit p.

It is easy then to express the preceding

theorem thus: The probability of the inequality

tends to 1 as n -+ oo

•

This proposition is of exactly the same type as Bernoulli's theorem, but applies to series of dependent trials.

8. Let a simple chain of N

=

ns trials be divided into n consecutive

series each consisting of s trials; also, let mr, m2, quencies of

E

in each of these series.

.

•

.

m,. be the fre­

When N is a large number, the

mean probability in N trials differs little from the quantity denoted by p. It is natural to 'modify the definition of the divergence coefficient given in Sec. 3 by taking p instead of the variable mean probability in N trials. Thus we define

226

[CHAP. XI

INTRODUCTION TO MATHEMATICAL PROBABILITY

In our case, the variables

X1 = (m1- 8p)2,

X2 = (m - 8p)2, 2

·

·

·

X,.= (m,.- sp)2

are neither identical nor independent, although the degree of dependence is evidently very slight.

These variables can also be presented in the

form (11)

(x..- p + Xcz+l - p +

taking successively a

•

•

. . . (n - 1)8 + 1.

1, 8 + 1, 28 + 1,

=

+ Xcz+s-1 - p)2

•

To find the mathematical expectation of (11) it suffices to notice that E(x,

_

p)2 = E(x,

E(x, -

p;)2 + (p,

_

_

p)2

=

pq + (q

_

p)(p1

_

p)�>-1

p)(xi- p) = E(x, - p ;)(x i - Pi) + (p; - p)(Pi - p) pq�-i + (p1- p)(q - p)�i-1 =

and then proceed exactly as in the approximate evaluation of B,. in Sec. 7. The final result is E(x.. - p + Xa+11 � - 8pq + 1 - � _

P+ _

Xa+s- 1-

p)2 = (q- p)(p1- p)(1 + �) 4_1 � (1- �)2 (1- �)2 +

2pq (1- �)2

·

·

2pq�

�·

·

+

_

(q - p)(p1- p) +1 [28(1 - �) + 1 + �]�a+•-1. (1 - �)2

For somewhat large 8 the two last terms in the right member are com­ pletely negligible; so is the third term if a � 8 + 1. approximation,

Hence, with a good

2pq� 1 � (q- p)(p1- p)(1 + �) 8pq + 1- � (1 - �)2 + (1 - �)2 , 2pq� 1 + � > 1 ,; if X E( ) = 8pq , (1 �)2 � 1 E(X1)

=

�

_

_

and

D

=

1 + �

1- �

_

(q- p)(p1- p)(1 + �). 2� Npq(1 - � )2 s(1 - �)2 +

Again, when N is large, the last term can be dropped and as a good approximation to D we can take (12)

D=

1 + � -

1- �

2� 8(1- �)2

·

It can be shown that the law of large numbers holds for variables X11 X2, . . . X,. and therefore when n (or the number of series) is large, the

SEc. 9]

APPLICATIONS OF THE LAW OF LARGE NUMBERS

227

empirical divergence coefficient is not likely to differ considerably from D as given by the above approximate formula. 9. In order to see how far the theory of simple chains agrees with

actual experiments, the author of this book himself has done extensive experimental work.

To form a chain of trials, one can take two sets of

cards containing red and black cards in different proportions, and proceed to draw one card at a time (returning it to the pack in which it belongs after each drawing) according to the following rules: At the outset one card is taken from a pack which we shall call the

first

set;

then, whenever a red card is drawn, the next card is taken from the first set; but after a black card, the next one is taken from the

second

set.

Evidently, these rules completely determine a series of trials possessing properties of a simple chain.

In the first experiment the first pack

contained 10 red and 10 black cards, while the second pack contained 5 red and 15 black cards.

Altogether, 10,000 drawings were made, and

following their natural order, they were divided into 400 series of 25 drawings each.

The results are given in Table III.

TABLE III.-DISTRIBUTION OF RED CARDS IN 400 SERIES OF 25 CARDS Frequency of red cards,

m

Difference, m

1 2 3 4 5 6 7 8

9 10 11 12 13 14 15 16 17

-8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7

8 9

Number of series with these frequencies

2 4 8 27 29 54 37 52 47 44 41 20 20 7 4 3 1

The sum of the numbers in column 3 is 400, as it should be.

Taking

the sum of the products of numbers in columns 1 and 3, we get 3,323, which is the total number of red cards.

The relative frequency of red cards in

10,000 trials is, therefore, 0.3323.

228

INTRODUCTION TO MATHEMATICAL PROBABILITY

[CHAP.

XI

In our case

i,

a =

f3

o

!,

=

=

l

and the mean probability p in an infinite series of trials p

1

{3

=

-

1

=

0

3

0.3333 .

=

Thus, the relative frequency observed differs from p only by 10-3 and in this respect the agreement between theory and experiment is very satisfactory.

Now let us consider the theoretical divergence coefficient

for which we have the approximate expression

D Here we must substitute

D

o - o

1 + =

o

1

_

>i and

=

s

2o s(1 - o)2 25.

=

.

The result is

1.631, approximately.

=

To find the empirical divergence coefficient we must first evaluate the sum S

:Z(m- -Y-)2

=

extended over all 400 series.

For the sake of easier calculation, we

presentS thus: S

:Z(m - 8)2 - i:Z(m -

=

8) +

-'4�.

Now from Table III we get

:Z(m-

8) 2

=

:Z(m -

3,521;

8)

=

123

whence

s

=

3,483.4.

2000%

Dividing this number by

=

2,222.2,

we find the empirical

divergence coefficient

D' which differs from D

=

=

1.568

1.631 by only about 0.06, well within reasonable

limits.

10. In two other experiments two packs were used: one containing

13 red and 7 black cards, and another 7 red and 13 black cards.

In

one experiment the pack with 13 red cards was considerPd as the

first

deck, and in the other experiment it became the

second

deck.

The

new experiments were conducted in the same way as that described in Sec. 9, but they were both carried to 20,000 trials divided into 1,000 series of 20 trials each. a =

In the first experiment, we have

f3

U,

=

'!"'--f,

o

=

T"'�r,

and D

=

1.796, approximately,

P

=

i

SEc. 10)

APPLICATIONS OF THE LAW OF LARGE NUMBERS

229

while the same quantities for the second experiment are a=

p=i

f3 = H,

y�,

and

D = 0.556, approximately. The results of these experiments are recorded in the following two tables: TABLE IV.-CONCERNING THE FIRST ExPERIMENT Frequency of red cards,

m

2 3 4 5 6 7 8 9 10 11

Difference, m

-10

3

-8 -7 -6 -5 -4

36 59

-3 -2 -1

93 103 117

0 2 3

128 121 101 93

4 5

48 39

6 7

26 7

8 9 10

1 1 1

1

12

13 14

15 16 17 18 19 20

Number of seties with these frequencies

5 18

TABLE V.-CONCERNING THE SECOND ExPERIMENT Frequency of red cards,

m

Difference, m-

10

Number of series with t.hese frequencies

2 10 48 112

8 9

-5 -4 -3 -2 -1

10 11 12 13

0 1 2 3

193 251 201 113 56

14 15

4 5

9 5

5 6 7

230

INTRODUCTION TO MATHEMATICAL PROBABILITY

[CHAP. XI

Taking the sum of the products of numbers in columns 1 and 3, we find 10,036

and

10,045

as the total number of red cards in the first and second experiments. Dividing these numbers by 20,000, we have the following relative frequencies of red cards: 0.50018 extremely near to p

=

0.5.

and

0.500225

From the first table we find that

2:(m - 10)2

=

8,924

summation being extended over all 1,000 series. by 20,000

·

�

=

Dividing this number

5,000, we find the empirical divergence coefficient in

the first experiment

D'

1.785

=

which comes close to

D

=

1.796.

Likewise, from the second table we find 2:(m - 10)2

=

2,709,

whence, dividing by 5,000,

D"

=

0.5418

again close to

D

=

0.5562.

Thus, all the essential circumstances foreseen theoretically, for simple chains of trials, are in excellent agreement with our experiments. Problems for Solution 1. From an urn originally containing a white and b black balls, in succession, each ball drawn being replaced by 1 + c(c > before the next drawing.

If

m

0)

n

balls are drawn

balls of the same color

is the frequency of white balls, show that the prob­

ability of the inequality

1

I<E

a n � - a+b -

does not tend to 1 as n increases indefinitely Indication of the Proof. If x, 1 or x; =

(Markoff, G. P6lya). =

O, according as a white or a black ball

appears in the ith drawing, we have E(x;)

2

=

E(x;)

=

a a+b'

E(x;x;)

=

a+c a a+b a+b+c

--

·

•

APPLIOATIONS OF THE LAW OF LARGE NUMBERS

231

Hence

B,.

=

E

(

x, + x, +

+x

)

na ' ,.-a+b

(a+b} l(a + b +c) +

+ nab

(a+b)(a+b +c)

.

2. Marbe' 8 Prob lem. A group of exactly m uninterrupted successesE or failures F in a Bernoullian series of trials with the probability p for a success is called an "m sequence." If N is the frequency of m sequences in n trials. show that the probability of the inequality

�� for a fixed

E

_

i

(pmq2 + plqm)
0) we find that cr2tl


oo

and, indeed, by setting a =

2l q = 1!'b in conformity with the result in Sec. 14. These examples may suffice to give an idea of problems in geometric Sylvester, Crofton, and others have enriched this field

probabilities.

by extremely ingenious methods of evaluating, or rather of avoiding However, from the evaluations, of very complicated multiple integrals. standpoint of principles, these investigations, ingenious as they are, do not contribute much to the general theory of probability. Problems for Solution

l whose middle 1. A point X is taken at random on a rectilinear segment AB point is 0. What is the probability that AX, BX, and AO can form a triangle? The x is assumed to be uniform. Ans. _%. distribution of AX l. 2. Two points X1, X2 are taken at random on AB X Assuming uniform distribution of probability, what is the mathe- A B ° between distance the of n power any of expectation matical X1 FIG. 15. =

=

=

and

X2?

dx1dx2 2ln . Ans. rz rz lx1- x � ln-2JoJo l (n + 1)(n + 2) 3. Three points X1, X2, X a are taken at random on AB. What is the probability that X a lies between X1 and X2? Ans. �.assuming uniform distribution of probability. 4. A rectilinear segment AB is divided into four equal parts AC CO OD DB. =

=

=

=

Supposing that the distribution of probability is symmetric with respect to 0, let p be the probability that a point selected at random on

0 D B

C

A

AB will

be between

C and

D.

Also, let Q be the probability that the middle point between two points selected at random will be between C and D. Prove

FIG. 16.

1 +P2 >- -· 2

thatQ

HINT: The middle point of a segment X1X2 is surely between C and D if : (i) X1 X2 are in CO; or (ii) X1 and X2 are in OD; or (iii) X1 and X2 are on opposite sides

and

of 0.

5. Two points

X1, X2

are chosen at random in a circle of radius

r.

Assuming

uniform distribution of probability, what is the mathematical expectation of their distance?

Ans.

Denoting the required mathematical expectation by

.,.2r•M

=

M,

fo2"fo2"F(r, 9, 9')d9d9'

where

F(r, r

Hence, varying

o,

9')

by

dr

d

F

=

=

fo'J:'·v> + foryr2

2rdr

+

P'•

p2

-

2pp' cos (9 - 9')pp'dpdp'.

- 2rp cos

(9 - 9')pdp

we have

INTRODUCTION TO MATHEMATICAL PROBABILITY

258

f2wJror

and

d(,.1r�M)

41rrdrJo

=

v'rt + p• - 2rp

cos

[CHAP. XII

c.>pdpdM.

By introduction of new polar coordinates the integral in the right member can be exhibited as

f�2 L "'d.,

0

--

FIG. 17.

Thus

2r

cos.,

u1du

=

d(...r'M)

16 -r 3 3

=

L

i

cos3

c.>d

will be > R tends to

oo no matter how large n is. Let Xt, X2 Xni y,, y2, ...Yn be components of w l r axesOX,OY. Then f.\M3

;�:. : ��::�� �1

Indication of Solution.

OM,, OM2, ...OMn

+ 02M2 +OaMa + . .. + OnMn

E (x D

=

EM)

=

•

•

•

Q

�

2"

M. Mh

� �

FIG. 18.

By Tshebysheff's lemma (Chap.X, Sec.1) the probabilities Q and Q' of the inequalities

!Yt

+ Y2 + • • • + Ynl >

are both less than

1/t2•

/r� + r� " Jr2 + r22 t "

+ XnI > t

!x1 + X2 +

1

+ r: + 2 + 12 . + 23

=

=

� �

t t

Now, if the length OM> R then either

!x1

+ X2 + • · · + Xnl >

IYt

+ Y2 + ' ' ' + Ynl >

or

� � =

_Ji_ V2

=

t

@. "2

t

Hence, the probability P for the length of OM to be >

that is,

R

is less than Q

+Q';

2G p 0 tends to

-

1 as

•

2 +x! ...:.--" ._ < 3 + • x + n

x�+x�+ < -!... _::...� ._ _;__ X , + x2 +

n->

__

oo.

References

E.CzUBER: "Geometrische Wahrscheinlichkeiten und Mittelwerte," Leipzig, 1884. E.CzUBER: "Wahrscheinlichkeitsrechnung,"

1,

pp.75- 109, Leipzig,

1908.

H.PoiNCARE: "Calcul des probabilites," pp. 118- 152, Paris, 1912. W.CROFTON: On the Theory of Local Probability Applied to Straight Lines Drawn at

Random in a Plane, the Method Used Being Also Extended to the Proof of

Certain New Theorems in Integral Calculus,

Philos.

Tr

ans., vol.

W.CRoFToN: Probability, "Encyclopaedi�t Brit�tnnica," 9th ed.

158, 1868.

CHAPTER XIII

THE GENERAL CONCEPT OF DISTRIBUTION 1. In dealing with continuous stochastic variables we have introduced the important concept of the function of distribution.

Denoting the

density of probability by f(z), this function was defined by F(t)

J� .J(z)dz

=

and it represents the probability of the inequality

:c

< t.

For a variable with a finite number of values the function of distribu­ tion can be defined as the sum F(t)

� P•

=

o:t<e

p11 p2, :c1, :c2, . :c,. where

•

•

•

p,.

are respective probabilities of all possible values

of the variable :c. The notation :c, < t is intended to show that the summation is extended over all values of :c less than t. •

•

Again, F(t) for any real t represents the probability of the inequality

:c

< t.

In this case F(t) is a discontinuous function, never decreasing and varying between F(- co) at the points :c1, :c2,

=

0 and F( +co) . . . :c,. and are

=

F (:c,

+ 0)

-

1.

Its discontinuities are located

such that

F (:c,

- 0)

=

p,,

denoting, in the customary way,

F(:c, + 0) F(:c, - 0)

=

=

lim F(:c, + lim F (:c, -

E) E)

when E1 through positive values, converges to

0.

To represent F(t)

graphically we note that

F(t) F(t)

=

P1. + P2 +

+p,.

=

F(t)

=

=

·

0 P1 P1 + P2

F(t)

260

for

t
8>0. HINT: Use Liapounoff's inequality. 3. A variable is distributed over the interval (0, + oo ) with a decreasing density of probability. Show that in this case moments M2 and M, satisfy the inequality

M: ;:i! IM, (Gauss) and that in general

if

-1 -1 [(p + 1)M,.]" ;:i! [(v + 1)M v]"

J' >I'> 0.

Indication of th£ Proof.

Show first that the existence of the integral

Jo .. x•f(x)d:l: in case /(:c) is a. positive and decreasing function implies the existence of the limit lim a•Hf(a)

=

0;

a

-+ + oo.

279

THE GENERAL CONCEPT OF DISTRIBUTION Hence, deduce that

fo .. xd.p(a;) where

.p(a;)

=

Jo '"z•+ld.p(a;)

1,

=

=

( v + 1)M

•

J(O) :...... J(z) and, finally, apply the inequality

[.fo

..

a;l'+ld.p(a;)

r [.fo

..

�

zv+1d.p(a;)

]" [.fo .. zd.p(a;) r-1'. •

4. Using the composition formula (1), page 269, prove Laplace's formula on page 278 by mathematical induction. 15. Prove that the distribution function of probability for a variable whose charac­ teristic function

.p(t) is given can be determined by the formula F (t)

=

.

1

C +lim

,.,_ h-o--

J

'"

.p(v)

... 2

1- 6-i•t

dv.

•

.. 1 + h -.,

-

�v

HINT: 'In carrying out Liapounoff's idea, take an auxiliary variable with the dis­ tribution

G(y)

=

Also make use of the integral

_!_f" 2h

- ..

-lildz.

e

Many definite integrals can be evaluated using the relation between characteristic and distribution functions, as the following example shows.

6. Let a; be distributed over ( istic function being in this case

oo,

+

oo

F(t)

=

J 1J

1

C +2-r

whence

1r

'"

1- e-ivt

- .. iv(1 + v1) oo

The character­

1 1 +t1

.p(t) we find

) with the density Y2e-lzl,

e-ivt

-- dv - ,.1 + v1

=

J

1

dv

= -

2

'

_

..

e-1•1da;,

e-ltl,

an integral due to Laplace.

7. A variable is said to have Poisson's distribution if it can have only integral 1, 2, ...and the probability of a; = k is

values 0,

the quantity

a

is called "parameter" of distribution. If

distribution with parameters

a1, as,

.

.•

distribution, the parameter of which is

a1

a,.,

+

n

variables have Poisson's

show that their sum has also Poisson's a2

+

·

·

·

+

a,..

280

INTRODUCTION TO MATHEMATICAL PROBABILITY

8. Prove the following result:

.!_f .. (�)

211"

-oo

v

"

sin tv v d

1 + -:::---- --2n 2·4·6·

_!

=

2

v

·

[

t ( +

n)"

1·2

n(n - 1)

+

-

� t (

1

(t + n

[CHAP. XIII

+ n - 2)" + - 4)"

...]

-

the series being continued as long as arguments remain positive.

HINT! Consider the sum of n uniformly distributed variables in the interval ( -1, +1) and express its distribution function in two different ways. 9. Establish the expression for the mathematical expectation of the absolute

value of the sum of

n

uniformly distributed variables in the interval

Am.

2

+ 3:nl

(2n + 2)

2·4·6· · ·

=

[nl nn 1 n n

- �(n 1

+

(

+

- )( 1. 2

( - �. + �).

2)"+1 + _

4)"+1

_

.

•

.

]·

the series being continued as long as the arguments remain positive.

HINT! Apply Laplace's formula on page 278, conveniently modified, to express the 3:t + Xs + · · · + x,. and that of lxt + X2 + + x,.l. 10. Show that under the same conditions as in Prob. 9

expectation of

·

Elx + x2 +

t

·

·

·

+ x,.l

n

=

211"

f'".. ( ) sint -t

_

"-1

·

·

sin t -t cost dt. ta

HINT: Prove and use the following formula lim

T-oo

I

T .

81'",.-

-T

1 -tWX . tk X2

=

-.,.lwl.

11. Let Xt and X2 be two identical and normally distributed variables with the 0 and the standard deviation tr. If xis defined as the greater of the values

mean

=

lxtl, lx2!,

that is,

x find the mean value of

x as

max.

well as that of

E(x) 12. Let

=

x

(ixtl, lx•l> x1•

Am.

2tr

=

=

y;• min.

(lxtl, lx21, . . . lx,.i)

. x,. are identical normally distributed variables with the mean 0 where Xt1 x2, and the standard deviation tr. Find the mean value of x. Am. Setting for brevity =

--

_ria' ... fo .. v2

--

we have

tTy;.

E(3:)

=

e

0

2.•

du

(1 -

=

B(t),

B(t) }"dt.

THE GENERAL CONCEPT OF DISTRIBUTION In particular for n

=

281

2

E =

;.cV2 - t).

For large n asymptotically

E(:c) "'

(v) -1 V1

f..

=

-

-oo

•

=

V,.

fl"t

The quotient

Ll

t2dF(t)

- x)ei•l•d:e

(1

0

is represented by a uniformly convergent integral; hence •

hm

tp(V) - 1

1

=

v1

--

2

J

00

-oo

=

t1dF(t)

1

--

2

or

0, we can take n so large that

I B(vi ) l < e;

i

=

0, 1,

•

•

•

n

whence

IOI < et•. Thus

l log q>(t) + it11 < et1 and since e can be taken arbitrarily small, log ..; ->.., >..; >.., l; l, + oo.

it into five integrals -oo,

that

I 1, I2, Ia, I 41 I 6

and

To estimate I3, we notice

taken respectively between limits

ltp.,(v) - e -21 � v2. ••

Hence

(8)

l >.vdv l

-IIal �21r

1

11"

=

0

To estimate 12 + I 4, we use the inequality get

}.,2

21r

-·

ltp.,(v) - e -21 � E.,(l) ••

(9) Finally, dealing with I 1 and

I 6,

we use the obvious inequality

ltp.,(v) - e -21 � 2 ••

and we

288

[CHAP. XIV

INTRODUCTION TO MATHEMATICAL PROBABILITY

and we obtain

(10) Taking into account results:

IH (t) n

_

In it, since

!. 2

_

(7), (8), (9),

!. f "'e -fsin tvdv v Jo

7f'

l

and

the following inequality

(10),

(Ill)•

En(l) + �e(hl)2 v:;hx

2

X2 + < h + 2'11' 4n-

4

•

7f'

X is still at our disposal, we can take X En(l)lh-1. =

The inequality thus obtained when combined with

(11)

I .. F

.. Jo I

�sin tv 1 1 dv (t) - 2- ?i f e -2 v-

crt


0, it will be satisfied for all smaller 8. Let

/i(t)

be the distribution function of

x;(i

=

1, 2, ... n).

The

sum

f(t)

=

ft(t) + f2(t) + . . . + f,.(t)

being a nondecreasing function of (Chap. XIII, Sec.

t,

the following inequality holds

5):

(f_ ,.Wdf(t) y-c � (f_ ,.ltj•df(t) y-b' (f_")l"df(t) y-•, ..

..

INTRODUCTION TO MATHEMATICAL PROBABILITY

290

provided a> b > c > 0.

We take here

a= 2 + 6, '

supposing 0 < 6 < 6.

f_ and

IWdf(t)

=

b

=

2 + 6',

c=2

Then

�Jl��a'; n

....

[CHAP, XIV

f_00,.1tjadj(t)

1

�Jl��6; n

=

f_ ,.ltl•df(t)

..

1

=

B ,.

(*Jl��6'Y B,.a-a'(*JlwaY'· �

But this inequality is equivalent to

and it shows that

if n

�r2+6 __a_�o, �,(k)

1 _ 1+ B" 2 provided 0 < 61 < 8.

Hence, in the proof we can assume that the funda-

mental condition is satisfied for some positive 6 � 1. b. Liapounoff's inequality (Chap. XIII, Sec. 5) with c= 0, b = 2, a

=

2 + 8 when applied to

Hence, (12)

x, gives ( u

'

we can write bk

,.

I - B t2 2

(I4) If w,.ltjH 2


=

v211' _

I

t

-oo

z,.

e

-

-�2

na

du,


,.(v)

in any fixed

v•

interval -l � v � l tends uniformly to e-2. It suffices to apply the fundamental lemma to conclude that the probability of the inequality m

< t,. �

-np,.

tends uniformly to the limit

--

¥2; 1

if t,. tends to t. Since B,. is asymptotic to

1 npq1

f'

e

_,.

2du

-�

+ 6 and Pn differs from p by a quantity of the order -6

1 /n, the inequality m-

np


0. p < 1 and hence the limit 1 and at the same time the

Show (a) that Liapounoff's condition is satisfied when theorem holds; (b) that this condition is not satisfied if p � limit theorem fails at least for p > 1. Solution. a. By using Euler's formula we find

c.�,."'

1+�

2

(2p. +1 - p) (2 + B)p. +1

_ P

' -

(n + a))2

{log

(p

-

1) •

Hence the first part is answered. b. The probability of the inequality

z, +z2+

·

·

+z.. �O

·

is less than

2�

� (k k�l

and this, in case

1 +a) {log (k + a)}P

p > 1, is less than 2 p-1

---- (log

a)I-P,

Hence, the probability of the equality Z1

+ Z2 +

•

·

·

+ :r;,.

=

0

2 >1----- (log a)1-P and the limit theorem cannot hold. p- 1 because 2p. -p + 1 > 0.

remains always that

B,. �

oo

Note

4. Prove the asymptotic formula

+

n" 1·2

·

•

•

n

1 "'-e"'

2

being a large integer. Hm: Apply Liapounoff's theorem ton variables distributed according to Poisson's law with parameter 1. 5. By resorting to the fundamental lemma, prove the following theorem due to Markoff: If for a variables,. with the mean 0 and the standard deviation 1 n

=

lim

E(s�)

n-+oo

1 =

--

y2w-

J

..

_

00

=

t�<e-it'dt

305

FUNDAMENTAL LIMIT THEOREMS for any given k

=

3, 4, 5,

. , then the probability of the inequality

s,.< t

tends

to the limit

6. In many special cases the limit of the error term can be considerably lower than :r;,. are identical and uni­ that given in Sec. 6. For instance, if variables x1, xs, formly distributed in the interval -�.� the probability F,.(t) of the inequality •

•

•

/n "\/12

Zt+z•+ · · · +x,.u-4pq(l-3pq)"2 + ...

for small u, while the other root tends to

0

be reached in the same way as in Examples

as

u-+

1 and 2,

0.

The final conclusion can now

pages

297 and 301.

References P. LAPLACE: "Thoorie analytique des probabiliMs," Oeuvres VII, pp. 309ff. S. PoissoN: "Recherches sur la probabilite des jugements en matiere criminelle et en matiere civile," pp.

246ff.,

Paris,

1837.

P. TsHEBYSHEFF: "Sur deux th�oremes relatifs aux probabilit�s," Oeuvres

481-491.

2,

pp.

A. MARKOFF: The Law of Large Numbers and the Method of the Least Squa1·es (Russian), Proc. Phys.-Math. Soc. Kazan, .Jl"e-z• --- : Sur les racines de }'equation e� dx"

1898. =

0,

Bull. Acad. Sci. St. Petersbourg,

9,

1898. --- :

"D�monstration du second thooreme limite du calcul des probabilit� par la

m�thode des moments," St. Petersbourg, --- :

1913. 1912.

"Wahrscheinlichkeitsrechnung," Leipzig,

A. LIAPOUNOFF: Sur une proposition de la thoorie des probabilit�, Bull. Acad. Sci. St. Petersbourg,

13, 1900.

---: Nouvelle forme du thooreme sur la limite de probabilite, M�m. Acad. Sci. St. Petersbourg,

12, 1901.

G. P6LYA: Ueber den zentralen Grenzwertsatz der Wahrscheinlichkeitsrechnung, Math. Z., 8, 1920. J. W. LINDEBERG: Eine neue Herleitung des Exponcntialgesetzes in der Wahrschein­ lichkeitsrechnung, Math. Z.,

15, 1922.

P. LEVY: "Calcul des probabilit�," Paris,

1925.

S. BERNSTEIN: Sur !'extension du thooreme limite du calcul des probabilites aux sommes de quantit� d�pendantes, Math. Ann., A.

KoLMOGOROFF:

Eine

97, 1927.

Verallgemeinerung des Laplace-Liapounoffschen Satzes,

Bull.·Acad. Sci. U.S.S.R., p.

959, 1931.

---: Uebe� die Grenzwertsii.tze der Wahrscheinlichkeitsrechnung, Bull. Acad. Sci.

. U.S.S.R., p. 363, 1933. A. KHINTcHiNE: "Asymptotische Gesetze der WahrscheinlichkeitsrechnutJ.g," Berlin,

1933.

CHAPTER XV

NORMAL DISTRIBUTION IN TWO DIMENSIONS. LIMIT THEOREM FOR SUMS OF INDEPENDENT VECTORS. ORIGIN OF NORMAL CORRELATION 1. The concept of normal distribution can easily be extended to two and more variables.

Since the extension to more than two variables

does not involve new ideas, we shall confine ourselves to the case of two-dimensional normal distribution. Two variables,

x, y, are said to be normally distributed if for them

the density of probability has the form (j'l'

where cp

ax2 + 2bxy + cy2 + 2dx + 2ey + f

=

is a quadratic function of x, y becoming positive and infinitely large together with lxl + IYI· This requirement is fulfilled if, and only if,

ax2 + 2bxy + cy2 is a positive quadratic form.

The necessary and sufficient conditions

for this are:

a> 0;

ac-b2

=A > 0.

Since A> 0 (even a milder requirement A ;;6 0 suffices), cbnstants

xo, y0

can be found so that cp

a(x-xo)2 + 2b(x-Xo)(y-Yo) + c(y - Yo)2 + g

=

identically in

x, y.

presented thus:

It follows that the density of probability tr"' may be

4

The expression in the right member depends on six parameters K; a, b, c; x0, yo. But the requirement

reduces the number of independent parameters to five.

We can take

a, b, c; xo, Yo for independent parameters and determine K by the condition

J_ ..,f_"',.e-,.(s-so)t-'Jb(z-so)(V:"•l-c(u-1/o>•dxdy

K

.

308

=

1

'

SEc. 2)

309

NORMAL DISTRIBUTION IN TWO DIMENSIONS

which, by introducing new variables

-

�= x can be exhibited thus

.. ..

f1 = Y

Xo,

- Yo

f_ .. J_ .. e-af._26�"'�"d�d., = 1.

K

To evaluate this and similar double integrals we observe that the positive quadratic form

a�2+ 2b�., + c.,2 can be presented in infinitely many ways as a sum of two squares

a�2+ 2b�., + c.,2 = (a�+ p.,)2+ ('Y�+ a .,)2, whence

b and

(aa

- P'Y)2

=

=

a{j+ 'Ya

1:1.

By changing the signs of a and {j if necessary, we can always suppose

a8

- P'Y

=

+Vil.

Now we take u =a�+ Pfli

for new variables of integration.

v

=

'Y�+ a.,

Since the Jacobian of u, v with respect

to ��.,is Vil, the Jacobian of�� f1 with respect to u, v will be by the known rules

f .. f .. -oo

-oo

e-afL-26�•'1"d�fl =

1

__

Vil

Thus

K'lf' Vil

=1

'

J

oo

-oo

f

oo

e-u•-v•dudv

-oo

=

1/Vil and, __!____ . Vil

Vil K=-· 7f'

That is, the general expression for the density of probability in two­ dimensional normal distribution is

vac 7f'

-

b2e- (..-zo)L- (..-zol l 1, we find as the expression of the probability for x,yto belong to the ring between

two ellipses l1 and l2.

If l1

=

0 and l2

1

l,

=

- e-1

gives the probability for x, yto belong to the ellipse l. If n numbers l, l1, l2, . . . ln-1 are determined by the conditions

1 - -1 e

=

-1 - -1 • e e

-Z. - -1• e e

=

=

·

the whole plane is divided into n +

·

1

·

=

e-1·-•

-

e-1·-•

=

1 --, n+l

regions of equal probability:

namely, the interior of the ellipse l, rings between l, l1; lt, l2; ••• ln-2, ln-l and, finally, part of the plane outside of the ellipse ln_1• 5. To find the distribution function of the variable x (without any regard toy), we must take forD the domain - oo <X< t;

-oo dz

we see that the probability of the inequality X< t is expressed by

1

---

IT1

-yl2;

f'

-

e 2dx1

·

·

·

dx,.dy1

·

·

·

dy,.

extended over the 2n-dimensional domain

2;(x, - 8)(y, - 81)

(3)


a

n

will be used to designate the sum extended over all integers which are >a and �b. It is an important problem to devise means for the approxi­ mate evaluation of the above sum when it contains a considerable number of terms..

Let [x], as usual, denote the largest integer contained in a real number x, so that x [x] + 0 where 0, so-called "fractional part" of x, satisfies the inequalities =

0�0(x)f'(x)dx =

n>a

which is known as "Euler's summation formula."

Proof.

Let k be the least integer

>a and l the greatest integer �b.

The sum in the left member of (1) is, by definition,

f(k) + f(k + 1) +

.

. . + f(l)

and we must show that this is equal to the right member. we write first 347

To this end

348

INTRODUCTION TO MATHEMATICAL PROBABILITY

(H f.abp(x)f'(x)dx f."a p(x)f'(x)dx J(z bp(x)f'(x)dx J i-A: i 1p(x)f'(x)dx. j s:+1p(x)f'(x)dx LH1(j- �)f'(x)dx _f(j) �(j 1) fiH1f(x)dx J i=l-1

+

=

Next, since

� ..,

+

is an integer,

+

X+

=

+

+

=

+

and

i=l-1 i 1 n=l-1 ) f( f( ) �i + p(x)f'(x)dx _ k t l - � f(n) Lf(x)dx. +

=

1=A:

n=A:+1

On the other hand,

i"p(x)f'(x)dx i"(k- - x �)f'(x)dx _f�)- p(a)f(a) if(x)dx .[bp(x)f'(x)dx .[b(z- x �)f'(x)dx _!�) p(b)f(b) .[bf(x)dx, 1

=

+

+

=

+

+

=

+

=

+

so that finally

J.>(x)f'(x)dx

=

-f(k) - f(k + 1)

-

·

+

f(l) p(b)f(b) - p(a)f(a) J)<x)dx; ·

·

+

-

+

whence

n:S:b a J)<x)dx p(b)f(b) - p(a)f(a) - J:bp(x)f'(x)dx, n>�f(n) +

=

which completes the proof of Euler's formula.

Corollary 1.

The integral

JCzp(z)dz u(x) =

represents a continuous and periodic function of

u(x

+ 1)

If 0 �X� 1,

x

with the period

- u(x) J:+1p(z)dz Jo1p(z)dz JC1(i - z)dz =

=

=

1.

=

0.

For

349

APPENDIX I

11(x)

=

f'"o (2l - z)dz = x(l - x) 2

J

and in general

O'(X) where

0 is

a fractional part of

x.

0 �

=

0(1

- 0} 2

Hence, for every real

O'(X) � j.

Supposing thatf"(x) exists and is continuous in parts, we get

x

(a, b) and integrating by

J>(x)f'(x}dx O'(b)f'(b} - q(a)f'(a) - J:bO'(x)f"(x)dx, =

which leads to another form of Euler's formula:

S:b "'£f(n) J)<x}dx + p(b)f(b) - p(a)f(a) - O"(b)f'(b) +

n

n>a

=

+ Corollary 2.

If

11(a)j'(a) + J:O'(x)f"(x)dx.

f(x) is defined for all x � a and possesses a continuous (a, + oo); if, besides, the integral J: p(x)f'(x}dx

derivative throughout the interval oo

exists, then for a variable limit

(2}

n

b

we have

S:b

"'£f(n)

n >a

= C+

jf(b}db + p(b)f(b) + !"'p(x)f'(x)dx

where C is a constant with respect to It suffices to substitute for

b.

J:p(x)f'(x}dx the difference

£ p(x)f'(x)dx - h p(x)f'(x)dx oo

oo

and separate the terms depending upon

2. Stirling's Formula.

b

from those involving a.

Factorials increase with extreme rapidity

and their exact computation soon becomes practically impossible.

The

question then naturally arises of finding . a convenient approximate

350

INTRODUCTION TO MATHEMATICAL PROBABILITY

expression for large factorials, which question is answered by a celebrated formula usually known as "Stirling's formula," although, in the main, it was established by de Moivre in connection with problems on proba­ bility.

De Moivre did not establish the relation to the number 11" =

3.14159 ...

of the constant involved in his formula; it was done by Stirling.

(2)

In formula

it suffices to take

a

=

�.

J(x)

= log

x,

and replace b

by an arbitrary integer n to arrive at the remarkable expression log

(1

·

2

·

3

·

·

·

( �)

+ n+

n) = C

log n - n+

i ..p(x�dx

For the sake of brevity we shall set

where C is a constant.

f ..p(x)dx. Jn X

w (n) =

Now

f ..p(x)dx Jn X

=

and

f k+1p(x)dx Jk x

=

=

i

f"+1p(x)dx + "+2p(x)dx + Jn X n+l X

r-p(u)du = f ' p(u)du f1p(u)du = Jo u + k +J; u +k Jo u + k f' Ci - u)du + f1(!- u)du = ! f 1 (1 - 2u)2du . 2 j o(k+u)(k+1-u) J; u+k Jo u+k

Hence

where

F (u) ,.

=

1

� (k +u)(k + 1- u)' k=n

Since

(k +u)(k + 1 - u) it follows that for 0 < u < � (k (k Thus for 0
0 for all real

X.

Hence, the roots of the polynomial in the left member are

imaginary, and this implies

J! Taking n for

J2m

2 ·4

1·

3 ·5

=

·

6

2m and n

J2m+l,

and ·

·

=




rp(x),

rp(Xo),

=

and the positive difference

rp(xo + 0) - rp(xo - 0) may be considered as a mass concentrated at the point x0• In all cases rp(x0 - 0) is the total mass on the segment (a, xo) excluding the end point Xo, whereas rp(xo + O) is the mass spread over the same segment including the point

Xo.

The points of discontinuity, if there are any, form an enumerable set, whence it follows that in any part of the interval (a, b) there are points of continuity. If for any sufficiently small positive

rp(xo + E)

>

E

rp(xo

-

)

E ,

xo is called a "point of increase" of rp(x). There is at least one point of increase and there might be infinitely many. For instance, if 856

357

APPENDIX II

IP(x) IP(x) then

c

0

for

mo

for

=

=

is the only point of increase.

1.0(x) every point of the interval

On the other hand, for

x-a b-a

mo-­

=

(a, b)

a�x�c C <X� b,

is a point of increase.

In case of a

finite number of points of increase the whole mass is concentrated in these points and the distribution function

IP(x)

is a step function with a

finite number of steps. Stieltjes' integrals

J:d1,0(x)

=

mo,

represent respectively the whole mass mo and its moments about the origin of the order 1, 2, . . . i. When the distribution function IP(x) is given, moments mo, mt, m2, . . . mi (provided they exist) are deter­ mined.

If, however, these moments are given and are known to originate

in a certain distribution of a mass over

(a, b),

the question may be raised

with what error the mass spread over an interval by these data?

(a, x) can be determined

In other words, given mo, m1, m2, . . . mi, what are the

precise upper and lower bounds of a mass spread over an interval

(a,

x)?

Such is the question raised by Tshebysheff in a short but important article "Sur les valeurs limites des integrates"

(1874).1

The results contained

in this article, including very remarkable inequalities which indeed are of fundamental importance, are given without proof. The first proof of these results and the complete solution of the question raised by Tsheby­ sheff was given by Markoff in his eminent thesis "On some applications of algebraic continued fractions"

(St.

Petersburg,

1884),

written in

Russian and therefore comparatively little known. Suppose that Pi is the limit of the error with which we can evaluate the (a, x) or, which is almost the same, the

mass belonging to the interval value of

IP(x),

when moments mo, m1, m2, . . . mi are given.

If, with i

tending to infinity, Pi tends to 0 for any given x, then the distribution function IP(x) will be completely determined by giving all the moments

One case of this kind, that in which mo

1

=

1,

m2,.

( 2_k_-_1...!_) , 1 _3 _. _5 _· ---=:---'-

__· =

Jour. Liouville, Ser. 2, T. XIX, 1874.

2k

APPENDIX II METHOD OF MOMENTS AND ITS APPLICATIONS 1. Introductory Remarks. To prove the fundamental limit theorem Tshebysheff devised an ingenious method, known as the "method of moments," which later was completed and simplified by one of the most prominent among Tshebysheff's disciples, the late Markoff. The simplicity and elegance inherent in this method of moments make it advisable to present in this Appendix a brief exposition of it. The distribution of a mass spread over a given interval (a, b) may be characterized by a never decreasing function cp(x), defined in (a, b) and varying from cp(a) = 0 to cp(b) = mo, where mo is the total mass con­ tained in (a, b). Since cp(x) is never decreasing, for any particular point xo, both the limits lim cp(xo - e) = cp(xo 0) lim cp(xo + E) = cp(xo + 0) -

exist when a positive number

cp(xo

-

E

tends to 0.

Evidently

0) ;;;;; cp(xo) ;;;;; cp(xo + 0).

If

cp(xo - 0)

=

cp(xo + 0)

then x0 is a "point of continuity" of cp(x).

cp(xo + 0)

>

cp(xo

=

cp(xo),

In case -

0),

x0 is a point of discontinuity of cp(x), and the positive difference cp(xo + 0) - cp(Xo

-

0)

may be considered as a mass concentrated at the point x0• In all cases cp(x0 - 0) is the total mass on the segment (a, xo) excluding the end point x0, whereas cp(x0 + 0) is the mass spread over the same segment including the point Xo. The points of discontinuity, if there are any, form an enumerable set, whence it follows that in any part of the interval (a, b) there are points of continuity. If for any sufficiently small positive E

cp(xo + E) > cp(xo

-

)

e ,

x0 is called a "point of increase" of cp(x). There is at least one point of increase and there might be infinitely many. For instance, if 356

357

APPENDIX II

IP(x)

=

IP(x)

=

for

0

a�x�c C <X� b,

for

mo

then c is the only point of increase. On the other hand, for IP(X) every point of the interval

=

x-a 1no;-b

-a

(a, b) is a point of increase. In case of

a

finite number of points of increase the whole mass is concentrated in these points and the distribution function IP(x) is a step function with a finite number of steps. Stieltjes' integrals

represent respectively the whole mass mo and its moments about the

origin of the order

1, 2, ...i.

When the distribution function IP(X)

is given, ·moments mo, m1, m2, . . . m. (provided they exist) are deter­ mined. If, however, these moments are given and are known to originate in a certain distribution of a mass over

(a, b), the question may be raised (a, x) can be determined

with what error the mass spread over an interval by these data?

In other words, given mo, m1, m2, . . . m., what are the

precise upper and lower bounds of a mass spread over an interval

(a, x)?

Such is the question raised by Tshebysheff in a short but important article 11 Sur

les valeurs limites des integrales"

{1874)

.

1

The results contained

in this article, including very remarkable inequalities which indeed are of fundamental importance, are given without proof. The first proof of these results and the complete solution of the question raised by Tsheby­ sheff was given by Markoff in his eminent thesis of algebraic continued fractions"

11 On

(St. Petersburg,

some applications

1884), written in

Russian and therefore comparatively little known. Suppose that P• is the limit of the error with which we can evaluate the mass belonging to the interval

(a, x) or, which is almost the same, the

value of IP(x), when moments mo, m1, m2, . . . m, are given. If, with i tending to infinity, P• tends to 0 for any given x, then the distribution

function IP(x) will be completely determined by giving all the moments

One case of this kind, that in which mo 1

=

1,

m2,.

=

5_ . · �>+I(z)) - P;-1 Q;(qi+1 - 4>>+1(z)) - Q'-1

=

=

P;+l - P;lf>;+J(z) Q'+1 - Q;lf>;+1(z)

we can write

q,(z)

=

P1+1 - P,q,,+l(z) Q;+1 - Q;lf>;+J(z)

in the sense that the formal development of the right-hand member is identical with

q,(z).

By virtue of relation

p,

4> (z) Q, The degree of

(3)

1 =

Q,(Q>+I - Q;l/>i+1)"

Q; being �� and that of Qi+1 being �i+1, the expansion of Q;(Q;+1 - Q;lf>i+l)

in a .series of descending powers of z begins with the power

z>-tH·1+�.

Hence,

p,

q,(z) - Q, and, since

M

=

z>.M->-1+1 +

�i+J ?; �; + 1, the expansion of q,(z) - p, Q;

begins with a term of the order characterizes the convergents

2�, + 1 in 1/z at least. This property P;/Q; completely. For let P/Q be a

rational fraction whose denominator is of the nth degree and such that in the expansion of

q,(z) - p

Q

361

APPENDIX II

the lowest term is of the order 2n + 1 in 1 Iz at least. Then PIQ coincides with one of the convergents to the continued fraction (1). Let i be determined by the condition

Then 4> (

z

)

P; - Q; P

4J(z) -Q

M zX.+X.; +

>-•+t in

1/z.

Hence, the

PQ;- P;Q in

z is

not greater than both the numbers

>.;-n-1

and

which are both negative while

is a polynomial.

Hence, identically, PQ;- P;Q

=

0

or

which proves the statement. 3. Continued Fraction Associated with

ib d'P(x)X

Let

·

a Z-

IP(X)

be a never

decreasing function characterizing the distribution of a mass over an interval (a, b). The moments of this distribution up to the moment of the order 2n are represented by integrals mo

=

.C d'P(x),

m1

=

J:bxd'P(x), m2

=

.Cx2d1P(X),

•

·

•

m2n

=

J:bx2nd'P(x).

INTRODUCTION TO MATHEMATICAL PROBABILITY

362 Let

If rp(x) has not less than

n + 1 points of increase, we must have Lll > o, . . . Ll,. > o,

.do > 0,

and conversely, if these inequalities are satisfied, rp(x) has at least points of increase. To prove this, consider the quadratic form

in

n + 1 variables to, t1, ,P

=

.

.

•

n + 1

t,.. Evidently

};mi+;t,t;

(i, j

=

0, 1, 2, . ..n)

so that � .. is the determinant of ,P and �o, Ll 1, � .. -1 its principal minors. The form ,P cannot vanish unless to 0. t1 t,. For if x 0, we must have also � is a point of increase and q, •

•

•

=

=

·

·

·

=

=

=

=

. J:e+•(to + t1x + f-• for an arbitrary positive

(to + t11/ +

'

•

•

E1

·

·

+ t,.x")2drp(x)

·

=

0

whence by the mean value theorem

J:e+•drp(X) e-e

+ t,.f/")2

=

0 (� -

E

< 1/ < � + E)

or

to + tlf/ + . . . + t,..,.. = 0 because

J:e+•d�p(x) e-e Letting

E

>

0.

converge to O, we conclude to + lt � +

at any point of increase.

·

·

·

+ t,.�"

=

Since there are at least

0

n + 1 points of increase

the equation to + t1x + would have at least

·

·

·

+ t,.x"

=

0

n + 1 roots and that necessitates to

=

t1

=

·

·

·

=

t,.

=

0.

363

APPENDIX II

Hence, the quadratic form !/>, which is never negative, can vanish only if all its variables vanish; that is, !/> is a definite positive form.

determinant � .. and all its principal minors �n-t, �..-2, positive, which proves the first statement.

•

•

Its

�0 must be

•

Suppose the conditions

At> 0, ... A,.> 0

Ao> 0, satisfied and let

have 8 < n+ 1 points of increase.

cp(x)

Then the

integral representing !/> reduces to a finite sum

+ t..��)2 + + t..��)2 + P2(to + t1�2 + + + p ( to + h�. + + t,.�:)2 ·

·

denoting by p1, p 2 ,

•

•

•

·

·

·

,

·

•

·

p. masses concentrated in the 8 points of

increase �1, �2, �.. Now, since 8� n constants t0, t1, all zero, can be determined by the system of equations •

·

·

•

+ t..�� to + h�t + . . + to + tt�2 . . . + t..��

to + tt�• +

·

·

+ t..�:

·

=

=

=

•

•

•

t,., not

0 0 0.

Thus !/> vanishes when not all variables vanish; hence, its determinant

�..

=

0, contrary to hypothesis.

From now on we shall assume that increase.

cp(x)

has at least n +

1

points of

The integral

rb dcp(x) Ja Z- X can be expanded into a formal power series of

rb dcp(x) Ja z X -

=

1/z,

thus

m2n mo mt m2 ... + 2 +t + + + 2 + z3 zn z z

and this power series can be converted into a continued fraction as explained in Sec. 2. Let

Pt P 2 Q/ Q21 be the first n +

1

•

P,. P..+t ' Q,. Qn+J

convergents to that continued fraction.

degrees of their denominators are, respectively,

I say that the

1, 2, 3, ... n+ 1.

Since these degrees form an increasing sequence, it suffices to show that there exists a convergent with the denominator of a given degree

8�n+l. This convergent PjQ is completely determined by the condition that in a formal expansion of the difference

INTRODUCTION TO MATHEMATICAL PROBABILITY

364

ibdlp(X)X

_

1/z, terms involving 1/z, 1/z2,

into a power series of absent.

!:_ Q

a Z-

•

•

•

1/z2" are

This is the same as to say that in the expansion of

ibdfP(X)X- P(z)

Q(z)

a Z-

1/z, 1/z2,

there are no terms involving

•

•

•

1jz•.

The preceding expres­

sion can be written thus:

ibQ(x)dfP(x) ibQ(z)- Q(x)dlp(x)- P(z) - X - X +

a

a

Z

Z

=

__!._ z•+l

+

Since

i bQ(z)- XQ(x)dfP(x)- P(z) a

is a polynomial in

z,

it must vanish identically.

P(z}

(4) To determine

Z-

That gives

ibQ(z)- XQ(x)dfP(X).

=

a

Z-

Q(z) we must express the conditions that in the expansion of

ibQ�)�fP;X) terms in 8

1/z, 1/z2,

•

•

•

ljz•

vanish.

These conditions are equivalent to

relations

which in turn amount to the single requirement that

(6) 8(x) of degree � 8- 1. Q(z) of degree 8 satisfying con­ ditions (5), and P(z) is determined by equation (4), then P(z)/Q(z) is a

for an arbitrary polynomial

Conversely, if there exists a polynomial

convergent whose denominator is of degree

ibdfP(X)

_

z- x

lacks the terms in

1/z, 1/z2,

•

•

•

1/z2•.

8.

P(z) Q(z)

For then the expansion of

365

APPENDIX II

Let

Q(z) Then equations

=

lo + l1z + l,z2 +

·

·

+ l,_1z•-1 + z•.

·

(5) become

molo + m1l1 + m2l2 + m1lo + m2l1 + mal2 +

·

·

·

·

·

·

+ m,_1l,_, + m, + m,la-1 + m,+l

ma-1lo + m.l1 + ma+1l2 + . . . + m2a-2la-1 + m2a-1

=

=

=

0 0 0.

This system of linear equations determines completely the coefficients lo, l1, . . . la-1 since its determinant �•-1 > 0. The existence of a convergent with the denominator of degree 8�n+l being established, it follows that the denominator of the sth convergent P,jQ, is exactly of degree 8. The denominator Q. is determined, except for a constant factor, and can be presented in the form:

Q,

c =

-

�a-1

1 z z2 mo m1m2 m1 m2ma

•

•

·

•

• •

•

•

•

z• m, ma+1

A remarkable result follows from equation (6) by taking Q 8

=

Q,.; namely, if

(7) while

(8 � n). In the general relation

Q.

=

q.Qa-1 - Q,_ll

the polynomial q, must be of the first degree

q,

=

a,z + fJ,,

which shows that the continued fraction associated with

ibdrp(x)

z-x

=

Q, and

366

INTRODUCTION TO MATHEMATICAL PROBABILITY

has the form 1

a1Z + �1-

1

a:aZ + ,.,R 2-

1

aaz +�a---:---=-

The next question is, how to determine the constants plying both members of the equation

a,

and �..

Multi­

(s � 2) Q,_2drp(z), (7), we get

by

integrating between limits

0

=

a

and b, and taking into account

a.J:zQ,_1Q,_zdrp(z) - .CQL2drp(z).

On the other hand, the highest terms in

Q,_l and Q,_2 are

Hence,

zQ,_z

.

1

=

-Q,_1 + 1/1 ao-1

where ![I is a polynomial of degree �s

-

2.

Referring to equation (6),

we have

and consequently

(8)

a, aa-1

J:bQ;_2drp(z)

=

J:bQ;_ldrp(z).

mz,.; how m1, a, can be found? Evidently a1 1/mo. Fur­ thermore, Qo = 1 and Q1 is completely determined given mo and m1. Relation (8) determines a2, and Q2 will be completely determined given mo, m1, m2, m8. The same relation again determines a8, and Q3 will be determined giyen mo, m1, ... m5. Proceeding in the same way, we conclude that, given mo, m1, m2, ... m2,., all the polynomials' Suppose that the following moments are given: mo,

many of the coefficients

as

well as constants

•

=

.

.

367

APPENDIX II

can be determined.

It is important to note that all these constants are

positive. Proceeding in a similar manner, the following expression can be found

{3,

=

J:b zQ!-1d111(z).

-a,

(b Ja Q:_ldiP(Z)

It follows that constants

f311 f3s1

•

•

•

f3n For if B

are determined by our data, but not f3n+l·

=

n + 1, the integral

J:bzQ�diP(z) can be expressed as a linear function of mo, coefficients.

m1,

•

•

•

m2n+l with known

But msn+l is not included among our data; hence, f3n+1 ·

cannot be determined.

4.. Properties of Polynomials Q,.

Q,(z)

=

0

Theorem.

Roots of the equat{on

(s ;;!! n)

are real, simple, ancl. contained within the interval (a, b). Proof. Let Q,(z) change its sign r < B times when z passes through points z1, z2, Zr contained strictly within (a, b). Setting •

•

•

8(z)

=

(z - Zt)(Z - Z2)

•

•

•

(z - Zr)

the product

B(z)Q,(z) does not change its sign when

z increases from a to b.

J:8(z)Q,(z)d!l'(z)

=

However,

0,

and this necessitates that

B(z)Q,(z) or

Q,(z) vanishes in all points of increase of !II(Z). But this is impossible, n + 1 points of increase,· whereas the degree B of Q, does not exceed n. Consequently, Q,(z) changes its sign in the interval (a, b) exactly 8 times and has all its roots real, simple, and located within (a, b).

since by hypothesis there are at lea,st

It follows from this theorem that the convergent

Pn

Q,.

368

INTRODUCTION TO MATHEMATICAL PROBABILITY

can be resolved into a sum of simple fractions as follows:

P,.(z) =�+�+ . Q,.(z) z - Zt z - Zs

(9)

. z,. are

. +� z - z,.

.

roots of the equation

Q,.(z)

=

0

and in general

,. P�(zr.). A = Q,.(zr.) The right member of coefficient of

1/z"'

can be expanded into power series of

(9)

being

1/z,

the

"

�A..z�1•

a=l

By the property of convergents we must have the following equations: n

�A..

=

mo

=

m1

a�l "

�A ..z..

a=l n

�A..z!n-1 = msn-l•

a-1

These equations can be condensed into one, n

�A..T(z..) = J:T(z)drp(z)

(10)

a=l

which should hold for any polynomial Let us take for

T(z)

T(z)

of degree � 2n

-

1.

a polynomial of degree 2n - 2:

T(z) -

[ (z

Q,.(z) z..)Q�(z..)

-

]2

•

Then

T(z..)

=

1,

T(zfl)

·=

0

and consequently, by virtue of equation

A.. = Thus constants At, A2,

fb[ (z .

.

.

.

(10),

]2

Q,.(z) drp(z) > 0. z.. )Q�(z..)

_

A,.

if

are. all positive, which shows that

.

P,.(zr.)

369

APPENDIX II

has the same sign as

Q�(zr.).

Now in the sequence

Q�(zt), Q�(z2), ...Q�(z,.) any two consecutive terms are of opposite signs.

The same being true of

the sequence

P,.(zt), P,.(z2), ...P,.(z,.), P,.(z)

it follows that the roots of

are all simple, real, and located in the

intervals

(Zt1 Z2); (z2, Za); . ..(Zn-11 z,.). Finally, we shall prove the following theorem:

Theorem.

For any real x

Q�(x)Qn-t(X) - Q�_1(x)Q,.(x) is a positive number. Proof. From the relations

Q.(z) Q.(x)

=

=

(a.z + {3.)Qs-t(Z) - Q._2(z) (a.x + {3.)Q,_t(x) - Qa-2(x)

it follows that

Q.(z)Q.-t(X) - Q.(x)Qa-t(Z) =

z-x

a.Q._1(z)Q._1(x) + +

whence, takings

Q .-t (z) Q .-2 (x)

-

Q. _t (x)Qa- 2(z)

z-x

1, 2, 3, ... nand adding results,

=

•=1

It suffices now to take z

=

x to arrive at the identity. n

Q�(x)Q,._t(X) - Q:,_1(x)Q,.(x) Since

Q0

=

1 and

a.

>

=

! a.Q._ (x) 1

2•

•-1

0, it is evident that

Q�(x)Q,._t (X) - Q:,_1(x)Q,.(x)

>

0

for every real x.

6. Equivalent Point Distributions.

If the whole mass can be con­

centrated in a finite number of points so as to produce the same l first moments as a given distribution, we have an "equivalent point distribu-

.

·370

INTRODUCTION TO MATHEMATICAL PROBABILITY

tion" in respect to the l first moments.

In what follows we shall suppose

that the whole mass is spread over an infinite interval

- oo,

oo

and that

the given moments, originating in a distribution with at least n + 1 points of increase, are

The question is: Is it possible to find an equivalent point distribution where the whole mass is concentrated inn + 1 points?

Let the unknown

points be ��. ��. . . . �..+1 and the masses concentrated in them

At, A2, ... A,.+l· Evidently the question will be answered in the affirmative if the system of 2n + 1 equations

n+l

�

Aa a=l n+l Aa�a a=l n+l Aae: a=l

�

(A)

�

=

=

=

mo m1 ms

n+l

� AaE!"

=

a=l

m2,.

can be satisfied by real numbers E1, � � • . . • �..+Ji A1, A2, • • • An+11 the last n + 1 numbers being positive. The number of unknowns being greater by one unit than the number of equations, we can introduce the additional requirement that one of the numbers ��. ��. . . . �..+1 should be equal to a given real number v.

(A) may be replaced by

The system

the single requirement that the equation

n+l (11)

� AaT(�a) f_"'.., T(x)dlf'(X) =

a-1

shall hold for any polynomial

T(x) of degree �2n. Let Q(x) be the �n+l and let B(x) be e2,

polynomial of degree n + 1 having roots��. an arbitrary polynomial of degree �n

-

1.

(11) to

T(x)

=

B(x)Q(x).

•

.

•

Then we can apply equation

APPENDIX II

Since

QU..)

(12)

= 0, we shall have J_"'.., 8(x)Q(x)d�P(x)

for an arbitrary polynomial see that requirement

(12)

371

0

=

8(x) of degree �n - 1. Presently we shall Q(v) 0 determines Q(x), save

=

together with

for a constant factor if

Q,.(v) ¢ 0. Dividing

Q(x) by Q,.(x), we have identically Q(x)

where

=

(Xx + p.)Q,.(x) + R,.-I(x)

R,._I(x) is a polynomial of degree �n - 2,

-

1.

If

8(x) is an arbi­

trary polynomial of degree �n

(Xx + p.)8(x) will be of degree �n

- 1.

Hence

.f<xx + p.)8(x)Q,.(x)diP(x) = 0 by

(6),

and

(12)

shows that

Lb8(x)R,._I(x)d�P(X)

=

0

8(x) of degree �n - 2. The last require­ R,.-I(x) differs from Q,._I(x) by a constant factor. Since the highest coefficient in Q(x) is arbitrary, we can set

for an arbitrary polynomial ment shows that

Rn-J(x)

=

-Q,._I(x).

In the equation

Q(x)

=

(Xx + p.)Q,.(x) - Q,._I(x)

X and p.. Multiplying both members by Q,._1(x)dfP(X) and integrating between - oo and oo, we get

it remains to determine constants

X _"'..,xQ,._lQndfP(X)

J

or

But

=

J_"',. Q!...1d!p(X)

372

INTRODUCTION TO MATHEMATICAL PROBABILITY

whence The equation

Q(v) (an+tV p,)Q,.(v) - Q,._t(V) serves to determine p, if Qn(v) The final expression of Q(x) will be Q(x) (an+t(X - v) QQ:(;)>)Q,.(x) - Q,._1(x). 0

+

=

=

¢ 0.

+

=

Owing to recurrence relations

Q2 (a2X f32)Q1 - Qo; Qs (asx f3s)Q2 - Q1; Q,. (a,.X (3,.)Qn-l - Qn-21 it is evident that Q, Q,., Q,._J, . Qt, Qo 1 +

=

+

=

·

+

=

·

x

in a Sturm series. For = oo only permanences.

x

=

·

1

- oo, it contains n + variations and for It follows that the equation

=

Q(x)

1

=

0

v.

has exactly n + distinct real roots and among them Thus, if the is solvable, the numbers h, , determined as problem + are roots of

Q(x)

�2

=

•

•

•

�n l

0.

Furthermore, all unknowns Aa will be positive. it follows that

(11)

In fact, from equation

a J_.. [<x _ ��)b'(�a)rd�(x) > o. .. Now we must show that constants Aa can actually be determined so A

=

to satisfy equations (A).

as

To this end let

_ P(x) f_.... Q(x) - Q(z) d�(z) [an+t(X- v) + Q,.Q,.(tv(v))JPn(X)- Pn-t(x). Then Q(x) J_.... :�z�- P(x) J_..,. Q�)��;z) =

X_ Z

=

=

(12),

and, on account of the expansion of the right member into power series of lacks the terms in Hence, the expan­ sion of

1/x

1/x, 1/x2, 1/x". f .... d�(z) P(x) _ x- z Q(x) •

_

•

•

373

APPENDIX II

lacks the terms in

P(x) Q(x)

1/x, 1/x2, =

mo

+

x

•

•

mt

•

1/x2"+1;

+ ... +

x2

that is, m2,. + x2"+1

On the other hand, resolving in simple fractions,

P(x) Q(x)

�

=

+

�

n

A +t . X - �n+l

+ ... +

X - �2

X - �1

1/x (A).

Expanding the right member into power series of

and comparing

with the preceding expansion, we obtain the system

By the previous

remark all constants A .. are positive.

Thus, there exists a point distribu­

tion in which masses concentrated in

mo,

m1, . . . m2,..

+

n

1

points produce moments

One of these points v may be taken arbitrarily, with

the condition

Q,. (v)

¢

0

being observed, however.

6. Tshebysheff's Inequalities.

In a note referred to in the introduc­

tion Tshebysheff made known certain inequalities of the utmost impor­ tance for the theory we are concerned with. proof of them was given by Markoff in

The first very ingenious

1884

and, by a remarkable

coincidence, the same proof was rediscovered almost at the same time by Stieltjes.

A few years later, Stieltjes found another totally different

proof; and it is this second proof that we shall follow. Let

-

oo,

IP(x)

oo.

be a distribution function of a mass spread over the interval

Supposing that a moment of the order i,

J_..

x'diP(x)

=

m;,

..

exists, we shall show first that lim

when

l

tends to

+ oo

.

Similarly

= =

0 0

For

J, .. xidiP(x) or

li(mo - IP(l)) lim liiP( -l)

f, .. diP(X)

!1;; li

=

li[IP(

+ oo

) - IP(Z)]

374

INTRODUCTION TO MATHEMATICAL PROBABILITY

or

Now both integrals

converge to 0 as ately.

l tends to + oo; whence both statements follow immedi­

Integrating by parts, we have

J:xidrp(x) = li[rp(l)- mo] - i.fo1[rp(x) - m0]xi-1dx fo xidrp(x) = ( -1)i-llirp(-l)- iJ-01xi-1rp(x)dx, -l

whence, letting

l converge to + oo, '

=

m;

f .. xidrp(x) =-iJro .. [rp(x)- mo]xi-ldx- if 0 -�

xi-lrp(x)dx.

-�

If the same mass

mo, with the same moment m;, is spread according to 1/t(x), we shall have

the law characterized by the function m;

�

J_.... xidl/t(x) = -ifo ..[l/t(x) - mo]xi-1dx- if� .. xi-11/t(x)dx,

whence

J_....xi-l[rp(x)- 1/t(x)]dx = 0.

(13)

Suppose the moments

rp(x) are known. Provided rp(x) + 1 points of increase, there exists an equivalent point

of the distribution characterized by ha.

dO dv

v < 0;

if

O

-
O

if

O

= 0 and :Xn(v) attains its maximum C2m, c2m+li H2m(O),

Referring to the above expressions of we find that

2 4·6··· 2m 3· 5 · 7 ··· ( 2m + 1) 2·4·6··· 2m :X2m+l(O) = • 3· 5 · 7 ··· ( 2m + 1) X2m(O) =

354, we find the inequality

In Appendix I, page

·

2·4·6 .. · 2m 1 .y; -: :----.---: -"'"F.'====:=� < 2 5;:;·- ··---= 1) ( 2m 1:;--· 3;oV 4m + 2 -

-

·

whence

•

-

2·4·6···2m 5 7 ( 2m + 1)

��=-����=
..

Suppose that each one of them, when instance

oo

),

tends to a certain limit (for

tends to the corresponding moment of normal distribution.

One can foresee that under such circumstances

1p(x)

F(x, >..)

"' e-u•du. J v;

will tend to

1

=

--

-oo

In fact, the following fundamental theorem holds:

If, for a variable distribution characterized

Fundamental Theorem.

by the function F(x, >..) , lim

for any fixed

J_

k

..

=

xkdF(x, >..)

=

�I-'" e-"'•xkdx;

..

0, 1,

lim

>..�

oo

..

2,

3,

F(v, >..)

..., then =

1 _ ,r: V 'll"

J

"

>..�

e-"'"dx;

oo

-oo

uniformly in v. Proof.

be

2n +

Let

1 moments corresponding to a normal distribution.

They

allow formation of the polynomials

Qo(x), Q1(x), ... Qn(x) and the function designated in Sec. 6 by

and

Q(x)

1/l(x).

Similar entities cor­

responding to the variable distribution will be specified by an asterisk. Since as

and since .!ln > O, we shall have

.1! for sufficiently large

>...

Then

> 0

F(x, >..)

will have not less than

n+1

points of increase and the whole theory can be applied to variable dis­ tribution.

In particular, we shall have

0 ;;! 1p(v) - 1/l(v

-

0) ;;! Xn(v)

(20) 0 ;;! F(v, >..) - 1/l*(v - 0) ;;! x!(v). Now

Q �(x) (s

=

0,

1,

2, ... n)

and

Q*(x)

depend rationally upon

385

APPENDIX II

mt(k

=

0,

1,

. 2n);

2,

hence, without any difficulty one can see that

Q!(x) --+ Q,(x); s 0, 1, 2, ... n Q*(x) --+ Q(x) =

as >.. --+ oo ; whence,

x:(v)

--+ x.. (v).

Again

1/t*(v

-

O)

--+

1/t(v

0)

-

as >.. --+ oo.

A few explanations are necessary to prove this.

Q,.(v) ¢

Then the polynomial

0.

�1

�2




---+

0.

B!+2 Liapounoff's result in regard to generality of conditions surpassed by far what had been established before by Tshebysheff and Markoff, whose proofs were based on the fundamental result derived in the preceding sec­ Since Liapounoff's conditions do not require the existence of

tion.

moments in an infinite number, it seemed that the method of moments was not powerful enough to establish the limit theorem in such a general form. ·

Nevertheless, by resorting to an ingenious artifice, of which we

made use in Chap. X, Sec. 8, Markoff finally succeeded in proving the limit theorem by the method of moments to the same degree of generality as did Liapounoff.

389

APPENDIX II

Markoff's artifice consists in associating with the variable

x, andy,. defined as follows:

variables

z, two new

Let N be a positive number which in the course of proof will be selected so as to tend to infinity together with

n.

Then

if if Evidently

z,., x,, y,. are connected by the relation z,. x, +y,. =

whence

E(x,) + E(y,.)

(21)

=

0.

Moreover

(22)

Elx�ol*'

+

EIYki2H

=

Elz,.l*'

I'L2+11,

=

x, and y,. x,. is bounded, mathematical expectations E(xt)

as one can see immediately from the definition of Since

exist for all integer exponents l

l, 2, 3, . . . and fork

=

=

1, 2, 3,

In the following we shall use the notations

IE(xDI

cL11; l 1, 2, 3, . . . c�2> + c�2> + . . . + c�2> B� 1'�2+1> + 1'�2+8> + . . . + 1'�2+6) c... =

=

=

=

Not to obscure the essential steps of the reasoning we shall first establish a few preliminary results.

Lemma 1.

Let

q,

represent the probability thaty, � . < c.. + q.. . q1 + q2 + ·

·

·

=

Proof. only if

lz�ol

Let

IP�,

Since y,. � 0

390

INTRODUCTION TO MATHEMATICAL PROBABILITY

But

J:

- lxl2+6d + c�•> +

·

·

•

+ c�>

B! tends to 0 for all positive integer exponents e except e = 2. 10. Let F,.(t) and q,,.(t) represent, respectively, the probabilities of the inequalities

.

+ Z2 + ··· + Z,. -v'2B.: X1 + X2 + ' ' ' + X,. -v'2B.: Z1

t


. + .

.

Apparently

w.

B" n

B"' n Besides

and

if not all subscripts X, p.,

But by Corollary

•

•

w

•

are equal to 1.

It follows that

2 B' _.!!

�

B..

and evidently H1,1,

•

•

1

G2,2,

= (;)I -

•

G2,2,

Hence

2·

;! · 2 �

•

•

1 .

B ..

1

(i}

and this in connection with (23) shows that for an even m

( +x. �. + "J �

E "'

2

•

·

Finally, no matter whether the exponent m is odd or even, we have

( + +Inn' ' ' + )

• X1 IliD E

X2

v2B,. _

Xn m

=

_1_

� V1T _

.. J .. -

X me- .,.dX.

APPENDIX II

395

Tshebysheff-Markoff's fundamental theorem can be applied directly and leads to the result: lim t/>n (t) uniformly in

t. t.

r:. V'll" _

f'

e-"'•dx oo

-

On the other hand, as has been established before, lim

uniformly in

1

=

[F,.(t) - q,,.(t)]

Hence, finally lim

F,.(t)

1

=

r:. V'll" _

f' -

=

0

e-"•dx oo

uniformly in t. And this is the fundamental limit tJ?.eorem with Liapounoff's condi­ tions now proved by the method of moments. Markoff, is simple enough and of high elegance.

This proof, due to However, preliminary

considerations which underlie the proof of the fundamental theorem, though simple and elegant also, are rather long.

Nevertheless, we must

bear in mind that they are not only useful in connection with the theory of probability, but they have great importance in other fields of analysis.

APPENDIX III ON A GAUSSIAN PROBLEM 1. In a letter to Laplace dated January 30, 1812,1 Gauss mentions a difficult problem in probability for which he could not find a perfectly satisfactory solution.

We quote from his letter:

Je me rappelle pourtant d'un probleme curieux duquel je me suis occup� il y a

12 ans,

mais lequel je n'ai pas r�ussi alors a r�soudre a rna satisfaction.

Peut­

�tre daignerez-vous en occuper quelques moments: dans ce cas je suis sur que vous La voici: Soit M une quantit� inconnue

trouverez une solution plus complete. entre les limites 0 et

1 pour laquelle toutes les valeurs sont ou �galement probabies

ou plus ou moins selon une loi donn�e: qu'on Ia suppose convertie en une fraction continue 1

M=-, + a

1 a"+

Quelle est la probabilite qu'en s'arr�tant dans le developpement a un terme fini > a