Introduction to Probability Models Ninth Edition
This page intentionally left blank
Introduction to Probability Models Ninth Edition Sheldon M. Ross University of California Berkeley, California
AMSTERDAM • BOSTON • HEIDELBERG • LONDON NEW YORK • OXFORD • PARIS • SAN DIEGO SAN FRANCISCO • SINGAPORE • SYDNEY • TOKYO Academic Press is an imprint of Elsevier
Acquisitions Editor Project Manager Marketing Managers Cover Design Composition Cover Printer Interior Printer
Tom Singer Sarah M. Hajduk Linda Beattie, Leah Ackerson Eric DeCicco VTEX Phoenix Color The Maple-Vail Book Manufacturing Group
Academic Press is an imprint of Elsevier 30 Corporate Drive, Suite 400, Burlington, MA 01803, USA 525 B Street, Suite 1900, San Diego, California 92101-4495, USA 84 Theobald’s Road, London WC1X 8RR, UK ∞ This book is printed on acid-free paper.
Copyright © 2007, Elsevier Inc. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the publisher. Permissions may be sought directly from Elsevier’s Science & Technology Rights Department in Oxford, UK: phone: (+44) 1865 843830, fax: (+44) 1865 853333, E-mail:
[email protected]. You may also complete your request on-line via the Elsevier homepage (http://elsevier.com), by selecting “Support & Contact” then “Copyright and Permission” and then “Obtaining Permissions.” Library of Congress Cataloging-in-Publication Data Application Submitted British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library. ISBN-13: 978-0-12-598062-3 ISBN-10: 0-12-598062-0 For information on all Academic Press publications visit our Web site at www.books.elsevier.com Printed in the United States of America 06 07 08 09 10 9 8 7 6
5
4
3
2
1
Contents
Preface xiii
1. Introduction to Probability Theory 1 1.1. 1.2. 1.3. 1.4. 1.5. 1.6.
Introduction 1 Sample Space and Events 1 Probabilities Defined on Events 4 Conditional Probabilities 7 Independent Events 10 Bayes’ Formula 12 Exercises 15 References 21
2. Random Variables 23 2.1. Random Variables 23 2.2. Discrete Random Variables 27 2.2.1. The Bernoulli Random Variable 28 2.2.2. The Binomial Random Variable 29 2.2.3. The Geometric Random Variable 31 2.2.4. The Poisson Random Variable 32 2.3. Continuous Random Variables 34 2.3.1. The Uniform Random Variable 35 2.3.2. Exponential Random Variables 36 2.3.3. Gamma Random Variables 37 2.3.4. Normal Random Variables 37
v
vi
Contents
2.4. Expectation of a Random Variable 38 2.4.1. The Discrete Case 38 2.4.2. The Continuous Case 41 2.4.3. Expectation of a Function of a Random Variable 43 2.5. Jointly Distributed Random Variables 47 2.5.1. Joint Distribution Functions 47 2.5.2. Independent Random Variables 51 2.5.3. Covariance and Variance of Sums of Random Variables 53 2.5.4. Joint Probability Distribution of Functions of Random Variables 61 2.6. Moment Generating Functions 64 2.6.1. The Joint Distribution of the Sample Mean and Sample Variance from a Normal Population 74 2.7. Limit Theorems 77 2.8. Stochastic Processes 83 Exercises 85 References 96
3. Conditional Probability and Conditional Expectation 97 3.1. 3.2. 3.3. 3.4.
Introduction 97 The Discrete Case 97 The Continuous Case 102 Computing Expectations by Conditioning 105 3.4.1. Computing Variances by Conditioning 117 3.5. Computing Probabilities by Conditioning 120 3.6. Some Applications 137 3.6.1. A List Model 137 3.6.2. A Random Graph 139 3.6.3. Uniform Priors, Polya’s Urn Model, and Bose–Einstein Statistics 147 3.6.4. Mean Time for Patterns 151 3.6.5. The k-Record Values of Discrete Random Variables 155 3.7. An Identity for Compound Random Variables 158 3.7.1. Poisson Compounding Distribution 161 3.7.2. Binomial Compounding Distribution 163 3.7.3. A Compounding Distribution Related to the Negative Binomial 164 Exercises 165
Contents
4. Markov Chains 185 4.1. 4.2. 4.3. 4.4. 4.5.
4.6. 4.7. 4.8. 4.9. 4.10. 4.11.
Introduction 185 Chapman–Kolmogorov Equations 189 Classification of States 193 Limiting Probabilities 204 Some Applications 217 4.5.1. The Gambler’s Ruin Problem 217 4.5.2. A Model for Algorithmic Efficiency 221 4.5.3. Using a Random Walk to Analyze a Probabilistic Algorithm for the Satisfiability Problem 224 Mean Time Spent in Transient States 230 Branching Processes 233 Time Reversible Markov Chains 236 Markov Chain Monte Carlo Methods 247 Markov Decision Processes 252 Hidden Markov Chains 256 4.11.1. Predicting the States 261 Exercises 263 References 280
5. The Exponential Distribution and the Poisson Process 281 5.1. Introduction 281 5.2. The Exponential Distribution 282 5.2.1. Definition 282 5.2.2. Properties of the Exponential Distribution 284 5.2.3. Further Properties of the Exponential Distribution 291 5.2.4. Convolutions of Exponential Random Variables 298 5.3. The Poisson Process 302 5.3.1. Counting Processes 302 5.3.2. Definition of the Poisson Process 304 5.3.3. Interarrival and Waiting Time Distributions 307 5.3.4. Further Properties of Poisson Processes 310 5.3.5. Conditional Distribution of the Arrival Times 316 5.3.6. Estimating Software Reliability 328 5.4. Generalizations of the Poisson Process 330 5.4.1. Nonhomogeneous Poisson Process 330 5.4.2. Compound Poisson Process 337 5.4.3. Conditional or Mixed Poisson Processes 343
vii
viii
Contents
Exercises 346 References 364
6. Continuous-Time Markov Chains 365 6.1. 6.2. 6.3. 6.4. 6.5. 6.6. 6.7. 6.8.
Introduction 365 Continuous-Time Markov Chains 366 Birth and Death Processes 368 The Transition Probability Function Pij (t) 375 Limiting Probabilities 384 Time Reversibility 392 Uniformization 401 Computing the Transition Probabilities 404 Exercises 407 References 415
7. Renewal Theory and Its Applications 417 7.1. 7.2. 7.3. 7.4. 7.5. 7.6. 7.7. 7.8. 7.9.
7.10.
Introduction 417 Distribution of N (t) 419 Limit Theorems and Their Applications 423 Renewal Reward Processes 433 Regenerative Processes 442 7.5.1. Alternating Renewal Processes 445 Semi-Markov Processes 452 The Inspection Paradox 455 Computing the Renewal Function 458 Applications to Patterns 461 7.9.1. Patterns of Discrete Random Variables 462 7.9.2. The Expected Time to a Maximal Run of Distinct Values 469 7.9.3. Increasing Runs of Continuous Random Variables 471 The Insurance Ruin Problem 473 Exercises 479 References 492
8. Queueing Theory 493 8.1. Introduction 493 8.2. Preliminaries 494 8.2.1. Cost Equations 495 8.2.2. Steady-State Probabilities 496
Contents
8.3. Exponential Models 499 8.3.1. A Single-Server Exponential Queueing System 499 8.3.2. A Single-Server Exponential Queueing System Having Finite Capacity 508 8.3.3. A Shoeshine Shop 511 8.3.4. A Queueing System with Bulk Service 514 8.4. Network of Queues 517 8.4.1. Open Systems 517 8.4.2. Closed Systems 522 8.5. The System M/G/1 528 8.5.1. Preliminaries: Work and Another Cost Identity 528 8.5.2. Application of Work to M/G/1 529 8.5.3. Busy Periods 530 8.6. Variations on the M/G/1 531 8.6.1. The M/G/1 with Random-Sized Batch Arrivals 531 8.6.2. Priority Queues 533 8.6.3. An M/G/1 Optimization Example 536 8.6.4. The M/G/1 Queue with Server Breakdown 540 8.7. The Model G/M/1 543 8.7.1. The G/M/1 Busy and Idle Periods 548 8.8. A Finite Source Model 549 8.9. Multiserver Queues 552 8.9.1. Erlang’s Loss System 553 8.9.2. The M/M/k Queue 554 8.9.3. The G/M/k Queue 554 8.9.4. The M/G/k Queue 556 Exercises 558 References 570
9. Reliability Theory 571 9.1. Introduction 571 9.2. Structure Functions 571 9.2.1. Minimal Path and Minimal Cut Sets 574 9.3. Reliability of Systems of Independent Components 578 9.4. Bounds on the Reliability Function 583 9.4.1. Method of Inclusion and Exclusion 584 9.4.2. Second Method for Obtaining Bounds on r(p) 593 9.5. System Life as a Function of Component Lives 595 9.6. Expected System Lifetime 604 9.6.1. An Upper Bound on the Expected Life of a Parallel System 608
ix
x
Contents
9.7. Systems with Repair 610 9.7.1. A Series Model with Suspended Animation 615 Exercises 617 References 624
10. Brownian Motion and Stationary Processes 625 10.1. Brownian Motion 625 10.2. Hitting Times, Maximum Variable, and the Gambler’s Ruin Problem 629 10.3. Variations on Brownian Motion 631 10.3.1. Brownian Motion with Drift 631 10.3.2. Geometric Brownian Motion 631 10.4. Pricing Stock Options 632 10.4.1. An Example in Options Pricing 632 10.4.2. The Arbitrage Theorem 635 10.4.3. The Black-Scholes Option Pricing Formula 638 10.5. White Noise 644 10.6. Gaussian Processes 646 10.7. Stationary and Weakly Stationary Processes 649 10.8. Harmonic Analysis of Weakly Stationary Processes 654 Exercises 657 References 662
11. Simulation 663 11.1. Introduction 663 11.2. General Techniques for Simulating Continuous Random Variables 668 11.2.1. The Inverse Transformation Method 668 11.2.2. The Rejection Method 669 11.2.3. The Hazard Rate Method 673 11.3. Special Techniques for Simulating Continuous Random Variables 677 11.3.1. The Normal Distribution 677 11.3.2. The Gamma Distribution 680 11.3.3. The Chi-Squared Distribution 681 11.3.4. The Beta (n, m) Distribution 681 11.3.5. The Exponential Distribution—The Von Neumann Algorithm 682 11.4. Simulating from Discrete Distributions 685 11.4.1. The Alias Method 688
Contents
11.5. Stochastic Processes 692 11.5.1. Simulating a Nonhomogeneous Poisson Process 693 11.5.2. Simulating a Two-Dimensional Poisson Process 700 11.6. Variance Reduction Techniques 703 11.6.1. Use of Antithetic Variables 704 11.6.2. Variance Reduction by Conditioning 708 11.6.3. Control Variates 712 11.6.4. Importance Sampling 714 11.7. Determining the Number of Runs 720 11.8. Coupling from the Past 720 Exercises 723 References 731
Appendix: Solutions to Starred Exercises 733 Index 775
xi
This page intentionally left blank
Preface
This text is intended as an introduction to elementary probability theory and stochastic processes. It is particularly well suited for those wanting to see how probability theory can be applied to the study of phenomena in fields such as engineering, computer science, management science, the physical and social sciences, and operations research. It is generally felt that there are two approaches to the study of probability theory. One approach is heuristic and nonrigorous and attempts to develop in the student an intuitive feel for the subject which enables him or her to “think probabilistically.” The other approach attempts a rigorous development of probability by using the tools of measure theory. It is the first approach that is employed in this text. However, because it is extremely important in both understanding and applying probability theory to be able to “think probabilistically,” this text should also be useful to students interested primarily in the second approach.
New to This Edition The ninth edition contains the following new sections. • Section 3.7 is concerned with compound random variables of the form SN = N i=1 Xi , where N is independent of the sequence of independent and identically distributed random variables Xi , i 1. It starts by deriving a general identity concerning compound random variables, as well as a corollary of that identity in the case where the Xi are positive and integer valued. The corollary is then used in subsequent subsections to obtain recursive formulas for the probability mass function of SN , when N is a Poisson distribution (Subsection 3.7.1), a binomial distribution (Subsection 3.7.2), or a negative binomial distribution (Subsection 3.7.3).
xiii
xiv
Preface
• Section 4.11 deals with hidden Markov chains. These models suppose that a random signal is emitted each time a Markov chain enters a state, with the distribution of the signal depending on the state entered. The Markov chain is hidden in the sense that it is supposed that only the signals and not the underlying states of the chain are observable. As part of our analysis of these models we present, in Subsection 4.11.1, the Viterbi algorithm for determining the most probable sequence of first n states, given the first n signals. • Section 8.6.4 analyzes the Poisson arrival single server queue under the assumption that the working server will randomly break down and need repair. There is also new material in almost all chapters. Some of the more significant additions being the following. • Example 5.9, which is concerned with the expected number of normal cells that survive until all cancer cells have been killed. The example supposes that each cell has a weight, and the probability that a given surviving cell is the next cell killed is proportional to its weight. • A new approach—based on time sampling of a Poisson process—is presented in Subsection 5.4.1 for deriving the probability mass function of the number of events of a nonhomogeneous Poisson process that occur in any specified time interval. • There is additional material in Section 8.3 concerning the M/M/1 queue. Among other things, we derive the conditional distribution of the number of customers originally found in the system by a customer who spends a time t in the system before departing. (The conditional distribution is Poisson.) In Example 8.3, we illustrate the inspection paradox, by obtaining the probability distribution of the number in the system as seen by the first arrival after some specified time.
Course Ideally, this text would be used in a one-year course in probability models. Other possible courses would be a one-semester course in introductory probability theory (involving Chapters 1–3 and parts of others) or a course in elementary stochastic processes. The textbook is designed to be flexible enough to be used in a variety of possible courses. For example, I have used Chapters 5 and 8, with smatterings from Chapters 4 and 6, as the basis of an introductory course in queueing theory.
Preface
xv
Examples and Exercises Many examples are worked out throughout the text, and there are also a large number of exercises to be solved by students. More than 100 of these exercises have been starred and their solutions provided at the end of the text. These starred problems can be used for independent study and test preparation. An Instructor’s Manual, containing solutions to all exercises, is available free to instructors who adopt the book for class.
Organization Chapters 1 and 2 deal with basic ideas of probability theory. In Chapter 1 an axiomatic framework is presented, while in Chapter 2 the important concept of a random variable is introduced. Subsection 2.6.1 gives a simple derivation of the joint distribution of the sample mean and sample variance of a normal data sample. Chapter 3 is concerned with the subject matter of conditional probability and conditional expectation. “Conditioning” is one of the key tools of probability theory, and it is stressed throughout the book. When properly used, conditioning often enables us to easily solve problems that at first glance seem quite difficult. The final section of this chapter presents applications to (1) a computer list problem, (2) a random graph, and (3) the Polya urn model and its relation to the BoseEinstein distribution. Subsection 3.6.5 presents k-record values and the surprising Ignatov’s theorem. In Chapter 4 we come into contact with our first random, or stochastic, process, known as a Markov chain, which is widely applicable to the study of many realworld phenomena. Applications to genetics and production processes are presented. The concept of time reversibility is introduced and its usefulness illustrated. Subsection 4.5.3 presents an analysis, based on random walk theory, of a probabilistic algorithm for the satisfiability problem. Section 4.6 deals with the mean times spent in transient states by a Markov chain. Section 4.9 introduces Markov chain Monte Carlo methods. In the final section we consider a model for optimally making decisions known as a Markovian decision process. In Chapter 5 we are concerned with a type of stochastic process known as a counting process. In particular, we study a kind of counting process known as a Poisson process. The intimate relationship between this process and the exponential distribution is discussed. New derivations for the Poisson and nonhomogeneous Poisson processes are discussed. Examples relating to analyzing greedy algorithms, minimizing highway encounters, collecting coupons, and tracking the AIDS virus, as well as material on compound Poisson processes, are included
xvi
Preface
in this chapter. Subsection 5.2.4 gives a simple derivation of the convolution of exponential random variables. Chapter 6 considers Markov chains in continuous time with an emphasis on birth and death models. Time reversibility is shown to be a useful concept, as it is in the study of discrete-time Markov chains. Section 6.7 presents the computationally important technique of uniformization. Chapter 7, the renewal theory chapter, is concerned with a type of counting process more general than the Poisson. By making use of renewal reward processes, limiting results are obtained and applied to various fields. Section 7.9 presents new results concerning the distribution of time until a certain pattern occurs when a sequence of independent and identically distributed random variables is observed. In Subsection 7.9.1, we show how renewal theory can be used to derive both the mean and the variance of the length of time until a specified pattern appears, as well as the mean time until one of a finite number of specified patterns appears. In Subsection 7.9.2, we suppose that the random variables are equally likely to take on any of m possible values, and compute an expression for the mean time until a run of m distinct values occurs. In Subsection 7.9.3, we suppose the random variables are continuous and derive an expression for the mean time until a run of m consecutive increasing values occurs. Chapter 8 deals with queueing, or waiting line, theory. After some preliminaries dealing with basic cost identities and types of limiting probabilities, we consider exponential queueing models and show how such models can be analyzed. Included in the models we study is the important class known as a network of queues. We then study models in which some of the distributions are allowed to be arbitrary. Included are Subsection 8.6.3 dealing with an optimization problem concerning a single server, general service time queue, and Section 8.8, concerned with a single server, general service time queue in which the arrival source is a finite number of potential users. Chapter 9 is concerned with reliability theory. This chapter will probably be of greatest interest to the engineer and operations researcher. Subsection 9.6.1 illustrates a method for determining an upper bound for the expected life of a parallel system of not necessarily independent components and (9.7.1) analyzing a series structure reliability model in which components enter a state of suspended animation when one of their cohorts fails. Chapter 10 is concerned with Brownian motion and its applications. The theory of options pricing is discussed. Also, the arbitrage theorem is presented and its relationship to the duality theorem of linear program is indicated. We show how the arbitrage theorem leads to the Black–Scholes option pricing formula. Chapter 11 deals with simulation, a powerful tool for analyzing stochastic models that are analytically intractable. Methods for generating the values of arbitrarily distributed random variables are discussed, as are variance reduction methods for increasing the efficiency of the simulation. Subsection 11.6.4 introduces the
Preface
xvii
important simulation technique of importance sampling, and indicates the usefulness of tilted distributions when applying this method.
Acknowledgments We would like to acknowledge with thanks the helpful suggestions made by the many reviewers of the text. These comments have been essential in our attempt to continue to improve the book and we owe these reviewers, and others who wish to remain anonymous, many thanks: Mark Brown, City University of New York Tapas Das, University of South Florida Israel David, Ben-Gurion University Jay Devore, California Polytechnic Institute Eugene Feinberg, State University of New York, Stonybrook Ramesh Gupta, University of Maine Marianne Huebner, Michigan State University Garth Isaak, Lehigh University Jonathan Kane, University of Wisconsin Whitewater Amarjot Kaur, Pennsylvania State University Zohel Khalil, Concordia University Eric Kolaczyk, Boston University Melvin Lax, California State University, Long Beach Jean Lemaire, University of Pennsylvania George Michailidis, University of Michigan Krzysztof Osfaszewski, University of Illinois Erol Pekoz, Boston University
xviii
Preface
Evgeny Poletsky, Syracuse University Anthony Quas, University of Victoria David Scollnik, University of Calgary Mary Shepherd, Northwest Missouri State University Galen Shorack, University of Washington, Seattle Osnat Stramer, University of Iowa Gabor Szekeley, Bowling Green State University Marlin Thomas, Purdue University Zhenyuan Wang, University of Binghamton Julie Zhou, University of Victoria Special thanks to Donald Minassian of Butler University, for his extremely careful reading of the text.
Introduction to Probability Theory
1 1.1.
Introduction
Any realistic model of a real-world phenomenon must take into account the possibility of randomness. That is, more often than not, the quantities we are interested in will not be predictable in advance but, rather, will exhibit an inherent variation that should be taken into account by the model. This is usually accomplished by allowing the model to be probabilistic in nature. Such a model is, naturally enough, referred to as a probability model. The majority of the chapters of this book will be concerned with different probability models of natural phenomena. Clearly, in order to master both the “model building” and the subsequent analysis of these models, we must have a certain knowledge of basic probability theory. The remainder of this chapter, as well as the next two chapters, will be concerned with a study of this subject.
1.2.
Sample Space and Events
Suppose that we are about to perform an experiment whose outcome is not predictable in advance. However, while the outcome of the experiment will not be known in advance, let us suppose that the set of all possible outcomes is known. This set of all possible outcomes of an experiment is known as the sample space of the experiment and is denoted by S. Some examples are the following. 1. If the experiment consists of the flipping of a coin, then S = {H, T } where H means that the outcome of the toss is a head and T that it is a tail.
1
2
1
Introduction to Probability Theory
2. If the experiment consists of rolling a die, then the sample space is S = {1, 2, 3, 4, 5, 6} where the outcome i means that i appeared on the die, i = 1, 2, 3, 4, 5, 6. 3. If the experiments consists of flipping two coins, then the sample space consists of the following four points: S = {(H, H ), (H, T ), (T , H ), (T , T )} The outcome will be (H, H ) if both coins come up heads; it will be (H, T ) if the first coin comes up heads and the second comes up tails; it will be (T , H ) if the first comes up tails and the second heads; and it will be (T , T ) if both coins come up tails. 4. If the experiment consists of rolling two dice, then the sample space consists of the following 36 points: ⎧ ⎫ (1, 1), (1, 2), (1, 3), (1, 4), (1, 5), (1, 6)⎪ ⎪ ⎪ ⎪ ⎪ ⎪(2, 1), (2, 2), (2, 3), (2, 4), (2, 5), (2, 6)⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎬ (3, 1), (3, 2), (3, 3), (3, 4), (3, 5), (3, 6) S= (4, 1), (4, 2), (4, 3), (4, 4), (4, 5), (4, 6)⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ (5, 1), (5, 2), (5, 3), (5, 4), (5, 5), (5, 6)⎪ ⎪ ⎪ ⎪ ⎪ ⎩ ⎭ (6, 1), (6, 2), (6, 3), (6, 4), (6, 5), (6, 6) where the outcome (i, j ) is said to occur if i appears on the first die and j on the second die. 5. If the experiment consists of measuring the lifetime of a car, then the sample space consists of all nonnegative real numbers. That is, S = [0, ∞)∗
Any subset E of the sample space S is known as an event. Some examples of events are the following. 1 . In Example (1), if E = {H }, then E is the event that a head appears on the flip of the coin. Similarly, if E = {T }, then E would be the event that a tail appears. 2 . In Example (2), if E = {1}, then E is the event that one appears on the roll of the die. If E = {2, 4, 6}, then E would be the event that an even number appears on the roll. ∗ The set (a, b) is defined to consist of all points x such that a < x < b. The set [a, b] is defined to consist of all points x such that a x b. The sets (a, b] and [a, b) are defined, respectively, to consist of all points x such that a < x b and all points x such that a x < b.
1.2.
Sample Space and Events
3
3 . In Example (3), if E = {(H, H ), (H, T )}, then E is the event that a head appears on the first coin. 4 . In Example (4), if E = {(1, 6), (2, 5), (3, 4), (4, 3), (5, 2), (6, 1)}, then E is the event that the sum of the dice equals seven. 5 . In Example (5), if E = (2, 6), then E is the event that the car lasts between two and six years. For any two events E and F of a sample space S we define the new event E ∪ F to consist of all outcomes that are either in E or in F or in both E and F . That is, the event E ∪ F will occur if either E or F occurs. For example, in (1) if E = {H } and F = {T }, then E ∪ F = {H, T } That is, E ∪ F would be the whole sample space S. In (2) if E = {1, 3, 5} and F = {1, 2, 3}, then E ∪ F = {1, 2, 3, 5} and thus E ∪ F would occur if the outcome of the die is 1 or 2 or 3 or 5. The event E ∪ F is often referred to as the union of the event E and the event F . For any two events E and F , we may also define the new event EF , sometimes written E ∩ F , and referred to as the intersection of E and F , as follows. EF consists of all outcomes which are both in E and in F . That is, the event EF will occur only if both E and F occur. For example, in (2) if E = {1, 3, 5} and F = {1, 2, 3}, then EF = {1, 3} and thus EF would occur if the outcome of the die is either 1 or 3. In Example (1) if E = {H } and F = {T }, then the event EF would not consist of any outcomes and hence could not occur. To give such an event a name, we shall refer to it as the null event and denote it by Ø. (That is, Ø refers to the event consisting of no outcomes.) If EF = Ø, then E and F are said to be mutually exclusive. We also define unions and intersections of more than two events in a similar
∞manner. If E1 , E2 , . . . are events, then the union of these events, denoted by n=1 En , is defined to be that event which consists of all outcomes that are in En for at least one value of n = 1, 2, . . . . Similarly, the intersection of the events En , denoted by ∞ n=1 En , is defined to be the event consisting of those outcomes that are in all of the events En , n = 1, 2, . . . . Finally, for any event E we define the new event E c , referred to as the complement of E, to consist of all outcomes in the sample space S that are not in E. That is, E c will occur if and only if E does not occur. In Example (4)
4
1
Introduction to Probability Theory
if E = {(1, 6), (2, 5), (3, 4), (4, 3), (5, 2), (6, 1)}, then E c will occur if the sum of the dice does not equal seven. Also note that since the experiment must result in some outcome, it follows that S c = Ø.
1.3.
Probabilities Defined on Events
Consider an experiment whose sample space is S. For each event E of the sample space S, we assume that a number P (E) is defined and satisfies the following three conditions: (i) 0 P (E) 1. (ii) P (S) = 1. (iii) For any sequence of events E1 , E2 , . . . that are mutually exclusive, that is, events for which En Em = Ø when n = m, then P
∞
En =
n=1
∞
P (En )
n=1
We refer to P (E) as the probability of the event E.
Example 1.1 In the coin tossing example, if we assume that a head is equally likely to appear as a tail, then we would have P ({H }) = P ({T }) =
1 2
On the other hand, if we had a biased coin and felt that a head was twice as likely to appear as a tail, then we would have P ({H }) = 23 ,
P ({T }) =
1 3
Example 1.2 In the die tossing example, if we supposed that all six numbers were equally likely to appear, then we would have P ({1}) = P ({2}) = P ({3}) = P ({4}) = P ({5}) = P ({6}) =
1 6
From (iii) it would follow that the probability of getting an even number would equal P ({2, 4, 6}) = P ({2}) + P ({4}) + P ({6}) =
1 2
1.3.
Probabilities Defined on Events
5
Remark We have chosen to give a rather formal definition of probabilities as being functions defined on the events of a sample space. However, it turns out that these probabilities have a nice intuitive property. Namely, if our experiment is repeated over and over again then (with probability 1) the proportion of time that event E occurs will just be P (E). Since the events E and E c are always mutually exclusive and since E ∪ E c = S we have by (ii) and (iii) that 1 = P (S) = P (E ∪ E c ) = P (E) + P (E c ) or P (E c ) = 1 − P (E)
(1.1)
In words, Equation (1.1) states that the probability that an event does not occur is one minus the probability that it does occur. We shall now derive a formula for P (E ∪ F ), the probability of all outcomes either in E or in F . To do so, consider P (E) + P (F ), which is the probability of all outcomes in E plus the probability of all points in F . Since any outcome that is in both E and F will be counted twice in P (E) + P (F ) and only once in P (E ∪ F ), we must have P (E) + P (F ) = P (E ∪ F ) + P (EF ) or equivalently P (E ∪ F ) = P (E) + P (F ) − P (EF )
(1.2)
Note that when E and F are mutually exclusive (that is, when EF = Ø), then Equation (1.2) states that P (E ∪ F ) = P (E) + P (F ) − P (Ø) = P (E) + P (F ) a result which also follows from condition (iii). [Why is P (Ø) = 0?]
Example 1.3 Suppose that we toss two coins, and suppose that we assume that each of the four outcomes in the sample space S = {(H, H ), (H, T ), (T , H ), (T , T )} is equally likely and hence has probability 14 . Let E = {(H, H ), (H, T )}
and F = {(H, H ), (T , H )}
That is, E is the event that the first coin falls heads, and F is the event that the second coin falls heads.
6
1
Introduction to Probability Theory
By Equation (1.2) we have that P (E ∪ F ), the probability that either the first or the second coin falls heads, is given by P (E ∪ F ) = P (E) + P (F ) − P (EF ) =
+
1 2
− P ({H, H })
=1−
1 4
=
1 2
3 4
This probability could, of course, have been computed directly since P (E ∪ F ) = P ({H, H ), (H, T ), (T , H )}) =
3 4
We may also calculate the probability that any one of the three events E or F or G occurs. This is done as follows: P (E ∪ F ∪ G) = P ((E ∪ F ) ∪ G) which by Equation (1.2) equals P (E ∪ F ) + P (G) − P ((E ∪ F )G) Now we leave it for you to show that the events (E ∪ F )G and EG ∪ F G are equivalent, and hence the preceding equals P (E ∪ F ∪ G) = P (E) + P (F ) − P (EF ) + P (G) − P (EG ∪ F G) = P (E) + P (F ) − P (EF ) + P (G) − P (EG) − P (F G) + P (EGF G) = P (E) + P (F ) + P (G) − P (EF ) − P (EG) − P (F G) + P (EF G) (1.3) In fact, it can be shown by induction that, for any n events E1 , E2 , E3 , . . . , En , P (Ei ) − P (Ei Ej ) + P (Ei Ej Ek ) P (E1 ∪ E2 ∪ · · · ∪ En ) = i
−
i<j
i<j m. Assume that in the count of the votes all possible orderings of the n + m votes are equally likely. Let Pn,m denote the probability that from the first vote on A is always in the lead. Find (b) P3,1 (c) Pn,1 (d) P3,2 (a) P2,1 (f ) Pn,2 (g) P4,3 (h) P5,3 (i) P5,4 ( j) Make a conjecture as to the value of Pn,m .
(e) P4,2
18
1
Introduction to Probability Theory
*25. Two cards are randomly selected from a deck of 52 playing cards. (a) What is the probability they constitute a pair (that is, that they are of the same denomination)? (b) What is the conditional probability they constitute a pair given that they are of different suits? 26. A deck of 52 playing cards, containing all 4 aces, is randomly divided into 4 piles of 13 cards each. Define events E1 , E2 , E3 , and E4 as follows: E1 = {the first pile has exactly 1 ace}, E2 = {the second pile has exactly 1 ace}, E3 = {the third pile has exactly 1 ace}, E4 = {the fourth pile has exactly 1 ace} Use Exercise 23 to find P (E1 E2 E3 E4 ), the probability that each pile has an ace. *27. Suppose in Exercise 26 we had defined the events Ei , i = 1, 2, 3, 4, by E1 = {one of the piles contains the ace of spades}, E2 = {the ace of spades and the ace of hearts are in different piles}, E3 = {the ace of spades, the ace of hearts, and the ace of diamonds are in different piles}, E4 = {all 4 aces are in different piles} Now use Exercise 23 to find P (E1 E2 E3 E4 ), the probability that each pile has an ace. Compare your answer with the one you obtained in Exercise 26. 28. If the occurrence of B makes A more likely, does the occurrence of A make B more likely? 29. Suppose that P (E) = 0.6. What can you say about P (E|F ) when (a) E and F are mutually exclusive? (b) E ⊂ F ? (c) F ⊂ E? *30. Bill and George go target shooting together. Both shoot at a target at the same time. Suppose Bill hits the target with probability 0.7, whereas George, independently, hits the target with probability 0.4. (a) Given that exactly one shot hit the target, what is the probability that it was George’s shot? (b) Given that the target is hit, what is the probability that George hit it? 31. What is the conditional probability that the first die is six given that the sum of the dice is seven?
Exercises
19
*32. Suppose all n men at a party throw their hats in the center of the room. Each man then randomly selects a hat. Show that the probability that none of the n men selects his own hat is 1 1 (−1)n 1 − + − +··· 2! 3! 4! n! Note that as n → ∞ this converges to e−1 . Is this surprising? 33. In a class there are four freshman boys, six freshman girls, and six sophomore boys. How many sophomore girls must be present if sex and class are to be independent when a student is selected at random? 34. Mr. Jones has devised a gambling system for winning at roulette. When he bets, he bets on red, and places a bet only when the ten previous spins of the roulette have landed on a black number. He reasons that his chance of winning is quite large since the probability of eleven consecutive spins resulting in black is quite small. What do you think of this system? 35. A fair coin is continually flipped. What is the probability that the first four flips are (a) H , H , H , H ? (b) T , H , H , H ? (c) What is the probability that the pattern T , H , H , H occurs before the pattern H , H , H , H ? 36. Consider two boxes, one containing one black and one white marble, the other, two black and one white marble. A box is selected at random and a marble is drawn at random from the selected box. What is the probability that the marble is black? 37. In Exercise 36, what is the probability that the first box was the one selected given that the marble is white? 38. Urn 1 contains two white balls and one black ball, while urn 2 contains one white ball and five black balls. One ball is drawn at random from urn 1 and placed in urn 2. A ball is then drawn from urn 2. It happens to be white. What is the probability that the transferred ball was white? 39. Stores A, B, and C have 50, 75, and 100 employees, and, respectively, 50, 60, and 70 percent of these are women. Resignations are equally likely among all employees, regardless of sex. One employee resigns and this is a woman. What is the probability that she works in store C? *40. (a) A gambler has in his pocket a fair coin and a two-headed coin. He selects one of the coins at random, and when he flips it, it shows heads. What is the probability that it is the fair coin? (b) Suppose that he flips the same coin a
20
1
Introduction to Probability Theory
second time and again it shows heads. Now what is the probability that it is the fair coin? (c) Suppose that he flips the same coin a third time and it shows tails. Now what is the probability that it is the fair coin? 41. In a certain species of rats, black dominates over brown. Suppose that a black rat with two black parents has a brown sibling. (a) What is the probability that this rat is a pure black rat (as opposed to being a hybrid with one black and one brown gene)? (b) Suppose that when the black rat is mated with a brown rat, all five of their offspring are black. Now, what is the probability that the rat is a pure black rat? 42. There are three coins in a box. One is a two-headed coin, another is a fair coin, and the third is a biased coin that comes up heads 75 percent of the time. When one of the three coins is selected at random and flipped, it shows heads. What is the probability that it was the two-headed coin? 43. Suppose we have ten coins which are such that if the ith one is flipped then heads will appear with probability i/10, i = 1, 2, . . . , 10. When one of the coins is randomly selected and flipped, it shows heads. What is the conditional probability that it was the fifth coin? 44. Urn 1 has five white and seven black balls. Urn 2 has three white and twelve black balls. We flip a fair coin. If the outcome is heads, then a ball from urn 1 is selected, while if the outcome is tails, then a ball from urn 2 is selected. Suppose that a white ball is selected. What is the probability that the coin landed tails? *45. An urn contains b black balls and r red balls. One of the balls is drawn at random, but when it is put back in the urn c additional balls of the same color are put in with it. Now suppose that we draw another ball. Show that the probability that the first ball drawn was black given that the second ball drawn was red is b/(b + r + c). 46. Three prisoners are informed by their jailer that one of them has been chosen at random to be executed, and the other two are to be freed. Prisoner A asks the jailer to tell him privately which of his fellow prisoners will be set free, claiming that there would be no harm in divulging this information, since he already knows that at least one will go free. The jailer refuses to answer this question, pointing out that if A knew which of his fellows were to be set free, then his own probability of being executed would rise from 13 to 12 , since he would then be one of two prisoners. What do you think of the jailer’s reasoning? 47. For a fixed event B, show that the collection P (A|B), defined for all events A, satisfies the three conditions for a probability. Conclude from this that P (A|B) = P (A|BC)P (C|B) + P (A|BC c )P (C c |B) Then directly verify the preceding equation.
References
21
*48. Sixty percent of the families in a certain community own their own car, thirty percent own their own home, and twenty percent own both their own car and their own home. If a family is randomly chosen, what is the probability that this family owns a car or a house but not both?
References Reference [2] provides a colorful introduction to some of the earliest developments in probability theory. References [3], [4], and [7] are all excellent introductory texts in modern probability theory. Reference [5] is the definitive work which established the axiomatic foundation of modern mathematical probability theory. Reference [6] is a nonmathematical introduction to probability theory and its applications, written by one of the greatest mathematicians of the eighteenth century.
1. L. Breiman, “Probability,” Addison-Wesley, Reading, Massachusetts, 1968. 2. F. N. David, “Games, Gods, and Gambling,” Hafner, New York, 1962. 3. W. Feller, “An Introduction to Probability Theory and Its Applications,” Vol. I, John Wiley, New York, 1957. 4. B. V. Gnedenko, “Theory of Probability,” Chelsea, New York, 1962. 5. A. N. Kolmogorov, “Foundations of the Theory of Probability,” Chelsea, New York, 1956. 6. Marquis de Laplace, “A Philosophical Essay on Probabilities,” 1825 (English Translation), Dover, New York, 1951. 7. S. Ross, “A First Course in Probability,” Sixth Edition, Prentice Hall, New Jersey, 2002.
This page intentionally left blank
Random Variables
2 2.1.
Random Variables
It frequently occurs that in performing an experiment we are mainly interested in some functions of the outcome as opposed to the outcome itself. For instance, in tossing dice we are often interested in the sum of the two dice and are not really concerned about the actual outcome. That is, we may be interested in knowing that the sum is seven and not be concerned over whether the actual outcome was (1, 6) or (2, 5) or (3, 4) or (4, 3) or (5, 2) or (6, 1). These quantities of interest, or more formally, these real-valued functions defined on the sample space, are known as random variables. Since the value of a random variable is determined by the outcome of the experiment, we may assign probabilities to the possible values of the random variable.
Example 2.1 Letting X denote the random variable that is defined as the sum of two fair dice; then P {X = 2} = P {(1, 1)} =
1 36 ,
P {X = 3} = P {(1, 2), (2, 1)} =
2 36 ,
P {X = 4} = P {(1, 3), (2, 2), (3, 1)} =
3 36 ,
P {X = 5} = P {(1, 4), (2, 3), (3, 2), (4, 1)} =
4 36 ,
P {X = 6} = P {(1, 5), (2, 4), (3, 3), (4, 2), (5, 1)} =
5 36 ,
P {X = 7} = P {(1, 6), (2, 5), (3, 4), (4, 3), (5, 2), (6, 1)} = P {X = 8} = P {(2, 6), (3, 5), (4, 4), (5, 3), (6, 2)} = P {X = 9} = P {(3, 6), (4, 5), (5, 4), (6, 3)} =
23
4 36 ,
5 36 ,
6 36 ,
24
2
Random Variables
P {X = 10} = P {(4, 6), (5, 5), (6, 4)} = P {X = 11} = P {(5, 6), (6, 5)} = P {X = 12} = P {(6, 6)} =
3 36 ,
2 36 ,
1 36
(2.1)
In other words, the random variable X can take on any integral value between two and twelve, and the probability that it takes on each value is given by Equation (2.1). Since X must take on one of the values two through twelve, we must have that 1=P
12
{X = n} =
i=2
which may be checked from Equation (2.1).
12
P {X = n}
n=2
Example 2.2 For a second example, suppose that our experiment consists of tossing two fair coins. Letting Y denote the number of heads appearing, then Y is a random variable taking on one of the values 0, 1, 2 with respective probabilities P {Y = 0} = P {(T , T )} = 14 , P {Y = 1} = P {(T , H ), (H, T )} = 24 , P {Y = 2} = P {(H, H )} =
1 4
Of course, P {Y = 0} + P {Y = 1} + P {Y = 2} = 1.
Example 2.3 Suppose that we toss a coin having a probability p of coming up heads, until the first head appears. Letting N denote the number of flips required, then assuming that the outcome of successive flips are independent, N is a random variable taking on one of the values 1, 2, 3, . . . , with respective probabilities P {N = 1} = P {H } = p, P {N = 2} = P {(T , H )} = (1 − p)p, P {N = 3} = P {(T , T , H )} = (1 − p)2 p, .. . P {N = n} = P {(T , T , . . . , T , H )} = (1 − p)n−1 p, n−1
n1
2.1.
Random Variables
25
As a check, note that P
∞
{N = n} =
n=1
∞
P {N = n}
n=1
=p
∞
(1 − p)n−1
n=1
=
p 1 − (1 − p)
=1
Example 2.4 Suppose that our experiment consists of seeing how long a battery can operate before wearing down. Suppose also that we are not primarily interested in the actual lifetime of the battery but are concerned only about whether or not the battery lasts at least two years. In this case, we may define the random variable I by 1, if the lifetime of battery is two or more years I= 0, otherwise If E denotes the event that the battery lasts two or more years, then the random variable I is known as the indicator random variable for event E. (Note that I equals 1 or 0 depending on whether or not E occurs.)
Example 2.5 Suppose that independent trials, each of which results m in any
of m possible outcomes with respective probabilities p1 , . . . , pm , i=1 pi = 1, are continually performed. Let X denote the number of trials needed until each outcome has occurred at least once. Rather than directly considering P {X = n} we will first determine P {X > n}, the probability that at least one of the outcomes has not yet occurred after n trials. Letting Ai denote the event that outcome i has not yet occurred after the first n trials, i = 1, . . . , m, then m
P {X > n} = P Ai i=1
=
m
P (Ai ) −
P (Ai Aj ) i<j
i=1
+ P (Ai Aj Ak ) − · · · + (−1)m+1 P (A1 · · · Am ) i<j < k
26
2
Random Variables
Now, P (Ai ) is the probability that each of the first n trials results in a non-i outcome, and so by independence P (Ai ) = (1 − pi )n Similarly, P (Ai Aj ) is the probability that the first n trials all result in a non-i and non-j outcome, and so P (Ai Aj ) = (1 − pi − pj )n As all of the other probabilities are similar, we see that P {X > n} =
m (1 − pi )n − (1 − pi − pj )n i<j
i=1
+ (1 − pi − pj − pk )n − · · · i<j < k
Since P {X = n} = P {X > n − 1} − P {X > n}, we see, upon using the algebraic identity (1 − a)n−1 − (1 − a)n = a(1 − a)n−1 , that P {X = n} =
m
pi (1 − pi )n−1 −
(pi + pj )(1 − pi − pj )n−1 i<j
i=1
+ (pi + pj + pk )(1 − pi − pj − pk )n−1 − · · ·
i<j < k
In all of the preceding examples, the random variables of interest took on either a finite or a countable number of possible values.∗ Such random variables are called discrete. However, there also exist random variables that take on a continuum of possible values. These are known as continuous random variables. One example is the random variable denoting the lifetime of a car, when the car’s lifetime is assumed to take on any value in some interval (a, b). The cumulative distribution function (cdf ) (or more simply the distribution function) F (·) of the random variable X is defined for any real number b, −∞ < b < ∞, by F (b) = P {X b} In words, F (b) denotes the probability that the random variable X takes on a value that is less than or equal to b. Some properties of the cdf F are (i) F (b) is a nondecreasing function of b, ∗ A set is countable if its elements can be put in a one-to-one correspondence with the sequence of positive integers.
2.2.
Discrete Random Variables
27
(ii) limb→∞ F (b) = F (∞) = 1, (iii) limb→−∞ F (b) = F (−∞) = 0. Property (i) follows since for a < b the event {X a} is contained in the event {X b}, and so it must have a smaller probability. Properties (ii) and (iii) follow since X must take on some finite value. All probability questions about X can be answered in terms of the cdf F (·). For example, P {a < X b} = F (b) − F (a)
for all a < b
This follows since we may calculate P {a < X b} by first computing the probability that X b [that is, F (b)] and then subtracting from this the probability that X a [that is, F (a)]. If we desire the probability that X is strictly smaller than b, we may calculate this probability by P {X < b} = lim P {X b − h} h→0+
= lim F (b − h) h→0+
where limh→0+ means that we are taking the limit as h decreases to 0. Note that P {X < b} does not necessarily equal F (b) since F (b) also includes the probability that X equals b.
2.2.
Discrete Random Variables
As was previously mentioned, a random variable that can take on at most a countable number of possible values is said to be discrete. For a discrete random variable X, we define the probability mass function p(a) of X by p(a) = P {X = a} The probability mass function p(a) is positive for at most a countable number of values of a. That is, if X must assume one of the values x1 , x2 , . . . , then p(xi ) > 0,
i = 1, 2, . . .
p(x) = 0,
all other values of x
Since X must take on one of the values xi , we have ∞ i=1
p(xi ) = 1
28
2
Random Variables
Figure 2.1. Graph of F (x).
The cumulative distribution function F can be expressed in terms of p(a) by F (a) = p(xi ) all xi a
For instance, suppose X has a probability mass function given by p(1) = 12 ,
p(2) = 13 ,
p(3) =
1 6
then, the cumulative distribution function F of X is given by ⎧ 0, a 0, α > 0 is said to be a gamma random variable with parameters α, λ. The quantity (α) is called the gamma function and is defined by ∞ (α) = e−x x α−1 dx 0
It is easy to show by induction that for integral α, say, α = n, (n) = (n − 1)! 2.3.4. Normal Random Variables We say that X is a normal random variable (or simply that X is normally distributed) with parameters μ and σ 2 if the density of X is given by 1 2 2 f (x) = √ e−(x−μ) /2σ , 2π σ
−∞ < x < ∞
This density function is a bell-shaped curve that is symmetric around μ (see Figure 2.2). An important fact about normal random variables is that if X is normally distributed with parameters μ and σ 2 then Y = αX + β is normally distributed with parameters αμ + β and α 2 σ 2 . To prove this, suppose first that α > 0 and note that
Figure 2.2. Normal density function.
38
2
Random Variables
FY (·)∗ the cumulative distribution function of the random variable Y is given by FY (a) = P {Y a} = P {αX + β a} a−β =P X α a−β = FX α (a−β)/α 1 2 2 = e−(x−μ) /2σ dx √ 2π σ −∞ a 1 −(v − (αμ + β))2 = exp √ 2α 2 σ 2 2π ασ −∞
dv
(2.9)
where the last equality a is obtained by the change in variables v = αx + β. However, since FY (a) = −∞ fY (v) dv, it follows from Equation (2.9) that the probability density function fY (·) is given by 1 −(v − (αμ + β))2 , −∞0
g(x)p(x)
2.4.
Expectation of a Random Variable
45
(b) If X is a continuous random variable with probability density function f (x), then for any real-valued function g, E[g(X)] =
∞
−∞
g(x)f (x) dx
Example 2.25 Applying the proposition to Example 2.23 yields E[X 2 ] = 02 (0.2) + (12 )(0.5) + (22 )(0.3) = 1.7 which, of course, checks with the result derived in Example 2.23.
Example 2.26 Applying the proposition to Example 2.24 yields E[X ] = 3
1
(since f (x) = 1, 0 < x < 1)
x 3 dx
0
=
1 4
A simple corollary of Proposition 2.1 is the following.
Corollary 2.2 If a and b are constants, then E[aX + b] = aE[X] + b
Proof In the discrete case,
E[aX + b] =
(ax + b)p(x)
x:p(x)>0
=a
xp(x) + b
x:p(x)>0
p(x)
x:p(x)>0
= aE[X] + b In the continuous case, E[aX + b] = =a
∞
(ax + b)f (x) dx
−∞ ∞
−∞
xf (x) dx + b
= aE[X] + b
∞
−∞
f (x) dx
46
2
Random Variables
The expected value of a random variable X, E[X], is also referred to as the mean or the first moment of X. The quantity E[X n ], n 1, is called the nth moment of X. By Proposition 2.1, we note that
E[X
n
⎧ ⎪ ⎪ ⎨
x n p(x),
x:p(x)>0 ]= ∞ ⎪ ⎪ ⎩ x n f (x) −∞
if X is discrete if X is continuous
dx,
Another quantity of interest is the variance of a random variable X, denoted by Var(X), which is defined by " # Var(X) = E (X − E[X])2 Thus, the variance of X measures the expected square of the deviation of X from its expected value.
Example 2.27 (Variance of the Normal Random Variable) Let X be normally distributed with parameters μ and σ 2 . Find Var(X). Solution:
Recalling (see Example 2.22) that E[X] = μ, we have that Var(X) = E[(X − μ)2 ] ∞ 1 2 2 =√ (x − μ)2 e−(x−μ) /2σ dx 2π σ −∞
Substituting y = (x − μ)/σ yields σ2 Var(X) = √ 2π
−∞
Integrating by parts (u = y, dv = ye−y σ2 Var(X) = √ 2π σ2 =√ 2π
∞
2 /2
y 2 e−y
−∞
∞
−∞
e−y
2 /2
dy
dy) gives
! 2 !∞ −ye−y /2 ! +
2 /2
∞
−∞
e−y
2 /2
dy
dy
= σ2 Another derivation of Var(X) will be given in Example 2.42.
2.5.
47
Jointly Distributed Random Variables
Suppose that X is continuous with density f, and let E[X] = μ. Then, Var(X) = E[(X − μ)2 ] = E[X 2 − 2μX + μ2 ] ∞ (x 2 − 2μx + μ2 )f (x) dx = −∞
=
∞ −∞
x 2 f (x) dx − 2μ
∞
−∞
xf (x) dx + μ2
∞
−∞
f (x) dx
= E[X 2 ] − 2μμ + μ2 = E[X 2 ] − μ2 A similar proof holds in the discrete case, and so we obtain the useful identity Var(X) = E[X 2 ] − (E[X])2
Example 2.28 Calculate Var(X) when X represents the outcome when a fair die is rolled. Solution: E[X 2 ] = 1
As previously noted in Example 2.15, E[X] = 72 . Also, $ % 1 6
+ 22
$ % 1 6
+ 32
$ % 1 6
+ 42
$ % 1 6
+ 52
$ % 1 6
+ 62
$ % 1 6
=
$ % 1 6
(91)
Hence, Var(X) =
2.5.
91 6
−
7 2 2
=
35 12
Jointly Distributed Random Variables
2.5.1. Joint Distribution Functions Thus far, we have concerned ourselves with the probability distribution of a single random variable. However, we are often interested in probability statements concerning two or more random variables. To deal with such probabilities, we define, for any two random variables X and Y , the joint cumulative probability distribution function of X and Y by F (a, b) = P {X a, Y b},
−∞ < a, b < ∞
48
2
Random Variables
The distribution of X can be obtained from the joint distribution of X and Y as follows: FX (a) = P {X a} = P {X a, Y < ∞} = F (a, ∞) Similarly, the cumulative distribution function of Y is given by FY (b) = P {Y b} = F (∞, b) In the case where X and Y are both discrete random variables, it is convenient to define the joint probability mass function of X and Y by p(x, y) = P {X = x, Y = y} The probability mass function of X may be obtained from p(x, y) by pX (x) = p(x, y) y:p(x,y)>0
Similarly,
pY (y) =
p(x, y)
x:p(x,y)>0
We say that X and Y are jointly continuous if there exists a function f (x, y), defined for all real x and y, having the property that for all sets A and B of real numbers f (x, y) dx dy P {X ∈ A, Y ∈ B} = B
A
The function f (x, y) is called the joint probability density function of X and Y . The probability density of X can be obtained from a knowledge of f (x, y) by the following reasoning: P {X ∈ A} = P {X ∈ A, Y ∈ (−∞, ∞)} ∞ = f (x, y) dx dy
−∞ A
=
fX (x) dx A
2.5.
where
fX (x) =
49
Jointly Distributed Random Variables
∞
−∞
f (x, y) dy
is thus the probability density function of X. Similarly, the probability density function of Y is given by ∞ f (x, y) dx fY (y) = −∞
A variation of Proposition 2.1 states that if X and Y are random variables and g is a function of two variables, then g(x, y)p(x, y) in the discrete case E[g(X, Y )] = y
=
x ∞ ∞
−∞ −∞
in the continuous case
g(x, y)f (x, y) dx dy
For example, if g(X, Y ) = X + Y , then, in the continuous case, ∞ ∞ (x + y)f (x, y) dx dy E[X + Y ] = −∞ −∞ ∞ ∞
=
−∞ −∞
xf (x, y) dx dy +
∞
∞
−∞ −∞
yf (x, y) dx dy
= E[X] + E[Y ] where the first integral is evaluated by using the variation of Proposition 2.1 with g(x, y) = x, and the second with g(x, y) = y. The same result holds in the discrete case and, combined with the corollary in Section 2.4.3, yields that for any constants a, b E[aX + bY ] = aE[X] + bE[Y ]
(2.10)
Joint probability distributions may also be defined for n random variables. The details are exactly the same as when n = 2 and are left as an exercise. The corresponding result to Equation (2.10) states that if X1 , X2 , . . . , Xn are n random variables, then for any n constants a1 , a2 , . . . , an , E[a1 X1 + a2 X2 + · · · + an Xn ] = a1 E[X1 ] + a2 E[X2 ] + · · · + an E[Xn ] (2.11)
Example 2.29 Calculate the expected sum obtained when three fair dice are rolled.
50
2
Random Variables
Solution: Let X denote the sum obtained. Then X = X1 + X2 + X3 where Xi represents the value of the ith die. Thus, E[X] = E[X1 ] + E[X2 ] + E[X3 ] = 3 72 = 21 2
Example 2.30 As another example of the usefulness of Equation (2.11), let us use it to obtain the expectation of a binomial random variable having parameters n and p. Recalling that such a random variable X represents the number of successes in n trials when each trial has probability p of being a success, we have that X = X1 + X2 + · · · + Xn where
1, Xi = 0,
if the ith trial is a success if the ith trial is a failure
Hence, Xi is a Bernoulli random variable having expectation E[Xi ] = 1(p) + 0(1 − p) = p. Thus, E[X] = E[X1 ] + E[X2 ] + · · · + E[Xn ] = np This derivation should be compared with the one presented in Example 2.17.
Example 2.31 At a party N men throw their hats into the center of a room. The hats are mixed up and each man randomly selects one. Find the expected number of men who select their own hats. Solution: Letting X denote the number of men that select their own hats, we can best compute E[X] by noting that X = X1 + X2 + · · · + XN where
1, Xi = 0,
if the ith man selects his own hat otherwise
Now, because the ith man is equally likely to select any of the N hats, it follows that P {Xi = 1} = P {ith man selects his own hat} = and so E[Xi ] = 1P {Xi = 1} + 0P {Xi = 0} =
1 N
1 N
2.5.
Jointly Distributed Random Variables
51
Hence, from Equation (2.11) we obtain that E[X] = E[X1 ] + · · · + E[XN ] =
1 N =1 N
Hence, no matter how many people are at the party, on the average exactly one of the men will select his own hat.
Example 2.32 Suppose there are 25 different types of coupons and suppose that each time one obtains a coupon, it is equally likely to be any one of the 25 types. Compute the expected number of different types that are contained in a set of 10 coupons. Solution: Let X denote the number of different types in the set of 10 coupons. We compute E[X] by using the representation X = X1 + · · · + X25 where Xi =
1, 0,
if at least one type i coupon is in the set of 10 otherwise
Now, E[Xi ] = P {Xi = 1} = P {at least one type i coupon is in the set of 10} = 1 − P {no type i coupons are in the set of 10} $ %10 = 1 − 24 25 when the last equality follows since each of the 10 coupons will (independently) not be a type i with probability 24 25 . Hence, & 10 ' E[X] = E[X1 ] + · · · + E[X25 ] = 25 1 − 24 25 2.5.2. Independent Random Variables The random variables X and Y are said to be independent if, for all a, b, P {X a, Y b} = P {X a}P {Y b}
(2.12)
In other words, X and Y are independent if, for all a and b, the events Ea = {X a} and Fb = {Y b} are independent.
52
2
Random Variables
In terms of the joint distribution function F of X and Y , we have that X and Y are independent if F (a, b) = FX (a)FY (b)
for all a, b
When X and Y are discrete, the condition of independence reduces to p(x, y) = pX (x)pY (y)
(2.13)
while if X and Y are jointly continuous, independence reduces to f (x, y) = fX (x)fY (y)
(2.14)
To prove this statement, consider first the discrete version, and suppose that the joint probability mass function p(x, y) satisfies Equation (2.13). Then P {X a, Y b} = p(x, y) yb xa
=
pX (x)pY (y)
yb xa
=
pY (y)
yb
pX (x)
xa
= P {Y b}P {X a} and so X and Y are independent. That Equation (2.14) implies independence in the continuous case is proven in the same manner and is left as an exercise. An important result concerning independence is the following.
Proposition 2.3 If X and Y are independent, then for any functions h and g E[g(X)h(Y )] = E[g(X)]E[h(Y )]
Proof Suppose that X and Y are jointly continuous. Then
E[g(X)h(Y )] = = =
∞
∞
−∞ −∞ ∞ ∞ −∞ −∞ ∞ −∞
g(x)h(y)f (x, y) dx dy g(x)h(y)fX (x)fY (y) dx dy
h(y)fY (y) dy
= E[h(Y )]E[g(X)] The proof in the discrete case is similar.
∞
−∞
g(x)fX (x) dx
2.5.
Jointly Distributed Random Variables
53
2.5.3. Covariance and Variance of Sums of Random Variables The covariance of any two random variables X and Y , denoted by Cov(X, Y ), is defined by Cov(X, Y ) = E[(X − E[X])(Y − E[Y ])] = E[XY − Y E[X] − XE[Y ] + E[X]E[Y ]] = E[XY ] − E[Y ]E[X] − E[X]E[Y ] + E[X]E[Y ] = E[XY ] − E[X]E[Y ] Note that if X and Y are independent, then by Proposition 2.3 it follows that Cov(X, Y ) = 0. Let us consider now the special case where X and Y are indicator variables for whether or not the events A and B occur. That is, for events A and B, define X=
1, 0,
if A occurs otherwise,
Y=
1, 0,
if B occurs otherwise
Then, Cov(X, Y ) = E[XY ] − E[X]E[Y ] and, because XY will equal 1 or 0 depending on whether or not both X and Y equal 1, we see that Cov(X, Y ) = P {X = 1, Y = 1} − P {X = 1}P {Y = 1} From this we see that Cov(X, Y ) > 0 ⇔ P {X = 1, Y = 1} > P {X = 1}P {Y = 1} ⇔
P {X = 1, Y = 1} > P {Y = 1} P {X = 1}
⇔ P {Y = 1|X = 1} > P {Y = 1} That is, the covariance of X and Y is positive if the outcome X = 1 makes it more likely that Y = 1 (which, as is easily seen by symmetry, also implies the reverse). In general it can be shown that a positive value of Cov(X, Y ) is an indication that Y tends to increase as X does, whereas a negative value indicates that Y tends to decrease as X increases. The following are important properties of covariance.
54
2
Random Variables
Properties of Covariance For any random variables X, Y, Z and constant c, 1. 2. 3. 4.
Cov(X, X) = Var(X), Cov(X, Y ) = Cov(Y, X), Cov(cX, Y ) = c Cov(X, Y ), Cov(X, Y + Z) = Cov(X, Y ) + Cov(X, Z).
Whereas the first three properties are immediate, the final one is easily proven as follows: Cov(X, Y + Z) = E[X(Y + Z)] − E[X]E[Y + Z] = E[XY ] − E[X]E[Y ] + E[XZ] − E[X]E[Z] = Cov(X, Y ) + Cov(X, Z) The fourth property listed easily generalizes to give the following result: ⎞ ⎛ n m n m ⎠ ⎝ Xi , Yj = Cov(Xi , Yj ) Cov i=1
j =1
(2.15)
i=1 j =1
A useful expression for the variance of the sum of random variables can be obtained from Equation (2.15) as follows: ⎞ ⎛ n n n Xi = Cov ⎝ Xi , Xj ⎠ Var i=1
i=1
=
n n
j =1
Cov(Xi , Xj )
i=1 j =1
=
n
Cov(Xi , Xi ) +
=
Cov(Xi , Xj )
i=1 j =i
i=1 n
n
Var(Xi ) + 2
i=1
n
Cov(Xi , Xj )
(2.16)
i=1 j < i
If Xi , i = 1, . . . , n are independent random variables, then Equation (2.16) reduces to n n Xi = Var(Xi ) Var i=1
i=1
2.5.
Jointly Distributed Random Variables
55
Definition 2.1 If X1 , . . . , Xn are independent and identically distributed, then the random variable X¯ =
n i=1 Xi /n
is called the sample mean.
The following proposition shows that the covariance between the sample mean and a deviation from that sample mean is zero. It will be needed in Section 2.6.1.
Proposition 2.4 Suppose that X1 , . . . , Xn are independent and identically distributed with expected value μ and variance σ 2 . Then, ¯ = μ. (a) E[X] ¯ = σ 2 /n. (b) Var(X) ¯ = 0, i = 1, . . . , n. ¯ Xi − X) (c) Cov(X,
Proof Parts (a) and (b) are easily established as follows: 1 E[Xi ] = μ, n i=1 n n 2 1 σ2 1 2 ¯ = Var(X) Var Xi = Var(Xi ) = n n n ¯ = E[X]
m
i=1
i=1
To establish part (c) we reason as follows: ¯ = Cov(X, ¯ Xi ) − Cov(X, ¯ X) ¯ ¯ Xi − X) Cov(X, 1 ¯ Xj , Xi − Var(X) = Cov Xi + n j =i
=
1 σ2 1 Cov(Xi , Xi ) + Cov Xj , Xi − n n n j =i
=
σ2 σ2 − =0 n n
where the final equality used the fact that Xi and thus have covariance 0.
j =i
Xj are independent and
Equation (2.16) is often useful when computing variances.
Example 2.33 (Variance of a Binomial Random Variable) Compute the variance of a binomial random variable X with parameters n and p.
56
2
Random Variables
Solution: Since such a random variable represents the number of successes in n independent trials when each trial has a common probability p of being a success, we may write X = X1 + · · · + Xn where the Xi are independent Bernoulli random variables such that 1, if the ith trial is a success Xi = 0, otherwise Hence, from Equation (2.16) we obtain Var(X) = Var(X1 ) + · · · + Var(Xn ) But Var(Xi ) = E[Xi2 ] − (E[Xi ])2 = E[Xi ] − (E[Xi ])2
since Xi2 = Xi
= p − p2 and thus Var(X) = np(1 − p)
Example 2.34 (Sampling from a Finite Population: The Hypergeometric) Consider a population of N individuals, some of whom are in favor of a certain proposition. In particular suppose that Np of them are in favor and N − Np are opposed, where p is assumed to be unknown. We are interested in estimating p, the fraction of the population that is for the proposition, by randomly choosing and then determining the positions of n members of the population. In such situations as described in the preceding, it is common to use the fraction of the sampled population that is in favor of the proposition as an estimator of p. Hence, if we let 1, if the ith person chosen is in favor Xi = 0, otherwise then the usual estimator of p is ni=1 Xi /n. Let us now compute its mean and variance. Now , n n E Xi = E[Xi ] i=1
1
2.5.
Jointly Distributed Random Variables
57
= np where the final equality follows since the ith person chosen is equally likely to be any of the N individuals in the population and so has probability Np/N of being in favor. n n Xi = Var(Xi ) + 2 Cov(Xi , Xj ) Var 1
1
i<j
Now, since Xi is a Bernoulli random variable with mean p, it follows that Var(Xi ) = p(1 − p) Also, for i = j , Cov(Xi , Xj ) = E[Xi Xj ] − E[Xi ]E[Xj ] = P {Xi = 1, Xj = 1} − p 2 = P {Xi = 1}P {Xj = 1 | Xi = 1} − p 2 =
Np (Np − 1) − p2 N N −1
where the last equality follows since if the ith person to be chosen is in favor, then the j th person chosen is equally likely to be any of the other N − 1 of which Np − 1 are in favor. Thus, we see that n / . n p(Np − 1) − p2 Var Xi = np(1 − p) + 2 N −1 2 1
= np(1 − p) −
n(n − 1)p(1 − p) N −1
and so the mean and variance of our estimator are given by , n Xi E = p, n 1 , n Xi p(1 − p) (n − 1)p(1 − p) − Var = n n n(N − 1) 1
Some remarks are in order: As the mean of the estimator is the unknown value p, we would like its variance to be as small as possible (why is this?), and we see by the preceding that, as a function of the population size N , the variance increases as N increases. The limiting value, as N → ∞, of the variance is p(1 − p)/n,
58
2
Random Variables
which is not surprising since for N large each of the Xi will be (approximately) independent random variables, and thus n1 Xi will have an (approximately) binomial distribution with parameters n and p. The random variable n1 Xi can be thought of as representing the number of white balls obtained when n balls are randomly selected from a population consisting of Np white and N − Np black balls. (Identify a person who favors the proposition with a white ball and one against with a black ball.) Such a random variable is called hypergeometric and has a probability mass function given by Np N − Np n k n−k P Xi = k = N 1 n It is often important to be able to calculate the distribution of X + Y from the distributions of X and Y when X and Y are independent. Suppose first that X and Y are continuous, X having probability density f and Y having probability density g. Then, letting FX+Y (a) be the cumulative distribution function of X + Y , we have FX+Y (a) = P {X + Y a} f (x)g(y) dx dy = = =
x+ya ∞ a−y
−∞ ∞
−∞
=
f (x)g(y) dx dy f (x) dx g(y) dy
−∞ −∞ ∞ a−y
−∞
FX (a − y)g(y) dy
(2.17)
The cumulative distribution function FX+Y is called the convolution of the distributions FX and FY (the cumulative distribution functions of X and Y , respectively). By differentiating Equation (2.17), we obtain that the probability density function fX+Y (a) of X + Y is given by ∞ d fX+Y (a) = FX (a − y)g(y) dy da −∞ ∞ d (FX (a − y))g(y) dy = −∞ da ∞ = f (a − y)g(y) dy (2.18) −∞
2.5.
Jointly Distributed Random Variables
59
Example 2.35 (Sum of Two Independent Uniform Random Variables) If X and Y are independent random variables both uniformly distributed on (0, 1), then calculate the probability density of X + Y . Solution:
From Equation (2.18), since 1, f (a) = g(a) = 0,
0 7 = P 0 $ % >0 $ % ⎪ ⎪ ⎪ 1 ⎪ ⎩ 10 1 ⎭ 10 12 12
≈ 1 − (2.2) = 0.0139
Example 2.52 The lifetime of a special type of battery is a random variable with mean 40 hours and standard deviation 20 hours. A battery is used until it fails, at which point it is replaced by a new one. Assuming a stockpile of 25 such batteries, the lifetimes of which are independent, approximate the probability that over 1100 hours of use can be obtained. Solution: If we let Xi denote the lifetime of the ith battery to be put in use, then we desire p = P {X1 + · · · + X25 > 1100}, which is approximated as follows: X1 + · · · + X25 − 1000 1100 − 1000 p=P > √ √ 20 25 20 25 ≈ P {N (0, 1) > 1} = 1 − (1) ≈ 0.1587 We now present a heuristic proof of the Central Limit theorem. Suppose first that the Xi have mean 0 and variance 1, and let E[etX ] denote their common moment generating function. Then, the moment generating function of X1 +√···n+ Xn is / . √ √ √ # " X1 + · · · + Xn = E etX1 / n etX2 / n · · · etXn / n E exp t √ n √ # " n by independence = E etX/ n Now, for n large, we obtain from the Taylor series expansion of ey that etX/
√ n
tX t 2 X2 ≈1+ √ + 2n n
2.8.
Stochastic Processes
83
Taking expectations shows that when n is large √ # " tE[X] t 2 E[X 2 ] E etX/ n ≈ 1 + √ + 2n n
=1+
t2 2n
because E[X] = 0, E[X 2 ] = 1
Therefore, we obtain that when n is large / n . t2 X1 + · · · + Xn ≈ 1+ E exp t √ 2n n When n goes to ∞ the approximation can be shown to become exact and we have that / . X1 + · · · + Xn 2 = et /2 lim E exp t √ n→∞ n Thus, the moment generating function of X1 +√···n+Xn converges to the moment generating function of a (standard) normal random variable with mean 0 and variance 1. Using this, it can be proven that the distribution function of the random variable X1 +√···n+Xn converges to the standard normal distribution function . When the Xi have mean μ and variance σ 2 , the random variables mean 0 and variance 1. Thus, the preceding shows that X1 − μ + X2 − μ + · · · + Xn − μ a → (a) P √ σ n
Xi −μ σ
have
which proves the central limit theorem.
2.8.
Stochastic Processes
A stochastic process {X(t), t ∈ T } is a collection of random variables. That is, for each t ∈ T , X(t) is a random variable. The index t is often interpreted as time and, as a result, we refer to X(t) as the state of the process at time t. For example, X(t) might equal the total number of customers that have entered a supermarket by time t; or the number of customers in the supermarket at time t; or the total amount of sales that have been recorded in the market by time t; etc. The set T is called the index set of the process. When T is a countable set the stochastic process is said to be a discrete-time process. If T is an interval of the real line, the stochastic process is said to be a continuous-time process. For instance, {Xn , n = 0, 1, . . .} is a discrete-time stochastic process indexed by the
84
2
Random Variables
nonnegative integers; while {X(t), t 0} is a continuous-time stochastic process indexed by the nonnegative real numbers. The state space of a stochastic process is defined as the set of all possible values that the random variables X(t) can assume. Thus, a stochastic process is a family of random variables that describes the evolution through time of some (physical) process. We shall see much of stochastic processes in the following chapters of this text.
Example 2.53 Consider a particle that moves along a set of m + 1 nodes, labeled 0, 1, . . . , m, that are arranged around a circle (see Figure 2.3). At each step the particle is equally likely to move one position in either the clockwise or counterclockwise direction. That is, if Xn is the position of the particle after its nth step then P {Xn+1 = i + 1|Xn = i} = P {Xn+1 = i − 1|Xn = i} =
1 2
where i + 1 ≡ 0 when i = m, and i − 1 ≡ m when i = 0. Suppose now that the particle starts at 0 and continues to move around according to the preceding rules until all the nodes 1, 2, . . . , m have been visited. What is the probability that node i, i = 1, . . . , m, is the last one visited? Solution: Surprisingly enough, the probability that node i is the last node visited can be determined without any computations. To do so, consider the first time that the particle is at one of the two neighbors of node i, that is, the first time that the particle is at one of the nodes i − 1 or i + 1 (with m + 1 ≡ 0). Suppose it is at node i − 1 (the argument in the alternative situation is identical). Since neither node i nor i + 1 has yet been visited, it follows that i will be the last node visited if and only if i + 1 is visited before i. This is so because in
Figure 2.3. Particle moving around a circle.
Exercises
85
order to visit i + 1 before i the particle will have to visit all the nodes on the counterclockwise path from i − 1 to i + 1 before it visits i. But the probability that a particle at node i − 1 will visit i + 1 before i is just the probability that a particle will progress m − 1 steps in a specified direction before progressing one step in the other direction. That is, it is equal to the probability that a gambler who starts with one unit, and wins one when a fair coin turns up heads and loses one when it turns up tails, will have his fortune go up by m − 1 before he goes broke. Hence, because the preceding implies that the probability that node i is the last node visited is the same for all i, and because these probabilities must sum to 1, we obtain P {i is the last node visited} = 1/m,
i = 1, . . . , m
Remark The argument used in Example 2.53 also shows that a gambler who is equally likely to either win or lose one unit on each gamble will be down n before being up 1 with probability 1/(n + 1); or equivalently P {gambler is up 1 before being down n} =
n n+1
Suppose now we want the probability that the gambler is up 2 before being down n. Upon conditioning on whether he reaches up 1 before down n, we obtain that P {gambler is up 2 before being down n} = P {up 2 before down n|up 1 before down n} = P {up 1 before down n + 1} =
n n+1
n n+1
n n+1 n = n+2n+1 n+2
Repeating this argument yields that P {gambler is up k before being down n} =
n n+k
Exercises 1. An urn contains five red, three orange, and two blue balls. Two balls are randomly selected. What is the sample space of this experiment? Let X represent the number of orange balls selected. What are the possible values of X? Calculate P {X = 0}.
86
2
Random Variables
2. Let X represent the difference between the number of heads and the number of tails obtained when a coin is tossed n times. What are the possible values of X? 3. In Exercise 2, if the coin is assumed fair, then, for n = 2, what are the probabilities associated with the values that X can take on? *4. Suppose a die is rolled twice. What are the possible values that the following random variables can take on? (i) (ii) (iii) (iv)
The maximum value to appear in the two rolls. The minimum value to appear in the two rolls. The sum of the two rolls. The value of the first roll minus the value of the second roll.
5. If the die in Exercise 4 is assumed fair, calculate the probabilities associated with the random variables in (i)–(iv). 6. Suppose five fair coins are tossed. Let E be the event that all coins land heads. Define the random variable IE 1, if E occurs IE = 0, if E c occurs For what outcomes in the original sample space does IE equal 1? What is P {IE = 1}? 7. Suppose a coin having probability 0.7 of coming up heads is tossed three times. Let X denote the number of heads that appear in the three tosses. Determine the probability mass function of X. 8. Suppose the distribution function of X is given by ⎧ b ε → 0 P !! n 67. Suppose that X is a random variable with mean 10 and variance 15. What can we say about P {5 < X < 15}? 68. Let X1 , X2 , . . . , X10 be independent Poisson random variables with mean 1. (i) Use the Markov inequality to get a bound on P {X1 + · · · + X10 15}. (ii) Use the central limit theorem to approximate P {X1 + · · · + X10 15}. 69. If X is normally distributed with mean 1 and variance 4, use the tables to find P {2 < X < 3}. *70. Show that lim e−n
n→∞
n nk k=0
k!
=
1 2
Hint: Let Xn be Poisson with mean n. Use the central limit theorem to show that P {Xn n} → 12 . 71. Let X denote the number of white balls selected when k balls are chosen at random from an urn containing n white and m black balls.
Exercises
95
(i) Compute P {X = i}. (ii) Let, for i = 1, 2, . . . , k; j = 1, 2, . . . , n, 1, Xi = 0, 1, Yj = 0,
if the ith ball selected is white otherwise if white ball j is selected otherwise
Compute E[X] in two ways by expressing X first as a function of the Xi s and then of the Yj s. *72. Show that Var(X) = 1 when X is the number of men who select their own hats in Example 2.31. 73. For the multinomial distribution (Exercise 17), let Ni denote the number of times outcome i occurs. Find (i) (ii) (iii) (iv)
E[Ni ]; Var(Ni ); Cov(Ni , Nj ); Compute the expected number of outcomes that do not occur.
74. Let X1 , X2 , . . . be a sequence of independent identically distributed continuous random variables. We say that a record occurs at time n if Xn > max(X1 , . . . , Xn−1 ). That is, Xn is a record if it is larger than each of X1 , . . . , Xn−1 . Show (i) (ii) (iii) (iv)
P {a record occurs at time n} = 1/n; n E[number of records by time n] = i=1 1/i; Var(number of records by time n) = ni=1 (i − 1)/i 2 ; Let N = min{n: n > 1 and a record occurs at time n}. Show E[N ] = ∞.
Hint: For (ii) and (iii) represent the number of records as the sum of indicator (that is, Bernoulli) random variables. 75. Let a1 < a2 < · · · < an denote a set of n numbers, and consider any permutation of these numbers. We say that there is an inversion of ai and aj in the permutation if i < j and aj precedes ai . For instance the permutation 4, 2, 1, 5, 3 has 5 inversions—(4, 2), (4, 1), (4, 3), (2, 1), (5, 3). Consider now a random permutation of a1 , a2 , . . . , an —in the sense that each of the n! permutations is equally likely to be chosen—and let N denote the number of inversions in this permutation. Also, let Ni = number of k : k < i, ai precedes ak in the permutation and note that N = ni=1 Ni .
96
2
Random Variables
(i) Show that N1 , . . . , Nn are independent random variables. (ii) What is the distribution of Ni ? (iii) Compute E[N ] and Var(N ). 76. Let X and Y be independent random variables with means μx and μy and variances σx2 and σy2 . Show that Var(XY ) = σx2 σy2 + μ2y σx2 + μ2x σy2 77. Let X and Y be independent normal random variables, each having parameters μ and σ 2 . Show that X + Y is independent of X − Y. Hint: Find their joint moment generating function. 78. Let φ(t1 , . . . , tn ) denote the joint moment generating function of X1 , . . . , Xn . (a) Explain how the moment generating function of Xi , φXi (ti ), can be obtained from φ(t1 , . . . , tn ). (b) Show that X1 , . . . , Xn are independent if and only if φ(t1 , . . . , tn ) = φx1 (t1 ) · · · φXn (tn )
References 1. W. Feller, “An Introduction to Probability Theory and Its Applications,” Vol. I, John Wiley, New York, 1957. 2. M. Fisz, “Probability Theory and Mathematical Statistics,” John Wiley, New York, 1963. 3. E. Parzen, “Modern Probability Theory and Its Applications,” John Wiley, New York, 1960. 4. S. Ross, “A First Course in Probability,” Seventh Edition, Prentice Hall, New Jersey, 2006.
Conditional Probability and Conditional Expectation
3 3.1.
Introduction
One of the most useful concepts in probability theory is that of conditional probability and conditional expectation. The reason is twofold. First, in practice, we are often interested in calculating probabilities and expectations when some partial information is available; hence, the desired probabilities and expectations are conditional ones. Secondly, in calculating a desired probability or expectation it is often extremely useful to first “condition” on some appropriate random variable.
3.2.
The Discrete Case
Recall that for any two events E and F , the conditional probability of E given F is defined, as long as P (F ) > 0, by P (E|F ) =
P (EF ) P (F )
Hence, if X and Y are discrete random variables, then it is natural to define the conditional probability mass function of X given that Y = y, by pX|Y (x|y) = P {X = x|Y = y} =
P {X = x, Y = y} P {Y = y}
=
p(x, y) pY (y)
97
98
3
Conditional Probability and Conditional Expectation
for all values of y such that P {Y = y} > 0. Similarly, the conditional probability distribution function of X given that Y = y is defined, for all y such that P {Y = y} > 0, by FX|Y (x|y) = P {X x|Y = y} pX|Y (a|y) = ax
Finally, the conditional expectation of X given that Y = y is defined by E[X|Y = y] = xP {X = x|Y = y} x
=
xpX|Y (x|y)
x
In other words, the definitions are exactly as before with the exception that everything is now conditional on the event that Y = y. If X is independent of Y , then the conditional mass function, distribution, and expectation are the same as the unconditional ones. This follows, since if X is independent of Y , then pX|Y (x|y) = P {X = x|Y = y} = P {X = x}
Example 3.1 Suppose that p(x, y), the joint probability mass function of X and Y , is given by p(1, 1) = 0.5,
p(1, 2) = 0.1,
p(2, 1) = 0.1,
p(2, 2) = 0.3
Calculate the conditional probability mass function of X given that Y = 1. Solution:
We first note that p(x, 1) = p(1, 1) + p(2, 1) = 0.6 pY (1) = x
Hence, pX|Y (1|1) = P {X = 1|Y = 1} =
P {X = 1, Y = 1} P {Y = 1}
=
p(1, 1) pY (1)
=
5 6
3.2.
The Discrete Case
99
Similarly, pX|Y (2|1) =
p(2, 1) 1 = pY (1) 6
Example 3.2 If X1 and X2 are independent binomial random variables with respective parameters (n1 , p) and (n2 , p), calculate the conditional probability mass function of X1 given that X1 + X2 = m. Solution:
With q = 1 − p,
P {X1 = k|X1 + X2 = m} = =
P {X1 = k, X1 + X2 = m} P {X1 + X2 = m} P {X1 = k, X2 = m − k} P {X1 + X2 = m}
P {X1 = k}P {X2 = m − k} P {X1 + X2 = m} n1 k n1 −k n2 p q p m−k q n2 −m+k k m−k = n1 + n2 m n1 +n2 −m p q m =
where we have used that X1 + X2 is a binomial random variable with parameters (n1 + n2 , p) (see Example 2.44). Thus, the conditional probability mass function of X1 , given that X1 + X2 = m, is n2 n1 k m−k P {X1 = k|X1 + X2 = m} = (3.1) n1 + n2 m The distribution given by Equation (3.1), first seen in Example 2.34, is known as the hypergeometric distribution. It is the distribution of the number of blue balls that are chosen when a sample of m balls is randomly chosen from an urn that contains n1 blue and n2 red balls. ( To intuitively see why the conditional distribution is hypergeometric, consider n1 + n2 independent trials that each result in a success with probability p; let X1 represent the number of successes in the first n1 trials and let X2 represent the number of successes in the final n2 trials. all trials have the same probability of being a success, Because 2 each of the n1 +n subsets of m trials is equally likely to be the success trials; m thus, the number of the m success trials that are among the first n1 trials is a hypergeometric random variable.)
100
3
Conditional Probability and Conditional Expectation
Example 3.3 If X and Y are independent Poisson random variables with respective means λ1 and λ2 , calculate the conditional expected value of X given that X + Y = n. Solution: Let us first calculate the conditional probability mass function of X given that X + Y = n. We obtain P {X = k|X + Y = n} =
P {X = k, X + Y = n} P {X + Y = n}
=
P {X = k, Y = n − k} P {X + Y = n}
=
P {X = k}P {Y = n − k} P {X + Y = n}
where the last equality follows from the assumed independence of X and Y . Recalling (see Example 2.36) that X + Y has a Poisson distribution with mean λ1 + λ2 , the preceding equation equals P {X = k|X + Y = n} =
. −(λ1 +λ2 ) / e−λ1 λk1 e−λ2 λn−k e (λ1 + λ2 )n −1 2 k! (n − k)! n!
λk1 λn−k n! 2 (n − k)!k! (λ1 + λ2 )n k n−k λ2 n λ1 = λ1 + λ2 λ1 + λ2 k =
In other words, the conditional distribution of X given that X + Y = n, is the binomial distribution with parameters n and λ1 /(λ1 + λ2 ). Hence, E{X|X + Y = n} = n
λ1 λ1 + λ2
Example 3.4 Consider an experiment which results in one of three possible outcomes with outcome i occurring with probability pi , i = 1, 2, 3, 3i=1 pi = 1. Suppose that n independent replications of this experiment are performed and let Xi , i = 1, 2, 3, denote the number of times outcome i appears. Determine the conditional expectation of X1 given that X2 = m.
3.2.
Solution:
The Discrete Case
101
For k n − m, P {X1 = k|X2 = m} =
P {X1 = k, X2 = m} P {X2 = m}
Now if X1 = k and X2 = m, then it follows that X3 = n − k − m. However, P {X1 = k, X2 = m, X3 = n − k − m} =
n! (n−k−m) pk pm p k!m!(n − k − m)! 1 2 3
(3.2)
This follows since any particular sequence of the n experiments having outcome 1 appearing k times, outcome 2 m times, and outcome 3 (n − k − m) times has probability p1k p2m p3(n−k−m) of occurring. Since there are n!/[k!m!(n − k − m)!] such sequences, Equation (3.2) follows. Therefore, we have n! (n−k−m) pk pm p k!m!(n − k − m)! 1 2 3 P {X1 = k|X2 = m} = n! p m (1 − p2 )n−m m!(n − m)! 2 where we have used the fact that X2 has a binomial distribution with parameters n and p2 . Hence, P {X1 = k|X2 = m} =
(n − m)! k!(n − m − k)!
p1 1 − p2
k
p3 1 − p2
n−m−k
or equivalently, writing p3 = 1 − p1 − p2 , P {X1 = k|X2 = m} =
k n−m−k p1 p1 n−m 1− 1 − p2 1 − p2 k
In other words, the conditional distribution of X1 , given that X2 = m, is binomial with parameters n − m and p1 /(1 − p2 ). Consequently, E[X1 |X2 = m] = (n − m)
p1 1 − p2
Remarks (i) The desired conditional probability in Example 3.4 could also
have been computed in the following manner. Consider the n − m experiments
102
3
Conditional Probability and Conditional Expectation
that did not result in outcome 2. For each of these experiments, the probability that outcome 1 was obtained is given by P {outcome 1, not outcome 2} P {not outcome 2} p1 = 1 − p2
P {outcome 1|not outcome 2} =
It therefore follows that, given X2 = m, the number of times outcome 1 occurs is binomially distributed with parameters n − m and p1 /(1 − p2 ). (ii) Conditional expectations possess all of the properties of ordinary expectations. For instance, such identities as , n n Xi |Y = y = E[Xi |Y = y] E i=1
i=1
remain valid.
Example 3.5 There are n components. On a rainy day, component i will function with probability pi ; on a nonrainy day, component i will function with probability qi , for i = 1, . . . , n. It will rain tomorrow with probability α. Calculate the conditional expected number of components that function tomorrow, given that it rains. Solution:
Let Xi =
1, 0,
if component i functions tomorrow otherwise
Then, with Y defined to equal 1 if it rains tomorrow, and 0 otherwise, the desired conditional expectation is obtained as follows. , n n Xi |Y = 1 = E[Xi |Y = 1] E t=1
i=1
=
n
pi
i=1
3.3.
The Continuous Case
If X and Y have a joint probability density function f (x, y), then the conditional probability density function of X, given that Y = y, is defined for all values of y
3.3.
The Continuous Case
103
such that fY (y) > 0, by fX|Y (x|y) =
f (x, y) fY (y)
To motivate this definition, multiply the left side by dx and the right side by (dx dy)/dy to get fX|Y (x|y) dx = ≈
f (x, y) dx dy fY (y) dy P {x X x + dx, y Y y + dy} P {y Y y + dy}
= P {x X x + dx|y Y y + dy} In other words, for small values dx and dy, fX|Y (x|y) dx is approximately the conditional probability that X is between x and x + dx given that Y is between y and y + dy. The conditional expectation of X, given that Y = y, is defined for all values of y such that fY (y) > 0, by ∞ E[X|Y = y] = xfX|Y (x|y) dx −∞
Example 3.6 Suppose the joint density of X and Y is given by f (x, y) =
6xy(2 − x − y), 0,
0 < x < 1, 0 < y < 1 otherwise
Compute the conditional expectation of X given that Y = y, where 0 < y < 1. Solution:
We first compute the conditional density fX|Y (x|y) =
f (x, y) fY (y)
= 1 0
6xy(2 − x − y) 6xy(2 − x − y) dx
=
6xy(2 − x − y) y(4 − 3y)
=
6x(2 − x − y) 4 − 3y
104
3
Conditional Probability and Conditional Expectation
Hence,
1 6x 2 (2 − x
E[X|Y = y] =
− y) dx 4 − 3y
0
=
(2 − y)2 − 4 − 3y
=
5 − 4y 8 − 6y
6 4
Example 3.7 Suppose the joint density of X and Y is given by f (x, y) =
4y(x − y)e−(x+y) , 0,
0 < x < ∞, 0 y x otherwise
Compute E[X|Y = y]. Solution:
The conditional density of X, given that Y = y, is given by
fX|Y (x|y) =
f (x, y) fY (y)
= ∞ y
= ∞ y
4y(x − y)e−(x+y) , 4y(x − y)e−(x+y) dx
x>y
(x − y)e−x (x − y)e−x dx
(x − y)e−x = ∞ −(y+w) , dw 0 we = (x − y)e−(x−y) ,
x>y
(by letting w = x − y)
x>y
∞ where the final equality used that 0 we−w dw is the expected value of an exponential random variable with mean 1. Therefore, with W being exponential with mean 1, E[X|Y = y] = y
∞
x(x − y)e−(x−y) dx
3.4.
Computing Expectations by Conditioning
∞
=
105
(w + y)we−w dw
0
= E[W 2 ] + yE[W ] =2+y
Example 3.8 The joint density of X and Y is given by 1
2 ye
f (x, y) =
−xy ,
0 < x < ∞, 0 < y < 2
0,
otherwise
What is E[eX/2 |Y = 1]? Solution:
The conditional density of X, given that Y = 1, is given by fX|Y (x|1) =
f (x, 1) fY (1) 1 −x e
= ∞ 21 0
−x dx 2e
= e−x
Hence, by Proposition 2.1, E[e
X/2
|Y = 1] =
∞
ex/2 fX|Y (x|1) dx
0
=
∞
ex/2 e−x dx
0
=2
3.4.
Computing Expectations by Conditioning
Let us denote by E[X|Y ] that function of the random variable Y whose value at Y = y is E[X|Y = y]. Note that E[X|Y ] is itself a random variable. An extremely important property of conditional expectation is that for all random variables X
106
3
Conditional Probability and Conditional Expectation
and Y
" # E[X] = E E[X|Y ]
If Y is a discrete random variable, then Equation (3.3) states that E[X|Y = y]P {Y = y} E[X] =
(3.3)
(3.3a)
y
while if Y is continuous with density fY (y), then Equation (3.3) says that ∞ E[X|Y = y]fY (y) dy (3.3b) E[X] = −∞
We now give a proof of Equation (3.3) in the case where X and Y are both discrete random variables.
Proof of Equation (3.3) When X and Y Are Discrete We must show that E[X] =
E[X|Y = y]P {Y = y}
(3.4)
y
Now, the right side of the preceding can be written E[X|Y = y]P {Y = y} = xP {X = x|Y = y}P {Y = y} y
y
= = =
x
y
x
y
x
P {X = x, Y = y} P {Y = y} P {Y = y}
xP {X = x, Y = y}
x P {X = x, Y = y} x
=
x
y
xP {X = x}
x
= E[X] and the result is obtained.
One way to understand Equation (3.4) is to interpret it as follows. It states that to calculate E[X] we may take a weighted average of the conditional expected value of X given that Y = y, each of the terms E[X|Y = y] being weighted by the probability of the event on which it is conditioned. The following examples will indicate the usefulness of Equation (3.3).
3.4.
Computing Expectations by Conditioning
107
Example 3.9 Sam will read either one chapter of his probability book or one chapter of his history book. If the number of misprints in a chapter of his probability book is Poisson distributed with mean 2 and if the number of misprints in his history chapter is Poisson distributed with mean 5, then assuming Sam is equally likely to choose either book, what is the expected number of misprints that Sam will come across? Solution:
Letting X denote the number of misprints and letting 1, if Sam chooses his history book Y= 2, if Sam chooses his probability book
then E[X] = E[X|Y = 1]P {Y = 1} + E[X|Y = 2]P {Y = 2} = 5 12 + 2 12 =
7 2
Example 3.10 (The Expectation of the Sum of a Random Number of Random Variables) Suppose that the expected number of accidents per week at an industrial plant is four. Suppose also that the numbers of workers injured in each accident are independent random variables with a common mean of 2. Assume also that the number of workers injured in each accident is independent of the number of accidents that occur. What is the expected number of injuries during a week? Solution: Letting N denote the number of accidents and Xi the number injured in the ithaccident, i = 1, 2, . . . , then the total number of injuries can be expressed as N i=1 Xi . Now , E
N
-
, ,
Xi = E E
N
1
-Xi |N
1
But E
,N
-
,
Xi |N = n = E
1
, =E
n
Xi |N = n
1 n 1
= nE[X]
Xi
by the independence of Xi and N
108
3
Conditional Probability and Conditional Expectation
which yields that , E
N
Xi |N = N E[X]
i=1
and thus , E
N
-
" # Xi = E N E[X] = E[N ]E[X]
i=1
Therefore, in our example, the expected number of injuries during a week equals 4 × 2 = 8. The random variable N i=1 Xi , equal to the sum of a random number N of independent and identically distributed random variables that are also independent of N , is called a compound random variable. As just shown in Example 3.10, the expected value of a compound random variable is E[X]E[N ]. Its variance will be derived in Example 3.17.
Example 3.11 (The Mean of a Geometric Distribution) A coin, having probability p of coming up heads, is to be successively flipped until the first head appears. What is the expected number of flips required? Solution:
Let N be the number of flips required, and let 1, Y= 0,
if the first flip results in a head if the first flip results in a tail
Now E[N] = E[N |Y = 1]P {Y = 1} + E[N |Y = 0]P {Y = 0} = pE[N |Y = 1] + (1 − p)E[N |Y = 0]
(3.5)
However, E[N |Y = 1] = 1,
E[N |Y = 0] = 1 + E[N ]
(3.6)
To see why Equation (3.6) is true, consider E[N |Y = 1]. Since Y = 1, we know that the first flip resulted in heads and so, clearly, the expected number of flips required is 1. On the other hand if Y = 0, then the first flip resulted in tails. However, since the successive flips are assumed independent, it follows that, after the first tail, the expected additional number of flips until the first head is
3.4.
Computing Expectations by Conditioning
109
just E[N]. Hence E[N|Y = 0] = 1 + E[N ]. Substituting Equation (3.6) into Equation (3.5) yields E[N ] = p + (1 − p)(1 + E[N ]) or E[N ] = 1/p
Because the random variable N is a geometric random variable with probability mass function p(n) = p(1 − p)n−1 , its expectation could easily have been computed from E[N ] = ∞ 1 np(n) without recourse to conditional expectation. However, if you attempt to obtain the solution to our next example without using conditional expectation, you will quickly learn what a useful technique “conditioning” can be.
Example 3.12 A miner is trapped in a mine containing three doors. The first door leads to a tunnel that takes him to safety after two hours of travel. The second door leads to a tunnel that returns him to the mine after three hours of travel. The third door leads to a tunnel that returns him to his mine after five hours. Assuming that the miner is at all times equally likely to choose any one of the doors, what is the expected length of time until the miner reaches safety? Solution: Let X denote the time until the miner reaches safety, and let Y denote the door he initially chooses. Now E[X] = E[X|Y = 1]P {Y = 1} + E[X|Y = 2]P {Y = 2} + E[X|Y = 3]P {Y = 3} = 13 E[X|Y = 1] + E[X|Y = 2] + E[X|Y = 3] However, E[X|Y = 1] = 2, E[X|Y = 2] = 3 + E[X], E[X|Y = 3] = 5 + E[X],
(3.7)
To understand why this is correct consider, for instance, E[X|Y = 2], and reason as follows. If the miner chooses the second door, then he spends three hours in the tunnel and then returns to the mine. But once he returns to the mine the problem is as before, and hence his expected additional time until safety is just E[X]. Hence E[X|Y = 2] = 3 + E[X]. The argument behind the other equalities in Equation (3.7) is similar. Hence E[X] = 13 2 + 3 + E[X] + 5 + E[X] or E[X] = 10
110
3
Conditional Probability and Conditional Expectation
Example 3.13 (The Matching Rounds Problem) Suppose in Example 2.31 that those choosing their own hats depart, while the others (those without a match) put their selected hats in the center of the room, mix them up, and then reselect. Also, suppose that this process continues until each individual has his own hat. (a) Find E[Rn ] where Rn is the number of rounds that are necessary when n individuals are initially present. (b) Find E[Sn ] where Sn is the total number of selections made by the n individuals, n 2. (c) Find the expected number of false selections made by one of the n people, n 2. Solution: (a) It follows from the results of Example 2.31 that no matter how many people remain there will, on average, be one match per round. Hence, one might suggest that E[Rn ] = n. This turns out to be true, and an induction proof will now be given. Because it is obvious that E[R1 ] = 1, assume that E[Rk ] = k for k = 1, . . . , n − 1. To compute E[Rn ], start by conditioning on Xn , the number of matches that occur in the first round. This gives E[Rn ] =
n
E[Rn |Xn = i]P {Xn = i}
i=0
Now, given a total of i matches in the initial round, the number of rounds needed will equal 1 plus the number of rounds that are required when n − i persons are to be matched with their hats. Therefore, E[Rn ] =
n (1 + E[Rn−i ])P {Xn = i} i=0
= 1 + E[Rn ]P {Xn = 0} +
n
E[Rn−i ]P {Xn = i}
i=1
= 1 + E[Rn ]P {Xn = 0} +
n (n − i)P {Xn = i} i=1
by the induction hypothesis = 1 + E[Rn ]P {Xn = 0} + n(1 − P {Xn = 0}) − E[Xn ] = E[Rn ]P {Xn = 0} + n(1 − P {Xn = 0}) where the final equality used the result, established in Example 2.31, that E[Xn ] = 1. Since the preceding equation implies that E[Rn ] = n, the result is proven.
3.4.
Computing Expectations by Conditioning
111
(b) For n 2, conditioning on Xn , the number of matches in round 1, gives E[Sn ] =
n
E[Sn |Xn = i]P {Xn = i}
i=0
=
n (n + E[Sn−i ])P {Xn = i} i=0
=n+
n
E[Sn−i ]P {Xn = i}
i=0
where E[S0 ] = 0. To solve the preceding equation, rewrite it as E[Sn ] = n + E[Sn−Xn ] Now, if there were exactly one match in each round, then it would take a total of 1 + 2 + · · · + n = n(n + 1)/2 selections. Thus, let us try a solution of the form E[Sn ] = an + bn2 . For the preceding equation to be satisfied by a solution of this type, for n 2, we need an + bn2 = n + E[a(n − Xn ) + b(n − Xn )2 ] or, equivalently, an + bn2 = n + a(n − E[Xn ]) + b(n2 − 2nE[Xn ] + E[Xn2 ]) Now, using the results of Example 2.31 and Exercise 72 of Chapter 2 that E[Xn ]=Var(Xn ) = 1, the preceding will be satisfied if an + bn2 = n + an − a + bn2 − 2nb + 2b and this will be valid provided that b = 1/2, a = 1. That is, E[Sn ] = n + n2 /2 satisfies the recursive equation for E[Sn ]. The formal proof that E[Sn ] = n + n2 /2, n 2, is obtained by induction on n. It is true when n = 2 (since, in this case, the number of selections is twice the number of rounds and the number of rounds is a geometric random variable
112
3
Conditional Probability and Conditional Expectation
with parameter p = 1/2). Now, the recursion gives that E[Sn ] = n + E[Sn ]P {Xn = 0} +
n
E[Sn−i ]P {Xn = i}
i=1
Hence, upon assuming that E[S0 ] = E[S1 ] = 0, E[Sk ] = k + k 2 /2, for k = 2, . . . , n − 1 and using that P {Xn = n − 1} = 0, we see that E[Sn ] = n + E[Sn ]P {Xn = 0} +
n
[n − i + (n − i)2 /2]P {Xn = i}
i=1
= n + E[Sn ]P {Xn = 0} + (n + n2 /2)(1 − P {Xn = 0}) − (n + 1)E[Xn ] + E[Xn2 ]/2 Substituting the identities E[Xn ] = 1, E[Xn2 ] = 2 in the preceding shows that E[Sn ] = n + n2 /2 and the induction proof is complete. (c) If we let Cj denote the number of hats chosen by person j, j = 1, . . . , n then n
Cj = S n
j =1
Taking expectations, and using the fact that each Cj has the same mean, yields the result E[Cj ] = E[Sn ]/n = 1 + n/2 Hence, the expected number of false selections by person j is E[Cj − 1] = n/2.
Example 3.14 Independent trials, each of which is a success with probability p, are performed until there are k consecutive successes. What is the mean number of necessary trials? Solution: Let Nk denote the number of necessary trials to obtain k consecutive successes, and let Mk denote its mean. We will obtain a recursive equation
3.4.
Computing Expectations by Conditioning
113
for Mk by conditioning on Nk−1 , the number of trials needed for k − 1 consecutive successes. This yields " # Mk = E[Nk ] = E E[Nk |Nk−1 ] Now, E[Nk |Nk−1 ] = Nk−1 + 1 + (1 − p)E[Nk ] where the preceding follows since if it takes Nk−1 trials to obtain k − 1 consecutive successes, then either the next trial is a success and we have our k in a row or it is a failure and we must begin anew. Taking expectations of both sides of the preceding yields Mk = Mk−1 + 1 + (1 − p)Mk or Mk =
1 Mk−1 + p p
Since N1 , the time of the first success, is geometric with parameter p, we see that M1 =
1 p
and, recursively M2 =
1 1 + , p p2
M3 =
1 1 1 + 2+ 3 p p p
and, in general, Mk =
1 1 1 + + ··· + k p p2 p
Example 3.15 (Analyzing the Quick-Sort Algorithm) Suppose we are given a set of n distinct values—x1 , . . . , xn —and we desire to put these values in increasing order or, as it is commonly called, to sort them. An efficient procedure for accomplishing this is the quick-sort algorithm which is defined recursively as follows: When n = 2 the algorithm compares the two values and puts them in the appropriate order. When n > 2 it starts by choosing at random one of the n values—say, xi —and then compares each of the other n − 1 values with xi , noting which are smaller and which are larger than xi . Letting Si denote the set of
114
3
Conditional Probability and Conditional Expectation
elements smaller than xi , and S¯i the set of elements greater than xi , the algorithm now sorts the set Si and the set S¯i . The final ordering, therefore, consists of the ordered set of the elements in Si , then xi , and then the ordered set of the elements in S¯i . For instance, suppose that the set of elements is 10, 5, 8, 2, 1, 4, 7. We start by choosing one of these values at random (that is, each of the 7 values has probability of 17 of being chosen). Suppose, for instance, that the value 4 is chosen. We then compare 4 with each of the other six values to obtain {2, 1}, 4, {10, 5, 8, 7} We now sort the set {2, 1} to obtain 1, 2, 4, {10, 5, 8, 7} Next we choose a value at random from {10, 5, 8, 7}—say 7 is chosen—and compare each of the other three values with 7 to obtain 1, 2, 4, 5, 7, {10, 8} Finally, we sort {10, 8} to end up with 1, 2, 4, 5, 7, 8, 10 One measure of the effectiveness of this algorithm is the expected number of comparisons that it makes. Let us denote by Mn the expected number of comparisons needed by the quick-sort algorithm to sort a set of n distinct values. To obtain a recursion for Mn we condition on the rank of the initial value selected to obtain: Mn =
n
E[number of comparisons|value selected is jth smallest]
j =1
1 n
Now if the initial value selected is the j th smallest, then the set of values smaller than it is of size j − 1, and the set of values greater than it is of size n − j . Hence, as n − 1 comparisons with the initial value chosen must be made, we see that Mn =
n
(n − 1 + Mj −1 + Mn−j )
j =1
2 Mk n
1 n
n−1
=n−1+
k=1
(since M0 = 0)
3.4.
Computing Expectations by Conditioning
or, equivalently, nMn = n(n − 1) + 2
n−1
Mk
k=1
To solve the preceding, note that upon replacing n by n + 1 we obtain (n + 1)Mn+1 = (n + 1)n + 2
n
Mk
k=1
Hence, upon subtraction, (n + 1)Mn+1 − nMn = 2n + 2Mn or (n + 1)Mn+1 = (n + 2)Mn + 2n Therefore, 2n Mn Mn+1 = + n + 2 (n + 1)(n + 2) n + 1 Iterating this gives Mn+1 2n 2(n − 1) Mn−1 = + + n+2 (n + 1)(n + 2) n(n + 1) n = ··· =2
n−1 k=0
n−k (n + 1 − k)(n + 2 − k)
since M1 = 0
Hence, Mn+1 = 2(n + 2)
n−1 k=0
= 2(n + 2)
n i=1
n−k (n + 1 − k)(n + 2 − k) i , (i + 1)(i + 2)
n1
115
116
3
Conditional Probability and Conditional Expectation
Using the identity i/(i + 1)(i + 2) = 2/(i + 2) − 1/(i + 1), we can approximate Mn+1 for large n as follows: , Mn+1 = 2(n + 2)
n i=1
. ∼ 2(n + 2)
1 2 − i +2 i +1 n
-
i=1
n+2
3
2 dx − x
n+1
2
1 dx x
/
= 2(n + 2)[2 log(n + 2) − log(n + 1) + log 2 − 2 log 3] / . n+2 + log 2 − 2 log 3 = 2(n + 2) log(n + 2) + log n+1 ∼ 2(n + 2) log(n + 2) Although we usually employ the conditional expectation identity to more easily enable us to compute an unconditional expectation, in our next example we show how it can sometimes be used to obtain the conditional expectation.
Example 3.16 In the match problem of Example 2.31 involving n, n > 1, individuals, find the conditional expected number of matches given that the first person did not have a match. Solution: Let X denote the number of matches, and let X1 equal 1 if the first person has a match and let it equal 0 otherwise. Then, E[X] = E[X|X1 = 0]P {X1 = 0} + E[X|X1 = 1]P {X1 = 1} = E[X|X1 = 0]
n−1 1 + E[X|X1 = 1] n n
But, from Example 2.31 E[X] = 1 Moreover, given that the first person has a match, the expected number of matches is equal to 1 plus the expected number of matches when n − 1 people select among their own n − 1 hats, showing that E[X|X1 = 1] = 2
3.4.
Computing Expectations by Conditioning
117
Therefore, we obtain the result E[X|X1 = 0] =
n−2 n−1
3.4.1. Computing Variances by Conditioning Conditional expectations can also be used to compute the variance of a random variable. Specifically, we can use that Var(X) = E[X 2 ] − (E[X])2 and then use conditioning to obtain both E[X] and E[X 2 ]. We illustrate this technique by determining the variance of a geometric random variable.
Example 3.17 (Variance of the Geometric Random Variable) Independent trials, each resulting in a success with probability p, are performed in sequence. Let N be the trial number of the first success. Find Var(N ). Solution: wise.
Let Y = 1 if the first trial results in a success, and Y = 0 other-
Var(N ) = E(N 2 ) − (E[N ])2 To calculate E[N 2 ] and E[N ] we condition on Y . For instance, " # E[N 2 ] = E E[N 2 |Y ] However, E[N 2 |Y = 1] = 1, E[N 2 |Y = 0] = E[(1 + N )2 ] These two equations are true since if the first trial results in a success, then clearly N = 1 and so N 2 = 1. On the other hand, if the first trial results in a failure, then the total number of trials necessary for the first success will equal one (the first trial that results in failure) plus the necessary number of additional trials. Since this latter quantity has the same distribution as N , we
118
3
Conditional Probability and Conditional Expectation
get that E[N 2 |Y = 0] = E[(1 + N )2 ]. Hence, we see that E[N 2 ] = E[N 2 |Y = 1]P {Y = 1} + E[N 2 |Y = 0]P {Y = 0} = p + E[(1 + N )2 ](1 − p) = 1 + (1 − p)E[2N + N 2 ] Since, as was shown in Example 3.11, E[N ] = 1/p, this yields E[N 2 ] = 1 +
2(1 − p) + (1 − p)E[N 2 ] p
or E[N 2 ] =
2−p p2
Therefore, Var(N ) = E[N 2 ] − (E[N ])2 2 2−p 1 = − p p2 =
1−p p2
Another way to use conditioning to obtain the variance of a random variable is to apply the conditional variance formula. The conditional variance of X given that Y = y is defined by " # Var(X|Y = y) = E (X − E[X|Y = y])2 |Y = y That is, the conditional variance is defined in exactly the same manner as the ordinary variance with the exception that all probabilities are determined conditional on the event that Y = y. Expanding the right side of the preceding and taking expectation term by term yield that Var(X|Y = y) = E[X 2 |Y = y] − (E[X|Y = y])2 Letting Var(X|Y ) denote that function of Y whose value when Y = y is Var(X|Y = y), we have the following result.
3.4.
Computing Expectations by Conditioning
119
Proposition 3.1 The Conditional Variance Formula " # Var(X) = E Var(X|Y ) + Var E[X|Y ]
(3.8)
Proof " # " # E Var(X|Y ) = E E[X 2 |Y ] − (E[X|Y ])2 " # # " = E E[X 2 |Y ] − E (E[X|Y ])2 # " = E[X 2 ] − E (E[X|Y ])2 and # " #2 " Var(E[X|Y ]) = E (E[X|Y ])2 − E E[X|Y ] # " = E (E[X|Y ])2 − (E[X])2 Therefore, " # E Var(X|Y ) + Var E[X|Y ] = E[X 2 ] − (E[X])2 which completes the proof.
Example 3.18 (The Variance of a Compound Random Variable) Let X1 , X2 , . . . be independent and identically distributed random variables with distribution F having mean μ and variance σ 2 , and assume that they are independent of the nonnegative integer valued random variable N. As noted in Example 3.10, where its expected value was determined, the random variable S = N i=1 Xi is called a compound random variable. Find its variance. Solution: Whereas we could obtain E[S 2 ] by conditioning on N , let us instead use the conditional variance formula. Now, N Var(S|N = n) = Var Xi |N = n = Var = Var
i=1 n
Xi |N = n
i=1 n i=1
= nσ 2
Xi
120
3
Conditional Probability and Conditional Expectation
By the same reasoning, E[S|N = n] = nμ Therefore, Var(S|N ) = N σ 2 ,
E[S|N ] = N μ
and the conditional variance formula gives that Var(S) = E[N σ 2 ] + Var(N μ) = σ 2 E[N ] + μ2 Var(N ) If N is a Poisson random variable, then S = N i=1 Xi is called a compound Poisson random variable. Because the variance of a Poisson random variable is equal to its mean, it follows that for a compound Poisson random variable having E[N ] = λ Var(S) = λσ 2 + λμ2 = λE[X 2 ] where X has the distribution F .
3.5.
Computing Probabilities by Conditioning
Not only can we obtain expectations by first conditioning on an appropriate random variable, but we may also use this approach to compute probabilities. To see this, let E denote an arbitrary event and define the indicator random variable X by 1, if E occurs X= 0, if E does not occur It follows from the definition of X that E[X] = P (E), E[X|Y = y] = P (E|Y = y),
for any random variable Y
Therefore, from Equations (3.3a) and (3.3b) we obtain P (E) = P (E|Y = y)P (Y = y), if Y is discrete y
=
∞ −∞
P (E|Y = y)fY (y) dy,
if Y is continuous
3.5.
Computing Probabilities by Conditioning
121
Example 3.19 Suppose that X and Y are independent continuous random variables having densities fX and fY , respectively. Compute P {X < Y }. Solution:
Conditioning on the value of Y yields P {X < Y } = = = =
∞
−∞ ∞ −∞ ∞ −∞ ∞ −∞
P {X < Y |Y = y}fY (y) dy P {X < y|Y = y}fY (y) dy P {X < y}fY (y) dy FX (y)fY (y) dy
where FX (y) =
y −∞
fX (x) dx
Example 3.20 An insurance company supposes that the number of accidents that each of its policyholders will have in a year is Poisson distributed, with the mean of the Poisson depending on the policyholder. If the Poisson mean of a randomly chosen policyholder has a gamma distribution with density function g(λ) = λe−λ ,
λ0
what is the probability that a randomly chosen policyholder has exactly n accidents next year? Solution: Let X denote the number of accidents that a randomly chosen policyholder has next year. Letting Y be the Poisson mean number of accidents for this policyholder, then conditioning on Y yields P {X = n} =
∞
P {X = n|Y = λ}g(λ) dλ
0
=
0
1 = n!
∞
e−λ
0
∞
λn −λ λe dλ n!
λn+1 e−2λ dλ
122
3
Conditional Probability and Conditional Expectation
However, because h(λ) =
2e−2λ (2λ)n+1 , (n + 1)!
λ>0
is the density function of a gamma (n + 2, 2) random variable, its integral is 1. Therefore, 1= 0
∞
2n+2 2e−2λ (2λ)n+1 dλ = (n + 1)! (n + 1)!
∞
λn+1 e−2λ dλ
0
showing that P {X = n} =
n+1 2n+2
Example 3.21 Suppose that the number of people who visit a yoga studio each day is a Poisson random variable with mean λ. Suppose further that each person who visits is, independently, female with probability p or male with probability 1 − p. Find the joint probability that exactly n women and m men visit the academy today. Solution: Let N1 denote the number of women, and N2 the number of men, who visit the academy today. Also, let N = N1 + N2 be the total number of people who visit. Conditioning on N gives
P {N1 = n, N2 = m} =
∞
P {N1 = n, N2 = m|N = i} P {N = i}
i=0
Because P {N1 = n, N2 = m|N = i} = 0 when i = n + m, the preceding equation yields that P {N1 = n, N2 = m} = P {N1 = n, N2 = m|N = n + m}e−λ
λn+m (n + m)!
Given that n + m people visit it follows, because each of these n + m is independently a woman with probability p, that the conditional probability that n of them are women (and m are men) is just the binomial probability of n
3.5.
Computing Probabilities by Conditioning
123
successes in n + m trials. Therefore, P {N1 = n, N2 = m} = =
λn+m n+m n p (1 − p)m e−λ (n + m)! n λn λm (n + m)! n p (1 − p)m e−λp e−λ(1−p) n!m! (n + m)!
= e−λp
(λp)n −λ(1−p) (λ(1 − p))m e n! m!
Because the preceding joint probability mass function factors into two products, one of which depends only on n and the other only on m, it follows that N1 and N2 are independent. Moreover, because P {N1 = n} =
∞
P {N1 = n, N2 = m}
m=0
= e−λp
∞ (λp)n −λ(1−p) (λ(1 − p))m (λp)n = e−λp e n! m! n! m=0
and, similarly, P {N2 = m} = e−λ(1−p)
(λ(1 − p))m m!
we can conclude that N1 and N2 are independent Poisson random variables with respective means λp and λ(1 − p). Therefore, this example establishes the important result that when each of a Poisson number of events is independently classified either as being type 1 with probability p or type 2 with probability 1 − p, then the numbers of type 1 and type 2 events are independent Poisson random variables.
Example 3.22 Let X1 , . . . , Xn be independent Bernoulli random variables,
with Xi having parameter pi , i = 1, . . . , n. That is, P {Xi = 1} = pi , P {Xi = 0} = qi = 1 − pi . Suppose we want to compute the probability mass function of their sum, X1 + · · · + Xn . To do so, we will recursively obtain the probability mass function of X1 + · · · + Xk , first for k = 1, then k = 2, and on up to k = n. To begin, let Pk (j ) = P {X1 + · · · + Xk = j }
124
3
Conditional Probability and Conditional Expectation
and note that Pk (k) =
k
pi ,
Pk (0) =
i=1
k
qi
i=1
For 0 < j < k, conditioning on Xk yields the recursion Pk (j ) = P {X1 + · · · + Xk = j |Xk = 1}pk + P {X1 + · · · + Xk = j |Xk = 0}qk = P {X1 + · · · + Xk−1 = j − 1|Xk = 1}pk + P {X1 + · · · + Xk−1 = j |Xk = 0}qk = P {X1 + · · · + Xk−1 = j − 1}pk + P {X1 + · · · + Xk−1 = j }qk = pk Pk−1 (j − 1) + qk Pk−1 (j ) Starting with P1 (1) = p1 , P1 (0) = q1 , the preceding equations can be recursively solved to obtain the functions P2 (j ), P3 (j ), up to Pn (j ).
Example 3.23 (The Best Prize Problem) Suppose that we are to be presented with n distinct prizes in sequence. After being presented with a prize we must immediately decide whether to accept it or reject it and consider the next prize. The only information we are given when deciding whether to accept a prize is the relative rank of that prize compared to ones already seen. That is, for instance, when the fifth prize is presented we learn how it compares with the first four prizes already seen. Suppose that once a prize is rejected it is lost, and that our objective is to maximize the probability of obtaining the best prize. Assuming that all n! orderings of the prizes are equally likely, how well can we do? Solution: Rather surprisingly, we can do quite well. To see this, fix a value k, 0 k < n, and consider the strategy that rejects the first k prizes and then accepts the first one that is better than all of those first k. Let Pk (best) denote the probability that the best prize is selected when this strategy is employed. To compute this probability, condition on X, the position of the best prize. This gives Pk (best) =
n
Pk (best|X = i)P (X = i)
i=1
1 Pk (best|X = i) n n
=
i=1
Now, if the overall best prize is among the first k, then no prize is ever selected under the strategy considered. On the other hand, if the best prize is in position i, where i > k, then the best prize will be selected if the best of the first
3.5.
Computing Probabilities by Conditioning
125
k prizes is also the best of the first i − 1 prizes (for then none of the prizes in positions k + 1, k + 2, . . . , i − 1 would be selected). Hence, we see that Pk (best|X = i) = 0,
if i k
Pk (best|X = i) = P {best of first i − 1 is among the first k} = k/(i − 1),
if i > k
From the preceding, we obtain that n k 1 Pk (best) = n i −1 i=k+1
n−1
1 dx x k n−1 k = log n k $ % n k ≈ log n k
≈
k n
Now, if we consider the function g(x) =
$n% x log n x
then g (x) =
$n% 1 1 log − n x n
and so g (x) = 0 ⇒ log(n/x) = 1 ⇒ x = n/e Thus, since Pk (best) ≈ g(k), we see that the best strategy of the type considered is to let the first n/e prizes go by and then accept the first one to appear that is better than all of those. In addition, since g(n/e) = 1/e, the probability that this strategy selects the best prize is approximately 1/e ≈ 0.36788.
126
3
Conditional Probability and Conditional Expectation
Remark Most students are quite surprised by the size of the probability of obtaining the best prize, thinking that this probability would be close to 0 when n is large. However, even without going through the calculations, a little thought reveals that the probability of obtaining the best prize can be made to be reasonably large. Consider the strategy of letting half of the prizes go by, and then selecting the first one to appear that is better than all of those. The probability that a prize is actually selected is the probability that the overall best is among the second half and this is 1/2. In addition, given that a prize is selected, at the time of selection that prize would have been the best of more than n/2 prizes to have appeared, and would thus have probability of at least 1/2 of being the overall best. Hence, the strategy of letting the first half of all prizes go by and then accepting the first one that is better than all of those prizes results in a probability greater than 1/4 of obtaining the best prize.
Example 3.24 At a party n men take off their hats. The hats are then mixed up and each man randomly selects one. We say that a match occurs if a man selects his own hat. What is the probability of no matches? What is the probability of exactly k matches? Solution: Let E denote the event that no matches occur, and to make explicit the dependence on n, write Pn = P (E). We start by conditioning on whether or not the first man selects his own hat—call these events M and M c . Then Pn = P (E) = P (E|M)P (M) + P (E|M c )P (M c ) Clearly, P (E|M) = 0, and so Pn = P (E|M c )
n−1 n
(3.9)
Now, P (E|M c ) is the probability of no matches when n − 1 men select from a set of n − 1 hats that does not contain the hat of one of these men. This can happen in either of two mutually exclusive ways. Either there are no matches and the extra man does not select the extra hat (this being the hat of the man that chose first), or there are no matches and the extra man does select the extra hat. The probability of the first of these events is just Pn−1 , which is seen by regarding the extra hat as “belonging” to the extra man. Because the second event has probability [1/(n − 1)]Pn−2 , we have P (E|M c ) = Pn−1 +
1 Pn−2 n−1
and thus, from Equation (3.9), Pn =
n−1 1 Pn−1 + Pn−2 n n
3.5.
Computing Probabilities by Conditioning
127
or, equivalently, 1 Pn − Pn−1 = − (Pn−1 − Pn−2 ) n
(3.10)
However, because Pn is the probability of no matches when n men select among their own hats, we have P1 = 0,
P2 =
1 2
and so, from Equation (3.10), 1 (P2 − P1 ) =− 3 3! 1 (P3 − P2 ) P4 − P3 = − = 4 4!
P3 − P2 = −
or or
1 1 − , 2! 3! 1 1 1 P4 = − + 2! 3! 4! P3 =
and, in general, we see that Pn =
1 1 (−1)n 1 − + − ··· + 2! 3! 4! n!
To obtain the probability of exactly k matches, we consider any fixed group of k men. The probability that they, and only they, select their own hats is 1 (n − k)! 1 1 ··· Pn−k = Pn−k nn−1 n − (k − 1) n! where Pn−k is the conditional probability that the other n −k men, selecting among their own hats, have no matches. Because there are nk choices of a set of k men, the desired probability of exactly k matches is 1 1 (−1)n−k − + ··· + Pn−k 2! 3! (n − k)! = k! k! which, for n large, is approximately equal to e−1 /k!.
Remark The recursive equation, Equation (3.10), could also have been obtained by using the concept of a cycle, where we say that the sequence of distinct individuals i1 , i2 , . . . , ik constitutes a cycle if i1 chooses i2 ’s hat, i2 chooses i3 ’s hat, . . . , ik−1 chooses ik ’s hat, and ik chooses i1 ’s hat. Note that every individual is part of a cycle, and that a cycle of size k = 1 occurs when someone chooses his
128
3
Conditional Probability and Conditional Expectation
or her own hat. With E being, as before, the event that no matches occur, it follows upon conditioning on the size of the cycle containing a specified person, say person 1, that Pn = P (E) =
n
P (E|C = k)P (C = k)
(3.11)
k=1
where C is the size of the cycle that contains person 1. Now call person 1 the first person, and note that C = k if the first person does not choose 1’s hat; the person whose hat was chosen by the first person—call this person the second person— does not choose 1’s hat; the person whose hat was chosen by the second person— call this person the third person—does not choose 1’s hat; . . . , the person whose hat was chosen by the (k − 1)st person does choose 1’s hat. Consequently, P (C = k) =
n−k+1 1 1 n−1 n−2 ··· = n n−1 n−k+2 n−k+1 n
(3.12)
That is, the size of the cycle that contains a specified person is equally likely to be any of the values 1, 2, . . . , n. Moreover, since C = 1 means that 1 chooses his or her own hat, it follows that P (E|C = 1) = 0
(3.13)
On the other hand, if C = k, then the set of hats chosen by the k individuals in this cycle is exactly the set of hats of these individuals. Hence, conditional on C = k, the problem reduces to determining the probability of no matches when n − k people randomly choose among their own n − k hats. Therefore, for k > 1 P (E|C = k) = Pn−k Substituting (3.12), (3.13), and (3.14) back into Equation (3.11) gives 1 Pn−k n n
Pn =
(3.14)
k=2
which is easily shown to be equivalent to Equation (3.10).
Example 3.25 (The Ballot Problem) In an election, candidate A receives n votes, and candidate B receives m votes where n > m. Assuming that all orderings are equally likely, show that the probability that A is always ahead in the count of votes is (n − m)/(n + m).
3.5.
Computing Probabilities by Conditioning
129
Solution: Let Pn,m denote the desired probability. By conditioning on which candidate receives the last vote counted we have n n+m m + P {A always ahead|B receives last vote} n+m
Pn,m = P {A always ahead|A receives last vote}
Now given that A receives the last vote, we can see that the probability that A is always ahead is the same as if A had received a total of n − 1 and B a total of m votes. Because a similar result is true when we are given that B receives the last vote, we see from the preceding that Pn,m =
n m Pn−1,m + Pn,m−1 n+m m+n
(3.15)
We can now prove that Pn,m = (n − m)/(n + m) by induction on n + m. As it is true when n + m = 1, that is, P1,0 = 1, assume it whenever n + m = k. Then when n + m = k + 1, we have by Equation (3.15) and the induction hypothesis that m n−m+1 n n−1−m + n+m n−1+m m+n n+m−1 n−m = n+m
Pn,m =
and the result is proven.
The ballot problem has some interesting applications. For example, consider successive flips of a coin that always land on “heads” with probability p, and let us determine the probability distribution of the first time, after beginning, that the total number of heads is equal to the total number of tails. The probability that the first time this occurs is at time 2n can be obtained by first conditioning on the total number of heads in the first 2n trials. This yields P{first time equal = 2n}
2n n = P {first time equal = 2n|n heads in first 2n} p (1 − p)n n
Now given a total of n heads in the first 2n flips we can see that all possible orderings of the n heads and n tails are equally likely, and thus the preceding conditional probability is equivalent to the probability that in an election, in which each candidate receives n votes, one of the candidates is always ahead in the counting until the last vote (which ties them). But by conditioning on whomever
130
3
Conditional Probability and Conditional Expectation
receives the last vote, we see that this is just the probability in the ballot problem when m = n − 1. Hence 2n n p (1 − p)n P {first time equal = 2n} = Pn,n−1 n 2n n p (1 − p)n n = 2n − 1 Suppose now that we wanted to determine the probability that the first time there are i more heads than tails occurs after the (2n + i)th flip. Now, in order for this to be the case, the following two events must occur: (a) The first 2n + i tosses result in n + i heads and n tails; and (b) The order in which the n + i heads and n tails occur is such that the number of heads is never i more than the number of tails until after the final flip. Now, it is easy to see that event (b) will occur if and only if the order of appearance of the n + i heads and n tails is such that starting from the final flip and working backwards heads is always in the lead. For instance, if there are 4 heads and 2 tails (n = 2, i = 2), then the outcome _ _ _ _T H would not suffice because there would have been 2 more heads than tails sometime before the sixth flip (since the first 4 flips resulted in 2 more heads than tails). Now, the probability of the event specified in (a) is just the binomial probability of getting n + i heads and n tails in 2n + i flips of the coin. We must now determine the conditional probability of the event specified in (b) given that there are n + i heads and n tails in the first 2n + i flips. To do so, note first that given that there are a total of n + i heads and n tails in the first 2n + i flips, all possible orderings of these flips are equally likely. As a result, the conditional probability of (b) given (a) is just the probability that a random ordering of n + i heads and n tails will, when counted in reverse order, always have more heads than tails. Since all reverse orderings are also equally likely, it follows from the ballot problem that this conditional probability is i/(2n + i). That is, we have shown that 2n + i n+i P {a} = p (1 − p)n , n P {b|a} =
i 2n + i
and so P {first time heads leads by i is after flip 2n + i} =
2n + i n+i i p (1 − p)n 2n + i n
3.5.
Computing Probabilities by Conditioning
131
Example 3.26 Let U1 , U2 , . . . be a sequence of independent uniform (0, 1) random variables, and let N = min{n 2: Un > Un−1 } and M = min{n 1: U1 + · · · + Un > 1} That is, N is the index of the first uniform random variable that is larger than its immediate predecessor, and M is the number of uniform random variables we need sum to exceed 1. Surprisingly, N and M have the same probability distribution, and their common mean is e! Solution: It is easy to find the distribution of N . Since all n! possible orderings of U1 , . . . , Un are equally likely, we have P {N > n} = P {U1 > U2 > · · · > Un } = 1/n! To show that P {M > n} = 1/n!, we will use mathematical induction. However, to give ourselves a stronger result to use as the induction hypothesis, we will prove the stronger result that for 0 < x 1, P {M(x) > n} = x n /n!, n 1, where M(x) = min{n 1: U1 + · · · + Un > x} is the minimum number of uniforms that need be summed to exceed x. To prove that P {M(x) > n} = x n /n!, note first that it is true for n = 1 since P {M(x) > 1} = P {U1 x} = x So assume that for all 0 < x 1, P {M(x) > n} = x n /n!. To determine P {M(x) > n + 1}, condition on U1 to obtain: P {M(x) > n + 1} =
1
P {M(x) > n + 1|U1 = y} dy
0
=
x
P {M(x) > n + 1|U1 = y} dy
0
=
x
P {M(x − y) > n} dy
0
=
0
x
(x − y)n dy n!
by the induction hypothesis
132
3
Conditional Probability and Conditional Expectation
x
= 0
=
un du n!
x n+1 (n + 1)!
where the third equality of the preceding follows from the fact that given U1 = y, M(x) is distributed as 1 plus the number of uniforms that need be summed to exceed x − y. Thus, the induction is complete and we have shown that for 0 < x 1, n 1, P {M(x) > n} = x n /n! Letting x = 1 shows that N and M have the same distribution. Finally, we have that E[M] = E[N ] =
∞ n=0
P {N > n} =
∞
1/n! = e
n=0
Example 3.27 Let X1 , X2 , . . . be independent continuous random variables with a common distribution function F and density f = F , and suppose that they are to be observed one at a time in sequence. Let N = min{n 2: Xn = second largest of X1 , . . . , Xn } and let M = min{n 2: Xn = second smallest of X1 , . . . , Xn } Which random variable—XN , the first random variable which when observed is the second largest of those that have been seen, or XM , the first one that on observation is the second smallest to have been seen—tends to be larger? Solution: To calculate the probability density function of XN , it is natural to condition on the value of N ; so let us start by determining its probability mass function. Now, if we let Ai = {Xi = second largest of X1 , . . . , Xi },
i2
then, for n 2, P {N = n} = P A2 A3 · · · An−1 Acn Since the Xi are independent and identically distributed it follows that, for any m 1, knowing the rank ordering of the variables X1 , . . . , Xm yields no
3.5.
Computing Probabilities by Conditioning
133
information about the set of m values {X1 , . . . , Xm }. That is, for instance, knowing that X1 < X2 gives us no information about the values of min(X1 , X2 ) or max(X1 , X2 ). It follows from this that the events Ai , i 2 are independent. Also, since Xi is equally likely to be the largest, or the second largest, . . . , or the ith largest of X1 , . . . , Xi it follows that P {Ai } = (i − 1)/i, i 2. Therefore, we see that P {N = n} =
n−2 1 1 123 ··· = 234 n − 1 n n(n − 1)
Hence, conditioning on N yields that the probability density function of XN is as follows: fXN (x) =
∞ n=2
1 fX |N (x|n) n(n − 1) N
Now, since the ordering of the variables X1 , . . . , Xn is independent of the set of values {X1 , . . . , Xn }, it follows that the event {N = n} is independent of {X1 , . . . , Xn }. From this, it follows that the conditional distribution of XN given that N = n is equal to the distribution of the second largest from a set of n random variables having distribution F . Thus, using the results of Example 2.37 concerning the density function of such a random variable, we obtain that fXN (x) =
∞ n=2
n! 1 (F (x))n−2 f (x)(1 − F (x)) n(n − 1) (n − 2)!1!
= f (x)(1 − F (x))
∞
(F (x))i
i=0
= f (x) Thus, rather surprisingly, XN has the same distribution as X1 , namely, F . Also, if we now let Wi = −Xi , i 1, then WM will be the value of the first Wi , which on observation is the second largest of all those that have been seen. Hence, by the preceding, it follows that WM has the same distribution as W1 . That is, −XM has the same distribution as −X1 , and so XM also has distribution F ! In other words, whether we stop at the first random variable that is the second largest of all those presently observed, or we stop at the first one that is the second smallest of all those presently observed, we will end up with a random variable having distribution F . Whereas the preceding result is quite surprising, it is a special case of a general result known as Ignatov’s theorem, which yields even more surprises.
134
3
Conditional Probability and Conditional Expectation
For instance, for k 1, let Nk = min{n k: Xn = kth largest of X1 , . . . , Xn } Therefore, N2 is what we previously called N , and XNk is the first random variable that upon observation is the kth largest of all those observed up to this point. It can then be shown by a similar argument as used in the preceding that XNk has distribution function F for all k (see Exercise 82 at the end of this chapter). In addition, it can be shown that the random variables XNk , k 1 are independent. (A statement and proof of Ignatov’s theorem in the case of discrete random variables are given in Section 3.6.6.) The use of conditioning can also result in a more computationally efficient solution than a direct calculation. This is illustrated by our next example.
Example 3.28 Consider n independent trials in which each trial results in one of the outcomes 1, . . . , k with respective probabilities p1 , . . . , pk , ki=1 pi = 1. Suppose further that n > k, and that we are interested in determining the probability that each outcome occurs at least once. If we let Ai denote the event that outcome
i does not occur in any of the n trials, then the desired probability is 1 − P ( ki=1 Ai ), and it can be obtained by using the inclusion–exclusion theorem as follows: k k
Ai = P (Ai ) − P (Ai Aj ) P i=1
i=1
+
i
i
j >i
P (Ai Aj Ak ) − · · · + (−1)k+1 P (A1 · · · Ak )
j >i k>j
where P (Ai ) = (1 − pi )n P (Ai Aj ) = (1 − pi − pj )n ,
i<j
P (Ai Aj Ak ) = (1 − pi − pj − pk ) , n
i<j 0 the resulting conditional probability will equal the probability that all of the outcomes 1, . . . , k − 1 occur at least once
3.5.
Computing Probabilities by Conditioning
135
when n − Nk trials are performed, and each results in outcome i with probability pi / k−1 j =1 pj , i = 1, . . . , k − 1. We could then use a similar conditioning step on these terms. To follow through on the preceding idea, let Am,r , for m n, r k, denote the event that each of the outcomes 1, . . . , r occurs at least once when m independent trials are performed, where each trial results in one of theoutcomes 1, . . . , r with respective probabilities p1 /Pr , . . . , pr /Pr , where Pr = rj =1 pj . Let P (m, r) = P (Am,r ) and note that P (n, k) is the desired probability. To obtain an expression for P (m, r), condition on the number of times that outcome r occurs. This gives j m pr m−j m pr P (m, r) = 1− P {Am,r |r occurs j times} Pr Pr j j =0
=
m−r+1 j =1
j pr m−j m pr 1− P (m − j, r − 1) Pr Pr j
Starting with P (m, 1) = 1, P (m, 1) = 0,
if m 1 if m = 0
we can use the preceding recursion to obtain the quantities P (m, 2), m = 2, . . . , n − (k − 2), and then the quantities P (m, 3), m = 3, . . . , n − (k − 3), and so on, up to P (m, k − 1), m = k − 1, . . . , n − 1. At this point we can then use the recursion to compute P (n, k). It is not difficult to check that the amount of computation needed is a polynomial function of k, which will be much smaller than 2k when k is large. As noted previously, conditional expectations given that Y = y are exactly the same as ordinary expectations except that all probabilities are computed conditional on the event that Y = y. As such, conditional expectations satisfy all the properties of ordinary expectations. For instance, the analog of ⎧ ⎪ E[X|W = w]P {W = w}, if W is discrete ⎪ ⎨ w E[X] = ⎪ ⎪ ⎩ E[X|W = w]fW (w) dw, if W is continuous w
is that E[X|Y = y] ⎧ E[X|W = w, Y = y]P {W = w|Y = y}, ⎪ ⎪ ⎨ w = ⎪ ⎪ ⎩ E[X|W = w, Y = y]fW |Y (w|y) dw, w
if W is discrete if W is continuous
136
3
Conditional Probability and Conditional Expectation
If E[X|Y, W ] is defined to be that function of Y and W that, when Y = y and W = w, is equal to E[X|Y = y, W = w], then the preceding can be written as ! # " E[X|Y ] = E E[X|Y, W ]!Y
Example 3.29 An automobile insurance company classifies each of its pol-
icyholders as being of one of the types i = 1, . . . , k. It supposes that the numbers of accidents that a type i policyholder has in successive years are independent Poisson random variables with mean λi , i = 1, . . . , k. The probability that a newly insured policyholder is type i is pi , ki=1 pi = 1. Given that a policyholder had n accidents in her first year, what is the expected number that she has in her second year? What is the conditional probability that she has m accidents in her second year? Solution: Let Ni denote the number of accidents the policyholder has in year i, i = 1, 2. To obtain E[N2 |N1 = n], condition on her risk type T . E[N2 |N1 = n] =
k
E[N2 |T = j, N1 = n]P {T = j |N1 = n}
j =1
=
k
E[N2 |T = j ]P {T = j |N1 = n}
j =1
=
k
λj P {T = j |N1 = n}
j =1
k
−λj λn+1 p j j n −λ jλ p j =1 e j j
j =1 e
= k
where the final equality used that P {T = j |N1 = n} =
P {T = j, N1 = n} P {N1 = n}
P {N1 = n|T = j }P {T = j } = k j =1 P {N1 = n|T = j }P {T = j } pj e−λj λnj /n!
= k
j =1 pj e
−λj λn /n! j
3.6.
Some Applications
137
The conditional probability that the policyholder has m accidents in year 2 given that she had n in year 1 can also be obtained by conditioning on her type. P {N2 = m|N1 = n} =
k
P {N2 = m|T = j, N1 = n}P {T = j |N1 = n}
j =1
=
k
e−λj
j =1
λm j m!
P {T = j |N1 = n}
k
=
−2λj λm+n p j j =1 e j k n −λ j m! j =1 e λj pj
Another way to calculate P {N2 = m|N1 = n} is first to write P {N2 = m|N1 = n} =
P {N2 = m, N1 = n} P {N1 = n}
and then determine both the numerator and denominator by conditioning on T . This yields k
j =1 P {N2 = m, N1 = n|T = j }pj k j =1 P {N1 = n|T = j }pj
P {N2 = m|N1 = n} =
k
j =1 e
=
k
k =
3.6.
m −λj λj m!
j =1 e
e−λj
λnj n! pj
n −λj λj p n! j
−2λj λm+n p j j =1 e j k n −λ m! j =1 e j λj pj
Some Applications
3.6.1. A List Model Consider n elements—e1 , e2 , . . . , en —that are initially arranged in some ordered list. At each unit of time a request is made for one of these elements—ei being requested, independently of the past, with probability Pi . After being requested the element is then moved to the front of the list. That is, for instance, if the present ordering is e1 , e2 , e3 , e4 and if e3 is requested, then the next ordering is e3 , e1 , e2 , e4 .
138
3
Conditional Probability and Conditional Expectation
We are interested in determining the expected position of the element requested after this process has been in operation for a long time. However, before computing this expectation, let us note two possible applications of this model. In the first we have a stack of reference books. At each unit of time a book is randomly selected and is then returned to the top of the stack. In the second application we have a computer receiving requests for elements stored in its memory. The request probabilities for the elements may not be known, so to reduce the average time it takes the computer to locate the element requested (which is proportional to the position of the requested element if the computer locates the element by starting at the beginning and then going down the list), the computer is programmed to replace the requested element at the beginning of the list. To compute the expected position of the element requested, we start by conditioning on which element is selected. This yields E [ position of element requested ] =
n
E [ position|ei is selected ]Pi
i=1
= =
n i=1 n
E [ position of ei |ei is selected ]Pi E [ position of ei ]Pi
(3.16)
i=1
where the final equality used that the position of ei and the event that ei is selected are independent because, regardless of its position, ei is selected with probability Pi . Now position of ei = 1 + Ij j =i
where
Ij =
1,
if ej precedes ei
0,
otherwise
and so, E [ position of ei ] = 1 +
E [Ij ]
j =i
=1+
j =i
P {ej precedes ei }
(3.17)
3.6.
Some Applications
139
To compute P {ej precedes ei }, note that ej will precede ei if the most recent request for either of them was for ej . But given that a request is for either ei or ej , the probability that it is for ej is P {ej |ei or ej } =
Pj Pi + Pj
and, thus, P {ej precedes ei } =
Pj Pi + Pj
Hence from Equations (3.16) and (3.17) we see that E{position of element requested} = 1 +
n i=1
Pi
j =i
Pj Pi + Pj
This list model will be further analyzed in Section 4.8, where we will assume a different reordering rule—namely, that the element requested is moved one closer to the front of the list as opposed to being moved to the front of the list as assumed here. We will show there that the average position of the requested element is less under the one-closer rule than it is under the front-of-the-line rule. 3.6.2. A Random Graph A graph consists of a set V of elements called nodes and a set A of pairs of elements of V called arcs. A graph can be represented graphically by drawing circles for nodes and drawing lines between nodes i and j whenever (i, j ) is an arc. For instance if V = {1, 2, 3, 4} and A = {(1, 2), (1, 4), (2, 3), (1, 2), (3, 3)}, then we can represent this graph as shown in Figure 3.1. Note that the arcs have no direction (a graph in which the arcs are ordered pairs of nodes is called a directed graph); and that in the figure there are multiple arcs connecting nodes 1 and 2, and a self-arc (called a self-loop) from node 3 to itself.
Figure 3.1. A graph.
140
3
Conditional Probability and Conditional Expectation
Figure 3.2. A disconnected graph.
Figure 3.3.
We say that there exists a path from node i to node j , i = j , if there exists a sequence of nodes i, i1 , . . . , ik , j such that (i, i1 ), (i1 , i2 ), . . . , (ik , j ) are all arcs. If there is a path between each of the n2 distinct pair of nodes we say that the graph is connected. The graph in Figure 3.1 is connected but the graph in Figure 3.2 is not. Consider now the following graph where V = {1, 2, . . . , n} and A = {(i, X(i)), i = 1, . . . , n} where the X(i) are independent random variables such that 1 P {X(i) = j } = , n
j = 1, 2, . . . , n
In other words from each node i we select at random one of the n nodes (including possibly the node i itself) and then join node i and the selected node with an arc. Such a graph is commonly referred to as a random graph. We are interested in determining the probability that the random graph so obtained is connected. As a prelude, starting at some node—say, node 1—let us follow the sequence of nodes, 1, X(1), X 2 (1), . . . , where X n (1) = X(X n−1 (1)); and define N to equal the first k such that X k (1) is not a new node. In other words, N = 1st k such that X k (1) ∈ {1, X(1), . . . , X k−1 (1)} We can represent this as shown in Figure 3.3 where the arc from X N −1 (1) goes back to a node previously visited. To obtain the probability that the graph is connected we first condition on N to obtain P {graph is connected} =
n k=1
P {connected|N = k}P {N = k}
(3.18)
3.6.
Some Applications
141
Now given that N = k, the k nodes 1, X(1), . . . , X k−1 (1) are connected to each other, and there are no other arcs emanating out of these nodes. In other words, if we regard these k nodes as being one supernode, the situation is the same as if we had one supernode and n − k ordinary nodes with arcs emanating from the ordinary nodes—each arc going into the supernode with probability k/n. The solution in this situation is obtained from Lemma 3.2 by taking r = n − k.
Lemma 3.1 Given a random graph consisting of nodes 0, 1, . . . , r and r arcs—namely, (i, Yi ), i = 1, . . . , r, where
Yi =
⎧ ⎪ ⎪ ⎨j
with probability
1 , r +k
⎪ ⎪ ⎩0
with probability
k r +k
j = 1, . . . , r
then P {graph is connected} =
k r +k
[In other words, for the preceding graph there are r + 1 nodes—r ordinary nodes and one supernode. Out of each ordinary node an arc is chosen. The arc goes to the supernode with probability k/(r + k) and to each of the ordinary ones with probability 1/(r + k). There is no arc emanating out of the supernode.]
Proof The proof is by induction on r. As it is true when r = 1 for any k, assume it true for all values less than r. Now in the case under consideration, let us first condition on the number of arcs (j, Yj ) for which Yj = 0. This yields P {connected} =
r i=0
P {connected|i of the Yj = 0}
i r−i r r k r +k r +k i
(3.19)
Now given that exactly i of the arcs are into the supernode (see Figure 3.4), the situation for the remaining r − i arcs which do not go into the supernode is the same as if we had r − i ordinary nodes and one supernode with an arc going out of each of the ordinary nodes—into the supernode with probability i/r and into each ordinary node with probability 1/r. But by the induction hypothesis the probability that this would lead to a connected graph is i/r. Hence, P {connected|i of the Yj = 0} =
i r
142
3
Conditional Probability and Conditional Expectation
Figure 3.4. The situation given that i of the r arcs are into the supernode.
and from Equation (3.19) i r−i r r k i r P {connected} = r i r +k r +k i=0 . / k 1 = E binomial r, r r +k =
k r +k
which completes the proof of the lemma.
Hence as the situation given N =k is exactly as described by Lemma 3.2 when r = n − k, we see that, for the original graph, P {graph is connected|N = k} =
k n
and, from Equation (3.18), P {graph is connected} =
E(N ) n
To compute E(N ) we use the identity E(N ) =
∞
P {N i}
i=1
which can be proved by defining indicator variables Ii , i 1, by 1, if i N Ii = 0, if i > N
(3.20)
3.6.
Some Applications
143
Hence, N=
∞
Ii
i=1
and so
, E(N ) = E
∞
Ii
i=1
=
∞
E[Ii ]
i=1
=
∞
P {N i}
(3.21)
i=1
Now the event {N i} occurs if the nodes 1, X(1), . . . , X i−1 (1) are all distinct. Hence, (n − i + 1) (n − 1) (n − 2) ··· n n n (n − 1)! = (n − i)!ni−1
P {N i} =
and so, from Equations (3.20) and (3.21), P {graph is connected} = (n − 1)!
n i=1
1 (n − i)!ni
(n − 1)! nj nn j! n−1
=
(by j = n − i)
(3.22)
j =0
We can also use Equation (3.22) to obtain a simple approximate expression for the probability that the graph is connected when n is large. To do so, we first note that if X is a Poisson random variable with mean n, then P {X < n} = e
−n
n−1 j n j =0
j!
Since a Poisson random variable with mean n can be regarded as being the sum of n independent Poisson random variables each with mean 1, it follows from the
144
3
Conditional Probability and Conditional Expectation
central limit theorem that for n large such a random variable has approximately a normal distribution and as such has probability 12 of being less than its mean. That is, for n large P {X < n} ≈
1 2
and so for n large, n−1 j n j =0
j!
≈
en 2
Hence from Equation (3.22), for n large, P {graph is connected} ≈
en (n − 1)! 2nn
By employing an approximation due to Stirling which states that for n large √ n! ≈ nn+1/2 e−n 2π We see that, for n large, 0 P {graph is connected} ≈
n−1 n π e 2(n − 1) n
and as lim
n→∞
n−1 n
n
1 n 1− = e−1 n→∞ n
= lim
We see that, for n large, 0 P {graph is connected} ≈
π 2(n − 1)
Now a graph is said to consist of r connected components if its nodes can be partitioned into r subsets so that each of the subsets is connected and there are no arcs between nodes in different subsets. For instance, the graph in Figure 3.5 consists of three connected components—namely, {1, 2, 3}, {4, 5}, and {6}. Let C denote the number of connected components of our random graph and let Pn (i) = P {C = i}
3.6.
Some Applications
145
Figure 3.5. A graph having three connected components.
where we use the notation Pn (i) to make explicit the dependence on n, the number of nodes. Since a connected graph is by definition a graph consisting of exactly one component, from Equation (3.22) we have Pn (1) = P {C = 1} (n − 1)! nj nn j! n−1
=
(3.23)
j =0
To obtain Pn (2), the probability of exactly two components, let us first fix attention on some particular node—say, node 1. In order that a given set of k − 1 other nodes—say, nodes 2, . . . , k—will along with node 1 constitute one connected component, and the remaining n − k a second connected component, we must have (i) (ii) (iii) (iv)
X(i) ∈ {1, 2, . . . , k}, for all i = 1, . . . , k. X(i) ∈ {k + 1, . . . , n}, for all i = k + 1, . . . , n. The nodes 1, 2, . . . , k form a connected subgraph. The nodes k + 1, . . . , n form a connected subgraph.
The probability of the preceding occurring is clearly k k n − k n−k Pk (1)Pn−k (1) n n and because there are 2 through n, we have
Pn (2) =
n−1 k−1
ways of choosing a set of k − 1 nodes from the nodes
k n−1 n−1 k n − k n−k k=1
k−1
n
n
Pk (1)Pn−k (1)
146
3
Conditional Probability and Conditional Expectation
Figure 3.6. A cycle.
and so Pn (2) can be computed from Equation (3.23). In general, the recursive formula for Pn (i) is given by Pn (i) =
n−i+1 k=1
n−1 k−1
k k n − k n−k Pk (1)Pn−k (i − 1) n n
To compute E[C], the expected number of connected components, first note that every connected component of our random graph must contain exactly one cycle [a cycle is a set of arcs of the form (i, i1 ), (i1 , i2 ), . . . , (ik−1 , ik ), (ik , i) for distinct nodes i, i1 , . . . , ik ]. For example, Figure 3.6 depicts a cycle. The fact that every connected component of our random graph must contain exactly one cycle is most easily proved by noting that if the connected component consists of r nodes, then it must also have r arcs and, hence, must contain exactly one cycle (why?). Thus, we see that E[C] = E[number of cycles] . / =E I (S) =
S
E[I (S)]
S
where the sum is over all subsets S ⊂ {1, 2, . . . , n} and 1, if the nodes in S are all the nodes of a cycle I (S) = 0, otherwise Now, if S consists of k nodes, say 1, . . . , k, then E[I (S)] = P {1, X(1), . . . , X k−1 (1) are all distinct and contained in 1, . . . , k and X k (1) = 1} =
1 1 (k − 1)! k−1k−2 ··· = n n nn nk
3.6.
Hence, because there are
n k
Some Applications
147
subsets of size k we see that
E[C] =
n n (k − 1)!
nk
k
k=1
3.6.3. Uniform Priors, Polya’s Urn Model, and Bose–Einstein Statistics Suppose that n independent trials, each of which is a success with probability p, are performed. If we let X denote the total number of successes, then X is a binomial random variable such that n k = 0, 1, . . . , n P {X = k|p} = p k (1 − p)n−k , k However, let us now suppose that whereas the trials all have the same success probability p, its value is not predetermined but is chosen according to a uniform distribution on (0, 1). (For instance, a coin may be chosen at random from a huge bin of coins representing a uniform spread over all possible values of p, the coin’s probability of coming up heads. The chosen coin is then flipped n times.) In this case, by conditioning on the actual value of p, we have that P {X = k} = 0
=
0
1
P {X = k|p}f (p) dp
1 n
k
p k (1 − p)n−k dp
Now, it can be shown that
1
p k (1 − p)n−k dp =
0
k!(n − k)! (n + 1)!
(3.24)
and thus n k!(n − k)! P {X = k} = k (n + 1)! =
1 , n+1
k = 0, 1, . . . , n
In other words, each of the n + 1 possible values of X is equally likely.
(3.25)
148
3
Conditional Probability and Conditional Expectation
As an alternate way of describing the preceding experiment, let us compute the conditional probability that the (r + 1)st trial will result in a success given a total of k successes (and r − k failures) in the first r trials. P {(r + 1)st trial is a success|k successes in first r} P {(r + 1)st is a success, k successes in first r trials} P {k successes in first r trials} 1 P {(r + 1)st is a success, k in first r|p} dp = 0 1/(r + 1) 1 r p k+1 (1 − p)r−k dp = (r + 1) k 0 r (k + 1)!(r − k)! by Equation (3.24) = (r + 1) (r + 2)! k =
=
k+1 r +2
(3.26)
That is, if the first r trials result in k successes, then the next trial will be a success with probability (k + 1)/(r + 2). It follows from Equation (3.26) that an alternative description of the stochastic process of the successive outcomes of the trials can be described as follows: There is an urn which initially contains one white and one black ball. At each stage a ball is randomly drawn and is then replaced along with another ball of the same color. Thus, for instance, if of the first r balls drawn, k were white, then the urn at the time of the (r + 1)th draw would consist of k + 1 white and r − k + 1 black, and thus the next ball would be white with probability (k + 1)/(r + 2). If we identify the drawing of a white ball with a successful trial, then we see that this yields an alternate description of the original model. This latter urn model is called Polya’s urn model.
Remarks (i) In the special case when k = r, Equation (3.26) is sometimes called Laplace’s rule of succession, after the French mathematician Pierre de Laplace. In Laplace’s era, this “rule” provoked much controversy, for people attempted to employ it in diverse situations where its validity was questionable. For instance, it was used to justify such propositions as “If you have dined twice at a restaurant and both meals were good, then the next meal also will be good with probability 34 ,” and “Since the sun has risen the past 1,826,213 days, so will it rise tomorrow with probability 1,826,214/1,826,215.” The trouble with such claims resides in the fact that it is not at all clear the situation they are describing can be modeled as consisting of independent trials having a common probability of success which is itself uniformly chosen.
3.6.
Some Applications
149
(ii) In the original description of the experiment, we referred to the successive trials as being independent, and in fact they are independent when the success probability is known. However, when p is regarded as a random variable, the successive outcomes are no longer independent because knowing whether an outcome is a success or not gives us some information about p, which in turn yields information about the other outcomes. The preceding can be generalized to situations in which each trial has more than two possible outcomes. Suppose that n independent trials, each resulting in one of m possible outcomes 1, . . . , m, with respective probabilities p1 , . . . , pm are performed. If we let Xi denote the number of type i outcomes that result in the n trials, i = 1, . . . , m, then the vector X1 , . . . , Xm will have the multinomial distribution given by P {X1 = x1 , X2 = x2 , . . . , Xm = xm |p} =
n! xm p x1 p x2 · · · pm x1 ! · · · xm ! 1 2
where x1 , . . . , xm is any vector of nonnegative integers that sum to n. Now let us suppose that the vector p = (p1 , . . . , pm ) is not specified, but instead is chosen by a “uniform” distribution. Such a distribution would be of the form c, f (p1 , . . . , pm ) = 0,
0 pi 1, i = 1, . . . , m, otherwise
m 1
pi = 1
The preceding multivariate distribution is a special case of what is known as the Dirichlet distribution, and it is not difficult to show, using the fact that the distribution must integrate to 1, that c = (m − 1)!. The unconditional distribution of the vector X is given by P {X1 = x1 , . . . , Xm = xm } =
···
× f (p1 , . . . , pm ) dp1 · · · dpm =
P {X1 = x1 , . . . , Xm = xm |p1 , . . . , pm }
(m − 1)!n! x1 ! · · · xm !
xm dp1 · · · dpm · · · p1x1 · · · pm
0p 1 m i 1 pi =1
Now it can be shown that
x1 ! · · · xm ! xm · · · p1x1 · · · pm dp1 · · · dpm = m 1 xi + m − 1 !
0p 1 m i 1 pi =1
(3.27)
150
3
Conditional Probability and Conditional Expectation
and thus, using the fact that
m 1
xi = n, we see that
n!(m − 1)! (n + m − 1)! n + m − 1 −1 = (3.28) m−1 m − 1 m − 1 Hence, all of the n + possible outcomes [there are n + possible nonm−1 m−1 negative integer valued solutions of x1 + · · · + xm = n] of the vector (X1 , . . . , Xm ) are equally likely. The probability distribution given by Equation (3.28) is sometimes called the Bose–Einstein distribution. To obtain an alternative description of the foregoing, let us compute the conditional probability that the (n + 1)st outcome is of type j if the first n trials have resulted in xi type i outcomes, i = 1, . . . , m, m 1 xi = n. This is given by P {X1 = x1 , . . . , Xm = xm } =
P {(n + 1)st is j |xi type i in first n, i = 1, . . . , m} P {(n + 1)st is j, xi type i in first n, i = 1, . . . , m} P {xi type i in first n, i = 1, . . . , m} n!(m − 1)! x +1 xm dp1 · · · dpm · · · p1x1 · · · pj j · · · pm x1 ! · · · xm ! = −1 n+m−1 m−1 =
where the numerator is obtained by conditioning on the p vector and the denominator is obtained by using Equation (3.28). By Equation (3.27), we have that P {(n + 1)st is j |xi type i in first n, i = 1, . . . , m} (xj + 1)n!(m − 1)! (n + m)! = (m − 1)!n! (n + m − 1)! =
xj + 1 n+m
(3.29)
Using Equation (3.29), we can now present an urn model description of the stochastic process of successive outcomes. Namely, consider an urn that initially contains one of each of m types of balls. Balls are then randomly drawn and are replaced along with another of the same type. Hence, if in the first n drawings there have been a total of xj type j balls drawn, then the urn immediately before
3.6.
Some Applications
151
the (n+1)st draw will contain xj +1 type j balls out of a total of m+n, and so the probability of a type j on the (n + 1)st draw will be given by Equation (3.29).
Remark Consider a situation where n particles are to be distributed at random among m possible regions; and suppose that the regions appear, at least before the experiment, to have the same physical characteristics. It would thus seem that the most likely distribution for the number of particles that fall into each of the regions is the multinomial distribution with pi ≡ 1/m. (This, of course, would correspond to each particle, independent of the others, being equally likely to fall in any of the m regions.) Physicists studying how particles distribute themselves observed the behavior of such particles as photons and atoms containing an even number of elementary particles. However, when they studied the resulting data, they were amazed to discover that the observed frequencies did not follow the multinomial distribution but rather seemed to follow the Bose–Einstein distribution. They were amazed because they could not imagine a physical model for the distribution of particles which would result in all possible outcomes being equally likely. (For instance, if 10 particles are to distribute themselves between two regions, it hardly seems reasonable that it is just as likely that both regions will contain 5 particles as it is that all 10 will fall in region 1 or that all 10 will fall in region 2.) However, from the results of this section we now have a better understanding of the cause of the physicists’ dilemma. In fact, two possible hypotheses present themselves. First, it may be that the data gathered by the physicists were actually obtained under a variety of different situations, each having its own characteristic p vector which gave rise to a uniform spread over all possible p vectors. A second possibility (suggested by the urn model interpretation) is that the particles select their regions sequentially and a given particle’s probability of falling in a region is roughly proportional to the fraction of the landed particles that are in that region. (In other words, the particles presently in a region provide an “attractive” force on elements that have not yet landed.) 3.6.4. Mean Time for Patterns Let X = (X1 , X2 , . . .) be a sequence of independent and identically distributed discrete random variables such that pi = P {Xj = i} For a given subsequence, or pattern, i1 , . . . , in let T = T (i1 , . . . , in ) denote the number of random variables that we need to observe until the pattern appears. For instance, if the subsequence of interest is 3, 5, 1 and the sequence is X = (5, 3, 1, 3, 5, 3, 5, 1, 6, 2, . . .) then T = 8. We want to determine E[T ]. To begin, let us consider whether the pattern has an overlap, where we say that the pattern i1 , i2 , . . . , in has an overlap if for some k, 1 k < n, the sequence of
152
3
Conditional Probability and Conditional Expectation
its final k elements is the same as that of its first k elements. That is, it has an overlap if for some 1 k < n, (in−k+1 , . . . , in ) = (i1 , . . . , ik ),
k j, (Xj +1 , . . . , Xj +n ) = (i1 , . . . , in )}
(3.30)
To verify (3.30), note first that T = j + n clearly implies both that T > j and that (Xj +1 , . . . , Xj +n ) = (i1 , . . . , in ). On the other hand, suppose that T >j
and
(Xj +1 , . . . , Xj +n ) = (i1 , . . . , in )
(3.31)
Let k < n. Because (i1 , . . . , ik ) = (in−k+1 , . . . , in ), it follows that T = j + k. But (3.31) implies that T j + n, so we can conclude that T = j + n. Thus we have verified (3.30). Using (3.30), we see that P {T = j + n} = P {T > j, (Xj +1 , . . . , Xj +n ) = (i1 , . . . , in )} However, whether T > j is determined by the values X1 , . . . , Xj , and is thus independent of Xj +1 , . . . , Xj +n . Consequently, P {T = j + n} = P {T > j }P {(Xj +1 , . . . , Xj +n ) = (i1 , . . . , in )} = P {T > j }p where p = pi 1 pi 2 · · · pi n Summing both sides of the preceding over all j yields 1=
∞
P {T = j + n} = p
j =0
∞
P {T > j } = pE[T ]
j =0
or E[T ] =
1 p
Case 2: The pattern has overlaps. For patterns having overlaps there is a simple trick that will enable us to obtain E[T ] by making use of the result for nonoverlapping patterns. To make the analy-
3.6.
Some Applications
153
sis more transparent, consider a specific pattern, say P = (3, 5, 1, 3, 5). Let x be a value that does not appear in the pattern, and let Tx denote the time until the pattern Px = (3, 5, 1, 3, 5, x) appears. That is, Tx is the time of occurrence of the new pattern that puts x at the end of the original pattern. Because x did not appear in the original pattern it follows that the new pattern has no overlaps; thus, E[Tx ] =
1 px p
1 where p = nj=1 pij = p32 p52 p1 . Because the new pattern can occur only after the original one, write Tx = T + A where T is the time at which the pattern P = (3, 5, 1, 3, 5) occurs, and A is the additional time after the occurrence of the pattern P until Px occurs. Also, let E[Tx |i1 , . . . ir ] denote the expected additional time after time r until the pattern Px appears given that the first r data values are i1 , . . . , ir . Conditioning on X, the next data value after the occurrence of the pattern (3, 5, 1, 3, 5), gives that ⎧ 1 + E[Tx |3, 5, 1], if i = 1 ⎪ ⎪ ⎨ if i = 3 1 + E[Tx |3], E[A|X = i] = 1, if i = x ⎪ ⎪ ⎩ if i = 1, 3, x 1 + E[Tx ], Therefore, E[Tx ] = E[T ] + E[A] = E[T ] + 1 + E[Tx |3, 5, 1]p1 + E[Tx |3]p3 + E[Tx ](1 − p1 − p3 − px ) (3.32) But E[Tx ] = E[T (3, 5, 1)] + E[Tx |3, 5, 1] giving that E[Tx |3, 5, 1] = E[Tx ] − E[T (3, 5, 1)] Similarly, E[Tx |3] = E[Tx ] − E[T (3)] Substituting back into Equation (3.32) gives px E[Tx ] = E[T ] + 1 − p1 E[T (3, 5, 1)] − p3 E[T (3)]
154
3
Conditional Probability and Conditional Expectation
But, by the result in the nonoverlapping case, E[T (3, 5, 1)] =
1 , p3 p5 p1
E[T (3)] =
1 p3
yielding the result E[T ] = px E[Tx ] +
1 1 1 = + p3 p5 p p3 p5
For another illustration of the technique, let us reconsider Example 3.14, which is concerned with finding the expected time until n consecutive successes occur in independent Bernoulli trials. That is, we want E[T ], when the pattern is P = (1, 1, . . . , 1). Then, with x = 1 we consider the nonoverlapping pattern Px = (1, . . . , 1, x), and let Tx be its occurrence time. With A and X as previously defined, we have that ⎧ if i = 1 ⎨1 + E[A], if i = x E[A|X = i] = 1, ⎩ if i = 1, x 1 + E[Tx ], Therefore, E[A] = 1 + E[A]p1 + E[Tx ](1 − p1 − px ) or E[A] =
1 1 − p1 − px + E[Tx ] 1 − p1 1 − p1
Consequently, E[T ] = E[Tx ] − E[A] =
px E[Tx ] − 1 1 − p1
=
(1/p1 )n − 1 1 − p1
where the final equality used that E[Tx ] = pn1px . 1 The mean occurrence time of any overlapping pattern P = (i1 , . . . , in ) can be obtained by the preceding method. Namely, let Tx be the time until the nonoverlapping pattern Px = (i1 , . . . , in , x) occurs; then use the identity E[Tx ] = E[T ] + E[A]
3.6.
Some Applications
155
to relate E[T ] and E[Tx ] = p 1px ; then condition on the next data value after P occurs to obtain an expression for E[A] in terms of quantities of the form E[Tx |i1 , . . . , ir ] = E[Tx ] − E[T (i1 , . . . , ir )] If (i1 , . . . , ir ) is nonoverlapping, use the nonoverlapping result to obtain E[T (i1 , . . . , ir )]; otherwise, repeat the process on the subpattern (i1 , . . . , ir ).
Remark We can utilize the preceding technique even when the pattern i1 , . . . , in includes all the distinct data values. For instance, in coin tossing the pattern of interest might be h, t, h. Even in such cases, we should let x be a data value that is not in the pattern and use the preceding technique (even though px = 0). Because px will appear only in the final answer in the expression px E[Tx ] = ppxxp , by interpreting this fraction as 1/p we obtain the correct answer. (A rigorous approach, yielding the same result, would be to reduce one of the positive pi by , take px = , solve for E[T ], and then let go to 0.) 3.6.5. The k-Record Values of Discrete Random Variables Let X1 , X2 , . . . be independent and identically distributed random variables whose set of possible values is the positive integers, and let P {X = j }, j 1, denote their common probability mass function. Suppose that these random variables are observed in sequence, and say that Xn is a k-record value if Xi Xn
for exactly k of the values i, i = 1, . . . , n
That is, the nth value in the sequence is a k-record value if exactly k of the first n values (including Xn ) are at least as large as it. Let Rk denote the ordered set of k-record values. It is a rather surprising result that not only do the sequences of k-record values have the same probability distributions for all k, these sequences are also independent of each other. This result is known as Ignatov’s theorem.
Ignatov’s Theorem Rk , k 1, are independent and identically distributed random vectors.
Proof Define a series of subsequences of the data sequence X1 , X2 , . . . by letting the ith subsequence consist of all data values that are at least as large as i, i 1. For instance, if the data sequence is 2, 5, 1, 6, 9, 8, 3, 4, 1, 5, 7, 8, 2, 1, 3, 4, 2, 5, 6, 1, . . .
156
3
Conditional Probability and Conditional Expectation
then the subsequences are as follows: 1:
2, 5, 1, 6, 9, 8, 3, 4, 1, 5, 7, 8, 2, 1, 3, 4, 2, 5, 6, 1, . . .
2:
2, 5, 6, 9, 8, 3, 4, 5, 7, 8, 2, 3, 4, 2, 5, 6, . . .
3:
5, 6, 9, 8, 3, 4, 5, 7, 8, 3, 4, 5, 6, . . .
and so on. Let Xji be the j th element of subsequence i. That is, Xji is the j th data value that is at least as large as i. An important observation is that i is a k-record value if and only if Xki = i. That is, i will be a k-record value if and only if the kth value to be at least as large as i is equal to i. (For instance, for the preceding data, since the fifth value to be at least as large as 3 is equal to 3 it follows that 3 is a five-record value.) Now, it is not difficult to see that, independent of which values in the first subsequence are equal to 1, the values in the second subsequence are independent and identically distributed according to the mass function P {value in second subsequence = j } = P {X = j |X 2},
j 2
Similarly, independent of which values in the first subsequence are equal to 1 and which values in the second subsequence are equal to 2, the values in the third subsequence are independent and identically distributed according to the mass function P {value in third subsequence = j } = P {X = j |X 3},
j 3
and so on. It therefore follows that the events {Xji = i}, i 1, j 1, are independent and P {i is a k-record value} = P {Xki = i} = P {X = i|X i} It now follows from the independence of the events {Xki = i}, i 1, and the fact that P {i is a k-record value} does not depend on k, that Rk has the same distribution for all k 1. In addition, it follows from the independence of the events {Xki = 1}, that the random vectors Rk , k 1, are also independent. Suppose now that the Xi , i 1 are independent finite-valued random variables with probability mass function pi = P {X = i},
i = 1, . . . , m
and let T = min{n : Xi Xn for exactly k of the values i, i = 1, . . . , n} denote the first k-record index. We will now determine its mean.
3.6.
Proposition 3.3 Let λi = pi /
m
j =i
Some Applications
157
pj , i = 1, . . . , m. Then
E[T ] = k + (k − 1)
m−1
λi
i=1
Proof To begin, suppose that the observed random variables X1 X2 , . . . take on one of the values i, i + 1, . . . , m with respective probabilities P {X = j } =
pj , pi + · · · + pm
j = i, . . . , m
Let Ti denote the first k-record index when the observed data have the preceding mass function, and note that since the each data value is at least i it follows that the k-record value will equal i, and Ti will equal k, if Xk = i. As a result, E[Ti |Xk = i] = k On the other hand, if Xk > i then the k-record value will exceed i, and so all data values equal to i can be disregarded when searching for the k-record value. In addition, since each data value greater than i will have probability mass function P {X = j |X > i} =
pj , pi+1 + · · · + pm
j = i + 1, . . . , m
it follows that the total number of data values greater than i that need be observed until a k-record value appears has the same distribution as Ti+1 . Hence, E[Ti |Xk > i] = E[Ti+1 + Ni |Xk > i] where Ti+1 is the total number of variables greater than i that we need observe to obtain a k-record, and Ni is the number of values equal to i that are observed in that time. Now, given that Xk > i and that Ti+1 = n (n k) it follows that the time to observe Ti+1 values greater than i has the same distribution as the number of trials to obtain n successes given that trial k is a success and that each trial is independently a success with probability 1 − pi / j i pj = 1 − λi . Thus, since the number of trials needed to obtain a success is a geometric random variable with mean 1/(1 − λi ), we see that E[Ti |Ti+1 , Xk > i] = 1 +
Ti+1 − 1 Ti+1 − λi = 1 − λi 1 − λi
Taking expectations gives that . E[Ti |Xk > i] = E
/ Ti+1 − λi !! E[Ti+1 ] − λi X > i = ! k 1 − λi 1 − λi
158
3
Conditional Probability and Conditional Expectation
Thus, upon conditioning on whether Xk = i, we obtain E[Ti ] = E[Ti |Xk = i]λi + E[Ti |Xk > i](1 − λi ) = (k − 1)λi + E[Ti+1 ] Starting with E[Tm ] = k, the preceding gives that E[Tm−1 ] = (k − 1)λm−1 + k E[Tm−2 ] = (k − 1)λm−2 + (k − 1)λm−1 + k m−1
= (k − 1)
λj + k
j =m−2
E[Tm−3 ] = (k − 1)λm−3 + (k − 1)
m−1
λj + k
j =m−2 m−1
= (k − 1)
λj + k
j =m−3
In general, E[Ti ] = (k − 1)
m−1
λj + k
j =i
and the result follows since T = T1 .
3.7.
An Identity for Compound Random Variables
Let X1 , X2 , . . . be a sequence of independent and identically distributed random variables, and let Sn = ni=1 Xi be the sum of the first n of them, n 0, where S0 = 0. Recall that if N is a nonnegative integer valued random variable that is independent of the sequence X1 , X2 , . . . then SN =
N
Xi
i=1
is said to be a compound random variable, with the distribution of N called the compounding distribution. In this subsection we will first derive an identity involving such random variables. We will then specialize to where the Xi are positive integer valued random variables, prove a corollary of the identity, and then
3.7.
An Identity for Compound Random Variables
159
use this corollary to develop a recursive formula for the probability mass function of SN , for a variety of common compounding distributions. To begin, let M be a random variable that is independent of the sequence X1 , X2 , . . . , and which is such that P {M = n} =
nP {N = n} , E[N ]
n = 1, 2, . . .
Proposition 3.4 The Compound Random Variable Identity For any function h E[SN h(SN )] = E[N ]E[X1 h(SM )]
Proof N E[SN h(SN )] = E[ Xi h(SN )] i=1
=
∞ n=0
N E[ Xi h(SN )|N = n]P {N = n} i=1
(by conditioning on N ) ∞ n E[ Xi h(Sn )|N = n]P {N = n} = n=0
=
∞ n=0
i=1 n E[ Xi h(Sn )]P {N = n} i=1
(by independence of N and X1 , . . . , Xn ) ∞ n E[Xi h(Sn )]P {N = n} = n=0 i=1
Now, because X1 , . . . , Xn are independent and identically distributed, and h(Sn ) = h(X1 + · · · + Xn ) is a symmetric function of X1 , . . . , Xn , it follows that the distribution of Xi h(Sn ) is the same for all i = 1, . . . , n. Therefore, continuing the preceding string of equalities yields E[SN h(SN )] =
∞
nE[X1 h(Sn )]P {N = n}
n=0
= E[N ]
∞ n=0
E[X1 h(Sn )]P {M = n}
(definition of M)
160
3
Conditional Probability and Conditional Expectation
= E[N ]
∞
E[X1 h(Sn )|M = n]P {M = n}
n=0
(independence of M and X1 , . . . , Xn ) = E[N]
∞
E[X1 h(SM )|M = n]P {M = n}
n=0
= E[N ]E[X1 h(SM )] which proves the proposition.
Suppose now that the Xi are positive integer valued random variables, and let αj = P {X1 = j },
j >0
The successive values of P {SN = k} can often be obtained from the following corollary to Proposition 3.4.
Corollary 3.5 P {SN = 0} = P {N = 0} 1 j αj P {SM−1 = k − j }, P {SN = k} = E[N] k k
k>0
j =1
Proof For k fixed, let h(x) =
1, 0,
if x = k if x = k
and note that SN h(SN ) is either equal to k if SN = k or is equal to 0 otherwise. Therefore, E[SN h(SN )] = kP {SN = k} and the compound identity yields kP {SN = k} = E[N ]E[X1 h(SM )] = E[N ]
∞ j =1
E[X1 h(SM ))|X1 = j ]αj
3.7.
= E[N ]
An Identity for Compound Random Variables ∞
161
j E[h(SM )|X1 = j ]αj
j =1
= E[N]
∞
j P {SM = k|X1 = j }αj
(3.33)
j =1
Now, P {SM = k|X1 = j } = P
M
! ! Xi = k !X1 = j
i=1
=P j+
M
! ! X i = k !X 1 = j
i=2
=P j+
M
Xi = k
i=2
=P j+
M−1
Xi = k
i=1
= P {SM−1 = k − j } The next to last equality followed because X2 , . . . , XM and X1 , . . . , XM−1 have the same joint distribution; namely that of M − 1 independent random variables that all have the distribution of X1 , where M − 1 is independent of these random variables. Thus the proof follows from Equation (3.33). When the distributions of M − 1 and N are related, the preceding corollary can be a useful recursion for computing the probability mass function of SN , as is illustrated in the following subsections. 3.7.1. Poisson Compounding Distribution If N is the Poisson distribution with mean λ, then P {M − 1 = n} = P {M = n + 1} =
(n + 1)P {N = n + 1} E[N ]
162
3
Conditional Probability and Conditional Expectation
=
λn+1 1 (n + 1)e−λ λ (n + 1)!
= e−λ
λn n!
Consequently, M − 1 is also Poisson with mean λ. Thus, with Pn = P {SN = n} the recursion given by Corollary 3.5 can be written P0 = e−λ λ j αj Pk−j , k k
Pk =
k>0
j =1
Remark When the Xi are identically 1, the preceding recursion reduces to the well-known identity for a Poisson random variable having mean λ: P {N = 0} = e−λ λ P {N = n} = P {N = n − 1}, n
n1
Example 3.30 Let S be a compound Poisson random variable with λ = 4 and P {Xi = i} = 1/4,
i = 1, 2, 3, 4
Let us use the recursion given by Corollary 3.5 to determine P {S = 5}. It gives P0 = e−λ = e−4 P1 = λα1 P0 = e−4 λ 3 P2 = (α1 P1 + 2α2 P0 ) = e−4 2 2 λ 13 P3 = (α1 P2 + 2α2 P1 + 3α3 P0 ) = e−4 3 6 λ 73 P4 = (α1 P3 + 2α2 P2 + 3α3 P1 + 4α4 P0 ) = e−4 4 24 λ 501 −4 P5 = (α1 P4 + 2α2 P3 + 3α3 P2 + 4α4 P1 + 5α5 P0 ) = e 5 120
3.7.
An Identity for Compound Random Variables
163
3.7.2. Binomial Compounding Distribution Suppose that N is a binomial random variable with parameters r and p. Then, (n + 1)P {N = n + 1} E[N ] r n+1 p n+1 (1 − p)r−n−1 = rp n + 1
P {M − 1 = n} =
=
r! n+1 p n+1 (1 − p)r−1−n rp (r − 1 − n)!(n + 1)!
=
(r − 1)! p n (1 − p)r−1−n (r − 1 − n)!n!
Thus, M − 1 is a binomial random variable with parameters r − 1, p. Fixing p, let N (r) be a binomial random variable with parameters r and p, and let Pr (k) = P {SN (r) = k} Then, Corollary 3.5 yields that Pr (0) = (1 − p)r k rp Pr (k) = j αj Pr−1 (k − j ), k
k>0
j =1
For instance, letting k equal 1, then 2, and then 3 gives Pr (1) = rp α1 (1 − p)r−1 rp [α1 Pr−1 (1) + 2α2 Pr−1 (0)] Pr (2) = 2 # rp " (r − 1)pα12 (1 − p)r−2 + 2α2 (1 − p)r−1 = 2 # rp " α1 Pr−1 (2) + 2α2 Pr−1 (1) + 3α3 Pr−1 (0) Pr (3) = 3 # α1 rp (r − 1)p " (r − 2)pα12 (1 − p)r−3 + 2α2 (1 − p)r−2 = 3 2 2α2 rp + (r − 1)p α1 (1 − p)r−2 + α3 rp(1 − p)r−1 3
164
3
Conditional Probability and Conditional Expectation
3.7.3. A Compounding Distribution Related to the Negative Binomial Suppose, for a fixed value of p, 0 < p < 1, the compounding random variable N has a probability mass function n+r −1 r n = 0, 1, . . . P {N = n} = p (1 − p)n , r −1 Such a random variable can be thought of as being the number of failures that occur before a total of r successes have been amassed when each trial is independently a success with probability p. (There will be n such failures if the rth success occurs on trial n + r. Consequently, N + r is a negative binomial random variable with parameters r and p.) Using that the mean of the negative binomial random variable N + r is E[N + r] = r/p, we see that E[N ] = r 1−p p . Regard p as fixed, and call N an NB(r) random variable. The random variable M − 1 has probability mass function (n + 1)P {N = n + 1} E[N ] (n + 1)p n + r r p (1 − p)n+1 = r(1 − p) r − 1 (n + r)! r+1 p (1 − p)n = r!n! n + r r+1 = p (1 − p)n r
P {M − 1 = n} =
In other words, M − 1 is an NB(r + 1) random variable. Letting, for an NB(r) random variable N , Pr (k) = P {SN = k} Corollary 3.5 yields that Pr (0) = p r Pr (k) =
k r(1 − p) j αj Pr+1 (k − j ), kp j =1
Thus, r(1 − p) α1 Pr+1 (0) p = rp r (1 − p)α1 ,
Pr (1) =
k>0
Exercises
165
# r(1 − p) " α1 Pr+1 (1) + 2α2 Pr+1 (0) 2p # r(1 − p) " 2 α1 (r + 1)p r+1 (1 − p) + 2α2 p r+1 = 2p # r(1 − p) " α1 Pr+1 (2) + 2α2 Pr+1 (1) + 3α3 Pr+1 (0) Pr (3) = 3p Pr (2) =
and so on.
Exercises 1. If X and Y are both discrete, show that pY (y) > 0.
x
pX|Y (x|y) = 1 for all y such that
*2. Let X1 and X2 be independent geometric random variables having the same parameter p. Guess the value of P {X1 = i|X1 + X2 = n} Hint: Suppose a coin having probability p of coming up heads is continually flipped. If the second head occurs on flip number n, what is the conditional probability that the first head was on flip number i, i = 1, . . . , n − 1? Verify your guess analytically. 3.
The joint probability mass function of X and Y , p(x, y), is given by p(1, 1) = 19 ,
p(2, 1) = 13 ,
p(3, 1) = 19 ,
p(1, 2) = 19 ,
p(2, 2) = 0,
p(3, 2) =
p(1, 3) = 0,
p(2, 3) = 16 ,
p(3, 3) =
1 18 , 1 9
Compute E[X|Y = i] for i = 1, 2, 3. 4.
In Exercise 3, are the random variables X and Y independent?
5. An urn contains three white, six red, and five black balls. Six of these balls are randomly selected from the urn. Let X and Y denote respectively the number of white and black balls selected. Compute the conditional probability mass function of X given that Y = 3. Also compute E[X|Y = 1]. *6. Repeat Exercise 5 but under the assumption that when a ball is selected its color is noted, and it is then replaced in the urn before the next selection is made.
166
3
Conditional Probability and Conditional Expectation
7. Suppose p(x, y, z), the joint probability mass function of the random variables X, Y , and Z, is given by p(1, 1, 1) = 18 ,
p(2, 1, 1) = 14 ,
p(1, 1, 2) = 18 ,
p(2, 1, 2) =
p(1, 2, 1) =
p(2, 2, 1) = 0,
1 16 ,
p(1, 2, 2) = 0,
p(2, 2, 2) =
3 16 ,
1 4
What is E[X|Y = 2]? What is E[X|Y = 2, Z = 1]? 8. An unbiased die is successively rolled. Let X and Y denote, respectively, the number of rolls necessary to obtain a six and a five. Find (a) E[X], (b) E[X|Y = 1], (c) E[X|Y = 5]. 9. Show in the discrete case that if X and Y are independent, then E[X|Y = y] = E[X]
for all y
10. Suppose X and Y are independent continuous random variables. Show that E[X|Y = y] = E[X]
for all y
11. The joint density of X and Y is f (x, y) =
(y 2 − x 2 ) −y e , 8
0 < y < ∞,
−y x y
Show that E[X|Y = y] = 0. 12. The joint density of X and Y is given by f (x, y) =
e−x/y e−y , y
0 < x < ∞,
0 0. The mass function of Xi is p(x) = θ x (1 − θ )1−x , x = 0, 1, 0 < θ < 1. The Xi are Poisson random variables with mean θ .
*19. Prove that if X and Y are jointly continuous, then ∞ E[X] = E[X|Y = y]fY (y) dy −∞
20. An individual whose level of exposure to a certain pathogen is x will contract the disease caused by this pathogen with probability P (x). If the exposure level of a randomly chosen member of the population has probability density function f , determine the conditional probability density of the exposure level of that member given that he or she (a) has the disease. (b) does not have the disease. (c) Show that when P (x) increases in x, then the ratio of the density of part (a) to that of part (b) also increases in x. 21. Consider Example 3.12 which refers to a miner trapped in a mine. Let N denote the total number of doors selected before the miner reaches safety. Also, let Ti denote the travel time corresponding to the ith choice, i 1. Again let X denote the time when the miner reaches safety. (a) (b) (c) (d) (e)
Give an identity that relates X to N and the Ti . What is E[N ]? What is E[T N ]? What is E[ N i=1 Ti |N = n]? Using the preceding, what is E[X]?
22. Suppose that independent trials, each of which is equally likely to have any of m possible outcomes, are performed until the same outcome occurs k consecutive times. If N denotes the number of trials, show that E[N] =
mk − 1 m−1
Some people believe that the successive digits in the expansion of π = 3.14159 . . . are “uniformly” distributed. That is, they believe that these digits have all the appearance of being independent choices from a distribution that is equally likely to be any of the digits from 0 through 9. Possible evidence against this hypothesis is the fact that starting with the 24,658,601st digit there is a run of nine successive 7s. Is this information consistent with the hypothesis of a uniform distribution? To answer this, we note from the preceding that if the uniform hypothesis were correct, then the expected number of digits until a run of nine of the same
Exercises
169
value occurs is (109 − 1)/9 = 111,111,111 Thus, the actual value of approximately 25 million is roughly 22 percent of the theoretical mean. However, it can be shown that under the uniformity assumption the standard deviation of N will be approximately equal to the mean. As a result, the observed value is approximately 0.78 standard deviations less than its theoretical mean and is thus quite consistent with the uniformity assumption. *23. A coin having probability p of coming up heads is successively flipped until two of the most recent three flips are heads. Let N denote the number of flips. (Note that if the first two flips are heads, then N = 2.) Find E[N ]. 24. A coin, having probability p of landing heads, is continually flipped until at least one head and one tail have been flipped. (a) Find the expected number of flips needed. (b) Find the expected number of flips that land on heads. (c) Find the expected number of flips that land on tails. (d) Repeat part (a) in the case where flipping is continued until a total of at least two heads and one tail have been flipped. 25. A gambler wins each game with probability p. In each of the following cases, determine the expected total number of wins. (a) The gambler will play n games; if he wins X of these games, then he will play an additional X games before stopping. (b) The gambler will play until he wins; if it takes him Y games to get this win, then he will play an additional Y games. 26. You have two opponents with whom you alternate play. Whenever you play A, you win with probability pA ; whenever you play B, you win with probability pB , where pB > pA . If your objective is to minimize the number of games you need to play to win two in a row, should you start with A or with B? Hint: Let E[Ni ] denote the mean number of games needed if you initially play i. Derive an expression for E[NA ] that involves E[NB ]; write down the equivalent expression for E[NB ] and then subtract. 27. A coin that comes up heads with probability p is continually flipped until the pattern T, T, H appears. (That is, you stop flipping when the most recent flip lands heads, and the two immediately preceding it lands tails.) Let X denote the number of flips made, and find E[X]. 28. Polya’s urn model supposes that an urn initially contains r red and b blue balls. At each stage a ball is randomly selected from the urn and is then returned along with m other balls of the same color. Let Xk be the number of red balls drawn in the first k selections.
170
3
Conditional Probability and Conditional Expectation
(a) Find E[X1 ]. (b) Find E[X2 ]. (c) Find E[X3 ]. (d) Conjecture the value of E[Xk ], and then verify your conjecture by a conditioning argument. (e) Give an intuitive proof for your conjecture. Hint: Number the initial r red and b blue balls, so the urn contains one type i red ball, for each i = 1, . . . , r; as well as one type j blue ball, for each j = 1, . . . , b. Now suppose that whenever a red ball is chosen it is returned along with m others of the same type, and similarly whenever a blue ball is chosen it is returned along with m others of the same type. Now, use a symmetry argument to determine the probability that any given selection is red. 29. Two players take turns shooting at a target, with each shot by player i hitting the target with probability pi , i = 1, 2. Shooting ends when two consecutive shots hit the target. Let μi denote the mean number of shots taken when player i shoots first, i = 1, 2. (a) Find μ1 and μ2 . (b) Let hi denote the mean number of times that the target is hit when player i shoots first, i = 1, 2. Find h1 and h2 . 30. Let Xi , i 0 be independent and identically distributed random variables with probability mass function p(j ) = P {Xi = j },
j = 1, . . . , m,
m
P (j ) = 1
j =1
Find E[N], where N = min{n > 0 : Xn = X0 }. 31. Each element in a sequence of binary data is either 1 with probability p or 0 with probability 1 − p. A maximal subsequence of consecutive values having identical outcomes is called a run. For instance, if the outcome sequence is 1, 1, 0, 1, 1, 1, 0, the first run is of length 2, the second is of length 1, and the third is of length 3. (a) Find the expected length of the first run. (b) Find the expected length of the second run. 32. Independent trials, each resulting in success with probability p, are performed. (a) Find the expected number of trials needed for there to have been both at least n successes and at least m failures.
Exercises
171
Hint: Is it useful to know the result of the first n + m trials? (b) Find the expected number of trials needed for there to have been either at least n successes and at least m failures. Hint: Make use of the result from part (a). i−1 R , 33. If Ri denotes the random amount that is earned in period i, then ∞ i i=1 β where 0 < β < 1 is a specified constant, is called the total discounted reward with discount factor β. Let T be a geometric random variable with parameter 1 − β that is independent of the Ri . Show that the expected total discounted reward is equal to the expected total (undiscounted) reward earned by time T . That is, show that ,∞ , T i−1 E β Ri = E Ri i=1
i=1
34. A set of n dice is thrown. All those that land on six are put aside, and the others are again thrown. This is repeated until all the dice have landed on six. Let N denote the number of throws needed. (For instance, suppose that n = 3 and that on the initial throw exactly two of the dice land on six. Then the other die will be thrown, and if it lands on six, then N = 2.) Let mn = E[N ]. (a) Derive a recursive formula for mn and use it to calculate mi , i = 2, 3, 4 and to show that m5 ≈ 13.024. (b) Let Xi denote the number of dice rolled on the ith throw. Find E[ N i=1 Xi ]. 35. Consider n multinomial trials, where each trial independently results in outcome i with probability pi , ki=1 pi = 1. With Xi equal to the number of trials that result in outcome i, find E[X1 |X2 > 0]. 36. Let p0 = P {X = 0} and suppose that 0 < p0 < 1. Let μ = E[X] and σ 2 = Var(X). Find (a) E[X|X = 0] and (b) Var(X|X = 0). 37. A manuscript is sent to a typing firm consisting of typists A, B, and C. If it is typed by A, then the number of errors made is a Poisson random variable with mean 2.6; if typed by B, then the number of errors is a Poisson random variable with mean 3; and if typed by C, then it is a Poisson random variable with mean 3.4. Let X denote the number of errors in the typed manuscript. Assume that each typist is equally likely to do the work. (a) Find E[X]. (b) Find Var(X). 38. Let U be a uniform (0, 1) random variable. Suppose that n trials are to be performed and that conditional on U = u these trials will be independent with a
172
3
Conditional Probability and Conditional Expectation
common success probability u. Compute the mean and variance of the number of successes that occur in these trials. 39. A deck of n cards, numbered 1 through n, is randomly shuffled so that all n! possible permutations are equally likely. The cards are then turned over one at a time until card number 1 appears. These upturned cards constitute the first cycle. We now determine (by looking at the upturned cards) the lowest numbered card that has not yet appeared, and we continue to turn the cards face up until that card appears. This new set of cards represents the second cycle. We again determine the lowest numbered of the remaining cards and turn the cards until it appears, and so on until all cards have been turned over. Let mn denote the mean number of cycles. (a) Derive a recursive formula for mn in terms of mk , k = 1, . . . , n − 1. (b) Starting with m0 = 0, use the recursion to find m1 , m2 , m3 , and m4 . (c) Conjecture a general formula for mn . (d) Prove your formula by induction on n. That is, show it is valid for n = 1, then assume it is true for any of the values 1, . . . , n − 1 and show that this implies it is true for n. (e) Let Xi equal 1 if one of the cycles ends with card i, and let it equal 0 otherwise, i =1, . . . , n. Express the number of cycles in terms of these Xi . (f) Use the representation in part (e) to determine mn . (g) Are the random variables X1 , . . . , Xn independent? Explain. (h) Find the variance of the number of cycles. 40. A prisoner is trapped in a cell containing three doors. The first door leads to a tunnel that returns him to his cell after two days of travel. The second leads to a tunnel that returns him to his cell after three days of travel. The third door leads immediately to freedom. (a) Assuming that the prisoner will always select doors 1, 2, and 3 with probabilities 0.5, 0.3, 0.2, what is the expected number of days until he reaches freedom? (b) Assuming that the prisoner is always equally likely to choose among those doors that he has not used, what is the expected number of days until he reaches freedom? (In this version, for instance, if the prisoner initially tries door 1, then when he returns to the cell, he will now select only from doors 2 and 3.) (c) For parts (a) and (b) find the variance of the number of days until the prisoner reaches freedom. 41. A rat is trapped in a maze. Initially it has to choose one of two directions. If it goes to the right, then it will wander around in the maze for three minutes and will then return to its initial position. If it goes to the left, then with probability 13 it will depart the maze after two minutes of traveling, and with probability 23 it
Exercises
173
will return to its initial position after five minutes of traveling. Assuming that the rat is at all times equally likely to go to the left or the right, what is the expected number of minutes that it will be trapped in the maze? 42. A total of 11 people, including you, are invited to a party. The times at which people arrive at the party are independent uniform (0, 1) random variables. (a) Find the expected number of people who arrive before you. (b) Find the variance of the number of people who arrive before you. 43. The number of claims received at an insurance company during a week is a random variable with mean μ1 and variance σ12 . The amount paid in each claim is a random variable with mean μ2 and variance σ22 . Find the mean and variance of the amount of money paid by the insurance company each week. What independence assumptions are you making? Are these assumptions reasonable? 44. The number of customers entering a store on a given day is Poisson distributed with mean λ = 10. The amount of money spent by a customer is uniformly distributed over (0, 100). Find the mean and variance of the amount of money that the store takes in on a given day. 45. An individual traveling on the real line is trying to reach the origin. However, the larger the desired step, the greater is the variance in the result of that step. Specifically, whenever the person is at location x, he next moves to a location having mean 0 and variance βx 2 . Let Xn denote the position of the individual after having taken n steps. Supposing that X0 = x0 , find (a) E[Xn ]; (b) Var(Xn ). 46. (a) Show that Cov(X, Y ) = Cov(X, E[Y | X]) (b) Suppose, that, for constants a and b, E[Y | X] = a + bX Show that b = Cov(X, Y )/Var(X) *47. If E[Y | X] = 1, show that Var(X Y ) Var(X)
174
3
Conditional Probability and Conditional Expectation
48. Give another proof of the result of Example 3.17 by computing the moment generating function of N i=1 Xi and then differentiating to obtain its moments. Hint: Let ,
φ(t) = E exp t , ,
N
Xi
i=1
= E E exp t
N i=1
! -! ! X i !N !
Now, ,
E exp t
N i=1
, n ! ! ! Xi !N = n = E exp t Xi = (φX (t))n ! i=1
since N is independent of the Xs where φX (t) = E[etX ] is the moment generating function for the Xs. Therefore, " # φ(t) = E (φX (t))N Differentiation yields " # φ (t) = E N (φX (t))N −1 φX (t) , " # φ (t) = E N (N − 1)(φX (t))N −2 (φX (t))2 + N (φX (t))N −1 φX (t) Evaluate at t = 0 to get the desired result. 49. A and B play a series of games with A winning each game with probability p. The overall winner is the first player to have won two more games than the other. (a) Find the probability that A is the overall winner. (b) Find the expected number of games played. 50. There are three coins in a barrel. These coins, when flipped, will come up heads with respective probabilities 0.3, 0.5, 0.7. A coin is randomly selected from among these three and is then flipped ten times. Let N be the number of heads obtained on the ten flips. Find (a) P {N = 0}. (b) P {N = n}, n = 0, 1, . . . , 10. (c) Does N have a binomial distribution? (d) If you win $1 each time a head appears and you lose $1 each time a tail appears, is this a fair game? Explain.
Exercises
175
51. Do Exercise 50 under the assumption that each time a coin is flipped, it is then put back in the barrel and another coin is randomly selected. Does N have a binomial distribution now? 52. Suppose that X and Y are independent random variables with probability density functions fX and fY . Determine a one-dimensional integral expression for P {X + Y < x}. *53. Suppose X is a Poisson random variable with mean λ. The parameter λ is itself a random variable whose distribution is exponential with mean 1. Show that P {X = n} = ( 12 )n+1 . 54. A coin is randomly selected from a group of ten coins, the nth coin having a probability n/10 of coming up heads. The coin is then repeatedly flipped until a head appears. Let N denote the number of flips necessary. What is the probability distribution of N ? Is N a geometric random variable? When would N be a geometric random variable; that is, what would have to be done differently? 55. Suppose in Exercise 42 that, aside from yourself, the number of other people who are invited is a Poisson random variable with mean 10. (a) Find the expected number of people who arrive before you. (b) Find the probability that you are the nth person to arrive. 56. Data indicate that the number of traffic accidents in Berkeley on a rainy day is a Poisson random variable with mean 9, whereas on a dry day it is a Poisson random variable with mean 3. Let X denote the number of traffic accidents tomorrow. If it will rain tomorrow with probability 0.6, find (a) E[X]; (b) P {X = 0}; (c) Var(X). 57. The number of storms in the upcoming rainy season is Poisson distributed but with a parameter value that is uniformly distributed over (0, 5). That is, is uniformly distributed over (0, 5), and given that = λ, the number of storms is Poisson with mean λ. Find the probability there are at least three storms this season. 58. A collection of n coins is flipped. The outcomes are independent, and the ith coin comes up heads with probability αi , i = 1, . . . , n. Suppose that for some value of j, 1 j n, αj = 12 . Find the probability that the total number of heads to appear on the n coins is an even number. 59. Let A and B be mutually exclusive events of an experiment. If independent replications of the experiment are continually performed, what is the probability that A occurs before B? *60. Two players alternate flipping a coin that comes up heads with probability p. The first one to obtain a head is declared the winner. We are interested in
176
3
Conditional Probability and Conditional Expectation
the probability that the first player to flip is the winner. Before determining this probability, which we will call f (p), answer the following questions. (a) Do you think that f (p) is a monotone function of p? If so, is it increasing or decreasing? (b) What do you think is the value of limp→1 f (p)? (c) What do you think is the value of limp→0 f (p)? (d) Find f (p). 61. Suppose in Exercise 29 that the shooting ends when the target has been hit twice. Let mi denote the mean number of shots needed for the first hit when player i shoots first, i = 1, 2. Also, let Pi , i = 1, 2, denote the probability that the first hit is by player 1, when player i shoots first. (a) Find m1 and m2 . (b) Find P1 and P2 . For the remainder of the problem, assume that player 1 shoots first. (c) (d) (e) (f)
Find the probability that the final hit was by 1. Find the probability that both hits were by 1. Find the probability that both hits were by 2. Find the mean number of shots taken.
62. A, B, and C are evenly matched tennis players. Initially A and B play a set, and the winner then plays C. This continues, with the winner always playing the waiting player, until one of the players has won two sets in a row. That player is then declared the overall winner. Find the probability that A is the overall winner. 63. Suppose there are n types of coupons, and that the type of each new coupon obtained is independent of past selections and is equally likely to be any of the n types. Suppose one continues collecting until a complete set of at least one of each type is obtained. (a) Find the probability that there is exactly one type i coupon in the final collection. Hint: Condition on T , the number of types that are collected before the first type i appears. (b) Find the expected number of types that appear exactly once in the final collection. 64. A and B roll a pair of dice in turn, with A rolling first. A’s objective is to obtain a sum of 6, and B’s is to obtain a sum of 7. The game ends when either player reaches his or her objective, and that player is declared the winner. (a) Find the probability that A is the winner.
Exercises
177
(b) Find the expected number of rolls of the dice. (c) Find the variance of the number of rolls of the dice. 65. The number of red balls in an urn that contains n balls is a random variable that is equally likely to be any of the values 0, 1, . . . , n. That is, P {i red, n − i non-red} =
1 , n+1
i = 0, . . . , n
The n balls are then randomly removed one at a time. Let Yk denote the number of red balls in the first k selections, k = 1, . . . , n. (a) Find P {Yn = j }, j = 0, . . . , n. (b) Find P {Yn−1 = j }, j = 0, . . . , n. (c) What do you think is the value of P {Yk = j }, j = 0, . . . , n? (d) Verify your answer to part (c) by a backwards induction argument. That is, check that your answer is correct when k = n, and then show that whenever it is true for k it is also true for k − 1, k = 1, . . . , n. 66. The opponents of soccer team A are of two types: either they are a class 1 or a class 2 team. The number of goals team A scores against a class i opponent is a Poisson random variable with mean λi , where λ1 = 2, λ2 = 3. This weekend the team has two games against teams they are not very familiar with. Assuming that the first team they play is a class 1 team with probability 0.6 and the second is, independently of the class of the first team, a class 1 team with probability 0.3, determine (a) the expected number of goals team A will score this weekend. (b) the probability that team A will score a total of five goals. *67. A coin having probability p of coming up heads is continually flipped. Let Pj (n) denote the probability that a run of j successive heads occurs within the first n flips. (a) Argue that Pj (n) = Pj (n − 1) + p j (1 − p)[1 − Pj (n − j − 1)] (b) By conditioning on the first non-head to appear, derive another equation relating Pj (n) to the quantities Pj (n − k), k = 1, . . . , j . 68. In a knockout tennis tournament of 2n contestants, the players are paired and play a match. The losers depart, the remaining 2n−1 players are paired, and they play a match. This continues for n rounds, after which a single player remains unbeaten and is declared the winner. Suppose that the contestants are numbered 1 through 2n , and that whenever two players contest a match, the lower numbered one wins with probability p. Also suppose that the pairings of the remaining players are always done at random so that all possible pairings for that round are equally likely.
178
3
Conditional Probability and Conditional Expectation
(a) What is the probability that player 1 wins the tournament? (b) What is the probability that player 2 wins the tournament? Hint: Imagine that the random pairings are done in advance of the tournament. That is, the first-round pairings are randomly determined; the 2n−1 firstround pairs are then themselves randomly paired, with the winners of each pair to play in round 2; these 2n−2 groupings (of four players each) are then randomly paired, with the winners of each grouping to play in round 3, and so on. Say that players i and j are scheduled to meet in round k if, provided they both win their first k − 1 matches, they will meet in round k. Now condition on the round in which players 1 and 2 are scheduled to meet. 69. In the match problem, say that (i, j ), i < j , is a pair if i chooses j ’s hat and j chooses i’s hat. (a) Find the expected number of pairs. (b) Let Qn denote the probability that there are no pairs, and derive a recursive formula for Qn in terms of Qj , j < n. Hint: Use the cycle concept. (c) Use the recursion of part (b) to find Q8 . 70. Let N denote the number of cycles that result in the match problem. (a) Let Mn = E[N ], and derive an equation for Mn in terms of M1 , . . . , Mn−1 . (b) Let Cj denote the size of the cycle that contains person j . Argue that N=
n
1/Cj
j =1
and use the preceding to determine E[N ]. (c) Find the probability that persons 1, 2, . . . , k are all in the same cycle. (d) Find the probability that 1, 2, . . . , k is a cycle. 71. Use the equation following (3.14) to obtain Equation (3.10). Hint: First multiply both sides of Equation (3.14) by n, then write a new equation by replacing n with n − 1, and then subtract the former from the latter. 72. In Example 3.25 show that the conditional distribution of N given that U1 = y is the same as the conditional distribution of M given that U1 = 1 − y. Also, show that E[N |U1 = y] = E[M|U1 = 1 − y] = 1 + ey
Exercises
179
Figure 3.7.
*73. Suppose that we continually roll a die until the sum of all throws exceeds 100. What is the most likely value of this total when you stop? 74. There are five components. The components act independently, with component i working with probability pi , i = 1, 2, 3, 4, 5. These components form a system as shown in Figure 3.7. The system is said to work if a signal originating at the left end of the diagram can reach the right end, where it can pass through a component only if that component is working. (For instance, if components 1 and 4 both work, then the system also works.) What is the probability that the system works? 75. This problem will present another proof of the ballot problem of Example 3.24. (a) Argue that Pn,m = 1 − P {A and B are tied at some point} (b) Explain why P {A receives first vote and they are eventually tied} = P {B receives first vote and they are eventually tied} Hint: Any outcome in which they are eventually tied with A receiving the first vote corresponds to an outcome in which they are eventually tied with B receiving the first vote. Explain this correspondence. (c) Argue that P {eventually tied} = 2m/(n + m), and conclude that Pn,m = (n − m)/(n + m). 76. Consider a gambler who on each bet either wins 1 with probability 18/38 or loses 1 with probability 20/38. (These are the probabilities if the bet is that a roulette wheel will land on a specified color.) The gambler will quit either when he or she is winning a total of 5 or after 100 plays. What is the probability he or she plays exactly 15 times?
180
3
Conditional Probability and Conditional Expectation
77. Show that (a) E[XY |Y = y] = yE[X|Y = y] (b) E[g(X, Y )|Y = y] = E[g(X, y)|Y = y] (c) E[XY ] = E[Y E[X|Y ]] 78. In the ballot problem (Example 3.24), compute P {A is never behind}. 79. An urn contains n white and m black balls which are removed one at a time. If n > m, show that the probability that there are always more white than black balls in the urn (until, of course, the urn is empty) equals (n − m)/(n + m). Explain why this probability is equal to the probability that the set of withdrawn balls always contains more white than black balls. [This latter probability is (n − m)/(n + m) by the ballot problem.] 80. A coin that comes up heads with probability p is flipped n consecutive times. What is the probability that starting with the first flip there are always more heads than tails that have appeared? 81. Let Xi , i 1, be independent uniform (0, 1) random variables, and define N by N = min{n: Xn < Xn−1 } where X0 = x. Let f (x) = E[N]. (a) (b) (c) (d)
Derive an integral equation for f (x) by conditioning on X1 . Differentiate both sides of the equation derived in part (a). Solve the resulting equation obtained in part (b). For a second approach to determining f (x) argue that P {N k} =
(1 − x)k−1 (k − 1)!
(e) Use part (d) to obtain f (x). 82. Let X1 , X2 , . . . be independent continuous random variables with a common distribution function F and density f = F , and for k 1 let Nk = min{n k: Xn = kth largest of X1 , . . . , Xn } (a) Show that P {Nk = n} = (b) Argue that
k−1 n(n−1) , n k.
fXNk (x) = f (x)(F¯ (x))k−1
∞ i +k−2 (F (x))i i i=0
Exercises
181
(c) Prove the following identity: a
1−k
∞ i +k−2 = (1 − a)i , i
0 < a < 1, k 2
i=0
Hint: Use induction. First prove it when k = 2, and then assume it for k. To prove it for k + 1, use the fact that ∞ i +k−1 i=1
i
(1 − a)i =
∞ i +k−2 i=1
i
(1 − a)i
∞ i +k−2 (1 − a)i + i −1 i=1
where the preceding used the combinatorial identity m m−1 m−1 = + i i i −1 Now, use the induction hypothesis to evaluate the first term on the right side of the preceding equation. (d) Conclude that XNk has distribution F . 83. An urn contains n balls, with ball i having weight wi , i = 1, . . . , n. The balls are withdrawn from the urn one at a time according to the following scheme: When S is the setof balls that remains, ball i, i ∈ S, is the next ball withdrawn with probability wi / j ∈S wj . Find the expected number of balls that are withdrawn before ball i, i = 1, . . . , n. 84. In the list example of Section 3.6.1 suppose that the initial ordering at time t = 0 is determined completely at random; that is, initially all n! permutations are equally likely. Following the front-of-the-line rule, compute the expected position of the element requested at time t. Hint: To compute P {ej precedes ei at time t} condition on whether or not either ei or ej has ever been requested prior to t. 85. In the list problem, when the Pi are known, show that the best ordering (best in the sense of minimizing the expected position of the element requested) is to place the elements in decreasing order of their probabilities. That is, if P1 > P2 > · · · > Pn , show that 1, 2, . . . , n is the best ordering. 86. Consider the random graph of Section 3.6.2 when n = 5. Compute the probability distribution of the number of components and verify your solution by using
182
3
Conditional Probability and Conditional Expectation
it to compute E[C]and then comparing your solution with E[C] =
5 5 (k − 1)!
k
k=1
5k
87. (a) From the results of Section 3.6.3 we can conclude that there are n+m−1 m−1 nonnegative integer valued solutions of the equation x1 +· · ·+xm = n. Prove this directly. (b) How many positive integer valued solutions of x1 + · · · + xm = n are there? Hint: Let yi = xi − 1. (c) For the Bose–Einstein distribution, compute the probability that exactly k of the Xi are equal to 0. 88. In Section 3.6.3, we saw that if U is a random variable that is uniform on (0, 1) and if, conditional on U = p, X is binomial with parameters n and p, then P {X = i} =
1 , n+1
i = 0, 1, . . . , n
For another way of showing this result, let U, X1 , X2 , . . . , Xn be independent uniform (0, 1) random variables. Define X by X = #i: Xi < U That is, if the n + 1 variables are ordered from smallest to largest, then U would be in position X + 1. (a) What is P {X = i}? (b) Explain how this proves the result of Exercise 88. 89. Let I1 , . . . , In be independent random variables, each of which is equally likely to be either 0 or 1. A well-known nonparametric statistical test (called the signed rank test) is concerned with determining Pn (k) defined by ⎧ ⎫ n ⎨ ⎬ j Ij k Pn (k) = P ⎩ ⎭ j =1
Justify the following formula: Pn (k) = 12 Pn−1 (k) + 12 Pn−1 (k − n) 90. The number of accidents in each period is a Poisson random variable with mean 5. With Xn , n 1, equal to the number of accidents in period n, find E[N ] when
Exercises
183
(a) N = min(n: Xn−2 = 2, Xn−1 = 1, Xn = 0); (b) N = min(n: Xn−3 = 2, Xn−2 = 1, Xn−1 = 0, Xn = 2). 91. Find the expected number of flips of a coin, which comes up heads with probability p, that are necessary to obtain the pattern h, t, h, h, t, h, t, h. 92. The number of coins that Josh spots when walking to work is a Poisson random variable with mean 6. Each coin is equally likely to be a penny, a nickel, a dime, or a quarter. Josh ignores the pennies but picks up the other coins. (a) Find the expected amount of money that Josh picks up on his way to work. (b) Find the variance of the amount of money that Josh picks up on his way to work. (c) Find the probability that Josh picks up exactly 25 cents on his way to work. *93. Consider a sequence of independent trials, each of which is equally likely to result in any of the outcomes 0, 1, . . . , m. Say that a round begins with the first trial, and that a new round begins each time outcome 0 occurs. Let N denote the number of trials that it takes until all of the outcomes 1, . . . , m − 1 have occurred in the same round. Also, let Tj denote the number of trials that it takes until j distinct outcomes have occurred, and let Ij denote the j th distinct outcome to occur. (Therefore, outcome Ij first occurs at trial Tj .) (a) Argue that the random vectors (I1 , . . . , Im ) and (T1 , . . . , Tm ) are independent. (b) Define X by letting X = j if outcome 0 is the j th distinct outcome to occur. (Thus, IX = 0.) Derive an equation for E[N ] in terms of E[Tj ], j = 1, . . . , m − 1 by conditioning on X. (c) Determine E[Tj ], j = 1, . . . , m − 1. Hint: See Exercise 42 of Chapter 2. (d) Find E[N ]. 94. Let N be a hypergeometric random variable having the distribution of the number of white balls in a random sample of size r from a set of w white and b blue balls. That is, w P {N = n} = m
b r−n w+b r
n
= 0 if either j < 0 or j > m. Now, consider N a compound random variable SN = i=1 Xi , where the Xi are positive integer valued random variables with αj = P {Xi = j }. where we use the convention that
j
184
3
Conditional Probability and Conditional Expectation
(a) With M as defined as in Section 3.7, find the distribution of M − 1. (b) Suppressing its dependence on b, let Pw,r (k) = P {SN = k}, and derive a recursion equation for Pw,r (k). (c) Use the recursion of (b) to find Pw,r (2).
Markov Chains
4 4.1.
Introduction
In this chapter, we consider a stochastic process {Xn , n = 0, 1, 2, . . .} that takes on a finite or countable number of possible values. Unless otherwise mentioned, this set of possible values of the process will be denoted by the set of nonnegative integers {0, 1, 2, . . .}. If Xn = i, then the process is said to be in state i at time n. We suppose that whenever the process is in state i, there is a fixed probability Pij that it will next be in state j . That is, we suppose that P {Xn+1 = j |Xn = i, Xn−1 = in−1 , . . . , X1 = i1 , X0 = i0 } = Pij
(4.1)
for all states i0 , i1 , . . . , in−1 , i, j and all n 0. Such a stochastic process is known as a Markov chain. Equation (4.1) may be interpreted as stating that, for a Markov chain, the conditional distribution of any future state Xn+1 given the past states X0 , X1 , . . . , Xn−1 and the present state Xn , is independent of the past states and depends only on the present state. The value Pij represents the probability that the process will, when in state i, next make a transition into state j . Since probabilities are nonnegative and since the process must make a transition into some state, we have that Pij 0,
i, j 0;
∞
Pij = 1,
i = 0, 1, . . .
j =0
Let P denote the matrix of one-step transition probabilities Pij , so that 3 3 3 P00 P01 P02 · · · 3 3 3 3 P10 P11 P12 · · · 3 3 3 3 .. 3 .. .. 3 . . . P=3 3 3 3 Pi0 Pi1 Pi2 · · · 3 3 3 3 . 3 .. .. 3 .. 3 . .
185
186
4
Markov Chains
Example 4.1 (Forecasting the Weather) Suppose that the chance of rain tomorrow depends on previous weather conditions only through whether or not it is raining today and not on past weather conditions. Suppose also that if it rains today, then it will rain tomorrow with probability α; and if it does not rain today, then it will rain tomorrow with probability β. If we say that the process is in state 0 when it rains and state 1 when it does not rain, then the preceding is a two-state Markov chain whose transition probabilities are given by 3 3α P=3 3β
3 1−α3 3 1−β3
Example 4.2 (A Communications System) Consider a communications system which transmits the digits 0 and 1. Each digit transmitted must pass through several stages, at each of which there is a probability p that the digit entered will be unchanged when it leaves. Letting Xn denote the digit entering the nth stage, then {Xn , n = 0, 1, . . .} is a two-state Markov chain having a transition probability matrix 3 3 3 p 1 −p3 3 P=3 31 − p p 3
Example 4.3 On any given day Gary is either cheerful (C), so-so (S), or glum (G). If he is cheerful today, then he will be C, S, or G tomorrow with respective probabilities 0.5, 0.4, 0.1. If he is feeling so-so today, then he will be C, S, or G tomorrow with probabilities 0.3, 0.4, 0.3. If he is glum today, then he will be C, S, or G tomorrow with probabilities 0.2, 0.3, 0.5. Letting Xn denote Gary’s mood on the nth day, then {Xn , n 0} is a three-state Markov chain (state 0 = C, state 1 = S, state 2 = G) with transition probability matrix 3 3 30.5 0.4 0.13 3 3 3 P=3 30.3 0.4 0.33 30.2 0.3 0.53
Example 4.4 (Transforming a Process into a Markov Chain) Suppose that whether or not it rains today depends on previous weather conditions through the last two days. Specifically, suppose that if it has rained for the past two days, then it will rain tomorrow with probability 0.7; if it rained today but not yesterday, then it will rain tomorrow with probability 0.5; if it rained yesterday but not today, then it will rain tomorrow with probability 0.4; if it has not rained in the past two days, then it will rain tomorrow with probability 0.2.
4.1.
Introduction
187
If we let the state at time n depend only on whether or not it is raining at time n, then the preceding model is not a Markov chain (why not?). However, we can transform this model into a Markov chain by saying that the state at any time is determined by the weather conditions during both that day and the previous day. In other words, we can say that the process is in state 0 state 1 state 2 state 3
if it rained both today and yesterday, if it rained today but not yesterday, if it rained yesterday but not today, if it did not rain either yesterday or today.
The preceding would then represent a four-state Markov chain having a transition probability matrix 3 3 30.7 0 0.3 0 3 3 3 30.5 0 0.5 0 3 3 P=3 30 0.4 0 0.63 3 3 30 0.2 0 0.83 You should carefully check the matrix P, and make sure you understand how it was obtained.
Example 4.5 (A Random Walk Model) A Markov chain whose state space is given by the integers i = 0, ±1, ±2, . . . is said to be a random walk if, for some number 0 < p < 1, Pi,i+1 = p = 1 − Pi,i−1 ,
i = 0, ±1, . . .
The preceding Markov chain is called a random walk for we may think of it as being a model for an individual walking on a straight line who at each point of time either takes one step to the right with probability p or one step to the left with probability 1 − p.
Example 4.6 (A Gambling Model) Consider a gambler who, at each play of
the game, either wins $1 with probability p or loses $1 with probability 1 − p. If we suppose that our gambler quits playing either when he goes broke or he attains a fortune of $N , then the gambler’s fortune is a Markov chain having transition probabilities Pi,i+1 = p = 1 − Pi,i−1 ,
i = 1, 2, . . . , N − 1,
P00 = PN N = 1 States 0 and N are called absorbing states since once entered they are never left. Note that the preceding is a finite state random walk with absorbing barriers (states 0 and N ).
188
4
Markov Chains
Example 4.7 In most of Europe and Asia annual automobile insurance premiums are determined by use of a Bonus Malus (Latin for Good-Bad) system. Each policyholder is given a positive integer valued state and the annual premium is a function of this state (along, of course, with the type of car being insured and the level of insurance). A policyholder’s state changes from year to year in response to the number of claims made by that policyholder. Because lower numbered states correspond to lower annual premiums, a policyholder’s state will usually decrease if he or she had no claims in the preceding year, and will generally increase if he or she had at least one claim. (Thus, no claims is good and typically results in a decreased premium, while claims are bad and typically result in a higher premium.) For a given Bonus Malus system, let si (k) denote the next state of a policyholder who was in state i in the previous year and who made a total of k claims in that year. If we suppose that the number of yearly claims made by a particular policyholder is a Poisson random variable with parameter λ, then the successive states of this policyholder will constitute a Markov chain with transition probabilities Pi,j =
e−λ
k:si (k)=j
λk , k!
j 0
Whereas there are usually many states (20 or so is not atypical), the following table specifies a hypothetical Bonus Malus system having four states. Next state if State
Annual Premium
0 claims
1 claim
2 claims
3 claims
1 2 3 4
200 250 400 600
1 1 2 3
2 3 4 4
3 4 4 4
4 4 4 4
Thus, for instance, the table indicates that s2 (0) = 1; s2 (1) = 3; s2 (k) = 4, k 2. Consider a policyholder whose annual number of claims is a Poisson random variable with parameter λ. If ak is the probability that such a policyholder makes k claims in a year, then ak = e−λ
λk , k!
k0
4.2.
Chapman–Kolmogorov Equations
189
For the Bonus Malus system specified in the preceding table, the transition probability matrix of the successive states of this policyholder is 3 3 3 a0 a1 a2 1 − a0 − a1 − a2 3 3 3 3 3 a0 0 a1 1 − a0 − a1 3 P=3 3 30 a0 0 1 − a0 3 3 3 30 0 a0 1 − a0
4.2.
Chapman–Kolmogorov Equations
We have already defined the one-step transition probabilities Pij . We now define the n-step transition probabilities Pijn to be the probability that a process in state i will be in state j after n additional transitions. That is, Pijn = P {Xn+k = j |Xk = i},
n 0, i, j 0
Of course Pij1 = Pij . The Chapman–Kolmogorov equations provide a method for computing these n-step transition probabilities. These equations are Pijn+m =
∞
Pikn Pkjm
for all n, m 0, all i, j
(4.2)
k=0
and are most easily understood by noting that Pikn Pkjm represents the probability that starting in i the process will go to state j in n + m transitions through a path which takes it into state k at the nth transition. Hence, summing over all intermediate states k yields the probability that the process will be in state j after n + m transitions. Formally, we have Pijn+m = P {Xn+m = j |X0 = i} =
∞
P {Xn+m = j, Xn = k|X0 = i}
k=0
=
∞
P {Xn+m = j |Xn = k, X0 = i}P {Xn = k|X0 = i}
k=0
=
∞ k=0
Pkjm Pikn
190
4
Markov Chains
If we let P(n) denote the matrix of n-step transition probabilities Pijn , then Equation (4.2) asserts that P(n+m) = P(n) · P(m) where the dot represents matrix multiplication.∗ Hence, in particular, P(2) = P(1+1) = P · P = P2 and by induction P(n) = P(n−1+1) = Pn−1 · P = Pn That is, the n-step transition matrix may be obtained by multiplying the matrix P by itself n times.
Example 4.8 Consider Example 4.1 in which the weather is considered as a two-state Markov chain. If α = 0.7 and β = 0.4, then calculate the probability that it will rain four days from today given that it is raining today. Solution:
The one-step transition probability matrix is given by 3 3 30.7 0.33 3 P=3 30.4 0.63
Hence, 3 3 3 3 30.7 0.33 30.7 0.33 3 3·3 P(2) = P2 = 3 30.4 0.63 30.4 0.63 3 3 30.61 0.393 3 =3 30.52 0.483 , 3 3 3 3 30.61 0.393 30.61 0.393 (4) 2 2 3 3 3 3 P = (P ) = 3 · 0.52 0.483 30.52 0.483 3 3 30.5749 0.42513 3 3 =3 0.5668 0.43323 4 equals 0.5749. and the desired probability P00
∗ If A is an N × M matrix whose element in the ith row and j th column is a and B is an M × K ij matrix whose element in the ith row and j th column is bij , then A · B is defined to be the N × K M matrix whose element in the ith row and j th column is k=1 aik bkj .
4.2.
Chapman–Kolmogorov Equations
191
Example 4.9 Consider Example 4.4. Given that it rained on Monday and Tuesday, what is the probability that it will rain on Thursday? Solution:
The two-step transition matrix is given by 3 3 3 3 30.7 0 0.3 0 3 3 30.7 0 3 3 3 3 0.5 0 0.5 0 3 · 30.5 0 P(2) = P2 = 3 3 30 30 0.4 0.4 0 0.6 3 3 3 30 0.2 0.2 0 0.83 30 3 3 30.49 0.12 0.21 0.183 3 3 30.35 0.20 0.15 0.303 3 3 =3 3 30.20 0.12 0.20 0.483 30.10 0.16 0.10 0.643
0.3 0.5 0 0
3 0 3 3 0 3 3 0.63 3 0.83
Since rain on Thursday is equivalent to the process being in either state 0 or 2 + P 2 = 0.49 + state 1 on Thursday, the desired probability is given by P00 01 0.12 = 0.61. So far, all of the probabilities we have considered are conditional probabilities. For instance, Pijn is the probability that the state at time n is j given that the initial state at time 0 is i. If the unconditional distribution of the state at time n is desired, it is necessary to specify the probability distribution of the initial state. Let us denote this by ∞ αi ≡ P {X0 = i}, i0 αi = 1 i=0
All unconditional probabilities may be computed by conditioning on the initial state. That is, P {Xn = j } =
∞
P {Xn = j |X0 = i}P {X0 = i}
i=0
=
∞
Pijn αi
i=0
For instance, if α0 = 0.4, α1 = 0.6, in Example 4.8, then the (unconditional) probability that it will rain four days after we begin keeping weather records is 4 4 + 0.6P10 P {X4 = 0} = 0.4P00
= (0.4)(0.5749) + (0.6)(0.5668) = 0.5700
192
4
Markov Chains
Suppose now that you want to determine the probability that a Markov chain enters any of a specified set of states A by time n. One way to accomplish this is to reset the transition probabilities out of states in A to P {Xm+1 = j |Xm = i} =
1,
if i ∈ A , j = i
0,
if i ∈ A , j = i
That is, transform all states in A into absorbing states which once entered can never be left. Because the original and transformed Markov chain follows identical probabilities until a state in A is entered, it follows that the probability the original Markov chain enters a state in A by time n is equal to the probability that the transformed Markov chain is in one of the states of A at time n.
Example 4.10 A pensioner receives 2 (thousand dollars) at the beginning of each month. The amount of money he needs to spend during a month is independent of the amount he has and is equal to i with probability Pi , i = 1, 2, 3, 4, 4 P = 1. If the pensioner has more than 3 at the end of a month, he gives the i i=1 amount greater than 3 to his son. If, after receiving his payment at the beginning of a month, the pensioner has a capital of 5, what is the probability that his capital is ever 1 or less at any time within the following four months? Solution: To find the desired probability, we consider a Markov chain with the state equal to the amount the pensioner has at the end of a month. Because we are interested in whether this amount ever falls as low as 1, we will let 1 mean that the pensioner’s end-of-month fortune has ever been less than or equal to 1. Because the pensioner will give any end-of-month amount greater than 3 to his son, we need only consider the Markov chain with states 1, 2, 3 and transition probability matrix Q = [Qi,j ] given by 3 3 1 3 3P3 + P4 3 3 P4
0 P2 P3
3 0 3 3 P1 3 3 P1 + P2 3
To understand the preceding, consider Q2,1 , the probability that a month that ends with the pensioner having the amount 2 will be followed by a month that ends with the pensioner having less than or equal to 1. Because the pensioner will begin the new month with the amount 2 + 2 = 4, his ending capital will be less than or equal to 1 if his expenses are either 3 or 4. Thus, Q2,1 = P3 + P4 . The other transition probabilities are similarly explained.
4.3.
Classification of States
193
Suppose now that Pi = 1/4, i = 1, 2, 3, 4. The transition probability matrix is 3 3 3 1 0 0 3 3 3 31/2 1/4 1/43 3 3 31/4 1/4 1/23 Squaring this matrix and then squaring the result gives the matrix 3 3 3 1 0 0 3 3 3 3 222 13 21 3 3 256 256 3 256 3 3 3 201 21 34 3 3 3 256
256
256
Because the pensioner’s initial end of month capital was 3, the desired answer is Q43,1 = 201/256. Let {Xn , n 0} be a Markov chain with transition probabilities Pi,j . If we let Qi,j denote the transition probabilities that transform all states in A into absorbing states, then ⎧ if i ∈ A , j = i ⎨ 1, if i ∈ A , j = i Qi,j = 0, ⎩ otherwise Pi,j , For i, j ∈ / A , the n stage transition probability Qni,j represents the probability that the original chain, starting in state i, will be in state j at time n without ever having entered any of the states in A . For instance, in Example 4.10, starting with 5 at the beginning of January, the probability that the pensioner’s capital is 4 at the beginning of May without ever having been less than or equal to 1 in that time is Q43,2 = 21/256. We can also compute the conditional probability of Xn given that the chain starts in state i and has not entered any state in A by time n, as follows. For i, j ∈ /A, P {Xn = j |X0 = i, Xk ∈ / A , k = 1, . . . , n} =
4.3.
Qni,j / A , k = 1, . . . , n|X0 = i} P {Xn = j, Xk ∈ = n P {Xk ∈ / A , k = 1, . . . , n|X0 = i} r ∈A / Qi,r
Classification of States
State j is said to be accessible from state i if Pijn > 0 for some n 0. Note that this implies that state j is accessible from state i if and only if, starting in i,
194
4
Markov Chains
it is possible that the process will ever enter state j . This is true since if j is not accessible from i, then ∞ !
! P {ever enterj |start in i} = P {Xn = j }!X0 = i n=0
∞
P {Xn = j |X0 = i}
n=0
=
∞
Pijn
n=0
=0 Two states i and j that are accessible to each other are said to communicate, and we write i ↔ j . Note that any state communicates with itself since, by definition, Pii0 = P {X0 = i|X0 = i} = 1 The relation of communication satisfies the following three properties: (i) State i communicates with state i, all i 0. (ii) If state i communicates with state j , then state j communicates with state i. (iii) If state i communicates with state j , and state j communicates with state k, then state i communicates with state k. Properties (i) and (ii) follow immediately from the definition of communication. To prove (iii) suppose that i communicates with j , and j communicates with k. Thus, there exist integers n and m such that Pijn > 0, Pjmk > 0. Now by the Chapman–Kolmogorov equations, we have that Pikn+m =
∞
m Pirn Prk Pijn Pjmk > 0
r=0
Hence, state k is accessible from state i. Similarly, we can show that state i is accessible from state k. Hence, states i and k communicate. Two states that communicate are said to be in the same class. It is an easy consequence of (i), (ii), and (iii) that any two classes of states are either identical or disjoint. In other words, the concept of communication divides the state space up into a number of separate classes. The Markov chain is said to be irreducible if there is only one class, that is, if all states communicate with each other.
4.3.
Classification of States
195
Example 4.11 Consider the Markov chain consisting of the three states 0, 1, 2 and having transition probability matrix 31 1 3 32 2 3 P = 3 12 14 3 30 1
3 03 3 13 43 3 23
3
3
It is easy to verify that this Markov chain is irreducible. For example, it is possible to go from state 0 to state 2 since 0→1→2 That is, one way of getting from state 0 to state 2 is to go from state 0 to state 1 (with probability 12 ) and then go from state 1 to state 2 (with probability 14 ).
Example 4.12 Consider a Markov chain consisting of the four states 0, 1, 2, 3 and having transition probability matrix 3 31 1 32 2 31 1 32 2 P=3 31 1 34 4 3 30 0
0 0 1 4
0
3 03 3 3 03 3 13 43 3 13
The classes of this Markov chain are {0, 1}, {2}, and {3}. Note that while state 0 (or 1) is accessible from state 2, the reverse is not true. Since state 3 is an absorbing state, that is, P33 = 1, no other state is accessible from it. For any state i we let fi denote the probability that, starting in state i, the process will ever reenter state i. State i is said to be recurrent if fi = 1 and transient if fi < 1. Suppose that the process starts in state i and i is recurrent. Hence, with probability 1, the process will eventually reenter state i. However, by the definition of a Markov chain, it follows that the process will be starting over again when it reenters state i and, therefore, state i will eventually be visited again. Continual repetition of this argument leads to the conclusion that if state i is recurrent then, starting in state i, the process will reenter state i again and again and again—in fact, infinitely often. On the other hand, suppose that state i is transient. Hence, each time the process enters state i there will be a positive probability, namely, 1 − fi , that it will never again enter that state. Therefore, starting in state i, the probability that the process will be in state i for exactly n time periods equals fin−1 (1 − fi ), n 1. In other words, if state i is transient then, starting in state i, the number of time periods
196
4
Markov Chains
that the process will be in state i has a geometric distribution with finite mean 1/(1 − fi ). From the preceding two paragraphs, it follows that state i is recurrent if and only if, starting in state i, the expected number of time periods that the process is in state i is infinite. But, letting 1, if Xn = i In = 0, if Xn = i ∞ we have that n=0 In represents the number of periods that the process is in state i. Also, ,∞ ∞ E In |X0 = i = E[In |X0 = i] n=0
n=0
=
∞
P {Xn = i|X0 = i}
n=0
=
∞
Piin
n=0
We have thus proven the following.
Proposition 4.1 State i is recurrent if
∞
Piin = ∞,
n=1
transient if
∞
Piin < ∞
n=1
The argument leading to the preceding proposition is doubly important because it also shows that a transient state will only be visited a finite number of times (hence the name transient). This leads to the conclusion that in a finite-state Markov chain not all states can be transient. To see this, suppose the states are 0, 1, . . . , M and suppose that they are all transient. Then after a finite amount of time (say, after time T0 ) state 0 will never be visited, and after a time (say, T1 ) state 1 will never be visited, and after a time (say, T2 ) state 2 will never be visited, and so on. Thus, after a finite time T = max{T0 , T1 , . . . , TM } no states will be visited. But as the process must be in some state after time T we arrive at a contradiction, which shows that at least one of the states must be recurrent. Another use of Proposition 4.1 is that it enables us to show that recurrence is a class property.
4.3.
Classification of States
197
Corollary 4.2 If state i is recurrent, and state i communicates with state j , then state j is recurrent.
Proof To prove this we first note that, since state i communicates with state j , there exist integers k and m such that Pijk > 0, Pjmi > 0. Now, for any integer n m+n+k Pjj Pjmi Piin Pijk
This follows since the left side of the preceding is the probability of going from j to j in m+n+k steps, while the right side is the probability of going from j to j in m + n + k steps via a path that goes from j to i in m steps, then from i to i in an additional n steps, then from i to j in an additional k steps. From the preceding we obtain, by summing over n, that ∞
m+n+k Pjj Pjmi Pijk
n=1
∞
Piin = ∞
n=1
n since Pjmi Pijk > 0 and ∞ n=1 Pii is infinite since state i is recurrent. Thus, by Proposition 4.1 it follows that state j is also recurrent.
Remarks (i) Corollary 4.2 also implies that transience is a class property. For if state i is transient and communicates with state j , then state j must also be transient. For if j were recurrent then, by Corollary 4.2, i would also be recurrent and hence could not be transient. (ii) Corollary 4.2 along with our previous result that not all states in a finite Markov chain can be transient leads to the conclusion that all states of a finite irreducible Markov chain are recurrent.
Example 4.13 Let the Markov chain consisting of the states 0, 1, 2, 3 have the transition probability matrix 3 30 3 31 P=3 30 3 30
0
1 2
0 1 1
0 0 0
13 3 23
03 3 03 3 03
Determine which states are transient and which are recurrent. Solution: It is a simple matter to check that all states communicate and, hence, since this is a finite chain, all states must be recurrent.
198
4
Markov Chains
Example 4.14 Consider the Markov chain having states 0, 1, 2, 3, 4 and 31 32 3 31 32 3 P=3 30 3 30 3 31
1 2 1 2
0
0
0
0
0 0
1 2 1 2
1 2 1 2
1 4
0
0
4
3 03 3 3 03 3 03 3 3 03 3 13 2
Determine the recurrent state. Solution: This chain consists of the three classes {0, 1}, {2, 3}, and {4}. The first two classes are recurrent and the third transient.
Example 4.15 (A Random Walk) Consider a Markov chain whose state
space consists of the integers i = 0, ±1, ±2, . . . , and have transition probabilities given by Pi,i+1 = p = 1 − Pi,i−1 ,
i = 0, ±1, ±2, . . .
where 0 < p < 1. In other words, on each transition the process either moves one step to the right (with probability p) or one step to the left (with probability 1−p). One colorful interpretation of this process is that it represents the wanderings of a drunken man as he walks along a straight line. Another is that it represents the winnings of a gambler who on each play of the game either wins or loses one dollar. Since all states clearly communicate, it follows from Corollary 4.2 that they are either all transient or all recurrent. So let us consider state 0 and attempt to n determine if ∞ n=1 P00 is finite or infinite. Since it is impossible to be even (using the gambling model interpretation) after an odd number of plays we must, of course, have that 2n−1 P00 = 0,
n = 1, 2, . . .
On the other hand, we would be even after 2n trials if and only if we won n of these and lost n of these. Because each play of the game results in a win with probability p and a loss with probability 1 − p, the desired probability is thus the binomial probability 2n n (2n)! 2n P00 = (p(1 − p))n , p (1 − p)n = n = 1, 2, 3, . . . n n!n! By using an approximation, due to Stirling, which asserts that √ n! ∼ nn+1/2 e−n 2π
(4.3)
4.3.
Classification of States
199
where we say that an ∼ bn when limn→∞ an /bn = 1, we obtain 2n ∼ P00
(4p(1 − p))n √ πn
Now it is easy to verify, for positive ∞ an , nbn , that if an ∼ bn , then n an < ∞ if and only if n bn < ∞. Hence, n=1 P00 will converge if and only if ∞ (4p(1 − p))n √ πn n=1
does. However, 4p(1 − p) 1 with equality holding if and only if p = 12 . Hence, ∞ n 1 1 n=1 P00 = ∞ if and only if p = 2 . Thus, the chain is recurrent when p = 2 and 1 transient if p = 2 . When p = 12 , the preceding process is called a symmetric random walk. We could also look at symmetric random walks in more than one dimension. For instance, in the two-dimensional symmetric random walk the process would, at each transition, either take one step to the left, right, up, or down, each having probability 14 . That is, the state is the pair of integers (i, j ) and the transition probabilities are given by P(i,j ),(i+1,j ) = P(i,j ),(i−1,j ) = P(i,j ),(i,j +1) = P(i,j ),(i,j −1) =
1 4
By using the same method as in the one-dimensional case, we now show that this Markov chain is also recurrent. Since the preceding chain is irreducible, it follows that all states will be recur2n . Now after 2n steps, the chain rent if state 0 = (0, 0) is recurrent. So consider P00 will be back in its original location if for some i, 0 i n, the 2n steps consist of i steps to the left, i to the right, n − i up, and n − i down. Since each step will be either of these four types with probability 14 , it follows that the desired probability is a multinomial probability. That is, 2n n 1 (2n)! 2n P00 = i!i!(n − i)!(n − i)! 4 i=0
=
2n n! n! 1 n!n! (n − i)!i! (n − i)!i! 4
n (2n)! i=0
2n n 2n n n 1 = n i n−i 4 i=0
2n 2n 2n 1 = n n 4
(4.4)
200
4
Markov Chains
where the last equality uses the combinatorial identity n 2n n n = n i n−i i=0
which follows upon noting that both sides represent the number of subgroups of size n one can select from a set of n white and n black objects. Now, (2n)! 2n = n n!n! √ (2n)2n+1/2 e−2n 2π by Stirling’s approximation ∼ n2n+1 e−2n (2π) 4n =√ πn Hence, from Equation (4.4) we see that 2n P00 ∼
1 πn
2n which shows that n P00 = ∞, and thus all states are recurrent. Interestingly enough, whereas the symmetric random walks in one and two dimensions are both recurrent, all higher-dimensional symmetric random walks turn out to be transient. (For instance, the three-dimensional symmetric random walk is at each transition equally likely to move in any of six ways—either to the left, right, up, down, in, or out.)
Remarks For the one-dimensional random walk of Example 4.15 here is a direct argument for establishing recurrence in the symmetric case, and for determining the probability that it ever returns to state 0 in the nonsymmetric case. Let β = P {ever return to 0} To determine β, start by conditioning on the initial transition to obtain β = P {ever return to 0|X1 = 1}p + P {ever return to 0|X1 = −1}(1 − p) (4.5) Now, let α denote the probability that the Markov chain will ever return to state 0 given that it is currently in state 1. Because the Markov chain will always increase by 1 with probability p or decrease by 1 with probability 1 − p no matter what its current state, note that α is also the probability that the Markov chain currently in state i will ever enter state i − 1, for any i. To obtain an equation for α, condition on the next transition to obtain
4.3.
Classification of States
201
α = P {every return|X1 = 1, X2 = 0}(1 − p) + P {ever return|X1 = 1, X2 = 2}p = 1 − p + P {ever return|X1 = 1, X2 = 2}p = 1 − p + pα 2 where the final equation follows by noting that in order for the chain to ever go from state 2 to state 0 it must first go to state 1—and the probability of that ever happening is α—and if it does eventually go to state 1 then it must still go to state 0—and the conditional probability of that ever happening is also α. Therefore, α = 1 − p + pα 2 The two roots of this equation are α = 1 and α = (1 − p)/p. Consequently, in the case of the symmetric random walk where p = 1/2 we can conclude that α = 1. By symmetry, the probability that the symmetric random walk will ever enter state 0 given that it is currently in state −1 is also 1, proving that the symmetric random walk is recurrent. Suppose now that p > 1/2. In this case, it can be shown (see Exercise 17 at the end of this chapter) that P {ever return to|X1 = −1} = 1. Consequently, Equation (4.5) reduces to β = αp + 1 − p Because the random walk is transient in this case we know that β < 1, showing that α = 1. Therefore, α = (1 − p)/p, yielding that β = 2(1 − p),
p > 1/2
Similarly, when p < 1/2 we can show that β = 2p. Thus, in general P {ever return to 0} = 2 min(p, 1 − p)
Example 4.16 (On the Ultimate Instability of the Aloha Protocol) Consider a communications facility in which the numbers of messages arriving during each of the time periods n = 1, 2, . . . are independent and identically distributed random variables. Let ai = P {i arrivals}, and suppose that a0 + a1 < 1. Each arriving message will transmit at the end of the period in which it arrives. If exactly one message is transmitted, then the transmission is successful and the message leaves the system. However, if at any time two or more messages simultaneously transmit, then a collision is deemed to occur and these messages remain in the system. Once a message is involved in a collision it will, independently of all else, transmit at the end of each additional period with probability p—the so-called Aloha
202
4
Markov Chains
protocol (because it was first instituted at the University of Hawaii). We will show that such a system is asymptotically unstable in the sense that the number of successful transmissions will, with probability 1, be finite. To begin let Xn denote the number of messages in the facility at the beginning of the nth period, and note that {Xn , n 0} is a Markov chain. Now for k 0 define the indicator variables Ik by ⎧ if the first time that the chain departs state k it ⎨ 1, directly goes to state k − 1 Ik = ⎩ 0, otherwise and let it be 0 if the system is never in state k, k 0. (For instance, if the successive states are 0, 1, 3, 4, . . . , then I3 = 0 since when the chain first departs state 3 it goes to state 4; whereas, if they are 0, 3, 3, 2, . . . , then I3 = 1 since this time it goes to state 2.) Now, ,∞ ∞ E Ik = E[Ik ] k=0
k=0
=
∞
P {Ik = 1}
k=0
∞
P {Ik = 1|k is ever visited}
(4.6)
k=0
Now, P {Ik = 1|k is ever visited} is the probability that when state k is departed the next state is k − 1. That is, it is the conditional probability that a transition from k is to k − 1 given that it is not back into k, and so P {Ik = 1|k is ever visited} =
Pk,k−1 1 − Pkk
Because Pk,k−1 = a0 kp(1 − p)k−1 , Pk,k = a0 [1 − kp(1 − p)k−1 ] + a1 (1 − p)k which is seen by noting that if there are k messages present on the beginning of a day, then (a) there will be k − 1 at the beginning of the next day if there are no new messages that day and exactly one of the k messages transmits; and (b) there will be k at the beginning of the next day if either
4.3.
Classification of States
203
(i) there are no new messages and it is not the case that exactly one of the existing k messages transmits, or (ii) there is exactly one new message (which automatically transmits) and none of the other k messages transmits. Substitution of the preceding into Equation (4.6) yields ,∞ ∞ a0 kp(1 − p)k−1 Ik E 1 − a0 [1 − kp(1 − p)k−1 ] − a1 (1 − p)k k=0
k=0
12 , there is a positive probability that the gambler’s fortune will increase indefinitely; while if p 12 , the gambler will, with probability 1, go broke against an infinitely rich adversary.
Example 4.24 Suppose Max and Patty decide to flip pennies; the one coming closest to the wall wins. Patty, being the better player, has a probability 0.6 ofwinning on each flip. (a) If Patty starts with five pennies and Max with ten, what is the probability that Patty will wipe Max out? (b) What if Patty starts with 10 and Max with 20? Solution: (a) The desired probability is obtained from Equation (4.14) by letting i = 5, N = 15, and p = 0.6. Hence, the desired probability is 1− 1−
2 5 3 2 15 ≈ 0.87 3
(b) The desired probability is 1− 1−
2 10 3 2 30 ≈ 0.98 3
For an application of the gambler’s ruin problem to drug testing, suppose that two new drugs have been developed for treating a certain disease. Drug i has a cure rate Pi , i = 1, 2, in the sense that each patient treated with drug i will be cured with probability Pi . These cure rates, however, are not known, and suppose we are interested in a method for deciding whether P1 > P2 or P2 > P1 . To decide upon one of these alternatives, consider the following test: Pairs of patients are treated sequentially with one member of the pair receiving drug 1 and the other
220
4
Markov Chains
drug 2. The results for each pair are determined, and the testing stops when the cumulative number of cures using one of the drugs exceeds the cumulative number of cures when using the other by some fixed predetermined number. More formally, let 1, if the patient in the j th pair to receive drug number 1 is cured Xj = 0, otherwise 1, if the patient in the j th pair to receive drug number 2 is cured Yj = 0, otherwise For a predetermined positive integer M the test stops after pair N where N is the first value of n such that either X1 + · · · + Xn − (Y1 + · · · + Yn ) = M or X1 + · · · + Xn − (Y1 + · · · + Yn ) = −M In the former case we then assert that P1 > P2 , and in the latter that P2 > P1 . In order to help ascertain whether the preceding is a good test, one thing we would like to know is the probability of it leading to an incorrect decision. That is, for given P1 and P2 where P1 > P2 , what is the probability that the test will incorrectly assert that P2 > P1 ? To determine this probability, note that after each pair is checked the cumulative difference of cures using drug 1 versus drug 2 will either go up by 1 with probability P1 (1 − P2 )—since this is the probability that drug 1 leads to a cure and drug 2 does not—or go down by 1 with probability (1 − P1 )P2 , or remain the same with probability P1 P2 + (1 − P1 )(1 − P2 ). Hence, if we only consider those pairs in which the cumulative difference changes, then the difference will go up 1 with probability p = P {up 1|up1 or down 1} =
P1 (1 − P2 ) P1 (1 − P2 ) + (1 − P1 )P2
and down 1 with probability q =1−p=
P2 (1 − P1 ) P1 (1 − P2 ) + (1 − P1 )P2
Hence, the probability that the test will assert that P2 > P1 is equal to the probability that a gambler who wins each (one unit) bet with probability p will go down
4.5.
Some Applications
221
M before going up M. But Equation (4.14) with i = M, N = 2M, shows that this probability is given by P {test asserts that P2 > P1 } = 1 − =
1 − (q/p)M 1 − (q/p)2M
1 1 + (p/q)M
Thus, for instance, if P1 = 0.6 and P2 = 0.4 then the probability of an incorrect decision is 0.017 when M = 5 and reduces to 0.0003 when M = 10. 4.5.2. A Model for Algorithmic Efficiency The following optimization problem is called a linear program: minimize cx, subject to Ax = b, x0 where A is an m × n matrix of fixed constants; c = (c1 , . . . , cn ) and b = (b1 , . . . , bm ) are vectors of fixed constants; and x = (x1 , . . . , xn ) is the n-vector of nonnegative values that is to be chosen to minimize cx ≡ ni=1 ci xi . Supposing that n > m, it can be shown that the optimal x can always be chosen to have at least n − m components equal to 0—that is, it can always be taken to be one of the so-called extreme points of the feasibility region. The simplex algorithm solves this linear program by moving from an extreme point of the feasibility region to a better (in terms of the objective function cx) extreme point (via the pivot n operation) until the optimal is reached. Because there can be as many as N ≡ m such extreme points, it would seem that this method might take many iterations, but, surprisingly to some, this does not appear to be the case in practice. To obtain a feel for whether or not the preceding statement is surprising, let us consider a simple probabilistic (Markov chain) model as to how the algorithm moves along the extreme points. Specifically, we will suppose that if at any time the algorithm is at the j th best extreme point then after the next pivot the resulting extreme point is equally likely to be any of the j − 1 best. Under this assumption, we show that the time to get from the N th best to the best extreme point has approximately, for large N , a normal distribution with mean and variance equal to the logarithm (base e) of N .
222
4
Markov Chains
Consider a Markov chain for which P11 = 1 and 1 , i−1
Pij =
j = 1, . . . , i − 1, i > 1
and let Ti denote the number of transitions needed to go from state i to state 1. A recursive formula for E[Ti ] can be obtained by conditioning on the initial transition: 1 E[Ti ] = 1 + E[Tj ] i −1 i−1
j =1
Starting with E[T1 ] = 0, we successively see that E[T2 ] = 1, E[T3 ] = 1 + 12 , E[T4 ] = 1 + 13 (1 + 1 + 12 ) = 1 +
1 2
+
1 3
and it is not difficult to guess and then prove inductively that E[Ti ] =
i−1
1/j
j =1
However, to obtain a more complete description of TN , we will use the representation TN =
N −1
Ij
j =1
where Ij =
1, 0,
if the process ever enters j otherwise
The importance of the preceding representation stems from the following:
Proposition 4.4 I1 , . . . , IN −1 are independent and P {Ij = 1} = 1/j,
1j N −1
Proof Given Ij +1 , . . . , IN , let n = min{i: i > j, Ii = 1} denote the lowest numbered state, greater than j , that is entered. Thus we know that the process enters
4.5.
Some Applications
223
state n and the next state entered is one of the states 1, 2, . . . , j . Hence, as the next state from state n is equally likely to be any of the lower number states 1, 2, . . . , n − 1 we see that P {Ij = 1|Ij +1 , . . . , IN } =
1/(n − 1) = 1/j j/(n − 1)
Hence, P {Ij = 1} = 1/j , and independence follows since the preceding conditional probability does not depend on Ij +1 , . . . , IN .
Corollary 4.5 (i) E[TN ] =
N −1
j =1 1/j . N −1 (ii) Var(TN ) = j =1 (1/j )(1 − 1/j ). (iii) For N large, TN has approximately a normal distribution with mean log N and variance log N .
Proof Parts (i) and (ii) follow from Proposition 4.4 and the representation TN =
N −1 j =1 Ij .
Part (iii) follows from the central limit theorem since 1
N
N −1 N −1 dx dx < 1/j < 1 + x x 1 1
or log N
1, or equivalently when p < 12 , the expected number of transitions to reach n is an exponentially increasing function of n. On the other hand, when p = 1 1 2 2 , E[N0,n ] = n , and when p > 2 , E[N0,n ] is, for large n, essentially linear in n. Let us now compute Var(N0,n ). To do so, we will again make use of the representation given by Equation (4.15). Letting vi = Var(Ni ), we start by determining the vi recursively by using the conditional variance formula. Let Si = 1 if the first transition out of state i is into state i + 1, and let Si = −1 if the transition is into state i − 1, i = 1, . . . , n − 1. Then, given that Si = 1:
N1 = 1
given that Si = −1:
∗ Ni = 1 + Ni−1 + Ni∗
Hence, E[Ni |Si = 1] = 1, E[Ni |Si = −1] = 1 + μi−1 + μi
4.5.
Some Applications
227
implying that Var(E[Ni |Si ]) = Var(E[Ni |Si ] − 1) = (μi−1 + μi )2 q − (μi−1 + μi )2 q 2 = qp(μi−1 + μi )2 ∗ Also, since Ni−1 and Ni∗ , the numbers of transitions to return from state i − 1 to i and to then go from state i to state i + 1 are, by the Markovian property, independent random variables having the same distributions as Ni−1 and Ni , respectively, we see that
Var(Ni |Si = 1) = 0, Var(Ni |Si = −1) = vi−1 + vi Hence, E[Var(Ni |Si )] = q(vi−1 + vi ) From the conditional variance formula, we thus obtain that vi = pq(μi−1 + μi )2 + q(vi−1 + vi ) or, equivalently, vi = q(μi−1 + μi )2 + αvi−1 ,
i = 1, . . . , n − 1
Starting with v0 = 0, we obtain from the preceding recursion that v1 = q(μ0 + μ1 )2 , v2 = q(μ1 + μ2 )2 + αq(μ0 + μ1 )2 , v3 = q(μ2 + μ3 )2 + αq(μ1 + μ2 )2 + α 2 q(μ0 + μ1 )2 In general, we have for i > 0, vi = q
i
α i−j (μj −1 + μj )2
j =1
Therefore, we see that Var(N0,n ) =
n−1
vi = q
i=0
where μj is given by Equation (4.16).
n−1 i i=1 j =1
α i−j (μj −1 + μj )2
(4.17)
228
4
Markov Chains
We see from Equations (4.16) and (4.17) that when p 12 , and so α 1, that μi and vi , the mean and variance of the number of transitions to go from state i to i + 1, do not increase too rapidly in i. For instance, when p = 12 it follows from Equations (4.16) and (4.17) that μi = 2i + 1 and vi =
1 (4j )2 = 8 j2 2 i
i
j =1
j =1
Hence, since N0,n is the sum of independent random variables, which are of roughly similar magnitudes when p 12 , it follows in this case from the central limit theorem that N0,n is, for large n, approximately normally distributed. In particular, when p = 12 , N0,n is approximately normal with mean n2 and variance Var(N0,n ) = 8
n−1 i
j2
i=1 j =1
=8
n−1 n−1
j2
j =1 i=j
=8
n−1
(n − j )j 2
j =1
≈8
n−1
(n − x)x 2 dx
1
≈
2 4 3n
Example 4.25 (The Satisfiability Problem) A Boolean variable x is one that takes on either of two values: TRUE or FALSE. If xi , i 1 are Boolean variables, then a Boolean clause of the form x1 + x¯2 + x3 is TRUE if x1 is TRUE, or if x2 is FALSE, or if x3 is TRUE. That is, the symbol “+” means “or” and x¯ is TRUE if x is FALSE and vice versa. A Boolean formula is a combination of clauses such as (x1 + x¯2 ) ∗ (x1 + x3 ) ∗ (x2 + x¯3 ) ∗ (x¯1 + x¯2 ) ∗ (x1 + x2 ) In the preceding, the terms between the parentheses represent clauses, and the formula is TRUE if all the clauses are TRUE, and is FALSE otherwise. For a
4.5.
Some Applications
229
given Boolean formula, the satisfiability problem is either to determine values for the variables that result in the formula being TRUE, or to determine that the formula is never true. For instance, one set of values that makes the preceding formula TRUE is to set x1 = TRUE, x2 = FALSE, and x3 = FALSE. Consider a formula of the n Boolean variables x1 , . . . , xn and suppose that each clause in this formula refers to exactly two variables. We will now present a probabilistic algorithm that will either find values that satisfy the formula or determine to a high probability that it is not possible to satisfy it. To begin, start with an arbitrary setting of values. Then, at each stage choose a clause whose value is FALSE, and randomly choose one of the Boolean variables in that clause and change its value. That is, if the variable has value TRUE then change its value to FALSE, and vice versa. If this new setting makes the formula TRUE then stop,4otherwise continue in the same fashion. If you have not stopped after n2 (1 + 4 23 ) repetitions, then declare that the formula cannot be satisfied. We will now argue that if there is a satisfiable assignment then this algorithm will find such an assignment with a probability very close to 1. Let us start by assuming that there is a satisfiable assignment of truth values and let A be such an assignment. At each stage of the algorithm there is a certain assignment of values. Let Yj denote the number of the n variables whose values at the j th stage of the algorithm agree with their values in A . For instance, suppose that n = 3 and A consists of the settings x1 = x2 = x3 = TRUE. If the assignment of values at the j th step of the algorithm is x1 = TRUE, x2 = x3 = FALSE, then Yj = 1. Now, at each stage, the algorithm considers a clause that is not satisfied, thus implying that at least one of the values of the two variables in this clause does not agree with its value in A . As a result, when we randomly choose one of the variables in this clause then there is a probability of at least 12 that Yj +1 = Yj + 1 and at most 12 that Yj +1 = Yj − 1. That is, independent of what has previously transpired in the algorithm, at each stage the number of settings in agreement with those in A will either increase or decrease by 1 and the probability of an increase is at least 12 (it is 1 if both variables have values different from their values in A ). Thus, even though the process Yj , j 0 is not itself a Markov chain (why not?) it is intuitively clear that both the expectation and the variance of the number of stages of the algorithm needed to obtain the values of A will be less than or equal to the expectation and variance of the number of transitions to go from state 0 to state n in the Markov chain of Section 4.5.2. Hence, if the algorithm has not yet terminated because it found a set of satisfiable values different from that of A , it will do so within an expected time of at most n2 and with a standard deviation of 4 2 2 at most n 3 . In addition, since the time for the Markov chain to go from 0 to n is approximately normal when n is large we can be quite certain that a satisfiable 4 assignment will be reached by n2 + 4(n2 23 ) stages, and thus if one has not been found by this number of stages of the algorithm we can be quite certain that there is no satisfiable assignment.
230
4
Markov Chains
Our analysis also makes it clear why we assumed that there are only two variables in each clause. For if there were k, k > 2, variables in a clause then as any clause that is not presently satisfied may have only one incorrect setting, a randomly chosen variable whose value is changed might only increase the number of values in agreement with A with probability 1/k and so we could only conclude from our prior Markov chain results that the mean time to obtain the values in A is an exponential function of n, which is not an efficient algorithm when n is large.
4.6.
Mean Time Spent in Transient States
Consider now a finite state Markov chain and suppose that the states are numbered so that T = {1, 2, . . . , t} denotes the set of transient states. Let ⎤ ⎡ P11 P12 · · · P1t ⎢ .. .. .. ⎥ PT = ⎣ ... . . . ⎦ Pt1
···
Pt2
Ptt
and note that since PT specifies only the transition probabilities from transient states into transient states, some of its row sums are less than 1 (otherwise, T would be a closed class of states). For transient states i and j , let sij denote the expected number of time periods that the Markov chain is in state j , given that it starts in state i. Let δi,j = 1 when i = j and let it be 0 otherwise. Condition on the initial transition to obtain sij = δi,j + Pik skj k
= δi,j +
t
Pik skj
(4.18)
k=1
where the final equality follows since it is impossible to go from a recurrent to a transient state, implying that skj = 0 when k is a recurrent state. Let S denote the matrix of values sij , i, j = 1, . . . , t. That is, ⎤ ⎡ s11 s12 · · · s1t ⎢ .. .. .. ⎥ S = ⎣ ... . . . ⎦ st1
st2
···
stt
In matrix notation, Equation (4.18) can be written as S = I + PT S
4.6.
Mean Time Spent in Transient States
231
where I is the identity matrix of size t. Because the preceding equation is equivalent to (I − PT )S = I we obtain, upon multiplying both sides by (I − PT )−1 , S = (I − PT )−1 That is, the quantities sij , i ∈ T , j ∈ T , can be obtained by inverting the matrix I − PT . (The existence of the inverse is easily established.)
Example 4.26 Consider the gambler’s ruin problem with p = 0.4 and N = 7. Starting with 3 units, determine (a) the expected amount of time the gambler has 5 units, (b) the expected amount of time the gambler has 2 units. Solution: follows:
The matrix PT , which specifies Pij , i, j ∈ {1, 2, 3, 4, 5, 6}, is as
PT =
1 2 3 4 5 6
1
2
3
4
5
6
0 0.6 0 0 0 0
0.4 0 0.6 0 0 0
0 0.4 0 0.6 0 0
0 0 0.4 0 0.6 0
0 0 0 0.4 0 0.6
0 0 0 0 0.4 0
Inverting I − PT gives ⎡
1.6149 ⎢ 1.5372 ⎢ ⎢ 1.4206 −1 S = (I − PT ) = ⎢ ⎢ 1.2458 ⎢ ⎣ 0.9835 0.5901
1.0248 2.5619 2.3677 2.0763 1.6391 0.9835
0.6314 1.5784 2.9990 2.6299 2.0763 1.2458
0.3691 0.9228 1.7533 2.9990 2.3677 1.4206
Hence, s3,5 = 0.9228,
s3,2 = 2.3677
0.1943 0.4857 0.9228 1.5784 2.5619 1.5372
⎤ 0.0777 0.1943 ⎥ ⎥ 0.3691 ⎥ ⎥ 0.6314 ⎥ ⎥ 1.0248 ⎦ 1.6149
232
4
Markov Chains
For i ∈ T , j ∈ T , the quantity fij , equal to the probability that the Markov chain ever makes a transition into state j given that it starts in state i, is easily determined from PT . To determine the relationship, let us start by deriving an expression for sij by conditioning on whether state j is ever entered. This yields sij = E[time in j |start in i, ever transit to j ]fij + E[time in j |start in i, never transit to j ](1 − fij ) = (δi,j + sjj )fij + δi,j (1 − fi,j ) = δi,j + fij sjj since sjj is the expected number of additional time periods spent in state j given that it is eventually entered from state i. Solving the preceding equation yields fij =
sij − δi,j sjj
Example 4.27 In Example 4.26, what is the probability that the gambler ever has a fortune of 1? Solution:
Since s3,1 = 1.4206 and s1,1 = 1.6149, then f3,1 =
s3,1 = 0.8797 s1,1
As a check, note that f3,1 is just the probability that a gambler starting with 3 reaches 1 before 7. That is, it is the probability that the gambler’s fortune will go down 2 before going up 4; which is the probability that a gambler starting with 2 will go broke before reaching 6. Therefore, f3,1 = −
1 − (0.6/0.4)2 = 0.8797 1 − (0.6/0.4)6
which checks with our earlier answer.
Suppose we are interested in the expected time until the Markov chain enters some sets of states A, which need not be the set of recurrent states. We can reduce this back to the previous situation by making all states in A absorbing states. That is, reset the transition probabilities of states in A to satisfy. Pi,i = 1,
i∈A
This transforms the states of A into recurrent states, and transforms any state outside of A from which an eventual transition into A is possible into a transient state. Thus, our previous approach can be used.
4.7.
4.7.
Branching Processes
233
Branching Processes
In this section we consider a class of Markov chains, known as branching processes, which have a wide variety of applications in the biological, sociological, and engineering sciences. Consider a population consisting of individuals able to produce offspring of the same kind. Suppose that each individual will, by the end of its lifetime, have produced j new offspring with probability Pj , j 0, independently of the numbers produced by other individuals. We suppose that Pj < 1 for all j 0. The number of individuals initially present, denoted by X0 , is called the size of the zeroth generation. All offspring of the zeroth generation constitute the first generation and their number is denoted by X1 . In general, let Xn denote the size of the nth generation. It follows that {Xn , n = 0, 1, . . .} is a Markov chain having as its state space the set of nonnegative integers. Note that state 0 is a recurrent state, since clearly P00 = 1. Also, if P0 > 0, all other states are transient. This follows since Pi0 = P0i , which implies that starting with i individuals there is a positive probability of at least P0i that no later generation will ever consist of i individuals. Moreover, since any finite set of transient states {1, 2, . . . , n} will be visited only finitely often, this leads to the important conclusion that, if P0 > 0, then the population will either die out or its size will converge to infinity. Let μ=
∞
j Pj
j =0
denote the mean number of offspring of a single individual, and let σ2 =
∞
(j − μ)2 Pj
j =0
be the variance of the number of offspring produced by a single individual. Let us suppose that X0 = 1, that is, initially there is a single individual present. We calculate E[Xn ] and Var(Xn ) by first noting that we may write
Xn−1
Xn =
Zi
i=1
where Zi represents the number of offspring of the ith individual of the (n − 1)st generation. By conditioning on Xn−1 , we obtain
234
4
Markov Chains
E[Xn ] = E[E[Xn |Xn−1 ]] , ,Xn−1 - =E E Zi |Xn−1 i=1
= E[Xn−1 μ] = μE[Xn−1 ] where we have used the fact that E[Zi ] = μ. Since E[X0 ] = 1, the preceding yields E[X1 ] = μ, E[X2 ] = μE[X1 ] = μ2 , .. . E[Xn ] = μE[Xn−1 ] = μn Similarly, Var(Xn ) may be obtained by using the conditional variance formula Var(Xn ) = E[Var(Xn |Xn−1 )] + Var(E[Xn |Xn−1 ]) Now, given Xn−1 , Xn is just the sum of Xn−1 independent random variables each having the distribution {Pj , j 0}. Hence, E[Xn |Xn−1 ] = Xn−1 μ,
Var(Xn |Xn−1 ) = Xn−1 σ 2
The conditional variance formula now yields Var(Xn ) = E[Xn−1 σ 2 ] + Var(Xn−1 μ) = σ 2 μn−1 + μ2 Var(Xn−1 ) = σ 2 μn−1 + μ2 σ 2 μn−2 + μ2 Var(Xn−2 ) = σ 2 (μn−1 + μn ) + μ4 Var(Xn−2 ) = σ 2 (μn−1 + μn ) + μ4 σ 2 μn−3 + μ2 Var(Xn−3 ) = σ 2 (μn−1 + μn + μn+1 ) + μ6 Var(Xn−3 ) = ··· = σ 2 (μn−1 + μn + · · · + μ2n−2 ) + μ2n Var(X0 ) = σ 2 (μn−1 + μn + · · · + μ2n−2 )
4.7.
Therefore,
Var(Xn ) =
σ 2 μn−1
1−μn 1−μ ,
Branching Processes
if μ = 1 if μ = 1
nσ 2 ,
235
(4.19)
Let π0 denote the probability that the population will eventually die out (under the assumption that X0 = 1). More formally, π0 = lim P {Xn = 0|X0 = 1} n→∞
The problem of determining the value of π0 was first raised in connection with the extinction of family surnames by Galton in 1889. We first note that π0 = 1 if μ < 1. This follows since μ = E[Xn ] = n
∞
j P {Xn = j }
j =1
∞
1 · P {Xn = j }
j =1
= P {Xn 1} Since μn → 0 when μ < 1, it follows that P {Xn 1} → 0, and hence P {Xn = 0} → 1. In fact, it can be shown that π0 = 1 even when μ = 1. When μ > 1, it turns out that π0 < 1, and an equation determining π0 may be derived by conditioning on the number of offspring of the initial individual, as follows: π0 = P {population dies out} =
∞
P {population dies out|X1 = j }Pj
j =0
Now, given that X1 = j , the population will eventually die out if and only if each of the j families started by the members of the first generation eventually dies out. Since each family is assumed to act independently, and since the probability that any particular family dies out is just π0 , this yields j
P {population dies out|X1 = j } = π0 and thus π0 satisfies π0 =
∞ j =0
j
π0 Pj
(4.20)
236
4
Markov Chains
In fact when μ > 1, it can be shown that π0 is the smallest positive number satisfying Equation (4.20).
Example 4.28 If P0 = 12 , P1 = 14 , P2 = 14 , then determine π0 . Solution:
Since μ =
3 4
1, it follows that π0 = 1.
Example 4.29 If P0 = 14 , P1 = 14 , P2 = 12 , then determine π0 . Solution:
π0 satisfies π0 =
1 4
+ 14 π0 + 12 π02
or 2π02 − 3π0 + 1 = 0 The smallest positive solution of this quadratic equation is π0 = 12 .
Example 4.30 In Examples 4.28 and 4.29, what is the probability that the population will die out if it initially consists of n individuals? Solution: Since the population will die out if and only if the families of each of the members of the initial generation die out, the desired probability is π0n . For Example 4.28 this yields π0n = 1, and for Example 4.29, π0n = ( 12 )n .
4.8.
Time Reversible Markov Chains
Consider a stationary ergodic Markov chain (that is, an ergodic Markov chain that has been in operation for a long time) having transition probabilities Pij and stationary probabilities πi , and suppose that starting at some time we trace the sequence of states going backward in time. That is, starting at time n, consider the sequence of states Xn , Xn−1 , Xn−2 , . . . . It turns out that this sequence of states is itself a Markov chain with transition probabilities Qij defined by Qij = P {Xm = j |Xm+1 = i} =
P {Xm = j, Xm+1 = i} P {Xm+1 = i}
=
P {Xm = j }P {Xm+1 = i|Xm = j } P {Xm+1 = i}
=
πj Pj i πi
4.8.
Time Reversible Markov Chains
237
To prove that the reversed process is indeed a Markov chain, we must verify that P {Xm = j |Xm+1 = i, Xm+2 , Xm+3 , . . .} = P {Xm = j |Xm+1 = i} To see that this is so, suppose that the present time is m + 1. Now, since X0 , X1 , X2 , . . . is a Markov chain, it follows that the conditional distribution of the future Xm+2 , Xm+3 , . . . given the present state Xm+1 is independent of the past state Xm . However, independence is a symmetric relationship (that is, if A is independent of B, then B is independent of A), and so this means that given Xm+1 , Xm is independent of Xm+2 , Xm+3 , . . . . But this is exactly what we had to verify. Thus, the reversed process is also a Markov chain with transition probabilities given by Qij =
πj Pj i πi
If Qij = Pij for all i, j , then the Markov chain is said to be time reversible. The condition for time reversibility, namely, Qij = Pij , can also be expressed as πi Pij = πj Pj i
for all i, j
(4.21)
The condition in Equation (4.21) can be stated that, for all states i and j , the rate at which the process goes from i to j (namely, πi Pij ) is equal to the rate at which it goes from j to i (namely, πj Pj i ). It is worth noting that this is an obvious necessary condition for time reversibility since a transition from i to j going backward in time is equivalent to a transition from j to i going forward in time; that is, if Xm = i and Xm−1 = j , then a transition from i to j is observed if we are looking backward, and one from j to i if we are looking forward in time. Thus, the rate at which the forward process makes a transition from j to i is always equal to the rate at which the reverse process makes a transition from i to j ; if time reversible, this must equal the rate at which the forward process makes a transition from i to j . If we can find nonnegative numbers, summing to one, that satisfy Equation (4.21), then it follows that the Markov chain is time reversible and the numbers represent the limiting probabilities. This is so since if xi Pij = xj Pj i for all i, j, xi = 1 (4.22) i
then summing over i yields xi Pij = xj Pj i = xj , i
i
xi = 1
i
and, because the limiting probabilities πi are the unique solution of the preceding, it follows that xi = πi for all i.
238
4
Markov Chains
Example 4.31 Consider a random walk with states 0, 1, . . . , M and transition probabilities Pi,i+1 = αi = 1 − Pi,i−1 ,
i = 1, . . . , M − 1,
P0,1 = α0 = 1 − P0,0 , PM,M = αM = 1 − PM,M−1 Without the need for any computations, it is possible to argue that this Markov chain, which can only make transitions from a state to one of its two nearest neighbors, is time reversible. This follows by noting that the number of transitions from i to i + 1 must at all times be within 1 of the number from i + 1 to i. This is so because between any two transitions from i to i +1 there must be one from i +1 to i (and conversely) since the only way to reenter i from a higher state is via state i + 1. Hence, it follows that the rate of transitions from i to i + 1 equals the rate from i + 1 to i, and so the process is time reversible. We can easily obtain the limiting probabilities by equating for each state i = 0, 1, . . . , M − 1 the rate at which the process goes from i to i + 1 with the rate at which it goes from i + 1 to i. This yields π0 α0 = π1 (1 − α1 ), π1 α1 = π2 (1 − α2 ), .. . πi αi = πi+1 (1 − αi+1 ),
i = 0, 1, . . . , M − 1
Solving in terms of π0 yields α0 π0 , 1 − α1 α1 α1 α0 π0 π2 = π1 = 1 − α2 (1 − α2 )(1 − α1 )
π1 =
and, in general, πi = Since
M 0
αi−1 · · · α0 π0 , (1 − αi ) · · · (1 − α1 )
πi = 1, we obtain , π0 1 +
M j =1
i = 1, 2, . . . , M
αj −1 · · · α0 =1 (1 − αj ) · · · (1 − α1 )
4.8.
Time Reversible Markov Chains
239
or , π0 = 1 +
M j =1
αj −1 · · · α0 (1 − αj ) · · · (1 − α1 )
-−1 (4.23)
and πi =
αi−1 · · · α0 π0 , (1 − αi ) · · · (1 − α1 )
i = 1, . . . , M
(4.24)
For instance, if αi ≡ α, then ,
j M α π0 = 1 + 1−α
-−1
j =1
=
1−β 1 − β M+1
and, in general, πi =
β i (1 − β) , 1 − β M+1
i = 0, 1, . . . , M
where β=
α 1−α
Another special case of Example 4.31 is the following urn model, proposed by the physicists P. and T. Ehrenfest to describe the movements of molecules. Suppose that M molecules are distributed among two urns; and at each time point one of the molecules is chosen at random, removed from its urn, and placed in the other one. The number of molecules in urn I is a special case of the Markov chain of Example 4.31 having αi =
M −i , M
i = 0, 1, . . . , M
240
4
Markov Chains
Hence, using Equations (4.23) and (4.24) the limiting probabilities in this case are , -−1 M (M − j + 1) · · · (M − 1)M π0 = 1 + j (j − 1) · · · 1 j =1
, =
M −1 M j =0
=
j
M 1 2
where we have used the identity 1 1 M 1= + 2 2 M M M 1 = j 2
j =0
Hence, from Equation (4.24) M M 1 , πi = 2 i
i = 0, 1, . . . , M
Because the preceding are just the binomial probabilities, it follows that in the long run, the positions of each of the M balls are independent and each one is equally likely to be in either urn. This, however, is quite intuitive, for if we focus on any one ball, it becomes quite clear that its position will be independent of the positions of the other balls (since no matter where the other M − 1 balls are, the ball under consideration at each stage will be moved with probability 1/M) and by symmetry, it is equally likely to be in either urn.
Example 4.32 Consider an arbitrary connected graph (see Section 3.6 for definitions) having a number wij associated with arc (i, j ) for each arc. One instance of such a graph is given by Figure 4.1. Now consider a particle moving from node to node in this manner: If at any time the particle resides at node i, then it will next move to node j with probability Pij where wij Pij = j wij and where wij is 0 if (i, j ) is not an arc. For instance, for the graph of Figure 4.1, P12 = 3/(3 + 1 + 2) = 12 .
4.8.
Time Reversible Markov Chains
241
Figure 4.1. A connected graph with arc weights.
The time reversibility equations πi Pij = πj Pj i reduce to wij wj i πi = πj j wij i wj i or, equivalently, since wij = wj i πj πi = j wij i wj i
which is equivalent to
πi =c j wij
or
πi = c
wij
j
or, since 1 =
i
πi
j wij πi = i j wij
Because the πi s given by this equation satisfy the time reversibility equations, it follows that the process is time reversible with these limiting probabilities. For the graph of Figure 4.1 we have that π1 =
6 32 ,
π2 =
3 32 ,
π3 =
6 32 ,
π4 =
5 32 ,
π5 =
12 32
242
4
Markov Chains
If we try to solve Equation (4.22) for an arbitrary Markov chain with states 0, 1, . . . , M, it will usually turn out that no solution exists. For example, from Equation (4.22), xi Pij = xj Pj i , xk Pkj = xj Pj k implying (if Pij Pj k > 0) that Pj i Pkj xi = xk Pij Pj k which in general need not equal Pki /Pik . Thus, we see that a necessary condition for time reversibility is that Pik Pkj Pj i = Pij Pj k Pki
for all i, j, k
(4.25)
which is equivalent to the statement that, starting in state i, the path i → k → j → i has the same probability as the reversed path i → j → k → i. To understand the necessity of this, note that time reversibility implies that the rate at which a sequence of transitions from i to k to j to i occurs must equal the rate of ones from i to j to k to i (why?), and so we must have πi Pik Pkj Pj i = πi Pij Pj k Pki implying Equation (4.25) when πi > 0. In fact, we can show the following.
Theorem 4.2 An ergodic Markov chain for which Pij = 0 whenever Pj i = 0 is time reversible if and only if starting in state i, any path back to i has the same probability as the reversed path. That is, if Pi,i1 Pi1 ,i2 · · · Pik ,i = Pi,ik Pik ,ik−1 · · · Pi1 ,i
(4.26)
for all states i, i1 , . . . , ik .
Proof We have already proven necessity. To prove sufficiency, fix states i and j and rewrite (4.26) as Pi,i1 Pi1 ,i2 · · · Pik ,j Pj i = Pij Pj,ik · · · Pi1 ,i Summing the preceding over all states i1 , . . . , ik yields Pijk+1 Pj i = Pij Pjk+1 i Letting k → ∞ yields πj Pj i = Pij πi which proves the theorem.
4.8.
Time Reversible Markov Chains
243
Example 4.33 Suppose we are given a set of n elements, numbered 1 through n, which are to be arranged in some ordered list. At each unit of time a request is made to retrieve one of these elements, element i being requested (independently of the past) with probability Pi . After being requested, the element then is put back but not necessarily in the same position. In fact, let us suppose that the element requested is moved one closer to the front of the list; for instance, if the present list ordering is 1, 3, 4, 2, 5 and element 2 is requested, then the new ordering becomes 1, 3, 2, 4, 5. We are interested in the long-run average position of the element requested. For any given probability vector P = (P1 , . . . , Pn ), the preceding can be modeled as a Markov chain with n! states, with the state at any time being the list order at that time. We shall show that this Markov chain is time reversible and then use this to show that the average position of the element requested when this one-closer rule is in effect is less than when the rule of always moving the requested element to the front of the line is used. The time reversibility of the resulting Markov chain when the one-closer reordering rule is in effect easily follows from Theorem 4.2. For instance, suppose n = 3 and consider the following path from state (1, 2, 3) to itself: (1, 2, 3) → (2, 1, 3) → (2, 3, 1) → (3, 2, 1) → (3, 1, 2) → (1, 3, 2) → (1, 2, 3) The product of the transition probabilities in the forward direction is P2 P3 P3 P1 P1 P2 = P12 P22 P32 whereas in the reverse direction, it is P3 P3 P2 P2 P1 P1 = P12 P22 P32 Because the general result follows in much the same manner, the Markov chain is indeed time reversible. [For a formal argument note that if fi denotes the number of times element i moves forward in the path, then as the path goes from a fixed state back to itself, it follows that element i will also move backward fi times. Therefore, since the backward moves of element i are precisely the times that it moves forward in the reverse path, it follows that the product of the transition probabilities for both the path and its reversal will equal f +r Pi i i i
where ri is equal to the number of times that element i is in the first position and the path (or the reverse path) does not change states.]
244
4
Markov Chains
For any permutation i1 , i2 , . . . , in of 1, 2, . . . , n, let π(i1 , i2 , . . . , in ) denote the limiting probability under the one-closer rule. By time reversibility we have Pij +1 π(i1 , . . . , ij , ij +1 , . . . , in ) = Pij π(i1 , . . . , ij +1 , ij , . . . , in )
(4.27)
for all permutations. Now the average position of the element requested can be expressed (as in Section 3.6.1) as Pi E[Position of element i] Average position = i
=
i
=1+
. / Pi 1 + P {element j precedes element i} i
=1+
j =i
Pi P {ej precedes ei }
j =i
[Pi P {ej precedes ei } + Pj P {ei precedes ej }]
i<j
=1+
[Pi P {ej precedes ei } + Pj (1 − P {ej precedes ei })]
i<j
=1+
(Pi − Pj )P {ej precedes ei } +
i<j
Pj
i<j
Hence, to minimize the average position of the element requested, we would want to make P {ej precedes ei } as large as possible when Pj > Pi and as small as possible when Pi > Pj . Now under the front-of-the-line rule we showed in Section 3.6.1 that P {ej precedes ei } =
Pj Pj + Pi
(since under the front-of-the-line rule element j will precede element i if and only if the last request for either i or j was for j ). Therefore, to show that the one-closer rule is better than the front-of-the-line rule, it suffices to show that under the one-closer rule P {ej precedes ei } >
Pj Pj + Pi
when Pj > Pi
Now consider any state where element i precedes element j , say, (. . . , i, i1 , . . . , ik , j, . . .). By successive transpositions using Equation (4.27),
4.8.
Time Reversible Markov Chains
245
we have π(. . . , i, i1 , . . . , ik , j, . . .) =
Pi Pj
k+1 π(. . . , j, i1 , . . . , ik , i, . . .)
(4.28)
For instance, P2 P1 P2 π(1, 3, 2) = π(3, 1, 2) P3 P3 P3 2 P2 P1 P1 P1 = π(3, 2, 1) = π(3, 2, 1) P3 P3 P2 P3
π(1, 2, 3) =
Now when Pj > Pi , Equation (4.28) implies that π(. . . , i, i1 , . . . , ik , j, . . .)
1 i1
To check this, and at the same time determine the stationary probabilities, we must see if we can find, with the Qi,j as previously given, positive numbers {πi } such that πi Pi,j = πj Qj,i To begin, let j = 1 and consider the resulting equations: πi Pi,1 = π1 Q1,i This is equivalent to πi
P {L = i} = π1 P {L = i} P {L i}
4.9.
Markov Chain Monte Carlo Methods
247
or πi = π1 P {L i} Summing over all i yields 1=
∞
πi = π1
i=1
∞
P {L i} = π1 E[L]
i=1
and so, for the preceding Qij to represent the reverse transition probabilities, it is necessary for the stationary probabilities to be πi =
P {L i} , E[L]
i1
To finish the proof that the reverse transition probabilities and stationary probabilities are as given, all that remains is to show that they satisfy πi Pi,i+1 = πi+1 Qi+1,i which is equivalent to P {L = i} P {L i + 1} P {L i} 1− = E[L] P {L i} E[L] and which is true since P {L i} − P {L = i} = P {L i + 1}.
4.9.
Markov Chain Monte Carlo Methods
Let X be a discrete random vector whose set of possible values is xj , j 1. Let the probability mass function of X be given by P {X = xj }, j 1, and suppose that we are interested in calculating θ = E[h(X)] =
∞
h(xj )P {X = xj }
j =1
for some specified function h. In situations where it is computationally difficult to evaluate the function h(xj ), j 1, we often turn to simulation to approximate θ . The usual approach, called Monte Carlo simulation, is to use random numbers to generate a partial sequence of independent and identically distributed random vectors X1 , X2 , . . . , Xn having the mass function P {X = xj }, j 1 (see Chap-
248
4
Markov Chains
ter 11 for a discussion as to how this can be accomplished). Since the strong law of large numbers yields lim
n→∞
n h(Xi ) i=1
n
=θ
(4.30)
it follows that we can estimate θ by letting n be large and using the average of the values of h(Xi ), i = 1, . . . , n as the estimator. It often, however, turns out that it is difficult to generate a random vector having the specified probability mass function, particularly if X is a vector of dependent random variables. In addition, its probability mass function is sometimes given in the form P {X = xj } = Cbj , j 1, where the bj are specified, but C must be computed, and in many applications it is not computationally feasible to sum the bj so as to determine C. Fortunately, however, there is another way of using simulation to estimate θ in these situations. It works by generating a sequence, not of independent random vectors, but of the successive states of a vector-valued Markov chain X1 , X2 , . . . whose stationary probabilities are P {X = xj }, j 1. If this can be accomplished, then it would follow from Proposition 4.3 that Equation (4.30) remains valid, implying that we can then use ni=1 h(Xi )/n as an estimator of θ . We now show how to generate a Markov chain with arbitrary stationary probabilities that may only be specified up to a multiplicative constant. Let b(j ), j = 1, 2, . . . be positive numbers whose sum B = ∞ b(j ) is finite. The folj =1 lowing, known as the Hastings–Metropolis algorithm, can be used to generate a time reversible Markov chain whose stationary probabilities are π(j ) = b(j )/B,
j = 1, 2, . . .
To begin, let Q be any specified irreducible Markov transition probability matrix on the integers, with q(i, j ) representing the row i column j element of Q. Now define a Markov chain {Xn , n 0} as follows. When Xn = i, generate a random variable Y such that P {Y = j } = q(i, j ), j = 1, 2, . . . . If Y = j , then set Xn+1 equal to j with probability α(i, j ), and set it equal to i with probability 1 −α(i, j ). Under these conditions, it is easy to see that the sequence of states constitutes a Markov chain with transition probabilities Pi,j given by Pi,j = q(i, j )α(i, j ), if j = i q(i, k)(1 − α(i, k)) Pi,i = q(i, i) + k=i
This Markov chain will be time reversible and have stationary probabilities π(j ) if π(i)Pi,j = π(j )Pj,i
for j = i
4.9.
Markov Chain Monte Carlo Methods
249
which is equivalent to π(i)q(i, j )α(i, j ) = π(j )q(j, i)α(j, i)
(4.31)
But if we take πj = b(j )/B and set π(j )q(j, i) α(i, j ) = min ,1 π(i)q(i, j )
(4.32)
then Equation (4.31) is easily seen to be satisfied. For if α(i, j ) =
π(j )q(j, i) π(i)q(i, j )
then α(j, i) = 1 and Equation (4.31) follows, and if α(i,j ) = 1 then α(j, i) =
π(i)q(i, j ) π(j )q(j, i)
and again Equation (4.31) holds, thus showing that the Markov chain is time reversible with stationary probabilities π(j ). Also, since π(j ) = b(j )/B, we see from (4.32) that b(j )q(j, i) α(i, j ) = min ,1 b(i)q(i, j ) which shows that the value of B is not needed to define the Markov chain, because the values b(j ) suffice. Also, it is almost always the case that π(j ), j 1 will not only be stationary probabilities but will also be limiting probabilities. (Indeed, a sufficient condition is that Pi,i > 0 for some i.)
Example 4.35 Suppose that we want to generate a uniformly distributed ele-
ment inS , the set of all permutations (x1 , . . . , xn ) of the numbers (1, . . . , n) for which nj=1 j xj > a for a given constant a. To utilize the Hastings–Metropolis algorithm we need to define an irreducible Markov transition probability matrix on the state space S . To accomplish this, we first define a concept of “neighboring” elements of S , and then construct a graph whose vertex set is S . We start by putting an arc between each pair of neighboring elements in S , where any two permutations in S are said to be neighbors if one results from an interchange of two of the positions of the other. That is, (1, 2, 3, 4) and (1, 2, 4, 3) are neighbors whereas (1, 2, 3, 4) and (1, 3, 4, 2) are not. Now, define the q transition probability function as follows. With N (s) defined as the set of neighbors of s, and |N (s)|
250
4
Markov Chains
equal to the number of elements in the set N (s), let q(s, t) =
1 |N (s)|
if t ∈ N (s)
That is, the candidate next state from s is equally likely to be any of its neighbors. Since the desired limiting probabilities of the Markov chain are π(s) = C, it follows that π(s) = π(t), and so α(s, t) = min(|N (s)|/|N (t)|, 1) That is, if the present state of the Markov chain is s then one of its neighbors is randomly chosen, say, t. If t is a state with fewer neighbors than s (in graph theory language, if the degree of vertex t is less than that of vertex s), then the next state is t. If not, a uniform (0,1) random number U is generated and the next state is t if U < |(N (s)|/|N (t)| and is s otherwise. The limiting probabilities of this Markov chain are π(s) = 1/|S |, where |S | is the (unknown) number of permutations in S . The most widely used version of the Hastings–Metropolis algorithm is the Gibbs sampler. Let X = (X1 , . . . , Xn ) be a discrete random vector with probability mass function p(x) that is only specified up to a multiplicative constant, and suppose that we want to generate a random vector whose distribution is that of X. That is, we want to generate a random vector having mass function p(x) = Cg(x) where g(x) is known, but C is not. Utilization of the Gibbs sampler assumes that for any i and values xj , j = i, we can generate a random variable X having the probability mass function P {X = x} = P {Xi = x|Xj = xj , j = i} It operates by using the Hasting–Metropolis algorithm on a Markov chain with states x = (x1 , . . . , xn ), and with transition probabilities defined as follows. Whenever the present state is x, a coordinate that is equally likely to be any of 1, . . . , n is chosen. If coordinate i is chosen, then a random variable X with probability mass function P {X = x} = P {Xi = x|Xj = xj , j = i} is generated. If X = x, then the state y = (x1 , . . . xi−1 , x, xi+1 , . . . , xn ), is considered as the candidate next state. In other words, with x and y as given, the Gibbs sampler uses
4.9.
Markov Chain Monte Carlo Methods
251
the Hastings–Metropolis algorithm with 1 p(y) q(x, y) = P {Xi = x|Xj = xj , j = i} = n nP {Xj = xj , j = i} Because we want the limiting mass function to be p, we see from Equation (4.32) that the vector y is then accepted as the new state with probability p(y)q(y, x) α(x, y) = min ,1 p(x)q(x, y) p(y)p(x) ,1 = min p(x)p(y) =1 Hence, when utilizing the Gibbs sampler, the candidate state is always accepted as the next state of the chain.
Example 4.36 Suppose that we want to generate n uniformly distributed points in the circle of radius 1 centered at the origin, conditional on the event that no two points are within a distance d of each other, when the probability of this conditioning event is small. This can be accomplished by using the Gibbs sampler as follows. Start with any n points x1 , . . . , xn in the circle that have the property that no two of them are within d of the other; then generate the value of I , equally likely to be any of the values 1, . . . , n. Then continually generate a random point in the circle until you obtain one that is not within d of any of the other n − 1 points excluding xI . At this point, replace xI by the generated point and then repeat the operation. After a large number of iterations of this algorithm, the set of n points will approximately have the desired distribution. Example 4.37 Let Xi , i = 1, . . . , n, be independent exponential random vari-
ables with respective rates λi , i = 1, . . . , n. Let S = ni=1 Xi , and suppose that we want to generate the random vector X = (X1 , . . . , Xn ), conditional on the event that S > c for some large positive constant c. That is, we want to generate the value of a random vector whose density function is 1 λi e−λi xi , P {S > c} n
f (x1 , . . . , xn ) =
i=1
xi 0,
n
xi > c
i=1
This is easily accomplished bystarting with an initial vector x = (x1 , . . . , xn ) satisfying xi > 0, i = 1, . . . , n, ni=1 xi > c. Then generate a random variable I that is equally likely to be any of 1, . . . , n. Next, generate an exponential random
252
4
Markov Chains
variable X with rate λI conditional on the event that X + j =I xj > c. This latter step, which calls for generating the value of an exponential random variable given that it exceeds c − j =I xj , is easily accomplished by using the fact that an exponential conditioned to be greater than a positive constant is distributed as the constant plus the exponential. Consequently, to obtain X, first generate an exponential random variable Y with rate λI , and then set + X=Y + c− xj j =I
The value of xI should then be reset as X and a new iteration of the algorithm begun.
Remark As can be seen by Examples 4.36 and 4.37, although the theory for the Gibbs sampler was represented under the assumption that the distribution to be generated was discrete, it also holds when this distribution is continuous.
4.10.
Markov Decision Processes
Consider a process that is observed at discrete time points to be in any one of M possible states, which we number by 1, 2, . . . , M. After observing the state of the process, an action must be chosen, and we let A, assumed finite, denote the set of all possible actions. If the process is in state i at time n and action a is chosen, then the next state of the system is determined according to the transition probabilities Pij (a). If we let Xn denote the state of the process at time n and an the action chosen at time n, then the preceding is equivalent to stating that P {Xn+1 = j |X0 , a0 , X1 , a1 , . . . , Xn = i, an = a} = Pij (a) Thus, the transition probabilities are functions only of the present state and the subsequent action. By a policy, we mean a rule for choosing actions. We shall restrict ourselves to policies which are of the form that the action they prescribe at any time depends only on the state of the process at that time (and not on any information concerning prior states and actions). However, we shall allow the policy to be “randomized” in that its instructions may be to choose actions according to a probability distribution. In other words, a policy β is a set of numbers β = {βi (a), a ∈ A, i = 1, . . . , M} with the interpretation that if the process is in
4.10.
Markov Decision Processes
253
state i, then action a is to be chosen with probability βi (a). Of course, we need have that 0 βi (a) 1, βi (a) = 1,
for all i, a for all i
a
Under any given policy β, the sequence of states {Xn , n = 0, 1, . . .} constitutes a Markov chain with transition probabilities Pij (β) given by Pij (β) = Pβ {Xn+1 = j |Xn = i}∗ = Pij (a)βi (a) a
where the last equality follows by conditioning on the action chosen when in state i. Let us suppose that for every choice of a policy β, the resultant Markov chain {Xn , n = 0, 1, . . .} is ergodic. For any policy β, let πia denote the limiting (or steady-state) probability that the process will be in state i and action a will be chosen if policy β is employed. That is, πia = lim Pβ {Xn = i, an = a} n→∞
The vector π = (πia ) must satisfy (i) πia 0 for all i, a, (ii) π = 1, i a ia (iii) π = i a πia Pij (a) for all j j a a
(4.33)
Equations (i) and (ii) are obvious, and Equation (iii) which is an analogue of Equation (4.7) follows as the left-hand side equals the steady-state probability of being in state j and the right-hand side is the same probability computed by conditioning on the state and action chosen one stage earlier. Thus for any policy β, there is a vector π = (πia ) which satisfies (i)–(iii) and with the interpretation that πia is equal to the steady-state probability of being in state i and choosing action a when policy β is employed. Moreover, it turns out that the reverse is also true. Namely, for any vector π = (πia ) which satisfies (i)–(iii), there exists a policy β such that if β is used, then the steady-state probability of being in i and choosing action a equals πia . To verify this last statement, ∗ We use the notation P to signify that the probability is conditional on the fact that policy β is β
used.
254
4
Markov Chains
suppose that π = (πia ) is a vector which satisfies (i)–(iii). Then, let the policy β = (βi (a)) be βi (a) = P {β chooses a|state is i} πia = a πia Now let Pia denote the limiting probability of being in i and choosing a when policy β is employed. We need to show that Pia = πia . To do so, first note that {Pia , i = 1, . . . , M, a ∈ A} are the limiting probabilities of the two-dimensional Markov chain {(Xn , an ), n 0}. Hence, by the fundamental Theorem 4.1, they are the unique solution of (i ) Pia 0, (ii ) Pia = 1, i a (iii ) Pj a = i a Pia Pij (a )βj (a) where (iii ) follows since P {Xn+1 = j, an+1 = a|Xn = i, an = a } = Pij (a )βj (a) Since πj a βj (a) = a πj a we see that (Pia ) is the unique solution of Pia 0,
i
Pia = 1,
a
Pj a =
i
a
πj a Pia Pij (a ) a πj a
Hence, to show that Pia = πia , we need show that i
πia 0, πia = 1,
a
πj a =
i
a
πj a πia Pij (a ) a πj a
4.10.
Markov Decision Processes
255
The top two equations follow from (i) and (ii) of Equation (4.33), and the third, which is equivalent to πj a = πia Pij (a ) a
a
i
follows from condition (iii) of Equation (4.33). Thus we have shown that a vector β = (πia ) will satisfy (i), (ii), and (iii) of Equation (4.33) if and only if there exists a policy β such that πia is equal to the steady-state probability of being in state i and choosing action a when β is used. In fact, the policy β is defined by β i (a) = πia / a πia . The preceding is quite important in the determination of “optimal” policies. For instance, suppose that a reward R(i, a) is earned whenever action a is chosen in state i. Since R(Xi , ai ) would then represent the reward earned at time i, the expected average reward per unit time under policy β can be expressed as . n / i=1 R(Xi , ai ) expected average reward under β = lim Eβ n→∞ n Now, if πia denotes the steady-state probability of being in state i and choosing action a, it follows that the limiting expected reward at time n equals lim E[R(Xn , an )] = πia R(i, a) n→∞
i
a
which implies that expected average reward under β =
i
πia R(i, a)
a
Hence, the problem of determining the policy that maximizes the expected average reward is maximize π=(πia )
i
πia R(i, a)
a
subject to πia 0, πia = 1, i
a
a
πj a =
for all i, a,
i
πia Pij (a),
for all j
(4.34)
a
However, the preceding maximization problem is a special case of what is known as a linear program and can be solved by a standard linear programming algorithm
256
4
Markov Chains
∗ ) maximizes the preceding, then the known as the simplex algorithm.∗ If β ∗ = (πia ∗ optimal policy will be given by β where
π∗ βi∗ (a) = ia ∗ a πia
Remarks (i) It can be shown that there is a π ∗ maximizing Equation (4.34)
∗ is zero for all but one value of a, which that has the property that for each i, πia implies that the optimal policy is nonrandomized. That is, the action it prescribes when in state i is a deterministic function of i. (ii) The linear programming formulation also often works when there are restrictions placed on the class of allowable policies. For instance, suppose there is a restriction on the fraction of time the process spends in some state, say, state 1. Specifically, suppose that we are allowed to consider only policies having the property that their use results in the process being in state 1 less than 100α percent of time. To determine the optimal policy subject to this requirement, we add to the linear programming problem the additional constraint π1a α
since
4.11.
a
a π1a
represents the proportion of time that the process is in state 1.
Hidden Markov Chains
Let {Xn , n = 1, 2, . . .} be a Markov chain with transition probabilities Pi,j and initial state probabilities pi = P {X1 = i}, i 0. Suppose that there is a finite set S of signals, and that a signal from S is emitted each time the Markov chain enters a state. Further, suppose that when the Markov chain enters state j then, independently of previous Markov chain states and signals, the signal emitted is s with probability p(s|j ), s∈S p(s|j ) = 1. That is, if Sn represents the nth signal emitted, then P {S1 = s|X1 = j } = p(s|j ), P {Sn = s|X1 , S1 , . . . , Xn−1 , Sn−1 , Xn = j } = p(s|j ) A model of the preceding type in which the sequence of signals S1 , S2 , . . . is observed, while the sequence of underlying Markov chain states X1 , X2 , . . . is unobserved, is called a hidden Markov chain model. ∗ It is called a linear program since the objective function R(i, a)π and the constraints are ia i a all linear functions of the πia . For a heuristic analysis of the simplex algorithm, see 4.5.2.
4.11.
Hidden Markov Chains
257
Example 4.38 Consider a production process that in each period is either in a good state (state 1) or in a poor state (state 2). If the process is in state 1 during a period then, independent of the past, with probability 0.9 it will be in state 1 during the next period and with probability 0.1 it will be in state 2. Once in state 2, it remains in that state forever. Suppose that a single item is produced each period and that each item produced when the process is in state 1 is of acceptable quality with probability 0.99, while each item produced when the process is in state 2 is of acceptable quality with probability 0.96. If the status, either acceptable or unacceptable, of each successive item is observed, while the process states are unobservable, then the preceding is a hidden Markov chain model. The signal is the status of the item produced, and has value either a or u, depending on whether the item is acceptable or unacceptable. The signal probabilities are p(u|1) = 0.01,
p(a|1) = 0.99,
p(u|2) = 0.04,
p(a|2) = 0.96
while the transition probabilities of the underlying Markov chain are P1,1 = 0.9 = 1 − P1,2 ,
P2,2 = 1
Although {Sn , n 1} is not a Markov chain, it should be noted that, conditional on the current state Xn , the sequence Sn , Xn+1 , Sn+1 , . . . of future signals and states is independent of the sequence X1 , S1 , . . . , Xn−1 , Sn−1 of past states and signals. Let Sn = (S1 , . . . , Sn ) be the random vector of the first n signals. For a fixed sequence of signals s1 , . . . , sn , let sk = (s1 , . . . , sk ), k n. To begin, let us determine the conditional probability of the Markov chain state at time n given that Sn = sn . To obtain this probability, let Fn (j ) = P {Sn = sn , Xn = j } and note that P {Xn = j |Sn = sn } =
P {Sn = sn , Xn = j } P {Sn = sn }
Fn (j ) = i Fn (i)
258
4
Markov Chains
Now, Fn (j ) = P {Sn−1 = sn−1 , Sn = sn , Xn = j } P {Sn−1 = sn−1 , Xn−1 = i, Xn = j, Sn = sn } = i
=
Fn−1 (i)P {Xn = j, Sn = sn |Sn−1 = sn−1 , Xn−1 = i}
i
=
Fn−1 (i)P {Xn = j, Sn = sn |Xn−1 = i}
i
=
Fn−1 (i)Pi,j p(sn |j )
i
= p(sn |j )
Fn−1 (i)Pi,j
(4.35)
i
where the preceding used that P {Xn = j, Sn = sn |Xn−1 = i} = P {Xn = j |Xn−1 = i} × P {Sn = sn |Xn = j, Xn−1 = i} = Pi,j P {Sn = sn |Xn = j } = Pi,j p(sn |j ) Starting with F1 (i) = P {X1 = i, S1 = s1 } = pi p(s1 |i) we can use Equation (4.35) to recursively determine the functions F2 (i), F3 (i), . . . , up to Fn (i).
Example 4.39 Suppose in Example 4.38 that P {X1 = 1} = 0.8. Given that the successive conditions of the first 3 items produced are a, u, a, (i) what is the probability that the process was in its good state when the third item was produced; (ii) what is the probability that X4 is 1; (iii) what is the probability that the next item produced is acceptable?
4.11.
Hidden Markov Chains
259
With s3 = (a, u, a), we have
Solution:
F1 (1) = (0.8)(0.99) = 0.792, F1 (2) = (0.2)(0.96) = 0.192 F2 (1) = 0.01[0.792(0.9) + 0.192(0)] = 0.007128, F2 (2) = 0.04[0.792(0.1) + (0.192)(1)] = 0.010848 F3 (1) = 0.99[(0.007128)(0.9)] ≈ 0.006351, F3 (2) = 0.96[(0.007128)(0.1) + 0.010848] ≈ 0.011098 Therefore, the answer to part (i) is 0.006351 ≈ 0.364 0.006351 + 0.011098 To compute P {X4 = 1|s3 }, condition on X3 to obtain P {X3 = 1|s3 } ≈
P {X4 = 1|s3 } = P {X4 = 1|X3 = 1, s3 }P {X3 = 1|s3 } + P {X4 = 1|X3 = 2, s3 }P {X3 = 2|s3 } = P {X4 = 1|X3 = 1, s3 }(0.364) + P {X4 = 1|X3 = 2, s3 }(0.636) = 0.364P1,1 + 0.636P2,1 = 0.3276 To compute P {S4 = a|s3 }, condition on X4 P {S4 = a|s3 } = P {S4 = a|X4 = 1, s3 }P {X4 = 1|s3 } + P {S4 = a|X4 = 2, s3 }P {X4 = 2|s3 } = P {S4 = a|X4 = 1}(0.3276) + P {S4 = a|X4 = 2}(1 − 0.3276) = (0.99)(0.3276) + (0.96)(0.6724) = 0.9698 To compute P {Sn = sn }, use the identity P {Sn = sn } = i Fn (i) along with the recursion (4.35). If there are N states of the Markov chain, this requires computing nN quantities Fn (i), with each computation requiring a summation over N terms. This can be compared with a computation of P {Sn = sn } based on conditioning on the first n states of the Markov chain to obtain P {Sn = sn } = P {Sn = sn |X1 = i1 , . . . , Xn = in }P {X1 = i1 , . . . , Xn = in } i1 ,...,in
=
i1 ,...,in
p(s1 |i1 ) · · · p(sn |in )pi1 Pi1 ,i2 Pi2 ,i3 · · · Pin−1 ,in
260
4
Markov Chains
The use of the preceding identity to compute P {Sn = sn } would thus require a summation over N n terms, with each term being a product of 2n values, indicating that it is not competitive with the previous approach. The computation of P {Sn = sn } by recursively determining the functions Fk (i) is known as the forward approach. There also is a backward approach, which is based on the quantities Bk (i), defined by Bk (i) = P {Sk+1 = sk+1 , . . . , Sn = sn |Xk = i} A recursive formula for Bk (i) can be obtained by conditioning on Xk+1 . Bk (i) =
P {Sk+1 = sk+1 , . . . , Sn = sn |Xk = i, Xk+1 = j }P {Xk+1 = j |Xk = i}
j
=
P {Sk+1 = sk+1 , . . . , Sn = sn |Xk+1 = j }Pi,j
j
=
P {Sk+1 = sk+1 |Xk+1 = j }
j
× P {Sk+2 = sk+2 , . . . , Sn = sn |Sk+1 = sk+1 , Xk+1 = j }Pi,j = p(sk+1 |j )P {Sk+2 = sk+2 , . . . , Sn = sn |Xk+1 = j }Pi,j j
=
p(sk+1 |j )Bk+1 (j )Pi,j
(4.36)
j
Starting with Bn−1 (i) = P {Sn = sn |Xn−1 = i} Pi,j p(sn |j ) = j
we would then use Equation (4.36) to determine the function Bn−2 (i), then Bn−3 (i), and so on, down to B1 (i). This would then yield P {Sn = sn } via P {Sn = sn } =
P {S1 = s1 , . . . , Sn = sn |X1 = i}pi
i
=
i
P {S1 = s1 |X1 = i}P {S2 = s2 , . . . , Sn = sn |S1 = s1 , X1 = i}pi
4.11.
=
Hidden Markov Chains
261
p(s1 |i)P {S2 = s2 , . . . , Sn = sn |X1 = i}pi
i
=
p(s1 |i)B1 (i)pi
i
Another approach to obtaining P {Sn = sn } is to combine both the forward and backward approaches. Suppose that for some k we have computed both functions Fk (j ) and Bk (j ). Because P {Sn = sn , Xk = j } = P {Sk = sk , Xk = j } × P {Sk+1 = sk+1 , . . . , Sn = sn |Sk = sk , Xk = j } = P {Sk = sk , Xk = j }P {Sk+1 = sk+1 , . . . , Sn = sn |Xk = j } = Fk (j )Bk (j ) we see that P {Sn = sn } =
Fk (j )Bk (j )
j
The beauty of using the preceding identity to determine P {Sn = sn } is that we may simultaneously compute the sequence of forward functions, starting with F1 , as well as the sequence of backward functions, starting at Bn−1 . The parallel computations can then be stopped once we have computed both Fk and Bk for some k. 4.11.1. Predicting the States Suppose the first n observed signals are sn = (s1 , . . . , sn ), and that given this data we want to predict the first n states of the Markov chain. The best predictor depends on what we are trying to accomplish. If our objective is to maximize the expected number of states that are correctly predicted, then for each k = 1, . . . , n we need to compute P {Xk = j |Sn = sn } and then let the value of j that maximizes this quantity be the predictor of Xk . (That is, we take the mode of the conditional probability mass function of Xk , given the sequence of signals, as the predictor of Xk .) To do so, we must first compute this conditional probability mass function, which is accomplished as follows. For k n P {Xk = j |Sn = sn } =
P {Sn = sn , Xk = j } P {Sn = sn }
Fk (j )Bk (j ) = j Fk (j )Bk (j ) Thus, given that Sn = sn , the optimal predictor of Xk is the value of j that maximizes Fk (j )Bk (j ).
262
4
Markov Chains
A different variant of the prediction problem arises when we regard the sequence of states as a single entity. In this situation, our objective is to choose that sequence of states whose conditional probability, given the sequence of signals, is maximal. For instance, in signal processing, while X1 , . . . , Xn might be the actual message sent, S1 , . . . , Sn would be what is received, and so the objective would be to predict the actual message in its entirety. Letting Xk = (X1 , . . . , Xk ) be the vector of the first k states, the problem of interest is to find the sequence of states i1 , . . . , in that maximizes P {Xn = (i1 , . . . , in )|Sn = sn }. Because P {Xn = (i1 , . . . , in )|Sn = sn } =
P {Xn = (i1 , . . . , in ), Sn = sn } P {Sn = ss }
this is equivalent to finding the sequence of states i1 , . . . , in that maximizes P {Xn = (i1 , . . . , in ), Sn = sn }. To solve the preceding problem let, for k n, Vk (j ) = max P {Xk−1 = (i1 , . . . , ik−1 ), Xk = j, Sk = sk } i1 ,...,ik−1
To recursively solve for Vk (j ), use that Vk (j ) = max max P {Xk−2 = (i1 , . . . , ik−2 ), Xk−1 = i, Xk = j, Sk = sk } i
i1 ,...,ik−2
= max max P {Xk−2 = (i1 , . . . , ik−2 ), Xk−1 = i, Sk−1 = sk−1 , i
i1 ,...,ik−2
Xk = j, Sk = sk } = max max P {Xk−2 = (i1 , . . . , ik−2 ), Xk−1 = i, Sk−1 = sk−1 } i
i1 ,...,ik−2
× P {Xk = j, Sk = sk |Xk−2 = (i1 , . . . , ik−2 ), Xk−1 = i, Sk−1 = sk−1 } = max max P {Xk−2 = (i1 , . . . , ik−2 ), Xk−1 = i, Sk−1 = sk−1 } i
i1 ,...,ik−2
× P {Xk = j, Sk = sk |Xk−1 = i} = max P {Xk = j, Sk = sk |Xk−1 = i} i
× max P {Xk−2 = (i1 , . . . , ik−2 ), Xk−1 = i, Sk−1 = sk−1 } i1 ,...,ik−2
= max Pi,j p(sk |j )Vk−1 (i) i
= p(sk |j ) max Pi,j Vk−1 (i) i
(4.37)
Exercises
263
Starting with V1 (j ) = P {X1 = j, S1 = s1 } = pj p(s1 |j ) we now use the recursive identity (4.37) to determine V2 (j ) for each j ; then V3 (j ) for each j ; and so on, up to Vn (j ) for each j . To obtain the maximizing sequence of states, we work in the reverse direction. Let jn be the value (or any of the values if there are more than one) of j that maximizes Vn (j ). Thus jn is the final state of a maximizing state sequence. Also, for k < n, let ik (j ) be a value of i that maximizes Pi,j Vk (i). Then max P {Xn = (i1 , . . . , in ), Sn = sn }
i1 ,...,in
= max Vn (j ) j
= Vn (jn ) = max P {Xn = (i1 , . . . , in−1 , jn ), Sn = sn } i1 ,...,in−1
= p(sn |jn ) max Pi,jn Vn−1 (i) i
= p(sn |jn )Pin−1 (jn ),jn Vn−1 (in−1 (jn )) Thus, in−1 (jn ) is the next to last state of the maximizing sequence. Continuing in this manner, the second from the last state of the maximizing sequence is in−2 (in−1 (jn )), and so on. The preceding approach to finding the most likely sequence of states given a prescribed sequence of signals is known as the Viterbi Algorithm.
Exercises *1. Three white and three black balls are distributed in two urns in such a way that each contains three balls. We say that the system is in state i, i = 0, 1, 2, 3, if the first urn contains i white balls. At each step, we draw one ball from each urn and place the ball drawn from the first urn into the second, and conversely with the ball from the second urn. Let Xn denote the state of the system after the nth step. Explain why {Xn , n = 0, 1, 2, . . .} is a Markov chain and calculate its transition probability matrix. 2. Suppose that whether or not it rains today depends on previous weather conditions through the last three days. Show how this system may be analyzed by using a Markov chain. How many states are needed?
264
4
Markov Chains
3. In Exercise 2, suppose that if it has rained for the past three days, then it will rain today with probability 0.8; if it did not rain for any of the past three days, then it will rain today with probability 0.2; and in any other case the weather today will, with probability 0.6, be the same as the weather yesterday. Determine P for this Markov chain. *4. Consider a process {Xn , n = 0, 1, . . .} which takes on the values 0, 1, or 2. Suppose P {Xn+1 = j |Xn = i, Xn−1 = in−1 , . . . , X0 = i0 } I Pij , when n is even = PijII , when n is odd where 2j =0 PijI = 2j =0 PijII = 1, i = 0, 1, 2. Is {Xn , n 0} a Markov chain? If not, then show how, by enlarging the state space, we may transform it into a Markov chain. 5. A Markov chain {Xn , n 0} with states 0, 1, 2, has the transition probability matrix ⎡1 1 1⎤ ⎢ ⎣0
2
3 1 3
1 2
0
6 2 3 1 2
⎥ ⎦
If P {X0 = 0} = P {X0 = 1} = 14 , find E[X3 ]. 6. Let the transition probability matrix of a two-state Markov chain be given, as in Example 4.2, by 3 3 3 p 1 − p3 3 3 P=3 1−p p 3 Show by mathematical induction that 3 3 1 + 1 (2p − 1)n 3 (n) P = 3 21 21 3 − (2p − 1)n 2 2
1 2 1 2
3 − 12 (2p − 1)n 3 3 3 + 1 (2p − 1)n 3 2
7. In Example 4.4 suppose that it has rained neither yesterday nor the day before yesterday. What is the probability that it will rain tomorrow? 8. Suppose that coin 1 has probability 0.7 of coming up heads, and coin 2 has probability 0.6 of coming up heads. If the coin flipped today comes up heads, then we select coin 1 to flip tomorrow, and if it comes up tails, then we select coin 2 to flip tomorrow. If the coin initially flipped is equally likely to be coin 1 or coin 2, then what is the probability that the coin flipped on the third day after the initial flip is coin 1?
Exercises
265
9. Suppose in Exercise 8 that the coin flipped on Monday comes up heads. What is the probability that the coin flipped on Friday of the same week also comes up heads? 10. In Example 4.3, Gary is currently in a cheerful mood. What is the probability that he is not in a glum mood on any of the following three days? 11. In Example 4.3, Gary was in a glum mood four days ago. Given that he hasn’t felt cheerful in a week, what is the probability he is feeling glum today? 12. For a Markov chain {Xn , n 0} with transition probabilities Pi,j , consider the conditional probability that Xn = m given that the chain started at time 0 in state i and has not yet entered state r by time n, where r is a specified state not equal to either i or m. We are interested in whether this conditional probability is equal to the n stage transition probability of a Markov chain whose state space does not include state r and whose transition probabilities are Qi,j =
Pi,j , 1 − Pi,r
i, j = r
Either prove the equality P {Xn = m|X0 = i, Xk = r, k = 1, . . . , n} = Qni,m or construct a counterexample. 13. Let P be the transition probability matrix of a Markov chain. Argue that if for some positive integer r, Pr has all positive entries, then so does Pn , for all integers n r. 14. Specify the classes of the following Markov chains, and determine whether they are transient or recurrent: 3 3 3 3 0 0 0 13 3 30 1 1 3 3 3 2 23 3 3 0 0 0 13 31 3 3 3 1 P2 = 3 1 1 P1 = 3 2 0 2 3 , 3, 3 3 2 2 0 03 3 3 1 1 03 3 3 3 0 0 1 03 2 2 31 32 3 31 34 31 P3 = 3 32 3 30 3 30
0
0
1 2 1 4 1 2
0
0
0
0
1 2 1 2
0 1 2
0 0
3 03 3 3 03 3 03 3, 13 3 23 13 2
31 3 34 31 32 3 P4 = 3 30 3 30 3 31
3 4 1 2
0 0 0
0
0
0 1
0 0
1 3
2 3
0
0
3 03 3 3 03 3 03 3 3 03 3 03
15. Prove that if the number of states in a Markov chain is M, and if state j can be reached from state i, then it can be reached in M steps or less.
266
4
Markov Chains
*16. Show that if state i is recurrent and state i does not communicate with state j , then Pij = 0. This implies that once a process enters a recurrent class of states it can never leave that class. For this reason, a recurrent class is often referred to as a closed class. 17. For the random walk of Example 4.15 use the strong law of large numbers to give another proof that the Markov chain is transient when p = 12 . Hint: Note that the state at time n can be written as ni=1 Yi where the Yi s are independent and P {Yi = 1} = p = 1 − P {Yi = −1}. Argue that if p > 12 , then, by the strong law of large numbers, n1 Yi → ∞ as n → ∞ and hence the initial state 0 can be visited only finitely often, and hence must be transient. A similar argument holds when p < 12 . 18. Coin 1 comes up heads with probability 0.6 and coin 2 with probability 0.5. A coin is continually flipped until it comes up tails, at which time that coin is put aside and we start flipping the other one. (a) What proportion of flips use coin 1? (b) If we start the process with coin 1 what is the probability that coin 2 is used on the fifth flip? 19. For Example 4.4, calculate the proportion of days that it rains. 20. A transition probability matrix P is said to be doubly stochastic if the sum over each column equals one; that is, Pij = 1, for all j i
If such a chain is irreducible and aperiodic and consists of M + 1 states 0, 1, . . . , M, show that the limiting probabilities are given by πj =
1 , M +1
j = 0, 1, . . . , M
*21. A DNA nucleotide has any of 4 values. A standard model for a mutational change of the nucleotide at a specific location is a Markov chain model that supposes that in going from period to period the nucleotide does not change with probability 1 − 3α, and if it does change then it is equally likely to change to any of the other 3 values, for some 0 < α < 13 . n = 1 + 3 (1 − 4α)n . (a) Show that P1,1 4 4 (b) What is the long run proportion of time the chain is in each state?
22. Let Yn be the sum of n independent rolls of a fair die. Find lim P {Yn is a multiple of 13}
n→∞
Exercises
267
Hint: Define an appropriate Markov chain and apply the results of Exercise 20. 23. Trials are performed in sequence. If the last two trials were successes, then the next trial is a success with probability 0.8; otherwise the next trial is a success with probability 0.5. In the long run, what proportion of trials are successes? 24. Consider three urns, one colored red, one white, and one blue. The red urn contains 1 red and 4 blue balls; the white urn contains 3 white balls, 2 red balls, and 2 blue balls; the blue urn contains 4 white balls, 3 red balls, and 2 blue balls. At the initial stage, a ball is randomly selected from the red urn and then returned to that urn. At every subsequent stage, a ball is randomly selected from the urn whose color is the same as that of the ball previously selected and is then returned to that urn. In the long run, what proportion of the selected balls are red? What proportion are white? What proportion are blue? 25. Each morning an individual leaves his house and goes for a run. He is equally likely to leave either from his front or back door. Upon leaving the house, he chooses a pair of running shoes (or goes running barefoot if there are no shoes at the door from which he departed). On his return he is equally likely to enter, and leave his running shoes, either by the front or back door. If he owns a total of k pairs of running shoes, what proportion of the time does he run barefooted? 26. Consider the following approach to shuffling a deck of n cards. Starting with any initial ordering of the cards, one of the numbers 1, 2, . . . , n is randomly chosen in such a manner that each one is equally likely to be selected. If number i is chosen, then we take the card that is in position i and put it on top of the deck—that is, we put that card in position 1. We then repeatedly perform the same operation. Show that, in the limit, the deck is perfectly shuffled in the sense that the resultant ordering is equally likely to be any of the n! possible orderings. *27. Determine the limiting probabilities πj for the model presented in Exercise 1. Give an intuitive explanation of your answer. 28. For a series of dependent trials the probability of success on any trial is (k + 1)/(k + 2) where k is equal to the number of successes on the previous two trials. Compute limn→∞ P {success on the nth trial}. 29. An organization has N employees where N is a large number. Each employee has one of three possible job classifications and changes classifications (independently) according to a Markov chain with transition probabilities ⎡ ⎤ 0.7 0.2 0.1 ⎣0.2 0.6 0.2⎦ 0.1 0.4 0.5 What percentage of employees are in each classification?
268
4
Markov Chains
30. Three out of every four trucks on the road are followed by a car, while only one out of every five cars is followed by a truck. What fraction of vehicles on the road are trucks? 31. A certain town never has two sunny days in a row. Each day is classified as being either sunny, cloudy (but dry), or rainy. If it is sunny one day, then it is equally likely to be either cloudy or rainy the next day. If it is rainy or cloudy one day, then there is one chance in two that it will be the same the next day, and if it changes then it is equally likely to be either of the other two possibilities. In the long run, what proportion of days are sunny? What proportion are cloudy? *32. Each of two switches is either on or off during a day. On day n, each switch will independently be on with probability [1 + number of on switches during day n − 1]/4 For instance, if both switches are on during day n − 1, then each will independently be on during day n with probability 3/4. What fraction of days are both switches on? What fraction are both off? 33. A professor continually gives exams to her students. She can give three possible types of exams, and her class is graded as either having done well or badly. Let pi denote the probability that the class does well on a type i exam, and suppose that p1 = 0.3, p2 = 0.6, and p3 = 0.9. If the class does well on an exam, then the next exam is equally likely to be any of the three types. If the class does badly, then the next exam is always type 1. What proportion of exams are type i, i = 1, 2, 3? 34. A flea moves around the vertices of a triangle in the following manner: Whenever it is at vertex i it moves to its clockwise neighbor vertex with probability pi and to the counterclockwise neighbor with probability qi = 1 − pi , i = 1, 2, 3. (a) Find the proportion of time that the flea is at each of the vertices. (b) How often does the flea make a counterclockwise move which is then followed by five consecutive clockwise moves? 35. Consider a Markov chain with states 0, 1, 2, 3, 4. Suppose P0,4 = 1; and suppose that when the chain is in state i, i > 0, the next state is equally likely to be any of the states 0, 1, . . . , i − 1. Find the limiting probabilities of this Markov chain. 36. The state of a process changes daily according to a two-state Markov chain. If the process is in state i during one day, then it is in state j the follow-
Exercises
269
ing day with probability Pi,j , where P0,0 = 0.4,
P0,1 = 0.6,
P1,0 = 0.2,
P1,1 = 0.8
Every day a message is sent. If the state of the Markov chain that day is i then the message sent is “good” with probability pi and is “bad” with probability qi = 1 − pi , i = 0, 1 (a) If the process is in state 0 on Monday, what is the probability that a good message is sent on Tuesday? (b) If the process is in state 0 on Monday, what is the probability that a good message is sent on Friday? (c) In the long run, what proportion of messages are good? (d) Let Yn equal 1 if a good message is sent on day n and let it equal 2 otherwise. Is {Yn , n 1} a Markov chain? If so, give its transition probability matrix. If not, briefly explain why not. 37. Show that the stationary probabilities for the Markov chain having transition probabilities Pi,j are also the stationary probabilities for the Markov chain whose transition probabilities Qi,j are given by k Qi,j = Pi,j
for any specified positive integer k. 38. Recall that state i is said to be positive recurrent if mi,i < ∞, where mi,i is the expected number of transitions until the Markov chain, starting in state i, makes a transition back into that state. Because πi , the long run proportion of time the Markov chain, starting in state i, spends in state i, satisfies πi =
1 mi,i
it follows that state i is positive recurrent if and only if πi > 0. Suppose that state i is positive recurrent and that state i communicates with state j . Show that state j is also positive recurrent by arguing that there is an integer n such that n πj πi Pi,j >0
39. Recall that a recurrent state that is not positive recurrent is called null recurrent. Use the result of Exercise 38 to prove that null recurrence is a class property. That is, if state i is null recurrent and state i communicates with state j, show that state j is also null recurrent.
270
4
Markov Chains
40. It follows from the argument made in Exercise 38 that state i is null recurrent if it is recurrent and πi = 0. Consider the one-dimensional symmetric random walk of Example 4.15. (a) Argue that πi = π0 for all i. (b) Argue that all states are null recurrent. *41. Let πi denote the long-run proportion of time a given irreducible Markov chain is in state i. (a) Explain why πi is also the proportion of transitions that are into state i as well as being the proportion of transitions that are from state i. (b) πi Pij represents the proportion of transitions that satisfy what property? (c) i πi Pij represent the proportion of transitions that satisfy what property? (d) Using the preceding explain why πj =
πi Pij
i
42. Let A be a set of states, and let Ac be the remaining states. (a) What is the interpretation of
πi Pij ?
i∈A j ∈Ac
(b) What is the interpretation of
πi Pij ?
i∈Ac j ∈A
(c) Explain the identity i∈A j ∈Ac
πi Pij =
πi Pij
i∈Ac j ∈A
43. Each day, one of n possible elements is requested, the ith one with probability Pi , i 1, n1 Pi = 1. These elements are at all times arranged in an ordered list which is revised as follows: The element selected is moved to the front of the list with the relative positions of all the other elements remaining unchanged. Define the state at any time to be the list ordering at that time and note that there are n! possible states. (a) Argue that the preceding is a Markov chain.
Exercises
271
(b) For any state i1 , . . . , in (which is a permutation of 1, 2, . . . , n), let π(i1 , . . . , in ) denote the limiting probability. In order for the state to be i1 , . . . , in , it is necessary for the last request to be for i1 , the last non-i1 request for i2 , the last non-i1 or i2 request for i3 , and so on. Hence, it appears intuitive that π(i1 , . . . , in ) = Pi1
Pin−1 Pi2 Pi3 ··· 1 − Pi1 1 − Pi1 − Pi2 1 − Pi1 − · · · − Pin−2
Verify when n = 3 that the preceding are indeed the limiting probabilities. 44. Suppose that a population consists of a fixed number, say, m, of genes in any generation. Each gene is one of two possible genetic types. If any generation has exactly i (of its m) genes being type 1, then the next generation will have j type 1 (and m − j type 2) genes with probability j m i m − i m−j , j m m
j = 0, 1, . . . , m
Let Xn denote the number of type 1 genes in the nth generation, and assume that X0 = i. (a) Find E[Xn ]. (b) What is the probability that eventually all the genes will be type 1? 45. Consider an irreducible finite Markov chain with states 0, 1, . . . , N . (a) Starting in state i, what is the probability the process will ever visit state j ? Explain! (b) Let xi = P {visit state N before state 0|start in i}. Compute a set of linear equations which the xi satisfy, i = 0, 1, . . . , N . (c) If j j Pij = i for i = 1, . . . , N − 1, show that xi = i/N is a solution to the equations in part (b) 46. An individual possesses r umbrellas which he employs in going from his home to office, and vice versa. If he is at home (the office) at the beginning (end) of a day and it is raining, then he will take an umbrella with him to the office (home), provided there is one to be taken. If it is not raining, then he never takes an umbrella. Assume that, independent of the past, it rains at the beginning (end) of a day with probability p. (i) Define a Markov chain with r + 1 states which will help us to determine the proportion of time that our man gets wet. (Note: He gets wet if it is raining, and all umbrellas are at his other location.)
272
4
Markov Chains
(ii) Show that the limiting probabilities are given by ⎧ q ⎪ , if i = 0 ⎪ ⎪ ⎨r + q where q = 1 − p πi = ⎪ 1 ⎪ ⎪ ⎩ , if i = 1, . . . , r r +q (iii) What fraction of time does our man get wet? (iv) When r = 3, what value of p maximizes the fraction of time he gets wet *47. Let {Xn , n 0} denote an ergodic Markov chain with limiting probabilities πi . Define the process {Yn , n 1} by Yn = (Xn−1 , Xn ). That is, Yn keeps track of the last two states of the original chain. Is {Yn , n 1} a Markov chain? If so, determine its transition probabilities and find lim P {Yn = (i, j )}
n→∞
48. Verify the transition probability matrix given in Example 4.20. 49. Let P (1) and P (2) denote transition probability matrices for ergodic Markov chains having the same state space. Let π 1 and π 2 denote the stationary (limiting) probability vectors for the two chains. Consider a process defined as follows: (i) X0 = 1. A coin is then flipped and if it comes up heads, then the remaining states X1 , . . . are obtained from the transition probability matrix P (1) and if tails from the matrix P (2) . Is {Xn , n 0} a Markov chain? If p = P {coin comes up heads}, what is limn→∞ P (Xn = i)? (ii) X0 = 1. At each stage the coin is flipped and if it comes up heads, then the next state is chosen according to P (1) and if tails comes up, then it is chosen according to P (2) . In this case do the successive states constitute a Markov chain? If so, determine the transition probabilities. Show by a counterexample that the limiting probabilities are not the same as in part (i). 50. In Exercise 8, if today’s flip lands heads, what is the expected number of additional flips needed until the pattern t, t, h, t, h, t, t occurs? 51. In Example 4.3, Gary is in a cheerful mood today. Find the expected number of days until he has been glum for three consecutive days. 52. A taxi driver provides service in two zones of a city. Fares picked up in zone A will have destinations in zone A with probability 0.6 or in zone B with probability 0.4. Fares picked up in zone B will have destinations in zone A with probability 0.3 or in zone B with probability 0.7. The driver’s expected profit for a trip entirely in zone A is 6; for a trip entirely in zone B is 8; and for a trip that involves both zones is 12. Find the taxi driver’s average profit per trip.
Exercises
273
53. Find the average premium received per policyholder of the insurance company of Example 4.23 if λ = 1/4 for one-third of its clients, and λ = 1/2 for two-thirds of its clients. 54. Consider the Ehrenfest urn model in which M molecules are distributed between two urns, and at each time point one of the molecules is chosen at random and is then removed from its urn and placed in the other one. Let Xn denote the number of molecules in urn 1 after the nth switch and let μn = E[Xn ]. Show that (i) μn+1 = 1 + (1 − 2/M)μn . (ii) Use (i) to prove that μn =
M −2 n M M E[X0 ] − + 2 M 2
55. Consider a population of individuals each of whom possesses two genes which can be either type A or type a. Suppose that in outward appearance type A is dominant and type a is recessive. (That is, an individual will have only the outward characteristics of the recessive gene if its pair is aa.) Suppose that the population has stabilized, and the percentages of individuals having respective gene pairs AA, aa, and Aa are p, q, and r. Call an individual dominant or recessive depending on the outward characteristics it exhibits. Let S11 denote the probability that an offspring of two dominant parents will be recessive; and let S10 denote the probability that the offspring of one dominant and one re2 . cessive parent will be recessive. Compute S11 and S10 to show that S11 = S10 (The quantities S10 and S11 are known in the genetics literature as Snyder’s ratios.) 56. Suppose that on each play of the game a gambler either wins 1 with probability p or loses 1 with probability 1 − p. The gambler continues betting until she or he is either winning n or losing m. What is the probability that the gambler quits a winner? 57. A particle moves among n + 1 vertices that are situated on a circle in the following manner. At each step it moves one step either in the clockwise direction with probability p or the counterclockwise direction with probability q = 1 − p. Starting at a specified state, call it state 0, let T be the time of the first return to state 0. Find the probability that all states have been visited by time T . Hint: Condition on the initial transition and then use results from the gambler’s ruin problem.
274
4
Markov Chains
58. In the gambler’s ruin problem of Section 4.5.1, suppose the gambler’s fortune is presently i, and suppose that we know that the gambler’s fortune will eventually reach N (before it goes to 0). Given this information, show that the probability he wins the next gamble is p[1 − (q/p)i+1 ] , 1 − (q/p)i
if p =
1 2
i+1 , 2i
if p =
1 2
Hint: The probability we want is P {Xn+1 = i + 1|Xn = i, lim Xm = N } m→∞
=
P {Xn+1 = i + 1, limm Xm = N |Xn = i} P {limm Xm = N |Xn = i}
59. For the gambler’s ruin model of Section 4.5.1, let Mi denote the mean number of games that must be played until the gambler either goes broke or reaches a fortune of N , given that he starts with i, i = 0, 1, . . . , N . Show that Mi satisfies M0 = MN = 0;
Mi = 1 + pMi+1 + qMi−1 ,
i = 1, . . . , N − 1
60. Solve the equations given in Exercise 59 to obtain Mi = i(N − i), =
i N 1 − (q/p)i , − q − p q − p 1 − (q/p)N
if p =
1 2
if p =
1 2
61. Suppose in the gambler’s ruin problem that the probability of winning a bet depends on the gambler’s present fortune. Specifically, suppose that αi is the probability that the gambler wins a bet when his or her fortune is i. Given that the gambler’s initial fortune is i, let P (i) denote the probability that the gambler’s fortune reaches N before 0. (a) Derive a formula that relates P (i) to P (i − 1) and P (i + 1). (b) Using the same approach as in the gambler’s ruin problem, solve the equation of part (a) for P (i). (c) Suppose that i balls are initially in urn 1 and N − i are in urn 2, and suppose that at each stage one of the N balls is randomly chosen, taken from whichever urn it is in, and placed in the other urn. Find the probability that the first urn becomes empty before the second.
Exercises
275
*62. In Exercise 21, (a) what is the expected number of steps the particle takes to return to the starting position? (b) what is the probability that all other positions are visited before the particle returns to its starting state? 63. For the Markov chain with states 1, 2, 3, 4 whose transition probability matrix P is as specified below find fi3 and si3 for i = 1, 2, 3. ⎡ ⎤ 0.4 0.2 0.1 0.3 ⎢ 0.1 0.5 0.2 0.2 ⎥ ⎥ P=⎢ ⎣ 0.3 0.4 0.2 0.1 ⎦ 0 0 0 1 64. Consider a branching process having μ < 1. Show that if X0 = 1, then the expected number of individuals that ever exist in this population is given by 1/(1 − μ). What if X0 = n? 65. In a branching process having X0 = 1 and μ > 1, prove that π0 is the smallest positive number satisfying Equation (4.16). j Hint: Let π be any solution of π = ∞ j =0 π Pj . Show by mathematical induction that π P {Xn = 0} for all n, and let n → ∞. In using the induction argue that P {Xn = 0} =
∞ (P {Xn−1 = 0})j Pj j =0
66. For a branching process, calculate π0 when (a) P0 = 14 , P2 = 34 . (b) P0 = 14 , P1 = 12 , P2 = 14 . (c) P0 = 16 , P1 = 12 , P3 = 13 . 67. At all times, an urn contains N balls—some white balls and some black balls. At each stage, a coin having probability p, 0 < p < 1, of landing heads is flipped. If heads appears, then a ball is chosen at random from the urn and is replaced by a white ball; if tails appears, then a ball is chosen from the urn and is replaced by a black ball. Let Xn denote the number of white balls in the urn after the nth stage. (a) Is {Xn , n 0} a Markov chain? If so, explain why. (b) What are its classes? What are their periods? Are they transient or recurrent?
276
4
Markov Chains
(c) Compute the transition probabilities Pij . (d) Let N = 2. Find the proportion of time in each state. (e) Based on your answer in part (d) and your intuition, guess the answer for the limiting probability in the general case. (f) Prove your guess in part (e) either by showing that Equation (4.7) is satisfied or by using the results of Example 4.31. (g) If p = 1, what is the expected time until there are only white balls in the urn if initially there are i white and N − i black? *68. (a) Show that the limiting probabilities of the reversed Markov chain are the same as for the forward chain by showing that they satisfy the equations πj =
πi Qij
i
(b) Give an intuitive explanation for the result of part (a). 69. M balls are initially distributed among m urns. At each stage one of the balls is selected at random, taken from whichever urn it is in, and then placed, at random, in one of the other M − 1 urns. Consider the Markov chain whose state at any time is the vector (n1 , . . . , nm ) where ni denotes the number of balls in urn i. Guess at the limiting probabilities for this Markov chain and then verify your guess and show at the same time that the Markov chain is time reversible. 70. A total of m white and m black balls are distributed among two urns, with each urn containing m balls. At each stage, a ball is randomly selected from each urn and the two selected balls are interchanged. Let Xn denote the number of black balls in urn 1 after the nth interchange. (a) Give the transition probabilities of the Markov chain Xn , n 0. (b) Without any computations, what do you think are the limiting probabilities of this chain? (c) Find the limiting probabilities and show that the stationary chain is time reversible. 71. It follows from Theorem 4.2 that for a time reversible Markov chain Pij Pj k Pki = Pik Pkj Pj i ,
for all i, j, k
It turns out that if the state space is finite and Pij > 0 for all i, j , then the preceding is also a sufficient condition for time reversibility. [That is, in this case, we need only check Equation (4.26) for paths from i to i that have only two intermediate states.] Prove this.
Exercises
277
Hint: Fix i and show that the equations πj Pj k = πk Pkj are satisfied by πj = cPij /Pj i , where c is chosen so that
j
πj = 1.
72. For a time reversible Markov chain, argue that the rate at which transitions from i to j to k occur must equal the rate at which transitions from k to j to i occur. 73. Show that the Markov chain of Exercise 31 is time reversible. 74. A group of n processors is arranged in an ordered list. When a job arrives, the first processor in line attempts it; if it is unsuccessful, then the next in line tries it; if it too is unsuccessful, then the next in line tries it, and so on. When the job is successfully processed or after all processors have been unsuccessful, the job leaves the system. At this point we are allowed to reorder the processors, and a new job appears. Suppose that we use the one-closer reordering rule, which moves the processor that was successful one closer to the front of the line by interchanging its position with the one in front of it. If all processors were unsuccessful (or if the processor in the first position was successful), then the ordering remains the same. Suppose that each time processor i attempts a job then, independently of anything else, it is successful with probability pi . (a) Define an appropriate Markov chain to analyze this model. (b) Show that this Markov chain is time reversible. (c) Find the long-run probabilities. 75. A Markov chain is said to be a tree process if (i) Pij > 0 whenever Pj i > 0, (ii) for every pair of states i and j , i = j , there is a unique sequence of distinct states i = i0 , i1 , . . . , in−1 , in = j such that Pik ,ik+1 > 0,
k = 0, 1, . . . , n − 1
In other words, a Markov chain is a tree process if for every pair of distinct states i and j there is a unique way for the process to go from i to j without reentering a state (and this path is the reverse of the unique path from j to i). Argue that an ergodic tree process is time reversible. 76. On a chessboard compute the expected number of plays it takes a knight, starting in one of the four corners of the chessboard, to return to its initial position if we assume that at each play it is equally likely to choose any of its legal moves. (No other pieces are on the board.)
278
4
Markov Chains
Hint: Make use of Example 4.32. 77. In a Markov decision problem, another criterion often used, different than the expected average return per unit time, is that of the expected discounted return. In this criterion we a number α, 0 < α < 1, and try to choose a policy choose i so as to maximize E[ ∞ i=0 α R(Xi , ai )] (that is, rewards at time n are discounted at rate α n ). Suppose that the initial state is chosen according to the probabilities bi . That is, P {X0 = i} = bi ,
i = 1, . . . , n
For a given policy β let yj a denote the expected discounted time that the process is in state j and action a is chosen. That is, ,∞ yj a = E β α n I{Xn =j,an =a} n=0
where for any event A the indicator variable IA is defined by 1, if A occurs IA = 0, otherwise (a) Show that
or, in other words, der β. (b) Show that
a
n
α I{Xn =j }
n=0
a yj a
j
yj a = E
a
,∞
is the expected discounted time in state j un-
yj a =
1 , 1−α
yj a = bj + α
a
i
yia Pij (a)
a
Hint: For the second equation, use the identity I{Xn=i ,an=a } I{Xn+1 =j } I{Xn+1 =j } = a
i
Take expectations of the preceding to obtain # " # " E I{Xn=i ,an=a } Pij (a) E IXn+1 =j } = i
a
Exercises
279
(c) Let {yj a } be a set of numbers satisfying 1 , yj a = 1−α a j
yj a = bj + α yia Pij (a) a
i
(4.38)
a
Argue that yj a can be interpreted as the expected discounted time that the process is in state j and action a is chosen when the initial state is chosen according to the probabilities bj and the policy β, given by yia βi (a) = a yia is employed. Hint: Derive a set of equations for the expected discounted times when policy β is used and show that they are equivalent to Equation (4.38). (d) Argue that an optimal policy with respect to the expected discounted return criterion can be obtained by first solving the linear program maximize yj a R(j, a), j
such that
a
j
yj a =
a
1 , 1−α
yj a = bj + α
a
i
yj a 0,
yia Pij (a),
a
all j, a;
and then defining the policy β ∗ by y∗ βi∗ (a) = ia ∗ a yia where the yj∗a are the solutions of the linear program. 78. For the Markov chain of Exercise 5, suppose that p(s|j ) is the probability that signal s is emitted when the underlying Markov chain state is j , j = 0, 1, 2. (a) What proportion of emissions are signal s? (b) What proportion of those times in which signal s is emitted is 0 the underlying state? 79. In Example 4.39, what is the probability that the first 4 items produced are all acceptable?
280
4
Markov Chains
References 1. K. L. Chung, “Markov Chains with Stationary Transition Probabilities,” Springer, Berlin, 1960. 2. S. Karlin and H. Taylor, “A First Course in Stochastic Processes,” Second Edition, Academic Press, New York, 1975. 3. J. G. Kemeny and J. L. Snell, “Finite Markov Chains,” Van Nostrand Reinhold, Princeton, New Jersey, 1960. 4. S. M. Ross, “Stochastic Processes,” Second Edition, John Wiley, New York, 1996. 5. S. M. Ross, “Probability Models for Computer Science,” Academic Press, 2002.
The Exponential Distribution and the Poisson Process
5 5.1.
Introduction
In making a mathematical model for a real-world phenomenon it is always necessary to make certain simplifying assumptions so as to render the mathematics tractable. On the other hand, however, we cannot make too many simplifying assumptions, for then our conclusions, obtained from the mathematical model, would not be applicable to the real-world situation. Thus, in short, we must make enough simplifying assumptions to enable us to handle the mathematics but not so many that the mathematical model no longer resembles the real-world phenomenon. One simplifying assumption that is often made is to assume that certain random variables are exponentially distributed. The reason for this is that the exponential distribution is both relatively easy to work with and is often a good approximation to the actual distribution. The property of the exponential distribution that makes it easy to analyze is that it does not deteriorate with time. By this we mean that if the lifetime of an item is exponentially distributed, then an item that has been in use for ten (or any number of) hours is as good as a new item in regards to the amount of time remaining until the item fails. This will be formally defined in Section 5.2 where it will be shown that the exponential is the only distribution which possesses this property. In Section 5.3 we shall study counting processes with an emphasis on a kind of counting process known as the Poisson process. Among other things we shall discover about this process is its intimate connection with the exponential distribution.
281
282
5.2.
5
The Exponential Distribution and the Poisson Process
The Exponential Distribution
5.2.1. Definition A continuous random variable X is said to have an exponential distribution with parameter λ, λ > 0, if its probability density function is given by −λx , x 0 λe f (x) = 0, x s + t | X > t} = P {X > s}
for all s, t 0
(5.2)
If we think of X as being the lifetime of some instrument, then Equation (5.2) states that the probability that the instrument lives for at least s + t hours given that it has survived t hours is the same as the initial probability that it lives for at least s hours. In other words, if the instrument is alive at time t, then the distribution of the remaining amount of time that it survives is the same as the original lifetime distribution; that is, the instrument does not remember that it has already been in use for a time t. The condition in Equation (5.2) is equivalent to P {X > s + t, X > t} = P {X > s} P {X > t} or P {X > s + t} = P {X > s}P {X > t}
(5.3)
Since Equation (5.3) is satisfied when X is exponentially distributed (for e−λ(s+t) = e−λs e−λt ), it follows that exponentially distributed random variables are memoryless.
5.2.
The Exponential Distribution
285
Example 5.2 Suppose that the amount of time one spends in a bank is expo1 nentially distributed with mean ten minutes, that is, λ = 10 . What is the probability that a customer will spend more than fifteen minutes in the bank? What is the probability that a customer will spend more than fifteen minutes in the bank given that she is still in the bank after ten minutes? Solution: If X represents the amount of time that the customer spends in the bank, then the first probability is just P {X > 15} = e−15λ = e−3/2 ≈ 0.220 The second question asks for the probability that a customer who has spent ten minutes in the bank will have to spend at least five more minutes. However, since the exponential distribution does not “remember” that the customer has already spent ten minutes in the bank, this must equal the probability that an entering customer spends at least five minutes in the bank. That is, the desired probability is just P {X > 5} = e−5λ = e−1/2 ≈ 0.604
Example 5.3 Consider a post office that is run by two clerks. Suppose that when Mr. Smith enters the system he discovers that Mr. Jones is being served by one of the clerks and Mr. Brown by the other. Suppose also that Mr. Smith is told that his service will begin as soon as either Jones or Brown leaves. If the amount of time that a clerk spends with a customer is exponentially distributed with mean 1/λ, what is the probability that, of the three customers, Mr. Smith is the last to leave the post office? Solution: The answer is obtained by this reasoning: Consider the time at which Mr. Smith first finds a free clerk. At this point either Mr. Jones or Mr. Brown would have just left and the other one would still be in service. However, by the lack of memory of the exponential, it follows that the amount of time that this other man (either Jones or Brown) would still have to spend in the post office is exponentially distributed with mean 1/λ. That is, it is the same as if he were just starting his service at this point. Hence, by symmetry, the probability that he finishes before Smith must equal 12 .
Example 5.4 The dollar amount of damage involved in an automobile accident is an exponential random variable with mean 1000. Of this, the insurance company only pays that amount exceeding (the deductible amount of) 400. Find the expected value and the standard deviation of the amount the insurance company pays per accident.
286
5
The Exponential Distribution and the Poisson Process
Solution: If X is the dollar amount of damage resulting from an accident, then the amount paid by the insurance company is (X − 400)+ , (where a + is defined to equal a if a > 0 and to equal 0 if a 0). Whereas we could certainly determine the expected value and variance of (X − 400)+ from first principles, it is easier to condition on whether X exceeds 400. So, let I=
1, 0,
if X > 400 if X 400
Let Y = (X − 400)+ be the amount paid. By the lack of memory property of the exponential, it follows that if a damage amount exceeds 400, then the amount by which it exceeds it is exponential with mean 1000. Therefore, E[Y |I = 1] = 1000 E[Y |I = 0] = 0 Var(Y |I = 1) = (1000)2 Var(Y |I = 0) = 0 which can be conveniently written as E[Y |I ] = 103 I,
Var(Y |I ) = 106 I
Because I is a Bernoulli random variable that is equal to 1 with probability e−0.4 , we obtain " # E[Y ] = E E[Y |I ] = 103 E[I ] = 103 e−0.4 ≈ 670.32 and, by the conditional variance formula " # Var(Y ) = E Var(Y |I ) + Var E[Y |I ] = 106 e−0.4 + 106 e−0.4 (1 − e−0.4 ) where the final equality used that the variance of a Bernoulli random variable with parameter p is p(1 − p). Consequently, 2
Var(Y ) ≈ 944.09
5.2.
The Exponential Distribution
287
It turns out that not only is the exponential distribution “memoryless,” but it is the unique distribution possessing this property. To see this, suppose that X is memoryless and let F¯ (x) = P {X > x}. Then by Equation (5.3) it follows that F¯ (s + t) = F¯ (s)F¯ (t) That is, F¯ (x) satisfies the functional equation g(s + t) = g(s)g(t) However, it turns out that the only right continuous solution of this functional equation is g(x) = e−λx ∗ and since a distribution function is always right continuous we must have F¯ (x) = e−λx or F (x) = P {X x} = 1 − e−λx which shows that X is exponentially distributed.
Example 5.5 Suppose that the amount of time that a lightbulb works before burning itself out is exponentially distributed with mean ten hours. Suppose that a person enters a room in which a lightbulb is burning. If this person desires to work for five hours, then what is the probability that she will be able to complete her work without the bulb burning out? What can be said about this probability when the distribution is not exponential? ∗ This is proven as follows: If g(s + t) = g(s)g(t), then
g
1 1 1 2 =g = g2 + n n n n
and repeating this yields g(m/n) = g m (1/n). Also 1 1 1 1 + + ··· + = gn g(1) = g n n n n
or
g
1 = (g(1))1/n n
Hence g(m/n) = (g(1))m/n , which implies, since g is right continuous, that g(x) = (g(1))x . Since g(1) = (g( 12 ))2 0 we obtain g(x) = e−λx , where λ = − log(g(1)).
288
5
The Exponential Distribution and the Poisson Process
Solution: Since the bulb is burning when the person enters the room it follows, by the memoryless property of the exponential, that its remaining lifetime is exponential with mean ten. Hence the desired probability is P {remaining lifetime > 5} = 1 − F (5) = e−5λ = e−1/2 However, if the lifetime distribution F is not exponential, then the relevant probability is P {lifetime > t + 5 | lifetime > t} =
1 − F (t + 5) 1 − F (t)
where t is the amount of time that the bulb had been in use prior to the person entering the room. That is, if the distribution is not exponential then additional information is needed (namely, t) before the desired probability can be calculated. In fact, it is for this reason, namely, that the distribution of the remaining lifetime is independent of the amount of time that the object has already survived, that the assumption of an exponential distribution is so often made. The memoryless property is further illustrated by the failure rate function (also called the hazard rate function) of the exponential distribution. Consider a continuous positive random variable X having distribution function F and density f . The failure (or hazard) rate function r(t) is defined by r(t) =
f (t) 1 − F (t)
(5.4)
To interpret r(t), suppose that an item, having lifetime X, has survived for t hours, and we desire the probability that it does not survive for an additional time dt. That is, consider P {X ∈ (t, t + dt)|X > t}. Now P {X ∈ (t, t + dt)|X > t} =
P {X ∈ (t, t + dt), X > t} P {X > t}
=
P {X ∈ (t, t + dt)} P {X > t}
≈
f (t) dt = r(t) dt 1 − F (t)
That is, r(t) represents the conditional probability density that a t-year-old item will fail. Suppose now that the lifetime distribution is exponential. Then, by the memoryless property, it follows that the distribution of remaining life for a t-year-old
5.2.
The Exponential Distribution
289
item is the same as for a new item. Hence r(t) should be constant. This checks out since r(t) = =
f (t) 1 − F (t) λe−λt =λ e−λt
Thus, the failure rate function for the exponential distribution is constant. The parameter λ is often referred to as the rate of the distribution. (Note that the rate is the reciprocal of the mean, and vice versa.) It turns out that the failure rate function r(t) uniquely determines the distribution F . To prove this, we note by Equation (5.4) that r(t) =
d dt F (t)
1 − F (t)
Integrating both sides yields log(1 − F (t)) = −
t
r(t) dt + k
0
or t 1 − F (t) = e exp − r(t) dt k
0
Letting t = 0 shows that k = 0 and thus t r(t) dt F (t) = 1 − exp − 0
The preceding identity can also be used to show that exponential random variables are the only ones that are memoryless. Because if X is memoryless, then its failure rate function must be constant. But if r(t) = c, then by the preceding equation 1 − F (t) = e−ct showing that the random variable is exponential.
Example 5.6 Let X1 , . . . , Xn be independent exponential random variables
with respective rates λ1 , . . . , λn , where λi = λj when i = j . Let T be independent
290
5
The Exponential Distribution and the Poisson Process
of these random variables and suppose that n
Pj = 1
where Pj = P {T = j }
j =1
The random variable XT is said to be a hyperexponential random variable. To see how such a random variable might originate, imagine that a bin contains n different types of batteries, with a type j battery lasting for an exponential distributed time with rate λj , j = 1, . . . , n. Suppose further that Pj is the proportion of batteries in the bin that are type j for each j = 1, . . . , n. If a battery is randomly chosen, in the sense that it is equally likely to be any of the batteries in the bin, then the lifetime of the battery selected will have the hyperexponential distribution specified in the preceding. To obtain the distribution function F of X = XT , condition on T . This yields 1 − F (t) = P {X > t} =
n
P {X > t|T = i}P {T = i}
i=1
=
n
Pi e−λi t
i=1
Differentiation of the preceding yields f , the density function of X. f (t) =
n
λi Pi e−λi t
i=1
Consequently, the failure rate function of a hyperexponential random variable is n −λj t j =1 Pj λj e r(t) = n −λi t i=1 Pi e By noting that P {T = j |X > t} =
P {X > t|T = j }P {T = j } P {X > t}
Pj e−λj t = n −λi t i=1 Pi e we see that the failure rate function r(t) can also be written as r(t) =
n j =1
λj P {T = j |X > t}
5.2.
The Exponential Distribution
291
If λ1 < λi , for all i > 1, then P1 e−λ1 t P1 e−λ1 t + ni=2 Pi e−λi t
P {T = 1|X > t} = =
P1 +
→1
P1 −(λi −λ1 )t i=2 Pi e
n
as t → ∞
Similarly, P {T = i|X > t} → 0 when i = 1, thus showing that lim r(t) = min λi
t→∞
i
That is, as a randomly chosen battery ages its failure rate converges to the failure rate of the exponential type having the smallest failure rate, which is intuitive since the longer the battery lasts, the more likely it is a battery type with the smallest failure rate. 5.2.3. Further Properties of the Exponential Distribution Let X1 , . . . , Xn be independent and identically distributed exponential random variables having mean 1/λ. It follows from the results of Example 2.38 that X1 + · · · + Xn has a gamma distribution with parameters n and λ. Let us now give a second verification of this result by using mathematical induction. Because there is nothing to prove when n = 1, let us start by assuming that X1 + · · · + Xn−1 has density given by fX1 +···+Xn−1 (t) = λe−λt Hence,
fX1 + ··· +Xn−1 +Xn (t) =
∞
0
=
t
(λt)n−2 (n − 2)!
fXn (t − s)fX1 + ··· +Xn−1 (s) ds
λe−λ(t−s) λe−λs
0
= λe−λt
(λs)n−2 ds (n − 2)!
(λt)n−1 (n − 1)!
which proves the result. Another useful calculation is to determine the probability that one exponential random variable is smaller than another. That is, suppose that X1 and X2 are independent exponential random variables with respective means 1/λ1 and 1/λ2 ;
292
5
The Exponential Distribution and the Poisson Process
what is P {X1 < X2 }? This probability is easily calculated by conditioning on X1 : ∞ P {X1 < X2 |X1 = x}λ1 e−λ1 x dx P {X1 < X2 } = 0
=
∞
P {x < X2 }λ1 e−λ1 x dx
0
=
∞
e−λ2 x λ1 e−λ1 x dx
0
=
∞
λ1 e−(λ1 +λ2 )x dx
0
=
λ1 λ1 + λ2
(5.5)
Suppose that X1 , X2 , . . . , Xn are independent exponential random variables, with Xi having rate μi , i = 1, . . . , n. It turns out that the smallest of the Xi is exponential with a rate equal to the sum of the μi . This is shown as follows: P {minimum(X1 , . . . , Xn ) > x} = P {Xi > x for each i = 1, . . . , n} =
n
P {Xi > x}
(by independence)
i=1
=
n i=1
e−μi x
= exp −
n
μi x
(5.6)
i=1
Example 5.7 (Analyzing Greedy Algorithms for the Assignment Problem) A group of n people is to be assigned to a set of n jobs, with one person assigned to each job. For a given set of n2 values Cij , i, j = 1, . . . , n, a cost Cij is incurred when person i is assigned to job j . The classical assignment problem is to determine the set of assignments that minimizes the sum of the n costs incurred. Rather than trying to determine the optimal assignment, let us consider two heuristic algorithms for solving this problem. The first heuristic is as follows. Assign person 1 to the job that results in the least cost. That is, person 1 is assigned to job j1 where C(1, j1 ) = minimumj C(1, j ). Now eliminate that job from consideration and assign person 2 to the job that results in the least cost. That is, person 2 is assigned to job j2 where C(2, j2 ) = minimumj =j1 C(2, j ). This procedure is then continued until all n persons are assigned. Since this procedure always selects the best job for the person under consideration, we will call it Greedy Algorithm A.
5.2.
The Exponential Distribution
293
The second algorithm, which we call Greedy Algorithm B, is a more “global” version of the first greedy algorithm. It considers all n2 cost values and chooses the pair i1 , j1 for which C(i, j ) is minimal. It then assigns person i1 to job j1 . It then eliminates all cost values involving either person i1 or job j1 [so that (n − 1)2 values remain] and continues in the same fashion. That is, at each stage it chooses the person and job that have the smallest cost among all the unassigned people and jobs. Under the assumption that the Cij constitute a set of n2 independent exponential random variables each having mean 1, which of the two algorithms results in a smaller expected total cost? Solution: Suppose first that Greedy Algorithm A is employed. Let Ci denote the cost associated with person i, i = 1, . . . , n. Now C1 is the minimum of n independent exponentials each having rate 1; so by Equation (5.6) it will be exponential with rate n. Similarly, C2 is the minimum of n − 1 independent exponentials with rate 1, and so is exponential with rate n − 1. Indeed, by the same reasoning Ci will be exponential with rate n − i + 1, i = 1, . . . , n. Thus, the expected total cost under Greedy Algorithm A is EA [total cost] = E[C1 + · · · + Cn ] =
n
1/i
i=1
Let us now analyze Greedy Algorithm B. Let Ci be the cost of the ith personjob pair assigned by this algorithm. Since C1 is the minimum of all the n2 values Cij , it follows from Equation (5.6) that C1 is exponential with rate n2 . Now, it follows from the lack of memory property of the exponential that the amounts by which the other Cij exceed C1 will be independent exponentials with rates 1. As a result, C2 is equal to C1 plus the minimum of (n − 1)2 independent exponentials with rate 1. Similarly, C3 is equal to C2 plus the minimum of (n − 2)2 independent exponentials with rate 1, and so on. Therefore, we see that E[C1 ] = 1/n2 , E[C2 ] = E[C1 ] + 1/(n − 1)2 , E[C3 ] = E[C2 ] + 1/(n − 2)2 , .. . E[Cj ] = E[Cj −1 ] + 1/(n − j + 1)2 , .. . E[Cn ] = E[Cn−1 ] + 1
294
5
The Exponential Distribution and the Poisson Process
Therefore, E[C1 ] = 1/n2 , E[C2 ] = 1/n2 + 1/(n − 1)2 , E[C3 ] = 1/n2 + 1/(n − 1)2 + 1/(n − 2)2 , .. . E[Cn ] = 1/n2 + 1/(n − 1)2 + 1/(n − 2)2 + · · · + 1 Adding up all the E[Ci ] yields that EB [total cost] = n/n2 + (n − 1)/(n − 1)2 + (n − 2)/(n − 2)2 + · · · + 1 =
n 1 i=1
i
The expected cost is thus the same for both greedy algorithms.
Let X1 , . . . , Xn be independent exponential random variables, with respective Equation (5.5), is that Xi is the rates λ1 , . . . , λn . A useful result, generalizing smallest of these with probability λi / j λj . This is shown as follows: P Xi = min Xj = P Xi < min Xj j =i
j
λi = n
j =1 λj
where the final equality uses Equation (5.5) along with the fact that minj =i Xj is exponential with rate j =i λj . Another important fact is that mini Xi and the rank ordering of the Xi are independent. To see why this is true, consider the conditional probability that Xi1 < Xi2 < · · · < Xin given that the minimal value is greater than t. Because mini Xi > t means that all the Xi are greater than t, it follows from the lack of memory property of exponential random variables that their remaining lives beyond t remain independent exponential random variables with their original rates. Consequently, ! ! P Xi1 < · · · < Xin ! min Xi > t = P Xi1 − t < · · · < Xin − t ! min Xi > t i
i
= P {Xi1 < · · · < Xin } which proves the result.
5.2.
The Exponential Distribution
295
Example 5.8 Suppose you arrive at a post office having two clerks at a moment when both are busy but there is no one else waiting in line. You will enter service when either clerk becomes free. If service times for clerk i are exponential with rate λi , i = 1, 2, find E[T ], where T is the amount of time that you spend in the post office. Solution: Let Ri denote the remaining service time of the customer with clerk i, i = 1, 2, and note, by the lack of memory property of exponentials, that R1 and R2 are independent exponential random variables with respective rates λ1 and λ2 . Conditioning on which of R1 or R2 is the smallest yields E[T ] = E[T |R1 < R2 ]P {R1 < R2 } + E[T |R2 R1 ]P {R2 R1 } = E[T |R1 < R2 ]
λ1 λ2 + E[T |R2 R1 ] λ1 + λ2 λ1 + λ2
Now, with S denoting your service time E[T |R1 < R2 ] = E[R1 + S|R1 < R2 ] = E[R1 |R1 < R2 ] + E[S|R1 < R2 ] = E[R1 |R1 < R2 ] + =
1 λ1
1 1 + λ1 + λ2 λ1
The final equation used that conditional on R1 < R2 the random variable R1 is the minimum of R1 and R2 and is thus exponential with rate λ1 + λ2 ; and also that conditional on R1 < R2 you are served by server 1. As we can show in a similar fashion that E[T |R2 R1 ] =
1 1 + λ1 + λ2 λ2
we obtain the result E[T ] =
3 λ1 + λ2
296
5
The Exponential Distribution and the Poisson Process
Another way to obtain E[T ] is to write T as a sum, take expectations, and then condition where needed. This approach yields E[T ] = E[min(R1 , R2 ) + S] = E[min(R1 , R2 )] + E[S] =
1 + E[S] λ1 + λ2
To compute E[S], we condition on which of R1 and R2 is smallest. E[S] = E[S|R1 < R2 ] =
2 λ1 + λ2
λ1 λ2 + E[S|R2 R1 ] λ1 + λ2 λ1 + λ2
Example 5.9 There are n cells in the body, of which cells 1, . . . , k are target cells. Associated with each cell is a weight, with wi being the weight associated with cell i, i = 1, . . . , n. The cells are destroyed one at a time in a random order which is such that if S is the current set of surviving cells then, independent of the order in which the cells not in S have been destroyed, the next cell killed is i, i ∈ S, with probability wi / j ∈S wj . In other words, the probability that a given surviving cell is the next one to be killed is the weight of that cell divided by the sum of the weights of all still surviving cells. Let A denote the total number of cells that are still alive at the moment when all the cells 1, 2, . . . , k have been killed, and find E[A]. Solution: Although it would be quite difficult to solve this problem by a direct combinatorial argument, a nice solution can be obtained by relating the order in which cells are killed to a ranking of independent exponential random variables. To do so, let X1 , . . . , Xn be independent exponential random variables, with Xi having rate wi , i = 1, . . . , n.Note that Xi will be the smallest of these exponentials with probability wi / j wj ; further, given that Xi is the smallest, Xr will be the next smallest with probability wr / j =i wj ; further, given that Xi and Xr are, respectively, the first andsecond smallest, Xs , s = i, r, will be the third smallest with probability ws / j =i,r wj ; and so on. Consequently, if we let Ij be the index of the j th smallest of X1 , . . . , Xn —so that XI1 < XI2 < · · · < XIn —then the order in which the cells are destroyed has the same distribution as I1 , . . . , In . So, let us suppose that the order in which the cells are killed is determined by the ordering of X1 , . . . , Xn . (Equivalently, we can suppose that all cells will eventually be killed, with cell i being killed at time Xi , i = 1, . . . , n.)
5.2.
The Exponential Distribution
297
If we let Aj equal 1 if cell j is still alive at the moment when all the cells 1, . . . , k have been killed, and let it equal 0 otherwise, then A=
n
Aj
j =k+1
Because cell j will be alive at the moment when all the cells 1, . . . , k have been killed if Xj is larger than all the values X1 , . . . , Xk , we see that for j > k E[Aj ] = P {Aj = 1} = P {Xj > max Xi } =
i=1,...,k
∞
P Xj > max Xi |Xj = x wj e−wj x dx i=1,...,k
0
=
∞
P {Xi < x for all i = 1, . . . , k}wj e−wj x dx
0
=
k ∞
(1 − e−wi x ) wj e−wj x dx
0
=
i=1 k 1
(1 − y wi /wj ) dy
0 i=1
where the final equality follows from the substitution y = e−wj x . Thus, we obtain the result E[A] =
n
k 1
j =k+1 0 i=1
(1 − y
wi /wj
) dy =
1
n k
(1 − y wi /wj ) dy
0 j =k+1 i=1
Example 5.10 Suppose that customers are in line to receive service that is provided sequentially by a server; whenever a service is completed, the next person in line enters the service facility. However, each waiting customer will only wait an exponentially distributed time with rate θ ; if its service has not yet begun by this time then it will immediately depart the system. These exponential times, one for each waiting customer, are independent. In addition, the service times are independent exponential random variables with rate μ. Suppose that someone is presently being served and consider the person who is nth in line. (a) Find Pn , the probability that this customer is eventually served. (b) Find Wn , the conditional expected amount of time this person spends waiting in line given that she is eventually served.
298
5
The Exponential Distribution and the Poisson Process
Solution: Consider the n + 1 random variables consisting of the remaining service time of the person in service along with the n additional exponential departure times with rate θ of the first n in line. (a) Given that the smallest of these n + 1 independent exponentials is the departure time of the nth person in line, the conditional probability that this person will be served is 0; on the other hand, given that this person’s departure time is not the smallest, the conditional probability that this person will be served is the same as if it were initially in position n − 1. Since the probability that a given departure time is the smallest of the n + 1 exponentials is θ/(nθ + μ), we obtain that Pn =
(n − 1)θ + μ Pn−1 nθ + μ
Using the preceding with n − 1 replacing n gives Pn =
(n − 1)θ + μ (n − 2)θ + μ (n − 2)θ + μ Pn−2 = Pn−2 nθ + μ (n − 1)θ + μ nθ + μ
Continuing in this fashion yields the result Pn =
θ +μ μ P1 = nθ + μ nθ + μ
(b) To determine an expression for Wn , we use the fact that the minimum of independent exponentials is, independent of their rank ordering, exponential with a rate equal to the sum of the rates. Since the time until the nth person in line enters service is the minimum of these n + 1 random variables plus the additional time thereafter, we see, upon using the lack of memory property of exponential random variables, that Wn =
1 + Wn−1 nθ + μ
Repeating the preceding argument with successively smaller values of n yields the solution: Wn =
n i=1
1 iθ + μ
5.2.4. Convolutions of Exponential Random Variables Let Xi , i = 1, . . . , n, be independent exponential random variables with respective rates λi , i = 1, . . . , n, and suppose that λi = λj for i = j . The random vari-
5.2.
The Exponential Distribution
299
able ni=1 Xi is said to be a hypoexponential random variable. To compute its probability density function, let us start with the case n = 2. Now, fX1 +X2 (t) =
t 0 t
=
fX1 (s)fX2 (t − s) ds λ1 e−λ1 s λ2 e−λ2 (t−s) ds
0
= λ1 λ2 e
−λ2 t
t
e−(λ1 −λ2 )s ds
0
=
λ1 λ2 e−λ2 t (1 − e−(λ1 −λ2 )t ) λ1 − λ2
=
λ2 λ1 λ2 e−λ2 t + λ1 e−λ1 t λ1 − λ2 λ2 − λ1
Using the preceding, a similar computation yields, when n = 3, fX1 +X2 +X3 (t) =
3
λi e
−λi t
j =i
i=1
λj λj − λi
which suggests the general result: fX1 + ··· +Xn (t) =
n
Ci,n λi e−λi t
i=1
where Ci,n =
j =i
λj λj − λi
We will now prove the preceding formula by induction on n. Since we have already established it for n = 2, assume it for n and consider n + 1 arbitrary independent exponentials Xi with distinct rates λi , i = 1, . . . , n + 1. If necessary, renumber X1 and Xn+1 so that λn+1 < λ1 . Now, fX1 + ··· +Xn+1 (t) = =
t
0 n i=1
fX1 + ··· +Xn (s)λn+1 e−λn+1 (t−s) ds
t
Ci,n 0
λi e−λi s λn+1 e−λn+1 (t−s) ds
300
5
The Exponential Distribution and the Poisson Process
=
n i=1
Ci,n
λi λn+1 λn+1 e−λn+1 t + λi e−λi t λi − λn+1 λn+1 − λi
= Kn+1 λn+1 e−λn+1 t +
n
Ci,n+1 λi e−λi t
(5.7)
i=1
where Kn+1 = ni=1 Ci,n λi /(λi − λn+1 ) is a constant that does not depend on t. But, we also have that t fX2 +···+Xn+1 (s)λ1 e−λ1 (t−s) ds fX1 +···+Xn+1 (t) = 0
which implies, by the same argument that resulted in Equation (5.7), that for a constant K1 fX1 +···+Xn+1 (t) = K1 λ1 e−λ1 t +
n+1
Ci,n+1 λi e−λi t
i=2
Equating these two expressions for fX1 +···+Xn+1 (t) yields Kn+1 λn+1 e−λn+1 t + C1,n+1 λ1 e−λ1 t = K1 λ1 e−λ1 t + Cn+1,n+1 λn+1 e−λn+1 t Multiplying both sides of the preceding equation by eλn+1 t and then letting t → ∞ yields [since e−(λ1 −λn+1 )t → 0 as t → ∞] Kn+1 = Cn+1,n+1 and this, using Equation (5.7), completes the induction proof. Thus, we have shown that if S = ni=1 Xi , then fs (t) =
n
Ci,n λi e−λi t
(5.8)
i=1
where Ci,n =
j =i
λj λj − λi
Integrating both sides of the expression for fS from t to ∞ yields that the tail distribution function of S is given by P {S > t} =
n i=1
Ci,n e−λi t
(5.9)
5.2.
The Exponential Distribution
301
Hence, we obtain from Equations (5.8) and (5.9) that rS (t), the failure rate function of S, is as follows: n Ci,n λi e−λi t rS (t) = i=1 n −λi t i=1 Ci,n e If we let λj = min(λ1 , . . . , λn ), then it follows, upon multiplying the numerator and denominator of rS (t) by eλj t , that lim rS (t) = λj
t→∞
From the preceding, we can conclude that the remaining lifetime of a hypoexponentially distributed item that has survived to age t is, for t large, approximately that of an exponentially distributed random variable with a rate equal to the minimum of the rates of the random variables whose sums make up the hypoexponential.
Remark Although 1=
∞
fS (t) dt =
0
n
Ci,n =
n i=1 j =i
i=1
λj λj − λi
it should not be thought that the Ci,n , i = 1, . . . , n are probabilities, because some of them will be negative. Thus, while the form of the hypoexponential density is similar to that of the hyperexponential density (see Example 5.6) these two random variables are very different.
Example 5.11 Let X1 , . . . , Xm be independent exponential random variables with respective rates λ1 , . . . , λm , where λi = λj when i = j . Let N be independent of these random variables and suppose that m n=1 Pn = 1, where Pn = P {N = n}. The random variable Y=
N
Xj
j =1
is said to be a Coxian random variable. Conditioning on N gives its density function: fY (t) =
m
fY (t|N = n)Pn
n=1
=
m n=1
fX1 + ··· +Xn (t|N = n)Pn
302
5
The Exponential Distribution and the Poisson Process
=
m
fX1 + ··· +Xn (t)Pn
n=1
=
m n=1
Pn
n
Ci,n λi e−λi t
i=1
Let r(n) = P {N = n|N n} If we interpret N as a lifetime measured in discrete time periods, then r(n) denotes the probability that an item will die in its nth period of use given that it has survived up to that time. Thus, r(n) is the discrete time analog of the failure rate function r(t), and is correspondingly referred to as the discrete time failure (or hazard) rate function. Coxian random variables often arise in the following manner. Suppose that an item must go through m stages of treatment to be cured. However, suppose that after each stage there is a probability that the item will quit the program. If we suppose that the amounts of time that it takes the item to pass through the successive stages are independent exponential random variables, and that the probability that an item that has just completed stage n quits the program is (independent of how long it took to go through the n stages) equal to r(n), then the total time that an item spends in the program is a Coxian random variable.
5.3.
The Poisson Process
5.3.1. Counting Processes A stochastic process {N (t), t 0} is said to be a counting process if N (t) represents the total number of “events” that occur by time t. Some examples of counting processes are the following: (a) If we let N (t) equal the number of persons who enter a particular store at or prior to time t, then {N (t), t 0} is a counting process in which an event corresponds to a person entering the store. Note that if we had let N (t) equal the number of persons in the store at time t, then {N (t), t 0} would not be a counting process (why not?). (b) If we say that an event occurs whenever a child is born, then {N (t), t 0} is a counting process when N (t) equals the total number of people who were born by time t. [Does N (t) include persons who have died by time t? Explain why it must.]
5.3.
The Poisson Process
303
(c) If N(t) equals the number of goals that a given soccer player scores by time t, then {N (t), t 0} is a counting process. An event of this process will occur whenever the soccer player scores a goal. From its definition we see that for a counting process N (t) must satisfy: (i) (ii) (iii) (iv)
N(t) 0. N(t) is integer valued. If s < t, then N (s) N (t). For s < t, N (t) − N (s) equals the number of events that occur in the interval (s, t].
A counting process is said to possess independent increments if the numbers of events that occur in disjoint time intervals are independent. For example, this means that the number of events that occur by time 10 [that is, N (10)] must be independent of the number of events that occur between times 10 and 15 [that is, N(15) − N(10)]. The assumption of independent increments might be reasonable for example (a), but it probably would be unreasonable for example (b). The reason for this is that if in example (b) N (t) is very large, then it is probable that there are many people alive at time t; this would lead us to believe that the number of new births between time t and time t + s would also tend to be large [that is, it does not seem reasonable that N (t) is independent of N (t + s) − N (t), and so {N (t), t 0} would not have independent increments in example (b)]. The assumption of independent increments in example (c) would be justified if we believed that the soccer player’s chances of scoring a goal today do not depend on “how he’s been going.” It would not be justified if we believed in “hot streaks” or “slumps.” A counting process is said to possess stationary increments if the distribution of the number of events that occur in any interval of time depends only on the length of the time interval. In other words, the process has stationary increments if the number of events in the interval (s, s + t) has the same distribution for all s. The assumption of stationary increments would only be reasonable in example (a) if there were no times of day at which people were more likely to enter the store. Thus, for instance, if there was a rush hour (say, between 12 P. M. and 1 P. M.) each day, then the stationarity assumption would not be justified. If we believed that the earth’s population is basically constant (a belief not held at present by most scientists), then the assumption of stationary increments might be reasonable in example (b). Stationary increments do not seem to be a reasonable assumption in example (c) since, for one thing, most people would agree that the soccer player would probably score more goals while in the age bracket 25–30 than he would while in the age bracket 35–40. It may, however, be reasonable over a smaller time horizon, such as one year.
304
5
The Exponential Distribution and the Poisson Process
5.3.2. Definition of the Poisson Process One of the most important counting processes is the Poisson process which is defined as follows:
Definition 5.1 The counting process {N (t), t 0} is said to be a Poisson process having rate λ, λ > 0, if (i) N (0) = 0. (ii) The process has independent increments. (iii) The number of events in any interval of length t is Poisson distributed with mean λt. That is, for all s, t 0 P {N (t + s) − N (s) = n} = e−λt
(λt)n , n!
n = 0, 1, . . .
Note that it follows from condition (iii) that a Poisson process has stationary increments and also that E[N (t)] = λt which explains why λ is called the rate of the process. To determine if an arbitrary counting process is actually a Poisson process, we must show that conditions (i), (ii), and (iii) are satisfied. Condition (i), which simply states that the counting of events begins at time t = 0, and condition (ii) can usually be directly verified from our knowledge of the process. However, it is not at all clear how we would determine that condition (iii) is satisfied, and for this reason an equivalent definition of a Poisson process would be useful. As a prelude to giving a second definition of a Poisson process we shall define the concept of a function f (·) being o(h).
Definition 5.2 The function f (·) is said to be o(h) if f (h) =0 h→0 h lim
Example 5.12 (i) The function f (x) = x 2 is o(h) since f (h) h2 = lim = lim h = 0 h→0 h h→0 h h→0 lim
5.3.
The Poisson Process
305
(ii) The function f (x) = x is not o(h) since lim
h→0
f (h) h = lim = lim 1 = 1 = 0 h→0 h h→0 h
(iii) If f (·) is o(h) and g(·) is o(h), then so is f (·) + g(·). This follows since lim
h→0
f (h) + g(h) f (h) g(h) = lim + lim =0+0=0 h→0 h h→0 h h
(iv) If f (·) is o(h), then so is g(·) = cf (·). This follows since f (h) cf (h) = c lim =c·0=0 h→0 h h lim
(v) From (iii) and (iv) it follows that any finite linear combination of functions, each of which is o(h), is o(h). In order for the function f (·) to be o(h) it is necessary that f (h)/ h go to zero as h goes to zero. But if h goes to zero, the only way for f (h)/ h to go to zero is for f (h) to go to zero faster than h does. That is, for h small, f (h) must be small compared with h. We are now in a position to give an alternate definition of a Poisson process.
Definition 5.3 The counting process {N (t), t 0} is said to be a Poisson process having rate λ, λ > 0, if (i) (ii) (iii) (iv)
N(0) = 0. The process has stationary and independent increments. P {N (h) = 1} = λh + o(h). P {N (h) 2} = o(h).
Theorem 5.1 Definitions 5.1 and 5.3 are equivalent. Proof We show that Definition 5.3 implies Definition 5.1, and leave it to you to prove the reverse. To start, fix u 0 and let
g(t) = E[exp{−uN (t)}] We derive a differential equation for g(t) as follows: g(t + h) = E[exp{−uN(t + h)}] = E[exp{−uN(t)} exp{−u(N (t + h) − N (t))}]
306
5
The Exponential Distribution and the Poisson Process
= E[exp{−uN(t)}]E[exp{−u(N (t + h) − N (t))}] by independent increments = g(t) E[exp{−uN (h)}]
by stationary increments
(5.10)
Now, assumptions (iii) and (iv) imply that P {N (h) = 0} = 1 − λh + o(h) Hence, conditioning on whether N (h) = 0 or N (h) = 1 or N (h) 2 yields E[exp{−uN (h)}] = 1 − λh + o(h) + e−u (λh + o(h)) + o(h) = 1 − λh + e−u λh + o(h)
(5.11)
Therefore, from Equations (5.10) and (5.11) we obtain that g(t + h) = g(t)(1 − λh + e−u λh) + o(h) implying that g(t + h) − g(t) o(h) = g(t)λ(e−u − 1) + h h Letting h → 0 gives g (t) = g(t)λ(e−u − 1) or, equivalently, g (t) = λ(e−u − 1) g(t) Integrating, and using g(0) = 1, shows that log(g(t)) = λt (e−u − 1) or g(t) = exp{λt (e−u − 1)} −u
That is, the Laplace transform of N (t) evaluated at u is eλt (e −1) . Since that is also the Laplace transform of a Poisson random variable with mean λt, the result follows from the fact that the distribution of a nonnegative random variable is uniquely determined by its Laplace transform.
5.3.
The Poisson Process
307
Figure 5.1.
Remarks (i) The result that N (t) has a Poisson distribution is a consequence of the Poisson approximation to the binomial distribution (see Section 2.2.4). To see this, subdivide the interval [0, t] into k equal parts where k is very large (Figure 5.1). Now it can be shown using axiom (iv) of Definition 5.3 that as k increases to ∞ the probability of having two or more events in any of the k subintervals goes to 0. Hence, N (t) will (with a probability going to 1) just equal the number of subintervals in which an event occurs. However, by stationary and independent increments this number will have a binomial distribution with parameters k and p = λt/k + o(t/k). Hence, by the Poisson approximation to the binomial we see by letting k approach ∞ that N (t) will have a Poisson distribution with mean equal to / . t to(t/k) t lim k λ + o = λt + lim k→∞ k→∞ t/k k k = λt by using the definition of o(h) and the fact that t/k → 0 as k → ∞. (ii) The explicit assumption that the process has stationary increments can be eliminated from Definition 5.3 provided that we change assumptions (iii) and (iv) to require that for any t the probability of one event in the interval (t, t + h) is λh + o(h) and the probability of two or more events in that interval is o(h). That is, assumptions (ii), (iii), and (iv) of Definition 5.3 can be replaced by (iii) The process has independent increments. (iv) P {N(t + h) − N (t) = 1} = λh + o(h). (v) P {N(t + h) − N (t) 2} = o(h). 5.3.3. Interarrival and Waiting Time Distributions Consider a Poisson process, and let us denote the time of the first event by T1 . Further, for n > 1, let Tn denote the elapsed time between the (n − 1)st and the nth event. The sequence {Tn , n = 1, 2, . . .} is called the sequence of interarrival times. For instance, if T1 = 5 and T2 = 10, then the first event of the Poisson process would have occurred at time 5 and the second at time 15. We shall now determine the distribution of the Tn . To do so, we first note that the event {T1 > t} takes place if and only if no events of the Poisson process occur
308
5
The Exponential Distribution and the Poisson Process
in the interval [0, t] and thus, P {T1 > t} = P {N (t) = 0} = e−λt Hence, T1 has an exponential distribution with mean 1/λ. Now, P {T2 > t} = E[P {T2 > t|T1 }] However, P {T2 > t | T1 = s} = P {0 events in (s, s + t] | T1 = s} = P {0 events in (s, s + t]} = e−λt
(5.12)
where the last two equations followed from independent and stationary increments. Therefore, from Equation (5.12) we conclude that T2 is also an exponential random variable with mean 1/λ and, furthermore, that T2 is independent of T1 . Repeating the same argument yields the following.
Proposition 5.1 Tn , n = 1, 2, . . . , are independent identically distributed exponential random variables having mean 1/λ.
Remark The proposition should not surprise us. The assumption of stationary and independent increments is basically equivalent to asserting that, at any point in time, the process probabilistically restarts itself. That is, the process from any point on is independent of all that has previously occurred (by independent increments), and also has the same distribution as the original process (by stationary increments). In other words, the process has no memory, and hence exponential interarrival times are to be expected. Another quantity of interest is Sn , the arrival time of the nth event, also called the waiting time until the nth event. It is easily seen that Sn =
n
Ti ,
n1
i=1
and hence from Proposition 5.1 and the results of Section 2.2 it follows that Sn has a gamma distribution with parameters n and λ. That is, the probability density of Sn is given by fSn (t) = λe−λt
(λt)n−1 , (n − 1)!
t 0
(5.13)
Equation (5.13) may also be derived by noting that the nth event will occur prior to or at time t if and only if the number of events occurring by time t is at least n.
5.3.
The Poisson Process
309
That is, N (t) n
Sn t
⇔
Hence, FSn (t) = P {Sn t} = P {N (t) n} =
∞
e−λt
j =n
(λt)j j!
which, upon differentiation, yields fSn (t) = −
∞ j =n
= λe−λt = λe−λt
λe−λt
∞
(λt)j −λt (λt)j −1 + λe j! (j − 1)! j =n
(λt)n−1 + (n − 1)!
∞
λe−λt
∞
(λt)j −1 −λt (λt)j − λe (j − 1)! j! j =n
j =n+1
(λt)n−1 (n − 1)!
Example 5.13 Suppose that people immigrate into a territory at a Poisson rate λ = 1 per day.
(a) What is the expected time until the tenth immigrant arrives? (b) What is the probability that the elapsed time between the tenth and the eleventh arrival exceeds two days? Solution: (a) E[S10 ] = 10/λ = 10 days. (b) P {T11 > 2} = e−2λ = e−2 ≈ 0.133.
Proposition 5.1 also gives us another way of defining a Poisson process. Suppose we start with a sequence {Tn , n 1} of independent identically distributed exponential random variables each having mean 1/λ. Now let us define a counting process by saying that the nth event of this process occurs at time S n ≡ T1 + T2 + · · · + Tn The resultant counting process {N (t), t 0}∗ will be Poisson with rate λ. ∗ A formal definition of N (t) is given by N (t) ≡ max{n: S t} where S ≡ 0. n 0
310
5
The Exponential Distribution and the Poisson Process
Remark Another way of obtaining the density function of Sn is to note that because Sn is the time of the nth event, P {t < Sn < t + h} = P {N (t) = n − 1, one event in (t, t + h)} + o(h) = P {N (t) = n − 1}P {one event in (t, t + h)} + o(h) = e−λt
(λt)n−1 [λh + o(h)] + o(h) (n − 1)!
= λe−λt
(λt)n−1 h + o(h) (n − 1)!
where the first equality uses the fact that the probability of 2 or more events in (t, t + h) is o(h). If we now divide both sides of the preceding equation by h and then let h → 0, we obtain fSn (t) = λe−λt
(λt)n−1 (n − 1)!
5.3.4. Further Properties of Poisson Processes Consider a Poisson process {N (t), t 0} having rate λ, and suppose that each time an event occurs it is classified as either a type I or a type II event. Suppose further that each event is classified as a type I event with probability p or a type II event with probability 1 − p, independently of all other events. For example, suppose that customers arrive at a store in accordance with a Poisson process having rate λ; and suppose that each arrival is male with probability 12 and female with probability 12 . Then a type I event would correspond to a male arrival and a type II event to a female arrival. Let N1 (t) and N2 (t) denote respectively the number of type I and type II events occurring in [0, t]. Note that N (t) = N1 (t) + N2 (t).
Proposition 5.2 {N1 (t), t 0} and {N2 (t), t 0} are both Poisson processes having respective rates λp and λ(1 − p). Furthermore, the two processes are independent.
Proof It is easy to verify that {N1 (t), t 0} is a Poisson process with rate λp by verifying that it satisfies Definition 5.3. • N1 (0) = 0 follows from the fact that N (0) = 0. • It is easy to see that {N1 (t), t 0} inherits the stationary and independent increment properties of the process {N (t), t 0}. This is true because the distribution of the number of type I events in an interval can be obtained
5.3.
The Poisson Process
311
by conditioning on the number of events in that interval, and the distribution of this latter quantity depends only on the length of the interval and is independent of what has occurred in any nonoverlapping interval. • P {N1 (h) = 1} = P {N1 (h) = 1 | N (h) = 1}P {N (h) = 1} + P {N1 (h) = 1 | N (h) 2}P {N (h) 2} = p(λh + o(h)) + o(h) = λph + o(h) • P {N1 (h) 2} P {N (h) 2} = o(h) Thus we see that {N1 (t), t 0} is a Poisson process with rate λp and, by a similar argument, that {N2 (t), t 0} is a Poisson process with rate λ(1 − p). Because the probability of a type I event in the interval from t to t + h is independent of all that occurs in intervals that do not overlap (t, t + h), it is independent of knowledge of when type II events occur, showing that the two Poisson processes are independent. (For another way of proving independence, see Example 3.20.)
Example 5.14 If immigrants to area A arrive at a Poisson rate of ten per
1 , then what week, and if each immigrant is of English descent with probability 12 is the probability that no people of English descent will emigrate to area A during the month of February?
Solution: By the previous proposition it follows that the number of Englishmen emigrating to area A during the month of February is Poisson distributed 1 −10/3 . = 10 with mean 4 · 10 · 12 3 . Hence the desired probability is e
Example 5.15 Suppose nonnegative offers to buy an item that you want to sell arrive according to a Poisson process with rate λ. Assume that each offer is the value of a continuous random variable having density function f (x). Once the offer is presented to you, you must either accept it or reject it and wait for the next offer. We suppose that you incur costs at a rate c per unit time until the item is sold, and that your objective is to maximize your expected total return, where the total return is equal to the amount received minus the total cost incurred. Suppose you employ the policy of accepting the first offer that is greater than some specified value y. (Such a type of policy, which we call a y-policy, can be shown to be optimal.) What is the best value of y? Solution: Let us compute the expected total return when you use the ypolicy, and then choose the value of y that maximizes this quantity. Let X de ∞ note the value of a random offer, and let F¯ (x) = P {X > x} = x f (u) du be its tail distribution function. Because each offer will be greater than y with probability F¯ (y), it follows that such offers occur according to a Poisson process with rate λF¯ (y). Hence, the time until an offer is accepted is an exponential
312
5
The Exponential Distribution and the Poisson Process
random variable with rate λF¯ (y). Letting R(y) denote the total return from the policy that accepts the first offer that is greater than y, we have E[R(y)] = E[accepted offer] − cE[time to accept] c = E[X|X > y] − ¯ λF (y) ∞ c xfX|X>y (x) dx − = ¯ λ F (y) 0 ∞ f (x) c x = dx − λF¯ (y) F¯ (y) y ∞ y xf (x) dx − c/λ = F¯ (y) Differentiation yields that d E[R(y)] = 0 ⇔ −F¯ (y)yf (y) + dy
∞
xf (x) dx −
y
c f (y) = 0 λ
Therefore, the optimal value of y satisfies ∞ c y F¯ (y) = xf (x) dx − λ y or ∞ ∞ c y f (x) dx = xf (x) dx − λ y y or
y
∞
(x − y)f (x) dx =
c λ
(5.14)
We now argue that the left-hand side of the preceding is a nonincreasing function of y. To do so, note that, with a + defined to equal a if a > 0 or to equal 0 otherwise, we have ∞ (x − y)f (x) dx = E[(X − y)+ ] y
− y)+
is a nonincreasing function of y, so is its expectation, thus Because (X showing that the left hand side of Equation (5.14) is a nonincreasing function of y. Consequently, if E[X] < c/λ—in which case there is no solution of Equation (5.14)—then it is optimal to accept any offer; otherwise, the optimal value y is the unique solution of Equation (5.14). It follows from Proposition 5.2 that if each of a Poisson number of individuals is independently classified into one of two possible groups with respective prob-
5.3.
The Poisson Process
313
abilities p and 1 − p, then the number of individuals in each of the two groups will be independent Poisson random variables. Because this result easily generalizes to the case where the classification is into any one of r possible groups, we have the following application to a model of employees moving about in an organization.
Example 5.16 Consider a system in which individuals at any time are classified as being in one of r possible states, and assume that an individual changes states in accordance with a Markov chain having transition probabilities Pij , i, j = 1, . . . , r. That is, if an individual is in state i during a time period then, independently of its previous states, it will be in state j during the next time period with probability Pij . The individuals are assumed to move through the system independently of each other. Suppose that the numbers of people initially in states 1, 2, . . . , r are independent Poisson random variables with respective means λ1 , λ2 , . . . , λr . We are interested in determining the joint distribution of the numbers of individuals in states 1, 2, . . . , r at some time n. Solution: For fixed i, let Nj (i), j = 1, . . . , r denote the number of those individuals, initially in state i, that are in state j at time n. Now each of the (Poisson distributed) number of people initially in state i will, independently of each other, be in state j at time n with probability Pijn , where Pijn is the n-stage transition probability for the Markov chain having transition probabilities Pij . Hence, the Nj (i), j = 1, . . . , r will be independent Poisson random variables with respective means λi Pijn , j = 1, . . . , r. Because the sum of independent Poisson random variables is itself a Poisson random variable, it follows that N the number of individuals in state j at time n—namely ri=1 j (i)—will be independent Poisson random variables with respective means i λi Pijn , for j = 1, . . . , r.
Example 5.17 (The Coupon Collecting Problem) There are m different types of coupons. Each time a person collects a coupon it is, independently of ones p = 1. Let N previously obtained, a type j coupon with probability pj , m j j =1 denote the number of coupons one needs to collect in order to have a complete collection of at least one of each type. Find E[N ]. Solution: If we let Nj denote the number one must collect to obtain a type j coupon, then we can express N as N = max Nj 1j m
However, even though each Nj is geometric with parameter pj , the foregoing representation of N is not that useful, because the random variables Nj are not independent.
314
5
The Exponential Distribution and the Poisson Process
We can, however, transform the problem into one of determining the expected value of the maximum of independent random variables. To do so, suppose that coupons are collected at times chosen according to a Poisson process with rate λ = 1. Say that an event of this Poisson process is of type j , 1 j m, if the coupon obtained at that time is a type j coupon. If we now let Nj (t) denote the number of type j coupons collected by time t, then it follows from Proposition 5.2 that {Nj (t), t 0}, j = 1, . . . , m are independent Poisson processes with respective rates λpj = pj . Let Xj denote the time of the first event of the j th process, and let X = max Xj 1j m
denote the time at which a complete collection is amassed. Since the Xj are independent exponential random variables with respective rates pj , it follows that P {X < t} = P {max Xj < t} = P {Xj < t, for j = 1, . . . , m} =
m
(1 − e−pj t )
j =1
Therefore, E[X] =
∞
0
=
∞
P {X > t} dt 1−
0
m
(1 − e−pj t ) dt
(5.15)
j =1
It remains to relate E[X], the expected time until one has a complete set, to E[N], the expected number of coupons it takes. This can be done by letting Ti denote the ith interarrival time of the Poisson process that counts the number of coupons obtained. Then it is easy to see that X=
N
Ti
i=1
Since the Ti are independent exponentials with rate 1, and N is independent of the Ti , we see that E[X|N ] = N E[Ti ] = N
5.3.
The Poisson Process
315
Therefore, E[X] = E[N ] and so E[N ] is as given in Equation (5.15). Let us now compute the expected number of types that appear only once in the complete collection. Letting Ii equal 1 if there is only a single type i coupon in the final set, and letting it equal 0 otherwise, we thus want , m m E Ii = E[Ii ] i=1
i=1
=
m
P {Ii = 1}
i=1
Now there will be a single type i coupon in the final set if a coupon of each type has appeared before the second coupon of type i is obtained. Thus, letting Si denote the time at which the second type i coupon is obtained, we have P {Ii = 1} = P {Xj < Si , for all j = i} Using that Si has a gamma distribution with parameters (2, pi ), this yields ∞ P {Xj < Si for all j = i|Si = x}pi e−pi x pi x dx P {Ii = 1} =
0
0
∞
=
P {Xj < x, for all j = i}pi2 x e−pi x dx
∞
= 0
(1 − e−pj x )pi2 xe−pi x dx
j =i
Therefore, we have the result , m - m ∞ E Ii = (1 − e−pj x )pi2 xe−pi x dx 0
i=1
=
i=1 j =i ∞
x 0
m j =1
(1 − e−pj x )
m i=1
pi2
e−pi x dx 1 − e−pi x
The next probability calculation related to Poisson processes that we shall determine is the probability that n events occur in one Poisson process before m events have occurred in a second and independent Poisson process. More formally let {N1 (t), t 0} and {N2 (t), t 0} be two independent Poisson processes having respective rates λ1 and λ2 . Also, let Sn1 denote the time of the nth event of
316
5
The Exponential Distribution and the Poisson Process
2 the time of the mth event of the second process. We seek the first process, and Sm ; < 2 P Sn1 < Sm
Before attempting to calculate this for general n and m, let us consider the special case n = m = 1. Since S11 , the time of the first event of the N1 (t) process, and S12 , the time of the first event of the N2 (t) process, are both exponentially distributed random variables (by Proposition 5.1) with respective means 1/λ1 and 1/λ2 , it follows from Section 5.2.3 that ; < P S11 < S12 =
λ1 λ1 + λ2
(5.16)
Let us now consider the probability that two events occur in the N1 (t) process before a single event has occurred in the N2 (t) process. That is, P {S21 < S12 }. To calculate this we reason as follows: In order for the N1 (t) process to have two events before a single event occurs in the N2 (t) process, it is first necessary for the initial event that occurs to be an event of the N1 (t) process [and this occurs, by Equation (5.16), with probability λ1 /(λ1 + λ2 )]. Now given that the initial event is from the N1 (t) process, the next thing that must occur for S21 to be less than S12 is for the second event also to be an event of the N1 (t) process. However, when the first event occurs both processes start all over again (by the memoryless property of Poisson processes) and hence this conditional probability is also λ1 /(λ1 + λ2 ); thus, the desired probability is given by 2 < ; 1 λ1 2 P S2 < S1 = λ1 + λ2 In fact this reasoning shows that each event that occurs is going to be an event of the N1 (t) process with probability λ1 /(λ1 + λ2 ) or an event of the N2 (t) process with probability λ2 /(λ1 + λ2 ), independent of all that has previously occurred. In other words, the probability that the N1 (t) process reaches n before the N2 (t) process reaches m is just the probability that n heads will appear before m tails if one flips a coin having probability p = λ1 /(λ1 + λ2 ) of a head appearing. But by noting that this event will occur if and only if the first n + m − 1 tosses result in n or more heads, we see that our desired probability is given by n + m − 1 λ1 k λ2 n+m−1−k ; < n+m−1 2 P Sn1 < Sm = k λ1 + λ2 λ1 + λ2 k=n
5.3.5. Conditional Distribution of the Arrival Times Suppose we are told that exactly one event of a Poisson process has taken place by time t, and we are asked to determine the distribution of the time at which the event occurred. Now, since a Poisson process possesses stationary and inde-
5.3.
The Poisson Process
317
pendent increments it seems reasonable that each interval in [0, t] of equal length should have the same probability of containing the event. In other words, the time of the event should be uniformly distributed over [0, t]. This is easily checked since, for s t, P {T1 < s, N (t) = 1} P {T1 < s|N (t) = 1} = P {N (t) = 1} =
P {1 event in [0, s), 0 events in [s, t]} P {N (t) = 1}
=
P {1 event in [0, s)}P {0 events in [s, t]} P {N (t) = 1}
λse−λs e−λ(t−s) λte−λt s = t This result may be generalized, but before doing so we need to introduce the concept of order statistics. Let Y1 , Y2 , . . . , Yn be n random variables. We say that Y(1) , Y(2) , . . . , Y(n) are the order statistics corresponding to Y1 , Y2 , . . . , Yn if Y(k) is the kth smallest value among Y1 , . . . , Yn , k = 1, 2, . . . , n. For instance if n = 3 and Y1 = 4, Y2 = 5, Y3 = 1 then Y(1) = 1, Y(2) = 4, Y(3) = 5. If the Yi , i = 1, . . . , n, are independent identically distributed continuous random variables with probability density f , then the joint density of the order statistics Y(1) , Y(2) , . . . , Y(n) is given by =
f (y1 , y2 , . . . , yn ) = n!
n
f (yi ),
y1 < y2 < · · · < yn
i=1
The preceding follows since (i) (Y(1) , Y(2) , . . . , Y(n) ) will equal (y1 , y2 , . . . , yn ) if (Y1 , Y2 , . . . , Yn ) is equal to any of the n! permutations of (y1 , y2 , . . . , yn ); and (ii) 1 the probability 1 density that (Y1 , Y2 , . . . , Yn ) is equal to yi1 , . . . , yin is n n j =1 f (yij ) = j =1 f (yj ) when i1 , . . . , in is a permutation of 1, 2, . . . , n. If the Yi , i = 1, . . . , n, are uniformly distributed over (0, t), then we obtain from the preceding that the joint density function of the order statistics Y(1) , Y(2) , . . . , Y(n) is n! 0 < y1 < y2 < · · · < yn < t f (y1 , y2 , . . . , yn ) = n , t We are now ready for the following useful theorem.
318
5
The Exponential Distribution and the Poisson Process
Theorem 5.2 Given that N (t) = n, the n arrival times S1 , . . . , Sn have the same distribution as the order statistics corresponding to n independent random variables uniformly distributed on the interval (0, t).
Proof To obtain the conditional density of S1 , . . . , Sn given that N (t) = n
note that for 0 < S1 < · · · < Sn < t the event that S1 = s1 , S2 = s2 , . . . , Sn = sn , N (t) = n is equivalent to the event that the first n + 1 interarrival times satisfy T1 = s1 , T2 = s2 − s1 , . . . , Tn = sn − sn−1 , Tn+1 > t − sn . Hence, using Proposition 5.1, we have that the conditional joint density of S1 , . . . , Sn given that N(t) = n is as follows: f (s1 , . . . , sn | n) =
which proves the result.
f (s1 , . . . , sn , n) P {N (t) = n}
=
λe−λs1 λe−λ(s2 −s1 ) · · · λe−λ(sn −sn−1 ) e−λ(t−sn ) e−λt (λt)n /n!
=
n! , tn
0 < s1 < · · · < sn < t
Remark The preceding result is usually paraphrased as stating that, under the condition that n events have occurred in (0, t), the times S1 , . . . , Sn at which events occur, considered as unordered random variables, are distributed independently and uniformly in the interval (0, t).
Application of Theorem 5.2 (Sampling a Poisson Process)
In Proposition 5.2 we showed that if each event of a Poisson process is independently classified as a type I event with probability p and as a type II event with probability 1 − p then the counting processes of type I and type II events are independent Poisson processes with respective rates λp and λ(1 − p). Suppose now, however, that there are k possible types of events and that the probability that an event is classified as a type i event, i = 1, . . . , k, depends on the time the event occurs. Specifically, suppose that if an event occurs at time y then it will be classified as a type i event, independently of anything that has previously oc curred, with probability Pi (y), i = 1, . . . , k where ki=1 Pi (y) = 1. Upon using Theorem 5.2 we can prove the following useful proposition.
Proposition 5.3 If Ni (t), i = 1, . . . , k, represents the number of type i events occurring by time t then Ni (t), i = 1, . . . , k, are independent Poisson random variables having means t E[Ni (t)] = λ Pi (s) ds 0
Before proving this proposition, let us first illustrate its use.
5.3.
The Poisson Process
319
Example 5.18 (An Infinite Server Queue) Suppose that customers arrive at a service station in accordance with a Poisson process with rate λ. Upon arrival the customer is immediately served by one of an infinite number of possible servers, and the service times are assumed to be independent with a common distribution G. What is the distribution of X(t), the number of customers that have completed service by time t? What is the distribution of Y (t), the number of customers that are being served at time t? To answer the preceding questions let us agree to call an entering customer a type I customer if he completes his service by time t and a type II customer if he does not complete his service by time t. Now, if the customer enters at time s, s t, then he will be a type I customer if his service time is less than t − s. Since the service time distribution is G, the probability of this will be G(t − s). Similarly, a customer entering at time s, s t, will be a type II customer with ¯ − s) = 1 − G(t − s). Hence, from Proposition 5.3 we obtain that probability G(t the distribution of X(t), the number of customers that have completed service by time t, is Poisson distributed with mean E[X(t)] = λ
t
G(t − s) ds = λ
t
G(y) dy
0
(5.17)
0
Similarly, the distribution of Y (t), the number of customers being served at time t is Poisson with mean E[Y (t)] = λ
t
¯ − s) ds = λ G(t
0
t
¯ G(y) dy
(5.18)
0
Furthermore, X(t) and Y (t) are independent. Suppose now that we are interested in computing the joint distribution of Y (t) and Y (t + s)—that is, the joint distribution of the number in the system at time t and at time t + s. To accomplish this, say that an arrival is type 1: type 2: type 3: type 4:
if he arrives before time t and completes service between t and t + s, if he arrives before t and completes service after t + s, if he arrives between t and t + s and completes service after t + s, otherwise.
Hence an arrival at time y will be type i with probability Pi (y) given by P1 (y) = P2 (y) =
G(t + s − y) − G(t − y),
if y < t
0,
otherwise
¯ + s − y), G(t
if y < t
0,
otherwise
320
5
The Exponential Distribution and the Poisson Process
P3 (y) =
¯ + s − y), G(t
if t < y < t + s
0,
otherwise
P4 (y) = 1 − P1 (y) − P2 (y) − P3 (y) Hence, if Ni = Ni (s + t), i = 1, 2, 3, denotes the number of type i events that occur, then from Proposition 5.3, Ni , i = 1, 2, 3, are independent Poisson random variables with respective means t+s E[Ni ] = λ Pi (y) dy, i = 1, 2, 3 0
Because Y (t) = N1 + N2 , Y (t + s) = N2 + N3 it is now an easy matter to compute the joint distribution of Y (t) and Y (t + s). For instance, Cov[Y (t), Y (t + s)] = Cov(N1 + N2 , N2 + N3 ) = Cov(N2 , N2 )
by independence of N1 , N2 , N3
= Var(N2 ) t t ¯ + s − y) dy = λ ¯ + s) du =λ G(t G(u 0
0
where the last equality follows since the variance of a Poisson random variable equals its mean, and from the substitution u = t − y. Also, the joint distribution of Y (t) and Y (t + s) is as follows: P {Y (t) = i, Y (t + s) = j } = P {N1 + N2 = i, N2 + N3 = j } =
min(i,j )
P {N2 = l, N1 = i − l, N3 = j − l}
l=0
=
min(i,j )
P {N2 = l}P {N1 = i − l}P {N3 = j − l}
l=0
Example 5.19 (Minimizing the Number of Encounters) Suppose that cars enter a one-way highway in accordance with a Poisson process with rate λ. The cars enter at point a and depart at point b (see Figure 5.2). Each car travels at a
5.3.
The Poisson Process
321
Figure 5.2. Cars enter at point a and depart at b.
constant speed that is randomly determined, independently from car to car, from the distribution G. When a faster car encounters a slower one, it passes it with no time being lost. If your car enters the highway at time s and you are able to choose your speed, what speed minimizes the expected number of encounters you will have with other cars, where we say that an encounter occurs each time your car either passes or is passed by another car? Solution: We will show that for large s the speed that minimizes the expected number of encounters is the median of the speed distribution G. To see this, suppose that the speed x is chosen. Let d = b − a denote the length of the road. Upon choosing the speed x, it follows that your car will enter the road at time s and will depart at time s + t0 , where t0 = d/x is the travel time. Now, the other cars enter the road according to a Poisson process with rate λ. Each of them chooses a speed X according to the distribution G, and this results in a travel time T = d/X. Let F denote the distribution of travel time T . That is, ¯ F (t) = P {T t} = P {d/X t} = P {X d/t} = G(d/t) Let us say that an event occurs at time t if a car enters the highway at time t. Also, say that the event is a type 1 event if it results in an encounter with your car. Now, your car will enter the road at time s and will exit at time s + t0 . Hence, a car will encounter your car if it enters before s and exits after s + t0 (in which case your car will pass it on the road) or if it enters after s but exits before s + t0 (in which case it will pass your car). As a result, a car that enters the road at time t will encounter your car if its travel time T is such that t + T > s + t0 , t + T < s + t0 ,
if t < s if s < t < s + t0
From the preceding, we see that an event at time t will, independently of other events, be a type 1 event with probability p(t) given by ⎧ if t < s ⎨P {t + T > s + t0 } = F¯ (s + t0 − t), p(t) = P {t + T < s + t0 } = F (s + t0 − t), if s < t < s + t0 ⎩ 0, if t > s + t0
322
5
The Exponential Distribution and the Poisson Process
Since events (that is, cars entering the road) are occurring according to a Poisson process it follows, upon applying Proposition 5.3, that the total number of type 1 events that ever occurs is Poisson with mean ∞ s s+t0 λ p(t) dt = λ F (s + t0 − t) dt F¯ (s + t0 − t) dt + λ 0
0
=λ
s+t0
F¯ (y) dy + λ
s t0
F (y) dy 0
t0
To choose the value of t0 that minimizes the preceding quantity, we differentiate. This gives ∞ d λ p(t) dt = λ{F¯ (s + t0 ) − F¯ (t0 ) + F (t0 )} dt0 0 Setting this equal to 0, and using that F¯ (s + t0 ) ≈ 0 when s is large, we see that the optimal travel time t0 is such that F (t0 ) − F¯ (t0 ) = 0 or F (t0 ) − [1 − F (t0 )] = 0 or F (t0 ) =
1 2
That is, the optimal travel time is the median of the travel time distribution. Since the speed X is equal to the distance d divided by the travel time T , it follows that the optimal speed x0 = d/t0 is such that F (d/x0 ) =
1 2
Since ¯ 0) F (d/x0 ) = G(x ¯ 0 ) = 1 , and so the optimal speed is the median of the distribuwe see that G(x 2 tion of speeds. Summing up, we have shown that for any speed x the number of encounters with other cars will be a Poisson random variable, and the mean of this Poisson will be smallest when the speed x is taken to be the median of the distribution G.
Example 5.20 (Tracking the Number of HIV Infections) There is a relatively long incubation period from the time when an individual becomes infected with the HIV virus, which causes AIDS, until the symptoms of the disease appear.
5.3.
The Poisson Process
323
As a result, it is difficult for public health officials to be certain of the number of members of the population that are infected at any given time. We will now present a first approximation model for this phenomenon, which can be used to obtain a rough estimate of the number of infected individuals. Let us suppose that individuals contract the HIV virus in accordance with a Poisson process whose rate λ is unknown. Suppose that the time from when an individual becomes infected until symptoms of the disease appear is a random variable having a known distribution G. Suppose also that the incubation times of different infected individuals are independent. Let N1 (t) denote the number of individuals who have shown symptoms of the disease by time t. Also, let N2 (t) denote the number who are HIV positive but have not yet shown any symptoms by time t. Now, since an individual who contracts the virus at time s will have symptoms by time t with probability G(t − s) ¯ − s), it follows from Proposition 5.3 that N1 (t) and will not with probability G(t and N2 (t) are independent Poisson random variables with respective means t t G(t − s) ds = λ G(y) dy E[N1 (t)] = λ 0
and
E[N2 (t)] = λ
0 t
¯ − s) ds = λ G(t
0
t
¯ G(y) dy
0
Now, if we knew λ, then we could use it to estimate N2 (t), the number of individuals infected but without any outward symptoms at time t, by its mean value E[N2 (t)]. However, since λ is unknown, we must first estimate it. Now, we will presumably know the value of N1 (t), and so we can use its known value as an estimate of its mean E[N1 (t)]. That is, if the number of individuals who have exhibited symptoms by time t is n1 , then we can estimate that t G(y) dy n1 ≈ E[N1 (t)] = λ 0
Therefore, we can estimate λ by the quantity λˆ given by = t λˆ = n1 G(y) dy 0
Using this estimate of λ, we can estimate the number of infected but symptomless individuals at time t by t ¯ estimate of N2 (t) = λˆ G(y) dy 0
t ¯ n1 G(y) dy = t0 0 G(y) dy
324
5
The Exponential Distribution and the Poisson Process
¯ For example, suppose that G is exponential with mean μ. Then G(y) = e−y/μ , and a simple integration gives that
estimate of N2 (t) =
n1 μ(1 − e−t/μ ) t − μ(1 − e−t/μ )
If we suppose that t = 16 years, μ = 10 years, and n1 = 220 thousand, then the estimate of the number of infected but symptomless individuals at time 16 is
estimate =
2,200(1 − e−1.6 ) = 218.96 16 − 10(1 − e−1.6 )
That is, if we suppose that the foregoing model is approximately correct (and we should be aware that the assumption of a constant infection rate λ that is unchanging over time is almost certainly a weak point of the model), then if the incubation period is exponential with mean 10 years and if the total number of individuals who have exhibited AIDS symptoms during the first 16 years of the epidemic is 220 thousand, then we can expect that approximately 219 thousand individuals are HIV positive though symptomless at time 16.
Proof of Proposition 5.3 Let us compute the joint probability to have P {Ni (t) = ni , i = 1, . . . , k}. To do so note first that in order for there been ni type i events for i = 1, . . . , k there must have been a total of ki=1 ni events. Hence, conditioning on N (t) yields P {N1 (t) = n1 , . . . , Nk (t) = nk }
k ! ! = P N1 (t) = n1 , . . . , Nk (t) = nk ! N (t) = ni
× P N (t) =
k
i=1
ni
i=1
Now consider an arbitrary event that occurred in the interval [0, t]. If it had occurred at time s, then the probability that it would be a type i event would be Pi (s). Hence, since by Theorem 5.2 this event will have occurred at some time uniformly distributed on (0, t), it follows that the probability that this event will be a type i
5.3.
The Poisson Process
325
event is Pi =
1 t
t
Pi (s) ds 0
independently of the other events. Hence,
k ! ! ni P Ni (t) = ni , i = 1, . . . , k ! N (t) =
i=1
will just equal the multinomial probability of ni type i outcomes for i = 1, . . . , k when each of ki=1 ni independent trials results in outcome i with probability Pi , i = 1, . . . , k. That is,
k ! ( ki=1 ni )! n1 ! n P N1 (t) = n1 , . . . , Nk (t) = nk ! N (t) = P · · · Pk k ni = n1 ! · · · nk ! 1 i=1
Consequently, P {N1 (t) = n1 , . . . , Nk (t) = nk } ni ( i ni )! n1 nk −λt (λt) i P1 · · · Pk e = n1 ! · · · nk ! ( i ni )! =
k
e−λtPi (λtPi )ni /ni !
i=1
and the proof is complete
We now present some additional examples of the usefulness of Theorem 5.2.
Example 5.21 Insurance claims are made at times distributed according to a Poisson process with rate λ; the successive claim amounts are independent random variables having distribution G with mean μ, and are independent of the claim arrival times. Let Si and Ci denote, respectively, the time and the amount of the ith claim. The total discounted cost of all claims made up to time t, call it D(t), is defined by D(t) =
N (t) i=1
e−αSi Ci
326
5
The Exponential Distribution and the Poisson Process
where α is the discount rate and N (t) is the number of claims made by time t. To determine the expected value of D(t), we condition on N (t) to obtain E[D(t)] =
∞
E[D(t)|N (t) = n]e−λt
n=0
(λt)n n!
Now, conditional on N (t) = n, the claim arrival times S1 , . . . , Sn are distributed as the ordered values U(1) , . . . , U(n) of n independent uniform (0, t) random variables U1 , . . . , Un . Therefore, , n −αU(i) E[D(t)|N (t) = n] = E Ci e i=1
=
n
E[Ci e−αU(i) ]
i=1
=
n
E[Ci ]E[e−αU(i) ]
i=1
where the final equality used the independence of the claim amounts and their arrival times. Because E[Ci ] = μ, continuing the preceding gives E[D(t)|N (t) = n] = μ
n
E[e−αU(i) ]
i=1
= μE
, n
e−αU(i)
i=1
= μE
, n
e
−αUi
i=1
The last equality follows because U(1) , . . . , U(n) are the values U1 , . . . , Un in increasing order, and so ni=1 e−αU(i) = ni=1 e−αUi . Continuing the string of equalities yields E[D(t)|N (t) = n] = nμE[e−αU ] μ t −αx e dx =n t 0 μ = n (1 − e−αt ) αt
5.3.
The Poisson Process
327
Therefore, E[D(t)|N (t)] = N (t)
μ (1 − e−αt ) αt
Taking expectations yields the result E[D(t)] =
λμ (1 − e−αt ) α
Example 5.22 (An Optimization Example) Suppose that items arrive at a processing plant in accordance with a Poisson process with rate λ. At a fixed time T , all items are dispatched from the system. The problem is to choose an intermediate time, t ∈ (0, T ), at which all items in the system are dispatched, so as to minimize the total expected wait of all items. If we dispatch at time t, 0 < t < T , then the expected total wait of all items will be λt 2 λ(T − t)2 + 2 2 To see why this is true, we reason as follows: The expected number of arrivals in (0, t) is λt, and each arrival is uniformly distributed on (0, t), and hence has expected wait t/2. Thus, the expected total wait of items arriving in (0, t) is λt 2 /2. Similar reasoning holds for arrivals in (t, T ), and the preceding follows. To minimize this quantity, we differentiate with respect to t to obtain . 2 / t d (T − t)2 λ +λ = λt − λ(T − t) dt 2 2 and equating to 0 shows that the dispatch time that minimizes the expected total wait is t = T /2. We end this section with a result, quite similar in spirit to Theorem 5.2, which states that given Sn , the time of the nth event, then the first n − 1 event times are distributed as the ordered values of a set of n − 1 random variables uniformly distributed on (0, Sn ).
Proposition 5.4 Given that Sn = t, the set S1 , . . . , Sn−1 has the distribution of a set of n − 1 independent uniform (0, t) random variables.
Proof We can prove the preceding in the same manner as we did Theorem 5.2, or we can argue more loosely as follows: S1 , . . . , Sn−1 | Sn = t ∼ S1 , . . . , Sn−1 | Sn = t, N (t − ) = n − 1 ∼ S1 , . . . , Sn−1 | N (t − ) = n − 1 where ∼ means “has the same distribution as” and t − is infinitesimally smaller than t. The result now follows from Theorem 5.2.
328
5
The Exponential Distribution and the Poisson Process
5.3.6. Estimating Software Reliability When a new computer software package is developed, a testing procedure is often put into effect to eliminate the faults, or bugs, in the package. One common procedure is to try the package on a set of well-known problems to see if any errors result. This goes on for some fixed time, with all resulting errors being noted. Then the testing stops and the package is carefully checked to determine the specific bugs that were responsible for the observed errors. The package is then altered to remove these bugs. Because we cannot be certain that all the bugs in the package have been eliminated, however, a problem of great importance is the estimation of the error rate of the revised software package. To model the preceding, let us suppose that initially the package contains an unknown number, m, of bugs, which we will refer to as bug 1, bug 2, . . . , bug m. Suppose also that bug i will cause errors to occur in accordance with a Poisson process having an unknown rate λi , i = 1, . . . , m. Then, for instance, the number of errors due to bug i that occurs in any s units of operating time is Poisson distributed with mean λi s. Also suppose that these Poisson processes caused by bugs i, i = 1, . . . , m are independent. In addition, suppose that the package is to be run for t time units with all resulting errors being noted. At the end of this time a careful check of the package is made to determine the specific bugs that caused the errors (that is, a debugging, takes place). These bugs are removed, and the problem is then to determine the error rate for the revised package. If we let 1, if bug i has not caused an error by t ψi (t) = 0, otherwise then the quantity we wish to estimate is (t) =
λi ψi (t)
i
the error rate of the final package. To start, note that E[(t)] =
λi E[ψi (t)]
i
=
λi e−λi t
(5.19)
i
Now each of the bugs that is discovered would have been responsible for a certain number of errors. Let us denote by Mj (t) the number of bugs that were responsible for j errors, j 1. That is, M1 (t) is the number of bugs that caused exactly
5.3.
The Poisson Process
329
one error, M2 (t) is the number that caused two errors, and so on, with j j Mj (t) equaling the total number of errors that resulted. To compute E[M1 (t)], let us define the indicator variables, Ii (t), i 1, by 1, bug i causes exactly 1 error Ii (t) = 0, otherwise Then, M1 (t) =
Ii (t)
i
and so
E[M1 (t)] =
E[Ii (t)] =
i
λi te−λi t
(5.20)
i
Thus, from (5.19) and (5.20) we obtain the intriguing result that . / M1 (t) E (t) − =0 t
(5.21)
Thus suggests the possible use of M1 (t)/t as an estimate of (t). To determine whether or not M1 (t)/t constitutes a “good” estimate of (t) we shall look at how far apart these two quantities tend to be. That is, we will compute , M1 (t) 2 M1 (t) from (5.21) = Var (t) − E (t) − t t 2 1 = Var((t)) − Cov((t), M1 (t)) + 2 Var(M1 (t)) t t Now, Var((t)) =
i
Var(M1 (t)) =
Var(Ii (t)) =
i
Cov((t), M1 (t)) = Cov
i
=
i
λ2i Var(ψi (t)) =
j
λ2i e−λi t (1 − e−λi t ),
i
λi te−λi t (1 − λi te−λi t ),
i
λi ψi (t),
Ij (t)
j
Cov(λi ψi (t), Ij (t))
330
5
The Exponential Distribution and the Poisson Process
=
i
=−
λi Cov(ψi (t), Ii (t))
λi e−λi t λi te−λi t
i
where the last two equalities follow since ψi (t) and Ij (t) are independent when i = j because they refer to different Poisson processes and ψi (t)Ii (t) = 0. Hence we obtain that , - M1 (t) 2 1 −λi t E (t) − λ2i e−λi t + λi e = t t i
i
E[M1 (t) + 2M2 (t)] = t2 where the last equality follows from (5.20) and the identity (which we leave as an exercise) 1 (λi t)2 e−λi t (5.22) E[M2 (t)] = 2 i
Thus, we can estimate the average square of the difference between (t) and M1 (t)/t by the observed value of M1 (t) + 2M2 (t) divided by t 2 .
Example 5.23 Suppose that in 100 units of operating time 20 bugs are discovered of which two resulted in exactly one, and three resulted in exactly two, errors. Then we would estimate that (100) is something akin to the value of a random variable whose mean is equal to 1/50 and whose variance is equal to 8/10,000.
5.4.
Generalizations of the Poisson Process
5.4.1. Nonhomogeneous Poisson Process In this section we consider two generalizations of the Poisson process. The first of these is the nonhomogeneous, also called the nonstationary, Poisson process, which is obtained by allowing the arrival rate at time t to be a function of t.
Definition 5.4 The counting process {N (t), t 0} is said to be a nonhomogeneous Poisson process with intensity function λ(t), t 0, if (i) (ii) (iii) (iv)
N (0) = 0. {N (t), t 0} has independent increments. P {N (t + h) − N (t) 2} = o(h). P {N (t + h) − N (t) = 1} = λ(t)h + o(h).
5.4.
Generalizations of the Poisson Process
331
Time sampling an ordinary Poisson process generates a nonhomogeneous Poisson process. That is, let {N (t), t 0} be a Poisson process with rate λ, and suppose that an event occurring at time t is, independently of what has occurred prior to t, counted with probability p(t). With Nc (t) denoting the number of counted events by time t, the counting process {Nc (t), t 0} is a nonhomogeneous Poisson process with intensity function λ(t) = λp(t). This is verified by noting that {Nc (t), t 0} satisfies the nonhomogeneous Poisson process axioms. 1. Nc (0) = 0. 2. The number of counted events in (s, s + t) depends solely on the number of events of the Poisson process in (s, s + t), which is independent of what has occurred prior to time s. Consequently, the number of counted events in (s, s + t) is independent of the process of counted events prior to s, thus establishing the independent increment property. 3. Let Nc (t, t + h) = Nc (t + h) − Nc (t), with a similar definition of N (t, t + h). P {Nc (t, t + h) 2} P {N (t, t + h) 2} = o(h) 4. To compute P {Nc (t, t + h) = 1}, condition on N (t, t + h). P {Nc (t, t + h) = 1} = P {Nc (t, t + h) = 1|N (t, t + h) = 1}P {N (t, t + h) = 1} + P {Nc (t, t + h) = 1|N (t, t + h) 2}P {N (t, t + h) 2} = P {Nc (t, t + h) = 1|N (t, t + h) = 1}λh + o(h) = p(t)λh + o(h) Not only does time sampling a Poisson process result in a nonhomogeneous Poisson process, but it also works the other way: every nonhomogeneous Poisson process with a bounded intensity function can be thought of as being a time sampling of a Poisson process. To show this, we start by showing that the superposition of two independent nonhomogeneous Poisson processes remains a nonhomogeneous Poisson process.
Proposition 5.4 Let {N (t), t 0}, and {M(t), t 0}, be independent nonhomogeneous Poisson processes, with respective intensity functions λ(t) and μ(t), and let N ∗ (t) = N (t) + M(t). Then, the following are true. (a) {N ∗ (t), t 0} is a nonhomogeneous Poisson process with intensity function λ(t) + μ(t). (b) Given that an event of the {N ∗ (t)} process occurs at time t then, independent of what occurred prior to t, the event at t was from the {N (t)} process with λ(t) . probability λ(t)+μ(t)
332
5
The Exponential Distribution and the Poisson Process
Proof To verify that {N ∗ (t), t 0}, is a nonhomogeneous Poisson process with
intensity function λ(t) + μ(t), we will argue that it satisfies the nonhomogeneous Poisson process axioms. 1. N ∗ (0) = N (0) + M(0) = 0. 2. To verify independent increments, let I1 , . . . , In be nonoverlapping intervals. Let N (I ) and M(I ) denote, respectively, the number of events from the {N (t)} process and from the {M(t)} process that are in the interval I . Because each counting process has independent increments, and the two processes are independent of each other, it follows that N (I1 ), . . . , N (In ), M(I1 ), . . . , M(In ) are all independent, and thus so are N (I1 ) + M(I1 ), . . . , N (In ) + M(In ), which shows that {N ∗ (t), t 0} also possesses independent increments. 3. In order for there to be exactly one event of the N ∗ process between t and t + h, either there must be one event of the N process and 0 events of the M process or the reverse. The first of these mutually exclusive possibilities occurs with probability P {N (t, t + h) = 1, M(t, t + h) = 0} = P {N (t, t + h) = 1} P {M(t, t + h) = 0} = (λ(t)h + o(h)) (1 − μ(t)h + o(h)) = λ(t)h + o(h) Similarly, the second possibility occurs with probability P {N (t, t + h) = 0, M(t, t + h) = 1} = μ(t)h + o(h) yielding that P {N ∗ (t + h) − N ∗ (t) = 1} = (λ(t) + μ(t))h + o(h) 4. In order for there to be at least two events of the N ∗ process between t and t + h, one of the following three possibilities must occur: there is at least two events of the N process between t and t + h; there is at least two events of the M process between t and t + h; or both processes have exactly one event between t and t + h. Each of the first two of these possibilities occurs with probability o(h), while the third occurs with probability (λ(t)h + o(h))(μ(t)h + o(h)) = o(h). Thus, P {N ∗ (t + h) − N ∗ (t) 2} o(h) Thus, part (a) is proven. To prove (b), note first that it follows from independent increments that which process caused the event at time t is independent of what occurred prior to t. To
5.4.
Generalizations of the Poisson Process
333
find the conditional probability that the event at time t is from the N process, we use that P {N(t, t + h) = 1|N ∗ (t, t + h) = 1} = = =
P {N (t, t + h) = 1, M(t, t + h) = 0} P {N ∗ (t, t + h) = 1} λ(t)h + o(h) (λ(t) + μ(t))h + o(h) λ(t) +
o(h) h
λ(t) + μ(t) +
Letting h → 0 in the preceding proves (b).
o(h) h
Now, suppose that {N (t), t 0} is a nonhomogeneous Poisson process with a bounded intensity function λ(t), and suppose that λ is such that λ(t) λ, for all t 0. Letting {M(t), t 0} be a nonhomogeneous Poisson process with intensity function μ(t) = λ − λ(t), t 0, that is independent of {N (t), t 0}, it follows from Proposition 5.4 that {N (t), t 0} can be regarded as being the process of time sampled events of the Poisson process {N (t) + M(t), t 0}, where an event of the Poisson process that occurs at time t is counted with probability p(t) = λ(t)/λ. With this interpretation of a nonhomogeneous Poisson process as being a timesampled Poisson process, the number of events of the nonhomogeneous Poisson process by time t, namely, N (t), is equal to the number of counted events of the Poisson process by time t. Consequently, from Proposition 5.3 it follows that N (t) is a Poisson random variable with mean E[N (t)] = λ 0
t
λ(y) dy = λ
t
λ(y) dy 0
Moreover, by regarding the nonhomogeneous Poisson process as starting at time s, the preceding yields that N (t + s) − N (t), the in its first t t number of events s+t time units, is a Poisson random variable with mean 0 λ(s + y) dy = s λ(y) dy. The function m(t) defined by m(t) =
t
λ(y) dy 0
is called the mean value function of the nonhomogeneous Poisson process.
Remark That N (s + t) − N (s) has a Poisson distribution with mean s+t s
λ(y) dy is a consequence of the Poisson limit of the sum of independent
334
5
The Exponential Distribution and the Poisson Process
Bernoulli random variables (see Example 2.47). To see why, subdivide the interval [s, s + t] into n subintervals of length nt , where subinterval i goes from s + (i − 1) nt to s + i nt , i = 1, . . . , n. Let Ni = N (s + i nt ) − N (s + (i − 1) nt ) be the number of events that occur in subinterval i, and note that n
P { 2 events in some subinterval} = P {Ni 2} i=1
n
P {Ni 2}
i=1
= no(t/n)
by Axiom (iii)
Because lim no(t/n) = lim t
n→∞
n→∞
o(t/n) =0 t/n
it follows that, as n increases to ∞, the probability of having two or more events in any of the n subintervals goes to 0. Consequently, with a probability going to 1, N(t) will equal the number of subintervals in which an event occurs. Because the probability of an event in subinterval i is λ(s + i nt ) nt + o( nt ), it follows, because the number of events in different subintervals are independent, that when n is large the number of subintervals that contain an event is approximately a Poisson random variable with mean n t t + no(t/n) λ s+i n n i=1
But, s+t n t t λ s +i λ(y) dy lim + no(t/n) = n→∞ n n s i=1
and the result follows.
The importance of the nonhomogeneous Poisson process resides in the fact that we no longer require the condition of stationary increments. Thus we now allow for the possibility that events may be more likely to occur at certain times than during other times.
Example 5.24 Siegbert runs a hot dog stand that opens at 8 A . M . From 8 until 11 A . M . customers seem to arrive, on the average, at a steadily increasing rate that starts with an initial rate of 5 customers per hour at 8 A . M . and reaches a maximum of 20 customers per hour at 11 A . M . From 11 A . M . until 1 P. M . the
5.4.
Generalizations of the Poisson Process
335
(average) rate seems to remain constant at 20 customers per hour. However, the (average) arrival rate then drops steadily from 1 P. M . until closing time at 5 P. M . at which time it has the value of 12 customers per hour. If we assume that the numbers of customers arriving at Siegbert’s stand during disjoint time periods are independent, then what is a good probability model for the preceding? What is the probability that no customers arrive between 8:30 A . M . and 9:30 A . M . on Monday morning? What is the expected number of arrivals in this period? Solution: A good model for the preceding would be to assume that arrivals constitute a nonhomogeneous Poisson process with intensity function λ(t) given by ⎧ 0t 3 ⎨5 + 5t, 3t 5 λ(t) = 20, ⎩ 20 − 2(t − 5), 5t 9 and λ(t) = λ(t − 9)
for t > 9
Note that N (t) represents the number of arrivals during the first t hours that the store is open. That is, we do not count the hours between 5 P. M . and 8 A . M . If for some reasons we wanted N (t) to represent the number of arrivals during the first t hours regardless of whether the store was open or not, then, assuming that the process begins at midnight we would let ⎧ 0, 0t 8 ⎪ ⎪ ⎪ ⎪ 8 t 11 ⎨5 + 5(t − 8), 11 t 13 λ(t) = 20, ⎪ ⎪ 20 − 2(t − 13), 13 t 17 ⎪ ⎪ ⎩ 0, 17 < t 24 and λ(t) = λ(t − 24)
for t > 24
As the number of arrivals between 8:30 A . M . and 9:30 A . M . will be Poisson 17 with mean m( 32 ) − m( 12 ) in the first representation [and m( 19 2 ) − m( 2 ) in the second representation], we have that the probability that this number is zero is 3/2 exp − (5 + 5t) dt = e−10 1/2
and the mean number of arrivals is 3/2 (5 + 5t) dt = 10 1/2
336
5
The Exponential Distribution and the Poisson Process
Suppose that events occur according to a Poisson process with rate λ, and suppose that, independent of what has previously occurred, an event at time s is a type 1 event with probability P1 (s) or a type 2 event with probability P2 (s) = 1 − P1 (s). If Ni (t), t 0, denotes the number of type i events by time t, then it easily follows from Definition 5.4 that {N1 (t), t 0} and {N2 (t), t 0} are independent nonhomogeneous Poisson processes with respective intensity functions λi (t) = λPi (t), i = 1, 2. (The proof mimics that of Proposition 5.2.) This result gives us another way of understanding (or of proving) the time sampling Poisson process result of Proposition 5.3 which states that N1 (t) and N2 (t) are indet pendent Poisson random variables with means E[Ni (t)] = λ 0 Pi (s) ds, i = 1, 2.
Example 5.25 (The Output Process of an Infinite Server Poisson Queue) It turns out that the output process of the M/G/∞ queue—that is, of the infinite server queue having Poisson arrivals and general service distribution G—is a nonhomogeneous Poisson process having intensity function λ(t) = λG(t). To verify this claim, let us first argue that the departure process has independent increments. Towards this end, consider nonoverlapping intervals O1 , . . . , Ok ; now say that an arrival is type i, i = 1, . . . , k, if that arrival departs in the interval Oi . By Proposition 5.3, it follows that the numbers of departures in these intervals are independent, thus establishing independent increments. Now, suppose that an arrival is “counted” if that arrival departs between t and t + h. Because an arrival at time s, s < t + h, will be counted with probability G(t − s + h) − G(t − s), it follows from Proposition 5.3 that the number of departures in (t, t + h) is a Poisson random variable with mean
t+h
λ
t+h
[G(t − s + h) − G(t − s)] ds = λ
0
[G (t − s + h)h + o(h)] ds
0
= λh
t+h
G (y) dy + o(h)
0
= λG(t)h + o(h) Therefore, P {1 departure in (t, t + h)} = λG(t)h e−λG(t)h + o(h) = λG(t)h + o(h) and P { 2 departures in (t, t + h)} = o(h) which completes the verification.
5.4.
337
Generalizations of the Poisson Process
If we let Sn denote the time of the nth event of the nonhomogeneous Poisson process, then we can obtain its density as follows: P {t < Sn < t + h} = P {N (t) = n − 1, one event in (t, t + h)} + o(h) = P {N (t) = n − 1}P {one event in (t, t + h)} + o(h) = e−m(t)
[m(t)]n−1 [λ(t)h + o(h)] + o(h) (n − 1)!
= λ(t)e−m(t)
[m(t)]n−1 h + o(h) (n − 1)!
which implies that fSn (t) = λ(t)e−m(t)
[m(t)]n−1 (n − 1)!
where m(t) =
t
λ(s) ds 0
5.4.2. Compound Poisson Process A stochastic process {X(t), t 0} is said to be a compound Poisson process if it can be represented as X(t) =
N (t)
Yi ,
t 0
(5.23)
i=1
where {N(t), t 0} is a Poisson process, and {Yi , i 1} is a family of independent and identically distributed random variables that is also independent of {N(t), t 0}. As noted in Chapter 3, the random variable X(t) is said to be a compound Poisson random variable. Examples of Compound Poisson Processes (i) If Yi ≡ 1, then X(t) = N (t), and so we have the usual Poisson process. (ii) Suppose that buses arrive at a sporting event in accordance with a Poisson process, and suppose that the numbers of fans in each bus are assumed to be independent and identically distributed. Then {X(t), t 0} is a compound Poisson process where X(t) denotes the number of fans who have arrived by t. In Equation (5.23) Yi represents the number of fans in the ith bus.
338
5
The Exponential Distribution and the Poisson Process
(iii) Suppose customers leave a supermarket in accordance with a Poisson process. If Yi , the amount spent by the ith customer, i = 1, 2, . . . , are independent and identically distributed, then {X(t), t 0} is a compound Poisson process when X(t) denotes the total amount of money spent by time t. Because X(t) is a compound Poisson random variable with Poisson parameter λt, we have from Examples 3.10 and 3.17 that E[X(t)] = λtE[Y1 ]
(5.24)
" # Var(X(t)) = λtE Y12
(5.25)
and
Example 5.26 Suppose that families migrate to an area at a Poisson rate
λ = 2 per week. If the number of people in each family is independent and takes on the values 1, 2, 3, 4 with respective probabilities 16 , 13 , 13 , 16 , then what is the expected value and variance of the number of individuals migrating to this area during a fixed five-week period? Solution: that
Letting Yi denote the number of people in the ith family, we have E[Yi ] = 1 · 16 + 2 · 13 + 3 · 13 + 4 · 16 = 52 , " # E Yi2 = 12 · 16 + 22 · 13 + 32 · 13 + 42 · 16 =
43 6
Hence, letting X(5) denote the number of immigrants during a five-week period, we obtain from Equations (5.24) and (5.25) that E[X(5)] = 2 · 5 ·
5 2
= 25
and Var[X(5)] = 2 · 5 ·
43 6
=
215 3
Example 5.27 (Busy Periods in Single-Server Poisson Arrival Queues) Consider a single-server service station in which customers arrive according to a Poisson process having rate λ. An arriving customer is immediately served if the server is free; if not, the customer waits in line (that is, he or she joins the queue). The successive service times are independent with a common distribution. Such a system will alternate between idle periods when there are no customers in the system, so the server is idle, and busy periods when there are customers in the system, so the server is busy. A busy period will begin when an arrival finds the system empty, and because of the memoryless property of the Poisson arrivals
5.4.
Generalizations of the Poisson Process
339
it follows that the distribution of the length of a busy period will be the same for each such period. Let B denote the length of a busy period. We will compute its mean and variance. To begin, let S denote the service time of the first customer in the busy period and let N(S) denote the number of arrivals during that time. Now, if N (S) = 0 then the busy period will end when the initial customer completes his service, and so B will equal S in this case. Now, suppose that one customer arrives during the service time of the initial customer. Then, at time S there will be a single customer in the system who is just about to enter service. Because the arrival stream from time S on will still be a Poisson process with rate λ, it thus follows that the additional time from S until the system becomes empty will have the same distribution as a busy period. That is, if N (S) = 1 then B = S + B1 where B1 is independent of S and has the same distribution as B. Now, consider the general case where N (S) = n, so there will be n customers waiting when the server finishes his initial service. To determine the distribution of remaining time in the busy period note that the order in which customers are served will not affect the remaining time. Hence, let us suppose that the n arrivals, call them C1 , . . . , Cn , during the initial service period are served as follows. Customer C1 is served first, but C2 is not served until the only customers in the system are C2 , . . . , Cn . For instance, any customers arriving during C1 ’s service time will be served before C2 . Similarly, C3 is not served until the system is free of all customers but C3 , . . . , Cn , and so on. A little thought reveals that the times between the beginnings of service of customers Ci and Ci+1 , i = 1, . . . , n − 1, and the time from the beginning of service of Cn until there are no customers in the system, are independent random variables, each distributed as a busy period. It follows from the preceding that if we let B1 ,B2 , . . . be a sequence of independent random variables, each distributed as a busy period, then we can express B as B =S+
N (S)
Bi
i=1
Hence, ⎡
⎤
E[B|S] = S + E ⎣
Bi |S ⎦
N (S) i=1
340
5
The Exponential Distribution and the Poisson Process
and
⎞
⎛
N (S)
Var(B|S) = Var ⎝
Bi |S ⎠
i=1
N (S) However, given S, i=1 Bi is a compound Poisson random variable, and thus from Equations (5.24) and (5.25) we obtain E[B|S] = S + λSE[B] = (1 + λE[B])S Var(B|S) = λSE[B 2 ] Hence, E[B] = E[E[B|S]] = (1 + λE[B])E[S] implying, provided that λE[S] < 1, that E[B] =
E[S] 1 − λE[S]
Also, by the conditional variance formula Var(B) = Var(E[B|S]) + E[Var(B|S)] = (1 + λE[B])2 Var(S) + λE[S]E[B 2 ] = (1 + λE[B])2 Var(S) + λE[S](Var(B) + E 2 [B]) yielding Var(B) =
Var(S)(1 + λE[B])2 + λE[S]E 2 [B] 1 − λE[S]
Using E[B] = E[S]/(1 − λE[S]), we obtain Var(B) =
Var(S) + λE 3 [S] (1 − λE[S])3
There is a very nice representation of the compound Poisson process when the set of possible values of the Yi is finite or countably infinite. So let us suppose that there are numbers αj , j 1, such that P {Y1 = αj } = pj , pj = 1 j
Now, a compound Poisson process arises when events occur according to a Poisson process and each event results in a random amount Y being added to
5.4.
Generalizations of the Poisson Process
341
the cumulative sum. Let us say that the event is a type j event whenever it results in adding the amount αj , j 1. That is, the ith event of the Poisson process is a type j event if Yi = αj . If we let Nj (t) denote the number of type j events by time t, then it follows from Proposition 5.2 that the random variables Nj (t), j 1, are independent Poisson random variables with respective means E[Nj (t)] = λpj t Since, for each j , the amount αj is added to the cumulative sum a total of Nj (t) times by time t, it follows that the cumulative sum at time t can be expressed as X(t) = αj Nj (t) (5.26) j
As a check of Equation (5.26), let us use it to compute the mean and variance of X(t). This yields / . αj Nj (t) E[X(t)] = E =
j
αj E[Nj (t)]
j
=
αj λpj t
j
= λt E[Y1 ] Also, / . αj Nj (t) Var[X(t)] = Var =
j
αj2 Var[Nj (t)]
by the independence of the Nj (t), j 1
j
=
αj2 λpj t
j
= λtE[Y12 ] where the next to last equality follows since the variance of the Poisson random variable Nj (t) is equal to its mean. Thus, we see that the representation (5.26) results in the same expressions for the mean and variance of X(t) as were previously derived.
342
5
The Exponential Distribution and the Poisson Process
One of the uses of the representation (5.26) is that it enables us to conclude that as t grows large, the distribution of X(t) converges to the normal distribution. To see why, note first that it follows by the central limit theorem that the distribution of a Poisson random variable converges to a normal distribution as its mean increases. (Why is this?) Therefore, each of the random variables Nj (t) converges to a normal random variable as t increases. Because they are independent, and because the sum of independent normal random variables is also normal, it follows that X(t) also approaches a normal distribution as t increases.
Example 5.28 In Example 5.24, find the approximate probability that at least 240 people migrate to the area within the next 50 weeks. Solution:
Since λ = 2, E[Yi ] = 5/2, E[Yi2 ] = 43/6, we see that E[X(50)] = 250,
Var[X(50)] = 4300/6
Now, the desired probability is P {X(50) 240} = P {X(50) 239.5} X(50) − 250 239.5 − 250 =P √ √ 4300/6 4300/6 = 1 − φ(−0.3922) = φ(0.3922) = 0.6525 where Table 2.3 was used to determine φ(0.3922), the probability that a standard normal is less than 0.3922. Another useful result is that if {X(t), t 0} and {Y (t), t 0} are independent compound Poisson processes with respective Poisson parameters and distributions λ1 , F1 and λ2 , F2 , then {X(t) + Y (t), t 0} is also a compound Poisson process. This is true because in this combined process events will occur according to a Poisson process with rate λ1 + λ2 , and each event independently will be from the first compound Poisson process with probability λ1 /(λ1 + λ2 ). Consequently, the combined process will be a compound Poisson process with Poisson parameter λ1 + λ2 , and with distribution function F given by F (x) =
λ2 λ1 F1 (x) + F2 (x) λ1 + λ2 λ1 + λ2
5.4.
Generalizations of the Poisson Process
343
5.4.3. Conditional or Mixed Poisson Processes Let {N(t), t 0} be a counting process whose probabilities are defined as follows. There is a positive random variable L such that, conditional on L = λ, the counting process is a Poisson process with rate λ. Such a counting process is called a conditional or a mixed Poisson process. Suppose that L is continuous with density function g. Because ∞ P {N(t + s) − N (s) = n} = P {N (t + s) − N (s) = n | L = λ}g(λ) dλ 0
=
∞
(λt)n g(λ) dλ n!
e−λt
0
(5.27)
we see that a conditional Poisson process has stationary increments. However, because knowing how many events occur in an interval gives information about the possible value of L, which affects the distribution of the number of events in any other interval, it follows that a conditional Poisson process does not generally have independent increments. Consequently, a conditional Poisson process is not generally a Poisson process.
Example 5.29 If g is the gamma density with parameters m and θ , g(λ) = θ e−θλ then
∞
(θ λ)m−1 , (m − 1)!
λ>0
(λt)n −θλ (θ λ)m−1 θe dλ n! (m − 1)! 0 ∞ t nθ m = e−(t+θ)λ λn+m−1 dλ n!(m − 1)! 0
P {N (t) = n} =
Multiplying and dividing by P {N(t) = n} =
e−λt
(n+m−1)! (t+θ)n+m
gives
t n θ m (n + m − 1)! n!(m − 1)!(t + θ )n+m
0
∞
(t + θ )e−(t+θ)λ
((t + θ )λ)n+m−1 dλ (n + m − 1)!
Because (t + θ )e−(t+θ)λ ((t + θ )λ)n+m−1 /(n + m − 1)! is the density function of a gamma (n + m, t + θ ) random variable, its integral is 1, giving the result m n θ n+m−1 t P {N (t) = n} = n t +θ t +θ Therefore, the number of events in an interval of length t has the same distribution of the number of failures that occur before a total of m successes are amassed, θ when each trial is a success with probability t+θ .
344
5
The Exponential Distribution and the Poisson Process
To compute the mean and variance of N (t), condition on L. Because, conditional on L, N (t) is Poisson with mean Lt, we obtain E[N (t)|L] = Lt Var(N (t)|L) = Lt where the final equality used that the variance of a Poisson random variable is equal to its mean. Consequently, the conditional variance formula yields Var(N (t)) = E[Lt] + Var(Lt) = tE[L] + t 2 Var(L) We can compute the conditional distribution function of L, given that N (t) = n, as follows. P {L x, N (t) = n} P {N (t) = n} ∞ P {L x, N (t) = n|L = λ}g(λ) dλ = 0 P {N (t) = n} x P {N (t) = n|L = λ}g(λ) dλ = 0 P {N (t) = n} x −λt e (λt)n g(λ) dλ = 0∞ −λt (λt)n g(λ) dλ 0 e
P {L x|N (t) = n} =
where the final equality used Equation (5.27). In other words, the conditional density function of L given that N (t) = n is fL|N (t) (λ | n) = ∞ 0
e−λt λn g(λ) , e−λt λn g(λ) dλ
λ0
(5.28)
Example 5.30 An insurance company feels that each of its policyholders has a rating value and that a policyholder having rating value λ will make claims at times distributed according to a Poisson process with rate λ, when time is measured in years. The firm also believes that rating values vary from policyholder to policyholder, with the probability distribution of the value of a new policyholder being uniformly distributed over (0, 1). Given that a policyholder has made n claims in his or her first t years, what is the conditional distribution of the time until the policyholder’s next claim?
5.4.
345
Generalizations of the Poisson Process
Solution: If T is the time until the next claim, then we want to compute P {T > x | N (t) = n}. Conditioning on the policyholder’s rating value gives, upon using Equation (5.28),
∞
P {T > x | N (t) = n} =
P {T > x | L = λ, N (t) = n}fL|N (t) (λ | n) dλ
0
1 0
=
e−λx e−λt λn dλ 1 −λt λn dλ 0 e
There is a nice formula for the probability that more than n events occur in an interval of length t. In deriving it we will use the identity ∞
(λt)j = j!
e−λt
j =n+1
t
λe−λx
0
(λx)n dx n!
(5.29)
which follows by noting that it equates the probability that the number of events by time t of a Poisson process with rate λ is greater than n with the probability that the time of the (n + 1)st event of this process (which has a gamma (n + 1, λ) distribution) is less than t. Interchanging λ and t in Equation (5.29) yields the equivalent identity ∞
e−λt
j =n+1
(λt)j = j!
λ
te−tx
0
(tx)n dx n!
(5.30)
Using Equation (5.27) we now have P {N(t) > n} =
∞
∞
e−λt
(λt)j g(λ) dλ j!
e−λt
(λt)j g(λ) dλ j!
j =n+1 0
=
∞
0
= 0
=
j =n+1
∞ λ 0
0
te−tx
∞ ∞
0
=
∞
(tx)n dxg(λ) dλ n!
g(λ) dλte−tx
x ∞
n
(tx)n dx n!
−tx (tx) ¯ dx G(x)te n!
(by interchanging)
(using (5.30)) (by interchanging)
346
5
The Exponential Distribution and the Poisson Process
Exercises 1. The time T required to repair a machine is an exponentially distributed random variable with mean 12 (hours). (a) What is the probability that a repair time exceeds 12 hour? (b) What is the probability that a repair takes at least 12 12 hours given that its duration exceeds 12 hours? 2. Suppose that you arrive at a single-teller bank to find five other customers in the bank, one being served and the other four waiting in line. You join the end of the line. If the service times are all exponential with rate μ, what is the expected amount of time you will spend in the bank? 3. Let X be an exponential random variable. Without any computations, tell which one of the following is correct. Explain your answer. (a) E[X 2 |X > 1] = E[(X + 1)2 ]; (b) E[X 2 |X > 1] = E[X 2 ] + 1; (c) E[X 2 |X > 1] = (1 + E[X])2 . 4. Consider a post office with two clerks. Three people, A, B, and C, enter simultaneously. A and B go directly to the clerks, and C waits until either A or B leaves before he begins service. What is the probability that A is still in the post office after the other two have left when (a) the service time for each clerk is exactly (nonrandom) ten minutes? (b) the service times are i with probability 13 , i = 1, 2, 3? (c) the service times are exponential with mean 1/μ? 5. The lifetime of a radio is exponentially distributed with a mean of ten years. If Jones buys a ten-year-old radio, what is the probability that it will be working after an additional ten years? 6.
In Example 5.3 if server i serves at an exponential rate λi , i = 1, 2, show that P {Smith is not last} =
λ1 λ1 + λ2
2
λ2 + λ1 + λ2
2
*7. If X1 and X2 are independent nonnegative continuous random variables, show that P {X1 < X2 | min(X1 , X2 ) = t} =
r1 (t) r1 (t) + r2 (t)
where ri (t) is the failure rate function of Xi . 8. Let Xi , i = 1, . . . , n be independent continuous random variables, with Xi having failure rate function ri (t). Let T be independent of this sequence, and
Exercises
suppose that is given by
n
i=1 P {T
347
= i} = 1. Show that the failure rate function r(t) of XT
r(t) =
n
ri (t)P {T = i|X > t}
i=1
9. Machine 1 is currently working. Machine 2 will be put in use at a time t from now. If the lifetime of machine i is exponential with rate λi , i = 1, 2, what is the probability that machine 1 is the first machine to fail? *10. Let X and Y be independent exponential random variables with respective rates λ and μ. Let M = min(X, Y ). Find (a) E[MX|M = X], (b) E[MX|M = Y ], (c) Cov(X, M). 11. Let X, Y1 , . . . , Yn be independent exponential random variables; X having rate λ, and Yi having rate μ. Let Aj be the event that the j th smallest of these n + 1 random variables is one of the Yi . Find p = P {X > maxi Yi }, by using the identity p = P (A1 · · · An ) = P (A1 )P (A2 |A1 ) · · · P (An |A1 · · · An−1 ) Verify your answer when n = 2 by conditioning on X to obtain p. 12. If Xi , i = 1, 2, 3, are independent exponential random variables with rates λi , i = 1, 2, 3, find (a) P {X1 < X2 < X3 }, (b) P {X1 < X2 | max(X1 , X2 , X3 ) = X3 }, (c) E[max Xi |X1 < X2 < X3 ], (d) E[max Xi ]. 13. Find, in Example 5.8, the expected time until the nth person on line leaves the line (either by entering service or departing without service). 14. Let X be an exponential random variable with rate λ. (a) Use the definition of conditional expectation to determine E[X|X < c]. (b) Now determine E[X|X < c] by using the following identity: E[X] = E[X|X < c]P {X < c} + E[X|X > c]P {X > c} 15. One hundred items are simultaneously put on a life test. Suppose the lifetimes of the individual items are independent exponential random variables with mean 200 hours. The test will end when there have been a total of 5 failures. If T is the time at which the test ends, find E[T ] and Var(T ). 16. Suppose in Example 5.3 that the time it takes server i to serve customers is exponentially distributed with mean 1/λi , i = 1, 2. What is the expected time until all three customers have left the post office?
348
5
The Exponential Distribution and the Poisson Process
17. A set of n cities is to be connected via communication links. The cost to construct a link between cities i and j is Cij , i = j . Enough links should be constructed so that for each pair of cities there is a path of links that connects them. As a result, only n − 1 links need be constructed. A minimal cost algorithm for solving this problem (known as the minimal spanning tree problem) first constructs the cheapest of all the n2 links. Then, at each additional stage it chooses the cheapest link that connects a city without any links to one with links. That is, if the first link is between cities 1 and 2, then the second link will either be between 1 and one of the links 3, . . . , n or between 2 and one of the links 3, . . . , n. Suppose that all of the n2 costs Cij are independent exponential random variables with mean 1. Find the expected cost of the preceding algorithm if (a) n = 3, (b) n = 4. *18. Let X1 and X2 be independent exponential random variables, each having rate μ. Let X(1) = minimum(X1 , X2 )
and X(2) = maximum(X1 , X2 )
Find (a) E[X(1) ], (b) Var[X(1) ], (c) E[X(2) ], (d) Var[X(2) ]. 19. Repeat Exercise 18, but this time suppose that the Xi are independent exponentials with respective rates μi , i = 1, 2. 20. Consider a two-server system in which a customer is served first by server 1, then by server 2, and then departs. The service times at server i are exponential random variables with rates μi , i = 1, 2. When you arrive, you find server 1 free and two customers at server 2—customer A in service and customer B waiting in line. (a) Find PA , the probability that A is still in service when you move over to server 2. (b) Find PB , the probability that B is still in the system when you move over to server 2. (c) Find E[T ], where T is the time that you spend in the system. Hint: Write T = S1 + S2 + W A + W B where Si is your service time at server i, WA is the amount of time you wait in queue while A is being served, and WB is the amount of time you wait in queue while B is being served.
Exercises
349
21. In a certain system, a customer must first be served by server 1 and then by server 2. The service times at server i are exponential with rate μi , i = 1, 2. An arrival finding server 1 busy waits in line for that server. Upon completion of service at server 1, a customer either enters service with server 2 if that server is free or else remains with server 1 (blocking any other customer from entering service) until server 2 is free. Customers depart the system after being served by server 2. Suppose that when you arrive there is one customer in the system and that customer is being served by server 1. What is the expected total time you spend in the system? 22. Suppose in Exercise 21 you arrive to find two others in the system, one being served by server 1 and one by server 2. What is the expected time you spend in the system? Recall that if server 1 finishes before server 2, then server 1’s customer will remain with him (thus blocking your entrance) until server 2 becomes free. *23. A flashlight needs two batteries to be operational. Consider such a flashlight along with a set of n functional batteries—battery 1, battery 2, . . . , battery n. Initially, battery 1 and 2 are installed. Whenever a battery fails, it is immediately replaced by the lowest numbered functional battery that has not yet been put in use. Suppose that the lifetimes of the different batteries are independent exponential random variables each having rate μ. At a random time, call it T , a battery will fail and our stockpile will be empty. At that moment exactly one of the batteries— which we call battery X—will not yet have failed. (a) (b) (c) (d) (e)
What is P {X = n}? What is P {X = 1}? What is P {X = i}? Find E[T ]. What is the distribution of T ?
24. There are 2 servers available to process n jobs. Initially, each server begins work on a job. Whenever a server completes work on a job, that job leaves the system and the server begins processing a new job (provided there are still jobs waiting to be processed). Let T denote the time until all jobs have been processed. If the time that it takes server i to process a job is exponentially distributed with rate μi , i = 1, 2, find E[T ] and Var(T ). 25. Customers can be served by any of three servers, where the service times of server i are exponentially distributed with rate μi , i = 1, 2, 3. Whenever a server becomes free, the customer who has been waiting the longest begins service with that server. (a) If you arrive to find all three servers busy and no one waiting, find the expected time until you depart the system. (b) If you arrive to find all three servers busy and one person waiting, find the expected time until you depart the system.
350
5
The Exponential Distribution and the Poisson Process
26. Each entering customer must be served first by server 1, then by server 2, and finally by server 3. The amount of time it takes to be served by server i is an exponential random variable with rate μi , i = 1, 2, 3. Suppose you enter the system when it contains a single customer who is being served by server 3. (a) Find the probability that server 3 will still be busy when you move over to server 2. (b) Find the probability that server 3 will still be busy when you move over to server 3. (c) Find the expected amount of time that you spend in the system. (Whenever you encounter a busy server, you must wait for the service in progress to end before you can enter service.) (d) Suppose that you enter the system when it contains a single customer who is being served by server 2. Find the expected amount of time that you spend in the system. 27. Show, in Example 5.7, that the distributions of the total cost are the same for the two algorithms. 28. Consider n components with independent lifetimes which are such that component i functions for an exponential time with rate λi . Suppose that all components are initially in use and remain so until they fail. (a) Find the probability that component 1 is the second component to fail. (b) Find the expected time of the second failure. Hint: Do not make use of part (a). 29. Let X and Y be independent exponential random variables with respective rates λ and μ, where λ > μ. Let c > 0. (a) Show that the conditional density function of X, given that X + Y = c, is fX|X+Y (x|c) =
(λ − μ)e−(λ−μ)x , 1 − e−(λ−μ)c
0<x Y , the random variables min(X, Y ) and X − Y are independent. (b) Use part (a) to conclude that for any positive constant c E[min(X, Y )|X > Y + c] = E[min(X, Y )|X > Y ] = E[min(X, Y )] =
1 λ+μ
(c) Give a verbal explanation of why min(X, Y ) and X − Y are (unconditionally) independent. 34. Two individuals, A and B, both require kidney transplants. If she does not receive a new kidney, then A will die after an exponential time with rate μA , and B after an exponential time with rate μB . New kidneys arrive in accordance with a Poisson process having rate λ. It has been decided that the first kidney will go to A (or to B if B is alive and A is not at that time) and the next one to B (if still living). (a) What is the probability that A obtains a new kidney? (b) What is the probability that B obtains a new kidney? 35. Show that Definition 5.1 of a Poisson process implies Definition 5.3. 36. Let S(t) denote the price of a security at time t. A popular model for the process {S(t), t 0} supposes that the price remains unchanged until a “shock” occurs, at which time the price is multiplied by a random factor. If we let N (t) denote the number of shocks by time t, and let Xi denote the ith multiplicative factor, then this model supposes that S(t) = S(0)
N (t)
Xi
i=1
1N (t) where i=1 Xi is equal to 1 when N (t) = 0. Suppose that the Xi are independent exponential random variables with rate μ; that {N (t), t 0} is a Poisson process with rate λ; that {N (t), t 0} is independent of the Xi ; and that S(0) = s. (a) Find E[S(t)]. (b) Find E[S 2 (t)].
352
5
The Exponential Distribution and the Poisson Process
37. Cars cross a certain point in the highway in accordance with a Poisson process with rate λ = 3 per minute. If Reb blindly runs across the highway, then what is the probability that she will be uninjured if the amount of time that it takes her to cross the road is s seconds? (Assume that if she is on the highway when a car passes by, then she will be injured.) Do it for s = 2, 5, 10, 20. 38. Let {Mi (t), t 0}, i = 1, 2 be independent Poisson processes with respective rates λi , i = 1, 2, and set N1 (t) = M1 (t) + M2 (t),
N2 (t) = M2 (t) + M3 (t)
The stochastic process {(N1 (t), N2 (t)), t 0} is called a bivariate Poisson process. (a) Find P {N1 (t) = n, N2 (t) = m}. (b) Find Cov N1 (t), N2 (t) . 39. A certain scientific theory supposes that mistakes in cell division occur according to a Poisson process with rate 2.5 per year, and that an individual dies when 196 such mistakes have occurred. Assuming this theory, find (a) the mean lifetime of an individual, (b) the variance of the lifetime of an individual. Also approximate (c) the probability that an individual dies before age 67.2, (d) the probability that an individual reaches age 90, (e) the probability that an individual reaches age 100. *40. Show that if {Ni (t), t 0} are independent Poisson processes with rate λi , i = 1, 2, then {N (t), t 0} is a Poisson process with rate λ1 + λ2 where N(t) = N1 (t) + N2 (t). 41. In Exercise 40 what is the probability that the first event of the combined process is from the N1 process? 42. Let {N (t), t 0} be a Poisson process with rate λ. Let Sn denote the time of the nth event. Find (a) E[S4 ], (b) E[S4 |N (1) = 2], (c) E[N (4) − N (2)|N (1) = 3]. 43. Customers arrive at a two-server service station according to a Poisson process with rate λ. Whenever a new customer arrives, any customer that is in the system immediately departs. A new arrival enters service first with server 1 and then with server 2. If the service times at the servers are independent exponentials with respective rates μ1 and μ2 , what proportion of entering customers completes their service with server 2?
Exercises
353
44. Cars pass a certain street location according to a Poisson process with rate λ. A woman who wants to cross the street at that location waits until she can see that no cars will come by in the next T time units. (a) Find the probability that her waiting time is 0. (b) Find her expected waiting time. Hint: Condition on the time of the first car. 45. Let {N (t), t 0} be a Poisson process with rate λ, that is independent of the nonnegative random variable T with mean μ and variance σ 2 . Find (a) Cov(T , N (T )), (b) Var(N (T )). 46. Let {N (t), t 0} be a Poisson process with rate λ, that is independent of the sequence X1 , X2 , . . . of independent and identically distributed random variables with mean μ and variance σ 2 . Find ⎛ Cov ⎝N (t),
N (t)
⎞ Xi ⎠
i=1
47. Consider a two-server parallel queuing system where customers arrive according to a Poisson process with rate λ, and where the service times are exponential with rate μ. Moreover, suppose that arrivals finding both servers busy immediately depart without receiving any service (such a customer is said to be lost), whereas those finding at least one free server immediately enter service and then depart when their service is completed. (a) If both servers are presently busy, find the expected time until the next customer enters the system. (b) Starting empty, find the expected time until both servers are busy. (c) Find the expected time between two successive lost customers. 48. Consider an n-server parallel queuing system where customers arrive according to a Poisson process with rate λ, where the service times are exponential random variables with rate μ, and where any arrival finding all servers busy immediately departs without receiving any service. If an arrival finds all servers busy, find (a) the expected number of busy servers found by the next arrival, (b) the probability that the next arrival finds all servers free, (c) the probability that the next arrival finds exactly i of the servers free.
354
5
The Exponential Distribution and the Poisson Process
49. Events occur according to a Poisson process with rate λ. Each time an event occurs, we must decide whether or not to stop, with our objective being to stop at the last event to occur prior to some specified time T , where T > 1/λ. That is, if an event occurs at time t, 0 t T , and we decide to stop, then we win if there are no additional events by time T , and we lose otherwise. If we do not stop when an event occurs and no additional events occur by time T , then we lose. Also, if no events occur by time T , then we lose. Consider the strategy that stops at the first event to occur after some fixed time s, 0s T. (a) Using this strategy, what is the probability of winning? (b) What value of s maximizes the probability of winning? (c) Show that one’s probability of winning when using the preceding strategy with the value of s specified in part (b) is 1/e. 50. The number of hours between successive train arrivals at the station is uniformly distributed on (0, 1). Passengers arrive according to a Poisson process with rate 7 per hour. Suppose a train has just left the station. Let X denote the number of people who get on the next train. Find (a) E[X], (b) Var(X). 51. If an individual has never had a previous automobile accident, then the probability he or she has an accident in the next h time units is βh + o(h); on the other hand, if he or she has ever had a previous accident, then the probability is αh + o(h). Find the expected number of accidents an individual has by time t. 52. Teams 1 and 2 are playing a match. The teams score points according to independent Poisson processes with respective rates λ1 and λ2 . If the match ends when one of the teams has scored k more points than the other, find the probability that team 1 wins. Hint: Relate this to the gambler’s ruin problem. 53. The water level of a certain reservoir is depleted at a constant rate of 1000 units daily. The reservoir is refilled by randomly occurring rainfalls. Rainfalls occur according to a Poisson process with rate 0.2 per day. The amount of water added to the reservoir by a rainfall is 5000 units with probability 0.8 or 8000 units with probability 0.2. The present water level is just slightly below 5000 units. (a) What is the probability the reservoir will be empty after five days? (b) What is the probability the reservoir will be empty sometime within the next ten days?
Exercises
355
54. A viral linear DNA molecule of length, say, 1 is often known to contain a certain “marked position,” with the exact location of this mark being unknown. One approach to locating the marked position is to cut the molecule by agents that break it at points chosen according to a Poisson process with rate λ. It is then possible to determine the fragment that contains the marked position. For instance, letting m denote the location on the line of the marked position, then if L1 denotes the last Poisson event time before m (or 0 if there are no Poisson events in [0, m]), and R1 denotes the first Poisson event time after m (or 1 if there are no Poisson events in [m, 1]), then it would be learned that the marked position lies between L1 and R1 . Find (a) (b) (c) (d)
P {L1 = 0}, P {L1 < x}, 0 < x < m, P {R1 = 1}, P {R1 > x}, m < x < 1.
By repeating the preceding process on identical copies of the DNA molecule, we are able to zero in on the location of the marked position. If the cutting procedure is utilized on n identical copies of the molecule, yielding the data Li , Ri , i = 1, . . . , n, then it follows that the marked position lies between L and R, where L = max Li , i
R = min Ri i
(e) Find E[R − L], and in doing so, show that E[R − L] ∼
2 nλ .
55. Consider a single server queuing system where customers arrive according to a Poisson process with rate λ, service times are exponential with rate μ, and customers are served in the order of their arrival. Suppose that a customer arrives and finds n − 1 others in the system. Let X denote the number in the system at the moment that customer departs. Find the probability mass function of X. Hint: Relate this to a negative binomial random variable. 56. An event independently occurs on each day with probability p. Let N (n) denote the total number of events that occur on the first n days, and let Tr denote the day on which the rth event occurs. (a) What is the distribution of N (n)? (b) What is the distribution of T1 ? (c) What is the distribution of Tr ? (d) Given that N (n) = r, show that the set of r days on which events occurred has the same distribution as a random selection (without replacement) of r of the values 1, 2, . . . , n.
356
5
The Exponential Distribution and the Poisson Process
*57. Events occur according to a Poisson process with rate λ = 2 per hour. (a) What is the probability that no event occurs between 8 P. M . and 9 P. M .? (b) Starting at noon, what is the expected time at which the fourth event occurs? (c) What is the probability that two or more events occur between 6 P. M . and 8 P. M .? 58. Consider the coupon collecting problem where there are m distinct types of mcoupons, and each new coupon collected is type j with probability pj , j =1 pj = 1. Suppose you stop collecting when you have a complete set of at least one of each type. Show that P {i is the last type collected} = E
.
/ (1 − U λj /λi )
j =i
where U is a uniform (0, 1) random variable. 59. There are two types of claims that are made to an insurance company. Let Ni (t) denote the number of type i claims made by time t, and suppose that {N1 (t), t 0} and {N2 (t), t 0} are independent Poisson processes with rates λ1 = 10 and λ2 = 1. The amounts of successive type 1 claims are independent exponential random variables with mean $1000 whereas the amounts from type 2 claims are independent exponential random variables with mean $5000. A claim for $4000 has just been received; what is the probability it is a type 1 claim? *60. Customers arrive at a bank at a Poisson rate λ. Suppose two customers arrived during the first hour. What is the probability that (a) both arrived during the first 20 minutes? (b) at least one arrived during the first 20 minutes? 61. A system has a random number of flaws that we will suppose is Poisson distributed with mean c. Each of these flaws will, independently, cause the system to fail at a random time having distribution G. When a system failure occurs, suppose that the flaw causing the failure is immediately located and fixed. (a) What is the distribution of the number of failures by time t? (b) What is the distribution of the number of flaws that remain in the system at time t? (c) Are the random variables in parts (a) and (b) dependent or independent? 62. Suppose that the number of typographical errors in a new text is Poisson distributed with mean λ. Two proofreaders independently read the text. Suppose that each error is independently found by proofreader i with probability pi , i = 1, 2.
Exercises
357
Let X1 denote the number of errors that are found by proofreader 1 but not by proofreader 2. Let X2 denote the number of errors that are found by proofreader 2 but not by proofreader 1. Let X3 denote the number of errors that are found by both proofreaders. Finally, let X4 denote the number of errors found by neither proofreader. (a) Describe the joint probability distribution of X1 , X2 , X3 , X4 . (b) Show that E[X1 ] 1 − p2 = E[X3 ] p2
and
E[X2 ] 1 − p1 = E[X3 ] p1
Suppose now that λ, p1 , and p2 are all unknown. (c) By using Xi as an estimator of E[Xi ], i = 1, 2, 3, present estimators of p1 , p2 , and λ. (d) Give an estimator of X4 , the number of errors not found by either proofreader. 63. Consider an infinite server queuing system in which customers arrive in accordance with a Poisson process and where the service distribution is exponential with rate μ. Let X(t) denote the number of customers in the system at time t. Find (a) E[X(t + s)|X(s) = n]; (b) Var[X(t + s)|X(s) = n]; Hint: Divide the customers in the system at time t + s into two groups, one consisting of “old” customers and the other of “new” customers. (c) Consider an infinite server queuing system in which customers arrive according to a Poisson process with rate λ, and where the service times are all exponential random variables with rate μ. If there is currently a single customer in the system, find the probability that the system becomes empty when that customer departs. *64. Suppose that people arrive at a bus stop in accordance with a Poisson process with rate λ. The bus departs at time t. Let X denote the total amount of waiting time of all those who get on the bus at time t. We want to determine Var(X). Let N (t) denote the number of arrivals by time t. (a) What is E[X|N (t)]? (b) Argue that Var[X|N (t)] = N (t)t 2 /12. (c) What is Var(X)? 65. An average of 500 people pass the California bar exam each year. A California lawyer practices law, on average, for 30 years. Assuming these numbers remain steady, how many lawyers would you expect California to have in 2050?
358
5
The Exponential Distribution and the Poisson Process
66. Policyholders of a certain insurance company have accidents at times distributed according to a Poisson process with rate λ. The amount of time from when the accident occurs until a claim is made has distribution G. (a) Find the probability there are exactly n incurred but as yet unreported claims at time t. (b) Suppose that each claim amount has distribution F , and that the claim amount is independent of the time that it takes to report the claim. Find the expected value of the sum of all incurred but as yet unreported claims at time t. 67. Satellites are launched into space at times distributed according to a Poisson process with rate λ. Each satellite independently spends a random time (having distribution G) in space before falling to the ground. Find the probability that none of the satellites in the air at time t was launched before time s, where s < t. 68. Suppose that electrical shocks having random amplitudes occur at times distributed according to a Poisson process {N (t), t 0} with rate λ. Suppose that the amplitudes of the successive shocks are independent both of other amplitudes and of the arrival times of shocks, and also that the amplitudes have distribution F with mean μ. Suppose also that the amplitude of a shock decreases with time at an exponential rate α, meaning that an initial amplitude A will have value Ae−αx after an additional time x has elapsed. Let A(t) denote the sum of all amplitudes at time t. That is, A(t) =
N (t)
Ai e−α(t−Si )
i=1
where Ai and Si are the initial amplitude and the arrival time of shock i. (a) Find E[A(t)] by conditioning on N (t). (b) Without any computations, explain why A(t) has the same distribution as does D(t) of Example 5.21. 69. For Example 5.21, let M(t) = E[D(t)] (a) Argue that M(t + h) = M(t) + e−αt M(h) (b) Argue that M(t + h) = M(h) + e−αh M(t) (c) Argue that M(h) = λhμ + o(h) (d) Use parts (a) and (c) to derive and then solve a differential equation for M(t). (e) Use parts (b) and (c) to derive and then solve a differential equation for M(t).
Exercises
359
70. For the infinite server queue with Poisson arrivals and general service distribution G, find the probability that (a) the first customer to arrive is also the first to depart. Let S(t) equal the sum of the remaining service times of all customers in the system at time t. (b) Argue that S(t) is a compound Poisson random variable. (c) Find E[S(t)]. (d) Find Var(S(t)). 71. Let Sn denote the time of the nth event of the Poisson process {N (t), t 0} having rate λ. Show, for an arbitrary function g, that the random variable N (t) i=1 g(Si ) has the same distribution as the compound Poisson random variN (t) able i=1 g(Ui ), where U1 , U2 , . . . is a sequence of independent and identically distributed uniform (0, t) random variables that is independent of N , a Poisson random variable with mean λt. Consequently, conclude that ⎤ ⎛ ⎞ ⎡ t t N (t) N (t) g(Si )⎦ = λ g(x) dx Var ⎝ g(Si )⎠ = λ g 2 (x) dx E⎣ i=1
0
i=1
0
72. A cable car starts off with n riders. The times between successive stops of the car are independent exponential random variables with rate λ. At each stop one rider gets off. This takes no time, and no additional riders get on. After a rider gets off the car, he or she walks home. Independently of all else, the walk takes an exponential time with rate μ. (a) What is the distribution of the time at which the last rider departs the car? (b) Suppose the last rider departs the car at time t. What is the probability that all the other riders are home at that time? 73. Shocks occur according to a Poisson process with rate λ, and each shock independently causes a certain system to fail with probability p. Let T denote the time at which the system fails and let N denote the number of shocks that it takes. (a) Find the conditional distribution of T given that N = n. (b) Calculate the conditional distribution of N , given that T = t, and notice that it is distributed as 1 plus a Poisson random variable with mean λ(1 − p)t. (c) Explain how the result in part (b) could have been obtained without any calculations. 74. The number of missing items in a certain location, call it X, is a Poisson random variable with mean λ. When searching the location, each item will independently be found after an exponentially distributed time with rate μ. A reward of R is received for each item found, and a searching cost of C per unit of search time is incurred. Suppose that you search for a fixed time t and then stop.
360
5
The Exponential Distribution and the Poisson Process
(a) Find your total expected return. (b) Find the value of t that maximizes the total expected return. (c) The policy of searching for a fixed time is a static policy. Would a dynamic policy, which allows the decision as to whether to stop at each time t, depend on the number already found by t be beneficial? Hint: How does the distribution of the number of items not yet found by time t depend on the number already found by that time? 75. Suppose that the times between successive arrivals of customers at a singleserver station are independent random variables having a common distribution F . Suppose that when a customer arrives, he or she either immediately enters service if the server is free or else joins the end of the waiting line if the server is busy with another customer. When the server completes work on a customer, that customer leaves the system and the next waiting customer, if there are any, enters service. Let Xn denote the number of customers in the system immediately before the nth arrival, and let Yn denote the number of customers that remain in the system when the nth customer departs. The successive service times of customers are independent random variables (which are also independent of the interarrival times) having a common distribution G. (a) If F is the exponential distribution with rate λ, which, if any, of the processes {Xn }, {Yn } is a Markov chain? (b) If G is the exponential distribution with rate μ, which, if any, of the processes {Xn }, {Yn } is a Markov chain? (c) Give the transition probabilities of any Markov chains in parts (a) and (b). 76. For the model of Example 5.27, find the mean and variance of the number of customers served in a busy period. 77. Events occur according to a nonhomogeneous Poisson process whose mean value function is given by m(t) = t 2 + 2t,
t 0
What is the probability that n events occur between times t = 4 and t = 5? 78. A store opens at 8 A . M . From 8 until 10 customers arrive at a Poisson rate of four an hour. Between 10 and 12 they arrive at a Poisson rate of eight an hour. From 12 to 2 the arrival rate increases steadily from eight per hour at 12 to ten per hour at 2; and from 2 to 5 the arrival rate drops steadily from ten per hour at 2 to four per hour at 5. Determine the probability distribution of the number of customers that enter the store on a given day. *79. Consider a nonhomogeneous Poisson process whose intensity function λ(t) is bounded and continuous. Show that such a process is equivalent to a process of
Exercises
361
counted events from a (homogeneous) Poisson process having rate λ, where an event at time t is counted (independent of the past) with probability λ(t)/λ; and where λ is chosen so that λ(s) < λ for all s. 80. Let T1 , T2 , . . . denote the interarrival times of events of a nonhomogeneous Poisson process having intensity function λ(t). (a) Are the Ti independent? (b) Are the Ti identically distributed? (c) Find the distribution of T1 . 81. (a) Let {N (t), t 0} be a nonhomogeneous Poisson process with mean value function m(t). Given N (t) = n, show that the unordered set of arrival times has the same distribution as n independent and identically distributed random variables having distribution function ⎧ ⎨ m(x) , xt F (x) = m(t) ⎩ 1, xt (b) Suppose that workmen incur accidents in accordance with a nonhomogeneous Poisson process with mean value function m(t). Suppose further that each injured man is out of work for a random amount of time having distribution F . Let X(t) be the number of workers who are out of work at time t. By using part (a), find E[X(t)]. 82. Let X1 , X2 , . . . be independent positive continuous random variables with a common density function f , and suppose this sequence is independent of N , a Poisson random variable with mean λ. Define N (t) = number of i N : Xi t Show that {N (t), t 0} is a nonhomogeneous Poisson process with intensity function λ(t) = λf (t). 83. Suppose that {N0 (t), t 0} is a Poisson process with rate λ = 1. Let λ(t) denote a nonnegative function of t, and let m(t) =
t
λ(s) ds 0
Define N(t) by N (t) = N0 (m(t)) Argue that {N (t), t 0} is a nonhomogeneous Poisson process with intensity function λ(t), t 0.
362
5
The Exponential Distribution and the Poisson Process
Hint: Make use of the identity m(t + h) − m(t) = m (t)h + o(h) *84. Let X1 , X2 , . . . be independent and identically distributed nonnegative continuous random variables having density function f (x). We say that a record occurs at time n if Xn is larger than each of the previous values X1 , . . . , Xn−1 . (A record automatically occurs at time 1.) If a record occurs at time n, then Xn is called a record value. In other words, a record occurs whenever a new high is reached, and that new high is called the record value. Let N (t) denote the number of record values that are less than or equal to t. Characterize the process {N (t), t 0} when (a) f is an arbitrary continuous density function. (b) f (x) = λe−λx . Hint: Finish the following sentence: There will be a record whose value is between t and t + dt if the first Xi that is greater than t lies between . . . 85. An insurance company pays out claims on its life insurance policies in accordance with a Poisson process having rate λ = 5 per week. If the amount of money paid on each policy is exponentially distributed with mean $2000, what is the mean and variance of the amount of money paid by the insurance company in a four-week span? 86. In good years, storms occur according to a Poisson process with rate 3 per unit time, while in other years they occur according to a Poisson process with rate 5 per unit time. Suppose next year will be a good year with probability 0.3. Let N(t) denote the number of storms during the first t time units of next year. (a) Find P {N (t) = n}. (b) Is {N (t)} a Poisson process? (c) Does {N (t)} have stationary increments? Why or why not? (d) Does it have independent increments? Why or why not? (e) If next year starts off with three storms by time t = 1, what is the conditional probability it is a good year? 87. Determine Cov[X(t), X(t + s)] when {X(t), t 0} is a compound Poisson process. 88. Customers arrive at the automatic teller machine in accordance with a Poisson process with rate 12 per hour. The amount of money withdrawn on each transaction is a random variable with mean $30 and standard deviation $50. (A negative withdrawal means that money was deposited.) The machine is in use for 15 hours daily. Approximate the probability that the total daily withdrawal is less than $6000.
Exercises
363
89. Some components of a two-component system fail after receiving a shock. Shocks of three types arrive independently and in accordance with Poisson processes. Shocks of the first type arrive at a Poisson rate λ1 and cause the first component to fail. Those of the second type arrive at a Poisson rate λ2 and cause the second component to fail. The third type of shock arrives at a Poisson rate λ3 and causes both components to fail. Let X1 and X2 denote the survival times for the two components. Show that the joint distribution of X1 and X2 is given by P {X1 > s, X1 > t} = exp{−λ1 s − λ2 t − λ3 max(s, t)} This distribution is known as the bivariate exponential distribution. 90. In Exercise 89 show that X1 and X2 both have exponential distributions. *91. Let X1 , X2 , . . . , Xn be independent and identically distributed exponential random variables. Show that the probability that the largest of them is greater than the sum of the others is n/2n−1 . That is, if M = max Xj j
then show
P M>
Hint: What is P {X1 >
n
n
Xi − M =
i=1
n 2n−1
i=2 Xi }?
92. Prove Equation (5.22). 93. Prove that (a) max(X1 , X2 ) = X1 + X2 − min(X1 , X2 ) and, in general, n (b) max(X1 , . . . , Xn ) = Xi − min(Xi , Xj ) i<j
1
+
min(Xi , Xj , Xk ) + · · ·
i<j t} = e−λπt , (b) E[X] = √1 . 2
2 λ
95. Let {N (t), t 0} be a conditional Poisson process with a random rate L. (a) Derive an expression for E[L|N (t) = n]. (b) Find, for s > t, E[N (s)|N (t) = n]. (c) Find, for s < t, E[N (s)|N (t) = n]. 96. For the conditional Poisson process, let m1 = E[L], m2 = E[L2 ]. In terms of m1 and m2 , find Cov(N (s), N (t)) for s t.
References 1. H. Cramér and M. Leadbetter, “Stationary and Related Stochastic Processes,” John Wiley, New York, 1966. 2. S. Ross, “Stochastic Processes,” Second Edition, John Wiley, New York, 1996. 3. S. Ross, “Probability Models for Computer Science,” Academic Press, 2002.
Continuous-Time Markov Chains
6 6.1.
Introduction
In this chapter we consider a class of probability models that has a wide variety of applications in the real world. The members of this class are the continuous-time analogs of the Markov chains of Chapter 4 and as such are characterized by the Markovian property that, given the present state, the future is independent of the past. One example of a continuous-time Markov chain has already been met. This is the Poisson process of Chapter 5. For if we let the total number of arrivals by time t [that is, N (t)] be the state of the process at time t, then the Poisson process is a continuous-time Markov chain having states 0, 1, 2, . . . that always proceeds from state n to state n + 1, where n 0. Such a process is known as a pure birth process since when a transition occurs the state of the system is always increased by one. More generally, an exponential model which can go (in one transition) only from state n to either state n − 1 or state n + 1 is called a birth and death model. For such a model, transitions from state n to state n + 1 are designated as births, and those from n to n − 1 as deaths. Birth and death models have wide applicability in the study of biological systems and in the study of waiting line systems in which the state represents the number of customers in the system. These models will be studied extensively in this chapter. In Section 6.2 we define continuous-time Markov chains and then relate them to the discrete-time Markov chains of Chapter 4. In Section 6.3 we consider birth and death processes and in Section 6.4 we derive two sets of differential equations— the forward and backward equations—that describe the probability laws for the system. The material in Section 6.5 is concerned with determining the limiting (or long-run) probabilities connected with a continuous-time Markov chain. In Section 6.6 we consider the topic of time reversibility. We show that all birth and death processes are time reversible, and then illustrate the importance of this ob-
365
366
6
Continuous-Time Markov Chains
servation to queueing systems. In the final section we show how to “uniformize” Markov chains, a technique useful for numerical computations.
6.2.
Continuous-Time Markov Chains
Suppose we have a continuous-time stochastic process {X(t), t 0} taking on values in the set of nonnegative integers. In analogy with the definition of a discrete-time Markov chain, given in Chapter 4, we say that the process {X(t), t 0} is a continuous-time Markov chain if for all s, t 0 and nonnegative integers i, j, x(u), 0 u < s P {X(t + s) = j |X(s) = i, X(u) = x(u), 0 u < s} = P {X(t + s) = j |X(s) = i} In other words, a continuous-time Markov chain is a stochastic process having the Markovian property that the conditional distribution of the future X(t + s) given the present X(s) and the past X(u), 0 u < s, depends only on the present and is independent of the past. If, in addition, P {X(t + s) = j |X(s) = i} is independent of s, then the continuous-time Markov chain is said to have stationary or homogeneous transition probabilities. All Markov chains considered in this text will be assumed to have stationary transition probabilities. Suppose that a continuous-time Markov chain enters state i at some time, say, time 0, and suppose that the process does not leave state i (that is, a transition does not occur) during the next ten minutes. What is the probability that the process will not leave state i during the following five minutes? Now since the process is in state i at time 10 it follows, by the Markovian property, that the probability that it remains in that state during the interval [10, 15] is just the (unconditional) probability that it stays in state i for at least five minutes. That is, if we let Ti denote the amount of time that the process stays in state i before making a transition into a different state, then P {Ti > 15|Ti > 10} = P {Ti > 5} or, in general, by the same reasoning, P {Ti > s + t|Ti > s} = P {Ti > t} for all s, t 0. Hence, the random variable Ti is memoryless and must thus (see Section 5.2.2) be exponentially distributed.
6.2.
Continuous-Time Markov Chains
367
In fact, the preceding gives us another way of defining a continuous-time Markov chain. Namely, it is a stochastic process having the properties that each time it enters state i (i) the amount of time it spends in that state before making a transition into a different state is exponentially distributed with mean, say, 1/vi , and (ii) when the process leaves state i, it next enters state j with some probability, say, Pij . Of course, the Pij must satisfy
Pii = 0,
all i
Pij = 1,
all i
j
In other words, a continuous-time Markov chain is a stochastic process that moves from state to state in accordance with a (discrete-time) Markov chain, but is such that the amount of time it spends in each state, before proceeding to the next state, is exponentially distributed. In addition, the amount of time the process spends in state i, and the next state visited, must be independent random variables. For if the next state visited were dependent on Ti , then information as to how long the process has already been in state i would be relevant to the prediction of the next state—and this contradicts the Markovian assumption.
Example 6.1 (A Shoeshine Shop) Consider a shoeshine establishment consisting of two chairs—chair 1 and chair 2. A customer upon arrival goes initially to chair 1 where his shoes are cleaned and polish is applied. After this is done the customer moves on to chair 2 where the polish is buffed. The service times at the two chairs are assumed to be independent random variables that are exponentially distributed with respective rates μ1 and μ2 . Suppose that potential customers arrive in accordance with a Poisson process having rate λ, and that a potential customer will enter the system only if both chairs are empty. The preceding model can be analyzed as a continuous-time Markov chain, but first we must decide upon an appropriate state space. Since a potential customer will enter the system only if there are no other customers present, it follows that there will always either be 0 or 1 customers in the system. However, if there is 1 customer in the system, then we would also need to know which chair he was presently in. Hence, an appropriate state space might consist of the three states 0, 1, and 2 where the states have the following interpretation: State 0 1 2
Interpretation system is empty a customer is in chair 1 a customer is in chair 2
368
6
Continuous-Time Markov Chains
We leave it as an exercise for you to verify that v0 = λ,
v 1 = μ1 ,
v 2 = μ2 ,
P01 = P12 = P20 = 1
6.3.
Birth and Death Processes
Consider a system whose state at any time is represented by the number of people in the system at that time. Suppose that whenever there are n people in the system, then (i) new arrivals enter the system at an exponential rate λn , and (ii) people leave the system at an exponential rate μn . That is, whenever there are n persons in the system, then the time until the next arrival is exponentially distributed with mean 1/λn and is independent of the time until the next departure which is itself exponentially distributed with mean 1/μn . Such a system is called a birth and ∞ death process. The parameters {λn }∞ n=0 and {μn }n=1 are called, respectively, the arrival (or birth) and departure (or death) rates. Thus, a birth and death process is a continuous-time Markov chain with states {0, 1, . . .} for which transitions from state n may go only to either state n − 1 or state n + 1. The relationships between the birth and death rates and the state transition rates and probabilities are v 0 = λ0 , v i = λi + μi ,
i>0
P0 1 = 1, λi , λi + μi μi , Pi,i−1 = λi + μi Pi,i+1 =
i>0 i>0
The preceding follows, because if there are i in the system, then the next state will be i + 1 if a birth occurs before a death, and the probability that an exponential random variable with rate λi will occur earlier than an (independent) exponential with rate μi is λi /(λi + μi ). Moreover, the time until either a birth or a death occurs is exponentially distributed with rate λi + μi (and so, vi = λi + μi ).
Example 6.2 (The Poisson Process) Consider a birth and death process for which μn = 0, λn = λ,
for all n 0 for all n 0
This is a process in which departures never occur, and the time between successive arrivals is exponential with mean 1/λ. Hence, this is just the Poisson process.
6.3.
Birth and Death Processes
369
A birth and death process for which μn = 0 for all n is called a pure birth process. Another pure birth process is given by the next example.
Example 6.3 (A Birth Process with Linear Birth Rate) Consider a population whose members can give birth to new members but cannot die. If each member acts independently of the others and takes an exponentially distributed amount of time, with mean 1/λ, to give birth, then if X(t) is the population size at time t, then {X(t), t 0} is a pure birth process with λn = nλ, n 0. This follows since if the population consists of n persons and each gives birth at an exponential rate λ, then the total rate at which births occur is nλ. This pure birth process is known as a Yule process after G. Yule, who used it in his mathematical theory of evolution.
Example 6.4 (A Linear Growth Model with Immigration) A model in which μn = nμ, λn = nλ + θ,
n1 n0
is called a linear growth process with immigration. Such processes occur naturally in the study of biological reproduction and population growth. Each individual in the population is assumed to give birth at an exponential rate λ; in addition, there is an exponential rate of increase θ of the population due to an external source such as immigration. Hence, the total birth rate where there are n persons in the system is nλ + θ . Deaths are assumed to occur at an exponential rate μ for each member of the population, so μn = nμ. Let X(t) denote the population size at time t. Suppose that X(0) = i and let M(t) = E[X(t)] We will determine M(t) by deriving and then solving a differential equation that it satisfies. We start by deriving an equation for M(t + h) by conditioning on X(t). This yields M(t + h) = E[X(t + h)] = E[E[X(t + h)|X(t)]] Now, given the size of the population at time t then, ignoring events whose probability is o(h), the population at time t + h will either increase in size by 1 if a birth or an immigration occurs in (t, t + h), or decrease by 1 if a death occurs in this interval, or remain the same if neither of these two possibilities occurs. That
370
6
Continuous-Time Markov Chains
is, given X(t), ⎧ ⎨X(t) + 1, X(t + h) = X(t) − 1, ⎩ X(t),
with probability [θ + X(t)λ]h + o(h) with probability X(t)μh + o(h) with probability 1 − [θ + X(t)λ + X(t)μ] h + o(h)
Therefore, E[X(t + h)|X(t)] = X(t) + [θ + X(t)λ − X(t)μ]h + o(h) Taking expectations yields M(t + h) = M(t) + (λ − μ)M(t)h + θ h + o(h) or, equivalently, M(t + h) − M(t) o(h) = (λ − μ)M(t) + θ + h h Taking the limit as h → 0 yields the differential equation M (t) = (λ − μ)M(t) + θ If we now define the function h(t) by h(t) = (λ − μ)M(t) + θ then h (t) = (λ − μ)M (t) Therefore, the differential equation (6.1) can be rewritten as h (t) = h(t) λ−μ or h (t) =λ−μ h(t) Integration yields log[h(t)] = (λ − μ)t + c or
(6.1)
6.3.
Birth and Death Processes
371
h(t) = Ke(λ−μ)t Putting this back in terms of M(t) gives θ + (λ − μ)M(t) = Ke(λ−μ)t To determine the value of the constant K, we use the fact that M(0) = i and evaluate the preceding at t = 0. This gives θ + (λ − μ)i = K Substituting this back in the preceding equation for M(t) yields the following solution for M(t): M(t) =
θ [e(λ−μ)t − 1] + ie(λ−μ)t λ−μ
Note that we have implicitly assumed that λ = μ. If λ = μ, then the differential equation (6.1) reduces to M (t) = θ
(6.2)
Integrating (6.2) and using that M(0) = i gives the solution M(t) = θ t + i
Example 6.5 (The Queueing System M/M/1) Suppose that customers arrive at a single-server service station in accordance with a Poisson process having rate λ. That is, the times between successive arrivals are independent exponential random variables having mean 1/λ. Upon arrival, each customer goes directly into service if the server is free; if not, then the customer joins the queue (that is, he waits in line). When the server finishes serving a customer, the customer leaves the system and the next customer in line, if there are any waiting, enters the service. The successive service times are assumed to be independent exponential random variables having mean 1/μ. The preceding is known as the M/M/1 queueing system. The first M refers to the fact that the interarrival process is Markovian (since it is a Poisson process) and the second to the fact that the service distribution is exponential (and, hence, Markovian). The 1 refers to the fact that there is a single server. If we let X(t) denote the number in the system at time t then {X(t), t 0} is a birth and death process with μn = μ, λn = λ,
n1 n0
372
6
Continuous-Time Markov Chains
Example 6.6 (A Multiserver Exponential Queueing System) Consider an exponential queueing system in which there are s servers available, each serving at rate μ. An entering customer first waits in line and then goes to the first free server. This is a birth and death process with parameters μn =
nμ, sμ,
λn = λ,
1ns n>s n0
To see why this is true, reason as follows: If there are n customers in the system, where n s, then n servers will be busy. Since each of these servers works at rate μ, the total departure rate will be nμ. On the other hand, if there are n customers in the system, where n > s, then all s of the servers will be busy, and thus the total departure rate will be sμ. This is known as an M/M/s queueing model. Consider now a general birth and death process with birth rates {λn } and death rates {μn }, where μ0 = 0, and let Ti denote the time, starting from state i, it takes for the process to enter state i + 1, i 0. We will recursively compute E[Ti ], i 0, by starting with i = 0. Since T0 is exponential with rate λ0 , we have that E[T0 ] =
1 λ0
For i > 0, we condition whether the first transition takes the process into state i − 1 or i + 1. That is, let 1, if the first transition from i is to i + 1 Ii = 0, if the first transition from i is to i − 1 and note that 1 , λi + μ i 1 E[Ti |Ii = 0] = + E[Ti−1 ] + E[Ti ] λi + μ i
E[Ti |Ii = 1] =
(6.3)
This follows since, independent of whether the first transition is from a birth or death, the time until it occurs is exponential with rate λi + μi ; now if this first transition is a birth, then the population size is at i + 1, so no additional time is needed; whereas if it is death, then the population size becomes i − 1 and the additional time needed to reach i + 1 is equal to the time it takes to return to state i (and this has mean E[Ti−1 ]) plus the additional time it then takes to reach i + 1
6.3.
Birth and Death Processes
373
(and this has mean E[Ti ]). Hence, since the probability that the first transition is a birth is λi /(λi + μi ), we see that E[Ti ] =
1 μi + (E[Ti−1 ] + E[Ti ]) λi + μ i λi + μ i
or, equivalently, E[Ti ] =
1 μi + E[Ti−1 ], λi λi
i1
Starting with E[T0 ] = 1/λ0 , the preceding yields an efficient method to successively compute E[T1 ], E[T2 ], and so on. Suppose now that we wanted to determine the expected time to go from state i to state j where i < j . This can be accomplished using the preceding by noting that this quantity will equal E[Ti ] + E[Ti+1 ] + · · · + E[Tj −1 ].
Example 6.7 For the birth and death process having parameters λi ≡ λ, μi ≡ μ,
1 μ + E[Ti−1 ] λ λ 1 = (1 + μE[Ti−1 ]) λ Starting with E[T0 ] = 1/λ, we see that μ 1 1+ , E[T1 ] = λ λ . 2 / μ μ 1 E[T2 ] = 1+ + λ λ λ E[Ti ] =
and, in general, E[Ti ] = =
. 2 i / μ μ 1 μ 1+ + + ··· + λ λ λ λ 1 − (μ/λ)i+1 , λ−μ
i0
The expected time to reach state j , starting at state k, k < j , is E[time to go from k to j ] =
j −1
E[Ti ]
i=k
=
(μ/λ)k+1 [1 − (μ/λ)j −k ] j −k − λ−μ λ−μ 1 − μ/λ
374
6
Continuous-Time Markov Chains
The foregoing assumes that λ = μ. If λ = μ, then i +1 , λ j (j + 1) − k(k + 1) E[time to go from k to j ] = 2λ E[Ti ] =
We can also compute the variance of the time to go from 0 to i + 1 by utilizing the conditional variance formula. First note that Equation (6.3) can be written as E[Ti |Ii ] =
1 + (1 − Ii )(E[Ti−1 ] + E[Ti ]) λi + μ i
Thus Var(E[Ti |Ii ]) = (E[Ti−1 ] + E[Ti ])2 Var(Ii ) = (E[Ti−1 ] + E[Ti ])2
μi λi (μi + λi )2
(6.4)
where Var(Ii ) is as shown since Ii is a Bernoulli random variable with parameter p = λi /(λi + μi ). Also, note that if we let Xi denote the time until the transition from i occurs, then Var(Ti |Ii = 1) = Var(Xi |Ii = 1) = Var(Xi ) =
1 (λi + μi )2
(6.5)
where the preceding uses the fact that the time until transition is independent of the next state visited. Also, Var(Ti |Ii = 0) = Var(Xi + time to get back to i + time to then reach i + 1) = Var(Xi ) + Var(Ti−1 ) + Var(Ti )
(6.6)
where the foregoing uses the fact that the three random variables are independent. We can rewrite Equations (6.5) and (6.6) as Var(Ti |Ii ) = Var(Xi ) + (1 − Ii )[Var(Ti−1 ) + Var(Ti )] so E[Var(Ti |Ii )] =
1 μi + [Var(Ti−1 ) + Var(Ti )] (μi + λi )2 μi + λi
(6.7)
6.4.
The Transition Probability Function Pij (t)
375
Hence, using the conditional variance formula, which states that Var(Ti ) is the sum of Equations (6.7) and (6.4), we obtain Var(Ti ) =
1 μi + [Var(Ti−1 ) + Var(Ti )] 2 μi + λi (μi + λi ) +
μi λi (E[Ti−1 ] + E[Ti ])2 (μi + λi )2
or, equivalently, Var(Ti ) =
μi 1 μi + Var(Ti−1 ) + (E[Ti−1 ] + E[Ti ])2 λi (λi + μi ) λi μ i + λi
Starting with Var(T0 ) = 1/λ20 and using the former recursion to obtain the expectations, we can recursively compute Var(Ti ). In addition, if we want the variance of the time to reach state j , starting from state k, k < j , then this can be expressed as the time to go from k to k + 1 plus the additional time to go from k + 1 to k + 2, and so on. Since, by the Markovian property, these successive random variables are independent, it follows that
Var(time to go from k to j ) =
j −1
Var(Ti )
i=k
6.4.
The Transition Probability Function Pij (t)
Let Pij (t) = P {X(t + s) = j |X(s) = i} denote the probability that a process presently in state i will be in state j a time t later. These quantities are often called the transition probabilities of the continuous-time Markov chain. We can explicitly determine Pij (t) in the case of a pure birth process having distinct birth rates. For such a process, let Xk denote the time the process spends in state k before making a transition into state k + 1, k 1. Suppose that the process is presently in state i, and let j > i. Then, as Xi is the time it spends in state i before moving to state i + 1, and Xi+1 is the time it then spends in state j −1 i + 1 before moving to state i + 2, and so on, it follows that k=i Xk is the time
376
6
Continuous-Time Markov Chains
it takes until the process enters state j . Now, if the process has not yet entered state j by time t, then its state at time t is smaller than j , and vice versa. That is, X(t) < j ⇔ Xi + · · · + Xj −1 > t Therefore, for i < j , we have for a pure birth process that
P {X(t) < j |X(0) = i} = P
j −1
Xk > t
k=i
However, since Xi , . . . , Xj −1 are independent exponential random variables with respective rates λi , . . . , λj −1 , we obtain from preceding and Equation (5.9), jthe −1 which gives the tail distribution function of k=i Xk , that
P {X(t) < j |X(0) = i} =
j −1
e
j −1
−λk t
r=k, r=i
k=i
λr λr − λk
Replacing j by j + 1 in the preceding gives that
P {X(t) < j + 1|X(0) = i} =
j
e
−λk t
k=i
j r=k, r=i
λr λr − λk
Since P {X(t) = j |X(0) = i} = P {X(t) < j + 1|X(0) = i} − P {X(t) < j |X(0) = i} and since Pii (t) = P {Xi > t} = e−λi t , we have shown the following.
Proposition 6.1 For a pure birth process having λi = λj when i = j
Pij (t) =
j k=i
e−λk t
j r=k, r=i
j −1
j −1
k=i
r=k, r=i
λr − e−λk t λr − λk Pii (t) = e−λi t
λr , λr − λk
i<j
6.4.
The Transition Probability Function Pij (t)
377
Example 6.8 Consider the Yule process, which is a pure birth process in which each individual in the population independently gives birth at rate λ, and so λn = nλ, n 1. Letting i = 1, we obtain from Proposition 6.1 P1j (t) =
j
r=k, r=1
k=1
=e
j
e−kλt
−j λt
j −1 r=1
j −1
j −1
k=1
r=k, r=1
r − e−kλt r −k j −1
r + e−kλt r −j
r=k, r=1
k=1
= e−j λt (−1)j −1 +
j −1
e−kλt
k=1
j
j −1 j −k
r r −k
r − r −k
j −1 r=k, r=1
j −1 r=k, r=1
r r −k
r r −k
Now, k j −k
j −1 r=k, r=1
(j − 1)! r = r −k (1 − k)(2 − k) · · · (k − 1 − k)(j − k)! = (−1)
k−1
j −1 k−1
so j j − 1 −kλt e P1j (t) = (−1)k−1 k−1 k=1
=e
−λt
=e
−λt
j −1 j − 1 −iλt e (−1)i i i=0
(1 − e−λt )j −1
Thus, starting with a single individual, the population size at time t has a geometric distribution with mean eλt . If the population starts with i individuals, then we can regard each of these individuals as starting her own independent Yule process, and so the population at time t will be the sum of i independent and identically distributed geometric random variables with parameter e−λt . But this means that the conditional distribution of X(t), given that X(0) = i, is the same as the distribution of the number of times that a coin that lands heads on each flip with probability e−λt must be flipped to amass a total of i heads. Hence, the
378
6
Continuous-Time Markov Chains
population size at time t has a negative binomial distribution with parameters i and e−λt , so
j − 1 −iλt Pij (t) = e (1 − e−λt )j −i , i −1
j i1
[We could, of course, have used Proposition 6.1 to immediately obtain an equation for Pij (t), rather than just using it for P1j (t), but the algebra that would have then been needed to show the equivalence of the resulting expression to the preceding result is somewhat involved.] We shall now derive a set of differential equations that the transition probabilities Pij (t) satisfy in a general continuous-time Markov chain. However, first we need a definition and a pair of lemmas. For any pair of states i and j , let qij = vi Pij Since vi is the rate at which the process makes a transition when in state i and Pij is the probability that this transition is into state j , it follows that qij is the rate, when in state i, at which the process makes a transition into state j . The quantities qij are called the instantaneous transition rates. Since vi =
j
vi Pij =
qij
j
and Pij =
qij qij = vi j qij
it follows that specifying the instantaneous transition rates determines the parameters of the continuous-time Markov chain.
Lemma 6.2 (a) (b)
1 − Pii (h) = vi h Pij (h) limh→0 = qij h
limh→0
when i = j
Proof We first note that since the amount of time until a transition occurs is exponentially distributed it follows that the probability of two or more transitions
6.4.
The Transition Probability Function Pij (t)
379
in a time h is o(h). Thus, 1 − Pii (h), the probability that a process in state i at time 0 will not be in state i at time h, equals the probability that a transition occurs within time h plus something small compared to h. Therefore, 1 − Pii (h) = vi h + o(h) and part (a) is proven. To prove part (b), we note that Pij (h), the probability that the process goes from state i to state j in a time h, equals the probability that a transition occurs in this time multiplied by the probability that the transition is into state j , plus something small compared to h. That is, Pij (h) = hvi Pij + o(h) and part (b) is proven.
Lemma 6.3 For all s 0, t 0, Pij (t + s) =
∞
Pik (t)Pkj (s)
(6.8)
k=0
Proof In order for the process to go from state i to state j in time t + s, it must be somewhere at time t and thus Pij (t + s) = P {X(t + s) = j |X(0) = i} =
∞
P {X(t + s) = j, X(t) = k|X(0) = i}
k=0
=
∞
P {X(t + s) = j |X(t) = k, X(0) = i} · P {X(t) = k|X(0) = i}
k=0
=
∞
P {X(t + s) = j |X(t) = k} · P {X(t) = k|X(0) = i}
k=0
=
∞
Pkj (s)Pik (t)
k=0
and the proof is completed.
380
6
Continuous-Time Markov Chains
The set of equations (6.8) is known as the Chapman–Kolmogorov equations. From Lemma 6.3, we obtain Pij (h + t) − Pij (t) =
∞
Pik (h)Pkj (t) − Pij (t)
k=0
=
Pik (h)Pkj (t) − [1 − Pii (h)]Pij (t)
k=i
and thus
/ . Pij (t + h) − Pij (t) Pik (h) 1 − Pii (h) = lim Pkj (t) − Pij (t) h→0 h→0 h h h lim
k=i
Now assuming that we can interchange the limit and the summation in the preceding and applying Lemma 6.2, we obtain qik Pkj (t) − vi Pij (t) Pij (t) = k=i
It turns out that this interchange can indeed be justified and, hence, we have the following theorem.
Theorem 6.1 (Kolmogorov’s Backward Equations) For all states i, j , and times t 0,
Pij (t) =
qik Pkj (t) − vi Pij (t)
k=i
Example 6.9 The backward equations for the pure birth process become Pij (t) = λi Pi+1,j (t) − λi Pij (t)
Example 6.10 The backward equations for the birth and death process become
Pij (t) = (λi
.
(t) = λ0 P1j (t) − λ0 P0j (t), P0j
/ μi λi + μi ) Pi+1,j (t) + Pi−1,j (t) − (λi + μi )Pij (t) λi + μ i λi + μi
or equivalently (t) = λ [P (t) − P (t)], P0j 0 1j 0j
Pij (t) = λi Pi+1,j (t) + μi Pi−1,j (t) − (λi + μi )Pij (t),
i>0
(6.9)
6.4.
The Transition Probability Function Pij (t)
381
Example 6.11 (A Continuous-Time Markov Chain Consisting of Two States) Consider a machine that works for an exponential amount of time having mean 1/λ before breaking down; and suppose that it takes an exponential amount of time having mean 1/μ to repair the machine. If the machine is in working condition at time 0, then what is the probability that it will be working at time t = 10? To answer this question, we note that the process is a birth and death process (with state 0 meaning that the machine is working and state 1 that it is being repaired) having parameters λ0 = λ, λi = 0, i = 0,
μ1 = μ, μi = 0, i = 1
We shall derive the desired probability, namely, P00 (10) by solving the set of differential equations given in Example 6.10. From Equation (6.9), we obtain (t) = λ[P10 (t) − P00 (t)], P00
(6.10)
P10 (t) = μP00 (t) − μP10 (t)
(6.11)
Multiplying Equation (6.10) by μ and Equation (6.11) by λ and then adding the two equations yields μP00 (t) + λP10 (t) = 0
By integrating, we obtain μP00 (t) + λP10 (t) = c However, since P00 (0) = 1 and P10 (0) = 0, we obtain c = μ and hence μP00 (t) + λP10 (t) = μ or equivalently λP10 (t) = μ[1 − P00 (t)] By substituting this result in Equation (6.10), we obtain P00 (t) = μ[1 − P00 (t)] − λP00 (t)
= μ − (μ + λ)P00 (t) Letting h(t) = P00 (t) −
μ μ+λ
(6.12)
382
6
Continuous-Time Markov Chains
we have . h (t) = μ − (μ + λ) h(t) +
μ μ+λ
/
= −(μ + λ)h(t) or h (t) = −(μ + λ) h(t) By integrating both sides, we obtain log h(t) = −(μ + λ)t + c or h(t) = Ke−(μ+λ)t and thus P00 (t) = Ke−(μ+λ)t +
μ μ+λ
which finally yields, by setting t = 0 and using the fact that P00 (0) = 1, P00 (t) =
λ μ e−(μ+λ)t + μ+λ μ+λ
From Equation (6.12), this also implies that P10 (t) =
μ −(μ+λ)t μ − e μ+λ μ+λ
Hence, our desired probability P00 (10) equals P00 (10) =
λ μ e−10(μ+λ) + μ+λ μ+λ
Another set of differential equations, different from the backward equations, may also be derived. This set of equations, known as Kolmogorov’s forward
6.4.
383
The Transition Probability Function Pij (t)
equations is derived as follows. From the Chapman–Kolmogorov equations (Lemma 6.3), we have Pij (t + h) − Pij (t) =
∞
Pik (t)Pkj (h) − Pij (t)
k=0
=
Pik (t)Pkj (h) − [1 − Pjj (h)]Pij (t)
k=j
and thus . / Pkj (h) Pij (t + h) − Pij (t) 1 − Pjj (h) Pik (t) = lim − Pij (t) h→0 h→0 h h h lim
k=j
and, assuming that we can interchange limit with summation, we obtain from Lemma 6.2 Pij (t) = qkj Pik (t) − vj Pij (t) k=j
Unfortunately, we cannot always justify the interchange of limit and summation and thus the preceding is not always valid. However, they do hold in most models, including all birth and death processes and all finite state models. We thus have the following.
Theorem 6.2 (Kolmogorov’s Forward Equations) Under suitable regularity conditions, Pij (t) =
qkj Pik (t) − vj Pij (t)
(6.13)
k=j
We shall now solve the forward equations for the pure birth process. For this process, Equation (6.13) reduces to Pij (t) = λj −1 Pi,j −1 (t) − λj Pij (t) However, by noting that Pij (t) = 0 whenever j < i (since no deaths can occur), we can rewrite the preceding equation to obtain Pii (t) = −λi Pii (t),
Pij (t) = λj −1 Pi,j −1 (t) − λj Pij (t),
j i +1
(6.14)
384
6
Continuous-Time Markov Chains
Proposition 6.4 For a pure birth process, Pii (t) = e−λi t , Pij (t) = λj −1 e−λj t
i0
t 0
j i +1
eλj s Pi,j −1 (s) ds,
Proof The fact that Pii (t) = e−λi t follows from Equation (6.14) by integrating and using the fact that Pii (0) = 1. To prove the corresponding result for Pij (t), we note by Equation (6.14) that " # eλj t Pij (t) + λj Pij (t) = eλj t λj −1 Pi,j −1 (t)
or # d " λj t e Pij (t) = λj −1 eλj t Pi,j −1 (t) dt Hence, since Pij (0) = 0, we obtain the desired results.
Example 6.12 (Forward Equations for Birth and Death Process) The forward equations (Equation 6.13) for the general birth and death process become (t) = Pi0
qk0 Pik (t) − λ0 Pi0 (t)
k=0
= μ1 Pi1 (t) − λ0 Pi0 (t) Pij (t) =
(6.15)
qkj Pik (t) − (λj + μj )Pij (t)
k=j
= λj −1 Pi,j −1 (t) + μj +1 Pi,j +1 (t) − (λj + μj )Pij (t)
6.5.
(6.16)
Limiting Probabilities
In analogy with a basic result in discrete-time Markov chains, the probability that a continuous-time Markov chain will be in state j at time t often converges to a limiting value which is independent of the initial state. That is, if we call this value Pj , then Pj ≡ lim Pij (t) t→∞
where we are assuming that the limit exists and is independent of the initial state i.
6.5.
Limiting Probabilities
385
To derive a set of equations for the Pj , consider first the set of forward equations Pij (t) = qkj Pik (t) − vj Pij (t) (6.17) k=j
Now, if we let t approach ∞, then assuming that we can interchange limit and summation, we obtain / . qkj Pik (t) − vj Pij (t) lim Pij (t) = lim t→∞
t→∞
=
k=j
qkj Pk − vj Pj
k=j
However, as Pij (t) is a bounded function (being a probability it is always between 0 and 1), it follows that if Pij (t) converges, then it must converge to 0 (why is this?). Hence, we must have that qkj Pk − vj Pj 0= k=j
or vj Pj =
qkj Pk ,
all states j
(6.18)
k=j
The preceding set of equations, along with this equation Pj = 1
(6.19)
j
can be used to solve for the limiting probabilities.
Remarks (i) We have assumed that the limiting probabilities Pj exist. A sufficient condition for this is that (a) all states of the Markov chain communicate in the sense that starting in state i there is a positive probability of ever being in state j , for all i, j and (b) the Markov chain is positive recurrent in the sense that, starting in any state, the mean time to return to that state is finite If conditions (a) and (b) hold, then the limiting probabilities will exist and satisfy Equations (6.18) and (6.19). In addition, Pj also will have the interpretation of being the long-run proportion of time that the process is in state j . (ii) Equations (6.18) and (6.19) have a nice interpretation: In any interval (0, t) the number of transitions into state j must equal to within 1 the number of tran-
386
6
Continuous-Time Markov Chains
sitions out of state j (why?). Hence, in the long run, the rate at which transitions into state j occur must equal the rate at which transitions out of state j occur. Now when the process is in state j , it leaves at rate vj , and, as Pj is the proportion of time it is in state j , it thus follows that vj Pj = rate at which the process leaves state j Similarly, when the process is in state k, it enters j at a rate qkj . Hence, as Pk is the proportion of time in state k, we see that the rate at which transitions from k to j occur is just qkj Pk ; thus
qkj Pk = rate at which the process enters state j
k=j
So, Equation (6.18) is just a statement of the equality of the rates at which the process enters and leaves state j . Because it balances (that is, equates) these rates, the equations (6.18) are sometimes referred to as “balance equations.” (iii) When the limiting probabilities Pj exist, we say that the chain is ergodic. The Pj are sometimes called stationary probabilities since it can be shown that (as in the discrete-time case) if the initial state is chosen according to the distribution {Pj }, then the probability of being in state j at time t is Pj , for all t. Let us now determine the limiting probabilities for a birth and death process. From Equation (6.18) or equivalently, by equating the rate at which the process leaves a state with the rate at which it enters that state, we obtain State 0 1 2 n, n 1
Rate at which leave = rate at which enter λ0 P0 = μ1 P1 (λ1 + μ1 )P1 = μ2 P2 + λ0 P0 (λ2 + μ2 )P2 = μ3 P3 + λ1 P1 (λn + μn )Pn = μn+1 Pn+1 + λn−1 Pn−1
By adding to each equation the equation preceding it, we obtain λ0 P0 = μ1 P1 , λ1 P1 = μ2 P2 , λ2 P2 = μ3 P3 , .. . λn Pn = μn+1 Pn+1 ,
n0
6.5.
Limiting Probabilities
387
Solving in terms of P0 yields P1 =
λ0 P0 , μ1
P2 =
λ1 λ1 λ0 P1 = P0 , μ2 μ2 μ1
P3 =
λ2 λ2 λ1 λ0 P2 = P0 , μ3 μ3 μ2 μ1
.. . Pn = And by using the fact that
λn−1 λn−1 λn−2 · · · λ1 λ0 Pn−1 = P0 μn μn μn−1 · · · μ2 μ1 ∞
n=0 Pn
= 1, we obtain
1 = P0 + P0
∞ λn−1 · · · λ1 λ0 n=1
μn · · · μ2 μ1
or P0 = 1+
1 ∞ λ0 λ1 · · · λn−1 n=1
μ1 μ2 · · · μn
and so Pn =
λ0 λ1 · · · λn−1 , ∞ λ0 λ1 · · · λn−1 μ1 μ2 · · · μn 1 + μ1 μ2 · · · μn
n1
(6.20)
n=1
The foregoing equations also show us what condition is necessary for these limiting probabilities to exist. Namely, it is necessary that ∞ λ0 λ1 · · · λn−1 n=1
μ1 μ2 · · · μn
0 is called a pure death process. Find Pij (t). 10. Consider two machines. Machine i operates for an exponential time with rate λi and then fails; its repair time is exponential with rate μi , i = 1, 2. The machines act independently of each other. Define a four-state continuous-time Markov chain which jointly describes the condition of the two machines. Use the assumed independence to compute the transition probabilities for this chain and then verify that these transition probabilities satisfy the forward and backward equations. *11. Consider a Yule process starting with a single individual—that is, suppose X(0) = 1. Let Ti denote the time it takes the process to go from a population of size i to one of size i + 1. (a) Argue that Ti , i = 1, . . . , j , are independent exponentials with respective rates iλ. (b) Let X1 , . . . , Xj denote independent exponential random variables each having rate λ, and interpret Xi as the lifetime of component i. Argue that
Exercises
409
max(X1 , . . . , Xj ) can be expressed as max(X1 , . . . , Xj ) = ε1 + ε2 + · · · + εj where ε1 , ε2 , . . . , εj are independent exponentials with respective rates j λ, (j − 1)λ, . . . , λ. Hint: Interpret εi as the time between the i − 1 and the ith failure. (c) Using (a) and (b) argue that P {T1 + · · · + Tj t} = (1 − e−λt )j (d) Use (c) to obtain that P1j (t) = (1 − e−λt )j −1 − (1 − e−λt )j = e−λt (1 − e−λt )j −1 and hence, given X(0) = 1, X(t) has a geometric distribution with parameter p = e−λt . (e) Now conclude that j − 1 −λti e (1 − e−λt )j −i Pij (t) = i −1 12. Each individual in a biological population is assumed to give birth at an exponential rate λ, and to die at an exponential rate μ. In addition, there is an exponential rate of increase θ due to immigration. However, immigration is not allowed when the population size is N or larger. (a) Set this up as a birth and death model. (b) If N = 3, 1 = θ = λ, μ = 2, determine the proportion of time that immigration is restricted. 13. A small barbershop, operated by a single barber, has room for at most two customers. Potential customers arrive at a Poisson rate of three per hour, and the successive service times are independent exponential random variables with mean 1 4 hour. What is (a) the average number of customers in the shop? (b) the proportion of potential customers that enter the shop? (c) If the barber could work twice as fast, how much more business would he do? 14. Potential customers arrive at a full-service, one-pump gas station at a Poisson rate of 20 cars per hour. However, customers will only enter the station for gas if there are no more than two cars (including the one currently being attended to) at the pump. Suppose the amount of time required to service a car is exponentially distributed with a mean of five minutes.
410
6
Continuous-Time Markov Chains
(a) What fraction of the attendant’s time will be spent servicing cars? (b) What fraction of potential customers are lost? 15. A service center consists of two servers, each working at an exponential rate of two services per hour. If customers arrive at a Poisson rate of three per hour, then, assuming a system capacity of at most three customers, (a) what fraction of potential customers enter the system? (b) what would the value of part (a) be if there was only a single server, and his rate was twice as fast (that is, μ = 4)? *16. The following problem arises in molecular biology. The surface of a bacterium consists of several sites at which foreign molecules—some acceptable and some not—become attached. We consider a particular site and assume that molecules arrive at the site according to a Poisson process with parameter λ. Among these molecules a proportion α is acceptable. Unacceptable molecules stay at the site for a length of time which is exponentially distributed with parameter μ1 , whereas an acceptable molecule remains at the site for an exponential time with rate μ2 . An arriving molecule will become attached only if the site is free of other molecules. What percentage of time is the site occupied with an acceptable (unacceptable) molecule? 17. Each time a machine is repaired it remains up for an exponentially distributed time with rate λ. It then fails, and its failure is either of two types. If it is a type 1 failure, then the time to repair the machine is exponential with rate μ1 ; if it is a type 2 failure, then the repair time is exponential with rate μ2 . Each failure is, independently of the time it took the machine to fail, a type 1 failure with probability p and a type 2 failure with probability 1 − p. What proportion of time is the machine down due to a type 1 failure? What proportion of time is it down due to a type 2 failure? What proportion of time is it up? 18. After being repaired, a machine functions for an exponential time with rate λ and then fails. Upon failure, a repair process begins. The repair process proceeds sequentially through k distinct phases. First a phase 1 repair must be performed, then a phase 2, and so on. The times to complete these phases are independent, with phase i taking an exponential time with rate μi , i = 1, . . . , k. (a) What proportion of time is the machine undergoing a phase i repair? (b) What proportion of time is the machine working? *19. A single repairperson looks after both machines 1 and 2. Each time it is repaired, machine i stays up for an exponential time with rate λi , i = 1, 2. When machine i fails, it requires an exponentially distributed amount of work with rate μi to complete its repair. The repairperson will always service machine 1 when it is down. For instance, if machine 1 fails while 2 is being repaired, then the repairperson will immediately stop work on machine 2 and start on 1. What proportion of time is machine 2 down?
Exercises
411
20. There are two machines, one of which is used as a spare. A working machine will function for an exponential time with rate λ and will then fail. Upon failure, it is immediately replaced by the other machine if that one is in working order, and it goes to the repair facility. The repair facility consists of a single person who takes an exponential time with rate μ to repair a failed machine. At the repair facility, the newly failed machine enters service if the repairperson is free. If the repairperson is busy, it waits until the other machine is fixed; at that time, the newly repaired machine is put in service and repair begins on the other one. Starting with both machines in working condition, find (a) the expected value and (b) the variance of the time until both are in the repair facility. (c) In the long run, what proportion of time is there a working machine? 21. Suppose that when both machines are down in Exercise 20 a second repairperson is called in to work on the newly failed one. Suppose all repair times remain exponential with rate μ. Now find the proportion of time at least one machine is working, and compare your answer with the one obtained in Exercise 20. 22. Customers arrive at a single-server queue in accordance with a Poisson process having rate λ. However, an arrival that finds n customers already in the system will only join the system with probability 1/(n + 1). That is, with probability n/(n + 1) such an arrival will not join the system. Show that the limiting distribution of the number of customers in the system is Poisson with mean λ/μ. 23. A job shop consists of three machines and two repairmen. The amount of time a machine works before breaking down is exponentially distributed with mean 10. If the amount of time it takes a single repairman to fix a machine is exponentially distributed with mean 8, then (a) what is the average number of machines not in use? (b) what proportion of time are both repairmen busy? *24. Consider a taxi station where taxis and customers arrive in accordance with Poisson processes with respective rates of one and two per minute. A taxi will wait no matter how many other taxis are present. However, an arriving customer that does not find a taxi waiting leaves. Find (a) the average number of taxis waiting, and (b) the proportion of arriving customers that get taxis. 25. Customers arrive at a service station, manned by a single server who serves at an exponential rate μ1 , at a Poisson rate λ. After completion of service the customer then joins a second system where the server serves at an exponential
412
6
Continuous-Time Markov Chains
rate μ2 . Such a system is called a tandem or sequential queueing system. Assuming that λ < μi , i = 1, 2, determine the limiting probabilities. Hint: Try a solution of the form Pn,m = Cα n β m , and determine C, α, β. 26. Consider an ergodic M/M/s queue in steady state (that is, after a long time) and argue that the number presently in the system is independent of the sequence of past departure times. That is, for instance, knowing that there have been departures 2, 3, 5, and 10 time units ago does not affect the distribution of the number presently in the system. 27. In the M/M/s queue if you allow the service rate to depend on the number in the system (but in such a way so that it is ergodic), what can you say about the output process? What can you say when the service rate μ remains unchanged but λ > sμ? *28. If {X(t)} and {Y (t)} are independent continuous-time Markov chains, both of which are time reversible, show that the process {X(t), Y (t)} is also a time reversible Markov chain. 29. Consider a set of n machines and a single repair facility to service these machines. Suppose that when machine i, i = 1, . . . , n, fails it requires an exponentially distributed amount of work with rate μi to repair it. The repair facility divides its efforts equally among all failed machines in the sense that whenever there are k failed machines each one receives work at a rate of 1/k per unit time. If there are a total of r working machines, including machine i, then i fails at an instantaneous rate λi /r. (a) Define an appropriate state space so as to be able to analyze the preceding system as a continuous-time Markov chain. (b) Give the instantaneous transition rates (that is, give the qij ). (c) Write the time reversibility equations. (d) Find the limiting probabilities and show that the process is time reversible. 30. Consider a graph with nodes 1, 2, . . . , n and the n2 arcs (i, j ), i = j , i, j, = 1, . . . , n. (See Section 3.6.2 for appropriate definitions.) Suppose that a particle moves along this graph as follows: Events occur along the arcs (i, j ) according to independent Poisson processes with rates λij . An event along arc (i, j ) causes that arc to become excited. If the particle is at node i at the moment that (i, j ) becomes excited, it instantaneously moves to node j , i, j = 1, . . . , n. Let Pj denote the proportion of time that the particle is at node j . Show that Pj = Hint: Use time reversibility.
1 n
Exercises
413
31. A total of N customers move about among r servers in the following manner. When a customer is served by server i, he then goes over to server j , j = i, with probability 1/(r − 1). If the server he goes to is free, then the customer enters service; otherwise he joins the queue. The service times are all independent, with the service times at server i being exponential with rate μ, i = 1, . . . , r. Let the . . . , nr ), where ni is the number of customers state at any time be the vector (n1 , presently at server i, i = 1, . . . , r, i ni = N . (a) Argue that if X(t) is the state at time t, then {X(t), t 0}, is a continuoustime Markov chain. (b) Give the infinitesimal rates of this chain. (c) Show that this chain is time reversible, and find the limiting probabilities. 32. Customers arrive at a two-server station in accordance with a Poisson process having rate λ. Upon arriving, they join a single queue. Whenever a server completes a service, the person first in line enters service. The service times of server i are exponential with rate μi , i = 1, 2, where μ1 + μ2 > λ. An arrival finding both servers free is equally likely to go to either one. Define an appropriate continuous-time Markov chain for this model, show it is time reversible, and find the limiting probabilities. *33. Consider two M/M/1 queues with respective parameters λi , μi , i = 1, 2. Suppose they share a common waiting room that can hold at most three customers. That is, whenever an arrival finds her server busy and three customers in the waiting room, she goes away. Find the limiting probability that there will be n queue 1 customers and m queue 2 customers in the system. Hint: Use the results of Exercise 28 together with the concept of truncation. 34. Four workers share an office that contains four telephones. At any time, each worker is either “working” or “on the phone.” Each “working” period of worker i lasts for an exponentially distributed time with rate λi , and each “on the phone” period lasts for an exponentially distributed time with rate μi , i = 1, 2, 3, 4. (a) What proportion of time are all workers “working”? Let Xi (t) equal 1 if worker i is working at time t, and let it be 0 otherwise. Let X(t) = (X1 (t), X2 (t), X3 (t), X4 (t)). (b) Argue that {X(t), t 0} is a continuous-time Markov chain and give its infinitesimal rates. (c) Is {X(t)} time reversible? Why or why not? Suppose now that one of the phones has broken down. Suppose that a worker who is about to use a phone but finds them all being used begins a new “working” period. (d) What proportion of time are all workers “working”?
414
6
Continuous-Time Markov Chains
35. Consider a time reversible continuous-time Markov chain having infinitesimal transition rates qij and limiting probabilities {Pi }. Let A denote a set of states for this chain, and consider a new continuous-time Markov chain with transition rates qij∗ given by qij∗ =
cqij , qij ,
if i ∈ A, j ∈ /A otherwise
where c is an arbitrary positive number. Show that this chain remains time reversible, and find its limiting probabilities. 36. Consider a system of n components such that the working times of component i, i = 1, . . . , n, are exponentially distributed with rate λi . When failed, however, the repair rate of component i depends on how many other components are down. Specifically, suppose that the instantaneous repair rate of component i, i = 1, . . . , n, when there are a total of k failed components, is α k μi . (a) Explain how we can analyze the preceding as a continuous-time Markov chain. Define the states and give the parameters of the chain. (b) Show that, in steady state, the chain is time reversible and compute the limiting probabilities. 37. For the continuous-time Markov chain of Exercise 3 present a uniformized version. 38. In Example 6.20, we computed m(t) = E[O(t)], the expected occupation time in state 0 by time t for the two-state continuous-time Markov chain starting in state 0. Another way of obtaining this quantity is by deriving a differential equation for it. (a) Show that m(t + h) = m(t) + P00 (t)h + o(h) (b) Show that m (t) =
λ μ + e−(λ+μ)t λ+μ λ+μ
(c) Solve for m(t). 39. Let O(t) be the occupation time for state 0 in the two-state continuous-time Markov chain. Find E[O(t)|X(0) = 1]. 40. Consider the two-state continuous-time Markov chain. Starting in state 0, find Cov[X(s), X(t)].
References
415
41. Let Y denote an exponential random variable with rate λ that is independent of the continuous-time Markov chain {X(t)} and let P¯ij = P {X(Y ) = j |X(0) = i} (a) Show that P¯ij =
1 λ δij qik P¯kj + vi + λ vi + λ k
where δij is 1 when i = j and 0 when i = j . (b) Show that the solution of the preceding set of equations is given by P¯ = (I − R/λ)−1 where P¯ is the matrix of elements P¯ij , I is the identity matrix, and R the matrix specified in Section 6.8. (c) Suppose now that Y1 , . . . , Yn are independent exponentials with rate λ that are independent of {X(t)}. Show that P {X(Y1 + · · · + Yn ) = j |X(0) = i} is equal to the element in row i, column j of the matrix P¯ n . (d) Explain the relationship of the preceding to Approximation 2 of Section 6.8. *42. (a) Show that Approximation 1 of Section 6.8 is equivalent to uniformizing the continuous-time Markov chain with a value v such that vt = n and then approximating Pij (t) by Pij∗n . (b) Explain why the preceding should make a good approximation. Hint: What is the standard deviation of a Poisson random variable with mean n?
References 1. D. R. Cox and H. D. Miller, “The Theory of Stochastic Processes,” Methuen, London, 1965. 2. A. W. Drake, “Fundamentals of Applied Probability Theory,” McGraw-Hill, New York, 1967. 3. S. Karlin and H. Taylor, “A First Course in Stochastic Processes,” Second Edition, Academic Press, New York, 1975. 4. E. Parzen, “Stochastic Processes,” Holden-Day, San Francisco, California, 1962. 5. S. Ross, “Stochastic Processes,” Second Edition, John Wiley, New York, 1996.
This page intentionally left blank
Renewal Theory and Its Applications
7 7.1.
Introduction
We have seen that a Poisson process is a counting process for which the times between successive events are independent and identically distributed exponential random variables. One possible generalization is to consider a counting process for which the times between successive events are independent and identically distributed with an arbitrary distribution. Such a counting process is called a renewal process. Let {N(t), t 0} be a counting process and let Xn denote the time between the (n − 1)st and the nth event of this process, n 1.
Definition 7.1 If the sequence of nonnegative random variables {X1 X2 , . . .}
is independent and identically distributed, then the counting process {N (t), t 0} is said to be a renewal process.
Thus, a renewal process is a counting process such that the time until the first event occurs has some distribution F , the time between the first and second event has, independently of the time of the first event, the same distribution F , and so on. When an event occurs, we say that a renewal has taken place. For an example of a renewal process, suppose that we have an infinite supply of lightbulbs whose lifetimes are independent and identically distributed. Suppose also that we use a single lightbulb at a time, and when it fails we immediately replace it with a new one. Under these conditions, {N (t), t 0} is a renewal process when N (t) represents the number of lightbulbs that have failed by time t. For a renewal process having interarrival times X1 , X2 , . . . , let S0 = 0,
Sn =
n i=1
417
Xi ,
n1
418
7
Renewal Theory and Its Applications
Figure 7.1. Renewal and interarrival times.
That is, S1 = X1 is the time of the first renewal; S2 = X1 + X2 is the time until the first renewal plus the time between the first and second renewal, that is, S2 is the time of the second renewal. In general, Sn denotes the time of the nth renewal (see Figure 7.1). We shall let F denote the interarrival distribution and to avoid trivialities, we assume that F (0) = P {Xn = 0} < 1. Furthermore, we let μ = E[Xn ],
n1
be the mean time between successive renewals. It follows from the nonnegativity of Xn and the fact that Xn is not identically 0 that μ > 0. The first question we shall attempt to answer is whether an infinite number of renewals can occur in a finite amount of time. That is, can N (t) be infinite for some (finite) value of t? To show that this cannot occur, we first note that, as Sn is the time of the nth renewal, N (t) may be written as N (t) = max{n : Sn t}
(7.1)
To understand why Equation (7.1) is valid, suppose, for instance, that S4 t but S5 > t. Hence, the fourth renewal had occurred by time t but the fifth renewal occurred after time t; or in other words, N (t), the number of renewals that occurred by time t, must equal 4. Now by the strong law of large numbers it follows that, with probability 1, Sn →μ n
as n → ∞
But since μ > 0 this means that Sn must be going to infinity as n goes to infinity. Thus, Sn can be less than or equal to t for at most a finite number of values of n, and hence by Equation (7.1), N (t) must be finite. However, though N (t) < ∞ for each t, it is true that, with probability 1, N (∞) ≡ lim N (t) = ∞ t→∞
This follows since the only way in which N (∞), the total number of renewals that occur, can be finite is for one of the interarrival times to be infinite.
7.2.
Distribution of N(t)
419
Therefore, P {N (∞) < ∞} = P {Xn = ∞ for some n} ∞ {Xn = ∞} =P n=1
∞
P {Xn = ∞}
n=1
=0
7.2.
Distribution of N(t)
The distribution of N (t) can be obtained, at least in theory, by first noting the important relationship that the number of renewals by time t is greater than or equal to n if and only if the nth renewal occurs before or at time t. That is, N (t) n ⇔ Sn t
(7.2)
From Equation (7.2) we obtain P {N (t) = n} = P {N (t) n} − P {N (t) n + 1} = P {Sn t} − P {Sn+1 t}
(7.3)
, i 1, are independent and have a common Now since the random variables Xi distribution F , it follows that Sn = ni=1 Xi is distributed as Fn , the n-fold convolution of F with itself (Section 2.5). Therefore, from Equation (7.3) we obtain P {N (t) = n} = Fn (t) − Fn+1 (t)
Example 7.1 Suppose that P {Xn = i} = p(1 − p)i−1 , i 1. That is, suppose
that the interarrival distribution is geometric. Now S1 = X1 may be interpreted as the number of trials necessary to get a single success when each trial is independent and has a probability p of being a success. Similarly, Sn may be interpreted as the number of trials necessary to attain n successes, and hence has the negative binomial distribution ⎧ ⎨ k − 1 p n (1 − p)k−n , kn P {Sn = k} = n−1 ⎩ 0, k t
we obtain from Equation (7.4) that t m(t) = [1 + m(t − x)]f (x) dx 0
t
= F (t) +
m(t − x)f (x) dx
(7.5)
0
Equation (7.5) is called the renewal equation and can sometimes be solved to obtain the renewal function.
Example 7.3 One instance in which the renewal equation can be solved is when the interarrival distribution is uniform—say, uniform on (0, 1). We will now present a solution in this case when t 1. For such values of t, the renewal function becomes t m(t) = t + m(t − x) dx
0 t
=t+
m(y) dy
by the substitution y = t − x
0
Differentiating the preceding equation yields m (t) = 1 + m(t) Letting h(t) = 1 + m(t), we obtain h (t) = h(t)
7.3.
Limit Theorems and Their Applications
423
or log h(t) = t + C or h(t) = Ket or m(t) = Ket − 1 Since m(0) = 0, we see that K = 1, and so we obtain m(t) = et − 1,
7.3.
0t 1
Limit Theorems and Their Applications
We have shown previously that, with probability 1, N (t) goes to infinity as t goes to infinity. However, it would be nice to know the rate at which N (t) goes to infinity. That is, we would like to be able to say something about limt→∞ N (t)/t. As a prelude to determining the rate at which N (t) grows, let us first consider the random variable SN (t) . In words, just what does this random variable represent? Proceeding inductively suppose, for instance, that N (t)= 3. Then SN (t) = S3 represents the time of the third event. Since there are only three events that have occurred by time t, S3 also represents the time of the last event prior to (or at) time t. This is, in fact, what SN (t) represents—namely, the time of the last renewal prior to or at time t. Similar reasoning leads to the conclusion that SN (t)+1 represents the time of the first renewal after time t (see Figure 7.2). We now are ready to prove the following.
Proposition 7.1 With probability 1, 1 N (t) → t μ
as t → ∞
Figure 7.2.
424
7
Renewal Theory and Its Applications
Proof Since SN (t) is the time of the last renewal prior to or at time t, and SN (t)+1 is the time of the first renewal after time t, we have SN (t) t < SN (t)+1 or SN (t) t SN (t)+1 < (7.6) N (t) N (t) N (t) (t) However, since SN (t) /N(t) = N i=1 Xi /N (t) is the average of N (t) independent and identically distributed random variables, it follows by the strong law of large numbers that SN (t) /N(t) → μ as N (t) → ∞. But since N (t) → ∞ when t → ∞, we obtain SN (t) →μ N (t)
as t → ∞
Furthermore, writing SN (t)+1 = N (t)
SN (t)+1 N (t) + 1
N (t) + 1 N (t)
we have that SN (t)+1 /(N(t) + 1) → μ by the same reasoning as before and N (t) + 1 →1 N (t)
as t → ∞
SN (t)+1 →μ N (t)
as t → ∞
Hence,
The result now follows by Equation (7.6) since t/N (t) is between two random variables, each of which converges to μ as t → ∞.
Remarks (i) The preceding propositions are true even when μ, the mean time between renewals, is infinite. In this case, we interpret 1/μ to be 0. (ii) The number 1/μ is called the rate of the renewal process. (iii) Because the average time between renewals is μ, it is quite intuitive that the average rate at which renewals occur is 1 per every μ time units. Proposition 7.1 says that the average renewal rate up to time t will, with probability 1, converge to 1/μ as t → ∞. What about the expected average renewal rate? Is it true that m(t)/t also converges to 1/μ? This result, known as the elementary renewal theorem, will be stated without proof.
7.3.
Limit Theorems and Their Applications
425
Elementary Renewal Theorem m(t) 1 → t μ
as t → ∞
As before, 1/μ is interpreted as 0 when μ = ∞.
Remark At first glance it might seem that the elementary renewal theorem should be a simple consequence of Proposition 7.1. That is, since the average renewal rate will, with probability 1, converge to 1/μ, should this not imply that the expected average renewal rate also converges to 1/μ? We must, however, be careful; consider the next example.
Example 7.4 Let U be a random variable which is uniformly distributed on (0, 1); and define the random variables Yn , n 1, by 0, if U > 1/n Yn = n, if U 1/n
Now, since, with probability 1, U will be greater than 0, it follows that Yn will equal 0 for all sufficiently large n. That is, Yn will equal 0 for all n large enough so that 1/n < U . Hence, with probability 1, Yn → 0 However,
as n → ∞
1 1 =n =1 E[Yn ] = nP U n n
Therefore, even though the sequence of random variables Yn converges to 0, the expected values of the Yn are all identically 1.
Example 7.5 Beverly has a radio that works on a single battery. As soon as the battery in use fails, Beverly immediately replaces it with a new battery. If the lifetime of a battery (in hours) is distributed uniformly over the interval (30, 60), then at what rate does Beverly have to change batteries? Solution: If we let N (t) denote the number of batteries that have failed by time t, we have by Proposition 7.1 that the rate at which Beverly replaces batteries is given by lim
t→∞
1 1 N (t) = = t μ 45
That is, in the long run, Beverly will have to replace one battery every 45 hours.
426
7
Renewal Theory and Its Applications
Example 7.6 Suppose in Example 7.5 that Beverly does not keep any surplus batteries on hand, and so each time a failure occurs she must go and buy a new battery. If the amount of time it takes for her to get a new battery is uniformly distributed over (0,1), then what is the average rate that Beverly changes batteries? Solution:
In this case the mean time between renewals is given by μ = EU1 + EU2
where U1 is uniform over (30, 60) and U2 is uniform over (0, 1). Hence, μ = 45 +
1 2
= 45 12
and so in the long run, Beverly will be putting in a new battery at the rate of That is, she will put in two new batteries every 91 hours.
2 91 .
Example 7.7 Suppose that potential customers arrive at a single-server bank in accordance with a Poisson process having rate λ. However, suppose that the potential customer will enter the bank only if the server is free when he arrives. That is, if there is already a customer in the bank, then our arriver, rather than entering the bank, will go home. If we assume that the amount of time spent in the bank by an entering customer is a random variable having distribution G, then (a) what is the rate at which customers enter the bank? (b) what proportion of potential customers actually enter the bank? Solution: In answering these questions, let us suppose that at time 0 a customer has just entered the bank. (That is, we define the process to start when the first customer enters the bank.) If we let μG denote the mean service time, then, by the memoryless property of the Poisson process, it follows that the mean time between entering customers is μ = μG +
1 λ
Hence, the rate at which customers enter the bank will be given by 1 λ = μ 1 + λμG On the other hand, since potential customers will be arriving at a rate λ, it follows that the proportion of them entering the bank will be given by λ/(1 + λμG ) 1 = λ 1 + λμG In particular if λ = 2 (in hours) and μG = 2, then only one customer out of five will actually enter the system.
7.3.
Limit Theorems and Their Applications
427
A somewhat unusual application of Proposition 7.1 is provided by our next example.
Example 7.8 A sequence of independent trials, each of which results in out-
come number i with probability Pi , i = 1, . . . , n, ni Pi = 1, is observed until the same outcome occurs k times in a row; this outcome then is declared to be the winner of the game. For instance, if k = 2 and the sequence of outcomes is 1, 2, 4, 3, 5, 2, 1, 3, 3, then we stop after nine trials and declare outcome number 3 the winner. What is the probability that i wins, i = 1, . . . , n, and what is the expected number of trials? Solution: We begin by computing the expected number of coin tosses, call it E[T ], until a run of k successive heads occurs when the tosses are independent and each lands on heads with probability p. By conditioning on the time of the first nonhead, we obtain E[T ] =
k (1 − p)p j −1 (j + E[T ]) + kp k j =1
Solving this for E[T ] yields (1 − p) j −1 E[T ] = k + jp pk k
j =1
Upon simplifying, we obtain E[T ] = =
1 + p + · · · + p k−1 pk 1 − pk p k (1 − p)
(7.7)
Now, let us return to our example, and let us suppose that as soon as the winner of a game has been determined we immediately begin playing another game. For each i let us determine the rate at which outcome i wins. Now, every time i wins, everything starts over again and thus wins by i constitute renewals. Hence, from Proposition 7.1, the rate at which i wins =
1 E[Ni ]
428
7
Renewal Theory and Its Applications
where Ni denotes the number of trials played between successive wins of outcome i. Hence, from Equation (7.7) we see that rate at which i wins =
Pik (1 − Pi )
(7.8)
1 − Pik
Hence, the long-run proportion of games that are won by number i is given by rate at which i wins proportion of games i wins = n j =1 rate at which j wins = n
Pik (1 − Pi )/(1 − Pik )
k k j =1 (Pj (1 − Pj )/(1 − Pj ))
However, it follows from the strong law of large numbers that the long-run proportion of games that i wins will, with probability 1, be equal to the probability that i wins any given game. Hence, P {i wins} = n
Pik (1 − Pi )/(1 − Pik )
k k j =1 (Pj (1 − Pj )/(1 − Pj ))
To compute the expected time of a game, we first note that the rate at which games end =
n
rate at which i wins
i=1
=
n P k (1 − Pi ) i
1 − Pik
i=1
[from Equation (7.8)]
Now, as everything starts over when a game ends, it follows by Proposition 7.1 that the rate at which games end is equal to the reciprocal of the mean time of a game. Hence, E[time of a game} =
1 rate at which games end
= n
1
k k i=1 (Pi (1 − Pi )/(1 − Pi ))
A key element in the proof of the elementary renewal theorem, which is also of independent interest, is the establishment of a relationship between m(t), the mean number of renewals by time t, and E[SN (t)+1 ], the expected time of
7.3.
Limit Theorems and Their Applications
429
the first renewal after t. Letting g(t) = E[SN (t)+1 ] we will derive an integral equation, similar to the renewal equation, for g(t) by conditioning on the time of the first renewal. This yields ∞ E[SN (t)+1 |X1 = x]f (x) dx g(t) = 0
where we have supposed that the interarrival times are continuous with density f . Now if the first renewal occurs at time x and x > t, then clearly the time of the first renewal after t is x. On the other hand, if the first renewal occurs at a time x < t, then by regarding x as the new origin, it follows that the expected time, from this origin, of the first renewal occurring after a time t − x from this origin is g(t − x). That is, we see that g(t − x) + x, if x < t E[SN (t)+1 |X1 = x] = x, if x > t Substituting this into the preceding equation gives t (g(t − x) + x) f (x) dx + g(t) =
0 t
=
xf (x) dx t
∞
g(t − x) f (x) dx +
0
∞
xf (x) dx 0
or
t
g(t) = μ +
g(t − x) f (x) dx
0
which is quite similar to the renewal equation t m(t − x)f (x) ds m(t) = F (t) + 0
Indeed, if we let g1 (t) = we see that
g1 (t) + 1 = 1 + 0
t
g(t) −1 μ
[g1 (t − x) + 1]f (x) dx
430
7
Renewal Theory and Its Applications
or
g1 (t) = F (t) +
t
g1 (t − x)f (x) dx
0
That is, g1 (t) = E[SN (t)+1 ]/μ − 1 satisfies the renewal equation and thus, by uniqueness, must be equal to m(t). We have thus proven the following.
Proposition 7.2 E[SN (t)+1 ] = μ[m(t) + 1] A second derivation of Proposition 7.2 is given in Exercises 13 and 14. To see how Proposition 7.2 can be used to establish the elementary renewal theorem, let Y (t) denote the time from t until the next renewal. Y (t) is called the excess, or residual life, at t. As the first renewal after t will occur at time t + Y (t), we see that SN (t)+1 = t + Y (t) Taking expectations and utilizing Proposition 7.2 yields μ[m(t) + 1] = t + E[Y (t)]
(7.9)
which implies that m(t) 1 1 E[Y (t)] = − + t μ t tμ The elementary renewal theorem can now be proven by showing that lim
t→∞
E[Y (t)] =0 t
(see Exercise 14). The relation (7.9) shows that if we can determine E[Y (t)], the mean excess at t, then we can compute m(t) and vice versa.
Example 7.9 Consider the renewal process whose interarrival distribution is the convolution of two exponentials; that is, F = F1 ∗ F2 ,
where Fi (t) = 1 − e−μi t ,
i = 1, 2
We will determine the renewal function by first determining E[Y (t)]. To obtain the mean excess at t, imagine that each renewal corresponds to a new machine being put in use, and suppose that each machine has two components—initially component 1 is employed and this lasts an exponential time with rate μ1 , and then
7.3.
Limit Theorems and Their Applications
431
component 2, which functions for an exponential time with rate μ2 , is employed. When component 2 fails, a new machine is put in use (that is, a renewal occurs). Now consider the process {X(t), t 0} where X(t) is i if a type i component is in use at time t. It is easy to see that {X(t), t 0} is a two-state continuous-time Markov chain, and so, using the results of Example 6.11, its transition probabilities are μ1 μ2 e−(μ1 +μ2 )t + P11 (t) = μ1 + μ2 μ1 + μ2 To compute the expected remaining life of the machine in use at time t, we condition on whether it is using its first or second component: for if it is still using its first component, then its remaining life is 1/μ1 + 1/μ2 , whereas if it is already using its second component, then its remaining life is 1/μ2 . Hence, letting p(t) denote the probability that the machine in use at time t is using its first component, we have that 1 − p(t) 1 1 p(t) + + E[Y (t)] = μ1 μ2 μ2 =
1 p(t) + μ2 μ1
But, since at time 0 the first machine is utilizing its first component, it follows that p(t) = P11 (t), and so, upon using the preceding expression of P11 (t), we obtain E[Y (t)] =
1 1 μ2 + e−(μ1 +μ2 )t + μ2 μ1 + μ2 μ1 (μ1 + μ2 )
(7.10)
Now it follows from Equation (7.9) that m(t) + 1 =
t E[Y (t)] + u μ
(7.11)
where μ, the mean interarrival time, is given in this case by μ=
1 1 μ1 + μ2 + = μ1 μ2 μ1 μ2
Substituting Equation (7.10) and the preceding equation into (7.11) yields, after simplifying, m(t) =
μ1 μ2 μ1 μ2 t− [1 − e−(μ1 +μ2 )t ] μ1 + μ2 (μ1 + μ2 )2
432
7
Renewal Theory and Its Applications
Remark Using the relationship of Equation (7.11) and results from the twostate continuous-time Markov chain, the renewal function can also be obtained in the same manner as in Example 7.9 for the interarrival distributions F (t) = pF1 (t) + (1 − p)F2 (t) and F (t) = pF1 (t) + (1 − p)(F1 ∗ F2 )(t) when Fi (t) = 1 − e−μi t , t > 0, i = 1, 2. An important limit theorem is the central limit theorem for renewal processes. This states that, for large t, N (t) is approximately normally distributed with mean t/μ and variance tσ 2 /μ3 , where μ and σ 2 are, respectively, the mean and variance of the interarrival distribution. That is, we have the following theorem which we state without proof. Central Limit Theorem for Renewal Processes
x N (t) − t/μ 1 2 lim P 2 <x = √ e−x /2 dx t→ ∞ 2 3 2π −∞ tσ /μ In addition, as might be expected from the central limit theorem for renewal processes, it can be shown that Var(N (t))/t converges to σ 2 /μ3 . That is, it can be shown that lim
t→∞
Var(N (t)) = σ 2 /μ3 t
(7.12)
Example 7.10 Two machines continually process an unending number of jobs. The time that it takes to process a job on machine 1 is a gamma random variable with parameters n = 4, λ = 2, whereas the time that it takes to process a job on machine 2 is uniformly distributed between 0 and 4. Approximate the probability that together the two machines can process at least 90 jobs by time t = 100. Solution: If we let Ni (t) denote the number of jobs that machine i can process by time t, then {N1 (t), t 0} and {N2 (t), t 0} are independent renewal processes. The interarrival distribution of the first renewal process is gamma with parameters n = 4, λ = 2, and thus has mean 2 and variance 1. Correspondingly, the interarrival distribution of the second renewal process is uniform between 0 and 4, and thus has mean 2 and variance 16/12.
7.4.
Renewal Reward Processes
433
Therefore, N1 (100) is approximately normal with mean 50 and variance 100/8; and N2 (100) is approximately normal with mean 50 and variance 100/6. Hence, N1 (100) + N2 (100) is approximately normal with mean 100 and variance 175/6. Thus, with denoting the standard normal distribution function, we have N1 (100) + N2 (100) − 100 89.5 − 100 P {N1 (100) + N2 (100) > 89.5} = P > √ √ 175/6 175/6 −10.5 ≈1− √ 175/6 10.5 ≈ √ 175/6 ≈ (1.944) ≈ 0.9741
7.4.
Renewal Reward Processes
A large number of probability models are special cases of the following model. Consider a renewal process {N (t), t 0} having interarrival times Xn , n 1, and suppose that each time a renewal occurs we receive a reward. We denote by Rn , the reward earned at the time of the nth renewal. We shall assume that the Rn , n 1, are independent and identically distributed. However, we do allow for the possibility that Rn may (and usually will) depend on Xn , the length of the nth renewal interval. If we let R(t) =
N (t)
Rn
n=1
then R(t) represents the total reward earned by time t. Let E[R] = E[Rn ],
E[X] = E[Xn ]
Proposition 7.3 If E[R] < ∞ and E[X] < ∞, then (a) with probability 1, (b)
R(t) E[R] = t E[X] E[R(t)] E[R] = lim t→ ∞ t E[X] lim
t→ ∞
434
7
Renewal Theory and Its Applications
Proof We give the proof for (a) only. To prove this, write R(t) = t
N (t) n=1
t
Rn
N (t) =
Rn N (t) n=1
N (t) t
By the strong law of large numbers we obtain N (t)
n=1 Rn → E[R] N (t)
as t → ∞
and by Proposition 7.1 N (t) 1 → t E[X] The result thus follows.
as t → ∞
Remark (i) If we say that a cycle is completed every time a renewal occurs, then Proposition 7.3 states that the long-run average reward per unit time is equal to the expected reward earned during a cycle divided by the expected length of a cycle. (ii) Although we have supposed that the reward is earned at the time of a renewal, the result remains valid when the reward is earned gradually throughout the renewal cycle.
Example 7.11 In Example 7.7 if we suppose that the amounts that the successive customers deposit in the bank are independent random variables having a common distribution H , then the rate at which deposits accumulate—that is, limt→∞ (total deposits by the time t)/t—is given by μH E[deposits during a cycle] = E[time of cycle] μG + 1/λ where μG + 1/λ is the mean time of a cycle, and μH is the mean of the distribution H .
Example 7.12 (A Car Buying Model) The lifetime of a car is a continuous random variable having a distribution H and probability density h. Mr. Brown has a policy that he buys a new car as soon as his old one either breaks down or reaches the age of T years. Suppose that a new car costs C1 dollars and also that an additional cost of C2 dollars is incurred whenever Mr. Brown’s car breaks down. Under the assumption that a used car has no resale value, what is Mr. Brown’s long-run average cost?
7.4.
Renewal Reward Processes
435
If we say that a cycle is complete every time Mr. Brown gets a new car, then it follows from Proposition 7.3 (with costs replacing rewards) that his long-run average cost equals E[cost incurred during a cycle] E[length of a cycle] Now letting X be the lifetime of Mr. Brown’s car during an arbitrary cycle, then the cost incurred during that cycle will be given by if X > T if X T
C1 , C 1 + C2 ,
so the expected cost incurred over a cycle is C1 P {X > T } + (C1 + C2 )P {X T } = C1 + C2 H (T ) Also, the length of the cycle is X, T,
if X T if X > T
and so the expected length of a cycle is 0
T
∞
xh(x) dx + T
T h(x) dx =
T
xh(x) dx + T [1 − H (T )]
0
Therefore, Mr. Brown’s long-run average cost will be
T 0
C1 + C2 H (T ) xh(x) dx + T [1 − H (T )]
(7.13)
Now, suppose that the lifetime of a car (in years) is uniformly distributed over (0, 10), and suppose that C1 is 3 (thousand) dollars and C2 is 12 (thousand) dollars. What value of T minimizes Mr. Brown’s long-run average cost?
436
7
Renewal Theory and Its Applications
If Mr. Brown uses the value T , T 10, then from Equation (7.13) his long-run average cost equals T 0
3 + 12 (T /10) (x/10) dx + T (1 − T /10)
= =
3 + T /20 − T 2 )/10
T 2 /20 + (10T 60 + T 20T − T 2
We can now minimize this by using the calculus. Toward this end, let g(T ) =
60 + T 20T − T 2
then g (T ) =
(20T − T 2 ) − (60 + T )(20 − 2T ) (20T − T 2 )2
Equating to 0 yields 20T − T 2 = (60 + T )(20 − 2T ) or, equivalently, T 2 + 120 T − 1200 = 0 which yields the solutions T ≈ 9.25and T ≈ −129.25 Since T 10, it follows that the optimal policy for Mr. Brown would be to purchase a new car whenever his old car reaches the age of 9.25 years.
Example 7.13 (Dispatching a Train) Suppose that customers arrive at a train depot in accordance with a renewal process having a mean interarrival time μ. Whenever there are N customers waiting in the depot, a train leaves. If the depot incurs a cost at the rate of nc dollars per unit time whenever there are n customers waiting, what is the average cost incurred by the depot? If we say that a cycle is completed whenever a train leaves, then the preceding is a renewal reward process. The expected length of a cycle is the expected time
7.4.
Renewal Reward Processes
437
required for N customers to arrive and, since the mean interarrival time is μ, this equals E[length of cycle] = N μ If we let Tn denote the time between the nth and (n + 1)st arrival in a cycle, then the expected cost of a cycle may be expressed as E[cost of a cycle] = E[c T1 + 2c T2 + · · · + (N − 1) c TN −1 ] which, since E[Tn ] = μ, equals cμ
N (N − 1) 2
Hence, the average cost incurred by the depot is cμ N (N − 1) c(N − 1) = 2N μ 2 Suppose now that each time a train leaves, the depot incurs a cost of six units. What value of N minimizes the depot’s long-run average cost when c = 2, μ = 1? In this case, we have that the average cost per unit time when the depot uses N is 6 6 + cμ N (N − 1)/2 =N −1+ Nμ N By treating this as a continuous function of N and using the calculus, we obtain that the minimal value of N is √ N = 6 ≈ 2.45 Hence, the optimal integral value of N is either 2 which yields a value 4, or 3 which also yields the value 4. Hence, either N = 2 or N = 3 minimizes the depot’s average cost.
Example 7.14 Suppose that customers arrive at a single-server system in accordance with a Poisson process with rate λ. Upon arriving a customer must pass through a door that leads to the server. However, each time someone passes through, the door becomes locked for the next t units of time. An arrival finding a locked door is lost, and a cost c is incurred by the system. An arrival finding the door unlocked passes through to the server. If the server is free, the customer enters service; if the server is busy, the customer departs without service and a cost K is incurred. If the service time of a customer is exponential with rate μ, find the average cost per unit time incurred by the system.
438
7
Renewal Theory and Its Applications
Solution: The preceding can be considered to be a renewal reward process, with a new cycle beginning each time a customer arrives to find the door unlocked. This is so because whether or not the arrival finds the server free, the door will become locked for the next t time units and the server will be busy for a time X that is exponentially distributed with rate μ. (If the server is free, X is the service time of the entering customer; if the server is busy, X is the remaining service time of the customer in service.) Since the next cycle will begin at the first arrival epoch after a time t has passed, it follows that E[time of a cycle] = t + 1/λ Let C1 denote the cost incurred during a cycle due to arrivals finding the door locked. Then, since each arrival in the first t time units of a cycle will result in a cost c, we have E[C1 ] = λtc Also, let C2 denote the cost incurred during a cycle due to an arrival finding the door unlocked but the server busy. Then because a cost K is incurred if the server is still busy a time t after the cycle began and, in addition, the next arrival after that time occurs before the service completion, we see that E[C2 ] = Ke−μt
λ λ+μ
Consequently, average cost per unit time =
λtc + λKe−μt /(λ + μ) t + 1/λ
Example 7.15 Consider a manufacturing process that sequentially produces items, each of which is either defective or acceptable. The following type of sampling scheme is often employed in an attempt to detect and eliminate most of the defective items. Initially, each item is inspected and this continues until there are k consecutive items that are acceptable. At this point 100% inspection ends and each successive item is independently inspected with probability α. This partial inspection continues until a defective item is encountered, at which time 100% inspection is reinstituted, and the process begins anew. If each item is, independently, defective with probability q, (a) what proportion of items are inspected? (b) if defective items are removed when detected, what proportion of the remaining items are defective?
7.4.
Renewal Reward Processes
439
Remark Before starting our analysis, note that the preceding inspection scheme was devised for situations in which the probability of producing a defective item changed over time. It was hoped that 100% inspection would correlate with times at which the defect probability was large and partial inspection when it was small. However, it is still important to see how such a scheme would work in the extreme case where the defect probability remains constant throughout. Solution: We begin our analysis by noting that we can treat the preceding as a renewal reward process with a new cycle starting each time 100% inspection is instituted. We then have proportion of items inspected =
E[number inspected in a cycle] E[number produced in a cycle]
Let Nk denote the number of items inspected until there are k consecutive acceptable items. Once partial inspection begins—that is, after Nk items have been produced—since each inspected item will be defective with probability q, it follows that the expected number that will have to be inspected to find a defective item is 1/q. Hence, E[number inspected in a cycle] = E[Nk ] +
1 q
In addition, since at partial inspection each item produced will, independently, be inspected and found to be defective with probability αq, it follows that the number of items produced until one is inspected and found to be defective is 1/αq, and so E[number produced in a cycle] = E[Nk ] +
1 αq
Also, as E[Nk ] is the expected number of trials needed to obtain k acceptable items in a row when each item is acceptable with probability p = 1 − q, it follows from Example 3.14 that E[Nk ] =
1 1 1 (1/p)k − 1 + 2 + ··· + k = p p p q
Hence we obtain PI ≡ proportion of items that are inspected =
(1/p)k (1/p)k − 1 + 1/α
To answer (b), note first that since each item produced is defective with probability q it follows that the proportion of items that are both inspected and found to be defective is qPI . Hence, for N large, out of the first N items produced there will be (approximately) N qPI that are discovered to be defective and
440
7
Renewal Theory and Its Applications
thus removed. As the first N items will contain (approximately) N q defective items, it follows that there will be N q − N qPI defective items not discovered. Hence, proportion of the nonremoved items that are defective ≈
N q(1 − PI ) N (1 − qPI )
As the approximation becomes exact as N → ∞, we see that proportion of the nonremoved items that are defective =
q(1 − PI ) (1 − qPI )
Example 7.16 (The Average Age of a Renewal Process) Consider a renewal process having interarrival distribution F and define A(t) to be the time at t since the last renewal. If renewals represent old items failing and being replaced by new ones, then A(t) represents the age of the item in use at time t. Since SN (t) represents the time of the last event prior to or at time t, we have that A(t) = t − SN (t) We are interested in the average value of the age—that is, in s lim
s→∞
0
A(t) dt s
To determine this quantity, we use renewal reward theory in the following way: Let us assume that at any time we are being paid money at a rate equal to the age of the renewal s process at that time. That is, at time t, we are being paid at rate A(t), and so 0 A(t) dt represents our total earnings by time s. As everything starts over again when a renewal occurs, it follows that s 0
A(t) dt E[reward during a renewal cycle] → s E[time of a renewal cycle]
Now since the age of the renewal process a time t into a renewal cycle is just t, we have
X
reward during a renewal cycle =
t dt 0
=
X2 2
7.4.
Renewal Reward Processes
441
where X is the time of the renewal cycle. Hence, we have that s average value of age ≡ lim
0
s→∞
=
A(t) dt s
E[X 2 ] 2E[X]
where X is an interarrival time having distribution function F .
(7.14)
Example 7.17 (The Average Excess of a Renewal Process) Another quantity associated with a renewal process is Y (t), the excess or residual time at time t. Y (t) is defined to equal the time from t until the next renewal and, as such, represents the remaining (or residual) life of the item in use at time t. The average value of the excess, namely, s lim
s→∞
0
Y (t) dt s
also can be easily obtained by renewal reward theory. To do so, suppose that we are paid at time t at a rate equal to Y (t). Then our average reward per unit time will, by renewal reward theory, be given by s
Y (t) dt s→∞ s E[reward during a cycle] = E[length of a cycle] 0
average value of excess ≡ lim
Now, letting X denote the length of a renewal cycle, we have that reward during a cycle =
X
(X − t) dt
0
=
X2 2
and thus the average value of the excess is average value of excess =
E[X 2 ] 2E[X]
which was the same result obtained for the average value of the age of renewal process.
442
7
7.5.
Renewal Theory and Its Applications
Regenerative Processes
Consider a stochastic process {X(t), t 0} with state space 0, 1, 2, . . . , having the property that there exist time points at which the process (probabilistically) restarts itself. That is, suppose that with probability one, there exists a time T1 , such that the continuation of the process beyond T1 is a probabilistic replica of the whole process starting at 0. Note that this property implies the existence of further times T2 , T3 , . . . , having the same property as T1 . Such a stochastic process is known as a regenerative process. From the preceding, it follows that T1 , T2 , . . . , constitute the arrival times of a renewal process, and we shall say that a cycle is completed every time a renewal occurs.
Examples (1) A renewal process is regenerative, and T1 represents the time of the first renewal. (2) A recurrent Markov chain is regenerative, and T1 represents the time of the first transition into the initial state. We are interested in determining the long-run proportion of time that a regenerative process spends in state j . To obtain this quantity, let us imagine that we earn a reward at a rate 1 per unit time when the process is in state j and at rate 0 otherwise. That is, if I (s) represents the rate at which we earn at time s, then 1, if X(s) = j I (s) = 0, if X(s) = j and
t
total reward earned by t =
I (s) ds 0
As the preceding is clearly a renewal reward process that starts over again at the cycle time T1 , we see from Proposition 7.3 that average reward per unit time =
E[reward by time T1 ] E[T1 ]
However, the average reward per unit is just equal to the proportion of time that the process is in state j . That is, we have the following.
Proposition 7.4 For a regenerative process, the long-run proportion of time in state j =
E[amount of time in j during a cycle] E[time of a cycle]
7.5.
Regenerative Processes
443
Remark If the cycle time T1 is a continuous random variable, then it can be shown by using an advanced theorem called the “key renewal theorem” that the preceding is equal also to the limiting probability that the system is in state j at time t. That is, if T1 is continuous, then lim P {X(t) = j } =
t→∞
E[amount of time in j during a cycle] E[time of a cycle]
Example 7.18 Consider a positive recurrent continuous time Markov chain that is initially in state i. By the Markovian property, each time the process reenters state i it starts over again. Thus returns to state i are renewals and constitute the beginnings of new cycles. By Proposition 7.4, it follows that the long-run proportion of time in state j =
E[amount of time in j during an i – i cycle] μii
where μii represents the mean time to return to state i. If we take j to equal i, then we obtain proportion of time in state i =
1/vi μii
Example 7.19 (A Queueing System with Renewal Arrivals) Consider a waiting time system in which customers arrive in accordance with an arbitrary renewal process and are served one at time by a single server having an arbitrary service distribution. If we suppose that at time 0 the initial customer has just arrived, then {X(t), t 0} is a regenerative process, where X(t) denotes the number of customers in the system at time t. The process regenerates each time a customer arrives and finds the server free.
Example 7.20 Although a system needs only a single machine to function, it maintains an additional machine as a backup. A machine in use functions for a random time with density function f and then fails. If a machine fails while the other one is in working condition, then the latter is put in use and, simultaneously, repair begins on the one that just failed. If a machine fails while the other machine is in repair, then the newly failed machine waits until the repair is completed; at that time the repaired machine is put in use and, simultaneously, repair begins on the recently failed one. All repair times have density function g. Find P0 , P1 , P2 , where Pi is the long-run proportion of time that exactly i of the machines are in working condition. Solution: Let us say that the system is in state i whenever i machines are in working condition i = 0, 1, 2. It is then easy to see that every time the system enters state 1 it probabilistically starts over. That is, the system restarts every
444
7
Renewal Theory and Its Applications
time that a machine is put in use while, simultaneously, repair begins on the other one. Say that a cycle begins each time the system enters state 1. If we let X denote the working time of the machine put in use at the beginning of a cycle, and let R be the repair time of the other machine, then the length of the cycle, call it Tc , can be expressed as Tc = max(X, R) The preceding follows when X R, because, in this case, the machine in use fails before the other one has been repaired, and so a new cycle begins when that repair is completed. Similarly, it follows when R < X, because then the repair occurs first, and so a new cycle begins when the machine in use fails. Also, let Ti , i = 0, 1, 2, be the amount of time that the system is in state i during a cycle. Then, because the amount of time during a cycle that neither machine is working is R − X provided that this quantity is positive or 0 otherwise, we have T0 = (R − X)+ Similarly, because the amount of time during the cycle that a single machine is working is min(X, R), we have T1 = min(X, R) Finally, because the amount of time during the cycle that both machines are working is X − R if this quantity is positive or 0 otherwise, we have T2 = (X − R)+ Hence, we obtain that P0 =
E[(R − X)+ ] E[max(X, R)]
P1 =
E[min(X, R)] E[max(X, R)]
P2 =
E[(X − R)+ ] E[max(X, R)]
That P0 + P1 + P2 = 1 follows from the easily checked identity max(x, r) = min(x, r) + (x − r)+ + (r − x)+
7.5.
Regenerative Processes
445
The preceding expectations can be computed as follows: ∞ ∞
E[max(X, R)] =
max(x, r)f (x) g(r) dx dr
0
=
+
0
E[(R − X) ] =
0
=
0
E[min(X, R)] =
∞ r
∞ ∞
rf (x) g(r) dx dr +
0
∞ ∞
xf (x) g(r) dx dr 0
r
(r − x)+ f (x) g(r) dx dr
0
∞ r
(r − x)f (x) g(r) dx dr
0
∞ ∞
min(x, r) f (x) g(r) dx dr
0
= +
0
0
E[(X − R) ] = 0
0
∞ r
xf (x) g(r) dx dr +
0
∞ ∞
∞ ∞
rf (x) g(r) dx dr 0
(x − r) f (x) g(r) dr dx
r
0
7.5.1. Alternating Renewal Processes Another example of a regenerative process is provided by what is known as an alternating renewal process, which considers a system that can be in one of two states: on or off. Initially it is on, and it remains on for a time Z1 ; it then goes off and remains off for a time Y1 . It then goes on for a time Z2 ; then off for a time Y2 ; then on, and so on. We suppose that the random vectors (Zn , Yn ), n 1 are independent and identically distributed. That is, both the sequence of random variables {Zn } and the sequence {Yn } are independent and identically distributed; but we allow Zn and Yn to be dependent. In other words, each time the process goes on, everything starts over again, but when it then goes off, we allow the length of the off time to depend on the previous on time. Let E[Z] = E[Zn ] and E[Y ] = E[Yn ] denote, respectively, the mean lengths of an on and off period. We are concerned with Pon , the long-run proportion of time that the system is on. If we let Xn = Yn + Zn ,
n1
446
7
Renewal Theory and Its Applications
then at time X1 the process starts over again. That is, the process starts over again after a complete cycle consisting of an on and an off interval. In other words, a renewal occurs whenever a cycle is completed. Therefore, we obtain from Proposition 7.4 that Pon = =
E[Z] E[Y ] + E[Z] E[on] E[on] + E[off]
(7.15)
Also, if we let Poff denote the long-run proportion of time that the system is off, then Poff = 1 − Pon =
E[off] E[on] + E[off]
(7.16)
Example 7.21 (A Production Process) One example of an alternating renewal process is a production process (or a machine) which works for a time Z1 , then breaks down and has to be repaired (which takes a time Y1 ), then works for a time Z2 , then is down for a time Y2 , and so on. If we suppose that the process is as good as new after each repair, then this constitutes an alternating renewal process. It is worthwhile to note that in this example it makes sense to suppose that the repair time will depend on the amount of time the process had been working before breaking down.
Example 7.22 The rate a certain insurance company charges its policyholders alternates between r1 and r0 . A new policyholder is initially charged at a rate of r1 per unit time. When a policyholder paying at rate r1 has made no claims for the most recent s time units, then the rate charged becomes r0 per unit time. The rate charged remains at r0 until a claim is made, at which time it reverts to r1 . Suppose that a given policyholder lives forever and makes claims at times chosen according to a Poisson process with rate λ, and find (a) Pi , the proportion of time that the policyholder pays at rate ri , i = 0, 1; (b) the long-run average amount paid per unit time. Solution: If we say that the system is “on” when the policyholder pays at rate r1 and “off” when she pays at rate r0 , then this on–off system is an alternating renewal process with a new cycle starting each time a claim is made. If X is the time between successive claims, then the on time in the cycle is the
7.5.
Regenerative Processes
447
smaller of s and X. (Note that if X < s, then the off time in the cycle is 0.) Since X is exponential with rate λ, the preceding yields that E[on time in cycle] = E[min(X, s)] s = xλe−λx dx + se−λs 0
1 = (1 − e−λs ) λ Since E[X] = 1/λ, we see that P1 =
E[on time in cycle] = 1 − e−λs E[X]
and P0 = 1 − P1 = e−λs The long-run average amount paid per unit time is r0 P0 + r1 P1 = r1 − (r1 − r0 )e−λs
Example 7.23 (The Age of a Renewal Process) Suppose we are interested in determining the proportion of time that the age of a renewal process is less than some constant c. To do so, let a cycle correspond to a renewal, and say that the system is “on” at time t if the age at t is less than or equal to c, and say it is “off” if the age at t is greater than c. In other words, the system is “on” the first c time units of a renewal interval, and “off” the remaining time. Hence, letting X denote a renewal interval, we have, from Equation (7.15), E[min(X, c)] E[X] ∞ P {min(X, c) > x} dx = 0 E[X] c P {X > x} dx = 0 E[X] c (1 − F (x)) dx (7.17) = 0 E[X]
proportion of time age is less than c =
where F is the distribution function of X and where we have used the identity that for a nonnegative random variable Y ∞ E[Y ] = P {Y > x} dx 0
448
7
Renewal Theory and Its Applications
Example 7.24 (The Excess of a Renewal Process) Let us now consider the long-run proportion of time that the excess of a renewal process is less than c. To determine this quantity, let a cycle correspond to a renewal interval and say that the system is on whenever the excess of the renewal process is greater than or equal to c and that it is off otherwise. In other words, whenever a renewal occurs the process goes on and stays on until the last c time units of the renewal interval when it goes off. Clearly this is an alternating renewal process, and so we obtain from Equation (7.16) that long-run proportion of time the excess is less than c =
E[off time in cycle] E[cycle time]
If X is the length of a renewal interval, then since the system is off the last c time units of this interval, it follows that the off time in the cycle will equal min(X, c). Thus, E[min(X, c)] E[X] c (1 − F (x)) dx = 0 E[X]
long-run proportion of time the excess is less than c =
where the final equality follows from Equation (7.17). Thus, we see from the result of Example 7.23 that the long-run proportion of time that the excess is less than c and the long-run proportion of time that the age is less than c are equal. One way to understand this equivalence is to consider a renewal process that has been in operation for a long time and then observe it going backwards in time. In doing so, we observe a counting process where the times between successive events are independent random variables having distribution F . That is, when we observe a renewal process going backwards in time we again observe a renewal process having the same probability structure as the original. Since the excess (age) at any time for the backwards process corresponds to the age (excess) at that time for the original renewal process (see Figure 7.3), it follows that all long-run properties of the age and the excess must be equal.
Figure 7.3. Arrowheads indicate direction of time.
7.5.
Regenerative Processes
449
Example 7.25 (The Busy Period of the M/G/∞ Queue) The infinite server queueing system in which customers arrive according to a Poisson process with rate λ, and have a general service distribution G, was analyzed in Section 5.3, where it was shown that the t number of customers in the system at time t is Poisson ¯ If we say that the system is busy when there distributed with mean λ 0 G(t)dt. is at least one customer in the system and is idle when the system is empty, find E[B], the expected length of a busy period. Solution: If we say that the system is on when there is at least one customer in the system, and off when ∞the system is empty, then we have an alternating ¯ = E[S], where E[S] is the mean of the renewal process. Because 0 G(t)dt service distribution G, it follows from the result of Section 5.3 that lim P {system off at t} = e−λE[S]
t→∞
Consequently, from alternating renewal process theory we obtain e−λE[S] =
E[off time in cycle] E[cycle time]
But when the system goes off, it remains off only up to the time of the next arrival, giving that E[off time in cycle] = 1/λ Because E[on time in cycle] = E[B] we obtain that e−λE[S] =
1/λ 1/λ + E[B]
or 1 E[B] = (eλE[S] − 1) λ If μ is the mean interarrival time, then the distribution function Fe , defined by x 1 − F (y) Fe (x) = dy μ 0 is called the equilibrium distribution of F . From the preceding, it follows that Fe (x) represents the long-run proportion of time that the age, and the excess, of the renewal process is less than or equal to x.
450
7
Renewal Theory and Its Applications
Example 7.26 (An Inventory Example) Suppose that customers arrive at a specified store in accordance with a renewal process having interarrival distribution F. Suppose that the store stocks a single type of item and that each arriving customer desires a random amount of this commodity, with the amounts desired by the different customers being independent random variables having the common distribution G. The store uses the following (s, S) ordering policy: If its inventory level falls below s then it orders enough to bring its inventory up to S. That is, if the inventory after serving a customer is x, then the amount ordered is S − x, 0,
if x < s if x s
The order is assumed to be instantaneously filled. For a fixed value y, s y S, suppose that we are interested in determining the long-run proportion of time that the inventory on hand is at least as large as y. To determine this quantity, let us say that the system is “on” whenever the inventory level is at least y and is “off” otherwise. With these definitions, the system will go on each time that a customer’s demand causes the store to place an order that results in its inventory level returning to S. Since whenever this occurs a customer must have just arrived it follows that the times until succeeding customers arrive will constitute a renewal process with interarrival distribution F ; that is, the process will start over each time the system goes back on. Thus, the on and off periods so defined constitute an alternating renewal process, and from Equation (7.15) we have that long-run proportion of time inventory y =
E[on time in a cycle] E[cycle time]
(7.18)
Now, if we let D1 , D2 , . . . denote the successive customer demands, and let Nx = min(n : D1 + · · · + Dn > S − x)
(7.19)
then it is the Ny customer in the cycle that causes the inventory level to fall below y, and it is the Ns customer that ends the cycle. As a result, if we let Xi , i 1, denote the interarrival times of customers, then on time in a cycle =
Ny
Xi
(7.20)
Xi
(7.21)
i=1
cycle time =
Ns i=1
7.5.
451
Regenerative Processes
Assuming that the interarrival times are independent of the successive demands, we have that E
, Ny
-
, , Ny - Xi |Ny =E E
Xi
i=1
i=1
= E[Ny E[X]] = E[X]E[Ny ] Similarly, E
,N s
Xi = E[X]E[Ns ]
i=1
Therefore, from Equations (7.18), (7.20), and (7.21) we see that long-run proportion of time inventory y =
E[Ny ] E[Ns ]
(7.22)
However, as the Di , i 1, are independent and identically distributed nonnegative random variables with distribution G, it follows from Equation (7.19) that Nx has the same distribution as the index of the first event to occur after time S − x of a renewal process having interarrival distribution G. That is, Nx − 1 would be the number of renewals by time S − x of this process. Hence, we see that E[Ny ] = m(S − y) + 1, E[Ns ] = m(S − s) + 1 where m(t) =
∞
Gn (t)
n=1
From Equation (7.22), we arrive at long-run proportion of time inventory y =
m(S − y) + 1 , m(S − s) + 1
sy S
For instance, if the customer demands are exponentially distributed with mean 1/μ, then long-run proportion of time inventory y =
μ(S − y) + 1 , μ(S − s) + 1
syS
452
7.6.
7
Renewal Theory and Its Applications
Semi-Markov Processes
Consider a process that can be in state 1 or state 2 or state 3. It is initially in state 1 where it remains for a random amount of time having mean μ1 , then it goes to state 2 where it remains for a random amount of time having mean μ2 , then it goes to state 3 where it remains for a mean time μ3 , then back to state 1, and so on. What proportion of time is the process in state i, i = 1, 2, 3? If we say that a cycle is completed each time the process returns to state 1, and if we let the reward be the amount of time we spend in state i during that cycle, then the preceding is a renewal reward process. Hence, from Proposition 7.3 we obtain that Pi , the proportion of time that the process is in state i, is given by Pi =
μi , μ1 + μ2 + μ3
i = 1, 2, 3
Similarly, if we had a process which could be in any of N states 1, 2, . . . , N and which moved from state 1 → 2 → 3 → · · · → N − 1 → N → 1, then the long-run proportion of time that the process spends in state i is Pi =
μi , μ1 + μ2 + · · · + μN
i = 1, 2, . . . , N
where μi is the expected amount of time the process spends in state i during each visit. Let us now generalize the preceding to the following situation. Suppose that a process can be in any one of N states 1, 2, . . . , N , and that each time it enters state i it remains there for a random amount of time having mean μi and then makes a transition into state j with probability Pij . Such a process is called a semi-Markov process. Note that if the amount of time that the process spends in each state before making a transition is identically 1, then the semi-Markov process is just a Markov chain. Let us calculate Pi for a semi-Markov process. To do so, we first consider πi the proportion of transitions that take the process into state i. Now if we let Xn denote the state of the process after the nth transition, then {Xn , n 0} is a Markov chain with transition probabilities Pij , i, j = 1, 2, . . . , N . Hence, πi will just be the limiting (or stationary) probabilities for this Markov chain (Section 4.4). That is, πi will be the unique nonnegative solution∗ of ∗ We shall assume that there exists a solution of Equation (7.23). That is, we assume that all of the
states in the Markov chain communicate.
7.6. N
Semi-Markov Processes
453
πi = 1,
i=1
πi =
N
πj Pj i ,
i = 1, 2, . . . , N
(7.23)
j =1
Now since the process spends an expected time μi in state i whenever it visits that state, it seems intuitive that Pi should be a weighted average of the πi where πi is weighted proportionately to μi . That is, πi μi , Pi = N j =1 πj μj
i = 1, 2, . . . , N
(7.24)
where the πi are given as the solution to Equation (7.23).
Example 7.27 Consider a machine that can be in one of three states: good condition, fair condition, or broken down. Suppose that a machine in good condition will remain this way for a mean time μ1 and then will go to either the fair condition or the broken condition with respective probabilities 34 and 14 . A machine in fair condition will remain that way for a mean time μ2 and then will break down. A broken machine will be repaired, which takes a mean time μ3 , and when repaired will be in good condition with probability 23 and fair condition with probability 13 . What proportion of time is the machine in each state? Solution: πi satisfy
Letting the states be 1, 2, 3, we have by Equation (7.23) that the
π1 + π2 + π3 = 1, π1 = 23 π3 , π2 = 34 π1 + 13 π3 , π3 = 14 π1 + π2 The solution is π1 =
4 15 ,
π2 = 13 ,
π3 =
2 5
454
7
Renewal Theory and Its Applications
Hence, from Equation (7.24) we obtain that Pi , the proportion of time the machine is in state i, is given by P1 =
4μ1 , 4μ1 + 5μ2 + 6μ3
P2 =
5μ2 , 4μ1 + 5μ2 + 6μ3
P3 =
6μ3 4μ1 + 5μ2 + 6μ3
For instance, if μ1 = 5, μ2 = 2, μ3 = 1, then the machine will be in good 5 of the time, in broken condition 16 condition 59 of the time, in fair condition 18 of the time.
Remark When the distributions of the amount of time spent in each state dur-
ing a visit are continuous, then Pi also represents the limiting (as t → ∞) probability that the process will be in state i at time t.
Example 7.28 Consider a renewal process in which the interarrival distribution is discrete and is such that P {X = i} = pi ,
i1
where X represents an interarrival random variable. Let L(t) denote the length of the renewal interval that contains the point t [that is, if N (t) is the number of renewals by time t and Xn the nth interarrival time, then L(t) = XN (t)+1 ]. If we think of each renewal as corresponding to the failure of a lightbulb (which is then replaced at the beginning of the next period by a new bulb), then L(t) will equal i if the bulb in use at time t dies in its ith period of use. It is easy to see that L(t) is a semi-Markov process. To determine the proportion of time that L(t) = j , note that each time a transition occurs—that is, each time a renewal occurs—the next state will be j with probability pj . That is, the transition probabilities of the embedded Markov chain are Pij = pj . Hence, the limiting probabilities of this embedded chain are given by πj = pj and, since the mean time the semi-Markov process spends in state j before a transition occurs is j , it follows that the long-run proportion of time the state is j is jpj Pj = i ipi
7.7.
7.7.
The Inspection Paradox
455
The Inspection Paradox
Suppose that a piece of equipment, say, a battery, is installed and serves until it breaks down. Upon failure it is instantly replaced by a like battery, and this process continues without interruption. Letting N (t) denote the number of batteries that have failed by time t, we have that {N (t), t 0} is a renewal process. Suppose further that the distribution F of the lifetime of a battery is not known and is to be estimated by the following sampling inspection scheme. We fix some time t and observe the total lifetime of the battery that is in use at time t. Since F is the distribution of the lifetime for all batteries, it seems reasonable that it should be the distribution for this battery. However, this is the inspection paradox for it turns out that the battery in use at time t tends to have a larger lifetime than an ordinary battery. To understand the preceding so-called paradox, we reason as follows. In renewal theoretic terms what we are interested in is the length of the renewal interval containing the point t. That is, we are interested in XN (t)+1 = SN (t)+1 − SN (t) (see Figure 7.2). To calculate the distribution of XN (t)+1 we condition on the time of the last renewal prior to (or at) time t. That is, P {XN (t)+1 > x} = E[P {XN (t)+1 > x|SN (t) = t − s}] where we recall (Figure 7.2) that SN (t) is the time of the last renewal prior to (or at) t. Since there are no renewals between t − s and t, it follows that XN (t)+1 must be larger than x if s > x. That is, P {XN (t)+1 > x|SN (t) = t − s} = 1
if s > x
(7.25)
On the other hand, suppose that s x. As before, we know that a renewal occurred at time t − s and no additional renewals occurred between t − s and t, and we ask for the probability that no renewals occur for an additional time x − s. That is, we are asking for the probability that an interarrival time will be greater than x given that it is greater than s. Therefore, for s x, P {XN (t)+1 > x|SN (t) = t − s} = P {interarrival time > x|interarrival time > s} = P {interarrival time > x}/P {interarrival time > s} 1 − F (x) = 1 − F (s) 1 − F (x)
(7.26)
456
7
Renewal Theory and Its Applications
Figure 7.4.
Hence, from Equations (7.25) and (7.26) we see that, for all s, P {XN (t)+1 > x|SN (t) = t − s} 1 − F (x) Taking expectations on both sides yields that P {XN (t)+1 > x} 1 − F (x)
(7.27)
However, 1 − F (x) is the probability that an ordinary renewal interval is larger than x, that is, 1 − F (x) = P {Xn > x}, and thus Equation (7.27) is a statement of the inspection paradox that the length of the renewal interval containing the point t tends to be larger than an ordinary renewal interval.
Remark To obtain an intuitive feel for the so-called inspection paradox, reason as follows. We think of the whole line being covered by renewal intervals, one of which covers the point t. Is it not more likely that a larger interval, as opposed to a shorter interval, covers the point t? We can explicitly calculate the distribution of XN (t)+1 when the renewal process is a Poisson process. [Note that, in the general case, we did not need to calculate explicitly P {XN (t)+1 > x} to show that it was at least as large as 1 − F (x).] To do so we write XN (t)+1 = A(t) + Y (t) where A(t) denotes the time from t since the last renewal, and Y (t) denotes the time from t until the next renewal (see Figure 7.4). A(t) is the age of the process at time t (in our example it would be the age at time t of the battery in use at time t), and Y (t) is the excess life of the process at time t (it is the additional time from t until the battery fails). Of course, it is true that A(t) = t − SN (t) , and Y (t) = SN (t)+1 − t. To calculate the distribution of XN (t)+1 we first note the important fact that, for a Poisson process, A(t) and Y (t) are independent. This follows since by the memoryless property of the Poisson process, the time from t until the next renewal will be exponentially distributed and will be independent of all that has previously occurred [including, in particular, A(t)]. In fact, this shows that if {N (t), t 0}
7.7.
457
The Inspection Paradox
is a Poisson process with rate λ, then P {Y (t) x} = 1 − e−λx
(7.28)
The distribution of A(t) may be obtained as follows P {A(t) > x} = =
P {0 renewals in [t − x, t]},
if x t
0,
if x > t
e−λx , 0,
if x t if x > t
or, equivalently, P {A(t) x} =
1 − e−λx , 1,
xt x>t
(7.29)
Hence, by the independence of Y (t) and A(t) the distribution of XN (t)+1 is just the convolution of the exponential distribution equation (7.28) and the distribution of equation (7.29). It is interesting to note that for t large, A(t) approximately has an exponential distribution. Thus, for t large, XN (t)+1 has the distribution of the convolution of two identically distributed exponential random variables, which by Section 5.2.3, is the gamma distribution with parameters (2, λ). In particular, for t large, the expected length of the renewal interval containing the point t is approximately twice the expected length of an ordinary renewal interval. Using the results obtained in Examples 7.16 and 7.17 concerning the average values of the age and of the excess, it follows from the identity XN (t)+1 = A(t) + Y (t) that the average length of the renewal interval containing a specified point is s lim
s→∞
0
XN (t)+1 dt E[X 2 ] = s E[X]
where X has the interarrival distribution. Because, except for when X is a constant, E[X 2 ] > (E[X])2 , this average value is, as expected from the inspection paradox, greater than the expected value of an ordinary renewal interval.
458
7
Renewal Theory and Its Applications
We can use an alternating renewal process argument to determine the long-run proportion of time that XN (t)+1 is greater than c. To do so, let a cycle correspond to a renewal interval, and say that the system is on at time t if the renewal interval containing t is of length greater than c (that is, if XN (t)+1 > c), and say that the system is off at time t otherwise. In other words, the system is always on during a cycle if the cycle time exceeds c or is always off during the cycle if the cycle time is less than c. Thus, if X is the cycle time, we have X, if X > c on time in cycle = 0, if X c Therefore, we obtain from alternating renewal process theory that E[on time in cycle] E[cycle time] ∞ xf (x) dx = c μ
long-run proportion of timeXN (t)+1 > c =
where f is the density function of an interarrival.
7.8.
Computing the Renewal Function
The difficulty with attempting to use the identity m(t) =
∞
Fn (t)
n=1
to compute the renewal function is that the determination of Fn (t) = P {X1 + · · · + Xn t} requires the computation of an n-dimensional integral. Following, we present an effective algorithm which requires as inputs only onedimensional integrals. Let Y be an exponential random variable having rate λ, and suppose that Y is independent of the renewal process {N (t), t 0}. We start by determining E[N(Y )], the expected number of renewals by the random time Y . To do so, we first condition on X1 , the time of the first renewal. This yields ∞ E[N (Y )] = E[N (Y )|X1 = x]f (x) dx (7.30) 0
where f is the interarrival density. To determine E[N (Y )|X1 = x], we now condition on whether or not Y exceeds x. Now, if Y < x, then as the first renewal occurs at time x, it follows that the number of renewals by time Y is equal to 0.
7.8.
Computing the Renewal Function
459
On the other hand, if we are given that x < Y , then the number of renewals by time Y will equal 1 (the one at x) plus the number of additional renewals between x and Y . But by the memoryless property of exponential random variables, it follows that, given that Y > x, the amount by which it exceeds x is also exponential with rate λ, and so given that Y > x the number of renewals between x and Y will have the same distribution as N (Y ). Hence, E[N (Y )|X1 = x, Y < x] = 0, E[N (Y )|X1 = x, Y > x] = 1 + E[N (Y )] and so, E[N (Y )|X1 = x] = E[N (Y )|X1 = x, Y < x]P {Y < x|X1 = x} + E[N (Y )|X1 = x, Y > x]P {Y > x|X1 = x} = E[N (Y )|X1 = x, Y > x]P {Y > x} since Y and X1 are independent = (1 + E[N (Y )])e−λx Substituting this into Equation (7.30) gives
∞
E[N (Y )] = (1 + E[N (Y )])
e−λxf (x) dx
0
or E[N (Y )] =
E[e−λX ] 1 − E[e−λX ]
(7.31)
where X has the renewal interarrival distribution. If we let λ = 1/t, then Equation (7.31) presents an expression for the expected number of renewals (not by time t, but) by a random exponentially distributed time with mean t. However, as such a random variable need not be close to its mean (its variance is t 2 ), Equation (7.31) need not be particularly close to m(t). To obtain an accurate approximation suppose that Y1 , Y2 , . . . , Yn are independent exponentials with rate λ and suppose they are also independent of the renewal process. Let, for r = 1, . . . , n, mr = E[N (Y1 + · · · + Yr )] To compute an expression for mr , we again start by conditioning on X1 , the time of the first renewal. ∞ mr = E[N (Y1 + · · · + Yr )|X1 = x] f (x) dx (7.32) 0
460
7
Renewal Theory and Its Applications
To determine the foregoing conditional expectation, we now condition on the j number of partial sums i=1 Yi , j = 1, . . . , r, that are less than x. Now, if all r r partial sums are less than x—that is, if i=1 Yi < x—then clearly the numr ber of renewals by time i=1 Yi is 0. On the other hand, given that k, k < r, of these partial sums are less than x, it follows from the lack of memory property of the exponential that the number of renewals by time ri=1 Yi will have the same distribution as 1 plus N (Yk+1 + · · · + Yr ). Hence, ,
j ! ! E N (Y1 + · · · + Yr )!X1 = x, k of the sums Yi are less than x
-
i=1
0, = 1 + mr−k ,
if k = r if k < r
(7.33)
To determine the distribution of the number of the partial sums that are less than j x, note that the successive values of these partial sums i=1 Yi , j = 1, . . . , r, have the same distribution as the first r event times of a Poisson process with rate λ (since each successive partial sum is the previous sum plus an independent exponential with rate λ). Hence, it follows that, for k < r, P k of the partial sums
j
! ! Yi are less than x !X1 = x
i=1
=
e−λx (λx)k
(7.34)
k!
Upon substitution of Equations (7.33) and (7.34) into Equation (7.32), we obtain
r−1 ∞
mr =
(1 + mr−k )
0
k=0
e−λx (λx)k f (x) dx k!
or, equivalently, r−1 mr =
k=1 (1 + mr−k )E[X
k e−λX ](λk /k!) + E[e−λX ]
1 − E[e−λX ]
(7.35)
If we set λ = n/t, then starting with m1 given by Equation (7.31), we can use Equation (7.35) to recursively compute m2 , . . . , mn . The approximation of m(t) = E[N(t)] is given by mn = E[N (Y1 + · · · + Yn )]. Since Y1 + · · · + Yn is the sum of n independent exponential random variables each with mean t/n, it follows that it is (gamma) distributed with mean t and variance nt 2 /n2 = t 2 /n. Hence, by
7.9.
Table 7.1
461
Approximating m(t) Exact
Fi
Applications to Patterns
Approximation
i
t
m(t)
n=1
n=3
n = 10
n = 25
n = 50
1 1 1 1
1 2 5 10
0.2838 0.7546 2.250 4.75
0.3333 0.8000 2.273 4.762
0.3040 0.7697 2.253 4.751
0.2903 0.7586 2.250 4.750
0.2865 0.7561 2.250 4.750
0.2852 0.7553 2.250 4.750
2 2 2 2 2 2
0.1 0.3 0.5 1 3 10
0.1733 0.5111 0.8404 1.6400 4.7389 15.5089
0.1681 0.4964 0.8182 1.6087 4.7143 15.5000
0.1687 0.4997 0.8245 1.6205 4.7294 15.5081
0.1689 0.5010 0.8273 1.6261 4.7350 15.5089
0.1690 0.5014 0.8281 1.6277 4.7363 15.5089
— — 0.8283 1.6283 4.7367 15.5089
3 3 3 3
0.1 0.3 1 3
0.2819 0.7638 2.0890 5.4444
0.2692 0.7105 2.0000 5.4000
0.2772 0.7421 2.0556 5.4375
0.2804 0.7567 2.0789 5.4437
0.2813 0.7609 2.0850 5.4442
— — 2.0870 5.4443
choosing n large, ni=1 Yi will be a random variable having most of its probability concentrated about t, and so E[N ( ni=1 Yi )] should be quite close to E[N (t)]. [Indeed, if m(t) is continuous at t, it can be shown that these approximations converge to m(t) as n goes to infinity.]
Example 7.29 Table 7.1 compares the approximation with the exact value for the distributions Fi with densities fi , i = 1, 2, 3, which are given by f1 (x) = xe−x , 1 − F2 (x) = 0.3e−x + 0.7e−2x , 1 − F3 (x) = 0.5e−x + 0.5e−5x
7.9.
Applications to Patterns
A counting process with independent interarrival times X1 , X2 , . . . is said to be a delayed or general renewal process if X1 has a different distribution from the identically distributed random variables X2 , X3 , . . . . That is, a delayed renewal process is a renewal process in which the first interarrival time has a different distribution than the others. Delayed renewal processes often arise in
462
7
Renewal Theory and Its Applications
practice and it is important to note that all of the limiting theorems about N(t), the number of events by time t, remain valid. For instance, it remains true that E[N (t)] 1 → t μ
and
Var(N (t)) → σ 2 /μ3 t
as t → ∞
where μ and σ 2 are the expected value and variance of the interarrivals Xi , i > 1. 7.9.1. Patterns of Discrete Random Variables Let X1 , X2 , . . . be independent with P {Xi = j } = p(j ), j 0, and let T denote the first time the pattern x1 , . . . , xr occurs. If we say that a renewal occurs at time n, n r, if (Xn−r+1 , . . . , Xn ) = (x1 , . . . , xr ), then N (n), n 1, is a delayed renewal process, where N (n) denotes the number of renewals by time n. It follows that 1 E[N (n)] → n μ
as n → ∞
σ2 Var(N (n)) → 3 n μ
as n → ∞
(7.36) (7.37)
where μ and σ are, respectively, the mean and standard deviation of the time between successive renewals. Whereas, in Section 3.6.4, we showed how to compute the expected value of T , we will now show how to use renewal theory results to compute both its mean and its variance. To begin, let I (i) 1 equal 1 if there is a renewal at time i and let it be 0 otherwise, i r. Also, let p = ri=1 p(xi ). Since, P {I (i) = 1} = P {Xi−r+1 = i1 , . . . , Xi = ir } = p it follows that I (i), i r, are Bernoulli random variables with parameter p. Now, N (n) =
n
I (i)
i=r
so E[N (n)] =
n
E[I (i)] = (n − r + 1)p
i=r
Dividing by n and then letting n → ∞ give, from Equation (7.36), μ = 1/p
(7.38)
7.9.
Applications to Patterns
463
That is, the mean time between successive occurrences of the pattern is equal to 1/p. Also, n n−1 2 Var(N(n)) 1 Var(I (i)) + Cov(I (i), I (j )) = n n n i=r
i=r nj >i
n−r +1 2 p(1 − p) + n n
n−1
=
Cov(I (i), I (j ))
i=r i<j min(i+r−1, n)
where the final equality used the fact that I (i) and I (j ) are independent, and thus have zero covariance, when |i − j | r. Letting n → ∞, and using the fact that Cov(I (i), I (j )) depends on i and j only through |j − i|, gives Var(N(n)) → p(1 − p) + 2 Cov(I (r), I (r + j )) n r−1
j =1
Therefore, using Equations (7.37) and (7.38), we see that
σ 2 = p −2 (1 − p) + 2p −3
r−1
Cov(I (r), I (r + j ))
(7.39)
j =1
Let us now consider the amount of “overlap” in the pattern. The overlap, equal to the number of values at the end of one pattern that can be used as the beginning part of the next pattern, is said to be of size k, k > 0, if k = max{j < r : (ir−j +1 , . . . , ir ) = (i1 , . . . , ij )} and is of size 0 if for all k = 1, . . . , r − 1, (ir−k+1 , . . . , ir ) = (i1 , . . . , ik ). Thus, for instance, the pattern 0, 0, 1, 1 has overlap 0, whereas 0, 0, 1, 0, 0 has overlap 2. We consider two cases. Case 1: The Pattern Has Overlap 0 In this case, N (n), n 1, is an ordinary renewal process and T is distributed as an interarrival time with mean μ and variance σ 2 . Hence, we have the following
464
7
Renewal Theory and Its Applications
from Equation (7.38): E[T ] = μ =
1 p
(7.40)
Also, since two patterns cannot occur within a distance less than r of each other, it follows that I (r)I (r + j ) = 0 when 1 j r − 1. Hence, Cov(I (r), I (r + j )) = −E[I (r)]E[I (r + j )] = −p 2 ,
if 1 j r − 1
Hence, from Equation (7.39), we obtain Var(T ) = σ 2 = p −2 (1 − p) − 2p −3 (r − 1)p 2 = p −2 − (2r − 1)p −1
(7.41)
Remark In cases of “rare” patterns, if the pattern hasn’t yet occurred by some time n, then it would seem that we would have no reason to believe that the remaining time would be much less than if we were just beginning from scratch. That is, it would seem that the distribution is approximately memoryless and would thus be approximately exponentially distributed. Thus, since the variance of an exponential is equal to its mean squared, we would expect when μ is large that Var(T ) ≈ E 2 [T ], and this is borne out by the preceding, which states that Var(T ) = E 2 [T ] − (2r − 1)E[T ].
Example 7.30 Suppose we are interested in the number of times that a fair coin needs to be flipped before the pattern h, h, t, h, t occurs. For this pattern, 1 r = 5, p = 32 , and the overlap is 0. Hence, from Equations (7.40) and (7.41) E[T ] = 32,
Var(T ) = 322 − 9 × 32 = 736,
and Var(T )/E 2 [T ] = 0.71875 On the other hand, if p(i) = i/10, i = 1, 2, 3, 4 and the pattern is 1, 2, 1, 4, 1, 3, 2 then r = 7, p = 3/625,000, and the overlap is 0. Thus, again from Equations (7.40) and (7.41), we see that in this case E[T ] = 208,333.33, Var(T ) = 4.34 × 1010 , Var(T )/E 2 [T ] = 0.99994
7.9.
Applications to Patterns
465
Case 2: The Overlap Is of Size k In this case, T = Ti1 ,...,ik + T ∗ where Ti1 ,...,ik is the time until the pattern i1 , . . . , ik appears and T ∗ , distributed as an interarrival time of the renewal process, is the additional time that it takes, starting with i1 , . . . , ik , to obtain the pattern i1 , . . . , ir . Because these random variables are independent, we have E[T ] = E[Ti1 ,...,ik ] + E[T ∗ ]
(7.42) ∗
Var(T ) = Var(Ti1 ,...,ik ) + Var(T )
(7.43)
Now, from Equation (7.38) E[T ∗ ] = μ = p −1
(7.44)
Also, since no two renewals can occur within a distance r − k − 1 of each other, it follows that I (r)I (r + j ) = 0 if 1 j r − k − 1. Therefore, from Equation (7.39), we see that r−1 ∗ 2 −2 −3 2 Var(T ) = σ = p (1 − p) + 2p E[I (r)I (r + j )] − (r − 1)p j =r−k
= p −2 − (2r − 1)p −1 + 2p −3
r−1
E[I (r)I (r + j )]
(7.45)
j =r−k
The quantities E[I (r)I (r + j )] in Equation (7.45) can be calculated by considering the particular pattern. To complete the calculation of the first two moments of T , we then compute the mean and variance of Ti1 ,...,ik by repeating the same method.
Example 7.31 Suppose that we want to determine the number of flips of a fair coin until the pattern h, h, t, h, h occurs. For this pattern, r = 5, p = the overlap parameter is k = 2. Because E[I (5)I (8)] = P {h, h, t, h, h, t, h, h} =
1 256
E[I (5)I (9)] = P {h, h, t, h, h, h, t, h, h} =
1 512
we see from Equations (7.44) and (7.45) that E[T ∗ ] = 32 Var(T ∗ ) = (32)2 − 9(32) + 2(32)3
1 256
+
1 512
= 1120
1 32 ,
and
466
7
Renewal Theory and Its Applications
Hence, from Equations (7.42) and (7.43) we obtain E[T ] = E[Th,h ] + 32,
Var(T ) = Var(Th,h ) + 1120
Now, consider the pattern h, h. It has r = 2, p = 14 , and overlap parameter 1. Since, for this pattern, E[I (2)I (3)] = 18 , we obtain, as in the preceding, that E[Th,h ] = E[Th ] +4, Var(Th,h ) = Var(Th ) + 16 − 3(4) + 2 64 8 = Var(Th ) + 20 Finally, for the pattern h, which has r = 1, p = 12 , we see from Equations (7.40) and (7.41) that E[Th ] = 2,
Var(Th ) = 2
Putting it all together gives E[T ] = 38,
Var(T ) = 1142,
Var(T )/E 2 [T ] = 0.79086
Example 7.32 Suppose that P {Xn = i} = pi , and consider the pattern 0, 1, 2, 0, 1, 3, 0, 1. Then p = p03 p13 p2 p3 , r = 8, and the overlap parameter is k = 2. Since E[I (8)I (14)] = p05 p15 p22 p32 E[I (8)I (15)] = 0 we see from Equations (7.42) and (7.44) that E[T ] = E[T0,1 ] + p −1 and from Equations (7.43) and (7.45) that Var(T ) = Var(T0,1 ) + p −2 − 15p −1 + 2p −1 (p0 p1 )−1 Now, the r and p values of the pattern 0, 1 are r(0, 1) = 2, p(0, 1) = p0 p1 , and this pattern has overlap 0. Hence, from Equations (7.40) and (7.41), E[T0,1 ] = (p0 p1 )−1 ,
Var(T0,1 ) = (p0 p1 )−2 − 3(p0 p1 )−1
For instance, if pi = 0.2, i = 0, 1, 2, 3 then E[T ] = 25 + 58 = 390,650 Var(T ) = 625 − 75 + 516 + 35 × 58 = 1.526 × 1011 Var(T )/E 2 [T ] = 0.99996
7.9.
Applications to Patterns
467
Remark It can be shown that T is a type of discrete random variable called new better than used (NBU), which loosely means that if the pattern has not yet occurred by some time n then the additional time until it occurs tends to be less than the time it would take the pattern to occur if one started all over at that point. Such a random variable is known to satisfy (see Proposition 9.6.1 of Ref. [4]) Var(T ) E 2 [T ] − E[T ] E 2 [T ]
Now, suppose that there are s patterns, A(1), . . . , A(s) and that we are interested in the mean time until one of these patterns occurs, as well as the probability mass function of the one that occurs first. Let us assume, without any loss of generality, that none of the patterns is contained in any of the others. [That is, we rule out such trivial cases as A(1) = h, h and A(2) = h, h, t.] To determine the quantities of interest, let T (i) denote the time until pattern A(i) occurs, i = 1, . . . , s, and let T (i, j ) denote the additional time, starting with the occurrence of pattern A(i), until pattern A(j ) occurs, i = j . Start by computing the expected values of these random variables. We have already shown how to compute E[T (i)], i = 1, . . . , s. To compute E[T (i, j )], use the same approach, taking into account any “overlap” between the latter part of A(i) and the beginning part of A(j ). For instance, suppose A(1) = 0, 0, 1, 2, 0, 3, and A(2) = 2, 0, 3, 2, 0. Then T (2) = T2,0,3 + T (1, 2) where T2,0,3 is the time to obtain the pattern 2, 0, 3. Hence, E[T (1, 2)] = E[T (2)] − E[T2,0,3 ] −1 + (p0 p2 )−1 − (p2 p0 p3 )−1 = p22 p02 p3 So, suppose now that all of the quantities E[T (i)] and E[T (i, j )] have been computed. Let M = min T (i) i
and let P (i) = P {M = T (i)},
i = 1, . . . , s
That is, P (i) is the probability that pattern A(i) is the first pattern to occur. Now, for each j we will derive an equation that E[T (j )] satisfies as follows: E[T (j )] = E[M] + E[T (j ) − M] = E[M] + E[T (i, j )]P (i), i:i=j
j = 1, . . . , s
(7.46)
468
7
Renewal Theory and Its Applications
where the final equality is obtained by conditioning on which pattern occurs first. But the Equations (7.46) along with the equation s
P (i) = 1
i=1
constitute a set of s + 1 equations in the s + 1 unknowns E[M], P (i), i = 1, . . . , s. Solving them yields the desired quantities.
Example 7.33 Suppose that we continually flip a fair coin. With A(1) = h, t, t, h, h and A(2) = h, h, t, h, t, we have E[T (1)] = 32 + E[Th ] = 34 E[T (2)] = 32 E[T (1, 2)] = E[T (2)] − E[Th,h ] = 32 − (4 + E[Th ]) = 26 E[T (2, 1)] = E[T (1)] − E[Th,t ] = 34 − 4 = 30 Hence, we need solve the equations 34 = E[M] + 30P (2) 32 = E[M] + 26P (1) 1 = P (1) + P (2) These equations are easily solved, and yield the values P (1) = P (2) = 12 ,
E[M] = 19
Note that although the mean time for pattern A(2) is less than that for A(1), each has the same chance of occurring first. Equations (7.46) are easy to solve when there are no overlaps in any of the patterns. In this case, for all i = j E[T (i, j )] = E[T (j )] so Equations (7.46) reduce to E[T (j )] = E[M] + (1 − P (j ))E[T (j )] or P (j ) = E[M]/E[T (j )]
7.9.
Applications to Patterns
469
Summing the preceding over all j yields 1 j =1 1/E[T (j )]
(7.47)
1/E[T (j )] P (j ) = s j =1 1/E[T (j )]
(7.48)
E[M] = s
In our next example we use the preceding to reanalyze the model of Example 7.8.
Example 7.34 Suppose that each play of a game is, independently of the
outcomes of previous plays, won by player i with probability pi , i = 1, . . . , s. Suppose further that there are specified numbers n(1), . . . , n(s) such that the first player i to win n(i) consecutive plays is declared the winner of the match. Find the expected number of plays until there is a winner, and also the probability that the winner is i, i = 1, . . . , s. Solution: Letting A(i), for i = 1, . . . , s, denote the pattern of ni consecutive values of i, this problem asks for P (i), the probability that pattern A(i) occurs first, and for E[M]. Because E[T (i)] = (1/pi )n(i) + (1/pi )n(i)−1 + · · · + 1/pi =
1 − pin(i) n(i)
pi
(1 − pi )
we obtain, from Equations (7.47) and (7.48), that 1 " n(j ) n(j ) # (1 − pj )/ 1 − pj j =1 pj
E[M] = s
pin(i) (1 − pi )/ 1 − pin(i) P (i) = s " n(j ) n(j ) # (1 − pj )/ 1 − pj j =1 pj
7.9.2. The Expected Time to a Maximal Run of Distinct Values Let Xi , i 1, be independent and identically distributed random variables that are equally likely to take on any of the values 1, 2, . . . , m. Suppose that these random variables are observed sequentially, and let T denote the first time that a run of m consecutive values includes all the values 1, . . . , m. That is, T = min{n : Xn−m+1 , . . . , Xn are all distinct}
470
7
Renewal Theory and Its Applications
To compute E[T ], define a renewal process by letting the first renewal occur at time T . At this point start over and, without using any of the data values up to T , let the next renewal occur the next time a run of m consecutive values are all distinct, and so on. For instance, if m = 3 and the data are 1, 3, 3, 2, 1, 2, 3, 2, 1, 3, . . . ,
(7.49)
then there are two renewals by time 10, with the renewals occurring at times 5 and 9. We call the sequence of m distinct values that constitutes a renewal a renewal run. Let us now transform the renewal process into a delayed renewal reward process by supposing that a reward of 1 is earned at time n, n m, if the values Xn−m+1 , . . . , Xn are all distinct. That is, a reward is earned each time the previous m data values are all distinct. For instance, if m = 3 and the data values are as in (7.49) then unit rewards are earned at times 5, 7, 9, and 10. If we let Ri denote the reward earned at time i, then by Proposition 7.3, lim n
E
"n
i=1 Ri
n
# =
E[R] E[T ]
(7.50)
where R is the reward earned between renewal epochs. Now, with Ai equal to the set of the first i data values of a renewal run, and Bi to the set of the first i values following this renewal run, we have the following: E[R] = 1 +
m−1
E[reward earned a time i after a renewal]
i=1
=1+
m−1
P {Ai = Bi }
i=1
=1+
m−1 i=1
=
m−1 i=0
i! mi
i! mi
(7.51)
Hence, since for i m E[Ri ] = P {Xi−m+1 , . . . , Xi are all distinct} =
m! mm
7.9.
Applications to Patterns
471
it follows from Equation (7.50) that E[R] m! = mm E[T ] Thus from Equation (7.51) we obtain E[T ] =
m−1 mm i!/mi m! i=0
The preceding delayed renewal reward process approach also gives us another way of computing the expected time until a specified pattern appears. We illustrate by the following example.
Example 7.35 Compute E[T ], the expected time until the pattern h, h, h, t, h, h, h appears, when a coin that comes up heads with probability p and tails with probability q = 1 − p is continually flipped. Solution: Define a renewal process by letting the first renewal occur when the pattern first appears, and then start over. Also, say that a reward of 1 is earned whenever the pattern appears. If R is the reward earned between renewal epochs, we have E[R] = 1 +
6
E[reward earned i units after a renewal]
i=1
= 1 + 0 + 0 + 0 + p 3 q + p 3 qp + p 3 qp 2 Hence, since the expected reward earned at time i is E[Ri ] = p 6 q, we obtain the following from the renewal reward theorem: 1 + qp 3 + qp 4 + qp 5 = qp 6 E[T ] or E[T ] = q −1 p −6 + p −3 + p −2 + p −1
7.9.3. Increasing Runs of Continuous Random Variables Let X1 , X2 , . . . be a sequence of independent and identically distributed continuous random variables, and let T denote the first time that there is a string of r
472
7
Renewal Theory and Its Applications
consecutive increasing values. That is, T = min{n r : Xn−r+1 < Xn−r+2 < · · · < Xn } To compute E[T ], define a renewal process as follows. Let the first renewal occur at T . Then, using only the data values after T , say that the next renewal occurs when there is again a string of r consecutive increasing values, and continue in this fashion. For instance, if r = 3 and the first 15 data values are 12, 20, 22, 28, 43, 18, 24, 33, 60, 4, 16, 8, 12, 15, 18 then 3 renewals would have occurred by time 15, namely, at times 3, 8, and 14. If we let N (n) denote the number of renewals by time n, then by the elementary renewal theorem E[N (n)] 1 → n E[T ] To compute E[N (n)], define a stochastic process whose state at time k, call it Sk , is equal to the number of consecutive increasing values at time k. That is, for 1 j k Sk = j
if Xk−j > Xk−j +1 < · · · < Xk−1 < Xk
where X0 = ∞. Note that a renewal will occur at time k if and only if Sk = ir for some i 1. For instance, if r = 3 and X5 > X6 < X7 < X8 < X9 < X10 < X11 then S6 = 1,
S7 = 2,
S8 = 3,
S9 = 4,
S10 = 5,
S11 = 6
and renewals occur at times 8 and 11. Now, for k > j P {Sk = j } = P {Xk−j > Xk−j +1 < · · · < Xk−1 < Xk } = P {Xk−j +1 < · · · < Xk−1 < Xk } − P {Xk−j < Xk−j +1 < · · · < Xk−1 < Xk } =
1 1 − j ! (j + 1)!
=
j (j + 1)!
where the next to last equality follows since all possible orderings of the random variables are equally likely.
7.10.
The Insurance Ruin Problem
473
From the preceding, we see that lim P {a renewal occurs at time k}= lim
k→∞
k→∞
∞
P {Sk = ir} =
i=1
∞ i=1
ir (ir + 1)!
However, E[N (n)] =
n
P {a renewal occurs at time k}
k=1
Because we can show that for any numbers ak , k 1, for which limk→∞ ak exists n k=1 ak = lim ak lim n→∞ k→∞ n we obtain from the preceding, upon using the elementary renewal theorem, 1 i=1 ir/(ir + 1)!
E[T ] = ∞
7.10.
The Insurance Ruin Problem
Suppose that claims are made to an insurance firm according to a Poisson process with rate λ, and that the successive claim amounts Y1 , Y2 , . . . are independent random variables having a common distribution function F with density f (x). Suppose also that the claim amounts are independent of the claim arrival times. M(t) Thus, if we let M(t) be the number of claims made by time t, then i=1 Yi is the total amount paid out in claims by time t. Supposing that the firm starts with an initial capital x and receives income at a constant rate c per unit time, we are interested in the probability that the firm’s net capital ever becomes negative; that is, we are interested in M(t) Yi > x + ct for some t 0 R(x) = P i=1
If the firm’s capital ever becomes negative, we say that the firm is ruined; thus R(x) is the probability of ruin given that the firm begins with an initial capital x. Let μ = E[Yi ] be the mean claim amount, and let ρ = λμ/c. Because claims occur at rate λ, the long run rate at which money is paid out is λμ. (A formal argument uses renewal reward processes. A new cycle begins when a claim occurs; the cost for the cycle is the claim amount, and so the long run average cost
474
7
Renewal Theory and Its Applications
is μ, the expected cost incurred in a cycle, divided by 1/λ, the mean cycle time.) Because the rate at which money is received is c, it is clear that R(x) = 1 when ρ > 1. As R(x) can be shown to also equal 1 when ρ = 1 (think of the recurrence of the symmetric random walk), we will suppose that ρ < 1. To determine R(x), we start by deriving a differential equation. To begin, consider what can happen in the first h time units, where h is small. With probability 1 − λh + o(h) there will be no claims and the firm’s capital at time h will be x + ch; with probability λh + o(h) there will be exactly one claim and the firm’s capital at time h will be x + ch − Y1 ; with probability o(h) there will be two or more claims. Therefore, conditioning on what happens during the first h time units yields that R(x) = (1 − λh)R(x + ch) + λh E[R(x + ch − Y1 )] + o(h) Equivalently, R(x + ch) − R(x) = λh R(x + ch) − λh E[R(x + ch − Y1 )] + o(h) Dividing through by ch gives R(x + ch) − R(x) λ λ 1 o(h) = R(x + ch) − E[R(x + ch − Y1 )] + ch c c c h Letting h go to 0 yields the differential equation R (x) =
λ λ R(x) − E[R(x − Y1 )] c c
Because R(u) = 1 when u < 0, the preceding can be written as λ x λ λ ∞ R (x) = R(x) − R(x − y)f (y) dy − f (y) dy c c 0 c x or, equivalently, R (x) =
λ λ R(x) − c c
x 0
R(x − y)f (y) dy −
λ ¯ F (x) c
(7.52)
where F¯ (x) = 1 − F (x). We will now use the preceding equation to show that R(x) also satisfies the equation λ x ¯ λ x R(x − y)F¯ (y) dy − F (y) dy, x 0 (7.53) R(x) = R(0) + c 0 c 0 To verify Equation (7.53), we will show that differentiating both sides of it results in Equation (7.52). [It can be shown that both (7.52) and (7.53) have unique solutions.] To do so, we will need the following lemma, whose proof is given at the end of this section.
7.10.
The Insurance Ruin Problem
475
Lemma 7.5 For a function k, and a differentiable function t d dx
x
x
t (x − y)k(y) dy = t (0)k(x) +
0
t (x − y)k(y) dy
0
Differentiating both sides of Equation (7.53) gives, upon using the preceding lemma, / . x λ ¯ ¯ ¯ R (x − y)F (y) dy − F (x) (7.54) R (x) = R(0)F (x) + c 0 Differentiation by parts [u = F¯ (y), dv = R (x − y) dy] shows that
x 0
R (x − y)F¯ (y) dy = −F¯ (y)R(x − y)|x0 −
= −F¯ (x)R(0) + R(x) −
x
R(x − y)f (y) dy
0
x
R(x − y)f (y) dy
0
Substituting this result back in Equation (7.54) gives Equation (7.52). Thus, we have established Equation (7.53). To obtain a more usable expression for R(x), consider a renewal process whose interarrival times X1 , X2 , . . . are distributed according to the equilibrium distribution of F . That is, the density function of the Xi is fe (x) = Fe (x) =
F¯ (x) μ
Let N(t) denote the number of renewals by time t, and let us derive an expression for # " q(x) = E ρ N (x)+1 Conditioning on X1 gives that q(x) = 0
∞
" # F¯ (y) dy E ρ N (x)+1 |X1 = y μ
Because, given that X1 = y, the number of renewals by time x is distributed as 1 + N(x − y) when y x, or is identically 0 when y > x, we see that E[ρ
N (x)+1
# " ρ E ρ N (x−y)+1 , |X1 = y] = ρ,
if y x if y > x
476
7
Renewal Theory and Its Applications
Therefore, q(x) satisfies
∞ ¯ F (y) F¯ (y) dy + ρ dy μ μ 0 x . ∞ / x λ λ x ¯ ¯ ¯ q(x − y) F (y) dy + = F (y) dy − F (y) dy c 0 c 0 0 λ x ¯ λ x ¯ q(x − y) F (y) dy + ρ − F (y) dy = c 0 c 0
q(x) =
x
ρq(x − y)
Because q(0) = ρ, this is exactly the same equation that is satisfied by R(x), namely Equation (7.53). Therefore, because the solution to (7.53) is unique, we obtain the following.
Proposition 7.6 # " R(x) = q(x) = E ρ N (x)+1
Example 7.36 Suppose that the firm does not start with any initial cap-
ital. Then, because N (0) = 0, we see that the firm’s probability of ruin is R(0) = ρ.
Example 7.37 If the claim distribution F is exponential with mean μ, then so is Fe . Hence, N (x) is Poisson with mean x/μ, giving the result ∞ " # R(x) = E ρ N (x)+1 = ρ n+1 e−x/μ (x/μ)n /n! n=0
=ρe
−x/μ
∞
(ρ x/μ)n /n!
n=0
= ρe
−x(1−ρ)/μ
To obtain some intuition about the ruin probability, let T be independent of the interarrival times Xi of the renewal process having interarrival distribution Fe , and let T have probability mass function P {T = n} = ρ n (1 − ρ),
n = 0, 1, . . .
Now consider P { Ti=1 Xi > x}, the probability that the sum of the first T of the Xi exceeds x. Because N (x) + 1 is the first renewal that occurs after time x, we
7.10.
The Insurance Ruin Problem
477
have N (x) + 1 = min n :
n
Xi > x
i=1
Therefore, conditioning on the number of renewals by time x gives that P
T
Xi > x =
∞ j =0
i=1
=
∞
P
T
! ! Xi > x !N (x) = j P {N (x) = j }
i=1
P {T j + 1|N (x) = j }P {N (x) = j }
j =0
=
∞
P {T j + 1}P {N (x) = j }
j =0
=
∞
ρ j +1 P {N (x) = j }
j =0
" # = E ρ N (x)+1 Consequently, P { Ti=1 Xi > x} is equal to the ruin probability. Now, as noted in Example 7.36, the ruin probability of a firm starting with 0 initial capital is ρ. Suppose that the firm starts with an initial capital x, and suppose for the moment that it is allowed to remain in business even if its capital becomes negative. Because the probability that the firm’s capital ever falls below its initial starting amount x is the same as the probability that its capital ever becomes negative when it starts with 0, this probability is also ρ. Thus if we say that a low occurs whenever the firm’s capital becomes lower than it has ever previously been, then the probability that a low ever occurs is ρ. Now, if a low does occur, then the probability that there will be another low is the probability that the firm’s capital will ever fall below its previous low, and clearly this is also ρ. Therefore, each new low is the final one with probability 1 − ρ. Consequently, the total number of lows that ever occur has the same distribution as T . In addition, if we let Wi be the amount by which the ith low is less than the low preceding it, it is easy to see that W1 , W2 , . . . are independent and identically distributed, and are also independent of the number of lows. Because the minimal value over all time of the firm’s capital (when it is allowed to remain in business even when its capital becomes negative) is x − Ti=1 Wi , it follows that the ruin probability of a firm
478
7
Renewal Theory and Its Applications
that starts with an initial capital x is R(x) = P
T
Wi > x
i=1
Because T " N (x)+1 # R(x) = E ρ =P Xi > x i=1
we can identify Wi with Xi . That is, we can conclude that each new low is lower than its predecessor by a random amount whose distribution is the equilibrium distribution of a claim amount. Let us now give the proof of Lemma 7.5.
Proof of Lemma 7.5
Let G(x) =
x 0
t (x − y)k(y) dy. Then
G(x + h) − G(x) = G(x + h) −
x
t (x + h − y)k(y) dy
0
x
+
t (x + h − y)k(y) dy − G(x)
0
=
x+h
x
x
+
t (x + h − y)k(y) dy [t (x + h − y) − t (x − y)]k(y) dy
0
Dividing through by h gives G(x + h) − G(x) 1 = h h
x+h
t (x + h − y)k(y) dy
x
+
x
0
t (x + h − y) − t (x − y) k(y) dy h
Letting h → 0 gives the result: G (x) = t (0) k(x) +
0
x
t (x − y) k(y) dy
Exercises
479
Exercises 1.
Is it true that (a) N(t) < n if and only if Sn > t? (b) N(t) n if and only if Sn t? (c) N(t) > n if and only if Sn < t?
2. Suppose that the interarrival distribution for a renewal process is Poisson distributed with mean μ. That is, suppose P {Xn = k} = e−μ
μk , k!
k = 0, 1, . . .
(a) Find the distribution of Sn . (b) Calculate P {N (t) = n}. *3. If the mean-value function of the renewal process {N (t), t 0} is given by m(t) = t/2, t 0, what is P {N (5) = 0}? 4. Let {N1 (t), t 0} and {N2 (t), t 0} be independent renewal processes. Let N(t) = N1 (t) + N2 (t). (a) Are the interarrival times of {N (t), t 0} independent? (b) Are they identically distributed? (c) Is {N (t), t 0} a renewal process? 5. Let U1 , U2 , . . . be independent uniform (0, 1) random variables, and define N by N = min{n : U1 + U2 + · · · + Un > 1} What is E[N]? *6. Consider a renewal process {N (t), t 0} having a gamma (r, λ) interarrival distribution. That is, the interarrival density is f (x) =
λe−λx (λx)r−1 , (r − 1)!
x>0
(a) Show that P {N (t) n} =
∞ e−λt (λt)i i!
i=nr
480
7
Renewal Theory and Its Applications
(b) Show that m(t) =
∞ . / −λt i e (λt)i i=r
r
i!
where [i/r] is the largest integer less than or equal to i/r. Hint: Use the relationship between the gamma (r, λ) distribution and the sum of r independent exponentials with rate λ, to define N (t) in terms of a Poisson process with rate λ. 7. Mr. Smith works on a temporary basis. The mean length of each job he gets is three months. If the amount of time he spends between jobs is exponentially distributed with mean 2, then at what rate does Mr. Smith get new jobs? *8. A machine in use is replaced by a new machine either when it fails or when it reaches the age of T years. If the lifetimes of successive machines are independent with a common distribution F having density f , show that (a) the long-run rate at which machines are replaced equals .
T
/−1 xf (x) dx + T (1 − F (T ))
0
(b) the long-run rate at which machines in use fail equals T 0
F (T ) xf (x) dx + T [1 − F (T )]
9. A worker sequentially works on jobs. Each time a job is completed, a new one is begun. Each job, independently, takes a random amount of time having distribution F to complete. However, independently of this, shocks occur according to a Poisson process with rate λ. Whenever a shock occurs, the worker discontinues working on the present job and starts a new one. In the long run, at what rate are jobs completed? 10. Consider a renewal process with mean interarrival time μ. Suppose that each event of this process is independently “counted” with probability p. Let NC (t) denote the number of counted events by time t, t > 0. (a) Is NC (t), t 0 a renewal process? (b) What is limt→∞ NC (t)/t?
Exercises
481
11. A renewal process for which the time until the initial renewal has a different distribution than the remaining interarrival times is called a delayed (or a general) renewal process. Prove that Proposition 7.1 remains valid for a delayed renewal process. (In general, it can be shown that all of the limit theorems for a renewal process remain valid for a delayed renewal process provided that the time until the first renewal has a finite mean.) 12. Events occur according to a Poisson process with rate λ. Any event that occurs within a time d of the event that immediately preceded it is called a d-event. For instance, if d = 1 and events occur at times 2, 2.8, 4, 6, 6.6, . . . , then the events at times 2.8 and 6.6 would be d-events. (a) At what rate do d-events occur? (b) What proportion of all events are d-events? 13. Let X1 , X2 , . . . be a sequence of independent random variables. The nonnegative integer valued random variable N is said to be a stopping time for the sequence if the event {N = n} is independent of Xn+1 , Xn+2 , . . . . The idea being that the Xi are observed one at a time—first X1 , then X2 , and so on—and N represents the number observed when we stop. Hence, the event {N = n} corresponds to stopping after having observed X1 , . . . , Xn and thus must be independent of the values of random variables yet to come, namely, Xn+1 , Xn+2 , . . . . (a) Let X1 , X2 , . . . be independent with P {Xi = 1} = p = 1 − P {Xi = 0},
i1
Define N1 = min{n : 3, N2 = 5, 3, N3 = 2,
X1 + · · · + Xn = 5} if X1 = 0 if X1 = 1 if X4 = 0 if X4 = 1
Which of the Ni are stopping times for the sequence X1 , . . .? An important result, known as Wald’s equation states that if X1 , X2 , . . . are independent and identically distributed and have a finite mean E(X), and if N is a stopping time for this sequence having a finite mean, then ,N E Xi = E[N ]E[X] i=1
To prove Wald’s equation, let us define the indicator variables Ii , i 1 by 1, if i N Ii = 0, if i > N
482
7
Renewal Theory and Its Applications
(b) Show that N
Xi =
∞
i=1
Xi Ii
i=1
From part (b) we see that , E
N
-
, =E
Xi
i=1
∞
Xi Ii
i=1
=
∞
E[Xi Ii ]
i=1
where the last equality assumes that the expectation can be brought inside the summation (as indeed can be rigorously proven in this case). (c) Argue that Xi and Ii are independent. Hint: Ii equals 0 or 1 depending on whether or not we have yet stopped after observing which random variables? (d) From part (c) we have , E
N
Xi =
i=1
∞
E[X]E[Ii ]
i=1
Complete the proof of Wald’s equation. (e) What does Wald’s equation tell us about the stopping times in part (a)? 14. Wald’s equation can be used as the basis of a proof of the elementary renewal theorem. Let X1 , X2 , . . . denote the interarrival times of a renewal process and let N(t) be the number of renewals by time t. (a) Show that whereas N (t) is not a stopping time, N (t) + 1 is. Hint: Note that N (t) = n ⇔ X1 + · · · + Xn t
and X1 + · · · + Xn+1 > t
(b) Argue that E
,N (t)+1 i=1
Xi = μ[m(t) + 1]
Exercises
483
(c) Suppose that the Xi are bounded random variables. That is, suppose there is a constant M such that P {Xi < M} = 1. Argue that t
0}, then N2 = min{n: XN1 +n > 0}.) Similarly, define a stopping time N3 on the sequence XN1 +N2 +1 , XN1 +N2 +2 , . . . that is identically defined on this sequence as is N1 on the original sequence, and so on.
Exercises
485
(b) Is the reward process in which Xi is the reward earned during period i a renewal reward process? If so, what is the length of the successive cycles? (c) Derive an expression for the average reward per unit time. (d) Use the strong law of large numbers to derive a second expression for the average reward per unit time. (e) Conclude Wald’s equation. 25. Suppose in Example 7.13 that the arrival process is a Poisson process and suppose that the policy employed is to dispatch the train every t time units. (a) Determine the average cost per unit time. (b) Show that the minimal average cost per unit time for such a policy is approximately c/2 plus the average cost per unit time for the best policy of the type considered in that example. 26. Consider a train station to which customers arrive in accordance with a Poisson process having rate λ. A train is summoned whenever there are N customers waiting in the station, but it takes K units of time for the train to arrive at the station. When it arrives, it picks up all waiting customers. Assuming that the train station incurs a cost at a rate of nc per unit time whenever there are n customers present, find the long-run average cost. 27. A machine consists of two independent components, the ith of which functions for an exponential time with rate λi . The machine functions as long as at least one of these components function. (That is, it fails when both components have failed.) When a machine fails, a new machine having both its components working is put into use. A cost K is incurred whenever a machine failure occurs; operating costs at rate ci per unit time are incurred whenever the machine in use has i working components, i = 1, 2. Find the long run average cost per unit time. 28. In Example 7.15, what proportion of the defective items produced is discovered? 29. Consider a single-server queueing system in which customers arrive in accordance with a renewal process. Each customer brings in a random amount of work, chosen independently according to the distribution G. The server serves one customer at a time. However, the server processes work at rate i per unit time whenever there are i customers in the system. For instance, if a customer with workload 8 enters service when there are three other customers waiting in line, then if no one else arrives that customer will spend 2 units of time in service. If another customer arrives after 1 unit of time, then our customer will spend a total of 1.8 units of time in service provided no one else arrives. Let Wi denote the amount of time customer i spends in the system. Also, define E[W ] by E[W ] = lim (W1 + · · · + Wn )/n n→∞
and so E[W ] is the average amount of time a customer spends in the system.
486
7
Renewal Theory and Its Applications
Let N denote the number of customers that arrive in a busy period. (a) Argue that E[W ] = E[W1 + · · · + WN ]/E[N ] Let Li denote the amount of work customer i brings into the system; and so the Li , i 1, are independent random variables having distribution G. (b) Argue that at any time t, the sum of the times spent in the system by all arrivals prior to t is equal to the total amount of work processed by time t. Hint: Consider the rate at which the server processes work. (c) Argue that N
Wi =
i=1
N
Li
i=1
(d) Use Wald’s equation (see Exercise 13) to conclude that E[W ] = μ where μ is the mean of the distribution G. That is, the average time that customers spend in the system is equal to the average work they bring to the system. *30. For a renewal process, let A(t) be the age at time t. Prove that if μ < ∞, then with probability 1 A(t) →0 t
as t → ∞
31. If A(t) and Y (t) are, respectively, the age and the excess at time t of a renewal process having an interarrival distribution F , calculate P {Y (t) > x|A(t) = s} 32. Determine the long-run proportion of time that XN (t)+1 < c. 33. In Example 7.14, find the long run proportion of time that the server is busy. 34. An M/G/∞ queueing system is cleaned at the fixed times T , 2T , 3T , . . . . All customers in service when a cleaning begins are forced to leave early and a cost C1 is incurred for each customer. Suppose that a cleaning takes time T /4, and that all customers who arrive while the system is being cleaned are lost, and a cost C2 is incurred for each one.
Exercises
487
(a) Find the long run average cost per unit time. (b) Find the long run proportion of time the system is being cleaned. *35. Satellites are launched according to a Poisson process with rate λ. Each satellite will, independently, orbit the earth for a random time having distribution F . Let X(t) denote the number of satellites orbiting at time t. (a) Determine P {X(t) = k}. Hint: Relate this to the M/G/∞ queue. (b) If at least one satellite is orbiting, then messages can be transmitted and we say that the system is functional. If the first satellite is orbited at time t = 0, determine the expected time that the system remains functional. Hint: Make use of part (a) when k = 0. 36. Each of n skiers continually, and independently, climbs up and then skis down a particular slope. The time it takes skier i to climb up has distribution Fi , and it is independent of her time to ski down, which has distribution Hi , i = 1, . . . , n. Let N (t) denote the total number of times members of this group have skied down the slope by time t. Also, let U (t) denote the number of skiers climbing up the hill at time t. (a) What is limt→∞ N (t)/t? (b) Find limt→∞ E[U (t)]. (c) If all Fi are exponential with rate λ and all Gi are exponential with rate μ, what is P {U (t) = k}? 37. There are three machines, all of which are needed for a system to work. Machine i functions for an exponential time with rate λi before it fails, i = 1, 2, 3. When a machine fails, the system is shut down and repair begins on the failed machine. The time to fix machine 1 is exponential with rate 5; the time to fix machine 2 is uniform on (0, 4); and the time to fix machine 3 is a gamma random variable with parameters n = 3 and λ = 2. Once a failed machine is repaired, it is as good as new and all machines are restarted. (a) What proportion of time is the system working? (b) What proportion of time is machine 1 being repaired? (c) What proportion of time is machine 2 in a state of suspended animation (that is, neither working nor being repaired)? 38. A truck driver regularly drives round trips from A to B and then back to A. Each time he drives from A to B, he drives at a fixed speed that (in miles per hour) is uniformly distributed between 40 and 60; each time he drives from B to A, he drives at a fixed speed that is equally likely to be either 40 or 60.
488
7
Renewal Theory and Its Applications
(a) In the long run, what proportion of his driving time is spent going to B? (b) In the long run, for what proportion of his driving time is he driving at a speed of 40 miles per hour? 39. A system consists of two independent machines that each functions for an exponential time with rate λ. There is a single repairperson. If the repairperson is idle when a machine fails, then repair immediately begins on that machine; if the repairperson is busy when a machine fails, then that machine must wait until the other machine has been repaired. All repair times are independent with distribution function G and, once repaired, a machine is as good as new. What proportion of time is the repairperson idle? 40. Three marksmen take turns shooting at a target. Marksman 1 shoots until he misses, then marksman 2 begins shooting until he misses, then marksman 3 until he misses, and then back to marksman 1, and so on. Each time marksman i fires he hits the target, independently of the past, with probability Pi , i = 1, 2, 3. Determine the proportion of time, in the long run, that each marksman shoots. 41. Each time a certain machine breaks down it is replaced by a new one of the same type. In the long run, what percentage of time is the machine in use less than one year old if the life distribution of a machine is (a) uniformly distributed over (0, 2)? (b) exponentially distributed with mean 1? *42. For an interarrival distribution F having mean μ, we defined the equilibrium distribution of F , denoted Fe , by Fe (x) =
1 μ
x
[1 − F (y)] dy
0
(a) Show that if F is an exponential distribution, then F = Fe . (b) If for some constant c, F (x) =
0, 1,
x0
That is, interarrivals are equally likely to be exponential with mean 1 or exponential with mean 2. (a) Without any calculations, guess the equilibrium distribution Fe . (b) Verify your guess in part (a). 44. An airport shuttle bus picks up all passengers waiting at a bus stop and drops them off at the airport terminal; it then returns to the stop and repeats the process. The times between returns to the stop are independent random variables with distribution F , mean μ, and variance σ 2 . Passengers arrive at the bus stop in accordance with a Poisson process with rate λ. Suppose the bus has just left the stop, and let X denote the number of passengers it picks up when it returns. (a) Find E[X]. (b) Find Var(X). (c) At what rate does the shuttle bus arrive at the terminal without any passengers? Suppose that each passenger that has to wait at the bus stop more than c time units writes an angry letter to the shuttle bus manager. (d) What proportion of passengers write angry letters? (e) How does your answer in part (d) relate to Fe (x)? 45. Consider a system that can be in either state 1 or 2 or 3. Each time the system enters state i it remains there for a random amount of time having mean μi and then makes a transition into state j with probability Pij . Suppose P12 = 1,
P21 = P23 = 12 ,
P31 = 1
(a) What proportion of transitions takes the system into state 1? (b) If μ1 = 1, μ2 = 2, μ3 = 3, then what proportion of time does the system spend in each state? 46. Consider a semi-Markov process in which the amount of time that the process spends in each state before making a transition into a different state is exponentially distributed. What kind of process is this? 47. In a semi-Markov process, let tij denote the conditional expected time that the process spends in state i given that the next state is j .
490
7
Renewal Theory and Its Applications
(a) Present an equation relating μi to the tij . (b) Show that the proportion of time the process is in i and will next enter j is equal to Pi Pij tij /μi . Hint: Say that a cycle begins each time state i is entered. Imagine that you receive a reward at a rate of 1 per unit time whenever the process is in i and heading for j . What is the average reward per unit time? 48. A taxi alternates between three different locations. Whenever it reaches location i, it stops and spends a random time having mean ti before obtaining another passenger, i = 1, 2, 3. A passenger entering the cab at location i will want to go to location j with probability Pij . The time to travel from i to j is a random variable with mean mij . Suppose that t1 = 1, t2 = 2, t3 = 4, P12 = 1, P23 = 1, P31 = 23 = 1 − P32 , m12 = 10, m23 = 20, m31 = 15, m32 = 25. Define an appropriate semi-Markov process and determine (a) the proportion of time the taxi is waiting at location i, and (b) the proportion of time the taxi is on the road from i to j , i, j = 1, 2, 3. *49. Consider a renewal process having the gamma (n, λ) interarrival distribution, and let Y (t) denote the time from t until the next renewal. Use the theory of semi-Markov processes to show that 1 Gi,λ (x) n n
lim P {Y (t) < x} =
t→∞
i=1
where Gi,λ (x) is the gamma (i, λ) distribution function. 50. To prove Equation (7.24), define the following notation: j
Xi ≡ time spent in state i on the j th visit to this state; Ni (m) ≡ number of visits to state i in the first m transitions In terms of this notation, write expressions for (a) the amount of time during the first m transitions that the process is in state i; (b) the proportion of time during the first m transitions that the process is in state i. Argue that, with probability 1, (c)
N i (m) j =1
j
Xi → μi Ni (m)
as m → ∞
(d) Ni (m)/m → πi as m → ∞. (e) Combine parts (a), (b), (c), and (d) to prove Equation (7.24).
Exercises
491
51. In 1984 the country of Morocco in an attempt to determine the average amount of time that tourists spend in that country on a visit tried two different sampling procedures. In one, they questioned randomly chosen tourists as they were leaving the country; in the other, they questioned randomly chosen guests at hotels. (Each tourist stayed at a hotel.) The average visiting time of the 3000 tourists chosen from hotels was 17.8, whereas the average visiting time of the 12,321 tourists questioned at departure was 9.0. Can you explain this discrepancy? Does it necessarily imply a mistake? 52. Let Xi , i = 1, 2, . . . , be the interarrival times of the renewal process {N (t)}, and let Y , independent of the Xi , be exponential with rate λ. (a) Use the lack of memory property of the exponential to argue that P {X1 + · · · + Xn < Y } = (P {X < Y })n (b) Use part (a) to show that E[N (Y )] =
E[e−λX ] 1 − E[e−λX ]
where X has the interarrival distribution. 53. Write a program to approximate m(t) for the interarrival distribution F ∗ G, where F is exponential with mean 1 and G is exponential with mean 3. 54. Let Xi , i 1, be independent random variables with pj = P {X = j }, j 1. If pj = j/10, j = 1, 2, 3, 4, find the expected time and the variance of the number of variables that need be observed until the pattern 1, 2, 3, 1, 2 occurs. 55. A coin that comes up heads with probability 0.6 is continually flipped. Find the expected number of flips until either the sequence thht or the sequence ttt occurs, and find the probability that ttt occurs first. 56. Random digits, each of which is equally likely to be any of the digits 0 through 9, are observed in sequence. (a) Find the expected time until a run of 10 distinct values occurs. (b) Find the expected time until a run of 5 distinct values occurs. 57. Let h(x) = P { Ti=1 Xi > x} where X1 , X2 , . . . are independent random variables having distribution function Fe and T is independent of the Xi and has probability mass function P {T = n} = ρ n (1 − ρ), n 0. Show that h(x) satisfies Equation (7.53). Hint: Start by conditioning on whether T = 0 or T > 0.
492
7
Renewal Theory and Its Applications
References The results in Section 7.9.1 concerning the computation of the variance of the time until a specified pattern appears are new, as are the results of Section 7.9.2. The results of Section 7.9.3 are from Ref. [3].
1. D. R. Cox, “Renewal Theory,” Methuen, London, 1962. 2. W. Feller, “An Introduction to Probability Theory and Its Applications,” Vol. II, John Wiley, New York, 1966. 3. F. Hwang and D. Trietsch, “A Simple Relation Between the Pattern Probability and the Rate of False Signals in Control Charts,” Probability in the Engineering and Informational Sciences, 10, 315–323, 1996. 4. S. Ross, “Stochastic Processes,” Second Edition, John Wiley, New York, 1996. 5. H. C. Tijms, “Stochastic Models, An Algorithmic Approach,” John Wiley, New York, 1994.
Queueing Theory
8 8.1.
Introduction
In this chapter we will study a class of models in which customers arrive in some random manner at a service facility. Upon arrival they are made to wait in queue until it is their turn to be served. Once served they are generally assumed to leave the system. For such models we will be interested in determining, among other things, such quantities as the average number of customers in the system (or in the queue) and the average time a customer spends in the system (or spends waiting in the queue). In Section 8.2 we derive a series of basic queueing identities which are of great use in analyzing queueing models. We also introduce three different sets of limiting probabilities which correspond to what an arrival sees, what a departure sees, and what an outside observer would see. In Section 8.3 we deal with queueing systems in which all of the defining probability distributions are assumed to be exponential. For instance, the simplest such model is to assume that customers arrive in accordance with a Poisson process (and thus the interarrival times are exponentially distributed) and are served one at a time by a single server who takes an exponentially distributed length of time for each service. These exponential queueing models are special examples of continuous-time Markov chains and so can be analyzed as in Chapter 6. However, at the cost of a (very) slight amount of repetition we shall not assume that you are familiar with the material of Chapter 6, but rather we shall redevelop any needed material. Specifically we shall derive anew (by a heuristic argument) the formula for the limiting probabilities. In Section 8.4 we consider models in which customers move randomly among a network of servers. The model of Section 8.4.1 is an open system in which customers are allowed to enter and depart the system, whereas the one studied
493
494
8
Queueing Theory
in Section 8.4.2 is closed in the sense that the set of customers in the system is constant over time. In Section 8.5 we study the model M/G/1, which while assuming Poisson arrivals, allows the service distribution to be arbitrary. To analyze this model we first introduce in Section 8.5.1 the concept of work, and then use this concept in Section 8.5.2 to help analyze this system. In Section 8.5.3 we derive the average amount of time that a server remains busy between idle periods. In Section 8.6 we consider some variations of the model M/G/1. In particular in Section 8.6.1 we suppose that bus loads of customers arrive according to a Poisson process and that each bus contains a random number of customers. In Section 8.6.2 we suppose that there are two different classes of customers—with type 1 customers receiving service priority over type 2. In Section 8.6.3 we present an M/G/1 optimization example. We suppose that the server goes on break whenever she becomes idle, and then determine, under certain cost assumptions, the optimal time for her to return to service. In Section 8.7 we consider a model with exponential service times but where the interarrival times between customers is allowed to have an arbitrary distribution. We analyze this model by use of an appropriately defined Markov chain. We also derive the mean length of a busy period and of an idle period for this model. In Section 8.8 we consider a single server system whose arrival process results from return visits of a finite number of possible sources. Assuming a general service distribution, we show how a Markov chain can be used to analyze this system. In the final section of the chapter we talk about multiserver systems. We start with loss systems, in which arrivals finding all servers busy, are assumed to depart and as such are lost to the system. This leads to the famous result known as Erlang’s loss formula, which presents a simple formula for the number of busy servers in such a model when the arrival process in Poisson and the service distribution is general. We then discuss multiserver systems in which queues are allowed. However, except in the case where exponential service times are assumed, there are very few explicit formulas for these models. We end by presenting an approximation for the average time a customer waits in queue in a k-server model which assumes Poisson arrivals but allows for a general service distribution.
8.2.
Preliminaries
In this section we will derive certain identities which are valid in the great majority of queueing models.
8.2.
Preliminaries
495
8.2.1. Cost Equations Some fundamental quantities of interest for queueing models are L, LQ , W, WQ ,
the average number of customers in the system; the average number of customers waiting in queue; the average amount of time a customer spends in the system; the average amount of time a customer spends waiting in queue.
A large number of interesting and useful relationships between the preceding and other quantities of interest can be obtained by making use of the following idea: Imagine that entering customers are forced to pay money (according to some rule) to the system. We would then have the following basic cost identity: average rate at which the system earns = λa × average amount an entering customer pays
(8.1)
where λa is defined to be average arrival rate of entering customers. That is, if N (t) denotes the number of customer arrivals by time t, then λa = lim
t→∞
N (t) t
We now present a heuristic proof of Equation (8.1).
Heuristic Proof of Equation (8.1) Let T be a fixed large number. In two different ways, we will compute the average amount of money the system has earned by time T . On one hand, this quantity approximately can be obtained by multiplying the average rate at which the system earns by the length of time T . On the other hand, we can approximately compute it by multiplying the average amount paid by an entering customer by the average number of customers entering by time T (and this latter factor is approximately λa T ). Hence, both sides of Equation (8.1) when multiplied by T are approximately equal to the average amount earned by T . The result then follows by letting T → ∞.∗ By choosing appropriate cost rules, many useful formulas can be obtained as special cases of Equation (8.1). For instance, by supposing that each customer pays $1 per unit time while in the system, Equation (8.1) yields the so-called ∗ This can be made into a rigorous proof provided we assume that the queueing process is regenerative in the sense of Section 7.5. Most models, including all the ones in this chapter, satisfy this condition.
496
8
Queueing Theory
Little’s formula, L = λa W
(8.2)
This follows since, under this cost rule, the rate at which the system earns is just the number in the system, and the amount a customer pays is just equal to its time in the system. Similarly if we suppose that each customer pays $1 per unit time while in queue, then Equation (8.1) yields LQ = λ a W Q
(8.3)
By supposing the cost rule that each customer pays $1 per unit time while in service we obtain from Equation (8.1) that the average number of customers in service = λa E[S]
(8.4)
where E[S] is defined as the average amount of time a customer spends in service. It should be emphasized that Equations (8.1) through (8.4) are valid for almost all queueing models regardless of the arrival process, the number of servers, or queue discipline. 8.2.2. Steady-State Probabilities Let X(t) denote the number of customers in the system at time t and define Pn , n 0, by Pn = lim P {X(t) = n} t→∞
where we assume the preceding limit exists. In other words, Pn is the limiting or long-run probability that there will be exactly n customers in the system. It is sometimes referred to as the steady-state probability of exactly n customers in the system. It also usually turns out that Pn equals the (long-run) proportion of time that the system contains exactly n customers. For example, if P0 = 0.3, then in the long run, the system will be empty of customers for 30 percent of the time. Similarly, P1 = 0.2 would imply that for 20 percent of the time the system would contain exactly one customer.∗ ∗ A sufficient condition for the validity of the dual interpretation of P is that the queueing process n
be regenerative.
8.2.
Preliminaries
497
Two other sets of limiting probabilities are {an , n 0} and {dn , n 0}, where an = proportion of customers that find n in the system when they arrive, and dn = proportion of customers leaving behind n in the system when they depart That is, Pn is the proportion of time during which there are n in the system; an is the proportion of arrivals that find n; and dn is the proportion of departures that leave behind n. That these quantities need not always be equal is illustrated by the following example.
Example 8.1 Consider a queueing model in which all customers have service times equal to 1, and where the times between successive customers are always greater than 1 [for instance, the interarrival times could be uniformly distributed over (1, 2)]. Hence, as every arrival finds the system empty and every departure leaves it empty, we have a0 = d0 = 1 However, P0 = 1 as the system is not always empty of customers.
It was, however, no accident that an equaled dn in the previous example. That arrivals and departures always see the same number of customers is always true as is shown in the next proposition.
Proposition 8.1 In any system in which customers arrive and depart one at a time the rate at which arrivals find n = the rate at which departures leave n and an = dn
Proof An arrival will see n in the system whenever the number in the system
goes from n to n + 1; similarly, a departure will leave behind n whenever the number in the system goes from n + 1 to n. Now in any interval of time T the number of transitions from n to n + 1 must equal to within 1 the number from n + 1 to n. [Between any two transitions from n to n + 1, there must be one from n + 1 to n, and conversely.] Hence, the rate of transitions from n to n + 1 equals the rate from n + 1 to n; or, equivalently, the rate at which arrivals find n equals the rate at which departures leave n. Now an , the proportion of arrivals finding n,
498
8
Queueing Theory
can be expressed as an =
the rate at which arrivals find n overall arrival rate
Similarly, dn =
the rate at which departures leave n overall departure rate
Thus, if the overall arrival rate is equal to the overall departure rate, then the preceding shows that an = dn . On the other hand, if the overall arrival rate exceeds the overall departure rate, then the queue size will go to infinity, implying that an = dn = 0. Hence, on the average, arrivals and departures always see the same number of customers. However, as Example 8.1 illustrates, they do not, in general, see the time averages. One important exception where they do is in the case of Poisson arrivals.
Proposition 8.2 Poisson arrivals always see time averages. In particular, for Poisson arrivals, Pn = an To understand why Poisson arrivals always see time averages, consider an arbitrary Poisson arrival. If we knew that it arrived at time t, then the conditional distribution of what it sees upon arrival is the same as the unconditional distribution of the system state at time t. For knowing that an arrival occurs at time t gives us no information about what occurred prior to t. (Since the Poisson process has independent increments, knowing that an event occurred at time t does not affect the distribution of what occurred prior to t.) Hence, an arrival would just see the system according to the limiting probabilities. Contrast the foregoing with the situation of Example 8.1 where knowing that an arrival occurred at time t tells us a great deal about the past; in particular it tells us that there have been no arrivals in (t − 1, t). Thus, in this case, we cannot conclude that the distribution of what an arrival at time t observes is the same as the distribution of the system state at time t. For a second argument as to why Poisson arrivals see time averages, note that the total time the system is in state n by time T is (roughly) Pn T . Hence, as Poisson arrivals always arrive at rate λ no matter what the system state, it follows that the number of arrivals in [0, T ] that find the system in state n is (roughly) λPn T . In the long run, therefore, the rate at which arrivals find the system in state n is λPn and, as λ is the overall arrival rate, it follows that λPn /λ = Pn is the proportion of arrivals that find the system in state n. The result that Poisson arrivals see time averages is called the PASTA principle.
8.3.
8.3.
Exponential Models
499
Exponential Models
8.3.1. A Single-Server Exponential Queueing System Suppose that customers arrive at a single-server service station in accordance with a Poisson process having rate λ. That is, the time between successive arrivals are independent exponential random variables having mean 1/λ. Each customer, upon arrival, goes directly into service if the server is free and, if not, the customer joins the queue. When the server finishes serving a customer, the customer leaves the system, and the next customer in line, if there is any, enters service. The successive service times are assumed to be independent exponential random variables having mean 1/μ. The preceding is called the M/M/1 queue. The two Ms refer to the fact that both the interarrival and the service distributions are exponential (and thus memoryless, or Markovian), and the 1 to the fact that there is a single server. To analyze it, we shall begin by determining the limiting probabilities Pn , for n = 0, 1, . . . . To do so, think along the following lines. Suppose that we have an infinite number of rooms numbered 0, 1, 2, . . . , and suppose that we instruct an individual to enter room n whenever there are n customers in the system. That is, he would be in room 2 whenever there are two customers in the system; and if another were to arrive, then he would leave room 2 and enter room 3. Similarly, if a service would take place he would leave room 2 and enter room 1 (as there would now be only one customer in the system). Now suppose that in the long run our individual is seen to have entered room 1 at the rate of ten times an hour. Then at what rate must he have left room 1? Clearly, at this same rate of ten times an hour. For the total number of times that he enters room 1 must be equal to (or one greater than) the total number of times he leaves room 1. This sort of argument thus yields the general principle which will enable us to determine the state probabilities. Namely, for each n 0, the rate at which the process enters state n equals the rate at which it leaves state n. Let us now determine these rates. Consider first state 0. When in state 0 the process can leave only by an arrival as clearly there cannot be a departure when the system is empty. Since the arrival rate is λ and the proportion of time that the process is in state 0 is P0 , it follows that the rate at which the process leaves state 0 is λP0 . On the other hand, state 0 can only be reached from state 1 via a departure. That is, if there is a single customer in the system and he completes service, then the system becomes empty. Since the service rate is μ and the proportion of time that the system has exactly one customer is P1 , it follows that the rate at which the process enters state 0 is μP1 . Hence, from our rate-equality principle we get our first equation, λP0 = μP1
500
8
Queueing Theory
Now consider state 1. The process can leave this state either by an arrival (which occurs at rate λ) or a departure (which occurs at rate μ). Hence, when in state 1, the process will leave this state at a rate of λ + μ.∗ Since the proportion of time the process is in state 1 is P1 , the rate at which the process leaves state 1 is (λ + μ)P1 . On the other hand, state 1 can be entered either from state 0 via an arrival or from state 2 via a departure. Hence, the rate at which the process enters state 1 is λP0 + μP2 . Because the reasoning for other states is similar, we obtain the following set of equations: State
Rate at which the process leaves = rate at which it enters λP0 = μP1 (λ + μ)Pn = λPn−1 + μPn+1
0 n, n 1
(8.5)
The set of Equations (8.5) which balances the rate at which the process enters each state with the rate at which it leaves that state is known as balance equations. In order to solve Equations (8.5), we rewrite them to obtain P1 = Pn+1 =
λ P0 , μ
λ λ Pn + Pn − Pn−1 , μ μ
n1
Solving in terms of P0 yields P0 = P0 , P1 =
λ P0 , μ
2 λ λ λ λ P1 + P1 − P0 = P1 = P0 , μ μ μ μ 3 λ λ λ λ P3 = P2 + P2 − P1 = P2 = P0 , μ μ μ μ 4 λ λ λ λ P4 = P3 + P3 − P2 = P3 = P0 , μ μ μ μ n+1 λ λ λ λ Pn+1 = Pn + Pn − Pn−1 = Pn = P0 μ μ μ μ P2 =
∗ If one event occurs at a rate λ and another occurs at rate μ, then the total rate at which either event occurs is λ + μ. Suppose one man earns $2 per hour and another earns $3 per hour; then together they clearly earn $5 per hour.
8.3.
Exponential Models
501
To determine P0 we use the fact that the Pn must sum to 1, and thus 1=
∞
Pn =
n=0
∞ n λ n=0
μ
P0 =
P0 1 − λ/μ
or λ , μ n λ λ 1− , Pn = μ μ P0 = 1 −
n1
(8.6)
Notice that for the preceding equations to make sense, it is necessary for λ/μ to n be less than 1. For otherwise ∞ n=0 (λ/μ) would be infinite and all the Pn would be 0. Hence, we shall assume that λ/μ < 1. Note that it is quite intuitive that there would be no limiting probabilities if λ > μ. For suppose that λ > μ. Since customers arrive at a Poisson rate λ, it follows that the expected total number of arrivals by time t is λt. On the other hand, what is the expected number of customers served by time t? If there were always customers present, then the number of customers served would be a Poisson process having rate μ since the time between successive services would be independent exponentials having mean 1/μ. Hence, the expected number of customers served by time t is no greater than μt; and, therefore, the expected number in the system at time t is at least λt − μt = (λ − μ)t Now if λ > μ, then the preceding number goes to infinity as t becomes large. That is, λ/μ > 1, the queue size increases without limit and there will be no limiting probabilities. Note also that the condition λ/μ < 1 is equivalent to the condition that the mean service time be less than the mean time between successive arrivals. This is the general condition that must be satisfied for limited probabilities to exist in most single-server queueing systems.
Remarks (i) In solving the balance equations for the M/M/1 queue, we obtained as an intermediate step the set of equations λPn = μPn+1 ,
n0
These equations could have been directly argued from the general queueing result (shown in Proposition 8.1) that the rate at which arrivals find n in the system— namely λPn —is equal to the rate at which departures leave behind n—namely, μPn+1 .
502
8
Queueing Theory
(ii) We can also prove that Pn = (λ/μ)n (1 − λ/μ) by using a queueing cost identity. Suppose that, for a fixed n > 0, whenever there are at least n customers in the system the nth oldest customer (with age measured from when the customer arrived) pays 1 per unit time. Letting X be the steady state number of customers in the system, because the system earns 1 per unit time whenever X is at least n, it follows that average rate at which the system earns = P {X n} Also, because a customer who finds fewer than n − 1 in the system when it arrives will pay 0, while an arrival who finds at least n − 1 in the system will pay 1 per unit time for an exponentially distributed time with rate μ average amount a customer pays =
1 P {X n − 1} μ
Therefore, the queueing cost identity yields that P {X n} = (λ/μ)P {X n − 1},
n>0
Iterating this gives P {X n} = (λ/μ)P {X n − 1} = (λ/μ)2 P {X n − 2} = ··· = (λ/μ)n P {X 0} = (λ/μ)n Therefore, P {X = n} = P {X n} − P {X n + 1} = (λ/μ)n (1 − λ/μ)
Now let us attempt to express the quantities L, LQ , W , and WQ in terms of the limiting probabilities Pn . Since Pn is the long-run probability that the system contains exactly n customers, the average number of customers in the system clearly is given by L=
∞
nPn
n=0
=
n ∞ λ λ 1− n μ μ n=0
=
λ μ−λ
(8.7)
8.3.
Exponential Models
503
where the last equation followed upon application of the algebraic identity ∞
nx n =
n=0
x (1 − x)2
The quantities W, WQ , and LQ now can be obtained with the help of Equations (8.2) and (8.3). That is, since λa = λ, we have from Equation (8.7) that W= =
L λ 1 , μ−λ
WQ = W − E[S] =W − =
1 μ
λ , μ(μ − λ)
LQ = λWQ =
λ2 μ(μ − λ)
(8.8)
Example 8.2 Suppose that customers arrive at a Poisson rate of one per every 12 minutes, and that the service time is exponential at a rate of one service per 8 minutes. What are L and W ? Solution:
Since λ =
1 12 ,
μ = 18 , we have L = 2,
W = 24
Hence, the average number of customers in the system is two, and the average time a customer spends in the system is 24 minutes. 1 . What is the Now suppose that the arrival rate increases 20 percent to λ = 10 corresponding change in L and W ? Again using Equations (8.7), we get L = 4,
W = 40
Hence, an increase of 20 percent in the arrival rate doubled the average number of customers in the system.
504
8
Queueing Theory
To understand this better, write Equations (8.7) as L=
λ/μ , 1 − λ/μ
W=
1/μ 1 − λ/μ
From these equations we can see that when λ/μ is near 1, a slight increase in λ/μ will lead to a large increase in L and W .
A Technical Remark We have used the fact that if one event occurs at an exponential rate λ, and another independent event at an exponential rate μ, then together they occur at an exponential rate λ + μ. To check this formally, let T1 be the time at which the first event occurs, and T2 the time at which the second event occurs. Then P {T1 t} = 1 − e−λt , P {T2 t} = 1 − e−μt Now if we are interested in the time until either T1 or T2 occurs, then we are interested in T = min(T1 , T2 ). Now P {T t} = 1 − P {T > t} = 1 − P {min(T1 , T2 ) > t} However, min(T1 , T2 ) > t if and only if both T1 and T2 are greater than t; hence, P {T t} = 1 − P {T1 > t, T2 > t} = 1 − P {T1 > t}P {T2 > t} = 1 − e−λt e−μt = 1 − e−(λ+μ)t Thus, T has an exponential distribution with rate λ + μ, and we are justified in adding the rates. Given that an M/M/1 steady-state customer—that is, a customer who arrives after the system has been in operation a long time—spends a total of t time units in the system, let us determine the conditional distribution of N , the number of others that were present when that customer arrived. That is, letting W ∗ be the
8.3.
Exponential Models
505
amount of time a customer spends in the system, we will find P {N = n|W ∗ = t}. Now, P {N = n|W ∗ = t} = =
fN,W ∗ (n, t) fW∗ (t) P {N = n}fW ∗ |N (t|n) fW∗ (t)
where fW ∗ |N (t|n) is the conditional density of W ∗ given that N = n, and fW ∗ (t) is the unconditional density of W ∗ . Now, given that N = n, the time that the customer spends in the system is distributed as the sum of n + 1 independent exponential random variables with a common rate μ, implying that the conditional distribution of W ∗ given that N = n is the gamma distribution with parameters n + 1 and μ. Therefore, with C = 1/fW ∗ (t) P {N = n|W ∗ = t} = CP {N = n}μe−μt
(μt)n n!
= C(λ/μ)n (1 − λ/μ)μe−μt =K
(μt)n n!
(by PASTA)
(λt)n n!
where K = C(1 − λ/μ)μe−μt does not depend on n. Summing over n yields 1=
∞
P {N = n|T = t} = K
n=0
∞ (λt)n
n!
n=0
= Keλt
Thus, K = e−λt , showing that P {N = n|W ∗ = t} = e−λt
(λt)n n!
Therefore, the conditional distribution of the number seen by an arrival who spends a total of t time units in the system is the Poisson distribution with mean λt. In addition, as a by-product of our analysis, we have that fW ∗ (t) = 1/C =
1 (1 − λ/μ)μe−μt K
= (μ − λ)e−(μ−λ)t
506
8
Queueing Theory
In other words, W ∗ , the amount of time a customer spends in the system, is an exponential random variable with rate μ − λ. (As a check, we note that E[W ∗ ] = 1/(μ − λ) which checks with Equation (8.8) since W = E[W ∗ ].)
Remark Another argument as to why W ∗ is exponential with rate μ − λ is as follows. If we let N denote the number of customers in the system as seen by an arrival, then this arrival will spend N + 1 service times in the system before departing. Now, P {N + 1 = j } = P {N = j − 1} = (λ/μ)j −1 (1 − λ/μ),
j 1
In words, the number of services that have to be completed before the arrival departs is a geometric random variable with parameter 1 − λ/μ. Therefore, after each service completion our customer will be the one departing with probability 1 − λ/μ. Thus, no matter how long the customer has already spent in the system, the probability he will depart in the next h time units is μh + o(h), the probability that a service ends in that time, multiplied by 1 − λ/μ. That is, the customer will depart in the next h time units with probability (μ − λ)h + o(h), which says that the hazard rate function of W ∗ is the constant μ − λ. But only the exponential has a constant hazard rate, and so we can conclude that W ∗ is exponential with rate μ − λ. Our next example illustrates the inspection paradox.
Example 8.3 For an M/M/1 queue in steady state, what is the probability that the next arrival finds n in the system? Solution: Although it might initially seem, by the PASTA principle, that this probability should just be (λ/μ)n (1 − λ/μ), we must be careful. Because if t is the current time, then the time from t until the next arrival is exponentially distributed with rate λ, and is independent of the time from t since the last arrival, which (in the limit, as t goes to infinity) is also exponential with rate λ. Thus, although the times between successive arrivals of a Poisson process are exponential with rate λ, the time between the previous arrival before t and the first arrival after t is distributed as the sum of two independent exponentials. (This is an illustration of the inspection paradox, which results because the length of an interarrival interval that contains a specified time tends to be longer than an ordinary interarrival interval—see Section 7.7.) Let Na denote the number found by the next arrival, and let X be the number currently in the system. Conditioning on X yields
8.3.
P {Na = n} =
∞
Exponential Models
507
P {Na = n|X = k}P {X = k}
k=0
=
∞
P {Na = n|X = k}(λ/μ)k (1 − λ/μ)
k=0
=
∞
P {Na = n|X = k}(λ/μ)k (1 − λ/μ)
k=n
=
∞
P {Na = n|X = n + i}(λ/μ)n+i (1 − λ/μ)
i=0
Now, for n > 0, given there are currently n + i in the system, the next arrival will find n if we have i services before an arrival and then an arrival before the next service completion. By the lack of memory property of exponential interarrival random variables, this gives P {Na = n|X = n + i} =
μ λ+μ
i
λ , λ+μ
n>0
Consequently, for n > 0, i n+i ∞ μ λ λ P {Na = n} = (1 − λ/μ) λ+μ λ+μ μ i=0
i ∞ λ λ = (λ/μ) (1 − λ/μ) λ+μ λ+μ n
i=0
= (λ/μ)n+1 (1 − λ/μ)
On the other hand, the probability that the next arrival will find the system empty, when there are currently i in the system, is the probability that there μ i ), are i services before the next arrival. Therefore, P {Na = 0|X = i} = ( λ+μ giving that
508
8
Queueing Theory
i i ∞ μ λ P {Na = 0} = (1 − λ/μ) λ+μ μ i=0
= (1 − λ/μ)
∞ i=0
λ λ+μ
i
= (1 + λ/μ)(1 − λ/μ) As a check, note that ∞
,
∞ P {Na = n} = (1 − λ/μ) 1 + λ/μ + (λ/μ)n+1
n=0
-
n=1
= (1 − λ/μ)
∞ (λ/μ)i i=0
=1 Note that P {Na = 0} is larger than P0 = 1 − λ/μ, showing that the next arrival is more likely to find an empty system than is an average arrival, and thus illustrating the inspection paradox that when the next customer arrives the elapsed time since the previous arrival is distributed as the sum of two independent exponentials with rate λ. Also, we might expect because of the inspection paradox that E[Na ] is less than L, the average number of customers seen by an arrival. That this is indeed the case is seen from E[Na ] =
∞ n=1
n(λ/μ)n+1 (1 − λ/μ) =
λ L 0
Interpretation No one in service Server busy; no one waiting n customers waiting
The transition diagram is shown in Figure 8.2 and the balance equations are State 0 0 n, n 1
Rate at which the process leaves = rate at which it enters λP0 = μP0 (λ + μ)P0 = λP0 + μP1 + μP2 (λ + μ)Pn = λPn−1 + μPn+2
Now the set of equations (λ + μ)Pn = λPn−1 + μPn+2 ,
n = 1, 2, . . .
has a solution of the form Pn = α n P0 To see this, substitute the preceding in Equation (8.13) to obtain (λ + μ)α n P0 = λα n−1 P0 + μα n+2 P0 or (λ + μ)α = λ + μα 3
Figure 8.2.
(8.13)
516
8
Queueing Theory
Solving this for α yields the three roots: √ √ −1 + 1 + 4λ/μ −1 − 1 + 4λ/μ , and α = α = 1, α= 2 2 As the first two are clearly not possible, it follows that √ 1 + 4λ/μ − 1 α= 2 Hence, Pn = α n P0 , μ P0 λ where the bottom equation follows from the first balance equation. (We can ignore the second balance equation as one of these equations is always redundant.) To obtain P0 , we use P0 =
P0 + P0 +
∞
Pn = 1
n=1
or
, P0
∞ μ n α =1 1+ + λ n=1
or
. P0
/ 1 μ + =1 1−α λ
or P0 =
λ(1 − α) λ + μ(1 − α)
and thus Pn =
α n λ(1 − α) , λ + μ(1 − α)
P0 =
μ(1 − α) λ + μ(1 − α)
where α=
√ 1 + 4λ/μ − 1 2
n0 (8.14)
8.4.
Network of Queues
517
Note that for the preceding to be valid we need α < 1, or equivalently λ/μ < 2, which is intuitive since the maximum service rate is 2μ, which must be larger than the arrival rate λ to avoid overloading the system. All the relevant quantities of interest now can be determined. For instance, to determine the proportion of customers that are served alone, we first note that the rate at which customers are served alone is λP0 + μP1 , since when the system is empty a customer will be served alone upon the next arrival and when there is one customer in queue he will be served alone upon a departure. As the rate at which customers are served is λ, it follows that λP0 + μP1 λ μ = P0 + P1 λ
proportion of customers that are served alone =
Also, LQ =
∞
nPn
n=1 ∞
=
λ(1 − α) n nα λ + μ(1 − α)
from Equation (8.14)
λα (1 − α)[λ + μ(1 − α)]
by algebraic identity
n=1
=
∞ 1
nα n =
α (1 − α)2
and WQ =
LQ , λ
W = WQ +
1 , μ
L = λW
8.4.
Network of Queues
8.4.1. Open Systems Consider a two-server system in which customers arrive at a Poisson rate λ at server 1. After being served by server 1 they then join the queue in front of server 2. We suppose there is infinite waiting space at both servers. Each server
518
8
Queueing Theory
Figure 8.3. A tandem queue.
serves one customer at a time with server i taking an exponential time with rate μi for a service, i = 1, 2. Such a system is called a tandem or sequential system (see Figure 8.3). To analyze this system we need to keep track of the number of customers at server 1 and the number at server 2. So let us define the state by the pair (n, m)— meaning that there are n customers at server 1 and m at server 2. The balance equations are Rate that the process leaves = rate that it enters λP0,0 = μ2 P0,1 (λ + μ1 )Pn,0 = μ2 Pn,1 + λPn−1,0 (λ + μ2 )P0,m = μ2 P0,m+1 + μ1 P1,m−1 (λ + μ1 + μ2 )Pn,m = μ2 Pn,m+1 + μ1 Pn+1,m−1 + λPn−1,m (8.15) Rather than directly attempting to solve these (along with the equation n,m Pn,m = 1) we shall guess at a solution and then verify that it indeed satisfies the preceding. We first note that the situation at server 1 is just as in an M/M/1 model. Similarly, as it was shown in Section 6.6 that the departure process of an M/M/1 queue is a Poisson process with rate λ, it follows that what server 2 faces is also an M/M/1 queue. Hence, the probability that there are n customers at server 1 is n λ λ 1− P {n at server 1} = μ1 μ1 State 0, 0 n, 0; n > 0 0, m; m > 0 n, m; nm > 0
and, similarly, P {m at server 2} =
λ μ2
m λ 1− μ2
Now if the numbers of customers at servers 1 and 2 were independent random variables, then it would follow that m n λ λ λ λ 1− 1− (8.16) Pn,m = μ1 μ1 μ2 μ2
8.4.
Network of Queues
519
To verify that Pn,m is indeed equal to the preceding (and thus that the number of customers at server 1 is independent of the number at server 2), all we need do is verify that the preceding satisfies the set of Equations (8.15)—this suffices since we know that the Pn,m are the unique solution of Equations (8.15). Now, for instance, if we consider the first equation of (8.15), we need to show that λ λ λ λ λ 1− = μ2 1 − 1− λ 1− μ1 μ2 μ1 μ2 μ2 which is easily verified. We leave it as an exercise to show that the Pn,m , as given by Equation (8.16), satisfy all of the Equations (8.15), and are thus the limiting probabilities. From the preceding we see that L, the average number of customers in the system, is given by (n + m)Pn,m L= n,m
m λ n λ λ λ 1− + 1− = n m μ1 μ1 μ2 μ2 n m =
λ λ + μ1 − λ μ2 − λ
and from this we see that the average time a customer spends in the system is W=
1 1 L = + λ μ1 − λ μ2 − λ
Remarks (i) The result (Equations (8.15)) could have been obtained as a direct consequence of the time reversibility of an M/M/1 (see Section 6.6). For not only does time reversibility imply that the output from server 1 is a Poisson process, but it also implies (Exercise 26 of Chapter 6) that the number of customers at server 1 is independent of the past departure times from server 1. As these past departure times constitute the arrival process to server 2, the independence of the numbers of customers in the two systems follows. (ii) Since a Poisson arrival sees time averages, it follows that in a tandem queue the numbers of customers an arrival (to server 1) sees at the two servers are independent random variables. However, it should be noted that this does not imply that the waiting times of a given customer at the two servers are independent. For a counter example suppose that λ is very small with respect to μ1 = μ2 ; and thus almost all customers have zero wait in queue at both servers. However, given that the wait in queue of a customer at server 1 is positive, his wait in queue at server 2 also will be positive with probability at least as large as 12 (why?). Hence, the wait-
520
8
Queueing Theory
ing times in queue are not independent. Remarkably enough, however, it turns out that the total times (that is, service time plus wait in queue) that an arrival spends at the two servers are indeed independent random variables. The preceding result can be substantially generalized. To do so, consider a system of k servers. Customers arrive from outside the system to server i, i = 1, . . . , k, in accordance with independent Poisson processes at rate ri ; they then join the queue at i until their turn at service comes. Once a customer is served by server i, he then joins the queue in front of server j , j = 1, . . . , k, with probability Pij . Hence, kj =1 Pij 1, and 1 − kj =1 Pij represents the probability that a customer departs the system after being served by server i. If we let λj denote the total arrival rate of customers to server j , then the λj can be obtained as the solution of λj = rj +
k
i = 1, . . . , k
λi Pij ,
(8.17)
i=1
Equation (8.17) follows since rj is the arrival rate of customers to j coming from outside the system and, as λi is the rate at which customers depart server i (rate in must equal rate out), λi Pij is the arrival rate to j of those coming from server i. It turns out that the number of customers at each of the servers is independent and of the form n λj λj P {n customers at server j } = 1− , n1 μj μj where μj is the exponential service rate at server j and the λj are the solution to Equation (8.17). Of course, it is necessary that λj /μj < 1 for all j . To prove this, we first note that it is equivalent to asserting that the limiting probabilities P (n1 , n2 , . . . , nk ) = P {nj at server j , j = 1, . . . , k} are given by P (n1 , n2 , . . . , nk ) =
k λj nj j =1
μj
1−
λj μj
(8.18)
which can be verified by showing that it satisfies the balance equations for this model. The average number of customers in the system is L=
k
average number at server j
j =1
=
k j =1
λj μ j − λj
8.4.
Network of Queues
521
The average in the system can be obtained from L = λW time a customer spends with λ = kj =1 rj . (Why not λ = kj =1 λj ?) This yields k j =1 λj /(μj − λj ) W= k j =1 rj
Remark The result embodied in Equation (8.18) is rather remarkable in that it says that the distribution of the number of customers at server i is the same as in an M/M/1 system with rates λi and μi . What is remarkable is that in the network model the arrival process at node i need not be a Poisson process. For if there is a possibility that a customer may visit a server more than once (a situation called feedback), then the arrival process will not be Poisson. An easy example illustrating this is to suppose that there is a single server whose service rate is very large with respect to the arrival rate from outside. Suppose also that with probability p = 0.9 a customer upon completion of service is fed back into the system. Hence, at an arrival time epoch there is a large probability of another arrival in a short time (namely, the feedback arrival); whereas at an arbitrary time point there will be only a very slight chance of an arrival occurring shortly (since λ is so very small). Hence, the arrival process does not possess independent increments and so cannot be Poisson. Thus, we see that when feedback is allowed the steady-state probabilities of the number of customers at any given station have the same distribution as in an M/M/1 model even though the model is not M/M/1. (Presumably such quantities as the joint distribution of the number at the station at two different time points will not be the same as for an M/M/1.)
Example 8.6 Consider a system of two servers where customers from outside the system arrive at server 1 at a Poisson rate 4 and at server 2 at a Poisson rate 5. The service rates of 1 and 2 are respectively 8 and 10. A customer upon completion of service at server 1 is equally likely to go to server 2 or to leave the system (i.e., P11 = 0, P12 = 12 ); whereas a departure from server 2 will go 25 percent of the time to server 1 and will depart the system otherwise (i.e., P21 = 14 , P22 = 0). Determine the limiting probabilities, L, and W . Solution: The total arrival rates to servers 1 and 2—call them λ1 and λ2 — can be obtained from Equation (8.17). That is, we have λ1 = 4 + 14 λ2 , λ2 = 5 + 12 λ1 implying that λ1 = 6,
λ2 = 8
522
8
Queueing Theory
Hence, P {n at server 1, m at server 2} = =
$ %n $ %m 1 4
3 4
1 20
4 5
1 5
$ %n $ %m 3 4
4 5
and L=
8 6 + = 7, 8 − 6 10 − 8
W=
L 7 = 9 9
8.4.2. Closed Systems The queueing systems described in Section 8.4.1 are called open systems since customers are able to enter and depart the system. A system in which new customers never enter and existing ones never depart is called a closed system. Let us suppose that we have m customers moving among a system of k servers, where the service times at server i are exponential with rate μi , i = 1, . . . , k. When a customer completes service at server i, she then joins the queue in front of server j, j = 1, . . . , k, with probability Pij , where we now suppose that k j =1 Pij = 1 for all i = 1, . . . , k. That is, P = [Pij ] is a Markov transition probability matrix, which we shall assume is irreducible. Let π = (π1 , . . . , πk ) denote the stationary probabilities for this Markov chain; that is, π is the unique positive solution of πj =
k
πi Pij ,
i=1 k
πj = 1
(8.19)
j =1
If we denote the average arrival rate (or equivalently the average service completion rate) at server j by λm (j ), j = 1, . . . , k then, analogous to Equation (8.17), the λm (j ) satisfy λm (j ) =
k i=1
λm (i)Pij
8.4.
523
Network of Queues
Hence, from (8.19) we can conclude that λm (j ) = λm πj ,
j = 1, 2, . . . , k
(8.20)
where λm =
k
(8.21)
λm (j )
j =1
From Equation (8.21), we see that λm is the average service completion rate of the entire system, that is, it is the system throughput rate.* If we let Pm (n1 , n2 , . . . , nk ) denote the limiting probabilities Pm (n1 , n2 , . . . , nk ) = P {nj customers at server j, j = 1, . . . , k} then, by verifying that they satisfy the balance equation, it can be shown that 1 if kj =1 nj = m Km kj =1 (λm (j )/μj )nj , Pm (n1 , n2 , . . . , nk ) = 0, otherwise But from Equation (8.20) we thus obtain that 1k Cm j =1 (πj /μj )nj , Pm (n1 , n2 , . . . , nk ) = 0,
if kj =1 nj = m otherwise
(8.22)
where ⎤−1
⎡ ⎢ Cm = ⎢ ⎣
n 1 ,...,nk : nj =m
k j =1
⎥ (πj /μj )nj ⎥ ⎦
(8.23)
Equation (8.22) is not as useful as we might suppose, for in order to utilize it we must determine the normalizing constant Cm given by Equation (8.23) which requires summing the products kj =1 (πj /μj )nj over feasible vec all the k tors (n1 , . . . , nk ): j =1 nj = m. Hence, since there are m+k−1 vectors this is m only computationally feasible for relatively small values of m and k. We will now present an approach that will enable us to determine recursively many of the quantities of interest in this model without first computing the normalizing constants. To begin, consider a customer who has just left server i and is headed to server j , and let us determine the probability of the system as seen * We are just using the notation λ (j ) and λ to indicate the dependence on the number of customers m m
in the closed system. This will be used in recursive relations we will develop.
524
8
Queueing Theory
by this customer. In particular, let us determine the probability that this customer observes, at that moment, nl customers at server l, l = 1, . . . , k, kl=1 nl = m − 1. This is done as follows: P {customer observes nl at server l, l = 1, . . . , k | customer goes from i to j } =
P {state is (n1 , . . . , ni + 1, . . . , nj , . . . , nk ), customer goes from i toj } P {customer goes from i toj }
Pm (n1 , . . . , ni + 1, . . . , nj , . . . , nk )μi Pij n: nj =m−1 Pm (n1 , . . . , ni + 1, . . . , nk )μi Pij 1 (πi /μi ) kj =1 (πj /μj )nj from (8.22) = K
=
=C
k
(πj /μj )nj
j =1
where C does not depend on n1 , . . . , nk . But because the preceding is a probability density on the set of vectors (n1 , . . . , nk ), kj =1 nj = m−1, it follows from (8.22) that it must equal Pm−1 (n1 , . . . , nk ). Hence, P {customer observes nl at server l, l = 1, . . . , k | customer goes from i to j } = Pm−1 (n1 , . . . , nk ),
k
ni = m − 1
(8.24)
i=1
As (8.24) is true for all i, we thus have proven the following proposition, known as the arrival theorem.
Proposition 8.3 (The Arrival Theorem) In the closed network system with m customers, the system as seen by arrivals to server j is distributed as the stationary distribution in the same network system when there are only m − 1 customers. Denote by Lm (j ) and Wm (j ) the average number of customers and the average time a customer spends at server j when there are m customers in the network. Upon conditioning on the number of customers found at server j by an arrival to that server, it follows that Wm (j ) = =
1 + Em [number at server j as seen by an arrival] μj 1 + Lm−1 (j ) μj
(8.25)
8.4.
Network of Queues
525
where the last equality follows from the arrival theorem. Now when there are m − 1 customers in the system, then, from Equation (8.20), λm−1 (j ), the average arrival rate to server j , satisfies λm−1 (j ) = λm−1 πj Now, applying the basic cost identity Equation (8.1) with the cost rule being that each customer in the network system of m − 1 customers pays one per unit time while at server j , we obtain Lm−1 (j ) = λm−1 πj Wm−1 (j )
(8.26)
Using Equation (8.25), this yields Wm (j ) = Also using the fact that tion (8.26):
1 + λm−1 πj Wm−1 (j ) μj
k
j =1 Lm−1 (j )
m − 1 = λm−1
(8.27)
= m − 1 (why?) we obtain, from Equa-
k
πj Wm−1 (j )
j =1
or λm−1 = k
m−1
(8.28)
i=1 πi Wm−1 (i)
Hence, from Equation (8.27), we obtain the recursion Wm (j ) =
(m − 1)πj Wm−1 (j ) 1 + μj μj ki=1 πi Wm−1 (i)
(8.29)
Starting with the stationary probabilities πj , j = 1, . . . , k, and W1 (j ) = 1/μj we can now use Equation (8.29) to determine recursively W2 (j ), W3 (j ), . . . , Wm (j ). We can then determine the throughput rate λm by using Equation (8.28), and this will determine Lm (j ) by Equation (8.26). This recursive approach is called mean value analysis.
Example 8.7 Consider a k-server network in which the customers move in a cyclic permutation. That is, Pi,i+1 = 1,
i = 1, 2 . . . , k − 1,
Pk,1 = 1
Let us determine the average number of customers at server j when there are two customers in the system. Now, for this network πi = 1/k,
i = 1, . . . , k
526
8
Queueing Theory
and as W1 (j ) =
1 μj
we obtain from Equation (8.29) that W2 (j ) = =
(1/k)(1/μj ) 1 + k μj μj i=1 (1/k)(1/μi ) 1 1 + k 2 μj μj i=1 1/μi
Hence, from Equation (8.28), λ2 =
2 k l=1
=
1 W2 (l) k
k l=1
2k 1 1 + k 2 μl μl i=1 1/μi
and finally, using Equation (8.26), 1 1 + μj μ2j ki=1 1/μi 1 L2 (j ) = λ2 W2 (j ) = k k 1 1 + μl μ2l ki=1 1/μi
2
l=1
Another approach to learning about the stationary probabilities specified by Equation (8.22), which finesses the computational difficulties of computing the constant Cm , is to use the Gibbs sampler of Section 4.9 to generate a Markov chain having these stationary probabilities. To begin, note that since there are always a total of m customers in the system, Equation (8.22) may equivalently be written as a joint mass function of the numbers of customers at each of the servers 1, . . . , k − 1, as follows:
Pm (n1 , . . . , nk−1 ) = Cm (πk /μk )m−
nj
k−1
(πj /μj )nj
j =1
=K
k−1 j =1
(aj )nj ,
k−1 j =1
nj m
8.4.
Network of Queues
527
where aj = (πj μk )/(πk μj ), j = 1, . . . , k − 1. Now, if N = (N1 , . . . , Nk−1 ) has the preceding joint mass function then, P {Ni = n|N1 = n1 , . . . , Ni−1 = ni−1 , Ni+1 = ni+1 , . . . , Nk−1 = nk−1 } Pm (n1 , . . . , ni−1 , n, ni+1 , . . . , nk−1 ) = r Pm (n1 , . . . , ni−1 , r, ni+1 , . . . , nk−1 ) = Cain , nm− nj j =i
It follows from the preceding that we may use the Gibbs sampler to generate the values of a Markov chain whose limiting probability mass function is Pm (n1 , . . . , nk−1 ) as follows: 1. Let (n1 , . . . , nk−1 ) be arbitrary nonnegative integers satisfying k−1 j =1 nj m. 2. Generate a random variable I that is equally likely to be any of 1, . . . , k − 1. 3. If I = i, set s = m − j =i nj , and generate the value of a random variable X having probability mass function P {X = n} = Cain ,
n = 0, . . . , s
4. Let nI = X and go to step 2.
The successive values of the state vector (n1 , . . . , nk−1 , m − k−1 j =1 nj ) constitute the sequence of states of a Markov chain with the limiting distribution Pm . All quantities of interest can be estimated from this sequence. For instance, the average of the values of the j th coordinate of these vectors will converge to the mean number of individuals at station j , the proportion of vectors whose j th coordinate is less than r will converge to the limiting probability that the number of individuals at station j is less than r, and so on. Other quantities of interest can also be obtained from the simulation. For instance, suppose we want to estimate Wj , the average amount of time a customer spends at server j on each visit. Then, as noted in the preceding, Lj , the average number of customers at server j , can be estimated. To estimate Wj , we use the identity Lj = λ j W j where λj is the rate at which customers arrive at server j . Setting λj equal to the service completion rate at server j shows that λj = P {j is busy}μj Using the Gibbs sampler simulation to estimate P {j is busy} then leads to an estimator of Wj .
528
8
8.5.
Queueing Theory
The System M/G/1
8.5.1. Preliminaries: Work and Another Cost Identity For an arbitrary queueing system, let us define the work in the system at any time t to be the sum of the remaining service times of all customers in the system at time t. For instance, suppose there are three customers in the system—the one in service having been there for three of his required five units of service time, and both people in queue having service times of six units. Then the work at that time is 2 + 6 + 6 = 14. Let V denote the (time) average work in the system. Now recall the fundamental cost equation (8.1), which states that the average rate at which the system earns = λa × average amount a customer pays and consider the following cost rule: Each customer pays at a rate of y/unit time when his remaining service time is y, whether he is in queue or in service. Thus, the rate at which the system earns is just the work in the system; so the basic identity yields that V = λa E[amount paid by a customer] ∗ denote respectively the service time and the time a given Now, let S and WQ customer spends waiting in queue. Then, since the customer pays at a constant rate of S per unit time while he waits in queue and at a rate of S − x after spending an amount of time x in service, we have . / S ∗ E[amount paid by a customer] = E SWQ + (S − x) dx 0
and thus ∗ ]+ V = λa E[SWQ
λa E[S 2 ] 2
(8.30)
It should be noted that the preceding is a basic queueing identity [like Equations (8.2)–(8.4)] and as such is valid in almost all models. In addition, if a customer’s service time is independent of his wait in queue (as is usually, but not always the case),* then we have from Equation (8.30) that V = λa E[S]WQ + * For an example where it is not true, see Section 8.6.2.
λa E[S 2 ] 2
(8.31)
8.5.
529
The System M/G/1
8.5.2. Application of Work to M/G/1 The M/G/1 model assumes (i) Poisson arrivals at rate λ; (ii) a general service distribution; and (iii) a single server. In addition, we will suppose that customers are served in the order of their arrival. Now, for an arbitrary customer in an M/G/1 system. Customer’s wait in queue = work in the system when he arrives
(8.32)
this follows since there is only a single server (think about it!). Taking expectations of both sides of Equation (8.32) yields WQ = average work as seen by an arrival But, due to Poisson arrivals, the average work as seen by an arrival will equal V , the time average work in the system. Hence, for the model M/G/1, WQ = V The preceding in conjunction with the identity λE[S 2 ] 2 yields the so-called Pollaczek–Khintchine formula, V = λE[S]WQ +
WQ =
λE[S 2 ] 2(1 − λE[S])
(8.33)
where E[S] and E[S 2 ] are the first two moments of the service distribution. The quantities L, LQ , and W can be obtained from Equation (8.33) as LQ = λWQ =
λ2 E[S 2 ] , 2(1 − λE[S])
W = WQ + E[S] = L = λW =
λE[S 2 ] + E[S], 2(1 − λE[S])
(8.34)
λ2 E[S 2 ] + λE[S] 2(1 − λE[S])
Remarks (i) For the preceding quantities to be finite, we need λE[S] < 1. This condition is intuitive since we know from renewal theory that if the server was always busy, then the departure rate would be 1/E[S] (see Section 7.3), which must be larger than the arrival rate λ to keep things finite. (ii) Since E[S 2 ] = Var(S) + (E[S])2 , we see from Equations (8.33) and (8.34) that, for fixed mean service time, L, LQ , W , and WQ all increase as the variance of the service distribution increases. (iii) Another approach to obtain WQ is presented in Exercise 38.
530
8
Queueing Theory
8.5.3. Busy Periods The system alternates between idle periods (when there are no customers in the system, and so the server is idle) and busy periods (when there is at least one customer in the system, and so the server is busy). Let us denote by In and Bn , respectively, lengths of the nth idle and the the n nth busy period, n 1. Hence, in the first j =1 (Ij + Bj ) time units the server will be idle for a time nj=1 Ij , and so the proportion of time that the server will be idle, which of course is just P0 , can be expressed as P0 = proportion of idle time I1 + · · · + In n→∞ I1 + · · · + In + B1 + · · · + Bn
= lim
Now it is easy to see that the I1 , I2 , . . . are independent and identically distributed as are the B1 , B2 , . . . . Hence, by dividing the numerator and the denominator of the right side of the preceding by n, and then applying the strong law of large numbers, we obtain P0 = lim
n→∞
=
(I1 + · · · + In )/n (I1 + · · · + In )/n + (B1 + · · · + Bn )/n
E[I ] E[I ] + E[B]
(8.35)
where I and B represent idle and busy time random variables. Now I represents the time from when a customer departs and leaves the system empty until the next arrival. Hence, from Poisson arrivals, it follows that I is exponential with rate λ, and so E[I ] =
1 λ
(8.36)
To compute P0 , we note from Equation (8.4) (obtained from the fundamental cost equation by supposing that a customer pays at a rate of one per unit time while in service) that average number of busy servers = λE[S]
8.6.
Variations on the M/G/1
531
However, as the left-hand side of the preceding equals 1 − P0 (why?), we have P0 = 1 − λE[S]
(8.37)
and, from Equations (8.35)–(8.37), 1 − λE[S] =
1/λ 1/λ + E[B]
or E[B] =
E[S] 1 − λE[S]
Another quantity of interest is C, the number of customers served in a busy period. The mean of C can be computed by noting that, on the average, for every E[C] arrivals exactly one will find the system empty (namely, the first customer in the busy period). Hence, a0 =
1 E[C]
and, as a0 = P0 = 1 − λE[S] because of Poisson arrivals, we see that E[C] =
8.6.
1 1 − λE[S]
Variations on the M/G/1
8.6.1. The M/G/1 with Random-Sized Batch Arrivals Suppose that, as in the M/G/1, arrivals occur in accordance with a Poisson process having rate λ. But now suppose that each arrival consists not of a single customer but of a random number of customers. As before there is a single server whose service times have distribution G. Let us denote by αj , j 1, the probability that an arbitrary batch consists of j customers; and let N denote a random variable representing the size of a batch and so P {N = j } = αj . Since λa = λE(N), the basic formula for work [Equation (8.31)] becomes . / E(S 2 ) V = λE[N ] E(S)WQ + (8.38) 2 To obtain a second equation relating V to WQ , consider an average customer. We have that
532
8
Queueing Theory
his wait in queue = work in system when he arrives + his waiting time due to those in his batch Taking expectations and using the fact that Poisson arrivals see time averages yields WQ = V + E[waiting time due to those in his batch] = V + E[WB ]
(8.39)
Now, E(WB ) can be computed by conditioning on the number in the batch, but we must be careful because the probability that our average customer comes from a batch of size j is not αj . For αj is the proportion of batches that are of size j , and if we pick a customer at random, it is more likely that he comes from a larger rather than a smaller batch. (For instance, suppose α1 = α100 = 12 , then half the batches are of size 1 but 100/101 of the customers will come from a batch of size 100!) To determine the probability that our average customer came from a batch of size j we reason as follows: Let M be a large number. Then of the first M batches approximately Mαj will be of size j , j 1, and thus there would have been approximately j Mαj customers that arrived in a batch of size j . Hence, the proportion of arrivals in the first M batches that were from batches of size j is approximately j Mαj / j j Mαj . This proportion becomes exact as M → ∞, and so we see that j αj proportion of customers from batches of size j = j j αj =
j αj E[N ]
We are now ready to compute E(WB ), the expected wait in queue due to others in the batch: j αj (8.40) E[WB | batch of size j ] E[WB ] = E[N ] j
Now if there are j customers in his batch, then our customer would have to wait for i − 1 of them to be served if he was ith in line among his batch members. As he is equally likely to be either 1st, 2nd, . . . , or j th in line we see that E[WB | batch is of size j ] =
j
(i − 1)E(S)
i=1
=
j −1 E[S] 2
1 j
8.6.
Variations on the M/G/1
533
Substituting this in Equation (8.40) yields E[WB ] =
E[S] (j − 1)j αj 2E[N ] j
=
E[S](E[N 2 ] − E[N ]) 2E[N ]
and from Equations (8.38) and (8.39) we obtain WQ =
E[S](E[N 2 ] − E[N ])/2E[N ] + λE[N ]E[S 2 ]/2 1 − λE[N ]E[S]
Remarks (i) Note that the condition for WQ to be finite is that λE(N)
0, then the situation when the nth repair has just occurred is that repair is about to begin on a machine, there are i − 1 other machines waiting for repair, and there are m − i working machines, each of which will (independently) continue to work for an exponential time with rate λ. Similarly, if Xn = 0, then all m machines are working and will (independently) continue to do so for exponentially distributed times with rate λ. Consequently, any information about earlier states of the system will not affect the probability distribution of the number of down machines at the moment of the next repair completion; hence, {Xn , n 1} is a Markov chain. To determine its transition probabilities Pi,j , suppose first that i > 0. Conditioning on R, the length of the next repair time, and making use of the independence of the m − i remaining working times, yields that for j m − i Pi,i−1+j = P {j failures during R} ∞ P {j failures during R | R = r}g(r) dr = 0
=
0
∞ m − i
j
(1 − e−λr )j (e−λr )m−i−j g(r) dr
If i = 0, then, because the next repair will not begin until one of the machines fails, P0,j = P1,j ,
j m−1
550
8
Queueing Theory
Let πj , j = 0, . . . , m − 1, denote the stationary probabilities of this Markov chain. That is, they are the unique solution of πi Pi,j , πj = i m−1
πj = 1
j =0
Therefore, after explicitly determining the transition probabilities and solving the preceding equations, we would know the value of π0 , the proportion of repair completions that leaves all machines working. Let us say that the system is “on” when all machines are working and “off” otherwise. (Thus, the system is on when the repairperson is idle and off when he is busy.) As all machines are working when the system goes back on, it follows from the lack of memory property of the exponential that the system probabilistically starts over when it goes on. Hence, this on–off system is an alternating renewal process. Suppose that the system has just become on, thus starting a new cycle, and let Ri , i 1, be the time of the ith repair from that moment. Also, let N denote the number of repairs in the off (busy) time of the cycle. Then, it follows that B, the length of the off period, can be expressed as B=
N
Ri
i=1
Although N is not independent of the sequence R1 , R2 , . . . , it is easy to check that it is a stopping time for this sequence, and thus by Wald’s equation (see Exercise 13 of Chapter 7) we have that E[B] = E[N ]E[R] = E[N ]μR Also, since an on time will last until one of the machines fails, and since the minimum of independent exponential random variables is exponential with a rate equal to the sum of their rates, it follows that E[I ], the mean on (idle) time in a cycle, is given by E[I ] = 1/(mλ) Hence, PB , the proportion of time that the repairperson is busy, satisfies PB =
E[N ]μR E[N ]μR + 1/(mλ)
8.8.
A Finite Source Model
551
However, since, on average, one out of every E[N ] repair completions will leave all machines working, it follows that π0 =
1 E[N ]
Consequently, PB =
μR μR + π0 /(mλ)
(8.58)
Now focus attention on one of the machines, call it machine number 1, and let P1,R denote the proportion of time that machine 1 is being repaired. Since the proportion of time that the repairperson is busy is PB , and since all machines fail at the same rate and have the same repair distribution, it follows that P1,R =
μR PB = m mμR + π0 /λ
(8.59)
However, machine 1 alternates between time periods when it is working, when it is waiting in queue, and when it is in repair. Let Wi , Qi , Si denote, respectively, the ith working time, the ith queueing time, and the ith repair time of machine 1, i 1. Then, the proportion of time that machine 1 is being repaired during its first n working–queue–repair cycles is: proportion of time in the first n cycles that machine 1 is being repaired n Si i=1 n = n n W + i=1 i i=1 Qi + i=1 Si n Si /n i=1 n = n n W /n + i=1 i i=1 Qi /n + i=1 Si /n Letting n → ∞ and using the strong law of large numbers to conclude that the averages of the Wi and of the Si converge, respectively, to 1/λ and μR , yields that μR P1,R = 1/λ + Q¯ + μR ¯ is the average amount of time that machine 1 spends in queue when it where Q fails. Using Equation (8.59), the preceding gives that μR μR = mμR + π0 /λ 1/λ + Q¯ + μR or, equivalently, that Q¯ = (m − 1)μR − (1 − π0 )/λ ¯ is Moreover, since all machines are probabilistically equivalent it follows that Q equal to WQ , the average amount of time that a failed machine spends in queue.
552
8
Queueing Theory
To determine the average number of machines in queue, we will make use of the basic queueing identity ¯ LQ = λ a W Q = λ a Q where λa is the average rate at which machines fail. To determine λa , again focus attention on machine 1 and suppose that we earn one per unit time whenever machine 1 is being repaired. It then follows from the basic cost identity of Equation (8.1) that P1,R = r1 μR where r1 is the average rate at which machine 1 fails. Thus, from Equation (8.59), we obtain that 1 r1 = mμR + π0 /λ Because all m machines fail at the same rate, the preceding implies that m λa = mr1 = mμR + π0 /λ which gives that the average number of machines in queue is LQ =
m(m − 1)μR − m(1 − π0 )/λ mμR + π0 /λ
Since the average number of machines being repaired is PB , the preceding, along with Equation (8.58), shows that the average number of down machines is L = LQ + PB =
8.9.
m2 μR − m(1 − π0 )/λ mμR + π0 /λ
Multiserver Queues
By and large, systems that have more than one server are much more difficult to analyze than those with a single server. In Section 8.9.1 we start first with a Poisson arrival system in which no queue is allowed, and then consider in Section 8.9.2 the infinite capacity M/M/k system. For both of these models we are able to present the limiting probabilities. In Section 8.9.3 we consider the model G/M/k. The analysis here is similar to that of the G/M/1 (Section 8.7) except that in place of a single quantity β given as the solution of an integral equation, we have k such quantities. We end in Section 8.9.4 with the model M/G/k for which unfortunately our previous technique (used in M/G/1) no longer enables us to derive WQ , and we content ourselves with an approximation.
8.9.
Multiserver Queues
553
8.9.1. Erlang’s Loss System A loss system is a queueing system in which arrivals that find all servers busy do not enter but rather are lost to the system. The simplest such system is the M/M/k loss system in which customers arrive according to a Poisson process having rate λ, enter the system if at least one of the k servers is free, and then spend an exponential amount of time with rate μ being served. The balance equations for this system are Rate leave = rate enter λP0 = μP1 (λ + μ)P1 = 2μP2 + λP0 (λ + 2μ)P2 = 3μP3 + λP1 (λ + iμ)Pi = (i + 1)μPi+1 + λPi−1 kμPk = λPk−1
State 0 1 2 i, 0 < i < k k Rewriting gives
λP0 = μP1 , λP1 = 2μP2 , λP2 = 3μP3 , .. . λPk−1 = kμPk or P1 =
λ P0 , μ
P2 =
λ (λ/μ)2 P1 = P0 , 2μ 2
P3 =
λ (λ/μ)3 P2 = P0 , 3μ 3!
.. . Pk = and using
k
0 Pi
λ (λ/μ)k Pk−1 = P0 kμ k!
= 1, we obtain (λ/μ)i /i! , Pi = k j j =0 (λ/μ) /j !
i = 0, 1, . . . , k
554
8
Queueing Theory
Since E[S] = 1/μ, where E[S] is the mean service time, the preceding can be written as (λE[S])i /i! Pi = k , j j =0 (λE[S]) /j !
i = 0, 1, . . . , k
(8.60)
Consider now the same system except that the service distribution is general— that is, consider the M/G/k with no queue allowed. This model is sometimes called the Erlang loss system. It can be shown (though the proof is advanced) that Equation (8.60) (which is called Erlang’s loss formula) remains valid for this more general system. 8.9.2. The M/M/k Queue The M/M/k infinite capacity queue can be analyzed by the balance equation technique. We leave it for you to verify that ⎧ (λ/μ)i ⎪ ⎪ ⎪ ⎪ ⎪ i! ⎪ , ik ⎪ k−1 ⎪ i ⎪ (λ/μ) (λ/μ)k kμ ⎨ + Pi = i! k! kμ − λ ⎪ i=0 ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ i k ⎪ ⎪ ⎩ (λ/kμ) k P0 , i>k k! We see from the preceding that we need to impose the condition λ < kμ. 8.9.3. The G/M/k Queue In this model we again suppose that there are k servers, each of whom serves at an exponential rate μ. However, we now allow the time between successive arrivals to have an arbitrary distribution G. To ensure that a steady-state (or limiting) distribution exists, we assume the condition 1/μG < kμ where μG is the mean of G.* The analysis for this model is similar to that presented in Section 8.7 for the case k = 1. Namely, to avoid having to keep track of the time since the last arrival, we look at the system only at arrival epochs. Once again, if we define Xn as the number in the system at the moment of the nth arrival, then {Xn , n 0} is a Markov chain. * It follows from the renewal theory (Proposition 7.1) that customers arrive at rate 1/μ , and as G the maximum service rate is kμ, we clearly need that 1/μG < kμ for limiting probabilities to exist.
8.9.
Multiserver Queues
555
To derive the transition probabilities of the Markov chain, it helps to first note the relationship Xn+1 = Xn + 1 − Yn ,
n0
where Yn denotes the number of departures during the interarrival time between the nth and (n + 1)st arrival. The transition probabilities Pij can now be calculated as follows:
Case 1: j > i + 1.
In this case it easily follows that Pij = 0.
Case 2: j i + 1 k. In this case if an arrival finds i in the system, then as i < k the new arrival will also immediately enter service. Hence, the next arrival will find j if of the i + 1 services exactly i + 1 − j are completed during the interarrival time. Conditioning on the length of this interarrival time yields Pij = P {i + 1 − j of i + 1 services are completed in an interarrival time} ∞ P {i + 1 − j of i + 1 are completed|interarrival time is t} dG(t) = 0
=
0
∞ i
+1 (i − e−μt )i+1−j (e−μt )j dG(t) j
where the last equality follows since the number of service completions in a time t will have a binomial distribution.
Case 3: i + 1 j k. To evaluate Pij in this case we first note that when all servers are busy, the departure process is a Poisson process with rate kμ (why?). Hence, again conditioning on the interarrival time we have Pij = P {i + 1 − j departures} ∞ P {i + 1 − j departures in time t} dG(t) = 0
=
∞
0
e−kμt
(kμt)i+1−j dG(t) (i + 1 − j )!
Case 4: i + 1 k > j . In this case since when all servers are busy the departure process is a Poisson process, it follows that the length of time until there will only be k in the system will have a gamma distribution with parameters i + 1 − k, kμ (the time until
556
8
Queueing Theory
i + 1 − k events of a Poisson process with rate kμ occur is gamma distributed with parameters i + 1 − k, kμ). Conditioning first on the interarrival time and then on the time until there are only k in the system (call this latter random variable Tk ) yields ∞ Pij = P {i + 1 − j departures in time t} dG(t) 0
=
0
=
0
∞ t 0
∞ t 0
P {i + 1 − j departures in t | Tk = s}kμe−kμs
(kμs)i−k ds dG(t) (i − k)!
k−j −μ(t−s) j k (kμs)i−k 1 − e−μ(t−s) ds dG(t) e kμe−kμs j (i − k)!
where the last equality follows since of the k people in service at time s the number whose service will end by time t is binomial with parameters k and 1 − e−μ(t−s) . We now can verify either by a direct substitution into the equations πj = i πi Pij , or by the same argument as presented in the remark at the end of Section 8.7, that the limiting probabilities of this Markov chain are of the form πk−1+j = cβ j ,
j = 0, 1, . . . . Substitution into any of the equations πj = i πi Pij when j > k yields that β is given as the solution of ∞ β= e−kμt (1−β) dG(t) 0
The values π0 , π1 , . . . , πk−2 , can be obtained by recursively solving the first k − 1 of the steady-state equations, and c can then be computed by using ∞ 0 πi = 1. ∗ denote the amount of time that a customer spends in queue, then If we let WQ in exactly the same manner as in G/M/1 we can show that ⎧ cβ ⎪ with probability k−1 πi = 1 − 1−β ⎨0, 0 ∗ WQ = ⎪ cβ ⎩Exp(kμ(1 − β)), with probability ∞ k πi = 1−β where Exp(kμ(1 − β)) is an exponential random variable with rate kμ(1 − β). 8.9.4. The M/G/k Queue In this section we consider the M/G/k system in which customers arrive at a Poisson rate λ and are served by any of k servers, each of whom has the service
8.9.
557
Multiserver Queues
distribution G. If we attempt to mimic the analysis presented in Section 8.5 for the M/G/1 system, then we would start with the basic identity V = λE[S]WQ + λE[S 2 ]/2
(8.61)
and then attempt to derive a second equation relating V and WQ . Now if we consider an arbitrary arrival, then we have the following identity: work in system when customer arrives = k × time customer spends in queue + R
(8.62)
where R is the sum of the remaining service times of all other customers in service at the moment when our arrival enters service. The foregoing follows because while the arrival is waiting in queue, work is being processed at a rate k per unit time (since all servers are busy). Thus, an amount of work k × time in queue is processed while he waits in queue. Now, all of this work was present when he arrived and in addition the remaining work on those still being served when he enters service was also present when he arrived— so we obtain Equation (8.62). For an illustration, suppose that there are three servers all of whom are busy when the customer arrives. Suppose, in addition, that there are no other customers in the system and also that the remaining service times of the three people in service are 3, 6, and 7. Hence, the work seen by the arrival is 3 + 6 + 7 = 16. Now the arrival will spend 3 time units in queue, and at the moment he enters service, the remaining times of the other two customers are 6 − 3 = 3 and 7 − 3 = 4. Hence, R = 3 + 4 = 7 and as a check of Equation (8.62) we see that 16 = 3 × 3 + 7. Taking expectations of Equation (8.62) and using the fact that Poisson arrivals see time averages, we obtain V = kWQ + E[R] which, along with Equation (8.61), would enable us to solve for WQ if we could compute E[R]. However there is no known method for computing E[R] and in fact, there is no known exact formula for WQ . The following approximation for WQ was obtained in Reference 6 by using the foregoing approach and then approximating E[R]: WQ ≈
λk E[S 2 ](E[S])k−1 ,k−1 (λE[S])n (λE[S])k 2 + 2(k − 1)!(k − λE[S]) n! (k − 1)!(k − λE[S]) n=0
(8.63) The preceding approximation has been shown to be quite close to the WQ when the service distribution is gamma. It is also exact when G is exponential.
558
8
Queueing Theory
Exercises 1.
For the M/M/1 queue, compute (a) the expected number of arrivals during a service period and (b) the probability that no customers arrive during a service period. Hint: “Condition.”
*2. Machines in a factory break down at an exponential rate of six per hour. There is a single repairman who fixes machines at an exponential rate of eight per hour. The cost incurred in lost production when machines are out of service is $10 per hour per machine. What is the average cost rate incurred due to failed machines? 3. The manager of a market can hire either Mary or Alice. Mary, who gives service at an exponential rate of 20 customers per hour, can be hired at a rate of $3 per hour. Alice, who gives service at an exponential rate of 30 customers per hour, can be hired at a rate of $C per hour. The manager estimates that, on the average, each customer’s time is worth $1 per hour and should be accounted for in the model. If customers arrive at a Poisson rate of 10 per hour, then (a) what is the average cost per hour if Mary is hired? if Alice is hired? (b) find C if the average cost per hour is the same for Mary and Alice. 4. Suppose that a customer of the M/M/1 system spends the amount of time x > 0 waiting in queue before entering service. (a) Show that, conditional on the preceding, the number of other customers that were in the system when the customer arrived is distributed as 1 + P , where P is a Poisson random variable with mean λ. ∗ denote the amount of time that an M/M/1 customer spends in (b) Let WQ queue. As a byproduct of your analysis in part (a), show that ∗ P {WQ
x} =
1− 1−
λ μ λ μ
if x = 0 +
λ −(μ−λ)x ) μ (1 − e
if x > 0
∗ is the amount 5. It follows from Exercise 4 that if, in the M/M/1 model, WQ of time that a customer spends waiting in queue, then ∗ WQ =
0, Exp(μ − λ),
with probability 1 − λ/μ with probability λ/μ
where Exp(μ − λ) is an exponential random variable with rate μ − λ. Using this, ∗ ). find Var(WQ
Exercises
559
6. Two customers move about among three servers. Upon completion of service at server i, the customer leaves that server and enters service at whichever of the other two servers is free. (Therefore, there are always two busy servers.) If the service times at server i are exponential with rate μi , i = 1, 2, 3, what proportion of time is server i idle? *7. Show that W is smaller in an M/M/1 model having arrivals at rate λ and service at rate 2μ than it is in a two-server M/M/2 model with arrivals at rate λ and with each server at rate μ. Can you give an intuitive explanation for this result? Would it also be true for WQ ? 8. A group of n customers moves around among two servers. Upon completion of service, the served customer then joins the queue (or enters service if the server is free) at the other server. All service times are exponential with rate μ. Find the proportion of time that there are j customers at server 1, j = 0, . . . , n. 9. A facility produces items according to a Poisson process with rate λ. However, it has shelf space for only k items and so it shuts down production whenever k items are present. Customers arrive at the facility according to a Poisson process with rate μ. Each customer wants one item and will immediately depart either with the item or empty handed if there is no item available. (a) Find the proportion of customers that go away empty handed. (b) Find the average time that an item is on the shelf. (c) Find the average number of items on the shelf. Suppose now that when a customer does not find any available items it joins the “customers’ queue” as long as there are no more than n − 1 other customers waiting at that time. If there are n waiting customers then the new arrival departs without an item. (d) Set up the balance equations. (e) In terms of the solution of the balance equations, what is the average number of customers in the system? 10. A group of m customers frequents a single-server station in the following manner. When a customer arrives, he or she either enters service if the server is free or joins the queue otherwise. Upon completing service the customer departs the system, but then returns after an exponential time with rate θ . All service times are exponentially distributed with rate μ. (a) Define states and set up the balance equations. In terms of the solution of the balance equations, find (b) the average rate at which customers enter the station. (c) the average time that a customer spends in the station per visit.
560
8
Queueing Theory
11. Consider a single-server queue with Poisson arrivals and exponential service times having the following variation: Whenever a service is completed a departure occurs only with probability α. With probability 1 − α the customer, instead of leaving, joins the end of the queue. Note that a customer may be serviced more than once. (a) Set up the balance equations and solve for the steady-state probabilities, stating conditions for it to exist. (b) Find the expected waiting time of a customer from the time he arrives until he enters service for the first time. (c) What is the probability that a customer enters service exactly n times, n = 1, 2, . . .? (d) What is the expected amount of time that a customer spends in service (which does not include the time he spends waiting in line)? Hint: Use part (c). (e) What is the distribution of the total length of time a customer spends being served? Hint: Is it memoryless? 12. Exponential queueing systems in which the state is the number of customers in the system are known as birth and death queueing systems. For such a system, let λn denote the rate at which a new customer joins the system when it is in state n, and let μn denote the rate at which there is a departure from the system when in state n. (a) Give the quantities λn and μn for the M/M/1 queue with finite capacity N . (b) Write down the balance equations. (c) Show how the balance equations can be reduced to the set of equations λn Pn = μn+1 Pn+1 ,
n0
(d) Give a direct argument for the preceding equations. (e) Solve the preceding equations, and in doing so, give the condition that is needed for there to be a solution. (f) What is the average arrival rate λa ? (g) What is the average amount of time that a customer spends in the system? *13. A supermarket has two exponential checkout counters, each operating at rate μ. Arrivals are Poisson at rate λ. The counters operate in the following way: (i) One queue feeds both counters.
Exercises
561
(ii) One counter is operated by a permanent checker and the other by a stock clerk who instantaneously begins checking whenever there are two or more customers in the system. The clerk returns to stocking whenever he completes a service, and there are fewer than two customers in the system. (a) Let Pn = proportion of time there are n in the system. Set up equations for Pn and solve. (b) At what rate does the number in the system go from 0 to 1? from 2 to 1? (c) What proportion of time is the stock clerk checking? Hint: Be a little careful when there is one in the system. 14. Customers arrive at a single-service facility at a Poisson rate of 40 per hour. When two or fewer customers are present, a single attendant operates the facility, and the service time for each customer is exponentially distributed with a mean value of two minutes. However, when there are three or more customers at the facility, the attendant is joined by an assistant and, working together, they reduce the mean service time to one minute. Assuming a system capacity of four customers, (a) what proportion of time are both servers free? (b) each man is to receive a salary proportional to the amount of time he is actually at work servicing customers, the rate being the same for both. If together they earn $100 per day, how should this money be split? 15. Consider a sequential-service system consisting of two servers, A and B. Arriving customers will enter this system only if server A is free. If a customer does enter, then he is immediately served by server A. When his service by A is completed, he then goes to B if B is free, or if B is busy, he leaves the system. Upon completion of service at server B, the customer departs. Assuming that the (Poisson) arrival rate is two customers an hour, and that A and B serve at respective (exponential) rates of four and two customers an hour, (a) what proportion of customers enter the system? (b) what proportion of entering customers receive service from B? (c) what is the average number of customers in the system? (d) what is the average amount of time that an entering customer spends in the system? 16. Customers arrive at a two-server system according to a Poisson process having rate λ = 5. An arrival finding server 1 free will begin service with that server. An arrival finding server 1 busy and server 2 free will enter service with server 2. An arrival finding both servers busy goes away. Once a customer is served by ei-
562
8
Queueing Theory
ther server, he departs the system. The service times at server i are exponential with rates μi , where μ1 = 4, μ2 = 2. (a) What is the average time an entering customer spends in the system? (b) What proportion of time is server 2 busy? 17. Customers arrive at a two-server station in accordance with a Poisson process with a rate of two per hour. Arrivals finding server 1 free begin service with that server. Arrivals finding server 1 busy and server 2 free begin service with server 2. Arrivals finding both servers busy are lost. When a customer is served by server 1, she then either enters service with server 2 if 2 is free or departs the system if 2 is busy. A customer completing service at server 2 departs the system. The service times at server 1 and server 2 are exponential random variables with respective rates of four and six per hour. (a) What fraction of customers do not enter the system? (b) What is the average amount of time that an entering customer spends in the system? (c) What fraction of entering customers receives service from server 1? 18. Customers arrive at a two-server system at a Poisson rate λ. An arrival finding the system empty is equally likely to enter service with either server. An arrival finding one customer in the system will enter service with the idle server. An arrival finding two others in the system will wait in line for the first free server. An arrival finding three in the system will not enter. All service times are exponential with rate μ, and once a customer is served (by either server), he departs the system. (a) Define the states. (b) Find the long-run probabilities. (c) Suppose a customer arrives and finds two others in the system. What is the expected times he spends in the system? (d) What proportion of customers enter the system? (e) What is the average time an entering customer spends in the system? 19. The economy alternates between good and bad periods. During good times customers arrive at a certain single-server queueing system in accordance with a Poisson process with rate λ1 , and during bad times they arrive in accordance with a Poisson process with rate λ2 . A good time period lasts for an exponentially distributed time with rate α1 , and a bad time period lasts for an exponential time with rate α2 . An arriving customer will only enter the queueing system if the server is free; an arrival finding the server busy goes away. All service times are exponential with rate μ. (a) Define states so as to be able to analyze this system. (b) Give a set of linear equations whose solution will yield the long run proportion of time the system is in each state.
Exercises
563
In terms of the solutions of the equations in part (b), (c) what proportion of time is the system empty? (d) what is the average rate at which customers enter the system? 20. There are two types of customers. Type 1 and 2 customers arrive in accordance with independent Poisson processes with respective rate λ1 and λ2 . There are two servers. A type 1 arrival will enter service with server 1 if that server is free; if server 1 is busy and server 2 is free, then the type 1 arrival will enter service with server 2. If both servers are busy, then the type 1 arrival will go away. A type 2 customer can only be served by server 2; if server 2 is free when a type 2 customer arrives, then the customer enters service with that server. If server 2 is busy when a type 2 arrives, then that customer goes away. Once a customer is served by either server, he departs the system. Service times at server i are exponential with rate μi , i = 1, 2. Suppose we want to find the average number of customers in the system. (a) Define states. (b) Give the balance equations. Do not attempt to solve them. In terms of the long-run probabilities, what is (c) the average number of customers in the system? (d) the average time a customer spends in the system? *21. Suppose in Exercise 20 we want to find out the proportion of time there is a type 1 customer with server 2. In terms of the long-run probabilities given in Exercise 20, what is (a) (b) (c) (d)
the rate at which a type 1 customer enters service with server 2? the rate at which a type 2 customer enters service with server 2? the fraction of server 2’s customers that are type 1? the proportion of time that a type 1 customer is with server 2?
22. Customers arrive at a single-server station in accordance with a Poisson process with rate λ. All arrivals that find the server free immediately enter service. All service times are exponentially distributed with rate μ. An arrival that finds the server busy will leave the system and roam around “in orbit” for an exponential time with rate θ at which time it will then return. If the server is busy when an orbiting customer returns, then that customer returns to orbit for another exponential time with rate θ before returning again. An arrival that finds the server busy and N other customers in orbit will depart and not return. That is, N is the maximum number of customers in orbit. (a) Define states. (b) Give the balance equations. In terms of the solution of the balance equations, find (c) the proportion of all customers that are eventually served.
564
8
Queueing Theory
(d) the average time that a served customer spends waiting in orbit. 23. Consider the M/M/1 system in which customers arrive at rate λ and the server serves at rate μ. However, suppose that in any interval of length h in which the server is busy there is a probability αh + o(h) that the server will experience a breakdown, which causes the system to shut down. All customers that are in the system depart, and no additional arrivals are allowed to enter until the breakdown is fixed. The time to fix a breakdown is exponentially distributed with rate β. (a) Define appropriate states. (b) Give the balance equations. In terms of the long-run probabilities, (c) what is the average amount of time that an entering customer spends in the system? (d) what proportion of entering customers complete their service? (e) what proportion of customers arrive during a breakdown? *24. Reconsider Exercise 23, but this time suppose that a customer that is in the system when a breakdown occurs remains there while the server is being fixed. In addition, suppose that new arrivals during a breakdown period are allowed to enter the system. What is the average time a customer spends in the system? 25. Poisson (λ) arrivals join a queue in front of two parallel servers A and B, having exponential service rates μA and μB (see Figure 8.4). When the system is empty, arrivals go into server A with probability α and into B with probability 1 − α. Otherwise, the head of the queue takes the first free server. (a) Define states and set up the balance equations. Do not solve. (b) In terms of the probabilities in part (a), what is the average number in the system? Average number of servers idle? (c) In terms of the probabilities in part (a), what is the probability that an arbitrary arrival will get serviced in A? 26. In a queue with unlimited waiting space, arrivals are Poisson (parameter λ) and service times are exponentially distributed (parameter μ). However, the server waits until K people are present before beginning service on the first customer; thereafter, he services one at a time until all K units, and all subsequent arrivals, are serviced. The server is then “idle” until K new arrivals have occurred. (a) Define an appropriate state space, draw the transition diagram, and set up the balance equations.
Figure 8.4.
Exercises
565
(b) In terms of the limiting probabilities, what is the average time a customer spends in queue? (c) What conditions on λ and μ are necessary? 27. Consider a single-server exponential system in which ordinary customers arrive at a rate λ and have service rate μ. In addition, there is a special customer who has a service rate μ1 . Whenever this special customer arrives, she goes directly into service (if anyone else is in service, then this person is bumped back into queue). When the special customer is not being serviced, she spends an exponential amount of time (with mean 1/θ ) out of the system. (a) What is the average arrival rate of the special customer? (b) Define an appropriate state space and set up balance equations. (c) Find the probability that an ordinary customer is bumped n times. *28. Let D denote the time between successive departures in a stationary M/M/1 queue with λ < μ. Show, by conditioning on whether or not a departure has left the system empty, that D is exponential with rate λ. Hint: By conditioning on whether or not the departure has left the system empty we see that Exponential(μ), with probability λ/μ D= Exponential(λ) ∗ Exponential(μ), with probability 1 − λ/μ where Exponential(λ) ∗ Exponential(μ) represents the sum of two independent exponential random variables having rates μ and λ. Now use momentgenerating functions to show that D has the required distribution. Note that the preceding does not prove that the departure process is Poisson. To prove this we need show not only that the interdeparture times are all exponential with rate λ, but also that they are independent. 29. Potential customers arrive to a single server hair salon according to a Poisson process with rate λ. A potential customer who finds the server free enters the system; a potential customer who finds the server busy goes away. Each potential customer is type i with probability pi , where p1 + p2 + p3 = 1. Type 1 customers have their hair washed by the server; type 2 customers have their hair cut by the server; and type 3 customers have their hair first washed and then cut by the server. The time that it takes the server to wash hair is exponentially distributed with rate μ1 , and the time that it takes the server to cut hair is exponentially distributed with rate μ2 . (a) Explain how this system can be analyzed with four states. (b) Give the equations whose solution yields the proportion of time the system is in each state.
566
8
Queueing Theory
In terms of the solution of the equations of (b), find (c) the proportion of time the server is cutting hair; (d) the average arrival rate of entering customers. 30. For the tandem queue model verify that Pn,m = (λ/μ1 )n (1 − λ/μ1 )(λ/μ2 )m (1 − λ/μ2 ) satisfies the balance equation (8.15). 31. Consider a network of three stations. Customers arrive at stations 1, 2, 3 in accordance with Poisson processes having respective rates, 5, 10, 15. The service times at the three stations are exponential with respective rates 10, 50, 100. A customer completing service at station 1 is equally likely to (i) go to station 2, (ii) go to station 3, or (iii) leave the system. A customer departing service at station 2 always goes to station 3. A departure from service at station 3 is equally likely to either go to station 2 or leave the system. (a) What is the average number of customers in the system (consisting of all three stations)? (b) What is the average time a customer spends in the system? 32. Consider a closed queueing network consisting of two customers moving among two servers, and suppose that after each service completion the customer is equally likely to go to either server—that is, P1,2 = P2,1 = 12 . Let μi denote the exponential service rate at server i, i = 1, 2. (a) Determine the average number of customers at each server. (b) Determine the service completion rate for each server. 33. Explain how a Markov chain Monte Carlo simulation using the Gibbs sampler can be utilized to estimate (a) the distribution of the amount of time spent at server j on a visit. Hint: Use the arrival theorem. (b) the proportion of time a customer is with server j (i.e., either in server j ’s queue or in service with j ). 34. For open queueing networks (a) state and prove the equivalent of the arrival theorem; (b) derive an expression for the average amount of time a customer spends waiting in queues. 35. Customers arrive at a single-server station in accordance with a Poisson process having rate λ. Each customer has a value. The successive values of customers are independent and come from a uniform distribution on (0, 1). The service time of a customer having value x is a random variable with mean 3 + 4x and variance 5.
Exercises
567
(a) What is the average time a customer spends in the system? (b) What is the average time a customer having value x spends in the system? *36. Compare the M/G/1 system for first-come, first-served queue discipline with one of last-come, first-served (for instance, in which units for service are taken from the top of a stack). Would you think that the queue size, waiting time, and busy-period distribution differ? What about their means? What if the queue discipline was always to choose at random among those waiting? Intuitively which discipline would result in the smallest variance in the waiting time distribution? 37. In an M/G/1 queue, (a) what proportion of departures leave behind 0 work? (b) what is the average work in the system as seen by a departure? 38. For the M/G/1 queue, let Xn denote the number in the system left behind by the nth departure. (a) If Xn+1 =
Xn − 1 + Yn , Yn ,
if Xn 1 if Xn = 0
what does Yn represent? (b) Rewrite the preceding as Xn+1 = Xn − 1 + Yn + δn
(8.64)
where δn =
1, 0,
if Xn = 0 if Xn 1
Take expectations and let n → ∞ in Equation (8.64) to obtain E[δ∞ ] = 1 − λE[S] (c) Square both sides of Equation (8.64), take expectations, and then let n → ∞ to obtain E[X∞ ] =
λ2 E[S 2 ] + λE[S] 2(1 − λE[S])
(d) Argue that E[X∞ ], the average number as seen by a departure, is equal to L.
568
8
Queueing Theory
*39. Consider an M/G/1 system in which the first customer in a busy period has the service distribution G1 and all others have distribution G2 . Let C denote the number of customers in a busy period, and let S denote the service time of a customer chosen at random. Argue that (a) a0 = P0 = 1 − λE[S]. (b) E[S] = a0 E[S1 ] + (1 − a0 )E[S2 ] where Si has distribution Gi . (c) Use (a) and (b) to show that E[B], the expected length of a busy period, is given by E[B] =
E[S1 ] 1 − λE[S2 ]
(d) Find E[C]. 40. Consider a M/G/1 system with λE[S] < 1. (a) Suppose that service is about to begin at a moment when there are n customers in the system. (i) Argue that the additional time until there are only n − 1 customers in the system has the same distribution as a busy period. (ii) What is the expected additional time until the system is empty? (b) Suppose that the work in the system at some moment is A. We are interested in the expected additional time until the system is empty—call it E[T ]. Let N denote the number of arrivals during the first A units of time. (i) Compute E[T |N ]. (ii) Compute E[T ]. 41. Carloads of customers arrive at a single-server station in accordance with a Poisson process with rate 4 per hour. The service times are exponentially distributed with rate 20 per hour. If each carload contains either 1, 2, or 3 customers with respective probabilities 14 , 12 , 14 compute the average customer delay in queue. 42. In the two-class priority queueing model of Section 8.6.2, what is WQ ? Show that WQ is less than it would be under FIFO if E[S1 ] < E[S2 ] and greater than under FIFO if E[S1 ] > E[S2 ]. 43. In a two-class priority queueing model suppose that a cost of Ci per unit time is incurred for each type i customer that waits in queue, i = 1, 2. Show that type 1 customers should be given priority over type 2 (as opposed to the reverse) if E[S1 ] E[S2 ] < C1 C2
Exercises
569
44. Consider the priority queueing model of Section 8.6.2 but now suppose that if a type 2 customer is being served when a type 1 arrives then the type 2 customer is bumped out of service. This is called the preemptive case. Suppose that when a bumped type 2 customer goes back in service his service begins at the point where it left off when he was bumped. (a) Argue that the work in the system at any time is the same as in the nonpreemptive case. 1. (b) Derive WQ Hint: How do type 2 customers affect type 1s? (c) Why is it not true that 2 VQ2 = λ2 E[S2 ]WQ
(d) Argue that the work seen by a type 2 arrival is the same as in the nonpreemptive case, and so 2 2 WQ = WQ (nonpreemptive) + E[extra time]
where the extra time is due to the fact that he may be bumped. (e) Let N denote the number of times a type 2 customer is bumped. Why is E[extra time|N ] =
N E[S1 ] 1 − λ1 E[S1 ]
Hint: When a type 2 is bumped, relate the time until he gets back in service to a “busy period.” (f) Let S2 denote the service time of a type 2. What is E[N |S2 ]? (g) Combine the preceding to obtain 2 2 = WQ (nonpreemptive) + WQ
λ1 E[S1 ]E[S2 ] 1 − λ1 E[S1 ]
*45. Calculate explicitly (not in terms of limiting probabilities) the average time a customer spends in the system in Exercise 24. 46. In the G/M/1 model if G is exponential with rate λ show that β = λ/μ. 47. Verify Erlang’s loss formula, Equation (8.60), when k = 1. 48. Verify the formula given for the Pi of the M/M/k. 49. In the Erlang loss system suppose the Poisson arrival rate is λ = 2, and suppose there are three servers, each of whom has a service distribution that is uniformly distributed over (0, 2). What proportion of potential customers is lost?
570
8
Queueing Theory
50. In the M/M/k system, (a) what is the probability that a customer will have to wait in queue? (b) determine L and W . ∗ given for the G/M/k model. 51. Verify the formula for the distribution of WQ
*52. Consider a system where the interarrival times have an arbitrary distribution F , and there is a single server whose service distribution is G. Let Dn denote the amount of time the nth customer spends waiting in queue. Interpret Sn , Tn so that if Dn + Sn − Tn 0 D n + S n − Tn , Dn+1 = 0, if Dn + Sn − Tn < 0 53. Consider a model in which the interarrival times have an arbitrary distribution F , and there are k servers each having service distribution G. What condition on F and G do you think would be necessary for there to exist limiting probabilities?
References 1. J. Cohen, “The Single Server Queue,” North-Holland, Amsterdam, 1969. 2. R. B. Cooper, “Introduction to Queueing Theory,” Second Edition, Macmillan, New York, 1984. 3. D. R. Cox and W. L. Smith, “Queues,” Wiley, New York, 1961. 4. F. Kelly, “Reversibility and Stochastic Networks,” Wiley, New York, 1979. 5. L. Kleinrock, “Queueing Systems,” Vol. I, Wiley, New York, 1975. 6. S. Nozaki and S. Ross, “Approximations in Finite Capacity Multiserver Queues with Poisson Arrivals,” J. Appl. Prob. 13, 826–834 (1978). 7. L. Takacs, “Introduction to the Theory of Queues,” Oxford University Press, London and New York, 1962. 8. H. Tijms, “Stochastic Models: An Algorithmic Approach,” Wiley, New York, 1994. 9. P. Whittle, “Systems in Stochastic Equilibrium,” Wiley, New York, 1986. 10. Wolff, “Stochastic Modeling and the Theory of Queues,” Prentice Hall, New Jersey, 1989.
Reliability Theory
9 9.1.
Introduction
Reliability theory is concerned with determining the probability that a system, possibly consisting of many components, will function. We shall suppose that whether or not the system functions is determined solely from a knowledge of which components are functioning. For instance, a series system will function if and only if all of its components are functioning, while a parallel system will function if and only if at least one of its components is functioning. In Section 9.2, we explore the possible ways in which the functioning of the system may depend upon the functioning of its components. In Section 9.3, we suppose that each component will function with some known probability (independently of each other) and show how to obtain the probability that the system will function. As this probability often is difficult to explicitly compute, we also present useful upper and lower bounds in Section 9.4. In Section 9.5 we look at a system dynamically over time by supposing that each component initially functions and does so for a random length of time at which it fails. We then discuss the relationship between the distribution of the amount of time that a system functions and the distributions of the component lifetimes. In particular, it turns out that if the amount of time that a component functions has an increasing failure rate on the average (IFRA) distribution, then so does the distribution of system lifetime. In Section 9.6 we consider the problem of obtaining the mean lifetime of a system. In the final section we analyze the system when failed components are subjected to repair.
9.2.
Structure Functions
Consider a system consisting of n components, and suppose that each component is either functioning or has failed. To indicate whether or not the ith component
571
572
9
Reliability Theory
is functioning, we define the indicator variable xi by 1, if the ith component is functioning xi = 0, if the ith component has failed The vector x = (x1 , . . . , xn ) is called the state vector. It indicates which of the components are functioning and which have failed. We further suppose that whether or not the system as a whole is functioning is completely determined by the state vector x. Specifically, it is supposed that there exists a function φ(x) such that 1, if the system is functioning when the state vector is x φ(x) = 0, if the system has failed when the state vector is x The function φ(x) is called the structure function of the system.
Example 9.1 (The Series Structure) A series system functions if and only if all of its components are functioning. Hence, its structure function is given by φ(x) = min(x1 , . . . , xn ) =
n
xi
i=1
We shall find it useful to represent the structure of a system in terms of a diagram. The relevant diagram for the series structure is shown in Figure 9.1. The idea is that if a signal is initiated at the left end of the diagram then in order for it to successfully reach the right end, it must pass through all of the components; hence, they must all be functioning.
Figure 9.1. A series system.
Example 9.2 (The Parallel Structure) A parallel system functions if and only if at least one of its components is functioning. Hence its structure function is given by φ(x) = max(x1 , . . . , xn ) A parallel structure may be pictorially illustrated by Figure 9.2. This follows since a signal at the left end can successfully reach the right end as long as at least one component is functioning.
9.2.
Structure Functions
573
Figure 9.2. A parallel system.
Example 9.3 (The k-out-of-n Structure) The series and parallel systems are both special cases of a k-out-of-n system. Such a system functions if and only if at least k of the n components are functioning. As ni=1 xi equals the number of functioning components, the structure function of a k-out-of-n system is given by
φ(x) =
⎧ ⎪ ⎪ ⎪ 1, ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩0,
if
if
n i=1 n
xi k xi < k
i=1
Series and parallel systems are respectively n-out-of-n and 1-out-of-n systems. The two-out-of-three system may be diagrammed as shown in Figure 9.3.
Figure 9.3. A 2-out-of-3 system.
Example 9.4 (A Four-Component Structure) Consider a system consisting of four components, and suppose that the system functions if and only if components 1 and 2 both function and at least one of components 3 and 4 function. Its structure function is given by φ(x) = x1 x2 max(x3 , x4 )
574
9
Reliability Theory
Pictorially, the system is as shown in Figure 9.4. A useful identity, easily checked, is that for binary variables,∗ xi , i = 1, . . . , n, max(x1 , . . . , xn ) = 1 −
n
(1 − xi )
i=1
Figure 9.4.
When n = 2, this yields max(x1 , x2 ) = 1 − (1 − x1 )(1 − x2 ) = x1 + x2 − x1 x2 Hence, the structure function in the example may be written as φ(x) = x1 x2 (x3 + x4 − x3 x4 )
It is natural to assume that replacing a failed component by a functioning one will never lead to a deterioration of the system. In other words, it is natural to assume that the structure function φ(x) is an increasing function of x, that is, if xi yi , i = 1, . . . , n, then φ(x) φ(y). Such an assumption shall be made in this chapter and the system will be called monotone. 9.2.1. Minimal Path and Minimal Cut Sets In this section we show how any system can be represented both as a series arrangement of parallel structures and as a parallel arrangement of series structures. As a preliminary, we need the following concepts. A state vector x is called a path vector if φ(x) = 1. If, in addition, φ(y) = 0 for all y < x, then x is said to be a minimal path vector.∗∗ If x is a minimal path vector, then the set A = {i : xi = 1} is called a minimal path set. In other words, a minimal path set is a minimal set of components whose functioning ensures the functioning of the system. ∗ A binary variable is one that assumes either the value 0 or 1. ∗∗ We say that y < x if y x , i = 1, . . . , n, with y < x for some i. i i i i
9.2.
Structure Functions
575
Example 9.5 Consider a five-component system whose structure is illustrated by Figure 9.5. Its structure function equals φ(x) = max(x1 , x2 ) max(x3 x4 , x5 ) = (x1 + x2 − x1 x2 )(x3 x4 + x5 − x3 x4 x5 ) There are four minimal path sets, namely, {1, 3, 4}, {2, 3, 4}, {1, 5}, {2, 5}.
Figure 9.5.
Example 9.6 In a k-out-of-n system, there are all of the sets consisting of exactly k components.
n k
minimal path sets, namely,
Let A1 , . . . , As denote the minimal path sets of a given system. We define αj (x), the indicator function of the j th minimal path set, by αj (x) = =
1, 0,
if all the components of Ai are functioning otherwise xi
i∈Aj
By definition, it follows that the system will function if all the components of at least one minimal path set are functioning; that is, if αj (x) = 1 for some j . On the other hand, if the system functions, then the set of functioning components must include a minimal path set. Therefore, a system will function if and only if all the components of at least one minimal path set are functioning. Hence, 1, if αj (x) = 1 for some j φ(x) = 0, if αj (x) = 0 for all j or equivalently φ(x) = max αj (x) j
= max j
i∈Aj
xi
(9.1)
576
9
Reliability Theory
Since αj (x) is a series structure function of the components of the j th minimal path set, Equation (9.1) expresses an arbitrary system as a parallel arrangement of series systems.
Example 9.7 Consider the system of Example 9.5. Because its minimal path sets are A1 = {1, 3, 4}, A2 = {2, 3, 4}, A3 = {1, 5}, and A4 = {2, 5}, we have by Equation (9.1) that φ(x) = max{x1 x3 x4 , x2 x3 x4 , x1 x5 , x2 x5 } = 1 − (1 − x1 x3 x4 )(1 − x2 x3 x4 )(1 − x1 x5 )(1 − x2 x5 )
Figure 9.6.
You should verify that this equals the value of φ(x) given in Example 9.5. (Make use of the fact that, since xi equals 0 or 1, xi2 = xi .) This representation may be pictured as shown in Figure 9.6.
Figure 9.7. The bridge system.
Example 9.8 The system whose structure is as pictured in Figure 9.7 is called the bridge system. Its minimal path sets are {1, 4}, {1, 3, 5}, {2, 5}, and {2, 3, 4}. Hence, by Equation (9.1), its structure function may be expressed as φ(x) = max{x1 x4 , x1 x3 x5 , x2 x5 , x2 x3 x4 } = 1 − (1 − x1 x4 )(1 − x1 x3 x5 )(1 − x2 x5 )(1 − x2 x3 x4 ) This representation φ(x) is diagrammed as shown in Figure 9.8.
9.2.
Structure Functions
577
Figure 9.8.
A state vector x is called a cut vector if φ(x) = 0. If, in addition, φ(y) = 1 for all y > x, then x is said to be a minimal cut vector. If x is a minimal cut vector, then the set C = {i : xi = 0} is called a minimal cut set. In other words, a minimal cut set is a minimal set of components whose failure ensures the failure of the system. Let C1 , . . . , Ck denote the minimal cut sets of a given system. We define βj (x), the indicator function of the j th minimal cut set, by ⎧ 1, if at least one component of the j th minimal ⎪ ⎪ ⎨ cut set is functioning βj (x) = 0, if all of the components of the j th minimal ⎪ ⎪ ⎩ cut set are not functioning = max xi i∈Cj
Since a system is not functioning if and only if all the components of at least one minimal cut set are not functioning, it follows that φ(x) =
k j =1
βj (x) =
k j =1
max xi
(9.2)
i∈Cj
Since βj (x) is a parallel structure function of the components of the j th minimal cut set, Equation (9.2) represents an arbitrary system as a series arrangement of parallel systems.
Example 9.9 The minimal cut sets of the bridge structure shown in Figure 9.9 are {1, 2}, {1, 3, 5}, {2, 3, 4}, and {4, 5}. Hence, from Equation (9.2), we may express φ(x) by φ(x) = max(x1 , x2 ) max(x1 , x3 , x5 ) max(x2 , x3 , x4 ) max(x4 , x5 ) = [1 − (1 − x1 )(1 − x2 )][1 − (1 − x1 )(1 − x3 )(1 − x5 )] × [1 − (1 − x2 )(1 − x3 )(1 − x4 )][1 − (1 − x4 )(1 − x5 )] This representation of φ(x) is pictorially expressed as Figure 9.10.
578
9
Reliability Theory
Figure 9.9.
Figure 9.10. Minimal cut representation of the bridge system.
9.3.
Reliability of Systems of Independent Components
In this section, we suppose that Xi , the state of the ith component, is a random variable such that P {Xi = 1} = pi = 1 − P {Xi = 0} The value pi , which equals the probability that the ith component is functioning, is called the reliability of the ith component. If we define r by r = P {φ(X) = 1},
where X = (X1 , . . . , Xn )
then r is called the reliability of the system. When the components, that is, the random variables Xi , i = 1, . . . , n, are independent, we may express r as a function of the component reliabilities. That is, r = r(p),
where p = (p1 , . . . , pn )
The function r(p) is called the reliability function. We shall assume throughout the remainder of this chapter that the components are independent.
9.3.
Reliability of Systems of Independent Components
579
Example 9.10 (The Series System) The reliability function of the series system of n independent components is given by r(p) = P {φ(X) = 1} = P {Xi = 1 for all i = 1, . . . , n} =
n
pi
i=1
Example 9.11 (The Parallel System) The reliability function of the parallel system of n independent components is given by r(p) = P {φ(X) = 1} = P {Xi = 1 for some i = 1, . . . , n} = 1 − P {Xi = 0 for all i = 1, . . . , n} =1−
n
(1 − pi )
i=1
Example 9.12 (The k-out-of-n System with Equal Probabilities) Consider a k-out-of-n system. If pi = p for all i = 1, . . . , n, then the reliability function is given by r(p, . . . , p) = P {φ(X) = 1} n Xi k =P i=1
=
n i=k
n i p (1 − p)n−i i
Example 9.13 (The Two-out-of-Three System) The reliability function of a two-out-of-three system is given by r(p) = P {φ(X) = 1} = P {X = (1, 1, 1)} + P {X = (1, 1, 0)} + P {X = (1, 0, 1)} + P {X = (0, 1, 1)} = p1 p2 p3 + p1 p2 (1 − p3 ) + p1 (1 − p2 )p3 + (1 − p1 )p2 p3 = p1 p2 + p1 p3 + p2 p3 − 2p1 p2 p3
580
9
Reliability Theory
Example 9.14 (The Three-out-of-Four System) The reliability function of a three-out-of-four system is given by r(p) = P {X = (1, 1, 1, 1)} + P {X = (1, 1, 1, 0)} + P {X = (1, 1, 0, 1)} + P {X = (1, 0, 1, 1)} + P {X = (0, 1, 1, 1)} = p1 p2 p3 p4 + p1 p2 p3 (1 − p4 ) + p1 p2 (1 − p3 )p4 + p1 (1 − p2 )p3 p4 + (1 − p1 )p2 p3 p4 = p1 p2 p3 + p1 p2 p4 + p1 p3 p4 + p2 p3 p4 − 3p1 p2 p3 p4
Example 9.15 (A Five-Component System) Consider a five-component system that functions if and only if component 1, component 2, and at least one of the remaining components function. Its reliability function is given by r(p) = P {X1 = 1, X2 = 1, max(X3 , X4 , X5 ) = 1} = P {X1 = 1}P {X2 = 1}P {max(X3 , X4 , X5 ) = 1} = p1 p2 [1 − (1 − p3 )(1 − p4 )(1 − p5 )]
Since φ(X) is a 0–1 (that is, a Bernoulli) random variable, we may also compute r(p) by taking its expectation. That is, r(p) = P {φ(X) = 1} = E[φ(X)]
Example 9.16 (A Four-Component System) A four-component system that functions when both components 1, 4, and at least one of the other components function has its structure function given by φ(x) = x1 x4 max(x2 , x3 ) Hence, r(p) = E[φ(X)] = E[X1 X4 (1 − (1 − X2 )(1 − X3 ))] = p1 p4 [1 − (1 − p2 )(1 − p3 )]
An important and intuitive property of the reliability function r(p) is given by the following proposition.
Proposition 9.1 If r(p) is the reliability function of a system of independent components, then r(p) is an increasing function of p.
9.3.
Reliability of Systems of Independent Components
581
Proof By conditioning on Xi and using the independence of the components, we obtain r(p) = E[φ(X)] = pi E[φ(X) | Xi = 1] + (1 − pi )E[φ(X) | Xi = 0] = pi E[φ(1i , X)] + (1 − pi )E[φ(0i , X)] where (1i , X) = (X1 , . . . , Xi−1 , 1, Xi+1 , . . . , Xn ), (0i , X) = (X1 , . . . , Xi−1 , 0, Xi+1 , . . . , Xn ) Thus, r(p) = pi E[φ(1i , X) − φ(0i , X)] + E[φ(0i , X)] However, since φ is an increasing function, it follows that E[φ(1i , X) − φ(0i , X)] 0 and so the preceding is increasing in pi for all i. Hence the result is proven.
Let us now consider the following situation: A system consisting of n different components is to be built from a stockpile containing exactly two of each type of component. How should we use the stockpile so as to maximize our probability of attaining a functioning system? In particular, should we build two separate systems, in which case the probability of attaining a functioning one would be P {at least one of the two systems function} = 1 − P {neither of the systems function} = 1 − [(1 − r(p))(1 − r(p ))] where pi (pi ) is the probability that the first (second) number i component functions; or should we build a single system whose ith component functions if at least one of the number i components functions? In this latter case, the probability that the system will function equals r[1 − (1 − p)(1 − p )] since 1 − (1 − pi )(1 − pi ) equals the probability that the ith component in the single system will function.∗ We now show that replication at the component level is more effective than replication at the system level. ∗ Notation: If x = (x , . . . , x ), y = (y , . . . , y ), then xy = (x y , . . . , x y ). Also, max(x, y) = n n n n 1 1 1 1 (max(x1 , y1 ), . . . , max(xn , yn )) and min(x, y) = (min(x1 , y1 ), . . . , min(xn , yn )).
582
9
Reliability Theory
Theorem 9.1 For any reliability function r and vectors p, p , r[1 − (1 − p)(1 − p )] 1 − [1 − r(p)][1 − r(p )]
Proof Let X1 , . . . , Xn , X1 , . . . , Xn be mutually independent 0–1 random variables with pi = P {Xi = 1}
pi = P {Xi = 1},
Since P {max(Xi , Xi ) = 1} = 1 − (1 − pi )(1 − pi ), it follows that r[1 − (1 − p)(1 − p )] = E(φ[max(X, X )]) However, by the monotonicity of φ, we have that φ[max(X, X )] is greater than or equal to both φ(X) and (X ) and hence is at least as large as max[φ(X), φ(X )]. Hence, from the preceding we have that r[1 − (1 − p)(1 − p )] E[max(φ(X), φ(X ))] = P {max[φ(X), φ(X )] = 1} = 1 − P {φ(X) = 0, φ(X ) = 0} = 1 − [1 − r(p)][1 − r(p )] where the first equality follows from the fact that max[φ(X), φ(X )] is a 0–1 random variable and hence its expectation equals the probability that it equals 1. As an illustration of the preceding theorem, suppose that we want to build a series system of two different types of components from a stockpile consisting of two of each of the kinds of components. Suppose that the reliability of each component is 12 . If we use the stockpile to build two separate systems, then the probability of attaining a working system is 1−
3 2 4
=
7 16
while if we build a single system, replicating components, then the probability of attaining a working system is 3 2 4
=
9 16
Hence, replicating components leads to a higher reliability than replicating systems (as, of course, it must by Theorem 9.1).
9.4.
9.4.
Bounds on the Reliability Function
583
Bounds on the Reliability Function
Consider the bridge system of Example 9.8, which is represented by Figure 9.11. Using the minimal path representation, we have that φ(x) = 1 − (1 − x1 x4 )(1 − x1 x3 x5 )(1 − x2 x5 )(1 − x2 x3 x4 ) Hence, r(p) = 1 − E[(1 − X1 X4 )(1 − X1 X3 X5 )(1 − X2 X5 )(1 − X2 X3 X4 )]
Figure 9.11.
However, since the minimal path sets overlap (that is, they have components in common), the random variables (1 − X1 X4 ), (1 − X1 X3 X5 ), (1 − X2 X5 ), and (1−X2 X3 X4 ) are not independent, and thus the expected value of their product is not equal to the product of their expected values. Therefore, in order to compute r(p), we must first multiply the four random variables and then take the expected value. Doing so, using that Xi2 = Xi , we obtain r(p) = E[X1 X4 + X2 X5 + X1 X3 X5 + X2 X3 X4 − X1 X2 X3 X4 − X1 X2 X3 X5 − X1 X2 X4 X5 − X1 X3 X4 X5 − X2 X3 X4 X5 + 2X1 X2 X3 X4 X5 ] = p 1 p 4 + p 2 p 5 + p 1 p 3 p 5 + p2 p 3 p 4 − p1 p 2 p 3 p 4 − p 1 p 2 p 3 p 5 − p1 p2 p4 p5 − p1 p3 p4 p5 − p2 p3 p4 p5 + 2p1 p2 p3 p4 p5 As can be seen by the preceding example, it is often quite tedious to evaluate r(p), and thus it would be useful if we had a simple way of obtaining bounds. We now consider two methods for this.
584
9
Reliability Theory
9.4.1. Method of Inclusion and Exclusion The following is a well-known formula for the probability of the union of the events E1 , E2 , . . . , En : n n
Ei = P (Ei ) − P (Ei Ej ) + P (Ei Ej Ek ) P i=1
i<j
i=1
i<j m, we can rewrite Equa-
n N i=1
i
(−1)i+1
(9.8)
586
9
Reliability Theory
The equality (9.3) and inequalities (9.4) now follow upon taking expectations of (9.7) and (9.8). This is the case since n
Ej , E[I ] = P {N > 0} = P {at least one of the Ej occurs} = P , E[N] = E
n
1
Ij =
j =1
n
P (Ej )
j =1
Also, . / N E = E[number of pairs of the Ej that occur] 2 / . Ii Ij =E i<j
=
P (Ei Ej )
i<j
and, in general . / N E = E[number of sets of size i that occur] i / . =E I j1 I j2 · · · I ji j1 < j2 0, α > 0, by G(t) = 1 − e−(λt) , α
t 0
The failure rate function for a Weibull distribution equals e−(λt) α(λt)α−1 λ λ(t) = = αλ(λt)α−1 α e−(λt) α
Thus, the Weibull distribution is IFR when α 1, and DFR when 0 < α 1; when α = 1, G(t) = 1 − e−λt , the exponential distribution, which is both IFR and DFR.
9.5.
System Life as a Function of Component Lives
597
Example 9.23 (The Gamma Distribution) A random variable is said to have a gamma distribution if its density is given, for some λ > 0, α > 0, by g(t) =
λe−λt (λt)α−1 (α)
for t 0
where
∞
(α) ≡
e−t t α−1 dt
0
For the gamma distribution, ¯ G(t) 1 = = λ(t) g(t)
∞
λe−λx (λx)α−1 dx λe−λt (λt)α−1 α−1 ∞ −λ(x−t) x = e dx t t t
With the change of variables u = x − t, we obtain 1 = λ(t)
∞ 0
u α−1 e−λu 1 + du t
Hence, G is IFR when α 1 and is DFR when 0 < α 1.
Suppose that the lifetime distribution of each component in a monotone system is IFR. Does this imply that the system lifetime is also IFR? To answer this, let us at first suppose that each component has the same lifetime distribution, which we denote by G. That is, Fi (t) = G(t), i = 1, . . . , n. To determine whether the system lifetime is IFR, we must compute λF (t), the failure rate function of F . Now, by definition, λF (t) = =
(d/dt)F (t) F¯ (t) ¯ (d/dt)[1 − r(G(t))] ¯ r(G(t))
where ¯ ¯ ¯ r(G(t)) ≡ r(G(t), . . . , G(t))
598
9
Reliability Theory
Hence, λF (t) =
¯ r (G(t)) G (t) ¯ r(G(t))
(G(t)) ¯ ¯ G (t) G(t)r ¯ ¯ r(G(t)) G(t) ! pr (p) !! = λG (t) r(p) ! ¯
=
(9.15)
p=G(t)
¯ Since G(t) is a decreasing function of t, it follows from Equation (9.15) that if each component of a coherent system has the same IFR lifetime distribution, then the distribution of system lifetime will be IFR if pr (p)/r(p) is a decreasing function of p.
Example 9.24 (The k-out-of-n System with Identical Components) Consider the k-out-of-n system which will function if and only if k or more components function. When each component has the same probability p of functioning, the number of functioning components will have a binomial distribution with parameters n and p. Hence, n n i p (1 − p)n−i r(p) = i i=k
which, by continual integration by parts, can be shown to be equal to r(p) =
n! (k − 1)!(n − k)!
p
x k−1 (1 − x)n−k dx
0
Upon differentiation, we obtain r (p) =
n! p k−1 (1 − p)n−k (k − 1)!(n − k)!
Therefore, . / pr (p) r(p) −1 = r(p) pr (p) , -−1 1 p x k−1 1 − x n−k = dx p 0 p 1−p
9.5.
System Life as a Function of Component Lives
599
Letting y = x/p, yields pr (p) = r(p)
,
1
y
k−1
0
1 − yp 1−p
-−1
n−k dy
Since (1 − yp)/(1 − p) is increasing in p, it follows that pr (p)/r(p) is decreasing in p. Thus, if a k-out-of-n system is composed of independent, like components having an increasing failure rate, the system itself has an increasing failure rate. It turns out, however, that for a k-out-of-n system, in which the independent components have different IFR lifetime distributions, the system lifetime need not be IFR. Consider the following example of a two-out-of-two (that is, a parallel) system.
Example 9.25 (A Parallel System That Is Not IFR) The life distribution of a parallel system of two independent components, the ith component having an exponential distribution with mean 1/i, i = 1, 2, is given by F¯ (t) = 1 − (1 − e−t )(1 − e−2t ) = e−2t + e−t − e−3t Therefore, λ(t) = =
f (t) F¯ (t) 2e−2t + e−t − 3e−3t e−2t + e−t − e−3t
It easily follows upon differentiation, that the sign of λ (t) is determined by e−5t − e−3t + 3e−4t , which is positive for small values and negative for large values of t. Therefore, λ(t) is initially strictly increasing, and then strictly decreasing. Hence, F is not IFR.
Remark The result of the preceding example is quite surprising at first glance. To obtain a better feel for it we need the concept of a mixture of distribution functions. The distribution function G is said to be a mixture of the distributions G1 and G2 if for some p, 0 < p < 1, G(x) = pG1 (x) + (1 − p)G2 (x)
(9.16)
Mixtures occur when we sample from a population made up of two distinct groups. For example, suppose we have a stockpile of items of which the fraction p
600
9
Reliability Theory
are type 1 and the fraction 1 − p are type 2. Suppose that the lifetime distribution of type 1 items is G1 and of type 2 items is G2 . If we choose an item at random from the stockpile, then its life distribution is as given by Equation (9.16). Consider now a mixture of two exponential distributions having rates λ1 and λ2 where λ1 < λ2 . We are interested in determining whether or not this mixture distribution is IFR. To do so, we note that if the item selected has survived up to time t, then its distribution of remaining life is still a mixture of the two exponential distributions. This is so since its remaining life will still be exponential with rate λ1 if it is type 1 or with rate λ2 if it is a type 2 item. However, the probability that it is a type 1 item is no longer the (prior) probability p but is now a conditional probability given that it has survived to time t. In fact, its probability of being a type 1 is P {type 1 | life > t} = =
P {type 1, life > t} P {life > t} pe−λ1 t
pe−λ1 t + (1 − p)e−λ2 t
As the preceding is increasing in t, it follows that the larger t is, the more likely it is that the item in use is a type 1 (the better one, since λ1 < λ2 ). Hence, the older the item is, the less likely it is to fail, and thus the mixture of exponentials far from being IFR is, in fact, DFR. Now, let us return to the parallel system of two exponential components having respective rates λ1 and λ2 . The lifetime of such a system can be expressed as the sum of two independent random variables, namely, ⎧ ⎪ ⎪ ⎨Exp(λ1 ) with probability
λ2 λ1 + λ2 system life = Exp(λ1 + λ2 ) + λ1 ⎪ ⎪ ⎩Exp(λ2 ) with probability λ1 + λ2 The first random variable whose distribution is exponential with rate λ1 + λ2 represents the time until one of the components fails, and the second, which is a mixture of exponentials, is the additional time until the other component fails. (Why are these two random variables independent?) Now, given that the system has survived a time t, it is very unlikely when t is large that both components are still functioning, but instead it is far more likely that one of the components has failed. Hence, for large t, the distribution of remaining life is basically a mixture of two exponentials—and so as t becomes even larger its failure rate should decrease (as indeed occurs).
9.5.
System Life as a Function of Component Lives
601
Recall that the failure rate function of a distribution F (t) having density f (t) = F (t) is defined by λ(t) =
f (t) 1 − F (t)
By integrating both sides, we obtain
t
t
λ(s) ds =
0
0
f (s) ds 1 − F (s)
= − log F¯ (t) Hence, F¯ (t) = e−(t)
(9.17)
where
t
(t) =
λ(s) ds 0
The function (t) is called the hazard function of the distribution F .
Definition 9.1 A distribution F is said to have increasing failure on the average (IFRA) if (t) = t
t 0
λ(s) ds t
(9.18)
increases in t for t 0. In other words, Equation (9.18) states that the average failure rate up to time t increases as t increases. It is not difficult to show that if F is IFR, then F is IFRA; but the reverse need not be true. Note that F is IFRA if (s)/s (t)/t whenever 0 s t, which is equivalent to (αt) (t) αt t
for 0 α 1, all t 0
602
9
Reliability Theory
But by equation (9.17) we see that (t) = − log F¯ (t), and so the preceding is equivalent to − log F¯ (αt) −α log F¯ (t) or equivalently log F¯ (αt) log F¯ α (t) which, since log x is a monotone function of x, shows that F is IFRA if and only if F¯ (αt) F¯ α (t)
for 0 α 1, all t 0
(9.19)
For a vector p = (p1 , . . . , pn ) we define pα = (p1α , . . . , pnα ). We shall need the following proposition.
Proposition 9.2 Any reliability function r(p) satisfies r(pα ) [r(p)]α ,
0α1
Proof We prove this by induction on n, the number of components in the system. If n = 1, then either r(p) ≡ 0, r(p) ≡ 1, or r(p) ≡ p. Hence, the proposition follows in this case. So assume that Proposition 9.2 is valid for all monotone systems of n − 1 components and consider a system of n components having structure function φ. By conditioning upon whether or not the nth component is functioning, we obtain r(pα ) = pnα r(1n , pα ) + (1 − pnα )r(0n , pα )
(9.20)
Now consider a system of components 1 through n − 1 having a structure function φ1 (x) = φ(1n , x). The reliability function for this system is given by r1 (p) = r(1n , p); hence, from the induction assumption (valid for all monotone systems of n − 1 components), we have r(1n , pα ) [r(1n , p)]α Similarly, by considering the system of components 1 through n − 1 and structure function φ0 (x) = φ(0n , x), we obtain r(0n , pα ) [r(0n , p)]α Thus, from Equation (9.20), we obtain r(pα ) pnα [r(1n , p)]α + (1 − pnα )[r(0n , p)]α
9.5.
System Life as a Function of Component Lives
603
which, by using the lemma to follow [with λ = pn , x = r(1n , p), y = r(0n , p)], implies that r(pα ) [pn r(1n , p) + (1 − pn )r(0n , p)]α = [r(p)]α which proves the result.
Lemma 9.1 If 0 α 1, 0 λ 1, then h(y) = λα x α + (1 − λα )y α − (λx + (1 − λ)y)α 0 for all 0 y x.
Proof The proof is left as an exercise. We are now ready to prove the following important theorem.
Theorem 9.2 For a monotone system of independent components, if each component has an IFRA lifetime distribution, then the distribution of system lifetime is itself IFRA.
Proof The distribution of system lifetime F is given by F¯ (αt) = r(F¯1 (αt), . . . , F¯n (αt)) Hence, since r is a monotone function, and since each of the component distributions F¯i is IFRA, we obtain from Equation (9.19) F¯ (αt) r(F¯1α (t), . . . , F¯nα (t)) [r(F¯1 (t), . . . , F¯n (t))]α = F¯ α (t) which by Equation (9.19) proves the theorem. The last inequality followed, of course, from Proposition 9.2.
604
9
9.6.
Reliability Theory
Expected System Lifetime
In this section, we show how the mean lifetime of a system can be determined, at least in theory, from a knowledge of the reliability function r(p) and the component lifetime distributions Fi , i = 1, . . . , n. Since the system’s lifetime will be t or larger if and only if the system is still functioning at time t, we have that ¯ P {system life > t} = r F(t) ¯ = (F¯1 (t), . . . , F¯n (t)). Hence, by a well-known formula that states that where F(t) for any nonnegative random variable X, E[X] =
∞
P {X > x} dx,
0
we see that∗ E[system life] =
∞
¯ r F(t) dt
(9.21)
0
Example 9.26 (A Series System of Uniformly Distributed Components) Consider a series system of three independent components each of which functions for an amount of time (in hours) uniformly distributed over (0, 10). Hence, r(p) = p1 p2 p3 and t/10, Fi (t) = 1,
0 t 10 t > 10
i = 1, 2, 3
Therefore, ⎧ ⎪ 10 − t 3 ⎨ , ¯ r F(t) = 10 ⎪ ⎩ 0,
0 t 10 t > 10
∗ That E[X] = ∞ P {X > x} dx can be shown as follows when X has density f : 0 ∞ 0
P {X > x} dx =
∞ ∞ 0
x
f (y) dy dx =
∞ y 0
0
f (y) dx dy =
∞ 0
yf (y) dy = E[X]
9.6.
Expected System Lifetime
605
and so from Equation (9.21) we obtain E[system life] =
10 10 − t 3
0
= 10
10 1
dt
y 3 dy
0
=
5 2
Example 9.27 (A Two-out-of-Three System) Consider a two-out-of-three system of independent components, in which each component’s lifetime is (in months) uniformly distributed over (0, 1). As was shown in Example 9.13, the reliability of such a system is given by r(p) = p1 p2 + p1 p3 + p2 p3 − 2p1 p2 p3 Since Fi (t) =
t, 1,
0t 1 t >1
we see from Equation (9.21) that E[system life] =
1
[3(1 − t)2 − 2(1 − t)3 ] dt
0
=
1
(3y 2 − 2y 3 ) dy
0
=1−
1 2
=
1 2
Example 9.28 (A Four-Component System) Consider the four-component system that functions when components 1 and 2 and at least one of components 3 and 4 functions. Its structure function is given by φ(x) = x1 x2 (x3 + x4 − x3 x4 ) and thus its reliability function equals r(p) = p1 p2 (p3 + p4 − p3 p4 )
606
9
Reliability Theory
Let us compute the mean system lifetime when the ith component is uniformly distributed over (0, i), i = 1, 2, 3, 4. Now, 1 − t, 0t 1 ¯ F1 (t) = 0, t >1 1 − t/2, 0t 2 F¯2 (t) = 0, t >2 1 − t/3, 0t 3 F¯3 (t) = 0, t >3 1 − t/4, 0t 4 F¯4 (t) = 0, t >4 Hence,
⎧ . / 3 − t 4 − t (3 − t)(4 − t) ⎨(1 − t) 2 − t + − , ¯ r F(t) = 2 3 4 12 ⎩ 0,
0t 1 t >1
Therefore, E[system life] =
1 24
1
(1 − t)(2 − t)(12 − t 2 ) dt
0
593 = (24)(60) ≈ 0.41 We end this section by obtaining the mean lifetime of a k-out-of-n system of independent identically distributed exponential components. If θ is the mean lifetime of each component, then F¯i (t) = e−t/θ Hence, since for a k-out-of-n system, r(p, p, . . . , p) =
n n
i
i=k
p i (1 − p)n−i
we obtain from Equation (9.21) E[system life] = 0
n ∞ n i=k
i
(e−t/θ )i (1 − e−t/θ )n−i dt
9.6.
Expected System Lifetime
607
Making the substitution 1 y dy = − e−t/θ dt = − dt θ θ
y = e−t/θ , yields E[system life] = θ
n n
i
i=k
1
y i−1 (1 − y)n−i dy
0
Now, it is not difficult to show that∗
1
y n (1 − y)m dy =
0
m!n! (m + n + 1)!
(9.22)
Thus, the foregoing equals E[system life] = θ
n i=k
=θ
(i − 1)!(n − i)! n! (n − i)!i! n!
n 1 i=k
i
(9.23)
Remark Equation (9.23) could have been proven directly by making use of special properties of the exponential distribution. First note that the lifetime of a k-out-of-n system can be written as T1 + · · · + Tn−k+1 , where Ti represents the time between the (i − 1)st and ith failure. This is true since T1 + · · · + Tn−k+1 equals the time at which the (n − k + 1)st component fails, which is also the first time that the number of functioning components is less than k. Now, when all n components are functioning, the rate at which failures occur is n/θ . That is, T1 is exponentially distributed with mean θ/n. Similarly, since Ti represents the time until the next failure when there are n − (i − 1) functioning components,
∗ Let
C(n, m) =
1 0
y n (1 − y)m dy
Integration by parts yields that C(n, m) = [m/(n + 1)]C(n + 1, m − 1). Starting with C(n, 0) = 1/ (n + 1), Equation (9.22) follows by mathematical induction.
608
9
Reliability Theory
it follows that Ti is exponentially distributed with mean θ/(n − i + 1). Hence, the mean system lifetime equals .
1 1 + ··· + E[T1 + · · · + Tn−k+1 ] = θ n k
/
Note also that it follows, from the lack of memory of the exponential, that the Ti , i = 1, . . . , n − k + 1, are independent random variables. 9.6.1. An Upper Bound on the Expected Life of a Parallel System Consider a parallel system of n components, whose lifetimes are not necessarily independent. The system lifetime can be expressed as system life = max Xi i
where Xi is the lifetime of component i, i = 1, . . . , n. We can bound the expected system lifetime by making use of the following inequality. Namely, for any constant c max Xi c + i
n (Xi − c)+
(9.24)
i=1
where x + , the positive part of x, is equal to x if x > 0 and is equal to 0 if x 0. The validity of inequality (9.24) is immediate since if max Xi < c then the left side is equal to max Xi and the right side is equal to c. On the other hand, if X(n) = max Xi > c then the right side is at least as large as c + (X(n) − c) = X(n) . It follows from inequality (9.24), upon taking expectations, that n ' & E[(Xi − c)+ ] E max Xi c + i
i=1
Now, (Xi − c)+ is a nonnegative random variable and so E[(Xi − c)+ ] =
∞
P {(Xi − c)+ > x} dx
0
=
∞
P {Xi − c > x} dx
0
=
c
∞
P {Xi > y} dy
(9.25)
9.6.
Expected System Lifetime
609
Thus, we obtain n ' & E max Xi c + i
∞
P {Xi > y} dy
(9.26)
i=1 c
Because the preceding is true for all c, it follows that we obtain the best bound by letting c equal the value that minimizes the right side of the preceding. To determine that value, differentiate the right side of the preceding and set the result equal to 0, to obtain: 1−
n
P {Xi > c} = 0
i=1
That is, the minimizing value of c is that value c∗ for which n
P {Xi > c∗ } = 1
i=1
n
Since i=1 P {Xi > c} is a decreasing function of c, the value of c∗ can be easily approximated and then utilized in inequality (9.26). Also, it is interesting to note that c∗ is such that the expected number of the Xi that exceed c∗ is equal to 1 (see Exercise 32). That the optimal value of c has this property is interesting and somewhat intuitive in as much as inequality 9.24 is an equality when exactly one of the Xi exceeds c.
Example 9.29 Suppose the lifetime of component i is exponentially distributed with rate λi , i = 1, . . . , n. Then the minimizing value of c is such that 1=
n
P {Xi > c∗ } =
i=1
n
e−λi c
∗
i=1
and the resulting bound of the mean system life is n ' & E max Xi c∗ + E[(Xi − c∗ )+ ] i
i=1
= c∗ +
n E[(Xi − c∗ )+ | Xi > c∗ ]P {Xi > c∗ ] i=1
+ E[(Xi − c∗ )+ | Xi c∗ ]P {Xi c∗ ] = c∗ +
n 1 −λi c∗ e λi i=1
610
9
Reliability Theory
In the special case where all the rates are equal, say, λi = λ, i = 1, . . . , n, then 1 = ne−λc
∗
or c∗ =
1 log(n) λ
and the bound is ' 1 & E max Xi (log(n) + 1) i λ That is, if X1 , . . . , Xn are identically distributed exponential random variables with rate λ, then the preceding gives a bound on the expected value of their maximum. In the special case where these random variables are also independent, the following exact expression, given by Equation (9.25), is not much less than the preceding upper bound: n ' 1 & 1 1 n1 dx ≈ log(n) E max Xi = 1/i ≈ i λ λ 1 x λ
i=1
9.7.
Systems with Repair
Consider an n component system having reliability function r(p). Suppose that component i functions for an exponentially distributed time with rate λi and then fails; once failed it takes an exponential time with rate μi to be repaired, i = 1, . . . , n. All components act independently. Let us suppose that all components are initially working, and let A(t) = P {system is working at t} A(t) is called the availability at time t. Since the components act independently, A(t) can be expressed in terms of the reliability function as follows: A(t) = r(A1 (t), . . . , An (t))
(9.27)
where Ai (t) = P {component i is functioning at t} Now the state of component i—either on or off—changes in accordance with a two-state continuous time Markov chain. Hence, from the results of Example 6.12
9.7.
Systems with Repair
611
we have Ai (t) = P00 (t) =
μi λi + e−(λi +μi )t μi + λi μ i + λi
Thus, we obtain
λ μ + e−(λ+μ)t A(t) = r μ+λ μ+λ
If we let t approach ∞, then we obtain the limiting availability—call it A—which is given by μ A = lim A(t) = r t→∞ λ+μ
Remarks (i) If the on and off distribution for component i are arbitrary continuous distributions with respective means 1/λi and 1/μi , i = 1, . . . , n, then it follows from the theory of alternating renewal processes (see Section 7.5.1) that Ai (t) →
1/λi μi = 1/λi + 1/μi μ i + λi
and thus using the continuity of the reliability function, it follows from (9.27) that the limiting availability is A = lim A(t) = r t→∞
μ μ+λ
Hence, A depends only on the on and off distributions through their means. (ii) It can be shown (using the theory of regenerative processes as presented in Section 7.5) that A will also equal the long-run proportion of time that the system will be functioning. 1 Example 9.30 For a series system, r(p) = ni=1 pi and so A(t) =
n . i=1
μi λi + e−(λi +μi )t μ i + λi μi + λi
and A=
n i=1
μi μi + λi
/
612
9
Reliability Theory
Example 9.31 For a parallel system, r(p) = 1 − A(t) = 1 −
n . i=1
1n i=1
(1 − pi ) and thus
/ λi −(λi +μi )t (1 − e ) μi + λi
and A(t) = 1 −
n i=1
λi μ i + λi
The preceding system will alternate between periods when it is up and periods when it is down. Let us denote by Ui and Di , i 1, the lengths of the ith up and down period respectively. For instance in a two-out-of-three system, U1 will be the time until two components are down; D1 , the additional time until two are up; U2 the additional time until two are down, and so on. Let U 1 + · · · + Un , U¯ = lim n→∞ n D1 + · · · + Dn D¯ = lim n→∞ n denote the average length of an up and down period respectively.∗ ¯ note first that in the first n up–down cycles—that is, To determine U¯ and D, in time ni=1 (Ui + Di )—the system will be up for a time ni=1 Ui . Hence, the proportion of time the system will be up in the first n up–down cycles is n U1 + · · · + Un i=1 Ui /n = n U 1 + · · · + Un + D 1 + · · · + D n U /n + ni=1 Di /n i=1 i As n → ∞, this must converge to A, the long-run proportion of time the system is up. Hence, U¯ μ =A=r λ+μ U¯ + D¯
(9.28)
However, to solve for U¯ and D¯ we need a second equation. To obtain one consider the rate at which the system fails. As there will be n failures in time ∗ It can be shown using the theory of regenerative processes that, with probability 1, the preceding
limits will exist and will be constants.
9.7.
n
i=1 (Ui
613
Systems with Repair
+ Di ), it follows that the rate at which the system fails is
rate at which system fails = lim n n→∞
i
n U1 + n1 D1
1 n n = ¯ U /n + D /n U + D¯ 1 i 1 i
= lim n n→∞
(9.29)
That is, the foregoing yields the intuitive result that, on average, there is one failure every U¯ + D¯ time units. To utilize this, let us determine the rate at which a failure of component i causes the system to go from up to down. Now, the system will go from up to down when component i fails if the states of the other components x1 , . . . , xi−1 , xi−1 , . . . , xn are such that φ(1i , x) = 1, φ(0i , x) = 0. That is the states of the other components must be such that φ(1i , x) − φ(0i , x) = 1
(9.30)
Since component i will, on average, have one failure every 1/λi +1/μi time units, it follows that the rate at which component i fails is equal to (1/λi + 1/μi )−1 = λi μi /(λi + μi ). In addition, the states of the other components will be such that (9.30) holds with probability P {φ(1i , X(∞)) − φ(0i , X(∞)) = 1} = E[φ(1i , X(∞)) − φ(0i , X(∞))] μ μ − r 0i , = r 1i , λ+μ λ+μ
since φ(1i , X(∞)) − φ(0i , X(∞)) is a Bernoulli random variable
Hence, putting the preceding together we see that . / μ μ λi μi rate at which component r 1i , − r 0i , = i causes the system to fail λi + μi λ+μ λ+μ Summing this over all components i thus gives / λi μi . μ μ r 1i , − r 0i , rate at which system fails = λi + μi λ+μ λ+μ i
Finally, equating the preceding with (9.29) yields / λi μi . μ μ 1 r 1i , − r 0i , = λi + μ i λ+μ λ+μ U¯ + D¯ i
(9.31)
614
9
Reliability Theory
Solving (9.28) and (9.31), we obtain μ λ+μ U¯ = n / , λi μi . μ μ r 1i , − r 0i , λi + μ i λ+μ λ+μ i=1 / . μ U¯ 1−r λ + μ D¯ = μ r λ+μ
r
(9.32)
(9.33)
Also, (9.31) yields the rate at which the system fails. ¯ we did not make use of Remark In establishing the formulas for U¯ and D, the assumption of exponential on and off times and in fact, our derivation is valid and Equations (9.32) and (9.33) hold whenever U¯ and D¯ are well defined (a sufficient condition is that all on and off distributions are continuous). The quantities λi , μi , i = 1, . . . , n, will represent, respectively, the reciprocals of the mean lifetimes and mean repair times.
Example 9.32 For a series system, 1
μi 1 μi + λi U¯ = = , λi μi 1 μj i λi i λi + μi j =i μj + λj 1 μi 1− i 1 μ i + λi D¯ = 1 × μi i λi i μi + λi i
whereas for a parallel system, 1 λi λi 1− i 1 μ i + λi μi + λi = × , U¯ = λi μi 1 1 λj λj i μi i j λi + μi j =i μj + λj μ j + λj 1−
1
1
i
λi 1 μ i + λi ¯ D¯ = U= 1 λi i μi 1− i μi + λi i
9.7.
Systems with Repair
615
The preceding formulas hold for arbitrary continuous up and down distributions with 1/λi and 1/μi denoting respectively the mean up and down times of component i, i = 1, . . . , n. 9.7.1. A Series Model with Suspended Animation Consider a series consisting of n components, and suppose that whenever a component (and thus the system) goes down, repair begins on that component and each of the other components enters a state of suspended animation. That is, after the down component is repaired, the other components resume operation in exactly the same condition as when the failure occurred. If two or more components go down simultaneously, one of them is arbitrarily chosen as being the failed component and repair on that component begins; the others that went down at the same time are considered to be in a state of suspended animation, and they will instantaneously go down when the repair is completed. We suppose that (not counting any time in suspended animation) the distribution of time that component i functions is Fi with mean ui , whereas its repair distribution is Gi with mean di , i = 1, . . . , n. To determine the long-run proportion of time this system is working, we reason as follows. To begin, consider the time, call it T , at which the system has been up for a time t. Now, when the system is up, the failure times of component i constitute a renewal process with mean interarrival time ui . Therefore, it follows that number of failures of i in time T ≈
t ui
As the average repair time of i is di , the preceding implies that total repair time of i in time T ≈
tdi ui
Therefore, in the period of time in which the system has been up for a time t, the total system downtime has approximately been t
n
di /ui
i=1
Hence, the proportion of time that the system has been up is approximately t +t
t n
i=1 di /ui
616
9
Reliability Theory
Because this approximation should become exact as we let t become larger, it follows that proportion of time the system is up =
1+
1
i di /ui
(9.34)
which also shows that proportion of time the system is down = 1 − proportion of time the system is up i di /ui = 1 + i di /ui Moreover, in the time interval from 0 to T , the proportion of the repair time that has been devoted to component i is approximately tdi /ui i tdi /ui Thus, in the long run, di /ui proportion of down time that is due to component i = i di /ui Multiplying the preceding by the proportion of time the system is down gives proportion of time component i is being repaired =
1+
di /ui i di /ui
Also, since component j will be in suspended animation whenever any of the other components is in repair, we see that i=j di /ui proportion of time component j is in suspended animation = 1 + i di /ui Another quantity of interest is the long-run rate at which the system fails. Since component i fails at rate 1/ui when the system is up, and does not fail when the system is down, it follows that rate at which i fails = =
proportion of time system is up ui 1/ui 1 + i di /ui
Exercises
617
Since the system fails when any of its components fail, the preceding yields that i 1/ui (9.35) rate at which the system fails = 1 + i di /ui If we partition the time axis into periods when the system is up and those when it is down, we can determine the average length of an up period by noting that if U (t) is the total amount of time that the system is up in the interval [0, t], and if N(t) is the number of failures by time t, then average length of an up period = lim
t→∞
= lim
t→∞
=
U (t) N (t) U (t)/t N (t)/t
1 i 1/ui
where the final equality used Equations (9.34) and (9.35). Also, in a similar manner it can be shown that di /ui average length of a down period = i (9.36) i 1/ui
Exercises 1.
Prove that, for any structure function, φ, φ(x) = xi φ(1i , x) + (1 − xi )φ(0i , x)
where (1i , x) = (x1 , . . . , xi−1 , 1, xi+1 , . . . , xn ), (0i , x) = (x1 , . . . , xi−1 , 0, xi+1 , . . . , xn ) 2.
Show that (a) if φ(0, 0, . . . , 0) = 0 and φ(1, 1, . . . , 1) = 1, then min xi φ(x) max xi
618
9
Reliability Theory
(b) φ(max(x, y)) max(φ(x), φ(y)) (c) φ(min(x, y)) min(φ(x), φ(y)) 3.
For any structure function, we define the dual structure φ D by φ D (x) = 1 − φ(1 − x) (a) Show that the dual of a parallel (series) system is a series (parallel) system. (b) Show that the dual of a dual structure is the original structure. (c) What is the dual of a k-out-of-n structure? (d) Show that a minimal path (cut) set of the dual system is a minimal cut (path) set of the original structure.
*4. Write the structure function corresponding to the following: (a)
Figure 9.16.
(b)
Figure 9.17.
(c)
Figure 9.18.
Exercises
5.
619
Find the minimal path and minimal cut sets for: (a)
Figure 9.19.
(b)
Figure 9.20.
*6. The minimal path sets are {1, 2, 4}, {1, 3, 5}, and {5, 6}. Give the minimal cut sets. 7. The minimal cut sets are {1, 2, 3}, {2, 3, 4}, and {3, 5}. What are the minimal path sets? 8. Give the minimal path sets and the minimal cut sets for the structure given by Figure 9.21. 9.
Component i is said to be relevant to the system if for some state vector x, φ(1i , x) = 1,
Otherwise, it is said to be irrelevant.
φ(0i , x) = 0
620
9
Reliability Theory
Figure 9.21.
(a) Explain in words what it means for a component to be irrelevant. (b) Let A1 , . . . , As be the minimal
path sets of a system, and let S denote the set of components. Show that S = si=1 Ai if and only if all components are relevant.
(c) Let C1 , . . . , Ck denote the minimal cut sets. Show that S = ki=1 Ci if and only if all components are relevant. 10. Let ti denote the time of failure of the ith component; let τφ (t) denote the time to failure of the system φ as a function of the vector t = (t1 , . . . , tn ). Show that max min ti = τφ (t) = min max ti
1j s i∈Aj
1j k i∈Cj
where C1 , . . . , Ck are the minimal cut sets, and A1 , . . . , As the minimal path sets. 11. Give the reliability function of the structure of Exercise 8. *12. Give the minimal path sets and the reliability function for the structure in Figure 9.22.
Figure 9.22.
Exercises
621
13. Let r(p) be the reliability function. Show that r(p) = pi r(1i , p) + (1 − pi )r(0i , p) 14. Compute the reliability function of the bridge system (see Figure 9.11) by conditioning upon whether or not component 3 is working. 15. Compute upper and lower bounds of the reliability function (using Method 2) for the systems given in Exercise 4, and compare them with the exact values when pi ≡ 12 . 16. Compute the upper and lower bounds of r(p) using both methods for the (a) two-out-of-three system and (b) two-out-of-four system. (c) Compare these bounds with the exact reliability when (i) pi ≡ 0.5 (ii) pi ≡ 0.8 (iii) pi ≡ 0.2 *17. Let N be a nonnegative, integer-valued random variable. Show that P {N > 0}
(E[N ])2 E[N 2 ]
and explain how this inequality can be used to derive additional bounds on a reliability function. Hint: E[N 2 ] = E[N 2 | N > 0]P {N > 0} (E[N | N > 0]) P {N > 0} 2
(Why?) (Why?)
Now multiply both sides by P {N > 0}. 18. Consider a structure in which the minimal path sets are {1, 2, 3} and {3, 4, 5}. (a) What are the minimal cut sets? (b) If the component lifetimes are independent uniform (0, 1) random variables, determine the probability that the system life will be less than 12 . 19. Let X1 , X2 , . . . , Xn denote independent and identically distributed random variables and define the order statistics X(1) , . . . , X(n) by X(i) ≡ ith smallest of X1 , . . . , Xn Show that if the distribution of Xj is IFR, then so is the distribution of X(i) . Hint: Relate this to one of the examples of this chapter.
622
9
Reliability Theory
20. Let F be a continuous distribution function. For some positive α, define the distribution function G by ¯ G(t) = (F¯ (t))α Find the relationship between λG (t) and λF (t), the respective failure rate functions of G and F . 21. Consider the following four structures: (i)
(ii)
Figure 9.23.
(iii)
Figure 9.24.
(iv)
Figure 9.25.
Figure 9.26.
Let F1 , F2 , and F3 be the corresponding component failure distributions; each of which is assumed to be IFR (increasing failure rate). Let F be the system failure distribution. All components are independent. (a) For which structures is F necessarily IFR if F1 = F2 = F3 ? Give reasons. (b) For which structures is F necessarily IFR if F2 = F3 ? Give reasons. (c) For which structures is F necessarily IFR if F1 = F2 = F3 ? Give reasons. *22. Let X denote the lifetime of an item. Suppose the item has reached the age of t. Let Xt denote its remaining life and define F¯t (a) = P {Xt > a} In words, F¯t (a) is the probability that a t-year old item survives an additional time a. Show that
Exercises
623
(a) F¯t (a) = F¯ (t + a)/F¯ (t) where F is the distribution function of X. (b) Another definition of IFR is to say that F is IFR if F¯t (a) decreases in t, for all a. Show that this definition is equivalent to the one given in the text when F has a density. 23. Show that if each (independent) component of a series system has an IFR distribution, then the system lifetime is itself IFR by (a) showing that λF (t) =
λi (t)
i
where λF (t) is the failure rate function of the system; and λi (t) the failure rate function of the lifetime of component i. (b) using the definition of IFR given in Exercise 22. 24. Show that if F is IFR, then it is also IFRA, and show by counterexample that the reverse is not true. *25. We say that ζ is a p-percentile of the distribution F if F (ζ ) = p. Show that if ζ is a p-percentile of the IFRA distribution F , then F¯ (x) e−θx , F¯ (x) e−θx ,
xζ xζ
where θ=
− log(1 − p) ζ
26. Prove Lemma 9.3. Hint: Let x = y + δ. Note that f (t) = t α is a concave function when 0 α 1, and use the fact that for a concave function f (t +h)−f (t) is decreasing in t. 27. Let r(p) = r(p, p, . . . , p). Show that if r(p0 ) = p0 , then r(p) p r(p) p
for p p0 for p p0
Hint: Use Proposition 9.2. 28. Find the mean lifetime of a series system of two components when the component lifetimes are respectively uniform on (0, 1) and uniform on (0, 2). Repeat for a parallel system.
624
9
Reliability Theory
29. Show that the mean lifetime of a parallel system of two components is 1 μ1 μ2 + + μ1 + μ2 (μ1 + μ2 )μ2 (μ1 + μ2 )μ1 when the first component is exponentially distributed with mean 1/μ1 and the second is exponential with mean 1/μ2 . *30. Compute the expected system lifetime of a three-out-of-four system when the first two component lifetimes are uniform on (0, 1) and the second two are uniform on (0, 2). 31. Show that the variance of the lifetime of a k-out-of-n system of components, each of whose lifetimes is exponential with mean θ , is given by θ2
n 1 i2 i=k
32. In Section 9.6.1 show that the expected number of Xi that exceed c∗ is equal to 1. 33. Let Xi be an exponential random variable with mean 8 + 2i, for i = 1, 2, 3. Use the results of Section 9.6.1 to obtain an upper bound on E[max Xi ], and then compare this with the exact result when the Xi are independent. 34. For the model of Section 9.7, compute for a k-out-of-n structure (i) the average up time, (ii) the average down time, and (iii) the system failure rate. 35. Prove the combinatorial identity n n n n−1 = − + ··· ± , in i −1 i i +1 n (a) by induction on i (b) by a backwards induction argument on i—that is, prove it first for i = n, then assume it for i = k and show that this implies that it is true for i = k − 1. 36. Verify Equation (9.36).
References 1. R. E. Barlow and F. Proschan, “Statistical Theory of Reliability and Life Testing,” Holt, New York, 1975. 2. H. Frank and I. Frisch, “Communication, Transmission, and Transportation Network,” Addison-Wesley, Reading, Massachusetts, 1971. 3. I. B. Gertsbakh, “Statistical Reliability Theory,” Marcel Dekker, New York and Basel, 1989.
Brownian Motion and Stationary Processes
10 10.1.
Brownian Motion
Let us start by considering the symmetric random walk, which in each time unit is equally likely to take a unit step either to the left or to the right. That is, it is a Markov chain with Pi,i+1 = 12 = Pi,i−1 , i = 0, ±1, . . . . Now suppose that we speed up this process by taking smaller and smaller steps in smaller and smaller time intervals. If we now go to the limit in the right manner what we obtain is Brownian motion. More precisely, suppose that each t time unit we take a step of size x either to the left or the right with equal probabilities. If we let X(t) denote the position at time t then X(t) = x(X1 + · · · + X[t/t] ) where
Xi =
+1,
if the ith step of length x is to the right
−1,
if it is to the left
(10.1)
and [t/t] is the largest integer less than or equal to t/t, and where the Xi are assumed independent with P {Xi = 1} = P {Xi = −1} =
1 2
As E[Xi ] = 0, Var(Xi ) = E[Xi2 ] = 1, we see from Equation (10.1) that E[X(t)] = 0,
.
Var(X(t)) = (x)
2
t t
/ (10.2)
We shall now let x and t go to 0. However, we must do it in a way such that the resulting limiting process is nontrivial (for instance, if we let x = t and
625
626
10
Brownian Motion and Stationary Processes
let t → 0, then from the preceding we see that E[X(t)] and Var(X(t)) would both converge to 0 and thus X(t) would equal 0 with probability 1). If we let √ x = σ t for some positive constant σ then from Equation (10.2) we see that as t → 0 E[X(t)] = 0, Var(X(t)) → σ 2 t We now √ list some intuitive properties of this limiting process obtained by taking x = σ t and then letting t → 0. From Equation (10.1) and the central limit theorem the following seems reasonable: (i) X(t) is normal with mean 0 and variance σ 2 t. In addition, because the changes of value of the random walk in nonoverlapping time intervals are independent, we have (ii) {X(t), t 0} has independent increments, in that for all t1 < t2 < · · · < tn X(tn ) − X(tn−1 ), X(tn−1 ) − X(tn−2 ), . . . , X(t2 ) − X(t1 ), X(t1 ) are independent. Finally, because the distribution of the change in position of the random walk over any time interval depends only on the length of that interval, it would appear that (iii) {X(t), t 0} has stationary increments, in that the distribution of X(t + s) − X(t) does not depend on t. We are now ready for the following formal definition.
Definition 10.1 A stochastic process {X(t), t 0} is said to be a Brownian motion process if (i) X(0) = 0; (ii) {X(t), t 0} has stationary and independent increments; (iii) for every t > 0, X(t) is normally distributed with mean 0 and variance σ 2 t. The Brownian motion process, sometimes called the Wiener process, is one of the most useful stochastic processes in applied probability theory. It originated in physics as a description of Brownian motion. This phenomenon, named after the English botanist Robert Brown who discovered it, is the motion exhibited by a small particle which is totally immersed in a liquid or gas. Since then, the process has been used beneficially in such areas as statistical testing of goodness of fit, analyzing the price levels on the stock market, and quantum mechanics. The first explanation of the phenomenon of Brownian motion was given by Einstein in 1905. He showed that Brownian motion could be explained by assuming that the immersed particle was continually being subjected to bombardment by
10.1.
Brownian Motion
627
the molecules of the surrounding medium. However, the preceding concise definition of this stochastic process underlying Brownian motion was given by Wiener in a series of papers originating in 1918. When σ = 1, the process is called standard Brownian motion. Because any Brownian motion can be converted to the standard process by letting B(t) = X(t)/σ we shall, unless otherwise stated, suppose throughout this chapter that σ = 1. The interpretation of Brownian motion as the limit of the random walks [Equation (10.1)] suggests that X(t) should be a continuous function of t. This turns out to be the case, and it may be proven that, with probability 1, X(t) is indeed a continuous function of t. This fact is quite deep, and no proof shall be attempted. As X(t) is normal with mean 0 and variance t, its density function is given by 1 2 e−x /2t ft (x) = √ 2πt To obtain the joint density function of X(t1 ), X(t2 ), . . . , X(tn ) for t1 < · · · < tn , note first that the set of equalities X(t1 ) = x1 , X(t2 ) = x2 , .. . X(tn ) = xn is equivalent to X(t1 ) = x1 , X(t2 ) − X(t1 ) = x2 − x1 , .. . X(tn ) − X(tn−1 ) = xn − xn−1 However, by the independent increment assumption it follows that X(t1 ), X(t2 ) − X(t1 ), . . . , X(tn ) − X(tn−1 ), are independent and, by the stationary increment assumption, that X(tk ) − X(tk−1 ) is normal with mean 0 and vari-
628
10
Brownian Motion and Stationary Processes
ance tk − tk−1 . Hence, the joint density of X(t1 ), . . . , X(tn ) is given by f (x1 , x2 , . . . , xn ) = ft1 (x1 )ft2 −t1 (x2 − x1 ) · · · ftn −tn−1 (xn − xn−1 ) / . 1 x12 (x2 − x1 )2 (xn − xn−1 )2 exp − + + ··· + 2 t1 t2 − t1 tn − tn−1 = (10.3) (2π)n/2 [t1 (t2 − t1 ) · · · (tn − tn−1 )]1/2 From this equation, we can compute in principle any desired probabilities. For instance, suppose we require the conditional distribution of X(s) given that X(t) = B where s < t. The conditional density is fs|t (x|B) =
fs (x)ft−s (B − x) ft (B)
= K1 exp{−x 2 /2s − (B − x)2 /2(t − s)} 1 Bx 2 1 = K2 exp −x + + 2s 2(t − s) t −s t sB x2 − 2 x = K2 exp − 2s(t − s) t (x − Bs/t)2 = K3 exp − 2s(t − s)/t where K1 , K2 , and K3 do not depend on x. Hence, we see from the preceding that the conditional distribution of X(s) given that X(t) = B is, for s < t, normal with mean and variance given by s E[X(s)|X(t) = B] = B, t s Var[X(s)|X(t) = B] = (t − s) t
(10.4)
Example 10.1 In a bicycle race between two competitors, let Y (t) denote the amount of time (in seconds) by which the racer that started in the inside position is ahead when 100t percent of the race has been completed, and suppose that {Y (t), 0 t 1} can be effectively modeled as a Brownian motion process with variance parameter σ 2 . (a) If the inside racer is leading by σ seconds at the midpoint of the race, what is the probability that she is the winner? (b) If the inside racer wins the race by a margin of σ seconds, what is the probability that she was ahead at the midpoint?
10.2.
629
Hitting Times, Maximum Variable, and the Gambler’s Ruin Problem
Solution: (a) P {Y (1) > 0|Y (1/2) = σ } = P {Y (1) − Y (1/2) > −σ |Y (1/2) = σ } = P {Y (1) − Y (1/2) > −σ } = P {Y (1/2) > −σ } √ Y (1/2) =P √ >− 2 σ/ 2 √ = ( 2)
by independent increments
by stationary increments
≈ 0.9213 where (x) = P {N (0, 1) x} is the standard normal distribution function. (b) Because we must compute P {Y (1/2) > 0|Y (1) = σ }, let us first determine the conditional distribution of Y (s) given that Y (t) = C, when s < t. Now, since {X(t), t 0} is standard Brownian motion when X(t) = Y (t)/σ , we obtain from Equation (10.4) that the conditional distribution of X(s), given that X(t) = C/σ , is normal with mean sC/tσ and variance s(t − s)/t. Hence, the conditional distribution of Y (s) = σ X(s) given that Y (t) = C is normal with mean sC/t and variance σ 2 s(t − s)/t. Hence, P {Y (1/2) > 0|Y (1) = σ } = P {N (σ/2, σ 2 /4) > 0} = (1) ≈ 0.8413
10.2.
Hitting Times, Maximum Variable, and the Gambler’s Ruin Problem
Let Ta denote the first time the Brownian motion process hits a. When a > 0 we will compute P {Ta t} by considering P {X(t) a} and conditioning on whether or not Ta t. This gives P {X(t) a} = P {X(t) a|Ta t}P {Ta t} +P {X(t) a|Ta > t}P {Ta > t}
(10.5)
Now if Ta t, then the process hits a at some point in [0, t] and, by symmetry, it is just as likely to be above a or below a at time t. That is P {X(t) a|Ta t} =
1 2
630
10
Brownian Motion and Stationary Processes
As the second right-hand term of Equation (10.5) is clearly equal to 0 (since, by continuity, the process value cannot be greater than a without having yet hit a), we see that P {Ta t} = 2P {X(t) a} ∞ 2 2 =√ e−x /2t dx 2πt a ∞ 2 −y 2 /2 =√ dy, √ e 2π a/ t
a>0
(10.6)
For a < 0, the distribution of Ta is, by symmetry, the same as that of T−a . Hence, from Equation (10.6) we obtain 2 P {Ta t} = √ 2π
∞ √
|a|/ t
e−y
2 /2
dy
(10.7)
Another random variable of interest is the maximum value the process attains in [0, t]. Its distribution is obtained as follows: For a > 0 P
max X(s) a = P {Ta t}
0st
by continuity
= 2P {X(t) a} from (10.6) ∞ 2 −y 2 /2 =√ dy √ e 2π a/ t Let us now consider the probability that Brownian motion hits A before −B where A > 0, B > 0. To compute this we shall make use of the interpretation of Brownian motion as being a limit of the symmetric random walk. To start let us recall from the results of the gambler’s ruin problem (see Section 4.5.1) that the probability that the symmetric random walk goes up A before going down B when each step is equally likely to be either up or down a distance x is [by Equation (4.14) with N = (A + B)/x, i = B/x] equal to Bx/(A + B)x = B/(A + B). Hence, upon letting x → 0, we see that P {up A before down B} =
B A+B
10.3.
10.3.
Variations on Brownian Motion
631
Variations on Brownian Motion
10.3.1. Brownian Motion with Drift We say that {X(t), t 0} is a Brownian motion process with drift coefficient μ and variance parameter σ 2 if (i) X(0) = 0; (ii) {X(t), t 0} has stationary and independent increments; (iii) X(t) is normally distributed with mean μt and variance tσ 2 . An equivalent definition is to let {B(t), t 0} be standard Brownian motion and then define X(t) = σ B(t) + μt 10.3.2. Geometric Brownian Motion If {Y (t), t 0} is a Brownian motion process with drift coefficient μ and variance parameter σ 2 , then the process {X(t), t 0} defined by X(t) = eY (t) is called geometric Brownian motion. For a geometric Brownian motion process {X(t)}, let us compute the expected value of the process at time t given the history of the process up to time s. That is, for s < t, consider E[X(t)|X(u), 0 u s]. Now, " # E[X(t)|X(u), 0 u s] = E eY (t) |Y (u), 0 u s " # = E eY (s)+Y (t)−Y (s) |Y (u), 0 u s " # = eY (s) E eY (t)−Y (s) |Y (u), 0 u s " # = X(s)E eY (t)−Y (s) where the next to last equality follows from the fact that Y (s) is given, and the last equality from the independent increment property of Brownian motion. Now, the moment generating function of a normal random variable W is given by E[eaW ] = eaE[W ]+a
2 Var(W )/2
632
10
Brownian Motion and Stationary Processes
Hence, since Y (t) − Y (s) is normal with mean μ(t − s) and variance (t − s)σ 2 , it follows by setting a = 1 that " # 2 E eY (t)−Y (s) = eμ(t−s)+(t−s)σ /2 Thus, we obtain E[X(t)|X(u), 0 u s] = X(s)e(t−s)(μ+σ
2 /2)
(10.8)
Geometric Brownian motion is useful in the modeling of stock prices over time when you feel that the percentage changes are independent and identically distributed. For instance, suppose that Xn is the price of some stock at time n. Then, it might be reasonable to suppose that Xn /Xn−1 , n 1, are independent and identically distributed. Let Yn = Xn /Xn−1 and so Xn = Yn Xn−1 Iterating this equality gives Xn = Yn Yn−1 Xn−2 = Yn Yn−1 Yn−2 Xn−3 .. . = Yn Yn−1 · · · Y1 X0 Thus, log(Xn ) =
n
log(Yi ) + log(X0 )
i=1
Since log(Yi ), i 1 are independent and identically distributed, {log(Xn )} will, when suitably normalized, approximately be Brownian motion with a drift, and so {Xn } will be approximately geometric Brownian motion.
10.4.
Pricing Stock Options
10.4.1. An Example in Options Pricing In situations in which money is to be received or paid out in differing time periods, we must take into account the time value of money. That is, to be given the amount v a time t in the future is not worth as much as being given v immediately. The reason for this is that if we were immediately given v, then it could be loaned out
10.4.
Pricing Stock Options
633
Figure 10.1.
with interest and so be worth more than v at time t. To take this into account, we will suppose that the time 0 value, also called the present value, of the amount v to be earned at time t is ve−αt . The quantity α is often called the discount factor. In economic terms, the assumption of the discount function e−αt is equivalent to the assumption that we can earn interest at a continuously compounded rate of 100α percent per unit time. We will now consider a simple model for pricing an option to purchase a stock at a future time at a fixed price. Suppose the present price of a stock is $100 per unit share, and suppose we know that after one time period it will be, in present value dollars, either $200 or $50 (see Figure 10.1). It should be noted that the prices at time 1 are the present value (or time 0) prices. That is, if the discount factor is α, then the actual possible prices at time 1 are either 200eα or 50eα . To keep the notation simple, we will suppose that all prices given are time 0 prices. Suppose that for any y, at a cost of cy, you can purchase at time 0 the option to buy y shares of the stock at time 1 at a (time 0) cost of $150 per share. Thus, for instance, if you do purchase this option and the stock rises to $200, then you would exercise the option at time 1 and realize a gain of $200 − 150 = $50 for each of the y option units purchased. On the other hand, if the price at time 1 was $50, then the option would be worthless at time 1. In addition, at a cost of 100x you can purchase x units of the stock at time 0, and this will be worth either 200x or 50x at time 1. We will suppose that both x or y can be either positive or negative (or zero). That is, you can either buy or sell both the stock and the option. For instance, if x were negative then you would be selling −x shares of the stock, yielding you a return of −100x, and you would then be responsible for buying −x shares of the stock at time 1 at a cost of either $200 or $50 per share. We are interested in determining the appropriate value of c, the unit cost of an option. Specifically, we will show that unless c = 50/3 there will be a combination of purchases that will always result in a positive gain. To show this, suppose that at time 0 we buy x units of stock, and buy y units of options
634
10
Brownian Motion and Stationary Processes
where x and y (which can be either positive or negative) are to be determined. The value of our holding at time 1 depends on the price of the stock at that time; and it is given by the following 200x + 50y, if price is 200 value = 50x, if price is 50 The preceding formula follows by noting that if the price is 200 then the x units of the stock are worth 200x, and the y units of the option to buy the stock at a unit price of 150 are worth (200 − 150)y. On the other hand, if the stock price is 50, then the x units are worth 50x and the y units of the option are worthless. Now, suppose we choose y to be such that the preceding value is the same no matter what the price at time 1. That is, we choose y so that 200x + 50y = 50x or y = −3x (Note that y has the opposite sign of x, and so if x is positive and as a result x units of the stock are purchased at time 0, then 3x units of stock options are also sold at that time. Similarly, if x is negative, then −x units of stock are sold and −3x units of stock options are purchased at time 0.) Thus, with y = −3x, the value of our holding at time 1 is value = 50x Since the original cost of purchasing x units of the stock and −3x units of options is original cost = 100x − 3xc, we see that our gain on the transaction is gain = 50x − (100x − 3xc) = x(3c − 50) Thus, if 3c = 50, then the gain is 0; on the other hand if 3c = 50, we can guarantee a positive gain (no matter what the price of the stock at time 1) by letting x be positive when 3c > 50 and letting it be negative when 3c < 50. For instance, if the unit cost per option is c = 20, then purchasing 1 unit of the stock (x = 1) and simultaneously selling 3 units of the option (y = −3) initially costs us 100 − 60 = 40. However, the value of this holding at time 1 is 50 whether
10.4.
Pricing Stock Options
635
the stock goes up to 200 or down to 50. Thus, a guaranteed profit of 10 is attained. Similarly, if the unit cost per option is c = 15, then selling 1 unit of the stock (x = −1) and buying 3 units of the option (y = 3) leads to an initial gain of 100 − 45 = 55. On the other hand, the value of this holding at time 1 is −50. Thus, a guaranteed profit of 5 is attained. A sure win betting scheme is called an arbitrage. Thus, as we have just seen, the only option cost c that does not result in an arbitrage is c = 50/3. 10.4.2. The Arbitrage Theorem Consider an experiment whose set of possible outcomes is S = {1, 2, . . . , m}. Suppose that n wagers are available. If the amount x is bet on wager i, then the return xri (j ) is earned if the outcome of the experiment is j . In other words, ri (·) is the return function for a unit bet on wager i. The amount bet on a wager is allowed to be either positive or negative or zero. A betting scheme is a vector x = (x1 , . . . , xn ) with the interpretation that x1 is bet on wager 1, x2 on wager 2, . . . , and xn on wager n. If the outcome of the experiment is j , then the return from the betting scheme x is
return from x =
n
xi ri (j )
i=1
The following theorem states that either there exists a probability vector p = (p1 , . . . , pm ) on the set of possible outcomes of the experiment under which each of the wagers has expected return 0, or else there is a betting scheme that guarantees a positive win.
Theorem 10.1 (The Arbitrage Theorem) Exactly one of the following is true: Either (i) there exists a probability vector p = (p1 , . . . , pm ) for which m
pj ri (j ) = 0,
for all i = 1, . . . , n
j =1
or (ii) there exists a betting scheme x = (x1 , . . . , xn ) for which n i=1
xi ri (j ) > 0,
for all j = 1, . . . , m
636
10
Brownian Motion and Stationary Processes
In other words, if X is the outcome of the experiment, then the arbitrage theorem states that either there is a probability vector p for X such that Ep [ri (X)] = 0,
for all i = 1, . . . , n
or else there is a betting scheme that leads to a sure win.
Remark This theorem is a consequence of the (linear algebra) theorem of the separating hyperplane, which is often used as a mechanism to prove the duality theorem of linear programming. The theory of linear programming can be used to determine a betting strategy that guarantees the greatest return. Suppose that the absolute value of the amount bet on each wager must be less than or equal to 1. To determine the vector x that yields the greatest guaranteed win—call this win v—we need to choose x and v so as to maximize v, subject to the constraints n
xi ri (j ) v,
for j = 1, . . . , m
i=1
−1 xi 1,
i = 1, . . . , n
This optimization problem is a linear program and can be solved by standard techniques (such as by using the simplex algorithm). The arbitrage theorem yields that m the optimal v will be positive unless there is a probability vector p for which j =1 pj ri (j ) = 0 for all i = 1, . . . , n.
Example 10.2 In some situations, the only types of wagers allowed are to
choose one of the outcomes i, i = 1, . . . , m, and bet that i is the outcome of the experiment. The return from such a bet is often quoted in terms of “odds.” If the odds for outcome i are oi (often written as “oi to 1”) then a 1 unit bet will return oi if the outcome of the experiment is i and will return −1 otherwise. That is, oi , if j = i ri (j ) = −1 otherwise Suppose the odds o1 , . . . , om are posted. In order for there not to be a sure win there must be a probability vector p = (p1 , . . . , pm ) such that 0 ≡ Ep [ri (X)] = oi pi − (1 − pi ) That is, we must have pi =
1 1 + oi
10.4.
Pricing Stock Options
637
Since the pi must sum to 1, this means that the condition for there not to be an arbitrage is that m (1 + oi )−1 = 1 i=1
Thus, if the posted odds are such that i (1 + oi )−1 = 1, then a sure win is possible. For instance, suppose there are three possible outcomes and the odds are as follows: Outcome Odds 1 1 2 2 3 3 That is, the odds for outcome 1 are 1 − 1, the odds for outcome 2 are 2 − 1, and that for outcome 3 are 3 − 1. Since 1 2
+
1 3
+
1 4
>1
a sure win is possible. One possibility is to bet −1 on outcome 1 (and so you either win 1 if the outcome is not 1 and lose 1 if the outcome is 1) and bet −0.7 on outcome 2, and −0.5 on outcome 3. If the experiment results in outcome 1, then we win −1 + 0.7 + 0.5 = 0.2; if it results in outcome 2, then we win 1 − 1.4 + 0.5 = 0.1; if it results in outcome 3, then we win 1 + 0.7 − 1.5 = 0.2. Hence, in all cases we win a positive amount. Remark If i (1 + oi )−1 = 1, then the betting scheme xi =
(1 + oi )−1 , 1 − i (1 + oi )−1
i = 1, . . . , n
will always yield a gain of exactly 1.
Example 10.3 Let us reconsider the option pricing example of the previous section, where the initial price of a stock is 100 and the present value of the price at time 1 is either 200 or 50. At a cost of c per share we can purchase at time 0 the option to buy the stock at time 1 at a present value price of 150 per share. The problem is to set the value of c so that no sure win is possible. In the context of this section, the outcome of the experiment is the value of the stock at time 1. Thus, there are two possible outcomes. There are also two different wagers: to buy (or sell) the stock, and to buy (or sell) the option. By the arbitrage theorem, there will be no sure win if there is a probability vector (p, 1 − p) that makes the expected return under both wagers equal to 0.
638
10
Brownian Motion and Stationary Processes
Now, the return from purchasing 1 unit of the stock is 200 − 100 = 100, if the price is 200 at time 1 return = 50 − 100 = −50, if the price is 50 at time 1 Hence, if p is the probability that the price is 200 at time 1, then E[return] = 100p − 50(1 − p) Setting this equal to 0 yields that p=
1 3
That is, the only probability vector (p, 1 − p) for which wager 1 yields an expected return 0 is the vector ( 13 , 23 ). Now, the return from purchasing one share of the option is 50 − c, if price is 200 return = −c, if price is 50 Hence, the expected return when p =
1 3
is
E[return] = (50 − c) 13 − c 23 =
50 3
−c
Thus, it follows from the arbitrage theorem that the only value of c for which there will not be a sure win is c = 50 3 , which verifies the result of section 10.4.1. 10.4.3. The Black-Scholes Option Pricing Formula Suppose the present price of a stock is X(0) = x0 , and let X(t) denote its price at time t. Suppose we are interested in the stock over the time interval 0 to T . Assume that the discount factor is α (equivalently, the interest rate is 100α percent compounded continuously), and so the present value of the stock price at time t is e−αt X(t). We can regard the evolution of the price of the stock over time as our experiment, and thus the outcome of the experiment is the value of the function X(t), 0 t T . The types of wagers available are that for any s < t we can observe the process for a time s and then buy (or sell) shares of the stock at price X(s) and then sell (or buy) these shares at time t for the price X(t). In addition, we will suppose that we may purchase any of N different options at time 0. Option i, costing ci per share, gives us the option of purchasing shares of the stock at time ti for the fixed price of Ki per share, i = 1, . . . , N . Suppose we want to determine values of the ci for which there is no betting strategy that leads to a sure win. Assuming that the arbitrage theorem can be gen-
10.4.
Pricing Stock Options
639
eralized (to handle the preceding situation, where the outcome of the experiment is a function), it follows that there will be no sure win if and only if there exists a probability measure over the set of outcomes under which all of the wagers have expected return 0. Let P be a probability measure on the set of outcomes. Consider first the wager of observing the stock for a time s and then purchasing (or selling) one share with the intention of selling (or purchasing) it at time t, 0 s < t T . The present value of the amount paid for the stock is e−αs X(s), whereas the present value of the amount received is e−αt X(t). Hence, in order for the expected return of this wager to be 0 when P is the probability measure on X(t), 0 t T , we must have that EP [e−αt X(t)|X(u), 0 u s] = e−αs X(s)
(10.9)
Consider now the wager of purchasing an option. Suppose the option gives us the right to buy one share of the stock at time t for a price K. At time t, the worth of this option will be as follows: X(t) − K, if X(t) K worth of option at time t = 0, if X(t) < K That is, the time t worth of the option is (X(t) − K)+ . Hence, the present value of the worth of the option is e−αt (X(t) − K)+ . If c is the (time 0) cost of the option, we see that, in order for purchasing the option to have expected (present value) return 0, we must have that EP [e−αt (X(t) − K)+ ] = c
(10.10)
By the arbitrage theorem, if we can find a probability measure P on the set of outcomes that satisfies Equation (10.9), then if c, the cost of an option to purchase one share at time t at the fixed price K, is as given in Equation (10.10), then no arbitrage is possible. On the other hand, if for given prices ci , i = 1, . . . , N , there is no probability measure P that satisfies both (10.9) and the equality ci = EP [e−αti (X(ti ) − Ki )+ ],
i = 1, . . . , N
then a sure win is possible. We will now present a probability measure P on the outcome X(t), 0 t T , that satisfies Equation (10.9). Suppose that X(t) = x0 eY (t) where {Y (t), t 0} is a Brownian motion process with drift coefficient μ and variance parameter σ 2 . That is, {X(t), t 0} is a geometric Brownian motion
640
10
Brownian Motion and Stationary Processes
process (see Section 10.3.2). From Equation (10.8) we have that, for s < t, E[X(t)|X(u), 0 u s] = X(s)e(t−s)(μ+σ
2 /2)
Hence, if we choose μ and σ 2 so that μ + σ 2 /2 = α then Equation (10.9) will be satisfied. That is, by letting P be the probability measure governing the stochastic process {x0 eY (t) , 0 t T }, where {Y (t)} is Brownian motion with drift parameter μ and variance parameter σ 2 , and where μ + σ 2 /2 = α, Equation (10.9) is satisfied. It follows from the preceding that if we price an option to purchase a share of the stock at time t for a fixed price K by c = EP [e−αt (X(t) − K)+ ] then no arbitrage is possible. Since X(t) = x0 eY (t) , where Y (t) is normal with mean μt and variance tσ 2 , we see that ∞ 1 2 2 (x0 ey − K)+ √ e−(y−μt) /2tσ dy ceαt = −∞ 2πtσ 2 ∞ 1 2 2 = (x0 ey − K) √ e−(y−μt) /2tσ dy 2 log(K/x0 ) 2πtσ Making the change of variable w = (y − μt)/(σ t 1/2 ) yields ∞ ∞ √ 1 2 αt μt 1 σ w t −w 2 /2 e e dw − K √ e−w /2 dw ce = x0 e √ 2π a 2π a where a= Now, 1 √ 2π
∞ a
√
eσ w t e−w
2 /2
log(K/x0 ) − μt √ σ t ∞ √ 2 1 e−(w−σ t) /2 dw √ 2π a √ ; < 2 = etσ /2 P N (σ t, 1) a √< ; 2 = etσ /2 P N (0, 1) a − σ t √ < ; 2 = etσ /2 P N (0, 1) − a − σ t √ 2 = etσ /2 φ σ t − a
dw = etσ
2 /2
(10.11)
10.4.
Pricing Stock Options
641
where N(m, v) is a normal random variable with mean m and variance v, and φ is the standard normal distribution function. Thus, we see from Equation (10.11) that ceαt = x0 eμt+σ
2 t/2
√ φ σ t − a − Kφ(−a)
Using that μ + σ 2 /2 = α and letting b = −a, we can write this as follows: √ c = x0 φ σ t + b − Ke−αt φ(b)
(10.12)
where b=
αt − σ 2 t/2 − log(K/x0 ) √ σ t
The option price formula given by Equation (10.12) depends on the initial price of the stock x0 , the option exercise time t, the option exercise price K, the discount (or interest rate) factor α, and the value σ 2 . Note that for any value of σ 2 , if the options are priced according to the formula of Equation (10.12) then no arbitrage is possible. However, as many people believe that the price of a stock actually follows a geometric Brownian motion—that is, X(t) = x0 eY (t) where Y (t) is Brownian motion with parameters μ and σ 2 —it has been suggested that it is natural to price the option according to the formula (10.12) with the parameter σ 2 taken equal to the estimated value (see the remark that follows) of the variance parameter under the assumption of a geometric Brownian motion model. When this is done, the formula (10.12) is known as the Black–Scholes option cost valuation. It is interesting that this valuation does not depend on the value of the drift parameter μ but only on the variance parameter σ 2 . If the option itself can be traded, then the formula of Equation (10.12) can be used to set its price in such a way so that no arbitrage is possible. If at time s the price of the stock is X(s) = xs , then the price of a (t, K) option—that is, an option to purchase one unit of the stock at time t for a price K—should be set by replacing t by t − s and x0 by xs in Equation (10.12).
Remark If we observe a Brownian motion process with variance parameter σ 2 over any time interval, then we could theoretically obtain an arbitrarily precise
642
10
Brownian Motion and Stationary Processes
estimate of σ 2 . For suppose we observe such a process {Y (s)} for a time t. Then, for fixed h, let N = [t/ h] and set W1 = Y (h) − Y (0), W2 = Y (2h) − Y (h), .. . WN = Y (Nh) − Y (N h − h) Then random variables W1 , . . . , WN are independent and identically distributed normal random variables having variance hσ 2 . We now use the fact (see Section 3.6.4) that (N − 1)S 2 /(σ 2 h) has a chi-squared distribution with N − 1 degrees of freedom, where S 2 is the sample variance defined by S2 =
N (Wi − W¯ )2 /(N − 1) i=1
Since the expected value and variance of a chi-squared with k degrees of freedom are equal to k and 2k, respectively, we see that E[(N − 1)S 2 /(σ 2 h)] = N − 1
and Var[(N − 1)S 2 /(σ 2 h)] = 2(N − 1) From this, we see that E[S 2 / h] = σ 2 and Var[S 2 / h] = 2σ 4 /(N − 1) Hence, as we let h become smaller (and so N = [t/ h] becomes larger) the variance of the unbiased estimator of σ 2 becomes arbitrarily small. Equation (10.12) is not the only way in which options can be priced so that no arbitrage is possible. Let {X(t), 0 t T } be any stochastic process satisfying,
10.4.
Pricing Stock Options
643
for s < t, E[e−αt X(t)|X(u), 0 u s] = e−αs X(s)
(10.13)
[that is, Equation (10.9) is satisfied]. By setting c, the cost of an option to purchase one share of the stock at time t for price K, equal to c = E[e−αt (X(t) − K)+ ]
(10.14)
it follows that no arbitrage is possible. Another type of stochastic process, aside from geometric Brownian motion, that satisfies Equation (10.13) is obtained as follows. Let Y1 , Y2 , . . . be a sequence of independent random variables having a common mean μ, and suppose that this process is independent of {N (t), t 0}, which is a Poisson process with rate λ. Let X(t) = x0
N (t)
Yi
i=1
Using the identity X(t) = x0
N (s) i=1
Yi
N (t)
Yj
j =N (s)+1
and the independent increment assumption of the Poisson process, we see that, for s < t, , N (t) E[X(t)|X(u), 0 u s] = X(s) E Yj j =N (s)+1
Conditioning on the number of events between s and t yields , N (t) ∞ Yj = μn e−λ(t−s) [λ(t − s)]n /n! E j =N (s)+1
n=0
= e−λ(t−s)(1−μ) Hence, E[X(t)|X(u), 0 u s] = X(s)e−λ(t−s)(1−μ) Thus, if we choose λ and μ to satisfy λ(1 − μ) = −α
644
10
Brownian Motion and Stationary Processes
then Equation (10.13) is satisfied. Therefore, if for any value of λ we let the Yi have any distributions with a common mean equal to μ = 1 + α/λ and then price the options according to Equation (10.14), then no arbitrage is possible.
Remark If {X(t), t 0} satisfies Equation (10.13), then the process {e−αt X(t), t 0} is called a Martingale. Thus, any pricing of options for which the expected gain on the option is equal to 0 when {e−αt X(t)} follows the probability law of some Martingale will result in no arbitrage possibilities. That is, if we choose any Martingale process {Z(t)} and let the cost of a (t, K) option be c = E[e−αt (eαt Z(t) − K)+ ] = E[(Z(t) − Ke−αt )+ ] then there is no sure win. In addition, while we did not consider the type of wager where a stock that is purchased at time s is sold not at a fixed time t but rather at some random time that depends on the movement of the stock, it can be shown using results about Martingales that the expected return of such wagers is also equal to 0.
Remark A variation of the arbitrage theorem was first noted by de Finetti in 1937. A more general version of de Finetti’s result, of which the arbitrage theorem is a special case, is given in Reference 3.
10.5.
White Noise
Let {X(t), t 0} denote a standard Brownian motion process and let f be a function having a continuous derivative in the region [a, b]. The stochastic integral b a f (t) dX(t) is defined as follows: b n f (t) dX(t) ≡ lim f (ti−1 )[X(ti ) − X(ti−1 )] (10.15) n→∞
a
i=1 max(ti −ti−1 )→0
where a = t0 < t1 < · · · < tn = b is a partition of the region [a, b]. Using the identity (the integration by parts formula applied to sums) n
f (ti−1 )[X(ti ) − X(ti−1 )]
i=1
= f (b)X(b) − f (a)X(a) −
n i=1
X(ti )[f (ti ) − f (ti−1 )]
10.5.
White Noise
645
we see that
b
b
f (t) dX(t) = f (b)X(b) − f (a)X(a) −
a
X(t) df (t)
(10.16)
a
b Equation (10.16) is usually taken as the definition of a f (t) dX(t). By using the right side of Equation (10.16) we obtain, upon assuming the interchangeability of expectation and limit, that . E
b
/ f (t) dX(t) = 0
a
Also, Var
n
f (ti−1 )[X(ti ) − X(ti−1 )] =
i=1
n
f 2 (ti−1 ) Var[X(ti ) − X(ti−1 )]
i=1
=
n
f 2 (ti−1 )(ti − ti−1 )
i=1
where the top equality follows from the independent increments of Brownian motion. Hence, we obtain from Equation (10.15) upon taking limits of the preceding that . Var
b a
/ f (t) dX(t) =
b
f 2 (t) dt
a
Remark The preceding gives operational meaning to the family of quantities {dX(t), 0 bt < ∞} by viewing it as an operator that carries functions f into the values a f (t) dX(t). This is called a white noise transformation, or more loosely {dX(t), 0 t < ∞} is called white noise since it can be imagined that a time varying bfunction f travels through a white noise medium to yield the output (at time b) a f (t) dX(t).
Example 10.4 Consider a particle of unit mass that is suspended in a liquid and suppose that, due to the liquid, there is a viscous force that retards the velocity of the particle at a rate proportional to its present velocity. In addition, let us suppose that the velocity instantaneously changes according to a constant multiple
646
10
Brownian Motion and Stationary Processes
of white noise. That is, if V (t) denotes the particle’s velocity at t, suppose that V (t) = −βV (t) + αX (t) where {X(t), t 0} is standard Brownian motion. This can be written as follows: eβt [V (t) + βV (t)] = αeβt X (t) or d βt [e V (t)] = αeβt X (t) dt Hence, upon integration, we obtain
t
eβt V (t) = V (0) + α
eβs X (s) ds
0
or V (t) = V (0)e
−βt
t
+α
e−β(t−s) dX(s)
0
Hence, from Equation (10.16), . / t V (t) = V (0)e−βt + α X(t) − X(s)βe−β(t−s) ds 0
10.6.
Gaussian Processes
We start with the following definition.
Definition 10.2 A stochastic process X(t), t 0 is called a Gaussian, or a normal, process if X(t1 ), . . . , X(tn ) has a multivariate normal distribution for all t1 , . . . , tn . If {X(t), t 0} is a Brownian motion process, then because each of X(t1 ), X(t2 ), . . . , X(tn ) can be expressed as a linear combination of the independent normal random variables X(t1 ), X(t2 ) − X(t1 ), X(t3 ) − X(t2 ), . . . , X(tn ) − X(tn−1 ) it follows that Brownian motion is a Gaussian process. Because a multivariate normal distribution is completely determined by the marginal mean values and the covariance values (see Section 2.6) it follows that
10.6.
647
Gaussian Processes
standard Brownian motion could also be defined as a Gaussian process having E[X(t)] = 0 and, for s t, Cov(X(s), X(t)) = Cov(X(s), X(s) + X(t) − X(s)) = Cov(X(s), X(s)) + Cov(X(s), X(t) − X(s)) = Cov(X(s), X(s)) =s
by independent increments
since Var(X(s)) = s
(10.17)
Let {X(t), t 0} be a standard Brownian motion process and consider the process values between 0 and 1 conditional on X(1) = 0. That is, consider the conditional stochastic process {X(t), 0 t 1|X(1) = 0}. Since the conditional distribution of X(t1 ), . . . , X(tn ) is multivariate normal it follows that this conditional process, known as the Brownian bridge (as it is tied down both at 0 and at 1), is a Gaussian process. Let us compute its covariance function. As, from Equation (10.4), E[X(s)|X(1) = 0] = 0,
for s < 1
we have that, for s < t < 1, Cov[(X(s), X(t))|X(1) = 0] = E[X(s)X(t)|X(1) = 0] = E[E[X(s)X(t)|X(t), X(1) = 0]|X(1) = 0] = E[X(t)E[X(s)|X(t)]|X(1) = 0] . / s = E X(t) X(t)|X(1) = 0 by (10.4) t s = E[X 2 (t)|X(1) = 0] t s = t (1 − t) by (10.4) t = s(1 − t) Thus, the Brownian bridge can be defined as a Gaussian process with mean value 0 and covariance function s(1 − t), s t. This leads to an alternative approach to obtaining such a process.
Proposition 10.1 If {X(t), t 0} is standard Brownian motion, then {Z(t), 0 t 1} is a Brownian bridge process when Z(t) = X(t) − tX(1).
648
10
Brownian Motion and Stationary Processes
Proof As it is immediate that {Z(t), t 0} is a Gaussian process, all we need verify is that E[Z(t)] = 0 and Cov(Z(s), Z(t)) = s(1 − t), when s t. The former is immediate and the latter follows from Cov(Z(s), Z(t)) = Cov(X(s) − sX(1), X(t) − tX(1)) = Cov(X(s), X(t)) − t Cov(X(s), X(1)) − s Cov(X(1), X(t)) + st Cov(X(1), X(1)) = s − st − st + st = s(1 − t) and the proof is complete.
If {X(t), t 0} is Brownian motion, then the process {Z(t), t 0} defined by t X(s) ds (10.18) Z(t) = 0
is called integrated Brownian motion. As an illustration of how such a process may arise in practice, suppose we are interested in modeling the price of a commodity throughout time. Letting Z(t) denote the price at t then, rather than assuming that {Z(t)} is Brownian motion (or that log Z(t) is Brownian motion), we might want to assume that the rate of change of Z(t) follows a Brownian motion. For instance, we might suppose that the rate of change of the commodity’s price is the current inflation rate which is imagined to vary as Brownian motion. Hence, d Z(t) = X(t), dt Z(t) = Z(0) +
t
X(s) ds 0
It follows from the fact that Brownian motion is a Gaussian process that {Z(t), t 0} is also Gaussian. To prove this, first recall that W1 , . . . , Wn is said to have a multivariate normal distribution if they can be represented as Wi =
m
aij Uj ,
i = 1, . . . , n
j =1
where Uj , j = 1, . . . , m are independent normal random variables. From this it follows that any set of partial sums of W1 , . . . , Wn are also jointly normal. The fact that Z(t1 ), . . . , Z(tn ) is multivariate normal can now be shown by writing the integral in Equation (10.18) as a limit of approximating sums.
10.7.
Stationary and Weakly Stationary Processes
649
As {Z(t), t 0} is Gaussian it follows that its distribution is characterized by its mean value and covariance function. We now compute these when {X(t), t 0} is standard Brownian motion. . t / E[Z(t)] = E X(s) ds =
0 t
E[X(s)] ds 0
=0 For s t, Cov[Z(s), Z(t)] = E[Z(s)Z(t)] . t / s =E X(y) dy X(u) du 0
=E
0
=
s
0 t
t
min(y, u) dy du 0
s 0
10.7.
/ X(y)X(u) dy du
0
s 0
=
t
E[X(y)X(u)] dy du 0
=
0
. s
u 0
y dy +
t
by (10.17)
u dy du = s
u
2
t s − 2 6
Stationary and Weakly Stationary Processes
A stochastic process {X(t), t 0} is said to be a stationary process if for all n, s, t, . . . , tn the random vectors X(t1 ), . . . , X(tn ) and X(t1 + s), . . . , X(tn + s) have the same joint distribution. In other words, a process is stationary if, in choosing any fixed point s as the origin, the ensuing process has the same probability law. Two examples of stationary processes are: (i) An ergodic continuous-time Markov chain {X(t), t 0} when P {X(0) = j } = Pj ,
j 0
where {Pj , j 0} are the limiting probabilities. (ii) {X(t), t 0} when X(t) = N (t + L) − N (t), t 0, where L > 0 is a fixed constant and {N (t), t 0} is a Poisson process having rate λ.
650
10
Brownian Motion and Stationary Processes
The first one of these processes is stationary for it is a Markov chain whose initial state is chosen according to the limiting probabilities, and it can thus be regarded as an ergodic Markov chain that we start observing at time ∞. Hence the continuation of this process at time s after observation begins is just the continuation of the chain starting at time ∞ + s, which clearly has the same probability for all s. That the second example—where X(t) represents the number of events of a Poisson process that occur between t and t + L—is stationary follows the stationary and independent increment assumption of the Poisson process which implies that the continuation of a Poisson process at any time s remains a Poisson process.
Example 10.5 (The Random Telegraph Signal Process) Let {N (t), t 0} denote a Poisson process, and let X0 be independent of this process and be such that P {X0 = 1} = P {X0 = −1} = 12 . Defining X(t) = X0 (−1)N (t) then {X(t), t 0} is called random telegraph signal process. To see that it is stationary, note first that starting at any time t, no matter what the value of N (t), as X0 is equally likely to be either plus or minus 1, it follows that X(t) is equally likely to be either plus or minus 1. Hence, because the continuation of a Poisson process beyond any time remains a Poisson process, it follows that {X(t), t 0} is a stationary process. Let us compute the mean and covariance function of the random telegraph signal " # E[X(t)] = E X0 (−1)N (t) " # = E X0 ]E[(−1)N (t) =0
by independence
since E[X0 ] = 0,
Cov[X(t), X(t + s)] = E[X(t)X(t + s)] # " = E X02 (−1)N (t)+N (t+s) = E[(−1)2N (t) (−1)N (t+s)−N (t) ] = E[(−1)N (t+s)−N (t) ] = E[(−1)N (s) ] =
∞ (λs)i (−1)i e−λs i! i=0
= e−2λs
(10.19)
For an application of the random telegraph signal consider a particle moving at a constant unit velocity along a straight line and suppose that collisions involving this particle occur at a Poisson rate λ. Also suppose that each time the
10.7.
Stationary and Weakly Stationary Processes
651
particle suffers a collision it reverses direction. Therefore, if X0 represents the initial velocity of the particle, then its velocity at time t—call it X(t)—is given by X(t) = X0 (−1)N (t) , where N (t) denotes the number of collisions involving the particle by time t. Hence, if X0 is equally likely to be plus or minus 1, and is independent of {N (t), t 0}, then {X(t), t 0} is a random telegraph signal process. If we now let t X(s) ds D(t) = 0
then D(t) represents the displacement of the particle at time t from its position at time 0. The mean and variance of D(t) are obtained as follows: t E[D(t)] = E[X(s)] ds = 0, 0
Var[D(t)] = E[D 2 (t)] . t / t =E X(y) dy X(u) du 0
=
t
0 t
E[X(y)X(u)] dy du 0
0
=2
E[X(y)X(u)] dy du
0X (w) = e−iws RX (s) ds R and >Y (w) = R
e−iws RY (s) ds
>X (w) denote the Fourier transforms, respectively, of RX and RY . The function R is also called the power spectral density of the process {X(t)}. Also, let > h(w) = e−iws h(s) ds denote the Fourier transform of the function h. Then, from Equation (10.22), > RY (w) = eiws RX (s − s2 + s1 )h(s1 )h(s2 ) ds1 ds2 ds =
eiw(s−s2 +s1 ) RX (s − s2 + s1 ) dse−iws2 h(s2 ) ds2 eiws1 h(s1 ) ds1
>X (w)> =R h(w)> h(−w)
(10.23)
Now, using the representation eix = cos x + i sin x, e−ix = cos(−x) + i sin(−x) = cos x − i sin x we obtain > h(w)> h(−w) =
. h(s) cos(ws) ds − i
h(s) sin(ws) ds
. ×
/
h(s) cos(ws) ds + i
. =
/
.
/2 h(s) cos(ws) ds
h(s) sin(ws) ds
+
! !2 ! ! −iws ! = ! h(s)e ds !! = |> h(w)|2
/2 h(s) sin(ws) ds
Exercises
657
Hence, from Equation (10.23) we obtain >X (w)|> >Y (w) = R h(w)|2 R In words, the Fourier transform of the covariance function of the output process is equal to the square of the amplitude of the Fourier transform of the impulse function multiplied by the Fourier transform of the covariance function of the input process.
Exercises In the following exercises {B(t), t 0} is a standard Brownian motion process and Ta denotes the time it takes this process to hit a. *1. What is the distribution of B(s) + B(t), s t? 2. Compute the conditional distribution of B(s) given that B(t1 ) = A, B(t2 ) = B, where 0 < t1 < s < t2 . *3. Compute E[B(t1 )B(t2 )B(t3 )] for t1 < t2 < t3 . 4.
Show that P {Ta < ∞} = 1, E[Ta ] = ∞,
a = 0
*5. What is P {T1 < T−1 < T2 }? 6. Suppose you own one share of a stock whose price changes according to a standard Brownian motion process. Suppose that you purchased the stock at a price b + c, c > 0, and the present price is b. You have decided to sell the stock either when it reaches the price b + c or when an additional time t goes by (whichever occurs first). What is the probability that you do not recover your purchase price? 7.
Compute an expression for P max B(s) > x t1 st2
8. Consider the random walk which in each t time unit either goes up or √ down the√amount t with respective probabilities p and 1 − p where p = 1 2 (1 + μ t). (a) Argue that as t → 0 the resulting limiting process is a Brownian motion process with drift rate μ.
658
10
Brownian Motion and Stationary Processes
(b) Using part (a) and the results of the gambler’s ruin problem (Section 4.5.1), compute the probability that a Brownian motion process with drift rate μ goes up A before going down B, A > 0, B > 0. 9. Let {X(t), t 0} be a Brownian motion with drift coefficient μ and variance parameter σ 2 . What is the joint density function of X(s) and X(t), s < t? *10. Let {X(t), t 0} be a Brownian motion with drift coefficient μ and variance parameter σ 2 . What is the conditional distribution of X(t) given that X(s) = c when (a) s < t? (b) t < s? 11. Consider a process whose value changes every h √time units; its new value σ h with probability p = being its√old value multiplied either √ by the factor e μ 1 −σ h with probability 1 − p. As h goes to zero, (1 + h), or by the factor e 2 σ show that this process converges to geometric Brownian motion with drift coefficient μ and variance parameter σ 2 . 12. A stock is presently selling at a price of $50 per share. After one time period, its selling price will (in present value dollars) be either $150 or $25. An option to purchase y units of the stock at time 1 can be purchased at cost cy. (a) (b) (c) (d)
What should c be in order for there to be no sure win? If c = 4, explain how you could guarantee a sure win. If c = 10, explain how you could guarantee a sure win. Use the arbitrage theorem to verify your answer to part (a).
13. Verify the statement made in the remark following Example 10.2. 14. The present price of a stock is 100. The price at time 1 will be either 50, 100, or 200. An option to purchase y shares of the stock at time 1 for the (present value) price ky costs cy. (a) If k = 120, show that an arbitrage opportunity occurs if and only if c > 80/3. (b) If k = 80, show that there is not an arbitrage opportunity if and only if 20 c 40. 15. The current price of a stock is 100. Suppose that the logarithm of the price of the stock changes according to a Brownian motion with drift coefficient μ = 2 and variance parameter σ 2 = 1. Give the Black-Scholes cost of an option to buy the stock at time 10 for a cost of (a) 100 per unit. (b) 120 per unit. (c) 80 per unit.
Exercises
659
Assume that the continuously compounded interest rate is 5 percent. A stochastic process {Y (t), t 0} is said to be a Martingale process if, for s < t, E[Y (t)|Y (u), 0 u s] = Y (s) 16. If {Y (t), t 0} is a Martingale, show that E[Y (t)] = E[Y (0)] 17. Show that standard Brownian motion is a Martingale. 18. Show that {Y (t), t 0} is a Martingale when Y (t) = B 2 (t) − t What is E[Y (t)]? Hint: First compute E[Y (t)|B(u), 0 u s]. *19. Show that {Y (t), t 0} is a Martingale when Y (t) = exp{cB(t) − c2 t/2} where c is an arbitrary constant. What is E[Y (t)]? An important property of a Martingale is that if you continually observe the process and then stop at some time T , then, subject to some technical conditions (which will hold in the problems to be considered), E[Y (T )] = E[Y (0)] The time T usually depends on the values of the process and is known as a stopping time for the Martingale. This result, that the expected value of the stopped Martingale is equal to its fixed time expectation, is known as the Martingale stopping theorem. *20. Let T = Min{t : B(t) = 2 − 4t} That is, T is the first time that standard Brownian motion hits the line 2 − 4t. Use the Martingale stopping theorem to find E[T ].
660
10
Brownian Motion and Stationary Processes
21. Let {X(t), t 0} be Brownian motion with drift coefficient μ and variance parameter σ 2 . That is, X(t) = σ B(t) + μt Let μ > 0, and for a positive constant x let T = Min{t : X(t) = x} x − μt = Min t : B(t) = σ That is, T is the first time the process {X(t), t 0} hits x. Use the Martingale stopping theorem to show that E[T ] = x/μ 22. Let X(t) = σ B(t) + μt, and for given positive constants A and B, let p denote the probability that {X(t), t 0} hits A before it hits −B. (a) Define the stopping time T to be the first time the process hits either A or −B. Use this stopping time and the Martingale defined in Exercise 19 to show that E[exp{c(X(T ) − μT )/σ − c2 T /2}] = 1 (b) Let c = −2μ/σ , and show that E[exp{−2μX(T )/σ }] = 1 (c) Use part (b) and the definition of T to find p. Hint: What are the possible values of exp{−2μX(T )/σ 2 }? 23. Let X(t) = σ B(t) + μt, and define T to be the first time the process {X(t), t 0} hits either A or −B, where A and B are given positive numbers. Use the Martingale stopping theorem and part (c) of Exercise 22 to find E[T ]. *24. Let {X(t), t 0} be Brownian motion with drift coefficient μ and variance parameter σ 2 . Suppose that μ > 0. Let x > 0 and define the stopping time T (as in Exercise 21) by T = Min{t : X(t) = x} Use the Martingale defined in Exercise 18, along with the result of Exercise 21, to show that Var(T ) = xσ 2 /μ3
Exercises
661
25. Compute the mean and variance of 1 (a) 0 t dB(t). 1 (b) 0 t 2 dB(t). 26. Let Y (t) = tB(1/t), t > 0 and Y (0) = 0. (a) What is the distribution of Y (t)? (b) Compare Cov(Y (s), Y (t)). (c) Argue that {Y (t), t 0} is a standard Brownian motion process. *27. Let Y (t) = B(a 2 t)/a for a > 0. Argue that {Y (t)} is a standard Brownian motion process. 28. For s < t, argue that B(s) − st B(t) and B(t) are independent. 29. Let {Z(t), t 0} denote a Brownian bridge process. Show that if Y (t) = (t + 1)Z(t/(t + 1)) then {Y (t), t 0} is a standard Brownian motion process. 30. Let X(t) = N (t + 1) − N (t) where {N (t), t 0} is a Poisson process with rate λ. Compute Cov[X(t), X(t + s)] *31. Let {N (t), t 0} denote a Poisson process with rate λ and define Y (t) to be the time from t until the next Poisson event. (a) Argue that {Y (t), t 0} is a stationary process. (b) Compute Cov[Y (t), Y (t + s)]. 32. Let {X(t), −∞ < t < ∞} be a weakly stationary process having covariance function RX (s) = Cov[X(t), X(t + s)]. (a) Show that Var(X(t + s) − X(t)) = 2RX (0) − 2RX (t) (b) If Y (t) = X(t + 1) − X(t) show that {Y (t), −∞ < t < ∞} is also weakly stationary having a covariance function RY (s) = Cov[Y (t), Y (t + s)] that satisfies RY (s) = 2RX (s) − RX (s − 1) − RX (s + 1) 33. Let Y1 and Y2 be independent unit normal random variables and for some constant w set X(t) = Y1 cos wt + Y2 sin wt,
−∞ < t < ∞
(a) Show that {X(t)} is a weakly stationary process. (b) Argue that {X(t)} is a stationary process.
662
10
Brownian Motion and Stationary Processes
34. Let {X(t), −∞ < t < ∞} be weakly stationary with covariance function > R(s) = Cov(X(t), X(t + s)) and let R(w) denote the power spectral density of the process. > > (i) Show that R(w) = R(−w). It can be shown that R(s) =
1 2π
∞
−∞
iws > dw R(w)e
(ii) Use the preceding to show that
∞
−∞
> R(w) dw = 2πE[X 2 (t)]
References 1. M. S. Bartlett, “An Introduction to Stochastic Processes,” Cambridge University Press, London, 1954. 2. U. Grenander and M. Rosenblatt, “Statistical Analysis of Stationary Time Series,” John Wiley, New York, 1957. 3. D. Heath and W. Sudderth, “On a Theorem of De Finetti, Oddsmaking, and Game Theory,” Ann. Math. Stat. 43, 2072–2077 (1972). 4. S. Karlin and H. Taylor, “A Second Course in Stochastic Processes,” Academic Press, Orlando, FL, 1981. 5. L. H. Koopmans, “The Spectral Analysis of Time Series,” Academic Press, Orlando, FL, 1974. 6. S. Ross, “Stochastic Processes,” Second Edition, John Wiley, New York, 1996.
Simulation
11 11.1.
Introduction
Let X = (X1 , . . . , Xn ) denote a random vector having a given density function f (x1 , . . . , xn ) and suppose we are interested in computing E[g(X)] = · · · g(x1 , . . . , xn )f (x1 , . . . , xn ) dx1 dx2 · · · dxn for some n-dimensional function g. For instance, g could represent the total delay in queue of the first [n/2] customers when the X values represent the first [n/2] interarrival and service times.∗ In many situations, it is not analytically possible either to compute the preceding multiple integral exactly or even to numerically approximate it within a given accuracy. One possibility that remains is to approximate E[g(X)] by means of simulation. (1) To approximate E[g(X)], start by generating a random vector X(1) = (X1 , . . . , (1) Xn ) having the joint density f (x1 , . . . , xn ) and then compute Y (1) = g(X(1) ). Now generate a second random vector (independent of the first) X(2) and compute Y (2) = g(X(2) ). Keep on doing this until r, a fixed number, of independent and identically distributed random variables Y (i) = g(X(i) ), i = 1, . . . , r have been generated. Now by the strong law of large numbers, we know that Y (1) + · · · + Y (r) = E[Y (i) ] = E[g(X)] r→∞ r lim
and so we can use the average of the generated Y s as an estimate of E[g(X)]. This approach to estimating E[g(X)] is called the Monte Carlo simulation approach. Clearly there remains the problem of how to generate, or simulate, random vectors having a specified joint distribution. The first step in doing this is to be ∗ We are using the notation [a] to represent the largest integer less than or equal to a.
663
664
11
Simulation
able to generate random variables from a uniform distribution on (0, 1). One way to do this would be to take 10 identical slips of paper, numbered 0, 1, . . . , 9, place them in a hat and then successively select n slips, with replacement, from the hat. The sequence of digits obtained (with a decimal point in front) can be regarded 1 n ) . as the value of a uniform (0, 1) random variable rounded off to the nearest ( 10 For instance, if the sequence of digits selected is 3, 8, 7, 2, 1, then the value of the uniform (0, 1) random variable is 0.38721 (to the nearest 0.00001). Tables of the values of uniform (0, 1) random variables, known as random number tables, have been extensively published [for instance, see The RAND Corporation, A Million Random Digits with 100,000 Normal Deviates (New York: The Free Press, 1955)]. Table 11.1 is such a table. However, this is not the way in which digital computers simulate uniform (0, 1) random variables. In practice, they use pseudo random numbers instead of truly random ones. Most random number generators start with an initial value X0 , called the seed, and then recursively compute values by specifying positive integers a, c, and m, and then letting Xn+1 = (aXn + c)
modulo m,
n0
where the preceding means that aXn + c is divided by m and the remainder is taken as the value of Xn+1 . Thus each Xn is either 0, 1, . . . , m − 1 and the quantity Xn /m is taken as an approximation to a uniform (0, 1) random variable. It can be shown that subject to suitable choices for a, c, m, the preceding gives rise to a sequence of numbers that looks as if it were generated from independent uniform (0, 1) random variables. As our starting point in the simulation of random variables from an arbitrary distribution, we shall suppose that we can simulate from the uniform (0, 1) distribution, and we shall use the term “random numbers” to mean independent random variables from this distribution. In Sections 11.2 and 11.3 we present both general and special techniques for simulating continuous random variables; and in Section 11.4 we do the same for discrete random variables. In Section 11.5 we discuss the simulation both of jointly distributed random variables and stochastic processes. Particular attention is given to the simulation of nonhomogeneous Poisson processes, and in fact three different approaches for this are discussed. Simulation of two-dimensional Poisson processes is discussed in Section 11.5.2. In Section 11.6 we discuss various methods for increasing the precision of the simulation estimates by reducing their variance; and in Section 11.7 we consider the problem of choosing the number of simulation runs needed to attain a desired level of precision. Before beginning this program, however, let us consider two applications of simulation to combinatorial problems.
Example 11.1 (Generating a Random Permutation) Suppose we are interested in generating a permutation of the numbers 1, 2, . . . , n that is such that all
11.1.
Table 11.1 04839 68086 39064 25669 64117 87917 62797 95876 29888 73577 27958 90999 18845 94824 35605 33362 88720 39475 06990 40980 83974 33339 31662 93526 20492 04153 05520 47498 23167 23792 85900 42559 14349 17403 23632
Introduction
665
A Random Number Table
96423 26432 66432 26422 94305 77341 56170 55293 88604 12908 30134 49127 49618 78171 81263 64270 82765 46473 67245 07391 29992 31926 25388 70765 38391 53381 91962 87637 49323 14422 98275 78985 82674 53363 27889
24878 46901 84673 44407 26766 42206 86324 18988 67917 30883 04024 20044 02304 84610 39667 01638 34476 23219 68350 58745 65381 14883 61642 10592 91132 79401 04739 99016 45021 15059 32388 05300 66523 44167 47914
82651 20848 40027 44048 25940 35126 88072 27354 48708 18317 86385 59931 51038 82834 47358 92477 17032 53416 82948 25774 38857 24413 34072 04542 21999 21438 13092 71060 33132 45799 52390 22164 44133 64486 02584
66566 89768 32832 37937 39972 74087 76222 26575 18912 28290 29880 06115 20655 09922 56873 66969 87589 94970 11398 22987 50490 59744 81249 76463 59516 83035 97662 88824 12544 22716 16815 24369 00697 64758 37680
14778 81536 61362 63904 22209 99547 36086 08625 82271 35797 99730 20542 58727 25417 56307 98420 40836 25832 42878 80059 83765 92351 35648 54328 81652 92350 24822 71013 41035 19792 69298 54224 35552 75366 20801
76797 86645 98947 45766 71500 81817 84637 40801 65424 05998 55536 18059 28168 44137 61607 04880 32427 69975 80287 39911 55657 97473 56891 02349 27195 36693 94730 18735 80780 09983 82732 35083 35970 76554 72152
14780 12659 96067 66134 64568 42607 93161 59920 69774 41688 84855 02008 15475 48413 49518 45585 70002 94884 88267 96189 14361 89286 69352 17247 48223 31238 06496 20286 45393 74353 38480 19687 19124 31601 39339
13300 92259 64760 75470 91402 43808 76038 29841 33611 34952 29080 73708 56942 25555 89356 46565 70663 19661 47363 41151 31720 35931 48373 28865 46751 59649 35090 23153 44812 68668 73817 11062 63318 12614 34806
87074 57102 64584 66520 42416 76655 65855 80150 54262 37888 09250 83517 53389 21246 20103 04102 88863 72828 46634 14222 57375 04110 45578 14777 22923 91754 04822 72924 12515 30429 32523 91491 29686 33072 08930
n! possible orderings are equally likely. The following algorithm will accomplish this by first choosing one of the numbers 1, . . . , n at random and then putting that number in position n; it then chooses at random one of the remaining n − 1 numbers and puts that number in position n − 1; it then chooses at random one of the remaining n − 2 numbers and puts it in position n − 2, and so on (where choosing a number at random means that each of the remaining numbers is equally likely
666
11
Simulation
to be chosen). However, so that we do not have to consider exactly which of the numbers remain to be positioned, it is convenient and efficient to keep the numbers in an ordered list and then randomly choose the position of the number rather than the number itself. That is, starting with any initial ordering p1 , p2 , . . . , pn , we pick one of the positions 1, . . . , n at random and then interchange the number in that position with the one in position n. Now we randomly choose one of the positions 1, . . . , n − 1 and interchange the number in this position with the one in position n − 1, and so on. To implement the preceding, we need to be able to generate a random variable that is equally likely to take on any of the values 1, 2, . . . , k. To accomplish this, let U denote a random number—that is, U is uniformly distributed over (0, 1)— and note that kU is uniform on (0, k) and so 1 P {i − 1 < kU < i} = , k
i = 1, . . . , k
Hence, if the random variable I = (kU ) + 1 will be such that P {I = i} = P {(kU ) = i − 1} = P {i − 1 < kU < i} =
1 k
The preceding algorithm for generating a random permutation can now be written as follows: Step 1: Step 2: Step 3: Step 4: Step 5: Step 6:
Let p1 , p2 , . . . , pn be any permutation of 1, 2, . . . , n (for instance, we can choose pj = j, j = 1, . . . , n). Set k = n. Generate a random number U and let I = (kU ) + 1. Interchange the values of pI and pk . Let k = k − 1 and if k > 1 go to Step 3. p1 , . . . , pn is the desired random permutation.
For instance, suppose n = 4 and the initial permutation is 1, 2, 3, 4. If the first value of I (which is equally likely to be either 1, 2, 3, 4) is I = 3, then the new permutation is 1, 2, 4, 3. If the next value of I is I = 2 then the new permutation is 1, 4, 2, 3. If the final value of I is I = 2, then the final permutation is 1, 4, 2, 3, and this is the value of the random permutation. One very important property of the preceding algorithm is that it can also be used to generate a random subset, say of size r, of the integers 1, . . . , n. Namely, just follow the algorithm until the positions n, n − 1, . . . , n − r + 1 are filled. The elements in these positions constitute the random subset.
Example 11.2 (Estimating the Number of Distinct Entries in a Large List) Consider a list of n entries where n is very large, and suppose we are interested
11.1.
Introduction
667
in estimating d, the number of distinct elements in the list. If we let mi denotethe number of times that the element in position i appears on the list, then we can express d by d=
n 1 mi i=1
To estimate d, suppose that we generate a random value X equally likely to be either 1, 2, . . . , n (that is, we take X = [nU ] + 1) and then let m(X) denote the number of times the element in position X appears on the list. Then . / n 1 1 1 d E = = m(X) mi n n i=1
Hence, if we generate k such random variables X1 , . . . , Xk we can estimate d by n ki=1 1/m(Xi ) d≈ k Suppose now that each item in the list has a value attached to it—v(i) being the value of the ith element. The sum of the values of the distinct items—call it v— can be expressed as v=
n v(i) m(i) i=1
Now if X = [nU ] + 1, where U is a random number, then / . n v(X) v(i) 1 v = = E m(X) m(i) n n i=1
Hence, we can estimate v by generating X1 , . . . , Xk and then estimating v by n v(Xi ) k m(Xi ) k
v≈
i=1
For an important application of the preceding, let Ai = {ai,1 , . . . , a
i,ni }, i = 1, . . . , s denote events, and suppose we are interested in estimating P ( si=1 Ai ). Since s ni s
P (ai,j ) P Ai = P (a) = m(ai,j ) i=1
a∈∪Ai
i=1 j =1
where m(ai,j ) is the number of events to
which the point ai,j belongs, the preceding method can be used to estimate P ( s1 Ai ).
668
11
Simulation
Note that the preceding procedure for estimating v can be effected without prior knowledge of the set of values {v1 , . . . , vn }. That is, it suffices that we can determine the value of an element in a specific place and the number of times that element appears on the list. When the set of values is a priori known, there is a more efficient approach available as will be shown in Example 11.11.
11.2.
General Techniques for Simulating Continuous Random Variables
In this section we present three methods for simulating continuous random variables. 11.2.1. The Inverse Transformation Method A general method for simulating a random variable having a continuous distribution—called the inverse transformation method—is based on the following proposition.
Proposition 11.1 Let U be a uniform (0, 1) random variable. For any continuous distribution function F if we define the random variable X by X = F −1 (U ) then the random variable X has distribution function F . [F −1 (u) is defined to equal that value x for which F (x) = u.]
Proof FX (a) = P {X a} = P {F −1 (U ) a}
(11.1)
Now, since F (x) is a monotone function, it follows that F −1 (U ) a if and only if U F (a). Hence, from Equation (11.1), we see that FX (a) = P {U F (a)} = F (a) Hence we can simulate a random variable X from the continuous distribution F , when F −1 is computable, by simulating a random number U and then setting X = F −1 (U ).
11.2.
General Techniques for Simulating Continuous Random Variables
669
Example 11.3 (Simulating an Exponential Random Variable) If F (x) = 1 − e−x , then F −1 (u) is that value of x such that
1 − e−x = u or x = − log(1 − u) Hence, if U is a uniform (0, 1) variable, then F −1 (U ) = − log(1 − U ) is exponentially distributed with mean 1. Since 1 − U is also uniformly distributed on (0, 1) it follows that − log U is exponential with mean 1. Since cX is exponential with mean c when X is exponential with mean 1, it follows that −c log U is exponential with mean c. 11.2.2. The Rejection Method Suppose that we have a method for simulating a random variable having density function g(x). We can use this as the basis for simulating from the continuous distribution having density f (x) by simulating Y from g and then accepting this simulated value with a probability proportional to f (Y )/g(Y ). Specifically let c be a constant such that f (y) c g(y)
for all y
We then have the following technique for simulating a random variable having density f . Rejection Method Step 1: Step 2:
Simulate Y having density g and simulate a random number U . If U f (Y )/cg(Y ) set X = Y . Otherwise return to Step 1.
Proposition 11.2 The random variable X generated by the rejection method has density function f .
670
11
Simulation
Proof Let X be the value obtained, and let N denote the number of necessary iterations. Then P {X x} = P {YN x} = P {Y x|U f (Y )/cg(Y )} P {Y x, U f (Y )/cg(Y )} K P {Y x, U f (Y )/cg(Y )|Y = y}g(y) dy = K x (f (y)/cg(y))g(y) dy = −∞ K x f (y) dy = −∞ Kc =
where K = P {U f (Y )/cg(Y )}. Letting x → ∞ shows that K = 1/c and the proof is complete.
Remarks (i) The preceding method was originally presented by Von Neumann in the special case where g was positive only in some finite interval (a, b), and Y was chosen to be uniform over (a, b) [that is, Y = a + (b − a)U ]. (ii) Note that the way in which we “accept the value Y with probability f (Y )/cg(Y )” is by generating a uniform (0, 1) random variable U and then accepting Y if U f (Y )/cg(Y ). (iii) Since each iteration of the method will, independently, result in an accepted value with probability P {U f (Y )/cg(Y )} = 1/c it follows that the number of iterations is geometric with mean c. (iv) Actually, it is not necessary to generate a new uniform random number when deciding whether or not to accept, since at a cost of some additional computation, a single random number, suitably modified at each iteration, can be used throughout. To see how, note that the actual value of U is not used—only whether or not U < f (Y )/cg(Y ). Hence, if Y is rejected—that is, if U > f (Y )/cg(Y )— we can use the fact that, given Y , U − f (Y )/cg(Y ) cUg(Y ) − f (Y ) = 1 − f (Y )/cg(Y ) cg(Y ) − f (Y ) is uniform on (0, 1). Hence, this may be used as a uniform random number in the next iteration. As this saves the generation of a random number at the cost of the preceding computation, whether it is a net savings depends greatly upon the method being used to generate random numbers.
11.2.
General Techniques for Simulating Continuous Random Variables
671
Example 11.4 Let us use the rejection method to generate a random variable having density function f (x) = 20x(1 − x)3 ,
0<x 1, return to step 1. Return the independent unit normals 0 X=
−2 log S V1 , S
0 Y=
−2 log S V2 S
The preceding is called the polar method. Since the probability that a random point in the square will fall within the circle is equal to π/4 (the area of the circle divided by the area of the square), it follows that, on average, the polar method will require 4/π = 1.273 iterations of step 1. Hence, it will, on average, require 2.546 random numbers, 1 logarithm, 1 square root, 1 division, and 4.546 multiplications to generate 2 independent standard normals. 11.3.2. The Gamma Distribution To simulate from a gamma distribution with parameters (n, λ), where n is an integer, we use the fact that the sum of n independent exponential random variables each having rate λ has this distribution. Hence, if U1 , . . . , Un are independent uniform (0, 1) random variables, n n 1 1 log Ui = − log Ui X= λ λ i=1
has the desired distribution.
i=1
11.3.
Special Techniques for Simulating Continuous Random Variables
681
When n is large, there are other techniques available that do not require so many random numbers. One possibility is to use the rejection procedure with g(x) being taken as the density of an exponential random variable with mean n/λ (as this is the mean of the gamma). It can be shown that for large n the average number of iterations needed by the rejection algorithm is e[(n − 1)/2π]1/2 . In addition, if we wanted to generate a series of gammas, then, just as in Example 11.4, we can arrange things so that upon acceptance we obtain not only a gamma random variable but also, for free, an exponential random variable that can then be used in obtaining the next gamma (see Exercise 8). 11.3.3. The Chi-Squared Distribution The chi-squared distribution with n degrees of freedom is the distribution of χn2 = Z12 + · · · + Zn2 where Zi , i = 1, . . . , n are independent standard normals. Using the fact noted in the remark at the end of Section 3.1 we see that Z12 + Z22 has an exponential distribution with rate 12 . Hence, when n is even—say n = 2k— 1 2 has a gamma distribution with parameters (k, 1 ). Hence, −2 log( k U ) χ2k i=1 i 2 has a chi-squared distribution with 2k degrees of freedom. We can simulate a chisquared random variable with 2k + 1 degrees of freedom by first simulating a standard normal random variable Z and then adding Z 2 to the preceding. That is, k 2 = Z 2 − 2 log Ui χ2k+1 i=1
where Z, U1 , . . . , Un are independent with Z being a standard normal and the others being uniform (0, 1) random variables. 11.3.4. The Beta (n, m) Distribution The random variable X is said to have a beta distribution with parameters n, m if its density is given by f (x) =
(n + m − 1)! n−1 x (1 − x)m−1 , (n − 1)!(m − 1)!
0<x x})m−1 = x n−1 (1 − x)m−1 Hence, as there are (n + m − 1)!/(n − 1)!(m − 1)! possible partitions, it follows that U(n) is beta with parameters (n, m). Thus, one way to simulate from the beta distribution is to find the nth smallest of a set of n + m − 1 random numbers. However, when n and m are large, this procedure is not particularly efficient. For another approach consider a Poisson process with rate 1, and recall that given Sn+m , the time of the (n + m)th event, the set of the first n + m − 1 event times is distributed independently and uniformly on (0, Sn+m ). Hence, given Sn+m , the nth smallest of the first n + m − 1 event times—that is Sn —is distributed as the nth smallest of a set of n + m − 1 uniform (0, Sn+m ) random variables. But from the preceding we can thus conclude that Sn /Sn+m has a beta distribution with parameters (n, m). Therefore, if U1 , . . . , Un+m are random numbers, 1 − log ni=1 Ui is beta with parameters (n, m) 1m+n − log i=1 Ui By writing the preceding as 1 − log ni=1 Ui 1 1 − log n1 Ui − log n+m n+1 Ui we see that it has the same distribution as X/(X + Y ) where X and Y are independent gamma random variables with respective parameters (n, 1) and (m, 1). Hence, when n and m are large, we can efficiently simulate a beta by first simulating two gamma random variables. 11.3.5. The Exponential Distribution—The Von Neumann Algorithm As we have seen, an exponential random variable with rate 1 can be simulated by computing the negative of the logarithm of a random number. Most computer programs for computing a logarithm, however, involve a power series expansion, and so it might be useful to have at hand a second method that is computationally easier. We now present such a method due to Von Neumann.
11.3.
Special Techniques for Simulating Continuous Random Variables
683
To begin let U1 , U2 , . . . be independent uniform (0, 1) random variables and define N, N 2, by N = min{n: U1 U2 · · · Un−1 < Un } That is, N is the first random number that is greater than its predecessor. Let us now compute the joint distribution of N and U1 : 1 P {N > n, U1 y} = P {N > n, U1 y|U1 = x} dx 0
=
y
P {N > n|U1 = x} dx
0
Now, given that U1 = x, N will be greater than n if x U2 · · · Un or, equivalently, if (a) Ui x, i = 2, . . . , n and (b) U2 · · · Un n−1 Now, (a) has probability x of occurring and given (a), since all of the (n − 1)! possible rankings of U2 , . . . , Un are equally likely, (b) has probability 1/(n − 1)! of occurring. Hence, P {N > n|U1 = x} = and so
y
P {N > n, U1 y} = 0
x n−1 (n − 1)!
yn x n−1 dx = (n − 1)! n!
which yields that P {N = n, U1 y} = P {N > n − 1, U1 y} − P {N > n, U1 y} =
yn y n−1 − (n − 1)! n!
Upon summing over all the even integers, we see that P {N is even, U1 y} = y −
y2 y3 y4 + − − ··· 2! 3! 4!
= 1 − e−y
(11.5)
We are now ready for the following algorithm for generating an exponential random variable with rate 1.
684
11
Step 1: Step 2: Step 3:
Simulation
Generate uniform random numbers U1 , U2 , . . . stopping at N = min{n: U1 · · · Un−1 < Un }. If N is even accept that run, and go to step 3. If N is odd reject the run, and return to step 1. Set X equal to the number of failed runs plus the first random number in the successful run.
To show that X is exponential with rate 1, first note that the probability of a successful run is, from Equation (11.5) with y = 1, P {N is even} = 1 − e−1 Now, in order for X to exceed x, the first [x] runs must all be unsuccessful and the next run must either be unsuccessful or be successful but have U1 > x − [x] (where [x] is the largest integer not exceeding x). As P {N even, U1 > y} = P {N even} − P {N even, U1 y} = 1 − e−1 − (1 − e−y ) = e−y − e−1 we see that P {X > x} = e−[x] [e−1 + e−(x−[x]) − e−1 ] = e−x which yields the result. Let T denote the number of trials needed to generate a successful run. As each trial is a success with probability 1 −e−1 it follows that T is geometric with means 1/(1 − e−1 ). If we let Ni denote the number of uniform random variables used on the ith run, i 1, then T (being the first run i for which Ni is even) is a stopping time for this sequence. Hence, by Wald’s equation, the mean number of uniform random variables needed by this algorithm is given by , T E Ni = E[N ]E[T ] i=1
Now, E[N ] =
∞
P {N > n}
n=0
=1+
∞
P {U1 · · · Un }
n=1
=1+
∞ n=1
1/n! = e
11.4.
and so E
, T
Simulating from Discrete Distributions
Ni =
i=1
685
e ≈ 4.3 1 − e−1
Hence, this algorithm, which computationally speaking is quite easy to perform, requires on the average about 4.3 random numbers to execute.
11.4.
Simulating from Discrete Distributions
All of the general methods for simulating from continuous distributions have analogs in the discrete case. For instance, if we want to simulate a random variable X having probability mass function j = 1, 2, . . . , Pj = 1 P {X = xj } = Pj , j
We can use the following discrete time analog of the inverse transform technique. To simulate X for which P {X = xj } = Pj let U be uniformly distributed over (0, 1), and set ⎧ x1 , if U < P1 ⎪ ⎪ ⎪ ⎪ x2 , if P1 < U < P1 + P2 ⎪ ⎪ ⎪ .. ⎪ ⎪ ⎨ . j −1 j X= ⎪ ⎪ x , if P < U < Pi ⎪ j i ⎪ ⎪ ⎪ ⎪ i 1 ⎪ ⎪ . ⎩ .. As, P {X = xj } = P
j −1
Pi < U
j − 1} = 1 − (1 − p)j −1
i=1
we can simulate such a random variable by generating a random number U and then setting X equal to that value j for which 1 − (1 − p)j −1 < U < 1 − (1 − p)j or, equivalently, for which (1 − p)j < 1 − U < (1 − p)j −1 As 1 − U has the same distribution as U , we can thus define X by X = min{j : (1 − p)j < U } = min j : j > .
log U =1+ log(1 − p)
/
log U log(1 − p)
As in the continuous case, special simulation techniques have been developed for the more common discrete distributions. We now present certain of these.
Example 11.8 (Simulating a Binomial Random Variable) A binomial (n, p) random variable can be most easily simulated by recalling that it can be expressed as the sum of n independent Bernoulli random variables. That is, if U1 , . . . , Un are independent uniform (0, 1) variables, then letting 1, Xi = 0,
if Ui < p otherwise
it follows that X ≡ ni=1 Xi is a binomial random variable with parameters n and p. One difficulty with this procedure is that it requires the generation of n random numbers. To show how to reduce the number of random numbers needed, note first that this procedure does not use the actual value of a random number U but only whether or not it exceeds p. Using this and the result that the conditional distribution of U given that U < p is uniform on (0, p) and the conditional distribution of U given that U > p is uniform on (p, 1), we now show how we can simulate a binomial (n, p) random variable using only a single random number:
11.4.
Step 1: Step 2: Step 3: Step 4: Step 5:
Simulating from Discrete Distributions
687
Let α = 1/p, β = 1/(1 − p). Set k = 0. Generate a uniform random number U . If k = n stop. Otherwise reset k to equal k + 1. If U p set Xk = 1 and reset U to equal αU . If U > p set Xk = 0 and reset U to equal β(U − p). Return to step 4.
This procedure generates X1 , . . . , Xn and X = ni=1 Xi is the desired random variable. It works by noting whether Uk p or Uk > p; in the former case it takes Uk+1 to equal Uk /p, and in the latter case it takes Uk+1 to equal (Uk − p)/ (1 − p).†
Example 11.9 (Simulating a Poisson Random Variable) To simulate a Poisson random variable with mean λ, generate independent uniform (0, 1) random variables U1 , U2 , . . . stopping at n −λ N + 1 = min n: Ui < e i=1
The random variable N has the desired distribution, which can be seen by noting that n N = max n : − log Ui < λ i=1
But − log Ui is exponential with rate 1, and so if we interpret − log Ui , i 1, as the interarrival times of a Poisson process having rate 1, we see that N = N (λ) would equal the number of events by time λ. Hence N is Poisson with mean λ. When λ is large we can reduce the amount of computation in the preceding simulation of N (λ), the number of events by time λ of a Poisson process having rate 1, by first choosing an integer m and simulating Sm , the time of the mth event of the Poisson process and then simulating N (λ) according to the conditional distribution of N (λ) given Sm . Now the conditional distribution of N (λ) given Sm is as follows: N (λ)|Sm = s ∼ m + Poisson(λ − s), λ , N (λ)|Sm = s ∼ Binomial m − 1, s
if s < λ if s > λ
where ∼ means “has the distribution of.” This follows since if the mth event occurs at time s, where s < λ, then the number of events by time λ is m plus the † Because of computer round-off errors, a single random number should not be continuously used when n is large.
688
11
Simulation
number of events in (s, λ). On the other hand given that Sm = s the set of times at which the first m − 1 events occur has the same distribution as a set of m − 1 uniform (0, s) random variables (see Section 5.3.5). Hence, when λ < s, the number of these which occur by time λ is binomial with parameters m − 1 and λ/s. Hence, we can simulate N (λ) by first simulating Sm and then simulate either P (λ − Sm ), a Poisson random variable with mean λ − Sm when Sm < λ, or simulate Bin(m − 1, λ/Sm ), a binomial random variable with parameters m − 1, and λ/Sm , when Sm > λ; and then setting m + P (λ − Sm ), if Sm < λ N (λ) = if Sm > λ Bin(m − 1, λ/Sm ), In the preceding it has been found computationally effective to let m be approximately 78 λ. Of course, Sm is simulated by simulating from a gamma (m, λ) distribution via an approach that is computationally fast when m is large (see Section 11.3.3). There are also rejection and hazard rate methods for discrete distributions but we leave their development as exercises. However, there is a technique available for simulating finite discrete random variables—called the alias method—which, though requiring some setup time, is very fast to implement. 11.4.1. The Alias Method In what follows, the quantities P, P(k) , Q(k) , k n − 1 will represent probability mass functions on the integers 1, 2, . . . , n—that is, they will be n-vectors of nonnegative numbers summing to 1. In addition, the vector P(k) will have at most k nonzero components, and each of the Q(k) will have at most two nonzero components. We show that any probability mass function P can be represented as an equally weighted mixture of n − 1 probability mass functions Q (each having at most two nonzero components). That is, we show that for suitably defined Q(1) , . . . , Q(n−1) , P can be expressed as 1 (k) Q n−1 n−1
P=
(11.6)
k=1
As a prelude to presenting the method for obtaining this representation, we will need the following simple lemma whose proof is left as an exercise.
Lemma 11.1 Let P = {Pi , i = 1, . . . , n} denote a probability mass function, then (a) there exists an i, 1 i n, such that Pi < 1/(n − 1), and (b) for this i, there exists a j , j = i, such that Pi + Pj 1/(n − 1).
11.4.
Simulating from Discrete Distributions
689
Before presenting the general technique for obtaining the representation of Equation (11.6), let us illustrate it by an example.
Example 11.10 Consider the three-point distribution P with P1 =
7 16 ,
1 . We start by choosing i and j such that they satisfy the conP2 = 12 , P3 = 16 ditions of Lemma 11.5. As P3 < 12 and P3 + P2 > 12 , we can work with i = 3 and j = 2. We will now define a 2-point mass function Q(1) putting all of its weight on 3 and 2 and such that P will be expressible as an equally weighted mixture between Q(1) and a second 2-point mass function Q(2) . Secondly, all of the mass of point 3 will be contained in Q(1) . As we will have
Pj =
1 2
(1) (2) Qj + Qj ,
j = 1, 2, 3
(11.7)
(2)
and, by the preceding, Q3 is supposed to equal 0, we must therefore take (1)
(1)
Q3 = 2P3 = 18 ,
(1)
(1)
Q2 = 1 − Q3 = 78 ,
Q1 = 0
To satisfy Equation (11.7), we must then set (2)
(2)
Q3 = 0,
Q2 = 2P2 −
7 8
(2)
= 18 ,
Q1 = 2P1 =
7 8
Hence, we have the desired representation in this case. Suppose now that the original distribution was the following 4-point mass function: P1 =
P2 = 14 ,
7 16 ,
P3 = 18 ,
P4 =
3 16
Now, P3 < 13 and P3 + P1 > 13 . Hence our initial 2-point mass function—Q(1) — will concentrate on points 3 and 1 (giving no weights to 2 and 4). As the final representation will give weight 13 to Q(1) and in addition the other Q(j ) , j = 2, 3, will not give any mass to the value 3, we must have that 1 (1) 3 Q3
= P3 =
1 8
Hence, (1)
Q3 = 38 ,
(1)
Q1 = 1 −
Also, we can write P = 13 Q(1) + 23 P(3)
3 8
=
5 8
690
11
Simulation
where P(3) , to satisfy the preceding, must be the vector (3) (1) P1 = 32 P1 − 13 Q1 = 13 21 , (3)
P2 = 32 P2 = 38 , (3)
P3 = 0, (3)
P4 = 32 P4 =
9 32
Note that P(3) gives no mass to the value 3. We can now express the mass function P(3) as an equally weighted mixture of 2-point mass functions Q(2) and Q(3) , and we will end up with P = 13 Q(1) + 23 12 Q(2) + 12 Q(3) = 13 Q(1) + Q(2) + Q(3) (We leave it as an exercise for you to fill in the details.)
The preceding example outlines the following general procedure for writing the n-point mass function P in the form of Equation (11.6) where each of the Q(i) are mass functions giving all their mass to at most 2 points. To start, we choose i and j satisfying the conditions of Lemma 11.5. We now define the mass function Q(1) concentrating on the points i and j and which will contain all of the mass (k) for point i by noting that, in the representation of Equation (11.6), Qi = 0 for k = 2, . . . , n − 1, implying that (1)
Qi = (n − 1)Pi ,
(1)
and so Qj = 1 − (n − 1)Pi
Writing P=
1 n − 2 (n−1) Q(1) + P n−1 n−1
(11.8)
where P(n−1) represents the remaining mass, we see that (n−1)
= 0,
(n−1)
=
n−1 1 1 n−1 (1) = Pj − Qj Pi + Pj − , n−2 n−1 n−2 n−1
(n−1)
=
n−1 Pk , n−2
Pi
Pj
Pk
k = i or j
That the foregoing is indeed a probability mass function is easily checked—for (n−1) instance, the nonnegativity of Pj follows from the fact that j was chosen so that Pi + Pj 1/(n − 1).
11.4.
Simulating from Discrete Distributions
691
We may now repeat the foregoing procedure on the (n − 1)-point probability mass function P(n−1) to obtain P(n−1) =
1 n − 3 (n−2) Q(2) + P n−2 n−2
and thus from Equation (11.8) we have P=
1 1 n − 3 (n−2) Q(1) + Q(2) + P n−1 n−1 n−1
We now repeat the procedure on P(n−2) and so on until we finally obtain P=
1 (1) Q + · · · + Q(n−1) n−1
In this way we are able to represent P as an equally weighted mixture of n − 1 2-point mass functions. We can now easily simulate from P by first generating a random integer N equally likely to be either 1, 2, . . . , n − 1. If the resulting value N is such that Q(N ) puts positive weight only on the points iN and jN , then we (N ) can set X equal to iN if a second random number is less than QiN and equal to jN otherwise. The random variable X will have probability mass function P. That is, we have the following procedure for simulating from P. Step 1: Step 2:
Generate U1 and set N = 1 + [(n − 1)U1 ]. Generate U2 and set X=
iN , jN ,
(N )
if U2 < QiN otherwise
Remarks (i) The preceding is called the alias method because by a renumber(k)
ing of the Qs we can always arrange things so that for each k, Qk > 0. (That is, we can arrange things so that the kth 2-point mass function gives positive weight to the value k.) Hence, the procedure calls for simulating N , equally likely to be 1, 2, . . . , n − 1, and then if N = k it either accepts k as the value of X, or it accepts for the value of X the “alias” of k (namely, the other value that Q(k) gives positive weight). (ii) Actually, it is not necessary to generate a new random number in step 2. Because N − 1 is the integer part of (n − 1)U1 , it follows that the remainder (n − 1)U1 − (N − 1) is independent of U1 and is uniformly distributed in (0, 1). Hence, rather than generating a new random number U2 in step 2, we can use (n − 1)U1 − (N − 1) = (n − 1)U1 − [(n − 1)U1 ].
692
11
Simulation
Example 11.11 Let us return to the problem of Example 11.1 which considers a list of n, not necessarily distinct, items. Each item has a value—v(i) being the value of the item in position i—and we are interested in estimating v=
n
v(i)/m(i)
i=1
where m(i) is the number of times the item in position i appears on the list. In words, v is the sum of the values of the (distinct) items on the list. To estimate v, note that if X is a random variable such that n = P {X = i} = v(i) v(j ),
i = 1, . . . , n
1
then
E[1/m(X)] =
= v(i)/m(i) v(j ) =v j v(j ) n
i
j =1
Hence, we can estimate v by using the alias (or any other) method to generate independent random variables X1 , . . . , Xk having the same distribution as X and then estimating v by v≈
n k 1 v(j ) 1/m(Xi ) k j =1
11.5.
i=1
Stochastic Processes
We can easily simulate a stochastic process by simulating a sequence of random variables. For instance, to simulate the first t time units of a renewal process having interarrival distribution F we can simulate independent random variables X1 , X2 , . . . having distribution F , stopping at N = min{n: X1 + · · · + Xn > t} The Xi , i 1, represent the interarrival times of the renewal process and so the preceding simulation yields N − 1 events by time t—the events occurring at times X1 , X1 + X2 , . . . , X1 + · · · + XN −1 . Actually there is another approach for simulating a Poisson process that is quite efficient. Suppose we want to simulate the first t time units of a Poisson process having rate λ. To do so, we can first simulate N (t), the number of events by t, and then use the result that given the value of N (t), the set of N (t) event times is
11.5.
Stochastic Processes
693
distributed as a set of n independent uniform (0, t) random variables. Hence, we start by simulating N (t), a Poisson random variable with mean λt (by one of the methods given in Example 11.9). Then, if N (t) = n, generate a new set of n random numbers—call them U1 , . . . , Un —and {tU1 , . . . , tUn } will represent the set of N(t) event times. If we could stop here this would be much more efficient than simulating the exponentially distributed interarrival times. However, we usually desire the event times in increasing order—for instance, for s < t, N (s) = number of Ui : tUi s and so to compute the function N (s), s t, it is best to first order the values Ui , i = 1, . . . , n before multiplying by t. However, in doing so you should not use an all-purpose sorting algorithm, such as quick sort (see Example 3.14), but rather one that takes into account that the elements to be sorted come from a uniform (0, 1) population. Such a sorting algorithm, of n uniform (0, 1) variables, is as follows: Rather than a single list to be sorted of length n we will consider n ordered, or linked, lists of random size. The value U will be put in list i if its value is between (i − 1)/n and i/n—that is, U is put in list [nU ] + 1. The individual lists are then ordered, and the total linkage of all the lists is the desired ordering. As almost all of the n lists will be of relatively small size [for instance, if n = 1000 the mean number of lists of size greater than 4 is (using the Poisson −1 approximation to the binomial) approximately equal to 1000(1 − 65 24 e ) 4] the sorting of individual lists will be quite quick, and so the running time of such an algorithm will be proportional to n (rather than to n log n as in the best all-purpose sorting algorithms). An extremely important counting process for modeling purposes is the nonhomogeneous Poisson process, which relaxes the Poisson process assumption of stationary increments. Thus it allows for the possibility that the arrival rate need not be constant but can vary with time. However, there are few analytical studies that assume a nonhomogeneous Poisson arrival process for the simple reason that such models are not usually mathematically tractable. (For example, there is no known expression for the average customer delay in the single-server exponential service distribution queueing model which assumes a nonhomogeneous arrival process.)‡ Clearly such models are strong candidates for simulation studies. 11.5.1. Simulating a Nonhomogeneous Poisson Process We now present three methods for simulating a nonhomogeneous Poisson process having intensity function λ(t), 0 t < ∞. ‡ One queueing model that assumes a nonhomogeneous Poisson arrival process and is mathemati-
cally tractable is the infinite server model.
694
11
Simulation
Method 1. Sampling a Poisson Process To simulate the first T time units of a nonhomogeneous Poisson process with intensity function λ(t), let λ be such that λ(t) λ
for all t T
Now as shown in Chapter 5, such a nonhomogeneous Poisson process can be generated by a random selection of the event times of a Poisson process having rate λ. That is, if an event of a Poisson process with rate λ that occurs at time t is counted (independently of what has transpired previously) with probability λ(t)/λ then the process of counted events is a nonhomogeneous Poisson process with intensity function λ(t), 0 t T . Hence, by simulating a Poisson process and then randomly counting its events, we can generate the desired nonhomogeneous Poisson process. We thus have the following procedure: Generate independent random variables X1 , U1 , X2 , U2 , . . . where the Xi are exponential with rate λ and the Ui are random numbers, stopping at n N = min n: Xi > T i=1
Now let, for j = 1, . . . , N − 1, ⎧ ⎨1, Ij = ⎩ 0,
if Uj λ
$
j i=1 Xi
%?
λ
otherwise
and set J = {j : Ij = 1} j Thus, the counting process having events at the set of times { i=1 Xi : j ∈ J } constitutes the desired process. The foregoing procedure, referred to as the thinning algorithm (because it “thins” the homogeneous Poisson points) will clearly be most efficient, in the sense of having the fewest number of rejected event times, when λ(t) is near λ throughout the interval. Thus, an obvious improvement is to break up the interval into subintervals and then use the procedure over each subinterval. That is, determine appropriate values k, 0 < t1 < t2 < · · · < tk < T , λ1 , . . . , λk+1 , such that λ(s) λi
when ti−1 s < ti , i = 1, . . . , k + 1 (where t0 = 0, tk+1 = T ) (11.9)
11.5.
Stochastic Processes
695
Now simulate the nonhomogeneous Poisson process over the interval (ti−1 , ti ) by generating exponential random variables with rate λi and accepting the generated event occurring at time s, s ∈ (ti−1 , ti ), with probability λ(s)/λi . Because of the memoryless property of the exponential and the fact that the rate of an exponential can be changed upon multiplication by a constant, it follows that there is no loss of efficiency in going from one subinterval to the next. In other words, if we are at t ∈ [ti−1 , ti ) and generate X, an exponential with rate λi , which is such that t + X > ti then we can use λi [X − (ti − t)]/λi+1 as the next exponential with rate λi+1 . Thus, we have the following algorithm for generating the first t time units of a nonhomogeneous Poisson process with intensity function λ(s) when the relations (11.9) are satisfied. In the algorithm, t will represent the present time and I the present interval (that is, I = i when ti−1 t < ti ). Step 1: Step 2: Step 3: Step 4:
t = 0, I = 1. Generate an exponential random variable X having rate λI . If t + X < tI , reset t = t + X, generate a random number U , and accept the event time t if U λ(t)/λI . Return to step 2. (Step reached if t + X tI ). Stop if I = k + 1. Otherwise, reset X = (X − tI + t)λI /λI +1 . Also reset t = tI and I = I + 1, and go to step 3.
Suppose now that over some subinterval (ti−1 , ti ) it follows that λi > 0 where λi ≡ infimum {λ(s): ti−1 s < ti } In such a situation, we should not use the thinning algorithm directly but rather should first simulate a Poisson process with rate λi over the desired interval and then simulate a nonhomogeneous Poisson process with the intensity function λ(s) = λ(s) − λi when s ∈ (ti−1 , ti ). (The final exponential generated for the Poisson process, which carries one beyond the desired boundary, need not be wasted but can be suitably transformed so as to be reusable.) The superposition (or, merging) of the two processes yields the desired process over the interval. The reason for doing it this way is that it saves the need to generate uniform random variables for a Poisson distributed number, with mean λi (ti − ti−1 ) of the event times. For instance, consider the case where λ(s) = 10 + s,
0<s 0 and tends to be smaller when t < 0. In certain cases the tilted distributions ft have the same parametric form as does f .
Example 11.22 If f is the exponential density with rate λ then ft (x) = Cetx λe−λx = λCe−(λ−t)x where C = 1/M(t) does not depend on x. Therefore, for t λ, ft is an exponential density with rate λ − t. If f is a Bernoulli probability mass function with parameter p, then f (x) = p x (1 − p)1−x ,
x = 0, 1
Hence, M(t) = Ef [etX ] = pet + 1 − p and so 1 (pet )x (1 − p)1−x M(t) x 1−x 1−p pet = pet + 1 − p pet + 1 − p
ft (x) =
That is, ft is the probability mass function of a Bernoulli random variable with parameter pt =
pet pet + 1 − p
We leave it as an exercise to show that if f is a normal density with parameters μ and σ 2 then ft is a normal density mean μ + σ 2 t and variance σ 2 .
11.6.
Variance Reduction Techniques
717
In certain situations the quantity of interest is the sum of the independent random variables X1 , . . . , Xn . In this case the joint density f is the product of onedimensional densities. That is, f (x1 , . . . , xn ) = f1 (x1 ) · · · fn (xn ) where fi is the density function of Xi . In this situation it is often useful to generate the Xi according to their tilted densities, with a common choice of t employed.
Example 11.23 Let X1 , . . . , Xn be independent random variables having re-
spective probability density (or mass) functions fi , for i = 1, . . . , n. Suppose we are interested in approximating the probability that their sum is at least as large as a, where a is much larger than the mean of the sum. That is, we are interested in
θ = P {S a} where S = ni=1 Xi , and where a > ni=1 E[Xi ]. Letting I {S a} equal 1 if S a and letting it be 0 otherwise, we have that θ = Ef [I {S a}] where f = (f1 , . . . , fn ). Suppose now that we simulate Xi according to the tilted mass function fi,t , i = 1, . . . , n, with the value of t, t > 0 left to be determined. The importance sampling estimator of θ would then be θˆ = I {S a}
fi (Xi ) fi,t (Xi )
Now, fi (Xi ) = Mi (t)e−tXi fi,t (Xi ) and so θˆ = I {S a}M(t)e−tS 1 where M(t) = Mi (t) is the moment generating function of S. Since t > 0 and I {S a} is equal to 0 when S < a, it follows that I {S a}e−tS e−ta and so θˆ M(t)e−ta
718
11
Simulation
To make the bound on the estimator as small as possible we thus choose t, t > 0, to minimize M(t)e−ta . In doing so, we will obtain an estimator whose value on each iteration is between 0 and mint M(t)e−ta . It can be shown that the minimizing t, call it t ∗ , is such that , n Et ∗ [S] = Et ∗ Xi = a i=1
where, in the preceding, we mean that the expected value is to be taken under the assumption that the distribution of Xi is fi,t ∗ for i = 1, . . . , n. For instance, suppose that X1 , . . . , Xn are independent Bernoulli random variables having respective parameters pi , for i = 1, . . . , n. Then, if we generate the Xi according to their tilted mass functions pi,t , i = 1, . . . , n then the importance sampling estimator of θ = P {S a} is θˆ = I {S a}e−tS
n t pi e + 1 − p i i=1
Since pi,t is the mass function of a Bernoulli random variable with parameter pi et /(pi et + 1 − pi ) it follows that , n n pi e t Xi = Et pi et + 1 − pi i=1
i=1
The value of t that makes the preceding equal to a can be numerically approximated and then utilized in the simulation. As an illustration, suppose that n = 20, pi = 0.4, and a = 16. Then Et [S] = 20
0.4et 0.4et + 0.6
Setting this equal to 16 yields, after a little algebra, ∗
et = 6 Thus, if we generate the Bernoullis using the parameter ∗
0.4et = 0.8 ∗ 0.4et + 0.6 then because ∗
M(t ∗ ) = (0.4et + 0.6)20
and e−t
∗S
= (1/6)S
11.6.
Variance Reduction Techniques
719
we see that the importance sampling estimator is θˆ = I {S 16}(1/6)S 320 It follows from the preceding that θˆ (1/6)16 320 = 81/216 = 0.001236 That is, on each iteration the value of the estimator is between 0 and 0.001236. Since, in this case, θ is the probability that a binomial random variable with parameters 20, 0.4 is at least 16, it can be explicitly computed with the result θ = 0.000317. Hence, the raw simulation estimator I , which on each iteration takes the value 0 if the sum of the Bernoullis with parameter 0.4 is less than 16 and takes the value 1 otherwise, will have variance Var(I ) = θ (1 − θ ) = 3.169 × 10−4 On the other hand, it follows from the fact that 0 θˆ 0.001236 that (see Exercise 33) Var(θˆ ) 2.9131 × 10−7
Example 11.24 Consider a single-server queue in which the times between successive customer arrivals have density function f and the service times have density g. Let Dn denote the amount of time that the nth arrival spends waiting in queue and suppose we are interested in estimating α = P {Dn a} when a is much larger than E[Dn ]. Rather than generating the successive interarrival and service times according to f and g, respectively, they should be generated according to the densities f−t and gt , where t is a positive number to be determined. Note that using these distributions as opposed to f and g will result in smaller interarrival times (since −t < 0) and larger service times. Hence, there will be a greater chance that Dn > a than if we had simulated using the densities f and g. The importance sampling estimator of α would then be αˆ = I {Dn > a}et (Sn −Yn ) [Mf (−t)Mg (t)]n where Sn is the sum of the first n interarrival times, Yn is the sum of the first n service times, and Mf and Mg are the moment generating functions of the densities f and g, respectively. The value of t used should be determined by experimenting with a variety of different choices.
720
11
11.7.
Simulation
Determining the Number of Runs
Suppose that we are going to use simulation to generate r independent and identically distributed random variables Y (1) , . . . , Y (r) having mean μ and variance σ 2 . We are then going to use Y (1) + · · · + Y (r) Y¯r = r as an estimate of μ. The precision of this estimate can be measured by its variance Var(Y¯r ) = E[(Y¯r − μ)2 ] = σ 2 /r Hence we would want to choose r, the number of necessary runs, large enough so that σ 2 /r is acceptably small. However, the difficulty is that σ 2 is not known in advance. To get around this, you should initially simulate k runs (where k 30) and then use the simulated values Y (1) , . . . , Y (k) to estimate σ 2 by the sample variance k (i) 2 Y − Y¯k /(k − 1) i=1
Based on this estimate of σ 2 the value of r that attains the desired level of precision can now be determined and an additional r − k runs can be generated.
11.8.
Coupling from the Past
Consider an irreducible Markov chain with states 1, . . . , m and transition probabilities Pi,j and suppose we want to generate the value of a random variable whose distribution is that of the stationary distribution of this Markov chain. Whereas we could approximately generate such a random variable by arbitrarily choosing an initial state, simulating the resulting Markov chain for a large fixed number of time periods, and then choosing the final state as the value of the random variable, we will now present a procedure that generates a random variable whose distribution is exactly that of the stationary distribution. If, in theory, we generated the Markov chain starting at time −∞ in any arbitrary state, then the state at time 0 would have the stationary distribution. So imagine that we do this, and suppose that a different person is to generate the next state at each of these times. Thus, if X(−n), the state at time −n, is i, then
11.8.
Coupling from the Past
721
person −n would generate a random variable that is equal to j with probability Pi,j , j = 1, . . . , m, and the value generated would be the state at time −(n − 1). Now suppose that person −1 wants to do his random variable generation early. Because he does not know what the state at time −1 will be, he generates a sequence of random variables N−1 (i), i = 1, . . . , m, where N−1 (i), the next state if X(−1) = i, is equal to j with probability Pi,j , j = 1, . . . , m. If it results that X(−1) = i, then person −1 would report that the state at time 0 is S−1 (i) = N−1 (i),
i = 1, . . . , m
(That is, S−1 (i) is the simulated state at time 0 when the simulated state at time −1 is i.) Now suppose that person −2, hearing that person −1 is doing his simulation early, decides to do the same thing. She generates a sequence of random variables N−2 (i), i = 1, . . . , m, where N−2 (i) is equal to j with probability Pi,j , j = 1, . . . , m. Consequently, if it is reported to her that X(−2) = i, then she will report that X(−1) = N−2 (i). Combining this with the early generation of person −1 shows that if X(−2) = i, then the simulated state at time 0 is S−2 (i) = S−1 (N−2 (i)),
i = 1, . . . , m
Continuing in the preceding manner, suppose that person −3 generates a sequence of random variables N−3 (i), i = 1, . . . , m, where N−3 (i) is to be the generated value of the next state when X(−3) = i. Consequently, if X(−3) = i then the simulated state at time 0 would be S−3 (i) = S−2 (N−3 (i)),
i = 1, . . . , m
Now suppose we continue the preceding, and so obtain the simulated functions S−1 (i), S−2 (i), S−3 (i), . . . ,
i = 1, . . . , m
Going backward in time in this manner, we will at some time, say −r, have a simulated function S−r (i) that is a constant function. That is, for some state j , S−r (i) will equal j for all states i = 1, . . . , m. But this means that no matter what the simulated values from time −∞ to −r, we can be certain that the simulated value at time 0 is j . Consequently, j can be taken as the value of a generated random variable whose distribution is exactly that of the stationary distribution of the Markov chain.
722
11
Simulation
Example 11.25 Consider a Markov chain with states 1, 2, 3 and suppose that simulation yielded the values ⎧ ⎨3, N−1 (i) = 2, ⎩ 2,
if i = 1 if i = 2 if i = 3
⎧ ⎨1, N−2 (i) = 3, ⎩ 1,
if i = 1 if i = 2 if i = 3
⎧ ⎨3, S−2 (i) = 2, ⎩ 3,
if i = 1 if i = 2 if i = 3
⎧ ⎨3, N−3 (i) = 1, ⎩ 1,
if i = 1 if i = 2 if i = 3
⎧ ⎨3, S−3 (i) = 3, ⎩ 3,
if i = 1 if i = 2 if i = 3
and
Then
If
then
Therefore, no matter what the state is at time −3, the state at time 0 will be 3.
Remark The procedure developed in this section for generating a random variable whose distribution is the stationary distribution of the Markov chain is called coupling from the past.
Exercises
723
Exercises *1. Suppose it is relatively easy to simulate from the distributions Fi , i = 1, 2, . . . , n. If n is small, how can we simulate from F (x) =
n
Pi Fi (x),
Pi 0,
Pi = 1?
i
i=1
Give a method for simulating from ⎧ 1 − e−2x + 2x ⎪ ⎪ , ⎨ 3 F (x) = −2x ⎪ ⎪ ⎩3 − e , 3 2.
0<x X3 |X1 > X2 } 2
× P {X1 − X2 − X3 > X4 |X1 > X2 + X3 } · · · × P {X1 − X2 · · · − Xn−1 > Xn |X1 > X2 + · · · + Xn−1 } n−1 1 = by lack of memory 2 Hence, P M>
n i=1
n Xi − M = P Xi > Xj = i=1
j =i
n 2n−1
Chapter 6 2. Let NA (t) be the member of organisms in state A and let NB (t) be the number of organisms in state B. Then {NA (t), NB (t)} is a continuous-Markov chain with ν{n,m} = αn + βm P{n,m},{n−1,m+1} =
αn αn + βm
P{n,m},{n+2,m−1} =
βm αn + βm
752
Appendix Solutions to Starred Exercises
4. Let N (t) denote the number of customers in the station at time t. Then {N (t)} is a birth and death process with λn = λαn , 7.
μn = μ
(a) Yes! (b) For n = (n1 , . . . , ni , ni+1 , . . . , nk−1 ) let Si (n) = (n1 , . . . , ni − 1, ni+1 + 1, . . . , nk−1 ),
i = 1, . . . , k − 2
Sk−1 (n) = (n1 , . . . , ni , ni+1 , . . . , nk−1 − 1), S0 (n) = (n1 + 1, . . . , ni , ni+1 , . . . , nk−1 ). Then qn,Si (n) = ni μ,
i = 1, . . . , k − 1
qn,S0 (n) = λ 11.
(b) Follows from the hint about using the lack of memory property and the fact that εi , the minimum of j − (i − 1) independent exponentials with rate λ, is exponential with rate (j − i − 1)λ. (c) From parts (a) and (b) P {T1 + · · · + Tj t} = P
max Xi t = (1 − e−λt )j
1i j
(d) With all probabilities conditional on X(0) = 1 P1j (t) = P {X(t) = j } = P {X(t) j } − P {X(t) j + 1} = P {T1 + · · · + Tj t} − P {T1 + · · · + Tj +1 t} (e) The sum of i independent geometrics, each having parameter p = e−λt , is a negative binomial with parameters i, p. The result follows since starting with an initial population of i is equivalent to having i independent Yule processes, each starting with a single individual. 16. Let the state be 2: an acceptable molecule is attached 0: no molecule attached 1: an unacceptable molecule is attached.
Chapter 6
753
Then this is a birth and death process with balance equations μ1 P1 = λ(1 − α)P0 μ2 P2 = λαP0 Since
2
0 Pi
= 1, we get
/ . μ2 1 − α μ2 −1 λαμ1 + = P2 = 1 + λα α μ1 λαμ1 + μ1 μ2 + λ(1 − α)μ2 where P2 is the percentage of time the site is occupied by an acceptable molecule. The percentage of time the site is occupied by an unacceptable molecule is P1 =
1 − α μ2 λ(1 − α)μ2 P1 = α μ1 λαμ1 + μ1 μ2 + λ(1 − α)μ2
19. There are four states. Let state 0 mean that no machines are down, state 1 that machine 1 is down and 2 is up, state 2 that machine 1 is up and 2 is down, and 3 that both machines are down. The balance equations are as follows: (λ1 + λ2 )P0 = μ1 P1 + μ2 P2 (μ1 + λ2 )P1 = λ1 P0 (λ1 + μ2 )P2 = λ2 P0 + μ1 P3 μ1 P3 = λ2 P1 + λ1 P2 P0 + P1 + P2 + P3 = 1 The equations are easily solved and the proportion of time machine 2 is down is P2 + P3 . 24. We will let the state be the number of taxis waiting. Then, we get a birth and death process with λn = 1, μn = 2. This is an M/M/1. Therefore: 1 1 = = 1. (a) Average number of taxis waiting = μ−λ 2−1 (b) The proportion of arriving customers that gets taxis is the proportion of arriving customers that find at least one taxi waiting. The rate of arrival of such customers is 2(1 − P0 ). The proportion of such arrivals is therefore λ λ 2(1 − P0 ) 1 = = = 1 − P0 = 1 − 1 − 2 μ μ 2
754
Appendix Solutions to Starred Exercises y
y
28. Let Pijx , vix denote the parameters of the X(t) and Pij , vi of the Y (t) y process; and let the limiting probabilities be Pix , Pi , respectively. By independence we have that for the Markov chain {X(t), Y (t)} its parameters are y
v(i,l) = vix + vl P(i,l)(j,l) =
vix
x y Pij
vix + vl y
P(i,l)(i,k) =
vl
y y Plk
vix + vl
and y
lim P {(X(t), Y (t)) = (i, j )} = Pix Pj
t→∞
Hence, we need to show that y
y
Pix Pl vix Pijx = Pjx Pl vjx Pjxi [That is, rate from (i, l) to (j, l) equals the rate from (j, l) to (i, l).] But this follows from the fact that the rate from i to j in X(t) equals the rate from j to i; that is, Pix vix Pijx = Pjx vjx Pjxi The analysis is similar in looking at pairs (i, l) and (i, k). 33. Suppose first that the waiting room is of infinite size. Let Xi (t) denote the number of customers at server i, i = 1, 2. Then since each of the M/M/1 processes {X1 (t)} is time reversible, it follows from Exercise 28 that the vector process {(X1 (t), (X(t)), t 0} is a time reversible Markov chain. Now the process of interest is just the truncation of this vector process to the set of states A where A = {(0, m): m 4} ∪ {(n, 0): n 4} ∪ {(n, m): nm > 0, n + m 5} Hence, the probability that there are n with server 1 and m with server 2 is m n λ1 λ2 λ2 λ1 Pn,m = k 1− 1− μ1 μ1 μ2 μ2 n m λ2 λ1 =C , (n, m) ∈ A μ1 μ2 The constant C is determined from where the sum is over all (n, m) in A.
Pn,m = 1
Chapter 7
42.
755
(a) The matrix P∗ can be written as P∗ = I + R/v and so Pij∗n can be obtained by taking the i, j element of (I + R/v)n , which gives the result when v = n/t. (b) Uniformization shows that Pij (t) = E[Pij∗N ], where N is independent of the Markov chain with transition probabilities Pij∗ and is Poisson distributed with mean vt. Since a Poisson random variable with mean vt has standard deviation (vt)1/2 , it follows that for large values of vt it should be near vt. (For instance, a Poisson random variable with mean 106 has standard deviation 103 and thus will, with high probability, be within 3000 of 106 .) Hence, since for fixed i and j , Pij∗m should not vary much for values of m about vt where vt is large, it follows that, for large vt, E[Pij∗N ] ≈ Pij∗n
where n = vt
Chapter 7 3. By the one-to-one correspondence of m(t) and F , it follows that {N (t), t 0} is a Poisson process with rate 12 . Hence, P {N (5) = 0} = e−5/2 6.
(a) Consider a Poisson process having rate λ and say that an event of the renewal process occurs whenever one of the events numbered r, 2r, 3r, . . . of the Poisson process occurs. Then P {N (t) n} = P {nr or more Poisson events by t} =
∞
e−λt (λt)i /i!
i=nr
(b)
E[N (t)] =
∞
P {N (t) n} =
n=1
=
∞ [i/r] i=r n=1
∞ ∞
e−λt (λt)i /i!
n=1 i=nr
e−λt (λt)i /i! =
∞ [i/r]e−λt (λt)i /i! i=r
756 8.
Appendix Solutions to Starred Exercises
(a) The number of replaced machines by time t constitutes a renewal process. The time between replacements equals T , if lifetime of new machine is T ; x, if lifetime of new machine is x, x < T . Hence, T xf (x) dx + T [1 − F (T )] E[time between replacements] = 0
and the result follows by Proposition 3.1. (b) The number of machines that have failed in use by time t constitutes a renewal process. The mean time between in-use failures, E[F ], can be calculated by conditioning on the lifetime of the initial machine as E[F ] = E[E[F | lifetime of initial machine]]. Now x, if x T E[F | lifetime of machine is x] = T + E[F ], if x > T Hence, T xf (x) dx + (T + E[F ])[1 − F (T )]
E[F ] = 0
or T E[F ] =
0
xf (x) dx + T [1 − F (T )] F (T )
and the result follows from Proposition 3.1. 18. We can imagine that a renewal corresponds to a machine failure, and each time a new machine is put in use its life distribution will be exponential with rate μ1 with probability p, and exponential with rate μ2 otherwise. Hence, if our state is the index of the exponential life distribution of the machine presently in use, then this is a two-state continuous-time Markov chain with intensity rates q1,2 = μ1 (1 − p),
q2,1 = μ2 p
Hence, P11 (t) =
μ1 (1 − p) exp{−[μ1 (1 − p) + μ2 p]t} μ1 (1 − p) + μ2 p μ2 p + μ1 (1 − p) + μ2 p
with similar expressions for the other transition probabilities [P12 (t) = 1 − P11 (t), and P22 (t) is the same with μ2 p and μ1 (1 − p) switching places]. Condition-
Chapter 7
757
ing on the initial machine now gives E[Y (t)] = pE[Y (t)|X(0) = 1] + (1 − p)E[Y (t)|X(0) = 2] / . / . P21 (t) P22 (t) P11 (t) P12 (t) + (1 − p) + + =p μ1 μ2 μ1 μ2 Finally, we can obtain m(t) from μ[m(t) + 1] = t + E[Y (t)] where μ = p/μ1 + (1 − p)/μ2 is the mean interarrival time. 22. Cost of a cycle = C1 + C2 I − R(T )(1 − I ) 1, if X < T I= where X = life of car 0, if X T Hence, E[cost of a cycle] = C1 + C2 H (T ) − R(T )[1 − H (T )] Also,
E[time of cycle] =
E[time | X = x]h(x) dx
=
T
xh(x) dx + T [1 − H (T )]
0
Thus the average cost per unit time is given by C1 + C2 H (T ) − R(T )[1 − H (T )] T 0 xh(x) dx + T [1 − H (T )] 30.
A(t) t − SN (t) = t t SN (t) =1− t SN (t) N (t) =1− N (t) t
The result follows since SN (t) /N(t) → μ (by the strong law of large numbers) and N (t)/t → 1/μ.
758 35.
Appendix Solutions to Starred Exercises
(a) We can view this as an M/G/∞ system where a satellite launching corresponds to an arrival and F is the service distribution. Hence, P {X(t) = k} = e−λ(t) [λ(t)]k /k! t where λ(t) = λ 0 (1 − F (s)) ds. (b) By viewing the system as an alternating renewal process that is on when there is at least one satellite orbiting, we obtain lim P {X(t) = 0} =
1/λ 1/λ + E[T ]
where T , the on time in a cycle, is the quantity of interest. From part (a) lim P {X(t) = 0} = e−λμ where μ =
∞ 0
(1 − F (s)) ds is the mean time that a satellite orbits. Hence, e−λμ =
1/λ 1/λ + E[T ]
so E[T ] = 42. (a) Fe (x) =
1 μ
x
1 − e−λμ λe−λμ
e−y/μ dy = 1 − e−x/μ .
0 x
1 x dy = , 0 x c. c 0 c (c) You will receive a ticket if, starting when you park, an official appears within one hour. From Example 7.23 the time until the official appears has the distribution Fe which, by part (a), is the uniform distribution on (0,2). Thus, the probability is equal to 12 . (b) Fe (x) =
49. Think of each interarrival time as consisting of n independent phases—each of which is exponentially distributed with rate λ—and consider the semi-Markov process whose state at any time is the phase of the present interarrival time. Hence, this semi-Markov process goes from state 1 to 2 to 3 . . . to n to 1, and so on. Also the time spent in each state has the same distribution. Thus, clearly the limiting probability of this semi-Markov chain is Pi = 1/n, i = 1, . . . , n. To compute lim P {Y (t) < x}, we condition on the phase at time t and note that if it is n − i + 1, which will be the case with probability 1/n, then the time until a renewal occurs will be sum of i exponential phases, which will thus have a gamma distribution with parameters i and λ.
Chapter 8
759
Chapter 8 2. This problem can be modeled by an M/M/1 queue in which λ = 6, μ = 8. The average cost rate will be $10 per hour per machine × average number of broken machines The average number of broken machines is just L, which can be computed from Equation (3.2): L=
λ μ−λ
=
6 =3 2
Hence, the average cost rate = $30/hour. 7. To compute W for the M/M/2, set up balance equations as follows: λP0 = μP1
(each server has rate μ)
(λ + μ)P1 = λP0 + 2μP2 (λ + 2μ)Pn = λPn−1 + 2μPn+1 ,
n2
These have solutions Pn = ρ n /2n−1 P0 where ρ = λ/μ. The boundary condition ∞ n=0 Pn = 1 implies P0 =
1 − ρ/2 (2 − ρ) = 1 + ρ/2 (2 + ρ)
Now we have Pn , so we can compute L, and hence W from L = λW : L=
∞
nPn = ρP0
n=0
= 2P0
n=0
∞ n=0
=2
∞ $ ρ %n−1 n 2
n
$ ρ %n 2
(2 − ρ) (ρ/2) (2 + ρ) (1 − ρ/2)2
=
4ρ (2 + ρ)(2 − ρ)
=
4μλ (2μ + λ)(2μ − λ)
[See derivation of Eq. (3.2).]
760
Appendix Solutions to Starred Exercises
From L = λW we have W = W (M/M/2) =
4μ (2μ + λ)(2μ − λ)
The M/M/1 queue with service rate 2μ has W (M/M/1) =
1 2μ − λ
from Equation (3.3). We assume that in the M/M/1 queue, 2μ > λ so that the queue is stable. But then 4μ > 2μ + λ, or 4μ/(2μ + λ) > 1, which implies W (M/M/2) > W (M/M/1). The intuitive explanation is that if one finds the queue empty in the M/M/2 case, it would do no good to have two servers. 1 = W (M/M/1) and One would be better off with one faster server. Now let WQ Q 2 = W (M/M/2). Then, WQ Q 1 = W (M/M/1) − 1/2μ WQ 2 WQ = W (M/M/2) − 1/μ
So, 1 WQ =
λ 2μ(2μ − λ)
(3.3)
and 2 WQ =
λ2 μ(2μ − λ)(2μ + λ)
Then, 1 2 > WQ ⇔ WQ
λ 1 > 2 (2μ + λ)
λ < 2μ 2 < W 1 whenever Since we assume λ < 2μ for stability in the M/M/1 case, WQ Q this comparison is possible, that is, whenever λ < 2μ. 13. (a) λP0 = μP1 (λ + μ)P1 = λP0 + 2μP2
(λ + 2μ)Pn = λPn−1 + 2μPn+1 ,
n2
These are the same balance equations as for the M/M/2 queue and have solution 2μ − λ λn P0 = , Pn = n−1 n P0 2μ + λ 2 μ
Chapter 8
761
(b) The system goes from 0 to 1 at rate λP0 =
λ(2μ − λ) (2μ + λ)
The system goes from 2 to 1 at rate 2μP2 =
λ2 (2μ − λ) μ (2μ + λ)
(c) Introduce a new state cl to indicate that the stock clerk is checking by himself. The balance equation for Pcl is (λ + μ)Pcl = μP2 Hence Pcl =
(2μ − λ) μ λ2 P2 = λ+μ 2μ(λ + μ) (2μ + λ)
Finally, the proportion of time the stock clerk is checking is Pcl +
∞
Pn = Pcl +
n=2
21.
2λ2 μ(2μ − λ)
(a) λ1 P10 . (b) λ2 (P0 + P10 ). (c) λ1 P10 /[λ1 P10 + λ2 (P0 + P10 )]. (d) This is equal to the fraction of server 2’s customers that are type 1 multiplied by the proportion of time server 2 is busy. (This is true since the amount of time server 2 spends with a customer does not depend on which type of customer it is.) By (c) the answer is thus (P01 + P11 )λ1 P10 λ1 P10 + λ2 (P0 + P10 )
24. The states are now n, n 0, and n , n 1 where the state is n when there are n in the system and no breakdown, and it is n when there are n in the system and a breakdown is in progress. The balance equations are λP0 = μP1 (λ + μ + α)Pn = λPn−1 + μPn+1 + βPn , (β + λ)P1 = αP1
n1
762
Appendix Solutions to Starred Exercises
(β + λ)Pn = αPn + λP(n−1) , ∞
Pn +
n=0
∞
n2
Pn = 1
n=1
In terms of the solution to the preceding, L=
∞
n(Pn + Pn )
n=1
and so W=
L L = λa λ
28. If a customer leaves the system busy, the time until the next departure is the time of a service. If a customer leaves the system empty, the time until the next departure is the time until an arrival plus the time of a service. Using moment generating functions we get λ E{esD | system left busy} μ λ E{esD | system left empty} + 1− μ μ λ λ + 1− E{es(X+Y ) } = μ μ−s μ
E{esD } =
where X has the distribution of interarrival times, Y has the distribution of service times, and X and Y are independent. Then E[es(X+Y ) ] = E[esX esY ] = E[esX ]E[esY ] by independence μ λ = λ−s μ−s So, E{esD } = =
μ λ λ μ λ + 1− μ μ−s μ λ−s μ−s λ (λ − s)
By the uniqueness of generating functions, it follows that D has an exponential distribution with parameter λ.
Chapter 8
763
36. The distributions of the queue size and busy period are the same for all three disciplines; that of the waiting time is different. However, the means are identical. This can be seen by using W = L/λ, since L is the same for all. The smallest variance in the waiting time occurs under first-come, first-served and the largest under last-come, first-served. 39. (a) a0 = P0 due to Poisson arrivals. Assuming that each customer pays 1 per unit time while in service the cost identity of Equation (2.1) states that average number in service = λE[S] or 1 − P0 = λE[S] (b) Since a0 is the proportion of arrivals that have service distribution G1 and 1 − a0 the proportion having service distribution G2 , the result follows. (c) We have P0 =
E[I ] E[I ] + E[B]
and E[I ] = 1/λ and thus, E[B] = =
1 − P0 λP0 E[S] 1 − λE[S]
Now from parts (a) and (b) we have E[S] = (1 − λE[S])E[S1 ] + λE[S]E[S2 ] or E[S] =
E[S1 ] 1 + λE[S1 ] + λE[S2 ]
Substituting into E[B] = E[S]/(1 − λE[S]) now yields the result. (d) a0 = 1/E[C], implying that E[C] =
E[S1 ] + 1/λ − E[S2 ] 1/λ − E[S2 ]
764
Appendix Solutions to Starred Exercises
45. By regarding any breakdowns that occur during a service as being part of that service, we see that this is an M/G/1 model. We need to calculate the first two moments of a service time. Now the time of a service is the time T until something happens (either a service completion or a breakdown) plus any additional time A. Thus, E[S] = E[T + A] = E[T ] + E[A] To compute E[A], we condition upon whether the happening is a service or a breakdown. This gives μ α + E[A | breakdown] μ+α μ+α α = E[A | breakdown] μ+α 1 α = + E[S] β μ+α
E[A] = E[A | service]
Since E[T ] = 1/(α + μ) we obtain that E[S] =
1 α 1 + + E[S] α+μ β μ+α
or E[S] =
α 1 + μ μβ
We also need E[S 2 ], which is obtained as follows. E[S 2 ] = E[(T + A)2 ] = E[T 2 ] + 2E[AT ] + E[A2 ] = E[T 2 ] + 2E[A]E[T ] + E[A2 ] The independence of A and T follows because the time of the first happening is independent of whether the happening was a service or a breakdown. Now, E[A2 ] = E[A2 | breakdown]
α μ+α
Chapter 8
765
α E[(downtime + S ∗ )2 ] μ+α < α ; = E[down2 ] + 2E[down]E[S] + E[S 2 ] μ+α . / 2 α α 2 1 + + E[S 2 ] = + μ + α β 2 β μ μβ =
Hence, . / α α 1 α 2 + + E[S ] = +2 β(μ + α) μ + α μ μβ (μ + β)2 . / 2 α α 2 1 + + E[S 2 ] + + μ + α β 2 β μ μβ 2
Now solve for E[S 2 ]. The desired answer is WQ =
λE[S 2 ] 2(1 − λE[S])
In the preceding, S ∗ is the additional service needed after the breakdown is over and S ∗ has the same distribution as S. The preceding also uses the fact that the expected square of an exponential is twice the square of its mean. Another way of calculating the moments of S is to use the representation S=
N (Ti + Bi ) + TN +1 i=1
where N is the number of breakdowns while a customer is in service, Ti is the time starting when service commences for the ith time until a happening occurs, and Bi is the length of the ith breakdown. We now use the fact that, given N , all of the random variables in the representation are independent exponentials with the Ti having rate μ + α and the Bi having rate β. This yields E[S|N ] = Var(S|N ) =
N +1 N + μ+α β N +1 N + 2 2 (μ + α) β
Therefore, since 1 + N is geometric with mean (μ + α)/μ [and variance α(α + μ)/μ2 ] we obtain E[S] =
α 1 + μ μβ
766
Appendix Solutions to Starred Exercises
and, using the conditional variance formula, .
1 1 Var(S) = + μ+α β
/2
α(α + μ) 1 α + + μ(μ + α) μβ 2 μ2
52. Sn is the service time of the nth customer; Tn is the time between the arrival of the nth and (n + 1)st customer.
Chapter 9 4. (a) φ(x) = x1 max(x2 , x3 , x4 )x5 . (b) φ(x) = x1 max(x2 x4 , x3 x5 )x6 . (c) φ(x) = max(x1 , x2 x3 )x4 . 6. A minimal cut set has to contain at least one component of each minimal path set. There are six minimal cut sets: {1, 5}, {1, 6}, {2, 5}, {2, 3, 6}, {3, 4, 6}, {4, 5}. 12. The minimal path sets are {1, 4}, {1, 5}, {2, 4}, {2, 5}, {3, 4}, {3, 5}. With qi = 1 − pi , the reliability function is r(p) = P {either of 1, 2, or 3 works}P {either of 4 or 5 works} = (1 − q1 q2 q3 )(1 − q4 q5 ) 17. E[N 2 ] = E[N 2 |N > 0]P {N > 0} (E[N |N > 0])2 P {N > 0},
since E[X 2 ] (E[X])2
Thus, E[N 2 ]P {N > 0} (E[N |N > 0]P [N > 0])2 = (E[N ])2 Let N denote the number of minimal path sets having all of its components functioning. Then r(p) = P {N > 0}. Similarly, if we define N as the number of minimal cut sets having all of its components failed, then 1 − r(p) = P {N > 0}. In both cases we can compute expressions for E[N ] and E[N 2 ] by writing N as the sum of indicator (i.e., Bernoulli) random variables. Then we can use the inequality to derive bounds on r(p). 22. (a) F¯t (a) = P {X > t + a | X > t} =
P [X > t + a} F¯ (t + a) = P {X > t} F¯ (t)
Chapter 9
767
(b) Suppose λ(t) is increasing. Recall that F¯ (t) = e−
t 0
λ(s) ds
Hence, t+a F¯ (t + a) λ(s) ds = exp − F¯ (t) t which decreases in t since λ(t) is increasing. To go the other way, suppose F¯ (t + a)/F¯ (t) decreases in t. Now when a is small F¯ (t + a) ≈ e−aλ(t) F¯ (t) Hence, e−aλ(t) must decrease in t and thus λ(t) increases. 25. For x ξ , 1 − p = F¯ (ξ ) = F¯ (x(ξ/x)) [F¯ (x)]ξ/x since IFRA. Hence, F¯ (x) (1 − p)x/ξ = e−θx . For x ξ , F¯ (x) = F¯ (ξ(x/ξ )) [F¯ (ξ )]x/ξ since IFRA. Hence, F¯ (x) (1 − p)x/ξ = e−θx . 30.
r(p) = p1 p2 p3 + p1 p2 p4 + p1 p3 p4 + p2 p3 p4 − 3p1 p2 p3 p4 ⎧ 2 2 ⎪ ⎨2(1 − t) (1 − t/2) + 2(1 − t)(1 − t/2) r(1 − F (t)) = −3(1 − t)2 (1 − t/2)2 , 0t 1 ⎪ ⎩ 0, 1t 2 1 " 2(1 − t)2 (1 − t/2) + 2(1 − t)(1 − t/2)2 E[lifetime] = 0
# − 3(1 − t)2 (1 − t/2)2 dt =
31 60
768
Appendix Solutions to Starred Exercises
Chapter 10 1. B(s) + B(t) = 2B(s) + B(t) − B(s). Now 2B(s) is normal with mean 0 and variance 4s and B(t) − B(s) is normal with mean 0 and variance t − s. Because B(s) and B(t) − B(s) are independent, it follows that B(s) + B(t) is normal with mean 0 and variance 4s + t − s = 3s + t. 3.
E[B(t1 )B(t2 )B(t3 )] = E[E[B(t1 )B(t2 )B(t3 )|B(t1 ), B(t2 )]] = E[B(t1 )B(t2 )E[B(t3 ) | B(t1 ), B(t2 )]] = E[B(t1 )B(t2 )B(t2 )] = E[E[B(t1 )B 2 (t2 ) | B(t1 )]] = E[B(t1 )E[B 2 (t2 ) | B(t1 )]] = E[B(t1 ){(t2 − t1 ) + B 2 (t1 )}]
(∗)
= E[B 3 (t1 )] + (t2 − t1 )E[B(t1 )] =0 where the equality (∗) follows since given B(t1 ), B(t2 ) is normal with mean B(t1 ) and variance t2 − t1 . Also, E[B 3 (t)] = 0 since B(t) is normal with mean 0. 5.
P {T1 < T−1 < T2 } = P {hit 1 before −1 before 2} = P {hit 1 before −1} × P {hit −1 before 2 | hit 1 before −1} = 12 P {down 2 before up 1} =
11 1 = 23 6
The next to last equality follows by looking at the Brownian motion when it first hits 1. 10.
(a) Writing X(t) = X(s)+X(t)−X(s) and using independent increments, we see that given X(s) = c, X(t) is distributed as c + X(t) − X(s). By stationary increments this has the same distribution as c + X(t − s), and is thus normal with mean c + μ(t − s) and variance (t − s)σ 2 . (b) Use the representation X(t) = σ B(t) + μt, where {B(t)} is standard Brownian motion. Using Equation (1.4), but reversing s and t, we see that the conditional distribution of B(t) given that B(s) = (c − μs)/σ is normal with mean t (c − μs)/(σ s) and variance t (s − t)/s. Thus, the conditional
Chapter 10
769
distribution of X(t) given that X(s) = c, s > t, is normal with mean . / t (c − μs) (c − μs)t σ + μt = + μt σs s and variance σ 2 t (s − t) s 19. Since knowing the value of Y (t) is equivalent to knowing B(t), we have E[Y (t) | Y (u), 0 u s] = e−c
2 t/2
E[ecB(t) | B(u), 0 u s]
= e−c
2 t/2
E[ecB(t) | B(s)]
Now, given B(s), the conditional distribution of B(t) is normal with mean B(s) and variance t − s. Using the formula for the moment generating function of a normal random variable we see that e−c
2 t/2
E[ecB(t) |B(s)] = e−c
2 t/2
= e−c
2 s/2
ecB(s)+(t−s)c
2 /2
ecB(s)
= Y (s) Thus {Y (t)} is a Martingale. E[Y (t)] = E[Y (0)] = 1 20. By the Martingale stopping theorem E[B(T )] = E[B(0)] = 0 However, B(T ) = 2 − 4T and so 2 − 4E[T ] = 0, or E[T ] = 12 . 24. It follows from the Martingale stopping theorem and the result of Exercise 18 that E[B 2 (T ) − T ] = 0 where T is the stopping time given in this problem and B(t) =
X(t) − μt σ
770
Appendix Solutions to Starred Exercises
Therefore, . E
/ (X(T ) − μT )2 − T =0 σ2
However, X(T ) = x and so the preceding gives that E[(x − μT )2 ] = σ 2 E[T ] But, from Exercise 21, E[T ] = x/μ and so the preceding is equivalent to Var(μT ) = σ 2
x μ
or Var(T ) = σ 2
x μ3
27. E[X(a 2 t)/a] = (1/a)E[X(a 2 t)] = 0. For s < t, 1 Cov Y (s), Y (t) = 2 Cov X(a 2 s), X(a 2 t) a 1 = 2 a2s = s a Because {Y (t)} is clearly Gaussian, the result follows. 30. (a) Starting at any time t the continuation of the Poisson process remains a Poisson process with rate λ. ∞
(b) E[Y (t)Y (t + s)] =
E[Y (t)Y (t + s)|Y (t) = y]λe−λy dy
0 s
= 0
yE[Y (t + s)|Y (t) = y]λe−λy dy
∞
+
y(y − s)λe−λy dy
s
s
= 0
1 y λe−λy dy + λ
∞
y(y − s)λe−λy dy
s
where the preceding used that E[Y (t)Y (t + s)|Y (t) = y] =
yE(Y (t + s)) = y(y − s),
y , λ
if y < s if y > s
Hence, Cov(Y (t), Y (t + s)) = 0
s
ye−λy dy +
s
∞
y(y − s)λe−λy dy −
1 λ2
Chapter 11
771
Chapter 11 1.
i (a) Let U be a random number. If i−1 j =1 Pj then simulate j =1 Pj < U i−1 from Fi . (In the preceding j =1 Pj ≡ 0 when i = 1.) (b) Note that 1 2 F (x) = F1 (x) + F2 (x) 3 3 where 0<x x} = P {X1 > m−1 (x)} = P {N (m−1 (x)) = 0} = e−m(m
−1 (x))
= e−x (b) P {m(Xi ) − m(Xi−1 ) > x|m(X1 ), . . . , m(Xi−1 ) − m(Xi−2 )} = P {m(Xi ) − m(Xi−1 ) > x|X1 , . . . , Xi−1 } = P {m(Xi ) − m(Xi−1 ) > x|Xi−1 } = P {m(Xi ) − m(Xi−1 ) > x|m(Xi−1 )} Now, P {m(Xi ) − m(Xi−1 ) > x|Xi−1 = y} Xi =P λ(t)dt > x|Xi−1 = y y
= P {Xi > c|Xi−1 = y}
where
c
λ(t) dt = x
y
= P {N (c) − N (y) = 0 | Xi−1 = y} = P {N (c) − N (y) = 0} c λ(t) dt = exp − y
=e 32.
773
−x
Var[(X + Y )/2] = 14 [Var(X) + Var(Y ) + 2Cov(X, Y )] =
Var(X) + Cov(X, Y ) 2
Now it is always true that √
Cov(V , W ) 1 Var(V )Var(W )
and so when X and Y have the same distribution Cov(X, Y ) Var(X).
This page intentionally left blank
Index
Bonus Malus automobile insurance system, 216–217 system, 188–189 Boole’s inequality, 16 Bose–Einstein statistics, 147–151 Box–Muller approach, 678 Branching processes, 233–236 Bridge Brownian, 647 structure, 587 system defined, 576 Brown, Robert, 626 Brownian bridge, 647 Brownian motion integrated, 648 standard, 627 variations on, 631–632 Brownian motion and stationary processes, 625–662 Gaussian processes, 646–649 harmonic analysis of weakly stationary processes, 654–657 hitting times and gambler’s ruin problem, 629–630 maximum variable and gambler’s ruin problem, 629–630 pricing stock options, 632–644 stationary and weakly stationary processes, 649–654 variations on Brownian motion, 631–632 white noise, 644–646 Busy period, 338, 530–531
Absorbing state of a Markov chain, 187 Accessibility of states in a Markov chain, 193–194 Age of the renewal process, 447 Algorithms, analyzing probabilistic, 224–230 Alias method in simulation, 688–692 Aloha protocol, 201–204 Alternating renewal process, 445–446 Aperiodic state of a Markov chain, 204 Arbitrage defined, 635 Arbitrage theorem, 635–638 Balance equations, 386, 500 Ballot problem, 128–130, 175 Bayes’ formula, 12–15 Bernoulli random variables, 28–29, 39, 50, 56–57, 286, 716 independent, 123–124 Best prize problem, 124–126 Beta distribution, 681–682 Beta random variable, 63 relation to gamma, 62–64 simulation of, 671, 681–682 Binomial random variables, 29–31, 39, 50 simulating, 686 sums of independent, 69–70 variance of, 55–56 Binomials, negative, 164–165 Birth and death model, 365 Birth and death processes, 368–375, 384 ergodic, 394 Bivariate exponential distribution, 363 Bivariate normal distribution, 167 Black–Scholes option pricing formula, 638– 644 Bonferroni’s inequality, 16
Cayley’s theorem, 590 Central limit theorem, 79, 82 Central limit theorem for renewal processes, 432–433
775
776
Index
Chapman–Kolmogorov equations, 189–193, 380 Chebyshev’s inequality, 77–78 Chi-squared distribution, 681 Chi-squared random variable with degree of freedom, 75 Class of Markov chain, 193 Closed class of a Markov chain, 266 Communications with Markov chain, 194 Complement of event, 3 Compound Poisson process, 337–342 Compound Poisson random variable, 120 Compound random variables defined, 108 identities, 159 variances of, 119–120 Compound random variables, identity for, 158–165 binomial compounding distribution, 163 compounding distribution related to negative binomials, 164–165 Poisson compounding distribution, 161– 162 Conditional expectation, 98, 102–103, 105– 106 Conditional expectation, conditional probability and, 97–184 computing expectations by conditioning, 105–120 computing probabilities by conditioning, 120–137 continuous case, 102–105 discrete case, 97–102 identity for compound random variables, 158–165 miscellaneous applications, 137–158 Conditional or mixed Poisson processes, 343–345 Conditional probability, 7–10, 136–137 Conditional probability and conditional expectation computing expectations by conditioning, 105–120 computing probabilities by conditioning, 120–137 continuous case, 102–105 discrete case, 97–102 identity for compound random variables, 158–165 miscellaneous applications, 137–158 Conditional probability density function, 102 Conditional variance, 118 Conditional variance formula, 119 Conditioning, computing probabilities by, 120–137
Connected graph defined, 145 Convolution, 58 Correlation, 167 Counting processes, 302–303 Coupling from past, 720–722 Coupon collecting problem, 313–316 Covariance, properties of, 54 Covariance and variance of sums of random variables, 53–61 Coxian random variable defined, 301 Craps, 16 Cumulative distribution function (cdf), 26, 28 Cut vector defined, 577 Decreasing failure rate (DFR), 596–597, 599–600 Delayed renewal process, 461–462, 481 Dependent events, 10 Dirichlet distributions defined, 149 Discrete distributions simulating from, 685– 692 Distributions, simulating from discrete, 685– 692 Doubly stochastic, 266 e, 131–132 Ehrenfest, P. and T., 239 Ehrenfest urn model, 239, 273 Einstein, A., 626 Elementary renewal theorem, 424–425 Ergodic birth and death process, 394 Ergodic define, 204 Ergodic Markov chain, 242 Erlang’s loss system, 553–554 Event, 2 Excess life of renewal process, 441, 448 Expectation, conditional, 97–184, 103 Expectation of sum of random number of random variables, 107 Expectations, computing by conditioning, 105–120 computing variances by conditioning, 117–120 Expected system lifetime, 604–610 Expected time to maximal run of distinct values, 469–471 Expected value, 38–39, 41, 92 of Bernoulli random variables, 39 of binomial random variables, 39–40, 50 of compound random variables, 107–108 of continuous random variables, 41 of discrete random variables, 38 of exponential random variables, 42, 282 of functions of random variables, 43–45, 49
Index of geometric random variables, 40–41, 108–109 of hypergeometric random variables, 56– 58 of normal random variables, 42–43 of Poisson random variables, 41 of the sum of a random number of random variables. See of compound random variables of sums of random variables, 49–51 tables of, 68–69 of time until k consecutive successes, 112– 113 of total discounted reward, 171, 283–284 of uniform random variables, 42 Exponential dissipation, properties of, 284– 291 Exponential distribution, 282–302, 682–685 further properties of, 291–298 Exponential distribution and Poisson process convolutions of exponential random variables, 298–302 definition, 282–283 exponential distribution, 282–302 further properties of exponential distribution, 291–298 properties of exponential distribution, 284– 291 Exponential models, 499–517 queueing system with bulk service, 514– 517 shoeshine shop, 511–514 single-server exponential queueing system, 499–511 Exponential queueing system, single-server, 499–511 Exponential random variables, 36, 42 convolutions of, 298–302 mixtures of, 599–600 rate of, 289–291 simulating, 669 Von Neumann algorithm, 682–685 Failure rate. See Decreasing failure rate; Increasing failure rate Failure rate function, 288, 302, 346 discrete time, 301 Gamblers ruin problem, 217–221 application to drug testing, 219–220 hitting times and, 629–630 maximum variable and, 629–630 Gamma distribution, 597, 680–681 Gamma random variables, 37 independent, 62
777
Gaussian processes, 646–649 General renewal process. See Delayed renewal process Geometric Brownian motion, 631–632, 658 Geometric distribution, mean of, 108–109 Geometric random variable, 31–32, 40–41 variance of, 117–119 Gibbs sampler, 250–251 Graph and components, 589 Graphs, random, 139–147, 588 Greedy algorithms, analyzing, 292–294 Hardy–Weinberg law, 207–209 Hastings–Metropolis algorithm, 248–250 Hazard function defined, 601 Hazard rate function, 288, 302 Hazard rate method, 673–676 discrete, 725–726 Hit-miss method, 729–730 Hyperexponential random variable, 290 Hypergeometric defined, 56, 58 Hypergeometric distribution defined, 99 Hypoexponential random variable, 299 Idle period, 338, 530 IFR, pillow system that is not, 599 IFR lifetime distribution, 598 Ignatov’s theorem, 133–134, 155 Impulse response function, 654–655 Inclusion and exclusion, method of, 584–592 Inclusion–exclusion bounds, 586 Inclusion–exclusion theorem, 134–136 Increasing failure rate (IFR), 596 Increasing failure rate on the average (IFRA), 571, 601 Independent events, 10–12 Independent events, pairwise, 11 Independent increments, 303 Independent random variables, 51–52, 314 sum of, 68 Indicator random variable for event, 25 Inspection paradox, 455–458 Instantaneous transmission rates defined, 378 Insurance, 121–122, 136–137, 325–327, 344–345, 446–447 Insurance ruin problem, 473–478 Interarrival time, sequence of, 307 Intersection of event, 3 Inventory example, 450–451 Inverse transformation method, 668–669 discrete analog, 685 Inversions, 95 Irreducible Markov chain, 194
778
Index
Jacobian determinant, 63 Joint cumulative probability distribution function, 47 Joint moment generating function, 96 Joint probability computing, 324–325 density function, 48 distribution of functions of random variables, 61–64 mass function, 48 of sample mean and sample variance from normal population, 74–76 Jointly continuous random variables, 48 Jointly distributed random variables, 47–64 Kolmogorov’s backward equation, 380 Kolmogorov’s forward equation, 382–383 k-out-of-n structure, 573 k-out-of-n system, 575, 606–607 with equal probability, 579 with identical components, 598 k-record index, 156 k-record values of discrete random variables, 155–158 L = λa W , 496 L = λa WQ , 496 Laplace transform, 72, 306 Limiting probabilities, 384–392 Linear growth model with immigration, 369– 371 Linear program defined, 255 List model, 137–139 Little o notation, 304 Little’s formula, 496 Long run proportion of time, 205 Markov chain generated data, mean pattern times in, 212–215 Markov chain in genetics, 207–209 Markov chains, 185–280 branching processes, 233–236 Chapman–Kolmogorov equations, 189– 193 classification of states, 193–204 continuous-time, 366–368, 381 ergodic, 242 hidden Markov chains, 256–263 independent time reversible continuoustime, 399 irreducible, 245 limiting probabilities, 204–217 Markov decision processes, 252–256 mean time spent in transient states, 230– 232
miscellaneous applications, 217–230 Monte Carlo methods, 247–252 time reversible, 236–247 time reversible Markov chains, 236–247 Markov chains, continuous-time, 365–415 birth and death processes, 368–375 computing transition probabilities, 404– 407 limiting probabilities, 384–392 time reversibility, 392–400 transition probability function, 375–384 uniformization, 401–404 Markov chains, hidden, 256–263 protecting states, 261–263 Markov decision processes, 252–256 Markov processes. See Semi-Markov processes Markovian property, 225, 227 Markov’s inequality, 77–78 Martingale process, 644, 659 Match problem, 126–128 Matching rounds problem, 110–112 Mean of geometric distribution, 108–109 joint distribution of sample, 74–76 pattern times in Markov chain generated data, 212–215 Poisson distribution with, 66 Poisson random variable width, 336 sample, 55 value function, 333 Mean time for patterns, 151–155 spent in transient states, 230–232 Mean value analysis of queing networks, 504 Memoryless random variable, 284, 366 Minimal cut set, 577 Minimal path and minimal cut sets, 574–578 Minimal path set, 574 Minimal path vector common, 574 Mixed Poisson processes, conditional or, 343–345 Mixture of distributions, 493 Moment, 46 Moment generating function determines distribution, 69 Moment generating functions, 64, 68 of binomial random variables, 65 of compound random variables, 174 of exponential random variables, 66–67, 282 of normal random variables, 67–68 of Poisson random variables, 66 of the sum of independent random variables, 68
Index tables of, 68–69 Monte Carlo methods, Markov chain, 247– 252 Monte Carlo simulation approach defined, 663 Monte Carlo’s simulation, 247 Moving average process, 654 Multinomial distribution, 87 Multivariate normal distribution, 72–73 Mutually exclusive events, 3 Negative binomial distribution defined, 88 Negative binomials, compounding distribution related to, 164–165 New better than used, 467 Nonhomogeneous Poisson process conditional distribution of event times, 361 mean value function of, 333 simulating, 693–703, 708–709 Normal random variables, 37–38, 42–43, 96 as approximations of the binomial, 80–81 simulation of, 671–673, 677–680 Null event, 3 Null recurrent state of a Markov chain, 270 Occupation time, 404, 414 Odds, 636–637 Options, 632–635, 637–638 Order statistics, 60–61 simulation of, 726–727 Ornstein-Uhlenbeck process, 652 Pairwise independent events, 11 Parallel structure, 572–573 Parallel system, 571, 579, 596, 612–615 expected life of, 608–610 that is not IFR, 599 Path vector, 574 Patterns of discrete random variables, 462–469 mean time for, 151–155 Patterns, applications to, 461–473 expected time to maximal run of distinct values, 469–471 increasing runs of continuous random variables, 471–473 patterns of discrete random variables, 462– 469 Period of a state of a Markov chain, 204 Poisson arrival queues, single-server, 338– 342 Poisson compounding distribution, 161–162 Poisson distribution, 60 Poisson distribution with mean, 66 Poisson paradigm, 70–72
779
Poisson process, 302–330 compound, 337–342 conditional distribution of arrival times, 316–327 counting processes, 302–303 definition of, 304–307 estimating software reliability, 328–330 interarrival and waiting time distributions, 307–310 miscellaneous properties of Poisson processes, 310–316 nonhomogeneous, 330–337 sampling, 694–697 sampling of, 318 simulating two-dimensional, 700–703 sums of, 352 Poisson process, exponential distribution and, 281–364 convolutions of exponential random variables, 298–302 definition, 282–283 exponential distribution, 282–302 further properties of exponential distribution, 291–298 properties of exponential distribution, 284– 291 Poisson process, generalization of, 330–345 compound Poisson process, 337–342 conditional or mixed Poisson processes, 343–345 Poisson process having rate, 304 Poisson processes conditional or mixed, 343–345 miscellaneous properties of, 310–316 Poisson queue, output process of infinite server, 336–337 Poisson random variables, 32–33, 100, 306, 320 approximation to the binomial, 32–33, 307, 333–334 expectation of, 41 maximum probability, 89 with mean, 336 random sampling, 122–123 simulating, 687–688 sums of independent, 70 Polar method defined, 680 Pollaczek–Khintchine formula, 529 Polya’s urn model, 147–151, 169–170 Positive recurrent state of a Markov chain, 204 Power spectral density, 656 Probabilistic algorithm, 229 Probabilistically process, 308
780
Index
Probabilities computing, 120–137 computing joint, 324–325 conditional, 7–10, 97–184, 136–137 defined on events, 4–6 identical component, 587 limiting, 204–217, 384–392 of matches, 126–128 stationary, 212 steady-state, 496–498 transition, 404–407 of union of events, 6 Probability density function, 34, 45 conditional, 102 joint, 48 Probability distribution function, joint cumulative, 47 Probability function, transition, 375–384 Probability mass function, 27 joint, 48 Probability theory, introduction to, 1–21 Bayes’ formula, 12–15 conditional probabilities, 7–10 independent events, 10–12 probabilities defined on events, 4–6 sample space and events, 1–4 Pure birth process, 365 Queueing system, 371 with bulk service, 514–517 G/M/k, 554–555 multiserver exponential, 372–375 with renewal arrivals, 443 simulating, 707 single-server exponential, 508–511 with finite capacity, 508–510 waiting times, 504 Queueing theory, 493–570 cost equations, 495–496 departure process, 394–395 exponential models, 499–517 finite source model, 549–552 M/G/k, 556–557 model G/M/1, 543–548 multiserver queues, 552–557 network of queues, 517–527 analysis of the Gibbs sampler, 526–527 the arrival theorem, 524–525 mean value analysis, 525 preliminaries, 494–498 steady-state probabilities, 496–498 system M/G/1, 528–531 optimization problem, 536–540 tandem quenes, 395 transition diagram, 512–513
variations on M/G/1, 531–543 work in quenes, 528 Queues infinite server, 319–320 multiserver, 552–557 output process, 336–337 single-server Poisson arrival, 338–342 Queues, network of, 517–527 closed systems, 522–527 open systems, 517–522 Quick sort algorithm, 113–116 Random graph, 139–147, 588 Random numbers, 664 Random permutations, generating, 664–666 Random subset, 726–727 Random telegraph signal process, 653 Random variable, expectation continuous case, 41–43 discrete case, 38–41 expectation of function of random variable, 43–47 Random variable identity, compound, 159 Random variable with mean, Poisson, 336 Random variables, 23–96 Bernoulli, 28–29, 39, 50, 56–57, 286, 716 binomial, 29–31, 39, 50 chi-squared, 75 compound, 108 continuous, 26, 34–38, 45, 471–473 convolutions of exponential, 298–302 covariance and variance of sums of, 53–61 Coxian, 301 discrete, 26, 155–158, 462–469 discrete random variables, 27–33 expectation of, 38–47 expectation of sum of random number of, 107–108 expectations of functions of, 43–47 exponential, 36, 42 gamma, 37 geometric, 31–32, 40–41 hyperexponential random, 290 hypoexponential, 299 independent, 51–52, 314 independent Bernoulli, 123–124 independent gamma, 62 joint density function of, 63 joint probability distribution of functions of, 61–64 jointly distributed, 47–64 jointly distributed random variables, 47–64 limit theorems, 77–83 moment generating functions, 64–76 normal, 37–38, 42–43
Index Poisson, 32–33, 306, 320 simulating binomial, 686 simulating normal, 671–673 simulating Poisson, 687–688 stochastic processes, 83–85 sum of independent, 68 sum of two independent uniform, 59 sums of independent binomial, 69–70 sums of independent normal, 70 sums of independent Poisson, 59, 70 uniform, 35–36, 42 variance of binomial, 55–56 variance of compound, 119–120 variance of geometric, 117–119 Random variables, identity for compound, 158–165 binomial compounding distribution, 163 compounding distribution related to negative binomials, 164–165 Poisson compounding distribution, 161– 162 Random variables, simulating continuous, 668–676, 677–685 chi-squared distribution, 681 data distribution, 681–682 exponential distribution, 682–685 gamma distribution, 680–681 hazard rate method, 673–676 inverse transformation method, 668–669 normal distribution, 677–680 rejection method, 669–673 Von Neumann algorithm, 682–685 Random walk, 224–230 symmetric, 199 Rate of distribution defined, 289 Rate of renewal process, 424 Rates, instantaneous transition, 378 Records, 95 Recurrent states, positive, 204 Recurring states, 195, 197 Regenerative processes, 442–451 alternating renewal processes, 445–451 Rejection method, 669–673 discrete, 725 Reliability, estimating software, 328–330 Reliability function, 602 Reliability function, bounds on, 583–595 method of inclusion and exclusion, 584– 592 second method for obtaining bounds, 593– 595 Reliability theory bounds on reliability function, 583–595 expected system lifetime, 604–610
781
reliability of systems of independent components, 578–582 structure functions, 571–578 system life as function of component lives, 595–603 Renewal, cycle and, 434 Renewal arrivals, queueing system with, 443 Renewal function computing, 458–461 estimating, 710–711 Renewal processes age of, 447 alternating, 445–451 average age of, 440 average excess of, 441 central limit theorem for, 432–433 defined, 417 excess of, 448 and interarrival distribution, 454 Renewal reward processes, 433–441 Renewal theory and its applications, 417–492 Reversible chain, time, 398 Sample mean defined, 55 Sample space, 1 Sample variance, 74 Satisfiability problem, 228–230 Second-order stationary process, 651 Semi-Markov processes, 452–454 Series system, 571, 579, 596, 611, 614 of uniformly distributed components, 604– 605 Shuffling, 267 Signed rank test, 182 Simulation, 663–731 coupling from past, 720–722 determining number of runs, 720 simulating continuous random variables, 668–676, 677–685 simulating from discrete distributions, 685–692 stochastic processes, 692–703 variance reduction techniques, 703–719 Six, 281–364 Snyder’s ratio of genetics, 273 Software reliability, estimating, 328–330 Spanning trees, 590 Standard Brownian motion, 627 Standard normal distribution function, 80 Stationary and weakly stationary processes, 649–654 Stationary increments, 303 Stationary probabilities, 212 Stationary processes harmonic analysis of weakly, 654–657
782
Index
second-order, 651 stationary and weakly, 649–654 weakly, 651 Stationary processes, Brownian motion and, 625–662 Stationary transition probabilities, 366 Stimulation, 663–731 Stirling’s approximation, 144, 203, 223 Stochastic processes, 83–85, 626, 646, 692– 703 simulating nonhomogenous Poisson process, 693–700 simulating two-dimensional Poisson process, 700–703 state space of, 84 Stopping time, 481, 675 Stratified sampling, 730 Strong law of large numbers, 78–79 Strong law for renewal processes, 423–424 Structure functions, 571–578 Suspended animation, series model with, 615–617 Symmetric random walk, 199 relation to Brownian motion, 625–626 Taylor series expansion, 82 Thinning algorithm in simulation, 694 Throughput rate, 523 Tilted density, 716 Time reversibility, 392–400 Time reversible chain, 398 Time reversible continuous-time Markov chains, independent, 399 Time reversible equations, 241 Time reversible Markov chains, 236–247 Times, conditional distribution of arrival, 316–327 Transient states, 195 mean time spent in, 230–232 Transition probabilities, computing, 404–407 Transition probability function, 375–384 Transition probability matrix, 197 Transition rates, instantaneous, 378 Tree process, 277 Truncated chain, 398 Two-dimensional Poisson process, simulating, 700–703 Two state continuous time Markov chain, 381–383
Uniform priors, 147–151 Uniform random variables, 35–36, 42, 59 Uniformization, 401–404 Union of event, 3 Union of events, probability of, 6 Unit normal distribution. See Standard normal distribution Variance reduction by conditioning, 708–712 Variance reduction techniques, 703–719 control variates, 712–714 importance sampling, 714–719 using antithetic variables, 704–707 variance reduction by conditioning, 708– 712 Variances of binomial random variables, 55–56 of compound random variables, 119–120 computing, 117–120 of exponential random variables, 282–284 of geometric random variables, 117–118 of hypergeometric random variables, 56– 58 joint distribution of sample, 74–76 of normal random variables, 46 of the number of renewals, 461 of Poisson random variables, 66 sample, 74 of sums of random variables, 53–61 tables of, 68–69 Viterbi algorithm defined, 263 Von Neumann algorithm, 682–685 Waiting time, 308 Waiting time distributions, interarrival and, 307–310 Wald’s equation, 482–485, 548, 675 defined, 481 Weak dependence, 71 Weak law of large numbers, 94 Weakly stationary process, 651 Weakly stationary processes harmonic analysis of, 654–657 stationary and, 649–654 Weibull distribution, 596 White noise, 644–646 Wiener, N., 626–627 Wiener process, 626 Work in queue, 528 Yule process, 377, 408–409
