APPLIED SEMI-MARKOV PROCESSES
APPLIED SEMI-MARKOV PROCESSES
By JACQUES JANS SEN Solvay Business School, Brussels, Bel...
71 downloads
1330 Views
10MB Size
Report
This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form
APPLIED SEMI-MARKOV PROCESSES
APPLIED SEMI-MARKOV PROCESSES
By JACQUES JANS SEN Solvay Business School, Brussels, Belgium RAIMONDO MANCA Universita di Roma "La Sapienza," Italy
Springer
Library of Congress Control Number:
ISBN-10: 0-387-29547-X
2005933791
e-ISBN 0-387-29548-8
ISBN-13: 978-0387-29547-3
Printed on acid-free paper.
AMS Subject Classifications: 60K05, 60J10, 60K15, 60K10, 60K20, 62P05, 65C40, 65C99, 91B30, 91B7Q
© 2006 Springer Science+Business Media, Inc.
All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science-t-Business Media, Inc., 233 Spring Street, New York, NY 10013, USA), except for brief excerpts m connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. Printed in the United States of America. 987654321 springeronline.com
Contents Preface 1 Probability tools for stochastic modelling 1 The sample space 2 Probability space 3 Random variables 4 Integrability, Expectation and Independence 5 Main distribution probabilities 5.1 The binomial distribution 5.2 The Poisson distribution 5.3 The normal (or Laplace-Gauss) distribution 5.4 The log-normal distribution 5.5 The negative exponential distribution 5.6 The multidimensional normal distribution 6 Conditioning (From independence to dependence) 6.1 Conditioning: introductory case 6.2 Conditioning: general case 6.3 Regular conditional probability 7 Stochastic processes 8 Martingales 9 Brownian motion
1 1 3 7 10 16 16 17 18 20 21 22 24 24 28 32 36 40 43
2 Renewal theory 1 Purpose 2 Main definitions 3 Classification of renewal processes 4 The renewal equation 5 The use of Laplace transform 5.1 The Laplace transform 5.2 The Laplace-Stieltjes transform 5.3 An application to the renewal function 6 Application of Wald' s identity 6.1 Wald's identity 6.2 A lower bound for renewal function R 7 Asymptotical behaviour of the N(t)-pYOCQSS 8 Recurrence times 8.1 Definitions 8.2 Distributions of recurrence times 8.3 Asymptotic behaviour
45 45 46 47 52 59 59 64 66 67 67 68 69 72 72 73 79
Contents
VI
8.4 Example: the Poisson process 8.5 Some parameters of the limit distributions 8.6 A characterization of the Poisson process 9 Delayed and stationary renewal processes 10 Numerical aspects 10.1 General quadrature method 10.2 Some particular formulas 10.3 Numerical approximation of the Poisson process 10.4 A real life example in motor-car accidents 10.4.1 Data description 10.4.2 The result distribution 3 Markov chains 1 Definitions 2 Markov chain state classification 2.1 Periodic and aperiodic states 2.2 Essential and inessential states - Irreducibility 2.3 Transient and recurrent states 3 Occupation times 4 Computation of absorption probabilities 5 Asymptotic behaviour 6 Examples 7 A case study in a social insurance problem (Janssen (1966)) 8 Asymptotic numerical treatment 8.1 An algorithm for MC Asymptotical Study 8.2 Irreducible real data example in motor-car insurance 8.3 Reducible and uni-irreducible examples-Canonical form connection 4 Markov renewal theory, Markov random w^alks and semiMarkov processes. 1 Positive (J-X) processes 2 Semi-Markov and extended semi-Markov chains 3 Primary properties 4 Examples 5 Markov renewal processes, semi-Markov processes and associated counting processes 6 Markov renewal functions 7 Classification of the state of a MRP 8 The Markov renewal equation 9 Asymptotic behaviour of a MRP 9.1 Asymptotic behaviour of Markov renewal functions 9.2 Asymptotic behaviour of solutions of Markov renewal
82 83 86 87 92 93 95 97 98 98 101 105 105 108 108 109 110 113 113 115 119 122 125 125 129 133
145 145 145 147 151 154 156 161 163 165 165
Contents
Vll
equations 10 Asymptotic behaviour of SMP 10.1 Irreducible case 10.2 Non-irreducible case 10.2.1 Uni-reducible case 10.2.2 General case 11 Recurrence times 11.1 Definitions 11.2 Transient results 11.3 Asymptotic results 12 Delayed and stationary MRP 13 Particular cases of MRP 13.1 Renewal processes and Markov chains 13.2 SMP an MRP of zero order (Pyke (1962)) 13.2.1 First type of zero order MRP 13.2.2 Second type of zero order MRP 13.3 Continuous Markov processes 14 A case study in social insurance (Janssen (1966)) 14.1 The semi-Markov model 14.2 Comparison of Markov and semi-Markov models 15 Asymptotical numerical examples
168 169 169 171 172 173 173 173 174 175 178 185 185 185 186 186 187 187 187 188 189
5 Functionals of (J-X) processes 1 (J-X) processes 2 Functionals of (J-X) processes 3 Functionals of positive (J-X) processes 4 Classical random walks and risk theory 4.1 Purpose 4.2 Basic notions on random walks 4.3 Renewal results on ladder variables 4.4 Classification of random walks 5 Defective positive (J-X) processes 6 Semi-Markov random walks 7 Indice variables for semi-Markov random walks 8 Distribution of the supremum for semi-Markov random walks
193 193 195 201 202 202 202 207 210 214 219 222
6 Non-homogeneous Markov and semi-Markov processes. 1 General definitions 1.1 Completely non homogeneous semi-Markov processes 1.2 Special cases 1.2.1 Non-homgeneous Markov additive process and semi-Markov process
224 227 227 227 231 231
Contents
Vlll
1.2.2 Non-homogeneous MC 1.2.3 Homogeneous Markov additive process 1.2.4 Non-homogeneous renewal process 1.3 Intensities for CNHSMP 1.3.1 Definition 1.3.2 Generalized Polya processes 2 Non-homogeneous Markov chains 3 Non-homogeneous Markov processes 3.1 Classical definition 3.2 The Chapman Kolmogorov equations 3.3 Intensity functions 3.4 The special case of constant intensity functions 3.5 NHMP as a special case of NHSMP 4 An NHMC transportation example
232 232 232 233 233 234 235 236 237 237 238 241 243 243
7 Markov and semi-Markov rew^ard processes. 1 Reward structure 1.1 Classification and notation 1.2 Discrete time fixed interest rate 1.3 Discrete time variable interest rate 1.4 Continuous time fixed interest rate intensity 1.5 Continuous time variable interest rate intensity 2 Discrete time Markov reward processes 2.1 Undiscounted case 2.1.1 First model 2.1.2 Second model 2.1.3 Third model 2.1.4 Fourth model 2.1.5 Fifth model 2.1.6 Sixth model 2.1.7 Seventh model 2.1.8 Eighth model 2.2 Discounted case 2.2.1 Immediate cases 2.2.2 Due cases 2.3 General algorithm for the DTMRWP 2.3.1 Homogeneous MRWP 2.3.2 Non-homogeneous MRWP 2.4 A DTMRWP applied example 3 Semi-Markov reward processes 3.1 Undiscounted CTSMRWP 3.1.1 First model 3.1.2 Second model
247 247 247 249 249 250 251 252 252 252 253 253 254 254 255 255 256 256 256 261 264 264 265 265 268 268 268 269
Contents 3.1.3 Third model 3.1.4 Fourth model 3.1.5 Fifth model 3.1.6 Sixth model 3.1.7 Seventh model 3.2 Discounted CTSMRWP 3.2.1 First model 3.2.2 Second model 3.2.3 Third model 3.2.4 Fourth model 3.2.5 Fifth model 3.2.6 Sixth model 3.2.7 Seventh model 3.2.8 Eighth model 3.2.9 Ninth model 4 Insurance applications of CTSMRWP 4.1 Two state examples 4.2 Three state examples 4.3 Four state examples
ix 270 271 272 272 273 273 273 275 275 276 277 277 278 278 279 280 282 286 290
References
295
Author index Subject index
303 305
PREFACE This book aims to give a complete and self-contained presentation of semi-Markov processes with finitely many states, divided into three major topics: theory, computation, and application in a correct approach for building applied stochastic models. To help assure the self-containedness of the book. Chapter 1 summarizes the basic probability tools that the reader will need to understand our presentation. The authors argue that semi-Markov processes furnish a framework for the construction of useful applied models, some of them giving fruitful alternative approaches to the classical models, as shown in finance with mixed semi-Markov diffusion models, for example. We strongly hope that practitioners with a variety of interests will have the opportunity to use these new types of models in their own fields. Another important feature of this book is its presentation of both homogeneous and non-homogeneous models. It is well known that the fundamental structure of many real-life problems is non-homogeneous in time and the application of homogeneous models to such problems gives, in the best case, only approximated results or, in the worst case, nonsensical results. This book addresses a very large public that includes undergraduate and graduate students in mathematics, applied mathematics, economics and business studies; actuaries and financial intermediaries; engineers and operations researchers; and also researchers in universities and RD departments in banking, insurance and industry. Readers who have mastered the material in this book will have a firm grasp of semi-Markov processes and be able to use them in the solution of their problems. They will indeed have a new approach giving a more competitive model related to the complexity of real-life problems. Let us now give some comments on the contents of the book. As we start from the fact that the semi-Markov processes are the children of a successful marriage between renewal theory and Markov chains, these two topics are presented in Chapters 2 and 3. The full presentation of Markov renewal theory, Markov random walks and semi-Markov processes is given in Chapter 4; functionals of (JX) processes and semi-Markov random walks are presented in Chapter 5 in the homogeneous case and are followed by a presentation of non-
xii
Preface
homogeneous Markov and semi-Markov processes in Chapter 6. Finally, continuous time semi-Markov reward processes are presented in Chapter 7, and in the last part of this chapter applications to insurance problems are given. Many parts of this book have been tought by the authors at several universities: Universite Libre de Bruxelles, Vrije Universiteit Brussel, Universite de Bretagne Occidentale (EURIA), Universites de Paris 1 (La Sorbonne) and Paris VI (ISUP), ENST-Bretagne, Universite de Strasbourg, and the Universities of Roma ("La Sapienza"), Firenze and Pescara. Our common experience has joined to create this book, taking into account the remarks of students in our various lectures and hoping to convince potential readers to enter an attractive mathematical field with new frontiers for real-life applications.
Jacques Janssen
Raimondo Manca
Chapter 1 PROBABILITY MODELLING
TOOLS
FOR
STOCHASTIC
In this chapter, the reader will find a short summary of the basic probability tools useful for understanding of the following chapters. Although there currently exist some very good presentations of stochastic processes from a "modem" point of view, that is to say with emphasis on sample paths or trajectories without using measure theory (see for example Cinlar (1975b) or Ross (2000), a deep understanding of stochastic processes requires the notion of integrability on a measurable space, well presented in Kolmogorov and Fomin (1961), Halmos (1974), Chung (2000) and many other standard textbooks. We will focus our attention on stochastic processes in discrete time and continuous time defined by sequences of random variables.
1 THE SAMPLE SPACE The basic concrete notion in probability theory is that of the random experiment, that is to say an experiment for which we cannot predict in advance the outcome. With each random experiment, we can associate the so-called elementary events CO, and the set of all these events Q is called the sample space. Some other subsets of Q will represent possible events, Let us consider the following examples. Example 1.1 If the random experiment consists in the flipping of a coin {H means that the outcome of the toss is heads and 7 that it is tails.), then we have as sample space ^ = {H,T]. (1.1) The family of all possible events is the set 3 = {0,{//},{r,}{//,r,}},
(1.2)
that is the family of all subsets of Q. Example 1.2 If the experiment consists in the throwing of two distinguishable dice, then the sample space consists of the 36 elementary events (1.3) ^ = {{Uj)JJ = \.....6], where the outcome (zj) is said to occur if / appears on the first die andy on the second one.
2
Chapter 1
Example 1.3 If the experiment consists in the throwing of two indistinguishable dice, then the sample space consists of the 21 elementary events = l,„.,6}, (1.4) Q=^{(Uj)J<jJ,j where the outcome (ij) is said to occur if / is the result of one of the two dice, andy the result of the second. Example 1.4 If the experiment consists in the measurement of the lifetime of an integrated circuit, then the sample space is the set of all non-negative real numbers R"^. Possible events are [a,6],(a,6),[a,Z?),(a,6] where for example the event [a,b) means that the lifetime is at least a and strictly inferior to b. Example 1.5 An insurance company is interested in the number of claims per year for its portfolio. In this case, the sample space is the set of natural numbers N. Example 1.6 A bank is to invest in some shares; so the bank looks to the history of the value of different shares. In this case, the sample space is the set of all nonnegative real numbers R"^. To be useful, the set of all possible events must have some properties of stability so that we can generate new events such as: (i)
the complement A"": A'' -{coeO.:co^
A],
(ii) the union A[jB : A[jB = {co:coe A orcoeB ],
(1.5) (1.6)
(1.7) (iii) the intersection AClB : AClB = {CO:COG A,COGB] . More generally, if {A^,n>V) represents a sequence of events, we can also consider the following events: n>\
n>\
representing respectively the union and the intersection of all the events of the given sequence. The first of these two events occurs iff at least one of these events occurs and the second iff all the events of the given sequence occur. The set Q is called the certain event and the set 0 the empty event. Two events A and B are said to be disjoint or mutually exclusive iff ^ n 5 =0. (1.9) Event A implies event B iff AczB. (1.10) Example 1.7 (a) In Example 1.2, the event "the sum of the dice points equals 7" is the set
Probability Tools
3
{a6),(2,5),(3,4),(4,3),(5,2),(6,l)}. (1.11) (b) In Example 1.3, the event "the sum of the dice points is seven" is the set {(1,6),(2,5X(3,4)}. (1.12) (c) In Example 1.6, the event "the value of the share is between "50$ and 75$" is given by the set [50,75] (1.13)
2 PROBABILITY SPACE Given a sample space Q , the set of all possible events will be noted by 3 , supposed to have the structure of a a-field or a a-algebra. Definition 2.1 The family 3 of subsets of Q is called a cr-field or a a algebra iff the following conditions are satisfied: (i) Q , 0 belong Xo 3 , (ii) Q is stable under denumerable intersection: ^„G3,V«>l=^f|^„€5, (2.1) n>\
(iii) 3 is stable for the complement set operation: ^G3=>^'e3,(2.2) --a-A). {with A' Then, using the well-known de Morgan's laws saying that
'u^"l -n\
)
n>\
\n>\
)
(2'3)
n>\
it is easy to prove that a G -algebra 3 is also stable under denumerable union: (2.4) ^„ G 3, V/? > 1 =:> IJ^^ G 3. Any couple (Q, 3) where 3 is a (T -algebra is called a measurable space, The next definition concerning the concept of probability measure or simply probability is an idealization of the concept of X\\Q frequency of an event. Let us consider a random experiment called E with which is associated the couple (Q, 3 ) ; if the set A belongs to 3 and if we can repeat the experiment E n times, under the same conditions of environment, we can count how many times A occurs. If n{A) represents this number of occurrences, the frequency of the event A is defined as f{A)^^^.
(2.5)
Chapter 1 In general, this number tends to become stable for large values of«. The notion of frequency satisfies the following elementary properties: (i) {A,Bs:5,AnB = 0=>fiA[jB)^f(A) + fiB), (2.6) (ii) / ( n ) - l , (2.7) (iii) A,Be3,:::>f(A[jB) = f(A) + f(B)-f(AnB), (2.8) (iv)A€3^f(A') = l-f(A). (2.9) To have a useful mathematical model for the theoretical idealization of the notion of frequency, we now introduce the following definition. Definition 2.2 a) The triplet (Q,3,P) is called a probability space if Q is a non-void set of elements, 3 a or-algebra of subsets of Cl and P an application from 3 /a[0,l] such that: (A„,n > l),A„ e 3,« > 1: (/ ^ y ^ A. fl Aj = (f) ( \ „ (2.10) (i)
^P\ U^" hS-^^^") {\
J «=1
(ii) P(Q) = 1. (2.11) b) The application P satisfying conditions (2.10) and (2.11) is called a probability measure or simply probability. Remark 2.1 1) The sequence of events {A^,n> 1) satisfying the condition that (A^,n>llA,e3,n>l:i^j=>A^nAj=0 (2.12) are called mutually exclusive, 2) The relation (2.11) assigns the value 1 for the probability of the entire sample space Q. There may exist events A' strictly subsets of Q such that (2.13) P{A') = l. In this case, we say that A is almost sure or that the statement defining A is true almost surely (in short a.s.) or holds for almost all CD . From axioms (2.10) and (2.11), we can deduce the following properties: Property 2.1 (i)If ^ , 5 e 3 , t h e n , P(A[jB) = P(A) + P(B)-P(AnB), (ii) If ^ E 5, then, P(A') = l-P(A).
(2.14) (2.15)
Probability Tools
5
(2.16) (iii) P(0) = O. (iv) If (B^,n>l) is a sequence of disjoint elements of 3 forming a partition of Q, then for all A belonging to 5 , P(A)^f^P(AnB„).
(2.17)
(v) Continuity property ofP: if (A^,n> 1) is an increasing (decreasing) sequence of elements of 3 , then
P\ \JA„ UlimP(^J; P\ H A UlimP(^J . \n>]
J
\
\n>\
J
(2.18)
J
Remark 2.2 a) Boole's inequality asserts that if {A^,n> 1) is a sequence of events, then
(2.19)
P\ U^ A U E ^ K ) ,A7>1
J
n>\
b) From (2.14), it is clear that we also have A^B=>P(A)l,j = \,...,l,k.^kj
P(A) = ZP,^,
if i^j
is given by
(2.24)
Example 1.1 is a particular case. For a "fair" coin, the probability measure P is defined as follows:
Chapter 1
P({//}) = P({r}) = i .
(2.25)
For Example 1.2, we have for two "fair" dice: ^({^i}) = ^ -
(2-26)
For Example 1.3, we get: P({''J}) = J^''<J'
^ ( { M - } ) - ^ , / = 1,...,6.
(2.27)
b) The continuous case Let Q be the real set R; It can be proven (Halmos (1974)) that there exists a minimal cr-algebra generated by the set of intervals: b}. (2.28) It is called the Borel a-algebra represented by>5 and the elements of J3 are called Borel sets. Given a probability measure P on (Q,>5), we can define the real function F, called the distribution function related to P, as follows. Definition 2.3 The function Ffrom R to [0,l] defined by: P((-oo,x]) = F(x),x G R
(2.29)
is called the distribution function related to the probability measure P. From this definition and the basic properties of P, we easily deduce that: P({a,b]) = F(b)-F(a), P({a,b)) = F(b-)-F(a), ^ ^ ^ ^ (2.30) P{[a,b]) = F(b)-F(a-). P{[a,b)) = Fib-)-F(a-l Moreover, from (2.29), any function F from R to [0,l] is a distribution function (in short d.f) iff it is a non-decreasing function satisfying the following conditions: F is right continuous at every point XQ, l\mF(x) = F(x,l (2.31) XTXQ
and moreover lim F(x) = 1, lim F(x) = 0 .
(2.32)
If the function F is derivable on R with/as derivative, we have F(x)=r
f(y)dy,xeR.
(2.33)
J-oo
The function/is called the density function associated with the d.f F and in the case of the existence of such a Lebesgue integrable function on R, F is called absolutely continuous.
Probability Tools
7
From the definition of the concept of integral, we can give the intuitive interpretation of/as follows; given the small positive real number Ax, we have: (2.34) P[{x,x + Ax})« fix)Ax . Using the Lebesgue Stieltjes integral, it can be seen that it is possible to define a probability measure P on (R,/?) starting from a d.f F on R by the following definition of P: P(A)=jdF(x)yAe^, (2.35) A
In the absolutely continuous case, we get P(A)^jf(y)dy.
(2.36)
A
Remark 2,3 In fact, it is also possible to define the concept of d.f in the discrete case if we set, without loss of generality, on (NQ,2^'), the measure P defined from the sequence (2.22). Indeed, if for every positive integer k, we set F(k) = f^Pj
(2.37)
7=1
and generally, for any real x,
then, for any positive integer k, we can write P{{l.,.,k]) = F{k)
(2.39)
and so calculate the probability of any event.
3 RANDOM VARIABLES Let us suppose the probability space (Q,3,P) and the measurable space (E,i//) are given. Definition 3.1 A random variable (in short r.v.) with values in E is an application Xfrom Q to E such that \/BGI/:X-'(B)G^,
(3.1)
where X\B) is called the inverse image of the set B defined by X-\B) = {co:X(cD)eB],X-\B)e3.
(3.2)
Particular cases a) If (£*, I//) = (R, J3), X is called a real random variable.
g
b) If(E,i//) = (R,j3),
Chapter 1
where
R
is the extended real line defined by
RU{+oo}U{-oo} and P the extended Borel cr-field of M, that is the minimal (J -field containing all the elements of /? and the extended intervals [-oo,a),(-oo,a],[-oo,a],(-oo,a), [a,+00),(a,+00],[a,+00],(a,+00), a e M , Xis called a rea/ extended value random variable. c) If £' = R"(«>l)with the product cr-field y5^"^of J3, X is called an «dimensional real random variable. d) If ^ = R^"^(n>l) with the product a-field yS^'^^of ^ , Z is called a real extended n-dimensional real random variable. A random variable Xis called discrete or continuous according as Stakes at most a denumerable or a non-denumerable infinite set of values. Remark 3.1 In measure theory, the only difference is that condition (2.11) is no longer required and in this case the definition of a r.v. given above gives the notion of measurable function. In particular a measurable function from (R,y5) to (R,/?) IS Qdi\\Qd?i Borel function. Let Jf be a real r.v. and let us consider, for any real x, the following subset of Q: [(D:X{CO)<X].
As, from relation (3.2), [co: X{(o) <x] = X-' ((-oo,x]), it is clear from relation (3.1) that this set belongs to the a -algebra 3 . Conversely, it can be proved that the condition {^:X(6;)<X}G3,
(3.4)
(3.5)
valid for every x belonging to a dense subset of R , is sufficient for X being a real random variable defined on Q. The probability measure P on (Q,3) induces a probability measure // on (R,y5) defined as V5 e y5: i^{B) = P{{o): X(co) e B}).
(3.6)
We say that ju is the induced probability measure on (R,>5), called the probability distribution of the r.v. X. Introducing the distribution function related to //, we get the next definition. Definition 3.2 The distribution function of the r.v. X, represented by F^, is the function from R ^ ' [ 0 , l ] defined by
Probability Tools
F^(X) = JU({-^,X])
9
= P({CD:X(CO)<X]),
In short, we write F,{x) = P{X<x),
(3.7)
(3.8)
This last definition can be extended to the multi-dimensional case with a r.v. X being an n-dimensional real vector: X = {X^,,,.,X^), a measurable appHcation from (Q,3,P) to (M%>5"). Definition 3.3 The distribution function of the r.v. X ~{X^,...,X^),
represented
by F^, is the function from W to [0,l] defined by F,{x„...,x„) = P({co:X,{co)<x„...,X„{co)<x„}).
(3.9)
In short, we write (3.10) FM^...,x„) = P{X,<x„...,X„<x„). Each component Xi (i=l,...,n) is itself a one-dimensional real r.v. whose d.f, called the marginal d.f, is given by F^(x.) = F^(+oo,...,+oo,x.,+oo,...,+oo). (3.11) The concept of random variable is stable under a lot of mathematical operations; so any Borel function of a r.v. Xis also a r.v. Moreover, if Xand 7 are two r.v., so are mf{X,Y},sup{Xj},X + Y,X-Y,X'Y,y,
(3.12)
provided, in the last case, that 7 does not vanish. Concerning the convergence properties, we must mention the property that, if (X^,n> 1) is a convergent sequence of r.v. - that is, for aWcoeQ, the sequence (A^^ (^y)) converges to X(co) - , then the limit X is also a r.v. on Q. This convergence, that may be called the sure convergence, can be weakened to give the concept of almost sure (in short a.s.) convergence of the given sequence. Definition 3.4 The sequence (X^(a))) converges a.s. to X(CD) if P[{cD:\imX^(co) = X(co)}) = l.
(3.13)
This last notion means that the possible set where the given sequence does not converge is a null set, that is a set A^ belonging to 3 such that P(N) = 0. (3.14)
IQ
Chapter 1
In general, let us remark that, given a null set, it is not true that every subset of it belongs to 5 but of course if it belongs to 3 , it is clearly a null set (see relation (2.20)). To avoid unnecessary complications, we will suppose from now on that any considered probability space is complete. This means that all the subsets of a null set also belong to 3 and thus that their probability is zero.
4 INTEGRABILITY, EXPECTATION AND INDEPENDENCE Let us consider a complete measurable space (Q, 5 , //) and a real measurable variable X defined on Q . To any set A belonging to 3 , we associate the r.v. I^ , called the indicator of ^4, defined as
If there exists partition {A^,n> 1) with all its sets measurable such that (D^A^:=^ X{co) = a^(a^ e R),n > 1,
(4.2)
then X is called a discrete variable. If moreover, the partition is finite, it is said to bQjinite. It follows that we can write JL"under the following form: 00
X(a)) = ^a„I,^(co).
(4.3)
Definition 4.1 The integral of the discrete variable X is defined by jXdM = Za„MiA„), n
(4.4)
n=\
provided that this series is absolutely
convergent,
Of course, i f X i s integrable, we have the integrability of \x\ too and
^X\d^i = Y}a„\^{A„).
(4.5)
To define in general the integral of a measurable function X, we first restrict ourselves to the case of a non-negative measurable variable X for which we can construct a monotone sequence {X^,n > 1) of discrete variables converging to X as follows:
^«(^) = ET7^f . , , . . . ! •
(4-6)
Probability Tools
11
Since for each n, XSco)<X^,,{co\ 1
(4.7)
0<X{cD)-XSco)\) of discrete variables converges well monotonically to X on Q . Deflnition 4.2 The non-negative measurable variable X is integrable on Q iff the elements of the sequence (X^,n > 1) of discrete variables defined by relation
\x„dP converges,
(4.6) are integrable and if the sequence
Vn
J
From this last definition, it follows that E{X) = \imE{XJ, where - k
(
(4.8)
^
(4.9)
(o\—<Xl)is a sequence of integrable independent and identically distributed r.v., then
-t,X,-^^E(X),
(4.49)
n k=\
The next section will present the most useful distribution functions for stochastic modelling.
5 MAIN DISTRIBUTION PROBABILITIES Here we shall restrict ourselves to presenting the principal distribution probabilities related to real random variables.
5.1 The Binomial Distribution Let us consider a random experiment E such that only two results are possible: a "success"(5) with probability/? and a "failure (F) with probability q=l'-p. If n independent trials are made in exactly the same experimental environment, the total number of trials in which the event S occurs may be represented by a random variable^whose distribution (p.J = 0,...,n) with p,=P(X = ili = l,...,n (5.1) is called a binomial distribution with parameters (n,p). From basic axioms of probability theory seen before, it is easy to prove that Pi
U^p^q"-\i
= 0,.,.,n,
(5.2)
\'J
a result from which we get (5.3) E(X) = np,yar(X) - npq. The characteristic function and the generating function, when it exists, of X respectively defined by
are given by
gAt) = {pe'+qy.
Probability Tools
17
Example 5.1 {The Cox and Rubinstein financial model) Let us consider a financial asset observed on n successive discrete time periods so that at the beginning of the first period, from time 0 to time 1, the asset starts from value SQ and has at the end of this period only two possible values, uS{) and dSo ( 00 (5.24) converges in law to a standard normal distribution. This means that the sequence of the distribution functions of the variables defined by (5.21) converges to O . This theorem was used by the Nobel Prize winner H. Markowitz (1959) to justify that the return of a diversified portfolio of assets has a normal distribution. As a particular case of the Central limit Theorem, let us mention the de Moivre 's theorem starting with f 1, with prob. p , X^=\ ^ ^ (5.25) [0, with prob. 1-/?, so that, for each n, the r.v. defined by relation (5.22) has a binomial distribution with parameters {n,p). By applying now the Central Limit Theorem, w e get the following result: S,-np
la^
>iV(0,l), ^np(\-p) called de Moivre's result.
(5.26)
5.4 The Log-Normal Distribution If the normal distribution is the most frequently used, it is nevertheless true that it could not be used for example to model the time evolution of a financial asset like a share or a bond, as the minimal value of these assets is 0 and so the support of their d.f is the half-real line [0,+oo). One possible solution is to consider the truncated normal distribution to be defined by in setting all the probability mass of the normal distribution on the negative half-real line on the positive one, but then all the interesting properties of the normal distribution are lost. Also, in order to have a better approach to some financial market data, w e have to introduce the log-normal distribution. The real non-negative random variable X has a lognormal distribution of parameters ju,cr - and w e will write X < LN(ju,cr) - if the r.v. l o g Z h a s a normal distribution with parameters Consequently, the density function of X i s given by
ju,a^.
21
Probability Tools 0,x0. (5.27)
Indeed, we can write P{X <x)^ P{logX
(6.7)
the so-called "theorem of
(6.8)
a relation expanding relation (6.7). P\C\AA
= P{A,),,,P{A^)
(6.9)
is true in the case the of independence of the n considered events. If the event B is fixed and of strictly positive probability, relation (6.4) gives the way to define a new probability measure on (Q, 3) denoted P^ as follows: P ( ^ ) = J^(iIL?),V^G5.
(6.10)
PB is in fact a probability measure as it is easy to verify that it satisfies conditions (2.10) and (2.11) and so P^is called the conditional probability measure given B. The integral with respect to this measure is called the conditional expectation EB relative to PB> From relation (6.10) and since PB{B)=^\, we thus obtain for any integrable r.v. Y\ E,(Y)=\Y{co)dP, n
=-L^JY(co)dP,
(6.11)
P\B) Q
For our next step, we shall now consider a countable event partition {B^,n>\) of the sample space Q. That is: Q = Q 5 „ 5 , n 5 , = 0 , V/,7:/^7. Then, for every event A, we have:
(6.12)
2^
Chapter 1
P(A) = Y.P(B„f]A)
(6.13)
and by relation (6.10): PiA) = XPiB„)P(A\B„).
(6.14)
n>\
Now, for any integrable r.v. Y, we can write: E(Y) = j;^JY(co)dP
(6.15)
and from relation (6.11): E{Y) = Y,P{B„)E,^{Y).
(6.16)
As the partition B^,n>\ generates a sub-a -algebra of 3 denoted 3, obtained as the minimal sub- a -algebra containing all the events of the given partition, we can write the relation (6.16) as E^ (7) called the conditional expectation of Y given 3 , . It is very important to understand that this conditional expectation is a function of CO and so a new random variable as: E^^(Y){co) = YEsSy)\{co), (6.17) n>\
So, the random variable E^ (7) assumes on each set B the value of E^ (7) that is constant and defined by relation (6.11) with B=Bn. Now, let us compute the expectation of this new random variable E^ (Y); from relation (6.17), we can deduce that: E(E^^ (Y)(CO)) - E\Y,ES„
(y)\
(CO) I
^Y,E(E,^(Y)\,^(CO)), (6.18)
= Y,E,^(Y)E(\(m)), n>\
^^E,^iY)P(B„), and finally from relation (6.16), we get: E(E^^(Y)(co)) = E(Y).
(6.19)
Furthermore, since for any set B belonging to 3 , , B is the union of a certain number of events J?„, finite or at the most denumerable, we obtain by integrating both members of relation (6.17):
Probability Tools
27
1^3, {Y){co)dP =X K (^)^«. (^)^^' ==Z^.„(^)jl«„(^)^^'
(6.20)
Using now relation (6.11), we get:
K ( r ) ( . , . P . [ i : ^ ^ J >•(.>,.]p(Br\B„), 5n5„
=S
J YicoW,
(6.21)
= JY((o)dP. In conclusion, we get: JE^^ {Y){co)dP = JY(co)dP,B e 3,. B
(6.22)
B
Of course, for5 = Q, this last relation is identical to (6.19). We shall focus our attention on the meaning of result (6.22) which equates two integrals on every set B belonging to 3i but presenting an essential difference: in the left member, the integrand E^ (Y) is 3, -measurable but in the right member, the integrand 7 is 3 -measurable and so not necessarily 3, -measurable since 3, cz3. Furthermore, the function E^ (Y) is a.s. unique; indeed, let us suppose that there exists another function 3j -measurable/so that relation (6.22) is still true. Consequently, we have: jf(co)dP = JY(o))dP,B e 3,. (6.23) B
B
From relations (6.22) and (6.23), we obtain: \f{co)dP - j^3, {co)dP,B e 3, B
(6.24)
B
SO that: ff((u)dP = j(f(co) - E^^ (co))dP, 5 e 3 , .
(6.25)
23
Chapter 1
As this relation holds for all B belonging to 3 , , it follows that E^ (Y) = / , a . s . ; otherwise, there would exist a set B belonging to 3, so that the difference E^ (7) - / would be different from 0 and so also the integral \{f{co)-E^^{co))dP
(6.26)
B
in contradiction with property (6.25).
6.2 Conditioning: General Case We can now extend the definition (6.17) to arbitrary sub-cr-algebras using property (6.22) as a definition with the help of the Radon Nikodym theorem, Halmos (1974). Definition 6.1 If 5, is a sub-a-algebra of 3 , the conditional expectation of the integrable r,v. Y given 3j, denoted by E^ (Y) or E(Y\'3A,
is any one r.v. of the
equivalence class such that: (i) E^ (Y) is 3, -measurable, (ii)
1^3^ {Y){co)dP = \Y{co)dP,B e 3,. B
(6.27)
B
In fact, the class of equivalence contains all the random variables a.s. equally satisfying relation (6.27). Remark 6.1 Taking B = Q in relation (6.27), we get: E(E^Y)) = E(Y),
(6.28)
a relation extending relation (6.17) to the general case. Particular cases (i) 3 , is generated by one r,v. X. This case means that 3, is the sub- cr -algebra of 3 generated by all the inverse images of X and we will use as notation: E,^(Y) = E(Y\X), (6.29) and this conditional expectation is called the conditional expectation of Y given X, (ii) 3, is generated by n r.v. X^,...,X^. This case means that 3, is the sub- cr -algebra of 3 generated by all the inverse images of X^,...,X^ and we will use as notation:
Probability Tools
29
E^^{Y) = E{Y\X„...,X„),
(6.30)
and this conditional expectation is called the conditional expectation of Y given In this latter case, it can be shown (Loeve (1977)) that there exists a version q){X^,..,,X^) of the conditional expectation so that ^ is a Borel function from R" to E and as such it follows that E{Y\X^,,..,X^)
is constant on each set
belonging to 3j for which X^{co) = X^,...,X^(CD) = x^, for instance. This justifies the abuse of notation E(Y\X,((o)^x„...,X„(co) = x„) = ,p{x„...,x„)
(6.31)
representing the value of this conditional expectation on all the co's belonging to the set [(D:X^(CD) = X^,...,X^(CO) = X^] .
Taking B = Q in relation (6.28), we get: E(Y)= j ^ ( 7 | X , ( ^ ) = x„...,Z,(^) = x„)^P(X,(^)<x„...,X,(^)<xJ, (6.32) a result often used in the sequel to evaluate the mean of a random variable using its conditional expectation with respect to some given event. (iii) If 5 i = { 0 , Q } , we get £(7|3,) = £(7)and if 3,={0,B,B\Q] , then on Band E(Y\:5,) = E(Y\B") on B', (iv) Taking as r.v. 7 the indicator of the event A, that is to say: (hcoe A, [0,co^A, the conditional expectation becomes the conditional probability of A given 3, denoted as follows: E(Y\:5,)
= E(Y\B)
P(A\3,) = E(l,(cop,) and then relation (6.27) becomes: jp{Ap,(co))dP = P(Af]BlBe:5,.
(6.34) (6.35)
B
Letting 5 = Q in this final relation, we get: = P{Al E(p(Ap,))
(6.36)
a property extending the theorem of total probability (6.14). If moreover, A is independent of 3 , , that is to say, if for all B belonging to 3 , : P{Ai^B) = P{A)P{B), (6.37) then we see from relation (6.34) that: (6.38) P[A\S,){o)) = P{A\coea.
30
Chapter 1
Similarly, if the r.v. Y is independent of 3 , , that is to say if for each event B belonging to 3, and each set A belonging to the cr-algebra generated by the inverse images of Y, denoted by 0VAe3,
P{A\^,)(CD)
K^i
J
Zt
(6.61) (6.62)
^
'
'
(6.63)
It is important to note here that the null events A^pA^2'^3' ^n which respectively these last three properties are not true, are generally not identical, so that for each CO, the random set function P(.|3,)((^)from 3 to [0,l] is not necessarily a probability measure since, to be so, these three sets must be identical. That is why we must introduce the concept of regular conditional probability (see Loeve (1977) or Gikhman and Skorokhod (1980)). Definition 6.2 The conditional probability PU3A(cD)is a regular conditional probability if there exists a function p(.,.) from 3 x Q to [0,l] so that: (i) for almost all co of Cl, p(.,co), as a set function on 3 , is a probability measure, (ii) for every fixed event A belonging to 3 , p(A,.) is "5-measurable and is a version of the given conditional probability, that is a.s., we have: (6.64) p{A,co) = P[Ap){co). The interest of such regular conditional probabilities is that we can express the related conditional expectation of an integrable r.v. X a.s. as an integral with respect to the measure j!?(.,(2;): E[Xp,){(D)=
\x{co')p{dco\co).
(6.65)
In many applications, it is sufficient to restrict the attention to all events of the sub- (J -algebra generated by a r.v. X, with values in the measurable space {E^ii/), and denoted by (J{X) . This means that we are only interested in the following conditional probabilities: P[Ap,),AscT{X). (6.66)
34
Chapter 1
If the conditional probability given 3i is regular, we can then define the function Cfrom o-(X)xQ to [0,l] as CiA,co) = P(A\3,)icD),Aea(X),cDeQ
(6.67)
satisfying (i) for almost all CD of Q, C(.,co), as a set function on or(X), is a probabihty measure, (ii) for every fixed events belonging to cr(X), C(A,.) is 3 -measurable, (iii) for every event A belonging to (j(X) and for every event B belonging to 3 , we have: \C(A, co)P{dco) = P{A n B). (6.68) B
C is called the conditional distribution of X given 3j and the mixed conditional distribution ofXgiven 5,is defined as the function Q{.,.) from defined by: Q{S,(D) = p[{co': X(co') e 5|3, ))(^),5 e y/.
y/xQ. Xo [0,l] (6.69)
The problem of the existence of regular conditional probability was solved by Loeve (1977) or Gikhman and Skorokhod (1980). For our goal, let us just say that this is the case if 3 is generated by a finite or countable family of random variables or if the space £" is a complete separable metric space. In the particular case of an /7-dimensional real r.v. X=(Xi, ...,X„),we can now introduce the very useful definition of the conditional distribution function ofX given 3j defined as follows:
= e({^'LY,(^')<x„...,X,(^')<xJ,^). Another useful definition concerns an extension of the concept of the independence of random variables to the definition of conditional independence of the n variables X^,,..,X^. For all (xi,...,x„) belonging toR"", we have the following identity: n
where of course we have: F^{x,M-P(X,<x,p,)
(6.72)
according to the definition (6.70) with n=\. Example 6.2 On the probability space(Q,3,P), let {X,Y)hQ di two-dimensional real r.v. whose d.f is given by:
Probability Tools
35
F{x,y) = P{X<xJ0 forAx:->0 with of course: fxi^)-\fix.y)dy.
(6.77)
R
Using formula (6.4), we thus obtain: P{y0}. (3.19)
50
Chapter 2
In reliability theory, the event {N = k} would mean that the (A:+l)th component introduced in the system would have an infinite lifetime! Also the probability distribution of A^is given by p(A^ = 0) = l-F(+oo), (3.20) P{N = l) = F(+oo){l-F(+oo)), (3.21) and in general, for A: G N : P{N = k) = (F(+oo))' (1 -F(+oo)). (3.22) Of course if F(+oo) = 1, we have, a.s., A^ = +oo.
(3.23)
In the case of a transient renewal process, we can write, using relation (3.22): E{N) = Y,k[F(+^)f{l-F(-^^)), As the function :; l-x
(3.24)
can be written for Ixl < 1 as a power series: '
A = Z^" >
(3-25)
which is analytical on (-1,+1) and thus derivable, we have
Writing relation (3.24) under the form 00
E{N) = F(+oo){l-F(+oo)).Y,k[F(+oD)f-'
(3.27)
we get, using relation (3.26):
So, it is possible to compute the mean of the total number of renewals very easily in the transient case. We can also give the distribution function of L. Indeed, we have: P{L
(4.6)
with
/"l(0 = / ( 0 ,
(4.7)
f^'\t)=jfit-x)f(x)dx,
(4.8)
fl"\t)=jf^"-'\t-x)f(x)dx.
(4.9)
0
From relation (4.5) or (4.6) we obtain the integral equation for h: h(t) = f(t) + f®h(tl with
(4.10)
f®h{t)=
(4.11)
jf(t - x)h{x)dx. 0
Or: f®h{t)^h®f{t), (4.12) (4.13) h{t) = f{t) + h®f{t). In fact, the renewal equation (4.2) is one particular case of the type of integral equations: (4.14) X{t) = G{t) + X^F{t\ where X is the unknown function, F and G being known measurable functions bounded on finite intervals and • the convolution product. Such an integral equation is said to be of renewal type. When G = F , we get the renewal equation. The study of these integral equations has a long history which includes contributions from Lotka (1940), Feller (1941), Smith (1954) and Cinlar (1969). Cinlar gave the two following propositions. Proposition 4,1 {Existence andunicity) The integral equation of renewal type (4.14) has one and only one solution, given by X(t)^R»G(t), (4.15) R being defined by relation (3.13).
54
Chapter 2
Proof (1) Existence In the second member of equation (4.14), we replace Xby expression (4.15): (4.16) G(t) + R^G^F(t), Using the commutative property of the convolution product, we get (4.17) G(t) + 7? • G • F(t) = (C/o(0 + R • F ( 0 ) • G(t), And by (3.14): (4.18) G(t) + i? • G • F(t) = R • G(t). So, function R • G(t) is a solution of the renewal-type equation (4.14). (2) Unicity Let X, and X2 be two solutions of equation (4.14), and let 7be defined by: r = X,-^2. (4.19) We then have 7 = 7#F(0, (4.20) and by induction, we get: Y^yF^''\ for all n>0. (4.21) As renewal function R can be defined by the function series (3.13), converging for all positive t, we know that: (4.22) limF^"^(0 = 0, for alW > 0 . n
Hence: lim Y • F^"> (0 = 0, for all / > 0,
(4.23)
n
and so by (4.21): 7(0 = 0 for all / > 0 .
(4.24)
It is also possible to study the asymptotical behaviour of solutions to renewaltype equations. The basic result is the so-called "key renewal theorem", proven by W.L. Smith (1954), and which is in fact mathematically equivalent to Blackwell's theorem (1948), given here as Corollary 4.2. A proof of the key renewal theorem using Blackwell's theorem can be found in Cinlar (1975b). Proposition 4.2 {Asymptotical behaviour. Key renewal theorem) (i) In the transient case, we have: limX(0 = i?(oo)G(oo)
(4.25)
provided the limit G(oo) = limG(0
(4.26)
exists.
Renewal Theory
55
(ii) In the case of recurrence, we have: \\mX{t) = - \G{x)dx,
(4.27)
provided that G is directly Riemann integrable on [0,oo), that F is not arithmetic, and supposing: m • E{X„)^\{\-F{x))dx.
(4.28)
Corollary 4.1 In the case of a recurrent renewal process with finite variance a^, we have: t \
lim ^(0
m^ -\-(J^
= \
2
'
(4-29)
Proof In the integral equation of renewal-type (4.14), let us choose X(t) = R(t)--^, m We shall compute G such that this integral equation is valid. We get: G(t) = X(t)-X^F(t) = R(t) - -^ - /? • F(t) + - \F(t - x)dx , m m^ From (3.14), we have for all / > 0: R(t) = Uo+H(t), so that:
(4.30)
(4.31) (4.32)
(4.33)
H(t) + -i- \F(t-x)dx. (4.34) G(t) = 1 + H(t)---F(t)-F• m m^ Using renewal equation (4.2) and posing x' = t-x in the integral of the second member, we get: G(t) = \
t m
1 V + - \F(x')dx\ mJ
(4.35)
And thus: G(t) = l
_1 \{l-F(x))dx. m mJ
(4.36)
Since 00
m= j{\-F(x))dx,
(4.37)
56
Chapter 2
we can also write: 1 °°
G(t) = ^ {(l-F(x))dx. (4.38) mf As a conclusion, the function G given by this last relation is the only function for which the integral equation of renewal-type has (4.30) as solution. Clearly, function G is monotonic non-increasing on [0,+oo) and:
JG(0^/ = -1 |
(4.39) j{l-F(x))dx dt, m, By permuting the order of integration (with the help of Fubini's theorem), we get: 1^ JG{t)dt = i . j x ( l - F{x))dx. (4.40) 0
^
0
But we have: 00
a'-^m'
= jx'dF{x)
(4.41)
0 00
=:-jxV(l-F(x)).
(4.42)
0
And integrating by parts, we find 00
a'+m'
=2Jx{l-F(x))dx.
(4.43)
0
Coming back to relation (4.40), we finally have
ym = ^ .
(4.44)
Corollary 4.1 is thus a direct consequence of result (ii) of Proposition 4,2. n Remark 4.1 1) From result (4.29), we immediately get an analogous result for the renewal function H, Indeed, we know, from relafion (3.14), that R(t) = H(t) + Uo(tl t>0. (4.45) Applying result (4.29), we get:
liJH(t)-L] = ^!^-l,
(4.46)
or
limfi/(0-^l = 4 ^ '
(4.47)
/-^ool^ ^' mj 2m^ 2) The two results (4.29) and (4.47) are often written under the following forms:
Renewal Theory 2
57 2
^(0 = - + ^ V - r - + o(i),
(4.48)
H(t)^-
+ ^ ^ - ^ + 0(l), (4.49) m 2m where 0(1) represents a function of ^ approaching zero as t approaches infinity. Corollary 4.2 In the case of a recurrent renewal process with finite mean m, we have: lim^^^l. r->cxD f rn
(4.50)
Proof Of course, if o"^ < oo the result is trivial by Corollary 4.1 if we divide both members of the relation by lim-. The result is also true if cr^ = oo .
n
Corollary 4.3 {BlackwelVs theorem) In the case of a recurrent process with finite mean m, we have, for every positive number T : \im( R(t)-R(t-T))
= ^.
(4.51)
Proof Let us consider the renewal-type equation (4.14) with function G defined as follows: G(/) = . '
^^'^^'
0,
(4.52)
T 0 : k(t) = h(t). (4.58) So, the probability defined above is given by h(t)dt and more generally by dH(t) with a precision error ofO(dt). This interpretation often simplifies the search for relations useful in renewal theory. 2) The variance of N(t) From Stein's lemma, we know that N(t) has, for all t, moments of any order. Also, let 6^2 (0 be the centred moment of order 2 of N(t): a,(t) = E{(N(t)y).
(4.59)
From results (3.5), we can write successively: a,(t)^j;^k'{U,-F)'F^^\t) 00
(4.60)
00
= ^k'F^'\t)-Y,k'F^'''\t) ^=1
k=\
00
00
(4.61)
= f,[v'-(y-iy]F^''\t)
(4.63)
= X(2v-l)F('-)(0
(4.64)
= 2|;(v-l)F"''(0 + i;F(^)(0.
(4.65)
Now if we compute H^^\t) by means of the relation: /
00
H^'\t) = \ Z^^'^t)
^
/^ °°
• ^F^''\t) V^''=i
we easily find:
^
,
(4.66)
Renewal Theory
59
if(2)(/)^£(K-l)F 0, A: e NQ ), is given by:
Renewal Theory
61
\k-\
/M(0 = ^ ^ e - ^
(5.14)
It is easy to see that /^^ is the density function of the sum of k independent, identically distributed, random variables, with the negative exponential distribution of parameter A. This meaning of the gamma distribution of parameters (A, A:) eases the search for the distribution of N(t). Indeed, the event {N(t) = n} is equivalent to the event of having nkov nk + 1, or ..., or nk + k-l occurrences of a Poisson process with parameter A on [0,^]. Let Np(t) represent the number of renewals for such a Poisson process. We have: ?(N(t) = n) = j;^?{N^(t)
= nk-^j)
.
(5.15)
j=0
Using result (3.41), we can now write:
^im->.)-e-'f^, =e
(5.16) y
^-^
In particular, for A: = 2, we get At (Atf ^i^N(t) = n) = e-''^^^^^\\ + ^^^\. (2n)\ 2n + \ To compute h{t) for A: = 2, we use result (5.4) with f{s)^X^\te-^^^'^'dt
(5 17)
(5.18)
(5.19)
0
{s + xy
(5.20)
This gives us: h{s) = ^_ , , , ^, . {s + Xf-X Since
*« 4-2(7721) •
(5.21)
P'22)
we can easily invert h{s) so that hit)-\-\e-'". As //(O) = 0, by integration we immediately get:
(5.23)
62
Chapter 2
This example shows that the difference H(t)
is here given by m
\{e-'"-l)
.
(5.25)
Thus, the error made in approximating H(t) by — does not approach zero as m « —> 00 . Indeed, we have At \ . (5.26) Hm H(ty 2 /->oo
By (4.49), this result means that, for a gamma distribution of parameters (/1,2), we have:
or CT^=^,
(5.28)
which is true, since from relations (5.38) of Chapter 1: m:^j,
(5.29)
^'=JT-
(5.30)
More generally, for the general T distribution of parameters (A,,n), we have that, if F^„ isthed.f:
^?i)(0 = ^...(0 ,
(5.31)
F(^^^j^) being the d.f of the sum of nk independent r.v. of negative exponential distribution with parameter A. So, we have for density function /^^/j^:
/(*/j)(0 = A„*(0 .
(5.32)
Or, by (5.14)
fi'll(t)-^^^e-- . This last result gives an explicit form of the renewal density:
which gives result (5.23) in the particular case of A: = 2. It can be shown (see Barlow & Proschan 1965) that:
(5.33)
Renewal Theory
63
ACO^IZ^-e^''^'-'),
(5.35)
where 6: = cos--- + zsin—- . k k And it can be shown by integrating (see Parzen (1962)) that:
(5.36)
Example 5.3 The "normal" distribution The support for any normal distribution is the entire real axis; this same support is not suitable in renewal theory. However, if / is the density function of a normal distribution with mean m and variance <j^, and if a is small with respect to m, then the truncated normal distribution on [0,oo], is very close to the true normal distribution. Indeed, the density function for the truncated normal distribution on [0,oo) of parameters {m,(j) is given by: 1 X
{u-mf
- [e~ 2-^ du , a •' " 0
(5.38)
where {u-mf
0.
(5.74)
Proof Let us suppose that F^ is the d.f corresponding to the renewal function aH . From the relation (5.58) pertinent to aH, we have: 1 + aH{s) Since, by (5.59): His) - - ^ , 1 - F{s) we can write:
F„W = - 2 ^ \-F{s)
(5.76)
L^ , 1 + - aF(s) l-F(s)
(5.77)
Renewal Theory
67
Or: FAs) = ^ .f%., • 1 - (1 - a)F{s) Since we assume that a < 1, we get: Q-.
(6.16)
m Proof By definition of NXt), we can write: t<S^yy Taking the expectation of both members, we get: t<E{S,,,^).
(6.17) (6.18)
Renewal Theory
69
Or, from Proposition 6.1: t<mR{t) , i.e., inequality (6.16)
(6.19) D
Remark 6.2 Later, we shalll see that the notion of the stationary renewal process will lead to an upper bound of R{t) for a large class of renewal processes.
7 ASYMPTOTICAL BEHAVIOUR OF THE iV(0-PROCESS This section will yield two important results concerning the counting process {N(t),t > 0) associated with a recurrent renewal process characterized by the d.f F. Proposition 7.1 (Strong law of large numbers). If m < CO, then, almost surely: hm—^ = ~. ^^°° t m
(7.1)
Proof For any sample path of a renewal process, we have: 5^(,) 0 V ^f^O)
h m - — = —. And as
J
is a subsequence of —, « > 0 , we also have: S„ (7.7)
70
Chapter 2
N(t) ^N(t) + l *^yv(/)+i
^;v(/)+i
N(t)
^^g^
^ ( 0 +1
we also get, by similar reasoning, that: hm—^-^ = —,
(7.9)
using result (7.4) again. Thus the proposition results from the double inequality (7.5), and from a classical a result on limits of sequences. Proposition 7.2 {Central limit theorem). If G^ < 00, then for ally E R : limP
^ N{f) -t/ ^ M00
/m^ Vy,
\y~
,
(7.10)
J
where O represents the standard normal df. Proof We know that successive hfetimes X^,X2,...,X„,.., of the renewal process are i.i.d. and with finite variance. We can thus apply the classical central limit theorem to the sequence of partial sums {T„,n>0), saying that:
limpf^^;^QO
(7.11)
By Definition 2.2, we can pass from the T„ -sequence to the N(t) -process: P{Nit)>n) = PiT„^
(7.17)
an application of the result (7.16) shows that the limit theory imposes that necessarily: l i m — = 1.
(7.18)
Consider now the probability in the first member of equality (7.12). We have successively: P{N(t)>n)
=P
N(t) - t/ m
m
FZ'~Ff.
(7.19)
mr' J
But • - t/ m
f
2 // mr'
nm-t •>Jm,
(7.20)
nm-t
(7.21)
cr^y7
\nm
If « -^ 00 and / -> 00, relations (7.14) and (7.18) imply that the quantity above approaches -y : limP
7^(0- t/ m > -y ^limP(Nit)>n), V\ /m
f
(7.22) (7.23)
This last result implies that limP t-^co
N(t) - t/ m
f
-2 / /
y
=
\-t
^N(t)+l
^.^(0
t
^N(,t)+l
X /. It is thus possible that the component in effect at time t is still the first one; in this case, the probabihty of the considered event is
F(x)-FiO
.
The other possibility is that there exists at least one renewal before or at t, If we suppose that the component in effect at time / was introduced into the system between u and u + du, we have to compute the probability that this component has its lifetime in interval(/-w, x). This result, combined with the probabilistic interpretation of the renewal function gives the integral of (8.24). Of course, if x < /, then there is necessarily at least one renewal before or at t, and n the probability of event ^A^(/)+I0, (8.29) F{t) = l-e-^\ and H(t) = At, (8.30) (8.31) R(t) = ;it + l, results (8.6), (8.7) and (8.8) of Proposition 8.1 yield f A f e-^^'-'^^du, x^ > max{x,/}, P(Z^(,),, >1, X,=y) = K{x,t-y ,ify oo : 1 °°
limi^(x,0 = ~ \[l-F(m^x{x,u})]du.
(8.66)
This last integral can be transformed using the following development and with the aid of integration by parts: 00
j[l-F{rmx{x,u})]du= 0
00
A;
j[l-F{x)]du+
j[\-F{u)]du
O
(8.67)
x 00
= [l-Fix)]x
+ [(l-F{u))uY^
+ judF(u). X
The fmiteness of the mean m implies that:
(8.68)
Renewal Theory
81
lim[l-F(w)]M = 0
(8.69)
(see for example Tucker (1962)). Consequently, the first two terms are opposite, and we get that: 00
CX3
j[l-F(m^ix{x,u})]du=
judF(u).
0
(8.70)
X
So, result (8.49) follows from (8.66). An immediate consequence is given by the following result.
n
Corollary 8.2 Ifm is finite, then (i) the limit distribution F^ of F^^^) and F^(^) has a density function f^ given by /.(") = ^ — ^ , m (ii) the limit distribution of F^^^^^^^ has a density distribution: uf{u) m provided F has f as density,
(8.71)
(8.72)
Proposition 8.5 (i) For all positive t,x,y,y < t, t-y
P{r{t)>x,
S(t)>y)=
JF'(t-u
+ x)dH(u) + F'(t + x) .
(8.73)
0
(ii) Ifm is finite, then l i m P ( K O > ^ , ^(0>y) /—>00
1 °°
=-
ff^
{[l-F(u)]du.
(8.74)
J
x+y
Proof (i) The figure below clearly shows the equivalence of the events {cD : r(^^^) > x,S(t,a)) > y} and {co: r(t-y)>x + y}.
(8.75) (8.76)
Chapter 2
82
y .
f-y
,
G^)
X
. r(t)
S(t)
^NOHl
rit-y) x+y Figure 8.4: excess and age events analysis Consequently, + y). P{r(0>x, S(t)>y) = P{rit-y)>x Using the relation (8.17), we obtain P{r(t)>x,
S(t)>y)
+ x)-
= l-F(t
j [\-F{t-u
+ x)]dHiu)
, (8.78)
[0./-;^]
That is, (8.73), since: F' =l-F. (ii) From the limit properties, we have > x + y) = \imP{y(t) > x + y), limP{/(t-y)
(8.79) (8.80)
An application of (8.77) and of result (8.48) of Proposition 8.4 yields: 1
\imP(r(t)>x,
S{t)>y)
x+y
= l--^ 1 °°
m
{[1-F(u)]du
(8.81)
\[l-F(u)]du,
(8.82)
since m=
\[\-F{u)]du.
D (8.83)
8.4 Example: The Poisson Process From (8.29), using (8.48) and (8.51), we easily get: X
/->00
^^
J
= \-e-
(8.84) (8.85)
Renewal Theory
83
limTv^^^^^^ (x) = A' jue-^"du
(8.86)
^-Au
= A'
2
(8.87)
i-Au-l)
(8.88) = l-e-'^(l + A;c). So, we see that F^, and consequently F^, have the same value as F . On the other hand, F^^ has a gamma distribution of second order. Proposition 8.6 explains this fact by showing that ^(0 asymptotically independent. Indeed result (8.74) of this proposition yields:
and /(t)
are
00
limP{r(t)>x,
S(t)>y)
= A \e-^"du
/—>oo
(8.89)
J
^e-'^'^'K Using (8.85) and results (8.48) of Proposition 8.4, we get \\mP{y{t) >x) = \\mP{S{t) >x) = e-^\ And so, from (8.90): limP(;'(0 > x,d{t) >y) = limP(KO > x)-\\mP{S{t) /—>Q0
^
/->00
> y).
(8.90) (8.91) (8.92)
r->00
From this result and from the fact that, for all /, ^;v(o.i=^(0 + KO, (8.93) Jf;y(^)+, is asymptotically the sum of two independent random variables having the same exponential distribution of parameter X. The distribution of X^ is indeed of type gamma and of second order. But more can be said by using results (8.74) and (8.39), which lead to: 00
P{ r(t) > X, S{t) >y)^X
\ e-^"du
(8.94)
x+y
= e-'^'^'\ (8.95) which is exactly the result (8.90). Consequently (8.92) is true for every t, and so S(t) and /(t) are independent for all t. This explains the fact that ^Ar(/)+i has a gamma distribution, as shown by (8.47)
8.5 Some Parameters Of The Limit Distributions Let us first consider the limiting of S(t) or /(t), say F^ . From Proposition 8.4, we know that for all % > 0 :
Chapter 2
84
Fs(x) = — m
(8.96)
{[\-F(u)]du.
The mean S^ of this distribution can be computed in the following way: C»
^
j[l-F^(x)]dx
00
00
=—
0
^
jdxj[l-F(u)]du. 0
(8.97)
X
Using Fubini's theorem, we have U
^ 0 0
S,
(8.98)
=-\[\-F{u)]du,\dx ^
0
0 1
00
= — Jw[l - F(u)]du,
(8.99)
We can compute this last integral by parts, so that -{\-F{u))
m
0
+
^ju'dF(u).
2m „
(8.100)
If we now suppose the finiteness of cr^, the variance related to d.f F, we know that the first term is zero, so that: ^ ^ ^ T2m. !;;!^'^^^^)
(8.101)
or (j^ +m^
(8.102)
2m that is: m
(J
2 2m' This last result shows, surprisingly, that S^ may be larger than m\ Indeed r, (J^ S^ > m 2m
m —
(8.103)
(8.104)
or S, >
mx,
S(t)>y)
= A je-'^^-^'^'^^du + e-'^^''^;
(8.119)
0
the computation of this integral gives as a final result: (8.120) P{r{t)>x, S(t)>y) = e-'^''^'\ Coming back to results (8.33) and (8.37), we can write that, for all positive t, X, y with y x, S(t)>y) = P{r(t)>x)-P{S(t)>y) . (8.121) As both members are equal to zero if ;^ > /, this last relation is true for all nonnegative x,y,t and shows that for all t, the r.v, /(t) and 5{t) are independent. The next proposition will show that this property of independence characterizes the Poisson process; that is, it is not only a necessary condition but also a sufficient one. Proposition 8.7 Suppose that for all t, the recurrence times y{t) and 5{t) are independent; then the renewal process is a Poisson process. Proof Taking the limits of both members of equality (8.121) with / - > o o , Propositions 8.4 and 8.6 yield, for all positive x and y: 00
. 0 0
00
- \[\-F{u)]du=-^\[\-F{u)]du'-\[\-F{u)]du. m ^ m^ m^ x-\-y
X
(8.122)
y
This relation shows that function K defined by K{x) = - \[\-F{u)]du
(8.123)
mJ X
must satisfy the following functional relation: K{x-\-y) = K{x)'K{y). Since, from (8.123): i^(0) = l, and since K is continuous on IR"^, AT must be exponential, i.e.
(8.124) (8.125)
Renewal Theory
87
K(x) = e-^\ Now, (8.123) can be rewritten as
(8.126)
e-'^ =-{[l-F(u)]du. mJ m
(8.127)
By derivation we get ^e-'' =l[l-F{x)l m so that F(x) = l-mAe-'\ It follows that F has a density f(x) = mA'e-'\ And as
(8.128)
(8.129) (8.130)
00
jf(x)dx
= l,
(8.131)
0
we must necessarily have: m=j , so that finally, from (8.129): F(x) = l-e-^'' .
(8.132) D (8.133)
9 DELAYED AND STATIONARY RENEWAL PROCESSES The notion of stationary renewal process is a particular case of a delayed renewal process, A delayed renewal process is a renewal process with the difference that the first r.v. X,, though still independent of the others, does not have the same distribution. More precisely, let (X„, /7 > 1) be a sequence of non-negative independent variables, G being the d.f of all other r.v. The corresponding sequence (r„, n>0), where T,=0 a.s., (9.1)
r„=x,+... + x „
(9.2)
is called a delayed renewal sequence or delayed renewal process, Clearly the basic definition of the "classical" renewal processes can be extended to the case of a delayed renewal process. For example if H^{t) represents the renewal function for a delayed renewal process, and if we pose the condition X, = X, then we have:
Chapter 2
HAt\X,=x) = \^ ' '\) (9.3) {\ + H{t -x) , x2 have F as d.f We thus have a particular delayed renewal process for which 1
^
G(x) = -{[\-F(u)]du,
x>0,
(9.8)
Such a process is called a stationary renewal process. The two main results concerning stationary renewal processes are related to renewal function H and to the distribution of the excess. Proposition 9.1 For every stationary renewal process characterized by d.f. (G,F) with a finite mean mforfi we have for all t\ HM--,
(9.9)
m H^ being the renewal function of the stationary renewal process. Proof If we take the Laplace Stieltjes transform of both members of (9.7), we get: (9.10) HAs)^G{s) + H{s)'G{sl with the convention that:
89
Renewal Theory
K(s) = je-^'dKix).
(9.11)
From the "classical" renewal equation (4.4), we deduce that: His) = F(s) + F(s)-H(s), or
(9.12)
(9.13) From (9.8) and the properties of the L-S transform, we obtain: G(s) =
1m J
-\e-'"[l-Fiu)]du
(9.14)
m,
1
-
je-'"F(u)du
(9.15)
An integration by parts leads to: (9.16) G(.) = . l ^ - ^ ( ^ ) m s Replacing H{s) and G(s) by expression (9.13) and (9.16) in equality (9.10), we find: 1 1 - F(s) (9.17) His) = 1 \-F{s), m Or, after simplification: ms But we know that [e-'^dx = -
(9.18)
(9.19)
It follows thus that the inverse Laplace Sfieltjes transform of (9.18) yields: m
u (9.20)
Proposition 9.1 has an interesting meaning. Indeed the value of H^ is asymptotically true for every renewal process, but here in the case of a stationary renewal process, the asymptotical expression is true for every t. Now, let /,(t) be excess at time t for the stationary renewal process and: F;^,„(X) = 1-FJX).
(9.21)
90
Chapter 2
Proposition 9.2 (i) For every delayed renewal process characterized by the distribution functions (G,F), with a finite mean mfor F, we have for all t: P{r(t)<x)
= G(t + x)-
j[l-F(t
+ x-u)]dH,(u)
.
(9.22)
[0,/]
(ii) If moreover, the renewal process is stationary, then, for all t: P{r(t) <x) = -j[l-F(u)]du
.
(9.23)
m Q
Proof (i) Conditioning with respect to the value of X,, (X.-t, t<X., {y{t-X,), t>X,. Since, for / > x: P(X,>y\X,>t)^^j^,
(9.25)
we deduce from (9.24) that: P{rs(0>x) = [ 1 - G ( 0 ] P ( ^ , -t>x\X,>t)+
' jP{r(t-y)
(9.26) > x)dG(y)
0
= [l-G(t)]-^-=^^^^+]pir(t-y)>x)dG(y).
(9.27)
Or: /V!,) (x) - 1 - G(t + x)+ | F ; , . , ) (x)dG(y).
(9.28)
0
This equahty expresses /Y^(^) as a function of F/(/). This last function is known from Proposition 8.2, and can be written (by (8.17)) under the form: 77f(,)(x) = l - F ( / + x)+ j[\-F(t-u-x)]dH(u). (9.29) [0./]
To simplify notation, let us write l-Fit + x) = FM (9.29) thus takes the form: . F;^,„(x) = F,(t) + F,(t-u)*H(t) Coming back now to (9.28), we obtain: F,:„(x) ^\-G(t + x) + F,» G(t) + F,»H* G(/) or
(9.30) (9.31) (9.32)
Renewal Theory
91
F^^^,)(x) = l - G ( / + x) + F, • [ G + / / # G ] ( 0
.
(9.33)
The last term may be written using renewal function H, if we use relation (9.7): F;^,,,{X)
= 1 - G ( / + X) + F , • HAtl
(9.34)
which is what we needed to prove. (ii) By Proposition 9.1, (9.34) becomes, in the stationary case: F^',,„{x) ^\-G{t
+ x)+ \FXt-u)— 0
.
(9.35)
^
The change of variable u^ = t -u yields: F;^^,^{X) =
\-G{t + x) + - \F,{u)du,
(9.36)
As function G is now defined by (9.8), we also have: \[\-F{u)]du
F;(.(X) = 1 - -
+ '^ \[\-F{u)]du,
0
(9.37)
X
By the additive property of the integral related to the integration domain, we get: X
F^^^,.,{x) = \-\[\-F{u)]du,
(9.38)
0
or (9.23).
D
Part (ii) of Proposition 9.2 leads to the same conclusion as the preceding theorem: in the stationary case, the asymptotical distribution of the excess y{t) is the exact distribution for all t. This result implies some interesting corollaries. Corollary 9.1 For every stationary renewal process, we have, for all t: (i)Fs\„(x)::^-][\-Fiu)]du, (ii)P{rit)>x,S(t)>y)^-
(9.39) \[\-F{u)]du,
(9.40)
x+y
(iii) P{X^,,^,, <x) = - judFiu),
(9.41)
Proof Results (i) and (ii) follow directly from relations (8.50), (8.77), and result (9.22) from Proposition 9.2. For (iii), we use relation (8.4) showing that X^^,>^+] is the sum of the two r.v. S(t) and y(t). From Proposition 9.2 and from (9.39), we know that the
92
Chapter 2
distribution of the two-dimensional r.v. {y,S) is independent of /, and consequently that the same is true for the sum y{t) + S{t). For r > X,, we have: '^N{t)+\
— ^N{t-X^)+\'>
(9.42)
so that: P{X^(,)„ <x\t>X,) = P{Z;v(,-^,)^i ^x\t>X,). Letting / tend to +00, we get: ?{X,,„,,<x)
(9.43)
= -\[\-F{u)]du,
(9.44)
because the event {co',t>X^{co)] is asymptotically of probability 1. As the distribution of Xj^^^.^^^ is independent of t, result (9.41)is equivalent to (9.44). n Remark 9.1 As for any two-dimensional random variable {X,Y), we have (9.45) P{X<x, Yx)'P{Y>y)^-P{X>x, Y > y), We can apply this elementary result to the joined distribution of {y{t),d{t)), given by (9.40). Doing so, it is easy to see that
\\r.j[l-F(w)]Jw „ , . . , + - r ^^[\-F{u)]du, ' P{r{t)<x,d{t)y,
=
(9.46) -\{\-F{u)\du^-
\\\-F{u)'\du,
\ixl, Lemma 10.2 Suppose that: A = hL0, then (l + Ay-"^ <e^ if p>q.
(10.7) (10.8)
Remark 10.1 From Lemmas 10.1 and 10.2, it results that: \^,\T^,
\fn,
(10.32) D (10.33)
//-»0
The obtained results show that it is possible to obtain the discrete time renewal equation by means of the discretization of the related continuous time, and that starting from the discrete one it is also possible to get the continuous one.
97
Renewal Theory
10.3 Numerical Approximation of the Poisson Process For the Poisson process for which F is an exponential distribution of parameter A we know that the renewal equation is: H(t) = ;it. (10.34) To evaluate our numerical method, we use the two rectangle formulas:
H(kh) = 1 - e-'"" + hXY, H{kh - rh) -Arhe
(10.35)
k-\
H{kh) = \-e-'''
+hXY,H{kh-Th)
e
Xrh
(10.36)
Taking into account the Mc Laurin expansion: (10.37) in both cases we get: H{kh) = ?ikh, that is the real value in our case. Here, relations (10.24) and (10.25) respectively become:
(10.38)
k
H(kh) = 1 - e - ' ' ' + J^H(kh -rh)
(1 - e - ' ' ' -1 + e-'^'-'^'),
(10.39)
H{kh) - 1 - e-^"' + ^7/(/t/? -vh)
(1 - e"^*^""' - 1 + e'^''
(10.40)
j-=0
And so k
H(kh) = 1 - e-^** + (e^* - 1 ) 2 ] H(kh - rh) f
(10.41)
e-''',
k-\
H{kh) = 1 - e-^'' + (1 - e-^'' )J^H(kh - Th)e-
(10.42)
Table 10.1 gives the results obtained by relations (10.35), (10.36), (10.41) and (10.42) respectively in columns 2, 4, 5, 6 with h = A, /I = 2; in column 3, we give values obtained by computing the value of the density function in the centre of the discretization intervals. (10.36) (10.41) (10.42) real value
beriods (10.35)
0 1 2 3 4 5 6
0
0
0
0
0
0.18127 0.35936 0.53433 0.70624 0.87513 1.04107
0.18127 0.36248 0.54364 0.72475 0.9058 1.0868
0.22659 0.45848 0.6958 0.93868 1.18724 1.44163
0.18127 0.36254 0.54381 0.72508 0.90635 1.08762
0.2214 0.44281 0.66421 0.88561 1.10701 1.32842
0 0.2 0.4 0.6 0.8 1 1.2
Chapter 2
98 1.20409 1.26774 1.70197 1 8 ^1.36426 ' 1.44862 1.9684
1.4 1.6 1.8 9 1.52163 2 10 1.67623 2.2 11 1.82813 2.4 12 1.97736 2.6 13 2.12398 14 2.26803 2.8 15 2.40956 3 16 2.54861 3.2 3.4 17 2.68522 18 2.81943 3.6 3.8 19 2.9513 4 20 3.08085 21 3.20813 4.2 4.4 22 3.33319 23 3.45605 4.6 ' 24 3.57676 J 4.8 3.69535 5 25 1 Table 10.1: comparison between numerical and process data 1.26888 1.45015 1.62946 2.24107 1.63142 1.81023 2.52013 1.81269 1.99096 2.80572 1.99396 2.17162 3.098 2.17523 2.35224 3.39712 2.3565 2.5328 3.70325 2.53777 2.7133 4.01654 2.71904 2.89375 4.33717 2.90031 3.07415 4.6653 3.08158 3.25449 5.00112 3.26285 3.43478 5.3448 3.44412 3.61501 5.69653 3.62538 3.79519 6.05649 3.80665 3.97531 6.42488 3.98792 4.15538 6.80189 4.16919 4.3354 j7.18774, 4.35046 4.51536 7.58261 4.53173
1.54982 I 1.77122 1.99262 2.21403 2.43543 2.65683 2.87824 3.09964 3.32104 3.54244 3.76385 3.98525 4.20665 4.42806 4.64946 4.87086 5.09226 5.31367 5.53507
10.4 A Real Life Example in Motor Car Accidents. 10.4.1 Data Description. In this paragraph the renewal equation is applied to an actuarial real data application. We wish to outline that the renewal process can be applied in a very general case utilising data from statistical observations. The formula that we shall use in this case is given by relation (10.28). We shall apply the renewal theory in the case of motorcar accidents. In a motorcar insurance contract each time that the insured causes an accident, the insurance company will pay the damage done. It is possible to think that once the insured car is repaired all the conditions are renewed as at the beginning of the contract and the renewal theory can be applied. We have the raw data of accidents that an insurance company observed over roughly a period of 50 years. From these data it is possible to construct the discrete time increasing d.f of the renewal time of the motorcar accident. As a whole we had the data of 156,428 insured persons, and among them 22,395 had at least one accident during the time of subscription. In the file there were no data concerning the insurance subscription dates. We construct two vectors. In the first one, we count the number of insured persons that had the second accident in 1 year, in 2 years,... from the first. In the
99
Renewal Theory
second we count the number of insured persons that had the third accident in 1 year, in 2 years,... from the second. More precisely, at the beginning, we set all the elements of the vectors equal to 0. We add 1 at the nth element of the first vector when the second accident is verified after n-1 and before n years by the first one. In the same way, we add 1 at the nth element of the second vector when the third accident is verified after n-l and before n years by the second one. At the end of this procedure, the nth element of vectors will give the number of accidents during the nth years from the previous one, respectively the second after the first and the third after the second. The results obtained in this way are reported in Table 10.2. For example, the number 1,576 in the third row of this table we have represented the number of the second accidents that took place after two years and before the third year by the first accident and similarly, the number 226 represents the number of the third accidents that took place after two years and before the third year by the second accident.
1 # years
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
1153^"^288-3 695 1576 1549 1344
928 619 386 278 189 139 101 77 40 36 15 11 11 10 7 14 6 8 7 7 2
126 226 224 172 176 138 107 86 69 61 33 21 15 16 7 3 4 3 2 1
1-2 + 2-3
241 821 1 802 1773 1516 1 104
757 493 364 258 200 134 98 55 52 22 14 15 13 9 15 6 8 7 7
2
1
Chapter 2
100 27 28 29 30 31 32 33 34 35 36 37
4 3 5 3 2 1 1 1
total
8228
4 3 5 3 2 1 1 1 1578
1
9 806
Table 10.2: number of accidents taking place in one year In Table 10.3, we report the yearly mean number and the variance related to the three distributions of Table 10.2.
1-2 + 2-3 1-2 2-3 mean 5.8676 5.4293 5.3453 1 variance 10.8680 11.7207 11.0421 Tab le 10.3: yeairly m e a n and variainee of accid
In Table 10.4, we report the distributions of the frequencies of the three vectors given in Table 10.2. The columns are obtained by dividing the number of each column by the total number of considered accidents in the vector (last element). Though mean and variance are unequal, the three histograms that can be obtained by Table 10.4 have a shape similar to that of the Poisson process. # years
1-2
2-3
1 2 3 4 5 6 7 8 9 10 11 12 13 14
0.018595 0.084468 0.191541 0.188260 0.163345 0.112786 0.075231 0.046913 0.033787 0.022970 0.016894 0.012275 0.009358 0.004861
0.055767 0.079848 0.143219 0.141952 0.108999 0.111534 0.087452 0.067807 0.054499 0.043726 0.038657 0,020913 0.013308 0.009506
1-2 + 2-3 0.02457679 0.08372425 0.183765042 0.180807669 0.154599225 0.112584132 0.077197634 0.050275342 0.037120131 0.026310422 0.020395676 0.013665103 0.009993881 0.005608811 1
Renewal Theory
101
1 16 ^^0.004375 0.001823 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
0.010139 0.004436 0.001337 0.001901 0.001337 0.002535 0.001215 0.001901 0.000851 0.001267 0.001702 0.000634 0.000729 0.000972 0.000851 0.000851 0.000243
0 0.000486 0.000365 0.000608 0.000365 1 0.000243 0.000122
0 0.000407914 0.000305935 0.000509892 0.000305935 0.000203957 0.000101978
0
0
0.000122
0.000101978
0
0
1 34
35 36
1 0.005302876 0.002243524 0.001427697 0.001529676 0.001325719 0.000917805 0.001529676 0.00061187 0.000815827 0.000713849 0.000713849 0.000203957
0.000101978 1 37 1 0.000122 1 TabI e 10.4: frequency dis tribution
The observation of data leads to the following considerations: i) all three distributions have the same shape, ii) these distributions have the Poisson shape but the variance is about the double of the mean, iii) the mean and the variance of the first two columns of Table 10.2 are similar. In this light, we can say that the renewal hypothesis is acceptable because of the similar shape and the parameters of the first two distributions. It is possible to suppose that after an accident the process is renewed and the behaviour of the insured people will be the same. To get more reliable data we decided to put together the observations of the first and second distributions, obtaining the third column of each table. 10.4.2 The Result Distribution Now we shall consider the frequencies of the last column of Table 10.4 as the "probability" that a new accident will happen after n-l and before n years from the previous one. The nth result can be considered as an evaluation of the "probability" that there will be an accident between the (nA)th and the ^7th years,
Chapter 2
102
given that at least one accident took place, as already said above, we do not have data related to the dates of first contract subscription. We use the time interval between two subsequent accidents. Values shown in the second column are "conditional probabilities" to the event of having at least one accident as described before. To apply the renewal model we need to construct the d.f that gives the probability of having an accident within n years. The "probability" of having at least one accident is obtained from the raw data. It is obtained by dividing the number of persons that had at least one accident (22,395) by the total number of the insured persons (156,428). 9,806 accidents are considered. They represent the number of accidents that were caused after another accident. We define the following events for an insured person: J?: cause at least two accidents during the observed period, (By{, cause another accident after the {nA)\h and before the nth year from the previous one, C cause at least one accident over the full subscription time (probability:22,395/156,428 (=0.14316)). We wish to estimate the following probabilities for the interarrival of successive accidents: (Dyi'. cause an accident after the («-l)th and before the ^th yQaxfrom the contract subscription or from the previous one. The nXh element of the second column Table 10.4 that is equal to the fourth of Table 10.3 gives the following value: n^lM (10.43) but we are interested in: P[0„/C]. (10.44) In the data, as already specified, we don't have the contract subscription date. We are interested to know the probabihties P[©^]. Under the renewal assumption, we can say that: Then by means of Bayes' theorem, it is possible to reconstruct the "probabilities" P[(D^] to have the first accident after one year, two years and so on. These results are reported in the third column of Table 10.5. year Cond. Freq Frequencies Cum. Freq 0.003519 1 0.024577 0.003519 2 0.083724 0.011986 0.015505 0.041814 0.183765 0.026309 3
mean numb. 0.003519 0.01551738 0.04191078
Renewal Theory
103
0.067699 1 0.06812506 4 1 0.180808 0.025885 5 0.09107341 0.089832 0.154599 0.022133 0.10866922 6 0.10595 0.112584 0.016118 0.12175334 0.117002 7 0.077198 0.011052 8 0.13130704 0.1242 0.050275 0.007198 9 0.13903198 0.03712 0.005314 0.129514 0.14506757 0.133281 10 0.02631 0.003767 0.00292 0.15000748 11 0.020396 0.136201 0.15371498 17 0.013665 0.001956 0.138157 0.15663962 0.009994 0.001431 0.139588 n 0.15870604 0.005609 0.000803 14 0.140391 0.16051876 15 0.005303 0.000759 0.14115 16 0.16170875 0.002244 0.000321 0.141471 0.16261899 17 0.001428 0.000204 0.141676 0.16340732 18 0.00153 0.000219 0.141895 19 0.16404851 0.001326 0.00019 0.142085 20 0.16453603 0.142216 0.000918 0.000131 21 0.16503802 0.00153 0.000219 0.142435 22 0.000612 8.76E-05 0.16535298 0.142523 23 0.16565263 0.000816 0.000117 0.142639 24 0.142742 0.16590744 0.000714 0.000102 25 0.142844 0.16613383 0.000714 0.000102 0.16626641 0.000204 2.92E-05 26 0.142873 27 0 0.0 0.142873 0.16635287 28 0.142931 0.000408 5.84E-05 0.16648298 29 0.000306 4.38E-05 0.16658543 0.142975 30 7.3E-05 0.16670596 0.00051 0.143048 31 0.143092 0.000306 4.38E-05 0.16679003 32 0.143121 0.000204 2.92E-05 0.16685396 33 0.000102 1.46E-05 0.16690042 0.143136 34 0 0.0 0.16692826 0.143136 0.000102 1.46E-05 35 0.14315 0.16696609 36 0.0 0 0.14315 0.16698588 37 1 0.000102 1 1.46E-05 1 0.143165 1 0.16701685 1 Table 10.5: the accidents renewal results In the fourth column of this last table the ' 'increasing d.f." obtained by the elements of the third column is reported. In this case the nth element represents the "probability" of having the first accident within n years. These data represent the d.f that we need in the discrete time renewal equation to evaluate the mean number of accidents that an insured person will have within 1, 2,..., n, years. These results are obtained by solving the discrete time renewal equation (36) and are reported in the last column of Table 10.4.
104
Chapter 2
Here our results show that the mean number of accidents is low. Let us recall that we used real data obtained by an insurance company and moreover this is in keeping with the value of the probability of having at least one accident during the full subscription time being, as we give above, 0.14316. If anyone thinks that the results are low in the light of his experience we will be glad to have other data and evaluate them by means of our model.
Chapter 3 MARKOV CHAINS This chapter presents briefly some fundamental results concerning the theory of Markov chains with a finite number of states. These results will be used in the following chapter. We will use the usual terminology introduced by Chung (1960) and Parzen (1962).
1 DEFINITIONS Let us consider an economic or physical system S with m possible states, represented by the set / : (1.1) I = {1 2,..., m } . Let the system S evolve randomly in discrete time (^ = 0, 1, 2,..., n,...), and let /„ be the r.v. representing the state of the system S at time n. Definition 1.1 The random sequence (J„,
/? G N ) w a Markov chain iff for all
7o J 7i J • • • J 7« ^ -^ •
P{Jn
= Jn I "^0 = Jo,
A = JW",
Jn-\ = Jn-\ ) = P{Jn=
Jn I ^n-\
= Jn-l ) ( 1 - 2 )
(provided this probability has meaning) Definition 1.2 A Markov chain {J„, n>0) is homogeneous iff the probabilities (1.2) do not depend on n and is non-homogeneous in the other cases. For the moment, we will only consider the homogeneous case for which we write: P{Jn=j\Jn-^-i)-P,, (1.3) and we introduce the matrix P defined as: The elements of the matrix P have the following properties: (i) p^. >0, for all/, y e / ,
(1.5)
( i i ) ^ ^ . , =1, for all / G / .
(1.6)
ye/
A matrix P satisfying these two conditions is called a Markov matrix or a transition matrix.
106
Chapters
To every transition matrix, we can associate a transition graph where vertices represent states. There exists an arc between vertices / andy iff p^j > 0.
Figure 1.1: py > 0 To fully define the evolution of a Markov chain, it is also necessary to fix an initial distribution for state jQ,i.Q. a vector P = iPl,'",Pm), such that: Pi>0, iel,
(1-7)
ZA=1.
(1.9)
(1.8)
iel
For all /,/?, represents the m///a//^roZ?aZ>///(y of starting from /: p,=P{J,=i), (1.10) For the rest of this chapter we will consider homogeneous Markov chains as being characterized by the couple ( p , P ) . If J„ = i a.s., that is if the system starts with probabihty equal to 1 from state /, then the components of vector p will be: Pj=S,. (1.11) We now introduce the transition probabilities of order pjf^, defined as: Plr'=PU.n=J\J.=i)' (1.12) From the Markov property (1.2), it is clear that conditioning with respect to J^^^, we get PIP=TP!^P'9-
0-13)
k
Using the following matrix notation:
p^^'-[pr]>
(1-14)
we find that relation (1.13) is equivalent to P^'^ = P ^ Using induction, it is easy to prove that if
(1.15)
P«=[/7W],
(1.16)
Markov Chain
107
then we obtain for all « > 1: P^") = P " . (1.17) Note that (1.17) implies that the transition probability matrix in n steps is equal to the nih power of the matrix P. For the marginal distributions related to J„, we define for / G / and n>0: p,{n) = P{J„=i). (1.18) These probabilities may be computed as follows:
Piin) = Y^Pjpf,
iel.
(1.19)
J
If we write: p^J^^=Sj,or¥ 0. Tij is said to be the hitting time of the singleton {j}, starting from state / at time 0. Supposing: fi"^=P{Ty=n\Jo=i), neNo (2.5) and f, =P{T,j 0: f
«^2.
Let: m,=E{T,\J,=i), with the possibility of an infinite mean. The value of my is given by:
(2.13) (2.14)
Markov Chain
111
m,=Y.nf^''-^{\-f,).^'
(2.15)
If / = 7 , then w,y is called the first passage time mean or the mean recurrence time of state /. For every 7, we define the sequence of successive return times to state j {^n^\ « > a ) as follows: ro^^)=0, rJ^)=sup{^ENo,
(2.16) /:>ri^?, / . ^ 7 , r i ^ ? < v < ^ } ,
n>0,
(2.17)
k
Using the Markov property and supposing JQ = j , the sequence of return times to state j is a renewal sequence with the r.v. ry^-r^l}, n>l (2.18) distributed according to TJJ . If JQ = /, / ^ j , then (/;^^\ n>0) is a general renewal sequence. In this case: ry'=T,, (2.19) and ry^-r^ll-T^, n>L (2.20) This shows that a Markov chain contains many embedded renewal processes. These processes are used to define the next classification of states. Definition 2.9 A state i is said to be transient (recurrent) if the renewal process associated with its successive return times to i is transient (recurrent). A direct consequence of this definition is that: / transient fa \) subsets C,,...,Q forming a partition, such that each subset Q is one and only one of the following types: (i) an essential recurrent positive closed set, ^*^ Using the following conventions: 00 + a = 00, a G R, oo • a = oo, (a> 0), and in this particular case, oo • 0 = 0 ,
112
Chapters
(ii) an inessential transient non-closed set. Remark 2.2 (1) If an inessential class reduces to a singleton {/}, there are two possibilities: a) There exists a positive integer N such that: 0
(4.1)
kel
or kel
As / j and state / is recurrent by assumption, we know by Proposition 2.1 that state y is also recurrent. Thus, by (3.9) gjj = 1. From (4.2), it follows that for all ^ > 0 : P^l\l-g,) =0, (4.3) As j > i, there exists a positive integer N such that: PI/"^ > 0. (4.4) Using relation (4.3) for k = i ,WQ get: ^,=1. (4.5) Hence, using relation (3.8): fy=l. (4.6) (ii) Since C(i) is a recurrent class, it is closed. Thus if j ^ C(i), then / does not lead toy and so: fij=0. D (4.7) Proposition 4.2 Let T be the set of all transient states of I, and let C be a recurrent class. For all 7, k e C, fo=ALabeling this common value asfc, linear system: f,c-t.P*fKc+Y,P.,, i^T. keT
the probabilities {f^c
i^T)
(4.8) satisfy the (4.9)
keC
Proof From relation (3.5), we have: fy=j;^P^,P{N,(^)>0\J,=k),
(4.10)
kel
or f.-ZP'^fv kel
Using the preceding proposition, we obtain:
(4-11)
Markov Chain
115
A C = Z A . A C + I;^.*, keT
i^T.
D(4.12)
keC
Remark 4.1 Parzen (1962) proved that under the assumption of Proposition 4.2, the linear system (4.9) has a unique solution. This shows, in particular, that if there is only one irreducible class C , then for all i eT : f,c=l(4.13) Definition 4.1 The probability f^c introduced in Proposition 4.2 is called absorption probability in class C, starting from state i. If class C is recurrent: ri i f / e C , fic=\ [0 if / is recurrent, / ^ C.
(4.14)
5 ASYMPTOTIC BEHAVIOUR Consider an irreducible aperiodic Markov chain which is positive recurrent. Suppose that the following limit exists: \impj(n) = /rj, jel (5.1) Starting with JQ = i. The relation (5.2) kel
becomes: Pij
~ ZjPik Pkj^
(5.3)
because Pji.n)-Pt'Since the state space / is finite, we obtain from (5.1) and (5.3):
(5.4) (5.5)
kel
and from (5.4): (5.6) iel
The result: lim;.«=;r,
(5.7)
is called an ergodic result, since the value of the limit in (5.7) is independent of the initial state /. From result (5.7) and (1.19), we see that for any initial distribution p:
116
Chapter 3
\imp,{n) = \^rnY^PjPf,
(5.8)
J
= Z/'y^-
(5.9)
SO that: lim/?,(f7) = ;r,.
(5.10)
/7->00
This shows that the asymptotic behaviour of a Markov chain is given by the existence (or non-existence) of the Umit of the matrix P". A standard result concerning the asymptotic behaviour of P" is given in the next proposition. The proof can be found in Chung (1960), Parzen (1962) or Feller (1957). Proposition 5.1 For any aperiodic Markov chain of transition matrix P and having a finite number of states, we have: a) if state j is recurrent (necessarily positive), then (i)
i^C{j)^\imp\p=—,
(5.11) "JJ
(ii) / recurrent and ^ C(j) => \impj"^ =0,
(5.12)
f
(iii) i transient limpj"^ ^:ii£Ul^
(5.13)
b) Ifj is transient, then for all i e I: limpj"^ =0.
(5.14)
Remark 5.1 1) Result (ii) of part a) is trivial since in this case: p^f' - 0 for all positive n, 2) Result (5.14) in part b) is identical to result (2.27) of Proposition 2.2. From Proposition 5.1, the following corollaries can be deduced. Corollary 5.1 {Irreducible case) If the Markov chain of transition matrix P is irreducible, then for all i,j e / : ]hnpf=7tj,
(5.15)
with 7tj= — .
(5.16)
It follows that for all j : 7tj>^.
(5.17)
Markov Chain
117
If we use Remark 4.1 in the particular case where we have only one recurrent class and where the states are transient (the so called uni-reducible case), then we have the following corollary: Corollary 5.2 (Uni-reducible case) If the Markov chain of transition matrix P has one essential class C {necessarily recurrent positive) and T as transient set, then we have: (5.18) (i) for all /, 7 e C : \\mp\"^ = TT:, with [TTJ,] e C] being the unique solution of the system: (5.19) ieC
(5.20) (ii) For all j
eT\
\\mp\f^ = O f o r a l l / G / .
(5.21)
(iii) For all j eC : limpjf ^TTj for a l l / s T .
(5.22)
Remark 5.2 Relations (5.19) and (5.20) are true because the set C of recurrent states can be seen as a Markov sub-chain of the initial chain. If the £ transient states belong to the set {!,.,.,£}, using a permutation of the set / , if necessary, then the matrix P takes the following form: 1 ••• £ £ + 1 ••• m Pn
Pi2
(5.23) £+ 1 O m
This proves that the sub-matrix P22 is itself a Markov transition matrix. Let us now consider a Markov chain of matrix P. The general case is given by a partition of/:
/ = ruc,U---UQ,
(5.24)
where T is the set of transient states and Ci,...,Q the r positive recurrent classes.
Chapter 3
118
By reorganizing the order of the elements of / , we can always suppose that T = {\,...,1), (5.25) (5.26) C,={£ + \,...,i + v,}, C,={l + v,+\,...,l + v,+v,}, (5.27) r-\
(5.28)
c = U+Zv,+i,-.., m j=\
where Vj is the number of elements in Cy, (y = 1,..., r ) and
^+Z-. y=i
(5.29)
m.
This results from the following block partition of matrix P: 0
0
0
P=
0
0 P
0
(5.30)
V2XV2
0 0 0 where, for j = 1,..., r : Vf^i is the transition sub-matrix for T, P<x^^ is the transition sub-matrix from T to Cj, Pyjxvj is the transition sub-matrix for the class Cj. From Proposition 5.1, we have the following corollary: Corollary 5.3 For a general Markov chain of matrix P, given by (5.30), we have: (5.31) (i) For all i e / and all j eT: limpf"^ = 0. in) For all j ^ C^
(v = l,..., r ) :
{KJ
if/eQ,
\imp\^ =\Q
if/GC,.
A7->00
v'^v,
(5.32)
J
[fi,cXj ifi^T. Moreover, for all v = 1,..., r : Z00
or 5,-lim£;.WS,.
(7.4)
This value can be found by applying Corollary 5.3 for / = 1,..., m . Numerical example Using real-Hfe data for sihcosis, Yntema (1962) began with the following intermediate degrees of disability:
s,----10% Sj = 30% ^3 - 50% ^4 = 70% ^5 = 100%
Using real observations recorded in the Netherlands, he considered the following transition matrix P: '.90 .10 0 0 0^ 0 .95 .05 0 0 P- 0 0 .90 .05 .05 ; (7.5) 0 0 0 .90 .10 0 0 .05 .05 .90 the transition graph associated with the matrix (7.5) is given in Figure 7.1: This immediately shows that: (i) all states are aperiodic, (ii) the set {5'3,iS'4,5'5} is an essential class (positive recurrent), (iii) the singleton {1} and {2} are two inessential transient classes. Hence a uni-reducible Markov chain can be associated with matrix P. We can thus apply Corollary 5.2. It follows from relation (7.4) that:
Chapter 3
124
(7.6) 7=3
where {TT^^.TT^.TI^) is the unique solution of the linear system: K-i
=
TC.
-
.05 • TT^ +
TTA
=
.05 • K^ +
1
.9'K2,
==
^3
+
O'TT^
+
.05-/Ts,
.9-7r,
+
.05-;r5,
.05 • K^ +
.9-7t„
+
n^.
+
^4
(7.7)
Figure 7.1 The solution is: 2 3 n.^-. Therefore:
(7.8)
Markov Chain
125
'^'^(|50 + |^0) (1.2) supposing X,=Q,a.s. (1.3) where the sequence (J^,/7>0) gives the successive states oi S in time and the sequence {X^,n> 0) gives the successive sojourn times. More precisely, X^ is the time spent by S in state J^-x (n>0). Times at which transitions occur are given by the sequence (T^,n > 0) where: T,=0,T,^X„...,T„^f^X^
(1.4)
and so X„=T„-T„_„n>l.
(1.5)
2 SEMI-MARKOV AND EXTENDED SEMI-MARKOV CHAINS On the complete probability space (Q,3,P), the stochastic dynamic evolution of the considered (J-X) process will be determined by the following assumptions: P(Xo==0)=l,a.s.,
146
P(Jo=i)=Ph /=l...,m with ^p.
Chapter 4
=1,
for all n>Oj'=l, ..,,m, WQ have: P(J^ =J,X^ <x\(J,,X,),k = 0,„.,n-l) = Q,^Jx),a.s.
(2.1) (2.2)
where any function Qy (ij=l, ...,w) is a non-decreasing real function null on M such that if j9 = limQ (x), ij'el, (2.3) then: m
t.p,-\.iel.
(2.4)
7=1
With matrix notation, we will write: Q = ta]' P = h ] ( = Q ( ^ ) ) . V = (P^,-,PJ• This leads to the following definitions.
(2.5)
Definition 2.1 Every matrix mxmQ of non-decreasing functions null on M satisfying properties (2.3) and (2.4) is called a semi-Markov matrix or a semiMarkov kernel Definition 2.2 Every couple (p,Q) where Q is a semi-Markov kernel and p a vector of initial probabilities defines a positive {J,X) process {J,X) ={{Jn.Xn), n>0) with / x R ^ as state space, also called a semi-Markov chain (in short a SMC), Sometimes, it is useful that the random variables X^,n>Q take their values in M instead of M , in this case, we need the next two definitions. Definition 2.3 Every matrixmxm Q of non-decreasing functions satisfying properties (2.3) and (2.4) is called an extended semi-Markov matrix or an extended semi-Markov kernel. Definition 2.4 Every couple (p,Q) where Q is an extended semi-Markov kernel andp a vector of initial probabilities defines a (J,X) process (J,X) =((Jn,X,),n>0)with IxR as state space, also called an extended semi-Markov chain (in short an ESMC). Let us come back to the main condition (2.2); its meaning is clear. For example let us suppose that we observe for a certain fixed n that Jn.\=i, then the basic relation (2.2) gives us the value of the following conditional probability: P(J„=j,X„<x\(J,,X,),k = 0,...,n-U„,,^i):^Q,jix). (2.6)
Markov Renewal Processes
147
That is, the knowledge of the value of Jn.\ suffices to give the conditional probabilistic evolution of the future of the process whatever the values of the other past variables might be. According to Kingman (1972), the event [co \ J^_^{co) = i] i^ regenerative in the sense that the observation of this event gives the complete evolution of the process in the future as it could evolve from n^O as / the initial state. Remark 2.1 The second member of the semi-Markov characterisation property (2.2) does not depend explicitly on n\ also we can be precise that we are now studying homogeneous semi-Markov chains in opposition with the nonhomogeneous case where this dependence with respect to n is valid.
3 PRIMARY PROPERTIES We will start by studying the marginal stochastic processes (J^,«>0), {X^,n > 0) called respectively the J-process and the X-process, (i) The y-process From the semi-Markov relation (2.2) and Lebesgue's theorem (see Chapter 1), we deduce that a.s.: p ( y „ = 7 | ( j , , x , ) , ) t = o,...,«-i) = a„_,y(+^). (3.1) Using the smoothing property (see Chapter 1) of conditional expectation, we get F(y„-7|(y,),/: = 0,...,«-l) = £(a„_,,(+^)|(A),^ = 0,...,^-l), (3.2) and as the r.v. Qj^ ,y("^^) ^^ (A^=0,...,«-l)-measurable, we finally get from relation (2.3) that: = 0,,„,n-\) = Pj^^j. (3.3) P{J„=j\{J,U Since relation (2.4) implies that the matrix P is a Markov matrix, we have thus proved the following result. Proposition 3.1 The J-process is a homogeneous Markov chain with P as its transition matrix. That is the reason why this J-process is called the imbedded Markov chain of the considered SMC in which the r.v. Jn represents the state of the system iSjust after the n\h transition. From results of Chapter 3, it follows that in the ergodic case there exists one and only one stationary distribution of probability n = (7r^,...,7r^) satisfying:
148
Chapter 4
(3.4) m
such that limP(J„ =j\j,^i)(^l\mpl;')^7rj,i,JBl,
(3.5)
where we know from Chapter 3 that
[4"^] = P".
(3.6)
(ii) The X-process Here, the situation is entirely different for the fact that the distribution of X^ depends on J^A- Nevertheless, we have an interesting property of conditional independence but before proving this property we must introduce some definitions. Definition 3.1 The two following conditional probability distributions: F,^,,„(x) = P(X„<x|j„_,,/„),
H,Jx) = P{X„<x\J„_,) are respectively called the conditional and unconditional distributions of the sojourn time X„. From the general properties of conditional recalled in Chapter 1, we successively get F,^^j^{x) = E(P{X„<x\{J,,X,\k(=r4"n), ^ ^ ' •^> A^»'=(^,t/o),A^=A. If all the functions A.J,B^J,iJ-\,.,,,m,wdim^h
(3.19) at - o o , we can also use an
integration by parts to express (3.18) as follows: m
(A.B),(0 = £j5,,(?-;.)«'4(>;) and moreover if A==B, we get:
(3.20)
m
{X.Bl.{t) = Y.\Mt-y)dA,{y).
(3.21)
Markov Renewal Processes
151
Proposition 3.4 For all n>0, we have: Q;=Q^.
(3-22)
Moreover, we also have: \imQ^"\t) = P".
(3.23)
/->oo
Proof From the definition (3.17), it is clear that = Qff{t),i,j = \,...,m,t>Q. Q^it) = ^{t),Q^{t)
(3.24)
Conditioning with respect to the first transition and using the regenerative property of the SMC, we have: Q}j{t) = P{J,=j,X,+X,Oj with: 7;(/|/) = 0
(6.2)
is a renewal process that could be possibly defective. From now on, the r.v. T^ (/|/) will be called the nth return time to state i. More generally, let us also fix statey, different from state / already fixed; we can also define the nth return or entrance time to state j , but starting from i as initial
Markov Renewal Processes
157
state. This time, possibly infinite too, will be represented by fr^(7|/),«>0j, using here too the convention that 7;(y|/) = 0.
(6.3)
Now, the sequence (r„(y|/),«>Oj is a delayed renewal process with values in K^ It is thus defined by two d.f.: Gy being that of 7;(y|/) and Gjj that of r2(y|/)-7](y|/),sothat: G,i,t) =
P(T\j\f)Q\.
Remark 6.1 From the preceding definitions, we can also write that:
p[T,(j\i) =
' +^)^\-G,{+^)
and for the mean of the T(i\i\,n>\,
(6.5) possibly infinite, we get:
lu,j=E(T\j\i)y]tdG..{,t),
(6.6)
0
with the usual convention that 0-(+oo) = 0.
(6.7)
The means /u^j.ij e / are called ihQfirst entrance or average return times. Lemma 6.1 Thefiinctions GyJ,jsI
satisfy the following relationships:
m
^=1
G,j{t) = Y,G^'»Q,,it) + {\-Gjj)*Q,j{t),iJsI,t>Q.
(6.8)
Proof By the regenerative property, we can decompose the event "first entrance intoy" starting from / following the state occupied at the first transition so that: m
G,(0-e,(0+
£
G,j*Q,,{t),iJ^I,t>().
(6.9)
By adding and subtracting the missing term for k^j in the sum of the second member of relation (6.9), we get relation (6.8). n
158
Chapter 4
For each possibly delayed renewal process defined by the couple (Gij,Gjj), ij belonging to /, we will represent by Ay and Ry the associated renewal functions defined by relations 2(2.4) and 2(3.12) so that: R,j{t) =
(6.10)
E{N]{t)\j,=i)
and by relations (5.8): + A,j{t). R,{t)^S,p,{t)
(6.11)
From relations 2(9.7), 2(3.9) and 2(3.14), we get: 00
R,{t)-G,.R..{t). Or equivalently, we have: R,(t)-dp,{t)
+ G^*fGf{t),i,j^I.
(6.13)
A7=0
Proposition 6.1 Assumption w < oo implies that: (i)
at least one of the renewal processes [T^ \j\j\n
^ 0j,7 e / is not defective,
(ii) for all i belonging to I, there exists a state s such that lim7;(^|/) = +oo, a.^.,
(6.14)
(iii) for the r.v. T^ defined by relation (1.4), given that Jo=i whatever i is, we have a.s. that lim7;=+oo. (6.15) n
Proof (i) As m is finite, there exists for the embedded MC (J^,n>0) at least one recurrent state, say k, reached by the chain in a finite time so that the state k will be visited infinitely often. It follows that the renewal process is not defective. (ii) Let us now consider / as initial state. We can associate a recurrent state, say s, depending on /, which will be hit starting from / so that the delayed renewal process defined by (G.^,G^J will not be defective. (iii) This is an immediate consequence of result (ii). n The following relations will express the renewal functions R.j, i,j el in function of the kernel Q instead of the m^ functions Gy. Proposition 6.2 For every i andj of I, we have that:
Markov Renewal Processes
R>j(t)-t.Qlf'(0.
159
(6.16)
Proof Counting variables N\{t)J e I, can also be defined as series of indicators as follows: ^ ; ( 0 = i:^K=y,r„.}-
(6.17)
From Proposition 6,1, we know that the series in this last relation has a.s. a finite number of terms, whatever 7 and / are. Consequently, we can integrate term by term to obtain:
^(A^;(0|Jo =0 = Z^(^K=.r„.,}ko =0'
(6-18)
Using now relation (3.16), we get that: 00
^(iv;.(o|Jo=0-£er(o>
(6.19)
A7=0
that is (6.15).
n
Using matrix notation with: R = [i?,],
(6.20)
relation (6.15) takes the form: 00
R = ^Q('^).
(6.21)
n=0
Let us now introduce the LS transform of matrices. For any matrix of suitable functions Ay from R'' to R represented by A=[4]
(6.22)
we will represent its L-S transform by: A=[ 4 ]
(6.23)
with A,j(s) = ]e-"dA,j(t).
(6.24)
0
Doing so for the matrix R, we get the matrix form of relation (6.16), 00
RW = Z(QW)"-
(6'25)
From this last relation, a simple algebraic argument shows that, for any s>0, relations R(^)(I - Q(^)) = (I - Q(s)R(s) = I (6.26)
160
Chapter 4
hold and so, we also have that: R(s) = (I-Q(s)r\ We have thus proved the following proposition.
(6.27)
Proposition 6.3 The Markov renewal matrix R is given by (6.28)
R = |;Q('",
the series being convergent in R"^. Moreover, the L-S transform of the matrix R has the form: R = (I-Qr', the inverse existing for all positive s.
(6.29)
The knowledge of the Markov renewal matrix R or its L-S transform R leads to useful expressions for d.f of the first entrance times. Proposition 6.4 For the L-S transforms of the first entrance time distributions, we have:
Inversely, we have: GAs) R>j(s) =
l-G,(s)
-,i^J,
(6.31)
Proof From relation (6.12), we have: Rj,.(s) = (l-Gj^.(s)r, so that:
(6.32)
G..(s) = l - ^ ^ .
(6.33)
Relation (6.11) allows us to write: R,(s) = G,.(s)R.j(s),i^j,
(6.34)
and consequently RJs) GJs) = ^^^—,i^j.
(6.35)
'
Rjjis)
Results (6.33) and (6.35) give (6.30). Relations (6.31) are immediate from relations (6.11) and (6.8).
n
Markov Renewal Processes
161
7 CLASSIFICATION OF THE STATES OF AN MRP To give the classification of the states here, we will proceed as we did in the case of Markov chains: that is, by considering the embedded renewal processes or delayed renewal processes of return times in the different states of/. This gives the following definition. Definition 7.1 The state J of I is said to be recurrent, transient or aperiodic or periodic with period d, Moreover,] is positive {or non-null) recurrent iff jUjj is finite. The next proposition establishes the interaction between classification of the states of an MRP and that of the same states but for the imbedded MC (J^,n>0), Proposition 7.1 (i) j is recurrent for the MRP iff j is recurrent, necessarily positive in the imbedded MC, (ii) // sup 6^. oo in relation (6.16) with i=j, we get: 00
^,(+«^) = I p r
(7.1)
because ^Q^{t)
(7.2)
=j ^ .
But we know (see relation (2.28) of Chapter 3) that in the imbedded MC, statey is recurrent iff the series in relation (7.1) diverges. Result (i) follows then from Proposition 3.3 of Chapter 2 since R,,(oo) = +00 G..(oo) = 1 (7.3) or equivalently that the renewal process \T^ ^i\j\n
^ Oj is recurrent.
(ii) Since sup 6^ < 00 '•J
we can write, for the mean ij,.. related to the d.f G-.:
(7.4)
162
Chapter 4
and moreover, we know from results of Chapter 3, that the mean recurrence time of statey in the imbedded MC is given by: m'^=Z"ZUP...'
(7.6)
!/
n=\
S„ , , A:=0
where ^njj ={{aQ,a^,...,a^),aQ = i,a^ = j,aj^ el,k = l„.,n-l,a,^
^ j].
Assuming now that i=j, we can write from relations (7.5) and (7.6): {rmjiP,,)mjj < ^i.. < irmx/3,,)mjj.
(7.7) (7.8)
The number of states m being finite, every recurrent state in the embedded MC is necessarily positive recurrent, so that mjj is finite and consequently (ii) is true, by relation (7.8), for jUjj . (iii)
From relation (3.16) of Chapter 2, we know that (7.9)
SO that, from relation (7.1):
This last relation proves that
Zp]f^j' (8.8) are two column vectors of functions having all their components in 5, the set of single-variable measurable functions, bounded on finite intervals or to B^ if all their components are non-negative. Proposition 8.1 The Markov integral equation of MRT, f = g + Q*f with f,g belonging to B^, has the unique solution: f = R*g.
(8.9) (8.10)
Proof a) Existence of the solution Replacing f of the second member of the equation (8.9) by (8.10), we get: g + Q«f = g + Q # R . g , = (I + Q . R ) . g , where the function matrix I is defined as in relation (8.2). Using relation (8.3), we obtain: (I + Q # R ) . g = R . g (8.12) or R.g = g+Q.(R.g). (8.13) This relation proves that the vector given by relation (8.10) is a solution of the given equation (8.9). b) Unicity of the solution Let fpf2 be two solutions of the equation (8.9) and let y be defined as: (8.14) y = f,-f,. Of course, the vector y satisfies the following equation: y = Q*y (8.15) and by iteration, we get for all positive n\ y = Q^"^*y. (8.16) As, from Proposition 6.2, the renewal matrix R can be defined by the function series R = |^Q^'^ which converges for all positive t, we must have
(8.17)
Markov Renewal Processes
165
limQ0.
(8.19)
And so by relations (8.17) and (8.14), we get: f\t) = f\tlt>Q, proving so the unicity of the considered solution.
(8.20) n
9 ASYMPTOTIC BEHAVIOUR OF AN MRP We will give asymptotical results, first for the Markov renewal functions and then for solutions to integral equations of an MRT. To finish, we will apply these results to transition probabilities of an SMP.
9.1 Asymptotic Behaviour Of Markov Renewal Functions We know that the renewal function Ry , ij belonging to /, is associated with the delayed renewal process, possibly transient, characterized by the couple {Gij,Gjj)A,f.on R\ Let us recall that ju^j represents the mean, possibly infinite, of the d.f Gij, Proposition 9.1 For all iJ of I, we have:
(i) iiinM^ = -L, RM)-RM-T)
(ii) lim^^
(9.1) T
= — , for every fixed r .
Proof (i) From relation (6.12), we have, for / ^ 7 : R,{t)_G,^R,{t) t
(9.2)
(9.3)
t
Ifj is transient, we know from Proposition 4.2 of Chapter 2, relation (4.25), that: \imG,j»Rjjit) = G,j(c^)-Rjj(oo) (9.4) SO that from relation (9.3): l i m - - ^ ^ = 0. The same proposition also gives us immediately that for ally transient:
(9.5)
166
Chapter 4
l i m ^ ^ = 0. /->oo
(9.6)
f
If y is recurrent, necessarily positive, we have from Corollary 4.2 of Chapter 2 that:
limM)=_L. '^»
t
(9.7)
Mjj
Moreover, from relation (6.11), we have in this case that: G,j.Rjj{t) = G,j>{U,+Hjj){t)
(9.8)
so that by relation (6.12), we get: lim—^^— = lim—^^— + /->00 lim—^^ /->Q0 f /->00 ^
f
— (9.9)
t
'->«•
Now, the key renewal theorem (Proposition 4.2 of Chapter 2) applied to the function G=l - Gy gives as a result that: \imG»Hjj{t)=~\{\-Gjj{x))dx, Mjj 0
(9.10)
_Mij ^jj
Hence l i m ( 7 / , ( 0 - ( ^ . . / / , ( 0 ) = 0.
(9.11)
From the renewal theory, we know that: l i m ^ ^ =—
(9.12)
and thus relation (9.11) gives:
lim^^M)=i t
(9.13)
lUjj
and so, from relation (9.9), we obtain result (9.1). (ii) The proof of (9.2) is based on the same argument as the one used for proving (i) and on Blackwell's theorem (see Chapter 2, Corollary 4.3). a The next proposition, due to Barlow (1962), is a useful complement to the last proposition as it gives a method for computing the values of the mean return times lUjjJ G / , in the ergodic case. Proposition 9.2 For an ergodic MRP, the mean return times satisfy the following linear system:
Markov Renewal Processes
167
In particular, for i=j, we have: jUjj=—Y,7r,7j„j = l,.,.,m, ^j
(9.15)
k
where the Tj^Jel are defined by relation (3.14), and where 7r = (;7r,,...,;7r^) is the unique stationary distribution of the imbedded Markov chain. Proof Relation (6.9) immediately gives: l^ij=Pijbij-^T.Pik^^ik^^kj)
(9.16)
k^j
or using relation (3.14) /^ij=Y,Pikf^kj-^ri.,
(9.17)
k^j
that is (9.14). Now, let ^i be the matrix \l^A , and let ^^,Ho b^ respectively the two matrices obtained from |LI as follows; for \i^, in setting all diagonal elements equal to 0 and for n^ in setting all non-diagonal elements of \i equal to 0. It follows that: fi = fi^+jio(9.18) Now, under matrix notation, the linear system (9.14) takes the form: ji = Pfio+il, (9.19) where ii is the mxmmatrix with all columns identical to the vector [T]],'",r/^) . From relations (9.18) and (9.19), we get ^,=(P-I)jio+il. (9.20) Premultiplying both sides of this equality by the vector n, we obtain : nii^=nr\, (9.21) since nP = n (9.22) by relation (5.5) of Chapter 3. Moreover, as for all j , TTJ is positive (see relation (5.17) of Chapter 3), relation (9.21) gives relation (9.15). Remark 9.1 In a similar manner. Barlow (1962) proved that if iuf\ij
n G / is the
second order moment related to the d.f Gtj , then:
//r=^/''+ZA.(/^,f+26,,//,.) k^j
and in particular for i=j\
(9.23)
168
Chapter 4
^ =^{Y.^,rif^ ^2YZ^,P,AM,) ^j
k
(9.24)
k^j I
with rif^= \x^dH,{x\keI,
(9.25)
[O.oo)
provided that these quantities are finite.
9.2 Asymptotic Behaviour Of Solutions Of Markov Renewal Equations Under the assumptions of Proposition 8.1, we know that the integral system (8.9), that is y;(0-g,(0 + Z \fj{t-s)dQ,j{s\ieI, (9.26) has the unique solution ^W = E \gj{t-s)dR,j{s\ieL
(9.27)
J [0./]
We will now study the asymptotical behaviour of this solution for t tending toward +oo, and we will establish the analogue of Proposition 4.2 of Chapter 2, i.e. the key renewal theorem. Proposition 9.3 {Key renewal theorem) For any ergodic MRP, we have:
l i m ^ \g(t-s)dR,(s)^^^ '"°° i [0,]
,
(9.28)
L^jm J
provided that the functions gu i belonging to /, are directly Riemann
integrable.
Proof From relation (6.12), we have for i^j\ \gj{t-s)dR,j{s) = gj*G,.*Rj.{t) [0,'] (9.29) =(g,«G,)./?,.(/). From properties of direct integrabihty (see Cinlar (1975b)), since G/, is a d.f, gj • G^j is also directly integrable. Consequently, we can apply the key renewal theorem (Proposition 4.2 of Chapter 2) saying that:
Markov Renewal Processes
169
= —]gj.G,j(T)dT,
\im(gj.G,j).Rjj(t)
(9.30)
Mjj 0
Permuting the order of integration in the double integral of the second member, and using after the change of variable T-U = y ,WQ get: 00
00
CO
jgj • G^j (T)dT = jdGy (u) jgj (T - u)dr °
°
"
00
00
=
(9.31)
jdG,j(u)lgj(y)dy. 0
0
But the assumption of irreducibility of the MRP implies that Gy is a d.f onR"^, and thus: ]gj'G,j(T)dT 0
= ]gj(y)dy.
(9.32)
0
Going back to equality (9.30) and using relation (9.15), we finally get (9.28).
10 ASYMPTOTIC BEHAVIOUR OF SMP 10.1 Irreducible Case Let us consider the SMP (Z(/), / > 0 ) associated with the MRP of kernel Q and defined by relation (5.8). Starting with Z(0) = /, it is important for the applications to know the probability of being in statey at time /, that is: (10.1) ^.(0 = P ( Z ( 0 = ;|Z(0) = /). A simple probabilistic argument using the regenerative property of the MRP gives the system satisfied by these probabilities as a function of the kernel Q:
,(t) = s,j(\-HXt))+z]M^-y)dQ,(y), k
Uj^i.
(10.2)
0
It is also possible to express the transition probabilities of the SMP with the aid of the first passage time distributions G^jJJ G / : ^.(0 = ^ ^ . G , ( 0 + ^.(l-//,(0)> hj^l(10.3) If we fix the valuey in relations (10.2), we see that the m relations for i=\, ..,,m form a Markov renewal type equation (in short MRE) of form (8.9). Applying Proposition 8.1, we immediately get the following proposition. Proposition 10.1 The matrix of transition probabilities
170
Chapter 4
,(t)=^A-,
i,jel,
the SMP associated with an ergodic
(10.8)
k
Proof For a fixed value of7, we know that the ^^., 7 =l,...,m satisfy the MRT integral equation (10.2). As the basic MRP is supposed to be ergodic, we can apply the key Markov renewal theorem (Proposition 9.3) to get: CO
lim4.(0 = ^
,
(10.9)
k
and since 00
\{\-Hj{y))dy
= VjJ^I,
(10.10)
0
relation (10.8) follows.
n
Markov Renewal Processes
171
Remark 10.2 (i) As the limit in relation (10.8) does not depend on /, Proposition 10.2 establishes an ergodic property saying that: lim 0 for ally (see relation (5.17) of Chapter 3), we also have n,>0,yG/.
(10.12)
So, asymptotically, every state is reachable with a strictly positive probability. (iii) In general, we have: \\mpf^\\m(j)M) (10.13) /7->oo
-^
/->00
•^
since of course 71J ^Uj J el,
(10.14)
This shows that the limiting probabilities for the imbedded Markov chain are not, in general, the same as taking limiting probabilities for the SMP. From Propositions 10.2 and 9.2, we inmiediately get the following corollary. Corollary 10.1 For an ergodic MRP, we have: n.=^.
(10.15)
This result says that the limiting probability of being in state y for the SMP is the ratio of the mean sojourn time in statey to the mean return time ofy. This intuitive result also shows how the different return times and sojourn times have a crucial role in explaining why we have relation (10.14) as, indeed, for the imbedded MC, these times have no influence.
10.2 Non-Irreducible Case It happens very often that stochastic models used for apphcations need nonirreducible MRP, as for example, in presence of an absorbing state, i.e. a statey such that Pjj=L (10.16)
172
Chapter 4
We will now see that the asymptotical behaviour is easily deduced from the irreducible case studied above. 10.2.1 Uni-Reducible Case As for Markov chains, this is simply the case in which the imbedded MC is unireducible so that there exist / {l<m) transient states, and so that the other m-l states form a recurrent class C. We always suppose aperiodicity both for the imbedded MC and the considered MRP. Let T= {l,...,/} be the set of transient states {T = 1 -C) .From Proposition 5.1 of Chapter 3, we know that: \im<j>,j{t) = 0,iJsT. (10.17) Moreover, from Proposition 10.2 and relation (10.3): lim(Z^(0-G,(a)) ^^"^^
, /,7GC,
(10.18)
where (;r/^,,...,;r^) represents the unique stationary probability distribution of the sub-Markov chain with C as state space. Since (10.19) G,M = f,, we get (10.20) Gy(^) = f,c^ whereyjc is the probability that the system, starting in state / will be absorbing by the recurrent class C. As there is only one essential class, we know that for all states / of/: fi.c=h (10.21) proving so the following proposition. Proposition 10.3 For any periodic uni-reducible MRP, we have: ]im(/>^j(t) = n\,jel,
(10.22)
where '0, 7 E r ,
n; = -IJH^J^C,
(10.23)
Here too, as the limit in (10.23) is independent of the initial state /, this result gives an ergodic property.
Markov Renewal Processes
173
10.2.2. General Case For any aperiodical MRP, there exists a unique partition of the state space /: / = rUC,U---UC,, r < m , (10.24) where T represents the set of transient states and C^,v = l,...,r represents the V th essential class necessarily formed of positive recurrent. From Chapter 3, we know that the system will finally enter one of the essential classes and will then stay in it forever. So a slight modification of the last proposition leads to the next result. Proposition 10.4 For any aperiodic MRP, we have :
lim^.(0 = n , , / , 7 e / ,
(10.25)
with, for any j eC^,v = l,...,r : f Yij,
n,=
0,
ieC^,v = l,..,,r,
/ E C , . , v ^ v > ' = l,...,r,
[f.cn'j,
(10.26)
ieT
where (TIJ ,j e C^)is the only stationary distribution of the sub-SMP whith C^as State space, that is :H. = ^
—,
(10.27)
where yTT^^k e C^ j is the unique stationary distribution of the sub-Markov chain with C^ as state space and [f^c,^^ eTjis the unique solution of the linear system
yi-lLp,yj-lLp,^i^T. J^T
00.28)
jeC,
Note that, in this proposition, the ergodic property is lost; this is due to the presence of the quantities fc ^^ relation (10.26).
11 RECURRENCE TIMES 11.1 Definitions For an MRP [(J^,TJ,n>0),
the random sequence of renewal epochs
((r^),^>0) still represents as in renewal theory, and the random sequence ((X^),/7>0) represents still successive Hfetimes and so the supplementary
174
Chapter 4
information with respect to renewal processes is given by the random sequence [(J^),n>0), the r.v. J^ giving the state in which the HfetimeX„+i was spent. Also, we can define recurrence times exactly as we did for renewal processes in section 8.1 of Chapter 2. Thus, r.v. 5{t) will give the age of the "component" actually living at time / and r.v. y{t) the time that this "component" has still to wait until its failure, so that we get the following definitions. Definition 11.1 For each t>0, the r.v. 5{t) and /(t) defined by the following relations:
are respectively called the age {or backward recurrence time) and the excess {or forward recurrence time or residual time) at time t.
11.2 Transient Results First we will give some transient results related to the two stochastic processes d = {5{t)j>Q), (11.2) r^{y{t\t>Q), respectively called the age and the excess processes. Proposition 11.1 {Distribution of the age) Let 7^.,(,)(x) = P(cJ(0<x,Z(0 = 7|Z(0) = /),
(11.3)
then \
P,j,x>t,
Pij,s(,)ix) = \ j (l-Hj(t-u))dH.j(u),x oo in relation (11.12), we get relation (11.17). n A similar argument gives the other result. Let us now define the function //and the number 77 as follows:
Markov Renewal Processes
177
'
(11.18)
Using Proposition 11.5 and Corollary 11.1, and bearing in mind that 77 is the mean related to the d.f. of//, the next result is immediate. Remark 11.1 As in renewal theory recurrence times ^(Oand y{t) have the same limit distribution, and moreover, this limiting distribution is independent of the initial state (ergodic property). Corollary 11.2 For any ergodic MRP, we have: .
00
limP(^(0>x,;'(0>>'|Z(0) = i) = - \{\-H{s))ds
(11.19)
and lim/^(cJ(0 < x|Z(0) = /) = limP(;^(0 < x|Z(0) = /)) = - j ( l " H{s)Ys.
(11.20)
Particular cases This last result shows that the common limiting distribution of the r.v. d{t) and y{t) has as density the function defined by: -{\-H{x))
(11.21)
and thus the mean of the limit distribution is given by 2
2/7
where cr^ is the variance related to the d.f //. To compute this variance a^ in terms of kernel Q or of doublet (P,F), we use the following notation : cc2jj--\x^dF^j{x),iJsI, 0 CXD
a,. = jx^dH.(x),i,eI,
(11.23)
0
^2 = fxV//(x), 0
so that from the second equality of relation (3.11), relations (11.18) and (11.23), we clearly have:
Chapter 4
178
J
(11.24) J
Using now relations (11.24) and (11.18), we can finally give the expression of (j^ as follows:
^_
V
v /
y
(11.25) /•
J
In the special case of F.j=F., i,JGl,
(11.26)
the preceding relations give for all / andy: ^2,ij
~
^2,i'
(11.27)
a^ = Z^/^2,/Similarly, for the second special case of
F.j=F.,i,jeI,
(11.28)
we get i
J
(11.29) So, in both particular cases (11.26) and (11.28), formula (11.25) becomes:
J
(11.30)
V J
12 DELAYED AND STATIONARY MRP Let us suppose we begin to observe the evolution of an economical or physical system 5* at time TQ and that the probabilistic evolution of this system is like a semi-Markov process. There is absolutely no reason that we should observe a transition of the system at time 0. In fact, we observe the state Z(0) while waiting for the first observed transition occurring at random time T\. The first lifetime Xi is a residual time and may have its own distribution function. This leads to the concept of delayed MRP. Definition 12.1 The bidimensionalprocess
(^(J^,T^),n>0^with
Markov Renewal Processes
179
T =0 ' ' (12.1) T„=X,+- + X„, n>\, is called a delayed Markov renewal sequence or delayed Markov renewal process {in short DMRP) of triplet (p,Q,Q) if (i)P(Jo=i) =
Pi,i^L
(ii)P(j, =j\X,<x\j,
=i) = aj(x), ij'el,
(12.2)
This definition is clearly based upon the supposition that the m-dimensional vector p = (/?j,...,j[?^)represents a probabihty distribution on / and that matrices Q,Q are two semi-Markov kernels. As in the case of renewal theory, there exist simple relations between renewal functions, marginal distributions, etc... of a DMRP of triplet (p,Q,Q) and the corresponding functions of the classical associated MRP of kernel Q. So, with the convention of adding a tilde to the functions related to the DMR, let Ry(t) be the Markov renewal functions of the DMRP, that is: R,j(t) = E(Nj(t)\j,^i).
(12.3)
We know that: R,j(t)^±P{j„=j,T„ O) is stationary and P{Z(t) = j\r(t)<x)
= ^]{l-H.(y))dy, ^J
(12.33)
0
Proof As the probabilistic interpretation of Markov renewal functions remains unchanged for SMRP, we have
p(z(t)^j,r(t)<x\z(0) = i) V
(12.34)
0
Using now relation (12.16) the value of the unconditional probability of the first member of relation (12.33) becomes: p{Z(t) = j\r(t)<x)
= ^^^(H,j(t '
1 ^
V
7 /=1
0
By Proposition 12.2, we know that
+ x)-H,^j(t)) (12.35)
184
Chapter 4 Yl
1 m
-YmRsAy)=—''
(12.36)
and by relations (12.17) and (3.17), we also have that: H,,{t) = — \{\-H,{y))dy. ^j
(12.37)
0
Inserting these last two results into result (12.35) and recalling from (12.28) that:
-Z^,'7,=In,.=l,
(12.38)
we obtain: P{Z(t) = j,r(t)<x) +-'£-iM.njU-(l-Hj(t ^U. ^
j(^i-H.(y))dy t
=
^l(l-Hj(y))dy + x-y)) + (l-Hj(t-y)))dy
+ -^\ j(l-Hj(z))dz+ ^J \0
So that finally, we get result (12.33).
(12.39)
J(1 -//,(z))c/z t+x
n
As an immediate consequence of this final proposition, we find that the marginal distributions of processes (Z(t),t>0),(y(t),t^O) are stationary, getting thus the next corollary. Corollary 12.2 Under the assumption of Proposition 12.3, we have for all positive t: njJeI, P{Z(t) = j) = 1 "r
(12.40)
P{r(t)<x)^-j(l-H(y))dy,x>0. V 0
Here too, we see that, in observing the excess process y , everything happens as though we had a stationary renewal process characterized by the d.f //, defined by relation (12.32). Remark 12.1 From relations (12.2), (12.16) and (12.17) and (12.29), we get:
Markov Renewal Processes
i
185
J
'•
j
'li 0
=-YLA^Pij-QiM))dy ^
i
7
/
J
(12.41)
0
0
^ 0
This is a result to be expected as in a stationary SMP, the first observed variable X\ is actually the excess of the running process at the origin of the observations. Nevertheless, as p^ ^TT , the process (J^,n>0), and consequently the process (X^,n > 0), are not usually stationary.
13 PARTICULAR CASES OF MRP We will devote this paragraph to particular cases of MRP having the advantage to lead to some explicit results.
13.1 Renewal Processes And Markov Chains For the sake of completeness, let us first say that with m=l, that is that the observed system has only one possible state, the kernel Q has only one element, say the d.f. F, and the process (Xn,n>0) is then a renewal process. Secondly, to obtain Markov chains studied in Chapter 3, it suffices to choose for the matrix F the following special degenerating case: F,j=U,yij'eI (13.1) and of course an arbitrary Markov matrix P. This means that all r.v. X„ have a.s. the value one, and so the single random component is the (/„) process, which is, from relation (3.4) a homogeneous MC of transition matrix P.
13.2 MRP Of Zero Order (PYKE (1962)) There are two types of such processes
186
Chapter 4
13.2.1 First Type Of Zero Order MRP This type is defined by the following semi-Markov kernel Q-[p>F^,
(13.2)
SO that:
p^=p^,F^=F,JeL
(13.3)
Naturally, we suppose that for every i belonging to I,piis strictly positive. In this present case, we have that the r.v. J^,^ > 0 are independent and identically distributed and moreover that the conditional interarrival distributions do not depend on the state to be reached, so that, by relation (3.11), H^=F,,isL (13.4) Moreover, since: P(X„ <x\(J„X,),kOare independent and equidistributed and moreover the conditional interarrival distributions do not depend on the state to be left, so that, by relation (3.11) m
H,=YpjFj{=F\i^L
(13.10)
y=i
Moreover, since: P{X„<x\{J,,X,\koo
or
188
Chapter 4
S;=Y,^im Here too, although we have uni-reducibility, the asymptotical state S. is independent of the initial state /. To pass from the Markov chain model to the semi-Markov one, the additional information needed is the knowledge of the matrix B = \Pij\ -
14.2 Comparison of Markov And Semi-Markov Models To compare both models in the same application for the obtained asymptotical results, let us introduce the following inequalities resulting from relation (14.6): inin^., 0,j=l,...,m, we have: P(J„ =j,X„ <x\iJ,,X,),k = 0,...,n-l)^Qj^^jix),a.s.,
(1.1) (1.2)
is called a (J,X) process or an extended semi-Markov chain (in short, ESMC). From this definition, it follows that we can no longer represent the sample paths of such a process with step functions as the r.v. X^ can be positive or negative but we can see the 5'-process defined by: S„=X,+X,+-+ X^,n = 0,l,... (1.3) as the successive positions of a particle moving on a real line and starting from the origin if the r.v. Xn n=0,\,... represents the successive steps of this random movement exactly as the interpretation of a classical random walk corresponding to the case of m=l. This leads to the following definition. Definition 1.2 The S-process defined by relation (1.3) is called a semi-Markov random walk (in short an SMRW).
194
Chapters
It is clear that basic results on positive {J^ processes given in the preceding chapter are still valid here provided that these properties do not involve the nonnegativity of the Xn, The following proposition summarises the basic properties. Proposition 1.1 {Basic properties of {J,X) processes) (i) The process [(J^,S^),n>0) is a Markov process with / x M a^ state space; more precisely, we have a.s. for allj of I and all real x: P(j„=j,S„<x\{J„S,),k = QX...,n-\) = Qj^^j{x-S„). (1.4) (ii) The process [(J^),n>0)is a homogeneous Markov chain with I as state space; more precisely, we have a.s. for allj of I: P{J„ =j,\j„k = 0,l,...,n-l)^pj^^j. (1.5) (iii) For all strictly positive n andfor all real x, we have: P(X„ <x,\Ji^,k = 0,l,...,«-l) = Hj^^(x),n>0,xeR, P(X„<x,\j„k^O,l,...,n) P(X„, <x„...,X„^
= Fj ^j (x),«>0,xeR, * <x„\j„k^QX...,n,yY{^j^,.J„M^
(1.6)
(0-/^J,Q = fe]>P = h ] > F - [ ^ ] -
(1-8)
Functionals of (J-X) Processes
195
2 FUNCTIONALS OF (J-X) PROCESSES This paragraph introduces the concept of functional ff of a given {J^K) process, fundamental for a lot of applications not only in finance and insurance but also in operations research. To define the functional W, we introduce a real and Lebesgue measurable function/of three variables defined on the set / x / x R . When they exist, we will use the following notation for the expectations with i,kel: 4 = jf(i,k,x)dQ,,(x),
^f> =
jf\iXx)dQ,,(x), (2.1)
k=\
k=\
Definition 2.1 Given a (J,X) process ((J^,X^),«>0) defined by (p,Q), and a Lebesgue measurable real function f on IxIxM., the functional W/ is defined as the stochastic process W^={W^{n),n = QX...) (2.2) where 0,« = 0, WAn):
Y,fiJ,-,,J,,X,\n>Q.
(2.3)
Janssen (1969b) extended for functionals of{Jy>C) processes key results obtained by Pyke and. Schaufele (1964) for functionals of positive {J^ processes: the strong law of large numbers (in short SLLN) and the Central Limit Theorem (in short CLT). The basic idea of the proofs is to decompose the sum W/ji) by introducing return times of return indices for the embedded Markov chain supposed to be ergodic (j^,/7>0) defined by relations (2.16) and (2.17) in Chapter 3, that is for ally belonging to /:
We know that the assumption of ergodicity implies that all states are positive recurrent so that, for ally, (2.5) ^-^oo=>rJ-^"^ ^ 0 0 . Moreover, if for ally we introduce the stochastic process in discrete time:
196
Chapters
{ul''\n>0)
(2.6)
with
^i''= Z
/(^„-P^„>^J>
(2.7)
then it is clear that this process (2.6) is a sequence of independent and identically r.v. with values in R , that is a random walk on the real line. The following propositions give some results concerning the moments of r.v. u(jeI,J^=j, Proposition 2.1 If the embedded Markov chain of the considered (J,X) process is ergodic and if the consideredfunctional is such that the expectations i,
(2.23)
(2.24)
and from relation (2.7): (2.25) From relations (2.1), we get: ^jk=PjkbjkJ^keI, and finally, from relation (2.8):
(2.26)
Functionals of (J-X) Processes 1
199
m
Eiu^^')^—Xmi-
Mjj)J 6 /•
(2.27)
Now applying Proposition 2.2 to the considered special case (2.23), we obtain the next fundamental result in particular for the asymptotical behaviour of semiMarkov random walks studied in the following section. Proposition 2.3 (Strong law of large numbers for (J,X) processes) For any ergodic {J,X) process so that the conditional means b.j, i,j e I are finite, we have the following result: O
m
-^^;^^llm,ci.s. n
(2.28)
i=]
The next result is related to a central limit theorem for functionals of (J^ processes. Starting still from the decomposition (2.9), Janssen (1969b) proved the following two propositions. Proposition 2.4 (Central limit theorem for functionals of (J^X) processes) If the expectations ^. exist for all i belonging to I, we have in the ergodic case andfor the convergence in law that: m. Wf(n)-n--^ r r W m.. m. (2.29) 0,vart/i \[n m V V JJ J Moreover, if jj... defined by relation (221) is non-null, then Wj-(n)-nAj.ju yjn where
->iV(0,//5)
(2.30)
Chapter 5
200
m
4=-^
T^^iVi 1
jPkr =—
(2.31) f
'
.hj.kel.
Remark 2.2 From Remark 2.1, we know that A/is independent of state 7, as of course jUjj is also independent ofy; the same is true for Bf. Proposition 2.5 {Central limit theorem for the two-dimensional process (j„,W^{n),n>0)) If the expectations ^. exist for all i belonging to I, we have in the ergodic case andfor the convergence in law that: f
Mir / , . \
,.A..
\
= ;r,0(x), (2.32) ^n where the function O is the distribution function of the normal law N(0,juB ) . Remark 2.3 An immediate consequence of this last proposition is that processes (J^,n> 0) and {Wf{n),n > 0) are asymptotically independent. These last two propositions immediately give the following one for the special case of {JyK) processes Proposition 2.6 {Central limit theorems for the two-dimensional {J^ processes) For any ergodic {J,X) process such that the conditional variances a^. related to the conditional df F.jJJ e / are finite, we have the following results: S^-n^
-Jn
T"
->iV(0,//5J,
' ^n -
••n^^{x\
(2.33) (2.34)
Functionals of (J-X) Processes
201
where the function O is the distribution function related to the normal law NiO,JjuBg jwith here, for all i and k belonging to I: g{i,k,x) = x-J^7r.ri., (2.35)
^ik=PikbikA=Vi^ Yu^iVi J
Pik^
V/=l
G\ {i,k G /) being the conditional variance related to the conditional distribution
3 FUNCTIONALS OF POSITIVE (J-X) PROCESSES It is clear that all the results of the preceding paragraph are valid for the special case of positive {J^ processes for which the r.v. X^ are a.s. non-negative. But moreover instead of considering the sum of the first n transitions to define Wj{n) in relation (2.3) we can reinsert the time with a sum up to A^(/), that is the total number of transitions in the semi-Markov process related to the considered semi-Markov kernel Q. In fact, this was the case originally considered by Pyke and Schaufele (1964) so that now, relation (2.3) takes the form: 0,A^(0 = 0, N{t) WAt) = (3.1)
X/(^._P^«,^J,A^(0>0.
Using the same decomposition as the one defined by relation (2.9), these authors proved the next three propositions corresponding to the strong law of large numbers and central limit theorem of the preceding sections. Proposition 3.1 {Strong law of large numbers for functionals of positive {Jy)C) processes) If the expectations ^. exist for all i belonging to I, we have in the ergodic case that: Wf{t) m.. M.S., (3.2) t mjj with:
202
Chapter 5 m
'='
(3.3)
m
the limit ratio rrij I rrijj being still independent of j . The propositions related to the central limit theorem have similar extensions for n replaced by N{t) to the numerator and by / to the denominator.
4 CLASSICAL RANDOM WALKS AND RISK THEORY 4.1 Purpose. In Chapter 4, we focused our attention on semi-Markov chains defined by a positive {J^ process. The case of an extended semi-Markov chain is considered in this chapter starting from a general ( J ^ process and having a very different interpretation, directly related to the classical notion of random walk, In next subsections, we will recall some basic notions concerning random walks that will be extended to the main results of what will be called Markov random walks in the next section. After that, we will develop the main classical models in risk theory, which is very useful for insurance companies.
4.2 Basic Notions On Random Walks Let [X^,n> l) be i.i.d. random variables, with F as common d.f, such that: i^(0) 0. (4.2) These two relations imply that for all n, the events {o):X„>Q],{co:X„ O) is called a random walk starting at xo, whose {X^,n > l) are the successive steps. If xo=0, the random walk is said to start at the origin. Example 4.1 If the distribution of r.v. Xn is concentrated on a two-point set
{-1,1} with p = P(X„ = 1), q(= l-p) = P(X^ ^ 1), (4.6) then the associated random walk is called the simple random walk or the Bernoulli random walk. The interpretation is quite simple: let us consider for instance a physical particle moving on a straight line starting at the origin. This particle takes a first unit step to the right with probability p or to the left with probabihty q and so on. Clearly, the r.v. S^ will give the position of the particle on the line after the nth step. Though very particular, the notion of a simple random walk has a lot of important applications in insurance, finance and operations research. A very classical application is the so-cMQd gambler's ruin problem. Let us consider a game with two players such that at each trial, each gambler wins 1 monetary unit with probability p and loses -1 monetary unit with probability q(= l-p), If u is the initial "fortune" of one player, he will be ruined at trial n iff, for the first time, his fortune just after this trial becomes strictly negative. He will be ruined before or at trial n iff he is ruined at one time k, k < n. The probability of this last event will be noted by ^(u,n) and the probability to be ruined precisely at time n will be noted by u(u,n) . Clearly, we have: '¥(u,n) = J^u(u,k)
(4.7)
and u(u,n) = '¥(u,n)-'¥(u,n-l).
(4.8)
The probability of not being ruined on [O,^], that is to say after any trial on [0,n], will be represented by y{u,n), and of course, we have: (4.9) y{u,n) = \-^{u,n). Probabilities y{u,n) and ^(u,n) WCQ cMQd VQspQctiyQly thQ non-ruin probability and the ruin probability on [O,^] starting at time 0 with an initial fortune - also called reserve or equities for insurance companies - of amount u.
204
Chapter 5
Now we will see how to express these two probabihties with the aid of events as functions of the variables X„, ^7=0,1,.. representing the "gain" (positive or negative) of the considered player just after the nth trial. Starting with xo=u, we can write: (4.10) u(u,n) = P{S,>0,k = l,..,,n-l,S„ O) are i.i.d, the sequence [H^,n> O) is still a random walk but eventually stopped at a certain step as it may be that p(7^=oo)>0. (4.31) The two-dimensional r.v. on N x R^ {(r„,H„),n>0) (4.32) are mutually independent and identically distributed with, as common d.f., P{r„S„...,S„_,>S,,S^<S,}
(4.34)
and that: H^=S^^.
(4.35)
As in the ascending case, we define by induction the construction of the following sequence of possibly defective r.v.: (r^,H^,n>0). (4.36) For the sake of completeness, let us mention too that we can also define weak ascending and descending ladder variables by replacing the strict inequality by a non-strict inequality in relations (4.25) and (4.34). These two sequences will be represented by:
(r;;,//;),(rf-,//f-).
(4.37)
4.3 Renewal Results On Ladder Variables Let us consider a process of strict ladder variables ((r^,H^),n > 0). The marginal process ((HJ,n > 0) is a possibly defective renewal process whose d.f. H is given by relation (4.21). If g is the renewal function associated with H, we have by (3.13) of Chapter 2 that: g(x) = f^M'"\x), By relation (3.16) of Chapter 2, we also have:
(4.38)
208
Chapter 5
1 - M(oo) a relation showing that the mean number of strict ladder heights on [0,oo) is finite iff//(oo)S,,k = 0,l„.„n-l and that it is a weak ascending ladder point iff: (4.41) S„>S„k = OX...,n-l. This distinction between strict and weak ladder points has no interest if the d.f H is continuous - but such is not the case for example for the Bernoulli random walk! Let M~ be analogue to M, defined by (4.19), in case of the weak ladder process. We have the following result: Proposition 4.1 If C-f,P(S,
SjJ {(D\Sl>Sl-Sl_jJ
= 0,..,,n-\]
= 0,...,n-\]
can be expressed as
or {co\Sl_j>QJ = 0,,„,n-\]
and finally by a
change of indices {co\S* >0,/ = l,...,/?}. We can now present the so-called duality lemma, first given by Feller (1971) and then by Heyman and.Sobel (1982).
210
Chapter 5
Proposition 4.4 {Duality lemma) If for the random walk generated by the sequence {Xj^,k > O) with XQ^O, we set: n(t) = card{5'^ >0,k = l,.,.,n,0<S^ then: E(n(t)) = g(t),
0,k^l,...,n,0<S„0,k
(4.57) and (Sl,S*,,..,Sl) are
= l„..,n,0<S„S„k
= l„,„n,0<S^oo
(5.4) X- +00,
ieI,xeR. (5.5)
-^
the matrix
P = [/;,]
(5.6)
is a sub-stochastic matrix satisfying conditions (5.2). The defective positive (J,X) process satisfies the conditions P(Xo=0)=l,a.s.,
(5.7)
216
Chapters
PiJo = 0 = P>i^\,...,m
with X A = 1 >
and for all n>OJ=\, .,,,m, we have: = 0,,„,n-\)^Q,^Jx\a,s, P{J^^j,X^<x\{J,,X,lk
(5-8) (5.9)
Let us recall that when the process enters for the first time state 0, it will stay forever and so we say that the process terminates or is terminated at the first entrance time in state 0; so, we can define the unconditional waiting time distributions as follows: e /, H,{x) = PiX„ < x\j„_, = i) = ZQIJMJ ye/
, H,(x) = P(X„<xJ„_,^0)
fO, xeM, =\ '
[I, X = +00,
and consequently: = P(J„=0\j„_,=i) = l-u,JeI, P(X^=^^\j„_,=i) and of course: \imH.(x) = u.,i€l. Now, let T.j(x,n)(i,jel)
(5.10)
(5.11) (5.12)
be the probability that the (J,X) process terminates at
step n with Jn^j\ before or at time x given that Jo'^iIt is clear that: T,j(x;n) = P(T„ <x,J„ =j,J„,, =0\J,=i), =(^-Oj)(Ql;\x)). The value of the probability 7]^. («)(/, y G / ) that the (JJ^ process terminates at step n with Jn=j, given that jQ=i, is (5.14) T^(n) = \imT^(x;n) = pl;^(l-Uj), the value of the probability T. {n)(i G 7) that the (J^X) process terminates at step n given that Jo==^*j is
T,(n) = Y,T,.(n) = Y^pl;\l-u.), Jel
(5.15)
jel
and finally the probability 7),(/ e I), that the (J^X) process terminates given that Jo=i, is given by
Replacing the Uj,j e 7 by (5.2), we find the result that:
TM)'?',:"'.
Functionals of (J-X) Processes
217
Using a result of Chung (1960, p.227), we have that the (7;,/e/) satisfy the following linear system: T,=ZP>^T,+p,„ieT, (5.18) keT
where 7 is the set of transient states in the imbedded Markov chain. Definition 5.2 (i) A defective positive {J,X) process is i-terminated iff, starting from state i, it terminates a.s., (ii) A defective positive {J,X) process defined by the triple (m,p,Q) is pterminatediff starting from initial distribution p, it terminates a.s., (iii) A class of defective positive (J,X) process defined by the doublet (m,Q) is terminated iff starting with any initial distribution p, // is p-terminated. The next proposition is now obvious. Proposition 5.1 (i) A defective positive (J,X) process is i-terminated iffTi=\, (ii) The defective positive {J,X) process (m, p, Q)is p-terminated
iff^p.T.=l, iel
(iii) The class defective positive {J,X) process is terminated iff Ti=\, for all i belonging to I. Remark 5.1 If state 0 is the only recurrent state, then Ti=l for all / of/and so we have a terminal class. In particular, this is the case when for all / of/, ^ < 1. For defective positive (J,X) processes, we can introduce the concept of lifetime of the process defined by the r.v. Mas follows Definition 5.3 The lifetime of a defective positive (J,X) process is the r.v. M defined as: M= sup r„. (5.19)
It is clear that the conditional distributions of M, having M as support, are given by: M, W = P(M < x\J, = i) = j;^f^T,j(x;n). Using relation (5.13), we get:
(5.20)
218
Chapters
(5.21) a result giving the theoretical explicit form of the lifetime conditional distributions. A simple probabilistic reasoning proves that the functions Mi, i=\,.,.,m satisfy the following integral equation system of renewal type: + X]Mj{x-y)dF,j(y),isI,
M,(x)^(l-o,)
(5.22)
ye/ 0
whose (5.21) is the unique solution. In the case of all the d.f Mf, being proper (i.e. limM.(x) = 1,/ = 1,...,w) and if, moreover, the following mean lifetimes exist: 00
M. = jxdM.(x) 0), we can associate a classical random walk, i.e. a sequence of i.i.d. random variables: (Uj'\l>0) (6.4) with
(6.5)
Uj^'= I X„.
From Proposition 2.1 and the Remark 2.1, it follows that these random variables have a mean ju given by 1 '" l^jj=—Y.^iVi^ (6.6) so that they are positive or negative for all j , depending on the sign of ju defined by m
M = Yj^ir]i-
(6-7)
i+\
Following Spitzer (1957) and Feller (1971) we will say that a random walk drifts to +00 (resp. -co) iff P(limsup{ft;: S„{co) < 0}) = 0, (6.8) (P(lim inf {6;: S„ {co) > 0}) = 0) n
and that it is oscillating iff P(limsup{^: S^{co) 0}) = 1. n
(6.9)
"
We then get the following theorem concerning the asymptotical behaviour of the semi-Markov random walk {Sn)> Theorem 6.1 If the semi-Markov random walk (Sn) has an irreducible M.C, and all the unconditional means 7]. J el are finite and, then if ju is null and if for one h P(Ul'^=0) 0} = X;'y^(limsup{5„ > 0} \J, = j), y=i
n
n
the first part of the theorem is proved. (ii) If // >0, let us introduce the following events: E^^io):
S„ n
Mjj ^Mjj >,n>Q. nijj Inijj
It is clear that for every positive integer n: {oy:S„{w)J^7,x>0,
(7.8)
n>l
and so P(H, = oo,y^_ ^y|y„ = / ) ^ j.„p(y^^ = oly^ = /) = Sj,P(F, = a,\j^ = i) = Sj,(l - limH'ix)).
^^'^^
From relations (7.8) and (7.9) we also deduce immediately that: P(j,-jJo-i)-\'^ „,, . . „ (7.10) ^ ^ \\-hmH (x), 7 = 0. We define the second strict ascending indice as tho first strict ascending indice
Functionals of (J-X) Processes
223
of the semi-Markov random walk induced by the following sequence: {^fl.,>^^,.2v..}, (7.11) or, equivalently, as the second indice n, if it exists, otherwise, we take oo, such that Sn>Sk,k=0,\,„„nA. (7.12) Of course, if the first indice F\ is infinite, the process of indices is considered as teminated. Otherwise, the second strict ascending indice is characterised by the couple {F^+F^Mx+H^) and for the occupied state, we write: ^I^JF,.F,' (7.13) Proceeding so for the following indices, we obtain the process of strict ladder indices
((//„,fj,^>0),
^^^^^
//o=0,fo=0, a.s. By construction, it is clear that this new process is a positive {J^ process eventually defective and characterised by the triple (m,p,H) where the matrix H is the matrix (7.15) H(x) = [//^(x)] with functions If^ defined by relation (7.8). The next proposition characterises i]\Q process of strict ladder indices, Proposition 7.1 If the semi-Markov random walk (Sn) has an irreducible M.C, and all the unconditional means r/.,ielare finite, then the positive {JyX) process: ((//„fj,«>0), ^^^^^ //o=0,^o=0.a.s. characterised by the triple (m,p,H) is defective or not following that iu0. Proof With the notation introduced in section 5, we have that: =/'(F„=+co|^„_,=/)
(7.17)
Now, from Proposition 7.1, we have that: F(F^ = +oo\j^ = 0 = 0 0, P(i^=+oo|Jo=0>Oo//Ol
^^^^^
//o=0,^o=0,a.s. characterised by the triple (w,p,H) is terminated for every initial distribution piffju 0 // < 0. (7.20) So, from relation (5.16), this implies that,for every state i, the probability Tj that the process ((H^,gJ,n > 0) terminates, is strictly positive. From this last result, we can deduce that on / , for the imbedded MC of the positive (J,X) process ((H^,g^),n > 0), all the states of the set / are transient and so that the only absorbing class is the set {0}. But from Chapter 3, we know that in this case, the absorption probability for this class is 1 and so: T.=iyieL D (7.21)
8 DISTRIBUTION OF THE SUPREMUM FOR SEMIMARKOV RANDOM WALKS Let us consider a semi-Markov random walk (SJ with an irreducible M.C. and all the unconditional means 77.,/e/finite. We are now interested in the distribution of the following supremum: (8.1) M = sup{S,,S,,„.}, Forju>0, under the assumptions of Proposition 7.2, it follows from this proposition that for all / of/ and all real x: = 0, (8.2) P(M<x\j,=i) This is also true for ju =0, as the positive (J,X) process ((H^,gJ,n > 0) is regular (see Fyke (1961a)) meaning that it has only a finite number of transitions on any time interval. Nor for /u 0 ) . From Proposition 7.2, we know that: hmM,.(x) = l , V / e / .
(8.3) for the process
(8.4)
X->oo
We also see that M.(0) = l-t;.,V/G/. (8.5) Nevertheless, to be useful, the "explicit" form (8.3) requires us to know the kernel of the positive {J,X) process ((H^,g^),n > 0) or the functions H"^ given by relation (7.8). Unfortunately, this is very difficult except in very particular cases. To avoid that, we can start from the following integral equations system of Wiener Hopf type given from an immediate probabilistic reasoning: ,^r ^
\l,]Mj(x-s)dQ,j(x),
x>0,
(8.6)
[ 0, X < 0. For m-l, we get the classical Wiener Hopf equation: M(X):
JM(x-s)dQ(x),
x>0,
(8.7)
0, x0) with values in / x M"^ such that: JQ =/,XQ P(J„
=0,a.s.JeI, =j,X„<x\(J„X,),k l) Consequently, the past influences the evolution of the process by the presence of r^_j andn in (1.1).
228
Chapter 6
Definition 1.2 The sequence Q=[^"~^^Q{sJ),n>\\
of mxm
matrices of
measurable functions of NQ x R"^ x R"^ h-> [0,l] where:
0) defined as
[ 0,N(t) 0 are independent of ^ or equivalently that Q(s,t) = F(s)-F(s,t). (1.22) Let us point out that, in this case, relations (1.18) become: (/>,j{s,t) = S,j(l-H,isj))^^l^,j{uj)Q,,^^^^^
(1.23)
If moreover, we have P(^) = P,^>0, (1.24) then the kernel Q is called a partially non-homogeneous semi-Markov kernel (in short PNHSMK) defining a partially non-homogeneous Markov additive process (in short PNHMAP) [(J^,TJ,n>0) and a partially non-homogeneous semiMarkov process (in short PNHSMP) Z = (Z(t),t > 0). This family was introduced in a different way by Hoem (1972).
232
Chapter 6
1.2.2 Non-Homogeneous MC If the sequences ^''~'^P(^),V^>0 are independent of ^, then (y^,^>0)is a classical non-homogeneous MC (in short NHMC) 1.2.3 Homogeneous Markov Additive Process A PNHSMK Q such that F{s,t) = F(t-s),s,t >0,t-s>0, (1.25) is of course a classical homogeneous SM kernel as in Chapter 4. Let us say that the case of non-homogeneous Markov processes will be treated in the next section. 1.2.4 Non-Homogeneous Renewal Process For m=l, The CNHMAP of kernel Q is given by Q(s,t) = C"-'Ms,t)),s,t >0,t-s>0 and characterizes the sequence (X^,n> 0) with, as in (1.1), XQ =0,a.s., P(X^<x\x„k0,t-s>0,n>\, (1.28) it follows that XQ =0,a.s., .(„-,)... ^. ., 0-29) P(X^<x\x„k0,coeQ.
(1.33)
Let us remark that for the classical homogeneous SMP, relations (1.31) and (1.28) become:
X.{t,co) = — ^ " '
"
,?>0,6;€Q.
(1.35)
More particularly, for homogeneous continuous Markov processes for which we know that: y:.(0 = ^ ^ - ^ ' , / > o , / E / (1.36) with /li,/l2,...,/l^ are strictly positive known constants. Consequently, in this particular case, relations (1.34) become: A,(/,^) = / 7 , ^ ^ / , , y E / , / > 0 and
(1.37)
234
Chapter 6
A(^dy) = I^.,7G/,^>0.
(1.38)
For m=l, we get the classical (homogeneous) Poisson process for which: A(t,co) = I,t>0, (1.39) 1.3.2 Generalised Polya Processes Let us come back to CNHDRP and CNHRP defined in section 1.2.4. We know that a CNHDRP is defined by the kernel (1.31): Q(s,t) = C"-'^F(sMs,t >0,t-s>0
(1.40)
and that the sequence {X^,n > O) satisfies relations (1.27). Assuming that there exists the density sequence C"-''f(sMs,t>Oj-s>0, "-'q(s,t) = fa \
(1.41)
^-V(^,o=^^"-'^^(^,oL \dt
J
the intensity function related to the kernel Q is given by formula (1.34) giving here: ^(^) f(T t)
Ut,co)=. . j y Z
.,^>Q,^eQ.
(1.42)
Inversely, from this last relation, we get: Q,b,s>Q,
(1.46)
we get the family of Polya processes defined by De Vylder (1977) for which by (1.43): /
(«)F ( ? „ , 0 - l - e x p
N-(o+i«)/i
\ + st \ + St„j
(1.47)
The special case ^==0 has also a sense and gives the particular Polya process defined for example in Buhlmann (1970) for which then, by relation (1.31): X{t,(D) = a + bN{t\ (1.48) and by relation (1.43):
Non-Homogeneous Semi-Markov Processes
235
SO that, in this case, the sequence {X^,n > O) is a CNHRP giving for the special case b=0, a classical Poisson process. Let us point out that the Polya processes are used for modelling epidemiological phenomena in which of course the number of cases already observed, say n, has strong influence in the structure of the future cases.
2 NON-HOMOGENEOUS MARKOV CHAIN In section 1.2.2, we define the particular case of non-homogeneous Markov chains (NHMC) as a particular completely non-homogeneous Markov chain (CNHMC) for which the kernel (^''"^^P(^)) is independent of ^ so that we have:
'"-"? = ["->,]
(2.1)
with, from relation (1.5) '"-''p,=P(J„-j\j„.,-i),i,J^I or with Doob 's (1953) notations:
(2.2)