Controlled Diffusion Processes

Stochastic Mechanics Random Media Signal Processing and Image Synthesis Mathematical Economics and Finance Stochastic O...

Author: Nikolai Vladimirovich Krylov | A.B. Aries

168 downloads 1447 Views 14MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Stochastic Mechanics Random Media Signal Processing and Image Synthesis Mathematical Economics and Finance Stochastic Optimization Stochastic Control Stochastic Models in Life Sciences

Stochastic Modelling and Applied Probability (Formerly: Applications of Mathematics)

14

Edited by B. Rozovski˘ı G. Grimmett Advisory Board D. Dawson D. Geman I. Karatzas F. Kelly Y. Le Jan B. Øksendal G. Papanicolaou E. Pardoux

N. V. Krylov

Controlled Diffusion Processes Translated by A. B. Aries

Reprint of the 1980 Edition

Author Nicolai V. Krylov School of Mathematics 127 Vincent Hall University of Minnesota Minneapolis, MN 55455 USA [email protected] Managing Editors B. Rozovski˘ı Division of Applied Mathematics Brown University 182 George St Providence, RI 02912 USA [email protected]

ISBN 978-3-540-70913-8 DOI 10.1007/978-3-540-70914-5

G. Grimmett Centre for Mathematical Sciences University of Cambridge Wilberforce Road Cambridge CB3 0WB United Kingdom [email protected]

e-ISBN 978-3-540-70914-5

Stochastic Modelling and Applied Probability ISSN 0172-4568 Library of Congress Control Number: 2008934473 Mathematics Subject Classification (2000): 93E20, 60J60, 60H10, 60H20 Soft cover reprint of the 1980 edition Translated from the Russian Edition published by Nauka, Moscow 1977 c 2009 Springer-Verlag Berlin Heidelberg c 1980 Springer-Verlag New York Inc. This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Cover design: WMXDesign GmbH, Heidelberg Printed on acid-free paper 9 8 7 6 5 4 3 2 1 springer.com

Preface

Stochastic control theory is a relatively young branch of mathematics. The beginning of its intensive development falls in the late 1950s and early 1960s. ~ u r i that n ~ period an extensive literature appeared on optimal stochastic control using the quadratic performance criterion (see references in Wonham [76]). At the same time, Girsanov [25] and Howard [26] made the first steps in constructing a general theory, based on Bellman's technique of dynamic programming, developed by him somewhat earlier [4]. Two types of engineering problems engendered two different parts of stochastic control theory. Problems of the first type are associated with multistep decision making in discrete time, and are treated in the theory of discrete stochastic dynamic programming. For more on this theory, we note in addition to the work of Howard and Bellman, mentioned above, the books by Derman [8], Mine and Osaki [55], and Dynkin and Yushkevich [12]. Another class of engineering problems which encouraged the development of the theory of stochastic control involves time continuous control of a dynamic system in the presence of random noise. The case where the system is described by a differential equation and the noise is modeled as a time continuous random process is the core of the optimal control theory of diffusion processes. This book deals with this latter theory. The mathematical theory of the evolution of a system usually begins with a differential equation of the form

at = f(t,xt) with respect to the vector of parameters x of such a system. If the function f (t,x) can be measured or completely defined, no stochastic theory is needed. However, it is needed if f(t,x) varies randomly in time or if the errors of measuring this vector cannot be neglected. In this case f(t,x) is, as a rule,

representable as b(t,x) + a(t,x)[, where b is a vector, cr is a matrix, and is a random vector process. Then

5,

It is convenient to write the equation in the integral form

where xo is the vector of the initial state of the system. We explain why Eq. (2) is preferable to Eq. (1). Usually, one tries to choose the vector of parameters x, of the system in such a way that the knowledge of them at time t enables one to predict the probabilistic behavior of the system after time t with the same certainty (or uncertainty) to the same extent as would knowledge of the entire prior trajectory xs (s 5 t). Such a choice of parameters is convenient because the vector x, contains all the essential information about the system. It turns out that if x, has this property, it can be proved under rather general conditions that the process 5, in (2) can be taken to be a Brownian motion process or, in other words, a Wiener process w,. The derivative of 5, is then the so-called "white noise," but, strictly speaking, 5, unfortunately cannot be defined and, in addition, Eq. (1) has no immediate meaning. However, Eq. (2) does make sense, if the second integral in (2) is defined as an Ito stochastic integral. It is common to say that the process x, satisfying Eq. (2) is a diffusion process. If, in addition, the coefficients b, o of Eq. (2) depend also on some control parameters, we have a "controlled diffusion process." The main subject matter of the book having been outlined, we now indicate how some parts of optimal control theory are related to the contents of the book. Formally, the theory of deterministic control systems can be viewed as a special case of the theory of stochastic control. However, it has its own unique characteristics, different from those of stochastic control, and is not considered here. We mention only a few books in the enormous literature on the theory of deterministic control systems: Pontryagin, Boltyansky, Gamkrelidze, and Mishchenko [60] and Krassovsky and Subbotin [27]. A considerable number of works on controlled diffusion processes deal with control problems of linear systems of type (2) with a quadratic performance criterion. Besides Wonham [76] mentioned above, we can also mention Astrom [2] and Bucy and Joseph [7] as well as the literature cited in those books. We note that the control of such systems necessitates the construction of the so-called Kalman-Bucy filters. For the problems of the application of filtering theory to control it is appropriate to mention Lipster and Shiryayev [51]. Since the theory of linear control systems with quadratic performance index is represented well in the literature, we shall not discuss it here.

Control techniques often involve rules for stopping the process. A general and rather sophisticated theory of optimal stopping rules for Markov chains and Markov processes, developed by many authors, is described by Shiryayev [69]. In our book, problems of optimal stopping also receive considerable attention. We consider such problems for controlled processes with the help of the method of randomized stopping. It must be admitted, however, that our theory is rather crude compared to the general theory presented in [69] because of the fact that in the special case of controlled diffusion processes, imposing on the system only simply verifiable (and therefore crude) restrictions, we attempt to obtain strong assertions on the validity of the Bellman equation for the payoff function. Concluding the first part of the Preface, we emphasize that in general the main aim of the book is to prove the validity of the Bellman differential equations for payoff functions, as well as to develop (with the aid of such equations) rules for constructing control strategies which are close to optimal for controlled diffusion processes. A few remarks on the structure of the book may be helpful. The literature cited so far is not directly relevant to our discussion. References to the literature of more direct relevance to the subject of the book are given in the course of the presentation of the material, and also in the notes at the end of each chapter. We have discussed only the main features of the subject of our investigation. For more detail, we recommend Section 1, of Chapter 1, as well as the introductions to Chapters 1-6. The text of the book includes theorems, lemmas, and definitions, numeration of which is carried out throughout according to a single system in each section. Thus, the invoking of Theorem 3.1.5 means the invoking of the assertions numbered 5 in Section 1 in Chapter 3. In Chapter 3, Theorem 3.1.5 is referred to as Theorem 1.5, and in Section 1, simply as Theorem 5. The formulas are numbered in a similar way. The initial constants appearing in the assumptions are, as a rule, denoted by Ki,ai.The constants in the assertions and in the proofs are denoted by the letter N with or without numerical subscripts. In the latter case it is assumed that in each new formula this constant is generally speaking unique to the formula and is to be distinguished from the previous constants. If we write N = N (Ki,Gi,. . .), this means that N depends only on what is inside the parentheses. The discussion of the material in each section is carried out under the same assumptions listed at the start of the section. Occasionally, in order to avoid the cumbersome formulation of lemmas and theorems, additional assumptions are given prior to the lemmas and theorems rather than in them. Reading the book requires familiarity with the fundamentals of stochastic integral theory. Some material on this theory is presented in Appendix 1. The Bellman equations which we shall investigate are related to nonlinear partial differential equations. We note in this connection that we do not

assume the reader to be familiar with the results related to differential equation theory. In conclusion, I wish to express my deep gratitude to A. N. Shiryayev and all participants of the seminar at the Department of Control Probability of the Interdepartmental Laboratory of Statistical Methods of the Moscow State University for their assistance in our work in this book, and for their useful criticism of the manuscript. N. V. Krylov

Contents

Notation

1 Introduction to the Theory of Controlled Diffusion Processes 1. The Statement of Problems-Bellman's Principle-Bellman's Equation 2. Examples of the Bellman Equations-The Normed Bellman Equation 3. Application of Optimal Control Theory-Techniques for Obtaining Some Estimates 4. One-Dimensional Controlled Processes 5. Optimal Stopping of a One-Dimensional Controlled Process Notes

2 Auxiliary Propositions Notation and Definitions Estimates of the Distribution of a Stochastic Integral in a Bounded Region Estimates of the Distribution of a Stochastic Integral in the Whole Space Limit Behavior of Some Functions Solutions of Stochastic Integral Equations and Estimates of the Moments Existence of a Solution of a Stochastic Equation with Measurable Coefficients Some Properties of a Random Process Depending on a Parameter The Dependence of Solutions of a Stochastic Equation on a Parameter The Markov Property of Solutions of Stochastic Equations Ito's Formula with Generalized Derivatives Notes

3 General Properties of a Payoff Function 1. Basic Results 2. Some Preliminary Considerations 3. The Proof of Theorems 1.5-1.7

Contents

4. The Proof of Theorems 1.8-1.11 for the Optimal Stopping Problem Notes

4 The Bellman Equation 1. Estimation of First Derivatives of Payoff Functions 2. Estimation from Below of Second Derivatives of a Payoff Function 3. Estimation from Above of Sewnd Derivatives of a Payoff Function 4. Estimation of a Derivative of a Payoff Function with Respect to t 5. Passage to the Limit in the Bellman Equation 6. The Approximation of Degenerate Controlled Processes by Nondegenerate Ones

7. The Bellrnan Equation Notes

5 The Construction of &-OptimalStrategies 1. &-OptimalMarkov Strategies and the Bellman Equation

2. &-OptimalMarkov Strategies. The Bellman Equation in

the Presence of Degeneracy 3. The Payoff Function and Solution of the Bellman Equation: The Uniqueness of the Solution of the Bellman Equation Notes

6 Controlled Processes with Unbounded Coefficients: The Nonned Bellman Equation 1. Generalizations of the Results Obtained in Section 3.1 2. General Methods for Estimating Derivatives of Payoff Functions 3. The Normed Bellman Equation 4. The Optimal Stopping of a Controlled Process on an Infinite Interval of Time

5. Control on an Infinite Interval of Time Notes

Appendices 1. Some Properties of Stochastic Integrals 2. Some Properties of Submartingales

Bibliography

Notation

Ed denotes a Euclidean space of dimension d with a fixed orthonormal basis, xi the ith coordinate of the point x E Ed ( i = 1,2, . . . ,d), xy = (x,y) the scalar product of vectors x, y E Ed, and x2 = xx the square of the length of x, 1x1 = p. a = (a") denotes a matrix with elements aij, a* the transpose of the matrix a, ay the vector equal to the product of the matrix a and the vector y, xay = (x,oy); tr a denotes the trace of the square matrix a, det a the determinant of the matrix a, and

where /loll is said to be the matrix norm of a. vfi = av/axi, gradxu is the vector with the coordinates vXi,and vxixj = a2v/ax' ax'. If a is a matrix of dimension d x dl, and b is a d-dimensional vector, then

where the matrix (aij) = iaa*. dij denotes the Kronecker delta, x,,, = xr(x) is the indicator of a set T, that is, the function equal to unity on r and equal to zero outside T. X~,,,, is the graph of the function x, given on [O,T]. z A t = min(z,t), t+ = t+ = +(It1 t),

+

r means equal by definition.

Notation

(SZ,9,P) is a probability space; SZ denotes a set whose points are denoted by the o with indices or without, 9is a o-algebra of the subset SZ, and P is a probability measure on 9. M< denotes the mathematical expectation of a random variable xs)dws,

(1)

where o(a,y), b(a,y) are given functions of y E Ed and of a control parameter a, x is the initial value of the process x,, w, is a dl-dimensional Wiener process, and dl an integer. Naturally, b(a,y) is a d-dimensional vector: b(a, y) = (bl(%y), . . . ,bd(a,y)); o(a,y) is a matrix of dimension d x dl, 4% Y)= (oij(a,y)). We denote by A the set of admissible controls, i.e., values of the pararneter a. Choosing appropriately the random process a, with values in A we can obtain various solutions of Eq. (1). We can thereby "control" the process x, considered. This gives rise to the question as to whether there exists a solution of Eq. (1) for the process {a,) chosen, and if this is the case, whether it is unique, that is, whether the process {x,) can be defined uniquely after {a,) has been chosen. We put off considering these questions and finding answers to them until later. From the practical viewpoint, it is reasonable to consider that the values of the control process a, at the time s are to be chosen on the basis of observations of the controlled process {x,) before time s. In other words, a, has to be a function of the trajectory xrp,., = {(t,xt):OI t I s): a, = a,(x~o,sl). Suppose that a cost functional is given for evaluating the control performance. Suppose also that on each trajectory of the process x, from the time t to the time t At the "cost" is f*t(x,) At + o(At), where f"(y) is the given function. Then the total loss for the control used is given by

+

for each individual trajectory of x,. Corresponding to the strategy a = {a,(x~o,,l)), the "mean" loss for the process x, with initial point x is given by vqx) = M

Somf".(x,) dt.

~,,~)) This gives rise to the problem of finding a strategy a0 = { a ~ ( ~ ~such that (for fixed x) vaO(x)= U(X)= inf va(x). (2) a

In the case where there exists no strategy a0 (the lower bound in (2) is not attained), we may wish to construct for each E > 0 a strategy a' = { a ~ ( ~ ~such ~ , , that ~ ) ) va"(x)I u(x) E. The strategy a&is said to be &-optimal for the (initial) point x; the strategy a0 is said to be optimal for the point x. The function v(x) is called a "performance function," determining which strategy will be of interest to us.

+

1. The Statement of Problems-Bellman's Principle-Bellman's

Equation

Setting aside temporarily the questions about the convergence of integrals which define pa and va(x), we show how to solve the problem of finding v ( x ) and aeusing Bellman's principle. Bellman's principle states that

-

for each t 2 0. To make the relation ( 3 ) clear we imagine that an interval of time t has elapsed from the initial instant of time. For the interval t the loss is given by

The trajectory of the process has reached a point, say y, at the instant of time t. What is to be done after the time t to minimize the total loss? Since the quantity (4) has already been lost, we should find out how to minimize the loss occurring after the instant of time t. We note that increments of a Wiener process after the time t, together with the point y, define completely the behavior of a trajectory of x, for s 2 t. The increments of ws after the time t do not depend on the "past" prior to the time t, and they behave in the same way as the pertinent increments do after the initial instant of time. Furthermore, coefficients of (1) do not depend explicitly on time. Hence obtaining as a trajectory after the time t, say a function ( y , : s 2 t), is equivalent to that of obtaining a function ( y , + , : s 2 0 ) as a trajectory after the initial instant of time (if we start from the point y). We note also that the loss f"(x) does not depend on time explicitly. Therefore, we can solve the problem of minimizing the loss after the time t, assuming that the trajectory starts from a point y at the initial instant of time. It can readily be seen that the mean loss after the time t, under the condition xi = y, cannot be smaller than v ( y ) and can be made arbitrarily close to v(y). Therefore, if we proceed after the time t in the optimal way, the mean loss during the entire control operation will be given by

-

-

In general, the quantity ( 5 ) is smaller than va(x). Even if there exists no optimal control, nevertheless we can get arbitrarily close to ( 5 ) by changing a, for s 2 t. Therefore the lower bounds of ( 5 ) and va(x) with respect to all strategies coincide, as stated in (3). Further, assume that v is a sufficiently smooth function. Applying Ito's formula to v(x,), we have1

Recall that x, = x.

1 Introduction to the Theory of Controlled Diffusion Processes

where

Therefore, it follows from Bellman's principle that

0 = inf {M u

= inf M a

[J: P ( ~ds,+) .(xi) J: [f g x S )+ LUsv(xt)]ds,

where we divide all the expressions by t and also we let t tend to zero, obtaining- thereby the equation inf [Lu(x)v(x)+ f"(x)] = 0. usA

Equation (6) is known as Bellman's differential equation for the optimal control problem under consideration. We started from vu(x)and arrived at v(x) and, then, at Eq. (6).We can do it in the backward direction, namely, we can show that if a function w satisfies Bellman's equation, the function w coincides with v. In doing so we can also see how to find optimal and &-optimalstrategies with the aid of Bellman's equation. For the function w inf [Law + f"] = 0. ~ E A

Therefore, -Law I f " ; using Ito's formula we obtain

We pass to the limit in the last inequality as t + oo,assuming that for the function w ( x ) and the process xi for any strategy a as t -+ oo Mw(x,) + 0,

M

J: p ( x s )ds + M Somf '(x,) ds.

This yieldsZ

We show, in turn, that w(x) 2 v(x). We assume that for each x the lower bound in (7)can be attained for some a = aO(x).We assume further that there We note that a sufficient condition for the inequality w 2 v to be satisfied is that the left-hand side of (7) be nonnegative.

1.

The Statement of Problems-Bellman's Principle-Bellman's Equation

exists a solution of the equation xp

=x

+ fi o(aO(x,O),x,O)dws + Ji b(aO(x,O),x,O)ds.

Since - ~ " ~ ( " ) ( x ) w=( xf )*O(")(x),by Ito's formula

= Mw(xp)

+ M fi f "O("~)(x~) ds,

from which it follows for a strategy aO = {aO(xs)}as t

+

co that

Therefore, w ( x ) = v(x) and a0 is the optimal strategy (for any point x). In the case where the lower bound in (7) cannot be attained, in order to prove the inequality w ( x ) 2 v(x) we take a function g(x) > 0 such that for an arbitrary x and a strategy a

For 8 > 0 we find the function aE(x)from the condition

and consider the strategy a&= {aE(x,)).Let xi be a process corresponding to the strategy a&and starting from a point x. By Ito's formula

Therefore, w ( x ) 2 va"(x)- E 2 v(x) - E, i.e., we have again that v(x) = w ( x ) and the strategy a&is &-optimal(for any point x). In fact, Bellman's equation provides a technique for finding the performance function v ( x ) as well as optimal and &-optimalstrategies. We note that the &-optimalstrategies constructed above determine the choice of a control at time t on the basis of the instantaneous value of x, rather than the entire segment of the trajectory xI0,,,.In other words, these strategies are characterized by the fact that the control at a point y is always the same, namely, it is equal to ae(y),regardless in what way and at which instant of time the trajectory has arrived at the point y. Intuitive reasoning suggests that we could have restricted ourselves to the aforementioned strategies from the very beginning. Indeed, the knowledge of how the trajectory has arrived at the point y cannot help us, by any means, to influence the "future" behavior of the trajectory x, because increments of the process w,, which determine this behavior, do not depend on the "past." Furthermore, the "cost" we have to pay after the trajectory has arrived at


the point y is not a function of the preceding segment of the trajectory. If it is therefore advantageous, for any reason, to use a control at least once after the trajectory has reached the point y, it will be advantageous, for the same reasons, to use this control each time when the trajectory reaches y. A strategy of the form {a(x,))is said to be Markov since its corresponding process is Markov; the behavior of the latter after the instant of time t depends only on the position at the time t and does not depend on the "prehistory." Therefore, one need seek optimal and &-optimalstrategies only among Markov strategies. However, it turns out that in order to prove the validity of our heuristic arguments justifying Bellman's equation, we need to consider all possible strategies. For example, in explaining Bellman's principle it was important that the controls applied after an instant of time t did not depend on the preceding ones. It becomes possible to apply various controls at a point y before time t as well as after time t. It turns out that it is sometimes convenient to broaden the notion of a strategy. Taking a strategy a = a , ( ~ ~ , , ,and ~ ) solving Eq. ( I ) , we obtain a process x, dependent on the trajectory w,:x, = X,(W[,,,~). Putting this solution into the expression for a,, we write a, as P,(W~,,~~). It is clearly desirable to include all processes P, = ~,(w[,,,~) with values in A in the set of the strategies considered, and to determine the resulting controlled process we solve in fact the equation

It can be seen that admitting strategies of the form P,(W[,,,~) is equivalent to being allowed to choose a control on the basis ofobservations ofthe process w,, the observations of the process w, providing us simultaneously with all the data about the process x,. It can also be seen that including these strategies in the set of all admissible strategies we leave intact the preceding arguments concerning Bellman's equation as well as Markov strategies. In particular, the inclusion of new strategies does not decrease the performance function, and can, thus, be approximated with the aid of Markov strategies. We make now a few remarks on the structure of Bellman's equation which may simplify the notation in some cases. Equation (6) can be written in a more expanded form as inf

asA

[

i,j= 1

d

aij(a,x)ux.,(x)

+ i=1 bi(w)ux.(x)+ f i x ) 1

1

= 0,

(8)

where x assumes values from that space in which the controlled process (1) assumes its values, and bi(a,x)is the velocity of the deterministic component of the motion of the ith component of the process, with the position of the process at a point x and with a control a applied. The matrix a(a,x) = (aij(a,x))= $o(a,x)o*(a,x)is symmetric; i.e., a* = i(oa*)* = too* = a and nonnegative definite, i.e., (al,1) = 3(0*1,0*1) = 310*112 2 0 and, in addition,

2. Examples of the Bellman Equations-The

Normed Bellman Equation

characterizes the diffusion component of the process. Furthermore, as can easilv be seen,

therefore add(a,x)= 0 for each x, a if and only if adk(a,x)= 0 for all k = 1, . . . , d l , x, a, i.e., if the last coordinate of the process has no diffusion component. In this case aid(ct,x)= 0, adi(a,x)= 0; the first summand in (8) becomes

1. Exercise Let o,(a,x), a2(a,x) be two square matrices of dimension d x d. To construct a new matrix 03(a,x)having d rows and 2d columns, suppose that the first d columns of the matrix a3(a,x)form a matrix a,(a,x) and that the last d columns form a matrix a2(a,x). We denote by ai(a,x)the matrices corresponding to oi(a,x)o:(a,x).Prove that

2. Examples of the Bellman EquationsThe Normed Bellman Equation The examples given in Section 2 are intended to show that in spite of the rather specialized form of the controlled system (1.1) and the performance functional many stochastic control problems can be reduced to the problem examined in Section 1. Suppose that we need to maximize M p a instead of minimizing it. Since - sup M a

Somfat(xt)dt

= inf M a

the payoff function v l ( x ) E sup M a

Som[- f Q ( x t ) ] dt,

Somf'.(xt)dt

satisfies the following Bellman equation:

+ (- f

= -SUP

sup [Lavl

+ f*]= 0.

0 = inf [L"(- v l ) asA

which yields

aeA

[Lav1 + f

~ E A

"I,

1 Intr-aduction to the Theory of Controlled Diffusion Processes

We note that in the minimization problem we derived Bellman's equation, (1.6), which contained inf; in the maximization problem (1.6) contains sup. In some cases, in order to ensure the existence of the functional pa, i.e., the convergence of the corresponding integral, one need introduce "discounting." For example, we consider the problem of finding vZ(x)= sup M a

So e - p ( x J dt, w

where x, is a solution of Eq. (1.1). If fa(x) is a bounded function, the integral in (2) exists. The multiplicative factor e-' is said to be a discounting multiplicative factor and can be interpreted as the probability that the trajectory of the process does not vanish before the instant of time t, and in fact we obtain the payoff f"f(x,)dt during the interval of time from t up to t + dt. We show how to reduce the last problem to the previous one. For x E Ed, y E (- C O , ~ )we , put f "(x,y) = e-yf "(x) and consider in Ed x El a control process whose first d coordinates move according to Eq. (1.1) and whose last coordinate is subject to the following "equation":

Let It is seen that vz(x,y)= e-yv2(x) and if our conclusions about the Bellman equation are valid, then

Putting vz(x,y)= e-Yv,(x), fa(x,y) = e-Yfa(x) into the last equality and canceling out by e-Y,we find SUP asA

[

I

d

1 a i j ( ~ , ~ ) ~ z x+i xCj ( ~bi(a,x)vzxi(x) ) - v ~ ( x+ ) fa(^) i=l

i,j=l

By the same token, for v3(x)= sup M a

J ' ~ ( x ,expl)

= 0.

Ji cas(xs)dsJdt,

where ca(x) is the given function of (a,x), we arrive at the equation

+ fa] = 0,

sup [LUv3- cav3 UEA

with the help of the equation

for the last coordinate y,.

(3)

2. Examples of the Bellman Equations-The Normed Bellman Equation

We see that introducing the "discounting" factor exp[-So cas(xs)ds] results in the appearance in Eq. (4) of the expression - cau,, which was absent in (1).

1. Exercise Let bl(a,x) be a dl-dimensional vector. Introducing the additional coordinate

explain why S P :

M

Morn

P(X~)~XP{J; bl(as,xS)dws-

1

1'

satisfies the equation

I

/bl(as~xs)~2ds dt

2. Exercise Using Ito's formula, show that if a function a(x) furnishes the upper bound in (1) (or in (3) or in (4)) for each x, the strategy {~(x,))is optimal in the corresponding problem.

An important class of optimal control is the problems of optimal stopping, in which one needs to choose, in addition to the strategy a, a random stopping time z such that the mean of the functional

J; P(XJ

dt + Q(x,)

is maximized. Surely, having made the decision to stop at time z,we stop observing the process after this time z. Hence we have to make the decision whether to stop the process at time t or not, on the basis of observation of the process only up to the time t. In other words, we shall treat Markov times as stopping times. On the set where z = oo, we assume, as usual, that g(x,) = 0, so that if stopping does not occur, we obtain

It turns out that the problem of optimal stopping can be reduced to the problem mentioned above via the technique of randomized stopping. We will illustrate this by an example of stopping a Wiener process (in (1.1) dl = d, o is a unit matrix, b = 0). Furthermore, we assume that f"(x) does not depend on a: f"(x) = f (x). Defining the nonnegative process r, = r(wI0,,,),we prescribe a rule for randomized stopping of the process w,. Let the trajectory w, stop with probability rt At + o(At) during the interval of time from t up to t + At, under

1 Tntroduction to the Theory of Controlled Diffusion Processes

the condition that it does not stop before the time indicated. Stopping w, at the instant of time t we have (compare with (5))

Then, as is easily seen, the probability that stopping does not occur on an individual trajectory before the time t is equal to exp(- Sf, rs ds) (in particular, exp( - 1 ; rs ds) for t = co). Therefore, the probability that stopping does in fact, occur in the interval (t,t + At), is given by

Hence the expected payoff on an individual trajectory is given by

f: [ J, p,rc exp = JoW

g (x

rs ds] dt

+

:s

[ J:

f (x + ws)ds exp -

+ w,)r,exp[- J, rs ds] dt

rs ds]

Integrating the last expression by parts we have

Therefore, using the above technique of randomized stopping, we obtain in the mean with the aid of the process r, M

f: + [ J: P(x

w,)exp -

crsds] dt,

+

where f' = f rg, cr = r. It can be readily seen that if it is really advantageous to stop the trajectory x w, at time t with nonzero intensity of r,, then one can as well stop the process at time t with probability one. Therefore, "randomized stopping cannot give a better result than "nonrandomized" stopping. On the other hand, an instantaneous stopping rule can be approximated in a reasonable sense by means of randomized stopping, if we increase the stopping intensity of r, after the time z. Hence

+

= sup M r

J: + [ J; P(x

w,)exp -

crsds] dt.

2. Examples of the Bellman Equations-The Normed Bellman Equation

Similarly, in the general case

= Sup M a,r

:J

[ ~i

f at3rt(x,) exp -

cas*rs ds] dt,

where = f" + rg, casr = r. If we regard the pair (a,r) as one parameter, we can easily notice the similarity between functions v, and v,. Hence, we write for v, a Bellman equation which is similar to (4).We note, however, that v, does not satisfy a Bellman equation in many cases. The point is that the functions f".' and casrin (6) are not bounded as functions of one of the control parameters r. With this in mind, let us go back to the derivation of the Bellman equation (1.6). Taking the last limit before (1.6), as t + co, we assumed that f"sr

+

is close to f"O(x) LaOv(x)in some sense uniformly with respect to strategies a. In order this to be the case, it is necessary at least that the process xs does not move far away from the initial point x in a short interval of time. We therefore assume that the coefficients o(a,x), b(a,x) are bounded, and furthermore the function f"(x) is bounded as well. As we saw above, the scheme which involves "discounting" reduces to a scheme without "discounting" if we introduce an additional coordinate y, and the equation

For the scheme with "discounting" we have therefore the requirement that ca(x)be bounded. Therefore, if we wish to consider the controlled process

with the payoff function v, and unbounded o, b, c, f , we have to apply, in general, methods other than those given in Section 1. One of the methods to be applied is based on random change of time and enables us to go from the unbounded o, b, c, f to the bounded a, b, c, f . We take a positive function m(a,x) for which the expressions a, b, mb, c, = me, f, mf are bounded functions of (a,x). Let

-

-

-

6,

and suppose that because of some special features of a controlled process, we can consider only those strategies a for which I++ 0 , x f 0 , respectively. Furthermore, u'(r) < 0. Hence u actually satisfies (12),(1I ) , (lo),and (9).Therefore v satisfies (8). The above implies, because v ( x ) is smooth, that for any t (compare with (4) and (5)). Letting t go to infinity and taking advantage of Fatou's lemma and the fact that v is nonnegative, we arrive at (14). We have thus solved our main problem. However, it is desirable to know to what extent the estimate (14) is exact and whether it can be strengthened. In other words, we wish to prove that v ( x ) in (13)is in fact a payoff function. Using Bellman's equation we find the optimal control a(x). For x 1 = r, x2=...= x d = 0 , the upper bound in (8) can be attained for the same a as that in (9) and (10).As our investigation demonstrates, this upper bound in (10) can be attained by a diagonal matrix a(a) such that aii(a)= p, i 2 2, al'(cc) = v. The eigenvectors of this matrix, except for the first vector, are orthogonal to the x1 axis, with eigenvalues equal to p; the first eigenvector is along the x 1 axis, with eigenvalue v. Because of the spherical symmetry of the problem, the upper bound in (8) can be attained for a different x on a matrix a,, with one eigenvector being We assume that y

f

0.

3. Application of Optimal Control Theory-Techniques for Obtaining Some Estimates

21

parallel to x and corresponding to the eigenvalue v, and the remaining eigenvalues being equal to p. Therefore, the matrix a, is characterized by the fact that a,x = vx, axy = py if (x,y) = 0. Since any vector

and

we have

Therefore,

Assuming a(x) = f i ,we obtain the function a(x) for which a(a(x),x) (=ax) yields the upper bound in (8). We can easily determine a(x) by noting that the eigenvectors of a(x) are the same as those of a,, and that the eigenvalues are equal to and f i We have aij(x) = ,pjP + (fi- f i ) 1x1

q.

The function a(x)is smooth everywhere except for x take a strategy a0 = {a,(x,)),the equation

fi

= 0. Therefore, if

we

has a solution uniquely defined up to the time of first entry at the zero point. Applying Ito's formula to the above solution, we have whence, as t V(X)

-+

co,

= Mv(x,,)

+ lim Mv(x,)x,, t'W

+

= vaO(x) lim Mv(x,)x,,,,. f+

m

In order to prove that v(x) from (13)is in fact a payoff function, a0 is an optimal strategy, and the estimate (14) is therefore exact, it sufficies to prove that the last summand is equal to zero. Since v is a bounded function, we need only to show that z, is finite with probability 1. Let g,(x) = - x 2 R2. By Ito's formula we have

+


which implies, by Fatou's lemma, that Mz,2(v + (d - 1)p) l gl(x) and Mz, < co. We make a few more remarks on the payoff function v(x) and the optimal process (15). Let y > 0, i.e., (d - 1)p < v, and let D(0) = S1\{O). From the equality vaO(x)= v(x) we obtain

from which it follows that the process (15) reaches with nonzero probability the zero point before it reaches as,. Furthermore, this probability tends to 1 when the initial point of the process tends to zero. We emphasize the fact that the process x, is nondegenerate and has no drift.

4. One-Dimensional Controlled Processes We shall prove in Sections 4 and 5 that a payoff function is twice continuously differentiable arid satisfies Bellman's equation if a one-dimensional controlled process is nondegenerate for no strategy at all, and if, in addition, some assumptions of a technical nature can be satisfied. Furthermore, we justify in these sections the rule for finding &-optimalstrategies using Bellman's equation. We wish to explain the relationship between the theory presented in Section 1.4 and the theory of multidimensional controlled processes which will be discussed in the subsequent chapters. One-dimensional controlled processes constitute a particular case of multidimensional controlled processes. Hence a general theory can provide a considerable amount of information about a payoff function. Bellman's equation, and &-optimal strategies. However, taking advantage of the specific nature of one-dimensional processes, we can prove stronger assertions in some cases. At the same time, we should point out that the results given below do not include everything that follows from the general theory. Let A be a (nonempty) convex subset of some Euclidean space, and let (T(~,x), b(a,x), ca(x),fa(x) be real functions given for a E A, x E (- m,co). Assume that ca(x)2 0; o(a,x), b(a,x), ca(x),and f"(x) are bounded and satisfy a Lipschitz condition with respect to (a,x), i.e., there exists a constant K such that for all a, p E A, x, y E El

where, as usual, a(a,x) = &[a(a,x)I2.

4. One-Dimensional Controlled Processes

Furthermore, we assume that the controlled processes are uniformly nondegenerate, that is, for some constant 6 > 0 for all a E A, x E El Let a Wiener process (wt,Ft) be given on some complete probability space (0,9,P), and let a-algebras of Ftbe complete with respect to measure P. 1. Definition. By a strategy we shall mean a random process a,(w)with values in A, which is progressively measurable with respect to the system of oalgebras of We denote by ?I the set of all strategies. x":

To each strategy a E ?I and a point x we set into correspondence a solution of the equation

By Ito's theorem the solution of Eq. (1) exists and is unique. We fix numbers r, < r, and a function g(x) given for x = r,, x = r,. We denote by za*" the first exit time of x?" from (rl,r2) and we set

V(X)= sup vU(x). ae9(

We shall frequently need to write mathematical expectations of the expressions which include repeatedly the indices a and x,where a is a strategy, and x is a point of the interval [rl,r2]. We agree to write these indices as subindices or superindices or the expectation sign. For example, we write M: S',f*t(x,) d t instead of f"'(xy) dt, etc. In addition, it is convenient to introduce the following notation: qt.' = ex:') ds.

1

The definition of va(x) thus becomes the following:

The definition given above of the strategy enables us to use the information about the behavior of the process w, in controlling the solution of Eq. (1). From the practical point of view, this situation is not natural. Hence we shall consider some other control techniques. Let C[O,co) be the space of continuous real functions x,, t E [O,co); let Jlr, be the smallest o-algebra of subsets of C[O,co), which contains all sets of the form {xl ,,: x, I a), s I t, a E (- co,co).


2. Definition. A function a,(x[,,,,) = at(xI0,,,)with values in A, given for t E [O,co), x[,,,) E C[O,co),is said to be a natural strategy admissible at the point x E [r1,r2]if this function is progressively measurable with respect to M, and if there exists at least one solution of the stochastic equation

which is Ft-measurable for each t. We denote by %,(x) the set of all natural strategies admissible at a point x. To each strategy a E 'LIE(x)we set into correspondence one (fixed)solution x?" of Eq. (2).

3. Definition. A natural strategy a,(~[,,,~) is said to be (stationary) Markov if a , ( ~ ~ , ,= , ~a(x,) ) for some function afx). We denote by %,(x) the set of all Markov strategies admissible at a point x. Note that to each natural strategy a,(~[,,,~) admissible at x, we can set into correspondence a strategy E 2I such that x:" = xf,". In fact, let us take a solution x,(w) = x;,"(w) of Eq. (2), and let us assume that Pt(w)= a , ( ~ ~ , , ~ ~It( wis)seen ) . that {P,) is a strategy and that the equation dx, = o(p,,x,)dw, + b(B,,xt)dt with the initial x , = x given can be satisfied for x, = x?". By the uniqueness theorem this equation has no other solutions; therefore, xf," = x:". It is clear that the inclusions 'UM(x)c %,(x) c 'U have precise meaning. we take a function a(x) with values In order to show that 'U,(x) # in A such that Ia(x) - a(y)l I Nlx - yl for all x, y for some constant N. Since the composition of functions satisfying a Lipschitz condition satisfies also a Lipschitz condition, there exists a solution of the equation

a,

therefore, { ~ ( x , )E)'U,(x). In the same way as we introduced the function v(x) for computing the upper bounds on the basis of %,(x), %,(x), we introduce here the functions v(,,(x), v(,,(x). It is seen that

4. Definition. Let E 2 0. A strategy a E 'U is said to be &-optimalfor a point x if v(x) I va(x)+ E.A 0-optimal strategy is said to be optimal.

Our objective is to prove the following theorem. 5. Theorem. V ( M , ( X ) = v ( ~ ) ( = x ) v(x)forx E [rl,r2],v(r,) = g(rl),v(r2)= g(r2), v(x)and its derivatives up to and including the second order are continuous on


[r1,r2],5and v"(x)satisfies a Lipschitz condition on [r,,r2]. For all x

+

sup [a(u,x)vt'(x) b(u,x)vl(x)- ca(x)v(x)+ f"(x)] = 0. asA

E

[r1,r2],

(3)

Furthermore, v is the unique solution of (3) in the class of functions which are twice continuously diflerentiable on [r1,r2]and equal to g at the end points of this interval. In order to prove the theorem we need four lemmas and additional notation. Let

+

F[u] = F[u](x)E sup [Lau(x) f"(x)] = F(x,u,ul,u"), aeA

6. Lemma. Let a(x),b(x),c(x),f ( x ) be continuous functions on [rl,r2]and let a 2 6, c 2 0, (a] lbl IK on this interval. Then there exists a unique function u(x)which is twice continuously diferentiable on [rl,r2],is equal to g at the end points of the interval [rl,r2],and is such that for all x E [r1,r2]

+ + IcI

+

a(x)ul'(x) b(x)ur(x)- c(x)u(x)+ f ( x )= 0.

(4)

Furthermore, IIu"llB

+ IIu'IIB + llullB

and, for g(rl) = g(r2)= 0,

llull~

N211f

Nl(llf

11s f 1)

ll-Yl9

where N , depends only on r,, r2, 6, K , g(rl), g(r2);N 2 depends only on r,, r2,6, K.

PROOF.Assertions of this kind are well known from the theory of differential equations (see, for example, [46]).Hence we shall just sketch the proof. First by considering instead of the function u the function u - $ in which $ is linear on [rl,r2] and $(ri) = g(ri),we convince ourselves that it suffices to prove the lemma for g = 0.Further, we can define a function y(x) explicitly By definition, v1(r1)(vU(rl))is assumed to be equal to the limit vf(x)(vU(x))as x fine v1(r2), v"(r2) in a similar way.

1 r,. We de-


such that the replacement of the unknown function with the aid of the formula u(x) = E(y(x))turns Eq. (4)into Dividing both sides of the last equation by a,, we arrive at the equation satisfying the boundary condition E(0) = E(1) = 0. Note that c,(y) 2 0 in (7). Analogous properties of the solution (4) can readily be derived from the properties of (7). Therefore, it suffices to prove our lemma for Eq. (7). Let g,(x,y) = ( x A y)(l - x v y) and for il > 0 let SA(X,Y) =

u""

1

&sh f i

s h f i ( x ~y)sh&(l

- x v y).

Elementary computation indicates that the combination of the relations - AE = - h, E(0) = E(1) = 0 is equivalent to the following:

In addition, (7) is equivalent to E" - ilE = (c, - A)E - f, and therefore to the equation

Let A = IIc211B[0,1]; then IITAul - TAu211B[0,11 5

llul

- u 2 1 1 B [ 0 , 1 ] ~max S o l g A ( x , ~ ) = d ~&llul - uZ/lB[O,l].

As can easily be verified,

Consequently, T Ais a contraction operator; Eqs. (8),(7),and (4)have unique solutions satisfying zero boundary conditions. We deduce the estimates (5) and (6) for the solution of (7).It follows from (8)that IIEllBIO,l]

max max gA(x,~)llf2/191[0,1] + &llEllB[0,l], x

Y

from which we find IIEllBro,ll 5 Nil f2119110,1,.Further, from (7)we obtain an [ ~ , ~ ] we . obtain the estimate E1 using the represenestimate for I l ~ ' ~ l l ~Finally, tation E'(x) = J", iil'(y)dy,where x, is a point in [0,1] at which ii' = 0. We have thus proved the lemma. 7. Lemma. There exists a constant N depending only on r,, r,, 6, K , and such that Mz: 5 N for each a E a,x E [rl,rz]. In particular, va(x) and v(x) are Jinite functions.


PROOF.We can consider without loss of generality that r1 = -r,. Let

Regardless of the fact that w(x) is the difference between two nondifferentiable functions, we can easily verify that w(x) is twice continuously differentiable and that for each a 2 6, b E [- K,K], x E [rl,r2]

In addition, w 2 0 on [rl,r2],w(ri)= 0. By Ito's formula, for each a E x E [r1,r2],t 2 0

a,

from which we conclude, using properties of the function w, that w(x) 2 M:(z A t ) and, as t -+ co,w(x) 2 M:z. 8. Lemma. Let the function a(x) satisfy a Lipschitz condition. We define a Markov strategy cc,(xl,,,,) by cct(xI,,,,) = a(x,). If f ( x ) is continuous on [rl,r2], the function

is twice continuously digerentiable on [rl,r2] and is the unique solution of the equation

+

La(")(x)u(x) f ( x )= 0,

x

E

[r1,r2],

(9)

in the class of twice continuously dlferentiable functions which are equal tog at the end points of Lrl,r2]. In particular, L"(")(x)v"(x) f "(x)= 0. Moreover, if some function w(x)has two continuous derivatives on [rl,r2],w(q) = g(ri)and

+

we have w 5 v"

+ ~ l l h l lwhere ~ ~ , N is the constant from (6).

PROOF.By Lemma 6, Eq. (9) satisfying the boundary conditions u(ri)= g(ri) has a smooth solution. Writing this solution as u , and applying Ito's formula to the expression ~ , ( x ~ , " ) e -we ~ : easily ' ~ find

from which the equality u,(x) = u(x)follows as t -+ co because zas",MET are finite, f , u , are bounded, and c" is nonnegative. We have thus proved the first assertion. In order to prove the second assertion, we set Law + f" = - h, and note that the function h, is continuous and La[w - r"] = - h,. Then,


according to the first assertion, we have

It remains to estimate u,(x). h: as well as h, is a continuous function. Therefore,u,(x) satisfies the equation Lau2= - h:, and, in addition, u2(ri)= 0. By Lemma 6 we have u2 i ~ l l h : l l ~Since ~ . h, 5 h, h: i h', Ilh:I19, I Ilhl12,, u2 i NllhllZl, thus proving the lemma.

9. Lemma. Let u(x),u,(x), u,(x) be bounded Bore1 functions on [rl,r2],E > 0. There exists a function a(x) with values in A, such that a(a(x),x)u2(x) + b(a(x),x)u,(x)- ca(")(x)u(x) + f"(")(x)+ E 2 F(x,u(x),ul(x),u2(x))for all x E [rl,r2].Furthermore, there exists a(x) satisfying the Lipschitz condition and also there exists a real-valued nonnegative function h(x) such that Ilhllst.,I E and for all x E [r1,r2]

+

+

+

a(a(x),x)u2(x) b(a(x),x)ul( x )- ca(")(x)u(x) f"(")(x) h(x) = F(x,u(x),u,(x),u,(x)).

PROOF.We fix some countable set {a(i))everywhere dense in A. Because a, b, c, f are continuous in the argument a

We conclude from the above that for each x

E

[rl,r2] there is an i such that

-

Next, we denote by i(x) the smallest value of i for which the last given inequality can be satisfied. It is seen that the (measurable) function a(x) a(i(x))yields the function stated in the first assertion. In order to prove the second assertion of the lemma, we extend the function i(x) outside [rl,r2] assuming i(x)= 1 for x $ [rl,r2].Let a,,,(x) = Ma(n A i(x

+ w,)) =

JzGSm-" a(n

-

A

i(y))e-(Y-x)2p'dt.

It is easily seen that a,,,(x) is an infinitely differentiable function. Furthermore, a,,Jx) E A due to the convexity property of the set A. Note that, as is well known (see, for example, [ l o ] ) ,for each measurable bounded function y(x) as t 0 the function My(x w,) + y(x) ( a s ) Hence a,,,(x) + a(n A i(x)) (as.) as t 10, and, clearly, a(n A i(x))-, a(i(x))as n + co. Defining

+


we obtain lim lim ht*"(x)I E

n-m

t10

ht,"(x)2 0.

Further, since u, u,, u, are bounded, the class of functions ht3" is class equibounded. Therefore, lim lim ~ ~ h t * nI ~&(r2 ~ 9 1- r,),

n + m t10

and t, n can be chosen so that (Iht3"ll,, I 2&(r2- r,). Thelemma has thus been proved. 10. Proof of Theorem 5. We use the so-called method of successive approximation in a space of strategies. This method, known as the Bellman-Howard method, enables us to find &-optimalstrategies and approximate values of a payoff function without solving nonlinear differential equations. We take as ao(x) any function with values in A satisfying the Lipschitz condition. We define the Markov strategy a, using the formula a,(~[,,~,) = aO(xt).Let vo(x)= vao(x).If a,,a,, . . . , a,, uo(x), v,(x), . . . , u,(x) have been constructed, we choose a function a,+ ,(x) such that it satisfies the Lipschitz condition and

,

Lan+lv,+ fOL"+l

+ h,+,

= F[v,],

(11)

1 191[1 ,r21

where h,+ is a function with a small norm: llhn+ I 1/(n + l ) ( n+ 2). Assume that v,+,(x) = van+'(x),where the strategy a,+, E %,(x) can be found with the aid of the function a,, ,. We prove that the sequence {v,(x)) has a limit and that this limit satisfies Eq. (3). We shall also prove that the limit of v, coincides with v. First, we investigate the behavior of v,, v; as n -+ co.Applying Lemma 8, we obtain F[v,] 2 0 since Lanv, + f"" = 0. We conclude from (11) that

Therefore

i.e., the sequence of functions

is monotone increasing. Furthermore, by Lemma 7 the totality of functions v, as well as u, is bounded, hence lim,,, u, exists. It is seen that v, has a limit as well. For


x

E

[rl,rZ]let iY(x)= lim v,(x). n+ m

By Lemma 6, it follows from the equality Lanv,

+ f"-= 0 that

where N does not depend on n. By the Lagrange theorem, Iv,(x) - v,(y)l I Nlx - yl. Therefore, the functions v, as well as u, are equicontinuous and uniformly bounded. By the Arzela theorem, some subsequence of the functions u, converges to the limit uniformly in x. Since the function u, increases as n increases, the entire sequence of u, converges to the limit uniformly in x. It then follows that v, converges to 5 uniformly in x. In particular. iY(x) is continuous on [rl,rz]. Further, using the Lagrange theorem, we derive from the uniform estimate llv;ll that Iv;(x) Nlx - yl, where N does not depend on n. By the Arzela theorem, the sequence {vi) is compact in the sense of uniform convergence on [rl,rz].Assume that {v;,) is a uniformly convergent subsequence and that iY1 is the limit of this subsequence. Taking the limit in the equality

V;(~)I

we have Therefore, 5' = a', {v;) has a single limit point, and v; -r 5' uniformly on [rl,r2]. In addition, lC'(x) - af(y)lI NIX - yl. Let us consider 5" and F[5]. We use Eq. (11). As was noted above, F[v,] 2 0 and f".+ + La.+la,+, = 0. Therefore, (11) yields

Dividing the last inequality by a(a,+ ,(x),x), we easily find

Also, we note that

From this representation of F it follows that the equation F = 0 is equivalent to r F1 = 0, and that 0 5 r + F1 I 6-la if 0 I F I E. Therefore, (12) yields

+

+

~ - ' K [ v ;- v;+, 6-1K(lvA 2 (v: F1(x,~,,v;)2 0.

+

+ Iv,

-

U , + ~ I ) + 6-1h,+l]


Integrating over x and letting n E'(x) - E'(rl)

+

+ lim

n+ m

co, F,(s,ds),uA(s))ds= 0

by virtue of the proven properties of v,, vh. Next we exploit a property of the function F,(x,y,p). Since the magnitude of the difference between the upper bounds does not exceed the upper bound of the magnitude of this difference, we have

$1 +

Iv, - El), and (13) In particular, IF,(s,v,,v;) - F,(s,E,E')l I (K/6)(lv;yields ~ , ( s , ~ s ) , T ( s ) )= d s0. (14)

Further, applying the elementary inequality

and the property of the upper bound noted, we find

from which we derive

In short, F1(x,$x),5'(x))satisfies a Lipschitz condition. Differentiating (14), we have first that E1'(x) F,(x,E(x),B'(x))= 0, and F[E] = 0; second, it follows from the equality v"" = -F,(x,E,E') that 5" satisfies a Lipschitz condition. In order to complete the proof of the theorem, we need only to show that for each twice continuously differentiable solution u(x) of the equation F[u] = 0 satisfying the boundary conditions u(rl) = g(rl), u(r2)= g(r2), the .. equalities u = v(,, = v(,, = v hold. First of all, if E > 0 and u is a function having the properties given above, by Lemma 9 there are u(x) and h(x) such that La(")u(x)+ f"(")(x)+ h(x) = 0

+


+

and Ilhl19, < E. This implies, by Lemma 8, that u I va NE and, therefore, < V ( M ) 5 v ( ~ ) 0. On the other hand, the equality F[u] = 0 for each a E N, x E [r,,r,] for t < z yields ~:'~; We apply Ito's formula to the expression ~ ( x ~ , " ) e - then

which gives the inequality u 2 va as t -t co, due to the finiteness of za,", M:z (Lemma 7), the boundedness off, u, the equality u(x2")= g ( x y ) , and 2 u. The theorem the nonnegativeness of P. Hence, u 2 v and u 2 v ( ~2, has thus been proved.

11. Remark. Since v" = v, v = limn,, van. Furthermore, the proof of the theorem provides us with a technique for finding &-optimalstrategies and the approximate value of the payoff function v. Using this technique, one has to know how to solve equations of the form La(")u(x)+ f '(.)(x) = 0 and + fB(")(x)2 F[u]x - E. The equality how to find P(x) such that LB(")u(x) v = lim,, ,vanenables us to estimate v",v',v as follows. By Lemmas 6 and 8,

where N , depends only on the maximum magnitudes of a(a,x), b(a,x), ca(x)g(rl), g(r,), r,, r,, 6. Therefore, by the Lagrange theorem, for all x, Y E Cr1721

Letting n -t m, we obtain a similar inequality for the function v. We divide the both sides of the inequality thus obtained by Ix - yl. Letting y go to x, we then have that for all x E [r1,r2]the sum Iv(x)l Iv'(x)I Ivl'(x)Idoes not exceed N , . Therefore,

+

+

12. Remark. (The smooth pasting condition.) At each point x

+

v(x - 0 ) = v(x O), vl(x - 0 ) = vl(x vl'(x - 0 ) = v"(x O),

+

+ O),

E (rl,r2),


which fact together with the boundary conditions v ( r l ) = g(rl), v(r2)= g(r2) helps us to find xo, e l , c 2 , d l , d,; if, for example, it has been proved that on some interval [ r l , x o ] the function v is representable as vl(x,cl,c2),and on [xo,r2] as v2(x,dl,d2),with v , and v , being known. 13. Exercise Let A

= [-

l,l],

Prove that the third derivative of the function v(x) = sup,,, at a point (r, r,)/2.

+

Mz"," is discontinuous

14. Exercise Using the inequality v r

+ F,(x,v,,vb) 2 0, prove that v" = lim,,, -

v; (as.).

We shall make a few more remarks on Theorem 5 proved above. We have first proved the existence of a solution of the equation F[u] = 0. Second, we have proved the fact that u is equivalent to v. We have thus obtained a theorem on the uniqueness of the solution of the equation F[u] = 0 satisfying the boundary condition u(ri)= g(ri).One should keep in mind that in the theory of differential equations the existence and uniqueness theorems are proved for a wider class of equations than that of equations of the type (3) (see [ 3 , 3 3 , 4 3 , 4 6 , 4 7 ] ) . The result of Exercise 13 shows that a payoff function need not have three continuous derivatives even if a, b, c, f is analytic in (a,x).We note here that if, for example, the function F,(x,y,p) has 10 continuous derivatives in (x,y,p), v has 12 continuous derivatives. We can easily deduce this by induction from the fact that the equation F [ v ] = 0 is equivalent to the equation V" + F1(x,v,vl)= 0. The next theorem follows from Remark 11 and the uniform convergence of vanto v. 15. Theorem. For each E > 0 there exists a function a(x) satisfying a Lipschitz condition and such that the Markov strategy cl,(xl0,,,)= cl(x,) is &-optimalfor all x.

If the payoff function v has been found, &-optimalMarkov strategies can easily be found with the aid of Lemmas 8 and 9. In fact, using Lemma 9, one can find a function a(x) satisfying a Lipschitz condition and such that In this case Ilhl191 I E and, by Lemma 8, u(x) I va(x)+ NE.


The problem with optimal strategies is much more complicated. To show that it is true we assume for E 2 0 that It is seen that for some x the set AO(x)can be empty. The next theorem is given without proof. 16. Theorem. (a) If a strategy a, is optimal for a point z E (rl,r,), the random vector a, E AO(xFZ) almost surely on a set {za3'> t) for almost all t. (b) A Markov strategy c l , ( ~ ~ = ~ ,a(x,) , ~ ) admissible for a point z E (rl,r,), is optimal for z if and only if a(x) E AO(x)for almost all x E (rl,r2). It follows from our theorem that the requirement for optimality imposes a very strict limitation on a strategy. The reader who has solved Exercise 13 can easily understand that the sets AO(x)(x # 0) are empty if A = (- 1,1), x;"= x + w, + a,ds, v(x) = sup, Mzz. Consequently, there is no optimal strategy in this case. The case where A = [- l,l] is more interesting (as in Exercise 13). Here AO(x)= (1) for x E [rl,(rl + r2)/2); AO((rl+ r,)/2) = [- 1,1], AO(x)= { - 1) for x E ((rl r2)/2,rZ]. In this case the function a(x) determining an optimal strategy is to satisfy (at least for almost all x) the following conditions: a(x) = + 1 for x E [rl,(r, + r2)/2]; a((r, + r2)/2)E [ - l,l], a(x) = - 1 for x E ((rl r,)/2, r,]. There arises the question of admissibility of the strategy a(x,) with the function a(x), that is, the question of solvability of the equation

fi

+

+

x,

=x

+ w,

+ sda(x,)ds

with a discountinuous drift coefficient. That the last equation is solvable can easily be proved with the aid of an appropriate transformation y, = f(x,) which reduces the initial equation to y, = f(x) o(y,)dw,, where o(y) satisfies a Lipschitz condition. Hence, there exists an optimal strategy in Exercise 13. Equations with coefficients which do not satisfy a Lipschitz condition have not been studied adequately (see, however, [75,78] and Section 11.6). In Section 1.1, we used Bellman's principle to deduce the Bellman equation. We now prove this principle.

+6

17. Theorem (Bellman's Principle). For all x E [rl,rz], a E a,let a Markov time ya9" I za3"be given. Then, for each function u(x) twice continuously differentiable on the interval [rl,r,], and such that F(u) = 0, we have on [rl,r2] the equality

which, in particular, holds for u = v.

5. Optimal Stopping of a One-DimensionalControlled Process

PROOF. We denote by ii(x)the right side of (16).Taking in (15)instead of z A t the expression y A t, we easily find that u(x)2 ii(x). On the other hand, for E > 0, a smooth function a(x) and a function h(x) such that Ilhl191 I E, we define the Markov strategy a,(xlO,,,)= a(x,). By Ito's formula

which yields We have used the fact that h 2 0 since 0 = F[u] 2 La(")u(x)+ f"(")(x)= -h(x). By Lemma 8, the last expectation as a function of x satisfies the equation La(")ul(x) h(x)= 0.

+

Therefore, by Lemma 6, the above expectation does not exceed Nllhl191 < NE.Finally, u(x) a iZ + NEfor each E > 0. Hence u I ii, which, together with the converse inequality proved before, yields the equality u = ii, thus proving the theorem. U 18. Exercise Prove that Eq. (16) will hold if we require only that f"(x) be measurable in x, continuous in a, and bounded with respect to (a,x).

19. Exercise For h E 9 1 [ r 1 , r Z let ]

u(x) = sup M:

Ji h(x.) dt.

Prove that lu(x)l< ~ l l h l l ~where , , N does not depend on h, x.

5. Optimal Stopping of a One-Dimensional Controlled Process

-

We consider the control scheme given in Section 1.4. To this end, we take the same set A and the functions o(a,x),b(a,x),ca(x),f"(x) satisfying the conditions given in Section 1.4. For simplicity of notation, we assume that ca(x) 0. In contrast to what we did in Section 1.4, we assume here that the function g(x) is given on the entire interval [r,,r,] and that g(x)is twice continuously


differentiable on this interval. As in Section 1.4, we denote by xFx a solution of Eq. (4.1), and assume that z = za9"is the time of first departure of ~ p ~ " ~ f r o m [r1,r21. For a Markov time y we set

and introduce a payoff function in the optimal stopping problem defined by w(x)= sup

V~,~(X).

as'll,~

In this section, we deal with the problem of finding a strategy a E ill and a Markov time y, such that V " , ~ ( X 2 )w(x) - E. 1. Definition. Let E 2 0. A Markov (with respect to {Ft}) time y to be &-optimalfor a point x if

= ya

is said

2 w(x) - E. sup vU,Y"(x) as%

A 0-optimal Markov time is said to be an optimal Markov time. We shall investigate the optimal stopping problem using the method of randomized stopping. We denote by %, the set of pairs (a,r), where a G ill, r = r,, is a nonnegative progressively measurable (with respect to {Pt}) process such that r,(m) 5 n for all (t,m). Let % = U, 23,. For a E A, r 2 0, let P r ( x )= f"(x) rg(x) and for (a,r)E 8 let

+

E,(x)

=

sup Ea*r(x), E(x) = lim En(x)= sup P r ( x ) . (ax)E 91n n-tm (a,r)s9

The main properties of functions v",(x)and the relationship of these functions with w(x) will be proved in the following lemma, whose first assertion justifies, in addition, the application of the method of randomized stopping. 2. Lemma. (a) w(x) = E(x) on [rl,r2]. (b) - E,,(x)I 5 (l/n)N for all x E [rl,rz],where N depends only on K and the function g. (c) E,(x) are twice continuously dzfferentiable on [rl,rz], E:(x) satisjes a Lipschitz condition, En(ri)= g(ri),and

Iw(x)

F[En] IIE;II~~rl,r2l

+ n(g-

En)+ = O

+ IIEAIIBrrl,r21 + Il~nll~rr1,r21

on [rl,r2].

(1)

I N, where N does not depend on n.

PROOF.The function i?,(x) is representable as the payoff function given in Section 1.4 if we take B, = A x [O,n] instead of the set A, and if we assume

5. Optimal Stopping of a One-Dimensional Controlled Process

that o(B,x) = o(a,x), b(P,x) = b(a,x),co(x)= r, fo(x) = f ' ( x )+ rg(x) for f i = (a,r) E B,. p becomes a control parameter, and 23, replaces a set of strategies %. Hence, Theorem 4.5 immediately implies the assertion on smoothness of E,(x) and the fact that E,(x) satisfies the corresponding Bellman equation. This equation is the following: 0=

+

+ b(cr,x)Ei(x)- rE,(x) + rg(x) f"(x)] sup [a(a7x)5L(x) asA,rs[O,nI

+

+

sup [LaEn(x) f '(x)] + sup r[g(x)- E,(x)] = F[En] n(g - En)+, asA . r s to,,] which proves (c). In order to prove (b), we write (1) as

+

+

sup [a(a,x)EL(x) b(a,x)Ek(x) f :(x)] = 0, asA

where f; = f" + n(g - En)+. From this, using Theorem 4.17, we have for all Markov times y = ya9"that

where En 2 g - (g - En)+ =- g , and f ; 2 fd;therefore, cn(x)n sup M: ae'll,?

[J;

fat(xt)dt

+ gn(xy

A

,)I.

On the other hand, if we take in (2) y = yo = yadX = inf {t :g(x;",")2 En(xFx)), then for 0 5 t 5 yy we have f,(x;,") = f ( x y ) and F,(x~.,",,) = g,(x;.,",,). Therefore, E,(x)

= sup Mz a e Pl

I

[J,Y"f ?xt) dt + gn(xYod A'

Comparing the last equality with (3),we obtain

A

.

which makes a crucial point in our proof. It turns out that if g is replaced with g, = g - (g - En)+, E,(x) will become a payoff function in the optimal stopping problem. Further, using the inequality connecting the magnitude of the difference between the upper bounds and the upper bound of the magnitude of the differences, we find

Therefore,


In order to estimate (g - En)+,we write (1) as follows:

where 4.17,

f":: = f a + n(g - v",), + n(En- g) + Lug, which En(x)- g(x) = sup M: u s e

yields, by Theorem

Ji e-"'f "nf (xt)dt.

fi 2 fa + Lug2 - (fa + Lag)- ; hence 1 En(x)- g(x)> -sup M e-"'dtsup(/. + L'~)- = --N, n

Note that

ae'U

Jc

u,x

1 g - En I - N , n

1 (g- En)+ y that


We divide the both sides of the last inequality by x - y and take the limit as y f x. Then, w" + F1(x,w,wt)I O(a.s.),i.e., F[w] I O(a.s.)on [rl,r2]. Further, let T = { x : w ( x )= g(x)). T is a closed nonempty (ri E T ) subset of the interval [rl,rz]. Let [pl,pz] be a subinterval not intersecting T . Then w(x) > g(x)for x E [ p l ,p3].Since En + w uniformly on [rl ,r2],the inequality Dn(x)> g(x) will be satisfied for x E [ P I ~ P ~beginning ], from some n. Therefore, (g - En)+ = 0, and by Lemma 2c F[&] = 0 on [p,,p,]. Hence 5: F,(~,i7~,5;) = 0 on [pl,p2] for sufficiently large n, which leads us, as in 4.10, to the assertion that w"(x) is continuous on [p,,pz], w" F1(x,w,wl)= 0 on [ ~ 1 , ~ 2 l . The above implies, in turn, two facts. First, w" = -Fl(x,w,w') on any subinterval not intersecting T : therefore w" outside r satisfies a Lipschitz condition. Second, F[w] 0 butside T . Finally, noting that it follows from w - g 2 0 and w - g = 0 on r that w' - g' = 0 on T n (rl,r2),we have the following theorem.

+

+

=

3. Theorem. (a) w together with its derivative is continuous on [rl,r2],w' is absolutely continuous, and w" is bounded on [rl,r2]. The function w" satisjes a Lipschitz condition outside the set r = ( x E [rl,rz]:w(x)= g(x)). (b) w 2 g, w(ri)= g(ri),F[w] I 0 (as.) and F[w] = 0 on [r,,r2]\r. (c) W' = g' on the set I' n (rl,rz). Next, we investigate &-optimalstrategies and optimal stopping times. 4. Theorem. (a) For a E a,x E [rl,rz] we denote by yo = y"sX the time ofjrst entry of a process x;"sXinto the set l';then, y o is an optimal stopping time. (b) We define for E > 0 a function a(x) satisfying a Lipschitz condition. In addition, we dejne a numerical function h(x) such that Ilhl121 I E and

(see Lemma 4.9). Also, we dejne a Markov strategy a using the formula a , ( ~ ~ , ,= , ~a(x,). ) Then, a is an NE-optimalstrategy for any point x and, moreover, W(X) I

Mi

N not depending on E and x.

[SF

j Y X t )dl

I

+ g (x,) + Ns,

PROOF.It is easily seen that only (7) is to be proved, and that the remaining assertions of our theorem follow from (7). It is obvious that (7) is to be proved only for x q! T . Let (pl,pz)be a subinterval not intersecting T . The function w is twice continuously differentiable and w" satisfies a Lipschitz condition on (p,,p2). Hence, the limit w"(p, + 0) and the limit wU(pz- 0) exist. Furthermore, it follows from (6) that on (p1,p2)


We apply Lemma 4.8 to the function w and the strategy a taking an interval [p,,p,] as an initial one. Then

A simple analysis of the deduction of (4.6) shows that the constant N can be taken to be the same for all p,, p, E [r1,r2]. It shows also that thus proving the theorem.

5. Exercise Let y"*" = y y be the first exit time of the process x:." from { x :w ( x ) > g(x) + E); then ya." is an &-optimalstopping time.

6. Exercise Let w ( x ) > g(x) on ( p l , p z ) and let w ( p i ) I g ( p i ) + E. Find the strategy a, which is &-optimalfor a point x , E ( p l , p , ) in the maximization problem

where z l is the first exit time from ( p l , p z ) (see Theorem 4.15). Then

that is, xl is a 2~-optimalstopping time and a is a 2~-optimalstrategy in the primary problem for the point x,.

We shall explain how the preceding results can be used for finding Eoptimal strategies and &-optimalstopping times. First, we find n such that [En - wl < ~ / (see 4 Lemma 2b). The function En is a solution of the equation sup

[a(a,x)E:(x)

a~A,r~[O,nl

+ b(a,x)EL(x)- rEn(x)+ rg(x) + f"(x)] = 0;

hence & can be found via the method of successive approximation in the space of strategies (see 4.10). Let 5; + En as m + co. We take m such that IEn(x)- Er(x)l I e/4 for x E [r1,r2]. Let

It can be easily verified that w(x) g(x) + E on [r,,r,]\G, and if (p ,,p2) c G, then w > g on (p1,p2).Therefore, the &-optimalstrategy for the points of [r,,r,]\G consists in instantaneous stopping. For the points of any


interval (pl,p2) c G the first exit time from G is an &-optimalstopping time (Exercise 5); we can find &-optimalstrategies using Exercise 6. In some cases it is difficult to do this. However, sometimes it is possible to define a function u(x) explicitly such that u(x) seems to coincide with w. In such cases the following theorem can be useful.

7. Theorem(0n Uniqueness). Let the function u(x) together with its Jirst derivative be dejined and continuous on [rl,r2]. Assume that ul(x)is absolutely continuous on [rl,rZ]. Finally, let u 2 g, u(ri) = g(ri),F[u] I 0 (as.). F(u) = 0 (as.) on the set {x E [rl,r2] :u(x)> g(x)}. Then u(x) = w(x). PROOF.We first prove that u I w. Let r = (x:u(x) = g(x)). Since g I w, it suffices to establish the inequality u I w on any subinterval (pl,p2) not intersecting r. Let the sequences pl be such that pl # pi,p f i pl, p i f p2. We immediately note that by the Lagrange theorem,

asn -+ co. Further, on (pl,pz)we have F[u] = 0 (as.), which yields u" Fl(x,u,ul) = 0, u" = - F1(x,u,uf)(a.s.). The expression F1(x,u,u1)is continuous in x, therefore u" coincides with a continuous function almost everywhere on (p1,p2). This readily implies that the function u" itself is continuous on (pl,pz). Next, we apply Theorem 4.5 to the function u on the interval [p'i,pg. Denoting by yn = yn,","the first exit time of x;," from [p;,p",, and noting that u(x;?) is equal to u(p;) or u(p",, we have

+

which implies, as n -+ co,that u(x) I w(x). Further, we prove the converse inequality. We denote by uU(x)a Bore1 function equal almost everywhere to a derivative of u'(x). We can take a function u"(x) such that the inequality F[u] I 0 is satisfied at all points [rl,rZ]. In fact, by assumption this inequality can be satisfied almost everywhere. We can redefine ul'(x) at the points at which F[u](x) > 0 for a randomly selected u", if we note that, in conjunction with the obvious inequality F(x,y,p,r) I Gr K(lp1 lyl 1) for r SO, for any x, y, p one can choose r I 0 so that F(x, y,p,r) I 0. In this case, Lau fOL I F[u] 5 0 everywhere on [rl,r2] for each a E A.

+

+ +

+


Since ul(x)is absolutely continuous, S::lurf(x)ldx< co. By Theorem 2.10.1, the fact that this condition is satisfied is sufficient for Ito's formula to be applied to the expression u(x;,"). Using Ito's formula and the inequalities u 2 g, Lau + f" 5 0 for each a E '$ and Ifor each Markov time y, we conclude that -

Therefore, u(x)2 sup,, V " , ~ (= X w(x). ) thus proving the theorem. The arguments in the above proof can prove as well the following theorem.

8. Theorem. Let a function u(x), together with its first derivative, be definite and continuous on [r,,r2].Assume that ul(x)is absolutely continuous on [r,,r2]. If u 2 g and F[u] 5 0 (a.s.), then u(x) 2 w(x) on [rl,r2]. In other words, w is the smallest function among those satisfying the inequalities u 2 g, F[u] 5 0 on [rl,rz]. 9. Exercise We take the function cU(x)from Section 1.4 and redefine II",~(X), letting

We encourage the reader to develop the actual arguments required in this section.

Notes Section 1. Girsanov in [25] was apparently the first who justified the application of Bellman's equation to some control problems, relying to a great extent upon differential equations theory. Using the same theory, Fleming in [13-161 made further steps in the development of optimal control theory; also, see Fleming and Rishel [17]. Speaking of the relationship between differential equations theory and optimal control theory, it is appropriate to draw the reader's attention to [20, 21, 30, 31, 35, 661. The control variables depending upon the entire "past" for processes with continuous time is first discussed by Fleming in [15]. Section 2. The normed Bellman equation was first introduced in [37]. The method of randomized stopping is developed in [29-31,371. The optimal stopping problem of a Markov (uncontrolled) process is discussed by Shiryayev in [69]. It is essential to note that the equations from optimal stopping theory are, in many cases, equivalent to the equations from the theory of differential (variational) inequalities; see Lions [49, SO], Lewy and Stampacchia [48], and Tobias [73,74]. The comparison of Exercise

Notes

4 with Example 3.7 in Chapter 1 in [SO] shows the relationship of other kinds between the optimal control theory of diffusion processes and the variational inequalitiestheory. Section 3. Other examples illustrating the application of optimal control theory for obtaining estimates which play an essential role in this theory may be found in [39,40]. Section 4. In a sense, our discussion consists of carrying the results and methods of Fleming in [13] over to the one-dimensional case. In the multivariate (d 2 3) case, using the methods mentioned above, it is possible to consider only optimal control problems in which the control parameter is not included in the diffusion coefficients; see [13-17,211. The reason for this is that it is impossible (in view of a well-known example due to N. N. Uraltseva) to prove a suitable analog of Lemma 6 for d 2 3. At the same time it is possible to work out a theory rather similar to that described in this section for the plane (d = 2), and, to allow the control parameter to be included in the coefficients of diffusion as well as drift; see [30, 31, 35,661. The method used in proving Theorem 5 differs, in fact, from the Bellman-Howard method to the extent that in the latter method, a,, is determined by the condition F[v,] = La-+lv,+ fb"+l. P. Mosolov drew the author's attention to the fact that the Bellman-Howard method follows the Newton-Kantorowich method for solving nonlinear function equations (see [3]).We note that the Bellman-Howard method applied to functional equations led to the quasilinearization method (see Bellman and Kalaba [5]). For one-dimensional control problems, see Mandl [52, 531, Prokhorov [64], Arkin, Kolemayev, and Shiryayev [ I ] , and Safonov [65]. Section 5. The methods developed in this section have been borrowed from [29,31]. Some hints as to how to find the set r can be found in Section 3.4 as well as in [56]. For the solution of equations arising from the sequential analysis, see also Shiryayev ~691.

,

Auxiliary Propositions

2

1. Notation and Definitions In addition to the notation given on pages xi and xii we shall use the following: T is a nonnegative number, and the interval [O,T] is interpreted as an interval of time; the points on this interval are, as a rule, denoted by t, s. D denotes an open set in Euclidean space, b the closure of D, and 2 0 the boundary of D. Q denotes an open set in Ed+,; the points of Q are expressed as (t,x) where t E El, x E Ed.d'Q denotes the parabolic boundary of Q (see Section 4.5). SR = {XE Ed: 1x1 < R),

CT,R= (0,T) x SR, HT = (0,T) x Ed.

C R = Cm,R,

If v is a countably additive set function, then Ivl is the variation of v, = ~ ( I v+ I v) is the positive part of v, and v- = )(IvI - V)is the negative part v. If T denotes a measurable set in Euclidean space, meas r is the Lebesgue measure of this set. For p 2 1 8 , ( r ) denotes a set of real-valued Bore1 functions f(x) on r such that

In the cases where the middle expression is equal to infinity, we continue to denote it by f llP,ras before. In general, we admit infinite values for various integrals (and mathematical expectations) of measurable functions. These values are considered to be defined if either the positive part or the

11

2 Auxiliary Propositions

negative part of the function has a finite integral. In this case the integral is assumed to be equal to m (- co) if the integral of the positive (negative) part of the function is infinite. For any (possibly, nonmeasurable) function f(x) on T we define an exterior norm in 2,(T), using the formula

+

where the lower bound is taken over the set of all Borel functions h(x) on T such that 1 I h on T. We shall use the fact that the exterior norm satisfies the triangle inequality: I]fl I f211p,r.Also, we shall use -+ 0 as n , co, there is a subsequence {n') for which the fact that if f,.(x) + 0 as n' -+ m (T-a.s.1. B(T) denotes the set of bounded Borel functions on I' with the norm

If

]I f n I P gr

+ f2l P,

]I

+ ]I

C(T) denotes the set of continuous (possibly, unbounded) functions on T. f is a smooth function means that f is infinitely differentiable. We say that f has compact support in a region D if it vanishes outside some compact subset of D. C;(D) denotes the set of all smooth functions with compact support in the region D.

We introduce A,,, . ..,., ( These elements are derivatives of f(t,x) along spacial directions. The time derivative is always expressed as (a/at)f(t,x). C2(D)denotes the set of functions u(x) twice continuously differentiable in D (i.e., twice continuously differentiable in D and such that u(x) as well as all first and second derivatives of u(x) have extensions continuous in D). C1,2(Q)denotes the set of functions u(t,x)twice continuously differentiable in x and once continuously differentiable in t in Q. Let D be a bounded region in Ed, and let u(x) be a function in D. We write u E W2(D)if there exists a sequence of functions un E C2(D)such that as n, m -+ co,where

Under the first condition of (1) and due to the continuity property of un, the functions in W2(D) are continuous in D. The second condition in (1)

1. Notation and Definitions

implies that the sequences u:ixj are fundamental in Yd(D). Hence there exist (Borel) functions ui, uij E Yd(D),to which uti, u:ixj converge in Yd(D). These sequences u",, utiXjconverge weakly as well to the functions given above. In particular, assuming cp E C," (D),and integrating by parts, we obtain

sD

cpu" dx = -

sD

cpxiundx,

Letting n + oo,we obtain

1. Definition. Let D c Ed, let v and h be Borel functions locally summable in D, and let l,, . . . , I, E Ed. The function h is said to be a generalized derivative (in the region D) of the function v of order n in the l,, . . . ,Indirections and this function h is denoted by u(,,, . . .,,( if for each cp E Cc(D) JD

cp(x)h(x)dx = (-

lr

JD

~(~)cp(,,, . . (I") dx.

In the case where the li direction coincides with the direction of the rith coordinate vector, the above function is expressed in terms of v,,, . . . ,,, U ( ~ l ). .. (In). The properties of a generalized derivative are well known (see [57,71,72]. We shall list below only those properties which we use frequently, without proving them. Note first that a generalized derivative can be defined uniquely almost everywhere. Equation (2) shows that ui = uxi in the sense of Definition 1. Similarly, uij = uXixj.Therefore, the functions u E W2(D) have generalized derivatives up to and including the second order. Furthermore, these derivatives belong to 2d(D). We assume that the values of first and second derivatives of each function u E W2(D) are fixed at each point. By construction, for the sequence un entering (I), 1lu:i - uxilld,D + 0,

1Iu:ixj - uxixj/ld,D+ 0. The set of functions W2(D)introduced resembles the well-known Sobolev space W;(D) (see [46,71,72]). If the boundary of the region D is sufficiently regular, for example, it is once continuously differentiable; Sobolev's theorem on imbedding (see [46,47]) shows that, in fact, W2(D)= W:(D). In this case u E W2(D)if and only if u is continuous in D,has generalized derivatives up to and including the second order, and, furthermore, these derivatives are summable in D to the power d. It is seen that if the function u is once continuously differentiable in D, its ordinary first derivatives coincide with its first generalized derivatives (almost everywhere). It turns out (a corollory of Fubini's theorem) that, for example, a generalized derivative u,~exists in the region D if for almost


all (xi, . . . ,x$) the function u(xl,x$, . . . ,x$) is absolutely continuous in x1 on (xl :(xl,xi, . . . ,x$) E D} and its usual derivative with respect to x1 is locally summable in D. The converse is also true. However, we ought to replace then the function u by a function equivalent with respect to Lebesgue measure. It is well known that if for almost all ( x r l , . . . ,x$) the function u(xl, . . . ,xi,x'b+l, . . . ,x$) has a generalized derivative on ((xl, . . . ,xi):(xl, . . . , xi,xcl, . . . ,x$) E D) and, in addition, this derivative is locally summable in D, u will have a generalized derivative in D. Using the notion of weak convergence, we can easily prove that if the functions cp, vn (n = 0,1,2, . . .) are uniformly bounded in D, v" -+ v0 (D-as.),. for some l,, . . . , 1, for n 2 1 the generalized derivatives v ~ ~ . . ., elk, , exist, and v . . .,,, 1scp (D-as.), the generalized derivative vg ,,.. also exists, V(1,).. 5 cp (D-a.s.1, and

.,!,)I

%).

. . (lk)

0 -+ V(l1)

. . . (lk)

weakly in LY2 in any bounded subset of the region D. In many cases, one needs to "mollify" functions to be smooth. We shall do this in a standard manner. Let c(x), cl(t), c(t,x) = il(t)i(x)be nonnegative, infinitely differentiable functions of the arguments x E Ed, t E El, equal to zero for 1x1 > 1, It1 > 1 and such that

For E # 0 and the functions u(x), u(t,x) locally summable in Ed, El x Ed, let

(3 (3

u(')(x)= E - ~ C u(O,.")(t,x) =E-~C -

* U(X) (convolution with respect to x), * u(t,x) (convolution with respect to x),

(: :) *

ds)(t,x)= ~-("'l)l -,-

u(t,x) (convolution with respect to (t,x)).

The functions u("(x), ~(~~")(t,x),u(~)(t,x) are said to be mean functions of the functions u(x), u(t,x). It is a well-known fact (see [10,71]) that u'" -+ u as E -+ 0: a. at each Lebesgue point of the function u, therefore almost everywhere; b. at each continuity point of the function u; uniformly in each bounded region, if u is continuous; c. in the norm Yp(D) if u E Yp(D) and in computing the convolution of u(') the function u is assumed to be equal to zero outside D. Furthermore, u(" is infinitely differentiable. If a generalized derivative u(,) exists in Ed, then [ u ( ~ , ] ( ~= ) [u(')](~). Finally, for p 2 1 IIu(~)IIP,E~5 I l ~ l l p , ~ ~ )

IIu(~)IIB(E~)

5 IIuIIB(E~).

1. Notation and Definitions

Considering the functions u'", we prove that the generalized derivative uxl of the function u(x) continuous in D does not exceed a constant N 1 almost everywhere if and only if the function u(x)satisfies in D the Lipschitz condition with respect to x1 having this constant, that is, if for any points xl,x2 E D such that an interval with the end points x,, x2 lies in D and xi = x i (i = 2, . . . ,d),the inequality (u(xl)- u(x2)(INllxl - x21 can be satisfied. It turns out that if a bounded function a has a bounded generalized derivative, o2 has as well a generalized derivative, and one can use usual formulas to find this generalized derivative. In addition to the space W2(D)we need spaces W2(D),W1s2(Q),and W1,2(Q),which are introduced for bounded regions D, Q in a way similar to the way W2(D)was, starting from sets of functions C2(D),C1,'(Q),and C1,2(Q),respectively, and using the norms

For proving existence of generalized derivatives of a payoff function another notion proves to be useful.

2. Definition. Let a function u(x)be given, and let it be locally summable in a region D. Let v(T)be a function of a set r which is definite, a-additive, and finite on the a-algebra of Bore1 subsets of each bounded region D' c D' c D. We say that the set function v on D is a generalized derivative of the function u in the l,, . . . , I, directions, and we write v(dx) = U(,,) . . . (,,)(x)(dx),

if for each function rp E C;(D),

The generalized derivative (d/dt)u(t,x)(dtdx) for the function u(t,x) locally summable in the region Q can be found in a similar way. The definitions given above immediately imply the following properties. It is easily seen that there exists only one function v(dx) satisfying (4)for all rp E C$(D).If the function u(,,,. . . (,,,(x)exists, which is a generalized derivative of u in the l,, . . . , 1, directions in the sense of Definition 1, assuming that v(dx) = u(,,,.. .(,,)(x)dx, we obtain in an obvious manner a set function


v, being the generalized derivative of u in the I,, . . . , I, directions in the sense of Definition 2. Conversely, if the set function v in Definition 2 is absolutely continuous with respect to Lebesgue measure, its Radon-Nikodym derivative will satisfy Definition 1 in conjunction with (4). Therefore, this Radon-Nikodym derivative is the generalized derivative u(,,,. . .(,,,(x). This fact justifies the notation of (3). In the case where the direction li coincides with the direction of the rith coordinate vector, we shall write

Using the uniqueness property of a generalized derivative, we easily prove that if the derivatives u(,,,. . . (,,,(x)(dx)for some k exist for all 11, . . . , I,, then

for I1,I . . . 11,1 # 0. Further, if the derivatives u(,,(,,(x)(dx)exist for all 1, all the derivatives u(,,,(,,,(x)(dx)exist as well. In this case, if I1,I . I1,I # 0, then

- (11 - 12)2u(11 - 1*)(11 - l2)(x)(dx)I. In fact, using Definition 2 we easily prove that the right side of this formula satisfies Definition 2 for k = 2. Theorem V of [67, Chapter 1, $11 constitutes the main tool enabling us to prove the existence of u(,,,. ..(,,,(x)(dx).In accord with this theorem from [67], the nonnegative generalized function is a measure. Regarding

as a generalized function, we have the following.

3. Lemma. Let u(x), v(T)be the same as those in thejrst two propositions of Definition 2. For each nonnegative cp E C;(D) let the expression (5) be nonnegative. Then there exists a generalized derivative u(~,, ,,,( in the sense of Definition 2. In this case, inside D ,,,

(- l)"(l,,. . . ,,,,(x)(dx) 2 (- l)kv(dx),

that is, for all bounded Bore1 T c fi c D

To conclude the discussion in this section we summarize more or less conventional agreements and notation. (w,,F,) is a Wiener process (see Appendix 1).

2. Estimates of the Distribution of a Stochastic Integral in a Bounded Region

Fzis the o-algebra consisting of all those sets A for which the set A n

{z I t) E Ftfor all t.

1UZ(t) denotes the set of all Markov (with respect to {Ft)) times z not exceeding t (see Appendix 1). C([O,T],E,) denotes a Banach space of continuous functions on [O,T] with range in Ed, Jlr, the smallest o-algebra of the subsets of C([O,T],E,) which contains all sets of the form where s 2 t, r denotes a Borel subset of Ed. 1.i.m. reads the mean square limit. ess sup reads the essential upper bound (with respect to the measure which is implied). inf l21 = a , f (xz) = f (xz)xz< a. When we speak about measurable functions (sets), we mean, as a rule, Borel functions (sets). The words "nonnegative," "nonpositive," "it does not increase," "it does not decrease," mean the same as the words "positive," "negative," "it decreases," "it increases," respectively. Finally, A=

az 1a(~')~

i= 1

denotes the Laplace operator. The operators La, F[u], F,[u], used in Chapters 4-6 are defined in the introductory section in Chapter 4.

2. Estimates of the Distribution of a Stochastic Integral in a Bounded Region Let A be a set of pairs (o,b),where a is a matrix of dimension d x d, and b is a d-dimensional vector. We assume that a random process (o,,b,) E A for all (w,t), and that the process

is defined. We shall see further that in stochastic control, estimates of the form

play an essential role, in (I) f is an arbitrary Borel function, z, is the first exit time of x, from the region D, and Q = ( 0 , ~x) D. A crucial fact here is that the constant N does not depend on a specified process (ot,bt),but is given


instead by the set A. In this section, our objective is to deduce a few versions of the estimate (1). We assume that D is a bounded region in Ed, x, is a fixed point of D, an integer dl 2 d, (w,,F,) is a dl-dimensional Wiener process, o,(o) is a matrix of dimension d x dl, b,(w) is a d-dimensional vector, and c,(w), r,(o) are nonnegative numbers. Assume in addition that o,, b,, c,, r, are progressively measurable with respect to (9,) and that they are bounded functions of (t,o). Let a, = 30,o:. Next, let p be a fixed number, p 2 d, and let y,,,

=

1

rudu,

q,,,=

1

cudu,

$, = c: -[(dil)l(pil)!(rtdet 4)11(pfI).

One should keep in mind that for p = d the expression cp-d)l(p+l)is equal to unity even if c, = 0; therefore $, = (r, det a,)ll(dil) for p = d. 1. Definition. A nonnegative function F(c,a) defined on the set of all nonnegative numbers c as well as nonnegative definite symmetric matrices a of dimension d x d is said to be regular if for each E > 0 there is a constant k ( ~such ) that for all c, a and unit vectors 1

2. Theorem. Assume that Ib,l I F(c,,a,) for all (t,w) for some regular function F(c,a). There exist constants Nl,N, depending only on d, thefunction F(c,a) and the diameter of the region D, and such that for all s 2 0, Bore1 f (t,x) and g(x), on a set { z D 2 s), almost surely

Before proving our theorem, we discuss the assertions of the theorem and give examples of regular functions. Note that the left sides of the inequalities (2) and (3) make sense because of the measurability requirements. It is seen that the function F(c,a) = c is regular. Next, in conjunction with Young's inequality,

if x , 2~ 0, p-l

+ q-l

=

1. Hence for a E ((),I),

E E (0,l)

Therefore, ca(tra)'-' is a regular function for a E (0,l).


We show that the function (det a)lid not depending on c is regular. Let p1 5 p2 I ...I pd be eigenvalues of a matrix a. We know that p, I (a2,L) if 121 = 1. Further, det a = p1p2 . pd, tra = p1 + p2 + . . . + pd. From this, in conjunction with the Young's inequality, we have +

Using the regular functions given above, we can construct many other regular functions, noting that a linear combination with positive coefficients of regular functions is a regular function. The function tr a is the limit of regular functions ca(tr a)'-' as LX 10. However, for d 2 2 the function tr a is not regular. To prove this, we suggest the reader should consider

3. Exercise For p

= d,

c, = 0, s = 0, g G 1 it follows from (3) that M

(det a,)llddt i N 2 (rneas D)'/~.

From the statement of Theorem 2 we take D = S,, F(c,a)= K tr a, with K > R-'. It is required to prove that for d 2 2 there exists no constant N 2 depending only on d, K, R, for which (4) can be satisfied.

This exercise illustrates the fact that the requirement Ib,l I F(c,,a,), where F is a regular function, is essential. In contrast to this requirement, we can weaken considerably the assumption about boundedness of o, b, c, r. For example, considering instead of the process x,, y,,, the processes

-

where z, is the time of first departure of x, from D, and noting that x, = X,, y,, = Y,,, for t < z,, we immediately establish the assertion of Theorem 2 in the case where a,,,,o,, ~,,,,b,, x,, c,, X, ,,,r, are bounded functions of (t,o). We think that the case where s = 0, r, 1, p = d is the most important particular case of Theorem 2. It is easily seen, in fact, that the proof of our theorem follows generally from the particular case indicated. The formal proof is rather difficult, however. It should be noted that according to our approach to the proof of the theorem, assuming s # 0, r, 1 makes the proving of estimates for s = 0, r, 1 essentially easier. In the future, it will be convenient to use the following weakened version of the assertions of Theorem 2.

-

+


4. Theorem. Let z be a Markov time (with respect to {SF,}), not exceeding 7,. Also, let there exist constants K , 6 > 0 such that for all t < z(o),;iE Ed d

Ibt(o)l I K,

C

ay(o)lliilj2 d1;i12.

i, j= 1

Then there exists a constant N depending only on d, K, 6, and the diameter of the region D such that for all s 2 0 and Bore1 f(t,x) and g(x) on the set {s I z ) , almost surely

This theorem follows immediately from Theorem 2 for r, p = d. In fact, we have x,,, = x ,

-

1, c, = 0,

+ Ji X . . P ~ ~ W ~ + Ji xU 0, P > ch(aR). W e note that u(x) is twice continuously differentiableand u(x)2 0 for x E D. Applying Ito's formula t o e-'+'s,tu(x,), we have for t 2 s on the set {z, 2 s) that


Assume that for all x E D, r 2 0 Then

which proves the assertion of the lemma as t -+ oo,with the aid of Fatou's lemma. Therefore, it remains only to choose constants a, j3 such that (5) is satisfied, assuming obviously that x # 0. For simplicity of notation, we shall not write the subscript r in c,, a,, a,, b,. In addition, let ;l = x/lxl, p = 1x1. A simple computation shows that

= (1

+ a sh up)-l{c(P - ch ap) + a sh ap(b,L)

ashap 1 - [tr a - (aA,A)] - F(c,a). +l+ashapp We note that ch ap r 1, chap 2 sh ap, a sh ap 2 aZp and for x E D the number p l R. Hence

Therefore, it follows from (6) that I r c

p - chaR 1 + ashaR

a2 a2 + (an,,?)+[tr a - (aA,L)] - F(c,a). 1 + a 1+a2R

We recall that F(c,a) is a regular function. Also, we fix some E < 1/R and ) E. Next, choose a large enough that a2/(1 + a2R) > E, a2/(1 + a) 2 k ( ~+ we take a number p so large that

Then 1 2 k(~)[c+ (al,il)] + E tr a - F(c,a)2 0,thus proving the lemma.


6. Corollary. Let G(c,a) be a regular function. There exists a constant N depending only on F(c,a), G(c,a) and the diameter of the region D such that

+

In fact,let Fl(c,a)= F(c,a) G(c,a).Then Ib,l I Fl(c,,a,),G(c,,a,)I Fl(c,,a,), and the assertion of our lemma is proved for Fl(c,a). 7. Lemma.LetR > O , h ( t , x ) 2 O , h ~ $ P ~ + ~ ( C ~ ) , h ( t , x ) =IO O,h(t,x)=O fort for 1x1 2 R. Then on (- oo,co) x Ed there exists a bounded function z(t,x) 2 0 equal to zero for t < 0 and such that for all sujiciently small if > 0 and nonnegative dejinite symmetric matrices a = (aij)on a cylinder C R .

where N(d) > 0. Furthermore, if the vector b and the number c are such that lbl I(R/2)c,then on the same set biz:) 2 cz("),i f if is sujiciently small. Finally, for all t 2 0, x E Ed

This lemma is proved in [42] by geometric arguments.

8. Lemma. Let lbtl < F(ct,at)for all (t,w)for a regular function F(c,a). There exists a constant N depending only on d, F(c,a), and the diameter of D, and such that for all s 2 0, f(t,x) on a set {z, 2 s}, almost surely

In other words, the inequality (2)holds for p = d.

PROOF. Let us use the notation introduced above:

s.,, = [c. du,

$t

= (r,det o,)ll(dt

l),

We denote by R the diameter of D and we consider without loss of generality that x , = 0.In this case D c S,. Also, we assume that z, is the first exit time of x, from S,. It is seen that z, 2 .z, Suppose that we have proved the inequality

((7,

2 s)-as.) for arbitrary s, f , where N = N(d,F,R). Furthermore, taking


in (7) the function f equal to zero for x !+ D, we obtain

< Nllfl l d + l , C n

=

Nllfl l d + l , Q

2 s)-as.) and, a fortiriori, ({zD2 s)-as.). It suffices therefore to prove (7). Usual reasoning (using, for example, the results given in [54, Chapter 1, $21) shows that it suffices to prove (7) only for bounded continuous nonnegative f(t,x). Noting in addition that by Fatou's lemma, for such a function ((2,

we conclude that it is enough to consider the case where r,(w) > 0 for all (t,~). We fix T > 0 and assume that h(y,x) = f ( T - y, x) for 0 < y < T, x E S,, and h = 0 in all the remaining cases. Using Lemma 7, we find an appropriate function z. Let z = z , , be the first exit time of a process (ys,,,xi)considered for t 2 s from a set [O,T) x S,. We apply Ito's formula to the expression e-vs~tz")(T- y,,,x,) for E > 0, t 2 s. Then

Using the properties of z(') for small E > 0, we find

Furthermore, z(') )10. Hence


in which we carry the term containing z(" from the right side to the left I Nllf lid+ l,cR: side. Also, we use the estimate Iz(")IISUP^,^ < Nllhlld+

IzI

~ ( d , ~ ) l l f l l ~ + ~ ,+~ M . ({1l n 7 e-n;ulbuldu19s

where y,,, E (0,T) for u E (s,z) by virtue of the condition r, > 0, and in addition, xu E S R ; hence the function h is continuous at a point ( T - yS,,,xu) and h(T - yS,,,x3 = f(y,,,x,). Letting E to zero in the last inequality, we obtain, using Fatou's lemma,

Further, on the set {zR2 s) it is seen that z I 7,. Therefore, by Lemma 5,

Finally, on the set {z, 2 s) for all T > 0,t > s, we obtain

It remains only to let first t + co,second T + co,and then to use Fatou's lemma as well as the fact that obviously z,,, + z, as T + co on the set {z, 2 s). We have thus proved the lemma. 9. Proof of Theorem 2. We note first that it suffices to prove Theorem 2 only for p = d. In fact, for p > d in accord with Holder's inequality, for example,

< -

(M{pe-".t(det

')I

a t ) l l d l g ( x t ) l p ~ d d t I ~ } e-ps.tctdtI% ~(M{~

-


In this case, "I! e-'Ps.tc,dt= 1 - e-Qsr'DI 1, and if we have proved the theorem for p = d, the first factor does not exceed [ N ( ~ F , DIlgP1dlJd,DldlP ) = NdlP(dJ,D)llgllp,D I (N(d,F,D)+ l)llgJJp,DThe inequality (2) was proved for p = d in Lemma 8. Therefore, it suffices to prove that

((zDI s)-a.s.) for all g. We can consider without loss of generality that g is a nonnegative bounded function. In this case, since (det a)lldis a regular function v = sup ess sup x, s,o

0

,,M

I

(det ~ , ) ~ l ~ g ( x , ) edt- ' P9 ~,, ~

is finite by Corollary 6. If v = 0, we have nothing to prove. Hence we assume that v > 0. Using Fubini's theorem or integrating by parts, we obtain for any numbers t , < t , and nonnegative functions h(t), r(t) that

+ l: exp{- J: r(u)d u I r ( t ) ( r h(u)du) dt. From this for s 2 0, A E FS, rt = (l/v)(g(x,)(deta,)lld,h, = (det ~ , ) ~ l ~ g ( x , ) , we find My,

ID,,

p he-",'

+, ,M ,,

dt

p

where the last term is equal to

exp{-l

r.

(r 1

hue-'Ps~u du dt,


Therefore,

% htf?-qs'tdf

M ~ ~ , r D a s

where f(t,x) = e-zgdl(d+')(x). Consequently, by Lemma 8,

where the constants N (which differ from one another) depend only on d, the function F(c,a), and the diameter of D. The last inequality is equivalent to the fact that {T, 2 s)-as.)

From this, taking the upper bounds, we find and v I ~ l l g l l ~thus , ~ ,completing the proof of Theorem 2. 10. Remark. Let 6 > 0. The function F(c,a) is said to be ®ular if for ) that for all c, a, and unit vectors A some E E (0,d) there is a constant k ( ~such In the sense of the above definition, the function which is ®ular for all 6 > 0, is a regular function. Repeating almost word for word the proofs of Lemmas 5 and 8 and the proof of Theorem 2, we convince ourselves that if the region D belongs to a circle of radius R, lbtl I F(ct,at)for all (t,o) and if F(c,a) is an R-'-regular function, there exist constants N l , N , depending only on d, F(c,a), and R such that the inequalities (2) and (3) are satisfied. 11. Exercise Let d 2 2, D = S,, E > 0.Give an example illustrating the (R-' + &)-regularfunction F(c,a) for which the assertions of Theorem 2 do not hold. (Hint: See Exercise 3.)

12. Exercise Let z(" be the function from Lemma 7. Prove that for sufficiently small E the function z("(t,x)decreases in t and is convex downward with respect to x on the cylinder C,.

3. Estimates of the Distribution of a Stochastic Integral in the Whole Space


I

In this section1 we shall estimate expressions of the form M f," f(t,x,)l dt using the 9,-norm off, that is, we extend the estimates from Section 2.2 to the case D = Ed. We use in this section the assumptions and notation introduced at the beginning of Section 2.2. Furthermore, let

Throughout this section we shall have two numbers K,,K, > 0 fixed and assume permanently that

-

for all (t,o).Note immediately that under this condition Ibtl does not exceed the regular function F(ct,at) K,c,. First we prove a version of Theorem 2.2. 1. Lemma. Let R > 0, let z be a Markov time with respect to {Ft},and let z, = inf { t 2 z: lx, - x,l 2 R}.' Then there exists a constant N = N(d,K,,R) such that for any Bore1 f(t,x)

PROOF. First, let z be nonrandom finite. For t 2 0 we set F ; = Fr+,, WI = Wr+t - wr,

z' is the first exit time of the process xj from S,. It is then seen that

=

M ( J ~e-Vi+if(y;

Also, see Theorem 4.1.8. infd = m.

1

+ y,, xi + x,)l d t l ~ b

(as.).


Furthermore, (w;,F;) is a Wiener process. In addition, by Theorem 2.2

for any x E Ed, y 2 0. In order to prove our lemma for the constant z, it remains to replace y, x by the Yo-measurable variables y,, x, in the last inequality. To do as indicated, we let rcn(t)= (k 1)/2"for t E (k/2",(k 1)/2"], rcn(x)= rcn(xl,. . . ,xd)= (IC,(X), . . . ,rcn(xd)).Note that rc,(t) 1 t for all t E (- c o , ~ ) K,(x) , --+ x for all x E Ed. From the very start, we can consider without loss of generality that f is a continuous nonnegative function. We denote by T:,Td, the sets of values of the functions rcn(t),rcn(x)respectively. Using Fatou's lemma, we obtain for the function f mentioned,

+

M

5N

{J:

e-Qi+;f(y; + y,, x;

2 ((s,,)J f

+ x,) dt l

+

1

~ b

l/(p+ 1)

P+

l(t,x)dx dt)

Further, we prove the lemma in the general case. Taking A E 9, and setting zn = rc,(z),

z; = inf{t 2 zn:lx, - x,.l 2 R), we can easily see that

zn 17 , x.<m

lim

n-t m

l'

lim z;

n+ m

e-Qt+tlf(~t,xt)l dt z

and that for s E r,l the set

2 z,,

e-*.$d f(yt,xJ1 dt,


Therefore, in accord with what has been proved,

thus completing the proof of our lemma. 2. Lemma. As in the preceding lemma, we introduce R, z, Z* Also, we denote by p = p(I) the positive root of the equation I - p K l - p2K2 = 0 for I > 0. Then 1 ~ { ~ ~ ~ < ~ e - I~ a;l;ji - l i ge -r *} - - ~ ~ < ~

PROOF.Let n(x) = ch plxl. Simple computations show that

Taking advantage of the fact that sh plxl 5 ch ~1x1,shplxl 5 plxl ch ~1x1,we obviously obtain Ic,n(x) - ~ * t , ~ ~ 2 n (c,xch ) p ) x l ( I - pKl - p2K2) = 0.

Further, using Ito's formula applied to e-A"tn(xt+ x), we have from the last inequality that


Using the continuity property of n(x),we replace x with a variable ( - x i , J in the last inequality. Then which yields for A E F1

We have proved the lemma; further, we shall prove the main theorem of Section 2.3. 3. Theorem. There exist constants N i = Ni(d,Kl,Kz)(i = 1,2) such that for all Markov times z and Bore1 functions f(t,x), g(x)

PROOF.We regard f , g as nonnegative bounded functions and in addition, we introduce the Markov times recursively as follows: Z0 zn+l

= Z,

= infit

2 zn:lxi- xrnl 2 1).

Note that by Lemma 2, M { ~ p + l < m-q*+ e l IFI)= M {M { X m+

R. In accord with Ito's formula applied to e-'t-Elxtl we obtain

+

+

where

Hence e - & R ~ e - r p q XI , se-"'"I. t Using Fatou's lemma, as t -+ co,we arrive at the former inequality. In order to prove the latter inequality, we note that under the assumption r, < K3ct we have on the set ( t , I y, + s} that t , - s I K3cp,, from which it follows that

Furthermore,


Having multiplied the extreme terms in the last two inequalities we establish the second assertion of the lemma, thus completing the proof of the lemma.

3. Theorem. There exists a finite function N(d,Kl) increasing with respect K , and such that for all f E 9p + (Ed+ ,)

,

PROOF.Suppose that we have proved the theorem under the condition that K 2 = K 3 = 1. In order to prove the theorem under the same assumption in the general case, we use arguments which replace implicitly the application of the self-scaling property of a Wiener process (see Exercise 3.5). If a E % = %(K1,K2,K3),let 1 1 rRa,Fa,Pa,&,wf,Ft,-a;, -b;,c;,

& 4G

It is seen that a' E % = %(K1/&,l,l). It is also seen that a' runs through the entire set % ( K 1 / G , 1 , 1 )when a runs through the entire set %(Kl,K2,K3). Further, for f E 9,. ,(Ed+,) let f '(t,x)= f (K3t,&x). We have

= sup as%

Ma

Sowe-"@ f ( y ~ " , x ; ,dt~ )

- K:/(P+' ) K ~ ( Pl )+sup

M"

a' E 'U'

Some-@'+trf (s + K , y;',', x + G x ; ' , O )dt

Therefore, if we have proved our theorem for K 2 = K 3 = 1, then 11v(K1,K2,K3,fr',.)Il:zi,Ed+l

S l ~ I ~ ( ~ , fll ,,s ,lx ), r l dxds

- K:KP12)d

4. Limit Behavior of Some Functions

Therefore,it suffices to prove this theorem only for K , in our proof in this case the expression

=K, =

1. We use

representable as the "sum" of terms each of which incorporates the change which occurs while the process (yr*",xy)moves across the region associated with the given term. We assume without loss of generality that f 2 0. Let R be such that the volume of S, is equal to unity. We denote by w(t,x) the indicator of a set C,,,. Let At,,xt,(t,x)= w ( t , - t,xl - x)f(t,x).It is seen that

Ia(s,x) =

S_" d t , f d x , Ma Som

e-qF$x t,,,,, ( y ; + , x g dt.

rn

In order to estimate the last expectation for fixed t,, x , , we note that

At ,,,, )(t,x) can be nonzero only for 0 I t , - t I 1, Ix, - xl r R. Hence, if

ya is the time of first entry of the process ( t , - yF%, - xFx) into the set C,,,, then

Furthermore, on the set {ya < co) 0 I t l - y;f I 1

R 2 Ix, - x;kXI = IXY-~~I.

and

The last inequality in the preceding lemma implies the inequality ya 2 z;*-"'. By Theorem 3.3 and the preceding lemma we obtain

{4

INi(lAtl,xl)llp+i,Ed+, exp - R

E

1

I

- -1x 2 - xll - -2( t i - s - 1) ,

where N , = N , ( d , K l , l ) is the constant given in Theorem 3.3. Also, we note that for t , < s the first expression in the above computations is equal to zero since t , - yFS I t l - s < 0 and ya = co. Hence where .n(t,x) = exp[(&/2)R- (~/2)1x1+ +(t + I ) ] for t I 0, n(t,x) = 0 for t > 0. Therefore, since v = sup, I",


In the right side of the last expression there is a convolution (with respect and z(tl,xl). It is a wellto (tl,xl)) of the two functions known fact that the norm of the convolution in 9, does not exceed the and the norm of the other function product of the norm of one function in 9p in 9 , . Using this fact, we conclude that

I l & t l , x l , I P +l , E d+l

To complete the proof of the theorem it remains only to show that the last constant N(d,Kl) can be regarded as an increasing function of K , . Let where the upper bound is taken over all f E zp+ l(Edtl)such that > 0. According to what has been proved above, N(d,Kl) < m. Il f l l p + i , ~ d + I In addition, the sets 'U increase with respect to K,. Hence v, R(d,K,) increase with respect to K,. Finally, it is seen that Iv(f,s,x)l I v(l f l,s,x) and IIv(K1,l,l,f,',')llp+l,Ed+l

I R(d,Kl)llf llp+l.Ea+l.

The theorem has been proved. We extend the assertions of Theorem 1 and 3 to the case where the function f(t,x) does not depend on t. However, we do not consider here the process r,, as we did in the previous sections. Let

=

sup

ae%(K1,K2,0)

Ma fow e-@(~f)(P-~)/P(det an1/Pg(x:,") dt.

4. Theorem. (a) Let g E Zp(Ed);then v(x) is a continuous function,

Iv(~)I N(d3K1,K2)11~IIp,Ed' (b) There exists ajinite function N(d,K1) increasing with respect to K , and such that for all g E YP(Ed)

This theorem can be proved in the same way as Theorems 1 and 3.


We proceed now to consider the main results of the present section. Let numbers K > 0, 6 > 0 be fixed, and let each point (t,x) E Ed+ (XE Ed) be associated with some nonempty set 23(t,x) (respectively, 23(x)) consisting of sets a of type (1). Let 23 be the union of all sets 23(t,x),23(x).We assume that a function ct(w) is bounded on 23 x [0,co) x UaSZa and that for all a E 23, U E [ o , ~ )O , E aa, y E Ed tr ot[at]* 5 K, r: = 1, ICCJtl*yl 2 61~1. It is useful to note that (2) can be rewritten as d

1 (a:)"yiyj

(a:y, y) =

1

2 2621y12,

i,j=l

1

since (a:y, y) = +(o:(o;)*y,y) = 4 ($)*yI2.

,,

5. Theorem. (a) Let 12 1, > 0, let Q c Ed+ let Q be an open set, and let f E z p + I(Q), Za = p , s , x - inf(t 2 O:(t + s,xFsX) 4 Q), =

Then

sup M'

a s B(s,x)

:J

e-qT-Uf(t

+ s, x

~dt.~ )

~lIz~Ip+5 i , aN(d,K,J,Ao)llf l l p + l , ~ . (b) Let A 2 1, > 0,let D c Ed, let D be an open set, and let g E Yp(D), za = T~~~ = inf (t 2 0 : ~ : '4~D),

:f

zyx) = sup Ma a e B(x)

Then

e-q;-ug(x:,x) dt.

PROOF.Since all eigenvalues of the matrix a: are greater than id2, det a: 2 2a62d..From this, assuming that f"=I f lxQ, i?: = :C + A, @: = q ~ :+ At and noting that i?: 2 1, we find Iz"s,x)( IN(6)A" -p)l(p

+l)

SUP a e B(s,x)

M' Jo

)

e - q-a( t F a (p-d)l(p+l)

x (r: det a:)ll(~+ly(y:,"x:,s) dt. It is seen that K tr aa < - i?:, '- 1 Therefore,

and

1

< - z.; '-1

f


which implies, by Theorem 3, that

thus proving assertion (a) of our theorem since

Proceeding in the same way, we can prove assertion (b) with the aid of Theorem 4. The theorem is proved. 6. Theorem. (a) Suppose that Q is a region in Ed+,, fl(t,x) is a bounded Borel function, f E 9,+ l(Q),1 > 0, Za = Za.s.x - infit 2 O:(s+ t, xpx) $ Q},

1

zA(s,x)= zA(f ,s,x)

= a ~sup ' t ) ( s , x )M a [ ~ ~ e - Q ~ - A y ( s + t , x ~ 3 d t + e - ~ ~ ~ - ". ~ l ( s + ~ , ~

Then, there exists a sequence 1, + co such that l,zL(s,x) + f (s,x) (Q-as.). (b) Suppose that D is a region in Ed, g,(x) is a bounded Borel function, g E T p ( D ) 2, > 07 Z~ = z

~=,inf{t ~ 2 O:xFx$ D},

Then, there exists a sequence 1, + co such that l,zAn(x)+ g(x)(D-as.). 7. Theorem. (a) W e introduce another element in Theorem 6a. Suppose that Q' is a bounded region Q' c Q' c Q. Then I(1.z" f f l p +l , Q , + O as 1 + co. If fl = 0,we can take Q' = Q. (b) Suppose that in Theorem 6b D' is a bounded region, D' c B' c D; then 1112~" gllp,D'-' 0 as 1 -t co. If g1 = 0, we can take D' = D. PROOFOF THEOREMS 6 AND 7. It was noted in Section 2.1 that the property of convergence with respect to an exterior norm implies the existence of a subsequence convergent almost everywhere. Using this fact, we can easily see that only Theorem 7 is to be proved. 7a. First, let fl = 0. We take a sequence of functions PROOFOF THEOREM f" E CF(Q)such that f" - f llp+l,Q+ 0. It is seen that

11

11z"f7s,x) - f(s7x)l I11z"f,s7x) - z"(f",s,x)l + Il.zA(f",s,x)- f"(s,x)l

+ If " ( 3 , ~-) f(s,x)l,


from which, noting that Iz"(f;s,x) - z" f ",s,x)l Iz"(Jf - f"l,s,x), we obtain, in accord with Theorem Sa,

In the last inequality the left side does not depend on n; the first and third terms in the right side can be made arbitrarily small by choosing an appropriate n. In order to make sure that the left side of the last inequality is equal to zero, we need only to show that for each n

-

In short, it suffices to prove assertion (a) for fl 0 in the case f E C:(Q). In conjunction with Ito's formula applied to f(s + t, X>~)I-~F-" for each a E B(s,x), t 2 0 we have

where

Since a;, b,: c; are bounded, IL; f(t,x)l does not exceed the expression

Denoting the last expression by h(t,x), we note that h(t,x) is a bounded finite function; in particular, h E Z p +,(Q). Using the Lebesgue bounded convergence theorem, we pass to the limit in (3) as t -+ co.Thus we have f(s,x)

= AMa

sz

e-ql-uf(s

which immediately yields

+ t,xex)dt - Ma :S

+

e - q ~ - * f L ~ ( st, x:,') dt,


In short, we have which, according to Theorem 5a yields

1

I Nllhllp+l,Qlim - = 0, A+m

1

thus proving Theorem 7a for fl E 0. In the general case IAz" f ,s,x) - f (s,x)l I il sup ~~e-@'~-"'"lfl(s as5(s,x)

1

+ il

a

sup M ,

Jr

+ za,x:,.) 1

e-'pF-"(s

+ t, xpx)dt - f(s,x) ,

where the exterior norm ofthe second term tends to zero; due to the boundedness of fl the first term does not exceed the product of a constant and the expression

Therefore, in order to complete proving Theorem 7a, it remains only to show that ]ln"lp+l,Qr -+ 0 as il -+ co for any bounded region Q' lying together with the closure in Q. To this end, it suffices in turn to prove that nA(s,x)-+ 0 uniformly on Q'. In addition, each region Q' can be covered with a finite number of cylinders of the type C,,,(s,y) = {(t,x): y - xl < R, It - sl < r}, so that C,,,,,(s,y) c Q. It is seen that we need only to prove that n"t,x) 0 uniformly on any cylinder of this type. We fix a cylinder C,,,(s,y) such that C,,,,,(s,y) c Q. Let zs(x) = infit 2 0: - xpXl 2 R}. Finally, we denote by p ( l ) the positive root of the equation il - pK - p2K = 0. Also, we note that for (t,x) E C,,,(s,y) we have za"*" 2 r A zf;c(x).Hence

1

-+

IX

Furthermore, by Lemma 3.23 the inequality holds true. Therefore, the function n"t,x) does not exceed Ae-" + I(ch p(il)R)-I on C,,,(s,y). Simple computations show that the last constant tends to zero as A -+ co.Therefore, n"t,x) tends uniformly to zero on C,,(s,y). This completes the proof of Theorem 7a. Theorem 7b can be proved in a similar way, which we suggest the reader should do as an exercise. We have thus proved Theorems 6 and 7. In Lemma 3.2, one should take .c = 0,c, = 1.

5. Solutions of Stochastic Integral Equations and Estimates of the Moments

8. Corollary. Let f

E

2,+l(Q), f 2 0 (Q-as.) and for all (s,x) E Q let

inf

Ma

a t B(s,x)

Sr e-qy(s + t, x;sx) dt

= 0.

Then f = 0 (Q-as.). In fact, by Theorem 2.4 the equality (4) still holds if we change f on the set of measure zero. It is then seen that for 2 2 0 inf

a E 5(s,x)

Ma

Sr e-qr-Ay(s + t, x:sX) dt

= 0.

Furthermore, for f1= 0 A z(-f,s,x)=

-

inf

Ma s r e - q ~ - "f (s

a E 5(s.x)

Therefore, za = 0 in Q and -f

= lim,,,

2,zk

=0

+ t, x?") dt.

(Q-as.).

5. Solutions of Stochastic Integral Equations and Estimates of the Moments In this section, we list some generalizations of the kind we need of wellknown results on existence and uniqueness of solutions of stochastic equations. Also, we present estimates of the moments of the above solutions. The moments of these solutions are estimated when the condition for the growth of coefficients to be linear is satisfied. The theorem on existence and uniqueness is proved for the case where the coefficients satisfy the Lipschitz condition (condition ( 2 ) ) . We fix two constants, T > 0, K > 0. Also, we adopt the following notation: (w,,F,) is a dl-dimensional Wiener process; x, y denote points of Ed; a,, a,(x), F,(x) are random matrices of dimension d x dl; b,(x), &x), t,, 5; are random d-dimensional vectors; r,, h, are nonnegative numbers. We assume all the processes to be given for t E [O,T], x E Edand progressively measurable with respect to {Fib If for all t E [O,T], w, x, y we say that the condition ( 9 ) is satisfied. If for all t E [O,T], w, x lla,(~)11~ I 2r:

+ 2K21x12,

we say that the condition (R) is satisfied. Note that we do not impose the condition ( 9 ) and the condition (R) on o",(x),&(x).Furthermore, it is useful to have in mind that if the condition ( 2 ) is satisfied, the condition (R) will be satisfied for r, = Ilot(0)ll, hi = Ibt(0)l


+

(with the same constant K) since, for example, IJat(x)I12I 2110,(0)11~ 211.t(x) - ~t(O)Il2. As usual, by a solution of the stochastic equation

process xt for we mean a progressively measuable (with respect to {Ft}), which the right side of (1) is defined4 and, in addition, xt(w) coincides with the right side of (1) for some set SZ' of measure one for all t E [O,T], w E SZ'. 1. Lemma. Let x, be a solution of Eq. (1) for

5, = 0. Then for q 2 1

We prove this lemma by applying Ito's formula to the twice continuously differentiable function and using the inequalities

I x ~ ~ ~

2. Lemma. Let the condition (R) be satisjed and let x, be a solution of Eq. (1) for tt = 0. Then, for all q 2 1, E > 0, t E [O,T]

+

-

where A = 4qK2 E A,, . If the condition (9) is satisjed, one can take in (2) hs = lbs(0)l,rs = llcs(O)ll.

PROOF. We fix q > 1, E > 0, to E [O,T]. Also, denote by $(t) the right side of (2). We prove (2) for t = to. We can obviously assume that $(to) < co.We make one more assumption which we will drop at the end of the proof. Assume that xt(w) is a bounded function of w, t. Using the preceding lemma and the condition (R), we obtain

Next, we integrate the last inequality over t. In addition, we take the expectation from the both sides of this inequality. In this case, the expectation of the stochastic integral disappears because, due to the boundedness of Recall that the stochastic integral in (1) is defined and continuous in t for t I T if


xt(o),finiteness of $(to), and, in addition, Holder's inequality, M

J:

~ x , l * ~ ~ o ( x t, ) sx ~NM ~

J:

llor(xt)llzdt

'

Furthermore, we use the following inequalities:

P 52

MIX,^^^ +

1 28

~ I ~ ~ 1 ~I q -MIX^^^^)' ~ r : -(l/q)(~r:q)l/q. Also, let

In accord with what has been said above for t 5 to, m(t) I

Ji [lqm(s) + w1-"~q'(s)] ds.

Further, we apply a well-known method of transforming such inequalities. Let 6 > 0. We introduce an operator F, on nonnegative functions of one variable, on [O,to], by defining

u2(t)for It is easily seen that F, is a monotone operator, i.e., if 0 I ul(t) I all t, then 0 5 F,ul(t) I F,uZ(t)for all t. Furthermore, if all the nonnegative functions u; are bounded and if they converge for each t, then limn,, F,un(t) = F, limn,, un(t). Finally, for the function v(t) = NeQt for all suffi1 we have F,u(t) I v(t) if t E [O,to]. In fact, ciently large N and 6 I

for It follows from (3) and the aforementioned properties, with N such that m(t) I v(t), that m(t) I F,m(t) I . . . I F:m(t) Iv(t). Therefore, the limit limn,, F:m(t) exists. If we denote this limit by v,(t), then m(t) I v,(t). Taking the limit in the equality F;+lm(t) = F,(F;m)(t), we conclude that v, = F,v,. Therefore, for each 6 E (0,l) the function m(t) does not exceed some non-


negative solution of the equation

We solve the last given equation, from which it follows that v,(t) 2 6, v,(O) = 6, and vb(t)= ilqv,(t) + gv; -'l/q'(t). (4) Equation (4),after we have multiplied it by v$114)- (which is possible due to the inequality v, 2 6) becomes a linear equation with respect to v;Iq. Having solved this equation, we find v,1/4(t)= 6'14 + $(t). Therefore, m(t) I (#Iq ll/(t))qfor all t E [O,to],6 E (0,l).We have proved the lemma for the bounded x,(w) as 6 -+ 0. In order to prove the lemma in the general case, we denote by z, the first exit time of xi from S,. Then xi ,,,(o) is the bounded function of (o,t),and, as is easily seen,

+

Therefore the process xi,,, satisfies the same equation as the process x, does; however, o,(x), b,(x) are to be replaced by x ,,,, o,(x), ~,,,,b,(x), respectively. In accord with what has been proved above, M 1 ~ , , , , 1 ~ ~ I [$(t)Iq. It remains only to allow R -+ co, to use Fatou's lemma, and in addition, to take advantage of the fact that due to the continuity of x, the time z, -, co as R -+ co. We have thus proved our lemma.

3. Corollary. Let fi llas112ds < co with probability 1, and let z be a Markov Then, ).for all q > 1 time with respect to (9,

In fact, we have obtained the second inequality using Holder's equality. The first inequality follows from the lemma, if we take o,(x) = a,X,,,, b,(x) = 0, write the assertion of the lemma with arbitrary K, E, and, finally, assume that K 10, E 10. %

4. Exercise In the proof of the lemma, show that the factor 2qin Corollary 3 can be replaced by unity.

5. Corollary. Let the condition (2) be satisfied, let xi be a solution of Eq. ( I ) , and let 2, be a solution of the equation 2, =

e + J; ~ , ( z , ) ~ w+, J; %(x;)ds.


Then, for all q 2 1, t E [O,T]

+ Ilos(as)- as(ns)112q) ds,

where p

= 4q2K2

PROOF.Let y,

+ q.

= ( x , - 2,) -

(5, - 5).Then, as is easily seen,

in this case From this, according to the lemma applied to the satisfy the condition (9). process y,, we have

+

+ 2(zq - 1 ) s ; e ( l ~ ~ ) * t [- ~M) ~ ~ o , ( xt, , -

5)- i?s(Ts)112q]1~q ds.

We raise both sides of the last inequality to the qthpower. We use Holder's inequality as well as the fact that Ibs(as

+ Ss - Es) - Ks(x"s)l IIbs(% + Ss - f s ) - bs('s)l + lbs(x"3- ~ s ( ~ s ) l I K21Ss - 5 1 + IbS(ZS)- Es(ns)l, (a + b)¶ I 2q '(aq + bq), -

which yields

+ 22q-11bs(~s) + 2q(2q - 1 ) q 2 ~ q - ~ ~ -~ q 1 ( ~ + P(2q - l)q22q- llas(Ts)- Zs(Ts)112q]ds. It remains to note that Ix, - T,l I lytl + ISt - &I , I x , - RtIzqI 22q-11~t12q + Q2q

Es(~s)124 l

2 2 q - 1 ) tt z l 2 q , thus proving Corollary 5.

6. Corollary. Let the condition (R)be satisfied, and let x, be a solution of (1). Then there exists a constant N = N(q,K) such that for all q 2 1, t E [O,T] M l ~ t' lq

< - N M I C , I ~ ~+ NtqP'M

S; [ICSIzq +'h

+ r:q]eN"-s)ds.

In fact, the process y, r x, - 5, satisfies the equation dy, = o(yt + t J d w , + b(y, + S,)dt,

Y O = 0,


the coefficients of this equation satisfying the condition (R), however with different h,, r,, K. For example, Ilo,(x

+ 5,)112 121: + 2K2[1x+ ttll212r: + 4K214;,12+ 4K21xI.

Therefore, using this lemma we can estimate Mlyt12q.Having done this, we need to use the fact that I 22q-11ytl + 2'4-I IttI* In our previous assertions we assumed that a solution of Eq. (1) existed , and we also wrote the inequalities which may sometimes take the form oo I oo.Further, it is convenient to prove one of the versions of the classical Ito theorem on the existence of a solution of a stochastic equation. Since the proofs of these theorems are well known, we shall dwell here only on the most essential points.

IX,/~~

7. Theorem. Let the condition (2' be )satisjed and let

fr

Then for t T Eq. (1) has a solution such that M Ixtl2dt < co. If x,, y, are two solutions of (I), then P { s ~ p , , ~ , , ~ ~ lytl x , > 0) = 0.

MIX,

PROOF.Due to Corollary 5, - y,12 = 0 for each t. Furthermore, the process x, - y, can be represented as the sum of stochastic integrals and ordinary integrals. Hence the process x, - y, is continuous almost surely. The equality x, = y, (a.s.) for each t implies that x, = y, for all t (a.s.), thus proving the last assertion of the theorem. For proving the first assertion of the theorem we apply, as is usually done in similar cases, the method of successive approximation. We define the operator I using the formula Ix, =

Ji os(xs)dw, + fi bs(xs)ds.

This operator is defined on those processes x, for which the right side of (5) makes sense, and, furthermore, this operator maps these processes into processes Ix, whose values can be found with the aid of the formula (5). Denote by V a space of progressively measurable processes x, with values in Ed such that

It can easily be shown that the operator I maps V into V. In addition, it can easily be deduced from the condition (2) that MIIx. - Iyt12I aM

Si

lxs - ys12ds,

where a = 2K2(1+ TK2). Let xjO)r 0, xj"") = 5, + Ixj") (n = 0,1,2, . . .). It follows from (6) that


Iterating the last inequality, we find

Since a series of the numbers (Ta)"12(n!)-n12 converges, it follows from (7) that a series of functions xj"") - xj") converges in V. In other words, the functions xj"") converge in V, and furthermore, there exists a process x", E V such that Ilxj") - %,I[ 0 as n -+ GO. Further, integrating (6),we obtain

-

IIIxt - IytIl 5 aTIlxt -

ill.

(8)

In particular, the operator I is continuous in V. Passing to the limit in the equality I(xj"") - (5, Ix = 0, we conclude 112, - (5, I%)[[= 0 , from which and also from (8) it follows that I%, = I(( + I%), for almost all t, w. However, the both sides of this eqbality are continuous with respect to t for almost all w. Hence they coincide for all t at once almost surely. Finally, taking xi = 5, + I%,, we have xi = 5, + I(< + I%)),= 5, + Ix, for all t almost surely. Therefore, x, is a solution of the primary equation, (I), thus completing the proof of the theorem.

+

+

8. Exercise Noting that as(x)= [os(x)- as(0)] + us(0),prove that the assertions of the theorem still hold if M j: 1q,12 dt < co,where

We continue estimating the moments of solutions of a stochastic equation.

9. Theorem. Suppose the condition (2) is satisjied, xi is a solution of Eq. ( I ) , and 2,is a solution of the equation

z,

Then, ij- the process 5, - is separable, the process x, - 2,is also separable, and for all q 2 1, t E [O,T] M sup s5t

Ix,

- %,IzqINeNiM sup

where N = N(q,K).

s5f

It, - ? , I z q

ti,

PROOF. It is seen that x, - x", is the sum of 5, - stochastic integrals and Lebesque integrals. Both types of integrals are continuous with respect to t. Hence, the separability property of 5, - 5 implies that xi - x", is separable, and in particular, the quantity sup,,, lxs - is measurable with respect to w.


As was done in proving Corollary 5, the assertion of the theorem in the general case can easily be deduced from that in the case where 5, = 5; = 2,= 0, FS(x)= 0, bs(x)= 0. It is required to prove in the latter case that

-

M sup lxs12qI Ntq-'eN'M S$t

Jd [b.(0)1~~+ 11~~~(0)11'~] ds.

(9)

Reasoning in the same way as in proving Lemma 2, we convince ourselves that it is possible to consider only the case with bounded functions x,(o) and to assume, in addition, that the right side of (9)is finite. First, we prove that the process

is a submartingale. We fix c > 0 and introduce an auxiliary function of the real variable r using the formula q(r) = d m . Note that cp(lx1) is a smooth function on Ed. In conjunction with Ito's formula,

[ I I g , ( X t ) I 1 2-

ICTt*(Xt~2]} Ixt I

dt

xtgt(x*) + eK2tqf(/xtl) dw,.

IxtI Let us integrate the last expression over t from s, up to s2 2 s,, and also, let us take the conditional expectation under the condition Fsl.In this case, the expectation of the stochastic integral disappears (see Proof of Lemma 2). In addition, we use the fact that since bt(xt)xt 2 - Ibt(xt)lIxtI 2 - K21xtI2- lb,(0)1Ixtl,

0 I cpl(r)I 1, lrl I q(r).

then

Furthermore, cp" 2 0,

r la:(xt)xt12.Therefore,

from which, letting E go to zero, we obtain, using the theorem on bounded 1 Fsl)2 )I,,. Therefore )I,is a submartingale. convergence, M {)I,, From well-known inequalities for submartingales (see Appendix 2) as well as Holder's inequality we have M sup

sst

I x I ~ M ~SUP ~ yiq~ < 4 ~ 1 7 ~ ' ~ S$t

< - 4 . 224- l e 2 4 K 2 t ~ I X , 1 2 4 + 4 . 22q- l e 2 q K 2 t tZ q - 1M

Jd /bs(0)12qds.


It remains only to use Lemma 2 or Corollary 6 for estimating MIX,^^^, and, furthermore, to note that tae" 2 N ( ~ , b ) e for ~ ~a' > 0, b > 0, t > 0. The theorem is proved. 10. Corollary. Let the condition ( R ) be satisfied, and let xt be a solution of Eq. (1). Then there exists a constant N(q,K) such that for all q > 1, t E [O,T] P NP-'eNtM

M sup lxS S l t

If

Si [1ts12q+ h:'I + 121d ~ .

5, is a separable process, then M sup lxslzq5 NM sup

sct

S l t

1tt12q + ~ t ' - ~ e ~Si' M

+ h 2 + I : ~ ]ds.

First, we note that the second inequality follows readily from the first expression. In order to prove the first inequality, we introduce the process y, = x, - 5,. It is seen that dy, = ot(yt + t,)dw, + bt(yt + C)

c+m

n

Further, for t, > t, xY2 - x:,

=

t 0, ItZ- tll I 1 (by Fatou's lemma), the relationship between the extreme terms of this inequality holds for n = 0 as well. Then, by Kolmogorov's theorem x": is a continuous process for all n 2 0. C:, being separable Wiener processes, are continuous as well.

%;,I4

MIX;,

In Corollary 5.3 we need take z = co,t = t,, a, = a&. 4 ),,.~

6. Existence of a Solution of a Stochastic Equation with Measurable Coefficients

Further, we fix some T > 0. The processes (x;;w,) are measurable with respect to pTfor t I T ; the increments w, after an instant of time T do not depend on F T . Therefore, the processes (x:;w,) (t I T ) do not depend on the increments w, after the instant of time T. Due to the coincidence of finite-dimensional distributions, the processes (x":;fi:) (t 5 T ) do not depend on the increments C: after the time T for n 2 1. This property obviously holds true for a limiting process as well, i.e., it holds for n = 0. This readily implies that for n 2 0 the processes fi: are Wiener processes with respect to o-algebras of Fj"), defined as the completion of o(x":,C::s I t ) . Furthermore, for n 2 0 and each s I t the variable x": is Fj")-measurable. Since 2: is continuous with respect to s, 2: is a progressively measurable process with respect to {Fj")). These arguments show that the stochastic integrals given below make sense; Let k-,(a) = 2-"[2"a], where [a] is the largest integer I a. Since o,(t,x";) for n 2 1 are bounded functions of (o,t),continuous with respect to t, and since k-,(t) t as m -+ a, then

-

lim M

m-t m

Jb l l c n ( t , a

-

- o n ( d t ) , + m . ) ) ~dt~ 2 0

for n 2 1 for each T 2 0. Hence for each t 2 0

So

6

Writing similar relations for a,(s,x:) dw,, Sfo b,(s,2:) ds, b,(s,x3 ds, and using the fact that the familiar finite-dimensional distributions coincide, we can easily prove that for all n 2 1, t 2 0

In other words,

for each t 2 0 almost surely. We have thus completed the first stage of proving Theorem 1. If we had so far the processes xi, the convergence property of which we knew nothing about, we have now the convergent processes 2:. However, in contrast to x;, the processes 57: satisfy an equation containing a Wiener process which changes as n changes. We take the limit in (2) as n -, co.For each no 2 1, we have

where o,,(s,x,) satisfies the Lipschitz condition with respect to (s,x).Hence llono(t2,x";2) - ~no(tl,x":l)ll N(lt2 - tll + - x":ll).


In addition, by virtue of (1) lim sup h10

n

sup

Itz-tll 8 ) = 0.

From this it follows, according to Lemma 3, that the first term in (3) tends in probability to So ono(s,jzsO)d@,O. Therefore, applying Chebyshev's inequality, we obtain

We estimate the last expression. It is seen that M

j,If (s72;)lds I etM J: e-'1 f(s,%;)1ds 5 etM Jr e-.l f(~,z;)lds.

Therefore, by Theorem 3 A 9

for n 2 1, where N does not depend on n. For n = 0 the last inequality as well holds, which fact we can can easily prove for continuous f taking the limit as n 4 co and using Fatou's lemma. Furthermore, we can prove it for all Bore1 f applying the results obtained in [54, Chapter 1, $21. Let w(t,x) be a continuous function equal to zero for t2 + 1x1' r 1 and such that w(0,O) = 1 , 0 I w(t,x) I 1. Then, for R > 0

%M

n-t w

Ji (la.

- anol/2(~,Z;) ds 5 NM

Ji [I

-w

(+,;)I

ds

+ n-r E rn M J ~R ~' R ( A ~ )

+ Nil lloo - ~nol1211d+~,~R,R. It should be noted that f(a,cr:i)

= ilo:A12

2 i621A12 since 61112 I(a,i,A) = (i,a:A) 5 1 1 1(cr:ll.

7. Some Properties of a Random Process Depending on a Parameter

Estimating M

Sb I o ,

- ~ ~ l l ~ ( r ,dr n :in)

similar fashion, we find

for each no > 0, R > 0. Finally, we note that the last expression tends to zero if we assume first that no -+ oo,and next, that R + co.Therefore,

in probability. We have a similar situation for the second integral in (2). Therefore it follows from (2) that

for each t 2 0 almost surely. It remains only to note that each side of the last equality is continuous with respect to t ; hence both sides coincide on a set of complete probability. We have thus proved Theorem 1.

7. Some Properties of a Random Process Depending on a Parameter In investigating the smoothness property of a payoff function in optimal control problems it is convenient to use theorems on differentiability in the mean of random variables over some parameter. It turns out frequently that the random variable in question, say J(p), depends on a parameter p in a complicated manner. For example, J ( p ) can be given as a functional of trajectories of some process xf which depends on p. In this section we prove the assertions about differentiability in the mean of such or other functionals of the process. Three constants T, K , m > 0 will be fixed throughout the entire section. 1. Definition. Let a real random process x , ( o ) be defined for t E [O,T].We write x, E 2 if the process x , ( o ) is measurable with respect to ( o , t ) and for all q 2 1

We write x, E 2 B if x, is a separable process and for all q 2 1 M sup Ix,Iq < m.

,ST

The convergence property in the sets 2 , 2 B can be defined in a natural way.


2. Definition. Let xp, x:, . . . , x;, . . . E 9 ( 9 B ) . We say that the 9-limit (9B-limit) of the process x: equals xp, and we write 9-limn,, x: = xp(9B-lim,,, x: = xp) if for all q 2 1

lim M

n+ m

Jo T ~x;- xplqdt = o

lim M sup x: - xp/q = o).

n+w

tsT

Having introduced the notions of the 9-limit (9B-limit), it is clear what is meant by 9-continuity (9B-continuity) of the process xf with respect to the parameter p at a point p,. 3. Definition. Suppose that p , E Ed, unit vector 1 E Ed, yt E 9 ( 9 B ) . Further, suppose that for each p from some neighborhood of the point p , a process xp E 9 ( 9 B ) is given. We say that y, is an 9-derivative (9B-derivative) of xf at the point p, along the 1 direction, and also, we write

We say that the process xf is once 9-differentiable (9B-differentiable) at the point p , if this process xp has 9-derivatives (9B-derivatives) at the point p, along all 1 directions. The process xf is said to be i times (i 2 2) 9-differentiable (9B-differentiable) at the point p , if this process xp is once 9-differentiable (9B-differentiable) in some neighborhoodlo of the point p, and, in addition, each (first) 9-derivative (9B-derivative) of this process xf is i - 1 times 9-differentiable (9B-differentiable) at the point p,. Definitions 1-3 have been given for numerical processes x, only. They can be extended to vector processes and matrix processes xi in the obvious way. Further, as is commonly done in conventional analysis, we write yf = 9-(a/al)xf if yp" = 9 - ( a / a l ) ~ p l ~for = ~all , p , considered, 9-(dial, d1,)xp 9-(a/al,)[9-(d/al,)xf], etc. We say that xp is i times 9-continuously 9-differentiable if all 9-derivatives of xp up to order i inclusive are 9continuous. We shall not dwell in future on the explanation of such obvious facts. We shall apply Definitions 1-3 to random variables as well as random processes, the former being regarded as time independent processes. In order to grow familiar with the given definitions, we note a few simple properties these definitions imply. It is obvious that the notion of 9continuity is equivalent to that of 9B-continuity for random variables. Furthermore, lMxP - MxP0lI MlxP - xPoI. Hence the expectation of an 9-

-

lo

That is, at each point of this neighborhood.

7. Some Properties of a Random Process Depending on a Parameter

continuous random variable is continuous. Since

the derivative of MxP along the 1 direction at a point p, is equal to the expectation of the 9-derivative of xP if the latter exists. Therefore, the sign of the first derivative is interchangeable with the sign of the expectation. Combining the properties listed in an appropriate way, we deduce that (a/dl)MxP exists and it is continuous at the point p, if the variable xP is 9-continuously 9-differentiable at the point p, along the 1 direction. A similar situation is observed for derivatives of higher orders. Since for z I T

X: is an 9-continuous variable if z(o) I T for all o , xp is an 9B-continuous process, and x: is a measurable function of o . A similar inequality shows that for the same z

if xp has an 9B-derivative along the 1 direction, and if x: and the right side of (1) are measurable functions of o. These arguments allow us to derive the properties of 9-continuity and 9-differentiability of the random variable x: from the properties of 2B-continuity and 9B-differentiability of the process xp. Furthermore, (1) shows that the order of the substitution oft for z and the order of the computation of derivatives are interchangeable. Suppose that the process xp is continuous with respect to t and is 9 B continuous with respect top at a point p,. Also, suppose that z (p) are random functions with values in [O,T], continuous in probability at the point p,. We assert that in this case x:&,, x:(,, are 9-continuous at the point p,. In fact, the difference IxS;",, -+ 0 in probability as p + p, and in addition, this difference is bounded by the summable quantity 2q-1 sup, IxpOlq. Therefore, the expectation of the difference indicated tends to zero, i.e., the variable x:tP, is 9-continuous. The 9-continuity of the second variable follows from the 9-continuity of the first variable and from the inequalities

~: ? ~' ( " , , 1 ~

In conjunction with Holder's inequality M sup t 0, t i , . . . , t i € [O,T], z,,. . . , z j € Ed. Next, we compare the vectors (6) and (8). Also, we find sequences of the processes Passing from vectors of type (6)to vectors of type (8),we prove by induction that the finite-dimensional distribution of x: is equivalent to that of 2:. Therefore, the finite-dimensional distributions of the limits of these processes in the mean square coincide, i.e., x, and j2;. The lemma is proved. 3. Corollary. If ti, a,(x), b,(x) are nonrandom and if, in addition, they are equal to F,(x), &x), respectively, for all t E [O,T], x E Ed, the processes x,, 2, have identical finite-dimensional distributions. Furthermore, the process x, does not depend on 9,, and the process jZ; does not depend on go.

c,

This corollary follows from Lemmas 1 and 2 and the fact that all Wiener processes have identical finite-dimensional distributions and that, for example, w, = w, - w, does not depend on F0. The formula mentioned at the beginning of the section can be found in the next theorem. In order not to complicate the formulation of the theorem, we list the conditions under which we shall prove the theorem. Let Z be a separable metric space with metric p and let (w;,F:) = (wt,Ft),ob;(x),b;(x) be defined for z E Z. We assume (in addition to the assumption mentioned at the beginning of the section) that the functions o:(x,o), b:(x,o) are continuous with respect to z for all t, o , x and

for all x. 4. Theorem. Suppose that the assumptions made before proving the theorem are satisfied. Let the totality of variables {w,,of(x),b:(x):t E [(),TI, x E Ed) be independent of 9,for all z E Z. Further, let 5 be an 9,-measurable random variable with values in Ed and afinite second moment, let be an 9,-measurable


random function with values in 2. Finally, let y, be a solution of the equation

W e denote by x:," a solution of the equation

Let F ( Z , X [ , , ~be ~ ) a nonnegative measurable function on Z x C([O,T],Ed). Then M { F ( ~ , ~1 [9 ~0 ,= )~@(5,5) ~) (a.s.), (11) where cD(z,x) MF(Z,XF$~).

-

PROOF.First we note that due to the conditions imposed, Eq. (9)and Eq. (10) are solvable and, in addition, are continuous with respect to t. Further, it suffices to prove Eq. (11) for functions of the form F(z,x,,, . . . ,x,,), where ti, . . . , tn E [O,T] and F(z,x,, . . . ,xn) is a bounded continuous function of (z,x,, . . . ,x,). In fact, in this case Eq. (11) extends in a standard manner to which are measurable with respect to a all nonnegative functions F(Z,X~,,,~), product of a o-algebra of Borel sets in z and the smallest a-algebra which contains cylinder sets of the space C([O,T],Ed).It is a well-known fact that the latter o-algebra is equivalent to the a-algebra of Borel sets of the metric space C([O, T ] , E d ) . In future, we shall consider functions F only of the type indicated. Let A = {z");i 2 1) be a countable everywhere dense subset in Z . For z E Z we denote by R,(z) the first member of the sequence {z")) for which p(z,z(i)I2-". It is easily seen that iS,(z)is the measurable function of z and that p(z,En(z))5 2-" for all z E Z. In addition, we define the function IC,(X) as in the proof of Lemma 1. By Lemma 1, almost surely

where we take the limit as n -+ oo. We agreed to consider only bounded continuous functions F(Z,X[,,,~)(moreover, of special type). Hence, the left side of (12)yields the left side of (1I ) , if we show that for some subsequence {n' lim sup Ix:n'"J3"n'(0- y,l = 0

>

n'4w t 0 such that Ilo,(o)II Ibt(o)l c,(o) 5 K , (a,A,A) 2 61;112for all A E Ed and (o,t), which satisfy the inequality t < z V 2'. Then for any u E W 2 ( 0 ) v, E W1,'(Q), t 2 0

+

+

+

r

+

e-* grad, u(xr)ordwr,

+ S: e-+'rgrad, v(s + r, xr)ordw,

(1)

almost surely on the sets {z' 2 t } , { z 2 t ) , respectively. Furthermore, for any u 'E WZ(D),v E W1,'(Q) u(xo)= - M J l e - + ' r ~ r u ( xdr r ) + Me-qT'u(xr.),

10. Ito's Formula with Generalized Derivatives

PROOF.We prove both the assertions of Theorem 1 in the same way via approximation of u, v by smooth functions. Hence we prove the first assertion only. Let a sequence vn E C1,2(&)be such that

IIv

- v"IIB(Q) 0,

112,

+

- v"IIw~,~((L) + 07

lllgradx(v - vn)I211d+1,Q 0. -)

Further, let Y, = xo

+ Ji ~r

< P r

dwr +

Ji

b0

0, 0

(12)

SR.

PROOF.We assume that each of the functions o, b, c, f , g, on, b,, c,, f,, g, is equal to zero for t 2 T. Then, in assertions (12) we can replace T-s by T. Further, as can easily be seen, the assertion that the left sides of (12) tend uniformly to zero is equivalent to the fact that relations (12) will still hold if we permit the values a, s, x in these relations to depend on n in an arbitrary way, provided a = a, E QI, s = s, E [O,T],x = x, E S,. In the future, we shall assume that in (12)the values a, s, x are replaced by similar values a,, s,, x, and also that R > 0 is fixed. Let

By Corollary 2.5.12 for any q 2 1 sup M sup ( n rsT

~< 3 co. ~

1. Basic Results

Furthermore, let

Obviously, as n

-+

co

In addition, Ih:(x) - h:(y)l 12w:(Jxl v Jyl),which implies in accord with Theorem 2.7.17 as n -+ co in 2' that

-

Replacing in the above arguments f by o, b, and using Theorem 2.5.9, we have that y: -+ 0 in 9 B as n -+ co. Further, the function wt(r,6)

sup

sup

a s A Ix-YI s 6 . 1 ~ 1 ,I y l s r

If

'(t,x) - f"(t,y)l

tends to zero as 6 1 0 due to the uniform continuity (with respect to a) of f"(t,x) with respect to x. In addition, the above function does not exceed 2K(1 r)". By Lebesgue's theorem

+

lim

6-0

i&

n+m

SoTwSn+.(r,6)dt

5 lim

6-'0 SOT

w.(r,S) dt = 0.

By Lemma 2.7.5, wsn+,(r,a:) -+ co for any r > 0 as n -+ co in measure dP x dt. This yields, by virtue of (13) and Chebyshev's inequality, lim

n-t m

JOT

P{w,+,( E ) dt 5 lim Iiiii

r + w n-+w

SoT~{lt:I > r) dt SoTP{w,+,(r,qD

+ rlim -tm n-rm

> E ) dt = 0.

In other words, ~~,+~(t:,q:) -+ 0 in measure dP x dt. Since, obviously, If""'(s. + t, 4') - P(s, + t, y:)l wsn+t(S:,r]:), the first of the above expressions tends to zero as n -+ co in measure dP x dt. Using (13), we can easily prove that this first expression indicated tends to zero in 2' as well (see the deduction of Lemma 2.7.6 from Lemma 2.7.5). Comparing the above with (14), we conclude that as n -+ co in 9.We would prove this theorem if the functions c;, c", g,, g were equal to zero. If the functions c;, ca, g,, g are not equal to zero, the reader will easily complete proving the theorem by noting that

- f2e-(P215 If1 - f 2 J + If1 + f21

Iv1 - vzl,

3 General Properties of a Payoff Function

if cp,, 402 , 0, and, in addition, applying the previous results as well as Holder's inequality. The theorem is proved.

13. Corollary. Suppose that the assumptions of Theorem 12 are satisjied. Also, suppose that we are given measurable functions g,(x) satisfying the inequality Ig,(x)l 5 K(l + 1x1)" and such that for each R > 0 lim sup Ig,(x) - g(x)l = 0.

n+ m 1x1 s R

Using the functions on, b,, c,, f,, g,(t,x), g,(x), we construct functions v,': v:, w,, v, in the same way as we constructed above the functions va", va, w, v on the basis of the functions o, b, c, f , g(t,x), g(x). Then, wds,x) -+ w(s,x), v,(s,x) -+ v(s,x) as n -+ co uniformly over s E [O,T], x E SR for each R > 0. Moreover, v~'"(~-")(s,x) -+ va, A ( T - S ) (s7x),v%,x) va(s,x) as n -+ co uniformly over a E a , z E m(T), s E [O,T), x E SR for each R > 0. Indeed, for example, +

Furthermore, it is seen that

+ M:,,

sup Ig,(s

tlT-s

+ t, ~,(n))e-~'("'- g(s + t, xt)e-+"I.

The last expression tends to zero as n -+ co by the theorem, uniformly over a E a , s E [O,T], x E SR. We once approximated the given functions with the aid of infinitely differentiable functions, using convolutions with smooth kernels. Let us see what this method of approximation can give in the case considered. We shall show that o,(a,t,x), etc., can be taken to be infinitely differentiable with respect to x. 14. Theorem. Let a sequence E, -+ 0 as n -+ co. Then the assertions of Theorem 12 and Corollary 13 hold for a,(a,t,x) = o'Oen)(a,t,x), b,(a,t,x) = b(07En)(a,t,~), etc. (For the notation, see Section 2.1).

PROOF.We assert that the functions on,b,, c,, f,satisfy, in general, inequalities

(1)-(3) with identical constants K and m, and that, in addition, these functions converge to o, b, c, f in 8,([O,T],B). Furthermore, we need verify (ll), (15) and the fact that Ig,(x)l Ig,(t,x)l 5 N(l \XI)",where N does not depend on n, t, x.

+

+

1. Basic Results

The desired estimates for on, b,, c,, f,, g, follow from the fact that, for example,

I sup Ilo(a,t, x - ~,z)llI K(l 121

s1

+ E, + 1x1)

Further, as in proving Theorem 12, we introduce here wt(r,6). We mentioned that wt(R,&)+ 0 as E + 0 for each t, R. From this, for each t we obtain

By Lebesgue's theorem,

1

sup sup f :(t,x) - f*(t,x)l dt a e A ixlsR

-+

0.

In a similar way we prove relations (1I), (15)and the fact that the functions on, b,, c, + a, b, c in Y1([O,T],B).The theorem is proved. In some cases one can take the functions a,, b,, c,, f,,g, to be infinitely differentiable with respect to (t,x). 15. Theorem. Suppose that the set A consists of aJinite number of points only and, in addition, that the sequence E, -+ 0 as n + oo, lenl I 1. Then the assertions of Theorem 12 and Corollary 13 hold for ~,(a,t,x)= o('")(a,t,x), b,(a,t,x) = b(")(a,t,x), etc. (in computing the convolution with respect to (t,x) we assume for t I0 that o(a,t,x) = o(a,O,x),etc.).

PROOF. Estimates of the growth of the functions a,, b,, c,, f,, g, can be obtained in the same way as in the preceding proof. Furthermore, Eqs. (11) and (15),as was mentioned in Section 2.1, are known. Therefore, it remains to show that on, b,, c,, f, + a, b, c, f in 9,([O,T],B).We shall prove only the convergence off,. For proving the convergence of a,, b,, c,, we repeat wordfor-word the corresponding arguments. Note that the definition of convergence in 9,[O,T], B) involves the upper bound with respect to A. Since A consists of a finite number of points, the upper bound mentioned does not exceed the sum (with respect to a E A) of expressions standing under the sign of the upper bound. Therefore, we prove


that f,

-+ f

in 9,([O,T],B)if we show that for each a E A for all R > 0 sup If :(t,x) - f*(t,x)l dt = 0. !! ,,,,, JOT

Let us take the function w,(r,d)from the proof of Theorem 12. Writing the convolution f :(t,x) in a complete form and recalling that [(t,x)= i,(t)[(x) (see Section 2.1),we easily prove that for 1x1, lyl < R, - yl I E

IX

w:(R + l , ~ , ) , ( ff(t,x)- f f(t,y)l Iwt(R + 1 , ~* ~&i151(t&i1) ) which implies convolution with respect to t. Let h: = f: - f '. For 1x1, lyl < R, lx - yl

<E

1

Ih:(t,x) - hxt,y) Iw:(R

+ 1,E ) + w,(R,E).

Next, we apply Lemma 2.7.15, taking in this lemma 5 instead of R. For each E > 0

Ex E

S, and R

+1

From this it follows that the limiting expression in (16)is smaller than

In the last expression the second term tends to zero as n -+ oo since the mean of any function from 9, converges in 9,.For the same reason the first term as n -+ oo tends to JOT

wt(R + 2, E ) dt

+

JOT

wt(R

+ 1. E ) dt.

(17)

Thus, (17)estimates the left side of (16)for any 8 > 0. In proving Theorem 12, we saw that (17)tended to zero as E -+ 0. We have thus proved Theorem 15.

2. Some Preliminary Considerations We shall prove the assertions of Theorems 1.5-1.11 by approximating an arbitrary strategy with the aid of step strategies, i.e., using strategies which are constant on each interval of a subdivision I of an interval of time [O,T]. We would expect that the upper bound of the payoffs given by all step strategies which have been constructed on the basis of subdivisions I = ( 0 = to, t,, . . . ,t, = T ) will tend to the corresponding payoff function as maxi(ti+, - ti) -+ 0. In this section, we prepare the proof of the above fact in a special formulation (see Theorem 3.2), and, furthermore, we prove that U ( S , X ) and w(s,x)are continuous with respect to x. Using the definitions, assumptions, and notations given in Section 1, we introduce here some new objects. Let us take E A, 0 5 s 4 t, and a function

2. Some Preliminary Considerations

u(x).Also, let us define a strategy j?,

= P and assume, in addition, that

In order to make oneself familiar with the operator G,,, we suggest the reader should work out the following exercise. 1. Exercise Let 0 $ so $ s, . . . 5 s, = T.Show that G,,,,, G,,,,,, . . . , G,n_,,,,, g(x) is the upper bound of va(so,x) with respect to all strategies a E '2I for which a, is constant on each semiinterval of time [si - so, si+ - so).

We shall repeatedly assume about the functions u(x) to be substituted in the operators GB,G, that for some constants K and m 2 0 for all x E Ed

+

In this case G[tlul(x)I N ( l Ixl)", where N does not depend on j?, s, t, x. As was seen in the discussion of Theorem 1.5-1.7, such inequalities readily follow from estimates of the moments of solutions of stochastic equations. 2. Theorem. Let the continuous function u(x) satisfy inequality (1). Then the function G!,,u(x) is continuous in x uniformly with respect to j? E A and s, t such that 0 I s t T. The functions va(s,x), va,'A(T-fl(s,x)are continuous in x uniformly with respect to a E 'U, s E [O,T], z E m ( T ) . In particular, the functions G,,u(x), v(s,x), w(s,x) are continuous in x uniformly with respect to s, t such that 0 I s 4 t I T .

PROOF.The last assertion follows from the fact that, for example, sup IGs,tu(xn)- Gs,tu(xo)l5 sup sup IG!,tu(x,) - G!,tu(xo)l. OssstsT Ossst5T B o A Furthermore, the right side of the last expression tends to zero as x, + x , according to the first assertion of the theorem. Next, we take a point xo E Ed, a sequence x, -+ x,, and also we assume that h, = x, - x,, o,(a,t,x) = o(a,t , x + h,). In a similar way, we introduce b,, c,, f,, g,, u,. For instance, u,(x) = u(x + h,). Since ca(t,x)is continuous in x uniformly with respect to a, for each t sup,,, sup,,, s R Ic;(t,x) - ~ ( t , x ) l - 0+ as n + oo.By Lebesgue's theorem, lim n-rm

JOT

sup sup lc;(t,x) - ca(t,x)ldt aoA I x I ~ R

= 0.

It is not hard to verify that the remaining assumptions of Theorem 1.12 are satisfied. Therefore, Theorem 1.12 is applicable in our case. Furthermore,


we note that the process xf,"."O(n)from Theorem 1.12 as well as the process ~ f , " ,-" h, ~ satisfy the same equation in an obvious way. Hence, x?"."O(n) = ~9%'"" - h,, c:(s + t, x:,',"O(n)) = cat(s+ t, x;,'"~), etc. By Theorem 1.12 we have that

as n + co uniformly with respect to a E %, s E [O,T]. Taking instead of g(s,x) the function u(x) in the last relation (this can be done because of continuity of u(x) and by virtue of (I)),we find as n + oo uniformly with respect to a E a,s E [O,T]. We derive the assertions of the theorem from the limiting relations proved above, in an elementary way. This completes the proof of Theorem 2. Further, we need continuity of va, Ga with respect to a. Let a metric in the set A be given by a function p(al,a2).We assume that p(a1,a2)< 1. for all a,, a, E A. We can easily satisfy this inequality if we replace, when needed, the initial metric by an equivalent metric, using the formula 2 p'(a,,a2) = - arctan p(a,,a2). .n

3. Definition. For a', a2 E % let

fla',a2)

= M JOT

p(a:,a:)

dt.

If a" E % (n = 0,1, . . .) and p"(an,aO) + 0 as n + co,we write an + aO. Since p(al,a2) < 1, p"(a',a2) will be defined for each a', a2 E %. 4. Exercise Using Theorem 2.8.1, prove that if p"(a1,a2) = 0, then for all (s,x).

By hypothesis, the set A is separable. We fix a countable subset {a(i)) dense everywhere in A. 5. Definition. Let I = {0 = to,t , , . . . ,t, = T ) be a subdivision of an interval of time [O,T], a E %, and let N be an integer. We write a E %,,(I,N), if


a,(o) E { a ( l ) ,. . . ,a(N)) for all o E S2, t E [O,T], and a, = ati for t E [ti,ti+,), i = 0, 1, . . . ,n - 1. Let %,,(I) = %,,(I,N), a,, = %,,(I). Strategies of the class a,,are said to be step strategies.

UN

0,

6. Lemma. Suppose that the diameter of the subdivision I , of the interval [O,T] tends to zero as n -+ co. Then for each strategy a E there is a sequence of strategies a" E 91cT(I,)converging to the strategy a E

a.

PROOF.The distance p" satisfies a triangle inequality. Hence it suffices to show that:

U,

%,,(I,) is dense in % ;, a. in the sense of the distance p" the set b. the set a,, is dense in a set of all the strategies each of which assumes only a finite number of values from {a(i)); c. the latter set is dense in %. Proof of (a). If a E acT, for some subdivision I = ( 0 = to,t i , . . . ,t, = T ) the equalities a, = ati can be satisfied for t E [ti,ti+,).Using the strategy a and the subdivision I,, we construct a strategy an so that a: will be right continuous, constant on each interval of the subdivision I,, and, in addition, coincide with a, at the left end points of the foregoing intervals. In this situation a: differs from a, only on those intervals of the subdivision I, each of which contains at least one point ti. It is seen that p(a:,a,) + 0 as n + co everywhere, except for, perhaps, the points ti. Hence p"(an,a)-, 0. Proof of (b). We take a strategy a, that assumes values in {a(l),. . . ,a(N)). In a Euclidean space EN let us choose N arbitrary points x,, . . . ,xN so that Ixi - xjl 2 1 for i # j. Let P,(o)= xi, if a,(o) = a(i), t E [O,T], P,(o)= 0 for t > T. It is easily seen that for s, t E [O,T]

Assuming K, = a,, fit = Po, we define completely the functions a,, P, for negative t. Let ic(n,t) = j2-" for j2-" 5 t < ( j + 1)2-", j = 0, 1, f2, . . . It is a well-known fact (see, for instance, the proof of Lemma 4.4 in [51]) that there exists a number s and a sequence consisting of integers n' -+ co, such that

+

+ 0 as By virtue of (2),we have for the functions a: = a,~n,,-,,+s:p"(an,a) n' -+ co. Furthermore, it is not hard to see that ~ ( nt ,- s) + s is a step function oft, ~ ( nt ,- s) + s 5 t. Hence a: is 9,-measurable and a" E % ., Proof of (c).Let us introduce on A the following functions:

It is seen that K,(u) is equal to that a(i),which is at a distance from a of not more than lln, and, which in addition, has the smallest possible index. Since


{a(i)) is dense everywhere in A, the functions in(a),lc,(a) are defined on A and p(lc,(a),a)) Illn for all a E A. Further, let lcn,,(a) = a(N A i,(a)). Obviously, lc,,(a) -+ a, if we let first N -+ co, second, n -+ co. Hence p"(lc,,(a),a) -+ 0 under the same conditions for each strategy a E %. It remains only to note that the strategy lcn,,(at) takes on values only in the set {a(l), . . . ,a(N)). We have thus proved the lemma.

7. Lemma. Let s E [O,T], let z,, z, be random variables with values in [0, T - s], and, jinally, let u(x) be a continuous function satisfying the condition (1). Then the random variable

is an 9-continuous function of (a,x)for a E %, x E Ed.

PROOF. Note first that if an -+ a, xn -+ x, the 9B-limit x:","."" is equal to xFhX,which fact follows, according to Theorem 2.8.1, from the continuity of o(a,t,y) and b(a,t,y) with respect to a, boundedness of o(a,t,y) and b(a,t,y) for fixed y, and, finally, convergence of a:(o) to at(@)in measure dP x dt. Further, reasoning in the same way as in the proof of Lemma 2.7.6, and, moreover, using condition (1.3) as well as the continuity of P(t,x) and f"(t,x) with respect to (a,x), we can prove that the processes cat(s t, x:,~,~), f Q ( s+ t, x:,"") are 2'-continuous with respect to (a,x). Also, applying the results obtained in Section 2.7 related to the 9B-continuity of integrals and the 9-continuity of products of 9-continuous processes, we immediately arrive at the assertion of the lemma. Lemma 7 is proved.

+

8. Corollary. For s E [O,T] and z E 9X(T - s) the functions va(s,x)and ua,'(s,x) are continuous with respect to (u,x) for a E a,x E Ed. For 0 I s I t I T the function G:,,u(x) is continuous with respect to (P,x) on A x Ed. Combining Corollary 8 with Lemma 6, we have

9. Corollary v(s,x) = lim

sup

n + m as'U,,(I,)

va(s,x)

for any sequence of subdivisions whose diameter tends to zero.

10. Exercise Prove that (3) is 9-continuous with respect to s and, next, deduce from that the continuity of va(s,x)with respect to s. This together with Theorem 2 enables one to conclude that va(s,x)is continuous with respect to (s,x)and that, in addition, v(s,x) is a Bore1 function of (s,x).


We prove some other properties of the operators G!,,. Letting the sequence E, -+ 0, E, # 0, we consider the mean functions for the functions a, b, c, J: Let a,(a,t,x) = a(")(a,t,x) (see the notation in Section 2.1), etc. In other words, we take a,, b,, c,, f, from Theorem 1.15. We denote by ~:,~,."(n) a solution of the equation dx, = a,(a,, s

+ t,x,) dw, + b,(a,, s + t,x,) dt,

Furthermore, let p"""(n) =

-

& e ( s + r, ~ : ~ , ' ( ndr) )

and, in addition, for a constant strategy P,

P for 0 I

s I t I T let

Regarding P as a single point of A and using Theorem 1.15 as well as the estimates of moments of solutions of stochastic equations, we have the following assertion.

11. Lemma. Let a continuous function u(x) satisfy condition ( I ) , and let u,(x) = u('")(x).Then Ie:u,(x)l for all s I t I T , x Furthermore,

E Ed, n

E

+ 1x1)"

> 0, P E A, where N does not depend on s, t, x, n, P. G!;:u,(x)

as n -, co for each t IT , 1x1 I R}.

IN(1

+

G!,tu(x)

A uniformly on each set of the form {(s,t,x):OI s I

The functions a,, b,, c,, f,, u, are smooth with respect to (t,x).In addition, their derivatives grow not more rapidly than (1 1x1)". For example,

I u,(x)l IE; ' J; =

21s1

+

u(x - e,z) : [(z)dz dl'

I

By Theorem 2.9.10, the function G!;:u,(x) is the unique solution of a certain equation. By Remark 2.9.1 1 the foregoing function is uniquely determined by the functions a, = +anax,b,, c,, f,,u,, for finding which it suffices, obviously, to give a, b, c, f, u. This, by Lemma 11, implies

12. Corollary. The function G!,,u(x) does not change i f we change the probability space and, furthermore, take another dl-dimensional Wiener process. The function G!,,u(x) can be determined uniquely by o, b, c, f , u.


Let us use the properties of the function G~;;u,(x) to a greater extent. 13. Corollary. Suppose that p E A, and that the function o(p,t,x)(b(P,t,x)) for each t E [O,T] is twice (once) continuously diflerentiable over x. Furthermore, suppose that second (jrst) derivatives of the foregoing function with respect to x are bounded on any set of the form [O,T] x SR. Let t E [O,T], and let q(s,x) be an injinitely differentiable function on Ed+,, which is equal to zero outside a certain cylinder [O,t] x SR. Then where

In fact, let

a + 1 akJ(p,s,x)a2 + axiaxj

Lfi(s,x)= as

i,j=l

d

bk(p,s,x)

-a axz

- cfll(s,x).

By Theorem 2.9.10, in a strip [O,t] x Ed. Multiplying the last equality by q, integrating by parts, and, in addition, introducing an operator Lfll*in the usual way, we have

It remains only to let n + co and to note that the integration is to be carried out over a bounded set and, for example, for almost all s, x. (For the properties of mean functions, see Section 2.1). The final property of the operator Gt,, which we give in this section follows immediately from Theorem 2.9.7 and Remark 2.9.9.

14. Lemma. Suppose that s E [O,T], 0 I tl I t2 IT - s and a strategy a E 2l is such that a, = a,, for t E [tl,t2). Let the continuous function u(x) satisfy condition (1). Then almost surely

Note that for proving the lemma we should take in Theorem 2.9.7, A, t,, t2, x:;~'~,at,, af+,(x), bf+,(x), instead of 2, s, T, l, 5, aF(x), b:(x) respectively.

3. The Proof of Theorems 1.5-1.7

3. The Proof of Theorems 1.5-1.7 In the preceding section we proved that some mathematical expectations of the form MZ,,Fa are continuous with respect to (a,x) on '? xI Ed. Furthermore, we learned how to approximate any strategy by means of step strategies. Also, we introduced the operators G!, G,,, which are crucial for the discussion in this section. Having thus completed the technicalities, we proceed now to prove Theorems 1.5-1.7. 1. Lemma. Let so < s, < . - . < s,

=

T. Then

PROOF.Let ui(x) = G ,,,, +,. . . G,,_,, ~ ( x (i) = O,l,. . . ,n - I), un(x)= g(x). Also, we fix E > 0. By Theorem 2.2, the function u,-,(x) = G,,-,,,, g(x) is continuous and furthermore, it satisfies the inequality l ~ , - ~ ( x ) l N(l + 1x1)". This implies in accord with Theorem 2.2 that the function u,-,(x) = Gs,-,,sn_l~n,(x) is continuous. Arguing in the same way, we convince ourselves that all the functions ui(x) are continuous. Further,

By Corollary 2.8, the functions G,,,, ,ui+,(x) are continuous with respect to p. Hence the last upper bound can be computed on any countable set everywhere dense in A. Noting in addition that Gt,s, ,ui+ ,(x) is continuous with respect to x according to Corollary 2.8, we conclude that there exists a (countable-valued)Bore1 function Pi(x) such that for all x +

+

In a space of continuous functions xIo,,, with values in Ed we define the function a,(x[,,,)) = a , ( ~ [ ~ using , , ~ ) the formula a , ( ~ ~ = ~ Pi(~,,_,o) ,,~) for t E [si - so,si+, - so), i = 0, . . . , n - 1, a , ( ~ ~ = ~ ,PO(0) , ~ ) for t 2 T - so. It is seen that the function a, is progressively measurable with respect to {Jlr,) and also it is seen that the equation

is equivalent to a sequence of equations


etc. Each of the equations given is solvable. Therefore, aiis a natural strategy admissible at each point (s,x). Finally, by Lemma 2.14 for i = 0, 1, . . . , n - 1

Adding up all such inequalities and collecting like terms, we find thus proving our lemma. In the theorem which follows we prove the first assertion of Theorem 1.7.

2. Theorem. (a) v(,, = u. (b) Let so = sb < s\ < . . . < st(, maxj(s; + - si) -t 0 for i --+ co. Then

=

T (i = 1,2, . . .),

PROOF.Assertion (a) follows from (b), Lemma 1, and the obvious inequality v(,, 5 v. Furthermore, it follows from Lemma 1 that the upper bound in (1) does not exceed v(so,x).Since the upper limit is smaller that the upper bound, to prove (b) we need only to show that

2.9, Using Corollary . . we construct step strategies so that vai(so,x)-+ v(so,x) as i + oo and a: = ~ l ~ i - , , for t E [s: - sO,sj+ - so). Also, we introduci functions u; according to the formulas U:(~,(X) = g(x), u~(x)= Gs;,s;+lu~+l(~) ( j = 0,1, . . . ,n(i) - 1). By Lemma 2.14

Adding up such inequalities with respect to j from j = 0 to j = n(i) - 1, and, in addition, collecting like terms, we obtain: vai(so,x)5 ub(x). Therefore, v(so,x) 5 b,,,ub(x), which is completely equivalent to (2). The theorem is proved.


3. Exercise Prove that if the subdivisions {sj} are embedded, the functions under the limit in (1) converge monotonically to v(s,,x).

4. Remark. The theorem proved above together with the constructions made in Lemma 1 provides a technique for finding &-optimalstrategies in the class of step natural strategies.

5. Lemma. (a) Let s E [O,T], x E Ed, cc E a.Then the processes

deJined for t E [0, T - s], are supermartingales with respect to IF,} thejrst , process being nonnegative (as.). (b) Gs,tv(t,x)5 U(S,X) for x E Ed, 0 5 s 5 t 5 T.

PROOF. It is seen that

where the right side is a martingale. Hence 6;'"'" is a supermartingale if ~ 2 " "is a supermartingale. The nonnegativity of 6 ; ~ "follows ~ from the fact that by the definition of a supermartingale, 62"3" 2 ~:,,{6~-,135)and g.,s,x T-s - 0. Further, by Theorem 2.2 the function v(s t, x) is continuous with respect to x. In addition, Iv(s + t, x)I I N ( l 1x1)". Therefore, by Lemma 2.7

+

+

for each t E [0, T - s] if an -+a. By Lemma 2.6, we can choose step an + a, which implies that the supermartingaleness of x;,"," needs to be proved only for step strategies. Since the constancy segments of a, can be considered one 5 K;;~,"( a s ) for t2 2 t1 if by one, it suffices to prove that ~:,,{x,~l9",~) cc, = a,, for t E [tl,t2). By Lemma 2.14, for such a strategy

from which it is seen that it remains now to prove assertion (b) of the lemma. We assume 8, E A, so = t, and, in addition, we construct a sequence of subdivisions so = sb < si < . . . < sico = T of an interval [t,T] so that


maxj(sj+, - sj) + 0. By Theorem 2 and Lemma 2.14, it is not hard to obtain

where the constants N do not depend on x. This implies that for each fi E A the magnitude of the sequence 8,"~) Gs;,s; ' ' ' Gs;(,)-t,%(,)g(~t-s

does not exceed N(l + I ~ f 5 ~ ] )the " , latter expression having a finite mathematical expectation. Therefore, recalling that

and also applying Lebesgue's theorem, we easily find G!,tv(t,x)

= lim

i+ m

G,qtGs;,s; . . - Gs:(,) - l,s;(,)g(x),

where the expression standing under the limit does not exceed v(s,x) in accord with Lemma 1 or Theorem 2. We have thus proved Lemma 5. 6. Theorem (Bellman's Principle). For s I t I T

in this case we can take the upper bound with respect to a E '21E(s,x)as well.

PROOF. The properties of supermartingales imply that U(S,X)= Mf,,icO 2

Mf,xict-s

2 M~,.IC~= - ~ v'(s,x).

Taking upper bounds with respect to a E '2l or a E %,(s,x), we prove the required result. The following lemma proves Theorem 1.5.

7. Lemma. Thefunction v(s,x) is continuous with respect to (s,x)for s E [O,T], x E Ed. PROOF.By Theorem 2.2, the function u(s,x) is continuous in x uniformly with respect to s E [O,T]. Therefore, it suffices to prove that v(s,x) is continuous in s for each x. We fix x,. We need to prove that if s,, t, E [O,T], Assuming that A consists of a single point

Po, we have in (3) equalities instead of inequalities.


t, - S, --t 0, then v(sn,xo)- v(tn,xo)+ 0. We consider without loss of generality that t, 2 s,. Further, we use Theorem 6 for x = xo, s = s,, t = t,, and, in addition, choose an E 'U such that the upper bound mentioned in the assertion of Theorem 6, attained for a = a" to within l/n. We have

where the superscript n attached to x, cp stands for (an,s,,xo).By Corollary 2.5.12, for any q 2 1 sup M sup Ix:Iq < co, n

t5T

from which it follows due to (1.3) that the limiting expression in the first term of (4) does not exceed N(t, - s,), and also that this term itself is equal to zero. If we replace f by c in the above arguments, and if, furthermore, we use Chebyshev's inequality, we can see that cp:,-sn -+ 0 in probability. By Corollary 2.5.12, x:~-,~-+ xO in probability. Due to the uniform continuity of v(t,x) with respect to x,

as y -, 0. It follows, in turn, that h(x:,-,, - xo) -+ 0 in probability. In particular, V(~.,X:~-~~) - v(tn,xO)--t 0 in probability. We can now easily prove that the expression standing under the sign of mathematical expectation in the second term in (4) tends to zero in probability. From (5) we conclude that the mathematical expectation of (5) as well tends to zero (compare with the deduction of Lemma 2.7.6 from Lemma 2.7.5). The lemma is proved. 8. Proof of Theorem 1.6. We shall drop the superscripts (a,s,x). Further, we take the supermartingale k-, = ".:-k from Lemma 5, which is, according to Lemma 7, continuous in t. Therefore, by the lemma given in Appendix 2, the processes

p,

=

Ji [/".(s

+ u, xu)+ r.v(s + u, xu)]exp

(

- cp. -

J,,r. du

)

du

are supermartingales for t E [0, T - s]. Therefore, 0 = M:,,Ck-o

- pol 2 M:,xClcr - p,l.

Applying the properties of supermartingales one more time, we obtain


It remains to take in the above inequalities upper bounds with respect to a E 8 , which completes the proof of Theorem 1.6. If we take in the above inequalities upper bounds with respect to cr E %,(s,x) and if, in addition, we use Theorem 2a, we arrive at the second assertion of Theorem 1.7, thus completing the proof of Theorem 1.7.

9. Exercise In proving Lemma 7, we introduced the function h(y). With the aid of h, we define a convex modulus of continuity of v(t,x) at a point xo according to the formula

*, h(y). where ?i(r)= suplul Prove that if N-

I1, where N = N(K,T,m).

10. Remark. From Theorem 2 and Corollary 2.12 it follows that the function v(s,x) can be defined uniquely after the functions o, b, c, f , g have been given. The function v(s,x) depends on neither the probability space nor the Wiener process involved.

4. The Proof of Theorems 1.8-1.1 1 for the Optimal Stopping Problem In this section we shall use the method of randomized stopping (see Section 1.2). Recall that this method consists in the introduction of the multiplier exp(-Sb r, du) into the functional which characterizes the payoff, and also in the replacement of the function f" by f" + rg. In accord with this remark we shall carry out the following construction. For n > 0 let B, = A x [O,n]. Furthermore, for P = (cr,7) E B, let

It is clear that for each n for p E B,, t 2 0, x and y E Ed the functions o(P,t,x), b(P,t,x) satisfy conditions (1.1) and (1.2) with the same constant K, and, in additon, the functions cp(t,x),f p(t,x),g(x) satisfy the growth condition (1.3) with the same constant m and a different constant K. Hence as in Section 1 where we introduced the concepts of a strategy, a natural strategy, and a payoff function with respect to A, o(a,t,x), b(a,t,x), ca(t,x)f"(t,x), g(x),

4. The Proof of Theorems

1.8-1.11 for the Optimal Stopping Problem

we can introduce here analogous quantities with respect to B,, a(P,t,x), b(P,t,x), cP(t,x),f P(t,x),g(x) = g(T,x). We denote by 23, the set of corresponding strategies, and denote by B,,,(s,x) the set of natural strategies. Let %, be a set of nonnegative processes F, which are progressively measurable with respect to {Ft}and such that Ft(o) I n for all (t,o), 8 = UB,, BE(s,x) = uBn,E(~,x), % = U%nEach strategy P E 23, is, obviously, a pair of processes (a,,r,), with a = {a,} E %, P = {F,}E %,. Conversely, each pair of this kind yields a strategy in 23,. It is easily seen that if P = (47)E En,xFS,"is a solution of the equation

In other words, xf3""= xr:",". Let Ws,x) = sup M s-J:[,! BE%

+

+

1

fPt(s t , ~ , ) e - ~ ~ dgt( T , ~ , - ~ ) e - * ~. - ~

Here, as well as above, the indices attached to the sign of the mathematical expectation imply the mathematical expectation of a expression in which these indices are used wherever possible. Theorems 1.5-1.7, as well as the results obtained in Sections 2 and 3, are applicable to the function Fn(s,x)as well as to a payoff function in the control problem without stopping. In particular Fn(s,x)is continuous with respect to (s,x), gn(T,x)= g(T,x) (Theorem 1.5). 1. Lemma. Let s E [O,T], x E Ed, P = (N,?)

E

23. Then the process

dejned for t E [0, T - s], is a continuous supermartingale.

PROOF. By Lemma 3.5a for P E 23, the process

is a supermartingale. In particular, (P = (a,O)),

is a supermartingale. It remains to apply the lemma from Appendix 2 to the last expression, thus completing the proof of our lemma. 2. Lemma. Let s E [O,t], x E Ed, y i E 1)32(T- s), = (ai,?) E B(i = 1,2, . . .). Further, let a Bore1 function u(t,x) satisfy the inequality lu(t,y)l I N(l Iyl)" with the same constant N for all t 2 0, y E Ed. In addition, let

+


Then

f 5:(s + t, xt)e-qtdt

lim M l x!:{J:

i-+m

+ u(s + yi,xyr)e-qyi

PROOF. It can easily be seen that ~f PI(, + t, x;i's'x)e-'li,s.x -

I

I faf(s

+

+ t, x~'s'x)le-

q:i3s3x

t, X;i,~.~)e-~:i~s~x~

(

(1 - exp -

J:

dp))

Integrating the both sides of the last expression over t E [O,yi], introducing the notation hi = sup (1 Ix;',S'xl)m

+ and finally, noting that I f"(t,x)l a K(l + Ixl)", te[O,T-s]

< KMhi -

[yi 0

(1 - exp(-

It is also seen that IM!,;U(S

J: Tbdp))

dt

Iq(t,x)I a K(l

+ J:

7:erp(-

+ IxI)", we find

J: 7; dp) dt]

+ yi,xYi)e-q~'- ~;f,u(s+ yi,xYi)e-qyil

Therefore, it suffices to show that the last expression tends to zero. Since ca(t,x)5 K(l + Ixl)", then cp;i,"x IKRT for h' a R. Hence

+ R G MxhiaR ~ ~ ~ ~ 7 ~ e x p ( - ~ : T ~ d p ) d t i-t m

1 R

5 - sup M(hi)' i

+ ReKRT

i+ m

(

MxhiSR yzT:exp -cp;i3s3x So'

J:%$) dt.

4. The Proof of Theorems 1.8-1.1 1 for the Optimal Stopping Problem

By hypothesis, the last term is equal to zero. Furthermore, it follows from estimates of moments of solutions of stochastic equations (see Corollary 2.5.12) that sup, M(hi)2< co. Therefore, letting R -. co in the inequality 1 I I- sup ~ ( h ' ) ~ , R i we obtain I = 0, which proves the lemma.

3. Lemma. (a) Let s E [O,T], x E Ed and for each a E 9.t let z" E %X(T- s), r" E 9' 3 be dejned. Then

(b) Let g, F,(s,x)

=g A

= sup

5,. Then sup

a s % re'Jn(T-s)

M:,LJ;

/"'(s

+ t, x,)e-"dt + gn(s+ 7, xde-*

1

Furthermore, we can replace in (a) and (b) the upper bound with respect to a E 9.t by an upper bound with respect to a E %,(s,x).

PROOF.By Theorem 1.6, Eq. (1) holds for any za, ra if it holds for z" = T - s, ra = 0. We deduce assertion (b) from (a) for r" = 0 in the same way as we deduced the corresponding assertion from Eq. (1.5.2) in proving Lemma 1.5.2. Theorem 1.7 implies that it is possible to replace % by %,(s,x) in the preceding considerations. Thus, it remains only to prove (1) for za = T - s, ra = 0.Let P = (a3) E 23,. Furthermore, let

-

According to Lemma 3.5 the process K, is a supermartingale. According to the lemma given in Appendix 2, the process p, K,@, - r,dG, is a supermartingale and, in addition,

So

F,,(s,x) = Mpo 2 MpT-, 2 e"(T-S'[ " ~ ~ - s- Fn(s9x)]+ V".(S,X).

(2)

Using Fubini's theorem, we easily prove that MpT-.

+

= ~;,{~(~,x,-,)e-q~-~

[f'

+l,(g - Q](s

1

+ t,xt)epmdt .


Obviously, the upper bound of the last expression with respect to f E %, is equal to

[f"' + n(g - E,,)+](s

1

+ t , ~ ~ ) e - ' ~+d gt ( T , ~ ~ - ~ ) e -. ' ~ - ~

Taking this fact into consideration, recalling the definition of En, and, finally, computing the upper bounds in (2) with respect to a E (II, 7 E %, we arrive at (1)for z" = T - s, r" = 0. The lemma is proved. 4. Corollary. Since g, I g, 5, I w.

5. Lemma. (a) The function w(s,x) is continuous with respect to s, x. (b) There exists a constant N such that IiT,(s,x)l < N ( l + 1x1)" for all n, s, x. (c) En(s,x)f W ( S , X ) uniformly on each set of the form {(s,x):s E [O,T], 1x1 5 R). PROOF.Assertion (a) follows from (b) as well as the continuity property of En(s,x). Since '23, c B,,,, the sequence E,(s,x) increzses. Moreover, in accord with Corollary 4, En I w and, obviously, E,,(s,x) I E,(s,x); in this case the function E,, does not differ essentially from the function v considered in Section 1. All this together with the estimates of v, w given in Section 1 proves assertion (b). Let ~ ( s , x=) limn,, E,(s,x). By Corollary 4, W(s,x) I w(s,x). On the other hand, for a E a,z E m ( T - s) let 'F, = nX,,,, fit = (at,Tf).Then, using Fubini's theorem, it is not hard to obtain

Let us write last relation in a different form. Let

for t I T - s, ~ " " ~ (=t )ya,S,X(T- s) for t > T - s. Furthermore, we introduce a random variable 5 which has an exponential distribution with a parameter equal to unity and which, in addition, does not depend on {ya3"."(t)}.What we have obtained can be written as follows: E,(s,x) 2 M:,,y(z + (l/n)(). W(s,x) 2 M:,,y(z + (l/n)t). Letting n tend to infinity, we note that the process ya."."(t)is continuous with respect to t,

4. The Proof of Theorems 1.8- 1.11 for the Optimal Stopping Problem

and, finally, we note that the last quantity is surnmable. Therefore, by Lebesgue's theorem, W(S,X)2 M~,,v(z)= va~'(s,x),

W(qx) 2 ~ ( q x ) .

We conclude that w(s,x) = w(s,x). From the last equality and the inequality w(s,x) 2 g(s,x) it follows, in particular, that the decreasing sequence of nonnegative continuous functions g(s,x) - gn(s,x) = g(s,x) - g(s,x) A En(s,x) + ~ ( s , x) ~(s,x)A W(S,X) = 0. By Dini's theorem, g(s,x) - g,(s,x) -+ 0 uniformly on each cylinder C,,,. In view of Lemma 3 (and Corollary 1.13), in order to prove (c) it suffices to show that Ign(s,x)(5 N(l + 1x1)" with the same constant N for all n, s, x. The last inequality follows from assertion (b), thus proving the lemma.

6. Remark. The proof of (a) completes the proof of Theorem 1.8. Theorems 1.9 and 1.10, as was seen in Section 1, follow from Theorem 1.11. For proving Theorem 1.11 we need an analog of Lemma 1, which is a combination of Lemma 1 and Lemma 5 (and Theorem 1.12). 7. Corollary. Let s E [O,T], x E Ed,B = (47) E .%2 p f , ~=, ~ W(S+ t, x:,S3x)e-q f , ~ , ~

Then the process

deJined for t E [0, T - s], is a continuous supermartingale. The process pf;;" is also a supermartingale for each z E %R(T- s). Subtracting from the process pff;" for B = (cr,O) the martingale

we arrive at a supermartingale which for t = T - s is equal to The last expression is nonnegative. Further, from the definition of the supermartingale it immediately follows that a supermartingale which is positive (a.s.) a t a certain moment of time, is positive (as.) at each preceding moment of time. Summing up what has been said above, we have the following result. 8. Corollary. Let s E [O,T], z E %R(T- s), x E Ed, cl E a.Then the process

is a nonnegative supermartingalefor t E [0, T - s].


9. Proof of Theorem 1.11. Corollary 7 and the properties of supermartingales imply

which proves inequality (1.10). Next, let za I z:S3x, E > 0. For P = ( a 7 )E (I) let zP = za. According to Bellman's principle (see Theorem 1.7) for each n Cn(s,x)=

sup

P E IB,.E(S,X)

M,:

(f;

fL(s

+ T,g(s + t, x,)] exp

+ t, x,)e-.'

dt

+ Cn(s+ 7 ,x,)e

-q=

J 9 Tpd p dt

- cp, -

Taking the limit in the last expression as n -+ cc and, furthermore, using the inequality g(s,x) s w(s,x), (3) and the fact that fT, t w, we have

Further, we take a sequence a'

E %,(s,x),

FiE %, for which

From the inequality g(s + t, x;,".") < w(s also from (3) and (4),we find E

lim ME,

i- m

fi7:exp(-

cp, -

+ t, xpqX)- E for t < z:"."

fdi: d p ) dt = 0.

and

4. The Proof of Theorems 1.8-1.1 1 for the Optimal Stopping Problem

By Lemma 2, the last expression and (4) yield

By Corollary 7, the process

is a continuous supermartingale. Therefore, according to the lemma given in Appendix 2, the process K:,"." - pt,S,"is a supermartingale for each p = (cl,r) E 23. In particular, M;,,K, IM!,xp,, which together with (5) and (3) yields w(s,x) r lim M&{J~ [/.:(s i-+ m

r

sup

M t {w (s

aG B E ~ s , ~ )

+ t, x,) + rtw(s + t, xt)] exp(-

rp, -

J,rpdp) dt

+ 7, *,I exp (- rp, - J; rpdp)

It only remains to prove that in (1.10) we have equality if A consists of a single point and if, in addition, za Iz?" In this case we do not write the superscript a since we deal with only one strategy. Let z < z y . For E > 0, as can be seen, z A z": I z:", and, further, in accord with what has been proved, W(S,X) = M.,{J;*

"

[f(s

+ t, XJ + rtw(s + t, xt)] exp

(-

rp.

-

J

r, dp dt

)

Letting E 10 and noting that z A z": t z A zzx = z IT - s, the function w(t,x) is continuous with respect to (t,x) and also, noting that the quantity

.-,

+

sup,, Iw(s p, x,)( has a finite mathematical expectation, we have proved what was required. This completes the proof of Theorem 1.11.

10. Remark. Applying Remark 3.10 to the functions iJ,,(s,x),we see that these functions are defined uniquely by the functions a(p,t, y), b(p,t,y), co(t,y),f P(t,y)


and g(T,y). The latter functions are expressible in terms of o(a,t,y), b(a,t,y), P(t,y),f"(t,y) and g(t,y), which together with Lemma 5c proves that in order to compute the function w(s,x) it is sufficient to give the functions o(a,t,y), b(a,t,y), cOL(t,y), f OL(t,y),and s(~,Y). 11. Exercise Examine the possibility of "pasting" to the strategy which "serves well" until a moment of time z, a strategy which "serves well" during an interval between z, and z,, and, furthermore, of extending this procedure, and show that the final assertion of Theorem 1.11holds in every case, not merely in the case where A consists of a single point.

12. Exercise Prove that for u

-

w and for u = g

w(s,x)= sup sup a ~ re91 $

M:{,J-s:

v..(~+ r, x,)

We conclude the discussion in this section by formulating two theorems. In the first theorem we shall estimate the rate of convergence of En to w. In the second theorem we shall give one connectivity property of a set Q., Both theorems mentioned will be proved in Section 5.3. However, here we note that the expressions appearing in the formulations of Theorems 13 and 14 which follow will be determined in the introduction to Chapter 4. Also, we note that the spaces W:d,2(HT)will be introduced in Section 5.3 (see Definition 5.3.1). 13. Theorem. Let g E y:l2(HT) n C(RT), F[g] 2 -K(1 Then in HT 1 Iw(s,x) - ~,,(s,x)l - N(K,m,T)(l 1x1)" n

+ 1 ~ 1 (H~-a.s.). )~

+

14. Theorem. Let g E W:d,2(HT)n C(HT),s E [O,T]. Furthermore, let there exist a function h(t,x) which is continuous in (s,T) x Ed and which coincides with F[g](t,x)almost everywhere in (s,T) x Ed. Further, let Q = ((t,x):t E (s,T),x

E Ed,h(t,x) > 01,

Qb = ((t,x):tE (s,T),x E Ed, w(t,x) > g(t,x)}.

Then Q c Qb and also each connected component of the region Qb contains at least one connected component of the region Q. In particular, if the set Q is connected, that is, if it consists of a single connected component, then the set Qb is connected as well.

Notes

Notes Section 1. The results valid for the general case appear here for the first time. Some of these results for particular cases can be found in Krylov [36] and Portenko and Skorokhod [61]. Sections 2,3. Some of the results in these sections can be found but without detailed proofs, in Portenko and Skorokhod 1611. The step strategies are considered in Fleming [141. Section 4. The methods for investigating the optimal stopping problem used in this section have been borrowed from [29-31, 361. Theorem 14 is the generalization of a result obtained in [56].

The Bellman Equation

4

In Chapter 3 we investigated general properties of controlled processes, such as continuity of a payoff function, the feasibility of passing to the limit from one process to another, the validity of Bellman's principle of various forms, etc. The assumptions we have made are rather weak. In this chapter we shall see that by making additional assumptions on the smoothness of initial objects, we can prove some smoothness of the payoff functions as well as the fact that the payoff functions satisfy the Bellman equation. The assumptions, definitions, and notations given in Section 3.1 are used throughout this chapter. Section 5 dealing with a passage to the limit in the Bellman equation is an exception, however, and self-contained in terms of assumptions and definitions. In addition to the main assumptions taken from Section 3.1, each section of this chapter contains assumptions which will be formulated or referred to at the beginning of each section, and which will be of use only in that section. We wish to give particular attention to one peculiarity of our assumptions. We make an assumption about a parameter m 2 0, which yields the rate of growth of functions as 1x1 + oo. The simple case when m = 0 is not excluded from our assumptions, and in this case the functions in question satisfy the boundedness assumption. For a first reading of Chapter 4, we therefore recommend the reader assume that m = 0. Furthermore, it will be easier to comprehend the material of this chapter under the assumption that ca(t,x)= 0. Let 1 a(a,t,x) = -o(a,t,x)o*(a,t,x), 2

4 The Bellman Equation

d

Fl(uij,t,x) = sup

C

acA i,j=l

aij(a,t,x)uij,

Note some properties of the quantities introduced. Since for each (t,x) the functions a(a,t,x), b(a,t,x), ca(t,x),f *(t,x)are uniformly bounded with respect to a (see (3.1.2) and (3.1.3)), the functions F, F, are finite. Since a(a,t,x), b(a,t,x), ca(t,x),f"(t,x) are continuous with respect to a E A and, since, in addition, the set A is separable, in determining F, F,, one can take the upper bound with respect to any countable set everywhere dense in A. This implies, for example, that the functions F(u0,uij,ui,u,t,x),Fl(uij,t,x), F[u](t,x) are measurable with respect to their arguments. Furthermore, if in a region Q for each a E A and a function u(t,x) (a.e. on Q), one can remove a set raof measure zero from Q for each a E A, so that the expression Lau + f a does not exceed zero on the remaining set. The union of rawith respect to a from any countable subset d of the set A has measure zero, in addition, outside this union Lau f" I0 on Q for each a E 1.If we take in the last inequality the upper bound with respect to a E d , and if we take the set 2 to be everywhere dense in A, it turns out that F[u](t,x) I 0 on Q r,. In particular, F[u] I 0 (Q-a.e.). This reasoning shows that if (1) is satisfied for each a E A, then F[u] I 0 (Q-a.e.). It is seen that the converse holds true as well. Finally, we mention that the function F, can be computed on the basis of the function F immediately according to the following simple formula:

+

UaEA

F1(uij,t,x)= lim r+m

1 F(uo,ruij,ui,u,t,x). r

-

1. Estimation of First Derivatives of Payoff Functions

1. Estimation of First Derivatives of Payoff Functions In addition to the assumptions made in Section 3.1, we assume here that for all t E [O,T], M E A, R > 0, x, y E SR Ica(t,x)- ca(t,y)(+ If"(t,x) - fa(t,y)l + Ig(x) - g(y)( + Ig(t,x) - g(t,y)( 2 K(l R)"lx - yl. (1)

+

1. Theorem. The functions v(s,x), w(s,x) have, for each s E [O,T], first-order generalized derivatives with respect to x. Furthermore, there exists a constant N = N(K,m) such that for each s E [O,T] for almost all x

Igrad,v(s,x)l

+ Igrad,w(s,x)l 5 N(l + I ~ 1 ) ~ ~ e ~ ' ~ - ~ ) ' .

PROOF.First, we prove the theorem under the assumption that for each t E [O,T], a E A the functions o, b, c, f , g are once continuously differentiable in x. Then it follows from our assumptions (see (1) and (3.1.1)) that for 1 E Ed Ila,l)(a,t,x)ll + Ib(l)(a,t?x)l5 K? Ici3t,x)J+ If :i,(t,x)l + Iga)(x)l+ I~cr)(t,x)l 2 K(1 + 1x1)". Relying upon the results obtained in Sections 2.7 and 2.8, we obtain that for any strategy a E %, s E [O,T] and z E %R(T - s) the functions va(s,x) and va9'(s,x)are continuously differentiable in x. In this case, for example,

In order to estimate v?;(s,x), we put ypS,"= 9-(d/dl)~;,~~.". We have

where the magnitude of the first term does not exceed KM:,,(l

+ Ixtl)"lytl

4 K[M:,,(l

+I~t1)~~l~~~[M:,xl~t1~l~~~,

which yields in turn, by Corollary 2.5.12 and Theorem 2.8.8 on estimation of moments xFSsX, yr'S'x,


In order to estimate the second term in (2),we apply the Cauchy inequality. The square of (2) is estimated in terms of the product of the quantity M:,,[ . P ( s

+ t,xt)I2 a K 2 ~ : , , ( 1+ I X , ~ ) ~a"N ( l + 1x1)

Zm N t

e

and the quantity

Therefore, the magnitude of the second term in (2) does not exceed N ( l + Ixl)2meNfEstimating in a similar way the expression

which resembles rather closely the left side of (2), we finally find

Next, for 1x1, lyl < R , according to the Lagrange theorem

< N ( l + R)'"Ix

- yleN(T-S).

(3)

As in Section 2.1, a function satisfying a Lipschitz condition has generalized derivatives, and in addition, the gradient of such a function does not exceed the Lipschitz constant. Hence (3) implies the existence of first-order generalized derivatives of w(s,x) and, in addition, the inequality

for almost all x

E

S,. The last inequality implies precisely that

with the same constant N . The function v(s,x)can be considered in a similar manner. We have thus proved the theorem for smooth functions o, b, c, f , g. For proving this theorem in the general case we make use of Theorem 3.1.14 and Corollary 3.1.13. We approximate the functions a, b, c, f , g(x), g(t,x) using smooth functions on,b,, c,, f,,g,(x), g,(t,x), which we have obtained from the initial functions by means of convolution with a function E ; ~ [ ( E ; ' x )


(see Theorem 3.1.14). Let

E,

= l/n. For x, y E S,, for example, let

It is seen from the above that on,b,, c,, f,,g,(x), g,(t,x) satisfy our assumptions for the same constant I?, m. Hence, if we denote by w,(s,x) the payoff function constructed on the basis of on, b,, c,, g,(t,x), then for 1x1, < R (see (3))

lyl

IX

Iw,(s,x) - w,(s, y) 1 5 N(K,m)(l + R)2meN(K,m)(T -') - YI.

Taking the limit in the last inequality as n + co,we arrive at (3) in the general case using Corollary 3.1.13 and Theorem 3.1.14. As we have seen above, inequality (3) implies the assertions of the theorem for the function w(s,x). Similar reasoning is suitable for v(s,x), thus proving the theorem.

2. Exercise Using Bellman's principle, prove that for some constant N

Iv(t,x) - v(s,x)l< ~

c

= N(K,m) for

t2s

( + I ~l l ) ~ " + ' e ~ ( ~ - " ) .

3. Remark. If ca(t,x) = 0, (2) contains no second term and, therefore

It is not in general permissible to assert that Bellman's equation F[v] = 0 holds for the function v(t,x) having only first derivatives with respect to x. In fact, the equation F[v] = 0, in addition to first derivatives with respect to x, involves second derivatives as well as a derivative with respect to t. It turns out that although the foregoing derivatives enter into the inequality F[v] < 0, we can make this inequality meaningful by integration by parts.

4. Theorem. Let R > 0, and, in addition, let q(t,x) be a nonnegative function which is injtnitely direrentiable on E d + , and is equal to zero outside [O,T] x S,. Then for u(t,x) = v(t,x) and for u(t,x) = w(t,x)for each P E A

where we assumefor the sake of simplicity that aij = aij(P,t,x),etc. Note that the assertion of the theorem makes sense since the functions wxi,vxi exist, o(P,t,x), b(P,t,x) satisfy a Lipschitz condition with respect to x, and, furthermore, they even have bounded first generalized derivatives. The


first-order generalized derivatives of the function a(B,t,x) = &r(p,t,x)o*(p,t,x) are bounded in each cylinder C,,,,. We first prove the theorem for differentiable o, b. In the lemma which follows assumption (1)will be absent.

5. Lemma. Let P E A and let the function a(P,t,x)for each t E [O,T] be twice continuously dflerentiable in x. Also, let b(P,t,x)be once continuously dlfSerentiable in x for each t E [O,T]. Furthermore, let the corresponding derivatives of these functions be bounded in each cylinder C,,,. Then for u(t,x) = v(t,x) and for u(t,x) = w(t,x)

where q is a function having the same properties as that in Theorem 4. PROOF. We introduce a constant strategy

P, = P.

For 1 2 0 let

+ M:,,U(T, ~ ~ - , ) e - ' ~ - ~ - ' ( ~ - ~ ) . If in Corollary 3.2.13 we take rb(t,x)= cb(t,x)+ 1 instead of cb(t,x)and a function f"8(t,x) = f b(t,x)+ Au(t,x) instead off a(t,x),and if, in addition, we

replace u(x)by u(T,x),we have

According to Bellman's principle (Theorems 3.1.6 and 3.1.11) w$ I u ; therefore, 0 I Aq(u - w$)and

for each 1 2 0. Further, let us take the limit in (4) as 1 -+ co. We note that due to the estimate

summability of the last quantity and, in addition, the continuity of u(s + t, xf,S,X) in t, the function h,,,(t)

= ~ ! , , e - ~ ~ u+(t,s x,)

is a continuous function oft. Therefore, if { is a random variable having an exponential distribution with exponent equal to unity, then Mh,,[(T - s) A ({/A)]-+ h,,,(O) as 1 -+ co. This implies precisely that for s E [O,T], 1 -+ co

Furthermore, it follows from the estimates of moments of solutions of stochastic equations that IhS,,(t)l I N(l + IxI)", where N does not depend on


s, x, t. Hence the left sides in (5)are bounded in C T ,uniformly with respect to I. From the estimates mentioned above and the inequality If P(t,s)lI K(l + 1x1)" it follows that for s E [O,T]

where N does not depend on s, x, I. Therefore, the totality of functions wf: is bounded on [O,T] x S, and wf: -+ u as I -+ co. Since in (4) we can take an integral over the set C,,,, we replace the function w t by u, letting I -+ co. The lemma is proved.

PROOF OF THEOREM 4. AS in proving Theorem 1, we approximate a, b, c, f , g(x) using smooth functions a,, b,, c,, f,,g,(x), and, in addition, taking the convolution of a, b, c, f , g(x) with the function nd[(nx)(see Theorem 3.1.14). We denote by v, the payoff function which has been constructed pn the basis of a,, b,, c,, f,, g,(x). According to Lemma 5,

where the operator L!* is constructed in the usual way on the basis of a,(p,t,x) = 4a(~,t,x)aX(p,t,x),b,(P,t7x),c~(t,x). Since the function v, has a generalized derivative with respect to x, we have, integrating by parts in (6), that

Let us take the limit in (7) as n + co. According to Theorem 3.1.14 and Corollary 3.1.13, the functions v,(t,x) converge to v(t,x)uniformly on [O,T] x S,. As was indicated in Section 2.1, c!(t,x) -+ cP(t,x) for all t, x due to the continuity of cS(t,x)in x and for almost all (t,x)because the generalized derivative bii exists. Furthermore, IbxilI K ; hence Ibnxil= Ibxi* nd[(nx)l2 K. This reasoning shows that the right side of (7) tends to

a s n + GO. Further, by Theorem 1, Igrad, v,(t,x)l I N(l

+1~1)~"


for t E [O,T], x E Ed, where N = N(K,T,m) does not depend on n. Then we obtain

-+ -+ a$ almost where N depends only on K, T, m, R. We know that everywhere. It can easily be seen as well that the union of the foregoing derivatives is bounded on [O,T] x S,. Hence the first term in the right side of (8) tends to zero as n -+ co. The second term tends to zero as well, since v, -+ v in 9Z(CT,R), and, in addition, because the norms ]gradxvnl in 8,(CT,,) are bounded and therefore (see Section 2.1) vnXj-+ vxj weakly in g2(CT,,). Therefore, the limit of the left side of (7) is equal to

asn -+ co. In a similar way this theorem can be proved for w(s,x), thus completing the proof of the theorem. In order to derive two corollaries from the theorem proved above, we need two simple facts. If in a region Q c [O,T] x Ed the bounded functions cp(t,x), $(t,x) have bounded generalized derivatives, cpxixj,qxi,$xj, then for any r E Cgm(Q)

JQ(W)xi$,

dx dt =

-JQ(oq)xixj$dx dt,

Inequalities (9) and (10) can be proved in the same way, namely, we need to take a function q E C$'(Q) such that it is equal to unity everywhere where q # 0. Next, we replace $xj by ($ql),j in (9) and, furthermore, we replace $ by mean functions $(" in both inequalities. Since the products $(')q,,$(')q E C,"(Q), by the definition of a generalized derivative, we can shift the derivatives from $("ql, $("q onto cp. Also, we pass to the limit as E -+ 0 using the theorem on bounded convergence. Finally, the presence of q1 has obviously no effect on values of the resulted expressions, which fact allows us to remove q, in general. 6. Corollary. Let a region Q c H,, P E A, and also let a(p,t,x) as a function of the variables (t,x) have second generalized derivatives with respect to x. Then for each nonnegative function q E C,"(Q) for u = v and for u E w

JQuLb*qdx dt

-

JQf

dx dt.


We have made use of the preceding remarks in order to remove the derivatives from the function u. Using the same remarks and shifting the derivatives on u whenever possible in the assertion of the theorem, we have

Taking arbitrary y 2 0, we arrive the following assertion.

7. Corollary. Let a region Q c H,. Furthermore, let a function v(w), as a function of (t,x), have two generalized derivatives in x and one derivative in t in the region Q. Then for any P E A almost everywhere on Q In other words, almost everywhere on Q

Lemma 5 (or Corollary 6) has a rather unusual application for proving assertions of the type of Theorems 2.3.3 and 2.3.4. We recall (see [40]) that for a fixed I > 0 a nonnegative infinitely differentiable function u(x) given on Ed is said to be I-convex if the matrix (Iu(x)dij- u,,,,(x)) is nonnegative definite for all x. According to Corollary 1 of Lemma 1 in [40],for each A-convex function /grad u(x)l a $ u(x)at each point x.

8. Theorem. Suppose that on a probability space measurable processes xi, a,, b,, c,, cp, are defined for t E [O,co),with xi E Ed. W e assume that a, is a nonnegative dejinite matrix of dimension d x d, b, is a d-dimensional vector, c,, cp, are nonnegative numbers. Assume that there exists a constant A > 0 such that c, 2 I tra, + $lbtl for all t, o.Finally, assume that for any smooth bounded function u(t,x)which decreases in t, is kconvex with respect to x and which has bounded derivatives (d/dt)u,u,,, u,,,, on [O,co) x Ed, the inequality

is satisfied, where

Then for any nonnegative Bore1 function f(t,x) M

Sowe-Ydet ai)ll(d+l ~ ( t , xdt~ 1) N(d,i)llf

PROOF.First we note that c,u 2 Iu tr a, + ,hulbtl 2 AU tr at This condition can be replaced by (12).

/id+ i,lO,rnl E d .

+ (grad u,b,).


Hence

is greater than zero since the trace of the product of positive matrices is positive; furthermore, by assumption, (a/&) u I 0. The above implies, in particular, that the right side of (11) is always definite. Next, we have from (11)that Mu(O,xo)2 - M Jom eKqtL;u(t,x,)dt,

(12)

where

We take a smooth function f (t,x) with compact support and some n > 0. Let A, be a set of all matrices a of dimension d x d such that traa* I 2n. For a E Anleta(a,t,x) = a, b(a,t,x) = 0, ca = A tr a(@), f" = (det a(a))li(d+l'f(t,x). We take as T a number such that f(t,x) = 0 for t 2 T - 2. Let g(x) = 0. On the basis of the quantities introduced, using some d-dimensional Wiener process, we define the payoff function v,,(t,x). It is seen that v,(t,x) increase as n increases. Let v(t,x) = lim,,, v,(t,x). By Theorem 2.3.3, for all x, n

Therefore, we shall prove the theorem for the function f chosen if we show that sup sup u(t,x) z M e-~.(detat)ll(d+l)f(t,xt)dt. (14) 220 X E E ~

Som

It should be mentioned that in accord with the results obtained in [54, Section 1, 21 it suffices to prove the assertion of the theorem for smooth nonnegative f(t,x) with compact support only. Thus, it remains only to prove (14). Using Lemma 5, we take as r](t,x)the function

which, being a function of (t,x), satisfies the conditions of Lemma 5 if s G [E,T - E]. Noting in addition that in our case the coefficients LB do not depend on (t,x),we can easily find using Lemma 5 for s E [E,T - E], n > 0 that d

C

i,j=l

aiiv$j,,,(s, y) - AU$)(S,~) tr a +

1

as

v$)(s,y)

+ (det a)lKd+l'f(e)(s,y) I 0

(15)

for all y E Ed and for nonnegative symmetric matrices a such that tr a I n. As can easily be seen, the functions v(t,x), v,(t,x) are equal to zero for t E [ T - E,T I . It is convenient to suppose that the functions v(t,x), v,(t,x) are

2. Estimation from Below of Second Derivatives of a Payoff Function

defined not only for t E [O,T] but for t 2 T as well, and, furthermore, that v(t,x) = v,(t,x) = 0 for t 2 T. Then f ("(s,~)= V ~ ) ( S=, ~ 0 )for s E [T - 1, co) (E< 1). Therefore, (15) holds not only for s E [E,T - E] but for s 2 T - E as well. By virtue of (13),according to the Lebesgue theorem, vf) -+ v(". Further, from (15) we have d

C

aijuEj(s,y)- ilv(&"(s,y) tra

i,j=l

+a

-

ds

v("(s,y)

+ (det a)'i'd+l''(s,~

5 0 (16)

for s 2 E and y E Ed for any nonnegative symmetric matrix a. If in (16) we take aij = nti(j, divide both sides of (16) by n, and, finally, let n -t a,we obtain n

In short, the matrix (ilv(')$j - v $ ) ~2, )0 and, furthermore, the function v(') is kconvex with respect to y. Taking a = 0, we obtain from (16) that v(') decreases in s. From (16) and (12) for u(t,x) = V'')(E t, x) we find

+

It remains to note that the left side of the last inequality does not obviously exceed the left side of (14). Also, we need to let E 10 and to obtain the right side of (14) from the right side of the inequality given above, using Fatou's lemma as well as the fact that f (')(t,x) -+ f (t,x) uniformly in (t,x). We have thus proved the theorem.

2. Estimation from Below of Second Derivatives of a Payoff Function In this section we shall estimate from below second derivatives of a payoff function. Also, using the estimates obtained, we shall represent the inequality

in a local form (see Lemma 1.5 or Corollary 1.6). Assume that the conditions given in Section 3.1 are satisfied. We also assume that the functions cr(a,t,x),b(a,t,x), ca(t,x),fOL(t,x),g(x), g(t,x) for each a E A, t E [O,T] are twice continuously differentiable over x. For all a E A, t E [O,T], x E Ed, 1 E Ed,let Ilo(l)(a,t,x)ll+ Ib(l,(a,t,x)l 5 K , Ilo(~)(~)(a,t,x)ll + Ib(,,(~)(a,tJ)l K(1 + 1 ~ 1 ) ~


-

and for

uU(t,x) g(x),

let

uU(t,x)E g(t,x)

I~:l)(t,x)l+ lut)(,)(t,x)l K(1 + 1x1)"

We prove under the foregoing assumptions that second-order generalized derivatives of the payoff functions v(s,x), w(s,x) with respect to x and firstorder derivatives of these functions with respect to s are countably additive functions of sets (see Definition 2.1.2). As was done in Section 1, we rely here upon the estimates of derivatives of functions va(s,x),v",~(s,x), which, according to the results obtained in Sections 2.7 and 2.8, are twice continuously differentiable in x. If we let 1 E Ed, a E JU, r E LMZ(T - s) and if, in addition, we write the derivative v~&;;(,)(s,x), using the differentiation rules of mathematical expectations, integrals, and composite functions, we obtain a rather cumbersome expression. In order to simplify it, we introduce the following notation:

E:;:(t)

=

1

e-(qr-qt'f"(s

+ r, xr)dr + e - ( q ~ - q ~ )+~ r,( sx,);

where xr = x:JJ, yr = ya,"," zr = za,s,x r r qr = q:,S,X. As can easily be seen, for each o and for almost all t E [0,r] 9

9

Differentiating the last expression, we find


Further, noting that tf;:(z) = g(s + z,x,), we conclude that for t E [O,z]

which constitutes an equation with respect to tf;:(t) for t E [O,z]. Since the reverse of the transformations we have carried out is true, t t : ( t ) is the unique solution of (4) on [O,z]. Further, it is not hard to see that the process tf;:(t A z ) is 8B-differentiable in x. According to the well-known rule of operations with derivatives, it follows from (4)that for almost all w for t E [O,z]

a az

- car(s+ r, xr)8B-- tf;:(r A z ) ) dr. It is convenient to regard (5) as an equation with respect to an 8 B derivative. Comparing (1)and (4)with (2) and (5),one easily sees that (?is,,(t) satisfies Eq. (5) for t E [O,T]. Since the solution of Eq. (5) is unique for the same reasons the solution of Eq. (4) is unique, then for almost all w for t E [O,zl

Differentiating (5) over 1 and, furthermore, regarding the relation thus obtained as an equation for YB-(aZ/d12)t:;:(t),we prove in a similar way that for almost all w for t E [O,z]

a2

YB-z

t:;3A 4 = t:;;(,),s,x(t).

1. Exercise Prove (6)and (7) by direct differentiation of (1).

We can deduce from (6)and (7) the estimates of

First, from our assumptions, for t I T ltf;:(t)l I K ( T - s Further,

IC?;~,(S

+ r, x,)t:;:(r)l

+ 1)?sup (1 + I X ; , ~ , ~ ~ ) ~ . ST-s

I K ~ ( T- s + 1) sup (1 + I X ~ * " ~ ~ ) ~ " . ?ST-s


Hence from (6) and (2)we obtain

where N ,

= K(T

- s + 1) + K 2 ( T - s + 1)'. Similarly,

+ N , tsT-s sup (1 + I X ; ~ " ~ ~ ) ~ "

sup

tsT-s

IZ:". ~,

(9)

where N 2 = N , + 2N,K(T - s). We also make use of the estimates of moments of derivatives of solutions of stochastic equations given in Theorem 2.8.8. In this case we obtain, for example,

where N = N(K,m). Estimating other expressions in the right sides of (8) and (9) in a similar way, and, in addition, noting that va''(~,x)= M:,,5:;:(0), v:i;(~,x)= M:,xt:fis,x(0)> v:fil)(~,x) = M:,xS$i(l),s,x(O),

we prove that the following assertion holds for vu,'(s,x).

2. Lemma. For each s E [O,T], cr E % and z E %R(T - s) the functions vu(s,x) and vu,"s,x) have second continuous derivatives with respect to x. There exists a constant N = N(K,m) such that for all 1 E Ed

Ix~)~", + Ix()~",

lo:i)(l)(s,x)l+ Iv$ii)(s7x)lINeN(T-S'(l+ NeN(T-s)(l ~ V : ~ ) ( S , X ) ~ Iv:i:(s,x)l IvU(s,x)I IvU,'(s,x)IINeN(T-s)(l+ 1x1)".

+ +

The proof of this lemma for vu(s,x)is the same as that for vu,'(s,x). However, we need to take z = T - s and, furthermore, replace g(t,x) by g(x).

3. Theorem. There exists a constant N the functions

= N(K,m) such

+l ~ 1 ~ ) ( ~ ~ ~ ~ ) + ~ , l~1~)(~~/~)+

v(s,x) + NeN(T-"(l w(s,x) + NeN(T-"(l + are convex downward with respect to x.

that for each s E [O,T]


PROOF. Let 1 # 0. Simple calculations show that [(I

+ I X ~ ~ ) ( ~ " I ~ ) I(l)(l) +

We take N from Lemma 2. Also, let N , = 23m/2N.By virtue of (10) and Lemma 2 a second directional derivative of the functions va(s,x)+ N,eN1(T-"(l + is positive for any a G 8, s E [O,T]. Therefore, these functions are convex downward. A fortiori, their upper bound with respect to a is convex downward: v(s,x)

+ N,eN1(T-"(l+ l ~ 1 ~ ) ( ~ " / ~ ' + ' .

We consider w(s,x) in a similar way, thus completing the proof of the theorem. 4. Corollary. Each of the functions v and w is representable as the digerence between two functions convex downward with respect to x. The subtrahend can be taken equal to h r e N ( T - S ) (1 + IX(2)(3m/2)+1 for a constant N

= N(K,m).

In fact, let us, for example, take N given in Theorem 5, and in addition, let us write In the lemma which follows we list general properties of differences of convex functions.

5. Lemma. Suppose that in a convex region Q c HT we are given a function u,(s,x) - u,(s,x), in which u , and u, are defined, measurable, locally bounded in Q, and, furthermore, convex downward with respect to x in Q, = {x:(s,x)E Q ) for each s. Then, for each l,, I , E Ed in the region Q there exist derivatives u~,,,(,,,(s,x)(dsdx)(see Definition 2.1.2). In this case inside Q

U(S,X) =

In addition, if the bounded function y(s,x) is measurable with respect to s, is twice continuously digerentiable with respect to x for each s, and, finally, i f this function is equal to zero outside some compact set lying in Q, then for any ll, l 2 Ed

PROOF. It is easy to obtain from the equality u = u , - u, and the properties of derivatives of u,, u, analogous properties of derivatives of u. We shall


see below that second derivatives along the I , direction of a function which is convex downward with respect to x are nonnegative. Hence inequality (11) readily follows from the equality u = u1 - u,. Therefore, it suffices to prove the lemma for the functions u which are convex downward with respect to x (u = u l , u , = 0). Further, it suffices, obviously, to prove that the assertions of the lemma hold in any bounded region Q' c Q' c Q. Note that by hypothesis the function u is bounded in any region Q'. Then, in proving the lemma it is possible to assume that the region Q is bounded, and that the function u is convex downward with respect to x and is bounded in Q. It will be convenient to assume as well that the function u is extended in some way outside Q. Let us take a unit vector 1 E Ed, s E (O,T), and also let us take a nonnegative q E C;(Q,). Furthermore, for a real r we introduce an operator AS, using the formula A:((s,x) = ((s, x

+ r l ) - 2((s,x) + 5(s,x - rl).

Integrating by parts, we easily prove that from which we have, by the mean value theorem, A:y(x) = r2q(l,(I,(x+ 0rl) where 101 < 1. In particular, for Irl I 1 the collection of functions ( l / r z )A:y(x) is bounded and for r + 0 it converges to q(,,(,,(x).Therefore,

The function q is equal to zero near the boundary Q,. Hence for a sufficiently small r the function A?q(x) is equal to zero near the boundary Q,. For sufficiently small r the last integral in (12) can be extended to Ed. Further, if we write Ju A:q dx as the sum of three integrals and if, in addition, we make in these integrals the change of variables of the type y = x + a, we easily obtain that for sufficiently small r

JQu(s,x)A:q(x) dx = J

Ed

q ( x )A:u(s,x) dx =

JQq ( x )A:u(s,x) dx.

Because u is convex, d?u(s,x) 2 0 if the distance between x and dQ, is larger than Irl. Furthermore, q 2 0 . Therefore, for small r

By virtue of (12),

Similarly, for any nonnegative function q E C g ( Q ) we have


By Lemma 2.1.3 the inequalities proved above imply the existence for each s of the measure u(,,(,,(s,x)(dx)on Q, as well as the existence of the measure u(l,(l,(s,x)(dsdx)on Q. By Fubini's theorem, for y E C,"(Q)

JQrlu(l)(l)(dsdx) JQuq(l)(l,ds dx =

In short,

This equality proved for y E C,"(Q) can be extended to all nonnegative Bore1 functions y if we apply usual arguments from measure theory. Further, by definition,

for all y E CF(Q,). Approximating the function y E CZ(Q,)which is equal to zero near aQ, uniformly in Q, along with its second derivative using the functions from C;(Q,), we can see that Eq. (13) holds for the functions q E C2(Qs)as well. Next, if the nonnegative function y is one taken from the formulation of the lemma, then

JQ

d~dx

~ ~ ( l ) ( l )

U(S7Xk(l)(I)(S,X) dx]

= JOT

dS [JQs

= JOT

ds [JQs ~(s~x)u(i)(l)(s~x)(dx)] = ~u(al)(dsdx).

J,

It remains only to recall (see Section 2.1) simple relations between u~,,,(~,, and u(,,+,,(, +,,, u(,,-,,,(,, -,,,,and also to represent a bounded function q which satisfies the conditions of the lemma as the difference between two nonnegative functions which also satisfy the conditions of the lemma. The lemma is proved.

6. Theorem. For any l,, 1, E Ed inside HT there exist generalized derivatives v ( ~ ~ ) ( ~ ~ ) dx), ( s , x~(,~,(,,,(s,x)(dsdx) )(~s (see Dejinition 2.1.2). There exists a constant N = N(K,m) such that for each 1 E Ed inside H , for u = v and for u = w ~(,,(~)(s,x)(ds dx) 2 - NeN(T-S)(l+

1 ~ 1 ) ~ "dsdx.

(14)

PROOF.From Lemma 5 and Corollary 4 follows the existence of derivatives as well as the fact that u(,,(,,(s,x)(dsdx) 2 - NeN(T-"[(l

+ l ~ 1 ~ ) ( ~ ~( ~ ~( 1 ds ~) ) dx.+

(15)


Using equality in (10)we have that

I (3m

+ 2)(3m+ 1)(1+ I x ~ ) ~ " ,

from which and from (15) we have (14), thus completing the proof of the theorem. Further, we can write the integro-differential inequalities given in Section 1, in a local form. Let

7. Theorem. In HT there exist generalized derivatives

v,i,j(ds dx),

a

w,i,j(ds dx),

Furthermore, for u = v and for u

-~

as

Ew

( ddx), s

d w(ds dx). as

-

inside HT for all p E A

- LPu(s,x)(dsdx) - f

P(s,x)ds dx 2 0.

In other words, [- LPu(s,x)(dsdx) - f P(s,x)ds dx] is a (positive) measure. PROOF. The existence of the derivatives vxi and wXiwas proved in Section 1. We have proved the existence of ~,,,~(dsdx) and ~ , , , ~ ( ddx) s in the preceding theorem. By Lemma 1.5 for j E A and for any nonnegative v] E CF(HT) JHT

[vLP*q+ f

q]ds dx d 0.

By Lemma 5, it follows from the last inequality that

where

- cP(s,x)v(s,x) ds dx

+ f P(s,x)ds dx.

By Lemma 2.1.3, the above enables us to conclude that the derivative (d/ds)v(s,x)(dsdx)exists and that it does not exceed [-vP(ds dx)] inside H,. Hence the theorem is proved for the function v. We can prove the theorem

3. Estimation from Above of Second Derivatives of a Payoff Function

for the function w in a similar way, thus completing the proof of our theorem.

8. Remark. In proving Theorem 7 we used no assumptions about nondegeneracy of a controlled process. In particular, all the assertions made in this section hold in the case where o(a,t,x) = 0.

3. Estimation from Above of Second Derivatives of a Payoff Function Inequalities of the form Lav(dsdx)

+ f a d s dx s 0

(see Theorem 2.7) enable us to estimate from above second derivatives v ( ~ ) ( ~dx). ) ( ~Such s estimation consists of the fact that we preserve the derivative v(,,(,,(dsdx) on the left side of (1)but we carry all the remaining expressions over to the right side of (1). It is necessary that the derivative v(,,(,,(dsdx) be "actually" present in some inequality of type (1) or that the derivative u(,,(~(s,x) "actually" belong to the operator F[u]. We assume that in addition to the assumptions made in this chapter, the assumptions made in Section 4.2 concerning the derivatives o, b, c, f , g(x), g(t,x) are satisfied. For t E [O,T], x E Ed, a E A, 1 # 0 let p(1) = p(t,x,Z) = inf sup na(t,x)(a(a,t,x)A,A), I:II=l ~ E A

We note that due to continuity of a(a,t,x), na with respect to a and due to separability of A we can, in determining p(t,x,l), compute the upper bound on a basis of a countable subset of A. Therefore, the upper bound mentioned is measurable with respect to (t,x). Furthermore, this upper bound is continuous with respect to A; therefore p(t,x,l) is measurable with respect to (t,x).In particular, Q(1)is a Bore1 set. Further, we introduced the function na(t,x)into the formula which gives p(t,x,l), for the sake of convenience. Since for each (t,x)the functions na(t,x), [na(t,x)]-' are bounded from above on the set A, p(t,x,l) > 0 if and only if inf sup (a(a,t,x)A,A)> 0,

in other words, if Therefore,

1:12=1 u s A

inf F1(AiAj,t,x)> 0.

I : Id= 1

( t , ~E)H T : inf Fl(AiAj,t,x) > 0 L:ld=l


Next, we shall explain in what sense Q(1)is a set on which the derivative u(,,(,, actually belongs to the operators F[u], Fl[u]. Let a point (to,xo)4 Q(1). Then, it can easily be seen that there is a vector I , such that 12, = 1 and Fl(~02$,to,xo) = 0. We can assume without loss of generality that the direction of ;lo is equivalent to that of a first coordinate vector. Then all(a,to,xo)= 0 for all a E A. The property of nonnegative definiteness of the matrices a(a,t,x) implies that al'(a,to,xo)= ail(a,to,xo)= 0 for all a E A, i = 1, ..., d. Therefore,

For computing Fl[u](to,xo)we need to know the derivatives u,,,~ only for i, j 2 2. At the same time, it is impossible to express

in terms of the derivatives u,,,, mentioned since 1' = l(l0/l2,l) # 0. For example, the functions u, u + ( X I ) ' have identical derivatives with respect to xixi (i,j 2 2); however, their derivatives witfi respect to (1)(1) are distinct. Operators F1 on the functions u, u (XI)' apparently coincide at the point (to,xo).An arbitrary variation of u(,,(,,has no effect on the value of Fl[u]. In the same sense, the derivatives uxlxl,u ~do znot ~belong ~ to the operator

+

In fact, assuming7 = (1,1),we easily prove that Lu = 2u(i,(i,.It is impossible, however, to express either U,I,I or uXzX2 in terms of u(i)ii,. The reader will understand how the estimation of the type v(,,(,,(dsdx) I I) ds dx depends on the equality p(t,x,l) = 0 from the following exercise.

1. Exercise Let d

= dl = 2,

T = 1,

Prove that p(1) = 0 for 1 X I = (1,1), p(7) > 0, vm(i)= 0, and that the function v(s,x) is a smooth function of s and ~(,,~,,(r) > 0 for 1x7, where r = [0,1] x { x : x l = x2}. Note that j, ds dx = 0.

Thus, if p(1) = 0, we need not know the value of u(,,(,,in order to compute F[u]. The derivative v(,,(,,(dsdx) is not in general absolutely continuous with respect to Lebesgue measure, that is, the generalized derivative v(,,(,,(s,x)

3. Estimation from Above of Second Derivatives of a Payoff Function

does not exist. We shall see below (see Theorem 5) that if p(t,x,l) > 0 in any region Q, the generalized derivative v(,,(,,(s,x)does exist in Q. 2. Lemma. Let u = (uij)be a matrix of dimension d x d, and let $ be a number. Assume that for all A E Ed (ul,A)2 *Ill2. Then for all (t,x)E HT and for all units 1 p(t,x,l)(ul,l) I sup na(t,x)tra(a,t,x)u aeA

+ I/-.

PROOF. We fix t, x. Furthermore, we denote by r the smallest closed convex set of matrices of dimension d x d, which contains all matrices na(t,x)a(cr,t,x) (a E A). We can obtain r, for example, of the closure as the set

which implies that the set r is bounded and, in addition, that rnax (aA,l)= sup na(t,x)(a(a,t,x)l,A), aeT

asA

rnax tr au = sup na(t,x)tr a(a,t,x)u. a sA

NET

Let us prove that p(t,x,l) = inf rnax (al,l)= rnax inf (al,l). a s T 1:11=1

rl:II=l a e T

The first equality in (3) follows from the first equality in (2). In order to prove the second equality in (3),we apply the main theorem of game theory. Let R > 0. The function (al,/Z)is given on I' x { l : M = 1,121 = R}, convex upward (linear) with respect to a, convex downward with respect to /Z because the matrices in r are nonnegative definite. Furthermore, the sets r, { A : M = 1,1A1 I R ) are convex, bounded, and closed. Therefore, for each R >0 ( )= min rnax (al,A) I:lI=l,III 0 on Q(1)

In other words, v(,,(,,(dtdx) is absolutely continuous with respect to Lebesgue measure on Q n Q(1). Further, since then inside Q -

a

dt dx

dt dx 5 - v(dt dx), at

which yields the lower estimate of (d/dt)v(dtdx) in terms of the Lebesgue measure dt dx. Next, taking the upper estimate from (5), we obtain, as above, that (a/at)v(dtdx) is absolutely continuous with respect to Lebesgue measure on Q. Therefore, if q E C,"(Q), then

and therefore, the measure density (a/dt)v(dtdx) with respect to Lebesgue measure is a generalized derivative of v with respect to t in the sense of Definition 2.1.1. The function w can be considered in the same way. 5. Theorem. Let a region Q c H,. Let a measure ((d/dt)v)-(dtdx) in a region Q be absolutely continuous with respect to a Lebesgue measure dt dx. Then, in Q there exists a derivative (a/at)v(t,x)and furthermore; a. if Q c Q(1)for some unit vector 1, a second generalized derivative v(,,(,,(t,x) exists, and also

where $, is a function satisfying the assumption of Theorem 3; b. i f for all (t,x) E Q, 1 # 0

0,

then Q c Q(l), all generalized derivatives of the type vXix,(t,x)exist in Q, and, in addition, F[v] 5 0 (Q-a.e.).

PROOF.According to the preceding corollary, (a/at)v(t,x)exists in Q and, furthermore, under conditions (a) the function of sets v(,,(,,(dtdx) in the region


Q is absolutely continuous with respect to Lebesgue measure. The RadonNikodym derivative of the function with respect to Lebesgue measure is a generalized derivative of the type v(,,(,,(t,x).Estimates for the former derivative follow immediately from (4) and the assumption that $1

dt dx I v(,,(,,(dtdx).

In order to prove (b), we note that the function

-

is continuous with respect to 1, which together with (7) implies that on Q ,u(t,x)

inf sup na(t,x)(a(a,t,x)l,l)> 0.

II/=1 a s A

It is seen that for all 2 E Ed

which yields for 111 = 1 p(t,x,l) 2 inf p(t,x)ll12 = p(t,x). 1:11= 1

Therefore, p(1) 2 p > 0 on Q. In other words, Q c nQ(1). Using assertion (a), we finally conclude that in Q there exist all generalized derivatives of the type v(,,(,,(t,x).As we have seen above, this implies the existence of all second mixed generalized derivatives in the region Q. Finally, the inequality F[v] I 0 (Q-a.e.) follows from Corollary 1.7, thus proving the theorem. 6. Remark. Obviously, Theorem 5 will hold if we replace in its formulation v and $, by w and $, $, being a function satisfying the assumption of Theorem 3.

4. Estimation of a Derivative of a Payoff Function with Respect to t We saw in Section 4.3 (see Theorem 3.5) that in order to prove the existence of second-order generalized derivatives of a payoff function with respect to space variables and to estimate them, we had to know how to estimate the derivatives of a payoff function with respect to t. In this section we estimate the absolute values of (a/at)v(t,x), (a/at)w(t,x), making more assumptions than in Sections 2 and 3. In addition to the main assumptions made in Chapter 4, we assume here that the functions a(a,t,x), b(a,t,x), ca(t,x),f '(t,x) for each a E A are once continuously differentiable with respect to (t,x) on the function g(x) is twice continuously differentiable with respect to x,

a,,

4. Estimation of a Derivative of a Payoff Function with Respect to t

and the function g(t,x) is once differentiable with respect to t and twice differentiable with respect to x, with the derivatives (d/dt)g(t,x),g,i(t,x), gXix,(t,x) being continuous in H T . Furthermore, for ail a E A, t E [O,T], 1 E Ed let

where the constants K and m are the same as those in (3.1.1)-(3.1.3). The last inequality follows readily from the foregoing and from (3.1.2) and (3.1.3) if in the right side of (1) we replace K ( l + by N(K,d) (1 + v2". It is seen also that (1 + 1x1)" I(1 + v2". Hence, if there exist constants K and m 2 0 for which all the assumptions except (1) can be satisfied, there will exist (other) constants K and m for which all the assumptions including (1) can be satisfied. Therefore, we can easily do without (1).We shall not omit ( I ) , however, because the estimate (1) is convenient in the case, for example, where m = 0 and, in addition, a(a,t,x) and b(a,t,x) are bounded functions. Furthermore, the right side of (1) written in a special form will be convenient for our computations. It is clear that we can always extend the functions a, b, c, f , and g(t,x) for t > T so that our assumptions are satisfied for all t E [O,co). However, in this case we need, perhaps, to replace the constant K by 2K for t > T. Let us assume that we have carried out the extension as described above. Due to the results obtained in Section 2.8 the process xf3"." is 9 B continuously 9B-differentiable over s for s E (0,T) on an interval [O,T]. Furthermore, if q:,"." = 9 B - ( d / d ~ ) x : " ~then , for each n 2 1 (see Theorem 2.8.7)

IXI)("+~)

1~1)~" IXI)("+~)

where N = N(K,m,n). It is seen that the process (s + t,xf9"'")is 9B-continuously 2B-differentiable as well. It follows from Section 2.7 that the function vu,'(s,x)is continuously differentiable over s on (0,T )for each a E a,x E Ed,z E YJl(T). Our objective consists in finding the formula for ( d / d s ) ~ ~ , ' "((~~, x- )~. ) 1. Lemma. Suppose that on a square (0,T) x (0,T) we are given a bounded function $(s,r) which is measurable with respect to (s,r). Also, suppose that for almost all r the function $(s,r) is absolutely continuous with respect to s


and that the function d

ds +(s,r) =

[

*

t-s

-

"'"

t -S

if the limit exists otherwise

satisjies the inequality

Then the function l;f-"(s,r)dr is absolutely continuous with respect to s and, in addition, a derivative of this function with respect to s coincides with

for almost all s.

The proof of the lemma follows from the fact that for those r for which $(s,r) is absolutely continuous with respect to s, the function (a/ds)$(s,r)is a derivative of $(s,r), is measurable with respect to (s,r), and, finally, as we can easily verify using Fubini's theorem,

We can better describe the method of using this lemma if from the definition of va,'(s,x),applying Ito's formula, we derive the following formula:

If we assume that the function g(t,y) is infinitely differentiable with its derivatives increasing not too rapidly for (yl + co,the functions Larg(s r, y) are continuously differentiable with respect to (s,y).Further, since the process (s + r, x;,".") is 2'B-continuously -%'B-differentiablewith respect to s, due to the results obtained in Section 2.7 the random variable

+

is 2'-continuously -%'-differentiablewith respect to s on (0,T) for each r E [O,T]. In this situation the mathematical expectation Mf,x~lbr[f

+ r,xr) + Larg(s+ r , ~ r ) ] e - ' ~

(3)

is continuously differentiable with respect to s on (0,T) for any a, x, r. If we compute the derivative of (3) with respect to s and, furthermore, if we make use of the familiar estimates of moments of x;,"." as well as inequality (2), we can easily prove (compare with the proof of Theotem 1.1) that the derivative of (3) with respect to s is bounded for s E (O,T), r E (0,T) for each x. Therefore, in this case the function va~"'(T-s)(s,x) is absolutely continuous with respect to s according to Lemma 1. Also, ((0,T)-a.e.)

4. Estimation of a Derivative of a Payoff Function with Respect to 6

Let us transform the last expression. Using Ito's formula as well as the rules given in Section 2.7, which enable us to interchange the order of derivatives and integrals, we conclude that

Thus, if g(t,y)is a sufficiently smooth function,

Immediate computation of the last derivative with respect to s and, next, the application of Fubini's theorem (or carrying out transformations identical to those given in Section 2) lead us to the following result.

2. Lemma. For each x E Ed, a E a, z E 'iIX(T)the function V ~ , " ' ( ~ - " ( S , X ) is absolutely continuous with respect to s and, in addition, almost everywhere on (0,T)its derivative coincides with 8'9 ' T - S ) (s,x)where

- M : , X ~ r , T - S [fJXT-'(T,xTTS) + LaT-sg(T,~~-S)]e-YT-s,

0 and, in addition, a continuous function x, defined on [to - 6, to] such that x,, = x,, (t,x,) E Q for t E [to - &to). It is easily seen that if the process (so t, x,) is continuous with respect to t and is inside Q at t = 0, this process (so + t,x,) can leave the region Q only across the parabolic boundary of Q. Two basic theorems of the present section are the following.

+

1. Theorem. Let the functions u, E W1*2(Q) (n = 0,1,2, . . .); also, n t0

IIunllB(8'Q)

lim

0 for all (qt,x) E A x Q, A E Edlet r"(t,x) 2 6.

(Z(a,t,x)A,A)2 61AI2, Let un E W1s2(Q)(n = 0,1,2, . . .); also,

Then for any function h E LZd+,(Q):

where N depends only on d, 6, and the maximal values of thefunctions @(a,t,x), zij(a,t,x),r"a(t,x)with respect to i, j = 1, . . . ,d, and (a,t,x) E A x Q. In particular, if G[un] + - h in the norm of LZd+,(Q), then G[uo] = - h (a.e.). It is essential to note that the hypotheses of the theorem include none on the convergence of derivatives of the functions un to derivatives of the function uo. In this connection, let us bring the reader's attention to 3. Exercise Let d = T = 1, Q = (0,l) x (-1,1), G[u] = Isup d 0 and each (t,x)E Q there exists a number i such that G[u](t,x)- E I p ( i ) u ( t , ~+)f"a(i)(t,~). We denote by i,(t,x) the smallest integer which satisfies the last inequality. It can easily be proved that the set {(t,x)E Q: i,(t,x) = i ) is measurable for each i. Hence the function i,(t,x) is measurable and, by this token, the function a",(t,x)= a(i,(t,x))is measurable as well, for which for (t,x)E Q. The foregoing (compare with (4)) implies that I E(t,x) + R ~ E ( ~I, xU"(t,x) ) + E ( T- t).

Here E is an arbitrary positive number; therefore, u I 5. This completes the proof of the lemma. For 1= 0 we derive from the lemma a probabilistic representation of a solutiop of the equation G[u] = - h:

6. Exercise Prove using (5) that if P(t,x) = 1, ul,u2 E W1,'(Q),

G[u,] 2 G[u2]

(a.e. on Q),

ulla.Q < ~ ~then u,l = u2 ~ everywhere ~ , in Q. In particular, if F[u,] = F[u,](a.e. on Q), ~ ~ = u21azQ, l then ~ u,, = u2 ~ in the region Q.

We note another simple consequence of Eq. (9,which is, however, rather irrelevant to the discussion in this section. 7. Theorem. Let (thefirst assumption of Theorem 2 be satisfied)

Q = CT,R, h = ess sup

CT,R

IG[u,]

-a

r(

1, - G[u2] )

1,

~

W1"(CT,R), ul(T,x)= u2(T,x).

1 ~2, E

Then for each n > 0 there is a constant N = N(K,n) such that for all (s,x)E CT,R


PROOF.Let hi = -G[ui]. Further, we write representations (5) for u1 and u, and, second, we subtract the representations obtained. Noting that the magnitude of the difference between the upper bounds does not exceed the upper bound of magnitudes of the differences, we have

Since Ih, - h,l 5 h (a.e.),Rglh, - h,l(s,x) I R$h(s,x) 5 h(T - s). Furthermore, since ul(T,x) = u2(T,x),

n; I L ,

- u,~(s,x)I M ~ U ,- uzl(s

+ 2,

I sup sup lu,(t,y) - u , ( t , ~ ) l ~ { l x : , = ~ ,R ~ I) . te[O,Tl I y l = R

It remains only to estimate the last probability. It is seen that it equals

According to Corollary 2.5.12 the last expression does not, in turn, exceed R-"NeN(*-"(1 + 1x1)" This completes the proof of the theorem. Using the probabilistic representation given in Lemma 5 in order to solve the equation G[u] = - h, we can give a probabilistic formula for the operatbr G.

8. Lemma. Let r"(t,x) = 1 and also let

+ n?u(t,x)]

F ~ , u ( ~ ,=x )S U ~ [ R ; ( A U + $ + h)(t,x) as%

Then for, all

where N depends only on d, 6, and the maximum of the moduli iZij(a,t,x),@(a,t,x). Furthermore, for the same h, u lim ]IA(Fh,u - U ) - G[u] - hlld+l,Q= 0.

2.- m

PROOF.By Lemma 5, u = F h l u , where h, ence between the upper bounds, we find

= - G[u].Estimating

F ~ ,u u = Fh,u - F ~ IuSUP R?(h - h l ) 5 sup R;(h oisl

Fh,u - u 2 inf R;(h - h,) as@

=

the differ-

- h,)+,

oi€l

-sup R;(hl - h) 2 -sup R:(h - hl)-, be%

csir

which together with Theorems 2.4.5 and 2.4.7 prove the assertions of the

5. Passage to the Limit in the Bellman Equation

theorem. In fact, by Theorem 2.4.5

s Allsup R:(h as%

d]l(Fiu - u ) + l l p + l , Q

- hl)*llp+l,Q5 Nll(h - hl)*llp+l,Q.

By Theorem 2.4.7, the expressions

-A

d sup R"',h - h,), .EL

sup R:(hl

-

as%

h)

converge in the sense of the upper norm $cb+,(Q)to h-h,. Since d ( ~ h , u u) is between the foregoing expressions, the former converges to h - h, as well. The lemma is proved.

-

9. Proof of Theorem 2. First we consider the case Fa(t,x) 1, h = 0. Let a region Q' c Q' c Q. According to Lemma 8

Further, it is seen that IF,0u - F;u,l I d sup R:lu - unl BE%

where No = sup, Ilu, -

+ No sup II: ae.

1,

~ ~ l l ~ By~ ~Theorem , ~ , . 2.4.5

Since the constant N does not depend on n, the right side of the last inequality tends to zero as n -+ oo. Therefore,

4 lim lim d]l(F,Oun - u,)A+m n + m

+ No lim A +

lid+

l,QT

/Id+l,I"

sup II! 1 E

where the first term does not exceed N lim.,, Il(G[u,])- lid+ ,, ,, in accord with Lemma 8, and the second term is equal to zero in accord with Theorem 2.4.7. Finally,

for each region Q' c Q' c Q, N depending only on d, 6 as well as the maximal magnitudes of Zij(cr,t,x),&a,t,x) with respect to a, t , x, i, j. Next, we choose an increasing sequence of regions Qf whose union is Q. Putting instead of Q' the region Q: in the left side of (6) and, in addition, letting i -+ oo, we complete proving assertion (b) of the theorem for h = 0. In this case assertion (a) of the theorem can be proved in a similar manner.


Using formal transformations, we can derive the general assertion from the particular case considered. Let h = 0, and also let Fa(t,x)be an arbitrary function satisfying the conditions of the theorem. We construct the operator G[u] on a basis of the functions (r^)-'iT,((9-'%, (f")-'c", ((r"-'7, 1 in the same way as we constructed the operator G[u] on a basis of the functions 4 6, z, F. Let N,

= sup

sup r"(t,x).

~ E (At , x ) c Q

Note that for any set of numbers I": 1. if 0 I sup,,, la, then SUP

asA

2. if sup,,,

1" 5 N, sup (~'(t,x))-'I" I N'6-l sup la; a sA

aeA

Id I 0, then sup 1" 2 N, sup Fu(t,x))'la2 N,6-' sup 1'. a€A

a, A

asA

The foregoing implies that

These inequalities together with the assertions of the theorem which hold for the operator G and h = 0 immediately prove the theorem for G and h = 0. In order to prove the theorem for an arbitrary h E Zd+,(Q), it suffices to note that G[u] h can be written as c[u] in an obvious way, if we construct c[u] on a basis of the functions i7, 6, 7 + h, and 7 in the same way as we construct G[u] on a basis of ii, 6, F, and F. This completes the proof of the theorem.

+

I?,

6. The Approximation of Degenerate Controlled Processes by Nondegenerate Ones Let (fi,g) be a (dl + d)-dimensional Wiener process, let E be a number, and, finally, let o,(a,t,x) be a matrix of dimension d x (dl + d), in which the first dl columns coincide with the respective columns of the matrix o(a,t,x) and, also, the block of last d columns gives &I,where I denotes a unit matrix of dimension d x d. Denote by % ' the set of all processes a" = a",(o)which are progressively measurable with respect to and which take on values s E [O,T], x E Ed we define the process X?,"(E) to be a from A. For a" E solution of the equation

a,

(6)

6. The Approximation of Degenerate Controlled Processes by Nondegenerate Ones

Furthermore, let q:,"."(&) =

Ji cZr(s+ r, x~,~.'(&)) dr,

v,(s,x) = sup v%(s,x). as%

For s E [O,T] we denote by @(T - s) the set of all Markov times (with respect to ,))f which do not exceed T - s,

(g})

w,(s,x) = sup

sup

a s % Ss(JR(T-S)

v$~(s,x).

The processes xFhX(&) for E # 0 are nondegenerate in the following sense. Let a,(a,t,x) = ia,(qt,x)o:(cr,t,x). It is seen that a,(cr,t,x) = a(a,t,x) + +&'I2 +&'I.

(2)

Hence for any 1 E Ed

The equality a, = a relations:

F,[u]

+ $&'I

immediately implies the following useful

= sup [LEu(t,x) + f"(t,x)]

= F[u]

UEA

+ -8'21

Au.

In the case where & -,c for all t, the set of strategies % c a. If, in addition, the first dl coordinates of the process fir form a process w,, due to the uniqueness of a solution of Eq. (1) we have xf*","= x?"."(O) for cr E %. Hence we can say that the nondegenerate controlled process xf3"."(&) as E + 0 approximates the (degenerate, in general) process x~~"'".

6

1. Theorem. As E + 0 v,(t,x)

+

v(t,x),

uniformly on each cylinder C,,,.


PROOF.According to Corollary 3.1.13 as E -, 0 uniformly on each cylinder C,,,. It is seen that for E = 0 the process xFS,"(~) can be defined to be a solution of the equation x, = x

+ J,a(ar,s + r, x.) diT: + Ji b(ar,s + r, x.) dr,

where G; is the vector composed of the first dl components of the vector fit.The last equation is equivalent to the equation for x;,",", in which, however, the Wiener process is (possibly) a different one and, in addition, it is allowed to choose strategies to be measurable with respect to rather large a-algebras. However, as we know from Remarks 3.3.10 and 3.4.10, a payoff function does not depend on a probability space and the fact that one dl-dimensional Wiener process is replaced by another dl-dimensional Wiener (with respect w, which to, possibly, very large a-algebras) process. Therefore, vo = v, w, together with (4) proves the theorem.

-

In some cases, for example, in finding numerical values of payoff functions, it is crucial to know how great the difference Iv,(s,x) - v(s,x)l is. 2. Theorem. For all s E [O,T], a E A, R > 0, x, y E S,, let

EE

Then there exists a constant N [- l,l]

= N(K,m)

such that for all (s,x) E HT,

Iv,(s,x) - v(s,x)l + IwXs,x) - w(s,x)l I lElN(1 + 1x1)2m eN(T-s) .

PROOF.We can prove the theorem by differentiating Eq. (1) over the parameter E. We prefer, however, a formal application of Theorem 1.1. We add the equation E, = E OdG, + Odr to Eq. (I), replacing in (1) E by E,, and furthermore, we regard E, as the last component of the controlled process (Xa,~.(~,e) &) t ,Eta , s , ( ~ , ). Note that for s E [O,T], x, y E Ed, el, c2 E El, a E A,

+ So

So

In other words, the function o,(a,s,x) satisfies a Lipschitz condition with respect to (x,E)uniformly in a, s. ~ "the ~ scheme ~ ~ , E con~ ~ ~ ~ " ~ Therefore, the controlled process ( X ~ ~ ~ ~ fits sidered in Chapter 4. Theorem 1.1 estimates the gradient of the functions v,(s,x), w,(s,x) with respect to the variables (x,~).In particular, the generalized lx12 I R2 do not exceed derivatives v,, W, with respect to E for

+

7. The Bellman Equation

N ( l + R)2meN(T-S) . A s was mentioned in Section 2.1, the boundedness of a generalized derivative yields a Lipschitz constant. Hence for c2 1x1' I R2

+

Iv,(s,x) - vo(s,x)l + Iw,(s,x) - wo(s,x)l I lclN(1 + R ) e

2m N ( T - s )

where N = N(K,m). It remains only to take R2 = x2 theorem is proved.

>

+ 1 for 181 5 1. The

7. The Bellman Equation The Bellman equation plays an essential role in finding a payoff function and &-optimalstrategies. It turns out that if the processes x;,","are nondegenerate, we can obtain the Bellman equation under the assumption of the existence of generalized derivatives of a payoff function. First, we prove two results of the type mentioned, and second, we derive the Bellman equation imposing restrictions only on o, b, c, f , g. These restrictions on o, b, c, f , g will be formulated after Theorem 2. Here as well as everywhere else in this chapter, we assume that the assumptions made in Section 3.1 are satisfied.

1. Theorem. Let a bounded region Q c H,, let w E W',2(Q),and, jinally, for each region Q' which together with its closure lies in Q let there exist a number 6 = 6(Q1)> 0 such that for all (t,x) E Q', a E A, A E Ed (a(a,t,x)i,A)2 @I2. Then F[w] I 0 (a.e. on Q), F[w] = 0 (Q n ((t,x):w(t,x)> g(t,x))-a.e.), w 2 g in the region Q. In short, (F[w] + w - g),

+g -w =0

(a.e. on Q).

PROOF.For P E A we introduce a constant strategy Pi r b. Let the region Q' c Q' c Q, let a point (s,x)E Q', and, finally, let Z' be the time of first exit of the process (s + t,xf,",") from the region Q'. By Theorem 3.1.11, for each i 2 0

By Ito's formula (Theorem .2.10.1)


Therefore, subtracting these two formulas, we obtain

o 2 M,!

Ji' [LPW(S+ t, x,) + f l(s + t, ~ , ) ] e - ~ ' - dt. "

Multiplying the last inequality by A and, in addition, letting 1+ co,we find according to Theorem 2.4.6 that LBw+ f P I 0 (a.e. on Q'). Then, F[w] I 0 (a.e. on Q). On the other hand, let E > 0 and let the region Q' c Q n {(s,x):w(s,x)> g(s,x)

+ E).

Then, according to the Bellman principle (according to Theorem 3.1.11)

where 2' is the time of first exit of the process (s + t, x:,".") from the region Q'. By Ito's formula,

which implies that

0 = sup M,: are%

Isup M,: OLE%

J:

[LU.W(S

+ t, x,) + fc(s + t, x,)]e-"dt

J:

F[W](S

+ t, x,)e-"

dt,

(1)

where F[w] I 0 (a.e. on Q). Hence the right side of (1) is equal to zero. By virtue of Corollary 2.4.8, we have F [w] = 0 (a.e. on Q'). In view of arbitrariness of Q' it means that F[w] = 0 (Q n {(s,x):w(s,x)> g(s,x) + 6)-a.e.) for each E > 0. The union of all these regions for all E > 0 constitutes a region Q n {(s,x):w(s,x) > g(s,x)). Therefore, in the last region F[w] = 0 almost everywhere. Finally, the inequality w 2 g is obvious (see, however, Theorem 3.1.8). We leave it as an exercise for the reader to prove the last assertion. The theorem is proved.

2. Theorem. Let a bounded region Q c H T , let v E W1,2(Q),and, finally, for each region Q' which together with its closure lies in Q let there exist a number 6 = 6(Q') > 0 such that for all (s,x) E Q', a E A, 1 E Ed (a(a,t,x)A,A)2 61A12. Then F[v] = 0 (a.e. on Q). The proof of this theorem follows exactly the proof of Theorem 1. We need only, instead of Theorem 3.1.11, to use Theorem 3.1.6 where it is necessary. We formulate the conditions which, in addition to the assumptions made in Section 3.1, we assume to be satisfied in the remaining part of this section.


+ +

Let us introduce a vector ya(t,x) of dimension d x (dl d 4), whose coordinates are given by the following variables: aij(a,t,x)(i= 1, . . . ,d,j = 1,. . . ,dl), b'(a,t,x) (i = 1,. . . ,d), ca(t,x),fa(t,x), g(x), g(t,x). For all a E A, 1 E Ed let the derivatives @,(t,x),y&(l,(t,x),(d/at)ya(t,x)exist and be continuous with respect to (t,x) on RT.Assume that the derivatives mentioned (they are vectors) do not exceed K(l + 1x1)" in norm for all a E A, 1 E Ed, (t,x) E RT. Also, it is convenient to assume that for all a E A, x E Ed We note that the relationship between this assumption and the preceding one was discussed in Section 4. We shall prove under the combined assumptions indicated that the functions v and w satisfy the corresponding Bellman equations in the region Q* = ((t,x) E H T :SUPaEA (a(a,t,x)A,A)> 0 for all A # 0). First we show that the set Q* is in fact a region. Let p = p(t,x) = inf sup na(t,x)(a(u,t,x)A,A). JIZI=l aEA

3. Lemma. The function p(t,x) is continuous in [O,T] x Ed, the equality is satisjied, the set Q* is open, and the function p-'(t,x) is locally bounded in Q*.

PROOF.The third and fourth assertions follow from the first and second assertions and also from well-known properties of continuous functions. Further, the derivatives (with respect to (t,x)) of the functions a(u,t,x), b(a,t,x), ca(t,x),and f"(t,x) are bounded on any set of the form A x [O,T] x (x: 1x1 I R). Therefore, these functions are continuous with respect to (t,x) uniformly with respect to a. By similar reasoning the function (a(a,t,x)A,A)is continuous with respect to (t,x) uniformly with respect to a E A, A E S , . Then we have that the function na(t,x)(a(a,t,x)A,A)is continuous with respect to (t,x) uniformly with respect to a E A, A E S,. Furthermore, we note that the modulus (or absolute value) of the difference between the lower (upper) bounds does not exceed the upper bound of the moduli of the differences. Therefore, if (tn,x,) -+ (to,xo),then

by the definition of uniform continuity.


In order to prove the second assertion, we make use of the fact that due to the inequality na(t,x)I 1 for = 1 we have ~up,,~(a(a,t,x)A,A) 2 p(t,x). Hence, if (t,x)E HT and p(t,x) > 0, then (t,x)E Q*. If p(t,x) = 0, there will be a sequence An E as1for which sup na(t,x)(a(a,t,x)An,An)-+ 0. USA

Therefore, (a(a,t,x)A,,A,) -+ 0 for all a E A. We can consider without loss of generality that the sequence {A,) converges to a limit. Ee denote this limit by A,. Then (a(a,t,x)Ao,Ao)= 0 for all a E A. Therefore, (t,x)f Q*, which completes the proof of the second assertion, thus proving the lemma. 4. Theorem. In HT (in the region Q*) the functions v(t,x), w(t,x) have all generalized jirst (respectively, second) derivatives with respect to x and a generalized first derivative with respect to t. The foregoing derivatives are locally bounded in HT (respectively in Q*). There exists a constant N = N(K,m) such that for u v and for u -= w for any 1 E Ed

-

+IxJ)~"~~(~-')

-~ ( 1

5

1

U ( ~ ) ( II) - N(1

P

+ I x J ) ~ ~ ~ ~ ( ~ - (a.e. ~ ) on Q*). (3)

This theorem is in some way a summary of the results obtained in Sections 1-4. The existence of (d/dt)u,u ( ~follows ) immediately from Theorem 1.1 and Theorem 4.3, from which, in addition, we have estimates of the foregoing derivatives. The existence of (d/dt)u implies that the measures (d/dt)u(dtdx) and ((d/dt)u)-(dtdx) are absolutely continuous with respect to the Lebesgue measure, and, furthermore, their Radon-Nikodym derivatives are equal to (a/dt)u and ((a/dt)u)-,respectively. By Theorem 3.5b and Remark 3.6, all generalized second derivatives of the functions v(t,x), w(t,x) with respect to x exist in Q*. Further, as was shown in proving Theorem 3.5, the function p(1) appearing in assertion (a) of that theorem is greater than p. Therefore, by Theorem 3.5 and Remark 3.6,

I

+ Igrad,u( + lul + 1 1~1)~"

(a.e.on Q*),

where $ = - ~ e ~ ( ~ - +' ) ( l is the right side of inequality (2.14). To complete the proof of inequality (3), it remains only to use inequality (2) and also, to recall (see Section 3.1) that lul I N ( l I ~ l ) ~ e ~ (The ~ - 'theorem ). is proved.

+


5. Theorem. F[v] = 0 (a.e. on Q*), F[w] I 0 (a.e. on Q*), F[w] = 0 (Q* n {(s,x):w(s,x) > g(s,x))-a.e.), w(s,x) 2 g(s,x), in the region Q*. The assertion concerning w can be written in short as follows: (F[w]

+ w - g), + g - w = 0

(a.e. on Q*).

PROOF.According to Corollary 1.7, F[v] 10, F[w] I 0 (a.e. on Q*). We prove that F[w] = 0 almost everywhere in any bounded region Q' which together with its closure lies in Q* n {(t,x):w(t,x) > g(t,x)}, which fact is obviously sufficient for proving the assertions of the theorem concerning w. Let us make use of the approximation of degenerate processes by means of the nondegenerate ones, which was described in Section 6. We take the matrix o,(a,s,x), the process X'~~'~(E), and the function w,(s,x) from Section 6. As was indicated in Section 6, the matrix a,(a,t,x) E io,(a,t,x)a:(a,t,x) is equal to a(a,t,x) + *e2Z and it also satisfies inequality (6.2):

Hence for E # 0 the set Q* associated with the matrix a, coincides with HT, which implies, according to Theorem 4, the existence of generalized first and second derivatives of w, with respect to x, a generalized first derivative with respect to t. Furthermore, it implies that the foregoing derivatives are locally bounded in HT. By Theorem 1, due to (4) for E # 0 the function w, satisfies the equation (see (6.3)) F[w,] + $c2Aw, = 0 almost everywhere in the region {(t,x)E HT:w,(~,x)> g(t,x)}. For all sufficiently small E these regions contain Q'. In fact, since Q' c {(s,~) E HT:W(S,X) > g(s,x)), the continuous function w(s,x) - g(s,x) > 0 on Q'. Since the set Q' is a compact, there exists a number 6 > 0 such that w(s,x) - g(s,x) 2 6 for (s,x) E Q'. By Theorem 6.1, the functions w,(s,x) g(s,x) -* w(s,x) - g(s,x) as E + 0 uniformly on Q'. Therefore, for all sufficiently small E on Q' (even on Q') the inequality w,(s,x) - g(s,x) 2 612 is satisfied. From the above we conclude that for all sufficiently small E

F[w,] 2 -i~~ A W,

(a.e. on Q').

(5)

Using Theorem 5.lb we take the limit in (5). However, before doing this, we need to estimate Aw,, (a/at)w,. Note that for /el I 1 the matrix o,(a,t,x) satisfies the same conditions as those the matrix o(a,t,x) satisfies, having, however, a different constant K. Indeed, the matrix norms of their derivatives with respect to t and x obviously coincide. Furthermore,

+

Ilo,(a,t,x)l12 = Ilo(a,t,x)l12 c2 I (K2

+ 1)(1+ 1~1)~.


Hence, applying Theorem 4 to the function w, for lei 5 1, E # 0, we find the constant N which depends only on K and m, for which

for all 1 E Ed, where p,(t,x) = inf sup %(t,x)(a,(a,t,x)A,A), I1I=1 a s A

nz(t,x) = (1

+ tra,(a,t,x) + Ib(a,t,x)l + cU(t,x)+ I fOL(t,x)I)-l.

Further, it is seen that (a,(a,t,x)A,A) 2 (a(a,t,x)A,A).Since e2 tr a,(a,t,x) = tr a(a,t,x) + - d I tr a(a,t,x) 2 n;(t,x) that

+ c2d,

r (111 + e2d))na(t,x).Hence p, 2 (1M1 + ~ ' d ) ) From ~ . (6) we conclude

Iw, ~ , ( , ~

S [1

+ i ( 1 + e2d)]N(1 + l~l)~'e"-')

(a.e. on Q*).

By virtue of Lemma 3 the last expression is bounded on Q' by a certain constant. Thus, there exists a constant N such that for any E E [- l,l], E f 0, the inequality IAw,l 5 N can be satisfied almost everywhere on Q'. Theorem 4 also implies the uniform boundedness of I(d/dt)w,l for e E [- 1,1]. Next, we take a sequence w,,. The above arguments and (5) yield F[w,] 2 inf F[wll,] 2 - N (a.e.on Q'), nz 1 1-1 (a.e. on Q'). in^ F[wlIn] r -- lim Awl,, 2 0 n+ w 2 n+m n The former inequality allows us to assert that the function inf,, 9d+ ,(Qf)(it is bounded on Q'). The latter inequality together with Theorem 5.lb yields F[w] 2

lim F[w,,,]

n+ w

20

,F[wIl,]

E

( a e o n Q').

Recalling that F[w] 0 (a.e. on Q*), we obtain F[w] = 0 (a.e. on Q'). We have proved the theorem for the function w. It remains only to prove that F[v] = 0 (a.e.on Q*). Let us consider the functions v,(s,x) introduced in Section 6. By inequality (4) and Theorem 4, for e # 0 generalized derivatives (a/dt)v,(t,x), vaxi(t,x)~,,~,,(t,x)exist and are locally bounded in HT. By Theorem 2 for E # 0 F[v,]

+ ie2Au, = 0

(a.e. in HT).

We fix a certain bounded region Q' c Q' c Q*. In the same way as we did before, we estimate on Q' the derivatives v,(,,(,,,(dldt)~,,using Theorem 4.


Then, using Theorem 5.lb, we can conclude that F[v] 2 l a F[vl,,] n-t m

=

1

!& (-p dv,,.)

n-t m

t0

(a.e. on Q1).

On the other hand, since F[v] I 0 (a.e. on Q*), F[v] = 0 (a.e. on Q'). Due to arbitrariness of Q', F[v] = 0 (a.e. on Q*). The theorem is proved. 6. Remark. Inequality (6) together with the estimate ,usgiven in the preceding proof, shows that for all E E [- 1,1], E # 0, 1 E Ed, ,u -Iw8(,,(,,I 1+,u

I (1

+ c2d)N(1+ I ~ 1 ) ~ ~ e ~ (a.e. ( ~ on - ~Q*), )

where N = N(K,m). From (6) it follows in general that inequality (7) holds almost everywhere in H T . However, the function ,u = 0 outside Q*. By Theorem 4, inequality (7) holds as well for E = 0. Absolutely similarly, for all E E [- l,l],1 E Ed ,u l+,u

-IvEcn(l)J I (1

+ ~ * d ) N (+l 1x1)3meN ( T - t )

(a.e. on Q*),

where N = N(K,m). In the case when for all (t,x)E RT and 1 # 0

the set Q* coincides with H T , and, in addition, the continuous function ,u(s,x) > 0 at each point [O,T] x Ed. Hence, the function ,u-' is bounded on each cylinder C,,,. Also, the above arguments show that the derivatives w,(,)(,, and v,(,,(,, are bounded (a.e.) in each cylinder C,,, by a constant not depending on E. The same remark applies to the mixed derivatives w,(,,,(,~,and vE(11)(12), which, as we know, can readily be expressed in terms of we(,,+,,,(,, +,,,, w&(ll-12)(11-12,, V&~11+12)(11 + l 2 b and V & ( l l-l2)(11-12). The next theorem follows immediately from Theorems 4 and 5, the results obtained in Section 3.1 on the continuity of v and w, as well as on the estimates of IvI and Iwl, and, finally, from the remarks made above about the properties of ,u if condition (8) is satisfied. Recall that the assumptions made in Section 3.1 and the assumptions about the smoothness of a, b, c, f , g(x), and g(t,x) which were formulated before Lemma 3 are assumed to be satisfied. 7. Theorem. For all (t,x)E HT and 1 #.O let inequality (8) be satisjied (i.e., FI(l'1',t,x) > 0). Then the functions v(t,x) and w(t,x) are continuous in H T , have in HT all generalizedjrst and second derivatives with respect to x and a generalizedjrst derivative with respect to t. These derivatives are bounded in each cylinder C,,,. There exists a constant N = N(K,m) such that for all (t,x) E RT Iv(t,x)l 5 N ( l I ~ l ) " e ~ ( ~ - " , Iw(t,x)l r N(1 + I ~ l ) " e ~ ( ~ - ~ ) .

+


Finally, a. F[v] = 0 (a.e. in HT), v(T,x) = g(x); b. F[w] I 0 (a.e. in HT),w(t,x) 2 g(t,x)for (t,x) E RT,F[w] = 0 almost everywhere on the set {(t,x)E H T ; w(t,x) > g(t,x)), w(T,x) = g(T,x).

It follows from this theorem that, in particular, the derivatives (a/at)v, (a/at)w,v,,, wxl, vXiij,and wxiX,are summable with respect to any cylinder C,,, to any power. Using theorems on embedding (see [47, Chapter 11, Lemma 3.31) we deduce from the foregoing the following. 8. Corollary. Under the assumptions of Theorem 7 grad,v(t,x) and grad, w(t,x) are continuous in R T . Moreover, for any R > 0, 1 E (0,l) there is a constant N such that for 1x1, lxzl 5 R, and t, t,, t, E [O,T] the inequalities

Ix,~,

Igrad,ui(t,x,) - grad,ui(t,xz)l I NIX, - xzl" (grad, ui(t,,x) - grad, ui(t2,x)l I N J t l - t2IN2, are satisfied, where u ,

i = 12, i = 1,2,

= v, u, = w.

Further, since the nonnegative function w(t,x) - g(t,x) is continuously differentiable with respect to x, its derivatives with respect to x vanish at the points of H T , at which this function vanishes. This implies 9. Corollary. The smooth pasting condition

grad, w(t,x) = grad, g(t,x) is satisfied, under the assumptions of Theorem 7 , everywhere on the set { ( t , ~E)

RT:w(t,x) = g(t,x)) and, in particular, on the boundary of this set.

10. Remark. The assertions of Theorems 5 and 7 will hold if in formulating the conditions imposed on ya after Theorem 2 we do not require continuity of y:,,, y:l,(,,, and (a/at)yaif by these derivatives we mean generalized derivatives, and if, finally, we drop the condition ILa(T,x)g(x)l+ ILag(T,x)l 5 K(1 +

1~1)~".

We shall explain this. In fact it suffices to show that Theorem 4 will still hold if we replace K, m by some other constants in the formulation of the theorem. Let us smooth the coordinates ya(t,x), assuming ya(t,x)= ya(O,x)for t I 0. Furthermore, using the vector ya(t,x,e) z [ya(t,x)](e), let us construct the payoff functions v(t,x,s) and w(t,x,s). For 0 < E < 1 the vector ya(t,x,e) satisfies all the conditions formulated after Theorem 2 containing the constants K' and m', which do not depend on E since, for example, yR(t,x,e) = [y:l,(t,x)](e). Hence for the functions v(t,x,e) and w(t,x,e) Theorem 4 holds true in which K , m, p, and Q* are replaced by K', m', ,u(t,x,e) and Q*(e),respectively, constructed on a basis of ya(t,x,e).

Notes

Next, in each cylinder C ,, the vector ya(t,x)satisfies a Lipschitz condition with respect to (t,x)with a constant not depending on a since estimates of the generalized derivatives yTl, and (8/8t)yado not depend on ol. This readily implies that ya(t,x,e) + ya(t,x) as e -+ 0 uniformly on A x C,,, for each R. In particular, p(t,x,e) -t p(t,x). Furthermore, by Theorem 3.1.12 and Corollary 3.1.13, v(t,x,e)+ v(t,x) and w(t,x,e) + w(t,x) as E -+ 0 uniformly on each cylinder C .,, The convergence of the payoff functions and the estimate given in Theorem 4 of their generalized first derivatives with respect to (t,x), which is uniform with respect to e E (0,1), enables us, as was mentioned in Section 2.1, to prove the existence and to estimate the generalized first derivatives of u(t,x) and w(t,x) with respect to (t,x).We can estimate v(,,(,,(t,x) and w(,,(,,(t,x) in the same way, if we take advantage of the fact that all p(t,x,e) 2 &(t,x) > 0 due to the uniform convergence of p(t,x,e) to p(t,x) in each bounded region Q' which together with its closure lies in Q*, starting from some instant of time.

Notes This chapter uses the methods and results of [34, 36, 371, and [58, 591. For control of jump processes, see Pragarauskas [63]. Section 1. If the set A consists of a single point, i.e., if we consider a diffusion process, it is possible to regard the functions v, - v as payoff functions. Therefore, in Theorem 4 we then have equality instead of inequality. Similar assertions can be found in Freidlin [18]. Theorem 8 is the generalization of a result obtained in [38, 621. Sections 2, 3. The method applied in these sections, involving the derivatives in the sense of Definition 2.1.2, enables us to do without the theorems on interior smoothness of solutions of elliptic as well as parabolic equations, that is, the theorems used by Krylov, Nisio, and Pragarauskas (see the references listed above). Section 4. In Theorem 2.9.10 the differentiability of v(t,x) with respect to t is derived from the existence of second derivatives of o, b, c, f with respect to x. Exercise 4 shows that in the presence of control in order to estimate (a/at)v(t,x),we need to require that the derivatives of o,b, c, f with respect to t exist. Section 5. The results obtained in this section for the time homogeneous case, can be found in [34]. It is well known that the limit of harmonic functions is harmonic, and the associated theory has much in common with the theory developed in this section. Section 6. The relationship between the payoff functions associated with a controlled process, as well as the nondegenerate approximation of this process, is investigated in Fleming [16], Krylov [37], Tobias [74]. Section 7. The fact that a payoff function satisfies the Bellman equation implies, in particular, that the Bellman equation is solvable. It is interesting to note that differential equations theory suggests no (other) methods for proving the solvability of the Bellman equations in question. The smooth pasting condition (Corollary 9) was first introduced by Shiryayev (see [69]).

The Construction of E-OptimalStrategies

5

The main objective of investigating a controlled process approached from a practical point of view is to construct optimal strategies or strategies close to optimal. In this chapter we show how one can find &-optimalstrategies in optimal control problems which were discussed in Chapters 3 and 4. Recall that we proved in Chapter 3 that one can always find &-optimal strategies in the class of natural strategies. In this chapter we focus our attention on constructing Markov (see Definition 3.1.3) &-optimalstrategies which is of interest from a practical point of view due to the simplicity of Markov strategies. Adjoint Markov strategies which are investigated in this chapter, are somewhat more complex, becoming thereby less applicable in engineering than Markov strategies (see Definition 3.17). However, from a theoretical point of view adjoint Markov strategies are more convenient in some respects than Markov strategies. Considering adjoint Markov strategies, we prove in Section 3 that a solution of the Bellman equation is a payoff function. In the arguments of this chapter the results obtained in Chapter 4 on payoff functions satisfying the Bellman equation play an essential role. Throughout Chapter 5 we use the assumptions, definitions, and notations given in Section 3.1.

1. &-OptimalMarkov Strategies and the Bellman Equation We showed in Sections 1.1, 1.4, and 1.5 the way to construct &-optimal strategies having the knowledge of the payoff function. In this section and

5 The Construction of &-OptimalStrategies

the sequel we carry out construction of these strategies in the following cases : a. for each R > 0 there exists a number 6, > 0 such that for all a E A, (t,x) E CT,R,jl E Ed the inequality (a(a,t,x)A,A)2 6R(AI2 is satisfied; b. for all t E [O,T],x E Ed, jl # 0 sup (a(a,t,x)il,jl)> 0 ; asA

c. a(a,t,x) does not depend on x. The technique for finding &-optimalMarkov strategies in the case (a) is given in this section. For (b) and (c) we shall prove the existence of &-optimal Markov strategies and construct randomized &-optimalMarkov strategies in the subsequent sections. The case (c) incorporates the control of a completely deterministic process when o(a,t,x) = 0. In this section, in addition to the assumptions made in Section 3.1, we impose the following conditions. Let A be a convex set in a Euclidean space. Also, for each (t,x) E R, let the functions a(a,t,x) and b(cc,t,x) satisfy the Lipschitz condition with respect to a, namely, for all a, P E A, (t,x) E RT let Ilo(a,t,x) - o(P,t,x)ll + Ib(a,t,x) - b(P,t,x)l 5 Kla - PI.

+ +

Furthermore, we introduce a vector ya(t,x)of dimension d x (dl d 4) whose coordinates are given by the followirg variables : oij(a,t,x)(i = 1, . . . ,d, j = 1, . . . ,dl), bi(a,t,x)(i = I, . . . ,d), ca(t,x),fa(t,x), g(x), g(t,x). We assume that for each a E A, 1 E Ed the derivatives ~:~,(t,x), y;,(,,(t,x), (d/dt)ya(t,x)exist and are continuous with respect to (t,x) on RT.In addition, for all a E A, 1 E Ed, ( t , ~E) RTlet

Finally, we assume for the sake of convenience that for all a E A, x f Ed As was shown in Section 4.4, we can always get rid of the last assumption by choosing appropriate constants K and m in the other assumptions made above. We regard the foregoing assumptions as satisfied throughout this section. 1. Lemma. Let a bounded region Q c HT and let a function u E W1*'(Q). For the function a = a(t,x) with values in A let

1. &-OptimalMarkov Strategies and the Bellman Equation

W e assert that for each

E

> 0 one can jind a function a(t,x) given on

(- co,co) x Ed which is injinitely differentiable with respect to (t,x) and, in.

addition, has values in A and a constant N such that

for all x, y

E Ed, t

2 0.

PROOF.We fix E > 0. We proceed in the same way as in proving Lemma 1.4.9. We choose a countable subset {a(i):i 2 1 ) which is everywhere dense in A. From the equality FLU] = sup i

+ ~ ( ~=' lim 1 max [ ~ ~ "+' uf""'] n+ao

isn

and the boundedness of Q we readily derive the existence of the measurable function a"(t,x)which assumes only a finite number of values from {a(i)) and is such that Ilh"ld+ l , Q I 4 2 . Assume that a"(t,x)is defined everywhere in Ed+,and equal to a(1) outside Q. We take the smoothing kernels nd+'[(nt,nx). Let a,(t,x) = nd+'c(nt,nx)* a"(t,x). As has repeatedly been noted, a,(t,x) are infinitely differentiable, a, + a" (a.e.). Moreover, a,(t,x) E A for all (t,x) because A is convex. Further, it follows from the continuity off" and the coefficients of La with respect to a that ha- -, h"a.e. on Q). Due to boundedness offa and the coefficients of La on Q there exists a constant N for which

for all a E A everywhere on Q. Hence the totalitv of the functions han is bounded by one function in gd+,(Q). By the Lebesgue theorem, Ilhanlld+l , Q + Ilh"lld+l , Q . Therefore, there exists a number n(e) such that

Next, we let a(t,x) = a,(,)(t,x) and prove that the function a(t,x) is the sought function. We embed the region Q into a cylinder C,,,. The function [(t,x) is equal to zero for 1x1 > 1. Outside of Q, therefore, outside of CT,,E = a(1).It is easy to derive from the foregoing properties of the functions [ and E that a,(t,x) = a(1) for 1x1 > R + 1 for all n. In particular, a(t,x) = a(1) for 1x1 > R + 1. For reasons similar to those above, a(t,x) = a(1) for t < - 1 and for t > T + 1. Due to the continuous differentiability of a(t,x) we have N 1 = sup sup Ia(,,(t,x)l < co. t,x

IsEd


Finally, let us show that for all x, y E Ed, t 2 0 We have

Ilo(a(t,x),t,x)- o(a(t,y),t,y)ll I K(1

+ Nl)lx

-

yl.

We can estimate the corresponding difference for the functions b in a similar way. The lemma is proved. Note that the function a(t,x) whose existence is asserted in the lemma depends on the combination Q, u, E. In the case where Q is a subregion of H T , u E W1,'(Q), E > 0, we denote the combination Q, u, E by p: It is convenient to write the function a(t,x) constructed on the basis of p as a[p](t,x).For a fixed so E [O,T] and the function a[p](t,x)we can define the Markov strategy a [ p ] using the formula

+

+

+

+

Since the functions o(a[p](so t, x), so t, x), b(a[p](so t, x), so t, x ) satisfy the Lipschitz condition with respect to x, the Markov strategy a [ p ] is admissible at a point (so,x)for each x E Ed.

2. Theorem. For each R > 0 let there exist a number 6 , > 0 such that for all a E A, ( t , ~E) CT,,, A E Ed (a(a,t,x)n,n)2 6~lL1'. Then v(,, = v on RT. Furthermore, we jix so E [O,T], assume that p = ( C T , R , ~ ,and, ~ ) , jinally, define the Markov strategy a [ p ] using Eq. (1). Then for each x E Ed lim lim v"[P1(so,x)= v(so,x). R+m e J 0

PROOF.First we note that since

the first assertion follows from the second assertion. We have from Theorem 4.7.7 that v E W1*2(CT,R) for all R. Therefore, we have defined the strategy a[p]. Furthermore, F[v] = 0 (HT-a.e.). Therefore, by the definition of the function cl [ p ](t,x) llhPlldf

where

1,CTSn

2

'9

1. &-OptimalMarkov Strategies and the Bellman Equation

According to Ito's formula

+ M::J

JiTsR hp(so + t,xt)e-vt dt,

where T T , R = inf{t

+

2 O:(so t , ~ , 4) [O,T) x SR).

By Theorem 2.2.4, the absolute value of the last mathematical expectation does not exceed NIIhPIId+I , C = , R < - NE, where N does not depend on E. Hence it tends to zero as E 1 0. Further, by virtue of the equality v(T,x) = g(x) the first expression in the right side of (2)equals

+ M$:V(SO + For proving the theorem it suffices to show that the two last terms tend to zero as R -, co uniformly with respect to E. By virtue of the growth estimates, as 1x1 + co, of the functions f"(t,x), g(x), v(t,x), to do as indicated we need only to prove that

Note that on the set

{ ~ a , [ % ~ ~ 3 "

time ofjirst exit of the process (so + t, xFIP1,sOO.") g(t,y)). Then for each x E Ed lim lim V " [ P ~ ~ ~ = ~ (w(so,x). S~,X)

R+m &LO


PROOF.This proof follows closely the preceding one. Ry Theorem 4.7.7, F[w] = 0 (Qo-a.e.).Therefore, by the definition of a[p](t,x) where According to Ito's formula

+ MYS:"" where

-

z,,,

= infit

hp(so+ t, xJe-'PCdt, 2 O:(so

+ t,x,) 4 Qo n ([O,T) x S,)).'

By Theorem 2.2.4, the last term does not exceed N~~hP~~,+,,QonCT,R and tends to zero as E 10. The first term in (4) is equal to

where the mathematical expectations tend to zero as R + GQ uniformly with respect to E,which fact can be proved in the same way as the corresponding fact from the preceding proof, since TT,, I zo I T - so, and since, in addition, the inequality T,,, < T - so is satisfied on the set {T,,, < 7,). The analysis carried on Eq. (4) implies the assertions of the theorem, thus completing the proof of the theorem.

2. &-OptimalMarkov Strategies. The Bellman Equation in the Presence of Degeneracy Theorems 1.2 and 1.3 provide the technique for finding &-optimalMarkov strategies if the strong nondegeneracy condition is satisfied: (a(a,t,x)A,A)2 6RlA12for all a E A, (t,x) E C,,, A E Ed, R > 0, where 6, > 0. If we reject this condition, we shall not know how to construct &-optimal"pure" Markov strategies. In some cases considered in this section it is possible, however, to construct &-optimal"mixed" Markov strategies making no assumption about nondegeneracy. In the cases mentioned we prove the existence of (usual) &-optimalMarkov strategies. The superscripts ~ [ p ]so, , x are omitted here and below in the proof.

2. The Bellman Equation in the Presence of Degeneracy

In this section we assume that the assumptions made in the preceding section are satisfied. In particular, we assume that A is a convex set in a certain Euclidean space. We denote by (so,xo)a fixed point of HT. In addition to the basic dl-dimensional Wiener process (wt,Ft)we shall need a d-dimensional Wiener process (9t7@J as well as a (d + dl)-dimensional Wiener process (fit,&). We assume that the processes listed can be defined on the probability spaces (Q,F,P), (fi,@,F), (a,g,p),respectively (we permit these spaces to be equivalent). The last d coordinates of the vector fit form a d-dimensional Wiener process which we denote by %if.We denote by iT; a dl-dimensional Wiener process composed of the first dl coordinates of the vector fit. Recall that we agreed in Section 1 to denote by the letter p the triple Q, u and E, where Q is the bounded subregion of HT, u E W1s2(Q),and E > 0. As in Section 1, we denote here by a[p](t,x) some smooth function on Ed+, with values in A such that its first derivatives with respect to x are bounded, the functions satisfy the Lipschitz condition with respect to x uniformly with respect to t, and E. IIF[u] - [La[P1~f"[P1]lld+ 1,

+

The existence of the function a[p] having the properties listed was proved in Lemma 1.1. If p = (Q,u,E),zt is a (nonrandom) continuous function which is given on [0, T - so] and assumes values in Ed, we can define the Markov strategy cc[p,z) using the formulas

We consider the equation

where br(x) = b(a[p](so

+ r, x + EZ,),so + r, x).

In the same way as we did in proving Lemma 1.1, we show here that the coefficients of Eq. (2) satisfy the Lipschitz condition with respect to x with a constant K(l N,), where N1 has been taken from the proof of Lemma 1.1. Therefore, Eq. (2) is solvable, and, furthermore, the Markov strategy a[p,z] is admissible at a point (so,xo).Using the notation from Section 3.1, we write the solution of Eq. (2) as x~[~,"~,"".~. Since so, xo are fixed, we write in short x:[P,'].

+


It is seen that va[P~'l(so,xo) 5 v(so,xo)everywhere on 6. We shall prove below (see Corollary 2) that ~ " [ P ~ " ~ ( sis~a, xrandom ~) variable. Hence for any sets p = (Q,u,E) &jva[p,*1 ( ~ 0 ~ x20 u(so,xo), ) (3) where M denotes the mathematical expectation associated with the measure

B.

We can interpret the mathematical expectation in (3) to be the payoff obtained by means of a mixed (weighted, randomized) Markov strategy. We explain this without defining concretely the mixed strategy. Assume that a probability measure is given on a set 'UM(so,xo).Also, we assume that first in correspondence with this measure we have a game involving the Markov strategy a and, next, the process is controlled by means of this strategy a. The average payoff of the control is equal to va(so,xo).The integral of va(so,xo) with respect to a probability measure on %,(so,xo) represents the total average payoff of the control of this type. In the case when the probability distribution on 'UM(so,xo)is given by a random element a[p,@], the total average payoff equals the left-hand side of (3). From a practical viewpoint, the technique of controlling a process by means of a random Markov strategy is no less legitimate than that of controlling a process by means of a (nonrandom, pure) Markov strategy. The fact that the left-hand side can be expressed in other terms is very important for the further discussion. On the probability space (fi,#,F) we consider the following equation:

where ar(x) = cr(a[p](so &(x) = b(a[p](so

+ r, x + ~fi:'),so + r, x), + r, x + ~fi:'),so + r, x).

Eq. (4) has a unique solution. In fact, o",(O)and &(o) are bounded, since a(a,t,O) and b(a,t,O) are bounded uniformly with respect to (a,t). Furthermore, the functions o",(x), Gr(x) satisfy the Lipschitz condition with respect to x, having the constant K(l + N,), where N, has been taken from the proof of Lemma 1.l.Finally, the processes Zr(x)and &(x)are progressively measurable with respect to We introduce a convenient notation for a solution of Eq. (4). If x, is a solution of Eq. (4), we put

{&I.

The process P[p] is a strategy with respect to a system of the o-algebras of in the sense of Definition 3.1.1. It is seen that x, satisfies the equation

(6)


Using the standard notations, we can write the following: x, = X,B[P~."~~"~. This also enables us to apply usual short notations, write the indices P[p], so, xo only for the sign of the mathematical expectation, and introduce v ~ [ ~ ~ ( s ~ , x ~ ) , in writing mathematical expectations of functionals of a solution of (4). In addition, since so, xo are fixed, we shall write xftP1 insteadof x ~ [ P 1 ~ s O ~ x O . The following lemma gives an obvious formula with which we can modify the left-hand side of (3). 1. Lemma. Let F(z,xIO,T-sol) be a measurable function given on CZ([O, T - so],Ed)and such that

for some constants N , n and also for all z, x[o,T-solE C([O,T - so],Ed).Let a bounded region Q c H T , let a function u E W1,'(Q) and let a number E > 0. Using Lemma 1.1., we construct the function a[p](t,x) on the basis of the set p = (Q,u,E).Using formula (1) we introduce Markov strategies a[p,z] for z E C([O,T - so],Ed). Furthermore, we dejine the strategy P[p] using the ), x, is a solution of Eq. (4). Then formula Pt[p] = a[p](so+ t, x, + ~ f f ; 'where the function is bounded and measurable with respect to z for z E C([O,T - so],Ed), where M (m) denotes the mathematical expectation associated with the measure

S (P).

PROOF. The boundedness of @(z)follows from (5) and the fact that due to familiar estimates of moments of solutions of stochastic equations

In proving (6) and the measurability of @(z),we use the results obtained in Section 2.9. We note that it is possible to solve Eq. (4) in a different way. We denote by g:the completion (with respect to measure p) ofthe smallest a-algebra containing & as well as all sets of the form { f i r E T } , where rI T - so and T denotes a Bore1 subset of Ed.We draw the reader's attention to the fact that r runs through the entire interval [0, T - so]. It is easily seen that the processes dt(x),&(x)from Eq. (4)are progressively measurable with respect to new a-algebras, and also that (%i,P,) is a dldimensional Wiener process. We note as well that the solution of Eq. (4)does not change on [0, T - so] when we pass from to (ffi,@,). This obvious fact follows, for instance, from the uniqueness of a solution of Eq. (4)and the fact that the former solution x, is progressively measurable with respect to

(ff;,g)


{&] and, in addition, is progressively measurable with respect to { p t } by virtue of the inclusion gtc p t ( t E [0, T - so]). Further, we use Theorem 2.9.4 in the case where 2 = C([O, T - so], Ed),

= %;6,T-so],

t = X~

and for z E Z o;(x) = o(a[pI(so + t, x + 4 , So + t, 4 , bf(x) = b(cr[p](so t,x &zt),s0 t,x).

+

+

+

Then the solution of Eq. (4) is a solution of the equation x, = t

+ si c!(xr) diVi + si bj(xr)dr.

By Theorem 2.9.4 (also, see Remark 2.9.9) ~ ~so,xo{F(~"~~[o, P[PI T-sol)l = 8(wt), where 8(z) = fiF(Z,Z;~~]-

q[P3z1 is a solution of Eq. (2),in which w, is replaced by tT:. 8(z) = @(z).Hence

By Corollary 2.9.3,

-

M ~ ~ ~ , F ( % "T-sol) , X [ ~=, fi@(%")= ~@(i?). We have proved Eq. (6). We can easily derive the measurability of 8 , therefore, of 8,from Remark 2.9.5, if we write F as F + - F -. This proves the lemma.

2. Corollary. For a = a[p,z] or a = p[p] let z = z" be the time of Jirst exit of the process (so t,xF)from a region Q1 c (- 1,T) x Ed. Then, the function v"[P~zl~~so,xo) is measurable, bounded with respect to z on C([O, T - so], Ed),and,Jinally,

+

Furthermore, the function v"[P~zl(so,xo) is measurable, bounded with respect to z on C([O, T - so], Ed),and also

Indeed, the second assertion is a particular case of the first assertion ( Q , = (- 1,T) x Ed, g(t,x) = g(x)). In order to prove the first assertion, we introduce the function z(z) for z E C([O, T - so],Ed) to be the time of first exit of the curve (so + t, z,) from Q,. Since Q1 c (- 1,T)x Ed,z(z) I T - so. It is easy to prove that limZn,,z(z,) 2 z(z). Therefore, the function z(z) is lower semicontinuous and, in addition, is measurable with respect to z. Further, we consider the function


where

[

exp - J:

P[PI(SO

+ r, ++

22.)

(so

+ r, x,) dr

I

dt.

The last function is obviously continuous on [0, T - so] x C2([0,T - so], Ed). Hence is measurable, being the composition of measurable functions. The application of the lemma to the function F(z,xto,T - s o l ) leads us immediately to the first assertion of the corollary.

3. Remark. We have discussed above the technique for controlling a process by means of an initial randomized Markov strategy. This technique provides an average payoff equal to the left-hand side of (7). From Eq. (7), we get another possibility to obtain the same payoff. Suppose that we have realized the d-dimensional Wiener process 6t so that it is observable and independent of w,. The pair (w,,%,)forms a (dl + d)-dimensional Wiener process. Furthermore, the pair ( X ~ ~ * " ~where , Z , ) , z, = %,, satisfies the equation

We have thus obtained a 2d-dimensional controlled process, for which the function a" = a[p](s, t,x, EZ,) is a Markov strategy (we mentioned the observability property of 6, because trajectories of a controlled process are taken as observable). If we assume the process (w,,%,) to be E,, we can easily see that

+

+

Therefore, we can obtain the left-hand side of (7) by mixing Markov strategies as well as by applying a strategy which is Markov with respect to a complete controlled process. In the next section we call strategies of this kind adjoint Markov strategies. 4. Lemma. W e take the functions v, from Section 4.6. Further, for E # 0 let p = ( C T , R , ~ , , ~Using l ) . Lemma 1.1, we construct the function a[p](t,x)on the basis of the set p. Using Eq. ( I ) , we introduce Markov strategies a[p,z] for z E C([O,T - so], Ed). On a probability space (O,F,P) we dejine the strategy p[p] using the formula fi,[p] = a[p](so+ t,x, + &it$'), where xi is a solution

- --


of Eq. (4). Finally, we assume that

[

x exp -

11

J: cfir[pl(so+ r, xr + E@:) dr

Then v~,)(so,xo)= v(so,xo).Moreover,

dt 2 v(so,xo).

~ v(so,xo), ~ ~ ( s ~ , x ~ ) lirn lirn - lirn -~ v ~ ~ ~ = &+O R + m

~ ~ l o

and alsofor each 6 > 0 lirn lirn lirn ~{va[p,'"l(so,xo) < v(so,xo)- 6) = 0. E+O R + m el10

PROOF. It follows from (10) that v ( ~ ) ( s ~= , xv(so,xo). ~) In turn, (10) follows from (9) since, according to Chebyshev's inequality, the probability given in (10) does not exceed

Therefore, we need to prove only (9). First we note that for E # 0 the nondegeneracy of the processes x?"."(&)(see Section 4.6, Inequality (4.6.2)) (a/dt)vEas well as ensures the existence of generalized derivatives vexi,veXixj, the boundedness of these derivatives in each cylinder C,,,. This implies, in particular, that the function a[p](t,x) is definite. We take an arbitrary strategy P = Pt which is progressively measurable with respect to {gt}. Further, we consider the expression

where x, is a solution of the following equation (having coefficients not depending on E):


Differentiating U ( E ) over E, bringing the notation of the derivative under the sign of the mathematical expectation and the integral, using the fact that the derivatives f , c, and g increase with respect to x not faster than a certain power, and, finally, applying the familiar estimates of moments of solutions of stochastic equations, we conclude that there exists a constant N(xo,K,T,m) for which /U'(E)~ I N(xo,K,T,m)for I 1 . Hence lu(0)- U ( E ) I ~ N(xO,K,T,m)lel for lei I 1. It is crucial that the constant N does not depend on the strategy j. Due to the result thus obtained, we can replace the expression x, 8%:' by x, everywhere in (8). However, by Corollary 2 the left-hand side of (8) thus modified coincides with the left-hand side of (9).Therefore, the left-hand side of (9) is not smaller than v(so,x,). Since va(so,x0)s v(so,x,) for each strategy a E %, the left-hand side of (9)is, on the other hand, not greater than v(so,x,). We have thus proved the lemma.

IE~

+

Now we can prove the main result of this section. 5. Theorem. Suppose that at least one of the following conditions is satisjied: a. o(a,t,x) and b(a,t,x)do not depend on x ; b. a(a,t,x)does not depend on x ; c. for all t E [O,T], x E Ed,IZ # 0 sup (a(a,t,x)A,A)> 0. a~ A

Then, using the notations of the preceding lemma, we have that inequality (8)as well as the assertions of this lemma hold true.

PROOF.Presumably, if condition (a) is satisfied, condition (b) will be satisfied. We included condition (a) in the statement of the theorem for completeness. If (a) is satisfied, the proof of the theorem becomes very simple. In fact, if (a) is satisfied, then, (see Eq. (4))the process xfLP1 E%;'is a solution of the equation

+

If we introduce the matrix 6, in the same way as we did in Section 4.6, we can easily turn the last equation into Eq. (4.6.1). Therefore, xfLP1 + E%;'= x ~ [ ~ ~ for , ~a11~ t ~almost ~ ~ (surely. E ) Further, from formulas (4.6.3) and the definition of a[p] = a[C,,,,u,,cl], we have F,[u] - [Ltu

IIFe[v&]- [ L t L P 1+~ &

+ f " ] = F[u] - [Lau + f"],

/Id+ I , C T , R

< El. -

(11)

From this, fixing E # 0 and applying Theorem 1.2 to the controlled process have that the expression in (8)under the sign of the lower limit

X ~ ' " * ~ ( Ewe ),


with respect to E is equal to v,(so,xo). By Theorem 4.6.1, v, + v; therefore, we have proved inequality (8) and, by this token, the assertions of Lemma 4 for (a). If (a) is not satisfied, the equality xtLP1 + 8%;' = X;[~~"".~(E) does not hold, in general. Thus, we cannot apply Theorem 1.2 for proving (8). In cases (b) and (c), to be considered at the same time, we can prove Eq. (8) almost in the same way as Theorem 1.2. We assume everywhere below that E # 0,lel I1. It can easily be seen that the process y: = xfrP1 E%;) satisfies the equation

+

By Theorem 4.7.7, F,[u,] follows that

-

=0

(H,-a.e.). From this as well as from (11) it

< &I,

IlhPl/d+ ~ , C T , R-

where hP - LfrP1u,- farp1.Since the matrix a,o: is uniformly nonsingular, we can apply Ito's formula to the expression

Using Ito's formula, for each R1 2 0 we obtain where, letting q: =

J; cMpl(so+ r, y:)

dr,

zP,' denotes the first exit time of the process (so + t,y[) from [O,T) x SR1, we write the variables If(R1) as follows:

- b(.[pI(so

x grad, v,(s,

+ t, y/),so + t, Y!

- aij(a[p](so

+

+ t, y[)e-vf

-

&%;)I

dt,

+ t, Y!), so + t, Y:

x v,xi,j(~o t, yf)e-'~fdt.

- &%;')I


As in proving Theorem 1.2, we show that lim,llo I,P(R1) = 0 if R > R1; lim sup sup sup

R1-+m l a l s l R > O e l > O

Next, we turn to the variables IP,(R1). By Theorem 4.1.1, it is easy to obtain for I 1 that Igrad,v,(t,x)l I N(K,T,m)(l + IXI)~"(HT-a.e.).Suppose that the last inequality is satisfied on a set T,such that means (HT\T,) = 0. We put the sum ~r,(so+ t, Y)! + X H ~ \ T ~+( St,O YP) before dt in the formula for IP,(R1). Furthermore, we split IP,(R1)into two terms in an appropriate manner. Applying Theorem 2.2.4 to the second term, we can see that it equals to zero. The first term as well as, therefore, IP,(R1)does not exceed

1

l&%;l(l+ y[l)'" dt.

N(K,T,~)~ Therefore,

lirn sup sup IIP,(R1)l = 0.

&+O R > O e l > O

If (b) is satisfied, then I$(R1) = 0. If (c) is satisfied, according to Remark 4.7.6 the derivatives v,,, are bounded in C,,,I (a.e.) by a constant not depending on E. In addition, for each 1 E Ed

Hence Ila(4t7x) - a(a,t,Y)ll IN(d,d1)K2(1 + 1x1 + lYl)lx - Yl,

where N does not depend on E,R, R1). This indicates that

(although N depends, for example, on

lim sup sup Il$(R1)l = 0 &+O R > O & l > 0

in both cases (b) and (c). Finally, from (12) and the properties investigated of IP(R1) we conclude that for each R1 2 0 lim v,(so,xo) 5 lim lim 0

&-t

,-to

lim

R-tm cl/O


where y(R1) is the expression in (13) under the sign of the limit. Letting R1 -t co and noting that, by Theorem 4.6.1, the left side of the last inequality is v(s,,x,), we arrive at inequality (8) (the notations have slightly been changed). The theorem is proved. We suggest the reader should prove a similar theorem for the optimal stopping problem as an exercise.

6. Exercise Let one of conditions (a), (b), or (c) of Theorem 5 be satisfied. Also, let p = ( C T , R , ~ , , ~ l ) , 7, = T ~ [ be ~the time ~ ~of first ~ exit . ~of the ~ process ~ ~ (so~ + t , x ~ [ ~ ~ from " ] ) Q, = {(t,x) E H,:w(t,x) > g(t,x) + 6 ) . As an analog of Lemma 4, prove that lim lim lirn lirn M V " [ ~ ~ " ~ ~ '= ~ (w(sO,xO) S~,X~) -810 E-0 R - r m z 1 l O

and that for each 6' > 0 --

-

610 E-0 R-m

8'10

..

lirn lirn lirn lirn ~ { v " [ ~ ~ " ] ~ ' ~ 0 were satisfied at a point (s,x) E HT, it is seen that F[u] u - g > 0 at this point, hence (F[u] u - g), + g - u = F[u] > 0. Therefore, F[u] < 0 (HT-a.e.), and also u is a superharmonic majorant of g(s,x). Theorem 5 and Corollary 7 enable us to find upper estimates for a payoff function. In order to prove the theorem on lower estimates, we need three auxiliary results.

+

+


8. Lemma. Let (so,xo)E HT,let a(s,x) be a Bore1 function on HT with values in A, and let 6 > 0. Further, let

n b,(s,z) = ---b (s z z). n + 1zI2 " ' ' Let us jnd a strategy an,' using the formula

where z?'(o) is a solution of the equation z, = x0

+ Ji an(so+ r, z,.)dw, + 6*, + Ji bn(so+ r, z,.) dr.

Then for all q 2 1

< a,

sup sup M sup

8 ~ [ 0 , 1n] z l

--

lim lim M sup 6 1 0 n+co

t5T-so

ZsT-so

'$I,

-~

~

~

'= 0. ~

~

~

~

,

~

PROOF. AS can easily be seen, the functions

are differentiable over z and, furthermore, the derivatives of these functions do not exceed Nn, with N not depending on n, s, x, z. Moreover, they satisfy the Lipschitz condition with respect to x since, for example, using the simple inequality we find

This implies that the coefficients of Eq. (4) satisfy the Lipschitz condition and Eq. (4)is solvable. Due to the familiar estimates of moments of solutions

~

l

~

3. The Payoff Function and Solution of the Bellman Equation

of stochastic equations (5) follows from the inequality

and a similar inequality for Ib,(t,z)l. Further, the process x;n'6~s0~x0 is a solution of the following equation:

Comparing the last equation with Eq. (4), we have in accord with Theorem 2.5.9 that

- b(a:s6,so + t, z:s6)12q dt, where N depends only on q, K, T - so. It remains to show that the last two terms tend to zero as n + co. We fix 6 > 0. Also, we consider the latter term only. Let zk be the time of first exit of the process z?"rom S,. We have

as R + co uniformly with respect to n. Therefore, we can complete proving the lemma if we prove that for each R > 0

as n -+ co.We note that the process z:pS is a solution of the equation zt = xo

+ Ji a",(s0 + r, z,.)di~,.+ J: bn(sO+ r, z,.)dr,

where the matrix a", is obtained as a result of writing to onfrom the right of the unit matrix of dimension d x d, which is multiplied by 4 1 , = (wr,5,).It is not hard to see that for each A, l5;1l2 = (5,5,*A,A) = (onaxA,A)+ 6A2 = la,*AI2 6L2 2 6A2. Therefore, by Theorem 2.2.4,

+


where N does not depend on n. Finally, the functions bounded in C,,, uniformly with respect to n

Therefore, according to the well-known properties of convolutions (see Section 2.1) the foregoing functions tend to zero as n -+ co for each t for almost all z. Thus, by the Lebesgue theorem, the right side of (7) tends to zero as n + co.The lemma is proved. Repeating the arguments given in the proof of Theorem 3.1.12 after Eq. (3.1.14) and, in addition, using the current lemma, we arrive at the following (also see Corollary 3.1.13). 9. Lemma. Let us carry out the construction as was described in the formulation of Lemma 8. W e take arbitrary Markov times znv6E 1)32(T- s,). Then

x exp

[-"J:

x exp[-

ca:s"so

+

fi P:'~(s,+ r,

z76)dr]

z*:l

+ "J:

I.:"(S~

+ ti z:")

dr] dt} - u a n ' ~ o , x= o )0.~

10. Lemma. Let a bounded region Q c HT. Let a function u satisfy inequality ( 1 ) in H T , u E W1,2(Q)n C(RT),let F[u] 2 0 (a.e. on Q), and let E > 0. We take a Bore1 function a(s,x) given on HT which assumes values from A and such that + f l l ( S , X ) ( ~2, ~- E) (a.e. on Q). (8)


W e jix (so,xo)E Q. Using Lemma 8 let us define the strategies an,< Then

+

u(so,xo)5 lim lirn M ~ ~ , ' ~ , ~ ( s 0xc,6)e-"rns6 6 1 0 n+m

where zn,' is an arbitrary Markov time not exceeding the time of first exit of the process (so + t, z;,') from the region Q.

PROOF.First we note that, in the same way as we did in proving Lemmas 1.4.9 and 4.5.5, we can establish here the existence of a function a(s,x) such that for all (s,x) E Q the left side of (8) is greater than F[u](s,x)- E. Since Flu] 2 0 (a.e. on Q), the function a satisfies inequality (8). Applying Ito's formula, we easily find

where

+ 1 [bi(a:,', so + t, zF6)- bL(s0 + t, z~~')]uxi(s0 + t, z:') d

i=l

1

dt,

By virtue of the inequality zn,Q T - so we obtain from the above as well as (8)that


We take the limit here as n + co, 6 1 0. Assuming g(s,x) = u(s,x) in Lemma 9, we conclude that for proving this lemma we need only to show that -lim lim I",' = 0. 810 n+m

Since the process z:,' is nondegenerate and, in addition,

for some constant N and all (t,z) E Q, n 2 1, by Theorem 2.2.2 (compare with the proof of Theorem 2.10.2)

where N does not depend on n, 6. Therefore, the estimated term belonging to In,'tends to zero as n + co, 6 1 0. Further, by Theorem 2.2.2,

where N does not depend on n, 6. In proving Lemma 8 (see the reasoning carried out after Eq. (7)) we showed that b,(t,z) + b(a(t,z),t,z) (HT-a.e.).In completely similar fashion, o,(t,z) + cr(a(t,z),t,z);therefore, a,(t,z) -+ a(a(t,z),t,z) (H,-a.e.). Since the totality of functions a,(t,z) is bounded on Q and, in addition, the derivatives uXixjE 9,+ l(Q), by the Lebesgue theorem the right side of (9) tends to zero as n + co for each 6. In the same way we can estimate the term containing b' - bb, which belongs to I"," We have proved the lemma. The following theorem enables us to find lower estimates for payoff functions.


11. Theorem. Let u E Wfd,2(HT)n C(RT) and satisfy inequality (1) in HT. Then :

+ +

a. if(F[u] u - w), then u < w in 8,; b. if(F[u] u - v), then u I v in 8,.

+ w - u 2 0 (HT-a.e.),u(T,x) I w(T,x)for all x E Ed, + v - u 2 0 (HT-a.e.),u(T,x) I v(Tx)for all x E Ed,

-

PROOF. Assertion (b) follows from (a). In fact, temporarily let us take g(s,x) r v(s,x). By Theorem 3.1.6, w v and also the inequality u l w from (a) implies that u I v. Proof of (a). For each E > 0 LU(u- E)+ f a = LUu+ f + &ca2 LC(u+ fOL. OL

Therefore, F[u - E] 2 F[u]. Furthermore, we note that for any real a the function (a + t), - t decreases with respect to t, thus implying that

+

+

(F[u - E] (U- E)- w)+ W - (U- 8) (HT-a.e.). 2 (F[u] + u - w), + w - u 2 0 It is seen that u(T,x) - E < w(T,x). Therefore the function u - E satisfies (a), for s = T this function being strictly smaller than w. If assertion (a) has been proved for such functions, then u - E I w, from which, in turn, we conclude as E 10 that u I w. Therefore, we may assume that u(T,x) < w(T,xj for all x E Ed. We note also that due to the continuity of u and w it suffices to prove the inequality u 5 w in HT. Let Q' = {(s,x)E HT:U(S,X) > w(s,x)). We wish to prove that the region Q' is an empty set. Assume the converse. We take (so,xo)E Q'; in addition, let R > lxol and Q = Q' n C,,, n {(s,x)E HT:s > so/2).. By virtue of the inequality u(T,x) < w(T,x) we have: Q c HT, hence u E W1,2(Q).The expression w - u is negative on the region Q. Therefore, it follows from the inequality (F[u] + u - w), + w - u 2 0 that (FLU] + u - w)+ > 0. Then, almost everywhere on Q Now we can apply the preceding lemma, thus obtaining for fixed E > 0

where zn,' is the time of first exit of (so + t, z:,"

from the region Q.

5 The Construction o f &-OptimalStrategies

-

If we take in Lemma 9 lu(s,x) - w(s,x)l instead of g(s,x) and if, in addition, we put ca(s,x) 0, we shall see that the last term in (10) is equal to --

lim lim M lu(so 6 1 0 n+m

+ ?"'a, z:&)

+ T " , ~z:A$)~. ,

- w(sO

In order to estimate the last expression, we note that if Iz:;f6( < R, a point (so zn3',z:~f6)lies on that part of the boundary of Q, where u = w. In short, if < R, then

+ Iz:;~.,I

Therefore, the last term in (10)does not exceed N sup M(1 + Iz:;~al)m+P~I,;A(j61 Z~ 6,n

I

1 1

+R

---

6,.

1

+

sup

lz:.'/)

m+p+l

t 0 we define the strategy P using the formula P, = a,,,,, where z, is the time of first exit of^^^'" from S,. It is seen that p E %n(R), where n(R) = na(R)has been taken from Definition 4. Furthermore, the processes xff;", and ,:::x satisfy the (same) equation :

Hence the foregoing processes coincide for all t E [0, T - s]. In particular, = x;,s,x for all t IZ, almost surely. Since B E for j = 1,2, . . . , we have by Theorem 3.1.6 that

xa,~,~

Let j -+ co, R -+ co. We note that when R co, T A T R each o for all sufficiently large R. By Fatou's lemma -+

-+

Z, Z A Z ,

= Z

foA

Furthermore, by virtue of inequalities (5) and Definition 4b,c the totality of variables Y : ~ S ~ ( V+j~, z( ~A ,zR)is bounded by a summable quantity. From this, using the Lebesgue theorem, we find lim lim M:,xY(-)(~n(R)+ j, z A z ~=) M:,xY(-,(v,~).

R-tmj-tm

Since Y = Y(+,- Y(-,, the above relations enable us to obtain (8) from (9), thus completing the proof of the theorem. 6. Exercise In Exercise 3 we take g(s,x) = [x2/()xl+ 1)](1 - s). Prove that the assumptions of Lemma 2a can be satisfied for u(s,x) = 1x1, Q = H,\{(s,O)}. Prove also that w(s,x) = Ixl(1 - s), and, furthermore, that if E 2 0, is the time of first exit of (s t, x;pqx) from the region

*","

+

then for (s,x)E Q, sup M:,,g(s KEU

+ r,, x,=) + 6 = sup M:,,w(s + z,,x,J = ~(1x1+ 1) < w(s,x). aePI

This exercise demonstrates that not all the assertions of Theorems 3.1.10 and 3.1.11 are true in general. In this connection, we do not prove theorems on &-optimalstopping times in the general case. We note nevertheless that it is convenient to search for &-optimalstopping times using Theorem 3.1.10 and approximating w(s,x) with the aid of w,(s,x). In the next theorem, it is useful to keep in mind the remark made before Theorem 5.

1. Generalizations of the Results Obtained in Section 3.1

7. Theorem. (a) For all (s,x) E HT W(S,X) =

SUP

sup

as%(s,x) r s'lR(T-s)

va3=(s,x).

(b) The function w(s,x)is lower semicontinuous on H T , w(s,x) 2 g(s,x) on RT. (c) For any a E %(s,x), z E %X(T- s) and the nonnegative bounded progressively measurable processes (with respect to (9,)) r, the inequality W(S,X)

t M:,

I

W(S

+ z, x,) ex(-

9, -

J; rpdp) dt

is satisjied. (d) W e have assertion (3.1.9) of Theorem 3.1.9, in which we can replace the upper bound with respect to a E % by the upper bound with respect to a E %(s,x) as well as by the upper bound with respect to a E %,(s,x).

PROOF. We prove (b) exactly in the same way as we proved (b) in the preceding theorem. Using the notation from the proof of Theorem 5, we can write (c) as follows: We can prove this in the same way as we prove an analogous inequality for v, making use, however, of Theorem 3.1.1 1 instead of Theorem 3.1.6, by which for all n, P E a,, w,(s,x) 2 M!,,Y(w,,z).

-

Proof of (d). For each a E %(s,x) let a time z = za E %R(T - S ) be defined. Putting in (c) rt 0 and also, noting that w(s,x) 2 g(s,x), we obtain w(s,x) 2 sup

sup

M:,,Y(w, z A y )

t sup

sup

M:,{J;AY

ae'U(s,x) ye!Ut(T-s)

aeQl(s,x)ye!Ut(T-s)

P ( +~t , ~ , ) e - ~ d t

Let us extend the above inequalities, replacing first %(s,x) by % c %(s,x), and next by %,(s,x) c %. Finally, let us take into account that the last resulting expression will not be smaller than the following expression: sup

sup

a e Q L , ~ ( s , x yem(T-s) )

M:,{f;^'

+ t, x,)e-vt dt

6 Controlled Processes with Unbounded Coefficients

for all n. Here we have equality by Theorem 3.1.9. To complete proving (d) we need to let n -+ co in the sequence of inequalities obtained if we proceed as indicated above. Assertion (a) follows from (d) if we assume za = T - s. We have proved the theorem. In Theorems 5 and 7 we asserted that v(T,x) = g(x) and w(T,x) = g(T,x). In other words, we know the boundary values of v and w for s = T. Regretfully, it may turn out that these boundary values of the functions u and w are weakly related to the values of these functions for s < T (see Exercise 11). Thus, the question arises as to when the boundary values are assumed, i.e., when lim W ( S , X ) = g(T,x). lim V ( S , X ) = g(x), sfT

sf T

The following lemma shows that always

lim v(s,x) 2 g(x), ST

T

lim w(s,x) 2 g(T,x). sf T

8. Lemma. For all R > 0 the limits lim sup [v(s,x)- g(x)]-,

lim sup [w(s,x) - g(T,x)]-

s f T lxlrR

sTT I x l r R

are equal to zero.

PROOF. By Theorem 3.1.5, lim sup Iv,(s,x)

-

g(x)l = 0.

sTT I x l c R

Hence the assertion of the lemma follows from We can consider the function w in a similar way, thus completing the proof of the lemma.

9. Theorem. (a) For each E > 0 and R > 0 let there be 6 > 0 and a superharmonic (or excessive) function u(s,x) in a region ( T - 6, T ) x Ed such that in this region u(s,x) 2 g(s,x) and u(T,x) < g(T,x) + E for 1x1 5 R. Then for all R >0 lim sup Iw(s,x) - g ( ~ , x )= l 0. s f T IxlrR

(b) For each E > 0 and R > 0 let there be 6 > 0 and a superharmonic (or excessive) function u(s,x) in a region ( T - 6, T ) x Ed such that g(x) I u(T,x) for all x, u(T,x) I g(x) + E for 1x1 < R. Then for all R > 0 lim sup Iv(s,x) - g(x)l = 0. s f T IxlrR

1. Generalizations of the Results Obtained in Section 3.1

PROOFOF (a). We take E > 0 and R > 0. Also, we find 6 > 0 and the corresponding function u(s,x). Applying Lemma 2a, we consider instead of the strip HT the strip ( T - 6, T) x Ed and, furthermore, we take the latter strip for Q. We thus have u(s,x) 2 w(s,x) in [ T - 6, T] x Ed, which yields Further, due to the continuity of u(s,x) and g(s,x) we have

lim

sup [w(s,x) - g(s,x)] + I sup [u(T,x) - g(T,x)] + .

sf T I x l c R

1x1 5 R

By hypothesis, the last expression does not exceed E. Since E is arbitrary, lim sup [w(s,x) - g(s,x)] + s f T Ixl5R

= 0.

Comparing this result with Lemma 8 and noting that la1 = a+ + a _ , we have proved assertion (a). Assertion (b) can be proved in a similar way, thus proving the theorem. 10. Remark. We have applied the version of Lemma 2 in which Q = H T . We could have used Lemma 2 in its general form but the statement of the appropriate theorem would then become too cumbersome. Using the scheme investigated in Section 3.1 as an example, we show how to apply Theorem 9. We shall see that the assumptions of Theorem 9 can always be satisfied if we take the controlled processes given in Section 3.1. We consider only the second assertion of Theorem 9. Assume that E > 0 and R > 0. First we find an infinitely differentiable function g(x) such that g(x) I g(x) for all x, g(x) I g(x) + E for 1x1 I R. It is seen that since 1g(x)1 2"K(1 + IX~~)"~', one can take g(x) = 2"K(1 + IX~')"~~ for 1x1 > 2R. Let

The computations, similar to those carried out after Lemma 2, yield

Using the assumptions made in Section 3.1 about the growth order of a and b, we easily find that in HT


The last expression is smaller than zero for s E ( T - 6, T ) if 6 is such that

Hence the function u' satisfies the hypotheses of Theorem 9b.

11. Exercise We consider a 1-dimensional case: d = d , = T = 1, A[O,co), A, b = c = f = 0. Let g(x) be a bounded continuous function. Prove that for s < 1 v(s,x) = sup g(y), Y

= [O,n],

a(a,s,x) = a,

lim v(s,x) = sup g(y). st 1

Y

12. Exercise Prove Theorems 3.1.10 and 3.1.11 for E > 0 in the general case if it is known that each connected component of the region Q , is bounded. (Hint:It is necessary to prove first that Q, is in fact a region, and second, that for each connected component of Q, there is a number no after which it is contained in the sets {(s,x):w,(s,x) > g(s,x) (~/2)}.)

+

13. Exercise Prove Theorems 3.1.10 and 3.1.11 for e > 0 in the general case if it is known that Iw(s,x)l + f"(s,x)l I N(l 1x1)" for all a, s, x and that, in addition, for some r > 0 for all s, x, E > 0

I

+

sup Mg,, DEN

SUP t5ze

Ixtlrn+* < co.

2. General Methods for Estimating Derivatives of Payoff Functions In Chapter 4 estimates of derivatives of payoff functions played an essential role. They were used, in particular, in deriving the Bellman equations. The proof of the estimates given in Chapter 4 was based on the estimates of moments of derivatives of solutions of stochastic equations with respect to the basis of the initial data in Section 2.8. In this section we give more precise estimates of derivatives of solutions of stochastic equations as well as more precise estimates of derivatives of payoff functions, omitting the proof, which will appear in a forthcoming publication. We introduce a vector yU(t,x) of dimension d x (dl d 4), whose coordinates are given by the variables oij(a,t,x)(i = 1, . . . ,d, j = 1,. . . ,dl),

+ +

2. General Methods for Estimating Derivatives of Payoff Functions

bi(a,t,x) (i = I,. . . ,d), ca(t,x), fOL(t,x), g(x), g(t,x). We assume repeatedly in this section that the assumptions made in the previous section have been satisfied. We also assume that for each a E A, t E [O,T] the vector ya(t,x)is twice continuously differentiable in x and, in addition, for all n, a E A,, 1 E Ed, ( t , ~E) HT

Besides these (tacit in the assertions of this section), we shall need additional assumptions. For convenience of reference we enumerate them. We fix 6 E (O,$]. Also, let

We fix five nonnegative functions u-,(t,x), u,(t,x), u-,(t,x), uz(t,x), uo(t,x), which together with two derivatives with respect to x and one derivative with respect to t are given and, in addition, are continuous in RT. On 8,let

We note that such functions always exist; for example, ui = 0. However, we shall require more than inequality (2). A list of the conditions follows. Not all are required to be satisfied simultaneously. The assertions indicate specifically the various sublists of conditions. 1. Assumption (on first derivatives of a and b). For all (t,x) E HT, a E A, 1 E Ed

where we have omitted for the sake of brevity the arguments (t,x) in the function u,(t,x) and the operator L",t,x), and also the arguments (a,t,x) in the functions a(a,t,x), b(a,t,x). We shall omit the arguments in the other assumptions in a similar way. 2. Assumption (on second derivatives of a and b). For all (t,x) E H,, a E A, a€A,lEEd


3. Assumption (on first derivatives of c, f , g). (a) For all (t,x) E RT, a E A,, l ~ E ~ , ~ = v , , w , ( n = ,...) 1,2

(b) ca(t,x)does not depend on x. There exists a linear subpace Ef of the space Ed such that for each 1' E Ef, 1" IEf, a E A, (t,x) E HT Furthermore, for each I'

E

Ez, 1 E Ed, a E A, (t,x) E RT

Finally, for all (t,x) E H,, a E A, I" IEf

4. Assumption (on second derivatives of c, f , g). For all (t,x) E RT, a E A,, 1 E Ed,u = u,, W , (n = 1,2, . . .) [f ;)(l) - C:~)(,)U] - I (- L ~ ; U ~-@I4)() ( ~ ' L; ~ 2)('I2 +(614),

We shall formulate later two more assumptions. However, right now we discuss some techniques for verifying the assumptions listed in various places. First we note that if in a specific case it is possible to choose functions u, so that each function satisfies only one of the inequalities (6), (8), or (11)-(14), the sum of these functions can be used as u, in all the inequalities. The functions u- ,, u-, appear only on the right-hand sides of the inequalities (4), (5), and (7). The left-hand sides of the inequalities contain u1 only, with the exception of inequality (7) in the left-hand side of which u, appears also. The greatest number of conditions is imposed on the function u,. Assumption 1 being the most stringent. Inequality (3) being satisfied depends only on the appropriate choice of the function u,. In the remaining inequalities, the right-hand sides contain functions which do not appear in the left-hand sides so that these inequalities can be satisfied by choosing these functions appropriately. In this connection, it is useful to bear in mind that since one can multiply or divide these inequalities by (112,(3) can be satisfied for all 1 E Ed if and only if (3) is satisfied for all unit vectors 1. Further, we can easily verify Assumption 1 if first derivatives of o and b with respect to x are bounded by the same constant for all a, t, x. In fact,


in this situation we assume that u,(t,x) = eeN'. Then, the left-hand side of (3) does not, obviously, exceed N,e-N'1112, where the constant N, does not depend on a, t, x. The right-hand side of (3) is equal to

Since ca 2 0, one can, for example, assume that 6 = 4,N = 4N1. In this case the function u, constructed satisfies (3). The constant T does not appear in an explicit manner in the assumptions made, which fact enables us to employ our assumptions for control over an infinite interval. Note that for T = m it is sometimes inconvenient to take a function of the form e-Nt as the function u,. In such cases one can keep in mind that Assumption 1will be satisfied if, for example, ca(t,x)is "sufficiently" large compared to first derivatives of a and b with respect to x, more precisely, if

for all (t,x) E RT,a E A and for unit 1 E Ed. Indeed, (15) coincides with (3) for u1 E 1, 111 = 1. (Inequality (15) is discussed in more detail in the notes to this chapter.) We have given two cases in which Assumption 1 is satisfied. We note that the first case holds in the scheme investigated in Chapters 3-5. Indeed, if the constant K, in (1.1) does not depend on n and, in addition, is equal to K, the assumption on differentiability of o and b enables us to rewrite (1.1) as Ib(i,l 1 K. In this connection, we show that if the conditions under which Theorems 4.7.4, 4.7.5, and 4.7.7 on smoothness of a payoff function and, also, on the Bellman equation were proved, Assumptions 1-4 will be satisfied as well. In other words, we wish to show that if A , = A, = . . . = A, K I --K - . . . -- K, ml = m2 = . . . = m, there always exist functions ui and a number 6 which satisfy Assumptions 1-4. Here as well as in a similar situation in Section 1, our objective is purely methodological. In order to have (3) satisfied, we take 6 = 4,u, = e-N1t,and furthermore, we choose an appropriate N,. Since

1 0(!,1 +

for verifying Assumption 2 we need only to choose u- from the condition where N, is a constant. Repeating the arguments given after Lemma 1.2' in Section 1, we easily convince ourselves that for u-, we can take a function of the form N,e-N4'(1 Ix~')~".

+

In these arguments one needs to take 8m instead of m.


We see from Chapter 3 that 101,

IwI

IN(l

+ 1x1)". Since for u = v,w

the function u , for inequality (6)can be sought from the condition where N is an appropriate constant. It is seen that the function u,, equal to N,e-N6'(l + I X ~ ~ )is~ applicable ", for a certain choice of constants N , , N , . Similarly, to have (13) satisfied, it suffices to solve an inequality of the form

L",,

+ N ( l + lxl)8m1(2+6) I 0.

As we saw in Section 1, this can easily be done. Finally, choosing functions of the form N,eCN@(l+ IxI2)" which satisfy inequalities (8), (14), and next adding up all the expressions thus found for u,, we obtain the function to be taken as u, in inequalities (6),(8),(13),and (14)in the large. Since we had dealt with an expression of the form N,e-N1Ot(l+ IxI2)",each time, the final version of the function u, thus obtained satisfies the inequality luzl I N ( l IxI)", using which we can easily find the function u - , satisfying (7). Therefore, Assumptions 1-4 have been satisfied.

+

5. Exercise Summarizing the above arguments, show that in the case considered one can take

Also, we write down what Assumptions 1-4 become if we take constants for the functions ui and if, in addition, we consider only Assumption 3a. Here (3)becomes (15).Inequality (4) can obviously be satisfied if there exists a constant N such that In other words, it suffices to take a constant N for which Ilo(oa)ll Nc". (16) Considering the remaining inequalities in a similar way, we can see that Assumptions 1-4 can be satisfied if inequalities (15)and (16)are satisfied and, furthermore, if there exists a constant N such that


for all (t,x) E RT, a E A,, 1 E Ed, u = v,, w, (n = 1,2, . . .). In this case we can say that c" is sufficiently large and that the derivatives of the functions g(t,x), g(x) are bounded. On the face of it, the last inequalities in (17) as well as inequalities (6) and (13) seem to be rather strange. These inequalities contain the functions v, and w,, which are in general unknown. In this connection, we note that

Hence, if, for example, we could, using (1.5) and Lemma 1.2, estimate the functions v, and w, and also, we could prove that Iv,l I u" and lw,l I u" for a function u", (6) and (13) will be satisfied if

The last remarks concerning (6), (13), and (17) are unnecessary if ca(t,x) does not depend on x. In such a case c;, = 0 and c;,(,, = 0, the functions u, and w, do not belong to (6),(13), and (17),and, finally, these functions need not be estimated. This fact shows how convenient it is to write (6) and (13) as we did above. In the case where ca(t,x)does not depend on x, Assumption 3 is regarded as satisfied if we succeed in satisfying conditions (b). Let us discuss this case. Inequalities (11) and (12) are particular cases of inequalities (6) and (8) since c(,, = 0 and also since in (11) and (12) we consider not all 1" E Ed but only those which are orthogonal to a subspace Ef. We can posit two extreme possibilities: Ef contains only a zero vector; EF coincides with the entire space Ed. If the first possibility is realized, equalities (9) and (10) hold for arbitrary 0 and b, any vector 2" 1Ef, and (11) and (12) must be satisfied for all (t,x) E R,, a E Ad, 11' E Ed. Also, in this case conditions (a) are satisfied. The advantage of considering conditions (b) along with conditions (a) is obvious when Ef = Ed. Here only the zero vector is orthogonal to Ef; furthermore, inequalities (11) and (12) are automatically satisfied (u(,,= 0 for 1 = 0), as well as Eq. (9). Therefore, if c" does not depend on x, Assumption 3 is satisfied if, for example, for all 1', 1 E Ed(10) is satisfied; in this case we can assume Ef = Ed. We note that, as can easily be seen, (10)holds for all 1', 1 E Ed,a E A, (t,x) E 8, if and only if the functions a(a,t,x) and b(a,t,x) are linear with respect to x for all a E A, t E [O,T]. It is seen that Assumption 2 is satisfied as well. In order to consider the intermediate possibility 0 # Ef # Ed, let us imagine that the space Ef is generated by the first coordinate vectors el, e,, . . . ,e,,, where 1 I do < d. Equalities (9) and (10) have then to be


satisfied for the vectors 1' with arbitrary first do coordinates as well as coordinates numbered d, + 1, . . . , d, which are equal to zero. From this condition it readily follows that for all 1" IEF, 1 E Ed, i = 1, . . . , do; j = 1, ..., dl of:',,)(a,t,x) = 0, ~&(~)(a,t,x) = 0,

bf,..)(a,t,x)= 0, bil)cl,(a,t,x)= 0.

The second relations in (18) imply that the first do rows of the matrix o and the first do coordinates of the vector b depend on the coordinates of x in a linear way. The first relations in (18) show that the elements of the matrix o and the elements of the vector b do not depend on xdo+l,. . . , xd. Therefore, the system dx, = o(a,, s

+ t, xi)dw, + b(a,, s + t, xi)dt

splits into two parts: the "upper" part for the coordinates x: ( i = 1, . . . ,do) and the "lower" part for the coordinates x: (i = do + 1, . . . ,d), the "upper" system being linear with respect to x:, . . . , x? and, in addition, containing no other unknowns. We shall complete this discussion of Assumptions 1-4 by suggesting the reader should do 6. Exercise Show that Assumptions 1-4 can be satisfied if a, b, c do not depend on x and, also, for all (t,x)E RT,C( E A, 1 E Ed

[f ;)(,)I

-

5 ( - ~ ; ~ ~ ) ( 1 / 2 ) - ( 6 1 4 )~( -; ~ ~ ) ( 1 / 2 ) + ( 6 / 4 ) ,

[g(,)(,)(t,x)] - 4 u \ ' / ~ ) - (u2 ~( /1~/ 2) ) + ( d / 4 ) ( t , ~ ) , [g(,)(,)(x)] $ u\'/~)-('/~) u2( 1 1 2 ) + ( 6 i 4 ) ( ~ , x ) .

Note that the last inequalities are automatically satisfied iff", g(t,x), g(x) are downward convex with respect to x.

7. Assumption (on derivatives with respect to t). (a) The functions f , g, o, b, c are continuously (with respect to (t,x)) differentiable with respect to (t,x) for all (t,x) E H T , n, a E A,,

for all (t,x) E IT,, a E A


(b) Furthermore, for all (t,x) E H T , a E A,, u = v,, w, (n = 1,2, . . .)

(c) Finally, the derivatives gXixj(t,x),gxixj(x)are continuous in H , and, also, for all x E Ed

Much of what has been said above could easily be repeated for Assumption 7. We note only that in verifying that Assumption 7 is satisfied, one need not attempt to find immediately the function u-, which simultaneously satisfies inequalities (20) and (21) (as well as inequalities (4) and (5)). One can find one function for (20) and another function for (21), and, next, one can take as u-, the sum of the functions thus obtained. The same holds for uo from (22)-(24). Moreover, one should keep in mind that Lag(T,x) in (24) is the quantity at a point (T,x), obtained as a result of application of the operator La(t,x) to the function g(t,x), and, in addition, that La(T,x)g(x)is to be regarded as the value of the operator La(T,x)evaluated on the function g ( ~ ) .

8. Assumption (summarizing). Assumptions 1, 2, 3b, 4, and 7 are satisfied, also, for 1' E Ed, a E A, (t,x) E H T ,

or Assumptions 1,2,3a, 4, and 7 are satisfied. We see from the results obtained in Chapter 4 that derivatives of the functions v and w can be estimated only on the sets on which these derivatives belong to the operator F[u]. In this connection, here as well as in Section 4.7, let Q* = {(t,x)E HT:~~pasA(a(a,t,x),A,A) > 0 for all il # 01, p

= p(t,x) =

inf sup na(t,x)(a(a,t,x)A,A).

111=1 a s A

We also introduce Q,*and p, (n = 1,2, . . .) using similar relations in which we put A, instead of A. 9. Lemma. p, I ,u,+~,p = limn,, tinuous in H,, the equalities

pn, the function p(t,x) is lower semicon-


are satisjied, the set Q* is open, and, jinally, the function pP1(t,x) is locally bounded on Q*.

PROOF. The inequality p, Ip, + is obvious. Further, due to the boundedness of na(t,x)a(a,t,x),which is uniform with respect to (a,t,x),the quadratic forms na(t,x)(a(a,t,x)l,l)and the functions

are continuous with respect to A E S, uniformly with respect to a, t, x, n. In accord with Dini's theorem it follows from the obvious relation

that in this case the convergence is uniform with respect to l E dS1 for each t, x. In particular, the lower bounds of the foregoing expressions converge, that is, p,(t,x) -, p(t,x) as n -, 0. Since the limit of an increasing sequence of continuous functions is lower semicontinuous and, furthermore, by Lemma 4.7.3 the functions p,(t,x) are continuous, p(t,x) is lower semicontinuous. The equality Q* = UQ,* follows from the fact that p,(t,x) f p(t,x). We can prove the other equality fc . d* by repeating the appropriate arguments from the proof of Lemma 4.7.3. Finally, the last assertions follow from the continuity of p.(t,x) and also from the monotone convergence of p,(t,x) to p(t,x). The lemma is proved. We formulate now the main results obtained in estimating derivatives of the functions v and w. It should be borne in mind that we shall give some assertions of Theorems 10-12 in a short form, in other words, we assert, for example, in Theorem 11 that for each x E Ed the function v(t,x) is left continuous with respect to t on [O,T], and that the function

increases (does not decrease) with respect to t on [O,T]. We fix x E Ed. The function $(t,x), as any increasing function, can have not more than a countable set of discontinuity points; all the discontinuities are of the first kind and, in addition, $(t - ,x) 5 $(t,x) s $(t + ,x) for each t. Since an integral with respect to r is a continuous function oft, we conclude that for each x E Ed the function v(t,x), being the function oft, can have not more than a countable set of discontinuity points, all of whose discontinuities are of first kind, and, in addition, that v(t,x) I v(t + ,x) for each t, that is, the graph of v(t,x) can have upward jumps only. The properties of v(t,x) listed above do not appear in Theorem 11. A similar situation is observed for the corresponding assertions of Theorems 10 and 12. In Theorem 10, the appropriate auxiliary function decreases, and hence v(t,x) has downward jumps only. Finally, we note that from the inequality $(t-,x) 4 $(t,x) follows the inequality v(t -,x) 4 v(t,x). Furthermore, by Theorem 1.5, the function v(t,x)


is lower semicontinuous. Therefore, v(t,x) I v(t-,x). Comparing the last inequality with the previous one, we obtain v(t,x) = v(t - ,x). Thus, the assertion about the left continuity of v(t,x) with respect to t in Theorem 11 is a consequence of Theorem 1.5 and of the fact that $(t,x) increases with respect to t. 10. Theorem. Suppose that a function v is bounded above in each cylinder

C,,, and also that Assumptions 1-4 are satisJied. Then: a. for each I,, 1, E Ed inside HT there exist generalized derivatives ~ ( ~ ~ ) ( ~ ~ , dx) ( t , x(see ) ( dDefinition t 2.1.2), and also for each 1 E Ed inside HT where

b. for each t E [O,T] the function v(t,x) is continuous with respect to x, has a generalized derivative with respect to x (in the sense of Definition 2.1.1), and, in addition, for each R > 0 , y E (0,1],t E [O,T] almost everywhere in S,

c. inside HT there exists a derivative (d/at)v(t,x)(dtdx),

a

- v(dtdx) I

at

inf ILBl(ti, + Igrad, vl

BsA

L B V ( ~dx) ~

+ v+ + 1)dt dx,

+ fp dt dx I o3

for each fi E A ; for each 1 E Ed ,uv(,,(,,(dtdx) I (Dl

+ Igrad,vl + v+ + 1)dt dx +

(a4 )-

v

(dt dx);

d. for each x E Ed the function v(t,x) is right continuous with respect to t on [O,T], and, furthermore, for each R > 0 there is a constant N 2 0 such thatfor all x E SR thefunction v(t,x)- N t decreases with respect to t o n [O,T]. Finally, if we replace v by w in the above formulations, assertions (a)-(d) will still hold.

11. Theorem. Let Assumptions 1, 3b, and 7 be satisfied; in this case for all 1' E E f , a E A, (t,x) E HT let equalities (25) be satisfied. Or let Assumptions 1,3a, and 7 be satisfied. Then: a. the function v(t,x) is bounded in each cylinder C,,,, for each x E Ed the function v(t,x) is left continuous with respect to t on [O,T], and, finally, The meaning of ILBJis explained in the statement of Theorem 4.3.3; L6u(dt dx) is defined prior to Theorem 4.2.7.


for each x E Ed the function v(t,x)

+ Sd ii2(r,x)dr

increases with respect to t on [O,T], where ii2= (76-1u-1u2)112+ u,; b. inside HT there exists a derivative (a/dt)v(dtdx) and

a u(t,x)(dtdx) 2

-

at

-

ii2(t,x)dt d x ;

c. for each R 2 0 lim sup Iv(t,x) - g(x)l = 0. ttT IxlsR Assertions (a)-(c) will remain true i f we replace in them v by w and g(x) by g(T,x). The next theorem holds i f the assumptions given in the preceding section only are satisfied. W e could have formulated it in Section 1; however, it is more closely related t o the discussion in this section.

-

12. Theorem. Let there exist a continuous function q(t,x) given on RT,and also a number 6 , > 0 such that for each t E (O,T] the function ut(s,x) v(t,x) + 1: q(r,x)dr, being a function of the variables (s,x),is superharmonic (or excessive) in the region HT n ( t - 6,,t) x Ed (see Dejinitions 5.3.2, 5.3.3). Then, we can prove Theorem 1la,b,c i f we replace ii2 by q.

13. Theorem. Let the summarizing Assumption 8 be satisjied. Then the functions v,, v, w, and w are uniformly bounded and also equicontinuous in each cylinder C,,,. Generalizedjirst derivatives of these functions with respect to (t,x) are uniformly bounded in each cylinder C,,,. Moreover, each of the foregoing functions satisJies the inequalities -Hz

a

I -u

at

I inf ILBl(iTl pea,

+ Igradxul + u+ + 1)

for any R > 0, y E (0,1]. Finally, v, f v and w, 7 w as n C,,, for each R > 0.

(HT-a.e.),

+

oo uniformly in

14. Theorem. Let the summarizing Assumption 8 be satisjied. Then in the region Q* the functions v(t,x) and w(t,x) have all generalized second derivatives with respect to x, which are locally bounded in Q*.Moreover, for each 1 E Ed


almost everywhere in Q*. Finally, for each no second derivatives of the functions v, and w, with respect to x for n 2 no are un$ormly bounded with respect to x and, also, n 2 no in each bounded region which together with its closure lies in Q,*,.

Theorems 10-12 are of an auxiliary nature in relation to Theorems 13 and 14. The former theorems can in turn be derived from the following estimates of derivatives of solutions of stochastic equations. For a E %, 111 = 1: (s,x)E RT let

Since by the definition of the set % it follows from the inclusion a E % that a E %, for a certain n, due to the results obtained in Section 2.8 the processes yFSsX,zFSgX exist and are continuous on [O,T]. Let

This process is defined for a E IU, (s,x) E HT on a time interval [0, T-s] if assumption 7a is satisfied.

15. Theorem. Let Assumptions 1 and 2 be satisjied. Then for each (s,x) E H T , z~!lX(T-s),a~%

Furthermore, for each t , E [0, T - s], y E [O, 8/(2 - 6)] on a set {z 2 t1) almost surely 26- l U l ( $ + t l , x;l.s.x)ly;;s?xl~e-2(1 -6) a cpt;"'"

ul(s

+ z, ~ J 1 y ~ l ~ e - ~ ' ~ - ' " + "


16. Theorem. Let Assumptions 1 and 7a be satisjied. Then for each (s,x)E H,,

z~1131(T-s),a~%

17. Exercise Making use of Exercise 5 and Theorems 13 and 14, prove Theorem 4.7.4 and, also, that in that theorem

3. The Normed Bellman Equation As is seen from Section 1.2 (also, see below Exercise 15), a payoff function need not satisfy the Bellman equation. In this case the initial functions o, b, c, f, g may be as smooth as desired. The objective of the present section consists in deriving a correct normed Bellman equation for a payoff function. We shall see that such an equation holds in a very broad class of cases. In addition to the assumptions given in Section 6.1, let us assume the following. We denote by ya(t,x)the vector of dimension d x (dl + d + 4), having the coordinates oij(a,t,x) (i = I , . . . ,d, j = 1, . . . ,dl),bi(a,t,x) (i = 1,. . . ,d), ca(t,x),fa(t,x), g(x), g(t,x). We assume that for each a E A, 1 E Ed the derivatives y:,(t,x), y;l,(,,(t,x) as well as the derivative (d/dt)ya(t,x)exist and are continuous on 8,. Assume that for all n, a E A,, 1 E Ed, ( t , ~E )

Let

I

1 1

Q,* = (t,x)E HT:sup (a(a,t,x)1,1)> 0 for all 1 # 0 , asA,

(t,x)E H,:SUP (a(a,t,x)1,1) > 0 for all 1 # 0 . a sA

By Lemma 2.9 the sets Q* and Q,*are open, Q,*+,2 QX, Q* = U, Q,*.To state our last assumption we assume that for each no and for each bounded

3. The Normed Bellman Equation

region Q' which together with its closure lies in Q,*,there exists a constant N such that for all n 2 no,j = 1,. . . , d

+

+ I;t

IvnXil 1vnXixjl

-

,I-

v 0 be so small that U = {u: < r} c r. Then G(u)l0 in U, G(0)= G(uo)= 0. We choose a sequence ai such that na,,hal + 0. This can be done since G(0)= 0. The vectors n,,,la' lie in a unit sphere. Hence, one can choose from the sequence of the vectors n,,,h". a convergent subsequence. We denote the limit of this sequence by e. Since ln,01a12 + lnaoha12= 1 for all a E A and since na,,P1 + 0, = 1. Finally, it follows from the inequalities n,,ola'u na,,hal I G(u)I 0 which hold for all u E U that eu I 0 for all u E U. However, if we take u = (l/r)e,we obtain e2 I 0, which is impossible because of the equality leI2 = 1. The contradiction thus obtained proves the lemma.

IuI

112

+

We fix (t,x) E H T , some function ma(t,x)> 0, and also we use this lemma in order to study the inequality Gma(uo,uij,ui,u,t,x) 5 0. It is natural to regard the set u = (uo,uij,ui,u)(i,j,= 1, . . . ,d) as a point of the Euclidean space Ed,, where d2 = 2 + d + d2. Let

Then, as can easily be seen,

The function Gma(u,t,x)is therefore good enough to replace F(u). It is seen that if mu > 0, the function naofrom Lemma 6 is equal to [ma(t,x)]-'mu0(t,x), which fact indicates that in the given case the function G(u)from Lemma 6 is equal to Gmao(u,t,x). Using Lemma 6, we immediately arrive at the following. 7. Lemma. Let (t,x)E HT and let some function ma(t,x)> 0 for all a E A. Then the set rma(t,x)= {(u~,u~~,u~,u):G~~(u~,u~~,u~,u,~,x) 10 ) being a set in E2+d+dz, is convex, closed, and, in addition, the set is the boundary of the former set. 8. Lemma. Let ma(t,x)be a normalizing multiplier. Then all solutions of the = 0 ( 2 0 ) are also solutions of the equation (inequality) GmaO(uo,uij,ui,u,t,x) equation (inequality) Gma(uo,uij,ui,u,t,x) = 0 ( 1 0 ) If m,(t,x) is a regular normalizing multiplier, the converse also holds, that is, the foregoing equations (inequalities) are equivalent.

PROOF.The assertions about the above inequalities easily follow from the fact that mao(t,x)> 0, and, further, if ma(t,x) is a regular multiplier, then m,(t,x) > 0.


Let us prove the first assertion of the lemma about the equations. We fix some t, x; but we do not write t, x in the arguments of the functions. By Lemma 7, in any neighborhood of a point (ub,u;,u:,ul) such that GmaO(ub,u:j,u:,u') =0 there are points (uo,uij,ui,u)at which F > 0. It is seen that Gmu2 0 at the same points. By Definition 1, the function Gmais finite. Furthermore, since the function Gmais convex, it is also continuous. Comparing the last two assertions, we conclude that Gmw(ub,ufj,u:,u') 2 0. The converse inequality is obvious, therefore Gma(ub,u~j,u:,u') = 0. We have thus proved the first assertion. In order to prove the second assertion of the lemma, we note that if Gmu(ub,u~,u:,u') = 0, the expression appearing in (5) under the sign of the upper bound is negative. Therefore, having taken N = N(t,x) from Definition 1, we have On the other hand, it follows from the equality Gmm = 0 that GmeI 0 and (since mu > 0) GmaO 5 0. The lemma is proved. The comparison of Lemma 5 with Lemma 8 immediately proves Theorem 3. Lemma 8 shows that all the information on payoff functions, which can be obtained from the normed Bellman equations having different m,(t,x), is contained in an equation corresponding to the normalizing multiplier m,,(t,x). Despite the fact that ma0(t,x)plays therefore a particularly essential role, it is frequently convenient to consider in practice other normalizing multipliers. The second assertion of Lemma 8 shows that the application of regular normalizing multipliers induces no loss of the information on payoff functions. 9. Example. Let d = 1, A = (-co,co), a(a,t,x) = 1, b(a,t,x) = 2a, cu=O, f"(t,x) = - a2f (t,x),where f (t,x) > 0. In this case

The equation GmzO = 0 seems to be, at least at first sight, inconvenient. Let m,(s,x) E 1. Since f > 0, 2au1 - a2f is the bounded function of a and, furthermore, 1 is a normalizing multiplier. Obviously, this multiplier is even regular. The calculations of G1 reduce to finding the vertex of the parabola 2aul - a2f. We have Therefore, in this case the equation G1 = 0 (equivalent to the equation Gmao= 0)becomes Uo u,, u: f -1'2(t,~)= 0.

+

+


It seems much more complicated to transform the equation GmaO = 0 to the equation given above when we do not use Lemma 8.

10. Exercise Prove that the function rn,(t,x) r 2 0,t~ [O,T),x€ Ed

E

sup {r[tr a(a,t,x) UEA

1 is a normalizing multiplier if and only if for any

+ Ib(cc,t,x)l + P(t,x)] + f"(t,x)) < co.

Turning again to Lemma 7, we see how the function Gmabehaves on a boundary of the set T. If the point (~b,u:~,uf,u') E T o and if, in addition, the function Gmuis finite in some neighborhood of the foregoing point, here as well as in the proof of Lemma 8 the function Gmais continuous at the point (~b,u~~,u:,u'), and, also, in any neighborhood of this point there are points at which Gma> 0 and Gmu(ub,u:j,u:,u',t,x) = 0. It is also possible that Gma(ub,u~j,u:,u',t,x) I0, and that in any neighborhood of the point (~b,u;~,u~,u') there are points at which GmQ= + a.

11. Definition. We say that at the point (~b,u:~,u:,u') the function G m a ( ~ o , ~ i j , ~(t, i ,x~are , t , fixed) ~ ) has a zero crossing if: = 0, the function Gmu is continuous at the point a. Gmu(ub,u:j,~:,~',t,x) ( u ~ , u ~ ~ , u ~and, , u ' )furthermore, , in any neighborhood of the point (ub,ujj, u:,u') there are points at which Gmu> 0 ; or b. G m u ( u b , ~ : j , ~ ~ ,5~ '0, , t ,and x ) in any neighborhood of the point (~b,u~~,u:,u') there are points at which Gma= + oo.

Recall that F(uo,uij,ui,u,t,x)= sup uo + asA

d

d

C

i,j=l

aii(a,t,x)uij

+ C bi(a,t,x)ui i= 1

It is seen that F(uo,uij,ui,u,t,x)= G1(uo,uij,ui,u,t,x). Combining what has been said prior to Definition 11 with the assertion of Lemma 7, we have that the function F ( u ~ , u ~ ~ , u ~ ,has u , ~a, xzero ) crossing at each point of the set To(t,x). Then, due to Lemma 5 the following theorem holds. 12. Theorem. (a) For almost all (t,x)E Q* the function F(u0,uij,ui,u,t,x),being the function of (uo,uij,ui,u),has a zero-crossing at the point


(b)FLw] I 0 (Q*-a.e.), w(t,x) 2 g(t,x) everywhere on Q*, and for almost all (t,x)E Q0 the function F(uo,uij,ui,u,t,x), being the function of (u0,uij,ui,u), has a zero crossing at the point ((d/dt)w(t,~),~~~,~(t,~),~~~(t,~),~(t,~)).

This theorem enables us to make the Bellman equation meaningful even if the equation cannot be satisfied. Indeed, we write if F crosses zero at the point (uo,uij,ui,u).Then assertion (a) in Theorem 12 implies that From Lemmas 5 and 7 we have the following. 13. Theorem. (a) For almost all (t,x)E Q* the point

lies on the boundary To(t,x)of the set T1(t,x). (b)For almost all (t,x) E Q0 the point

lies on the boundary To(t,x)of the set T1(t,x).

Let us indicate here an example of the application of Theorem 12. 14. Example. d = 1, A = [ o , ~ )f"(t,x) , = ag(t,x), La = ( a p t )

The Bellman equation is here the following: sup

;[

UE[O,~)

v

I

+ vxx + a(g(t,x)- v)

+ (d2/ax2)- a.

= 0.

This equation may be not satisfied. Theorem 12 provides the correct interpretation of Eq. (11).According to Theorem 12 the function

has a zero-crossing at the point ((d/dt)v,vx,,v,,v). According to Definition 11 there exist two ways for a function to cross zero. The first way implies that

and, furthermore, for any (uo,u,,,ul,u) close to ((a/at)v,v,,,v,,v) the function F(u0,u1,,u,,u,t,x) is finite. It immediately follows from (12) that g(t,x) 5 v,


+

(a/at)v v,, = 0. Since any small variation of v in (12) need not make the left side of (12)go to infinity, g(t,x) < v. Therefore, the first way for a function to cross zero implies the relations

a

-v+v,,=O at

to be satisfied. In the other case, i.e., on the set (t,x) where (13) is not satisfied, the left side of (12) does not exceed zero, and, furthermore, some arbitrarily small variations of (a/at)v,v,,, v make the left side of (12)go to infinity, which is possible only if g(t,x) = v. The inequality F[v] < 0 yields (d/dt)v + v,, I 0. Therefore, either (13)is satisfied or

a + v,, < 0. at

-v

-

It is clear that (13) or (14)hold almost everywhere in H , rather than for all (t,x) E HT.We also note that by Theorem 12 the function w(t,x) constructed on the basis of g(t,x), f"(t,x) 0, La = (dldt) (az/ax2)satisfies either (13)or (14)almost at each point (t,x). We can obtain (13) and (14) in a somewhat different way if we take the normalizing multiplier ma(s,x)= 1/(1 a) and, in addition, if we let P = a/(l a). Then the equation Gma[v]= 0 can be written as

+

+

+

The expression under the sign of the upper bound is a linear function of

p. Using the fact that a linear function on an interval attains the upper

bound at either end point of the interval, we immediately have

which precisely means that either (13)or (14)is satisfied. 15. Exercise Show that in Example 14, Eq. (11) cannot be satisfied almost everywhere in HT if g(T,x) = 0 and, also, if g(t,x) is a function which is bounded and continuous in RT and satisfies the inequality g(t,x) > 0 for (t,x)E H,.

16. Exercise Let f"(t,x) 2 0 for all a, t, and x, and let m,(t,x) be a normalizing multiplier. Prove that there exists a function N(t,x) such that m,(t,x) 5 N(t,x)m,,(t,x) for all a, t, x.

4. The Optimal Stopping of a Controlled Process

17. Exercise Prove all the theorems given in Section 6.3 in the case where, in the assumption associated with inequalities (2)-(4), inequalities (2) and (3) are replaced respectively by (a.e. on Q').

4. The Optimal Stopping of a Controlled Process on an Infinite Interval of Time In this section we investigate the limiting behavior of a payoff function as T + co in the problem of optimal stopping of a controlled process. We assume that the basic inequalities (1.1)-(1.4) are satisfied for all a E A, in each trip RT for constant K,, m,, K, m, which depend in general on T. We vary T as well as the sets Nn and N and also the functions w,(t,x) and w(t,x) (see Section 1) associated with the controlled process in HT. Hence, it is natural to attribute the index T to a n , 8 , w,, w, defined in Section 1. Thus, we consider the sets of strategies N;, NT as well as the functions w,(T,t,x), w(T,t,x). In addition to the assumptions that Kn = Kn(T),m, = m,(T), K = K(T), m = m(T), we state and use some other assumptions which will be given after Exercise 11 in this section. We denote by 1)31 the set of all Markov times with respect to {Ft}, N= NT. It is seen that N is the set of all progressively measurable (with respect to {Ft}) functions a, having values in A, for each of which for T > 0 there is a number n such that a,(o) E A, for all t I T, o E 52. Obviously, 1)31(T - s) 2 1)31(T1- s) for T 2 T'. Therefore,

nT

w(T,s,x) = sup

sup

va3'(s,x)

aeBT r ~ m ( T - s )

If z E 1)31(T' - s), the values of a, for t > T' (even for t > T' - s) are of no consequence for the computation of va*'(s,x).This implies that the right side of (1) is equal to SUP V~'~(S,X) = w(T1,s,x). sup aeBT' r ~ m ( T ' - s )

Therefore, w(T,s,x) increases with respect to T and has the (possibly, infinite) limit lim w(T,s,x) = w(s,x). T+m

It is seen that for s I S w(s,x) = sup w(T,s,x) = sup sup w,(T,s,x). TtS

T2S

n


It is clear that the function w(s,x) is lower semicontinuous with respect to (s,x) as the upper bound of continuous functions w,(T,s,x). In particular, w(s,x) is a Bore1 function. It is seen that for arbitrary a E %, r E 1)32 M:,{J~

e-'P(s

+ t, x,) dt + e-'T X

~

u,(T',x) uZ(T',X)' E ~

-

Under assumption (a), E ( T )L 0 as T + CQ. Using Lemma 1, we find M:,,x,~T-,u~(s

+ ~,x,)e-" I e(T)M;,,u,(s + ~ , x , ) e - ~ ' I ~(T)u,(s,x).

We have thereby proved equality (4).By Lemma 1

if (b) is satisfied. Since the integral with respect to t converges in the last equation, the integrand tends to zero. Therefore, (5) is satisfied. This completes the proof of the lemma. 3. Remark. Lemma 2b will be satisfied if, for example, u, 5 Nlca for all a for some constant N,. In this case we may take u2 = N,. Lemma 2a will


+

be satisfied if u , E W:d,2((0,co)x Ed) and Lau, E U , I 0 ((0,co) x Ed-a.e.) for all a for some E > 0, in which case the function u,(s,x)esEcan be taken for u,. It is convenient to use Lemma 2 in verifying the next theorem. 4. Theorem. Let s 2 0, x

E

Ed and, also, for each a E % let

Then w(s,x) =

sup

(a,r)s IW x(lRl(s.x)

vaJ(s,x).

PROOF. It is seen that

W(S,X) = lim sup

sup va,'(s,x) r~m(T-s) lim sup sup va~'(s,x)

T - t m asWT

=

T - t m a s % rs(lR(T-s)

5 sup

T > 0 (a,,)€

SUP

1%

v ~ , ~ ( s= ,x)

x(lRl(s,x)

(@,I)€

sup

V~'~(S,X).

IN x (DII(s,x)

Let us prove the converse. To this end, as can easily be seen, it suffices to show that for all (a,z)E 1% x )IJZI(s,x) By definition, g(s + z, x,) convergence

=0

if z

=

co. Then, by the theorem on monotone

Similarly,

-

lim T-tm

va. 1 ~ G - s )

-

(~5x1.

Subtracting the last inequalities, we find (6). The theorem is proved.


5. Exercise Consider the process x, = -ewc-('12)twhere w, is a one-dimensional Wiener process. The process x, is a solution of the stochastic equation

Prove that sup Mx,

ZCDI

= 0,

sup Mx,

reFm(T)

= -1

for each T > 0. This exercise shows that the assertion of Theorem 4 is false, if we impose no restrictions on the controlled process.

Theorem 4 answers the question as to when a payoff function in the problem of optimal stopping of a controlled process on an infinite time interval can be obtained as the limit of payoff functions associated with finite time intervals. The presence of the set 1% x %RI(s,x)in the statement of this theorem is rather inconvenient. In this connection we consider the case where 1% x %Rl(s,x)= 2l x %R.

6. Theorem.Let there be nonnegative functions a,, W 2 E W:?((O,co) x Ed) continuous in [O,W) x Ed and such that a, 2 g in [O,W) x Ed and, in addition, for all a E A let

+

Then 1% x %Rl(s,x)= 2l x %R for each s, x. Furthermore, vT;',(s,x)s W1(s,x) W2(s,x)for all a E 2l, z E %R, s 2 0, x E Ed;w IW 1 W 2 , and also the function w is Jinite.

+

The assertions of this theorem follow immediately from the inequalities

Here the second inequality has been obtained by Lemma 1. 7. Exercise Let p, q > 0, and let w, be a d-dimensional Wiener process w(T,s,x)=

sup

M

ZEW~DI(T-S)

In this case a denotes a unit matrix, b Using Theorem 4, show that

IX

+ YIP (l+S+'T)"

= 0, c = f = 0, g(s,x) =

w(s,x) = sup M TEW

Ix + w.Ip

(1

+ s + 7)'.

IxlP/(l

+ slq.


Prove that w(s,x) = co for p 2 2q and w(s,x) < co for p < 2q for all s, x. (In the first case use the law of the iterated logarithm; in the second case use Theorem 6, assuming

w 2 = 0,

and choosing an appropriate N.)

Theorem 4 together with the equality w(s,x) = lim lim w,(T,s,x) T + m n+m

as well as the results obtained in Chapter 5 enables us to find strategies and Markov times for which va"(s,x)differs arbitrarily slightly from a payoff function in the problem of optimal stopping of a controlled process on an infinite time interval. In some cases, one can indicate an optimal stopping time. 8. Theorem. Let a set A consist of a single point. Assume that there exists a nonnegative function iij E W:d,2((0,m) x Ed) continuous in [O,m)x Ed and such that

Ig(T7x)l - 0, lim sup ----~ ~E( T~ , X )

T+m X

Also, we assume that for each s 2 0, x is satisfied:

L* I O5 ((0,m) x Ed)-a.e. E Ed either

of the following inequalities

We denote by z y the time of first exit of the process (s + t, xi*")from Qo = {(s,x) [O,m) E d : W ( S , X ) > ~ ( s , x ) ) . Then, for each z E 1)32 the variable vE(s,x)is defined, vT(s,x)< m(vZ(s,x)> - m),if the first (second) inequality (7) is satisfied, W(S,X)

for all (s,x).

= sup vT(s,x), rsm

PROOF. First we note that, since the function w(s,x) - g(s,x) is lower semicontinuous, the set Qo is in fact a region. Further, by Lemma 2

as T

+

m . From this it follows, in particular, that for each z E

for large T

+

Ms,xe-(P'~T> T-slg(~ z, x,)I < a. We omit here and hencefbrth the index cc, which assumes only one value; we assume that the expression O00is equal to zero.


Furthermore, since lg(s,x)l s K ( T ) ( l + /xl)m(T) for s 5 T, due to the estimates of moments of stochastic equations or any T we have Ms,xe-(Pz~r5 T - s l ~ (+~ 7 , xT)I< a.

Therefore, for any z

+

~ ~ , , e - * ~ l g (2, s xT)l< co.

It is also seen that

The last two inequalities immediately imply our assertions concerning vT(s,x). Now we can write 1% x %Rl(s,x)= % x %R. Further, by Theorem 4, w(s,x)= sup vT(s,x). rem We prove that the time z y is optimal. We denote by z y ( T ) the time of first exit of (s t, x,) from Q,(T) = {(s,x)E RT:w(T,s,x) > g(s,x)). By Theorem 3.1.10 for each T on RT

+

wT(s,;) = M,,,{J~

ro(T)

-

e .Y(s

+ t, x,) dt + e - " ~ ( ~ ' g (+s r,(T), xr0(,)

Let T + co.By virtue of the relation w(T,s,x) w(s,x)we have Q,(T) c Q,(Tf) for T' > T , Q , = Q,(T), z y ( T ) f z y The continuity of x;," with respect to t also implies that on a set {z;" < co)

UT

lim x:+(~)= x:p.

T+ w

By the monotone convergence theorem, M,,

So

ro(T)

-

e * Y b + t, xd dt = M , ,

So

ro(T)

e *Y+(s+ t, xd dt -

Applying here the estimates of solutions of stochastic equations as well as the dominated convergence theorem, we conclude that for each T'


As T' -+ co the last expression tends to zero by virtue of (8). By the dominated convergence theorem, it follows from (9) that

+

lim ~ s , x ~ - " ' o ~ , o , T ~ - sz0,x,,) g(~

T'+ ao

= Ms,,e-"=og(s,

+ z,, x,,).

Thus, letting T' in (10) to infinity, we obtain: w(s,x) = v70(s,x), which proves the theorem. However, the condition associated with (7) cannot be removed.

9. Example. Let c = 0, f (s,x) = (d/ds)(ssins), g(s,x) = 0. Then, as can easily be seen, w(s,x) = co, z, = co,and, in addition,

is not defined.

+

10. Example. Let c = 0, f = 0, g(s,x) = - 1/(1 s)'. In this case we can take 1/(1+ s) for W(s,x). Obviously, w(s,x) = 0, 7, = co, and, in addition, by Theorem 8,

11. Exercise Let us take p < 2q in Exercise 7. Show that in such case one can take W l given in Hint to Exercise 7 for iij in Theorem 8. Using the fact that the process fi(,,,, is a Wiener process for any constant c > 0 (self-scaling property of a Wiener process), prove that

Deduce from this, noting that

and also applying the fact that the problem is spherically symmetric, that. Qo together with each point (so,xo)contains a part of the paraboloid

which lies in [O,oo) x Ed. Using the almost obvious inequality w(s,x) 2 (1 By definition.

+ so)4(1+ s)-(w(sO,x)

for s 2 so


prove that if (so,xo)E Qo (i.e., w(so,xo)> g(so,xo)),then (s,xo)E Q0 for all s 2 so. Combining this result with the preceding one, we arrive at the assertionthat for some constant

Prove that w(s,O) > 0, and then Qo # (a, c, > 0.

We proceed now to derive the normed Bellman equation. In addition to the assumptions formulated at the start of this section, we shall impose the following conditions. Let ya(s,x) be a vector of dimension d x (dl + d + 3) having coordinates aij(a,s,x) (i = 1, . . . ,d, j = I, . . . ,dl), bi(a,s,x) (i = 1, . . . ,d), ca(s,x), f"(s,x), g(s,x). Assume that the vector ya(s,x) is one time continuously differentiable with respect to s, twice continuously differentiable with respect to x on [O,m)x Ed for each a~ A, and for all n = 1,2,. . . , a € A,, 1 E Ed, T > 0, (s,x)E RT

Let QX(T) = {(s,x)E HT:sup (a(a,s,x)A,1) > 0 for all A # 01, asAn

Q*

=

{(s,x)E ( 0 , ~x) Ed:sup (a(a,s,x)A,1)> 0 for all 1 # 0). aeA

We assume that for each no, T o > 0 and for each bounded region Q' which together with its closure lies in Q:,,(To), there exists a constant N such that for all n 2 no, T > T o , i, j = 1, . . . ,d almost everywhere in Q'

Thus, we assume that the assumptions made in Section 3 are satisfied in each strip HT.In this connection we note that we can obtain the required estimates of derivatives in various cases if we apply the results obtained in Section 2. Finally, we introduce the concept of a normalizing multiplier in the same way as we did in Section 3, letting T = co in Definition 3.1.

12. Theorem. The function w(s,x) in the region Q* is continuous, has a generalized jirst derivative with respect to s, and two generalized derivatives with respect x ; all these derivatives are locally bounded in Q*. For any normalizing multiplier ma(s,x) we have Gma[w]< 0 (Q*-a.e.), w(s,x) 2 g(s,x) in the region Q*, Gmu[w]= 0 (QO= {(s,x)E Q*:W ( S , X ) > g(s,x))-a.e.). The proof of this theorem follows the proof of Theorem 3.3b. Hence we restrict ourselves to some hints only. One can easily derive the assertion of


the theorem for an arbitrary normalizing multiplier from the assertion for m,(s,x) = mao(s,x)with the aid of Lemma 3.8. According to Theorem 3.3 this theorem holds if we replace w by w ( T ) = w(T,s,x), and, furthermore, if we replace Q* by Q*(T) = U, Q,*(T). By Theorem 4.5.1, it is possible to = 0 and in inequalities of take the limit in equalities of the type GmaO[w(T)] the type Gmao[w(T)] I 0 as T -t KI.

13. Remark. As in the previous section assertions of the type of Theorems 3.12b and 3.13b hold here. The translation of the preceding arguments requires no changes. We consider an important particular case of the problem of optimal stopping of a controlled process on an infinite time interval where the functions o(a,s,x), b(a,s,x), ca(s,x),f "(s,x),g(s,x) are time homogeneous, i.e., they do not depend on s. In this case for any a E IU and for bounded z E 1111 the function va~'(s,x)does not depend on s. In order to convince oneself that this is the case, it suffices to write (2) in an explicit form and to note that x~"~ does " not depend on s. It follows from the equality w(T,s,x) = sup

sup

a ~ 2 IreiUl(T-s)

va,"(s,x)

that w(T,s,x) depends only on x and T - s. This, in turn, implies that w(s,x) = lim,,, w(T,s,x) does not depend on s. It is clear that if ma(s,x)= ma(x),then G m a ( ~ o , ~ i j , ~does i , ~ not , ~ , depend ~) on s. Here we put Gm,(uij,ui,u,x)= Gm~(O,uij,ui,u,s,x), Gma[u]( x )= Gm~(~,ixj(~),~xi(~),~(~),~). It is natural that we omit now the argument s in the functions which do not depend on s in the case considered. 14. Theorem. The function w(x) in the region

(a(a,x)I,A) > 0 for all I # 0 is continuous and, furthermore, has two generalized derivatives with respect to x, which are locally bounded in D*. In addition, if the nonnegative function ma(x)is such that for all x, uij,ui, u sup ma(x)< KI,

Gma(uij,ui,u,x)< oo,

(13)

u

then Gme[w]5 0 (D*-a.e.), w(x) r g(x) in D*, P a [ w ]= 0 (Do = ( x E D*:

W ( X ) > g(x))-a.e.).

This theorem follows from Theorem 12 since, obviausly, Q* = ((),a) x D*,

Q0 = (O,CO) x Do,

a

-

as

w(x) = 0

5. Control on an Infinite Interval of Time

and, by virtue of the inequality

+

Gma(u0,uij,ui,u,x) 4 luOlSUP ma(x) Gma(uij,ui,u,x)< oo, a

the function m,(s,x)

= m,(x) is a normalizing multiplier.

15. Remark. In the stationary case one can write K , and m, instead of K , ( T ) and mn(T)in ( l l ) ,since the vector y does not depend on s. In verifying (12) using the results obtained in Section 2, it is natural to seek the functions ui depending only on x. The same is applicable to the functions w, and w2 from Theorem 6. 16. Exercise Returning to Exercises 7 and 11, we assume that p < 2q. Show that we can change the function g(s,x) inside the region {(s,x):s2 0, 1x1 < (co/2) in such a way (mainly, smooth out) that the payoff function does not change and, furthermore, Assumption 2.8 can be satisfied for u, = u , = u, = W , (w, given in Hint to Exercise 7), u - , = u - , = 0 for each T. Derive from the above that w(s,x) has one generalized derivative with respect to s and two generalized derivatives with respect to x, these derivatives being locally bounded in ( 0 , ~ x) Ed,

m)

where h = 0 in QO,h = - (aglas)- $ Ag outside QO.Regarding (14)as an equation with respect to w and noting that w I(1 + for some E > 0, prove that w(0,x) = M

Putting here x

= x , = (c,,O,

Jow

h(t,x + v,)dt.

. . . ,O),

and using the fact that w(O,xo)= g(O,xo)and also, that the distribution of w, is known, write an equation for c,. Prove that such an equation has a unique solution with respect to c,.

5. Control on an Infinite Interval of Time Let the functions o(a,t,x), b(a,t,x), ca(t,x),fa(t,x) be given for all cc E A, t 2 0, x E Ed. Also, in each strip of H T let the functions given satisfy Assumptions (1.1)-(1.3), with constant K , and m,, which depend, generally speaking, on T . As in the previous section, we introduce here the sets of strategies N$, 'UT = U, U ',; 'U = ( U T . Let

nT

v,(~,s,x)= sup M,:

JOT-'

ae%,T

v(T,s,x) = sup v,(T,s,x). n

e-'Yat(s

+ t, x~ dt,


The objective of this section consists in investigating the limit behavior of v(T,s,x) as T + co.We denote by I%l(s,x) the set of all a E '3 for which at least one of the following expressions is finite:

For a E I%I(s,x)let va(s,x)= v?+)(s,x)- v:-)(s,x). Throughout this section we assume that there exists a E A and also, that u(s,x) andgl(s,x)are nonnegative continuous functions given on [O,co) x Ed, which belong to Wfd,2((0,co)x Ed)and such that

,--

v(T7x) lim sup = 0, X E E (T,x) ~~!

T-tm

where a relation of the form 010 is assumed to be equal to zero, and, in addition, for all a E A Lavl 0 ( ( 0 , ~x )Ed-a.e.). This assumption was discussed in Remark 4.3. Furthermore, after Theorem 4 we add some conditions to those listed above. 1. Theorem. For all s 2 0, x V(S,X)

-- lim v(T,s,x), T+ oo

E Ed we

have

v(s,x)2 --g(s,x),

u(s,x)=

sup

a s I%I(s,x)

va(s,x). (1)

PROOF.By Lemma 4.2, for any s 2 0, x E Ed lim sup M~,,Q(T,X~-,)~-"T-S = 0.

T+oo a s %

Let

I

+ t, x,) dt - e-QT-sg(T7xT-s). (3)

i7(T7s,x)= sup M:,, aeBT

Due to (2), lim Iv(T,s,x) - E(T,s,x)l = 0.

T+m

Further, let T' > T, and let a E By Theorem 2.9.7, almost surely

(4)

aT'and at = g for t E [ T - s, T' - s].


By Lemma 4.1, the last expression is greater than (- l)g(T,x$?g. Therefore, for strategies a of the given type

Computing here the upper bounds, we obtain, obviously, 5(T,s,x) 5 5(T1,s,x).Therefore, the function E(T,s,x) increases with respect to T. From this and (4) it follows that the limits lim E(T,s,x), lim v(T,s,x) T- m

T-r m

exist and are equal. Furthermore, these limits are greater than E(s,s,x) = - v(s,x). Let us prove (1). On one hand, by the monotone convergence theorem for a E I'%I(s,x) vU(s,x)= v?+)(s,x)- 0:-)(s,x)

I lim v(T,s,x) = v(s,x). T-r m

Thus, the right side of (1) does not exceed v(s,x). On the other hand, let s 2 0, x E Ed, T > s, a E 21T. Also, we define a' E 2l using the formula a: = a, for t < T - s, a; = for t 2 T - s. For Tf > T, as above,

+ Mt,,eCvT -'g(T,xT-3, where, by virtue of (2) for a sufficiently large T, the right side is finite and, therefore, the primary expression is bounded uniformly with respect to T'. Hence, if T is sufficiently large, a' E I%l(s,x). From (2) and (5) we have


Therefore, for sufficiently large T sup

va(s,x)2 iT(T,s,x).

a = IVI(S,X)

As T -* co we have that v(s,x) does not exceed the right side of ( I ) , thus proving the theorem. 2. Remark. The above proof can be used in finding &-optimalstrategies on an infinite time interval if v(s,x) < co. Namely, we take first a large T such that the limiting expression in (2) becomes smaller than &/3 and that Iv(T,s,x) - v(s,x)(< 43. Next, we choose a E N such that

Further, as we did in the proof given, we costruct a' on the basis of a. Then, by (6)

3. Remark. We can write (1) as follows : v(s,x) r lim lim vn(T,s,x) = sup T - m n-m

a~ I(DII(s,x)

va(s,x).

It turns out that we can interchange the limit with respect to n and the limit with respect to T . In fact, let us denote by iTn(T,s,x)the right side of (3), in which N T is replaced by N:. Here as well as in our theorem we prove that F,(T,s,x) increases with respect to T , if A, 3 a. Let this condition be satisfied for n 2 n*. Then from (2)we have v(s,x) = lim lim Fn(T,s,x) = sup sup iT,(T,s,x) T + m n-m

Tzs nsn*

=

sup sup iTn(T,s,x)= sup lim Fn(T,s,x) nzn* T g s n t n * T+m

=

sup lim v,(T,s,x) = lim lim v,(T,s,x). nsn* T - m n-+m T+m

Remark 2 contains the inequality v(s,x) < co. The sufficient condition for v(s,x) to be finite can easily be found from Lemma 4.1. 4. Theorem. Let there exists a nonnegative function iJ E W:d,2((O,co) x Ed) which is continuous in [O,co) x Ed and such that for all a E A Laa f"+ 5 0 (a.e.). Then - v -(s,x) Iv(s,x) 5 V(s,x) lNl(s,x) = N ,

+

for all s 2 0, x E Ed. We derive now the normed Bellman equation. We shall assume in future that the assumptions made in the preceding section after Exercise 4.11 are


satisfied if we replace in those assumptions g(s,x) by 0, and, in addition, if we replace in (4.12) the function w,(T,s,x) by v,(T,s,x). Here as well as in Section 4, the following holds true. 5. Theorem. The function v(s,x) in the region Q* is continuous, has a generalized jirst derivative with respect to s and two generalized derivatives with respect to x, and, in addition, all these derivatives are locally bounded in Q*. For each normalizing multiplier ma(s,x) Gmm[v] =0

(a.e. on Q*).

We consider now in more detail the case where the functions o, b, c, f do not depend on s. It is seen that here the function v(s,x) does not depend on s: v(s,x) = v(x). Here as well as in the preceding section, we introduce Gmz(uij,ui,u,x) Gm=[u](x) and a region D*. From Theorem 5 we immediately have the following.

6. Theorem. The function v(x) in the region D* is continuous and has two generalized derivatives with respect to x. These derivatives are locally bounded in D*. If the nonnegative function ma(x)is such that for all x, uij,ui, u Gma(uij,ui,u,x) < co,

sup ma(x)< oo, a

then Gmu[v]= 0 (D*-a.e.).

The first condition in (7)(also, (4.13))is superfluous in some cases. 7. Theorem. For all n 2 1, x

inf [tr a(a,x)

aeA,

E D*,

let

+ Ib(a,x)l + ca(x)+ If"(x)l] > 0.

(8)

Then G m f v ]= 0 (D*-a.e.)for any nonnegative function ma(x)such that the inequality Ga(uij,ui,u,x) < 03 is satisjied for all x, uij,ui, u.

PROOF. Let

v,(x)

=

lim v,(T,s,x). T+m

As follows from Theorem 1, for all n for which g

GF(uij,ui,u,x)= sup ma(x) a € A,

i,j= I

aij(a,x)uij+

d

i= 1

E A,,

this limit exists. Let

I

bi(a,x)ui- ca(x)u+ f ' ( x ) ,


We note that by (8)

sup rTi,,(x) < co.

aeA,

Let n be such that g E A, and let temporarily A = A,. Then, by Theorem 6 we have that G?[v,] = 0 (D*-a.e.),where D,*= {x E Ed:sup (a(a,x)il,A) > 0 for all il # 0). aeA,

We take a bounded region D' c D' c D*. Also, we choose a large number no such that c D,*,,g E A,,. For n 2 no hence

Let here n + co. By hypothesis, the ordinary derivatives of the functions v,(T,s,x) as well as the functions themselves are bounded in (0,l) x D' uniformly for n 2 no, T 2 2. This implies that for n 2 no the function v,(x) has two generalized derivatives which are uniformly bounded in D'. According to Remark 3 and Theorem 4.5.1 we have from (9) as n + co

Since D' is an arbitrary subregion of D* and furthermore, G r O1 Gaa0 as 0 '

+

O07

GfiaO[u]= 0

(D*-a.e.).

Next, repeating almost word-for-word the proof of Lemma 3.8, it is easy to see that all solutions of the equation GfimO(uij,u_,,u,x) = 0 are solutions of the equation Gma(uij,ui,u,x) = 0 if the function Gmais finite. Therefore, in this case Gma[v] = 0 (D*-a.e.),thus proving the theorem. Sometimes the application of Theorem 7 yields more extensive information about a payoff function than the application of Theorem 6 does.

8. Example. Let A = (0,1], A, = [l/n,l], and let the function f(x) 2 0. It is not hard to visualize a situation in which Fa[u](x)

= sup ma(x)[adu(x) - au(x) ~ E A

+ ~ '(x)]. f

If the function m,(x) is bounded with respect to a for each x and if, in addition, m,(x) > 0, the relation Gma[v](x)= 0 holds if and only if Av(x) - v(x) f(x) 1 0. In particular, this inequality is equivalent to the Bellman equation (m, = 1). At the same time, if we take ma(x)= l/a, it is easily seen that Gmais a finite function and also (by Theorem 7 if the appro-

+

priate conditions are satisfied) 0 = sup(Av - v a6A

+ af)= A v - v + f .

9. Exercise Let A = (0,1], A, support, and let

= [l/n,l].

Furthermore, let f(x) be a smooth function with compact

cma[u](x)= sup m,(x)[a Au(x) - au(x) asA

+ af(x)].

Prove that A v - v + f 5 0 (a.e.), v 2 0 and A v - v + f = 0 almost everywhere in the set {x:v(x) > 0). Prove also that iff 5 0, then v = 0 and, in addition, if we take the normalizing multiplier a-', we obtain the false relation A v - v + f = 0. Explain why Theorem 7 is inapplicable in this case. (Hint: See the assumptions made at the start of this section.)

10. Remark. If condition (8) is satisfied, it is possible to prove the theorem on zero crossing or the theorem that (v,,,,,v,,,v) belongs to a boundary of some set, which are similar to Theorems 3.12 and 3.13.

Notes Section 1. The idea of Exercise 6 is similar to that of an example due to Dynkin. Section 2. In [36] and [37], it is required that c be sufficiently large compared to the first and second derivatives of a, b, c, f. It is thus assumed that (15)and (17) are satisfied. In [22], an example illustrates that if T = co and, in addition, inequality (15) is violated, the second derivative of the payoff function can be unbounded. It is known about a diffusion process (A consists of a single point; see Freidlin [19]) that if "killing" is insignificant, the "payoff function" for T = co need not possess smoothness. It is also known that as c as well as the smoothness of initial functions increases, the smoothness of the "payoff function" increases. (See Freidlin [19]). It is interesting to note that the increase of smoothness need not occur in controlled processes. For instance, let (w,,g,) be a one-dimensional Wiener process, A = [O,v], T = co,

where 1> 0. It is not hard to show that 1 v(x) = - cos x,

1

x E [O,z],


where z is the solution on ( 0 , ~of ) the equation

Hence v"(z-) = -(l/A)cosz # 0 = v"(z + ) and, further, for each 1 the second derivative v is discontinuous. Concerning the increase of smoothness of a payoff function, see also the remarks made prior to Theorem 1.4.15. Sections 3,4,5. In these sections the results obtained in [37] have been developed further. Theorem 4.8 can be obtained by the methods used by Shiryayev in [69]. There are other well-known methods for finding a stopping boundary, different from the method described in Exercise 4.16, see [44, 56,681.

Appendix 1

Some Properties of Stochastic Integrals

We shall mention some facts from the stochastic integral theory, omitting the proof. The latter can however be found in Doob [9], Dynkin [ll], and Shiryayev [51], and Gikhman and Skorokhod [23,24]. Let (52,F) be a measurable space, and let {F,, t 2 0) be an increasing family of G-algebras(a flow of o-algebras) which satisfies the condition F, c F for all t 2 0. A process 5,(0) given for t 2 0, w E 52 with values in Ed, is if) for each s > 0 said to be progressively measurable (with respect to {F,) the function &(w) considered for t E [O,s], w E 52 is measurable with respect to the direct product of a o-algebra of Bore1 subsets of an interval [O,s] and 9,. It is a known fact that a continuous process t,, which is measurable with respect to F,for each t, is progressively measurable. A nonnegative function z given on D is said to be a Markov time (with respect to {F,}), if for any s 2 0 the set {w:z(w) > s) E F s . It is also a known fact that the times of first exit of continuous progressively measurable processes 5, from open sets are Markov times. Let a probability measure P be given on (52,F). A continuous process w, = (w:, . . . ,wtl) defined for t 2 0, o E D is referred to as a dl-dimensional Wiener process if w, = 0, the increments w, on nonoverlapping intervals are independent, and, finally, if the distribution w, - w,(t > s) is normal with parameters 0, (t - s)I, where I denotes the unit matrix of dimension dl x dl. If, furthermore, for any t 2 0 the variable w, is F,-measurable, the increment w,,, - w, for h 2 0 does not depend on Ft and if the o-algebras of 9,are complete, we say that the pair (w,,F,) is a dl-dimensional Wiener process or that w, is a dl-dimensional Wiener process with respect to {F,}. It should be mentioned here that any Wiener process w, is Wiener with respect to the completion of its own o-algebras of 9; = o{w,:s I t).

Appendix 1 Some Properties of Stochastic Integrals

Let (w,,F,) be a dl-dimensional Wiener process, and let a, be a random matrix of dimension d x dl which is progressively measurable with respect to ( 9 , ) and such that for each t 2 0

So

Then the stochastic integral a,dw, is defined, and is a continuous progressively measurable process (with respect to (9,)) satisfying the condition

so

The stochastic integral a, dw, can be constructed in the following way. Let a process a, be a step process, i.e., there exist numbers 0 = to < t , < t2 < . . . < t, = m such that a, = ati for t E [ti,ti+ i = 0, . . . , n - 1. Then Soa,dw, can be defined as the corresponding sum: if t E [ti,ti+,), then

In this case equality (2) can immediately be verified. In the general case we prove that there exists a sequence of progressively measurable step processes o,(n) such that for each t > 0

Hence, by the Cauchy criterion and equality (2), the sequence of stochastic integrals a,(n)dw, is Cauchy in L, (mean square sense), and, hence, convergent in L,. Let us denote the limit a, dw,, where the limit for each t is determined of course only to within an equivalence. We also prove that for each t, w one can choose values of Sf, a,dw, such that the process thus obtained becomes continuous with respect to t. Hence by the integral o, dw, we usually mean a continuous process. It turns out that for any Markov time z for all t 2 0 almost surely

So

So

fi

A stochastic integral is in general not the limit of integral sums, similar to the Riemann-Stieltjes sums. It is, however, known that for each process a, there exists a sequence of integers i(n) which tends to infinity as n + m and is such that for all T > 0 and almost all s E [0,1]

where xi(t) = 2-'[t2'], [a] denoting the largest integer I a. We note that the second integral in (4) is an integral of a step function and, therefore, is of the form (3).

Appendix 1 Some Properties of Stochastic Integrals

A stochastic integral can be defined not only for the functions a, satisfying condition (1).The characteristic feature of a stochastic integral under condition (1) is that

almost surely on the set {w:z(w) 2 t) for any bounded Markov time z and for any t. Let z be a Markov time, let a, be a process with values in a set of matrices of dimension d x dl, and let b, be a d-dimensional process. We assume that xs5ps, ~~~~b~ are progressively measurable. Furthermore, we assume that the integrals ~,,,o,dw,, ~,,,b,ds are defined. If the process 5, satisfies the relation

so

Sf,

it is convenient to write dt, = a, dw, + b, dt, t I z, to= x. The formal expression a,dw, + b,dt is called the stochastic differential of t,. The notation dt, = a, dw, b, dt, t Iz, to= x is only the short-hand representation of Eq. (6), and is more convenient than (6) when we encounter the need to write u(tt), where u is some function, using the stochastic integrals. It is sufficient to find the stochastic differential du(t,). It turns out that for any twice continuously differentiable function u given on Ed, we have the following Ito's formula:

+

where the first term is understood as

) or grad ~ ( 5 , ) We can write the first term in (7) in a short form: grad ~ ( 5 ,dt, a, dw, + grad u([,)b, dt. In order to compute dtf d t j in the second term in (7), one is to apply the usual rules for removing parentheses as well as the following rules for the product of stochastic differentials: ( d ~ f=) dt, ~ dwf dw: = 0 for i # j, dwfdt = 0, (dt)2 = 0. In short, dtf d5j =

d1

1 afkajkdt = (o,a:)'j

dt.

k= 1

Putting together in (7) the terms with dt and the terms with dw,, respectively, we can rewrite Ito's formula as follows: du(t,) = grad u(t,)a, dw,

+ L"t,btu(

Controlled Diffusion Processes

Diffusion in Solids: Fundamentals, Methods, Materials, Diffusion-Controlled Processes

Multidimensional Diffusion Processes

Schroedinger diffusion processes

Multidimensional diffusion processes

Multidimensional Diffusion Processes

Diffusion Processes and Fertility Transition

Controlled Markov Processes and Viscosity Solutions

Controlled Markov Processes and Viscosity Solutions

Controlled Markov Processes and Viscosity Solutions

Controlled Markov Processes and Viscosity Solutions

Diffusion, Markov processes and martingales. Ito calculus

High Dimensional Nonlinear Diffusion Stochastic Processes

Schrödinger diffusion processes (Probability and its Applications)

Schrödinger Diffusion Processes (Probability and its Applications)

Diffusion Processes and their Sample Paths

High Dimensional Nonlinear Diffusion Stochastic Processes

Diffusion Processes During Drying of Solids

Stochastic differential equations and diffusion processes

Multidimensional Diffusion Processes (Classics in Mathematics)

Diffusion processes in advanced technological materials

Controlled Environment

Controlled Desires

Stochastic Differential Equations and Diffusion Processes, Second Edition

Stochastic Differential Equations and Diffusion Processes, second edition

Kinetic Processes: Crystal Growth, Diffusion, and Phase Transitions in Materials

Stochastic Differential Equations and Diffusion Processes (North-Holland mathematical library)

Controlled Cravings

Controlled Response

Stochastic Differential Equations and Diffusion Processes, Second Edition

Controlled Diffusion Processes

Diffusion in Solids: Fundamentals, Methods, Materials, Diffusion-Controlled Processes

Multidimensional Diffusion Processes

Schroedinger diffusion processes

Multidimensional diffusion processes

Multidimensional Diffusion Processes

Diffusion Processes and Fertility Transition

Controlled Markov Processes and Viscosity Solutions

Controlled Markov Processes and Viscosity Solutions

Controlled Markov Processes and Viscosity Solutions

Controlled Markov Processes and Viscosity Solutions

Diffusion, Markov processes and martingales. Ito calculus

High Dimensional Nonlinear Diffusion Stochastic Processes

Schrödinger diffusion processes (Probability and its Applications)

Schrödinger Diffusion Processes (Probability and its Applications)

Diffusion Processes and their Sample Paths

High Dimensional Nonlinear Diffusion Stochastic Processes

Diffusion Processes During Drying of Solids

Stochastic differential equations and diffusion processes

Multidimensional Diffusion Processes (Classics in Mathematics)

Diffusion processes in advanced technological materials

Controlled Environment

Controlled Desires

Stochastic Differential Equations and Diffusion Processes, Second Edition

Stochastic Differential Equations and Diffusion Processes, second edition

Kinetic Processes: Crystal Growth, Diffusion, and Phase Transitions in Materials

Stochastic Differential Equations and Diffusion Processes (North-Holland mathematical library)

Controlled Cravings

Controlled Response

Stochastic Differential Equations and Diffusion Processes, Second Edition

Recommend Documents