Stochastic optimal control Discrete-time Case

Stochastic Optimal Control: The Discrete- Time Case Dimitri P. Bertsekas and Steven E. Shreve WWW site for book inform...

Author: Bertsekas

84 downloads 1400 Views 4MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Stochastic Optimal Control: The Discrete- Time Case Dimitri P. Bertsekas and Steven E. Shreve

WWW site for book information and orders http://world.std.com/-athenasc/

Athena Scientific, Belmont , Massachusetts

Contents Preface Acknowledgments

Chapter I 1.1 1.2 1.3

Introduction

Structure of Sequential Decision Models Discrete-Time Stochastic Optimal Control Problems-Measurability Questions The Present Work Related to the Literature

Part I ANALYSIS O F DYNAMIC PROGRAMMING MODELS Chapter 2 2.1 2.2 2.3

Notation and Assumptions Problem Formulation Application to Specific Models 2.3.1 Deterministic Optimal Control 2.3.2 Stochastic Optimal Control-Countable Disturbance Space 2.3.3 Stochastic Optimal Control-Outer Integral Formulation 2.3.4 Stochastic Optimal Control-Multiplicative Cost Functional 2.3.5 Minimax Control

Chapter 3 3.1 3.2 3.3

Monotone Mappings Underlying Dynamic Programming Models

Finite Horizon Models

General Remarks and Assumptions Main Results Application to Specific Models

viii

CONTENTS

Chapter 4 4.1 4.2 4.3

4.4

General Remarks and Assumptions Convergence and Existence Results Computational Methods 4.3.1 Successive Approximation 4.3.2 Policy Iteration 4.3.3 Mathematical Programming Application to Specific Models

Chcrpter 5 5.1 5.2 5.3 5.4

5.5

Infinite Horizon Models under Monotonicity Assumptions

General Remarks and Assumptions The Optimality Equation Characterization of Optimal Policies Convergence of the Dynamic Programming Algorithm-Existence of Stationary Optimal Policies Application to Specific Models

Chupter 6 6.1 6.2 6.3

Infinite Horizon Models under a Contraction Assumption

A Generalized Abstract Dynamic Programming Model

General Remarks and Assumptions Analysis of Finite Horizon Models Analysis of Infinite Horizon Models under a Contraction Assumption

Purt II

STOCHASTIC OPTIMAL CONTROL THEORY

Chapter 7 Borel Spaces and Their Probability Measures 7.1 7.2 7.3 7.3

7.5 7.6

7.7

Notation Metrizable Spaces Borel Spaces Probability Measures on Borel Spaces 7.4.1 Characterization of Probability Measures 7.4.2 The Weak Topology 7.4.3 Stochastic Kernels 7.4.4 Integration Semicontinuous Functions and Borel-Measurable Selection Analytic Sets 7.6.1 Equivalent Definitions of Analytic Sets 7.6.2 Meas'urability Properties of Analytic Sets 7.6.3 An Analytic Set of Probability Measures Lower Semianalytic Functions and Universally Measurable Selection

Chnpter 8 8.1

The Finite Horizon Borel Model

The Model

CONTENTS 8.2 8.3

The Dynamic Programming Algorithm-Existence and T k ( ~I) Tk+'(J), k = o,i,. . . , J 2 T(J) =, T,(J) 2 T,+'(J), J I T,(J)

V~EM=,(T,;~~T,~)(J)I(T,,~~~T,~~~)(J k = 0,1,. . . , n

J 2

T,(J)

VPEM=>(T,;

.

= (po,pl,. . .)EII,

. T,,)(J) 2 up;. . T,,+,)(J), k = 0 , 1 , . . . , n = ( p O , p l,...) E n .

28

2.

MONOTONE MAPPINGS IN DYNAMIC PROGRAMMING MODELS

Another fact that we shall be using frequently is that for each J E F and > 0, there exists a p, E M such that

E

+E

T(J)(x) -I/E

if if

T ( J ) ( x )> - co, T ( J ) ( x )= - co.

In particular, if J is such that T ( J ) ( x )> - co for V x E S, then for each E > 0, there exists a p, E M such that

2.2 Problem Formulation

We are given a function J , E F satisfying , . .) E II and positive integer and we consider for every policy n = ( p O pl,. N the functions J N , , E F and J , E F defined by

J,(x)

=

lim (T,,TP1 . . . T,,

N-+m

_ l)(Jo)(x)

V x E S.

(6)

For every result to be shown, appropriate assumptions will be in effect which guarantee that the function J , is well defined (i.e., the limit in ( 6 ) exists for all X E S ) . We refer to J N . , as the N-stage cost function for n and to J , as the cost function for n. Note that J N , , depends only on the first N functions in n while the remaining functions are superfluous. Thus we could have considered policies consisting of finite sequences of functions in connection with the N-stage problem, and this is in fact done in Chapter 8. However, there are notational advantages in using a common type of policy in finite and infinite horizon problems, and for this reason we have adopted such a notation for Part I. Throughout Part I we will be concerned with the N-stage optimization problem minimize J,, ,( x ) (F) subject to n E II, and its infinite horizon version minimize J,(x) subject to n E n. We refer to problem (F)as the N-stagefinite horizon problem and to problem ( I ) as the injinite horizon problem.

For a*fixed XES, we denote by Jg(x) and J*(x) the optimal costs for these problems, i.e., Vx E S, (7) JA(x) = inf JN, ,(x) X

E

~

J*(x) = inf J,(x)

Vx E S.

ria II

We refer to the function JA as the N-stage optimal cost function and to the function J * as the optimal cost function. We say that a policy n* E ll is N-stage optimal at x E S if JN, ..(x) = J$(x) and optimal at x E S if J,*(x) = J*(x). We say that n* E l 7 is N-stage optimal (respectively optimal) if JN.,* = JR (respectively J,, = J*). A policy TC*= (pg ,p:,. . .) will be called uniformly N-stage optimal if the policy (p:, p:+ ,, . . .) is (N - i)-stage optimal for all i = 0,1,. . . ,N - 1. Thus if a policy is uniformly N-stage optimal, it is also N-stage optimal, but not conversely. For a stationary policy n = (p, p, . . .) E ll, we write J, = J,. Thus a stationary policy n* = (p*,p*, . . .) is optimal if J * = J,,. Given E > 0, we say that a policy n, E fl is N-stage &-optimalif J ~n ,. ( ~ ) 5

JR(x)

+E

if J ~ ( x >)- co, if JX(x) = - co.

We say that n, E ll is E-optimalif

If [E,) is a sequence of positive numbers with E, 10, we say that a sequence of policies {n,) exhibits {&,I-dominatedconvergence to optimality if lim JN, =, and, for n = 2,3,. . . ,

n+

= JA,

w

2.3 Application to Specific Models

A large number of sequential optimization problems of practical interest may be viewed as special cases of the abstract problems (F) and (I). In this section we shall describe several such problems that will be of continuing interest to us throughout Part I. Detailed treatments of some of these problems can be found in DPSC.+ + We denote by DPSC the textbook by Bertsekas. "Dynamic Programming and Stochastic Control." Academic Press. Ncw York, 1976.

30

2.


2.3.1 Deterministic Optimal Control Consider the mapping H : SCF

-+

R* defined by

Our standing assumptions throughout Part I relating to this mapping are: (1) The functions g and f map SC into [- co, a ] and S, respectively. (2) The scalar cx is positive. The mapping H clearly satisfies the monotonicity assumption. Let Jo be identically zero, i.e.,

Then the corresponding N-stage optimization problem (F)can be written as ..

minimize J,,,(xo)

akg[xk,pk(xk)]

= k=O

subject to xk+

=f

[ x k ,pk(xk)],

pk E M ,

k

= 0, . . . , N

-

1.

This is a finite horizon deterministic optimal control problem. The scalar a is known as the discount factor. The infinite horizon problem (I) can be written as minimize J,(xo)

=

lim N+m

subject to

xk+

=f

1 a k g [ x kpk(xk)] ,

k=o

[ ~ kpk(xk)], ,

pkE M , k

= 0, 1,.

This limit exists if any one of the following three conditions is satisfied:

cc

< 1,

g(x, u) 2 0 g(x, u) I 0 0 I g(x, u) I b

v x € S, u € U(x), (12) v x € S, u € U(x), (13) for some b~ (0, co) and all x E S, u E U(x). (14)

Every result to be shown for problem (11)will explicitly assume one of these three conditions. Note that the requirement 0 < g(x, u) I b in (14) is no more strict than the usual requirement Ig(x,u)l I b/2. This is true because adding the constant b/2 to g increases the cost corresponding to every policy by b/2(1 - x ) and the problem remains essentially unaffected. Deterministic optimal control problems such as (10) and ( 1 1 ) and their stochastic counterparts under the countability assumption of the next subsection have been studied extensively in DPSC (Chapters 2, 6, and 7). They are given here in their stationary form in the sense that the state and control spaces S and C , the control constraint U ( . ) ,the system function f , and the

cost per stage g do not change from one stage to the next. When this is not the case, we are faced with a nonstationary problem. Such a problem, however, may be converted to a stationary problem by using a procedure described in Section 10.1 and in DPSC (Section 6.7). For this reason, we will not consider further nonstationary problems in Part I. Notice that within our formulation it is possible to handle state constraints of the form x ~ E X , k = 0,1,. . . , by defining g(x, u) = cc whenever x $X.This is our reason for allowing g to take the value oo. Generalized versions of problems (10) and (11) are obtained if the scalar a is replaced by a function a : SC -+ R* with 0I ~ ( xu), for all x E S, u E U(x), SO that the discount factor depends on the current state and control. It will become evident to the reader that our general results for problems (F) and (I) are applicable to these more general deterministic problems. 2.3.2

Stochastic Optimal Control-Countable

Disturbance Space

Consider the mapping H : SCF -+ R* defined by

where the following are assumed: (1) The parameter w takes values in a countable set W with given probability distribution p(dwlx,u) depending on x and u, and E{.lx,u} denotes expected value with respect to this distribution. (See a detailed definition below.) (2) The functions g and f map SCW into [ - co,co] and S, respectively. (3) The scalar cc is positive. Our usage of expected value in (15) is consistent with the definition of the usual integral (Section 7.4.4) and the outer integral (Appendix A), where the o-algebra on W is taken to be the set of all subsets of W. Thus if wi, i = 1,2,. . . , are the elements of W, (p',p2,. . .) any probability distribution on W , and z: W -+ R* a function, we define

where z'(wi) z-(w,)

= max [O, z(wi)), =

max(0, - z(wi)>,

i = 1,2,. . . , i = 1,2,. . . .

In view of our convention cx: - oo = co,the expected value E (z(w)) is well defined for every function z: W -+ R* and every probability distribution (p1,p2,. . .) on W. In particular, if we denote by (pl(x, u), p2(x,u), . . .) the

32

2.


probability distribution p(dwlx, u) on W = {w', w2,. . .), then (15) can be written as m

H(x, u, J ) =

1pi(x,U)max{O, g(x, u, w') + aJ[f

(x, u, wi)])

i=l

A point where caution is necessary in the use of expected value defined this way is that for two functions z, : W + R* and 5,: W + R*, the equality need not always hold. It is guaranteed to hold if (a) E(z:(w)} < cl;: and E{z; (w)) < oo,or (b) E(z;(w)) < co and E{z; (w)} < co,or (c) E (z: (w)) < co and E{z;(w)) < co (see Lemma 7.11). We always have, however,

It is clear that the mapping H of (15) satisfies the monotonicity assumption. Let Jobe identically zero, i.e.,

Then if g(x, u, w) > - co for all x, u, w, the N-stage cost function can be written as

where the states x,,x,, . . . , x N - , satisfy

The interchange of expectation and summation in (17) is valid, since g(x, u, w) > - oo for all x, u, w. and we have for any measure space (Q, F, v),

measurable h: R

-1

R*, and iE (- oo,+ oo],

When Eq. (18) is used successively to express the states x,,x,, . . . ,x,-, exclusively in terms of wo, w,, . . . ,wN- and x,, one can see from (17) that JN,,(xo) is given in terms of successive iterated integration over w,- ,, . . . ,wo: For each xo E S and 71 E rI the probability distributions pi(xo, pO(xO)), . . . ,pL(xN- pN- l ( ~ N -,)), i = 1,2,. . . , over W specify, by the product measure theorem [ A l , Theorem 2.6.21, a unique product measure on the cross product W N of N copies of W. If Fubini's theorem [ A l , Theorem 2.6.41 is applicable, then from (17) the N-stage cost function JN,,(xo) can be alternatively expressed as

,,

where this expectation is taken with respect to the product measure on wN and the states x , , x,, . . . ,x,-, are expressed in terms of w,, w,, . . . ,w,-, and xo via (18). Fubini's theorem can be applied if the expected value in (19)is not of the form oo - oo,i.e., if either

In particular, this is true if either

E(max(0, - g [ x k , p k ( x k ) w , , ] ) ) < oo,

k

= 0,.

. . ,N - 1

or if g is uniformly bounded above or below by a real number. If JN,,(xo) can be expressed as in (19)for each xo E S and TC E II,then the N-stage problem can be written as N-1

minimize J,, ,(xo) = E{

1

rkg[q, pk(xk),wk] k=O

subjectto x k + , = f [ x k , p k ( x k ) , w k ] , p k € M , k = O , . . . , N - 1 : which is the traditional form of an N-stage stochastic optimal control problem and is also the starting point for the N-stage model of Part I1 (Definition 8.3).

34

2.


The corresponding infinite horizon problem is (cf. Definition 9.3) minimize

J,(x,)

subject to x,,

=

lim E

akg[x,, ~,(x,),w,]

, = f [x,, p,(x,), w,],

p, E M, k

= 0,1,.

...

This limit exists under any one of the conditions:

a

< 1,

g(x, u, w) 2 0 g(x, u, w) l 0 0I g(x, u, w) l b

vx € S, u € U(x), w € w, Vx € S, u E U(x), w € w, for some b E (0, co) and all x E S, u E U(X),w E W .

(21) (22) (23)

Every result to be shown for problem (20) will explicitly assume one of these three conditions. Similarly as for the deterministic problem, a generalized version of the stochastic problem is obtained if the scalar a is replaced by a function x : S C W + R" satisfying 0 I a(x, u, w) for all (x, u, w). The mapping H takes the form

H(x, u, J) = E jg(x, u, w) + a(x, u, w)J[f (x, u, w)] lx, u } . This case covers certain semi-Markov decision problems (see [J2]). We will not be further concerned with this mapping and will leave it to the interested reader to obtain specific results relating to the corresponding problems (F) and (I) by specializing abstract results obtained subsequently in Part I. Also, nonstationary versions of the problem may be treated by reduction to the stationary case (see Section 10.1 or DPSC, Section 6.7). The countability assumption on W is satisfied for many problems of interest. For example, it is satisfied in stochastic control problems involving Markov chains with a finite or countable number of states (see, e.g., [D3], [K6]). When the set W is not countable, then matters are complicated by the need to define the expected value

E(g[x, P(x), wl + aJ[f (4P(x), w)Ilx, u ) for every EM. There are two approaches that one can employ to overcome this difficulty. One possibility is to define the expected value as an outer integral, as we do in the next subsection. The other approach is the subject of Part I1 where we impose an appropriate measurable space structure on S, C, and W and require that the functions , uM ~ be measurable. Under these circumstances a reformulation of the stochastic optimal control problem into the form of the abstract problems (F) or (I) is not straightforward. Nonetheless, such a reformulation is possible as well as useful as we will demonstrate in Chapter 9.

2.3.3

Stochastic Optimal Control-Outer

Consider the mapping H: SCF

-i

Integral Fornzulation

R* defined by

H(x, u, J) = E* [ g b , u, W ) + aJ[f (x, u, w)]lx, u},

(24)

where the following are assumed: (1) The parameter w takes values in a measurable space (W, F ) . For each fixed (x, u) E SC, a probability measure p(dw]x,u) on (W, 9)is given and E*(.lx, u ) in (24) denotes the outer integral (see Appendix A) with respect to that measure. Thus we may write, in the notation of Appendix A,

H(n, u, J ) =

J*{g(x,u, w) i nJ[/(x,

u. w)])p(dwlx, u).

(2) The functions g and f map SCW into [ - co,co] and S, respectively. (3) The scalar cc is positive. We note that mappings (9) and (15) of the previous two subsections are special cases of the mapping H of (24). The mapping (9) (deterministic problem) is obtained from (24) when the set W consists of a single element. The mapping (15) (stochastic problem with countable disturbance space) is the special case of (24) where W is a countable set and F is the o-algebra consisting of all subsets of W. For this reason, in our subsequent analysis we will not further consider the mappings (9) and (Is),but will focus attention on the mapping (24). Clearly H as defined by (24) satisfies the monotonicity assumption. Just as for the models of the previous two sections, we take J,(x)

=0

VXES

and consider the corresponding N-stage and infinite horizon problems (F) and (I). If appropriate measurability assumptions are placed on S, C, f , g, and p, then the N-stage cost JN.li(x) = (Two. . T , N - I ) ( ~ o ) ( ~ ) can be rewritten in terms of ordinary integration for every policy n = (p,, p,, . . .) for which p,, k = 0,1,. . . , is appropriately measurable. To see this, suppose that S has a o-algebra 9, C has a o-algebra %',and @ is the Bore1 o-algebra on R*. Suppose f is ( Y V F , 9')-measurable and g is (YV.F,.g)measurable, where 9 V . F denotes the product o-algebra on SCW. Assume that for each fixed B E 9, p(Blx, u) is .YV-measurable in (x, u) and consider a policy n = (p,, p,, . . .), where p, is (Y,'#)-measurable for all k. These conditions guarantee that T,, (J)given by '

36

2.


is 9-measurable for all k and J E F that are Y-measurable. Just as in the previous section, for a fixed x o E S and n = (p,, p,, . . .) E ll, the probability measures p(. Ixo, po(xo)),. . . ,p(. IxN-, ,pN- ,(xN - ,)) together with the system equation

define a unique product measure p(d(wo, . . . ,w N - ,)lxo,n) on the cross product W Nof N copies of W . [Note that x,, k = 0,1,. . . ,N - 1, can be expressed as a measurable function of (w,, . . . ,w N - , ) via (25)l. Using the calculation of the previous section, we have that if g ( x , u, w) > - cc for all x , u, w, and Fubini's theorem is applicable, then

where x , , x , , . . . , x N - , are expressed in terms of w o , w,, . . . ,w,-, and x o via (25). Also, as in the previous section, Fubini's theorem applies if either

Thus if appropriate measurability conditions are placed on S, C, W ,f , g, and p(dwlx, u) and Fubini's theorem applies, then the N-stage cost J N , , corresponding to measurable zreduces to the traditional form

This observation is significant in view of the fact that inf J,,.(x) rren

I inf J,,.(x)

V ~ S.E

rrsn

where

Thus, if an optimal (c-optimal) policy n* can be found for problem (F)and

n* E fi .(i.e., is measurable), then n* is optimal (eoptimal) for the problem minimize J,, ,(x) subject to n E fi, which is a traditional stochastic optimal control problem. These remarks illustrate how one can utilize the outer integration framework in an initial formulation of a particular problem and subsequently show via further (and hopefully simple) analysis that attention can be restricted to the class of measurable policies fi for which the cost function admits a traditional interpretation. The main advantage that the outer integral formulation offers is simplicity. One does not need to introduce an elaborate topological and measure-theoretic structure such as the one of Part I1 in an initial formulation of the problem. In addition the policy iteration algorithm of Chapter 4 is applicable to the problem of this section but cannot be justified for the corresponding model of Part 11. The outer integral formulation has, however, important limitations which become apparent in the treatment of problems with imperfect state information by means of sufficient statistics (Chapter 10). 2.3.4 Stochastic Optimal Control-Multiplicati2;e Cost Functional

Consider the mapping H :SCF -t R* defined by We make the same assumptions on w, g, and f as in Section 2.3.2, i.e., w takes values in a countable set W with a given probability distribution depending on x and u. We assume further that

In view of (27), the mapping H of (26) satisfies the monotonicity assumption. We take

and consider the problems (F) and (I). Problem (F) corresponds to the stochastic optimal control problem minimize

JN,~(xO)=E(g[xO~~O(xO)~wO]~~~~[xN-l~~N-1(xN-I)~wN-1])

(28) subject to x,,,

= f [x,,p,(x,),

w,],

,U,E

M, k = O,l,. . . ,

and problem (I) corresponds to the infinite horizon version of (28). The limit as N + cc in (28) exists if g(x, u, w) 2 1 for every x, u, w or 0 I g(x, u, w) I 1

38

2.


for every x, u, w. A special case of (28) is the exponential cost functional problem minimize

i , [I::

E exp

subject to x,,

g'[x,, ~,(x,),w,]

=f

11 y , M~ , k

[x,, p,(x,), w,],

= 0,1,.

..,

where g' is some function mapping S C W into (- a , a ] . 2.3.5 Minimax Control Consider the mapping H: SCF

H(x, u, J ) =

-+ R*

defined by

sup { q ( x ,u, w)

w s W(x, U)

+ aJ[ f ( x , u, w ) ] )

where the following are assumed:

(1) The parameter w takes values in a set W and W ( x ,u) is a nonempty subset of W for each x E S, u E U(x). (2) The functions g and f map S C W into [ - co, co] and S respectively. (3) The scalar a is positive. Clearly the monotonicity assumption is satisfied. We take If g(x, u, w) > - cc for all x, u, w, the corresponding N-stage problem (F) can also be written as minimize J , .(I,)

sup

=

N j liakg[xk,,uk(xk),

W k € W[xk, lrk(xk)l

subject to x,,

, = f [x,, ,u,(x,), w,],

W,]}

k= 0

pk E M ,

k

= 0,1,.

. .,

(30)

and this is an N-stage minimax control problem. The infinite horizon version is minimize J.(x,)

=

lim

sup

(Nil

a k g [ X k,uk(xk), . wk]

N + m w k E W[X~. p~(xk)I k = 0

subjectto x,+,=f[x,,,u,(x,),w,],

EM, k = 0 , 1 , . . . .

(31)

The limit in (31) exists under any one of the conditions (21), (22), or (23). This problem contains as a special case the problem of infinite time reachability examined in Bertsekas [B2]. Problems (30)and (31)arise also in the analysis of sequential zero-sum games.

Chapter 3

Finite Horizon Models

3.1 General Remarks and Assumptions

Consider the N-stage optimization problem minimize JN,,(x) = (T,; . . T,,_ ,)(Jo)(x) subject to n = ( p o ,p l , . . .) E l3, where for every p E M, J E F , and x E S we have T , ( J ) ( x ) = H [ x , p(x), J ] ,

T ( J ) ( x )= inf H ( x , u, J ) . U€

U(x)

Experience with a large variety of sequential optimization problems suggests that the N-stage optimal cost function J$ satisfies JR

=

inf J N , , = TN(J,), n s II

and hence is obtained after N steps of the DP algorithm. In our more general setting, however, we shall need to place additional conditions on H in order to guarantee this equality. Consider the following two assumptions. Assumption F.l If [ J , ) c F is a sequence satisfying J,., and H ( x , u, J , ) < co for all x E S , u E U ( x ) ,then

lim H ( x , u, J,) k-

n,

=

H

I J k for all k

V xES, u E U(x).

3.

40

. FINITE HORIZON MODELS

Assumption F.2 There exists a scalar u E (0, oo) such that for all scalars r E (0, oo) and functions J E F, we have

We will also consider the following assumption, which is admittedly somewhat complicated. It will enable us to obtain a stronger result on the existence of nearly optimal policies (Proposition 3.2) than can be obtained under F.2. The assumption is satisfied for the stochastic optimal control problem of Section 2.3.3, as we show in the last section of this chapter. Assumption F.3 There is a scalar and {E,) c R satisfy

J

=

lim J,,

JIJ,,

P E (0, oo) such that if J E F, {J,)

c F,

n = 1 , 2 ,...,

n+ m

Jn(4

5

J ( x ) + En, Jn-,(x)+E,,

n = 1,2,. . . and X E S with J ( x ) > - oo, n = 2 , 3 ,... and X E S with J ( x ) = - o o ,

then there exists a sequence {p,) c M such that

T ( J ) ( x )+PC, , Tp n - , (J,_,)(X)+PE,,

n = 1,2,. . . , X E S with T ( J ) ( x )> - oo, n=2,3,. . . , X E Swith T ( J ) ( x ) = - oo.

Each of our results will require at most one of the preceding assumptions. As we show in Section 3.3, at least one of these assumptions is satisfied by every specific model considered in Section 2.3. 3.2 Main Results The central question regarding the finite horizon problem is whether J$ = T N ( J o ) ,in which case the N-stage optimal cost function J,?j can be obtained via the DP algorithm that successively computes T ( J o ) ,T 2 ( J o ) ,. . . . A related question is whether optimal or nearly optimal policies exist. The results of this section provide conditions under which the answer to these questions is affirmative. Proposition 3.1 (a) Let F.l hold and assume that J,,,(x) < oo for all x € S , n € l I , a n d k = 1 , 2, . . . , N.Then

J,?j = T N ( J o ) .

(b) Let F.2 hold and assume that J z ( x ) > - cc for all x 1,2,. . . ,N. Then

and for every such that

E

E

S and k

=

> 0, there exists an N-stage &-optimalpolicy, i.e., a ~ , E I I

+ E.

J$ I JN,.c I J A

Proof (a) For each k = 0,1,. . . ,N - 1, consider a sequence { p i ) c M such that

lim T,; [ T N - k - l ( J o ) ] = T N - k ( J O ) ,

k=O,.. . , N - 1 ,

1-m

By using F.l and the assumption that Jk,.(x) < cc,we have

J g I inf. . . inf (T,;. . . T , N-1 ~ Nl -) ( J O ) io

= inf -

iN- 1

. . inf (T,:. . - T

io

= inf . . . io

i

~

%-2

i ~ - 2

[

-

I

inf) Tpi.w-i(Jo) x-1

~

iN-,

inf (T,: . . - T,~N -2)[T(Jo)] N -2

ix-2

where the last equality is obtained by repeating the process used to obtain the previous equalities. On the other hand, it is clear from the definitions of Chapter 2 that T N ( ~ I , ) J;, and hence JA = TN(J,). (b) We use induction. The result clearly holds for N = 1. Assume that it holds for N = k, i.e., J: = T k ( J o )and for a given E > 0, there is a X,E II with J,, I J,* + E. Using F.2 we have for all p E M,

.=

Hence J,*+, I T ( J Z ) , and by using the induction hypothesis we obtain J:+, I T k + l ( J 0 ) .On the other hand, we have clearly T ~ + ' ( JI~J,*+ ) and hence T k + ' ( J 0 )= J,*+,. For any F > 0, let ?? = ( p o , p l , . . .) be such that J,., I J,* + (F/2a), and let ,iiE M be such that TF(J,*)I T ( J f )+ (Z/2). Consider the policy 7f, = @,Po, & , . . .). Then

The induction is complete.

Q.E.D.

Proposition 3.l(a) may be strengthened by using the following assumption in place of F.1.

3.

FINITE HORIZON MODELS

Assumption F.l' The function J o satisfies

and if jJ,) c F is a sequence satisfying J,,

, I J k I J O for all k, then

The following corollary is obtained by verbatim repetition of the proof of Proposition 3.l(a).

Corollary 3.1.1 Let F.1' hold. Then

Proposition 3.1 and Corollary 3.1.1 may fail to hold if their assumptions are slightly relaxed.

COUNTEREXAMPLE 1 Take S = { O ) , C = U ( 0 )= ( - 1,0], Jo(0) = 0, H(0, u, J ) = u if - 1 < J(O), H(0, u, J ) = J ( 0 ) u if J(0) I - 1. Then (T,, . . . T,,_ l ) ( J o ) ( 0 = ) po(0) and Jg(0) = - 1, while T N ( J o ) ( 0= ) - N for every N. Here the assumptions J,, .(O) < co and J:(O) > - co are satisfied, but F.l, F.lf, and F.2 are violated.

+

COUNTEREXAMPLE 2 Take S = {O, I ) , C = U ( 0 )= U(1)= (- co,01, Jo(0)= J o ( l ) = 0 , H(0, u, J ) = u if J ( l ) = - co, H(0, u, J ) = 0 if J ( l ) > - co, and H ( l , u , J ) = u. Then (T,; . . T ,,_, ) ( J o ) ( 0 )= 0, (T,; . . T,,_,)(J0)(1) = PO(^) for all N 2 1. Hence, Jg(0) = 0 , Jg(1) = - co. On the other hand, we have P(J,)(O) = T N ( J o ) ( l= ) - co for all N 2 2. Here F.2 is satisfied, but F.l, F.l', and the assumptions J,. .(x) < co and J t ( x ) > - co for V x E S are all violated. The following counterexample is a stochastic optimal control problem with countable disturbance space as discussed in Section 2.3.2. We use the notation introduced there.

COUNTEREXAMPLE 3 Let N = 2, S = jO, I ) , C = U ( 0 )= U ( 1 )= R, W = ( 2 , 3, . . .), p ( ~ = k l x , u ) = k - ~ ( ~ , " _ , n - ~for ) - ~k = 2 , 3 ,..., ~ E S U, E C , f(O,u,w)= f ( l , u , w ) = 1 for V U E C ,W E W , ~ ( O , U , W ) = Wg,( l , u , w ) = u for V U EC , W E W . Then a straightforward calculation shows that J$(O) = w, J S ( 1 ) = - m, while T 2 ( J o ) ( 0 = ) - co, T 2 ( J o ) ( 1 )= - m. Here F.l and F.2 are satisfied, but F.l' and the assumptions Jk.,(x) < so for all x , ~k,, and J:(x) > - x for all x and k are all violated. The next counterexample is a deterministic optimal control problem as discussed in Section 2.3.1. We use the notation introduced there.

3.2

43

MAIN RESULTS

COUNTEREXAMPLE 4 Let N = 2, S = (0,1,. . .), C = U(x)= (0,co) for Vx E S, f ( x ,U ) = 0 for V X E S, u E C , g(0,U ) = - u for V UE U(O),g(x,U ) = x for V U EU ( x )if x # 0. Then for Z E ll and x # 0, we have J,,.(x) = x - ,u,(O), so that J:(x) = - oo for all X E S. On the other hand clearly there is no two-stage E-optimalpolicy for any E > 0. Here F.l, F.2, and the assumption J,,.(x) 4 oo for all x, n, k are satisfied, and indeed we have J;(x) = T ~ ( J , ) (=~-) co for Vx E S. However, the assumption Jz(x) > - co for all x and k is violated. As Counterexample 4 shows there may not exist an N-stage E-optimal policy ifwe have J,*(x)= - oo for some k and x E S. The following proposition establishes, under appropriate assumptions, the existence of a sequence of nearly optimal policies whose cost functions converge to the optimal cost function. Proposition 3.2 Let F.3 hold and assume J,,,(x) < cc for all x E S, n E II, and k = 1,2,. . . ,N. Then J$ = T N ( J o ) .

Furthermore, if (8,) is a sequence of positive numbers with E, LO, then there exists a sequence of policies {n,) exhibiting {E,}-dominated convergence to optimality. In particular, if in addition J$(x) > - co for all X E S, then for every E > 0 there exists an E-optimal policy. Proof We will prove by induction that for K N we have Jg = T ~ ( J , ) , and furthermore, given K and {en)with E, LO, E, > 0 for Vn, there exists a sequence {n,} c II such that for all n,

.,,

lim JK, n-+ cc

JK,

=Jf,

J g ( x )+ En J K .,n_, ( x )

.,(x)

+ E,

We show that this holds for K

JT(x)= inf J,..(x) Z

E

~

=

=

V x e S with J i ( x ) > - co, V X ES with J ~ ( x=) - a.

(2) (3)

1. We have

inf HEX,,LL(x), J O ]= T ( J O ) ( x ) V X ES. PEM

It is also clear that, given [E,), there exists a sequence (n,) c II satisfying (1)-(3) for K = 1. Assume that the result is true for K = N - 1. Let /3 be the scalar specified in F.3. Consider a sequence {en) c R with E, > 0 for Vn and limn,, en = 0, and let (it,} c ll,it, = (p;,,u;, . . .), be such that lim J N - l , e n = J $ - l , n+ m,

J N - I. %,,(x)

J A - l ( ~+) Pv1en V X E S with J z - l ( x ) > -so, ( 5 ) J . v - ~ . ~ ~ _ , ( x ) + P -V' cX, E S with J z - l ( x ) = - ~ . ( 6 )

3.

44


The assumption J,,,(x) < co for all x E S, n EkII =, 1,2,. . . ,N, guarantees that we have

I,"=,

Without loss of generality we assume that E, < co. Then Assumption F.3 together with ( 4 ) implies that there exists a sequence {p", c M such that, for all n,

T ( J g - 1)(x)+ En 5( , X

Tp;(JN-

)

if T ( J $ - l ) ( x )> - co, (9) if T(J;- I)(.) = - co. (10)

+E

We have by the induction hypothesis J g - , T N ( J o )s J $ . Hence,

=

T N - ' ( J , ) , and it is clear that

We also have J $ 5 lim T,;(JN-l.,n) n+m

Combining (8), ( l l ) ,and (12),we obtain

Let nn = ( p ; , p ; , & , . . .). Then from (8)-(10) and (13), we obtain, for all n, lim JN,."

=J$,

n - +co

JN,

, (XI

J $ ( x ) + ~n JN,,n_ ( x )

+ E,

V XE S V xES

and the induction argument is complete.

with J $ ( x ) > - CD, with J ~ ( x =) - a,

Q.E.D.

Despite the need for various assumptions in order to guarantee J$ = T N ( J o ) ,the following result, which establishes the validity of the DP algorithm as a means for constructing optimal policies, requires no assumption other than monotonicity of H. Proposition 3.3 A policy n* if and only if

Proof

= ( p t ,pT,

. . .) is uniformly N-stage optimal

Let (14)hold. Then we have, for k = 0,1,. . . ,N - 1,

On the other hand, we have J g - , I (T,:. . . T,;- l)(Jo),while T N - k ( J oI) J g - k . Hence, Jg = (T,;. . . T,; - ,)(Jo)and z*is uniformly N-stage optimal. Conversely, let z*be uniformly N-stage optimal. Then

-,

by definition. We also have for every EM, (T,T)(Jo)= (T,T,;_,)(J0), which implies that

T 2 ( ~= o )inf (T,T)(Jo)= inf (T,T,;_ $ J o ) 2 J $ = (T,;_ 2TP;-, ) ( J o ) 2 T2(Jo).

Therefore T ~ ( J=~J $) = (Tpr;_2TP;,)(Jo)= ( T P ; - ~ T ) ( J o ) . Proceeding similarly, we show all the equations in (14).

Q.E.D.

As a corollary of Proposition 3.3, we have the following. Corollary 3.3.1 (a) There exists a uniformly N-stage optimal policy if and only if the infimum in the relation

Tk+'(Jo)(x) = inf H[x,u, T k ( J 0 ) ]

(15)

u s U(x)

.

is attained for each x E S and k = 0,1,. . . N - 1. (b) If there exists a uniformly N-stage optimal policy, then We now turn to establishing conditions for cxistcnce of a uniformly N-stage optimal policy. For this we need compactness assumptions. If C is a Hausdorff topological space, we say that a subset U of C is compact if every collection of open sets that covers U has a finite subcollection that covers U . The empty set in particular is considered to be compact. Any sequence {u,} belonging to a compact set U c C has at least one accumulation point U E U,i.e., a point i i U ~ every (open) neighborhood of which contains an infinite number of elements of [u,). Furthermore, all accumulation points of (u,} belong to U. If [U,} is a sequence of nonempty compact subsets of C and U, 3 U,, for all n, then the intersection U, is nonempty and compact. This yields the following lemma, which will be useful in what follows.

,

Lemma 3.1 Let C be a Hausdorff space, f:C U a subset of C. Assume that the set U ( i )defined by

+ R*

a function, and

is compact for each iE R. Then f attains a minimum over U

46

3.


Proof If f(u) = oo for all U E U, then every U E U attains the minimum. Iff * = inf( f(u)lu E U) < oo,let {in}be a scalar sequence such that An > An+ for all n and 3,, + f *. Then the sets U(in) are nonempty, compact, and satisfy U(/?,) 3 U(/Zn+,)for all n. Hence, the intersection U(1,,) is nonempty and compact. Let u* be any point in the intersection. Then I,, for all n, and it follows that f(u*) < f *. Hence, f u* E U and f(u*) I attains its minimum over U at u*. Q.E.D.

,

Direct application of Corollary 3.3.1 and Lemma 3.1 yields the following proposition. Proposition 3.4 Let the control space C be a Hausdorff space and assume that for each ~ E S%, E R ,and k = 0,1,. . . ,N - 1, the set

is compact. Then

Jg

=

TN(Jo),

and there exists a uniformly N-stage optimal policy. The compactness of the sets U,(x, 1")of (16) may be verified in a number of important special cases. As an illustration, we state two sets of assumptions which guarantee compactness of U,(x,3.) in the case of the mapping

corresponding to a deterministic optimal control problem (Section 2.3.1). a(x, u), b I g(x, u) < oo for some b E R and all x E S, Assume that 0 I u E U(x), and take J, 0. Then compactness of U,(x, i ) is guaranteed if: (a) S = Rn (n-dimensional Euclidean space), C = Rm, U(x) E C, f , g, and a are continuous in (x, u), and g satisfies lim,,, g(x,, u,) = oo for every bounded sequence (x,) and every sequence (u,: for which lukl + a (1.1 is a norm on Rm); (b) S = Rn, C = Rm,f, g, and a are continuous, U(x) is compact and nonempty for each x E Rn, and U(.) is a continuous point-to-set mapping from Rn to the space of all nonempty compact subsets of Rm.The metric on this space is given by (3) of Appendix C. The proof consists of verifying that the functions Tk(J,), k = 0,1,. . . , N - 1, are continuous, which in turn implies compactness of the sets U,(x, 2 ) of (16). Additional results along the lines of Proposition 3.4 will be given in Part I1 (cf. Corollary 8.5.2 and Proposition 8.6).

3.3

APPLICATION TO SPECIFIC MODELS

3.3 Application to Specific Models We will now apply the results of the previous section to the models described in Section 2.3. Stochastic Optimal Control-Outer

Integral Formulation

Proposition 3.5 The mapping H ( x , u, J ) = E* [ g ( x ,u, w)

+ a J [ f ( x ,u, w)]Ix,u>

of Section 2.3.3 satisfies Assumptions F.2 and F.3. We have

Proof

H(x. u. J ) = J*.[g(x,u, w) + a J [ f ( x , u, w)])p(dwlx,u), where 1" denotes the outer integral as in Appendix A. From Lemma A.3(b) we obtain for all x E S, u E C , J E F , r > 0,

Hence, F.2 is satisfied. We now show F.3. Let J E F, { J , ) c F , [E,) c R satisfy E, > 0, and for all n, J

=

lim J,, n-oc

Let

jp,)

ln)

I,"=, < co, E,

J I J,,

M be such that for all n, T,,iJ)(x) 5

i

T ( J ) ( x )i- E , - 1/&n

if T ( J ) ( x )> - co, if T ( J ) ( x )= - a,

T p n ( J )5 TZ,,- ,(J).

(22) (23) (24)

Consider the set A ( J ) = jx

E Slthere exists u E

U ( x )with p*(.(wIJ [ f ( x ,u, w)] = - a]lx, u) > 01,

where p* denotes p-outer measure (see Appendix A). Let P*(jw)J[f ( x , p ( x ) ,w)] = - m ) I x , p ( x ) )> 0

p E M be such that V X EA(J).

(25)

48

3.


Define for all n ( x )

if

P,(x)

if x $ A ( J ) .

X E

A(J),

We will show that {p,) thus defined satisfies the requirement of F.3 with /3 = 1 + 2a. For X E A(J),we have, from Corollary A.l.l and (18)-(21), lim sup TPn(Jn)(x) = lim sup TE(J,)(x) n-t

n+ m

m

= lim sup n-t

m

J*{g[x,

P(X)

wl

+ ~ J . [ f ( xP(x), . w)1)

It follows from Lemma A.3(g) and the fact that T,(J)(x)< co [cf. ( 1 8 ) and (21)]that lim sup Tp,(J,)(x)= - co I T(J)(x). n-+ m

For x $ A(J),we have, for all n,

Take B, E F to contain {wlJ[ f ( x ,pn(x),w)] = - co) and satisfy Using Lemma A.3(e) and (b) and (19), we have

Hence, for x $ A(J) we have from (28),(22),and (23)that I lim sup T p n ( J ) ( x=) T ( J ) ( x ) . lim sup Tpn(Jn)(x) n-tm

n+ m

Combining (27)and this relation we obtain lim sup Tp,(Jn)(x) I T(J)(x) n- rn

Vx E S,

(27)

3.3


and since T,,(J,) 2 T(J) for all n, it follows that lim Tp,(J,) = T(J).

(29)

n+ m

If x is such that T(J)(x)> - co, it follows from (27) and (29) that we must have x $ A(J). Hence, from (28), (22), and Lemma A.3(b), T,,(J,)(x) l T,,(J)(x)

+

~ C I E ,I

+ + 2a)En

T(J)(x) (1

if

T(J)(x) > - 00. (30)

If x is such that T(J)(x) = - m, there are two possibilities: (a) x 4 A ( J ) and (b) X E A(J). If x 4 A(J), it follows from (28), (24), and (18) that Tp,(Jn)(x)

T,,(J)(x)

+ 2%

I Tp,-,(J)(x) + HE, I Tp,_,(Jn-l)(x)+ ~ u E , .

(31)

If x E A(J), then by (18)-(20) and Lemma A.3(b),

-

Tp,-l(Jn-l)(~)+ 2ae,.

(32)

It follows now from (29)-(32) that (p,) satisfies the requirement of F.3 Q.E.D. with p = 1 + 2a. As mentioned earlier, mapping (17)contains as special cases the mappings of Sections 2.3.1 and 2.3.2. In fact, for those mappings F.l is satisfied as well, as the reader may easily verify by using the monotone covergence theorem for ordinary integration. Direct application of the results of the previous section and Proposition 3.5 yields the following. Corollary 3.5.1

Let H be mapping (17) and let J,(x)

=0

for V ~ S.E

(a) I f J , ~ , ( x ) < c o f o r a l l x ~ S , ~ ~ ~ , a n d,k. .=. ,lN, 2, t h e n J E = TN(Jo)and for each sequence {en)with en -10, en > 0 for Vn, there exists a sequence of policies (n,) exhibiting [en)-dominated convergence to optimality. In particular, if in addition Jz(x) > - m for all x E S, then for every E > 0 there exists an E-optimal policy. (b) If J,*(x)> - oc for all ~ E Sk ,= 1,2,. . . ,N, then J; = TN(J0)and for each e > 0 there exists an N-stage e-optimal policy. (c) Propositions 3.3 and 3.4 and Corollary 3.3.1 apply.

50

3.


As Counterexample 3 in the previous section shows, it is possible to have J i # TN(Jo)in the stochastic optimal control problem if the assumptions of parts (a) and (b) of Corollary 3.5.1 are not satisfied. Naturally for special classes of problems it may be possible to guarantee the equality J i = TN(Jo) in other ways. For example, if the problem is such that existence of a uniformly N-stage optimal policy is assured, then we obtain J i = TN(Jo)via Corollary 3.3.l(b). An important special case where we have J i = TN(Jo) without any further assumptions is the deterministic optimal control problem of Section 2.3.1. This fact can be easily verified by the reader by using essentially the same argument as the one used to prove Proposition 3.l(a). However, if J i ( x ) = - co for some x E S, even in the deterministic problem there may not exist an N-stage &-optimalpolicy for a given E (see Counterexample 4). Stochastic Optimal Control-Multiplicative

Cost Functional

Proposition 3.6 The mapping

of Section 2.3.4 satisfies F.1. If there exists a b~ R such that 0 I g(x, u, w) I b for all x E S, u E U(x), w E W , then H satisfies F.2. Proof Assumption F.l is satisfied by virtue of the monotone convergence theorem for ordinary integration (recall that W is countable). Also, if 0 I g(x, u, w) I b, we have for every J E F and r > 0, H(x, u, J

+ r) = E{g(x, u, w)(J[f = El&,

Thus F.2 is satisfied with cx

t ~w)J[f ,

= b.

(x, u, w)] + r)lx, u> (x, u, w)] Ix, u ) + rE jg(x, u, w)/x,u).

Q.E.D.

By combining Propositions 3.6 and 3.1, we obtain the following. Corollary 3.6.1

Let H be the mapping (33) and Jo(x)= 1 for V ~ E S .

(a) IfJ,.,(x)< c o f o r a l l x ~ S , n ~ n , k1,2 = , . . . ,N , t h e n J i = T N ( J o ) . (b) If there exists a b E R such that 0 I g(x, u, w) I b for all x E S, u E U(x), W E W , then J i = TN(Jo)and there exists an N-stage &-optimalpolicy. (c) Propositions 3.3 and 3.4 and Corollary 3.3.1 apply. We now provide two counterexamples showing that the conclusions of parts (a) and (b) of Corollary 3.6.1 may fail to hold if the corresponding assumptions are relaxed. COUNTEREXAMPLE 5 Let everything be as in Counterexample 3 except that C = (0. cc) instead of C = R (and, of course, Jo(0) = J o ( l )= 1 instead

3.3

51


of Jo(0)= J o ( l )= 0). Then a straightforward calculation shows that J;(O) = a,J;(1) = 0, while T2(Jo)(0)= T2(Jo)(1)= 0. Here the assumption that J,. .(x) < cc for all x, n, k is violated, and g is unbounded above. COUNTEREXAMPLE 6 Let everything be as in Counterexample 4 except for the definition of g. Take g(0,u) = u for V U EU(0) and g(x,u) = x for Vu E U ( X )if x # 0. Then for every .n E I7 we have J 2 ,.(x) = xp,(O) for every x # 0, and J;(x) = 0 for V x E S. On the other hand, there is no two-stage E-optimal policy for any E > 0. Here the assumption J,.,(x) < cc for all x,n,k is satisfied, and indeed we have J z ( x ) = T 2 ( ~ , ) ( x=) 0 for V x e S . However, g is unbounded above.

Minimax Control Proposition 3.7 The mapping

H ( x , u , J ) = sup j y ( x , u , w ) + a J [ f ( x , u , w ) ] }

(34)

w e W ( x ,U )

of Section 2.3.5 satisfies F.2.

Proof

We have for r > 0 and J EF ,

H(x, u, J

+ r) =

SUP w e W ( x .u)

{g(x,U ,W )

= H(x, u, J )

Corollary 3.7.1

+ ar.

+ a J [ f ( x ,U ,w)] + ctr) Q.E.D.

Let H be mapping (34)and Jo(x)= 0 for V x E S.

), (a) If J,*(x)> - cc for all X E S, k = 1,2,. . . ,N , then J; = T N ( ~ , and for each E > 0 there exists an N-stage E-optimal policy. (b) Propositions 3.3 and 3.4 and Corollary 3.3.1 apply. , it is clearly possible that If we have J:(x) = - cc for some ~ E Sthen there exists no N-stage &-optimalpolicy for a given E > 0, since this is true even for deterministic optimal control problems (Counterexample 4). It is also possible to construct examples very similar to Counterexample 3 which show that it is possible to have Jg f T N ( J o )if J,*(x)= - cc for some x and k.

Chapter 4

Infinite Horizon Models under a Contraction Assumption

4.1 General Remarks and Assumptions Consider the infinite horizon problem minimize J,(x)

=

lim (T,,T,, . . . T,,_ ,)(J,)(x) N+

subject to n

02

= (pO,,uI,.. .)E

ll.

The following assumption is motivated by the contraction property of the mapping associated with discounted stochastic optimal control problems with bounded cost per stage (cf. DPSC, Chapter 6). Assumption C (Contraction Assumption) There is a closed subset B of the space B (Banach space of all bounded real-valued functions on S with the supremum norm) such that J, E B, and for all J E B, ,u E M, the functions T(J) and T,(J) belong to B. Furthermore, for every n = (p,, p , , . . .)E ll, the limit

exists and is a real number for each x E S. In addition, there exists a positive integer m and scalars p, a, with 0 < p < 1, 0 < a, such that

Condition (3)implies that the mapping (T,,T,, . . . T,__ ,)is a contraction ~ k = 0,1,. . . ,m - 1. When nz = 1, the mapping mapping in B for all p , M, T, is a contraction mapping for each y~ M. Note that (2) is required to hold on a possibly larger set of functions than (3). It is often convenient to take B = B. This is the case for the problems of Sections 2.3.1, 2.3.2, and 2.3.5 assuming that a < 1 and g is uniformly bounded above and below. We will demonstrate this fact in Section 4.4. In other problems such as, for example, the one of Section 2.3.3, the contraction property (3) can be verified only on a strict subset B of B. 4.2 Convergence and Existence Results We first provide some preliminary results in the following proposition. Proposition 4.1 Let Assumption C hold. Then: (a) For every J E B and J,

=

E I'I, we

have

lim (T,, . . . T,,_ ,)(Jo)= lim (T,, . . . T,,_ ,)(J). N - no

N+m

(b) For each positive integer N and each J EB, we have inf (T,,. . . T,,

_

,)(J) = TN(J)

nsn

and, in particular,

Ji =

inf (T,,. . . T,,_ l)(Jo) = TN(~,). Z

E

~

(c) The mappings Tw'and T;, y E M, are contraction mappings in B with modulus p, i.e.,

I T"(J) I~T;(J)

T~(J')~I - T;(J')I -

J'I I pJIJ - ~ ' 1 1 I p l l ~-

VJ, J'EB,

VJ, J'EB,

+

EM

Proof' (a) For any integer k 2 0, write k = nm q, where y, n are nonnegative integers and 0 q < nl. Then for any J. J'EB, using (2) and (3), we obtain

54

4.

INFINITE HORIZON MODELS UNDER A CONTRACTION ASSUMPTION

from which, by taking the limit as k (and hence also n) tends to infinity, we have lim(T,;,.T

k-+ w

,,_, )(Jo)= k-+ lim(T,;..T ,,-, )(J) m

VJEB.

(b) Since T ~ ( J ) E Bfor all k by assumption, we have Tk(J)(x)> - oo forallx~Sandk.Forany& > O , l e t p k ~ M , k = O , l ., .. , N - 1,besuchthat

Using (2) we obtain

2 inf (T,; KC

. . T,,_ ,)(J) -

II

Since E > 0 is arbitrary, it follows that TN(J)2 inf (T,; rr E n

. . T,,_ ,)(J).

The reverse inequality clearly holds and the result follows. (c) The fact that T r is a contraction mapping is immediate from (3). We also have from (3) for all pkeM, k = 0,. . . ,m - 1, and J , J'EB, (T,;

. . T,,,_,)(J) 5 (T,;

. . T,_-,)(J') + pllJ - J'11.

Taking the infimum of both sides over pk E M, k using part (b) we obtain

= 0,

1,. . . , m - 1, and

A symmetric argument yields

Combining the two inequalities, we obtain lITm(J)- Tm(J')III pl(J- J'Il. Q.E.D.

In what follows we shall make use of the following fixed point theorem. (See [ 0 5 , p. 3831-the proof found there can be generalized to Banach spaces.) Fixed Point Theorem If B is a closed subset of a Banach space with norm denoted by 11. 11 and L : B -+ B is a mapping such that for some positive integer m and scalar p ~ ( 0I), , IILm(z)- Lm(zr)llI ~ [ l-z zlll for all z, ~ ' E Bthen , L has a unique fixed point in B, i.e., there exists a unique vector z* E B such that L(z*) = z*. Furthermore, for every Z E B, we have lim [ILN(z)- z*[l = 0. N+ w

The following proposition characterizes the optimal cost function J * and the cost function J, corresponding to any stationary policy (p, p, . . .) E II.It also shows that these functions can be obtained in the limit via successive application of T and T, on any J E B. Proposition 4.2 Let Assumption C hold. Then: (a) The optimal cost function J * belongs to B and is the unique fixed point of T within B, i.e., J * = T(J*),and if J' €Band J' = T(Jf),then J' = J*. Furthermore, if J' E B is such that T(J1)I J', then J * I J', while if J' I T(Jf), then J' I J*. (b) For every p E M, the function J, belongs to B and is the unique fixed point of T, within B. (c) There holds

- J,I[ = 0 lim lITfj(~)

VJEB,

EM.

N-+ w

Proof From part (c) of Proposition 4.1 and the fixed point theorem, we have that T and T, have unique fixed points in B. The fixed point of T, is clearly J,, and hence part (b) is proved. Let J"* be the fixed point of T. We . any P > 0, take p E M such that have J"* = ~ ( 1 " )For

+ 2. From (2) it follows that T;(J"*) I T~(J"*)+ aZ I J"* + (1 + a)Z. Continuing T ~ ( J *I ) J*

in the same manner, we obtain u

T$(J*) I j*

+ (1 + u + . . . + am-')P.

Using (3) we have T;~(J*)I T;(J*) + p(i + a + . . . + am-l ) ~ I J * + (1 + p)(l + u + . . . + am-')z.

56

4.


Proceeding similarly, we obtain, for all k 2 1,

+

+ +

+

T$(J"*) I J"* + (1 + + . . . pk-l)(l CI . . . am-')z. Taking the limit as k -. a3 and using the fact that J, = lim,,, T;~(J"*),we have

+ cc + . . . + am-')-'&, we obtain J, I J * + Since J * I J, and E > 0 is arbitrary, we see that J * 2. We also have J * = inf lim (T,, . . . T,,_ ,)(J*)2 lim T~(J"*)= J*.

Taking Z= (1 - p)(l

8.

N-r m

npIIN+o,

Hence J * = J * and J * is the unique fixed point of T. Part (c) follows immediately from the fixed point theorem. The remaining part of (a) follows easily from part (c) and the monotonicity of the mapping T. Q.E.D. The next proposition relates to the existence and characterization of stationary optimal policies. Proposition 4.3 Let Assumption C hold. Then: (a) A stationary policy n* = (p*,p*, . . .)EII is optimal if and only if T,,(J*) = T(J*). Equivalently, n* is optimal if and only if T,*(J,*) = T(J,*). (b) If for each X E S there exists a policy which is optimal at x, then there exists a stationary optimal policy. (c) For any E > 0, there exists a stationary &-optimal policy, i.e., a n, = (p,, p,, . . . ) ~such n that JIJ* -

J,*I

I E.

Proof (a) If TC*is optimal, then J,, = J * and the result follows from parts (a) and (b) of Proposition 4.2. If T,,(J*) = T(J*), then T,,(J*) = J*, and hence J,, = J * by part (b) of Proposition 4.2. If T,,(J,,) = T(J,,), then J,, = T(J,,) and J,, = J * by part (a) of Proposition 4.2. (b) Let 71: = (p:,., p?.,, . . .) be a policy which is optimal at x E S. Then using part (a) of Proposition 4.1 and part (a) of Proposition 4.2, we have

Hence T,;, ,(J*)(x) = T(J*)(x)for each x. Define p* E M by means of p*(x) = ~ : , ~ ( xThen ) . T,,(J*) = T(J*)and the stationary policy (p*, p*, . . .) is optimal by part (a). (c) This part was proved earlier in the proof of part (a) of Proposition 4.2 [cf. (4)]. Q.E.D. Part (a) of Proposition 4.3 shows that there exists a stationary optimal policy if and only if the infimum is attained for every x E S in the optimality equation J*(x) = T(J*)(x) = inf H(x, u, J*). UE U ( X )

Thus if the set U(x) is a finite set for each x E S , then there exists a stationary optimal policy. The following proposition strengthens this result and also shows that stationary optimal policies may be obtained in the limit from finite horizon optimal policies via the D P algorithm, which for any given J E B successively computes T(J), T2(J),. . . . Proposition 4.4 Let Assumption C hold and assume that the control space C is a Hausdorff space. Assume further that for some J E B and some positive integer k, the sets

are compact for all x E S, 3. E R, and k 2 k. Then: (a) There exists a policy TC*= (pg, p?, . . .)Ell attaining the infimum for all x E S and k 2 k in the D P algorithm with initial function J , i.e., (T,;T"(J)

=

'

T k f (J)

V k 2 k.

(6)

(b) There exists a stationary optimal policy. (c) For every policy n* satisfying (6), the sequence {pt(x)J has at least one accumulation point for each s E S. (d) If p* :S + C is such that p*(x) is an accumulation point of [pt(x)) for each x E S, then the stationary policy (p*, p*, . . .) is optimal. Proof

(a) We have

T,+ '(J)(x) = inf H[x, u, TYJ)], UE

U(x)

and the result follows from compactness of sets (5) and Lemma 3.1. (b) This part will follow immediately once we prove (c) and (d). (c) Let n* = (p:, p?,. . .) satisfy (6) and define

58

4.


We have from (2), (6), and the fact that T ( J * ) = J*,

From these two relations we obtain

1 all n 2 k and k 2 5 and {p:(x)) It follows that p , * ( x ) ~U,[x, J * ( x ) + 3 ~ x 8 ~for has an accumulation point by the compactness of U k [ x ,J * ( x ) + 3ask]. (d) If p*(x) is an accumulation point of {p:(x)), then p * ( x ) ~ U,[x, J * ( x ) 3ae,] for all k 2 E, or equivalently,

+

By using (2), we have, for all k,

II(T,*Tk)(J) - T,*(J*)II I allTk(J)- J*ll 5 as,. Combining the preceding two inequalities, we obtain

Since ck -+ 0 [cf. Proposition 4.2(c)], we obtain T,,(J*) < J*. Using the fact that J* = T ( J * ) I T,*(J*), we obtain T,,(J*) = J*, which implies by Proposition 4.3 that the stationary policy (p*,p*, . . .) is optimal. Q.E.D. Examples where compactness of sets (5) can be verified were given at the end of Section 3.2. Another example is the lower semicontinuous stochastic optimal control model of Section 8.3. 4.3 Computational Methods

There are a number of computational methods which can be used to obtain the optimal cost function J* and optimal or nearly optimal stationary policies. Naturally, these methods will be useful in practice only if they require a finite number of arithmetic operations. Thus, while "theoretical" algorithms which require an infinite number of arithmetic operations are of

4.3

59

COMPUTATIONAL METHODS

some interest, in practice we must modify these algorithms so that they become computationally implementable. In the algorithms we provide, we assume that for any J E B and E > 0 there is available a computational method which determines in a finite number of arithmetic operations functions J, E B and p, E M such that For many problems of interest, S is a compact subset of a Euclidean space, and such procedures may be based on discretization of the state space or the control space (or both) and piecewise constant approximations of various functions (see e.g., DPSC, Section 5.2). Based on this assumption (the limitations of which we fully realize), we shall provide computationally implementable versions of all "theoretical" algorithms we consider. 4.3.1 Successive Approximation

The successive approximation method consists of choosing a starting function J E B and computing successively T(J), TZ(J),. . . , Tk(J),. . . . By part (c) of Proposition 4.2, we have lim,,, IITk(J)- J*ll = 0, and hence we obtain in the limit the optimal cost function J*. Subsequently, stationary optimal policies (if any exist) may be obtained by minimization for each x E S in the optimality equation J*(x) = inf H(x,u, J*). x€ U(x)

If this minimization cannot be carried out exactly or if only an approximation to J * is available, then nearly optimal stationary policies can still be obtained, as the following proposition shows. Proposition 4.5 Let Assumption C hold and assume that

y* EB and

p E M are such that

2 0,

where

E~

2 0 are scalars. Then

J* I J, I J * + [(2ael Proof

Using (2) we obtain

and it follows that

+ eZ)(l+ a + . . . + am-')/(l - p)].

60

4.


Using this inequality and an argument identical to the one used to prove ( 4 ) in Proposition 4.2, we obtain our result. Q.E.D. An interesting corollary of this proposition is the following. Corollary 4.5.1 Let Assumption C hold and assume that S is a finite set and U ( x )is a finite set for each x E S. Then the successive approximation method yields an optimal stationary policy after a finite number of iterations in the sense that, for a given J E B,if n* = (pg,p:, . . .) E II is such that

then there exists an integer k such that the stationary policy ( p t ,pz, . . .) is optimal for every k 2 k. Proof Under our finiteness assumptions, the set M is a finite set. Hence there exists a scalar E* > 0 such that J , I J* E* implies that ( p ,p,. . .) is optimal. Take k sufficiently large so that IITk(J)- J*ll 5 P for all k 2 E, wherez satisfies 2aP(1 + a + . . . + am- ' ) ( l - p)-' I E * , and use Proposition 4.5. Q.E.D.

+

The successive approximation scheme can be sharpened considerably by making use of the monotonic error bounds of the following proposition. Proposition 4.6 Let Assumption C hold and assume that for all scalars r # 0, J E B, and x E S, we have a, 5 [ T m ( J

+ r ) ( x )- T m ( J ) ( x ) ] / rI

a,,

(7)

where a,, a , are two scalars satisfying 0 I a , I a , < 1. Then for all J E B , ~ E Sand , k = 1,2,. . . ,we have

where

6, = max d,

=

inf [ T k m ( J ) ( x) T(k-')m(J)(x)], XES

-----a,,-----

1 a

I

- a,

a, = s u p [ T k m ( J ) ( x-) T ( k - ' ) m ( J ) ( x ) ] . xcs

Note If B = B we can always take a , = p, a , = 0, but sharper bounds are obtained if scalars cn, and a , with 0 < a , and/or a , < p are available. Proof It is sufficient to prove (8) for k = 1, since the result for k > 1 then follows by replacing J by T ( k - ' ) m ( J ) .In order to simplify the notation. we assume m = 1. In order to prove the result for the general case simply

4.3

61


replace T by Tmin the following arguments. We also use the notation dl

= d,

-

dl

=a,

d2 = d',

Relation (7) may also be written (for m T(J) + min[nlr,a2r] I T ( J

=

-

d2 = 2.

1) as

+ r) I T(J) + max[a1r,a2v].

(9)

We have for all x E S, J(x)

+ d I T(J)(x).

(10)

Applying T on both sides of (10) and using (9) and (lo), we obtain

By adding min[aId, a;d] to each side of these inequalities, using (9) (with J replaced by T(J) and v = min[a,d, a2d]), and then again (ll), we obtain

Proceeding similarly, we have for every k

Taking the limit as k

+ co,we

=

1,2,. . . ,

have

I

J(x) + min ------d, -d I T(J)(x) min [ l a 1-a2

+

r T~(J)(x)+ min ----d, 11 :al

I J*(x).

Also, we have from (11) that

and by taking the infinum over x E S, we see that

-d l 1 - n2

62

4.


It is easy to see that this relation implies min

[

a;

---, d

,

1 -a2

]

I min [-dl, 1-a

~ d ' 1-a2

]

.

(13)

Combining (12)and (13)and using the definition of b, and b,, we obtain Also from (12) we have T ( J ) ( x )+ b, I J*(x), and an identical argument ) b, I J*(x). Hence the left part of (8) is proved for shows that T 2 ( J ) ( x+ k= 1, m = 1. The right part follows by an entirely similar argument. Q.E.D. Notice that the scalars b, and 5, in (8)are readily available as a byproduct of the computation. Computational examples and further discussion of the error bounds of Proposition 4.6 may be found in DPSC, Section 6.2. By using the error bounds of Proposition 4.6, we can obtain J* to an arbitrary prespecified degree of accuracy in a finite number of iterations of the successive approximation method. However, we still do not have an implementable algorithm, since Proposition 4.6 requires the exact values of the functions Tk(J).Approximations to T k ( J )may, however, be obtained in a cornputationally implementable manner as shown in the following proposition, which also yields error bounds similar to those of Proposition 4.6. Proposition 4.7 Let Assumption C hold. For a given J E B and consider a sequence (J,) c B satisfying

E

> 0,

+

T ( J )I J , I T ( J ) E, T(J,) 5 J k f l I T ( J k )+ E,

k

=

1,2,. . . .

Then

T ~ ~ ( J ) - J , ~ Ik =, o , i , . . . , where -

E = &(I

+ + . . . + CLm-l)/(l- p) %

Furthermore, if the assumptions of Proposition 4.6 hold, then for all x E S and k = 1,2,.. .

4.3


Proof

We have

An identical argument yields

and we also have

IITm(Jm)- T2"(J)llI p l l J m - Tm(J)IJ. Using the preceding three inequalities we obtain l J J z m - T2"(J)ll I IIJzm - ~ ~ ( ~ r I ( 1 p)E(l + a

+

Proceeding similarly we obtain, for k

+ JJTm(Jm) - T2"(J)ll

n ) J l

+ - . .+am-').

=

1,2,. . . ,

and (14)follows. The remaining part of the proposition follows by using (14) and the error bounds of Proposition 4.6. Q.E.D. Proposition 4.7 provides the basis for a computationally feasible algorithm to determine J* to an arbitrary degree of accuracy, and nearly optimal stationary policies can be obtained using the result of Proposition 4.5.

4.3.2 Policy Iteration The policy iteration algorithm in its theoretical form proceeds as follows. An initial function p, E M is chosen, the corresponding cost function J,, is computed, and a new function p , E M satisfying T,,(J,,) = T(J,,) is obtained. More generally, given pk E M, one computes J,, and a function p,+ E M satisfying T,,,xn), ... and assume B = B. From part (a) of Proposition 4.2, we have that whenever J EB and J 5 T(J), then J 5 J*. Hence the values J*(x,),. . . ,J*(x,) solve the mathematical programming problem fl

maximize

C ,Ii i= 1

subject to

H(xi,u, J,),

i = 1,. . . ,n, u E U(xi),

where J, is the function taking values J,(xj) = /Zi,i = 1,. . . ,n. If U(xj) is a finite set for each i, then this problem is a finite-dimensional (possibly nonlinear) programming problem having a finite number of inequality constraints. In fact, for the stochastic optimal control problem of Section 2.3.2, this problem is a linear programming problem, as the reader can easily verify (see also DPSC, Section 6.2). This linear program can be solved in a finite number of arithmetic operations.

68

4.


4.4 Application to Specific Models

The results of the preceding sections apply in their entirety to the problems of Sections 2.3.3 and 2.3.5 if a < 1 and g is a nonnegative bounded function. Under these circumstances Assumption C is satisfied, as we now show. Stochastic Optimal Control-Outer Proposition 4.10

Integral Formulation

Consider the mapping

H(x, u, J ) = E* ( g ( x ,u, w) + u J [ f ( x ,u, w)]lx, u )

(30)

of Section 2.3.3 and let J o ( x )= 0 for V x E S. Assume that cc < 1 and for some b E R there holds Then Assumption C is satisfied with B equal to the set of all nonnegative functions J E B, the scalars in (2) and (3)equal to 2a and a, respectively, and m = 1. Note If the special cases of the mappings of Sections 2.3.1 and 2.3.2 are considered, then B can be taken equal to B, and the scalars in (2) and (3) can both be taken equal to a. Proof Clearly Jo E B and T ( J ) , T,(J) E B for all J E B and y E M. We also have, for any = ( y o ,y l , . . .)~ll,

and hence limN,,(T,; . . T P N ]- ) ( J O ) ( xexists ) for all x verify inductively using Lemma A.2 that

E

S. It is also easy to

N- 1

(T,, . . . T,,_ , ) ( J o ) ( x I)

C

k= 0

akb

b / ( l - a)

VSES, N

=

Hence lim,,,(T,; . . T,,_l)(Jo)(x)is a real number for every x. We have for all X E S ,J. J ' E B , EM, and W E W ,

Hence, using Lemma A.3(b),

1,2,. .

4.4


Hence

A symmetric argument yields T,(J')(x) - T,(J)(x) 5 2allJ

-

J'll. Therefore,

Taking the supremum of the left side over x E S, we have IIT,(J) - T,(J1)II s 2~115- J'II

V,UEM , J , J ' E B,

(33)

which shows that (2)holds. If J , J ' E B, then from (31), Lemma A.2, and Lemma A.3(a), we obtain in place of (32) E*{glx, A x ) , wl + ccJ[f(x,A x ) , w)l lx, u ) I E*(g[x,A x ) , wl + a J 1 l f ( x~ , ( w)] 4 ,Ix, u } + allJ - J'll,

and proceeding as before, we obtain in place of (33) ( J - T ( J ) J - J

This shows that (3)holds with p

= a.

V ~ E M ,J , J ' E B .

Q.E.D.

Minimax Control Proposition 4.11

Consider the mapping

of Section 2.3.5 and let J,(x) b E R, there holds

=0

for V x E S. Assume that a < 1 and for some

Then Assumption C is satisfied with (2)and (3)both equal to cc. Proof

B equal to B, m = 1, and the scalars in

The proof is entirely similar to the one of Proposition 4.10. Q.E.D.

For additional problems where the theory of this chapter is applicable, we refer the reader to DPSC. An example of an interesting problem where Assumption C is satisfied with m > 1 is the first passage problem described in Section 7.4 of DPSC.

Chapter 5

Infinite Horizon Models under Monotonicity Assumptions

5.1 General Remarks and Assumptions Consider the infinite horizon problem minimize J,(x)

=

lim (T,,T,, - . . T,,_ ,)(J,)(x)

N+ m

subject to

n = (p,, p l , . . .) E Il.

In this chapter we impose monotonicity assumptions on the function J , which guarantee that J, is well defined for all TCEll.For every result to be shown in this chapter, one of the following two assumptions will be in effect. Assumption I (Uniform Increase Assumption) There holds

J0(x) 5 H(x, U, Jo)

VX E S, u E U(X).

(2)

Assumption D (Uniform Decrease Assumption) There holds

J,(x) 2 H(x, u, J,)

Vx E S, u E U(x).

(3)

It is easy to see that under each of these assumptions the limit in (1)is well defined as a real number or co.Indeed, in the case of Assumption I we have

+

from (2)that J o I T,,(Jo)

(T,,T,,)(Jo) 5 - . . 5 (T,,TP1.. . T,,-,)(JO)

. . .,

while in the case of Assumption D we have from (3) that Jo 2 T,,(Jo) 2 (T,,T,,)(Jo) 2 . . . 2(T,,T,,. . . T,,-,)(Jo) 2 . . . . In both cases, the limit in ( 1 ) clearly exists in the extended real numbers for each x E S. In our analysis under Assumptions I or D we will occasionally need to assume one or more of the following continuity properties for the mapping H. Assumptions 1.1 and 1.2 will be used in conjunction with Assumption I, while Assumptions D.l and D.2 will be used in conjunction with Assumption D. Assumption 1.1 If ( J , ) c F is a sequence satisfying J o I J , I J k + lfor all k, then

(

)

lim H(x,u, J,) = H x, u, lim Jk k - oo

k l m

Vx E S,

u E U(x).

(4)

Assumption 1.2 There exists a scalar a > 0 such that for all scalars r > 0 and functions J E F with Jo I J, we have Assumption D.l all k, then

,

If (J,) c F is a sequence satisfying Jk+ I J , I J o for

lim H(x,u, J,) = H

QxE S, u E U(x).

k+ w

(6)

Assumption D.2 There exists a scalar a > 0 such that for all scalars r > 0 and functions J E F with J I J,, we have

5.2 The Optimality Equation We first consider the question whether the optimality equation J* = T(J*) holds. As a preliminary step we prove the following result, which is of independent interest. Proposition 5.1 Let Assumptions I, 1.1, and 1.2 hold. Then given any E

> 0, there exists an E-optimal policy, i.e., a TC, E II,such that

5.

72

INFINITE HORIZON MODELS UNDER MONOTONICITY ASSUMPTIONS

Furthermore, if the scalar u in 1.2 satisfies a < 1, the policy to be stationary.

Proof

Let

(E,)

be a sequence such that

E~

TC, can

be taken

> 0 for all k and

For each x E S, consider a sequence of policies { T C , [ X ] ) c ll of the form xk[xl such that for k

= (,ukg[~I> ,u![xI,.

..I,

= 0,1,. . .

Such a sequence exists, since we have J*(x) > - cc under our assumptions. The (admittedly confusing) notation used here and later in the proof should be interpreted as follows. 'The policy n,[x] = (,ukg[x],y:[x], . . .) is associated with x. Thus ,u:[x] denotes, for each x E S and k, a function in M, while ,u:[x](z)denotes the value of ,u:[x] at an element z E S. In particular, y f [ x ] ( x )denotes the value of ,uf[x] at x. Consider the functions ,iik E M defined by F k ( ~ 1=

~ k [XI o (x)

'VxE S

and the functions 7, defined by

By using (lo),( 1 I), I , and 1.1, we obtain

We have from (12),(13) and 1.2 for all k

=

1,2,. . . and X E S

T,-, _ ,(JL)(x)= H [ X F, k - 1 ( ~Jkl ), I H[x, F k - ,(XI, ( J * + &)I 2 H[x,

and finally,

,(x),J*]

+

%EL

Using this inequality and 1.2, we obtain

Continuing in the same manner, we obtain for k

=

1,2,. . .

Since J, I J,, it follows that

Denote n, = (Po,P I , . . .). Then by taking the limit in the preceding inequality and using (9), we obtain

If r < 1, we take c, = ~ ( -1 x ) for all k and nL[x] = (,uo[x],pl[x], . . .) in (10). The stationary policy 71, = (p,p,. . .), where p(.u) = po[.u](s) for all s ~ S , s a t i s f i e s J , ~ ~ J * + r . . Q.E.D. It is easy to see that the assumption cx < 1 is essential in order to be able to take n, stationary in the preceding proposition. As an example, take S = jOj, U(0) = (0, a), Jo(0)= 0, H(0, u, J ) = u + J(0). Then J*(O) = 0, but for any ,u E M, we have J,(O) = x. By using Proposition 5.1 we can prove the optimality equation under I, 1.1, and 1.2. Proposition 5.2 Let I, 1.1, and 1.2 hold. Then

Furthermore, if J'EF is such that J' 2 J, and J' 2 T(J1),then J' 2 J*. Proof

For every n = (pO,p l , . . .) E n and x E S, we have, from I. 1,

J,(s)= lim (T,,T,, . . . T,,)(J,)(x) k-

n;

(T,, . . . T,,)(J,)

I

(x) 2 T,,(J*)(x) 2 T(J*)(x).

By taking the infimum of the left-hand side over R E n,we obtain

74

5.


To prove the reverse inequality, let E, and E, be any positive scalars and let E = (p,, p,, . . .) be such that

where E l

= (p1,p2,.. .).

=

Such a policy exists by Proposition 5.1. We have

,

lirn (T,, . . T ,)(Jo)]

T,,

[k+m

Since J * I J,-and

E,

and

E,

can be taken arbitrarily small, it follows that

Hence J * = T(J*). Assume that J'EF satisfies J' 2 J, and J' 2 T(J'). Let {E,) be any sequence with E~ > 0 and consider a policy E = (Po,PI,. . .) E n such that We have, from 1.2, J*

=

inf lirn (T,, . . . T,,)(Jo) a~IIk+m

2 inf lim inf(T,, . . , T,,)(Jf) Z

E

k-m ~

I lirn inf(T,, . . . T,,)(J') k- cc

I lirn lnf(T,, . . . T,, _ ,) [T(J')

+ E,]

k-m

I lirn inf(T,, . . . Tpk_ ,T,, k- m

Since we may choose

_

,)(J' + 8,)

CEOaiEi as small as desired, it follows that J * I J'. Q.E.D.

The following counterexamples show that 1.1 and 1.2 are essential in order for Proposition 5.2 to hold.

COUNTEREXAMPLE 1 Take S = ( 0 , I ) , C = U ( 0 )= U ( 1 )= ( - 1, 01, J,(O) = J o ( l ) = - 1, H(0, u, J ) = u if J(1) I - 1, H(0, u, J ) = 0 if J ( l ) > - 1, and H(1, u, J ) = u. Then (T,, . . . T,,_ l ) ( J o ) ( 0 = ) 0 and (T,, . . . T,,- , ) ( J o ) ( l )= y O ( l )for N 2 1. Thus J*(O) = 0 , J * ( l ) = - 1, while T(J*)(O)= - 1, T ( J * ) ( l )= - 1, and hence J* # T ( J * ) . Notice also that J , is a fixed point of T, while J , I J* and J , f J*, so the second part of Proposition 5.2 fails when J , = J'. Here I and 1.1 are satisfied, but 1.2 is violated. COUNTEREXAMPLE 2 Take S = {O, I}, C = U ( 0 )= U ( 1 )= (01, J,(O) = J o ( l ) = 0 , H(O,O, J ) = 0 if J ( 1 ) < co,H(O,O, J ) = co if J(1) = oo,H(1,0, J ) = J ( l ) + 1. Then (T,, . . . T,,_ J(J,)(O) = 0 and ( T w o . . T,,- J(J0)(1)= N. Thus J*(O) = 0, J * ( l )= oo.On the other hand, we have T(J*)(O)= T ( J * ) ( l )= oo and J X # T ( J * ) . Here I and 1.2 are satisfied, but 1.1 is violated. As a corollary to Proposition 5.1 we obtain the following. Corollary 5.2.1 Let I, 1.1, and 1.2 hold. Then for every stationary policy n = ( y ,y,. . .), we have

Furthermore, if J ' E F is such that J' 2 J , and J' 2 T,(J1), then J' 2 J,. Proof Consider the variation of our problem where the control constraint set is U,(x) = ( y ( x ) j rather than U ( x ) for V x E S. Application of Q.E.D. Proposition 5.2 yields the result.

We now provide the counterpart of Proposition 5.2 under Assumption D . Proposition 5.3 Let D and D.l hold. Then

T ( J 1 ) ,then J' I J*. Furthermore, if J ' E F is such that J' I J , and J' I Proof

We first show the following lemma.

Lemma 5.1 Let D hold. Then

J*

=

lim J g , N+ rn

where J i is the optimal cost function for the N-stage problem. Proof Clearly we have J* I J g for all N , and hence J* I lim,,, Also, for all n = (y,,p,,. . . ) ~ l lwe , have

and by taking the limit on both sides we obtain J , 2 lim,,, J*>lim,,,Jg. Q.E.D.

Jg.

J $ , and hence

76

5.


Proof (continued) We return now to the proof of Proposition 5.3. An argument entirely similar to the one of the proof of Lemma 5.1 shows that under D we have for all x E S lim inf H ( x , u, J;) N-m U E U ( X )

=

inf lim H ( x , u, J;). u~U(x)N+m

Using D . l , this equation yields

(I6)

>

lirn T ( J $ ) = T lirn J ; .

(17) Since D and D.l are equivalent to Assumption F.1' of Chapter 3, by Corollary 3.1.1 we have J ; = T N ( J o )from , which we conclude that T ( J ; ) = T N +' ( J O = ) J $ + , . Combining this relation with (15) and (17), we obtain J* = T ( J * ) . T o complete the proof, let J ' E F be such that J' I J o and J' I T ( J 1 ) . Then we have N- m

J*

=

(N+w

inf lirn (T,, . . . T,, _ i ) ( J o ) Z E ~ N - m

2 lirn inf (T,, N - m acn

. . . T,, _ l ) ( J o )

2 lirn inf (T,, . . . T,,- , ) ( J ' ) 2 lim T N ( J ' )2 J' N-m

Hence J* 2 J'.

nen

N- m

Q.E.D.

In Counterexamples 1 and 2 of Section 3.2, D is satisfied but D.l is violated. In both cases we have J* # T ( J * ) ,as the reader can easily verify. A cursory examination of the proof of Proposition 5.3 reveals that the only point where we used D.l was in establishing the relations

and JE = T N ( J o ) .If these relations can be established independently, then the result of Proposition 5.3 follows. In this manner we obtain the following corollary. Corollary 5.3.1 Let D hold and assume that D.2 holds, S is a finite set, and J * ( x ) > - co for all X E S. Then J* = T ( J * ) . Furthermore, if J ' E F is such that J' I J o and J' I T ( J ' ) ,then J' I J*. Proof A nearly verbatim repetition of the proof of Proposition 3.l(b) shows that, under D , D.2, and the assumption that J * ( x ) > - co for all x E S, we have J ; = T N ( J o )for all N = 1,2,. . . . We will show that lim H ( x , u, J X ) = H

V x E S, u E U ( x ) .

N-ta,

Then using (16) we obtain (17), and the result follows as in the proof of Proposition 5.3. Assume the contrary, i.e., that for some x " S,~ E E U(,Y), and

5.2 E

THE OPTIMALITY EQUATION

> 0, there holds H(x",u", J?) - & > H

k

,

=

1,2,. . . .

From the finiteness of S and the fact that J*(x) = limN,, J g ( x ) > - co for all x , we know that for some positive integer ii J

-

( a )I 1

N+ m

Vk 2 E.

J

By using D.2 we obtain for all k 2 ii H(x",u",J;) - E I H(R,u",JZ - ( & / a ) I ) H

Q.E.D.

which contradicts the earlier inequality.

Similarly, as under Assumption I, we have the following corollary. Corollary 5.3.2 Let D and D.l hold. Then for every stationary policy we have

TL = (p,p, . . .),

J,

=

Tp(Jp).

Furthermore, if J ' E F is such that J' I J o and J' I T,(J1), then J' 5 J,. It is worth noting that Propositions 5.2 and 5.3 can form the basis for computation of J* when the state space S is a finite set with n elements denoted by x,,x,,. . . ,x,. It follows from Proposition 5.2 that, under I, 1.1, and 1.2, J * ( x , ) , . . . ,J*(x,) solve the problem n

minimize

I-i i= 1

subject to IVi2 inf H ( x i ,u, J J , E

i = 1,. . . ,n,

U(XI)

I.i 2 Jo(xi),

i = 1,. . . ,n,

where J , is the function taking values J,(xi) = ii,i = 1,. . . ,n. Under D and D.l, or D, D.2, and the assumption that J * ( x ) > - co for V X ES, the corresponding problem is maximize

i= 1

l.i

subject to lbiI H ( x , , u , J,), lLi5 J O ( x i ) ,

i = 1 , . . . ,n, i = 1,. . . ,n.

UE

U(xi)

When U ( x , )is also a finite set for all i, then the preceding problem becomes a finite-dimensional (possibly nonlinear) programming problem.

78

5.


5.3 Characterization of Optimal Policies We have the following necessary and sufficient conditions for optimality of a stationary policy. Proposition 5.4 Let I, 1.1, and 1.2 hold. Then a stationary policy TC*= (p*,p*, . . .) is optimal if and only if

Furthermore, if for each x~ S, there exists a policy which is optimal at x , then there exists a stationary optimal policy. Proof If TC*is optimal, then J,, = J* and the result follows from Proposition 5.2 and Corollary 5.2.1. Conversely, if T,,(J*) = T(J*), then since J* = T ( J * )(Proposition 5.2), it follows that T,,(J*) = J*. Hence by Corollary 5.2.1, J,, I J* and it follows that n* is optimal. If T C ,=* ( p z ,,, pT, . . .) is optimal at x E S, we have, from 1.1,

.,

J*(x) = J,,(x)

=

lim (T,;,; . . T,:, x ) ( J o ) ( x ) k- a,

Hence T,*o.x(J*)(x)= T ( J * ) ( x )for all x E S. Define p* E M by p*(x) = p;j;,,(x). Then T,,(J*) = T ( J * ) and, by the result just proved, the stationary policy Q.E.D. (p*, p*, . . .) is optimal. Proposition 5.5 Let D and D.l hold. Then a stationary policy n* (p*, p*, . . .) is optimal if and only if T,*(J,*)

=

T(J,*).

=

(19)

Proof If n* is optimal, then J,, = J*, and the result follows from Proposition 5.3 and Corollary 5.3.2. Conversely, if T,,(J,,) = T(J,,), then we obtain, from Corollary 5.3.2, that J,, = T(J,,), and Proposition 5.3 yields J,, I J*. Hence TC*is optimal. Q.E.D. Examples where n* satisfies (18) or (19) but is not optimal under D or I, respectively, are given in DPSC, Section 6.4. Proposition 5.4 shows that there exists a stationary optimal policy if and only if the infimum in the optimality equation J * ( x ) = inf H ( x , u, J*) u

LT(x)

5.3

79

CHARACTERIZATION OF OPTIMAL POLICIES

is attained for every x E S. When the infimum is not attained for some x E S, the optimality equation can still be used to yield an &-optimalpolicy, which can be taken to be stationary whenever the scalar a in 1.2 is strictly less than one. This is shown in the following proposition. Proposition 5.6 Let I, 1.1, and 1.2 hold. Then: (a) If E > 0, (ci) satisfies (pz, pT,. . .)EI7 is such that

I,"=, aksk= E, E~ > 0, i = 0,1,. . . , and n* =

then

(b) If E > 0, the scalar a in 1.2 is strictly less than one, and p* E M is such that T,*(J*) I T(J*)

+ ~ ( -1 a),

then

Proof (a) Since T(J*) = J*, we have T,;(J*) T , ; , to both sides we obtain

< J* + E,, and applying

Applying T,:_2 throughout and repeating the process, we obtain, for every

Since JoI J*, it follows that

By taking the limit as k + co,we obtain J,, I J * + E. (b) This part is proved by taking 6, = ~ ( -1 a) and p: Q.E.D. the preceding proof.

= p*

for all k in

A weak counterpart of part (a) of Proposition 5.6 under D is given in Corollary 5.7.1. We have been unable to obtain a counterpart of part (b) or conditions for existence of a stationary optimal policy under D.

80

5.


5.4 Convergence of the Dynamic Programming AlgorithmExistence of Stationary Optimal Policies The D P T(Jo),T'(J,), while under can define a

algorithm consists of successive generation of the functions . . . . Under Assumption I we have T k ( ~ , I ) Tk+l(J0)for all k, Assumption D we have Tk"(Jo) I Tk(Jo)for all k. Thus we function J, E F by J,(x) = lim TN(~,)(x) N*

Vx E S.

(20)

02

We would like to investigate the question whether J, = J*. When Assumption D holds, the following proposition shows that we have J, = J * under mild assumptions. Proposition 5.7 Let D hold and assume that either D.l holds or else J$ = TN(Jo)for all N, where J g is the optimal cost function of the N-stage problem. Then J, = J*. Proof By Lemma 5.1 we have that D implies J * = lirn,, , JR. Corollary 3.1.1 shows also that under our assumptions J$ = T,(J,). Hence J * = lim,, ,TN(Jo)= J, . Q.E.D.

The following corollary is a weak counterpart of Proposition 5.1 and part (a) of Proposition 5.6 under D. Corollary 5.7.1 Let D hold and assume that D.2 holds, S is a finite set, and J*(x) > - cc for all x E S. Then for any E > 0, there exists an r-optimal 7 such that policy, i.e., a 7c, E l

+

Proof For each N , denote r, = 42(1 + cx . . . + aN-I) and let 7c, = N ,r p No , pNl , . . . ,pN-l,,u,p,.. .) be such that EM, and fork = 0,1,. . . ,N- 1, u ,; E M and ( T , ~ T ~ '-)~( J- ~I) T,-$(J~) EN.

+

We have T,;_ ,(Jo) I T(Jo)+ r,, and applying T,:_ Continuing in the same manner, we have

from which we obtain, for N

= 0,1,. . . ,

+

J,, I TN(Jo) (42).

to both sides, we obtain

5.4

CONVERGENCE OF THE DYNAMIC PROGRAMMING ALGOR~THM

81

As in the proof of Corollary 5.3.1 our assumptions imply that J $ = T N ( ~ , ) for all N , so by Proposition 5.7, lim,,, T N ( ~= , )J*. Let IV be such that T'(J,) I J* + ( 4 2 ) . Such an IV exists by the finiteness of S and the fact that J * ( x ) > - co for all X E S . Then we obtain J,, I J* + c, and T C is~ the Q.E.D. desired policy. Under Assumptions I, 1.1, and 1.2, the equality J , = J* may fail to hold even in simple deterministic optimal control problems, as shown in the following counterexample.

COUNTEREXAMPLE 3 Let S = [0, co), C = U ( x )= (0,co) for V X E S J, o ( x )= 0 for V x E S, and Then it is easy to verify that

J * ( x ) = inf J,(x) Z

E

=

co

V x E S,

~

while

Hence J,(O)

=0

and J , ( 0 ) < J*(O).

In this example, we have J*(x) = co for all ~ E SOther . examples exist where J* # J , and J * ( x ) < co for all X E S (see [S14, p. 8801). The following preliminary result shows that in order to have J , = J*, it is necessary and sufficient to have J , = T(J,). Proposition 5.8 Let I, 1.1, and 1.2 hold. Then

J , I T(J,)

T ( J * ) = J*.

(21)

Furthermore, the equalities

hold if and only if

J,

=

T(J,).

(23)

Proof Clearly we have J , I J , for all TC E ll,and it follows that J , i J*. Furthermore, by Proposition 5.2 we have T ( J * ) = J*. Also, we have, for all k 2 0,

T(J,)

=

inf H ( x , u, J,) 2 inf H [ x , u, Th(J,)] = T k + '(Jo). u U(x)

ue

U(x)

Taking the limit of the right side, we obtain T(J,) 2 J,, which, combined J* and T ( J * ) = J*, proves (21).If (22)holds, then (23)also holds. with J , I

82

5.


Conversely, let (23) hold. Then since we have J , 2 J o , we see from Proposition 5.2 that J , 2 J*, which combined with (21) proves (22). Q.E.D. In what follows we provide a necessary and sufficient condition for J , = T(J,) [and hence also (22)] to hold under Assumptions I, 1.1, and 1.2. We subsequently obtain a useful sufficient condition for J , = T(J,) to hold, which at the same time guarantees the existence of a stationary optimal policy. For any J E F, we denote by E ( J ) the epigraph of J , i.e., the subset of S R given by E ( J ) = ( ( x ,]")I J ( x ) l A). Under I we have T k ( J o )IT k + ' ( J o )for all k and also J , = lim,,, so it follows easily that

(24) T~(J,),

Consider for each k 2 1 the subset C k of S C R given by C k = ( ( x ,u, i,)IH[x, U, T k - ' ( J o ) ]I A, X E S , U E U ( X ) ) .

(26)

Denote by P ( C k )the projection of C k on S R , i.e.,

Consider also the set is obtained from P ( C k )by adding for each x the point [ x , z ( x ) ] The set P(C,) where K(x) is the perhaps missing end point of the half-line (Al(x,A) E P ( C k ) ) . We have the following lemma. Lemma 5.2 Let I hold. Then for all k 2 1

Furthermore, we have P(Ck) = P O = EITk(Jo)l if and only if the infimum is attained for each x~ S in the equation T k ( ~ o ) (= x ) inf H [ x , u, T k - ' ( J o ) ] . u€

U(x)

' The symbol 3 means "there exists" and the initials "s.t." stand for "such that."

(30)

5.4

CONVERGENCE OF THE DYNAMIC PROGRAMMING ALGORITHM

Proof

If ( x ,3.) E E [ T k ( J o ) ]we , have Tk(Jo)(x)= inf H [ x , u, Tk-'(J,)] I 3,. uE

U(X)

Let {cn)be a sequence such that sequence such that

E,

> 0, E,

-+ 0,

and let (u,) c U ( x ) be a

+ E, IA + E ,

H [ x , u,, T k - ' ( J O ) ]I Tk(JO)(x)

Then ( x ,u,, iL+ E,) E C, and ( x ,3, + 8,) E P(Ck) for all n. Since j. + E , by (28)we obtain ( x ,i)E P(Ck).Hence EITk(Jo)l P O .

+i

,

(32)

Conversely, let ( x ,A) E P(C,). Then by (26)-(28)there exists a sequence {A,) with A, -, 3, and a corresponding sequence (u,) c U ( x )such that T k ( J o ) ( xl ) H [ x , u,, T k - ' ( J ~ )1 ] I,,. Taking the limit as n Hence

-+

co, we obtain Tk(Jo)(x) l A and (x,I,)EE[T~(J,)].

PO c E[Tk(JO)I, and using (32) we obtain (29). To prove that (30)is equivalent to the attainment of the infimum in (31), assume first that the infimum is attained by pz- ,(x) for each x E S. Then for each ( x ,A)E EITk(Jo)], which implies by (27) that (X,;L)EP(C,)Hence EITk(Jo)]c P(C,) and, in E view of (29),we obtain (30).Conversely, if (30)holds, we have [ x , Tk(Jo)(x)] P(C,) for every x for which T k ( J o ) ( x< ) oo.Then by (26)and (27),there exists a pi- ,(x)E U ( x )such that H [ x ,pf- l(x),T ~ - ' ( J ~I) Tk(~,,)(x) ] = inf H [ x , u, T,-'(J,)]. u E U(x)

Hence the infimum in (31) is attained for all x for which T k ( ~ , ) ( x 0, E,

+ 0.

Then there exists a sequence

and so H[x, u,, Tk-'(Jo)] I3,

+ E,,

k,n = 1,2,. . . .

It follows that (x, u,, i+ r,) E Ck for all k, n and (x, u,, i+ r,) E all n. Hence (x, i+ E,) E P(n,"= Ck) for all n, and since 3, + r, obtain (x, A)EP(nP= Ck).Therefore

C, for we

+ 1,

and using (36) we obtain (37). Conversely, let (37) hold. Then by (36) we have P(n,"=, C,) = E(J,). Let ~ E beS such that J,(x) < co. Then [x, J,(x)] E P(n,"=, C,), and there

5.4


85

exists a sequence (I,,) with in-+J,(x) and a sequence {u,) c U ( x )such that k, n = l,2,. . . .

H[x, u,, T k - ' ( J 0 ) ]I A,,

Taking the limit with respect to k and using 1.1, we obtain

H(x, u,, J,), it follows that and since T(J,)(x) I

Taking the limit as n + oo,we obtain for all x E S such that J,(x) < oo. Since the preceding inequality holds also for all x E S with J,(x) = co, we have T(J,) I J, . On the other hand, by Proposition 5.8, we have

Combining the two inequalities, we obtain J , = T(J,). (b) Assume J , = T(J,) and that the infimum in (38) is attained for each x E S. Then there exists a function p* E M such that for each (x,I,) E E(J,) ~ [ xp*(x), , J,] I i . Hence H [ x , ~ * ( x T ) ,k - l ( J O ) I ] 2 for k = 1,2,. . . , and we have [ x ,p*(x),A] r),"=, C,. As a result, ( x , ? , )P(n,"= ~ Ck).Hence

and (39)follows from (35). Conversely, let (39) hold. We have for all ~ E with S J,(x) < co that [ x ,J,(x)] EE(J,)

=P

Thus there exists a p*(x) E U ( x )such that

from which we conclude that

0 1 tkZ1 c k

.

E

86

5.


Taking the limit and using 1.1, we see that T(J,)(x) l H [ x , P*(x), J,] 1 Jm(x). It follows that T(J,) 1 J,, and since J , I T(J,) by Proposition 5.8, we finally obtain J , = T(J,). Furthermore, the last inequality shows that p*(x) attains the infimum in (38) when J,(x) < co. When J,(x) = co, every U E U ( x ) attains the infimum, and the proof is complete. Q.E.D. In view of Proposition 5.8, the equality J , = T ( J , ) is equivalent to the validity of interchanging infimum and limit as follows J,

=

lim inf (T,, . . . T,,)(Jo) = inf lim(T,, . . . T,,)(Jo) asnk-w

= J*

k-m nsII

Thus Proposition 5.9 states that interchanging infimum and limit is in fact equivalent to the validity of interchanging projection and intersection in the manner of (37) or (39). The following proposition provides a compactness assumption which guarantees that (39) holds. Proposition 5.10 Let I, 1.1, and 1.2 hold and let the control space C be a Hausdorff space. Assume that there exists a nonnegative integer E such that for each x E S, i E R, and k 2 E, the set

U k ( x ,i)= { uE U ( x ) l H[ x , u, T k ( J o ) ]I 1")

(40)

is compact. Then

and (by Propositions 5.8 and 5.9) J , = T(J,)

=

T ( J * ) = J*.

Furthermore, there exists a stationary optimal policy. Proof

By (35)it will be sufficient to show that

( ) fi

k= 1

Let (x,?.) be in that

m

P

C

CO

n p(ck) n m =

k= 1

P ( C k ) .Then there exists a sequence

H [ x , u,, T k ( J o ) ]5 H [ x , u,, T n ( J o ) ]I 3. or equivalently

(42)

k= 1

(u,) c

v n 2 k,

U ( x ) such

5.4

87


Since U,(x, II) is compact for k 2 E, it follows that the sequence {u,) has an accumulation point ii and ii E U,(X, A)

But Uo(x,A)

3

U,(x, A) 3 . . . , so

E

U,(x, A) for k = 0,1,. . . . Hence

H[x,ii,Tk(Jo)] - oo for all X E S and k = 1,2,. . . ,N. Then

JA and for every such that

E

=

TN(Jo),

> 0 there exists an N-stage &-optimalpolicy, i.e., a J; I J N , = , I JA

+

T C , E ~

E.

(b) Let Assumptions A.l -A.4 and F.3 hold and assume that J,, Jx) < co f o r a l l x ~n~~,l ? , a n dk = 1,2,. . . , N . Then J;

=

TN(Jo).

Furthermore, given any sequence ( E , ) with &,LO,r, > 0 for Qn, there exists a sequence of policies exhibiting (&,)-dominated covergence to optimality. In particular, if in addition Jg(x) > - co for all x E S, then for every E > 0 there exists an &-optimalpolicy. Similarly, by modifying the proofs of Proposition 3.3 and Corollary 3.3.l(b), we obtain the following. Proposition 6.2 Let Assumptions A.l -A.4 hold.

(a) A policy TC*= (,u;f:,pT, . . .) E I? is uniformly N-stage optimal if and only if (T,:TN-k-')(Jo) = TN-k(Jo),k = 0,1,. . . ,N - 1. (b) If there exists a uniformly N-stage optimal policy, then Analogs of Corollary 3.3.l(a) and Proposition 3.4 can be proved if is rich enough so that the following assumption holds.

fi

Exact Selection Assumption For every J E F*, if the infimum in

T(J) = inf H(x, u, J) u0U(x)

is attained for every x E S, then there exists a p* E fi such that T,,(J)

=

T(J).

In Examples 1 and 4 of the previous section the exact selection assumption is satisfied (see Propositions 7.50 and 7.33). The following proposition is proved similarly to Corollary 3.3.l(a) and Proposition 3.4. Proposition 6.3 Let Assumptions A.l-A.4 and the exact selection assumption hold.

(a) There exists a uniformly N-stage optimal policy if and only if the infimum in the relation T ~ + ~ ( J ~= ) ( Xinf ) H[x, u, Tk(Jo)] uE U(X)

i s a t t a i n e d f o r e a c h x ~ S a n d k = O , l , .. . , N - 1. (b) Let the control space C be a Hausdorff space and assume that for e a c h x ~ S , % ~ R , a n d0,1,.. k = . , N - 1, theset Uk(x,i ) = {u E U(x)lH[x, u, Tk(Jo)]5 i ) is compact. Then

and there exists a uniformly N-stage optimal policy.

6.3 Analysis of Infinite Horizon Models under a Contraction Assumption

We consider the following modified version of Assumption C of Section 4.1. Assumption

There is a closed subset B of the space B such that:

~ (a) J , E B F*, (b) For all J E Bn F*, the function T(J) belongs to B n F*, (c) For all J EB n F and P E f i , the function T,(J) belongs to B n F. Furthermore, for every n

= (p,,

-

p, ,. . .) E ll,the limit

exists and is a real number for each x E S. In addition, there exists a positive integer m and scalars p and a with 0 < p < 1 , 0 < a such that IIT,(J)-T,(J1)IIIallJ-J'll

~ p ~ f Ji ,, J ' E B ~ ,

II(T,,T,~. . T,_- ,)(J) - (T,OT,I . . T,_- ,)(J1)II2 pllJ - J' l 'dp,,.. .,p,-,EM, J , J ' E B ~ F .

-

If Assumptions A.l-A.5 and C" are made, then almost all the results of Chapter 3 have counterparts within our extended framework. The key fact is that, since F" and F* are closed under pointwise limits (Assumption AS), it follows that B n F, B n F, B n F*, and B n F* are closed subsets of B. This is true in view of the fact that convergence of a sequence in B (i.e., in sup norm) implies pointwise convergence. As a result the contraction mapping fixed point theorem can be used in exactly the same manner as in Chapter 3 to establish that, for each p E l@,J, is the unique fixed point of T, in B n F and J * is the unique fixed point of T in B n F*. Only the modified policy iteration algorithm and the associated Proposition 4.9 have no counterparts in this extended framework. The reason is that our assumptions do not guarantee that Step 3 of the policy iteration algorithm can be carried out. Rather than provide a complete list of the analogs of all propositions in Chapter 4 we state selectively and without proof some of the main results that can be obtained within the extended framework.

Proposition 6.4 Let Assumptions A.l-A.5 and

hold.

(a) The function J * belongs to B n F* and is the unique fixed point of T within B n F*. Furthermore, if J' E B n F* is such that T(J') I J', then J * I J', while if J' I T(Jt),then J' I J*. (b) For every p E the function J, belongs to B n F" and is the unique fixed point of T, within B n E. (c) There holds

a,

(d) A stationary policy TC*= (p*, p*, . . .)Efi is optimal if and only if Equivalently, IT*is optimal if and only if J,, E B n F* and

-

(e) For any E > 0, there exists a stationary &-optimal policy, i.e., a (pE,,us,.. .)Ell such that

T C ~=

Proposition 6.5 Let Assumptions A.l-A.5 and hold. Assume further that the exact selection assumption of the previous section holds.

(a) If for each X E S there exists a policy which is optimal at x, then there exists an optimal stationary policy.

(b) Let C be a Hausdorff space. If for some J E B n F* and for some positive integer 7t the set

U,(x, i) = { u E U(x) 1 H[x,u,Tk(J)] I i) is compact for all X E S, /ZE R, and k 2 E, then there exists an optimal stationary policy.

Part II

Stochastic Optimal Control Theory

Chapter 7

Borel Spaces and Their Probability Measures

This chapter provides the mathematical background required for analysis of the dynamic programming models of the subsequent chapters. The key concept, which is developed in Section 7.3 with the aid of the topological concepts discussed in Section 7.2, is that of a Borel space. In Section 7.4 the set of probability measures on a Borel space is shown to be itself a Borel space, and the relationships between these two spaces are explored. Our general framework for dynamic programming hinges on the properties of analytic sets collected in Section 7.6 and used in Section 7.7 to define and characterize lower semianalytic functions. These functions result from executing the dynamic programming algorithm, so we will want to measurably select at or near their infima to construct optimal or nearly optimal policies. The possibilities for this are also discussed in Section 7.7. A similar analysis in a more specialized case is contained in Section 7.5, which is presented first for pedagogical reasons. Our presentation is aimed at the reader who is acquainted with the basic notions of topology and measure theory, but is unfamiliar with some of the specialized results relating to separable metric spaces and probability measures on their Borel o-algebras.

7.

BOREL SPACES AND THEIR PROBABILITY MEASURES

7.1 Notation

We collect here for easy reference many of the symbols used in Part 11. Operations on Sets Let A and B be subsets of a space X. The complement of A in X is denoted by A". The set-theoretic diference A - B is A n Bc. We will sometimes write X - A is place of A". The symmetric diference A A B is ( A - B) u ( B - A). If X is a topological space, A will denote the closure of A. If A,, A,, . . . is a sequence of sets such that A, c A, c . . . and A = U,", ,An, we write An? A . I f A , 2 A, = . . . a n d A = n , " = , ~ , , w e w r i t e A ~ J A . I f X , , X ,,... isa sequence of spaces, the Cartesian products of X I , X,, . . . ,X , and of X , , X,, . . . are denoted by X,X,. . . Xn and X,X,. . . , respectively. If the given spaces have topologies, the product space will have the product topology. Under this topology, convergence in the product space is componentwise convergence in the factor spaces. If the given spaces have o-algebras 9,,,9,,,.. . , the product o-algebras are denoted by BX,9,,.. .9," and 9,,9,,. . . , respectively. If X and Y are arbitrary spaces and E c X Y , then for each x E X, the x-section of E is

If 9'is a class of subsets of a space X , we denote by o ( 9 ) the smallest the class of all subsets which o-algebra containing 9.We denote by 9, or 9, can be obtained by union or intersection, respectively, of countably many If 9 is the collection of closed subsets of a topological space X, sets in 9. then Fa= -9, and the members of 9, are called the F,-subsets of X. If 9 is the collection of open subsets of X, the members of 9, are called the G,-subsets of X. If (X, .Y) is a paved space, i.e., 9 is a nonempty collection of subsets of X, and S is a Suslin scheme for .Y(Definition 7.15), then N ( S ) is the nucleus of S. The collection of all nuclei of Suslin schemes for .P is denoted Y ( 9 ) . Special Sets The symbol R represents the real numbers with the usual topology. We use R* to denote the extended real numbers [- co,+ co] with the topology discussed following Definition 7.7 in Section 7.3. Similarly, Q is the set of rational numbers and Q* is the set of extended rational numbers Q u j fco). If X and Y are sets and f :X + Y , the graph off is

7.1

103

NOTATION

If A c X and %? is a collection of subsets of X, we define .f(A) = ( f ( x ) J xA~} and

If B c Y and (?? is a collection of subsets of Y , we define f -'(B) = {xEXIf ( x ) ~ B and )

, is the collection of closed subsets of X If X is a topological space, 9 and gxthe Borel o-algebra on X (Definition 7.6). The space of probability measures on (X, Bx)is denoted by P(X); C(X) is the Banach space of bounded, real-valued, continuousfunctions on X with the supremum norm

(If I1 = xsup(f(x>I sx ; for any metric d on X which is consistent with its topology, U,(X) is the space of bounded real-valued functions on X which are unijormly continuous with respect to d. If X is a Borel space (Definition 7.7), dxis its analytic a-algebra (Definition 7.19) and @, its universal a-algebra (Definition 7.18). We let N denote the set of positive integers with the discrete topology. The Baire null space JV is the product of countably many copies of N. The Hilbert cube Y? is the product of countably many copies of [0, 11. We will denote by C the collection ofjnite sequences of positive integers. We impose no topology on C. If s E C and z = (c,, [,, . . . ) E Mwe , write s < z to mean s = (C1,[,,. . . ,ik) for some k. Mappings If X and Y are spaces, proj, is the projection mapping from XY onto X If E is a subset of X, the indicator function of E is given by

Iff : X

+

[ - m, +a],the positive and negative parts off are the functions

If f,:X -t Y is a sequence of functions, Y is a topological space, and limn, ,f,(x) = f (x) for all x E X, then we write f, -,f . If, in addition, Y = [ - oo,+a]and fl(x) I f2(x) I . . . for all x E X , we write f, f , while if J;(x) 2 J2(x) 2 . . . for all x E X, we write fi J f'. In general, when the arguments of extended real-valued functions are omitted, the statements are to be

104

7.


interpreted pointwise. For example, (sup,.JJ(x) = sup,f,(x) for all X E X , fl I ,f2 if and only if fl(x) 5 .f2(x) for all EX, and f + E is the function ( j ' E)(x)= f(x) + E for all x E X.

+

Miscellaneous If (X, d) is a nonempty metric space, x E X, and Y is a nonempty subset of X, we define the distance from x to Y by d(x, Y)

=

inf d(x, y). YEY

We define the diameter of Y by diam(Y) = sup d(x, y). X,YEY

If (X, .F)is a measurable space and .Fcontains all singleton sets, then for .xE X we denote by p, the probability measure on (X, 9)which assigns mass one to the set (x). 7.2 Metrizable Spaces Definition 7.1 Let (X, F ) be a topological space. A metric d on X is consistent with .Fif every set of the form { y E Xld(x, y) < c), x E X, c > 0, is in F, and every nonempty set in F is the union of sets of this form. The is metrizable if such a metric exists. space (X, 9)

The distinction between metric and metrizable spaces is a fine one: In a metric space we have settled on a metric, while in a metrizable space the choice is still open. If one metric consistent with the given topology exists, then a multitude of them can be found. For example, if d is a metric on X consistent with F , the metric p defined by

is also consistent with F. In what follows, we abbreviate the notation for metrizable spaces, writing X in place of (X, F ) . I f (X, .F) is a topological space and Y c X, unless otherwise specijed, we will understand Y to be a topological space with open sets G n Y , where G ranges over .FThis . is called the relative topology. If ( Z , Y ) is another topological space, cp : Z -t X is one-to-one and continuous, and cp- is continuous on cp(Z) with the relative topology, we say that cp is a homeomorphism and Z is homeomorphic to cp(Z). When there exists a homeomorphism from Z into X, we also say that Z can be homeomorphically embedded in X. Given a metric d on X consistent with its topology and a homeomorphism cp:Z -+ X

'

as just described, we may define a metric dl on Z by It can be easily verified that the metric dl is consistent with the topology Y . This implies that every topological space homeomorphic to a metrizable space (or subset of a metrizable space) is itself metrizable. Our attention will be focused on metrizable spaces and their Bore1 g-algebras. The presence of a metric in such spaces permits simple proofs of facts whose proofs are quite complicated or even impossible in more general topological spaces. We give two of these as lemmas for later reference. Lemma 7.1 (Urysohn's lemma) Let X be a metrizable space and A and B disjoint, nonempty, closed subsets of X. Then there exists a continuous function f :X -+ [O, 11 such that f (a) = 0 for every a E A, f (b) = 1 for every b E B, and 0 < f (x) < 1 for every x #A v B. If d is a metric consistent with > 0, then f can be chosen to be the topology on X and inf,,,,,,,d(a,b) uniformly continuous with respect to the metric d.

Proof

Let d be a metric on X consistent with its topology and define

where the distance from a point to a nonempty closed set is defined by (8). This distance is zero if and only if the point is in the set, and the mapping of (8) is Lipschitz-continuous by (6) of Appendix C. This f has the required b) > 0, then d(x, A) d(x, B) is bounded away properties. If inf,,,,,,,d(a, from zero, and the uniform continuity off follows. Q.E.D.

+

Lemma 7.2 Let X be a metrizable space. Every closed subset of X is a G, and every open subset is an F,.

Proof We prove the first statement; the second follows by complementation. Let F be closed. We may assume without loss of generality that F is nonempty. Let d be a metric on X consistent with its topology. The continuity of the function x -t d(x, F) implies that

isopen.ButF=n,"=,G,.

Q.E.D.

Definition 7.2 Let X be a metrizable topological space. The space X is separable if it contains a countable dense set.

It is easily verified that any subspace of a separable metrizable space is separable and metrizable. A collection of subsets of a topological space (X, F)is a base for the topology if every open set can be written as a union of sets from the collection. It is a subbase if a base can be obtained by taking

106

7.


finite intersections of sets from the collection. If 5 has a countable base, (X, F)is said to be second countable. A topological space is Lindelof if every collection of open sets which covers the space contains a countable subcollection which also covers the space. It is a standard result that in metrizable spaces, separability, second countability, and the Lindelof property are equivalent. The following proposition is a direct consequence of this fact. Proposition 7.1 Let (X, .F)be a separable, metrizable, topological space and 8 a base for the topology 5.Then 8 contains a countable subcollection gowhich is also a base for F . Proof Let %? be a countable base for the topology 5.Every set C E % has the form C = B,, where I ( C ) is an index set and B, €8for every a~ l(C). Since C is Lindelof, we may assume l(C) is countable. Let 8, = Ucs,e{Bala~I(C)). Q.E.D.

U,E,,(c,

The Hilbert cube 2 is the product of countably many copies of the unit interval (with the product topology). The unit interval is separable and metrizable, and, as we will show later (Proposition 7.4), these properties carry over to the Hilbert cube. In a sense, 2 is the canonical separable metrizable space, as the following proposition shows. Proposition 7.2 (Urysohn's theorem) Every separable metrizable space is homeomorphic to a subset of the Hilbert cube S. Proof Let (X, d) be a separable metric space with a countable dense set (x,). Define functions cp,(x)=min{l,d(x,x,)), and cp:X

-+

k = 1 , 2 ,...,

H by

Each cp, is continuous, so cp is continuous. (Convergence in H is componentd(y, x,,) = 0, so wise.) If cp(x) = cp(y), then letting x,! -+ x, we see that lim x = y and cp is one-to-one. It rema~nsto show that cp-' is continuous, i.e., cp(y,) + cp(y) implies y, + y. But if cp(y,) + cp(y), choose E > 0 and x, such that d(y, x,) < E. Since d(y,, x,) -+ d(y, x,) as n -+ oo, for n sufficiently large d(y,,x,) < E. Then d(y,y,) < 2 ~ . Q.E.D.

,

If X is a separable metrizable space and cp :X + 2 is the homeomorphism whose existence is guaranteed by Proposition 7.2, then by identifying x E X with q ( x ) ~ Hwe , can regard X as a subset of H . Indeed, we can regard X as a topological subspace of 2,since the images of open sets in X under the mapping cp are just the relatively open subsets of cp(X) considered as a subspace of &?. Note, however, that although X is both open and closed in itself,

p(X) may be neither open nor closed in 2.In fact, it may have no topological characterization at all. Likewise, a set with special structure in X, say a G,, may not have this structure when considered as a subset of 2.The next definition and proposition shed some light on this issue. Definition 7.3 Let X be a topological space. The space X is topologically complete if there is a metric d on X consistent with its topology such that the metric space (X,d) is complete, i.e., if {x,) c X is a d-Cauchy sequence [d(xn7x,) -+ 0 as n, m -+ co], then {x,) converges to an element of X. Proposition 7.3 (Alexandroff's theorem) Let X be a topologically complete space, Z a metrizable space, and p : X -+ Z a homeomorphism. Then p(X) is a G,-subset of Z. Conversely, if Y is a G,-subset of Z and Z is topologically complete, then Y is topologically complete. Proof For the proof of the first part of the proposition, we treat X as a subset of Z. There are two metrics to consider, a metric d on Z consistent with its topology and a metric dl on X which makes it complete. Define

U,

=

For n

{z~Zld(s, X ) < l/n and 3 an open neighborhood V(z) of z such that

=

1,2,. . . , given z E U, and V(z) as just defined, we have

so U, is open. We show X = For z EX, define

U,

Then W(z) is relatively open in X, thus of the form W(z) = V(z) n X, where V(z) is an open neighborhood in Z of z. Also,

so z E U, . Therefore X c (),"=, U,. Now suppose z E (),"= U, . Then d(z, X ) = 0, and since X is closed, we have z E X. There is a sequence {x,) c X such that x, -+ s. Let V,(z)be an open neighborhood in Z of z for which

For each n, there is an index k, such that X,E V,(Z)for k 2 k,. From (11) we see that d,(xi,xj) < l/n for i,j 2 k,, so {x,) is Cauchy in the complete space (X, dl) and hence has a limit in X. But the limit is z by assumption, so X = n;=l Un.

108

7.


For the converse part of the theorem, suppose (2, d) is a complete metric U,, where each Un is open in Z. Define a metric dl space and Y = on Y by

If {y,) is Cauchy in (Y,d,), then it is also Cauchy in (Z,d), and thus has a limit y E Z. For each n,

as i, j + oo,so [ l / d ( y , , Z - U,)] remains bounded as k y E U, for every n, hence y E Y. Q.E.D.

-+

oo.It follows that

As we remarked earlier without proof, the Hilbert cube inherits metrizability and separability from the unit interval. It also inherits topological completeness. This is a special case of the fact, which we now prove, that completeness and separability of metrizable spaces are preserved when taking countable products. Proposition 7.4 Let X I , X,, . . be a sequence of metrizable spaces and ;

Y, = X1X2.. . X n , Y = XIX,. . - . Then Y and each Y, is metrizable. If each X, is separable or topologically complete, then Y and each Y, is separable or topologically complete, respectively.

Proof

If d, is a metric on X, consistent with its topology, then

. . .), is a metric on Y consistent with the where y = (y,, y,, . . .), y^ = (y^,,ij,, product topology. If each (X,, d,) is complete, clearly (Y, d) is complete. If 9,is a countable base for the topology on X,, the collection of sets of the form GIG2. . . G,X,+ ,Xn+, . . . , where G, ranges over 9, and n ranges over the positive integers, is a countable base for the product topology on Y. The arguments for the product spaces Y, are similar. Q.E.D.

Combining Propositions 7.2-7.4, we see that every separable, topologically complete space is homeomorphic to a G,-subset of the Hilbert cube, and conversely, every G,-subset of the Hilbert cube is separable and topologically complete. We state a second consequence of these propositions as a corollary. Corollary 7.4.1 Every separable, topologically complete space can be homeomorphically embedded as a dense G,-set in a compact metric space.

Proof Let X be separable and topologically complete and let cp be a -is a G,-subset homeomorphism from X into 3.Since 3 is metrizable, cp(X) of (Proposition 7.3) and thus a dense G,-subset of cp(X). Tychonoff's Q.E.D. theorem implies that 2 is compact, so cp(X) is compact. If X and Z are topological spaces, cp a homeomorphism from Z onto X, and d a metric on X consistent with its topology such that (X, d) is complete, then dl defined by (10) is a metric on Z consistent with its topology, and (Z, dl) is also complete. Thus topological completeness is preserved under homeomorphisms. The same is true for separability, as is well known. Topological completeness is somewhat different from separability, however, in that one must produce a metric to verify it. It is quite possible that a space has two metrics consistent with its topology, is a complete metric space with one, but is not a complete metric space with the other. For example, let X = {I,+,*, . . .) have the discrete topology,

and

Then (X, dl) is complete, but (X, d,) is not. A more surprising example is that the set N o of irrational numbers between 0 and 1 with the usual topology is topologically complete. To see this, write .No= n,,,([O, 11 - {r)), where Q is the set of rational numbers. It follows that N o is a G,-subset of [O,1] and is thus topologically complete by Proposition 7.3. Another proof is obtained as follows. Let N be the set of positive integers with the discrete topology and N the product of countably many copies of N. The space .N is called the Baire null space and is topologically complete (Proposition 7.4). The topological completeness of N o follows from the fact that N and N o are homeomorphic. We give the rather lengthy proof of this because it is not readily available elsewhere. This homeomorphism will be used only to construct a counterexample (Example 1 in Chapter 8), so it may be skipped by the reader without loss of continuity. Proposition 7.5 The topological spaces N o and JV are homeomorphic.

CN, Proof Let C be the set offinite sequences ofpositive integers. 1 f z ~ u we will represent its components by Ck. Similarly, 9, will represent the components of an element 2 of C u N . The length of z E C u .N is defined to be the number of its components. If s has length greater than or equal to k, we define zk = ( 0,

A totally bounded metric space is necessarily separable, since U,"=, F,,, is a countable dense subset. Total boundedness depends on the metric, however, and a space which is totally bounded (and separable) with one metric may not be totally bounded with another. Like separability, total boundedness is preserved under passage to subspaces, i.e., if (X, d ) is totally bounded and Y c X , then ( Y ,d ) is totally bounded. To see this, take E > 0 and let F,,, be a finite subset of X such that

Choose a point, if possible, in each of the sets

Y n { Y ~ x l d ( x , Y 0, the

contains a finite cover of X . Hence, ( X ,d ) is totally bounded. If ( X ,d ) is complete and totally bounded and S = { s j ) is a sequence in ( X , d ) , then an infinite subsequence S , c S must lie in some set B , = { y ~ X l d ( x y) , , < 1 ) . Since B , is totally bounded, an infinite subsequence S2 c S , must lie in some set B2 = { Y E B l l d ( x z ,y ) < 3). Continuing in this manner, we have for each n an infinite sequence S,, c S, lying in B,+ = ( Y E B,ld(x,+ y) < l / ( n + 1 ) ) . Let j , < j2 < . . . be such that sj, E S,. Then ( s j n ) is Cauchy and thus convergent. Therefore S has an accumulation Q.E.D. point, and the compactness of ( X ,d ) follows.

,,

,

,

Corollary 7.6.1 The Hilbert cube is totally bounded under any metric consistent with its topology, and every separable metrizable space has a totally bounded metrization.

Proof The Hilbert cube is compact by Tychonoff's theorem. Urysohn's theorem (Proposition 7.2) can be used to homeomorphically embed a given separable metrizable space into the Hilbert cube. Q.E.D.

As mentioned previously, total boundedness implies separability. By combining this fact with Proposition 7.6, we obtain the following corollary. Corollary 7.6.2 A compact metric space is complete and separable.

If X is a metrizable space, the set of all bounded, continuous, realvalued functions on X is denoted C ( X ) .As is well known, C ( X )is a Banach space under the norm

and we will always take C ( X )to have the metric and topology corresponding to this norm. If d is a metric on X consistent with its topology, we denote by U d ( X )the collection of functions in C ( X )which are uniformly continuous with respect to d. We take U d ( X )to have the relative topology of C ( X ) . We conclude this section with a discussion of the properties C ( X )and U,(X) inherit from X . Proposition 7.7 If X is a compact metrizable space, then C ( X ) is separable.

Proof The space X is separable (Corollary 7.6.2). Let {x,} be a countable dense subset of X and let F,, F,, . . . be an enumeration of the collection of sets of the form .( y E X I d ( x k , y ) I l l n ) , where k and n range over the positive

114

7.


xj

integers. For any disjoint pair F , and F j , let be a continuous function taking values in [O,1] such that J j ( x ) = 0 for x E Fi and J j ( x ) = 1 for x~ F j . If F iand F j are not disjoint, let J j be identically one. Let %' consist of the functions J j as i and j range over the positive integers. The collection %' clearly separates points in X , i.e., given x # y, there exists f E%' for which .f(x) # .f(y). Let 9 be the collection of finite-degree polynomials over %', i.e., a typical element in .VP has the form

E %',and the summation is finite. where a(i,, . . . ,in;j,, . . . ,jn) E R, J . . ,in Then 9 is a vector space under addition and the product of two elements in -9'is again in 9.With these operations 9 is an algebra, and by the StoneWeierstrass theorem, .VP is dense in C ( X ) .Let Pobe the collection of finitedegree polynomials over %? with rational coefficients. An easy approximation argument shows that Pois dense in 9, and thus dense in C ( X )as well. Since 9, is countable, C ( X ) is separable. Q.E.D.

Definition 7.5 Let (X, d l ) and ( Y ,d,) be metric spaces. A mapping cp :X -+ Y is an isometry if

In this case we say that ( X ,d l ) and (cp(X),d2) are isometric spaces. If ( X ,d l ) and ( Y ,d,) are as in Definition 7.5, we may regard the former as a subspace of the later, and the distances between points in X are unaffected by this embedding. Thus an isometry is a metric-preserving homeomorphism. Proposition 7.8 Let ( X ,d ) be a metric space. There exists a complete metric space (X,, d l ) , called the completion of ( X ,d ) , and an isometry cp:X + X , such that q ( X ) is dense in X,. Proof The construction of the completion of a metric space is standard, so we content ourselves with a sketch of it. Given the metric space ( X ,d ) , define an equivalence relation on the set of Cauchy sequences in ( X ,d ) by ( x ,j

-

-

-

(x;)

lim d(x,, x;)

n+

= 0.

oC

Let X , be the set of equivalence classes of Cauchy sequences in ( X ,d ) under this relation and let dl be defined on X d X , by d l ( x ,y) = lim d ( x n ,y,), n- x

where (x,} and ( y,} are chosen to represent the equivalence classes x and y. It is straightforward to verify that the limit in (15) exists for every pair of Cauchy sequences {x,) and {y,), and it is independent of the particular sequences chosen to represent the equivalence classes x and y. Furthermore, (Xd,dl)can be shown to be a complete metric space, and the mapping cp which takes X E Xinto the equivalence class in X, containing the Cauchy sequence (x, x,. . .) is an isometry. The image of X under cp is dense in X,. Q.E.D. We can regard X, as consisting of X together with limits of all Cauchy sequences in X. We are really interested in the case in which (X, d) is totally bounded, for which we have the following result. Corollary 7.8.1 Let (X,d) be a totally bounded metric space. There exists a compact metric space (X,, dl) and an isometry cp:X -,X, such that cp(X) is dense in X,.

Proof In light of Propositions 7.6 and 7.8, it suffices to prove that the completion (X,,d,) of (X, d) is totally bounded. Choose E > 0. Regarding (X, d) as a subspace of (X,, dl), choose a finite set F , of X for which

Since X is dense in X,, we have X,

=

U { y E X,ld,(x, y) < E}.

Q.E.D.

x~ F ,

If X is a separable metrizable space, it is not necessarily true that C(X) is separable (unless X is compact, in which case we have Proposition 7.7). For example, let f : R + [O,1] be defined as

and given an infinite sequence b = (PI,P,, . . .) of zeroes and ones, define

We have constructed an uncountable collection of functions f, in C(R) such that if b1 # b2, then I/&, - f,,ll = 1. Therefore, C(R) cannot be separable. It is true, however, that given a separable metrizable space X, there is a metric d on X consistent with its topology such that U,(X) is separable. This is a consequence of the next proposition and the fact that separability implies the existence of a totally bounded metrization (Corollary 7.6.1). We prove this proposition with the aid of the following lemma.

116

7.


Lemma 7.3 Let Y be a metrizable space, d a metric on Y consistent with its topology, and X c Y. If g E U,(X), then g has a continuous extension to Y , i.e., there exists @ E C ( Y )such that g(x) = g^(x)for every x E X , and the extension g^ can be chosen to satisfy lgll = Qll. If X is dense in Y, g^ is unique.

1

11

Proof Since g is uniformly continuous on X , given E > 0 there exists 6 ( ~>) 0 such that if x , , x , E X and d(x,, x,) < then Ig(x,) - g(x,)l < r. Suppose EX. Then there exists a sequence {x,) c X for which x, + y. Given r > 0, there exists N ( E )such that d(x,, x,) l 6(r) for all n, m 2 N(E), so {g(x,)) is Cauchy in R. Define g^(y)= lirn,,, g(x,). Note that n 2 N ( E ) implies Ig(x,) - g^( y) I E. Suppose now that X E X and d(x, y) 5 6 ( ~ ) / 2Choose . n 2 N ( E )SO that d(x,, y) < 6 ( ~ ) / 2Then . d(x,x,) I a(&)and

a(&),

1

This shows that for any sequence ( x ; ) c X with x; + y, we have g^(y)= lim,,,g(x~), so the definition of g^(y) is independent of the particular sequence (x,) chosen. If y E X , we can take x, = y, n = 1,2, . . . , and obtain g ( y ) = g^(y),so g^ is an extension of g. If { y,) is a sequence in X which converges to y e X , then there exist sequences {x,,) in X with y, = lim,,, x,,. . . Choose n, < n, < . . . so that lim,,, x,,_ = y and d(x,,,,,, y,) < 6(l/m)/2. Then

Letting m + oo in (18) and using (17),we conclude that g^(y)= lirn,, and g^ is continuous on X . It is clear that

,g^(y,)

If X = Y, g^ is clearly unique and we are done. If X is a proper subset of Y, use the Tietze extension theorem (see, e.g., Ash [All or Dugundji [D7]) to extend g^ to all of Y so that

Proposition 7.9 If ( X ,d ) is a totally bounded metric space, then U,(X) is separable. Proof Corollary 7.8.1 tells us that ( X ,d ) can be isometrically embedded as a dense subset of a compact metric space (X,,d,). We regard X as a

7.3

117

BOREL SPACES

subspace of X,. Given any g E U,(X), by Lemma 7.3, g has a unique extension Il @ l . The mapping g -+ g^ is linear and normpreserving, thus an isometry from U,(X) to C(X,). The latter space is sepQ.E.D. arable by Proposition 7.7, and the separability of U,(X) follows. @ E C(X,) such that 11g11 =

7.3 Borel Spaces

The coi~structionsnecessary for the subsequent theory of dynamic programming are impossible when the state space and control space are arbitrary sets or even when they are arbitrary measurable spaces. For this reason, we introduce the concept of a Borel space, and in this and subsequent sections we develop the properties of Borel spaces which permit these constructions. Definition 7.6 If X is a topological space, the smallest o-algebra of subsets of X which contains all open subsets of X is called the Borel o-algebra and is denoted by B,.The members of Bxare called the Borel subsets of X.

If X is separable and metrizable and F is a o-algebra on X containing a subbase Y for its topology, then F contains Bx. This is because, from Proposition 7.1, any open set in X can be written as a countable union of finite intersections of sets in Y . Thus we have Bx= o(Y) for any subbase Y. We will often refer to the smallest a-algebra containing a class of subsets as the o-algebra generated by the class. Thus, Bxis the o-algebra generated by the class of open subsets of X. Note that 93, is the class of Borel subsets of the real numbers in the usual sense, i.e., the o-algebra generated by the intervals. Given a class of real-valued functions on a topological space X, it is common to speak of the weakest topology with respect to which all functions in the class are continuous. In a similar vein, one can speak of the smallest o-algebra with respect to which all functions in the class are measurable. If X is a metrizable space, it is easy to show that its topology is the weakest with respect to which all functions in C(X) are continuous. The following proposition is the analogous result for 9,. In the proof and in subsequent proofs, we will use the fact that for any two sets R, R', any collection V of subsets of Q', and any function f :R + R', we have

.[f

-l(@)l = f -'[o(V)l-

is the smallest Proposition 7.10 Let X be a metrizable space. Then o-algebra with respect to which every function in C(X) is measurable, i.e., ' g=~ f E C ( X ) . f - '(,@R)].

118

7.


Proof Denote 9 = o[UfeCcX, f -l(gR)] and let F R be the topology of R. We have

To prove the reverse containment $9, c 9 we need only establish that 9 contains every nonempty open set. By Lemma 7.2, it suffices to show that 9contains every nonempty closed set. Let A be such a set. We may assume without loss of generality that A # X, so there exists X E X - A. Let B = ( x ) , Q.E.D. and let f be given by Lemma 7.1. Then A = f - ' ( { O ) ) E 9. We use Lemma 7.2 to prove another useful characterization of the Borel o-algebra in a metrizable space. Proposition 7.11 Let X be a metrizable space. Then 93, is the smallest class of sets which is closed under countable unions and intersections and contains every closed (open) set.

Proof Let 9 be the smallest class of sets which contains every closed set and is closed under countable unions and intersections, i.e., 9 is the intersection of all such classes. Then 9 c 98, and it suffices to prove that 9 is closed under complementation. Let 9' be the class of complements of Then 9 ' is also closed under countable unions and intersections. sets in 9. Lemma 7.2 implies that 9 contains every open set, so 9' contains every Given D E 9, we have D E 9', SO closed set, and consequently 9 c 9'. DC€9. Q.E.D. Definition 7.7 Let X be a topological space. If there exists a complete separable metric space Y and a Borel subset BE.@, such that X is homeomorphic to B, then X is said to be a Bovel space. The empty set will also be regarded as a Borel space.

Note that every Borel space is metrizable and separable. Also, every complete separable metrizable space is a Borel space. Examples of Borel spaces are R, Rn, and R* with the weakest topology containing the intervals [-GO, a), (P, a ] , (a, P), or, P E R . (This is also the topology that makes the function cp defined by

7.3

119

BOREL SPACES

a homeomorphism from R* onto [- 1,1]). Any countable set X with the discrete topology (i.e., the topology consisting of all subsets of X) is also a Borel space. We will show that every Borel subset of a Borel space is itself a Borel space. For this we shall need the following two lemmas. The proof of the first is elementary and is left to the reader. Lemma 7.4 If Y is a topological space and E c Y, then the o-algebra BE generated by the relative topology coincides with the relative o-algebra, i.e., the collection {E n C~CEB,).In particular, if E € B y , then $8, consists of the Borel subsets of Y contained in E. Lemma 7.5 If X and Y are topological spaces and cp is a homeomorphism of X into Y , then cp(Bx)= B,(,,. Proof If Fxis the topology of X, then cp(.Fx) is the topology of p(X). Since cp is one-to-one, we have that cp is the inverse of a mapping, and

Proposition 7.12 If X is a Borel space and B E B,, then B is a Borel space. Proof Let cp be a homeomorphism of X into some complete separable metric space Y such that cp(X)~.@,.From Lemma 7.5 and the fact that B E.gX,we obtain cp(B)E ,Q,(,). It follows from Lemma 7.4 that q(B)E By. Q.E.D.

Like separability and completeness, the property of being a Borel space is preserved when taking countable Cartesian products. Proposition 7.13 Let X I , X,, . . . be a sequence of Borel spaces and Y, = X l X 2 . . . X,, Y = X l X 2 . . . . Then Y and each Y, with the product topology is a Borel space and the Borel o-algebras coincide with the product = BxI.Qx2. . . Bxnand By = Bx,Bx2 ' . o-algebras, i.e., BYn '

'

Proof As in Proposition 7.4, we focus our attention on the more difficult infinite product. Consider the last statement of the proposition. Each X, has a countable base 9, for its topology, and the collection of sets of the form G,G, . . . GnX,+,Xn+, . . . , where G, ranges over 9, and n ranges over the positive integers, is a base for the product topology on Y. The o-algebra generated by this topology is 98,.Recall that the product o-algebra ,%,,B,,... is the smallest o-algebra containing all finite-dimensional measurable rectangles, i.e., all sets of the form BIB2 . . . B,X,+ ,, X,+, . . . , where BkE B,,, k = 1, . . . , n. It is clear that each basic set of the product topology on Y is a finite-dimensional measurable rectangle, and since each open subset of Y is a countable union of these basic open sets, every open subset of Y is .gx,9Zlx2 . . . measurable. We conclude that By c @ x I ~ x. .2. . (Note that

120

7.


this argument relies only on the separability of the spaces X I , X 2 , . . . . Without this separability assumption, the argument fails and the conclusion is false.) The reverse set containment follows from the observation that for each kand B,E&?,,, X , X 2 ~ ~ ~ X k ~ , B k X k + , ~ ~ ~ ~ ~ y . To prove that Y is a Borel space, note that X , can be mapped by a homeomorphism cp, onto a Borel subset of a separable topologically complete space 2,.The product = 2,z2. . . is separable and topologically complete, and cp : Y + I;defined by cp(x1,x2, . -

-1 = (cp,(x,), cp2(x2),. . .)

is a homeomorphism from Y onto cp,(X,)cp2(X2). . . . This last set is in L@z,&?z,' ' . = g F ,and the conclusion follows. Q.E.D. Definition 7.8 Let X and Y be topological spaces. A function f : X + Y is Borel-measurable iff - ' ( B )E 3, for every B E B y . In many respects, Borel-measurable functions relate to Borel o-algebras as continuous functions relate to topologies. We have already used the fact, for example, that if f , : X -+ Y , is continuous from a topological space X to a topological space Y,, k = 1,2, . . . , then F : X + Y, Y2 . . . defined by F(x) = ( f l ( x )f2(x), , . . .) is also continuous. This follows from the componentwise nature of convergence in product spaces. There is an analogous fact for Borel-measurable functions and Borel spaces. Proposition 7.14 Let X be a Borel space, Y,, Y2, . . . a sequence of Borel spaces, and f,: X + Y , a sequence of functions. If each f, is Borel-measurable, k = 1,2,. . . , then the function F:X + Y, Y , - - - defined by and the functions F,:X

-t

Y, Y 2 .. . Y, defined by

are Borel-measurable. Conversely, if F is Borel-measurable, then each f, is Borel-measurable, k = 1,2, . . . , and if some F, is Borel-measurable, then f,, f 2 , . . . ,f, are Borel-measurable. Proof Again we consider only the infinite product. The Borel o-algebra in Y 1 Y 2 . .. is generated by sets of the form B I B 2 . . . , where B k ~ g y k , k = 1 , 2 , . . . .Now F - ' ( B I B 2 . . .) = f ;'(B,) nf ;'(B2) n . . . .

(19)

The left side of (19) is in B, for each B k € B Y , ,k = 1,2,. . . , if and only if the sets f , '(B,) are in % , , for each B, E g y k ,k = 1,2,. . . , and the result follows. Q.E.D.

7.3

BOREL SPACES

121

Corollary 7.14.1 Let X and Y be Borel spaces, D a Borel subset of X, and f :D + Y Borel-measurable. Then

Proof The mappings (x, y) + f (x) and (x, y) + y are Borel-measurable from DY to Y , so the mapping F(x, y) = (f(x), y) is Borel-measurable from D Y to Y Y . Then

Since {(y,y ) j y ~Y ) is closed in Y Y , Gr(f) is Borel-measurable.

Q.E.D.

The concept of homeomorphism is instrumental in classifying topological spaces, since it allows us to identify those which are "topologically equivalent." We can also classify measurable spaces by identifying those which, when regarded only as sets with o-algebras, are indistinguishable. We specialize this concept to Borel spaces. Definition 7.9 Let X and Y be Borel spaces and cp:X -+ Y a Borelmeasurable, one-to-one function such that cp-' is Borel-measurable on cp(X). Then cp is called a Borel isomorphism, and we say that X and cp(X) are Borel-isomorphic (or simply isomorphic). If X and Y are Borel spaces and cp : X + Y is a Borel isomorphism, it is tempting to think of X and q(X) as identical measurable spaces. The difficulty with this is that X is a Borel space, but q(X) is not required to be. This discrepancy is eliminated by the following intuitively plausible proposition, the rather lengthy proof of which can be found in Chapter I, Section 3 of Parthasarathy [PI]. We will not have occasion to use this result. Proposition 7.15 (Kuratowski's theorem) Let X be a Borel space, Y a separable metrizable space, and cp :X -t Y one-to-one and Borel-measurable. Then cp(X) is a Borel subset of Y and cp-' is Borel-measurable. In particular, if Y is a Borel space, then X and p(X) are isomorphic Borel spaces. The advantage of classifying spaces by means of Borel isomorphisms is illustrated by the following result. We need this proposition for the subsequent development, but the proof is rather lengthy and is relegated to Appendix B, Section 2. Proposition 7.16 Let X and Y be Borel spaces. Then X and Y are isomorphic if and only if they have the same cardinality. Proposition 7.16 leads to a consideration of the possible cardinalities of Borel spaces. Of course, Borel spaces which are countably infinite are

122

7.


possible, as are Borel spaces which consist of a given finite number of elements. In both these cases, the Borel o-algebra is the power set and the conclusion of Proposition 7.16 is trivial. Because every Borel space can be homeomorphically embedded in the Hilbert cube, every Borel space has cardinality less than or equal to c. Even if one were to admit the possibility of an uncountable cardinality strictly less than c, the proof of Proposition 7.16 as given in Appendix B shows that every uncountable Borel space has cardinality c. By combining this fact with Proposition 7.16, we obtain the following corollary. Corollary 7.16.1 Every uncountable Borel space is Borel-isomorphic to every other uncountable Borel space. In particular, every uncountable Borel space is isomorphic to the unit interval [O,1] and the Baire null space M . 7.4

Probability Measures on Borel Spaces

If X is a metrizable space, we shall refer to a probability measure p on the measurable space (X,B,) as simply a probability measure on X. The set of all probability measures on X will be denoted by P(X). A probability measure p~ P(X) determines a linear functional 1,: C(X) + R defined by

On the other hand, a function f Of: P(X) + R defined by

E

C(X) determines a real-valued function

These relationships and the metrizability of the underlying space X allow us to show several properties of P(X). In particular, we will prove that there is a natural topology on P(X), the weakest topology with respect to which every mapping of the form of (21) is continuous, under which P(X) is a Borel space whenever X is a Borel space. 7.4.1 Characterization of Probability Measures Definition 7.10 Let X be a metrizable space. A probability measure p E P(X) is said to be regular if for every B EBx,

p(B) = s u p j p ( ~ ) I c F B, F closed)

= inf

j p ( G ) l ~c G, G open).

(22)

Proposition 7.17 Let X be a metrizable space. Every probability measure in P(X) is regular.

7.4

123

PROBABILITY MEASURES ON BOREL SPACES

Proof Let p E P(X) be given and let Q be the collection of Be.BX for which (22) holds. If H c X is open, then H = F,, where {F,) is an increasing sequence of closed sets (Lemma 7.2), so

U,"=,

inf [p(G)IH c G, G open)

= p(H) =

lim p(F,) n- m

I sup (p(F)IF c H, F closed) I p(H).

Therefore & contains every open subset of X. We show that & is a o-algebra and conclude that Q = 99,. If B E 8,then p(Bc) = 1 - p(B) = 1 - supjp(F)IF c B, F closed) = inf .( p(G)IBc c G, G open), and similarly, p(Bc) = sup{p(F)IF c Bc, F closed), so 8 is closed under complementation. Now suppose {B,: is a sequence of sets in 6.Choose E > 0 and F, c B, c G, such that Fn is closed, Gn is open, and p(Gn - F,) I ~12".Then

and since E is arbitrary, p

U B, (nyl

) { = inf

It is also apparent from (23) that

so for N sufficiently large,

p(G)

U B, !nyl

I

c G, G open .

124

7.

The finite union

Uk


Fn is a closed subset of

U,"=,Bn and E is arbitrary, so

This shows that 8 is closed under countable unions and completes the proof. Q.E.D. From Proposition 7.17 we conclude that a probability measure on a metrizable space is completely determined by its values on the open or closed sets. The following proposition is a similar result. It states that a probability measure p on a metric space ( X ,d ) is completely determined by the values Jg dp, where g ranges over U d ( X ) . Proposition 7.18 Let X be a metrizable space and d a metric on X consistent with its topology. If p,, p2 E P ( X ) and

then p,

= p2 .

Proof

Let F be any closed proper subset of X and let G,

=

( x E Xld(x, F )

< lln). For sufficiently large n, F and G,' are disjoint nonempty closed sets for which inf,.,,,,,,.nd(x, y) > 0, so by Lemma 7.1, there exist functions

f,E U,(X) such that f,(x) = 0 for x E G,', f,(x) = 1 for x E F, and 0 I f , ( x ) I 1 for every x E X . Then

and so

Reversing the roles of p1 and p2, we obtain pl(F) = p2(F). Proposition 7.17 Q.E.D. implies p,(B) = p2(B)for every B E 97,. 7.4.2 The Weak Topology

We turn now to a discussion of topologies on P ( X ) , where X is a metrizable space. Given E > 0, p E P ( X ) , and f E C ( X ) , define the subset of P ( X ): V , ( p : f )= { q t ~ ( ~ ) I J f d qJfdPlp <Ej.

7.4


I f D c C ( X ) ,consider the collection of subsets of P ( X ) :

Let F ( D ) be the weakest topology on P ( X ) which contains the collection V ( D ) ,i.e., the topology for which V ( D )is a subbase. Lemma 7.6 Let X be a metrizable space and D c C ( X ) .Let {p,) be a net in P ( X ) and p E P ( X ) . Then p, + p relative to the topology F ( D ) if and only if J ,f dpa + J f dp for every ,f E D. Proof Suppose p, -t p and f E D. Then, given E > 0, there exists /3 such that a 2 p implies y, E T/,(p;f).Hence dp, -+ J f d p . Conversely, if i f dp, -+ i f dp for every f E D, and G EF ( D ) contains p, then p is also contained in I/,,(p;f,)c G, where E, > 0 andf, E D, k = 1,. . . ,n. some basic open set Choose /3 such that for all a 2 /3 we have 1j.L dp, - Jf, dpl < E,, k = 1,. . . ,n. Then ~ , E Gfor a 2 P, so pa +p. Q.E.D.

i,f

n;=,

We are really interested in . T [ C ( X ) ] , the so-called weak topology on P ( X ) .The space C ( X )is too large to be manipulated easily, so we will need a . a set D is produced countable set D c C ( X )such that 3 ( D ) = 3 [ C ( X ) ] Such by the next three lemmas. Lemma 7.7 Let X be a metrizable space and d a metric on X consistent with its topology. If f E C ( X ) , then there exist sequences {g,) and {h,] in U,(X) such that g, .T. f and h, f . Proof We need only produce the sequence {g,}, since the other case follows by considering - f . In Lemma 7.14 under weaker assumptions we will have occasion to utilize the construction about to be described, so we are careful to point out which assumptions are being used. I f f E C ( X ) ,then f is bounded below by some b E R, and for at least one x , E X, f(x,) < c ~ . Define

Note that for every x E X ,

b I g,(x) 5 f ( x )

+ nd(x,x ) = f ( x ) ,

and

b I g,(x) l f ( x o ) + nd(x, x,) c a. Thus

7.4

PROBABILITY MEASURES O N BOREL SPACES

Lemma 7.9 Let X be a metrizable space and d a metric on X consistent with its topology. If D is dense in U,(X), then F [ U , ( X ) ] = J"(D).

Proof It is clear that .F(D) c . T [ U , ( X ) ] . To prove the reverse set containment, we choose a set I/,(p;g ) ^lr[U,(X)], ~ select a point po in this set, and construct a set in T ( D )containing po and contained in I/,(p;g).Let

3 . for any q~ T/,,,,(p0; h), we have Let h~ D be such that llg - hll < ~ ~ 1Then

Proposition 7.19 Let X be a separable metrizable space. There is a metric d on X consistent with its topology and a countable dense subset D of U,(X) such that F ( D ) is the weak topology F [ C ( X ) ]on P ( X ) .

Proof Corollary 7.6.1 states that the separable metrizable space X has a totally bounded metrization d. By Proposition 7.9, there exists a countable dense set D in U,(X). The conclusion follows from Lemmas 7.8 and 7.9. Q.E.D. From this point on, whenever X is a metrizable space, we will understand P ( X ) to be a topological space with the weak topology F [ C ( X ) ] . We will show that when X is separable and metrizable, P ( X ) is separable and metrizable; when X is compact and metrizable, P ( X ) is compact and metrizable; when X is separable and topologically complete, P ( X ) is separable and topologically complete; and when X is a Borel space, P ( X ) is a Borel space. Proposition 7.20 If X is a separable metrizable space, then P ( X ) is separable and metrizable.

7.

128


Proof Let d be a metric on X consistent with its topology and D a countable dense subset of Ud(X) such that F ( D ) is the weak topology on P(X) (Proposition 7.19). Let R" be the product of countably many copies of the real line and let cp :P(X) + R" be defined by

where {g,,g,,. . .) is an enumeration of D. We will show that cp is a homeomorphism, and since Rw is metrizable and separable (Proposition 7.4), these properties for P(X) will follow. Suppose that cp(pl) = cp(p2), SO that Jg,dp, = Jg,dp, for every ED. If g~ Ud(X),then there exists a sequence {g,,} c D such that Ilg,, - gll + 0 as j -+ co. Then

< 2 lim supllgkj - gll = 0, j- ca

so Jg dp, = Jg dp,. Proposition 7.18 implies that p1 = p,, so cp is one-to-one. ~ the mapping p + Jg,dp is continuous by Lemma 7.6, so For each g , D, cp is continuous. To show that cp-' is continuous, let (p,) be a net in P(X) such that cp(p,) + cp(p) for some p~ P(X). Then Jg,dp, + Jg, dp for every gk~D,andbyLernma7.6,p,+p. Q.E.D. Proposition 7.20 guarantees that when X is separable and metrizable, the topology on P(X) can be characterized in terms of convergent sequences rather than nets. We give several conditions which are equivalent to convergence in P(X). Proposition 7.21 Let X be a separable metrizable space and let d be a metric on X consistent with its topology. Let (pn} be a sequence in P(X) and p E P(X). The following statements are equivalent:

(a) Pn+P; (b) Jfdp,+Jfdpforevery f EC(X); (c) j"g dpn jg dp for every g E U d ( X ) ; (d) lim sup,,, p,(F) I p(F) for every closed set F c X; (e) lim inf,,, , p,(G) 2 p(G) for every open set G c X. +

7.4

129


Proof The equivalence of (a), (b), and (c) follows from Lemmas 7.6 and 7.8. The equivalence of (d) and (e) follows by complementation. To show that (b) implies (d), let F be a closed proper nonempty subset , < llk). For k sufficiently large, F and G; of X and let Gk = ( x ~ X l d ( xF) are disjoint nonempty sets, and there exist functions f k e C ( X ) such that f,(x) = 1 for x E F, f,(x) = 0 for x E G;, and 0 I f(x) I 1 for every x E X. Using (b) we have

and letting k + oo,we obtain (d). To show that (d) implies (b), choose f E C(X) and assume without loss of generality that 0 I f I 1. Choose a positive integer K and define closed sets

Define p :X -t [O, 11 by

where FK+ =

a.Then f - (1/K) Ip If , and, for any q

EP ( X ) ,

Using (d) we have

S

lim sup f dpn - (1/K) 5 lirn sup S p dpn n+ m

n+ m

K

lim sup

=

n+m

k=l

~n(~k)

and since K is arbitrary, we obtain lim sup J-f dp. 5 J f dp n+ w

for every f~C ( X ) .In particular, (29) holds for -f , SO

S

S

lim inf f dp. = - lim sup (- f )dpn 2 n+a.

n+m

Combine (29) and (30) to conclude (b).

S(-f )dp Sf dp.

Q.E.D.

=

(30)

7.

130


When X is a metrizable space, we denote by p, the probability measure on p(X) which assigns unit point mass to x, i.e., p,(B) = 1 if and only if x E B. Corollary 7.21.1 Let X be a metrizable space. The mapping 6 :X + P(X) defined by S(x) = p, is a homeomorphism.

Proof It is clear that 6 is one-to-one. Suppose {x,) is a sequence in X and x E X. If X, -+ x and G is an open subset of X, then there are two possibilities. Either X E G , in which case X,EG for sufficiently large n, so p,.(G) = 1 = px(G),or else x $ G, in which case lim inf,, , p,,,(G) 2 lim inf,, , 0 = p,(G). Proposition 7.21 implies pXn-+ p,, SO 6 is continuous. On the other hand, if pXn-+ p, and G is an open neighborhood of x, then since liminf,,, pXn(G)2 px(G) = 1, we must have X,E G for sufficiently large n, i.e., x, -+ x. This shows that 6 is a homeomorphism. Q.E.D. From Corollary 7.21.1 we see that p, can converge to p in such a way that strict inequality holds in (d) and (e) of Proposition 7.21. For example, let G c X be open, let x be on the boundary of G, and let x, converge to x through G. Then pXn(G)= 1 for every n, but p,(G) = 0. We now show that compactness of X is inherited by P(X). Proposition 7.22 If X is a compact metrizable space, then P(X) is a compact metrizable space.

Proof If X is a compact metrizable space, it is separable (Corollary 7.6.2) and C(X) is separable (Proposition 7.7). Let { fk) be a countable set in C(X) such that f, = 1,II I 1 for every k, and .( fk) is dense in the unit sphere (f E C(X)lllf I 1). Let [ - 1, l I m be the product of countably many copies of [- 1,1] and define q : P ( X ) -+ [- 1,1]" by

11

fkl l

A trivial modification of the proof of Proposition 7.20 shows q is a homeomorphism. We will show that q[P(X)] is closed in the compact space [- 1, I]", and the compactness of P(X) will follow. Suppose (p,) is a sequence in P(X) and q(p,) + (a,, a,, . . . ) E [ - 1,1Im. Given F > 0 and f E C(X) with 11 f I 1, there is a functionf, with f - , f k l < ,513. There is a positive integer N such that n,m 2 N implies IJf,dp, - Sfkdpm/< 4 3 . Then

11

11

7.4

131


11 11 > 1, define

so { jf d p n ) is Cauchy in [ - 1,1]. Denote its limit by E( f ) . If f

It is easily verified that E is a linear functional on C ( X ) ,that E( f ) 2 0 whenever f 2 0, IE(f)l I 11 f for every f e C ( X ) , and E ( f , ) = 1. Suppose (h,) is a sequence in C ( X )and hn(x)-1 0 for every x E X . Then for each E > 0 , the set K,(E) = {xlh,(x) 2 E ) is compact, and K,(E) = Therefore, for n sufficiently large, K,(E) = which implies Ilhn/lL O . Consequently, E(h,) L O . This shows that the functional E is a Daniel1 integral, and by a classical theorem (see, e.g., Royden [R5, p. 299, Proposition 211) there exists a unique f -l(BR)] which satisfies E( f ) = j fdp for probability measure on o[ufEc(x, every f E C ( X ) . Proposition 7.10 implies p ~ P ( X ) . We have

11

a,

so q ( p n )+ q(p). This proves q [ P ( X ) ] is closed.

a.

Q.E.D.

In order to show that toplogical completeness and separability of X imply the same properties for P ( X ) , we need the following lemma. Lemma 7.10 Let X and Y be separable metrizable spaces and cp :X + Y a homeomorphism. The mapping $: P ( X ) -+ P ( Y ) defined by

is a homeomorphism. Proof Suppose p,, p2 E P ( X ) and p1 # p, . Since p1 and p2 are regular, there is an open set G c X for which pl(G) f p2(G). The image q ( G ) is relatively open in q ( X ) , so q ( G ) = q ( X )n B, where B is open in Y. It is clear that

so $ is one-to-one. Let [p,) be a sequence in P ( X ) and p ~ P ( X ) . If p, -+ p, then since q - ' ( H ) is open in X for every open set H c Y , Proposition 7.21 implies lim inf$(p,)(H) = lim i n f p n [ q - ' ( H ) ]2 p [ q - ' ( H ) ] n-

K,

= $(p)(H),

n-cc

so $ ( p H )-+ $ ( p ) and $ is continuous. If we are given {p,) and p such that $(pn) -+ IC/(p),a reversal of this argument shows that p, -+ p and I+-' is continuous. Q.E.D. Proposition 7.23 If X is a topologically complete separable space, then P ( X ) is topologically complete and separable.

132

7.


Proof By Urysohn's theorem (Proposition 7.2) there is a homeomorphism cp:X -+ 2 , and the mapping $ obtained by replacing Y by 2 in Lemma 7.10 is a homeomorphism from P(X)to P ( 2 ) .Alexandroff's theorem (Proposition 7.3) implies q ( X ) is a G,-subset of I f ,and we see that We will show) I [P(X)] is a G,-subset of the compact space P(X)(Proposition 7.22) and use Alexandroff's theorem again to conclude that P(X) is topologically complete. Since q ( X ) is a G,-subset of 2,we can find open sets G, 3 G, 3 . . . such that q ( X ) = (-),"=, G,. It is clear from (31) that

~ 2 c) is But for any closed set F and real number c, the set { p P(X)lp(F) 2 Gn)< llk) is the comclosed by Proposition 7.21(d), and ( p ~ P ( 2 ) l p ( plement of such a set. Q.E.D. We turn now to characterizing the a-algebra B,(,, when X is metrizable and separable. From Lemma 7.6, we have that the mapping Qf:P(X)-+ R given by

is continuous for every f E C(X). One can easily verify from Proposition 7.21 that the mapping 8,:P(X) -+ [O, 11 defined by'

is Borel-measurable when B is a closed subset of X. (Indeed, in the final stage of the proof of Proposition 7.23, we used the fact that when B is closed the upper level sets { p P(X)IB,(p) ~ 2 c} are closed.) Likewise, when B is open, 8, is Borel-measurable. It is natural to ask if 8, is also Borel-measurable when B is an arbitrary Bore1 set. The answer to this is yes, and in fact, B,(,, is the smallest a-algebra with respect to which 8, is measurable for every B E B ~A. useful aid in proving this and several subsequent results is the concept of a Dynkin system. The use of the symbol 0 , here is a slight abuse of notation. In keeping with the definition of Of,the technically correct symbol would be Ox,#.

7.4

133


Definition 7.11 Let X be a set and 9 a class of subsets of X. We say 9 is a Dynkin system if the following conditions hold:

(a) X E ~ . (b) I f A , B ~ 9 a n dB c A, then A- BE^. (c) If A,,A,,. . . ~ and 9 A, c A, c..., then

U,"=,A , E ~ .

Proposition 7.24 (Dynkin system theorem) Let 9 be a class of subsets of a set X, and assume 9 is closed under finite intersections. If 9 is a Dynkin then 9 also contains o ( 9 ) . system containing 9,

Proof This is a standard result in measure theory. See, for example, Ash [Al, page 1691. Q.E.D. Proposition 7.25 Let X be a separable metrizable space and 8 a collection of subsets of X which generates B, and is closed under finite intersections. Then Bp(,, is the smallest o-algebra with respect to which all functions of the form

are measurable from P(X) to [0, 11, i.e.,

Proof Let 9 be the smallest o-algebra with respect to which 8, is measurable for every E € 6 . To show 9 c Bp(,,, we show that O, is ,@,(,,measurable for every BE.@,. Let 9 = (BE,%~,IQ,is .gPo,-measurable). It is easily verified that 9 is a Dynkin system. We have already seen that 9 contains every closed set, so the Dynkin system theorem (Proposition 7.24) implies 9 = 93,. It remains to show that B,,,, c 9 . Let 9' = {BEB,IO, is B-measurable) As before, 9'is a Dynkin system, and since & c 9', we have 9' = W,.Thus the function Qs(p)= J.f dp is 9-measurable when f is the indicator of a Bore1 set. Therefore Of is .F-measurable when f is a Borel-measurable simple function. Iff E C(X), then there is a sequence of simple functions f, which are uniformly bounded below such that f,t f . The monotone convergence theorem implies Of, t Of, SO Of is 9-measurable. It follows that for E > 0, p~ P(X), and f E C(X), the subbasic open set

is 9-measurable. It follows that Bp(,, = .F (see the remark following Definition 7.6). Q.E.D.

134 Corollary 7.25.1

7.


If X is a Borel space, then P(X) is a Borel space.

Proof Let cp be a homeomorphism mapping X onto a Borel subset of a topologically complete separable space Y. Then, by Lemma 7.10, P(X) is homeomorphic to the Borel set {pE P(Y) 1 p[cp(~)]= 1). Since P(Y) is topologically complete and separable (Proposition 7.23), the result folQ.E.D. lows. 7.4.3 Stochastic Kernels

We now consider probability measures on a separable metrizable space parameterized by the elements of another separable metrizable space. Definition 7.12 Let X and Ybe separable metrizable spaces. A stochastic kernel q(dy1x) on Y given X is a collection of probability measures in P(Y) parameterized by EX. If F is a a-algebra on X and y-'[&!,(,,] c F, where y :X + P(Y) is defined by

(32) then q(dyjx) is said to be .F-measurable. If y is continuous, q(dylx) is said to be continuous. Y (x) = 4(dy(x),

Proposition 7.26 Let X and Y be Borel spaces, d a collection of subsets of Y which generates 98, and is closed under finite intersections, and q(dy1x) a stochastic kernel on Y given X. Then q(dy1x) is Borel-measurable if and only if the mapping I,,: X -+ [O, 11 defined by

is Borel-measurable for every E E 6. Proof Let y : X -+ P(Y) be defined by y(x) = q(dy1x). Then for E E $ , we have I., = 8,. y. If q(dylx) is Borel-measurable (i.e., y is Borel-measurable), , is Borel-measurable for every E E 8.Conthen Proposition 7.25 implies i c versely, if ,?, is Borel-measurable for every E € b , then a[i,JE,, jLE 92,. Proposition 7.25 implies

so q(dylx) is Borel-measurable.

Q.E.D.

Corollary 7.26.1 Let X and Y be Borel spaces and q(dylx) a Borelmeasurable stochastic kernel on Y given X. If BE.@^,, then the mapping

7.4


A, :X + [O,1] defined by

where B,

=

[ y E Y l(x, y) E B), is Borel-measurable.

Proof If B EBxy and x~ X, then B, c Y is homeomorphic to B n [jx) Y] E B ~ It follows ~ . that B,EB,, so q(B,lx) is defined. It is easy to show that the collection 3 = [BE.%,,IA, is Borel-measurable) is a Dynkin system. Proposition 7.26 implies that 9 contains the measurable rectangles, so 3 = Bx,. Q.E.D.

We now show that one can decompose a probability measure on a product of Borel spaces into a marginal and a Borel-measurable stochastic kernel. This decomposition is possible even when a measurable dependence on a parameter is admitted, and, as we shall see in Chapter 10, this result is essential to the filtering algorithm for imperfect state information dynamic programming models. As a notational convenience, we use _X to denote a typical Borel subset of a Borel space X. Proposition 7.27 Let (X, F) be a measurable space, let Y and Z be Borel spaces, and let q(d(y,z)lx) be a stochastic kernel on YZ given X. Assume that q(B1x) is 9-measurable in x for every BE$,,. Then there exists a stochastic kernel r(dzlx, y) on Z given XY and a stochastic kernel s(dy1x) on Y given X such that r(Zlx, y) is 9.%,-measurable in (x, y ) for every _2 E B,, , s(Z/x)is 9-measurable in x for every Z E B ~and

Proof We prove this proposition under the assumption that Y and Z are uncountable. If either Y or Z or both are countable, slight modifications (actually simplifications) of this proof are necessary. From Corollary 7.16.1, we may assume without loss of generality that Y = Z = (0,1]. Let s(dy1x) be the marginal of q(d(y, z)(x)on Y, i.e., s(Y(x)= q(_YZlx)for every YEW,. For each positive integer n, define subsets of Y

M(j, n) = ( ( j- 1)/2",,j/2"],

j = I,. . . ,2".

Then each M(j,n + 1) is a subset of some M(k,n), and the collection jM(j, n)ln = 1,2,. . . ; j = 1,. . . ,2") generates 8,. For Z E Q n Z, define q(dy(0,z] Ix) to be the measure on Y whose value at Y E By is q(_Y(O,z] lx). Then q(dy(0,z]lx) is absolutely continuous with respect to s(dy1x) for every

136

7.


Z E Q n Z and EX. Define for z e Q n Z

Gn(zlx,Y ) =

q [ M ( j ,n)(O,zl Ixlls[M(j,n)lxl if y E M ( j , n) and s [ M ( j ,n)lx] > 0, 0 if Y E M ( j , n) and s [ M ( j ,n)lx] = 0.

The functions Gn(zlx,y) can be regarded as generalized difference quotients of q(dy(0,z]Ix) relative to s(dy1x). For each z, the set ( x ,y ) E X Yl lim G,(zlx, y) exists in R n+ m

is .FBy-measurable. Theorem 2.5, page 612 of Doob [D4] states that

and if we define lim Gn(zlx,y)

if ( x ,Y ) E B(z),

G(zlx,Y ) = otherwise, then

It is clear that for any z, G(zlx,y) is 9-8,-measurable in ( x ,y).t A comparison of (34) and (35) suggests that we should try to extend G(zlx,y) in such a way that for fixed ( x ,y), G(zlx,y) is a distribution function. For the reader familiar with martingales, we give the proof of the theorem just referenced. Fix x and J] and observe that for rn 2 n.

Since {M(j,n)lj = 1,. . . ,2") is the 0-algebra generated by G,(ZX,y ) regarded as a function of y, we conclude that G,(+ y), n = 1,2, . . . is a martingale on Y under the measure s(~/ylx). Each G,(slx,y) is bounded above by 1, so by the martingale convergence theorem (see, e.g., Ash [Al, p. 2921) G,(s/x, J ) converges for s(dylx) almost every y. Thus s[B(z),ls] = 1 and the definition of G(zlx, y) given above is possible. Let m + x in (*) to see that (35) holds whenever 1 = M( j. n) for some j and n. The collection of sets Y for which (35) holds is a Dynkin system, and it follows from Proposition 7.24 that (35) holds for every Y E B , .

7.4


137

Toward this end, for each z, E Q n Z , we define C(zo)= { ( x ,y) ~ x Y 1 3 - Q 7 ~n Z with z I zo and G(zlx,y) > G(zolx,y ) ) ,

D(zo)= { ( x ,y) E X Y i G ( . lx, y) is not right-continuous at z o )

E

=

{ ( x ,Y )E X Y ~ G ( ~y)~does X , not converge to zero as z 10)

and

For fixed X E X and z, then

E

Q n Z , (35) implies that whenever Z E Q n Z , z I z,,

Therefore G(zlx,y) I G(zolx,y) for s(dylx)almost all y, so s[C(zo),lx] = 0 and

Equation (36) implies that G(zlx,y) is nondecreasing in z for s(dylx) almost all y. This fact and (35)imply that if z J zo ( Z E Q n Z ) , then

and

fbr s(dy1x) almost all y. Therefore s[~(z,),lx] = 0 and

7.

138


Equation (35) also implies that as z J 0 (z E Q n Z )

Since G(zlx,y) is nondecreasing in z for s(dylx) almost all y, we must have G(zlx,y) i0 for s(dy1x) almost all y, i.e.,

Substituting z = 1 in (35), we see that

so G(ljx, y) = 1 for s(dylx) almost all y, i.e.,

s(F,lx)

= 0.

For z E Z, let {z,) be a sequence in Q n Z such that z, 1zand define, for every EX, Y, lim G(z,lx, y)

if (x, y ) t X Y - (C u D u E u F),

F(zlx, y) =

(40) otherwise.

For (x, y) E XY - (C u D u E u F), G(zlx, y) is a nondecreasing rightcontinuous function of z E Q n Z, SO F(zlx, y) is well defined, nondecreasing, and right-continuous. It also satisfies for every (x, y) E X Y,

and lim F(zlx, y) = 0. z i o

It is a standard result of probability theory (Ash [Al, p. 241) that for each (x, y) there is a probability measure r(dzlx, y) on Z such that

~ which r(Zlx, y) is Fc3-A,-measurable in The collection of subsets Z E B for (x, y) forms a Dynkin system which contains {(O,z]jzt Z } , so r(Zlx, y) is S.3,-measurable for every _Z E BZ Relations (35)-(40) and the monotone convergence theorem imply

7.4


139

The collection of subsets ZE% . , for which (34)holds forms a Dynkin system which contains {(O, z] lz E Z } , so (34)holds for every _ZE BZ. Q.E.D. If B = B,,an application of Proposition 7.26 reduces Proposition 7.27 to the following form. Corollary 7.27.1 Let X , Y , and Z be Borel spaces and let q(d(y,z)lx) be a Borel-measurable stochastic kernel on Y Z given X. Then there exist Borel-measurable stochastic kernels r(dzlx,y) and s(dy1x)on Z given X Y and on Y given X,respectively, such that (34)holds. If there is no dependence on the parameter x in Corollary 7.27.1, we have the following well-known result for Borel spaces. Corollary 7.27.2 Let Y and Z be Borel spaces and q E P ( Y Z ) .Then there exists a Borel-measurable stochastic kernel r(dz1y) on Z given Y such that

where s is the marginal of q on Y

7.4.4 Integration As in Section 2.1, we adopt the convention

With this convention, for a, b,

CE

R* the associative law

still holds, since if either a, b, or c is co,then both sides of (42)are co,while if neither a, b, nor c is co,the usual arithmetic involving finite numbers and - co applies. Also, if a, b, c~ R* and a + b = c, then a = c - b, provided b # f m. It is always true however that if a b I c, then a I c - b. We use convention (42) to extend the definition of the integral. If X is a metrizable space, p E P ( X ) , and f :X + R* is Borel-measurable, we define

+

Note that if 1f dp < co or if Sf - dp < co, (43) reduces to the classical definition of f dp. We collect some of the properties of integration in this extended sense in the following lemma. +

S

Lemma 7.11 Let X be a metrizable space and let p~ P ( X ) be given. Let ,f; g and f ; , n = 1.2,. . . , be Borel-measurable, extended real-valued functions on X.

140

7.


(a) Using (42) to define f J(f

+ g, we have

+9 ) d< ~ Jf

dP + Jg dP.

(b) If either (bl) j f dp < cc and jg+dp < co,or (b2) - dp < co and Jg- dp < co,or (b3) jg+ dp < co and jgdp < cc,then +

Sf

J(f + 9)dP = Jf d~ + Jg (c) (d) (e) (f )

dP.

I f 0 < a < co,thenj(af)dp = ajfdp. Iff I g, then f dp < jg dp. If f, t f and j fi dp > - co,then dp t dp. Iff, I f and dp < a, then dp I j f dp.

1

Sfs

Sf,

Sf,

Sf

Proof We prove (b) first and then return to (a). Under assumption (bl), we have f(x) < cc and g(x) < co for p almost every x, so the sum f(x) g(x) can be defined without resort to the convention (42) for p almost every x. Furthermore, j f dp < co and jgdp < co, so (45) follows from the additivity theorem for classical integration theory (Ash [Al, p. 451). The proof of (45) under assumption (b2) is similar. Under assumption (b3), either J f dp = co, in which case both sides of (45) are co, or else j f dp < co, in which case assumption (bl) holds. Returning to (a), we note that if assumption (bl) holds, then (45) implies (44). If assumption (bl) fails to hold, then

+

+

+

so (44) is still valid. Statements (c) and (d) are simple consequences of (42) and (43). Statement (e) follows from the extended monotone convergence theorem (Ash [Al, p. 471) if j f ; dp < co.If J f ; dp = co,then j f, dp > - cc implies dp = Jf , dp = cc,and the conclusion follows from (d). Statement (f) follows from the extended monotone convergence theorem. Q.E.D.

Sf:

We saw in Corollary 7.27.2 that a probability measure on a product of Borel spaces can be decomposed into a stochastic kernel and a marginal. This process can be reversed, that is, given a probability measure and one or more Borel-measurable stochastic kernels on Borel spaces, a unique probability measure on the product space can be constructed. Proposition 7.28 Let X I , X,, . . . be a sequence of Borel spaces, Y, = X s X 2 . . . X , a n d Y = X , X , . . . . L e t p ~ P ( X ~ ) b e g i v e n , a n d , f o=r n1,2,..., let q,(dx,+ ( y,) be a Borel-measurable stochastic kernel on X,, given 5 .

,

,

7.4


141

Then for n = 2,3,.. . , there exist unique probability measures rn E P(Y,) such that

rn(xl_x2' x n ) = Jx- I x x

J

Xz ...JX n -

1

9n-l(~nx1,x2~...~xn-1)

qn-2(d~n-lIx1,~2,. . . ,xn-2).. . ql(dx2lxl)p(dxl) V X l €Bxl,. . . ~ X ~ E B X(46) ,.

1

I f f : Y, -t R* is Borel-measurable and either f + cir, < ac, or J f - dr, < co, then

.. Furthermore, there exists a unique probability measure r on Y = XlX2. such that for each n the marginal of r on Y,is r,.

Proof The spaces Y,,n = 2,3,.. . , and Yare Bore1 by Proposition 7.13. If there exists rn E P(Y,) satisfying (46),it must be unique. To see this, suppose r; E P(Y,) also satisfies (46).The collection 9 = jB €By,lrn(B)= YL(B)) is a and rn = Dynkin system containing the measurable rectangles, so 9 = Byn rb. We establish the existence of rn by induction, considering first the case n = 2.For BEB,,, use Corollary 7.26.1to define

It is easily verified that r 2 € P(Y2)and r2 satisfies (46).Iff is the indicator of B € B y 2the , I,, f(xl,x2)ql(dx21xl) is Borel-measurable and, by (48),

Linearity of the integral implies that (49)holds for Borel-measurable simple functions as well. I f f : Y2-t [0,co] is Borel-measurable, then there exists an increasing sequence of simple functions such that f,t f.By the monotone convergence theorem,

is Borel-measurable and so J x 2 f(xl,x2)ql(dx,lxl) tir2 = n-t lirnm Jx, J x 2 ~ ( x~1 ~> ) q ~ ( d x ~ l x ~ ) ~ ( d x ~ ) lim Jy2,fn

n-

30

142

7.


But !,,f, dr, t jy2 f dr,, so (49)holds for any Borel-measurable nonnegative f . For a Borel-measurable f : Y2+ R* satisfying f f dr, < ar, or j f - dun < oo, we have +

J y 2 f + dr2

= Jxl

L,f +(x1,x2)q~(dx,lx,)~(dx~),

and

L,f-

dr2 =

jxlJ x 2 f - ( x 1 ,~ 2 ) q l ( d x 2 ~ ~ l ~ ~ ( d ~ l ) .

Assume for specificity that jy2 f

-

dr2 < oo.Then the functions

and

satisfy condition (b2) of Lemma 7.11, so

where the last step is a direct result of the definition of jx, f ( x , , x2)q,(dx,lxl). Assume now that r k € P ( Y , )exists for which (46) and (47) hold when n = k. For B E Y,+,, let

Then rk+,E P(Y,+,). If B

= _ X 1 X 2 . ._XkX,+,, where

+

X j ~ . 9 3 x jthen ,

by (47) when n = k. This proves (46) for n = k 1. Now use (50) to prove (47) when n = k + 1 and f is an indicator function. As before, extend this to the case of f : Y,, + [0, oo]. If f : Y,+ + R* is Borel-measurable and either J f dr,+ < co or f - dr,, < oo, then the validity of (47) for non+

,

,

1

,

,

7.4


negative functions and the induction hypothesis imply

qi(dxzlxi)~(dxi) =~ x l , , . x k ~ x k + l f ~ ( x l ~ ~ ~ ~ ~ x k + l ) ~ k ( d x k + l ~ x l ~ x 2 ~ ~ ~ ~ ~ x k

and likewise Jyk+

- drk+l

= Jx,,,,xkfxk+

Assume for specificity that

- ( X I > .. . ? x k + l ) q k ( d x k + l l x l ~ x 2 ? .

I,,+, f

-

. . ,xk)drk.

drk+, < co.Then the functions

and

satisfy condition (b2) of Lemma 7.11, so as before

Since

we can apply the induction hypothesis to the right-hand side of (51) to conclude that (47)holds in the generality stated in the proposition. To establish the existence of a unique probability measure r E P(Y)whose marginal on Y,is r,,n = 2,3,. . . ,we note that the measures r, are consistent, i.e., if m 2 n, then the marginal of r, on Y, is r,. If each X k is complete, the Kolmogorov extension theorem (see, e.g., Ash [Al,p. 1911)guarantees the existence of a unique r~ P(Y)whose marginal on each Y, is r,. If X k is not complete, it can be homeomorphically embedded as a Borel subset in a complete separable metric space 2,. As in Proposition 7.13,each Y,, is homeomorphic to a Borel subset of the complete separable metric space = .flz2. . and Y is homeomorphic to a Borel subset of the complete separable metric space = . . . Each r , P(Y,) ~ can be identified with

.z,

zl.f2.

144

7.


P,,EP(K) in the manner of Lemma 7.10, and, invoking the Kolmogorov extension theorem, we establish the existence of a unique PEP(Y) whose marginal on each is P,,. It is straightforward to show that P assigns probability one to the image of Y in 7,so 7 corresponds to some Y E P(Y) whose marginal on each Y,, is v,. The uniqueness of P implies the uniqueness of v. Q.E.D. In the course of proving Proposition 7.28, we have also proved the following. Proposition 7.29 Let X and Y be Borel spaces and q(dylx) a Borelmeasurable stochastic kernel on Y given X. I f f : X Y -t R* is Borel-measurable, then the function il:X -+ R* defined by

Corollary 7.29.1 Let X be a Borel space and let f :X measurable. Then the function 8f :P(X) R* defined by

+ R*

be Borel-

-+

Proof Define a Borel-measurable stochastic kernel on X given P(X) by q(dx1p) = p(dx). ~ e f i n e f ? : ~-+( R* ~ )by~ f?(p,X) = f(x). Then

is Borel-measurable by Proposition 7.29.

Q.E.D.

I f f E C(XY) and q(dylx) is continuous, then the mapping I. of (52) is also continuous. We prove this with the aid of the following lemma. Lemma 7.12 Let X and Y be separable metrizable spaces. Then the mapping o:P(X)P(Y)+ P(XY) defined by

where pq is the product of the measures p and q, is continuous. Pvoof' We use Urysohn's theorem (Proposition 7.2) to homeomorphically embed X and Y into the Hilbert cube 2 , and, for simplicity of notation, we treat X and Y as subsets of 2.Let d be a metric on 2%consistent with its topology. If g~ U , ( X Y ) , then Lemma 7.3 implies that g can be extended to a function @ E C ( A f 2 ) . The set of finite linear combinations of the form Cjk,lf3.(~)hj(y), where and hj range over C ( 2 ) and k ranges over the

f3.

7.5

BOREL-MEASURABLE SELECTION

145

positive integers, is an algebra which separates points in .XX, so given > 0, the Stone-Weierstrass theorem implies that such a linear combination x h j - gll < E. If (pn) is a sequence in P(X) can be found satisfying converging to p E P(X), (9,) a sequence in P(Y) converging to q E P(Y), and f,and hj the restrictions of and hj to X and Y, respectively, then E

lxT=

The continuity of o follows from the equivalence of (a) and (c) of Proposition 7.21. Q.E.D. Proposition 7.30 Let X and Y be separable metrizable spaces and let q(dy1x) be a continuous stochastic kernel on Y given X. Iff E C(XY), then the function 1,:X -+ R defined by

is continuous. Proof The mapping v:X + P(XY) defined by v(x) = p,q(dy)x) is continuous by Corollary 7.21.1 and Lemma 7.12. We have iL(x)= (Of. v)(x), where Of:P(XY) + R is defined by Of(r) = Sf dr. By Proposition 7.21, Of is Q.E.D. continuous. Hence, 1, is continuous. 7.5 Semicontinuous Functions and BorelMeasurable Selection In the dynamic programming algorithm given by (17) and (18) of Chapter 1, three operations are performed repetitively. First, there is the evaluation of a conditional expectation. Second, an extended real-valued function in two variables (state and control) is infimized over one of these variables (control). Finally, if an optimal or nearly optimal policy is to be constructed, a "selector" which maps each state to a control which achieves

146

7.


or nearly achieves the infimum in the second step must be chosen. In this section, we give results which will enable us to show that, under certain conditions, the extended real-valued functions involved are semicontinuous and the selectors can be chosen to be Borel-measurable. The results are applied to dynamic programming in Propositions 8.6-8.7 and Corollaries 9.17.2-9.17.3. Definition 7.13 Let X be a metrizable space and f an extended realvalued function on X. If { x E XI f ( x ) < c ) is closed for every c E R, f is said to be lower semicontinuous. If { xE XI f ( x )2 c ) is closed for every C E R, f is said to be upper semicontinuous.

Note that f is lower semicontinuous if and only if -f is upper semicontinuous. We will use this duality in the proofs of the following propositions to assert facts about upper semicontinuous functions given analogous facts about lower semicontinuous functions. Note also that i f f is lower semicontinuous, the sets { X E XI f ( x ) = - co) and { x ~f ( x ) < co) are closed, since the former is equal to ,(x E XI f ( x )< - n) and the latter is X. There is a similar result for upper semicontinuous functions. The following lemma provides an alternative characterization of lower and upper semicontinuous functions.

XI

Lemma 7.13 Let X be a metrizable space and f : X -+ R*.

(a) The function f is lower semicontinuous if and only if for each sequence jx,] c X converging to X E X lim inf f ( x , ) 2 .f(x). n+ m

(b) The function f is upper semicontinuous if and only if for each sequence (x,) c X converging to x E X lirn sup f (x,) < f (x). DO',

Proof Suppose f is lower semicontinuous and x, a subsequence {x,,) such that x,, -+ x as k -+ co and

lim f (x,,) k+ m

= lim inf f

-+

x. We can extract

(x,).

n+m

Given E > 0, define lim inf f (x,) + E

if lim inf f (x,) > - a, ,-.a

otherwise.

7.5

147


There exists a positive integer k ( ~ such ) that f(x,,) 5 8(&)for all k 2 k(c). The set (YEXIf(y) < $(E))is closed, and hence it contains x. Inequality (53) follows. Conversely, if (53) holds and for some C E R, (x,) is a sequence in ( y E XI f(y) < c) converging to x, then f(x) I c, so f is lower semicontinuous. Part (b) of the proposition follows from part (a) by the duality mentioned Q.E.D. earlier. I f f and g are lower semicontinuous and bounded below on X and if x,

-+ X, then

lim inf [f (x,) n-rm

+ g(x,)] 2 lim inf f (x,) + lim infg(x,) n- m m n+

so f + g is lower semicontinuous. I f f is lower semicontinuous and cc > 0, then uf is lower semicontinuous as well. Upper semicontinuous functions have similar properties. It is clear from (53) and (54) that f : X -+ R* is continuous if and only if it is both lower and upper semicontinuous. We can often infer properties of semicontinuous functions from properties of continuous functions by means of the next lemma. Lemma 7.14 Let X be a metrizable space and f : X -+ R*. (a) The function f is lower semicontinuous and bounded below if and only if there exists a sequence {f,) c C ( X )such that f , t f . (b) The function f is upper semicontinuous and bounded above if and only if there exists a sequence (f,) c C(X) such that f , If. Proof We prove only part (a) of the proposition and appeal to duality for part (b). Assume f is lower semicontinuous and bounded below by b E R, and let d be a metric on X consistent with its topology. We may assume without loss of generality that for some x, E X , f(xo) < co, since the result is trivial otherwise. (Take f,(x) = n for every x E X.) As in Lemma 7.7, define

Exactly as in the proof of Lemma 7.7. we show that jg,) is an increasing sequence of continuous functions bounded below by b and above by f . The characterization (53) of lower semicontinuous functions can be used in place of continuity to prove g, T f'. In particular, (28) becomes ,f(x) I lirn inf f(y,) I lirn g,(x) n+ m

+ E.

n- m

Now define f" = min jn,g,j.. Then each f , is continuous and bounded and jl, t f . This concludes the proof of the direct part of the proposition. For

7.

148


the converse part, suppose { f , )c C(X) and f , t f . For

is closed.

CE

R,

Q.E.D.

The following proposition shows that the semicontinuity of a function of two variables is preserved when one of the variables is integrated out via a continuous stochastic kernel. Proposition 7.31 Let X and Y be separable metrizable spaces, let q(dy1x) be a continuous stochastic kernel on Y given X, and let f :XY + R* be Borel-measurable. Define

(a) Iff is lower semicontinuous and bounded below, then A is lower semicontinuous and bounded below. (b) Iff is upper semicontinuous and bounded above, then iis upper semicontinuous and bounded above. Proof We prove part (a) of the proposition and appeal to duality for part (b). Iff :XY -+ R* is lower semicontinuous and bounded below, then by Lemma 7.14 there exists a sequence (f,) c C(XY) such that f, T f . Define /Zn(x)= Jf,(x, y)q(dylx). By Proposition 7.30, we have that A, is continuous, and by the monotone convergence theorem in? A. By Lemma 7.14, iis lower semicontinuous. Q.E.D. An important operation in the execution of the dynamic programming algorithm is the infimization over one of the variables of a bivariate function. In the context of semicontinuity, we have the following result related to this operation. Proposition 7.32 Let X and Y be metrizable spaces and let f : X Y + R* be given. Define

f *(x) = inf f (x, y). YGY

(a) I f f is lower semicontinuous and Y is compact, then f * is lower semicontinuous and for every x E X the infimum in (55) is attained by some yE Y. (b) Iff is upper semicontinuous, then f * is upper semicontinuous. Proof (a) Fix x and let { y,} c Y be such that f (x, y,) L f *(x). Then (y,) accumulates at some y , Y, ~ and part (a) of Lemma 7.13 implies that f(x, yo) = f *(x). To show that f * is lower semicontinuous, let (x,) c X be

7.5


such that x,

+ x,.

Choose a sequence { y,) c Y such that

There is a subsequence of {(x,, y,)), call it ((x,,,~,~)), such that lim inf,, , f (x,, y,) = lim,, , f (x,, ,y,,). The sequence { y,,) accumulates at some yo E Y , and, by Lemma 7.13(a), lim inf f *(x,) = lim inf f (x,, y,) n-r

w

n-r

=

w

lim f (x,,, y,,) 2 f (x, ,yo) 2 f *(xo), k+ m

* is lower semicontinuous. (b) Let dl be a metric on X and d, a metric on Y consistent with their topologies. If G c XY is open and x, E proj,(G), then there is some yo E Y for which (x,, yo)E G, and there is some r > 0 such that so f

N,(xo, yo) = {(x,Y)EXYldl(x, xo) < E> dz(y, yo) < 8)

G.

Then

so proj,(G) is open in X. For

CE

R,

The upper semicontinuity of f implies that {(x,y)l f(x, y) < c) is open, so Q.E.D. {XEXI f "(x) < c) is open and f * is upper semicontinuous. Another important operation in the dynamic programming algorithm is the choice of a measurable "selector" which assigns to each x E X a y E Y which attains or nearly attains the infimum in (55). We first discuss Borelmeasurable selection in case (a) of Proposition 7.32. For this we will need the Hausdorff metric and the corresponding topology on the set 2' of closed subsets of a compact metric space Y (Appendix C). The space 2' under this topology is compact (Proposition C.2) and, therefore, complete and separable. Several preliminary lemmas are required. Lemma 7.15 Let Y be a compact metrizable space and let g: Y + R* be lower semicontinuous. Define g* :2' + R* by

Then g" is lower semicontinuous. Proof Since the empty set is an isolated point in 2', we need only prove that g* is lower semicontinuous on 2' - (@). We have already shown [Proposition 7.32(a)] that, given a nonempty set A E ~ ' ,there exists Y E A

7.

150


such that g*(A) = g(y). Let {A,) c 2' be a sequence of nonempty sets with ~ be such that g*(A,) = g(y,), n = 1,2,. . . . Choose limit A E ~ ' ,and let y , A, a subsequence { y,,) such that lim g(y,,)

= liminfg(y,) = liminfg*(A,).

k- m

n+m

n+ a,

The subsequence (y,,) accumulates at some yo E Y, and, by Lemma 7.13(a), g(yo) i lim g(y,,)

= liminfg*(A,).

k+ m

n+ m

From (14) of Appendix C and from Proposition C.3, we have (in the notation of Appendix C) -

yo E lim A,

= A,

n-t m

The result follows from Lemma 7.13(a).

Q.E.D.

Lemma 7.16 Let Y be a compact metrizable space and let g: Y -+ R* be lower semicontinuous. Define G:2'R* -+ 2' by G(A, c) = A n { Y EYlg(y) i c).

(57)

Then G is Borel-measurable. Proof We show that G is upper semicontinuous (K) (Definition C.2) and apply Proposition C.4. Let {(A,, c,)) c 2 ' ~ " be a sequence with limit then (A, c). If limn,, G(A,, c,) =

a,

lim G(A,, c,) c G(A, c). n-+m

Otherwise, choose y~lim,,, G(A,, c,). There is a sequence n, < n, < . . . of positive integers and a sequence y , , ~G(A,,, c,,),k= 1,2,. . . , such that ynk-+ y. By definition, y , , ' ~A,, for every k, so y~lim,,, An = A. We also have g(ynk)i c,,, k = 1,2,. . . , and using the lower semicontinuity of g, we obtain g(y ) i lim inf g( y,,) i lim c,, k* m

= c.

k- n

Therefore y E G(A, c), (58) holds, and G is upper semicontinuous (K).

Q.E.D.

7.5

151


Lemma 7.17 Let Y be a compact metrizable space and let g : Y -+ R* be lower semicontinuous. Let g*:2'+ R* be defined by (56) and define G*:2' -+ 2' by G*(A) = A n ( Y E Y l g ( y ) g*(A)).

(59)

Then G* is Borel-measurable. Proof Let G be the Borel-measurable function given by (57).Lemma 7.15 implies g* is Borel-measurable. A comparison of (57)and (59) shows that

It follows that G* is also Borel-measurable.

Q.E.D.

Lemma 7.18 Let Y be a compact metrizable space. There is a Borelmeasurable function o:2' - (@) + Y such that ~ ( A )AE for every A E ~ '- (@). Proof Let {g,ln = 1,2,. . .) be a subset of C ( Y ) which separates points in Y (for example, the one constructed in the proof of Proposition 7.7). As in Lemma 7.15, define :2' + R* by

and, as in Lemma 7.17, define G;:2'

Let H,:2'

-+

-+ 2'

by

2' be defined recursively by

Then for A # @, each H,(A) is nonempty and compact, and

Therefore, have

n,"=,H,(A), then for n

H,(A) # @. If y, y ' ~

=

Since {g,ln = 1,2,. . .) separates points in Y, we have y = yr, and must consist of a single point, which we denote by o(A).

1,2,. . . , we

n,"=,H,(A)

152

7.


We show that for A # @ m

lim H,(A)

=

n-r m

n =0

H,(A)

=

{o(A)).

(60)

Since the sequence {H,(A)) is nonincreasing, we have from (14) and (15) of Appendix C that

-

If y ~ l i m , , , H,(A), then there exist positive integers n, < n, < . . . and a sequence y , , ~H,,(A), k = 1,2,. . . , such that y,, -+ y. For fixed k, y,, EH,, for all j 2 k, and since H,,(A) is closed, we have Y E H,,(A). Therefore, yE .,(A) and

n,?=

m

lim H,(A) n+ m

c

0 H,(A).

n=O

(62)

From relations (61)and (62),we obtain (60). Since GX and H , are Borel-measurable for every n, the mapping v:2' 1%) 2' defined by v ( A ) = { a ( A ) )is Borel-measurable. It is easily seen that the mapping t : Y -+ 2' defined by t ( y ) = { y ) is a homeomorphism. Since Y is compact, t ( Y ) is compact, thus closed in 2', and t - ' : t ( Y ) -+ Y is Borel-measurable. Since o = t-' ov, it follows that o is Borel-measurable. Q.E.D. -+

Lemma 7.19 Let X be a metrizable space, Y a compact metrizable space, and let f : X Y -+ R* be lower semicontinuous. Define F : X R * -+ 2' by

Then F is Borel-measurable. Proof The proof is very similar to that of Lemma 7.16. We show that F is upper semicontinuous (K) and apply Proposition C.4. Let (x,, c,) -+ ( x ,c) in X R * and let y be an element of En,,F(x,, c,), provided this set is nonempty. There exist positive integers n, < n, < . . . and y , , ~F(xnk,en,) such that y,, -+ y. Since f(x,,, y,,) I c,, and f is lower semicontinuous, we conclude that f ( x ,y) 5 c, so that limn,, F(x,, c,) c F(x, c). The result follows. Q.E.D.

Lemma 7.20 Let X be a metrizable space, Y a compact metrizable space, and let f :X Y -+ R* be lower semicontinuous. Let f * :X -+ R* be given by f * ( x ) = min,,, f ( x , y ) , and define F * : X + 2' by

Then F* is Borel-measurable.

7.5

153


Proof Let F be the Borel-measurable function defined by (63). Proposition 7.32(a)implies that f * is Borel-measurable. From (63)and (64)we have F*(x) = F[x, f *(x)]. It follows that F* is also Borel-measurable.

Q.E.D.

We are now ready to prove the selection theorem for lower semicontinuous functions. Proposition 7.33 Let X be a metrizable space, Y a compact metrizable space, D a closed subset of X Y, and let f : D -+ R* be lower semicontinuous. Let f * :proj,(D) -+ R* be given by

f *(x) = min f(x, y).

(65)

YE&

Then proj,(D) is closed in X, f * is lower semicontinuous, and there exists a Borel-measurable function cp:projx(D)-+ Y such that Gr(cp) c D and

f

[x, 4o(x)l = f *(XI

Vx E projx(D).

(66)

Proof We first prove the result for the case where D = XY. As in Lemma 7.18, let o:2' - ((21) -t Y be a Borel-measurable function satisfying ~ ( A )AE ' ((21). As in Lemma 7.20, let F*:X -+ 2' be the Borelfor every A E ~ measurable function defined by Proposition 7.32(a) implies that f * is lower semicontinuous and F*(x) f (21 for every x E X. The composition cp = o o F* satisfies (66). Suppose now that D is not necessarily XY. To see that projx(D) is closed, note that the function g = - X, is lower semicontinuous and

,

where g*(x) = min,, g(x, y). By the special case of the proposition already proved, g* is lower semicontinuous, projx(D) is closed, and there is a Borelmeasurable function q , : X -+ Y such that g[x,q,(x)] = g*(x) for every x E X or, equivalently, (x, cpl(x))E D for every x E projx(D). Define now the lower semicontinuous function ~ : X -+ Y Ji* by

y(&y) = For all c E R,

Y)

if (x, Y)ED, otherwise.

154

7.


Since min,,,J?(x, g) is lower semicontinuous, it follows that f * is also lower semicontinuous. Let cp2:X -+ Y be a Borel-measurable function satisfying

Clearly (x, cp,(x)) E D for all x in the Bore1 set

Define cp :projx(D) + Y by (el(x)

if min Ax,y ) = m, vaY

The function qn is Borel-measurable and satisfies (66).

Q.E.D.

We turn our attention to selection in the case of an upper semicontinuous function. The analysis is considerably simpler, but in contrast to the "exact selector" of (66) we will obtain only an approximate selector for this case. Lemma 7.21 Let X be a metrizable space, Y a separable metrizable space, and G an open subset of XY. Then projx(G) is open and there exists a Borel-measurable function cp:proj,(G) + Y such that Gr(cp) c G. Proof Let { g,ln = 1,2,. . .) be a countable dense subset of Y. For fixed Y, the mapping x -,(x, y) is continuous, so {x E X/(x,y) E G) is open. Let G, = {x E XI(x, y,) E G), and note that projx(G) = G, is open. Define cp :projx(G) -+ Y by

YE

U,"=,

Then cp is Borel-measurable and Gr(cp) c G.

Q.E.D.

Proposition 7.34 Let X be a metrizable space, Y a separable metrizable space, D an open subset of X Y , and let f : D -+ R* be upper semicontinuous. Let f * :proj,(D) + R* be given by

f *(x) = inf f(x,y).

(67)

psDz

Then projx(D) is open in X, f * is upper semicontinuous, and for every F: > 0, there exists a Borel-measurable function cp,:projx(D) + Y such that

7.5

155


Gr(cp,) c D and for all x E projx(D)

Proof The set projx(D) is open in X by Lemma 7.21. To show that f * is upper semicontinuous, define an upper semicontinuous function x Y + R* by

7:

fi,

=

Y)

if (x, Y)E D, otherwise.

For c E R, we have

and this set is open by Proposition 7.32(b). Let E > 0 be given. For k = 0, 11, i 2 , . . . , define (see Fig. 7.1) A(k) = {(x,Y) E Dlf(x, Y) < k&), B(k) = j x ~ p r o j , ( ~ ) l ( k l ) I~f *(x) < kc), B( - co) = jx E projx(D)lf *(x) = - co), B(w) = .jx E projx(D)lf *(x) = co). The sets A(k), k = 0, i l , 12,. . . , are open, while the sets B(k), k = 0, 1 1 , 2 2 , . . . ,B(- a), and B(co) are Borel-measurable. By Lemma 7.21,

FIGURE 7.1

7.

156


there exists for each k = 0, If: 1, f2,. . . a Borel-measurable cp,: projx(A,) -+ Y such that Gr(cp,) c A,, and there exists a Borel-measurable q :projx(D) + Y such that Gr(q) c D. Let k* be an integer such that k* 5 Define cpE:projx(D) Y by +

cp,(x) @(x) cp,,(x)

if x E B(k), k = 0, & 1, &2,. . . , if X E B(co), if X E B(- co).

Since B(k) c projx[A(k)] and B(- co) c proj,[A(k)] for all k, this definition is possible. It is clear that cp, is Borel-measurable and Gr(cp,) c D. If x E B(k), then, since (x, cp,(x))E A(k), we have

f [x, ~ , ( x ) = ] f [x, qk(~)]< k& 5 f *(XI+ &. If x E B(co), then f(x, y) = co for all y E D, and f [x, cp,(x)] x E B(- a), we have

f [x, cp,(x)] = f [x,cp,*(x)l < k*&5 Hence cp, has the required properties.

=

co = f "(x). If

- I/&.

Q.E.D.

7.6 Analytic Sets

The dynamic programming algorithm is centered around infimization of functions, and this is intimately connected with projections of sets. More specifically, iff: X Y + R* is given and f * :X + R* is defined by

then for each c E R

If f is a Borel-measurable function, then ((x,y)lf(x, y) < c ) is a Borelmeasurable set. Unfortunately, the projection of a Borel-measurable set need not be Borel-measurable. In Borel spaces, however, the projection of a Borel set is an analytic set. This section is devotcd to development of properties of analytic sets. 7.6.1 Equivalent Dejinitions of Analytic Sets

There are a number of ways to define the class of analytic sets in a Borel space X. One possibility is to define them as the projections on X of the Borel subsets of XY, where Y is some uncountable Borel space. Another

possibility is to define them as the images of the Baire null space Jlr under continuous functions from Jlr into X. Still another possibility is to define them as all sets of the form

where .A'" is the set of all sequences of positive integers (the Baire null space) and the sets S(ol, o, , . . . ,on)are closed in X. All these definitions are equivalent, as we show in Proposition 7.41. We will take the third definition as our starting point, since this is the most convenient analytically. We first formalize the set operation just given in terms of the notion of a Suslin scheme in a paved space. Definition 7.14 Let X be a set. A paving B of X is a nonempty collection of subsets of X. The pair (X, B) is called a paved space.

If (X, .P) is a paved space, we denote by o(B) the o-algebra generated by 9, we denote by 9, the collection of all intersections of countably many members of 9 , and we denote by 9, the collection of all unions of countably many members of 9 . Recall that N is the set of positive integers, Jlr is the set of all infinite sequences of positive integers, and C is the set of all finite sequences of positive integers. Definition 7.15 Let (X, 9)be a paved space. A Suslin scheme for 9 is a mapping from C into rY.The nucleus of a Suslin scheme S:C + B is

The set of all nuclei of Suslin schemes for a paving B will be denoted by Y(B). In order to simplify notation, we write, for s = (o,, a,, . . . ,on)€C and z = (il,12,. - .)E.M,

With this notation, (69) can also be written as

We will use both expressions interchangeably. Note that the union in (69) is uncountable, so if .Y is a o-algebra and S is a Suslin scheme for 9, N(S) may be outside .Y. Several properties of 9(.Y) are given below.

158

7.


Proposition 7.35 Let X be a space with pavings 9 and 2 such that

.9'c 2. Then

Proof (a) Obvious. Now ). choose (b) It is clear that .Y(.9'),3 9'(,9 N(S,)EY(,Y),. where S, is a Suslin scheme for 9, k = 1,2,. . . . It suffices to construct a Suslin scheme S for .Ysuch that

For k = 1, 2, . . . ,let II, = ((2j - 1)2,- ' I j = 1,2, . . .). Then II, , n 2 , . . . is a partition of N into infinitely many infinite sets. For each positive integer k, let cp, :A'"-t A'" be defined by

i.e., cp, picks out the components of (i,, i 2 , . . .)with indices in II,. We want to construct a Suslin scheme S for which

We may rewrite (71) as

Given (i,, C2, . . . ,in) E C, we have n positive integers j and k. Define

= (2j -

1)2k-' for exactly one pair of

This defines a Suslin scheme S for which (72),and hence (71), is easily verified. We now use (71) to prove (70). Choose

For some 2 , E .M, we have

Thus, for every k,

It follows that x E

n,"=,N(S,) and Uzp.Nn s < z S k ( for ~ ) every k, so 0,,,, Sk(s).Let z, E JV be such 02,ns,,k(zo, Sk(s).An application

If we are given X E (),"=, N(Sk), then X E for every k, there exists z, E JV such that x E that qo,(z,) = z,, k = 1,2,. . . . Then X E of (71)shows that

and

Relation (70)follows from (74)and (75). (c) It is clear that Y(,Y),I> Y(9).Choose N(S,) E Y ( Y ) , , where S, is a Suslin scheme for .Y, k = 1,2,. . . . It suffices to construct a Suslin scheme S for .Yfor which

UP=,

Given (i,, i , , . . . ,[,)EX, we have positive integers j and k. Define

i, = (2j - 1)2k-1 for exactly one pair of

This defines a Suslin scheme S for which

U z e Nns 0 is arbitrary, we have f (z,) = x,x E f ( M , ) , and N(R) c f ( M I ) . Q.E.D. We have thus characterized analytic sets as the continuous images of closed subsets of JV. We will obtain an even sharper characterization, for which we need the following lemma. Lemma 7.24 If ,MI is a nonempty closed subset of N , then there exists a continuous function g:.M -, N such that N , = ~ ( J V ) .

Proof Use the Lindelof property to cover M , with a countable collection of nonempty closed sets {S(i,)l[,E N )which satisfy

N,

2

S(il),

diam S ( i l )l1,

1, = 1,2,. . . ,

where d is a metric on JV consistent with its topology and diamS([,) is given by (9). Cover each S((',) with a countable collection of nonempty E N ) which satisfy closed sets is([,,

[, ) I [ ,

7.

164


Continue in this manner so that, for any

(i,,i 2 ,. . . , i n - ,),

50

S ( i l , i ~ , .,in-,) . =

U Sn=

S ( i l , i 2 , .. . , i n ) ,

S(i13C2,. . . ,Cn-l,in), in= 1,2,. . . . diam S ( i 1 ,i2,.. . ,in) 5 l/n, s(C12129..

.,in-l)

3

(83)

1 in

= 1,2,. . . ,

The completeness of .N and (82)-(85) imply that for each z~ M , consists of a single point. Define g(z) to be this point. Then

n,,

(84) (85) S(s)

The continuity of g follows by an argument similar to the one used in the Q.E.D. proof of Lemma 7.23. Proposition 7.37 Let X be a Borel space. A nonempty set A c X is analytic if and only if A = f ( M ) for some continuous function f : M --+ X .

Proof If X is complete, the proposition follows from Lemmas 7.23 and 7.24. If X is not complete, it is still homeomorphic to a Borel subset of a complete separable space, and the result follows from Corollary 7.36.1. Q.E.D. Proposition 7.37 gives a very useful characterization of nonempty analytic sets in terms of continuous functions and the Baire null space M . The Baire null space has a simple description and its topology allows considerable flexibility. We have already shown, for example, that it is homeomorphic to M , , the space of irrationals in [O,l]. Another important homeomorphism is the following. Lemma 7.25 The space M is homeomorphic to any finite or countably infinite product of copies of itself.

Proof We prove the lemma for the case of a countably infinite product. Let II,, IT2, . . . be a partition of N, the set of positive integers, into infinitely many infinite sets. Define cp: M + , N M N . . . by

where z, consists of the components of z with indices in II,. Then cp is oneto-one and onto and, because convergence in a product space is componentwise, cp is a homeomorphism. Q.E.D. Combination of Lemma 7.25 with Proposition 7.37 gives the following.

Proposition 7.38 Let X I , X, ,. . . be a sequence of Borel spaces and A, an analytic subset ofX,, k = 1,2,. . . .Then the sets AlA2. . . and AlA2. . .A,, n = 1,2,. . . , are analytic subsets of XIX,. . . and XlX2- . .X,, respectively.

Proof Let f,:N + X, be continuous such that A, = f,(N), k Let cp be given by (86) and F : N N . . . + X l X 2 . . . be given by

=

1,2,. . . .

F(z1, z22. . = ( f l ( ~ l ) > f 2 ( ~- 2 ) , . Then F 0 cp is continuous and maps .ilr onto A,A,. . . . The finite products are handled similarly. Q.E.D. Another consequence of Proposition 7.37 is that the continuous image of an analytic set, in particular, the projection of an analytic set, is analytic. As discussed at the beginning of this section, this property motivated our inquiry into analytic sets. We formalize this and a related fact to obtain another characterization of analytic sets. Proposition 7.39 Let X and Y be Borel spaces and A an analytic subset of XY. Then proj,(A) is analytic. Conversely, given any analytic set C c X and any uncountable Borel space Y, there is a Borel set B c XY such that C = proj,(B). If Y = N , B can be chosen to be closed.

Proof If A = f ( N ) c X Y is analytic, where f is continuous, then proj,(A) = (proj, 0 f ) ( N ) is analytic by Proposition 7.37. If C = f ( N ) c X is nonempty and analytic, then

c = projX[Gr(f 11, where Gr(f )= {( f (z),z) E X N I ZE N ) is closed because f is continuous. If Y is any uncountable Borel space, then there exists a Borel isomorphism cp from JV onto Y (Corollary 7.16.1). The mapping (I? defined by

is a Borel isomorphism from X N onto X Y, and C = projx(~[Gr(f)]).

Q.E.D.

So far we have treated only the continuous images of analytic sets. With the aid of Proposition 7.39, we can consider their images under Borelmeasurable functions as well. Proposition 7.40 Let X and Y be Borel spaces and f : X + Y a Borelmeasurable function. If A c X is analytic, then f(A) is analytic. If B c Y is analytic, then f -'(B) is analytic.

Proof Suppose A c X is analytic. By Proposition 7.39, there exists a Borel set B c X.M such that A = proj,(B). Define I): B + Y by $(x, z) = f (x).

166

7.


Then $ is Borel-measurable, and Corollary 7.14.1 implies that G ~ ( $ ) BXNY. E Finally, f(A) = proj,[Gr($)] is analytic by Proposition 7.39. If B c Y is analytic, then B = N(S), where S is some Suslin scheme for F , . Then f -'(B) = N(f 0 S), where f o S is the Suslin scheme for 33, defined by

-'

-'

The analyticity off -'(B) follows from Proposition 7.36.

Q.E.D.

We summarize the equivalent definitions of analytic sets in Borel spaces. Proposition 7.41 Let X be a Borel space. The following definitions of the collection of analytic subsets of X are equivalent:

(a) Y ( F x ) ; (b) Y(Bx); (c) the empty set and the images of A'" under continuous functions from A'" into X ; (d) the projections into X of the closed subsets of X M ; (e) the projections into X of the Borel subsets of XY, where Y is an uncountable Borel space; (f) the images of Borel subsets of Y under Borel-measurable functions from Y into X, where Y is an uncountable Borel space. Proof The only new characterization here is (f). If Y is an uncountable Borel space and f : Y + X is Borel-measurable, then for every B E B y , f (B) is analytic in X by Proposition 7.40. To show that every nonempty analytic set A c X can be obtained this way, let q be a Borel isomorphism from Y onto X.M and let F c X M be closed and satisfy proj,(F) = A. Define B = ~ - ' ( F ) E B , . Then (proj, q)(B) = A. If A = @, then f(@) = A for any Q.E.D. Borel-measurable f : Y -+ X. 0

7.6.2 Measurability Properties of Analytic Sets

At the beginning of this section we indicated that extended real-valued functions on a Borel space X whose lower level sets are analytic arise naturally via partial infimization. Because the collection of analytic subsets of an uncountable Borel space is strictly larger than the Borel G-algebra (Appendix B), such functions need not be Borel-measurable. Nonetheless, they can be integrated with respect to any probability measure on (X,Bx). To show this, we must discuss the measurability properties of analytic sets. If X is a Borel space and p~ P(X), we define p-outer measure, denoted by p*, on the set of all subsets of X by

Outer measure on an increasing sequence of sets has a convergence property, namely, p*(En)t p*(U,"=, En)if El c E , c . . . . This is easy to verify from (87) and also follows from Eq. ( 5 ) and Proposition A.1 of Appendix A (see also Ash [Al, Lemma 1.3.3(d)]).The collection of sets Bx(p) defined by

is a o-algebra (Ash [Al, Theorem 1.3.5]), called the completion of .Bxwith respect to p. It can be described as the class of sets of the form B u N as B ranges over 8, and N ranges over all subsets of sets of p-measure zero in B, (Ash [Al, p. 18]), and we have

Furthermore, p* restricted to .9Yx(p)is a probability measure, and is the only extension of p to Bx(p) that is a probability measure. In what follows, we denote this measure also by p and write p(E) in place ofp*(E) for all E E Bx(p). Definition 7.18 Let X be a Borel space. The universal o-algebra axis defined by %, = Bx(p). If E E %,, we say E is universally measurable.

n,.,(,,

The usefulness of analytic sets in measure theory is in large degree derived from the following proposition. Proposition 7.42 (Lusin's theorem) Let X be a Borel space and S a Suslin scheme for a,. Then N(S) is universally measurable. In other words, Y ( a X )= a x .

Proof Denote A = N(S), where S is a Suslin scheme for a , . For (o,, . . . ,o,) E C, define

and

Define also

Then where

7.

168


As o1 oo,M(ol) T N , so R(ol) T A. Likewise, as ok co,M(ol,. . . ,ok- l, ok) f M(ol,. . . ,ok- 1),SO R(ol,. . . ,ok- l, ok)T R(nl,. . . ,ok- l). Given p~ P(X) and E > 0, choose TI ,C,, . . . such that

Then

rk)is universally measurable, so (91) and (93) imply

The set K(r,, . . . ,

We show that

an argument by contradiction will be used to show that for some we have

If no such zl existed, then for every integer k(zl) such that

If li = max,, 0 is arbitrary, this implies that

+ p*(X - A). that p*(E) + p*(X - E ) 2 1, so 1 2 p*(A)

It is true for any E c X

and A is measurable with respect tGp.

Q.E.D.

Corollary 7.42.1 Let X be a Borel space. Every analytic subset of X is universally measurable.

Proof The closed subsets of X are universally measurable, so Y (9, c) 42, by Proposition 7.42. Q.E.D. As remarked earlier, the class of analytic subsets of an uncountable Borel space is not a o-algebra, so there are universally measurable sets which are not analytic. In fact, we show in Appendix B that in any uncountable Borel space, the universal o-algebra is strictly larger than the o-algebra generated by the analytic subsets. 7.6.3 An Analytic Set of Probability Measures

In Proposition 7.25 we saw that when X is a Borel space, the function 8, : P ( X )-t [O, 11 defined by QA(p)= p(A) is Borel-measurable for every Borel-measurable A c X . We now investigate the properties of this function when A is analytic. The main result is that the set { p P~( X ) ( p ( A )2 c) is analytic for each real c. Proposition 7.43 Let X be a Borel space and A an analytic subset of X . For each C E R, the set { p~ P ( X ) l p ( A )2 c ) is analytic.

170

7.


Proof Let S be a Suslin scheme for Fx,the class of closed subsets of X, such that A = N(S). For S E E , let N(s), M(s), R(s), and K(s) be defined by (88)-(90) and (92).Then (91)and (95)hold and each K(s) is closed. We show that for c E R

-

c,,

-

If p(A) 2 c, then for any n 2.1, there exists (i,, . . . ) E M such that (93) is satisfied with p = p and E = lln. Then by (91),for k = 1,2, . . .

On the other hand, given m

PE

n u n iP

n = l zeNs Lemma 8.3 ( F + ) If J o : S -+ R* is identically zero, then TK(Jo)(x) for every x E S, K = 1,. . . ,N, where T Kdenotes the composition of T with itself K times. - co

Proof Suppose for some K I N and ZES that for every x E S, and

8.2

197

THE DYNAMIC PROGRAMMING ALGORITHM

By Proposition 7.50, there are universally measurable selectors pj:S + C, j = 1,. . . ,K - 1, such that p j ( x )U(x) ~ and

+

( T, K - J Tj-')(Jo)(x) s Tj(Jo)(x) 1,

j

=

1,. . . ,K - 1,

for every x E S. Then

where the last inequality is obtained by repeating the process used to obtain the first two inequalities. By Lemma 8.2, there is a stochastic kernel p0 E U(CIS) such that (T,,TK- ')(Jo)(n) = - a. Then (T,,TPl.. . T,,_ ,)(J0)(x) 5 T,,[TK-'(Jo)

Choose any p~ U(CIS)and let By Lemma 8.1,

TC

+ 1 + a + ...+

= ( p o , .. . ,pK-

= - oo.

',p, . . . ,p), SO that T C En.

so for some k I K - 1, jg- dqk(n,pi)= oo. This contradicts the (F+) assumption. Q.E.D. Lemma 8.4 Let (5,) be a sequence of extended real-valued, universally measurable functions on S and let p be an element of U(C1S).

(a) If T,(J,)(x) < oo for every X E S and J k .1J , then T,(Jk).1T,(J). (b) If T,(J;)(x) < oo for every x E S, g 2 0, and J k f J, then T,(Jk)7 T,(J). (c) If { J,) is uniformly bounded, g is bounded, and J, + J , then T,(Jk) T,(J). +

Proof

Assume first that T,(J,) < oo and Jk .1J . Fix x. Since

we have

S

g(x,u)+ a ~,(x')t(dx'lx,u) < co

for p(dulx) almost all u. By the monotone convergence theorem [Lemma 7.1 l(f)I,

for ,u(dulx)almost all u. Apply the monotone convergence theorem again to conclude T,(Jk)(x).lT,(J)(x). If T,(J;) < co,g 2 0, and Jk J, the same type of argument applies. If { J k )is uniformly bounded, g bounded, and J k -+ J, a similar argument using Q.E.D. the bounded convergence theorem applies. The dynamic programming algorithm over a finite horizon is executed by beginning with the identically zero function on S and applying the operator T successively N times. The next theorem says that this procedure generates the optimal cost function. In Proposition 8.3, we show how &-optimalpolicies can also be obtained from this algorithm. Proposition 8.2 ( F + ) ( F - ) Let Jo be the identically zero function on S. Then J; = T ~ ( J ~ ) K , = 1, . . . , N . (18) Proof It suffices to prove (18) for K = N, since the horizon N can be chosen to be any positive integer. For any TC = (,uo,. . . ,pN- E II and K I N, we have J ~ , a=

' '

TpK- l)(JO)

2 (Tpo'

' '

TfiK-2T)(J0)

2

TK(JO),

(I9)

where the last inequality is obtained by repeating the process used to obtain the first inequality. Infimizing over TC E II when K = N and using Corollary 8.1.1, we obtain J; 2 T ~ ( J ~ ) . (20) E

If (F+)holds, then, by Lemma 8.3, Tk(Jo)> - co, k = 1,. . . ,N. For > 0, there are universally measurable selectors Gk:S+ C, k = 0,. . . ,N - 1,

with

G,(x)E

U(x)and

for every x E S (Proposition 7.50).Then

where the last inequality is obtained by repeating the process used to obtain the first two inequalities. It follows that

Combining (20)and (22),we see that the proposition holds under the ( F f ) assumption. If ( F - ) holds, then JK,,(x)< oo for every X E S, Z E IT, K = 1, . . . , N. Use Proposition 7.50 to choose nonrandomized policies ni = (pb, . . . , ph- , ) € I T such that

as i -+ oo. By (19) and Lemma 8.4(a), (T,:. . . T'iN- J ( J O )

J; I inf

(Lo, . . . , i~

-

N - 1

11

= inf. . . inf (T,:. . . T,i?- J ( J 0 ) io

= inf.

h-1

IN-

I

. , inf (T,:. . . T

i~-2 = inf. . . inf (T,:. . . T,:--;T)(Jg) io

io

ix-2

where the last equality is obtained by repeating the process used to obtain the previous equality. Combining (20)and (23),we see that the proposition Q.E.D. holds under the ( F - ) assumption. When the state, control, and disturbance spaces are countable, the model of Definition 8.1 falls within the framework of Part I. Consider such a model, and, as in Part I, let M be the set of mappings p: S -+ C for which p(x)E U(x) for every x E S. In Section 3.2, it was often assumed that for every x E S and p j € M , j = O , . . . , K - 1,wehave

or else for every x E S inf pjeM,O<j - a,

K

=

1,. . . ,N .

(25)

Under the (F') assumption, Lemma 8.3 implies that

-a < T ~ ( J ,I ) pjaM.

inf (T,,...T,K_l)(JO), OsjsK- 1

so (25)is satisfied. Under ( F - ) , we have from Lemma 8.1 that

where 71 = ( p O., . . ,pK- ,), SO (24)holds. The primary reason for introducing the stronger ( F f )and ( F - ) assumptions is to enable us to prove Lemma 8.1. If one chooses instead to take (16)as the definition of JK,, (as is done in

200

8.

THE FINITE HORIZON BOREL MODEL

Section 3.1), then (24) or (25) suffices to prove Proposi.tion 8.2 along the lines of the proof of Proposition 3.1 of Part I. Proposition 8.2 implies the following property of the optimal cost function. Corollary 8.2.1 (F+)(F-) For K lower semianalytic.

=

1,2,. . . ,N, the function

Jg

is

Proof As observed following Definition 8.5, T(J) is lower semianalytic whenever J is. Since Jg = T K ( ~ , )and JoE 0 is lower semianalytic, the result follows. Q.E.D.

We give an example to show that even when l- = SC and the one-stage cost g: SC + R* is Borel-measurable, JT can fail to be Borel-measurable. EXAMPLE 1 Let A be an analytic subset of [0, 11 which is not Borelmeasurable (Appendix B). By Proposition 7.39, there is a closed set F c [O, 11.M such that A = projlo,,](F). Let S = [O, I], C = N , r = SC, and g = X F e Then JT(x) = inf g(x, u) = xAc(x)

Vx E S,

UEC

which is a lower semianalytic but not Borel-measurable function. We could also choose C = [O, 11, l- = SC, B a G,-subset of the unit square SC, and g = z B c . This is because Jlr and N o , the space of irrational numbers in [O, 11, are homeomorphic (Proposition 7.5). But NO=

n(LO,11

- {r})

reQ

is a G,-subset of [0, 11, so there is a homeomorphism cp:N + [O, 11 such that q ( N ) is a G,-subset of [O,l]. Let @:[O, 1 ] N -+ [O, 1][0,1] be the homeomorphism defined by

Then @([O,11.N) = [0, l]q(.N) is a G6-subset of SC = [O, 11[O, I], and since F is a G,-set in [O,l]./V, B = @(F) is a G,-subset of SC which satisfies proj,(B) = A. If g = x B c , then again JT = xAC. We now use Proposition 8.2 to establish existence of &-optimalpolicies. Proposition 8.3 ( F f ) For each E > 0, there exists a nonrandomized Markov E-optimal policy. (F-) For each E > 0, there exists a nonrandomized semi-Markov 8optimal policy and a (randomized) Markov &-optimalpolicy. Proof If ( F+) holds, then the policy (,Lo,. . . ,L,_ ,) constructed in the proof of Proposition 8.2 is &-optimal,nonrandomized, and Markov.

Assume (F-) holds. We show first the existence of an &-optimal,nonrandomized, semi-Markov policy. Let ni = ( p i , . . . , p h - ,) be as in the proof of Proposition 8.2. Then inf (T,?. . . Tpi,- ~ ) ( J o ) N - 1 (io.. . . .i w - 1 ) = inf JN,n(Lo,.. . .,,,(io, . . . .i~ - 1) where #03. . . . i~ - l ) = ( p t , . . . ,pkN:,'). Choose E > 0 and define J;

=

T"(J,)=

Order linearly the countable set ( ~ ( ' 0.,... ' N - 1 ) (i,, . . . , i N - , are positive integers) and define n ( x ) to be the first x ( ' o . . . . . ' ~ - such that

Let the components of n ( x ) be

..

( ~ 6 ( ~ 0 ) > ~ ; ( ~ 1>P;) , . ~(xN-1)).

The set ( x l n ( x )= ~ ( ' ~. .3.' N. -

I))

is universally measurable for each (i,, ..., iN- ,),

SO (PO(XO),PI(XO,XI),.

..,PN-I(XO.XN-I)),

where po(xo) = p","xo) and p k ( x o ,x,) = p 2 ( x k ) , k = 1,. . . ,N - 1, is an Eoptimal nonrandomized semi-Markov policy. We now show the existence of an &-optimal(randomized) Markov policy. By Lemma 8.2, there exist p , - , ~ U(CIS)such that for k = 1,. . . ,N

+

(T,, _ , T k - - ' ) ( J o )s T k ( J o ) &/(I

Proceed as in (21).

+ a + a2 + . . . + a N - I ) .

Q.E.D.

If the (F-) assumption holds and E > 0, it may not be possible to find a nonrandomized Markov &-optimal policy, as the following example demonstrates. N

EXAMPLE^ L e t s = jO,1,2,.. . j , C = ( 1 , 2, . . .), W = ( w 1 , w 2 ) , r = SC, and define

=2

8.

202


The ( F - ) assumption is satisfied. Let z = (,uo,,ul) be a nonrandomized Markov policy. If the initial state x, is neither zero nor one, then regardless of the policy employed, x , = 0 with probability 1 - (llx,), and x , = 1 with probability llx,. Once the system reaches zero, it remains there at no further cost. If the system reaches one, it moves to x , = 0 at a cost of - ~ ~ ( 1 ) . Thus JN,.(x0)= - y l ( l ) / x o if x, # 0, xo # 1, and J$(x,) = - oo if x, # 0, x, # 1. For any E > 0, n cannot be &-optimal. In Example 2, it is possible to find a sequence of nonrandomized Markov policies (n,) such that JN.n. 1 J$ . This example motivates the idea of policies exhibiting {&,}-dominated convergence to optimality (Definition 8.3) and Proposition 8.4, which we prove with the aid of the next lemma. Lemma 8.5 Let (J,} be a sequence of universally measurable functions from S to R* and ,u a universally measurable function from S to C whose graph lies in T. Suppose for some sequence {s,) of positive numbers with E, < m, we have, for every x E S,

I,"=,

S~ : ( x ' ) t ( d x ' ,u(x)) x, < m, and for k

= 2,3,.

lim Jk(x)= J(x) k+ m

..

J(x) < Jk(x)l J(x)

+ E~

J d x ) < J k - 1(x)+ &k

if J(x) > - CO, if J(x) = - m.

Then ) T,(J). lim T p ( J k= k- m

Proof

Since J

< J, for every k, it is clear that T,(J) I lirn inf T,(Jk). k-m

For x E S,

Now lim k - sup X, J f X , J i x , )> - E:~,(x')t(dx'lx, AX)

If J(x')

= - co,then

and since J[J: (x') + lim sup k+

C,"=,E,] t(dx'lx, p(x)) < oo,we have

S;

- m} Jk(x')t(dx'lx,P(x))

x'lJ(x')=

If follows that lim sup T,(Jk) I T,(J). k+ m

Combine (27) and (28) to conclude (26).

(28)

Q.E.D.

Proposition 8.4 (F-) Let {r,) be a sequence of positive numbers with 10. There exists a sequence of nonrandomized Markov policies {n,) exhibiting (&,)-dominated convergence to optimality. In particular, if Jg(x) > - co for all x E S, then for every E > 0 there exists an &-optimal nonrandomized Markov policy. E,

Proof For N = 1, by Proposition 7.50 there exists a sequence of nonrandomized Markov policies n, = (y", such that for all n T,;(Jo)(x)

+

I

T(Jo)(x) - 1/&n

E,

if T(Jo)(x) > - oo, if T(J,)(x) = - a.

We may assume without loss of generality that

Therefore {z,) exhibits (&,)-dominated convergence to one-stage optimality. Suppose the result holds for N - 1. Let z, = (p;, . . . ,p$- ,) be a sequence of (N - 1)-stage nonrandomized Markov policies exhibiting {~,/2cx)dominated convergence to ( N - 1)-stage optimality, i.e., lim J N - l , n=nJ g - l , n+ w

We assume without loss of generality that C,"=, E, < co.By Proposition 7.50, there exists a sequence [pn}of universally measurable functions from S to

204

8.

C whose graphs lie in


r such that

We may assume without loss of generality that T ( J 1 )S T

(

J

)

n

= 2,3,.

...

(32)

By Proposition 7.48, the set

is analytic in SC, and the Jankov-von Neumann theorem (Proposition 7.49) ) ]C implies the existence of a universally measurable y : p r ~ j ~ [ A ( J $ - ~+ whose graph lies in A(J;- ,). Define lii"(x) =

~(x) yn(x)

if x E projs[A(J$ otherwise.

-

,)I,

Then it, = (,P7 TC,) is an N-stage nonrandomized Markov policy which will be shown to exhibit {&,)-dominated convergence to optimality. For X E projs[A(J;- ,)I, we have, from Lemma 8.5 and the choice of y, lim sup JN, i i n ( ~=) lirn sup T,(JNn+ m

n+ m

,,,J(x)

For x $projs[A(J;- ,)I, we have t((xllJ$- ,(xf) = - co) lx, u) = 0 for every u E U(x), SO by (29) JN.%,(XI= TJJN- l,ri,)(x) S Tp(J$- ,)(XI + &n/2,

(34)

and lim sup JN.%,(x)5 lim sup T,,(J;n--'

1)(x)S JE(x)

n+ m

3c

by (31). It follows that lirn JN.

= J;.

n-rm

Suppose for fixed x E S we have JR(x) > - m. Then x $ projs[A(J;- ,)I, and we have, from (31) and (34),

Suppose now that J$(x) = - oo. If x$proj,[A(J$-,)I, imply, for n 2 2, JN,

then (32) and (34)

~z"(x) 5 TAJR- 1 ) ( ~+) &,I"? 5 TPn-I(J$-,)(x) f &,/2 I T,nJN, ii,

we have, from (29) and (30),

while if x E projs[A(J$ JN,

,,,- ,)(XI+ &,I2 - 1 ( ~+ ) &n/2,

~ J N - 1,

ii,(x)

Tp(J, - I , n,)(x) 5 T,(J.N - I , z,, - ,)(x) + &n/2 = JN. en+ &n/2. =

In either case, JN.*,(x) I JN.en- 1 ( ~+) ~ n .

(37)

From (35)-(37) we see that (2,) exhibits (&,)-dominated convergence to Q.E.D. optimality. We conclude our discussion of the ramifications of Proposition 8.2 with a technical result needed for the development in Chapter 10. Lemma 8.6 (F+)(F-) For every p E P(S),

Proof

For p E P(S) and

E

II,

which implies

Choose

E

> 0 and let 2

and it follows that

~ ben &-optimal.If p((x1JY,(x)=

-

a))= 0, then

~fp({xI J;(x) = - a } )> 0 , then

1

J $ ( x ) p ( d x ) = co, then J $ ( x ) p ( d x )= co and (39) follows. If S{x,JX(x)> Otherwise, the right-hand side of (40)can be made arbitrarily small by letting E approach zero, so inf,,,J JN,,(x)p(dx) = - co and (39) is again valid. The lemma follows from (38) and (39). Q.E.D.

We now consider the question of constructing an optimal policy, if this is at all possible. When the dynamic programming algorithm can be used to construct an optimal policy, this policy usually satisfies a condition stronger than mere optimality. This condition is given in the next definition. Definition 8.6 Let .n = ( po, . . . , p, - ,) be a Markov policy and n N - k = ( p k ,. . . , pN- ,), k = 0 , . . . , N - 1. The policy n is uniformly N-stage optimal if

Lemma 8.7 ( F + ) ( F - ) The policy n = (,uo,. . . ,,uN - ,) E II is uniformly N-stage optimal if and only if

Proof

If n

= (,uo,.

where Jo,,o = J ; then for all k

= 0.

. . , p N - is uniformly N-stage optimal, then

If ( T , k T N - k - l ) ( J O )= T N - k ( ~ Ok) = , 0 , . . . ,N

-

1,

where the next to last equality is obtained by continuing the process used to obtain the previous equalities. Q.E.D. Lemma 8.7 is the analog for the Bore1 model of Proposition 3.3 for the model of Part I. Because (F') or (F-) is a required assumption in Lemma 8.1, one of them is also required in Lemma 8.7, as the following example shows. If we take (16) as the definition of Jk. ., then Lemma 8.7, Proposition 8.5, and

Corollaries 8.5.1 and 8.5.2 hold without the (F') and (F-) assumptions. The proofs are similar to those of Section 3.2.

EXAMPLE 3 Let S = { s , t ) u { ( k , j ) / k= 1,2,. . . ; j = 1,2), C = [a,b ) , U(s)= {a,b), U ( t )= U(k,j) = { b ) , k = 1,2,. . . ,j = 1,2, W = S, and cc = 1. Let the disturbance kernel be given by p(sls, a) = 1,

p[tl(k, 2),b] = 1, k = 1,2,. . . , and p(tlt, b ) = 1. Let the system function be f ( x , u, w) = w. Thus if the system begins at state s, we can hold it at s or allow it to move to some state (1, I ) , from which it subsequently moves to some (k,2) and then to t. Having reached t, the system remains there. The relevant costs are g(s, a) = g(s, b ) = g(t, b ) = 0, g[(k,I ) , b] = k, g[(k,2),b] = - k, k = 1,2,. . . . Let .rr = ( p O , p l , p Z )be a policy with p0(s) = b, pl(s) = p2(s)= a. Then

( 0 k T(Jo)(xz)= T,,(Jo)(xz)

T ~ ( J ~= ((T ~, ~ )

-k

=

1: i 0

I

- oo

T 3 ( J o ) ( x o= ) (T,,T,IT,2)(Jo)(xo) = - k

if x , if x , if x ,

= s, = (k, 1), = (k,2),

if x , = ( k , l ) , if x , = (k,2), if x , = t , if x , if x ,

= (k, 1), = (k,2),

However, J,. ,(s) = oo > J,, ,(s) = 0, where 3 = ( P o ,P I , ,Liz) and &(s) = pl(s) = p2(s) = a, so .rr is not optimal and T ~ ( J , )# J g . It is easily verified that 7C is a uniformly three-stage optimal policy, so Corollary 3.3.l(b) also fails to hold for the Bore1 model of this chapter. Here both assumptions (F+) and (F-) are violated. Proposition 8.5 ( F + ) ( F - ) If the infimum in

is achieved for each ~ E Swhere , J g is identically zero, then a uniformly Nstage optimal (and hence optimal) nonrandomized Markov policy exists. This policy is generated by the dynamic programming algorithm, i.e., by measurably selecting for each x a control u which achieves the infimum. Proof Let 7~ = (p,,, . . . ,pNwhere p,-,- , :S + C achieves the infimum in (41)and satisfies p, - ,(x) E U(x) for every x E S, k = 0,. . . ,N - 1 (Proposition 7.50). Apply Lemma 8.7. Q.E.D.

-,

Corollary 8.5.1 (F+)(F-) If U(x) is a finite set for each x E S, then a uniformly N-stage optimal nonrandomized Markov policy exists. Corollary 8.5.2 (F+)(F-) If for each XES,/?E R, and k the set

= 0,.

. . ,N

-

1,

is compact, then there exists a uniformly N-stage optimal nonrandomized Markov policy. Proof

Apply Lemma 3.1 to Proposition 8.5.

Q.E.D.

8.3 The Semicontinuous Models

Along the lines of our development of lower and upper semicontinuous functions in Section 7.5, we can consider lower and upper semicontinuous decision models. Our models will be designed to take advantage of the possibility for Borel-measurable selection (Propositions 7.33 and 7.34), and in the case of lower semicontinuity, the attainment of the infimum in (55) of Chapter 7. We discuss the lower semicontinuous model first. Definition 8.7 The lower semicontinuous, jinite horizon, stochastic, optimal control model is a nine-tuple (S,C, U , W,p, f,cc,g, N) as given in Definition 8.1 which has the following additional properties:

(a) The control space C is compact. (b) The set r defined by (1) has the form l- = Uj"=,Tj, where l-' c T2 c . . . , each Tj is a closed subset of SC, and lim

inf

j + m (x. U ) E P - rJ- 1

g(x,u) = co.?

(c) The disturbance kernel p(dwlx, u) is continuous on T. By convention. the infimum over thc cmpty set is x,so this condition is satisfied if the

rJare all identical for j larger than some index k .

(d) The system function f is continuous on.J W. (e) The one-stage cost function g is lower semicontinuous and bounded below on T. Conditions (c) and (d) of Definition 8.7 and Proposition 7.30 imply that t(dxrlx,u) defined by (3) is continuous on T,since for any h E C(S) we have

Condition (e) implies that the (F') assumption holds. Proposition 8.6 Consider the lower semicontinuous finite horizon model of Definition 8.7. For k = 1,2,. . . ,N, the k-stage optimal cost function Jt is lower semicontinuous and bounded below, and Jz = Tk(J,). Furthermore, a Borel-measurable, uniformly N-stage optimal, nonrandomized Markov policy exists.

Proof Suppose J: S + R* is lower semicontinuous and bounded below, and define K: l- + R* by

K(x, U)= g(x, u) + ~1 S~(x')t(dx'lx, u).

(42)

Extend K to all of SC by defining Z(x, u) =

if (x, u) $ T.

By Proposition 7.31(a) and the remarks following Lemma 7.13, the function K is lower semicontinuous on r. For C E R, the .set {(x,u) E SC/&(X,u) I C) must be contained in some Tkby Definition 7.8(b), so the set {(x,u) E S C ] ~ ( X u), 5 c) = {(x,u) E Tk1 ~ ( xU), I C) is closed in Tk and thus closed in SC as well. It follows that &(x, u) is lower semicontinous and bounded below on SC and, by Proposition 7.32, the function T(J)(x)= inf k(x, u) UGC

is as well. In fact, Proposition 7.33 states that the infimum in (43) is achieved for every x E S, and there exists a Borel-measurable q : S -+ C such that For j = 1,2, . . . , let qj: proj,(rj) + C be a Borel-measurable function with graph in Tj.(Set D = T j in Proposition 7.33 to establish the existence of such p(x) = ql(x) a function.) Define p: S + C so that p(x) = ~ ( xif) T(J)(x) < a, if T(J)(x)= E and x ~ p r o j , ( r ' ) ;and for j = 2,3, . . . , define p(x) = cpj(x) if

T(J)(x) = co and xEprojS(rj)- proj,(Tj-I). Then p is Borel-measurable, p(x) E U(x) for every x E S, and T,(J) = T(J). Since J, r 0 is lower semicontinuous and bounded below, the above argument shows that J,*= Tk(J,) has these properties also, and furthermore, for each k = 0, . . . ,N - 1, there exists a Borel-measurable p,: S -+ C such that pk(x)E U(x) for every x E S and (T,*T~-~-')(J,)= TN-k(JO).The Q.E.D. proposition follows from Lemma 8.7. We note that although condition (a) of Definition 8.7 requires the compactness of C, the conclusion of Proposition 8.6 still holds if C is not compact but can be homeomorphically embedded in a compact space in such a way that the image of Tj, j = 1,2, . . . , is closed in sC. That is to say, the conclusion holds if there is a compact space C and a homeomorphism cp: C -, such that for j = 1,2, . . . , @(Tj)is closed in sC, where

c

The continuity o f f and p(dwlx, u) and the lower semicontinuity of g are unaffected by this embedding. In particular, if Tj is compact for each j, we can take = Y? and use Urysohn's theorem (Proposition 7.2) and the fact that the continuous image of a compact set is compact to accomplish this transformation. We state this last result as a corollary.

c

Corollary 8.6.1 The conclusions of Proposition 8.6 hold if instead of assuming that C is compact and each Tj is closed in Definition 8.7, we assume that each Tj is compact. Definition 8.8 The upper semicontinuous, jinite horizon, stochastic, optimal control model is a nine-tuple (S, C, U , W,p, f,cc,g, N) as given in Definition 8.1 which has the following additional properties: (a) (b) (c) (d)

The set r defined by (1) is open in SC. The disturbance kernel p(dwlx, u) is continuous on T. The system function f is continuous on T W The one-stage cost g is upper semicontinuous and bounded above

on T. As in the lower semicontinuous model, the stochastic kernel t(dx'lx, u) is continuous in the upper semicontinuous model. In the upper semicontinuous model, the (F-) assumption holds. If J : S + R* is upper semicontinuous and bounded above, then K : T + R* defined by (42) is upper semicontinuous and bounded above. By Proposition 7.34, the function T(J)(x)= inf K(x, u) uEU(X)

is upper semicontinuous, and for every E > 0 there exists a Borel-measurable

y :S -+ C such that y ( x )E U ( x )for every x E S, and

-

Since J o 0 is upper semicontinuous and bounded above, so is J z = Tk(Jo), k = 1,2, . . . , N. The following proposition is obtained by using these facts to parallel the proof of the (F-) part of Proposition 8.3. Proposition 8.7 Consider the upper semicontinuous finite horizon model of Definition 8.8. For k = 1,2,. . . ,N, the k-stage optimal cost function J z is upper semicontinuous and bounded above, and J z = Tk(Jo).For each E > 0, there exists a Borel-measurable, nonrandomized, semi-Markov, E-optimal policy and a Borel-measurable, (randomized) Markov, E-optimal policy.

Actually, it is not necessary that S and C be Bore1 spaces for Proposition 8.7 to hold. Assuming only that S and Care separable metrizable spaces, one can use the results on upper semicontinuity of Section 7.5 and the other assumptions of the upper semicontinuous model to prove the conclusion of Proposition 8.7. It is not possible to parallel the proof of Proposition 8.4 to show for the upper semicontinuous model that given a sequence of positive numbers {E,) with E, LO,a sequence of Borel-measurable, nonrandomized, Markov policies exhibiting {&,)-dominatedconvergence to optimality exists. The set A(JX- ,) defined by (33) may not be open, so the proof breaks down when one is restricted to Borel-measurable policies. We conclude this section by pointing out one important case when the disturbance kernel p(dw)x,u) is continuous. If W is n-dimensional Euclidean space and the distribution of w is given by a density d(w(x,u ) which is jointly continuous in ( x ,u) for fixed w, then p(dw(x,u) is continuous. To see this, let G be an open set in W and let (x,, u,) + ( x ,u) in SC. Then lim infp(Glx,, u,)

,-a

= lim inf k-+ a2

L

d(wlxk,uk)dw

by Fatou's lemma. The continuity ofp(dwlx, u)follows from Proposition 7.21.+ Note that by the same argument.

so p(Gjs,.u,) + p(G1x.u). Under this condition, the assumption that the systcm function is continuous in the state (Definitions 8.7(d)and 8.8(c)) can bc weakcned. See [H3] and [S5].

212

8.


In fact, it is not necessary that d be continuous in ( x ,u) for each w, but only that (x,, u,) 4 ( x ,u) imply d(w(x,,u,) + d(w(x,u) for Lebesgue almost all w. For example, if W = R, the exponential density d(wlx,U ) =(-[P:{

w - m(x,u ) ) ]

if w 2 m(x,u), if w < m(x,u),

where m:SC -+ R is continuous, has this property, but need not be continuous in ( x ,u) for any w E R.

Chapter 9

The Infinite Horizon Bore1 Models

A first approach to the analysis of the infinite horizon decision model is to treat it as the limit of the finite horizon model as the horizon tends to infinity. In the case (N) of a nonpositive cost per stage and the case (D) of bounded cost per stage and discount factor less than one, this procedure has merit. However, in the case (P) of nonnegative cost per stage, the finite horizon optimal cost functions can fail to converge to the infinite horizon optimal cost function (Example 1 in this chapter), and this failure to converge can occur in such a way that each finite horizon optimal cost function is Borel-measurable, while the infinite horizon optimal cost function is not (Example 2). We thus must develop an independent line of analysis for the infinite horizon model. Our strategy is to define two models, a stochastic one and its deterministic equivalent. There are no measurability restrictions on policies in the deterministic model, and the theory of Part I or of Bertsekas [B4], hereafter abbreviated DPSC, can be applied to it directly. We then transfer this theory to the stochastic model. Sections 9.1-9.3 set up the two models and establish their relationship. Sections 9.4-9.6 analyze the stochastic model via its deterministic counterpart. 9.1 The Stochastic Model Definition 9.1 An infinite horizon stochastic optimal control model, denoted by (SM), is an eight-tuple ( S ,C, U , W ,p, f , a, g) as described in

9.

214

THE INFINITE HORIZON BOREL MODELS

Definition 8.1. We consider three cases, where l- is defined by (1) of Chapter 8: (P) 0 I g(x,u) for every ( x ,u) E r. ( N ) g(x, u) I 0 for every ( x ,U ) E l-. (D) 0 < a < 1, and for some b E R, - b I g(x, u) I b for every ( x ,u) E r. Thus we are really treating three models: (P), ( N ) ,and (D). If a result is applicable to one of these models, the corresponding symbol will appear. The assumptions (P), ( N ) , and (D) replace the (F+)and (F-) conditions of Chapter 8. Definition 9.2 A policy for (SM) is a sequence n = (p,, p,, . . .) such that for each k, pk(duklxo,u,, . . . ,u,-, ,x,) is a universally measurable stochastic kernel on C given SC . . . CS satisfying

,,

for every (x,, uo, . . . ,u,- x,). The concepts of semi-Markov, Markov, nonrandomized, and F-measurable policies are the same as in Definition 8.2. W e denote by n' the set of all policies for ( S M )and by II the set of all Markov policies. If n is a Markov policy of the form n = (p,p,. . .), it is said to be stationary. As in Chapter 8, we often index S and C for clarity, understanding S, to be a copy of S and Ck to be a copy of C. Suppose p E P ( S )and n = ( p , ,p, ,. . .) is a policy for (SM). By Proposition 7.45, there is a sequence of unique probability measures r,(n,p) on SoCo. - . S N - ,CN- N = 1,2,. . . , such that for any N and any universally measurable function h : S o C o .. . S N - C N - + R* which satisfies either Jh+ drN(n,p)< co or Jh- drN(n,p)< CO, (4) of Chapter 8 is satisfied. Furthermore, there exists a unique probability measure r(n,p) on SoCoS,C,... such that for each N the marginal of r(n,p) on SoCo. . . S N - C N - is rN(n,p). With rN(n,p) and r(n,p) determined in this manner, we are ready to define the cost corresponding to a policy.

,,

, ,

, ,

Definition 9.3 Suppose n is a policy for (SM). The (infinite horizon) cost corresponding to n at x E S is

+ T h e interchange of integration and summation is justified by appeal to the monotone convergence theorem under (P)and (N), and the bounded convergence theorem under (D).

9.1

215

THE STOCHASTIC MODEL

If TC = (p,p,. . .) is stationary, we sometimes write J, in place of J,. The (infinite horizon) optimal cost at x is J*(x) = inf J,(x). XEII'

If

E

> 0, the policy

TC

is E-optimal at x provided

If J,(x) = J*(x),then n is optimal at x. If n is &-optimalor optimal at every x E S, it is said t o be E-optimalor optimal, respectively. It is easy to see, using Propositions 7.45 and 7.46, that, for any policy n, J,(x) is universally measurable in x. In fact, if n = ( p o , p l , .. .) and nk = (p,, . . . ,pk- then J,. ,,(x) defined by ( 5 ) of Chapter 8 is universally measurable in .Y and lim J,, ,,(x)

k* m

V x E S.

= J,(x)

(3)

If TC is Markov, then (3) can be rewritten in terms of the operators T,, of Definition 8.4 as lim (T,, . . . T,,_ ,)(J,)(x)= J,(x)

k+ w

V x E S,

(4)

which is the infinite horizon analog of Lemma 8.1. If n is a Borel-measurable policy and g is Borel-measurable, then J,(x) is Borel-measurable in x (Proposition 7.29). It may occur under (P),however, that lirn,,, J z ( x ) # J*(x),where J:(x) is the optimal k-stage cost defined by (6) of Chapter 8. We offer an example of this.

EXAMPLE 1 Let S

=

(0,1,2,. . .}, C

=

{1,2,. . .), U(x) = C for every

~ E SC( ,= 1,

The problem is deterministic, so the choice of Wand p(dwlx, u) is irrelevant. Beginning at x , = 0, the system moves to some positive integer u, at no cost. It then successively moves to u, - 1, u, - 2,. . . , until it returns to zero and the process begins again. The only transition which incurs a nonzero cost is the transition from one to zero. If the horizon k is finite and u, is chosen larger than k, then no cost is incurred before termination, so J,*(O)= 0. Over the infinite horizon, the transition from one to zero will be made infinitely often, regardless of the policy employed, so JK(0)= x.

216

9.


For 71 = ( p Op, l , . . .) E IIf and p E P(S),let q,(n, p) be the marginal of r(n,p) on S,C,, k = 0,1, . . . . Then (7) of Chapter 8 holds, and if 71 is Markov, (8) holds as well. Furthermore, from (1)we have

which is the infinite horizon analog of (9) of Chapter 8. Using these facts to parallel the proof of Proposition 8.1, we obtain the following infinite horizon version.

Proposition 9.1 ( P ) ( N ) ( D ) If x E S and 71' E ll',then there is a Markov policy 71 such that

Corollary 9.1.1 ( P ) ( N ) ( D ) We have J*(x) = inf J,(x)

V x E S,

where II is the set of all Markov policies for (SM).

9.2 The Deterministic Model Definition 9.4 Let (S, C, U , W , p , f ,a,g) be an infinite horizon stochastic optimal control model as given by Definition 9.1. The corresponding infinite horizon deterministic optimal control model, denoted by ( D M ) , consists of the following : P ( S ) State space. P(SC) Control space. 0 Control constraint. A function from P ( S ) to the set of nonempty subsets of P ( S C ) defined for each p E P ( S ) by U(p)= ( q P(SC)lq(T) ~ = 1 and the marginal of q on S is p),

where l- is given by (1)of Chapter 8. J; System function. The function from P(SC) to P ( S ) defined by

where t(dx1lx,u) is given by (3) of Chapter 8. a Discount factor. One-stage cost function. The function from P(SC) to R* given by

(6)

The model ( D M )inherits considerable regularity from (SM).Its state and control spaces P ( S ) and P ( S C ) are Bore1 spaces (Corollary 7.25.1). The system function 7 is Borel-measurable (Proposition 7.26 and Corollary 7.29.1), and the one-stage cost function g is lower semianalytic (Corollary 7.48.1). Furthermore, under assumption (P) in ( S M ) , we have 2 0 , while under ( N ) ,g < 0, and under ( D ) , 0 < cc < 1 and - b I g I b. Definition 9.5 A policy for ( D M ) is a sequence of mappings E = (p,, ,El,. . .) such that for each k, p k : P ( S )+ P ( S C ) and pk(p)e U ( p )for every p~ P(S). The set of all policies in ( D M ) will be denoted by fl. We place no measurability requirements on these mappings. A policy n of the form ?t= (p,p,. . .) is said to be stationary. Definition 9.6 Given p, E P ( S ) and a policy the cost corresponding to 71 at p, is

= ( P o , ,El, . . .)

for ( D M ) ,

where the control sequence {q,) is generated recursively by means of the equation

and the system equation If n = ( p , p , . . .) is stationary, we write at p, is

in place of 3,. The optimal cost

J*(po) = inf TZ(po). Tren

The concepts of &-optimaland optimal policies for ( D M )are the same as those given in Definition 9.3 for ( S M ) . Definition 9.7 A sequence ( p , ,q, ,q , ,. . .) E P(S)P(SC)P(SC) . . is admissible in ( D M )if q, E D(po)and q,, E D[T(qk)],k = 0,1,. . . . The set of all admissible sequences will be denoted by A.

,

The admissible sequences are just the sequences of controls go, q,, . . . together with the initial state p, which can be generated by some policy for ( D M )via (10)and (11).Except for p,, the measures pk are not included in the sequence, but can be recovered as the marginals of the measures qk on S [cf. (6)l. Definition 9.8 Let 7: P ( S ) + R* be given and let ,E: P ( S ) + P ( S C ) be such that p ( p ) D~ ( p ) for every p~ P(S). The operator T , mapping 7 into

218

9.


T,(J):P ( S ) + R* is defined by The operator T mapping J into T ( J ) : P ( S )+ R* is defined by

+

T ( J ) ( p )= inf .(g(q) a J [ T ( q ) ] )

Vp€P(S).

4E U(P)

Because (DM) is deterministic, it can be studied using results from Part I, Chapters 4 and 5 or from DPSC. This is because there is no need to place measurability restrictions on policies in a deterministic model. The operators T , and T of Definition 9.8 are special cases of those defined in Section 2.1. In the present case, we take H(p, q, J ) to be

The monotonicity assumption of Section 2.1 is satisfied by this choice of H. The cost corresponding to a policy 75 = (Po,,El, . . .) as given by (9) is easily seen to be of the form (cf. Section 2.2)

7, = lim (T,;

. . T,,-

,)(J,),

N+ m

where J,(p) = 0 for every p~ P(S). It is a straightforward matter to verify that under (D) the contraction assumption of Section 4.1 is satisfied when B is taken to be the set of bounded real-valued functions on P(S), m is taken to be one, and p = a. Under (P), Assumptions I, 1.1, and 1.2 of Section 5.1 are satisfied, while under (N), Assumptions D, D.l, and D.2 of the same section are in force. 9.3 Relations between the Models

Definition 9.9 Let n = ( p , ,pl,.. .) E Il be a Markov policy for (SM) and a policy for (DM). Let p, E P(S) be given. If for all k 75 = (p,, p l , . . .) E

where p, is generated from p, by 75 via (10) and ( l l ) ,then n and n are said to correspond at p,. If n and 75 correspond at every p~ P(S), then n and ? are i said to correspond. If and E correspond at p,, then the sequence of measures [q,(n, p,), ql(n,p,), . . .] generated from p, by n via (8) of Chapter 8 is the same as the sequence (q,, q,, . . .) generated from p, by 7i via (10) and (11). If n and if correspond, then they generate the same sequence (q,, q,, . . .) for any initial p,.

9.3

219

RELATIONS BETWEEN THE MODELS

Proposition 9.2 ( P ) ( N ) ( D ) Given a Markov policy 7-1 E II, there is a corresponding Z E fl. If 7 1 fZ~ and p, E P ( S ) are given, then there is a Markov policy n E II corresponding to ?? at p, .

Proof If n = ( p o ,p,, . . .) E II is given, then for each k and any pkE P(S), there is a unique probability measure on SC, which we denote by p,(p,), satisfying (12) (Proposition 7.45). Furthermore,

so 71 = (Po,P I , . . .) is in fj[ and corresponds t o n. If Z = (p,, p,, . . .) E and p, E P ( S ) are given, let ( p , ,p, ,pz ,. . .) be generated from p, by 71 via (10)and (11). For each k, choose a Borel-measurable stochastic kernel pk(dulx)which satisfies (12)for this particular p, (Corollary 7.27.2). Then (13)holds, so

for pk almost every x. Altering p,(dulx) on a set of pk-measure zero if necessary, we may assume that (14) holds for every X E S and (12) is satisfied. Then Q.E.D. n = ( p o , p l , . . . ) E I I corresponds to Z at p,. Proposition 9.3 ( P ) ( N ) ( D ) Let p E P(S), n E II, and n and ?? correspond at p, then

EE

n be given. If

Proof We have from (7) of Chapter 8, (5), (8), (9),and the monotone or bounded convergence theorems

= J,(p).

Corollary 9.3.1

Q.E.D.

( P ) ( N ) ( D ) Let x E S, n E II, and E E fl be given. If n and

71 correspond at p,, then

220

9.

Corollary 9.3.2 Proof x € S,


(P)(N)(D) For every x E S,

Corollaries 9.1.1,9.3.1, and Proposition 9.2 imply that, for every J*(px) = inf A(&)= inf J,(x) = J*(x). ?ten Z E ~

Q.E.D.

Corollary 9.3.2 shows that J * and I* are related, but in a rather weak ) . Proposition 9.5 we way that involves 5* only on S = { ~ , E P ( S ) ( X E SIn strengthen this relationship, but in order to state that proposition we must show a measurability property of J*. This is the subject of Proposition 9.4, which we prove with the aid of the following lemma.

Lemma 9.1 The set A of admissible sequences in (DM) is an analytic subset of P(S)P(SC)P(SC).. . . Proof

The set A is equal to A. n [n?==,B,], where

By Corollary 7.35.2, it suffices to show that A , and B,, k = 0,1, . . . , are analytic. Using the result of Proposition 7.38, this will follow if we show that

are analytic. Let P(T) = (qeP(SC)(q(T)= I), where r is given by (1) of Chapter 8. Then P(T) is analytic (Proposition 7.43). Equation (6) implies that A is the intersection of the analytic set P(S)P(T)(Proposition 7.38) with the graph of the function a:P(SC) + P(S) which maps q into its marginal on S. It is easily verified that o is continuous (Proposition 7.21(a) and (b)), so Gr(o) is Bore1 (Corollary 7.14.1). Therefore, A is analytic. The set B is the inverse image ofA under the Borel-measurable mapping (q,, q,) + [T(qo),q,], Q.E.D. so is also analytic (Proposition 7.40). Proposition 9.4 (P)(N)(D) The function T* :P(S) + R* is lower semianalytic. Proof

Define G:A

+

R* by

9.3

221


where A is the set of admissible sequences (Definition 9.7). Then G is lower semianalytic by Lemma 7.30(2), (4) and Lemma 9.1. By the definition of J * and A, we have J*(po)=

inf (40 , a ,

. . .)€Apo

PO^...

so J* is lower semianalytic by Proposition 7.47. Corollary 9.4.1 analytic.

Proof

(P)(N)(D) The function J * : S

VPOEP(S),

(16)

Q.E.D. + R*

is lower semi-

By Corollary 9.3.2,

where 6(x) = p, is the homeomorphism defined in Corollary 7.21.1. Apply Lemma 7.30(3) and Proposition 9.4 to conclude that J * is lower semianalytic. Q.E.D. Lemma 9.2 Given p~ P(S) and such that

1

E

J*(x)p(dx)+&

(N)

TAP) 5

> 0, there exists a policy 71 for (DM)

S if J *(x)p(dx) S

if J*(x)p(dx)>-KI, = - KI.

Proof As a consequence of Corollary 9.4.1, J*(x)p(dx) is well defined. Let p E P(S) and E > 0 be given. Let G: A -+ R* be defined by (15).Proposition 7.50 guarantees that under (P)and (D) there exists a universally measurable selector cp :P(S) -+ P(SC)P(SC). . . such that (p, cp(p))E A for every p E P(S) and ~PEP(S). G[P, cp(p)l 5 J*(P) + & Let a : S + P(SC)P(SC). . . be defined by ~ ( x=) ~ ( p , ) Then . o is universally measurable (Proposition 7.44) and Under (N),there exists a universally measurable o :S + P(SC)P(SC). . . such that for every x E S, (p,, o(x))E A and G[P,> d x ) ] 5 wherep,(J*) 1 otherwise.

J*(x) + F - (1 + F')/E~,(J*)

if J*(x) > - co, if J*(x) = - co,

= p({xlJ*(x) = - co))ifp({xlJ*(x) = - a

) ) > 0 and P,(J*)

(18) =

9.

222


Denote o(x) = [qO(d(xO, '0)lx), ql(d(xl, ul)lx), . . Each qk(d(xk,uL)Ix) is a universally measurable stochastic kernel on SkCkgiven S. Furthermore, qo(d(x0, uo)lx)E U(P,)

vx E S,

and,for k = 0,1,. . . , qk+l(d(xk+l,Uk+l)Ix)Eu(~[qk(d(xk~uk)lx)])

Fork

= 0,1,.

. . , defineq,~P(SC)by

T h e n a ( T ) = l , k = 0 , 1 , . . . . Weshowthat(p,qo,q,,... ) ~ A . S i n c e t h e m a r ginal of q,(d(xo, uo)lx) on So is p,, we have

so q 0 € O(p). For k

= 0,1,.

. . , we have

Therefore qk+,E U[y(qk)] and (p,qo,q,, . . .) E A. Let 71 be any policy for (DM) which generates the admissible sequence (p,qO,q l , . . .). Then under (P) and (D), we have from (17) and the monotone or bounded convergence theorem

9.3


Under (N), we have from the monotone convergence theorem

If p((xlJ*(x) = - co)) = 0, (18) and (19) imply

where both sides may be - co. If p({xl J*(x) = - a ) ) > 0, then JJ*(x) p(dx) = - co and we have, from (18) and (19), P

)

> -z

J

< E - (1 Proposition 9.5

Proof

[J*(x)

+ &]p(dx)- (1 + e2)/&

+ E ~ ) / E= -I/&.

Q.E.D.

(P)(N)(D) For every p E P(S),

Lemma 9.2 shows that

For the reverse inequality, let p be in P(S) and let E be a policy for (DM). There exists a policy TC l ll corresponding to 7C at p (Proposition 9.2), and, by Proposition 9.3,

By taking the infimum of the left-hand side over n e fI, we obtain the desired result. Q.E.D. Propositions 9.3 and 9.5 are the key relationships between (SM) and (DM). As an example of their implications, consider the following corollary. Corollary 9.5.1 (P)(N)(D) Suppose TC lll and E E fI are corresponding policies for (SM) and (DM). Then 71 is optimal if and only if E is optimal.

Proof

If TC is optimal, then

If 7C is optimal, then J,(x)

= J,(p,)

= y*(p,) = J*(x)

VxcS.

Q.E.D.

The next corollary is a technical result needed for Chapter 10.

9.

224


Corollary 9.5.2 ( P ) ( N ) ( D ) For every p E P(S), J ~ * ( x ) p ( d x=) inf J ~ , ( x ) ~ ( d x ) . Proof

By Propositions 9.2 and 9.3, J*(p) = inf

Apply Proposition 9.5.

S~ , ( x ) ~ ( d x ) Vp

E

P(S).

Q.E.D.

We now explore the connections between the operators T, and T pand the operators T and T. The first proposition is a direct consequence of the definitions. We leave the verification to the reader. Proposition 9.6 ( P ) ( N ) ( D ) Let J : S -+ R* be universally measurable and satisfy J 2 0, J 1 0, or - c 5 J 1 c, c < co, according as (P), ( N ) , or ( D )is in force. Let Y :P ( S ) + R* be defined by

J

VP P(S),

J ( P )= J(x)p(dx) and suppose p: P ( S )+ P(SC) is of the form

for some p ~ U(CIS)t.Then p ( p ) D(p) ~ for every p ~ P(S), and

Proposition 9.7 ( P ) ( N ) ( D ) Let J : S + R* be lower semianalytic and satisfy J 2 0, J 1 0, or - c I J I c, c < co, according as (P), (N), or ( D ) is in force. Let 7 :P ( S ) + R* be defined by

Then

Proof -

For p E P ( S ) and q E D(p) we have

g b ) + u J [ f ( q ) l = Js,[s(r 4

+

J, J(x0t(dxf\x,u)]q(d(x,u ) )

The set U(CIS),defined in Section 8.2. is the collection of universally measurable stochastic kernels p on C given S which satisfy p ( U ( x ) l x )= 1 for every X E S. +

9.4


which implies

Given E > 0, Lemma 8.2 implies that there exists y E U(CIS) such that

Let q E D(p) be such that

Then

where ST(J)(X)~(~X) + E may be - co.Therefore,

9.4 The Optimality Equation-Characterization of Optimal Policies

As noted following Definition 9.8, the model (DM) is a special case of that considered in Part I and DPSCt. This allows us to easily obtain many results for both (SM) and (DM). A prime example of this is the next proposition. Proposition 9.8 (P)(N)(D) We have J* = f(J*),

JY= T(JY).

Proof The optimality equation (21) for (DM) follows from Propositions 4.2(a), 5.2, and 5.3 or from DPSC, Chapter 6, Proposition 2 and Chapter 7, + Whereas we allow Z j t o be extended rcal-valued, in Chapter 7 of DPSC thc one-stage cost function is assumed to be real-valued. This more restrictive assumption is not essential to any of the results we quote from DPSC.

226

9.


Proposition 1. We have then, for any x E S,

by Propositions 9.5 and 9.7, so (22) holds as well.

Q.E.D.

Proposition 9.9 (P)(N)(D) If 71 = (p, p,. . .) is a stationary policy for (DM), then 4 = T,(J,). If n = (p,,u,. . .) is a stationary policy for (SM), then J, = T,(J,).

Proof For (DM) this result follows from Proposition 4.2(b), Corollary 5.2.1, and Corollary 5.3.2 or from DPSC, Chapter 6, Corollary 2.1 and Chapter 7, Corollary 1.1. Let n = (,u,p,. . .) be a stationary policy for (SM) and let Ti = (p, p, . . .) be a policy for (DM) corresponding to n. Then for each XES,

by Propositions 9.3 and 9.6.

Q.E.D.

Note that Proposition 9.9 for (SM) cannot be deduced from Proposition 9.8 by considering a modified (SM) with control constraint of the form

as was done in the proof of Corollary 5.2.1. Even if ,u is nonrandomized so that (23) makes sense, the set

may not be analytic, so U , is not an acceptable control constraint. The optimality equations are necessary conditions for the optimal cost functions, but except in case (D) they are by no means sufficient. We have the following partial sufficiency results. Proposition 9.10

(P) If J:P(s) -t [0, oo] and J 2 T(J), then J 2 J*. If J : S -+ [0, m] is lower semianalytic and J 2 T(J), then J 2 J*. (N) If J:P(S) -t [ - co,01 and J I T(J), then 7 I J*. J*. If J : S -+ [ - co,0] is lower semianalytic and J I T(J), then J I ]:P(S) + [ c, c], c < m, and J = T(J), then J = 5". If (D) If J : S -+[-c, c], c < a,is lower semianalytic and J = T(J), then J = J*. Proof We consider first the statements for (DM). The result under (P) follows from Proposition 5.2, the result under (N) from Proposition 5.3, and the result under (D) from Proposition 4.2(a). These results for (DM)

9.4

227


follow from Proposition 2 and trivial modifications of the proof of Proposition 9 of DPSC, Chapter 6. We now establish the (SM) part of the proposition under (P). Cases (N) and (D) are handled in the same manner. Given a lower semianalytic function J : S + [0, co] satisfying J 2 T(J), define J : P(S) + [0, a] by (20).Then

by Proposition 9.7. By the result for (DM), J 2 J*. In particular, J(x)

= J(P,)

2 J*(p,)

= J*(x)

Proposition 9.11 Let Ti = ( F , p, . . .) and policies in (DM) and (SM), respectively.

Vx E S. 71 = (p, p,

Q.E.D.

. . .) be stationary

(P) If 2 : P(S) + [0, co] and 7 2 T,(J), then J 2 I,. If J : S + [0, co] is universally measurable and J 2 T,(J), then J 2 J,. T,(J), then J I7,. (N) I f J : P ( S ) + [-co,0] and J I If J:S + [ - oo,0] is universally measurable and J I T,(J), then J C J,. (D) If J : P(S) + [- c, c], c < co, and J = T,(J), then J = 7,. If J : S + [- c, c], c < co,is universally measurable and J = T,(J), then J = J,. Proof The (DM) results follow from Proposition 4.2(b) and Corollaries 5.2.1 and 5.3.2 or from DPSC, Corollary 2.1 and trivial modifications of Corollary 9.1 of Chapter 6. The (SM) results follow from the (DM) results and Proposition 9.6 in a manner similar to the proof of Proposition 9.10. Q.E.D. Proposition 9.11 implies that under (P), J, is the smallest nonnegative universally measurable solution to the functional equation Under (D), J, is the only bounded universally measurable solution to this equation. This provides us with a simple necessary and sufficient condition for a stationary policy to be optimal under (P) and (D). Proposition 9.12 (P)(D) Let 7t = (I*, p, . . .) and = (p,p, . . .) be stationary policies in (DM) and (SM), respectively. The policy 7t is optimal if and only if I* = T,(J*). The policy TC is optimal if and only if J * = T,(J*).

Proof If 7t is optimal, then J, = J*. By Proposition 9.9, J * = T,(J*). Conversely, if J * = T,(J*), then, by Proposition 9.11, J * 2 J, and n is

228

9.


optimal. The proof for (SM) follows from the (SM) parts of the same propositions. Q.E.D. Corollary 9.12.1 (P)(D) There is an optimal nonrandomized stationary policy for (SM) if and only if for each x E S the infimum in

is achieved. Proof If the infimum in (24) is achieved for every x E S, then by Proposition 7.50 there is a universally measurable selector p: S -t C whose graph lies in r and for which

Then by Proposition 9.8 T,(J*)

=

T(J*) = J*,

so n = (p, p,. . .) is optimal by Proposition 9.12. If TC = (p, p, . . .) is an optimal nonrandomized stationary policy for (SM), then by Propositions 9.8 and 9.9

so p(x) achieves the infimum in (24) for every x E S.

Q.E.D.

In Proposition 9.19, we show that under (P) or (D), the existence of any optimal policy at all implies the existence of an optimal policy that is nonrandomized and stationary. This means that Corollary 9.12.1 actually gives a necessary and sufficient condition for the existence of an optimal policy. Under (N) we can use Proposition 9.10 to obtain a necessary and sufficient condition for a stationary policy to be optimal. This condition is not as useful as that of Proposition 9.12, however, since it cannot be used to construct a stationary optimal policy in the manner of Corollary 9.12.1. Proposition 9.13 (N)(D) Let Z = ( p ,p, . . .) and n = (p,p, . . .) be stationary policies in (DM) and (SM), respectively. The policy n is optimal if and only if 7, = T(Jp).The policy n is optimal if and only if J , = T(J,). Proof

If E is optimal, then 7,

= J*.

By Proposition 9.8

9.5


229

Conversely, if J, = T(J,), then Proposition 9.10 implies that 7, I 5" and 7t is optimal. If 71 is optimal, J , = T(J,) by the ( S M ) part of Proposition 9.8. The converse is more difficult, since the ( S M ) part of Proposition 9.10 cannot be ., . .) be invoked without knowing that J, is lower semianalytic. Let n = ( A,ii a policy for ( D M ) corresponding to 71 = ( p ,p,. . .), so that J,(p) = JJ,(x)p(dx) for every p E P(S). Then for fixed p E P ( S ) and q E D(p),

provided the integrand

is universally measurable in x. But T(J,) = J , by assumption, which is universally measurable, so

By taking the infimum of the left-hand side over q E g ( p ) and using Proposition 9.9, we see that T(7,) 2 7, = T,(J,). The reverse inequality always holds, and by the result already proved for ( D M ) ,7tis optimal. The optimality ofn follows from Corollary 9.5.1. Q.E.D. 9.5

Convergence of the Dynamic Programming AlgorithmExistence of Stationary Optimal Policies

Definition 9.10 The dynamic programming algorithm is defined recursively for ( D M ) and ( S M ) by

We know from Proposition 8.2 that this algorithm generates the k-stage optimal cost functions J:. For simplicity of notation, we suppress the Y here. At present we are concerned with the infinite horizon case and the possibility that J , may converge to J Y as k + co.

230

9.


Under (P),JoI J1and so 7, = T(Jo)I T(J,) = 7,. Continuing, we see that 7, is an increasing sequence of functions, and so J, = lirn,,, Jkexists and takes values in [0, + a ] . Under (N), Jkis a decreasing sequence of functions and 7, exists, taking values in [ - oo,01. Under (D), we have

and, in general,

As k -t m, we see that b b/(l - a), so J, = lirn,,,

6 aj + Tk(Jo)increases to a limit. But b xj"=,aj = Tk(Jo)exists and satisfies - b/(l - a) I J,.

Similarly, we have

7, < b/(l

- a).

then Now if J : P ( S ) -t [ - c, c], c < a,

JoI J + c,

T(7,) I T(J + c) = ac + T(7),

and, in general,

It follows that

7,

lirn inf Tk(J), k - t rn

and by a similar argument beginning with J - c < J o , we can show that lim sup,,, Tk(J)< 7,. This shows that under (D), if J is any bounded realvalued function on P(S), then J, = lim,,, Tk(J). The same arguments can be used to establish the existence of J, = lirn,, , J, . Under (P), J, :S -+ [0, oo] ; under (N), J, :S -+ [ - m, 0] ; and under (D), J, = lirn,,, Tk(J)takes values in [ - b/(l - a), b/(l - a)] where J:S -+ [ - c, c], c < oo, is lower semianalytic. Note that in every case, J, is lower semianalytic by Lemma 7.30(2).

+

Lemma 9.3 (P)(N)(D) For every p e P(S),

9.5


231

Proof For k = 0,1,. . . , the lemma follows from Proposition 9.7 by induction. When k = co,the lemma follows from the monotone convergence theorem under (P) and (N) and the bounded convergence theorem under (D). Q.E.D. Proposition 9.14 (N)(D) We have

1, = J*, J,

= J*.

Indeed, under (D) the dynamic programming algorithm can be initiated from any 7: P(S) [- c, c], c < co, or lower semianalytic J : S -+ [ - c, c], c < co,and converges uniformly, i.e., -+

lim sup ITk(J)(p)- J*(p)l = 0, k + w p E P(S)

lim suplTk(J)(x)- J*(x)l = 0. k+m XES

(27) (28)

Proof The result for (DM) follows from Proposition 4.2(c) and 5.7 or from DPSC, Chapter 6, Proposition 3 and Chapter 7, Proposition 4. By Lemma 9.3,

so (25)implies (26).Under (D), if a lower semianalytic function J : S + [ - c, c], c < co, is given, then define J : P(S) -+ [- c, c] by (20). Equation (28) now follows from (27) and Propositions 9.5 and 9.7. Q.E.D. Case (D) is the best suited for computational procedures. The machinery developed thus far can be applied to Proposition 4.6 or to DPSC, Chapter 6, Proposition 4, to show the validity for (SM) of the error bounds given there. We provide the theorem for (SM). The analogous result is of course true for (DM). Proposition 9.15 (D) Let J : S Then for all X E Sand k = 0,1,. . . ,

-+

[-c, c], c < co,be lower semianalytic.

where

bk = [%/(I- a)] i n f [ ~ ~ ( ~ )-( Tkx ) '(J)(x)], xes

6,

=

[%/(I- a)] sup[Tk(J)(x)- Tk- '(J)(x)]. xeS

(30)

9.

232


Proof Given a lower semianalytic function J: S -+ [ - c, c], c < co,define 1 :P(S) + [- c, c] by (20). By Proposition 9.7,

Therefore bk

=

[a/(l - a)] inf [Tk(J)(p)- Tk-l(J)(p)], P PP(S)

6,

=

[a/(l - a)] sup [Tk(J)(p)- Tk-l(J)(p)], P E P(S)

where b, and 5, are defined by (30) and (31). Taking a, = a, = a in Proposition 4.6 or using the proof of Proposition 4, Chapter 6 of DPSC, we obtain

Substituting p = p, in this equation, we obtain (29).

Q.E.D.

It is not possible to develop a policy iteration algorithm for (SM) along the lines of Proposition 4.8 or 4.9. One difficulty is this. If at the kth iteration ~ then J,, is we have constructed a policy (p,,pk,. . .), where p , U(CIS), universally measurable but not necessarily lower semianalytic. We would like to find p, + E ~ ( ~ 1such s ) that TPk+ ,(J,,) 5 T(J,,) + 6 , where E > 0 is some prescribed small number, but Proposition 7.50 does not apply to this case. We turn now to the question of convergence of the dynamic programming algorithm under (P). Without additional assumptions, we have only the following result.

,

Proposition 9.16 (P) We have

7, 5 J*, J, I J*. Furthermore, the following statements are equivalent:

Jr, T(J,),

(a) = (b) J, = J*, ( 4 J,= T(J,), (d) J, = J*.

Proof It is clear that (32) holds and, by Proposition 9.10, implies the equivalence of (a) and (b). Lemma 9.3, Proposition 9.5, and (32) imply (33). Conditions (a) and (c) are equivalent by Lemma 9.3 and Proposition 9.7. Conditions (b) and (d) are equivalent by Lemma 9.3 and Proposition 9.5. Q.E.D.

9.5


233

In Example 1, we have J,(O) = 0 and J*(O) = co,so strict inequality in (32) and (33) is possible. We present now an example in which not only is J, different from J*,but J, is Borel-measurable while J * is not. EXAMPLE 2 (Blackwell) Let C be the set of finite sequences of positive integers and H the set of functions h from C into {O,l). Then H can be regarded as the countable Cartesian product of copies of {O,1) indexed by C.Let (0,l) have the discrete topology and H the product topology, so H is a complete separable metrizable space (Proposition 7.4). A typical basic open set in H is { h ~ H l h ( s=) 1 VseC,, h(s) = 0 VSEC,), where C1 and C2 are finite subsets of C. Consider a Suslin scheme R:C + B, defined by R(s)={h~Hlh(s)=l)

VSEC.

Then N(R) = ( h H13(11, ~ c2,. . .) E JV such that h( 0, there exists an &-optimal nonrandomized semi-Markov policy for (SM). If for each x E S there exists a policy for (SM) which is optimal at x, then there exists a semi-Markov (randomized) optimal policy. Proof Under (N) we have Jk.1J * (Proposition 9.14), so, given the analytically measurable sets

E

> 0,

converge up to S as k + co. By Proposition 8.3, for each k there exists a k-stage nonrandomized semi-Markov policy nk such that for every x E S

+

Jk(x) (42) -1 / ~

if Jk(x)> - m, if Jk(x)= - co.

Then for x E A, we have either J*(x) > - oo and Jk,

&(x) I Jk(x)

+ (~12)5 J*(x)

E,

or else J*(x) = - co. If J*(x) = - m, then either Jk(x)= - m and Jk,

."(XI I - 1/&,

or else Jk(x)> - m and

,,

k Choose any p~ U(CIS) and define itk = (pk,,. . . ,pkp, p,. . .), where nk = ( p i , . . . ,pi- For every x E Ak, we have

J,(x)

J*(x)

5 Jk, =k(x)I

+E

if J*(x) > -a, if J*(x) = - a,

so itk is a nonrandomized semi-Markov policy which is &-optimalfor every x~ A,. The policy TC defined to be itk when the initial state is in A,, but not in Aj for any j < k, is semi-Markov, nonrandomized, and &-optimalat every X E U F = ~=AS.~

9.

240


Suppose now that for each x E S there exists a policy z x for (SM) which is optimal at x. Let 75" be a policy for (DM) which corresponds to zx, and let ( p x ,q& q:, . . .) be the sequence generated from p, by E x via (10) and ( 1 1). If G : A -, [ - m, 0] is defined by (15),then we have from Proposition 9.3 that

J * ( x )= J,(x)

= J,(P,)

=

G(p,, q"o q;,. . .I.

(42)

We have from Proposition 9.5 and (16) that

J*(x) = J*(p,)

inf

=

(qo,ql..

G(px,q,, q l , . . .).

..)€Apx

(43)

Therefore the infimum in (43)is attained for every p, E S, where S = (p,l y E S), so by Proposition 7.50, there exists a universally measurable selector $ : S -, P(SC)P(SC) . . such that $(p,) E A,, and

Let 6 : s -+ S be the homeomorphism 6 ( x )= p, and let cp(x) = $[S(x)].Then cp is universally measurable, q ( x )E A,, , and

J*(x)=G[p,,cp(x)]

V~ES.

Denote

cp(x) = [ ~ O ( ~ ( ~ O ~ U O ) I41(d(xl,ul)Ix),. X ) > . .I. For each k 2 0, q,(d(x,, u,)~x)is a universally measurable stochastic kernel on S,C, given S, and by Proposition 7.27 and Lemma 7.28(a),(b),q,(d(x,, u,)Ix) can be decomposed into its marginal p,(dx,lx), which is a universally measurable stochastic kernel on S, given S, and a universally measurable stochastic kernel p,(du,lx, x,) on C, given SS,. Since po(dxolx)= p,(dx,), the stochastic kernel p,(du,lx, x,) is arbitrary except when x = x,. Set

The sequence z = ( P o ,p,, p, ,. . .) is a randomized semi-Markov policy for (SM). From (7)of Chapter 8, we see that for each x E S

qk(n,P,) = qk(d(Xk,U L ) ~ X ) From (5), (15),and (44),we have

so 71 is optimal.

VXE S, k

= 031,.

...

Q.E.D.

Although randomized polcies may be considered inferior and are avoided in practice, under (N) as posed here they cannot be disregarded even in deterministic problems, as the following example demonstrates.

EXAMPLE 3 (St. Petersburg paradox) Let S = (0,1,2,. . .), C = (0, I), U(x) = C for every x E S, CI = 1,

f (x, 4 =

1

if u = 1 , x # 0 , otherwise, -2"

i f x # O , u=O, otherwise.

Beginning in state one, any nonrandomized policy either increases the state by one indefinitely and incurs no nonzero cost or else, after k increases, jumps the system to zero at a cost of -2,+l, where it remains at no further cost. Thus J*(1) = -a,but this cost is not achieved by any nonrandomized policy. On the other hand, the randomized stationary policy which jumps the system to zero with probability 5 when the state x is nonzero yields an expected cost of - cc and is optimal at every x E S. The one-stage cost gin Example 3 is unbounded, but by a slight modification an example can be constructed in which g is bounded and the only optimal policies are randomized. If one stipulates that J * must be finite, it may be possible to restrict attention to nonrandomized policies in Proposition 9.20. This is an unsolved problem. If (SM) is lower semicontinuous, then Propositiq 9.19 can be strengthened, as Corollary 9.17.2 shows. Similarly, if (SM) is upper semicontinuous, a stronger version of Proposition 9.20 can be proved. Proportional 9.21 Assume (SM) satisfies conditions (a)-(d) of Definition 8.8 (the upper semicontinuous model).

(D) For each E > 0, there exists a Borel-measurable, &-optimal,nonrandomized, stationary policy. (N) For each E > 0, there exists a Borel-measurable, &-optimal,nonrandomized, semi-Markov policy. Under both (D) and (N), J * is upper semicontinuous. Proof Under (D) and (N) we have lim,,, J, = J * (Proposition 9.14), and each J, is upper semicontinuous (Proposition 8.7). By an argument similar to that used in the proof of Corollary 9.17.2, J * is upper semicontinuous. By using Proposition 7.34 in place of Proposition 7.50, the proof of Proposition 9.19 can be modified to show the existence of a Borel-measurable, E-optimal,nonrandomized, stationary policy under (D).By using Proposition 8.7 in place of Proposition 8.3, the proof of Proposition 9.20 can be modified to show the existence of a Borel-measurable, &-optimal, nonrandomized, semi-Markov policy under (N). Q.E.D.

Chapter 10

The Imperfect State Information Model

In the models of Chapters 8 and 9 the current state of the system is known to the controller at each stage. In many problems of practical interest, however, the controller has instead access only to imperfect measurements of the system state. This chapter is devoted to the study of models relating to such situations. In our analysis we will encounter nonstationary versions of the models of Chapters 8 and 9. We will show in the next section that nonstationary models can be reduced to stationary ones by appropriate reformulation. We will thus be able to obtain nonstationary counterparts to the results of Chapters 8 and 9. 10.1 Reduction of the Nonstationary Model-State Augmentation The finite horizon stochastic optimal control model of Definition 8.1 and the infinite horizon stochastic optimal control model of Definition 9.1 are said to be stationary, i.e., the data defining the model does not vary from stage to stage. In this section we define a nonstationary model and show how it can be reduced to a stationary one by augmenting the state with the time index.

10.1

REDUCTION OF THE NONSTATIONARY MODEL

243

We combine the treatments of the finite and infinite horizon models. Thus when N = cc and notation of the form S o ,S,, . . . ,S,-, or k = 0,. . . , N - 1 appears, we take this to mean S o , S,, . . . and k = 0, 1, . . .,respectively. Definition 10.1 A nonstationary stochastic optimal control model, denoted by (NSM), consists of the following objects: N Horizon. A positive integer or co. S,, k = 0,. . . ,N - 1 State spaces. For each k, S, is a nonempty Borel space. C,, k = 0,. . . ,N - 1 Control spaces. For each k, C , is a nonempty Borel space. U,, k = 0,. . . ,N - 1 Control constraints. For each k, U , is a function from S, to the set of nonempty subsets of C,, and the set rk

=

{ ( ~ 2 k~ k ) l E~ Skk > uk

Uk(xk)}

(1)

is analytic in S,C,. W,, k = 0,. . . ,N - 1 Disturbance spaces. For each k, W, is a nonempty Borel space. p,(dw,lx,, u,), k = 0,. . . ,N - 1 Disturbance kernels. For each k, P k ( d ~ k lu,) ~ kis, a Borel-measurable stochastic kernel on W, given S,C,. f,,k = 0,. . . ,N - 2 System functions. For each k, f, is a Borel-measurable function from S,C, W, to S,, . Discount factor. A positive real number. cl g,, k = 0,. . . ,N - 1 One-stage cost functions. For each k, g, is a lower semianalytic function from T, to R*.

,

We envision a system which begins at some x , E S, and moves successively through state spaces S,, S,+ ,. . . and, if N < cc, finally terminates in S N - , . A policy governing such a system evolution is a sequence nk = (p,, pk+ . . ,pN- ,), where each pj is a universally measurable stochastic kernel on C j given S,C,. . . C j - ,Sj satisfying

,, ,

,,

for every (x,, u,, . . . ,u j - x j ) Such a policy is called a k-originating policy and the collection of all k-originating policies will be denoted by nk.The concepts of senzi-Markov, Markov, nonrandomizell and 9-measurable policies are analogous to those of Definitions 8.2 and 9.2. The set 11° is also written as II', and the subset of II' consisting of all Markov policies is denoted by II. Define the Borel-measurable state transition stochastic kernels by

244

10.

THE IMPERFECT STATE INFORMATION MODEL

Given a probability measure p, E P(S,) and a policy nk = (p,,. . . ,pN- E n k , defineforj= k , k + 1, . . . , N - 1

x tj-l(dxjlxj-

,, uj-

~ j - l ( ~ ~ j - l I ~ k. ., ,uj-2,xj-l)' ~k,. ' . ~ k ( ~ ~ k l ~ k ) ~ k ( ~ ~ k ) VS~E Bsj, Cj€BCj. (2) There is a unique probability measure qj(nk,p , ) ~P(SjCj) satisfying (2). If the horizon N is finite, we treat (NSM) only under one of the following assumptions :

If N

=

co,we treat (NSM) only under one of the assumptions:

(P) OIg,(x,,u,) f o r e v e r y ( x k , u k ) ~ r k , k =,..., O N-1. (N) gk(xk,u,)I 0 forevery(x,,u,)~l-,, k = 0 , . . . , N - 1. (D) 0 < a < 1, and for some b E R, - b I gk(xk,uk) I b for every (x,, uk)E r,, k = 0, . . . , N - 1. As in Chapters 8 and 9, the symbols (F+),(F-), (P),(N), and (D) will be used to indicate when a result is valid under the appropriate assumption. We define the k-originating cost corresponding to nk at x, E S, to be

and the k-originating optimal cost at x, E S, to be J*(xk,k) = inf Jffk(x,,k). ak~nk A policy n E 11° is E-optimalat x,

E S,

if

The policy TC is optimal at x, if J,(x,, 0) = J"(x,, 0). We say TC E nois &-optimal (optimal) if it is &-optimal(optimal) at every x, E So. Let (E,) be a sequence of positive numbers with F, 10. A sequence of policies jn,} c 11° is said to

10.1

REDUCTION OF THE NONSTATIONARY MODEL

exhibit {E,)-dominated conuergence to optimality if lim J,,(xo, 0 ) = J*(xo,0)

VxOE S O ,

n+ m

and for n = 2,3,. . . J,,(xo, 0 )

J*(xo,O) f En J ,,_,( x o , O ) + ~ ,

if J * ( x o ,0 ) > - co, if J * ( x O , O ) = - a .

Definition 10.2 Let a nonstationary stochastic optimal control model as defined by Definition 10.1 be given. The corresponding stationary stochastic optimal control model, denoted by (SSM), consists of the following objects. ( T is both a terminal state and the only control available at that state. If N = oo,the introduction of T is unnecessary.):

~)Ix,E

S = U ~ Z{(x,, ; S k ) v { T ) State space. C= [(u,, k)lukE C,) u { T ) Control space. U Control constraint. A function from S to the set of nonempty subsets of C defined by U(x,, k ) = ((u,, k)lukE U,(xk)), U ( T )= { T ) . W= ((w,, k)Iw, E &) Disturbance space.

U:Z,

ULil

p(dwlx, u) Disturbance kernel. If

f

vk B,,,

System function. We define for k

',E

we define

= 0, . . . ,N -

2

and for the remaining two stages

f

[(xN-l,N - l),(uN-l,N - l),(wN-l,N - 111 = T , f ( T , T , w) = T . a Discount factor. g One-stage cost function. We define g[(xk k),( ~ k)] k = gk(~k~ g ( T , T ) = 0. 2

N

Horizon.

k ) ,

@I (9)

Consider the mapping cp, :S, + S given by cpk(xk)= (xk,k). We endow S with the topology that makes each cp, a homeomorphism, and we endow C and W with similar topologies. The spaces S, C, and Ware Borel. The set

246

10.


is analytic, and g defined on l- by (8) and (9) is lower semianalytic. The disturbance kernel p(dwlx, u) is not defined on all of SC by (4), but it is defined on a Bore1 subset of SC containing I- - {(T, T)),which is all that is necessary. Likewise, the system function f is not defined on all of SCW by (5)-(7), but the set of points where it is not defined has probability zero under any policy governing the system evolution. Both p(dwlx, u) and f are Borel-measurable on their domains. Thus (SSM) is a special case of the stochastic optimal control model of Definition 8.1 (N < co)or Definition 9.1 (N = co). If N < co, the ( F f ) and (F-) assumptions on (SSM) are given in Section 8.1. These are equivalent to the respective (F') and (F-) assumptions on (NSM)given earlier in this section. If N = co, the (P),(N),and (D)assumptions on (SSM) of Definition 9.1 are equivalent to the respective (P),(N), and (D) assumptions on (NSM) given earlier in this section. The reader can verify that there is a correspondence of policies between S (SSM) is J*(x,, k) (NSM) and (SSM), and the optimal cost at (x,,~ ) E for given by (3). Because of these facts, results already proved for (SSM) with either a finite or infinite horizon have immediate counterparts for (NSM). An illustration of this is the nonstationary optimality equation. Proposition 10.1 (P)(N)(D) Let J*(x,, k) be defined by (3). For fixed k, J*(x,, k) is lower semianalytic on S,, and

We do not list all the results for (NSM) that can be obtained from (SSM). The reader may verify, for example, that the existence results of Propositions 8.3 and 8.4, are valid for (NSM) in exactly the form stated. From Propositions 9.19 and 9.20 we conclude that, under (P) and (D), an &-optimalnonrandomized Markov policy exists for (NSM), while under (N), an &-optimal nonrandomized semi-Markov policy exists. In what follows, we make use of these results and reference only their stationary versions. 10.2 Reduction of the Imperfect State Information Model-Sufficient Statistics

Before defining the imperfect state information model, we give without proof some of the standard properties of conditional expectations and probabilities we will be using. For a detailed treatment, see Ash [All. Throughout this discussion, (R,.F,P) is a probability space and X is an extended realvalued random variable on R for which either E[X +]or E[X-] is finite. If 9 c .Fis a o-algebra on R, then the expectation of X conditioned on 9 is any %measurable, extended real-valued, random variable E [ x I ~ ] ( . )

10.2

REDUCTION OF THE IMPERFECT STATE INFORMATION MODEL

247

on Q which satisfies

It can be shown that at least one such random variable exists. Any such random variable will be called a version of E[XI9]. If X(co) 2 b for some b E R and every co E Q, then it can be shown that for any version E [XI9](.) the random variable E [ x I ~ ](. ) defined by g[xl9](,)

= max(E[Xl9](w), b),

is also a version of E[XI9]. If 8 c 9 is a collection of sets which is closed under finite intersections and generates the o-algebra 9 and if Y is an extended real-valued, %measurable, random variable satisfying

then Y satisfies (10) for every D E 9 , and Y is a version of E[XI9]. If 8 c 9 is a o-algebra, then E{E[XI91 I ~ ) ( u= ) E[XJ8l(w) (11) for P almost every w. Suppose now that (Q,,.F1) and ( Q 2 , F 2 )are measurable spaces and Y, :Q + 0, and Y2 :Q -+ R2 are measurable. Let g:QlR2 + R* be measurable and satisfy either E[g+(Y,, Y,)] < co or E[gp(Y1, Y2)] < co.We define E[XI Y,l(w) = E[XI@-(Y1)l(o), where F(Y,)

=

{Y;~(F)~FEF,}.

We define for y, E Q,

E[xIY,]

where w(y,) is any element of Y;'({yl)). Since is F(Y,)measurable, it is constant on Y; '(( y,)), and this definition makes sense. Note that E[XI Y, = y,] is a function of y,, not of w. We have for any y, E Y,

for P almost every y,. We use the phrase "for P almost every y," to indicate that, in this case, P((w ~ R l ( 1 2fails ) when y, For F E F 2 , define

=

Y,(w))) = 0.

248

10.


Suppose t(dy21y,) is a stochastic kernel on (R,, P2)given R , such that for every F E P2 PLY2 €Fly1 = y11

=

t(Fly1)

for P almost every y,. Then (12)can be extended to

for P almost every y,. We will find (11) and (13) particularly useful in our treatment of the imperfect state information model. They will be used without reference to this discussion. Definition 10.3 The impevfect state information stochastic optimal control model (ISI) is the ten-tuple (S, C, (U,, . . . , U,-,), Z , a, g, t, s,, s, N ) described as follows: S, C, a, g, t State space, control space, discount factor, one-stage cost function, and state transition kernel as given in Definition 8.1 and (3) of Chapter 8. We assume that g is defined on all of SC. Z Observation space. A nonempty Bore1 space. U,, k = 0,. . . ,N - 1 Control constraints. Define for k = 0,. . . ,N - 1,

An element of I, is called a kth information vector. For each k, U , is a mapping from I, to the set of nonempty subsets of C such that

analytic. s, Initial observation kernel. A Borel-measurable stochastic kernel on given S. s Obserziation kernel. A Borel-measurable stochastic kernel on Z given CS. N Horizon. A positive integer or co. For the sake of simplicity, we have eliminated the system function, disturbance space, and disturbance kernel from the model definition. In what follows, our notation will generally indicate a finite N. If N = a,the appropriate interpretation is required. The system moves stochastically from state x, to state x,,, via the state transition kernel t(dx,+ ,lx,. u,) and generates cost at each stage of g(x,, u,). The observation z,, is stochastically generated via the observation kernel s(dz,+ ,ju,, x,, ,) and added to the past observations and controls (z,, u,, . . . , z,,u,) to form the (k + 1)stinformation vector i, = (z,, u,, . . . ,z,, u,, z,, ,). The first information vector i, = ( s o )is generated by the initial observation

,

,

10.2

249


kernel so(dzo~xo), and the initial state xo has some given initial distribution p. The goal is to choose u, dependent on the kth information vector i, so as to minimize

Definition 10.4 A policy for (ISI) is a sequence z = ( y o ,. . . ,y,- ,) such that, for each k, ,u,(du,lp; i,) is a universally measurable stochastic kernel on C given P(S)Iksatisfying

If for each p, k, and i,, y,(du,lp; ik) assigns mass one to some point in C, n is nonrandomized. The concepts of Markov and semi-Markov policies are of no use in (ISI), since the initial distribution, past observations, and past controls are of genuine value in estimating the current state. Thus we expect policies to depend on the initial distribution p and the total information vector. In the remainder of this chapter, H will denote the set of all policies in (ISI). Just as we denote the set of all sequences of the form (z,, uo,. . . ,u,- ,z,) E Z C . . . C Z by I, and call these sequences the kth information vectors, we find it notationally convenient to denote the set of all sequences of the form ( x o ,zo ,u o ,. . . ,x,, z,, u,) E SZC . . . S Z C by Hk and call these sequences the kth history vectors. Except for u,, the kth information vector is that portion of the kth history vector known to the controller at the kth stage. Given p E P(S) and z = ( y o , . . . ,u ,-, ,)E H, by Proposition 7.45 there is a sequence of consistent probability measures P,(n,p) on H,,k = 0,. . . , N - 1, defined on measurable rectangles by

,

Definition 10.5 Given p E P(S), a policy n = ( y o , . . . ,y N - ,) E Il,and a positive integer K I N, the K-stage cost corresponding to n at p is

If N < m, the cost corresponding lo 71. is J,,

.,

and we assume either

250

10.


If N = cc, the cost corresponding to 7c is J, = lim,,, J,,,, and to ensure that this limit is a well-defined extended real number, we impose one of the following conditions: g(x, u) for every (x, u) E SC. (P) 0 I (N) g(x, u) 5 0 for every (x, u) E SC. (D) O < a < l , a n d f o r s o m e b ~ R , - b 5 g ( x , u ) 1 b f o r e v e r y ( x , u ) ~ S C . The optirnal cost at p is J%P) = inf JN, .(PI. n e Il

The concepts of optimality at p, optimality, E-optimalityat p, and E-optimality of policies are analogous to those given in Definition 8.3. If N < oo and (F') or (F-) holds, then by Lemma 7.11(b)

If N

=

oo and (P), (N), or (D) holds, then

To aid in the analysis of (ISI), we introduce the idea of a statistic sufficient for control. This statistic is defined in such a way that knowledge of its values is sufficient to control the model. Definition 10.6 A statistic for the model (ISI) is a sequence (yo,. . . ,yN- ,) of Borel-measurable functions yk:P(S)Ik+ Y,, where Y , is a nonempty Bore1 space, k = 0,. . . ,N - 1. The statistic (yo,. . . ,y,- ,) is suficient for control provided : (a) For each k, there exists an analytic set c YkC such that projyk($k)= Y , and for every p E P(S)

rk

rk

=

(20)

{(ik2u)I[~k(~; ik),"] ~ ~ k ) ,

where r, is defined by (15). We define O k ( ~ k )=

($k)yr.

(21)

(b) There exist Borel-measurable stochastic kernels Fk(dyk+1 y,, u,) on = 0,. . . ,

,

Y,+ given Y,C such that for every p E P(S), n E ll,_Y,+ EB,, ,, k +

10.2


251

N - 2, we have

for Pk(n,p) almost every (L,, B,).+ (c) There exist lower semianalytic functions fying for every p€P(S),n ~ l lk, = 0,. . . , N - 1,

gk:rk[- co, co] -t

satis-

for Pk(n,p)almost every (%,a,), where the expectation is with respect to Pk(n, PICondition (a) of Definition 10.6 guarantees that the control constraint set Uk(ik)can be recovered from q,(p; i,). Indeed, from (15), (20), and (21), we have for any p€P(S), i k € I k k, = 0,. . . ,N - 1,

If U,($ = C for every i , I,, ~ k = 0,. . . ,N - 1, then condition (a) is satisfied with r, = KC. This is the case of no control constraint. Condition (b) guarantees that the distribution of y,,, depends only on the values of y, and u,. This is necessary in order for the variables y, to form the states of a stochastic optimal control model of the type considered in Section 10.1. Condition (c) guarantees that the cost corresponding to a policy can be computed from the distributions induced on the (y,, u,) pairs. We temporarily postpone discussion on the existence and the nature of particular statistics sufficient for control, and consider first a perfect state information model corresponding to model (ISI) and a given sufficient statistic. Definition 10.7 Let the model (ISI) and a statistic sufficient for control (qO,.. . ,yN-J be given. The perfect state information stochastic optimal control model, denoted by (PSI), consists of the following (we use the notation of Definitions 10.3 and 10.6):

Y,, k = 0,. . . ,N - 1 C Control space. 0 , k = 0,. . . ,N - 1 a Discount factor. , k = 0,. . . ,N - 1 k = 0,. . . ,N - 2 N Horizon.

rk,

State spaces. Control constraints. One-stage cost functions. State transition kernels.

+ In this context "for P,(?r,p) almost every (?,,a,)" means that the set { ( x , , z, u,, z k . uk)E Hkl ( 2 2 ) holds when 7, = q k ( p ;i,) Z , = u,) has Pk(x,p)-measure one.

. . . ,x , ,

252

10.


Thus defined, (PSI) is a nonstationary stochastic optimal control model in the sense of Definition 10.l.+The definitions of policies and cost functions for (PSI) are given in Section 10.1. We will use ( ^ ) to denote these objects in (PSI). For example, I?' is the set of all (0-originating) policies and fi is the set of all Markov (0-originating) policies for (PSI). If it = (Go, . . . , fiN- ,) is a policy for (PSI), then by (24) and Proposition 7.44 the sequence

where

,,

i, = (zO,u,,. . . ,uk- z,),

k

= 0,.

. . ,N - 1,

(25)

is a policy for (ISI). We call this policy E also, and can regard fit as a subset of II in this sense. If it is a nonrandomized policy for (PSI), then it is also nonrandomized when considered as a policy for (ISI).We will see in Proposition 10.2 that il results in the same cost for both (PSI) and (ISI). Define cp :P(S) + P(Y,) by

Thus defined, cp(p) is the distribution of the initial state yo in (PSI) when the initial state x, in (ISI) has distribution p. By Corollary 7.26.1, for every _Y, E By,the mapping

is Borel-measurable. Define a Borel-measurable stochastic kernel on S given P(S) by q(dxolp)= p(dxo). Then (26) can be written as cp(P)(_Y,)= S*_Yo(x07 P ) ~ ( ~ ~ O I P ) . It follows from Propositions 7.26 and 7.29 that cp is Borel-measurable. For p E P(S), define the mapping I/,,,:H, + YoCo . . . Y,C, by

where (25) holds. For q E P(Yo) and il = (fi,, . . . ,fiN- ,) E fi', there is a sequence ofconsistent probability measures P,(il, q) generated on YoCo.. . KC,, k = 0,. . . ,N - 1, defined on measurable rectangles by

'

The disturbance spaces, disturbance kernels. and system functions in ( P S I ) can be taken to be W, = Y,+ p,(dw,l~.,. u,) = ~,(djl,+, J,, u,), and f,(y,,u,, w,) = w,. respectively.

,,

10.2


253

For a Markov policy ?€I?, these objects are related to the probability measures P,(it,p) defined by (16) in the following manner. Lemma 10.1 Suppose p E P(S) and it E fi.Then for k for every Bore1 set B c YoCo.. . &Ck,we have

= 0,.

. . ,N - 1 and

Pk(?, P ) [ ~ L : ( ~ ) = ] pk[it> (P(P)](~). Proof

(29)

It suffices to prove that if _YO E B ~ , ,COE.%~,,,. . . ,& ~ , 9 8 ~ ~ ,

CkE BCk,then

P , ( ~ , ~ ) ( ( l l o ( ~ ; i o ) ~uoECo,. - Y o . . . ,rk(P;ik)E-Yk>~ k € C k ) ) = Pk[% (P(p)](&CO. . . &Ck).+

(30)

For k = 0, (30) follows from (16), (26), and (28). If (29) holds for some k < N, then using (16), (22), (28), and (29), we obtain

As noted earlier, (PSI) is a model of the type considered in Section 10.1. The (F+)and (F-) conditions of Section 10.1, when specialized to the (PSI) model, will be denoted by (Pf) and (E-), respectively. These conditions are not to be confused with the ( F + ) and (F-) conditions for the IS1 model given in this section. In a particular problem it is often possible to see the relationship between these finiteness conditions on the two models. In the general case, the relationship is unclear. We point out, however, that if g is bounded below or above, then ( F + )or (F-) is satisfied for (ISI), respectively, and given any statistic sufficient for control, the corresponding Gk can be chosen so as to be bounded below or above, respectively. If a particular result holds when we assume (F+)on the (ISI) model and ( p + )on the (PSI) model, the notation ( F + , F + ) will appear. The notation (F-,P-) has a similar meaning. In this context, we define

.

where ij = ( z o , u o , . . . u j _ ,.z,). We will often use this notation t o indicate a set which depends on functions of some or all of the components of a Cartesian product.

254

10.


If N = co, we consider conditions (P), (N), and (D) for (ISI) and the corresponding conditions (p),(N), and @) for (PSI). In this case, however, if (P) holds for (ISI) and lower semianalytic functions g k : f k+ [- co,co] satisfying (23) exist, there is no loss of generality in assuming that Zjk 2 0 for every k, i.e., (p) holds for (PSI). Likewise, if (N) or (D) holds for (ISI), we may assume without loss of generality that (N) or @), respectively, holds for (PSI). As in the finite horizon case, we adopt the notation (P, P), (N, N), and (D, D) to indicate which assumptions are sufficient for a result to hold. holds, From Section 10.1, we have that when (pi), (p-), (p),(Q),or (8) then the (0-originating) cost covvesponding to a policy il for (PSI) at y e Yo is

where N may be infinite. The (0-originating) optimal cost for (PSI) at y e Yo is j;( y) = inf

jN,

%en'

,(y).

(32)

The remainder of this section is devoted to establishing relations between costs, optimal costs, and optimal and nearly optimal policies for the (ISI) and (PSI) models. Proposition 10.2 (F+,F+)(F-, F-)(P, F)(N, R)(D,D) For every peP(S) and il E fi, we have JN.

Froof

%(P)= L

~(Yo)P(P)(~Yo).

. ~ N ,

(33)

From (31), (28), (23), (IS), (19), and Lemma 10.1, we have

where the ( F + ) or (F-) assumption is used to interchange integration and summation when N < co, and the monotone or bounded convergence Q.E.D. theorem is used when N = co. Corollary 10.2.1 ( F + , ~ + ) ( F p-)(P,~)(N,N)(D,D) -, For every p~ P(S), we have

10.2


255

Proof The function &yo) is lower semianalytic, so the integral in (34) is defined. From Proposition 10.2, we have

so it suffices to show that

This follows from Lemma 8.6 and Corollary 9.5.2.

Q.E.D.

We wish now to establish a relationship similar to (33) between the optimal cost functions for (ISI) and (PSI). In light of Corollary 10.2.1, it suffices to show that given any policy for (ISI), a policy for (PSI) can be found which does at least as well. This is formalized in the next lemma, and the analog of (33) is given as part of Proposition 10.3.

Lemma 10.2 (F', F'+)(F-,F-)(P,P)(N,R)(D, D) Given p~ P(S) and n E l7,there exists it E fi such that

Proof L e t p € P ( S ) a n d n = ( p O,..., , u N . . . , ) € ~ b e g i v e n . F o r k =,..., 0 N - 1, let Qk(z,p) be the probability measure on Y,Ckdefined on measurable rectangles to be There exists a Borel-measurable stochastic kernel fi,(dukly,) on C, given Y , such that for every Bore1 set B c Y,C, we have

In particular, 1 = Pk(n, ~ ) ( { ( ~ukk,) ~ r k ) )

so, altering fik(duklyk) on a set of measure zero if necessary, we may assume =y1kforevery ) y , Y,. ~ Letit = (fi,,. . . ,fiN-,). that (38) h o l d s a n d f i k ( ~ k ( y k ) ~ Then it is a Markov policy for (PSI). We show by induction that for _Y, €By,, C,E Bc, k = 0,. . . ,N - 1,

256

10.


We see from (26) and (37) that the marginal of Qo(n,p) on Yo is cp(p).Equation (39) for k = 0 follows from (28) and (38). Assume that (39) holds for k. From (38), (37), (22), and the induction hypothesis, we have

= Jyoc0

yk,

Lk+, P ~ + ~ ( G + ~ ~ Y ~ + ~ ) ~ ~ ( ~ Y ~ + ~ Y ~ , ~ ~

= P k + l [ ~ , c p ( P ) l ( ~ ~~k_+Yl , + l , u k ~ _ C k + l ) ) .

Taken together, (37)and (39)imply that for& E By,,C, we have

E

a,, k = 0,. . .,N - 1,

If (40) is used in place of Lemma 10.1, the proof of Proposition 10.2 can now be used to prove (36). Q.E.D. Definition 10.8 Given q E P(Yo) and r > 0, a policy it E I?' is said to be weakly q-&-optimalif

The policy it is said to be q-optimal if q((yOEY ~ ~ J ~ = , % Jj$(y0))) ( ~ ~= ) 1. Equation (35) shows that given any p E P(S) and E > 0, a weakly cp(p)-Eoptimal Markov policy exists. The next proposition shows that such a policy is &-optimalat p when considered as a policy in (ISI). Proposition 10.3 (F+$+)(F-, F-)(P, P)(N,R)(D, D) We have

Furthermore, if it is optimal, cp(p)-optimal, or weakly cp(p)-&-optimalfor (PSI), then it is optimal, optimal at p, or &-optimal at p, respectively, for (ISI). If it is e-optimal for (PSI) and (F+,p'), (P,P), or (D,D) holds, then it is also &-optimalfor (ISI).

10.2


257

Proof Equation (41) follows from Corollary 10.2.1 and Lemma 10.2. Let it be &-optimalfor (PSI). It is clear that under (P, P) and (D, D), we have

j;(Yo)>-m ~N,~(Y ) 5o YO) +

VYOEYO,

v

~

~

(42)

E

(43)

Under (F+,p+), (42) follows from Lemma 8.3 and Proposition 8.2, so again (43) holds. We have from (41) and Proposition 10.2 that

so 2 is e-optimal for (ISI). The remainder of the proposition follows from (41) and Proposition 10.2. Q.E.D. We shall show shortly that a statistic sufficient for control always exists, and indeed, in many cases it can be chosen so that (PSI) is stationary. The existence of such a statistic for (ISI) and the consequent existence of the corresponding model (PSI) enable us to utilize the results of Chapters 8 and 9. For example, we have the following proposition. Proposition 10.4 (F', P+)(F-, F-)(P, P)(N,R)(D, D) If (yo, . . . , y,- ,) is a statistic sufficient for control for (ISI), then for every E > 0, there exists an E-optimal nonrandomized policy for (ISI) which depends on i, = (zo,uo,. . . ,uk- zk)only through y,(p; i,), i.e., has the form 71. =

( ~ o [ ~ ; ~ o ( ~ ; i. o. ,)Pl N , .- ~ [ P ; Y N - I ( P ; ~ N - ~ ) I ) .

(44)

Under (F', p'), (P, p), or (D, D), we may choose this &-optimal policy to have the simpler form

Proof Under (F', p+),(P, p), or (D, D), there exists an e-optimal, nonrandomized, Markov policy it = (Go,. . . ,fi,- ,) for (PSI) (Propositions 8.3 and 9.19). This policy it is &-optimalfor (ISI) by Proposition 10.3, and the second part of the proposition is proved. Assume (F-, E-) holds and let {en) be a sequence of positive numbers E , < o~ and E, .LO. Let it, = (fi;,, . . . , ,) be a sequence of nonwith randomized Markov policies for (PSI) exhibiting {en)-dominated convergence to optimality (Proposition 8.4). By Proposition 10.2 and the (F-, p-)

I,"=,

rN-

10.


assumption, we have

Since

we have

Let

E

> 0 be given and let n(p) be the smallest positive integer n for which

Define ,uk(p;y,) = fi;(P)(yk),k = 0,. . . ,N - 1. Then by Propositions 10.2 and 10.3, n given by (44)is an &-optimalnonrandomized policy for (ISI). Assume (N, R)holds. Consider the nonstationary stochastic optimal control model ( N P S I ) for which the initial state space is P(Y,), the initial control space is a singleton set {u,), the initial cost function is g,(q, u,) = 0 for every q~ P(Yo),and the initial transition kernel is given by t(dy,lq, u,) = q(dyo)for every q~ P(Yo). For k 2 0, the (k + 1)st state and control spaces, control constraint, cost function, and transition kernel are Y,, C, o,, g^,, and Ek(dyk+,I y,, u,) of (PSI), respectively. The discount factor is a and the horizon is infinite. By definition, the optimal cost for ( N P S I ) at q~ P(Y,) is

which, by Corollaries 9.1.1 and 9.5.2, is the same as JYo

J*(Yo)q(dyo).

Now ( N P S I ) has a nonpositive one-stage cost function, so, by Proposition 9.20, for each E > 0 there exists an &-optimal,nonrandomized, semi-Markov policy 77 = (P(4X Po(9;YO),Pl(9;Y I ) . . . .I. For fixed q E P(Yo),let ?(q) be the policy for (PSI)given by

Then

i.e., it(q) is weakly q-s-optimal for (PSI). By Proposition 10.3, the policy n defined by (44), where ,u,(p; y,) = p(cp(p);y,), is &-optimalfor (ISI). Q.E.D. The other specific results which can be derived for (ISI) from Chapters 8 and 9 are obvious and shall not be exhaustively listed. We content ourselves with describing the dynamic programming algorithm over a finite horizon. By Proposition 8.2, the dynamic programming algorithm has the following form under ( F + , F + ) or (FP,F-), where we assume for notational simplicity that (PSI) is stationary:

If the infimum in (47) is achieved for every y and k = 0,. . . ,N - 1, then there exist universally measurable functions fi,: Y -, C such that for every y and k = 0,. . . ,N - 1, f i , ( y ) ~Zi(y) and fi,(y) achieves the infimum in (47). Then it = (Go,. . . ,fi,- ,) is optimal in (PSI) (Proposition 8.5), so it is optimal in (ISI) as well (Proposition 10.3). If (F+,i?+) holds and the infimum in (47) is not achieved for every y and k = 0, . . . , N - 1, the dynamic programming algorithm (46)and (47)can still be used in the manner of Proposition 8.3 to construct an &-optimal, nonrandomized, Markov policy it for (PSI). We see from Proposition 10.3 that it is an &-optimalpolicy for (ISI) as well. In many cases, r],, ,(p; i, ,) is a function of r],(p; i,), u,, and z,, . The computational procedure in such a case is to first construct (Go,. . . ,fi,- ,) via (46) and (47), then compute yo = yo(p;io) from the initial distribution and the initial observation, and apply control uo = fio(yo).Given y,, u,, and z,.,, compute y,,, and apply control u,,, = fi,. ,(y,+ ,), k = 0,. . . ,N - 2. In this way the information contained in (p; i,) has been condensed into y,. This condensation of information is the historical motivation for statistics sufficient for control.

,

10.3 Existence of Statistics Sufficient for Control

Turning to the question of the existence of a statistic sufficient for control, it is not surprising to discover that the sequence of identity mappings on P(S)I,, k = 0,. . . ,N - 1, is such an object (Proposition 10.6). Although this

260

10.


represents no condensation of information, it is sufficient to justify our analysis thus far. We will show that if the constraint sets T, are equal to IkC, k = 0,. . . ,N - 1, then the functions mapping P(S)I, into the distribution of x , conditioned on ( p ;i,), k = 0,. . . ,N - 1, constitute a statistic sufficient for control (Proposition 10.5). This statistic has the property that its value at the (k + 1)st stage is a function of its value at the kth stage, u, and z,, [see (52)],so it represents a genuine condensation of information. It also results in a stationary perfect state information model and, if the conditional distributions can be characterized by a finite set of parameters, it may result in significant computational simplification. This latter condition is the case, for example, if it is possible to show beforehand that all these distributions are Gaussian.

,

10.3.1 Filtering and the Conditional Distributions of the States

We discuss filtering with the aid of the following basic lemma. Lemma 10.3 Consider the (ISI) model. There exist Borel-measurable stochastic kernels ro(dxolp;z,) on S given P(S)Z and r(dx1p;u, z) on S given P(S)CZ which satisfy

Jso ~ o ( ~ o ~ x o ) = ~ (Jsd Jzo ~ oro(~0l ) P;~o)~o(d~oxo)~(dxo)

Proof For fixed ( p ; u )P(S)C, ~ define a probability measure q on S Z by specifying its values on measurable rectangles to be (Proposition 7.28)

By Propositions 7.26 and 7.29, q(d(x,z)jp;u) is a Borel-measurable stochastic kernel on S Z given P(S)C. By Corollary 7.27.1, this stochastic kernel can be decomposed into its marginal on Z given P(S)C and a Borel-measurable stochastic kernel r(dxlp;u, z) on S given P(S)CZ such that (49) holds. Q.E.D. The existence of ro(dxolp;z,) is proved in a similar manner. It is customary to call p, the given distribution of x,, the a priori distribution of the initial state. After z, is observed, the distribution is "up-dated", i.e., the distribution of x , conditioned on z, is computed. The up-dated distribution is called the a posteriori distribution and, as we will show in Lemma,10.4 is just ro(dxolp;so). At the kth stage, k 2 1, we will have some a priori distribution p; of x , based on i,-, = (z,, u,, . . . ,u,-, ,z,- ,). Control

u,-, is applied, some z, is observed, and an a posteriori distribution of x, conditioned on (i,- u,- ,, z,) is computed. We will show that this distribution is just r(dxlp;; u,- 2,). The process of passing from an a priori to an a posteriori distribution in this manner is calledfiltering, and it is formalized next. Consider the function 7 : P(S)C + P(S) defined by

,,

,,

Equation (50) is called the one-stage prediction equation. If x, has an a posteriori distribution p, and the control u, is chosen, then the a priori distribution of x,, isT(p,, u,). The mapping f is Borel-measurable (Propositions 7.26 and 7.29). Given a sequence i, E I, such that i,, = (i,, u,, z,, ,), k = 0,. . . ,N - 2, and given p E P(S), define recursively

,

,

PO(P;io) = ro(dx01P; 201, ~ k + l ( p ; ~ k= + lr(dxlJf[~k(~;ik),uk];u,,zk+l), )

(51) = O,.

..

- 2.

(j2)

Note that for each k, p,: P(S)I, + P(S) is Borel-measurable. Equations (48)-(52) are called the filtering equations corresponding to the (ISI) model. For a given initial distribution and policy, they generate the conditional distribution of the state given the current information, as the following lemma shows, Lemma 10.4 Let the model (ISI) be given. For any peP(S), n = (po, . . . , p ~ ,)-E rI and 3, E .$,, we have Pk(n, ~ for P,(n,p) almost every i,, k

) [ ~ k ~ ik] S k = l P~(P ik)(Sk) ;

= 0,.

(j3)

. . ,N - 1.

Prooft We proceed by induction. For any SoE .92,and Zo - E .B,, we have from (51), (16), and (48), that

Equation (53) for k probability.

=0

follows from (54) and the definition of conditional

In this and subsequent proofs, the reader may find the discussion concerning conditional expectations and probabilities at the beginning of Scction 10.2 helpful.

262

10.


Assume now that ( 5 3 ) holds for k. For any _ I k € B,,, C,E B,, Z,+, E B , and 3,. € B Swe , have from (16), the induction hypothesis, Fubini's theorem, ( 5 0 ) , ( 5 2 ) , and (49)that

,

l)dPk+l(n,~)

P ~ + ~ ( Pi k ;> " k , z k +

~ i k ~ ~ k ~ ~ k lE€ z~ k + k 11 9 ~ k +

=

J JJ (iks!kl

ck

s k

+ 1~

( z k l+I " k ?

~

k

+

= P k + l ( n , ~ ) ( { i k ~ l k , ~ l k ~ C k , x k + lE

llxk,

u k ) ~ k ( d u k l ~ ik) ; dPk(n, P )

~ ~ + ~ , z ~ + ~ E Z ~ + ~ ) . ) . (j5)

It follows from ( 5 5 ) and the definition of conditional probability that Pk+l(n,~)[xk+l

E S ~ + ~= IP ~~+ ~~ ( P+; ~~~ +~I ) ( _ S ~ + ~ )

for Pk+,(n, p ) almost every i,, and the induction step is completed.

Q.E.D.

Proposition 10.5 Consider the ( I S I ) model and assume that U , ( x ) = C for every X E S and k = 0,. . . ,N - 1. Then the sequence [ p o ( p ; i o ) , . . . , p N - l ( p ; iN- ,)I defined by ( 5 1 ) and ( 5 2 ) is a statistic sufficient for control, and the resulting perfect state information model is stationary.

10.3

EXISTENCE OF STATISTICS SUFFICIENT FOR CONTROL

263

Proof Let Y , in Definition 10.6 be P(S), k = 0,. . . ,N - 1. We have already seen that the mappings p,: P(S)I, + P(S) are Borel-measurable, so (_Po,. . . ,pN-,) is a statistic. Condition (a) of Definition 10.6 is satisfied with r , = P ( S ) C , k = O ,..., N - 1 . define , For y E P(S), u E C and _Y E B,(,, Z ( Y , ~ , Y=) i z ~ Z l r [ d x l T ( y , u ) ; u , z ]I ~) ,

Y)

Note that Z(y, u, is the (y, u)-section of the inverse image of Y under a Borel-measurable function. The stochastic kernel

is Borel-measurable by Propositions 7.26 and 7.29, so the stochastic kernel

is Borel-measurable by the same propositions. It follows from Proposition 7.26 and Corollary 7.26.1 that i(dyljy,u) is a Borel-measurable stochastic and k = 0,. . . ,N - 2, kernel on P(S) given P(S)C. For n E Il,p E P(S), _YE a,(,, we have from Lemma 10.4

= i(Zl~k,

for Pk(n,p) almost every (B,, a,), where the expectations are with respect to P,+ ,(n,p). Thus (22) is satisfied. For TCEII,p~ P(S), and k = 0,. . . ,N - 1, we have from Lemma 10.4

264

10.


for P,(n, p) almost every (L,, ti,), where the expectations are with respect to Pk(n,p). The function g^: P(S)C -,R* defined by

(58)

8 7 , ti) = Js g(x,t i ) ~ ( d x ) is lower semianalytic (Proposition 7.48),and, by (57),g^ satisfies (23).

Q.E.D.

If the horizon is finite, then the transition kernel f and the one-stage cost function g^ defined by (56) and (58) can be substituted in the dynamic programming algorithm (46)-(47) to compute the optimal cost function j: for (PSI). The optimal cost function J z for (ISI) can then be determined from (41).Ifthe horizon is infinite, in the limit the dynamic programming algorithm (46)-(47) yields j* under (fi) and (D) and under (p)in some cases (Propositions 9.14 and 9.17). The determination of J* from 3'"is again accomplished by using (41). 10.3.2 The Identity Mappings Proposition 10.6 Let the model (ISI)be given. The sequence of identity mappings on P(S)I,, k = 0,. . . ,N - 1, is a statistic sufficient for control. Proof Let Y, in Definition 10.6 be P ( S ) I k ,k = 0,. . . ,N - 1, and let y, be the identity mapping on P(S)I,. Then ( y o ,. . . ,y,is a statistic. Condi= P ( S ) T k , k = 0,. . . ,N - 1. tion (a) of Definition 10.6 is satisfied with If _Y,+ E B ~ ( ~yk )E P(S)I,, ~ ~ +and E C k , we adopt the notation

,)

rk

-

- -

1 (X+i)(y, + tik)~k(7k)(dxk)

where p,(y,) is given by (51)and (52).By an argument similar to that used in Proposition 10.5, it can be shown that fk is Borel-measurable. For p~ P ( S ) , and ~EII,~,,EB~(,,,,+ , , k = 0,. . . , N - 2, we have from Lemma 10.4

= ?k(&+

l I ? k ? ~ ~ k ) ~

for P,(n, p ) almost every (?,,ti,),

SO

(22)is satisfied.

For k

= 0,.

. . ,N - 1, define lj,: P(S)I,C + R* by

By Proposition 7.48, g^, is lower semianalytic for each k. For p E P(S), n E IT, and k = 0,. . . ,N - 1, we have from Lemma 10.4

for P,(n,p) almost every (F,,E,), where the expectation is with respect to P,(n, p), SO (23) is satisfied. Q.E.D. The transition kernels ?, and one-stage cost functions g, defined by (59) and (60) can be used in the nonstationary version of the dynamic programming algorithm (46)-(47). See the discussion following Proposition 10.5.

Chapter 11

Miscellaneous

11.1 Limit-Measurable Policies

In this section we strengthen the results of Section 7.7 concerning universally measurable functions. In particular, we show that these results are still valid if limit-measurable functions (Definitions B.2 and B.3) are used in place of universally measurable functions. This allows us to replace all the results on the existence of universally measurable policies in Chapters 8 and 9 by stronger results on the existence of limit-measurable policies. We now rework the main results of Section 7.7 with the aid of the concepts and results of Appendix B. Proposition 11.1 Let X, Y , and Z be Borel spaces, D E Yx, and E E 2,. Suppose f :D -+ Y and g: E -t Z are limit-measurable and f(D) c E. Then the composition g is limit-measurable.

Proof

This follows from Corollary B.ll.l.

Q.E.D.

Corollary 11.1.1 Let X and Y be Borel spaces, let f : X -t Y be a function, and let q(dYlx)be a stochastic kernel on Y given X such that, for each x, q(dylx) assigns probability one to the point f ( x ) ~ Y. Then q(dylx) is limitmeasurable if and only iff is limit-measurable.

Proof

See the proof of Corollary 7.44.3.

Q.E.D.

11.1

267

LIMIT-MEASURABLE POLICIES

Proposition 11.2 Let X and Y be Borel spaces and let q(dy1x) be a stochastic kernel on Y given X. The following statements are equivalent: (a) The stochastic kernel q(dy1x) is limit-measurable. (b) For every B EB y ,the mapping I,,: X + R defined by

is limit-measurable. (c) For every Q E Y y, the mapping IbQof (1) is limit-measurable. Proof We prove (a) => (c) (b) =.(a). Suppose (a) holds and Q E Y y. = 8, o y, where y:X -+ P(Y) is given by Now y(x) = q(dylx) and BQ:P(Y)-t R is given by

We have assumed that y is limit-measurable, and 8, is limit-measurable by Proposition B.12. Therefore (c) holds. It is clear that (c) => (b). Suppose now that (b) holds. Then

Letting y and 8, be defined by (2) and (3), we have from Proposition 7.25

so q(dy1x) is limit-measurable.

Q.E.D.

Proposition 11.3 Let X and Y be Borel spaces and let f :XY -+ R* be limit-measurable. Let q(dylx) be a limit-measurable stochastic kernel on Y given X. Then the mapping I.:X R* defined by -+

is limit-measurable. Proof The mapping 6(x) = p, is continuous (Corollary 7.21.1), as is the mapping o: P(X)P(Y) -+ P(XY) defined by o(p, q) = pq, where pq is the

268

1 1.

product measure (Lemma 7.12). Suppose Q E 5FXuand f x € X,

MISCELLANEOUS

= ZQ.

For every

where y and QQ are given by (2) and (3). Since all the functions on the righthand side of (4) are limit-measurable, iis limit-measurable. It follows that i, is limit-measurable when f is a limit-measurable simple function. The extension to the general limit-measurable, extended real-valued function f is straightforward. Q.E.D. Corollary 11.3.1 Let X be a Borel space and let f :X-+ R* be limitmeasurable. Then the function Of :P ( X ) -, R* defined by

Q,(P)

=

Jf

dP

is limit-measurable. We have the following sharpened version of the selection theorem for lower semianalytic functions. Proposition 11.4 Let A' and Y be Borel spaces, D c XY an analytic set, and f : D -+ RXa lower semianalytic function. Define f * : proj,(D) -+ R* by

f *(x) = inf f(x, y). Y E D ~

The set

I = {xE projx(D)I for some y , Dx, ~ f(x, y,)

= f *(x))

is limit-measurable, and for every E > 0 there exists a limit-measurable function 9: projx(D) -+ Y such that Gr(cp) c D and for all x E projx(D)

Proof The proof is the same as in Proposition 7.50(b), except that at the points where Corollaiy 7.44.2 is invoked to say that the composition of analytically measurable functions is universally measurable, we use ProposiQ.E.D. tion 11.1 to say that the composition is limit-measurable.

By the remark following Corollary B.ll.l, we see that I and the selector obtained in Proposition 11.4 are in fact 8:-measurable. This remark further suggests that the constructions in Chapters 8 and 9 of optimal and E-optimal

11.2

ANALYTICALLY MEASURABLE POLICIES

269

policies can be done more carefully by keeping track of the minimal 9'; with respect to which policies and costs are measurable. We do this to some extent in the next section, but do not pursue this matter to any great length. Propositions 11.1-1 1.4are sufficient to allow us to replace every reference to a "(universally measurable) policy" in Chapters 8 and 9 by the words "limit-measurable policy." It does not matter which class of policies is considered when defining J ; and J*; the proof of Proposition 8.1 together with Proposition 11.5 given below can be used to show that these functions are determined by the analytically measurable Markov policies alone. Corollary 11.1.1 tells us that the nonrandomized limit-measurable policies are just the set of sequences of limit-measurable functions from state to control which satisfy the control constraint (cf. Definition 8.2). This fact and Proposition 11.2 are needed for the proof of the limit-measurable counterpart of Lemma 8.2. From Proposition 11.3 we can deduce that the cost corresponding to a limit-measurable policy is limit-measurable (cf. Definitions 8.3 and 9.3). This fact was used, for example, in proving that under (F-) a nonrandomized, semi-Markov, &-optimalpolicy exists (Proposition 8.3). Proposition 11.4 allows limit-measurable &-optimaland optimal selection. The &-optimal selection property for universally measurable functions is used in practically every proof in Chapters 8 and 9. The exact selection property is used in showing the existence under certain conditions of optimal policies (Propositions 8.5, 9.19, and 9.20). 11.2 Analytically Measurable Policies

Some of the existence results of Chapters 8 and 9 can be sharpened to state the existence of &-optimalanalytically measurable policies. This is due to Proposition 7.50(a) and the following propositions. Proposition 11.5 is the analog of Corollary 7.44.3 for universally measurable policies and of Corollary 11.1.1 for limit-measurable policies. Proposition 11.5 Let X and Y be Bore1 spaces, let f : X -, Y be a function, and let q(dy1x) be a stochastic kernel on Y given X such that, for each x, q(dy(x)assigns probability one to the point f ( x ) ~ Y. Then q(dylx) is analytically measurable if and only iff is analytically measurable.

Proof

We sharpen the proof of Corollary 7.44.3. Let y(x) = q(dylx)and that g = 6 f and f = 6-' y. NOW6 is a homeomorphism from Y to Y = jp,ly E Y), so S and 6-': F + Y are both Borel-measurable. Iff is analytically measurable and C E ,97,(,,,then

6(y) = py,

SO

0

0

~ - ' ( c= )f - ' [ 6 - ' ( C ) ] ~ d ~

270

11. MISCELLANEOUS

because 6 - '(C) E B y . If y is analytically measurable and B Eg,, then

f -l(B) = y-l[G(B)] e d x because 6(B)E Bp(,).

Q.E.D.

Proposition 11.6 Let X and Y be Bore1 spaces and let q(dy1x) be a stochastic kernel on Y given X. The following statements are equivalent: (a) The stochastic kernel q(dy1x) is analytically measurable. (b) For every B E B y , the mapping I.,: X + R defined by is analytically measurable. Proof Assume (a) holds and define y(x) = q(dy1x). Then for B E B ~ , C E B R ,and 8,: P(Y) + R defined by (3), we have

because 0, '(C) E BP(,) (Proposition 7.25). Therefore (b) holds. If (b) holds, we can show that (a) holds by the same argument used in the proof of Proposition 11.2. Q.E.D. We know from Corollary B.ll.l that the composition of analytically measurable functions need not be analytically measurable, so the cost corresponding to an analytically measurable policy for a stochastic optimal control model may not be analytically measurable. To see this, just write out explicitly the cost corresponding to a two-stage, nonrandomized, Markov, analytically measurable policy (cf. Definition 8.3). A review of Chapters 8 and 9 shows the following. Proposition 8.3 is still valid if the word "policy" is replaced by "analytically measurable policy," except that under (F-) an analytically measurable, nonrandomized, semiMarkov, &-optimalpolicy is not guranteed to exist. However, an analytically measurable nonrandomized E-optimal policy can be shown to exist if g I 0 [B12]. The proof of the existence of a sequence of nonrandomized Markov policies exhibiting [&,)-dominated convergence to optimality (Proposition 8.4) breaks down at the point where we assume that a sequence of one-stage policies {p", exists for which

This occurs because T,--l(J,) may not be analytically measurable. In the first sentence of ~ r o ~ o 4 t i o9.19, n the word "policy" can be replaced by "analytically measurable policy." The &-optimal part of Proposition 9.20 depends on the (F-) part of Proposition 8.3, so it cannot be strengthened in

11.3

MODELS WITH MULTIPLICATIVE COST

271

this way. Under assumption (N),an analytically measurable, nonrandomized, &-optimalpolicy can be shown to exist [B12], but it is unknown whether this policy can be taken to be semi-Markov. The results of Chapters 8 and 9 relating to existence of universally measurable optimal policies depend on the exact selection property of Proposition 7.50(b). Since this property is not available for analytically measurable functions, we cannot use the same arguments to infer existence of optimal analytically measurable policies.

11.3 Models with Multiplicative Cost In this section we revisit the stochastic optimal control model with a multiplicative cost functional first encountered in Section 2.3.4. We pose the finite horizon model in Borel spaces and state the results which are obtainable by casting this Borel space model in the generalized framework of Chapter 6. This does not permit a thorough treatment of the type already given to the model with additive cost in Chapters 8 and 9, but it does yield some useful results and illustrates how the generalized abstract model of Chapter 6 can be applied. The reader can, of course, use the mathematical theory of Chapter 7 to analyze the model with multiplicative cost directly under conditions more general than those given here. We set up the Bovel model with multiplicative cost. Let the state space S, the control space C, and the disturbance space W be Borel spaces. Let the control constraint U mapping S into the set of nonempty subsets of C be such that

is analytic. Let the disturbance kernel p(dwlx,u) and the system function f : SC W + S be Borel-measurable. Let the one-stage cost function g be Borelmeasurable, and assume that there exists a b E R such that 0 I g(x, u, w) I b for all X E S, u E U(X),W E W. Let the horizon N be a positive integer. In the framework of Section 6.1, we define F to be the set of extended real-valued, universally measurable functions on S and F* to be' the set be I the set of functions in F which are lower semianalytic. We let & of universally measurable functions from S to C with graph in T. Define H: SCF + [0, co] by

wherewedefineO.co= c o . O = O . ( - a ) = ( - a ) . O = O . W e t a k e J , : S - t R * to be identically one. Then Assumptions A.l -A.4, F.2, and the Exact Selection Assumption of Section 6.1 hold. (Assumption A.2 follows from Lemma 7.30(4) and Propositions 7.47 and 7.48. Assumption A.4 follows from Proposition

11 .

272

MISCELLANEOUS

7.50.) From Propositions 6.l(a), 6.2(a), and 6.3 we have the following results, where the notation of Section 6.1 is used. Proposition 11.7 In the finite horizon Bore1 model with multiplicative cost, we have and for every E > 0 there exists an N-stage E-optimal (Markov) policy. A policy n* = (y;, . . . ,y$- ,) is uniformly N-stage optimal if and only if ( T , ; T ~ - ~ -)(Jo) ' = TN- (J,), k = 0,. . . ,N - 1, and such a policy exists if and only if the infimum in the relation T ~ '(J,)(x) +

=

inf H[x, u, Tk(~,,)] uEU(X)

is attained for each x~ S and k = 0,. . . ,N - 1. A sufficient condition for this infimum to be attained is for the set

, k to be compact for each ~ E S~, E Rand

= 0,.

. . ,N

- 1.

Appendix A

The Outer Integral

Throughout this appendix, (X, 98,p) is a probability space. Unless otherwise specified, f , g, and h are functions from X to [ - co,co]. Definition A.l

Iff 2 0, the outer integral off with respect to p is defined

by

Iff is arbitrary, define

where f'(x)=max{O,,f(x)),

f-(x)=maxjO,-f(x)),

andweset co - co = co. Lemma A.l such that

I f f 2 0, then there exists a .%-measurable g with g 2 f ,

APPENDIX A

Proof

Choose g, 2 f,g, 98-measurable, so that

We assume without loss of generality that g , 2 9 , 2 . . . . Let g Then g 2 f , g is 93-measurable, and (3) holds. Q.E.D.

= limn,,

g,.

Lemma A.2 I f f 2 0 and h 2 0, then

If either f or h is %-measurable, then equality holds in (4). Proof Suppose 9 , 2 f , 9 , 2 f , g, and 9 , are 93-measurable, and I* f dp = J g , dp, J*h dp = 19, dp. Then 9 , g2 2 f + h and (4) follows from (1). Suppose h is W-measurable and Shdp < a.[If Shdp = oo, equality is g, where g is 93-measurable and easily seen to hold in (4).] Suppose f + h I

+

Then f 5 g

-

h and g - h is 5?-measurable, so

which implies

Therefore equality holds in (4).

Q.E.D.

We provide an example to show that strict inequality can occur in (4), even if f + h is B-measurable. For this and subsequent examples we will need the following observation: For any E c X,

where p*(E) is p-outer measure defined by

and x, is the indicator function of E defined by

275

THE OUTER INTEGRAL

To verify (5), note that if 1, I g and g is .93-measurable, then { x ] ~ ( x2) 1) is a B-measurable set containing E and consequently

Definition A.l implies

On the other hand, if {B,) is a sequence of B-measurable sets with E c B, and p(B,) l p*(E), then p(n'i,"=,B,) = p*(E). By construction, xn:,, B, 2 xE. But xn;= , B , is 63-measurable, and

The reverse of inequality (6) follows. Note that the preceding argument shows that for any set E, there exists a set B e.93 such that E c B and p(B) = p*(E). EXAMPLE 1 Let X = [O, 11, let .93be the Bore1 o-algebra, and let p be Lebesgue measure restricted to 93.Let E c X be a set for which p*(E) = p*(X - E) = 1 (see [HI, Section 16, Theorem El). Then

and strict inequality holds in (4). Lemma A.2 cannot be extended to (possibly negative) bounded functions, even if h is .%-measurable, as the following example demonstrates. EXAMPLE 2 Let (X,93,p) and E be as before. Let f Then j*(f

+ h)dp = j * 21, dp = 2,

Lemma A.3 (a) I f f I y, then j*fdp I j*g dp. (b) If E > 0 and J' g f + r , then

=

zE - zx- E ,

h = 1.

276

APPENDIX A

(c) If j* f dp < co or j* f - dp < co,then +

J*(-.f)dp

=

-J*f

(8)

dp.

(d) If A, B E 93 are disjoint, then for any f J*

~~~~f d~ = J*

dp + J* iB/dp.

XJ

(9)

(e) If E c X satisfies p*(E) = 0, then for any f J * f d p = J*%X-Ef d ~ . (f) If p*(jxl f(x) = co)) > 0, then for every g, S*(g + f ) d p = m. (g) If p*({xlf(x) = - co)) > 0, then for every g either j*(g + f ) d p or j*(g + f ) d p = - oo. Proof

(a) Iff I g, then f

+

=

cc

5 g + and f - 2 g-. By (I),

J*f'dp5J*gidp,

J*f-dpt S*gdp.

The result follows from (2). (b) In light of (a), it remains only to show that

For g, 2 f +,g, 93-measurable, and J*f

+

dp = J B ~ ~ P ?

we have

J*(f For g2 2 ( f

+ E)-,

+ E)'

dp 5 Jgl dp

+

E =

J* f dp + E . +

g2 .%-measurable, and J*(f

+ &I-

dp = 1 9 2 d ~ ,

we have .q2

+ E 2 (J' + E)- + E = maxf f ' -

-E,O)

+ E >

J'-,

SO

E

+ J*(f + E)-

dp = E

+ Jg2 dp = Jig2 + ~ ) d p J * f

Combine (11) and (12) to conclude (10).

dp.

(12)

THE OUTER INTEGRAL

(c) We have

where the assumption that J*f dp < co or J*f - dp < EI is necessary for the next to last equality. (d) Suppose f 2 0. Let g be a W-measurable function with g 2 x,, f and +

,

Now suppose 9, P

zAf , g2 2 X B f are W-measurable and

Combine (13) and (14) to conclude (9) for f 2 0. The extension to arbitrary f is straightforward. (e) Suppose f 2 0. Choose BEW with p(B) = p*(E) = 0, B 2 E. By (d),

Hence J"*f dp = f dp. The extension to arbitrary f is straightforward. (f) We have (g + f )+(x) = oo iff (x) = oo,so that

+

Hence J*(g f ) + dp = co,and it follows that J*(g + f ) d p = co. (g) Consider the sets E = f(x) = - EI) and E, = (xif(x) = g(x) < co). If p*(E,) = 0, then

.[XI

p*(E - E,)

= p*(E - E,)

+ p*(E,) 2 p*(E) > 0.

-

oo,

278

APPENDIX A

Since we have f ( x ) + g ( x ) = co for X E E - E,, it follows from ( f ) that J*(g + f ) d p = co. If p*(E,) > 0, then p*((xl(g f ) - ( x ) = co))2 p*(E,) > 0 and hence, by (f), ("(9 f ) - dp = m. Hence, if [*(g + f)' dp = co, then J*(g + f ) dp = m, while if J*(g + f ) + dp < co, then J*(g + f ) dp = - m. Q.E.D.

+

+

The bound given in ( 7 ) is the sharpest possible. To see this, let f be as defined in Example 2, g = f + 1, and E = 1. Despite these pathologies of outer integration, there is a monotone convergence theorem, which we now prove. Proposition A . l f ,t

If { f , ) is a sequence of nonnegative functions and

f then >

If { f , )is a sequence of nonpositive functions and f,J f , then

Proof We prove the first statement of the theorem. The second follows from the first and Lemma A.3(c). Assume f, 2 0 and f ,f f . Let {g,) be a sequence of B-measurable functions such that g, 2 f, and

If, for some n, {g,dp = {*f,dp = m, then (15) is assured. If not, then for every n,

Suppose (17) holds for every n and for some n,

,

,

p(.Ix)gn(x)> gn+ I ( x ) ) )> 0.

Then since g, + 2 f,+ 2 f,, we have that

defined by

satisfies g, 2 g 2 f, everywhere and ij < g, on a set of positive measure. This contradicts (16). We may therefore assume without loss of generality that (17)holds and g, 5 g, . . . . Let g = lim,,, g,. Then g 2 f and

But f , 5 f for every n, so the reverse inequality holds as well.

Q.E.D.

279

THE OUTER INTEGRAL

One might hope that if {f,) is a sequence of functions which are bounded below andf, f f , then (15) remains valid. This is not the case, as the following example shows. EXAMPLE 3 Let X = [O, I), W be the Bore1 o-algebra, and p be Lebesgue measure restricted to 98. Define an equivalence relation on X by

x

-

-

y o x - y is rational.

Let Fo be constructed by choosing one representative from each equivalence class. Let Q = {q,, q,, . . .) be an enumeration of the rationals in [O,1) with qo = 0 and define Then F o , F,, . . . is a sequence of disjoint sets with

U:Z~

If for some n < oo,we have p*(U,",, F,) < 1, then E = Fk contains a 98-measurable set with measure 6 > 0. For k = 1,. . . ,n - 1, let q, = vk/sk, where r, and s, are integers and vk/skis reduced to lowest terms. Let ( p , , p2 ,. . .) be a sequence of prime numbers such that max s, < p , < p2 < .

lsksn-l

'

Then the sets E, E + p ; [mod I], E + p;' [mod 11, . . . are disjoint, and by the translation invariance of p, each contains a 9-measurable set with measure 6 > 0. It follows that [O,1) must contain a W-measurable set of infinite measure. This contradiction implies

for every n. Define

Then f,T 0, but (5) and (19) imply that for every n

By a change of sign in Example 3, we see that the second part of Theorem A.l cannot be extended to functions which are bounded above unless additional conditions are imposed. We impose such conditions in order to prove a corollary.

280

APPENDIX A

A.l.l xr=Corollary < { f,) =, E,

m. Let

lim fn n-r

Let {E,) be a sequence of positive numbers with be a sequence with

=f,

w

Then

Proof F r o m ( 2 0 ) w e h a v e l i m n + , f ~ = f + a n d l i m , ~ , f ~ = f - . N o w infksnf k I f; s f - and inf,,, f; f f - as n --+ co. By Proposition A.l, J* f - d p = lirn J* inff;dp$ n+w

k2n

lim J*

lim J* f ; d p i J *

f-dp,

n-r m

/;

dp = J* f - dp.

n-w

Let A = {xif(x) and (e) imply

= -a

) . Ifp*(A) = 0, then (21),(22),(24),and Lemmas A.3(b)

Combine (26) and (27) to conclude (25). If p*(A) > 0, then S*f - dp = - m and (26) will imply (25) provided that

and lim sup J*

f.i dp < ca.

n-r m

Conditions (21) and (24) imply (28). Conditions (21)-(23) imply for every XEX

THE OUTER INTEGRAL

J*f: d p ~ 2 ~ . + J * f ~ l d ~

and

The finiteness of

I,"=, .zk and (24)imply (29).

Q.E.D.

Appendix B

Additional Measurability Properties of Bore1 Spaces

This appendix supplements Section 7.6. The notation and terminology used here is the same as in that section and, in most cases, is defined in Section 7.1.

B.l

Proof of Proposition 7.35(e)

Our first task is to give a proof of Proposition 7.35(e). To do this, we introduce the space N* = {l,2 , . . .) u {a)with the topology induced by the metric

where we define l/a = 0. Let JV* = N*N*. . . with the product topology. The space .Af of sequences of positive integers is a topological subspace of .N*. The space JV* is compact by Tychonoff's theorem, while JV is not. If ( X ,9 )and ( Y ,9)are paved spaces, we denote by 9'2 the paving of XY:

Proposition B.l Let ( X , P ) be a paved space and .f the collection of compact subsets of .N*.Then the projection on X of a set in Y(.P.Y/C)is in Y(9). Conversely, every set in Y(.9)is the projection on X of some set in

[(9.,f)o]d.

ADDITIONAL MEASURABILITY PROPERTIES OF BOREL SPACES

283

Proof Let S be a Suslin scheme for .Y,X. Then for every s E C, S(s) has the form S(s) = Sl(s)S2(s), where S,(S)E and S2(s)E X . Now

SO

where

Since each S,(s) is compact, we have

Define a Suslin scheme R for 9 by

otherwise.

lizr Then

so projx[N(S)] E Y ( 9 ) . For the second part of the proposition, suppose S is a Suslin scheme Define a Suslin scheme R for I f by for 9'.

For fixed z , E .A'",we have

ns,zo R(s)

=

{z,),

SO

284

APPENDIX B

Therefore,

and it remains only to show that

If we can show that

where C, is the set of elements in C having k components, then (3) will follow. Let x E X and z, = ((7,[',. . . ) E N * be given. Suppose

We see from (2) that zo E A'" and ( x ,z,) E r)s,,,[S(s)R(s)], so for every k 2 1, (x,zo) S K Y , . . . ,c ~ o ) R (. ~. .Y,i;). , hi^ implies (x,Z O ) E .,kr~(s)~(s)~, and

nr=us

On the other hand, if (x,z,)E OF=, ~s,,,[S(s)R(s)],then for each k 2 1, ( x ,z,) E ~ s , x k [ S ( ~ ) ~This ( ~ ) can ] . happen only if z, E N and ( x ,z,) E S([:,. . . ,[~O)R([:, . . . ,it).Therefore,

which proves the reverse of set containment (5). Equality (4)follows. Q.E.D.

285


is a paved space, Y is another space, and Q c Y, we define a If (X,,!?) paving of X Y by YQ Lemma B.l

=

(PQlPeY).

Let (X, 9 ) and ( Y ,A?) be paved spaces. Then:

(a) Y(,P)Q = Y(YQ) for every Q c Y ; (b) Y(9)2? c 9(.9'2). Proof

Part (a) is trivial and part (b) follows from (a).

Q.E.D.

We are now in a position to prove part (e) of Proposition 7.35. Proposition B.2 Let (X, .9) be a paved space. Then Y(.P) = Y [Y(Y)].

Proof

In light of Proposition 7.35(d), we need only prove

Let .M* and .X be as in Proposition B.1. If AeY[9(.9')], then by the second part of Proposition B.l, A = projx(B) for some set Be([Y(P)X],),. By Lemma B.l(b) and Proposition 7.35(b) and (c), we have

The first part of Proposition B.l implies that A = projX(B)eY(9)and (6) follows. Q.E.D. B.2 Proof of Proposition 7.16 In Proposition 7.16 we stated that Borel spaces X and Y are Borelisomorphic if and only if they have the same cardinality. A related result is that every uncountable Borel space is Borel-isomorphic to every other uncountable Borel space. We used the latter fact in Proposition 7.27 to assume without loss of generality that the Borel spaces under consideration were actually copies of (O,l], we used it in Proposition 7.39 to transfer a statement about JV to a statement about any uncountable Borel space, and we will use it again in Proposition B.7 to allow our treatment of the limit o-algebra to center on the space .M. The proofs of Proposition 7.16 and Corollary 7.16.1 depend on the following lemma, which is an immediate consequence of Propositions 7.36 and 7.37. The reader may wish to verify that these propositions depend only on Propositions 7.35, B.l, and B.2, so no circularity is present in the arguments. Lemma B.2 Let X be a nonempty Borel space. There is a continuous function f' from .M onto X.

286

APPENDIX B

Define A! to be the set of infinite sequences of zeroes and ones. We can regard A as the countable product of copies of {O, 1 ) and endow it with the product topology, where { O , l ) has the discrete topology. By Tychonoff's theorem, A! is compact with this topology. It is also metrizable as a complete separable space. Our proof of Proposition 7.16 consists of three parts. We show first that every uncountable Borel space contains a Borel subset homeomorphic to A!, we show second that every uncountable Borel space is isomorphic to a Borel subset of A, and we show finally that these first two facts imply that every uncountable Borel space is isomorphic to A.

Lemma B.3 Let X be an uncountable Borel space. There exists a compact set K c X such that A!and K are homeomorphic. Proof Let J ' : N + X be the continuous, onto function of Lemma B.2. For each x E X , choose an element z, E JV such that x = f(z,). Let S = {z,lx E X ) , SO that f is a one-to-one function from S onto X. For ~ E S , if possible choose an open neighborhood T ( z ) of z such that S n T ( z ) is countable. Let R be the set of all Z E S for which such a T ( z ) can be found. Since separable metrizable spaces have the Lindelof property, there exists a countable subset R' of R such that T ( z )= T ( z ) ,SO

U,,,

uZsRf

and R is countable. Since S is uncountable, S - R must be infinite. Furthermore, if z E S - R , then every open neighborhood of z contains infinitely many points of S - R. Let d be a metric on A'" consistent with its topology for which ( M ,d ) is complete. For 3~ M , the closed sphere of radius r centered at 7 is the set ( z E .NId(z,?) 5 1.1.The interior ofthis sphere, denoted Int { Z E .N"Id(z,z)I u), is the set { z ~ . N I d ( z , ? < ) Y). Let z(0) and z(1) be distinct points in S - R. Then f [z(O)] f f [ z ( l ) ] , so there exist disjoint open neighborhoods U and V o f f [z(O)] and f [ z ( l ) ] respectively. Let S(0) and S ( l ) be disjoint closed spheres of radius no greater than one centered at z(0) and z(1) and contained in f - ' ( U ) and f - ' ( V ) respectively. We have that f [S(O)] and f [ S ( l ) ] are disjoint. Note also that for every z E (S - R ) n Int S(O),every open neighborhood of z contains infinitely many points of ( S - R ) n Int S(O), and the same is true of S(1). By the same procedure we can choose distinct points z(0,O) and z ( 0 , l ) in ( S - R ) n Int S(0) and distinct points z(1,O) and z ( 1 , I ) in ( S - R ) n Int S(1), and we can also choose dirjoint closed spheres S(O,O), S(0, I ) , S(1,O) and S ( 1 , l ) of radius no greater than 5 centered at z(0, O), z(0, I ) , z(1,O) and z(1, I), respectively, so that f[S(O, 0 ) ] , f [S(O,I ) ] , f [S(1,O)] and ,f [S(1, I ) ] are all disjoint. We can choose these spheres so that S(0,O) and


287

S ( 0 , l ) are contained in S(O), while S(1,O) and S ( 1 , l ) are contained in S(1). At the kth step of this process, we choose a collection of disjoint closed spheres S ( p , , . . . , p,) of radius no greater than l l k centered at distinct points z ( p l , . . . , p,) in S - R, where each pj is either zero or one. Furthermore, we can choose the spheres so that for each ( p , , . . . , p,- ,)

(i)

(1

=@,

f[s(~l>.~.>pk-l>o)]nf[s(~l>...>pk-l>l)] -

I

.

.

-

~,=0,1.

For fixed m = ( p , , p,, . . .) E A', the sets { S ( p , , . . . , p,)} form a decreasing sequence of closed sets with radius converging to zero, so { z ( p l , . . . , p,)) is S ( p l , . . . , p,). Cauchy and thus has a limit ~ ( mE ) We show that y:.&' -+ ./V is a homeomorphism. If ( p , , p , , . . .) and (v,, v,, . . .) are distinct elements of A', then for some integer k, we have pk # V , . Since y ( p l , p 2 , . . .) E S ( p l , . . . ,p,), y ( v l , v 2 , . . .) E S ( v I ,. . . ,vk), and S ( p , , . . . , p,) is disjoint from S(v,, . . . , v,), we see that cp(p,, p,, . . .) # cp(v,, v,, . . .), so y is one-to-one. To show cp is continuous, let {m,) be a sequence converging to m E A'. Choose E > 0 and let k be a positive integer such that 2/k < E. There exists an n such that whenever n 2 Ti, the elements m, and m = ( p , , p,, . . .) agree in the first k components, so both cp(m,) and ~ ( mare ) in S ( p l , . . . , p,). This implies d(cp(m,), cp(m))I 2/k < E, so cp is continuous. To show that y-' is continuous, it suffices to show that q ( F ) is closed in q(&) whenever F is closed in A'. This follows from the fact that A' is compact and cp is continuous. Define M , c .M to be the compact homeomorphic image of A' under cp. We now show that f : M I-+ X is a homeomorphism. To see that f is one-to-one, choose distinct points z and 2 in M I . Then there exist distinct points m = ( p , , p,, . . .) and f i = ( f i , , fi,, . . .) in .Asuch that z = cp(m) and z^ = y ( f i ) . For some k, we have p, # fi,, so by (i), f [ S ( p o , .. . ,pk)] n f [S(fio,. . . , fi,)] = @. Since z E S ( p O ,. . . , p,) and 2 E S(fio, . . . , fi,), we see that f( z ) # f(B),so f is one-to-one. Just as in the case of cp, the continuity of f follows from the fact that f is continuous and has a compact domain. ) a compact subset of X homeomorphic to .A. Q.E.D. The set K = f ( M I is

Or=

-'

Lemma B.4 Let X be an uncountable Borel space. There exists a Borel subset L of .Asuch that X and L are Borel-isomorphic.

Proof By definition, X is homeomorphic to a Borel subset B of a complete separable metric space Y. By Urysohn's and Alexandroff's theorems (Propositions 7.2 and 7.3), Y is homeomorphic to a G,-subset of the Hilbert cube .X, so B and hence X are homeomorphic to a Borel subset of I f . It suffices then to show that 2 is Borel-isomorphic to a Borel subset of .A'. The idea of the proof is this. Each element in Yf is a sequence of real numbers in [0, 11. Each of these numbers has a binary expansion, and bv

288

APPENDIX B

mixing all these expansions, we obtain an element in A . Let us first define $ :[O,1] -, A which maps a real number into a sequence of zeroes and ones which is its binary expansion. It is easier to define $-I, which we define on A, u {(O,O,0, . . .)), where

Al = {(pl, p 2 , . . .)EA l p k = 1 for infinitely many k). It is given by

and it is easily verified that is one-to-one, continuous, and maps onto [O, 11. Since A - 4, is countable, the domain of I)-' is a Borel subset of A, and Proposition 7.15 tells us that )I is a Borel isomorphism. Since we have not proved Proposition 7.15, we show directly that )I is Borel-measurable. Consider the collection of sets k=1,2,..., k = 1,2,. . . .

R ( k ) = { ( ~ 1 , ~ 2 , . . . ) ~ A [ ~ k = o ) >

( k ) = { ( p 1 p 2 ..

)

A=1

These sets form a subbase for the topology of A , so by the remark following Definition 7.6, we need only prove that $-'[R(k)] and $-'[k(k)] are Borelmeasurable to conclude that IC/ is. Since one of these sets is the complement of the other, we may restrict attention to tj-'[R(k)]. Remembering that the domain of I / - ' is A, u {O,O,0, . . .)), we have

and k- 1

{j = f1 ~ ~ ( ~ l , p 2 , . ) ~ ~ ~ , p k = O = (PI,. .U ., ~

k 1)-

{x+

j= 1

310<x 0 for which S,(a,) c X - K. For B E Y , ( A ) ,we have d(a,, B) I max d(a,B) < E, acA

which implies B n S,(a,) # @ and B E 2' - 2K. Therefore Y , ( A ) c 2' - 2K, and 2' - 2K is open. Having thus shown that the sets 2G and 2' - 2K are open in (2', p) when G is open and K is closed, we must now show that given any open subset 9 of (2X,p) and any nonempty A E 9,we can find open sets G,, G,, . . . ,G, and closed sets K , , K,, . . . ,K , in X for which Since 9 is open in (2', p), there exists E > 0 such that Y , ( A ) c 9.Since A is closed in the compact set X, there exist points { x , , . . . ,x,} in A such that A c U;=, S,,,(x,). Let and ~ ' since for each k, A n S,,,(xk) # @, we have By construction, A E ~ and, A E 2' - 2Kk.Therefore Suppose B is another set in 2G1n (2' - 2K1)n . . . n (2' - 2Kn).The fact that B E 2G implies max d(b,A ) < E. beB

306

APPENDIX

c

If for some a, E A we had d(ao,B) 2 r, then we would also have S,(ao) c X - B. But for some X,E A, a0 E S,,,(x,) and this would imply in succession S,,,(x,) c X - B, B c K,, and B $ 2' - 2Kk.This contradiction shows that max d(a,B) < E. ~ E A

Inequalities (11) and (12) establish that p(A, B) < r, and as a consequence 2G1n (2' - 2 K 1n ) . . - n (2' - 2"',)

S,(A)

9.

Q.E.D.

If a cover of a space contains no finite subcover, we say the cover is essentially injnite. To show that (2', p ) is compact when X is compact, we must show that no essentially infinite open cover of 2' exists. As a consequence of the following lemma, this will be accomplished if we can show that the subbase $9 u K contains no essentially infinite cover. We remind the reader that a topological space in which every open cover has a countable subcover is called Lindelof, and in metrizable spaces this property is equivalent to separability. Lemma C.l Let Q be a Lindelof space and let Y be a subbase for the topology on a. If there exists an essentially infinite open cover of 0, then there exists one which is a subset of Y. Proof Let 93 be the base for the topology on Q constructed by taking finite intersections of sets in Y and let 9 be an essentially infinite open cover of R. Each C E @has a representation C = UaEA(,,B,,where B , E ~for every a E A(C). The collection U,,,(~,la E A(C)) is an essentially infinite open cover of Q, and, by the Lindelof property, it contains a countable, essentially infinite, open subcover 9 = {B,, B,, . . .). Each B, has a repreS k j ,where S k j €Y ,j = 1,. . . ,n(k).If for each j the cover sentation B, = 9j= { S l j B2,B3,. , . .} is not essentially infinite, then there exists a finite subcollection Bj which also covers Q. But then (B,) v

[lu

(lj - (sIj))] C 9

j= 1

is a finite subcover of Q. This contradiction implies that for some index j,, the cover gj0is essentially infinite. Denote R , = SljO.In general, given R,, R,, . . . ,R, in 9 such that B, c R,, k = 1,. . . ,n, and ( R , , R,, . . . ,R,, B,,,, B,,,,. . .) is an essentially infinite open cover of Q, we can use the preceding argument to construct R,+, E Y for which B,,, c R,, and ( R 1 ,R z , . . . ,R,, R,+ ,,BE+,,Bn+3,.. .) is an essentially infinite open cover of Q. The collection { R , ,R,, . . .j is an essentially infinite open cover conQ.E.D. tained in Y .

,

Proposition C.2 Let ( X , d ) be a compact metric space and p the Hausdorff metric on 2'. The metric space (2', p ) is compact.

THE HAUSDORFF METRIC AND THE EXPONENTIAL TOPOLOGY

307

Proof We first show that (2', p) is separable. Since ( X , d ) is compact, it is separable. Let D be a countable dense subset of X and let Let 9 consist of finite unions of sets in '3. Then 9 is countable and, as we now show, is dense in (2',p). Given A E ~ ' and E > 0, choose a positive integer n satisfying 2/n < E. The collection of sets { S l j , ( x ) l xD~) covers the ~ which also compact set A, so there is a finite subcollection { S , d x ) l x F) covers A and which satisfies SlIn(x)n A # @ for every X E F. The set B = Ux,,Slj,(x) is in 9 and satisfies p(A, B) < E . As a result of Proposition C.l, Lemma C.l, and the separability of (2', p), to show that (2',p) is compact we need only show that every open cover of 2' which is a subset of 99 u contains a finite subcover of 2'. Thus let (Gala€A ) be a collection of open sets and { K & ~ EB) a collection of closed sets in X , and suppose

nDEB

Define the closed set KO = K D .By definition, KO$ U D , B ( 2 X- 2Kfl),so KOE U r e A 2 GThus . for some a,€ A, we have KOE ~ ~i.e.,~KO Oc,G,,. This means that X - G,, c X - K O = ( X - K,),

U

DEB

and since X - G,, is compact, there exists a finite set {P1,P2,.. . ,p,) c B for which n

To complete the proof, we show 2'

= 2G=0 y

rk:

I

U (2X - 2K0k) 1

.

If C E 2', then either C c Gro,in which case C E 2'.0, or else C n ( X - G,,) # @. In the latter case, (13) implies that for some k, C n ( X - KBk)# @, i . e . , C ~ 2 ' - 2 ~ ~ ~ .Q.E.D.

We now develop some convergence notions in (2',p). Let (A,} be a sequence of sets in 2'. Define

308

APPENDIX

c

Forexample,ifX = [-1,1]andA, = {(-l)"),wehavelim,,,A, = {-1,l) and b,,, A, = @. If X = [- 1,1] and A, = [ - lln, lln], we have

lim A,

=

n+ m

lim A,

=

(0).

n+ m

-

-

Clearly we have lim,,, A, c limn,, A,. It is also true that limn,, A, and limn,, A, are closed. To see this for &,,,A,, let {x,) be a sequence in lirn,,, A, converging to x. Then from (6) we have for each rn

+ lim inf d(x, ,A,) = d(x, x,), m

lim inf d(x, A,) I d(x, x,) n-m

n-+

and since d(x, x,) can be made arbitrarily small by choosing m sufficiently A,. Replace lim inf,, ,by lim sup,, ,in large, we conclude that x E En,, b,,, A, is closed. the preceding argument to show that If lirn,,, A, = h,,, A,, we denote their common value by lirn,,, A,. This notation is justified by the following proposition. Proposition C.3 Let (X, d) be a compact metric space and p the Hausdorff metric on 2'. Let {A,) be a sequence in 2'. Then -

lirn A,

=

lim A,

=A

n+ m

n+ rn

if and only if lirn p(A,, A) = 0. n+ cc

Proof Assume for the moment that A # and suppose (16) holds. Then for each x in the compact set A, d(x, A,) + 0 as n -+ co.Given E > 0, j = 1,. . . , k let (x,, . . . , x,) be points of A such that the open spheres Sel2(xj), cover A. Choose N large enough so that for all n 2 N d(xj, A,) I ~ 1 2 ,

j = 1,. . . ,k.

Now use the Lipschitz continuity [cf. (6)] of the function x + d(x, A,) to conclude that d(x, A,) I E

Vx E A.

This implies that lim max d(x, A,) n+m

= 0.

xsA

This equation and (3) imply that (17) will follow if we can show lim max d( y, A) = 0. n-tw Y E A ,


309

If (18) fails to hold, then for some E > 0 there exists a sequence y , A,, ~ such that n, < n, < . . . and

The compactness of X implies that { y,} accumulates at some yo E X which, by (19) and the continuity of x + d(x,A), must satisfy d(yo,A) 2 e. But y0~lim,,, A, by (14),and this contradicts (16). Hence (18) holds. Still assuming A # 121, we turn to the reverse implication of the proposition. If (17) holds, then lim d(x, A,)

=0

V x E A,

n+ m

and lim max d( y, A) = 0. n+m YEA,,

Equation (20)implies that

If x EK,,,A,, then by definition there exists a sequence y, E A,, such that n, < n, < . , . and lim d(x,y,)

= 0.

k+ m

We have from (6) that

and, letting k -+ ar, and using (21)and (23),we conclude d(x,A) = 0. Since A is closed, this proves X E A and -

lim A, c A.

n+ m

(24)

Combine (22)and (24)to obtain (16). Assume finally that A = (Zr If (16) holds, then all but finitely many of the sets A, must be empty, for otherwise one could find y, E A,, ,n , < n, < . . . , and {y,} would accumulate at some yo ~lim,,, A,. If all but finitely many of the sets A, are empty, then ( 5 ) implies that (17) holds. Conversely, if (17) holds and A = 121, then (4) implies that all but finitely many of the sets A, are empty. Equation (16) follows from (2), (14), and (15). Q.E.D. For the proof of Proposition 7.33 in Section 7.5 we need the concept of a function which is upper semicontinuous in the sense of Kuratowski, or in abbreviation, upper semicontinuous (K).

310

APPENDIX

c

Definition C.2 Let Y be a metric space and X a compact metric space. A function F: Y -+ 2X is upper semicontinuous (K) if for every convergent sequence { y,) in Y with limit y, we have limn,, F( y,) c F( y). The similarity of Definition C.2 to the idea of an upper semicontinuous real or extended real-valued function is apparent [Lemma 7.13(b)]. Although we will not discuss functions which are lower semicontinuous (K), it is interesting to note that such a concept exists and has the obvious definition, namely, that the function F : Y -+ 2' is lower semicontinuous (K) if for every convergent sequence {y,) in Y with limit y, we have b,,, F(y,) 2 F(y). It can be seen from Proposition C.3 that a function F : Y -+ 2' is continuous in the usual sense (where 2' has the exponential topology) if and only if it is both upper and lower semicontinuous (K). We carry the analogy with real-valued functions even farther by showing that an upper semicontinuous (K) function is Borel-measurable, and the remainder of the appendix is devoted to this. Lemma C.2 Let Y be a metric space and X a compact metric space. If F : Y + 2' is upper semicontinuous (K), then for each open set G c X, the set

is open. Proof The openness of F-'(2') for every open G is in fact equivalent to upper semicontinuity (K), but we need only the weaker result stated. To prove it, we show that for G open, the set F-'(2X - 2G)is closed. If {y,) is a sequence in this set with limit y E Y , then

and so there exists a sequence {x,) in the compact set X - G such that . . . This sequence has an accumulation point x E X - G, x, E F(yn),n = 1,2,. and, by (14), x~lim,,, F(y,). The upper semicontinuity (K) of F implies x~F(y),andsoF(y)n(X-G)#ja,i.e.,y~F-'(2'-2'). Q.E.D. Proposition C.4 Let Y be a metric space, (X, d) a compact metric space, and let 2' have the exponential topology. Let F: Y -+ 2' be upper semicontinuous (K). Then F is Borel-measurable. Proof If F: Y -+ 2' is upper semicontinuous (K) and G is an open subset of X, then F-'(2G) is Borel-measurable in Y by Lemma C.2. If K is a closed subset of X, define open sets G, = (xjd(x, K) < lln). We have K = G,, and so a closed set A is a subset of K if and only if A c G,, n = 1,2,. . . .


This implies 2K =

2'", and

is a G,-set, thus Borel-measurable in Y. It follows that for any set 9 in the subbase9 u .X for the exponential topology on 2X,F - ' ( 9 ) is Borel-measurable in Y. By Proposition 7.1, any open set in 2X can be represented as a countable union of finite intersections of sets in 9 u .X and so its inverse image under F is Borel-measurable. Q.E.D.

References

R. Ash, "Real Analysis and Probability." Academic Press, New York, 1972. K. J. Astrom, Optimal control of Markov processes with incomplete state information, J. Math. Anal. Appl. 10 (1965), 174-205.

R. Bellman, "Dynamic Programming." Princeton Univ. Press, Princeton, New Jersey, 1957. D. P. Bertsekas, Infinite-time reachability of state-space regions by using feedback control, IEEE Trans. Automatic Control AC-17 (1972), 604-613. D. P. Bertsekas, On error bounds for successive approximation methods, IEEE Trans. Automatic Control AC-21 (1976), 394-396. D. P. Bertsekas, "Dynamic Programming and Stochastic Control." Academic Press, New York, 1976. D. P. Bertsekas, Monotone mappings with application in dynamic programming, S I A M J. Control Optimization 15 (1977), 438-464. D. P. Bertsekas and S. Shreve, Existence of optimal stationary policies in deterministic optimal control, J. Math. Anal. Appl. (to appear). P. Billingsley, Invariance principle for dependent random variables, Trans. Amer. Math. Soc. 83 (1956), 250-282. D. Blackwell, Positive dynamic programming, Proc. Fifth Berkeley Sympos. Math. Statist. and Probability, 1965, 415-418. D. Blackwell, Discounted dynamic programming, Ann. Math. Statist. 36 (1965), 226-235 D. Blackwell, On stationary policies, J. Roy. Statist. Soc. 133A (1970), 33-37. D. Blackwell, Borel-programmable functions, Ann. Prob. 6 (1978). 321-324. D. Blackwell, D. Freedman, and M. Orkin, The optimal reward operator in dynamic programming, Ann. Probability 2 (1974), 926-941. N. Bourbaki, "General Topology." Addison-Wesley. Reading, Massachusetts, 1966. D. W. Bressler and M. Sion, The current theory of analytic sets, Canad. J. Math. 16 (1964), 207-230.

REFERENCES

313

L. D. Brown and R. Purves, Measurable selections of extrema, Ann. Statist. 1 (1973), 902-912. D. Cenzer and R. D. Mauldin, Measurable parameterizations and selections, Trans. Amer. Math. Soc. (to appear). C. Dellacherie, "Ensembles Analytiques, Capacittts, Mesures de Hausdorff." SpringerVerlag, Berlin and New York, 1972. E. V. Denardo, Contraction mappings in the theory underlying dynamic programming, S I A M Rev. 9 (1967), 165-177. C. Derman, "Finite State Markovian Decision Processes." Academic Press, New York, 1970. J. L. Doob, "Stochastic Processes." Wiley, New York, 1953. L. Dubins and D. Freedman, Measurable sets of measures, Pacific J. Math. 14 (1964), 1211-1222. L. Dubins and L. Savage, "Inequalities for Stochastic Processes (How to Gamble if you Must)." McGraw-Hill, New York, 1965. (Republished by Dover, New York, 1976.) J. Dugundji, "Topology." Allyn & Bacon, Rockleigh, New Jersey, 1966. E. B. Dynkin and A. A. Juskevic, "Controlled Markov Processes and their Applications." Moscow, 1975. (English translation to be published by Springler-Verlag.) D. Freedman, The optimal reward operator in special classes of dynamic programming problems, Ann. Probability. 2 (1974), 942-949. E. B. Frid, On a problem of D. Blackwell from the theory of dynamic programming, Theor. Probability Appl. 15 (1970), 719-722. N. Furukawa, Markovian decision processes with compact action spaces, Ann. Math. Statist. 43 (1972) 1612-1622. N. Furukawa and S. Iwamoto, Markovian decision processes and recursive reward functions, Bull. Math. Statist. 15 (1973), 79-91. N. Furukawa and S. Iwamoto, Dynamic programming on recursive reward systems, Bull. Math. Statist. 17 (1976), 103-126. K. Godel, The consistency of the axiom of choice and of the generalized continuumhypothesis, Proc. Nut. Acad. Sci. U.S.A. 24 (1938), 556-557. P. R. Halmos, "Measure Theory." Van Nostrand-Reinhold, Princeton, New Jersey, 1950. F. Hausdorff, "Set Theory." Chelsea, Bronx, New York, 1957. C. J. Himmelberg, T. Parthasarathy, and F. S. Van Vleck, Optimal plans for dynamic programming problems, Math. Operations Res. 1 (1976). 390-394. K. Hinderer, "Foundations of Nonstationary Dynamic Programming with Discrete Time Parameter." Springler-Verlag, Berlin and New York, 1970. J. Hoffman-Jorgensen, "The Theory of Analytic Spaces." Aarhus Universitet, Aarhus, Denmark, 1970. A. Hordijk, "Dynamic Programming and Markov Potential Theory." Mathematical Centre Tracts, Amsterdam, 1974. R. Howard, "Dynamic Programming and Markov Processes." MIT Press, Cambridge, Massachusetts, 1960. B. Jankov, On the uniformisationof A-sets, Dokl. Akad. Nauk S S S R 30 (1941), 591 -592 (in Russian). W. Jewell, Markov renewal programming I and 11, Operations Res. 11 (1963), 938-971. A. A. Juskevit (Yushkevich), Reduction of a controlled Markov model with incomplete data to a problem with complete information in the case of Borel state and control spaces, Theor. Probability Appl. 21 (1976), 153-158. L. Kantorovich and B. Livenson. Memoir on analytical operations and projective sets, Fund. Math. 18 (1932), 214-279.

314

REFERENCES

K. Kuratowski, "Topology I." Academic Press, New York, 1966. K. Kuratowski, "Topology 11." Academic Press, New York, 1968. K. Kuratowski and A. Mostowski. "Set Theory." North-Holland, Amsterdam, 1976. K. Kuratowski and C. Ryll-Nardzewski, A general theorem on selectors, Bull. Polish Acad. Sci. 13 (1965), 397-411. H. Kushner, "Introduction to Stochastic Control." Holt, New York, 1971. M. Loeve, "Probability Theory." Van Nostrand-Reinhold, Princeton, New Jersey, 1963. N. Lusin, Sur les ensembles analytiques, Fund. Math. 10 (1927), 1-95. N. Lusin and W. Sierpinski, Sur quelques proprittts des ensembles (A), Bull. Acad. Sci. Cracovie (1918), 35-48. G. Mackey, Bore1 structure in groups and their duals, Trans. Amer. Math. Soc. 85 (1957), 134-165. A. Maitra, Discounted dynamic programming on compact metric spaces, Sankhya 30A (1968), 211-216. J. McQueen, A modified dynamic programming method for Markovian decision problems, J. Math. Anal. Appl. 14 (1966), 38-43. P. A. Meyer, "Probability and Potentials." Ginn (Blaisdell), Boston, Massachusetts, 1966. P. A. Meyer and M. Traki, Reduites et jeux de hasard (Seminaire de Probabilites VII, Universite de Strasbourg, in "Lecture Notes in Mathematics," Vol. 321), pp. 155-171. Springer, Berlin, 1973. J. von Neumann, On rings of operators. Reduction theory, Ann. of Math. 50 (1949), 401 -485. P. Olsen, Multistage stochastic programming with recourse: The equivalent deterministic problem, SIAM J. Control Optimization 14 (1976), 495-517. P. Olsen, When is a multistage stochastic programming problem well-defined?, SIAM J. Control Optimization 14 (1976), 518-527. P. Olsen, Multistage stochastic programming with recourse as mathematical programming in an L, space, SIAM J. Control Optimization 14 (1976), 528-537. D. Ornstein, On the existence of stationary optimal strategies, Proc. Amer. Math. Soc. 20 (1969), 563-569. J. M. Ortega and W. C. Rheinboldt, "Iterative Solutions of Nonlinear Equations in Several Variables." Academic Press, New York, 1970. K. Parthasarathy, "Probability Measures on Metric Spaces." Academic Press, New York, 1967. Yu. V. Prohorov, Convergence of random processes and limit theorems in probability theory, Theor. Probability Appl. 1 (1956), 157-214. D. Rhenius, Incomplete information in Markovian decision models, Ann. Statist. 2 (1974), 1327-1334. R. T. Rockafellar, Integral functionals, normal integrands and measurable selections, in "Nonlinear Operators and the Calculus of Variations." Springer-Verlag, Berlin and New York, 1976. R. T. Rockafellar and R. Wets, Stochastic convex programming: relatively complete recourse and induced feasibility, SIAM J. Control Optimization 14 (1976), 574-589. R. T. Rockafellar and R. Wets, Stochastic convex programming: basic duality, Pacific J. Math. 62 (1976). 173-195. H. L. Royden, "Real Analysis." Macmillan, New York, 1968. S. Saks, "Theory of the Integral." Stechert, New York, 1937. Y. Sawaragi and T. Yoshikawa, Discrete-time Markovian decision processes with incomplete state information, Ann. Math. Statist. 41 (1970), 78-86.

REFERENCES

315

M. Schal, On continuous dynamic programming with discrete time parameter, 2. Wahrscheinlichkeitstheorie und Verw. Gebiete 21 (1972), 279-288. M. Schal, On dynamic programming: Compactness of the space of policies, Stochastic Processes Appl. 3 (1975). 345-364. M. Schal, Conditions for optimality in dynamic programming and for the limit of n-stage optimal policies to be optimal, 2.Wahrscheinlichkeitstheorie und Verw. Gebiete 32 (1975), 179-196. E. Selivanovskij, Ob odnom klasse effektivnyh mnozestv (mnozestva C), Mat. Sb. 35 (1928), 379-413. S. Shreve, A General Framework ,for Dynamic Programming with Specializations, M . S. thesis (1977), Dept. of Elec. Eng., Univ. of Illinois, Urbana. S. Shreve, Dynamic Programming in Complete Separable Spaces, Ph.D. thesis (1977), Dept. of Math., Univ. of Illinois, Urbana. S. Shreve and D. P. Bertsekas, A new theoretical framework for finite horizon stochastic control, Proc. Fourteenth Annual Allerton Conf. Circuit and System Theory, ANerton Park, Illinois, October, 1976, 336-343. S. Shreve and D. P. Bertsekas, Equivalent stochastic and deterministic optimal control problems, Proc. 1976 IEEE Conf. Decision and Control, Clearwater Beach, Florida, 705-709. S. Shreve and D. P. Bertsekas, Alternative theoretical frameworks for finite horizon discrete-time stochastic optimal control, SIAM J. Control Optimization 16 (1978). S. Shreve and D. P. Bertsekas, Universally measurable policies in dynamic programming Mathematics of'Operations Research (to appear). R. Solovay, A model of set-theory in which every set of reals is Lebesgue measurable, Ann. Marh. 92 (1970), 1-56. R. E. Strauch, Negative dynamic programming, Ann Marh. Statist. 37 (1966), 871-890 C. Striebel, Sufficient statistics in the optimal control of stochastic systems, J. Math. Anal. Appl. 12 (1965), 576-592. C. Striebel, "Optimal Control of Discrete Time Stochastic Systems." Springer-Verlag, Berlin and New York, 1975. M. Suslin (Souslin), Sur une definition des ensembles measurables B sans nombres transfinis, C. R. Acad. Sci. Paris 164 (1917), 88-91. V. S. Varadarajan, Weak convergence of measures on separable metric spaces, Sankhya 19 (1958), 15-22. D. H. Wagner, Survey of measurable selection theorems, SIAM J. Control Optimization 15 (1977), 859-903. A. Wald, "Statistical Decision Functions." Wiley, New York, 1950. H. S. Witsenhausen, A standard form for sequential stochastic control, Math. Systems Theory 7 (1973), 5-1 1.

Table of Propositions, Lemmas, Definitions, and Assumptions

Chapter 2

Monotonicity Assumption Chapter 3

Proposition Proposition Proposition Proposition Proposition Proposition Proposition

3.1 3.2 3.3 3.4 3.5 3.6 3.7

Chapter 5

Lemma 3.1

Chapter 4

4.1 4.2 4.3 4.4 4.5

4.6 4.7 4.8 4.9 4.10 4.1 1

Assumption C (Contraction Assumption) Fixed Point Theorem

Assumption F. I Assumption F.2 Assumption F.3

Proposition Proposition Proposition Proposition Proposition

Proposition Proposition Proposition Proposition Proposition Proposition

Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition

5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 5.10 5.1 1 5.12 5.13

TABLE O F PROPOSITIONS

Proposition 5.14 Proposition 5.15 Lemma 5.1 Lemma 5.2 Assumption I (Uniform Increase Assumption) Assumption D (Uniform Decrease Assumption) Assumption I. I Assumption 1.2 Assumption D. 1 Assumption D.2 Chapter 6

Proposition Proposition Proposition Proposition Proposition

6.1 6.2 6.3 6.4 6.5

Assumption A. 1 Assumption A.2 Assumption A . 3 Assumption A . 4 Assumption A.5 Assumption P.2 Assumption F.3 Exact Selection Assumption Assumption C Chapter 7

Proposition 7.1 Proposition 7.2 (Urysohn's Theorem) Proposition 7.3 (Alexandroff's Theorem) Proposition 7.4 Proposition 7.5 Proposition 7.6 Proposition 7.7 Proposition 7.8 Proposition 7.9 Proposition 7.10 Proposition 7.1 1 Proposition 7.12 Proposition 7.13 Proposition 7.14 Proposition 7. I5 (Kuratowski's Theorem)

Proposition 7.16 Proposition 7.17 Proposition 7.18 Proposition 7.19 Proposition 7.20 Proposition 7.21 Proposition 7.22 Proposition 7.23 Proposition 7.24 (Dynkin System Theorem) Proposition 7.25 Proposition 7.26 Proposition 7.27 Proposition 7.28 Proposition 7.29 Proposition 7.30 Proposition 7.31 Proposition 7.32 Proposition 7.33 Proposition 7.34 Proposition 7.35 Proposition 7.36 Proposition 7.37 Proposition 7.38 Proposition 7.39 Proposition 7.40 Proposition 7.41 Proposition 7.42 (Lusin's Theorem) Proposition 7.43 Proposition 7.44 Proposition 7.45 Proposition 7.46 Proposition 7.47 Proposition 7.48 Proposition 7.49 (Jankov-von Neumann Theorem) Proposition 7.50 Lemma Lemma Lemma Lemma Lemma Lemma Lemma Lemma Lemma Lemma Lemma Lemma Lemma

7.1 (Urysohn's Lemma) 7.2 7.3 7.4 7.5 7.6 7.7 7.8 7.9 7.10 7.11 7.12 7.13

TABLE OF PROPOSITIONS Lemma Lemma Lemma Lemma Lemma Lemma Lemma Lemma Lemma Lemma Lemma Lemma Lemma Lemma Lemma Lemma Lemma

7.14 7.15 7.16 7.17 7.18 7.19 7.20 7.21 7.22 7.23 7.24 7.25 7.26 7.27 7.28 7.29 7.30

Definition Definition Definition Definition Definition Definition Definition Definition Definition Definition Definition Definition Definition Definition Definition Definition Definition Definition Definition Definition Definition

Lemma Lemma Lemma Lemma Lemma

Definition Definition Definition Definition Definition Definition Definition Definition

Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition

Chapter 8

Lemma 8.1 Lemma 8.2

8.1 8.2 8.3 8.4 8.5 8.6 8.7

8.1 8.2 8.3 8.4 8.5 8.6 8.7 8.8 Chapter 9

7.1 7.2 7.3 7.4 7.5 7.6 7.7 7.8 7.9 7.10 7.1 1 7.12 7.13 7.14 7.15 7.16 7.17 7.18 7.19 7.20 7.21


8.3 8.4 8.5 8.6 8.7

9.1 9.2 9.3 9.4 9.5 9.6 9.7 9.8 9.9 9.10 9.11 9.12 9.13 9.14 9.15 9.16 9.17 9.18 9.19 9.20 9.21

Lemma 9.1 Lemma 9.2 Lemma 9.3 Definition Definition Definition Definition Definition Definition Definition Definition Definition Definition

9.1 9.2 9.3 9.4 9.5 9.6 9.7 9.8 9.9 9.10

TABLE OF PROPOSITIONS

Chapter 10

Proposition Proposition Proposition Proposition Proposition Proposition Lemma Lemma Lemma Lemma

Appendix B

10.1 10.2 10.3 10.4 10.5 10.6

Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition Proposition

10.1 10.2 10.3 10.4

Definition Definition Definition Definition Definition Definition Definition Definition

10.1 10.2 10.3 10.4 10.5 10.6 10.7 10.8

245 248 249 249 250 251 256 Chapter 11


11.1 11.2 11.3 11.4 11.5 11.6 11.7 Appendix A

Proposition A. 1 Lemma A. 1 Lemma A.2 Lemma A.3 Definition A. 1

Lemma Lemma Lemma Lemma Lemma Lemma Lemma Lemma

B. 1 B.2 B.3 B.4 B.5 B.6 B.7 B.8 B.9 B. 10 B. 11 B. 12

B.l B.2 B.3 B.4 B.5 B.6 B.7 B.8

Definition B. 1 Definition B.2 Definition B.3

Appendix C

Proposition Proposition Proposition Proposition

C. 1 C.2 C.3 C.4

Lemma C.l Lemma C.2 Definition C. 1 Definition C.2

Index

A

Alexandroff s theorem, 107 Analytic measurability of a function, 171 Analytic set, 160 Analytic u-algebra, 171 A posteriori distribution, 260ff A priori distribution, 260ff Axiom of choice, 301

B Baire null space, 103, 109 Borel isomorphism, 121 Borel measurability of a function. 120 Borel programmable, 2 1 Borel a-algebra, 117 Borel space, I18

C Cantor's continuum hypothesis, 301 Completion of a metric space, 114 of a a-algebra, 167 Composition of measurable functions, 298 Contraction assumption, 52

Control constraint, 2,26, 188,216,243,245,248, 251,271 space, 2,26, 188,2 16,243,245,248,251, 27 1 Cost corresponding to a policy, 2,28, 191,217, 244,249,254 one-stage, 2, 189,2 16,243,245,248,25 1, 27 1 optimal, 2,29, 191,217,244,246,250,254 C-sets, 20

D Disturbance kernel, 189,243,245,271 Disturbance space, 189,243,245,271 Dynamic programming (DP) algorithm, 3,6, 39,57,80, 198,229,259 Dynkin system, 133 Dynkin system theorem, 133

Epigraph, 82 Exact selection assumption. 95 Exponential topology, 304

F Filtering, 261 Fixed point theorem (Banach), 55 F,-set, 102

N Nonstationary model, 243

0

H Hausdorff metric, 303 Hilbert cube, 103 Homeomorphism, 104 Horizon finite, 28, 189,243,245,248,251,271 infinite, 70,213,216,243,245,248,251

I Imperfect state information model, 248 Indicator function, 103 Information vector, 248 Isometry, 144

J Jankov-von Neumann theorem, 182

K Kuratowski's theorem, 121

Observation kernel, 248 Observation space, 248 Optimality equation, 4, 57, 71, 73, 78ff, 225 nonstationary, 246 Outer integral, 273 monotone convergence theorem for, 278

P Paved space, 157 Policy,2,6,26,91, 190,214,217,243,249 analytically measurable, 190,269ff Borel-measurable, 190 €-optimal, 29, 191,215,244 {en}-dominated convergence to optimality, 29, 191,245 k-originating, 243 limit-measurable, 190,266ff Markov, 6, 190 nonrandomized, 190,249 optimal, 29,191,215,244 p-e-optimal, 12 q-optimal, 256 semi-Markov, 190 stationary, 214 uniformly N-stage optimal, 29,206 universally measurable, 190 weakly q-e-optimal, 256 p-outer measure, 166,274 Projection mapping, 103

L Limit measurability, 298 Limit a-algebra, 293 Lindelof space, I06 Lower semianalytic function, 177 Lower semicontinuous function, 146 Lower semicontinuous model, 208 Lusin's theorem, I67 M Metrizable space, 104 Monotonicity assumption. 27

Regular probability measure, 122 Relative topology, 104 R-operator, 21 5

Second countable space, 106 Semi-Markov decision problems, 34 Separable space, 105 State space, 2,26. 188,216,243,245,248, 251,271

State transition kernel, 189,243,248, 25 1 Statistic sufficient for control, 250 existence of, 259ff Stochastic kernel, 134 Stochastic programming, I Iff Suslin scheme, 157 nucleus of, 157 regular, 161 System function, 189,216,243,245,271

Topologically complete space, 107 Totally bounded space, 112

Uniform decrease assumption, 70 Uniform increase assumption. 70 Universal function, 290 Universal measurability of a function, 171 Universal a-algebra, 167 Upper semicontinuous function, 146 Upper semicontinuous model, 210 Upper semicontinuous (K) function, 3 10 Urysohn's lemma, 105 Urysohn's theorem, 106

Weak topology on space of probability measures, 125

Stochastic optimal control Discrete-time Case

Stochastic optimal control: the discrete time case

Stochastic optimal control

Optimal Control

Optimal control

Optimal Control: An Introduction

Optimal Control: An Introduction

Stochastic Optimal Control and the U.S. Financial Debt Crisis

Robust and optimal control

Robust and optimal control

Optimal Control Systems

Stochastic Optimal Control, International Finance, and Debt Crises

Optimal control and estimation

Optimal Control: An Introduction

Optimal Adaptive Control Systems

Optimization and Optimal Control

Optimal Control: An Introduction

Linear Optimal Control

Optimal Periodic Control

Stochastic Optimal Control: The Discrete-Time Case (Optimization and Neural Computation Series)

Stochastic Optimal Control: The Discrete-Time Case (Optimization and Neural Computation Series)

Robust control design. An optimal control approach

Applied Optimal Control: Optimization, Estimation and Control

Stochastic stability and control

Stochastic Stability and Control

Applied Optimal Control: Optimization, Estimation, And Control

Optimal Control Theory: An Introduction

Optimal Control Theory. An Introduction

Stochastic dynamics and control

Optimal control of viscous flow

Optimal, Predictive and Adaptive Control

Stochastic optimal control Discrete-time Case

Stochastic optimal control: the discrete time case