Introduction to Applied Optimal Control (Mathematics in Science and Engineering, 159)

An Introduction to Applied Optimal Control

This is Volume 159 in MATHEMATICS IN SCIENCE AND ENGINEERING A Series of Monographs and Textbooks Edited by RICHARD BELLMAN, University of Southern California The complete listing of books in this series is available from the Publisher upon request.

An Introduction to Applied Optimal Control Greg Knowles Department of Mathematics Carnegie-Mellon University Pittsburgh. Pennsylvania

@

1981

ACADEMIC PRESS A Subsidiary of Harcourt Brace Jovanovich. Publishers

New York

London

Toronto

Sydney

San Francisco

COPYRIGHT © 1981, BY ACADEMIC PRESS, INC. ALL RIGHTS RESERVED. NO PART OF THIS PUBLICATION MAY BE REPRODUCED OR TRANSMITTED IN ANY FORM OR BY ANY MEANS, ELECTRONIC OR MECHANICAL, INCLUDING PHOTOCOPY, RECORDING, OR ANY INFORMATION STORAGE AND'RETRIEVAL SYSTEM, WITHOUT PERMISSION IN WRITING FROM THE PUBLISHER.

ACADEMIC PRESS, INC.

111 Fifth Avenue, New York, New York 10003

United Kingdom Edition published by

ACADEMIC PRESS, INC. (LONDON) LTD. 24/28 Oval Road, London NWI 7DX

Library of Congress Cataloging in Publication Data

Knowles. Greg. An introduction to applied opt1Jllal control. (Mathematics in science and engineering)

Includes bibliographies and index.

1. Control theory. 2. Mathematical optimization. I. Title. II. Series. QA402.3.K56 629.8'312 81-7989 ISBN Q..12-416960-0 AACR2

PRINTED IN THE UNITED STATES OF AMERICA

81 82 83 84

9 8 7 6 S 4 3 2 1

Contents

Preface

ix

Chapter I Examples of Control Systems;

the Control Problem

General Form of the Control Problem

Chapter II

5

The General Linear Time Optimal Problem

I. Introduction

9

2. Applications of the Maximum Principle

13

3. Normal Systems-Uniqueness of the Optimal Control

17

20

29

34

4. Further Examples of Time Optimal Control 5. Numerical Computation of the Switching Times References

Chapter III The Pontryagin Maximum Principle I. The Maximum Principle

2. Classical Calculus of Variations 3. More Examples of the Maximum Principle References

35

40

44

57

Chapter IV The General Maximum Principle; Control Problems with Terminal Payoff 1. Introduction

2. Control Problems with Terminal Payoff 3. Existence of Optimal Controls References

59

61

64

69

V

vi

Contents

Chapter V Numerical Solution of Two-Point Boundary-Value Problems 1. 2. 3. 4. 5. 6.

Linear Two-PointBoundary-Value Problems NonlinearShooting Methods Nonlinear ShootingMethods: Implicit Boundary Conditions Quasi-Linearization Finite-Difference Schemes and Multiple Shooting Summary . References

71

74

77

79

81

84

85

Chapter VI Dynamic Programming and Differential Games 1. Discrete Dynamic Programming 2. Continuous Dynamic Programming-Control Problems 3. Continuous Dynamic Programming-Differential Games References

87

90

96

105

Chapter VII Controllability and Observability 1. ControllableLinear Systems 2. Observability References

107

116

121

Chapter VIII State-Constrained Control Problems 1. 2. 3. 4.

The Restricted MaximumPrinciple Jump Conditions' The Continuous Wheat Trading Model withoutShortselling Some Models in Productionand Inventory Control References

123

126

132

135

141

Chapter IX Optimal Control of Systems Governed by Partial Differential Equations 1. SomeExamp~sofEllipticControIProb~ms 2. Necessary and SufficientConditionsfor Optimality 3. Boundary Control and ApproximateControllability

of Elliptic Systems 4. The Control of Systems Governedby ParabolicEquations 5. Time Optimal Control 6. ApproximateControllabilityfor Parabolic Problems References

Appendix I Geometry of R"

143

146

152

156

159

161

164

165

Contents

Appendix II

vii

Existence of Time Optimal Controls

and the Bang-Bang Principle

169

Appendix III Stability

175

Index

179

This page intentionally left blank

Preface

This book began as the notes for a one-semester course at Carnegie­ Mellon University. The aim of the course was to give an introduction to deterministic optimal control theory to senior level undergraduates and first­ year graduate students from the mathematics, engineering, and business schools. The only prerequisite for the course was a junior level course in ordinary differential equations. Accordingly, the backgrounds of the students were widely dissimilar, and the common denominator was their interest in the applications of optimal control theory. In fact, one of the most popular aspects of the course was that students were able to study problems from areas that they would not normally cover in their respective syllabi. This text differs from the standard ones in that we have not attempted to prove the maximum principle, since this was beyond the background and interest of most of the students in the course. Instead we have tried to show its strengths and limitations through examples. In Chapter I we introduce the concept of optimal control by means ofexamples. In Chapter II necessary conditions for optimality for the linear time optimal control problem are derived geometrically, and illustrations are given. In .Chapters ill and IV we discuss the Pontryagin maximum principle, its relation to the calculus of variations, and its application to various problems in science, engineering, and business. Since the optimality conditions arising from the maximum principle can often be solved only numerically, numerical techniques are discussed in Chapter V. In Chapter VI the dynamic programming approach to the solution of optimal control problems and differential games is considered; in Chapter VII the controllability and observability of linear control systems are discussed, and in Chapter VIII the extension of the maximum principle to state-constrained ix

x

Preface

control problems is given. Finally, for more advanced students with a background in functional analysis, we consider in Chapter IX several problems in the control of systems governed by partial differential equations. This could serve as an introduction to research in this area. The support of my colleagues and students at Carnegie-Mellon University has been invaluable during this project; without it this text would almost certainly not have appeared.

Chapter I

Examples of Control Systems; the Control Problem

Example 1 Consider a mechanism, such as a crane or trolley, of mass m which moves along a horizontal track without friction. If x( t) re­ presents the position at time t, we assume the motion of the trolley is governed by the law t > 0,

mx(t) = u(t),

(1)

where u(t) is an external controlling force that we apply to the trolley (see Fig. 1). Assume that the initial position and velocity of the trolley are given as x(O) = xo, x(O) = Yo, respectively. Then we wish to choose a function u (which is naturally enough called a control function) to bring the trolley to rest at the origin in minimum time. Physical restrictions will usually require that the controlling force be bounded in magnitude, i.e., that !u(t)! s M.

(2)

For convenience, suppose that m = M = 1, and rewrite Eq. (1) X2

= u(t),

where x 1(t) and X2(t) are now the position and velocity of the body at time t. Equation (1) then becomes

2

I.

Examples of Control Systems m 1·. . 1ee

ll =UC>:.L_ _

01---------"-''---~---------

Fig••

or x(t)

= Ax(t) + bu(t),

x(O) =

[;:l

where A

=

[~ ~]

and

b=[~l

x(t) =

(3)

[Xt(t)], X2(t)

and the control problem is to find a function u, subject to (2), which brings the solution of (3), x(t), to [8J in minimum time t. Any control that steers us to [8J in minimum time is called an optimal control. Intuitively, we should expect the optimal control is first a period of maximum acceleration(u = + 1),and then maximum braking(u = -1), or vice versa. Example 2 (Bushaw [1 J) A control surface on an aircraft is to be kept at rest at a fixed position. A wind gust displaces the surface from the desired position. We assume that if nothing were done, the control surface would behave as a damped harmonic oscillator. Thus if () measures the deviation from the desired position, then the free motion of the surface satisfies the differential equation

o+ afJ + w

2

()

=0

with initial conditions ()(O) = ()o and fJ(O) = ()~. Here ()o is the displace­ ment of the surface resulting from the wind gust and ()~ is the velocity imparted to the surface by the gust. On an aircraft the oscillation of the control surface cannot be permitted, and so we wish to design a servomechanism to apply a restoring torque and bring the surface time. The equation then becomes back to rest in ~inimum O(t) + afJ(t)

+ w 2 ()(t) =

u(t),

fJ(O) = ()~,

(4)

where u(t) represents the restoring torque at time t. Again we must suppose that lu(t)1 s C, where C is a constant, and by normalization

Examples of Control Systems

3

can be taken as 1. The problem is then to find such a function u, so that the system will be brought to (J = 0, tJ = 0 in minimum time. It is clear that if (Jo > 0 and (J~ > 0, then the torque should be directed initially in the direction of negative (J and should have the largest possible magnitude. Thus u(t) = -1 initially. However, if u(t) = -1 is applied for too long a time, we shall overshoot the desired terminal condition (J = 0, = O. Therefore at some point there should be a torq ue reversal to + 1 in order to brake the system. The following questions occur:

e

(1) Is this strategy indeed optimal, and ifso, when should the switch take place? (2) Alternatively, is it better to remove the torque at some point, allow a small overshoot, and then apply + 1? (3) In this vein, we could ask whether a sequence of -1, + 1, -1, + 1, ... of n steps is the best, and if so, what is n and where do the switches occur? Again we are led to controls that take on (only) values ± 1; such controls are called bang-bang controls. Note that as before, setting Xl = () and X 2 = iJ, we can write the system equation (4) Xl = x 2 ,

x2 = i

-ax 2

Xl(O) = -

W2Xl

+ u,

= Ax + bu,

xiO)

=

x(O) =

(Jo,

(J~,

[~:J

where

A=[O _w

2

1J b=[~J

-a '

and u is chosen with ju(t)1 =:;; 1 and to minimize C(u) =

III Jo 1 dt,

Example 3 (Isaacs [3J) Let x(t) be the amount of steel produced by a mill at time t. The amount produced at time t is to be allocated

4

I. Examples of Control Systems

to one of two uses: (1) production of consumer products; (2) investment. It is assumed that the steel allocated to investment is used to increase

productive capacity-by using steel to produce new steel mills, trans­ port facilities, or whatever. Let u(t), where 0::;; u(t) ::;; 1, denote the fraction of steel produced at time t that is allocated to investment. Then 1 - u(t) represents the fraction allocated to consumption. The assump­ tion that the reinvested steel is used to increase the productive capacity could be written dx dt = ku(t)x(t),

where

x(O)

= C - initial endowment,

where k is an appropriate constant (i.e., rate of increase in production is proportional to the amount allocated to investment). The problem is to choose u(t) so as to maximize the total consumption over some fixed period of time T > O. That is, we are to maximize

f: (1 -

u(t»x(t)dt.

For this problem, do we consume everything produced, or do we invest some at present to increase capacity now, so that we can produce more and hence consume more later? Do we follow a bang-bang procedure of first investing everything and then consuming everything? Example 4 Moon-Landing Problem (Fleming and Rishel [2J) Consider the problem of a spacecraft attempting to make a soft landing on the moon using the minimum amount of fuel. For a simplified model, let m denote the mass, h the height, v the vertical velocity of the space­ craft above the moon, and u the thrust of the spacecraft's engine (m, h, v, and u are functions of time). Let M denote the mass of the spacecraft without fuel, ho the initial height, Vo the initial velocity, F the initial amount of fuel, oc the maximum thrust of the engine, k a constant, and g the gravity acceleration of the moon (considered constant). The equations of motion are

h = v, v=-g+m-1u,

m=

-ku,

General Form of the Control Problem

and the control u is restricted so that are

°::; u(t) ::;;

h(O)

0(.

5

The end conditions

= ho ,

v(O) = vo, m(O) - M - F = 0,

h(t l ) =0, v(t l ) = 0,

where t 1 is the time taken for touchdown. With Xl

= h,

x(O) x(t l)

Xl

= V,

= m,

X3

= (ho, vo, M + F)T, = (O,O,anything)T,

this problem becomes, in matrix form, -g

X= [

+X l X3"lu] -ku

and we wish to choose u so that

is a minimum. However, x3

= f(t,x,U),

°

;$;

u(t)

;$;

0(

and

= - ku, so the above becomes -M - F

+ k f~1

u(r)dr,

and this is minimized at the same time as C(u)

CII = Jo

u(r) dt,

Note that although these problems come from seemingly completely different areas of applied mathematics, they all fit into the following general pattern.

GENERAL FORM OF THE CONTROL PROBLEM (1) The state equation is i = 1, ... ,n,

6

I.

Examples of Control Systems

or in vector form i = f(t, x, U),

where and (2) The initial point is x(O) = Xo ERn, and the final point that we wish to reach is Xl ERn. The final point Xl is often called the target (point), and mayor may not be given. (3) The class .1 of admissible controls is the set of all those control functions u allowed by the physical limitations on the problem. (In Examples 1 and 2 we had .:\ = {u : lu(t)1 s 1} and m = 1.) Usually we shall be given a compact, convex set Q c B" (the restraint set) and we shall take .:\ =

{u = (ut> ... ,um) : u, piecewise continuous and u(t) E Q}.

(4) The cost function or performance index quantitatively compares the effectiveness of various controllers. This is usually of the form

ill

C(u) = Jo fo(t, x(t), u(t»dt, where fo is a given continuous real-valued function, and the above integral is to be interpreted as: we take a control u E .:\, solve the state equations to obtain the corresponding X, calculate fo as a function of t, and perform the integration. If a target point is given (so called fixed­ end-point problem), then t 1 must be such that X(tl) = Xl' In particular, if fo == 1, then C(u) = t 1 , and we have the minimum-time problem. Ifa target point is not given (free-end-point problem), then t 1 will be a fixed given time, -and the integration is performed over the fixed interval [0, t 1 ]. The optimal control problem can now be formulated: Find an admissible control u* that minimizes the cost function, i.e., for which C(u*) S C(u) for all u E .:\. Such controls u* are called optimal controls. We shall first investigate in depth in Chapter 2 the linear (i.e., state equations are linear in X and u) time optimal control problem, deriving a necessary condition for optimality known as Pontryagin's maximum principle [4].

References

7

References [I] D. Bushaw, Optimal discontinuous forcing terms, in "Contributions to the Theory

of Non-linear Oscillations," pp. 29-52. Princeton Univ. Press, Princeton, New Jersey, 1958. [2] W. Fleming and R. Rishel, "Deterministic and Stochastic Optimal Control." Springer-Verlag, Berlin and New York, 1975. [3] R. Isaacs, "Differential Games." Wiley, New York, 1965. [4] L. Pontryagin, V. Boltyanskii, R. Gramkrelidze, and E. Mischenko, "The Mathe­ matical Theory of Optimal Processes." Wiley (Interscience), New York, 1962. For extensive bibliographies on control theory, the reader should consult E. B. Lee and L. Markus, "Foundations of Optimal Control Theory." Wiley, New York, 1967. M. Athans and P. Falb, ."Optimal Control: An Introduction to the Theory and Its Applications. McGraw-Hill, New York, 1965.

This page intentionally left blank

Chapter II

1.

The General Linear Time Optimal Problem

INTRODUCTION

Consider a control system described by the vector differential equation x(t) = A(t)x(t)

+ B(t)u(t),

x(O)

= X o ERn,

(1)

where

a2n(t)

aln(t)l . ,

(t ) bblm(t)] 2m . ,

anit)

bnm(t)

and we assume the elements of A(t), B(t) are integrable functions over any finite interval of time. The set of admissible controls will be ~

= {u = (ut>. . . ,Um)T : lui(t)1 ::;; 1, i = 1, ... ,m}.

(2)

A target point x, ERn is given, and the control problem is to mini­ mize the time t l for which x(t l ) = Xl' 9

lO

II.

The General Linear Time Optimal Problem

From the theory of ordinary differential equations, the solution of (1) can be written x(t: u) = X(t)x o + X(t) f~

X-1(l:)~(r)u(l:)dl:,

(3)

where X(t) is the principal matrix solution ofthe homogeneous system X(t) = A(t)X(t), X(O) = I, the identity matrix [3,7]. At a given time t we define the attainable set at time t to be just the set of all those points x E R" that we can reach in time t using all of our admissible controls, i.e., d(t) = {x(t: u): U E Ll}. A knowledge of the attainable set at time t will give us a complete description ofthe points we can hit in time up to t (see Fig. 1). Further we can reformulate our control problem: Minimize the time t 1 for which

Xl E

d(t 1 ) .

Before we proceed to characterize time optimal controls, we should really at this point ask ourselves, Does an optimal control exist? Otherwise we lay ourselves open to the possibility of constructing something that may not exist. This point should be stressed, for in a control problem we are usually trying to force a physical system to behave according to our requirements, and there is generally no reason to expect that nature will be sympathetic. For the linear time optimal control problem, it is shown in Appen­ dix 1 that optimal controls exist if the target point Xl can be hit in some time. This last condition brings us to the subject known as con­ trollability, which will be further discussed in Chapter VII. From now on we shall assume that an optimal control u* exists, where t* is the minimum time, and we shall consider the problem of

• "1

Fig.l

1.

II

Introduction

characterizing u*. Further we shall consider only the case of one control variable, i.e., m = 1. B is then n x 1. The general case is only notationally different. Define y(t) = X-l(t)B(t), i.e., y(t) = (Yl(t), ... ,Yn(tW, and Yl = X-l(t*)Xl - Xo ERn, i.e., Yl = J~ y(r)u*(r) dt, Set 9P(t)

=

{f~

=

{(f~

y(r)u(r)dr: u h(r)u(r),

f~

E~} Y2(r)u(r), ... ,

f~

Yn(r)u(r) dt

r:

uE

~}.

9l(t) is called the reachable set in time t, and we show in Appendix I that 9P(t) is closed, bounded, and convex for all t 2 0. In fact d(t) = X(t)[xo

and 9P(t)

+ 9P(t)] =

{X(t)[xo

+ y]: y E 9P(t)}

= X-l(t)d(t) ­ x o'

Since Xl E d(t*), we have Yl E 9P(t*), and clearly t* is the smallest time for which Yl E 9l(t) (otherwise we would have Xl E d(t) for t < t*, a contradiction). If we think of the control problem in terms of these sets 9P(t), t > 0, we know that at t = 0, 9P(t) = {OJ, and as time increases 9P(t) grows and eventually intersects the point Yi - Our problem becomes, find the first time t* at which 9P(t) intersects Yl. We should expect, as is shown in Hermes [3] that the first contact occurs when 9P(t*) just touches Y1, or that Yl must belong to the boundary of 9l(t*) (see Fig. 2). The supporting hyperplane theorem (Appendix 1) then implies that there is a nontrivial hyperplane H (with outward normal", say) sup­ porting 9P(t*) at Yl' In other words "T Y1 2 "T y

that is

"T(f~·

Yl(~)u*(r)dr, 2

tl(f~·

for all Y E 9P(t*) and

... ,

f~·

n « 0,

Yn(r)u*(r)dr)

ytCr)u(r) dt, ... ,

f~·

Yn(r)U(r)dr)

for all u E~.

Rearranging gives (with "T = ('11' ... ,'1n))

f~·('11Yl(r)

+ '12Y2(r) +... + '1nYn(r) )(u*(r)-

u(r))dr 2 0,

for all u E~,

12

II.

The General Linear Time Optimal Problem

Fig. 2

or

f~*

("Ty(t»(U*(t)-u(t»dt

This can happen only if u*(t) = sgn("Ty(t», r +1 sgn(x}= -1 { undefined

E

~ O.

(0, t*). Here

if x>O if x < 0 if x = O.

Hence we have just proven the following theorem, a special case of the Pontryaqin maximum principle [6]. Xl

Theorem 1 If u* is any optimal control transfering (1) from Xo to in minimum time t*, then there exists a nonzero" E R" such that t

E

[0, t*].

(4)

In the general case, with u* = (uf, ... , U;::)T, we can similarly show (5)

for i = 1, ... , m, t E [0, t*], where [ ]; denotes the ith component of the vector. To simplify the notation we shall henceforth abbreviate (5) by t

E

[0, t*].

(6)

2. Applications of the Maximum Principle

13

We can place this theorem in the form it is usually stated by noting that the function t/f(t) = "TX- 1(t) is the solution of the adjoint equation

if,(t) = - t/f(t)A(t) with initial condition t/f(0) = ing gives

"T. For, taking inverses and differentiat­

0= dd ("T) = dd (t/f(t)X(t)) = if,x . . t t = if,x

+ t/fk

+ t/fAX = (if, + t/fA)X

and consequently, if, = -t/fA. Further t/f(0) tions (4) and (6) then become

=

"T since X(O) = I. Rela­

0:::;; t:::;; t*.

u*(t) = sgn(t/f(t)B(tW,

(7)

If we define the Hamiltonian by

H(t/f, x, u) = t/f(Ax

+ Bu),

then the optimal control (7) satisfies

H(t/f, x, u*) = max H(t/f, x, u). ued

We shall return later to an in-depth discussion of the maximum prin­ ciple, but first we shall see how the characterization (7) allows us to determine time optimal controls. One immediate observation is that u* will be bang-bang whenever "TX- 1(t)B(t) is nonzero, and the times at which the control changes value are just the zeros of this function. These times are usually called the switching times of u*.

2. APPLICAnONS OF THE MAXIMUM PRINCIPLE Consider the control problem in Example 1 in Chapter I. The state equation is '

i(t)

=

[~ ~JX(t)

+ GJU(t),

Hence A

=

[~ ~J

B=[~J

X(O) = (Xo, Yo).

(1)

14

ll.

The General Linear Time Optimal Problem

and X(t) is the solution of X(t)

= AX(t),

X(O)

= I.

Then X(t) = eAt = I

+

Ant"

L ­n! 00

n= 1

and so

X-l(t)=e-At=[~

~tl

Consequently, from the maximum principle the optimal control has the form u*(t) = sgn(171t

+ 172)

for some (171)172) E R 2 •

(2)

We see immediately that u* has only one switch between 1 and -1 (as we conjectured). Note that when u* = + 1 we can solve (1) with initial condition x(O) = (xo, Yo) and obtain dX 1

(ft = X2'

dX 2 = 1 dt ' Xl

.

1

dX2

i.e., dXl = X2' x~

y~

=2+ XO- 2

Fig. 3

x(O) = (xo, Yo),

2. Applications of the Maximum Principle

15

(see Fig. 3). Note that X2 is always increasing, so we traverse the path in the direction indicated. When u = -1, we have Xl

= -tx~

+ (xo + ty~),

which is shown in Fig. 4. This time X2 is decreasing. The maximum principle tells us that the optimal control is either u* = + 1 (then u* = -1), or u* = -1 (then u* = + 1). So, bearing in mind that we have to end up at 0 by traversing arcs of the types in Figs. 3 and 4, we can easily construct the optimal trajectory (see Fig. 5).

- - - - - - f B - - - - - - - - - - - - - + - - - -.... Xl

Fig. 4

-":lIIo.

....

"'" "

..... .... 2bifJ 2

° 0,

24

N

X

25

26

II.

The General Linear Time Optimal Problem

where IX, f3 are constants (initial conditions for the adjoint equation) and 2 - k • In any case, r/J2 can have at most one zero, and u* has at most one switch. If we denote by J1. = Jb 2

+ 1 in the lower right­

the solution of(3) passing through (0,0) with u = hand quadrant and by

the solution of (3) passing through (0,0) with u = - 1 in the upper left­ hand quadrant, then the switching locus X2 = W(x 1 ) is as pictured in Fig. 11. The optimal control synthesizer is then for for

X2 X2

> W(xd and on (I' _) < W(x 1) and on (r +).

The verification of these details is exactly the same as in Section 2 Example 1.

... u

~-1

-------------t-------------~Xl

u

~

----_ ...

+1

Fig. 11

27

4. Further Examples of Time Optimal Control

Consider now the case of underdamping b2 the adjoint system, and we get

J

t/J 2(t) =

ae" sin(wt

-

k2 < 0. We can solve

+ f3),

where w = k - b . In other words, the switches of the optimal con­ trol are exactly nfca seconds apart. Each solution of the state equations (with u = + 1) 2

2

[.X X2lJ =

[0 1J[XlJ X2 + [OJ _k 2

-2b

1

is a spiral approaching a critical equilibrium point 0 + = (llk 2 , 0) as t ---. + (f) (see Fig. 12). Similarly for u = -1, each solution of

is a spiral approaching 0 _ = (-liP, 0) as t ---. (f) (see Fig. 13). We construct the switching locus as follows: Find the solution of S + passing through the origin and unfold it, that is, take S 1 and reflect it back to Sz to etc. (see Fig. 14). This defines the switching locus = W(Xl) for ~ 0. For < 0, we set W(X l) = - W( This gives us the result shown in Fig. 15. Then it can be shown [5] that if P 1 and P 2 are colinear with 0 +, it takes exactly nk» seconds to traverse the optimal trajectory (S+) between PI and P 2, so as in the case of the undamped oscillator, we start off at our initial point (xo, Yo), with u = + 1 if it lies below W, u = -1 if above W, and continue until we hit W. Then the control switches sign, and by

s;

s;

Xl)'

X2

Xl

Xl

....--+------+---+-+-Herl-+-+--f-- x 1

Fig. 12

----+--I-~~t_t1---Xl

Fig. 13

-- .... S1

I

/

-------:-~+_--+-__._--~-x, I

,

\

S' 2

Fig. 14

u = +1

Fig. IS

S.

Numerical Computation of the Switching Times

29

the maximum principle we go for n/w seconds with the control at this value. However, it takes us exactly nk» seconds to rehit W, and so every time we cross W the control changes sign. This process continues until we hit r _ or I' +, and then we come into 0 by switching to u = - 1 or u = + 1, respectively. The synthesizer can be defined by '¥(Xb X2 ) = {

for x2>W(xdandonr_ for X2 < W(Xt) and on I' +

- I +1

and the optimal trajectories are solutions of x

+ 2bx + k 2 x

= t/J(x, x),

x(O)

=

Xo,

x(O) = Yo.

5. NUMERICAL COMPUTATION OF THE SWITCHING TIMES In this section we discuss a method for the numerical solution of time optimal control problems

x = Ax + bu.

(1)

Suppose the optimal control u* transferring Xo to 0 in minimum time is bang-bang with r switches. Without loss of generality, suppose that the first action of u* is -1 (see Fig. 16). Then, we must have

o = ~+At{XO

+

£r e-Atbu*(t)dt)

i.e.,

+1

-11------

Fig. 16

30

II.

The General Linear Time Optimal Problem

In other words, multiplying both sides by A and integrating yields Ax o = _(e- At1b - b) + (e- At2b - e- At1b) + ... + (-l)'(e-Atrb _ e- Atr- 1b) i.e.,

Ax o = b - 2e- At1b + 2e- At2b + ... + (_l)'e-Atrb.

(2)

If matrix Ais normal (commutes with its adjoint), has eigenvalues AI" .. , An' and corresponding orthonormal eigenvectors {x., .. , Xn } [7], then n

eAtb =

L

eAjt(b' X)Xj

j= 1

i.e.,

e-Atb =

n

L j= 1

e-Ajt(b' X)Xj .

So we can expand (2) as n

L Aj(XO • Xj)Xj = L [(b' X)Xj -

2e- Ajt1(b • X)Xj

j= 1

+ 2e- AjI2(b' X)Xj + .. , + (_l)'e-Ajtr(b' x)xil

(3)

Now using the orthonormality of the {x), (3) can be alternatively written as a set of n nonlinear equations in r unknowns t 1 , ••• , t.:

+ Aj(Xo'Xj) = (b' xj)(l - 2e- Ajt, + 2e- Ajt2 - 2e Ajl3 + ... +( -1)'e­ Ajtr) for j = 1, 2, ... , n; that is, -AI(XO' x.) + [b : x 1)(1 - 2e- A,t, + 2e- A,t 2 + ... + (_lYe-A,t r) = 0 -Aix o' Xn) + (h : xn)(l - 2e- Ant, + 2e- Ant2 + '" + (_l)'e- Antr) = O.

Note. It has been shown by Feldbaum [6, Chap. 3, Theorem 10] that if {AI' ... , An} are real, then r ~ n - 1.

Take as an example the harmonic oscillator, Example 1, Section 4,

x=

Ax

+ bu,

where x(O) =

G:J.

5.

31

Numerical Computation of the Switching Times

Here we have and Hence Ax o = [

YOJ

-x o

and from Example 1, e-Atb = [-sintJ.

cost

Then (2) becomes

YoJ = [OJ - 2[-sint 1J + 2[-sint 2J + ... + (_l y [ - sin tr J [ -x o 1 cos r, cos r, cos r, or "

smt 1 - s m t2 1 - Zcos r,

+ ·· , +

Yo

( -1)'+ 1 .

2

smtr = 2

+ 2cost 2 + ... + (-I)'cost r

= -Xo.

These equations in general have multiple solutions; however, by normality, the problem minimize t,

subject to

0< t 1 < t 2 < ... < t,

and

(-Iy+ 1 . . . f 1(t1,···,t r ) =smt 1 - s m t2 + · · · + 2 smtr f2(t1,"" t r ). = 1 - Zcos r,

( -1)'

-

Yo 2

°

= ,

+ 2 cos t 2 + ... + -2- cos r, -

Xo

=

°

must have a unique solution, namely, the switching times of our optimal control u*. So, setting t = (t 1> ' •• , tr ) , the problem has the general form P: minimize h(t)

subject to

gi(t) >

° and

jj(t) = 0, j = 1, 2,

i = 1, ... , r.

32

II.

The General Linear Time Optimal Problem

This is a mathematical programming problem that could be solved by Lagrange multiplier methods. A better method is to make the following transformation: t 1 = yi,

tz =

YI + yL

then setting fj(y) = jj(t), P is equivalent to pT:

minimize

YI + ... + Y;

subject to Jj(y) = 0, j = 1, 2,

and we have removed the inequality constraints. One method, which seems to work, for solving pT is to solve

pTT:

minimize YI + ... +

Y; + A(f!(y)Z + f!(Y)Z),

increasing A until successive solutions y coincide to a desired accuracy (this is an example of a penalty method) [2,3]. For example, with X o = 1, Yo = 0, the switching times are t 1 = 0.505 sec, t z = 1.823 sec.

Problems 1. Consider a control process described by x + bx = u for a real constant b, with the restraint lu(t)/ ::s; 1. Verify that the response x(t) with x(O) = Xo to a control u(t) is x(t) = e-btx o + e- bt

f:

ebSu(s)ds.

(a) If b ~ 0, show that every initial point can be controlled to Xl = O. (b) If b < 0, describe precisely those points Xo that can be steered to Xl = O. 2. In the control process in Problem 1, show that the control trans­ ferring Xo to Xl = 0 in minimum time (when this is possible) is bang-bang, indicate how many switches it can have, and show that it can be synthesized by u(t)

=

-sgn(x(t)).

Compute the minimum time t in terms of X o and b.

Problems

33

3. Suppose that you have been given a contract to ship eggs by rocket from New York to Los Angeles, a distance of 2400 miles. Find the shortest time in which you can do this without breaking any eggs. You may assume the path traveled is a straight line, and neglect friction, the rotation of the earth, etc. The only stipulation is that the eggs break if the acceleration exceeds 100 ft/sec 2 . 4. Calculate the minimum time to transfer from the initial point (1, 0) to the origin (0, 0) for the system

x+X=U,

IU(t)1 ~ 1.

What are the switching times of the optimal control? 5. Find the optimal trajectories and switching locus for the problem of reaching the origin in minimum time for the control system Xl = X z

lUll ~ Iuzl s

+ Ut>

Xz = -Xl + Uz ,

1,

1.

6. Find the control that steers the system Xl(O) = 1,

xz{O) = 0,

to the origin (0,0) in t = 1 and minimizes the cost C(u) = sup IU(t)l· 0:51:51

°

[Hint: If for the optimal control u*, C(u*) = k, show that this problem is equivalent to finding that number k > for which the minimum time t* to reach (0,0) from (Ilk, 0) for the system Xz =

lui

U,

~ 1

is t* = 1.J 7. For the control system

lUi ~

1,

°

show that the minimum time control steering X o = - 1 to X o = is u == t, which is not bang-bang. 8. Prove the Corollary to Theorem 3.1. 9. Discuss the time optimal control to the origin for the system,

= Xz = Xl

U1

+ Uz,

Ul -

Uz ,

lUll ~

1,

Iuzl ~ 1.

34

Il,

The General Linear Time OptimalProblem

10. Show that the maximum of the Hamiltonian defined in Section 1 is constant in time if the control system i = Ax + Bu is

autonomous. 11. Prove that a point Yl E flt(t) is hit by a unique trajectory if and only if y, is an extreme point of 9l(t). References

[lJ D. Bushaw, Optimal discontinuous forcing terms, in "Contributions to the Theory [2J [3J [4J [5J [6J

[7J

of Non-linear Oscillations," pp. 29-52. Princeton Univ. Press, Princeton, New Jersey, 1958. A. V. Fiacco and G. P. McCormick, "Non-linear Programming: Sequentially Unconstrained Minimization Techniques." Wiley, New York, 1968. H. Hermes and J. P. LaSalle, "Functional Analysis and Time-Optimal Control." Academic Press, New York, 1969. M. R. Hestenes, Multiplier and gradient methods, J. Optim. Theory Appl. 4, 303-320 (1969). E. B. Lee and L. Markus, "Foundations of Optimal Control Theory." Wiley, New York, 1967. L. Pontryagin, V. Boltyanskii, R. Gramkrelidze, and E. Mischenko, "The Mathe­ matical Theory of Optimal Processes." Wiley (lnterscience), New York, 1962. G. Strang, "Computational Linear Algebra." Prentice-Hall, Englewood Cliffs, New Jersey, 1980.

Chapter III

1.

The Pontryagin Maximum Principle

THE MAXIMUM PRINCIPLE

Consider the autonomous control problem: (1) Xi = h(Xl>' .. ,Xn , Ub' differentiable in R" x n.

.. ,Um ) ,

i = 1,2, ... n, with f continuously

We are given (2) the initial point Xo and (possibly) final point Xl' (3) the class Ll of admissible controls, which we take to be all piece­ wise continuous functions u, with u(t) E n,n a given set in B", (4) the cost functional C(u) =

f~' fo(x(t), u(t)) dt,

where fo is continuously differentiable in W x n. Define for

X"

u, cP

E

R" x B" x R", CPo

E

R,

H(x, u, cp) = CPofo(x, u) + CPt!I(X, u) + CP2f2(X, u) = ofo(x, u) + cP • f(x, u)

(H is called the Hamiltonian) and M(x, cp) = max H(x, v, cp). YEn

Then the maximum principle can be stated. 35

+ ... + CPnfn(x, u)

36

III.

The Pontryagin Maximum Principle

Tbeorem 1 (Pontryagin [4]) Suppose u* is an optimal control for the above problem and x* is the corresponding trajectory. Then there exists a nonvanishing function cp*(t) = (cpt(t), . . . , cp:(t» and cp~ such that oH i = 1,2, ... n, (a) x:", = -OCPi = i(x* u*), 1 , (b) 0 is a given constant. In this example the control u is not constrained. In our previous formulation,

1 = IXX + f3u, 10 = ax 2 + u2 , H(x, U, l(P - S) + q>z( -AP)

and . oH q> 1 = - ill = - (- h) = h, . oH qJz = - oS = -( -qJd =

qJl'

qJl(T) = qJz(T) = 0, as this is a free-end-point problem. Solving the adjoint equations gives C1

= -hT,

so qJl(t) = ht - hT, ¢z(t) = qJl(t) = ht - hT, qJit) =

ht Z

2 -

(hT)t

+ c z ; qJz(T) = 0.

Therefore Cz

hT z

hT z

(hT)t

+ 2·

= hT Z --2-=2'

qJz(t) =

ht Z

2 -

hT Z

Now, applying the last stage of the maximum principle, we see that H(I*(t), S*(t), P*(t), q>t(t), q>~(t))

=

max H(I*(t), S*(t), P, qJ!(t), qJ~(t»,

l'sPsP

for all

t E (0,T).

This maximum is achieved at the same time as the maximum of max (- cP + qJ!(t)P -

l'sPsP

= max P( -

AqJ~(t)P)

l'sPsP

That is, denoting

W) =

-

C

+ qJ!(t) -

AqJ~(t),

C

+ qJ!(t) -

AqJ~(t».

m. The Pontryagin MaximumPrinciple

46

the optimal control for 0

t

~

~

T is given by

p P*(t) = _upnknown {

when

W»O

when when

~(t)

= 0

~(t)


0, the only optimal strategy is P*(t) =

E,

0

~

t

~

T.

Example 2 (Isaacs [3J) We return to Example 3 of Chapter 1. For simplicity, we assume that the constant k = 1 and

x = ux, max

x(O) = x o,

fo (1 T

u(t))x(t)dt.

The Hamiltonian is #(x, u, cp)

= (u -

l)x

+ tpux,

and the adjoint equation is 0. For the moment, we shall assume that Ud and X d are sufficiently large that the minimization of the cost (3) keeps us in the region x 2 0, u 2 0. In Chapter 9 we shall consider inventory problems with state and control constraints. For this problem, the Hamiltonian is H(x, u, t/J) = - c(u - UdV - h(x - Xd)2

+ t/J(u - d),

and the adjoint equation is.

. oR t/J* = - ax = 2h(x*

t/J*(T)

- Xd),

= 0,

(4)

since (2), (3) is a free-end-point control problem. The maximum of the Hamiltonian over all U E R will occur at

oH au

= - 2c( u - Ud)

+ t/J

= 0,

that is, u*(t)

t/J*(t)

=~

+ uit),

05, t 5, T.

(5)

Substituting (5) into (2) gives x*(t) =

t/J;~t)

+ uit) -

d(t),

05, t 5, T,

x*(O)

= xo,

(6)

3. More Examples of the Maximum Principle

49

which, together with (4) forms a two-point boundary-value problem for the optimum trajectory and adjoint variable. Rather than solve this problem directly, we attempt a solution for the adjoint variable in the form ljJ*(t) = a(t) + b(t)x*(t),

(7)

0,

t~

for some functions a and b. The advantage is again that once the func­ tions a and b are determined, relation (7) and (5) gives the optimal con­ trol in feedback fomi. Differentiating (7) gives

tfr* = d + bx* + bx", and substituting (6) and (4) for X, tfr results in . 2h(x* - xd ) = d + bx*

+ b (ljJ* 2c + Ud

-

d) .

Plugging in (7) for ljJ* results in d(t) + b(t)(uit) - d(t»

+ 2hxd(t)

+ a(t)b(t) + (b(t) + b 2c

2(t)

2c

_ 2h) x*(t)

= 0 for all 0 ~ t ~ T.

(8)

This will be satisfied if we choose a and b such that b(t) d(t) + b(t)(Ud(t) - d(t»

From ljJ*(T)

=

+ b;~)

_=

0,

(9)

= O.

(10)

2h

a(t)b(t)

+ 2hxit) + ~

0, without loss of generality we can also suppose

= 0, a(T) = O.

(11)

b(T)

(12)

The equation for b is a Ricatti equation, which can be solved by the to give substitution b(t) = ~(t)g(t)

b(t)=-2cj!jtanh(j!j(T-t)}

O~t~T.

(13)

When (13) is substituted in (10) we are left with a linear equation for a, which can be solved by variation of parameters.

50

HI.

The Pontryagin Maximum Principle

For simplicity, we take a particular case Ud(t) = d(t), xd(t) = Cd' a constant, for 0 ::;; t ::;; T. That is, the firm wishes to have the production rate match the demand rate while maintaining a constant level of in­ ventory. Now a satisfies .( )

at

a(t)b(t) 2h +2 - + Cd =

0

a(T) = 0,

,

which, when solved, gives (14) substitution of (14) and (13) into (5) gives the feedback control law

U*(t'X)=~[Cd-X]tanh(~(T-t))+d(t),

O::;;t::;;T.

(15)

As a consequence the optimum control rate is equal to the demand rate plus an inventory correction factor, which tends to restore the inven­ tory to the desired level Cd. Further computation gives the optimal inventory level as x*(t) =

Cd

+

(xo -fi:T:. Cd) cosh [~ - (T - t)] , cosh('\I hlc T) C

0::;; t

s:

T.

(16)

We can see from (15) and (16) that if we start out with Xo = Cd' the de­ sired inventory, we remain at that level and meet the demand through production. If we start out away from Cd' the inventory level asymptot­ ically moves toward the desired level.

Example 4 Moon-Landing Problem (Fleming and Rishel [2]) This problem was described in Chapter 1. The state equations were

h= v, iJ = -g

m=

+ ulm

with cost

C(u) = min IT u

Jo

-ku,

where

os

u(t) ::;; 1,

(h(O), v(O), m(O)) = (ho , Vo, M h(T) = 0,

v(T) = 0,

+ F),

m(T) > O.

u(~) d~,

(17)

3. More Examples of the Maximum Principle

51

The Hamiltonian is H(h, D, m,u, t/J 1> t/J2' t/J3)

U + t/J1 D + t/J2( -g

=-

+ u/m) -

kt/J3U.

The adjoint equations become

.

t/Jl

aH -a;;- =

=

.

0,

aH

t/J2 = -Ji;= -t/J1>

tfr 3 -

(18)

aH _ t/J22U am - m

-

'

with t/J3(T) = 0 since m(T) is not specified. The maximum with respect to u of the Hamiltonian will occur at the same point as max [-u

O'::;u:sl

+ t/J2u/m -

kt/J3U],

that is,

u'(t)

when when when

~ {:dCfined

1 - t/J2(t)/m + kt/Jit) < 0 1 - t/Jz(t)/m + kt/Jit) = 0 1 - t/J2(t)/m + kt/J 3(t) > O.

Note that for the problem to be physically reasonable, max thrust> gravitational force, that is,

1> (M In general, if max thrust

+ F)g

1 --->g M+F .

or

= a (0 ~

u(t)

~

a), then

IX

M+F>g·

Our intuition would lead us to expect that the optimal control is first a period offree fall (u* == 0) followed by maximum thrust (u* == 1), culminating, we hope, in a soft landing. Assuming this to be the case, we shall first show how to construct such an optimal control and then

52

m.

The Pontryagin Maximum Principle

use this information to show that it is the unique solution of the maximum principle. Suppose that u*(t) = + lover the last part of the trajectory [e, T]' Remember that h(T) = 0, v(T) = 0, m(T) is unknown, and m(e) = M + F since we have coasted to this point. Then the solution of (17) is h(e) =

_ ~ g(T _ 2

v(e) = g(T m(e) = M

1')2 _

...

e) + ~

k

In

M+ (M + (M + k2

FIn

F - k(T - e)) _ T M +F k

F - k(T - e)) M+F '

e '

(19)

+ F.

If we plot h(e) againstetc), we get the curve shown in Fig. 4. Clearly this curve is the locus of all (height, velocity) pairs that we can steer to (0, 0) with full thrust + 1. There are some physical restrictions on the length of this curve. Namely, as the spacecraft is burning fuel at a rate k, the total amount of fuel will be burned in time F/k seconds. Consequently,

°s

T-

es

F/k.

Over the first part of the trajectory [0, e], we "free-fall," u* = 0, and we have h(t) = -tgt 2 + vot + ho,

v(t) = -gt + vo, m(t) = M + F,

\

\

h \

Fig. 4

(20)

3. More Examples of the Maximum Principle

53

or, in the phase plane, 1 h(t) = ho - 2g [VZ(t) - v~J,

os

t

~~.

(21)

As expected, this curve is a parabola. Using this strategy we free-fall following trajectory (21) until we hit the curve constructed previously in Fig. 4., where we switch to full thrust (see Fig. 5). The switching time ~ is the time at which (19) and (20) cross. We now show that this choice for u* satisfies the maximum principle. Let From the adjoint equations (18)

= Az - Alt, J/J3(t) = A3' m(t) = k(~ - t) + M + F, J/Jz(t)

O~t~T,

o~

t

~~,

~ ~

t

~

T,

we see that J/J3(r) = A3

it

+ J~ [k(~

(Az - Alt)

_ t) + M + FJz dt

Since the switching function

os is zero at t =

t

~

T,

~,

r(J 0, and switch to maximum thrust when first 'P(h, v) = o.

Problems 1. Find the production rate P(t) that will change the inventory level l(t) from 1(0) = 2 to 1(1) = 1 and the sales rate S(t) from S(O) = 0 to S(I) = 1 in such a way that the cost C = gP 2 (t)dt is a minimum. Assume j(t) = P(t) - S(t),

S(t) = (P is unrestricted).

2. Minimize C(u) = t

- P(t)

g U(t)4 dt subject to

x = x + U,

x(O) = x o ,

x(l)

= O.

3. Find the extremals for (a)

So" «y')2 -

(b)

SOl

«y')2

y2)dx,

+ 4xy')dx,

y(O)

= 0,

y(O) = 0,

y(n)

= 0,

y(l) = 1.

56

m.

The Pontryagin Maximum Principle

Remember that the extremals for

S:

j(x, y'(x), y(x))dx,

= Ya,

y(a)

y(b) = Yb

are the solutions of the Euler-Lagrange equation d

/y - dx (/y,) = 0 that satisfy the boundary conditions. 4. Consider the following model for determining advertising expendi­ tures for a firm that produces a single product. If S(t) is the sales rate at time t and A(t) the advertising rate at time t, restricted so that 0::;; A(t)::;; A, then we assume S(t) =

- AS(t) + Y S~ A(t - r)e- dt,

A, y > O.

t

We wish to choose A to maximize g S(t)dt, the total sales over period [0, TJ. 5. Find the optimal control which maximizes the cost SOlOO

x(t)dt

subject to the system dynamics x(t) = -O.1x(t)

+ u(t),

os

u(t) ::;; 1,

x(O)

= X o'

6. Apply the maximum principle to the control problem

2(t)) dt,

maximize S02 (2x(t) - 3u(t) subject to

x(t)

IXU

= x(t) + u(t),

x(O)

IX

~ 0

= 5,

0 ::;; u(t) ::;; 2.

7. Discuss the optimal control of the system X2 =-u

from x(O) = (x?, x~) to (0,0), which minimizes the cost

S~

xi(t) dt.

8. What is the solution of the maximum principle for the problem of minimum fuel consumption minimize

S;' .,)1 + u(tf dt,

References

57

for the system

x = u, x(O)

= x?,

X(O) = xg,

lui ~

1,

x(t 1 ) ~ 0,

Does the solution of this problem change if the cost is replaced by

f~l lu(t)1 dt ? 9. A function E(x*, u*, u, t) is defined by

E(x*, u*, u, t) = f(x*, u, t) - f(x*, u*, t) + (u* - u)fu(x*, u*, t). Show that if (x*, u*) is an optimal solution of the calculus of variations problem of Section 2, then

E(x*, u*, u, t) ~ 0

for all

u, t.

(This is the Weierstrass E function necessary condition for a strong extremum.)

References [1] A. Bensoussan, E. Hurst, and B. Naslund, "Management Applications of Modern Control Theory." North-Holland Publ., New York, 1974. [2] W. Fleming and R. Rishel. "Deterministic and Stochastic Optimal Control." Springer-Verlag, Berlin and New York, 1975. [3] R. Isaacs, "Differential Games." Wiley, New York, 1965. [4] L. Pontryagin, V. Boltyanskii, R. Gramkrelidze, and E. Mischenko, "The Mathe­ matical Theory of Optimal Processes." Wiley (Interscience), New York, 1962. [5] H. Sagan, "Introduction to the Calculus of Variations." McGraw-Hili, New York, 1969.

This page intentionally left blank

Chapter IV

The General Maximum Principle; Control Problems with Terminal Payoff

1. INTRODUCTION In this chapter we consider the maximum principle for the general non autonomous control problem. Namely, suppose that (1) x = f(x, t u) is the control process and f is continuously dif­ ferentiable in R" + 1+ m; (2) we are given sets X 0, X 1 in R" to be interpreted as sets of allowable initial and final values, respectively; (3) d = {u: u is bounded, piecewise continuous u(t) Ene B", and u steers some initial point Xo E X 0, at a fixed time to, to some final point Xl E Xl' at time t d; (4) The cost of control u is C(u) =

I

II

10

fo(x(t), t, u(t)) dt,

fo continuously differentiable in W+ m+ 1 .

Define the Hamiltonian for x=(x1, ... ,xn),U=(Ul""'Um) , and t/J = (l{IO,l{Il,'" ,l{In) as H(x, u, t, t/J) = l{IofO(x, t, u) + l{I tfl(X, t, u) + ...

and define M(x, t, t/J) = max H(x, u, t, t/J). UEC!

59

+ l{Infn(x, t, u)

60

IV. The General Maximum Principle

Theorem 1 (Lee and Marcus [2J) If u*, x* are optimal for the above problem over [to, tt], then there exists a nontrivial adjoint response t/f* such that (i) x*(t) = f(x*(t), t, u*(t)), (ii) tiJj(t) = (iii) "'~

:~

=

ito

"'t(t)

:~j

(x*(t), t, u*(t)),

is a nonpositive constant (~O),

(iv) H(x*(t), u*(t), t, t/f*(t)) tinuity. Further

j = 1, ... , n,

and

= M(x*(t), t, t/f*(t)), at points of con­

It

~

M(x*(t), t, t/f*(t)) = Jt'

LJ

t'i=O

"'{(s)

oj;

ot (x*(s),s, u*(s)) ds

----!.

and, hence

M(x*(tt), tt, t/f*(tT)) = O. If X 0 and Xl (or just one of them) are manifolds in R" with tangent spaces To and T l at x*(to) and x*(tt), respectively, then t/f* can be selected to satisfy the transversality conditions (or at just one end) and In the most usual case Xl (and possibly X 0) will be of the form (1)

Xl = {X:lh(X)=O,k= 1,2, ... ,1}.

for some given real-valued functions {gl,' .. , g/}' For example, our fixed-end-point problem could be formulated x*(tf) = Xl or

x: = 0,

gr(X*(tl)) = X:(tl) -

r = 1,...

,no

(2)

If the functions {g 1" •• , gl} are differentiable, then the transversality condition can be simply written

t/f*(tt)

=

I

L AkVgk(X*(tt)),

k=l

for some constants Vb ... , AI}' By (1)atthe terminal point we have also

gk(X*(tf))=O

for

k= 1,... ,1.

For example, in the fixed-end-point problem,

Vgr(x*(tf)) = (0,... ,1,0 ... 0). rth

2. Control Problems with Terminal Payoff

61

So (3) becomes t/1*(tf) = (A'I,A2,'" ,An), and in this case (as we have been assuming all along) t/1*(td is unknown. In the other case that we have been using, the free-end-point problem, Xl = W; consequently the only vector orthogonal to the tangent space is 0, that is, t/1*(tf) = O.

2. CONTROL PROBLEMS WITH TERMINAL PAYOFF In many applications of control theory the cost takes the form

°

C(u) = g(x(T»

+ SOT fo(x(t), t, u(t»

(1)

dt,

where T> is fixed and qi R" -+ R is a given continuously differentia­ ble function that represents some terminal payoff, or salvage cost, at the final time T. We can convert such problems into the standard form by adding an extra state variable X n + I (t) satisfying

xn+1(t) = 0,

rr;

xn+I(T)

= g(x(T»/T.

Then g(x(T» = x n+ l(t) dt, and the cost (1) is now in purely integral form. When the maximum principle is applied to this new problem, one finds the only change is in the transversality conditions, which now become t/11(T)

= - ~g (x*(T», ox,

i

(2)

= 1,2, ... ,no

Example 1 A Continuous Wheat-Trading Model (Norstrom [3J) We consider the following model for a firm engaged in buying and selling wheat, or a similar commodity. The firm's assets are of two types, cash and wheat, and we represent the balance of each quantity at time t by xl(t) and X2(t), respectively. The initial assets Xl(O) and xiO) are given. The price of wheat over the planning period [0, TJ is t ::; T. assumed to be known in advance and is denoted it by p(t), The firm's objective is to buy and sell wheat over the period [0, TJ so as to maximize the value of its assets at time T, that is, it wishes to maximize

°::;

(3)

62

IV.

The General Maximum Principle

Ifthe rate of buying or selling wheat at time t is denoted by u(t), (u(t) > 0 indicates buying; u(t) < 0, selling), then we could model the operation of the firm by the differential equations

xlet) = - axit) x2 (t) = u(t),

(4)

p(t)u(t),

(5)

where a > 0 in (4) is the cost associated with storing a unit of wheat, and the term p(t)u(t) indicates the cost (or revenue) of purchases (or sales) of wheat at time t. We have also the natural control constraint M

s

u(t)

s M,

M, M

given constants.

(6)

The Hamiltonian for this problem is

H=

r/Jl( -ax2 -

pu)

+ r/J2U,

(7)

and the adjoint equations are (8)

(9) with transversality conditions

by (2). In this case (8) and (9) are independent of the state variables and can be solved directly to give r/Jl(t)

=

r/J 2(t) =

(10)

-1,

o s t s T.

- a(t - T) - peT),

(11)

This enables us to write out the Hamiltonian explicitly as

H = aX2 + p(t)u

+ u( -

a(t - T) - peT))

= u(p(t) - a(t - T) - p(T))

+ aX2'

H will be minimized as a function of u if u*(t) =

M (buy) M (sell) { undetermined

when when when

pet)


peT) - a(T - t) pet)

= peT) -

a(T - t).

(12)

63

ControlProblems withTerminal Payoff

2. 7

6

5

4

3

2

x

Sell

.. I

Buy

I

I

1

1

I

I

I

o

5

4

3

2

7

6

Fig.!

Figure 1 illustrates this solution for a particular price function, T=7,

O(=t,

M=-I,

M=I,

- 2t + 7 p(t) =

{

_

2:: 1~ t-

2

x 1 (0) = 50, X2(0)

= 1,

Osts2

~ ~ :~ 5s t

s

: 7.

From (11), it follows that ljJ 2(t) = - (tt + !). The optimal control is seen to be Os t < 4.6 u*(t) = {-I (sell) 4.6 < t < 7. 1 (buy) The simple optimal policy (12) has some shortcomings, particularly for long planning periods. It is very much dependent on the final price of wheat p(T), and not on the price between t and T. If T is very large,

64

IV.

The General Maximum Principle

then for small t the function t -+ p(T) - iY.(T - t) will be negative, and (12) would require us to sell wheat. This would mean (for T sufficiently large) that the supply of wheat x 2 (t) would eventually become negative, which would gain us cash with the state equation having the form (4). So for long planning periods, we should modify (4) to assign a cost for short-selling (that is, X2 < 0) or else forbid short-selling by adding the state constraint.

t

~

o.

We shall return to this problem in Chapter VIII, when we discuss state­ constrained control problems.

3. EXISTENCE OF OPTIMAL CONTROLS In this section we summarize some of the main existence theorems of optimal control theory. From now on we will assume the set of admissible controls is

11 = {u: u is bounded, measurable, and u(t) E n, for all t}, for some compact set n c R". In order to have a reasonable existence theory, it is necessary to consider measurable controllers, not just piecewise continuous. This may seem to lead us away from physical reality. On the other hand, once the existence of an optimal control is guaranteed, the necessary conditions, such as the maximum principle, can be applied without qualm, and in many cases they will show that the optimal controls are indeed piecewise continuous or better. The first result is essentially due to Fillipov. For a detailed proof, we refer to [2].

Theorem 1 Consider the nonlinear control process

x = f(x, t, u),

t> 0,

x(O) E X o ,

where f is continuously differentiable, and (a) the initial and target sets X 0, X 1 are nonempty compact sets in R"; (b) the control restraint set n is compact, and there exists a control transferring X 0 to X 1 in finite time; (c) the cost for each u E 11 is C(u) = g(x(ttl)

+ I~'

fo(x(t),t,u(t))dt

+

max {y(x(t))},

lello,lll

3. Existence of Optimal Controls

65

where fo is continuously differentiable in R n + 1 +m and g and yare continuous on R". Assume (d)

there exists a uniform bound Ix(t)!:$;; b

O:$;; i

s: t 1 ,

For all responses x(t) to all controllers uEA; (e)

the extended velocity set

V(x, t)

=

{(fO(x, t, u), f(x, t, u) : UEO}

n

is convex in R + 1 for each fixed (x, t). Then there exists an optimal controller u*(t) on [0, t1], u* E A, minimizing C(u).

Corollary 1 For the time optimal problem if (a), (b), (c), (d) hold, and V(x,t) = {f(x,t,U)IUEO} (the velocity set) is convex in W for each fixed (x, t), then time optimal controls exist. Applications of the theorem and its corollary are plentiful. Consider our moon-landing problem: minimize

S~l

subject to

u(r)dr

h = v,

iJ

=

-g

+ m-1u,

-g

+ m-1u,

rh = -ku.

Hence V«h, v,m),t)

= {(u, v,

-ku): O:$;; U:$;; l},

which is convex for each fixed «h, v,m), t). Hence an optimal control for this problem exists, and all our previous computations are justified. Similarly for Example 3 in Chapter I, minimize

S:.(1 - u)xdt

subject to

x=

kux,

O:$;; u :$;; 1.

We have V(x,t) = {«I - u)x,kux): O:$;; U:$;; I} = {(x - ux, kux): O:$;; U:$;; l},

There are, however, many problems where optimal controls do not exist [1].

66

IV.

The General Maximum Principle

Example 1 Consider the time optimal problem of hitting the fixed target (1,0) with the system

Xl = (1 - X~)U2, x 2 = u,

Xl(O)

= 0,

xiO)

= 0,

IU(t)1

s

1.

We show first that (1,0) E d(1) but (1,0) fI d(1). For each positive integer n, subdivide [0, 1J into 2n equal subinter­ vals. Let I j = (jj2n, (j + 1)/2n),j = 0, 1, ... , 2n - 1. Define U

(t) =

n

{I

if t E I j , j odd if t E I j , j even.

-1

For example (see Fig. 2), let x(t; un) denote the solution corresponding to Un' Then

xiI: un) = I~ and

Ix it : un)1 s as n --+ 00. Thus Xt(1: un) =

=

un(.)d.

=

°

II; Un(.)d·l--+ °

for all n

uniformly in

t E [0, 1]

I: [1 - xi.: unfJu;(.)d.

fo (1 l

xi.: un(.W d. --+ 1

as

n--+ 00.

Hence x(1 : un) --+ (1,0); however, (1,0) fI d(I), since this would require a control u such that X2(t: u) == 0, or U == and then Xl(t: u) == 0.

°

Fig. 2

67

Problems

As an exercise, show that for any t > 1, (1,0) E d(t), and hence inf{t: (1,0) is reached in time t} = 1, but this infimum cannot be attained. As the final existence theorem we consider the time optimal problem when the state equations are linear in x. In this case we do not need any convexity. Theorem 2 (Olech and Neustadt [4J) control problem with state equation x(t) = A(t)x(t)

+ f(t, u(t),

Consider the time optimal x(O) = x o,

where f is (separately) continuous and the entries of A are integrable over finite intervals. Suppose /)., X 1 are as before. If the target set Xl is hit in some time by an admissible control, then it is hit in minimum time by an optimal control. Problems

1.

Use the maximum principle to solve the following problem: maximize (8Xl(18) + 4x2(18»

subject to Xl = 2x l + X2

x2 = 4x

2.

l -

+ u,

2u,

Resolve the control problem in Chapter 3, Problem 5, with the cost maximize x(l00).

3.

Discuss optimal control of the system . . . 1 ('II 2 d mllllIlllze"2 Jo Xl t

subject to Xl = X2

X2

+ u,

= -u,

and X(tl) = O. (Note that a singular arc u* =

!(xTf = const, is possible in this example.)

-4 -

x!, xfx!

+

4. Discuss the minimum time control to the origin for the system

lui ~

1.

68

IV.

The General Maximum Principle

5. Discuss the possible singular solution of the system Xl = X2, X2 = -X2 - XtU, x(O) = x'', with terminal restraint set X t

and cost

1

It

C(u) = "2 Jo (xi(t) + x~(t»

lui ~

1,

= {(Xt, X2): xi + x~ = I}, dt.

(Show that the singular solution trajectory is x! = ± x!, and that the optimal control there is u* = - (x!

+ x!)/x!­

6. Discuss the optimal solution of the fixed time control problem

lui s

1,

T

C(u) = fo (xi + luj)dt. Include the possibility of singular arcs.

7. Consider the following model for a rocket in plane flight. Let Xt(t) and X2(t) be the cartesian coordinates of the rocket; X3(t) = dxfd: and X4(t) = dx 2/dt the velocity components; X5(t) the mass of the rocket; Ut(t) and U2(t) the direction cosines of the thrust vector; U3(t) = - dx 5/dt the mass flow rate; c the effective exhaust gas

speed (positive constant); and g the magnitude of gravitational acceleration (positive constant). Define Ut

U2

= cos at, = cosfJ.

The state equations are then Xl

= x 3 ,

x2 = X4'

x3 = CUtU3/X5' x4 = CU2U3/X5 x5 = -u 3 ,

x(O) =

X o .

g,

The control restraint set is defined by

ui + u~

=

1,

0 ~ U3 ~ M.

References

69

Say as much as you can about the control that transfers the rocket from Xo to [" A, " " B] (A, B fixed) and minimizes

- f~

X3

dt .

(That is, we wish to transfer a rocket of given initial mass and position to a given altitude while expending a prescribed amount of fuel and attaining maximum range.) 8. Show that time optimal control transfering (0,0) to (1,0) does not exist in the example in Section 3.

References [1] H. Hennes and J. P. LaSalle, "Functional Analysis and Time-Optimal Control." Academic Press, New York, 1969. [2] E. B. Lee and L. Marcus, "Foundations of Optimal Control Theory." Wiley, New York, 1967. [3] C. Norstrom, The continuous wheat trading model reconsidered. An application of mathematical control theory with a state constraint. Working Paper, Graduate School of Industrial Administration, Carnegie-Mellon University, 1978. [4] C. Olech, Extremal solutions of a control system, J. Differential Equations 2, 74-101 (1966).

This page intentionally left blank

Chapter V

1.

Numerical Solution of Two-Point Boundary-Value Problems

LINEAR TWO-POINT BOUNDARY-VALUE PROBLEMS

We shall consider methods for the numerical solution of the following types of problem: (i) where (ii) (iii)

n first-order equations are to be solved over the interval [to, tr], to is the initial point, and t r is the final point; r boundary conditions are specified at to; (n - r) boundary conditions are specified at tro

Without loss of generality we shall take the problem in the form

Yi = gi(YI, Y2' ... ,Yn' t), Yi(tO) = c., Yi(t r) = Ci ,

= 1,

, n, i = 1, , r, i = r + 1, ... , n, i

(1)

where each gi is twice differentiable with respect to Yj' There can be considerable numerical difficulties in solving such problems. It has been shown for

Y = 16 sinh 16y,

Y(O)

= y(l) = 0,

that if we choose the missing initial condition, Y(O) = s, 71

72

V.

Numerical Solution of Two-Point Boundary-Value Problems

for s > 10- 7 , then the solution goes to infinity at some point in (0,1). (This point is ~ l6In(8/s).) The true solution is of course y(x) == 0, s x ~ 1 [2]. Let us first consider the linear case

°

y(t)

= A(t)y(t) + f(t),

(2)

where

= 1, ... , r, i = r + 1, ... , n.

Yi(tO) = c.; Yi(tr) =

i

Ci,

So c and f(t) are given. The adjoint system is defined as the solution of the homogeneous equation z(t)

=-

(3)

AT(t)z(t).

As usual we can write the general solution of (2) as (4)

where

k = A(t)X,

X(to)

= I.

Furthermore, the solution of (3) can be written (5)

and so (4) gives z(tr)Ty(ta

= z(trl X(tay(t o) + z(taT Jlo fIr X(tr)X(s)-lf(s) ds.

(6)

Substituting the transpose of (5) into (6) gives

z(taTy(t~)

= Z(to)Ty(t o) + fIr z(taTX(taX(s)-lf(s)ds. Jlo

Now, again from (5), z(s) = X(s)-TX(tr)Tz(t r),

and so (7)

1. Linear Two-Point Boundary-Value Problems

or in component form, itl Zj(tC)Yi(tC) -

it

Zi(tO)Yi(tO) =

i:

f

73

(8)

itl Zi(S)!;(S) ds.

Equations (7)and (8)are the basic identities in the method of adjoints, the first of the shooting methods we shall investigate. The method begins by integrating the adjoint equations (3)backward n - r times, with the terminal boundary conditions

z(ll(t c) =

-

0 0

0 0

0 0 0 1 0 0

rth,

Z(2)(t C) =

-

0 0 1 0

Z(3 l(tc) =

0

0

-

0 0 0 1

rth, (8')

0 0

z(n-r)(tc) =

-

0 0 1

This gives us n - r functions z(l)(t), Z(2 l(t), ... , z(n-r)(t), to

~ t ~ tc (of course in practice we do the integration at a discrete set of points). Then

z(m)(tc)Ty(tc)

=

n

L zlm)(tc)Yi(tc) = Yr+m(tC) =

i,,: 1

C

m = 1, ... , n - r,

r+m,

and so (8) becomes, with some rearrangement,

i=~

n 1

zlm)(to)Yi(to)

= Cr+m - i~l

i: i~l

n

r

z!ml(tO)Yi(tO) -

zlm)(s)!;(s) ds

(9)

for m = 1, ... , n - r. The set of equations (9) is a set ofn - r equations in n - r unknowns {Yr+ 1(to), Yr+ ito), ... , Yn(tO)}, which we can solve and thereby obtain a full set of initial conditions at t = to.

74

V.

Numerical Solution of Two-Point Boundary-Value Problems

Note that (9) can be rewritten

l

z ~ ~ 1(tO) Z~~2(tO) z~~ 1(tO)

z~~ 2(tO)

z~n+-[)(to)

z~n+-{)(to)

···

Cr+ 1

Z~1)(tO)]

z~2)(to) z~n-r)(to)

±

-

i= 1

zP)(tO)Ci -

lYr+ 1(to)J Yr+ 2(tO)

.. . .. .

Yn(t o)

ff ~>P)(t)};(t)

dt

0

(10)

c; -

it

1

i- L:f L

zln-r)(tO)C

zln-r)(t)};(t) dt

The inverse of the z matrix exists since {Z(1)(t r), ... ,z(n-r)(tr)} are linearly independent, and consequently so are {z(1)(t), . . . , z(n-r)(t)}

for any

t

E

[to,

trl

(This is a known fact from linear ordinary differential equations.) We can code this method as follows: 1. Set m = 1. 2. Integrate the adjoint equations backward from t r to to for the mth set of boundary conditions (8'). 3. Evaluate the mth row of (10). 4. If m = n - r, solve Eq. (10) for the missing initial conditions {Yr+1(tO),' .. , Yn(tO)}; go to item 6. 5. If m < n - r, set m = m + 1; return to item 2. 6. Now using the full set of initial conditions {Yi(tO)}i= 1, integrate (1) forward to obtain the solution of the boundary-value problem. Note that if n - r-> r, we save work by reversing the roles of the initial and terminal points. 2. NONLINEAR SHOOTING METHODS

Nonlinear two-point boundary problems can be solved by an iter­ ative process. We start with an initial guess {

(0) ( ) (OJ ( ) (0)( )} Yr+ 1 to , Yr+ 2 to , ... ,Yn to .

2. Nonlinear Shooting Methods

This will allow us to solve the Eqs. (1.1) to find y(O)(t), to ::; t we iterate according to the following scheme: set

s;

75

t r. Then (1)

Then by the usual Taylor-series expansion, we have as a first-order approximation i = 1, ... , n,

that is, (2)

(where J is the gradient of g), which is just a set of linear ordinary differential equations with variable coefficients. Furthermore, bYlk)(to) = 0, bYlk)(tr) = Ci - Ylk)(t r),

i = 1, ... , r, k = 0, 1, 2, i = r + 1, ... ,n, k = 0, 1,2,

, (3) .

Equations (2) and (3) define a linear two-point boundary-value problem for the correction vector by(k)(t), to ::; t ::; t., which we can solve by the previous method, the method of adjoints; namely, as before, we define the adjoint system to (2) i(k)(t)

= -Jnt)z(k)(t),

(4)

where J k = J(y(k)(t», and solve (4) backward n - r times with end­ point conditions

o o Q rth

J

o

o o

o o

o

Q

Q

Q

o 1

o o

o

1

o

o

o

1

Denote the solutions thus obtained at the kth iteration step by {z(l)(t), Z(2)(t),. . . ,z(n- r)(t)} (k)'

76

V. Numerical Solution of Two-Point Boundary-Value Problems

Then the fundamental identity tells us that

[

Z ~ ~ l(tO) z~~\(to)

Z~I)(tO)

Z~~2(tO) z~~ ito)

Z~2)(tO)

z~n-r)(to)

z~n+-{)(to)

lbY~k21(tO)]

]

bY~~.2(tO)

(k)

[t5yn

..

by~k)(tO)

l(tr)]

== t5y~k~ 2(tr) . (5) .. t5y~k)(tr)

[Note that (2) has no forcing term, simplifying Eqs. (5).J Solving (5) gives us a complete set of initial conditions t5y(k)(t O)' and we get the next choice for the initial conditions for y by setting y(k+l)(tO)

= y(k)(tO) + t5y(k)(t o).

Now We solve for y(k+ 1)(t),. t E [to, trJ, and return to (2) to calculate J(y(k+ 1», t5y(k+ 1), etc. We terminate whenever max{t5Ylk)(t r): i = r, r + 1, ... , n} is sufficiently small or k is too large. This is called the shooting method; we guess the unknown initial values, solve the equations, and then, on the basis of this solution, make corrections to the previous initial values. So in shooting methods our aim is to find the missing initial data. It can be shown that shooting methods are a special case of the Newton-Raphson method, and so, provided that our initial guess is sufficiently close, Jk is nonsingular, and the interval [to, trJ is not too large, the approximations converge quadratically. To recapitulate, the method of adjoints for nonlinear ordinary dif­ ferential equations is carried out as follows: 1. Determine analytically the gradient (og;/oy). 2. Initialize the counter on the iterative process. Set k = 0. 3. For k = 0, guess the missing initial conditions yIO)(to), i = r + 1, ... , n. 4. Integrate (1) of Section 1 with initial conditions Ylk)(t o) = ylk)(to),

Cj,

i = 1,2, , r, i = r + 1, , n,

and store y(k)(t). 5. Set the counter on the integration of the adjoint equations, m = 1.

3. Nonlinear Shooting Methods: Implicit Boundary Conditions

77

6. Calculate zlm)(t o), i = r + 1, ... , n, by integrating the adjoint equations (4) backward from t f to to, with final data z\m)(t c) as in Eq. (8) of Section 1, i = 1, ... , n. Note that in this integration the stored profiles y(k)(t) are used to i, j = 1, ... , n. evaluate the partial derivatives

og;j(JYj,

7. For the mth row of (5) form the right-hand side of (5) by sub­ tracting the specified terminal value Yi(tC) = Ci from the calculated value Ylk)(tf ) , i = r + 1,. , . , n, found in item 4. 8. If m < n - r, set m = m + 1 and go to 6. 9. Form the set ofn - r linear algebraic equations (5) and solve for (iy\k)(t O)' i = r + 1, ... , n. 10. Form the next set of trial values by ylk+ 1)(t o) = y\k)(t O)

+ by\k)(t O),

i= r

+ 1, ... , n.

11. Set k = k + 1; return to 4. 12. Terminate, whenever max{(iYlk)(tc) : i = r + 1, ... ,n} is suffi­ ciently small, or whenever k exceeds a maximum value. 3. NONLINEAR SHOOTING METHODS: IMPLICIT BOUNDARY CONDITIONS In this section we consider the two-point boundary-value problem of Eqs. (1.1) with implicit boundary conditions that are functions of both the initial and the terminal conditions: q;(Yl(tO) , Yz(t o),·· ., Yn(t O), Yl(t C),·· . Yn(tC))

=

0,

i

= 1, ... , n. (1)

Let us define the variation in qi as i = 1,2, ... , n,

(2)

Since qlrue = 0. Up to second order we can approximate (iqi by n

bqi =

.L

J~ 1

Oqi

~y.(t) U J

n

0

(iYNo)

Oqi

+ J~.L1 UY ~ .(t) (iYitc), C

i= 1,2, ... ,n, (3)

J

where

(o~~:o)

and

(o:j~~c)

are the gradients evaluated at y(t o) and y(ta, respectively. Equations (3) are n equations in 2n variables {(iy(to), (iy(tc)}. However, from the

78

V. Numerical Solution of Two-Point Boundary-Value Problems

fundamental identity of adjoints we can relate by(t o) and by(tc) as in Eq. (2.5) n

I

i= 1

zP)(tO)bYi(tO) = bYj(td,

j =

J, 2, ... , n,

(4)

where {z'!', ... , z(nl} are the solutions of the adjoint equations (5.2.4) with terminal conditions 1 0 z(ll(t c) =

Z(2

l(t c) =

0 0

0 0

0 1 0

z(nl(td =

(5)

1

0

If we substitute (4) into (3), we get bq; =

Oqi

n

I

j= 1

+

-0 ( ) bYj(to)

Yj to

.f

J= 1

O~(ti

0 ) YJ c

(± z~jl(tO) s= 1

bYs(to»),

i

= 1,2, ... ,no

(6)

i = 1,2, ... .n.

(7)

On rearranging (6), we find that

~~(jl bq, _~(~ - L. 0 ( ) + L. 0 ( ) zp (to)) bYp(to), p=l Yp to j=l Yj t r

Equation (7) is a set of n equations in n unknowns {by(t o)}, which can be solved, and from by(t o) weget our new guess for the initial data. To recapitulate: 1. Determine analytically the partial derivatives

Oqi ) ( °Yj(t o) ,

2. Initialize the counter on the iterative process. Set k = O. 3. For k = 0, guess the missing initial conditions y\O)(t o),

i=I,2, ... ,n.

4. Integrate (1.1) with initial conditions ylkl(to) and store the profiles

y\kl(t), i = 1,2, ... ,no

5. Using initial values Ylkl(to) and calculated final values ylk)(t r), evaluate bqi' i = 1,2, ... , n by (1) and (2).

Quasi-Linearization

4.

79

6. If max {c5q 1, c5q2' ... , c5qn} is sufficiently small, or if k is greater than the maximum terminate. Otherwise go to 7. 7. For each Ylt r) appearing in the implicit boundary conditions (1), integrate the adjoint equations (2.4) backward with terminal data (5). The profiles y!k)(t), i = 1,2, ... , n, that are stored in 4 are used to evaluate the derivatives (ogi/oYj)' Save {z(1)(to), ... , z(n)(t o)}. 8. Using the expressions in 1 for (oq;/oy), evaluate oq; oYj(to)'

Oqi oYNr)'

,n,

i,j= 1,2, ...

and form the left-hand side of (7). 9. Solve (7) for c5Yi(t O) and call the solution c5y!k)(t O) for the kth iteration step, i = 1,2, ... , n. 10. Form the next set of trial conditions y!k+ 1)(to) = y!k)(to)

+ c5y!k)(t o),

i = 1, ... ,no

11. Set k = k + 1; return to 4. 4. QUASI-LINEARIZATION Let us reconsider the system of n nonlinear ordinary differential equations:

Yi = gi(Yl, Y2"'" Yn, t),

i = 1,2,

.n,

(1)

Yi(tO) = c.,

i = 1,2,

.r,

(2)

Yi(tr) = c.,

i = r

+ 1,

, n.

(3)

Suppose we have the kth nominal solution to Eqs. (1)-(3), y(k)(t) over [to, trJ, in the sense that the initial and terminal conditions are satisfied exactly, but the profiles y(k)(t) satisfy the differential equation (1) only approximately. We expand the right-hand side of (1) in a Taylor series up through first-order terms around the nominal solution y(k)(t); namely, approximate

ess"" 1)) ~ where

g;(y(k»)

+ Jly(k»)(y(k+ 1)

_

y(k»),

i

= 1, ... , n,

(4)

80

V. Numerical Solution of Two-Point Boundary-Value Problems

is the ith row of the gradient evaluated at y(k). Since we want ylk+ 1)

= ess": 1»,

i = 1,... .n,

(5)

by substituting (4) in (5) we arrive at the ordinary differential equation for y(k+ 1) t) +

ylk+ ll(t) = gi(ylk)(t), . . • ,y~k)(t),

og In -0 (y)k+1)(t) i

j=1 Yj

y)k)(t)).

(6)

On rearranging terms in (6), we have y(k+

ll(t)

= J(y(k)(t))y(k+ 1)(t) + f(t),

k = 0,1,2, ... ,

(7)

where J(y(k)(t)) is an n x n matrix with elements ogi/oYj evaluated at y(k)(t), and f(t) is an n x 1 vector with elements gi( Yl(k)() t, ... , Yn(k)(» t, t -

a

~ Ogi Yj(k,() L. t, j=1 Yj

i = 1,2, ... ,no

Since we are clamping the boundary conditions at each iteration, we set y!k)(t O) = Yi(tO) = c., y!k)(t r) = Yi(tr) = Ci'

i = 1,2, i

= r + 1,

,r, ,n.

(8)

Equations (7)and (8)define a linear two-point boundary-value prob­ lem that can be solved by the method of adjoints to give the (k + l)th approximation to Eqs. (1)-(3). Theoretically, for a solution to the nonlinear problem, we require lim y\k)(t) = Yt(t),

k ..... a:

i = 1,2, ... ,n,

to:S; t :s; tr.

Numerically it is sufficient for

Iy!k+ 1)(t) - y!k)(t) I < s,

i = 1,2, ... , n, to:S; t :s; t r .

Recapitulating, quasi-linearization consists ofthe following steps: 1. Linearize the right-hand side of (1) to obtain (7). 2. For k = 0, provide nominal profiles ylO), ... ,y~O)(t), to :s; t :s; tr, that satisfy the boundary conditions. 3. For the kth iteration, using as nominal profiles y(k)(t), solve the linear two-point boundary-value problem (7) and (8). 4. Test whether

Iy!k+ 1)(t) - y\k)(t)1 < e,

i = 1,2, ... ,n,

If satisfied, exit; otherwise set k = k

+ 1 and

to:S; t:s; t r .

go to 3.

5. Finite-Difference Schemes and Multiple Shooting

81

It can be shown that when quasi-linearization converges it does so quadratically, but again it is a Newton-type method (now in q[t o , t r ] ) and so the initial guess is very important.

5. FINITE-DIFFERENCE SCHEMES AND MULTIPLE SHOOTING Finite-difference schemes have proved successful for numerically unstable problems since the finite-difference equations incorporate both the known initial and final data, as does quasi-linearization. There­ fore the solution is constrained to satisfy these boundary conditions. The disadvantages are that the preparation ofthe problem is arduous and the solution involves solving large numbers of (in general) non­ linear simultaneous equations by say Newton-Raphson or other itera­ tive methods. To see why this arises, consider a simple linear problem

d2y

dt 2 = Y

+ t,

y(a) = c l ,

a ~ t y(b) =

~

b,

(1)

C2'

(2)

The interval [a, b] is divided into N + 1 intervals of length h = + 1), discrete points are given by

(b - a)/(N

t j = to

b-a

+j N + l '

j = 0, ... , N

+ 1,

where to = a and t N + 1 = b. Set Yj = y(t). Then if we replace y" by second central difference

y"

=

Yj+l -

~i + Yj-l + O(h 2),

and the discrete approximation to (1) is -Yj-l +(2 + h2)Yj - Yj+l = -h 2t j ,

j

= 1,2, ... , N.

(3)

Then (1) and (2) can be approximated by the N equations in N un­ knowns {Yl" .. , YN}, that is, 2 (2 + h )Yl - Y2 = -h 2t 1 + c 1 , -Yl + (2 + h2)Y2 - Y3 = -h 2t 2 ,

82

V. Numerical Solution of Two-Point Boundary-Value Problems

Or in matrix form, Yz

-h Zt1 -hZt z

YN-I

-hZtN _ 1

Yt

-1 -1 -1

2

+ hZ

+ c1 (4)

-hZtN + Cz

YN

since Yo = y(a) = C I, YN+I = y(b) = Cz. Since this problem is linear, we have here a linear set of equations. In general, for nonlinear ordinary differential equations we shall get a set of nonlinear equations that must be solved by iteration. For example, the central difference approximation to _y"

+ yZ =

t,

y(a)

= C I,

y(b)

= Cz ,

is - Yj-t

+ 2Yj + hZyJ -

Yj+ I = -

hZtj ,

j = 1,2, ... ,N.

Finally, for unstable problems a combination of shooting and finite­ difference ideas, called multiple shooting, has given promising results. Here the interval [to, tf ] is divided into equal parts, over each of which a shooting method is applied (see Fig. 1). However, in contrast to finite-difference methods, the number of intervals is usually relatively small. Note that in solving our general problem by shooting over [to, t r], if we use integration scheme of order p and step size h, the errors in the numerical solution of the differential equations is of the order K I = const,

and so reducing the size of the interval over which we shoot can substantially reduce the errors [1]. For instance, consider the simple

Fig. 1 Multiple Shooting.

5.

Finite-Difference Schemes and Multiple Shooting

83

two-point boundary-value problem (TPBVP) Yi(t)] = [fi(t,y(t»] [ Y2(t) f2(t, y(t»

(5)

over 0 ::; t::; 1, where Yi(O) = a, Y2(1) = b. If we break up the interval [0, 1] into two equal subintervals [O,!] and [!, 1] (see Fig. 2), and apply shooting over each, guessing Y2(0), Yi(!), Y2(!), then we can formulate this problem as a four-dimensional TPBVP of a type considered earlier. First, we make a time change to reduce the equation (6) y = f(t, y), to an equation over [0,

n That is, if we set. = t -

o::; r ::; !, then z is the solution of z(.) = f(.

with

+!, z(.»

!, z(.) =

y(.

+ i),

: = g(., z(.», Z2(!) = b.

Furthermore, continuity across t

=

(7)

! requires that Y2(!) = Z2(0).

(8)

Therefore our original problem can be equivalently stated

(9)

o

1/2 Fig. 2

84

V.

Numerical Solution of Two-Point Boundary-Value Problems

with boundary conditions (5), (7), (8), or

~r OI ~ ! ~lr :i~ J o

0 0

zz(O)

+

r- L~ 0

(10)

0

which is of the type considered in Section 3. Keller [1] has shown that if we partition the interval [to, tc] into N equal subintervals, then we must now solve a nN system of ordinary differential equations by shooting; however, the error can be bounded by K 1 = const.

6. SUMMARY Method

Advantages

Disadvantages

Shooting

1. Easy to apply. 2. Requires only (N-r) missing initial data points. 3. Converges quadratically when it converges.

1. Stability problems, particularly over long intervals. 2. Must solve nonlinear differential equations.

Quasi-linearization

1. Only need to solve linear differential equations 2. Converges quadratically 3. For numerically sensitive problems quasi-linear equations may be stable.

1. Must select initial profiles. 2. Need to store y(k)(t) and

1. Good for numerically unstable problems.

1. Arduous to code. 2. May need to solve large numbers of nonlinear equations.

Finite-Difference Schemes

ylk+ l)(t).

Problems 1. Apply the method of adjoints to the equation ji

= y + t,

y(O) = 0,

y(1) =

tX.

References

85

2. Apply the method of Section 3 to the nonlinear TPBVP 3yji

+ y2

=

0,

with the boundary conditions

y(O) =

0(,

3. Write a program to solve the TPBVP Y(t) By(O)

= A(t)y(t) + f(t), 0 s t ~ 1

y(t) = (Yl(t), ... ,Yit)),

+ Cy(l) = d,

for given A(t), f(t), B, C, and d. As an example, solve

with Y3(0)

=

-0.75,

Yl(l)

= 13.7826,

Y3(1) = 5.64783,

over 0 ~ t ~ 1, and give the resultant y values at 0.05 intervals. [2]

References [1] H. B. Keller, "Numerical Methods for Two-Point Boundary Value Problems. Ginn (Blaisdell), Boston, Massachusetts, 1968. [2] S. Roberts and J. Shipman, "Two-Point Boundary Value Problems" Elsevier, Amsterdam, 1972.

This page intentionally left blank

Chapter VI

Dynamic Programming and Differential Games

The fundamental principle of dynamic programming, the so-called principle of optimality, can be stated quite simply.

If an optimal trajectory is broken into two pieces, then the last piece is itself optimal.

A proof of this principle, sufficient for many applications, can be given quite simply (see Fig. 1). Suppose path (2) is not optimal. Then we can find a "better" trajectory (2') beginning at (x', t'), which gives a smaller cost than (2). Now, tracing (1) to (x', t') and (2') from then on must give a smaller overall cost, which contradicts the supposed optimality of (1)-(2). (Of course this proof requires that pieces of admissible trajectory can be concatenated to form an admissible trajectory. This is not necessarily always true, and indeed the principle of optimality may fail to hold.) 1. DISCRETE DYNAMIC PROGRAMMING

Let us consider the network with associated costs shown in Fig. 2. We can regard this as an approximation to a fixed-time problem, where we have partitioned the time interval into three parts, and we have restricted the number of states that are attainable to three after the first time step, three after the second time step, and four after 87

88

VI. Dynamic Programming and DifferentiaJ Games

T

t'

Fig. I

---/,[2]

ill 2

3

Fig. 2

third. The number on the path refers to the payoff when that path is chosen. Furthermore, not all future states need be attainable from previous states. Finally, the numbers in boxes refer to the extra payoff we get for finishing up at that point. This corresponds to a term g(x(T)) in the cost functional. Of course they may be zero. Let us work out the path that maximizes the total payoff (Fig. 3). Remembering the idea of dynamic programming, we start at the rear (i.e., state 3) and move backward. Now calculate all possible payoffs in going from state 2 to state 3. Remembering that the final section of an optimal trajectory is itself optimal, we can immediately disregard the lower path from P 1 since this gives only 5 units of payoff, whereas the upper path gives 17 units. Another way of looking at this is to

1. Discrete Dynamic Programming

@] P3

89

r---_ - - - - - ' 0

2

3

Fig. 3

suppose our optimal trajectory (from 0) lands us at P 1; then clearly over our final stretch we will pick up 17 units and not 5. To indicate that this path is the best from this point, we mark it with an arrow and put 17 next to Pl' Continuing similarly with P 2 and P 3' we find 15 and 13 as the best payoffs from these points. Then we apply the same argument in going from step 1 to step 2, and we obtain the result shown in Fig. 4. Note that both paths from the middle point at step 1 give the same cost; hence we arrow both paths. Now, with one final application (Fig. 5), we have finished. We see that the maximum payoff is 24, and we can follow our arrows backward to get the optimal path: it is the upper path in this case. ~

100::,--- - - - - - - l

§I""'----

---~~ 2

Fig. 4

~

90

VI. Dynamic Programming and Differential Games

1

®J

Fig.S

Unfortunately, if we discretized our time interval into many pieces (n), and allowed many control values at each step (m), then we can

see that the dimension of the set of all possible final paths will be enormous (mn ) . This is the main disadvantage of dynamic programming: it is often called the curse of dimensionality. 2. CONTINUOUS DYNAMIC PROGRAMMING­ CONTROL PROBLEMS We now use dynamic programming to derive necessary conditions for optimality for a control problem of the type minimize f.~ fo(x(t), u(t)) dt

(1)

subject to i(t) = f(x(t),u(t»,

x(to) = xo,

to ~ t

~ T.

(2)

Let V(Xl> tt) be the optimal cost for this control problem with initial condition x(tt) = x., to ~ t 1 ~ T. In the following we shall suppose that V, f, and fo are sufficiently differentiable. This is a strong assumption. Suppose that, u* is an optimal control over [to, TJ for (1) and (2), and that {) > 0 is sufficiently small. Then V(x o, to) =

fT

Jto

fo(x(t:u*),u*(t»dt

= ftoH fo(x(t: u*),u*(t» dt Jto

+

fT Jto+,s

fo(x(t: u*),u*(t» dt.

(3)

However, by the principle of optimality, the final term in (3) is just V(x(to + {) : u*), to + {», that is, V(x o, to) =

i

to + ,s

to

fo(x(t: u*),u*(t» dt

+ V(x(t o + {) : u*), to + {».

(4)

2.

91

Continuous Dynamic Programming-Control Problems

Now, using the regularity of f and V, we can expand in a Taylor series and see that

+ 15 : u*) = V(x(t o + 15 : u*), to + b) = x(to

Xo

+ bf(x o, u*(to)) + 0(15),

(5)

b{V x V(x o, to) • f(x o, u*(t o))

+ V;(xo , to}} + V(xo, to) + 0(15),

(6)

i:o+O JoJx(t: u*), u*(t)) dt = bJo(x o, u*(to)) + 0(15),

(7)

where lim(o(b)/b)

=

~~o

o.

IfEqs. (5)-(7) are substituted in (4)and 15 --+ 0, we arrive at the following partial differential equation for V: V;(x o, to) = -{VxV(xo, to)· f(xo,u*(t o))

+ Jo(xo,u*(t o))}.

(8)

If we return to Eq. (3), we can derive a little more information about the optimal control. Thus V(x o, to) = min{ ued

~to+o

Jto

Jo(x(t: u), u(t)) dt

+ V(x(t o + 15 : u), to + b)}.

Now expanding as before in a Taylor series gives V;(xo, to) = - min {VxV(xo,t o)· f(xo,u(t o)) u(tolen.

+ Jo(xo,u(to))}.

(9)

Since u(to) can take any value in n, and since X o and to are arbitrary, (9) implies that V;(x, t)

= - min {V x V(x, t) • f(x, w) + Jo(x, w)}, wen

(10)

for all x, t > 0, and the optimal control u* is just the value of WEn that achieves the minimum in (10); u* = u*(x, t). Equation (10) is called the Hamilton-Jacobi-Bellman (HJB) equation, and it imme­ diately leads to an optimal control in feedback form. The partial differential equation (10) has the natural terminal condition V(x, T) = 0

(11)

When the control problem has a cost of the form minimize {9(X(T))

+ i~

Jo(X,U)dt},

92

VI. Dynamic Programming and Differential Games

Eq. (11) is replaced by V(x, T) = g(x),

x ERn.

In general it can be shown that the following theorem holds [3].

Theorem 1 Suppose that f and io are continuously differentiable and the solution V of (10) is twice continuously differentiable. If u* achieves the minimum in (10) and u* is piecewise continuous, then u* is an optimal control. Conversely, if there exists an optimal control forall(xo,t 0 and S, Q ~ 0, an optimal control exists, is unique, and is given by u*(t) = R -1 BT(g(t) - K(t)x*(t»,

(30)

where K is the solution of the Riccati differential equation K(T)

= S.

(31)

The vector g(t) is the solution of the linear equation

g = -(A - BR-1BTK)T g - Q,(t),

(32)

The optimal trajectory satisfies X o.

(33)

+ ,(T) . K(T)'(T).

(34)

x(O) = The minimum value of.the cost is C(u*)

= tx*(T) • Sx*(T) -

,(T) • STx*(T)

Example 2 An Infinite Horizon Control Problem (Lee and Marcus [3]) When the basic interval [0, T] becomes infinite, that is, T = + 00, the theory given above leads to the linear regulator problem: find the control that minimizes the total error over all time. To simplify the analysis, we consider the linear autonomous system

x = Ax + Bu,

x(O) =

X o,

(35)

for A and B constant matrices, and cost functional C(u)

= 2"1 Jor'"

{x(t)· Qx(t)

+ u(t) • Ru(t)} dt,

where Q, R > 0 are constant symmetric matrices.

(36)

96

VI.

Dynamic Programming and Differential Games

The first problem that immediately arises is finding when (36) will be finite. Clearly we want the solutions of(35) (or at least the optimum trajectory) to decay to zero as t --+ 00. The set of admissible controls will be the space of all m-vector functions that are square integrable over (0, (0). It turns out that the required assumption on A and B is rank{B,AB,A 2B, ... ,A"-lB}

=

n.

(37)

(This condition is called controllability, and we shall consider it in some detail in Chapter VII.) Theorem 3 Consider the autonomous linear control problem (35) for which (37) holds, and the cost functional (36). There exists a unique symmetric positive definite matrix E such that (38)

For each initial point X o in R", there exists a unique optimal control u* that can be synthesized in feedback form as u*(t) = -R-1BTEx*(t).

(39)

The optimal response x* satisfies

x* = (A

- BR-1BTE)x*,

x*(O)

= Xo,

(40)

and the minimal cost is C(u*) =

txo . Exo·

(41)

Note that once the existence of E is assumed (basically this involves showing that limt_C(> K(t)· E exists, using the controllability as­ sumption), then by a well-known lemma of Liapunov, the matrix A ­ BR-1BTE is a stability matrix (all its eigenvalues have negative real parts), and so solutions of (40) decay to zero as t --+ 00, as required [3].

3. CONTINUOUS DYNAMIC PROGRAMMING­ DIFFERENTIAL GAMES In many situations we should like to consider two individually acting players in our control model. Associated with the process would again be a cost functional; however, the players will now be antagonistic, in the sense that one player would try to maximize this cost and the other to minimize it. If the dynamics of the process are

3. Continuous Dynamic Programming-Differential Games

97

again given by a differential equation, we could model it as

x=

f(x,,p, !/I),

x(O) = Xo,

(1)

where e is the first player's control variable and !/I is the second player's control variable. We assume each of these controls takes its values in some restraint set; typically -1~cPi~l,

i=I, ... ,m

-1~!/Ii~l,

(2)

or ,p and !/I could be unrestrained [2]' To determine when the game ends, we suppose some terminal set F in the state space R" is given, and we terminate the game whenever F is reached. To simplify the analysis, we always assume that F can be reached from any point in the state space by admissible controls. Given any two controls ,p and "" we associate a cost of the form C(,p,!/I) = f~f

fo(x(t), ,p(t),!/I(t)) dt

+ g(x(tr)),

(3)

where the integration extends over the trajectory and ends (t = t r) when we hit F. The function g is a terminal cost, which needs to be defined only on F. (We shall show later that this formulation actually includes the fixed-time problem considered earlier.) The novel aspect is that now t/J will try to minimize (44) and !/I will try to maximize it. The value of the game starting from X o would then be naturally defined as

.

V(xo) = min max C(,p,!/I), ~

~

where the minimum and maximum are over all admissible controls ,p and !/I. Several mathematical problems involved in such a definition as (4) are immediately obvious. Is the ming max., well defined, and does it equal max~ min.? A full investigation of these problems involves deep concepts (see [1]), and we shall from here on assume the value exists, and min max = max min whenever required. In the simple problems we shall solve here, this will not be a difficulty. We can derive necessary conditions for optimality by using dynamic programming in essentially the same way as before. Assume we have a trajectory starting at Xo at time to, of the following type (see Fig. 6). We break the trajectory at time to + fJ for small fJ > 0, and suppose that we use arbitrary admissible controls ,p,!/I over [to, to + fJ), and optimal controls over the final are, to + fJ --+ F. Then the cost calculated

98

VI.

Dynamic Programming and Differential Games F

Fig. 6

along this trajectory is just rIO+~

Jro

fo(x(r), VV(xo) . f(xo,tP(to), "'(to)) + o(fJ). (5)

In other words, the cost of this trajectory from Xo is V(xo) + f>( VV ·f(x o, tP(to), "'(to))

+~

l:oH fo(x,tP, "')

dt

+ O~f»}

(6)

Since the value of the ga.me starting at Xo is the min, maJC,j, of the cost (6) (by the principle of optimality), we have V(xo) = min max[Eq (6)J, so canceling V(xo) and letting f>



--+ 0

'"

gives

min max[VV . f(xo, tP(to), "'(to))

+ fo(x o, tP(t o), "'(to))] = O.

(7)

'"

Since (7) holds {or any Xo in the state space and for tP(to) and "'(to) in their respective restraint sets, we have min max [V V(x) . f(x, O. Suppose player II, if left unhindered, can manufacture weapons at a rate m2 • He also loses them at a rate proportional to the number x 1 that his enemy is devoting to that purpose. We shall assume therefore

°

(15)

where C2 may be regarded as a measure of effectiveness of player I's weapons against player II's defenses. By reversing the role of the players, we obtain the second state equation (16)

Suppose we plan on the war lasting some definite time T. Each day (say) player I puts (1 - 0, and so xf is increasing over ( T - l/cl, T ) . However, it is certainly possible that xT(t) = 0 for t I T - l/c, . This corresponds to player I being annihilated. There will be a critical trajectory that just hits the point A in Fig. 8. All trajectories which start off below this will have x:(to) = 0 for some t o < T - l/cl. It is clear that if x f ( t o ) = 0, all player I1 must do is keep 2: = 0; XT can be kept at 0 only if

(28)

xz* 2 m1lc1,

and in this case I1 should play **(t) = m l / c l x 2 ( t ) ,

until point A, when both egies (+ 1).

+* and $I*

to I t IT -

l/cl,

revert to their old optimum strat-

104

VI. Dynamic Programming and Differential Games

X,

T-

c,!.­

T ~.

=

0

v" = 1

~.

=

0

"'. = 0

Fig. 8

Finally, for t ::; to, we can resolve the HJB equation with Xl(t O) = 0, and we find that II's optimal strategy is unchanged; however, player I should switch slightly earlier, along the dotted line in Fig. 8. Here we see that since I's final attack is going to be nullified through the annihi­ lation of his forces, he tries to compensate by starting earlier. Of course, the value of the game will have to be recomputed for these new opti­ mum trajectories. The case in which II is annihilated can be handled symmetrically.

Problems

1. Find the feedback control and minimal cost for the infinite horizon control problem

x=

-x

+ u,

x(O) = Xo,

with cost

C(u) = fo'" [(X(t))2 + (u(tW] dt. 2. Find the control u = (Ul' U2) that minimizes the cost

~2 Jor'" (x 2 + u21 + u2)dt 2 for the system x(O) = 1.

References

105

3. Calculate the feedback control that minimizes -1

2

IT ((Xl-IX) 0

2

+U2)dt .

for the system 4. Calculate the symmetric feedback matrix

for the optimal control of the system

x-X=U, with cost

where X

=

[~l

w=

:J>

[:1

0,

y

> 0.

5. By use of dynamic programming, derive the optimality conditions for the discrete control problem . . .

1 ~ 2 f.., Uk 2 k =o

mmumze subject to

Xo

= 1.

References [1] [2] [3]

A. Friedman, "Differential Games." Wiley, Interscience, New York, 1971. R. Isaacs, "Differential Games." Wiley, New York, 1965. E. Lee and L. Markus, "Foundations of Optimal Control Theory." Wiley, New York, 1967.

This page intentionally left blank

Chapter VII

Controllability and Observability

One of our central assumptions in the time-optimal control problem was that the target point could be reached in finite time by an admissible control. In this chapter we investigate this problem of controllability of systems.

1. CONTROLLABLE LINEAR SYSTEMS To begin with, suppose that there are no restrictions on the magni­ tude of the control u(t). Let the restraint set be n = R". Definition 1 The linear control process x(t)

= A(t)x(t) + B(t)u(t),

(1)

with n = B", is called (completely) controllable at to if to each pair of points Xo and X'l in R" there exists a piecewise continuous control u(t) on some finite interval to ~ t ~ t 10 which steers X o to x.. In the case (1) is autonomous, we shall see that controllability does not depend upon the initial time, and so we just say the system is controllable. Theorem 1 The autonomous linear process x=Ax+Bu 107

(2)

108

VII.

Controllability and Observability

is controllable if and only if the n x nm matrix [B,AB,A 2B, ... ,An-1BJ

has rank n. Proof

Assume that (2) is controllable, but that rank[B,AB, ... ,An-1BJ < n.

Consequently, we can find a vector v e R" such that v[B,AB,A 2B, ... ,An-1BJ = 0

or vB

= vAB = ... = vAn-lB = O.

(3)

By the Cayley-Hamilton theorem [1J, A satisfies its own characteristic equation, so for some real numbers Ct> C2, • • • , Cm An

=

c1A n- 1 + C2An-2

+ ... + cnI.

Thus clvAn-lB

+ c2vAn-2B + ... + cnvB = 0

by (2). By induction, vAn+kB for all m = 0, 1,2, ... , and so,

= 0 for all k = 0,1,2, ... , or

vAnB

=

veAIB

= V[I + At + ~ A 2t2 + .. .JB = .

2!

vAmB

0

=0 (4)

for all real t. However, the response starting from X o = 0 with control u is just x(t)

= eAI f~ e-ASBu(s)ds,

so (4) implies that v· x(t)

= f~

veA(I-S)Bu(s)ds

= 0,

for all u and t > O. In other words, all points reachable from Xo = 0 are orthogonal to v, which contradicts the definition of controllability. Next suppose that rank[B,AB,A 2B,. " ,An- 1BJ = n. Note first that (1) is controllable if and only if

U {Jo II eA(I-s)Bu(s)ds : u is piecewise continuous} = s:

1>0

(5)

1.

Controllable Linear Systems

109

Suppose (4) is false; then certainly l

{fo eA(I-S)Bu(s) ds: u is piecewise continuous}

t= R"

or there must exist some nonzero v E R" such that l

v fo eA(I-s)Bu(s)ds = 0

for all piecewise continuous D, and so veA(I-s)B = 0,

O~s~l.

(6)

Setting s = 1, we see that vB = O. Differentiating (6) and setting s = 1 give vAB = O. Accordingly, by induction, vB = vAb = vA 2B = ... = VA"-l B = 0, which contradicts the rank condition. • Corollary 1

If (1) is normal, then it is controllable.

The converse is true only in the case m = 1; in fact, for m = 2, we have a simple counter example. Example 1

Hence b 2 and Ab2 are linearly dependent, so system is not normal. However . [B,ABJ

=

G

o

-1

I

-2

is certainly rank 2. Note also that the proof of Theorem 1 shows that if the system is controllable, then it is controllable in any arbitrarily small time. The concept of controllability for autonomous systems is indepen­ dent of the coordinate system chosen for R". For if y = Qx, where Q

11 0

VII.

Controllability and Observability

is a real nonsingular constant matrix, and iJI = QB,

then

x = Ax + Bu

(7)

y = dy + iJlu

(8)

is controllable if and only if is controllable. This follows easily from the identity rank[B,AB, ... ,An-lB]

=

rankQ[B,AB,

,An-lB]

= rank[iJI,d~,

,dn-liJI].

We shall say (7) and (8) are equivalent control systems. Example 2

The autonomous linear process

r'" + aly(n-l) + a2ytn-2) + ... + anY = u

(L)

is controllable with u E R. In phase space, (L) is just

Y=

0 0

1 0

0 1

0 0

0

0

0

1

y+

-al

-an -an-l -an-2

0 0 0

u,

1

and the rank condition is easily checked. For autonomous systems with one control variable, this example is typical.

Theorem 2 Every autonomous controllable system

x=

Ax

+ bu,

(9)

with u E R, is equivalent to a system of the form (L). Proof

Define the real nonsingular (by Theorem 1) n x n matrix Q

= [A n -

1b,A n

-

2b,

... , Ab,b].

Direct calculation shows that b= Qd

and

AQ = Qd,

1. Controllable Linear Systems

III

where IXl

1X2

1 0 0 1

0 0

0 0 0 0

1 0

0 0

a=

d= IXn-l IXn

and the real constants IXl' 1X2,'" istic equation for A by

,lX n

0 1

are determined by the character­ (10)

The analogous change of coordinates applied to (L) would give system matrices .91' and d, where -al -a2

1 0 0 1

0 0

-an-l -an

0 0 0 0

1 0

.91'=

Consequently, (9) will be equivalent to (L) if we choose In other words, (9) is equivalent to the system y(n) _

where lXI, •.. by (to). •

, tin

1X1ln - 1) _ 1X2 y(n- 2) -

••• -

IXnY = u,

are obtained from the characteristic equation for A

The next concept, which occurs often, is the domain of null control­ lability at time to. This is the set of all points in R" that can be steered to the origin in time t l > to by an admissible controller u(t) taking values in a given restraint set n c R". In the case n = R" and the system is autonomous, it is easy to show that null controllability is equivalent to controllability (Problem 2). In Section 11.3 we showed directly, by constructing time optimal controllers, that certain second­ order systems were null controllable with controls restricted by lu(t)1 :-s; 1. The following theorem essentially generalizes this construc­ tion.

112

VII.

Controllability and ObservabiJity

Theorem 3 (Hermes and LaSalle [1]) Consider the autonomous linear process in R"

x=

Ax

+ Bu,

(11)

for which (a) 0 belongs to the interior of Q; rank[B,AB,An-tB] = n; (c) every eigenvalue A of A satisfies Re(A)

(b)

~

O.

Then the domain of null controllability at any time T equals B", Proof Without loss of generality, assume that Q contains the unit cube in W. Then the theorem will be proven if we can show that [J£ = Ut~o[J£(t)

= {y(t: u): Iud ~

1, t> O} = R",

where [J£(t) is the reachable set introduced in Appendix 1, and y(t: u)

= f~ e-A<Bu(r) dt.

Since fJt is convex, it is enough to show that for each" #- 0 in R", there exists an admissible control u* such that a- y(t: u*) -+ 00 as t -+ 00. (Geometrically this is clear; a detailed proof can be obtained by sup­ posing fJt #- R", then using the supporting hyperplane theorem to derive a contradiction.) We show that the admissible control u*(t) = sgn["Te-AtB],

t > 0,

suffices. Note first that the rank condition implies that at least one component of"Te-AtB, say "Te-Atbt> is not identically zero. Then "Ty(t: u*) = f~ \"Te-A<Bu*(r)1 dt,

and hence

"T y(t:

u*) -+

00,

if we can show that

fo'" I"Te-A X2) : X2 ~

p(x,u)

=

-u.

(5)

O}. For this problem, (6)

Now let x*(t) = (xf(t), x!(t)) be an optimal trajectory with control u*(t), and suppose x*(t) lies entirely on the boundary of B, for t k S t S t k + 1. Trivially, from (6) the optimal control must be u*(t)

= 0,

(7)

Substituting (7) into (2) and (3) gives xf(t) t

See Nostrom [3].

= xf(tk),

x!(t)

= 0,

3. The Continuous Wheat Trading Model without ShortseUing

133

Since the control constraints are now no longer binding, their respective multipliers in (1.12) are zero, and so we have A(t) = 0,

y(x, t) = 0,

XEr,

t> 0,

0,

XEn,

-=

y(x,O)

=

ily

i= 1

°

(3)

with gi = ox" the Dirac function centered at Xi' i = 1,2, ... ,m. Alter­ natively, with m = 1, gl(X) = I, (3) would model the problem of con­ trolling the temperature on the rod n by the ambient temperature Ul(t). IfA. j and ({Jj,j = 1,2, ... , are the eigenvalues and eigenfunctions of the system y" = A.y in. n with homogeneous Dirichlet boundary condi­

•

U (t) 1

u (t )

~

~

• x

1

2

u (t) m

~

x•2

X

n

• m

•

6. Approximate Controllability for Parabolic Problems

163

tions, then for gi E £2(Q), the solution of (3) can be written y(x, t: u) =

j~l

Ctl

(gio CPj)

f~

exp[ - Aj(t - t)]uJr) d't) CPj(x),

(4)

where (gi,CPj) = Jll9i(x)cpix)dx are the Fourier coefficients of the {gil with respect to the {cpj}. The solution (4) is also valid for gi = bX i if we interpret te.. CPj) = cpixJ As before, (3) is approximately controllable in time T if and only if

In I]y(T:

u) dx

=0

for all

u

= (Ui),

u, E L 2(0, T),

i=I,2, ... ,m

implies that I]

= 0 (I] E L ~Q».

j~l(g,cp)(I:

exp[ -ilN

-

If m

(5)

= 1, (5) is equivalent to

't)]U('t)d't}I],CP) = 0

for all u E L 2(0, T),

and hence 00

I

j=l

(g,cp)(I],cpj)exp(-Aj't) == 0

for

0< r

~

T.

(6)

However, the functions {exp( -Aj't)} are linearly independent, and so (6) must imply that (g, CPj)(I], cp) = 0

for all j = 1,2,. .. .

(7)

The necessary and sufficient condition for approximate controllability can now be read from (7). If(g, CPjo) = 0 for somejx, then we can certainly find a nonzero I] satisfying (7); for instance, take I] = CPj for any j i= jo· Conversely, if(g,cpj) i= 0 for allj = 1,2, ... , then from (7) (I],cP) = 0 for all j, and hence I] = 0, since the {cp j} span L 2(Q). Consequently, with m = 1, (3) is approximately controllable if and only if (g, cP j) i= 0 for every j = 1,2, .... In the general ,case, with m controls one can similarly show (5) is equivalent to (gioCP)(I],CP) = 0 for all i = 1,2, ... .m, j = 1,2, ... ,m. Consequently, (4.4) is approximately controllable in any time T> 0 if and only if for every j = 1,2, ... , (gi, cP j) i= 0 for some i = 1,2, ... , m. (8) Note that in the case gi = forevery j=I,2, ... ,

t)Xi'

(8) becomes

CPj(x;)i=O

forsome

i=I,2, ... ,m,

164

IX.

Controlof Partial Differential Equations

or equivalently, for every j, at least one 1,2, ... .m.

Xi

is not a nodal point of qJj' i =

Finally, in the case in which the state equations (3) are over a region the distinct eigenvalues of

o eRN, if Aj are

Liy = Ay

y=o

in 0, on F,

and {qJk,j}k'4, 1 are corresponding eigenfunctions (mj is the multiplicity of Aj), then the solution of (3) becomes

y(x,t: u) =

itl (f~

j~l k~l

exp[ -Ai t - 't")]u;(-r)d't") (gi,qJk)qJkix)

and the same argument now shows (3) is approximately controllable in any time T if and only if

rm k

(g1> qJlj) (gl,qJ2j)

r ~mj) .

(g 1,

(g2' qJlj) (g2,qJ2) (g 2, qJmj)

(gm' qJlj)l (gm,qJ2j) (gm,

=~

~mj)

for every j = 1,2,. .. . In particular, one needs at least as many controls as the largest multiplicity of the eigenvalues. Consequently, any system (3) with infinite multiplicity of its eigenvalues cannot be approximately controllable in L 2(0) with "point controls." For further discussion of this subject, we refer the reader to Triggiani [6]. References [I] A. Butkovskiy, "Distributed Control Systems." Elsevier, Amsterdam, 1969. [2] A. Friedman, "Partial Differential Equations." Holt, New York, 1969. [3] G. Knowles, Time optimal control in infinite dimensional spaces. SIAM J. Control Optim. 14,919-933 (1976). [4] G. Knowles, Some problems in the control of distributed systems, and their numerical solution, SIAM J. Control Optim. 17, 5-22 (1979). [5] J. Lions, "Optimal Control of Systems Governed by Partial Differential Equa­ tions." Springer-Verlag, Berlin and New York, 1971. [6] R. Triggiani, Extensions of rank conditions for controllability and observability to Banach spaces and unbounded operators, SIAM J. Control Optim. 14, 313-338 (1976).

Appendix I

Geometry of R"

Suppose that C is a set contained in W, and that for x E C, Ilxll = (xi + x~ + ... + X;)l/2. C is called bounded if there exists a number (X < 00 such that IIxll ::;; (X for all x E C. C is called closed if it contains its boundary; i.e., if x, E C and Ilxn - xII ~ 0, then we must have xE C. C is convex ifit contains every line segment whose end points lie in C; i.e., ifx, y E C, then tx h E C. (See Fig. 1.) An element x E C is called an extreme point of C if it does not lie on any (nontrivial) line segment contained in C (alternatively, if it cannot be written as a proper linear combination of elements of C). For example, the vertices of a square are extreme points (see Fig. 2). Given a vector" ERn, the hyperplane through a point Xl ERn with normal" is H = {x E R": "TX = "TXtJ (see Fig. 3). A point Xl E C is supported (Fig. 4).by this hyperplane H if C lies on one side of H, i.e., "TX l ~ "TX for all x E C. The point Xl is called an exposed point of C if Xl is supported by a hyperplane H (with normal", say) and if Xl is the only point of C supported by H. In other words "TX l > "TX for all x E C, and x =f. Xl' In Fig. 5, Xl is an exposed point. In Fig. 6, Xl is a support point but not exposed point, and in Fig. 7, Xl is an extreme point and a support point, but not exposed point.

+

165

166

Appendix I

not convex

Fig.!

d ....

... c

c a...- - - - - - -... b Fig. 2

Fig. 3.

Hyperplane through 0 with normal If.

1)

Fig. 4.

XI

is a support point of C.

c Fig.S

1)

-----~-r--------_r_-----H

c Fig. 6

C Fig. 7

,)

168

Appendix I

Theorem 1 If C is a closed, bounded convex set, and if Xl belongs to the boundary of C, then we can find a hyperplane H (with normal q ¥- 0) that supports C at Xl. A set C is called strictly convex if every boundary point of C is an exposed point (e.g., circles, ellipsoids). A rectangle is not strictly convex. In general every exposed point is a support point (trivial), and every exposed point is an extreme point (why?), but not conversely.

Appendix II

1.

Existence of Time Optimal Controls and the Bang-Bang Principle

LINEAR CONTROL SYSTEMS

We shall now suppose that we are given a control system of the form x(t)

= A(t)x(t) + b(t)u(t),

x(O) = Xo,

(1)

where x(t) E R", u(t) ER, and A and bare n x nand n x 1 matrices of functions that are integrable on any finite interval. The controls u will be restricted to the set U

= {u: u is Lebesgue measurable and lu(r)1

~

1 a.e.},

(2)

where U is commonly called the set of admissible controls; note that U c LOO(R). A "target" point Z E R" is given, and we shall consider the control problem of hitting z in minimum time by trajectories of (1), using only controls u E U. Given a control u, denote the solution of (1) at time t by x(t : u). In fact, by variation of parameters, x(t: u) = X(t)xo

+ X(t)

S; X- (r)b(r)u(r) dt, 1

(3)

where X(t) is the principal matrix solution of the homogeneous system X(t) = A(t)X(t) with X(O) = I. 169

170

Appendix II

The attainable set at time t [namely, the set of all points (in R") reachable by solutions of (1) in time t using all admissible controlsJ is just

(4)

d(t) = {x(t: u): u E U} cz R",

It will also be of use to consider the set 9l(t) =

{S~

X-1(r)b(r)u(r) dt : u E U} cz R",

(5)

Note that by (3) d(t)

X(t)[ x o + 9l(t)]

=

=

{X(t)xo

+ X(t)y: y E 9l(t)}

(6)

and 9l(t) = X-1(t)d(t) - xo,

(7)

so that ZE d(td if and only if(X-1(t1)z - xo) E Bl(tl)' Finally, we define the bang-bang controls to be elements of U bb = {u :lu(r)1 == 1 a.e. r},

and we denote the set of all points attainable by bang-bang controls by

dbb(t)

=

t > O.

{x(t: u): u E Ubb} ,

2. PROPERTIES OF THE ATIAINABLE SET In this section we derive the properties of d(t) that will be central to the study of the control problem. Define a mapping l: L 00([0, TJ) ~ Rn , T> 0, by leu) =

S:

X-1(r)b(r)u(r)dr,

UE

Loo([O, TJ).

(1)

Lemma 1 lisacontinuouslinearmappingbetween U c Loo([O, TJ) with the weak-star topology and R" with the usual topology. Proof The linearity follows directly from the additivity of inte­ gration. For the continuity, recall that on bounded subsets of L 00([0, TJ) the weak-star topology is metrizable, and u, ~ U if and only if

SOT y(r)ur(r) dt ..... S: y(r)u(r) dr

as

r

~

00

for all

y

E

U([O, TJ).

Existence of Time Optimal Controls and the Bang-Bang Principle

171

Suppose that (ur ) E U and U r ~ u. Let X- 1 (r)b(r) = (Yi(r)). Then 1([0, E L T]) by assumption, and so

{y;}

fo

T

Yi(r)u.(r)dr

fo

T

--+

Yi(r)u(r)dr

for all i = 1,2, ... , n. In other words I(ur )

--+

I(u). •

As an immediate consequence of this lemma, we have the following theorem. Theorem 1 d(T) is a compact, convex subset of W for any T> O. Proof The set of admissible controls is just the unit ball in L 00([0, T]), and so is w*-compact and convex (Banach-Alaoglu

theorem). Consequently, 9f(T) = I(u)

is the continuous, linear image of a w*-compact, convex set, so must be compact and convex. However, d(T) = X(T)[xo

+ 9f(T)]

is just an affine translate of 9l(T) in W, and so itself must be compact and convex. • A much deeper result about the structure of the attainable is con­ tained in the following theorem, usually called the bang-bang principle. It says essentially that any point reachable by some admissible control in time T is reachable by a bang-bang control in the same time. Theorem 2 (Bang-Bang Principle)

For any T> 0, d(T) = d bb(T).

Proof Note that d(T) is just an affine translate of 9f(T). It will then be sufficient to show that for every point x E 9l(T), x = I(u*) for some bang-bang control u*. Let B

=r

1

({x }) (') U

=

{u:I(u)

= x}

(') U.

(B is just the set of all admissible controls in U hitting x in time T.) By Lemma 1, B is a weak-star compact, convex subset of L 00([0, T]),

and so by Krein-Milman theorem it has an extreme point, say u*. If we can show ju*1 == 1 a.e., then we shall have found our bang-bang control and be finished.

172

Appendix

n

Suppose not. Then there must exist a set E c [0, T] of positive Lebesgue measure such that lu*(r)/ < 1

for r

E

E.

In fact we can do a little better. Namely, let

Em = {r E E: lu*(r)1 < 1 - 11m},

m = 1,2, ....

Then U::.'= 1 Em = E, and since E has positive measure, at least one Em must also have positive measure. So there must exist an s > 0 and a nonnull set F c [0, T] with lu*(r)1 < 1 - s

for rEF.

(2)

Since F is nonnull, the vector space L OCl(F) (again with respect to Lebesgue measure) must be infinite dimensional, and as a consequence, the integration mapping 1F:LOCl(F) ~ R", 1F(v)

= IF X- 1(r)b(r)v(r) dr,

cannot be 1-1. (IF maps an infinite-dimensional vector space into a finite-dimensional vector space.) Consequently, Ker(IF) "1= 0, so we can choose a bounded measurable function v ¢ 0 on F with 1F(v) = O. We now set v = 0 on [0, T] - F so that 1(v) = 0, (3) and by dividing through by large-enough constant we can certainly suppose Ivl ::;; 1 a.e. on [0, T]. Then by (2), lu* ± svl ::;; 1,so that u* ± €V E U, and by (3)1(u* ± €V) = 1(u*) ± l:1(v) = 1(u*) = x, i.e., u* ± l:V E B. Since clearly, u* = !(u* + l:v) + !(u* - €V)

a

and v ¢ 0, u* cannot be an extreme point of B, a contradiction. •

3. EXISTENCE OF TIME OPTIMAL CONTROLS We return to the time optimal control considered in Section 1 of this Appendix. We shall assume throughout the rest of this section that the target point z is hit in some time by an admissible control, that is: (*) There exists a t 1 > 0 and u E U such that x(t 1 : u) = z.

173

Existence of Time Optimal Controls and the Bang-Bang Principle

Assumptions of the type (*) are called controllability assumptions, and, needless to say, are essential for the time-optimal control problem to be well posed. If we set t* = inf{t : 3u E U with x(t : u) = z} (1) [this set is non empty by (*) and bounded below by 0, so we can talk about its infimum], then t* would be the natural candidate for the minimum time, and it remains to construct an optimal control u* E U such that x(t* : u*) = z. Theorem 1 If (*) holds, then there exists an optimal control u* E U such that x(t* : u*) = z in minimum time t*.

Proof Define t* as in (1). By this definition we can choose a sequence of times t n t t* and of controls {un} C U such that x(t n : un) = z, n = 1,2, .... Then Iz - x(t* : un)1

= Ix(tn: un) -

(2)

x(t* : un)l.

However, from the fundamental solution, x(tn: un) - x(t* : Un) = X(tn)XO - X(t*)xo

- X(t*) f~'

+ X(tn) f~"

X- 1(r)b(r)un(r)dt

X- 1(r)b(r)un(r)dt:

Consequently, Ix(tn: un) - x(t* : un)1 ::::;; IX(tn)xo - X(t*)xol

+ IX(tn)

f" X-

+ IX(t*)

-

1(r)b(r)Un(r)dr!

X(tn)l f~'

(3)

X- 1(r)b(r)U,,(r)dr!

The first term on the right-hand side of (3) clearly tends to zero as n --+ 00. [X(·) is continuous.] The second term can be bounded above by IIX(tn)11

f" IX-

1(r)b(r)llu

)II .I::" IX-

n(r)1 dt ::::;; IIX(t n

1(r)b(r)1

dt

since Iunl ::::;; 1. Consequently, as n --+ 00 this term also tends to zero. Finally, the third term tends to zero, again by the continuity of X(·). Plugging (3) back into (2), we get x(t* : un) --+ z as n --+ 00,

174

Appendix II

i.e., Z E d(t*). However, by Theorem 2.1, $'(t*) is compact, and so $'(t*). Consequently,

ZE

Z =

x(t*: u*)

for some u* E U. • Note that once we have shown that an optimal control exists, the bang-bang principle guarantees that an optimal bang-bang control exists. The above proof also carries over with only cosmetic changes to continuously moving targets z(t), 0 ~ t ~ T, in fact to "continuously moving" compact target sets and to the general state equation

x = A(t)x(t) + B(t)u(t), where B(t) is n x m and u(t) = (u 1(t), . . . ,um(tW.

Appendix III

Stability

Unstable control systems that magnify small errors into large fluctuations as time increases are rarely compatible with good design. Consequently, tests for the stability of systems of differential equations are extremely important. The system of differential equations i

= f(x)

(1)

is said to be stable about x = 0 if, for every B > 0, there exists a (j > 0 such that Ilxoll < (j implies that the solution x(t) starting at x(O) = Xo obeys Ilx(t)11 < B for 0 ~ t < 00. A necessary condition for stability about 0 is clearly that f(O) = 0, or that 0 is an equilibrium point of (1). Usually, for control applications, one requires more than just stability; one would prefer that small fluctuations be damped out as time increases. The system (1) is called asymptotically stable about the origin if, for every B > 0, there exists a (j > 0 such that Ilxoll < (j implies that x(t) ..... 0 as t ..... 00, where x(t) is the solution of (1) starting at x(O) = Xo' Further­ more, (1) is globally asymptotically stable about 0 if every solution of (1) converges to 0 as t ..... 00. Global asymptotic stability is the most useful concept for practical design of control systems. If (1) is linear, that is, if

i=Ax, 175

(2)

176

Appendix III

it is easy to see that a necessary and sufficient condition for global asymptotic stability is that A have eigenvalues with negative real parts only. For if p.j}}=1 are the distinct eigenvalues of A, then any solution of (2) can be written p

X(t) =

L gj(t) exp(AA

(3)

j= 1

where the gN) are polynomials in t. From (3) it is easy to see that x(t) -+ 0 if and only ifReA.j < O,j = 1,2, ... ,P. The eigenvalues of A are determined as the roots of the characteristic equation det(A.r - A) = O. Once this equation has been computed it is not necessary to find all of the roots; the famous Routh-Hurwitz test gives necessary and sufficient conditions for the roots of a polynomial

d(A) = aoAn + alAn- l

+ ... + an-1A + an,

ao > 0,

(4)

to have negative real parts (for a proof, see [1, Vol. II, Chapter XV]. First, define the following determinants

al ao 0 0 L\k = 0 0 0

a3 a2 al ao 0 0

as

a4

a 3

a2 a4

al a 3

0 'a2

0

0

k = 1,2, ... ,n,

0

ak

where we substitute zero for ak if k > n.

Theorem 1 The polynomial (4) with real coefficients has all its roots with negative real parts if and only if the coefficients of (4) are positive and the determinants L\k > 0, k = 1,2, ... ,n. Example 1

For the characteristic equation

aoA 4 + alA 3 + a2A2

+ a3A + a4 =

the conditions for asymptotic stability are

i = 0,1,2,3,4,

0,

Stability

and

a3 0 a2 a4 = a3(ata2 - aOa3) - aia4 > at a3 Note that since a, > 0, ~2 > 0 follows from

~3

>

o.

o.

177

This page intentionally left blank

Index A Adjoint equation, 13,37,92

Admissible controls, 6, 169

Approximate controllability

of elliptic systems, 153

of parabolic systems, 161, 162

Attainable set, 10

B Bang-bang controls, 3, 4, 18, 158, 159, 161

Bang-bang principle, 171

Bilinear form, 146

coercive, 147

Boundary control

of elliptic systems, 152

of parabolic systems, 158

Bounded set, 165

Brachistochrone, 42

D Differential game, 96

Dynamic programming

continuous, 90, 96

discrete, 87

E Equivalent control systems, 110

Euler-Lagrange equation, 41

Exposed point, 19, 165

Extremals, 56

Extreme point, 165

F Feedback control, 17,94

First integral of Euler equation, 42

H C Calculus of variations, 40

Closed loop controls, 17

Closed set, 165

Controllability, 10, 96, 107

Control problem

fixed-end point, 6

fixed time, 4

free-end point, 6, 61

general form, 5, 6

infinite horizon, 95

linear regulator, 95

minimum time, 1,2,9, 159, 172

normal, 17, 19

quadratic cost, 93

state constrained, 123

Convex set, 165

Cost function, 6

Cycloids, 44

Hamiltonian, 13, 35, 59

Hamilton-Jacobi-Bellman equation, 91, 98,

99

Hyperplane, 165

I

Inventory control, 44, 46, 48, 135, 137

J

Jump conditions, 126

M Marginal costs, 92

Maximum principle

autonomous problems, 12, 35, 36

nonautonomous problems, 59

Moon-landing problem, 4, 50

179

180

Index

N Null controllability, 111

o Observability, 116, 118

Optimal control, 6

p Performance index, 6

Planning horizon, 137

Principle of optimality, 87

R Reachable set, II

Restraint set, 6

Restricted maximum principle, 125

Riccati equation, 94

Routh-Hurwitz theorem, 176

s Salvage cost, 61

Stability, 115

Stabilizability, 113

State equations, 5

Strictly convex set, 19, 168

Supporting hyperplane theorem, 168

Support point, 165

Switching locus, 16, 23, 26, 27

Switching times, 13

numerical solution, 29

T Target point, 6

Terminal payoff, 61

Transversality conditions, 37, 60

Two-point boundary-value problems, 71

finite difference scheme, 81

implicit boundary conditions, 77

method of adjoints, 73

multiple shooting, 82

quasi-linearization, 79

shooting methods, 76

v

Value of a differential game, 97

Variational inequalities, 149

w Weierstrass E function, 57