The Approximate Minimization of Functionals

THE APPROXIMATE MINIMIZATION

OF FUNCTIONALS

THE APPROXIMATE MINIMIZATION

OF FUNCTIONALS

JAMES W. DANIEL Computer Sciences Department University of Wisconsin

PRENTICE-HALL, INC.

ENGLEWOOD CLIFFS, N. J.

PREFACE

This material was prepared for use as a text in support of my lectures at the 1969 session on "Minimization problems" of l'Ecole d'Ete Analyse Numerique

held at Le Breau-sans-Nappe, France, and sponsored by the Commissariat a l'Energie Atomique, Gaz de France, and l'Electricite de France; I gratefully acknowledge the financial support of those organizations and their cooperation in reproducing this text. Of course those ideas in this text which are primarily the author's did not come to light only during the period of writing; therefore I must also acknowledge several other sources of support in recent years, particularly the National Science Foundation, the- Office of Naval Research, the Mathematics Research Center at the University of Wisconsin, and the Computer Sciences Department at the University of Wisconsin, Madison. It is a great pleasure fpr me to acknowledge the invaluable coopera-

tion of my three assistants at I'Ecole d'Ete, Mssrs. Chavent, Yvon, and Tremolieres, and the students there, especially Mr. Lascaux, whose questions, comments, and suggestions were of great help to me. This book represents an attempt at a general presentation of problems

involved in the approximate minimization of functionals. As such it treats in the first chapter the general area of variational problems and the question of the existence of solutions, in the next two chapters the question of the convergence of approximate solutions of discretized variational problems in general and in specific important cases, in the next two chapters the theory of gradient methods of minimization in general spaces, and in the final four chapters practical computational methods of minimization in ER'. Although a significant amount of material is presented concerning constrained minimization problems, the primary emphasis of this text is on unconstrained problems, particularly in the last four chapters; this is not a text on mathe-

Vill

PREFACE

matical programming. In the last four chapters on "practical" methods, detailed computational algorithms or computer programs are not presented. The rather "theoretical" presentation of this "practical" material reflects my feeling that analysis and understanding of an algorithm, even under specialized

hypotheses, is all-important in the creation of new methods; whenever pos-

sible, however, the references include sources of computer programs or detailed computational descriptions of the implementations of algorithms. The references presented here give a reasonably thorough coverage of the most important English language literature on minimization methods, although no claim is made of completeness. Although many of the important results of Russian papers can also be found in English, the largest gaps in the references are with respect to the very extensive literature in Russian. I hope to correct this shortcoming at some future time. JAMES W. DANIEL

Madison, Wisconsin

CONTENTS

1

VARIATIONAL PROBLEMS IN AN ABSTRACT SETTING

I

I.I. Introduction

1

1.2. Typical Problems 1.3. Basic Functional Analysis 1.4. General Functional Analysis for Minimization Problems 1.5. The Role of Convexity 1.6. Minimizing Sequences

2

THEORY OF DISCRETIZATION

3

7 13

17

28

2.1. Introduction 2.2. Constrained Minimization 2.3. Unconstrained Minimization 2.4. Remarks on Operator Equations

3

3

EXAMPLES OF DISCRETIZATION

28 28 31

34

37

3.1. Introduction 3.2. Regularization 3.3. A Numerical Method for Optimal Control Problems 3.4. Chebyshev Solution of Differential Equations 3.5. Calculus of Variations 3.6. Two-Point Boundary Value Problems 3.7. The Ritz Method ix

37

37 41

52 55

59 65

X

CONTENTS

4 GENERAL BANACH-SPACE METHODS OF GRADIENT TYPE

70

4.1. Introduction 4.2. Criticizing Sequences and Convergence in General 4.3. Glgbal Minimum along the Line 4.4. First Local Minimum along the Line: Positive Weight Thereon 4.5. A Simple Interval along the Line 4.6. A Range Function along the Line 4.7. Search Methods along the Line 4.8. Specialization to Steepest Descent 4.9. Step-Size Algorithms for Constrained Problems 4.10. Direction Algorithms for Constrained Problems 4.11. Other Methods for Constrained Problems

70 71

76 77

80 84 89 93 95 105

110

5 CONJUGATE-GRADIENT AND SIMILAR METHODS IN HILBERT SPACE

5.1. Introduction 5.2. Conjugate Directions for Quadratic Functionals 5.3. Conjugate Gradients for Quadratic Functionals 5.4. Conjugate Gradients as an Optimal Process 5.5. The Projected-Gradient Viewpoint 5.6. Conjugate Gradients for General Functionals 5.7. Local-Convergence Rates 5.8. Computational Modifications

6

GRADIENT METHODS IN IR'

6.1. Introduction 6.2. Convergence of x,+ i - x, to zero 6.3. The Limit Set of {x,} 6.4. Improved Convergence Results 6.5. Constrained Problems 6.6. Minimization along the Line

114 114 114 117 119 122 125

127

136

142 142 142 145

150 154 154

7 VARIABLE-METRIC GRADIENT METHODS IN IR'

7.1. Introduction 7.2. Variable-Metric Directions 7.3. Exact Methods for Quadratics 7.4. Some Particular Methods 7.5. Constrained Problems

159 159 159 165 168

178

CONTENTS A

8 OPERATOR-EQUATION METHODS

8.1. Introduction 8.2. Newton and Newton-like Methods 8.3. Generalized Linear Iterations 8.4. Least-Squares Problems

9

AVOIDING CALCULATION OF DERIVATIVES

9.1, Introduction 9.2. Modifying Davidon's First Method 9.3. Modifying Newton's Method 9.4. Modifying the Gauss-Newton Method 9.5. Modifying the Gauss-Newton-Gradient Compromise 9.6. Methods Ignoring Derivatives

180 180 181

185

190

194 194

194 197

200 201

203

EPILOGUE

211

REFERENCES

213

INDEX

225

1

VARIATIONAL PROBLEMS IN AN ABSTRACT SETTING

1.1. INTRODUCTION

Many problems of pure and applied mathematics either arise or can be formulated as variational problems, that is, as problems of locating a minimizes

ing point for some (nonlinear) real-valued functional over a certain set. Such a setting is often beneficial from the analytic viewpoint of determining the existence and uniqueness of such points; however, we shall primarily emphasize here the computational aspects of this approach, that is, how we can compute a solution to the problem by actually minimizing the appropriate functional. In some cases, in which the problem was originally formulated as a variational problem (e.g., calculus-of-variations problems), this may well be one of the best computational approaches, rather than considering the problem in some equivalent form (e.g., the Euler-Lagrange differential equation). In other cases, in which the problem is artificially converted into a variational one, this approach provides new computational methods for consideration. General references: Kantorovich-Krylov (1958), Courant-Hilbert (1953), Kantorovich (1948). 1.2. TYPICAL PROBLEMS

The variety of minimization proI ems is indeed immense; we consider some typical problems. A large class is that of optimal-control problems, which arise so often in the highly technological industries. Here one seeks to minimize a cost functional

f(x, u) = f c[t, x(t), u(t)] dt u

1

2

VARIATIONAL PROBLEMS IN AN ABSTRACT SETTING

SEC.

1. 2

over the collection of points (functions) x, u satisfying constraints such as x - d1 = s[t, x(t), u(t)) X(O) E X, X(t p) E Xp

x(t) E X(t) U(1) E U(t)

where X Xp, X(t), and U(t) are certain specified sets and set functions. Such problems arise, for example, in determining the minimal time or minimal expenditure for striking a given missile target.

As a special case with s(t, x, u) - u, we have a basic problem of the calculus of variations, in which we seek to minimize

f(x) = f c[t, x(t), *(t)] dt 0

subject to some boundary conditions on x(t), such as x(0) = x(1) = 0

Many problems of applied mathematics are of this latter form if we allow the variable t to represent a vector of dimension higher than one and x to represent the vector of first partial. derivatives of x with respect to those variables. The classical problem of finding the height x(t,> t2) of a surface of minimal area stretched across a closed space curve C = c(t t2) for (t t2) E G, a simple closed curve in the plane, for example, is most naturally described mathematically as the problem of finding x(t t2) to minimize

f rl

+

(d)2

+

(dX)2] 112

)

dt, dt2

(G = region enclosed by G)

over the set of x(t t2) satisfying

x(t t2) = c(t t2) for (t t2) E G A discrete analogue of the continuous optimal-control problem described above is the mathematical programming problem in which one seeks to minimize some function of a finite number of real variables subject to finitely many constraints; these problems arise, for example, very commonly in the petroleum industry. As we shall later see, it is also often solved as a discretized approximation to a continuous optimal-control problem. Many problems in data approximation ultimately are of this form also and provide perhaps one of the most common forms of such problems. Where one has

SEC. 1. 3

VARIATIONAL PROBLEMS IN AN ABSTRACT SETTING

3

some data y, depending on a parameter t measured at certain points t;, i = 1, 2, . . . , N, it is desired to approximate the formula generating the data by some expression

y(t) = g(t; a) where the choice of the parameter a determines the expression. We can do this by picking a so as to minimize some norm of the vector in E^' with components

Y,-g(ti;a). Essentially, this technique has been applied lately in a somewhat novel fashion for the solution of such problems as differential or integral equations. For example, one seeks to solve the differential equation

Du =f in

C,

where C is some domain. If q

u = 0 on dC = boundary of C . .

., 97,Y are some functions satisfying the boun-

dary condition on dC, we try to choose numbers a ... , a,,, to minimize some norm of the vector in EM with components given by

[Du](t) -f(t),

i= 1, ... , M

where {t,) is some grid of points over the domain C; alip, + ... + axqv is then taken as an approximate solution to the differential equation. This represents only a few of the types of variational problems commonly of practical interest at pcesent; rather than proceed to list more types, we shall develop tools to allow us to consider such problems in a more general setting.

General references: Hestenes (1966), Pontryagin et al. (1962), Balakrishnan-Neustadt (1964), Courant-Hilbert (1953), Morrey (1966), Akhiezer

(1962), Abadie (1967), Fiacco-McGormick (1968), Hadley (1964), Mangasarian (1969), Zangwill (1969), Lorentz (1966), Rosen-Meyer (1967), Rabinowitz (1968), Mikhlin-Smolitskiy (1967). 1.3. BASIC FUNCTIONAL ANALYSIS

We shall consider variational problems which can be stated as problems in a general function-space setting; for simplicity, we restrict ourselves to problems in Banach spaces. We recall that a real or complex Banach space E (hereafter assumed to be real unless otherwise stated) is a vector space over the real or complex numbers which is complete in the topology generated by the norm I

I defined on E. A Hilbert space is a Banach space in which the norm is given via an inner product < -, -), that is, IIxl I = `x, x>'-z. I

-

I

4

SEC. 1. 3

VARIATIONAL PROBLEMS IN AN ABSTRACT SETTING

The Banach space E* of bounded (real-valued) linear functionals on E is called the dual or adjoins space; it generates another topology for E, the weak topology, for which a basis at the origin 0 is given by

(0,0 =(x;If,(x)I 0,

i= 1,2,...,n

I

A set C in E is said to be convex if every convex combination of every finite subset of C is itself an element of C. The following important result is known:

If x - x, then for each n = 1, 2, ... , there is a convex combination y of .the points x ... , x such that y. x. This is implied by the fact that a convex set which is closed in the norm topology must be closed in the weak topology; a weakly closed set is always norm-closed of course since a norm convergent sequence is weakly convergent.

SEC. 1. 3

VARIATIONAL PROBLEMS IN AN ABSTRACT SETTING

5

EXERCISE. Prove that the fact that a convex set is norm-closed if and only if it is weakly closed implies the fact that for every weakly convergent sequence that norm-conof convex combinations of there is a sequence verges to the same limit.

In the sequel we shall be considering (real-valued) nonlinear functionals f on a space E and various continuity properties of those functionals. Recall

that a functional f is lower semicontinuous in a topology if and only if (x; f(x) > a) is open for each real a. From this we deduce that if f is weakly. lower semicontinuous-that is lower semicontinuous in the weak topology-

then from x, -, x it follows that f(x) < lim inf f(x.) A similar statement holds for norm lower semicontinuity, of course, under the hypothesis that x -. x. Since most of our arguments depend only on the above sequential property, we define a functional to be sequentially lower semicontinuous (for some topology) if and only if the convergence of (xe) to x impliesf(x) < lim inf Sequential continuity properties are generally weaker than the usual continuity properties; the two kinds of properties are equivalent in the norm topology since the norm topology has a countable basis at each point. EXERCISE. Prove that a functional on a normed space is norm sequentially lower semicontinuous if and only if it is norm lower semicontinuous.

We shall often consider linear mappings from one Banach space E, into another E2; If A is such a mapping, then it is continuous (between the norm topologies) if and only if it is bounded, that is, II AxIIE,< IIAI1=sup IIAxIIE,=sup IxIIE,=I xeE,

0o

IIXIIE,

The Banach space of all such bounded linear operators is denoted by L(E E2); thus E* = L(E, IR). If A E L(E E2) and f2 E E2, we can define

f, EE*by

f1(x) =f2(Ax) The mapping of f2 into f, is linear and is denoted by At, the adjoint of A. If E is a Hilbert space, add if A E L(E, E) and At = A, then A is called self-adjoint; in this case we have _ (x, A*y> = <x, Ay> for all

x,yEE.

If E is a real (complex) Banach space and A E L(E, E), then a(A), the spectrum of A, is defined via

6

SEC. 1.3

VARIATIONAL PROBLEMS IN AN ABSTRACT SETTING

a(A) = [A E IR (Q); Ax - Al - A is not one to one, L[R(A,), E]] Ax' or R(A,) # E, or

where R(A,) denotes the range of A, and S denotes the norm closure of any set S. If Eis a Hilbert space and A is self-adjoint, then all points irl a(A) are real and lie in the interval [m, M] where

m = IEf

and

<x, z>

M = sup E

<X, >

Then we can write

ml < <x, Axy < M<x, x>

It follows that A 11 = max (I m I, I MI). If m > 0 (> 0), A is called positivedefrnite (semidefinite) or coercive (semicoercive). Examples of simple spaces 1. E = IR", dimensional real Euclidean space with any norm whatsoever since all norms on IR" are topologically equivalent. Some standard

norms are, for x = (x,, ... , x"): IIxII",ve =

maxlx,l

and

1SiSi

IIxII",, -

lEIx1I,]1/,

in the second equation particularly for p = 1 or p = 2, in which latter case E is a Hilbert space. 2. E = C[a, b] is the real Banach space of real-valued continuous functions defined on [a, b] c iR with the norm I If I I- = max I f(t) 1; norm converla bl

gence here is equivalent to the usual uniform convergence. E is not reflexive.

3. E = L,(a, b), the Banach space of real-valued pth-power Lebesgue integrable (if p = oo, then essentially bounded instead) functions defined on (a, b) c IR, with norm III III, = [J: I f(t)11, dt]

'P

or

IIxII-

esssuplf(t)I (a, b)

For I < p < co, L,(a, b) is reflexive; L,(a, b) is a Hilbert space. 4. E = W;(a, b), the Banach space of all real-valued functions defined on [a, b], having absolutely continuous derivatives of orders less than or equal to (m - 1) on [a, b], and having the mth derivative (and of course all

SEC.

VARIATIONAL PROBLEMS IN AN ABSTRACT SETTING

1. 4

7

lower-order derivatives) in LD(a, b). Many equivalent norms are possible for this Sobolev space, one of which is I/P

IIf1I =

{ 1-0

Ifs"(a)1°+ JlfT(t)I°dt o

)

Ifp = 2, WZ is a Hilbert space with the obvious inner product. (Such spaces can be defined in higher dimensions, although the concept of the derivatives of absolutely continuous functions must be replaced by that of generalized or distributional derivatives, and the norm should be the sum of the L, norms of all the derivatives of order i for 0 < i < m.) For clarity we consider the Hilbert space WZ(0, 1). Here we have I If I

I = [=]12 ° {If(o)12 + f o If'(t)12 dt)12

Thus, if f. -f in WZ, then f. converges to f' in L2(0, 1) and f converges to f uniformly, that is, in C[0, 1]. Even weak convergence in W21 implies a

strong type of convergence. For, if f -f in W;,, then f(0) converges to f(O) and f;, converges to f' weakly in L2(0, 1). From this it follows that (f }, is equicontinuous and uniformly bounded, which implies that f,, converges to f uniformly on [0, I], that is, it converges in C[0, 1].

EXERCISE. Provide the details proving that weak convergence in WZ implies, norm convergence in C[0, 1].

Since many algorithms yield sequences only weakly convergent to some desired solution, it is important to realize, as this example indicates, that weak convergence can sometimes be quite "strong."

General references: Dunford-Schwartz (1962), Kantorovich (1948),. Kantorovich-Akilov (1964), Morrey (1966), Taylor (1961). 1.4. GENERAL FUNCTIONAL ANALYSIS FOR MINIMIZATION PROBLEMS

Perhaps the fundamental result concerning the existence of a solution for a variational problem is the well-known general fact that a sequentially lower semicontinuous real-valued function defined on a countably compact set must achieve its infimum there; the important special case for our use is the following.

THEOREM 1.4.1. Let f be a weakly sequentially lower semicontinuous functional defined on a weakly sequentially compact set C; then there exists

xQ in C such that f(xo) = inff(x) = min f(x). xEC

xEr

8

SEC.

VARIATIONAL PROBLEMS IN AN ABSTRAO SETTING

1. 4

Proof: Let m = inff(x); then there exists a sequence of points x E C, xEC m. Since C is weakly sequentially compact, there is an with xo E C and a subsequence x such that lim f(x,,) = m and x - xo. By the semicontinuity,

f(xo) < lim inf

m < f(xo)

Thus f(xo) = m. Q.E.D. COROLLARY 1.4.1. A weakly sequentially lower semicontinuous functional achieves its infimum on every convex, bounded, norm-closed subset of a reflexive space. Proof: From earlier statements we know that a convex, norm-closed set is weakly closed and that a weakly closed, bounded set is weakly sequentially compact in reflexive space; the corollary then follows immediately from the theorem. Q.E.D. EXERCISE. Prove that a lower semicontinuous functional achieves its infimum on each countably compact set.

Theorem 1.4.1, of course, directly applies only to constrained minimization problems, that is, where x is considered only in some-weakly sequentially

compact, and hence bounded, set C; our maim concern here is with unconstrained problems, but for proving existence of a solution it is often possible and wise to reduce them to constrained problems. (An interesting sidelight is that for computational purposes one often tries to reduce the constrained

problem to an unconstrained one.) Here one tries to find a bound-say B-on the norm of the solution if it exists, and then consider the problem over S. = {x; I I x I I S B). This is possible if, for example, the functional satisfies a T-property in a reflexive space.

DEFINITION 1.4.1. A functional f is said to satisfy a T-property (at xo

with T = To) if there exists an xo and To > 0 such that I Ix implies f(x) > f(xo).

-- xo I1

To

THEOREM 1.4.2. A weakly sequentially lower semicontinuous functional f satisfying a T-property in a reflexive space E achieves its infimum over E.

Proof: Let S(xo; T0) be the norm-closed sphere of radius To about xo; then S(xo; To) is weakly sequentially compact. Since f(xo) > 0, and Q.E.D. Vf(x) E E*, this implies f(x) < lim We can deduce growth conditions in a similar manner. THEOREM 1.4.4. Let f have first and second derivatives throughoui.

E and let f X satisfy at each x E E the condition > I I h I Ig(I I h I I) where g is a nonnegative continuous function for t > 0 tending to infinity

with t. Then lim f(x) _ +oc I;x i -

Proof: Let s(t) = <x, Vf(tx)> for 0 < i < 1; then s(1) = s(0) + f0 s'(t) dt, that is, <x, Vf(x)* = <x, Vf(0)> + f ( <x, f;xx> dt

Therefore, <x, Vf(x)> >- I I x I I I Vf(0) I I+ I I

I x I Ig(I I x 11). Since

f(x) = f(0) + f <x, Vf(tx)> dt 0

= f(0) + f o t dt

= f(0) - IIxIl IIVf(0)II + IIxII f ' g(tIIxll)dt >_ f(0) + 11X11 I- I I

VP I I + f " g(:I I x 11) dt, 0

and since limg(t) = oo, we then conclude that lim f g(t1Ix11) dt = 011. OxU-.

0

Q.E.D.

EXERCISE. Consider

f(x) = f [ [1(t)]2 + c[t, x(t)1} dt,

1(1) = d x(t)

for x in c

W21

(See Example 3, Section 1.3.) We let 11 x112 = f o [x(1)12 dt. Suppose

12

VARIATIONAL PROBLEMS IN AN ABSTRACT SETTING 2

cum(t, u)

=

for all t, u. Prove that lim f(x)

SEC.

1. 4

t, u ) > y > -7c2

dug

+oo. Hint: Use the fact that for all

x E E we have f0 [z(t)]2 dt > nz f o [x(t)]z dt to prove that Z e, for small enough e > 0.

As we stated earlier, we wish to see how minimization problems are related to the solution of operator equations. The fundamental statement is as follows. THEOREM 1.4.5. Let the functional f defined on a set Sin a Banach space

E be minimized at a point xo E S, with x0 an interior point in the norm topology. If f has a derivative Vf(xo) at x0, then Vf(xo) = 0. Proof: For any fixed h E E, f(xo + th) is a real-valued function of the real variable t on some open interval containing zero, having a derivative at zero, and being minimized there. Therefore,

f(xo + th)

0=

d

G-o

=

Since h E E was arbitrary, = 0 for all h c= E and, therefore, Vf(xo) = 0. Q.E.D. Similarly, minimization problems over certain types of sets can be related to generalized eigenvalue problems via Lagrange multiplier theorems. As an example, we merely state one type of such result.

PROPOSITION 1.4.2. Let f and g be functionals on a Banach space E, both of which are differentiable at the point xo which is local minimum off on the set [x; g(x) = g(x0)}. If I I Vg(xo) I I > 0, then there exists a real number A such that Vf(x(,) = t,Vg(x,,). Proposition 1.4.2 gives a necessary condition for a point to give a local minimum over a set defined by one real equality constraint. Much of the theory of optimal control and mathematical programming involves determining necessary conditions and also additional sufficient conditions for a point to be an extremizing point over sets with more complicated constraints. Another simple necessary condition for xo to minimize a differentiable f(x) over a convex set C is that <x - x0, Vf(x,,)> > 0 for all x in C. The amount of literature in this extremely complex area is vast. Since our main concern here is with the theory of and methods for approximate solutions primarily

to unconstrained problems, we shall pursue this no further. General references: Levitin-Poljak (1966a), Vainberg (1964).

VARIATIONAL PROBLEMS IN AN ABSTRACT SETTING

SEC. 1. 5

13

1.5. THE ROLE OF CONVEXITY

We saw already in Corollary 1.4.1 that convexity of the set over which we seek a minimum can be of importance in proving existence of a solution; various kinds of convexity assumptions about the functional to be minimized also play a vital role. We shall consider next some examples of how these properties are related to continuity properties of the functional and to the nature of the points solving the minimization problem. Although more general definitions are possible, for the following let C be a convex subset of a Banach space E. DEFINITION 1.5.1. A functional f is convex (strictly convex) on C if and

only if for each x x2 in C and A in (0, 1),

f[Ax + (1 - x)x2l < (_ (>) <x2 for each x x2 in C, x,

x., Vf(x1)>

x2. Another equivalent condition is that

<X2 - x1,

Vf(x2) - Vf(x1)> >_ (>) 0

for x x2 in C, x, $ x2. If f x exists in the convex open set C, then f is convex if and only if 0 with 6(t) = 0 if and only if t = 0, such that

f(x 2 Y) S max (f(x),f(y)} - b(II x - y1D for all x, y in C. EXERCISE. Find a condition equivalent to the one in Definition 1.6.2 for f[2x + (1 - A)y], A in [0, 1].

THEOREM 1.6.1. Let C be a norm-closed, bounded, convex subset of a reflexive space E; f a weakly sequentially lower semicontinuous uniformly quasi-convex functional on a norm neighborhood of C; and an approximate minimizing sequence. Then x, x*, the unique point in C minimiz-

ingf. Proof. By Proposition 1.6.1, there exists at least one subsequence x,,, weakly converging to a minimizing point x'. But the minimizing point x' = x* clearly is unique, just as in Theorem 1.5.2, so x -, x*. Since f(xn) converges tof(x*), since (x + x*)/2 -- x*, and since f is weakly sequentially lower semicontinuous, for all c > 0 there is an N, such that n > N, implies

I f(xn) -f(x*) I < E

and

fx

*) > f(x*) - E

19

VARIATIONAL PROBLEMS IN AN ABSTRACT SETTING

SEC. 1. 6

By the uniform quasi-convyity, a(I l x

f (x*)} - f

x* I I) < max (f

X.

±

x*) < 2e for n

N,

and hence I I x. - x* I I -- 0. Q.E.D.

It is also possible to prove strong convergence recalts on the basis of properties of C, not f. DEFINrrION 1.6.3. A set C is called uniforrr;y convex if and only if there

exists a real-valued, continuous, monoton function 8(t) for t > 0 with 8(t) =0 if and only if t = 0, such that ., y c- C and I I z I I < S(I I x - y I ) imply (x + y)/2 + z E C. THEOREM 1.6.2. Let C be a no .n-closed, bounded, uniformly convex subset of a reflexive space E; f a v sequentially lower semicontinuous functional on a norm neighborhoc i of C with a unique point x* minimizing f on C; x* on the norm boundary )f C; and (xc} an approximate Minimizing sequence. Then x. -* x*. Proof.: By Proposition 1.6 L and the uniqueness of x*, we have x, -> x*. Since d(x,,, C) ---. 0, there i, a sequence x; E C with Jim I Ix. - x; I I = 0.

Suppose there exists e > ' and a subsequence (x;,} with I x', - x* I I > e for all i. By the uniform onvexity of C, if I I z I I < b(e), then I

x",+x*+zCC 2

for all i. Since

x',,+x* 2 -l- z -- x+ z and C i, weakly closed, x* + z E C for all z satisfying I z I I < b(e) > 0. Then c* is not a boundary point, yielding a contradiction. Thus x. (and henc also converges in norm to x*. Q.E.D. I

EXERCISE. Consider f(x) = f (t[x(t)]2 + V0171 dt on o W](0, 1), the 0 set of x in W2(0, 1) with x(0) = x(1) = 0 and with IIx112

f 1 [x(t)]2 dt. Let 0

n1/2t xw(t)

for 0 y > -x2 for all u in (-oo, oo), tin [0, 1]. Let (u, v) = f u(t)v(t) dt. Then 0

flu + h) =f(u) + (-D2U + c(t, u), h) + 4([- D2 + c (t, u)]h, h) + small terms Thus

_ (-D2u + c(t, u), h)

VARIATIONAL PROBLEMS IN AN ABSTRACT SETTING

SEC. 1. 6

25

and

2, and I/p + 1/q = 1; that K(t, T) = K(r, t), that the spectrum of A as an operatodfrom Lq into Lq is positive and that A maps bounded sets into precompact sets (that is, A is a compact operator). We suppose that the operator

c(u) = c[.,

is norm-continuous from L,(0, 1) into L,(0, 1)-i.e., that I c(t, u) I < a(t) + b I u li JF, where a E L,,(0, 1), and b > 0; that c(t, u) is continuous in u for almost all t and measureable in t for all u; and that C(t,u)

fc(t,s)ds 0

satisfies

C(t, U) < Gcu2 + fi(t)' u jr + d(t) where

a E (0, m),

m = inf {A; A E Q(A)},

0 < f E L2,(,_,)(0, 1) for some F in (0, 2),

and

0 < b E L1(0, 1)

Then we can write

A = GG*

where G is positive and compact from L2(0, 1) into L,(0, 1), and deduce that the functional

f(x) = <x, x> - 2 f 1 C[t, G ,,:(t)] dt 0

is defined on L2(0, 1) and achieves its minimum there. At the minimum x*, Vf vanishes, and so

THEORY OF DISCRETIZATION

SEC. 2. 4

35

0 = Vf(x*) = 2x* - 2G* c(Gx*) Defining u* = Gx* E Lp(0, 1), we see that u* = Ac(u*) and that u* solves the integral equation. if we define

J(w) = - 2 f t C[t, w(t)) dt 0

for w e L,(O, 1), we see that f(x) = J[G(x)]

as described by the Aubin-Lions work. Thus the theory described in Section 2.3 applies to numerical solution of integral equations, where integration is, for example, discretized by means of quadrature formulas. A particularly attractive feature of integral equations is the compactness of the operator A. If A is approximated by a quadrature sum, A. u = t=t w.,, K(., T;) u(zt) = Q,,[Ku)

where the quadrature formula Q. gatisfies Jim --.m

f l = f ' f(t) at o

for all continuous f, then the operators A. turn out to be collectively compact in many cases; that is, the union of the images by A. of each bounded set is

compact. This fact has been exploited greatly [Anselone (1965, 1967), Anselone-Moore (1964), Moore (1966)] to analyze numerical methods for linear integral equations. Essentially, the same viewpoint has been used to analyze nonlinear equations given by variational problems [Daniel (1968a)]. A typical result using this viewpoint is as follows. If f and f., n = 1 , 2, ... , are weakly lower semicontinuous functionals such that for each x in a weakly compact subset C of a reflexive space E we have

lim f (x) =f(x) and such that {Vf - Vf } is a collectively compact set of norm-continuous mappings of E into E*, then if x,* E C satisfies f(x.) < inf c,, x E C

36

sec. 2.4

THEORY OF DISCRETIZATION

0, it follows that every weak limit point of {x,*), at least one of with e. which exists, minimizes f over C. EXERCISE. Show that the collective compactness referred to above merely serves to guarantee the consistency of the discretization with C. = C, E. = E,

p. = r = the identity map. Thus most results of Daniel (1968a) follow from either of the above Theorems 2.2.1 or 2.3.1. It is a trivial exercise further to deduce from these theorems results concerning the solution of operator equations via discretizations; one need only recall that Vf(x*) = 0 at an interior minimum of f. Thus one is led to results concerning the weak convergence of p x,* to x* where

Vf.(x.) = 0

and

Vf(x*) = 0

Stronger convexity hypotheses on f will then give norm convergence. The analysis of convergence for discretization methods of solution of nonlinear equations has been carried much further, however, than can be covered from the variational viewpoint. Rather than give such an incomplete picture of the subject, therefore, we merely refer the interested reader to the literature. We proceed, in the following chapter, to examine a number of examples to which the variational viewpoint naturally applies in order to demonstrate some particular cases of discretization methods. General references: Aubin (1967a, b; 1968), Browder.(1967), Petryshyn (1968).

3

EXAMPLES OF DISCRETIZATION

3.1. INTRODUCTION

In the well-developed theory of discretizations for operator' equations, many examples of particular discretization schemes can be found, particularly

for partial differential equations (see the General References at the end of this section for such general examples). In this chapter we shall examine some specific types of problems or methods which, by our considering a particular form for the discretization, can be analyzed from the viewpoint of the discretization theory of variational problems and from the theorems presented in Chapter 2 or extensions of those theorems. In some cases this leads to new results, in some it provides a different way of looking at well-known results, and in one it shows how the approach can be used to guide the direction of one's research on a new method. General references: Aubin (1967a, b; 1968).

3.2. REGULARIZATION

The idea of regularization has been studied from at least two different viewpoints. Under the name of regularization it was developed theoretically largely by the Russian school [Levitin-Poljak (1966b), Tikhonov (1965)] for the situation in which one seeks to minimize a functional g and, out of all the solutions to this problem, find the one which is "smoothest" or "most regular" with respect to another functional h-that is, which minimizes h over the set of solutions of the first problem. If g represents a calculus-of-variations problem, for example, one might take. 37

38

SEC. 3. 2

EXAMPLES OF DISCRETIZATION

h(x) = f' I z(t) V z dr 0

to make x "smooth." This goal can be accomplished in many cases by minimizing

g + a"h,

a" > 0, and lima" =0

where

(3.2.1)

and noting that the solutions to these problems converge to the desired regular solution. This same technique has also been studied as a form of the penalty function method, since minimization of the functional in Equation 3.2.1 is equivalent to minimizing

h + 1a"g,

where

a" > 0,

lima" = 0

and

In this form we recognize the procedure as a form of the penalty-function technique to minimize h over the set of x satisfying g(x) = 0, if g(x) > 0 for all x [Courant (1943), Butler-Martin (1962)]. We shall briefly consider this method (from the regularization viewpoint) as a discretization. First we shall generalize it somewhat because of its relevance for numerical work, and then we shall specialize to the above description. Suppose we seek to minimize the nonnegative weakly sequentially lower semicontinuous functional h over the set of points which minimize the weakly sequentially lower semicontinuous functional g over a weakly sequentially compact subset C of a reflexive space E. Suppose we have a discretization for this problem described by [E", g", h", C", p", r"}. For a sequence of positive a" tending to zero we shall define

.f"(x") = g"(x") + a,h"(x"),

for

x" E E"

We also define

f(x) = g(x),

for

xEE

and thus. we have a discretization [E", f", C", p", r,j. We list assumptions for this example corresponding to the consistency definition (Definition 2.2.2), but stronger:

1. lim sup [g"(r,rx*) + a"h"(r"x;) = g(x*) - ah(x`)] = lim sup Z > 0

" " A--

for every x minimizing g over C; 2. lira sup a,h(p-x*) - g"(x.) - a,h"(x,*)] = lim sup 8" S 0 if x,* erapproximately minimizesf" over C";

3. the sets C" = p"C" U C are uniformly bounded, and, if z", E C with z", z, then z c C;

EXAMPLES OF DISCRETIZATION

SEC. 3. 2

39

4. solutions x; exist; E C. for every x* minimizing g over C. 5. THEOREM 3.2.1. Suppose x.* satisfies j,

inf [g.(x.) + a h.(x.)) + C. g.(x.) + a.h.(x ) < x.CC. Under hypotheses 1-5 (above) on the discretization for the weakly sequentially lower semicontinuous functionals g and h, with h > 0, h. > 0, and C weakly sequentially compact, if in addition all > 0 converges to zero slowly enough

that urn supC, n-.o^

+§.+En+y,, 0, lim a" = 0. For each n, let x* satisfy

\

S

g"(x*) + a,,h"(x*) S g"(x") + anh"(x") + a"8"

for.all x" E Q.

where 6" Z 0, lim 6" = 0. Then all weak limit points x' of p"x*, at least one

of which exists, solve the C-problem-i.e., if x' = (y, u), then y = s[t, y(t), u(t)] almost everywhere, x' E Q0, and h(x') < h(x) for all x in Q0. Proof: We wish to apply Theorem 3.2.1, if possible; we check five numbered hypotheses preceding that theorem with C =_ Q0, C. = Q". Number 1 is true by Assumption A4, but only for some x*, not all x*; No. 2 is valid by Assumption A5; No. 4 is assumed above; No.5 is valid by Assumption A4,'

but only for some, not all, x*. For condition 3, we note that C" - p"C" U

C e Q". If z", e C and z", -- - z, since Q"+' c Q" and Q" is weakly sequentially compact because of Assumption A2, we conclude that z E Q for all i and hence z e n Q. c Q0, as demanded by condition 3. Although we cannot exactly apply Theorem 3.2.1, we can follow the lines of its proof, making use of additional information we have in this case. Thus we can conclude, as in Theorem 3.2.1, that a weak limit point x' of exists, must lie in Q0, and minimizes g over C-that is, g(x') = 0.

The hypotheses I through 5 and that on the decay rate of a" were used in Theorem 3.2.1 only to show that h(x') < h(x*) (where x* solves the Cproblem in our case); we can handle this differently. We have

0= g"(r"x*) < g"(x:) < g"(x*) +- ah"(x*) < g"(r"x*) 4 a,h"(r"x. ) + a"v".< g,,(xn) -

anh"(r,x*) + a"a"

which implies

h"(x*) < h"(r"x*) + J. Then

h(x') < lim inf

lim inf h"(x*)

< lim inf [h"(r"x*) + 8"] = h(x*)

Since h(x') < h(x*) and x* minimizes h, so must x'. Q.E.D.

SEC. 3.3

EXAMPLES OF DISCRETIZATION.

47

EXERCISE. Provide all the details for the Proof of Theorem 3.3.1 above.

The results of Section 3.2 concerning stronger convergence properties of course apply here also, but we shall not state them again. Rather we must consider conditions under which the assumptions in the theorem are true. The theorem above (Theorem 3.3.1) merely serves to identify conditions sufficient to guarantee the applicability of the numerical method of Rosen (1966). We note that the existence of p. is important only to the proof, while

the existence of r. and the numbers d related thereto are crucial to the numerical algorithm itself; we are required to treat the P-problem over Q., a set defined via d., and we must therefore know d. in order actually to compute. In Cullum (1969), it is shown that, for certain problems, if the sets Y(t) are expanded by distances y., U(t) by distances a and a. discretized step size k of length k = 1/m is used, then sequences m(n) and 1(n) exist such that maps

p., r. exist for the problem defined by y., a,(.), k = 1/m(n), with d S y. + 0g.). This does not really yield a computational procedure, since for a given sequence of step sizes k we still do not know by how much to expand the constraint sets. Now we shall attempt to make the numerical method really implementable; another approach to this can be found in Cullum (1970) for problems lacking state constraints. First, however, we remark that the assumptions other than A4 and A5 are reasonable assumptions insofar as the existence of the solution to the C-problem and the computability of approximate solutions of the P-problem are concerned. In Rosen (1966), for example, in order

to prove that the numerical method used there for minimizing g. + a.h. works, it is assumed that s(t, y, u) and c(t, y, u) are convex jointly in y and u; it is a simple matter to show-using this assumption, the assumptions in the

paragraph following A2, and the additional one that s,(t, y, u), S.(t, y, u), c,(t, y, u), c.(t, y, u) exist and as functions of t are in L2(0, l) for fixed (y, u) E E-that f, h, Q satisfy their needed assumptions. We therefore do not discuss these assumptions further. EXERCISE. Indicate how the assumptions of the preceding paragraph can be

used to deduce that f, h, and Q satisfy the assumptions demanded by the theory developed so far.

Let us consider the mapping p.; we must apply it to points X. _ (y., u.) satisfying

y,..,+, _ y,,; + ks(t, y.,,, u.,,) - kb.,, with b.,, > 0 for 0 < i

n-I

If we define w.(t) = p.u. as a step function constant on each interval (t t,+,) with value u.,, and b.(t) similarly, then y. looks like the numerical solution of the equation 1 = s[t, z(t), w.(t)) - b.(t), z(0) = y..o; if we define v.(t) = p.y. as the solution of this equation, then we are asking y. and v.(t) to be close

48

SEC. 3. 3

EXAMPLES OF DISCRETIZATION

in some sense uniformly in u, and b,. Even then, one needs to know that Y(t) and U(t) are continuous enough that v,(t,) near Y(t) for all i will imply the nearness for all t, and similarly for w,, U. Finally, to conclude satisfying h,(V,) and g(p,V,) - g,(V,) to tend to Assumption A5, we need h(p,V,) zero. We give some conditions under which Assumption A5 is valid via this approach. For any set T and positive number c, let N(T, e) = [z; d(z, 7) < e}. We.shall say a set function T(t) is continuous on 0 < t < 1 if and only if for e > 0 there exists b > 0 such that I t' - t" I < 6 implies T(t') c N[T(t"), e]. ASSUMPTION A6. Assume that Y(t) and U(t) are continuous set functions.

ASSUMPTION A7. Suppose that for each w E LZ(0, 1) with w(t) E U(t) almost everywhere and z(0) E Y there exists a unique solution of i(t) _

s[t, z(t), w(t)], z(O) = 0 for almost all t E [0, 1], and that the set of such solutions z(t) is bounded uniformly in such w and zo. ASSUMPTION AS. Assume there exists a function q(t, y) continuous in (t, y) for (t, y) in [0, 1] x (- oo, cc) and such that, if 1(t, y, u) is either of the functions s(t, y, u) or c(t, y, u), we have I 1(t', y', u)

- 1(t", y", u) I S I q(t', y') -

q(t",

y") I

for all u E U* _ fu; u E U(t) for some t E [0, 1]).

Remark: If both Y(t) and U(t) are of the form Y(t) _ [y; m(t) S y S M(t)) for continuous m, M, then they are continuous set functions. If U* and Yare compact, if s(t, y, u) is Lipschitz-continuous in y uniformly in

(t, u) E [0, 1] x U*, and if I s(t, y, u) I < p(t) a(l y 1) for u E U* where µ(t) is integrable on [0, 1] and a(I y ) = O(1 y ) as Iy I -- oo, then Assumption A7 is valid [Roxin (1962)].

EXERCISE. Prove that a set function of the form Y(t) = [y; m(t) < y < M(t)} is continuous if m and M are continuous real-valued functions.

THEOREM 3.3.2. Under Assumptions A6, A7, A8, the mapping p, described above satisfies Assumption A5.

Sketch of proof: Letting (v w,,) = p,(y u,) as described above, it is easy to show that I v,(t,) - y,,, I = o(1) uniformly in i as k the difference equation for y,,,, the equation

v,(t,,.,) = v,(t,) +

0 by examining

s[t, v,(t), u,.,] dt

and using the continuity assumptions on s(t, y, u). Since d, > d[y,,,, Y(t)] tends to zero, we have d[v,(t,), Y(t,)] = o(l) + O(d,) which, by the continuity

EXAMPLES OF DISCRETIZATION

sEc. 3. 3

49

of Y(t), yields d[a,(t), Y(t)] = o(1). Similarly, we find d[w,(t), U(t)] = o(1). Writing

h(v., w.) - h.(y,,, u.) = E

j

(c[t, v.(t), u..,] - c[t,, y..,, u,.,]) dt

and using the continuity property of c(t, y, u), we find that

Jim I h(v w,) - h,(y u,) I = 0 and similarly for g - g,,. Q.E.D. EXERCISE. Supply the details in the Proof of Theorem 3.3.2 above.

We remark that the estimates "o(l)" above are satisfactory for p,, since we have no need for the actual bounds; dealing with r,,, however, we must have computable numbers d,. Consider the definition now of an operator r,,, to be applied to x* = (y*, u*), the solution of the C-problem. Thus y* satisfies y*(t) = s(t, y*(t), u*(t)]

almost everywhere. Suppose for the moment we can define u, - r,u* via u,,, = u*(t,). Then y, = r, y* can be defined via

y.,,+, = y.,, + ks(t,, y,,,, u,,,) for 0 < i < n - 1, y,,, = y*(0) that is, so that y, is a numerical solution of the differential equation for y*; under suitable hypotheses we can then bound y,,, - y*(t,). If u* is only

measurable, we cannot estimate d but can only show that, for certain problems, there exist satisfactory d using the techniques of Cullum (1969) as sketched in the first paragraph of this section. To derive computable d,, we need more continuity assumptions on u*(t). Using these hypotheses we can

bound y; , - y*(t,) and hence bound d while Assumption A8 is more than sufficient to guarantee lim I h,(x,) - h(x*) I= 0. ASSUMPTION A9. Assume that s(t, y, u) is Lipschitz-continuous with respect to y uniformly in (t, u) E [0, 1) x U* and continuous in (t, y, u) e [0, 1] x (-oo, oo) x U*. Assume u*(t) is piecewise continuous, having only finitely many discontinuities, each of finite-jump type. ASSUMPTION AlO. Assume in addition to Assumption A9 that At, y, u) is continuously differentiable with respect tot and u, and that u*(t) is piecewise

continuously differentiable, both u* and 9* having only finitely many discontinuities, each of finite jump type. THEOREM 3.3.3. Under Assumptions A8 and A9, r, as described above

satisfies Assumption A4, with d, = O[k + m(k)], where w(k) - sup Is[t',

50

EXAMPLES OF DISCRETIZATION

ssc. 3. 3

y, u*(t')] - s[t", y, u*(t")] with the supremum taken over all t', t" with 0 < t' < t" < 1, 1 t' - t" I < k, t' and t" in the same-interval of continuity of u*, and y in a certain bounded set R. If Assumption A10 holds, then w(k) = 0(k), and we may take the computable value dA = k' e > 0. Sketch of proof: The only real task is to bound l y.,, --- y*(t,) I. Were it not for the discontinuities in the equation, we could immediately write that ly.., - y*(t,) I = 0[k + co(k)] uniformly in i by the standard theory in Henrici (1962); it is trivial to generalize this to allow the discontinuities. Essentially the argument is as follows. Up to the first discontinuity r, the 0[k + co(k)] result is valid. One can consider the calculation between T, and the next discontinuity T, as the solution of a new initial-value problem in which the initial data used in the numerical method-that is, y*(t,) for the last t, < T,are inaccurate of order 0[k + co(k)]. Since the initial error propagates in a bounded fashion, the error on is also 0[k + co(k)]. The argument proceeds in this manner throughout the finitely many discontinuities Ti. Q.E.D. EXERCISE. Supply the details for the Proof to Theorem 3.3.3, above.

The reader should note that we have only partly attained our goal of finding computable constants d,,. Our estimates-saying that we may take d = 0(k'-,), for example-only mean that the numerical method will thus work for sufficiently small k; we do not have a computable expression for d guaranteed to work for all k. Although one would like to be able to prove convergence of the numerical

solutions without the continuity requirements in Assumptions A9 and A10, this does not seem possible in general (for a special case, see Daniel [1970]); however, very broad classes of problems do have solutions satisfying A10, and one might even call this a typical situation. Thus the assumptions in

A10 do not appear to be unreasonably strong. As a simple special case, the optimal-time problem for y = Ay + Bu,, with A and B constant, with y(0) = yo given, and with u restricted by I I u II,. S 1, can be treated by making use of the classical theory of optimal-time processes; and it can be shown that, if a solution exists, it will be approximated by approximate solutions of the discretized problem with k' expanded constraint sets, extending slightly a result in Krasovskii (1957). More generally, under Assumptions Al-A10, we have proved that approximate solutions to a penalty-function form of the P-problem have weak limit points solving the C-problem.

Another approach for defining the mapping r without assuming the control u*(t) to be piecewise continuous is as follows (only the outline of the procedure is given). Suppose that, for each e > 0, u*(t) can be approximated

by a continuous function u,(t) "nearly" satisfying the constraints-say,

EXAMPLES OF DISCRETIZATION

SEC. 3. 3

d[u,(t), U(t)] < bl(f) with bl(f)

51

0 as e , 0; and suppose that y,(t),

defined as the solution to y, = s(t, y u,), y,(0) = y*(0), is also "near" constraints-say, d[y,(t), Y(t)] sb2(e), d[y,(1), Yp] < b2(E), with b2(e) ---i 0 as f ---' 0-and "near" y* so that the

I h(y*, u*) - h(y,, u,) 1 < 63(E)

with

b,(E) - . 0 as c --+ 0

Pick n so large that the oscillation of u, over intervals of length kis less than

c, and define u, as the piecewise constant interpolant of u, at the points 0, k, 2k.... and y, as the solution to y,,. = s(t, y,,,,, u,,.), v,,.(0) = T*(0). Again we can argue that y,,,, and u, are "near" the constraint sets and I h(y*, u*) - h(y,,., u,,.) I < b,(E). For each n, let (z., w.) be the solution (assuming it exists) of the original C-problem only with the control restricted

to be constant on each interval

[t. t

)

and define (y u) = r (y*

u*) via u.,, = w.(tr), y.,r+1 = y.,, + ks[t ,Y.,r, w.(tr)], Y.,0 = z.(0). If, for example, s(t, y, u) is (uniformly) Lipschitzian in y and t, then it is simple to see that l y.,, - z.(t,) I 0(k) uniformly in n and i. EXERCISE. Prove that l y.,, - z.(t) I = 0(k) uniformly in n and i if s(t, y, u) is uniformly Lipschitzian in y and t, as asserted in the preceding paragraph.

From this estimate for y. - z. one can conclude that I h(z., w.) - h.(y., u.) - ) 0

Because of the minimal property of (z., w.) and the fact that (y,, u,,.) is "near" the constraint set, one can conclude that h(z., w.) < h(y,,., u,,.) + bs(E)

Therefore, we can write h(y*, u*) < h(z., w.) < h(y,,., u,,.) + b,(f). Since h(y*, u*) __ h(y,,., u,,.) I < b,(e) and

I h(z., w.) - h. [r.(y*, u*)] I

)0

we conclude that I h(y*, u*) - h.[r.(y*, u*)] 1 --, 0. Thus Assumption A4 is satisfied for this r. and d. can be taken to be k' for any fixed f > 0. EXERCISE. Consider the simpler C-problem in which Y, = [y0], Yp = Y(t) = (-oo, oo), u(t) _ [-a, a] for some fixed a. Provide the detailed and precise hypotheses and arguments for the above construction of r.. [For the solution of this problem, see Budak et al. (1968-69)].

52

sEC. 3.4

EXAMPLES OF DISCRETIZATION

3.4. CHEBYSHEV SOLUTION OF DIFFERENTIAL EQUATIONS A

We wish to consider at this point a problem which can be examined best from the discretization viewpoint, although the theorems of Chapter 2 are not directly applicable. An attempt to apply the concepts of that chapter, however, will reveal the fundamental difficulties and research areas in the particular problem. This will show, as stated in Section 3.1, how the abstract discretization can be useful in guiding one's research. Suppose one seeks to solve Au = b where A is a uniformly elliptic linear (for simplicity here only) differential operator in two variables over a bounded

domain D, under the condition u = 0 on r, the boundary of D, assumed to be sufficiently smooth; more general types of equations may also be treated by the method to, be presented. A numerical method of recent popularity [Krabs (1963), Rosen (1968)], given a sequence of functions (qrr} satisfying the boundary data, consists in choosing numbers aw,,, . . . , aw,w to minimize II ImaxM

A

(r aw,# )] (x,) - b(x,)

where the M points (x,) form a "grid" over D. Strictly for convenience we take M = cn for fixed c (experience indicates that c = 4 is a good choice [Rosen (1968)]) and suppose that the grid is such that any point in D is at a distance of at most hw from a grid point xi. We wish to find conditions under which the miminizing point u,* a.,r#, will converge, in some sense, to the solution u* of our problem. Since we seek to minimize a supremum norm, the norm must be defined; therefore let

E = (u; u = 0 on r, all partial derivatives of u through second-order are continuous on b = D u r)

For u E E, let 1 1u1 1 = II u I I,, = max I u(x) I. Let zED

f(u) = II Au - b I I where we now need to assume that b is continuous and bounded on D. Let . E w be that subset of E spanned by the functions 9,, ... , qw, assumed to lie in E; let pw be the identity mapping, and rw be at the moment undefined. Define f1(uw) = I I Auw - b I I

m = max I [Auw] (xr) - b(x;) I 1 SrScw

SEC. 3. 4

EXAMPLES OF DISCRETIZATION

53

We now seek conditions for consistency. Consider condition 2 of Definition 2.2.2:

f(paua) -fa(un) =11 Au. - b 110. - 11,4u.* - b Ilta,w

Since this quantity is always nonnegative, the requirement lim sup [f(pau?) - fa(u: )] < 0

in fact demands convergence; in order to compare suprema over discrete and continuous sets, we need to know something about the growth of the functions Aug' - b between grid points. Hence we now assume that Ac, satisfies a Lipschitz condition with Lipschitz constant A, (this restricts A somewhat also) and that b satisfies one with a constant Ao. From this it follows that

If(pau.) -fa(u.)I 1 is rather "weaker." For a thorough analysis of the calculus of variations in the reader is referred to Morrey (1966); relevant approximation concepts are in DiGuglielmo (1969). Consider the problem of minimizing the functional

f(x) = J o g(t, x, x) dt

subject to

x(0) = x(1) = 0

where x = dxldt. The following simple case of a general numerical method has been suggested [Greenspan (1967)]: minimize (or nearly minimize)

56

SEC. 3.5

EXAMPLES OF DISCRETIZATION

subject to

hig(tr- , x".,

f"(x")

X.,0 =

0,

h, = t, - t,-,

where the minimization is over the set of values of x" ,, ... , x,, ,,_, ; this method can be fitted neatly into the theory of Theorem 2.3.1. In Greenspan (1967), under the assumption that there exist unique minimizing points x* for f (in C' [0, 1]) and xA for f" satisfying the spike condition-

for some constant A independent of n-it was purportedly proved that p"x,*, the piecewise linear interpolation to x;, converges uniformly to x*; because the author inadvertently left out an assumption guaranteeing a lower semicontinuity property for the functional f, the proof is in fact incorrect. However, as we shall-show below by use of Theorem 2.3.1, the usual assumptions guaranteeing a unique minimizing point for f, in conjunction with an assump-

tion guaranteeing the satisfaction of a type of spike condition, yield a convergence proof.

For convenience, let us take h, = h = 1/n for all i. For a fixed p > 1, let

E = (x; x(O) = x(1) = 0, x is absolutely continuous on [0, 1], x E L,[0, l]}

For x E E, let 11X11= Ilxil,

For each n, let E. be (n the norm Ilx.II

k(')1 ' dt}'i°

1)-dimensional Euclidean space where X. E E. has

={hE[Ix".(-x

h

I11I

where x,,,0 = x,," = 0 by definition. Let p" be the mapping defined by piecewise linear joining of the values x",, at t, = ih, so p"x" E E. Define the mapping =x(t,),i = 1,...,n - 1. We now make the standard type of assumption in the calculus of variations [Akhiezer (1962)] in order to guarantee the existence of a minimizing point for f. Note that E, as a closed linear subspace of W 1(0, 1), is reflexive, and that weak convergence in E implies uniform convergence-that is, convergence in C[0, l]-as noted in Section 1.3.

SEC. 3. 5

EXAMPLES OF DISCRETIZATION

S7

ASSUMPTIONS: Al. g(t, x, w) is jointly continuous in its variables for

0 S t < 1 and - oo < x, w < 00. A2. There exist constants a, b with b > 0 such that g(t, x, w) > a + b l`w I' for all tin [0,1], x finite. A3. g iF differentiably convex in w; i.e.,

g(t, x, wI) - g(t, x, w2) > (WI - w:)g.(t, x, w2) with g continuous in x, uniformly for (t, w) bounded. PROPOSITION 3.5.1. The functional f is weakly sequentially lower semicontinuous on E, bounded below, and satisfies a T-condition. Proof: For the last two assertions in this proposition, note that

f(x) = f Ig(t,x,2)dt> fa[a+blzlljdt = a+bllxll° The proof of the weak sequential lower semicontinuity is straightforward using the convexity of g; details may be found in Akhiezer (1962), pp. 137-139. Q.E.D.

THEOREM 3.5.1. The discretization scheme defined above is stable and

satisfies a uniform-growth condition. Proof:

l

II° = f o ! (p"x")' 11 dt dt

=Ji

h

= Ilx"Ila

proving stability. For the growth condition, " f"(x")=h±g

x .!,I -- x",,_ I h

)>h!Ir, Ia + b I x" , hx"

= a + bllx"II: Q. Q.E.D.

The only remaining ingredient for application of Theorem 2.3.1 is the Consistency; in Greenspan (1967), the spike condition was needed for this. In our case, we must make the following assumptions.

58

EXAMPLES OF DISCRETIZATION

SEC. 3. 5

ASSUMPTIONS: A4. Some solution x* minimizing f(x) lies in C1[0, 1];

i.e., z* is continuous. A5. There exist constants c and d and a continuous function s(t, v) such that 19(t I Iv., z) - g(12, v2, z) I < (c + dI z 1P) I s(t., v) - s(t2, V2)1

where t, t2 are aTibtrary points in [0, 1] and v v2, z are arbitrary real numbers.

Remarks. If

g(t, x, w) _ (w2/2) + r(t, x) then Assumption A5 is satisfied with s = r. If

g(t, x, w) = l(w)m(t, x) with I l(w)I < c + dl wI

then Assumption AS is satisfied with r = m; many actual problems are of the above types. Assumption A4 is probably superfluous in many cases. THEOREM 3.5.2. The discretization described above is consistent.

Proof: For condition 1 of Definition 2.2.2 we prove lim A- I f,(rrx*) - f(x*) I = 0

Since, by assumption, x* is in C'[0, 1], given e, for sufficiently large n, I x*(t,_,) - x*(i) I < e

and

*

z*(t) - xf h xr-. < e

for tr_, < t 0, there exists N such that n > N implies hx.,I- I

)

Since c > 0 is arbitrary, condition 2 follows. Q.E.D. EXERCISE. Show that I x,,, the Proof of Theorem 3.5.2.

is bounded independently of n, i as asserted

We now can state the following theorem which follows immediately from Theorem 2.3.1 and the above theorems.

THEOREM 3.5.3. Let Assumptions Al - A5 be valid and let the discretization method described above be used. Then all weak limit points of at least one of which exists, minimize f. If the solution x* is unique, then, in particular, p.x,* converges uniformly to x* and the derivatives converge L; weakly. EXAMPLE [Greenspan (1965)]. Consider minimizing

f' I xI (1 + *,)'n dt,

subject to

x(0) = 1, x(1) = cosh 1

having solution x(t) = cosh I. Using h - 0.2, a maximum error of 0.046 is found, while for h = 0.01 the error is 0.0015. Q

For similar results, see Simpson (1968, 1969). 3.6. TWO-POINT BOUNDARY-VALUE PROBLEMS

The problem discussed in the previous section is of course essentially a two-point boundary-value problem for a second-order ordinary differential

60

EXAMPLES OF DISCRETIZATION

sEC. 3. 6

equation; the method described is only one of many possible for use on this problem. Another recent method of great interest is the application of the Ritz method to this problem, using certain special classes of functions as basis functions. In Section 3.7 we shall examine the general Ritz method, but in this section we wish to look at the more special problem indicated above. For clarity we shall consider only simple boundary conditions, ahhough more complex ones can be treated [Ciarlet et al. (1968a, b)]. The method has been thoroughly analyzed [Ciarlet (1966), Ciarlet et al. (1967)] for solving J=O

(-1)1"D'[q,{t)D'x(t)l = g[t, x(t)], t c- (0, 1)

Dkx(0) = Dkx(1) = 0, k = 0, I,

... , n - I

where Dy = dy/dt. The results in this general case, if q (t) Z E > 0, are more complicated to state, but just the same as those for the equation D2x(t) = g[t, x(t)], t E (0, 1) (3.6.1)

X(0) = x(1) = 0

that is, for n = 1, q0(t) = 0, q,(t) - 1; therefore, we shall present only this simpler but sufficiently representative problem.

Let the Hilbert space E = o WZ = [x; x is absolutely continuous, x E L2(0, 1), x(O) = x(l) = 0), and, for x, y E E, define <x, y> =f ` Dx(t) Dy(t) dt 0

Assume that g(t, x) is continuous in (t, x) in [0, 1) x (-oo, oo), and satisfies

(1) g(t,xx_g(:,y)}Y>-n2 (2)

if x#Y

g(t, z_y(t,Y),oo

As a further example, the four-point Gaussian scheme, with ko = 8, is compatible with H2(PN), so that the error using this subspace and quadrature formula, if x* E C4[0, 1], is of order I PN 13. As a concrete example, consider D2x(t) = e"('",

x(0) = x(l) = 0

to be solved using HI(PN), PM: t; = ih, h =1/(N + 1), and the compatible four-point Gaussian integration scheme. The errors 11x* are:

3.13 x 10-' at h=-, 4.40 x 10-6 at

-

and 7.15 x 10-7 at h=g

[Ciarlet et al. (1967), Herbold (1968)]. In this special case, one can show that Ilx* - 0;,211. G Kh7"2 [Perrin-Price-Varga (1969)]. General references: Herbold (1968), Simpson (1968). 3.7. THE RITZ METHOD

The method discussed in the previous section was simply a special case of the general Ritz method. Suppose we wish to minimize a functional f(x) over a Hilbert space E. Suppose that for each n we have a finite dimensional subspace E. of E with the property that for each x in E (actually, we need this only for the point x* minimizing f),

lim d(x, E.) = 0 n--

66

SEC. 3.7

EXAMPLES OF DISCRETIZATION

where d(x, En) = min lax - x,II. We can write "min" rather than "inf" x.E E. above since in a Hilbert space the distance to a closed linear subspace is always attained, and uniquely so. For each n, we then find x.* minimizing f over E. and hope that x,* converges to x* in some sense. We can describe this easily as a discretization method. For each n let

fn - f and let p be the identity mapping; let r. be the best approximation mapping-that is, x - rnx I I = min H H x - xn 1I. If f is norm-upper semix.E E.

continuous, then this discretization is consistent. Employing the conditions of Definition 2.2.2 to check this, for condition 1 we need lim sup f(x*), which follows directly from the norm-upper semicontinuity, the definition of r,,, and the assumption that d(x*, En) approaches zero. For condition 2, we have fn(x,*) f(x,*) -- f(x.) = 0; the other conditions are irrelevant here. Thus by a simple modification in Theorem 2.3.1, we have the following; essentially the same theorem is valid for minimization over a set C, when the Ritz problem is then solved over C r) E.. PROPOSITION 3.7.1. Let f be a weakly sequentially lower semicontinuous,

norm-upper semicontinuous functional on the Hilbert space E and let f satisfy a T-prcperty. Let {E.) be a sequence of closed linear subspaces such that lim d(x*, En) = 0, and for each n let x,* satisfy

f(x.) G f(xn) -}- e for all xn E E. with Jim c. = 0. Then all weak limit points of x:, at least one of which exists, minimize f over E, and lim f(x,*) =f(x*). EXERCISE. Give a rigorous Proof for Proposition 3.7.1.

As mentioned before and shown in a previous section, if f satisfies some

type of a uniform-convexity assumption, then x* is unique and x; -. x*. As also shown in a previous sectio}t, in practice one cannot compute f precisely but must use some approxitn. tion to it; for the particular example of the preceding section-namely. two-point boundary-value problemswe saw that this still cotiki lead to satisfactory results under suitable hypotheses. Clearly we could consider this problem in greater generality via the discretization viewpoint; we prefer to leave this as an exercise and look briefly instead at some known results [Mikhlin-Smolitskiy (1967)] in this direction for the special case in which f is a quadratic functional. Suppose we seek to solve the equation

Ax = k where A is a bounded, positive-definite, self-adjoint linear operator in a

EXAMPLES OF DISCRETIZATION

SEC. 3.7

67

Hilbert space E; this equation has a unique solution, x*, which clearly must also be the unique point minimizing the functional

f(x)

- <x, k>

over E. Since f = A for all x, f is convex and in fact X

fix + h) = f(x) + + I f(x) + + 4mII h 112

where 0 < mI < A and Vf(x) = Ax - k. Thus if lim f(x,,) = f(x*), we have IIx,,-x*II2

f(x.)-f(x*)>+ 2

x*. Thus if we use the Ritz method on this f, we find that

which implies x,,

x,* - x*. We suppose that, for the Ritz method, E. is, for each n, the linear subspace spanned by V . . . , l). where (q; ...} is a complete basis for E-that E is, a set of linearly independent elements with d(x, E. is precisely equivalent to solving

f

k,,

where k E IR", k _ (, , matrix, A. = ((A, ,1)),

k>)T and A. is an n x n

A,,, = 0, M, < 00. Suppose the complete basis (c ...) satisfies

i=j

I

if

0

if i

J,

i,JZI

Then the basis is strongly minimal and moreover the condition number of A is bounded by M, /m, and thus that of A' is uniformly bounded for I F.1. and 16.1. sufficiently small. In particular, these statements hold if f 9p, ...) is a complete orthonormal basis in E.

SEC. 3.7

EXAMPLES OF DISCRETIZATION

69

The Ritz method has been very popular for use in solving certain differential equations of physical interest, and much theory has been developed in this area. For more details and excellent examples the reader is referred to Mikhlin-Smolitskiy (1967). A special feature of the finite dimensional subspaces E., namely that E. is the span of Vl, . , p., allowed us to derive the special results above; often, however, one does not have such expanding subspaces. For example, in ER', if E. is the set of piecewise linear functions on [0, 11 with nodes at i/n, i = 1, 2, . . ., n - 1, we have no such expanding basis; another feature in this case makes analysis easy as we shall now see. More generally, in ER', let P(x)

be a function with compact support and define

for x in RR' and 1-integers j; if we let E. (for h = I/n) be the space of linear combinations of these functions, we have the finite-element method. Some steps have been taken to analyze this very general method [Fix-Strang (1969)]. For example, on the sample problem

-ux'xi - uxu. + u =f(xl, x2) in IR2, the relevant square matrices important in the Ritz method (finiteelement method) have uniformly bounded inverses (in the 1Z norm) if and only if there is no 0a in IR2 such that 0(2 rj + 0o) = 0 for all 2-integers j, where 0() is the Fourier transform

f

dx

Moreover, the resulting numerical method is convergent if and only if for some

integer p > 1, 0(0) $ 0 but 0 has zeros of order at least p + I at

= 2a j

for all other 2-integers j. More widely applicable theory is under development. EXERCISE. In 1R' rather than {R2, find some functions r having Fourier transforms satisfying the above necessary-and-sufficient condition for convergence.

4

GENERAL BANACH-SPACE METHODS OF GRADIENT TYPE

4.1. INTRODUCTION

We now wish to consider iterative methods for minimizing a functional f in some real Banach space E; primarily, we shall be concerned with the unconstrained problem-that is, minimization over all of E-but we shall also briefly consider methods for the constrained problem when they are natural extensions of earlier methods. If f is differentiable, then from the formula dt

f(x + tp)



we see that f is instantaneously decreasing most rapidly in the direction p (that is, with I I p II = 1) if

° -11 Vf(x) II

In a Hilbert space this gives

Vfx)

p II

f(x) Ti

the direction of steepest descent [Cauchy (1847), Curry (1944)]. More generally [Altman (1966a, b)], we consider a steepest-descent direction to be any direction p E E, 1 1 p1 1 =1, such that = -11 Vf(x)11. If the unit sphere in E is strictly convex-that is, if II x II = I I Y II = 1 and 0 < A < I imply

ll Ax + (1 - A)Y II < 1-then such a direction p is unique. The function f of course instantaneously decreases in any direction p

satisfying < 0. We shall consider iterative methods which, at each point x in the iterative sequence, provide such a direction p 70

GENERAL BANACH-SPACE METHODS OF GRADIENT TYPE

SEC. 4.2

71

along which we move to the next point x.., = x + t.p where the distance t of movement must be expeditiously chosen. We must be sure that the directions do not become nearly orthogonal to Vf(x) too rapidly. DEFINITION 4.1.1. A sequence of vectors

if and only if < 0, and II P.(x.) 11 P.(X.)71

Vf(x.))

will be called admissible II --. 0 whenever

-- 0

For example a sequence of steepest-descent directions is admissible, where B: E* --' E satisfies as is a sequence generated by p.(x.) _ > m II c 112, m > 0, for all c E E*. We shall be able to analyze the iteration x + t.p.(x.) for admissible direction sequences (p.(x.)) and various methods of choosing t > 0. 4.2. CRITICIZING SEQUENCES AND CONVERGENCE IN GENERAL

Throughout this chapter we shall denote by W(xo) the following set:

W(xo) = the intersection of all norm-closed convex sets containing L(xo) - (x; f(x) < f(xo)} as a subset Thus W(xo) is the closed convex hull of the level set L(x,,). The problem of minimizing f over E can of course be reduced to that over W(x,,), which we shall often assume is bounded. We shall always assume that f is bounded below, so that we can speak of trying to minimize f. The goal of our analysis of each method will be to compute a sequence (x.} such that (f(x.)} is decreasing, hopefully toward the infimum off. Generally we shall discover that, for some 6 > 0, we have

f(x.) -

f(x.+ l) ?

-SI l p.(x.) I I-1

If f is bounded below, then f(x.) - f(x.+,) and hence

0 with II x111 - x'J II > E if r j; for large enough n, x must lie in some one of the spheres of radius e/3 0, this implies in fact about the x"', i = 1, 2, ... , N. Since JJ x.11 that all the x must be in some one fixed sphere, since to jump to another x. J J > e/3 which is never true for large n. Although we shall improve this result later, we have proved the following theorem. one requires I I

THEOREM 4.2.3. If Vf(x) is norm-continuous in x, if Vf(x) = 0 has only finitely many solutions in W(xo), and if [xn} is a criticizing sequence with JJ

x JJ --+ 0, then

either has no norm-limit points or x, -. x*

with Vf(x*) = 0. We do not wish to give the impression that the only way to treat minimization is from the criticizing-sequence point of view; other approaches also

can be taken. For example [Yakovlev (1965)], suppose the directions p are generated via p = where for each n, H. is a bounded, positivedefinite self-adjoint linear operator from the Hilbert space E into itself. Suppose

0 < a for all x, y in E, and suppose we take

0 < E, < t

X.+1 = X. + taPs'

A

- EZ

Then f of course is uniquely minimized at a point x* and one can prove that x, -- p x*. Arguing much as we shall in Section 4.6, one can show that f(x,,

1) -Ax.)


< 0 for 0 < z < t}

I- = SUP {t; KVf(x. + ?p.), A,> -

We assume 0 < at. < a < 1. Let

x+

then (x") is a criticizing

sequence and

f(x.) -f(x..,,) >_ s(cy,.)(l - c)y,. for all c E (0, 1 - (x) with Vf (x")'

I I P I I\

Proof. In any determination of t", clearly - a. = 0 and

dt [f(x. + tP.) - a.t ] G 0

for 0 < t < t", implying f(x. + t. P.) - a.t. (Vf(x"), P.> d(y.)s(c J(1 ,,

c)Y.

for all cin(0,1-a). Proof.- From the proof of the preceding theorem we know that t;, 11 p, I I

s(cy,) for all c in (0, 1 - a); therefore, t;, I I p. I I >_ t, I I p. I I >_ d(YJs(cy.)

II p, I I - d(y,)s(cy,) yields a criticizing sequence by part A of Theorem 4.2.4 with c,(t) - d(t)s(ct) and c2(t) - ct. Since Therefore,

dt l f (x, + tp,) - a,t ] < 0 for 0 < t < t, then

f(x.+J - a.t. < - < -a2 Then {x.} is a criticizing sequence and I I x.+, - x. I

0.

Proof: By Proposition 1.5.1 we have -T. > T.b,

Thus tf(x.)} is decreasing and hence convergent and T.S, tends to zero. If infinitely often we have

- > e > 0 then it cannot occur under condition l since then we have

f(x,)

-f(x.+1) >_ 612 >_ 6"F1

in contradiction to the boundedness below off. Under condition 2, however, we have

E(1 - b2) < (1 - a2) S the right-hand side of which tends to zero since Vf is uniformly continuous and since T. must tend to zero if does not. This gives a contradiction, leading us to conclude that and hence I I Vf(x.) II tends to zero. Finally, Ii x.+1 - x.lI = II T.P.II = I T.I _< I I

0

Q.E.D.

General references: Altman (1966a), Elkin (1968), Goldstein (1964b,

1965, 1966, 1967), Levitin-PoIjak (1966a).

84

GENERAL BANACH-SPACE METHODS OF GRADIENT TYPE

SEC. 4.6

4.8. A RANGE FUNCTION ALONG THE LINE

We shall now describe another way of selecting IN by making use of a function g(x, t, p) which will determine the range of values t can assume.

The method is similar to that in Theorem 4.5.3 except that a different measure of the distance to be moved is used. The main idea is to pick t to guarantee that the decrease in f dominates

as discussed in Section 4.2. We shall determine admissible values of tM in terms of the range function

g(x, t, P) = (x) - f(x + zP)

-t< f(x),P>

which is continuous at t = 0 if we define g(x, 0, p) = I. We shall assume that an admissible sequence of directions pp is given satisfying II p II = 1. Given a number 6 satisfying 0 < 6 < # and a forcing function d satisfying d(t) < 8t,

we shall attempt to move from x to x,+, = xn + and x,-= x. + p. we find g(x., then we set and also

t.

as follows: if, for

d( !At)

for all t

if we have g(x,,, 1,

where z = A,d() where A. = I if t = 1 and A. = s(d()) if t :p6 1, where s is the reverse modulus of continuity of Vf. Proof: By Equation 4.6.1, f(x,) is decreasing and

f(x.) -f(x.+,) > t.d(< - Vf(x.), p.>)

(4.6.3)

If t = I does not satisfy Equation 4.6.1, then t, E (0, 1). For these n, we write

f(x.+,) -f(x.) _ for some A. E (0, 1) Thus, from Equation 4.6.2,

d() < g(x., t., P.) -II x.), P. x.), P.

d() and hence

t. = II x.+, - X. II > II A.t.P. II > soI Vf(x. + 2.t.P.) - Vf(x.) II] > s[d()] (4.6.4) Hence, using Equation 4.6.3, we conclude that

f(x,) - f(x.+.) >_ d()s[d()] Thus

f(x.) -f(x.+) >_ A.d() as asserted, which implies, as before, --, 0. Q.E.D.

86

GENERAL BANACH-SPACE METHODS OF GRADIENT TYPE

SEC. 4.6

Computationally one needs a procedure for computing a tM E (0, 1) satisfying Equations 4.6.1 and 4.6.2 if tM = 1 does not satisfy Equation 4.6.1. We consider doing this [Armijo (1966), Elkin (1968)] by successively trying the values

tM = a, a2, a', ... , for some a E (0, 1) THEOREM 4.6.2. Under the hypotheses of Theorem 4.6.1, t may be chosen

as the first of the numbers a°, al, a2, ... satisfying Equation 4.6.1, and then (x,) is a criticizing sequence,

f(xM) -

f(xM+ 1) >_ 1d()

where

A. = 1

if tM = 1, aM = as[(1 - a)]

ift,$1. Proof. As in the previous theorem, tM = 1 yields no problem. in the other case, we have xM+1 = xM + alp,,, j > 1. Let xM = x, + a' 'p,. Then we have

f(x,) - f(xr) < I I xM - xM I I d() f(XM) - f(xM+ t )

II

XM+ I - xM I I d(< - Vf(x,), PM>)

Therefore,

f(x,+,) -

f(x:) < (1 - a) I I xM - xM I I d()

We can write

f(xa+,)

-{y J X.,) =

This leads to > -d()

PM>

Hence II

Vf[2MXM + (I - ~M)xn+I]

>-

- Vf(x,) II



> (1 - a)

(4.6.5)

We then have I I XM+

1

X. I I

a 112M x, + (1 -

1 - XM I I >_ as[(1 -

aK-Vf(x,), PM>]

GENERAL BANACH-SPACE METHODS OF GRADIENT TYPE

SEC. 4.6

87

Therefore, from this and from Equation 4.6.1 we have f(xw)

-f(x.+1) > as[(1 - 6)]d()

0. Q.E.D.

which implies that

In particular, one can consider this algorithm with d(t) = 8t and, instead of Equation 4.6.2, the stronger condition

1-6

g (x,,, t.,

This method has been considered often [Goldstein (1964b, 1965, 1966, 1967)].

The two theorems above can be extended somewhat. For example, rather than demanding, in Theorem 4.6.1, that lip. II = 1, suppose we assume that lip. 11 >_ d,

Vf(x.).

I I P.11

))

for some forcing function d and that

(_Vf(x ) , ..

P.

lip. II

tends to zero whenever

d() IIP.II

tends to zero. EXERCISE. Show that the latter condition immediately above is valid, for example, if 11 p. II is bounded above or d(t) = qt, q # 0.

Looking at the proof of Theorem 4.6.1, we see that under these conditions Equation 4.6.3 with tw = 1 becomes

f(xw) - f(x.+J >_ d() = d((-Vf(x.3,

II P. Pw

fl>

IIP.II)

so that either (-Vf(xw),11P.

II

or

lip. 11 >_

d1\-Vf(x.).

IIp. [I))

must tend to zero, yielding U Vf(xw) II -- 0. For rw E (0, 1), Equation 4.6.4 becomes instead t. I I P. II >_

Sd()

IIP.II

88

GENERAL BANACH-SPACE METHODS OF GRADIENT TYPE

SEC. 4.6

and thus d() d( - Vf(x.), PR )

f(x,) - f(x.+ 1) Z s

IIP.I

IIPRII

which implies that d() II P. II

and thereby II Vf(x,) II tends to zero. Thus we have proved the following corollary. COROLLARY 4.6.1. Theorem 4.6.1 is valid [except for the bound on

f(x,+,)] with the assumption that Iip.Il = I being replaced by I. Ip.II >_

11 d,\(-Vf(x.),

IIP.II/)'

and

2. (- Vf(x,J, I I - \

0 whenever d. ( and thence 1 xw+ 1 - X. 11 >

P.

L(I - b) (- Vf(x,),

lip. 11

»

and

f(x.) - f(x.+l)

acsl (I - O)(_Vf(x.),II l

\

1]

E 11p. 11, c > 0, then II

x. II , 0, since to is bounded; this is true in particular for p _ -Vf(x.), as we saw above.

General references: Altman (1966a), Elkin (1968), Goldstein (1964b, 1965, 1966, 1967). 4.7. SEARCH METHODS ALONG THE LINE

In actual computation it is of course necessary to deal with discrete data; this means, for example, that one cannot generally minimize f(x;, + tpn) over all t > 0 but only over some discrete set of t-values. In this section we shall indicate how, in some cases, we can guarantee convergence for practical, computationally convenient choices of step size. For theoretical analysis, we shall restrict ourselves to strictly unimodal functions-that is, to those that have a unique minimizing point along each

straight line; from Section 1.5 we know that this is equivalent to strong quasi-convexity. EXERCISE. Prove the equivalence of strict unimodality and strong quasiconvexity as asserted above.

This equivalence implies that if we have three t-values t, < t2 < t, such that

f(x + t2 P) < f(x + t, p) and f(x + t2p) < f(x + t,p), then f(x + tp) is minimized at a value of t between t, and t,. EXERCISE. Prove the preceding assertion concerning the location of the t-value minimizing the strictly unimodal function f(x + tp).

We combine this fact with Theorem 4.4.2 for a. = a = 0 to prove the following.

THEOREM 4.7.1. Let f be strongly quasi-convex and bounded below on W(x,), let Vf be uniformly continuous on W(xa), and let pn = define an admissible direction sequence. Suppose that for each n there are values such that tn, 1 , tn.29 ' ' ' f tn.ko

90

GENERAL BANACH-SPACE METHODS OF GRADIENT TYPE

SEC. 4.7

f(x.) >f(x. + t..tP.) > ... >f(x. + t., k. P.) t..k.->>2 ,

tn. k.+1

and 2. < 1. Thus) Theorem 4.4.2 with d(t) = 2 implies our theorem for t. = t..k._I- Since

f(x. + t..k.P.) 2 is sufficient to guarantee that t. _ (k. - 1)h. or t. = kh. will make (x.) a criticizing sequence. Proof. In this case,

t..k.-I =k. - 1> 1 =2 3 k. + 1

t..k.+I Q.E.D.

COROLLARY 4.7.2 [Cea {1969)]. Under the hypotheses of Theorem 4.7.1, if in addition

f(x. + 2h.p.) > f(x. + h.p.) 0 such that

f(x,, + h.p.) < f(x. + 2h.p.) _ +s(cy,)(1 - c)y. for all c in (0, 1), with

(-Vf(x),II Proof.-.The point t;, providing the global minimum for f(x. + tp,) must satisfy 0 < t; < 2h and of course t = t; would yield a criticizing sequence. Since f is convex, for 0 < t < h we have

f(x. + tp.) > 2f(x. + h.p.) - f(x. + 2h. p.) + fix. + 2h. p.) -f(xx + h. p.)

h

> 2f(x. + h.p.) -f(x. + 2h.p.) > 2f(x. + h. p.) - f(x,) while arguing similarly for h < t G 2h we deduce

f(x. + tp.)

2f(x. + h. p.) -f(x.)

Setting t; = t thus gives

f(x. + t.'p.) > 2f(x. + h.p.) -f(x.)

92

SEC. 4.7

GENERAL BANACH-SPACE METHODS OF GRADIENT TYPE

and therefore

f(x.) -f(x. + h.p.) >.[f(x.) -f(x. + t,p.)] The theorem follows from part B of Theorem 4.2.4 with 8 = }. Q.E.D. We can now give a practical algorithm of a search type to locate a suitable value of We assume that the algorithm is entered with a point x4, direction p,,, and a number h > 0 given. We write in a pseudo-ALGOL language for convenience.

Search routine [Cea (1969)]

if f(x + hp.) < reduce:

h

then go to first;

-T;

if f(x + hp,) > (x. + h pal > f(x4 + hp.) then EXIT FROM ROU-

f

TINE NOW WITH t = h; if f IS CONVEX then EXIT FROM ROUTINE NOW

WITH t. --2 h T;

loop:

while f (x + h

first:

EXIT FROM ROUTINE NOW WITH t = h; if f(x + f(x. + hp.) then go to oldway;

f(x + hp.) do h -

T;

t.- 2h; change: oldway:

while f(x + (t + f(x. + tp.) do t t + h; EXIT FROM ROUTINE NOW WITH t = t; if f IS CONVEX and f(x. + 2hp,J 0 and a2 > 0 be chosen and choose t to satisfy S,

and set xn+, = x - t.Vf(xn). Then x -+ x*, the unique point minimizing f over E, starting at any x0. Given any e > 0 there exists an N such that for

n > N,

-t.M1)

11x*-x.... II 0, then - 0.

98

GENERAL BANACH-SPACE METHODS OF GRADIENT TYPE

SEC. 4.9

Proof: As in Theorem 4.2.4, we easily find f(x.) --.1(x. + t:P.) > tu. II P.11 [Y. - c2(Y.)] > c,(Y.)[Y. - c2(Y.)] where

/-Vf(x.),

-\

Y. =

P. IIP.II

If t = t", we then have -f(x.) - f(x.+,) > cl(Y.)[Y. - c2(Y.)] If t = t',, then t;, I I P. 11 < t.u 11p, 11, and arguing as for t.0 we get

f(x.) - f(x. + t.P.) >- tI[P.II IF. - c2(Y.)] d2(Y.)d1(IIP.IDEV. - c20.)]

Thus y.

0 or

11 P. 11

0; since 11 p.11= I I x. - x.11 and 11 Vf(x.) 11 are

bounded, this gives t.a', which implies t. = t:' and thus t > t;,; in the second case, clearly t > t;,. In either case, by the defining property of t. and the fact that t. > t,', we have

f(xa) - f(xa + tap.) > f(X.) - f(x. + t:P.) + a.(t. - t.X

f(x.) -

P.>

f(x. + to P.)

so that the theorem follows from part 2 of Theorem 4.9.2 with f

1. Q.E.D.

EXERCISE. Fill in the details in the above Proof.

Remark. Setting a - a = 0 yields the usual method. THEOREM 4.9.4. Let f and C be as in Theorem 4.9.3 and let C be normclosed. Let t be either: (1) the smallest positive t providing a local minimum for

f(xa + t.P.) - a t(Vf(x.), P.>

over the set of t such that x + tp E C, t > 0; or (2) the first positive root r of

- aa(Vf(x.),

if x. + rp E C, otherwise t = sup {t; X. + tP. (=- C) or (3) the following:

t = sup {t;

d((-Vf(xn), II

A P.

d,(1) = t, d2(t) = d(t), /3 = 1. Q.E.D. EXERCISE. Fill in the details in the above Proof.

As in the unconstrained case, if Vf is Lipschitz-continuous, while the 0, it is above theorems define a range of t-values leading to possible to double the size of this range by a more careful analysis. THEOREM 4.9.6. Let f be bounded below on C, let Vf satisfy

11Vf(x) - Vf(y)II /' L

for all n. For each n let xn+, = xn + to pA where to is defined via

to = min (1, ynl I I Pn 112

J

Then f (xn) decreases to a limit. If 1I PA 11 is uniformly bounded-for example,

if C is bounded-then lim = 0

If 1lpAI l -0 implies (Vf(x,i),

P. IIPAII

-> 0

then

Jim (Vf(xn), 11 P.

II = 0

SEC. 4.9

GENERAL BANACH-SPACE METHODS OF GRADIENT TYPE

101

Proof.

f(xn+ 1)- f(xn) 0 such that f(xw) - f(x., ) z a' for either r = 1 or r = 2, which implies lim = 0. Since + M II x:-x*11 [_]

1/2

using Equation 4.10.2 and the positive-definiteness of A,,. Thus

0 where f;,' denotes the derivative f', . If x" is chosen to minimize g"(x)

+ + < 0

for convex f, implying that p is a direction of nonincreasing f-values as needed.

THEOREM 4.10.3. Let f be convex, bounded below on the norm-closed,

bounded convex set C, and attain its minimum over C at x*; let f', exist in C and I I f' I I < B > 0 for all x in C. Let [xj be a sequence in C such that lim = 0 where p = xn - x and xn minimizes

gn(x)=

f(x*).

over C. Then

Proof: By the definition of xn as we saw above, we have >_ - <x - x, f.'(`v.' - X.) > - <x - x'., V n(x'n)> -

since 0 and 11 s(x, t) I( S Kt2 for some K. Thus only a small perturbation of the linear motion keeps us in C. Algorithms have been given for determining t-values,

and convergence proofs are known. The methods for computing s(x, t), however, are very complex and do not appear to lend themselves to practical computation; therefore, we consider the method no further. One further type of method for constrained problems which we wish to consider is the penalty function method. We have met this approach before in Sections 3.2 and 3.3 in a more specialized form. In fact, the whole approach fits into the discretization analysis if one makes some extensions in those results, but this adds but little to the general applicability of those theorems; therefore, we treat the penalty-function method briefly in the more classical fashion. We seek to minimizef(x) over C = {x; g(x) < 0), where g is some nonlinear functional. Instead, we shall approximately minimize f(x) + PP[g(x)] over E, where the penalty functions P. are such that, fort > 0, lim PP(t) = 00; n-.«

uniformly for

t > 6 > 0 for all 6 > 0

Thus P. will penalize us for having an x with g(x) > 0. EXERCISE.. Give some examples of penalty functions that satisfy the conditions immediately above.

GENERAL BANACH-SPACE METHODS OF GRADIENT TYPE

sEc. 4.11

111

What we can hope will occur, then, is that our computed sequence xp will satisfy

lim sup g(xp) < 0 p+w

This, however, is not enough in general to guarantee that d(xp, C) = inf I I xp - x l l rEC

is tending to zero. DEFINITION 4.11.1 [Levitin-Poljak (1966a)]. The constraint defined by g

is called correct if lim sup p-.w

0 implies lim d(xp, C) = 0. 11-.w

EXERCISE. Find some explicit conditions under which constraints are correct.

THEOREM 4.11.1. Let g define a correct constraint; for some e > 0 let I f(x) - f(y) I < L II x - yli if d(x, C) < E and d(y, C) < e; let Pp[g(x)] > 0

for all x E E; let lim PP(t) = oo for t > 0, uniformly for t > a > 0 for all 6 > 0; and let lim PP[g(x)] = 0 for all x r= CO, a dense subset of C. Define mp = inf {f(x) + PP[g(x)]}, xEE

m = inf f(x) xEC

and assume inff(x) = rn > - oo. For a sequence ep > 0, satisfy

xEE

Ep ---p

0, let xp E E

f(x,) + PP[g(x,)] < mp + ep Then {xp} is an approximate minimizing sequence for f over C in the sense of Definition 1.6.1.

Proof: Let wp E C, lim f(wp) w, E Co with

m. Since f is continuous, there exists

If(w) -f(wp)I m, sequence bounded below by zero, we write

<xm - xn, N(xm - xn)> = E(xm) -

2<M*Hrn, xm - X.>

Since n > m, xm - x is in B.; since x minimizes E(x) over B., which equals -2M*Hr,,, is orthogoral to B. and therefore to xm - x,,. Thus we conclude that <xm - xn, N(xm - xn)> = E(xm) -

which tends to zero, since {E(x,)} is a Cauchy sequence. Since N is positivex'. definite, {x,} is a Cauchy sequence and there exists x' E B such that x By a continuity argument we find, setting r' = k - Mx', that <M*Hr', z> = 0 for all z E B. Since again VE(x') = -2M*Hr', this implies that x' minimizes E over B. Q.E.D. EXERCISE. Prove that <M *Hr', z> = 0 for all z E B, where r' is defined it the Proof of Theorem 5.2.1 above.

As a practical matter, the minimization is easier if B. is finite-dimensional-if, say, B. is spanned by the linearly independent vectors [po, p ... for all n. Then of course x a,,,jpj. It would be convenient if a., j

were independent of n, so that xn+, = x + a,p,,. It is a simple matter to or

= 0 show that this is the case if and only if either x = for i = 0, 1, . . . , j - 1 [Antosiewicz-Rheinboldt (1962)]. We include the proof in one direction as a part of the following.

116

CONJUGATE-GRADIENT AND SIMILAR METHODS IN HILBERT SPACE

SEC. 5.2

THEOREM 5.2.2. Let {p,); be a sequence of linearly independent vectors

satisfying

= 0 if i x,+ i = X. + c P,W

j and let x0 be arbitrary. Let c,

_
'

r, - k - Mx,

Let B. be spanned bypo, ... , p,-, and let B = closure of U B,. Then x, -- 'x' minimizing E over B.

Proof: For i < n - 1, <M*Hr P+) _ <M*Hr,-t, P,> - c,-, 0, and for any such a, A, we have

a < 4u(p) < I < ,u(Kg,) < A

and a< v(g)
= <M-'(M*H)-'M*Hr M*Hr,>

_ = y(gi) 1

E(x,) - E(xr+,) = c, = E(x,)c,v(g,)

The estimates of the theorem follow from cr > 1/A, Y(g) > a. If K and N commute, then ary(gr)

y(g,)

K)

,u , gr

_-

[Kg,, Kg,]2

[Kg,, TKg,][Kg,, T -'

where Ix, y] - <x, K-' y>. It is easy to see that T is self-adjoint positivedefinite relative to with spectral bounds a, A; thus 4aA

c.v(g) > (A+a)2 by the inequality of Kantorovich [Faddeev-Faddeeva (1963), Kantorovich (1948)]. Now let fi > 0 be the lower spectral bound for N. Then

fill x.-

I12S=E(x.)

SEC. 5.4

CONJUGATE-GRADIENT AND SIMILAR METHODS IN HILBERT SPACE

119

In some cases the method can be shown to converge even when a = 0, but examples are known in whit, i we then have 11x - x* 11 > (inn)-' for some ).. > 0, showing that no geometric convergence rate is possible [Odloleskal (1969), Poljak (1969a)]. General references: Antosiewicz-Rheinboldt (1962), Daniel (1965, 1967b), Hayes (1954), Hestenes (1956), Hestenes-Stiefel (1952). 5.4. CONJUGATE GRADIENTS AS AN OPTIMAL PROCESS

Much-improved bounds on the convergence rate can be obtained by viewing the conjugate-gradient method in a different light, one which shows more clearly the great power of the method-as opposed, say, to the steepestdescent method, which also has a convergence factor like (A - a)/(A + a).

Suppose we seek to solve Mx = k-that is, M*HMx = M*Hk-by

some sort of gradient method; for more generality we allow ourselves to multiply gradients also by an operator K, where M, H, N, K, Tare as defined earlier. If at each step we allow ourselves to make use of all previous information, we are lead to consider iterations of the form

x.+I =

h = M -'k

P JA) are polynomials of degree less than or equal to n. If we should by chance have x0 = h, we would want x = h for all n. This leads, since h should be considered arbitrary, to the requirement that where

xo + P.(T)T(h - x0)

(5.4.1)

where P,.(.t) is a polynomial of degree less than or equal to n. We wish to use methods of spectral analysis to discuss such methods,

so we are forced to assume that

N = p(T) where pa,) is a positive function continuous on some neighborhood of the spectrum of T. As we shall later see, this is satisfied in the practical methods, where usually p(..) ) or p(A) -- 1. For each n, we wish to choose so that E(x..,I) is the least possible under any method of the form of Equation 5.4.h According to the spectral theorem, we can write 1) _ A p(2)[1 - 2PP(1)12 ds(2) J

(5.4.2)

0

where s(.) is a known increasing function. The fact that there is a polynomial PP(2) yielding this least value follows from a straightforward generalization

120

CONJUGATE-GRADIENT AND SIMILAR METHODS IN HILBERT SPACE

SEC. 5.4

[Daniel (1965, 1967b)] of the theorem in finite dimensions as proved in Stiefel (1954, 1955).

is minimized by setting

PROPOSITION 5.4.1. The error measure

1-

to be the (n + 1)st element of the orthogonal [on

[a, A] relative to the weight function .p(A) ds(A)} set of polynomials R,(A) satisfying R,(0) = 1. EXERCISE. Prove Proposition 5.4.1.

We shall now show that, for each n, the vectors generated by the conjugate-gradient method are precisely those generated by this optimal process. THEOREM 5.4.1. For each 'n, the vector x generated by the conjugategradient (CG) method coincides with that generated by the optimal process of the form in Equation 5.4.1. Proof: Given n, the vectors p0, ... , p,-, in the CG method are independent. Since p0 = Kgo and p,+I = Kg,,.., + b,p it is clear that any linear combination of p0, ... , can be written as a linear combination of Kg0, ... , Thus the n vectors Kg0, ... , Kg, span at least the ndimensional space B.- sp[po, . . . , hence B = sp[Kg0, . . . , Kg._,].

Now Kg0 =g°Kg0; assume that for j< i, Kg, can be written as a linear combination of T°Kg0, T'Kg...... T'Kgo. Then Kgr+, = K(g, - c1Np,) = Kg, - c,Tp, We can write p, as a linear combination of Kg0, ... , Kgt, each of which, by

the inductive assumption, is a linear combination of T°Kg0, ... , T'Kgo. Therefore, Kg,+, is a linear combination of T°Kg0, ... , T'*'Kg,,. Reasoning as above, we have

B. = sp[T°Kg0,... ,

T"-'Kg.)

Now x, minimizes E(x) on x0 + B. if x is generated by the CG method. By what we have shown above, this says that the x generated by the CG method minimizes E(x) on the set of points ,.-1

x = x0 + E1-0s1T`Kg0 = x0 + P,,-,(T)T(h - x0) where P,,_,(A) is the (n - 1)st-degree polynomial P.-,(A)

among all iterations of the form

X. I = x0 + P (T)T(h - x°) the CG method makes

the least. Q.E.D.

_'E-,

1-0

s,V'. That is,

SEC. 5.4

CONJUGATE-GRADIENT AND SIMILAR METHODS IN HILBERT SPACE

121

Thus, if we insert any polynomial into Equation 5.4.2, we can get a where x,+, is generated by the conjugate-gradient method, bound for If we choose for comparison since that method gives the least value of as the (n + 1)st Chebyshev polynomial relative to )p()) ds(1) 1on [a, A], we find the following bound.

PROPOSITION 5.4.2. Let a - a/A. Then, for the conjugate-gradient method,

E(x.) S w.E(x0) S 4(1 \1 +

/a

a ) 2"E(xo)

and 11 x. - h 112 converges to zero at this same rate, where

"'. = (1 +

2(1 - ar )2. +

EXERCISE. Prove Proposition 5.4.2.

By this result we have reduced our estimate of the convergence factor from (1 - a)/(1 + a) to at least (1 - /-,% )/(1 _ix ). When one uses the steepest-descent algorithm to solve Mx = k by minimizing E, one moves from xx to x.., in the direction M*Hr.. Therefore, the steepest-descent method has the form of Equation 5.4.1 and, therefore, reduces the error E(x) by less than the conjugate-gradient method for every n. Since the best-known and in

certain cases best possible convergence estimates for steepest descent [Akaike (1959)] are of the form (1 - a)/(1 + a), while we have at least (1 - ^/ T )/(1 + ,/ _a), we see that the convergence of the conjugate-gradient method is also asymptotically better. For clarity, we now state the form that the conjugate-gradient algorithm takes in certain special cases. The iteration takes its simplest form in the case in which the operator M is itself positive-definite and self-adjoint; it was this case for which the method was originally developed. Here we may now take H = M-' and K = I. Thus

N=T=M,

E(x)=

Since N = T, we have p(A) = A, and the analysis of this section applies. The iteration becomes as follows:

Given x0, let po = ro = k - Mxo. For n = 0, 1,.. . , let

=

11r.11'

_

P., MP.>

r.+i = r. - c.MP.,

-

b"

I I r" 112

A second special case which is simple enough for practical use arises from setting H = K = I, so that T = N = M*M. Again, p(2) - A, and we have E(x) = 11 r 112. Fortunately, for computational purposes one can avoid

the actual calculation of M*M and can put the iteration in the following form :

Given x0, let ro = k - Mxo, Po = go = M*ra. For n = 0, 1, C.

_

- 11 MP" I I2'

r.+1 = r" - C. MP",

... , let

x"+ 1 = X. + C.P.

g"+ 1= M r"+ 1

P"+1 = g"+1 + b"P"

where

_<MP Mgr+I>=11 g"+1112

b"

II MP"112

11&11,

A third special case arises from H = (M*M)-', K - M*M, so that N = I, T = M*M, p(2) - 1, E(x) = II h - x112. By some manipulation, the iteration takes the following form:

Given x0, let r,, = k -- Mx0, p0 = M*ro. For n = 0, 1, . c" =

11 r" 112 I

IP P.

rn .1 = r

X" 1

112'

c"MP",

- X.

. .

, let

CnPn

P"+ I_ M*r"+ t

'?

b"P"

where b" . _.

II r"+1 E. Ilr"II2

EXERCISE. Show that the last two algorithms above generate the desired iterates.

General references: Daniel (1965, 1967b), Faddeev-Faddeeva (1963). S.S. THE PROJECTED-GRADIENT VIEWPOINT

It has been widely believed that the CG method exhibits superlinear convergence-that is, that II x" - h II tends to zero faster than any geometric

SEC. 5.5

CONJUGATE-GRADIENT AND SIMILAR METHODS IN HILBERT SPACE

123

sequence An with ) > 0-although the best error estimates in general only yield

,,/A

,/a

!

If we view the method as one of projecting the gradient direction onto the space conjugate to all preceding directions, we obtain an indication that the convergence might in fact be superlinear; the result we obtain in this way is also needed later for the analysis of nonquadratic functionals. For simplicity of notation, we restrict ourselves to the simplest special case of the CG method

with M itself positive-definite and self-adjoins, with N = T == Al, K 1. Without loss of generality, we consider the CG iteration starting with a first guess x0 - 0. Suppose we are given a vector d $ 0 such that 'd, k; = 0. by We define an equivalent inner product [x, y] = <x, My; Then we have [h, d] = 0-that is, h is M-conjugate to d. Let P, be the orthogonal (in the sense of the inner product projection onto the linear

subspace spanned by d, and let P, = I - PF Define the Hilbert space A', = P,. with inner product and define the operator M, = P,M in. ,. EXERCISE. Prove that M, is a bounded, self-adjoint, positive-definite linear operator from .7f', onto . V , and that, therefore, h is the unique solution of the equation

M,x -- k, = P,k Show that the spectral bounds a,, A, of M, are related to those a, A of M by a < a, A, A. Hint: For example, to solve Mix k' for k',,- Al',, let

aM-ld

x, - A4-1k' If

- = 48

soft = 0 and xa = x'.

To solve M,x = k, in .Y° we consider the general form of the CG method obtained by letting

K=M H=M2, sothat N=I,T=M, All the theory of the,CG method applies here, and we can in particular deduce

that E,(x") S w.zE,(xo)

where W. = (XI

2(1-a)" (1 + ax" + (I - -777A,

E,(x) = [h - x, h - x] = A straightforward calculation shows that the iterates x, generated by this general algorithm on M, in .", are precisely the same as the iterates generated by using the standard simple algorithm on M in ,' if the initial direction po

in the simple algorithm is not chosen as ro = k - Mxo = k as usual, but by the formula

Po=P,r0=ro

b_,d,

b_,=-

ro, Md>

that is, by the usual way of generating CG directions if we identify d with p_,. EXERCISE. Prove the assertion in the preceding paragraph.

All that the preceding paragraph says is that the standard CG method, modified to require the first direction po to be conjugate to d, is equivalent to a general CG method in a space M-conjugate to d; therefore, the modification of the standard method converges and, in fact, since

E,(k)=[h-x,h-x)=

E(x)

we have

E(x") < w2E(xo)

More generally, if we have proceeded through standard CG directions

SEC. 5.6

CONJUGATE-GRADIENT AND SIMILAR METHODS IN HILBERT SPACE

125

P09 pI, ... , PL-, to arrive at xL = 0, then the solution his M-conjugate to pi, 0 < i < L - 1, and we can define P,, as the orthogonal projection (in the [ ,

sense) onto the span of [p0, ... , PL-,}, P, = I - PA, .'I = P,.,°,

]

M, = PM. Then the remainder of the standard CG iterates are precisely the same as those generated by the more general CG method applied to M, in A, and, therefore, our convergence estimates can make use of the spectral bounds of M. on A e, rather than of M on A. Since the projections P, are "contracting" as we do this analysis after each new standard CG step, the spectral bounds on the operators M, might be contracting, allowing a proof of superlinear convergence. While we have not been successful in accomplishing this, it seems a worthwhile approach. 5.0. CONJUGATE GRADIENTS FOR GENERAL FUNCTIONALS

We now wish to consider minimizing a general functional f(x) over a

Hilbert space .° by some analogue of the conjugate-gradient method. In this case, Vf(x) plays the role of 2(Mx - k) and fx plays the role of 2M. For notational convenience we shall write J(x) _ Vf(x), P. = f we shall also write r" _ -J(x), J',.. Thus, in analogy to the quadratic problem, given x0, let pa = ro = -J(x0); for n = 0, 1, . . . , let x.+, = x" + c" to be determined; set r.+, = -J(x.+,), and p.+, = b"p", where

_ b"

- r"+ , J.+ i P" P., J.'. P.>

If the sequence of vectors p" that we generate in this manner is admissible,

then all the results of Chapter 4 apply to determine the choice of c,,; we consider the admissibility. If we desire

>allr"II2, then

a>0

precisely what we need is

b.-Xr.,P.-J > -(1 - a)IIr.II2 This follows, for example, if b.-

(I -a)

Ilr"II

IIP.-i Il

for which and

_ <J(x) - J(y), x - y> > a II x -

1I2

as does the error estimate with x = x', y = x,,. Q.E.D. This theorem by itself does not indicate any special value for the method; all of the methods of Chapter 4 behave essentially in this fashion. The advan-

tage of the method for quadratic functions is its rapid convergence rate; we show that, asymptotically, this same rate is obtained in general. 5.7. LOCAL-CONVERGENCE RATES

In examining the local-convergence rate, we discover that estimates can

be found simultaneously for a larger class of methods-namely, without

128

CONJUGATE-GRADIENT AND SIMILAR METHODS IN HILBERT SPACE

SEC. 5.7

choosing b. via the conjugacy requirement. We assume instead that I I b.-, P.-, I I < D 11 r.11 for some D; then I I P.112

1 1r

. 1 1I2 + I I b.-, P.-, III < (1 + D2) II r.112

which yields

r.,.-=11r.11Z 11r.1111P.11

IIP.II

121 (1+D)

so that the p are admissible directions. (This assumption can be weakened via Remark I following Theorem 4.2.4.) If we examine the effect of this change on the Proof of Theorem 5.6.1, we find instead that 1

A(1+D2) f(x.+,) _< f(x.) - A(1

I

D2 II r.li2

so that the conclusions of the theorem follow. Thus we have proved the following. THEoi

i 5.7.1. Let 0 < aI < Jx < Al for x E .-°, J = Df.

Given x0, let po = ro = -J(xp). For n = 0, 1, . . ., let

(5.7.1)

Then x. converges to the unique x* minim;zing f over .*°, and

iix-x*II \ 1 + >7.i

JJ)P., P.>dt

_ - II r. I I2 + C<J,P., P.> -+- c J This gives

-II r. 112 + c<J.'p.,p.> -+c2BIIP.1I' + c2B

I I P II3

On the interval

p0

if

d 0 small enough (independent of x0) and cw and b" determined as described above, it follows that xw - x*. Proof..

r(xw) -

f(xw+,) _ <J(xw+,), -cwPa> + 2 C.NPw, Jxw+rc.P.> C. IIP"112

{

II rw+, Il IPw

dIIP"112[Ta-6

II I1Pwil

ll

c"a}

(A+cw(1+6L)IJ

Because of the lower bound for cw, if 6 is small enough, then f(xw) - f (xw+,) > d, I I Pw 112

+

138

CONJUGATE-GRADIENT AND SIMILAR METHODS IN HILBERT SPACE

SEC. 5.8

for some d, > 0, and hence I I p.11 > (1 - SD) I I r I I tends to zero, implying x*. Q.E.D.

X.

The above theorem is somewhat similar to Theorem 4.5.1. In order to obtain good estimates of the local-convergence' rate, we need to determine c more accurately. According to Lemma 5.7.2, c. is approximately given by I r I J2

I

c - P", .,P">

then the asymptotic convergence rate-that is, for e,, small enough-is described as follows: for every m there exists N;" such that for n > Nm, we have E"+m(x.,+m) < [wn, ± O(e.11(4m-3)) 1. O(e+/(4m-»)]En(x")

where wm is given in Theorem 5.7.3. When J(x) is linear, we know that

b" = II r"+,IIZ IIr"II Since this formula does not involve Jn in any way, it is computationally useful

and has been used in practice for general problems; a computer program can be found in Fletcher-Reeves (1964). If b" satisfies II b"p" II < DII r"+, II, then convergence is guaranteed. by previous theorems; such an inequality

does not appear to be valid in general, however. It can be guaranteed by setting

b"=min{11r"+,IIZ A ,IIr"+,II I1r"IIZ

' a

IIP"II

Another way to compute a b" which is just as convenient from the computa-. tional viewpoint as that above, but more easily analyzed, is via the formula [Poljak (1969a)] IIr"II2

which is a correct formula for quadratics. EXERCISE. Prove that the three determinations of b", namely

11 r"+, III 11r"IIZ

r

Z

and

f,

P

,

are equivalent on quadratics.

140

CONJUGATE-GRADIENT AND SIMILAR METHODS IN HILBERT SPACE

SEC. 5.8

For the global convergence question, we have x.D l

I l b"P" I I =11 P I I

I N, we see that

a"fin-N

1

IIPNI2

and hence

But then, according to Remark I after Theorem 4.2.4, this implies that II VJ(x.)11-> 0, a contradiction. Therefore Vfix,,,) - 0 for some subsequence. Theorem 4.2.1 applied to this subsequence implies that the subsequence is

minimizing, while the inequality f(x",,,) j > n, implies that {x.) is a minimizing sequence. Q.E.D.

6

GRADIENT METHODS IN

(RI

6.1. INTRODUCTION

Since ER' under any norm (all of which are equivalent) is a Banach space, and is in fact a Hilbert space under the usual inner-product, all the results of Chapters 4 and 5 apply here. In fact, of course, more detailed results can be obtained for gradient methods in R' because of the especially simple structure of this space; in this chapter we examine some of these results.

First, because of the finite dimensionality of E', the weak and norm topologies coincide, and any closed, bounded set is (sequentially) compact and vice versa; thus the existence theory of Cnapter I is simplified, the precise simplifications being left to the reader. Second, because of the nature of the topology in R', criticizing sequences 1x,.) for a functional f are generally more valuable since, if W(x,,) is bounded (see Section 4.2), then limit points x' of {xn} exist and must be critical points off; in the following sections we shall examine the consequences of this more closely.

Finally, the asymptotic convergence rates of particular methods can be studied in more detail in ER'; w_- describe some of these results. 6.2. CONVERGENCE OF x,,, i - X. TO ZERO

We mentioned in Section 4.2, particularly in Theorem 4.2.3, that the

convergence of x,t, - x to zero could be of great value; in Section 6.3 we shall examine this in some detail. In the present section we shall examine

situations in which one can assert that x,,, - xn does converge to zero. We have already seen in Chapter 4 -according to Theorems 4.6.1, 4.6.2, x tends to zero when is determined by use of

and 4.6.3-that

142

SEC. 6.2

143

GRADIENT METHODS IN IR'

simple intervals along the line. For the methods of Section 4.7 involving a

range function along the line, we could not in general prove that 0, as indicated by Theorems 4.7.1 and 4.7.2 and their exxII tended versions in Corollaries 4.7.1 and 4.7.2. As shown in Corollary 4.7.3, II --+ 0 implies more generally, whenever II where p = IIP. II - . 0-in many special cases of this general method we can assert that II x.11 -- 0. It is not true in general, however, that the algorithms of Sections 4.3, 4.4, and 4.5 involving minimization along the line necessarily II

yield 11 x.+, - x II - 0; contradicting examples- can -be created. We can,

however, show that for many methods and certain kinds of functions we must always have I I x. (I --. 0.

If W(xo) is compact, then

has limit points x' and Vf(x') = 0.

Hence the following proposition follows. PROPOSITION 6.2.1. If Vf is continuous on the compact set W(xo) and

Vf(x) = 0 has only one solution x*, then x -- x*. We seek more significant results.

THEOREM 6.2.1 [Elkin (1968)]. If W(xo) is compact, if there exists a

a > 0 such that

f(x.+,) <x2 - x Vf(x,)> if x, :e-x2 Thus, if <x2 - x V1(x,)> > 0, we conclude f(x2) > f(x,); a function satisfying this property is called strictlypseudo-con vex [Elkin (1968), Ponstein (1967)].

GRADIENT METHODS IN IR'

SEC. 6.3

145

THEOREM 6.2.4 [Elkin (1968)]. For all x, y in the compact set W(x,,), f(x,,) for all n, let let (x - y, V f ( y ) > > 0 imply f(x) > f (y). Let f 0. Then D such that I I x,, +, I I 0 let no be the greatest integer not exceeding n which is congruent to zero modulo m. Then it is easy to see that

=I

. Yr

r,r

Since Wo = {x; f(x) g(to, 2); and in [to.1, to, 2] if g(to, 1) = g(to, 2). Thus we have located the minimum

in an interval [a b1] smaller than [ao, bo] and we can proceed iteratively. The method would be most efficient if we only need to evaluate g at one new point each time-that is, if either t,,, or t1,2 would equal whichever of to,1 and to,2 lies in (a b,); to allow this, we never choose a, = to,,, b1 = to,2 but in that case of g(to,,) = g(to,2) we define a, = ao, b, = 111,2. If one seeks the smallest final interval [a,,,, bm] for a given m, then it is known [Kiefer (1957), Spang (1962)] that one should choose tj, I

___

Fm- I -1

m+I -i

(b, - a1) + a

tt.2 =

F. -, Fm+1-t

(b, - a) + a,

where Fo = 1, F, = 1, F; = F1-1 + F,-2 are the Fibonacci numbers. This Fibonacci search always requires only one evaluation of g per step. On the

final step, one takes

tm- i,I = (# +

am-I) + am_,

t.-J,2 - 3-I

+ am-i

in order to isolate the minimum best. The final interval has width bm-am=(b,-ao)-1-E

2F.

156

sEC. 6.6

GRADIENT METHODS IN IR'

Since F20 > 104, we see that the intervals shrink rapidly. It is known that for large i, we have

F!-'

0.382,

F,r i

F' F,+t

- 0.618

which allows one to use the simpler formulas:

, = 0.382(b, - a,) + a, t, z = 0.618(b, - a,) + a, t,

The final interval in this way satisfies

b. - a.

= (0.618)m(bo - ao)

Thus one can isolate the minimum in this way as accurately as desired.

Next we turn to methods using interpolation, although some of our remarks apply to direct-search techniques as well. Some of the procedures first seek an interval in which the minimizing point t* lies. Usually this is done by taking some number t, as an estimate of t* and then evaluating g

at 0, to a2t a,t ... , for some sequence a, (often a, = 2') and stopping at the first instance that the values of g do not decrease; if one is willing to evaluate g'(a,t,) as well, one can also stop whenever g'(a, t,) becomes positive.

If the termination procedure occurs at t = t then t, is reduced and the process restarted. Thus we tnally find a, t, with g(a,t,) < g(a,_,t,), g(a,t,) < g(a,+I t,) and t* is isolated in [a,_, t a,+, t,]. The number of evaluations of g will be reduced if t, at least near the solution x*, is a good estimate, for then one would expect to isolate t* in [0, a, t,] every time. In fact if near the soution x* one sets t'. = it, where t, is asymptotically correct, then we should isolate t* easily in [t:, 3t,] and a choice of t* = t' or 2t,', whichever gave the lower f-value, would lead to convergence, as we saw at the start of this section. In this light we see that Theorem 5.8.1 on the convergence of the conjugate-gradient method with c determined as C.

_

.P P

can be considered as providing a good estimate t, which is asymptotically the correct t*; this has been used [Daniel (1967a)] as t, and has given good results. If is any admissible sequence of directions and the functional f on RI satisfies 0 < aI < f x < Al, the analogous choice for t, is P.>

fPP>

GRADIENT METHODS IN R'

sEC. 6.6

157

It has been shown [Elkin (1968)] that one obtains global convergence with t = /,t, where 0 < e < f, < 2 - E, and of course that t, is asymptotically correct. Thus linearization can always be used to get a good estimate t, if but has an estimate one can afford to evaluate f'.. If one cannot compute x f* for the minimum value off, then

f

f(x.) - f

t`

+ a(t) 159

160

VARIABLE-METRIC GRADIENT METHODS IN RI

SEC. 7.2

Suppose Qo is some self-adjoint positive-definite operator (I x I matrix, in IR'. Then we can write

f(xu + tP) =f(xo) + t + o(t) =f(xo) + t-orthogonal to p, and-asymptotically, of course-to po, . . . , also. Thus we can consider that the power of the conjugate-gradient method compared to the steepest-descent method comes from the former's use of a

VARIABLE-METRIC GRADIENT METHODS IN R'

SEC. 7.L

161

good variable metric. In Yakolev (1965), gradient-type methods are considered

strictly in the setting of variable-metric methods-that is, x"+! = X. - t"H"VJ lx")

for some sequence of operators H. and steps t Most of the results there concern convergence under various choices of t" given certain properties of H. such as

0 < a These correspond, with some minor changes, to the methods of Chapter 4, although more detailed convergence rates are often given in Yakovlev (1965). Thus we consider the methods in this completely general setting no further.

In a sense the best metric would be one which turns the level curves J(x) = c into spheres so that the interior normal direction to the surface-that

is, -Vf(x)-points to the point minimizing f. For quadratic functionals

f(x)_Kh-x,M(h-x)>=[h-x,h-x) where

[u, v] _ and thus generates the direction

P. = -J.'-'AX.) This is the direction of Newton's method. Because of this intuitive viewpoint and because Newton's method leads to quadratic convergence [KantorovichAkilov (1964), Rail (1969)], one often tries to pick the variable-metric formulation to mimic Newton's method; thus variable-metric methods are also called quasi-Newton methods [Broyden (1965, 1967), Zeleznik (1968)]. Because of the situation in the constrained case (see Section 4.10), one might

not greatly expect quadratic convergence from mimicking the Newton process if one proceeds close to the Newton direction to the minimum off along that line rather than using the pure Newton step -1

However, the value of t. which minimizes f(x + tp") is asymptotically P") = I r., J. r.>

162

VARIABLE-METRIC GRADIENT METHODS IN IR,

SEC. 7.2

in this case, and thus near the solution x* the minimization along x + tp, nearly leads to the normal Newton step. While one should then hope for quadratic convergence, most results known to us guarantee only superlinear convergence [Levitin-Poljak (1966a), Yakovlev (1965)]. From what we have done, this can most easily be seen from the viewpoint of conjugate gradients. In Sections 5.3, 5.4, and 5.5 we considered a very general form of conjugategradient methods involving arbitrary self-adjoint positive-definite operators H and K, while in Section 5.6 such extra operators were missing. Clearly one may define a general method using operators H., K. at each point x and develop convergence theory and error estimates in terms of the operator T, _ just as in the quadratic-functional case; this is done in Daniel (1965, 1967a, b), and the convergence rates are given via the spectral bounds a, A of T, as usual. If one takes Hz = K, = J` 1, where J', is self-adjoint, uniformly positive-definite, and uniformly bounded, one gets T, = I and a = A = 1, which implies superlinear convergence. In this case, of course, p, = J; 'r, = -J,''J(x,) and we have the minimization modification of Newton's method and a proof of superlinear convergence. It is possible, however, to show that the convergence is actually quadratic. If we let

so that f(x +,) < f(x,), from 0 < aI < J,,< AI for scalars and pick a, A, one can conclude a 11X.., - x* 112 . COROLLARY 7.3.1 [Vercoustre (1969)]. Suppose that H.6, = x,+, - x, for 0 < i < n - 1, 0 < n < I - 1, where 6, = M(x,+, - x,);, suppose that

x for 0 < n < I. Then x,+i = h, the solution for quadratics

f(x)=$<x-h,M(x-h)>.

n-1

Proof. If for some n we have 6. _ E T,8 then JQo .-I

.-1

1-0

(-0

M-'

r=o

.-s

HB,=ET,H.S.-ETJ(x,+1

-X)=ET,M'a, 1-0

T,6, = M-'6. = x.+,

- x.

which is a contradiction. Thus [b( ..... 6.) is linearly independent for all n. Q.E.D. We now suppose that Hn+, is symmetric and that t is always chosen to

minimize f(x. + tp,J, so that

0 = _ J.>

- , for n = 0, 1, ... , r Under these hypotheses for the two-parameter methods of Equation 7.3.1 it can then be proved [Broyden (1967)] that = 0 if i :*j for 0 < i, j < r and, therefore, that we have a conjugate-direction method. The proof goes roughly as follows. From the definitions of and it easily follows that

P.+. = a.P. +

for some scalars a,,, 8.

168

SEC. 7.4

VARIABLE-METRIC GRADIENT METHODS IN (R,

Now,

= 0

r = 0,

n=0,1,...,r

The induction then proceeds easily to give

i = 0, 1,...,n,

=0,

n =0, 1,...,r

Thus we have found a two-parameter class of exact variable metric methods; the admissibility of the direction sequence for nonquadratic functionals still remains unknown, however. Since a study of the admissibility requires considerable specialization of the vectors z,,, we consider this question for special methods, although little information is available even in special cases. 7.4. SOME PARTICULAR METHODS

We consider first the class of variable-metric methods [Broyden (1967)] defined via

q. ° (a.P. - R.H.a.) Z.

+ P.P.)

an = (I + )

(7.4.1)

P.>

a.,P. ) ., H..> Y.=(1-Q.t.

where P. is arbitrary, t is chosen so PA> = 0, and Ho is symmetric. By a straightforward inductive argument [Broyden (1967)] or by using Corollary 7.3.1, one can show that, if M is symmetric and nonsingular, then the b, are linearly independent and hence the method is exact.

VARIABLE-METRIC GRADIENT METHODS IN IRt

SEC. 7.4

169

THEOREM 7.4.1. If Ho and M are positive-definite and fin > 0, then it follows that H. is positive-definite for each n. Proof: The proof goes by induction. If H. is positive-definite, let LL* _ v =/L*x, and w/= L\*5N, we then have H,,. If we let u = Kx, H., 1X> = -

+

V,''

2

&t. 12 [l ;, U>

The only possible negative terms come from

v, v

lv w>2 .w, w>

which in fact is nulinegative by the Schwarz inequality and is positive unless

v = )w. If the term 2 is also zero, then = 0 but = -Ku, u> # 0. Therefore, Kx, H.+ x> > 0 if x:;& 0 and hence definite. Q.E.D.

is positive-

Since only finitely many iterations need be used for quadratics, we have with

">E>0

An

in that case. If we use the algorithm for more general functionals, however,

we cannot immediately conclude such bounds (see, however, Theorems 7.4.2 and 7.4.5). Similarly, it is not known whether or not such bounds exist for quadratics in infinite-dimensional spaces, a result which would at least give some indications for the nonquadratic case in IR'. At this time numerical experience testing various choices of the parameters f, is rather limited; thus the method for arbitrary P. requires further study both theoretically and computationally. In practice, when one uses this method one seldom actually performs an exact minimization along the direction p to reach.-(.,, -that is, p,> = 0; it is striking to note, however, that if one seldom has = 0 for all n, then the directions are independent of the parameters {f,). Computationally, however, one finds great dependence on the choice of these parameters.

THEOREM 7.4.2. Under the assumptions of this section, if pp> = 0, then the direction n+1

11p.+' 11

determined by Equation 7.4.1 is independent of f,,.

170

SEC. 7.4

VARIABLE-METRIC GRADIENT METHODS IN R,

Proof: ,, Hf(j)> H,,Vf(xw+,) - Hwaw +

Nwi'w

where tfpf{<JP H"a"9 - I)

-Pw+I = HUVf(xw+I) - H"b"

-Pw

H"aw

H,,a,i

-Pw

`a". H.Vf(x")> H a w

H"aw>

w

aw, H"(- aa>S -+HA> = + Therefore,

a.-, g., H.- g. l P. = Hg. - S.,[H.H.-I g. + g.- H.-. g. Then for n < m, we have

_ + fig,., H.- I [g.- - g,J>

g.,H.- g. g.- H.-Ig.=0+- X

X

g.,

1, H.

g., H.- g. + g,,

I g.

172

VARIABLE-METRIC GRADIENT METHODS IN (R'

SEC. 7.4

is positive-definite, we must have

Therefore, since

= 0 for n < m But then, from the definitions of H. and B,,, we have for n < m that

H.g..=H.-1g.=... =Hog. and hence g,,, is an eigenvector with eigenvalue 1 for Ho 1 H. if m > n. There-

fore, we find that P.

[I + H0g.g" dP.-

8.. Hog.> + g.-1,p.-1>

For the conjugate-gradient method, po = Kg0 = po, and therefore x', = x1.

If p, = 2,p; and x;+1 = x,+, for i = 0, 1, ... , n, as is true for n = 0, then P.+1 =

Kg.. , Hog.+1>I + Hog... 1g:J2.P., 8.+1, Hog.+1/ + g.,P.

while

P;.+1 = Hog.+1 + 1>P;.

H0g.+, + p g., Hog. 8.+1, Hg.+1> + AX g.' Haa.> If + Hog.+3 X f

1

+ Vg.' H0g.>H0g +1- Hog.+,$., A.P., + P.+

(where a...

is defined by the equality) since = - 6.-1 =

Thus

2.+1P:+1 and hence

x,+:. Q.E.D.

Thus, because of Theorems 7.4.2 and 7.4.3 and the results of Chapter 5 and Section 6.4, we know that the methods of Equation 7.4.1 (and in particular, Davidon's method) yield global convergence when implemented

with exact minimization along x + tp and applied to uniformly convex

SEC. 7.4

VARIABLE-METRIC GRADIENT METHODS IN IR'

173

quadratic functionals-for another proof see [Horwitz-Sarachik (1968)]-and we have estimates on the local-convergence rate. Similarly, for x, near x*, for uniformly convex nonquadratic functionals, their relationship to conjugate-gradient methods gives a local-convergence result for these methods as well. Only very recently [Powell (1970)] has a global-convergence result for the Davidon method been found; we give this result and outline the proof. THEOREM 7.4.4 (Powell (1970)]. Let f be twice continuously differenti-

able, let f > aI, a > 0, and let the Davidon algorithm be used with exact x minimization along x + tp,,, starting with an arbitrary x, and positivedefinite symmetric H,. Then the sequence tion 2.

converges to the unique solu-

Proof. Since H. is positive-definite for each n [Fletcher-Powell (1963)],

we can define r. - H;'. Writing

7,.x.+, -x. one can check that

r.+, _(I-)r"\I -

Using the fact that = t.

0

and 1

1

I(x,.+, ), H.+, f(x.+,)> = < f(x,.+, ), H. f(x.+, )> I

+

Ax.), H. f(x,.)>

we conclude that

tr(r.) +

11 ofx.+,) I I;

-

Ilof(x.+ IIZ + 1i H. x.+,) I_I

I I Vf(x.)112

H (x,.)

174

SEC. 7.4

VARIABLE-METRIC GRADIENT METHODS IN IR,

Solving this recursion thus yields tr

tr (ro) + 1-0

I I Vf(x

I I VAX. + J II{2

)112

J (x0), Ho J (x0)



0 and a,j < 0 if i # j,

and that 0: RI -p R' is isotope if -x < y implies O(x) < ra(y); is diagonal if, for I < i < 1, the ith component of O(x) depends only on the ith component of 0; and is convex if

0 [,lx + (1 - ))Y) < AO(x) + (I - AWY) for all x, y in [R'. Such operators commonly arise in the numerical solution of boundary-value problems for mildly nonlinear differential equations [Bers (1953), Greenspan-Parter (1965), Schechter (1962)]. The following two results are typical of what is known for the full nonlinear methods. PROPOSITION 8.3.2 [Bers (1953), Ortega-Rheiboldt (1967a, 1968)]. Let A be an M-matrix, 0: IR' IR''a continuous, diagonal isotone mapping, and set

G(x) = Ax + O(x)

Then for any xu in

IR',

the nonlinear Jacobi and nonlinear SOR (for

189

sEc. 8.3

OPERATOR-EQUATION METHODS

0 < w S 1 (and hence, for w = 1, the nonlinear Gauss-Seidel)] methods all yield sequences xa converging to the unique solution R of G(x) = 0.

PROPOSITION 8.3.3 (Caspar (1968), Kellogg (1969), Ortega-Rheinboldt

(1968)]. Let G(x) = H(x) + V(x) with H and V continuous, let G(R) = 0, let

Z 0 for all x, y in 1, and for each bounded set B let there exist positive constants

L, and a, such that

IH(x)-H(Y)IISL,llx-YII and Z a, I I x - y 112

for all x, y in B. Suppose that

0 0.

PROPOSITION 8.3.6 [Caspar (1968)]. Let G(x) = H(x) + V(x) with H

and V continuously differentiable, let G(x) - G(y) < G',(x - y) if x < y or y < x, let rI + Hx and rI + V' be M-matrices for all r > 0 and all x, let G(R) = 0 and xo > 2, G(xo) > 0. Then, for the Newton-l-step-ADI method, if

d(x.) 0

d,

where e, is the vector with e,, = 6,,, the Kronecker delta. It is clear that if the d, for i = 1, 2, . . ., I are small enough, this method will behave essentially

as well as the method with derivatives; the problem here, as with all the

other methods wherein derivatives are replaced by differences, lies in how to choose the d,. An excellent analysis of this problem has been given for Davidon's method [Stewart (1967)]; since the viewpoint is of interest for use on any method, we present the ideas here once and for all. The whole basis of most gradient methods is to treatf(x) as a quadratic, locally; thus we shall consider the problem of approximating the derivative

y = f'(0) of the quadratic f(t) =f(0) + yt + 40&t2 by the difference

f(d)

f(0 d

= Ya

The two sources of error in approximating the scalar y by the scalar Yd are the truncation error in the divided difference and the cancellation produced in computing f(d) - f(0) for small d; clearly we should balance these errors. We can estimate the relative truncation error by

YdY

YI

1I

Ild Y

If we assume that, in computing f(t), we actually compute

f +(t) = f(t)(l + E),

l e I < it known

then the relative cancellation error can be estimated as 21

f(o) f(d) - f(0)

I

q

If we equate the two estimates and solve for the number d, we find that I dl should be the positive root of

3a2z3 + lal ly! z2 - 41f(0)I IYI n =0 and sign (d) = sign (ay).

196

ssc. 9.2

AVOIDING CALCULATION OF DERIVATIVES

EXERCISE. Show that the above choice of d is correct.

To avoid solving the above cubic, experimentally it has been satisfactory to ignore the cubic or quadratic term, depending on which gives a smaller root, to solve the resulting simpler equation, and to refine the result by one Newton step applied to the original cubic. This gives the following:

2{ILI}u2'

Idl=r[1-

z = 211 fao)I}"'

Idl = s[1 -

JaIr 1a T1+14Iy1]

if

y2ZIaf(o)I17

if

y2 < Iaf(O)I,

This computation requires a crude estimate of y, which can easily be obtained, and also an estimate a ,of f "(0). For the Davidon method we are considering,

f(xx + te,) =f(xJ + t + 3g,'12+ o(t2) where a,, is the ith diagonal element of f' s'. Recall that H. - f' - '; thus we seek the diagonal elements of H.'. These can in fact be computed recursively (see Proof of Theorem 7.4.4) without knowledge of the off-diagonal elements H.-' from the equations [Stewart (1967)]

Hi+'i = Hr ' + ( ltr

-

f) a,a* +

I

[Vf(x)o' + a,vf(xi)*]

where

P, = (Vf(x), P,>,

.8r =

and we replace Vf(x,) by its approximation. EXERCISE. Show that the above recursion for H;+', is correct.

Thus we have a rule for determining the size of the numbers d, in the difference approximation to Vf(xJ. Certainly we have ignored many problems, implying that our analysis is far from rigorous, but the ideas in practice appear to lead to good results. For further computational details and examples the reader is referred to Stewart (1967), where the method is shown to be quite powerful in practice. A similar approach has been applied for a Davidonlike method applied to minimize I I J(x) 112 for a nonlinear operator J [Fletcher (1968)]; here J'- I usually does not exist, so one is led to finding a Davidontype approximation to a pseudo-inverse J'+ using differences. The method as proposed in Fletcher (1968) is exact for quadratics. EXAMPLE [Stewart (1967)]. The Davidon modification without derivatives was used to minimize

AVOIDING CALCULATION OF DERIVATIVES

SEC. 9.3

197

f(x, y) = 100(y - x2)2 + (1 - x)2 starting with xo = -1.2, yo = 1.0. After 163 function evaluations, f was reduced to 9 x 10-12 with (x, y) = (1.000002, 1.006003)

9.3. MODIFYING NEWTON'S METHOD

Computationally, at least two problems are involved in Newton's method: the need for solving a linear system at each step, and the need for evaluating roughly 12,derivatives at each step. One can eliminate most of the derivative evaluation by using the derivatives at one fixed point throughout,

but this eliminates the powerful feature of quadratic convergence. An alternative, of course, is to use differences to evaluate the derivatives. If the step size used for the differences is small enough, one should maintain the rapid convergence, it would seem. The local-convergence properties can be

analyzed by means of Proposition 8.2.1; if we replace the derivative of J(x) by AJ(x, f) ^ y, where the components yr =

E

[J(x + fe) - J(x.)]

as in Section 9.2, we have the following local result [Dennis (1969)]. PROPOSITION 9.3.1. Let

Ao' ° [AJ(xo, fo)] exist with

IIA0 'J'-A0 1J;II_KIIx-yIl for x, y E S(xo, r). Let e > 0 be such that

K(e + Suppose that

fo) < 1 and

2

>h=

KII Ao'J(xo) II

(1 - Ke - Kfo)2

2

r ro =

1

-

h

x

IIAo'J(xo)II

1-Kf-+Kfo

and that f is a sequence of numbers such that If. I < f and x + f.e, E S(xo, r) f o r i = 1, ... , 1. Then the sequence

x.+, = X. - [AJ(x., fJ]-'J(xJ

198

sEc. 9.3

AVOIDING CALCULATION OF DERIVATIVES

is well defined and converges to 2, solving J(x) = 0. If, for a constant C, we have I E. I S C I I J(x.)11, then the convergence is quadratic. For the global properties the situation is somewhat Less simple; we have to be sure that the directions generated are descent directions. Thus let us suppose that J(x) = Vf(x) and that AJ(x, e) is as defined before; let us consider the follwoing direction-generating algorithm [Goldstein-Price (1967)]: If AJ(x1, e.) is singular, or if

0, setting x' = x, starting with i = 1 up to i = 1, if f(x' + a,e,) < f(x'), then x' is replaced by x' + b,e, and i by i + 1; if f(x' + b,e) Z f(x'), but f(x' - 26,e) < f(x'), then x' is replaced by x' 26,e, and i by i + 1; otherwise x' is not changed. Finally, when i reaches 1-+- 1, we set EM(x) = x'. The entire algorithm proceeds from x to as follows, starting with some initial x, and x, = EM(xo) # x0. We compute 4 = EM(x,); if x ' . = x then is cut in half f o r i = 1, 2, ... , l and the

-

iteration restarts at x,,. Otherwise, if f[EM(2x f(x,), we set xx+, = EM(2x, if the latter inequality is invalid, we set x = EM(x ). Iff is strictly convex with Vf continuous and lim f(x) = +oo, then it can be shown [Cea (1969)] that II x - 2 -.0, where .2 minimizes f over IR'. A computer implementing this method can be found in Kaupe (1963).

Since the coordinate directions, which are used in the above algorithm, need not be the best ones, the process has been modified as follows (Rosen-

204

AVOIDING CALCULATION OF DERIVATIVES

SEC. 9.6

brook (1960)]. Given a vector x and I orthonormal directions d,(x),I step sizes 8,(x), and two parameters a > 1, ft E (0, 1), the exploratory-move operator EM(x) is defined as follows. For i cycling through the values 1, 2, ... ., 1setting

x' = x, if f(x' + 6,d,) < f(x'), then we replace x' by x' + 6, d,, i by i + 1 (or 1 by 1), b; by a3 and record a success; otherwise b, is replaced by -,86, and a failure is recorded. After one success and one failure have been recorded for each value of i, we set EM(x) = x'. The iteration, starting with xa and

and from directions d,(xa), ... , d,(xo), now proceeds from x to EM(x ). Let 2., be the sum of the steps in the as follows. Set to 2.,d,(x ). New directions direction d,(x,) and define vectors d,(x,+,) are now obtained by orthonormalizing the vectors q,; this completes the description of the method. Roughly speaking, we can say that is the most successful motion found so far, d2(x,+,) is the most successful direction orthogonal to d,(x,+,), and so on. A further modification of this to move to the point minmethod [Swann (1964)) is, for each direction imizing f in that direction, and then to compute new directions as before. We do not consider these methods further since we believe the methods to be considered next to be of greater importance and usefulness. We have seen several times in earlier chapters that there is great advantage to using conjugate directions of some type; the above methods, however,

all deal with orthogonal directions-that is, 1-conjugate directions. It is possible, however, to generate directions that are conjugate (at least for quadratics) without dealing with derivatives. These methods, which appear to be the best of those that ignore derivatives, are based on the fact that if one minimizes a quadratic

f(x) = in the direction p from two points x, and x2, arriving at the points x', and x'2, then x', - x'2 is M-conjugate to p, since

E

let dk+ I., = dk,,

for r # s,

dk+1,, = dk,,+,

and let

vk+1 - tk.sak ak yy

If, however, tk.syk <E ak

let dk+,,, = dk,, for r = 1, 2, . . . ,1 and set vk+I = 6kConsider the method applied to minimize the quadratic

f(x)_ Suppose each time that the last direction dk,, is replaced by dk,,+,. Then the last step of the k-iteration and that of the (k + 1)-iteration are in the same direction and, therefore, because of our earlier remark, we shall next- introduce a conjugate direction. After 1 + 1 steps we would have I conjugate directions and, if they are linearly independent, we shall therefore get the correct solution on the next iteration. The method of choosing the direction to be

206

sec. 9.6

AVOIDING CALCULATION OF DERIVATIVES

eliminated is the technique that keeps the directions dk....... dk,, independent, as we shall see; it also determines which direction, if any, is eliminated at each step and thereby invalidates the above argument. Thus it does not appear possible to prove that this method is exact for quadratics, although we can prove convergence. First we show that the directions dk,, (for arbitrary functionals) are linearly independent.

[Zangwill (1967)]. The directions dk....... dk,, are

THEOREM 9.6.1

linearly independent: In fact, their determinant satisfies det [(dk....... dk.,)] ak > E.

Proof: The result is true for k = 1; assume it for k. Then, since xk.J - xk.0 = akdk.J+1, we have det [(dk,,, ..., dk..-1, dk.,+I,

I

'om'' det [(dk,,,

ak

dk.,+I, ..., dk.,)]

..., dk.J)] = tk.sak for all s ak

The choice of s-that is, the direction to try to replace-gives us the greatest chance of replacement, while the criterion for replacing or not yields {k+1 > E. Q.E.D.

Having the above fact, we can prove convergence. THEOREM 9.6.2. Let f be a continuous and strongly quasi-convex functional on IR', and let the above method be applied starting with an arbitrary x1,,, ; suppose the sequence {xk,,],

r =0, 1,...,1,

k = 1,2,...,

is bounded. Then any limit point x' of xk,,, as k - oo for any ro = 0, 1,

... ,1 is also a limit point of xk.,, r:# r0, as k

oo, and for each such limit point there exist I linearly independent directions dl, ... , d, such that f(x') S

f(x'+td,)foralltandr=1,2,...,1.

Proof Since I I dk., II = I also, given any subsequence K of integers k, we can find a further subsequence K, such that dk,, d, for r = 1, 2, . . . , I as k , oo with k E KI, and xk,, -+ x, for r = 0, 1,. .. , I as k oo with k E K, ; we show next that x,+ 1 = x, for r = 0, 1, ... , 1- 1. Recalling that

f(xk,,+,) = 0. Since the d, are linearly independent, we have

Vf(x') = 0. Q.E.D. COROLLARY 9.6.2. If, in addition to the hypotheses of Corollary 9.6.1,

we know that [x; Vf(x) = 0} contains no continuum, then the sequence Xk., converges.

Proof. We have shown that the difference of successive elements in the sequence xk.0, ... , xk,, tends to zero; the same argument shows that I I xk,1 -

I

II I I =I I xk.1 - xk+1,011- -> 0

Thus we may apply Theorem 6.3.1. Q.E.D. COROLLARY 9.6.3. If the continuously differentiable, strongly quasiconvex functional f is strictly pseudo-convex-that is, if <x - y, Vf(y)> Z 0

SEC. 9.6

AVOIDING CALCULATION OF DERIVATIVES

implies f(x) > f(y) for x # y-and if [xk,,} is bounded, then xk,, --+ z, the unique minimizer for f.

Proof: Limit points x' exist with Vf(x') = 0 by Corollary 9.6.1; then, by the strict pseudo-convexity, for any x, we have 0 = <x - x', Vf(x )> and hence f(x) > f(x'), so x' minimizes f. By the strong quasi-convexity, such a minimizer is unique. Q.E.D. COROLLARY 9.6.4. If f is uniformly quasi-convex, strictly pseudo-con-

vex, and continuously differentiable, then for any x,,0, the sequence (xk,,} converges to the unique k minimizing f. COROLLARY 9.6.5 [Zangwill (1967)]. If 0 < aI S f' in IR', then for any x1,0, the sequence [xk,,) converges to the unique z minimizing f. EXERCISE. Prove Corollaries 9.6.4 and 9.6.5.

We have not, however, been able to prove that the method is exact for quadratics; Zangwill (1967) has developed a modification of the method that is exact. The Zangwill method is as follows. Let e i = 1, . . . , I be the coordinate directions and let an initial point x0., and directions dl,,, . . . , d,,,, II d,,, II = I be given. Let to,, minimize

f(x0,, + td,,,) and let xo.I+1 = x0,1 + to,ldl,,

Set n = 1 and iteratively apply the basic k-iteration starting with k = 1; the basic k-iteration is as follows, given xk_ I.,+ dk....... dk,, and n: (1) compute a' to minimize f(xk_1,,+, + let n' = n, and replace n by n(modulo 1) + 1. If a' :?,- 0, let xk,o = xk_,,,+1 + a'e,; ; if, however, a' = 0, return to the start of step 1, noting that if step 1 is performed I times, we may consider xk_,,,+, to be the solution. (2) For r = 1,. .. ,1, compute tk,, to minimize f(xk,,_ 1 + tdk,,) and let

Define dk,,+ l

I I xk,, - xk- 1.1+

1

compute tk,,+, to minimize f(xk., + tdk,J+,), and setxk,l+1 = xk,, + tk.,+1 dk,,+l

Define directions dk+,,, ° dk,,+, f o r r = 1 , 2, ... ,1.

SEC. 9.6

AVOIDING CALCULATION OF DERIVATIVES

209

This method differs from the preceding primarily in its feature of minimizing over the coordinate directions as well as the directions dk,,; this feature allows us to revise the directions dk,, in the simple manner of the algorithm and thus obtain exact convergence for quadratics, as the following theorem shows. THEOREM 9.6.3

[Zangwill (1967)].

Let f(x) = ,

where M is self-adjoint and positive-definite, and let the initial point xo,, be given. Then the iteration stops during step 1, with xk_,,,+, = h, the solution, for some k < 1. Proof: Assume that at the start of the basic k-iteration at step k for

k < n - 1, the directions

dk.1-k+I, dk.1-k+2, ..., dk,,

are mutually M-conjugate and linearly independent; clearly this is true for k = 1, starting the induction. If we do not stop in step I this time, then xk-1,1+I : Xk.o and, since M is positive-definite, f(xk,o)