Nondifferentiable optimization (Mathematical programming study)

MATHEMATICAL PROGRAMMING STUDIES Editor-in-Chief M.L. BALINSKI, International Institute for Applied Systems Analysis, ...

Author: M.L. Balinski | Philip Wolfe (editors)

35 downloads 928 Views 7MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

MATHEMATICAL PROGRAMMING STUDIES

Editor-in-Chief M.L. BALINSKI, International Institute for Applied Systems Analysis, Laxenburg, Austria, and City University of New York, N.Y., U.S.A. Senior Editors E.M.L. BEALE, Scientific Control Systems, Ltd., London, Great-Britain GEORGE B. DANTZIG, Stanford University, Stanford, Calif., U.S.A. L. KANTOROVICH, National Academy of Sciences, Moscow, U.S.S.R. TJALLING C. KOOPMANS, Yale University, New Haven, Conn., U.S.A. A.W. TUCKER, Princeton University, Princeton, N.J., U.S.A. PHILIP WOLFE, IBM Research, Yorktown Heights, N.Y., U.S.A. Associate Editors PETER BOD, Hungarian Academy of Sciences, Budapest, Hungary VACLAV CHVATAL, Stanford University, Stanford, Calif., U.S.A. RICHARD W. COTTLE, Stanford University, Stanford, Calif., U.S.A. J.E. DENNIS, Jr., Cornell University, Ithaca, N.Y., U.S.A. B. CURTIS EAVES, Stanford University, Stanford, Calif., U.S.A. R. FLETCHER, The University, Dundee, Scotland D.R. FULKERSON, Cornell University, Ithaca, N.Y., U.S.A. ARTHUR M. GEOFFRION, University of California, Los Angeles, Calif., U.S.A. TERJE HANSEN, Norwegian School of EconomIcs and Business Administration, Bergen, Norway ELI HELLERMAN, Bureau of the Census, Washington, D.C., U.S.A. PIERRE HUARD, Electricite de France, Paris, France ELLIS L. JOHNSON, IBM Research, Yorktown Heights, N.Y., U.S.A. C.E. LEMKE, Rensselaer Polytechnic Institute, Troy, N.Y., U.S.A. GARTH P. McCORMICK, George Washington University, Washington, D.C., U.S.A. GEORGE L. NEMHAUSER, Cornell University, Ithaca, N.Y., U.S.A. WERNER OETTLI, Universitiit Mannheim, Mannheim, West Germany MANFRED W. PADBERG, New York University, New York, U.S.A. L.S. SHAPLEY, The RAND Corporation, Santa Monica, Calif., U.S.A. K. SPIELBERG, IBM Scientific Center, Philadelphia, Pa., U.S.A. D.W. WALKUP, Washington University, Saint LOUIS, Mo., U.S.A. C. WITZGALL, National Bureau of Standards, Washington, D.C., U.S.A.

MATHEMATICAL PROGRAMMING STUDY 3 N ondifferentiable Optimization Edited by M.L. BALINSKI and Philip WOLFE

November (1975)

NORTH-HOLLAND PUBLISHING COMPANY - AMSTERDAM

© The Mathematical Programming Society, 1975 All rights reserved. No part of thIs publication may be reproduced, stored in a retrieval system, or transmItted in any form or by any means, electronic, mechanical, photocopying. recording or otherwIse, without the prIor permission of the copyrIght owner.

This STUDY is also available to non-subscribers in a book edition.

Prmted in The Netherlands

PREFACE We will attempt to describe nondifferentiable optimization (NDO), give some reasons for doing it, and sketch the contributions of the eight papers constituting the major part of this volume. NDO is concerned with the minimization of a continuous real-valued function, of several real variables, which is not necessarily differentiable. The nondifferentiability is not due to properties of the domain of the function, which in most cases can be taken to be all of Euclidean n-space, but to the function itself; yet we will rule out truly wild functions. The examples below indicate the scope we consider, and they are difficult enough. There are at least three good reasons for concern with NDO: curiosity; the practical importance of some problems having seriously nondifferentiable objective functions; and the possibility of recasting some familiar problems as NDO problems so as to improve our understanding or our ability to handle them. On the first count, the assumption of differentiability was once important in the existence theory of optimization-necessary conditions and duality. Now, thanks in large part to Fenchel and Rockafellar, we realize that it is really irrelevant to most of that theory. On the other hand, most algorithms for optimization problems seem to make heavy use of derivatives, or at least of differentiability, and the study of convergence of the algorithms, and especially of their rate of convergence, seems to require more than that: it is customary to assume that the functions involved have positive lower and upper bounds for the eigenvalues of their matrices of second derivatives. Are these assumptions necessary for useful results? Is there an effective algorithm for minimizing an arbitrary convex function? Do the methods of steepest descent, the conjugate gradient methods, and the quasi-Newton methods have reasonable extensions to the kinds of problem described below? The most general class of NDO problem much studied is the Minimax Problem: let g(x, y) be defined on a suitable subset of X x Y(X and Ybeing, usually, subsets of Euclidean space); we seek the minimum of f(x) for x E X, where f(x) = max {g(x, y): y E Y}. No matter how smooth gis, v

vi

Preface

j is almost surely significantly nondifferentiable. In the simplest case Y is finite and we may write j(x) = max {gi(X): i = 1, ..., m}, taking each gi

differentiable. Such functions will be generally differentiable almost everywhere, as convex functions are, but the minimum sought is mostly likely to fall at a nondifferentiable point: j(x) = Ixl = max {x, - x} fails only for x = 0, but if one depends on differentiability at the minimum, he can say little about this problem. Danskin* originated the computational study of the minimax problem in the context of some military problems of "one-sided" games, and much subsequent work is in that spirit. It forms an important part of approximation theory, for example in posing the problem of finding the best "Chebycheff' approximation of a given function h(y) by linear combinations of functions!;: for that problem g(x,y) = Ih(y) - LiXi!;(y)l. Further, since any convex function j has the form max {g(x, y): y E Y} with g linear in x, problems of the minimization of general convex functions overlap largely with minimax problems. Each paper in this STUDY deals essentially with a minimax problem, although that fact is not always immediately obvious. Paper 3 offers an example of a problem whose statement as a minimax problem-a convex one, at that---_0 for a l l x ~ Q ;

yi = 1

/ff(x) > O,

(28)

/ff, f~) = 0.

Proposition 3.1 together with Proposition 2.1 and its corollaries may be used to provide a simple proof of an optimality condition for problem (16). We shall say that ~ is a local minimum for problem (16) if ~ is a minimizing point of g[x, y[fl(x)], ..., ~[fm(x)]] over a set of the form Q c~ {x: [x - ~] _< e}, where [.I denotes the Euclidean norm and e > 0 is some positive scalar. If~ is a unique minimizing point over Q n {x: Ix - ~] _< e}, we shall say that ~ is an isolated local minimum. Proposition 3.2. Let Q be a closed convex set and ~ be an isolated local minimum for problem (16). Then there exists a multiplier vector = (-ill,..., ym) satisfying

I

1

i=1 ~ y vJi Ix=" (x - x--) >_ 0 for all x e Q,

(29)

y' = 0

if f~(2) < O,

i = 1..... m,

(30)

yi= 1

/ff~(2)>O,

i = 1..... m,

(31)

0 < y' 0, i = 1,2 ..... m}, 1- = { i : f ~ ( 2 ) < 0 , i = 1 , 2 .... ,m}, I ~ = {i: f~(2) = 0, i = 1,2 .... ,m}. Assume that 5 > 0 is taken sufficiently small to guarantee that fi(x) > 0 ft(x) < 0

for all x ~ S(2; 5), for all x s S(2; 5),

i e 1 +, i e 1-.

Let us first consider the case where the objective function O in problem (16) has the particular form

g[x, 7[f,(x)],...,

7[fm(x)]] =

go(x) + i=1 ~ gi(x)7[fi(x)],

(35)

where g~: R" ~ R, i = 0 . . . . , m, are continuously differentiable functions. Now if we make the assumption gi(2) 4:0

for all i s I ~

(36)

14

D.P. Bertsekas

/ Nondifferentiable

optimization

via a p p r o x i m a t i o n

we have that, when g has the form (35), problem (16), locally within a neighborhood of ~, may be written as

min~go(x) + ~ g,(x) fi(x) ieI + ( + ,~,o+ ~ max[-O, g,(x) f(x)] + ,~,o ~ min[O, g,(x) f~(x)]} (37) where I ~247 = {i:

gi(~) >

O, f~(x) = 0},

I ~ = {i:

gi~)

< O, f(x) = 0}.

Since 2 is an isolated local minimum of the above problem, it follows under the mild assumption

V[gi(x) fi(x)]

x=x'

i ~ 10

are linearly independent vectors

(38)

that the set I ~ is empty and we have

gift)

> 0

for all i ~ I ~

(39)

Notice that the previous assumption (36) is implied by assumption (39). This fact can be verified by noting that 2 is an optimal solution of the problem min

~go(X)+ i~, gi(x) fi(x)+ i e~l o + z,+ i~, gi(x)fi(x)~ d + ~ "o -

g i ( x ) f t (x) k, where k is sufficiently large, are identical to the ones that would be generated by a method of multipliers for problem (41) for which: (a) Only the constraints g ~ ( x ) f ( x ) < zi, i e I ~ are eliminated by means of a generalized quadratic penalty. (b) The penalty parameter for the (k + 1)'hminimization corresponding to the ith constraint, i e I ~ depends continuously on x and is given by = ck/g

(x).

(c) The multiplier vector ~ at the beginning at the (k + 1)th minimization is equal to Yr. Alternatively, the vectors Xk, Yk for k > k, where k is sufficiently large, are identical to the ones that would be generated by the method of multipliers for problem (41) for which: (a) Both constraints g~(x) f ( x ) < z~, i e I ~ and 0 < z~, i e I ~ are eliminated by means of a generalized quadratic penalty.

D.P. Bertsekas / Nondifferentiable optimization via approximation

17

(b) The penalty parameter for the (k + 1)th minimization corresponding to the ith constraints depends continuously on x and is given by ~(x) =

2cdgi(x). (c) The multiplier vectors ~ , ~ at the beginning of the (k + 1)thminimization (where ~ corresponds to the constraints #~(x) f~(x) < zi, i ~ I ~ and ~ corresponds to the constraints 0 < z~, i t I ~ satisfy ~ = Yk-and ~ = 1 -- y~.

The equivalence described above may be seen by verifying the following relations which hold for all scalars y ~ [0, 1], c > 0, g > 0, f . ~ ( f , y, c) = min[z + (g/2c) {[max(0,y + (c/g)(gf - z))] 2 - y2}] 0 0, {y~}, {W~,} satisfy 0 n,

a,(m,m) -- 2 Isin (i)1 i/m + Y~ la,(m,j)l, j~p rn

bi(m)

=

e "/i

s i n ( / " m),

i =

We represented max{f1, f2 ..... fs} by

1.....

5,

m =

1, ..., 10.

D.P. Bertsekas / Nondifferentiable

optimization

0

.=.~. I I I I I I I I

~

~ t r l

I I I I I 1 ~ 1 r

r162

N

I

.~

I

I

I r

I

I

,...~ t'q

0

tr

tr

I

~'%

t

I

.8 t"q ("4

II

.=.

I

t,r

II

I

I

tr

I

I

it-i

I

I

O 0

0,.,.

[.-o

via approximation

23

24


m a x { A , . . . , fs} = A + 7[f2 - A

+ ~:[f3 - A + ~'[A - f3 + ~:[f5 - A ] ] ] ]

and used our approximation procedure in conjunction with iteration (34). The starting points were x ~ = x z = ..- = x 1~ = 0 and y01 = y~ = yo3 = yo4 = 0. The optimal value obtained is -0.51800 and the corresponding multiplier vector was = (1.00000, 0.99550, 0.89262, 0.58783). It is worth noting that for minimax problems of this type the optimal values of the approximate objective obtained during the computation constitute useful lower bounds for the optimal value of the problem. Table 3 shows the results of the computation for the case where unconstrained minimization was "exact" (i.e., e = 10- 5 in the D F P routine% It also shows the results of the computation when the unconstrained minimization was inexact in the sense that the k th minimization was terminated when t h e / t - n o r m of the direction vector in the D F P was less than max[lO -s, lO-k].

References [17 D.P. Bertsekas and S.K. Mitter, "A descent numerical method for optimization problems with nondifferentiable cost functionals", SlAM Journal on Control 11 (1973) 637-652. [2] D.P. Bertsekas and S.K. Mitter, "Steepest descent for optimization problems with nondifferentiable cost functionals", Proceedinos of the 5th annual Princeton conference on information sciences and systems, Princeton, N.J., March 1971, pp. 347-351. [3] D.P. Bertsekas, "Combined primal-dual and penalty methods for constrained minimization", SIAM Journal on Control to appear. [4] D.P. Bertsekas, "On penalty and multiplier methods for constrained minimization", SIAM Journal on Control 13 (1975) 521-544. [5] J. Cullum, W.E. Donath and P. Wolfe, "An algorithm for minimizing certain nondifferentiable convex functions", RC 4611, IBM Research, Yorktown Heights, N.Y. (November 1973). [6] V.F. Demyanov, "On the solution of certain minimax problems", Kibernetica 2 (1966). [7] V.F. Demyanov and A.M. Rubinov, Approximate methods in optimization problems (American Elsevier, New York, 1970). [8] A.V. Fiacco and G.P. McCormick, Nonlinear programming : sequential unconstrained minimization techniques (Wiley, New York, 1968). [9] A.A. Goldstein, Constructive real analysis (Harper & Row, New York, 1967). [10] M. Held, P. Wolfe and H.P. Crowder, "Validation of subgradient optimization", Mathematical Programmin0 6 (1) (1974) 62-88. [11] M.R. Hestenes, "Multiplier and gradient methods", Journal of Optimization Theory and Applications 4 (5) (1969) 303-320. [12] C. Lemarechal, "An algorithm for minimizing convex functions", in : J.L. Rosenfeld, ed., Proceedings of the IFIP Conoress 74 (North-Holland, Amsterdam, 1974) pp. 552-556.


25

[13] B.W. K o n and D.P. Bertsekas, "Combined primal-dual and penalty methods for convex programming", SIAM Journal on Control, to appear. [14] B.W. Kort and D.P. Bertsekas, "Multiplier methods for convex programming", Proceedings of 1973 IEEE conference on decision and control, San Diego, Calif., December 1973, pp. 428-432. [15] B.T. Polyak, "Minimization of unsmooth functionals", ~,urnal Vy~islitel'noi Matematiki i Matemati~eskoi Fiziki 9 (3) (1969) 509-521. [16] M.J.D. Powell, "A method for nonlinear constraints in minimization problems", in : R. Fletcher, ed., Optimization (Academic Press, New York, 1969) pp. 283-298 [17] B.N. Pschenishnyi, "Dual methods in extremum problems", Kibernetica I (3) (1965) 89-95. [18] R.T. Rockafr "The multiplier method of Hestenes and Powell applied to convex programming", Journal of Optimization Theory and Applications 12 (6) (1973). [19] N.Z. Shot, "On the structure of algorithms for the numerical solution of planning and design problems", Dissertation, Kiev (1964). ['20] P. Wolfe, "A method of conjugate subgradients for minimizing nondifferentiable functions", Mathematical Programming Study 3 (1975) 145-173 (this volume).

Mathematical Programming Study 3 (1975) 26-34. North-Holland Publishing Company

O N I M P R O V I N G R E L A X A T I O N M E T H O D S BY M O D I F I E D GRADIENT TECHNIQUES*

P.M. CAMERINI, L. F R A T T A and F. M A F F I O L I lstituto di Elettrotecnica ed Elettronica, Politecnico di Milano, Milano, Italy

Received 14 November 1974 Revised manuscript received 4 August 1975

Relaxation methods have been recently shown to be very effectivefor some large scale linear problems. The aim of this paper is to show that these procedures can be considerably improved by following a modified gradient step direction.

1. Introduction Given the constants Ck ~ R, #k e R" (k = 1..... KL n and K being two positive integers, and a variable vector ~ ~ R", find V. 1

maxw(n) = rnkin {c ~ + ~ . / ~ ) ~--- Ck(~) "4- 7C" ~ktn ).

It has been shown [9, 16] that whenever K is large and an efficient m e t h o d exists to find w(Tr) for each value of ~ (this happens for instance for some combinatorial optimization problems [1,9, 16]) a m e t h o d related to gradient techniques provides a very effective attack on P. 1. Specifically one can a d o p t the following iterative scheme:

{~

o

-----0,

~m + 1 = 7,cm "Jr tm Sin,

(1)

{ t.,} being a sequence of scalars, and sm the gradient of worm) such that s" = Vw(Tr") = #k(,..J.

(2)

* Partially supported by the Centro di Telecomunicazioni Spaziali of CNR (Italy). A provisional version of this work was presented at the International Conference on Operation Research (Eger, August 1974).

P.M. Camerini et al. / Improving relaxation methods by gradient techniques

27

Since k(n) is in general a multivalued function, there exist some points where the gradient is undefined. At such points a set of subgradients can be defined and any of them can be used as s" [16]. (Note that this iteration scheme is not applied as in conventional gradient procedures, but rather, as in relaxation methods, in order to come closer and closer to the optimal region; the objective function need not be improved at each step.) Computational experience with some large scale problems (assignment, traveling salesman, multicommodity max flow) [1, 9, 10, 16] has shown that the above relaxation method works better than classical gradient procedures or column generation techniques [7, 15]. The aim of this paper is to prove that the efficiency of the relaxation technique is further improved by selecting the modified gradient direction s" = ~ ' + / ~ . s m- 1,

(3)

where #m = #kand tim is a suitable scalar (sm- 1 = 0 for m = 0). Note that (3) is in fact equivalent to a weighted sum of all preceding gradient directions, which has been successfully used by Crowder [5] in order to avoid some possible troublesome effects due to the "subgradient's alternating components". In [5], two kinds of modified gradient directions are proposed and, although their general properties have not been investigated, computational experience shows their advantage over use of the simple gradient direction. In the following section we give a rationale for choosing the iteration scheme (1) and (3), leading to a policy for determining tim and t m at each step. We also show that the properties given in [9] for (2) can be extended to (3) and that (3) is always at least as good a direction as (2). In Section 3, some computational experiments are discussed which confirm the theoretical results.

2. Properties of modified gradient directions

Lemma 1. [9]. L e t ~z and n m be such that ~ = w(~z) >>. w(rc'~) = wm. Then (~ - n"). #m I> g, _ wm i> 0.

(4)

Let, f o r all m, ~V--Wm

0 < t.
10.

(5)

28


Then (~ -

n').

s '~ > / ( n

-

(6)

re"). ~u=

for all m. Theorem 1. Let S,n- 1 .pm

-7./is._,112

fl1r

/ys--I

0

- 1

(9)

I1 '1t

Theorem 2.

(10) Proofs. Lemma 2, and Theorems 1 and 2 are proved in the Appendix. Lemma 1 guarantees that the direction of/~" forms an acute angle with the direction leading from rg" to the optimum ~, while Lemma 2 extends this property to sm. Theorem 1 shows that by a proper choice of tim, s" is always at least as good a direction as/~". Fig. 1 attempts to illustrate such a behaviour in a two-dimensional case. K

"~m =sm

Sm,, p:O

rim

P~ S ~-1

I

1~r n ' l

l~rn- 1

Fig. 1.

r

'

~

m IBm> O

P.M. Camerini et aL / Improving relaxation methods by 9radient techniques

29

Theorem 2 guarantees that a point closer and closer to the optimum is obtained at each iteration, and that the following convergence property holds. If (5) holds and, for some e > 0,

t,,

--

W m

rorallm,

(ll)

where ~, < max,, w(r0, then the sequence ( ~ } either includes a point n I e Pw or converges to a point on the boundary of Pw, where Pw denotes the polyhedron of feasible solutions to ~ ~< ck + n" p~ for all k. In fact it can be shown exactly as in [-9] and [14] that {n m} is F6jer-monotone relative to Pw and hence converges. From (1), (5) and (11) it follows that if no n~ in {n"} exists such that rrt e Pw, then w (lim,._~~ n ' ) = ~ and the limit point is on the boundary of Pw. Similar results were proved in [9] for the iteration scheme (1) and (2) with a condition on t., less restrictive than (5), namely 2(~, - w,,,)

o < t.
0, i.e., from (1) tim+ 1(~ - rim+ 1). s" >i 0. Then Lemma 2 follows from (16), Lemma 1 and (3).

Proof of Theorem 1. The proof is trivial when ~m = 0. When/~, > 0,

IIs-II 2 - II~," II2 = ~. IIs - - ills + 2flm(S'-l" #m) 0, as

II~ - ~112 + t~ IIs-II s - 2tm(n - ~m)" sm < I1~ - ~112.

(16)

34


References [1] P.M. Camerini, L. Fratta and F. Maffioli, "A heuristically guided algorithm for the traveling salesman problem", Journal of the Institution of Computer Science 4 (1973) 31-35. [2] P.M. Camerini and F. Maffioli, "Bounds for 3-matroid intersection problems", Information Processing Letters 3 (1975) 81-83. [3] P.M. Camerini, L. Fratta and F. Maffioli, "Traveling salesman problem: heuristically guided search and modified gradient techniques", to appear. [4] G.A. Croes, "A method for solving traveling salesman problems", Operations Research 6 (1958) 791-812. [5] H. Crowder, "Computational improvements for subgradient optimization", IBM Research Rept. RC 4907 (No. 21841). Thomas J. Watson Research Center (June 1974). [6] G.B. Dantzig, D.R. Fulkerson and S.M. Johnson, "Solution of a large scale traveling salesman problem", Operations Research 2 (1954) 393-410. 1-7] J.B. Dantzig, Linear programming and extensions (Princeton University Press, Princeton, N.J., 1963) ch. 23. [8] M. Held and R.M. Karp, "A dynamic programming approach to sequencing problems", SIAM Journal on Applied Mathematics l0 (1962) 195-210. [9] M. Held and R.M. Karp, "The traveling salesman problem and minimum spanning trees: part II", Mathematical Programming 1 (1971) 6-25. [10] M. Held, R.M. Karp and P. Wolfe, "Large scale optimization and the relaxation methods", in: Proceedings of the 25th Conference of the ACM, August 1972, pp. 507-509. 1-11] L.L. Karg and G.L. Thompson, "A heuristic approach to solving traveling salesman problems", Management Science 10 (1964) 225-248. [12] S. Lin, "Computer solution of the traveling salesman problem", The Bell System Technical Journal 44 (1965) 2245-2269. El 3] F. Maffioli, "Shortest spanning hypertrees", in: Symposium on optimization problems in engineering and economics, Naples, December 1974. [14] T. Motzkin and I.J. Schoenberg, "The relaxation method for linear inequalities", Canadian Journal of Mathematics 6 (1954) 393-404. [15] J.F. Shapiro, "A decomposition algorithm for integer programming problems with many columns", in: Proceedings of the 25th Conference of the ACM, August 1972, pp. 528-533. [16] P. Wolfe, M. Held and H. Crowder, "Validation of subgradient optimization", Mathematical Programming 6 (1974) 62-88.


THE MINIMIZATION OF CERTAIN NONDIFFERENTIABLE SUMS OF EIGENVALUES OF SYMMETRIC MATRICES

Jane CULLUM, W.E. DONATH and P. WOLFE IBM Thomas J. Watson Research Center, Yorktown Heights, New York, U.S.A.

Received Revised manuscript received 28 April 1975 Properties of the sum of the q algebraically largest eigenvalues of any real symmetric matrix as a function of the diagonal entries of the matrix are derived. Such a sum is convex but not necessarily everywhere differentiable. A convergent procedure is presented for determining a minimizingpoint of any such sum subject to the condition that the trace of the matrix is held constant. An implementation of this procedure is described and numerical results are included. Minimization problems of this kind arose in graph partitioning studies [8]. Use of existing procedures for minimizing required either a strategy for selecting, at each stage, a direction of search from the subdifferential and an appropriate step along the direction chosen [-10,13] or computationally feasible characterizations of certain enlargements of subdifferentials [1,6] neither of which could be easily determined for the given problem. The arguments use results from eigenelement analysis and from optimization theory.

I. Introduction

This paper is concerned with the problem of minimizing a certain convex but not necessarily differentiable function, the sum of the q largest eigenvalues of a real symmetric matrix as a function of the diagonal entries of the matrix, constrained only by the requirement that the sum of these entries is constant. Use of existing procedures for the minimization of a general convex function requires either a strategy for selecting directions and steps at each stage [10, 13] or computationally tractable characterizations of certain enlargements of subdifferentials [1, 6], neither of which can be easily determined for this particular function. The analog for convex functions of the method of steepest descent is known not to work in general.

36

J. Cullum et al. / Minimization of nondifferentiable sums of eigenvalues

The practical origin of this problem is sketched in the next section. The simplest problem of this kind (and not a practical one) comes from taking the entries of the matrix all zero. The function to be minimized is then just the sum of the q largest components of the vector d = (dl ..... d~) of diagonal entries, constrained by ~7= t dj = 0. The unique answer is d = 0, a point at which the function is not differentiable. In general, we must cope not only with the likelihood of nondifferentiability at a minimizing point, but also with the fact that our highly nonlinear objective function has no simple analytical expression. Fortunately, the objective function is convex, so that we can use, as in [1] and E6], some of the machinery of "convex analysis". After specifying notation in Section 3, we develop in Section 4 the essential properties of the objective function and find a computationally tractable description of its subdifferential. Section 5 is devoted to the algorithm itself. As discussed there, the procedure is motivated by the method of steepest descent; its outstanding feature is its use of computationally tractable enlarged subdifferentials to yield a convergent and implementable procedure. The proof of convergence is given in Section 6. Section 7 is devoted to a more detailed discussion of the computational steps, while Section 8 gives some results from using the algorithm on a problem of reasonable test size. This report is a condensed version of another [3]; we have, for conciseness, omitted a discussion of the possible failure of steepest descent on a general convex function, some details on the implementation of the algorithm, and a large number of test results. As presented in Section 5 the minimization procedure is applicable only to the specific class of problems described. However, the basic idea in this algorithm of adaptively modifying the subdifferentials to anticipate any nondifferentiability has general application.

2. Origin of the problem The problem studied here--that of minimizing the sum of the q algebraically largest eigenvalues of a symmetric matrix, as a function of the diagonal entries--arose in a problem of graph theory: partition the n vertices of a given graph into q sets of specified cardinalities ml < ... < mq in such a way as to minimize the total number of edges connecting different sets. In problems of practical interest, lsuch as the packaging of computer circuitry, the graph may have as many as 2 000 vertices, while 2 < q < 20.

J. Cullum et al. / Minimization of nondifferentiable sums of eioenvalues

37

A procedure due to Donath and Hoffman [7] for obtaining a suboptimal partitioning uses the eigenelements of a matrix A + D* that achieves the following minimization. - 89 i=

mi2i(A+D): D diagonal, Tr(D) =

.

(2.1)

In [8] Donath and Hoffman showed that this minimum is a lower bound on the minimum number of connecting edges in all such partitions. In (2.1), the n-order matrix A is the incidence matrix of the given graph excepting along the diagonal, where Au = - ~ i , ~ Aii, so it is real, symmetric and usually sparse; 2~(A + D) is the ith eigenvalue of A + D with the ordering 2~(A + D) > ... > 2,(A + D) and Tr(D) is the trace of the matrix D. In the sequel we assume that m~ is independent of i, in which case the problem posed in (2.1) can be written

min{~2~(A+D):Tr(D)=O, Ddiagonal}.i=

(2.2)

This corresponds to a partition into q groups of equal cardinality. We note that the only property of A required below is its symmetry; its provenance in a graph problem has no bearing on the minimization procedure. The sparsity of A makes the eigenvalue, eigenspace computations required by the algorithm feasible for large values of n.

3. Notation and definitions Given any d = (dl .... , d,)~ E", D is the corresponding diagonal matrix of order n with Du = d~ 1 < i < n. A is a fixed real, symmetric matrix of order n. The quantities 2j(A + D), 1 < j < n are the n eigenvalues of A + D arranged in descending order: 21(A + D) > 22(A + D) > ... > 2,(A + D). The objective function is q

f(d)= j =~ l " 2j(A + O).

(3.1)

Given the ordering of the eigenvalues of A + D above, we define the integer r(e) by supposing 21(A + O), ..., 2q_ a-r~)(A + D) to be all of those eigenvalues greater than 2q + e, and define

38


Yl(d, e) = {Yl ..... Yq -1 -rtO} as a set of corresponding eigenvectors. The integer s(e) is then defined by supposing that Aq-r~(A + D) ..... 2q+~lo(A + D) comprise all the eigenvalues lying in the closed interval + o ) - e,

q(a + D) + el,

and Y2(d, e) = {Yq-r (u, Ad> for all Ad~E"}. (3.3) (a, b> denotes the inner product of the two vectors a and b. Any u ~ Of(d) is subgradient o f f The directional derivative o f f at d in the direction w, f'(d, w) = lim [(f(d + t w) - f(d))/t]. ttO +

(3.4)

A finite-valued convex function on E" is continuous and has a directional derivative at each point in each direction. Moreover, for each d, Of(d) is nonempty, convex, compact and for each w f'(d, w) = max <w, u>: u ~ Of(d).

(3.5)

Relation (3.5) expresses the directional derivative in terms of the support function of the subdifferential and is used extensively in the development of the minimization procedure.

J. Cullum et aL / Minimization of nondifferentiable sums of eigenvalues

39

For any set G, conv G is the convex hull of G. For any matrix E, Tr(E) is the trace of E. For any vector z, z r is the transpose of z, and Ilzll is the Euclidean norm of z. e will always denote the vector (1, 1, ..., 1).

4. Properties of the objective function In this section we develop some of the properties of the function f in (3.1) that we want to minimize. Recall that q < n.

Theorem 4.1. ( i ) f i s bounded below on the set C = {d: (d, e> = 0}, (ii)The intersection of C with any level set {d: f(d) < oe} o f f is bounded, (iii) f assumes its minimum value on C. Proof. (i) Suppose that, on the contrary, f ( d k ) ~ - ov for a sequence {dk} c C. Then 2 i ( A + D k ) ~ - - G o for i > q , and so T r ( A + D k ) = ~7= 1 2i(A + Dk) ~ -- oo, a contradiction since Tr(A + Dk) = Tr(A) on C. (ii) Since f is bounded on the intersection and Tr(A + D) is constant there, ~7>q 2i(A + D), and hence each eigenvalue 2~(A + D) is bounded on the intersection. Let Ck = A + Dk, then by the Hoffman-Wielandt theorem [17], i=1

i=1

Thus, since the 2i(Ck) are uniformly bounded for all dk ~ C, so are the 2i(Dk) which range over all components of vectors dk ~ C. (iii) This is an immediate consequence of (ii) and the continuity off. Lemma 4.2 was proved in [8] using inequalities from matrix theory. Lemma 4.2 [8]. The function f is convex. Proof. Ky Fan [11] has shown that for each d, f(d) = max

(xj, (A + D ) x j ) : X = (xl .... ,Xq) ,

(4.1)

J

where X is any set of orthonormal vectors. For fixed X the summand in (4.1) is a linear function of d, s o f i s the pointwise supremum of a family of convex functions and hence is convex [15].


40

For the rest of this section the vector d and the corresponding diagonal matrix D will be fixed, as will the associated eigenvalues 2j = 2j(A + D), 1 < j < n. Y = {Yl ..... y,} is any corresponding orthonormal basis of eigenvectors of A + D. For each such Y, Y1 = Y~(d, 0) is the set of eigenvectors in Y corresponding to eigenvalues exceeding 2q--see Section 3 - and Y2 = Y2(d, 0) is the set of eigenvectors in Y corresponding to 2q. Let r and s denote respectively the interior and exterior multiplicities of 2q. Lemma 4.3. The maximand of(4.1), q

, j=l

attains its maximum value on a set X = { x l . . . . . xq} of orthonormal vectors if and only if

X ___ sp{Y,, Y2}

and

sp{Y,} ___ sp{X}.

Proof. Since Y is a set of orthonormal vectors, for each 1 < k < q, XR = ~,3= 1 <XR, yj> yj. Let trj = ~ = 1 <Xk, yj>2. Then )-', (Xk, (A + D) x , ) = )-" k=l

k=l

2 i <x,, yj)2 + 2q q j=l

q

<Xk, yj)2 k=l

j=l

q

= E &k+ E ( - - 2 d + 2 q ) [ 1 - - t r j ] k=l

j=l

+ j=q+l

k=l

Since each term in the j sums is nonpositive, equality holds if and only if each term vanishes. Thus, f o r j > q + s and 1 < k < q, (Xk, Y~> ---- 0, SO each Xk~Sp{Y1, Y2}. Similarly, for e a c h j < q - r - 1, trj = ~ = 1 (Xk, Yj> 2 = 1 so each such y j ~ s p { X } . N o w let M be the family of all 'maximizing' sets of orthonormal vectors X = {xl ..... xq} described by Lemma 4.3. Define G(d) = { u ~ E": u = T ( X ) for some X ~ M }.

(4.2)

Lemma 4.4. Of(d) = c o n v G(d).

Proof. By a theorem of Danskin [5], the directional derivative o f f at any point d in any direction w is given by

J. Cullum et al. / Minimization o f nondifferentiable sums o f eiflenvalues

f'(d, w) = max {(u, w>: u e G(d)}.

41

(4.3)

By (3.5) and (4.3), f'(d, w) is the support function of the set G(d) and of the convex set Of(d). Therefore, Of(d) = conv G(d). L e m m a 4.5 expresses G(d) in terms of any orthonormal set of eigenvectors Y = {Yl ..... y,} of A + D. Lenuna 4.5. For any orthonormal set of eigenvectors Y = {Yl ..... y,} of

A+D, f.lr+ 1 G(d) = {u: u = T(Yt(d,O)) + T(Y2(d,O)H)for some ,,l-I e--r+s+l}.

(4.4)

Recall that, Yl(d, 0 ) = {Yl, ..-,Yq-,-1} and Y2(d, 0 ) = {y~_~..... y~+~}, where r and s are the interior and exterior multiplicities of 2q(A + D), and H ,r §+ s + l is the set of all (r + s + 1) x (r + 1) matrices with orthonormal columns. Proof. Clearly, for any H e ,t-t,+ , , + s +1 l the set of vectors { Yl(d, 0), Y2(d, 0)H} is orthonormal and by Lemma 4.3 is in M. Conversely, let u e G(d) in (4.2) be generated by X = {Xl .... , xq} e M. Since sp{ Y1, Y2} - sp{X} _ sp{ }'1 }, there is a matrix Ve Hg such that X V = { Yl(d, 0), Z} witlf'sp{ Z} _c sp{ Y2} ~.irr + 1 and the columns of Z orthonormal. Therefore, there exists an H e ,,,+~+ 1 such that Z = Y2(d, O)H. Observe that the components of u, ui = ~ = 1 x2~, 1 < i < n arethe diagonal entries of the matrix X X T. Since V is orthogonal, X V v T x T = X X T. Therefore, X and X V generate the same vector. That is, every u e G(d) has a representation T(YI(d, 0)) + T(Y2(d, O)H) for some H = ~tJtr+s+ t-/r+ 1 1" Lemmas 4.2 to 4.5 yield the following characterization of the differentiability off. Theorem4.6. For any

real symmetric matrix A, the function f = ~,~= 1 21(A + D) is differentiable except at those points d such that 2q(A + D) = 2q+ I(A + D).

ProoL By Lemma 4.4, f is differentiable at a point d if and only if G(d) contains exactly one vector. Let u e G(d). By Lemma 4.5, for any Y there is t_tr + 1 1 such that some matrix n e --,+s+

u -- T(Y~(d, 0)) + T(Y2(d, O)H).

42

J. Cullum et al. / Minimization of nondifferentiabte sums of eigenvalues

Casea. If2q+l(A + D) ~ )~q(A + D), then s = 0, r + 1 = r + s + 1 and H is an orthogonal matrix. But then, as in the proof of Lemma 4.5, the sets { Yl(d, 0), Y2(d,0)} and { Y~(d, 0), Y2(d, 0)H} generate the same vector. Thus, any u = T(YI(d, 0)) + T(Y2(d, 0)). So G(d) consists of the single vector, q

ui = ~ y f,

1 0 and for any /,,4 ~--r+~+ 1, sp{ Y2(d, 0)} #- sp{ Y2(d, 0)H}. In particular, G(d) contains all the vectors. q-1

ui = ~ yj2 + (yqicosO + y~q+a)isinO)2,

1 _ O. Proof. By (3.4),

f'(d~, w~) = lim [f(doo + t woo) - f(d~)]/t. t~O +

But for any t ~ [0, T], f ( d ~ + t w~) = limk f(dk + t Wk) > limk f(dk+ 1) = f(d~), since f is continuous. The result is now clear. The following L e m m a is in Wilkinson [17]. Lemma 6.2. Let the sequence (dR}, k E K have the limit doo, and let ~ be any eigenvector for 2 ~(A + DR~ 1 O, any u ~ P ~f(doo) is the limit of a sequence {Uk} with Uke P S(dk, e). Proof. By L e m m a 4.4, for any d, P df(d) = P conv G(d) = c o n v P G(d). Let r and s be the interior and exterior multiplicities of 2q(A + D~). Let yk be an ordered orthonormal basis of eigenvectors of A + Dk. Let yk(oo) denote the first q - r - 1 vectors in yk and y2k(oo)denote the next r + s + 1 vectors. By L e m m a 6.2 there exists a subsequence K' _ K and a set of vectors {Y1, Y2} such that for k e K ' , {Yk(oO)} ~ Y1, {Yff(~)} ~ Y2, and { Y1, II2} is an orthonormal set of eigenvectors of A + D~ corresponding to 2j{A + O~o), 1 < j < q + s. + 1 By L e m m a 4.5, # ~ P G(d~o) if and only if for some .~./. . .~. T-,/r,+s+ 1,

g = T(Y1) + T(Y2H) - e q/n where e = (1, 1,..., 1). Clearly, {Ok} -o #, k e K' for

gk = T(yk(~176 + T(yk(~

H) -- e q/n.

But, by Lemma 6.2, for any e > 0, for large k, Yl(dk, e) contains at most q -- r -- 1 vectors and Y2(dk, e) contains at least r + s + 1 vectors. Therefore, for large k the set

P Gk(OO) = ( U : U = T(yk(oo)) + T(yk(oo)H) - e q/n for some H e/-/~, ++1s+ 1}

(6.1)

is contained in P G(dk, e). The desired result follows immediately since any


46

u ~ P Of(d~) has the representation n+l

n+l

u= Y~#,~j, #j->0, j=l

~#j=l,

gj ~ P G(d~).

(6.2)

j=l

T h e o r e m 6.4 is the converse of T h e o r e m 6.3. Together these theorems provide the weak continuity arguments needed in the p r o o f of convergence of the minimization procedure. Theorem 6.4. For k ~ K let {dk} ~ d~o, Uk 6 P S(dk, e), {Uk} --* U~o. I f e < e* = m i n { [ 2 / -

)],j[" "~i ~- "~j, eigenvalues

o f A + Do},

then u~ ~ P 3 f(d~o). Proof. Each Uk = ~"~+=l #ktgkb where gkt e P G(dk, e), #k, >-- O, ~,"Z+l #k, = 1. By compactness there exists a subsequence K ' _~ K such that for 1 _< l _< n + 1 and k e K'{Pkt} --' #t and {gkt} ~ g~. Clearly, it is sufficient to prove that each g~ e P Of(doo), since uo~ = ~_-+ 1 #tgl. Let r and s be the interior and exterior multiplicities of 2~(A + Do~). By L e m m a 6.2, since e < e* for large k, Yl(dk, e) has q - r - 1 m e m b e r s and Y2(dk, e) has r + s + 1 members. By compactness and L e m m a 6.2, there exists a further subsequence K" ~_ K' such that Yl(dk, e)--, I"1 and Y2(dk, e)--, II2, where Y1 is an o r t h o n o r m a l basis for the eigenspace of A + Doo corresponding to 2j(A + Doo), 1 _ j ___ q - r - 1 and similarly I12 corresponds to 2~(A + D~o) = I~(A + D), q - r _< j _< q + s. By L e m m a 4.5, for each 1 -< l < n + 1 and k ~ K", gkt = r(Yl(dk, e)) + T(Y2(dk, e)Hu) u , + 1 . 1. But there exists K"' c K" such that for each e q/n for s o m e Hkl e--r+s+ _ l _< n + 1, Hkl--* ,u, t e n- t ~,r+ Therefore, {gu} --* gl = T(YO + T(Y2H,) 1< + s + 1 1.

= e q/n. By L e m m a 4.5, gt ~ P G(d~), 1 < I < n + 1. Therefore, uoo ~ P Of(d~o). T h e o r e m s 6.3 and 6.4 are used to prove the following convergence theorem.

Theorem 6.5. (i) The sequence {dR} of iterates generated by the sum of eigenvalues algorithm is bounded. (ii) f(dk), k = 1, 2 . . . . converges to the minimum value o f f on the space (e, d ) = 0. (iii) Any limit point do~ of the iterates is a minimizing point o f f on (e, d ) = 0.

J. Cullum et al. / Minimization o f nondifferentiable sums o f eigenvalues

47

Proof. The boundedness of the iterates is a consequence of Theorem 4.1. (iii) is an immediate consequence of (ii) and the continuity of f, so consider (ii). We first show that {sk} ~ 0. Otherwise, for large k, trk< - 5 k = - e #- 0. Let d* be any limit point of the iterates. Let K' =__K and w* be such that for k e K ' , {dk} ~ d* and {Wk} --" W*, where Wk is the direction of search used at step k. Since the sets P S(dk, 5) are uniformly bounded, for some R and large k ~ K', max (u, u~P S(dk, e)

w*)
--ek. Thus, for some subsequence K" _ K' and some point d*, ak ~ 6 --> 0 and dk ~ d*, k ~ K". Now, we need only to prove that for any direction w with (e, w) = 0, f'(d*, w) >__O. By construction, for any w with Ilwll = 1, ~k__ O"k. Choose K"' ___ K" such that gk ---)"g*" Since 5k --~ 0 for large k, gk ~ P S(dk, 5*), where 5" -- m i n { [ 2 i - )~)[: 2i ~- 2), 2i eigenvalues of A + D* }. So by Theorem 6.4, g* ~ P Of(d*). Therefore, for any (w, e) = 0, f'(d*, w) >_ (g*, w) >_ 0, so d* is a minimizing point of f on (d, e) = 0. Since f is continuous, and f(dk), k ~ K is m o n o t o n e decreasing, f ( d k ) ~ f(d*), the minimum value of f on (e, d) = 0. Therefore, any limit point of iterates is a minimizing point of f

48

J. Cullum et aL / Minimization of nondifferentiable sums of eigenvalues

7. Implementation of the algorithm Use of the algorithm requires the repeated computation of the q algebraically largest eigenvalues and associated sets of eigenvectors of real symmetric matrices. In the applications described in [-7] these matrices are large (n > 500) but very sparse, approximately 5 nonzero entries in each row and column. Because of the large amount of storage space required by similarity methods only eigenelement procedures, such as simultaneous iterations [16], that do not explicitly modify the given matrix can be used. For the matrices in [,7], the algebraically largest eigenvalues are usually not largest in magnitude and the matrices are not usually definite. A typical eigenvalue configuration is 0.01, -0.015, - 0 . 0 2 ..... - 1 0 , and one expects to encounter multiple eigenvalues near a minimizing point. Application of a simultaneous iteration scheme to such a matrix requires heuristic matrix shifting (A ~ A - aI, a a scalar) to achieve magnitude dominance of the desired eigenvalues, and there is no guarantee of convergence to the desired eigenelements. Single vector Lanczos procedures [-14] that do not require dominance are available but other types of difficulties such as losses of orthogonality between the vectors generated arise. A block Lanczos algorithm suggested by W. Kahan was developed for the eigenelement computations. The use of blocks alleviates the problems of orthogonality associated with the single vector Lanczos schemes and allows multiple eigenvalues to be handled easily. Since it is a Lanczos procedure the desired eigenvalues need not be dominant. The sparsity of the matrices was used to make the matrix-vector multiplications required by the method very efficient. At each iterate dk the direction of search Wk is a solution of the following min-max problem: min

max

Ilwll-< 1 u~PS(dk,~k)

(u, w).

To simplify the notation, let P S(k) = P S(dk, ek) and P G(k) = P G(dk, 8k). By [-1], wk = -z /llz ll, where z~ is the element in P S ( k ) c l o s e s t to the origin. By [,12] this element can be determined computationally if contact points of P S(k) can be computed. A point u* ~ P S(k) is a contact point of P S(k) corresponding to the direction w, if the plane through u* with normal w is a supporting hyperplane of P S(k). Algebraically, (u*, w) = max (u,w). ueP S(k)

(7.1)

The following lemma demonstrates the ease with which contact points of


49

P S(k) can be computed. Fix k, let Y = {Yl ..... y,} be any ordered orthonormal basis of eigenvectors for the matrix A + Dk. Let r(k) and s(k) denote, respectively, the interior and exterior ek-multiplicities of ~,q(A + Dk). Let t = q - r(k) - 1 a n d m = r ( k ) + s ( k ) + 1. Lemma 7.1. For any direction w, a contact point u of P S(dk, ek) correspondin9 to w can be computed from any ordered set Z = {zl ..... Zr(k)+1} of orthonormal eioenvectors of the symmetric m • m matrix M, where

Mkj = ~ Y(j+t)iYtk+t)iWi,

1 < k , j < m.

i=1

In fact, for any such set Z, the vector u = T(Yl(dk, ek)) + T(Y2(dk, ek)H) -- e q/n, where H is the matrix whose fh column is z i, is a contact point of P S(dk, ek) corresponding to the direction w. Proof. From the linearity in (7.1) and the convexity of P S(k), the maximum in (7.1) must occur on a generator u ~ P G(k). By construction, u ~ P G(k) only if for some H ~ H~ k) § 1,

u = T(Yx(dk, ek)) + T(Y2(dk, ek)n) -- e q/n. Let ~lo, 1 < l < m, 1 < j < r(k) + 1 be the entries in H. Then (u, w ) = ~ i=1

yj2 + j=l

E t=l

rhu-oY;,

-q/n

w,.

(7.2)

\j=q-rtk)

The only variables in (7.2) are the orthonormal vectors t/j, 1 < j < r(k) + 1, so only the second summation must be considered in the maximization of (7.1). A rearrangement of this sum yields the expression, ~, /=1

j,k=l

th~

Yu+t) iY(k+t)iWi

?hk

9

(7.3)

i=1

If we define M(w, y) to be the real, symmetric d x d matrix whose j, k element is

~ Y(j+t) iYtk+t) iWi,

(7.4)

i=1

then (7.3) can be written as r(k) + 1

Z 1=1

Qh, M (w, Y)n,).

(7.5)

50

J. Cullum et al. / Minimization of nondifferentiable sums o f eigenvalues

Lemma 4.3 is applicable with A replaced by M(w, y) and q by r(k) + 1. The maximum of (7.5)over all sets Z = {zl ..... Z,tk)§ of orthonormal vectors is assumed when Z is any set of vectors satisfying the conclusions of Lemma 4.3. For any such set Z = {r/1..... t/,~k)+1}, let H = (the), 1 _< I < m, 1 0} contains all optimal dual solutions. Thus, it is plausible to construct ascent algorithms for the dual problem of the form u k§ 1 = u k + Ok?k, where ?k is a subgradient of L at u k and Ok satisfies L(U k A- Ok7k) = max L(u k + o])k). 0~--0

f o r a l l t ~ T ( ~ ) } . (2.3)

Moreover, the directional derivative V L ( ~ ; v) of L at fi in the direction v is given by [22] V L ( ~ ; v) = min v 7'. t~ Tfff)

(2.4)

This last result establishes the existence of directional derivatives in all directions v ~ R m at all points ~ ~ R', as well as characterizing them. The following theorem, stated without proof, gives us the constructive mechanism for using the characterizations. In words, the theorem says that

M . L . F i s h e r et al, / Using d u a l i t y to solve discrete o p t i m i z a t i o n p r o b l e m s

61

a point ~ > 0 is optimal if and only if VL~; v) < 0 for all feasible directions v from ~. For any ~ > 0, let I(-fi) = {i: ~i

=

O}

and let IC(~) denote the complementary set. Theorem

2.1. The point ~ >_ 0 is optimal in the dual problem (1.2) /f and

only if

max VL(fi; v) = O, vEV

where V c_ R " is the set of points satisfying -1

O 0 ,

i = 1,...,m.

Let v*, v* = a* denote an optimal solution to (2.5), and let 2* denote an optimal solution to (2.6). Theorem 2.2 below gives necessary and sufficient conditions that ~ > 0 is optimal in the dual in terms of the dual linear programs (2.5) and (2.6). The proof of theorem 2.2 is self contained and does not rely on the charac-

62

M.L. Fisher et al. / Using duality to solve discrete optimization problems

terizations (2.3) and (2.4) except for the obvious fact that a convex combination of subgradients is a subgradient. The difficult part of (2.3) is to establish that all subgradients can be generated as convex combinations of the 7 t, t ~ T(-fi). Theorem 2.2. The dual solution ~ > 0 is optimal in the dual problem (1.2) (equivalently (2.1)) if and only if a* = 0 in the dual L P problems (2.5) and (2.6). Proof. Sufficiency: If a* = 0 in (2.6), then the subgradient t~ T(~)

satisfies 7* < 0, i z I(fi), ~* = 0, ir lCC~). Thus, for all u > 0,

L(u) < LCu) + (u - -~)~* = LCu) + ~ (u, - -fii)?*

~

= L(fi) +

i~lC~)

uiy* < L(fi).

i~I(~

The-first inequality follows because 7" is a subgradient, the first equality because ~* = 0, i ~ IC(fi), the second equality because ui = 0, i 6 I(~) and the second inequality because ~* < 0, i~ I(-fi) and ui > 0. This establishes sufficiency. Necessity: We prove necessity by showing that a* > 0 in (2.5) implies is not optimal. If a* > 0, then v*7 t > 0 for all t 6 T(fi) and therefore v* ~ 0. Consider now the problem max LCu + Or*). 0>/0

If the maximum of L(u) along the half-line fi + Or* for 0 -> 0 is greater than L(fi), or if L(u) is u n b o u n d e d along the half-line, then clearly fi is not optimal in the dual problem and the theorem is proven. Suppose then that for all 0 _> 0,

LC~ + Or*) < LC~).

(2.7)

We will show a contradiction. For 0 > 0, we have L(~ + Or*) = rain f ( x 9 + -~ 7t + Ov*~,', t=l

.....

T

= min f ( x 9 + -u 7t + Ov*~'~, tr T(~)

where the second equality follows from (2.7) and the fact that v*y t > 0 for all t ~ T~). Consider a sequence {0 k}k=l ~ such that Ok --* 0 +. Since the


63

number of x t is finite, we can select 01 sufficiently small that there will be an x s, s r T(fi), satisfying for all k,

Lffi + Okv:#) = f ( x s) + ~ 7s + Okv*7s.

(2.8)

Taking limits on both sides of (2.8) and using the continuity of L, we have L~) = f ( x s) + fi7 s or s ~ T(-fi), a contradiction. This proves necessity and the theorem. Suppose fi is optimal in the dual problem (2.2). It can easily be shown that 2 ~ X satisfying g ( 2 ) < b is optimal in the primal problem (1.1) if L(fi) = L(~, 2) and fi(g(2) - b) = 0. There is no guarantee, however, that such an ~ exists. Even if one does, it might still be quite difficult to find it since the number ofx ~ X satisfying L(~) = L(fi, x) may be large. In Section 6, we show how the dual problem can be used in a "fail-safe" manner as an analytic tool in a general branch and bound algorithm. The problems (2.5) and (2.6) are the ones to use to solve the dual problem in the linear programming form (2.1) or (2.2) using the primal-dual simplex algorithm. The set T~) can be extremely large, however, and it should be generated as needed. Details are given in the next section. There are some additional observations to be made about the dual problem. First, it is possible to compute an upper bound on the optimal dual objective function value w(b). The following result, stated without proof, will be used in the branch and bound algorithm in Section 6. Corollary 2.3. Consider the points x 1, ..., x K ~ X and suppose there exist non-negative weights 21 .... , ~,r satisfying ~ = 1 2k = 1 and ~kr= 1 g(xR) 2k -< b, then K

w(b) w'(b)). There is, of course, an increased likelihood that v(b) > w'(b~ or more generally that v(b) - w'(b) > v(b) - w(b) > O, but the larger duality gap may be compensated by the shorter time to calculate w'(b). Reference [I2] describes an application, where this phenomenon occurs. A more complex form of relaxation, called data relaxation, has been found for integer programming [20]. This type of relaxation consists of a change in the functions O~(x) - b~ so that subsequent Lagrangean minimizations can be more efficiently accomplished. We remark that it is possible to use the duality theory developed here to try to minimize or reduce the error introduced by the data change (see [20, ~4]).

3. Primal-dual ascent algorithm for dual problem In this section we give a primal-dual ascent algorithm for solving the dual problem in the LP form (2.2). The algorithm we propose is an adaptation of the primal-dual simplex algorithm applied to (2.2) where the large number of points in X are handled implicitly. The algorithm may also be interpreted as an ascent algorithm in which the successive restricted primal problems solved in the primal-dual for a given u value generate a subgradient which may be used in a manner similar to the gradient in differentiable ascent algorithms to determine a direction of ascent. Our algorithm bears a close resemblance to the algorithm of Grinold in [23] for large scale linear programming and also to the algorithm of Bradley in [3]. It was demonstrated in Theorem 2.2 that the point fi > 0 is optimal in (2.1) if and only if there exists a convex combination y* of the subgradients ~t, t ~ T(~), such that 7" = 0 if ~ > 0 and 7~ --- 0 if ~ = 0. The algorithm proceeds by generating successive yt, t ~ T(-ff) until either fi is proven optimal in the dual by the test of Theorem 2.2 or a new ~ # ~ is found such that L~) > L(~). Thus, we consider a generic step of the algorithm beginning at a point > 0 and a set T'(~) ___ Tfa). Recall that I(-a) = {i: ~ = 0}. We solve the following LP problem which is simply problem (2.6) with a restricted number of columns: rtl

t r * = min )-" s/- + i= 1

subject to

~ IcT'('~)

~

s~/, (3.1)

i~tct~)

2,7 ~-s~- + s + = 0 ,

i = 1..... m,


65

2,=1, teT'(-~)

,~,>0,

teT'(fi),

si- > 0 ,

s+ > 0 ,

i = 1,...,m.

The dual of (3.1) is: a* = max v, subject to v < v ?',

- 1 ~vi 0 in {3.2c) could be adjusted to reflect differences in scaling on different rows. By Theorem 2.2, ~ is optimal in the dual if a* = 0. We describe a generic step of the algorithm for the case a* > 0. Notice from (3.1) and (3.2) that ?* = ~t~r'(~)2"? t is a subgradient such that by complementary slackness v*~* = a* > 0 and an increase in the Lagrangean function is indicated in the direction v* namely, for 0 >- 0,

L ~ + Or*) 0. This interval is the one in which u remains non-negative. It is possible, of course, that V(h-)= 0 and 0max = + ~ . As long as w(b) < + ~ , 0,,ax = + ~ does not cause any real difficulty because we can easily devise a finite search of the half-line ~ + Or*, for 0 > 0, for the maximum of L along it. If w(b) = + ~ , then a finite search to discover this fact can be constructed because there are a finite number of pieces in the piecewise linear function L being maximized. In this regard, it is important to mention that when the primal-dual ascent algorithm is used in conjunction with branch and bound as described in Section 6, we can also impose the restriction 0max< (~ - LC~))/a*, where ~ is the incumbent cost

66

M.L. Fisher et al. / Usin 0 duality to solve discrete optimization problems

as defined in Section 6. In our discussion henceforth Of the primal-dual ascent algorithm, we assume 0max< + ~ . Let 0* ~ [0, 0m,~ denote the maximal value of 0 such that L(~ + Or*) = Lfu) + Oa*. The procedure for finding 0* is the following. Let 01 = 0max and at iteration k, compute L(fi + OkV*) = Lf6 + Ok?)* , X k)

for some x k ~ X.

Notice that L(-d + Ok?)*) = L(~, x k) + OkV*Yk. IfL(~ + OkV*) = LC6) + OkV*7*, then Ok = 0* and the search terminates. If L(~ + OkV*) < Lf6) + Okv*?*,

(3.4)

then compute L~, x k) - L ~ ) Ok+ 1 =

?)*7* - - ?)*~)k "

(3.5)

The denominator in this fraction is positive because of (3.4) and the fact that L(-6, x k) _> L~). If0k+ 1 = 0, then set 0* = 0 and the search terminates. If Ok+l > 0, the basic iteration is repeated. We show now that (i) the solution of the LP (3.1) defined at ~ + O'v* can begin with the optimal basis for (3.1) at fi, (ii) the value of the objective function at ~ + O'v* will be (barring degeneracy) less than a*. Finite convergence of the primal-dual algorithm is thereby assured by the usual simplex criterion (see [38; pp. 128-134]). There are three cases to consider. Case l: 0 < 0* < 0m~x- In this case, we want to show that the optimal basic variables from (3.1) at ~ remain in the problem at K + O'v* and with the same objective function coefficients. First, for t E T'~) such that 2t is in the optimal basis for (3.1), we have by complementary slackness (see (3.2)) that v*? t = a*. This implies that t ~ T(-6 + O'v*) since by construction, Lfd + O'v*) = L(-U-) + O'a* = L(~) + O'v*? t = L(~ + O'v*, xt).

To complete the argument, we need to consider the s7 and s + variables. They clearly remain in (3.1) at ~ + O'v* and can be in a starting feasible basis. We must simply show that the objective is unchanged for the s + that are basic. For i z i ~ we have that ICfu)~_ ICf~ + O'v*) because 0 " < 0m~x; the coefficient of s + for such an i thus remains equal to 1. Consider s + for i ~ I(-6). If s + is basic in (3.1), then by complementary slackness,


67

v* = 0 which implies i ~ I(~ + O'v*) and the objective function coefficient remains equal to zero. In short, we use the previously optimal basis as a starting feasible basis for (3.1) at ~ + O'v* and the objective function value equals tr*. To show that this value of a* can be reduced for (3.1) at fi + O'v*, note that 0* > 0 means we have found an x ~ X such that L~, x ~) > L ~ ) = L(~, x t) for 2, basic in the starting basis. By constructio n, L(~ + O'v*) = L(fi, x ~) + 0*v*7~ = L(fi) + O'a* and therefore v*y" < tr*. In other words, the LP reduced cost relative to the starting basis of the subgradient 7~ of L at fi + O'v* is v*7~ - a* < 0. The activity 7' should be pivoted into the starting basis and (barring degeneracy) the objective function will be decreased. Case 2: 0* = 0max. This case is the same as case 1 except for the possibility that for some i e Ic~), we have s~ basic and ui + O*vi = 0 implying i e I ~ + O'v). If this is so, t h e objective function coefficient of s~- needs to be changed from 1 to 0 which (again barring degeneracy) causes a reduction in the objective function. Problem (3.1) is then reoptimized. Case 3: 0* = 0. In this case, we have found an s ~ T(-~) such that L~, x ") = L~) and v'y* - v*?s > 0 (see (3.5)). Thus, v*~~ < tr*, we have s ~ T~) T'(~) and the non-basic 7~ can enter the basis causing (barring degeneracy) the objective function to be reduced. The primal-dual rule of moving to the next breakpoint of the L function in the direction v* ensures convergence of the ascent algorithm. It is also reasonable to move to the maximum of L in the direction v*, but then convergence cannot be established. This is a curious point which needs more study from both a theoretical and computational viewpoint (see also [-23, p. 452]).

4. Simplicial approximation algorithm for dual problem In this section, we present a method for approximating an optimal solution to the dual problem (1.2) by approximating the optimality conditions of Theorem 2.2 as expressed by problem (2.6) with a* = 0. The motivation for this approximation is the slow increase in the dual objective function value due to small ascent steps using the primal-dual algorithm on some problems. The simplicial approximation algorithm takes steps of fixed size which can be preselected. Since Scarf's original work in reference [36] on fixed point approxima-

68

M.L. Fisher et al. / Usino duality to solve discrete optimization problems

tion, there have been a number of important theoretical and computational extensions. For expositional reasons, however, we will ignore these extensions here. Our method draws on some of the construction of Hansen and Scarf for simplicial approximation in nonlinear programming [-25, pp. 14-18]. The major conceptual difference between our method and theirs is that we are approximating optimal dual solutions while they are approximating optimal primal solutions. Wagner [39] has also applied simplicial approximation to nonlinear programming. It is convenient to bound the region in which the simplicial approximation algorithm searches for an approximate solution by adding the constraint ~'=1 ui < U for U sufficiently large. Lemma 4.1 gives sufficient conditions under which a valid U can be computed. Lemma 4.1. L e t x k, k = 1, ..., m, be points in X such that gi(Xk) ~ b i f o r all i and gk(X k) < b k. Then without loss o f optimality the constraint ~ = 1 ui < U can be added to (4.1), where ~,

f ( x k) - Wo

v =

and w o is a lower bound on w(b) (e.g., w o = L(u) f o r some u > 0).

Proof. Let u*, w* = w(b) be any optimal solution in (2.1). We have u~(bk -- gk(Xk)) < ~ u*(bi -- gi(xk)) --< f ( x k) -- W* < f ( x k) -- WO, i=1 or

u~ < -

f ( x k) - w o bk -- gk(Xk) '

k=

1,...,m.

Summation over i gives us ~7'=, u* < U. Thus, the addition of the constraint ~T= ~ ui < U does not cut off any optimal solution of (2.1) and the lemma is proven. With the addition of the bounding constraint, the dual problem (2.1) becomes w(b) = max

subject to

w, w < - u b + f ( x ' ) + u g(x'),

~

i=1

ui < U,

u > 0.

(4.1) t = 1,..., T,


69

The necessary and sufficient condition for optimality of fi in (4.1) from Theorem 2.2 becomes a* = 0 in a*=min

~ st+

subject to

~

si+,

i= 1

i~IC(u~

~

2t~-si-+s~

(4.2) +-0=0,

i = 1,...,m,

t~ T ( ~

~

2,=1,

t~T(~)

2i > 0,

t e T(fi),

sT>O,

s+>0,

i=

1 . . . . . m,

where m

0=0

.if ~ f i i
0

if ~ R i =

U.

i=1

For technical reasons, it is convenient to add a z e r o th c o m p o n e n t to the u vector and consider our search to be over the simplex

This modification holds only for this section. In [36], Scarf selects points to form a set P = {u ~ u 1, ..., uK}, where u k ~ S, k > m and where u ~ = (0, Mo,..., Mo), u I = (M1, O, Mx,..., M1), u" = (M,, ..... M,,, 0) and Mo > M1 > ... > M,, > U. In addition, the uk are chosen so that u/k1 r u~2 for any i and any kl ~ k2, k~ > m and k2 > m. With the points uk so chosen, Scarf defines a primitive set u k~ . . . , u k" to be one such that there is no u k satisfying u~ >

min

(u~J),

i = 0 . . . . , m.

j=O, 1 . . . . . m

A primitive set describes a subsimplex {u: u i >I minju~, i = 0, ..., m} in which no other u k lie. F r o m a computational viewpoint, the simplex S can be partitioned into a regular grid of subsimplices and moreover, the points uk can be generated as required (see [25]).

70


The main tool in Scarf's simplicial approximation algorithm is the following result. Lemma 4.2. [-36, p. 1332]. L e t u k~..... u k" be a primitive set, and let u k~ be a specific one o f these vectors. Then, aside f r o m one exceptional case, there is a unique vector u k different f r o m u k~ and such that uk~ Uk~- ', Uk, Uk~+ 1, . .~ Uk", f o r m a primitive set. The exceptional case occurs when the m vectors u k', i ~ ct, are all selected f r o m u ~. . . . . u s, and in this case no replacement is possible.

To apply this result to solving approximately (4.1), we need to apply another theorem which extends the above result. Theorem 4.3. [36, p. 1341]. L e t

/1 A=

0

0

ao,,,+l

ao.K\

am,m+t

a,n,K

" 0 0 0

0

1

)

be an (m + 1) x (K + 1) matrix and b T = (bo . . . . , bin) a (m + 1) non-negative vector, such that the k th column o f A is associated with the k th vector uk ~ P. A s s u m e that the set o f non-negative vectors y, satisfying A y = b, is bounded. Then there exists a primitive set u k~ ..., u k~ so that the columns ko, ..., k,, f o r m a feasible basis f o r A y = b.

The theorem is applied to (4.1) by specifying the appropriate matrix A and vector b. The matrix A is given below and the appropriate uk are shown above their associated columns. U0

UI

U rn

Um + l

UK

1

0

0

1

1

0

1 0

y~'+~ + 1

?~c + 1

0 1 7ss + ~ + 1

?sg + 1

A =

0

0

)


71

where 7 k is any subgradient of L at u k. The vector b x = (1 ..... 1). Clearly, the set {y: A y = b, y > 0} is bounded in this case. We sketch the constructive proof of Theorem 4.3 because it implies an algorithm which we need to discuss in some detail. After the discussion of the algorithm we will give a discussion of the results of the algorithm and their interpretation as an approximation to the optimality conditions of Theorem 2.2. The algorithm begins with the set u ~ u 1, ..., um which is not a primitive set. A primitive set is uniquely formed by replacing u ~ with the vector u k*, where =

max

k>m

We then take the feasible basis corresponding to the first m + 1 columns of A and pivot in column k*. This pivot is the ordinary simplex pivot. If the column of A corresponding to u ~ is pivoted out of the starting basis, then the algorithm terminates. If another column is pivoted out of the starting basis, say column ki, then the primitive set u k*, u 1. . . . , u m and the new basis correspond except for u i and the zero th column of A which is still in the basis. In this case, we remove u ~ from the primitive set, and replace it with a unique u ~ P according to Lemma 4.2. If u ~ replaces u i, then the algorithm terminates. Otherwise, the algorithm continues by repeating these same operations. At an arbitrary iteration, we have a primitive set u k~ u kl, . . . , u km and a feasible basis corresponding to the columns 0, kl . . . . . kin. S t e p 1. Pivot column ko into the basis. If column 0 is pivoted out of the basis, terminate the algorithm. Otherwise, go to step 2. S t e p 2. If column k, was pivoted out of the basis in step 1, take u k' out of the primitive set. If u ~ comes into the primitive set, terminate the algorithm. Otherwise, repeat step 1. The finiteness of the algorithm is due to the uniqueness of the changes at each of the above steps a n d the finiteness of the set P and the possible bases of A y = b. Thus, the algorithm implicit in Theorem 4.3 terminates with a primitive set u k~ . . . , u k" and a non-negative basic solution of A y = b using the basis columns k0,..., km of A. Let s0,..., sm denote the variables corresponding to the first m + 1 columns of A, and let 2m+ 1, ..., 2r denote the remaining variables. Let ~ and ~[k denote the specific values of these variables in the terminal solution; we will show that the ~'k and any u in the final simplex satisfy an approximate sufficient condition for optimality that tr* -- 0 in

72


(4.2). The condition is approximated because the subgradients in the desired convex combination are subgradients of L chosen not at a single point but from rn + 1 points which are close together. There are two cases to consider. Case 1 : 0 r {ko, ..., k.,}. In this case, we have s0 = 0 and row 0 o f A y = b gives us K

Z

L = 1.

(4.4)

k=m+l

Also, in this case ~7'= 1 < u so we require 0 = 0. For row i, i = 1, ..., m, we have K

si+

)--

(~/k+l)2k=

1,

k=ra+ 1

and using (4.4), we have K

si+

~

yik~k = 0 .

(4.5)

k=ra+ 1

If si > 0, then u ~ is in the final primitive set and u k~ ~ 0, j = 0 .... , m and ~kX=,.+ 1 Y~k~k= --Si < 0 is the appropriate approximate optimality condition for row i. On the other hand, if u ~ is not in the final primitive set, st is not a basic variable implying ~ = 0 and ~kr=.,+ 1 ~'ik~k = 0 which is again the appropriate approximate optimality condition. Case 2 : 0 ~ {k 0. . . . , kin}. In this case, the algorithm has progressed from the region of S, where ~7'= 1 ui ~ 0 to the b o u n d a r y ~7'= 1 us ~ U. By the definition of primitive sets, there must be some u ~, 1 < l _< m, not in the final primitive set implying g, = 0. Since the right-hand side on row I is 1, there must be at least one ~k > 0 and therefore ~kr=.,+ 1 ~'k > 0. We norrealize the non-negative weights to sum to one by taking k ~

K

~k

k=m+

1,...,K.

E k = ra + 1 ~[k

F r o m row 0, we have ~ = , . + l ~ [ k = 1 --30 < 1, where the inequality is strict (barring degeneracy) because Do is basic. F r o m row i, we have K

K

k=m+ l

k=m+ l

M.L. Fisher et at. / Usino duality to solve discrete optimization problems

73

and dividing by ~ = , , + 1 ~[k, 1

X

~[k~ , +

~

S0

~,~[k--

2k=O"

(4.6)

Since the uk~,j = 0, 1. . . . , m, in the final primitive set all satisfy ~7'= 1 u~~ ~ U, it is appropriate to allow the non-negative quantity 3o on every row i, i = 1, ..., m. As before, ~i > 0 implies u i in the final primitive set and the condition ~ = m + 1 ~'~-k - ff -< 0 is appropriate. The above analysis establishes the nature of the approximation in terms of approximating optimality conditions. It is also natural to ask: what is an approximately optimal solution t7 in (4.1)? It is reasonable to use any point a in the terminal simplex described by uk~. . . . , u*-. The best choice, however, appears to be any optimal solution in the linear programming problem ~(b) = max w, subject to w < - u b + f ( x ' ) + u g(x'), ~u~O,

i=1

where T is the index set of the yt corresponding to basic 2 variables in the terminal basic feasible solution. We have argued above that ~ contains at least one t. L e t a be an optimal solution in (4.7) and compute L(tT). It is easy to see that t7 is optimal in the dual problem (4.1) if ~(b) = L(t2). This is because L(~) w(b) and v~(b) - L(a) > 0 is an upper bound on the objective function error of stopping with a. Let us now consider briefly the convergence properties of the simplicial approximation as the uk, k > m are increased in number and the diameter of the subsimplices defined by primitive sets approaches zero. For r = 1, 2, ..., let u k~ ..., u kin'" denote the defining points for the terminal primitive set at the r ta partition of S. As r goes to infinity, the diameter of these subsimplices converges to zero, and they contain a sequence with a subsequence converging to a single point u*. Any such point must be optimal in (4.1). To see this, recall that there is only a finite number of distinct columns possible

74


for the system A y = b. Thus, there is some basic set repeated infinitely often in the converging subsequence. Since the sets Q~ = {u: u > 0 and L(u) = L(u, x')} are closed and convex, the approximate optimality conditions represented by this fixed basic set become exact at u*.

5. Subgradient relaxation methods for dual problem In this section we discuss the very interesting work of Held and Karp [27] and Held, Wolfe and Crowder [28] on the application and extension of "The relaxation method for linear inequalities" of Agmon [1] and Motzkin and Schoenberg [32] to the traveling salesman and other optimization problems. We will refer to this method as subgradient relaxation to distinguish it from the data and problem relaxation methods discussed in Section 2. Our discussion in this section will be brief and the development will follow the development of [27] incorporating an observation from [28]. The idea behind subgradient relaxation is to generate a sequence {u'} of non-negative dual variables by the rule Ur+l

=

max{u~. + 0,77,0},

i = 1..... m,

where ?r is any subgradient of L at ur and 0r is a positive constant to be chosen from a specified range. There is no guarantee that L(u r+l) > L(u r) but if w(b) is known, the method can be made to converge (in a certain case, finitely) to a fi such that L(fi) > ~, where ~ is arbitrarily close to w(b). Of course, w(b) is generally not known although it can often be closely estimated. Another point worth noting is that in using the subgradient relaxation method in the branch and bound algorithm of Fig. 6.1, we try to take ~ to be the lower bound required for fathoming the subproblem by bound in step 5. In any case, the justification for the subgradient relaxation method rests with its computation effectiveness. The method has proven effective for the traveling salesman problem [27], the assignment problem and multi-commodity flow problems [28]. Consider now a generic step from the point u > 0 to the point u + 07 + 6, where 0 is some positive number to be chosen, ? is a subgradient of L at u andbi=0ifui+0?i>_0,6i= - u i - OTi > O if ui + O?i < O, i = 1. . . . . m. The following lemma stated without proof summarizes the results of [27, p. 9] and [28]. In words, it says that if a sufficiently small step in

M.L. Fisher et al. / Usin 0 duality to solve discrete optimization problems

75

the direction of the subgradient 7 is taken followed by a projection onto the non-negative orthant, then the new point u + 07 + 6 is closer to a maximal u* than u. Lemma 5.1 (Held, Karp and Wolfe). Let u > 0 with dual value L(u) be given. Suppose -~ is such that ~ >_ 0 and L(-d) > L(u). I f 0 is chosen to satisfy

0 0 is usually available from previous computations. Step 4: Computing the Lagrangean is a network optimization problem; shortest route computation for integer programming, minimum spanning tree for the traveling salesman problem, dynamic programming shortest route computation for the resource constrained network scheduling problem, etc. Step 5: As a result of step 4, the lower bound L(fi; X k) on v(b; X k) is available, where LN; X*) = - ~b + mirn{f(xt) + ~g(x')}.

(6.3)

It should be clear that (6.1) is fathomed if L(fi; X*) > 2 since L(~; X*) _
0. An upper bound on a non-basic variable can be explicitly added to the set X in which case the Lagrangean shortest route problem needs to be modified. Of course, upper bounds affect the Lagrangean subgradients and analysis of Section 2; we omit details. A second practical need are data relaxation methods to control the size of the shortest route problems [-20]. Our experimentation on the IP dual problem has been limited to solution by the primal-dual algorithm of Section 3 with the modification that ascent steps are to the maximum of L along the indicated direction rather than


81

the next breakpoint. Although the modified algorithm can no longer be proven to converge, this has not appeared to present practical problems. The basis B used in the transformation of (7.1) to (7.2) was always an optimal LP basis and the starting dual solution was always u = 0. The algorithm was encoded for the MULTICS time sharing system on the G.E. 645 computer at M.I.T. The group theoretic IP algorithm IPA [21] was also established on MULTICS, and we used it to find feasible and optimal IP solutions while the primal-dual provided lower bounds. Segment size on MULTICS effectively limits search space and, as a result, IPA would either find a feasible or optimal IP solution within 10 to 60 seconds after the LP solution was found, or be forced to quit because the available space was filled. The IP dual approach has not yet been combined with branch and bound except on an ad hoc basis. The first IP test problem is a covering problem due to Day [5] consisting of 30 rows and 28 zero-one variables. A number of test problems were constructed from this problem by altering the objective function. Fig. 7.1 illustrates the performance of the primal-dual algorithm on one of the first test problems. The abscissa represents the number of iterations of the primal-dual algorithm and the ordinate the lower bound. The lower bounds have been normalized so that the LP cost is 100. The circled points on the figure are the successive lower bounds. The first two points are on the ordinate axis because the primal-dual had not yet begun to pivot. The arrow around iteration 50 on Fig. 7.1 indicates that a feasible IP solution to the IP problem was found while maximizing along a line on which lie the points yielding the indicated adjacent bounds. Coincidentally, it took 48 simplex iterations to find an optimal LP solution. The feasible IP solution was in fact optimal, but it was not possible to prove it because complementary slackness did not hold (see Section 6). Optimality was established by a separate run with IPA. We did not allow the algorithm to converge to an optimal dual solution because it was taking too much time, but the asymptote seems clear. The behavior illustrated in Fig. 7.1 has typical and atypical characteristics. The typical characteristics are the rapid rise in dual lower bound during the first few steps, the long tail towards the maximal dual lower bound, and the existence of a duality gap. The atypical characteristic is the discovery of a feasible solution. The discovery of a feasible IP solution as a result of a Lagrangean minimization was a rare event for all of the IP problems tested thus far. To see why this might be so for any discrete dual problem, consider the general model (1.1) and let ~ E X be an arbitrary

M.L. Fisher et al.

82

Using duality to solve discrete optimization problems

MINIMAL

106

|

(9

|

|

|

|

IP C O S T

(9

|

|

105

|

C3 Z E3 O r

104

rY 103 LU O d 102

101

I00

,

I

50

100 PRIMAL- DUAL

150

200

250

ITERATIONS

Fig.

7.1.

feasible solution (i.e., g(~) _< b). Since J(~) _> L(u) for any u _> 0, the scalar u(g(Yc) - b) may need to be a negative number fairly large in magnitude in order that L(u) = f(YO + u(g(YO - b). This is particularly true when there is a significant duality gap between the primal and dual problems, i.e., when f(~) :~ L(u) for all u _> 0. Table 7.1 summarizes our experience with six test problems derived from the problem of Fig. 7.1 by altering the objective function. Problem 1 is the one depicted in Fig. 7.1. Note that there was, or appeared to be, a duality gap in all cases between the IP problem and its dual (a ">_" or "_ 0, (g,+i, s~) = 0, i ~< n,

(11)

that is to say, x. + 1 minimizes f in the affine manifold spanned by {So, ..., s. }. We consider those methods which, in addition to (10), satisfy, for each n,

C. Lemarechal / An extension of Davidon methods

100

s. = ~ 27g,.

2." • 0.

(12)

i=0

In other words, we seek the n 'h direction in the subspace spanned by the gradients which are already known. It is quite natural to proceed in this way. Lemma 2.1.

For each method satisfying (10), (12), one has, for each n,

(g.+ a, gi) = O,

Proof.

i =.0,..., n.

The proof is by recurrence on i. For i = 0,

(g.+ 1, go) = (g.+ 1, So) = 0

For i

0 = (gn+ 1, Si) =

> 0,

from (11). 2}gj

i-1

=

g,) + j =2O 2}(g.+,,

But the last sum is supposed to be zero, and 21 :~ 0, so the proposition is proven. Thus, in a Davidon method, (12) is a sufficient condition for the gradients gi to be orthogonal to one another. Theorem 2.2.

s.=

For each method satisfying (10) and (12), one has

k,__~~ gi "=

(13)

igii 2 ,

where {k.}, a scalar sequence, is the only possible freedom allowed in such methods. Furthermore, if-g. is the projection of 0 onto the convex hull of{go . . . . . g.}, then the directions s, generated by all such methods are colinear to -g..

Proof. F r o m (10), 0 = (s.; A si) = (s., gi+ l - gi), a constant with respect to i. F r o m (12), (s,, gi) = ~. 2~(g,, g j) = k,,

i < n. Thus, (s., gi) is

i ~ n.

j=O

This is a linear system with respect to 2, the solution of which, by L e m m a 2.1,


101

is obviously 27 = kJlg;[ 2. Now, adjust kn = kn, so that sn = L~q,/lo, I2 belongs to the convex hull of {go. . . . . 0n}. kn is defined by 1.

Is.I 2 = _~.~21/]g,I ~ = kn = (s.,o,), i ]~.12 for i ~< n, and this point is exactly the stated projection. But in this case,

Remark 2.3. Adachi [1] mentions twelve Davidon methods which are equivalent. Hence the question is: since "(10) and (12) =~ (13)" and since one does not know any method satisfying "(10) and not (13)", is (12) really useful to prove (13)? The answer is obviously yes: for n = 1 there are many directions Sl such that (sx, A So) = 0, but only a few of them verify sl = 2~go + 2~ga. In fact, the phenomenon mentioned by Adachi justifies a posteriori the hypothesis (12): if no method is known verifying "not (12)", perhaps it is because (12) is really natural. Theorem 2.4.

Proof.

F o r each n, 9, ~ O~,f(Xo), w h e r e e. = f ( X o ) - f ( x , ) .

From (6), we know that the theorem holds if f ( x . ) >f f ( X o ) + (9., x . - Xo) - ( f ( X o ) - f ( x . ) ) ,

i.e., if (9,, x. - x0) ~< 0. But from (11) we have n--1

(0n, X. -- XO) =

~

Pi(O., Si) = 0.

i=0

We conclude this section by claiming that when applied to a quadratic objective function all methods defined by (10) and (12), i.e., in practice all Davidon methods, have the following properties: Sn = Proj O / c o n v { - 9 o . . . . . --gn} (with Pn > 0), On~ ~,nf(Xo), where en = f ( x o ) -- f(Xn).

3. Extension to non-differentiable functions

Both the properties exhibited in the preceding section are intrinsic and do not depend on the nature of the objective function. Hence, we are able to


102

define an extension of conjugate gradient methods in the following way: Sn is calculated as the projection of O onto the convex hull of { - g0 . . . . . - g . } as long as g. e 8,nf(xo). This latter can easily be checked; if it does not hold, then the sequence is cut off and, for example, sn is taken as -On. In order to ensure convergence, we have to slightly modify e. by defining en=f(xo)-f(x.)+e,

wheree>O.

We propose the following algorithm. Algorithm 3.1. Let Xo 9 H, e > 0 be given. Let go e ~3f(xo). Set i = n = 0, Pi = O. Step 1. Calculate sn, being the projection of O onto the convex hull of . . . ,

- g n ) .

Step 2. Calculate Pn such that f ( x n + pnsn) ~ f(xn + psn), Vp >10. Set Xn+ l = Xn -F pnSn.

Step 3. Determine gn+ 1 9 df(xn+ 1) such that (gn+ 1, sn) = 0. Set n = n + 1. Step 4. If (gn, x. - xp) -%<e, then go to step 1. Step 5. Set i = i + 1, p~ = n, and go to step 1. In step 2, note that since (s,, g,) < 0, f ( x . + ps,) >1 f ( x . ) if p < 0. Our convergence theorem will be: f ( x . ) (which is decreasing) has a limit f * , such that f * ~< inf{f(x): x 9 H} + e. This property is easily proven if the sequence of cuts is finite. Suppose for example that Pi = 0. Then s, ---, 0 from Corollary 1.5. But en is an increasing sequence; hence, if m ,f ( x ) + (g, y - x) + d([y Furthermore, if d(t) ~ O, then t ~ O.

x[).


103

The proof of this classical result is not given. Theorem 3.3.

When applied to a uniformly convex function, Algorithm 3.1 is such that f(x,) ~ f*, where f * 0 if a cut must appear, g.+l will of course be sought by a dichotomic search in the presence of: (J) Pl 9 [0, p.], x l = x. + p l s . and gi 9 Of(x1) such that (s., 91) ~< O, (JJ) P2 9 ]P., + ~ [, x2 = x . + p2s. and g2 9 Of(x2) such that (s., 02) >/O. We seek g.+l as a convex combination of gl and g2. Let 21, 22/> O, 21 + 2 2 = 1 and set X -~- )~IX1 "-~ ,~.2X2,

g -- "~lgl q- )]'292-

x and g are candidates for x. + 1 and g. + 1, respectively, Lemma 4.1.

g 9 O~,f(x), where

e' = ,~l,~2(Xl - x2, gl - g2). Proof.

Vz e H, f ( z ) >~ f ( x 0 + (gl, z - Xl) f ( z ) >~ f ( x 2 ) "+" (g2, Z -- X2).

By convex combination, we deduce

(14)

105


f ( z ) >~ f ( x ) + (9, z - x) = ~1(41 "~ 42)(91, Xl) -- 42(~1 -~ /1,2)(92, X2)

+ 4~(91, xl) + 41~2[(9,, x2) + (92, x0] + ~(92, x2) and a direct calculation gives the stated value for e'. As in Theorem 1.2, it is easy to see that 9 ~ ~ +,f(xp,), with e', +1 = f(Xv,) - f ( x ) + (g, x -- xv, ) + e'

and e' as in (14). Therefore, since e ~ d , f ( x ) is an increasing mapping, we see that we have (iii) if e'n+ 1 ~< f(xp,) - f ( x ) + e, i.e., if (g, x - xp,) +

41,~,2(X 1 -- X2, 91

--

g2) ~< 8"

(15)

If (15) does not hold, we should cut the chain. But, before doing this, we have to be sure that Ix - xp,[ remains b o u n d e d from below (see (iv)). Note that e'/> 0 in (14). Therefore, if fl~]0,1[, then, as soon as 2142(xl - x2, gl - 92) ~< fl e; "not (15)",

(16)

the inequalities (9, x - xp,) >/(1 - fl)e > 0 obtain. As the dichotomic search proceeds, e' ~ 0. So, if (15) never obtains, we are sure that (16) obtains after a finite number of iterations and then we can safely cut the chain. The precise choices for 41 and 42 will be given in view of (i) and (ii). Lemma 4.2.

Suppose f is uniformly convex with d convex (cf. (2)). I f

41P1(S,, 91) "~- 42P2(Sn, 92) ~ 0,

(17)

then f ( x , ) >1 f ( x ) + d([xn - x[).

Proof.

We have f ( x , ) >~ f(Xl) + (91, Xn -- Xl) "~ d(p 1 ISn[) f ( x , ) >t f(x2) + (92, x , -- x2) + d(p2 Is.I).

By convex combination making use of the convexity of f, we obtain f ( x , ) >i f ( x ) - 2,PI(gl, s,) - 22P2(g2, s,) + ).,d(pl

Is,l) +

4~d(p~Is,l).

But since d is convex,

41d(p, Is,I) + ~=d(p= Is.I) f> dr(~lp~ + ~2p2)Is.I] -- d(Ix - x,I) and thus the desired property is obtained when (17) holds.

106


To summarize this section, let us choose e > 0, 0 < ~ < 1, 0 < fl < 1. The dichotomic line-search is terminated as soon as we have simultaneously (see (j), (jj) for notations),

2,(g1, s.) + a2(g , s.) 1> •lPl(gl,

(18)

]s.I 2,

(19)

Sn) + "~'2P2(92, Sn) 0 for all p r e c e d i n g d i r e c t i o n s sj. T h u s , zig-zagging is a v o i d e d ; the a l g o r i t h m c a n n e v e r " r e t r a c e its steps". But it is n o t sure that, as the a l g o r i t h m proceeds, the m i n i m u m p o i n t is n e v e r o v e r s t e p p e d . H e n c e the necessity of a cut if Ixn - xp, I b e c o m e s t o o large.

References [-1] N. Adachi, "On the uniqueness of search directions in variable-metric algorithms", Journal of Optimization Theory and Applications 11 (6) (1973) 590-604. [2] D.P. Bertsekas and S.K. Mitter, "A descent numerical method for optimization problems with nondifferentiable cost functionals", S I A M Journal on Control 11 (4) (1973) 637-652. [3] W.C. Davidon, "Variable-metric algorithms for minimization", A.E.C. Research and Development Report ANL 5990 (1959). 14] V.F. Demjanov, "Algorithms for some minimax problems", Journal of Computer and Systems Science 2 (1968) 342-380. [5] P.J. Laurent, Approximation et optimisation (Hermann, Paris, 1972). 1"6] C. Lemarechal, "An Algorithm for minimizing convex functions", in: J.L. Rosenfeld, ed., Information processing "74 (North-Holland, Amsterdam, 1972) pp. 552-556. [7] G.P. McCormick and K. Ritter, "Projection method for unconstrained optimization", Journal of Optimization Theory and Applications 10 (2) (1972) 57-66. ['8] R.T. Rockafellar, Convex analysis (Princeton University Press, Princeton, N.J., 1970). 1"9] P. Wolfe, A method of conjugate suhgradients for minimizing nondifferentiable functions, Mathematical Programming Study 3 (1975) 145-173 (this volume). [10] G. Zoutendijk, "Some algorithms based on the principle of feasible directions", in: J.B. Rosen, O.L. Mangasarian and K. Ritter, eds., Nonlinear Programming (Academic Press, New York, 1970).


MINIMAX SOLUTION OF NON-LINEAR EQUATIONS WITHOUT CALCULATING DERIVATIVES Kaj MADSEN Technical University of Denmark, Lyngby, Denmark Received 8 November 1974 Revised manuscript received 12 May 1975

The problem of minimizing the maximum residual of a set of non-linear algebraic equations is considered in the case where the functions defining the problem have continuous first derivatives, but no expression for these derivatives is available. The method is based on successive linear approximations of the functions, and solution of the resulting linear systems in the minimax sense subject to bounds on the solutions. Approximations to the matrix of derivatives are updated by using the Broyden l-2] rank-one updating formula. It is shown that the method has good convergence properties. Some numerical examples are given.

1. Introduction T h e p r o b l e m of minimizing the m a x i m u m residual

F(x) =- m a x J

If (x)l

(1,1)

of a set of nonlinear algebraic functions fj(x) = f j ( x l , . . . , x,),

j = 1. . . . . m

(1.2)

can be solved by a m e t h o d described by M a d s e n [6]. As m o s t other m e t h o d s for o p t i m i z a t i o n this m e t h o d is iterative and ifxk is an estimate of a solution, the increment hk is found as a vector which minimizes the linear a p p r o x imation maxlfj(xk) + ~ 9 J

i=l

, I Tdfj,xtx jh,

(1.3)

subject to the constraint

IIh[I = I](hl,..., h.)ll

- max

Ih, I _

F(Xk) -- F(Xk + hk) >_ F(Xk) -- IIf(xk) + dx~hk + O(hk)iI F ( X k ) - I/Ath~)ll F(x~)- IIf(x~) + J~,~hkl[ § [IJ~ -- B~II" rlh~[I

o(llh~ll)

=1+

F ( x k ) - [If(xk) + Jxkhkll +

o(llh~l[)

= 1 + o(1).

(3.6)

Therefore inequality (2.12) is never satisfied when k is sufficiently large, and the lemma is proved. Theorem 3.2.

I f xk ~ x*, then x* is a stationary point of F.

Proof. Suppose x* is not a stationary point. In this case there exists a vector h* such that

d - II/(x*)ll -II:(x*) + J.,h* II > O.

(3.7)

Because of continuity this implies that for k sufficiently large, k > ko say, we have

Ilf(xk)/I - I[f(xk)

+

ax~n*ll - d/2.

(3.8)

If we let tk be defined by the equation [Ihk/I = tk [[h*l[,

(3.9)

then we deduce from inequality (3.8) and the definition of hk that for k_koandtk< 1,

Ilf(x~)l[ - [If(x~) + J~hkll --- [[f(x~)[I -IIf(x~) + Jx~(tdi*)lt = i]f(Xk)[j -- [[f(Xk)(1 -- tk) + tk(f(Xk) + ,J,.h*)il >-

t~(]l f(x~)ll - [If(x~) + -/~h*il)

>- tk" d/2.

(3.10)

Now (3.8), (3.10) and the definition ofhk imply that for k > ko we have the

K. Madsen / Minimax solution of nonlinear equations

117

inequality

IIf(xk)ll

-

Ilf(xk) + J~)ik[[

> min(tk, 1)' d/2,

(3.11)

and therefore it follows from Lemma 3.1 that

~a+x -Ileal[

(3.12)

for k sufficiently large. Further, Theorem 3.1 implies that B a ~ J~,, and therefore inequality (3.7) gives that for k sufficiently large we have

[If(xk)ll - Ilf(xa) + eah* I[ > d/2.

(3.13)

However, hk --} 0 and consequently,

IIf(x~)[I -IIf(xa) + B~hall-~ 0.

(3.14)

(3.13) and (3.14) can only hold if ha is restricted for all large values of k, and therefore

Ilha[I = ~k

(3.15)

when k is large. Now (3.12) and (3.15)imply that Hhk+~ 11 >-- I/hall for all large values of k. This contradicts the fact that hk ~ 0 and therefore x* must be a stationary point of F. In the remaining part of this section we assume that m > n, and that the objective function F is positive at the solution x*. In this situation the rate of convergence for the algorithm depends on the following condition. Definition 3.2. If every n x n submatrix of,/x is nonsingular, then we say that ,/~, satisfies the Haar-condition.

Lemma 3.2. Let hx be the unconstrained minimax solution of the linear system Ij(h) = fj(x) + ,=~ ~xitx) h, = O,

j = 1. . . . . m.

(3.16)

I f the Haar-condition holds at a solution x* of the non-linear problem (1.1), then there exist 6 > 0 and c > 0 such that for IIx - x* If - ~, F(x) -

[If(x) + ,L, hx[t ->

F(x) -

F ( x * ) >_ cllx

cllhxll

(3.17)

and -

x*ll

(3.18)


118

Proof. It follows from the theory of linear minimax approximation that when the matrix J , satisfies the Haar-condition, the solution to the problem (3.16) is the minimax solution to (n + 1) of the equations in (3.16) and the maximum residual is attained by each of these equations. Such a set of (n + 1) equations is called a reference, and we use the notation refz for the set of indices defining the reference at the point x. Without loss of generality we can suppose that A(x*) = . . .

= A(I*) > If,<x*)l,

i > k.

(3.19)

Then the Haar-condition implies that k > n (see [-3J). It follows that if x is close enough to x*, then the reference defining the solution of (3.16) will consist of functionsfi with j _< k, and further the signs of the residuals corresponding to functions in the reference will all be positive. This means that there exists 51 > 0 such that

IIx - x*ll < 5, ~ I[f(x) + J~h~ll = f/x) § (vf~<x), h~),

j~ref~, (3.20)

where (Vfj(x), h) is the inner product, (Vfj(x), h) = ~ ~fJ(x) h,.

(3.21)

i= 1 (~Xi

Again, from the theory of linear minimax approximation it follows that if x satisfies the condition in (3.20) then there exist positive constants r such that

Z

je ref=

~JVL(x) = O.

(3.22)

For any vector h # 0 we deduce from this jeref=

~j(Vfj(x), h) = 0

(3.23)

and consequently one of the terms in the sum must be positive, since tl~ Haar-condition implies that at least two of the inner products are non-zero. Therefore c~, =

rain ~ max (Vf~(x), h) ~ > 0 )

Ilhl= l ~ j~refx

(3.24)

and since c~ is continuous as a function of x there exists cl > 0 such that i[x - x * l l -< 51 ~ c x >- c~. From (3.20) and (3.24), with h = - / i x

(3.25) inserted, it follows that for


IIx - x*ll ~ ~1

we

119

have

max f j ( x ) = max

jr refx

j~ refx

{ IIf(x) + a~hxll

-(vf,<x), hx))

-> Itf(x) + Jxh~ll + Cl IIh~ll

(3.26)

and since the left-hand side of this inequality does not exceed F(x), inequality (3.17) is proved. Now it is easy to prove (3.18). Inequality (3.24) implies that

max _ cl Ilhll

(3.27)

j_k

and consequently max {fi(x*) + (Vfj(x*), h)} >- [If(x*)[] + cl j max {fj{x* + h)} > F(x*) + j-- IIf(x~)ll -[If(x~) + Jx.(t~h=.)ll _> t~(ilf(xk)ll- ilf(xk) + J,,h,,,]l) >_ t~c llh,.ll -- c iln~ll.

(3.35)

Therefore we can use Lemma 3.1 and we find that 2k +, > IIh~ II for k > ko (say). Note that we can choose k0 such that inequality (2.11) always holds for k > ko. N o w hk --* 0 for k --* oo, and this can be true only because (3.31) holds for infinitely many values of k. In order to prove that (3.32) implies (3.33) we let e;,, k = 1, 2 . . . . , be a sequence of positive numbers satisfying ][f(Xk +.hk) -- {f(Xk) + J=khk}]l < e'k IIh~l[,

t[ J,,. - B R[I _ ka, then it will hold also in the next 2 iterations. Therefore, since every special iteration is followed by a normal iteration, (3.33) will hold for every k e N following the first index k 2 ~ k 1 for which (3.31) holds. N o w the result is a consequence of (3.44) since

IIx +2 - x*ll -< max{ek+1, ek}

C- llxk- x*[I

(3.49)

cl

for every k > k2.

4. Numerical results

We have solved several test problems with our algorithm, and we illustrate the results by giving the number of function evaluations for some of these. The calculations were performed in double precision on an IBM 370/165 computer. The first problem is a data fitting problem given in [1]: Find the parameters xi, i = 1, 2 . . . . . 5 such that the rational expression Xl + x2y 1 +Xay+x4y 2+xsy 3

(4.1)

is the best fit to e y at the points yj = - 1 (0.1) 1. Thus the functions are

xl + xzyj - e yJ, f ~ ( x l , . . . , x 5 ) = 1 + x3y j + x4y~ + xsy 3

j = 1, 2 . . . .

21

(4.2)


123

and we use the starting point Xo = (0.5, 0, 0, 0, 0) and the initial bound 20 = 0.125. The second example is the Chebyquad equations as defined by Fletcher [-4]. Here we find zeros for n = 2, 4, 6 and 9, whereas no zero exists in the case n = 8. Here a local m i n i m u m x* with F ( x * ) > 0 is found, the convergence being rather slow because the Jacobian is singular at the solution. We used the starting vectors given by Fletcher and the initial values 2o = 0.01 for n < 6 and 20 = 0.005 for n = 8, 9. Finally we consider the functions fl(x)

= x21 + xg -

1,

f2(x) = 3 -

x21 -

x~,

f3(x) = x 4 + x 4 + 2 x ~ x ~ - 3(x~ + x~2) + 1.

(4.3)

For these functions the m a x i m u m residual attains its minimum at every point on the circle with centre (0, 0) and radius x/2. We tried several starting points and in all the cases the sequence of vectors generated converged to a point on the circle though not always the same point. We also tried the method where the derivatives are given [-6] on these test examples and here the sequence of vectors corresponding to the functions (4.3) always converges to the set of points on the circle but not always to a specific point. We give the number of function evaluations used when Xo = (3, 1) and 2o = 0.2. In Table 1 we quote the number of function evaluations used by the two methods to find the solutions with 6 decimal accuracy. The figures for this m e t h o d are considerably smaller than the figures which would be the result of a derivative-approximation of the form (1.5) at each step in the iteration, since these figures would be at least (n -~ 1) times the figures in the last column. Table 1 Number of function evaluations 6 decimal accuracy

Functions (4.2) Chebyquad

n

m

This method

Derivativesgiven

5 2

21 2

31 13

9 7

--

4

4

18

8

--

6

6

29

9

--

8

8

123

40

--

9

9

2

3

Functions (4.3)

60 29

12 16


124

Our results indicate that the method is also efficient for finding zeros of non-linear equations in the case where m = n. For comparison we give the number of function evaluations used by the method of Powell [-9] to find the zeros of the Chebyquad functions, and also to find the zeros of the trigonometric functions

fj(x) = ~ {Aijsin xi + Bijcos xi} - Ej,

j = 1. . . . . n

(4.4)

i=1

whose parameters are given by Fletcher and Powell [-5]. Table 2 Number of functionevaluationsused to meet the criterion F(Xk) < n Chebyquad ----

Functions(4.4)

This method

2

11

4 6 9

16 26 54

5

10 - 4

Powell 7 14 34 46

16

16

--

10

21

22

--

20

33

45

These results, and the results from other test examples, show that our method is efficient for solving overdetermined systems of non-linear equations in the minimax sense. This includes minimax data fitting. Further, the algorithm is a good alternative to existing methods for finding zeros of systems of non-linear equations.

Appendix Proof

of

Theorem 3.1. Define the matrix C k by

(

B k + , - J x , = (Bk--Jx,) I - h ~ h k / i - Ok,

(5.1)

that is, Ck = (Bk -- dx,) ( I - - h k h T k ~ - - ( B k + (f(Xk + hk)--_h~ffkkf(Xk)- Bkhk)hTk)

h hJ

(f(Xk + hk) -- f(Xk) -- d~*hk)h T

hrhk

+4. (5.2)


125

Since hk -~ 0 for k ~ ~ the smoothness condition (3.1) implies that 1]f(Xk + hk) -- f(Xk) -- Jx, hk il _~ 0

for k ---, oo

(5.3)

iFhkll and therefore

[ICk[[ ~ 0

for k ~ ~ .

(5.4)

Now suppose that ilck[I < ~ for k >__ ko. Then it follows from (5.1) and (2.16) that for k > ko

J,,*l[ -< (Bk

-

-

J'*)"

+

IIC [I

3n'F,.

(5.5)

F r o m this it follows that Bk ~ Jx* and Theorem 3.1 is proved.

Acknowledgment Most of this work was carried out during my stay at AERE Harwell, England, and I wish to express my thanks to M.J.D. Powell for m a n y helpful discussions and suggestions.

References [1] I. Barrodale, M.J.D. Powell and F.D.K. Roberts, "The differential correction algorithm for rational l~-approximation", SIAM Journal on Numerical Analysis 9 (1972) 493. [2] C.G. Broyden, "A class of methods for solving non-linear simultaneous equations", Mathematics of Computation 19 (1965) 577. [-3] A.R. Curtis and M.JD. Powell, "Necessary conditions for a minimax approximation", The Computer Journal 8 (1966) 358-361. [-4] R. Fletcher, "Function minimization without evaluating derivatives--a review", The Computer Journal 8 (1965) 33. [-5] R. Fletcher and M.J.D. Powell, "A rapidly convergent descent method for minimization", The Computer Journal 6 (1963) 163. [-6] K. Madsen, "An algorithm for minimax solution of overdetermined systems of nonlinear equations", Journal of the Institute of Mathematics and its Applications, to appear. [7"] D.W. Marquardt, "An algorithm for least squares estimation of non-linear parameters", SIAM Journal on Applied Mathematics 11 (1963) 431. [8] M.R. Osborne and G.A. Watson, "An algorithm for.minimax approximation in the non-linear case", The Computer Journal 12 (1969) 63.

126


[9] M.J.D. Powell, "A hybrid method for non-linear equations", in: P. Rabinowitz, ed., Numerical methods for non-linear algebraic equations (Gordon and Breach, New York, 1970) p. 87. [10] M.J.D. Powell, "A new algorithm for unconstrained optimization", in: Nonlinear programminff (Academic Press, New York, 1970). [11] M.J.D. Powell, "A Fortran subroutine for unconstrained minimization, requiring first derivatives of the objective function", Rept. R6469, A.E.R.E., Harwell, England (1970). [12] M.J.D. Powell, "The minimax solution of linear equations subject to bounds on the variables", Rept. CSS11, A.E.R.E., Harwell, England (1974).


THE USE OF THE BOXSTEP METHOD IN DISCRETE OPTIMIZATION Roy E. MARSTEN Massachusetts Institute of Technology, Cambridge, Mass., U.S.A. Received 10 March 1975 Revised manuscript received 9 April 1975 The Boxstep method is used to maximize Lagrangean functions in the context of a branch-and-bound algorithm for the general discrete optimization problem. Resuits are presented for three applications: facility location, multi-item production scheduling, and single machine scheduling. The performance of the Boxstep method is contrasted with that of the subgradient optimization method.

1. Introduction

The Boxstep method [-15] has recently been introduced as a general approach to maximizing a concave, nondifferentiable function over a compact convex set. The purpose of this paper is to present some computational experience in the use of the Boxstep method in the area of discrete optimization. The motivation and context for this work, originated by Held and Karp [10], is provided by Geoffrion [-9] and by Fisher, Northup and Shapiro [-5, 6], who have shown how the maximization of concave, piecewise linear (hence nondifferentiable) Lagrangean functions can provide strong bounds for a branch-and-bound algorithm. We shall consider three applications: facility location, multi-item production scheduling, and single machine scheduling. Our experience with these applications, while limited, is quite clear in its implications about the suitability of Boxstep for this class of problems. We shall also take this * The research reported here was partially supported by National Science Foundation grants GP-36090X (University of California at Los Angeles), GJ-1154X2 and GJ-1154X3 (National Bureau of Economic Research).

128

R.E. Marsten / The use of the Boxstep method in discrete optimization

opportunity to introduce two refinements of the original Boxstep method which are of general applicability. Three appendices give complete data for the test problems used.

2. The Boxstep method We present here a specialized version of the Boxstep method which is adequate for maximizing the Lagrangean functions which arise in discrete optimization. We address the problem max w(lr),

(2.1)

w(~) = m i n ( f k + ~ ok).

(2.2)

~t>O

where

fk is a scalar, it, gk ~ R m, and K is a finite index set. Thus w(ir) is a concave, piecewise linear function. The Boxstep method solves (2.1) by solving a finite sequence of local problems. Using (2.2), the local problem at 7rt with box size fl may be written as P(Tr'; fl)

maximize subject to

a, fk+~gk>a fork~K, t t 7ri--fl~< lri--< ~ i + f l f o r i = 1,...,m, 7r>0.

This local problem may be solved with a cutting plane algorithm [7, 12, 14]. If a global optimum lies within the current box, it will be discovered. If not, then the solution of the local problem provides a direction of ascent from 7rt. The Boxstep method seeks out a global optimum as follows. Let P(Trt; fl) denotes p(;~t; fl) with K replaced by some K" _~ K.

Step 1 (start). C h o o s e 7~1 ~__ 0,/~ ~_~ 0, j~ > 0. S e t t = 1. Step 2 (cutting plane algorithm). (a) (initialization). Choose K" ___K. (b) (reoptimization). Solve P(rrt; fl). Let ~r, # denote an optimal solution. (c) (function evaluation). Determine k* ~ K such that w(~) = f k , + ~rgk, (d) (local optimality test). If w(r > # - e, go to step 3; otherwise set = K w {k*} and return to (b). Step 3 (line search). Choose n t+l as any point on the ray {$ + 0c($ - it'): >- 0} such that w(rct+ 1) >_ wffr). Step 4 (global optimality test). If w(Trt+ 1) < w(rct) + e, stop. Otherwise sett=t+ 1 and go to step 2.


129

The convergence of the method is proved in [,15]. In the piecewise linear case (finite K) we may take e = 0, at least in theory. The implementation of the method works with the dual of P(~r'; fl) so that new cuts can be added as new columns and the primal simplex method used for reoptimization at step 2(b). The motivation behind the method is the empirical observation that the number of cutting plane iterations required to solve P(~zt; fl) is a monotonically increasing function of B. This presents the opportunity for a trade-off between the computational work per box (directly related to fl) and the number of boxes required to reach a global optimum (inversely related to B). Computational results reported in [,15] demonstrate that, for a wide variety of problems, the best choice of B is "intermediate", i.e., neither very small nor very large. If B is sufficiently small, then (in the piecewise linear case) we obtain a steepest ascent method; while if fl is sufficiently large, Boxstep is indistinguishable from a pure cutting plane method. (For ~1 = 0 and B = oo we recover the Dantzig-Wolfe method [2].) For intermediate values of fl we have something "between" these two extremes. The three applications which follow are all of the form: v* = min fix), x~X

s.t. 9(x) < b,

(2.3)

where f : X - - . R, g : X ~ R m, and X = {x*: k s K } is a finite set. The Boxstep method will be used to maximize a Lagrangean function w(rO, defined for rr ~ R~ as

w(n) = min f(x) + n[g(x) - b]. xeX

(2.4)

Any branch-and-bound algorithm for (2.3) can compute lower bounds by evaluating this Lagrangean, since w(n)< v* for all n > 0. Finding the greatest lower bound, i.e., maximizing w(n) over all n > 0, is a dual problem for (2.3). Thus we shall be using Boxstep to solve a Lagrangean dual of the discrete program (2.3). By defining fk = f ( x k) and gk = g(x k) _ b for all k ~ K we obtain the form assumed above, (2.2).

3. Facility location with side constraints The first application is a facility location model [-8] of the form:

130


min ~, f x i + ~ i=l

~ ciYij,

(3.1)

i=1 j=l

subject to

~ y~j = 1,

j = 1..... n~

(3.2)

i=1

A x + B y _< r,

(3.3)

vixi < Z djyij 101~176 which is typical. A number of proposals have been made for treating nondifferentiable optimization problems by some version of steepest descent. Outstanding among them are those of Demjanov [-4] and Bertsekas and Mitter, whose paper [1] gives an excellent survey of the area and an extensive bibliography. None of these proposals, however, seems to have a straightforward implementation in the general case. Demjanov calls for knowledge of the complete subdifferential or, worse, an e-subdifferential [20, ch. 23] at any point, which seems impractical, while Bertsekas and Mitter confine their procedure to objective functions whose conjugate functions can be given analytically, and require the solution of a (smooth) constrained nonlinear programming problem at each step of their method. We know of three alternatives to the present method which are fully articulated for our class of problem. The convergence of the "cutting-

P. Wolfe / A method of conjugate subgradients

147

plane" method does not depend on differentiability [23]. For smooth problems, even quadratic, it can have quite slow convergence, but it seems more attractive in the nondifferentiable case: the closer the epigraph o f f can be approximated by a polyhedron, the better it should work. A recent refinement of the method [11] seems even more promising. A second alternative is the "subgradient algorithm" mentioned above which has, we think, worked well, but whose generally effective implementation and rate of convergence is still a mystery. (Shot's modification [21, 22] of subgradient optimization does, however, allow some convergence rate estimates to be made.) A third we should mention is the recent very interesting procedure of Lemarechal [14, 15], which of all these is closest to ours in spirit, and has shown encouraging computational results. We expect the present method to work well even on smooth problems, for it constitutes for them a version of the method of conjugate gradients. Indeed, w h e n f i s quadratic and the obvious specialization of the procedure to that case is made, this method is exactly that of Hestenes and Stiefel [12], and terminates in the solution of the problem. Our belief in the effectiveness of the present method is based largely on the above observation, for our practical experience with it is small. It will be seen that there is great freedom of choice in how the algorithm can be run, and we have only begun to explore some of the possibilities. Feuer [7] has tried it in its simplest form (i.e., as presented below) on several of the small but hard differentiable problems in the literature, and found that it required from seven to ten times as much work, measured by number of function evaluations, as standard good methods. Our few experiments, employing a good line search adapted to the differentiable case, seem to bring the required work to within a factor of two of that taken by a standard good implementation of conjugate gradients [8]. This limited use has been encouraging, but we are not ready to offer the procedure as the answer to our class of problem without more study. We have confined our study to convex functions. While those to which this procedure applies are many and useful, some extensions are possible; for example, Feuer [7] has developed some of the ideas of this work into an effective algorithm for the type of minimax problem studied by Demjanov [4]. We can hope that they will be useful in any problem for which a suitable analogue of the notion of subgradient is at hand. In Section 2 below we review some notions from convex analysis which are not yet household words in optimization and develop formulas needed in the sequel. The bulk of the paper, Sections 3~5, concentrates on the


148

"main algorithm": a procedure which, given certain positive "tolerances", terminates in a point which approximates the solution to a degree measured by the tolerances. Section 3 gives an informal account of the main algorithm and reasons for its several parts. Section 4 presents the main algorithm, while Section 5 discusses its convergence. Section 6 shows its connection with the method of conjugate gradients. Section 7 is devoted to some theoretical observations on how the main algorithm may be run sequentially, reducing its tolerances to zero, in an "outer algorithm" which converges to the solution of the problem if one exists. The main results are that the minimum is approached as k -3 in k line searches in the general case, and as A -k (A > 1) if f is twice differentiable and strongly convex.

2. Preliminaries

We first summarize some notions we need from convex analysis, as presented by Rockafellar [20]. Let f be a closed, proper convex function on E" (i.e., f is convex and lower semicontinuous and never assumes the value - Go, although + oo is allowed). The effective domain o f f is the convex set dora f = {x:f(x) < + ~ } . For any x E d o m f a n d nonnull d~ E", the

directional derivative if(x;d) = lim

t~0 +

f ( x + t d) - f ( x ) t

(2.1)

exists. It is a convex function of d, and is closed and proper if x belongs to the relative interior of d o m f The vector u ~ E" is a subgradient of f a t x if

f(y) > f(x) + (u, y - x)

(2.2)

for all y. The set of all subgradients at x is the subdifferential Of(x) of f a t x, which is a closed, convex set. Denoting by f'(x; d) the closure of f '(x; d) as a function of d (that is, the function whose epigraph is the closure of that of f '(x; "), which is the greatest lower semicontinuous function majorized by f'(x; ")), J~(x; d) = sup{{u, d): u E c3f(x)}.

(2.3)

If x belongs to the relative interior of d o m f , then Of(x) is nonempty, and if(x; ") is closed (i.e., ~' = f ' ) and proper; further, Of(x) is nonempty and bounded if and only if x belongs to the interior of d o m f in which case

if(x; d) is finite for all d, and closed.

P. Wolfe / A method of conjugate suboradients

149

The convex function f is called polyhedral if it has the form

f(x) = max{f/(x), i = 1. . . . . m},

(2.4)

where the functions f~ are affine, so that Vf~(x) = gi is constant for each i; f and f'(x; ") are closed and proper 9 Letting I(x) = {i: f/(x) = f(x)}

(2.5)

Of(x) = conv{9i: i e I(x)},

(29

for all x, where for any set S, conv S denotes the convex hull of S. Suppose that x e dom f, which consists of more than a single point, and that f~(x; d) r - o o for all d. Then f~(x; d) has a finite minimum value M on the unit ball {d: Idl < 1}, where I l denotes the Euclidean norm 9 If M > 0, then x minimizes f; otherwise, the value M is assumed for a unique d. To prove the latter statement, note that when M < 0 any minimizer d has norm 1, for Idl < 1 would imply f~(x; d/ldl) = if(x; d)/Id I = M/Id I < M. If there were two different minimizers dl, d2, we should have 189 + 89 < 1, while iT(x,9 8 9 89 < M since f ~ X( ; ' ) is convex. We will call the direction d thus defined the direction of steepest descent for f at x, and correspondingly define the gradient o f f at x to be

Vf(x) = if(x; d)d,

where Idl _< 1 minimizes J~(x; .).

(2.7)

(Note Vf(x) = 0 just when x minimizes f ) Of course, i f f is differentiable at x, then Vf(x) is just the ordinary gradient. For any set S g E" there is a unique point v in the closure of the convex hull of S having minimum norm; it will be denoted by Nr S. Algebraically, the point is characterized by the relation (v, s) >_ 1/312 for all s sS. The characterization below of the gradient was essentially given by Demjanov [4]: Vf(x) = - Nr af(x).

(2.8)

The proof uses the minimax theorem for a bilinear function on the product of two sets, one of which is bounded [20, Corollary 37.3.2], and (2.3): min f~(x; d) = min sup{(u, d): u E 0f(x)}

[dl < 1

[d[ < 1

=

~SoU~ ,

min ( u , d ) =

sup

l a l -< 1

=

-

min

u~i~Of (x)

lu[ =

...

-]Nr~f(x)].

m

lul


150

The saddlepoint of(u, d) is given by u = Nr 0f(x), d = -u/[u], from which (2.8) is immediate. Using formula (2.8) to calculate Vfis quite feasible w h e n f i s polyhedral and I(x) can be found, for by (2.6) Vf(x) = Nr{gi: i t I(x)},

(2.9)

so the problem of determining Vf(x) is the quadratic programming problem of finding the point of smallest norm in the convex hull of a given finite point set in E". We note that an algorithm especially designed for that calculation has been given [26]; it requires 89 + O(r/) cells of auxiliary storage to carry out. At a typical step of the algorithm of this paper, Nr G has already been found for some finite set G; a point 9 is determined, and Nr[G w {g}] must be found. The latter step requires O(n z) multiplications to perform. Unfortunately, the operator V does not have many useful properties. While V~ f = ~Vf for real ~, V(f + 9) # Vf + Vg (although 0(f + g) = Of + 0g in general), nor has Vf(x) any useful continuity properties we know of although 0fis upper semicontinuous [20, Corollary 24.5.1]). It may be worth mentioning that we use the Euclidean norm throughout only for convenience; but the convenience is great. We believe that all the theoretical results hold if]'] is taken as any strictly convex norm on R", and the induced dual norm is used for points of the dual space, in which Of(x) lies. The definition of Vf(x) would then be modified to designate that d for which minld I _ 0.

(2.12)

P. Wolfe / A method o f conjuoate subgradients

151

Note that Q is continuous and m o n o t o n e nondecreasing, and that lim Q(t) = if(x; d)/Id] 2.

(2.13)

t-+O+

We have supposed that for any x E d o m f we can obtain, in a uniquely determined way, some member 9 of Of(x). Given x and d :~ 0 and t > 0, let that member of Of(x + t d) be denoted by 9(0; we define

M(t)-

(9(t),d)

id[2

for t > 0.

(2.14)

M(') is m o n o t o n e nondecreasing, and

- f ' ( x + td; - d ) O. In that case we find in the course of doing the line search that (3.3) cannot be satisfied even for rather small t, and settle for 9+ ~ t?f(x + t d) for some small t, while taking x + = x. The line search algorithm of Section 4 shows, by exhibition, the existence of a finite process which can satisfy the requirements of step 4 of the algorithm. Being a bisection search, that is about its only virtue; for particular classes of problem we can devise much better searches, but we suppose they would all use the same idea: since the functions Q and M are nondecreasing, the set of all t > 0 satisfying (3.3) is an interval (possibly empty) and the set of all t > 0 satisfying (3.2) is an interval which, if not empty, extends to infinity. We show in Section 5 that if these intervals are nonempty, then they overlap in an interval of positive length, and bisection will find a point in their overlap, giving Case (A); otherwise, "stepping in" will end in Case (B) or "stepping out" will cause f ( x + t d)--, - o o .


157

4. The main algorithm The parameters e, 6, b, m 2 < m I < i, all positive, are fixed throughout. Initially we have the starting point Xo, some 9o ~ Of(xo), Go = {9o}, and the scalar ao = 0. The k 'h step of the algorithm constitutes one execution of the procedure below. See the end of Section 2 for the notation used.

Step 1. Set d = - N r G. If ]d[ < e, do step 2; else step 4. Step 2. If a < 8, stop; else do step 3. Step 3 (reset). Set x+ = x, 9+ = 9, G+ = {9}, and a+ = 0. Step k is finished. Step 4 (Line search: see below for details). Find t > 0 and g + ~ 8f(x + t d) such that (4.1) >- - m , l d l 2 and either Case A

f ( x + t d ) - f(x) 0, Q(t) < - m z } , R = { t : t > 0, M(t) >_ - m l } .

(4.4)

(a) Choose t > 0. Go to (b), (f), or (c) according as t ~ L\R, L ~ R, or R~L. (b) Replace t by 2t until t ~ R. Then if also t ~ L, go to ( f ) ; otherwise go to (d). (c) Replace t by i t until t[d[ 0, b > 0, | 6 > 0, 0 < mz < m~ < 89are given. Let S be the set of all points of R" lying within Euclidean distance 2b of (5.1) {x: f ( x ) 0. The Lemma is proved. Remark 5.1. The number of steps (each consisting of one function and one subgradient evaluation) the line search requires can be bounded if a lower bound for f is known. Let

f ( x ) - inf{f(y): y E S} _< D,

(5.2)

and choose the initial value of t in step (a) so that

b < t ld I [d[-2 (1

+

]d[2/4C 2)

=

[d[ -2 + 1/4C 2.

Repeated application of this inequality for k = K . . . . . L yields IdL[-z > ]dr[ -z + (L - K ) / 4 C 2, so that 1> 12>L-K e-~ - ~ - ~ 4C

1 + idol z

(5.6)

as long as no resets occur. The result follows. (ii) In Case A of step 4 of the algorithm we have f(Xk+l)

-- f ( X k ) 0, ~ 2~ = 1. Set in caseA,

yj=xj+tfl~

yj=xj

in C a s e B ,

for all j. Then g~ e Of(yj) for all j, and ]yj - x~] < b for all j, so [x - yj[ _< + b for allj. Let C' = m a x { [ g j [ : j e J } . Take TEE". By convexity, f ( Y ) - f(Yi) > ( g J, Y -- YJ} = (9i, Y - x} + (g j, x - yj} > ( g j, y - x ) -

c ' ( 6 + b)

for all j, whence multiplying by 2j and summing, f ( y ) - ~ 2H(yj) > - (d, y - x ) - C'(5 + b) >_ - I d [ ' [ y - x[ - C'(6 + b).

(5.9)

N o w f(yj) - f ( x ) > - C ' [ y j - x[ >>_ - C ' ( 6 + b) for all j, so ).jf(yj) - f ( x ) >_ - C'(6 + b).

(5.10)

Adding (5.9) and (5.10) we have f(y) -f(x)>

-]d I 9 ]y - x I - 2C'(6 + b),

(5.11)

which proves the theorem. Remark 5.2. The proof gives a crude bound on the number of steps required by the algorithm when it terminates. By (5.8), the total "path length" is bounded by [ f ( X o ) - f(Xk)]/m28. The path length between resettings must be at least 6, so [f(Xo) - f(xk)]/m2e6

(5.12)

162

P. Wolfe / A method of conju,qate subgradients

bounds the number of resettings and, by (i), 4C 2[f(xo) - f(xk)]/me~a6

(5.13)

bounds the total number of steps. When the bound D of (5.2) is known, the product of the expression (5.4), with e substituted for ]d[, and the bound on steps 4C2D/m2e3•

(5.14)

bounds the total number of function and gradient evaluations for the main algorithm. Remark 5.3. Paralleling a suggestion of Hestenes and Stiefel [12, Section 7], the point ~ = ~ 2jyj (see (iii) above for 2j) may be better than x = Xk: adding (5.9) to the relation ~ 2J(yj) > f(:~), we find that for any y ~ R" f ( y ) - f ( 2 ) >_ - ~ [ y - x[ - C'(,5 + b).

(5.15)

It seems sensible to replace x by 2 at each resetting when f(2) < f(x); the proof is not affected. While it is not clear that doing this will have a great effect in the general case, it appears to be a very good idea whenfis smooth. S u p p o s e f t o be twice differentiable at the solution x* of the problem and to have a nonsingular Hessian H there; then gj = Vf(yj) = H . (yj - x*) + O(]Y~ - x'l), whence - d = H . ( 2 - x*) + O(max~ ]yj - x'l). The degree of improvement using 2 over Xk is roughly the degree to which Idl is smaller than IVf(xk)l, which could be considerable. Remark 5.4. When the conjugate gradient method is used to minimize a strongly convex differentiable function, it is known that resetting every n or more iterations makes the procedure superlinearly convergent [16, 17], while otherwise it will usually converge linearly [3]. The proof above is not altered by such resettings, provided they are done infrequently enough to allow the resetting of Step 3 of the algorithm to happen at least once every I iterations for some fixed number I. However, a valuable example due to Powell [18] briefly mentioned in [19, Section 4] shows that we cannot in general expect better than linear convergence, even when f is piecewiselinear and exact line search is done. Let f ( x ) = max(ui, x), where ui = (cos 72i, sin 72i) for i = 0, 1, 2, 3, 4 (all angles in degrees). Contours o f f and a track of the main algorithm between restarts are shown in Fig. 6. Using polar coordinates Jr, 0], choose x0 = [ro, 36], so that (Uo, Xo) = ( u l , x l ) , and choose 9o = Uo. We find


163

X1 [/'1, 108], g , - U2, dl = -Nr{uo, u2} = Is, 252], x2 = [r2, 180], g2 U3, d2 = -Nr{u0, u2, u3} = 0. The isosceles triangles (0, Xo, x0, (0, xl. x2) are similar, the ratio of side to base being 89 + x/5), so that Ix] has been reduced by that factor in each of the two steps. We restart at Xz and find a similar path being traced again. =

=

u0

Fig. 6.

Powell's example.

Remark 5.5. The above example suggests strongly that one should not restart with the bundle G reset to only a single subgradient but rather to the set of all "active" subgradients--those which actually serve to determine the descent direction in the neighborhood of x. (That determination is easy if the subgradients generated after restarting are retained, as well as the corresponding numbers f(Xk) -- (gk, Xk), since 9k is active at x just when f ( x ) = f ( X k ) + (gk, X - Xk).) Since u2, ua are both active at x2 in the example, and - N r { u2, u3 } points directly from x2 to the origin, this device would solve the problem in one step after restarting. Such a modification seems promising and could hardly do any harm, but we do not think that it will in general yield a terminating procedure for the piecewise-linear case.

6. The algorithm and conjugate gradients The algorithm of Section 4 constitutes an extension of the method of conjugate gradients developed by Hestenes and Stiefel [12] for the solution of linear equation systems or, equivalently, for the minimization of convex quadratic functions. That fact was not at all apparent at first, but emerged empirically.

164


Having done most of the work reported above, we thought it as well to make sure that the algorithm would not do something actually ridiculous on smooth problems. Applying it to a quadratic function is particularly easy (the appropriate version of the algorithm for quadratics is given below). The algebra for a general problem of two variables is not hard to do, and we were surprised to find that the procedure terminated with the exact solution in two steps. The algebra for problems of more than two variables is tedious, so we formulated several numerical problems of three and four variables and ran them with a simple interactive computer routine. We were dumbfounded to find that the algorithm terminated in an apparently exact solution in each case, in as many steps as the number of variables. Since every known optimization scheme requiring only n linear minimizations to solve any quadratic problem in n variables involves conjugate directions, we knew where to look for an explanation. Suppose for a bundle G that - d ~ Nr G is not one of the vectors in G; then there is some subset G' ~_ G, having at least two members, such that - d E conv G' and (d, g') = (d, g")

for all g', g" ~ G'.

(6.1)

In the differentiable case. taking g' = Vf(x'), g" = Vf(x"), and x" = x' + t d' for suitable t, d' the relation (6.1) constitutes a workable definition of "conjugacy" between the directions d and d' even when f is not quadratic; for when f is the quadratic function (Q is a symmetric matrix)

f(x) = (p, x ) + 89

Q x),

(6.2)

we have V f ( x ) = p + Q x, so g " - g ' = t Qd', and (6.1) becomes just d Q d' = 0. (We find (6.1) first given essentially by Zoutendijk [28, Section 11.2].) As a matter of fact, our procedure for a quadratic problem is exactly the method of conjugate gradients. We could show that most quickly by referring to Hestenes and Stiefel: the points x and gradients g Cr" in [12]) generated are the same, and using the scaling of their Modification I [12, Section 9], our "d" is their "p'. We give, however, a self-contained demonstration, since we want the stronger than customary result that the method makes sense even whenfis not convex. (That fact does not seem to be generally known about the conjugate gradient method, but it emerges from a thoughtful reading [12, Section 10, first paragraph].) We restate the algorithm of Section 4 for the case t h a t f i s the quadratic function given by (6.2). It can be greatly simplified; effectively the parameters e, 6, rn~, m2 are all set to zero, and resetting is not needed.


165

Algorithm for a quadratic objective T h e starting point x o is given. Set Go = {go} = {Vf(xo)}, and p e r f o r m the steps b e l o w for each k = 0, 1. . . . 'until stopped in step 2 or step 3.

Step 1. Set d = - N r G. Step 2. I f ( d , Qd> / ( d , Qd>, x+ = x + td, g+= Vf(x+)=g + tQd. If g+ = 0, stop. Step 4. C h o o s e H _ G a n d s e t G + = H w { - d , g + } .

Lemma 2. Let the vectors go. . . . . gk be nonnull and mutually orthogonal, and let u = Nr{go . . . . . gk-,},V = Nr{go, . . . , gk}. Then (u, gi> = lul ~ # 0 for j < k, and v = Nr{u, gk}. Proof.

W e m a y write (uniquely) u = )-'4 < k 2fli with ~ 2 i = 1 and 2 i > 0,

>_ lul 2 for a l l j < k. Since (u, gi) = ~ilgil 2 and Igil r 0 for allj < k and 2 i # 0 for some j, we k n o w u # 0, and thus 2 i > 0 for j < k. gi) - In[ 2] = (u, u> - ( E 2j)lul 2 = 0 then implies (u, gi> [ul a for a l l j < k. N o t e 2 i = lul2/Igil ~ for j < k Similarly writing v = ~iXk#jgj, we have ~j --Ivl=/Igil 2 for j < k, so

2j

Nondifferentiable optimization (Mathematical programming study)

Combinatorial Optimization (Mathematical programming study)

Methods of Descent for Nondifferentiable Optimization

Optimality and Stability in Mathematical Programming (Mathematical programming study)

Point-to-set Maps and Mathematical Programming (Mathematical programming study)

Mathematical programming

Introduction to mathematical optimization

Mathematical Theories of Optimization

Combinatorial Optimization II (Mathematical Programming Studies, Vol 13)

Introduction to Mathematical Optimization: From Linear Programming to Metaheuristics

Integer Programming and Combinatorial Optimization

Mathematical Programming at Oberwolfach

Applied Mathematical Programming

Mathematical Foundations of Programming

Network models and associated applications (Mathematical programming study, No. 15)

Mathematical Optimization and Economic Theory

Mathematical optimization and economic analysis

Mathematical optimization and economic theory

Mathematical optimization and economic theory

Mathematical optimization & Economic theory

Quasidifferential Calculus (''Mathematical Programming Studies '')

Mathematical Programming: Theory and Methods

The Mathematical Experience, Study Edition

Mathematical Programming and Electrical Networks

Decomposition techniques in mathematical programming

Mathematical Programming with Data Perturbations

Mathematical Programming: Theory and Methods

Integer Programming and Combinatorial Optimization, 15 conf

Mathematical optimization in computer graphics and vision

Mathematical Optimization in Computer Graphics and Vision

Combinatorial Data Analysis: Optimization by Dynamic Programming

Nondifferentiable optimization (Mathematical programming study)

Combinatorial Optimization (Mathematical programming study)

Methods of Descent for Nondifferentiable Optimization

Optimality and Stability in Mathematical Programming (Mathematical programming study)

Point-to-set Maps and Mathematical Programming (Mathematical programming study)

Mathematical programming

Introduction to mathematical optimization

Mathematical Theories of Optimization

Combinatorial Optimization II (Mathematical Programming Studies, Vol 13)

Introduction to Mathematical Optimization: From Linear Programming to Metaheuristics

Integer Programming and Combinatorial Optimization

Mathematical Programming at Oberwolfach

Applied Mathematical Programming

Mathematical Foundations of Programming

Network models and associated applications (Mathematical programming study, No. 15)

Mathematical Optimization and Economic Theory

Mathematical optimization and economic analysis

Mathematical optimization and economic theory

Mathematical optimization and economic theory

Mathematical optimization & Economic theory

Quasidifferential Calculus (''Mathematical Programming Studies '')

Mathematical Programming: Theory and Methods

The Mathematical Experience, Study Edition

Mathematical Programming and Electrical Networks

Decomposition techniques in mathematical programming

Mathematical Programming with Data Perturbations

Mathematical Programming: Theory and Methods

Integer Programming and Combinatorial Optimization, 15 conf

Mathematical optimization in computer graphics and vision

Mathematical Optimization in Computer Graphics and Vision

Combinatorial Data Analysis: Optimization by Dynamic Programming

Recommend Documents