Introduction to Sensitivity and Stability Analysis in Nonlinear Programming (Mathematics in Science and Engineering, 165)

Introduction to Sensitivity and Stability Analysis in Nonlinear Programming This is Volume 165 in MATHEMATICS IN SCIE...

Author: Anthony Fiacco

72 downloads 666 Views 4MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Introduction to Sensitivity and Stability Analysis in Nonlinear Programming

This is Volume 165 in MATHEMATICS IN SCIENCE AND ENGINEERING A Series of Monographs and Textbooks Edited by RICHARD BELLMAN, University o f Southern California The complete listing of books in this series is available from the Publisher upon request.

Introduction to Sensitivity and Stability Analysis in Nonlinear Programming ANTHONY

V.

FIACCO

Operations Research Department Institute for Management Science and Engineering School of Engineering and Applied Science The George Washington University Washington, D.C.

1983

A C A D E M I C PRESS A Subsidiary of Harcourt Brace Jovanovich, Publishers

Paris San Diego

New York London O Sydney Tokyo Toronto San Francisco S ~ Paul0

Academic Press Rapid Manuscript Reproduction

COPYRIGHT @ 1983, BY ACADEMIC PRESS, INC. ALL RIGHTS RESERVED. NO PART OF THIS PUBLICATION MAY BE REPRODUCED OR TRANSMITTED IN ANY FORM OR BY ANY MEANS, ELECTRONIC OR MECHANICAL, INCLUDING PHOTOCOPY, RECORDING, OR ANY INFORMATION STORAGE AND RETRIEVAL SYSTEM, WITHOUT PERMISSION IN WRITING FROM THE PUBLISHER.

ACADEMIC PRESS, INC.

111 Fifth Avenue, New York. New York 10003

Uniied Kingdom Edition published by ACADEMIC PRESS, INC. (LONDON) LTD.

24/28 Oval Road, London

NWI 7DX

Library of Congress Cataloging in Publication Data

Fiacco. Anthony V . Introduction to sensitivity and stability analysis in nonlinear programming. (Mathematics in science and engineering) Bibliography: p . Includes index. I . Nonlinear programming. I . Title. I I . Series. T57.8. F53 1982 519.7'6 82-1 I642 ISBN 0-12-254450-1

PRINTED IN THE UNITED STATES OF AMERICA 83 84 85 86

9 8 7 6 5 4 3 2 1

To Sarah, Alicia, Tonino, Ma, Pa, and Clara

This Page Intentionally Left Blank

Contents

Preface

xi

I . Overview Chapter I . Motivation and Perspective

3

Chapter 2 . Basic Sensitivity and Stability Results 2.1 2.2 2.3 2.4 2.5 2.6 2.7

Introduction 8 Objective Function and Solution Set Continuity Differential Stability 22 36 Implicit Function Theorem Results 54 Optimal Value and Solution Bounds 60 General Results from RHS Results Summary 62

11

II. Theory and Calculation of Solution Parameter Derivatives Chapter 3 . Sensitivity Analysis under Second- Order Assumptions 3.1 Introduction 67 3.2 First-Order Sensitivity Analysis of a Second-Order Local Solution 68 80 3.3 Examples 3 . 4 First- and Second-Order Parameter Derivatives of the Optimal Value Function 82

Chapter 4 . Computational Aspects of Sensitivity Calculations: The General Problem 4.1 Introduction

91 vii

viii

Contents

4.2 Formulas for the Parameter First Derivatives of a Karush-KuhnTucker Triple 91 4.3 Applications and Examples 106

Chapter 5 . Computational Aspects: RHS Perturbations 5.1 Introduction I16 5.2 The Use and lnitial Interpretation of Lagrange Multipliers 117 5.3 Examples of Early Sensitivity Interpretations of Lagrange Multipliers 123 125 5.4 Supporting Theory 5.5 Formulas for the Parameter First Derivatives of a Karush-KuhnTucker Triple and Second Derivatives of the Optimal Value Function 127 I36 5 . 6 Examples and Applications

ZIZ. Algorithmic Approximations Chapter 6 . Estimates of Sensitivity Information Using Penalty Functions

6.1 Introduction 155 6.2 Approximation of Sensitivity Information Using the Logarithmic156 Quadratic Mixed Barrier-Penalty Function Method 6.3 Examples of Estimates of Solution Point and Lagrange Multiplier Parameter Derivatives 168 6.4 Extensions 171 6.5 Sensitivity Calculations Based on the Perturbed Karush-KuhnTucker System 173 179 6 . 6 Optimal Value Function Sensitivity Estimates 6.7 Example of Estimates of Optimal Value and First- and SecondParameter Derivatives 182 185 6.8 Sensitivity Approximations for RHS Perturbations 6.9 Recapitulation I90

Chapter 7 . Calculation of Sensitivity Information Using Other Algorithms

7.1 Introduction I94 7.2 Connections between Algorithmic and Sensitivity 196 Calculations 7.3 Algorithmic Calculations of the Inverse of the Jacobian of the Karush-Kuhn-Tucker System 20 1 206 7.4 Sensitivity Results for Augmented Lagrangians 7.5 Conclusions and Extensions 218

ix

Contents

N. Applications and Future Research Chapter 8. An Example of Computational Implementations: A Multi-Item Continuous Review Inventory Model 8. I 8.2 8.3 8.4 8.5

Introduction 225 226 Screening of Sensitivity Information 229 Example Sensitivity Calculations by SENSUMT 232 A Multi-Item Inventory Model Additional Computational Experience with Applications

243

Chapter 9. Computable Optimal Value Bounds and Solution Vector Estimates for Parametric NLP Programs 9.1 Introduction 246 9.2 Computable Piecewise Linear Upper and Lower Optimal Value Bounds 247 9.3 Estimates of a Parametric Solution Vector and a Sharper Convex 249 Upper Bound 9.4 Connections between Optimal Value Bounds and Duality 255 9.5 Nonlinear Dual Lower Bounds 259 9.6 Extensions 265 268 9.7 Bounds on a Solution Point 273 9.8 Further Extensions and Applications

Chapter 10. Future Research and Applications 10. I Recapitulation and Other Research Directions 10.2 Future Research Directions and Applications 10.3 Conclusions 285

Appendix I . Notation, Conventions, and Symbols Appendix II. References

Lemmas, Theorems, Corollaries, Definitions, and 32 1 Examples 329

Selected Bibliography of Works Not Cited Author Index Subject Index

289

357 36 1

346

277 28 1


Preface

Mathematical programming sensitivity and stability results are now abundant and far-reaching, but they are scattered in the literature. This has seriously diluted the impact of these results and impaired their systematic implementation, a deficiency compounded by the relatively small effort that has been made to make these results intelligible and accessible to nonmathematicians. Our purpose with this volume is to initiate seriously the process of unification and explication and to communicate the practical applicability and usefulness of many of these results to practitioners. It is now feasible to develop software that allows for the calculation of sensitivity and stability information whenever a solution to a nonlinear programming (NLP) problem is calculated. It seems clear that nonlinear programming will not be a widely used or practical decision-making technique without this capability. (Witness the routine use of post-optimality analysis in linear programming.) Further, sensitivity and stability calculations and solution algorithm calculations often involve similar data and the same manipulations, so it appears natural and efficient to calculate the former as a by-product of the latter. We strongly support this approach and illustrate it via the detailed calculation of sensitivity and stability information by a classical penalty function method. Similar exploitation is indicated for Newton and projected- and reduced-gradient methods and for augmented Lagrangian algorithms. It should be stressed, however, that the basic sensitivity and stability results are presented in terms of the given problem functions and independently of any algorithmic manipulations. Our goal is therefore the advancement of both theoretical unification and practical implementation. Numerous results and treatments are published here for the first time. This book will therefore be of equal interest to practical problem solvers and developers of methodology. The treatment is sweeping with respect to general developments and detailed for results that follow under well-known smoothness assumptions. The latter results are highly structured and readily implemented, also providing a rich foundation and a clear perspective for results under more general assumptions.

xi

xii

Preface

This book is directed primarily, but certainly not exclusively, to researchers and practitioners who utilize mathematical programming models. Both mathematicians and operations researchers should find the developments of particular interest in applications. The numerous references, supplemented by the bibliography of works not cited in the text, should be of great value. The book has been used by the author as the text for a graduate course in NLP sensitivity and stability analysis and could also be utilized as a supporting text for mathematical programming courses or seminars on advanced topics. It should provide a valuable reference for anyone interested in the sensitivity and stability analysis of nonlinear algebraic equations. In this treatment, the term sensitiviry information is often taken to mean program parameter derivatives of the optimal value function or a “Karush-Kuhn-Tucker triple” of a parametric nonlinear program. Srabiliry information here refers primarily to parametric bounds on the optimal value function or solution points. To our knowledge this is the first book devoted entirely to general sensitivity and stability results in nonlinear programming. Others will surely follow as the results are further expounded and unified. Important results in mathematical programming theory and algorithms have been and undoubtedly will continue to be developed using perturbation methods. It is a pleasure to acknowledge several vital sources of encouragement and support. Richard Bellman originally suggested the idea of collecting these results into a single treatment. My dissertation students have contributed significantly, as collaborators, implementers, and constructive critics. Section 3.4, Chapters 4 and 5, Sections 6.5-6.8, and Sections 8.1-8.4 are based on papers coauthored with Robert Armacost. Sections 2.1-2.4 are based largely on a survey coauthored with William Hutzler. The optimal value bounds results reported in Section 9.2 were first implemented in collaboration with Abolfazl Ghaemi. Several valuable suggestions and numerous corrections were obtained from Jerzy Kyparisis, who carefully read the entire manuscript. He also collaborated in the development of the results reported in Section 9.7 and in collecting the list of references in the selected bibliography. Continued support has been forthcoming from the Operations Research Department and the Institute for Management Science and Engineering of the George Washington University. This work was also partially supported by the Army Research Office, the Office of Naval Research, and the National Science Foundation.

Overview


Chapter I

Motivation and Perspective

A

methodology for conducting a local perturbation

(sensitivity) analysis and finite perturbation (stability) analysis of solution behavior with respect to problem changes is a well established requirement of any scientific discipline A sensitivity and stability analysis should be an integral

part of any solution methodology.

The status of a solution

cannot be understood without such information.

This has been

well recognized since the inception of scientific inquiry and has been explicitly addressed from the beginning of mathematics.

Important branches of mathematics have been inspired

by perturbation and approximation questions, and entire journals and mathematical treatises are devoted to such subjects.

In mathematical programming, sensitivity and stability

techniques have been used to obtain optimality conditions, duality results, solution algorithms, convergence and rate of convergence proofs, and acceleration of convergence of algorithms, in addition to their more obvious and immediate applications in estimating near solutions with different data. Such techniques are also essential in more mundane applications, e.g., in model validation and cost-benefit analysis.

3

4

I Motivation and Perspective

In linear programming (LP), the theoretical and computational results regarding the effects of parameter changes have been reasonably standardized under appropriate nondegeneracy assumptions, with the now well known and routinely used "postoptimality" (parametric) sensitivity and "range" (solution invariance) analysis being standard options in most LP computer codes.

In the evolution of linear programming method-

ology, the concern with a parametric analysis of a solution came early, and the theoretical and computational aspects were developed and refined along with the solution procedures. This did not happen in nonlinear programing (NLP). Concentration of effort was initially devoted to the development of solution algorithms.

Unlike linear programming, where the

simplex method soon dominated and unified theoretical and computational efforts, no "optimal" computational approach was dominant in nonlinear programming.

Numerous solution strate-

gies were proposed, and a host of theoretical and computational problems arose in developing first- and second-order optimality conditions, competitive numerical algorithms, and convergence and rate of convergence characterizations. The fragmentation of effort in the development of nonlinear programming methodology was reflected in the neglect of the development of sensitivity and stability results, except sporadically and somewhat abstractly.

Certainly, a unified

theory of stability analysis required some unification in the characterization of optimality and convergence theory.

Fortu-

nately, this unification began about a decade ago and with it came a serious concentrated effort to develop a sensitivity and stability analysis methodology.

The references corrob-

orating this fact are now, happily, too numerous to cite, but

5


it is obvious to anyone who has followed developments in nonlinear programming that sensitivity and stability analysis has reached a sophisticated and respectable level. An impressive body of theoretical results has been developed for nonlinear programming.

As for computational imple-

mentations, many of the most incisive results are too abstract and general to be of immediate use in practical applications, and much work is needed to interpret and specialize these to obtain useful computable methods.

We feel there is a great

need for a systematic exploitation of problem structure and for the efficient interfacing of perturbation calculations with "algorithmic" (solution algorithm) calculations.

The

development and computational implementation of standard accessible techniques for conducting nontrivial sensitivity analyses in NLP has not been widespread and lags appreciably behind the present level of sophistication cf the theory. Nonetheless, nontrivial implementations do exist, and it is time to collect and unify the basic procedures as well as the existing rich body of theory. It is hoped that this book will contribute to this unification.

It is only a first step, and much remains to be done.

We begin by providing an overview of many of the important basic results in Chapter 2, using a common terminology and notation and making some efforts to indicate important relationships.

Given this perspective, Chapters 3

-

8 are con-

cerned essentially with the calculation of the parameter derivatives of the optimal value solution vector and associated optimal Lagrange multipliers of general classes of parametric nonlinear programs satisfying standard ideal second-order


6

(twice continuous differentiability) assumptions.

Nonalgo-

rithmic (problem-oriented) and algorithmic (solution-methodoriented) results are given, the emphasis being ultimately on the provision of simple computable algorithmic methods and results that might be useful in practical applications. Chapter 9 provides a simple but effective approach for calculating parametric bounds on the optimal value function and a parametric feasible solution vector estimate, and also presents preliminary results that utilize the optimal value bounds to derive parametric solution vector bounds.

These re-

sults are based essentially on the exploitation of convexity properties, particularly the convexity (or concavity) of the optimal value function, under appropriate assumptions.

As

with the calculation of the parameter derivatives, the bounds results are computable with information already typically generated by most NLP algorithms.

Thus, sensitivity and stabil-

ity calculations can invariably be closely tailored to solution algorithm calculations and performed as side calculations as progress is made towards a solution.

This approach and its

practical applicability are demonstrated in detail using a well known classical penalty-function algorithm.

The idea is

readily applied to any solution algorithm and adaptations to augmented Lagrangian methods, reduced gradient methods, and projected gradent methods are given, in addition to immediate connections with recursive techniques based on Newton’s method. Chapter 10 concludes with a brief survey of potentially fruitful research directions and applications. It is hoped that the results presented in Chapter 2 will convince the reader that the theory of sensitivity and stability in mathematical programming is sophisticated, incisive,


7

and c o m p r e h e n s i v e .

T h e r e i s no a s p e c t o f t h e s u b j e c t t h a t t h e

r e s u l t s do n o t touch

-

i n d e e d , it i s c l e a r t h a t t h e e n t i r e

s u b j e c t c a n be d e v e l o p e d from t h e p e r t u r b a t i o n a l p o i n t of view. T h i s i s n o t s u r p r i s i n g , o n c e it i s r e a l i z e d t h a t many fundamental r e s u l t s i n mathematics can be proved u s i n g p e r t u r b a t i o n techniques.

The p o i n t w e w i s h t o make i s t h a t s e n s i t i v i t y and

s t a b i l i t y r e s u l t s cannot and w i l l n o t be c o n f i n e d simply t o t h e c l a s s i c a l c o n c e r n of w h e t h e r " t h e s o l u t i o n c h a n g e s c o n t i n u o u s l y w i t h continuous changes i n t h e d a t a . " w i l l o f c o u r s e h a v e t o be a d d r e s s e d .

Such q u e s t i o n s

But a c o m p l e t e p e r t u r b a -

t i o n t h e o r y i s completely encompassing, and hence w e e x p e c t t h a t e v e r y i m p o r t a n t r e s u l t i n m a t h e m a t i c a l programming w i l l be r e v i s i t e d from t h i s p e r s p e c t i v e . W e a l s o hope t h a t C h a p t e r s 2

-

1 0 convince t h e r e a d e r t h a t

u s e f u l c o m p u t a b l e t e c h n i q u e s a r e known a n d c a n now be implemented a n d i n t e r f a c e d w i t h any s t a n d a r d NLP a l g o r i t h m i c c a l c u lations.

T h e r e would a p p e a r t o b e no t h e o r e t i c a l o b s t a c l e t o

d e v e l o p i n g t h e s o f t w a r e , f o r t h e i n d i c a t e d s e n s i t i v i t y and s t a b i l i t y calculations, t h a t interfaces with.the calculations r e q u i r e d by a n y o f t h e s t a n d a r d NLP c o d e s .

Chapter 2

Basic Sensitivity and Stability Results

2.1.

INTRODUCTION Comprehensive treatments of linear programming sensitivity

analysis and parametric linear programming (i.e., obtaining a solution as a function of problem parameters) have been forthcoming (Dinkelbach, 1969; Noiieka, Guddat, Hollatz, and Bank, 1974; Gal, 1979) along with basic sensitivity results (parameter derivatives) in quadratic programming (Boot, 1963). Unified treatments of nonlinear parametric programming methodology have only very recently begun to emerge (Bank, Guddat, Klatte, Kummer, and Tamer, 1982; Brosowski, 1982). Activity in the study of sensitivity and stability of solutions of general mathematical programs to problem perturbations has been sporadic and results are scattered in the literature.

This chapter endeavors to give a flavor of the

state of the art in this area.

Many important contributions

are necessarily omitted in this overview, which is confined to parametric perturbations of nonlinear programming problems. This and the following section, and most of Sections 2.3 and 2.4, are based on Fiacco and Hutzler (1979b), though numerous definitions, additional results, and refinements have been introduced. X

9

2. I Introduction

the next two examples demonstrate, the solution of very

As

simple mathematical programs may vary smoothly or change drastically for arbitrarily small perturbations of the problem parameters. Example 2 . 1 . 1

Consider the nonlinear program minx(xl

-

s.t. x2

+ (x, + 1) 2

€1 xl,

x2 2 -x1' The problem may be viewed as seeking x(f) region) that minimizes the distance from

E

(E,

R(E)

(the feasible

-1) to R ( f ) .

A

depiction and three solutions are illustrated in Fig. 2.1.1.

(c

=-2)

Fig. E

= -2

J

2.1.1. 0, 2.

D e p i c t i o n o f Example 2 . 1 . 1

and s o l u t i o n s f o r

10

2 Basic Sensitivity and Stability Results

The a n a l y t i c a l s o l u t i o n of t h i s p r o b l e m is e a s i l y s e e n t o be

1

( ( 1+ E ) / 2 ,

T

~ ( € 1

[ X l ( E ) 1 x ~ ( E ) I=

=

X(E)

-

1)/2,

< -1

E

- 1 ( E ( 1

(0, 0) r ((E

It is clear that

-(1 + E ) / 2 ) ,

(E

-

1)/2),

> 1.

E

i s p i e c e w i s e l i n e a r , c o n t i n u o u s , and

d i f f e r e n t i a b l e everywhere e x c e p t f o r

E

= 21.

I t is r e a d i l y

shown t h a t t h e o p t i m a l v a l u e f u n c t i o n o f t h i s p r o b l e m , f

*

(E)

= f[x(E),

€1,

i s convex ( d e f i n e d j u s t a f t e r Theorem

and t w i c e d i f f e r e n t i a b l e everywhere e x c e p t f o r

2.2.4),

E

= 21,

where it i s o n l y o n c e d i f f e r e n t i a b l e . U n f o r t u n a t e l y , as t h e n e x t example i l l u s t r a t e s , t h e s o l u t i o n s o f m a t h e m a t i c a l p r o g r a m s do n o t a l w a y s b e h a v e s o n i c e l y . E x a m p l e 2 . I. 2 min

X

EX

s . t . x 1. -1. The s o l u t i o n o f t h i s p r o b l e m i s g i v e n by E

> 0;

and i f

X(E) E

X(E)

c a n b e c h o s e n a s any v a l u e i n [-1,

m)

= -1 i f

if

E =

0;

< 0 t h e r e i s no f i n i t e s o l u t i o n of t h i s p r o b l e m .

Thus, f * ( E ) =

-E,

v a l u e s of

Thus, a s

E.

0 and h a s no l o w e r bound, f o r t h e r e s p e c t i v e E

v a r i e s i n a s m a l l neighborhood o f t h e

o r i g i n i n E1 t h e s o l u t i o n may b e f i n i t e and u n i q u e , it may b e unbounded, o r t h e r e may b e i n f i n i t e l y many s o l u t i o n s . I t s h o u l d b e c l e a r from t h e s e two s i m p l e e x a m p l e s t h a t

v e r y s m a l l p e r t u r b a t i o n s of t h e parameters o f a mathematical program c a n c a u s e a w i d e v a r i e t y o f r e s u l t s .

The p u r p o s e o f

t h i s c h a p t e r i s t o summarize and i l l u s t r a t e t h e work t h a t h a s b e e n d o n e t o d a t e i n p r o v i d i n g c o n d i t i o n s u n d e r which t h e

II

2.2 Objective Function and Solution Set Continuity

s o l u t i o n s o f n o n l i n e a r programs a r e l o c a l l y w e l l behaved and i n e s t i m a t i n g s o l u t i o n p r o p e r t i e s a s a f u n c t i o n o f problem parameters. The r e a d e r n o t f a m i l i a r w i t h o t h e r s p a c e s and norms may s u b s t i t u t e En f o r a l l m o r e g e n e r a l s p a c e s and t h e u s u a l E u c l i d e a n norm f o r o t h e r norms i n t h e f o l l o w i n g r e s u l t s . 2.2

O B J E C T I V E FUNCTION AND SOLUTION SET C O N T I N U I T Y

Some o f t h e e a r l i e s t work i n s t a b i l i t y a n a l y s i s f o r nonl i n e a r programming was c o n c e r n e d w i t h t h e v a r i a t i o n of t h e o p t i m a l v a l u e f u n c t i o n w i t h changes i n a parameter appearing i n t h e r i g h t hand s i d e o f t h e c o n s t r a i n t s , i . e . ,

involving

problems of t h e form:

s . t . g ( x ) 1.

min X f ( x ) , where f : E n point-to-set

-+

E

1

,

g: En

+

P1

E,

Em and

i s i n Em.

E

(E)

The t h e o r y of

maps ( B e r g e , 1963) h a s been u s e d f o r much o f t h e

a n a l y s i s o f t h i s problem.

Hogan (1973d) h a s p r o v i d e d a n

e x c e l l e n t development o f t h o s e p r o p e r t i e s of p o i n t - t o - s e t

maps

t h a t are e s p e c i a l l y u s e f u l i n d e r i v i n g such r e s u l t s . Next w e p r e s e n t s e v e r a l d e f i n i t i o n s and p r o p e r t i e s rel a t i n g t o point-to-set important r e s u l t s .

maps t h a t are needed i n a number o f

The terms "map," "mapping," and " p o i n t - t o -

s e t map" a r e u s e d i n t e r c h a n g e a b l y i n t h e s e q u e l . Given t w o t o p o l o g i c a l s p a c e s T and X I a p o i n t - t o - s e t mapping

r

from T t o X i s a f u n c t i o n which associates w i t h

e v e r y p o i n t i n T a s u b s e t o f X. say t h a t t h e point-to-set

mapping

F o l l o w i n g Berge ( 1 9 6 3 ) , w e

r

i s c o n t i n u o u s a t to

E

T

i f it i s b o t h u p p e r s e m i c o n t i n u o u s and lower s e m i c o n t i n u o u s a t

12

2 B a w Sensitivity and Stability Reaulta

These l a s t two n o t i o n s a r e e s t a b l i s h e d by t h e f o l l o w i n g

to.

d e f i n i t i o n , which w e i n c l u d e f o r c o m p l e t e n e s s .

r

Let

Definition 2.2.1.

be a p o i n t - t o - s e t

mapping from T

t o s u b s e t s of X.

r

(i)

open s e t S

i s lower s e m i c o n t i n u o u s a t t o e T i f , f o r e a c h

c

X satisfying S

n

r ( t0) #

t h e r e e x i s t s a neigh-

borhood N of t o , N ( t O ) , s u c h t h a t f o r e a c h t i n N ( t O ) ,

r(t)

n

s

(ii)

z

pr.

r

i s upper s e m i c o n t i n u o u s a t t o E T i f , f o r e a c h

open s e t S C X c o n t a i n i n g

r ( t0 )

t h e r e e x i s t s a neighborhood N

of t o , N ( t O ) , s u c h t h a t f o r e a c h t i n " t o ) ,

r

Furthermore, i f

r ( t )C

S.

i s lower s e m i c o n t i n u o u s a t e a c h p o i n t o f

t h e n it i s s a i d t o be lower s e m i c o n t i n u o u s i n T; and i f

T,

i s upper s e m i c o n t i n u o u s a t e a c h p o i n t of T w i t h

f o r each t , t h e n

r

r(t)

r

compact

i s s a i d t o be upper s e m i c o n t i n u o u s i n T.

The mapping l' is s a i d t o be c o n t i n u o u s i n T i f i t is b o t h upper and lower s e m i c o n t i n u o u s i n T. D e f i n i t i o n s , s i m p l e r and g e n e r a l l y easier t o a p p l y , t h a t a r e closely r e l a t e d t o these semicontinuity properties f o r point-to-set

r

Let

maps a r e based on t h e p r o p e r t i e s o f t h e i r g r a p h s .

be a p o i n t - t o - s e t

t h e g r a p h G of

r

is

mapping from T t o s u b s e t s o f X.

G 5

UteTUxEr(t)( t , x ) , i . e . ,

Then

G = { ( t , x)

T x X ~ EX r ( t ) ) .

L e t tn e T be s u c h t h a t tn + to. Then

D e f i n i t i o n 2.2.2.

(i)

r

i s open a t to e T i f , f o r e a c h xo

E

r ( t O ) there

e x i s t s a v a l u e m and a sequence {xn) C X s u c h t h a t xn e T ( t n ) f o r e a c h n > m and xn (ii)

xn

.+

r

-f

xo.

i s c l o s e d a t to e T i f xn

xo t o g e t h e r imply t h a t xo

E

E

l'(tn) f o r e a c h n and

r (to).

E

13

2.2 Objective Function and Solution Sct Continuity

The map

r

i s s a i d t o b e o p e n ( c l o s e d ) i n T i f i t i s open

( c l o s e d ) a t e a c h p o i n t o f T.

r

I t may a l s o b e n o t e d t h a t

is

c l o s e d i n T i f and o n l y i f i t s graph is a c l o s e d set i n T

x

X.

An a l t e r n a t i v e t o t h e Berge d e f i n i t i o n o f c o n t i n u i t y o f t h e map

r

i s g i v e n i n t h e s e t t i n g o f open a n d c l o s e d maps by

Hogan ( 1 9 7 3 d ) .

The map

r

i s s a i d t o be c o n t i n u o u s a t t o E T

i f it i s b o t h open a n d c l o s e d a t t o , and i t i s s a i d t o b e c o n t i n u o u s i n T i f i t i s b o t h o p e n a n d c l o s e d i n T.

The d e f i -

n i t i o n t h a t i s a p p l i c a b l e i n t h e f o l l o w i n g r e s u l t s w i l l be

c l e a r from t h e c o n t e x t , i . e . ,

t h e framework f o r a g i v e n r e s u l t

w i l l i n v o l v e e i t h e r s e m i c o n t i n u i t y c o n d i t i o n s o r open-closed c o n d i t i o n s , b u t n o t both.

I n t h e f o r m e r case, t h e Berge

d e f i n i t i o n s w i l l a p p l y a n d i n t h e l a t t e r case t h e Hogan d e f i n i t i o n s w i l l apply. The r e l a t i o n s h i p between t h e s e d e f i n i t i o n s may b e found i n Hogan ( 1 9 7 3 d ) , a n d i s summarized a s f o l l o w s :

( i )t h e d e f i n i -

t i o n s o f l o w e r s e m i c o n t i n u o u s a n d o p e n a t a p o i n t are e q u i v a l e n t , a n d ( i i )i f

r

i s u n i f o r m l y compact n e a r t o , i . e . ,

if

t h e r e i s a n e i g h b o r h o o d N of to s u c h t h a t t h e c l o s u r e o f t h e

setuteNI'(t) i s compact, t h e n l' i s c l o s e d a t t o i f a n d o n l y i f

r ( t0 )

is compact a n d

r (t)is

I t is clear t h a t i f

upper semicontinuous a t to.

r(tO)is

upper semicontinuous a t to, t h e n

a c l o s e d s e t and

r

e v e r , t h e converse does n o t hold.

r(t)

is

m u s t b e c l o s e d a t t o . HowThe f o l l o w i n g v e r y s i m p l e

example e x h i b i t s t h i s d i s t i n c t i o n between u p p e r s e m i c o n t i n u o u s and c l o s e d , i n t h e absence of t h e compactness assumptions. C o n s i d e r f : E1 0 i f x = 0.

-t

E1 d e f i n e d by f ( x ) = l / x i f

x # 0 and f (x) =

I t may r e a d i l y b e v e r i f i e d t h a t f i s c l o s e d b u t

n o t upper semicontinuous a t x = 0.

14


F o l l o w i n g Berge ( 1 9 6 3 ) , f o r r e a l - v a l u e d t h e n o t a t i o n l s c and

USC

functions we use

f o r lower and u p p e r s e m i c o n t i n u i t y ,

These a r e d e f i n e d , a c c o r d i n g t o t h e c u s t o m a r y

respectively.

convention, a s follows.

L e t @ be a r e a l - v a l u e d

f i n e d on t h e t o p o l o g i c a l s p a c e X.

f u n c t i o n de-

Then

( i ) @ i s s a i d t o be l s c a t a p o i n t x o

E

X if

E

X if

@ ( x o ,)

l i m inf $(x)

x'x 0 ( i i ) @ i s s a i d t o be u s c a t a p o i n t x o

lim

SUP

X'Xo

@ ( X I 2 @(xo).

The f u n c t i o n @ i s s a i d t o b e c o n t i n u o u s a t xo i f i t i s b o t h lsc and

USC

e a c h p o i n t of X.

a t xo and l s c ( u s c ) on X i f @ i s l s c ( u s c ) a t

I f @ i s continuous a t xo, then

l i m i n f @ ( x ) = l i m s u p @ ( x )= l i m @ ( X I . X'X0 x+xo x+xO Using t h e s e d e f i n i t i o n s , t h e f o l l o w i n g r e s u l t s a r e e a s i l y established f o r real-valued

f u n c t i o n s (Berge, 1 9 6 3 ) .

Assume

X and T a r e t o p o l o g i c a l s p a c e s i n t h e n e x t two r e s u l t s .

Theorem 2.2.1.

d e f i n e d on X

x

I f f is a real-valued

USC

(lsc) function

T and i f R i s a l o w e r s e m i c o n t i n u o u s

(upper

s e m i c o n t i n u o u s ) mapping from T i n t o X s u c h t h a t f o r e a c h T, R ( E )

#

PI,

then t h e (real-valued) function f

f * ( E ) = i n f x { f ( x , € 1 ~x

is

USC

E

*

,

E

in

d e f i n e d by

~ ( € I1

(lsc).

Theorem 2 . 2 . 2 .

If f is a continuous real-valued

d e f i n e d on t h e s p a c e X i n t o X such t h a t R ( E ) f

x T

function

and R i s a c o n t i n u o u s mapping o f T f o r each

E

i n T, then t h e

15


(real-valued) function f f*(E) = infx{f(x,

i s c o n t i n u o u s i n T. s(E)

= {X

E)

* , defined ~x e

R(E)I

F u r t h e r m o r e , t h e mapping S , d e f i n e d by

R(E)I~(X,

E

by

E)

= f*(E)},

i s a n u p p e r s e m i c o n t i n u o u s mapping of T i n t o X. C o n d i t i o n s t h a t imply t h e c o n t i n u i t y o f t h e s o l u t i o n o f a m a t h e m a t i c a l program h a v e b e e n g i v e n by D a n t z i g , Folkman, and S h a p i r o ( 1 9 6 7 ) a n d by Robinson a n d Day ( 1 9 7 4 ) .

L e t t i n g f be a

f u n c t i o n from a m e t r i c s p a c e X t o E l , w i t h R ( E ) C X , and d e f i n i n g S ( E ) = {x D a n t z i g e t aZ.

E

R(E) If(x) = infz{f(z)I z E

(1967) o b t a i n c o n d i t i o n s f o r

S

R(E)}}

t o vary i n a

When R i s d e f i n e d by l i n e a r i n e q u a l i t i e s , t h e y

c l o s e d manner.

o b t a i n u n d e r a p p r o p r i a t e c o n d i t i o n s t h a t S i s a c l o s e d mapping. Under t h i s same h y p o t h e s i s , l e t t i n g S

*

d e n o t e t h e mapping S

when it i s a s i n g l e t o n , t h e y o b t a i n c o n d i t i o n s which y i e l d t h e c o n t i n u i t y of S domain.

*

a s a function of t h e parameter

The domain D o f a p o i n t - t o - s e t

defined a s D = { t e T I T ( t ) #

map

r:

in its

E,

T

-+

X is

PI.

The f o l l o w i n g c l a s s i c a l c o n c e p t of p o i n t w i s e t o p o l o g i c a l

l i m i t s of a s e q u e n c e {R 1 o f s u b s e t s o f a m e t r i c s p a c e X i s n utilized. D e f i n e t h e i n n e r l i m i t and t h e o u t e r l i m i t o f {Rn}, respectively, a s

9Rn n-

and

{

: x

-

l i m Rn : { x n+m

If

9Rn n+m

-

E

XI 3 xn

E

XI 3 xn

= l i m Rn = R ,

n+m

Rn

E

E

j

Rnj

f o r n 2 n* and xn

x}

+.

f o r j 1. j* and x n

+

j

x}.

t h e n t h e l i m i t R o f {Rn} i s s a i d t o

16


e x i s t and w e w r i t e

l i m Rn = R n+m

or

Rn

R.

+

A c o n n e c t e d s e t M i s s u c h t h a t t h e r e do n o t e x i s t d i s j o i n t

open sets A1 and A2 s u c h t h a t M C A1 U A 2 , M

# 9, a n d

A1

8.

M n A 2 #

L e t R be a point-to - s et

Theorem 2 . 2 . 3 .

mapping from t h e

metric s p a c e T t o t h e s e t o f s u b s e t s o f E n , w i t h R ( E ) a c l o s e d set f o r each

E

i n T.

L e t D be t h e domain o f R.

is a connected s e t f o r each

*

R ( E ) i s compact.

{ c n } C D,

-+

t h e mapping S

*

E

i n D, a n d f o r some

+

E

*

i n D,

F u r t h e r m o r e , assume t h a t f o r e v e r y s e q u e n c e i m p l i e s R(cn)

E*

is continuous a t

+

E

*

R(E*).

,

if n

I f g i s a n a f f i n e f u n c t i o n from E Ax

Suppose R ( E )

b , where A i s a n m

E

*

Then, i f f

c o n s t a n t v e c t o r , a n d i f F$,(g)

= {x

C(En),

i s i n i t s domain.

t o E ~ i, . e . ,

n c o n s t a n t m a t r i x and b

x

E

i f g(x) = E

Em i s a

Mlg(x) 2 0 ) where M C En,

E

t h e f u n c t i o n g is s a i d t o be nondegenerate w i t h r e s p e c t t o t h e set M i f %(g)

h a s a nonempty i n t e r i o r and no component of g

i s i d e n t i c a l l y zero. SM(g) = { x

E

The c o n t i n u i t y o f t h e map d e f i n e d by

F$,(g) I f ( x ) = i n f z { f ( z ) I z

E

F $ , ( g ) ) } and S i ( t h e

mapping SM when i t i s a s i n g l e t o n ) a s f u n c t i o n s o f g i s g i v e n by t h e f o l l o w i n g t h e o r e m s . x, y

M imply t h a t a x

E

+

(1

The s e t M i s s a i d t o b e convex i f

-

cx)y

E

M f o r any

CL E

[o, 11.

I f f i s c o n t i n u o u s , g i s a f f i n e , and M i s

Theorem 2 . 2 . 4 .

c l o s e d and c o n v e x , t h e n SM i s c l o s e d a t e v e r y n o n d e g e n e r a t e p o i n t g. A f u n c t i o n 4 i s s a i d t o be convex o n a convex s e t M i f

@ ( a x+ (1 any a

E

-

10,

a @ ( x )+ (1

a)y) 11.

-

a ) @ ( y )f o r any x , y

A f u n c t i o n @ i s quasi-convex

i f and o n l y i f { x

E

E

M and

on a convex s e t M

M l @ ( x )5 k} i s convex f o r a n y r e a l number

17


E q u i v a l e n t l y , $ ( x ) i s q u a s i - c o n v e x on M i f m a x { $ ( x ) , $ ( y ) }

k.

>

$(Ax

+

(1

-

X)y) f o r e v e r y x , y

M and every X

E

[O,

E

11.

It

f o l l o w s t h a t a convex f u n c t i o n i s q u a s i - c o n v e x . I f t h e a s s u m p t i o n s o f Theorem 2 . 2 . 4

Theorem 2 . 2 . 5 .

hold

*

and f i s quasi-convex or % ( g )

i s bounded, t h e n SM i s con-

t i n u o u s a t a n o n d e g e n e r a t e p o i n t g of i t s domain. M o t i v a t e d by a p p l i c a t i o n s t o t h e t h e o r y o f economic c h o i c e , Robinson and Day ( 1 9 7 4 ) , c o n s i d e r i n g a g e n e r a l c o n s t r a i n t s e t R ( E ) , p r o v i d e c o n d i t i o n s t h a t g u a r a n t e e t h e c o n t i n u i t y of t h e

point-to-set

mapping whose v a l u e a t

is t h e set of solutions

E

o f t h e m a t h e m a t i c a l program minx f ( x ,

E)

s.t. x

E

R(E)

E

E

T,

where R ( E ) r e p r e s e n t s a c o n s t r a i n t s e t as a f u n c t i o n o f t h e parameter

E,

a n d X a n d T are t o p o l o g i c a l v e c t o r s p a c e s .

t h a t end, l e t f mopping S: T

+

*

(€1

=

infx{f (x,

X by S ( E ) = { x E

E)

Ix

E

To

R ( E ) ~ ,and d e f i n e t h e

R ( E ) If ( x , E ) = f* ( E )

1.

A f u n c t i o n $ is s a i d t o be s t r i c t l y quasi-convex on a

convex s e t M i f , f o r a l l xl, and a l l h

E

( 0 , l), $(Ax,

+

x2 e M such t h a t $(x,) (1

-

A)xZ) < $ ( x 2 ) .

$(x2)

A convex

f u n c t i o n i s also s t r i c t l y quasi-convex. A s s u m e t h a t t h e s p a c e X i s l o c a l l y convex

Theorem 2.2.6.

and t h a t R ( E ) # fl f o r e a c h convex-valued o n T ( i . e . f (x,

E)

= min{f

1

(x,

E)

,

,

E

T and t h a t R i s continuous and

R ( E ) i s convex f o r e a c h E i n T ) .

If

f 2 ( E ) 1 , where f l i s c o n t i n u o u s o n X x

T and s t r i c t l y q u a s i - c o n v e x

c o n t i n u o u s o n T.

E

i n x f o r each f i x e d

E,

a n d f 2 is

t h e n S i s c o n t i n u o u s a n d convex-valued on T.

18


A s might be expected,

stronger s t a b i l i t y characterizations

w e r e i n i t i a l l y found f o r t h e o s t e n s i b l y s i m p l e r r i g h t hand s i d e ( r h s ) problems P1(€)

and P 2 ( € ) .

(However, i t i s n o t e d i n

may be r e f o r m u l a t e d S e c t i o n 2 . 6 t h a t t h e g e n e r a l problem P 3 ( ~ ) a s a n e q u i v a l e n t r h s problem o f t h e form P2(€). ) R e c a l l t h a t t h e i n e q u a l i t y - c o n s t r a i n e d r h s p r o b l e m is

min X f ( x ) where now f : En P

(E)

1

are:

s . t . g ( x ) 1. E1

+

P1 ( € 1

E

a n d g: En

-+

A s s o c i a t e d w i t h problem

Em.

( i )t h e f e a s i b l e r e g i o n , R ( E ) = { x

(ii)t h e set D =

E E m l R ( ~ )#

{E

t h e e f f e c t i v e domain o f f * ;

g } , i.e.,

E

n

E Ig(x) 2 E);

t h e domain o f R a n d

( i i i )a s e t I ( E )= { x E E n l g ( x ) >

€ 1 a s s o c i a t e d w i t h t h e i n t e r i o r of t h e f e a s i b l e r e g i o n ; and ( i v ) t h e o p t i m a l v a l u e f u n c t i o n f * ( E ) = i n f x I f ( x ) Ix

R ( E ) ~ ,

E

a l s o c a l l e d t h e " m a r g i n a l f u n c t i o n " when t h e p e r t u r b a t i o n s are rhs perturbations. The f o l l o w i n g t h r e e t h e o r e m s , d u e t o Evans a n d Gould

(1970), p r o v i d e c o n d i t i o n s f o r t h e s t a b i l i t y o f t h e c o n s t r a i n t set, i.e.,

t h e f e a s i b l e r e g i o n R ( E ) , a s w e l l a s f o r t h e con-

t i n u i t y of the optimal value function.

I n the statements

of t h e f o l l o w i n g r e s u l t s i n t h i s s e c t i o n w e w i l l d e n o t e by R a point-to-set

mapping from t h e s e t D C Em t o t h e s e t o f a l l

s u b s e t s of En, w i t h t h e v a l u e a t

E

i n D g i v e n by R ( E ) .

The

i n t e r i o r o f t h e s e t D i s d e n o t e d by D 0 , a n d t h e c l o s u r e of I (E)

-

is d e n o t e d by I ( E )

.

A s s u m e g is continuous.

T h e o r e m 2.2.7. ( i ) The mapping R i s u p p e r s e m i c o n t i n u o u s a t

only i f t h e r e e x i s t s a vector

E'

0.

C2 f o r a l l j , They a l s o

show t h a t t h e c o n d i t i o n s g i v e n i n ( i ) ,a l o n g w i t h f

E

C'

and

22


convex imply t h a t t h e map S 6 of t h e s e t S 6 ( € ) o f & - o p t i m a l s o l u t i o n s i s u n i f o r m l y l i n e a r l y c o n t i n u o u s on To f o r e a c h

6 > 0 and a l s o o b t a i n t h e n e x t c o n t i n u i t y r e s u l t f o r S 6 . Theorem 2 . 2 . 1 3 .

quasi-convex, continuous a t

I f f a n d -gi,

i = 1,

R ( F ) i s bounded, a n d I ( T )

..., m,

# 0 , then

are s t r i c t l y S6

is

f o r each 6 > 0.

Hager (1979) r e c e n t l y o b t a i n e d L i p s c h i t z r e s u l t s f o r q u a d r a t i c programs w i t h u n i q u e s o l u t i o n s , assuming t h a t t h e s e e x i s t f c r s m a l l p e r t u r b a t i o n s a n d t h a t t h e p e r t u r b a t i o n parame t e r v a r i e s o v e r a convex s e t .

Lipschitz c o n t i n u i t y of t h e

s o l u t i o n and m u l t i p l i e r v e c t o r o f s u c h programs f o l l o w s u n d e r t h e hypothesis t h a t t h e g r a d i e n t s of t h e binding c o n s t r a i n t s s a t i s f y t h e l i n e a r i n d e p e n d e n c e c r i t e r i o n (CQ3, S e c t i o n 2 . 3 ) . I n both i n s t a n c e s an e s t i m a t e of t h e Lipschitz constant is provided. L i p s c h i t z p r o p e r t i e s of t h e o p t i m a l v a l u e f u n c t i o n and e s t i m a t e s o f t h e C l a r k e ( 1 9 7 5 ) g e n e r a l i z e d g r a d i e n t w e r e obt a i n e d by Gauvin ( 1 9 7 9 ) f o r problem P 2 ( € ) ( S e c t i o n 2.3)

and

e x t e n d e d by Gauvin a n d Dubeau ( 1 9 7 9 ) t o problem P 3 ( € ) ( S e c t i o n 2.3).

E x t e n s i o n s i n a more g e n e r a l s e t t i n g h a v e b e e n p r o v i d e d

r e c e n t l y by G o l l a n ( 1 9 8 1 a , b ) and R o c k a f e l l a r ( 1 9 8 2 a , b ) . 2.3.

DIFFERENTIAL STABILITY I n t h i s s e c t i o n w e c o n c e n t r a t e on t h e o r y t h a t h a s b e e n

developed t o a n a l y z e v a r i o u s s t a b i l i t y p r o p e r t i e s of t h e o p t i mal v a l u e f u n c t i o n , e v o l v i n g from a n a n a l y s i s of t h e r a t e o f change of t h i s f u n c t i o n a t a s o l u t i o n p o i n t .

W e begin w i t h a

b r i e f d i s c u s s i o n o f s e v e r a l w e l l known c o n s t r a i n t q u a l i f i c a t i o n s t h a t a r e t y p i c a l l y i n v o k e d i n o b t a i n i n g many o f t h e results.

23

2.3 Differential Stability

The constraint qualifications used in mathematical programming are regularity conditions which are generally imposed to insure that the set of Karush-Kuhn-Tucker multipliers (Karush, 1939; Kuhn and Tucker, 1951) corresponding to an optimal solution of a mathematical program is nonempty.

We

present here five qualifications that are frequently applied. These and a number of others are treated in some detail by Mangasarian (1969).

Throughout this discussion we shall assume

the constraint set R E {x g : En

Em and h: En

-+

vector functions. V' and , ' V

-+

En( g(x) 2 0, h(x) = GI, where

E

Ep are once continuously differentiable

Define B(x) E {il g 1 . (x) = 0).

The operators

without subscripts, denote differentiation with

respect to x. The Mangasarian-Fromovitz constraint qualification

CQl.

(Mangasarian and Fromovitz, 1967) is said to hold at a point

*

x

R if:

E

(i) there exists a vector z for all i

E

En such that Vgi(x*)z > 0

..., p, and 1, ..., are linearly

B(x*), Vh. (x*)z = 0 for j = 1,

E

7

(ii) the gradients {Vh.(x*), j = 3

independent. This condition is equivalent to (CQ1)': C

+ CP

3=1

x

E

*

ieB(x ) w.Vh.(x*) = 0 has no nonzero solution ui 2 0, w 3

3

*

uivgi(x 1 1'

for

R. If the gi are concave (or even pseudo-concave) functions

and the h . are affine, then CQ1 is equivalent to the well known 3

"Slater condition" (Slater, 1950), a general form of which we give as CQ2.

24


If the function 4: En

Definition 2.3.1.

+

E1 is differ-

entiable on the convex set M and @(y) 2 $(XI for all x, y with V@(x) (y

-

E

M,

x) 1. 0, then 4 is said to be pseudo-convex on M.

(The function 4 is said to be pseudo-concave if - 4 is pseudoconvex. ) A differentiable convex function is pseudo-convex and a

pseudo-convex function is both quasi-convex and strictly quasi-convex. The Slater constraint qualification is satisfied at

CQ2.

*

x

R if h

is affine for each j, gi is pseudo-concave for j each i and t.here exists a point jT e R with gi(x) > 0 for each E

i in B(x CQ3.

*

at x j

=

E

*

).

The linear independence assumption is said to hold

*

R if the gradients cVgi(x 1 ,

l,...,p} CQ4.

c

*

i

E

*

B(x 1; Vhj(x*),

are linearly independent.

If there are no equality constraints and uiVgi(x

*

)

=

*

0 has no nonzero solution ui L 0, for x

ieB(x ) * R, the Cottle constraint qualification is said to hold at x

B

.

(In the absence of equality constraints, CQ1 is equivalent to CQ4 1 CQ5.

The Kuhn-Tucker constraint qualification (Kuhn and

*

Tucker, 1951) is satisfied at x E R if, for each nonzero * * vector z e En satisfying Vgi(x ) z 2 0 for each i E B(x ) and

*

Vh.(x ) z = 0, j = 1, 3

..., p,

tiable arc originating at x

*

z is tangent to a once-differen-

and contained in R.

The relationships that hold among these qualifications, in addition to those already mentioned, are that CQ3 implies CQ1, which, in turn, is sufficient for CQ5. which also implies CQ5.

Also, CQ2 implies CQ4,

For a proof and further discussion of

25


the relationships among these constraint qualifications see Arrow, Hurwicz, and Uzawa (1961), Bazaraa, Goode, and Shetty (1972), Mangasarian (1969), and Peterson (1.973). Robinson (1976b) has shown the equivalence of CQ1 and a form of local stability of the set of solutions of a system of inequalities.

Gauvin (1977) has shown that CQ1 is both neces-

.

sary and sufficient for the set of Lagrange multiplier vectors corresponding to a given local solution of a general NLP problem to be nonempty, compact, and convex.

In addition, Gauvin

and Tolle (1977) established that CQ1 is preserved under rhs perturbations.

If CQ3 holds, it is well known that the

Lagrange multiplier vector, corresponding to a given solution, exists and is unique. One of the earliest characterizations of the differential stability of the optimal value function of a mathematical program was provided by Danskin (1966, 1967). problem of minimizing f(x,

E)

Addressing the

subject to x e R, R some topo-

logical space, Danskin derived conditions under which the directional derivative of the optimal value function exists, and also determined its representation. D e f i n i t i o n 2. 3.2.

The (one-sided) directional derivative

of the function $ at the point t in the direction z is defined to be: DZ$(t) = lim [@(t + B z ) B+O+

-

$(t)l/B

if the limit exists. Theorem 2 . 3 . 1 .

Let R be nonempty and compact and let f

and the partial derivatives af/aEi be continuous. point

E

Then at any

in Ek and for any direction z e Ek the directional

26


d e r i v a t i v e of f D f

*

Z

(E)

=

*

e x i s t s a n d i s g i v e n by

min VEf(x, X€S(E)

where S ( E ) = cx E R l f ( x ,

E)

E)Z,

= f*(E)).

This r e s u l t has wide a p p l i c a b i l i t y i n t h e s e n s e t h a t t h e c o n s t r a i n t s p a c e R c a n b e any compact t o p o l o g i c a l s p a c e .

It

h a s b e e n e x t e n d e d by a number o f a u t h o r s , i n c l u d i n g Dem'yanov a n d Rubinov ( 1 9 6 8 ) , t o o t h e r s p a c e s a n d a v a r i e t y o f f u n c t i o n a l forms.

The p r i n c i p a l r e s t r i c t i o n o f t h i s r e s u l t i s t h a t t h e

set R does n o t v a r y w i t h t h e parame-ter

E.

However, s i n c e i n -

e q u a l i t y a n d e q u a l i t y c o n s t r a i n t s c a n be " a b s o r b e d " i n t o t h e o b j e c t i v e f u n c t i o n o f a program t h r o u g h t h e u s e of a n a p p r o p r i a t e a u x i l i a r y function, (Lagrangian, p e n a l t y f u n c t i o n , e t c . )

,

Danskin's r e s u l t is o f t e n a p p l i c a b l e t o a u x i l i a r y f u n c t i o n I t c a n a l s o be r e a d i l y a p p l i e d t o t h e o b j e c t i v e

methods.

f u n c t i o n o f t h e d u a l o f a convex program w i t h r h s p e r t u r b a tions. E

T u r n i n g t o t h e p r o b l e m where R d e p e n d s o n a p a r a m e t e r

through t h e i n e q u a l i t y g ( x ,

-gi

E)

L

0 , a n d f a n d t h e components

a r e convex, Hogan (197333) h a s shown t h a t D z f ^ ( e ) e x i s t s k The f o l l o w i n g t h e o r e m p r e s e n t s and i s f i n i t e f o r a l l z e E

.

t h e d e t a i l s of t h i s r e s u l t f o r f*(E) = i n f x { f ( x , g(xi

€ 1 2 0 1 , where

M is a s u b s e t of En.

t h i s problem i s d e f i n e d a s L ( x , u ,

E)

E)

Ix

E

M,

The L a g r a n g i a n f o r

:f ( x ,

E)

-

T

u g ( x , E).

F o r c o n v e n i e n c e a n d w i t h o u t l o s s o f g e n e r a l i t y , w e s h a l l assume t h a t t h e parameter v a l u e of i n t e r e s t i s

E

= 0,

unless otherwise

stated. Theorem 2 . 3 . 2 .

f a n d t h e -gi

L e t M be a c l o s e d and convex s e t .

are convex on M f o r e a c h f i x e d

t i n u o u s l y d i f f e r e n t i a b l e on M borhood o f

E

k = 0 in E

.

x

N(0)

,

E

Suppose

and a r e con-

where N ( 0 ) i s a n e i g h -

I f S(0) : { x E MI g ( x , 0) 1. 0 a n d

27


f * ( O ) = f ( x , 0)) i s nonempty and bounded, there is a point

X

E

M s u c h t h a t g ( Z , 0) > 0 , t h e n D Z f * ( 0 )

e x i s t s and i s f i n i t e f o r a l l z

*

DZf (0) =

f * ( O ) i s f i n i t e , and

E

E

k , and

min max VJEL(x,u , 0 ) z x e S ( 0 ) ueK(x,O) min

-

max

{ ( v E f ( x , 0)

-

uTv E g ( x , o ) ) z } ,

x e S ( 0 ) ueK(x,O) where K ( x , 0 ) i s t h e s e t o f o p t i m a l L a g r a n g e m u l t i p l i e r s f o r t h e given x

E

S(0).

F o r convex programs t h e s e t K ( x , 0 ) i s t h e same f o r e x h x

This does n o t allow us t o drop t h e minimization

S(0).

E

over x

E

S(0)

*

i n t h e e x p r e s s i o n f o r DZf (0), however, s i n c e

t h e q u a n t i t y VEL(x, u , 0)z b e i n g m i n i m i z e d w i l l g e n e r a l l y deHowever, it d o e s a l l o w f o r a c o n s i d e r a b l e economy

pend on x .

i n t h e c a l c u l a t i o n of DZf

*

s i n c e t h e c o n s t r a i n t s e t K ( x , 0)

a s s o c i a t e d w i t h t h e i n n e r m a x i m i z a t i o n p r o b l e m d o e s n o t depend on x . Some r e c e n t i n v e s t i g a t i o n s of t h i s s o r t h a v e f o c u s e d on t h e extremal value function inequality-equality o p t i m i z a t i o n problems w i t h r h s p e r t u r b a t i o n s ,

of t h e form

c o n s i s t e n t w i t h o u r n o t a t i o n f o r P 3 ( ~ ,) where x E~

E

EpI

E

=

( E ~ ,E

~ E

)E

k

( ~k = m

+

constrained

E

En,

E~

E

Eml

p) i n t h e following r e s u l t s

i n t h i s section, unless otherwise specified.

28


W e a l s o define, f o r R ( E ) #

8,

S ( E ) = {X E R ( E ) If(x) =

f * ( E ) l , and t h e L a g r a n g i a n L ( x , u , w,

+ wT [ h ( x )

-

E

*I .

f(x)

E)

- uT [ g ( x ) -

= 0 , and i f CQ1 h o l d s f o r some x* E

1

*

.

I f R ( 0 ) # jlf, w i t h R u n i f o r m l y compact n e a r

Theorem 2 . 3 . 3 .

continuous a t

1

Given t h e s e d e f i n i t i o n s , Gauvin and T o l l e

(1977) have proved t h e f o l l o w i n g c o n t i n u i t y p r o p e r t y o f f

E

E

E

S(O),

then f * is

= 0.

F i a c c o (1980b,c) and Gauvin and Dubeau (1979) showed t h a t t h i s r e s u l t h o l d s f o r t h e g e n e r a l problem minx f ( x ,

E)

s.t. g ( x ,

E)

2 0,

h(x,

E)

= 0,

where t h e problem f u n c t i o n s a r e C1 i n ( x ,

E),

R ( E ) , and S ( E )

a r e d e f i n e d t o be t h e f e a s i b l e r e g i o n and s o l u t i o n sets o f P 3 ( ~ ) ,r e s p e c t i v e l y , and t h e p a r a m e t e r

E

is a vector i n E

k

.

I n t h e a b s e n c e o f e q u a l i t y c o n s t r a i n t s , R o c k a f e l l a r (1974) h a s shown t h a t , under c e r t a i n second-order c o n d i t i o n s , t h e function f

*

o f P 2 ( ~ )s a t i s f i e s a s t a b i l i t y o f d e g r e e two

condition, i.e.,

i n a neighborhood o f

twice-differentiable and f * ( O ) = Cp(0).

f u n c t i o n Cp:

Em+'

E

= 0 there e x i s t s a

-t

E1 w i t h f * ( E ) 2 Cp ( E )

Under t h i s s t a b i l i t y p r o p e r t y , bounds on t h e

d i r e c t i o n a l d e r i v a t i v e s o f f * (when t h e y e x i s t ) c a n be d e r i v e d . For convex programming problems o f t h e form P 3 ( ~ ) ,G o l ' s t e i n (1972) h a s shown t h a t a s a d d l e - p o i n t c o n d i t i o n i s s a t i s f i e d by t h e d i r e c t i o n a l d e r i v a t i v e of f*.

Gauvin and T o l l e ( 1 9 7 7 ) ,

n o t assuming c o n v e x i t y , b u t l i m i t i n g t h e i r a n a l y s i s t o problem P * ( E ) , e x t e n d t h e work o f G o l ' s t e i n and p r o v i d e s h a r p bounds

on t h e D i n i upper and lower d e r i v a t i v e s o f f

*

, also

r e q u i r i n g t h e e x i s t e n c e o f second-order d e r i v a t i v e s .

without These

29


r e s u l t s w e r e e x t e n d e d by F i a c c o a n d H u t z l e r ( 1 9 7 9 a ) t o t h e general inequality-constrained

problem a n d by F i a c c o ( 1 9 8 0 ~ )

and Gauvin and Dubeau (1979) t o t h e more g e n e r a l problem The Gauvin-Dubeau e x t e n s i o n

P 3 ( ~ ) ,a n d a r e p r e s e n t e d n e x t .

follows t h e approach of Auslender (1979), w h i l e t h e a u t h o r ' s a p p r o a c h i s b a s e d o n e l i m i n a t i n g t h e e q u a l i t y c o n s t r a i n t and using elementary arguments t o d e a l w i t h t h e r e s u l t i n g s i m p l e r i n e q u a l i t y - c o n s t r a i n e d problem. Let L(x, u, w,

E)

:f ( x ,

E)

-

T

u g(x,

E)

+ wTh ( x ,

denote

E)

t h e u s u a l L a g r a n g i a n of p r o b l e m P 3 ( ~ )a n d l e t K ( x , 0) d e n o t e t h e s e t o f Karush-Kuhn-Tucker a s o l u t i o n x of P

3

f * ( O ) ] / B , where z

(E) E

at

E

v e c t o r s (u, w) corresponding t o

= 0,

*

l e t Q ( z , B ) = [f (62)

-

A s above, R ( E )

Ek i s a u n i t v e c t o r .

d e n o t e s t h e f e a s i b l e r e g i o n , S ( E ) t h e s o l u t i o n s e t , and t h e f u n c t i o n s f , g , h a r e assumed j o i n t l y o n c e c o n t i n u o u s l y d i f f e r e n t i a b l e i n (x, problem P 3 ( E )

E).

The f o l l o w i n g r e s u l t s h o l d f o r

.

The D i n i u p p e r a n d l o w e r d e r i v a t i v e s are d e f i n e d a s

l i m sup B+O+

Q ( z , B ) and l i m i n f Q ( z , B ) B+0+

, respectively.

I f R ( 0 ) # g, R i s u n i f o r m l y compact n e a r k E = 0 , a n d C Q 1 h o l d s f o r some x* E S(0) t h e n , f o r any z E E , Theorem 2 . 3 . 4 .

l i m inf Q(z, 8) 2 min VEL(X B +O+ ( u , w ) E K ( x * ,0) The n e x t c o r o l l a r y ,

*

,

u , w,

0)z.

(2.3.1)

f o l l o w i n g i m m e d i a t e l y from t h e theorem,

g i v e s a r e s u l t t h a t i s weaker, b u t u s e f u l i n t h e s e q u e l . I f R ( 0 ) # 8, R i s u n i f o r m l y compact n e a r k = 0 , and CQ1 h o l d s a t e a c h x E S O), t h e n , f o r any z E E , CoroZZary 2.3.5.

E

l i m inf Q ( z , B) 2 B+O+

inf min V E L ( X , u , w , 0)z. ( 2 . 3 . 2 ) x e S ( 0 ) ( u , w ) e K ( x 0)

30

Z Basic Sensitivity and Stability Results

Under the conditions of Corollary 2.3.5,

T h e o r e m 2.3.6.

for any z

E

Ek ,

lim s u p Q(z, 0 ) 5 0+0+

inf max VEL(xf u , w, 0 ) z . x e S ( 0 ) (u,w)EK(x,O)

CoroZZary 2.3.7.

(2.3.3)

If, in the hypotheses of Corollary 2.3.5,

CQ1 is replaced with CQ3, then for each z

E

Ek , DZf*(0) exists

and (2.3.4) where [u(x), w(x) I is the unique optimal Lagrange multiplier vector associated with x

E

S(0).

Auslender (1979) also obtained these bounds for the rhs perturbation problem P2(€), extending the results of Gauvin and Tolle (1977) by using a weaker form of the MangasarianFromovitz constraint qualification and adapting implicit func tion theorem results due to Hestenes (1975).

This allows him

to replace the differentiability assumption on the objective and inequality functions with the weaker requirement that they be locally Lipschitz.

Fontanie (1980) used the author's re-

duction approach in addressing an extension of Auslender's results to allow general perturbations.

Utilizing concepts of

subdifferential analysis and generalized derivatives, Rockafellar (1982a) has provided general bounds and sharper results under weaker assumptions than those given heretofore. Further significant extensions of these results have been provided by Rockafellar (1982b), utilizing a second-order constraint qualification and saddle-point properties of augmented Lagrangians.

31


The following theorem (Fiacco and Hutzler, 1979a; Fiacco, 1980b) corresponds, under slightly different assumptions, to results obtained by Gol'stein (1972) and Hogan (197333) (Theorem 2.3.2) for a general class of problems of the form P 3 ( ~ )that are convex in x.

..., m be convex functions in x, and let the functions h j = 1, ..., p be jr Corollary 2 . 3 . 8 .

Let f and -gi, i = 1,

affine in x, with all functions jointly C1 in (x,

# pI, R is uniformly compact near for each x

*

DZf (0) =

E

*

E

E).

If R(0)

= 0, and CQ1 is satisfied

S(O), then DZf (0) exists for each z

E

Ek , and

inf max VEL(x, u, w, 0)z. xeS(0) (u,w)~K(x,O)

(2.3.5)

As noted following Theorem 2.3.2, the expression for (2.3.5) is theoretically and computationally simplified by noting that K(x, 0)

Z

K(O), a set that is the same for each x

E

S(0).

Although we are focusing attention on programs for which the spaces involved are finite dimensional, we note that most of these sensitivity results have been extended to infinite dimensional programs.

For example, Maurer (1977a,b) obtained

a characterization of the directional derivative of the extremal value function subgradient for problem P 3 ( ~ , ) and has applied his results to a class of optimal control problems. Lempio and Maurer (1980) have obtained similar bounds under analogous assumptions that are required to handle general perturbed infinite-dimensional programs of the form minimize f(x,

E),

subject to x

E

M1 and g(x,

are arbitrary closed convex sets.

E)

E

M2, where M1 and M2

Other extensions in

32


infinite-dimensional spaces may be found in the works of Dem'yanov and Pevnyi (1972), Gollan (1981a,b), Levitin (1976), Rockafellar (1982a,b), and others. If the inequalities (2.3.2) and (2.3.3) are applied to problem P2(€), then we obtain the result

(

inf min uTzl xeS(0) (u,w)~K(x,O)

< lim sup Q(z, B ) B+O+

5

+

wTz2) 5 lim inf ~ ( z ,B ) +O'B

inf

xeS(0)

max (uTzl (u,w)eK(x,O)

+ wTz 2 )

obtained by Gauvin and Tolle (19771, where we have defined z = (zl, z2)T The equation (2.3.4) resulting from the addi-

.

tional assumption of linear independence (CQ3) becomes DZf*(0) =

T T inf (u(x) z1 + w(x) z2). xes (0)

These relationships explicitly link the rate of change of the optimal value function f * with the optimal Lagrange multipliers (u, w).

In particular, they show that, under the given

*

assumptions, the rate of change of f

in any direction z is

bounded if the union of the set of optimal Lagsange multipliers associated with S(0) is bounded.

It is clear that many addi-

tional valuable insights might be obtained from this connection, in deducing optimal value stability properties from optimal multiplier properties and conversely. This line of inquiry has been extremely fertile.

We next

indicate some results, relating to this connection, that provide an elegant unification of important necessary optimality conditions. For the problem min

X

f(x)

s.t. g(x) 2 x e M

E,

Pi(€)

33


where M i s a convex s u b s e t o f En,

f i s convex and t h e gi a r e

concave, G e o f f r i o n (1971) g i v e s a clear p r e s e n t a t i o n of s o m e basic results. f

*

H e f i r s t notes t h a t t h e optimal value function

i s convex.

For t h e r e s u l t of i n t e r e s t , t h e following d e f i -

n i t i o n s are needed.

$(W)

A v e c t o r v i s s a i d t o be a s u b g r a d i e n t o f 6 a t t h e

is finite. point

L e t 6 be a convex f u n c t i o n a n d assume

W if @ ( Z )+ (x

@(x)

-

W)Tv f o r a l l x .

The program P ' ( E ) i s c a l l e d " s t a b l e " i f f 1 t h e r e e x i s t s c > 0 such t h a t

*

(E)

i s f i n i t e and

T h i s c o n d i t i o n was a p p a r e n t l y f i r s t i n t r o d u c e d by G a l e (1967) t o obtain duality results.

Assuming P i ( € ) h a s a s o l u t i o n ,

G e o f f r i o n shows t h a t P i ( € ) i s s t a b l e i f and o n l y i f a n o p t i m a l Karush-Kuhn-Tucker

multiplier u

*

e x i s t s , and u

m u l t i p l i e r f o r P i ( € ) i f and o n l y i f u f

*

at

*

*

is an optimal

i s a subgradient of

E.

V a r i a n t s o f t h e l a s t r e s u l t h a v e a p p a r e n t l y been known f o r

some t i m e .

F o r example, R o c k a f e l l a r ( 1 9 7 0 ) shows t h a t i f

P i ( € ) i s a convex program s a t i s f y i n g S l a t e r ' s c o n d i t i o n a n d

has a s o l u t i o n , t h e n t h e s t a t e d r e s u l t h o ld s . R o c k a f e l l a r (1967) u s e d a n e q u i v a l e n t n o t i o n of s t a b i l i t y o f a problem o f t h e f o r m min

X

-

f (x)

s.t. x

E

M

g(Ax) and

Ax

f

To.

Here, A i s a l i n e a r t r a n s f o r m a t i o n from a r e a l ( f i n i t e o r

infinite-dimensional) vector space X t o another similar space T,

f i s a f i n i t e - v a l u e d convex f u n c t i o n o n M C X I where M Z

34


and c o n v e x , a n d g i s a f i n i t e - v a l u e d c o n c a v e f u n c t i o n o n To C T f o r To # fl a n d convex. be c a l l e d

The p e r t u r b e d p r o b l e m c o n s i d e r e d w i l l

F ( E )a n d i s t h e above problem w i t h t h e o b j e c t i v e

-

function f ( x )

g(Ax

-

E),

where

E

f i r s t t h a t t h e o p t i m a l v a l u e f* o f Suppose t h a t f * ( O ) i s f i n i t e .

i s i n T.

R o c k a f e l l a r shows

F ( E ) i s convex o n

T.

H e t h e n d e f i n e s P(0) t o be

"stably set" i f the d i r e c t i o n a l derivative

and t h e c o n s t r a i n t s o f by a t l e a s t o n e x ) .

P(0) a r e

"consistent"

(i.e., s a t i s f i e d

(Geoffrion ( 1 9 7 1 ) n o t e s t h a t t h i s condi-

t i o n i s equivalent to G a l e ' s condition, given i n t h e l a s t Rockafellar proves t h a t i f t h e r e e x i s t s a t least

paragraph.) one x

E

X where f i s f i n i t e a t x and g i s f i n i t e a n d c o n t i n u -

ous a t Ax, t h e n P(0) i s s t a b l y s e t a n d f * i s c o n t i n u o u s n e a r c

=

0.

H e p r o c e e d s t o p r o v e many s t r o n g d u a l i t y r e l a t i o n s h i p s

between t h e g i v e n convex program P ( 0 ) and a c o n c a v e program

s i m i l a r i n form t o B ( 0 ) a n d c o n s t r u c t e d by means o f t h e conjugate f u n c t i o n theory of Fenchel (1949). C l a r k e ( 1 9 7 6 ) h a s shown t h a t i f X i s a Banach s p a c e and f

i s l o c a l l y L i p s c h i t z , t h e n programs o f t h e form P 1 ( € ) , w i t h x

E

M C X a r e "normal" i n t h e s e n s e t h a t g e n e r a l i z e d Karush-

Kuhn-Tucker c o n d i t i o n s c a n be shown t o h o l d , e v e n i n t h e a b s e n c e o f d i f € e r e n t i a b i l i t y and c o n v e x i t y a s s u m p t i o n s .

t e r m s t h e program min

X

f (x)

s . t . g ( x ) 2. x

E

M,

E,

Clarke

35


where M is a closed subset of En, normal if Karush-Kuhn-Tucker type multipliers exist for any solution x.

*

is said to be "calm" if f lim inf[f

*

(E')

- f

*

(E)

The problem P i ( € )

is finite and

E)l/llE'

-

Ell

>

(2.3.6)

-m

E'+E

where as usual

x

E

M, g(x) 1. € 1 ,

R(E) # B R(E) = 8 .

(We note that the limit quotient in Eq. (2.3.6) stability also used by Rockafellar (1967).)

is a form of

Using these no-

tions, Clarke showed that if the problem is calm it is also

*

E

=

0,

then the problem is calm and normal for almost all

E

in a

normal, and if f

is finite in a neighborhood of

Conditions sufficient for the calmness

neighborhood of 0.

(and hence the normality) of the problem are given in the following theorem. Th eor em 2 . 3 . 9 .

(i) -gi, i

If =

1,

..., m, are convex,

(ii) M is convex and bounded, (iii) f is bounded and Lipschitz continuous on M, and (iv) there exists a point x

E

M such that g(x) >

E,

then

P ' ( E ) is calm.

1

Extensions, refinements, and generalizations of many of the results in this section may be found in Hiriart-Urruty (1979) and Gollan (1981a,b), and a further sharpening and more general and unified treatment of these and related results in Rockafellar (1982a,b).

36 2.4.


IMPLICIT FUNCTION

THEOREM RESULTS

There a r e many forms of i m p l i c i t f u n c t i o n theorems t h a t have found e x t e n s i v e a p p l i c a t i o n i n f u n c t i o n a l a n a l y s i s . These theorems t r e a t t h e g e n e r a l problem o f s o l v i n g a n e q u a t i o n o f t h e form $(x, Y) = 0

(2.4.1)

f o r x i n t e r m s o f y.

w e l l known.

The c l a s s i c a l r e s u l t s i n t h i s a r e a a r e

For c o m p l e t e n e s s w e p r e s e n t two forms o f t h e A more complete d i s c u s s i o n o f

i m p l i c i t f u n c t i o n theorem.

t h e s e and o t h e r theorems i s c o n t a i n e d i n Bochner and M a r t i n ( 1 9 4 8 ) , H e s t e n e s ( 1 9 6 6 , 1 9 7 5 ) , and i n most advanced t e x t s i n functional analysis.

These r e s u l t s e x t e n d t o more g e n e r a l

spaces. Theorem 2 . 4 . 1 .

Function. 1

[ I m p l i c i t F u n c t i o n Theorem f o r Ck

Suppose $: En+m

+

En is a k t i m e s c o n t i n u o u s l y

d i f f e r e n t i a b l e mapping whose domain i s T. $

(F, y )

Suppose (E, 7)E T,

= 0 , and t h e J a c o b i a n w i t h r e s p e c t t o x , Vx$(T,

TI, is

Then t h e r e e x i s t s a neighborhood of y , N ( 7 ) C Em, k and a u n i q u e f u n c t i o n y E C [ N ( y ) ] , y: N(y) + En, w i t h y(y) = nonsingular.

x

and $ [ y ( y ) , yI = 0 for a l l y e

N(y).

The f u n c t i o n y i s s a i d t o be d e f i n e d i m p l i c i t l y by t h e e q u a t i o n $ [Y (y), y1 = 0.

i s used t o d e n o t e t h e j Ek w i t h r e s p e c t t o i t s j - t h argu-

I n t h e n e x t theorem, t h e n o t a t i o n $ j-th component of 9: Ek+'

..., $,)/a(x,, ..., x k ) d e n o t e s the J a c o b i a n ..., $ k ) w i t h r e s p e c t t o (xl, ..., x k ) .

ment, and

of

($I~,

-+

a($l,

Theorem 2 . 4 . 2 .

Function. 1

[ I m p l i c i t F u n c t i o n Theorem f o r A n a l y t i c

I f $ j (xl,

..., xk;

yl,

..., y,)

neighborhood o f t h e o r i g i n w i t h $ . ( O ,

I

is a n a l y t i c i n a

0) = 0 f o r j = 1,

..., k ,

37

2.4 Implicit Function Theorem Results

..., 4Q/a(x1, ..., xk)l -1 exists at x = y = 0, then the system of equations @.(xl, ..., Xki y1r Y,) = 7 for j = 1, ... k, has a unique solution x = xj(y,, ..., y , ) , j

and [a(@,,

. - . r

which vanishes for y

=

O r

0 and which is analytic in a neighbor-

hood of the or gin. Results of this type have particular applicability to sensitivity analysis in nonlinear optimization and have only recently been exploited in NLP.

Hildebrandt and Graves (1927)

have provided results on the existence and differentiability of solutions of Eq. 2.4.1.

Cesari (1966) has established condi-

tions under which the equation @(y, y) = 0 has at least one solution, and discusses the continuous dependence of y on parameters of the equation.

Rheinboldt (1969) has given

global existence theorems for the solution of (2.4.1), which leads to a "continuation property."

This continuation property

has been applied to the solution of parametric optimization programs.

Recent applications of implicit function theory re-

sults due to Hestenes (1975) have been made in differential stability by Auslender (1979) and Gauvin and Dubeau (19791, as noted in the previous section. Fiacco and McCormick (1968) and Duffin, Peterson, and Zener (1967) (for geometric programming (GP)) provided some of the first applications of an implicit function theorem to obtaining sensitivity information about the solution of a mathematical program.

Since then, additional results in this

area have been obtained by Bigelow and Shapiro (1974), Armacost and Fiacco (1974, 1975, 1976, 1977, 1978), Armacost (1976a,b), Fiacco (1976), Jittorntrum (1978, 1981) , Robinson (1974), and Spingarn (1977).

38


Robinson (1979) provided an implicit function theorem for a "generalized equation" where we seek a "solution" of 0

F(Y, € 1 + T(y)

E

where F: Y

x

P

-+

(2.4.2)

ES is a vector function, Y is an open set in

ES, P is a topological space, T is a closed point-to-set mapping from E Given

E,

S

into itself, and

E

is a parameter vector.

the problem is to find y such that the point-to-set

map on the rhs of ( 2 . 4 . 2 ) contains 0, or equivalently, such that the vector -F(y,

E)

T(y).

E

It is assumed that F(y,

E)

is (jointly) continuous and the partial Frgchet derivative

V F is continuous on Y x P. Y Given 5 E P , the linearization of F at T around yo

-

E

Y is

-

given by LF (y) = F ( y o , E ) + VyF(yO, E ) (y - yo). It is YO assumed that Yo is a nonernpty bounded convex set and that Y = Yo + y B C Y for some y > 0, where B is the (open) unit Y ball in ES. Conditions are given such that: (i) the set of

solutions

h

S(E)

of ( 2 . 4 . 2 ) in Yo

+ 6B is upper semicontinuous

in some neighborhood N(F) of T for some 6 = C$

if

E

#

;(El

(0, y l

(with

N(T)); (ii) g ( F ) = Yo; and (iii) for each

some neighborhood Na(F) and each

0 #

E

E

E

c1

h

S(E)

> 0 and

N,(T),

c 2CS)

+ ( A + a) max (IIF(y, Y

E)

-

F(y, T )11 Iy

E

YO}B

(2.4.3)

for some A > 0. Nonlinear complementarity and equilibrium problems and the Karush-Kuhn-Tucker conditions for P 3 ( ~ )can be expressed as realizations of ( 2 . 4 . 2 ) .

The latter constitute our interest

here and are expressed in the form ( 2 . 4 . 2 ) when we take T T s = n + m + p, y = (x, u, w) , F = (CxL, g , -hTIT, and

39


T =

a$,,

w h e r e JIC d e n o t e s t h e i n d i c a t o r f u n c t i o n o f C d e f i n e d

by

and

a d e n o t e s t h e " s u b d i f f e r e n t i a l o p e r a t o r " (i.e., a$c(y) is

t h e s e t o f a l l s u b g r a d i e n t s o f $, a t y ) [ R o c k a f e l l a r (197O)J and C = E n x EY x Ep, where = { u E ?lu 1. 0 ) . The a d d i t i o n a l c o n d i t i o n s r e q u i r e d f o r t h e s e r e s u l t s a r e t h e e x i s t e n c e o f rl > 0 , a l o n g w i t h t h e f o l l o w i n g a s s u m p t i o n s f o r each yo (i) Y (ii) Y (iii) Y

E

Yo:

n

G-'(o)

yo

= yo;

n ~ - l ( y )c y Y n yo

G-'(o)

YO

+

~ l l y l lf o~r e a c h y e

n G - l ( y ) # 0 a n d convex f o r e a c h y yo

E

n ~ ;

nB,

where G = LF + T, a n d t h e i n v e r s e r-l o f a p o i n t - t o - s e t YO yo -1 map r: X + 2 a t Z E 2 i s d e f i n e d t o b e r (?) = { x E X ~ Z E r(x)1. Robinson a l s o shows t h a t a s s u m p t i o n (iii) may be r e p l a c e d by ( i i i ) 'VyF(yO,

E)

i s p o s i t i v e s e m i d e f i n i t e and T i s a

maximal monotone o p e r a t o r .

An o p e r a t o r T i s c a l l e d "monotone"

i f f o r e a c h (yl, v l ) , ( y 2 , v 2 ) i n t h e g r a p h o f T, i t f o l l o w s T t h a t (y, - y,) (v, v,) 2 0. I t i s c a l l e d "maximal monotone"

-

i f i t i s monotone a n d i t s g r a p h i s n o t p r o p e r l y c o n t a i n e d i n t h a t o f any o t h e r monotone o p e r a t o r . Many o t h e r r e s u l t s f o r p a r a m e t r i c n o n l i n e a r programs a r e d e v e l o p e d by Robinson, making u s e o f t h e g e n e r a l i z e d e q u a t i o n framework b r i e f l y i n d i c a t e d h e r e .

40


An i l l u s t r a t i o n o f a n a p p l i c a t i o n o f t h e above r e s u l t s i s p r o v i d e d by Robinson t o a n a l y z e t h e s o l u t i o n s e t o f t h e l i n e a r generalized equation 0

E

+

Ay

+

a

T(y)

(2.4.4)

where A i s a p o s i t i v e s e m i d e f i n i t e s T =

aqc a n d

x

s matrix, a

E

ES,

C i s a nonempty p o l y h e d r a l convex s e t i n E

i s shown t h a t t h e s o l u t i o n s e t o f

s

.

It

i s nonempty and

(2.4.4)

bounded i f a n d o n l y i f t h e r e e x i s t s 6 o > 0 s u c h t h a t f o r e a c h

s

x

[la' =

8.

s m a t r i x A ' and each a

-

a l l } < 60,

E

ES w i t h 6 '

t h e set ;(A',

:max(llA'

a ' ) : {y10

E

+

A'y

-

All,

a'

+

Under t h e a s s u m p t i o n t h a t p b e a bound o n ; ( A ,

a$JC(y)}

a) # 8,

i t is shown t h a t 3X > 0 s u c h t h a t

+ SA(A', +

a')

h6'(1

-

n

M

A6')

c S(A, a) -1

(1

+

(2.4.5)

p)B

where M i s any open bounded s e t c o n t a i n i n g

a ) and 6 ' < 61

(A,

for some 6

1 > 0 . L i n e a r a n d q u a d r a t i c programming and l i n e a r c o m p l e m e n t a r i t y p r o b l e m s can b e p o s e d i n t h e f o r m ( 2 . 4 . 4 ) . A p p l i e d t o a q u a d r a t i c program, f o r example,

(2.4.5)

provides an e x t e n s i o n of a r e s u l t due t o Daniel ( 1 9 7 3 ) .

It

concerns t h e s o l u t i o n s t a b i l i t y f o r p o s i t i v e d e f i n i t e q u a d r a t i c p r o g r a m s , which i s g i v e n n e x t f o r c o m p a r i s o n w i t h ( 2 . 4 . 5 ) . D a n i e l c o n s i d e r s a program i n E T

m i n x ( l / 2 ) x Kx

-

n

o f t h e form

k Tx

s.t. cx 5 c Dx = d , where K i s p o s i t i v e d e f i n i t e a n d s y m m e t r i c w i t h

smallest e i g e n v a l u e . c a s e of

(2.4.5).

7

> 0 its

Daniel obtained t h e f o llo win g s p e c i a l

41


where xo solves the program above and x;J solves that program when K' and k' replace K and k, respectively. Using additional assumptions, a number of stronger results have been obtained which characterize more completely the relationship between a solution set and the optimal value function of a mathematical program to general perturbations appearing simultaneously in the objective function and anywhere in the constraints.

These problems generally have the

form min

X

f(x,

E)

s.t. g(x, € 1 2 0 h(x, where f: En

x

E)

Ek

= -+

0, El, g: En

x

Ek

+

Em, and h: En

x

Ek

-+

EpI

unless specifically stated otherwise. McCormick (Fiacco and McCormick, 1968) obtained conditions that guarantee the existence of a differentiable function of

E

that locally solves a particular form (problem P;(E), Section 3.4) of P3(€).

Robinson (1974) and Fiacco (1976) extended

this result to programs in which the perturbations appear generally, as in P 3 ( ~ ) . These results are established by applying an appropriate implicit function theorem to the firstorder necessary conditions that must hold at a local solution of P 3 ( € ) to prove the existence of a continuous or differentiable Karush-Kuhn-Tucker triple.

The author's extension,

given in the next theorem, establishes the existence of a once continuously differentiable (local) solution of P 3 ( ~ ) ,along

42


with associated unique differentiable optimal Lagrange multipliers.

This is the basic theorem for the sensitivity results

developed in this book, and is proved (with slightly weaker differentiability assumptions) in Chapter 3 (Theorem 3.2.2). T h e o r e m 2.4.4.

If

(1) f, g , h are C2 in (x, E) in a neighborhood of (x*, O),

*

[x

(2) the second order sufficiency conditions hold at

* * , u ,w I

(see Lemma 3.2.11,

*

(3) the linear independence assumption holds at x

complementary slackness with respect to u

*

holds at x

*

, and

,

then

*

(i) x

is a local isolated (i.e., locally unique)

minimizing point of P3(0) with unique Lagrange multipliers

*

*

U i W r

(ii) for

E

near 0, there exists a unique C1 function

[x(E), U(E), W(E) I T satisfying the second-order suffi* * * T ciency conditions for problem P 3 ( ~ ,) with y(D) = [x , u , w I , Y(E)

=

hence

X(E)

is an isolated local minimizing point of P 3 ( ~ )with

associated unique Lagrange multipliers u (E) and w ( E) (iii) for

E

,

and

near 0, the gradients of the binding con-

straints are linearly independent, and strict complementary slackness holds for U(E) and g(x, E). Fiacco (1976) also provided an explicit formula for the partial derivatives, as well as approximations based on classical penalty functions (Theorem 2 . 4 . 9 ) .

These results

provide the basis for the methodology presented in Part I1 of this book.

43


Jittorntrum (1978, 1981), essentially complecing the development of results pursued by Bigelow and Shapiro (1974), obtained the following results, which do not require the strict complementarity condition (4) of Theorem 214.4.

How-

ever, the second-order part of the second-order sufficiency conditions (2) of the theorem must be strengthened to: v2L(x

ZT

*

,

*

u

* ,w ,

0)z > 0 for all z # 0 such that Vgi(x*, 0)z

= 0, for all i such that gi(x*) = 0 and u; > 0 and such that

*

Vh.(x 7

,

0)z = 0 for all j .

Following Robinson (1980a), we call

this condition, taken together with the first-order KarushKuhn-Tucker conditions (see Lemma 3.2.11,

“the strong second-

order sufficient conditions for problem P 3 (O).“ (i)

-

Conclusions

(v) of the next theorem were obtained by Jittorntrum in

the cited 1978 doctoral dissertation, Conclusion (vi) in the cited 1981 paper. Theoraern 2 . 4 . 5 .

*

hood of (x

,

If f, g, h are C L in (x,

*

in a neighbor-

0), if the strong second-order sufficient condi-

*

*

tions for problem P (0) hold at (x , u , w 3 * independence condition holds at x , then (i) x

E)

*

)

,

and if the linear

is a local isolated minimizing point of P3(0)

and the associated Lagrange multipliers u (ii) for

E

*

and w

*

are unique,

in a neighborhood of 0, there exists a unique

c o n t i n u o u s vector function y(E) = [x(E), u(E) , w(E) l T satis-

fying the strong second-order sufficiency condition for a local * * * T r minimum of the problem P 3 ( ~ )such that- y ( 0 ) = (x , u , w and hence, X(E) is a locally unique minimizer of P 3 ( € ) with associated unique Lagrange multipliers U(E) and w(E), (iii) linear independence of the binding constraint gradients holds at X(E) for

E

near 0,

44 (iv) every

E

t h e r e e x i s t 0 < a,

( 1 ~ 1 1
0, X(E, r) is a locally unique unconstrained local minimizing point of W(x, for each i = 1,

..., m,

E,

and V:W(x,

r) with gi[x(E, r), € 1 > 0 E,

r) is positive definite

at x = X(E, r). The existence of higher-order derivatives of Y(E) depends on the degree of (continuous) differentiability of the problem functions.

This follows directly from an application of the

classical implicit function theorem to the first-order necessary conditions for a solution of P 3 ( € ) .

An analogous result

holds for Y(E, r), the solution of (2.4.6).

In fact, it

51


f o l l o w s e a s i l y t h a t under t h e a p p r o p r i a t e c o n d i t i o n s n o t o n l y do h i g h e r - o r d e r

d e r i v a t i v e s of

Y(E,

r ) e x i s t , but these deriva-

t i v e s converge t o t h e corresponding d e r i v a t i v e s of Y ( E ) .

This

r e s u l t i s s t a t e d n e x t and l a t e r subsumed i n C o r o l l a r y 6.4.1. L e t f , g , and h have c o n t i n u o u s d e r i v a -

Theorem 2 . 4 . 1 0 .

t i v e s o f a l l o r d e r s up t o p

+

1.

Assume t h a t the conditions

Then, i n a neighborhood o f

of Theorem 2 4 . 4 a r e s a t i s f i e d .

r ) = ( 0 , 0 ) , t h e r e e x i s t s a unique f u n c t i o n

(E,

r)

Y(E,

[x

=

E,

r) ,

U(E,

r),

W(E,

Y(E,

r)

E

Cp,

r ) l Ts a t i s f y i n g ( 2 . 4 . 6 1 ,

with

r)

Y(Er

+

Y(E)r

and t h e j - t h p a r t i a l d e r i v a t i v e o f y w i t h r e s p e c t t o

at

E

(E,

r ) c o n v e r g e s t o t h e j - t h p a r t i a l d e r i v a t i v e of y a t

r

0 for

+

(E,

r ) n e a r ( 0 , 0 ) , where j = 1,

...

r

E

as

p.

w a s s u b s e q u e n t l y g i v e n by Buys

An a n a l o g o f Theorem 2 . 4 . 9

and Gonin (1977) and Armacost and F i a c c o ( 1 9 7 7 ) , u s i n g an augmented L a g r a n g i a n p r e v i o u s l y u t i l i z e d by Buys ( 1 9 7 2 ) . gives “exact” s e n s i t i v i t y r e s u l t s , i.e., w i t h t h o s e of Theorem 2 . 4 . 4 . a t e d w i t h problem P

3

(E)

r e s u l t s coinciding

The augmented L a g r a n g i a n a s s o c i -

i s defined a s

h

L(xi

Ur

Wr

= f(x,

E r

E)

C)

-

2

ieJ(E

ieK(E)

where J ( E )= { i l u cgi(x’

E)

< 01.

i

-

cgi(xr

It

E)

L 0 ) and K ( E ) = { i l ui

-


52

Under t h e a s s u m p t i o n s of Theorem 2 . 4 . 4 ,

Theorem 2 . 4 . 1 1 . there exists c

*

such t h a t f o r

n e a r 0 and c > c

*

,

there

f u n c t i o n y ( E , c) = [ x ( E , c ) , u ( E , c ) ,

c1

e x i s t s a unique

E

c ) I ' s~ a t i s f y i n g

W(E,

( i ) v ~ ( x ,u, w,

( i i ) u1 .g1 . (x, (iii) h . ( x , 3

with

= 0,

w e have t h a t

..., m, ..., p ,

j = 1,

and

F u r t h e r m o r e , f o r any

c) = Y ( E ) .

Y(E,

0,

=

i = 1,

= 0,

E)

E)

c)

E,

n e a r 0 and c > c*

E

c) is a l o c a l l y unique unconstrained l o c a l

X(E,

h

minimizing p o i n t o f L [ x ,

U(E,

c),

W(E,

c),

*

p o s i t i v e d e f i n i t e f o r [ x , u, w] n e a r [x

2^

cl and V L i s

E,

* * , u ,w I.

This

A

r e s u l t , r e a d i l y f o l l o w i n g from t h e d e f i n i t i o n o f L and Theorem 2.4.4,

is proved and f o r m u l a s f o r V E y ( € , c ) are developed i n

C h a p t e r 7. Armacost and F i a c c o (1975) also o b t a i n e d f o r m u l a s f o r t h e f i r s t - and second-order d e r i v a t i v e s o f t h e o p t i m a l v a l u e f u n c t i o n of t h e problem P 3 ( ~ ) . These f o l l o w from t h e n e x t r e s u l t , which i s a n e a s y consequence o f Theorem 2.4.4. Suppose t h e a s s u m p t i o n s of Theorem 2.4.4

Theor em 2 . 4 . 1 2 .

h o l d f o r P 3 ( ~ ) . Then, i n a neighborhood of

E

= 0, t h e optimal

value function f*(E) is t w i c e continuously d i f f e r e n t i a b l e a s a f u n c t i o n of

E,

and

(i) f*(E) = L [ x ( E ) ,

U(E)

,~

( € 1 ,€ 1

( i i ) VEf*(E) = V E L [ x ( ~ )U, ( E )

,~

( € 1 , €1, and

(iii) v E2 f * ( E ) = V ~ ~ V ~ L [ X ( u E ()E, ) , C o n s i d e r t h e problem minx f ( x )

s.t. gi(x) L

E

h.(x) =

E

3

~ ~

..., m, = 1, ..., p.

i = 1,

I +

j

~

I

w(E)

,

ElT].

53

2.4 Implicit Function Theorem R e h u h

The L a g r a n g i a n f o r P 2 ( ~ )i s g i v e n by m L ( x , u, w ,

E)

=

f(x)

2

+

i=l

-

wj [hj(x)

j=1

E .

7 +m

I.

The f o l l o w i n g r e s u l t f o l l o w s i m m e d i a t e l y from Theorem 2 . 4 . 1 2 . I t i s s t a t e d s e p a r a t e l y because of i t s importance.

Theorem 2 . 4 . 1 3 .

L e t f , g , and h b e t w i c e continuously

d i f f e r e n t i a b l e i n x i n a neighborhood of x a s s u m p t i o n s o f Theorem 2 . 4 . 4

*

.

Suppose t h e Then, i n a

hold f o r P 2 ( € ) .

neighborhood of E = 0,

(i)

VEf*(E) = [u(EIT,

Theorem 2 . 4 . 1 2

and 2.4.13

-w(E)

T

I

T

a r e p r o v e d i n C h a p t e r 3 (Theorem

3.4.1 and C o r o l l a r y 3.4.4).

Formulas f o r VEf

*

i n t e r m s of t h e

d e r i v a t i v e s o f t h e o r i g i n a l p r o b l e m f u n c t i o n s are d e v e l o p e d i n C h a p t e r 3 , S e c t i o n 3.4. A p p l i c a t i o n s o f t h e s e n s i t i v i t y r e s u l t s g i v e n i n Theorems 2.4.4,

2.4.9,

and 2.4.12

have b e e n implemented v i a a p e n a l t y

f u n c t i o n a l g o r i t h m and a r e r e p o r t e d subsequently.

Here, w e

b r i e f l y s u m m a r i z e some o f t h e i n i t i a l c o m p u t a t i o n a l e x p e r i e n c e . Armacost a n d F i a c c o (1974) i l l u s t r a t e d c o m p u t a t i o n a l a s p e c t s o f some o f t h e r e s u l t s o u t l i n e d i n t h i s s e c t i o n . Using t h e SUMT ( S e q u e n t i a l U n c o n s t r a i n e d M i n i m i z a t i o n T e c h n i q u e ) V e r s i o n 4 computer c o d e d e v e l o p e d by Mylander, H o l m e s , a n d McCormick (1971) a n d a s u b r o u t i n e f o r s e n s i t i v i t y

analysis coded by Armacost and Mylander (1973) that implemented a procedure for calculating the solution parameter derivatives based on the results given in Theorems 2.4.4, 2.4.9, and 2.4.12, they obtained results that computationally corroborated the theory, e.g., the convergence of the first partial derivatives of the optimal solution and the optimal value function of several problems.

The resulting code was further developed and

refined by Armacost (1976a,b).

Subsequently, Armacost and

Fiacco (1978) used this computer program, which came to be known as SENSUMT, to analyze the behavior of the solution of an inventory problem relative to changes in several problem parameters.

This application is discussed in Chapter 8.

Fur-

ther development of SENSUMT and other applications will be discussed as the theory is developed. 2.5.

OPTIMAL VALUE AND SOLUTION BOUNDS The foregoing results are concerned mainly with the con-

tinuity or differentiability properties of the optimal value

*

function f

(E)

or a Karush-Kuhn-Tucker triple [x(E), u(E) ,

~ ( € 1 1 . Among the results we have given, only a few, e.g., Robinson's generalized implicit function theorem result (2.4.3) for the general system (2.4.21, the particular realization of this obtained earlier for the positive definite quadratic program by Daniel in Theorem 2.4.3, Robinson's bounds on a Karush-Kuhn-Tucker triple given in Theorem 2.4.8, and Stern and Topkis' Lipschitz condition on the optimal value function of P1 ( E ) given in Theorem 2.2.12 provide parametric solution bounds. All of these results involve valuable existence proofs,

providing incisive characterizations of solution properties

2 . 5 Optiinal Valuc

dnd

55

Solution Hound\

r e l a t i v e t o parametric perturbations.

N o n e t h e l e s s , much

remains t o b e done t o d e v e l o p computable t e c h n i q u e s f o r c a l c u l a t i n g u s e f u l n u m e r i c a l e s t i m a t e s o f bounds f o r g e n e r a l c l a s s e s of problems. The q u e s t i o n a r i s e s w h e t h e r p r a c t i c a l l y u s e f u l computable p a r a m e t r i c s o l u t i o n bounds c a n b e c a l c u l a t e d f o r g e n e r a l c l a s s e s of NLP p r o b l e m s .

P r a c t i c a l i t y demands t h a t t h e re-

q u i r e d e f f o r t b e f e a s i b l e and t h e p e r t u r b a t i o n s b e f i n i t e , while usefulness suggests a d d i t i o n a l l y t h a t a s u i t a b l e degree of a c c u r a c y b e a c h i e v e d .

W e find that generally applicable

bounds r e s u l t s m e e t i n g t h e p r e s c r i b e d p r o p e r t i e s a r e r a r e indeed.

I n f a c t , t h e r e a p p e a r t o b e two f u n d a m e n t a l d i r e c -

t i o n s , o n e d u e t o K a n t o r o v i c h ( 1 9 4 8 ) and o n e d u e t o Moore (1966, 1977).

Both i n v o l v e N e w t o n ' s method, t h e l a t t e r

i n c o r p o r a t i n g r e s u l t s from i n t e r v a l a r i t h m e t i c .

Most g e n e r a l

a p p r o a c h e s f o r o b t a i n i n g p a r a m e t r i c s o l u t i o n bounds and e r r o r bounds s e e m t o b e a s s o c i a t e d w i t h one o f t h e s e b a s i c r e s u l t s . Examples o f i n t r i g u i n g a n d i m p o r t a n t r e c e n t e r r o r bounds r e s u l t s o f t h e K a n t o r o v i c h t y p e may b e found i n t h e work o f M i e l ( 1 9 8 0 ) , a n d o f a m i x t u r e o f t h e s e t y p e s i n t h e work o f Mancini and McCormick ( 1 9 7 6 , 1 9 7 9 ) , and McCormick ( 1 9 8 0 ) . T h e o r e t i c a l a n d n u m e r i c a l c o m p a r i s o n s o f t h e two b a s i c t e c h n i q u e s f o r c o m p u t i n g e r r o r bounds h a v e b e e n r e p o r t e d by Rall (1979). A n o t h e r s t r a t e g y f o r o b t a i n i n g s o l u t i o n bounds i s t h e e x p l o i t a t i o n of s t r u c t u r e of p a r t i c u l a r c l a s s e s of f u n c t i o n s . F o r e x a m p l e , i f t h e g i v e n p r o b l e m i s a l i n e a r programming problem, t h e n under u s u a l nondegeneracy assumptions one c a n e s s e n t i a l l y c o n s t r u c t an e x a c t parametric s o l u t i o n f o r f i n i t e and r a t h e r g e n e r a l c l a s s e s o f p a r a m e t r i c p e r t u r b a t i o n s .

Of

56


c o u r s e , t h i s may e n t a i l an i n o r d i n a t e amount of e f f o r t a n d , i f t h i s i s p r o h i b i t i v e , q u e s t i o n s a r i s e a s t o whether s u i t a b l y a c c u r a t e p a r a m e t r i c bounds on a s o l u t i o n might be c a l c u l a t e d with t o l e r a b l e e f f o r t .

I n n o n l i n e a r programming an i m p o r t a n t

c o m p u t a t i o n a l l y e x p l o i t a b l e p r o p e r t y is s e p a r a b i l i t y o f t h e problem f u n c t i o n s , i . e . ,

t h e r e p r e s e n t a t i o n of a f u n c t i o n a s

t h e sum of f u n c t i o n s o f one v a r i a b l e .

S i n c e e a c h one-

d i m e n s i o n a l "component" o f a s e p a r a b l e f u n c t i o n can g e n e r a l l y b e s e p a r a t e l y e s t i m a t e d o r bounded by s i m p l e r f u n c t i o n s , t h i s l e a d s t o t h e p o s s i b i l i t y of e s t i m a t i n g o r bounding a m u l t i d i m e n s i o n a l s e p a r a b l e problem by a s i m p l e r problem.

Assuming

t h a t t h e one-dimensional e s t i m a t i o n e r r o r s o r e r r o r bounds a r e t r a c t a b l e , t h e n t h e s o l u t i o n and e r r o r bound i n f o r m a t i o n a s s o c i a t e d w i t h t h e a p p r o x i m a t i o n problem c a n p o s s i b l y b e r e l a t e d t o t h e s o l u t i o n - and e r r o r - b o u n d i n f o r m a t i o n o f t h e o r i g i n a l problem. The most s t r i k i n g and w e l l known a p p l i c a t i o n o f t h e e x p l o i t a t i o n of s e p a r a b i l i t y i s undoubtedly t h e e x t e n s i v e l y developed and w i d e l y u s e d branch and bound methods f o r nonconvex s e p a r a b l e programming p i o n e e r e d by F a l k and Soland ( 1 9 6 9 ) , based on g l o b a l l y s o l v i n g t h e g i v e n problem by g e n e r -

a t i n g a sequence of s e p a r a b l e convex u n d e r e s t i m a t i n g programs. C l o s e r t o t h e s p i r i t of s o l u t i o n e r r o r bounds, however, G e o f f r i o n (1977) n o t e d t h a t t h e e r r o r bounds i n v o l v e d i n f i t t i n g f u n c t i o n s ( o f one v a r i a b l e ) t o t h e r e s p e c t i v e compon e n t s of a g i v e n s e p a r a b l e problem c o u l d b e v e r y p r e c i s e l y r e l a t e d t o t h e d i s c r e p a n c y between t h e s o l u t i o n v a l u e o f t h e g i v e n problem and t h a t o f t h e a p p r o x i m a t i o n problem.

H e gave

s e v e r a l t h e o r e t i c a l r e s u l t s f o r bounding t h e d i f f e r e n c e o f t h e o p t i m a l v a l u e and c o n s t r u c t i v e c u r v e - f i t t i n g t e c h n i q u e s f o r

2.5 Optimal Value and Solution Bounds

57

reducing t h e r e s u l t i n g deviation.

More r e c e n t l y , Meyer ( 1 9 7 9 ,

1980) d e v e l o p e d t e c h n i q u e s f o r c a l c u l a t i n g error bounds on t h e o p t i m a l v a l u e of l a r g e convex s e p a r a b l e programs and h a s been i n v e s t i g a t i n g t h e e x p l o i t a t i o n of s e p a r a b i l i t y t o c a l c u l a t e

error bounds on s o l u t i o n p o i n t s .

Thakur ( 1 9 7 8 , 1 9 8 0 , 1 9 8 1 )

computes o p t i m a l v a l u e f u n c t i o n and s o l u t i o n p o i n t e r r o r - b o u n d i n f o r m a t i o n f o r convex ( h i g h l y n o n l i n e a r ) s e p a r a b l e p r o g r a m s , b a s e d on a n a n a l y s i s o f a c o r r e s p o n d i n g p i e c e w i s e l i n e a r a p p r o x i m a t i o n problem.

An i n t r i g u i n g u s e of t h i s i n f o r m a t i o n

i s t o a c c e l e r a t e c o n v e r g e n c e by r e d u c i n g t h e r e g i o n known t o c o n t a i n a s o l u t i o n i n a scheme b a s e d on s o l v i n g a s e q u e n c e o f p r o g r e s s i v e l y t i g h t e r l i n e a r a p p r o x i m a t i o n problems.

Again, i n

t h e c o n t e x t of s e p a r a b l e , b u t now s e p a r a b l e p a r a m e t r i c p r o g r a m s , Benson ( 1 9 8 0 ) h a s e x t e n d e d t h e u s u a l b r a n c h a n d bound a p p r o a c h t o a l l o w f o r t h e s i m u l t a n e o u s c a l c u l a t i o n of a s e t of s o l u t i o n s of a g i v e n s e p a r a b l e program c o r r e s p o n d i n g t o a f i n i t e number of r e s o u r c e l e v e l s ( r i g h t - h a n d s i d e s ) . D i n k e l and K o c h e n b e r g e r ( 1 9 7 7 , 1 9 7 8 ) and D i n k e l , K o c h e n b e r g e r , a n d Wong ( 1 9 7 8 , 1 9 8 2 ) h a v e c o n d u c t e d numerous experiments i n c a l c u l a t i n g t h e parameter d e r i v a t i v e s of a Karush-Kuhn-Tucker

triple [x(E)

,

u ( E ) , W ( E ) ] and e x t r a p o l a t i n g

f o r new s o l u t i o n s , i n a v a r i e t y o f a p p l i e d G P p r o b l e m s ( e s s e n t i a l l y u n d e r t h e a s s u m p t i o n s of Theorem 2 . 4 . 4 ) .

A distinctive

f e a t u r e o f t h e i r work i s t h e u s e o f a t e c h n i q u e g e n e r a l l y a t t r i b u t e d t o Davidenko ( 1 9 5 3 ) :

extrapolation over a small

parameter i n t e r v a l u s i n g f i r s t p a r a m e t e r - d e r i v a t i v e information f o l l o w e d by a r e e v a l u a t i o n o f f u n c t i o n s i n v o l v e d a n d a r e c a l c u l a t i o n o f d e r i v a t i v e s a t t h e new p o i n t , t i o n , etc.

f o l l o w e d by e x t r a p o l a -

They h a v e o b t a i n e d much g r e a t e r a c c u r a c y u s i n g t h i s

i n c r e m e n t a l a p p r o a c h t h a n i s o b t a i n e d by u s i n g one l i n e a r

58


extrapolation, the accuracy apparently being proportional to the number of increments

- though

obviously, there is a trade-

off with computational effort that dictates a compromise strategy.

They have also developed heuristics for estimating

the range over which extrapolation is valid. Our computational approach to the calculation of optimal value and Karush-Kuhn-Tucker triple parameter derivatives is discussed in detail in Chapters 3

-

8,

so we confine ourselves

here to a few remarks concerning a central topic of this section, the calculation of optimal solution bounds.

Our point

of view has been to start with the highly structured problem and to focus our attention explicitly on the calculation of parametric bounds on the optimal value function corresponding to finite parameter changes.

We have found the class of prob-

lems with convex (or concave) optimal value function to have many computationally exploitable properties, while being general enough to be of wide practical interest, and have thus been naturally led to the jointly convex parametric NLP P 3 ( c ) as a logical starting point on which to build a computable bounding methodology.

The convexity of the problem in x leads

to immediate connections with duality theory and the determination of dual parametric bounds.

The convexity of the optimal

value function allows for the immediate calculation of a global lower parametric linear bound whenever a solution and a subgradient is calculated at any specified parameter value, and an upper linear bound via linear interpolation between the optimal values whenever solutions for two parameter values have been calculated.

The joint convexity of the feasible

region also easily yields a feasible parametric vector, whenever solutions are calculated for two parameter values, by

59

2.5 Optimal Value and Solution Bounds

l i n e a r i n t e r p o l a t i o n between t h e g i v e n s o l u t i o n p o i n t s . t h e bounds on f

*

(E)

Also,

open t h e d o o r t o computable t e c h n i q u e s for

bounding a Karush-Kuhn-Tucker

triple [x(E)

,

U(E)

,

W(E)

1.

Finally, the p o s s i b i l i t y of estimating corresponding r e s u l t s f o r nonconvex p a r a m e t r i c programs i s r e a l i z e d f o r t h o s e programs t h a t c a n be e s t i m a t e d by j o i n t l y convex programs.

In

p a r t i c u l a r , r e s u l t s c a n be o b t a i n e d f o r j o i n t l y s e p a r a b l e nonconvex programs s i n c e , a s n o t e d e a r l i e r , an i m p o r t a n t approach f o r s o l v i n g s t a n d a r d ( i . e . , n o n p a r a m e t r i c ) nonconvex s e p a r a b l e problems i s b r a n c h and bound methodology t h a t i s based on s o l v i n g a s e q u e n c e of convex u n d e r e s t i m a t i n g problems. I t s u f f i c e s t o o b s e r v e t h a t t h e same t e c h n i q u e s can be used

j o i n t l y i n (x,

E)

t o g e n e r a t e p a r a m e t r i c j o i n t l y convex under-

e s t i m a t i n g problems o f a p a r a m e t r i c j o i n t l y s e p a r a b l e nonconvex program.

Further, s i n c e t h e s a m e i d e a s involving s e p a r a b i l i t y

c a n b e e x t e n d e d t o a s i g n i f i c a n t l y more g e n e r a l s t r u c t u r e c a l l e d " f a c t o r a b i l i t y " (McCormick, 1976), encompassing most f u n c t i o n s commonly e n c o u n t e r e d i n p r a c t i c e , t h e a 2 p l i c a b i l i t y o f r e s u l t s b a s e d on j o i n t l y convex programs t o nonconvex programs i s clear. The d e t a i l s of t h e i n d i c a t e d r e s u l t s and t h e i r a p p l i c a t i o n i n d e v e l o p i n g s o l u t i o n bounds a n d r e f i n e d p a r a m e t r i c s o l u t i o n estimates w i l l n o t b e p u r s u e d f u r t h e r h e r e , b u t a r e p r e s e n t e d i n C h a p t e r 9. W e close by n o t i n g t h a t t h e c l a s s o f problems t h a t have a n

o p t i m a l v a l u e f u n c t i o n w i t h c o n v e x i t y or c o n c a v i t y p r o p e r t i e s is c o n s i d e r a b l y l a r g e r t h a n m i g h t be imagined from a c u r s o r y

analysis.

R e v e r s e convex programs ( e . g . ,

P2(€) w i t h f concave,

t h e gi convex a n d t h e h . a f f i n e i n x ) may be e x p e c t e d t o have a 3

piecewise concave o p t i m a l v a l u e f u n c t i o n .

Also, large classes

60


of parametric posynomial GP problems have a convex optimal value function (all the more intriguing since signomial GP problems may be underestimated or overestimated by posynomial GP problems).

Finally, it is noted that GP duality theory can

be exploited to calculate global parametric bounds on the optimal value function. 2.6.

GENERAL RESULTS FROM RHS RESULTS

The general parametric problem P 3 ( ~ )and the rhs parametric problem P 2 ( ~ )are more closely associated than their formulations may suggest.

In fact, surprisingly, any general problem

of the form minX f(x,

E)

s.t. g(x,

E)

2 0, h(x,

E)

P3 ( € 1

= 0

may be formulated as an e q u i v a l e n t rhs problem minX f(x)

s.t. g(x) 2 E ~ h(x) , =

by simply redefining

E

E

2

P2 ( € 1

in P3(€) to be a variable and intro-

ducing a new parameter a such that E = a.

This results in the

problem min

(XI€)

f(x,

E)

s.t. g(x,

E)

2 0, h(x,

E)

= 0,

€ = a

P (0.1

which is clearly equivalent to P3(€) and of the form P2(~). The relevance and utility of this equivalence is its use in organizing more tightly and relating more precisely the body of results that hold for the various formulations.

It is

obvious that results for P 3 ( ~ )can be applied to P2(~). The reformulation P(a) of P3(€) implies that the converse of this fact is true:

results for P 2 ( ~ )obviously apply to P(a) and

can be applied to P3(~),when the assumptions on P3(") can be

61

2 6 General R e w h troni RHS Studkc\

shown t o imply t h e r e q u i r e d P2(f) a s s u m p t i o n s v i a P(a).

This

h a s many i n t e r e s t i n g a p p l i c a t i o n s i n i m m e d i a t e l y e x t e n d i n g c e r t a i n r e s u l t s f o r P2(€) to t h e i r n a t u r a l g e n e r a l i z a t i o n f o r P3(c).

F o r example, C Q 1 h o l d s f o r P 3 ( E ) a t x = I w i t h

w i t h r e s p e c t t o t h e v e c t o r 7 , i f and o n l y i f V x q ( S r , Vxh(St, F ) y = 0 , and t h e r o w s o f Vxh(F, pendent.

But t h i s means t h a t V

Also, d e f n i n g H E

(,”),

(X,

E

F)y

= E,

> 0,

y ) a r e l i n e a r l y inde-

E )g (-x , T I ( ; ) = Vxg(X, E ) y > 0 .

it follows t h a t

(x, E and t h e rows o f

a r e l i n e a r l y independent. problem P(a) a t ( x , the vector

(y, 0 ) .

E)

=

T h i s means t h a t C Q 1 h o l d s f o r t h e

-

(2, E ) w i t h a

=

T, w i t h r e s p e c t t o

Thus, C Q 1 i n P ( E ) i s i n h e r i t e d by a

r e a l i z a t i o n of P * ( E ) .

A s a n immediate a p p l i c a t i o n ,

t h a t t h e Gauvin-Tolle r e s u l t s , t h e c o n t i n u i t y o f f

*

it f o l l o w s (E)

and t h e

d i f f e r e n t i a l s t a b i l i t y bounds on t h e D i n i d e r i v a t i v e s o f P 2 ( € ) described i n S e c t i o n 2.3,

e a s i l y e x t e n d t o P 3 ( ~ )v i a P(a), i f

t h e n a t u r a l g e n e r a l i z a t i o n s o f t h e Gauvin-Tolle a s s u m p t i o n s a r e assumed t o h o l d f o r P3(€).

This p a r t i c u l a r application

was n o t e d by J a n i n ( p e r s o n a l communication, 1981) and R o c k a f e l l a r ( 1 9 8 2 a ) and i n d e p e n d e n t l y a p p l i e d by G o l l a n (1981a) t o p r o v e some g e n e r a l r e s u l t s . A n o t h e r i m p o r t a n t example i s b a s e d on w e l l known r e s u l t s and i n v o l v e s d e d u c i n g t h e c o n v e x i t y o f t h e o p t i m a l v a l u e funct i o n of t h e g e n e r a l problem P 3 ( € ) from t h e c o n v e x i t y o f t h e

62

2 Basic Sensitivity and Stahility Results

o p t i m a l v a l u e f u n c t i o n f o r P2(c).

A c l a s s i c a l r e s u l t is t h e

c o n v e x i t y o f t h e o p t i m a l v a l u e f u n c t i o n o f P2(€), i f t h i s problem i s convex i n x.

It follows t h a t t h e optimal value of

P(a) i s convex if t h i s p r o b l e m i s j o i n t l y convex i n ( x , E). But c l e a r l y , i f P 3 ( € ) i s convex i n ( x ,

E)

,

t h e n so i s P ( a ) .

Hence, t h e f o l l o w i n g r e s u l t , p r o v e d i n d e p e n d e n t l y o f t h e r e s u l t

f o r P2 ( E ) by Mangasarian a n d Rosen ( 1 9 6 4 ) , i s s e e n t o be i m m e d i a t e l y i m p l i e d by t h e P 2 ( ~ )r e s u l t : P

3

(E)

t h e optimal value of

i s convex i f t h i s problem i s j o i n t l y convex i n ( x ,

E).

W e make i m p o r t a n t u s e o f t h i s r e s u l t i n t h e b o u n d i n g p r o c e d u r e

d e s c r i b e d i n Chapter 9 , as i n d i c a t e d i n t h e l a s t s e c t i o n . The s i t u a t i o n i s n o t a l w a y s s o s t r a i g h t f o r w a r d , b u t c l e a r l y c a r e f u l examination of extensions t o P 3 ( c ) v i a P(a) i s w a r r a n t e d whenever a r e s u l t f o r P2(c) h a s b e e n o b t a i n e d .

Ob-

v i o u s l y , i f a r e s u l t f o r P 3 ( ~ )c a n be p r o v e d d i r e c t l y w i t h l i t t l e more e f f o r t t h e n t h a t r e q u i r e d f o r t h e c o r r e s p o n d i n g

P 2 ( € ) r e s u l t , t h i s would p r o b a b l y be p r e f e r a b l e .

However,

c o n s i d e r a b l e t h e o r e t i c a l s i m p l i f i c a t i o n s and a more c o n s t r u c t i v e p r o o f m i g h t make i t d e s i r a b l e t o d e v e l o p a r e s u l t f o r

.

P2(E) and e x t e n d t o P 3 ( € ) t h r o u g h P ( a )

I t would b e i n t e r -

e s t i n g t o e x p l o r e e x i s t i n g r e s u l t s from t h i s p o i n t of view. 2.7.

SUMMARY

T h i s c h a p t e r p r o v i d e s a c o n c i s e s u r v e y o f a number o f b a s i c s e n s i t i v i t y and s t a b i l i t y r e s u l t s f o r g e n e r a l classes o f n o n l i n e a r p a r a m e t r i c problems.

I t s u r e l y does n o t c o n t a i n a l l

t h e i m p o r t a n t f i n d i n g s , b u t h o p e f u l l y g i v e s a good i d e a o f s e v e r a l mainstream d i r e c t i o n s of r e s e a r c h t h a t have evolved and a r e s t i l l b e i n g d e v e l o p e d .

In the past several years, a s

63

2.7 Summary

the references will attest, developments have accelerated and the level of sophistication of results has been appreciably elevated. We shall endeavor to anticipate some of the future research directions in the final chapter.

Otherwise, except for the

hounds results in Chapter 9, the rest of this book is devoted to a rather detailed treatment of a fundamental segment of the methodology that has been developed rather thoroughly in the past few years.

This body of results is perhaps the most

highly structured for the general problem of interest.

It is

based on one of the strongest collections of assumptions that might be invoked, while still remaining in the province of assumptions that are typically utilized to obtain the strongest consequences in other important mathematical programming applications, e.g.,

strong characterizations of optimality,

and convergence and rate of convergence of algorithms. The assumptions needed for these ideal results are the assumptions of Theorem 2.4.4:

second-order sufficiency, linear

independence, and strict complementary slackness conditions. These provide an abundance of theoretical sensitivity and stability results and sufficient structure to readily allow the development of simple formulas and solution-algorithmbased computational approximation techniques.

It is perhaps

to be expected that this collection of results would be one of the first to be most fully developed and applied.

The results

are of intrinsic interest and are useful, since practical problems meet the requisite conditions surprisingly often. They also provide an essential interface with more general

64


r e s u l t s and a v i t a l p e r s p e c t i v e and p o i n t of d e p a r t u r e , e s t a b l i s h i n g e s s e n t i a l l y t h e most t h a t c a n be e x p e c t e d under r a t h e r i d e a l circumstances.

Theory and Calculation of Solution Parameter Derivatives


Chcipter--3

Sensitivity Analysis under Second-Order Assumptions

3.1.

INTRODUCTION In this chapter we give the theoretical validation of the

result stated as Theorem 2 . 4 . 4 in Chapter 2, a basic characterization of the sensitivity of a local solution of a general mathematical programming problem to a small parametric variation of the problem functions.

This material is based largely

on Fiacco (1976) and Armacost and Fiacco (1975).

Although

existence results for higher-order derivatives are immediate, the analysis concentrates on the explicitly tractable firstorder (i.e., first partial derivative) sensitivity information. As noted in Chapter 2, the theorem is an extension of a result presented by Fiacco and McCormick (1968, Theorem 6). The distinguishing feature of the Fiacco-McCormick result was a demonstration of the relationship of the second-order sufficient optimality conditions (for a strict local solution) to the existence and behavior of first-order variations of a local solution and the associated Lagrange multipliers, when the problem functions are subject to parametric variations, and an explicit representation of the first partial derivatives of the local solution point and the associated Lagrange

67

68

3 Sensitivity Analysis under Second-Order Assumptions

multipliers with respect to the problem parameters. 3

-

Chapters

5 describe the extension of this theory to incorporate a

larger class of parametric problems, and Chapter 6 provides a synthesis of this sensitivity theory with classical penalty function theory. The basic sensitivity result is presented in Section 3.2 and applies to a large class of twice-differentiable parametric programs.

The partial parameter derivatives of a

Karush-Kuhn-Tucker triple are shown to exist and are explicitly obtained, under well known assumptions.

Existence of higher-

order partial parameter derivatives follows readily. small examples illustrate the results in Section 3.3.

Two The

existence and representation of the first and second partial derivatives of the optimal value function follow easily, and are given in Section 3.4. 3.2.

FIRST-ORDER SENSITIVITY ANALYSIS OF A SECOND-ORDER LOCAL SOLUTION Consider the problem of determining a local solution

X(E)

of

,..., P), k is a parameter vector in E . h.(x, 3

where x

E

En and

E

E)

= 0 (j = 1

P3(E)

The Lagrangian of P 3 ( ~ )is defined as

(3.2.1)

3.2 Analysis of a Second-Order Local Solution

T h r o u g h o u t t h e book, i f t h e r e are no i n e q u a l i t y cons t r a i n t s , s i m p l y s u p p r e s s r e f e r e n c e t o t h e f u n c t i o n s gi and t h e a s s o c i a t e d m u l t i p l i e r s ui,

and do l i k e w i s e f o r t h e h

1

and

i f t h e r e are no e q u a l i t i e s . Recall t h a t t h e gradient vector j a n d H e s s i a n m a t r i x o p e r a t o r s , V and V2, a r e t a k e n t h r o u g h o u t

w

with r e s p e c t t o x i f they a r e not subscripted. W e are i n t e r e s t e d i n analyzing t h e behavior of a l o c a l

s o l u t i o n x ( T ) o f P ( T ) when T i s s u b j e c t t o p e r t u r b a t i o n . 3

For

s i m p l i c i t y i n n o t a t i o n w e s h a l l assume t h a t F = 0 ( w i t h o u t l o s s C o n d i t i o n s w i l l be g i v e n t h a t g u a r a n t e e t h e

of g e n e r a l i t y ) .

existence of a l o c a l s o l u t i o n of x ( 0 ) f o r

E

X(E)

of P 3 ( € ) i n a neighborhood

i n a neighborhood of 0 , a l o n g w i t h t h e a s s o c i -

a t e d o p t i m a l Lagrange m u l t i p l i e r s

U(E)

and w ( f ) .

Under t h e s e

c o n d i t i o n s t h e components o f a l l t h e s e q u a n t i t i e s a r e shown t o b e u n i q u e l y d e f i n e d d i f f e r e n t i a b l e f u n c t i o n s of neighborhood of

E

f

in a

= 0.

W e s h a l l make c r i t i c a l u s e o f t h e s e c o n d - o r d e r

sufficient

c o n d i t i o n s f o r a s t r i c t l o c a l s o l u t i o n of a mathematical programming p r o b l e m .

T h e s e c o n d i t i o n s a r e now w e l l known,

a l t h o u g h t h e y have been s e r i o u s l y e x p l o i t e d o n l y i n t h e p a s t several years.

They were d e v e l o p e d a n d v e r i f i e d by F i a c c o and

McCormick ( 1 9 6 8 ) a n d o t h e r s .

For t h e problem P j ( 0 ) t h e

c o n d i t i o n s may b e s t a t e d a s f o l l o w s . Lemma 3 . 2 . I .

[Second-order s u f f i c i e n t c o n d i t i o n s f o r a

s t r i c t l o c a l m i n i m i z i n g p o i n t of p r o b l e m P (O).] 3

I f the

f u n c t i o n s d e f i n i n g problem P 3 ( 0 ) are t w i c e c o n t i n u o u s l y d i f f e r e n t i a b l e i n a neighborhood o f x

*

,

minimizing p o i n t o f problem P ( 0 ) hood o f x

*

3

then x

*

is a strict local

( i . e . , t h e r e i s a neighbor-

s u c h t h a t t h e r e d o e s n o t e x i s t any f e a s i b l e x # x

*

I0

3 Sensitivity Analysis under Second-Order Assilrnptions

*

where f ( x , 0 ) 5 f ( x v e c t o r s u*

,

Em a n d w*

E

0 ) ) i f t h e r e e x i s t (Lagrange m u l t i p l i e r ) E

Ep s u c h t h a t t h e f i r s t - o r d e r

Karush-

Kuhn-Tucker c o n d i t i o n s ( K a r u s h , 1939; Kuhn a n d T u c k e r , 1 9 5 1 ) hold, i.e., gi(X hj(x

*

ui g i ( x

* I

* *

0) 2 0,

, 0)

,

= 0,

0) = 0 ,

*

ui 2 0 ,

*

*

..., m , j = 1, ..., P , i 1, ..., m, i 1, ..., m, i = 1,

=

=

m

*

i=l

+

P

1

j =1

w*Vh.(x*, 0) = 0 3 3

and, f u r t h e r , i f

T 2 * * * z V L(x , u , w , C)z > 0 Vgi(x Vgi(x

*

*

f o r a l l z # 0 such t h a t

,

0)z 1. 0

f o r a l l i , where g i ( x

,

0)z

f o r a l l i , where ui

,

0)z = 0 ,

=

0

*

*

,

0) = 0.

> 0,

and

*

Vh.(x 3

j = 1,

..., p.

Han a n d Mangasarian ( 1 9 7 9 ) n o t e d t h a t t h e r e s t r i c t i o n s o n z are e q u i v a l e n t t o z

+

0, v f ( x * , 0)z = 0, v g i ( x * ,

' *

0)z 1.

o

if

g i ( x * , 0 ) = 0 , and Vh. ( x , 0) z = 0 f o r e v e r y j . 3 These c o n d i t i o n s a r e a p p l i c a b l e w h e t h e r o r n o t c o n s t r a i n t s a r e p r e s e n t , and w h e t h e r o r n o t t h e r e e x i s t s a v e c t o r z a s indicated.

I f t h e r e a r e no c o n s t r a i n t s , t h e a b o v e c o n d i t i o n s

7I

3 . 2 A n a l y w of a Second-Order Local Solution

a r e l o g i c a l l y v a l i d i f r e f e r e n c e t o t h e c o n s t r a i n t s i s supT h i s l e a d s t o t h e w e l l known s u f f i c i e n t c o n d i t i o n s

pressed. that x

*

be a n i s o l a t e d

(i.e.,

l o c a l l y u n i q u e ) l o c a l uncon-

*

s t r a i n e d minimizing p o i n t o f f ( x , 0 ) : Vf(x T 2

z V f

* (x ,

# 0.

0)z > 0 for all z

,

0 ) = 0 and

I f t h e r e a r e no z # 0 s a t i s -

fying the indicated relationships with the constraint gradients a n d t h e f i r s t - o r d e r Karush-Kuhn-Tucker

*

(x

,

u

*

,

*

conditions hold, then

w ) a g a i n l o g i c a l l y s a t i s f i e s t h e s e second-order

conditions.

A s a p o i n t of i n t e r e s t i t i s o b s e r v e d t h a t i n

s u c h a case t h e r e m u s t b e n l i n e a r l y i n d e p e n d e n t b i n d i n g constraint gradients a t x

*

.

For example, i f P 3 ( 0 ) is a l i n e a r

programming p r o b l e m a n d t h e s e c o n d - o r d e r

*

(x

,

u

*

,

*

w ),

2

then, since V L

3

conditions hold a t

0 , t h e r e c a n be n o z # 0

satisfying the given inequalities, i.e.,

x

*

must be a v e r t e x

o f t h e f e a s i b l e r e g i o n d e f i n e d by t h e c o n s t r a i n t s o f P 3 ( 0 ) . I t i s c a u t i o n e d t h a t t h e c o n c l u s i o n o f Lemma 3 . 2 . 1

be s t r e n g t h e n e d t o assert t h a t x

*

can not

i s a l o c a l l y unique and hence

i s o Z a t e d l o c a l minimum o f p r o b l e m P 3 ( 0 ) , a s e r r o n e o u s l y a s s e r t e d i n F i a c c o a n d McCormick ( 1 9 6 8 , Theorem 4 )

is proved t h a t x

*

i s a s t r i c t l o c a l minimum).

counterexample i n E l ,

P:

( w h e r e it

A simple

min x L / 2 s . t . h ( x ) = 0 , w h e r e h ( x ) =

x 6 s i n ( l / x ) i f x # 0 a n d h ( x ) = 0 i f x = 0 w a s p r o v i d e d by Robinson 3.2.1

(1980b).

I n t h i s e x a m p l e , t h e a s s u m p t i o n s o f Lemma

hold a t x = 0 and 0 is a s t r i c t local ( i n f a c t , g l o b a l )

minimum o f p r o b l e m P.

However, i t f o l l o w s t h a t xk = l / k r i s a

l o c a l s o l u t i o n ( i n f a c t , a n i s o l a t e d f e a s i b l e p o i n t and h e n c e a n i s o l a t e d l o c a l s o l u t i o n ! ) of p r o b l e m P f o r a n y i n t e g e r k # 0 , a n d e v e r y n e i g h b o r h o o d of x = 0 c o n t a i n s xk f o r Ikl s u f f i ciently large.

Hence, x

=

9 i s c e r t a i n l y n o t a n i s o l a t e d local

72

3 \cn\itir

(or global) minimum.

ity Andly\i\ under Second-Order A\surnption\

Robinson then shows that x

*

is an iso-

lated local minimum if the second-order sufficient conditions of Lemma 3 . 2 . 1 are strengthened by assuming these hold for all optimal Lagrange multipliers (u, w) associated with x^ and if it is also assumed that CQ1 holds at x

*

.

Note that this

extended form of the second-order sufficient conditions in Lemma 3 . 2 . 1 is n o t enough per se to imply that the local

*

minimum x

is isolated, since this stronger condition holds in

Robinson's example.

Hence, an additional regularity assumption

such as CQ1 is needed. If we add the assumptions of linear independence (CQ3) and strict complimentary slackness to the second-order strict

*

sufficiency conditions at (x

,

01,

and the appropriate differ-

entiability assumptions, we obtain the following result. [Basic Sensitivity Theorem (Fiacco, 1976).

Theorem 3 . 2 . 2 .

First-order sensitivity results for a second-order local

*

minimizing point x . I

If

(i) the functions defining P 3 ( E ) are twice continuously differentiable in x and if their gradients with respect to x and the constraints are once continuously differentiable in

*

in a neighborhood of (x

,

E

0),

(ii) the second-order sufficient conditions for a local minimum of P3(0) hold at x*, with associated Lagrange multipliers u

*

and w

*

,

(iii) the gradients Vgi(x

*

= 0) and Vh. ( x

3

,

*

,

0) (for i such that gi(x

*

,

0) (all j) are linearly independent (CQ3),

0)

73

3.2 Analysis of a Second-Order Local Solution

and

*

(iv) ui > 0 when gi(x

*

,

0) = 0 (i = 1,

...,m)

(i.e.,

strict complementary slackness holds), then (a) x

*

is a local i s o l a t e d minimizing point of

problem P3(0) and the associated Lagrange multipliers u (b) for

E

*

and w

*

are unique,

in a neighborhood of 0, there exists a

unique, once continuously differentiable vector function Y(E) = [x(E), ~ ( € 1 ~,( € 1 T 1 satisfying the second-order sufficient conditions for a local minimum of problem P(E) such * * * T * that y(0) = (x , u , w ) = y , and hence X(E) is a l o c a l l y u n i q u e local minimum of problem P3(c) with associated unique Lagrange multipliers U(E) and ~ ( € 1 , and (c) for

E

near 0, the set of binding inequalities

is unchanged, strict complementary slackness

holds, and the binding constraint gradients are linearly independent at Part (a) follows if (b) is true.

Proof.

~(€1. It is stated

separately because it is of intrinsic interest.

That x

*

is a

from assumption (ii), strict local minimum of ~ ~ ( follows 0 ) which also implies that VL(x of u* and w

*

*

,

u

*

* ,w ,

0) = 0.

The uniqueness

follows from this and assumption (iii).

The proof of (b) follows from a straightforward application of the implicit function theorem to the first-order necessary optimality conditions of P3(€), as follows.

A s s u m p t i o n ( i i ) i m p l i e s t h e s a t i s f a c t i o n of t h e K a r u s h Kuhn-Tucker

first-order

VL(x, u , w ,

uigi(x,

a t ( x , u, w,

E)

0

E)

=

E)

= 0,

(x

=

conditions

*

,

i = 1, . . . , m ,

u

*

* ,w ,

(3.2.2)

Assumption ( i )i m p l i e s

0).

i s once continuously

t h a t t h e system of equations (3.2.2)

d i f f e r e n t i a b l e i n a l l t h e arguments; so, i n p a r t i c u l a r , Jacobian m a t r i x of defined.

with respect t o (x, u, w) i s w e l l

(3.2.2)

( i i ) ,( i i i ) ,a n d ( i v )

I t follows t h a t assumptions

imply t h e e x i s t e n c e of t h e i n v e r s e o f t h i s m a t r i x a t ( x

w

*

,

0)

the

*

,

u

*

,

( a n i m m e d i a t e e x t e n s i o n o f Theorem 1 4 o f F i a c c o a n d

McCormick ( 1 9 6 8 ) , a n d now a w e l l known r e s u l t . ) The a s s u m p t i o n s o f t h e i m p l i c i t f u n c t i o n t h e o r e m (Theorem w i t h r e s p e c t to t h e equations (3.2.2)

2.4.1)

*

solution (x

,

u

*

*

,w ,

and t h e p a r t i c u l a r

0 ) are s a t i s f i e d and w e can c o n c l u d e

*

t h a t i n a neighborhood of

(x

,

u

*

,

w

*

),

for

E

i n a neighborhood

of 0 , t h e r e e x i s t s a unique once continuously d i f f e r e n t i a b l e f u n c t i o n [ x ( E ), u ( E ) ,

*

u ( O ) , w ( O ) ] = (x that for

E

,

near 0,

u

*

W(E)

,

X(E)

w

*

1 s a t i s f y i n g ( 3 . 2 . 2 ) w i t h [ x ( O ),

).

The s a t i s f a c t i o n o f

(3.2.2)

means

i s a f i r s t - o r d e r Karush-Kuhn-Tucker

p v i n t of p r o b l e m P 3 ( € ) , w i t h a s s o c i a t e d L a g r a n g e m u l t i p l i e r s U(E)

and

w(E).

To p r o v e

( c ), w e f i r s t n o t e t h a t t h e b i n d i n g c o n s t r a i n t

s e t a t x ( 0 ) r e m a i n s t h e same f o r

E

n e a r 0.

This i s seen

immediately f o r t h e e q u a l i t i e s h . [ x ( E ) , € 1 = 0, s i n c e X ( E ) 3 s a t i s f i e s ( 3 . 2 . 2 ) n e a r E = 0. For t h e i n e q u a l i t i e s , w e h a v e from ( 3 . 2 . 2 ) E

= 0.

t h a t u ~ ( E ) ~ = [ x ( E ) E, ] = 0 ( i = 1,

If gi[x(0),

..., m )

01 = 0 f o r some i , t h e n u i ( 0 )

> 0

near

75

3 . 2 Analysis of' a Second-Order Local Solution

( b y s t r i c t complementary s l a c k n e s s ) , h e n c e ui E

= 0 by c o n t i n u i t - y o f

> 0 near

and w e c o n c l u d e t h a t g . [ x ( f ) ,

U(E)

I f g i [ x ( 0 ) , 01 > 0 f o r s o m e i , t h e n g i [ x ( E ) ,

= 0. E

(E)

= 0 by c o n t i n u i t y .

E]

> 0 near

E]

Therefore, defining

B ( E ) :{ i l g . [ x ( E ) ,

E]

=

01,

w e have concluded t h a t B ( E ) = B ( 0 ) f o r

E

n e a r 0.

(The a r g u -

ment a l s o shows t h a t s t r i c t complementary s l a c k n e s s i s p r e served f o r

E

n e a r 0 , p r o v i n g t h e f i r s t p a r t of

W e now show t h a t t h e s e c o n d - o r d e r

conditions E

near 0.

any

E

I E ~
0 f o r any v e c t o r

= 0 for a l l i

Z(E)

Then t h e r e m u s t e x i s t Vgi [ x ( E ~ ) E,

~

f o r a l l k.

E~

V L[x(E), u(E), [x(E),

E]Z(E) = 0 for a l l j. 3 Suppose t h e a s s e r t i o n i s f a l s e .

E

2

... .

T 2

# 0 s u c h t h a t Vg.

> 0 and zk

zk] = 0 f o r a l l i

f o r a l l j , and

Z(E)

B ( 0 ) and Vh. [ x ( E ) ,

E

T h i s may b e p r o v e d a s f o l l o w s .

k = 1, 2 ,

sufficient optimality

W e must show t h a t t h e r e e x i s t s 6 > 0 s u c h t h a t f o r

such t h a t

w(E)]z(E) E]Z(E)

(Lemma 3 . 2 . 1 )

(c).)

# 0 such t h a t

B(0)

,

E~

Oh. [ x ( E ~ ) , 3

E ~ z kI

V L [ x ( E ~ ) ,u ( E ~ ) , w ( E ~ ) , ~

~

W i t h o u t l o s s o f g e n e r a l i t y , assume

S e l e c t a c o n v e r g e n t s u b s e q u e n c e of { z

t h e subsequence { z

k

Taking l i m i t s a s k

1

0,

+

k

= 0

5 01 f o r2 k

112

11

= 1

1, relabel

f o r c o n v e n i e n c e and c a l l t h e l i m i t Z.

and r e c a l l i n g a s s u m p t i o n ( i ) y i e l d s 2 * * * t h e c o n c l u s i o n t h a t ZT V L ( x , u , w , 0 ) Z L 0 f o r some F s u c h + +m

t h a t 11Tl1 = 1, Vgi(x*,

0)Z = 0 for a l l i

E

*

B ( 0 ) and V h . ( x 3

,

But t h i s i s a c o n t r a d i c t i o n of a s s u m p t i o n

0)Z = 0 f o r a l l j .

( i i ) and t h e p r o o f o f t h e a s s e r t i o n i s c o m p l e t e . Since it w a s established t h a t s o l v e s (3.2.2)

for

E

[x(E), u(E), w(E)]

near 0 , it follows t h a t

X(E)

uniquely

is a locally

~

76


u n i q u e l o c a l minimum of P ( E ) w i t h a s s o c i a t e d u n i q u e L a g r a n g e multipliers

U(E)

and

W(E)

,

completing t h e proof o f p a r t ( b ) .

The p r e s e r v a t i o n of s t r i c t complementary s l a c k n e s s w a s p r o v e d above.

+

the (say) r

The p r e s e r v a t i o n o f t h e l i n e a r i n d e p e n d e n c e o f p binding c o n s t r a i n t g r a d i e n t s a t

X(E)

n e a r 0 f o l l o w s d i r e c t l y from t h e f a c t t h a t a n ( r

+

for E

p ) by

( r + p ) J a c o b i a n o f t h e s y s t e m o f e q u a t i o n s d e f i n e d by t h e c o n s t r a i n t s t h a t a r e b i n d i n g a t x ( 0 ) must be n o n s i n g u l a r , a l o n g w i t h t h e assumed c o n t i n u i t y o f t h e f i r s t d e r i v a t i v e s . The p r o o f i s c o m p l e t e . Robinson ( 1 9 7 4 ) p r o v e d a n a n a l o g o u s t h e o r e m (Theorem 2.4.7, C h a p t e r 2 ) u n d e r weaker a s s u m p t i o n s ( s p e c i f i c a l l y , a s s u m p t i o n ( i ) i s r e p l a c e d by t h e a s s u m p t i o n t h a t t h e s e c o n d p a r t i a l

d e r i v a t i v e s o f t h e problem f u n c t i o n s w i t h r e s p e c t t o x are j o i n t l y c o n t i n u o u s i n (x, c o n t i n u i t y of

[x(E)

,

U(E)

,

W(E)

bounds o n t h e v a r i a t i o n of changes i n

E

and d e m o n s t r a t e d t h e r e s u l t i n g

E ) ) ,

] near

[x(E)

,

E

U(E)

H e also o b t a i n s

= 0.

, w(E)]for

small

a n d u s e s t h e s e t o d e t e r m i n e con-

(Theorem 2.4.8)

vergence r a t e s f o r a l a r g e class of algorithms. Theorem 3.2.2

reduces to t h e following statement i f t h e r e

a r e nu c o n s t r a i n t s p r e s e n t or b i n d i n g i n P 3 ( 0 ) . C o r o l l a r y 3.2.3.

[ S e n s i t i v i t y a n a l y s i s f o r t h e uncon-

s t r a i n e d problem: minx f (x,

E)

.I

I f f (x,

o u s l y d i f f e r e n t i a b l e i n x and V x f ( x , differentiable i n

E,

E)

is t w i c e c o n t i n u -

E)

i s once c o n t i n u o u s l y

i n a neighborhood of

*

(x

,

01,

and if t h e

s e c o n d - T r d e r s u f f i c i e n t c o n d i t i o n s f o r a n u n c o n s t r a i n e d local minimum of f ( x , 0 ) h o l d a t x

*

then, f o r

E

near 0, there e x i s t s

a unique once c o n t i n u o u s l y d i f f e r e n t i a b l e v e c t o r f u n c t i o n s a t i s f y i n g t h e second-order s u f f i c i e n t c o n d i t i o n s for an

X(E)

77

3.2 Analysih of a Second-Order Local Solution

u n c o n s t r a i n e d l o c a l minimum o f f ( x , hence,

E)

such t h a t x ( 0 ) = x

*

and

i s a l o c a l l y u n i q u e u n c o n s t r a i n e d l o c a l minimum o f

X(E)

f(xr €1 * With (x, u , w ) = [ x ( E ) , u ( E ) , w ( E ) ] , ( 3 . 2 . 2 ) i s i d e n t i c a l l y satisfied for

to

E

n e a r 0 and c a n b e d i f f e r e n t i a t e d w i t h r e s p e c t

t o y i e l d e x p l i c i t expressions f o r the f i r s t p a r t i a l

E

d e r i v a t i v e s of t h i s v e c t o r f u n c t i o n . It follows t h a t

M ( E ) V ~ Y ( E ) = N(E),

(3.2.3)

where -

v2L

I

T

-Vg1,

... -vgmT

91

UlV91

Ohl, T

... VhTP

0

0

M E

0

UmVgm

gm

I

r

0

m

t h e J a c o b i a n m a t r i x o f ( 3 . 2 . 2 ) w i t h r e s p e c t t o y = (x, u , w ) ’ , e v a l u a t e d a t [ Y ( E ) , E ] and N

f

[-v2

L ~ -ulvEgl, , T

EX

...

r

-umVEgm, T -VEhl, T

..., -V

hTIT E P

t h e n e g a t i v e of t h e J a c o b i a n m a t r i x o f ( 3 . 2 . 2 ) w i t h r e s p e c t t o E,

evaluated a t

[Y(E),

€1.

Since M i s nonsingular f o r

E

near

0, it follows t h a t

vEy(E) = M(E)-~N(E)

(3.2.4)

78


where the quantities are evaluated as in (3.2.3) and neighborhood of 0. v,y(o)

In particular, we have that (3.2.5)

(M*)-~N*,

=

where y(0) = [ x ( O ) , u(O), w 0)IT and N

*

is in a

E

= N(0).

S

*

(x

,

u

*

,w

* T )

,

M

*

= M(0)

Also, since the quantities involved in (3.2.4)

are all continuous in their arguments (from the assumptions of the theorem and the consequences of the implicit function theorem), (3.2.5)

is also the limit of

3.2.4)

as

0, i.e.

E +

(3.2.6

lim VEy(~)= VEy(0). E+O

It is important to note that VEy(~), a first-order estimate of the variation of an isolated local solution

X(E)

of problem P 3 ( € ) and the associated unique Lagrange multipliers U(E) and w(E), can b e c a Z c u Z a t e d from (3.2.4) once [ x ( E ) , u(E), w(E)] has been determined.

In particular, VEy(0), and hence a

first-order Taylor's series approximation of in a neighborhood of u

*

,w

*

)

is known.

E =

[x(E),

u(E), W(E)

*

0 is available from (3.2.5) once (x

,

This is summarized in the following

statement. CoroZZary 3 . 2 . 4 .

w(E)] near

= 0.1

E

[First-order estimation of [x(E), u(E), Under the assumptions of Theorem 3.2.2,

a

first-order approximation of [ x ( E ) , u(E), w(E)] in a neighborhood of

E

= 0 is given by

(3.2.7)

*

*

, w* )

*

where (x

,

and

and N ( E ) are defined as in (3.2.31,

M(E)

means that

u

= [x(O), u(O), ~ ( 0 1 1 , M

@ ( ~ ) / l l ~ l0l as +

E

+

0.

= M(O),

and

N

@(El

*

= N(O), =

O(llEll)

3 . 2 t\naly\i\ 01

:i

79

Second-Ordcr I.ocal Soltilitin

i s a n immediate

The f o l l o w i n g c o r o l l a r y o f Theorem 3 . 2 . 2 i m p l i c a t i o n o f t h e i m p l i c i t f u n c t i o n theorem.

[Exist2nce of higher-order derivatives.

Corollary 3.2.5.

I f t h e c o n d i t i o n s o f Theorem 3 . 2 . 2

hold, with t h e respective

assumed o r d e r s o f d i f f e r e n t i a b i l i t y b e i n g p a s s u m e d , w i t h p 1. 1, t h e n i n a n e i g h b o r h o o d of

E

:[x(E),

Y(E)

= 0.

I

-

1 m o r e than t h a t

u(E),

w ( E ) I ~E

Cp

I f t h e problem f u n c t i o n s a r e

a n a l y t i c i n (x, c ) i n a n e i g h b o r h o o d o f a n a l y t i c i n a n e i g h b o r h o o d of

E

*

(x

,

then

O),

Y(E)

is

= 0.

The e x i s t e n c e a n d c o n t i n u i t y c f t h e h i g h e r - o r d e r

Proof.

p a r t i a l d e r i v a t i v e s and a n a l y t i c i t y r e s u l t from t h e i m p l i c i t f u n c t i o n theorems,

Theorem 2 . 4 . 1

system of e q u a t i o n s (3.2.2) If f(x,

applied to the

a s i n t h e p r o o f o f Theorem 3 . 2 . 2 .

i s n o t c o n s t r a i n e d , and t h e c o n d i t i o n s o f

E)

C o r o l l a r y 3.2.3 E

and 2.4.2,

2 * hold, then V f ( x

,

0)

-1

e x i s t s and hence n e a r

= 0 t h e r e e x i s t s a once continuously d i f f e r e n t i a b l e f u n c t i o n

t h a t l o c a l l y u n i q u e l y s a . t i s f i e s Vxf [ x ( E ) ,

X(E)

f e r e n t i a t i o n by VXf 2

VEX

+

E

v:xf

and t h e r e f o r e n e a r

v

x

=

E]

= 0.

Dif-

yields = 0, E

= o

-v X2 f - l v 2E X f

where t h e q u a n t i t i e s are e v a l u a t e d a t x ( E ) .

The a p z l o g y of

80


EXAMPLES

3.3.

The a p p l i c a t i o n o f t h e p r e v i o u s t h e o r e t i c a l r e s u l t s f o r e s t i m a t i n g s e n s i t i v i t y i n f o r m a t i o n may be i l l u s t r a t e d by t h e f o l l o w i n g s i m p l e examples. E x a m p Z e 3 . 3.1 min X x

s.t. x

(x e E ~ ) .

E

The s o l u t i o n o f t h e problem i s g i v e n u n i q u e l y by X C E ) f o r any

E.

The h y p o t h e s e s o f Theorem 3.2.2

v e r i f i e d f o r t h i s problem.

=

E,

may b e r e a d i l y

(Note t h a t t h e second-order

suf-

f i c i e n t optimality conditions a r e s a t i s f i e d i n a l o g i c a l sense, s i n c e t h e r e a r e no nonzero v e c t o r s o r t h o g o n a l t o t h e b i n d i n g constraint gradients.) L(x, u,

x

E)

-

u(x

-

E)

Tucker c o n d i t i o n s y i e l d dx ( € 1 dU'E) 7 = 1 , - dE

i s g i v e n by

The Lagrangian (3.2.1)

and t h e f i r s t - o r d e r Karush-Kuhn-

U(E)

= 0,

= 1.

We t h u s f i n d

for a l l

E.

E x a m p l e 3. 3.2 min where

E~

X1

+

E1X2

> 0 and

S.t.

# 0 (x

E~

>

X2

E

2 2

E 2 X 1, - X I

2

2 0

The problem i s d e p i c t e d i n

E ).

F i g . 3.3.1. The f i r s t o r d e r Karush-Kuhn-Tucker

c o n d i t i o n s imply t h a t

t h e s o l u t i o n and t h e a s s o c i a t e d Lagrange m u l t i p l i e r s a r e u n i q u e l y g i v e n by 2

) 1 x ~ ( E ) = - 1 / 2 ~ ~ x~ 2~( ~ , =

/

2 2 4

E

=

and u ~ ( E= ) E ~ ,

~ ~ ( =€ 01, where

~

~

~

~

81

3.3 Examples

Fig.

3.3.1.

D e p i c t i o n of ExampZe,-%. 3 . 2 .

It is easily verified that the second-order conditions hold for [x(E)

,

U(E)

]

as given, and hence it is observed that all

the hypotheses of Theorem 3.2.2 E

are satisfied for any value of

as indicated. It follows that 1

2 2 / 2

3 ~~ ~ / E ~ ~ ~E ~~

and

These matrices,.whose coefficients give the desired firstorder sensitivity information, can also be calculated directly from (3.2.4).

82

3 Sensitivity Analysis under Second-Order Assumption$

The n e x t s e c t i o n d e v e l o p s f o r m u l a s f o r t h e d e r i v a t i v e s o f t h e "optimal value function," f

*

(E)

P 3 ( ~ a) n d P 2 ( € ) a l o n g t h e Karush-Kuhn-Tucker tX(E) r

3.4.

~ ( € r1

W(E)

I

€ 1 , of problems

= f[x(E),

-

point trajectory

FIRST- AND SECOND-ORDER PARAMETER DERIVATIVES O F THE OPTIMAL VALUE FUNCTION Because of i m p o r t a n t c o n n e c t i o n s w i t h Lagrange m u l t i p l i e r s

and d u a l i t y t h e o r y , f i r s t - o r d e r

changes i n t h e c p t i m a l v a l u e

f u n c t i o n have t r a d i t i o n a l l y been a n a l y z e d w i t h r e s p e c t t o v a r i a t i o n s i n t h e " r i g h t hand s i d e " of t h e c o n s t r a i n t s . (1972) a l s o d e r i v e s second-order

Buys

changes i n c o n n e c t i o n w i t h a n

a n a l y s i s of t h e b e h a v i o r o f t h e o p t i m a l v a l u e o f a n a s s o c i a t e d A more c o m p l e t e t r e a t m e n t was

augmented L a g r a n g i a n f u n c t i o n .

g i v e n s u b s e q u e n t l y by A r m a c o s t and F i a c c o ( 1 9 7 6 ) . a s s u m p t i o n s of Theorem 3.2.2, extended these f i r s t - o r d e r

Under t h e

Armacost a n d F i a c c o ( 1 9 7 5 )

r e s u l t s a n d a l s o d e v e l o p e d second-

o r d e r r e s u l t s f o r a g e n e r a l class of p a r a m e t r i c v a r i a t i o n s . The o t h e r r e f e r e n c e d r e s u l t s a r e o b t a i n e d as s p e c i a l cases. Theorem 3.4.1

a n d C o r o l l a r y 3.4.4

Theorems 2 . 4 . 1 2 Let Y(E)

a n d 2.4.13.

= [x(E)

t r i p l e , where

were c i t e d i n C h a p t e r 2 as

X(E)

,

U(E)

,

W(E)

I T b e a Karush-Kuhn-Tucker

s o l v e s p r o b l e m P 3 ( ~ )f o r

E

n e a r 0.

Then

t h e ( l o c a l ) "optimal value function" i s defined a s f*(E)

f[X(E)r

€1,

(3.4.1)

and t h e " o p t i m a l v a l u e L a g r a n g i a n " i s d e f i n e d a s L*(€) = L [ x ( E ) , u ( E ) ,

w(E)

, €1.

(3.4.2)

83

3.4 Derivatives of the Optimal Value Function

Theorem 3 . 4 . I .

[ F i r s t - and s e c o n d - o r d e r

I f t h e c o n d i t i o n s o f Theorem

optimal value function of P ( E ) ] . 3 3.2.2

changes i n t h e

h o l d f o r p r o b l e m P 3 ( € ) , and i f t h e problem f u n c t i o n s a r e

t w i c e continuously d i f f e r e n t i a b l e i n (x, i n a neighborhood of (a)

(b)

f

*

(E)

= L

*

E)

*

near ( x

,

0) then,

= 0,

E

(3.4.3)

(E);

m

2

-

V E f * ( E ) = VEL = VEf

ui(c)VEgi

i=l P

(3.4.4)

= VEf

-

T

U(E)

VEg

+

W(E)

T

VEh;

and h e n c e a l s o 2 *

= V2 XELVEX(E)

-

m

1

(3.4.5)

VECJiTVEUi(E)

i=l

+

VEh;VEwj(E)

+

VEL. 2

j=1

Proof.

R e c a l l t h a t i n a neighborhood o f

u ~ ( E ) ~ ~ [ x ( EE ]) , E

0 , i = 1,

slackness holds, hj[x(E), [x(E),

f

*

u(E),

w ( E ) ]E ~cl.

( € 1 :f [X(E), €1

E]

..., m,

E

=

0,

s t r i c t complementary

: 0, j = 1,

..., p ,

and Y ( E ) =

I t follows immediately t h a t

84

3 Fcn\iu\ ity

AndI)\i\

under Second-Order A\\umption\

yielding conclusion (a). Furthermore, we can differentiate (3.4.6) to obtain

*

VEf

(E)

=

*

€1

VEf[X(E),

= VEL

(E)

= V€L[X(E), U(E), W(E), E l =

(3.4.7)

+ VULVEU(€) + VWLVEW(€) + VEL,

VXLVEX(E)

where L is evaluated as in (3.4.6). Since the Karush-Kuhn-Tucker conditions hold at y ( ~ ) ,it follows that V L = 0. X

Complementary slackness implies ui(0) or

0, i = I,

..., m.

gi[x(0), 0

=

continuity

and differentiability then imply one of two con(i) gi[x(0), 01 > 0, implying

sequences, respectively: gi[x(E),

V €u. (E) 1

E]

> 0 for

Strict complementary slackness,

near 0, implying u ~ ( E ): 0, implying

E

0; or (ii) ui(0) > 0, implying u ~ ( E )> 0 for

0, implying gi[x(E), E]

5

0.

VULVEu(E) = (-gl[x(E), €1, since h.[x(E), E] 3

,..., hP[x(E),

E]

*

(3.4.7) that VEf

5

0 for

=

0.

Also,

near 0, VWLVEw(~)= (hl[x(E),

E])V~W(E)= 0. (E)

near

From this it follows that

..., -grn[x(E), EI)VEu(~)=

E

E

VEL for

E

We therefore conclude from near 0, proving conclusion

(b). Differentiation with respect to

E

of the result obtained

in (b) gives

v E2 f* ( E )

+ V&LVEU(€) = VXELVEX(E) 2

+ VWELVEW(E) 2 + VEL. 2

Calculation of the derivatives yields conclusion (c).

To be

perfectly clear about what is involved in calculating this

85


Hessian, we w r i t e (3.4.5)

i n t e r m s o f t h e o r i g i n a l problem

W e have

functions.

I

+

m

1

-

)V:f

1

1 P

+

ui(E)VZgi

i=l

\

'I

wj(E)V2h.(

j=l

m VEgiVEui(E) T

+

P

1

VEhTVEwj( € 1

(3.4.8)

1

j=1

The f o l l o w i n g two c o r o l l a r i e s a r e i m m e d i a t e , f o r two important r e a l i z a t i o n s of problem P 3 ( € ) . [ F i r s t - and s e c o n d - o r d e r

Coro 1 l a r y 3 . 4 . 2 ,

changes i n t h e

optimal value function of P 3 ( ~ ) w , i t h c o n s t r a i n t s independent of E.]

I f t h e c o n d i t i o n s o f Theorem 3 . 4 . 1

h o l d and t h e con-

s t r a i n t s o f P 3 ( ~ )a r e f u n c t i o n s o n l y o f x , t h e n i n a n e i g h b o r hood o f

(a)

E

= 0,

f

*

*

= L (E);

(E)

(b)

V E f * ( E ) = VEf;

(c)

2 * VEf ( € 1 =

and 2

VXftX(E),

€IBEX(€)

or

where VEx i s o b t a i n e d from ( 3 . 2 . 4 1 , N = [-VEXf

,

Corollary 3 . 4 . 3 .

El

with

[ F i r s t - and second-order changes i n t h e

unconstrained optimal value function.] C o r o l l a r y 3.2.3

2

+ VEf[X(E),

are s a t i s f i e d and f ( x ,

ously d i f f e r e n t i a b l e i n (x,

E)

I f t h e assumptions of E)

n e a r (x*,

is t w i c e continu-

o),

then i n a

86


neighborhood of

*

(a) VEf ( € 1

= 0,

E

=

VEf,

=

-VxEf 2 Vxf2 1 VEXf 2

+ VZfI [x(E) r

~

l

Another useful expression for VZf* is obtained by recalling that Vf(x(E) ,

E)

2 = 0 implies that Vxf VEx

+ V2xf

= 0, so that

Using this fact, it is easy to see that

v:f*

T

where

(3.4.9)

A formu a analogous to

3.4.9)

for the general problem,

with the Lagrangian L replacing f, will be obtained in Section 4.2 after manipulations of (3.2.3) that considerably simplify the analysis. Equation (3.4.4) immediately yields previously established results when certain problem structures are considered.

One

realization of (3.4.4) gives the well known "Lagrange multiplier sensitivity result" and then (3.4.5) also establishes relations for the Hessian of the optimal value function, with

87


respect to the rhs perturbations of the constraints.

For this

case, problem P 3 ( € 1 reduces to min

X

f (x)

~ . t .gi(x) 1. he 3

(XI

=

€j+,,,r

i = 1,

...

j = 1,

..., p.

r

P2 ( € 1

m,

Results for this problem will be developed in some detail in Chapter 5.

However, two important results are noted here

since they are immediate corollaries of Theorem 3.4.1. Coro Z Zary 3 . 4 . 4 .

[Optimal value function derivatives for

rhs perturbations.]

If

(i) the functions defining P 2 ( ~ )in a neighborhood of E

=

0 are twice continuously differentiable in x, in a neigh-

*

borhood of x

,

and

(ii) conditions (ii) in a neighborhood of

Proof.

i = 1,

E

E)

(iv) of Theorem 3.2.2 hold, then,

= 0,

We let f (x,

..., m, h3.(x,

-

=

E)

,

= f (x)

h. 3 (x)

-

gijx,

E)

=

j+m’ j = 1,

E.

gi (x)

-

E ~ ,

..., p, and apply

Because the functions defining

the results of Theorem 3.4.1.

P 2 ( ~ )are twice continuously differentiable in x, in a neigh-

*

borhood of x

,

the functions defining P 2 ( ~ )are twice continu-

ously differentiable in (x,

E)

in a neighborhood of (x*, 0 ) .

This, together with the satisfaction of conditions (ii)

-

(iv)

88


of Theorem 3.2.2,

satisfies the conditions of Theorem 3.4.1,

and hence the conclusions of Theorem 3.4.1

apply to P 2 ( E )

.

Let ek denote the m + p vector with a 1 in the k-th component and 0 ' s elsewhere. Theorem 3.4.1,

v E f*(E)

Then, recalling conclusion (b) of

we have that for

=

VEL = VEf

-

E

near 0,

m

2

uivEgi

i=l

and applying this to problem P 2 ( ~ )yields m

P

and hence VEf* ( E l T

for c near 0,

.= JE ); ( : [

as asserted in conclusion (a). Conclusion (b) results immediately from the differentiability of [u(E),

W(E)

I (conclusion (b) of Theorem 3.2.2) ,

differentiating the result of (a) with respect to

E.

A second, more general, corollary extends a result

established by Fiacco and McCormick (1968, Theorem 6 ) for the problem minx

f(x)

+ eoa0(x)

s.t. gi(x) + Eibi(x) 1. 0,

i = 1,

. . . I

m,

89


C o r o l l a r y 3.4.5.

[Optimal value derivatives for perturba-

tions that are linear in the parameters.]

If

(i) the functions of problem P L ; ( E ) are twice continu-

*

ously differentiable in x, in a neighborhood of (x (ii) conditions (ii)

,

O), and

- (iv) of Theorem 3.2.2 hold, then,

and

Proof.

+

We let f (x,

~ . b(x), . i = 1, 1 1

j = 1, ...,p,

E)

=

f (x)

..., m, h.(x, 7

E)

+ Eoao(x), gi (x, E ) = h. (x)

3

= g . (x)

+ ~~+~c~(x),

and apply conclusion (b) of Theorem 3.4.1,

having verified as in Corollary 3.4.4 that the conditions of Theorem 3.2.2

are satisfied.

In particular, with

*

denoting

90


evaluation at VEf*(0)

=

E

= 0, we have that

* *

(a*,, -Ulblr

* *

. * - I

* *

-Umbmr ~ 1 ~ - 1 . . r , W*C*)r P P

the result obtained by Fiacco and McCormick.

Differentiation

of (a) yields (b). Note (from Theorem 3.4.1) that the values of the optimal value function and its gradient can be calculated once the Karush-Kuhn-Tucker triple [x( E ) mined.

, u(E) ,

w ( E ) ] has been deter-

However, for the general problem, the value of the

Hessian matrix of the optimal value function requires the determination of both the triple and its first derivatives. Applications and examples of these results for the optimal value function and its derivatives are deferred to Section 4.3 (Chapter 4), where formulas are developed for the calculation of V Y(E) from (3.2.4).

These result from the various con-

ditions that can hold, under the assumptions of Theorem -1 3.2.2, leading to corresponding expressions for M(E) as a function of block components of this matrix.

The development

of these calculations follows from results originally presented in Armacost and Fiacco (1975).

However, the results

involving the connection between the optimal value Hessian and the second-order optimality conditions of the given parametric problem, viewing (x, for the first time.

E)

as the variable, are presented here

Chapter 4

Computational Aspects of Sensitivity Calculations: The General Problem

4.1.

INTRODUCTION In this chapter, we examine the numerical aspects of

calculating the parameter derivatives of the Karush-KuhnTucker triple for the general problem P(E), in particular seeking computational efficiencies.

Aside from their use in

calculating the optimal value function Hessian, they are of considerable importance in other applications, e.g., in characterizing the stability of the solution subject to perturbation and in providing a first-order estimate of KarushKuhn-Tucker triples of problems involving different values of the parameters, once one such triple has been determined. Formulas for the Karush-Kuhn-Tucker parameter-derivatives are given in Section 4 . 2 .

A small NLP example and a general

parametric LP example illustrate the results in Section 4.3. FORMULAS FOR THE PARAMETER FIRST DERIVATIVES OF A KARUSH-KUHN-TUCKER TRIPLE

4.2.

As noted in Section 3.2 (Chapter 3), when u(E),

W(E)

IT

Y(E)

=

is available, VEy(~)can be calculated.

[x(E), We

briefly recapitulate and then derive expressions for various

91

92

4 Computational Aspects of Sensitivity Calculations

cases.

R e c a l l t h a t t h e Karush-Kuhn-Tucker s a t i s f i e d by

n e c e s s a r y c o n d i t i o n s (3.2.2)

s.t. g(x,

min X f ( x , E)

2 0, h ( x ,

E)

first-order for

Y(E) E)

= 0

are ~ , L [ Y ( E ) , € 1 :V X f [ x ( E ) ,

m

1

-

€1

ui(E)VXgi[x(E),

€1

i=l

P

+

1

w.(E)V

j=1

3

0,

i = 1,

..., m,

hj[x(E), €1 = 0,

j = 1,

..., p ,

u ~ ( E ) ~ ~ [ x ( E E) ], =

along with

h . [ x ( E ) , € 1 = 0, X I

u ~ ( E )2

0 and g i [ x ( E ) , € 1 2 0 , i = 1,

(4.2.1)

..., m.

Before p r o c e e d i n g w i t h an a n a l y s i s o f ( 4 . 2 . 1 ) , that for

E

n e a r 0 , Theorem 3.2.2

assumed t h r o u g h o u t ) a s s u r e s t h a t

we note

(whose c o n d i t i o n s a r e b e i n g Y(E)

E

mentary s l a c k n e s s w i l l c o n t i n u e t o hold.

C1 and s t r i c t compleIn particular, t h i s

means t h a t nonbinding c o n s t r a i n t s remain nonbinding n e a r and

u ~ ( E )>

0 near

E

s o l u t i o n s o f (4.2.1)

= 0 i f u . (0) > 0. 1

near

E =

E

= 0

T h e r e f o r e , i n examining

0 , under t h e g i v e n a s s u m p t i o n s ,

w e can ( i )d e l e t e nonbinding c o n s t r a i n t i n f o r m a t i o n and ( i i ) d i v i d e o u t t h e nonzero these appear, i.e., gi = 0.

u ~ ( E ) from

t h o s e e q u a t i o n s i n which

w e c a n f o r s u c h i r e p l a c e uigi

= 0 by

I n t h e s u b s e q u e n t a n a l y s i s w e do i n f a c t d e a l w i t h

and e l i m i n a t e nonbinding c o n s t r a i n t i n f o r m a t i o n immediately. However, we choose n o t t o t a k e a d v a n t a g e o f b i n d i n g c o n s t r a i n t information, i.e.,

of t h e f a c t t h a t u i ( E ) g i [ x ( E ) ,

be r e p l a c e d by g i [ x ( E ) ,

E]

= 0 i f ui(0)

> 0.

E]

= 0 can

Our p u r p o s e i s

t o p r o v i d e comparisons v i a a p a r a l l e l t r e a t m e n t o f t h e c a l c u l a t i o n s with those t h a t t y p i c a l l y a r i s e i n "algorithmic"

93

4.2 Derivatives of a Karush-Kuhn-Tucker Triple

implementations, i.e., when a solution algorithm provides a solution

7

of a perturbation of (4.2.1) or when an estimate of

Y(E) solving (4.2.1) near

E

= 0 is given.

In such computa-

tions, the identification of the set of binding constraints at any given iteration may be at best tenuous and determined only in some limiting sense.

It is desirable to have each step of

the "nonalgorithmic" calculations available for comparison. An example of parallel algorithmic calculations of this information is given in terms of the penalty function computations in Chapter 6, Section 6.5. analysis of (4.2.1)

.

As in (3.2.3), let N(E)

-V hT, € 1

.IT

We return to a direct

=

[-v:xLT,

..., -u.v g., T 1

E

. . . I

l

and let

Since the system (4.2.1) is identically satisfied for 0, it can be differentiated with respect to

M(E)V~Y(E)=

"€1.

Under the conditions of Theorem 3.2.2,

E

E

near

to obtain (4.2.2)

M(E)

has an inverse;

thus vEy(E) = M(E)-~N(E); is an n

M(E)

n

+

m

+ p

x

(4.2.3)

+ m + p square matrix and VEy(€) and N(E) are

k matrices.

94


Clearly, any method of solving ( 4 . 2 . 2 ) ,

a system of linear

equations, is satisfactory and M ( E ) need not be inverted as in -1 (4.2.3). However, the work involved in calculating M ( E ) can be significantly reduced, as will become evident. throughout that the conditions of Theorem 3.2.2

Assume

hold, and

suppose, without loss of generality, that the first r inequality constraints are binding.

For simplicity, we now proLet

ceed to eliminate nonbinding constraint information.

m

be defined as follows:

Then, rearranging the rows and columns corresponding to the last m

-

r (nonbinding) inequality constraints, it follows

that

M=[:

Let Y ( E )

=

[x(E), Ti(€),

w(€)IT and G ( E )

..., -V hT3 ' . IT, where i = T [ul(€),..., ur ( € 1 1 , and S

T -uiVEgi,

U(E)

=

E

gi[x(0), 01 > 0, i = r

1,

=

1,.

.., m,

continuity imply that u.(E) = 0, i

*

..., r,

= (gl,

+

I-v:XLT, .., j = 1,

..., 4,) . T

..., p, Since

complementary slackness and =

r

+

1,

..., m

for

E

near 0.

Hence, the corresponding components of V Y(E) are zero, i.e.,

95


VEui = 0, i = r

+

1,

..., m.

It is thus assumed that non-

binding constraint information is suppressed in the Lagrangian or its derivatives with respect to x or It follows that ( 4 . 2 . 2 )

(Y(E), E).

E,

when evaluated at

may be reduced to solving

the system

-

M ( E ) V ~ ~ ( E=) E ( E ) .

-

Let, G since

5

E

diag(g.) and

(4.2.4) E

diag (ui), i = 1,

..., r.

Note that,

= 0,

0 0

0

Thus

(4.2.5)

-- 1

where U

=

diag(l/ui) for i = 1,

-

.

- ,r.

Thus far we have assumed no special structure on the problem.

However, in order to make further progress in calcu2 -1 , several special cases: (1) VxL lating B ( E ) - ~ consider

exists; (2) ViL = 0; ( 3 ) there are n linearly independent constraint gradients; and ( 4 ) r

+ p

< n.

Let

96

4 Computational Aspect\ 01 Scnvtivity Calculation\

an (r

+

p)

x

n matrix.

Let

k ( ~ ) -denote l

the left matrix on

3.

the right-hand side of (4.2.5); thus

P-["i"

(4.2.6)

Now suppose that (4.2.7)

I.

Our task is to determine the Aij for the various cases. eral general properties of the A.. are noted. 13

-1

Sev-

Clearly since

must be symmetric, All and A22 must be symmetric and T Aal = A12. Also, it is easy to show that All is positive semidefinite, under the given assumptions. This and zTAllz > 0 if Allz # 0 follow from the fact that All = AllV 2L All, M

PAll

=

0 and the second-order sufficiency condition.

It also

follows immediatly from our general Case 4 result below. We T 2 also note that A22 = -A12 V L A12 (Section 5.5), hence A22 is negative semidefinite if V 2L is positive semidefinite. Case 1.

V;L-'

Exists

It is easily shown that

- VXL 2 -1PT IPVXL 2 -1PTI-' , -

A12

=

A21

A22

=

-[PVxL 2 -1PTI-'

-'

Note that [PVxL P'1-l

. exists by our assumptions.

(4.2.8)

91


Case 2 .

2

V L = 0 X

T h e r e a r e two p o s s i b l e s i t u a t i o n s :

there are r

+

p < n or

r + p = n l i n e a r l y independent binding c o n s t r a i n t gradients. I f t h e r e a r e f e w e r t h a n n , a s s u m p t i o n ( i i )o f Theorem 3.2.2

f i ( ~ ) -dIo e s

v i o l a t e d and i t i s e a s i l y s e e n t h a t

is

not exist.

For e x a m p l e , t h i s c o r r e s p o n d s t o t h e s i t u a t i o n c h a r a c t e r i z i n g a d e g e n e r a t e s o l u t i o n i n l i n e a r programming.

y (

W e s h a l l n o t pursue t h i s

~ may ) n o t even b e continuous.

p o s s i b i l i t y f u r t h e r here.

I n t h i s case,

When t h e r e are n l i n e a r l y independ-

e n t b i n d i n g c o n s t r a i n t g r a d i e n t s w e have a s p e c i a l i n s t a n c e o f Case 3, which i s d e v e l o p e d n e x t . Case 3 .

There a r e n LinearZy Independent Binding C o n s t r a i n t Gradients

The J a c o b i a n of t h e n c o n s t r a i n t s w i t h r e s p e c t t o t h e n -1 v a r a i a b l e s i s nonvanishing, i.e., P e x i s t s . Hence, All

= 0,

A12

= A l:

= P

-1

Note t h a t i f V:L-’

,

(4.2.9)

exists,

(4.2.9)

i s c o m p u t a b l e from ( 4 . 2 . 8 ) ;

however, h e r e t h e e x i s t e n c e of V2L-l X

i s n o t assumed.

Also,

t h e r e m a i n i n g C a s e 2 p o s s i b i l i t y m e n t i o n e d above g i v e s All

= A22 = 0

a n d A12

= AT1 = P-’.

N o t e also t h a t t h e

s t a n d a r d l i n e a r programming problem n o n d e g e n e r a t e s o l u t i o n c a s e f a l l s i n t o t h i s l a t t e r category, w i t h n l i n e a r l y independent b i n d i n g c o n s t r a i n t g r a d i e n t s and V L: Case 4 .

r

+

= 0.

p < n

T h i s r e p r e s e n t s t h e more g e n e r a l s i t u a t i o n a n d i s t r e a t e d i n d e t a i l here. matrix.

I n (4.2.4),

E(E) i s a n n + r + p s q u a r e

By a s s u m p t i o n (iii)o f Theorem 3.2.2,

t h e Vxgi,

98


..., r and

...,

the V h. j = 1, p are linearly independx 3' By assumption (ii) of Theorem 3.2.2 the second-order

i = 1, ent.

sufficient conditions for a local minimum are satisfied, and by assumption (iv), strict complementary slackness holds. Thus, the Hessian of the Lagrangian is positive definite with respect to those nonzero vectors orthogonal to the binding constraint gradients, and since r + p vectors is nonempty. z

T VxLz 2 > 0,

n, the set of such

Hence

for all z # 0 such that m

'I'

(4.2.10)

PZ = [-VxqT, VxhT) z = 0

and at least one such z exists.

Under the assumption of

linear independence of the binding constraint gradients, P has rank r

+ p, and without

l o s s of generality, it can be assumed

that the submatrix involving the first r nonsingular. =

p columns of P is

Therefore, we partition P as follows:

[pa, pbl I

where Pa is an (r an (r

+

+ p)

x

+

p)

x

(r

+

p) nonsingular matrix and Pb is

q matrix, where q

=

n

- r

-

p.

This partition

and (4.2.10) allow us to express the first r z in terms of the remaining q components. where z E Er and zb E E q . a tion given in (4.2.10) yields +

and hence za = -P,lPbzb.

+

p components of T Let z = (za, zb) ,

Then the orthogonality condi-

99


Then

s a t i s f i e s (4.2.10). T

=

Letting

(pij1

-T

= pa

1..

we can f u r t h e r conclude t h a t

+:.[

-P;lPb

.;[-.;,TI

I

' 0.

(4.2.11)

Thus, t h e m a t r i x

Dxx

T 2 = T VxL T I

where T =

is p o s i t i v e d e f i n i t e and hence has an inverse. l e a d s us t o a r e p r e s e n t a t i o n of M(E) A

Recalling t h a t $ ( e ) - l right-hand s i d e of

(4.2.5)

-1

.

This f a c t

d e n o t e s t h e l e f t m a t r i x on t h e I

and g i v e n t h a t

we can w r i t e

A

M =

(4.2.12)

I00


Using the partitions defined above, the result is All = T Dii TT,

(4.2.13) -1 A22

=

-A.]21 :[L2Vx

We have now developed analytical expressions for

G ( E ) -1

for all of the specified cases, and now turn to the calculation of VET(€), using the block components Aij of

[

i-’.

Defining N = -VzxLT, VETT, -VEhTIT and recalling that N = [-VExL 2 T , -(EVEg) T, -VEh , it follows that A

0

-u--1

0

0

0 N,

I,

(4.2.14)

1

-A1lV,xL 2 E)

+

=

-A21VExL 2 +

-

A12[-::”,1

““3

I

-VEh

J

.

(4.2.15)

101


Two aspects of Eqs. (4.2.14) and (4.2.15) may be noted. First, the diagonal matrix involving the Lagrange multipliers (4.2.5) cancels out (as already anticipated) ,

in B(E)-'

resulting in a computational savings, since we need only concern ourselves with

in solving (4.2.15).

Second, the

portion of (4.2.15) corresponding to the partial derivatives of the Lagrange multipliers with respect to the parameters is given in conglomerate form, i.e., we have not obtained separate expressions for the quantities VEU(~)and V€W(E) at this level of generality, although V E X(E) may be calculated separately. Since we would generally be interested in estimating either changes in X(E) or changes in the Karush-Kuhn-Tucker triple [x(E), u(E), w(E)], this poses no problem.

Also, there is no

difficulty in applying (4.2.15) intact to calculate the 2 * * Hessian VEf (E) of the op.tima1 value function f ( € 1 ,

since it

follows easily from (3.4.5) that = -fiTV

V:f* where

fi

E

7

+

1^ 2 V2 L = -N M N + VEL E

is defined as in (4.2.14) and

VET is

(4.2.16) calculated as in

(4.2.15). Equation (4.2.14) holds for the general problem whenever (4.2.4) is well defined, in particular when the conditions of Theorem 3.2.2 hold.

For the (exhaustive) cases treated in

this section, V EY ( E ) may then be calculated from (4.2.15) by first evaluating the Aij as given in (4.2.81, (4.2.91, or (4.2.131, depending on which conditions apply.

102


Defining D

(x, E ) L

(VxL, V E L ) , we observe t h a t V 2 f *

=

(E)

may

be expressed d i r e c t l y i n t e r m s of t h e j o i n t H e s s i a n

of t h e L a q r a n q i a n ( 3 . 2 . 1 )

of problem P 3 ( € ) .

T h e r e s u l t may

be d e r i v e d f r o m t h e p r e v i o u s r e l a t i o n s , b u t w e s i m p l y s t a t e it

here a n d t h e n v e r i f y it.

v;f*

The f o r m u l a i s

:I.

v x =

[ V E XT I

I,,.&[2

(4.2.17)

To c h e c k i t s v a l i d i t y , we e x p a n d t h e e x p r e s s i o n and u s e t h e known f a c t s t o show t h a t t h e r e s u l t agrees w i t h ( 3 . 4 . 5 ) . Expanding (4.2.17)

v;f*

=

yields

V E X TVXLVEX 2

From ( 4 . 2 . 6 )

+

V;€LvEX

and (4.2.14)

+

vEXTv:XL

+

VEL. 2

(4.2.18)

we r e c a l l t h a t

which yields

(4.2.19)

4.2 Derivativeb of a Karush-Kuhn-Tucker Triple

103

Multiplying the first equation by VExT , using the second equation, and rearranging gives

Using this in (4.2.18) and rearranging gives V:f*

=

VXELVEx 2 - (VEgT, -VEhT]

which corresponds precisely with (nontrivial) components of (3.4.5). The formula (4.2.17) explicitly provides the precise relationship between the Hessian of the optimal value function and the joint (variable-parameter) Hessian of the Lagrangian of the general parametric program P 3 ( ~ ) ,under the given assumptions.

Some well known results are corroborated im-

mediately and some new enlightening relationships may be inferred from this expression. If VEx is defined and the Lagrangian joint Hessian is

defined and positive semidefinite along

[Y(E),

E]

on some

domain, then it follows immediately from (4.2.17) that the optimal value Hessian V 2 f"

(6)

of problem P3(€) is defined and

positive semidefinite over that domain.

This is consistent

with the well known fact (Mangasarian and Rosen (1964)) that

*

if P3 ( E ) is a jointly convex program, then f

(E)

is convex.

I04


Of c o u r s e ,

(4.2.17)

t e l l s u s more.

I t i s n e c e s s a r y and s u f f i T 2

c i e n t f o r V:f*

t o be p o s i t i v e s e m i d e f i n i t e , t h a t z V

f o r a l l z such t h a t z

T

= y

T

(VEX

T

,

11.

LZ 0 (x, E ) This extends t h e noted

r e s u l t , under t h e g i v e n a s s u m p t i o n s , and e x p l i c i t l y shows why t h e o p t i m a l v a l u e o f a l a r g e r ( t h a n j o i n t l y convex) c l a s s o f problems i s convex.

I t would be i n t e r e s t i n g t o d e v e l o p t h e

i m p l i c a t i o n s o f t h i s r e s u l t and, i n p a r t i c u l a r , t o o b t a i n analogs of (4.2.17)

under weaker a s s u m p t i o n s .

Rephrasing t h e l a s t o b s e r v a t i o n l e a d s t o a n o t h e r v e r y i n t r i g u i n g c h a r a c t e r i z a t i o n of what i m p l i e s t h e c o n v e x i t y of f * ( E ) .

Recall t h a t

T [ x ( E ) , €1

which w e deduce t h a t V x g E x Defining PE =

(-VETT,

+ VET =

0 and VxhVEx

+

VEh = 0 .

VEhTIT and r e c a l l i n g t h a t P =

V hTIT w e c o n c l u d e t h a t PVEx X

: 0 and h [ x ( E ) , € 1 = 0 , from

+

[-VxqT,

PE = 0 , o r e q u i v a l e n t l y t h a t

(4.2.20)

T h i s r e l a t i o n s h i p i s o f c o u r s e e q u i v a l e n t t o t h e second equat i o n of (4.2.19)

,

and w e have chosen t o r e d e r i v e i t t o r e c a l l

i t s o r i g i n f o r o u r p r e s e n t purpose. 0 (>O)

T 2

Suppose t h a t z V

2.

f o r e v e r y v e c t o r z(z # 0 ) such t h a t ( P , P E ) z = 0.

t h e n c o n c l u d e i n p a r t i c u l a r from (4.2.17) 2 * VEf i s p o s i t i v e s e m i d e f i n i t e ( d e f i n i t e ) .

and ( 4 . 2 . 2 0 )

We

that

(The d e f i n i t e p a r t

of t h e s t a t e m e n t f o l l o w s a s w e l l , n o t i n g t h a t no column of

(

VEXT,

11'

is the zero vector.)

But r e c a l l i n g t h e second-order

s u f f i c i e n t c o n d i t i o n s (Lemma 3.2.1)

f o r a s t r i c t l o c a l minimum,

and n o t i n g t h a t t h e second-order n e c e s s a r y c o n d i t i o n s ( w i t h a p p r o p r i a t e r e g u l a r i t y ) a r e t h e same w i t h t h e s t r i c t i n e q u a l i t y

sTV 2L s > 0 r e p l a c e d by i n e q u a l i t y , w e o b s e r v e t h e f o l l o w i n g

4.2 Derivatives of a Karush-Kuhn-Tucker

fact:

For given

E

Triple

105

such that the assumptions of Theorem 3 . 2 . 2

hold, the Hessian V:f*

is positive definite (semidefinite) if

the second-order part of the second-order sufficient (necessary) conditions for a local minimum of P: min (XrE) f (xr E ) s.t. g(x, E ) 2 0, h(x, E ) = 0 hold, relative to (x, E ) viewed jointly as the variables, at (x,

E)

= [x(E),

€1.

This

establishes a direct connection between the convexity of the

*

optimal value f

(E)

of P 3 ( ~ )and second-order sufficient

(necessary) conditions for P 3 ( € ) , where (x,

E)

is viewed as a

variable. This relationship should lead to additional insights regarding the optimal value Hessian. is suggested. tion over (x,

An immediate application

Suppose the given problem P requires minimizaE)

jointly, and we attempt to solve the problem

by a decomposition scheme based on fixing a minimum xo to update

E

=

=

E =

solving for

x ( E ~ )over x, using the accumulated information E ~ etc. ,

(An example application of this type This might be viewed as a scheme

is given in Section 5.6.)

for solving P based on solving a sequence of parametric pro, = 0 , 1, grams ( P ( E ~ ) ) k

(51,

... .

Suppose the solution of P is

E) and that the given scheme successfully generates a - - ( X r ur Wr E ) with

minimizing sequence (xk r

ukr wkr E ~ ) (uk, wk) the associated optimal Lagrange multipliers. +

suppose the assumptions of Theorem 3 . 2 . 2

Finally,

hold for problem P ( E )

for large k, and the second-order conditions for a strict local minimum of problem P hold at (F, Ti, W, F ) . 2 * large k , the Hessian VEf

( E ~ )of

Then, for

the optimal value of P ( E ~ )

(with nonbinding constraint information deleted) will be positive definite, a fact that can be exploited; e.g., justifying applications of Newton's method to update E ~ and ,

106


i n t r o d u c i n g t h e p o s s i b i l i t y of c a l c u l a t i n g bounds on t h e o p t i ma1 v a l u e f

*

of P ( E ) near

(E)

Chapter 9 1 .

T ( e . g . , see S e c t i o n 2.5 and

=

E

Many more a p p l i c a t i o n s o f t h i s c h a r a c t e r i z a t i o n

a r e anticipated. 4.3.

APPLICATIONS AND EXAMPLES An i n d i c a t e d a p p l i c a t i o n o f t h e s e n s i t i v i t y a n a l y s i s

r e s u l t s o b t a i n e d h e r e i s a c a l c u l a t i o n o f f i r s t - and secondorder estimates of t h e optimal value function, using a T a y l o r ' s series e x p a n s i o n . P

3

(E)

If

X(E)

i s a s o l u t i o n of problem

s a t i s f y i n g t h e c o n d i t i o n s of Theorem 3 . 2 . 2

(Chapter 3 ) ,

t h e n a f i r s t - o r d e r e s t i m a t e of t h e o p t i m a l v a l u e f u n c t i o n , f [ x ( ~ ) €, 1 ,

f* ( E)

f*(E)

+

f*(O)

f*(O)

*

where VEf

o f VEf

VEf*(0)E,

+

1 VEf*(0)E+ 3

E

T 2 * VEf ( O ) E ,

(4.3.2) and (3.4.51,

A s an i l l u s t r a t i v e example of t h e c a l c u l a t i o n

2 * and VEf ( E ) ,

(E)

(4.3.1)

2 * and VEf ( E ) a r e d e f i n e d by ( 3 . 4 . 4 )

(E)

respectively.

*

i n a neighborhood of 0 , i s g i v e n by

E

e s t i m a t e i s g i v e n by

and a second-order f*(E)

for

c o n s i d e r t h e f o l l o w i n g convex ( i n x )

problem, which was d i s c u s s e d i n Armacost and F i a c c o ( 1 9 7 4 ) .

Example 4. 3. 1 min

X

f(x,

s.t. gl(x, for

-

E~

> 0.

ulgl(x,

E)

L ( x , u,

E)

= x1

+

~~x~

E)

= El

2

-

2 x1

-

2 x2

L

0

The Lagrangian i s d e f i n e d a s L ( x , u, and i s g i v e n h e r e by E)

E

x1

+

E2X2

-

ul(

E;

-

2 x1

-

2 x2).

E)

= f(x,

E)

107

4.3 Applications and Examples

By application of the Karush-Kuhn-Tucker conditions the solution may be found algebraically to be -&

(1

+

1

€;y2

-€lE2

(1

= U1(E) =

U(E)

+

f*(d = -El(l

(1

+

1/2

€3

+

(4.3.3)

2E1 E2 2

The surface depicting the optimal value function is difficult to portray graphically in three dimensions.

However,

isovalue contours of the function in parameter space are shown in Fig. 4.3.1.

Since we have the optimal value function

analytically, we can determine its gradient and Hessian directly, giving

vEf*

(€1 =

(af*

af* (€)lac2) (4.3.4)

and -2

0 2 *

VEf

(E)

(1

=

-‘2 (1

+

€;y2

-I

+ -El

(4.3.5)

108

3 Computational Aspects of Senbitivity Calculations

=2

2

1

0

-1

-2

Fig.

4.3.1.

I s o v a l u e c o n t o u r s of

f*fE)

=

-E~(I

+

1/2 E:)

To i l l u s t r a t e t h e a p p l i c a t i o n o f t h e g e n e r a l e q u a t i o n s

d e r i v e d i n Theorem 3.4.1,

w e s h a l l u s e them t o d e r i v e t h e

g r a d i e n t and t h e H e s s i a n of t h e o p t i m a l v a l u e f u n c t i o n f o r

and hence

which i s p r e c i s e l y (4.3.4)

a t t h e s o l u t i o n p o i n t (4.3.3).

.

109


From (3.4.5), VEf 2 * (E)

t h e H e s s i a n of f

+

= V2 XELVEX(E)

evaluated a t [ x ( E ) , u ( E ) ] , W e obtain

v 2f * ( € 1

[ o0

=

E

,] 0

VEX(€)

VEL 2

-

*

(E)

i s g i v e n by

vEgTvEU1(E)

where,

from a b o v e , V L = (-2u1c1, E

;] ";.I -

+ [-2u;(E)

and s i n c e from t h e above s o l u t i o n for [ x ( E ) , u

late that

v EX ( E )

= 1

L

(1 +

E

y

+ €$I2

(1

w e have t h a t 0 2 * V E f (E) =

-E,

0 -E-

..

.

.;y2 2

'1

+

-1 (1 +

E$/

2

x2).

VEU1(E)

(E)]

1

we calcu-

4 Computational Abpects of Sensitivity Calculations

I10 which c o r r e s p o n d s w i t h (4.3.5), differentiating the solution f

*

t h e r e s u l t o b t a i n e d by t w i c e (E)

obtained analytically.

Of c o u r s e , w e c a n n o t g e n e r a l l y e v a l u a t e t h e r e q u i r e d V X(E) E

and V

E

U(E)

a n a l y t i c a l l y a n d must t u r n t o some n u m e r i c a l

technique f o r s o l v i n g t h e system (4.2.3 ) ,

last section, f o r these quantities.

t h e s u b j e c t of t h e

The example s e r v e s f u r -

t h e r t o i l l u s t r a t e t h e usefulness of t h e r e s u l t s obtained there.

2:j

F o r example, w e f i n d t h a t

v 2L X

= ";I

= 2UlI,

and s i n c e u ~ ( E ) > 0 f o r a l l

E

t h e r e s u l t s given i n (4.2.8)

u n d e r c o n s i d e r a t i o n , C a s e 1 and apply.

i n v e r s e i s r e q u i r e d i s g i v e n by ( 4 . 2 . 6 )

h=[

,"

V2L

o]l 2 x 2 ) f o r t h e example.

W e wish t o d e t e r -

such t h a t

and t h e n c a l c u l a t e t h e V € X ( E ) S i n c e V$,-'

as

PT

where P = Vxg = (2x1, mine t h e A i j

h

The m a t r i x M ( E ) whose

and V € U ( E )

from ( 4 . 2 . 1 5 ) .

= ( 1 / 2 u 1 ( ~ ) ) 1 ,w e f i n d from ( 4 . 2 . 8 )

that


A12

= AT

1

21

A22 -

1

=-I 2ul2 ( x f

-2 ( x f

+

1

+

x;

x;).

We now proceed to evaluate ( 4 . 2 . 1 5 ) .

vXL

= (1

+

2U1X1.

+

E2

Since

2u1x2)

it f o l l o w s that 2 EX

0 =

[o

0 1 1 1

and since g l ( x r

E)

=

2

E~

-

2

x1

-

2 x2 ' we have that VEgl = ( 2 ~ ~ ~ 0 ) .

We now have all of the information necessary to calculate V€Y(E) from ( 4 . 2 . 1 5 ) ,

and obtain

r =I

1

-1 2(x;

+ x;)

0 x2)[o

0 11

+

2(xf

-1

+

XC)

(2c1r

0)

2 Since x1

+ x22

=

E

2

~

,

x1x2

7 2 9 1

2 -xl 2 2UlEl

VEY(E) =

-x2

7 2E1

Therefore, we may conclude that

VEX(€) =

12

J

,

-x:

VEU(E)

=

(-2 ->). 2E1

7 2UlE1

It may readily be verified (with the understanding that u ~ ( E ) )that this corresponds exactly with the result obtained above by explicit differentiation. All

x1

=

x l ( ~ )and u1

=

the required calculations can be made for any admissible value of E, once

X(E)

and

U(E)

are known.

As another example, consider a parametric linear programn ming problem in E .

I13


Example 4 . 3 . 2 minx

f(x,

E)

= C

E)

= gi(x)

s.t. g i ( x ,

~

+~

E

-

*X. * ~

+

c

E x n n n

1. 0 ,

E~

i = 1,

..., m,

LP

(E)

where m 2 n , t h e ci are c o n s t a n t s , t h e gi are l i n e a r , a n d t h e c o n d i t i o n s o f Theorem 3.2.2

hold.

W e now c a l c u l a t e V € X ( E )

and

using t h e formulas developed i n t h e l a s t s e c t i o n .

V€U(E),

The L a g r a n g i a n f o r problem L P ( E ) i s L(x, u,

=

E)

c

T

(E)X

-

m

1

uitgi(x)

-

Eil,

i=l

where c

T

(E)

...,

= ( C ~ E ~ ,

2

c ~ E ~ ) S . i n c e VxL =

0, C a s e 2 a p p l i e s .

The s a t i s f a c t i o n o f t h e c o n d i t i o n s of Theorem 3.2.2

requires

t h a t t h e r e be n l i n e a r l y i n d e p e n d e n t b i n d i n g c o n s t r a i n t g r a d i -

A(€)-'

e n t s , a n d h e n c e w e o b t a i n t h e components o f

from

(4.2.9).

L e t u s assume t h a t t h e f i r s t n c o n s t r a i n t s a r e

binding.

T h i s i s o n e o f s e v e r a l p o s s i b l e cases.

V a r i a n t s of

t h i s s i t u a t i o n i n v o l v i n g o t h e r sets o f b i n d i n g c o n s t r a i n t s c a n a l s o b e h a n d l e d i n a s t r a i g h t f o r w a r d manner, though t h e c a l c u -

l a t i o n s a r e somewhat more t e d i o u s . Consistent with our p r i o r notation, g ( x ,

€1,

..., g n ( x ,

E)I

Equation (4.2.9) A 2 2 = -P-TViLP-l,

fi-l

= [p;T

T

uiVxgi,

f o r i = 1,

a n d P = [-V XQ1 = 1-V x g 1'

g i v e s All

"1.

...,

-Vxg:lT. -1 = P , and

T = A21

= 0 , A12

= [g,(x,

E)

and hence

To u s e ( 4 . 2 . 1 5 )

-

T

,

w e r e q u i r e V 2 L and EX

VET.

S i n c e V X L = cT

w e f i n d t h a t VZxL = ( C , 01, where C = d i a g ( c i )

..., n.

Since g . (x,

E)

=

gi(x)

-

E

~

,i t

follows

I14 that

4 Computarional Ahpecth of Senbitivity Calculations

VET =

W e c a n now c a l c u l a t e

(-1, 0).

= [p-?

-P-TC ,

...,

E)

and o b t a i n

11

where G ( E ) = [ u ~ ( E, ) gi(xr

(4.2.15)

+

1. 0 for i = n

u ~ ( E ])

T

.

Since the constraints

l , . . . , m are n o t b i n d i n g , i t f o l l o w s

t h a t u1 . ( E ) = 0 and V E u . l ( E ) = 0 f o r i = n

+

1,

..., m.

This

, V€U(E) I . c o n s t i t u t e s t h e c o m p l e t e s o l u t i o n for [V€X(E) W e can a l s o e a s i l y c a l c u l a t e t h e g r a d i e n t and t h e Hessian

of t h e o p t i m a l v a l u e f u n c t i o n s , using (3.4.4) obtaining

v

f*(E)

T

= VEL =

+

(x (E)C, 0)

= (XT(E)C, 0) = (XT(E)C

+

+

(iiT(E)

TiT(€),

T

u (€1

,

0)

0)

and

y

-1

=

=[

-cP-l

,

0)

0

-

P-TC, 0

1

(P-TC, 0)

"1.

and ( 3 . 4 . 5 ) ,

115


This example illustrates how we can easily calculate sensitivity results for many classes of parametric problems, using the general relationships developed. The next chapter continues the analysis that has been conducted here for the general parametric nonlinear programming problem P 3 ( E ) , pursuing ramifications for important realizations of P3(€), where the parameters appear only in the righthand sides of the constraints. on Armacost and Fiacco (1976).

This material is based largely

Chapter 5

Computational Aspects: RHS Perturbations

5.1.

INTRODUCTION It is clear from the previous results and is also a long-

established fact that optimal Lagrange multipliers are associated with a measure of the sensitivity of the optimal value of the objective function with respect to rhs changes in the constraints of a mathematical programming problem.

Corol-

lary 3.4.4 of Chapter 3 gives a precise result under the assumptions of Theorem 3.2.2 of that chapter.

Recent work in

sensitivity and stability analysis for nonlinear programming, as indicated in Chapter 2 , provides incisive characterizations under very general assumptions.

Given the assumptions of

Theorem 3.2.2, Theorem 3.4.1 gives closed form expressions for the gradient VEf* and Hessian VZf* of the optimal value of P3W.

As a matter of historical interest, this chapter first sketches the development of the interpretation of Lagrange multipliers as sensitivity estimates in optimization problems subject to rhs perturbations.

Following this brief chrono-

logical perspective, additional results are developed that are based on the general treatment presented in the last two

I I6

I17

5.2 Use and Initial Interpretation of Lagrange Multipliers

chapters.

The reader not wishing to make this historical

digression may proceed directly to Section 5.4, where the development of technical results is resumed. The brief recapitulation of developments in the use and interpretations of Lagrange multipliers in linear and nonlinear programming problems is given in Section 5.2.

In Section 5.3,

several developments of the analytical interpretation of Lagrange multipliers as sensitivity estimates not covered previously are quickly reviewed.

In Section 5.4, the given

theoretical results that hold for general parametric nonlinear programming problems are specialized and easily yield the Lagrange multiplier sensitivity result and the second-order change in the optimal value function for problem P2(~). Formulas for calculating the first derivatives of a Karush-KuhnTucker triple and the Hessian of the optimal value function are developed in Section 5 . 5 .

In Section 5.6, several appli-

cations of first- and second-order sensitivity information of

*

the optimal value function f

(E)

of problem P 2 ( ~ )are indi-

* (E)

cated, including the second-order estimation of f values of

E

for

near the starting value, and the use of sensitivity

information in developing a cyclical (decomposition) scheme for solving a "standard" NLP problem. 5.2.

THE USE AND INITIAL INTERPRETATION OF LAGRANGE MULTIPLIERS

A classical problem that has been examined by numerous mathematicians is the determination of the maxima or minima of a function f(x) of several variables required to satisfy one or more equations, h.(x) = 0, j = l,..., 7

p.

One approach to

solving this problem was proposed by Lagrange (1881) and

I18

5 Computational Aspects: RHS Perturbations

involved a function he introduced, now known as the Lagrangian. For the indicated problem, the Lagrangian is defined as L(x, w) = f(x)

+

P

2

wjhj(x).

(5.2.1)

j=1

The constants w

j

were often called undetermined multipliers,

and are now generally known as Lagrange multipliers.

The

"Lagrange Multiplier Rule" consists of solving the system of equations VxL(x, w) = 0, h .( x ) = 0, j = 1, 7

..., p, for x and w.

One of the resulting solutions will sometimes yield a solution of the given problem.

Lagrange did not provide a detailed

explanation of when or why. Incredibly, not until an extensive work published by Hancock (1917), do we find a rigorous unified treatment of necessary and sufficient conditions, based on Lagrange's approach, that characterizes relative extrema subject to subsidiary equalities.

Caratheodory (1935) likewise established

necessary and sufficient conditions involving Lagrange multipliers for a "minimum in the small," conditions less cumbersome in their development than Hancock's.

Both authors

established the important second-order sufficient condition for an isolated local minimum, i.e., that at a local solution the Hessian of the Lagrangian be positive definite with respect to all nonzero vectors orthogonal to the gradients of the constraints. Hancock also developed a test involving determinants to determine if the sufficient condition is satisfied.

Mann

(1943) developed a procedure that was simpler than that given by Hancock to test the satisfaction of the second-order sufficient condition, again based on evaluating certain determinants.

I19


Curiously, many current texts in analysis and advanced calculus still replicate the classical approach and examine only equality side conditions in extremal problems. order conditions are often not developed.

Second-

In addition, most

of these classical mathematical treatments do not appear to offer an i n t e r p r e t a t i o n of the Lagrange multipliers.

Even

most books in nonlinear programming have not until very recently developed or applied the second-order conditions. A primary use and subsequent development of the "sensitivity" interpretation of Lagrange multipliers and associated terminology appears to derive from an analysis of economic models.

Allen's (1938) text for economists used Lagrange

multipliers to solve various models, though it did not offer a specific interpretation of them.

Hicks (1939), while not di-

rectly employing Lagrange multipliers in his mathematical treatment, makes frequent reference to the corresponding concepts, the "marginal utility of money" (attributed to Marshall (1920)) and "shadow prices." Samuelson (1947) gave several interpretations of Lagrange multipliers in an economic setting and developed rigorous approaches to different economic structures using analytical methods, including an analysis of mathematical models involving extrema subject to equality constraints (with all functions twice continuously differentiable).

The solutions were ob-

tained by Lagrange's method of undetermined multipliers.

For

a cost minimization problem in the theory of the firm, the Lagrange multiplier is clearly identified as a marginal cost, in the theory of consumer behavior as the marginal utility of money, and in welfare economics as a price.

In a section

dealing with transformations of independent variables Samuelson

I20


showed a n a l y t i c a l l y t h a t t h e p a r t i a l d e r i v a t i v e o f t h e o p t i m a l v a l u e f u n c t i o n w i t h r e s p e c t t o t h e r h s p a r a m e t e r o f an e q u a l i t y c o n s t r a i n t i s t h e Lagrange m u l t i p l i e r a s s o c i a t e d w i t h t h a t constraint. Following t h i s i n i t i a l r i g o r o u s i n t e r p r e t a t i o n o f Lagrange m u l t i p l i e r s , s u b s e q u e n t developments f o l l o w e d a l o n g t h e l i n e s of t h e newly emerging t h e o r y o f l i n e a r and n o n l i n e a r programming.

Fundamental r e s u l t s i n l i n e a r programming i n 1947-1948

were p r e s e n t e d t o s e v e r a l m e e t i n g s o f t h e Econometric S o c i e t y i n 1948 and a p p e a r e d i n s e v e r a l p a p e r s p u b l i s h e d under t h e e d i t o r s h i p o f Koopmans ( 1 9 5 1 a ) .

Possibly t h e f i r s t proof o f a

d u a l i t y theorem f o r l i n e a r programming by Gale, Kuhn, and Tucker ( 1 9 5 1 ) , i s c o n t a i n e d t h e r e i n .

A r e l a t i o n s h i p between

d u a l v a r i a b l e s and Lagrange m u l t i p l i e r s i s c u r i o u s l y n o t n o t e d . I n t h e same volume, Koopmans (1951b) f o r m u l a t e d a l i n e a r p r o d u c t i o n model i n t h e t e r m i n o l o g y o f " a c t i v i t y a n a l y s i s .

"

I n t h i s c o n t e x t , t h e v e c t o r s normal t o t h e " a c h i e v a b l e cone" a t an " e f f i c i e n t p o i n t " a r e i d e n t i f i e d a s "shadow p r i c e s . " The e x i s t e n c e o f p r i c e s i s shown t o be n e c e s s a r y and s u f f i c i e n t f o r t h e a c t i v i t y v e c t o r t o l e a d t o an e f f i c i e n t p o i n t .

The

p r i c e v e c t o r i s shown t o d e f i n e t h e " m a r g i n a l r a t e o f subs t i t u t i o n " and i s u s a b l e under a regime o f d e c e n t r a l i z e d decisions. Goldman and Tucker ( 1 9 5 6 ) , i n a volume e d i t e d by Kuhn and Tucker ( 1 9 5 6 ) , p r e s e n t e d a t h e o r e t i c a l development o f l i n e a r programming, and began by i n t r o d u c i n g d u a l v a r i a b l e s a s p r i c e s i n t h e c o n t e x t o f an economic problem.

I n t h a t paper they

developed t h e e x p l i c i t r e l a t i o n s h i p between d u a l i t y and Lagrange m u l t i p l i e r s f o r l i n e a r programs.

Another p a p e r i n

t h a t volume by M i l l s (1956) examined t h e m a r g i n a l v a l u e o f

121


matrix games (and simultaneously the marginal value of linear programs due to the equivalence between matrix games and linear programs) when a particular kind of perturbation was allowed.

It was shown that with this perturbation Lagrange

multipliers correspond to dual variables.

Under certain

specified conditions the marginal value with respect to a constraint is the corresponding dual variable and is also the "generalized" Lagrange multiplier.

It was also pointed out

that this interpretation was easily extended to nonlinear programming.

In their preface, Kuhn and Tucker (1956) identified

this examination of marginal values as a sensitivity analysis. Charnes and Cooper

1961) indicated that a sensitivity inter-

pretation should be extended to "dual evaluators,'' their term for dual variables. [e.g., Dantzig (196

In subsequent works on linear programming ),

Gale (1960)], liberal economic inter-

pretations are made and the relationship to Lagrange multipliers is often explicit.

Dantzig makes extensive use of the

term "sensitivity analysis" when describing what is also called "post-optimality analysis" for linear programs. As noted above, the development of Lagrange's method was

essentially limited to the problem where the constraints are equations.

Necessary conditions that allowed inequality con-

straints were obtained by Karush (1939) using the calculus of variations, though this result in his unpublished master's thesis became widely known only several years ago.

Fritz John

(1948) established general necessary conditions for the solution of an optimization problem involving i n e q u a l i t y side conditions, using a Lagrangian approach.

However, John's re-

sult allowed for the nonexistence of finite multipliers.

At a

1950 symposium Kuhn and Tucker (1951) presented their paper on

122


nonlinear programming that extended the Lagrange Multiplier Rule to guarantee finite multipliers for the inequality constrained problem, and gave convexity conditions such that a mathematical programming problem would be equivalent to the problem of finding a saddle point of the Lagrangian.

This

extension involved a now well-known "constraint qualification," requiring once differentiability of the problem functions. Slater (1950) obtained analogous results for convex programs without requiring differentiability, his constraint qualification being an interiority assumption characterizing the feasible region.

Arrow and Hurwicz (1956) extended the Kuhn-

Tucker result to nonlinear programs that did not satisfy the convexity property. Subsequently, considerable effort was made to develop and relate various types of constraint qualifications.

For

example, Uzawa (1958) proved the Kuhn-Tucker theorem under different qualifications on the constraints.

Mangasarian and

Fromovitz (1967) developed another necessary optimality condition involving a constraint qualification that accommodates both inequality and equality constraints.

Additional impor-

tant constraint qualifications were developed by Abadie, Cottle, Gould, and, most recently, by Rockafellar.

The inter-

ested reader may begin by consulting Section 2 . 3 and one of the sources given in the next paragraph. Hestenes (1966) and Fiacco and McCormick (1968) presented a comprehensive examination of necessary and sufficient optimality conditions and associated constraint qualifications. In addition, the books by Canon, Cullum, and Polak (1970) and Mangasarian (1969) give detailed treatments of optimality conditions and constraint qualifications and, along with

123

5.3 Early Sensitivity Interpretations of Lagrange Multipliers

Bazaraa e t aZ.

( 1 9 7 2 ) a n d o t h e r s , h a v e reexamined a n d f u r t h e r

d e v e l o p e d t h e i n t e r r e l a t i o n s h i p s among v a r i o u s c o n s t r a i n t qualifications. 5.3.

EXAMPLES O F EARLY SENSITIVITY INTERPRETATIONS OF LAGRANGE MULTIPLIERS A s w i t h t h e g e n e r a l d e v e l o p m e n t a n d u e o f Lagra ge m u l t i -

p l i e r s , a n a l y t i c a l i n t e r p r e t a t i o n s o f t h e Lagrange m u l t i p l i e r s

w e r e i n i t i a l l y made f o r p r o b l e m s w i t h o n l y e q u a l i t y constraints.

One o f t h e f i r s t a f t e r Samuelson ( 1 9 4 7 ) a p p e a r s t o

be d u e t o P h i p p s ( 1 9 5 2 ) , who c o n s i d e r e d a problem e q u i v a l e n t

t o P 2 ( € ) w i t h m = 0 and p

The f u n c t i o n s a r e assumed t o

n.

be twice c o n t i n u o u s l y d i f f e r e n t i a b l e , and t h e m a t r i x o f f i r s t p a r t i a l d e r i v a t i v e s of t h e h have r a n k p a t a s o l u t i o n .

w i t h r e s p e c t t o x i s assumed t o j Phipps developed t h e u s u al f i r s t -

o r d e r necessary conditions involving t h e s t a t i o n a r i t y of t h e Lagrangian, i . e . , V,L(X,

=

0

h.(x) =

E

W,

E)

3

~

,

j = 1,

+

a set of n + p equations i n n

wj,

j = 1,

L(x, w,

E)

..., p )

= f (x)

p unknowns ( x i ,

and p parameters

(E

-

E.

+ ZPZl w j [ h . ( x ) 3

(5.3.1)

..., p ,

3

j r

I.

j = 1,

i = 1,

..., n;

..., p ) ,

where

H e used t h e suf-

f i c i e n t c o n d i t i o n of Hancock (1917) w i t h t h e t e s t d e v e l o p e d by Mann ( 1 9 4 3 ) .

Along w i t h t h e assumed l i n e a r i n d e p e n d e n c e o f t h e

g r a d i e n t s o f t h e c o n s t r a i n t s , t h i s a s s u r e s t h a t t h e Jacobian of (5.3.1) (5.3.1).

w i t h r e s p e c t t o (x, w) i s n o n s i n g u l a r a t a s o l u t i o n o f Phipps concluded t h a t t h e c o o r d i n a t e s of each sta-

t i o n a r y p o i n t and t h e a s s o c i a t e d o p t i m a l m u l t i p l i e r s , as w e l l

a s t h e " s t a t i o n a r y v a l u e " o f f , are a l l w e l l d e f i n e d f u n c t i o n s

I24


of t h e p a r a m e t e r s .

Note t h a t u n d e r t h e assumed c o n d i t i o n s

t h i s r e s u l t follows r e a d i l y s i n c e t h e i m p l i c i t f u n c t i o n t h e o r e m g u a r a n t e e s t h e e x i s t e n c e of u n i q u e , o n c e c o n t i n u o u s l y d i f f e r e n t i a b l e f u n c t i o n s x i ( € ) , ~ ~ ( € 1i ,= 1,

..., n ,

j = 1,

..., p.

P h i p p s o f f e r e d a n i n t e r p r e t a t i o n of t h e Lagrange m u l t i p l i e r s a n d e x h i b i t e d t h e now w e l l known Lagrange m u l t i p l i e r s e n s i t i v i t y r e s u l t , i.e.,

af

*

/a€

*

= -wjr

j = 1,

..., p.

Wilde (1962) u s e d d i f f e r e n t i a l c a l c u l u s t o d e r i v e t h i s r e s u l t , a l o n g w i t h c o n d i t i o n s f o r a l o c a l minimum o f a g e n e r a l c l a s s of p r o b l e m s i n v o l v i n g i n e q u a l i t y c o n s t r a i n t s a n d o n c e continuously d i f f e r e n t i a b l e functions.

"Slack" v a r i a b l e s a r e

i n t r o d u c e d t o c o n v e r t t h e i n e q u a l i t i e s t o e q u a l i t i e s a n d it i s assumed t h a t t h e r e i s a n o n v a n i s h i n g J a c o b i a n o f t h e r e s u l t i n g s y s t e m w i t h r e s p e c t t o s u b s e t s o f t h e s l a c k and " d e c i s i o n " variables.

Using t h i s s t r u c t u r e a n d t h e i m p l i c i t f u n c t i o n

theorem t o i d e n t i f y f u n c t i o n a l d e p e n d e n c e between s u b s e t s o f variables, the optimal value function i s d i f f e r e n t i a t e d with r e s p e c t t o t h e "independent" v a r i a b l e s and t h e r e s u l t i n g part i a l d e r i v a t i v e s are t e r m e d " s e n s i t i v i t i e s . "

e q u i v a l e n t t o t h e Karush-Kuhn-Tucker

Conditions

necessary c o n d i t i o n s are

developed and, i n p a r t i c u l a r , a s u b s e t of t h e s e n s i t i v i t i e s i s shown t o c o r r e s p o n d t o o p t i m a l L a g r a n g e m u l t i p l i e r s .

In a

s u b s e q u e n t p a p e r ( 1 9 6 5 ) Wilde n o t e s t h a t a l t h o u g h t h i s p r o c e d u r e i s s t r a i g h f o r w a r d f o r l i n e a r c o n s t r a i n t s and s i m p l e obj e c t i v e f u n c t i o n s , i t becomes i m p r a c t i c a l i n more r e a l i s t i c situations. E v e r e t t ( 1 9 6 3 ) p r e s e n t e d a somewhat h e u r i s t i c method o f Lagrange m u l t i p l i e r s t o s o l v e c e r t a i n c l a s s e s ( p a r t i c u l a r l y t h o s e i n v o l v i n g c e r t a i n t y p e s o f s e p a r a b l e f u n c t i o n s ) o f nonl i n e a r programming p r o b l e m s a n d showed t h a t t h e L a g r a n g e

I25

5.4 Supporting Theory

multipliers can sometimes be used to bound changes in the optimal value.

The optimal multipliers are identified as the

derivatives of the optimum "payoff" with respect to each resource. Boot (1963) presented a thorough examination of sensitivity analysis for convex quadratic programming and analytically computed the many partial derivatives involved.

He developed

formulas that can be used to show that the partial derivative of the optimal value function, with respect to the components of the right-hand side (parameters) of the constraints, are optimal Lagrange multipliers. Contemporary developments are reported in Chapter 2 and are further documented in the works cited and in the bibliography. 5.4.

SUPPORTING THEORY

As noted in Section 3.4 (Chapter 3), the results for the general problem immediately yield the well known Lagrange multiplier sensitivity result and the not so well known second-order changes in the optimal value function for the problem P2(&).

Since this problem is the main concern of this

chapter, we restate the resulting specialization of Theorem 3.2.2 and Theorem 3.4.1 in full.

Conclusions (a), (b) and (d)

follow immediately from Theorem 3.2.2, conclusion (c) from Theorem 3.4.1, and conclusions (e) and (f) from Corollary 3.4.4.

Recall that the problem is

min X f(x)

s.t. gi(x) 1. E ~ , i = 1,

..., m,

I26


with associated Lagrangian L(X, u, w, € 1 = f(x)

f

-

ui(gi(x)

-

Ei)

i=l

2 P

+

w.(h.(x) 3

j=l Theorem 5 . 4 . 1 .

3

(5.4.1)

E~+~).

[First-order changes in a Karush-Kuhn-

Tucker triple and first- and second-order changes in the optimal value function for the problem with rhs perturbations.]

If

(i) the functions defining P2(€) are twice continuously

*

differentiable in x in a neighborhood of x , (ii) the second-order sufficient condition for a local

*

minimum of P2(0) hold at x pliers u

*

and w

*

with associated Lagrange multi-

,

(iii) the gradients Vxgi(x

*

)

(for i such that gi(x

*

)

= 0)

and V h.(x*) (all j) are linearly independent, and X I * * (iv) ui > 0 when gi(x ) = 0, i = 1, m,

...,

then (a) x

*

is a local isolated minimizing point of P2(0) and

the associated Lagrange multipliers u* and w* are unique, (b) for

E

in a neighborhood of 0 there exists a unique,

once continuously differentiable vector function Y(E) [ x (E) , u ( E )

, w ( E)]

=

satisfying the second-order sufficient

conditions for a local minimum of problem P2(€) such that

*

y ( 0 ) = (x

,

u

*

,

and hence X(E) is a locally unique mini-

mum of P 2 ( ~ )with associated Lagrange multipliers

U(E)

and

W(E) I (c) for

E

in a neighborhood of 0, (5.4.2)

127

5.5 Karush-Kuhn-Tucker Triple and Optimal Value Function

(d) strict complementarity and linear independence of the binding constraint gradients hold at X(E) for (e)

for

E

E

near 0,

in a neighborhood of 0 the gradient of the

optimal value function is (5.4.3) (f)

for

E

in a neighborhood of 0, the Hessian of the

optimal value function is (5.4.4)

Equations (5.4.3) and (5.4.4) give us the means, at least

*

in theory, to calculate the gradient VEf

2 *

and the Hessian VEf

of the optimal value function for problem P*(E).

In practice,

estimates of the values of the Lagrange multipliers are

*

usually available and we can thus estimate VEf

.

However, we

may not generally have available a functional relationship for the Lagrange multipliers in terms of the parameters and therefore may not be able to calculate the gradient of the multipliers (the Hessian of the optimal value function) directly. However, the general procedure for obtaining the derivatives of a Karush-Kuhn-Tucker triple of P3(€) given in Chapter 4 is readily specialized to problem P 2 ( ~ )in the next section. 5.5.

FORMULAS FOR THE PARAMETER FIRST DERIVATIVES OF A KARUSH-KUHN-TUCKER TRIPLE AND SECOND DERIVATIVES OF THE OPTIMAL VALUE FUNCTION For the general parametric problem minx f(x,

E)

s.t. gi(x,

E)

2 0,

i = 1,

..., m,

128


it was shown i n S e c t i o n 4 . 2 t h a t t h e n o n t r i v i a l p a r a m e t e r d e r i v a t i v e s o f a Karush-Kuhn-Tucker W(E)

triple

Y(E)

:[x(E)

,

U(E)

,

I T a r e g i v e n by

where t h e A.. a r e t h e b l o c k components o f t h e i n v e r s e o f 13

(4.2.6)

the matrix

L d e n o t e s t h e Lagrangian

(3.2.1)

o f P 3 ( ~ ) ,and t h e s u p e r b a r

d e n o t e s r e f e r e n c e t o t h e s e t o f i n d i c e s i = 1,

..., r

associ-

a t e d with t h e i n e q u a l i t y c o n s t r a i n t s t h a t a r e binding a t t h e given solution

x(E).

obtained f o r t h e Aij,

Furthermore, e x p l i c i t formulas w e r e i n t e r m s o f t h e f i r s t and second p a r t i a l

d e r i v a t i v e s o f t h e problem f u n c t i o n s w i t h r e s p e c t t o x. These r e s u l t s h o l d i n a neighborhood of

E

= 0 for the

g e n e r a l problem P 3 ( ~ ) ,and s i m p l i f y c o n s i d e r a b l y f o r problem P2(€).

I n p a r t i c u l a r , w e o b t a i n g r e a t l y s i m p l i f i e d expres-

s i o n s f o r t h e Hessian of t h e optimal value f u n c t i o n s f o r t h e various cases specified.

I g n o r i n g t h e nonbinding i n e q u a l i t y


I29

constraints, the Lagrangian for problem P 2 ( ~ )is L(x, u, w, € 1 = f(x)

-

2 r

-

ui(gi(x)

Ei)

i=l

j=1

and hence

2

r VXL

=

P

VXf

j=1

i=l which is not

n explicit function of

the zero matrix.

w.V h.(x), 1 x 1 2 Therefore, VEXL is

E.

In addition, with gi

5

gi(x)

-

E i'

v,gi

-

..., 0, -1, 0, ..., 0), where the -1 is in the i-th column, hj (XI - E ~ + V~ h, i = 1, ..., r. Similarly, with h (0, ..., 0, -1, 0, ..., 0), where the -1 is in the ( j + r)-th column, j = 1, ..., p. Note that there are r + p parameters, (0,

=

the right-hand side of the binding constraints.

Therefore,

and (4.2.15) reduces to

€1 =

(5.5.1)

I

L

[Y( € 1

,E l

It is interesting to note that V€X(E) involves only A12 2 "

and VEf

(E)

(recalling (5.4.4)) involves only A22 (although

these may be interrelated, depending on the characteristics of the particular problem).

130


Suppressing the rows and columns corresponding to the (zero) partial derivatives associated with the nonbinding inequality constraints, we now calculate the Hessian of the optimal value function with respect to the right-hand side of the binding constraints. dimension r

ip .

Let H(E) denote this Hessian of

From (5.4.4)

11

[T 11.

Therefore, from ( 5 . 5 . 1 ) , H(E)

=

-[:

it follows that

A22(E)

(5.5.2)

In particular, the following result is an immediate impliAll functions are evaluated at [x(E), u(E),

cation of ( 5 . 5 . 2 ) .

w(E), E] unless otherwise indicated.

For emphasis we

explicitly denote the functional dependence of the matrix AZ2 on the parameter vector

E

as A22 ( E )

.

[Positive (semi)definiteness of the

Corollary 5.5.1.

Hessian V2f* of the optimal value function.] E

tions of Theorem 5 . 4 . 1 neighborhood of

E

=

If the assump-

hold for problem P 2 ( ~ ) ,then f o r

in a

E

0,

(a) A 2 2 ( ~ )is negative semidefinite if and only if V 2 f* ( E ) E

is positive semidefinite, and (b) A 2 2 ( ~ )is negative definite if and only if H(E) is positive definite. (c) If A22(0) is negative definite and if a l l the constraints are binding at x(0) positive definite.

=

*

x

,

2 *

then V E f

(E)

= H(E) is

131


Thus, i n a convex n e i g h b o r h o o d o f

E

= 0,

f

*

(E)

i s convex

if and o n l y i f A 2 2 ( ~ ) i s n e g a t i v e s e m i d e f i n i t e , and f

s t r i c t l y convex f u n c t i o n o f t h o s e components o f

E

* (E)

is a

associated

w i t h t h e c o n s t r a i n t s t h a t a r e b i n d i n g a t x ( 0 ) = x* i f A 2 2 ( 0 )

is negative d e f i n i t e . The e x p l i c i t H e s s i a n s and V € X ( E )

a r e d i s p l a y e d below f o r

t h e t h r e e ( a n d o n l y p o s s i b l e ) cases t r e a t e d p r e v i o u s l y .

Note

t h a t from ( 5 . 5 . 1 ) ,

Case 1 .

V:L-'

Exists

From ( 4 . 2 . 8 )

i t f o l l o w s t h a t A12

: hence (5.5.1)

= VxL 2 -1PTI PVxL 2 -1PT I - 1

and (5.5.2)

and

yield

and

Case 2 .

There a r e n L i n e a r l y I n d e p e n d e n t Binding C o n s t r a i n t Gradients

From ( 4 . 2 . 9 ) , (5.5.1)

A12

and (5.5.2)

v EX ( E )

= P-1 a n d A2 2 = -P-TV:LP-l;

hence

yield

= p-l[-ol

:]I

[Y( € 1 , E l

and (5.5.4)

132

S Computational Aspects: RHS Perturbations

Note that if P 2 ( € ) is a linear programming problem, then V€X(E) is constant and H(E) = 0, and if P2(€) is a quadratic programing problem, then VEx(€) is constant and H(E) is constant. Case 3 .

P

+

p

n

From (4.2.13),

where -1 T

All = TDxxT

and hence (5.5.1

and (5.5.2) yield

and

(5.5.5)

where -1

T = [palpb]

and

Dxx

=

TTVxLT. 2

An equivalent expression for the Hessian of the optimal value function in Case 1 was obtained by Buys (1972) by utilizing an augmented Lagrangian function. The following corollary follows easily and summarizes some of the implications of (5.5.2)

-

(5.5.5).

133

S . 5 Karush-Kuhn-Tucker Triple and Optimal Value Function

C o r o l l a r y 5.5.2.

2 [Relationship between AZ2 ( E ) and VxL.l

If the assumptions of Theorem 5.4.1 hold for problem P 2 ( ~ ) , then for any

E

in a neighborhood of

(a) Case 1 .

E =

0:

2

If VxL is positive definite, then AZ2 is

negative definite. (b) Case 2 .

2

The matrix VxL is positive definite (semi-

definite) if and only if A22 is negative definite (semidefinite). (c) Case 3 .

If V 2L is positive semidefinite, then A22 is X

negative semidefinite. Proof.

Conclusions (a) and (b) are immediate.

Conclusion

(c) is less obvious but may be readily proved as follows.

It

suffices to show that the matrix

[

B = L:v

-

V ~ L -1 T DT~ V2~ T LJ

is positive semidefinite.

2 Since V XL is symmetric and positive

semidefinite, it can be factored into the product of its 2 T "square root. " Let C be the square root: thus, V XL = C C and B is rewritten as B

=

CT(I

-

-1 TCT IC. CTDxxT

T Calling the matrix in brackets D, it follows that D D

=

D2 = D;

therefore D and hence also B are positive semidefinite, and it follows that AZ2(€) is negative semidefinite. In particular, the next result is an immediate implication of Corollaries 5.5.1 and 5.5.2

and subsumes (under the stipu-

*

lated conditions) the well known fact that f

(E)

problem P 2 ( ~ )is a convex programming problem.

is convex if


134 C o r o l Z a r y 5.5.3.

[Local convexity of the optimal value

function f*(E) of problem P2(~).1 If the assumptions of Theorem 5.4.1 hold for problem P 2 ( ~ ) ,then for neighborhood of

E

E

in a

= 0:

L (a) H(E) is positive definite if VXL[x(O)

, ~ ( 0 ,) W(0) r

is positive definite, if Case 1 or Case 2 holds, and 2 * 2 (b) VEf ( E ) is positive semidefinite if VXL[x(€), W(E) ,

E]

01

U(E)r

is positive semidefinite.

Hence, in particular, if the conditions of Theorem 5.4.1 hold for any

E

in a convex set S, then f*(E) will be twice

differentiable in S and, if P 2 ( ~ )is a convex program, then

*

f

(E)

will be convex in

S.

We note that an alternative derivation of the Hessian H(E) is provided by the general expression (4.2.17) that we verified in Section 4.2, where we recall that

It follows immediately for P2(~) that

so that

135


We first observe that this expression for H ( E ) is consistent with the formula given in (5.5.2).

To check this,

note from (5.5.1) that (5.5.6) yields

T 2 Consistency with (5.5.2) requires that A22 = -A12VxLA12. expressions (4.2.6) and ( 4 . 2 . 7 )

The

,

M =

imply, in particular, that 2 VXLAl2

+

PTA22 = 0, PAl2 = I.

Multiplying the first equation by A'i2 and using the second equation gives T 2 0 = A12VxLA12

T T + A12P A22

= A

T 2 12V xLAl2 + A22r

which yields the desired result. The results we have derived earlier in this section for H ( E ) can, of course, be readily derived from (5.5.6) as well.

In particular, the convexity of the optimal value function for the convex problem P 2 ( ~ )follows immediately from (5.5.6). Many other implications follow immediately from the various formulas that have been derived.

For example, if P2(€) is

a quadratic programming problem, then it is easy to see from the given results that Y(E) is locally affine and f* locally quadratic, under the given assumptions, and in fact closedform explicit expressions for these quantities may readily be derived.

Other structured problem classes, e.g., linear,

136


geometric, and separable programs, may also be readily addressed to derive respective realizations of the sensitivity formulas from the general formulas obtained in this and the previous chapter. EXAMPLES AND APPLICATIONS

5.6.

First-order sensitivity information in the form of Lagrange multipliers for problem P2(€) is often used to estimate the value of the objective function when the constraints are changed slightly.

An

immediate application of the prior

sensitivity results for problem P2(€) is the incorporation of second-order sensitivity information to provide a better estimate of the value of the objective function when the constraints are perturbed.

The value of the objective function

in a neighborhood of

0 may be approximated by means of a

E =

truncated Taylor series including terms of the second degree, i.e. , ftX(E)I

A

f[x(O)l + Vxf[x(0)l tX(E)

-

X(0)l (5.6.1)

x

[X(E)

-

X(0)l.

Since we can measure the change in f directly as a function of E,

*

using the optimal value function f

(E)

3

f[x(~)] we obtain (5.6.2)

Recall from (5.4.3) that in a neighborhood of VEf*(E) =

[U(E)

TI

and from (5.4.4) that

-W(E)

TI ,

E

= 0

137

5.6 Examples and Applications

Therefore, for problem P 2 ( ~ )in a neighborhood of

E

=

0 we can

write

+

+ ET [ : : ]

f[X(O)l

f*(E)

(112)E ~ [ vEu(o)]~. -VEW(0)

(5.6.3)

The derivatives of the multipliers can be calculated using (5.5.1) and the Case 1

-

3 formulas for A22, once u(0) and w(0)

are known. Note that we can also estimate [x(E), u(E), ~ ( € 1 1 by a first-order Taylor'.~series at about

E =

0, since we can cal-

culate VEx(0), VEu(0), and VEw(0) once [x(O), u(O), w(0)I is known. We illustrate the usefulness of this second-order estimate, using computational results obtained from a well known test problem, the Shell Dual (Colville, 1968). E x a m p l e 5. 6.1

[The Shell Dual:

first- and second-order estimates of

.

f* ( E ) ] The Shell Primal is the minimization of a cubic objective function subject to linear constraints,

2 n

minX

ejxj

+

n

n

xicijxj

i=l j=1

j=l

2

2 2 n

aijxj 2 Ei,

i = 1, 2,

xj,O,

j=l,2

j=1

1

djxi,

j=l

n

s.t.

+

...,m,

,..., n,

for specified values of the coefficients, ej, cij, dj, aij, and rhs levels

E ~ .

138


A dual of this problem, with cubic objective function and

quadratic constraints, may be written

n

m

n

n

m

1

s.t.

ajiuj = ei + 2

j=l

+

cjixj

2 3dixi,

j=l

i = l , 2

,..., n,

xi)O,

i=l, 2

,..., n,

ui)O,

j=l,2

,..., m.

In our example n = 5 and m = 10. shown in Table 5.6.1.

The problem data are

As is obvious from the notation used to

designate the problems, the right-hand sides of the nontrivial primal constraints ci (i parameters.

=

1,

..., 10) are treated here as

The values E O and c1 that are assigned to

E

in

this analysis are given in Table 5.6.2, along with A € = E ~ E-0. Table 5.6.1. i\j ej

' ij

a

i j

S h e l l P r o b l e m Data 4

5

- 36

-18

-12

-20 39 - 6 -31 32

-10 - 6 70 - 6 -10

32 -31 - 6 39 -20

-10 32 -10 -20 30

4

8

10

6

2

-16

2 - 2

- 2 2

1 0.4 0 - 4 1 0 - 1

0

- 2 - 9 0

0 0 2 0 - 2 - 4 - 3 3

2

3

-15

-27

2 3 4

5

30 -20 -10 32 -10

1

1

2 3 4

1

-

0

3.5

5

0 0

10

- 1 - 1 1 1

6 7 8 9

2

0

- 1

1

- 1

1

2

0 - 1

-

2.8

- 2 4

0 - 1 - 1 5

1

1

139


Table 5.6.2.

A l t e r n a t e Primal Right-Hand S i d e s

i

0 Ei

1

Ei

1

2

-40

-2

-40

-2

0

0

3

7

8

9

10

-1.5

-40

-60

3.5

1

-5

-1.25

-40

-60

3.5

1

1

0.25

0

0

0

0

4

5

6

-0.35

-4

-6

-0.3

-4

0

0.05

Denote the primal objective function by f(x).

The optimal are there-

value functions of the primal problem at EO and fore denoted by f" ( E0 ) and f* (E1) , respectively.

For purposes of experimentation it was decided to solve the dual, rather than the simpler primal. exercise was then to solve the dual at

E

* the (common primal-dual) optimal value f

The object of this =

E

0

,

thus yielding

o ( E ) , and thence to

obtain a first- and second-order estimate of f

*

(E

1 *

pare these results to the actual solution value f primal problem at

E

=

E

1

.

It follows that the variables u

) (E

and com1

of the

)

...,

(j = 1, 10) in the j dual, at a Karush-Kuhn-Tucker point, are the (primal) Lagrange

multipliers associated with the nontrivial primal constraints. Thus, since Theorem 5.4.1 was found to be applicable, corresponds to the gradient V E f *

U(E

0

o ) , and the partial deriva-

(E

tives of U(E0 ) with respect to the parameters give the 2 * components of the Hessian VEf

function f*

o

(E ) .

0

(E )

of the optimal value

The Hessian was obtained directly from an

estimate of these derivatives.

I40


The solutions of the dual problems at E O and E~ were estimated by the SENSUMT program, applying the algorithm based on the logarithmic-quadratic penalty function W(x, will be described in the next chapter.

The first derivatives

of the u components of the solution point the dual at

E',

using (6.8.3).

with respect to

E,

r) that

E,

[X(E

0

)

,

U(E

0

)

I of

were then estimated directly

Estimates of all the quantities required

for the Taylor's series approximations of f* ( E 1) were then available. The x component of the solution obtained for the dual 0 T problem at E' was X ( E ) = (0.44, 0.532, 0.595, 0.453, 0.169) ,

yielding f*

o)

(E

=

-43.729.

The u component was given by

0

0 = 4.11, U ~ ( E ) = 2.299, U ~ ( E ) = 10.28, and the re-

0

U~(E )

maining components equaling zero.

(This corresponds to the

fact that the third, fifth, and sixth nontrivial primal constraints are binding at

X(E

0

The calculated value of

) .)

as desired. That portion H(E0 ) 2 * ( E 0) = VEu(€0) inof the optimal value function Hessian VEf "

VEf

0

(E

)

coincided with

U(E

0

volving only the parameters

)

,

E3,

[

E

~

and ,

E

~

corresponding , to

the right-hand sides of the binding primal constraints, was estimated to be au3/aE3

au6/aE3 Let E~

=

au3/aE5

au6/aE5

au3/aE6

au6/aE6

(0.05, 1.0, 0.25), i.e.,

the nonzero components of AE.

5.210

-0.725

4.382

-0.725

0.702

-0.884

4.382

-0.884

5.984

the vector composed of

1

.

141


A f i r s t - o r d e r estimate o f f f * ( E 1 ) = f * ( Eo ) + V E f 1

*

(E

*

(E

1

)

i s g i v e n by

o )Eb

+ (0.05, 1 . 0 , 0 . 2 5 ) ( 4 . 1 1 , 2 . 2 9 9 ,

= -43.729

10.28)

T

= -38.655,

a n d a s e c o n d - o r d e r estimate by

= -38.655

+

0.337

= -38.318.

1 * 1 By s o l v i n g D ( E ) i t w a s c a l c u l a t e d t h a t f ( E ) = -38.330.

In

t h i s example t h e a b s o l u t e p e r c e n t a g e error f o r t h e f i r s t - o r d e r

estimate i s 0 . 8 5 % , w h i l e for t h e s e c o n d - o r d e r e s t i m a t e t h e p e r c e n t a g e error i s 0.03%. A s noted, H(E

0

1.

the duality relation was exploited t o calculate

S u b s e q u e n t l y , Armacost

(1976a) s o l v e d t h e S h e l l Primal

d i r e c t l y , and t h e components o f t h e H e s s i a n o f t h e o p t i m a l v a l u e f u n c t i o n o b t a i n e d by means o f

(6.8.2)

a n d (6.8.31,

a g r e e d c l o s e l y w i t h t h e estimates o b t a i n e d above by means o f t h e S h e l l Dual. I n a d d r e s s i n g problem P2 ( E )

,

a t t e n t i o n was f o c u s e d on an

a n a l y s i s of t h e s e n s i t i v i t y of a s o l u t i o n corresponding t o a p a r t i c u l a r v a l u e E O of

E.

But q u i t e obviously,

i f t h e re-

quired conditions hold with respect t o a d i f f e r e n t value E,

E

of

t h e n s i m i l a r s e n s i t i v i t y i n f o r m a t i o n c o u l d be o b t a i n e d a t

.

1

E~

T h i s l e a d s t o a n e x t e n s i o n of t h e t y p e o f a p p l i c a t i o n

j u s t d i s c u s s e d , where values.

E

may b e a l l o w e d t o t a k e on a number o f

142


C o n s i d e r t h e problem of f i n d i n g a s o l u t i o n t o

m in (XI€)

f (XI

s . t . g 1. ( x )

with

E

E

~

i = 1,

..., m ,

A s s u m e t h a t f o r any

i n a set S C Em+'.

d i t i o n s o f Theorem 5 . 4 . 1

,

Q

E

i n S t h e con-

apply.

One approach t o s o l v i n g problem Q i s t o t r e a t x and

E

w i t h o u t d i s t i n c t i o n and s o l v e t h e r e s u l t i n g problem i n n

+ m +

p variables.

phase approach.

Another p o s s i b i l i t y i s t o u s e a two-

F o r a g i v e n v a l u e of

E

0

= EO i n S, f i n d X ( E )

t h a t solves Q, along with t h e associated optimal m u l t i p l i e r s U(E U(E

0

)

0),

and W(E

W(E

0)

0

).

0 Then, u s i n g d e r i v a t i v e i n f o r m a t i o n a t [ X ( E 1,

I , determine

E~

t h a t d e c r e a s e s f [ x ( E0 ) I

s u b j e c t t o c1 b e i n g r e s t r i c t e d t o S.

w ( E') ] and c o n t i n u e .

1 Solve f o r [x(E )

W e can i n c o r p o r a t e second-order

f*

(E

o)

,

U(E

1

),

optimal

v a l u e s e n s i t i v i t y i n f o r m a t i o n i n t o t h i s approach. (Given an 0 0 U ( E ) , W ( E ) I , w e can a l s o o b t a i n a f i r s t estimate of [x(E'), 1 1 o r d e r e s t i m a t e of [ X ( E ) , u ( E ~ ) , w ( E ) ] and b e g i n t h e second i t e r a t i o n from t h i s a p p r o x i m a t i o n . ) I m p o r t a n t a p p l i c a t i o n s r e q u i r i n g t h e s o l u t i o n o f problems h a v i n g t h e same g e n e r a l s t r u c t u r e a s problem Q can be found. For example, G e o f f r i o n ( 1 9 7 0 ) and Silverman ( 1 9 7 2 a , b ) enc o u n t e r s u c h problems i n an approach t o s o l v i n g a c l a s s of

143


decomposable n o n l i n e a r programming problems h a v i n g t h e form k

k

s.t.

1

gRi(xR) 2 bit

i = 1,

xR

R = 1,

R=l

Xi,

E

..., m, ..., k.

The g e n e r a l a p p r o a c h , c a l l e d r e s o u r c e - d i r e c t i v e by G e o f f r i o n a n d r e s o u r c e a l l o c a t i o n by S i l v e r m a n , i s t o d e t e r m i n e t h e m-component v e c t o r s

E~

=

( E % ~ ,

...,

E

~

s u~ c h ) t h a t t h e

o p t i m a l s o l u t i o n s of t h e subproblems min

XR

fR(xR)

s . t . gLi(XR) 2 xi

where R = 1,

t-

E

L i

i = 1,

..., m,

XR'

..., k ,

ar

r e s p e c t i v e o p t i m a l compon

s o l u t i o n of t h e o r i g i n a l problem.

ts o f t h e

This usually involves t h e

s o l u t i o n of a "master" p r o b l e m k

MP

k

s.t.

*

1

E~~

L bi,

i = 1,

R=l

..., m ,

where f R ( E R ) i s d e f i n e d a s t h e o p t i m a l v a l u e o f t h e subproblem S P L ( ~ R ) . The f u n c t i o n s a n d c o n s t r a i n t s i n v o l v e d are s u c h t h a t t h e s u b p r o b l e m s a r e convex programs.

Both a u t h o r s solve t h e

master problem by u t i l i z i n g some s o r t o f a d i r e c t i o n - f i n d i n g procedure i n v o l v i n g d i r e c t i o n a l d e r i v a t i v e s o r s u b g r ad ien ts of

*

f L , and hence of t h e subproblem o p t i m a l v a l u e f u n c t i o n s . C l e a r l y S P Q ( ~ R ) ,a c c o u n t i n g f o r t h e c o n s t r a i n t s of MP, g i v e s

r i s e t o a problem h a v i n g t h e g e n e r a l form Q.


144

I t i s p o s s i b l e t h a t t h e second-order

r e s u l t s developed

above can be used t o d e v e l o p a s e c o n d - o r d e r d i r e c t i o n - f i n d i n g p r o c e d u r e t o s o l v e decomposable n o n l i n e a r programs when t h e necessary hypotheses a r e s a t i s f i e d . The f o l l o w i n g s m a l l example i l l u s t r a t e s t h e two i n d i c a t e d approaches t o o b t a i n i n g a s o l u t i o n of a problem of t h e t y p e Q.

Ezarnple 5 . 6 . 2 [ D i r e c t s o l u t i o n v e r s u s s o l u t i o n by d e c o m p o s i t i o n u s i n g

sensitivity results.] min

(X,

f(x,

€1

E)

= x2

E)

= x2

-

(xl

E)

= x2

-

( x l + 1)

s.t. gl(x, g2(x, where

( E ~ , E ~ E)

C o n s i d e r t h e problem

{

S =

1

E:

-

2

E~

-

-

2

-

El

2 0,

2

-

E2

1. 0,

1)

E;

Q

1. 0 ) .

W e s o l v e t h i s f i r s t a s a problem i n f o u r v a r i a b l e s .

The

Lagrangian i s d e f i n e d a s L ( x , u,

E)

-

= x2

-

u1(x2

(x,

-

1) E;).

The f i r s t - o r d e r n e c e s s a r y o p t i m a l i t y c o n d i t i o n s t h a t ( x ,

E)

(x, + 1)2

s o l v e Q r e q u i r e t h a t t h e r e e x i s t ul, b e f e a s i b l e and s a t i s f y 2U1(Xl

-

1)

+

2u2(x1 1

-

u1

+

1) = 0 ,

-

u2 = 0 ,

u1

+

2U3E1

= 0,

u

+

2U3E2

= 0,

2

-

-

-

u2(x2

-

-

E2)

u3(1

-

€1 2

u2 1. 0 s u c h t h a t ( x ,

E)

145


u1(x2

-

(x,

112

-

El)

u2(x2

-

(x, + 1)2

-

E

-

E;)

-

u3(1

-

E:

2

= 0,

)=

0,

(5.6.4)

= 0.

The r e s u l t u1 = u2 = 0 i s i m p o s s i b l e , w h i l e u1 = 1 and u2 = 0

i s a l s o excluded s i n c e t h i s l e a d s t o g 2 ( x I u

1

= 0 and u2 = 1 l e a d s t o g l ( x ,

< 0.

Similarly,

Therefore, t h e only

< 0.

E)

E)

remaining p o s s i b i l i t y i s t h a t u1 > 0 , u2 > 0 , and u3 > 0. may be v e r i f i e d t h a t t h e s y s t e m ( 5 . 6 . 4 )

and t h e f e a s i b i l i t y

r e q u i r e m e n t s a r e u n i q u e l y s a t i s f i e d by x1 = 0 , x2 = 1 E~

= E~

= -fi/2,

u1 = u2 = 1 / 2 ,

It

and u3 = f i / 4 .

-

fi/2,

T h i s is t h e

g l o b a l minimum b e c a u s e of t h e c o n v e x i t y of problem Q i n (x,

E)

.

The o p t i m a l s o l u t i o n v a l u e i s g i v e n by f ( x ,

E)

=

1

-

n/2. W e now view problem Q a s a problem i n t h e two v a r i a b l e s x1

and x 2 , p a r a m e t e r i z e d on t h e r i g h t - h a n d straints.

s i d e of t h e con-

To emphasize t h e d i s t i n c t i o n i n t r e a t m e n t , w e

r e l a b e l t h e problem

As noted,

Q(E).

t h i s corresponds t o t h e

g e n e r a l s t r u c t u r e o f a subproblem i n v o l v e d i n t h e i n d i c a t e d approach t o s o l v i n g c e r t a i n decomposable n o n l i n e a r programs. The L a g r a n g i a n i s now d e f i n e d a s L ( x , u,

E)

= x2

-

-

u1(x2

U2(X2

-

-

(xl

2

(xl + 1)

112

-

-

E 2 b

El)

and t h e f i r s t - o r d e r n e c e s s a r y o p t i m a l i t y c o n d i t i o n s t h a t solve Q ( E ) r e q u i r e t h a t , f o r given

E

i n S, there e x i s t

X(E)

146


u2 ( E ) 1. 0 s u c h t h a t

u ~ ( E ) ,

-

2u1(x1

1)

+

u1(x2

-

(xl

u2(x2

-

(xl

2u ( x 2 1

= 0

+

1) = 0 ,

-

u2 = 0,

1

-

-

1)2

-

El)

+

1)2

-

E

=

u2 = 0.

I t i s impossible f o r u

u1

u1

1

(5.6.5) = 0,

)=

2

and u2 = 1 i m p l i e s t h a t

< 0 for a l l

E

be f e a s i b l e and s a t i s f y

X(E)

i n S, i.e.,

X(E)

0.

I t may be v e r i f i e d t h a t

X(E)

= (-1, E ~ and )

i s not feasible.

u1 = 0 and u2 = 1 i m p l i e s t h a t g 2 [ x ( E ) , The o n l y r e m a i n i n g p o s s i b i l i t y i s t h a t for a l l

E

i n S.

E]

g l [ x ( ~ ) IE ]

Similarly,

< 0 for a l l

u ~ ( E )

E

i n S.

> 0 and ~ ~ ( >€ 01

I t f o l l o w s t h a t t h e unique s o l u t i o n of

(5.6.5)

c o m p a t i b l e w i t h t h e f e a s i b i l i t y r e q u i r e m e n t s i s g i v e n by =

X1(E)

€1

-

€2

,

+

-

E;

+ xl I x ( E )

U1(E)

=

2

U2(E)

=

l - X1I -

~

--

E

4 + E 1 - E

- 4

-

8

-

that

E

i s i n S.

X(E)

El

8

X(E)

where

+ ~8~~ +E 8~~~ +

+

2

16) = f*(E),

I

E2 I

The c o n v e x i t y o f Q ( E ) i n x f o r a l l

i s t h e unique g l o b a l s o l u t i o n of

Note t h a t f o r any

E

E

assures

Q(E).

i n S t h e s e c o n d - o r d e r s u f f i c i e n t con-

d i t i o n s a r e s a t i s f i e d , t h e b i n d i n g c o n s t r a i n t g r a d i e n t s Vxgl = (2

-

2x, 1) and Vxg2 = (-2

-

2x, 1) a r e l i n e a r l y i n d e p e n d e n t ,

and s t r i c t complementary s l a c k n e s s h o l d s .

S i n c e t h e problem

f u n c t i o n s a r e t w i c e c o n t i n u o u s l y d i f f e r e n t i a b l e , Theorem 5.4.1 applies.

147


S i n c e w e h a v e t h e s o l u t i o n e x p l i c i t l y i n terms o f t h e parameters, w e can c a l c u l a t e t h e p a r t i a l d e r i v a t i v e s d i r e c t l y , obtaining V x E

l

(E)

= (1/4,

-1/4), E2

-

r

VEu2 ( E ) = (-1/8, N o t e t h a t VE€*

VEf

*

E

8

+

4, = VEf*(E),

1/8).

= u ( E ) ~a ,s

W e may a l s o c a l c u l a t e

expected.

by t h e c h a i n r u l e

VEf

*

-1/4 (E)

= vxfs

E2

-

El

+

4

8

-

(El

-

E2

+ 4

2 *

N o t e a l s o t h a t VEf

(E)

E2

r

8 =

-

El

+

4)

8

V E u ( ~ ) a, s e x p e c t e d .

F o r p u r p o s e s o f i l l u s t r a t i o n and comparison w e s h a l l der i v e t h e same r e s u l t s u s i n g t h e g e n e r a l t h e o r y o f o b t a i n i n g t h e g r a d i e n t of t h e Karush-Kuhn-Tucker

t r i p l e presented i n

S e c t i o n 5.5. D i f f e r e n t i a t i n g (5.6.5) t h e J a c o b i a n m a t r i x M.

with r e s p e c t t o (x, u ) , we obtain

F o r c o n v e n i e n c e w e do n o t i n d i c a t e t h e

f u n c t i o n a l dependence on E a n d p e r f o r m t h e s u b s t i t u t i o n a t t h e

148


end o f t h e c a l c u l a t i o n s . 2Ul

+

I t follows t h a t

1)

2 ( x 1 + 1)

-1

0

u1

l)Ul

-

2(x1

0

2u2

x2

-

u2-

-

(x,

-1

u2

E1

0

x2

W e n o t e from t h e s e c o n d e q u a t i o n i n (5.6.5)

0

(x,

t h a t u1

+ 1)2 - E 2 +

u 2 = 1.

w e can w r i t e 0 0 -U

0

" 0 ];.

1

-u2

where 0

-

2(x1

0

-1

-1

0

-1

0

Since t h e r e a r e t w o l i n e a r l y independent binding c o n s t r a i n t g r a d i e n t s , Case 2 a p p l i e s , and w e o b t a i n 1 -_ 4 1 -z(xl

I f we let

I-

u2 > 0 and t h e l a s t t w o e q u a t i o n s

Using a l s o t h e f a c t t h a t ul, i n (5.6.51,

-

+ 1)

1 4

+(X1

-

--18

1 8

1 8

--1 8


I49

then

-

+

1) - 2 ( x 1

1)

1

Hence,

1

0

0

1

o r (5.5.1)

(4.2.14)

yields -

-a -2(x1 1 -

+ 1)

T1( X 1

VEY(E)

1

1 4

=

evaluated a t x = x (E)

VEY(E)

+

-

El

€2

8

=

4

-

+

El

-8

1

-1 8

-

-

1 -_ 4

4

8

-8

Thus,

-1

4

.

1)

1

1 -

-

-

E2

8

1 -_ 8 These r e s u l t s c o r r e s p o n d e x a c t l y w i t h t h o s e o b t a i n e d by d i r e c t differentiation. The g r a d i e n t and H e s s i a n , o b t a i n e d d i r e c t l y o r a p p r o x i m a t e l y , can b e u s e d t o d e v e l o p a s e c o n d - o r d e r a p p r o x i m a t i o n f o r F [ x ( E ) ] a s i n (5.6.2).

To s o l v e problem Q ( E ) w e would

t h e n s e e k t o minimize f

s u b j e c t t o t h e c o n s t r a i n t s on

*

(E)

E.

I n g e n e r a l , w e would p r o c e e d i t e r a t i v e l y , as i n d i c a t e d p r e viously.

I t i s o b v i o u s t h a t f o r t h e problem Q ( E ) t r e a t e d

above, t h e second-order approximation of f f

*

(E)

i s quadratic.

*

(E)

is exact, since

Consider t h e approximation a t

E

= 0.

150

5 Computational A\pect\. RHS Perturbation\

Then

and it f o l l o w s t h a t f

*

= f

(E)

*

0 ) + u ( O ) T E + ZE 1 TVEU(0)E

+

- mlE1

c2

2

-

~

E

+

~8~~

+

E

8~~~ + 1 6 1 .

To c o m p l e t e t h e s o l u t i o n o f problem Q ( E )

, we

now r e q u i r e

t h e m i n i m i z a t i o n of f * ( E ) i n p a r a m e t e r s p a c e , where 1 E;

S p e c i f i c a l l y , w e must d e t e r m i n e

2 0.

E~

and

E~

-

E~

1

-

t h a t solve

t h e problem,

s.t. 1

-

E2

1

E

-

> 0.

E2

2 -

The L a q r a n q i a n i s g i v e n by L(E, A)

=

-(161 -

E~2

+

-

A1(l

E~2

E l2

-

~

-

E

+ ~8c1 + E

B E 2~

16)

+

EZ),

and t h e f i r s t - o r d e r n e c e s s a r y c o n d i t i o n s t h a t q u i r e t h e e x i s t e n c e o f A 1. 0 s u c h t h a t

E

E

s o l v e E re-

b e f e a s i b l e and

satisfy El

-

E2

+

4

8 E 2 - E

8

A1(l

1

+ 4

-

E;

+

AIEl

= 0,

+

A1E2

= 0,

-

EZ)

= 0.

(5.6.6)

I5 I

5.6 Examples and Application5

By symmetry, w e must h a v e

E~

= E ~ ,from

f o l l o w s t h a t t h e u n i q u e s o l u t i o n of g i v e n by

E

= (-fl/2,

-fi/2)

which it r e a d i l y

(5.6.6)

and X = f l / 2 ,

(for X

0) i s

with f*(E) = 1

-

S i n c e problem E i s convex, t h i s g i v e s t h e unique g l o b a l

fi/2.

s o l u t i o n o f p r o b l e m E, a n d o u r c o m p o s i t e s o l u t i o n i s i d e n t i c a l t o t h a t o b t a i n e d by s o l v i n g problem Q ( E ) a s a problem i n f o u r variables. I t s h o u l d b e n o t e d t h a t a t y p i c a l c h a r a c t e r i s t i c of decom-

p o s a b l e p r o b l e m s n o t p r e s e n t i n t h i s example i s t h a t t h e cons t r a i n t s on t h e r h s p a r a m e t e r s o f e a c h subproblem u s u a l l y i n v o l v e t h e r h s p a r a m e t e r s from m o r e t h a n o n e subproblem. Thus, i n a d d i t i o n t o t h e g i v e n c o n s t r a i n t , o n e must a l s o i n t r o d u c e a d d i t i o n a l c o n s t r a i n t s on t h e v a l u e s of t h e right-hand s i d e s which i n s u r e t h a t t h e p a r a m e t e r v a l u e s o b t a i n e d i n t h e s o l u t i o n o f a g i v e n subproblem a r e c o m p a t i b l e w i t h t h e requirements of a l l "coupled" subproblems.

This complication

w a s n o t i n t r o d u c e d i n t h e i l l u s t r a t i v e problem Q. i.e.,

I f it w e r e ,

i f t h e p a r a m e t e r c o n s t r a i n t s d e f i n i n g S i n c l u d e d param-

e t e r s from o t h e r s u b p r o b l e m s , t h e " c o n t i n g e n t " subproblem f e a s i b i l i t y c o n s t r a i n t s would h a v e r e q u i r e d e x p l i c i t and j o i n t consideration.

O b v i o u s l y , t h i s d i f f i c u l t y would have t o b e

s u r m o u n t e d by any a l g o r i t h m i c a p p r o a c h .

Nonetheless, t h e

a v a i l a b i l i t y of second-order s e n s i t i v i t y i n f o r m a t i o n should s t i l l o f f e r a d i s t i n c t a d v a n t a g e i n making m o r e e f f e c t i v e adj u s t m e n t s of t h e p a r a m e t e r s . Numerous a d d i t i o n a l a p p l i c a t i o n s o f t h e u s e o f s e n s i t i v i t y i n f o r m a t i o n w i l l u n d o u b t e d l y c o n t i n u e t o b e found.

F o r ex-

ample, numerous t h e o r e t i c a l r e s u l t s i n d u a l i t y t h e o r y and s t a b i l i t y a n a l y s i s have been concerned w i t h t h e f i r s t - o r d e r

152


*

behavior of the optimal value function f

(E)

of P 2 ( E ) .

Compu-

tational implementations and, quite possibly, additional theoretical insights, might well be further enhanced by incorporating the full power of second-order sensitivity information. A

computational procedure (and the computer program

SENSUMT) for estimating the first- and second-order sensitivity

of the optimal value function and the first-order sensitivity of a Karush-Kuhn-Tucker triple has evolved, through several stages of refinement.

It is based on a well known class of

penalty function algorithms for solving the usual nonlinear programming problem and provides a model for demonstrating how standard NLP algorithms can effectively estimate sensitivity information in the normal course of solving a problem.

This

algorithmic approach is developed fully in the next chapter.

111 Algorithmic Approximations


Chapter 6

Estimates of Sensitivity Information Using Penalty Functions

6.1

INTRODUCTION

I n Chapters 3

-

5 a first-order

a n a l y s i s is given of t h e

v a r i a t i o n of a l o c a l i s o l a t e d s o l u t i o n

X(E)

of a general

p r o b l e m P 3 ( ~ ) ,a l o n g w i t h t h e a s s o c i a t e d Lagrange m u l t i p l i e r s [u(E),

w(E)],

function.

and a s e c o n d - o r d e r a n a l y s i s of t h e o p t i m a l v a l u e

*

As i n d i c a t e d , o n c e ( x

,

u

*

,w

*

= [x(O), u(O), w(0)l

i s a v a i l a b l e , a l l t h e p a r a m e t e r d e r i v a t i v e s c a n ( i n t h e o r y ) be calculated. Although t h e s e r e s u l t s a r e o f c o n s i d e r a b l e i n t e r e s t , i n p r a c t i c e i t i s d e s i r a b l e t o e x p l o i t t h e f a c t t h a t it i s poss i b l e t o e s t i m a t e s e n s i t i v i t y i n f o r m a t i o n from i n f o r m a t i o n g e n e r a t e d by a s o l u t i o n a l g o r i t h m a s a s o l u t i o n i s a p p r o a c h e d . I t seems a p p a r e n t t h a t any s o l u t i o n a l g o r i t h m must g e n e r a t e ,

o r a t l e a s t manipulate,

an abundance o f i n f o r m a t i o n t h a t i s

r e l e v a n t t o s e n s i t i v i t y and s t a b i l i t y a n a l y s i s . I t f o l l o w s t h a t t h e c l a s s o f a l g o r i t h m s b a s e d on t w i c e -

d i f f e r e n t i a b l e p e n a l t y f u n c t i o n s can r e a d i l y be adapted t o p r o v i d e e s t i m a t e s of t h e s e n s i t i v i t y information developed i n Chapters 3

-

5 , without additional assumptions.

t h e b a s i s f o r t h i s r e s u l t i n Theorem 6 . 2 . 1

We establish

i n the next section,

156

6 Sensitivity Information Using Penalty Functions

formulating an appropriate penalty function for the problem P 3 ( ~ )and absorbing the problem parameter directly into the penalty function.

It follows that the assumptions of Theorem

of Chapter 3 guarantee the existence of a trajectory of

3.2.2

unconstrained local minima of the penalty function converging

*

to x

.

For changes in

E

near

E

=

0, perturbations of this

trajectory are shown to relate closely to perturbations of x(E).

The material in this chapter is based on Fiacco (1976)

and Armacost and Fiacco (1975, 1976). Though we shall confine our analysis to a class of penalty function algorithms in this chapter, it is reiterated that any valid solution algorithm utilizes and generates analogous information that can be manipulated to provide corresponding sensitivity estimates.

This fact will be further corroborated

in the next chapter. 6.2.

APPROXIMATION OF SENSITIVITY INFORMATION USING THE LOGARITHMIC-QUADRATIC MIXED BARRIER-PENALTY FUNCTION METHOD For convenience and specificity the results will be given

in terms of the logarithmic barrier function combined with a quadratic penalty term to handle the equality constraints. For the problem P3 ( E ) , minx f(x,

E)

s.t. gi (x,

E)

2 0,

i. = 1,

h.(x,

E)

=

0,

j = 1,

3

..., m, P

P3 ( € 1

157

6.2 Logarithmic-Quadratic Mixed Barrier-Penalty Function Method

t h i s f u n c t i o n i s d e f i n e d as

2 P

+ (1/2r)

(6.2.1) 2

hj

(Xr

E ) r

j=1

where r i s a p a r a m e t e r d e f i n e d on t h e p o s i t i v e r e a l s .

The

a n a l y s i s c a n b e c a r r i e d o u t i n a s i m i l a r f a s h i o n f o r any twice-differentiable penalty function.

Numerous i n t e r e s t i n g

p r o p e r t i e s i n v o l v i n g s u c h p e n a l t y f u n c t i o n s have been documented and are w e l l known.

F o r o u r p r e s e n t p u r p o s e s , w e need

t h e f o l l o w i n g r e s u l t s , which are e x t e n s i o n s o f r e s u l t s o b t a i n e d by F i a c c o a n d McCormick (1968, Theorems 6 a n d 1 7 ) . W e r e c a l l t h a t t h e p e n a l t y f u n c t i o n a l g o r i t h m b a s e d on

(6.2.1)

c o n s i s t s e s s e n t i a l l y of approximating a local s o l u t i o n

of p r o b l e m P 3 ( ~ )by a n u n c o n s t r a i n e d l o c a l minimum W(x,

E,

X(E,

r ) of

r ) f o r r > 0 and s m a l l t h a t s t r i c t l y s a t i s f i e s t h e

i n e q u a l i t i e s of P 3 ( c ) .

Under t h e p r e s e n t a s s u m p t i o n s , t h e

v a l i d i t y o f t h e a l g o r i t h m i s e s t a b l i s h e d a s an immediate c o r o l l a r y o f t h e n e x t theorem. To a v o i d v a r i o u s t r i v i a l e x c e p t i o n s , i t i s assumed i n t h e f o l l o w i n g t h a t a t l e a s t one c o n s t r a i n t i s p r e s e n t i n P 3 ( c ) . Theorem 6.2.1.

[Approximation of f i r s t - o r d e r

r e s u l t s a n d d e t e r m i n a t i o n of estimates from W(x, 19761.

I f t h e a s s u m p t i o n s o f Theorem 3.2.2

neighborhood about

(E,

sensitivity E,

r ) (Fiacco,

hold, then i n a

r ) = ( 0 , 0) t h e r e e x i s t s a u n i q u e o n c e

158


r) , u ( E , r) ,

c o n t i n u o u s l y d i f f e r e n t i a b l e v e c t o r f u n c t i o n [x (E, r ) ] satisfying

W(E,

u, w,

E)

= 0

uigi(xr

E)

= r,

VL(X,

i = 1,

..., m , *

w i t h [ x ( O , O), u ( 0 , O), w ( 0 , O ) ] = ( x f o r any

r ) n e a r ( 0 , 0 ) and r > 0 ,

(E,

,

(6.2.2)

u

*

,w

*

1 , and such t h a t

r) is a locally

X(E,

u n i q u e u n c o n s t r a i n e d l o c a l m i n i m i z i n g p o i n t o f W(x, E, r ) , 2 w i t h g i [ x ( E r r ) , E ] > 0 ( i = 1, m ) a n d V W [ X ( E , r ) , E, rl

...,

positive definite. [x(E, r ) ,

The e x i s t e n c e o f

Proof.

U(E,

r),

W(E,

r ) l as

d e s c r i b e d f o l l o w s f r o m t h e i m p l i c i t f u n c t i o n t h e o r e m (Theorem u s i n g t h e same a r g u m e n t a s i s u s e d i n Theorem 3.2.2

2.4.1)

prove p a r t ( b ) , n o t i n g t h a t t h e J a c o b i a n m a t r i c e s of

The f a c t t h a t g i [ x ( E , gi(x

,

gi[x(E,

(6.2.2)

c o i n c i d e when r = 0 .

and ( 3 . 2 . 2 )

*

to

r),

E]

> 0 f o r a l l i such t h a t

0 ) > 0 f o l l o w s f o r a l l (E, r ) n e a r ( 0 , 0) s i n c e

r ) , E]

+

gi(x*,

0) a s

(E,

0 ) = 0 , t h e f a c t t h a t u ~ ( E r, )

t h a t gi(x*,

For i such

r) * (0, 0). +

> 0 (the latter

u;

i n e q u a l i t y d e r i v i n g from t h e a s s u m p t i o n o f s t r i c t complementary s l a c k n e s s ) a s (E, r ) * ( 0 , 0 ) i m p l i e s f r o m ( 6 . 2 . 2 ) gi[x(E, r ) , €1 = r/ui(E, Thus, g i [ x ( E ,

( 0 , 0).

r > 0 and

(E,

r ) > 0 f o r r > 0 and r),

E]

vw

..., m )

r) near provided t h a t

r ) i s s u f f i c i e n t l y close t o (0, 0 ) .

By t h e d e f i n i t i o n s ( 6 . 2 . 1 ) (3.2.1)

> 0 ( i = 1,

(E,

that

o f t h e p e n a l t y f u n c t i o n W and

o f t h e L a g r a n g i a n L, w e h a v e t h a t = Vf

-

m

1

i=l

P

(r/gi)Vgi

+

1

j=l

(hj/r)Vh j

(6.2.3)

6.2

159

ogarithmic-Quadratic Mixed Barrier-Penalty Function Method

and VL = Vf

m

1

-

2 P

+

uiVgi

(6.2.4)

wjVhj.

j=l

i=l

C o n s e q u e n t l y , p r o v i d e d o n l y t h a t gi # 0 ( a l l i ) and r # 0 ,

SO

t h a t W i s w e l l d e f i n e d , a n y s o l u t i o n (x, u , w ) o f t h e s y s t e m (6.2.2)

y i e l d s VW = VL = 0.

(x, u , w ) = [ x ( E , r ) ,

I n particular, t h i s is true for

r),

U(E,

W(E,

r )I

,

and w e c a n c o n c l u d e

( u s i n g also t h e f a c t proved i n t h e p r e v i o u s paragraph) t h a t VW[x(E, r ) , f o r r > 0 and first-order

r ] :V L [ X ( E , r ) ,

E,

(E,

r),

U(E,

W(E,

r ) , €1 = 0

r) s u f f i c i e n t l y close t o (0, 0).

necessary condition t h a t

s t r a i n e d m i n i m i z i n g p o i n t o f W(x,

E,

X(E,

Thus, t h e

r ) b e a n uncon-

r) is s a t i s f i e d f o r

(E,

r)

as indicated. I n t h e r e m a i n d e r o f t h e p r o o f assume, f o r c o n v e n i e n c e of n o t a t i o n , t h a t a l l f u n c t i o n s are e v a l u a t e d a t U(E,

r),

W(E,

r ) ] unless otherwise specified.

d e f i n i t e n e s s of VLW f o r

[x(E,

r),

The p o s i t i v e

n e a r 0 and r > 0 and s m a l l can be

E

shown a s f o l l o w s . D i f f e r e n t i a t i n g (6.2.3)

and ( 6 . 2 . 4 )

w i t h r e s p e c t t o x and

comparing y i e l d s m

P

i=l

and p r e -

j=1

a n d p o s t m u l t i p l y i n g by any v e c t o r

z T v 2wz = z

v

LZ

+

1 m

(r/g:)(vgiz)

i=l

2 P

(Vhjz) j=1

+ (l/r)

Z

gives

2 (6.2.5)

.

2,

I60


The s e c o n d - o r d e r

,

s u f f i c i e n t c o n d i t i o n s (Lemma 3.2.1)

t o g e t h e r w i t h s t r i c t complementary s l a c k n e s s , i m p l y t h a t

* , u* , w* , 0 ) z * Vgi(x , 0 ) z

T 2

z V L(x

> 0

f o r z = 0 such t h a t

= 0

for all i e B

*

(6.2.6)

- { i l gi(x*, 0) = Vh.(x*, 0 ) z = 0 , 7

j = 1,

01,

..., p .

C o n s i d e r a n y s e q u e n c e C E ~ , r k } w i t h rk > 0 a n d ( c k , r k )

+

T

2 W e s h a l l show t h a t z VW z > 0 f o r k l a r g e , w h e r e {zk} (0, 0 ) . k k k 2 i s a n a r b i t r a r y s e q u e n c e o f u n i t v e c t o r s i n En and V Wk d e n o t e s t h e Hessian of W e v a l u a t e d a t ( f k , r k ) . gi*

-z

gi(x

*

,

0),

*

*

h . ( , ~ 01, 3

h. 3

L

*

For convenience, l e t 5 L(x

*

,

u

*

,

w

*

,

0).

S e l e c t a c o n v e r g e n t s u b s e q u e n c e of { z k } , r e l a b e l it {zk} f o r c o n v e n i e n c e , and c a l l t h e l i m i t ( u n i t v e c t o r ) ?.

If

VgzZ # 0 f o r some i e B* o r i f Vh*Z # 0 f o r some j , t h e n t a k i n g 3 l i m i t s i n (6.2.5) w i t h z = z k and ( E , r ) = ( f k , r k ) and rec a l l i n g t h e a s s u m p t i o n of s t r i c t c o m p l e m e n t a r y s l a c k n e s s , T 2 *I f Vg.2 = y i e l d s t h e c o n c l u s i o n t h a t zkV Wkzk + +m a s k -+ +m. 1 0 for a l l i

E

B

*

*-

a n d Vh.z = 0 f o r j = 1, . . . , p , 7

then we again

t a k e l i m i t s o v e r t h e a p p r o p r i a t e s u b s e q u e n c e i n (6.2.5) t o conclude t h a t

T 2 T 2 -T 2 *l i m i n f k zkV W z > l i m i n f zkV Lkzk = z V L z > 0 , k k b e c a u s e of

(6.2.6)

.

T 2 T h i s shows t h a t z V Wz > 0 f o r a l l z # 0 , p r o v i d i n g r > 0 and

(f,

r) i s close t o (0, 0)

f o r a l l such

(E,

r).

,

i.e.,

VLW i s p o s i t i v e d e f i n i t e

This a l s o implies t h a t

u n i q u e l o c a l m i n i m i z i n g p o i n t o f W(x,

E,

X(E,

r) i s a

r ) f o r any

and r > 0 and s m a l l , c o m p l e t i n g t h e p r o o f .

E

near 0

161


) =~ [ x ( E , r ) ,

With ( x , u , w

t i o n of

(6.2.2)

M ( E f

r),

W(E,

r)lT E

are i d e n t i c a l l y s a t i s f i e d f o r

the equations (6.2.2) a neighborhood o f

U(E,

(0, 0 ) .

r) i n

(E,

E

yields

r)r

N ( E r

t h e p r e c i s e a n a l o g y of

r),

Using t h e c h a i n r u l e , d i f f e r e n t i a -

with respect t o

r ) V E Y ( E r r) =

Y(E,

(3.2.3).

(6.2.7) The q u a n t i t i e s i n t h i s

e q u a t i o n c o r r e s p o n d t e r m by t e r m w i t h t h o s e a p p e a r i n g i n (3.2.3)

,

except t h a t the p

x

p zero matrix ( t h e last p rows

and columns) t h a t a p p e a r e d i n t h e J a c o b i a n m a t r i x M ( E ) i s now r e p l a c e d i n t h e new J a c o b i a n m a t r i x M ( E , r ) by a p m a t r i x w i t h e a c h d i a g o n a l e l e m e n t e q u a l t o -r. t o (3.2.4)

and (3.2.5)

,

x

p diagonal

Corresponding

r e s p e c t i v e l y , w e have t h a t (6.2.8)

-1

*

(6.2.9)

N r

where M

*

and N

*

a r e a s d e f i n e d i n (3.2 5 ) .

S i n c e t h e s y s t e m s of e q u a t i o n s ( 3 . 2 . 2 )

and ( 6 . 2 . 2 )

when r = 0 , t h e c o n c l u s i o n s o f Theorems 3.2.2 that

0) = Y(E)

Y(E,

imply

Y(E,

0 ) = V E y ( ~ f) o r

E

sufficiently

Hence, t h e f o l l o w i n g c o r o l l a r y i s i m m e d i a t e .

c l o s e t o 0.

Corollary 6 . 2 . 2 .

[Convergence o f p e n a l t y f u n c t i o n s o l u -

t i o n s and s e n s i t i v i t y e s t i m a t e s . ] Theorem 6 . 2 . 1

r)

and 6 . 2 . 1

( r e t a i n i n g t h e n o t a t i o n w e have i n t r o d u c e d ,

f o r convenience) and V

Y(Er

coincide

hold, then f o r

I f t h e assumptions of

F near 0,

0) = y(F)

+

(6.2.10)

and VEy ( E ,

r)

+

VEy ( T , 0 ) = VEy (5)

(6.2.11)

162 as

6 Sensitivity Information Using Penalty Functions E

(0,

-+

T and r

+

0, with

In particular, for

0).

r ) c o n f i n e d to a neighborhood of

(Er

(E,

r)

(0,

-+

01,

and (6.2.13) Based on t h e s e r e s u l t s , i t i s a p p a r e n t t h a t w e c a n e s t i -

mate

Y(E)

= [x(E),

desired for

E

u(E),

w ( € ) I T a n d V E y ( ~ as ) c l o s e l y as

n e a r 0 , by Y ( E ,

r) = [x(E,

r),

U(E,

r),

T

r)]

W(E,

and V E y ( € , r ) , r e s p e c t i v e l y , p r o v i d i n g r i s s u f f i c i e n t l y close t o 0.

Having o b t a i n e d a s o l u t i o n Y ( E ,

s u f f i c i e n t l y c l o s e t o ( 0 , O),

r ) of (6.2.2)

for

(E,

r)

V E y ( € , r ) c a n b e c a l c u l a t e d from

(6.2.8). Though, i n p r i n c i p l e , any t e c h n i q u e c o u l d be u t i l i z e d t h a t y i e l d s a s o l u t i o n of

(6.2.2)

near

r ) = (0,

(E,

-

obviously formulated t h e system (6.2.2)

01, w e h a v e

which s h o u l d p r o p e r l y

be viewed a s a p e r t u r b a t i o n of t h e f i r s t - o r d e r

n e c e s s a r y con-

d i t i o n s f o r a local minimizing p o i n t of problem P 3 ( € ) t h a t i t i s s a t i s f i e d by [ x ( E ,

r),

U(E,

r),

W(E,

-

such

r ) ] for r > 0

i f a n d o n l y i f t h e l o g a r i t h m i c - q u a d r a t i c mixed b a r r i e r - p e n a l t y f u n c t i o n W(x, pliers

U(E,

E,

r ) i s minimized by

r ) and

W(E,

X(E,

r ) (with the multi-

r ) a p p r o p r i a t e l y d e f i n e d , as s p e c i f i e d ) .

Thus, t h e i n d i c a t e d a l g o r i t h m i s t h e u s u a l p e n a l t y f u n c t i o n procedure based on determining, minimizing p o i n t

rk 4 0 , k = 1, 2 ,

X(E,

for fixed

r k ) o f W(x,

..., f o r

E,

a unconstrained

E,

r k ) f o r rk > 0 a n d

x such t h a t g i ( x r

an a p p r o p r i a t e n e i g h b o r h o o d o f x ( 0 , 0 ) = x

*

E)

.

> 0 ( a l l i) i n

I63


Having d e t e r m i n e d

X(E,

r k ) , a n estimate of

l a r g e , t h e Lagrange m u l t i p l i e r s from t h e r e l a t i o n s

u ~ ( E r , k) =

[u ( E )

U(E,

rk) =

W.(E,

3

It follows t h a t the vector

3

r k ) ,W ( E ,

rk) = [ x ( E , r k ) , ( 6 . 2 . 2 ) and Y ( E , r k ) * [ x ( E ) ,

c a n be e s t i m a t e d

rk), €1,

rk/gi[x(E,

h . [ x ( E , r k ) , € ] / r k ( a l l i and j ) . Y(E,

, w (E) I

when k i s

X(E)

u(E),

r ) I i s a s o l u t i o n of k

'I'

w ( E ) ] as k *

rk) with respect t o

m.

The f i r s t

c a n t h e n be

p a r t i a l d e r i v a t i v e s of

Y(E,

o b t a i n e d from ( 6 . 2 . 8 1 ,

and would c o n s t i t u t e a n estimate o f

E

V € Y ( E ) ( a c c o r d i n g t o ( 6 . 2 . 1 1 ) ) , p r o v i d e d t h a t k i s l a r g e and

E

i s n e a r 0. An a l t e r n a t i v e t o u t i l i z i n g (6.2.8)

to calculate

V E y ( ~ r, ) i s a v a i l a b l e , u s i n g t h e f a c t t h a t t h e H e s s i a n of W(x,

E,

r ) is n o n s i n g u l a r a t a minimizing p o i n t

X(E,

r)

.

This

w i l l be s e e n t o have t h e advantage of i n v o l v i n g a s m a l l e r

-

matrix inverse t h e (n

+

m

i n (6.2.8)

t h a t of t h e n

+

+

x

n Hessian of W r a t h e r th an

+

p)

-

and r e q u i r e s o n l y i n f o r m a t i o n t h a t i s r e a d i l y

x

(n

m

p ) Jacobian matrix M ( E , r ) appearing

a v a i l a b l e from t h e W f u n c t i o n a t a minimizing p o i n t .

In t h i s

approach, t h e p e n a l t y f u n c t i o n i s minimized t o o b t a i n

X(E,

which d e te r min es V € X ( E , from W [ X ( E , r ) ,

E,

r),

r ) i n terms o f q u a n t i t i e s a v a i l a b l e

r ] and i t s H e s s i a n .

The m u l t i p l i e r s and

t h e i r d e r i v a t i v e s c a n t h e n be c a l c u l a t e d t o s a t i s f y ( 6 . 2 . 2 1 , w h i c h d e t e r m i n e s t h e s e q u a n t i t i e s a s f u n c t i o n s of x, The m a t r i x V € X ( E ,

E,

and r.

r) i s o b t a i n e d from t h e f o l l o w i n g r e s u l t .

To a v o i d any a m b i g u i t y , d e n o t e by X ( E , r ) t h e t r a j e c t o r y o f l o c a l minima o f W(x, Theorem 6.2.1.

X(E, r)

E,

r) whose e x i s t e n c e i s p r o v e d by

W e showed t h a t when r > 0 X(E,

r)

and

VEX(€, r)

V€X(E,

r)

(6.2.14)

I64

6 Sensitivity Information Using Penally Functions

for

The f o l l o w i n g c o r o l l a r y i s a d i r e c t

r) near (0, 0 ) .

(E,

consequence o f Theorems 3.2.2 f o r e s t i m a t i n g changes i n

-

by c o r r e s p o n d i n g c h a n g e s i n

[Approximation of

CoroZZary 6 . 2 . 3 .

VEX(^,

r) a n d

6.2.1 X(E,

X(E)

and p r o v i d e s a b a s i s

r).

X ( E r

X(E,

and 6 . 2 . 1

for

r)

by

Under t h e a s s u m p t i o n s o f Theorem

x ( T ) and V € K ( E , r)

(E,

and V € X ( E )

r ) n e a r ( 0 , 0 ) and r > 0 , i t f o l l o w s t h a t

(E,

+

VEX

r).]

X(E)

2

-

-+

r) = -V W [ X ( E , r ) ,

VEx(F) as E,

E

+

F and r

rl -1 V 2E X W [ X ( ~ , r),

0 , and

+

rl

E,

(6.2.15) S i n c e t h e system o f e q u a t i o n s (3.2.2)

Proof.

and (6.2.2)

c o i n c i d e when r = 0 , it f o l l o w s from t h e c o n c l u s i o n s o f Theorems 3 . 2 . 2

X(E,

r)

t h a t , f o r r > 0 and

and 6 . 2 . 1

(E,

r ) n e a r (0, 01,

x ( E , 0) = x ( F )

-+

(6.2.16)

and V€X(E,

as

E -+

r)

VEx(E, 0 ) = VEx(EI)

-+

h and r

-+

0 (where c o n v e r g e n c e i s component by compoFrom t h e f a c t t h a t VW[X(E, r ) ,

nent, i n a l l cases).

rl = 0

E,

a n d K ( E , r ) i s o n c e c o n t i n u o u s l y d i f f e r e n t i a b l e f o r any 0 a n d r > 0 and s m a l l , v~w[x(E,

r),

E,

it follows t h a t

E,

r1

= 0,

2 a n d s i n c e w e showed t h a t ' V W i s p o s i t i v e d e f i n i t e f o r

(E,

n e a r ( 0 , 0 ) and r > 0 , w e o b t a i n t h e e x p r e s s i o n g i v e n i n f o r VEST(€,

near

~ I V ~ X ( E r, )

+ V ; ~ W [ X ( Er, ) ,

(6.2.15)

E

r).

r)

165

6.2 Loearithmic-Quadratic Mixed Barrier-Penalty Function Method

T h i s r e s u l t gives t h e b a s i s f o r a p p r o x i m a t i n g u ( T ) a n d

w ( F ) as w e l l .

With

-u ~ ( E r, )

r ) near ( 0 , 0 ) and r > 0 , d e f i n i n g

(E,

r), €1,

r/gi[X(E,

i = 1,

...

m,

(6.2.17)

. . . I

p r

(6.2.18)

r

and

W.

3

h . [ S t ( E , r ) , ~ l / r , j = 1, 3

r)

(E,

since VL[P(E, r ) ,

-

U(E,

= VW[T(E, r ) ,

it f o l l o w s t h a t s o l u t i o n of

Y(E,

r),

-

rl = 0

E,

[X(E, r ) ,

r)

(6.2.2).

r ) , €1

W(Er

-

U(E,

T h e r e f o r e , T h e o r e m s 3.2.2

imply that

-

u ~ ( E ,r )

+

W.

+

iii(Er 0 )

= ui(E),

as

E

-+

(E,

r)

E and r

ij

-+

( T , 0) =

j

...

r

m,

(6.2.19)

3

t h e s-th component o f

(6.2.17) E

j = 1,

...

p,

(6.2.20)

r

and ( 6 . 2 . 1 8 )

w i t h respect t o

yields

r)/aEs

= ( - r / g i )2 ( v' g i [ a S i l ( E r

+

(3,

0.

D i f f e r e n t i a t i o n of

aiii(E,

W.

a n d 6.2.1

i = 1,

and 1

r ) , W ( E , r ) l T is a

i = 1,

agi/aEs),

...

r)/aEsr

s = 1,

r

aFn(E,

r)/aes1 T

..., m; ..., k,

(6.2.21)

and

aw.7 ( E ,

= (l/r)(vhj[aF,(E,

+

ah./ass) 3

r

j = 1,

s

..., axn(€,

r)/acsr

= 1,

...

r

p;

..., k,

r)/aEsl

T

(6.2.22)

I66


where t h e f u n c t i o n s are a l l e v a l u a t e d a t [ Y ( E , r ) , € 1 .

The

p r e v i o u s r e s u l t s a l s o i m p l y t h a t t h e matrices whose components a r e g i v e n r e s p e c t i v e l y by (6.2.21)

and (6.2.22)

converge

component-wise as f o l l o w s :

v ~ U ( E ,r )

.+

(6.2.23)

V E u ( F r 0 ) = VEu(F)

(6.2.24) as

E

7 and r

+

0.

+

C o n s e q u e n t l y , w e c a n estimate Y ( E ) a n d V E y ( € ) by and V € Y ( E ,

r > 0.

r) f o r

(E,

F o r any s u c h

Y(E, r)

r ) s u f f i c i e n t l y close t o ( 0 , 0 ) a n d (E,

r ) , t h e s e q u a n t i t i e s can a l l be c a l -

c u l a t e d o n c e a l o c a l u n c o n s t r a i n e d m i n i m i z i n g p o i n t X ( E , r ) of W(x,

E,

r ) h a s been determined ( i n t h e r e g i o n such t h a t

gi[x(E, r ) , containing

E]

> 0 ( i = 1,

..., m ) ) i n

a s u i t a b l e open s e t

0).

X(E,

Returning t o t h e problem P 3 ( 0 ) , it f o l l o w s t h a t t h e usual p e n a l t y f u n c t i o n approach u t i l i z i n g W(x, 0 , r ) t o f i n d a l o c a l s o l u t i o n o f t h i s problem can be used t o approximate y ( 0 ) =

*

(x

,

u

*

w * ) a~n d V E y ( 0 ) .

A minimizing sequence

{x(O, r k ) }

of

{W(x, 0 , r k ) Ic o n v e r g i n g t o x* i s g u a r a n t e e d by Theorem 6 . 2 . 1 f o r rk > 0 and small. The p o i n t F ( 0 , r k ) may b e c o n s i d e r e d a n

estimate of x

*

.

The q u a n t i t i e s i n v o l v e d i n t h e r i g h t - h a n d

s i d e of ( 6 . 2 . 1 5 ) c a n be e v a l u a t e d a t E = 0 , r = r k r o n c e x ( 0 , r ) has been determined, y i e l d i n g t h e e s t i m a t e

k

v Ex ( 0 ) a t [?(Or

A

v E~

( 0 ,r k ) =

r k ) , O r rkl.

2 -v 2w-1 vEXw

(6.2.25)

167


The a s s o c i a t e d L a g r a n g e m u l t i p l i e r s f i r s t partial derivatives a t U(O) W(O)

v EU ( O )

f

E

[ u ( E ) , w ( E ) I and t h e i r

= 0 c a n t h e n be e s t i m a t e d from

U(O, r k ) ,

(6.2.26)

~ ( 0 ,r k ) ,

(6.2.27)

A vEii(O,

rk),

(6.2.28)

(6.2.29) where t h e r e s p e c t i v e q u a n t i t i e s o n t h e r i g h t - h a n d o b t a i n e d from t h e r e l a t i o n s ( 6 . 2 . 1 7 ) , (6.2.22)

evaluated a t

E

6.2.18),

s i d e are

6.2.211,

and

= 0 , r = r k'

It i s important t o note,

from t h e p o i n t o f v i e w o f compu-

t h a t an estimate o f t h e H e s s i a n 2 matrix of t h e penalty function V W t h a t i s involved i n calcut a t i o n a l implementation,

l a t i n g t h e estimate VEW(O,

r k ) o f V E x ( 0 ) i n (6.2.251,

wi22

a l r e a d y b e a v a i l a b Z e i f o n e of many v a r i a n t s o f Newton's method

i s u s e d t o compute t h e u n c o n s t r a i n e d m i n i m i z i n g p o i n t X ( 0 , r k ) o f W(x, 0 , r k ) . Thus, u t i l i z i n g any o f t h e s e w e l l known proc e d u r e s f o r t h e u n c o n s t r a i n e d m i n i m i z a t i o n s , much o f t h e i n f o r m a t i o n r e q u i r e d t o c a l c u l a t e t h e s e n s i t i v i t y i n f o r m a t i o n by t h i s t e c h n i q u e w i l l h a v e a l r e a d y been g e n e r a t e d i n implementing t h e usual penalty function algorithm. V'

W appearing i n (6.2.25)

EX

(Of c o u r s e , t h e o t h e r t e r m

i s a l s o known o n c e t h e problem i s

s p e c i f i e d and n e e d o n l y b e e v a l u a t e d a t [ W ( O ,

rk), 0 , r k l . )

The a n a l o g o u s o b s e r v a t i o n h o l d s f o r t h e c a l c u l a t i o n o f t h e f i r s t p a r t i a l d e r i v a t i v e s o f t h e problem f u n c t i o n s a t [Z(O,

r k ) , 01 w i t h r e s p e c t t o

E.

L e t y(0, r k )

f [ z ( O , r k ) , 01.

Using t h e c h a i n r u l e , w e o b t a i n z

v E f ( 0 , r k ) = VXfVEX + V E f ,

(6.2.30)

168


where t h e f u n c t i o n s are e v a l u a t e d a t ( x ,

r ) = lTI(0, r k ) , 0 ,

E,

The vector of p a r t i a l s V f w i l l b e known o n c e t h e prob-

rkl.

E

l e m i s s p e c i f i e d a n d need o n l y be e v a l u a t e d .

The g r a d i e n t Vxf

w i l l normally have a l r e a d y been c a l c u l a t e d , i n a p p l y i n g t h e

usual penalty function algorithm.

Thus, t h e work r e q u i r e d t o

o b t a i n t h e estimate

v Ef*(O)

VXfVETt + VEf,

where a l l f u n c t i o n s o n t h e r i g h t - h a n d s i d e a r e e v a l u a t e d as i n (6.2.30)

,

and h

t h e d e r i v a t i v e s o f which a p p e a r i n ( 6 . 2 . 2 1 ) a n d

j'

is c o n s i d e r a b l y reduced.

The same a p p l i e s t o t h e gi

(6.2.22).

EXAMPLES OF ESTIMATES O F SOLUTION POINT AND LAGRANGE MULTIPLIER PARAMETER DERIVATIVES

6.3.

W e use t h e examples o f S e c t i o n 3 . 3

(Chapter 3) to i l l u s -

t r a t e the application of these algorithmic results. reader is r e f e r r e d t o Section 3 . 3 f o r t h e values of VEY

(E)

The and

Y(E)

.

E x a m p l e 3. 3 . 1

s.t. x

min X x

x

E,

E

E

1

The l o g a r i t h m i c b a r r i e r f u n c t i o n f o r t h i s p r o b l e m i s W(x,

E,

r)

x

any r > 0 by X(E,

r)

+

E

-

r ln(x

X(E,

=

X(E)

r) =

-

+ r.

E

as r

E),

-+

u n i q u e l y minimized f o r x > Hence, f o r any v a l u e o f

0 and V € X ( E ,

U(E,

r ) :r / g [ x ( E , r ) ] , w i t h g [ x ( E , r )

have t h a t a l l r.

U(E,

r)

=

1=

U(E)

and V € U ( E

E

r)

r)

illus-

since

Also,

X(E,

and

E,

r) = 1 = V€X(E),

t r a t i n g t h e c o n c l u s i o n s of C o r o l l a r y 6 2 . 3 .

E

-

E

= r, we

= 0 = V U(E)

for

I69

6.3 Solution Point and Laprange Multiplier Parameter Derivatives

F o r t h i s example, t h e s o l u t i o n d i f f e r e n t i a b l e f o r any v a l u e o f

E,

X(E)

o f P 3 ( ~ )i s u n i q u e and

and i t i s n o t e d t h a t t h e

c o r r e s p o n d i n g estimates u s i n g t h e p e n a l t y f u n c t i o n a r e v a l i d f o r any

T h i s m i g h t h a v e been a n t i c i p a t e d by Theorems 3 . 2 . 2

E.

and 6 . 2 . 1 and C o r o l l a r y 6 . 2 . 3 ,

are s a t i s f i e d f o r any

s i n c e t h e r e q u i r e d assumptions

E.

Examvle 3.3.2

min

x x1

+

€lX2 g2(x,

where

> 0 and

E~

E~

E

- -xl

0,

# 0.

To estimate t h e s o l u t i o n and s e n s i t i v i t i e s , w e u s e t h e

logarithmic b a r r i e r function (6.2.1), W(x,

E,

As u s u a l ,

r)

+

x

E1x2

-

r 1n(x2

which f o r o u r example is

-

E2x1) 2 2

-

r ln(-xl).

r > 0 and t h e f u n c t i o n i s minimized o v e r t h e set o f

p o i n t s s a t i s f y i n g t h e c o n s t r a i n t s w i t h s t r i c t i n e q u a l i t y as

r * 0.

From t h e r e q u i r e m e n t o f s t a t i o n a r i t y , VW(x,

w e f i n d t h a t W(x,

E,

r)

E,

r ) i s minimized u n i q u e l y ( o v e r t h e i n d i -

c a t e d r e g i o n ) by

4

f o r any r > 0 .

r) =

4

As e x p e c t e d from t h e t h e o r y ,

€;XI

x ~ ( E ,

r)

+ +

~

2

= 0,

- 1 / 2 ~= ~~ ~~ ~( and € 1 x ~ ( E r, )

2 2

* 1 / 4 ~ ~= 6~ ~ ( as € 1 r * 0.

170


The Lagrange m u l t i p l i e r s a s s o c i a t e d w i t h

X(E,

r ) are given

and

Calculating f i r s t derivatives with respect t o

E

-1/2

a x 1 ( € , r ) - -16~~~:(1 + 8 ~ ~ ~ : r )r + 4 ~ ;( 1 + 8 2 4 aEl 1 6 ~ ~

we obtain 2

1/2

~

~

~

~

~

r

)

~

and ax1(€, r ) 3E.2

-1/2

-

- 3 21 ~2 ~ +~ 8 ~ ~ ( ~1 ~ ; rr )+

ax1(€,

r)/aE2

+ ~

/

+

1

E

2 4

1

I t f o l l o w s t h a t a x 1 ( € , ‘)/as1

~

6

I+ 1

~

(1+E 8

~

~

/ 2 22 =~ a x~l ( E~) / a E ~l

E3 ~= E ax1(E)/aE2, ~

as r

+

~

2 )1/2]

~

~

~

~

and

0.

One c a n a l s o v e r i f y t h a t

and ax2(€, r) aE2

-

ax1(€, r )

--2~~

aE2

Taking l i m i t s y i e l d s a x 2 ( € , ax2(€,

r)/aE2

Thus, V € X ( E ,

+

r)

/

r)/aEl

+

2 3 2 ~= ~a x ~ 2 ( E~ ) / a E 2 as r

-

1

+

V€X(E)

as r

c o n c l u d e d from t h e t h e o r y .

+

3 2

- 1 / 2 ~= ~a x~2 ( E ~ ) / a E 2 and -t

0 , as d e s i r e d .

0 , component by component, a s

~

~

171

6.4 Extensions

F i n a l l y , from t h e p r e v i o u s c a l c u l a t i o n o f

r ) , w e have

U(E,

that

v E u ( ~ r, ) =

as r

+

[ ]

-+

1

0

0

0

= VEU(E)

0 , and t h e f i r s t - o r d e r r e s u l t s a r e complete.

Note t h a t , a s i n t h e p r e v i o u s example, t h e r e s u l t s a r e v a l i d f o r a s e t of v a l u e s o f

In t h i s case t h e conditions

E.

i m p l y i n g e x i s t e n c e and convergence o f t h e q u a n t i t i e s o b t a i n e d above a r e s a t i s f i e d a s l o n g as

E~

0 and

E

~

# E 0.

~

Thus, f o r

b o t h examples t h e r e s u l t s go w e l l beyond t h e c a l c u l a t i o n and e s t i m a t i o n of s e n s i t i v i t y information a t a p o i n t ( f o r a given value of

E)

,

and e s s e n t i a l l y p r o v i d e a " p a r a m e t r i c a n a l y s i s . "

The i n d i c a t e d f u n c t i o n s e x i s t f o r a l a r g e s e t o f v a l u e s o f

E

and t h e f u n c t i o n s depending on t h e p e n a l t y p a r a m e t e r r converge point-wise i n E t o t h e i r indicated l i m i t s a s r

+

0.

EXTENSIONS

6.4.

These examples s u g g e s t t h a t t h e t h e o r e t i c a l r e s u l t s o b t a i n e d h e r e can be e x t e n d e d , under a p p r o p r i a t e c o n d i t i o n s , t o allow f o r a parametric analysis (i.e., ysis i n the large).

a s e n s i t i v i t y anal-

Such would a p p e a r t o f o l l o w , n o t i n g t h a t

t h e p r o o f s of t h e main r e s u l t s h i n g e p r i m a r i l y on t h e i m p l i c i t f u n c t i o n theorem a p p l i e d t o t h e Karush-Kuhn-Tucker

c o n d i t i o n s of

problem P3(0) and on an a p p r o p r i a t e p e r t u r b a t i o n o f t h o s e conditions. o f any

E

The r e s p e c t i v e r e s u l t s a r e v a l i d i n a neighborhood or

(E,

r ) f o r which t h e c o n d i t i o n s ( s a t i s f y i n g t h e

h y p o t h e s e s o f t h e i m p l i c i t f u n c t i o n theorem) a r e assumed t o

172

6 Senqitivity Informalion Using Penalty Functions

hold.

I t i s c l e a r t h a t w e need o n l y i n v o k e t h e a n a l o g o u s

conditions f o r every

or

E

r ) i n a p p r o p r i a t e sets i n which

(E,

t h e s e parameters are allowed t o vary, t o e s t a b l i s h t h e v a l i d i t y o f t h e c o n c l u s i o n s i n t h e s e sets. The f o l l o w i n g c o r o l l a r y o f Theorem 6 . 2 . 1 i s a l s o , a n a l o g o u s l y , an immediate consequence o f t h e same e x t e n s i o n s o f t h e c l a s s i c a l i m p l i c i t f u n c t i o n theorem and t h e f a c t t h a t t h e s y s t e m s of e q u a t i o n s ( 3 . 2 . 2 )

and ( 6 . 2 . 2 )

c o i n c i d e when r = 0 .

The f i r s t p a r t i s a n e x t e n s i o n o f Theorem 2 . 4 . 1 0

and p r e c i s e l y

p a r a l l e l s C o r o l l a r y 3.2.5. [ E x i s t e n c e and convergence o f h i g h e r -

Coro l l a r y 6 . 4 . I.

r ) .]

order d e r i v a t i v e s of Y ( E , l a r y 3.2.5

hold, then

y ( ~ r, ) : [ X ( E ,

r),

i n a neighborhood o f

(E,

r),

U(E,

w(E,

r) = (0, 0 ) .

t i o n s a r e a n a l y t i c i n (x,

r ) l T E CP I f t h e problem func-

i n a neighborhood of

E)

r ) i s a n a l y t i c i n a neighborhood of

then Y ( E ,

Furthermore,

Y(E,

y with respect t o

r) E

at

(E,

...,

Proof.

(E,

(x*, O ) ,

r)

=

(0, 0 ) .

and t h e j - t h p a r t i a l d e r i v a t i v e s of

+ Y(E)

r ) converge t o t h e c o r r e s p o n d i n g

j-th partial derivatives a t j = 1,

I f t h e a s s u m p t i o n s of Corol-

E

as r

+

0, for

E

n e a r 0 , where

p. The e x i s t e n c e and c o n t i n u i t y o f t h e h i g h e r - o r d e r

p a r t i a l d e r i v a t i v e s and a n a l y t i c i t y r e s u l t from t h e i m p l i c i t f u n c t i o n theorems,

Theorems 2 . 4 . 1

system of e q u a t i o n s ( 6 . 2 . 2 )

and 2 . 4 . 2 ,

applied t o the

as i n t h e p r o o f o f Theorem 6 . 2 . 1 .

From t h i s and t h e f a c t t h a t ( 3 . 2 . 2 ) and ( 6 . 2 . 2 )

r = 0 , w e conclude t h a t Y(Er

r)

+.

Y(E,

0) = Y ( E )

c o i n c i d e when

173

6.5 Perturbed Karush-Kuhn-Tucker System

and t h e p a r t i a l d e r i v a t i v e s a t

(E,

r ) also converge t o t h e

corresponding p a r t i a l d e r i v a t i v e s a t hood o f

E

=

as r

E

+

0, i n a neighbor-

0.

These r e s u l t s a r e n o t p u r s u e d f u r t h e r i n t h i s book. could have a p p l i c a t i o n , e.g.,

i n providing a b a s i s f o r esti-

mating t h e para m e t r i c v e c t o r f u n c t i o n component o f mate E~(E).

X(E)

They

by d e v e l o p i n g e a c h

X(E)

a s a power series i n

E,

y i e l d i n g t h e esti-

This i s p r e c i s e l y analogous t o t h e e x t r a p o l a t i o n

t h e o r y developed f o r e s t i m a t i n g t h e c o u r s e of a minimizing t r a j e c t o r y i n t e r m s of t h e involved parameter i n p en alty f u n c t i o n methodology ( F i a c c o and McCormick, 1 9 6 8 ) . X(E,

r ) c o u l d b e d e v e l o p e d i n a power series

Similarly,

r) i n

;(E,

(E,

F o r r n e a r 0 , G ( E , r ) may b e an a d e q u a t e e s t i m a t e o f G ( E ) . e s t i m a t e G(E,

r). To

r ) , one could u t i l i z e t h e p e n a l t y f u n c t i o n

method b a s e d on W(x, s a t i s f y i n g (6.2.2)

E,

r ) t o o b t a i n s e v e r a l v a l u e s of

X(E,

r)

for r > 0 and f i x e d , corresponding to

s e v e r a l v a l u e s o f E i n a s u i t a b l e domain. 6.5.

SENSITIVITY CALCULATIONS BASED ON THE PERTURBED KARUSH-KUHN-TUCKER SYSTEM W e now t u r n b r i e f l y t o t h e p r o b l e m o f o b t a i n i n g a d i r e c t

s o l u t i o n o f t h e system V E y ( ~ r, ) =

M(E,

r)-'N(E,

(6.2.8)

r),

which r e s u l t s from d i f f e r e n t i a t i o n w i t h r e s p e c t t o p e r t u r b e d Karush-Kuhn-Tucker

s y s t e m (6.2.2)

at Y(E,

E

of t h e

r ) , and

show how t h i s r e l a t e s t o t h e s o l u t i o n o b t a i n e d i n S e c t i o n 6 . 4 by way o f t h e p e n a l t y f u n c t i o n minimum

X(E,

r).

Again, assume

174


t h a t t h e c o q d i t i o n s of Theorem 3.2.2

6.2.1

guarantees t h a t f o r

(E,

hold.

As shown, Theorem

r ) n e a r ( 0 , O),

Y(E,

r ) i s i n C1

and s a t i s f i e s t h e f o l l o w i n g s y s t e m : VXL(X,

u, w,

uigi(x, h . (x, 3

E)

= 0

E)

=

r,

i = 1,

..., m ,

E)

=

w.r,

j = 1,

..., p .

3

L e t U = d i a g ( u i ) and G = d i a g ( g . ) . 1

(6.2.2)

Then, t h e J a c o b i a n of

(6.2.2) t a k e n w i t h r e s p e c t t o ( x , u, w ) i s g i v e n by

M(E,

r)

=

uivxgi

gi

0

0

-r

L

(6.5.1)

Then, d e f i n i n g T

-vx9

r)

-U-lG

0

we obtain

M(E,

r) E

[

0

6

-U

0

0

1

vtl

(6.5.2)

-rI

(6.5.3)

I75


A s b e f o r e , w e s h a l l see t h a t w e n e e d o n l y c a l c u l a t e

G(E,

r)

gi[x(E,

1,

.

-1

r ) , € 1 > 0 , and u i ( € , r )

[r :I,

..., m ) . Q E

R e c a l l t h a t Theorem 6 . 2 . 1

provides t h a t r > 0,

r ) , € 1 (i = -1 rU and d e f i n e

= r/gi[x(E,

T h e r e f o r e , w e can w r i t e G = where U2 = UU.

Then (6.5.4) e x i s t s and

(6.5.5)

where P = (-Vxg T , VxhTjT.

Letting

+

i t i s e a s i l y shown t h a t i f (V:L All

= (V:L

A12

=

T

+

-1

(l/r)PTQP)

= (l/r)AllP

T

(l/r)PTQP)-’

exists,

,

Q,

(6.5.6)

2 A 2 2 = - ( l / r ) Q+ ( l / r )QPAllPTQ. f o r r > 0 and (E, r ) n e a r

To show t h e e x i s t e n c e o f All

(0, 0) w e now make u s e of t h e p e n a l t y f u n c t i o n W(x, t h e r e s u l t s o f Theorem 6.2.1. Lagrangian L ( x , u, w ,

E)

E,

r ) and

Recalling t h e d e f i n i t i o n of t h e

(3.2.1)

and t h e p e n a l t y f u n c t i o n

I76 W(x,

6 Sensitivity Information Using Penalty Functions E,

r ) (6.2.11,

it may e a s i l y be v e r i f i e d by d i r e c t

calculations that, a t

2

X(E,

r),

m

V W = VX 2L + X 2

(r/g:)VxgTVxgi

+ (l/r)

i=1

using (6.2.17) VxW 2 = VxL 2

P

VxhTVxhj

j=l

and h e n c e

+

(l/r)PTQP,

where P and Q a r e d e f i n e d above. 2

t h a t V W is p o s i t i v e d e f i n i t e f o r X

and h e n c e All

2

= V;W-'

I t f o l l o w s f r o m Theorem 6 . 2 . 1 (E,

r ) n e a r ( 0 , 0 ) and r > 0

e x i s t s , as d e s i r e d .

F o l l o w i n g t h e same p r o c e d u r e as b e f o r e f o r M ( E ) ,

letting

so t h a t

w e have (6.5.7)

177


L e t N ( E , r) :

(-VExL 2 T , -(UVEg)T , -VEh

Jacobian of t h e system (6.2.2) VEY(Er

b e t h e n e g a t i v e of t h e

taken with respect t o

E.

Then

r ) = M ( E , = ) - I N ( €r ,)

A

and h e n c e , w i t h N ( E ,

r)

= I-VExL 2 T

2 -A21VExL + A 2 2

E q u a t i o n (6.5.8)

,

VEgT, -VEhTIT, w e o b t a i n

(-;I:)

i s a d i r e c t analog of

(4.2.15).

S i n c e w e h a v e g i v e n o s t e n s i b l y t w o methods f o r c a l c u l a t i n g t h e u n i q u e p a r t i a l d e r i v a t i v e s V E y ( € , r ) , o n e by (6.5.8) a n d a s e c o n d by ( 6 . 2 . 1 5 ) ,

(6.2.211,

and ( 6 . 2 . 2 2 )

,

it i s

n a t u r a l t o i n q u i r e which p r o c e d u r e m i g h t be p r e f e r a b l e a n d how t h e c a l c u l a t i o n s are r e l a t e d .

I t t u r n s o u t t h a t t h e pro-

c e d u r e s are e s s e n t i a l l y e q u i v a l e n t :

w e n e e d o n l y compute All

t o solve ( 6 . 5 . 6 ) ,

t h e o t h e r components A i j

of

A(E,

S i n c e All

and h e n c e (6.5.81,

r)-’ b e i n g a f u n c t i o n o f All = V;W-’,

and known q u a n t i t i e s .

t h i s i s seen to be proportionate to t h e

e f f o r t n e e d e d t o s o l v e t h e s y s t e m g i v e n by (6.2.151, and (6.2.22).

(6.2.211,

I n f a c t , t h e two p r o c e d u r e s c a n be p l a c e d i n 2 p r e c i s e c o r r e s p o n d e n c e by e x p r e s s i n g VxW a n d VxW a s f u n c t i o n s

178


2 of VxL and VxL,

respectively, a t

X(E,

r).

I n particular, note

that

m

V E2X W =

v 2E XL +

D

1

(r/gf)VxgTVEgi i=l

+ (l/r)

$

VxhTVEhj

j=1

using previously defined notation.

From ( 6 . 2 . 1 5 )

and t h e

above r e s u l t s , w e h a v e t h a t V ~ X ( E ,

r)

2 -1 2

= -V XW

=-A

VEXW

V 2E XL +

11

and t h e r e f o r e V€X(E,

r ) = -A

V2 L

11

EX

+

A12( -VEh vE9)

where A12 i s g i v e n by ( 6 . 5 6 ) .

This r e s u l t coincides with

t h a t o b t a i n e d i n t h e upper p a r t of

(6.5.8).

Using t h e e x p r e s -

s i o n f o r VEx(€, r ) a n d t h e f o r m u l a s g i v e n f o r V E u ( € , r ) a n d VEw(c, r ) i n (6.2.21)

a n d (6.2.22), w e a l s o o b t a i n , t e r m by

t e r m , t h e bottom p a r t of

(6.5.8)

,

a f t e r regrouping t e r m s i n a

s t r a i g h t f o r w a r d manner. Thus , t h e p r o c e d u r e s b a s e d o n ( 6 . 5 . 8 )

(6.2.21), a n d ( 6 . 2 . 2 2 ) ,

, or

(6.2.15)

,

a r e s e e n t o be v i r t u a l l y i d e n t i c a l ,

t h e only d i f f e r e n c e being whether t h e d a t a i s o r g an ized i n

179

6.6 Optimal Value Function Sensitivity Estimates

terms o f t h e L a g r a n g i a n o r t h e p e n a l t y f u n c t i o n , r e s p e c t i v e l y . N o t e t h a t any t e c h n i q u e f o r o b t a i n i n g s o l u t i o n s o f

c o u l d be u s e d .

However, s i n c e t h e p e r t u r b e d Karush-Kuhn-

Tucker system (6.2.2) first-order

(6.2.2)

is simply another r e p r e s e n t a t i o n of t h e

n e c e s s a r y c o n d i t i o n s t h a t hold a t an unconstrained

minimum o f t h e l o g a r i t h m i c - q u a d r a t i c l o s s f u n c t i o n W(x, (6.2.1),

t h e o r g a n i z a t i o n o f d a t a b a s e d o n (6.2.151,

and (6.2.22)

r)

(6.2.211,

would p r o b a b l y b e more e f f i c i e n t if a n a l g o r i t h m

based on W w e r e used t o o b t a i n t h e s o l u t i o n o f 6.6.

E,

(6.2.2).

OPTIMAL VALUE FUNCTION SENSITIVITY ESTIMATES C o n s i d e r t h e c l a s s o f a l g o r i t h m s b a s e d on t w i c e c o n t i n u -

o u s l y d i f f e r e n t i a b l e p e n a l t y f u n c t i o n s and, as b e f o r e , f o r s p e c i f i c i t y , c o n s i d e r t h e l o g a r i t h m i c - q u a d r a t i c loss p e n a l t y function (6.2.1). hold. X(E,

Let Y(E,

A s s u m e t h a t t h e c o n d i t i o n s o f Theorem 3.2.2

r ) be d e f i n e d as i n Theorem 6.2.1,

where

r ) i s a l o c a l l y u n i q u e m i n i m i z i n g p o i n t o f W(x,

E,

r).

Define t h e " p e n a l t y f u n c t i o n o p t i m a l value" a s w * ( E , r) z w[X(E, r ) , Since l i m y r+O

lim W

r+O+

*

E,

E,

r) =

Y(E),

E,

rl.

(6.6.1)

it r e a d i l y f o l l o w s t h a t

r ) = l i m W[X(E, r ) , r+O+ = L[x(E),

U ( E ) ,

E,

W(E),

r]

EI

= f[X(E), E l = f*(E)I

t h e o p t i m a l v a l u e f u n c t i o n o f p r o b l e m Pj ( E ) .

W e m i g h t con-

I80


2

2 *

l i m V E W ( ~ , r ) = VEf ( E ) . r+O+ R e c a l l t h a t VXW[x(€, r ) , VEW

*

= VXWVEX

= VEf

+

VEW =

-

These r e s u l t s do i n f a c t f o l l o w , E,

rl

v Ew

(r/gi)VEgi

-

5

2

+

(hj/r)VEhj

j=1

i=l = VEf

Thus

= 0.

uiVEgi

f

+

wjVEhj

j=1

i=l = VEL,

2 " from which i t a l s o f o l l o w s t h a t V E W ( E ) = V E [ V E L T ] .

r e l a t i o n s c h a r a c t e r i z e t h e behavior of W

*

(E,

These

r ) , essentially

t h e optimal value function of t h e associated penalty function problem : min

X

W(x,

h.(x, 3

E,

E)

r)

= 0,

,..., p ,

j = 1

and p r e c i s e l y p a r a l l e l t h o s e g i v e n i n Theorem 3.4.1

f o r f*CE).

As a c o r o l l a r y , w e immediately o b t a i n a v e r i f i c a t i o n o f t h e c o n j e c t u r e s t a t e d above. W e have t h u s proved t h e f o l l o w i n g r e s u l t s .

[ F i r s t - and second-order

Theorem 6.6.1.

penalty function optimal value W o f Theorem 3.2.2 hood o f (a)

E

*

(E,

r).]

changes i n t h e

I f the conditions

h o l d f o r problem P 3 ( ~ ) ,t h e n i n a neighbor-

= 0,

*

VEW ( E ,

r)

= V€L[X(E,

r),

U(E,

r),

W(E,

r), €1, (6.6.2)

181

6.6 Optimal Value Function Sensitivity Estimates

(6.6.3)

i=l

Using these results and recalling the assumed twice continuous differentiability of the functions in (x,

*

neighborhood of (x

,

(r),

E)

in a

the following corollary follows

immediately. [Convergence of the penalty function

Corollary 6.6.2.

optimal value, gradient and Hessian to the respective quantities associated with the optimal value function.]

Given the

conditions of Theorem 3.2.2 of Chapter 3, then in a neighborhood of

E

= 0,

(6.6.4)

(b)

lim VEW*(e, r) = VEL[x(~),U(E) r+O+ * = f (€1,

, w(E),

€1 (6.6.5)

and hence also

We can therefore estimate the value of the gradient and Hessian of the optimal value function of problem P3(€) by evaluating the corresponding quantities obtained from the penalty function at a minimizing point near (0, 0) and r

.+

0.

X(E,

r), for

(E,

r)

182

6 Senhitivity Information Using Penalty Functions

EXAMPLE OF ESTIMATES OF OPTIMAL VALUE AND FIRST- AND SECONDPARAMETER DERIVATIVES

6.7.

The second example of S e c t i o n 3 . 3 w i l l be used t o i l l u s t r a t e t h e a p p l i c a t i o n s of t h e s e r e s u l t s . Example 3.3.2

m i n x x1 i-€ l X 2 s.t. gl(x, g2(x,

2 2

E)

= -E2X1

E)

= -xl

+ x2

2 0,

> 0 -

where

E~

> 0 and

E~

# 0.

The s o l u t i o n p o i n t was found t o be

X(E)

I.

=

U(E)

=

[b']

The o p t i m a l v a l u e f u n c t i o n i s f

*

(E)

=

-1

2' 4E1E2

Hence

and

2 * VEf ( € 1 =

-

1 2E1E2

32

"1 2E1E2

T h i s example was used i n S e c t i o n 6 . 3 t o i l l u s t r a t e t h e u s e o f p e n a l t y f u n c t i o n methods t o o b t a i n e s t i m a t e s o f t h e p a r t i a l d e r i v a t i v e s of t h e s o l u t i o n p o i n t and t h e Lagrange m u l t i p l i e r s .

183

6.7 Optimal Value and First- and Second-Parameter Derivatives

W e u s e it i n a s i m i l a r way h e r e t o i l l u s t r a t e t h e u s e o f

p e n a l t y f u n c t i o n methods t o o b t a i n estimates o f t h e o p t i m a l v a l u e f u n c t i o n a n d f i r s t - and s e c o n d - o r d e r c h a n g e s w i t h r e s p e c t to parametric perturbations.

The l o g a r i t h m i c - q u a d r a t i c l o s s p e n a l t y f u n c t i o n

~ ( x E, , r ) = x1

+

- r

E1x2

In(x2

-

2 2 E2x1

)-

r ln(-xl).

The g r a d i e n t o f W w i t h r e s p e c t t o x i s T

2

v Xw =

-

X2-E:Xf

S i n c e V XW = 0 a t a m i n i m i z i n g p o i n t

X(E,

r ) , w e can use

r )~ =

E

t h i s f a c t to calculate x ~ ( E ,

- l+,/l +

8 2 4E1E:2

r) =

Clearly

X(E,

2 r

r ) = [ x ~ ( E r, ) ,

~ and

~

x*(E,

x ~ ( E , r ) l T+ X ( E )

2~ 2 x (E, r) 2 1

as r

+

-.€1

+ r

0, and,

from t h e t h e o r y o f p e n a l t y f u n c t i o n s , w e know a l s o t h a t W

*

(E,

r) z W[X(E, r ) ,

From Theorem 6.6.1

E,

1-1

+ x ~ ( E )

and C o r o l l a r y 6 . 6 . 2 ,

n e a r ( 0 , 0 ) a n d r > 0 a n estimate o f V E f VEW

*

(E,

r) = V L

I

[x(E,r) ,u(E,r)

,EI

+

E ~ x ~ ( E=)

f

*

(E).

w e know t h a t f o r

*

(E)

i s g i v e n by

(E,

r)

I84


where t h e L a g r a n g i a n f o r o u r problem i s d e f i n e d as L(x, u,

+

x1

E)

E1X2

-

-

u1(x2

E;x;)

+

u2x1.

This y i e l d s VEW

*

( E r

r) =

(X2(Er

r ) , 0)

w e have t h a t

From ( 6 . 2 . 2 1 ,

-

r)(O,

U1(Er

u ~ ( E ,r) =

r/gl[x(E,

r ) , € 1 and

from t h e s o l u t i o n f o r x 2 ( ~ r, ) w e f i n d t h a t g1 E x 2 Consequently,

r/El.

* vEw

As r + O

* vEw

(E,

U ~ ( E ,r) = E

r)).

- 2 E 2 X 1 (2E ,

-

2 2 ~

=X

~

by c a l c u l a t i n g V E2 W"

(E,

r)

E

a, n d h e n c e

~

r) =

r

( E r

r)

+

a s e x p e c t e d from C o r o l l a r y 6.6.2. S i m i l a r l y , w e c a n e s t i m a t e VEf 2 " for

(E,

(E)

r ) n e a r ( 0 , 0) a n d r > 0 , by Theorem 6 . 6 . 1 a n d obtaining

Corollary 6.6.2,

where u1 = u1

(E

,

r ) and x1

[VEx(E, r ) , V € U ( E , l a t e d from ( 6 . 5 . 8 ) .

r ) l of

= x1 ( E [x(E,

,

r).

r),

U(E,

The p a r t i a l d e r i v a t i v e s

r ) ] c a n be c a l c u -

A l l r e q u i r e d q u a n t i t i e s c a n be computed

once t h e u n c o n s t r a i n e d m i n i m i z i n g p o i n t

X(E,

r ) i s known.

I85

6.8 Sensitivity Approximations for RHS Perturbations

W e do n o t c a r r y o u t t h e c a l c u l a t i o n s ,

2 * c o r o l l a r y t h a t VEW

(E,

r)

b u t we know f r o m t h e

Vgf*(E) a s r

-f

+

0.

P u r s u i n g t h e a l g o r i t h m i c r e s u l t s p a r a l l e l i n g t h o s e obt a i n e d f o r t h e r h s p e r t u r b a t i o n problem P 2 ( € ) i n Chapter 5, w e n e x t c o n s i d e r t h e a p p l i c a t i o n o f t h e f o r e g o i n g p e n a l t y funct i o n r e s u l t s t o t h i s i m p o r t a n t r e a l i z a t i o n of t h e g e n e r a l problem P j ( € 1 . 6.8.

SENSITIVITY APPROXIMATIONS FOR RHS PERTURBATIONS The p r o b l e m o f i n t e r e s t i s min

where Em + p ) ,

=

E

s.t. g ( x ) 2

f(x)

X

(E

1

,

E

2

1,

E~

=

E

~

( E ~ ,

h, ( x ) =

...,

E

m

E

2

,

P2 ( E l

and c 2 =

...

r

a n d f o r which t h e g e n e r a l r e s u l t s g i v e n i n S e c t i o n 6.5

simplify considerably. from ( 6 . 5 . 8 )

I n p a r t i c u l a r , s i n c e V 2 L = 0 , w e have EX

that

VEy(E, r ) =

where t h e A i j

are g i v e n i n ( 6 . 5 . 6 ) .

p r e c i s e a n a l o g y of All

2

= V X W [ x ( ~ r, ) ,

(5.5.1). E,

r1-l.

T h i s i s of c o u r s e t h e

I t i s shown i n S e c t i o n 6.5 t h a t

Thus,

from ( 6 . 5 . 6 1 ,

(6.8.1)

I86


where w e r e c a l l t h a t P =

[-:I]

and

where U = d i a g ( u . ) .

Q

=

[

U2

: ] I

Hence, (6.8.2)

Also, p r e c i s e l y analogous t o (5.5.2)

,

( 6 6.3)

E E,r

"I

yields

r)

where

I-$ + Q

=

7 1 QPVxW-'PTQ][ r

:].

W e c a n a l s o c a l c u l a t e t h e s e d e r i v a t i v e s i n t e r m s of V ~ X ( E ,

r ) , r e c a l l i n g t h a t ui = r / g i

[vIw(E,I:] V

U(Er

=

-$

:]

+

and G = d i a g ( g i ) , o b t a i n i n g

$ QPVEX(Er

I87

6.X Sensitivity Approxiinalions for RHS Perturbation\

V€W(E,

r ) = r1[ ( O ,

-I)

+

VXhVEx(E, r ) l .

(6.8.4)

W e may u s e t h e p e n a l t y f u n c t i o n m i n i m i z i n g p o i n t d i r e c t l y

a s i n Section 6.2

t o o b t a i n t h e same r e s u l t .

This approach,

which w e now i n d i c a t e , i s t h e o n e t h a t was implemented comput a t i o n a l l y v i a SENSUMT by Armacost and Mylande? Armacost ( 1 9 7 6 a ) .

I t a l s o y i e l d s (6.8.2)

-

( 1 9 7 3 ) a n d by

( 6 . 8 . 4 ) and i n -

volves manipulating information t h a t i s r e a d i l y a v a i l a b l e a t t h e s o l u t i o n p o i n t of t h e penalty function.

F o r problem P 2 ( E ) ,

t h e logarithmic-quadratic loss p e n a l t y function i s

i=l

j=l

and t h e g r a d i e n t t a k e n w i t h r e s p e c t t o x is

j=1

Taking p a r t i a l d e r i v a t i v e s w i t h r e s p e c t t o t h e r h s parameters E,

w e have a t

a (vxw) -aEi

X(E,

r)

rVxgi [ x ( E , r ) l [gi[x(E, r ) l

-

ci1 2,

i = 1,

...

r

m,

I88

6 Scn\iti\ity Inlorination Ubing Pcnalty Functlclns

and

a (VxW) a€. 3 +m

--

-

-Vxh.

[ X ( E r

r )1

,

r

..., p .

j = 1,

S i n c e VxW i s i d e n t i c a l l y z e r o i n a n e i g h b o r h o o d o f

r) =

(E,

( 0 , O), r > 0 , w e c a n t a k e t h e t o t a l d e r i v a t i v e o f VxW = 0

with respect t o

E

t o obtain

U s i n g t h e above r e l a t i o n s h i p s , w e o b t a i n

[gi[x(E, r ) l

which i s e x a c t l y ( 6 . 8 . 2 ) . i f i # k.

-

ci1

... ...

2

= 1 i f

D e f i n e 6ik

W e know t h a t

a[gi[x(Er r ) l

-

E ~ ] / =~ -cSik, E ~

i = 1,

k = 1,

a [ h 7. [ x ( E , r ) l - E ~ + ~ I / ~ =E ,- &m,k'

Vxh. T

-

E ~ =I VXqi[x(E, r ) l , i

VX[hj[x(€, r ) ]

-

E ~ += ~V ]h . [ X ( E r r ) l ,

u ~ ( E r, ) = r / [ g i [ x ( E ,

X

..., m ,

..., m

r)l - ~ i l r i

w.(E, r ) = [ h j [ x ( E , r ) ] 7

-

=

+ p;

..., p , ..., m

1,

...

~ ~ + ~ l j/ =r ,1,

r

+

p;

...

= 1,

m;

r

...

j = 1,

I

..]

i = k a n d €iik = 0

j = 1,

r)]

r )1

r

k = 1,

VX[qi[x(E,

[X(Er

r

m;

...

r

P.

p;

I89

6.X Sensitivity Approximations lor RHS Pcrturbationh

Using a l l of t h e s e r e l a t i o n s h i p s , w e o b t a i n

aw.

(E,

r)

(x(E,

aEk i = 1,

..., m ,

r))aX r ,r a E(kE

j = 1,

..., p t

-

6j+m,k

1

'

k = 1, . . . r

m + P.

Using t h e n o t a t i o n i n t r o d u c e d a b o v e , w e o b t a i n V E u ( ~ r, ) = r G - 2 [(I, 0 )

-

VX9VEX(Er

r)]

and

which a r e e x a c t l y ( 6 . 8 . 3 ) a n d ( 6 . 8 . 4 ) . The matrices d e t e r m i n e d by ( 6 . 8 . 3 ) a n d t h e n e g a t i v e o f ( 6 . 8 . 4 ) y i e l d a n a p p r o x i m a t i o n o f t h e H e s s i a n of t h e o p t i m a l

value function f right-hand

*

(E)

w i t h r e s p e c t t o t h e components of t h e

s i d e of t h e c o n s t r a i n t s f o r p r o b l e m P 2 ( ~ ) . T h i s

f o l l o w s d i r e c t l y from c o n c l u s i o n ( f ) o f Theorem 5 . 4 . 1 a n d Corollary 6 . 2 . 2 .

T h i s r e s u l t may a l s o b e s e e n by r e c a l l i n g 2 *

from C o r o l l a r y 6 . 6 . 2 t h a t VEW*(€, r ) + V E f * ( c ) and V E W ( E , r) 2 * + Since it i s readily calculated t h a t + V f (E) a s r 0 E -+

*

VEW ( E ,

and

r) =

.

I90

6 Sensitivity Informalion Using Penalty Functions

we conclude t h a t lim

r+O+

and

lim

r+O+

vE‘

-vEw

E, E,

r)

=

r)

2 *

Vcf

(E)

=

VEU(E) [-vEw

(E)

as state 6.9.

RECAPITULATION In Chapters 2

-

5 , t h e o r e t i c a l and c o m p u t a t i o n a l a s p e c t s

of s e n s i t i v i t y a n a l y s i s have been d e v e l o p e d i n p a r a l l e l .

The

t h e o r e t i c a l r e s u l t s have been shown t o be e a s i l y i m p l e m e n t a b l e computationally v i a a penalty function technique.

In the

example above, t h e e s t i m a t e s by way of a p e n a l t y f u n c t i o n were d e r i v e d a n a l y t i c a l l y ; however, i n p r a c t i c e , w e o b t a i n n u m e r i c a l e s t i m a t e s f o r a d e c r e a s i n g s e q u e n c e o f p o s i t i v e v a l u e s of r. I t r e q u i r e s o n l y a modest e f f o r t t o modify t h e u s u a l p e n a l t y

f u n c t i o n a l g o r i t h m t o o b t a i n t h e r e s u l t s d e v e l o p e d above. The r e s u l t s o f C h a p t e r s 3

-

5 are a l g o r i t h m - i n d e p e n d e n t .

They r e q u i r e o n l y t h e c o n d i t i o n s o f Theorem 3 . 2 . 2 . once a Karush-Kuhn-Tucker

Clearly,

t r i p l e i s known, o b t a i n i n g t h e

gradient of t h e optimal value function is a t r i v i a l matter c o m p u t a t i o n a l l y , r e q u i r i n g o n l y t h e e v a l u a t i o n of t h e p a r t i a l d e r i v a t i v e o f t h e Lagrangian w i t h r e s p e c t t o t h e parameters. I f t h e g r a d i e n t o f t h e Karush-Kuhn-Tucker

t r i p l e i s known,

t h e n t h e c a l c u l a t i o n of t h e H e s s i a n o f t h e o p t i m a l v a l u e funct i o n i s a l s o a r e l a t i v e l y s i m p l e matter u s i n g ( 3 . 4 . 5 ) . The c e n t r a l d i f f i c u l t y a r i s e s i n o b t a i n i n g t h e g r a d i e n t o f t h e Karush-Kuhn-Tucker

t r i p l e f o r t h e g e n e r a l case.

A general

191

6.9 Recapitulation

e x p r e s s i o n was g i v e n i n ( 4 . 2 . 1 5 )

i n t e r m s of t h e inverse of

~ ( E IT .h r e e d i f f e r e n t sets o f e q u a t i o n s ( 4 . 2 . 8 ) , (4.2.13),

lem.

(4.2.91,

or

a r i s e , d e p e n d i n g o n t h e c h a r a c t e r i s t i c s o f t h e probe x i s t s o r when t h e r e a r e n l i n e a r l y independ-

When V;L-’

ent constraints,

(4.2.8)

and ( 4 . 2 . 9 )

may b e implemented t o

p e r m i t t h e c a l c u l a t i o n o f V E y ( € ) w i t h modest e f f o r t .

However,

when n e i t h e r o f t h e s e c o n d i t i o n s a p p l i e s , t h e e f f o r t i n v o l v e d

t o c o n s t r u c t t h e matrices r e q u i r e d i n ( 4 . 2 . 1 3 )

i s much g r e a t e r .

N o n e t h e l e s s , t h e p r o c e d u r e d e s c r i b e d by t h e e q u a t i o n s i n (4.2.13)

i s much more e f f i c i e n t t h a n o n e s i m p l y i n v o l v i n g t h e

inversion of

G(E)

t h a t does not e x p l o i t t h e r e l a t i o n s

stipulated i n the optimality conditions. W e h a v e shown t h a t p e n a l t y f u n c t i o n methods e a s i l y p r o v i d e

the desired estimates.

The g r a d i e n t o f t h e p e n a l t y f u n c t i o n

i s t h e estimate o f t h e g r a d i e n t o f t h e o p t i m a l v a l u e f u n c t i o n . S i m i l a r l y , t h e H e s s i a n o f t h e p e n a l t y f u n c t i o n i s a n estimate of t h e H e s s i a n o f t h e o p t i m a l v a l u e f u n c t i o n a n d i s e a s i l y c a l c u l a t e d when a n estimate o f t h e g r a d i e n t o f t h e KarushKuhn-Tucker known. M(E,

t r i p l e taken with respect t o t h e parameters i s

A t h e o r y f o r o b t a i n i n g VEy(€, r ) w a s developed using

r ) , by way of s t r a i g h t f o r w a r d m a n i p u l a t i o n o f ( 6 . 2 . 8 ) .

However, a more e f f i c i e n t a l t e r n a t i v e a p p r o a c h u s i n g E q . (6.2.15)

is a v a i l a b l e , i f t h e given penalty function algorithm

i s being u t i l i z e d . the solution of

Armacost a n d Mylander ( 1 9 7 3 ) implemented

(6.2.15)

t o obtain V€X(E,

r ) , u s i n g t h e SUMT

V e r s i o n 4 computer c o d e w i t h t h e l o g a r i t h m i c - q u a d r a t i c l o s s penalty function. (6.2.21)

I t i s t h e n a t r i v i a l m a t t e r t o implement

and 6.2.22)

t o o b t a i n V E u ( € , r ) and V € W ( E ,

r ) , thus

o b t a i n i n g a n estimate o f t h e g r a d i e n t o f t h e Karush-Kuhn-Tucker

192


t r i p l e , a n d a l l o w i n g f o r e a s y c a l c u l a t i o n o f a n estimate of t h e Hessian of t h e o p t i m a l v a l u e f u n c t io n . S e v e r a l a p p l i c a t i o n s of t h e s e r e s u l t s have been mentioned. W e r e c a p i t u l a t e h e r e and d i s c u s s s e v e r a l o t h e r p o t e n t i a l ap-

plications.

The estimate o f t h e s e n s i t i v i t y o f t h e Karush-

Kuhn-Tucker t r i p l e g i v e s a measure o f t h e s t a b i l i t y o f t h e solution point.

I n a d d i t i o n , t h i s i n f o r m a t i o n c a n be u s e d t o

g i v e a f i r s t - o r d e r esti.mate o f Karush-Kuhn-Tucker

t r i p l e s of

problems i n v o l v i n g d i f f e r e n t v a l u e s o f t h e parameters, once one s u c h t r i p l e h a s b e e n d e t e r m i n e d . s i t i v i t y of t h e Karush-Kuhn-Tucker

An estimate o f t h e s e n -

t r i p l e i s necessary t o

o b t a i n t h e Hessian of t h e o p t i m a l v a l u e f u n ctio n . The estimates o f t h e s e n s i t i v i t y o f t h e o p t i m a l v a l u e f u n c t i o n c a n b e u s e d t o o b t a i n f i r s t - and s e c o n d - o r d e r e s t i mates o f t h i s f u n c t i o n , u s i n g a t r u n c a t e d T a y l o r ' s s e r i e s . This s e n s i t i v i t y information f o r t h e optimal value function and t h e Karush-Kuhn-Tucker

t r i p l e may b e u s e d i n a c y c l i c

p r o c e d u r e f o r s o l v i n g P 3 ( € ) , where t h e p a r a m e t e r s c o r r e s p o n d t o a s u b s e t o f t h e variables.

F o r problem P 2 ( € ) , where t h e

parameters are on t h e right-hand

s i d e of t h e c o n s t r a i n t s , such

a procedure u s i n g t h e f i r s t - and second-order

sensitivity

i n f o r m a t i o n may have a p p l i c a t i o n i n s o l v i n g p r o b l e m s by r h s d e c o m p o s i t i o n , a well-known

technique.

idea i s given i n Chapter 5, Section 5.6. p a r a m e t e r i z e d on t h e r i g h t - h a n d

An example o f t h i s Problems

s i d e o f t h e c o n s t r a i n t s are

i n t i m a t e l y i n v o l v e d i n much of d u a l i t y t h e o r y and t h i s s e n s i t i v i t y i n f o r m a t i o n may h a v e a p p l i c a t i o n i n a l g o r i t h m s s o l v i n g P ( E ) using various duals.

193

6.9 Recapitulation

While w e h a v e c o n c e n t r a t e d on p a r a m e t r i c s e n s i t i v i t y a n a l y s i s and d e v e l o p e d n e i g h b o r h o o d r e s u l t s , many of t h e s e r e s u l t s e x t e n d t o p a r a m e t r i c n o n l i n e a r programming, where t h e param-

eters are p e r m i t t e d t o range i n a p r e s c r i b e d set.

This w i l l

u n d o u b t e d l y be a s u b j e c t of c o n t i n u i n g f u t u r e i n v e s t i g a t i o n s .

Chapter 7

Calculation of Sensitivity Information Using Other Algorithms

7.1.

INTRODUCTION

I n t h e p r e v i o u s c h a p t e r i t i s shown t h a t a l l t h e s e n s i t i v i t y i n f o r m a t i o n t h a t was developed i n P a r t I1 o f t h i s book, namely t h e f i r s t and second p a r a m e t e r d e r i v a t i v e s o f t h e o p t i m a l v a l u e f u n c t i o n and t h e f i r s t p a r a m e t e r d e r i v a t i v e s of a Karush-Kuhn-Tucker

t r i p l e , can be e s t i m a t e d u s i n g informa-

t i o n t h a t i s a v a i l a b l e i n t h e u s u a l a p p l i c a t i o n of a s o l u t i o n algorithm.

The a n a l y s i s t h e r e c o n c e n t r a t e s on an i m p o r t a n t

c l a s s o f a l g o r i t h m s b a s e d on c l a s s i c a l p e n a l t y f u n c t i o n s and

i s s p e c i a l i z e d t o p e r h a p s t h e most p o p u l a r and most h e a v i l y u t i l i z e d r e p r e s e n t a t i v e of t h i s c l a s s , t h e logarithmicq u a d r a t i c mixed b a r r i e r - p e n a l t y f u n c t i o n ( 6 . 2 . 1 ) .

However, i t

was n o t e d t h a t a l g o r i t h m s g e n e r a l l y use and g e n e r a t e informat i o n t h a t i s r e l e v a n t t o t h e c a l c u l a t i o n of s e n s i t i v i t y estimates

.

W e support t h i s f a c t i n t h i s chapter, indicating s p e c i f i c

and e x p l o i t a b l e c o n n e c t i o n s w i t h t y p i c a l l y a v a i l a b l e a l g o r i t h m i c i n f o r m a t i o n and b r i e f l y p u r s u i n g t h e s e f o r s e v e r a l i m p o r t a n t c l a s s e s of a l g o r i t h m s , i n c l u d i n g Newton, p r o j e c t e d - g r a d i e n t , reduced-gradient,

and augmented Lagrangian methods.

I94

Before

195

7.1 Introduction

doing so, w e d i g r e s s b r i e f l y t o s u p p o r t t h e i d e a t h a t s o l u t i o n a l g o r i t h m c a l c u l a t i o n s may o f t e n be d i r e c t l y r e l a t e d t o s e n s i t i v i t y and s t a b i l i t y estimates. I t i s a p p a r e n t t h a t t h e c o n n e c t i o n between s o l u t i o n algo-

r i t h m c a l c u l a t i o n s and s e n s i t i v i t y c a l c u l a t i o n s may b e cons i d e r a b l y d e e p e r t h a n t h e f a c t t h a t t h e s e c a l c u l a t i o n s simply r e q u i r e common i n f o r m a t i o n .

Solution algorithms t y p i c a l l y

s o l v e o r e s t i m a t e a s o l u t i o n o f a sequence o f subproblems o f t h e g i v e n problem whose s o l u t i o n s "converge" i n some s e n s e t o t h e g i v e n problem.

The subproblems may o f t e n be viewed a s

p e r t u r b a t i o n s o f t h e o r i g i n a l problem, a s o l u t i o n o f which would t h e n presumably be a l i m i t of s o l u t i o n s o f an a s s o c i a t e d sequence o f t h e p e r t u r b e d problems.

Thus, once t h e l a t t e r

sequence o f problems h a s been i d e n t i f i e d , a p r e d i c t i o n of a s o l u t i o n t o t h e n e x t problem, g i v e n a s o l u t i o n t o t h e problem a t hand, might be p r o v i d e d by a p e r t u r b a t i o n a l a n a l y s i s o f t h e problem a t hand and e x t r a p o l a t i o n .

Thus, i t seems p l a u s i b l e

t h a t such an a l g o r i t h m might be r a t h e r c o m p l e t e l y c h a r a c t e r i z e d i n t e r m s of e s t i m a t e s d e r i v e d from a n a p p r o p r i a t e s e n s i t i v i t y o r s t a b i l i t y a n a l y s i s of t h e g i v e n problem. I n f a c t , i m p o r t a n t f a m i l i e s o f a l g o r i t h m s a r e known t o d e f i n e s o l u t i o n t r a j e c t o r i e s o r s e q u e n c e s o f an a s s o c i a t e d f a m i l y o f subproblems t h a t have been a n a l y z e d v i a s e n s i t i v i t y and s t a b i l i t y t e c h n i q u e s .

Such r e s u l t s have been r e p o r t e d by

Meyer ( 1 9 7 0 ) , F i a c c o and McCormick (19681, and Robinson ( 1 9 7 4 ) , i n t h e f i r s t i n s t a n c e t o a n a l y z e and p r o v e convergence, i n t h e second t o c h a r a c t e r i z e and a c c e l e r a t e convergence, and i n t h e t h i r d t o e s t a b l i s h t h e r a t e o f convergence. Section 6.2,

F o r example, i n

w e n o t e d t h a t t h e p e n a l t y f u n c t i o n method u t i l i z e d

196

7 Calculation of Sensitivity Using Other Algorithms

i n Chapter 6 could be l o c a l l y completely c h a r a c t e r i z e d i n

t e r m s o f s o l u t i o n s of a p e r t u r b a t i o n of t h e Karush-Kuhn-Tucker c o n d i t i o n s , t h u s immediately p l a c i n g t h e p e n a l t y f u n c t i o n minimization i n correspondence w i t h a s o l u t i o n of a p er tu r b at i o n of t h e g i v e n problem.

This correspondence w a s f i r s t

r e p o r t e d by F i a c c o and McCormick ( 1 9 6 8 ) . Thus, t h e f a c t t h a t s e n s i t i v i t y a n d s o l u t i o n a l g o r i t h m a n a l y s i s may c o n c e r n t h e same p r o b l e m - p e r t u r b a t i o n

solution

t r a j e c t o r y h a s been o b s e r v e d a n d e x p l o i t e d h e r e and e l s e w h e r e . I n c o n c l u s i o n , w e n o t e s t i l l a n o t h e r i n t e r e s t i n g correspondence. Suppose f r C 2 and w e w i s h t o s o l v e Vf ( x ) = 0 , g i v e n such t h a t Vf(F) =

F #

2

0 and s u p p o s e V f ( F ) - l e x i s t s .

x

I t is

n o t d i f f i c u l t t o show t h e f o l l o w i n g e q u i v a l e n c e between s e n s i S t a r t i n g a t K, t h e

t i v i t y a n a l y s i s a n d Newton's method:

Newton estimate ( i . e . , t h e f i r s t i t e r a t e ) o f a s o l u t i o n o f (Taylor's series)

V f ( x ) = 0 i s t h e same as t h e f i r s t - o r d e r e s t i m a t e o f x ( 0 ) a t x ( S ) = X, where s o l u t i o n o f V ~ [ X ( E )= ] E n e a r

E

X(E)

= 7.

E

C1

is t h e unique

Thus, any i t e r a t i o n o f

Newton's a l g o r i t h m i s t h e s a m e a s a n e x t r a p o l a t i o n b a s e d on a p a r a m e t r i c s e n s i t i v i t y a n a l y s i s , a n d h e n c e Newton's a l g o r i t h m may be c h a r a c t e r i z e d i n t h i s s e t t i n g . 7.2.

CONNECTIONS BETWEEN ALGORITHMIC AND SENSITIVITY CALCULATIONS The m o s t o b v i o u s and c o m p e l l i n g way t o see t h e c o n n e c t i o n

between t h e v a l u e s a n d t h e p a r a m e t e r d e r i v a t i v e s o f t h e o p t i -

m a l v a l u e f u n c t i o n o r a Karush-Kuhn-Tucker

t r i p l e and algo-

r i t h m i c information i s through t h e d e r i v a t i o n o f t h e s e quantities.

Though v a r i o u s s u f f i c i e n t c o n d i t i o n s h a v e b e e n

i n v o k e d t o p r o v i d e e x i s t e n c e p r o o f s and t h e d e t e r m i n a t i o n of

197

7.2 Algorithmic and Sensitivity Calculations

e x p l i c i t f o r m u l a s and c o n t i n u i t y p r o p e r t i e s , n o t e t h a t c r u c i a l s e n s i t i v i t y measurements a r e b a s e d o n t h e s a t i s f a c t i o n o f necessary c o n d i t i o n s c h a r a c t e r i z i n g o p t i m a l i t y .

This leads t o

s e v e r a l r e l e v a n t connections. Assuming d i f f e r e n t i a b i l i t y o f t h e problem f u n c t i o n s and any o f a v a r i e t y o f w e l l known a n d t y p i c a l l y i n v o k e d c o n s t r a i n t q u a l i f i c a t i o n s (see S e c t i o n 2 . 3 ) , any v a l i d a l g o r i t h m a p p l i e d

*

to P2(0) w i l l , i f successful, generally yield a t r i p l e (x

u

*

,w

*

)

,

s a t i s f y i n g t h e f i r s t - o r d e r necessary conditions ( 3 . 2 . 2 ) I t i s w e l l known a n d h a s b e e n amply d e m o n s t r a t e d

for P2(0).

h e r e i n i n Chapter 5 and elsewhere t h a t t h e o p t i m a l m u l t i -

*

p l i e r s (u

,w

*

)

are i n t i m a t e l y r e l a t e d t o t h e r a t e o f change

of t h e o p t i m a l v a l u e f u n c t i o n w i t h r e s p e c t t o c o n s t r a i n t perturbations.

F o r example, u n d e r t h e a s s u m p t i o n s o f C o r o l l a r y * T * T 3 . 4 . 4 o r Theorem 5.4.1, w e know t h a t V E f * ( 0 ) = [ ( u ) , (-w 1 1.

F o r t h e g e n e r a l problem P 3 ( 0 ) a n d u n d e r t h e a s s u m p t i o n s of Theorem 3 . 2 . 2 ,

*

VEf (0) would o f c o u r s e b e g i v e n by ( 3 . 4 . 4 1 ,

*

which c a n be c a l c u l a t e d o n c e ( x

,

u

*

,w

*

)

i s known.

i s t h a t t h i s s e n s i t i v i t y i n f o r m a t i o n , namely u

*

and w

The p o i n t

*

,

is not

m e r e l y a v a i l a b l e b u t i s o f t e n e x p l i c i t l y required t o d e t e r m i n e whether a c a n d i d a t e s o l u t i o n has been o b t a i n e d .

Thus, t h e

c o n n e c t i o n between a l g o r i t h m i c c a l c u l a t i o n s a n d s e n s i t i v i t y analysis is inevitable. T u r n i n g o u r a t t e n t i o n t o t h e p a r a m e t e r d e r i v a t i v e s VEy ( € 1 o f a Karush-Kuhn-Tucker

t r i p l e [ x ( E ),

U(E)

, w ( E ) ]of

P3(€)

We

recall t h a t M ( E ) V ~ Y ( E )= N(E)

where M a n d -N

(3.2.3)

are, r e s p e c t i v e l y , t h e Jacobians of t h e first-

o r d e r n e c e s s a r y c o n d i t i o n s ( 3 . 2 . 2 ) o f P 3 ( ~ ,) w i t h r e s p e c t t o

198


( x , u, w ) and

E.

S i n c e M and N i n v o l v e o n l y t h e v a l u e s o r

p a r t i a l d e r i v a t i v e s o f t h e o r i g i n a l problem f u n c t i o n s and t h e multipliers,

i t i s a p p a r e n t t h a t M ( 0 ) and N ( 0 ) a r e a l s o com-

*

p l e t e l y d e t e r m i n e d , once ( x

* * ,u ,w )

i s known.

I n view o f

( 3 . 2 . 3 ) and o u r a s s u m p t i o n s , t h i s i n f o r m a t i o n c o m p l e t e l y de-

t e r m i n e s V E y ( 0 ) . T h e r e f o r e , any v a l i d a l g o r i t h m a p p l i e d t o P 3 ( 0 ) c a n p r o v i d e an e s t i m a t e of

*

(x

,

u

*

,

w

*

),

i f successful,

and t h i s can b e used i n ( 3 . 2 . 3 ) t o c a l c u l a t e an estimate o f 2 * V E y ( ~ a) t E = 0 . S i n c e t h e c a l c u l a t i o n of VEf (0) ( 3 . 4 . 5 ) c a n a l s o be made once VEy(0) i s a v a i l a b l e , t h i s means t h a t any p r o c e d u r e t h a t can e s t i m a t e a Karush-Kuhn-Tucker u

*

,

w

*

),

i.e.,

*

t r i p l e (x

,

s a t i s f y t h e f i r s t - o r d e r necessary conditions

( 3 . 2 . 2 ) f o r a l o c a l s o l u t i o n , provides s u f f i c i e n t information

t o estimate a l l t h e s o l u t i o n parameter-derivative t h a t w e have p r e s e n t e d .

calculations

Summarizing, t h e s o l u t i o n s e n s i t i v i t y

c a l c u l a t i o n s a r e b a s e d on t h e s o l u t i o n i n f o r m a t i o n t h a t a s o l u t i o n algorithm i s t y p i c a l l y g e n e r a l l y endeavoring t o estimate, under t h e g i v e n assumptions. A c t u a l l y , t h e c o n n e c t i o n i s c o n s i d e r a b l y d e e p e r and more e x p l i c i t , as may r e a d i l y b e e s t a b l i s h e d .

The f i r s t - o r d e r

necessary c o n d i t i o n s ( 3 . 2 . 2 ) f o r a l o c a l s o l u t i o n of P ( 0 ) a r e system o f e q u a t i o n s i n y = ( x , u , w )

T

.

A w e l l known t e c h n i q u e

f o r s o l v i n g a system o f equations, p a r t i c u l a r l y a t t r a c t i v e because of e x c e l l e n t convergence p r o p e r t i e s

(under t h e given

c o n d i t i o n s and when a good e s t i m a t e o f a s o l u t i o n i s a v a i l a b l e ) ,

i s Newton's method.

L e t t i n g F = (VL, u 1g 1,

..., umgml h l , ...,

AsT I it f o l l o w s t h a t ( 3 . 2 . 2 ) i s e q u i v a l e n t t o F = 0 . hP) * suming t h a t yk n e a r a s o l u t i o n y i s a v a i l a b l e , t h e n e x t

I99

7.2 Algorithmic and Sensitivity Calculations

v e c t o r yk+l d e t e r m i n e d by Newton's method i s Yk+l = Yk

-

(7.2.1)

MilFk

where t h e s u b s c r i p t d e n o t e s e v a l u a t i o n a t y = y k ,

E

= 0 , and

t h e matrix M is t h e usual Jacobian of F with respect t o (x, u, w)

,

t h e same m a t r i x a p p e a r i n g i n t h e e q u a t i o n ( 3 . 2 . 3 ) t h a t

d e t e r m i n e s V E y ( ~ ) . R e c a l l i n g t h a t M ( 0 ) i s n o n s i n g u l a r under o u r a s s u m p t i o n s (Theorem 3.2.21,

w e a r e t a c i t l y assuming t h a t

yk is c l o s e enough t o y* t o a s s u r e t h a t systems o f e q u a t i o n s ,

q1e x i s t s .

( 3 . 2 . 3 ) and ( 7 . 2 . 1 ) ,

Both

may b e w r i t t e n i n

t h e form

Mz = b

(7.2.2)

where z i s t h e v a r i a b l e .

C o n s e q u e n t l y , any t h e o r e t i c a l re-

s u l t s o r numerical techniques f o r analyzing o r s o l v i n g (7.2.2)

w i l l be a p p l i c a b l e t o s o l v i n g ( 3 . 2 . 3 ) ,

and v i c e - v e r s a .

In

p a r t i c u l a r , it i s e v i d e n t t h a t an a n a l y s i s o f t h e s t r u c t u r e o f M and o f e f f i c i e n t t e c h n i q u e s f o r s o l v i n g (7.2.2)

becomes a

common c o n c e r n o f any a l g o r i t h m i c p r o c e d u r e s b a s e d on i m i t a t i n g (7.2.1)

and of any n u m e r i c a l t e c h n i q u e s f o r c a l c u l a t i n g

t h e s o l u t i o n p a r a m e t e r d e r i v a t i v e s V E y ( ~ from ) (3.2.3).

For

example, t h e d e c o m p o s i t i o n s o f M g i v e n i n C h a p t e r s 4 and 5 a r e r e l e v a n t t o any a l g o r i t h m b a s e d on ( 7 . 2 . 1 ) . s o l u t i o n a l g o r i t h m b a s e d on (7.2.1) solve (7.2.2)

C o n v e r s e l y , any

w i l l a l r e a d y be geared t o

and hence w i l l be a b l e t o s o l v e (3.2.31,

and

most i m p o r t a n t l y , t h e same m a t r i x would be i n v o l v e d i n making a given e s t i m a t e .

This f a c t i s obviously e x p l o i t a b l e , both

t h e o r e t i c a l l y and c o m p u t a t i o n a l l y .

F o r example, it g r e a t l y

e a s e s t h e c o d i n g , s t o r a g e , and d a t a p r o c e s s i n g r e q u i r e m e n t s needed t o add t h e s e n s i t i v i t y a n a l y s i s c a p a b i l i t y t o any

e x i s t i n g NLP code b a s e d on v a r i a n t s o f ( 7 . 2 . 1 ) .

200


The i d e a o f u s i n g ( 7 . 2 . 1 )

a s a b a s i s f o r an a l g o r i t h m , o r

a s a " c l o s i n g t e c h n i q u e , " i n t h e f i n a l s t a g e s o f a n y NLP a l g o r i t h m i s a t t r a c t i v e and p o p u l a r .

E f f i c i e n t NLP a l g o r i t h m s

must u t i l i z e second-order i n f o r m a t i o n and must u l t i m a t e l y comp e t e w i t h t h e performance o f t h e i t e r a t i v e p r o c e s s ( 7 . 2 . 1 ) . I n d e e d , t h e second-order a l g o r i t h m s o f t e n a p p r o x i m a t e t h i s p r o c e s s a s a s o l u t i o n i s approached.

S t r o n g and e x p l o i t a b l e

c o n n e c t i o n s t h u s e x i s t between t h e s e n s i t i v i t y a n a l y s i s c a l c u l a t i o n s and t h e most e f f e c t i v e NLP a l g o r i t h m s . Thus, i t would a p p e a r t h a t e s t i m a t e s o f t h e m a t r i x M and t h e p r o c e s s d e f i n e d by ( 7 . 2 . 1 )

a r e i n e v i t a b l y involved i n t h e

c a l c u l a t i o n s of second-order algorithms, a t l e a s t n e a r a solution.

C l e a r l y , e s t i m a t e s o f a Karush-Kuhn-Tucker

triple are

e s s e n t i a l and s u f f i c e t o p r o v i d e e s t i m a t e s o f t h e c o n s t i t u e n t s of M near a s o l u t i o n of t h e necessary conditions ( 3 . 2 . 2 ) . Algorithms d i f f e r i n how t h e d a t a t h e y m a n i p u l a t e a r e o r g a n i z e d and how t h e e s t i m a t e s o f t h e b i n d i n g c o n s t r a i n t s and t h e Lagrange m u l t i p l i e r s a r e made.

This l e a d s t o d i f f e r e n t repre-

s e n t a t i o n s o f t h e m a t r i x M and i t s i n v e r s e , f o r d i f f e r e n t algorithms.

W e have a l r e a d y s e e n one s u c h r e p r e s e n t a t i o n ,

f a c t , given i n (6.5.6) mixed b a r r i e r - p e n a l t y

in

f o r t4-l from t h e l o g a r i t h m i c - q u a d r a t i c function algorithm.

I n the next

s e c t i o n , w e i n v e s t i g a t e t h e s t r u c t u r e o f M-1 f o r a g e n e r a l a l g o r i t h m i c s t r a t e g y t h a t i n c l u d e s a s r e a l i z a t i o n s two a d d i t i o n a l important c l a s s e s of algorithms, projected-gradient reduced-gradient algorithms.

and

20 I

7.3 Inverse of the Jacobian of the Karush-Kuhn-Tucker System

ALGORITHMIC CALCULATIONS O F THE INVERSE O F THE JACOBIAN OF THE KARUSH-KUHN-TUCKER SYSTEM

7.3.

I n C h a p t e r 4, f o r m u l a s f o r t h e b l o c k components o f t h e i n v e r s e o f t h e J a c o b i a n M o f t h e f i r s t - o r d e r n e c e s s a r y condit i o n s (3.2.2)

f o r a s o l u t i o n of t h e g e n e r a l problem P ( E ) w e r e

o b t a i n e d , i n terms o f t h e f u n c t i o n s and d e r i v a t i v e s o f t h e f u n c t i o n s d e f i n i n g t h e problem.

T h i s was done i n t h e c o u r s e

of developing expressions f o r t h e s o l u t i o n V € Y ( E )

of t h e

system M(E)V~Y(E) = N(E)

(3.2.3)

t h a t r e s u l t e d from t h e d i f f e r e n t i a t i o n of t o t h e parameter

E

(3.2.2) w i t h r e s p e c t

a t t h e Karush-Kuhn-Tucker

t r i p l e ~ ( € 1 . In

C h a p t e r 4 i t was n o t e d t h a t t h e e l i m i n a t i o n o f nonbinding cons t r a i n t i n f o r m a t i o n and an a p p r o p r i a t e f a c t o r i z a t i o n l e d t o a r e d u c t i o n (4.2.4)

of

(3.2.31,

so t h a t i t was a c t u a l l y o n l y

n e c e s s a r y t o c a l c u l a t e t h e b l o c k components A i j of a reduced m a t r i x

h ( ~d)e r i v e d

of t h e inverse

from M ( E ) and g i v e n i n (4.2.6).

I n t h i s s e c t i o n w e give expressions f o r t h e Aij

in terms

of a l g o r i t h m i c c a l c u l a t i o n s a s s o c i a t e d with procedures based on t h e s t r a t e g y o f m a i n t a i n i n g t h e s a t i s f a c t i o n , up t o a f i r s t o r d e r e s t i m a t e , o f t h e c o n s t r a i n t s t h a t a r e c u r r e n t l y deemed binding.

W e f i r s t g i v e g e n e r a l r e s u l t s , f o l l o w e d by s p e c i a l i -

z a t i o n s t o r e d u c e d - g r a d i e n t and p r o j e c t e d - g r a d i e n t c a l c u l a tions.

W e f o l l o w t h e development g i v e n p r e v i o u s l y by t h e

author (Fiacco, 1980a).

W e c o n f i n e o u r a t t e n t i o n t o t h e Case

4 s i t u a t i o n of C h a p t e r 4, S e c t i o n 4.2, where t h e number of b i n d i n g i n e q u a l i t y c o n s t r a i n t s r and e q u a l i t y c o n s t r a i n t s p i s

less t h a n t h e number of v a r i a b l e s n.

202


T h i s i s t h e l e a s t s t r u c t u r e d , and hence m o s t g e n e r a l , s i t u a t i o n t h a t c a n be e n c o u n t e r e d u n d e r t h e g i v e n a s s u m p t i o n s . Many r e p r e s e n t a t i o n s of d a t a are o r g a n i z e d .

fi-'

a r e p o s s i b l e , d e p e n d i n g on how t h e

However, a g e n e r a l r e p r e s e n t a t i o n t h a t i s

t a i l o r e d t o t h e a s s u m p t i o n s w e are making h e r e w a s o b t a i n e d by McCormick ( 1 9 7 7 ) and w i l l s e r v e o u r p u r p o s e e x t r e m e l y a p t l y . W e f o l l o w h i s development h e r e r a t h e r c l o s e l y .

F i r s t , n o t e t h a t t h e second-order s u f f i c i e n t c o n d i t i o n s and s t r i c t complementary s l a c k n e s s , a s s u m p t i o n s ( i i )and ( i v )

*

o f Theorem 3.2.2,

imply t h a t , a t ( x

S i s any n by

VxhTIT, a n ( r

[n

-

u

*

,w

2 zTVxLz > 0 for all z

must have (4.2.10) : where P = [-Vx7jT,

,

+

*

1, with

we

E = 0,

# 0 s u c h t h a t Pz

p ) by n m a t r i x .

= 0,

Hence, i f

(r + p ) ] matrix t h a t generates t h e n u l l

s p a c e o f P , t h e n z = Sv f o r some v i n E n - ( r + P )

implies t h a t

Pz = 0 ( s i n c e PS = 0) and a l s o t h a t z # 0 i f v # 0 ( s i n c e w e

are t a c i t l y assuming t h a t S h a s f u l l column r a n k [ n Thus, zTV2Lz = v TSTV 2 LSv > 0 , p r o v i d i n g t h a t v # 0. X

T 2 c l u d e t h a t D = S V LS i s a p o s i t i v e d e f i n i t e [n

[n

X

-

(r

+

p ) ] matrix.

-

Further, s i n c e P has rank r

-

(r

+

( r + PI]).

W e con-

+ p ) ] by p , by t h e

l i n e a r i n d e p e n d e n c e a s s u m p t i o n , a s s u m p t i o n (iii) o f Theorem 3.2.2,

an n by ( r

rank ( r

+

+

p ) pseudo-inverse

p ) m a t r i x P'

s a t i s f y i n g PP'P

of P e x i s t s , i.e., = P.

a

These c o n s t r u c t s

and o b s e r v a t i o n s l e a d t o a g e n e r a l r e p r e s e n t a t i o n of t h e b l o c k components

of

2-l.

I t i s assumed t h a t P# and S a r e g e n e r a t e d by some m a t r i x

t e c h n i q u e which r e l a t e s t h e s e q u a n t i t i e s by t h e e x p r e s s i o n I

-

P'P

=

s i n c e PP'

SW, where W i s some [ n = I

-

and Vxf(x*, O ) T = -P

(r T

+

p ) ] by n m a t r i x .

* ( u , w * ) a~ t o p t i m a l i t y

Also,

1.3 Inverse of the Jacobian of the Karush-Kuhn-Tucker System

203

it may be o f i n t e r e s t t o n o t e t h a t

This m o t i v a t e s t h e widely used e s t i m a t i o n

- (P#)TVxfTf o r

the

Lagrange m u l t i p l i e r s i n a l g o r i t h m s i n v o l v i n g t h e s e c o n s t r u c t s . of

The r e s u l t , i n t h e form o f t h e b l o c k components A i j

h-’,

f o l l o w s r e a d i l y a n d i s g i v e n by -1 T

,

All

= SD

S

A12

= Afl

= [I

A 2 2 = -A

V 2 LP # 21 x

-

(7.3.1)

A~~v:LJP#,

.

T h e r e a r e many good t e c h n i q u e s c u r r e n t l y a v a i l a b l e f o r c a l c u l a t i n g S a n d P # , m o t i v a t e d by v a r i o u s n u m e r i c a l e f f i c i e n c y , s t a b i l i t y and a l g o r i t h m i c c o n s i d e r a t i o n s .

W e mention

two h e r e f o r c o m p l e t e n e s s and b e c a u s e t h e y a r e p r e c i s e l y t a i l o r e d t o t h e c a l c u l a t i o n s a s s o c i a t e d w i t h two i m p o r t a n t f a m i l i e s o f m a t h e m a t i c a l programming a l g o r i t h m s , reducedg r a d i e n t and p r o j e c t e d - g r a d i e n t

type algorithms.

The f i r s t t e c h n i q u e i s a s s o c i a t e d w i t h t h e r e d u c e d - g r a d i e n t

o r v a r i a b l e - r e d u c t i o n t y p e a l g o r i t h m s f o r n o n l i n e a r programming, a n d i s a c r u c i a l p a r t o f t h e s i m p l e x method f o r l i n e a r programming.

I t i s b a s e d on t h e s i m p l e o b s e r v a t i o n t h a t t h e

l i n e a r i n d e p e n d e n c e a s s u m p t i o n i m p l i e s t h e e x i s t e n c e o f an

(r

+

p ) by ( r

+

p) nonsingular submatrix P

for simplicity t h a t the f i r s t r

+

a

o f P.

p columns o f P are l i n e a r l y

i n d e p e n d e n t , w e c a n p a r t i t i o n P a s P = [Pa, Pb] an ( r

+

p ) by [n

-

(r

+

Assuming

p ) ] matrix.

decomposition o f t h e v a r i a b l e x = ( x

, where

Pb i s

This induces a n a t u r a l

x ~ ) and ~ , since

204


* g(x ,

*

0 ) = 0 and h ( x

,

0 ) = 0 , a l l o w s a p p l i c a t i o n of t h e i m -

p l i c i t f u n c t i o n theorem t o conclude t h a t t h e r e e x i s t s a t w i c e

,

d i f f e r e n t i a b l e v e c t o r f u n c t i o n xa ( x b ) s u c h t h a t g [ x a (x,)

= x

*

.

xb,

The xa and xb may be t h o u g h t o f as " d e p e n d e n t " a n d

"independent" v a r i a b l e s , r e s p e c t i v e l y .

Once t h e b i n d i n g con-

s t r a i n t s a r e i d e n t i f i e d it s u f f i c e s t o m i n i m i z e f [ x a ( x b ) , xb, 01 o v e r xb u s i n g a n y a p p r o p r i a t e u n c o n s t r a i n e d method,

*

d e t e r m i n e xb, a n d h e n c e x

*

.

to

The i n d i c a t e d a l g o r i t h m s a c t u a l l y

i n v o k e t h e l i n e a r i n d p e n d e n c e a s s u m p t i o n f o r aZZ f e a s i b l e boundary p o i n t s , a n d h e n c e a t any g i v e n i t e r a t i o n c a n e i t h e r reduce f ( x ) wit h o u t e n c o u n t e r i n g c o n s t r a i n t s or w i l l be i n a s i t u a t i o n completely analogous t o t h e one d e s c r i b e d a t t h e o u t s e t , a n d c a n p r o c e e d t o minimize f o v e r t h e c u r r e n t l y i n d e p e n d e n t v a r i a b l e s i n t h e s p a c e of c u r r e n t l y b i n d i n g constraints. R e t u r n i n g t o t h e d e t e r m i n a t i o n o f S and P# f o r t h i s t y p e o f a l g o r i t h m , w e observe t h a t S : IS:, PS = [Pa, P b l [ s , l = 0, -1

so Sa = -Pa PbSb a n d h e n c e

S i m i l a r l y , s i n c e PP'P

= PI defining

must s a t i s f y

205

7 . 3 Inverse of the Jacobian of the Karush-Kuhn-Tucker System

gives the result p# =

[

p2 - pIp2)]

=

[

+

[-'rPb]

p2'

I n t e r m s o f t h e q u a n t i t i e s d e f i n e d , t h e b l o c k components of

Aij

fi-'

a r e t h e r e f o r e g i v e n by ( 7 . 3 . 1 ) , where

S = TSb, P # =

[;'I

+

TP2,

and

a n n by [ n any t n [n

-

-

-

(r

( r + p ) ] m a t r i x w i t h r a n k [n

-

( r + p ) ] , Sb i s

+ p ) ] s q u a r e n o n s i n g u l a r m a t r i x , and P2 i s any

( r + p ) ] by ( r

+

p) matrix.

For t h e projected-gradient type algorithms t h e g r a d i e n t s

of b i n d i n g c o n s t r a i n t s a r e a g a i n assumed l i n e a r l y i n d e p e n d e n t a t f e a s i b l e boundary p o i n t s , w i t h t h e d a t a b e i n g o r g a n i z e d m a i n l y t o accommodate a p r o j e c t i o n m a t r i x o f t h e form, PR = I

-

PT(PPT)-'P,

used t o p r o j e c t a given d i r e c t i o n v e c t o r i n t o

t h e l i n e a r s u b s p a c e a s s o c i a t e d w i t h t h e c u r r e n t l y b i n d i n g constraints.

Here, t h e rows o f P would r e p r e s e n t t h e g r a d i e n t s

o f t h e c o n s t r a i n t s c u r r e n t l y deemed t o r e m a i n b i n d i n g i n t h e n e x t i t e r a t i o n , and PPT i s a n ( r

+

p ) by ( r

+

p ) m a t r i x , non-

s i n g u l a r under t h e l i n e a r independence assumption. t h a t P# = PT(PPT)-'

We find

s a t i s f i e s t h e r e q u i r e m e n t s f o r a pseudo-

i n v e r s e o f P , a n d it f o l l o w s e a s i l y t h a t a s u i t a b l e c h o i c e f o r S i s any m a t r i x SR formed by s e l e c t i n g a n y [n

-

(r

+ p)l

l i n e a r l y i n d e p e n d e n t columns of PR. The b l o c k components of 1 M f o r g r a d i e n t - p r o j e c t i o n type c a l c u l a t i o n s are t h e r e f o r e T -1 t h e same as ( 7 . 3 . 1 ) , w i t h S = SR and P# = PT(PP ) A-

.

206 7.4.


SENSITIVITY RESULTS FOR AUGMENTED LAGRANGIANS R e c e n t l y , t h e augmented L a g r a n g i a n a p p r o a c h h a s b e e n viewed

more and more i n t h e s p i r i t o f a Quasi-Newton method r e l a t e d

t o i t e r a t i o n s of t h e g e n e r a l form ( 7 . 2 . 1 ) .

Thus, w e may an-

t i c i p a t e a c l o s e c o n n e c t i o n between t h e a l g o r i t h m i c c a l c u l a t i o n s and t h e s e n s i t i v i t y c a l c u l a t i o n s ( 3 . 2 . 3 1 , i n t h e l a s t two s e c t i o n s .

as i n d i c a t e d

A t h e o r e t i c a l connection i s r e a d i l y

e s t a b l i s h e d w i t h o u r p r i o r r e s u l t s , w i t h t h e augmented Lagrangian p l a y i n g t h e role of t h e u s u a l Lagrangian.

W e con-

clude t h i s chapter with s e n s i t i v i t y r e s u l t s f o r a typical augmented L a g r a n g i a n f u n c t i o n .

The Karush-Kuhn-Tucker

param-

e t e r d e r i v a t i v e s t u r n o u t t o be e x a c t f o r t h i s c l a s s o f a l g o r i t h m s , i f t h e o p t i m a l Lagrange m u l t i p l i e r s a r e known. T h i s d e v e l o p m e n t i s b a s e d l a r g e l y o n t h a t g i v e n by A r m a c o s t and Fiacco (1977). P e n a l t y f u n c t i o n a l g o r i t h m s , s u c h as t h e l o g a r i t h m i c b a r r i e r q u a d r a t i c - l o s s p e n a l t y f u n c t i o n s t u d i e d i n Chapter 6, b e l o n g t o a g e n e r a l c l a s s o f a l g o r i t h m s whereby t h e c o n s t r a i n e d p r o b l e m i s t r a n s f o r m e d i n t o a s e q u e n c e o f u n c o n s t r a i n e d prob-

l e m s by means o f a n a u x i l i a r y f u n c t i o n .

M o t i v a t e d i n p a r t by

an e f f o r t e i t h e r t o " r e g u l a r i z e " or "convexify" t h e u s u a l L a g r a n g i a n o r t o overcome t h e now w e l l u n d e r s t o o d and w e l l documented i l l - c o n d i t i o n i n g t y p i c a l l y a s s o c i a t e d w i t h t h e c l a s s i c a l p e n a l t y f u n c t i o n p r o c e d u r e s a s a s o l u t i o n i s app r o a c h e d , o t h e r t y p e s o f a u x i l i a r y f u n c t i o n s h a v e more r e c e n t l y received considerable attention.

These i n c l u d e " e x a c t p e n a l t y

f u n c t i o n s " and " g e n e r a l i z e d Lagrangians"

[ A r r o w , Gould, a n d

Howe (1973) ] a n d "augmented L a g r a n g i a n s " [ H e s t e n e s ( 1 9 6 9 ) a n d Buys ( 1 9 7 2 ) I .

207

7.4 Sensitivity Results for Augmented Lagrangians

G e n e r a l i z e d L a g r a n g i a n s g e n e r a l l y r e f e r t o more g e n e r a l forms o f t h e u s u a l Lagrangian a s s o c i a t e d w i t h problem P: minx f ( x ) s . t . g ( x ) 1. 0 , h ( x ) = 0 .

T y p i c a l l y , f , gi,

a r e r e p l a c e d by f u n c t i o n s o f t h e s e q u a n t i t i e s .

or w

h j , ui,

j

The o b j e c t i s

t o s t r u c t u r e t h e new f u n c t i o n so t h a t it behaves " l i k e " a Lagrangian (e.g.,

i n characterizing optimality) but acquires

c e r t a i n des irable p r o p e r t i e s (e.g.,

convexity o r strict

c o n v e x i t y ) t h a t t h e u s u a l Lagrangian d o e s n o t p o s s e s s .

A

p o p u l a r e x t e n s i o n i n t r o d u c e s g e n e r a l i z e d Lagrange m u l t i p l i e r s t h a t may a l s o be f u n c t i o n s of x.

Augmented Lagrangians a r e

u s u a l l y formed by a d d i n g a p e n a l t y term t o t h e u s u a l L a g r a n g i a n , though t h i s c l a s s can e a s i l y be e n l a r g e d by u s i n g t h e g e n e r a l i z e d L a g r a n g i a n s d e s c r i b e d e a r l i e r i n t h e paragraph. The m o t i v a t i o n was a p p a r e n t l y f i r s t " c o n v e x i f i c a t i o n , " t h e n a way o f overcoming i l l - c o n d i t i o n i n g and a l s o a way o f developi n g an " e x a c t " method, a s d e s c r i b e d i n t h e n e x t p a r a g r a p h .

In

t h i s c o n t e x t , t h e t e r m "Method o f M u l t i p l i e r s ' ' r e f e r s t o a p a r t i c u l a r a p p r o a c h u s i n g a p a r t i c u l a r form o f an augmented Lagrangian.

The t e r m i s due t o Hestenes ( 1 9 6 9 ) , who proposed

an a l g o r i t h m b a s e d on s e q u e n t i a l l y improving t h e e s t i m a t e s of t h e Lagrange m u l t i p l i e r s .

S i m i l a r i d e a s w e r e a l s o proposed by

Powell ( 1 9 6 9 ) and o t h e r s a t a b o u t t h e same t i m e . A p e n a l t y f u n c t i o n would be c o n s i d e r e d e x a c t i f , f o r p a r t i c u l a r v a l u e s o f c e r t a i n p a r a m e t e r s , an u n c o n s t r a i n e d ( l o c a l ) minimum o f t h e f u n c t i o n ( l o c a l l y ) s o l v e s t h e g i v e n programming problem.

S i n c e t h e " o p t i m a l v a l u e s " of t h e s e

p a r a m e t e r s a r e g e n e r a l l y unknown, t h e y must be e s t i m a t e d , and s i n c e t h e s e p a r a m e t e r s o f t e n c o r r e s p o n d t o Lagrange m u l t i p l i e r s , t h e r e s u l t i s t h a t augmented Lagrangian and e x a c t penalty function algorithms a r e q u i t e s i m i l a r i n s p i r i t .

208


I t i s much e a s i e r t o d e a l w i t h e q u a l i t y c o n s t r a i n t s i n

t h e s e methods.

When c o n s i d e r i n g i n e q u a l i t y c o n s t r a i n t s ,

p e n a l t y f u n c t i o n s may b e i n t r o d u c e d f o r which t h e same o r d e r of d i f f e r e n t i a b i l i t y i s n o t i n h e r i t e d from t h e problem functions.

Also,

s e r i o u s c o m p l i c a t i o n s arise i n c o n s t r u c t i n g

e f f e c t i v e s t r a t e g i e s a s s o c i a t e d w i t h t h e i d e n t i f i c a t i o n of t h e binding inequality constraints. Arrow e t a l .

(1973) developed a s a d d l e - p o i n t t h e o r y f o r an

"extended" L a g r a n g i a n , d e f i n i n g m u l t i p l i e r f u n c t i o n s w i t h certain limiting properties.

R e a l i z a t i o n s of t h e i r extended

Lagrangian y i e l d many o f t h e augmented L a g r a n g i a n s and e x a c t p e n a l t y f u n c t i o n s found i n t h e l i t e r a t u r e .

As i n d i c a t e d , a p r i m a r y m o t i v a t i o n f o r t h e s t u d y of augmented L a g r a n g i a n s and e x a c t p e n a l t y f u n c t i o n s h a s been t o overcome t h e problems o f i l l - c o n d i t i o n i n g a s s o c i a t e d w i t h ordinary penalty functions.

Because of t h i s , c o n s i d e r a b l e

e f f o r t h a s been made t o i n v e s t i g a t e t h e c o m p u t a t i o n a l a s p e c t s of t h e s e a l g o r i t h m s [ e . g . ,

Bertsekas (1975), Rockafellar

( 1 9 7 3 ) , and Rupp ( 1 9 7 6 ) J .

Buys ( 1 9 7 2 ) p r o v i d e d p e r h a p s t h e

e a r l i e s t b e s t d e t a i l e d a n a l y s i s o f a d u a l a p p r o a c h , u s i n g an augmented Lagrangian f i r s t i n t r o d u c e d by R o c k a f e l l a r ( 1 9 7 1 ) . Buys and Gonin ( 1 9 7 7 ) and Armacost and F i a c c o (1977) used t h e same approach t o o b t a i n s e n s i t i v i t y a n a l y s i s r e s u l t s i n t e r m s o f t h i s augmented Lagrangian f o r m u l a t i o n .

S i n c e o u r i n t e n t is

t o address t h e s e n s i t i v i t y r e s u l t s , f u r t h e r descriptions of the computational approaches a r e n o t pursued. For s i m p l i c i t y , t h e s p e c i f i c augmented Lagrangian used by Buys and Gonin (1977) w i l l be u t i l i z e d h e r e .

I t i s noted,

however, t h a t o u r approach i s d i r e c t l y a p p l i c a b l e t o t h e

209


g e n e r a l form o f t h e e x t e n d e d Laqrangian g i v e n by Arrow e t a l . , t h u s encompassing a l a r g e f a m i l y o f augmented L a g r a n g i a n s and exact penalty functions.

W e a d d r e s s problem P 3 ( ~ ) .

R a t h e r t h a n u s e t h e d u a l approach o f Buys, t h e s e n s i t i v i t y r e s u l t s t h a t o b t a i n u s i n g augmented L a q r a n g i a n s f o l l o w d i r e c t l y by c o n s i d e r i n g t h e e q u a t i o n s t h a t a r e s a t i s f i e d a t a solution point.

The key p o i n t i n t h e f o l l o w i n g development i s

t h a t t h e g r a d i e n t o f t h e augmented Lagrangian i s e q u a l t o t h e g r a d i e n t o f t h e o r d i n a r y Laqranqian n e a r a s o l u t i o n p o i n t , when t h e problem p a r a m e t e r

E

i s perturbed.

L e t c > 0 be a c o n s t a n t , J

i = 1,

...,m }

-

and K E { i l ui

-

{ i l ui cgi(x,

E)

cqi(x,

E)

2 0,

< 0 , i = 1,

As u s u a l , w e suppose t h e a s s u m p t i o n s o f Theorem 3.2.2 problem P 3 ( ~ ) . L e t x

*

{ i l q i ( x * , 0 ) = 0 , i = 1, ui

-

-

c g i ( x * , 0 ) = u;

hold f o r

be a l o c a l minimum o f problem P 3 ( 0 )

w i t h a s s o c i a t e d Laqrange m u l t i p l i e r s u

*

..., m}.

..., m).

*

-

*

..., m

*

.

Then, f o r i

> 0 and f o r i = 1,

qi ( x * , 0 ) > 0 and hence u;

and w

L e t B (0) =

B*(O),

E

f B*(O),

cgi ( x * , 0) = - c g i ( x

*

,

0) < 0.

Thus, a t t h e s o l u t i o n p o i n t , J i s d e f i n e d w i t h s t r i c t i n e q u a l i t y and c o r r e s p o n d s t o B * ( O ) , and K c o n t a i n s a l l i such t h a t qi(x*,

0) > 0 .

The augmented Lagranqian f o r problem Pj ( E ) Section 2.4,

,

as

in

is defined a s

+

7 P

(wj

+

(1/2)ch.(x, E ) ) h . ( x , 3

j=l

-

(1/2c)

2

2

Ui'

7

E)

(7.4.1)

210


The g r a d i e n t o f @ t a k e n w i t h r e s p e c t t o x i s h

VxL(x, u, w ,

c) =

E,

v XL

+ ieJ

P

2

chjVxhj.

*

*

+

(7.4.2)

j=l At

(x, u, w ,

*

c) = (x

E,

, u , w , 0,

c ) , s i n c e g.(x*, 0) = 0, 1

i e J , and h . ( x * , 0) = 0 €or a l l j , i t f o l l o w s t h a t

I

A

*

VXL(X

,

u

*

*

,w ,

0 , c) = VXL(X

*

,

u

*

* ,w ,

(7.4.3)

0).

The augmented L a g r a n g i a n i s t w i c e c o n t i n u o u s l y d i f f e r e n t i a b l e

-

i n x, e x c e p t a t p o i n t s where ui t i o n ( i v ) of Theorem 3 . 2 . 2 , and hence

*

near (x

,

2 is u

*

,

ui

*

cgi(x, cgi(x

*

E)

,

= 0.

By assump-

0) # 0 f o r a l l i,

t w i c e c o n t i n u o u s l y d i f f e r e n t i a b l e f o r (x, u, 0).

Differentiating (7.4.2)

E)

with respect to x

yields 2*

VXL(X,

u, w,

E,

2

c) = VXL + ieJ

2 P

+

ieJ 2

ch.V h 1 x 1

j=l

+

(7.4.4)

cVxhTVxhj. j=1

Again, a t (x, 2 A *

VXL(X

Ur

w,

, u* , w * ,

*

E,

c ) = (x

* * ,u ,w ,

0 , c) ,

*T

0, c ) =

*

ieJ

+

2 P

c

j=l

*T * Vxhj Vxhj,

(7.4.5)

21 1


*

where t h e e x p o n e n t

at

* * * (x , u , w , *

J = B

(0) = 11,

denotes evaluation of t h e given functions W i t h o u t loss o f g e n e r a l i t y , assume t h a t

0).

..., r ) .

Then, r e c a l l i n g t h a t

it follows t h a t 26 2 VXL = VXL

T

+

CP P.

(7.4.6)

T 2 * S i n c e z VxL z > 0 f o r a l l z # 0 s u c h t h a t Pz = 0 , by a s s u m p t i o n s ( i i )a n d ( i v ) o f Theorem 3.2.2,

then f o r a l l c

26 s u f f i c i e n t l y l a r g e i t f o l l o w s e a s i l y t h a t VxL i s p o s i t i v e

*

d e f i n i t e n e a r (x

c

*

,

u

> 0 such t h a t f o r

*

* , w , 0 , c ) . Thus, t h e r e i s a number * 2 n * * * c > c , VxL(x , u , w , 0 , c ) i s p o s i t i v e *

d e f i n i t e (and hence n o n s i n g u l a r ) .

Assume t h a t c > c

and a s

b e f o r e , t h a t t h e f i r s t r i n e q u a l i t y c o n s t r a i n t s are assumed b i n d i n g , t h e s u p e r b a r i n d i c a t i n g e v a l u a t i o n a t i = l , . . . , r. For convenience w e s h a l l u s e a b a r underscore t o denote evaluation a t i = r

+

l , . . . , m.

The main r e s u l t c a n b e s t a t e d as

follows. [ S e n s i t i v i t y r e s u l t s u s i n g a n augmented

Theorem 7 . 4 , l .

Lagrangian f o r problem P ( ~ 1 . 1 I f t h e a s s u m p t i o n s o f Theorem 3.2.2 then f o r

E

n e a r 0 and c > c

*

h o l d f o r p r o b l e m P3(€),

t h e r e e x i s t s a u n i q u e , o n c e con-

tinuously d i f f e r e n t i a b l e vector function y a ( € , c) =

-u ( E ,

c), n

w(E,

VXL(X,

c),

u, w,

g(E, E,

uigi(x, h.(x, 3 uigi(x,

[x(E,

c),

c)lT satisfying

c ) = 0, E)

= 0,

i = 1,

E)

= 0,

j = 1

E)

= 0,

..., r ,

,..., P , i = r + 1, ..., m,

(7.4.7) (7.4.8 (7.4.9 (7.4.10

212


with

c),

[x(E,

c),

U(E,

( t h e Karush-Kuhn-Tucker f o r any

[x(E),

u(E),

t r i p l e of Theorem 3.2.2)

n e a r 0 and c > c

E

c ) l T=

W(E,

*

,

A

E,

c ] and VxL[x, u, w,

*

n e a r (x

*

,u ,w

*

u),

(7.4.7)

-VxgT

VxhT

G

0

0

Wr

gr

= (X

C)

E,

V?:* :M

u

r

r

w

*

*

I

0

, is

..., r

..., m.

and

G

=

E v a l u a t e d a t (x, U,

2 , 0, c ) , Ma becomes

-Vx?J*T Vxh*T

=

taken

G

d i a g ( u i ) , i = r + 1,

* - *

7.4.10)

system (3.2.2)

= d i a g ( g i ) , U = d i a g ( u i ) , i = 1,

1=

-

-Vx9

-

diag(gi),

c) ,

W(E,

t h e p r e c i s e analogy of t h e

J a c o b i a n M ( E ) o f t h e Karush-Kuhn-Tucker

where

c),

).

w i t h r e s p e c t t o (x, U, w ,

V?:

U(E,

cl i s p o s i t i v e d e f i n i t e f o r (x, u , w )

E,

The J a c o b i a n m a t r i x o f

Proof.

and s u c h t h a t

c ) i s a l o c a l l y u n i q u e un-

X(E,

c o n s t r a i n e d l o c a l m i n i m i z i n g p o i n t of L[x, 2"

w ( E ) ]: ~Y ( E )

0

0

0

0

0

0

-Vxg (7.4.11)

0

G

Under t h e g i v e n a s s u m p t i o n s i t f o l l o w s t h a t MZ-'

e x i s t s and

h e n c e , by t h e i m p l i c i t f u n c t i o n theorem, t h e r e e x i s t s a u n i q u e , once c o n t i n u o u s l y d i f f e r e n t i a b l e v e c t o r f u n c t i o n y [X(Er

c ) r

-

U(Er

c ) r

W(E,

c),

U(Erc)lT

(E,

s a t i s f y i n g (7.4.7)

c) =

-

n e a r 0 a n d c > c*. 2A AS i n d i c a t e d above, V x ~i s p o s i t i v e d e f i n i t e a t , a n d h e n c e

(7.4.10) near ,

x

for

*

,

u

E

*

* ,w ,

0, c ) .

I t follows t h a t

A

u n i q u e l o c a l minimum of L[x, n e a r 0 and c > c

*

.

U(E,

c),

W(E,

X(E,

c) is a l o c a l l y

c ) , E, cl f o r

E

213


Observing t h a t near

E

= 0,

Vxt

V L at [x(E, c), U(E, X

a comparison o f (7.4.7)

Kuhn-Tucker s y s t e m ( 3 . 2 . 2 1 ,

u(E),

w ( E ) ]for

E

(7.4.10)

c),

[x(E,

h

c),

W(E,

c)l f o r

E

c)]

and t h e Karush-

U(E,

c),

W(E,

c)]

n e a r 0 , and t h e proof i s complete.

I t may a l s o b e n o t e d i n p a s s i n g t h a t L U(E,

W(E,

and t h e u n i q u e n e s s o f t h e s o l u -

t i o n s of both systems, implies t h a t [x(E),

-

c),

n e a r 0.

f a t [x(E, c),

Along w i t h t h e c o n c l u s i o n s

given i n t h e l a s t p a r t of t h e p r o o f , t h i s immediately i m p l i e s t h e r e s u l t s t h a t p r e c i s e l y p a r a l l e l t h o s e g i v e n i n Theorem f o r t h e o p t i m a l v a l u e f u n c t i o n and i t s f i r s t and s e c o n d

3.4.1

A

d e r i v a t i v e s , where t h e augmented L a g r a n g i a n L r e p l a c e s t h e u s ua l Lagrangian L i n t h e given e x p r e s sio n s . S i n c e t h e system (7.4.7)

-

(7.4.10)

i s once continuously

d i f f e r e n t i a b l e and i d e n t i c a l l y e q u a l t o z e r o f o r

c > c

*

,

Ma(€,

i t c a n be d i f f e r e n t i a t e d w i t h r e s p e c t t o c)VEya(~c , ) = Na(€,

where M a ( € ,

v ~a y

and

(E,

c) is defined f o r

c)

=

c) E

n e a r 0 and c > c

E

n e a r 0 and

E

to y i e l d (7.4.12)

*

,

214


N o t i c e f i r s t t h a t (7.4.12) since

G

L(E,

c) E 0 f o r i

-N a ( € ,

y i e l d s ~ V € U ( E , c ) = 0 , and

> 0 i t f o l l o w s t h a t V E ~ ( € , c) = 0 , as expected,

{

B*(O).

c ) b e t h e p o r t i o n s of

inequality constraints.

Let

since

Z a ( € , c ) , VETa(€, c ) , a n d exclu d in g t h e nonbinding

(7.4.12)

Then w i t h

and l e t t i n g

it follows t h a t

Letting (7.4.13) w e have t h a t

The e l e m e n t s C i j

Recall t h e computations f o r t h e g e n e r a l

given i n Chapter 4. p r o b l e m when V$-' i d e n t i c a l , w i t h; : V Cll

= VXL 2

C12

=

c21 T

yI =

are i m m e d i a t e l y d e t e r m i n e d u s i n g a r e s u l t

e x i s t s (4.2.8). 2 r e p l a c i n g VxL,

-

The s i t u a t i o n h e r e i s and w e o b t a i n

P'( P v y P T j 1 P v ; i - l ) ,

v y P T ( P v x 2A L - lpT)-'

r

(7.4.14)

215


Following t h e development i n Chapter 4 , S e c t i o n 4.2 p a r t i c u l a r , u s i n g (4.2.1511,

[in

t h e estimate o f t h e f i r s t - o r d e r

s e n s i t i v i t y o f t h e n o n t r i v i a l components o f a Karush-KuhnT u c k e r t r i p l e f o r problem P 3 ( ~ )i s g i v e n by

-VEh

.

(7.4.15)

-VEh c ) :Y ( E ) n e a r

r

V E y ( ~ f) o r

E

(E, c) ~a The s t r u c t u r e of t h e o r i g i n a l p r o b l e m

n e a r 0.

E

= 0 , w e must h a v e V y

a n d t h e augmented L a g r a n g i a n c a n b e u s e d t o d e m o n s t r a t e exp l i c i t l y t h a t V E y a ( ~ ,c ) : V E y ( ~ ) . R e c a l l t h a t f o r t h e g i v e n p r o b l e m P 3 ( ~,)

where

a n d t h e components A i j (4.2.13)

,

a r e d e f i n e d by ( 4 . 2 . 8 ) ,

d e p e n d i n g on which c o n d i t i o n s a p p l y .

D i f f e r e n t i a t i n g (7.4.2) with r e s p e c t t o

v 2E XAL

(4.2.9),

r

= VEXL 2

+

+

c

1

i=l

j=1

yields

r

9iv:x9i

h.V 3 2E Xh 3.

c

E

+

+

c

c

1

j=1

1

i=l

Vx9TVEgi

VxhTVEhj.

or

!


216

T h e s e and t h e f o l l o w i n g e q u a t i o n s are e v a l u a t e d a t y a ( € , c ) for

E

n e a r 0 , and hence, qi[x(E,

hj[x(E, c ) ,

= 0 , j = 1,

E]

..., p.

c),

E]

= 0,

i

E

*

J = B (0) a n d

This y i e l d s

v2

VEXL 2 " =

L

-

CpT[

EX

vE'],

-VEh

w i t h P d e f i n e d as b e f o r e .

An e x p l i c i t e x p r e s s i o n f o r VEya(E, c )

c a n now b e d e r i v e d , which s i m p l i f i e s (7.4.15). 5 1v2E X2 = c l l v ~ x L

-

ccllpT[-;:;],

a n d s i n c e (7.4.14) i m p l i e s t h a t CllPT

= 0,

i t follows t h a t

cllv;x2

= cllv;xL.

Similarly, 2 "

-VEh a n d s i n c e (7.4.14) i m p l i e s t h a t CZIPT = I ,

we obtain

Since

217

7.4 Sensitivity Results for Augmented Lagranpians

The f i r s t - o r d e r s e n s i t i v i t y o f t h e Karush-Kuhn-Tucker

triple

may t h e n be w r i t t e n a s

=

for

E

[ i . &

c1-1 + [O0

ol]i,(E1

C I

(7.4.16 1

n e a r 0.

S i n c e i t was p r e v i o u s l y concluded t h a t V E f T ( ~ ) i V ~a 7 for

E

n e a r 0 and c > c

*

,

(E,

c)

i t must b e t r u e t h a t

(7.4.17) To show e x p l i c i t l y t h a t t h i s r e l a t i o n s h i p h o l d s , n o t e t h a t (definition)

( u s i n g (7.4.6) 1

P

= (I

+

0

cPT

(0

0

I)[

V2L P

PT 0 1

218


P r e m u l t i p l y i n g t h e l a s t e q u a t i o n by k a ( c , c ) - l a n d p o s t m u l t i p l y i n g by f i ( ~ 1 - l y i e l d s 7.5.

(7.4.17).

CONCLUSIONS AND EXTENSIONS Under t h e g i v e n c o n d i t i o n s , t h e e q u a t i o n s ( 7 . 4 . 1 4 )

and

( 7 . 4 . 1 5 ) o b t a i n e d u s i n g t h e augmented L a g r a n g i a n may b e viewed

as a n a l t e r n a t i v e c o m p u t a t i o n a l d e v i c e f o r c a l c u l a t i n g t h e f i r s t - o r d e r s e n s i t i v i t y o f a Karush-Kuhn-Tucker problem P 3 ( f ) , e n c o m p a s s i n g ( 4 . 2 . 1 5 ) equations (4.2.8),

(4.2.9),

v a r i o u s cases s p e c i f i e d .

triple for

together with the

and (4.2.13)

t h a t apply i n t h e

T h a t i s , u s i n g t h e augmented

L a g r a n g i a n i n s t e a d of t h e u s u a l L a g r a n g i a n , o n l y Case 1 ( 4 . 2 . 8 ) can a r i s e , and, hence, o n l y t h e one set o f e q u a t i o n s ( 7 . 4 . 1 4 ) f o r t h e i n v e r s e o f t h e J a c o b i a n i s needed u n d e r t h e g i v e n conditions.

Note t h a t b o t h a p p r o a c h e s r e q u i r e knowledge o f

t h e b i n d i n g i n e q u a l i t y c o n s t r a i n t i n d i c e s and i n f a c t r e q u i r e

t h e d e t e r m i n a t i o n o f a Karush-Kuhn-Tucker (7.4.14)

definite.

triple.

In addition,

2

r e q u i r e s a v a l u e o f c f o r which V X L i s p o s i t i v e Using e i t h e r t h e u s u a l L a g r a n g i a n o r t h e augmented

Lagrangian, t h e i n d i c a t e d i n f o r m a t i o n per mits an e x a c t calcul a t i o n of t h e f i r s t - o r d e r

s e n s i t i v i t y o f t h e Karush-Kuhn-

Tucker t r i p l e . The r e q u i r e m e n t s f o r e x a c t s e n s i t i v i t y i n f o r m a t i o n are r a t h e r s e v e r e , and i n e f f e c t r e s u l t i n p o s t o p t i m a l i t y s e n s i t i v i t y analysis calculations (i.e., a solution is required b e f o r e s e n s i t i v i t y c a l c u l a t i o n s c a n b e made).

I f inexact

s e n s i t i v i t y information i s considered, t h e n one obvious p o s s ib i l i t y i s t o u s e estimates of t h e l o c a l s o l u t i o n a n d i t s a s s o c i a t e d o p t i m a l Lagrange m u l t i p l i e r s i n t h e g i v e n e q u a t i o n s .

219

7.5 Conclusions and Extensions

The augmented Lagrangian approach i s a l r e a d y " a l g o r i t h m i c , i n v o l v i n g f i r s t e s t i m a t i n g t h e c o n s t a n t c and t h e o p t i m a l Lagrange m u l t i p l i e r s

U(E,

c) ,

h

t h e n minimizing L o v e r x.

c ) f o r problem P 3 ( € ) , and

W(E,

Of c o u r s e , any v a l i d u n c o n s t r a i n e d

m i n i m i z a t i o n a l g o r i t h m c o u l d be used, a s c o u l d any a p p r o p r i a t e procedure f o r e s t i m a t i n g t h e optimal m u l t i p l i e r s .

i t may be o b s e r v e d t h a t i f

r e g a r d , from Theorem 7.4.1 W(E)

clo se t o 0, then [ u ( E , c ) , E

u(E),

a r e t h e ( l o c a l l y ) unique o p t i m a l Lagrange m u l t i p l i e r s f o r

problem P 3 ( ~ )a s s o c i a t e d w i t h

as

In t h i s

+

and i f

X(E)

W(E,

c)l =

E

[u(E),

is s u f f i c i e n t l y

*

~ ( € 1 1* ( u , w

0 and hence t h e l o c a l l y unique l o c a l minimum

?[x,

u(E),

X(E)

+

w(E),

x(0) = x

*

(with res pect t o

c ] i s g i v e n by G ( E , c ) =

.

E)

X(E,

c) =

*

)

;(E,

c ) of

X(E)

and

Clearly, the f i r s t p a r t i a l derivatives of

[x(E,

c),

U(E,

c),

W(E,

c ) l a l s o con-

v e r g e (component by component) t o t h e f i r s t p a r t i a l d e r i v a t i v e s

*

o f [ x ( O ) , u ( O ) , w ( O ) ] = (x

*

, u ,w

*

).

A c t u a l l y , any p r o c e d u r e

t h a t d e t e r m i n e s t h e o p t i m a l m u l t i p l i e r s of problem P 3 ( 0 ) as E +

0 can be used.

U n c o n s t r a i n e d m i n i m i z a t i o n o f t h e augmented

Lagrangian ( i n t h e a p p r o p r i a t e neighborhood) w i l l t h e n y i e l d an e s t i m a t e o f t h e l o c a l s o l u t i o n x ( 0 ) o f problem P 3 ( 0 ) , and t h e equations given i n (7.4.14)

and (7.4.15)

can be used t o

c a l c u l a t e t h e c o r r e s p o n d i n g estimates o f t h e f i r s t p a r t i a l d e r i v a t i v e s o f t h e Karush-Kuhn-Tucker

t r i p l e [x(O) , u ( 0 ) r W ( 0 ) l

o f problem P 3 (0). An a l t e r n a t i v e t o t h e above p r o c e d u r e s f o r e s t i m a t i n g t h e d e s i r e d s e n s i t i v i t y i n f o r m a t i o n i s t h e p e n a l t y f u n c t i o n proc e d u r e developed i n C h a p t e r 6 .

Analogous t o t h e augmented

Lagrangian r e s u l t , t h e p e n a l t y f u n c t i o n Hessian was shown t o b e p o s i t i v e d e f i n i t e n e a r a s o l u t i o n , so t h a t one set o f

220


e q u a t i o n s f ( 6 . 5 . 6 ) and (6.5.8) (7.4.1511

, analogous

to (7.4.14)

and

suffices to calculate the p a r t i a l derivatives of the

Karush-Kuhn-Tucker

triple.

With t h e p e n a l t y f u n c t i o n , t h e r e

i s n o n e e d t o make a p r i o r c a l c u l a t i o n o f t h e o p t i m a l L a g r a n g e m u l t i p l i e r s o r o f any o t h e r i n f o r m a t i o n , e x c e p t f o r t h e un( h a v i n g p r e s e t a s c a l a r param-

c o n s t r a i n e d minimizing p o i n t s

e t e r ) n o r m a l l y r e q u i r e d by t h e a l g o r i t h m , t o estimate t h e T h u s , t h e r e q u i r e d e f f o r t i s com-

t r i p l e and i t s d e r i v a t i v e s .

p a r a t i v e l y m o d e s t , w i t h r e s p e c t t o t h e L a g r a n g i a n a n d augmented L a g r a n g i a n c a l c u l a t i o n s , and makes t h i s p r o c e d u r e a p p e a l i n g a s a preoptimazity s e n s i t i v i t y a n a l y s i s e s t i m a t i o n technique. T h i s a p p e a l i s somewhat o f f s e t by t h e t y p i c a l i l l - c o n d i t i o n i n g t h a t c h a r a c t e r i z e s t h e p e n a l t y f u n c t i o n Hessian n ear a local s o l u t i o n o f t h e g i v e n problem.

Thus, a l l t h e i n d i c a t e d

approaches i n v o l v e compensating f a c t o r s , each o f f e r i n g advantages and disadvantages. A s a f i n a l o b s e r v a t i o n , t h e e x p r e s s i o n d e v e l o p e d above f o r

V E y a ( ~ ,c ) i n ( 7 . 4 . 1 4 )

and (7.4.15)

c a n be p l a c e d i n p r e c i s e

correspondence w i t h t h e s e n s i t i v i t y e x pr es s io n o b t a i n e d by Buys and Gonin ( 1 9 7 7 ) .

Our method of p r o o f i s s i m p l e r ,

however, u t i l i z i n g as shown a r e s u l t p r e v i o u s l y o b t a i n e d f o r t h e usual Lagrangian.

F u r t h e r , t h e r e l a t i o n s h i p between t h e

L a g r a n g i a n a n d augmented L a g r a n g i a n c a l c u l a t i o n s i s demons t r a t e d e x p l i c i t l y a n d allows a p p l i c a t i o n o f a l l p r i o r s e n s i t i v i t y r e s u l t s i n v o l v i n g t h e usual Lagrangian. Although a s p e c i f i c augmented L a g r a n g i a n f u n c t i o n was u s e d above t o o b t a i n t h e s e n s i t i v i t y r e s u l t s , t h e a n a l y s i s a n d a n a l o g o u s r e s u l t s o b t a i n f o r a more g e n e r a l f u n c t i o n , s u c h a s t h e Arrow e t aZ.

(1973) " e x t e n d e d " L a g r a n g i a n , which encom-

p a s s e s m o s t of t h e p o p u l a r f u n c t i o n s o f t h e augmented t y p e .

221

7.5 Conclusions and Extensions

I t seems c l e a r t h a t any v a l i d a l g o r i t h m w i l l y i e l d s e n s i t i v i t y

r e s u l t s a n a l o g o u s t o t h o s e d e v e l o p e d h e r e t h a t may be u t i l i z e d t o p r o v i d e e s t i m a t e s of t h e s e n s i t i v i t y of t h e o p t i m a l v a l u e f u n c t i o n and t h e Karush-Kuhn-Tucker

triple.


IV Applications and Future Research


Chapter 8

An Example of Computational Implementations: A Multi-Item Continuous Review Inventory Model

8.1.

INTRODUCTION

This c h a p t e r i s d i r e c t e d t o t h e p r a c t i c a l uses of s e n s i t i v i t y and s t a b i l i t y i n f o r m a t i o n .

Sensitivity analysis

r e s u l t s o b t a i n e d by Armacost and F i a c c o (1978) a r e r e p o r t e d , f o r a m u l t i - i t e m i n v e n t o r y model developed by Schrady and Choe ( 1 9 7 1 ) f o r t h e U.S.

Navy.

The computer program SENSUMT, i m -

p l e m e n t i n g t h e p e n a l t y f u n c t i o n method and s e n s i t i v i t y e s t i mates d e s c r i b e d i n C h a p t e r 6 , was used t o s o l v e t h e problem and a p p r o x i m a t e t h e s e n s i t i v i t i e s .

The example a n a l y z e d was

t h e one t r e a t e d by Schrady and Choe, though r e a d i l y e x t e n d e d t o a l a r g e - s c a l e model.

These r e s u l t s p r o v i d e d one of t h e

f i r s t v e r i f i c a t i o n s o f t h e f e a s i b i l i t y and p o t e n t i a l v a l u e o f a d e t a i l e d automated s e n s i t i v i t y a n a l y s i s c a p a b i l i t y f o r NLP i n practical situations.

They i l l u s t r a t e t h e numerous r i c h

i n t e r p r e t a t i o n s and i n s i g h t s t h a t can be d e r i v e d from t h i s i n f o r m a t i o n , a l s o i n d i c a t i n g t h e c a u t i o n t h a t must be t a k e n i n i n t e r p r e t i n g t h e r e s u l t s and making v a l i d i n f e r e n c e s . W e commence o u r d i s c u s s i o n w i t h an i s s u e t h a t w i l l be

r e c o g n i z e d a s v i t a l by anyone who h a s s e r i o u s l y pursued t h e generation of r e l e v a n t s e n s i t i v i t y information:

22s

indiscriminate

226

8 Multi-Item Continuous Review Inventory Model

p r o l i f e r a t i o n of d a t a .

An o b v i o u s f i l t e r i s t h e e f f e c t o f t h e

g i v e n p e r t u r b a t i o n s on t h e o p t i m a l v a l u e f u n c t i o n .

Algorith-

mic c a l c u l a t i o n s can i n e v i t a b l y be e x p l o i t e d t o e s t i m a t e t h e g r e a t e s t e f f e c t s and hence t h e most i n f l u e n t i a l p e r t u r b a t i o n s . W e i n d i c a t e how t h i s can be done, i n e x p e n s i v e l y and a s a

n a t u r a l byproduct o f a l g o r i t h m i c c a l c u l a t i o n s , u s i n g t h e c l a s s i c a l p e n a l t y f u n c t i o n methods o f t h e t y p e u t i l i z e d i n C h a p t e r 6.

Extension of t h e i d e a t o o t h e r f a m i l i e s of algorithms i s

q u i t e evident. 8.2.

SCREENING OF SENSITIVITY INFORMATION For problems i n v o l v i n g a l a r g e number of p a r a m e t e r s , a

very l a r g e number o f p a r t i a l d e r i v a t i v e s may be c a l c u l a t e d i f one p r o c e e d s i n d i s c r i m i n a t e l y .

This i s not only t i m e -

consuming, b u t may a l s o be q u i t e burdensome t o a u s e r who must evaluate t h e o v e r a l l significance of t h e r e s u l t s .

One impor-

t a n t measure o f t h e l a t t e r i s t h e e f f e c t o f a p e r t u r b a t i o n on the s o l u t i o n v a l u e , i.e.,

t h e optimal value function.

I t is

q u i t e p o s s i b l e and i s o f t e n o b s e r v e d i n p r a c t i c e t h a t t h e o p t i mal o b j e c t i v e f u n c t i o n v a l u e i s much more s e n s i t i v e t o v e r y few o f t h e many p a r a m e t e r s p r e s e n t .

With t h i s i n mind, a s i m p l e

technique f o r estimating t h e f i r s t - o r d e r s e n s i t i v i t y of t h e o p t i m a l v a l u e f u n c t i o n i s i n c o r p o r a t e d i n t h e SENSUMT computer program t o p r o v i d e an o p t i o n f o r p r e l i m i n a r y s c r e e n i n g o f t h e parameters i n order t o e l i m i n a t e f u r t h e r c a l c u l a t i o n s involving p e r t u r b a t i o n s o f p a r a m e t e r s deemed t o have r e l a t i v e l y l i t t l e e f f e c t on t h e o p t i m a l v a l u e f u n c t i o n .

A u s e r may i n t r o d u c e

h i s own c r i t e r i a o f s i g n i f i c a n c e i n t h i s d e t e r m i n a t i o n . t h e f o r m u l a s d e v e l o p e d by F i a c c o C1976), a s e c o n d o p t i o n

Using

221

8.2 Screening of Sensitivity Information

i s i n c l u d e d which p e r m i t s t h e c a l c u l a t i o n o f t h e s e n s i t i v i t y

estimates f o r t h e L a g r a n g e m u l t i p l i e r s .

This screening device

and t h e i n d i c a t e d o p t i o n s are d i s c u s s e d n e x t . C o r o l l a r y 6.6.2

provides a j u s t i f i c a t i o n f o r estimating

f * ( ~ )V , E f * ( E ) , and VEf 2 * ( E ) by W*(E, r ) , VEW*(€, r ) , a n d 2 *

VEW

(E,

r), r e s p e c t i v e l y , when r i s p o s i t i v e and s m a l l enough.

Since C o r o l l a r y 6.2.2 f[x(E, r ) ,

E]

=

f * (E)

,

and c o n t i n u i t y imply t h a t l i m r*O+ a n o t h e r estimate o f t h e o p t i m a l v a l u e

f u n c t i o n f * ( E ) i s p r o v i d e d by

r)

?(E,

E f[x(E,

r),

E]

when

r > 0 and s m a l l , a s noted i n t h e f i n a l p a r t of S e c t i o n 6.2. As i n ( 6 . 2 . 3 0 )

,

d i r e c t application of t h e chain r u l e f o r dif-

ferentiation yields, for x =

r),

X(E,

N

VEf(E, r) = Vxf(x, E ) V ~ X ( E , r )

+ VEf(x,

(8.2.1)

E).

Under t h e g i v e n a s s u m p t i o n s c o n t i n u i t y a l s o a s s u r e s t h a t

V E T ( ~ r, ) * VEW

(E,

-f

V E f * ( c ) as r * 0

+.

r ) a r e estimates of V E f

*

Thus, b o t h (E)

VET(€,

r ) and

for r sufficiently small.

I t s h o u l d be n o t e d t h a t t h e s e estimates are f u n c t i o n a l l y

related, since m

N

From t h i s e x p r e s s i o n i t i s c l e a r t h a t V E f ( E , r ) i s t h e b e t t e r

estimate o f V E f * ( E ) , t h e r e m a i n i n g terms i n VEW*(€, r ) s i m p l y constituting "noise" t h a t i s eliminated a s r by u s i n g t h e e x p r e s s i o n ( 6 . 6 . 2 ) 6.6.1,

f o r VEW

*

(E,

-+

0

+

.

However,

r) g i v e n i n Theorem

V E W * ( ~ ,r ) c a n b e e v a l u a t e d w i t h o u t n e c e s s i t a t i n g t h e

c a l c u l a t i o n of V € X ( E ,

r ) , which i s r e q u i r e d t o compute ( 8 . 2 . 1 ) .

228


Thus, t h e c r u d e r b u t c o m p u t a t i o n a l l y much c h e a p e r e s t i m a t e of V f E

*

( € 1 g i v e n by V E W

*

(E,

r) i n (6.6.2)

(noting (6.6.5)) has

been i n t r o d u c e d a s a n o p t i o n i n t h e computer program a s t h e indicated preliminary screening device t o identify c r u c i a l R e s t r i c t i o n of subsequent c a l c u l a t i o n s t o t h e s e

parameters.

p a r a m e t e r s , and o t h e r c a l c u l a t i o n s , s u c h a s t h e s h a r p e r e s t i -

m a t e o f VEf

*

are p r o v i d e d a s a d d i t i o n a l

g i v e n by ( 8 . 2 . 1 ) ,

(E)

options. I n summary, t h e b a s i s f o r t h e e s t i m a t i o n p r o c e d u r e u t i l i z e d h e r e i s t h e m i n i m i z a t i o n o f t h e p e n a l t y f u n c t i o n W(x,

E,

r)

T h i s y i e l d s a p o i n t K ( E , r ) t h a t is viewed

g i v e n by ( 6 . 2 . 1 ) .

a s a n e s t i m a t e o f a ( l o c a l ) s o l u t i o n x* o f t h e g i v e n p r o b l e m when r > 0 i s s m a l l .

The a s s o c i a t e d o p t i m a l Lagrange m u l t i -

p l i e r s u* a n d w* a r e t h e n e s t i m a t e d by U(E, r ) a n d W ( E , using t h e r e l a t i o n s h i p s given i n (6.2.17)

and (6.2.18)

r ) by

,

respectively. I f d e s i r e d , a l l of t h e f i r s t p a r t i a l d e r i v a t i v e s of =

[x(E, r ) ,

U(E,

r),

W(E,

r ) l Tw i t h r e s p e c t t o

E,

and ( 6 . 2 . 2 2 ) .

i s c a l c u l a t e d , t h e n VEf

*

(E)

r)

an estimate

of V E y ( ~ )may , b e o b t a i n e d by f i r s t s o l v i n g ( 6 . 2 . 1 5 ) applying ( 6 . 2 . 2 1 )

Y(E,

and t h e n

I f the f u l l matrix V € X ( E ,

i s e s t i m a t e d by ( 8 . 2 . 1 ) .

r)

However,

i f i t i s d e s i r e d t o e l i m i n a t e c a l c u l a t i o n s i n v o l v i n g param-

*

e t e r s h a v i n g less e f f e c t on l o c a l c h a n g e s of f ( € 1 , s c r e e n i n g d e v i c e d e s c r i b e d above i s u s e d . e s t i m a t i o n o f VEf

*

(E)

by V E W * ( ~ ,r ) .

the

This e n t a i l s i n i t i a l

Components o f

s p o n d i n g t o t h e r e s p e c t i v e components o f VEf

*

(E)

E

corre-

t h a t are deemed

i n c o n s e q u e n t i a l may t h e n b e d e l e t e d and would n o t e n t e r i n t o any s u b s e q u e n t c a l c u l a t i o n s .

*

that (x

,

u

*

,w

*

),

f

*

(E),

I n p a r t i c u l a r , it i s emphasized

and VEf

*

(E)

may b e e s t i m a t e d by

229

8 . 3 Example of Sensitivity Calculations by SENSUMT Y(E,

r ) , f [ x ( E , r ) , €1, and DEW

c a l c u l a t i n g a n y components o f V

*

(E,

r ) , respectively, without

Y(E,

r ) , once

X(E,

r ) i s known.

T h e s e , and o t h e r c a l c u l a t i o n s t h a t have b e e n i n c l u d e d i n t h e computer c o d e s i m p l e m e n t i n g t h e p r o c e d u r e , are i l l u s t r a t e d i n t h e n e x t example a n d d i s c u s s e d i n t h e i n v e n t o r y example a n d i n t h e f i n a l s e c t i o n of C h a p t e r 9. 8.3.

EXAMPLE SENSITIVITY CALCULATIONS BY SENSUMT B r i e f l y , t h e conduct o f a s e n s i t i v i t y a n a l y s i s i s

c o n t r o l l e d by f o u r o p t i o n s :

no s e n s i t i v i t y a n a l y s i s , a s e n s i -

t i v i t y a n a l y s i s a t t h e f i n a l "subproblem," i . e . ,

minimization

o f t h e p e n a l t y f u n c t i o n , a s e n s i t i v i t y a n a l y s i s a t e a c h subproblem a l o n g t h e p e n a l t y f u n c t i o n minimizing t r a j e c t o r y , o r a s e n s i t i v i t y a n a l y s i s a t t h e f i n a l subproblem, f o r a range of d i f f e r e n c i n g increments. A s a n i l l u s t r a t i o n o f t h e k i n d o f i n f o r m a t i o n t h a t c a n be

g e n e r a t e d , c o n s i d e r t h e f o l l o w i n g s i m p l e convex ( i n x ) p a r a m e t r i c n o n l i n e a r programming problem. E x a m p l e 8.3.1 min

X

f(x,

s.t. g l ( x , f o r E~ > 0 a n d

E)

= x

1

+

-

E2X2

x 1 - x 22 - > 0

E)

=

E~

not restricted.

El

W e f i r s t g i v e t h e s o l u t i o n i n c l o s e d form.

t h e Karush-Kuhn-Tucker

Application of

conditions y i e l d s t h e following:


230 U1(E)

= (1 +

€22) 1/

2

/2E1.

and gl[X(E),

=

El

0.

I t may b e r e a d i l y v e r i f i e d t h a t t h e a s s u m p t i o n s o f Theorem

are s a t i s f i e d .

3.2.2

f*(E)

= -E1(1

+

From t h e e x p r e s s i o n f o r

€;)Ii2.

1..)J

Taking p a r t i a l d e r i v a t i v e s w i t h r e s p e c t t o

v EX ( E )

€:y2

(1 +

1

=

(1

+

= (-(1

X(E)

+

(1

+

and

E~

E

=

(

-(1

+

€2

.;y2,

-E1E2/(1

1/2 E;)

w, e o b t a i n

€3

+ €$I2),

and

v E U*(E) l

~

1 2 4 E2/(2El(l

Suppose w e are i n t e r e s t e d i n t h e v a l u e s

+

E;)

E~

1/2 = 2,

)). E~

= 1.

E v a l u a t i o n of t h e above e x p r e s s i o n s y i e l d s

[-"I,

f* = - 2 0 ,

VEf* =

x* =

VEX

-fi

u;

= n/4,

* --

(-0, -a), [-W2

12/21r

-a/2 - n / 2

VEu: = ( - n / 8 ,

n / 8 ) .

F o r i l l u s t r a t i v e p u r p o s e s , n u m e r i c a l r e s u l t s o b t a i n e d by t h e computer program are i n c l u d e d i n T a b l e 8 . 3 . 1 f o r t h e o p t i m a l v a l u e f u n c t i o n and Lagrange m u l t i p l i e r s e n s i t i v i t y . The subproblem k c o r r e s p o n d s t o m i n i m i z a t i o n o f t h e p e n a l t y

T a b l e 8.3.1.

S e n s i t i v i t y E s t i m a t e s f o r Example 8.3.1, where Lagrangian

E

= (2, 1 )

Chain R u l e

Subproblem

f

af/acl

w a E 2

w a c 1

af/ac2

U1

aUl/aEl

aUl/aE 2

1

-1.9999

-1.9999

-0.9999

-1.3333

-1.3333

0.4999

-0.3333

0.1666

2

-2.5393

-1.5440

-1.2947

-1.4087

-1.4088

0.3860

-0.2200

0.1761

3

-2. 7765

-1.4439

-1.3833

-2.4139

-1.4139

0.3609

-0.1844

0.1767

4

-2.8128

-2.4243

-1.4064

-1.4142

-1.4142

0.3560

-0.1790

0.1768

5

-2.8245

-1.4127

-1.4123

-1.4142

-1.4142

0.3531

-0.1768

0.1768

6

-2.8274

-1.4137

-1.4137

-1.4142

-1.4142

0.3532

-0.1767

0.1768

Ana Zy t i c a l

-2.8282

-1.4142

-1.4142

-1.4142

-1.4142

0.3537

-0.1769

0.1768

232


f u n c t i o n W(x,

E,

r k ) , where rk > rk+l > 0.

The v a l u e s o f t h e

f i r s t - o r d e r o p t i m a l v a l u e f u n c t i o n s e n s i t i v i t y computed b o t h by t h e c h a i n r u l e (8.2.1) a n d by t a k i n g p a r t i a l d e r i v a t i v e s o f t h e Lagrangian w i t h r e s p e c t t o t h e parameters p r e s e n t e d f o r comparison.

(6.6.2) a r e

As i n d i c a t e d p r e v i o u s l y a n d a s

e x p e c t e d , t h e c h a i n r u l e estimates are s h a r p e r . t h e estimates o b t a i n e d by ( 6 . 6 . 2 ) tions.

Nonetheless,

g i v e v e r y good approxima-

The v a l u e s o f f are g i v e n by f [ x ( E , r ) , € 1 , w h i l e t h e

v a l u e s o f u1 a n d VEul w e r e o b t a i n e d by a p p l y i n g ( 6 . 2 . 1 7 ) (6.2.21).

The components o f

X(E)

and

and V E x ( € )w e r e c a l c u l a t e d

also b u t are n o t l i s t e d . The r e s u l t s a r e t y p i c a l .

Computational ex p er ien ce i n d i -

cates t h a t t h e s e n s i t i v i t y estimates w i l l c o n v e r g e s a t i s f a c t o r i l y whenever t h e g i v e n a l g o r i t h m s u c c e e d s i n d e t e r m i n i n g a good estimate o f a Karush-Kuhn-Tucker 8 4.

A MULTI-ITEM

point.

INVENTORY MODEL

T r a d i t i o n a l l y , i n v e n t o r y models h a v e b e e n f o r m u l a t e d t o minimize some f u n c t i o n of t h e o r d e r i n g , h o l d i n g , a n d s h o r t a g e

( o r b a c k o r d e r ) costs s u b j e c t t o v a r i o u s c o n s t r a i n t s .

Schrady

a n d Choe (1971) f o r m u l a t e d a n i n v e n t o r y model t h a t a p p e a r s t o h a v e much g r e a t e r r e l e v a n c e f o r a n i n v e n t o r y s y s t e m i n a noncommercial e n v i r o n m e n t , s u c h a s i n s t i t u t i o n a l o r m i l i t a r y . The c o s t s u s e d i n t h e t r a d i t i o n a l models may be q u i t e a r t i f i -

c i a l a n d t h e r e a l o b j e c t i v e o f t h e s y s t e m i s o f t e n maximizat i o n o f a m e a s u r e o f r e a d i n e s s o r service, h e r e assumed t o be equivalent t o minimization of stockouts.

In addition, t h e

s t o c k p o i n t s of s u c h s u p p l y s y s t e m s are i n e v i t a b l y c o n s t r a i n e d by i n v e s t m e n t a n d reorder work load l i m i t a t i o n s .

233

8.4 Multi-item Inventory Model

Schrady and Choe's m u l t i - i t e m i n v e n t o r y system assumes t h e s e c o n s t r a i n t s a l o n g w i t h t h e s p e c i f i c o b j e c t i v e o f minim i z i n g t h e t o t a l time-weighted

shortages.

The d e c i s i o n v a r i -

a b l e s a r e t a k e n t o b e t h e " r e o r d e r q u a n t i t i e s " and t h e " r e o r d e r p o i n t s , " r e s p e c t i v e l y , how much t o o r d e r and when t o o r d e r each i t e m i n t h e inventory.

A t h r e e - i t e m example problem was

s o l v e d by Schrady and Choe ( 1 9 7 1 ) u s i n g t h e SUMT computer code (Mylander e t a l . ,

1971).

S u b s e q u e n t l y , McCormick (1972)

showed how t h e s p e c i a l s t r u c t u r e o f t h i s i n v e n t o r y model can be used t o f a c i l i t a t e t h e u s e o f t h e SUMT code t o s o l v e v e r y l a r g e i n v e n t o r y problems.

H e a l s o e x t e n d e d t h e model t o i n -

c l u d e c o n s t r a i n t s on s t o r a g e volume and t h e p r o b a b i l i t y o f depletion of c r i t i c a l i t e m s . The model and example p r e s e n t e d h e r e a r e t h e o r i g i n a l o n e s due t o Schrady and Choe.

The computer program d e s c r i b e d i n

t h e p r e c e d i n g s e c t i o n was used t o s o l v e t h e example and c a l c u l a t e t h e p a r t i a l d e r i v a t i v e s of various q u a n t i t i e s of i n t e r e s t , w i t h r e s p e c t t o e a c h p a r a m e t e r i n v o l v e d i n d e f i n i n g t h e model. The a n a l y s i s can be a p p l i e d t o t h e e x t e n d e d model w i t h o u t difficulty. D e t a i l e d development o f t h e model i s n o t g i v e n h e r e .

The

i n t e r e s t e d r e a d e r i s r e f e r r e d t o t h e Schrady-Choe and McCormick papers.

Here, w e g i v e a summary t r e a t m e n t o f t h e v a r i o u s con-

d i t i o n s and r e l a t i o n s h i p s upon which t h e model is based.

We

then t a b u l a t e t h e r e s u l t s obtained i n solving the r e s u l t i n g n o n l i n e a r programming problem and a p p l y i n g t h e s e n s i t i v i t y a n a l y s i s methodology.

A number o f o b s e r v a t i o n s and i n t e r p r e -

t a t i o n s a r e o f f e r e d t o i l l u s t r a t e t h e many u s e s t o which t h e s e n s i t i v i t y i n f o r m a t i o n might be a p p l i e d .

234


I t i s assumed t h a t t h e amount o f e a c h i t e m i n i n v e n t o r y i s

a l w a y s known, t h a t a l l demand t h a t o c c u r s when t h e on-hand s t o c k i s zero i s b a c k - o r d e r e d ,

and t h a t t h e demand t h a t o c c u r s

d u r i n g t h e t i m e between t h e p l a c e m e n t o f a n o r d e r and i t s r e c e i p t by t h e s t o c k p o i n t ( i . e . , t h e " l e a d t i m e demand") i s n o r m a l l y d i s t r i b u t e d w i t h known mean For t h e i - t h i t e m ,

c

i

ui

and v a r i a n c e u

1'

let

= i t e m u n i t cost ( i n d o l l a r s ) ,

Xi

= mean demand p e r u n i t t i m e ( i n u n i t s ) ,

ri

=

Qi

= reorder quantity,

reorder point,

$ ( x ) = t h e N o r m a l ( 0 , 1) d e n s i t y f u n c t i o n , @ ( z )=

/z

@ ( x ) d x= t h e N o r m a l

(0,

1) complementary

cumulative d i s t r i b u t i o n function. I n a d d i t i o n , l e t K1 b e t h e i n v e s t m e n t l i m i t i n d o l l a r s , K2 be t h e number o f o r d e r s p e r u n i t o f t i m e t h a t c o n s t i t u t e s reo r d e r w o r k l o a d l i m i t , a n d N b e t h e t o t a l number o f i t e m s i n the inventory. I t c a n b e shown t h a t t h e e x p e c t e d t i m e - w e i g h t e d s h o r t a g e

o f i t e m i a t any p o i n t i n t i m e i s g i v e n by B ~ ( Q ~r i, ) = L [ B (r.1 . Qi 1 1

where

-

B ~ ( + Q r~i ) l

235

8.4 Multi-Item Inventory Model

+

The e x p e c t e d on-hand i n v e n t o r y o f i t e m i i s g i v e n by ri

-

pi

+

Qi/2

r i ) a n d t h e e x p e c t e d number of o r d e r s p l a c e d p e r

Bi(Qi,

u n i t t i m e f o r i t e m i i s A./Q 1

i'

U s i n g t h e above e x p r e s s i o n s a n d a s s u m p t i o n s , S c h r a d y and Choe ( 1 9 7 1 ) i n d i c a t e t h a t m e a n i n g f u l a p p r o x i m a t i o n s o f t h e g i v e n q u a n t i t i e s a r e o b t a i n e d even when t h e s e c o n d t e r m i s d r o p p e d from t h e e x p r e s s i o n f o r t h e e x p e c t e d s h o r t a g e s , and when t h e l a s t t e r m i s d r o p p e d from t h e e x p r e s s i o n f o r e x p e c t e d on-hand i n v e n t o r y .

The g i v e n a s s u m p t i o n s and s i m p l i f i c a t i o n s

t h e n l e a d r e a d i l y t o t h e f o l l o w i n g n o n l i n e a r programming p r o b l e m (which S c h r a d y and Choe ( 1 9 7 1 ) p r o v e d c o n v e x ) : N

N

i=1

w i t h ri u n r e s t r i c t e d i n s i g n , Qi 2 0 , i = l r . . . , N , (Q1r

- - I

QN)

T

,

r = (rl,

..., r N ) , a n d g1 T

Q

=

and g 2 r e p r e s e n t i n g

t h e i n v e s t m e n t and w o r k l o a d c o n s t r a i n t s , r e s p e c t i v e l y . The p r o b l e m d a t a f o r t h e Schrady-Choe

t h r e e - i t e m example

a n d t h e i n i t i a l s t a r t i n g p o i n t f o r t h e SENSUMT program a r e shown i n T a b l e 8.4.1.

As i n d i c a t e d i n t h e t a b l e , t h e l e a d t i m e

demands a n d s t a n d a r d d e v i a t i o n s , t h e i t e m u n i t c o s t s and mean demands, a n d t h e i n v e s t m e n t a n d w o r k l o a d l i m i t s are a l l t r e a t e d a s parameters i n conducting t h e s e n s i t i v i t y a n a l y s i s .

Our c a l -

c u l a t i o n s i n d i c a t e d t h a t t h e a s s u m p t i o n s o f Theorem 3.2.2 were s a t i s f i e d f o r (SC) a t t h e g i v e n s o l u t i o n , f o r t h e g i v e n d a t a .

236


Table 8.4.1.

I n v e n t o r y P r o b l e m Data Item i

Quantity

1

2

3

(mean o f l e a d t i m e demand)

100

200

300

(S.D. o f l e a d t i m e demand)

100

100

200

1

10

20

I , 000

I, 5 0 0

Parameters

'i U

i

ci

( i t e m u n i t c o s t , $)

'i

(mean demand/ unit time)

Kl

(investment l i m i t ) ( r e o r d e r work load limit )

K2

2,000

$8,000 15 r e o r d e r s / u n i t time

Var i a b 1 e s 0

(amount o r d e r e d )

600

270

300

0

(reorder level)

200

260

400

Qi P

i

T a b l e 8 . 4 . 2 g i v e s t h e computer s o l u t i o n and T a b l e 8 . 4 . 3 the f i n a l e s t i m a t e of t h e f i r s t p a r t i a l d e r i v a t i v e s of t h e optimal value function Z

eters.

*

w i t h r e s p e c t t o t h e problem param-

R e l a t i v e t o t h e c r i t e r i o n used i n t h e computer program,

t h e T a b l e 8 . 4 . 3 r e s u l t s i n d i c a t e t h a t t h e o p t i m a l v a l u e funct i o n i s s e n s i t i v e t o p a r a m e t e r s KZ, c l , a2, c 2 , a 3 , and c3. Many i n f e r e n c e s a r e p o s s i b l e .

For example, t h e f a c t t h a t t h e

s o l u t i o n i s p a r t i c u l a r l y s e n s i t i v e t o t h e v a l u e s of t h e s t a n d a r d d e v i a t i o n s o f t h e l e a d t i m e demand of i t e m s 2 and 3 might i n d i c a t e t h a t , s i n c e t h e s e p a r a m e t e r s w e r e o b t a i n e d by sampling, a d d i t i o n a l s a m p l i n g o f t h e s e l e a d t i m e demands may v e r y w e l l be w a r r a n t e d t o r e d u c e t h e a s s o c i a t e d s t a n d a r d deviations.

237


T a b l e 8.4.2.

S o l u t i o n and Lagrange M u l t i p l i e r s

Item i Quantity

Variables

* Qi * ri

1

2

3

533

246

285

253

277

437

Lagrange m u l t i p l i e r s

* *

0.0052

u3

0.6230

u2 Value

z Table 8.4.3

*

12.987

a l s o suggests t h a t t h e optimal s o l u t i o n value

i s v e r y s e n s i t i v e t o a l l of t h e i t e m c o s t s .

If the structure

of problem ( S C ) i s examined, t h i s r e s u l t may a t f i r s t a p p e a r

c o n t r a d i c t o r y s i n c e t h e ci a p p e a r o n l y i n t h e i n v e s t m e n t c o n s t r a i n t and t h e o p t i m a l v a l u e f u n c t i o n , a c c o r d i n g t o T a b l e 8.4.3.

O p t i m a l Value F u n c t i o n Derivatives Item i

P a r t i a 1s

az

*

1

2

3

lapi

-0.0000

az*/aoi

0.0119

0. 0897a

0.172ga

az

2.1713a

I. 034.Sa

1.445Za

0.0012

0.0025

0.0022

*

laci

a z* / a xi * az /aKl * az /aK2

-0.0003

-0.0008

-0.0052

-0. 6230a

*

I/?

*

aDeemed " s i g n i f i c a n t " b y c r i t e r i o n , [ A l Z > 0.001 f o r a u n i t change i n t h e g i v e n parameter, where A I Z i s t h e e s t i mated f i r s t - o r d e r change in Z*. T h i s criterion #as s e l e c t e d Criteria appropriate a r b i t r a r i l y f o r i l l u s t r a t i v e purposes. t o t h e p a r t i c u l a r a p p Z i c a t i o n can be s e l e c t e d b y a u s e r .

238


Table 8.4.3,

l i m i t K1.

i s apparently not very s e n s i t i v e t o t h e investment

The p r o b l e m i s o n e o f p r e c i s e i n t e r p r e t a t i o n .

p a r t i a l d e r i v a t i v e s measure r a t e o f change. t h e i n v e s t m e n t c o n s t r a i n t g l ( Q , r ) a t (Q

*

,

*

The

But i n s p e c t i o n o f

r ) reveals t h a t

t h e change i n a n i t e m cost ci by any amount A c i

h a s t h e same

e f f e c t on t h e c o n s t r a i n t as a c h a n g e i n t h e i n v e s t m e n t l i m i t K1 of -(r:

+

Q;/2

-

Since t h e quantity i n parentheses

pi)dci.

may be v e r i f i e d f r o m T a b l e s 8 . 4 . 1

and 8.4.2

t o b e much g r e a t e r

t h a n 1 f o r a l l i , i t f o l l o w s t h a t t h e e f f e c t o f c h a n g i n g any

c

by any i n c r e m e n t 6 w i l l b e much g r e a t e r o n t h e c o n s t r a i n t i * (and hence, on t h e o p t i m a l v a l u e Z , s i n c e t h e c o n s t r a i n t i s

b i n d i n g ) t h a n t h e e f f e c t o f c h a n g i n g K1 by t h e same amount 6 . This implies t h a t

I aZ*/aci I

>

I aZ*/aK1l

f a c t , i t c a n be shown h e r e t h a t aZ*/aci

*

a 2 /aK,,

f o r each i and, i n = -(r:

+

Q;/2

-

pi)

x

so t h a t t h e r e l a t i o n s h i p s i n d i c a t e d a r e i n d e e d

precisely verified. The above o b s e r v a t i o n s m i g h t a l s o s u g g e s t t h a t some care must b e t a k e n i n i n t e r p r e t i n g t h e r e s u l t s .

Changes i n t h e

parameter a s s o c i a t e d w i t h t h e l a r g e s t ( i n a b s o l u t e value) p a r t i a l d e r i v a t i v e w i l l g i v e t h e g r e a t e s t l o c a l change i n t h e o p t i m a l v a l u e o f t h e o b j e c t i v e f u n c t i o n , compared t o a c h a n g e o f t h e same m a g n i t u d e i n any o t h e r p a r a m e t e r t a k e n i n d i v i d u a l l y . This f o l l o w s because t h e o b j e c t i v e f u n c t i o n and/or s o m e o f t h e c o n s t r a i n t s (as above) are m o s t s i g n i f i c a n t l y a f f e c t e d by t h i s p a r a m e t e r change a t t h e c u r r e n t s o l u t i o n , i n t e r m s o f r a t e o f change.

G e n e r a l r u l e s have n o t b e e n g i v e n f o r s e l e c t i o n o f

optimal l o c a l changes i n t h e parameters, i.e.,

f o r determining

t h e o p t i m a l m a g n i t u d e and c o m b i n a t i o n o f s u c h c h a n g e s .

W e do

n o t pursue t h i s "marginal a n a l y s i s " d e t e r m i n a t i o n , though it s h o u l d be n o t e d t h a t t h e g r e a t e s t l o c a l r a t e of decrease i n

239


t h e optimal value f u n c t i o n i s along t h e d i r e c t i o n of t h e negative of t h e g r a d i e n t of t h i s f u n c t i o n i n parameter space

( i . e . , a l o n g t h e v e c t o r composed o f t h e n e g a t i v e o f t h e compon e n t s of t h e p a r t i a l d e r i v a t i v e s w i t h r e s p e c t t o t h e v a r i o u s A u s e r would n o n e t h e l e s s have t o d e t e r m i n e t h e

parameters).

f e a s i b i l i t y o f t h i s d i r e c t i o n o f change and, i f f e a s i b l e , t h e o p t i m a l move a l o n g t h i s v e c t o r , t a k i n g i n t o a c c o u n t o t h e r f a c t o r s s u c h a s t h e r e l a t i v e " c o s t e f f e c t i v e n e s s " o f any s c h e d u l e o f changes i n any model p a r a m e t e r . R e f e r r i n g back t o T a b l e 8.4.2,

*

w e n o t e t h a t t h e Lagrange

*

m u l t i p l i e r u2 i s much g r e a t e r t h a n ul.

Recalling the "sensi-

t i v i t y " i n t e r p r e t a t i o n o f Lagrange m u l t i p l i e r s , which h o l d s under t h e p r e s e n t c o n d i t i o n s , i t f o l l o w s t h a t u:

*

*

u2 = - 3 2 / a K 2 .

= -32

*

/aK1and

This conclusion i s consistent with t h e r e s u l t

o b t a i n e d i n T a b l e 8.4.3,

and i t means t h a t t h e workload con-

s t r a i n t g2 i s by f a r t h e more e f f e c t i v e i n d e t e r m i n i n g t h e minimum number of e x p e c t e d time-weighted r e n t v a l u e of t h e p a r a m e t e r s , e . g . , have a g r e a t e r e f f e c t on r e d u c i n g Z K1.

*

s h o r t a g e s a t t h e cur-

a s m a l l i n c r e a s e i n K2 w i l l than a small increase i n

N o n e t h e l e s s , a u s e r must a g a i n s i m u l t a n e o u s l y c o n s i d e r

t h e c o m p a r a t i v e c o s t s i n v o l v e d i n making f i n i t e changes, i n c o n j u n c t i o n w i t h t h e i r e x p e c t e d e f f e c t s , t o a r r i v e a t an o p t i mal m a r g i n a l improvement b a s e d on t h i s f i r s t - o r d e r i n f o r m a t i o n . The s e n s i t i v i t y i n f o r m a t i o n i s v a l u a b l e , b u t r e q u i r e s some care i n exploiting. T a b l e 8.4.4

gives t h e e s t i m a t e s of t h e f i r s t d e r i v a t i v e s

o f t h e o p t i m a l r e o r d e r q u a n t i t i e s Qi

and r e o r d e r p o i n t s ri w i t h

r e s p e c t t o e a c h o f t h e problem p a r a m e t e r s .

This i s extremely

d e t a i l e d i n f o r m a t i o n t h a t g i v e s an i n d i c a t i o n o f how t h e comp o n e n t s o f t h e s o l u t i o n v e c t o r i t s e l f w i l l change a s t h e

240


T a b l e 8.4.4.

Solution Point Sensitivity Item

Par t i aIs

i

2

1 -47.3187

-18. 7610

3

-14.9065

5.2265

6.1961

9.9671

,208.8688

15.3140

14.3755

-31. 7918

-10.3425

-20.0020

-0.8469

0.2271

-0.1084

-0.1273

1.0337

-0.4919

8.1908

-4.9719

3.8522

-2.6783

-7.2676

-7.1087

-1.2374

-0.4033

0.5843

-0.261 1

-0. 3 8 3 9

0.0446

-0. 1 6 7 0

0.4442

-0.4251

-3.1702

-12.1523

-2.3072

v a r i o u s p a r a m e t e r s change.

I n p a r t i c u l a r , t h i s information

can b e used t o o b t a i n a f i r s t - o r d e r e s t i m a t e o f t h e s o l u t i o n v e c t o r o f a problem i n v o l v i n g d i f f e r e n t p a r a m e t e r v a l u e s , once a s o l u t i o n f o r a g i v e n s e t o f p a r a m e t e r s h a s been o b t a i n e d . The p a r t i a l d e r i v a t i v e s o f t h e Lagrange m u l t i p l i e r s w i t h r e s p e c t t o t h e parameters a r e given i n Table 8 . 4 . 5 .

Again,

t h e s e can be used t o o b t a i n f i r s t - o r d e r e s t i m a t e ' s o f t h e Lagrange m u l t i p l i e r s o f a problem w i t h d i f f e r e n t p a r a m e t e r values.

I n p a r t i c u l a r , t h e r e l a t i v e e f f e c t s of t h e c o n s t r a i n t s

on t h e o p t i m a l v a l u e o f problems i n v o l v i n g d i f f e r e n t p a r a m e t e r v a l u e s can b e e s t i m a t e d .

F u r t h e r m o r e , it f o l l o w s t h a t t h e

p a r t i a l d e r i v a t i v e s o f t h e o p t i m a l m u l t i p l i e r s u1 and u2 w i t h r e s p e c t t o K1 and K2, r e s p e c t i v e l y , y i e l d t h e s e c o n d p a r t i a l

241


L . M.

T a b l e 8.4.5. Derivatives

Lagrange M u l t i p l i e r S e n s i t i v i t y :

Optimal

Constraint i

Par t i a 1s

Investment

*

aUi/aK2

*

aui/ac3

Workload

-0.0002

-0,1382

0.0006

0.1635

0.0000

0.0006

0.0002

0.0489

0.0000

0.0020

0.0003

0.0323

d e r i v a t i v e s of t h e o p t i m a l v a l u e f u n c t i o n w i t h r e s p e c t t o t h e p a r a m e t e r s K1

and K 2 u n d e r t h e p r e s e n t c o n d i t i o n s .

c a n be u s e d t o pro-

kind of information given i n Table 8 . 4 . 5 vide a second-order

Thus, t h e

estimate of t h e o p t i m a l v a l u e f u n c t i o n Z

*

f o r d i f f e r e n t v a l u e s of t h e s e parameters. T o i l l u s t r a t e and t e s t t h e a p p l i c a t i o n o f t h e t y p e o f

information provided here, t h e f i r s t p a r t i a l d e r i v a t i v e s with r e s p e c t t o c1 o f t h e o p t i m a l v a l u e f u n c t i o n Z components Q

i

*

,

the solution

and ri, and t h e Lagrange m u l t i p l i e r s u1 and u2

w e r e u s e d t o g i v e a f i r s t - o r d e r estimate o f t h e c o r r e s p o n d i n g s o l u t i o n v a l u e s a s s o c i a t e d w i t h t h e problem i n which t h e g i v e n v a l u e of c1 w a s i n c r e a s e d by 1 d o l l a r .

These estimates w e r e

compared w i t h t h e r e s p e c t i v e v a l u e s of t h e s o l u t i o n o b t a i n e d by a c t u a l l y s o l v i n g t h e p e r t u r b e d problem. summarized i n T a b l e 8.4.6.

The r e s u l t s are

Though t h e p e r t u r b a t i o n i s l a r g e

( t h e p a r a m e t e r b e i n g i n c r e a s e d by 100% o f i t s c u r r e n t v a l u e ) , t h e estimates are s e e n t o b e e x t r e m e l y a c c u r a t e , w i t h t h e e x c e p t i o n of t h e e s t i m a t e d r e o r d e r q u a n t i t y Q1.

Many u s e s

242


T a b l e 8.4.6. F i r s t - O r d e r E s t i m a t e s f o r a U n i t I n c r e a s e of Parameter E = e l , where F = 1 and ? = 2 Quantity F(e)

E s t i m at e a

F(e)

Actual

X Abs. e r r o r b

15.159

14.996

1.08

324

412

21.36

221

229

3.49

261

257

1.56

267

268

0.37

299

297

0. 67

417

420

0.71

0,0058

0.0057

1.75

0. 7865

0. 7 6 7 1

1.94

c o u l d b e made o f t h e e s t i m a t e d s o l u t i o n ; e . g . ,

s h o u l d it be

d e s i r a b l e t o s o l v e t h e p e r t u r b e d p r o b l e m a c c u r a t e l y , i t would g e n e r a l l y be computationally extremely advantageous t o use t h e e s t i m a t e d s o l u t i o n as a s t a r t i n g p o i n t . The c o m p l e t e e x p l o i t a t i o n o f t h i s s e n s i t i v i t y a n a l y s i s i n f o r m a t i o n now a v a i l a b l e w i l l depend l a r g e l y o n u s e r i n t e r e s t and i n g e n u i t y .

8.5 Additional Computational Experience with Applications

8.5.

243

A D D I T I O N A L COMPUTATIONAL EXPERIENCE W I T H APPLICATIONS

A s i n d i c a t e d i n C h a p t e r 5 , S e c t i o n 5.6,

t h e SENSUMT

program was used t o s o l v e and g e n e r a t e s e n s i t i v i t y i n f o r m a t i o n f o r an a n a l y s i s o f t h e S h e l l Dual problem (Armacost and F i a c c o , 1 9 7 6 1 , following i n i t i a l s u c c e s s f u l experiments with a v a r i e t y

o f s m a l l t e s t problems, i n c l u d i n g a d e t e r m i n i s t i c e q u i v a l e n t o f a c a t t l e f e e d problem (Armacost and F i a c c o , 1 9 7 4 ) .

Our

s u c c e s s w i t h t h e s e problems m o t i v a t e d u s t o e s t a b l i s h t h e p r a c t i c a l a p p l i c a b i l i t y o f NLP s e n s i t i v i t y c a l c u l a t i o n s i n g e n e r a l , and t h e g i v e n a l g o r i t h m i c approach i n p a r t i c u l a r , by a p p l y i n g o u r methods t o w e l l known models f o r m u l a t e d by o t h e r s and p u b l i s h e d i n t h e open l i t e r a t u r e .

The f i r s t s u c h e x p e r i -

ment was t h e a p p l i c a t i o n o f SENSUMT t o t h e Schrady-Choe i n v e n t o r y model (Armacost and F i a c c o , 1 9 7 8 ) ; r e s u l t s o f t h a t analysis a r e reported i n t h i s chapter.

Two a d d i t i o n a l models

have been t h e s u b j e c t o f numerous a d d i t i o n a l s e n s i t i v i t y s t u d i e s u s i n g SENSUMT:

a w e l l known G P model o f a stream-

w a t e r p o l l u t i o n abatement system, and a n o n l i n e a r s t r u c t u r a l d e s i g n model.

W e b r i e f l y i n d i c a t e t h e s t u d i e s t h a t have been

conducted w i t h t h e s e models. For a s t r e a m w a t e r p o l l u t i o n GP model, whose p r o t o t y p e was f o r m u l a t e d by Charnes and Gemmell ( 1 9 6 4 ) and e x t e n d e d and a n a l y z e d by Ecker (1975) u s i n g an upper Hudson R i v e r d a t a b a s e , F i a c c o and Ghaemi (1979a) f i r s t o b t a i n e d t h e o p t i m a l w a s t e t r e a t m e n t l e v e l s minimizing t h e a n n u a l c o s t .

(Ecker's r e s u l t s

w e r e obtained using a GP algorithm, while t h e l a t t e r study makes u s e o f t h e p e n a l t y f u n c t i o n a l g o r i t h m a p p l i e d t o t h e u s u a l convex e q u i v a l e n t o f t h e g i v e n GP problem.)

I t i s shown

t h a t t h e p o l i c y o f f i x e d d i s s o l v e d oxygen r e q u i r e m e n t y i e l d s

244


t h e minimum a n n u a l w a s t e t r e a t m e n t c o s t , r e l a t i v e t o two o t h e r environmental c o n t r o l p o l i c i e s .

Using SENSUMT, F i a c c o and

Ghaemi (197933) s u b s e q u e n t l y a n a l y z e d t h e s e n s i t i v i t y o f t h e o p t i m a l waste t r e a t m e n t c o s t t o c h a n g e s i n t h e 6 7 p a r a m e t e r s i n v o l v e d i n t h e f o r m u l a t i o n o f t h e model.

This a n a l y s i s i s

c o n d u c t e d f o r t h e f i x e d d i s s o l v e d oxygen r e q u i r e m e n t p o l i c y . I t i s shown t h a t t h e maximum a l l o w a b l e d i s s o l v e d oxygen de-

f i c i t i s u n e q u i v o c a l l y t h e p a r a m e t e r t o which t h e o p t i m a l

w a s t e t r e a t m e n t c o s t i s most s e n s i t i v e .

A l l of t h e many

p a r a m e t e r s i n v o l v e d i n t h e model are a n a l y z e d a n d , r e l a t i v e t o s t i p u l a t e d a s s u m p t i o n s , are shown t o have from s i g n i f i c a n t t o n e g l i g i b l e i m p a c t on t h e w a s t e t r e a t m e n t c o s t . The SENSUMT code w a s u t i l i z e d by F i a c c o and Ghaemi (1982) t o c o n d u c t a s e n s i t i v i t y a n a l y s i s o f a n o n l i n e a r programming problem a s s o c i a t e d w i t h t h e o p t i m i z a t i o n o f t h e s t r u c t u r a l d e s i g n o f a v e r t i c a l l y c o r r u g a t e d t r a n s v e r s e b u l k h e a d of a n o i l tanker.

Major a s s u m p t i o n s and r e s u l t s a s s o c i a t e d w i t h t h e

b a s i c s e n s i t i v i t y a n a l y s i s a p p r o a c h are v e r i f i e d f o r t h e subj e c t problem.

F i r s t - o r d e r s e n s i t i v i t y information i s used t o

estimate measures a s s o c i a t e d w i t h v a r i o u s p e r t u r b a t i o n s of t h e g i v e n problem d a t a .

D e t a i l s o f s t a b i l i t y and convergence

c h a r a c t e r i s t i c s o f t h e s o l u t i o n p o i n t s and t h e i r p a r t i a l der i v a t i v e s w i t h r e s p e c t t o t h e problem p a r a m e t e r s a r e a n a l y z e d . Using t h e r e s u l t s o f t h e p r e v i o u s a n a l y s i s o f t h e c o r r u g a t e d b u l k h e a d model, a c l o s e d form o f t h e o p t i m a l s o l u t i o n o f t h i s model i s d e r i v e d by F i a c c o and Ghaemi (1981a) as a funct i o n of t h e many d e s i g n p a r a m e t e r s .

The a n a l y t i c s o l u t i o n i s

shown t o b e v a l i d o v e r l a r g e c h a n g e s i n t h e p a r a m e t e r s .

This

d e m o n s t r a t e s t h e p o s s i b i l i t y o f making v e r y a c c u r a t e ( i n t h i s i n s t a n c e , e x a c t ) l o c a l estimates o f a markedly n o n l i n e a r

245

8.5 Additional Computational Experience with Applications

p a r a m e t r i c s o l u t i o n , once one s o l u t i o n i s found, even though t h e o r i g i n a l problem s t a t e m e n t might a p p e a r t o make t h i s prohibitive.

I t i s shown t h a t t h e l o c a l p a r a m e t r i c s o l u t i o n

o b t a i n e d c a n be e x t e n d e d t o i n c l u d e l a r g e changes i n t h e p a r a m e t e r s , p a r t i c u l a r l y when t h e p a r a m e t r i c p e r t u r b a t i o n s a r e r e s t r i c t e d and h i g h l y s t r u c t u r e d , a s i s t y p i c a l i n p r a c t i c a l applications.

A s i d e from t h e p r a c t i c a l v a l u e o f s u c h a r e s u l t ,

i t i s r e p o r t e d b e c a u s e t h i s problem i s w e l l p u b l i c i z e d and has been u t i l i z e d f o r some t i m e a s a t e s t problem.

Knowledge o f

t h e p r e c i s e s o l u t i o n f o r a range o f parameter values provides i d e a l information f o r f u r t h e r studies.

Chapter 9

Computable Optimal Value Bounds and Solution Vector Estimates for Parametric NLP Programs

9.1.

INTRODUCTION

In Chapter 2 , Section 2.5,

w e gave a b r i e f p e r s p e c t i v e on

s e v e r a l a p p r o a c h e s t o t h e t h e o r e t i c a l and c o m p u t a t i o n a l c a l c u l a t i o n o f o p t i m a l v a l u e and s o l u t i o n bounds.

W e briefly

summarized t h e d i r e c t i o n w e have t a k e n i n s e e k i n g computable p r o c e d u r e s f o r f i n i t e p a r a m e t e r changes, b a s e d e s s e n t i a l l y on e x p l o i t i n g t h e r i c h s t r u c t u r e o f t h e j o i n t l y convex p a r a m e t r i c program.

T h i s i d e a was developed i n F i a c c o (1981) and i s

presented i n t h i s chapter. Summarizing, a s i m p l e t e c h n i q u e i s p r o p o s e d f o r c a l c u l a t i n g p i e c e w i s e - l i n e a r c o n t i n u o u s g l o b a l upper and lower p a r a -

metric bounds on t h e o p t i m a l v a l u e of n o n l i n e a r p a r a m e t r i c programs t h a t have a convex o r concave o p t i m a l v a l u e f u n c t i o n . This provides a procedure f o r c a l c u l a t i n g parametric optimal v a l u e bounds f o r g e n e r a l nonconvex p a r a m e t r i c programs, whene v e r convex o r concave u n d e r e s t i m a t i n g o r o v e r e s t i m a t i n g problems can be c o n s t r u c t e d .

For t h e j o i n t l y convex program,

t h i s approach l e a d s immediately t o t h e c o n s t r u c t i o n o f a p a r a m e t r i c f e a s i b l e v e c t o r y i e l d i n g a computable and g e n e r a l l y s h a r p e r n o n l i n e a r o p t i m a l v a l u e upper p a r a m e t r i c bound.

246

247

9.2 Piecewise Linear Upper and Lower Optimal Value Bounds

Connections and e x t e n s i o n s o f w e l l known d u a l i t y r e s u l t s a r e developed t h a t l e a d t o c o n s t r u c t i v e i n t e r p r e t a t i o n s o f t h e bounds r e s u l t s and g e n e r a l l y s h a r p e r n o n l i n e a r p a r a m e t r i c lower An approach i s proposed f o r t h e c a l c u l a t i o n o f s o l u -

bounds.

t i o n p o i n t p a r a m e t r i c bounds, b a s e d on o u r a b i l i t y t o c a l c u l a t e t h e o p t i m a l v a l u e f u n c t i o n bounds. COMPUTABLE PIECEWISE LINEAR UPPER AND LOWER OPTIMAL VALUE BOUNDS

9.2.

The a u t h o r r e c e n t l y p r o p o s e d a s i m p l e p r o c e d u r e f o r o b t a i n i n g computable g l o b a l upper and lower p a r a m e t r i c bounds on t h e o p t i m a l v a l u e o f g e n e r a l n o n l i n e a r p a r a m e t r i c programs f o r which o v e r e s t i m a t i n g and u n d e r e s t i m a t i n g problems can be c o n s t r u c t e d t h a t have e i t h e r a convex o r concave o p t i m a l v a l u e function.

The c l a s s o f programming problems t h a t can be s o

bounded t u r n s o u t t o be e x t r e m e l y l a r g e , and hence t h e potent i a l applications are vast.

The i d e a i s b a s e d on p r o p e r t i e s

t h a t a r e e l e m e n t a r y and w e l l known b u t a p p a r e n t l y have n o t been p r e v i o u s l y e x p l o i t e d o r s y s t e m a t i c a l l y developed f o r t h e i n d i c a t e d purpose. The bounding p r o c e d u r e may p e r h a p s b e most e a s i l y conveyed g e o m e t r i c a l l y , f o r problems whose o p t i m a l v a l u e f u n c t i o n i s convex. min

C o n s i d e r t h e g e n e r a l problem X

f(x,

E)

s.t. g ( x ,

E)

2 0, h(x,

I f P 3 ( € ) i s j o i n t l y convex, i . e . , and

E,

E)

= 0.

i f f i s j o i n t l y convex i n x

t h e components of g a r e j o i n t l y concave i n x and

t h o s e o f h j o i n t l y a f f i n e i n x and

P3 ( € 1

E,

E,

and

t h e n i t h a s been known

f o r some t i m e (Mangasarian and Rosen, 1 9 6 4 ) t h a t t h e o p t i m a l

248

9 Optimal Value Bounds and Solution Vector Estimates

/ Y =e2

?

y = f*(c)

--L/

--/

Fig.

C onv e x o p t i m a l v a l u e f u n c t i o n and b o u n d s .

9.2.2.

*

v a l u e f u n c t i o n f ( € 1 o f P 3 ( E ) i s a convex f u n c t i o n of

Under

E.

t h e s e circumstances a t y p i c a l graph o f t h e optimal value f f r e l a t i v e t o a g i v e n component E~

i s shown i n F i g . 9 . 2 . 1 . Suppose w e have e v a l u a t e d f values of

E,

*

(E)

of

with

E

E

j

*

of

fixed, j # i,

and i t s s l o p e a t two

t h o s e f o r which component

E~

( f o r s i m p l i c i t y ) t h e o t h e r components o f

= 0 and E

Fit and

a r e f i x e d a t 0.

Then, under g e n e r a l c i r c u m s t a n c e s , t h e r e e x i s t t h e s u p p o r t s y = a , ( € ) and y = ! L 2 ( ~ ) o f ( t h e e p i g r a p h o f ) f * ( ~ a) t and

E~

= Ti,

r e s p e c t i v e l y , above which f

l i n e y = ! L 3 ( ~ ) through LO, f

*

(E)

must l i e , so t h a t f

*

f * (0)I (E)

,

[Ti,

*

(E)

Ei.

= 0

must l i e , and t h e

f * (Fi) I

,

below which

must l i e i n t h e c r o s s - h a t c h e d

t r i a n g u l a r a r e a i n d i c a t e d i n F i g . 9 . 2 . 1 when 0 and

E~

E~

v a r i e s between

I f f * ( ~ i) s d i f f e r e n t i a b l e , t h e n c o n v e x i t y o f f * ( ~ )

249

9.3 Parametric Solution Vector and Sharper Convex Upper Bound

implies t h a t f*(E) 2 f*(T)

+

V-f*(T) t

(E

-

T ) , and i t f o l l o w s t h a t

1

f

*

* af (0) + aEi

(0)

i'

n o t d i f f e r e n t i a b l e , t h e n any s u b g r a d i e n t of f any v e c t o r v s u c h t h a t f * ( E ) 2 f * ( E ) +

through [ T I f * (T) I. derivative of f

*

(E)

- E) ,

Vt ( E

would s u f f i c e t o y i e l d a g l o b a l l i n e a r lower bound on f

*

*

(E)

I n p a r t i c u l a r , w e can use t h e d i r e c t i o n a l

a t any p o i n t i n t h e i n t e r i o r o f i t s domain,

s i n c e t h i s i s known t o e x i s t ( R o c k a f e l l a r , 1 9 7 0 b ) . s i m i l a r c o n s t r u c t s apply i f f

*

(E)

i s concave.)

(Obviously,

Thus, i n t h e

i n t e r i o r o f i t s domain, e a s i l y computable upper and lower bounds on f

*

f o r any f i n i t e changes i n

E

can be p r o v i d e d by

computing two s o l u t i o n s o f P 3 ( E ) c o r r e s p o n d i n g t o two v a l u e s of

E

and t h e a s s o c i a t e d s u p p o r t s o f t h e e p i g r a p h o f f

t h e s e l e c t e d values of t h e parameters.

*

(E)

at

A s n o t e d , most of t h e

r e q u i r e d i n f o r m a t i o n i s p r o v i d e d by most s t a n d a r d NLP a l g o r i t h m s when a s o l u t i o n h a s been d e t e r m i n e d f o r t h e g i v e n parameter v a l u e s and t h e a s s o c i a t e d o p t i m a l Lagrange m u l t i pliers exist. 9.3.

ESTIMATES OF A PARAMETRIC SOLUTION VECTOR AND A SHARPER CONVEX UPPER BOUND

An e x t r e m e l y i m p o r t a n t b y p r o d u c t o f t h i s bounds c a l c u l a t i o n i s i t s immediate a p p l i c a b i l i t y t o r n u l t i p a r a m e t r i c p e r t u r -

bation and t h e a v a i l a b i l i t y o f an a s s o c i a t e d f e a s i b l e parametric vector.

Denote by

parameter v e c t o r

E

e E

E~

k

.

and

E,

two d i s t i n c t v a l u e s o f t h e

Suppose F ( E ~i )s f e a s i b l e t o problem

P 3 ( ~ 1 ) and x ( E ~ ) s o l v e s P 3 ( ~ l ) l w h i l e X ( E , ) i s f e a s i b l e t o

250


P3(~2)w r ith

+

Define € ( a )

*

3 (E) is jointlyconvex. x ( a ) = a ? i ( ~ , ) + (1 - ~ ) X ( E ~ ) ,

its solution.

x ( E ~ )

-

(1

Assume P

N

+ (1 - a ) x ( s l ) , where

and x ( a ) =

0 5 a 5 1.

I t can

e a s i l y be shown t h a t ? ( a ) (hence a l s o x * ( a ) ) i s i n R [ E ( ~ ) ] f o r a 5 1 where R ( E ) i s t h e f e a s i b l e r e g i o n o f P 3 ( ~ ) . ( I n

any 0

t h e t e r m i n o l o g y o f p o i n t t o s e t maps (see S e c t i o n 2 . 2 )

,

means t h e g r a p h G o f R i s a convex s u b s e t o f Ek

i.e.,

G =

{(E,

x)

IE

E

E

k

,x *

E

En,

x

R ( E ) ) i s a convex s e t . )

this

Furthermore,

i t f o l l o w s t h a t f [ x ( a ), € ( a ) ] i s a convex f u n c t i o n o f a t h a t l i e s above t h e (unknown) o p t i m a l v a l u e f

*

[ € ( a ) ]and on o r be-

*

l o w t h e known l i n e a r upper bound y = R 3 [ ~ ( a ) ]o f f [ ~ ( a ) ] f,o r 0 5 a 5 1, w i t h f [ x * ( O ) , ~ ( 0 1 1= f*(E1) and f [ x * ( l ) , ~ ( 1 1 1= f

*

(E2).

These f a c t s a r e a c t u a l l y i n v o l v e d i n d e m o n s t r a t i n g t h e convexity of f

*

and i t s domain (Mangasarian and Rosen, 1 9 6 4 ) S i n c e P 3 ( ~ )i s j o i n t l y

and a r e r e a d i l y shown a s f o l l o w s . convex, i t f o l l o w s t h a t g [ ? ( a ) , € ( a ) ] :g [ a X ( E 2 )

+

(1

>

ag[51(E2),

>

0.

-

E21

~ ) X ( E ~ ) , + (1

+

(1

-

a)g[X(E,)

r

Ell

The f i r s t i n e q u a l i t y (meaning t h a t t h e i n e q u a l i t y h o l d s f o r e a c h component o f g ) f o l l o w s from j o i n t c o n c a v i t y o f e a c h gir and t h e second from t h e f e a s i b i l i t y o f X ( E ~ f) o r P 3 ( ~ 1 ) and f o r P 3 ( ~ 2 ) . Since t h e equality c o n s t r a i n t s a r e j o i n t l y a f f i n e i f P3(€) i s j o i n t l y convex, t h e a n a l o g o u s r e l a t i o n s h o l d f o r h [ z ( a ), equalities.

E

( a )]

Hence,

, with

z(a) E

t h e i n e q u a l i t i e s r e p l a c e d by

R [ E ( a )1.

25 I


If f(x,

E)

i s j o i n t l y convex, t h e n

*

t h e f i r s t i n e q u a l i t y f o l l o w i n g from t h e f a c t t h a t x ( a ) E R [ ~ ( a ) l which , f o l l o w s from t h e p r e c e d i n g r e s u l t , and t h e second from t h e c o n v e x i t y assumption. (x,

E)

and t h e f a c t t h a t ( x * ,

€ ( a ) ] i s convex i n a .

*

E)

The c o n v e x i t y o f f i n

i s a f f i n e i n a imply f [ x * ( a ) ,

Thus, t h e s t a t e d r e s u l t s h o l d .

The d e p i c t i o n o f f [ € ( a ) ] and i t s bounds a r e shown i n F i g . 9.3.1.

For s i m p l i c i t y , w e d e n o t e L i [ e ( a ) ] by a i ( a ) .

f i g u r e i s comparable t o Fig.

*

i - t h component o f x ( a )

I

E

9.2.1.

This

W e have a l s o d e p i c t e d t h e

R [ E ( ~ ) ]i n t h e f i g u r e .

I

1

*

Note t h a t

a

B o u n d s o n t h e o p t i m a l 3 a l u e f [€(a)] and t h e F i g . 9.3.1. i - t h component of t h e f e a s i b l e v e c t o r x ( a ) , when t h e problem i s s o l v e d for a = 0, 1. A l l s o l i d l i n e s r e p r e s e n t computable quantities.

252


t h e a s s u m p t i o n t h a t P J ( ~ )i s j o i n t l y convex c a n be r e l a x e d . I t s u f f i c e s t h a t P [ ~ ( a ) be l j o i n t l y convex i n x and a t o o b t a i n

t h e s e r e s u l t s and t h i s d e p i c t i o n f o r

E

=

E

(a).

T h i s remarkably r e g u l a r b e h a v i o r i s e x p l o i t a b l e i n many

*

Since x ( a ) is t h e r e s u l t of l i n e a r i n t e r p o l a t i o n

ways.

between two g i v e n s o l u t i o n s , i f a d d i t i o n a l s o l u t i o n s are obt a i n e d c o r r e s p o n d i n g t o i n t e r m e d i a t e v a l u e s o f a , 0 < a < 1, then it is apparent t h a t a piecewise-linear

continuous f e a s i b l e

p a r a m e t r i c v e c t o r s ( a ) can be o b t a i n e d by l i n e a r i n t e r p o l a t i o n between c o n t i g u o u s s o l u t i o n s .

Further, t h e function f

*

[E

( a )I

w i l l l i e between p i e c e w i s e - l i n e a r c o n t i n u o u s u p p e r and l o w e r p a r a m e t r i c bounds t h a t c a n be r e a d i l y computed, and t h a t w i l l become t i g h t e r w i t h e a c h new i n t e r m e d i a t e s o l u t i o n t h a t i s computed, e . g . ,

a t a v a l u e o f a where t h e d e v i a t i o n between

t h e upper and l o w e r bounds i s g r e a t e s t .

*

Hence, i f a p r e s c r i b e d

e r r o r t o l e r a n c e i n e s t i m a t i n g f [ € ( a ) ]f o r 0 5 a 5 1 i s g i v e n , t h e n w e need o n l y s o l v e P 3 [ (~a )1 f o r s e v e r a l v a l u e s o f a i n

[O, 1 1 , u n t i l t h e bounds on f * [ E ( a )1 do n o t d e v i a t e by more than t h e given t o l e r a n c e . I t f o l l o w s from what w a s shown t h a t t h e o b j e c t i v e f u n c t i o n

f(x,

E)

a l o n g [ ; ( a ) , € ( a ) ] i s a piecewise-convex

continuous

f u n c t i o n , does n o t underestimate t h e a c t u a l optimal value

*

f u n c t i o n f [ € ( a ) ,] and h a s i t s g r a p h on o r below t h e known piecewise-linear

upper bound, o v e r t h e s p e c i f i e d r a n g e o f a .

Thus, w i t h e v e r y s o l u t i o n c o r r e s p o n d i n g t o an i n t e r m e d i a t e v a l u e o f a w e may e a s i l y c a l c u l a t e , i n a d d i t i o n t o b e t t e r piecewise-linear

c o n t i n u o u s upper and lower bounds, a more

a c c u r a t e p i e c e w i s e - l i n e a r c o n t i n u o u s f e a s i b l e estimate o f t h e p a r a m e t r i c s o l u t i o n v e c t o r a l o n g w i t h a more a c c u r a t e piecewise-convex

continuous overestimate of t h e optimal value

253


II

\

*

F i g . 9.3.2. B o u n d s on t h e o p t i m a l _ v a l u e f [ ~ ( a ) a] n d t h e - t h c o m p o n e n t of t h e f e a s i b l e v e c t o r x ( a l , when t h e p r o b l e m s s o l v e d f o r a = 0, a, 1 . A l l s o l i d l i n e s r e p r e s e n t comput a b 1 e quan t i t i e s

.

function.

W e i l l u s t r a t e t h e p a r a m e t r i c o p t i m a l v a l u e bounds

and a s s o c i a t e d f e a s i b l e v e c t o r s o l u t i o n e s t i m a t e a v a i l a b l e a f t e r s o l u t i o n s have been o b t a i n e d f o r t h r e e v a l u e of a , i n F i g . 9.3.2. T h i s p r o v i d e s a g e n e r a l approach f o r g e n e r a t i n g o p t i m a l v a l u e bounds and an a s s o c i a t e d p a r a m e t r i c s o l u t i o n v e c t o r e s t i m a t e f o r g e n e r a l n o n l i n e a r p a r a m e t r i c problems.

Most o f

t h e s e c o n c e p t s a r e w e l l known and s i m p l e , t h e e s s e n t i a l n o v e l t y b e i n g t h e i r e x p l o i t a t i o n , a d a p t a t i o n , and t a i l o r i n g t o

254


t h e p u r p o s e a t hand.

The c a l c u l a t i o n s i n v o l v e t h e manipula-

t i o n of i n f o r m a t i o n g e n e r a l l y a l r e a d y p r o v i d e d by s t a n d a r d so lution algorithms. There a r e , however, numerous i m p o r t a n t t h e o r e t i c a l and a l g o r i t h m i c r e f i n e m e n t s t h a t must b e made t o s h a p e t h e s e t e c h n i q u e s i n t o a c o h e r e n t , i n c i s i v e and e f f i c i e n t methodology. T h e i r development, v a l i d a t i o n , and c o m p u t a t i o n a l i m p l e m e n t a t i o n a r e b e i n g pursued.

But i t a l r e a d y a p p e a r s p l a u s i b l e t h a t an

abundance of s e n s i t i v i t y and s t a b i l i t y i n f o r m a t i o n can be c a l c u l a t e d f o r any s o l v a b l e NLP problem, w i t h r e l a t i v e l y l i t t l e a d d i t i o n a l computational e f f o r t .

The k i n d o f i n f o r m a t i o n t h a t

can be c a l c u l a t e d b e g i n s t o be somewhat comparable i n s c o p e and d e t a i l t o t h e e s s e n t i a l p o s t - o p t i m a l i t y p a r a m e t r i c s e n s i t i v i t y and range a n a l y s i s i n f o r m a t i o n t h a t i s h e a v i l y u t i l i z e d i n l i n e a r programming.

I n f a c t , it seems v e r y l i k e l y t h a t NLP

s t a b i l i t y r e s u l t s w i l l b r i n g new t e c h n i q u e s f o r e s t i m a t i n g t h e e f f e c t s o f l a r g e p e r t u r b a t i o n s i n l i n e a r programming a s w e l l . Many r e s e a r c h d i r e c t i o n s r e q u i r e e x p l o r a t i o n i n d e v e l o p i n g t h e proposed bounds t e c h n i q u e s .

W e mention a few:

(1) s y s -

t e m a t i c a c c u m u l a t i o n o f c o m p u t a t i o n a l e x p e r i e n c e w i t h moderates i z e d j o i n t l y convex programs, a l l o w i n g f o r t h e s i m u l t a n e o u s perturbation of s e v e r a l parameters;

( 2 ) development o f t h e

o p t i m a l v a l u e bounding p r o c e d u r e f o r s e p a r a b l e and f a c t o r a b l e programs, based on known p r o c e d u r e s f o r f o r m u l a t i n g a j o i n t l y convex u n d e r e s t i m a t i n g problem f o r t h e g i v e n p a r a m e t r i c nonconvex problem, a p p l y i n g t h e bounding t e c h n i q u e t o t h e convex problem, and b r a n c h and bound and o t h e r t e c h n i q u e s f o r i m proving accuracy;

( 3 ) a d e e p e r a n a l y s i s o f c o n d i t i o n s on t h e

problem f u n c t i o n s t h a t a r e s u f f i c i e n t f o r t h e c o n v e x i t y o r

255

9.4 Connections between Optimal Value Bounds and Duality

concavity of the optimal value function f o r s p e c i f i e d , s i m p l e , e. g.

, vector)

(often

p e r t u r b a t i o n s of problem p a r a m e t e r s ,

a l l o w i n g f o r t h e a p p l i c a b i l i t y of t h e o p t i m a l v a l u e bounding p r o c e d u r e f o r nonconvex problems w i t h o u t r e q u i r i n g t h e cons t r u c t i o n o f convex u n d e r e s t i m a t i n g o r o v e r e s t i m a t i n g problems; ( 4 ) c o n t i n u e d d e m o n s t r a t i o n s o f p r a c t i c a l a p p l i c a b i l i t y ; and

(5) t h e f u r t h e r development o f c o m p u t a t i o n a l t e c h n i q u e s t h a t e x p l o i t s t a n d a r d NLP a l g o r i t h m c a l c u l a t i o n s analogous t o t h o s e i n t r o d u c e d by t h e a u t h o r f o r p e n a l t y f u n c t i o n a l g o r i t h m s , t h u s , h o p e f u l l y , s t i m u l a t i n g t h e r o u t i n e and w i d e s p r e a d use of s e n s i t i v i t y and s t a b i l i t y i n f o r m a t i o n f o r a l l s t a n d a r d NLP a l g o r i t h m s j u s t a s i n l i n e a r programming. CONNECTIONS BETWEEN OPTIMAL VALUE BOUNDS AND DUALITY

9.4.

The f o r e g o i n g o p t i m a l v a l u e bounds r e s u l t s have many p o i n t s of contact with d u a l i t y theory.

I n f a c t , t h e bounding pro-

c e d u r e and i t s c o m p u t a b i l i t y w e r e i n i t i a l l y s u g g e s t e d t o t h e a u t h o r by t h e f o l l o w i n g w e l l known c o n n e c t i o n s w i t h o p t i m a l i t y Suppose t h e problem i s a r h s

c o n d i t i o n s and d u a l i t y r e s u l t s . p e r t u r b a t i o n problem o f t h e form min X f ( x )

s . t . g ( x ) 1.

P1

E,

An immediate c o n n e c t i o n e a s i l y f o l l o w s :

i f t h e Karush-Kuhn-

Tucker c o n d i t i o n s h o l d a t [ x ( E ) , u ( E ) ] and i f unique and d i f f e r e n t i a b l e n e a r E =

0.

E =

[x(E),u(E)I

0 , t h e n V,f*(E) =

is

u ( E ) n~e a r

Suppose a l s o t h a t P 1 ( ~ ) i s convex i n x and f o r

s i m p l i c i t y assume t h a t o n l y one p a r a m e t e r E

(€1

1. 0 , t h a t a s o l u t i o n

X(E)

E~

i s perturbed, t h a t

of P 1 ( ~ ) e x i s t s ( f o r a l l

E

~

)

256


s a t i s f y i n g t h e Karush-Kuhn-Tucker

c o n d i t i o n s , and t h a t g [ x ( f ) ]

*

Then i t f o l l o w s t h a t t h e o p t i m a l v a l u e f

= E.

and n o n d e c r e a s i n g a s a f u n c t i o n of

E

,as

i s convex

i n Fig. 9.2.1.

At

= Ti and f = 0, j # i, i j and y = Q 2 ( ~ )a s d e p i c t e d i n F i g .

t h e s o l u t i o n s x ( 0 ) and x ( Y ) , where t h e lower bounds y = Q 1 ( € )

~

(f)

f

a p p l y , and have n o n n e g a t i v e s l o p e s c o r r e s p o n d i n g t o 1 2 n o n n e g a t i v e o p t i m a l Lagrange m u l t i p l i e r s , s a y ui and u i r * 1 respectively. C l e a r l y , w e o b t a i n a , ( € ) = f (0) + u i ~ i a n d 9.2.1

Q2(f)

= f

*

(F) +

u

2

~

(

-E T~i ) .

I t i s c l e a r from F i g .

9.2.1

that

z

any s u c h t a n g e n t t o t h e g r a p h o f f " ( E ) w i l l i n t e r s e c t t h e y a x i s a t o r below t h e o p t i m a l v a l u e f

*

(E)

at

E

= 0.

I n f a c t , i t may be i n f e r r e d from t h e a s s u m p t i o n s and t h e figure that m

y

-

2

UiEi

= f

*

(El

-

(9.4.1)

m i=l

= L t X ( E ) r u ( T ) r 01,

where x ( 0 ) s o l v e s P1 ( 0 ) w i t h a s s o c i a t e d o p t i m a l Lagrange 1 m u l t i p l i e r u ( 0 ) s u c h t h a t u i ( 0 ) = u i r and x ( E ) s o l v e s P1(F) w i t h a s s o c i a t e d o p t i m a l Lagrange m u l t i p l i e r u(.) u.

(z)

P1(0).

2 = uir

such t h a t

L d e n o t i n g t h e u s u a l Lagrangian a s s o c i a t e d w i t h

Hence, t h e 111 and R 2 y - i n t e r c e p t s are L [ x ( O ) , u ( o ) , 0 1

and L[x(EI) , u ( 5 ) , 0 1 , r e s p e c t i v e l y .

I t i s o b v i o u s from t h e

f i g u r e and t h e assumptions t h a t f * ( O ) = L [ x ( O ) ,

U(0)r

01.

257

9.4 Connections between Optimal Value Bounds and Duality

Dual p r o b l e m s i n v o l v e d t h e d e t e r m i n a t i o n o f t h e s l o p e (Lagrange m u l t i p l i e r ) of a s u p p o r t i n g l i n e ( o r p l a n e , i n general) l i k e y =

a,(€)

s u c h t h a t t h e minimal L a g r a n g i a n v a l u e

o v e r x ( i . e . , t h e y i n t e r c e p t ) i s maximized.

For example, t h e

Wolfe d u a l (Wolfe, 1 9 6 1 ) o f t h e convex program minx f ( x )

s. t. g ( x ) 1. 0

(PI

i s g i v e n by

max

(x,u)

s . t . VXL(X, u ) = 0 , u 2 0 ,

L(x, u)

where L(x, u ) = f ( x )

-

(D) It is

Cu.g. ( x ) , t h e u s u a l Lagrangian. 1 1

e a s i l y shown t h a t (9.4.2)

w h e r e R i s t h e f e a s i b l e r e g i o n of p r o b l e m P a n d t o t h e f e a s i b l e s e t F$, o f D.

(G, G)

belongs

Under a p p r o p r i a t e r e g u l a r i t y

conditions it a l s o follows t h a t min f ( x ) = max L(y, u ) , xeR ( Y , u ) '% a n d t h e s e c o n d i t i o n s e s s e n t i a l l y summarize t h e d u a l i t y relationship. To e s t a b l i s h a c o n n e c t i o n between t h i s d u a l i t y c o n c e p t and

t h e bounds d i s c u s s e d e a r l i e r a n d d e p i c t e d i n F i g . 9 . 2 . 1 ,

con-

s i d e r t h e Wolfe d u a l o f P 1 ( € ) , g i v e n by

s.t. VXL(X, u,

E)

-

= 0, u 2 0 ,

E ] and VxL = Vxf

-

For s i m p l i c i t y , s u p p o s e P 1 ( ~ ) h a s a s o l u t i o n f o r any

E.

where L ( x , u ,

E)

= f (x)

T u [g(x)

-

D1

i m p o r t a n t p r o p e r t i e s o f D 1 ( ~ ) s h o u l d be n o t e d : s t r a i n t s of D 1 ( € )

do n o t depend o n

E

(€1

T

u Vxg.

Two

( i )t h e con-

a n d ( i i )t h e d u a l i t y

258


r e s u l t s j u s t g i v e n imply t h a t t h e o p t i m a l v a l u e f

*

( € 1 of P 1 ( ~ )

i s bounded below by t h e o b j e c t i v e f u n c t i o n of D 1 ( E ) a t any i.e.,

f e a s i b l e p o i n t of D l ( 0 ) , f * ( E ) 1. L ( 2 ,

6,

E

= f(2) = GTE

f o r every

(2, 6)

-

GT[g(2)

+ L ( 2 , 6,

E

= 0,

El

(9.4.3)

0)

t h a t s a t i s f i e s VxL = 0 and u 1. 0.

e r a l i z e s t h e d u a l i t y r e s u l t (9.4.2) for

-

T h i s gen-

j u s t g i v e n (which a p p l i e s

e s s e n t i a l l y ) and p r o v i d e s immediate c o n n e c t i o n s and

e x t e n s i o n s o f t h e bounds r e s u l t s g i v e n e a r l i e r . Feasible p o i n t s of D1(€)

Suppose t h e problem P 1 ( ~ ) i s s o l v e d

perturbations of P1(0). a t T by x ( T ) .

may be found by s o l v i n g

Then, under a p p r o p r i a t e r e g u l a r i t y c o n d i t i o n s

t h e r e e x i s t s u ( T ) 1. 0 s u c h t h a t VXL[x(E), u ( T ) ,

-

E]

= 0 ; hence

T

[ x ( T ) , u ( E ) I i s d u a l f e a s i b l e , and f u r t h e r , u ( S ) [ g [ x ( ? ) I W e n o t e i n p a s s i n g t h a t i f t h e s e Karush-Kuhn-Tucker

= 0.

-

d i t i o n s hold, then f * (E) = L [ x ( Z ) , u ( s ) , € 1 ,

- El con-

and hence t h e

above r e l a t i o n s imply t h a t [ x ( T ) , u ( T ) 3 s o l v e s t h e d u a l D1 (T) of P1(E).

R e t u r n i n g t o t h e bounds r e l a t i o n s h i p , w e o b s e r v e

t h a t [ x ( T ) , u ( T ) ] i s a f e a s i b l e point of the dual D 1 ( ~ ) of P1(€)

f o r any

E.

Thus, from (9.4.3) w e c o n c l u d e t h a t

f * ( E ) 2 L[X(E), u ( + ) ,

El

= f[x(T)I T = u(T) E

-

+

u(T)T(S

-

E)

L [ x ( T ) , U ( E ) , 01.

(9.4.4)

As o b s e r v e d , under a p p r o p r i a t e r e g u l a r i t y a s s u m p t i o n s t h i s bound i s m e t when

strict for a l l

E

E

= 7 , with t h e i n e q u a l i t y generally being

# T i n t h e domain o f f

*

(E).

I t i s c l e a r t h a t i f P 1 ( € ) s a t i s f i e s t h e c o n d i t i o n s imposed

a t t h e b e g i n n i n g o f t h i s s e c t i o n t h e n t h e lower bound on f t h a t w e have j u s t o b t a i n e d i n (9.4.4)

*

(E)

i s t h e same a s ! L z ( ~ ) i n

259

9.5 Nonlinear Dual Lower Bounds

t h e bound y =

i n F i g . 9.2.1

t h a t w e previously calculated

The bound (9.4.4) i s more g e n e r a l i n a l l o w i n g t h e

i n (9.4.1).

p e r t u r b a t i o n o f t h e e n t i r e parameter v e c t o r

E,

one component a s i n (9.4.1) and F i g . 9 . 2 . 1 . c o r r e s p o n d e n c e w i t h t h e bound y =

a,(€)

r a t h e r than only

Of c o u r s e , t h e

i n F i g . 9.2.1

(9.4.1) is a l s o e s t a b l i s h e d a s above, r e p l a c i n g

F

and

by 0.

Sum-

m a r i z i n g , i t f o l l o w s t h a t a l i n e a r lower bound on t h e o p t i m a l value of f*(E) of P 1 ( ~ ) a t 7, supporting t h e set of (y,

E

(E)

-

E)

such

,

c o r r e s p o n d s t o t h e Lagrangian Wolfe d u a l v a l u e

, where

x is e v a l u a t e d a t a s o l u t i o n x ( F ) o f P1 ( € 1 f o r

t h a t y 1. f o f P1

*

(E)

= E , w i t h u e q u a l t o t h e a s s o c i a t e d o p t i m a l Lagrange m u l t i -

Assuming t h a t [ x ( T ) , u ( F ) I s a t i s f i e s t h e Karush-

p l i e r u(E).

Kuhn-Tucker c o n d i t i o n s , i t f o l l o w s t h a t f* ( E ) = L [ x ( E ) , u ( s ) ,

E]

and [ x ( F ) , u ( F ) ] i s a f e a s i b l e p o i n t of D 1 ( € ) *

for a l l

E

in

t h e domain o f f " ( € 1 . 9.5.

NONLINEAR DUAL LOWER BOUNDS

This d u a l i t y i n t e r p r e t a t i o n leads t o f u r t h e r extensions of t h e bounds r e s u l t s .

Observe a g a i n t h a t t h e two e s s e n t i a l

properties of t h e dual D1(€) bounds on f

*

(E)

t h a t l e a d t o computable lower

a r e t h e d u a l lower bound (9.4.3) and t h e l a c k

o f dependence of t h e c o n s t r a i n t s o f D 1 ( € )

on

E.

This permits

t h e c a l c u l a t i o n o f t h e p a r a m e t r i c lower bound (9.4.3) whenever

any feasible point

( 2 , 6) of

D1(€)

h a s been computed.

t h e s e c o n d i t i o n s and r e s u l t s a l s o h o l d f o r t h e d u a l t h e more g e n e r a l p a r a m e t r i c problem min

X

where ? =

f(x)

s . t . g ( x ) 1. F ( E ) ,

T(E)is

an m-component v e c t o r f u n c t i o n .

But

s l ( ~of)

260


Assuming

pl(~) is

ma x

L(xr

(Xru)

Feasible points

ur

convex i n x , t h e Wolfe d u a l i s

-

8) s . t . V X ~ ( x U, ,

(x, u) o f

El(€)

E)

-

~ ~ ( € 1

u 2 0.

= 0,

a g a i n do n o t depend on

E

and a

bound o f t h e form (9.4.3) may be c a l c u l a t e d a n a l o g o u s l y , y i e l d i n g f o r any d u a l f e a s i b l e p o i n t

(z, 5 )

El(€)

of

the

inequality f*(E) 2 L G , 5, 7 ) = f ( z )

-

= -Tu g(E) i n t e r m s of t h e c o n s t i t u e n t s of

GT[9(Z)

+

-

LG,

G,

F1(€)

and

?(E)

1

0)

(9.5.1)

C1 ( € 1 .

In particu-

l a r , by analogy t o t h e p r e v i o u s t r e a t m e n t w i t h r e s p e c t t o P1(€) and D 1 ( ~ )t h a t g i v e s (9.4.4), w e c o n c l u d e from (9.5.1) t h a t f * ( E ) 2 L [ x ( ' i ) , U(E) ,

-

= U ( a T g E )

El

+ L[X(E)r u(E),

-

(9.5.2)

OIr

where f * and L now r e f e r t o t h e o b j e c t i v e f u n c t i o n s o f D1, r e s p e c t i v e l y , and x ( T ) now s o l v e s

o p t i m a l Lagrange m u l t i p l i e r u ( F ) .

f[x(E)I

-

I f t h e Karush-Kuhn-Tucker

e q u a l i t y when

E

= F.

g e n e r a l i z e (9.4.3) for a l l

E

- a ( € ) ] and ,

u ( S ) [:(+)

i n t h e domain o f f

o f a l l components o f

(9.5.2)

E,

*

becomes

o f c o u r s e (9.5.2) h o l d s w i t h

The bounds g i v e n by (9.5.1)

and (9.4.4).

and

1 (a) w i t h a s s o c i a t e d

conditions hold, then t h e right-hand s i d e of T

p1

and (9.5.2)

They a r e g l o b a l ( i . e . , v a l i d allow f o r the perturbation

(E)),

and do n o t r e q u i r e any c o n v e x i t y re-

s t r i c t i o n s on ? ( € ) . I t i s n o t e d , however, t h a t i f t h e components o f

F ( E )a r e

convex o r concave, t h e n t h e (more g e n e r a 1 ) l o w e r bound (9.5.1)

i s a c c o r d i n g l y convex o r concave. Lagrangian ( d u a l ) lower bound on f

*

I n t h e former c a s e , t h e (E)

w i l l obviously coincide

261


w i t h o r b e s h a r p e r t h a n t h e bound d e r i v e d from i t s l i n e a r i z a t i o n (as i n Fig. 9.2.1) f

*

a t any v a l u e o f

E

( € 1 , o v e r a n y s u b s e t o f t h e domain o f f

*

i n t h e domain o f (E).

In the latter

case, l i n e a r i n t e r p o l a t i o n between t w o p a r a m e t e r v a l u e s would y i e l d a l i n e a r lower bound on ( 9 . 5 . 1 )

-

and h e n c e o n f

*

(E)

.

I t s h o u l d be n o t e d t h a t t h e d u a l i t y r e s u l t s g i v e n f o r

P1(€) e x t e n d i n a n o b v i o u s manner t o t h e more g e n e r a l inequality-equality

s . t . g ( x ) 1. q

rnin X f ( x )

- (€1

c o n s t r a i n e d r h s convex program of t h e form

( ~, )h ( x )

=

G(E),

P2

-

where h ( x ) i s a f f i n e i n x , r e c a l l i n g t h a t t h e Wolfe d u a l o f P 2 ( ~ )i s

where L now d e n o t e s t h e L a g r a n g i a n L ( x , U,

of

W,

7)

F 2 ( € )and 7

=

d o n o t depend on

f(x)

-

T

u [g(x)

[ G ( E ), h ( E ) 1. ,.s

E,

-

T(E)I

+ wT [ h ( x )

-

c(E)1

A g a i n , t h e c o n s t r a i n t s of

and a bound o f t h e form ( 9 . 5 . 2 )

Ej2(€)

applies.

The f a c t t h a t n o c o n v e x i t y ( o r c o n c a v i t y ) r e s t r i c t i o n s a r e needed on

-

;(E)

or

X(E)

t o c a l c u l a t e bounds may g r e a t l y f a c i l i -

t a t e t h e c a l c u l a t i o n of u s e f u l bounds f o r p r o g r a m s o f t h e form

P1(€) o r

F 2 ( ~ t)h a t

are n o t convex i n x.

In particular,

it

s u f f i c e s t o o b t a i n a convex u n d e r e s t i m a t i n g problem i n x ,

i.e.

, without

altering

F(E)o r

%(E).

The d u a l i t y r e s u l t s j u s t

g i v e n c a n be a p p l i e d t o t h i s p r o b l e m , y i e l d i n g a lower bound of t h e form ( 9 . 5 . 1 )

o n i t s o b j e c t i v e f u n c t i o n v a l u e , hence on

262


t h e o p t i m a l v a l u e of t h e o r i g i n a l problem, whenever a d u a l f e a s i b l e p o i n t of t h e u n d e r e s t i m a t i n g problem h a s been calculated. I t i s s t r e s s e d t h a t it i s n o t necessary t o solve t h e primal

problem i; ( E ) o r 1

P 2 ( & )f o r

some v a l u e o f

i n t e r e s t t o o b t a i n a lower bound on f

*

i n t h e domain of

E

It suffices t o

(E).

o b t a i n any d u a l - f e a s i b l e p o i n t t o e v a l u a t e t h e p a r a m e t r i c lower bound g i v e n i n ( 9 . 5 . 1 ) .

T h i s f a c t i s e x p l o i t a b l e by any

NLP a l g o r i t h m t h a t c a l c u l a t e s a sequence o f d u a l f e a s i b l e p o i n t s i n approaching a s o l u t i o n .

I n p a r t i c u l a r , t h e penalty

f u n c t i o n c a l c u l a t i o n s t h a t w e have h e a v i l y u t i l i z e d f o r t h e a l g o r i t h m i c e s t i m a t i o n o f s e n s i t i v i t y a n a l y s i s ( F i a c c o , 1976) a l s o p r o v i d e an e x c e l l e n t example o f t h i s b a s i c i d e a o f an i t e r a t i v e a l g o r i t h m i c a p p r o x i m a t i o n o f o p t i m a l v a l u e bounds. For example, i f a sequence o f d u a l - f e a s i b l e p o i n t s I [ x k ( F ) , uk (F)] 1 i s g e n e r a t e d f o r

El (T)when

k

=

1, 2 ,

s o l u t i o n algorithm i n t h e process of s o l v i n g

... by

t h e given

F1(E),a s

is

c h a r a c t e r i s t i c a l l y t r u e o f p e n a l t y f u n c t i o n s ( F i a c c o and McCormick, 1968; see a l s o Theorem 6 . 2 . 1

i n t h i s book), then w e

c o u l d c o n c l u d e from ( 9 . 5 . 1 ) t h a t f o r e a c h k f * ( d 2 Uk(T)tiJ(E)+ L[Xk(E),

Uk(S),

o v e r t h e domain o f f * ( ~ ) p, r o v i d e d t h a t

01

F1(&) i s

convex i n x.

(The i n e q u a l i t y a p p l i e s even i f F i s r e p l a c e d by Fk, i . e . , even i f F changes f o r e a c h k . )

[x, (El, uk ( E ) 11 con-

Suppose

v e r g e s t o [ x (F), u ( F ) ] , where x ( F ) s o l v e s

Pl ( F )

with associated

d u a l m u l t i p l i e r u ( F ) , a s g e n e r a l l y happens f o r p e n a l t y f u n c t i o n d u a l - f e a s i b l e c o n v e r g e n t subsequences.

Then o f c o u r s e t h e

above bound a p p r o a c h e s f * ( ~ ) u ( E ) ~ ~ ~+ ( LE[ )x ( F ) , u ( T ) , 0 1 if

263


t h e problem f u n c t i o n s a r e c o n t i n u o u s . s a t i s f i e s t h e Karush-Kuhn-Tucker

[ x ( F ) , u(F)1

If

c o n d i t i o n s , t h e n t h e bound

i s s a t i s f i e d e x a c t l y a t F. The d u a l problem and a r e s u l t i n g lower bound can a l s o be g i v e n f o r t h e g e n e r a l p a r a m e t r i c program P 3 ( ~ )t h a t i s convex i n x.

The problem and i t s d u a l a r e

minx f ( x ,

s.t. g ( x ,

E)

2 0, h(x,

E)

P3 (€1

= 0

E)

and max

(X?U?W)

L ( x , u, w ,

E)

s . t . V X L ( X , u, w ,

E)

= 0,

u 1. 0 , where now L ( x , u , w ,

E)

= f (x,

E)

-

u g(x,

E)

+ wTh ( x ,

D3(c)

As

E).

b e f o r e , t h e o p t i m a l v a l u e of P 3 ( ~ )is bounded below by any d u a l - f e a s i b l e v a l u e o f L ( x , u, w , f * ( E ) 1. ~

[ f i ( E ) ,

i.e.

€1,

~ ( € 1 ,G ( E ) ,

(2, G, G)

f o r any p o i n t

E),

satisfying the dual constraints.

d i f f e r e n c e now i s t h a t t h e d u a l - f e a s i b l e p o i n t e r a l l y depends on

E

gen-

and may n o t be computable, even o v e r a

s m a l l s u b s e t o f t h e domain o f noted i n Section 9.2, can be c a l c u l a t e d i f f L1

(2, 6, G)

The

E

of i n t e r e s t .

Nonetheless, a s

p i e c e w i s e - l i n e a r upper and lower bounds

*

(E)

i s convex.

The l i n e a r s u p p o r t s

and L 2 f o r P 3 ( ~ )were based s o l e l y on t h i s c o n v e x i t y

assumption.

A d u a l i t y i n t e r p r e t a t i o n c a n a g a i n be g i v e n ,

however, when P 3 ( € ) i s j o i n t l y convex.

Motivation f o r t h i s i s

t h e i d e a t h a t w e may b e a b l e t o c h a r a c t e r i z e any l i n e a r lower bound s u c h as g i v e n i n F i g . 9 . 2 . 1

a s t h e graph of t h e optimal

v a l u e a s s o c i a t e d w i t h some p a r a m e t r i c l i n e a r program. convexity of f

*

(E)

The

and i t s r e l a t i o n s h i p t o t h e Lagrangian

s u p p o r t t h i s i d e a and i n f a c t immediately y i e l d an e q u i v a l e n t

264


L a g r a n g i a n bound.

Suppose [ x ( E ) ,

Karush-Kuhn-Tucker

c o n d i t i o n s and f

u(E),

*

(6)

w(E)]satisfies the

is d i f f e r e n t i a b l e .

These c o n d i t i o n s a n d t h e f o l l o w i n g c o n s e q u e n c e s h o l d u n d e r t h e f * ( E ) = L [ x ( E ) , u ( E ) , ~ ( € 1 ,€ 1

a s s u m p t i o n s o f Theorem 3.2.2: a n d VEf

*

= V € L [ X ( E ) ,u ( E ) ,

(E)

it f o l l o w s t h a t f * ( E ) 2 f * ( F ) domain o f f

*

(E).

w(E),

+

€1.

Since f

V E f * ( T )( E

-

*

i s convex,

(E)

T),f o r

E,

F i n the

The g i v e n r e l a t i o n s h i p s i m p l y t h a t t h i s

) L[x(E), u(E), convexity condition i s equivalent to f * ( ~ = El

W(E),

(E

- T).

+

2 L [ x ( T ) , U(E), w m , FI

VEL[X(T),

U(F),

w ( T ) , TI

The g i v e n l o w e r bound o n f * ( E ) is p r e c i s e l y a n a l o g o u s

t o t h e l i n e a r lower bounds d e r i v e d e a r l i e r and s u g g e s t s t h e a p p r o p r i a t e l i n e a r program a n d d u a l i t y i n t e r p r e t a t i o n . C o n s i d e r t h e l i n e a r program d e r i v e d from t h e j o i n t l y convex program P3 ( E ) by l i n e a r i z i n g t h e o b j e c t i v e f u n c t i o n and b i n d i n g c o n s t r a i n t s a b o u t [ x ( F ) , F ] , t h u s y i e l d i n g ( w i t h nonbinding constraints deleted)

-

x(31

E l (E

-

F)

s . t . g [ x ( T ) , TI + V x g [ x ( E ) , F l [ x

-

x(S)I

(E

-

F ) 1. 0 ,

x ( F ) , TI [ x

-

x(F)I

-

F ) = 0.

min X f [ x ( F ) , F ]

+

VXf[x(F), F l [ x

+

VEf [ x ( F ) ,

-

x ( F ) , FI

X(F),

FI

(E

LP~(E)

S i n c e P 3 ( E ) i s j o i n t l y convex, i t f o l l o w s t h a t L P 3 ( ~ )i s a n u n d e r e s t i m a t i n g problem f o r P3

-* f

(E)

(E),

i.e.,

i s t h e optimal value of LP3(€).

-* f

(E) 5 f *

(€1,

The Wolfe d u a l of

where

265

9.6 Extensions

L P 3 ( ~ )i s g i v e n by

max

(x,u,w)

f +

- r)

V E R E

+

Vx9(x

-

-

+ wT[li +

VXii(X

-

F) +

s . t . vxF

- X) +

VXF(X

-

uT[q

hTV

7

X

t-

x)

+ VET(€ - 3 1 VEFi(E

=

3 1

T w VXE = 0 , u 1. 0 ,

where t h e s u p e r b a r d e n o t e s e v a l u a t i o n a t x =

x

-

D L P (~€ 1

X,

E

= F , and

~ ( 7 ) .T h i s r e d u c e s t o max

(u,w)

L

+ PEL(€ - -a

s.t. V,f;

= 0 , u 2 0.

Thus, w e deduce from t h e d u a l i t y r e l a t i o n s h i p t h a t f*(E) where

z*=

T * ( E ) 1. E.*+ V C E * ( E

-

F),

L [ x ( F ), u ( E ) , w ( E ) , F ]

.

This corresponds p r e c i s e l y

t o t h e bound d e r i v e d j u s t p r e v i o u s l y , r e s u l t i n g simply from t h e c o n v e x i t y of f

*

(E).

Summarizing, f o r t h e g e n e r a l j o i n t l y convex program P 3 ( ~ a) l i n e a r lower bound on f t h e l i n e a r i z a t i o n of P j

*

(E)

(E)

with optimal m u l t i p l i e r s

i s p r o v i d e d by t h e d u a l v a l u e of a t some (F, F ), where 51 s o l v e s P 3 (5)

(u, w).

EXTENSIONS

9.6.

A p a r t from d u a l i t y r e s u l t s and t h e e x p l o i t a t i o n of conv e x i t y , w e n o t e t h a t o t h e r problem s t r u c t u r e s may a l l o w t h e c a l c u l a t i o n of bounds.

F o r example, suppose t h e problem i s

s . t . x e R ( E ) , where R i s a convex p o i n t k t o - s e t map ( i . e . , zl, z2 i n t h e graph G = { ( E , x ) I E E E ,

P ( E ) : minx f ( x ,

x

E

0 5

E)

R ( E ) ) of R i m p l i e s t h a t a z l c1

+

(1

-

a ) z Z e G for every

5 1) a s it i s i n t h e j o i n t l y convex program, b u t f ( x ,

h a s no p a r t i c u l a r c o n v e x i t y s t r u c t u r e .

E)

W e can s t i l l c a l c u l a t e

266


a p a r a m e t r i c u p p e r bound on f when

E~

and

*

+

[ ~ ( a ) f]o r € ( a ) =

(1

a r e i n a convex s u b s e t of t h e domain of f

E~

and 0 5 a 5 1, once X ( E ~ E) R ( E ~ )and

*

-

a)E1,

(E)

c R ( E ~ )a r e a v a i l -

'~E(E,)

a b l e , s i n c e t h e c o n v e x i t y o f t h e map R i m p l i e s , a s o b s e r v e d i n S e c t i o n 9.3,

t h a t z ( a ) = ~ T I ( E ~+) (1

f o r 0 5 a 5 1.

I t follows t h a t f

*

-

~ ) Z ( E E~ R) [ ~ ( a ) l

[ ~ ( a ) 5] f [ Z ( a ) , € ( a ) ] f o r

a l l 0 5 a 5 1. Another c l a s s o f problems, one n o t r e q u i r i n g c o n v e x i t y i n x, f o r which b o t h upper and lower bounds on f

*

(E

can be c a l -

c u l a t e d , a r e problems o f t h e form minx f ( x ,

E)

s.t. x

E

where R does n o t depend on

R

If f(x,

E.

t h e n i t i s w e l l known t h a t f

*

(E)

E)

i s concave i n

i s concave.

E,

This allows f o r

t h e c a l c u l a t i o n o f p i e c e w i s e - l i n e a r upper and lower g l o b a l bounds, u s i n g l i n e a r s u p p o r t s t o p r o v i d e upper bounds and l i n e a r i n t e r p o l a t i o n t o p r o v i d e a lower bound on f

*

(E),

analo-

gous t o t h e bounding p r o c e d u r e t h a t was d i s c u s s e d a t t h e b e g i n n i n g o f t h i s s e c t i o n f o r t h e convex o p t i m a l v a l u e function. The t e c h n i q u e s d e s c r i b e d h e r e p r o v i d e an approach f o r o b t a i n i n g computable o p t i m a l v a l u e p a r a m e t r i c bounds and an associated parametric f e a s i b l e vector f o r l a r g e c l a s s e s of p a r a m e t r i c n o n l i n e a r programs.

A b a s i s f o r o b t a i n i n g upper

and lower bounds on t h e o p t i m a l v a l u e i s t h e c o n v e x i t y o r conc a v i t y of t h e o p t i m a l v a l u e f u n c t i o n o f t h e problem o f i n t e r e s t , o r o f an a u x i l i a r y problem t h a t u n d e r e s t i m a t e s or o v e r e s t i m a t e s t h e o p t i m a l v a l u e o f t h e g i v e n problem.

The d e r i v a t i o n o f an

a s s o c i a t e d f e a s i b l e p a r a m e t r i c v e c t o r f u n c t i o n i s an immediate consequence o f t h e c o n v e x i t y o f t h e p o i n t t o s e t map t h a t def i n e s t h e f e a s i b l e region R ( E ) a s a function of the parameter€.

261

9.6 Extensions

The bounding approach h a s d i r e c t c o n n e c t i o n s w i t h d u a l i t y t h e o r y , a s n o t e d , l e a d i n g t o f u r t h e r r e s u l t s f o r lower bounds. Namely, t h e c o n v e x i t y of t h e g i v e n problem i n t h e v a r i a b l e s becomes e x p l o i t a b l e , g i v i n g a d d i t i o n a l and sometimes s h a r p e r bounds f o r problems i n v o l v i n g more g e n e r a l r i g h t - h a n d - s i d e perturbations.

The a p p l i c a t i o n of t h e s e t e c h n i q u e s t o non-

convex programs t o o b t a i n g l o b a l p a r a m e t r i c lower bounds v i a convex u n d e r e s t i m a t i n g problems i s immediate and l e a d s t o t h e i d e a o f e x p l o i t i n g nonconvex s o l u t i o n methodologies based on convex u n d e r e s t i m a t i n g problems, e . g . , for calculating global solutions. developed.

branch and bound methods

T h i s approach remains t o be

Other d i r e c t i o n s of r e s e a r c h t h a t s h o u l d be c a r e -

f u l l y e x p l o r e d are t h e i d e n t i f i c a t i o n of a d d i t i o n a l c l a s s e s of problems t h a t have convex o r concave o p t i m a l v a l u e f u n c t i o n s , t h e f u r t h e r e x p l o i t a t i o n o f problem s t r u c t u r e , such a s s e p a r a b i l i t y , t o c o n s t r u c t r e l a t e d c o m p u t a t i o n a l l y t r a c t a b l e programs w i t h convex o r concave o p t i m a l v a l u e f u n c t i o n s , t h e u s e o f o t h e r convex and nonconvex d u a l i t y r e s u l t s Ce.g.,

Rockafellar, 1970,

1971, 1974) and t h e development o f bounds on t h e p a r a m e t r i c s o l u t i o n v e c t o r and t h e a s s o c i a t e d o p t i m a l m u l t i p l i e r s .

Re-

s u l t s b a s e d on weaker c o n d i t i o n s i m p l y i n g any e x p l o i t a b l e p r o p e r t y o f t h e o p t i m a l v a l u e f u n c t i o n , e.g., pseudo-convexity,

quasi- o r

would b e e x t r e m e l y u s e f u l t o accommodate

l a r g e r classes o f problems b u t remain t o be developed. The r e a d e r i n t e r e s t e d i n i n t e r p r e t a t i o n s a s s o c i a t e d w i t h Fig. 9.2.1

i s a l s o r e f e r r e d t o Lasdon (1970) and Bazaraa and

S h e t t y (1979).

268


BOUNDS ON A SOLUTION P O I N T

9.7.

W e t u r n b r i e f l y t o t h e problem o f c o n s t r u c t i n g computable

bounds on a s o l u t i o n p o i n t min X f ( x ,

s.t. g ( x ,

E)

o f t h e g e n e r a l problem

X(E) E)

2 0, h(x,

Suppose P 3 ( ~ )i s j o i n t l y convex, x ( E ~ )s o l v e s P 3 ( ~ 2 ) .

R e c a l l i n g t h e n o t a t i o n and r e s u l t s o f

*

0 5 a 2 1, where € ( a ) =

-

P3(E)

s o l v e s P 3 ( ~ 1 ) , and

i t was concluded t h a t x ( a )

S e c t i o n 9.3,

(1

x ( E ~ )

= 0.

E)

+

aE2

(1

-

E

R [ E ( ~ ) f]o r any

*

a)E1 and x ( a ) =

+

a ) x ( ~ ~ )F .u r t h e r m o r e , w e found t h a t m a x [ ( l l ( a ) , ! L 2 ( a ) l 5 Thus, g i v e n any v a l u e o f a

f * [ ~ ( a ) 5] f [ x * ( a ) , € ( a ) ] .

E

*

[O, 11,

w e a r e a b l e t o c a l c u l a t e a f e a s i b l e p o i n t x ( a ) o f P 3 [ € ( a )1 and upper and lower bounds on t h e o p t i m a l v a l u e f m[ E ( a )I o f P 3 [ ~ ( a ) ] once , s o l u t i o n s o f P 3 [ € ( 0 ) ] and P 3 [ c ( l ) l a r e a v a i l .We a d d r e s s t h e problem o f e s t i m a t i n g t h e d i s t a n c e o f

able.

x* ( a ) t o F* ( a ) , where F* ( a ) is a s o l u t i o n o f P 3 [ E ( a )1 f o r some i n ( 0 , 1).

fixed value of a =

The f o l l o w i n g a p p r o a c h h a s been s u g g e s t e d by F i a c c o and Kyparisis (Fiacco, 1981).

A s s u m e t h e problem f u n c t i o n s a r e

t w i c e continuously d i f f e r e n t i a b l e .

Let

z

*

=

-* -

x ( a ) and

Then t h e c o n v e x i t y o f f i m p l i e s t h a t f o r any v e c t o r z

E

= E(Z). En,

the

first-order inequality

AT holds.

F f(z*

+ z, F)

*

* , -E )

f(y

2 Vxf(y*, F ) z

A second-order i n e q u a l i t y a l s o holds.

conditions hold a t z

Karush-Kuhn-Tucker and w

-

, Taylor's

L* F L ( Z * = f

+

2 ,

u

+

with multipliers u

*

* , w * , -E ) T 2

*

Assuming t h e

series y i e l d s

(z*, 8 ) + ( 1 / 2 ) 2 VxL(rI, u

where q = z

*

(9.7.1)

Bz f o r some B

E

*

* ,w ,az, *

[0, 1 1 .

If z

+ z

is a feasible

269

9.7 Bounds on a Solution Point

p o i n t of P 3 ( F ) , t h e n f ( z

*

+

z,

E)

2. L

*

a n d h e n c e w e may con-

clude t h a t

AT

E f(z*

-

+ z, E )

f ( z * , 5)

*

T 2

L ( 1 / 2 ) 2 VxL(rl, u , w

llZll

W e are s e e k i n g a bound on

-

x*(E)

that i f z = Z

observed, and (9.7.1)

z*,

* , -€12. IIx* ( E l

t h e n z*

and (9.7.2)

(9.7.2)

+Z

- z*(a)11.

Note

= x * ( E ) E R ( F ) , as

apply.

F u r t h e r , w e can

c a l c u l a t e a n o v e r e s t i m a t i o n of t h e l e f t - h a n d s i d e o f t h e s e i n e q u a l i t i e s f r o m o u r p r e v i o u s o p t i m a l v a l u e bounds c a l c u l a tions, obtaining i n particular t h a t B(Z) E f [ x * ( a ) ,

-

€1

-

max[tl(iji), f i 2 ( i j i ) 1

1. f [ X * ( E : ) , 51 f(z

*

+ z,

5)

-

-

f(z

W e now h a v e from (9.7.11,

Vxf(z*,

*

, 5). (9.7.21,

a n d (9.7.3)

that

5 B(a),

(9.7.4)

* * -T 2 ( 1 / 2 ) 2 V x L ( q , u , w , F)? 5 B ( a ) .

(9.7.5)

I t r e m a i n s t o u n d e r e s t i m a t e t h e l e f t - h a n d sides of

(9.7.4)

and (9.7.5) p1

(9.7.3)

f [ F * ( E i ) , FI

i n terms o f 11211.

A n a t u r a l approach is t o seek

> 0 and p2 > 0 such t h a t

Vxf(z

* , E)E

1.

lJ1ll~ll

(9.7.6)

and (9.7.7)

270


Having c a l c u l a t e d p1 a n d p z r w e c o u l d t h e n c o n c l u d e from (9.7.4)

(9.7.7)

a n d ( 9 . 7 . 6 ) t h a t ))1)1 5 B(Z)/ul 2 t h a t 115711 5 B ( E ) / v 2 .

However, n o t i n g t h a t ( 9 . 7 . 6 )

i s n o t r e p r e s e n t e d i n (9.7.7)

v2

0 so

gram V'L

combining ( 9 . 7 . 1 )

= 0 and

(9.7.7)

and (9.7.2)

and

d o e s n o t a c c o u n t f o r second-

o r d e r e f f e c t s and t h a t t h e f i r s t - o r d e r (9.7.6)

and from ( 9 . 7 . 5 )

information yielding (e.g.,

for a linear pro

y i e l d s no bound)

,

suggests

t o y i e l d a bound i n c o r p o r a t i n g

b o t h f i r s t - a n d s e c o n d - o r d e r i n f o r m a t i o n a t t h e o u t s e t and a t h e r e f o r e h o p e f u l l y more c o m p r e h e n s i v e a n d s h a r p e r bound. n o t e f i r s t t h a t simply adding (9.7.1)

and ( 9 . 7 . 2 )

t o a bound o f t h e t y p e g i v e n i n ( 9 . 7 . 6 )

We

does n o t l e a d

o r (9.7.71,

because

of t h e l a c k o f homogeneity w i t h r e s p e c t t o z o f t h e t e r m s involving

However, a s i m p l e m o d i f i c a t i o n d o e s l e a d t o t h i s

2.

t y p e of bound.

N o t e t h a t t h e right-hand

+

n o n n e g a t i v e i f z*

z

E

RE),

t h e r e f o r e s q u a r e b o t h s i d e s of (9.7.2),

a

+

since z

*

(9.7.1)

s i d e of

is

(9.7.1)

s o l v e s P ~ ( E ) . W e may and a d d t h e r e s u l t t o

w e i g h t e d r e s p e c t i v e l y by a 1. 0 and B 2 0 , where

= 1, f o r a d d i t i o n a l f l e x i b i l i t y , o b t a i n i n g f o r a l l

s u c h t h a t z*

+

z

E

z

R(E),

As b e f o r e , t h i s e x p r e s s i o n a n d ( 9 . 7 . 3 )

imply t h a t

-

I f w e c a n f i n d any v ( a , 8 ) > 0 s u c h t h a t

(9.7.8)

271

9.7 Bounds on a Solution Point

t h e n we can conclude t h a t (9.7.9) The p a r a m e t e r s a a n d B h a v e b e e n i n t r o d u c e d f o r a d d i t i o n a l f l e x i b i l i t y and g e n e r a l i t y , i n c o r p o r a t i n g t h e bounds b a s e d on (9.7.6)

and ( 9 . 7 . 7 )

a s s p e c i a l cases o f ( 9 . 7 . 9 )

and a l l o w i n g

t h e p o s s i b i l i t y o f i m p r o v i n g on t h e f o r m e r bounds.

Ideally,

w e would l i k e t o f i x a a n d B s u c h t h a t t h e l a r g e s t v a l u e o f

B ) w i l l r e s u l t , a n d t h e t h e o r e t i c a l i m p l i c a t i o n s of t h i s

p(ct,

o p t i m a l s e l e c t i o n might be i n t e r e s t i n g t o e x p l o r e .

I n prac-

t i c e , a p o s s i b l e s t r a t e g y m i g h t be f i r s t t o check t h e bounds o b t a i n e d f r o m ( a , B ) = (1, O), and B = 0 t h e n p = p l ,

( 0 , 1). Obviously, i f a = 1

and i f a = 0 and B = 1, t h e n p = u2.

Suppose w e a l l o w a t m o s t o n e a t t e m p t t o improve t h e bound.

If

e i t h e r p l o r p 2 i s 0 , b u t n o t b o t h , w e c o u l d a c c e p t t h e non= p2 = 0 , w e

z e r o v a l u e f o r p t o c a l c u l a t e t h e bound.

I f pl

might t r y "unbiased" w e i g h t s , a = 8 = 1 / 2 .

Finally, i f

u1 > 0

a n d u2 > 0 , w e m i g h t s e l e c t a a n d B p r o p o r t i o n a t e t o p1 a n d p z , r e s p e c t i v e l y , e.g., p2/(pl

+

p2).

w e may t a k e a = p l / ( p l

+

p2)

and B =

The v a r i o u s p o s s i b i l i t i e s a n d t h e i r i m p l i c a t i o n s

w i l l be e x p l o r e d . Having s e l e c t e d v a l u e s f o r a a n d B, w e a r e l e d t o t h e s t u d y o f a p r o b l e m o f t h e form

o v e r a n a p p r o p r i a t e set c o n t a i n i n g ( z

*

* * , u , w , z).

The cal-

c u l a t i o n of p l e a d s t o an a n a l y s i s of v a r i o u s n o n l i n e a r

212


s.t.

-

iil

2

Or

i

V h.(x, E)z =

O r

j = 1,

Vxgi(x,

E)Z

X I

1lZ1l2

E

gi(xr F) = . . . I

01,

p,

= 1,

w i t h p o s s i b l y some a d d i t i o n a l r e s t r i c t i o n s .

As n o t e d , t h e

development o f bounds r e s u l t s u s i n g t h i s a p p r o a c h i s c u r r e n t l y being i n v e s t i g a t e d . Two i n t r i g u i n g f a c t s a s s o c i a t e d w i t h t h e P ( a , 8 ) bound are worth mentioning.

One i s a c o n n e c t i o n between t h e p o s i t i v i t y

o f t h e o b j e c t i v e f u n c t i o n o f P ( a , B ) a t ?I f o r

c1

> 0,

8 > 0,

w i t h ( a n e q u i v a l e n t form o f ) t h e s e c o n d - o r d e r s u f f i c i e n t con-

X b e a s t r i c t l o c a l minimum o f P 3 ( T ) (see comment

ditions that

f o l l o w i n g Lemma 3 . 2 . 1 ) .

The s e c o n d i s t h e s i m i l a r i t y of P(c1,

6 ) w i t h a q u a d r a t i c a u x i l i a r y p r o b l e m o f t h e form minz ~

~

f F (E )+x~( 1 ~ / 2 ) z~T v ; ~ ( x o , u o r wor

V h.(xo, E ) z = 0, X

I

j = 1,

E)Z

..., p

t h a t arises i n c o n n e c t i o n w i t h t h e r e c u r s i v e programming methods o f Wilson ( 1 9 6 3 ) , L e v i t i n a n d Polyak ( 1 9 6 6 1 , Robinson ( 1 9 7 2 ) , and o t h e r contemporary v e r s i o n s , f o r s o l v i n g P 3 ( T ) w i t h

( x o r u o r wo) g i v e n a n d k = 1, 2 ,

... .

These i n t e r e s t i n g c l o s e

correspondences w i t h important r e s u l t s m e r i t f u r t h e r i n v e s t i g a t i o n b u t w i l l n o t be pursued here.

273

9 . 8 Further Extensions and Applications

9.8.

FURTHER EXTENSIONS

AND A P P L I C A T I O N S

If

t h e g i v e n problem P 3 ( € ) d o e s n o t have a convex o p t i m a l

f u n c t i o n , t h e n w e c a n s t i l l g e n e r a t e upper o r lower bounds on f

*

i f w e have a t o u r d i s p o s a l a j o i n t l y convex u n d e r e s t i m a t i n g

o r o v e r e s t i m a t i n g problem of P 3 ( € ) , i . e . ,

a problem whose

o p t i m a l v a l u e must u n d e r e s t i m a t e o r o v e r e s t i m a t e f parameter values of i n t e r e s t .

*

for the

For s t a n d a r d ( n o n p a r a m e t r i c ) NLP

programs t h e r e e x i s t s a v a r i e t y o f t e c h n i q u e s f o r g e n e r a t i n g convex u n d e r e s t i m a t i n g problems i f t h e f u n c t i o n s o f t h e g i v e n nonconvex problem a r e s e p a r a b l e ( F a l k and Soland, 1 9 6 9 ) o r f a c t o r a b l e (McCormick, 1 9 7 6 ) , t h e l a t t e r c l a s s i n c l u d i n q a l l t h e f u n c t i o n s w e have e n c o u n t e r e d i n p r a c t i c e . F o r p a r a m e t r i c programs i t i s s i m p l y a m a t t e r o f a p p l y i n g t h e s e t e c h n i q u e s jointly o v e r b o t h t h e d e c i s i o n v a r i a b l e x and t h e parameter

E.

For a l l s u c h p r o c e d u r e s , t h e c a l c u l a t i o n o f

bounds o f t h e form 111 o r !La f o r t h e j o i n t l y convex undere s t i m a t i n g problem w i l l p r o v i d e p i e c e w i s e - l i n e a r lower bounds on t h e o p t i m a l v a l u e o f t h e g i v e n nonconvex problem.

This

t e c h n i q u e seems e s p e c i a l l y e x p l o i t a b l e by branch and bound methods t h a t a r e based on s o l v i n g a sequence o f convex undere s t i m a t i n g problems. The methodology f o r g e n e r a t i n g convex o v e r e s t i m a t i n g problems i s less d e v e l o p e d , though some r e s u l t s e x i s t .

In-

t e r i o r p e n a l t y ( b a r r i e r ) f u n c t i o n methods ( F i a c c o and McCormick, 1968) a r e based on d e v i s i n g an u n c o n s t r a i n e d o v e r e s t i m a t i n g problem t h a t i s convex i f t h e problem i s convex and l o c a l l y convex o t h e r w i s e .

Meyer (1970) developed a l g o r i t h m s f o r re-

v e r s e convex c o n s t r a i n t s b a s e d on c o n s t r u c t i n g a sequence o f

214


l i n e a r o v e r e s t i m a t i n g problems.

A v r i e l , Dembo, and Passy

(1975) have d e v e l o p e d an a l g o r i t h m f o r s o l v i n g a g e n e r a l i z e d g e o m e t r i c programming problem, u t i l i z i n g a sequence o f posynomial ( h e n c e , e q u i v a l e n t t o convex) o v e r e s t i m a t i n g problems i n c o n j u n c t i o n w i t h an " i n t e r i o r " c u t t i n g p l a n e method.

Ghaemi

(1980) h a s d e r i v e d f o r m u l a s f o r c a l c u l a t i n g a convex o v e r e s t i m a t i n g problem of a f a c t o r a b l e program.

I f a j o i n t l y con-

vex o v e r e s t i m a t i n g problem c a n be c o n s t r u c t e d , t h e n t h e c a l c u l a t i o n of a bound of t h e form i 3 f o r t h i s problem w i l l p r o v i d e a l i n e a r p a r a m e t r i c upper bound on t h e o p t i m a l v a l u e o f t h e o r i g i n a l problem o v e r t h e r a n g e of p a r a m e t e r v a l u e s c o n s i d e r e d . Analogous bounds can o b v i o u s l y be c o n s t r u c t e d from programs whose o p t i m a l v a l u e i s concave, and some p r e l i m i n a r y r e s u l t s a r e a l s o r e p o r t e d by Ghaemi ( 1 9 8 0 ) . The p r o c e d u r e f o r c a l c u l a t i n g p i e c e w i s e - l i n e a r upper and lower bounds f o r t h e convex o p t i m a l v a l u e f u n c t i o n was f i r s t implemented by F i a c c o and Ghaemi and r e p o r t e d by Ghaemi (1980) i n h i s d o c t o r a l d i s s e r t a t i o n .

Ghaemi added t h e bounds

calculation c a p a b i l i t y t o the penalty-function s e n s i t i v i t y a n a l y s i s computer program SENSUMT and used t h i s program s u c c e s s f u l l y t o o b t a i n bounds f o r s e v e r a l s m a l l convex and nonconvex problems.

Ghaemi a p p l i e d t h e program t o c a l c u l a t e

bounds on t h e o p t i m a l v a l u e o f a convex e q u i v a l e n t of a g e o m e t r i c programming model of a stream-water p o l l u t i o n abatement system r e l a t i v e t o t h e most i m p o r t a n t p a r a m e t e r , a s d e t e r m i n e d by t h e Fiacco-Ghaemi s e n s i t i v i t y s t u d y (1979b) (see S e c t i o n 8.5 o f t h i s b o o k ) .

F i a c c o and Ghaemi (1980) a l s o p r e -

p a r e d a d e t a i l e d u s e r s ' manual f o r SENSUMT t h a t i n c l u d e s t h e

275

9.8 Further Extensions and Applications

o p t i m a l v a l u e bounds c a l c u l a t i o n a s a u s e r o p t i o n , a s w e l l a s a comprehensive s e n s i t i v i t y a n a l y s i s c a p a b i l i t y . The i n d i c a t e d w a t e r p o l l u t i o n bounds c a l c u l a t i o n i n v o l v e s t h e p e r t u r b a t i o n of a s i n g l e r h s parameter, t h e allowable oxygen d e f i c i t l e v e l i n t h e f i n a l r e a c h o f t h e s t r e a m , t h a t proved unequivocably t o b e t h e s i n g l e most i n f l u e n t i a l param-

e t e r i n t h e p r i o r s e n s i t i v i t y s t u d y by F i a c c o and Ghaemi (197933).

S u b s e q u e n t l y , F i a c c o and K y p a r i s i s (1981a) u t i l i z e d

SENSUMT t o c a l c u l a t e bounds on t h e o p t i m a l v a l u e c o s t f u n c t i o n

when t h e 30 most i n f l u e n t i a l p a r a m e t e r s w e r e p e r t u r b e d simultaneously.

I n t h i s a p p l i c a t i o n , t h e optimal value function

( t h e a n n u a l w a s t e t r e a t m e n t c o s t ) i s n o t convex i n a f u l l neighborhood o f t h e b a s e v a l u e o f t h e p a r a m e t e r v e c t o r . e v e r , it was p o s s i b l e t o show t h a t t h e r e s t r i c t i o n o f f

*

How(E)

to

t h e s u b s e t of parameters involved i n t h e d e s i r e d p e r t u r b a t i o n

i s convex.

I n a c o n t i n u a t i o n o f t h i s s t u d y , F i a c c o and

K y p a r i s i s (1981b) implemented f o r t h e f i r s t t i m e t h e r e f i n e ment p r o c e d u r e d e s c r i b e d i n S e c t i o n 9.3,

o b t a i n i n g much s h a r p e r

o p t i m a l v a l u e bounds and a p i e c e w i s e - l i n e a r solution estimate, a s w e l l .

Also,

f e a s i b l e parametric

s o l u t i o n v e c t o r bounds were

c a l c u l a t e d , a g a i n an i n i t i a l i m p l e m e n t a t i o n , u s i n g some of t h e r e s u l t s o f S e c t i o n 9.7. I n a n o t h e r s t u d y , which i n v o l v e d t h e convex e q u i v a l e n t of a g e o m e t r i c programming model o f a power system e n e r g y model d e v e l o p e d by Ecker and Wiebking (1976) t o f i n d t h e t u r b i n e e x h a u s t a n n u l u s and c o n d e n s e r s y s t e m d e s i g n t h a t minimizes t o t a l a n n u a l f i x e d p l u s o p e r a t i n g c o s t , F i a c c o and Ghaemi (1981b) u s e d SENSUMT t o o b t a i n p a r a m e t e r d e r i v a t i v e s and bounds on t h e o p t i m a l v a l u e f u n c t i o n f o r a v a r i e t y o f s i n g l e

216


o b j e c t i v e f u n c t i o n and c o n s t r a i n t p a r a m e t e r changes.

A

novelty of t h i s a n a l y s i s i s t h e use of t h e concavity r e s u l t g i v e n i n t h e second p a r a g r a p h o f S e c t i o n 9 . 6 and an e x p l o i t a t i o n o f problem s t r u c t u r e t o c a l c u l a t e a nonlinear lower bound on t h e o p t i m a l v a l u e f u n c t i o n . The s e r i o u s i m p l e m e n t a t i o n o f t h e bounds p r o c e d u r e f o r s e p a r a b l e and f a c t o r a b l e nonconvex programs, implemented f o r some examples by Ghaemi ( 1 9 8 0 ) , remains t o be developed.

Chapter 10

Future Research and Applications

10.1.

RECAPITULATION AND OTHER

RESEARCH DIRECTIONS

C h a p t e r 2 p r e s e n t s a number o f b a s i c c o n t r i b u t i o n s t o t h e t h e o r y o f s e n s i t i v i t y and s t a b i l i t y a n a l y s i s f o r g e n e r a l c l a s s e s o f n o n l i n e a r programming problems.

Hopefully, i t

c a p t u r e s a f l a v o r o f t h e main t h r u s t and r a n g e of developments f o r t h e s t a t i c nonconvex d e t e r m i n i s t i c problem t h a t r e l a t e s t h e behavior of t h e optimal value o r s o l u t i o n set t o perturbat i o n s o f problem p a r a m e t e r s .

That b r i e f a c c o u n t n e c e s s a r i l y

o m i t s many i m p o r t a n t r e s u l t s t h a t weave t h e f a b r i c more t i g h t l y and t h a t e x t e n d t h e framework of t h e t h e o r y t o more general spaces.

Also, r e s u l t s t h a t s i g n i f i c a n t l y e x p l o i t

s p e c i a l problem s t r u c t u r e have been d e t a i l e d o n l y f o r param-

e t e r d e r i v a t i v e s under r a t h e r i d e a l second-order c o n d i t i o n s (Chapters 3

-

8 ) and f o r p a r a m e t r i c bounds f o r j o i n t l y convex

programs ( C h a p t e r 9 ) .

I n t h i s s e c t i o n w e i n d i c a t e important

r e s e a r c h d i r e c t i o n s t h a t have been touched upon v e r y l i g h t l y o r not a t a l l . Numerous i m p o r t a n t r e s u l t s r e l e v a n t t o c o n v e x i t y and s t a b i l i t y a r e known f o r convex programs and t h e i r d u a l s , o t h e r t h a n t h o s e n o t e d i n C h a p t e r 2 , under v a r i o u s smoothness and 211

278

10 Future Research and Applications

nonsmoothness a s s u m p t i o n s .

S e e , f o r example, R o c k a f e l l a r

(1970, 1 9 8 2 a , b ) G o l ' s t e i n

( 1 9 7 2 ) , o r Hogan ( 1 9 7 3 ~ ) . L i n e a r

and q u a d r a t i c p a r a m e t r i c programming c h a r a c t e r i z a t i o n s h a v e been r a t h e r thoroughly developed:

see, e . g . ,

the references

c i t e d a t t h e b e g i n n i n g o f C h a p t e r 2 , a s w e l l a s Bereanu ( 1 9 7 6 ) , M a r t i n (19751, and Robinson ( 1 9 7 3 a , 1975, 1 9 7 7 ) f o r c o n t i n u i t y and s t a b i l i t y c h a r a c t e r i z a t i o n s .

P a r a m e t r i c r a n g e and p o s t -

o p t i m a l i t y s e n s i t i v i t y a n a l y s i s t e c h n i q u e s f o r l i n e a r programming h a v e b e e n known a n d r o u t i n e l y implemented f o r s o m e t i m e ( D a n t z i g , 1963; B r a d l e y , Hax, and M a g n a n t i , 1 9 7 7 ) .

A

recent

novel development a p p e a r s t o be a " s e n s i t i v i t y t o l e r a n c e " a p p r o a c h by Wendell ( 1 9 8 1 a , b ) t h a t a l l o w s f o r t h e s i m u l t a n e o u s and i n d p e n d e n t p e r t u r b a t i o n o f s e l e c t e d p a r a m e t e r s a n d t h e comp u t a t i o n o f a maximum t o l e r a n c e r a n g e w i t h i n which p a r a m e t r i c v a r i a t i o n does n o t a l t e r a given o p t i m al b a s i s .

Parametric

g e o m e t r i c programs h a v e a s t r o n g s t r u c t u r e t h a t c a n r e a d i l y b e e x p l o i t e d f o r s e n s i t i v i t y a n d g l o b a l bound i n f o r m a t i o n , a s n o t e d by D u f f i n e t a l .

(19671, B e i g h t l e r a n d P h i l l i p s (19761,

Dembo ( 1 9 8 2 ) , and i n some o f o u r r e c e n t work ( e . g . , applications discussed i n Section 9.8).

see t h e

The numerous computa-

t i o n a l i m p l e m e n t a t i o n s by D i n k e l a n d Kochenberger (1977, 1 9 7 8 ) a n d D i n k e l , K o c h e n b e r g e r , and Wong (1978, 1 9 8 2 ) m e n t i o n e d i n S e c t i o n 2.5,

using an incremental predictor-corrector

approach

t o e x t r a p o l a t e s e n s i t i v i t y i n f o r m a t i o n , are r e l e v a n t i n t h i s context. I n s e e k i n g t o e x p l o i t s t r u c t u r e i n d e v i s i n g computable methods f o r c a l c u l a t i n g m e a s u r e s o f s t a b i l i t y , it i s clear t h a t q u a n t i t i e s s u c h as bounds t h a t c a n be d e r i v e d i n t e r m s o f f u n c t i o n s o f a s i n g l e v a r i a b l e are o f s p e c i a l i n t e r e s t .

This

i m m e d i a t e l y b r i n g s t o mind s e p a r a b l e p r o g r a m s , e x t e n s i v e l y

219

10. I Recapitulation

e x p l o i t e d by F a l k a n d S o l a n d ( 1 9 6 9 ) a n d s u b s e q u e n t l y by many o t h e r s t o d e v i s e g l o b a l s o l u t i o n s t r a t e g i e s b a s e d o n branchand-bound and convex u n d e r e s t i m a t i o n .

There h a s been sur-

p r i s i n g l y l i t t l e e x p l o i t a t i o n of t h i s class o f problems i n calculating s e n s i t i v i t y o r s t a b i l i t y information, except f o r t h e r e l a t e d p r e l i m i n a r y s t u d y of G e o f f r i o n ( 1 9 7 7 ) , t h e r e c e n t u s e o f error bound i n f o r m a t i o n by Meyer ( 1 9 7 9 , 1 9 8 0 ) a n d Thakur (1978, 1 9 8 0 , 1 9 8 1 ) f o r convex p r o g r a m s , as b r i e f l y n o t e d i n S e c t i o n 2.5.

The e x t e n s i o n by Benson (1980) o f a

l a r g e c l a s s o f b r a n c h a n d bound methods t o p a r a m e t r i c nonconvex s e p a r a b l e programming w i t h r h s p e r t u r b a t i o n s i s a n i n t e r e s t i n g development. I t s h o u l d be e m p h a s i z e d t h a t i m p o r t a n t g e n e r a l c o n t e m p o r a r y

treatments of parametric s t a b i l i t y r e s u l t s obtained recently, e.g.,

t h e many i n c i s i v e r e s u l t s o b t a i n e d i n t h e s t u d y of gen-

e r a l i z e d e q u a t i o n s by Robinson ( 1 9 7 3

-

1 9 8 0 ) and o t h e r s , and

deep d i f f e r e n t i a l s t a b i l i t y r e s u l t s of R o c k a f e l l a r (1982a,b) have been mentioned o n l y i n p a s s i n g i n Chapter 2.

These re-

s u l t s are f a r - r e a c h i n g a n d w i l l u n d o u b t e d l y i n s p i r e and i n f l u e n c e many f u t u r e d e v e l o p m e n t s . W e h a v e n o t i n c l u d e d any r e s u l t s t a i l o r e d e s p e c i a l l y t o

o p t i m a l c o n t r o l t h e o r y , e x c e p t t o m e n t i o n t h e p a p e r s o f Maurer ( 1 9 7 7 a , b ) i n S e c t i o n 2.3. infinite-dimensional

The r e s u l t s g i v e n i n C h a p t e r 2 f o r

s p a c e s a r e g e n e r a l l y a p p l i c a b l e , of

course.

Some r e c e n t r e s u l t s h a v e b e e n r e p o r t e d by G o l l a n

(1979).

W e h a v e a l s o n o t a d d r e s s e d zero-one o r i n t e g e r pro-

gramming s e n s i t i v i t y a n a l y s i s , t h o u g h many r e s u l t s have been developed, e . g. ,

by Roodman (1972) , S h a p i r o ( 1 9 7 7 ) , M a r s t e n

and Morin ( 1 9 7 7 ) , H o l m a n d K l e i n (1978, 19821, Wolsey (19811, a n d Graves a n d S h a p i r o ( 1 9 8 2 ) .

280


I t i s r e l e v a n t t o mention t h e " d e c i s i o n v a r i a b l e s e n s i t i v -

i t y a n a l y s i s " t e c h n i q u e s d e v e l o p e d by McKeown ( 1 9 8 0 ) .

The

motivation i s t o determine t h e s e n s i t i v i t y o f t h e optimal v a l u e t o changes i n t h e components o f a s o l u t i o n v e c t o r . leads t o a "curvature analysis,

'I

i.e.,

This

an a n a l y s i s o f t h e

e i g e n v a l u e s and e i g e n v e c t o r s o f t h e Hessian o f t h e o b j e c t i v e f u n c t i o n , i n t h e u n c o n s t r a i n e d problem.

Up t o s e c o n d - o r d e r

i n f o r m a t i o n , t h e o p t i m a l v a l u e f u n c t i o n i s l o c a l l y most s e n s i t i v e t o a change i n t h e d i r e c t i o n o f t h e e i g e n v e c t o r a s s o c i a t e d w i t h an e i g e n v a l u e whose a b s o l u t e v a l u e i s l a r g e r t h a n t h a t o f any o t h e r e i g e n v a l u e , and l e a s t s e n s i t i v e t o a change i n t h e d i r e c t i o n o f an e i g e n v e c t o r a s s o c i a t e d w i t h an e i g e n value with l e a s t absolute value.

For t h e c o n s t r a i n e d problem,

t h e r e l e v a n t Hessian i s t h e r e s t r i c t i o n of t h e H e s s i a n o f t h e Lagrangian t o t h e c o n s t r a i n t t a n g e n t s u b s p a c e a t a s o l u t i o n . McKeown h a s d e v e l o p e d d e t a i l e d n u m e r i c a l p r o c e d u r e s t h a t i m plement t h i s a n a l y s i s , i n a computer program whose s u c c e s s f u l use h a s been r e p o r t e d i n p r a c t i c a l a p p l i c a t i o n s ( e . g . ,

Archetti,

B a l l a b i o , and V e r c e l l i s , 1 9 8 0 ) . W e s h o u l d a l s o mention t h a t an e n t i r e body o f t h e o r y ,

somewhat more g e n e r a l b u t t o a g r e a t e x t e n t a n a l o g o u s t o t h e t h e o r y i n v o l v i n g p a r a m e t r i c v a r i a t i o n , h a s d e v e l o p e d more o r

less i n p a r a l l e l w i t h t h e p a r a m e t r i c t h e o r y .

This involves

t h e s t u d y of t h e e f f e c t s o f g e n e r a l p e r t u r b a t i o n s o f t h e prob-

l e m f u n c t i o n s on s o l u t i o n b e h a v i o r . For example, c o n s i d e r Pk: k k k k minx f ( x ) s . t . g ( x ) 2 0 , h ( x ) = 0 , where t h e f u n c t i o n s f , g k l and hk converge i n some s p e c i f i e d s e n s e t o f , g , h , respectively, as k

+

=.

Questions concerning t h e r e l a t i o n s h i p

281

10.2 Future Research Directions and Applications

o f s o l u t i o n s o f Pk t o s o l u t i o n s o f P:

minx f ( x ) s.t. g ( x ) 2

0 , h ( x ) = 0 , are of i n t e r e s t and o b v i o u s l y r e l a t e t o s e n s i t i v -

i t y and s t a b i l i t y q u e s t i o n s .

Noteworthy background r e a d i n g s

i n t h i s a r e a are t h e p a p e r s by Wijsman (1966) a n d Mosco ( 1 9 6 9 ) . The p a p e r s by D a n t z i g e t aZ. quite relevant.

(1967) a n d F i a c c o (1974) are a l s o

Numerous r e f e r e n c e s c o u l d be g i v e n h e r e , i n

a d d i t i o n t o those already provided f o r t h e parametric perturb a t i o n s , which a r e c e r t a i n l y r e l e v a n t ( n o t i n g t h a t t h e problem P ( E ) may be a n a l y s e d a t E = Z by c o n s i d e r i n g p r o b l e m s o f t h e k f o r m Pk, where f ( x ) = f ( x , E ~ ) ,e t c . , w h e r e E~ * E a s k * -1. The i n t e r e s t e d r e a d e r i s a l s o r e f e r r e d t o t h e r e c e n t work o f

Wets ( 1 9 7 7 ) a n d S a l i n e t t i a n d W e t s (1979) t h a t g i v e s a number of i n t e r e s t i n g r e s u l t s i n v o l v i n g s e q u e n c e s of convex sets and t h e i r a p p l i c a t i o n t o convex s t o c h a s t i c p r o g r a m i n g a s w e l l as many r e f e r e n c e s t o o t h e r work i n t h i s area.

This application

r e m i n d s u s t h a t t h e g e n e r a l area of s t o c h a s t i c programming h a s n o t b e e n a d d r e s s e d i n t h i s book e i t h e r , a l t h o u g h t h e i n e v i t a b l e presence o f u n c e r t a i n t y , e.g.,

p a r a m e t e r s t h a t are random

v a r i a b l e s , would o b v i o u s l y s u g g e s t t h a t p e r t u r b a t i o n a n a l y s i s r e s u l t s c h a r a c t e r i z i n g s o l u t i o n s e n s i t i v i t y o r s t a b i l i t y would c l e a r l y be a p p l i c a b l e .

E x p l i c i t c o n n e c t i o n s h a v e a l r e a d y been

made, as s u g g e s t e d by S a l i n e t t i a n d W e t s (19791, Bereanu (1976) and o t h e r s . 10.2.

FUTURE RESEARCH DIRECTIONS

AND APPLICATIONS

W e e x p e c t t o see a n e v e r - i n c r e a s i n g e m p h a s i s on computa-

b i l i t y a n d p r a c t i c a l a p p l i c a b i l i t y i n t h e f u t u r e development of s e n s i t i v i t y and s t a b i l i t y a n a l y s i s methodology i n mathe-

matical p r o g r a m i n g .

I t i s i n t h e s p i r i t of o u r t i m e s a n d i s

282


a l s o p a r t o f a n a t u r a l m a t u r a t i o n p r o c e s s o f an a p p l i e d science.

As p r a c t i t i o n e r s become more aware o f what a d d i t i o n a l

v a l u a b l e s o l u t i o n i n f o r m a t i o n can r e a d i l y be g e n e r a t e d by a g i v e n s o l u t i o n a l g o r i t h m t o r e l a t e s o l u t i o n changes t o d a t a changes, t h e y w i l l want t h i s i n f o r m a t i o n .

W e i n d i c a t e some

r e s e a r c h d i r e c t i o n s and a r e a s o f a p p l i c a t i o n s t h a t have cont r i b u t e d o r t h a t w e e x p e c t w i l l c o n t r i b u t e t o t h i s development. The i n d i c a t e d r e s u l t s i n s e p a r a b l e programming mentioned i n t h e l a s t s e c t i o n a l s o s u g g e s t an approach, b r i e f l y ment i o n e d i n S e c t i o n s 2 . 5 and 9 . 8 ,

f o r generating s t a b i l i t y

i n f o r m a t i o n f o r nonconvex s e p a r a b l e o r f a c t o r a b l e programs. E r r o r bounds a r e f r e q u e n t l y o b t a i n e d by g e n e r a t i n g s i m p l e r f u n c t i o n s t h a t bound t h e g i v e n problem f u n c t i o n s , e . g . ,

convex

e n v e l o p e s of t h e c o n s t r a i n t f u n c t i o n s , a s i n t h e s e p a r a b l e nonconvex programming branch-and-bound

approach o f F a l k and

Soland ( 1 9 6 9 ) o r convex u n d e r e s t i m a t i n g o r concave o v e r e s t i mating f u n c t i o n s , a s i n t h e nonconvex f a c t o r a b l e programming approach o f McCormick ( 1 9 7 6 ) .

But once a p r o c e d u r e f o r gener-

a t i n g s i m p l e r "bounding problems" i s a t hand, i t can g e n e r a l l y be a p p l i e d t o a p e r t u r b a t i o n o f t h e o r i g i n a l problem t o o b t a i n "simple" (e.g.,

convex) bounding problems.

I f t h e perturba-

t i o n a n a l y s i s o f t h e s i m p l e r a s s o c i a t e d problems i s t r a c t a b l e , t h e n once t h e r e l a t i o n s h i p between t h e o p t i m a l s o l u t i o n o f t h e p e r t u r b e d bounding problems and t h e o r i g i n a l p e r t u r b e d problem

i s u n d e r s t o o d , w e have a p o s s i b l e p r o c e d u r e f o r g e n e r a t i n g bounds on t h e p e r t u r b e d s o l u t i o n o f t h e o r i g i n a l problem.

As n o t e d i n C h a p t e r 9 , t h e bounds approach e x p l o i t i n g t h e p r o p e r t i e s o f t h e j o i n t l y convex program i s p a r t i c u l a r l y w e l l s u i t e d t o t h e g e n e r a t i o n o f p a r a m e t r i c bounds f o r p a r a m e t r i c nonconvex s e p a r a b l e programs.

Once a j o i n t l y convex s e p a r a b l e

283

10.2 Future Research Directions and Applications

u n d e r e s t i m a t i n g p r o b l e m i s a t hand, t h e n w e c a n c a l c u l a t e p a r a m e t r i c o p t i m a l v a l u e l o w e r bounds f o r t h i s problem.

Ob-

v i o u s l y , t h e s e w i l l c o n s t i t u t e lower bounds f o r t h e o p t i m a l v a l u e f u n c t i o n o f t h e g i v e n nonconvex program. d e v e l o p e d branch-and-bound

The w e l l

methodology f o r c a l c u l a t i n g t h e

u n d e r e s t i m a t i n g p r o b l e m c a n o b v i o u s l y b e exploited.

Prelimi-

n a r y c a l c u l a t i o n s u s i n g t h i s i d e a h a v e b e e n made by Ghaemi (1980).

T h i s bounds c a l c u l a t i o n c o u l d r e a d i l y be i n t e r f a c e d

w i t h t h e a p p r o a c h p r o p o s e d by Benson (1980) f o r s i m u l t a n e o u s l y c a l c u l a t i n g t h e s o l u t i o n s of a g i v e n problem c o r r e s p o n d i n g t o s e v e r a l s p e c i f i e d resource ( r h s ) l e v e l s i n one p as s o f a b r a n c h - and-bound a l g o r i t h m

.

Bounds r e s u l t s i n t e r m s o f o r i g i n a l p r o b l e m d a t a , s u c h as t h o s e i n d i c a t e d i n C h a p t e r 2 by D a n i e l (1973) f o r q u a d r a t i c programming a n d , a t a h i g h e r l e v e l o f g e n e r a l i t y , by Robinson ( 1 9 7 6 a ) , are o f c o n s i d e r a b l e i n t e r e s t b u t remain t o b e comput a t i o n a l l y implemented.

A study of t h e connection with sensi-

t i v i t y a n d s t a b i l i t y t h e o r y of p r o c e d u r e s f o r g e n e r a t i n g s o l u t i o n error bounds, e . g . ,

t e c h n i q u e s u t i l i z i n g t h e u s e of

i n t e r v a l a r i t h m e t i c c o n c e p t s s u c h as t h o s e p r o p o s e d by Robinson (1973b) , Mancini and McCormick (1979) , and McCormick (1980) , s h o u l d be a s u b j e c t of f r u i t f u l r e s e a r c h . C o m p u t a t i o n a l t e c h n i q u e s f o r o p t i m a l v a l u e and KarushKuhn-Tucker t r i p l e s e n s i t i v i t y a n d s t a b i l i t y bounds m i g h t be e x p e c t e d t o d e v e l o p from t h e i n c i s i v e r e s u l t s d e v e l o p e d i n t h e f o l l o w i n g areas:

( i ) g e n e r a l i z e d e q u a t i o n s (Robinson, 1 9 7 6 a , b ,

1979, 1 9 8 0 a ) ; ( i i ) d i f f e r e n t i a l s t a b i l i t y a n d s u b d i f f e r e n t i a l a n a l y s i s o f t h e o p t i m a l v a l u e f u n c t i o n (Gollan, 1981a,b; R o c k a f e l l a r , 1 9 8 2 a , b ) ; (iii) d e f o r m a t i o n , c o n t i n u a t i o n a n d p a r a m e t r i c imbedding (Mine, Fukushima, a n d Ryang, 1 9 7 7 , 1978;

284


K o j i m a and Hirabayashi, 1981); ( i v ) higher-order

( t h a n two)

o p t i m a l i t y c o n d i t i o n s a p p l i e d t o s t a b i l i t y a n a l y s i s and para-

metric programming ( G o l l a n , 1 9 8 1 ~ ) ;( v ) g e n e r i c c o n d i t i o n s f o r o p t i m a l i t y a n d s t a b i l i t y ( S p i n g a r n and R o c k a f e l l a r , 1 9 7 9 ) ; ( v i ) c o n n e c t i o n s between nonconvex d u a l i t y r e s u l t s a n d s t a b i l i t y ( B a l d e r , 1977; D o l e c k i a n d Kurcyusz, 1 9 7 8 ) ; ( v i i ) d e t e r m i n a t i o n of r e g i o n s o f s t a b i l i t y (Zlobec, Gardner, and Ben-Israel,

1982);

( v i i i ) r e c e n t a d v a n c e s i n p a r a m e t r i c l i n e a r a n d n o n l i n e a r programming ( G a l , 1979; NoiiEka e t a Z . , 1974; Lommatzsch, 1979; Bank e t a l . ,

1982; Brosowski, 1 9 8 2 ) ; a n d ( i x ) t h e a p p l i c a t i o n

of c o n t e m p o r a r y t e c h n i q u e s i n nonsmooth o p t i m i z a t i o n methodo l o g y t o s e n s i t i v i t y , s t a b i l i t y , a n d p a r a m e t r i c programming methodology

(Gauvin, 1 9 7 8 ) .

Some a p p l i c a t i o n s o f s e n s i t i v i t y and s t a b i l i t y a n a l y s i s i n m a t h e m a t i c a l programming a r e r a t h e r o b v i o u s , e . g . ,

estimation

of s o l u t i o n s o f p e r t u r b e d p r o b l e m s , g i v e n a s o l u t i o n o f a p r o b l e m w i t h g i v e n p a r a m e t e r v a l u e s , a n d d e t e r m i n a t i o n of parameters t o which t h e o p t i m a l v a l u e or s o l u t i o n p o i n t i s most s e n s i t i v e .

Numerous a p p l i c a t i o n s h a v e b e e n i n d i c a t e d

t h r o u g h o u t t h i s book. here:

W e c o l l e c t a number o f t h e s e a n d o t h e r s

( i )o p t i m a l i t y c o n d i t i o n s a n d a l g o r i t h m d e r i v a t i o n a n d

a c c e l e r a t i o n ( F i a c c o a n d McCormick, 1968; F i a c c o , 1 9 7 6 ) ; (ii) c o n v e r g e n c e o f a l g o r i t h m s (Meyer, 1 9 7 0 ) ; (iii)rate o f conv e r g e n c e o f a l g o r i t h m s (Robinson, 1 9 7 4 ) ; ( i v ) d e c o m p o s i t i o n ( G e o f f r i o n , 1970; Lasdon, 1970, Gauvin, 1 9 7 8 ) ; ( v ) i m p l i c i t l y defined-function

m i n i m i z a t i o n (Hogan, 1973a; d e S i l v a , 1 9 7 8 ) ;

( v i ) model v a l i d a t i o n a n d c o s t b e n e f i t a n a l y s i s ( A r c h e t t i e t aZ., 1980; F i a c c o a n d Ghaemi, 197933, 1 9 8 2 ) ; a n d homotopy c o n t i n u a t i o n methods

( R h e i n b o l d t , 1969; Garcia a n d Gould, 1978; Garcia

and Z a n g w i l l , 1 9 7 9 ; A l l g o w e r a n d Georg, 1 9 8 0 ) .

Conversely,

285

10.3 Conclusions

a l l t h e areas m e n t i o n e d w i l l u n d o u b t e d l y c o n t r i b u t e t o t h e development o f m a t h e m a t i c a l programming s e n s i t i v i t y a n d s t a b i l i t y methodology. 10.3.

CONCLUSIONS

The f u t u r e s h o u l d see a c o n c e r t e d e f f o r t t o w a r d s t h e f u r t h e r u n i f i c a t i o n of t h e powerful c o l l e c t i o n o f s e n s i t i v i t y a n d s t a b i l i t y r e s u l t s t h a t are s c a t t e r e d t h r o u g h o u t t h e l i t e r ature.

U n i f i c a t i o n i n e v i t a b l y f o l l o w s t h e development o f a

c r i t i c a l mass of r e s u l t s t h a t are i n c i s i v e enough a n d s u f f i c i e n t l y c o h e s i v e t o p r o v i d e a s i g n i f i c a n t body o f t h e o r y and a methodology.

The m a t u r a t i o n o f t h e s t a t e of t h e a r t i s s i g -

n a l l e d by t h e emergence o f books t h a t u l t i m a t e l y p r o v i d e t h e u n i f i c a t i o n of t h e s c i e n c e .

Only a few books h a v e b e e n w r i t t e n

i n t h i s g e n e r a l s u b j e c t a r e a i n "continuous" mathematical programming.

The f i r s t , by D i n k e l b a c h (19691, t r e a t s p a r a -

metric s e n s i t i v i t y a n a l y s i s i n l i n e a r p r o g r a m i n g .

Books i n

p a r a m e t r i c l i n e a r programming h a v e a l s o b e e n w r i t t e n , by Gal (1979) ; NoiiEka e t aZ.

( 1 9 7 4 ) ; Lommatzsch ( e d i t o r , 1 9 7 9 ) :

Kausmann, Lommatzsch, a n d NoiiEka ( 1 9 7 6 ) ; a n d Lorenzen ( 1 9 7 4 ) . Books i n p a r a m e t r i c n o n l i n e a r programming h a v e o n l y r e c e n t l y b e e n c o m p l e t e d , by Bank e t aZ.

( 1 9 8 2 ) a n d Brosowski (19821, t h e

f o r m e r e m p h a s i z i n g l i n e a r a n d q u a d r a t i c p a r a m e t r i c programming, the latter semi-infinite

( l i n e a r ) p a r a m e t r i c programming.

This

book is a p p a r e n t l y t h e f i r s t d e v o t e d e n t i r e l y t o g e n e r a l cont i n u o u s m a t h e m a t i c a l programming s e n s i t i v i t y a n d s t a b i l i t y analysis results. Of c o u r s e , s e n s i t i v i t y a n d s t a b i l i t y r e s u l t s are d i r e c t l y

a p p l i c a b l e a n d i n d e e d are t h e b u i l d i n g b l o c k s f o r c o n s t r u c t i n g and c h a r a c t e r i z i n g p a r a m e t r i c s o l u t i o n s .

U n i f i c a t i o n of

286


s e n s i t i v i t y and s t a b i l i t y methodology w i l l a c c e l e r a t e t h e u n d e r s t a n d i n g o f b a s i c t h e o r y , s t i m u l a t e a l g o r i t h m s and s o f t w a r e developments, and h a s t e n w i d e s p r e a d , r o u t i n e implementations i n a l l a r e a s of a p p l i c a t i o n . S e n s i t i v i t y and s t a b i l i t y a n a l y s i s r e s u l t s are r e a d y f o r e x t e n s i v e computational implementation.

Experimental r e s u l t s

have u n e q u i v o c a l l y d e m o n s t r a t e d t h e p r a c t i c a l a p p l i c a b i l i t y o f v a r i o u s c o m p u t a t i o n a l schemes t h a t can g e n e r a t e a w e a l t h o f i n f o r m a t i o n t h a t s h o u l d be e x t r e m e l y v a l u a b l e t o u s e r s .

As

n o t e d i n t h i s book, e x t e n s i v e c o m p u t a t i o n s have been s u c c e s s f u l l y performed w i t h t h e SENSUMT computer program t h a t i n t e r f a c e s subprograms c a l c u l a t i n g s e n s i t i v i t y i n f o r m a t i o n w i t h t h e

w e l l known SUMT program.

The p r o c e d u r e u t i l i z e s i n f o r m a t i o n

g e n e r a t e d by a s o l u t i o n a l g o r i t h m t o c a l c u l a t e s e n s i t i v i t y and s t a b i l i t y bound i n f o r m a t i o n a s a s o l u t i o n i s approached.

This

i d e a i s n a t u r a l and e f f e c t i v e and i s b a s e d on t h e t h e o r e t i c a l l y d e m o n s t r a b l e f a c t t h a t s o l u t i o n and s e n s i t i v i t y and s t a b i l i t y i n f o r m a t i o n r e q u i r e e s s e n t i a l l y t h e same a s s u m p t i o n s and a n a l ogous c a l c u l a t i o n s .

The p a r t i c u l a r a l g o r i t h m u t i l i z e d t o

o b t a i n t h e c o m p u t a t i o n a l r e s u l t s r e p o r t e d i n t h i s book i s based on a c l a s s i c a l p e n a l t y f u n c t i o n .

However, t h e approach i s

c l e a r l y a p p l i c a b l e t o any v a l i d s o l u t i o n a l g o r i t h m and s h o u l d y i e l d an e f f i c i e n t p r o c e d u r e f o r a d d i n g a s e n s i t i v i t y and s t a b i l i t y a n a l y s i s c a p a b i l i t y t o any n o n l i n e a r programming code, once t h e f o r m u l a s a p p r o p r i a t e t o t h e g e n e r a l a l g o r i t h m i c m a n i p u l a t i o n s and d a t a o r g a n i z a t i o n a r e developed. W e have p r o v i d e d f i r s t - and second-order

formulas €or projected-gradient,

sensitivity

reduced-gradient,

and

augmented-Lagrangian c a l c u l a t i o n s , and have shown t h e i n t i m a t e and n a t u r a l c o n n e c t i o n between s e n s i t i v i t y c a l c u l a t i o n s and

287

10.3 Conclusions

c a l c u l a t i o n s b a s e d on Newton's method.

Thus, t h e e x p l o i t a t i o n

of s o l u t i o n algorithm c a l c u l a t i o n s t o e s t i m a t e s e n s i t i v i t y i n f o r m a t i o n a s a s o l u t i o n i s approached h a s been demonstrated f o r l a r g e c l a s s e s o f t h e most e f f e c t i v e a l g o r i t h m s .

Aside

from t h e e x p l o i t a t i o n o f a l g o r i t h m i c c a l c u l a t i o n s , w e have g i v e n s e n s i t i v i t y and s t a b i l i t y bound f o r m u l a s d i r e c t l y i n

t e r m s o f t h e g i v e n problem f u n c t i o n s . Computational e x p e r i e n c e d e m o n s t r a t e s t h e p r a c t i c a l f e a s i b i l i t y and a p p l i c a b i l i t y o f g e n e r a t i n g r a t h e r i n t r i c a t e s e n s i t i v i t y and s t a b i l i t y i n f o r m a t i o n f o r n o n l i n e a r programming problems o f p r a c t i c a l i n t e r e s t .

Indeed, given t h e c l o s e

c o n n e c t i o n between s o l u t i o n a l g o r i t h m and s e n s i t i v i t y and s t a b i l i t y c a l c u l a t i o n s , it s e e m s a p p a r e n t t h a t i f an a l g o r i t h m can d e t e r m i n e a s o l u t i o n , t h e n a l g o r i t h m i c m a n i p u l a t i o n s can be u t i l i z e d t o y i e l d an abundance o f s e n s i t i v i t y and s t a b i l i t y i n f o r m a t i o n a s t h e s o l u t i o n i s approached.

Practitioners

s h o u l d know t h a t t h e r e i s no t h e o r e t i c a l impediment t o t h e c o m p u t a t i o n a l i m p l e m e n t a t i o n and r o u t i n e e x p l o i t a t i o n of t h i s fact. A s a f i n a l comment, it s h o u l d be s t r e s s e d t h a t s e n s i t i v i t y

and s t a b i l i t y methodology i s n o t merely a c o l l e c t i o n of t e c h niques f o r generating i n s i g h t s concerning t h e s t a t u s of a s o l u t i o n r e l a t i v e t o d a t a changes.

I t i s a p o i n t of view, an

a p p r o a c h , from which a l l a s p e c t s o f m a t h e m a t i c a l programming can and undoubtedly w i l l be r e d e v e l o p e d and e x t e n d e d .


Appendix I

Notation, Conventions, and Symbols

1.

INTRODUCTION Definitions of many key concepts are given in the text as

they are first introduced, or appropriate references are suggested.

Here, for convenience, we summarize frequently used

conventions.

We have drawn freely from other sources, especi-

ally from the lucid summary of standard conventions given by Mangasarian (1969, Chap. 1 and App. A-D).

Several references

that have been consulted or that may be recommended are Berge (1963) for topology and point-to-set maps; Fiacco and McCormick (1968) for first- and second-order optimality conditions, convex programming, and Wolfe duality and penalty functions; Hestenes (1966) for results in the calculus of variations and optimal control; Hestenes (1975) and Luenberger (1969) for functional analysis results that are applicable to optimization; Luenberger (1973) for basic results in linear and nonlinear programming and for an introduction to nonlinear programming algorithms; Mangasarian (1969) for first-order optimality conditions, constraint qualifications, and convexity notions; Ortega and Rheinboldt (1970) for general results in functional analysis, linear algebra, and iterative processes;

290

Appendix I

a n d R o c k a f e l l a r (1970) f o r convex a n a l y s i s .

The books by

L u e n b e r g e r a n d O r t e g a and R h e i n b o l d t a r e l i s t e d i n t h e s e l e c t e d b i b l i o g r a p h y and t h e o t h e r s i n t h e r e f e r e n c e s . SETS, VECTORS, AND SPACES

2.

S e t s a n d s p a c e s are u s u a l l y d e n o t e d by c a p i t a l A r a b i c

l e t t e r s , v e c t o r s a n d c o n s t a n t s by s m a l l A r a b i c l e t t e r s , a n d t h e empty s e t by fl.

The i n t e r s e c t i o n o f t h e t w o sets M a n d N

i s t h e set o f e l e m e n t s b e l o n g i n g t o b o t h M a n d N a n d i s d e n o t e d The u n i o n o f M a n d N i s t h e s e t o f e l e m e n t s be-

by M C7 N.

l o n g i n g t o e i t h e r M o r N a n d i s d e n o t e d by M U N .

I f M is a

s u b s e t o f N , w e w r i t e M C N o r N 3 M.

T h i s means t h a t e a c h

e l e m e n t of M i s a l s o a n e l e m e n t o f N .

I f x i s an element of M,

t h i s i s d e n o t e d by x x f M.

E

and i f x i s n o t a n e l e m e n t o f M, by

M,

W e sometimes w r i t e x , y

E

M, f o r x

M and y

E

E

M.

In

g e n e r a l , a s l a s h across a symbol d e n o t e s i t s n e g a t i o n . I f M i s t h e set o f x

E

N having an a d d i t i o n a l property,

s a y f ( x ) = 0 , where f i s a r e a l - v a l u e d f u n c t i o n , t h e n w e w r i t e M = {x

E

N l f ( x ) = 0) or M = { x l f ( x ) = 0) i f x

from t h e c o n t e x t .

E

N is understood

I f t h e e l e m e n t s of M c a n b e l i s t e d , e . g . ,

i f M i s t h e s e t o f i n t e g e r s from 1 t h r o u g h m , w e may w r i t e M = 11, 2 ,

..., m).

I f M C N and N C M w e w r i t e M = N, and i f

t h e r e a r e no e l e m e n t s i n M w e w r i t e M = fl. M

#

Thus, M # N o r

means M a n d N are n o t t h e same s e t s , o r t h a t M i s n o t

empty, r e s p e c t i v e l y . The ( C a r t e s i a n ) p r o d u c t o f two sets M a n d N i s d e n o t e d by M x N and i s d e f i n e d a s t h e s e t o f o r d e r e d p a i r s

by M

x

N = {(x, y ) ( x

E

M,

y

E

N).

(x, y) given

291

Appendix I

The symbol En r e p r e s e n t s n - d i m e n s i o n a l E u c l i d e a n s p a c e , whose b a s i c p r o p e r t i e s are assumed known. x = (xl,

..., x n I T I a n

Thus, x e En means

o r d e r e d n - t u p l e of r e a l numbers x l ,

...,

xn, d e f i n i n g a n n-component column "vector," t h e s u p e r s c r i p t T The v e c t o r x e En i s a l s o c a l l e d a

denoting transposition.

F r e q u e n t l y u s e d p r o p e r t i e s of En w i l l b e 1 q u i c k l y reviewed. I f x , y e En a n d a E , then x + y = (x 1 " p o i n t " i n En.

yl,

..., xn

+ ynIT and a x

t h i s i s aM = I x l x = az, z

z

E NI,

..., ax,).T

= (axl, E

MI a n d M

+

+

Consistent with

N = {xlx = y

+ z, y

e M,

i f M, N c E". T

The " s c a l a r " o r " i n n e r " p r o d u c t x y of two v e c t o r s T

+

x , y E En i s d e f i n e d by x y = xlyl

8 . .

+

xnyn.

The E u c l i d e a n

norm llxll of x E En is d e f i n e d a s llxll = (xTx)l'*. t a n c e between x

En a n d y e En i s g i v e n by

E

11 x -

The d i s y ( I , which

d e f i n e s a metric on En. A

" n e i g h b o r h o o d " of a p o i n t xo i n a g i v e n s p a c e i s d e n o t e d

by N ( x o ) a n d i s t a k e n t o mean a n o p e n s u b s e t o f t h e s p a c e c o n t a i n i n g xo.

I n En,

a n e i g h b o r h o o d of xo E En may be t a k e n

t o b e a n "open b a l l " o f r a d i u s A ( c e n t e r e d ) a t xo g i v e n by B A ( x O )= { x E En

I

IIx

-

xoII < A 1 f o r some r e a l number A > 0.

W e may t h e n d e f i n e x t o b e a n " i n t e r i o r p o i n t " of M C En i f

some n e i g h b o r h o o d o f x i s c o n t a i n e d i n M.

I f e v e r y p o i n t of M

i s a n i n t e r i o r p o i n t of M, t h e n M i s s a i d t o be open.

A point

x i s s a i d t o b e a p o i n t of c l o s u r e o f a s e t M i f e v e r y n e i g h borhood of x c o n t a i n s a t l e a s t o n e e l e m e n t of M. need n o t i t s e l f be a n e l e m e n t of M . )

(Note that x

The s e t M i s s a i d t o b e

c l o s e d i f e v e r y p o i n t o f c l o s u r e o f M i s i n M.

The s e t of

p o i n t s of c l o s u r e o f M i s c a l l e d t h e c l o s u r e of M a n d i s den o t e d by

E.

The s e t o f i n t e r i o r p o i n t s o f M i s c a l l e d t h e

i n t e r i o r o f M a n d i s d e n o t e d by M

0

.

A boundary p o i n t o f M i s

292

Appendix I

any p o i n t x s u c h t h a t e v e r y n e i g h b o r h o o d o f x c o n t a i n s a t l e a s t one e l e m e n t of M a n d o n e e l e m e n t n o t i n M.

A set M i s c a l l e d

bounded i f t h e r e e x i s t s a r e a l number c s u c h t h a t IIxII 5 c f o r each x

M.

E

a n d bounded.

The s e t M

c

En i s c a l l e d compact i f i t i s c l o s e d

An i m p o r t a n t p r o p e r t y o f a compact s e t M i s t h a t

e v e r y s e q u e n c e o f p o i n t s i n M h a s a l i m i t p o i n t i n M. Although xi may b e u s e d t o d e s i g n a t e t h e i - t h component o f t h e v e c t o r x, t h i s n o t a t i o n w i l l be more f r e q u e n t l y u s e d t o denote a p a r t i c u l a r v e c t o r i n t h e given space.

w i l l be clear from t h e c o n t e x t . t h e n {x,) o f {x,)

d e n o t e s t h e "Sequence" x l ,

k-*m

- -X I [

o r xk

x2,

+

-

...,

En f o r k = 1, 2 ,

... .

A "subsequence"

... .

The 1 , where j = 1, 2 , j E" c o n v e r g e s t o a # * l i m i t " x i f and o n l y i f

= 0 , which means t h a t , g i v e n 6 > 0 ,

t h e r e e x i s t s SF

XI\ < 6 f o r a l l k 1. F.

W e a l s o w r i t e l i m xk = kTI t o e x p r e s s t h i s c o n v e r g e n c e . A p o i n t 2 E En i s a

such t h a t ( ( x k

x

E

w i l l b e d e n o t e d by {x,

sequence {xk) c

l i m IIxk

I f xk

The meaning

" l i m i t p o i n t " o f {x,)

i f and o n l y i f t h e r e e x i s t s a subsequence

s u c h t h a t xk + 2 . I f {xk} c o n v e r g e s t o X, it i s j j c l e a r t h a t a n y c o n v e r g e n t s u b s e q u e n c e Ix, 1 must a l s o c o n v e r g e {xk 1 o f {x,)

t o TI, i.e.,

x is

j t h e u n i q u e l i m i t p o i n t o f { x k l i f and o n l y i f

K is t h e l i m i t of {xk). The f o l l o w i n g c o n s t r u c t s p r o v i d e a d d i t i o n a l v e r y i m p o r t a n t a n d f r e q u e n t l y u s e d l i m i t i n g p r o p e r t i e s o f a s e q u e n c e {x,)

real numbers t h a t n e e d n o t b e c o n v e r g e n t .

of

They e s s e n t i a l l y

d e f i n e t h e smallest a n d l a r g e s t l i m i t p o i n t s o f t h e s e q u e n c e . L e t Sk = { x i ( i 1. k ) and zk = i n f Sk, f o r k = 1, 2 ,

Sk+l C Sk, w e must h a v e z

... .

~ 1.+ zk~ f o r e a c h k, i.e.,

n o n d e c r e a s i n g s e q u e n c e o f r e a l numbers.

(2,)

Hence, if {x,)

bounded, it f o l l o w s t h a t zk * 8, a number c a l l e d t h e

Since

is a

is

293

Appendix I

lim xk

"limit inferior" of {xk}, and denoted by is not bounded below, we write superior" of {x,) If

lim k-tw

X,

=

k+m

lim xk = Z or xk

,.

k

k+m

=

-m.

is denoted and defined by

+ J xk = Z,

k-+m

lim x

k-tm

-+

= S.

If {x,)

The "limit k+m

xk = -=(-xk).

then {xk} converges to S, i.e.

k+m

,

S, in terms of our previous notation.

also write "lim inf" for

lim and

We

"lim sup" for E.

The notation lim $(x) = $(T)or $(x)

+

$ ( S ) as x

+

Z is

X-+Z

frequently used, where x and S may be vectors or numbers and $ is a real-valued vector function.

The symbol x

+

H means that

x converges to Z in any manner, i.e., we may take x = xk where {x,}

S

is any sequence (in the set to which x may be

restricted). -+

+

For each such {xk),

$L

(x) + $ ( S ) means that @(xk)

$ (9).

It is convenient to have a convention €or limiting an operation to elements that are a subset of a given set of elements.

The practice we follow is common and is illustrated

by the following examples:

if B C N = 11,

..., n},

then

CieB zi means sum those elements zi over those i belonging to the subset B of integers 1, n; denotes the

...,

UtEN(~-r(t)

union of those sets r(t) over those values of t belonging to

N < F ) C TI where N(F) may, for example, represent a neighborhood of

F

in some space TI or T may be an index set and we may

have r(t) =

r t'

We close this section with one additional very important concept, "linear independence" of several vectors. combination" of xl,

..., xm

E

"linear

En is a vector y E En such that Am E E1 If y = 0 implies that

...,

y = Cmi=l X.X. for some hi, 1 1 each Xi = 0, then the vectors xl, linearly independent.

A

.

..., xm

Otherwise, i.e.

,

are said to be

if Cy=l Xixi = 0 and

294

Appendix I

some A

..., xm a r e

i # 0, t h e n xl,

dependent."

s a i d t o be " l i n e a r l y

I t f o l l o w s t h a t any m v e c t o r s i n En a r e l i n e a r l y

dependent i f m > n. 3.

FUNCTIONS AND MAPPINGS "Maps" ( o r mappings) and o p e r a t o r s a r e g e n e r a l l y d e n o t e d

by c a p i t a l A r a b i c o r Greek l e t t e r s , w h i l e s m a l l A r a b i c o r Greek l e t t e r s a r e u s u a l l y used f o r r e a l - o r v e c t o r - v a l u e d f u n c t i o n s . A "mapping"

r

from t h e s e t X t o t h e s e t Y i s a correspond-

r

e n c e a s s o c i a t i n g a s u b s e t ( c a l l e d t h e "image" o r " v a l u e " o f a t x) r ( x ) of Y with every x

E

A " f u n c t i o n " $ from a s e t X

X.

i n t o a set Y i s a correspondence t h a t a s s o c i a t e s with every element x

E

X a s i n g l e element y

E

Y.

W e w r i t e 4:

X

-+

Y to

d e s i g n a t e t h i s c o r r e s p o n d e n c e and @ ( x )= y t o d e n o t e t h e f a c t t h a t t h e image ( o r v a l u e ) $ ( x ) o f $ a t x i s t h e s i n g l e e l e m e n t y.

A scalar-valued o r real-valued

from a s e t X i n t o E

.

1

function $ is a function

An m-dimensional v e c t o r f u n c t i o n $ i s a

f u n c t i o n from X i n t o Em.

The components $ l ( x ) ,

$ ( x ) a r e r e a l - v a l u e d f u n c t i o n s on X.

..., $,(x)

of

I f e a c h component of

$(x) h a s a g i v e n p r o p e r t y , t h e n u s u a l l y $ ( x ) i s s a i d t o have t h a t property.

The n o t a t i o n $

0 , $ > 0, $ = 0, $ # 0 means

t h a t e a c h component o f 4 i s n o n n e g a t i v e , o r p o s i t i v e , o r 0 , o r t h a t some component i s n o t 0 , r e s p e c t i v e l y . A

real-valued

f u n c t i o n $ d e f i n e d on an open s e t M C E

s a i d t o be c o n t i n u o u s a t TI

E M

quence {xk} C M such t h a t xk

-+

i f $(xk)

x.

-+

$(?)

n

is

f o r e a c h se-

The f u n c t i o n $ i s s a i d t o

be c o n t i n u o u s on M i f i t i s c o n t i n u o u s a t e a c h p o i n t o f M. C o n t i n u i t y p r o p e r t i e s o f f u n c t i o n s and maps a r e f u r t h e r d i s c u s s e d i n C h a p t e r 2 , pp. 11-14.

295

Appendix I

Consistent with the above, an m-component vector-valued function @ = (@l,

.'I'

..., @m)

is said to be continuous if each

component function @i (i = 1,

..., m)

is continuous.

If each

component of @ is continuous on an open subset of the space containing the set M we write

@

C(M), or simply

E

@

E

C if

restriction to M is understood or if M is the entire space. If the component functions have continuous partial derivatives of order k, this is designated analogously by @ k @ E C .

The infimum of a real-valued function is a number f3 such that @(x) 1. 6 for all x

E

Ck (M) or

defined on a set M

@ E

M (i.e., @ is

bounded from below on M by 8 ) and further, given any 6 > 0, there exists x @(x)

= f3

E

M such that @(x) < B

or f3 = infx @(x) s.t. x

ject to." x

E

If there exists T

E

E

+

6.

We write infxeM

M, where "s.t." means "sub-

M such that @(F) 5 @(XI forevery

M, then @(F) is called the "minimum value" or "global

minimum value" or "optimal value" of @ on M I and I is called a "global minimum'' of

@(F) = minX @(x)

s

@

on M.

t. x

e

We write @(F) = minxeM @(x) or If

M.

is a global minimum of @ on

M n B X (I)for some X > 0, then @(TI) is called a "local minimum value" or (locally optimal value and Z a "local minimum" or "local solution. 'I Supremum and maximum are defined analogously to infimum Specifically, supxeM @ (x) = -infxeM[-@(x) I and

and minimum.

-minxeM [-@(x)], so that it suffices to study either minimization or maximization. An important result is

maxxeM @(x)

=

that a lower (upper) semicontinuous function (see definition on p. 12) @ defined on a nonempty compact set M C En is bounded from below (above) and there exists x global minimum (maximum) of

@

on M.

e

M such that W is a

296

Appendix I A real-valued

f u n c t i o n $ d e f i n e d on a n open s e t M C En i s

s a i d t o be " d i f f e r e n t i a b l e " a t P

+ 8 ( X , x ) llxll f o r a l l x

E En

E

M if

such t h a t

$(P + x )

X + x

a v e c t o r ( w i t h f i n i t e components) and B ( f ,

$(P)+ v(P)'x

M, where v ( X ) i s

E

x)

=

-P

0 as x

-P

0.

The f u n c t i o n 4 i s s a i d t o be d i f f e r e n t i a b l e on M i f i t i s The f u n c t i o n $ i s s a i d t o

d i f f e r e n t i a b l e a t e a c h p o i n t i n M. have a p a r t i a l d e r i v a t i v e a t

x with

ponent xi o f x i f t h e q u o t i e n t t o a f i n i t e number a s 6

+.

r e s p e c t t o t h e i - t h com-

[$(P+ 6 ei)

0 , where ei

E

-

$(P)]/6c o n v e r g e s

En i s a v e c t o r h a v i n g

one i n t h e i - t h component and z e r o i n e a c h r e m a i n i n g component. When i t e x i s t s and i s f i n i t e , t h e l i m i t o f t h e g i v e n q u o t i e n t

i s c a l l e d t h e " p a r t i a l d e r i v a t i v e " o f $ w i t h r e s p e c t t o xi a t

Z and i s d e n o t e d by a $ ( P ) / 3 x i . V $ (X) = 1 3 4 (P)/ax,,

..., a $ ( P ) / a x n ]

(row) v e c t o r

i s c a l l e d the "gradient"

I f $ i s d i f f e r e n t i a b l e a t R, t h e n V $ ( f I )

o f $ a t Z.

w e may t a k e v ( E ) = V $ ( f I )

@(K+ x).

The n-component

e x i s t s and

i n t h e e x p r e s s i o n g i v e n above f o r

Another i m p o r t a n t f a c t i s t h a t i f V$(W) e x i s t s and

V $ i s c o n t i n u o u s a t R, t h e n C$ i s d i f f e r e n t i a b l e a t X. I f 4 = ($1,

...,

an m-component v e c t o r f u n c t i o n

d e f i n e d on a n open s e t M C En,

t h e n $ i s s a i d t o be d i f f e r -

M i f e a c h o f i t s components $i i s d i f f e r e n -

entiable a t W

E

t i a b l e a t P.

A s before,

$ i s s a i d t o be d i f f e r e n t i a b l e on M

i f it i s d i f f e r e n t i a b l e a t e v e r y p o i n t o f PI.

The v e c t o r func-

t i o n $ i s s a i d t o have p a r t i a l d e r i v a t i v e s a t W w i t h r e s p e c t t o t h e components xl,

..., xn

of x

E

E n i f e a c h component Qi

h a s p a r t i a l d e r i v a t i v e s a t R w i t h r e s p e c t t o xl,

..., xn.

The

Appendix I

297

n o t a t i o n V$(H) t h e n r e p r e s e n t s

an m

x

n m a t r i x c a l l e d t h e J a c o b i a n o f $ a t K.

The r e a l - v a l u e d f u n c t i o n $ d e f i n e d on an open set M

c En

$(X + x )

i s s a i d t o be " t w i c e d i f f e r e n t i a b l e " a t X E M i f

=

~ $ ( ~ r +) x( 1 / 2 ) x T v 2 $ ( z ) x + y ~ x)llx112, , f o r a l l x E E" 2 such t h a t + x E M, where V is an n x n matrix (with

+

x

$(x)

f i n i t e e l e m e n t s ) and y ( Z , x )

+

0 as x

i s c a l l e d t h e H e s s i a n o f $ a t Z.

+

0.

The m a t r i x V2$(F)

I t f o l l o w s t h a t i f V$ is

d i f f e r e n t i a b l e a t E o r i f V$ h a s c o n t i n u o u s ( f i r s t p a r t i a l ) d e r i v a t i v e s a t K, t h e n $ i s t w i c e d i f f e r e n t i a b l e a t K and

1

a 2 $(rr)/ax,ax, /axlaxn a2$ (TI)

- 9 .

a 2 $(rr)/axnaxl a2$

( a/axnaxn

so t h a t t h e i j - t h e l e m e n t of V 2 $(K) i s g i v e n by f o r i , j = 1, of $ a t I.

..., n ,

a 2$(K)/ax.axi

7 c a l l e d t h e "second p a r t i a l d e r i v a t i v e s "

2

~f V $ i s c o n t i n u o u s a t

X (i.e., i f t h e elements

of V 2 $ a r e c o n t i n u o u s ) , t h e n V2$ (K) i s symmetric. p a r t i a l derivatives of

+

The k-th

a t H a r e defined analogously.

298

Appendix I

d e f i n e d , w e may g i v e t h e well-known

With V $ a n d V$ '

"mean-

v a l u e " a n d T a y l o r ' s s e c o n d - o r d e r t h e o r e m s , e a c h of which w e use f r e q u e n t l y .

The "mean-value theorem,

'I

which w e a l s o r e f e r

to a s a f i r s t - o r d e r Taylor's expansion, states t h a t i f $ i s a r e a l - v a l u e d d i f f e r e n t i a b l e f u n c t i o n o n a n open convex s e t

M C En, t h e n $(XI = $(Y) + V$[Q(X, y , 6 ) 1 ( x f o r any g i v e n x , y

E

M,

-

Y)

where q ( x , y , 6 ) = y

some r e a l number 6 s u c h t h a t 0 < 6 < 1.

+

6(x

-

y) for

I f , additionally, $

i s t w i c e d i f f e r e n t i a b l e o n M, t h e n T a y l o r ' s t h e o r e m a s s u r e s t h e v a l i d i t y of t h e " s e c o n d - o r d e r T a y l o r ' s e x p a n s i o n , " $ ( x ) = $(Y) + V$(Y) ( x + 1/2(x

where ~ ( x y, ,

-

B) = y

Y)

+

-

Y)

T 2

v $[n

B(x

-

(x,

Yr

B) 1 (x

-

Y)

I

y ) f o r s o m e r e a l number B s u c h

t h a t 0 < 6 < 1. 4.

MATRICES

Matrices, o t h e r t h a n t h e H e s s i a n V 2 f of a r e a l - v a l u e d f u n c t i o n f or t h e J a c o b i a n V$ of a v e c t o r - v a l u e d

f u n c t i o n $,

w i l l g e n e r a l l y b e d e n o t e d by c a p i t a l A r a b i c l e t t e r s .

An m

x

where a i j ,

p m a t r i x A i s d e f i n e d as

..., m, j = 1, ..., p , (ail, ..., a ) i s c a l l e d iP

i = 1,

The r o w vector

t h e column v e c t o r ( a l j , o f A.

..., am j ) T i s

a r e real numbers. t h e i - t h r o w o f A and

c a l l e d t h e j - t h column

The " t r a n s p o s e " o f A i s d e n o t e d by AT a n d i s t h e p

x

m

299

Appendix I

m a t r i x d e f i n e d as

The m a t r i x A i s s a i d t o b e "symmetric" i f A = A

T

.

A submatrix

of A i s a m a t r i x w i t h rows a n d columns e x t r a c t e d , r e s p e c t i v e l y , from rows a n d columns o f A. The r a n k o f a m a t r i x A i s t h e l a r g e s t number o f l i n e a r l y i n d e p e n d e n t r o w s ( c a l l e d t h e r o w r a n k ) or columns ( c a l l e d t h e column r a n k , which a l w a y s e q u a l s t h e r o w r a n k ) .

An

m

x

m

m a t r i x A i s s a i d t o be " n o n s i n g u l a r " i f it h a s r a n k m , i . e . , i f it h a s "full r a n k . "

m

x

I n t h i s i n s t a n c e , t h e r e e x i s t s a unique

m m a t r i x c a l l e d t h e i n v e r s e o f A a n d d e n o t e d by A - l

t h a t A A-l m a t r i x"

1 =

= A-lA

[:

- 0 .

= I, where I d e n o t e s t h e m x

:I,

such

m "identity

where t h e z e r o e n t r i e s mean t h a t a l l t h e "of f - d i a g o n a l " eleW e sometimes w r i t e A - ~f o r ( A - ~ ) ~ .

ments are zero. I f A is an m

x

p m a t r i x and B i s a p

x

q m a t r i x d e f i n e d as

follows,

where ai,

b.

3

E

Ep,

i = 1,

..., m

a n d j = 1,

..., q,

then t h e

300

Appendix I

product AB, A times B, exists and is defined as

It follows that (AB)T = BTA T If A and B are m

x

.

p matrices, then the sum A

+

B

exists and is defined as

A + B =

I

where A is defined as above and B analogously. product aA for any a

E

Also, the

E1 is the matrix whose ij-th element is

aaijm

An

T z Az

x

m matrix A is said to be positive semidefinite if

0 for all z

E

Em and positive definite if zTAz > 0 for

Em such that z # 0.

Similarly, an m x m matrix A is said to be negative semidefinite if zTAz 5 0 €or all z E Em and negative definite if zTAz < 0 for all z c Em such that all z

z # 0.

E

If A is positive or negative definite, then it follows

that A is nonsingular and A-l is, respectively, positive or negative definite. A diagonal m

x

m matrix with elements zl,

denoted by diag(zi), i = 1,

..., zm

..., m, and defined to be

is

301

Appendix I

The i n v e r s e [ d i a g ( z i ) ] - l e x i s t s i f a n d o n l y i f zf'

..., m, d i a g ( zf') .

a n d it i s e a s i l y s e e n t h a t [ d i a g ( z i ) 1 - l =

i = 1,

t h e norm of t h e m a t r i x A i s d e f i n e d a s IlAll =

If x e En,

I

supx{IIAxII

exists for

llxll = 1 ) .

This definition extends t o general

normed s p a c e s . 5.

PARAMETRIC NLP The r e s u l t s i n t h e book a r e m a i n l y c o n c e r n e d w i t h r e a l i z a -

t i o n s or g e n e r a l i z a t i o n s of a p a r a m e t r i c n o n l i n e a r p r o g r a m i n g (NLP) problem o f t h e f o r m minx f ( x , where x e En, h: En

x

E E

* Ep.

Ek

R ( E ) = {x

s.t.

E)

E

E

g(x,

k , f : En

E

x

E)

h(x,

2 0,

k * E 1, g : En

x

E)

P3(€)

= 0

Ek -+ Em, and

The f e a s i b l e r e g i o n o f P 3 ( € ) i s d e n o t e d by

Enlg(x,

0, h(x,

E)

E)

= 0 ) and d e f i n e s a p o i n t -

to-set map ( o r mapping) R from Ek t o s u b s e t s o f En.

The

o p t i m a l v a l u e o f P J ( ~ )i s d e f i n e d a s

f

*

(E)

=

infx{f (x,

E)

Ix

R(E)1

E

{+-

# Jl

if

R(E)

if

R(E) =

Jl

*

w i t h e f f e c t v e domain o f € a n d domain o f R d e f i n e d t o be k D = { E E E R ( E ) # Jl} a n d t h e s e t o f ( g l o b a l ) s o l u t i o n s by S(E) = {x

E

R ( ~ ) l f ( x E, ) = f * ( E ) } ,

a l s o defining a point-to-set A "local s o l u t i o n "

by

X(E)

map S f r o m Ek t o s u b s e t s of En.

o r " l o c a l minimum" o f P 3 ( € ) w i l l b e d e n o t e d

a n d many r e s u l t s w i l l a l s o c o n c e r n t h e l o c a l o p t i m a l

v a l u e f u n c t i o n f * ( ~ )= f [ x ( E ) ,

€1.

The ( u s u a l ) L a g r a n g i a n

a s s o c i a t e d w i t h problem P 3 ( € ) i s d e f i n e d t o b e L(x,

U, W, E )

= f(x,

E)

-

T

u g(x,

E)

T + w h ( x , €1,

302

Appendix I

where u

Em and w

E

Ep are called Lagrange multipliers and

E

the superscript T denotes transpose, so that we have L: En Em

x

Ep

Ek

x

-+

El.

A local solution

X(E)

x

along with so-called

optimal multipliers (that satisfy the well-known Karush-KuhnTucker first-order necessary conditions) define what will be called a Karush-Kuhn-Tucker triple, y ( E ) = [x(E), U(E), w ( € 1 I T

.

Important realizations of P3(€) are: (i) f and the components of g and h are affine functions of x, (ii) f is a quadratic function of x [i.e., f(x, - ) = xTAx + bTx and the matrix A and vector b do not depend on XI, and the components of g and h are affine functions of x; (iii) f is convex in x, the components of g are concave functions of x, and the components of h are affine functions of x.

In the first instance

P3 is called a linear program, in the second instance a quadratic program, and in the third a convex program. Important specializations of problem P 3 ( ~ )will be considered, e.g.,

the so-called right-hand-side (rhs) perturbation

problem minx f(x) where

E~

E

' E g(x) 2 ,

s.t.

Em, c2

E

Epr

E = (E

h(x) =

E

2

,

1, E ~ ) and ~ , the rhs inequality-

constrained perturbation problem minx f(x) where

E

E

Em.

P1(E)

Also, we shall give many results for generaliza-

tions of P 3 ( ~,) e.g. minx f(x,

g(x) 2 E,

s.t.

E)

,

for

s.t.

x

E

R(E)

where R(E) is not necessarily defined in terms of functional inequalities or equalities, results for more general spaces, and results encompassing the possibility of an abstract set

303

Appendix 1

inclusion x tioned.

E

M, i n a d d i t i o n t o t h e v a r i o u s c o n s t r a i n t s men-

However, t h e g e n e r a l framework w i l l be a s d e s c r i b e d

f o r t h e p r o t o t y p e model P 3 ( ~ ) . Our main c o n c e r n s w i l l be t h e c o n t i n u i t y p r o p e r t i e s o f R and S and c o n t i n u i t y and d i f f e r e n t i a b i l i t y p r o p e r t i e s o f f

*

and y , a l o n g w i t h bounds on t h e s e q u a n t i t i e s , r e l a t i v e t o changes i n t h e problem p a r a m e t e r

E.

W e s h a l l a l s o be i n t e r e s t e d

i n c o n v e x i t y p r o p e r t i e s o f R, S , and f*. W e u s e t h e term bounds r a t h e r l o o s e l y t o d e s c r i b e any s e t

c o n t a i n i n g any numbers or p o i n t s a s s o c i a t e d w i t h s o l u t i o n i n formation of i n t e r e s t , e.g.,

upper and lower bounds on t h e

o p t i m a l v a l u e f u n c t i o n on a g i v e n s e t o f p a r a m e t e r s , a n upper bound on t h e d i s t a n c e between s o l u t i o n p o i n t s f o r d i f f e r e n t parameter values, etc. t h e problem p a r a m e t e r s .

are.

Bounds may o r may n o t b e a f u n c t i o n of " P a r a m e t r i c bounds" are t h o s e t h a t

" E r r o r bounds" a r e bounds on t h e d i s c r e p a n c y between a

c a l c u l a t e d s o l u t i o n v a l u e and a n a c t u a l s o l u t i o n v a l u e . F i n a l l y , " s e n s i t i v i t y " i n f o r m a t i o n h e r e g e n e r a l l y refers t o any measure a s s o c i a t e d w i t h a s o l u t i o n change c o r r e s p o n d i n g t o an a r b i t r a r i l y s m a l l change i n t h e problem p a r a m e t e r s , w h i l e " s t a b i l i t y " i n f o r m a t i o n i s any such measure c o r r e s p o n d i n g t o a f i n i t e change.

W e s h a l l o f t e n mean o p t i m a l v a l u e o r Karush-

Kuhn-Tucker t r i p l e p a r a m e t e r d e r i v a t i v e s by s e n s i t i v i t y i n f o r m a t i o n , when t h e s e q u a n t i t i e s e x i s t , and w i l l o f t e n be r e f e r r i n g t o o p t i m a l v a l u e o r KKT-triple bounds, r e l a t i v e t o f i n i t e p a r a m e t e r changes, i n t h e c o n t e x t o f s t a b i l i t y information.

304 6.

Appendix I

SYMBOLS For c o n v e n i e n c e w e l i s t t h e p r i n c i p a l symbols a n d d e f i n i -

t i o n s t h a t a r e a d o p t e d , f o l l o w e d by t h e number o f t h e p a g e on which t h e symbol o r c o n c e p t i s f i r s t d e f i n e d o r d e s c r i b e d .

In

cases where t h e symbol i s n o t d i s c u s s e d i n t h i s a p p e n d i x , t h e t e x t p a g e o n which it f i r s t a p p e a r s i s g i v e n .

I n t h o s e few

i n s t a n c e s where a p a g e number i s n o t g i v e n , t h e g i v e n c o n c e p t

is n o t u s e d i n t h i s book b u t i s w e l l known a n d i s i n c l u d e d f o r completeness. S e c t i o n s 2-5,

The symbols are g r o u p e d u n d e r t h e t i t l e s of where t h e r e s p e c t i v e n o t i o n or r e l a t e d c o n s t r u c t s

are d e s c r i b e d . SETS, VECTORS, AND SPACES

PI X E

t h e empty set, 2 9 0 M

x i s a n e l e m e n t of M,

290

x is n o t a n e l e m e n t of M,

290

M

t h e c l o s u r e o f t h e s e t M,

291

MO

t h e i n t e r i o r of t h e s e t M,

M U N

t h e u n i o n of t h e s e t s M a n d N ,

M n N

t h e i n t e r s e c t i o n of M a n d N , 290

x i

-

"ieIMi

nieI'i M C N

291 290

t h e u n i o n a n d i n t e r s e c t i o n of t h e sets Mi, r e s p e c t i v e l y , where i i s a n e l e m e n t of t h e set I,

293

M i s a subset of N, i . e . ,

i s a l s o i n N , 290

e a c h e l e m e n t of M

Appendix I

305 t h e e l e m e n t s o f M a r e t h e same a s t h o s e of N ,

M = N

290

t h e e l e m e n t s o f M a r e n o t t h e same a s t h o s e o f

N, 2 9 0 t h e s e t of x f o r which t h e s t a t e m e n t f o l l o w i n g t h e v e r t i c a l bar is t r u e , 290

M x N

t h e C a r t e s i a n p r o d u c t o f t h e s e t s M and N ,

[a, b l

= {X E E la

(a, b)

= {x

290

1

x 5 b}, t h e c l o s e d r e a l i n t e r v a l from a t o b , 1 6 E

E 1( a < x < b}, t h e open real i n t e r v a l

from a t o b , 1 7 ( a , bl

= ( x e E1 ( a < x 5 b ) ,

38

[a, b ) En

n-dimensional E u c l i d e a n s p a c e , 2 9 1

T

a s a s u p e r s c r i p t , denotes t r a n s p o s i t i o n , 291

X

= (xl,

...

= xlyl

+

, x n l T i f x e E", vector, 291

T

X Y

* a *

o f x,' y

-

x y t h e scalar or i n n e r p r o d u c t n n'

En,

291

norm of t h e v e c t o r x; i f x e En, t h e E u c l i d e a n norm i s g i v e n by )I x1I = (xTx)1'2, 291

llxll

IIX

E

+

an n-component

YII

d i s t a n c e between x En,

291

E

En and y e En,

a metric on

306

Appendix I

= {x

BA(xo)

I

IIx

-

xoII < A ) ,

an open b a l l o f r a d i u s

X > 0 a t xo, 2 9 1

. ..,

c1X

= (axl,

aM

= {xlx = a z ,

X + Y

=

where a z

E

MI, where a

..., xn

( x l + yl, y E En, 2 9 1

+

E1 and x

E

E

E

1

,

E

En, 2 9 1

291

y n I T , where x

En and

E

M + N

a sequence o f v e c t o r s xk

lxk

2,

{x, j

I

..., 2 9 2

E

E

n

, where

k = 1,

a subsequence o f { x 1 , where j = 1, 2 , k

l i m xk = 51

k-+m

t h e l i m i t of t h e sequence {x,}

l i m IIxk

k-

l i r n xk

- -X I [

=

..., 2 9 2

i s Z, i - e . ,

0 , a l s o w r i t t e n xk

+

I, 2 9 2

t h e l i m i t i n f e r i o r of t h e sequence {x,) o f r e a l numbers, a l s o w r i t t e n l i m i n f x k , 293

G

k+m

-

l i m xk

t h e l i m i t s u p e r i o r of t h e sequence { x k l o f r e a l

k-

numbers, a l s o w r i t t e n l i m s u p xk, 293 k-

l i m @(XI

t h e l i m i t o f t h e r e a l - v a l u e d v e c t o r f u n c t i o n $I a s x c o n v e r g e s t o xo i n any manner, ( i . e . , f o r

X'X0

e a c h sequence {xn} such t h a t xn

+

xo, t h e

g i v e n l i m i t i n g q u a n t i t y i s d e f i n e d and u n i q u e ) , 293

l i m inf x'xO

@

(x)

t h e l i m i t i n f e r i o r of t h e real-valued function @ ( x )a s x c o n v e r g e s t o xo i n any manner, 1 4

l i m s u p @ ( x ) t h e l i m i t s u p e r i o r o f @ ( x )a s x c o n v e r g e s t o x+xo x0, 1 4

Appendix I

l i m Rn n+-

301 t h e i n n e r l i m i t of a sequence o f sets (R,},

15

t h e o u t e r l i m i t o f a sequence o f s e t s {Rn), 1 5

l i m Rn n-

t h e l i m i t of a sequence o f sets {R,},

{ z i , i e 1)

t h e s e t of e l e m e n t s zi s u c h t h a t t h e i n d e x i

w r i t t e n Rn

-+

also

R, 1 6

b e l o n g s t o some s e t I, 2 4

1

i eI

5

zi

AiXi

i=l

t h e sum o f t h e e l e m e n t s zi s u c h t h a t i e I , 293

=

A x + 11 xl, 293

+ Amxm, a l i n e a r combination o f

..., xm

e En,

where A l l

..., Am

e E

1

,

FUNCTIONS A N D MAPPINGS

$ 2 0

o r d e r i n g r e l a t i o n s , meaning r e s p e c t i v e l y t h a t

$'O $ = O

e a c h component of t h e r e a l - v a l u e d v e c t o r f u n c t i o n $ i s nonnegative, p o s i t i v e , o r zero,

$ # O

o r t h a t some component i s n o t z e r o , 2 9 4

r : ~ + x

g e n e r a l symbol used f o r a p o i n t - t o - s e t

map, a

correspondence t h a t a s s o c i a t e s with every e l e m e n t t of T a s u b s e t o f X I 2 9 4 v a l u e (image) of t h e map = {t e Tlr(t)

(Note:

r

at t

E

T, 2 9 4

# 01, c a l l e d t h e domain o f T , 1 5

T h i s i s a l s o used f o r t h e domain of

p a r t i c u l a r maps, e.g., point-to-set

map R, 301

t h e f e a s i b l e region

308

Appendix I

-

G

"t€T"xer graph of

t ) ( t , x ) = { ( t , x ) E T x Xlx E r ( t ) ) , 1 2 (also used f o r t h e graph of

r

o t h e r maps =

r-l

(ti

e.g.,

t h e map R,

U t E A r ( t ) , t h e image o f r on

250 A C D,

called t h e

range of

r

i f A = D, t h e domain o f

r

= {x E x l x

E

r ( t )1 ,

at t

r

t h e inverse of

E

T,

39

a@ ( x ) / a x i

t h e p a r t i a l d e r i v a t i v e o f t h e r e a l - v a l u e d funct i o n @ w i t h r e s p e c t t o xi a t t h e p o i n t X, 2 9 6

a L @ (Z) /axiax

j

t h e second p a r t i a l d e r i v a t i v e o f t h e r e a l - v a l u e d f u n c t i o n @ a t 51 w i t h r e s p e c t t o xi, =

[af (X)/axl,

..., a f ( Z ) / a x n i , t h e

(TI) T ,

..., V@m(51)T]T,t h e

m

j r

297

g r a d i e n t of

f u n c t i o n f a t T? E En,

t h e real-valued = [Val

x

296

n Jacobian at m a t r i x o f t h e m-component vector f u n c t i o n E

En,

V @ is

297

a(@l,

.

...

(Note: r

x

+

A c l a s s i c a l symbol f o r

@m)/a(xlr

...

r

x n ) r 36)

t h e Hessian of t h e real-v alu ed f u n c t i o n @ a t

x

E

En,

an n

x

n m a t r i x whose i j - t h e l e m e n t i s

t h e s e t of f u n c t i o n s t h a t a r e d e f i n e d a n d cont i n u o u s on a n o p e n s u b s e t ( o f t h e g i v e n s p a c e ) t h a t c o n t a i n s t h e s e t M,

295

t h e s et o f f u n c t i o n s t h a t are k t i m e s c o n t i n u o u s l y d i f f e r e n t i a b l e on a n open s u b s e t t h a t c o n t a i n s M,

295

309

Appendix I

infxeM$(x)

t h e infimum o f t h e r e a l - v a l u e d s e t M, 295

minx,,$(x)

t h e minimum ( v a l u e ) o f t h e r e a l - v a l u e d $ on t h e

f u n c t i o n $ on t h e

function

s e t M. 295

o(llhll)

s m a l l o of l l h l l , r e p r e s e n t i n g a r e a l - v a l u e d f u n c t i o n $ ( h ) s u c h t h a t $ ( h ) / l l h l I * 0 as h + 0, 78

lcl

a b s o l u t e v a l u e of t h e number c, 2 1

0 * o+

t h e l i m i t a s 0 c o n v e r g e s t o 0 a l o n g any s e q u e n c e

of p o s i t i v e r e a l numbers, 25 = lim[$(t

B+O+ .~

+ 02)

-

$ ( t ) ] / B , t h e (one-sided)

d i r e c t i o n a l d e r i v a t i v e of t h e r e a l - v a l u e d

func-

t i o n $ a t t h e p o i n t t i n t h e d i r e c t i o n z , 25 - $ 1 ( x ) I $ 2 ( Z ) = $ 1[ $ 2 (z)], it b e i n g u n d e r s t o o d

t h a t a l l functions preceding the vertical bar i n t h e g i v e n e x p r e s s i o n are e v a l u a t e d a t t h e s p e c i f i e d argument u n l e s s o t h e r w i s e s t a t e d , 4 4

M A TRICES

A

,

an m

p m a t r i x , 298

x

AT

t h e t r a n s p o s e of t h e m m a t r i x , 299 I

the i d e n t i t y matrix, 299

x

p m a t r i x A, a p

x

m

310

Appendix I

=I

AB

i

r

the product of the m x p matrix amlT and B = [blr . . . I m, where air b. E Ep, i = 1, 3 j = 1, q, 3 0 0

A = [a,,

...,

...,

...,

bqlr

A + B

the sum of the m x p matrices A and B , whose ij-th elements are, respectively, hij and bij, 300

i

the product of the matrix A by a E E1 , a matrix whose ij-th element is aaij, where the ij-th element of A is aij, 300

aA

diag ( zi)

a diagonal m x m matrix whose i-th diagonal m, 300 element is zi, i = 1,

...,

C = [AIB]

c

A-

=

121

the matrix C is equal to the partitioned matrix given, where the vertical bar denotes the partition and A and B are matrices, also written , [ A B ] , 7 7 same as above, where the horizontal bar denotes the partition, also written without the horizontal bar, 77 inverse matrix, 2 9 9

31 I

Appendix I

A

-T

t h e transpose of t h e inverse of t h e matrix A,

i.e.,

= supx{IIAxII Illxll =

IlAll

A,

= (AT)-’,

-2

-1 -1 A

= A

= AB,

A * B

299

1 1 , t h e norm of t h e m a t r i x

301

= AA, where A is m x m,

A2

A

= (A-lIT

A-T

-

(A2)-l,

175

where A i s m

X

m, 175

t h e p r o d u c t of t h e m a t r i c e s A and B, t h i s

symbol b e i n g u s e d t o d i s t i n g u i s h [A B] = [ A I B ] from [AB] = [A

a m b i g u i t y, 1 29

B ] where t h e r e may be

PARAMETRIC NLP E

=

(El,

. . . I

a k-component parameter, f : En

X

Ek

-+

column v e c t o r i n E

k , t h e problem

301

E1

t h e problem o b j e c t i v e f u n c t i o n , 301 g: En

X

Ek

-f

Em

t h e problem i n e q u a l i t y - c o n s t r a i n t f u n c t i o n g = (g,, h: E n x Ek

-+

..., g,IT.

vector

301

Ep

t h e problem e q u a l i t y - c o n s t r a i n t h = (hl, P3(€): minx f ( x ,

..., hp) T ,

vector function,

301

s.t. g(x, E) 0 h(x, E) = 0 t h e g e n e r a l p a r a m e t r i c NLP problem: f o r a given v a l u e o f E , f i n d an x t h a t minimizes f ( x , E ) s u b j e c t t o g i ( x , E ) 1. 0 ( i = 1, m ) and h . ( x , E) = 0 ( j = 1, p ) , 301 E)

3

...,

...,

312

Appendix I

1 2 s.t. g(x) E , h(x) = E an important special case of P 3 ( € ) called a right-hand-side (rhs) (perturbation) problem, where E~ e Em, E~ E Ep, and E = ( E ~ ,E ~ ) ~302 ,

P~(E):minx f(x)

P1(~):minX f(x) s.t. g(x) L E an important special case of P 3 ( ~ ) ,302 E En(g(x, E ) 2 0 , h(x, E ) = 01, 301 (also used for a general constraint set)

R(E)

= Ex

R

point-to-set map whose value is R ( E ) , 301

f* ( € 1

f*

optimal value function, whose value is given by * ( N o t e : The same f ( E ) as defined, 301. symbol will be used for a l o c a l optimal value function in many results. The meaning will be clear from the context)

S

point-to-set map whose value is

S(E),

301

X(E) = [X1(E), . . . I Xn(E)l T a local solution of P3(~),302

the local optimal value function, when exists, 301 L(x, u, w, € 1 = f(x, € 1 - u g(x, E ) + wTh(x, Lagrangian of P 3 ( ~ ) ,where u

X(E)

the usual Em, w E Ep,

E), E

301

an optimal Lagrange (inequality constraint) multiplier vector associated with ~ ( € 1 , 3 0 2

313

Appendix I

a n o p t i m a l Lagrange ( e q u a l i t y c o n s t r a i n t ) m u l t i p l i e r a s s o c i a t e d with x ( E ) , 302

W(E)

= [ x ( E ), U ( E )

, W ( E ) IT,

a Karush-Kuhn-Tucker

t r i p l e , 302 =

L [ Y ( E ) , €1, t h e o p t i m a l v a l u e L a g r a n g i a n ,

t h e d i r e c t i o n a l d e r i v a t i v e of f z a t the point

D:f*

(E)

E,

i n the direction

26

t h e d i r e c t i o n a l second d e r i v a t i v e of f direction z a t the point

V.

*

82

E,

*

i n the

44

t h e g r a d i e n t operator, subscripted i f necessary f o r c l a r i t y , to denote t h e v a r i a b l e s of t h e partial differentiation.

The g r a d i e n t o f a

s c a l a r f u n c t i o n i s d e f i n e d h e r e t o be a row vector.

I f n o t s u b s c r i p t e d , it i s assumed

t h a t d i f f e r e n t i a t i o n is with respect t o t h e f i r s t argument, 296

t h e g r a d i e n t of t h e optimal value function, 4 4

Appendix I

2

VX~(x,U r W,

E)

Vx[VxL(x, u, w, €IT], the Hessian (matrix) of L(x, u, w, E) with respect to x, an n x n 2 matrix whose ij-th element is a L(x, u, w, E)/ ax.axi = a/ax.[aL(x, U, W, ~ ) / a x ~ Ifor , i, 7 3 j = 1, n. If not subscripted, it is assumed that differentiation is with respect to the first argument, 43

=

...,

= VE[VEf* (E) T1,

VZf* € 1

respect to VL (x u, w, (i = 1,

E)

= 0

..., m;

E,

uigi(xr

j = 1,

...

r

the k

x

k Hessian of f(x, E) with

52 E)

P)

= 0

h . (x, 3

E)

=

0

the Karush-Kuhn-Tucker conditions associated with a local solution x of problem P3(€), 7 4

M

the Jacobian matrix of the Karush-Kuhn-Tucker (first-order) optimality conditions of P3(~), with respect to (x, u, w), 77

315

Appendix I

=

[:

a L./ a x1

1

JI

aL/axn) :

(X,U,W r

T

2

VxEL(xr ur wr (Note:

=

[:;lL

E)

E)

For simplicity, we write

v VEL ixL]r

102

N

-V hT]',

the negative of the Jacobian matrix E P of the Karush-Kuhn-Tucker equations for P 3 ( ~ ) , with respect to E, 77 M(E)V~Y(E)= N ( E ) basic first-order sensitivity equation, 77

-

g(Xr E)

= [gl(xl

...,

E), gr(x, E)lT, the vector of binding inequality constraints, 94

316

Appendix I

P

A t h e r e d u c e d form o f t h e J a c o b i a n o f t h e Karush-Kuhn-Tucker

equations with respect t o

( x , u, w ) , w i t h nonbinding c o n s t r a i n t informat i o n d e l e t e d , and r e a r r a n g e d f o r symmetry, 9 6

A- 1 t h e b l o c k component r e p r e s e n t a t i o n o f t h e inverse of

i, 9 6

317

Appendix I

2 T = I-VExL I

fi

T

VETTI -VEhT] ,

t h e r e d u c e d form o f

t h e n e g a t i v e o f t h e J a c o b i a n o f t h e KarushKuhn-Tucker e q u a t i o n s , w i t h r e s p e c t t o

=

100

A ( E ) - ~ ( E ), g e n e r a l e x p r e s s i o n f o r t h e f i r s t p a r t i a l d e r i v a t i v e s o f a Karush-Kuhn-Tucker t r i p l e , r e d u c e d t o s u p p r e s s n o n b i n d i n g cons t r a i n t information, 100

21

EX

-VEh

f o r m u l a f o r t h e d e r i v a t i v e s o f a r e d u c e d KKT t r i p l e of problem P 3 ( ~ )i n terms of t h e b l o c k components of v:f

*

=

iTVEy+

i-',

2 AT"VEL = -N M

100

'i +

2 VEL, formula f o r t h e

o p t i m a l v a l u e H e s s i a n i n terms o f

i-', 1 0 1

formula f o r t h e o p t i m a l v a l u e Hessian as a f u n c t i o n of V

2

(XI € 1 L,

102

318

Appendix I

formula for the derivatives of a (reduced) KKT triple of problem P 2 ( ~ , ) 129

(reduced) Hessian of the optimal value of P2 ( € 1 r 130 TV 2LVEx, (reduced) Hessian of the optimal X 2 value of P2(€) in terms of VxL, 134

= VEx

VL x, u, w, (i = 1,

M(E,

E)

=

0

u.g. (x, 1 1

E)

=

r

hj(xr

E)

= w.r

I j = 1, p) the necessary conditions associated with a local minimizing point of the barrier-penalty function W(x, E , r), and a perturbation of the Karush-Kuhn-Tucker conditions associated with a local solution of problem P3(€), 158

..., m;

...,

r)VEy(c, r) = N ( E r r) first-order sensitivity equation derived from the perturbed Karush-Kuhn-Tucker equations and associated with W(x, E , r), 161

i=l

j=1

the logarithmic-quadratic mixed barrierpenalty function for P3(€), 50

319

Appendix I

W

*

a l o c a l minimizing p o i n t o f W(x,

r)

X(Er

(E,

r)

E,

r ) , 50

r ) , E, r ] , t h e p e n a l t y f u n c t i o n o p t i m a l v a l u e , 179

= W[X(E,

VEX(€, r)

=

r),

-v~w[?E(E,

E,

r1 -1v E2 X w [ y ( E , r ) ,

E r

r1,

a formula f o r t h e f i r s t d e r i v a t i v e s o f a l o c a l minimum o f W(x,

E,

r ) , 164

%I

SM

the point-to-set

sM

*

t h e mapping SM when SM i s a s i n g l e t o n , 1 6

sM

S6

map d e f i n e d by S M ( g ) , 1 6

E R ( E ) I f ( x ) 5 f * ( E ) + 6 , 6 1. 01, t h e s e t o f 6-optimal s o l u t i o n s o f problem P 1 ( € ) , 20

(€1

= {x

t h e s e t o f o p t i m a l Lagrange m u l t i p l i e r s a s s o c i -

K(x, 0)

a t e d with a solution point x, 27

s-t.

P ( a ) : min ( x , E ) f ( x r € 1 h(x,

E)

= 0,

E

g(x,

E)

L 0,

= a

a r h s p e r t u r b a t i o n problem e q u i v a l e n t t o t h e g e n e r a l p a r a m e t r i c problem P 3 ( ~ ) ,6 0

EXAMPLES OF THE CHAIN RULE FOR D I F F E R E N T I A T I O N

Assuming t h e i n v o l v e d q u a n t i t i e s and d e r i v a t i v e s e x i s t :

320

Appendix I

A B BRE V I A T I 0 NS

GP

g e o m e t r i c programming, 37

KKT

Karush-Kuhn-Tucker,

lsc

lower s e m i c o n t i n u o u s ( u s e d f o r r e a l - v a l u e d f u n c t i o n s only), 1 4

NLP

n o n l i n e a r programming, 3 0 1

rhs

right-hand s i d e , 302

s.t.

s u b j e c t t o , 295

USC

upper semicontinuous (used f o r r e a l - v a l u e d f u n c t i o n s only), 12

303

Appendix II

Lemmas, Theorems, Corollaries, Definitions, and Examples

Example 2 . 1 . 1

[ f * ( E ) and ~ ( € 1 1 , 9

Example 2 . 1 . 2

[ f * ( E ) a n d x ( E ) ] , 10

Definition 2.2.1

[lower a n d u p p e r s e m i c o n t i n u o u s p o i n t - t o - s e t map], 1 2

D e f i n i t i o n 2.2.2

[open a n d closed p o i n t - t o - s e t

Theorem 2 . 2 . 1

[ f * u s c (1sc) f o r ~ ( € 1 : min f ( x , € 1 s . t .

Theorem 2.2.2

x

E

R(E).

[f*

E

C and S ( E ) u p p e r s e m i c o n t i n u o u s f o r

P(E).

Theorem 2.2.3

B e r g e (19631.1,

E

R(E)

(1967). I ,

.

E

{x

E

(1967) . I ,

D a n t z i g , Folkman, and S h a p i r o 16

M l g ( x ) 2 0).

Dantzig e t aZ.

16

[Solution set continuous f o r P ( g ) .

e t aZ. Theorem 2 . 2 . 6

14

[ S o l u t i o n s e t c l o s e d f o r P ( g ) : min f ( x ) s . t . x

Theorem 2.2.5

Berge (19631.1, 14

[ x ( E ) i s c o n t i n u o u s f o r P ( E ) : min f ( x ) s . t . x

Theorem 2 . 2 . 4

map], 1 2

[S

Dantzig

(1967).1, 1 7

i s c o n t i n u o u s and convex v a l u e d f o r P ( E ) :

min f ( x ,

E)

s.t. x

E

R(E) r l X,

E

E

T.

Robinson a n d Day (19741.1, 1 8 Theorem 2.2.7

[ E q u i v a l e n t c o n d i t i o n s f o r u p p e r a n d lower s e m i c o n t i n u i t y of R ( E ) , f o r P 1 ( € ) . Evans a n d Gould ( 1 9 7 0 ) . ] , 18

32 I

322

Appendix I1

*

Theorem 2.2.8

[Conditions for f lsc or USC, for P1(&). Evans and Gould (1970).I , 19

Theorem 2.2.9

[Conditions for f , R ( E ) continuous, for P1(&). Evans and Gould (1970). I , 19

Theorem 2.2.10

[f continuous for P(E): min f(x) s.t. g(x) E , x E X. Hogan (1973c).], 20

Theorem 2.2.11

[f continuous for P(E): min f(x, E ) s.t. g(x, E ) 1. 0, x E X. Hogan (1973c).l, 20

Definition 2.2.3

[Lipschitz continuous function], 21

Definition 2.2.4

[Uniformly linearly continuous point-to-set

*

*

*.

map1

21

Theorem 2.2.12

[f”( E ) Lipschitz for P1(~). Stern and Topkis (1976).], 21

Theorem 2.2.13

[Continuity of &-optimal solutions of P1(&). Stern and Topkis (1976). I , 22

Definition 2.3.1

[Pseudoconvex function. I , 24

CQ1

[Mangasarian-Fromovitz Constraint Qualification], 23

CQ2

[Slater Constraint Qualification], 24

CQ3

[Linear Independence], 24

CQ4

[Cottle Constraint Qualification], 24

CQ 5

[Kuhn-Tucker Constraint Qualification] , 24

Definition 2.3.2

[Directional derivative], 25

Theorem 2.3.1

[Directional derivative DZf of f ( E ) for P(E): min f(x, E ) s.t. x E R. Danskin (1966, 19671.1, 25

Theorem 2.3.2

[Existence of DZf*(E) for P(E): min f(Xr E) s.t. g(x, E ) 0 , x E X, P(E) convex in x. Hogan (197323).I , 26

Theorem 2.3.3

[f (E) E C with CQ1 for P2(~). Gauvin and Tolle (1977).], 28

*

*

*

323

Appendix 11

Theorem 2.3.4

[Optimal v a l u e D i n i d e r i v a t i v e lower bound for P3(€).

Gauvin and Dubeau (1979) and

F i a c c o ( 1 9 8 0 ~ )I ., 2 9 C o r o l l a r y 2.3.5

[Weaker optimal v a l u e D i n i d e r i v a t i v e l o w e r Gauvin and Dubeau ( 1 9 7 9 )

bound f o r P 3 ( € ) .

a n d F i a c c o ( 1 9 8 0 ~. )I , Theorem 2 . 3 . 6

[ O p t i m a l v a l u e Din1 d e r i v a t i v e u p p e r bound for P3(€).

Gauvin a n d Dubeau (1979) and

F i a c c o ( 1 9 8 0 ~. )I C o r o l l a r y 2.3.7

29

,

30

[DZf* f o r P 3 ( € ) w i t h CQ3.

Gauvin and Dubeau

(1979) a n d F i a c c o ( 1 9 8 0 ~. )I ,

C o r o l l a r y 2.3.8

*

[DZf

f o r P 3 ( € ) convex i n x.

30 F i a c c o and

H u t z l e r (1979a) and F i a c c o (1980b) . I , Theorem 2.3.9

[ C o n d i t i o n s s u c h t h a t Pi(€) i s c a l m .

31

Clarke

(19761.1, 35 Theorem 2 . 4 . 1

[ I m p l i c i t F u n c t i o n Theorem f o r Ck F u n c t i o n . Bochner a n d M a r t i n ( 1 9 4 8 ) . I ,

Theorem 2 . 4 . 2

[ I m p l i c i t F u n c t i o n Theorem f o r A n a l y t i c Function.

Theorem 2.4.3

Bochner a n d M a r t i n ( 1 9 4 8 ) . I

,

36

[ S o l u t i o n bound f o r a p o s i t i v e d e f i n i t e q u a d r a t i c program.

Theorem 2 . 4 . 4

36

Daniel (1973). I

[ B a s i c S e n s i t i v i t y Theorem:

, 41

Sufficient

c o n d i t i o n s f o r a l o c a l l y u n i q u e C1 KKT t r i p l e of P 3 ( € ) . Theorem 2.4.5

Fiacco (1976). I

,

42

[ S u f f i c i e n t c o n d i t i o n s for local uniqueness, c o n t i n u i t y and d i r e c t i o n a l d i f f e r e n t i a b i l i t y o f a KKT t r i p l e and d i f f e r e n t i a b i l -

*

for P 3 ( € ) , without strict complementary s l a c k n e s s . J i t t o r n t r u m i t y of f

(E)

(1978, 1 9 8 1 ) . 1 , 43 Definition 2.4.1

[ C y r t o h e d r o n of class C k (19771.1,

D e f i n i t i o n 2.4.2

.

Spingarn

46

[Strong second-order

conditions for P i ( € ) .

Spingarn (1977). I , 47

324 Theorem 2 . 4 . 6

Appendix 11

[ S u f f i c i e n t c o n d i t i o n s f o r a l o c a l l y unique 1 C K K T t r i p l e of P i ( € ) . S p i n g a r n (1977) . I ,

Theorem 2 . 4 . 7

48

[Local u n i q u e n e s s a n d c o n t i n u i t y o f a KKT

t r i p l e f o r P3 ( E ) . Theorem 2.4.8

Robinson ( 1 9 7 4 ) . 1 ,

[Bounds o n a KKT t r i p l e .

48

Robinson ( 1 9 7 4 ) . ] ,

49

Theorem 2 . 4 . 9

[ L o c a l l y u n i q u e C1-KKT

t r i p l e associated

with a l o c a l l y unique unconstrained l o c a l minimum o f t h e b a r r i e r - p e n a l t y f u n c t i o n W(x, Theorem 2.4.10

E,

Fiacco (1976) . I ,

r).

50

[ L o c a l l y u n i q u e a p p r o x i m a t i o n of a CP KKT t r i p l e for C P + l problem f u n c t i o n s , a s s o c i a t e d w i t h P 3 ( € ) a n d W(x,

E,

r).

Fiacco

(19761.1, 51 Theorem 2.4.11

[Locally unique

c1

KKT t r i p l e of P ( € 1 u s i n g

a n augmented L a g r a n g i a n .

3 Buys a n d Gonin

( 1 9 7 7 ) a n d A r m a c o s t a n d F i a c c o (1977) . I , 52

Theorem 2 . 4 . 1 2

[Formulas f o r t h e o p t i m a l v a l u e f P 3 ( € ) , its gradient V f

*

E

*

(E)

of

2 * a n d H e s s i a n VEf

Armacost a n d F i a c c o ( 1 9 7 5 ) . 1, 52 Theorem 2.4.13

[Lagrange m u l t i p l i e r s e n s i t i v i t y r e s u l t s ,

for P2 ( E ) 53 Lemma 3 . 2 . 1

.

.

A r m a c o s t and F i a c c o (1975) . I ,

[Second-order s u f f i c i e n t c o n d i t i o n s f o r a

s t r i c t l o c a l m i n i m i z i n g p o i n t of p r o b l e m P3 ( 0 ) . F i a c c o a n d McCormick ( i 9 6 8 ) . I , 6 9 Theorem 3.2.2

[ B a s i c S e n s i t i v i t y Theorem, a s l i g h t l y

s t r o n g e r v e r s i o n of Theorem 2 . 4 . 4 .

Easy

e x t e n s i o n of main r e s u l t i n F i a c c o (1976) . I ,

C o r o l l a r y 3.2.3

[Sensitivity analysis for the unconstrained problem:

(1976). I ,

C o r o l l a r y 3.2.4

72-76 minx f ( x ,

E).

[ F i r s t - o r d e r e s t i m a t i o n of near

E =

Based on F i a c c o

76-77 0.

[ x ( E ) , ~ ( € 1 ,w ( E ) 1

F i a c c o (1976) . I ,

78

Appendix I1

325

Corollary 3.2.5

[Existence of higher-order KKT-triple derivatives. Fiacco (1976). 1 , 79

Example 3.3.1

[One variable, calculation of solution derivative] , 80

Example 3.3.2

[Two variables, calculation of solution derivatives], 8 0

Theorem 3.4.1

[First- and second-order changes in the optimal value function of P3(~): Equivalent to Theorem 2.4.12. Armacost and Fiacco (1975).I , 83-85

Corollary 3.4.2

[First- and second-order changes in the optimal value function of P3(~), with constraints independent of E , following easily from Armacost and Fiacco (19751.1, 85

Corollary 3.4.3

[First- and second-order changes in the unconstrained optimal value function. New result, following easily from Armacost and Fiacco (1975).I , 85-86

Corollary 3.4.4

[Optimal value function derivatives for rhs perturbations. Equivalent to Theorem 2.4.13. Armacost and Fiacco (19751.1, 87-88

Corollary 3.4.5

[Optimal value derivatives for perturbations that are linear in the parameters. Armacost and Fiacco (1975).], 89-90 2 *1, 106-112 [Calculation of VEf* and VEf

Example 4.3.1 Example 4.3.2

[Calculation of VET, VEf*, and Vzf* for a parametric LP], 113-114

Theorem 5.4.1

[First-order changes in a KKT triple and first- and second-order changes in the optimal value function for the problem with rhs perturbations. Fiacco (1976) and Armacost and Fiacco (1975).], 126-127

3 26

Appendix 11

Corollary 5.5.1

[Positive (semi) definiteness of the Hessian V,2 f * of the optimal value function. Armacost and Fiacco (1976).I

,

130

Corollary 5.5.2

: L . [Relationship between A22 ( E ) and V Armacost and Fiacco (1976).I , 133

Corollary 5.5.3

[Local convexity of the optimal value func* tion f ( E ) of problem P2(~). Armacost and Fiacco (1976). I , 134

Example 5.6.1

[The Shell Dual: first- and second-order * estimates of f ( E ) ] , 137-141

Example 5.6.2

[Direct solution vs solution by decomposition using sensitivity results], 144-151

Theorem 6.2.1

[Approximation of first-order sensitivity results and determination of estimates from W(x, E, r). A slightly stronger version of Theorem 2.4.9. Based on Fiacco (1976).I , 157-160

Corollary 6.2.2

[Convergence of penalty function solutions and sensitivity estimates. Fiacco (1976). I , 161-162

Corollary 6.2.3

[Approximation of X(E) and VEx(c) by Z(E, r) and V E y ( ~ ,r). Fiacco (1976). I , 164

Example 3.3.1 continued

,

[Penalty function sensitivity estimates], 168

Example 3.3.2 continued

,

[Penalty function sensitivity estimates], 169

Corollary 6.4.1

[Existence and convergence of higher-order derivatives of y ( E , r) , an extension of Theorem 2.4.10. Fiacco (1976).I , 172-173

Theorem 6.6.1

[First- and second-order changes in the * penalty function optimal value W ( E , r). Armacost and Fiacco (1975).I

Corollary 6.6.2

,

180

[Convergence of the penalty function optimal value, gradient, and Hessian to the respective quantities associated with the optimal value function. Armacost and Fiacco (1975). 1 , 181

327

Appendix II

Example 3.3.2, continued

[Penalty function estimates of optimal value

Theorem 7.4.1

[Sensitivity results using an augmented Sharpened Lagrangian for problem P ( E ) version of Theorem 2.4.11. Armacost and Fiacco (1977). J , 211-213

Example 8.3.1

[Illustration of sensitivity calculations by SENSUMT], 229

and its derivatives], 182-185

.


References

ALLEN, R. G. D. (1938). M a t h e m a t i c a l A n a l y s i s f o r E c o n o m i s t s . Macmillan, New York. ALLGOWER, E. , AND GEORG, K. (isso). "Simplicia1 and Continuation Methods for Approximating Fixed Points and Solutions to Systems of Equations." SIAM R e v . 22, 28-85. ARCHETTI, F. , BALLABIO, D. , and VERCELLIS, C. (1980). "CostBenefit Analysis of Insulation of Buildings Via Nonlinear Optimization." In N u m e r i c a l O p t i m i z a t i o n of Dynamic S y s t e m s (L. C. W. Dixon and G. P. Szegs, eds.), NorthHolland, Amsterdam, pp. 363-375. ARMACOST, R. L. (1976a). "Computational Experience with Optimal Value Function and Lagrange Multiplier Sensitivity in NLP." Technical Paper T-335, Institute for Management Science and Engineering, George Washington Univ., Washington, D.C. ARMACOST, R. L. (1976b). "Sensitivity Analysis in Parametric Nonlinear Programming." Ph.D. dissertation, School of Engineering and Applied Science, George Washington Univ., Washington, D.C. ARMACOST, R. L., and FIACCO, A. V. (1974). "Computational Experience in Sensitivity Analysis for Nonlinear Programming," Math. Programming 6 (3), 301-326. ARMACOST, R. L., and FIACCO, A. V. (1975). "Second-Order Parametric Sensitivity Analysis in NLP and Estimates by Penalty Function Methods." Technical Paper T-324, Institute for Management Science and Engineering, George Washington Univ., Washington, D.C. ARMACOST, R. L., and FIACCO, A. V. (1976). "NLP Sensitivity Analysis for RHS Perturbations: A Brief Survey and Second Order Extensions." Technical Paper T-334, Institute for Management Science and Engineering, George Washington Univ., Washington, D.C.

329

330

References

ARMACOST, R. L . , and FIACCO, A. V. ( 1 9 7 7 ) . "Exact Sensitivity A n a l y s i s U s i n g Augmented Lagrangians." T e c h n i c a l Paper T - 3 4 9 , I n s t i t u t e f o r M a n a g e m e n t Science and E n g i n e e r i n g , G e o r g e Washington U n i v . , Washington, D.C. ARMACOST, R. L . , and F I A C C O , A. V. ( 1 9 7 8 ) . "Sensitivity A n a l y s i s f o r P a r a m e t r i c N o n l i n e a r Programming U s i n g P e n a l t y Methods." I n C o m p u t e r s and MathematicaZ Programming, N a t i o n a l B u r e a u of S t a n d a r d s Special P u b l i c a t i o n 5 0 2 , 261-269. R. L . , and MYLANDER, W. C. ( 1 9 7 3 ) . "A G u i d e t o t h e V e r s i o n 4 C o m p u t e r Subroutine for I m p l e m e n t i n g Sensitivity A n a l y s i s i n N o n l i n e a r Programming." Technical Paper T - 2 8 7 , I n s t i t u t e f o r M a n a g e m e n t S c i e n c e and E n g i n e e r i n g , G e o r g e Washington U n i v . , Washington, D.C.

ARMACOST, SUMT

-

"A G e n e r a l ARROW, K. J . , GOULD, F. J., and HOWE, S. M. ( 1 9 7 3 ) . Saddle P o i n t R e s u l t f o r C o n s t r a i n e d O p t i m i z a t i o n . " Math. Programming 5 ( 2 ) , 2 2 5 - 2 3 4 . K. J . , a n d HURWICZ, L. ( 1 9 5 6 ) . " R e d u c t i o n of C o n s t r a i n e d M a x i m a t o Saddle P o i n t P r o b l e m s . " I n Proceedings o f t h e T h i r d B e r k e l e y Symposium o n MathematicaZ S t a t i s t i c s and P r o b a b i l i t y (J. N e y m a n , e d . ) , U n i v . C a l i f o r n i a ,

ARROW,

Berkeley,

Calif.

"Constraint ARROW, K. J . , HURWICZ, L., and UZAWA, H. ( 1 9 6 1 ) . Q u a l i f i c a t i o n s i n Maximization Problems." Naval R e s . L o g i s t i c s Q . 8 ( 2 ) , 175-190. AUSLENDER, A. ( 1 9 7 9 ) . " D i f f e r e n t i a b l e Stability i n Nonconvex and N o n d i f f e r e n t i a b l e P r o g r a m m i n g . " I n MathematicaZ Programming S t u d i e s , V o l . 1 0 ( P . H u a r d , e d . ) , N o r t h - H o l l a n d , A m s t e r d a m , pp. 2 9 - 4 1 . AVRIEL, M., DEMBO, R . , and PASSY, U. ( 1 9 7 5 ) . G e n e r a l i z e d G e o m e t r i c P r o g r a m s . I' I n t . J . Eng. 9 , 1 4 9 - 1 6 8 .

" S o l u t i o n of NurnericaZ Meth.

BALDER, E. J . ( 1 9 7 7 ) . " A n E x t e n s i o n of D u a l i t y - S t a b i l i t y R e l a t i o n s to Nonconvex O p t i m i z a t i o n Problems. SIAM J . ControZ O p t i m i z . 1 5 ( 2 ) , 3 2 9 - 3 4 3 . BANK, B . , GUDDAT, J . , KLATTE, D., KUMMER, B., and TAMMER, K. NichtZineare Parametrishe Optimierung. (1982). S e m i n a r b e r i c h t N r . 31, H u m b o l d t U n i v e r s i t y , B e r l i n , S e k t i o n M a t h e m a t i k , 1981. T o be p u b l i s h e d a s N o n l i n e a r Parametric Optimization, Akademie-Verlag, B e r l i n . BAZARAA, M. S . , GOODE, J. J . , and SHETTY, C. M. ( 1 9 7 2 ) . " C o n s t r a i n t Q u a l i f i c a t i o n s R e v i s i t e d . 'I Management S c i . 2 8 ( l o ) , 567-573. BAZARAA, M. S . , and SHETTY, C. M. ming: Theory and A l g o r i t h m s .

(1979).

N o n l i n e a r ProgramWiley, New Y o r k .

BEIGHTLER, C. S., and P H I L L I P S , D. T . ( 1 9 7 6 ) . m e t r i c Programming. Wiley, New Y o r k .

A p p l i e d Geo-

331

References

BENSON, H. P. ( 1 9 8 0 ) . " A l g o r i t h m s f o r P a r a m e t r i c Nonconvex Programming." D i s c u s s i o n P a p e r #20, C e n t e r f o r Econometric a n d D e c i s i o n S c i e n c e , Univ. F l o r i d a . (1976). "The C o n t i n u i t y o f t h e Optimum i n P a r a metric Programming a n d A p p l i c a t i o n s t o S t o c h a s t i c Programming." J . O p t i m i z . Theory A p p l . 1 8 ( 3 1 , 319-333.

BEREANU, B.

BERGE, C. ( 1 9 6 3 ) . T o p o l o g i c a l Spaces M a c m i l l a n , New York.

( t r a n s . E. M.

Patterson).

.

BERTSEKAS, D . P. ( 1 9 7 5 ) "Combined P r i m a l - D u a l a n d P e n a l t y SIAM J . C o n t r o l Methods f o r C o n s t r a i n e d M i n i m i z a t i o n . 1 3 ( 3 ) , 521-544. BIGELOW, J. H . , a n d SHAPIRO, N . 2. ( 1 9 7 4 ) . "Implicit Function Theorems f o r M a t h e m a t i c a l Programming a n d f o r S y s t e m s o f Inequalities." Math. Programming 6 ( 2 1 , 141-156. BOCHNER, S . , a n d MARTIN, W. ( 1 9 4 8 ) . S e v e r a l Complex V a r i a b l e s . P r i n c e t o n Univ. P r e s s , P r i n c e t o n , N. J.

BOOT, J. C. G. ( 1 9 6 3 ) . "On S e n s i t i v i t y A n a l y s i s i n Convex Q u a d r a t i c Programming P r o b l e m s . " O p e r a t i o n s R e s . I 1 (51, 7 71- 786.

BRADLEY, S. P . ,

HAX,

A.

C.,

a n d MAGNANTI, T. L. ( 1 9 7 7 ) . Addison-Wesley, R e a d i n g ,

A p p l i e d Mathematical Programming.

Mass.

.

BROSOWSKI , B. ( 1 9 8 2 ) P a r a m e t r i c Semi- I n f i n (E),47, 48 Structural design model, 244, 245 Subdifferential operator, 39 Subgradient, 33 SUMT, 53, 191, 233, 286

T Taylor’s expansion first-order, 298 second-order, 298 Transposition of matrices, 298 symbol, T. 291 of vectors, 291

U Unconstrained problem sensitivity, 76. 79 Unconstrained solution point, see Solution point and Local minimum Underestimating program. 56, 57, 59, 60. 254, 267. 273, 279. 283 Uniformly linearly continuous point-to-set map, 21 Upper semicontinuity of a function, 14

V Vector conventions. 291 -295 distance between, 291 inner product, 291 norm, 291 ordering relations, 294 scalar product. 29 I sum. 291 Vector function. 294-298

W Water pollution GP model. 243. 244. 274, 275 Wolfe dual. 138. 255-265


Introduction to Sensitivity and Stability Analysis in Nonlinear Programming (Mathematics in Science and Engineering, 165)

Introduction to sensitivity and stability analysis in nonlinear programming

Mathematics in Engineering and Science

Mathematics in Engineering and Science

Mathematics in engineering and science

An Introduction to Continuum Mechanics (Mathematics in Science and Engineering)

Stability of Motion, (Mathematics in Science and Engineering, Volume 30)

An Algorithmic Approach to Nonlinear Analysis and Optimization (Mathematics in Science and Engineering 63)

Nonlinear System Theory (Mathematics in Science and Engineering, 175)

Approximation of Nonlinear Evolution Systems (Mathematics in Science and Engineering)

Introduction to Nonlinear Analysis

Nonserial dynamic programming (Mathematics in Science and Engineering, Volume 91)

An Introduction to Nonlinear Boundary Value Problems (Mathematics in Science and Engineering, V. 109)

Design Sensitivity Analysis of Structural Systems (Mathematics in Science and Engineering, Volume 177)

Introduction to nonlinear science

Numerical Methods in Sensitivity Analysis and Shape Optimization (Modeling and Simulation in Science, Engineering & Technology)

Dynamic programming in chemical engineering and process control, Volume 12 (Mathematics in Science and Engineering)

Analysis and control of nonlinear infinite dimensional systems, Volume 190 (Mathematics in Science and Engineering)

Variational methods in mathematics, science and engineering

Nonlinear Partial Differential Equations in Engineering, Volume II (Mathematics in Science and Engineering)

Nonlinear systems: analysis, stability, and control

Nonlinear Programming (Classics in Applied Mathematics)

Nonlinear Programming (Classics in Applied Mathematics)

Methods of Nonlinear Analysis, Vol. 1 (Mathematics in Science and Engineering Series, Vol. 61) (v. 1)

Sensitivity analysis in linear regression

An introduction to nonlinear analysis

An introduction to nonlinear analysis

An Introduction to Nonlinear Analysis

Sensitivity Analysis in Linear Regression

Structural sensitivity analysis and optimization 2, Nonlinear systems and applications

Structural Sensitivity Analysis and Optimization 2, Nonlinear Systems and Applications

Introduction to Sensitivity and Stability Analysis in Nonlinear Programming (Mathematics in Science and Engineering, 165)