Optimal Design of Control Systems: Stochastic and Deterministic Problems (Pure and Applied Mathematics: A Series of Monographs and Textbooks 221)

OPTIMAL DESIGN OF CONTROL SYSTEMS

PURE AND APPLIED MATHEMATICS A Program of Monographs, Textbooks, and Lecture Notes

EXECUTIVE EDITORS Earl J. Taft Rutgers University New Brunswick, New Jersey

Zuhair Nashed University of Delaware Newark, Delaware

EDITORIAL BOARD M. S. Baouendi University of California, San Diego Jane Cronin Rutgers University Jack K. Hale Georgia Institute of Technology

Anil Nerode Cornell University Donald Passman University of Wisconsin, Madison Fred S. Roberts Rutgers University

S. Kobayashi University of California, Berkeley

Gian-Carlo Rota Massachusetts Institute of Technology

Marvin Marcus University of California, Santa Barbara

David L. Russell Virginia Polytechnic Institute and State University

W. S. Massey Yale University

Walter Schempp Universitat Siegen

Mark Teply University of Wisconsin, Milwaukee

MONOGRAPHS AND TEXTBOOKS IN PURE AND APPLIED MATHEMATICS 1. K. Yano, Integral Formulas in Riemannian Geometry (1970) 2. S. Kobayashi, Hyperbolic Manifolds and Holomorphic Mappings (1970) 3. V. S. Vladimirov, Equations of Mathematical Physics (A. Jeffrey, ed.; A. Littlewood, trans.) (1970) 4. B. N. Pshenichnyi, Necessary Conditions for an Extremum (L. Neustadt, translation ed.; K. Makowski, trans.) (1971) 5. L. Narici et a/., Functional Analysis and Valuation Theory (1971) 6. S. S. Passman, Infinite Group Rings (1971) 7. L. Domhoff, Group Representation Theory. Part A: Ordinary Representation Theory. Part B: Modular Representation Theory (1971,1972) 8. W. Boothbyand G. L. Weiss, eds., Symmetric Spaces (1972) 9. Y. Matsushima, Differentiate Manifolds (E. T. Kobayashi, trans.) (1972) 10. L. E. Ward, Jr., Topology (1972) 11. A. Babakhanian, Cohomological Methods in Group Theory (1972) 12. R. Gilmer, Multiplicative Ideal Theory (1972) 13. J. Ye/7, Stochastic Processes and the Wiener Integral (1973) 14. J. Barros-Neto, Introduction to the Theory of Distributions (1973) 15. R Larsen, Functional Analysis (1973) 16. K. Yano and S. Ishihara, Tangent and Cotangent Bundles (1973) 17. C. Procesi, Rings with Polynomial Identities (1973) 18. R. Hermann, Geometry, Physics, and Systems (1973) 19. N. R. Wallach, Harmonic Analysis on Homogeneous Spaces (1973) 20. J. Dieudonne, Introduction to the Theory of Formal Groups (1973) 21. /. Vaisman, Cohomology and Differential Forms (1973) 22. B.-Y. Chen, Geometry of Submanifolds (1973) 23. M. Marcus, Finite Dimensional Multilinear Algebra (in two parts) (1973,1975) 24. R. Larsen, Banach Algebras (1973) 25. R. O. Kujala and A. L Vitter, eds., Value Distribution Theory: Part A; Part B: Deficit and Bezout Estimates by Wilhelm Stall (1973) 26. K. B. Stolarsky, Algebraic Numbers and Diophantine Approximation (1974) 27. A. R. Magid, The Separable Galois Theory of Commutative Rings (1974) 28. B. R. McDonald, Finite Rings with Identity (1974) 29. J. Satake, Linear Algebra (S. Koh et al., trans.) (1975) 30. J. S. Go/an, Localization of Noncommutative Rings (1975) 31. G. Klambauer, Mathematical Analysis (1975) 32. M. K. Agoston, Algebraic Topology (1976) 33. K. R. Goodearl, Ring Theory (1976) 34. L. E. Mansfield, Linear Algebra with Geometric Applications (1976) 35. N. J. Pullman, Matrix Theory and Its Applications (1976) 36. B. R. McDonald, Geometric Algebra Over Local Rings (1976) 37. C. W. Groetsch, Generalized Inverses of Linear Operators (1977) 38. J. E. Kuczkowski and J. L. Gersting, Abstract Algebra (1977) 39. C. O. Christenson and W. L. Voxman, Aspects of Topology (1977) 40. M. Nagata, Field Theory (1977) 41. R. L. Long, Algebraic Number Theory (1977) 42. W. F. Pfeffer, Integrals and Measures (1977) 43. R. L Wheeden and A. Zygmund, Measure and Integral (1977) 44. J. H. Curtiss, Introduction to Functions of a Complex Variable (1978) 45. K. Hrbacek and T. Jech, Introduction to Set Theory (1978) 46. W. S. Massey, Homology and Cohomology Theory (1978) 47. M. Marcus, Introduction to Modern Algebra (1978) 48. £ C. Young, Vector and Tensor Analysis (1978) 49. S. B. Nadler, Jr., Hyperspaces of Sets (1978) 50. S. K. Sega/, Topics in Group Kings (1978) 51. A. C. M. van Rooij, Non-Archimedean Functional Analysis (1978) 52. L. Corwin and R. Szczarba, Calculus in Vector Spaces (1979) 53. C. Sadosky, Interpolation of Operators and Singular Integrals (1979) 54. J. Cronin, Differential Equations (1980) 55. C. W. Groetsch, Elements of Applicable Functional Analysis (1980)

56. /. Vaisman, Foundations of Three-Dimensional Euclidean Geometry (1980) 57. H. /, Freedan, Deterministic Mathematical Models in Population Ecology (1980) 58. S. B. Chae, Lebesgue Integration (1980)

59. 60. 61. 62.

C. S. Rees ef a/., Theory and Applications of Fourier Analysis (1981) L Nachbin, Introduction to Functional Analysis (R. M. Aron, trans.) (1981) G. Orzech and M. Orzech, Plane Algebraic Curves (1981) R. Johnsonbaugh and W. E. Pfaffenberger, Foundations of Mathematical Analysis

(1981) 63. W. L. Voxman and R. H. Goetschel, Advanced Calculus (1981) 64. L J. Corwin and R. H. Szczarba, Multivariable Calculus (1982) 65. V. /. Istratescu, Introduction to Linear Operator Theory (1981)

66. R. D. Jarvinen, Finite and Infinite Dimensional Linear Spaces (1981) 67. J. K. Been? and P. E. Ehriich, Global Lorentzian Geometry (1981)

68. 69. 70. 71.

D. L Armacost, The Structure of Locally Compact Abelian Groups (1981) J. W. Brewer and M. K. Smith, eds., Emmy Noether: A Tribute (1981) K. H. Kim, Boolean Matrix Theory and Applications (1982) T. W. Wieting, The Mathematical Theory of Chromatic Plane Ornaments (1982)

72. D. B.Gauld, Differential Topology (1982)

73. R. L. Faber, Foundations of Euclidean and Non-Euclidean Geometry (1983) 74. M. Carmeli, Statistical Theory and Random Matrices (1983) 75. J. H. Carruth et a/., The Theory of Topological Semigroups (1983) 76. R. L. Faber, Differential Geometry and Relativity Theory (1983)

77. S. Bamett, Polynomials and Linear Control Systems (1983) 78. G. Karpilovsky, Commutative Group Algebras (1983) 79. F. Van Oystaeyen and A. Verschoren, Relative Invariants of Rings (1983)

80. /. Vaisman, A First Course in Differential Geometry (1984) 81. G. W. Swan, Applications of Optimal Control Theory in Biomedicine (1984) 82. T. Petrie andJ. D. Randall, Transformation Groups on Manifolds (1984) 83. K. Goebel and S. Reich, Uniform Convexity, Hyperbolic Geometry, and Nonexpansive Mappings (1984) 84. T. Albu and C. Nastasescu, Relative Finiteness in Module Theory (1984) 85. K. Hrbacek and T. Jech, Introduction to Set Theory: Second Edition (1984) 86. F. Van Oystaeyen and A. Verschoren, Relative Invariants of Rings (1984) 87. B. R. McDonald, Linear Algebra Over Commutative Rings (1984)

88. M. Namba, Geometry of Projective Algebraic Curves (1984) 89. G. F. Webb, Theory of Nonlinear Age-Dependent Population Dynamics (1985) 90. M. R. Bremner et ai, Tables of Dominant Weight Multiplicities for Representations of

Simple Lie Algebras (1985) 91. A. E. Fekete, Real Linear Algebra (1985) 92. S. B. Chae, Holomorphy and Calculus in Normed Spaces (1985) 93. 94. 95. 96. 97.

A. J. Jem, Introduction to Integral Equations with Applications (1985) G. Karpilovsky, Projective Representations of Finite Groups (1985) L. Narici and E. Beckenstein, Topological Vector Spaces (1985) J. Weeks, The Shape of Space (1985) P. R. Gribik and K. O. Kortanek, Extremal Methods of Operations Research (1985)

98. J.-A. Chao and W. A. Woyczynsk!, eds., Probability Theory and Harmonic Analysis (1986)

99. 100. 101. 102.

G. D. Crown et a/.. Abstract Algebra (1986) J. H. Carruth et a/., The Theory of Topological Semigroups, Volume 2 (1986) R. S. Dora/7 and V. A. Belfi, Characterizations of C*-Algebras (1986) M. W. Jeter, Mathematical Programming (1986)

103. M. Altman, A Unified Theory of Nonlinear Operator and Evolution Equations with Applications (1986)

104. A. Verschoren, Relative Invariants of Sheaves (1987) 105. R. A. Usmani, Applied Linear Algebra (1987)

106. P. Blass and J. Lang, Zariski Surfaces and Differential Equations in Characteristic p > 0(1987)

107. J. A. Reneke et ai, Structured Hereditary Systems (1987) 108. H. Busemann and B. B. Phadke, Spaces with Distinguished Geodesies (1987) 109. R. Harte, Invertibility and Singularity for Bounded Linear Operators (1988) 110. G. S. Ladde et at., Oscillation Theory of Differential Equations with Deviating Arguments (1987) 111. L. Dudkin et a/., Iterative Aggregation Theory (1987) 112. T. Okubo, Differential Geometry (1987) 113. D. L. Stand and M. L. Stand, Real Analysis with Point-Set Topology (1987)

114. T. C. Card, Introduction to Stochastic Differential Equations (1988) 115. S. S. Abhyankar, Enumerative Combinatorics of Young Tableaux (1988) 116. H. Sfracfe and R. Famsteiner, Modular Lie Algebras and Their Representations (1988) 117. J. A. Huckaba, Commutative Rings with Zero Divisors (1988)

118. W. D. Wallis, Combinatorial Designs (1988) 119. W. W;{?s/aw, Topological Fields (1988) 120. G. Karpilovsky, Field Theory (1988)

121. S. Caenepeel and F. Van Oystaeyen, Brauer Groups and the Cohomology of Graded Rings (1989) 122. W. Kozlowski, Modular Function Spaces (1988) 123. E. Lowen-Colebunders, Function Classes of Cauchy Continuous Maps (1989)

124. M. Pave/, Fundamentals of Pattern Recognition (1989) 125. V. Lakshmikantham et a/., Stability Analysis of Nonlinear Systems (1989) 126. R. Sivaramakrishnan, The Classical Theory of Arithmetic Functions (1989) 127. N. A. Watson, Parabolic Equations on an Infinite Strip (1989)

128. K. J. Hastings, Introduction to the Mathematics of Operations Research (1989) 129. B. Fine, Algebraic Theory of the Bianchi Groups (1989)

130. D. N. Dikranjan et a/., Topological Groups (1989) 131. J.C. Morgan II, Point Set Theory (1990) 132. P. Biter and A. Witkowski, Problems in Mathematical Analysis (1990)

133. H. J. Sussmann, Nonlinear Controllability and Optimal Control (1990) 134. J.-P. Florens et a/., Elements of Bayesian Statistics (1990) 135. N. Shell, Topological Fields and Near Valuations (1990)

136. B. F. Doolin and C. F. Martin, Introduction to Differential Geometry for Engineers 137. 138. 139. 140. 141. 142. 143. 144. 145. 146.

(1990) S. S. Holland, Jr., Applied Analysis by the Hilbert Space Method (1990) J. Okninski, Semigroup Algebras (1990) K. Zhu, Operator Theory in Function Spaces (1990) G. B. Price, An Introduction to Multicomplex Spaces and Functions (1991) R. B. Darst, Introduction to Linear Programming (1991) P. L Sachdev, Nonlinear Ordinary Differential Equations and Their Applications (1991) T. Husain, Orthogonal Schauder Bases (1991) J. Foran, Fundamentals of Real Analysis (1991) W. C. Brown, Matrices and Vector Spaces (1991) M. M. RaoandZ. D. Ron, Theory of Oriicz Spaces (1991)

147. J. S. Go/an and T. Head, Modules and the Structures of Rings (1991) 148. 149. 150. 151. 152.

C. Small, Arithmetic of Finite Fields (1991) K. Yang, Complex Algebraic Geometry (1991) D. G. Hoffman et a/., Coding Theory (1991) M. O. Gonzalez, Classical Complex Analysis (1992) M. O. Gonzalez, Complex Analysis (1992)

153. L W. Baggett, Functional Analysis (1992) 154. M. Sniedovich, Dynamic Programming (1992) 155. R. P. Agarwal, Difference Equations and Inequalities (1992) 156. 157. 158. 159.

C. Brezinski, Biorthogonality and Its Applications to Numerical Analysis (1992) C. Swartz, An Introduction to Functional Analysis (1992) S. 8. Nadler, Jr., Continuum Theory (1992) M. A. AI-Gwaiz, Theory of Distributions (1992)

160. E. Perry, Geometry: Axiomatic Developments with Problem Solving (1992) 161. E. Castillo and M. R. Ruiz-Cobo, Functional Equations and Modelling in Science and Engineering (1992)

162. A. J. Jerri, Integral and Discrete Transforms with Applications and Error Analysis (1992) 163. A. Charlieret a/., Tensors and the Clifford Algebra (1992)

164. 165. 166. 167. 168. 169. 170.

P. Bilerand T. Nadzieja, Problems and Examples in Differential Equations (1992) E. Hansen, Global Optimization Using Interval Analysis (1992) S. Guerre-Delabriere, Classical Sequences in Banach Spaces (1992) Y. C. Wong, Introductory Theory of Topological Vector Spaces (1992) S. H. Kulkami and B. V. Limaye, Real Function Algebras (1992) W. C. Brown, Matrices Over Commutative Rings (1993) J. LoustauandM. Dillon, Linear Geometry with Computer Graphics (1993)

171. W. V. Petryshyn, Approximation-Solvability of Nonlinear Functional and Differential Equations (1993) 172. E. C. Young, Vector and Tensor Analysis: Second Edition (1993) 173. T. A. Bick, Elementary Boundary Value Problems (1993)

174. M. Pave/, Fundamentals of Pattern Recognition: Second Edition (1993) 175. S. A. Albeverio et al., Noncommutative Distributions (1993) 176. W. Fulks, Complex Variables (1993)

177. M. M. Rao, Conditional Measures and Applications (1993) 178. A. Janicki and A. Weron, Simulation and Chaotic Behavior of a-Stable Stochastic Processes (1994) 179. P. Neittaanmaki and D. Tiba, Optimal Control of Nonlinear Parabolic Systems (1994) 180. J. Cronin, Differential Equations: Introduction and Qualitative Theory, Second Edition (1994)

181. S. HeikkilS and V. Lakshmikantham, Monotone Iterative Techniques for Discontinuous Nonlinear Differential Equations (1994) 182. X. Mao, Exponential Stability of Stochastic Differential Equations (1994)

183. B. S. Thomson, Symmetric Properties of Real Functions (1994) 184. J. E. Rub/o, Optimization and Nonstandard Analysis (1994) 185. J. L Bueso et al., Compatibility, Stability, and Sheaves (1995) 186. A. N. Michel and K. Wang, Qualitative Theory of Dynamical Systems (1995) 187. M. R. Darnel, Theory of Lattice-Ordered Groups (1995)

188. Z. Naniewicz and P. D. Panagiotopoulos, Mathematical Theory of Hemivariational Inequalities and Applications (1995) 189. L. J. Co/w/n and R. H. Szczarta, Calculus in Vector Spaces: Second Edition (1995) 190. L. H. Erbe et al., Oscillation Theory for Functional Differential Equations (1995)

191. S. Agaian etal.. Binary Polynomial Transforms and Nonlinear Digital Filters (1995) 192. 193. 194. 195. 196.

M, I. Gil', Norm Estimations for Operation-Valued Functions and Applications (1995) P. A. Grillet, Semigroups: An Introduction to the Structure Theory (1995) S. Kichenassamy, Nonlinear Wave Equations (1996) V. F. Krotov, Global Methods in Optimal Control Theory (1996) K. /. Beidaret al.. Rings with Generalized Identities (1996)

197. V. I. Amautov et al., Introduction to the Theory of Topological Rings and Modules 198. 199. 200. 201. 202. 203. 204.

(1996) G. S/erfcsma, Linear and Integer Programming (1996) R. Lasser, Introduction to Fourier Series (1996) V. Sima, Algorithms for Linear-Quadratic Optimization (1996) D. Redmond, Number Theory (1996) J. K. Beem et al., Global Lorentzian Geometry: Second Edition (1996) M. Fontana et al., Priifer Domains (1997) H. Tanabe, Functional Analytic Methods for Partial Differential Equations (1997)

205. C. Q. Zhang, Integer Flows and Cycle Covers of Graphs (1997) 206. E. Spiegel and C. J. O'Donnell, Incidence Algebras (1997)

207. B. Jakubczyk and W. Respondek, Geometry of Feedback and Optimal Control (1998) 208. T. W. Haynes et al., Fundamentals of Domination in Graphs (1998) 209. T. W. Haynes et al., Domination in Graphs: Advanced Topics (1998) 210. L. A. D'Alotto et al., A Unified Signal Algebra Approach to Two-Dimensional Parallel

Digital Signal Processing (1998) 211. F. Halter-Koch, Ideal Systems (1998) 212. N. K. Goviletal., Approximation Theory (1998) 213. R. Cross, Multivalued Linear Operators (1998) 214. A. A. Martynyuk, Stability by Liapunov's Matrix Function Method with Applications (1998) 215. A. FaviniandA. Yagi, Degenerate Differential Equations in Banach Spaces (1999)

216. A. Illanes and S. Nadler, Jr., Hyperspaces: Fundamentals and Recent Advances (1999)

217. G. Kato and D. Struppa, Fundamentals of Algebraic Microlocal Analysis (1999) 218. G. X.-Z. Yuan, KKM Theory and Applications in Nonlinear Analysis (1999)

219. D. Motreanu and N. H. Pavel, Tangency, Flow Invariance for Differential Equations, and Optimization Problems (1999) 220. K. Hrbacek and T. Jech, Introduction to Set Theory, Third Edition (1999) 221. G. £ Kolosov, Optimal Design of Control Systems (1999) 222. A. I. Prilepko et al., Methods for Solving Inverse Problems in Mathematical Physics (1999)

Additional Volumes in Preparation

OPTIMAL DESIGN OF CONTROL SYSTEMS Stochastic and Deterministic Problems

G. E. Kolosov Moscow University of Electronics and Mathematics Moscow, Russia

MARCEL

MARCEL DEKKER, INC.

NEW YORK • BASEL

Library of Congress Cataloging-in-Publication Data Kolosov, G. E. (Gennadil Evgen'evich) Optimal design of control systems: stochastic and deterministic problems / G. E.

Kolosov. p. cm.— (Monographs and textbooks in pure and applied mathematics; 221) Includes bibliographical references and index.

ISDN 0-8247-7537-6 (alk. paper) 1. Control theory. 2. Mathematical optimization. I. Title. II. Scries.

QA402.3.K577 1999 629.8312—dc21

99-30940 CIP

This book is printed on acid-free paper. Headquarters

Marcel Dekker, Inc. 270 Madison Avenue, New York, NY 10016 tel: 212-696-9000; fax: 212-685-4540 Eastern Hemisphere Distribution Marcel Dekker AG Hutgasse 4, Postfach 812, CH-4001 Basel, Switzerland tel: 41-61-261-8482; fax: 41-61-261-8896

World Wide Web http://www.dekker.com

The publisher offers discounts on this book when ordered in bulk quantities. For more information, write to Special Sales/Professional Marketing at the headquarters address above. Copyright © 1999 by Marcel Dekker, Inc. AH Rights Reserved.

Neither this book nor any part may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, microfilming, and recording, or by any information storage and retrieval system, without permission in writing from the publisher.

Current printing (last digit): 10 9 8 7 6 5 4 3 2 1 PRINTED IN THE UNITED STATES OF AMERICA

PREFACE

The rise of optimal control theory is a remarkable example of interaction between practical needs and mathematical theories. Indeed, in the middle of this century the development of various automatic control systems in technology and of systems for control of motion of mechanical objects (in particular, of flying objects such as airplanes and rockets) gave rise to specific mathematical problems concerned with finding the conditional extremum of functions or functionals, which could not be solved by means of the methods of classical mathematical analysis and the calculus of variations. Extreme urgency of these problems for practical needs stimulated the efforts of mathematicians to develop methods for solving these new problems. At the end of the fifties and at the beginning of the sixties, these efforts were crowned with success when new mathematical approaches such as Pontryagin's maximum principle, Bellman's dynamic programming, and linear and convex programming (developed somewhat earlier by L. Kantorovich, G. Dantzig, and others) were established. These new approaches greatly affected the research carried out in control theory at that time. It should be noted that these approaches have played a very important role in the process of formation of optimal control theory as an independent branch of science. One can say that the role of the maximum principle and dynamic programming in the theory of optimal control is as significant as that of Maxwell's equations in electromagnetic theory in physics. Optimal control theory evolved most intensively at the end of the sixties and during the seventies. This period showed a very high degree of cooperation and interaction between mathematicians and all those dealing with applications of control theory in technology, mechanics, physics, chemistry, biology, etc. Later on, a gap between the purely mathematical and the practical approach to solving applied problems of optimal control began to emerge and is now apparent. Although the appearance of this gap can be explained by quite natural reasons, nevertheless, the further growth of this trend seems to be undesirable. The author hopes that this book will to some extent reduce the gap between these two branches of research. iii

IV

Preface

This book is primarily intended for specialists dealing with applications of control theory. It is well known that the use of such approaches as, say, the maximum principle or dynamic programming often leads to optimal control algorithms whose implementation for actual real-time plants encounters great (sometimes insurmountable) difficulties. This is the reason

that for solving control problems in practice one often employs methods based on various simplifications and heuristic concepts.

Naturally, this

results in losses in optimality but makes it possible to obtain control algorithms that allow simple technological implementations. In some cases the use of simplifications and heuristic concepts can also result in significant deviations of the system performance index from its optimal value (Chapter VI). In this book we describe ways for constructing simply realizable algorithms of optimal (suboptimal) control, which are based on the dynamic

programming approach. These algorithms are derived on the basis of exact, approximate analytical, or numerical solutions of differential and functional Bellman equations corresponding to the control problems considered.

The book contains an introduction and seven chapters. Chapter I deals with some general concepts of control theory and the description of mathematical approaches to solving problems of optimal control. We consider both deterministic and stochastic models of controlled systems and discuss

the distinguishing features of stochastic models, which arise due to possible ambiguous interpretation of solutions to stochastic differential equations describing controlled systems with white noise disturbances. We define the synthesis problem as the principal problem of optimal control theory and give a general scheme of the dynamic programming approach. The Bellman equations for deterministic and stochastic control problems (for Markov models and stochastic models with indirect observations) are studied. For problems with infinite horizon we introduce the

concept of stationary operating conditions, which is widely used in further chapters of the book. Exact methods of synthesis are considered in Chapter II. We describe the exceptional cases in which the Bellman equations have exact solutions, and

hence the optimal control algorithms can be obtained in explicit analytical forms.

First (in §2.1), we briefly discuss some well-known results concerned with solution of the so-called LQ-problems. Next, in §§2.2-2.4, we write exact solutions for three specific problems of optimal control with bounded control actions. We consider deterministic and stochastic problems of control of

the population size and the problem of constructing an optimal servomechanism. In these systems, the optimal controllers are of the "bang-bang"

form, and the switch point coordinates are given by finite formulas.

Preface

v

The following four chapters are devoted to the description of approximate methods for synthesis. In this case, the design of suboptimal control systems is based, as a rule, on using the approximate solutions of the corresponding Bellman equations. To obtain these approximate solutions, we mainly use various versions of small parameter methods or successive approximation procedures. In Chapter III we study weakly controlled systems. We consider control problems with bounded controls and assume that the values of admissible control actions are small. This stipulates the appearance of a small parameter in the nonlinear term in the Bellman equation. This, in turn, makes it possible to propose a natural successive approximation procedure for solving the Bellman equation, and thus the synthesis problem, approximately. This procedure is a modification of the well-known Picard and Bellman procedures which provide a way for obtaining approximate solutions of nonlinear differential equations by solving a sequence of linear equations. Chapter III is organized as follows. First (in §3.1), we describe the general scheme of approximate synthesis for controlled systems under stationary operating conditions. Next (in §3.2), by using this general scheme, we calculate a suboptimal controller for an oscillatory system with one degree of freedom. Later (in §3.3 and §3.4), we generalize our approach to nonstationary problems and to the case of correlated disturbances; then we estimate the error obtained. In §3.5 we prove that the successive approximation procedure in question converges asymptotically. Finally (in §3.6), we apply this approach to an approximate design of a stochastic system with distributed parameters. Chapter IV is about stochastic controlled systems with noises of small intensities. In this case, the diffusion terms in the Bellman equation contain small coefficients. Under certain assumptions this allows us to replace the initial stochastic problem by a sequence of auxiliary deterministic problems of optimal control whose solutions (i) can be calculated more easily and (ii) give a way for designing suboptimal control systems (with respect to the initial stochastic problem). This approach is used for calculating suboptimal controllers for two specific servomechanisms. In Chapter V we consider a class of controlled systems whose dynamics are quasiharmonic. The trajectories of such systems are close to harmonic oscillations, and this is the reason that the well-developed techniques of the theory of nonlinear oscillations can be effectively applied for studying these systems. By using polar coordinates as the phase variables, we describe the system state in terms of slowly changing amplitude and phase. The presence of a small parameter on the right-hand sides of the differential equations for these variables allows us to elaborate different versions of approximate solutions for the various problems of optimal control. These

vi

Preface

solutions are based on the use of appropriate asymptotic expansions of the performance index, the optimal control algorithm, etc. in powers of the small parameter. We illustrate these techniques by solving four specific problems of optimal damping of deterministic and stochastic oscillations in a biological predator-prey system and in a mechanical system with oscillatory dynamics. In Chapter VI we discuss some special asymptotic methods of synthesis which do not belong to the classes of control problems studied in Chapters III-V. We consider the problems of control of plants with unknown parameters (the adaptive control problems), in which the a priori uncertainty of their values is small. In addition, we study stochastic control problems with bounded phase variables and a problem of optimal control of the population size whose behavior is governed by a stochastic logistic equation with a large value of the medium capacity. We use small parameter approaches for solving the problems mentioned above. For the construction of suboptimal controls, we employ the asymptotic series expansions for the loss functions and the optimal control algorithms. The error obtained is estimated. Numerical methods of synthesis are covered in the final Chapter VII. We discuss the problem of the assignment of boundary conditions to grid

functions and propose some different schemes for solving specific problems of optimal control. The numerical methods proposed are used for solving specific synthesis problems. The presentation of all the approaches studied in the book is accompanied by numerous examples of actual control problems. All calculations are carried out up to the accuracy level sufficient for comparatively simple implementation of the optimal (suboptimal) algorithms obtained in actual devices. In many cases, the algorithms are presented in the form of analogous circuits or flow charts. The book can be helpful to students, postgraduate students, and specialists working in the field of automatic control and applied mathematics. The book may be of interest to mechanical and electrical engineers, physicists and biologists. Only knowledge of the foundations of probability theory is required for assimilating the subject matter of the book. The reader should be acquainted with basic notions of probability theory such as random events and random variables, the probability distribution function and the probability density of random variables, the mean value of a random variable, inconsistent and independent random events and variables, etc. It is not compulsory to know the foundations of the theory of random processes, since Chapter I provides all necessary facts about the methods for describing random processes that are encountered further in

Preface

vii

the book. This makes the book accessible to a wide circle of students and specialists who are interested in applications of optimal control theory. The author's intention to write this book was supported by R. L. Stratonovich, who was the supervisor of the author's Ph.D thesis and for many

years till his sudden death in 1997 remained the author's friend. The author wishes to express his deep gratitude to V. B. Kolmanovskii,

R. S. Liptser, and all participants of the seminar "Stability and Control" at the Moscow University of Electronics and Mathematics for useful remarks and advice concerning the contents of this book. The author's special thanks go to M. A. Shishkova for translating the manuscript into English and keyboarding.

G. E. Kolosov

CONTENTS

Preface

v

Introduction

1

Chapter I. Synthesis Problems for Control Systems and the Dynamic Programming Approach 1.1. Statement of synthesis problems for optimal control systems 1.2. Differential equations for controlled systems with random functions 1.3. Deterministic control problems. Formal scheme of the dynamic programming approach 1.4. The Bellman equations for Markov controlled processes 1.5. Sufficient coordinates in control problems with indirect observations Chapter II. lems

7 7

32 48 57

75

Exact Methods for Synthesis Prob-

2.1. Linear-quadratic problems of optimal control (LQproblems) 2.2. Problem of optimal tracking a wandering coordinate 2.3. Optimal control of the population size 2.4. Stochastic problem of optimal fisheries management Chapter III. Approximate Synthesis of Stochastic Control Systems With Small Control Actions 3.1. Approximate solution of stationary synthesis problems 3.2. Calculation of a quasioptimal regulator for the oscillatory plant

93

93 103 123 133

141

144 154 IX

Contents 3.3. Synthesis of quasioptimal controls in the case of correlated noises 3.4. Nonstationary problems. Estimates of the quality of approximate synthesis 3.5. Analysis of the asymptotic convergence of successive

approximations (3.0.6)-(3.0.8) as k —>• oo 3.6. Approximate synthesis of some stochastic systems with distributed parameters Chapter IV.

164 175

188 199

Synthesis of Quasioptimal Systems

in the Case of Small Diffusion Terms in the

Bellman Equation 4.1. Approximate synthesis of a servomechanism with small-intensity noise

4.2. Calculation of a quasioptimal system for tracking a discrete Markov process

219 221

233

Chapter V. Control of Oscillatory Systems 5.1. Optimal control of a quasiharmonic oscillator. An

247

asymptotic synthesis method 5.2. Control of the "predator-prey" system. The case of a poorly adapted predator 5.3. Optimal damping of random oscillations 5.4. Optimal control of quasiharmonic systems with noise in the feedback circuit

248

Chapter VI.

267 276 298

Some Special Applications of

Asymptotic Synthesis Methods 6.1. Adaptive problems of optimal control 6.2. Some stochastic control problems with constrained

311 312

phase coordinates 6.3. Optimal control of the population size governed by the stochastic logistic model

328 341

Chapter VII. Numerical Synthesis Methods 7.1. Numerical solution of the problem of optimal damping of random oscillations 7.2. Optimal control for the "predator-prey" system (the general case)

355

Conclusion References Index

383 387 401

356 368

INTRODUCTION

The main problem of the control theory can be formulated as follows. In the design of control systems it is assumed that each control system (see Pig. 1) consists of the following two principal parts (blocks or subsystems): the subsystem P to be controlled (the plant) and the controlling subsystem C (the controller). The plant P is a dynamical system (mechanical, electrical, biological, etc.) whose behavior is described by a well-known operator mapping the input (controlling) actions u(t) into the output trajectories x(t}. This operator can be denned by a system of ordinary differential, functional, functional-differential, or integral equations or by partial differential equations. It is important that the operator (or, in technical terms, the structure or the construction) of the plant P is assumed to be given and fixed from the outset.

x(t)

U(t)

P

C

FIG. 1 As for the controller C, no preliminary restrictions are imposed on its structure. This block must be constructed in such a way that the output

trajectories { x ( t ) : 0 < t < T] (the case T = +00 is not excluded) possess, in a sense, sufficiently "good" properties. Whether the trajectories are "good" or not depends on the specifications imposed on the control system in question. These assumptions are often stated by using the concept of a support (or standard) trajectory x ( t ) , and the control system itself is constructed so that the deviation x(t) — x(t)\

on the time interval 0 < t < T does not exceed a value given in advance. If the "quality" of an individual trajectory {x(t): 0 < t < T} can be estimated by the value of some functional /[»(