Selected Topics in Approximation and Computation (International Series of Monographs on Computer Science)

Selected Topics in Approximation and Computation

This page intentionally left blank

Selected Topics in Approximation and Computation

Marek A. Kowalski UNIVERSITY OF WARSAW

Krzysztof A. Sikorski UNIVERSITY OF UTAH

Frank Stenger

UNIVERSITY OF UTAH

New York Oxford OXFORD UNIVERSITY PRESS 1995

Oxford University Press Oxford New York Athens Auckland Bangkok Bombay Calcutta Cape Town Dar es Salaam Delhi Florence Hong Kong Istanbul Karachi Kuala Lumpur Madras Madrid Melbourne Mexico City Nairobi Paris Singapore Taipei Tokyo Toronto and associated companies in Berlin Ibadan

Copyright © 1995 by Oxford University Press, Inc. Published by Oxford University Press, Inc., 198 Madison Avenue, New York, New York 10016 Oxford is a registered trademark of Oxford University Press, Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior permission of Oxford University Press. Library of Congress Cataloging-in-Publication Data Kowalski, Marek A., 1953Selected topics in approximation and computation/ Marek A. Kowalski, Krzysztof A. Sikorski, Frank Stenger. p. cm. Includes bibliographical references (p. - ) and index. ISBN 0-19-508059-9 1. Approximation theory. I. Sikorski, Krzysztof A., 1953- . II. Stenger, Frank. III. Title. QA221.K69 1995 511'.4—dc20 95-6224

9 8 7 6 5 4 3 2 1 Printed in the United States of America on acid-free paper

To our wives and children

This page intentionally left blank

Preface This text covers classical basic results of approximation theory. It also obtains new developments in the theory of moments and Sine approximation, as well as n-widths, s-numbers, and the relationship of these concepts to computational complexity. In addition, the text also contains several computational algorithms. Chapter 1 covers basic concepts of classical approximation. In Section 1.1, the classical and basic concepts of approximation theory are couched in the language of functional analysis. Thus, it is most convenient to cover the concepts of existence and uniqueness of best approximation in a normed space setting. From the point of both theory and application, the inner product space is an important normed space in which to study best approximation. In this space it is convenient to study approximation in various polynomial settings such as via classical orthogonal polynomials and via Cardinal, or Sine approximation. These concepts are covered in Section 1.2 of the text. Concepts of approximation in the uniform norm are covered in Section 1.3. These concepts are also conveniently presented in a Banach space setting. Important examples from a classical standpoint include polynomial approximation with respect to the uniform norm. Chapters 2 deals with spline methods of approximation. Polynomials are historically the most popular tools of approximation since they are easy to compute. However, an interpolatory polynomial of high degree does not do a good job of approximating arbitrary data, since it is then nearly always the case that the polynomial also has large overshoots and undershoots between data points. On the other hand, splines, which are piecewise polynomials discussed in Chapter 2, are very convenient for approximating data, particularly data that is contaminated with noise. This is especially true of the practically important B-splines, which have variationdiminishing properties when used to approximate data ( i.e., the total variation of the spline approximant is no more than that of the

viii

PREFACE

data). Thus splines provide particularly useful methods of approximation in the important areas of computer-aided geometric design and for representing computer graphics displays. Sine methods discussed in Chapter 3 are ideal for approximating functions that may have singularities at end-points of an interval. Spline methods are ideal for the approximation of data, and polynomial methods are ideal for approximating analytic functions that have no singularities on the interval of approximation. For example, if a function is analytic in a region containing an interval /, then we can achieve an C9(exp{—bn}} error in a degree n polynomial approximation of the function in 7, where 6 is some positive constant. This rapid exponential rate of decrease of the error reduces to a drastically slow O(n~b'} rate in the case when the function has a singularity on /. On the other hand, using ra-point Sine approximation, we can achieve an (9(exp{ — frn1/2}) error of approximation whether or not the function has an end-point singularity on /. Also, it turns out that Sine methods provide simple-to-use, accurate approximation tools for every operation of calculus, including the approximation of Hilbert transforms, the approximation of derivatives, the approximation of definite and indefinite integration, the approximation and inversion of Laplace transforms, and the approximation of definite and indefinite convolution integrals. While an 0(exp{ —fr'n 1 / 2 }) error is also possible via spline approximation, by suitable choice of both the mesh and the degree of the spline on each subinterval, the constant b' in this exponential rate of convergence is usually not as large as the corresponding constant 6 for sine approximation. In Chapter 4 we present a family of simple rational functions, which make possible the explicit and arbitrarily accurate rational approximation of the filter, the step, and impulse functions! Moment problems are conveniently discussed in the setting of approximation theory in Chapter 5. Included among the well known moment problems are the discrete and continuous moment problems named after Hausdorff, Stieltjes, and Hamburger, as well as the discrete and continuous trigonometric moment problems. We also include the Sine moment problem, whose solution in the appropriate Sine space is relatively easy and devoid of the difficulties that one encounters using the usual monomial bases. Chapter 6 deals with some rather "deep" concepts of approximation — n-widths and s-numbers. An important and historic problem of approximation theory is to achieve a practically accurate approximation to a function / by a polynomial of a certain degree. Next, one might want to know the error of best approximation of the func-

PREFACE

ix

tion / by polynomials of degree n. Following this, we could identify a class of functions F to which the function / to be approximated belongs, and we could then determine the maximum error of best approximation, as / is varied throughout F. Finally, we could vary our tools of approximation, i.e., the classes of n-dimensional subspaces (e.g., not just polynomials), and so deduce the best method of approximation of functions in F. Thus we are able to explore limits of approximability, a knowledge which is both practically and theoretically worthwhile. Chapter 7 discusses optimal methods of approximation and optimal algorithms for general, nonlinear approximation problems. It relates approximability with the amount of work required to achieve a certain accuracy. These concepts are discussed in the general setting of normed spaces. Later, they are connected with splines and also with the concepts of n-widths and s-numbers discussed in the previous chapter. Chapter 8 illustrates applications of the approximation theory of the previous chapters. There we discuss the solution of Burgers' equation, the approximation of band-limited signals, and a nonlinear zero-finding problem. Each section of the text ends with a set of exercises. Each chapter closes with annotations, which include historical remarks that indicate the source of the material. References follow at the end of each chapter. Exercises are numbered from 1 to 180 globally throughout the text. Theorems, lemmas, corollaries, examples, and figures are numbered consecutively on each page, and have the page number attached to their name. For example, Lemma 105.1 is the name of the first lemma on page 105, and similarly, Example 45.2 is the name of the second example on page 45. There are no references to the literature inside of the text. All references are discussed in the annotations to each chapter. Numbered formulas are almost eliminated to provide more structured text. In the few remaining cases they are numbered again according to the page system described above (i.e., formula 15.1 refers to first formula on page 15). We believe that the special format chosen will best serve the reader by providing more structured and self contained text.

x

PREFACE

We are very grateful for criticisms and remarks to this manuscript that were given to us by: L. Plaskota, G. Wasilkowski, and H. Wozniakowski. We also wish to acknowledge the support of the IBM corporation (Sikorski, Stenger), the NSF, and the University of Utah under the ACERC center (Sikorski), and the Committee for Scientific Research - KBN (Kowalski) during parts of the duration of this project. M. A. Kowalski, K. Sikorski, and F. Stenger, December 1994-

Contents 1

Classical Approximation

1

1.1 General results 1 1.1.1 Exercises 12 1.2 Approximation in unitary spaces 13 1.2.1 Computing the best approximation 17 1.2.2 Completeness of orthogonal systems 20 1.2.3 Examples of orthogonal systems 21 1.2.4 Remarks on convergence of Fourier series . . . 34 1.2.5 Exercises 36 1.3 Uniform approximation 39 1.3.1 Chebyshev subspaces 42 1.3.2 Maximal functionals 47 1.3.3 The Remez algorithm 56 1.3.4 The Korovkin operators 58 1.3.5 Quality of polynomial approximations 63 1.3.6 Converse theorems in polynomial approximation 66 1.3.7 Projection operators 72 1.3.8 Exercises . 83 1.4 Annotations 87 1.5 References 89 2

Splines

2.1

Polynomial splines 2.1.1 Exercises 2.2 B-splines 2.2.1 General spline interpolation 2.2.2 Exercises 2.3 General splines 2.3.1 Exercises 2.4 Annotations 2.5 References

93

93 102 103 109 110 Ill 114 . 114 115

xii

CONTENTS'

3

Sine Approximation 117 3.1 Basic definitions 117 3.1.1 Exercises .125 3.2 Interpolation and quadrature 126 3.2.1 Exercises 132 3.3 Approximation of derivatives on T 134 3.3.1 Exercises 136 3.4 Sine indefinite integral over F . . 136 3.4.1 Exercises 139 3.5 Sine indefinite convolution over F 139 3.5.1 Derivation and justification of procedure . . . . 141 3.5.2 Multidimensional indefinite convolutions . . . . 146 3.5.3 Two dimensional convolution . 147 3.5.4 Exercises 149 3.6 Annotations 150 3.7 References 150

4

Explicit Sine-Like Methods 153 4.1 Positive base approximation 153 4.1.1 Exercises 158 4.2 Approximation via elliptic functions 158 4.2.1 Exercises 160 4.3 Heaviside, filter, and delta functions 161 4.3.1 Heaviside function 162 4.3.2 The filter or characteristic function . . . . . . . 163 4.3.3 The impulse or delta function 164 4.3.4 Exercises 166 4.4 Annotations 166 4.5 References 166

5

Moment Problems 5.1 Duality with approximation 5.1.1 Exercises 5.2 The moment problem in the space Co(D] 5.3 Classical moment problems 5.3.1 Exercises 5.4 Density and determinateness 5.4.1 Exercises 5.5 A Sine moment problem 5.5.1 Exercises 5.6 Multivariate orthogonal polynomials 5.6.1 Exercises

169 170 -. 175 175 178 185 189 203 205 206 206 218

CONTENTS 5.7 5.8

Annotations References

6 n-Widths and s-Numbers 6.1 n-Widths 6.1.1 Relationships between n-widths 6.1.2 Algebraic versions of an and cn 6.1.3 Exercises 6.2 s-Numbers 6.2.1 s-Numbers and singular values 6.2.2 Relationships between s-numbers 6.2.3 Exercises 6.3 Annotations 6.4 References 7

xiii 219 220 223 223 229 235 236 237 240 246 255 255 256

Optimal Approximation Methods 259 7.1 A general approximation problem 262 7.1.1 Radius of information—optimal algorithms . .264 7.1.2 Exercises 270 7.2 Linear problems 270 7.2.1 Optimal information 276 7.2.2 Relations to n-widths 281 7.2.3 Exercises 285 7.3 Parallel versus sequential methods 286 7.3.1 Exercises 290 7.4 Linear and spline algorithms 291 7.4.1 Spline algorithms 295 7.4.2 Relations to linear Kolmogorov n-widths . . . 302 7.4.3 Exercises 304 7.5 s-Numbers, minimal errors 304 7.5.1 Exercises 309 7.6 Optimal methods 310 7.6.1 Optimal complexity methods for linear problems312 7.6.2 Exercises 314 7.7 Annotations 314 7.8 References . 316

8 Applications 8.1 Sine solution of Burgers' equation 8.2 Signal recovery 8.2.1 Formulation of the problem 8.2.2 Relations to n-widths

319 319 321 321 322

xiv 8.2.3 Algorithms and their errors 8.2.4 Asymptotics of minimal cost 8.2.5 Exercises 8.3 Bisection method . 8.3.1 Formulation of the problem . 8.3.2 Optimality theorem 8.3.3 Exercises 8.4 Annotations 8.5 References . Index

CONTENTS 325 332 333 334 .334 335 . 340 340 340 343

Selected Topics in Approximation and Computation

This page intentionally left blank

Chapter 1

Classical Approximation In this chapter we acquaint the reader with the theory of approximation of elements of normed spaces by elements of their finite dimensional subspaces. The theory of best approximation was originated between 1850 and 1860 by Chebyshev. His results and ideas have been extended and complemented in the 20th century by other eminent mathematicians, such as Bernstein, Jackson, and Kolmogorov. Initially, we present the classical theory of best approximation in the setting of normed spaces. Next, we discuss best approximation in unitary (inner product) spaces, and we present several practically important examples. Finally, we give a reasonably complete presentation of best uniform approximation, along with examples, the Remez algorithm, and including converse theorems about best approximation.

1.1

General results

The goal of this section is to present some general results on approximation in normed spaces. Let T be a linear space (over the field of complex or real numbers) endowed with the norm || • || and let V be a, finite dimensional subspace of f'. Suppose that / is an element of f. We wish to determine an element v in V satisfying

At this moment it is natural to ask whether such an element v exists. It turns out that the answer is positive regardless of the structure of the spaces T and V. Namely, the following theorem holds.

2

CHAPTER 1. CLASSICAL

APPROXIMATION

Theorem 2.1 Under the above assumptions, the set

is nonempty, convex, and compact. Proof Since 0 G V we have

Let us now consider an arbitrary element g in V satisfying \\y\\ > 2||/||. We note that

Hence, g $ 0(f,V) and (3(f,V) C B = {g 6 V : \\g\\ < 2||/||}. Consequently, e(f, V) — inf^ 6j g | / — z/||. We remind the reader that any bounded and closed subset of a finite dimensional space is compact and that any continuous function on a compact set attains its infimum on the set. Since B is compact and the function

is continuous, we get the existence of a v G V such that ||/ — v = e ( f , V ) . Thus, /3(f,V) is nonempty. For arbitrary elements v\, v% G /?(/, V) and for any number a 6 [0,1] we have

Hence, the set /3(/, V7) is convex. Since it is also closed, the proof is complete. • Henceforth, an element v e /?(/, V] will be called a best approximation of / with respect to V, and the quantity e(f, V) will be referred to as the best approximation error of / with respect to V . Given a positive number r, we denote by B ( f , r] the ball of radius r about /. From the proof of Theorem 2.1 we get the following result.

1.1. GENERAL RESULTS

3

Corollary 3.1 The best approximation error e(/, V) can be rewritten as Moreover, if the above minimum is attained at r = p, then

Some properties of an element / G T can be carried over to its best approximation, as shown in the following theorem. Theorem 3.1 If f 6 F and A : F-+T is a linear operator such that \\A\\ d= sup^ii^! Ay11 < 1, A(V) C V and Af = f, then there is a best approximation h of f with respect to V satisfying Ah — h. Proof Indeed, for any best approximation g € V of / we have

Thus, the operator A continuously maps the set fl(f,V) into itself. If /?(/, V) is a singleton, then, of course, Ag = g. Otherwise, we use the Brouwer fixed point theorem , which reads as follows. Let D be a convex and compact subset of TV1 and let F : D —> D be a continuous mapping. Then there exists a point x in D such that F ( x ) = x. Since the dimension n is arbitrary and 7£2n can be regarded as Cn, the Brouwer theorem readily extends toward subsets of Cn. Let K be the field of scalars in T (K = 71 or K = C) and let the elements v\, v % , . . . , vn form a basis of V. Since A(V) C V, we have

where Ck, d^ € K and each dk depends linearly and continuously on GI, c 2 , . . . , cn. Thus, given a point (GI, c 2 , . . . , cn] € Kn the equation defines a linear and continuous mapping of Kn into Kn. By Theorem 2.1, the set /?(/, V) is nonempty convex and compact, and so is the set

4

CHAPTER 1. CLASSICAL

APPROXIMATION

Since A(ft(f,V)) C 0(/,V), we get F(D) C D. Thus, by the Brouwer theorem, there exists a point a; = (xi, x-2,.,., xn] € D such that F(x) = x. Finally, setting h = 53£=i xkvk we obtain h € P(f, V) and Ah = /i. The proof is complete. • Let us now consider the following two examples that illustrate the problem of uniqueness of best approximation. Example 4.1 Let f be the Cartesian plane 7£2 with the norm Let V be an arbitrary one dimensional subspace of T and let / be any element in f. Thus, any ball in T and any such a subspace V represent on the plane a disk and a straight line containing the origin, respectively. The situation is shown in Figure 4.1. From this figure and Corollary 3.1 we see that there exists a unique best approximation element to each / 6 T with respect to V. •

Figure 4.1: Approximation on Cartesian plane Example 4.2 As in the previous example let T = Ti2. We redefine the norm in T by the equation Thus, balls in T are now squares on the plane. Two sides of these squares are always parallel to the straight lines y — ±x (see Figure 5.1). Let V be a one dimensional subspace of T. Since any point in V is clearly its unique best approximation with respect to V, we assume that the element / to be approximated is in T \ V.

1.1. GENERAL RESULTS

5

Based on Corollary 3.1 and Figure 5.2 we see that /3(f, V) is a proper

Figure 5.1: Unit ball in 7£2 with the norm ||(x,y)|| = |xj + |y| closed segment of V if the slope of V is ±7r/4, and that /?(/, V) is a singleton if the slope of V is diiferent from ±7r/4. •

Figure 5.2: Sets of best approximations The examples above show that the uniqueness of best approximation depends on the geometry of the space F (induced by the norm), as well as on the subspace V and the element /. We should like to point out that the approximation problem in Example 4.2 has many solutions because the unit sphere

6

CHAPTER 1. CLASSICAL

APPROXIMATION

contains intervals, i.e., sets of the form

where f,g£F and / / g. We shall now characterize all normed spaces T such that approximation problem has a unique solution regardless of the choice of V and /. Theorem 6.1 The following conditions are equivalent: (a) For any finite dimensional subspace V of T and any element f of F there exists a unique best approximation of f with respect to V. (b) For any two distinct elements f and g such that \\f\\ = \\g\\ = 1 we have \\f + g\\ < 2. (c) The unit sphere dB(Q, 1) in T does not contain any interval. Proof We shall first prove the equivalence between (b) and (c). We assume that the condition (c) is not satisfied. Hence, the sphere 8B(Q,l) contains some interval, say [/,