Matrix algebra and its applications to statistics and econometrics

Matrix Algebra and Its Applications to Statistics and Econometrics This page is intentionally left blank Matrix Alge...

Author: C Radhakrishna Rao; M Bhaskara Rao

75 downloads 1249 Views 23MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Matrix Algebra and Its Applications to Statistics and Econometrics

This page is intentionally left blank

Matrix Algebra and Its Applications to Statistics and Econometrics

c. Radhakrishna Rao Pennsylvania State University, USA

M. Bhaskara Rao North Dakota State University, USA

, •

World Scientific Singapore· New Jersey· London· Hong Kong

Published by

World Scientific Publishing Co. Pte. Ltd. POBox 128, Farrer Road, Singapore 912805 USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE

Library of Congress Cataloging-in-Publication Data Rao, C. Radhakrishna (Calyampudi Radhakrishna), 1920Matrix algebra and its applications to statistics and econometrics / C. Radhakrishna Rao and M. Bhaskara Rao. p. cm. Includes bibliographical references and index. ISBN 9810232683 (alk. paper) I. Matrices. 2. Statistics. 3. Econometrics. I. Bhaskara Rao, M. QA188.R36 1998 98-5596 512.9'434--dc21 CIP

British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library.

First published 1998 Reprinted 2001, 2004

Copyright © 1998 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, or parts thereof, nwy not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any infornwtion storage and retrieval system now known or to be invented, without written permission from the Publisher.

For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher.

Printed in Singapore by Utopia Press Pte Ltd

To our wives

BHARGAVl (Mrs. C.R. Rao) JAYASRI (Mrs. M.B. Rao)


PREFACE

Matrix algebra and matrix computations have become essential prerequisites for study and research in many branches of science and technology. It is also of interest to know that statistical applications motivated new lines of research in matrix algebra, some examples of which are generalized inverse of matrices, matrix approximations, generalizations of Chebychev and Kantorovich type inequalities, stochastic matrices, generalized projectors, Petrie matrices and limits of eigenvalues of random matrices. The impact of linear algebra on statistics and econometrics has been so substantial, in fact, that a number of books devoted entirely to matrix algebra oriented towards applications in these two subjects are now available. It has also become a common practice to devote one chapter or a large appendix on matrix calculus in books on mathematical statistics and econometrics. Although there is a large number of books devoted to matrix algebra and matrix computations, most of them are somewhat specialized in character. Some of them deal with purely mathematical aspects and do not give any applications. Others discuss applications using limited matrix theory. We have attempted to bridge the gap between the two types. We provide a rigorous treatment of matrix theory and discuss a variety of applications especially in statistics and econometrics. The book is aimed at different categories of readers: graduate students in mathematics who wish to study matrix calculus and get acquainted with applications in other disciplines, graduate students in statistics, psychology, economics and engineering who wish to concentrate on applications, and to research workers who wish to know the current developments in matrix theory for possible applications in other areas. This book provides a self-contained, updated and unified treatment of the theory and applications of matrix methods in statistics and econ~ metrics. All the standard results and the current developments, such as the generalized inverse of matrices, matrix approximations, matrix vii

viii

MATRIX ALGEBRA THEORY AND APPLICATIONS

differential calculus and matrix decompositions, are brought together to produce a most comprehensive treatise to serve both as a text in graduate courses and a reference volume for research students and consultants. It has a large number of examples from different applied areas and numerous results as complements to illustrate the ubiquity of matrix algebra in scientific and technological investigations. It has 16 chapters with the following contents. Chapter 1 introduces the concept of vector spaces in a very general setup. All the mathematical ideas involved are explained and numerous examples are given. Of special interest is the construction of orthogonal latin squares using concepts of vector spaces. Chapter 2 specializes to unitary and Euclidean spaces, which are vector spaces in which distances and angles between vectors are defined. They playa special role in applications. Chapter 3 discusses linear transformations and matrices. The notion of a transformation from one vector space to another is introduced and the operational role of matrices for this purpose is explained. Thus matrices are introduced in a natural way and the relationship between transformations and matrices is emphasized throughout the rest of the book. Chapters 4, 5, 6 and 7 cover all aspects of matrix calculus. Special mention may be made of theorems on rank of matrices, factorization of matrices, eigenvalues and eigenvectors, matrix derivatives and projection operators. Chapter 8 is devoted to generalized inverse of matrices, a new area in matrix algebra which has been found to be a valuable tool in developing a unified theory of linear models in statistics and econometrics. Chapters 9, 10 and 11 discuss special topics in matrix theory which are useful in solving optimization problems. Of special interest are inequalities on singular values of matrices and norms of matrices which have applications in almost all areas of science and technology. Chapters 12 and 13 are devoted to the use of matrix methods in the estimation of parameters in univariate and multivariate linear models. Concepts of quadratic subspaces and new strategies of solving linear equations are introduced to provide a unified theory and computational techniques for the estimation of parameters. Some modern developments in regression theory such as total least squares, estimation of parameters in mixed linear models and minimum norm quadratic estimation are discussed in detail using matrix methods. Chapter 14

ix

Preface

deals with inequalities which are useful in solving problems in statistics and econometrics. Chapter 15 is devoted to non-negative matrices and Perron-Frobenius theorem which are essential for the study of and research in econometrics, game theory, decision theory and genetics. Some miscellaneous results not covered in the main themes of previous chapters are put together in Chapter 16. It is a pleasure to thank Marina Tempelman for her patience in typing numerous revisions of the book.

March 1998 C.R. Rao M.B. Rao


NOTATION

The following symbols are used throughout the text to indicate certain elements and the operations based on them.

Scalars

R C F x

= Xl + iX2

X=

Xl -

iX2

Ixi = (xi + X~)1/2

real numbers complex numbers general field of elements a complex number conjugate of X modulus of X

General

{an}

A,B, ... ACB xEA

A+B AuB AnB

a sequence of elements sets of elements set A is contained in set B x is an element of set A {Xl + X2 : Xl E A, X2 E B} {x : X E A and/or B} {x : X E A and x E B}

Vector Spaces

(V, F) dimV at, a2,··· Sp(at, ... , ak) Fn Rn

Cn

vector space over field F dimension of V vectors in V the set {alaI + ... + akak : al, .. · , ak E C} n dimensional coordinate (Euclidean) space same as Fn with F = R same as Fn with F = C xi

xii


direct sum, {x + y : x E V, yEW; V n W = O} inner product semi inner product

El1

< .,. >

(., .) Transformations

transformation from space V to space W the range of T, i.e., the set {Tx : x E V} the kernel of T, i.e., the set {Tx = 0 : x E V} nullity (dimension of K{T))

T:V-+W

R(T) K(T)

v{T) Matrices

general matrices or linear transformations m x n order matrix

A,B,C, ... A

mxn Mm,n Mm,n(-) Mn A = [aij]

A E Mm,n

Sp(A)

A* =.A' A* = A A*A = AA* A*A = AA* A# A-I

AA+ ALMN In

I

=I

the class of matrices with m rows and n columns m x n order matrices with specified property 0 the class of matrices with n rows and n columns aij is the (i,j) the entry of A (i-th row and j-th column) A is a matrix with m rows and n columns the vector space spanned by the column vectors of A, also indicated by R(A) considering A as transformation iiij is complex conjugate of aij obtained from A interchanging rows and columns, i.e., if A = (aij) then A' = (aji) Conjugate transpose or transpose of .A defined above Hermitian or self adjoint unitary normal adjoint (A E Mm,n, < Ax, Z >m= < x, A#z >n) inverse of A E Mn such that AA-I = A-I A = I generalized or g-inverse of A E Mm,n, (AA- A = A) Moore-Penrose inverse Rao-Yanai (LMN) inverse identity matrix of order n with all diagonal elements as unities and the rest as zeros identity matrix when the order is implicit

Notation

o

xiii

IIxll

zero scalar, vector or matrix rank of matrix A spectral radius of A vector of order mn formed by writing the columns of A E Mm,n one below the other matrix partitioned by colwnn vectors aI, ... ,an matrix partitioned by two matrices Al and A2 trace A, the sum of diagonal elements of A E Mn determinant of A Hadamard-Schur product Kronecker product Khatri-Hao product matrix with < bi , aj > as the (j, i)-th entry, where A = (all ... laftl ) , B = (bll· . ·Ibn ) norm of vector x

IIxll ..e

semi-norm of vector x

II All

norm or matrix norm of A

IIAIIF

Frobenius norm of A = ([tr(A* A)Jl/2

II Allin

induced matrix norm: max II Axil for

IIAlls

spectral norm of A

IIAllui

unitarily invariant norm, IIU* AVII = for all unitary U and V, A E Mm,n

IIAllwui

weakly unitarily invariant norm, IIU* AUII for all unitary U, A E Mn

IIAIIMNi

M, N invariant norm

m(A)

matrix obtained from A = (aij) by replacing aij by laijl, the modulus of the number aij E C positive definite matrix (x* Ax > 0 for x i= 0) non-negative definite matrix (x" Ax ~ 0) singular value decomposition

p(A) p.. (A) vec A

(all·· . Ian) [AIIA2J tr A IAI or det A

A·B A®B A0B AoB

pd nnd s.v.d. B~L A

IIxll =

1

IIAII = IIAII

or simply B ~ A to indicate (LOwner partial order) A - B is nnd


xiv

B ~e A

xi ~ Yi, i = 1, ... ,n, where x' = (x}, .. . ,xn ) and y' = (y}, . . . ,Yn) entry wise inequality bij ~ aij, A = (aij), B = (b ij )

B ~e A

entry wise inequality bij

X

~e Y

A

~e

A

>e 0

0

Y

«

y

«w x «8 x

y

x

~

aij

non-negative matrix (all elements are non-negative) positive matrix (all elements are positive) vector x majorizes vector y vector x weakly majorizes vector y vector x soft majorizes vector y

{Ai(A)}

eigenvalues of A E M n , [A} (A) ~ ... ~ An(A)]

{O'i(A)}

singular values of A E Mm,n, [O'}(A) ~ ... ~ O'r(A)], r = min{m,n}

CONTENTS

Preface . ....... ... .... . .. . ... . . . .. . .......... . ..... .. .. . . .. ... . . vii Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. xi

CHAPTER 1. VECTOR SPACES 1.1 Rings and Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 1.2 Mappings ... . . . ......... . .... . ................ .... .. . . . .... . 1.3 Vector Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 1.4 Linear Independence and Basis of a Vector Space . . . . . . . . . . . . . . . . 1.5 Subspaces ....... .. .. . ........... .. . . . ... .... .. . ... . . . .. .. . . 1.6 Linear Equations .. . .. . ... ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 1. 7 Dual Space ...... .. ......... .. ... . ..... . ....... . ....... .. ... 1.8 Quotient Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 1.9 Projective Geometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . CHAPTER 2. UNITARY AND EUCLIDEAN SPACES 2.1 Inner Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Orthogonality .... . ..... . .. .. ... . .. . ...................... . . . 2.3 Linear Equations .. ........... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Linear Functionals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Semi-inner Product ............. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Spectral Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Conjugate Bilinear Functionals and Singular Value Decomposition . .. ......... .. .. .. . . .. . . ... . . . . . .. ...... . ....

51 56 66 71 76 83 101

CHAPTER 3. 3.1 3.2 3.3 3.4

LINEAR TRANSFORMATIONS AND MATRICES Preliminaries . . ...... . . . ... ... .. . .... ..... . ... . . .. .. . . ...... Algebra of Transformations .. .. .... . .. . . ... ..... . .. .. . ... . . .. Inverse Transformations ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. Matrices . .. . . .. . ...... .. . . . .. .... .. .... . ....... . .... . .. .. ..

1 14 16 19 24 29 35 41 42

xv

107 110 116 120

XVI

MATRIX ALGERBA THEORY AND APPLICATIONS

CHAPTER 4. CHARACTERISTICS OF MATRICES 4.1 Rank and Nullity of a Matrix .......... .... .................. 4.2 Rank and Product of Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 4.3 Rank Factorization and Further Results .... .......... ..... .. . 4.4 Determinants . . .. ......... ... ............ ......... ...... . . . 4.5 Determinants and Minors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..

128 131 136 142 146

CHAPTER 5. FACTORIZATION OF MATRICES 5.1 Elementary Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . .. .......... 5.2 Reduction of General Matrices ......... .. ........ .. ... ... . . . . 5.3 Factorization of Matrices with Complex Entries . . . . . . . . . . . . . . . . . 5.4 Eigenvalues and Eigenvectors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Simultaneous Reduction of Two Matrices . . . . . . . . . . . . . . . . . . . . .. 5.6 A Review of Matrix Factorizations . . . . . . . . . . . . . . . . . . . . . . . . . . . .

157 160 166 177 184 188

CHAPTER 6. OPERATIONS ON MATRICES 6.1 Kronecker Product. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 6.2 The Vec Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 6.3 The Hadamard-Schur Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 6.4 Khatri-Roo Product. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 6.5 Matrix Derivatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..

193 200 203 216 223

CHAPTER 7.

PROJECTORS AND IDEMPOTENT OPERATORS Projectors ......... . ... .. ................... . ... . .. ... .... . Invariance and Reducibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Orthogonal Projection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. Idempotent Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. Matrix Representation of Projectors . . . . . . . . . . . . . . . . . . . . . . . . . . .

239 245 248 250 256

CHAPTER 8. GENERALIZED INVERSES 8.1 Right and Left Inverses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Generalized Inverse (g-inverse) ..... .. .... .. ..... .. . . . .. ...... 8.3 Geometric Approach: LMN-inverse .... .... .. ........... . .. . .. 8.4 Minimum Norm Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..

264 265 282 288

7.1 7.2 7.3 7.4 7.5

Contents

8.5 8.6 8.7 8.8 8.9

Least Squares Solution ... .... . .. . ..... . .......... . . . ....... . Minimum Norm Least Squares Solution . . . . . . . . . . . . . . . . . . . . . . .. Various Types of g-inverses ............................ . ..... G-inverses Through Matrix Approximations. . . . . . . . . . . . . . . . . . .. Gauss-Markov Theorem. . . .. . . .. . . . . . .... .. ... .. . . . . . .. . . . . .

xvii

289 291 292 296 300

CHAPTER 9. MAJORIZATION 9.1 Majorization.. ............................... . ........ . . . .. 303 9.2 A Gallery of Functions ..... .... ..... ............ ...... ... . .. 307 9.3 Basic Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 308 CHAPTER 10. INEQUALITIES FOR EIGENVALUES 10.1 Monotonicity Theorem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 10.2 Interlace Theorems ... . ... ... .. ........................... 10.3 Courant-Fischer Theorem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . lOA Poincare Separation Theorem ...... . ...... ...... . .. . . ... ... 10.5 Singular Values and Eigenvalues . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.6 Products of Matrices, Singular Values, and Horn's Theorem . . ........... . ........... . ... .................. . 10.7 Von Neumann 's Theorem . . . . .. . . . . .. .. . . . .. .. .. . .. . .. .. ... CHAPTER 11. MATRIX APPROXIMATIONS 11.1 Norm on a Vector Space. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 11.2 Norm on Spaces of Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 11.3 Unitarily Invariant Norms . ............................. ... 11.4 Some Matrix Optimization Problems . . . . . . . . . . . . . . . . . . . . . . .. 11.5 Matrix Approximations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 11.6 M, N-invariant Norm and Matrix Approximations . . . . . . . . . . . .. 11.7 Fitting a Hyperplane to a Set of Points . . . . . . . . . . . . . . . . . . . . ..

340 342

361 363 374 383 388 394 398

CHAPTER 12. 12.1 12.2 12.3 1204 12.5

OPTIMIZATION PROBLEMS IN STATISTICS AND ECONOMETRICS Linear Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Some Useful Lemmas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Estimation in a Linear Model ...................... .. .. . .. . A Trace Minimization Problem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . Estimation of Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..

322 328 332 337 339

403 403 406 409 413

xviii


12.6 The Method of MIN QUE: A Prologue. . . . . . . . . . . . . . . . . . . . . .. 12.7 Variance Components Models and Unbiased Estimation . . . . . .. 12.8 Normality Assumption and Invariant Estimators . . . . . . . . . . . . . 12.9 The Method of MIN QUE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 12.10 Optimal Unbiased Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.11 Total Least Squares. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

415 416 419 422 425 428

CHAPTER 13. QUADRATIC SUBSPACES 13.1 Basic Ideas 13.2 The Structure of Quadratic Subspaces . . . . . . . . . . . . . . . . . . . . . .. 13.3 Commutators of Quadratic Subspaces . . . . . . . . . . . . . . . . . . . . . . . 13.4 Estimation of Variance Components . . . . . . . . . . . . . . . . . . . . . . . . .

433 438 442 443

CHAPTER 14.

INEQUALITIES WITH APPLICATIONS IN STATISTICS Some Results on nnd and pd Matrices . . . . . . . . . . . . . . . . . . . . . .. Cauchy-Schwartz and Related Inequalities . ... ... .... ...... .. Hadamard Inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. Holder's Inequality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. Inequalities in Information Theory . . . . . . . . . . . . . . . . . . . . . . . . .. Convex Functions and Jensen's Inequality . . . . . . . . . . . . . . . . . . . . Inequalities Involving Moments. . . . . . . . . . . . . . . . . . . . . . . . . . . .. Kantorovich Inequality and Extensions . . . . . . . . . . . . . . . . . . . . . .

449 454 456 457 458 459 461 462

CHAPTER 15. NON-NEGATIVE MATRICES 15.1 Perron-Frobenius Theorem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 15.2 Leontief Models in Economics . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 15.3 Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 15.4 Genetic Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 15.5 Population Growth Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..

467 477 481 485 489

CHAPTER 16. MISCELLANEOUS COMPLEMENTS 16.1 Simultaneous Decomposition of Matrices . . . . . . . . . . . . . . . . . . . .. 16.2 More on Inequalities .. . . ... . .. .......... ..... . . .... .... ... 16.3 Miscellaneous Results on Matrices ... . .... . ...... .. .... . .... 16.4 Toeplitz Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16.5 Restricted Eigenvalue Problem . .... . ...... .. ...... . ... . ....

493 494 497 501 506

14.1 14.2 14.3 14.4 14.5 14.6 14.7 14.8

Contents

16.6 Product of Two Raleigh Quotients . . . . . . . . . . . . . . . . . . . . . . . . . . 16.7 Matrix Orderings and Projection . . ..................... . ... 16.8 Soft Majorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 16.9 Circulants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 16.10 Hadamard Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 16.11 Miscellaneous Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

xix

507 508 509 511 514 515

REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 519 INDEX ..... . .. . .. . .. . ....... . .... .. .. . ... . .. . . . .. . . . .. . . .. ... . 529

CHAPTER 1 VECTOR SPACES The use of matrix theory is now widespread in both physical and s~ cial sciences. The theory of vector spaces and transformations (of which matrices are a special case) have not, however, found a prominent place, although they are more fundamental and offer a better understanding of applied problems. The concept of a vector space is essential in the discussion of topics such as the theory of games, economic behavior, prediction in time series, and the modern treatment of univariate and multivariate statistical methods.

1.1. Rings and Fields Before defining a vector space, we briefly recall the concepts of groups, rings and fields. Consider a set G of elements with one binary operation defined on them. We call this operation multiplication. If a and f3 are two elements of G, the binary operation gives an element of G denoted byaf3. The set G is called a group if the following hold:

(gd (g2)

a(f31') = (af3h for every a, f3 and l' in G (associative law). The equations ay = f3 and ya = f3 have unique solutions for y for all a and f3 in G. From these axioms, the following propositions (P) follow. (We use the symbol P for any property, proposition or theorem. The first two digits after P denote the section number.)

P 1.1.1 There exists a unique element, which we denote by 1 (the unit element of G), such that al

=a

and la

=a

for every a in G.

P 1.1.2 For every a in G, there exists a unique element, which we denote by a-I (multiplicative inverse of a, or simply, the inverse of a) 1

2


such that

aa- I

= a-Ia = l.

A group G is said to be cormnutative if a{3 = {3a for every a and {3 in G. If the group is cormnutative, it is customary to call the binary operation as addition and use the addition symbol + for the binary operation on the elements of G. The unit element of G is called the zero element of G and is denoted by the symbol o. The inverse of any element a in G is denoted by -a. A cormnutative group is also called an abelian group. A simple example of an abelian group is the set of all real numbers with the binary operation being the usual addition of real numbers. Another example of an abelian group is the set G = (0, (0), the set of all positive numbers, with the binary operation being the usual multiplication of real numbers. We will present more examples later. A subgroup of a group G is any subset H of G with the properties that a{3 E H whenever a, {3 E H . A subgroup is a group in its own right under the binary operation of G restricted to H . If H is a subgroup of a group G and x E G, then xH = {xy : y E H} is called a left coset of H. If x E H, then xH = H . If xIH and X2H are two left cosets, then either xIH = X2H or xIH n X2H = 0. A right coset Hx is also defined analogously. A subgroup H of a group G is said to be invariant if xH = Hx for all x E G . Let H be an invariant subgroup of a group G. Let GIH be the collection of all distinct cosets of H . One can introduce multiplication between ele~ of GIH. If HI and H2 are two cosets, define HIH2 = {a{3 : a E HI and {3 E H2}. Under this binary operation, GIH is a group. Its unit element is H. The group GIH is called the quotient group of G modulo H. It can also be shown that the union of all cosets is G. More concretely, the cosets of H form a partition of G. There is a nice connection between finite groups and latin squares. Let us give a formal definition of a latin square. DEFINITION 1.1.3. Let T be a set of n elements.

A latin square of order n based on T is a square grid of n 2 elements t ~J' -- 1 < i ~ n, 1 ~ j ~ n arranged in n rows and n columns such that (1) tij E T for every i and j, (2) each element of T appears exactly once in each row, (3) each element of T appears exactly once in each column.

L

= (tiJ-)

3

Vector SpacetJ

In a statistical context, T is usually the set of treatments which we wish to compare for their effects over a certain population of experimental units. We select n 2 experimental units arranged in n rows and n columns. The next crucial step is the allocation of treatments to experimental units. The latin square arrangement of treatments is one way of allocating the treatments to experimental units. This arrangement will enable us to compare the effects of any pair of treatments, rows, and columns. Latin squares are quite common in parlor games. One of the problems is to arrange the kings (K), queens (Q), jacks (J) and aces (A) of a pack of cards in the form of a 4 X 4 grid so that each row and each column contains one from each rank and each suit. If we denote spades by S, hearts by H, diamonds by D and clubs by C, the following is one such arrangement.

SA CQ DJ HK

DK HJ SQ CA

HQ DA CK SJ

CJ SK HA DQ

The above arrangement is a superimposition of two latin squares. The suits and ranks each form a latin square of order 4. We now spell out the connection between finite groups and latin squares. P 1.1.4 Let G be any group with finitely many elements. Then the table of the group operation on the elements of G constitutes a latin square of order n on G. PROOF. Assume that G has n elements. Let G = {01' 02, ... ,On}. Assume, without loss of generality, that the group is commutative with the group operation denoted by +. Let us consider a square grid of size n x n, where the rows and columns are each indexed by 01,02,··· ,On and the entry located in the i-th row and j-th column is given by 0i +OJ. This is precisely the table of the group operation. We claim that no two elements in each row are identical. Suppose not. If 0i + OJ = 0i + Ok for some 1 ~ i,j, k ~ nand j 1= k, then OJ = Ok. This is a contradiction. Similarly, one can show that no two elements in each column are identical.

4


It is not difficult to construct latin squares on any n symbols. But it is nice to know that the group table of any finite group gives a latin square. However, it is not true that every latin square arises from a group table. We will talk more about latin squares when we discuss fields later. We now turn our attention to rings. Let K be a set equipped with two binary operations, which we call as addition and multiplication. The set K is said to be a ring if the following hold: (1) With respect to addition, K is an abelian group. (2) With respect to multiplication, the associative law holds, i.e.,

0:(f3'Y) = (o:f3h for every 0:, f3 and 'Y in K. (3) The multiplication is distributive with respect to addition, i.e.,

0:(f3 + 'Y) = 0:f3 + O:'Y for every 0:, f3 and 'Y in K. If the multiplication operation in the ring K is commutative, then K is called a commutative ring. As a simple example, let K = {a, 1,2,3,4, 5,6}. The addition and multiplication on K are the usual addition and multiplication of real numbers but modulo 7. Then K is a commutative ring. Let Z be the set of all integers with the usual operations of addition and multiplication. Then Z is a commutative ring. Finally, we come to the definition of a field. Let F be a set with the operations of addition and multiplication (two binary operations) satisfying the following: (1) With respect to the addition, F is an abelian group. (2) With respect to the multiplication, F - {a} is an abelian group. (3) Multiplication is distributive with respect to addition, i.e.,

0:(f3 + 'Y) = 0:f3 + O:'Y for every 0:, f3 and 'Y in F. The members of a field F are called scalars. Let Q be the set of all rational numbers, R the set of all real numbers, and C the set of all complex numbers. The sets Q, Rand C are standard examples of a field. The reader may verify the following from the properties of a field. P 1.1.5

If 0: + f3

= 0: + 'Y for 0:, f3 and 'Y in F, then f3 = 'Y.

Vector Spaces

P 1.1.6

5

(-1)0 = -0 for any a in F.

P 1.1.7 0 a = 0 for any a in F. P 1.1.8 If a # 0 and {3 are any two scalars, then there exists a unique scalar x such that ox = {3. In fact, x = 0-1{3, which we may also write as {3/ o. P 1.1.9 If 0{3 = 0 for some a and {3 in F, then at least one of a and {3 is zero. Another way of characterizing a field is that it is a commutative ring in which there is a unit element with respect to multiplication and any non-zero element has a multiplicative inverse. In the commutative ring K = {O, 1,2, 3} with addition and multiplication modulo 4, there are elements a and {3 none of which is zero and yet 0{3 = o. In a field, 0{3 = 0 implies that at least one of a and {3 is zero.

EXAMPLE 1.1.10. Let p be any positive integer. Let F = {O, 1,2, ... ,p - I}. Define addition in F by 0+ {3 = 0+ {3 (modulo p) for a and {3 in F. Define multiplication in F by 0{3 = 0{3 (modulo p) for a and {3 in F. More precisely, define addition and multiplication in F by a + {3 = a + {3 = a

+ {3 -

if a p

0{3 = 0{3

if a

+ {3 ~ p + {3 > p -

if 0{3

~

r

1;

p -1,

if 0{3 = rp

="Y

1,

2: 1 and 0

+ "y for some integers ~ "y ~

p - 1.

If p is a prime number, then F is a field. EXAMPLE 1.1.11. Let F = {O, 1,0,{3}, addition and multiplication on the elements of F be as in the following tables.

Multiplication table

Addition table

0 1 a {3

0

1

a

{3

0 1 a {3

1 0 {3 a

a {3 0 1

{3 a 1 0

0 1 a {3

0

1

a

{3

0 0 0

0 1 a {3

0 a {3 1

0 {3 1 a

°

6


The binary operations so defined above on F make F a field. Firute fields, i.e., fields consisting of a firllte number of elements are called Galois fields. One of the remarkable results of a Galois field is that the number of elements in any Galois field is pm for some prime number p and positive integer m. Example 1.1.10 is a description of the Galois field, GF(p), where p is a prime number. Example 1.1.11 is a description of the Galois field, GF(22 ). As one can see, the description of GF(p) with p being a prime number is easy to provide. But when it comes to describing G F(pm) with p being prime and m ~ 2, additional work is needed. Some methods for construction of such fields are developed in papers by Bose, Chowla, and Roo (1944, 1945a, 1945b). See also Mann (1949) for the use of GF(pm) in the construction of designs. Construction of orthogonal latin squares and magic squares are two of the benefits that accrue from a study of firllte fields. Let us start with some defirutions. DEFINITION 1.1.12. Let L1 and L2 be two latin squares each on a set of n symbols. They are said to be orthogonal if we superimpose one latin square upon the other, every ordered pair of symbols occurs exactly once in the composite square. The following are two latin squares, one on the set S1 = {S, H, D, C} and the other on the set S2 = {K, Q, J, A}.

L1

:

S C D

D H S

H D C

C S H

L 2·.Q J

H C S

D

K

A

K Q

J

Q

A K

K A

A

J

Q

J

The latin squares L1 and L2 are orthogonal. Way back in 1779, Leonard Euler posed the following famous problem. There are 36 officers of six different ranks with six officers from each rank. They also come from six different regiments with each regiment contributing six officers. Euler conjectured that it is impossible to arrange these officers in a 6 x 6 grid so that each row and each column contains one officer from each regiment and one from each rank. In terms of the notation introduced ab0're , can one build a latin square L1 on the set of regiments and a latin square L2 on the set of ranks such that L1 and L2 are orthogonal? By an exhaustive enumeration, it has been found that Euler was right. But if n > 6, one can always find a pair of orthogonal latin squares as shown

Vector Spaces

7

by Bose, Shrikhande and Parker (1960). In the example presented after Definition 1.1.3, the suits are the regiments, the kings, queens, jacks and aces are ranks, and n = 4. The problem of finding pairs of orthogonal latin squares has some statistical relevance. Suppose we want to compare the effect of some m dose levels of a drug, Drug A say, in combination with some m levels of another drug, Drug B say. Suppose we have m 2 experimental units classified according to two attributes C and D each at m levels. The attribute C, for example, might refer to m different age groups of experimental units and the attribute D might refer to m different social groups. The basic problem is how to assign the n = m 2 drug combinations to the experimental units in such a way that the drug combinations and the cross-classified experimental units constitute a pair of orthogonal latin squares. If such an arrangement is possible, it is called a graeco-Iatin square. As an illustration, consider the following example. Suppose Drug A is to be applied at two levels: High (At) and Low (A 2 ), and Drug B at two levels: High (Bd and Low (B2). The four drug combinations constitute the first set 8 1 of symbols, i.e.,

for which a latin square L1 is sought with n = 4. Suppose the attribute C has two age groups: C1(~ 40 years old) and C2 (> 40 years old), and D has two groups: Dl (White) and D2 (Black). The second latin square L2 is to be built on the set

Choosing L1 and L2 to be orthogonal confers a distinct statistical advantage. Comparisons can be made between the levels of each drug and attribute. The concept of orthogonality between a pair of latin squares can be extended to any finite number of latin squares.. DEFINITION 1.1.13. Let L 1 , L2, ... ,Lm be a set oflatin squares each of order n. The set is said to be mutually orthogonal if Li and L j are orthogonal for every i =1= j.

8


The construction of a set of mutually orthogonal latin squares is of statistical importance. Galois fields provide some help in this connection. Let GF(s) be a Galois field of order s. Using the Galois field, one can construct a set of s - 1 mutually orthogonal latin squares. Let G F( s) = {OO, 01, ... ,Os-I} with the understanding that 00 = o. P 1.1.14 Let Lr be the square grid in which the entry in the i-th row and j-th column is given by

for 1 ::; r ::; s - 1. Then L}, L2, ... ,Ls- 1 is a set of mutually orthogonal latin squares. PROOF. First, we show that each Lr is a latin square. We claim that any two entries in any row are distinct. Consider the i-th row, and p-th and q-th elements in it with p =F q. Look at

Consequently, no two entries in any row are identical. Consider now the j-th column, and p-th and q-th entries in it with p =F q. Look at

in view of the fact that r 2: 1 and Or =f:. O. Now, we need to show that Lr and Ls are orthogonal for any r =f:. sand r, s = 1,2, ... ,s - 1. Superimpose Lr upon L 8 • Suppose (oij(r),Oij(S)) = (opq(r),opq(s)) for some 0 ::; i, j ::; s-1 and 0 ::; p, q ::; s-1. Then 0rOi +OJ = orop+Oq and 0sOi + OJ = osop + Oq. By subtracting, we obtain

or, equivalently, (Or -

Os)(Oi - op) = O.

Since r =F s, we have 0i - op = 0, or i = p. We see immediately that j = q. This shows that Lr and La are orthogonal. This completes the proof.

Vector Spaces

9

Pairs of orthogonal latin squares are useful in drawing up schedules for competitions between teams. Suppose Teams A and B each consisting of 4, players want to organize chess matches between members of the teams. The following are to be fulfilled. (1) Every member of Team A plays every member of Team B. (2) All the sixteen matches should be scheduled over a span of four \ days with four matches per day. (3) Each player plays only one match on any day. (4) On every day, each team plays an equal number of games with white and black pieces. (5) Each player plays an equal number of games with white and black pieces. Drawing a 16-match schedule spread over four days fulfilling Conditions 1, 2, and 3 is not difficult. One could use a latin square on the set of days the games are to be played. The tricky part is to have the schedule fulfilling Conditions 4 and 5. A pair of orthogonal latin squares can be used to draw up a schedule of matches. Let Di stand for Day i, i = 1, 2, 3, 4. Let L1 and L2 be the pair of orthogonal latin squares on the sets

S1

= {Dl, D2, D3, D4}

and

S2

= {1,2, 3,4},

respectively, given by

L1 :

D2 D3 D4 D4 D3 D2 Dl D2 Dl D4 D3 D3 D4 Dl D2

L2 :

1 2 3 4 3 4 1 2 4 3 2 1 2 1 4 3

Dl

Replace even numbers in L2 by white (W), odd numbers by black (B) and then superimpose the latin squares. The resultant composition is given by


10

Team A/Team B

1 2 3 4

1

2

3

4

(Dl, B) (D4,B) (D2, W)

(D2, W)

(D3,B)

(D3, W) (Dl, B)

(D3, W) (Dl, W) (D3, B)

(D3, W)

(D4, B)

(D2,B) (D4, W) (Dl, W)

(D2,B)

The schedule of matches can be drawn up using the composite square. Day

Team A players

Dl

1 2 3 4

vs vs vs vs

2 3

1 2 3 4

vs vs vs vs

2 3 1 4

W

1

vs vs vs vs

3 2 4 1

B W

vs vs vs vs

4 1 3

D2

D3

2 3 4

D4

1

2 3 4

Team B players

Color of pieces by team A players

1

B

4

W B W

2

B W B

B W W B

W B

This schedule fulfills all the five requirements 1 to 5 stipulated above. A pair of orthogonal latin squares can also be used to build magic squares. Let us define formally what a magic square is.

11

Vector Spaces

DEFINITION 1.1.15. A magic square of order n is an n x n square grid consisting of the numbers 1, 2, ... , n 2 such that the entries in each row, each column and each of the two main diagonals sum up to the same number. We can determine what each row in a magic square sums up to. The sum of all integers from 1 to n 2 is n 2 (n 2 + 1)/2. Then each row in the magic square sums up to (l/n)n 2 (n 2 + 1)/2 = n(n2 + 1)/2. The following are magic squares of orders 3 and 4:

294 7 5 3 618

16

3

2

5

10

11

13 8

9 4

6 15

7 14

12 1

(from an engraving of Albrecht Diller entitled "Melancholia" (1514)). Many methods are available for the construction of magic squares. What we intend to do here is to show how a pair of orthogonal latin squares can be put to use to pullout a magic square. Let L1 = (il j ) and L2 = (ifj) be two orthogonal latin squares on the set {O, 1, 2, . .. , n - I}. Let M = (mij) be an n x n square grid in which the entry in the i-th row and j-th column is given by

for i,j = 1,2, ... , n. What can we say about the numbers mi/s? Since L1 and L2 are orthogonal, every ordered pair (i,j),i,j,= 0,1,2, ... , n - 1 occurs exactly once when we superimpose Ll upon L2. Consequently, each of the numbers 0, 1,2, ... , n 2 -1 will appear somewhere in the square grid M. We are almost there. Define a new grid M' = (m~j) of order n x n with mi j = mij + 1. Now each of the numbers 1,2, . . . , n 2 appears somewhere in the grid M'. P 1.1.16 In the grid M', each row and each column sums up to the same number.

12


PROOF . Since LI and

L2 are latin squares, for any i = 1, 2, . . , , n,

n

L

mij

= Sum of all entries in the i-th row of M

j=I

n

n

j=1

j=1

= n ~ (I . + ~ (2 , L.J 'J L.J 'J = n(O

+ 1 + .. . + n -

I)

+ (0 + 1 + ... + n -

= n(n - l}n/2 + (n - l}n/2 = n(n

2

-

I)

1}/2,

which is independent of i. In a similar vein, one can show that each column of M sums up to the same number n(n2 - 1}/2. Thus M' has the desired properties stipulated above. The grid M' we have obtained above is not quite a magic square. The diagonals of M' may not sum up to the same number. We need to select the latin squares LI and L2 carefully.

P 1.1.17 Let Ll and ~ be two orthogonal latin squares of order n each on the same set {O, 1,2, ... ,n - I}. Suppose that each of the two main diagonals of each of the latin squares Ll and L2 add up to the same number (n 2 - l}n/2. Then the grid M' constructed above is a magic square. PROOF . It is not hard to show that each of the diagonals of M sums

up to n(n2

-

1}/2. We now have M' truly a magic square.

EXAMPLE 1.1.18. In the following L} and L2 are two latin squares of order 5 on the set {O, 1,2,3, 4}. These latin squares satisfy all the conditions stipulated in Proposition 1.1.17. We follow the procedure outlined above. 01234 01234 2340 1 34012 Ll : 40 1 23 L 2 :12340 12340 40 123 34012 23401 o 6 12 1824 1 7 13 19 25 13 1921 1 7 142022 2 8 M: 21 2 8 14 15 M' : 22 3 9 15 16 9 10 1622 3 10 11 1723 4 1723 4 5 11 1824 5 6 12

Vector Space8

13

Note that M' is a Magic square of order 5. We will wind up this section by talking about sub-fields. A subset Fl of a field F is said to be a sub-field of F if Fl is a field in its own right under the same operations of addition and multiplication of F restricted to Fl. For example, the set Q of all rational numbers is a subfield of the set R of all real numbers. A field F is said to be algebraically closed if every polynomial equation with the coefficients belonging to F has at least one root belonging to the field. For example, the set C of all complex numbers is algebraically closed, whereas the set R of all real numbers is not algebraically closed.

Complements As has been pointed out in P 1.1.4, the multiplication table of a finite group provides a latin square. We do not need the full force of a group to generate a latin square. A weaker structure would do. Let G be a finite set with a binary operation. The set G is said to be a quasi group if each of the equations, oy = (3 and yo = (3 has a unique solution in y for every 0, (3 in G.

1.1.1 Show that the multiplication table of a quasigroup with n elements is a latin square of order n. 1.1.2 Show that every latin square of order n gives rise to a quasigroup. (li we look at the definition of a group, it is clear that if the binary operation of a quasi group G is associative, then G is a group.) 1.1.3 Let G = {O, 1, 2} be a set with the following multiplication table. 012

o 1 2

1 2 0 1 2 0

0 2 1

Show that G is a quasi group but not a group. 1.1.4 Let n be an integer 2 2 and G = {O, 1,2, ... ,n - I}. Define a binary operation * on G by o

* (3 = ao + b(3 + c (modulo n)

for all 0 and (3 in G, where a and b are prime to n . Show that G is a quasi group.

14


1.1.5 If L., L 2 , ••• ,Lm is a set of mutually orthogonal latin squares of order n, show that m ~ n - 1. Let X = {I, 2, ... ,n}, say, be a finite set. Let G be the collection of all subsets of X. Define a binary operation on G by o.{3 = o.6{3, a, {3 E G, where 6 is the set-theoretic operation of symmetric difference, i.e., o.6{3 = (a - {3) U ({3 - a), where 0.-

{3 = {x EX: x E a, x ~ {3}.

1.1.6 How many elements are there in G? 1.1.7 Show that G is a group. 1.1.8 Set out the multiplication table of the group G when n = 3. 1.1.9 Let F = {a + bv'2; a, b rational}. The addition and multiplication of elements in F are defined in the usual way. Show that F is a field. 1.1.10 Show that the set of all integers under the usual operations of addition and multiplication of numbers is not a field. 1.2. Mappings

In the subsequent discussion of vector spaces and matrices, we will be considering transformations or mappings from one set to another. We give some basic ideas for later reference. Let 8 and T be two sets. A map, a mapping, or a function 1 from 8 to T is a rule which associates to each element of 8 a unique element of T. If 8 is any element of 8, its associate in T is denoted by 1(8). The set 8 is called the domain of 1 and the set of all associates in T of elements of 8 is called the range of I. The range is denoted by 1(8). The map 1 is usually denoted by 1 : 8 --t T. Consider a map 1 : 8 --t T. The map 1 is said to be surjective or onto if 1(8) = T, i.e., given any t E T, there exists 8 E 8 such that I(s) = t. The map 1 is said to be injective or one-to-one if any two distinct elements of 8 have distinct associates in T, i.e., 81,82 E 8 and 1(81) = I(S2) imply that 81 = 82· The map f is said to be bijective if f

Vector Spaces

15

is one-to-one and onto or surjective and injective. If f is bijective, one can define the inverse map which we denote by f- 1 : T ---+ S; for t in T, f-l(t) = s, where s is such that f(s) = t. The map f- 1 is called the inverse of f.

f be a mapping from a group G 1 to a group is said to be a homomorphism if

DEFINITlON 1.2.1. Let

G 2 • Then

f

f(o{3) = f(o)f({3) If f is bijective, phic.

f

for every

0

and {3 in G 1 .

is said to be an isomorphism and Gland G 2 isomor-

Suppose G is a group and H an invariant subgroup of G, i.e., xH = Hx for all x E G. Let GIH be the quotient group of G modulo H, i.e., GIH is the collection of all distinct cosets of H. [Note that a coset of H is the set {xy : y E H} as defined in Section 1.1]. There is a natural map 71' from G to GIH. For every 0 in G, define 71'(0) = the coset of H to which 0 belongs. The map 71' is surjective and a homomorphism from G onto GIH. This map 71' is called the projection of G onto the quotient group GIH.

f be a mapping from a field Fl to a field F 2 • is said to be a homomorphism if

DEFINITION 1.2.2. Let

Then

f

f(o + (3) = f(o) + f({3) , f(o{3) = f(o)f({3) for every 0 and {3 in Fl. If f is bijective, then f is called an isomorphism and the fields Fl and F2 are called isomorphic.

Complements

1.2.1 Let Sand T be two finite sets consisting of the same number of elements. Let f : S ---+ T be a map. If f is surjective, show that f is bijective. 1.2.2 Let S = {1, 2, 3, 4} and G be the collection of all bijective maps from S to S. For any two maps f and 9 in G, define the composite map fog by (J 0 g)(x) = f(g(x)),x E S. Show that under the binary operation of composition of maps, G is a group. Let H be the collection of all maps f in G such that f(1) = 1. Show that H is a subgroup but not invariant. Identify all distinct left cosets of H. Is this a group under the usual multiplication of cosets?

16


1.3. Vector Spaces

The concept of a vector space is central in any discussion of multivariate methods. A set of elements (called vectors) is said to be a vector space or a linear space over a field of scalars F if the following axioms are satisfied. (We denote the set of elements by V(F) to indicate its dependence on the underlying field F of scalars. Sometimes, we denote the vector space simply by V if the underlying field of scalars is unambiguously clear. We denote the elements of the set V(F) by Roman letters and the elements of F by Greek letters.) (1) To every pair of vectors x and y, there corresponds a vector x + y in such a way that under the binary operation +, V(F) is an abelian group. (2) To every vector x and a scalar a, there corresponds a vector ax, called the scalar product of a and x, in such a way that a} (a2x) = (a}a2)x for every aJ, a2 in F and x in V(F), and Ix = x for every x in V(F), where 1 is the unit element of F. (3) The distributive laws hold for vectors as well as scalars, i.e., a(x + y) = ax + ay for every a in F and x, y in V(F), and (a} + a2)x = a}x + a2x for every a}, a2 in F and x in V(F). We now give some examples. The first example plays an important role in many applications. EXAMPLE 1.3.1. Let F be a field of scalars and k 2: 1 an integer. Consider the following collection of ordered tuples:

Define addition and scalar multiplication in Fk by

for every 8 in F and (a}, a2, ... ,ak) in Fk. It can be verified that Fk is a vector space over the field F with (0,0, ... ,0) as the zero-vector. We call Fk a k-dimensional coordinate space. Strictly speaking, we should

Vector Spaces

17

write the vector space Fk as Fk (F). We will omit the symbol in the parentheses, which will not cause any confusion. Special cases of Fk are Rk and C k , i.e., when F is the field R of real numbers and C of complex numbers, respectively. They are also called real and complex arithmetic spaces. EXAMPLE 1.3.2. Let n ~ 1. The collection of all polynomials of degree less than n with coefficients from a field F with the usual addition and scalar multiplication of polynomials is a vector space. Symbolically, we denote this collection by

Pn(F)(t)

= {ao + a1t + a2t2 + ... + an_1tn-1 : ai E F, i=O,1,2, ... ,n-I},

which is a vector space over the field F. The entity ao + a1t + a2t2 + ... + a n -1 t n - 1 is called a polynomial in t with coefficients from the field

F. EXAMPLE 1.3.3. Let V be the collection of all real valued functions of a real variable which are differentiable. If we take F = R, and define sum of two functions in V and scalar multiplication in the usual way, then V is a vector space over the field R of real numbers. EXAMPLE 1.3.4. Let V = {(a,,B) : a > 0 and f3 addition and scalar multiplication in V as follows.

(1) (aI, f3.)

+ (a2' (32) =

> O}.

Define vector

(a1 a 2, f31(32) for every (aI, (31) and (a2, (32)

inV.

(2) 8(a,f3) = (a 6,f36) for every 8 in Rand (a,f3) in V. Then V is a vector space over the field R of real numbers. EXAMPLE 1.3.5. Let p be an odd integer. Let V = {(a, (3) : a and f3 real}. Define vector addition and scalar multiplication in Vas below: (1) (aI, f3.)+(a2, (32) = ((af+a~)l/p,(f3f+f3~)l/P) for every (a1,f3.) and (a2' f32) in V. (2) 8(a, (3) = (8 1/ Pa, 81/ p(3) for every 8 in R and (a, (3) in V.

Then V is a vector space over the field R of real numbers. This statement is not correct if p is an even integer.

18


EXAMPLE 1.3.6. Let F = {a, 1, 2}. With addition and multiplication modulo 3, F is a field. See Example 1.1.10. Observe that the vector space Fk has only 3 k elements, while R k has an uncountable number of elements. The notion of isomorphic vector spaces will be introduced now. Let V I and V 2 be two vector spaces over the same field F of scalars. The spaces V I and V 2 are said to be isomorphic to each other if there exists a bijection h : V I ---+ V 2 such that

h(x + y) = h(x)

+ h(y)

h(ax) = ah(x)

for all a E F and x E VI .

for all x, y E VI,

Complements

1.3.1 Examine which of the following are vector spaces over the field C of complex numbers. Explain why or why not? (1) V Addition:

= {(a,;3); a E R, f3 E C}.

Scalar multiplication:

8(a,f3) = (8a,8f3), 8 E C,(a,f3) E V.

(2) V = {(a,f3): a +f3 = O,a,f3 E C}. Addition:


8(a, f3) = (8a, 8f3) , 8 E C, (a, f3)

E V.

1.3.2 Let V I = (0,00) and F = R. The addition in VIis the usual operation of multiplication of real numbers. The scalar multiplication is defined by ax = xO:, a E R, x E VI.

19

Vector Spaces

Show that VIis a vector space over R . Identify the zero vector of VI. 1.3.3 Show that VI of Complement 1.3.2 and the vector space V 2 = R over the field R of real numbers are isomorphic. Exhibit an explicit isomorphism between VIand V 2. 1.3.4 Let V(F) be a vector space over a field F. Let, for any fixed positive integer n,

Define addition in Vn(F) by

for (X},X2,'" ,xn ), (Y},Y2,'" tion in Vn(F) by 0(XI,X2' •••

,Yn)

E vn(F). Define scalar multiplica-

,xn ) = (OXl,OX2, ... ,oxn ), 0 E F (X},X2,'"

and

,xn ) E Vn(F).

Show that Vn(F) is a vector space over the field F. 1.4. Linear Independence and Basis of a Vector Space

Through out this section, we assume that we have a vector space V over a field F of scalars. The notions of linear independence, linear dependence and basis form the core in the development of vector spaces. DEFINITION 1.4.1. A finite set X}, X2, ... ,Xk of vectors is said to

be linearly dependent if there exist scalars O}, 02, ... ,Ok, not all zeros, such that 0IX} + 02X2 + ... + 0kXk = O. Otherwise, it is said to be linearly independent. P 1.4.2 The set consisting of only one vector, which is the zero vector 0, is linearly dependent. P 1.4.3 The set consisting of only one vector, which is a non-zero vector, is linearly independent. P 1.4.4 dependent.

Any set of vectors containing the zero vector is linearly


20

P 1.4.5 A set Xl, X2, .. • ,Xk of non-zero vectors is linearly dependent if and only if there exists 2 ~ i ~ k such that

for some scalars f3l, f32, ... ,f3i-I, i.e., there is a member in the set which can be expressed as a linear combination of its predecessors. Let i E {I, 2, ... ,k} be the smallest integer such that the set of vectors Xl, X2, ••. ,Xi is linearly dependent. Obviously, 2 ~ i ~ k. There exist scalars Ql, Q2, ... ,Qi, not all zero, such that Ql Xl + Q2 X 2 + ... + QiXi = o. By the very choice of i, Qi =1= o. Thus we can write PROOF.

P 1.4.6 Let A and B be two finite sets of vectors such that A c B. If A is linearly dependent, so is B. If B is linearly independent, so is A. DEFlNlTlON 1.4.7. Let B be any subset (finite or infinite) of V. The set B is said to be linearly independent if every finite subset of B is linearly independent. DEFlNlTlON 1.4.8. (Basis of a vector space) A linearly independent set B of vectors is said to be a (Hamel) basis of V if every vector of V is a linear combination of the vectors in B. The vector space V is said to be finite dimensional if there exists a Hamel basis B consisting of finitely many vectors.

It is not clear at the outset whether a vector space possesses a basis. Using Zorn's lemma, one can demonstrate the existence of a maximal linearly independent system of vectors in any vector space. (A discussion of this particular feature is beyond the scope of the book.) Any maximal set is indeed a basis of the vector space. From now on, we will be concerned with finite dimensional vector spaces only. Occasionally, infinite dimensional vector spaces will be presented as examples to highlight some special features of finite dimensional vector spaces. The following results play an important role. P 1.4.9 If XI, X2, ••• ,Xk and Yl, Y2,· the vector space V, then k = s.

•• ,Ys

are two sets of bases for

Vector Spaces

21

PROOF. Suppose k =1= s. Let s > k. It is obvious that the set Yl! Xl! X2, ••• , Xk is linearly dependent. By P 1.4.5, there is a vector Xi which is a linear combination of its predecessors in the above set. Consequently, every vector in V is a linear combination of the vectors y},X}'X2, ••• ,Xi-},Xi+J, ••. ,Xk. ObservenowthatthesetY2,YI,XI,X2, ••• , Xi-I, Xi+!, ••• , Xk is linearly dependent. Again by P 1.4.5, there exists a j E {I, 2, ... , i-I, i + 1, ... , k} such that Xj is a linear combination of its predecessors. (Why?) Assume, without loss of generality, that i < j. It is clear that every vector in V is a linear combination ofthevectorsY2,y),x),x2,.·· ,Xi-),Xi+l, ... ,Xj_I,Xj+), ... ,Xk. Continuing this process, we will eventually obtain the set Yk, Yk-l, .. . , Y2, YI such that every vector in V is a linear combination of members of this set. This is a contradiction to the assumption that s > k. Even if we assume that s < k, we end up with a contradiction. Hence s = k. In finite dimensional vector spaces, one can now introduce the notion of the dimension of a vector space. It is precisely the cardinality of any Hamel basis of the vector space. We use the symbol dim(V) for the dimension of the vector space V.

P 1.4.10 Any given set Xl, X2, ... tors can be enlarged to a basis of V.

, Xr

of linearly independent vec-

Let YI , Y2, • •• , Y k be a basis of V, and consider the set X I, YI , Y2, ... , Yk of vectors, which is linearly dependent. Using the same method as enunciated in the proof of P 1.4.9, we drop one of y/s and then add one of xi's until we get a set X r ,Xr -1!.·· ,Xl'Y(I)'Y(2)'···' Y(k-r), which is a basis for V, where Y(i)'S are selections from Y1! Y2,· .. , Yk. This completes the proof. PROOF.

P 1.4.11 Every vector of any given basis of V. PROOF.

Let

X

x),x2, ... ,Xk

in V has a unique representation in terms be a basis for V. Let

and, also X

for some scalars

Qi'S

= {3IXI + {32 x 2 + ... + {3k x k,

and {3j's. Then

22


from which it follows that ai - f3i = 0 for every i, in view of the fact that the basis is linearly independent.

In view of the unique representation presented above, one can define a map from V to Fk. Let XI,X2, ... ,Xk be a basis for V. Let x E V, and x = aIxI + a2x2 + ... + akXk be the unique representation of x in terms of the vectors of the basis. The ordered tuple (aI, a2, ... , ak) is called the set of coordinates of x with respect to the given basis. Define

which one can verify to be a bijective map from V to Fk. Further, r.p(-) is a homomorphism from the vector space V to the vector space Fk. Consequently, the vector spaces V and Fk are isomorphic. We record this fact as a special property below. P 1.4.12 Any vector space V(F) of dimension k is isomorphic to the vector space Fk. The above result also implies that any two vector spaces over the same field of scalars and of the same dimension are isomorphic to each other. It is time to take stock of the complete meaning and significance of P 1.4.12. If a vector space V over a field F is isomorphic to the vector space Fk for some k ~ 1, why bother to study vector spaces in the generality they are introduced? The vector space Fk is simple to visualize and one could restrict oneself to the vector space Fk in subsequent dealings. There are two main reasons against pursuing such a seemingly simple trajectory. The isomorphism that is built between the vector spaces V(F) and Fk is based on a given basis of the vector space V(F). In the process of transformation, the intrinsic structural beauty of the space V(F) is usually lost in its metamorphism. For the second reason, suppose we establish a certain property of the vector space Fk. If we would like to examine how this property comports itself in the space V(F), we could use anyone of the isomorphisms operational between V(F) and Fk, and translate this property into the space V(F). The isomorphism used is heavily laced with the underlying basis and an understanding of the property devoid of its external trappings provided by the isomorphism would then become a herculean task.

Vector Spaces

23

As a case in point, take F = R and V = Pk, the set of all polynomials with real coefficients of degree < k. The vector space P k is isomorphic to R k • Linear functionals on vector spaces are introduced in Section 1.7. One could introduce a linear functional f on P k as follows. Let JL be a measure on the Borel 17-field of [a, bj, a non-degenerate interval. For x E Pk, let

f(x) =

lb

x(t)JL(dt).

One can verify that

f(x

+ y) = f(x) + f(y), x, Y E P k

and

f(ax) = af(x),

a E

R, x E P k •

Two distinct measures JLI and JL2 on [a, bj might produce the same linear functional. For example, if

for m

= 0,1,2, ...

, k - 1, then

h(x) =

rb

Ja

x(t)JLI(dt) =

1b

x(t)JL2(dt) = h(x)

for all x E P k . A discussion of features such as this in Pk is not possible in Rk. The vector space P k has a number facets allied to it which would be lost if we were to work only with R k using some isomorphism between Pk and Rk. We will work with vector spaces as they come and ignore P 1.4.12.

Complements. 1.4.1 Let V = C, the set of all complex numbers. Then V is a vector space over the field C of complex numbers with the usual addition and multiplication of complex numbers. What is the dimension of the vector space V? 1.4.2 Let V = C, the set of all complex numbers. Then V is a vector space over the field R of real numbers with the usual addition of complex

24


numbers. The scalar multiplication in V is the usual multiplication of a complex number by a real number. What is the dimension of V? How does this example differ from the one in Complement 1.4.1? 1.4.3 Let V = R, the set of all real numbers. Then V is a vector space over the field Q of rational numbers. The addition in V is the usual addition of real numbers. The scalar multiplication in V is the multiplication of a real number by a rational number. What is the dimension of V? 1.4.4 Let R be the vector space over the field Q of rational numbers. See Complement 1.4.3. Show that J2 and v'3 are linearly independent. 1.4.5 Determine the dimension of the vector space introduced in Example 1.3.4. Identify a basis of this vector space. 1.4.6 Let F = {O, 1,2,3,4} be the field in which addition and multiplication are carried out in the usual way but modulo 5. How many points are there in the vector space F3? 1.5. Subspaces In any set with a mathematical structure on it, subsets which exhibit all the features of the original mathematical structure deserve special scrutiny. A study of such subsets aids a good understanding of the mathematical structure itself. DEFINITION 1.5.1. A subset S of a vector space V is said to be a subspace of V if ax + f3y E S whenever x, yES and a, f3 E F.

P 1.5.2 A subspace S of a vector space V is a vector space over the same field F of scalars under the same definition of addition of vectors and scalar multiplication operational in V. Further, dim(S) ::; dim(V). PROOF. It is clear that S is a vector space in its own right. In order to show that dim(S) ::; dim(V), it suffices to show that the vector space S admits a basis. For then, any basis of S is a linearly independent set in V which can be extended to a basis of V. It is known that every vector space admits a basis.

If S consists of only the zero-vector, then S is a zero-dimensional subspace of V. If every vector in S is of the form ax for some fixed non-zero vector x and for some a in F, then S is a one-dimensional subspace of V. If every vector in S is of the form aXl + f3x2 for some

25

Vector Spaces

fixed set of linearly independent vectors Xl and X2 and for some 0: and f3 in F, then S is a tw~dimensional subspace of V . The schematic way we have described above is the way one generally obtains subspaces of various dimensions. The sets {O} and V are extreme examples of subspaces of V. P 1.5.3 The intersection of any family of subspaces of V is a subspace of V. P 1.5.4 Given an r-dimensional subspace S of V, we can find a basis XI,X2, ••• ,Xr ,Xr +I,X r +2, ... ,Xk of V such that XI,X2, •.. ,Xr is a basis of S. The result of P 1.5.4 can also be restated as follows: given a basis Xt, X2, .. . ,Xr of S, it can be completed to a basis of V. The subspaces spanned by a finite set of vectors need special attention. If Xl, X2, ••• ,X r is a finite collection of vectors from a vector space V(F), then the set

is a subspace of V(F). This subspace is called the span of Xl, X2, ... ,Xr and is denoted by Sp(Xt,X2, ... ,xr ). Of course, any subspace of V(F) arises this way. The concept of spanning plays a crucial role in the following properties. P 1.5.5 Given a subspace S of V, we can find a subspace sc of V such that S n SC = {O}, dim(S) + dim(SC) = dim(V), and V

= S EB SC = {x + y: XES, Y ESC}.

Further, any vector X in V has a unique decomposition with Xl E Sand X2 ESc.

X

= Xl

+ X2

PROOF. Let XI,x2, .•• ,XnXr+I, ... ,Xk constitute a basis for the vector space V such that Xl, X2, .. . ,Xr is a basis for S. Let SC be the subspace of V spanned by Xr+b X r +2, ••• ,Xk. The subspace sc meets all the properties mentioned above.

We have introduced a special symbol EB above. The mathematical operation S EB sc is read as the direct sum of the subspaces Sand Sc.

26


The above result states that the vector space V is the direct sum of two disjoint subspaces of V. We use the phrase that the subspaces 8 and 8 c are disjoint even though they have the zero vector in common! We would like to emphasize that the subspace 8 c is not unique. Suppose V = R2 and 8 = {(X, 0) : x E R}. One can take 8 c = {(x,x) : x E R} or 8 c = {(x,2x) : x E R}. We will introduce a special phraseology to describe the subspace SC : 8 c is a complement of 8. More formally, two subspaces 8 1 and S2 are complement to each other if 8 1 n 82 = {a} and {x+y: xE8 1 ,yE8 2 }=V. P 1.5.6 Let K = {Xl, X2, . .. ,xr } be a subset of the vector space V and Sp(K) be the vector space spanned by the vectors in K, i.e., Sp(K) is the space of all linear combinations of the vectors Xl, X2, . . . ,Xr . Then

where the intersection is taken over all subspaces 8 v of V containing K. Let 8 1 and 8 2 be two subspaces of a vector space V. Let

The operation + defined between subspaces of V is analogous to the operation of direct sum 61. We reserve the symbol 61 for subspaces 8 1 and 8 2 which are disjoint, i.e., 8 1 n 8 2 = {a}. The following results give some properties of the operation + defined for subspaces. P 1.5.7 Let 8 1 and 8 2 be two subspaces of a vector space V. Let 8 be the smallest subspace of V containing both 8 1 and 8 2 . Then (1) S = 8 1 + 82, (2) dim(S) = dim(8 1 )

+ dim(S2) that 8 1 + 8 2 S

dim(Sl

n 8 2),

PROOF. It is clear 8. Note that 8 1 + 8 2 is a subspace of V containing both 8 1 and 8 2. Consequently, 8 S 8 1 + S2. This establishes (1). To prove (2), let Xl,X2, ..• ,X r be a basis for 8 1 n 8 2, where r = dim(SI n 8 2). Let Xl, X2, .•. ,Xr , X r +l, X r +2, .. . ,Xm be the completion of the basis of 8 1 n S2 to 8 1 , where dim(8 l ) = m. Refer to P 1.5.4. Let x}, X2, ' " ,XT) Yr+l> Yr+2,. " ,Yn be the completion of the basis of 8 1 n 8 2 to S2, where dim(8 2) = n. It now

27

Vector Spaces

follows that a basis of 8 1 + 8 2 is given by XI, x2, .• • X m , Yr+lt Yr+2,'" ,Yn' (Why?) Consequently, dim(8 l

,X r , X r + 1, X r +2, ... ,

+ 82) = r + (m - r) + (n - r) =m+n-r = dim(8d + dim(8 2) - dim(8 l n 8 2).

P 1.5.8 Let 8 1 and 82 be two subspaces of V. Then the following statements are equivalent. (1) Every vector X in V has a unique representation Xl E 8 1 and X2 E 8 2. (2) 8 1 n 8 2 = {O}. (3) dim(8 1 ) + dim(82) = dim(V).

Xl

+ X2

with

Complements. 1.5.1 Let x, Y and z be three vectors in a vector space V satisfying + Y + z = 0. Show that the subspaces of V spanned by X and y and by X and z are identical. 1.5.2 Show that the subspace 8 = {O} of a vector space V has a unique complement. 1.5.3 Consider the vector space R3. The vectors (1,0,0), (0, 1,0) generate a subspace of R3, say 8. Show that 8{(0, 0, I)} and 8{(1, 1, I)} are two possible complementary one-dimensional subspaces of 8. Show that, in general, the choice of a complementary subspace 8 c of 8 c V is not unique. 1.5.4 Let 8 1 and 8 2 be the subspaces of the vector space R3 spanned by {(I, 0, 0), (0,0, I)} and {(O, 1, 1), (1,2, 3)}, respectively. Find a basis for each of the subspaces 8 1 n 8 2 and 8 1 + 8 2. 1.5.5 Let F = {O, 1, 2} with addition and multiplication defined modulo 3. Let 8 be the subspace of F3 spanned by (0,1,2) and (1,1,2). Identify a complement of 8. 1.5.6 Let F = {O, 1, 2} with addition and multiplication modulo 3. Make a complete list of all subspaces of the vector space F3. Count how many subspaces are there for each of the dimensions 1,2, and 3. 1.5.7 Show that the dimension of the subspace of R6 spanned by the X

28


following row vectors is 4. 1 1 1 1 1 1

0 0 0 1 1 1

1 1 1 0 0 0

0 1 0 0 1 0

1 0 0 1 0 0

0 0 1 0 0 1

1.5.8 Consider pq row vectors each consisting of p + q + 1 entries arranged in q blocks of p rows each in the following way. The last p columns in each block have the same structure with ones in the diagonal and zeros elsewhere.

Block 1

Block 2

Block q

{!

{! {j

1 0 1 0

0 0

1 0 0 1

0 0

1

0

0

0

0

1

0 0

1 1

0 0

1 0

0 1

0 0

0

1

0

0

0

1

o o

0 0

1 1

o

1 0

o

1

o

o

0

1

o

0

1

Show that the subspace of RP+q+l spanned by the row vectors is of dimension p + q - l. 1.5.9 If pq numbers aij, i = 1,2, ... ,p; j = 1,2, ... ,q are such that the tetra difference

Vector Spaces

29

for all i, j, r, and s, show that

for all i and j for some suitably chosen numbers aI, a2, ... ,ap and bl , b2 , •• • ,bq • (Complements 1.5.7-1.5.9 are applied in the analysis of variance of two-way-dassified data in statistics.)

1.6. Linear Equations Let Xl, X2, ... ,Xm be fixed vectors in any vector space V(F}. Consider the following homogeneous linear equation, (1.6.1) with f3i's in F. The word "homogeneous" refers to the vector 0 that appears on the right hand side of the equality (1.6.1). If we have a non-zero vector, the equation is called non-homogeneous. The basic goal in this section is to determine f3i's satisfying equation (1.6.1). Let b = (f3l,/32, ... ,13m) be a generic symbol which is a solution of (1.6.1). The entity b can be regarded as a vector in the vector space Fm. Let S be the collection of all such vectors b satisfying equation (1.6.1). We will establish some properties of the set S. Some comments are in order before we spell out the properties of S. The vector (0,0, ... ,O) is always a member of S. The equation (1.6.1) is intimately related to the notion of linear dependence or independence of the vectors Xl, X2, ... ,Xm in V (F). If Xl> X2, ... ,Xm are linearly independent, f31 = 0,132 = 0, ... ,13m = 0 is the only solution of (1.6.1). The set S has only one vector. If Xl, X2,' .. ,Xm are linearly dependent, the set S has more than one vector of Fm. The objective is to explore the nature of the set S. Another point of inquiry is why one is confined to only one equation in (1.6.1). The case of more than one equation can be handled in an analogous manner. Suppose Xl, X2,'" ,Xm and YI, Y2, ... ,Ym are two sets of vectors in V (F). Suppose we are interested in solving the equations

f3lYI

+ f32Y2 + ... + f3mYm

= 0


30

in unknown 131, /32, ... as a single equation

,13m

in F. These two equations can be rewritten

with (x}, yd, (X2' Y2), ... ,(xm , Ym) E y2(F). The treatment can now proceed in exactly the same way as for the equation (1.6.1).

P 1.6.1

S is a subspace of Fm.

P 1.6.2 Let Y 1 be the vector subspace of Y spanned by Then dimeS) = m - dim(Yd.

x},

X2,

••• ,Xm •

PROOF. If each Xi = 0, then it is obvious that S = Fm, dimeS) = m, and dim(Y l ) = 0. Consequently, dimeS) = m - dim(Y l ). Assume that there exists at least one Xi =I 0. Assume, without loss of generality, that Xl, X2,." ,Xr are linearly independent and each of Xr+l, Xr+2,'" ,Xm is a linear combination of x}, X2, ... ,Xr . This implies that dim(Y l) = r. Accordingly, we can write

(1.6.2) for each j = r vectors, bl

+ 1, r + 2, ... ,m

and for some f3j,s'S in F. Then the

= (f3r+l,l,f3r+l,2,'"

~ = (f3r+2,l,f3r+2,2,'"

bm- r

= (f3m,l,

f3m,2,""

,f3r+l,r, - 1, 0, ... , 0), ,f3r+2,r, f3m,r,

0, -1, ... , 0),

0,

0, ... ,-1),

(1.6.3)

are all linearly independent (why?) and satisfy equation (1.6.1). If we can show that the collection of vectors in (1.6.3) spans all solutions, then it follows that they form a basis for the vector space S, and consequently, dimeS)

If b = (f3}, /32, ...

,13m)

=m

- r

=m

- dim(Y I ).

is any solution of (1.6.1), one can verify that

Vector Spaces

31

i.e., b is a linear combination of bl, b2 , ••• ,bm - r . Use the fact that Xl! X2, •• • ,X r are linearly independent and equation (1.6.2). This completes the proof. A companion to the linear homogeneous equation (1.6.1) is the socalled non-homogeneous equation,

for some known vector y =I O. Note that while a homogeneous equation {1.6.1} always has a solution, namely, the null vector in Fm, a nonhomogeneous equation may not have a solution. Such an equation is said to be inconsistent. For example, let Xl = (I, 1, 1), X2 = {I, 0, I} and X3 = {2, 1, 2} be three vectors in the vector space R 3 {R). Then the non-homogeneous equation,

has no solution. P 1.6.3

The non-homogeneous equation, (1.6.4)

admits a solution if and only if y is dependent on

Xl, X2, • .. ,Xm ·

The property mentioned above is a reformulation of the notion of dependence of vectors. We now identify the set of solutions {1.6.4} if it admits at least one solution. If {1.6.4} admits a solution, we will use the phrase that {1.6.4} is consistent. P 1.6.4 Assume that equation {1.6.4} has a solution. Let bo = (f31,/h . . . ,13m) be any particular solution of (1.6.4). Let 8 1 be the set of all solutions of {1.6.4}. Then

8 1 = {bo

+ b : b E 8},

{1.6.5}

where 8 is the set of all solutions of the homogeneous equation {1.6.1}. It is clear that for any b E 8, bo + b is a solution of {1.6.4}. Conversely, if c is a solution of {1.6.4}, we can write c = bo + {c - bo}. Note that c - bo E 8. PROOF.

32


Note that the consistent non-homogeneous equation (1.6.4) admits a unique solution if and only if the subspace S contains only one vector, namely, the zero vector. Equivalent conditions are that dim(S) = 0 = m - dim(Vt} or Xl, X2, . .. , Xm are linearly independent. A special and important case of the linear equation (1.6.4) arises when Xl, X2, ... , Xm belong to the vector space V (F) = Fk, for some k ~ 1. If we write Xi = (Xli, X2i, ..• , Xki) for i = 1,2, ... , m, with each Xji E F, and Y = (YI, Y2, • .. , Yk) with each Yi E F, then the linear equation (1.6.4) can be rewritten in the form, Xll{31 +XI2{32 x21{31

+ .. . +Xl m {3m

= YI,

+ x22{32 + ... + x2m{3m = Y2, (1.6.6)

which is a system of k simultaneous linear equations in m unknowns {3t.{32, ... ,(3m. Associated with the system (1.6.6), we introduce the following vectors: Ui=(Xil,Xi2, ... ,Xim), i=1,2, ... Vi

= (X i l,Xi2, ...

,Xim,Yi),

i

,k,

= 1,2, ...

,k.

For reasons that will be clear when we take up the subject of matrices, we call Xl, X2, ... , Xm and y as column vectors, and UI, U2,· .. , Uk, VI, V2, .•• , Vk as row vectors. The following results have special bearing on the system (1.6.6) of equations.

P 1.6.5 The maximal number, g, of linearly independent column vectors among Xl, X2, ... , Xm is the same as the maximal number, s, of linearly independent row vectors among UI, U2, ... , Uk. PROOF. The vector Y has no bearing on the property enunciated above. Assume that each Yi = o. If we arrange mk elements from F in the form of a rectangular grid consisting of k rows and m columns, each row can be viewed as a vector in the vector space Fm and each column can be viewed as a vector in the vector space Fk. The property under discussion is concerned about the maximal number of linearly independent rows and of independent columns. We proceed with the

Vector Spaces

33

proof as follows. The case that every Ui = 0 can be handled easily. Assume that there is at least one vector Ui i= O. Assume, without loss of generality, that 'ILl, U2, .•. ,us are linearly independent and each Uj, for j = s + 1, s + 2, ... , k, is a linear combination of UI, U2, •.. ,Us' Consider the subsystem of equations (1.6.6) with y/s taken as zeros consisting of the first s equations Xil{31

+ xi2f32 + ... + xim{3m =

0, i = 1, ... , s.

(1.6.7)

Let S be the collection of all solutions of (1.6.6) and S* that of (1.6.7). It is clear that S = S*. Let V I be the vector space spanned by Xl, X2, •.. , X m . Let dim(V)) = g. By P 1.6.2, dim(S) = m-dim(V)) = m - g. The reduced system of equations (1.6.7) can be rewritten in the fonnat of (1.6.1) as

with xi, x2' ... ,x~, now, in F S • Let Vi be the subspace of F S spanned by xi, x2, ... , x~. (Observe that the components of each xi are precisely the first s components of xd Consequently, dim(Vi) ::; dim(F S ) = s. By P 1.6.2, dim(S*) = m - dim(Vi) ~ m - s, which implies that m - 9 ~ m - s, or, 9 ::; s. By interchanging the roles of rows and columns, we would obtain the inequality s ::; g. Hence s = g. The above result can be paraphrased from an abstract point of view. Let the components x), X2, ... ,X m be arranged in the form of a rectangular grid consisting of k rows and m columns so that the entries in the i-th column are precisely the entries of Xi. We have labelled the rows of the rectangular grid by UI, U2, ... ,Uk. The above result establishes that the maximal number of linearly independent vectors among Xl, X2, .• . , Xm is precisely the maximal number of linearly independent vectors among U), U2, ••. , Uk. We can stretch this analogy a little further. The type of relationship that exists between x), X2,' .. , Xm and UI, U2, ... , Uk is precisely the same that exists between Xl, X2, ... , X m , Y and Vb V2, ... , Vk. Consequently, the maximal number of linearly independent vectors among x I, X2, . .. , X m , Y is the same as the maximal number of linearly independent vectors among VI, V2, ... , Vk. This provides a useful characterization of consistency of a system of nonhomogeneous linear equations.


34

P 1.6.6 A necessary and sufficient condition that the non-homogeneous system (1.6.6) of equations has a solution is that the maximal number, g, of linearly independent vectors among UI, U2, • .. ,Uk is the same as the maximal number, h, of linearly independent vectors among the augmented vectors VI, V2,··. ,Vk·

(1.6.6) admit a solution if and only if the maximal number of linearly independent vectors among Xl, X2,'" ,Xm is the same as the maximal number of linearly independent vectors among Xl, X2, ••• ,Xm , y. But the maximal number of linearly independent vectors among Xl , X2, •.• ,Xm , Y is the same as the maximal number of linearly independent vectors among VI,V2,'" ,Vk. Consequently, a solution exists for (1.6.6) if and only if 9 = s = h. PROOF. By P 1.6.3, equations

The systems of equations described in (1.6.6) arises in many areas of scientific pursuit. One of the pressing needs is to devise a criterion whose verification guarantees a solution to the system. One might argue that P 1.6.5 and P 1.6.6 do provide criteria for the consistency of the system. But these criteria are hard to verify. The following proposition provides a necessary and sufficient condition for the consistency of the system (1.6.6). At a first glance, the condition may look very artificial. But time and again, this is the condition that becomes easily verifiable to check on the consistency of the system (1.6.6). P 1.6.7 The system (1.6.6) of non-homogeneous linear equations admits a solution if and only if

whenever

( 1.6.8)

(1.6.6) admits a solution. Suppose = 0 for some el, e2, ... ,ek in F. Multiply the i-th equation of (1.6.6) by ei and then sum over i. It now follows that elYI + e2Y2 + ... + ekYk = O. Conversely, view PROOF. Suppose the system

el UI

+ e2U2 + ... + ekUk

Vector Spaces

as a system of homogeneous linear equations in k unknowns 10 1, 102, • •• Consider the system of homogeneous linear equations CIVI

35 ,10k .

+ C2V2 + ... + CkVk = 0

in k unknowns 101, 102, ••• ,10k. By (1.6.8), these two systems of equations have the same set of solutions. The dimensions of the spaces of solutions are k - sand k - h, respectively. Thus we have k - s = k - h, or s = h. By P 1.6.6, the system (1.6.6) has a solution.

Complements 1.6.1 Let Q be the field of rational numbers. Consider the system of equations. 2{31

+ {33 -

{34 = 0

{32 - 2{33 - 3{34 = 0

in unknown {31, /32, {33, {34 E Q. Determine the dimension of the solution subspace S of Q4. Show that 2{31

+ {33 -

{34 = Yl

{32 - 2{33 - 3{34

= Y2

admit a solution for every Yl and Y2 in Q. 1.6.2 Consider the system (1.6.6) of equations with Yl = Y2 ~ = ... Yk = O. Show that the system has a non-trivial solution if k < m.

=

1.7. Dual Space One way to understand the intricate structure of a vector space is to pursue the linear functionals defined on the vector space. The duality that reigns between the vector space and its space of linear functionals aids and reveals what lies inside a vector space. DEFINITION 1.7.1. A function! defined on a vector space V(F} taking values in F is said to be a linear functional if

!(alxl + a2 x 2) = at/(xt} + a2/(x2} for every

Xl, X2

in V(F} and

aI, a2

in F.

One can view the field F as a vector space over the field F itself. Under this scenario, a linear functional is simply a homomorphism from the vector space V(F} to the vector space F(F}.


36

1.7.2. Consider the vector space Rn. Let aI , a2, .. . , an be fixed real numbers. For x = (6,6, ... , ~n) ERn. let EXAMPLE

The map I is a linear functional. If ai = 1 and aj = 0 for j i= i for some fixed 1 ::; i ::; n, then the map I is called the i-th co-ordinate functional. EXAMPLE 1.7.3. Let P n be the collection of all polynomials xC) of degree < n with coefficients in the field C of complex numbers. We have seen that P n is a vector space over the field C. Let aO be any complexvalued integrable function defined on a finite interval [a, bJ. Then for x( ·) in P n , let

I(x) Then

=

lb

a(t)x(t) dt.

I is a linear functional on P n.

It is time to introduce the notion of a dual space. Later, we will also determine the structure of a linear functional on a finite dimensional vector space.

1.7.4. Let V(F) be any vector space and V' the space of all linear functionals defined on V(F). Let us denote by 0 the linear functional which assigns the value zero of F for every element in V(F). The set Viis called the dual space of V(F). DEFINITION

We will now equip the space V' with a structure so that it becomes a vector space over the field F. Let II. h E V' and aI, a2 E F. Then the function I defined by

I(x)

= al/I(x) + a2h(x) ,

x

E

V(F)

is clearly a linear functional on V(F). We denote the functional I by adl + a2l2 . This basic operation includes, in its wake, the binary operation of addition and scalar multiplication on V' by the elements of the field F. Under these operations of addition and scalar multiplication, V' becomes a vector space over the field F.

Vector Spaces

37

P 1.7.5 Let X},X2, ... ,Xk be a basis of a finite dimensional vector space V(F}. Let at, 02, ... , Ok be a given set of scalars from F. Then there exists one and only one linear functional f on V(F} such that

PROOF. Any vector x in

V(F} has a unique representation x =

{tXt +6X2 + ... + {kXk for some scalars any linear functional on V(F}, then

6,6, ... ,{k in F.

If f is

which means that the value f(x} is uniquely determined by the values of f at XI, X2,· .. , Xk. The function f defined by

for X = {tXt + 6X2 + ... + {kXk E V(F} is clearly a linear functional satisfying f(Xi} = 0i for each i. Thus the existence and uniqueness follow.

P 1.7.6 Let Xt,X2, ... ,Xk be a basis of a finite dimensional vector space V. Then there exists a unique set It, 12, ... ,fk of linear functionals in Vi such that if z =], if i

=1=

j,

(1.7.1)

and these functionals form a basis for the vector space V'. Consequently,

dim(V}

= dim(V/}.

PROOF. From P 1.7.5, the existence of k linear functionals satis-

fying (1.7.1) is established. We need to demonstrate that these linear functionals are linearly independent and form a basis for the vector space V'. Let f be any linear functional in V'. Let f(Xi} = ai, i = 1,2, ... , k. Note that f = otlt +0212 + ... +oklk. The linear functionals It, 12, ... ,Jk do indeed span the vector space V'. As for their linear independence, suppose {3I1t + fhh + ... + {3klk = 0 for some scalars {3t, {32, ... , {3k in F. Observe that 0 = ({3I1t + fhh + ... + {3kfk)(Xi} = {3i

38


for each i = 1,2, ... ,k. Hence linear independence of these functionals follows. The result that the dimensions of the vector space V and its dual space are identical is obvious now. The basis h, 12, ... ,Ik so arrived above is called the dual basis of x}, X2, . . . ,Xk. Now we are ready to prove the separation theorem. P 1.7.7 Let u and v be two distinct vectors in a vector space V. Then there exists a linear functional I in V' such that I(u} 1= I(v} . Equivalently, for any non-zero vector X in V , there exists a linear functional I in V' such that I(x} 1= O. PROOF . Let Xl,x2,'" ,Xk be a basis of V and II! 12,.·· , Ik its dual basis. Write x = 6Xl + 6X2 + . .. + ~kXk for some scalars 6,6, .. . ,~k in F. If x is non-zero, there exists 1 ::; i ::; k such that ~i is non-zero. Note that li(X} = ~i =1= O. The first statement of P 1.7.7 follows if we take x = u - v .

Since V'is a vector space, we can define its dual vector space V" as the space of all linear functionals defined on V'. From P 1.7.6, we have dim(V} = dim(V/} = dim(V"}. Consequently, all these three vector spaces are isomorphic. But there is a natural isomorphic map from V to V", which we would like to identify explicitly. P 1.7.8 For every linear functional Zo in V", there exists a unique Xo in V such that

zo(J} = I(xo} The correspondence Zo

¢:>

for every I in V'.

Xo is an isomorphism between V" and V.

PROOF. Let 11,12, ... ,!k be a basis of V'. Given Xo in V , there exists a unique Zo in V" such that

We refer to P 1.7.5. Consequently, zo(J} = I(xo} for all I in V' . If Xl and X2 are two distinct vectors in V, then the corresponding vectors Zl and Z2 in V" must be distinct. If not, (Zl - Z2)(J} = 0 = /(XI) - I(X2} = I(XI - X2} for all I in V'. But this is impossible in view of P 1.7.7. Thus we observe that the correspondence Xo ¢:> Zo

Vector Spaces

39

enWlciated above is an injection. It is also clear that this association is a homomorphism. 'The isomorphism of this map now follows from the fact that dim(V) = dim(V"). Now that we have dual spaces in circulation, we can introduce the notion of annihilator of any subset (subspace or not) of a vector space. DEFINITION 1.7.9. The annihilator of a subset S of a vector space

sa of linear fWlctionals given by sa = {f E V' : f(x) = 0 for every x in S}. It is clear that the annihilator sa is a subspace of the vector space

V is the set

V' regardless of whether S is a subspace or not. If S contains only the null vector, then Sa = V'. If S = V, then sa = {O}. If S contains a non-zero vector, then sa i= V' in view of P 1.7.7. P 1.7.10 If S is a subspace of a vector space V, then dim(Sa) = dim V - dim(S). PROOF. Let xl, X2, ... ,Xr be a basis of the subspace S which can be extended to a full basis Xt, X2, .. . ,Xn Xr+b Xr+2, ... ,Xk of V. Let It, 12, ... ,ik be the dual basis of V'. Let f E sa. We can write f = Qllt + Q2h + ... + Qkfk for some scalars Qt, Q2,··· ,Qk in F. Observe that for each 1 ::; i ::; r,O = f(xd = Qi. Consequently, f is a linear combination of fr+l ,Jr+2, . .. ,fk only, i.e., f = Qr+!fr+l + Qr+2fr+2 + ... + Qkik. This implies that sa is a subspace of the span Sp(Jr+1 ,Jr+2,··· ,!k) of fr+l ,Jr+2, ... ,Jk. By the very construction of the dual basis, fi(xj) = 0 for every 1 ::; j ::; rand r + 1 ::; i ::; k. Consequently, each fi' r + 1 ::; i ::; k, belongs to sa. Thus we observe that Sp(Jr+1 ,Jr+2, ... ,ik) ~ sa. We have now identified precisely what sa is, i.e., Sa = Sp(Jr+!, fr+2, ... ,fk). From this it follows that dim(Sa) = k - r = dim(V) - dim(S).

The operation of annihilation can be extended. We start with a subspace S of a vector space V, and arrive at its annihilator sa which is a subspace of V'. Now we can look at the annihilator saa of the subspace sa. Of course, saa would be a subspace of V". This chain could go on forever. P 1.7.11 If S is a subspace of a vector space V, then morphic to S.

saa

is

is~

40


PROOF. Consider the bijection as identified between V and V" in P 1.7.S. For every Xo in V there exists a unique zo in V" such that zo(f) = I(xo) for every I in V'. If zo E 8 aa , then zo(J) = I(xo) = 0 for every I in 8 a • Since 8 is a subspace, this implies that Xo E 8. In a similar vein, one can show that if Xo E 8, then zo E 8 aa • The isomorphism that has been developed between V and V" in P 1.7.S when restricted to the subspace 8 is an isomorphism between 8 and 8 aa •

P 1.7.12

If 8 1 and 8 2 are subspaces of a vector space V, then

(8 1 + 8 2 )a

= 8~ n 8 2.

These identities follow from the definition of annihilator. Complements

1.7.1 If I is a non-zero linear functional from a vector space V(F) to a field F, show that I is a surjection (onto map). 1.7.2 If 11 and 12 are two linear functionals on a vector space V(F) satisfying It (x) = 0 whenever h(x) = 0 for x in V(F), show that It = 0:12 for some 0: in F. 1.7.3 Let F be a field and Pn(t) the vector space of all polynomials of n-l

degree less than n with coefficients from the field F. For any x =

E

dit i

i=O

in P n with di's in F, define n-l

I(x) = Ldi/3i i=O

for any fixed choice f3o, f31 , f32, . .. , f3n-l of scalars from F. Show that I is a linear functional. Show that any linear functional on P n (t) arises this way. 1.7.4 Let F = {O, 1, 2} with addition and multiplication modulo 3. Spell out all the linear functionals explicitly on F3. 1.7.5 The vectors (1,1,1,1), (1,1,-1,-1), (1,-1,1,-1), and (1,-1,-1,1) form a basis of the vector space R 4 . Let It, 12, h, and 14 be the dual basis. Evaluate each of these linear functionals at x = (1,2,3,4).

Vector Spaces

1.7.6

Let

f

41

be a linear functional on a vector space V(F) and

S = {x E V(F) : f(x) = O}. Show that S is a subspace of V(F). Comment on the possible values of the dimension of S. 1.7.7 If S is any subset of a vector space V, show that saa is isomorphic with the subspace spanned by S. 1.7.8 If S1 and S2 are two subsets of a vector space V such that S1 C S2, show that S2 c S'1.

1.8. Quotient Space There are many ways of generating new vector spaces from a given vector space. Subspaces are one lot. Dual spaces are another. In this section, we will introduce quotient spaces. DEFINITION 1.8.1. Let S be a subspace of a vector space V. Let x be an element of V. Then Sx = x + S is said to be a coset of S.

We have seen what cosets are in the context of groups. The idea is exactly the same. The group structure under focus here is the addition of vectors of a vector space. We define the following operations on the cosets of S. Addition:

For x, y in V, let


For a in F and x in V, let if a

# 0,

if a

= O.

The operation of addition defined above is nothing new. We have introduced this operation in the context of complementary subspaces. The following properties of these operations can be verified easily.

(1) Sx + Sy = Sx+y for any x, y in V. This means that Sx also a coset of S. (2) Sx + S = Sx for all x in V. (3) Sx + S-x = S for all x in V.

+ Sy

is

42


In addition to the above properties, the operation of addition satisfies commutative and associative laws. The set of all distinct cosets of 8 thus becomes a commutative group. The zero element of the group is 8 . The negative of 8 x is 8- x • The scalar multiplication introduced above on the cosets satisfies all the rules of a vector space. Consequently, the set of all cosets form a vector space which is called the quotient space associated with the subspace 8 and is denoted by V /8. The following result identifies what the quotient space is like. P 1.8.2 The quotient space is isomorphic to every complement of the subspace 8 of V. Let 8 c be a complement of 8. Let f : 8 c ---+ V /8 be defined by f(x) = 8 x ,x E 8 c • We show that f is an isomorphism. Let Xl and X2 be two distinct points of 8 c . Then 8 xi i= 8 X2 • If not, for any given Zl in 8 there exists Z2 in 8 such that Xl + Zl = X2 + Z2 = X, say. What this means is that X has two distinct decompositions, which is not possible. Consequently, f is an injection. Let K be any coset of 8. Then K = X + 8 for some X in V. Since X admits a unique decomposition, we can write X = Xo + Xl with Xo E 8 and Xl E 8 c • Consequently, K = X + 8 = Xo + Xl + 8 = Xl + (xo + 8) = Xl + 8. Thus K is of the form Xl + 8 for some Xl in 8 c . This shows that f is a surjection. It can be verified that f is a linear map. Hence f is an isomorphism. PROOF.

P 1.8.3 For any subspace 8 of a vector space V, dim(V /8) dim(V) - dim(8).

=

This result follows from the fact that V /8 is isomorphic to a complement of 8. If one is searching for a complement of 8, V /8 is a natural candidate! Complements 1.8.1 Let F = {O, 1, 2} with addition and multiplication modulo 3. Let 8 be the subspace of F3 spanned by (1,0,0) and (1,0,2) . Construct the quotient space F3 / S. Exhibit a complement of the subspace 8 different from the quotient space F 3 /8. 1.9. Projective Geometry It is time to enjoy the fruits of labor expended so far. We will present some rudiments of projective geometry just sufficient for our needs.

Vector Spaces

43

Some applications of projective geometry include the construction of orthogonal latin squares and balanced incomplete block designs. DEFINITION 1.9.1. Let S be a set of elements and S a collection of subsets of S. The pair (S, S) is said to be a projective geometry if (1) given any two distinct elements in S there is a unique set in S containing these two points, and (2) any two distinct sets in S have only one member of S in common. m the picturesque language of geometry, the members of S are called points and the sets in S lines. The condition (1) translates into the dictum that there is one and only one line passing through any two given distinct points. The condition (2) aligns to the statement that any two distinct lines meet at one and only one point. If the set S is finite, the associated geometry is called a finite projective geometry. m this section, we show that such geometries can be constructed from vector spaces. Consider a three-dimensional vector space V(F) over a finite field F consisting of s elements, say. If Xl, X2, and X3 are linearly independent vectors in V(F), we can identify V(F) as

Since each 0i can be chosen in different ways, the number of distinct vectors in V(F) is s3. We now build a finite projective geometry out of V(F). Let S be the collection of all one-dimensional subspaces (points) of V(F). Consider any two-dimensional subspace of V(F). Any such subspace can be written as the union of all its one-dimensional subspaces. mstead of taking the union, we identify the two-dimensional subspace by the set of all its one-dimensional subspaces. With this understanding, let S be the collection of all two-dimensional subspaces (lines) of V(F). We provide examples in the later part of the section. The important point that emerges out of the discussion carried out so far is that the pair (S, S) is a projective geometry. P 1.9.2

The pair (S, S) is a finite projective geometry. Moreover:

(1) The number of points, i.e., the cardinality of S is s2 + s + 1. (2) The number of lines, i.e., the cardinality of S is s2 + S + 1. (3) The number of points on each line is s + 1.

44


PROOF. A one-dimensional subspace of V(F) is spanned by a nonzero vector of V(F). For each non-zero vector x in V(F), let M(x) be the one-dimensional subspace spanned by x. There are s3 - 1 nonzero vectors in V(F). But the one-dimensional subspaces spanned by each of these vectors are not necessarily distinct in view of the fact that M(x) = M(ax) for each non-zero a in F and non-zero x in V(F). There are s - 1 vectors giving rise to the same one-dimensional subspace. Consequently, the total number of one-dimensional subspaces is (s3 - 1)/(s - 1) = s2 + s + 1. Thus the cardinality of the set 8 is s2 + s + 1. This proves (1). Any two-dimensional subspace of V(F) is spanned by two linearly independent vectors of V(F) . For any two linearly independent vectors Xt, X2 of V (F), let M (Xl, X2) be the twodimensional subspace of V(F) spanned by Xl, x2. The total number of pairs of linearly independent vectors is (s3 - 1)(s3 - s)/2. (Why?) The total number of non-zero vectors in any two-dimensional subspace is s2 - 1. The subspace M(XI' X2) can also be spanned by any two linearly independent vectors in M (Xl, X2). The total number of pairs of linearly independent vectors in M(XI' X2) is (s2 -1)(s2 - s)/2. (Why?) Consequently, the total number of different two-dimensional subspaces is [(s3-1)(s3- s )/2]/[(s2_1)(s2- s)/2] = s2+s +1. This proves (2). Using the argument similar to the one used in establishing (1), the total number of distinct one-dimensional subspaces of a two-dimensional subspace is (s2 -l)/(s -1) = s + 1. This proves (3). It remains to be shown that (8, S) is a projective geometry. Let M(XI) and M(X2) be two distinct one-dimensional spaces of V(F), i.e., distinct points of 8. Let M(X3, X4) be any two-dimensional subspace containing both M(XI) and M(X2). Since Xl, X2 E M(X3, X4), it follows that M(xI, X2) = M(X3, X4). Consequently, there is one and only one line containing any two points. Consider now two different two-dimensional subspaces M (Xl, X2) and M(X3, X4) of V(F). The vectors x}, X2, X3 and X4 are linearly dependent as our vector space is only three dimensional. There exist scalars a}, a2, not both zero, and a3, a4, not both zero, in F such that

(Why?) Obviously, y is non-zero. Clearly, the point y belongs to both the lines M(XI' X2) and M(X3, X4). This means that there is at least one point common to any two distinct lines. Suppose M(YI) and M(Y2)

Vector Spaces

45

are two distinct one-dimensional subspaces common to both M(x}, X2) and M(X3, X4). Then it follows that

M(y}, Y2) = M(x}, X2) = M(X3, X4). This is a contradiction. This shows that any two distinct lines intersect at one and only one point. The proof is complete. The projective geometry described above is denoted by PG(2, s). The relevance of the number s in the notation is clear. The number two is precisely the dimension of the vector space F3 less one. The number s could not be any integer. Since the cardinality of the Galois field F is s, s = pm for some prime number p and integer m ;::: 1. See Section 1.1. A concrete construction of the projective geometry PG(2, s) is not hard. For PG(2, s), what we need is a Galois field F with s elements, which we have described in Example 1.1.10 in a special case. Let the s elements of F be denoted by ao = 0, a} = 1, a2, ... ,a s -} . The underlying three-dimensional vector space V(F) over the field F can be taken to be F3. We now identify explicitly one- and two-dimensional subspaces of F3.

(a)

one-dimensional subspaces Anyone-dimensional subspace of F3 is one of the following types:

(1) Span {(1,ai,ajn, i,j = 0, 1,2, . . . ,s-1. (2) Span {(O, 1, ain, i = 0, 1,2, ... , s - 1. (3) Span {(O, 0, One way to see this is to observe, first, that the vectors (1, ai, aj), (0,1, ak) and (0,0,1) are always (Why?) linearly independent for any i,j, k = 0,1,2, ... ,s -1. Secondly, the totality of all one-dimensional subspaces listed above is exactly s2 + s + 1. In our projective geometry PG(2, s), each of these one-dimensional subspaces constitute points of the set S. For ease of identification, we codify these one-dimensional subspaces.

In.

Code assigned

One-dimensional subspace Span{(l, ai, ajn Span{(O, 1, ain Span{(O, 0,

In

S2

+ is + j, s + i, I

i, j = 0, 1,2, ... ,s - 1 i = 0, 1,2, ... ,s - 1

46


The integer codes for the points of S are all different, although they may not be successive numbers. We now try to identify the tw~dimensional subspaces of F3. Two and associated one dimensional subspaces

Type

Two-dimensional subspaces

1

{(O,O,l),(O,l,O)}

Constituent one-dimensional subspaces

{(O, 1, O)};

{(O, OJ, I)},

j = 0,1,2, ... , s - 1

2

{(O, 0,1), (1, 0i, O)} i

3

= 0, 1, 2, ...

,s - 1

{(O, 1,0), (1 , 0, Oi)} = 0, 1,2, ... , s - 1

i 4

{(O,O, I}; {(1, 0i, OJ)}, = 0, 1,2, ... , s - 1

j

{(O, 1, O};

{(I, OJ, Oi)}'

j = 0, 1,2, ... , s - 1

{(O, 1, Oi), (1, Ok, O)} {(O,I,Oi}; {(l,Ok +OJ,OiOj)}, i = 1,2,3 ... ,s -1, j = 0,1,2, ... , s - 1 k =0,1,2, ... ,s- 1

(For typographical convenience, the qualifying phrase "span" is omitted for the subspaces indicated.) The above table needs some justification. When we wrote down all one-dimensional subspaces of F3 in a systematic fashion, the following vectors in F3 arranged in three distinct groups played a pivotal role. Group 1 (0,0,1) Group 2 (0,1,0)

(0,1,1),

Group 3 (1,0,0)

(1,0,1), (1,1,1), (1,02,1),

(1,1,0) (1,02,0)

.. . ; (0,1, os-d (1,0,02); (1,1,02); (1,02,02);

... ; (1,0, os-d ... ; (1,1, os-d ... ; (1,02, os-d

(l,os-},O) (1, Os-I, 1), (1,Os-1,02); ... ; (l,Os-l,Os-l) Take any two vectors from anywhere in the above pool. Their span will give a tw~dimensional subspace of F3. But there will be duplications. We need to select carefully pairs of vectors from the above

Vector Spaces

47

so as to avoid duplications. Let us begin with the vector from Group 1 and find partners for this vector to generate a two-dimensional subspace. To start with, take the vector (0, 1,0) from Group 2 as a partner. Their span would give us a two-dimensional subspace of F3. The one-dimensional subspaces of Span {(O, 0,1), (0, 1, O)} can be identified in terms of the notation of the one-dimensional subspaces we have employed earlier. The one-dimensional subspaces are given by Span {(0,1,0)}, Span {OJ (0, 1,0) + (O,O,l)},j = 0,1,2, ... ,8 - 1. These one-dimensional subspaces can be rewritten in succinct from as: Span {(0,1,0)}, Span {(O,oj,I)},j = 0,1,2, ... ,8 -1, which are 8 + 1 in number, as expected. This particular two-dimensional subspace is categorized as of Type 1. Let us consider the span of (0,0,1) and anyone of the remaining vectors in Group 2. We will not get anything new. Let us now consider the span of the vector in Group 1 and any vector from Group 3. Consider, in particular, the span of (0,0,1) and any vector in the first column of Group 3. Any such two-dimensional subspace is categorized as of Type 2. You might ask why. Before we answer this question, observe that there are 8 such two-dimensional subspaces and they are all distinct. If we consider the span of (0,0, 1) with any vector in any of the remaining columns of vectors in Group 3, it would coincide with one of the two-dimensional subspaces we have stored under Type 2. The operation of finding mates for the vector (0,0,1) ends here. Let us work with the vector (0,1,0) from Group 2. The vector space spanned by (0,1,0) and anyone of the remaining vectors in Group 2 would coincide with the one we have already got under Type 1. Consider the vector space spanned by (0,1,0) and anyone of the vectors in the first row of Group 3. Each of these two-dimensional subspaces is categorized as of Type 3. There are 8 many of these two-dimensional spaces. These are all distinct among themselves and are also distinct from what we have got under Types 1 and 2. Further, the vector space spanned by (0,1,0) and anyone the vectors from the remaining rows of Group 3 would coincide with one of those we have already got. This completes the search of mates for the vector (0,1,0). Consider any vector (0,1, Oi), i = 1,2, ... ,8-1 from Group 2. The vector space spanned by (0, 1, Oi) and anyone of the vectors from Group 3 is categorized as of Type 4. All these two-dimensional spaces are distinct and are also distinct from what we have already got. There are 8(8-1) vector subspaces in Type 4. So far, we have got 8(8 -1) + 8 + 8 + 1 = 82 + 8 + 1 distinct

48


two-dimensional subspaces. We have no more! The identification of the one-dimensional subspaces in any two-dimensional subspace listed above is similar to the one we have explained for the two-dimensional subspace listed under Type 1. It would be quite instructive to use the integer codes for the one-dimensional subspaces listed under each twodimensional subspace delineated above. EXAMPLE 1.9.3. Let us look at a very specific example. Take s = 3. The Galois field F can be identified as {O, 1, 2} with the usual operations of addition and multiplication modulo 3. The total number of onedimensional subspaces (points) in PG(2, 3) is 13 and the total number of two-dimensional subspaces (lines) is 13. They are identified explicitly in the accompanying table along with the integer codes of the onedimensional subspaces.

The numbers involved in the integer code are:l,3,4,5,9,10,11,12,13,14, 15,16 and 17. For an inkling of what is to come, suppose we want to compare the performance of 13 treatments on some experimental units. Suppose that the treatments are numbered as 1,3,4,5,9,10,11,12,13,14, 15,16 and 17. In the above description of the projective geometry PG(2,3), each line can be viewed as a block and each point as a treatment. Corresponding to each line which has four points (integer codes) on it, we create a block with four experimental units and assign to them the treatments associated with the integer codes. We thus have 13 blocks corresponding to 13 lines, with four treatments in each block. We then have what is called a balanced incomplete block design! Each treatment is replicated exactly four times and every pair of treatments appears exactly in one block. We will now describe formally what a balanced incomplete block design is and explain how projective geometry provides such a design. One of the primary goals in design of experiments is to compare the effect of some 1/ treatments over a certain population of experimental units. The experimental units can be arranged in blocks in such a way that the units in each block are homogeneous in all perceivable aspects. Ideally, we would like to have blocks each containing 1/ experimental units, i.e., of size 1/, so that each treatment can be tried once in each block. In practice, blocks of size k with k < 1/ may only be available. Suppose we have b blocks each of size k. A design is simply an allocation of treatments to experimental units in the blocks. One of the basic

Vector Spaces

49

problems is to allocate the treatments to units in a judicious manner so that we can compare the performance of any two treatments statistically with the same precIsion. A design is said to be a balanced incomplete block design if the following conditions are met: (1) Each treatment appears in r blocks, i.e., each treatment is replicated r times. (2) Every pair of treatments appears together in A blocks. Such a design is denoted by BIBD with parameters b, k, 1/, rand A. The basic question is how to construct such a design. Projective geometry is a vehicle to realize our goal. If the number of treatments 1/ = s2 + s + 1 for some s of the form pm, where p is a prime number and m is a positive integer, and the number of blocks available is b = s2 + s + 1 each of size s + 1, then the projective geometry PG(2, s) will deliver the goods. Identify the points of the projective geometry with treatments and the lines with blocks. We have a balanced incomplete block design with parameters b = s2 + s + 1, k = s + 1, 1/ = s2 + s + 1, 1· = S + 1 and A = 1. The example presented in the accompanying Table is a BIBD with b = 13, k = 4, 1/ = 13, r = 4 and A = 1. After having gone through the gamut outlined above, one gets the uncomfortable feeling that the technique of projective geometries in the construction of balanced incomplete block designs is of limited scope. In practice, the number of blocks available, the number of treatments to be tested and block size may not conform to the above specifications. (At this juncture, we would like to point out that a BIBD may not be available for any given b, k and 1/.) In order to have more flexibility, we need to extend the ideas germane to the projective geometry PG(2, s). Start with a Galois field F consisting of s elements. Consider the vector space Fm+l. The projective geometry PG(m, s) consists of the set S of all one-dimensional subspaces of the vector space Fm+l and the set S of all k-dimensional subspaces of F m + 1 for some k 2: 2. The elements of S are called points and elements of S are called k-planes. One could treat each point as a treatment and each k-plane as a block. This is a more general way of developing a BIBD. We will not pursue in detail the general construction. We only want to provide a rudimentary introduction to what vector spaces and their ilk can solve a variety of statistical problems. For a discussion of finite projective geometries of dimensions more


50

than two, the reader is referred to Rao (1945c, 1946a) and to the references to other papers given there. TABLE: BIBD DESIGN Type

Tw~dimensional

subspaces

1

{(0,0,1),(0,1,0)}

2

{(0,0,1),(1,0,0)} {(0,0,1),(1,1,0)} {(0,0,1),(1,2,0)}

3

{(0,1,0),(1,0,0)} {(O, 1,0),(1,0, I)} {(0,1,0),(1,0,2)}

4

{(0,1,1),(1,0,0)} {(0,1,1),(1,1,0)} {(0,1,1),(1,2,0)} {(0,I,2),(I,0,0)} {(O,I,2),(I,I,0)} {(0,1,2),(1,2,0)}

One-dimensional subspaces

{(0,0,1)}: {(O, 1, I)}: {(O,O, I)}: {(1,0,1)}: {(O, 0, I)} : {(I, 1, I)} : {(O, 0, I)}: {(1,2,1)}: {(O, 1,0)}: {(I, 1, O)}: {(O, 1, O)} : {(I, 1, I)} : {(O, 1, O)} : {(I, 1, 2)} : {(O, 1, I)} : {(I, 1, I)} : {(0,1,1)}: {(I, 1, O)} : {(O, 1, I)}: {(I, 1, 2)} : {(0,1,2)}: {(I, 1, 2)}: {(O, 1, 2)}: {(I, 1, O)} : {(0,I,2)}: {(I, 1, I)} :

I nteger codes

{(0,1,0)} 1,3,4,5 {(O, 1, 2)}; {(1,0,0)} 1,9,10,11 {(1,0,2)}; {(I, 1, O)} 1,12,13,14 {(I, 1, 2)}; {(I, 2, O)} 1,15,16,17 {(1,2,2)}; {(1,0,0)} 3,9,12,15 {(I, 2, O)}; {(I, 0, I)} 3,10,13,16 {(I, 2, I)}; {(I, 0, 2)} 3,11,14,17 {(I, 2, 2)}; {(I, 0, O)} 4,9,13,17 {(I, 2, 2)}; {(1,0,2)} 4,11,12,16 {(I, 2, I)}; {(I, 0, I)} 4,10,14,15 {(I, 2, O)}; {(1,0,0)} 5,9,14,16 {(I, 2, I)}; {(I, 0, I)} 5,10,12,17 {(I, 2, 2)}; {(1,0,2)} 5,11,13,15 {(I, 2, O)}.

Note: Some references to material covered in this Chapter, where further details can be obtained, are Bose, Shrikhande and Parker (1960), Halmos (1958), Raghava Rao (1971) and Rao (1947, 1949).

CHAPTER 2 UNITARY AND EUCLIDEAN SPACES So far we have studied the relationship between the elements of a vector space through the notion of independence. It would be useful to consider other concepts such as distance and angle between vectors as in the case of two and three dimensional Euclidean spaces. It appears that these concepts can easily be extended to vector spaces over the field of complex or real numbers by defining a function called the inner product of two vectors.

2.1. Inner Product However abstract a vector space may be, when it comes to practicality, we would like to relate the vectors either with real numbers or complex numbers. One useful way of relating vectors with real numbers is to associate a norm, which is a non-negative real number, with every vector. We will see more of this later. Another way is to relate pairs of vectors with complex numbers leading to the notion of inner product between vectors. We will present rudiments of these ideas now. DEFINITION 2.1.1. Let V be a vector space over a field F, where F

is either the field C of complex numbers or R of real numbers. A map < ', ' > from V x V to F is called an inner product if the following properties hold for all x, y, z in V and a, {3 in F .

(1) < x, y >

(2) < X,x >

= {

> 0, if x =I 0, (positivity)

= 0, (3)

< y, x >, if F = C, (anti-symmetry) < y, x >, if F = R. (symmetry) if x

= 0.

< ax + {3y, z > = a < x, z > + {3 < y, z > . (linearity in the first argument) 51

52


The bar that appears in (1) above is the operation of conjugation on complex numbers. A vector space furnished with an inner product is called an inner product space. It is customary to call such a space unitary when F = C, and Euclidean when F = R. We have the following proposition as a consequence of the conditions (1), (2) and (3) of Definition 2.1.1. In the sequel, most of the results are couched with reference to the field C of complex numbers. Only minor modifications are needed when the underlying field is R.

P 2.1.2

For any x, y, z in V(C) and a, /3 in C, the following hold.

(a) < x, ay + /3z > = a < x, y > +!3 < x, z >. (b) < x, 0> = < 0, x >= O. (c) = a<x,/3y>= a!3<x,y>. Some examples of inner product spaces are provided below. EXAMPLE 2.1.3. Consider the vector space Rk for some k ~ 1. For any two vectors x = (a1,a2,'" ,ak) and y = (/31,/32,'" ,/3k) in Rk, define

< x, y > = a1/31

+ a2/32 + ... + ak/3k,

which can be shown to be an inner product on the vector space Rk. This is the standard inner product of the space R k • 2.1.4. Consider the vector space C k for some k ~ 1. Let 81 ,82 , ••• ,15k be fixed positive numbers. For any two vectors x = (at, a2, .. · ,ak) and y = (/31, /32, ... ,/3k) in Ck, define EXAMPLE

which can be shown to be an inner product on C k • If 81 = 82 = ... = 15k = 1, the resultant inner product is the so-called standard inner product on the space Ck. One might wonder about the significance of the way the inner product is defined above on C k • If one defines

then one of the conditions (which one?) of Definition 2.1.1 is violated.

Unitary and Euclidean Spaces

53

2.1.5. Let P n be the space of all polynomials of degree less than n with coefficients from the field C of complex numbers. For any two polynomials x(-) and y(.) in P n , define EXAMPLE

1

< x, y > =

J

x(t)y(t) dt,

o which can be shown to be an inner product on P n . EXAMPLE 2.1.6. Let Xl,x2, .. . ,Xk be a basis of a vector space V(C). For any two vectors x and y in V, we will have unique representations

in terms of the basis vectors. Let 81 ,82 , ... ,8k be some fixed positive numbers. Define

which can be shown to be an inner product on the vector space V. Note that an inner product on a vector space can be defined in many ways. The choice of a particular inner product depends on its usefulness in solving a given problem. We will see instances of several inner products in subsequent sections and chapters. Every inner product gives rise to what is known as a norm. DEFINITION 2.1.7. (Norm) Let < ., . > be an inner product on a vector space V. The positive square root of < x, x > for any x in V is called the norm of x and is denoted by IIx II. There is a more general definition of a norm on a vector space. The norm we have introduced above arises from an inner product. The more general version of a norm will be considered in a later chapter. As a consequence of Definitions 2.1.1 and 2.1.7, the following inequality follows.

P 2.1.8 (Cauchy-Schwartz Inequality). Let (V, < ., . » be an inner product space with the associated norm II . II in V. Then for any two vectors x and y in V, the inequality

1< x,y > I ~ IIxlillyll,

(2.1.1)


54

holds. Moreover, equality holds in the above if and only if for some I and 8, not both zero, in C.

,X + 8y = 0

Let (3 = < x,x > and a = - < y,x >. Observe that < x, y >. We are required to establish that lal 2 = ao. =

PROOF.

a

= -

I < x, y > 12 :::; (3 < y, y >.

By the definition of an inner product,

o :::; < ax + (3y, ax + (3y >

= a < x, ax

= = =

+ (3y > + (3 < y, ax + (3y >

< x,x > +ajj < X,y > + (30. < y,x > +(313 < y,y > ao.

lal 2(3 -laI 2(3 -laI 2(3 + (32 < y,y > -laI 2(3 + (32 < y, y >, (2.1.2)

,X

from which (2.1.1) follows. If + 8y = 0 for some I and 8, not both zero, in C, it is clear that equality holds in (2.1.1). On the other hand, if equality holds in (2.1.1), equality must hold in (2.1.2) throughout. This implies that ax + (3y = O. If x = 0, take I to be any non-zero scalar and 8 = O. If x i= 0, take I = a and 8 = (3. This completes the proof.

P 2.1.9 For any two vectors x and y in a vector space V equipped with an inner product < .,. >, the following inequality holds.

< X,y > + < y,x >:::; 2I1x·lIlIyll. PROOF. Observe that 21 < X,y > I ~ < X,y > + < y,x >. The result now follows from the Cauchy-Schwartz inequality. We now establish some properties of a norm. For any two vectors x and y, IIx - yll can be regarded as the distance between x and y.

P 2.1.10 Let x and y be any two vectors in an inner product space V with inner product < .,. > and norm 11·11. Then the following hold.

(I) IIx + yll :::; IIxll + lIyll· (2) IIx - yll + IIYII ~ IIxll (triangle inequality of distance). (3) IIx + Yll2 + IIx - Yll2 = 211xl12 + 211Yl12 (parallelogram law). (4) IIx +y1l2 = IIxll 2+ lIyll2 if < x, y > = 0 (Pythagorous theorem).


55

PROOF. By the definition of the norm,

II x + Yll2

=

~ =

IIxll 2+ IIYIl2+ < x,y > + < y,x > IIxll 2+ lIyl12 + 211 xll Ilyll (by Cauchy-Schwartz inequality) (lIxll + lIyll)2,

from which (1) follows. In (1), if we replace x by y and y by x - y, we obtain (2). The relations expostulated in (3) and (4) can be established in an analogous fashion. We now formally define the distance and angle between any two vectors in any inner product space. DEFINITION 2.1.11. Let x and y be any two vectors in a vector space

V equipped with an inner product < " -> and the associated norm 11 · 11. The distance 8(x, y) between x and y is defined by 8(x, y) = IIx - YII.

P 2.1.12 The distance function 8(-'·) defined above has the following properties. (1) 8(x, y) = 8(y, x) for any x and y in V. (2) 8(x, y) ? 0 for any x and y in V, =0 if and only if x = y. (3) 8(x,y) ~ 8(x,z) + 8(y,z) for any X,y and z in V (triangle inequality). PROOF. The properties (1) and (2) follow from the very definition of the distance function. If we replace x by x - y and y by x - z in P 2.1.10 (2), we would obtain the triangle inequality (3) above. DEFINITION 2.1.13. Let V be a Euclidean space equipped with an inner product < ',' > and the associated norm II . II. For any two non-zero vectors x and y in V, the angle () between x and y is defined by cos() = < x, y > f[lI xll lIyll].

Observe that, in view of the Cauchy-Schwartz inequality, cos () always lies in the interval [-1, 1]. This definition does not make sense in unitary spaces because < x, y > could be a complex number. The notion of angle between two non-zero vectors of a Euclidean vector space is consonant with the usual perception of angle in vogue in the two-dimensional Euclidean space. Let x = (Xl, X2) and y = (yI! Y2)

56


be two non-zero vectors in the first quadrant of the two-dimensional Euclidean space R2. Let Ll be the line joining the vectors = (0,0) and x = (Xl,X2) and L2, the line joining 0= (0,0) and y = (Yl,Y2). Let 01 and O2 be the angles the lines Ll and L2 make with the x-axis, respectively. Then the angle 0 between the lines Ll and L2 at the origin is given by 0 = 01 - O2 • Further,

°

cos 0 = cos ( 01

xl

-

Yl

(

2 ) = cos 01 cos O2 + sin 0 1 sin O2 x2 Y2 < x, Y > \lx\l\ly\I·

=W·TIYIT+W ·TIYIT= Complements

2.1.1 Let V be a real inner product space and 0:, /3 be two positive real numbers. Show that the angle between two non-zero vectors x and y of V is the same as the angle between the vectors o:x and /3y. 2.1.2 Compute the angle between the vectors x = (3, -1, 1,0) and y = (2,1, -1, 1) in R4 with respect to the standard inner product of the space R4. 2.1.3 Let 0:,/3" and 8 be four complex numbers. For x = (6'~2) and y (1]b1J2) in C 2, define < X,y > O:~lih /36ih ,6ii2 86ii2. Under what conditions on 0:, /3", and 8, is < -,. > an inner product on C2? 2.1.4 Suppose Ilx + Yl12 = \lxl1 2 + \lyl12 for some two vectors x and y in an unitary space V. Is it true that < x, y > = o? What happens when V is a Euclidean space?

=

=

+

+

+

2.2. Orthogonality Let us, for a moment, look at two points x = (Xl, X2) and Y = (y!, Y2) in the two-dimensional Euclidean space R2. Draw a line joining the points = (0,0) and x and another line joining and y. We would like to enquire under what circumstances the angle between these lines is 90°. Equivalently, we ask under what conditions the triangle .6.0xy formed by the points 0, x, and y is a right-angled triangle with the angle LxOy at the origin = 90°. It turns out that the condition is < x, y > = XlYl + X2Y2 = 0. (Draw a picture.) This is the idea that we would like to pursue in inner product spaces.

°

°


57

2.2.1. Two vectors X and y in an inner product space V are said to be orthogonal if the inner product between x and y is zero, i.e., < x, y > = O. In the case of a Euclidean space, orthogonality of x and y implies that the angle between x and y is 90°. Trivially, if x = 0, then x is orthogonal to every vector in V. The notion of orthogonality can be extended to any finite set of vectors. DEFINITION

DEFINITION 2.2.2. A collection, Xl, X2, ... product space V is said to be orthonormal if

,x r,

of vectors in an inner

if i =1= j, if i = j.

If a vector x is such that < x, x > = IIxll2 = 1, then x is said to be of unit length. If we drop the condition that each vector above be of unit length, then the vectors are said to be an orthogonal set of vectors. P 2.2.3 Let Xl, X2, ... ,Xr be an orthogonal set of non-zero vectors in an inner product space V. Then Xl, X2, ••• ,X r are necessarily linearly independent. PROOF. Suppose y = 0lXI + 02X2 + ... + OrXr = O. Then for each 1 ~ i ~ r, 0 = < y,Xi >= 0i < Xi,Xi >= O. Since Xi is a non-zero vector, 0i = O. This shows that the orthogonal set under discussion is linearly independent. One of the most useful techniques in the area of orthogonality is the well-known Gram-Schmidt orthogonalization process. The process transforms a given bunch of vectors in an inner product space into an orthogonal set.

P 2.2.4 (Gram-Schmidt Orthogonalization Process). Given a linearly independent set Xl, X2, ... ,Xr of vectors in an inner product space, it is possible to construct an orthonormal set Zl, Z2, ... ,Zr of vectors such that Sp(xt, ... ,Xi) = Sp(Zl'" . ,Zi) for every i = 1, ... ,r. [Note the definition: Sp( aI, ... ,ak) = {Ol al + ... + 0kak : 01, ... ,Ok E C}.] PROOF.

= Xl, Y2 = X2 -

Define vectors Yl, Y2,

YI

02.IYlt

. .. ,Yr

in the following way:

58


Yr = Xr - Or,r-lYr-l - Or,r-2Yr-2 - ... - Or,lYl,

for some scalars Oi,/S. We will choose Oi,/S carefully so that the new vectors Yl, Y2, . .. , Yr form an orthogonal set of non-zero vectors. The determination of Oi,/S is done sequentially. Choose 02,1 so that Yl and Y2 are orthogonal. Setting

o

=

< Y2,Yl >

=

< X2,Xl >

-02,1

< YbYl >,

we obtain 02,1 = < X2, Xl > / < Yb Yl >. (Note that < Y}, Y1 > > 0.) Thus Y2 is determined. FUrther, the vector Y2 is non-zero since Xl and X2 are linearly independent. Choose 03,2 and 03,1 so that Y1, Y2 and Y3 are pairwise orthogonal. Set

o = < Y3,Y2 > = < X3,Y2 > o = = <X3,Yl > -

03,2 03,1

< Y2,Y2 >. .

From these two equations, we can determine 03,1 and 03,2 which would meet our requirements. Thus Y3 is determined. Note that the vector Y3 is a linear combination of the vectors x}, X2, and X3. Consequently, Y3 =1= O. (Why?) Continuing this process down the line successively, we will obtain a set Y1, Y2, .. , , Yr of orthogonal non-zero vectors. The computation of the coefficients oi./s is very simple. For the desired orthonormal set, set Zi = YdilYill, i = 1,2 ... , r. From the above construction, it is clear that (1) each Yi is a linear combination of X1,X2, ... ,Xi, and (2) each Xi is a linear combination of Y1, Y2, ... , Yi, from which we have

2.2.5. Let P4 be the vector space of all real polynomials of degree less than 4. The polynomials 1, X, x2, x3 form a linearly independent set of vectors in P 4. For p(.) and q(.) in P 4, let EXAMPLE

+1

< pO, qO >=

J

p(X) q(x) dx.

-1


59

Observe that < ., . > is an inner product on P 4. The vectors 1, X, x2, x3 are not orthogonal under the above inner product. We can invoke the Gram-Schmidt orthogonalization process on these vectors to obtain an orthonormal set. The process gives

= 1, p2(X) = x, P3(X) = x2 - 1/3, P4(X) = x3 -

Pl(X)

(3/5)x.

This process can be continued forever. The sequence PI. P2, ... of polynomials so obtained is the well-known Legendre orthogonal polynomials. We can obtain an orthonormal set by dividing each polynomial by its norm. We can create other sequences of polynomials from 1, x, x2, ... by defining inner products of the type

choosing a suitable function f(x). We know that every vector space has a basis. If the vector space comes with an inner product, it is natural to enquire whether it has a basis consisting of orthonormal vectors, i.e., an orthonormal basis. The Gram-Schmidt orthogonalization process provides the sought-after basis. We will record this in the form of a proposition. P 2.2.6 Every inner product space has an orthonormal basis. If we have a basis Xl,X2, ... ,Xk for an inner product space V, we can write every vector x in Vasa linear combination of Xl, X2, ... ,Xk,

for some 01,02, ... ,Ok in C. Determining these coefficients Oi'S is a hard problem. If Xl, X2, ... ,Xk happen to be orthonormal, then these coefficients can be calculated in a simple way. More precisely, we have OJ = < X,Xi > for each i and

X = <X,Xl >Xl

+

<X,X2 >X2

+ ... +

<X,Xk >Xk.

This is not hard to see. There are other advantages that accrue if we happen to have an orthonormal basis. The inner product between any


60

two vectors x and y in V can be computed in a straightforward manner. More precisely,

>< XI,Y > + < X,X2 >< + ... + < X,Xk >< Xk,Y >.

< x,y > =
(2.2.1)

The above is the well-known Parseval Identity. Once we know the c0efficients Oi ' S in the representation of x in terms of Xl , X2 , ••• , Xk and the coefficients f3/s in the representation of Y in terms of Xl, X2, ••• , Xk, we can immediately jot down the inner product of X and Y, courtesy Parseval identity, as

One consequence of (2.2.1) is that the norm of X can be written down explicitly in terms of these coefficients. More precisely,

In this connection, it is worth bringing into focus the Bessel Inequality. The statement reads as follows: if Xl, X2, • •• , Xr is a set of orthonormal vectors in an inner product space, then for any vector X in the vector space, the following inequality holds

IIxII 2 2:

r

L 1
12.

i =l

The identity (2.2.1) is not hard to establish. What is interesting is that the characteristic property (2.2.1) of an orthonormal basis characterizes the orthonormal property. We record this phenomenon in the following proposition.

P 2.2.7

Let V be an inner product space of dimension k. Let be some k vectors in V having the property that for any two vectors X and Y in V, Xl, X2, • •• , Xk

= < X,XI >< XI,Y > + < X,X2 >< + ... + < X , X2 >< Xk,Y > .

X2,Y

> (2.2.3)


Then

XI,X2,'"

,Xk

61

is an orthonormal basis for V.

PROOF. Let us see what we can get out of the property (2.2.3) enunciated above. By plugging x = Xl and y = Xl in (2.2.3), we observe that

Since all the terms involved are non-negative, the only way that the above equality could hold is that IIxIIi :::; 1. As a matter of fact, we have IIxili :::; 1 for every i. Let Ul, U2,' .. ,Uk be an orthonormal basis of the vector space V. Each Xi has a unique representation in terms of the given orthonormal basis. By (2.2.2),

for each i. By plugging

= Y=

X

Ui

in (2.2.3), we observe that

for each i. Summing (2.2.5) over i, we obtain k

k=

k

L L 1
12

i=l j=l

k

k

k

= LLI < Xj,Ui > 12 = LIIXjI12. j=l i=l

(2.2.6)

j=l

We have seen earlier that each IIxili :::; 1. This can coexist with (2.2.6) only when each IIxili = 1. In that case, if we look at (2.2.4) and related identities, it follows that < Xi, Xj > = 0 for all i =I- j. This completes the proof. IT we look at the proof, one wonders whether the assumption that the vector space has dimension k can be dropped at all. It is not feasible. Try to prove the above proposition by dropping the assumption on the dimension! In Chapter 1, we talked about complements of subspaces of a vector space. We have also seen that the complement need not be unique. IT we

62


have, additionally, an inner product on the vector space, the whole idea of seeking a complement for the given subspace has to be reexamined under the newer circumstances. This is what we propose to do now. P 2.2.8 Let V be a vector space equipped with an inner product < .,. > and S a subspace of V. Then there exists a subspace S.L of V with the following properties. < x, Y > = 0 whenever xES and yES.L. S n S.L = {O} and V = S $ S.L. dim(S) + dim(S.L) = dim(V).

(1) (2) (3)

PROOF. Let Xl, X2, ... , Xr be a basis of the subspace S, and extend it to a full basis XI,X2, ... ,XnXr+I, ... ,xk ofV . Letzl,Z2, ... ,ZnZr+b .•. ,Zk be the orthonormal basis of V obtained by the Gram-Schmidt orthogonalization process carried out on the Xi'S. We now have a natural candidate to fit the bill. Let S.L be the vector subspace spanned by Zr+b Zr+2, ... ,Zk. Trivially, (1) follows. To prove (2), note that every vector x in V has a unique representation, x

= (OIZI

+ 02Z2 + ... + OrZr) + (Or+IZr+1 + ... + OkZk)

= YI + Y2,

say,

for some scalars oi's in C, where YI = 0IZI + 02Z2 + .. . + 0rZr and Y2 = X - YI . It is clear that YI E Sand Y2 E S.L. By the very construction, we have S n S.L = {O} and dim(S) + dim(S.L) = dim(V). We have talked about complements of a subspace in Chapter 1. The subspace S.L is a complement of the subspace S after all. But the subspace S.L is special. It has an additional property (1) listed above. In order to distinguish it from the plethora of complements available, let us call the subspace S.L an orthogonal complement of the subspace S. When we say that S.L is an orthogonal complement, we sound as though it is not unique. There could be other subspaces exhibiting the properties (1), (2) and (3) listed above. The proof given above is not much of help to settle the question of uniqueness. The subspace S.L is indeed unique and can be characterized in the following way. P 2.2.9 Let S be a subspace of an inner product space V. Then any subspace S.L having the properties (1) and (2) of P 2.2.8 can be characterized as S.L

= {x

E V:< X,Y >

=

0

for every

YES}.


63

PROOF. Let S .. be a subspace of V having the properties (1) and (2) of P 2.2.8. Let S .... = {x E V :< x, y >= 0 for every YES}. We will show that S .. = S ..... As in the proof of P 2.2.8, let Zl, Z2, ••• ,Zr, Zr+1' ••• ,Zk be an orthononnal basis for V such that Zl, Z2, .•• ,Zr is a basis for the subspace S. Then for xES,

x = < x, zl > +

Zl

+ < x, Z2 >

Z2

+ ... + < x, Zr >

Zr

< x, zr+l > zr+l + ... + < x, Zk > zk,

with < x, Zl > Zl + < x, Z2 > Z2 + ... + < x, Zr > Zr E S. If XES", then, by (2), < x, Zi > = 0 for every i = 1,2, ... ,1·. Consequently, xES ..... Conversely, if xES .... , < x, Zi > = 0 for every i = 1,2, ... ,T, in particular. Hence xES". This completes the proof. To stretch matters beyond what was outlined in P 2.2.8, one could talk about the orthogonal complement (S.1).1 of the subspace S.1. If we look at the conditions, especially (1), of P 2.2.8 for the orthogonal complement S.1 of a given subspace S should meet, we perceive some symmetry in the way the condition (1) is arraigned. P 2.2.9 provides a strong motivation for the following result.

P 2.2.10 For any subspace S of an inner product space V, the relation, (S.1).1 = S, holds true. PROOF. Does this really require a proof? Well, let us try one. By P 2.2.8 (1) and (2), it follows that (S.1).1 C S. Since dim(S)+dim(S.1) = dim((S.1).1)+dim(S.1) = dim(V), we have dim((S.1).1) = dim(S). This together with (S.1).1 c S implies that (S.1).1 = S. In the absence of an inner product on a vector space, we could talk about complements of subspaces of the vector space. One could also talk about the complement of a given complement of a given subspace of a vector space. There is no guarantee that the second generation complement will be identical with the given subspace. Starting with a given subspace, one can keep on taking complements no two of which are alike! We are now in a position to introduce Orthogonal Projections. These projections have some bearing in some optimization problems. A more general definition of a projection will be provided later. First, we start with a definition.

64


DEFINITION 2.2.11. (Orthogonal Projection) Let S be a subspace of

an inner product space V and S.l its orthogonal complement. Let x be any vector in V. Then the vector x admits a unique decomposition, x = Y + z, with yES and z E S.l. Define a map Ps from V to S by Ps(x) = y. The map Ps is called an orthogonal projection from the space V to the space S. The orthogonal projection is really a nice map. It is a homomorphism from the vector space onto the vector space S. It is idempotent. These facts are enshrined in the following proposition.

P 2.2.12 Let P s be the orthogonal projection from the inner product space V to its subspace S. Then it has the following properties. (1) The map P s is a linear map from V onto S. (2) The map Ps is idempotent, i.e., Ps(Ps(x)) = Ps(x) for every x in the vector space V. PROOF. The uniqueness of the decomposition of any vector x as a sum of two vectors y and z, with y in Sand z in S.l, is the key ingredient for the map P s to have such nice properties as (1) and (2). For the record, observe that the map Ps is an identity map when it is restricted to the subspace S. In order to define the projection map we do not need an inner product on the vector space. The projection map can always be defined on V with respect to some fixed complement sc of the subspace S of V. Such a map will have properties (1) and (2) of P 2.2.12. The orthogonal projection expostulated above arises in a certain optimization problem. Given any vector x in an inner product space V and any subspace S of V, we would like to compute explicitly the distance between the point x and the subspace S. The notion of distance between any two vectors of an inner product space can be extended to cover vectors and subsets of the vector space. More precisely, if x is a vector in V and A a subset of V, the distance between x and A can be defined as infYEA IIx - YII. Geometrically speaking, this number is the shortest distance between x and points of A. Generally, this distance is hard to compute and may not be attained. If the subset happens to be a subspace of V, the computation is simple, and in fact, the distance is attained at some point of the subspace.

P 2.2.13 Let x be any vector in an inner product space V and Sa


65

subspace of V. Then the distance S(x, S) between x and S is given by

+

PROOF.Since V = S @ S', we can write x = x l x2 with x l E S and x2 E S'. Of course, x l = Ps(x). For any vector y in S, observe that

The last equality requires justification. First, the vector xl - y belongs to S, and of course, x2 E SL. Consequently, < x l - y,x2 > = 0. Pythagorous theorem now justifies the last equality above. See P 2.1.10 (4). After having split llx - Y112 into two parts, we minimize llxl - yll 2 over all y E S. Since x1 belongs to S, the minimum occurs at y = x l = Ps(x). - This completes the proof. The Pythagorous theorem and the decomposition of an inner product space into two subspaces which are orthogonal to each other are two sides of the same coin. If S and SI are complementary subspaces of a inner product space V , and x E S , y E S', then llz y1I2 = 1 1 ~ 1 1 ~ llY112 2 lly112. The above inequality can be paraphrased as follows. For the inequality, llx yll 2 Ilyll, holds for every x E S. any fixed y E sL1 Does this property characterize membership of y in s I ? Yes, it does.

+

+

+

P 2.2.14 Let y be any vector in an inner product space V and S a subspace of V. Then y E S' if and only if llx

+ yll

2 llyll for every x in S.

(2.2.7)

PROOF.We have already checked the "only if" part of the above statement. To prove the "if part", let y = yl y;! be the orthogonal decomposition of y with yl E S and y2 E SL. It suffices to show that yl = 0. Observe that

+

A word of explanation is in order. In the above chain of equalities and inequalities, Pythagorous theorem is used as well as (2.2.6). Observe

66


also that (-y) E S. Thus equality must prevail everywhere in the above chain and hence Y1 = O.

Complements 2.2.1

Show that two vectors x and yare orthogonal if and only if

for all pairs of scalars a and {3. Show that two vectors x and y in a real irmer product space are orthogonal if and only if

2.2.2 If x and yare vectors of unit length in a Euclidean space, show that x + y and x - yare orthogonal. 2.2.3 Let Xl, X2,'" ,Xk be an orthonormal basis of an irmer product space and Y1 = Xl, Y2 = Xl + X2,· .. ,Yk = Xl + ... + Xk. Apply the Gram-Schmidt orthogonalization process to Y1, Y2, ... ,Yk· 2.2.4 Let Sl and S2 be two subspaces of an irmer product space. Prove the following.

cst·

(1) If Sl C S2, then Sf (2) (Sl n S2)1- = + Sf . nSf· (3) (Sl + S2)1- =

st st

2.2.5 Let S be a subspace of V and consider the set of points H = {xo + x: XES} for fixed Xo. Find min IIY - zll for given Y with respect to z E H. 2.3. Linear Equations In Section 1.6, we have considered a linear equation, homogeneous or non-homogeneous, in the environment of vector spaces involving unknown scalars belonging to the underlying field. A special case of such a linear equation we have considered is the one when the vector space was Fk, for some k ~ 1. The linear equation gave rise to a system of linear equations (1.6.6), which, upon close scrutiny, gives the feeling that there is some kind of inner product operation involved. In this section, we will indeed consider irmer product spaces and equations involving

67


the underlying inner products. Let a1, a2,' .. ,am be given vectors in an inner product space V. Let a1, a2, ... ,am be given scalars in the underlying field of the vector space. Consider the system of equations, <X,ai> = ai, i = 1,2, ...

,m,

(2.3.1)

in unknown x E V. If V = C k or R k, and < -, - > is the usual inner product on V, then the above system of equations (2.3.1) identifies with the system (1.6.6). The above system is, in a way, more general than the system (1.6.6). Of course, in (1.6.6), the underlying field F is quite arbitrary. We now need to explore some methods of solving equation (2.3.1). P 2.3.1 The system (2.3.1) of equations has a solution (Le., the equations (2.3.1) are consistent} if and only if m

m

L,Biai = 0 whenever L.8iai = 0 i=1 i=1 for any scalars .8}' .82, ... ,.8m in C.

(2.3.2)

PROOF. Suppose the system (2.3.1) admits a solution x, say. Supm

pose for some scalars .81, .82, ... ,.8m in C,

i=1

.8iai

= O.

m

m

< x, L

L

.8i ai > = 0 -

i=l

L,Bi < X,ai

Then

m

> = L,Biai.

i=l

i=1

For the converse, consider the following system of linear equations < a},al > "Y1 + < a1, a2 > "Y1 +

+ < am,al > "Ym = + < am, a2 > "Ym =

aI, a2,

(2.3.3)

in unknown scalars "Yl, "Y2, ... ,"Ym in C. Our immediate concern is whether the system (2.3.3) admits a solution. We are back into the fold of the system of linear equations (1.6.6). We would like to use

68


P 1.6.7 which provides a necessary and sufficient condition for the system (1.6.6) to have a solution. Let Ui = « aI, ai >, < a2, ai >, ... , < am, ai », i = 1,2, . .. ,m. We need to verify whether the condition,

is satisfied whenever f31Ul

+ f32u2 + ... + f3mum

for any

= 0

(2.3.4)

131,132, . . ·, 13m E C,

is satisfied to guarantee a solution to the system (2.3.3). If 131 Ul + f32U2 + .. . +f3mum = 0, then this is equivalent to 131 < ai, al > + 132 < ai, a2 > + ... + 13m < ai, am > = 0 for each i = 1,2, ... ,m. This, in turn, m

is equivalent to < ai,

L: !3jaj > =

0 for each i = 1,2, ... ,m. Suppose j=1 f31ul + f32u2 + ... + f3mum = 0 for some scalars 131,132,' .. ,13m. By what m

we have discussed above, this is equivalent to < ai,

L: !3jaj > =

0 for

i=1

each i

= 1,2, ... m

,m. This then implies that

m

= < L: !3i ai, L: !3i ai > . m

i=1

i=1

m

m

i=1

j=1

L: !3i ai =

O.

L:!3i < ai, L: !3jaj

m

Consequently,

>= 0

By (2.3.2),

i=l

L: f3i o i =

O. Thus (2.3.4) is verified. The system (2.3.3) admits a solui=1 tion. Denote by, with an apology for an abuse of notation, ,1,,2, ... "m a solution to the system (2.3.3) of equations. Let

One can verify that Xo is a solution of the system (2.3.1) of equations. The verification process merely coincides with the validity of the system (2.3.3) of equations. As has been commented earlier, there is an uncanny resemblance between the systems (2.3.1) and (1.6.6). In view of P 1.6.7, the above result is not surprising. Suppose the system (2.3.1) of equations is consistent, i.e., the system admits a solution. An immediate concern is the identification of a s~ lution. If we scrutinize the proof of P 2.3.1 carefully, it will certainly

Unitary and Eu.clidean Spaces

69

provide an idea of how to obtain a solution to the system. This solution is built upon the solution of the system (2.3.3) of equations operating in the realm of the field of complex numbers. Solving the system (2.3.3) is practical since we are dealing with complex numbers only. There may be more than one solution. We need to determine the stmcture of the set of all solutions of (2.3.1). The following proposition is concerned about this aspect of the problem.

P 2.3.2 Let 8 1 be the collection of all solutions to the system (2.3.1) of equations, assumed to be consistent. Let 8 be the collection of all solutions to the system

< x, ai > = 0

i

= 1,2, ...

,m,

of equations. Let Xo be any particular solution of (2.3.1). Then (1) 8 is a subspace of V, and (2) 8 1 = Xo + 8 = {xo + y : y E 8}. The above proposition is modeled on P 1.6.4. The same kind of humdrum argument carries through. Among the solutions available to the system (2.3.1), we would like to pick up that solution x for which IIxll is minimum. We could label such a solution as a minimum norm solution. The nicest thing about the solution which we have offered in the proof of P 2.3.1 is that it is indeed a minimum norm solution. Let us solemnize this fact in the following proposition.

P 2.3.3 The unique minimum norm solution of (2.3.1), when it is consistent, is given by Xo

= Ilal + 12 a2 + ... + Imam,

where 11,12, ... ,1m is any solution to the system (2.3.3) of equations. Further, IIxoll = ilal + i2 a 2 + ... + imam· PROOF. We have already shown that Xo is a solution to the system

(2.3.1) in P 2.3.1. Any general solution to the system (2.3.1) is of the form Xo + y, where y satisfies the conditions that < y, ai > = 0 for each i = 1,2, ... , m. See P 2.3.2. This y is orthogonal to xo! (Why?) Consequently, by Pythagorous theorem,

70


which shows that Xo is a minimwn nonn solution. FUrther, in the above the equality is attained only when Y = O. This shows that the solution Xo is unique with respect to the property of minimum nonn. As for the norm of Xo, we note that m

IIxoll2 = < Xo, Xo > = < Xo, 2: "/iai > = i=l

m

2:

m

i'i

< Xo, ai > =

i=l

2:

i'i(}:i.

i=l

This completes the proof. Complements

2.3.1 Let V be an inner product space and x E V. Let S be a subspace ofV. (1) Show that IIx - YII is minimized over all yES at any Y = ii E S for which (x - ii) is orthogonal to S, i.e., < x - ii, Y > = 0 for all yES. (This is an alternative formulation of P 2.2.13.) (2) Suppose S is spanned by the vectors Y1. Y2, ... , Yr' Show that the problem of determining ii E S such that x - ii is orthogonal to S is equivalent to the problem of determining scalars /31. /32, ... , /3r such that x - (/31Y1 + /3?Y2 + ... + /3rYr) is orthogonal to S which, in turn, is equivalent to solving the equations

< Y1,Y1 > /31 + < Y2,Y1 > /32

< Y1,Y2 > /31 + < Y2,Y2

+ ... + < Yr,Y1 > /3r = < X,Y1 >, > /32 + ... + < Yr,Y2 > /3r = < X,Y2 >,

< YllYr > /31 + < Y2,Yr > /3? + ... + < YTlYr > /3r

=

< X,Yr >,

in unknown scalars /31, /32, .. , /3r. (3) Show that the system of equations is solvable. (The method outlined in (2) is a practical way of evaluating

Ps(x).) (4) Let V = R n with its standard inner product. Let x = (X1! X 2, ... ,xn ) and Yi = (Yil,Yi2, . . . ,Yin), i = 1,2, ... ,T. Show that the steps involved in (1), (2), and (3) above lead to the least squares theory of approximating the vector x by a vector from the vector space spanned by Y1, Y2, . . . ,Yr'


71

2.3.2 Let P 4 be the vector space of all polynomials of degree less than 4 with real coefficients. The inner product in P 4 is defined by

< Yl (x), Y2(X) > =

ill Yl (X)Y2(X) dx

for Yl (x), Y2 (x) E P 4. Let S be the vector space spanned by the polynomials Yl(X) = 1,Y2(X) = x, and Y3(X) = x 2. Determine the best approximation of the polynomial 2x + 3x 2 - 4x 3 by a polynomial from

S. 2.4. Linear Functiollals

In Section 1.7, we presented some discussion on linear functionals of vector spaces and dual spaces. Now that we have an additional structure on our vector spaces, namely, inner products, we need to reexamine the concept of linear functional in the new environment. The definitions of a linear functional and that of the dual space remain the same. If we use the inner product available on the underlying vector space, we see that the linear functionals have a nice structural form and we get a clearer understanding of the duality between the vector space and its dual space. First, we deal with linear functionals. P 2.4.1 (Representation Theorem of a Linear Functional) Let V be an inner product space and J a linear functional on V. Then there exists a unique vector z in V such that

J(x) = < x, z > for every x in V.

(2.4.1)

PROOF. If J(x) = 0 for every x in V, take z = O. Otherwise, let S.L be the orthogonal complement of the subspace S = {x E V: J(x) = O} of V. The subspace S.L has at least one non-zero vector. (Why?) Choose any vector u in S.L such that < u,u > = 1. Set z = J(u)u . The vector z is the required candidate. Since u rt s, J(u) =F 0 and hence z =F 0 and z E S.L. Let us see whether the vector z does the job. To begin with, we verify

J(z) = J(u)J(u) = < z, z > . Let x E V. Set Xl

=X

-

[J(x)/ < z, z >lz.

(2.4.2)

72


We show that x = Xl + X2 is the orthogonal decomposition of X with respect to the subspaces Sand S.l, where X2 = [j(x}j < z, Z >Jz. It is clear that X2 E S.l. Further, j(XI} = o. Consequently, Xl E S. From

(2.4.2),

o = < XI. Z > = < X, z >

- [j(x}j < z, z >lJ(z}

= < X, z

> - j(x}.

Hence j(x} = < X, z > . The uniqueness of the vector z is easy to establish. If Zl is another vector such that < x, Zl > = < X, z > for all X in V, then < x, Zl - Z > = 0 for all X in V. Hence, we must have Zl = z. This completes the proof. Thus with every linear functional j on the inner product space V, we have a unique vector z in V satisfying (2.4.1). This correspondence is an isomorphism between the vector space V and its dual space V'. This is not hard to establish. We can reap some benefits out of the representation theorem presented above. We recast P 1.7.6 in our new environment.

P 2.4.2 Let Xl, X2, ... ,Xk constitute a basis for an inner product space V. Then we can find another basis Zl, Z2, ... ,Zk for V such that if i

<x o z o>={l " ] 0 for all i and j. Further, for any

X

= < X, Zl

>

Xl

X

if i

= j, =f: j,

in V, we can write

+ < X, Z2

> X2

Also, Zi = Xi for every i if and only if basis for V.

+ ... + < X, Zk

Xl,

> Xk.

X2, ... ,Xk is an orthonormal

There are several ways of establishing the veracity of the above proposition. One way is to use the result of P 1.7.6 to obtain a dual basis of linear functionals It, h, ... ,!k for V' and then use P 2.4.1 above to obtain the associated vectors z}, Z2, . .. ,Zk. We would like to describe another way (really?) which is more illuminating. Let X be any vector in V and (6,6, ... '~k) its co-ordinates with respect to the basis Xl,X2, ... ,Xk, i.e., PROOF.


73

For each 1 :S i :S k, define Ii : V --+ F by Ji(x) = ~i' One can verify that each Ii is a linear functional. By P 2.4.1, there exists a unique vector Zi in V such that li(X) = < X, Zi > for every x in V, and for each 1 :S i :S k. One can verify that Zl, Z2, ••• ,Zk constitute a basis for the vector space V. Since the co-ordinates of Xi are (0,0, ... ,0,1,0, ... ,0) with 1 in the i-th position, we have if j if j

= i, # i.

The other statements of the proposition follow in a simple manner. As an application of the above ideas, let us consider a statistical prediction problem. Let (0, A, P) be a probability space and Xl, X2, ••. ,Xk be square integrable real random variables (defined on the probability space). Then the collection of all random variables of the form 0lXl + 02X2 + ... + 0kXk for all real scalars ot, 02, ... ,ok is a vector space V over the field R of real numbers with the usual operations of addition and scalar multiplication of random variables. We introduce the following inner product on the vector space V. For any x, y in V,

< X,y > =

E(xy),

where E stands for the expectation operator. The above expectation is evaluated with respect to the joint distribution of X and y. In statistical parlance, E( xy) is called the product moment of X and y. Assume that Xl! X2, ••• ,Xk are linearly independent. What this means, in our context, is that if 0lXl + 02X2 + ... + 0kXk = almost surely for some scalars 01, Q2, ... ,Qk then each Qi must be equal to zero. This implies that none of the random variables is degenerate almost surely and dim(V) = k. For applications, it is convenient to adjoin the random variable Xo which is equal to the constant 1 almost surely to our lot Xl, X2, .. . ,Xk if it is not already there. Let V* be the vector space spanned by xo, Xl," . ,Xk. Let p be any positive integer less than k. Let S be the vector space spanned by xo, xl, ... ,xp. Let y be any random variable in V*. Now we come to the prediction problem. Suppose we are able to observe Xl! X2, .•• ,xp. We would like to predict the value of the random variable y. Mathematically, we want to propose a linear predictor f30 + f31Xl + ... + f3pxp as our prediction of the random

°

74


variable y. Practically, what this means is that whenever we observe Xl, X2, ... ,x ' plug in the observed values into the predictor and then P declare that the resultant number is our preructed value of y. Now the question arises as to the choice of the scalars /30, /31, ... ,/3p' Of course, we all feel that we must choose the scalars optimally, optimal in some sense. One natural optimality criterion can be developed in the following way. For any choice of the scalars /30, /31, ... ,/3p,

y - /30 - /31 Xl

-

/32 X2 - .. , - /3px p

can be regarded as prediction error. We need to minimize the preruction error in some way. One way of doing this is to choose the scalars /30, /3t, ... ,/3p in such a way that

liy - /30 - /3t Xt - ... /3px pl1 2 = E(y - /30 - /3t Xt - ... -

/3px p)2 (2.4.3)

is a minimum. This kind of scenario arises in a variety of contexts. In Econometrics, Xi could denote the price of a particular stock at time period i, i = 1,2, ... ,k. After having observed Xt, X2, . .. ,xp at p successive time points, we would like to predict the price of the stock at the time point p + 1. In such a case, we take y = Xp-t-l' In the spatial preruction problem, the objective is to predict a response variable y at a new site given observations x}, X2, ... ,xp at p existing sites. The spatial prediction problem is known as kriging in geostatistics literature. Let us get back to the problem of choosing the scalars /30, /3t, ... ,f3p in (2.4.3). Observe that for any choice of scalars /30, /31, ... ,/3p, the vector f30 + /31 XI + ... + /3pxp belongs to the subspace S. The problem now reduces to finrung a vector X in S such that lIy - xII is a minimum. We have already solved this problem. See the result of P 2.2.13. The solution is given by X = Ps(y), the orthogonal projection of y onto the subspace S. Let us try to compute explicitly the orthogonal projection of y onto the subspace S. Observe that X must be of the form f30 + /31 x I + ... + /3pxp for some scalars /30, /31, ... ,/3p' Write the orthogonal decomposition of y with respect to the subspaces Sand S..L as

y = X + (y - x)

= Ps(y) + (y - x).

Observe that (y - x) E S..L if and only if < y - X,Xi > = 0 for every i = 0,1,2, . .. ,po But < y-x,Xi >= 0 means that E((y-x)xd =


75

0= E«y - f30 - f31Xl - ... - f3pXp)Xi). Expanding the expectation, we obtain the following equations in f3o, f31, .. • , f3p: f30 f30E(Xl) f3oE(X2)

+ f3 1E(xt) + f32E(X2) + + f3 1E(xi) + f32E(XIX2) + + f31E(X2Xd + f32E(x~) +

+ f3pE(xp) = + f3pE(XIXp) = + f3pE(X2Xp) =

E(y), E(X1Y)' E(X2Y),

We need to solve these equations in order to build the required predictor. These linear equations can be simplified further. We can eliminate f30 by using the first equation above, i.e.,

from each of the remaining equations. Now we will have p equations in p unknowns f31, f32, . .. , f3p:

+ f3pSl p = + f3pS2p =

SOl, S02,

where Sij = E(XiXj) - E(Xi)E(xj) = covariance between Xi and Xj, 1 :::; i,j :::; p, and SOi = E(yxd - E(y)E(Xi) = covariance between y and Xi, i = 1,2, ... , p. The problem of determining the optimal predictor of y reduces to the problem of solving the above p linear equations in p unknowns!

Complements

2.4.1 Let R3 be the three-dimensional vector space equipped with the standard inner product. Let f : R 3 ---+ R be the linear functional defined by f(x) = f(xl, X2, X3) = 2Xl + X2 - X3 for X = (Xl,X2,X3) E R3. Determine the vector z E R3 such that

f(x)

=< X,2 >, X E

R3 .

76


2.4.2 Let V be a real vector space with an inner product and T a transformation from V to V. Define a map f: V ---+ R by

f(x) = < Tx,y >, x E V for some fixed vector y E V. Show that Determine z E V such that

f(x) = < x, z >

f

is a linear functional on V.

for all x E V.

2.5. Semi-inner Product We have seen what inner products are in Section 2.1. We do come across some maps on the product space V x V of a vector space which are almost like an inner product. A semi-inner product, which is the focus of attention in this section, is one such map relaxing one of the conditions of an inner product. In this section, we outline some strategies how to handle semi-inner product spaces. All the definitions and results presented in this section are designed for vector spaces over the field C of complex numbers. The modifications should be obvious if the underlying field is that of real numbers. DEFINITION 2.5.1. Let V be a vector space. A complex valued function (., .) defined over the product space V x V is said to be a semi-inner product if it meets the following conditions:

(1) (x,y) = (y,x) for all x and y in V. (2) (x,x) 2: 0 for all x in V. (3) (O:IXl + 0:2X2,y) = O:I(Xl,y) + 0:2(X2,y) for all Xl,X2 and y in V, and

0:1,0:2

in C.

These conditions are the same as those for an inner product except for (2) which admits the possibility of (x, x) vanishing for x i= O. We use the notation (.,.) for a semi-inner product, and < .,. > for a regular inner product for which < x, x >= 0 only when x = O. In the same vein, we define the positive square root of (x, x) as the semi-norm of x and denote it by Ilxll se . Note that IIxlise could be zero for a non-zero vector. The vector space V equipped with a semi-inner product is called a semi-inner product space.


77

Most of the results that are valid for an inner product space are also valid for a semi-inner product space. There are, however, some essential differences. In the following proposition, we highlight some of the salient features of a semi-inner product space. P 2.5.2 Let (".) be a semi-inner product on a vector space V. Then the following are valid. (1) (0,0) = O.

(2) (x, y) = 0 if either IIxli se = 0 or IIYlIse = 0 (3) (Cauchy-Schwartz Inequality) I(x, y)1 ~

IIxlisellYlise for all x and yin V. (4) (Triangle Inequality) II (x + Y)lIse ~ Ilxlise + IIYlIse for all x and yin V. (5) The set N = {x E V: IIxli se = O} is a subspace of V.

PROOF. Before attempting a proof, it will be instructive to scan some parts of Section 2.1 to get a feeling about where the differences lie. To prove (1), choose a1 = a2 = 0 and Y = O. For (2), one could use the Cauchy-Schwartz inequality stated in (3). But this is not the right thing to do. In the proof of (3), we make use of the fact that the assertion of (2) is true. Suppose IIYlIse = O. Then for any complex number a,

o ~ (x + ay, x + ay) =

IIxll~e

+ a(y, x) + a(x, y) + aa(y, y)

= IIxll~e + a(y, x) + a(y, x). If a

= 'Y + i8 and (x, y) = ~ + i"1 for real numbers 'Y, 8, ~ and "1, IIxll~e + a(y, x) + a(y, x) = Ilxll~e + 2'Y~ + 28"1 ~ 0

for all real numbers 'Y and 8. Set 'Y

= O.

then

(2.5.1)

Then

for all real numbers 8. This is possible only if "1 (2.5.1). Then

=

O. Set 8

=

0 in

for all real numbers 'Y. This is possible only if ~ = O. Consequently, (x,y)=~+i"1=O.

78


To prove (3), we follow the same route as the one outlined in the proof of (2). The statement of (3) is valid when lIyllse = 0 by virtue of (2). Assume that lIyllse =f:. o. For any complex number 0:, observe that

o$ Set

0:

o

(x+o:y,x+o:y) = IIxll~e+o:(y,x)+a(x,y)+ao:llyll~e.

= -(x, Y)/lIyll~e. $

Then

IIxll~e -I(x, yW /lIyll~e -I(x, y)12 /lIyll~e + I(x, y)12 /lIyll~e,

from which the Cauchy-Schwartz inequality follows. In the above, we have used the fact that (x,y)(y,x) = l(x,y)j2. This proof is essentially the same as the one provided for the inequality in inner product spaces. Observe the role played by the statement (2) in the proof. The proof of (4) hinges on an application of the Cauchy-Schwartz inequality and is analogous to the one provided in the inner product case. Finally, we tackle the statement (5). To show that N is a subspace, let x and y belong to N, and 0: and {3 be complex numbers. We need to show that

(o:x + {3y, o:x + {3y) = O. But (o:x + {3y, o:x + {3y) = o:allxll~e + {3t1llyll~e +o:t1(x, y) +a{3(y, x) = 0, in view of (2). This completes the proof. We would like to bring into focus certain differences between inner product spaces and semi-inner product spaces. We look at an example. 2.5.3. Consider the k-dimensional Euclidean space Rk for some k 2: 2. Let 1 $ r < k be an integer. For x = (~1,6, ... ,~k) and y = (7]b7]2, ... ,7]k) in Rk, define EXAMPLE

(x,y) = 67]1 +67]2 + ... +~r7]r. The map (.,.) is a semi-inner product on the vector space norm of x works out to be

Rk.

The

The subspace N = {x E Rk : IIxli se = O} consists of vectors of the form (0, 0, . .. '~r+ 1, . .. '~k). Consequently. dim(N) = k - r.


79

The dimensions of orthogonal vector spaces under a semi-inner product may not be additive. To illustrate this point, let Xl = (1,0,0, ... ,0) and X2 = (0, ... ,0,1,0, . . . ,0) where 1 in X2 appears at the (r + 1)th position. Let < ., . > be any inner product on Rk and (., .) the semi-inner product on Rk introduced above. Let S be the vector space spanned by Xl and X2. Obviously, the dimension of the subspace S of Rk is two. Let U = {x E Rk : (x,xt) = (X,X2) = O}. Every vector X in U is of the form (0'~2,6, .. . '~k) for some real numbers ~2' 6, ... '~k. Observe that the dimension of the subspace U of R k is k - 1. The subspace U can be regarded as the orthogonal complement of the subspace S with respect to the semi-inner product (., .). But dimeS) + dim(U) =-I k. Also, S n U =-I {O}. On the other hand, if we define U = {x E Rk : < X,XI > = < X,X2 >= O}, then the subspace U is the orthogonal complement of the subspace S with respect to the inner product < ., . >. There are two ways of manufacturing an inner product from a semiinner product. Let V be a semi-inner product space with semi-inner product (., l The critical ideas are based on the subspace N defined above. Take any complement NC of N. Look up P 1.5.5. The restriction of the semi-inner product (-, .) to the vector space N C is an inner product! This is rather easy to see. Another way is to use the idea of a quotient space. Let W = V IN be the quotient space of V with respect to the subspace N . Look up Section 1.8. The space W is the collection of all cosets of N of the form X + N,x E V . We define a map < ., > on the product space W X W by

<x+N,y+N>= (x,y) for any two distinct cosets of N. It is not hard to show that the map < .,. > is an inner product on the quotient space W. We will record this fact in the form of a proposition for future reference. P 2.5.4 The map < -, . > defined above on the quotient space W is an inner product on W. It is time to define, formally, the orthogonal complement of a subspace S of a vector space V equipped with a semi-inner product (-, .). Following the procedure in the case of inner product spaces, define S; = {x E V : (x,y) = 0 for every yES} as the orthogonal complement of S with respect to the semi-inner product (. , .). As we have

80


seen earlier , S n S1. s may contain non-zero vectors. It is clear that S; is a subspace of V, and then that S n S; is a subspace. The dimension of the space S n S; could be more than one. But one could always decompose any given vector x in V into a sum Xo + Xoo with Xo E S and Xoo E S;. Let us put this down as a proposition.

P 2.5.5 For any given vector x in a semi-inner product space V, there exists Xo in Sand Xoo in S; such that x = Xo

+ Xoo·

Further, the subspaces Sand S; together span the space V. PROOF. Let Xl, X2, ... ,X r be a basis of the vector space S. The basic problem is to determine scalars 0:1,0:2, ... ,O:r such that X-O:lXl0:2X2 - ... - O:rXr belongs to S;. If we succeed in this mission, we simply let Xo = O:lXl + 0:2X2 + ... + O:rXr and Xoo = x - Xo. We will then have the desired decomposition. The scalars could be found. The condition that x - O:lXl - 0:2X2 - ... - O:rXr belongs to S; is equivalent to (x - O:lXl - 0:2X2 - ... - O:rXr, Xi) =

0

for each

i = 1,2, ... ,r.

These equations can be rewritten in an illuminating way. (Xl, Xl)O:l

(Xl, X2)0:1

+ (X2' Xd0:2 + .. . + (Xr, xdO:r = (x,xd, + (X2' X2)0:2 + .. . + (Xr, X2)O:r = (X,X2),

We have r linear equations in r unknowns 0:1, 0:2, ... ,O:r. This system of equations is analogous to the one presented in (1.6.6). We can invoke P 1.6.7 to check whether this system admits a solution. The case Iixlise = o is trivial. The decomposition is: x = 0 + x. Assume that Iixlise =1= o. Let Ui = «XI, Xi), (X2,Xi), ... ,(xr,xi)),i = 1,2, ... ,r. Suppose elUl + e2 U 2 + ... + erUr = 0 for some scalars el, e2, ... ,er. This is equivalent to r

(Xj, L€iXd i=l

=0

for each

j

=

1,2, ... ,r.

Unitary and Euclidean Spares r

This implies that

(L

81

r

€jXj,

j=l

r

L

€iXi)s = 0, from which we have

i=l

II L

€ixill

i=l

= o. Following the line of thought outlined in P 1.6.7, we need only to verify that

But r

Ci (y, Xl)

+ c2(y, X2) + ... + cr(y, Xr) = (y, I:: €iXi) = 0, i=l r

in view of the fact that

II L

€iXili = 0 and the result of P 2.5.2 (2).

i=l

Consequently, the above system of equations is consistent. Finally, the mere fact that the decomposition is possible is good enough to conclude that the spaces Sand S; together span V. This completes the proof. As has been pointed out earlier, the dimensions of Sand S; need not add up to the dimension of V. Further, we would like to point out that the decomposition is not unique. This makes it difficult to define the projection of V onto the subspace S. We need not worry about it. We can get around this difficulty. Let us consider an optimization problem in the context of a semiinner product space similar to the one considered in Section 2.2. Let V be a vector space equipped with a semi-inner product (., .) and S a subspace of V. Let X be any vector in V. We raise the question whether the minimum of IIx - zllse is attained over all z E S, or in other words, whether there exists a vector Xo E S such that inf

zES

Ilx -

zllse

=

IIx - xollse.

This has a solution and it is not hard to guess the vector. The experience we have had with inner product spaces should come handy. In fact, the same kind of proof as in the case of regular inner products works. In P 2.5.5 we showed that any given vector X E V admits a decomposition

X = Xo +xoo with Xo E Sand Xoo E S;. The decomposition is not unique as stated earlier. But any Xo and Xoo with the stated inclusion properties will do.

82


The vector Xo is indeed the solution to the optimization problem above. Note that for any vector z in S,

IIx -

zll~e

= II (x - xo) + (xo - z)ll;e

+ (xo - z), (x - xo) + (xo - z)) = lI(x - xo)ll~e + II(xo - z)ll~e + (x - Xo, Xo + (xo - z, x - xo) = II (x - xo)ll~e + lI(xo - z)ll~e

= ((x - xo)

z)

(2.5.2) This inequality establishes that the vector Xo is a desired solution to our optimization problem. The semi-inner products that appear above vanish in view of the facts that x - Xo E S; and Xo - z E S . But we must admit that the solution vector Xo need not to be unique. The solution vector Xo can be characterized in the following way. P 2.5.6 Let x be any vector in any semi-inner product space V. Let S be a subspace of V. The following two statements are equivalent. (1) There exists a vector Xo in S such that inf

zES

(2)

IIx - zllse = IIx -

xoll se .

There exists a vector Xo in S such that x - Xo E S;.

PROOF. The statement (2) implies the statement (1) from the in-

equality established in (2.5.2). Suppose (1) is true. Then for any complex number 0: and for any vector z in S.

IIx -

Xo - o:zll~e = ~

IIx - xoll~e + o:allzll~e IIx - xoll;e,

a(x - xo, z) - o:(z,x - xo)

since Xo + o:z E S. Since the above inequality is true for any scalar 0:, it follows that (x - Xo, z) = O. But this equality is true for any z in S. Hence x - Xo E S;. We consider another problem of minimization. Let S be a subspace of a semi-inner product space. Let H be a coset of S. We want to minimize IIxli se over all x in H. A solution to this problem can be characterized in the following way.


83

P 2.5.7 Let H be a coset of the subspace S of a semi-inner product space V. Suppose that there exists Xo in H such that inf

xEH

IIxlise = IIxollse.

Then Xo E S;-. PROOF. Suppose Xo is a solution to the minimization problem. Then for any scalar a and any vector z in S,

The above inequality follows if we observe that Xo + az E H. Since the above inequality is valid for any scalar a, it follows that (xo, z) = O. But this equality is valid for any z in S. Hence Xo E S;-.

Complements

2.5.1 Let (.,.) be a semi-inner product on a vector space V and N = {x E V : IIxli se = O}. Show that N.L = V. 2.5.2 Let (.,.) be a semi-inner product on a vector space V and N = {x E V : IIxlise = O}. If x + N = Y + N for x, y E V, show that IIxli se = IIYlIse. Show that for the coset x + N of N, Inf

zEx+N

IIzlise = IIxlise.

2.6. Spectral Theory The spectral theory of conjugate bilinear functionals, to be introduced shortly, can be regarded as a crowning achievement of the theory of vector spaces. We will see numerous instances of the pivotal role that the spectral theory plays in a variety of problems. Let us begin with a definition. DEFINITION 2.6.1. Let V be a vector space over the field C of complex numbers. A map K(-'·) from VxV into C is said to be a Hermitian conjugate bilinear functional if it has the following properties. (1) K(x, y) = K(y, x) for all x and y in V. (Hermitian property) (2) K(alxl + a2x2,y) = alK(xl,Y) + a2K(x2,Y) for all vectors Xl, X2, Y in V and scalars al and a2. (Conjugate bilinearity)

84


A few words of explanation are needed about the terminology used. If we look at the conditions (1) and (2) carefully, they are part of the ones that define an inner product or a semi-inner product. The only defining condition of a semi-inner product that is missing from the list above is that K(x, x) need not be non-negative. The goal of spectral theory is to express any given Hermitian conjugate bilinear functional as a linear combination of semi-inner products by breaking up the vector space V into orthogonal subspaces with respect to a specified inner product on V. Another point worth noting is the following. By combining (1) and (2), one can show that

for all vectors x, Yl, Y2 in V and scalars aI, a2- The map K(·,·) is not quite bilinear! We see that the phrase "conjugate bilinearity" is very apt to describe the property (2). The final remark is that the Hermitian property gives us immediately that K(x, x) is real for all vectors x. We now develop a body of results eventually culminating in the spectral theorem for a Hermitian conjugate bilinear functional. We always assume that the vector space V comes with an inner product < ', . >. Whenever we talk about orthogonality it is always with respect to the underlying inner product on the vector space.

P 2.6.2 Let K(·,·) be a Hermitian conjugate bilinear functional and < .,. > an inner product on a vector space. Then the following are valid. (1) The supremum of K(x, x)/ < x, x > over all non-zero vectors x in V is attained at some vector Xl in V. (2) K(y, xd = 0 for Y E (Sp(Xl)).L, where Xl is the vector under focus in (1) and Sp(xd is the vector space spanned by Xl. PROOF. Let Zl, Z2, ... ,Zk be an orthogonal basis of the vector space V and X = "YlZl + "Y2 Z2 + ... + "YkZk an arbitrary vector in V in its usual representation in terms of the given basis. Let us compute K(x, x) and < x, x >. Note that

k

k

K(x, x) = "L"L"Yi"'YjK(Zi,Zj), i=l i=l


85

and k

< X,X >

=

L 'Yi'Yi· i=l

We want to simplify the problem. Observe that for any non-zero scalar a, and non-zero x E V, K(x,x)/ < x,x >= K(ax,ax)/ < aX,ax >. Consequently,

K(x,x)/<x,x>=

sup xEV,x;.W

sup

K(x,x).

xEV,<x,x>=l

Let kij = K (Zi' Zj) for all i and j. The maximization problem can now be recast as follows. Maximize the objective function k

k

LL 'Yiijkij i=l j=l

over all complex numbers "11, "12, . . . ,'Yk subject to the condition k

Lbil 2 =

1.

i=l

The set D = {hl,'Y2, ... ,'Yk) E C k : bll 2 + b212 + ... + bkl2 I} is a compact subset of Ck. further, the objective function is a continuous function on D. By a standard argument in Mathematical Analysis, the supremum of the objective function is attained at some vector hi, "12,· .. of D. Let Xl = "Ii Zl + 'Y2Z2 + ... + 'YZZk. It is clear that the supremum of K(x, x)/ < x, x> over all non-zero x in V is attained at Xl. This completes the proof of Part (1). For Part (2), let a be any complex number and y any vector in (Sp(xd)l.. If y = 0, (2) is trivially true. Assume that y # O. Then for any complex number a, aXl + y is non-zero. In view of the optimality of the vector Xl, we have

,"In

Let us expand the ratio that appears on the left hand side above and then perform cross- multiplication On writing a = al +ia2 and K (Xl, y)

86


= 61

+ i62

for some real numbers

aI, a2,

61 , and 62 , and observing that

< Xl,Y > = 0, we have

More usefully, we have the following inequality:

This inequality is true for all real numbers al and a2. But the number that appears on the right hand side of the inequality is fixed. Consequently, we must have 61 = 62 = O. Hence K(Xl, Y) = 61 + i62 = o. Some comments are in order on the above result. The attainment of the supremum as established in Part (1) is purely a topological property. Even though the Hermitian bilinear conjugate functional is not an inner product, it inherits some properties of the inner product. If Xl E Sp(Xl) and Y E (Sp(xt})1., it is clear that < Xl, Y > = O. Part (2) says that the same property holds for K(-, .), i.e., K(Xl, y) = o. But more generally, we would like to know whether K(x, y) = 0 whenever X E 8 and Y E 81., where 8 and 81. are a pair of orthogonal complementary subspaces ofV. P 2.6.2 is just a beginning in response to this query. In the following proposition, we do a better job. P 2.6.3 Let K(·,·) be a Hermitian conjugate bilinear functional on a vector space V equipped with an inner product < .,. >. Then there exists a basis x}, X2, .. . , Xk for the vector space V such that

< Xi, Xj > = K(Xi, Xj) = 0 for all i i- j.

(2.6.1)

PROOF. Choose Xl as in P 2.6.2 and apply the result to the vector space (Sp(xt})1. with K(-' ·) and restricted to the subspace (Sp(Xl))1.. There exists a vector X2 E (Sp(Xl))1. such that

(1)

sup

K(x, x)/ < x, X> = K(X2, X2)/ < X2, X2 >,

xE(Sp(xI).L ,x;i:O

(2) K(Xl1X2) = < Xl,X2 > = 0, (3) K(u, v) = < u, v > = 0 whenever u E Sp(xt, X2) and v E (Sp(Xl,X2))1., where, as usual, Sp(Xl, X2) is the vector space spanned by the vectors Xl and X2. Reflect a little on (3) and see why it holds. Now the focus


87

of attention is the vector space (Sp(XI' X2)).L. A repeated application of P 2.6.2 yields the desired basis. It is possible that the Hermitian conjugate bilinear functional could be an inner product or a semi-inner product in its own right. What P 2.6.3 is trying to convey to us is that we can find a cormnon orthononnal basis under both the inner products K(·,·) and < .,. >. Once we have obtained a basis XI, X2, . .. ,Xk havi ng the property (2.6.1), it is a simple job to normalize them, i.e., have them satisfy < Xi, Xi > = 1 for every i. Assume now that XI, X2, . .. ,Xk is an orthononnal basis under the inner product satisfying (2.6.1). Let K(Xi' Xi) = Ai, i = 1,2, ... ,k. Assume, without loss of generality, that >'1 2: A2 2: ... 2: Ak. The numbers Ai'S are called the eigenvalues of K(·,·) with respect to the inner product < .,. >, and the corresponding vectors Xl, X2, .•• ,Xk, the eigenvectors of K(-,·) corresponding to the eigenvalues All A2, .• . ,Ak. There is no reason to believe that all Ai'S to be distinct. Let A(l)' A(2)' .•. ,A(s) be the distinct eigenvalues with multiplicities rl, r2, .. . ,rs , respectively. We tabulate the eigenvalues, the corresponding eigenvectors, and the subspaces spanned by the eigenvectors in a systematic fashion.

Eigenvalues Al

= A2 = ... = Arl = A(l) = Arl +2 = ... = Arl +r2 = A(2)

Arl +1

Corresponding eigenvectors Xl, X2,··· 'X rl

X rl +1, X r1 +2, ... ,Xrj +r2

The subspace spanned by the i-th set of vectors is denoted by E i , i = 1, ... ,s. We want to introduce another phrase. The subspace Ei is called the eigenspace of K ( ., .) correspondi ng to the eigenvalue A( i). From the way these eigenspaces are constructed, it is clear that the eigenspaces E I , E 2 , ... ,Es are mutually orthogonal. What this means is that if x E Ei and y E Ej, then < x, y > = 0 fQr any two distinct i and j. Moreover, the vector space V can be realized as the direct sum of the subspaces E I , E2, ... ,Es. More precisely, given any vector y in V, we can find Yi in Ei for each i such that Y = YI + Y2 + ... + Ys. This

88


decomposition is unique. Symbolically, we can write V = El E!1 E2 E!1" . EI1 Ea· Some more properties of eigenvalues, eigenvectors and eigenspaces are recorded in the following proposition. P 2.6.4 Let K(·, .), < following are valid.

',' >, .Vs and E/s be as defined above. The

(1) K(x, y) = < x, Y > = 0 for every x in Ei and Y in E j for any two distinct i and j.

(2) K(x, x)/ < x, x> =

A(i) for every x in Ei and for every i. (3) If X,Y E Ei for any i and < X,Y > = 0, then K(x,y) = O. (4) If x, y E Ei for any i, then K(x, y) = A(i) < x, y > . (5) If Yil, Yi2, ... ,Yir, is an orthonormal basis for the subspace Ei, i = 1,2, ... ,8, then the k vectors, Yll,Y12,··· ,Ylrl;Y2bY22, ..• ,Y2r2;'" ; Ysl, Ys2,'" ,Ysr., constitute an orthonormal basis for the vector space V.

One can establish the above assertion by a repeated application of (2.6.1) and the defining properties of K( ·,·) and < ',' >. The property (5) above has an interesting connotation. The property (2.6.1) is very critical in understanding the structure of any Hermitian conjugate bilinear functional. Once we obtain the subspaces Ei 's, one could generate a variety of orthonormal bases for the vector space V satisfying (2.6.1) by piecing together a variety of orthonormal bases for each subspace Ei. If the eigenvalues are all distinct, or equivalently, each subspace Ei is one-dimensional, we do not have such a kind of freedom. In this case, the normalized vectors Xl,X2, ..• ,Xk satisfying (2.6.1) are unique. Of course, we need to demonstrate that any orthonormal basis of the vector space V satisfying (2.6.1) arises in the way Part (5) of the above proposition outlines. Let us put that down as a proposition. Let K( ·, .), , xi's and Ei's be as defined above. Let be an orthonormal basis having the property (2.6.1). Then each Zi must belong to some subspace E j . Equivalently, every orthonormal basis of V satisfying (2.6.1) is generated as outlined in Part (5) of P 2.6.4.

P 2.6.5

Z}, Z2, '"

,Zk

PROOF. Since V = El EI1 E2 EI1 ... EI1 E s , each vector Zi in the given

orthonormal basis has a unique decomposition Zi

= Z{l + Zi2 + ... + Zis,

Zij

E Ej , j

= 1,2, '"

,8.


Let us work with the vector we have

Zl.

Since for every )

89

i= 1, < Zl, Zj

< Zll, Zjl > + < Z12, Zj2 > + .. . + < Zls, Zjs > = Since for every)

i= 1, K(ZI' Zj) =

> = 0,

o.

0, we have

This implies, from Part (4) of P 2.6.4,

Consequently,

As this is true for every) i= 1, and Zt, Z2, • .. ,Zs is an orthonormal basis, the vector A(I)Zll + A(2)ZI2 + ... + A(s)Zls must be a multiple of the vector Zl. Since

for some scalar a, we have

(A(l) - a)zll

+ (A(2)

- a)zl2

+ ... + (A(s)

- a)zls

=

o.

Now, we claim that Zlj i= 0 for exactly one) E {1, 2, ... ,s}. Suppose not. Then there are distinct indices )1,)2, ... ,)r E {1, 2, ... ,s} with r ? 2 such that Zlj. i= 0 for i = 1,2, ... ,r and Zlj = 0 for every ) E {1, 2, ... ,s} - {jl, )2, ... ,)r}. Since Zljl' ZIi2, ... ,Zljr are linearly independent, we must have A(M - a = 0 for every i = 1,2, ... ,r. But the A(i) 's are all distinct. This contradiction establishes the claim. In view of the claim, we have that ZI = Zlj. Hence ZI E E j . The same story can be repeated for the other members of the given basis. One important consequence of the above results is that the eigenvalues A(i) 's and the eigenspaces E/s are uniquely defined. If we look at the process how the normalized vectors Xl, X2, ••• ,Xk are chosen satisfying (2.6.1), we had some degree of freedom in the selection of these

90


vectors at every stage of optimization. In the final analysis, it does not matter how the vectors are selected. They lead to the same eigenvalues A(i)'S and eigenspaces Ei's. We need the following terminology. The rank of semi-inner product (', .) on V with respect to an inner product < ',' > on V is defined to be the dimension of the subspace N.L, where N = {x E V : (x, x) = O}. The orthogonal complement N.L is worked out with respect to the inner product < ',' >. Now we are ready to state and prove the spectral theorem for Hermitian conjugate bilinear functionals. The main substance of the spectral theorem is that every Hermitian conjugate bilinear functional is a linear combination of semi-inner products. More precisely, we want to write any given Hermitian conjugate bilinear functional K(·,·) in the following form: (2.6.2) having the following features. (In the background, we have an inner product < " > on the vector space V.) (1) The numbers 81 , 82, . .. ,8m are strictly decreasing. (2) The semi-inner products (-, 'h, (', ·h, ... ,(', ')m are all of nonzero ranks. (3) The subspaces F 1 , F2, ... ,F m are pairwise orthogonal, where Fi = Nt and Ni = {x E V : (X,X)i = O}. (The orthogonality is with respect to the inner product < .,. > .) (4) For any pair of vectors x and y in V, we have

< X,y >= (x,yh + (x,yh + ... + (x,y)m' In abstract terms, when we say we have a spectral form for a Hermitian conjugate bilinear functional with respect to a given inner product, we mean a form of the type (2.6.2) exhibiting all the features (1), (2), (3), and (4) listed above. For such a form, we demonstrate that the vector space V is the direct sum of the subspaces F 1 , F2, ... ,F m. Suppose Fm+l is the subspace of V orthogonal to each of F 1 ,F 2, ... ,Fm. We show that the subspace F m+l is zerc~·dimensional. Observe that the vector space V is the direct sum of the subspaces F},F 2, ... ,Fm+I' Any vector x in V has a unique decomposition x=

Ul

+ U2 + ... + U m +l,


91

with Ui E Fi. By (4) above,

< x,x > = (x,xh + (x,xh + ... + (x,x)m = =

(Ut,Udl

< Ul, Ul

+ (U2,U2h + ... + (um,um)m > + < U2, U2 > + ... + < Um , Um > .

On the other hand,

Consequently, < Um+l, Um+l >= O. Since x is arbitrary, it follows that the subspace F m+l is zero-dimensional. In the following result, we identify explicitly the spectral form of a Hermitian conjugate bilinear functional. P 2.6.6 (Spectral Theorem). Let K(·,·) be a Hermitian conjugate bilinear functional and < " . > an inner product on a vector space V. Then there exist semi-inner products (', ')i, i = 1,2, ... ,s of nonzero ranks, and distinct real scalars A(i)' i = 1,2, ... ,s such that the following hold.

(1) The subspaces F i , i = 1,2, ... ,s of V are pairwise orthogonal, where Fi = Nt and Ni = {x E V: (X,X)i = O}. (2) For every x, y E V,

< X,y > = (x,yh + (x,yh + ... + (x,y)s' (3) K(x, y) = A(1)(X, yh

+ A(2)(X, yh + .,. + A(s) (x, y)s.

PROOF. The spade work we have carried out so far should come in handy. It is not hard to guess the identity of the scalars A(i) 'So We need to identify precisely the subspaces F/s before we proceed further. Let the distinct eigenvalues A(i) 's and the eigenspaces E/s be those as outlined above. We prove that Fi = Ei for every i. We define first the semi-inner products. Let x, y E V. Since the vector space V is the direct sum of the vector spaces E 1 , E2,'" ,E s , we can write

x= y=

+ U2 + ... + Us, Ul + U2 + .. . + v s , Ul

92


with Ui and Vi in E i , in a unique way. For each 1 :S i :S s, define

One can check that ("')i is a semi-inner product on the vector space V. Next we show that (X,X)i = 0 for x in V if and only if x E Ef-. If (x, X)i = 0, then < Ui, Ui >= 0 which implies that Ui = O. Thus we observe that x =

UI

+ U2 + ... + Ui-l + Ui+l + ... + Us·

Consequently, x E Et-. (Why?) The converse follows in the same way if we retrace the steps involved above. Thus we identify the null space N i of the semi-inner product (-, .) i as Et-. Hence Nt- = E i . In view of this identification, (1) follows. By the very definition of the semi-inner products, we have for any x, y E V,

< x,y > = < U}'VI > + < U2,V2 > + ... + < us,Vs >

=

(x,yh

+ (x,yh + ... + (x,Y)s.

This establishes (2). Finally, by P 2.6.4 (4), we have

+ K(U2, V2) + ... + K(u s , VS) = A(I) < UI,VI > +A(2) < U2,V2 > + ... + A(s) < Us,Vs > = A(I)(X, yh + A(2)(X, yh + ... + A(s) (x, Y)s,

K(x, y) = K(uI, VI)

from which (3) follows. It is clear that each semi-inner product introduced above is of non-zero rank. The set P(I), A(2), ... , A(s)} of eigenvalues is called the spectrum of K(·, .). In the following result, we show that the representation given above is unique.

P 2.6.7 Let K(·,·) be a Hermitian conjugate bilinear functional and < .,. > an inne~ product on a vector space V. Let A{1) > A(2) > ... > A(s) be the eIgenvalues and E 1 , E 2 , ... , Es the corresponding eigenspaces of K ( ., .). Suppose


93

is a spectral form of K C,, .) for some real numbers 81 > 82 > .. . > 8m and semi-inner products (., -fl, C" .)~, . . . , (., embodying the three features (1), (2), and (3) of the spectral theorem P 2.6.6 outlined above. Then m = s, 8i = A(i), and (., .): = (., ')i for every i , where the semiinner product C,,·) is the same as the one defined in P 2.6.6.

·rm

PROOF. The ideas are essentially contained in the discussion preced-

ing P 2.6.6. Let Pi = {x E V : (X,Xfi = O} and G i = Pr for each 1 ::; i ::; m. By hypothesis (1), G l , G 2 , . . . , G m are pairwise orthogonal. First, we show that the vector space V is the direct sum of the subspaces G l , G 2 , •• • , G m . Let G m + l be a subspace of V orthogonal to G l , G 2 , •• • , G m so that

Let x E V. We can write x =

Xl

+ 1.

with Xi E G i , 1 ::; i ::; m < x,X > -

=

+ X2 + ... + Xm + Xm+l (x,x)~

+ ... + (x,xfm (Xl,XSl + (X2,X2f2 + ... + (xm,xmfm < X}, Xl > + < X2 , X2 > + ... + < Xm,Xm > . CX,Xfl

+

By (2),

But <X,X>= <Xl,Xl>+<X2,X2>+'"

+ < Xm,Xm > + < Xm+l,Xm+l >, which implies that < Xm+l, Xm+l > = O. Since x is arbitrary, it follows that G m + l is zero-dimensional. If Yil, Yi2, ... , Yiri is an orthonormal basis of G i , i = 1,2, ... , m , then Yl},Y12, • ..

,Ylrl,Y2},Y22,·· · ,Y2r2" " ,Yml,Ym2,··· ,Ymr",

is an orthonormal basis of V. Also < Yij,Yrt >

=

0


94

for every (i, j) =f (r, t). By P 2.6.5, each Yij belongs to some E r . This immediately leads to the verification of the result. There is an alternative way of writing down the spectral form of a Hermitian conjugate bilinear functional. In this form, the functional is written explicitly in terms of the underlying inner product. P 2.6.8 Let K(·,·) be a Hermitian conjugate bilinear functional and < ., - > an inner product on a vector space V. Then there exist orthonormal vectors Xl, X2, ... , Xr in V and real numbers >'1 ~ >'2 ~ ... ~ >'r such that for any pair X and Y of vectors in V, we have K(x,y)

=

>'1 < X,Xl >< +>'r

where r

~

> + >'2 < X,X2 >< X2,Y > + ... < X, Xr >< Xr, Y >, (2.6.3) Xl,Y

dim(V).

PROOF. Choose orthonormal vectors Xl, X2, . .. , Xk in V satisfying (2.6.1), where k = dim(V). Let >'i = K(Xi, Xi) for each i. For given vectors X and Y in V, write the decompositions of X and Y as X

=

< X, Xl > Xl + < X, X2 > X2 + ... + < X, Xk > Xk

Y

=

< Y,Xl > Xl + < Y,X2 > X2 + ... + < Y,Xk > Xk.

and

Consequently, k

K(x,y)

= K ( L < X,Xi > Xi, i=l

k

L

< y,Xi > Xi)

j=l

k

= L>'i < X,Xi >< Xi,y >. i=l

In the above representation, we omit those >'i'S which are zero. Thus we have the desired representation (2.6.3). The statement of P 2.6.8 can be reworded in the following way.

P 2.6.9 Let K(·,·) be a Hermitian conjugate bilinear functional and < .,. > an inner product on a vector space V. Then there exist vectors Xl,X2,··· ,Xk in V and real numbers >'1,>'2, .. . ,>'k such that


95

for any pair x and Y of vectors in V, we have

K(x,y) = Al
< XbY > +A2 < X , X2 >< X2,Y > + .. . +Ak < X,Xk >< Xk,Y >

X,Xl

and

< X,Y >

=

< X,Xl >< Xl,Y > + < X , X2 >< X2 , Y > + (2.6.4) + < x, Xk >< Xk , Y > .

The properties (2.6.1), (2.6.3) and (2.6.4) are all equivalent. The second part of Property (2.6.4) is equivalent to the fact that Xl, X2, . .. ,Xk should constitute an orthonormal basis of the vector space V . See P 2.2.7. Consequently, the properties (2.6.3) and (2.6.4) are equivalent. It is clear that (2.6.1) and (2.6.4) are equivalent. It is time to take stock of what has been accomplished. One crucial point we need to discuss is that what happens to the spectral form when the Hermitian conjugate bilinear functional K(·, ·) is itself a semi-inner product or, more restrictively, an inner product. In that case, we will get an additional bonus. If the Hermitian conjugate bilinear functional is a semi-inner product, then all its eigenvalues are non-negative. (Why?) IT the Hermitian conjugate bilinear functional is an inner product , then all its eigenvalues are positive. (Again, why?) The spectral representation of a Hermitian conjugate bilinear functional K(·,·) in (2.6.2) naturally depends on the inner product < ., - > chosen. It is, therefore, of some interest to examine how the representations differ for different choices of the inner product. The following theorems shows that in any representation the number of positive, negative and zero eigenvalues are the same, while the actual eigenvalues and the corresponding eigenvectors may not be the same. We give a representation of K( ·,·) in terms of some basic linear functionals which brings into focus the stated facts above. P 2.6.10 Let K(· ,·) be a Hermitian conjugate bilinear functional on a vector space V. Then there exist p + q (s dim(V)) linearly independent linear functionals L l , L2 , . . . ,Lp +q defined on V such that for every pair X and Y of vectors in V , we have p

K(x, y)

= I: Li(X)Li(Y) i=l

q

I: Lp+j(x)Lp+j(Y)· j=l

(2.6.5)

96


Moreover, the numbers p and q in (2.6.5) are unique for a given K( ·, .), while the choice of linear functionals is not unique. PROOF. Consider the representation of K( ·,·) given in (2.6.4). Assume, without loss of generality, that AI, A2, ... ,Ap are positive Ap+l' Ap+2, . . . ,Ap+q are negative, and Ap+q+b Ap+q+2, . .. ,Ak are zeros. Let J-Lp+j = -Ap+j,j = 1,2, ... ,q. Then for any pair x and Y of vectors in V, p

K(x,y) = LAi < X,Xi >< Xi,Y > i=l q

- LJ-LP+j < x,xp+j >< xp+j,Y > j=l p

= L < x, (Ai)1/2xi >< (Ad 1 / 2x i' Y > i=l q

< x, (J-LP+j)1/2 xp+j >< (J-Lp+j )1/2xp+j , Y >

- L j=l p

q

= LLi(X)Li(Y) - L Lp+j(x)Lp+j(Y)' j=l

i=l

where Li(x) =< x,(Ad1/2xi >, i = 1,2, ... ,p and, similarly we have Lp+j(x) = < x, (J-Lp+j)I/2 xp+j >, j = 1,2, ... ,q. Observe that for each i, LiO is a linear functional on the vector space V . Since Xi 'S are orthononnal, the linear functionals Li (.), i = 1,2, ... ,p + q are all linearly independent. Now let us settle the question of uniqueness. Suppose we have two representations of K(·, ·) given by p

K(x, y)

= L Li(X)Li(Y) - L Lp+j (X) Lp+j (y) i=l

and

q

r

j=1 s

K(x,y) = LMi(X)Mi(Y) - LMr+j(x)Mr+j(y) i=l

j=l


97

for all x and y in V, and for some sets {Li} and {Mi} of linearly independent linear functionals on V. Set x = y. Then for any x in V,

K(x, x) = =

Suppose

l'

p

q

i=1

j=1

L Li(X)Li(X) - L Lp+j(x)Lp+j(x) r

s

i=1

j=1

L Mi(X)Mi(X) - L Mr+j(x)Mr+j(x).

(2.6.6)

(2.6.7)

> p. Consider the following linear equations, Li(X) =0, i= 1,2, ... ,p, Mr+j(x) = 0, j = 1,2, ... ,s,

(2.6.8)

in x. For any x satisfying (2.6.8), K(x, x) is non-positive as per the representation (2.6.6) and is non-negative as per the representation (2.6.7). This apparent anomaly will not arise if

Mi(X) = 0, i = 1,2, ... ,T, Lp+j(x) = 0, j = 1,2, ... ,q,

(2.6.9)

whenever the vector x satisfies equations (2.6.8). What this means is that each of the linear functionals M 1, M 2, ... ,Mr is a linear combination of the functionals £1, £2,"" L p, Mr+ll M r+2, ... ,Mr+s' See Complement 1.7.2. As a consequence, each of the linear functionals MI, M 2, ... ,Mr , M r+1, ... ,Mr+8 is a linear combination of the funetionals L 1, L2, .. . ,Lp, Mr+l' M r+2, .. . ,Mr+s . This is not possible since M 1, M 21 ··· I Mn M r+11 ··· I Mr+s are linearly independent and p < T. Thus we must have 1" ::; p. By similar argument, we can show that p ::; T. Hence p = T. In a similar vein, it follows that q = s. This completes the proof. The numbers p + q and p - q are called the rank and signature of the Hermitian conjugate bilinear functional K(-, .), respectively. If we recall how we obtained the first eigenvalue >'1 of the Hermitian conjugate bilinear functional, we identify >'1 as the largest value of the ratio K(x, x)/ < x, x> as x varies over all non-zero vectors of V. The remaining eigenvalues also do have such optimality properties. In the

98


following theorem, we characterize the intermediate eigenvalues. Before we present the theorem, we would like to rei terate the basic framework under which we operate. Let Al 2: A2 2: ... 2: Ak be the eigenvalues of the Hermitian conjugate bilinear functional K( ·, ·) with respect to an inner product < -, - >, and Xl, X2, .. • ,Xk the corresponding eigenvectors which form an orthonormal basis for the vector space V . All these facts have been handed. down to us from P 2.6.2 and the discussion that ensued..

P 2.6.11 (Minimax Theorems) Let Ms be the vector space spanned by the first s eigenvectors of K(-'·) and M- s the vector space spanned by the last s eigenvectors for each 1 :::; s :::; k. The following hold: (1) inf K(x, x)/ < X, X > = As and the infimum is attained xEM.,x,eO

at

= XS. sup K(x, x)/

X

>=

Ak-s+1

and the supremum is

attained at x = Xk-s+1. (3) inf sup K(x, x)/ < x, x > =

Ak- s +1

where the infimum is

(2)

= As where the supremum is S

xES ,x,eO

taken over all subspaces S of V with dim(S) 2: s, and the supremum is attained at S = Ms. PROOF. We begin by proving (1). Let x be any non-zero vector in the vector subspace Ms. Write down its representation in terms of the given orthonormal basis of Ms:

x=

0lXl + 02X2 + ... + OsXs,

for some scalars o}, 02, ...

K(x, x)

< x, x> =

_

,Os .

_

We compute

_

_

[01 0 1 A 1 +02 0 2 A2+ · · .+OsOsAsl![0101 +020:2+ .. . +osO:s].

The above computation indicates that we are taking a weighted average of the numbers Al 2: A2 2: ... 2: As with the non-negative weights 010:}' 020:2,·· . , O:S0:8· It is clear that the weighted average is always


99

~ >'s. It is also clear that when x = x s, the weighted average is exactly equal to >'s. This proves (1). A similar argument establishes the truth of (2). Let us now prove (3). Let S be any subspace with dimeS) 2: s. Observe that

K(x, x) > :eES,:e:FO < x, X > sup

> >

sup xESnM k -0+1 ,X:F O

inf

xESnMk -0+1 ,x:F O xE M

inf

k_ .+l,X:F 0

K(x,x)/ < x,x > K(x,x)/ < x,x >

K(x,x)/ < x,x > =

(2.6.10) Ak-s+l

by (1).

The first inequality above holds as long as S n M k - s + 1 =/:. {O}. This is so, in view of the fact that

+ dim(M k- s+1) - dimeS + Mk-s+d 2: (s) + (k - s + 1) - k = 1.

dimeS n Mk-s+t} = dimeS)

Taking the infimum of (2.6.10) over all subspaces S of dimensions 2: s, we obtain inf sup K(x,x)/ < x,x > 2: Ak-s+1' S xES,x:FO

The subspace S = M- s is one such subspace with dimension equal to s and for which, by (2), sup

K(x, x)/ < x, x> =

Ak-s+l.

:eES,X:FO

Consequently, the infimum above is attained at the subspace S = M- s • This completes the proof of (3). In order to prove (4), repeat the above argument used in the proof of (3) with only one modification: in the chain of inequalities, use S n M-(k-s+1) instead of S n M k - s+ 1 ' There is another line of thinking when it comes to analyzing a Hermitian conjugate functional K (', .) defined on a vector space V. Suppose Q is a subspace of V. One can restrict the given bilinear functional K(·, ·) to the Cartesian product Q x Q. The restriction still remains a Hermitian conjugate bilinear functional on the vector space Q. Now the question arises as to the type of relationship that prevails between the eigenvalues of K ( " .) as defined on V x V and the eigenvalues of K(·, .) as defined on Q x Q. In the following proposition, we establish some inequalities.

100


P 2.6.12 Let K(-'·) be a Hermitian conjugate bilinear functional on a vector space V of dimension k. Let >'1 2 >'2 2 ... 2 >'k be its eigenvalues. Let Q be a subspace of V of dimension t. Let ILl 2 IL2 2 ... 2 ILt be the eigenvalues of K(-'·) restricted to Q x Q. Then the following inequalities hold: (1) >'s 2 j.Ls for s = 1,2, ... ,t; (2) j.Lt-s+1 2 >'k-s+1 for s = 1,2, ... ,t. PROOF. This result follows by an application of minimax theorems established above. By Part (3) of P 2.6.11 (Minimax Theorems), ILt-s+1

=

inf

sup

seQ xES,x,eO

K(x, x)! < x, x >,

where the infimum is taken over all subspaces S of the vector space Q with dim(S) 2 s . But this infimum, 'nf ) seQ

K(x,x).

sup xES,x,eO

< x, X >

2 )nf

sup

K(x,x)

sev xES,x,eO

< x, X >

= >'k-s+1,

where the infimum on the right hand side of the above inequality is taken over all subspaces S of the vector space V with dim(S) 2 s. This is true for any s = 1,2, ... ,t. This proves Part (2) above. Using Part (4) of P 2.6.11, one can establish Part (1) above. The following is a simple consequence of the above result. This result is usually known as the interlace theorem for eigenvalues. P 2.6.13 (Interlace Theorem) Let K(·, ·) be a Hermitian conjugate bilinear functional on a vector space V of dimension k . Let >'1 2 >'2 2 ... 2 >'k be the eigenvalues of K(·, .). Let Q be a subspace of V of dimension (k - 1). Let ILl 2 IL2 2 ... 2 ILk-1 be the eigenvalues of K(·,·) restricted to the subspace Q. Then

>'1 2

ILl

2 >'2 2

IL2

2 ... 2

ILk-1

2

>'k .

COlnplemellts

2.6.1 Let V = C 3 and consider the following function on the product space V x V:

+ (1 + i)6172 + (1 - i)6173 + (1 - i)6171 + 6173 + (1 + i)6171 + ~3T]-}. + 6173

K(x, y) = 6171


101

for x = (~1,6,6) and Y = (171,"72,173) E V.

(1) Show that K(·,·) is a Hermitian conjugate bilinear functional onV. (2) Obtain the spectral form of K(·, .).

2.7. Conjugate Bilinear Functionals and Singular Value Decomposition It may be puzzling to the reader that the phrase "Conjugate bilinear functional" is cropping up again in a new section. Are we not done with it in Section 2.6? The functionals we are entertaining in this section are defined in a more general framework than hitherto considered. You will see the difference when we introduce the definition. DEFINITION 2.7.1. Let VI and V 2 be two vector spaces both over the field C of complex numbers. A map B( ·, ·) from VI x V 2, the Cartesian product of Viand V 2, into C is said to be a conjugate bilinear functional if

(1) B(OIXI + 02X2, y) = oIB(XI' y) + o2B(X2' Y), (2) B(x, /3IYl + /32Y2) = i3 I B(x, yt} + i32 B (X, Y2) hold for every X,X},X2 in VI; Y,Yl,Y2 in V 2; and O},02,/3I,/32 in C. Some remarks are in order. In the above framework, the vector spaces V 1 and V 2 need not be identical. If V I = V 2 and B (., .) is a Hermitian conjugate bilinear functional on VI, then B(· ,·) is a conjugate bilinear functional on VI x VI in the sense portrayed above. If VI = V2 and B(·,·) is a conjugate bilinear functional on VI x V 2, it is not necessary that B(·,·) be Hermitian. Try an example. In what follows, we choose and fix an inner product < .,. >1 on the vector space Viand an inner product < .,. >2 on V 2. We establish what is called the Singular Value Decomposition of the conjugate bilinear functional B(·, ·) with respect to the inner products < ., . >1 and < ., . >2. This decomposition is in the same spirit as the spectral fonn of a Hermitian conjugate bilinear functional on a vector space. In the proof of the singular value decomposition, we make use of the spectral form of a certain Hermitian conjugate bilinear functional on a vector space. Even though the singular value decomposition sounds

102


more general than the spectral form but it is a derivative of the spectral form. P 2.7.2 (Singular Value Decomposition) Let Be, .), < .,. >1 and < .,. >2 be as defined above. Then there exist positive real numbers 0"1 ~ 0"2 ~ ... ~ O"r with r ~ min{dim(Vd, dim(V2)}, orthonormal vectors Xl, X2, ... ,Xr in VIand orthonormal vectors Yl, Y2, ... ,Yr in V 2 such that for any two vectors X in VIand Y in V 2,

B(x,y) = 0"1 <X,Xl>11 + < X, X2 >1 < X2, ",(y) >1 + ... + < X,Xk >1< Xk,"'(Y) >1,

which, with the help of (2.7.1), becomes

< x,,,,(y) >1

=

< X,Xl >1< ~(xd,Y >2 + < X,X2 >1< ~(X2)'Y >2 + .,. + < X, Xk >1 < ~(Xk)' Y >2 . (2.7.4)

We need to procure an orthonormal set of vectors for the vector space V 2. A natural candidate is:

Note that by (2.7.3), for i,j E {1,2, ... ,r}, if '{,

=),

if i =1= j. Thus Zl! Z2,' .. ,Zr are orthogonal. Let Ui be the positive square root of ut and Yi = (Ui)-1 Zi, i = 1,2, ... ,r. Then Y}, Y2, .. . , Yr is a set of orthonormal vectors in V 2 • FUrther, Zi = 0 if i = l' + 1, r + 2, ... ,k.

104


Using (2.7.1) and (2.7.4), the puzzle is solved. For any vector x in VI and yin V2, we have

B(x, y)

= < x, TJ(Y) >1 = < X,X1 >1< Z},Y >2 + < X,X2 >1< Z2,Y >2 + ... + < X,Xk >1< Zk,Y >2 - CT1 < X,X} >1< YI,Y >2 + CT2 < X,X2 >1 < Y2,Y >2 + ... + CTr < x, Xr >1 < Yn Y >2 .

This completes the proof. In the development of a spectral form for a Hermitian conjugate bilinear functional defined on a vector space, the critical point was to demonstrate the existence of an orthonormal basis for the vector space satisfying (2.6.4). In the context of a conjugate bilinear functional defined on the Cartesian product of two vector spaces, we can exhibit two orthonormal bases one for each of the underlying vector spaces satisfying a property similar to (2.6.4). Such bases are called canonical bases. Let us state formally what was discussed above.

P 2.7.3 (Canonical Bases of Two Vector Spaces) Let (VI, < -,' >d and (V2' < -,' >2) be two inner product spaces with dimensions k and m, say, respectively. Let B( ·,·) be a conjugate bilinear functional defined on V I X V 2. Then there exists an orthonormal basis x}, X2," . ,Xk for the vector space VI and an orthonormal basis YI, Y2,'" ,Ym for V 2 such that B(Xi,Yj)

=/=j,

= 0

for all i

=/= 0

for j = i.

(2.7.5)

PROOF. Most of the spade work needed for this result was already done in the proof of P 2.7.2. We have already obtained orthonormal vectors Xl, X2, ... ,Xr in V I and orthonormal vectors YI, Y2, ... ,Yr in V 2 such that (2.7.5) is satisfied for all i,j E {1,2, . .. ,r}. Extend Xl, X2, ... ,X r to an orthonormal basis Xl, X2, ... ,Xk of V I, and extend the same courtesy to the set YI, Y2, ... ,Yr' The property (2.7.5) covers all the basis vectors in view of the singular value decomposition of the bilinear form Be, .).

105


It is time we name the numbers 0"1 ~ 0"2 ~ ... ~ O"r > O. These numbers are called the singular values of the conjugate bilinear form B(-, .). Further, the vectors Xi in VI and Yi in V2 are called canonical vectors associated with the singular value O"i, i = 1,2, ... ,T. In the context of a Hermitian conjugate bilinear functional defined on a vector space, the eigenvalues were obtained as a solution to a certain optimization problem. The singular values also have a similar optimality property. P 2.7.4 The largest singular value 0"1 of a conjugate bilinear functional B(-,·) defined on the Cartesian product VI x V 2 of two vector spaces has the following property:

0"1 =

IB(x, Y)I/[IIxlh liyli2j.

sup xEV l ,x#O,yEV 2 ,y#O

Moreover, the supremum above is attained at X = Xl and Y = Yl, where Xl and Yl are a set of canonical vectors associated with the singular value 0"1PROOF. This result is a simple consequence of the singular value

decomposition of B(-, -). For any vectors

X

in VI and Y in V 2, we have

B(x,y) = 0"1 < X,Xl >1< Yl,Y >2 +0"2 < X,X2 >1< Y2,Y >2

+ ...

+O"r < X,X r >1< Yr,Y >2-

Expand each of X and Y with respect to their respective orthonormal bases stemming from the singular value decomposition:

+ < X, X2 >1 X2 + ... + < X, Xk >1 Xk = nlxl + n2x2 + ... + nkxk, say, = < y, Yl >2 Yl + < y, Y2 >2 Y2 + _.. + < y, Ym >2 Ym = f3lYl + f32Y2 + ... + f3mYm, say.

X = < X, Xl >1 Xl Y

By the Cauchy-Schwartz inequality, it now follows that r

i=l r

r

k

m

:::; 0"1 (L IniI2)1/2(L lf3iI 2)1/2 i=l i=l :::; 0"1 (L IniI2)1/2(L lf3jI2)1/2 = 0"1 Ii x lil liyli2i=l

j=l

106


Consequently, the supremum under discussion is ::; B(Xl' Yl) = 0"1. This completes the proof.

0"1.

It is clear that

Complements

2.7.1 Let Vt defined by

= R2 and V2 = R3.

Let B be the functional on VI XV2

for x = (6,6) E VIand Y = ("11, 'TJ2, '(/3) E V 2· (1) Show that B is a conjugate bilinear functional. (2) Obtain the singular value decomposition of B.

Note: The material covered in this Chapter is based on Halmos (1958), Rao and Mitra (1968a, 1971a, 1971b) and Rao (1973c).

CHAPTER 3 LINEAR TRANSFORMATIONS AND MATRICES In Chapters 1 and 2, we have looked at entities called linear funetionals. They were maps from a vector space into the associated field of the vector space. The notion of a linear functional can be extended to a more general setting. In this chapter, we study linear transformations from one vector space into another and their representation in matrix form.

3.1. Preliminaries Let V and W be two arbitrary vector spaces over the same field F. A map T from V to W, written as T : V ---+ W, is said to be a linear transformation if T(ax

+ f3y)

= aT(x)

+ f3T(y)

for every a and f3 in F, and x and y in V. There are other names used for the type of map introduced above: a linear operator, a homomorphism, or a linear mapping. In the sequel, any linear transformation is simply called a transformation, and a general transformation will be referred to as a map. The vector space V is called the domain of the transformation T. If S is a subspace of V, we can restrict the map T to S, and the restriction is usually denoted by TIS. For the restricted map, the domain is obviously the space S. With T : V ---+ W, one can associate two sets R(T)

= {T(x)

E W : x E V}, K(T)

= {x

E V : T(x)

= O}.

The set R(T) is a subset of the vector space Wand is called the range of the transformation T . FUrther, one can show that R(T) is a subspace of 107

108


w.

The set K(T) is a subset of the space V and is called the kernel of the transfonnation T. One can show that K(T) is a subspace of V. To round up the discussion on the range and kernel of a transformation, let us introduce two more notions. The dimension of the subspace R(T) is called the rank of the transformation T and is usually denoted by peT). The dimension of the subspace K(T) is called the nullity of the transformation T and is usually denoted by v(T). If the transformation T is such that R(T) = W, then the transformation is labeled as onto. Otherwise, it is labeled as an into transformation. T is said to be an isomorphism if T is one-to-one and onto. If T is an isomorphism, we declare that the vector spaces V and Ware isomorphic. Let us state a few facts surrounding these notions.

P 3.1.1 Let T: V -+ Wand KC be a complement of the subspace K(T) in V. Then the following hold. (1) The transformation TIKc, i.e., T restricted to KC, has the range R(T), i.e., R(TIKC) = R(T). (2) The transformation TIKc : KC -+ R(T) is one-to-one and onto. (3) dim(KC) = peT) = dim(V) - veT). PROOF. It is obvious that R(TIKC) C R(T). Let y E R(T). There exists a vector x in V such that T(x) = y. Since K(T) EB KC = V, we can write x = Xl + X2 with Xl E K(T) and X2 E K C. As y = T(x) = T(XI)+T(X2) = T(X2) andx2 E KC, we have y E R(TIKC). This proves (1). For (2), we need to show that the map TIKc is one-to-one. Let u and v be two vectors in KC such that T(u) = T(v). As T(u - v) = 0, we have u - v E K(T). The vector u - v also belongs to K C. Hence u - v = 0. This shows that TIKc is one-to-one. We have already seen that dim(V) = dim(K(T)) + dim(KC). (See P 1.5.8.) Since K C and R(T) are isomorphic, (3) follows. The above result can be rephrased as follows . Given any transformation from one vector space V to another W, we have two subspaces K(T) and R(T) with K(T) being a subspace of V and R(T) a subspace of W. The dimensions of these subspaces match the dimension of the vector space V, i.e., dim(K(T)) + dim(R(T)) = dim(V) . We look at some examples. EXAMPLE 3.1.2. Let F be any field. Define a map T from F3 to F2 by T(';1 ,6,6) = (6,6) for (';b6,6) in F3. Note that F3 and F2 are

109

Linear 'l'ransformations and Matrices

three-dimensional and two-dimensional vector spaces, respectively, over the same field F. Further, T is a linear transformation. EXAMPLE 3.1.3. Consider the vector space P n of all polynomials of degree less than n with complex numbers as coefficients. Then the map (Differential operator) T defined by, n-l

n-l

n-l

T(L~iXi) = (d/dx)(L~iXi) = (Li~iXi-l) i=O

i=O

i=l

n-l

is a linear transformation from P n to P n-}, where mial in x of degree less than n with

~o, 6,

...

'~n-l

I: ~ixi is a polyno-

i=O

E C.

EXAMPLE 3.1.4 . Consider the vector space P n of all polynomials of degree less than n with complex coefficients. Then the map (Integral operator) S defined as

n-l n-l S(L ~ixi) = (L(~d(i + l)))xi+1 i=O

i=O

is a linear transformation from P n to P n+l. EXAMPLE 3.1.5. Let V and W be arbitrary vector spaces over the same field. Let WI, W2,'" ,Wr be any r vectors in Wand YI, Y2,··· ,Yr any r linear functionals on V. Then the map T defined by,

is a transformation from V to W. 3.1.6. Let S be a subspace of a vector space V. Let SC be a complement of Sin V. Then any vector x in V can be written as x = Xl + X2 with Xl E Sand X2 E SC in a unique way. Define a map T from V to S by, T(x) = Xl' Then T is a transformation from V onto S. Such an operator is called a projection onto S along SC. We have come across this notion when we were discussing inner product spaces and orthogonal complements of subspaces. We do not have to have an inner product on the vector space in order to have the notion of projection feasible. EXAMPLE

110


Finally, we state some results which connect the notions of range, kernel and isomorphism. P 3.1.7 Let T : V ---+ W . Then the following are valid. (1) The transformation T is an isomorphism if and only if dim(K(T)) = 0, i.e., K(T) contains only the zero vector, and R(T) = W. (2) If the vector spaces V and W have the same dimension, then K(T) = {O} if and only if R(T) = W. (3) If T is an isomorphism from V to W, and S is a subspace of V, then TIS is an isomorphism from S onto R(TIS). PROOF. Part (1) is easy to establish. For part (2), one can use the dimensional identity, dim(V) = dim(K(T)) + dim(R(T)). The proof of part (3) is trivial.

Complements 3.1.1 Let V be the set of all complex numbers regarded as a vector space over the field of real numbers. Let T be the map from V to V defined by T(x + iy) = x - iy, the conjugate of the complex number x isomorphism of V.

+ iy

E V. Show that T is an

3.2. Algebra of Transformations In this section, we look at the collection of all transformations from one vector space to another. This collection can be endowed with a structure so as to make it a vector space. Of course, the underlying structure of the vector space involved plays a crucial role in passing its features to the collection. In the sequel, assume that all vector spaces are over the same field F. DEFINITION 3.2.1. Let T : V ---+ Wand S : V ---+ W . Define a map T+S by, (T + S)(x) = T(x) + S(x) , x E V. For each a in F, define a map aT by,

(aT)(x)

= aT(x), x

E V.

Linear

nunsfo~ations

111

and Matrices

It is clear that the maps T + S and aT are transformations. Thus addition and scalar multiplication of transformations are naturally available on the collection of all transformations. The following proposition clearly spells what these operations mean from a structural point of view. P 3.2.2 Let L(V, W) be the collection of all transformations from the vector space V to the vector space W. With the operations of addition and scalar multiplication of transformations defined above, L(V, W) is a vector space over the same field F. One can define one more operation on transformations, namely, composition, subject to some compatibility conditions. DEFINITION 3.2.3. Let

T :V

~

Wand S : W

~

U. Define a map

ST:V~Uby

(ST)(x) = S(T(x)), x E V. It is clear that the map ST is a transformation. The transformation is, sometimes, called the product of the transformations Sand T. The product TS may not be defined. If V = W = U, one can always define the product ST as well as T S, and they need not be the same. After having defined the space L(V, W), the next item on the agenda is to determine the magnitude of its dimension. The following proposition addresses this problem. P 3.2.4 If dim (V) mn.

=m

and dim(W)

= n,

then dim(L(V, W))

=

PROOF. Let Xl, X2, • •• ,Xm be a basis of the vector space V and

a basis ofW. Let T be any transformation from L(V, W). First, we observe that the value of T(x), for any X E V, is determined by the set T(xt}, T(X2), ... ,T(xm ) of vectors in W. For, we can write

WI, W2, ••• ,Wn

for some aI, a2,." ,am in F in a unique fashion. It follows that

Once a basis for the vector space V is fixed, knowing the vector X is equivalent to knowing its co-ordinates ai, a2, ... ,am' Once we know

112


the values of the transfonnation T at Xt,X2, .•. ,Xm , we can immediately write down the value of T at x. This innocuous observation has deep implications. One can build a transfonnation from V to W demanding it to take certain values in the vector space W at certain vectors in V! With this in view, for every fixed i E {I, 2, ... ,n} and j E {I, 2, . .. ,m}, let tij be a transformation from V to W satisfying TijXj = Wi, TijXk = 0 for all k i= j. In other words, we want a transfonnation Tij such that TijXI = 0, TijX2 = 0, ... ,Tijxj-l = 0, TijXj = Wi, TijXj+1 = 0, ... ,Tijxm = O. Note that Tij is the only transformation that takes the value 0 at Xl, X2, ... ,Xj-l' Xj+I, . .. ,Xm and the value Wi at Xj. (Why?) We claim that the set T ij , 1 :::; i :::; n, 1 :::; j :::; m of transformations constitutes a linearly independent set in L(V, W). Suppose n

m

LL

Ciij7ij

=0

i=l j=l

for some scalars 1:::; k:::; m, n

0=

Ciij,l :::;

i :::; n, 1 :::; j :::; m. In particular, for each

m

(L L

Ciij7ij) (Xk)

=

CilkWI

+ Ci2kW2 + ... + CinkWn·

i=l j=l

Since WI, W2,· .. , Wn are linearly independent, Ciik = 0 for every i 1,2, ... ,n. Since k is arbitrary, it follows that Ciij = 0 for every i and j. This establishes the linear independence of the transformations. Next, we show that any transformation T in L(V, W) is a linear combination of the transformations Til'S. Let T(Xi) = Yi, 1 :::; i :::; m. Expand each Yi in terms of the basis of W, i.e.,

for some scalars

(3ij'S.

One simply verifies that m

T =

n

""(3~~

-tJ-TJt·

i=l j=l

It now follows that dim(L(V, W)) = mn.

Linear TransjofTnations and Matrices

113

The notion of a linear transformation is not that much different from the notion of a lin~ar functional. By piecing together linear functionals in some suitable manner we can construct a linear transformation from one vector space to another. An inkling of this phenomenon has already been provided in Example 3.1.5. A fuller version of this example is the substance of the following proposition. P 3.2.5 Let V and W be two vector spaces. Let WI, W2, ... ,Wn be a basis for W. Let T be a map from V to W. Then T is a transformation if and only if there exist linear functionals Yl, Y2, ... ,Yn on V such that

for every x in V. PROOF. If T = Yl WI + Y2 W2 + ... + Yn Wn for some linear functionals Yl, Y2, . .. ,Yn on the vector space V, it is clear that T is a transformation. Conversely, let T be a transformation from V to W. Let xl, X2,." ,Xm be a basis for the vector space V. Let Yi = T(xd, i = 1,2, ... ,m. Write for each i,

for some scalars f3ij'S. Define for each j = 1,2, ...

,n,

m

Yj(x) = L oif3ij, X E V, i=1

where x = 01 Xl + 02X2 + ... + OmXm is the unique representation of x in terms of the basis of V. It is clear that each Yi(-) is a linear functional on V. Finally, for any x in V, m

T(x)

m

Tn

= T(LoiXi) = LOiT(Xi) = LOiYi = i=1 n

m

i=1

i=1

m

n

i=l

j=1

LOi Lf3ijWj

n

= L ( L o if3ij)Wj = LYj(x)Wj. j=1 i=1

j=1

Thus we are able to write T as a combination of linear functionals! This completes the proof.

114


Linear transformations from a vector space V into itself are of special interest. For any two transformations T and S from V to V, one can define the product TS as the composition of the maps T and S taken in this order. Moreover, one can define the identity transformation I from V to V by I(x) = x for every x in V. It is clear that for any transformation T from V to V, T I = IT = T. Thus the space L(V, V) has an additional structure which arises from the operation of product of transformations defined above. The following proposition provides the necessary details what this additional structure entails. P 3.2.6 The space L(V, V) of all transformations from a vector space V into itself is an algebra with an identity. We have already seen that the space L(V, V) is a vector space. The additional structure on L(V, V) comes from the binary operation of product or composition of transformations. We need to identify the identity of this new binary operation. The obvious candidate is the identity transformation I. In order to show that the space L(V, V) is an algebra, we need to verify the following. (1) For any transformation T,OT = TO = 0 holds, where 0 is the transformation which maps every vector of V into the zero vector. The map 0 is the additive identity of the additive operation of the vector space L(V, V) . (2) For every transformation T, IT = T I = T holds. (3) (Associative law). For any three transformations T, Sand U, (TS)U = T(SU) holds. ( 4) (Distributive laws). For any three transformations T, Sand U, (T+ S)U = TU +SU and T(S + U) = TS +TU hold. These properties are easy to establish. As for the distributive law and associative law, they are valid in a more general framework. For example, suppose T is a transformation from a vector space V I into a vector space V 2, and Sand U are transformations from the vector space V 2 into a vector space V 3, the transformations (S + U)T, ST and UT are all well-defined from the vector space VI to V 3 • Moreover, we have the following distributive law: PROOF.

(S

+ U)T =

ST + UTe

One could write down any number of identities of the above type. As an example of another variety, suppose T is a transformation from a vector

Linear Transformations and Matrices

115

space VItO a vector space V 2, S is a transformation from the vector space V 2 to a vector space V 3 , and U a transformation from the vector space V 3 to a vector space V 4. Then the transformations U (ST) and (U S)T make sense and they are indeed transformations from the vector space VI to V4. Further, they are identical, i.e., U(ST) = (US)T. It is customary to denote this transformation by U ST. Complements 3.2.1 Let Q be the field of all rational numbers. Let V = Q2 and T a transformation from V to V. The only clues we have about T are that T(1,0) = (2, -3), T(O, 1) = (3,1).

Determine T(x, y) for any (x, y) E V. Is T an isomorphism? Justify your answer. 3.2.2 Let V = Q2, W = Q3, and T a transformation from V to W. The only clues we have about T are that T(1,0)

= (2,3, -2),

T(1, 1)

= (4, -7,8) .

Determine T(x, y) for any (x, y) E V. 3.2.3 Let V = R 3 , W = R4, and T a transformation from V to W. The only clues we have about T are that T(1,0,0) = (1,-1,2,1), T(0,1,0) = (0,1,1,0), T(0,0,1) = (2,0,0,0).

Determine the range and kernel of the transformation T along with their dimensions. 3.2.4 Let P be the collection of all polynomials viewed as a vector space over the field of real numbers. (Note that P is an infinitedimensional vector space.) Let T be the differential operator on P defined by n-l

n-l

T(L~iXi) = (Li~iXi-l), i=O

i=l

and S the transformation on P defined by n-l

n-l

S(L ~ixi) = L(~i/(i + 1))xi+l, i=O

~o,6,

...

,~n real, n

i=O

2: 1. Compute TS and ST. Show that ST i= TS.

116


3.3. Inverse Transformations Let T : V --+ W. We raise the question whether given any vector y in W, is it possible to recover the vector x in V through a linear transformation, related to T in some way. The answer to the question depends on the nature of the transformation T. The following are two crucial properties that the answer depends on.

(1) The map is injective or one-t~one: recall that the map T is injective if Xl and X2 are any two distinct vectors in V, then the vectors T(xI) and T(X2) are distinct. (2) The map is surjective or onto: recall that the map T is surjective if for every vector y in W, there exists at least one vector x in V such that T(x) = y, i.e., R(T) = W. P 3.3.1

Let T : V

--+

W.

(1) If the map T is injective, then there exists a linear transformation S : W --+ V such that ST = I, the identity transformation from V to V. (Such a transformation S is called the left inverse of T and is denoted by Ti I.) (2) If the map T is surjective, then there exists a linear transformation S : W --+ V such that TS = I, the identity transformation from W to W. (Such a transformation S is called the right inverse of T and is denoted by Tii. I.) (3) If the map T is bijective, i.e., T is injective and surjective, then there exists a transformation S : W --+ V such that ST = I and T S = I with the identity transformation I operating on the appropriate vector space. The transformation S is unique and is called the inverse of T. It is denoted by T-I. A bijective map is also called invertible. (4) There always exists a transformation S : W --+ V such that TST = T. Such a transformation S is called a generalized inverse (g-inverse) of T and is denoted by T-. PROOF. Before we embark on a proof, let us keep in line the entities we need. Let R(T) be the range of the transformation T. Choose and fix a complement RC of R(T) in W. See P 1.5.5 for details. Let K(T) be the kernel of the transformation T. Choose and fix a complement KC of K(T) in V. We now proceed to prove every part of the above proposition. We basically make use of the above subspaces,

Linear 'Pransformations and Matrices

117

their complements, and the associated projections. Let y be any vector in W. We can write y = Yo + YI uniquely with Yo E R(T) and YI E R C • Since T is injective, there exists a unique xo in V such that T(xo) = Yo. Define Til(y) = Xo. Thus Til is a well-defined map from W to V. It is clear that TilT = I, the identity transformation from V to V. It remains to show that the map Til is a transformation. This essentially follows from the property that any projection is a linear operation. Let a and j3 be two scalars and u and W be any two vectors in W. If we decompose u = Uo + UI and W = Wo + WI with uo, Wo E R(T) and UI, WI E R c, then we can identify the decomposition of au + j3w as

au + j3w = (auo

+ j3wo) + (aul + j3wt)

with auo + j3wo E R(T) and aUI + j3wl E RC. Let Xo and vo be the unique vectors in V such that T(xo) = Uo and T(vo) = WOo It is now clear that

This shows that the map Ti 1 is a transformation. This proves (1). We now work with K(T) and a complement K C of K(T) . Let y be any vector in W. Since T is surjective, there exists a vector x in V such that T(x) = y. But there could be more than one vector x satisfying T(x) = y. But there is one and only one vector Xo in KC such that T(xo) = y. This is not hard to see. Define

Til(y) = Xo · By the very nature of the definition of the map Til, TTii. 1 = I, the identity map from W to W. It remains to show that Tii. l is a transformation. In order to show this, one can craft an argument similar to the one used in Part (1) above, which proves (2). If the map is bijective, both the definitions of the maps Til and Til coincide. Let the common map be denoted by T- I . Of course, we have TT-I = I and T-IT = I. As for uniqueness, suppose S is a map such that TS = I, then

S

= IS = T- I T S = T- 1 I = T- 1 •

118


Another interesting feature is that if Sand U are two maps from W to I V satisfying T S = I and UT = I, then we must have S = U = T- . This proves (3). Let yEW. Write uniquely y = Yo +YI with Yo E R(T) and YI E RC. Determine x E V such that T(x) = Yo. Decompose uniquely x = XO+XI with Xo E K C and Xl E K(T). Define T-(y) = Xo. T- is well-defined and indeed is a transformation from W to V. It is clear that rr-T = T, which proves (4). The transformations Til, Till and T- are not, in general, unique. Different possible choices arise from different choices of the complements R C and KC. We will exploit this fact when we explore the world of ginverses in one of the subsequent chapters. We record a few facts about inverse transformations for future reference. These results can easily be verified.

P 3.3.2 Let T be a bijective transformation (isomorphism) from a vector space V to a vector space W, and S a bijective transformation from the vector space W to a vector space U. Let a be a non-zero scalar. Then the following are valid.

(1) The transformation ST is a bijective transformation from V to U and (ST)-l = T-IS- I . (2) The transformation aT is a bijective transformation from V to Wand (aT)-1 = a-IT-I. (3) The transformation T-I is a bijective transformation from W to V and (T- I = T.

tl

Let us specialize about transformations that operate from a vector space V into itself. If T is such a transformation, T-I exists if and only if p(T) = dim(V). This follows from the identity v(T)+p(T) = dim(V). See P 3.1.7. Complements 3.3.1 Let V be the set of all complex numbers viewed as a vector space over the field of real numbers. Let T be the transformation from V to V defined as T(x + iy) = x - iy, the complex conjugate of the complex number x T-I.

+ iy

E V. Determine


3.3.2

119

Let T be the transfonnation on the vector space C 2 defined by T{XI,X2} = {axi +,8x2,,),XI +8x2},

for {Xl, X2} E C 2 and a,,8, ,)" and 8 E C. Show that T is an isomorphism if a8 - ,8')' =1= o. In such an event, determine T- I . If a8 - ,8')' = 0, determine a g-inverse T- of T. 3.3.3 Let P be the vector space of all polynomials with real coefficients, which is viewed as an infinite-dimensional vector space over the field of real numbers. Let T be the differential operator on P defined by n-l

n-l

T{L aixi} = L i=O

iaixi-l,

i=l

real and n ~ 1. Show that T is surjective. Let S be the transformation on P defined by aO, aI, a2, ... ,an-l

n-l

n-l

S{L aixi} = L i=O

i=O

~xi+l. 'I,

+1

Show that S is a right inverse of T. Show that there is no left inverse for T. 3.3.4 If T 1 , T2, ... ,Tr are invertible transfonnations from a vector space V into V itself, show that T 1 T 2 ... Tr is also invertible. 3.3.5 Let T be a transfonnation from V to W. Show that for the existence of a left inverse of T, it is necessary that T is injective. If T is injective, show that any left inverse of T is surjective. 3.3.6 Let T be a transfonnation from V to W. Show that for the existence of a right inverse of T, it is necessary that T is surjective. If T is surjective, show that any right inverse of T is injective. 3.3.7 If T and S are two transformations from a finite-dimensional vector space V into itself such that TS = I, show that T is invertible and S = T-I. {Why do we need the finite-dimensionality condition on the vector space?} 3.3.8 Let T be a transformation from a finite-dimensional vector space V into itself enjoying the property that T{~J), T{X2}, . . . ,T{xr } are linearly independent in W whenever XI, X2, ... ,Xr are linearly independent in V for any r ~ 1. Show that T is invertible. Is the finitedimensionality condition needed?

120


3.4. Matrices Let V and W be two finite-dimensional vector spaces over the same field F. Let T be a transformation from V to W. Let Xl. X2,··· ,Xm be a basis of the vector space V and Y1, Y2, . . . ,Yn that of W. For each i = 1,2, ... ,m, note that T(Xi) E W, and consequently, we can write (3.4.1) for some scalars Oij'S in F. These scalars and the transformation T can be regarded as two sides of the same coin. Knowing the transformation T is equivalent to knowing the bunch of scalars Oij'S. We explain why. The transformation T provides a rule or a formula which associates every vector in V with a vector in W . This rule can be captured by the set of scalars Oij'S. First of all, let us organize the scalars in the form of an n X m grid AT, which we call a matrix.

Let x be any vector in V. We can write

for some scalars 131, 132, . .. ,13m in F. In view of the uniqueness of the representation, the vector x can be identified with the m-tuple (131,{32 , ••• ,13m) in Fm. The transformed value T(x) of x under T can be written as

T(x) = I1Y1

+ 12Y2 + ... + InYn

(3.4.2)

for .som~ scala~s ,1,,2, ... "n in F. The transformed vector T(x) can be IdentIfied wIth the n-tuple (,1,,2, ... "n) in Fn. Thus the transformation T can be identified with a transformation from Fm to Fn. Once we know the vector x, or equivalently, its m-tuple (131,132, ... ,13m), the coordinates (,1,,2, ... "n) of the transformed vector T( x) can be obtained as m

Ii =

L i=l

O ij13j,

i = 1,2, ... ,no

(3.4.3)


121

For, we note that

T(x) = T(f31Xl + f32X2 + ... + f3m Xm) = f31 T (Xt) + f32T(X2) + ... + f3mT(xm) n

n

n

= f31(L OilYi) + f32(L Oi2Yi) + ... + f3m(L OimYi) i=1

i=1

m

i=1

m

= (L Oljf3j)Yl

j=1

m

+ (L 0 2jf3j)Y2 + .. . + (L Onjf3j)Yn. j=1

j=1

(3.4.4)

using (3.4.1) in step 3 above. Then identifying (3.4.4) with (3.4.2), we obtain (3.4.3), which transforms f3i's to I/S. Equations (3.4.3) can be written symbolically as:

[0

11

012

aIm]

021

022

02m

°nl

°n2

°nm

[::] [:] -

or, in short ATb = c,

(3.4.5)

where b is the column vector consisting of entries f3i 's and c is the column vector consisting of entries I i'S. The symbolic representation etched above can be made algebraically meaningful. Operationally, the symbolic representation given above can be implemented as follows . Start with any vector x in V. Determine its coordinates b with respect to the given basis of V. Combine the entries of the matrix AT and the vector b as per the arithmetic set out in the equation (3.4.3) in order to obtain the coordinates c of the transformed vector T(x). Once we know the coordinates of the vector T(x), we can write down the vector T(x) using the given basis of the vector space W. There are two ways of giving an algebraic meaning to the symbolic representation (3.4.5) of the transformation T . In the representation, we seem to multiply a matrix of order n x m (i.e., a matrix consisting nm entries from the underlying field arranged in n rows and m columns) and a matrix of order m x 1 resulting in a matrix of order n x 1. One could spell out the rules of multiplication in such a scenario by invoking the

122


equations (3.4.3). Another way is to define formally the multiplication of two matrices and then exclaim that in the symbolic representation (3.4.5) we are actually carrying out the multiplication. We will now spend some time on matrices and some basic operations on matrices. 3.4.1. Let F be a field. A matrix A of order m x n is an array of mn scalars from F arranged in m rows and n columns. The array is presented in the following form: DEFINITION

with the entries aij 's coming from the field F. Frequently, we abbreviate the matrix A in the form A = (aij), where aij is the generic entry located in the matrix at the junction of the i-th row and j-th column. The scalar aij is also called the (i,j)-th entry of the matrix A. The origin of the word "matrix" makes an interesting reading. In Latin, the word "matrix" means - womb, pregnant animal. The use of the word "matrix" in Linear Algebra, perhaps, refers to the way a matrix is depicted - it is a womb containing objects in an orderly fashion. The Indo-European root of the word "matrix" is ma which means mother. It is time to introduce some operations on matrices. In what follows, we assume that all matrices have scalars from one fixed field. DEFINITION

3.4.2.

(1) Addition. Let A = (aij) and B = (f3ij) be two matrices of the same order m x n. The matrix C = (/ij) of order m x n is defined by the rule that the (i,j)-th entry lij of C is given by lij = aij + f3ij for all 1 ~ i ~ m and 1 ~ j ~ n. The matrix C is called the sum of A and B, and is denoted by A + B. (2) Scalar Multiplication. Let A = (aij) be a matrix of order m x n and a a scalar. The matrix D = (8ij ) of order m x n is defined by the rule that the (i,j)-th entry 8ij of D is given by 8ij = aaij for all i and j. The matrix D is called a scalar multiple of A and is denoted by aA. If a = -1, the matrix aA is denoted by

-A.

(3) Multiplication. Let A = (aij) and B = (f3ij) be two matrices of order m x nand p x q, respectively. Say that A and Bare

123

Linear TIunsformations and Matrices

conformable for multiplication in the order they are written if the number of columns of A is the same as the number of rows of B, i.e., n = p. Suppose A and B are conformable for multiplication. The matrix E = (eij) of order m x q is defined by the rule that n

the (i,j)-th entry

eij

of E is given by

eij

=

:L (Xikf3kj

for all

k=l

1 :::; i :::; m and 1 :::; j :::; q. The matrix E is called the product of A and B, and is denoted by AB. Of all the operations defined on matrices defined above, the multiplication seems to be the most baffling. The operation of multiplication occurs in a natural way when we consider a pair of transformations and their composition. If two matrices A and B are conformable for multiplication, it is not true that B and A should be conformable for multiplication. Even if B and A are conformable for multiplication with the orders of AB and B A being identical, it is not true that we must have AB = BA! We introduce two special matrices. The matrix of order m x n in which every entry is zero is called the zero matrix and is denoted by Omxn. If the order of the matrix is clear from the context, we will denote the zero matrix simply by o. The matrix of order n x n in which every diagonal entry is equal to 1 and every off-diagonal entry is 0 is called the identity matrix of order n x n and is denoted by In. We now record some of the properties of the operations we have introduced above.

P 3.4.3

(1)

Let A, Band C be matrices of the same order. Then

(A+B)+C=A+(B+C), A+B = B+A, A+O=A, A+(-A)=O. (2) Let A, Band C be three matrices of orders m x n, n x p, and p X q, respectively. Then A(BC) = (AB)C of order m x q, ImA= Aln = Orxm A = AOnxs =

A, A, Orxn, Omxs.


124

(3) Let A and B be two matrices of the same order m x n, C a matrix of order n x p and D a matrix of order q X m. Then

(A+B)C= AC+BC D(A+B) = DA+ DB. The above properties are not hard to establish. Some useful pointers emerge from the above deliberations. If Mm,n denotes the collection of all matrices of order m X n, then it is a vector space over the field F. If m = n, then Mm,m is an algebra. The operations are addition, scalar multiplication and multiplication of matrices. This algebra has a certain peculiarity. For two matrices A and B in Mm,m, it is possible that AB = 0 without each of A and B being the zero matrix. Construct an example yourself. We introduce two other operations on an m X n matrix A. One is called transpose, which is obtained by writing the columns of A as rows (i-th column as the i-th row, i = 1, ... ,n) and denoted by A' . It is seen that the order of A' is n X m. The following is an example of A and A'. 2

A=C

3

The following results concerning the transpose operation are easily established. P 3.4.4

(1) (AB)' = B'A', (ABC)' = C'B'A'. (2) (A + B)' = A' + B'. (3) (A2)' = (A')2. (4) If A = (aij) is a square symmetric matrix of order n

n

1

1

LL

aijXiXj

n,

then

= x' Ax

where x' = (Xl, ... ,x n ). Note that X is a column vector. (5) (A-I), = (A')-l if A is invertible, i.e., AA-l = A-I A = I holds.


125

Another is called the conjugate tmnspose, applicable when the elements of A are from the field of complex numbers, which is obtained by first writing the columns of A as rows as in the transpose and replacing each element by its complex conjugate, and denoted by A*. Thus if A = Al +iA2, where Al and A2 are real, then A* = A~ -iA~. The following results concerning the conjugate transpose are easily established. P 3.4.5 (1) (AB)* = B* A*, (ABC)* = C* B* A*. (2) (A+B)* = A* +B*. (3) (aA)* = oA*, (0 is the complex conjugate of a). (4) (A-I)* = (A*)-I if A is invertible. We showed that associated with a transformation T: V ~ W, there exists an n x m matrix AT which provides a transformation from Fffl to Fn through the operation of matrix multiplication ATx, x E Fffl. As observed earlier, the transformations T and AT are isomorphic and AT does the same job as T. We quote some results which are easy to establish. P 3.4.6 Let T : V ~ Wand S : V -- W be two transformations with the associated matrices AT and As. Then

Aa:T+,BS = aAT + f3As. Let T: V ~ Wand S : W ~ U. Then AST = AsAT, which justifies the matrix multiplication as introduced in Definition 3.4.2 (3).

Complements 3.4.1 Let A = (aij) be a matrix of order n x n with entries from a field. Define n Trace A = Tr A = 2:aii i=I

i.e., the sum of diagonal elements. The following results are easily established: (1) Tr(A+B)=TrA+TrB (2) TraA = a Tr A, for any scalar a. (3) Tr(AB) = Tr(BA) for A of order m x nand B of order n x m. (4) Let x be an n-vector and A be n x n matrix. Then x' Ax = Tr(Axx ' ).

126


Let us recall that in Section 3.1, we have introduced the range R(T) and the nullity K(T) of a transformation T and defined dimR(T) = p(T) as the rank of T and dim K(T) = v(T) as the nullity of T, satisfying the condition dim V = v(T) + p(T). The matrix AT associated with T, for chosen complements of R(T) and K(T), is a transformation on its own right from Fm to F n as represented in (3.4.5) with rule of multiplication given in (3.4.3). So we have V(AT) and p(AT) associated with AT. It is easy to establish the following proposition. P 3.4.7 Let T: V -+ Wand AT: Fm with T be as described above. Then

-+

Fn, the matrix associated

(1) p(T) = p(AT) (2) v(T) = V(AT) = dimSp(AT) = dimSp(A~) + V(AT) = m + v(A~) = n where Sp(AT) is the vector space generated by the column vectors of AT and Sp(A~) in the vector space generated the row vectors of AT, i.e., the column vectors of A~. In the rest of the chapters, we develop the algebra of matrices as a set of elements with the operations of addition, scalar multiplication, matrix multiplication, transpose and conjugate transpose as defined in this section. The results of Chapter two on spectral theory in the context bilinear forms are proved with special reference to matrices for applications to problems in statistics and econometrics.

(3) p(AT) (4) p(AT) (5) p(A~)

Complements 3.4.2 Let A E M n , i.e., a square matrix of order nand p(a) = ao + ala + ... + akak be a polynomial in a of degree k. Define P(A) = a01 + alA + ... + akAk E Mn. Show that if p(a) + q(a) = h(a) and p(a)q(a) = t(a) for scalar polynomials, then p(A) + q(A) = h(A) and p(A)q(A) = t(A). 3.4.3 Let p(,X) = ('x - 1)('x + 2) and consider the matrix equation P(A) = o. Show that A = 1 and -21 are roots of P(A) = o. Construct an example to show that there can be roots other than 1 and -21.

CHAPTER 4 CHARACTERISTICS OF MATRICES In Chapter 3, linear transformations and their associated matrices have been at the center of attraction. The associated matrix itself can be viewed as a linear transformation in its own right. To bring matters into proper perspective, let us recapitulate certain features of Chapter 3. Let V and W be two vector spaces of dimensions nand m, respectively, over the same field F. Let T be a linear transformation from V to W. Let AT be the matrix of order m x n associated with the transformation T. See Section 3.4. The entries of the matrix are all scalars belonging to the underlying field F. Once we arrive at the matrix AT, the flesh and blood of the transformation T, we can ignore the underlying transformation T for what goes on, and concentrate solely on the matrix AT. The matrix AT can now be viewed as a linear transformation from the vector space F n to Fffl by the following operational device: b -+ ATb, b E F n ,

where, as usual, we identify members of the vector spaces Fffl and Fn by column vectors. The subscript T attached to the associated matrix now becomes superficial and we drop it. In this chapter, we are solely concerned with matrices of order m x n with entries belonging to a field F of scalars. Let us recall that a matrix A of order m x n is an arrangement of mn elements in m rows and n columns with the (i,j)-th element indicated by aij' Addition and multiplication operations are as in Definition 3.4.2. Such matrices can be regarded as linear transformations from the vector space F n into the vector space Fffl. A variety of decomposition theorems for matrices will be presented along with their usefulness. In the initial part of this chapter, we will rehash certain notions introduced in the environment of linear transformations for matrices. 127

128


4.1. Rank and Nullity of a Matrix Let A be a matrix of order m x n. Unless otherwise specified, the entries of the matrices are always from a fixed field F of scalars. The range space R(A) of A is defined by,

R(A) = {Ax: x E Fn}.

(4.1.1)

The set (4.1.1) or the subspace (4.1.1) has the equivalent expression

R(A)

= {alaI + ... +ana n : al, .. . ,an E F}

where aI, ... ,an are the n column vectors of A. It is also called the span of the column vectors of A, and alternatively written as Sp(A). It is time to move on to other entities. The rank p(A) of the matrix A is defined by,

p(A)

= dim[R(A)] = dim[R(A')].

( 4.1.2)

The number p(A) can be interpreted as the maximal number of linearly independent column vectors of A, or equivalently, as the maximal number of linearly independent row vectors of A. Another object of interest is the kernel K(A) of the matrix, which is defined by, K(A) = {x E F n : Ax = O}, (4.1.3) which can be verified to be a subspace of the vector space Fn, and the nullity v(A) of the matrix A by,

v(A) = dim[K(A)].

(4.1.4)

The kernel is also called by a different name: null space of A. From P 3.1.1, dim[R(A)] + dim[K(A)] = n = dim[Fn], (4.1.5) or equivalently,

p(A)

+ v(A) = n.

(4.1.6)

If we look at the identity (4.1.5), there is something odd about it. The range space R(A) is a subspace of the vector space F1n whereas the null space K( A) is a subspace of the vector space Fn, and the dimensions of

Chamcteristic8 of Matrices

129

these two subspaces add up to n! Let us rewrite the identity (4.1.5) in a different form: dim[R(A')]

+ dim[K(A)]

= n.

(4.1.7)

Now, both R(A') and K(A) are subspaces of the vector space Fn. The identity (4.1.7) reads better. Then one is immediately led to the question whether R(A') n K(A) = {O}? (4.1.8) The answer to this question depends on the make-up of the field F. If F is the field of real numbers, (4.1.8) certainly holds. There are fields for which (4.1.8) is not valid. For an example, let F = {O, I} and

A=[~ ~]. Note that dim[K(A)] = 1 and dim[R(A')] = 1. Further, K(A) n R(A') is a one-dimensional subspace of F2. Even the field F = C of complex numbers is an oddity. We can have (4.1.8) not satisfied for a matrix A. The following is a simple example. Fix two real numbers a and b, both not zero. Let

A

=

[aa+'tb + ~b

bb-

iaia] .

It is clear that the subspace R(A') of F2 is one-dimensional. Let x' = (a+ib, b-ia). One can check that Ax = O. By (4.1.7), dim[K(A)] = 1. But K(A) n R(A') is a one-dimensional subspace providing a negative response to (4.1.8). Does it puzzle you why the answer to the question in (4.1.8) is in the affirmative when F = R but not when F = C or {O, I}? It is time for some introspection. Let us search for a necessary and sufficient condition on a matrix A for which K(A) n R(A') = {O}. Let A be a matrix of order n x k and x E R(A'). Let aI, a2, ... , an be the columns of A'. We can write (4.1.9)

130


for some scalars at, a2, ... , an in F. Suppose x E K(A). This is equivalent to:

0= Ax = (at,a2, ... ,an)'x, a~x

= 0

for

i = 1,2, ... , n.

(4.1.10)

Combining (4.1.9) and (4.1.10), we obtain

Let us rewrite the above equations in a succinct matrix form: AA' a = 0, where a' = (at, a2, ... , an). Note that (4.1.9) can be rewritten as A'a = x. The deliberations carried out so far make us conclude that K(A) n R(A') = {O} if and only if AA'a = 0 for any vector a implies that A' a = o. We can enshrine this result in the form of a proposition.

P 4.1.1 Let A be any matrix of order n x k. Then K(A) n R(A') = {O} if and only if

AA' a

=0

for any

a ~ A' a

= o.

(4.1.11)

The condition (4.1.11) is the one that sorts out the fields. If F is the field of real numbers, then (4.1.11) is always true for any matrix A. To see this, suppose AA'a = 0 for some a. Then a' AA'a = O. Let Y = A'a. We can rewrite aAA'a = 0 as y'y = O. But y'y is a sum of squares of real numbers, which implies that y = A' a = O. If F = {O, 1}, there are matrices for which (4.1.11) is not true. Take

A

= [~

~]

and

a'

= (1,0).

H F is the field of complex numbers, there are matrices A for which (4.1.11) is not true. It all boils down to the following query. Suppose Yt, Y2, ... , Yn are complex numbers such that yi + y~ + ... + y~ = O. Does this mean that each Yi = O?

Characteristics of Matrices

131

Complements 4.1.1 Let A and B be matrices of orders m x nand s x m, respectively. Show that R{A) = K{B) if and only if

R{B') = K{A'). 4.1.2 Show that a subset S of the vector space Fn is a subspace of Fn if and only if S is the null space of a matrix. 4.2. Rank and Product of Matrices Let us extend the range of ruscussion carried above to products of matrices. Suppose A and B are two matrices such that the product AB is meaningful. Observe that every column of AB is a linear combination of the columns of A. In a similar vein, every row of AB is a linear combination of the rows of B . This simple observation leads to the fruitful inclusion relations:

R{AB) C R{A), R{B' A') C R{B').

p{AB)

~

p{A), p{AB)

~

p{B).

(4.2.1) (4.2.2)

We can combine both the inequalities into a proposition. P 4.2.1

p{AB} ~ min{p(A}, p(B)}.

( 4.2.3)

We are led to another inquiry: when does equality hold in (4.2.3)? We have a precise answer for this question. For this, we need to take a detour on inverses of one kind or the other. In Section 3.3, we spoke about inverse transformations of various types. We need to refurbish these notions in the environment of matrices. There are two ways to do this. One way is to view each matrix as a linear transformation from one vector space to another, work out a relevant inverse transformation and then obtain its associated matrix. This process is a little tortuous. Another way is to define inverses of various kinds for a given matrix directly. We pursue the second approach.

132


DEFINITION 4.2.2. Let

A be a matrix of order m x n .

(1) A left inverse of A is a matrix B of order nxm such that BA = I, where I is the identity matrix of order n x n, i.e., I is a diagonal matrix in which every diagonal entry is equal to the unit element 1 of the field F. (If B exists, it is usually denoted by A'Ll.) (2) A right inverse of A is a matrix C of order n x m such that AC = I, where I is the identity matrix of order m x m. (If C exists, it is usually denoted by A RI.) (3) A regular inverse of A of order n x n is a matrix C such that AC = I. IT C exists, it is denoted by A-I. In such a case AA-I = 1= A-IA. We know precisely the conditions under which a transformation admits a left inverse, a right inverse, or an inverse. These conditions, in the realm of matrices, have a nice interpretation. The following results are easily established. p 4.2.3

Let A be a matrix of order m x n.

(1) A admits a left inverse if and only if p(A) = n. (2) A admits a right inverse if and only if p(A) = m. (3) A admits a regular inverse if and only if m = nand p(A)

= n.

P 4.2.4 Let A and B be two matrices of orders m x nand n x s, respectively. Then:

(1) p(AB)

= p(B)

if A'Ll exists, i.e., (2) p(AB) = p(A) if Bi/ exists, i.e.,

p(A) = n. p(B) = n.

PROOF. Suppose that the left inverse of A exists. Note that B = (A'Ll A)B. Also, by (4.2.3),

p(AB) ~ p(B) = p(A'LI AB) ~ p(AB), from which the professed equality follows. One can establish (2) in a similar vein. The above proposition has very useful applications in a variety of contexts. A standard scenario can be described as follows. Suppose B is any arbitrary matrix. If we pre-multiply or post-multiply B by any non-singular matrix A, the rank remains unaltered, i.e., p(AB) ~ p(B) or p(BA) = p(B), as the case may be .

Charncteristics of Matrices

133

Once equality in (4.2.3) holds, it sets a chain reaction, as the following proposition exemplifies. P 4.2.5 Let A and B be two matrices of order m x nand n x s, respectively. (I) If p(AB} = peA}, then p(CAB} = p(CA} for any matrix C for which the multiplication makes sense. (2) If p(AB} = pCB}, then p(ABD} = p(BD} for any matrix D for which the multiplication makes sense. PROOF . (I) Each column of the matrix AB is a linear combination of the columns of A. The condition p(AB} = peA} implies that the subspaces spanned by the columns of AB and of A individually are identical. What this means is that every column of A is a linear combination of the columns of AB. (Why?) Consequently, we can write A = ABT for some suitable matrix T. Now, for any arbitrary matrix C for which the multiplication involved in the following makes sense,

p(CA} = p(CABT)

~

p(CAB)

~

p(CA),

from which the desired equality follows. The result (2) follows in a similar vein. The condition given in P 4.2.5 for the equality of ranks in (4.2.3) is sufficient but not necessary. Let F be the field of real numbers. Further let

A=B= Then p(AB) = pCB) but peA} =1= 3. We are still in search of a necessary and sufficient condition for the equality in (4.2.3). Let us pinpoint the exact relationship between p(AB) and pCB). P 4.2.6 Let A and B be two matrices of order m x nand n x s , respectively. Then

+ dim[K(A} n R(B)], p(AB} + dim[R(A'} n K(B'}].

(1)

pCB} = p(AB)

(2)

peA) =

suffices to prove the first part. The second part is a consequence of the first part. (Why?) To prove (1) , we observe that PROOF. It

{Bx E F n

:

ABx = O} = K(A} n R(B).

134


We will now find the dimension of the subspace {Bx E Fn : ABx = O} of Fn. Note that {Bx E F n

:

ABx = O} = {Bx E F n

:

x E K{AB)}.

Let V = K{ AB) and W = Fn. The matrix B can be viewed as a transformation from the vector space V to the vector space W. The set {Bx E Fn : x E K{AB)} then becomes the range of this transfonnation. By P 3.1.1 (3), dim{{Bx E F n

:

x E K{AB)}) +dim{{x E K{AB): Bx

= O})

= dim{K{AB)), which gives

dim{{Bx E F n : x E K{AB)})

= =

dim{K{AB)) - dim{{x E K(AB) : Bx

= O})

s - dim{R{AB)) - [8 - dim{R{B))] = p{B) - p{AB).

This completes the proof. One can officially close the search for a necessary and sufficient condition if one is satisfied with the following result. P 4.2.7 Let A and B be two matrices of orders m x nand n x respectively. Then

= p(B) p(AB) = p(A)

(1)

p{AB)

(2)

8,

= {O}; R(A') n K(B') = {O}.

if and only if K(A) n R(B) if and only if

PROOF. (1) This is an immediate consequence of P 4.2.5. The second statement (2) is a consequence of (1). We now spend some time on a certain celebrated inequality on ranks, namely, Frobenius inequality.

P 4.2.8 (Frobenius Inequality) Let A, Band C be three matrices such that the products AC and C B are defined. Then p(ACB) PROOF.

+ p(C)

~ p(AC)

+ p(CB).

( 4.2.4)

Consider the product of partitioned matrices,

0 AC] [ I 01 = C -B I

I -A] [ [o I CB

[-ACB 0

0] C·

Characteristics of Matrices

135

Since the matrices

are non-singular, it follows that

[°

p CB

°

AC]_ [-ACB C - P

But p

CO]

p(ACB)

+ p(C).

[C; Ag] ~ p [dB J + P[Acf] -

p(CB)

+ p(AC),

(why?)

from which the desired inequality follows. The following inequality is a special case of Frobenius inequality. P 4.2.9 (Sylvester's Inequality) Let A and B be two matrices of orders m x nand n x s, respectively. Then p(AB) ;::: p(A)

+ p(B) -

n,

with equality if and only if K(A) C R(B). In the Frobenius inequality, take C = In. The inequality follows. From P 4.2.6 (1), p(B) = p(AB)+dim[K(A)nR(B)]. If equality holds in the Sylvester's inequality, then we have dim[K(A) n R(B)] = n - p(A) = n - dim[R(A)] = dim[K(A)]. See (4.1.5). Consequently, equality holds in Sylvester's inequality if and only if K(A) c R(B). PROOF.

Complements 4.2.1 Reprove P 4.2.3 using results of Sections 3.3 and 3.4. 4.2.2 Let A and B be square matrices of the same order. Is p(AB) = p(BA)? 4.2.3 Let A and B be two matrices of orders m x nand m x s, respectively. Let (AlB) be the augmented matrix of order m x (n + s). Show that p(AIB) = p(A)

if and only if B = AC for some matrix C.

136


4.3. Rank Factorization and Further Results In a variety of applications in engineering and statistics, we need to express a matrix as a product of simple and elegant matrices. In thls section , we consider one of the simplest such factorizations of matrices. . P 4.3.1 (Rank Factorization Theorem) Let A be a matrIX of order m x n with rank a. Then A can be factorized as A=RF,

where R is of order m x a, F is of order a x n, Sp(A) Sp( F'), and p( R) = p( F) = a . Alternatively,

(4.3.1)

= Sp(R),

A=SDG,

Sp(A') =

( 4.3.2)

where Sand G are non-singular matrices of order m x m and n x n, respectively, and D is a block matrix of the following structure: _ D -

[

a Ia xa

a x (n0 - a)

0

0 (m - a) x (n - a)

(m - a) x a

1 .

PROOF. The main goal in the above factorization is to write the matrix A of rank a as a product of two matrices one of them has a full column rank and the other full row rank. This is like removing the chaff from the grain so as to get to the kernel. A proof can be built by imitating the winnowing process. Take any basis of the vector space Sp(A). The column vectors of the basis are taken as the column vectors of the matrix R of order m x a. Every column vector of A is now a linear combination of the columns of R . Consequently, we can write A = RF for some matrix F. Once R is chosen, there is only one matrix F whlch fulfills the above rank factorization. (Why?) It is obvious that p(R) = a. It is also true that p(F) = a. Thls can be seen as follows. Note that

a = p(A) = p(RF) ~ p(F) ~ min{a,n} ~ a. The fact that Sp( A') = Sp( F') follows from the facts that every row vector of A is a linear combination of the rows of F and p( F) = a.

Chamcteristic8 of Matrices

137

To obtain the form (4.3.2), one simply needs to augment matrices in (4.3.1) to get non-singular matrices. For example, let S = (R : Ro), where the columns of the matrix Ro is any basis of a complementary subspace of Sp(R) in Fm. The matrix G is obtained by adjoining F with (n - a) rows so that G becomes non-singular. Take any basis of the complementary subspace of the vector space spanned by the rows of F. The basis vectors are adjoined to F to form the matrix G. A direct multiplication of (4.3.2) gives (4.3.1), which justifies the validity of (4.3.2). That is all there is in rank factorization. Such a simple factorization is useful to answer some pertinent questions on ranks of matrices. One of the perennial questions is to know under what condition does the equality p(A + B) = p(A) + p(B) holds for two given matrices A and B. The following result goes some way to answer this question. P 4.3.2

Let A and B be two matrices of the same order with ranks

a and b, respectively. Then the following statements are equivalent.

(1)

p(A + B) = p(A)

(2)

p(AIB)

(3)

+ p(B).

(4.3.3)

= p [~ ] = p(A) + p(B). Sp(A) n Sp(B) = {O} and Sp(A') n Sp(B') = {O}.

(4.3.4) (4.3.5)

(4)

The matrices A and B can be factorized in the following style:

A= S

Ia

0 0] Gj

(4.3.6)

0 0 0 [000

where the zeros are matrices of appropriate order, and

lSI # 0, IGI # O.

We start with rank factorization of A and B. Write A = RtFt and B = R2F2 with Sp(A) = Sp(Rt), Sp(A') = Sp(FD, Sp(B) = Sp(R2) and Sp(B') = Sp(FD. It is obvious that PROOF.

Sp(AIB) = Sp(RtIR2)' Sp(A'IB') = Sp(F{lF~), from which it follows that

p(AIB) = p(R t IR2),

p(~) = p(~:).

138


We are now ready to prove the equivalence of the statements. Let us assume (1) is true. Then

p(A)

+ p(B) = p(A + B) = P(RI FI + R2F2) = p((RIIR2) (~:)) ::; P(RIIR2) = p(AIB) ::; p(Rd + p(R2) = p(A)

+ p(B),

from which p(AIB) = p(A) + p(B) follows. The equality p(~) = p(A) + p(B) follows in a similar vein. Thus (1) => (2). Since p(AIB) = dim[Sp(AIB)] = the maximal number of linearly independent columns of the augmented matrix (AlB) = p(A) + p(B) = dim[Sp(A)]+dim[Sp(B)] = the maximal number oflinearly independent columns of A+ maximal number of linearly independent columns of B , it follows that Sp(A) n Sp(B) = {O}. It also follows in a similar vein that Sp(A') n Sp(B') = {O} . Thus (3) follows from (2). Let us augment the matrix (R 1 IR2) of order m x (a + b) to a nonsingular matrix of order m x m. Since Sp(A) = Sp(R 1 ) , Sp(B) = Sp(R2)' Sp(RdnSp(R2) = {O} (by hypothesis), P(Rl) = a, and p(R2) = b, all the columns of the matrix (R 1 IR2) are linearly independent. Consequently, we can find a matrix Ro of order m x (m - (a + b)) such that the augmented matrix S = (RIIR2IRo) is nonsingular. Following the same line of reasoning, we can find a matrix Fo of order (n - (a + b)) x n such that

is non-singular. The professed equalities of expressions in (4) follow now routinely. Finally, (4) => (1) obviously. The equality p(A + B) = p(A) + p(B) is very hard to fulfill. What the above result indicates is that if no column vector of A is linearly dependent on the columns of B , no column vector of B is linearly dependent on the columns of A, no row vector of A is linearly dependent on the rows of B, and no row vector of B is linearly dependent on the rows of A, only then the purported equality in (1) will hold. P 4.3.2 can be generalized. Looking at the proof a little critically, it seems that what is good for two matrices should be good for any finite number of matrices.

Chamcteristics of Matrices

139

P 4.3.3 Let AI, A2, ... , Ak be any finite number of matrices each of order mxn with ranks aI, a2, . .. , ak, respectively. Then the following statements are equivalent.

(1) P(AI + A2 + ... + A k) = p(At} + p(A2) + ... + p(Ak). (2) P(AIIA21 .. ·IAk) = p(A~IA21 .. ·IA~) = P(Al) + ... + p(Ak). (3) Sp(Ai)nSp(Aj) = {O} and Sp(ADnSp(Aj) = {O} for all i =/=j. (4) There exist nonsingular matrices Sand G of orders m x m and n x n, respectively, such that Ai

= SDiG, i =

1,2, ... ,k,

where Di is a block matrix of order m x n with the following structure. Row Block No.

Column Block No.

1 2

z

2

o o

0 0

o o

i

o

0

k+1

o

0

1

... (k

+ 1) o

o o

o

o

The zeros appearing in the above are zero matrices of appropriate order. We now consider the problem of computing the rank of a matrix utilizing the notion of a Schur complement. Let A be a matrix of order (n + s) x (n + t) partitioned in the following style.

A --

[nFin nf S, the elementary matrix Ers(j3) is a unit lower-triangular matrix. The notion of a unit upper-triangular matrix is analogous. IT r < s, the elementary matrix Ers(j3) is a unit upper-triangular matrix. IT A and B are two unit upper-triangular matrices of the same order, it can be checked that the product AB is also a unit upper-triangular matrix. A similar assertion is valid for unit lower-triangular matrices. P 5.2.1 Let A = (O:ij) be a matrix of order m x n . Then there exists a unit lower-triangular matrix B of order m x m such that BA is in echelon form.

Factorization of Matrices

161

PROOF. Start wjth the first non-zero row of the matrix A. Assume, without loss of generality, that the first row of A is non-zero. Let ali be the first non-zero entry in the first row. Take any 2 ~ 8 ~ m. If asi = 0, we do nothing. If asi =I 0, we multiply the first row by -asda1i and add this to the s-th row. This operation makes the (8, i)-th entry zero. This operation is equivalent to pre-multiplying A by the elementary matrix Est ( -asd a1 i) which is a unit lower-triangular matrix. Thus we have used the (1, i)-th entry, namely ali, as a pivot to liquidate or sweep out all the other entries in the i-th column. The resultant matrix or reduced matrix is mathematically obtainable by pre-multiplying A successively by a finite number of unit lower-triangular matrices whose product is again a unit lower-triangular matrix. Now start with the reduced matrix. Look at the second row. If all of its entries are equal to zero, move on to the third row. If we are unable to find any non-zero vectors among the second, third, ... ,m-th rows, the process stops. The reduced matrix is clearly in echelon form. Otherwise, locate the first non-zero vector among the m - 1 rows of the reduced matrix starting from the second. Repeat the process of sweeping out all the entries below the first non-zero entry (pivot) of the chosen non-zero vector. Repeat this process until we could find no more non-zero vectors in the reduced matrix. The reduced matrix is clearly in echelon form. The promised matrix B is simply the product of all pre-multiplying unit lower-triangular matrices employed in the sweep-out process. Clearly, B is a unit lower-triangular matrix. This completes the proof. COROLLARY 5.2.2. Let A be a matrix of order m x n. Let B be the unit lower-triangular matrix obtained in the proof of P 5.2.1 such that B A is in echelon form. Then the rank of A is equal to the total number of non-zero row vectors in BA, or equivalently to the total number of pivots. We have already established the rank factorization of A in P 4.3.1. This result can be obtained as a corollary of P 5.2.1. Let BA = C. Write A = B- 1 C. Let p(A) = a. The matrix C will have exactly a non-zero rows. Eliminate all zero rows. Let F denote the resultant matrix. The matrix F will be of order aXn. Eliminate the corresponding columns from B- 1 • Let R be the resultant matrix which will be of order m x a. It is clear that p(R) = p(F) = a. Further, A = B-1C = RF. We will jot this result as a corollary.

162


COROLLARY 5.2.3. (Rank Factorization) Any given matrix A of order m x n and of rank a -# 0 admits a rank factorization

A=RF

(5.2.1)

where R is of order m x a with rank a and F is of order a x n with rank a. We can polish the echelon form a little. For the given matrix A, determine a lower-triangular matrix B such that BA = G is in echelon form. By interchanging the columns of G, one can produce a matrix in which all the entries below the leading diagonal are zeros, For example, if

G=

01 10 21 02 31] , [o 0 2 0 1

interchange columns 1 and 2 to produce the matrix

1o 01 21 02 31] , [o 0 2 0 0 in which we notice that all the entries below the leading diagonal are zeros. In some cases, one may have to do more than one interchange of columns. Any interchange of columns in a matrix is equivalent to post-multiplying the matrix with an elementary matrix of the type Era. When the matrix G in echelon form is changed into another matrix D in echelon form with the additional property that all entries below the leading diagonal of D are zeros, one can write D = GG, where G is a product of matrices of the type Era. In the final analysis, we will have BAG = GG = D. Let A = AG. Observe that A is obtainable from A by some interchanges of columns of A. Let us put down all these deliberations in the form of a corollary. 5.2.4. Let A be a given matrix and B a unit lowertriangular matrix such that B A = G is in echelon form. Then one can construct a matrix 11 by interchanging columns of A such that BA is in echelon form with the additional property that all entries below the leading diagonal are zeros. COROLLARY


163

5.2.5. (Upper Echelon Form) For any given matrix A there exists a non-singular matrix B such that B A is in upper echelon form. COROLLARY

PROOF. If A is the zero matrix, it is already in upper echelon form. Asswne that A is a non-zero matrix. Identify the i-th colwnn such that it is a non-zero vector and all the first (i - 1) columns are zero vectors. Identify the first entry aji in the i-th colwnn which is nonzero. Add j-th row to the first row. In the resulting matrix, the (1, i)th element is non-zero. Using this element as a pivot, sweep out the rest of the entries in the i-th colwnn. Disregard the first row and the first i colwnns of the reduced matrix. Repeat the above operation on the submatrix created out of the reduced matrix. Continue this process until no non-zero colwnn is left. All these matrices may not be unit lower-triangular matrices. The addition of a row at a lower level of the matrix to a row at a higher level of the matrix cannot be done by pre-multiplying the matrix with a unit lower-triangular matrix. In any case, the final reduced matrix is in upper echelon form and the product of all elementary matrices involved is the desired non-singular matrix

B. For the next result, we need the notion of a principal minor of a square matrix. Let A be a square matrix of order m x m. Let 1 ~ i l < i2 < ... < ip ~ m. The determinant of the submatrix of A obtained from A by deleting the il-th, i 2-th, ... , ip-th rows and i l th, i 2-th, ... , ip-th colwnns from A is called a principal minor of A. The submatrix itself is called a principal submatrix. If iI, i2, ... ,ip are consecutive integers and ip = m, the associated principal minor is called a leading principal minor of A. The corresponding submatrix of A is called a leading principal submatrix. P 5.2.6 (LU Triangular Factorization) Let A be an m x m matrix such that all its leading principal minors are non-zero. Then A can be factorized as (5.2.2) A=LU, where L is a unit lower-triangular matrix and U is a non-singular uppertriangular matrix each of order m x m. PROOF. Let A = (aij). Since all =1= 0, it can be used as a pivot to sweep out all the other elements in the first colwnn. Let us identify the

164


(2,2)-th element a~V in the reduced matrix. Observe that

= The determinant on the left-hand side of the above expression is a leading principal minor of A and by hypothesis, it is non-zero. The second determinant above is the result of the sweep-out process and this does not change the value of the minor. Now it follows that a~;) =1= O. Further, all the leading principal minors of the reduced matrix are non-zero. (Why?) Using as a pivot, sweep out all the entries below the second row in the second column of the reduced matrix. Continue this process until we end up with an upper triangular matrix U. All the operations involved are equivalent to pre-multiplying the matrix A with a series of unit lower-triangular matrices whose product B is clearly a unit lower-triangular matrix. Thus we have B A = U, from which we have A = B- I U = LU, say. Observe that B- 1 is a unit lower-triangular matrix. It is clear that U is non-singular. This completes the proof.

aW

COROLLARY 5.2.7. Let A be a matrix of order m x m for which every leading principal minor is non-zero. Then it can be factorized as A=LDU,

(5.2.3)

where L is unit lower-triangular, U is unit upper-triangular, and D is a non-singular diagonal matrix. PROOF. First, we write A = LUI following (5.2.2). Divide each row of U l by its diagonal element. This operation results in a matrix U which is unit upper-triangular. This operation is also equivalent to premultiplying Ut by a non-singular diagonal matrix D- t . Thus we can write U = D-t Ut . The result now follows. The assumption that the matrix A should have leading principal minors non-zero looks a little odd. Of course, every such matrix is non-singular. Consider the following matrix:

A=[~ ~] .


165

The matrix A is non-singular but the assumption of P 5.2.6 on leading principal minors is not met. There is no way we could write A = LU with L unit lower-triangular and U non-singular upper-triangular. (Try.) In the following corollary we record that this assumption is inviolable. COROLLARY 5.2.8. Let A be a non-singular matrix. Then A admits a factorization of the form (5.2.3) if and only if all the leading principal minors of A are non-zero.

The critical observation is that if A can be factorized in the form (5.2.3), then any leading principal submatrix of A can be factorized in the form (5.2.3). Now the result becomes transparent. The notion of leading minors and leading submatrices can be defined for any general matrix A not necessarily square. A result analogous to P 5.2.6 should work out for rectangular matrices too. PROOF.

5.2.9. Let A be a matrix of order m x n for which all its leading principal minors are non-zero. Then we have the factorization COROLLARY

A=LU,

(5.2.4)

where L is unit lower-triangular of order m x m and U is in upper echelon form. The hypothesis that all leading principal minors are non-zero can be relaxed, but the conclusion will have to be diluted. COROLLARY 5.2.10. Let A be matrix of order m x m such that its il-th, i 2-th, ... , ir-th columns are dependent on the previous columns for some 1 < i l < i2 < ... < ir ~ m. Let B be the matrix obtained from A by deleting the il-th, i2-th, ... , ir-th rows and il-th, i2- th , ... , ir-th columns of A. Suppose all the leading principal minors of B are non-zero. Then we have the factorization

A=LU,

(5.2.5)

where L is unit lower-triangular and U is upper-triangular (with some diagonal elements possibly zero). PROOF. The factorization (5.2.5) can be established by following the same argument that is used in the proof of P 5.2.6. If we encounter a

166


zero leading element at any stage of the reduction process, skip the row containing the leading element and then move on to the next row. The hypothesis of the corollary ensures that when a leading element is zero at any stage, all the elements below it in the same coltunn are zero. When the matrix involved is special, the factorization becomes special too. In the following, we look at symmetric matrices. COROLLARY 5.2.11. Let A be a symmetric matrix of order m x m with all its leading principal minors to be non-zero. Then we have the factorization (5.2.6) A= L6L',

where L is unit lower-triangular and 6 is diagonal with non-zero diagonal entries. PROOF. First, obtain the factorization A = LU following P 5.2.6, where L is unit lower-triangular and U non-singular and upper-triangular. Since A is symmetric,

A = LU = A' = U' L',

from which we note that U = show that 6 is diagonal, then result follows. The diagonality upper-triangular and L-lU' is

L-lU'L'. Let 6 = L-lU'. If we can we can write A = LU = L6L' and the of 6 follows from the fact that U L- 1 is lower-triangular. (Check.)

5.3. Factorization of Matrices with Complex Entries In addition to the factorization results given in the last section, some special results are available when the elements of the matrices belong to the field of complex numbers. Let A be a square matrix with complex numbers as entries. The complex conjugate of A is defined to be that matrix A * obtained from A by replacing the entries of A by their complex conjugates and then taking the transpose of the matrix. If the entries of A are real, the complex conjugate of A is merely the transpose A' of A. A matrix A is said to be Hermitian if A* = A. If A has real entries, recall that A is said to be symmetric if A' = A. Two coltunn vectors al and a2 of the same order with complex entries are said to be orthogonal if aia2 = a2al = O. They are said to be orthonormal if, in addition, aia1 = 1 = a2a2. These notions are not different from what


167

we already know about orthogonal and orthonormal vectors in an inner product space. The relevant inner product space in this connection is equipped with the standard inner product. A square matrix A is said to be tmitary if A* A = AA* = I. A square matrix A with real entries is said to be orthogonal if A' A = AA' = I. We now introduce formally Householder matrices.

en

DEFINITION 5.3.1. Let w be a column vector of order n x 1 with complex entries satisfying w*w = 1. Let

E(w) = In - 2ww*. The matrix E(w) is called a Householder matrix. The following are some of the properties of Householder matrices. P 5.3.2 (1) Every Householder matrix is Hermitian and tmitary. (2) Let the column vectors a and b be such that a*a = b*b, a*b = b*a (which is automatically true if the vectors are real), and distinct. Then there exists a vector w of unit length such that E(w)a = b. (3) Let a and b be two distinct column vectors of the same order n x 1 with the following structure: a' = (a~, a2) and b' = (ai, b2), where al is a column vector of order t x 1. (What this means is that the first t entries of a and b are identical.) Suppose that b2b2 = a2a2 and b2a2 = a2b2. Then there exists a vector w of unit length such that E(w}a = band E( w)( c', 0') = (c', 0') for any column vector c of order t x 1. PROOF. The verification of (1) is easy. For (2), take w = r(a - b), where r > 0 is such that the length of w is tmity. In fact l/r 2 = (a - b)*(a - b). The given conditions on the vectors a and b imply that the vectors (a - b) and (a + b) are orthogonal, i.e., (a - b)*(a + b) = o. We are now ready to prove the assertion:

E(w)a = (In - 2ww*)a -

2 * a+b [In - 2r (a - b)(a - b) ][-2-

- -a+b + -a-b 2 2.

a-b + -2-1

(a- b) = b.

As for (3), take w = r(a - b), where r > 0 is such that the vector w has unit length. This would do the trick. The following is one of the most useful factorizations of a matrix. It is similar to the rank factorization of a matrix presented earlier.

168


P 5.3.3 (QR Factorization) Let A be a matrix of order m x n with complex entries. Let the rank of A be a. Then A can be factorized as (5.3.1) A=QR, where Q is an m x a matrix of rank a such that Q*Q a X n matrix in upper echelon form.

= Ia

and R is an

PROOF. If A = 0, the avowed factorization of A can be set down easily. Assume that A is non-zero. Identify all non-zero coluIIlllS of A. Let 1 :::; it < i2 < ... < ir :::; n be such that ail' ai2' ... ,air are all those columns of A which are non-zero. Let d be the first entry in the column ail' If d is real, let bt be the column vector given by b~ = (Jail ail ' 0, ... ,0). If d is complex, let b~ = (~Jail ail' 0, .., 0). Note that bibt = ail ail and biail = ail bl . By P 5.3.2 (2), there exists a vector Wt of unit length such that E(wdail = bt . Let E(wdA = At. Let us examine closely the structure of AI . The first (it - 1) colUIIlllS of Al are zero vectors. The it-th column of At is bl . Let us work on AI. Identify all non-zero columns of AI. It turns out that il-th, i2-th, . . . , ir-th columns of At are the only ones which are non-zero. Let us work with i 2-th column of At. Let us denote this column by c. Partition c' = (cIlc2' C3,'" ,c..n) = (Ctlc'), say. Let b2 = (cIlvc*c, 0, ... ,0) if C2 is real. If C2 is complex, b2 is modified in the same way as in bt . The vectors C and b2 satisfy the conditions of P 5.3.2(3) with t = 1. There exists a vector W2 of length one such that E(W2)C = b2 . Also, E(W2)b t = bl . Let A2 = E(W2)A t . Let us look at the structure of A2. The it-th, i 2-th, ... ,ir-th columns are the only vectors of A2 which are non-zero. In the it-th column, all entries below the first element are zeros. In the i2-th column, all entries below the first two elements are zeros. The trend and strategy should be clear by now. We now have to work with the matrix A2 and its i3-th column. Continuing this way, we will have Householder matrices E(wt) , E(W2), ... ,E(wr ) such that

is in upper echelon form. Let QI = E(wr)E(wr_l)'" E(W2)E(Wl)' It is clear that Qt is unitary and A = QiRl . Since p(A) = a, exactly a rows of Rl will be non-zero. Delete these rows from Rl and let R be the resultant matrix. Delete


169

the correspondingly numbered columns from Qi and let the resultant matrix be Q. Note that A = QR and thls is the desired factorization. The QR factorization of a matrix is computationally very important. It can be used to determine the rank of the matrix, a g-inverse of the matrix, and a host of other features of the matrix. The method outlined in the proof of P 5.3.3 is called Householder method and used in many computational routines. There is another method of obtaining QR factorization of a matrix based on the Gram-Schmidt orthogonalization process which was outlined in Chapter 2.

Gram-Schmidt Method: Let us denote the column vectors of A by aI, a2, ... ,an. The Gram-Schmidt process is designed to provide vectors bl , ~, ... ,bn such that b;b s = 0 for every r i= s, and Sp(aI, a2, ... ,ai) = Sp(b 1, b2 , ... ,bi )

for each i = 1,2, ... , n, where Sp(at, ... ,ai) denotes the subspace spaIUled by al, ... ,ai. The relationship between aI, a2, ... ,an and bI, b2 , •• . ,bn can be written in the form at

= bt

a3 = rl3 bt

+ b2 + r23 b2 + b3

an = rtnb1

+ r 2n b2 + ... + r n-l,n bn-1 + bn ,

a2 = rt2 bt

(5.3.2)

where rij's and bi' S are determined so as to satisfy bibj = 0 for all i i= j. One computational procedure for obtaining bt , b2, ... ,bn is as follows. Suppose bt , ~, ... ,bi - 1 are determined. We will determine r1 i, r2i, ... , Ti-1,i, and bi. Consider the equation ai = 1"lib1

Since b;b s = 0 for every

1"

+ T2i b2 + ...

i= s, r

,1"i-1,i bi-t

+ bi·

E {I, 2, ... ,i -I},

(5.3.3) If b;br = 0, set 1"ri = O. If b;br Finally, bi is obtained from

i= 0, set rri =

b;adb;br ,

1"

= 1, .. . ,i-l.

(5.3.4)

170


Now that Tli, T2i,'" ,Ti-1,i, and bi are determined, we can proce.ed to the next step of determining T1 ,i+b T2,i+b ... ,Ti,i+1, and bi+l III an analogous fashion. The process is continued until all bb b2, . .. ,bn are determined. Equations (5.3.2) can, indeed, be written in a succinct form. Let A = (at, a2,'" ,an), B = (bt, b2,'" ,bn ), and

C= [

1

T12

0

1

o

0

o

Note that A = BC and C is upper triangular. If some of the bi's are zero, we can omit these columns from B and the correspondingly numbered rows from C resulting in matrices Q1 and R 1, respectively. Thus we will have A = Q1Rt, where the columns of Ql are orthogonal and R1 is in upper echelon form. We can normalize the column vectors of Q1 so that the resultant vectors are orthonormal. Let Q be the matrix so obtained from Q1 by the process of normalization. The matrix RI is also modified to absorb the constants involved in the normalization process with the resultant matrix denoted by R. Thus we have the desired QR decomposition of A. Some computational caution is in order in implementing the GramSchmidt method. From the nature of the formulae (5.3.3) and (5.3.4), some problems may arise if bi bi = 0 or close to zero indicating that ai depends on or closely related to a1, a2, ... ,ai-I. In such a case it may be advisable to shift ai to the last position and consider ai+1 after ai-I' Such a rearrangement of ai decided upon at different stages of the Gram-Schmidt process does not alter the nature of the problem. We can always restore the order by shifting the columns of A and the corresponding rows of R under these circumstances, the resulting matrix Q will still be orthonormal and R will be in echelon form. Modified Gram-Schmidt Method: This is a slightly different way of transforming a given set at, a2, ... ,an of vectors into an orthogonal set of vectors. The procedure is carried out in n stages as detailed in the following.


171

Stage 1 Set bl =

al'

Compute

Stage 2 Set

b2

= a~l). Compute

b*a(l) l'

24 -

2 4

bb

22

and

and

Stage

en - 1)

Set bn-l =

(n-2) an_I'

Tn-l,n

=

Compute

b*n-I an(n-2) b* b n-l n-l

d an

(n-l) _ (n-2) b an - an - Tn -l,n n-l'

Stage n Set

bn

= a~n-l).

If, at any stage, b i = 0, set Ti,i+l = Ti,i+2 = ... = Ti,n = O. One can check that bi bj = 0 for all i i= j. The set bl , b2, ... , bn of vectors and the coefficients Tik'S can be used to set out the QR decomposition of the matrix A = (aI, a2, ... , an).

172


The singular value decomposition (SVD) of a matrix of order m x n is a basic result in matrix algebra. A matrix A of order ~ x n with rank a can be

P 5.3.4 (SVD) factorized as

A = PL:::.Q*

(5.3.5)

where P is of order m x a such that P* P = la, Q is of order n x a such that Q*Q = Ia and L:::. is a diagonal matrix of order a x a with all positive entries. PROOF. A simple proof of (5.3.5) depends on what is called SD (spectral decomposition) of a Hermitian matrix derived later in (5.3.9). Note that AA* is Hermitian of order m x m and using (5.3.9), we have

\2 * + ... + AaPaP a [ = PIPI*+ ... + PaPa*+ Pa+IPa+1 * + ... + PmP m *

\2 * AA * = AIPIPI

where PI, ... ,Pm are orthonormal and \2 1'f 2. = J,..2 = 1, ... , a Pi*AA* Pj = Ai =Oifi#j,i=1, ... ,m,j=1, ... ,m As a consequence of the above two equations, we have

= 0, i = a + 1, ... ,m and with q; = \-lp;A,i = 1, ... ,a P; A

q;qj = 1 ifi =j = 0 if i

# j.

Now consider A = (PIP~

+ ... + Pmp~)A

+ Pap~A + Pa+1P~+1A + ... + Pmp:nA AIPlqi + ... + AaPaq~

= PIPiA + ... =

= PL:::.Q*

where P = (PI: ... : Pa), Q = (ql : ... : ql) . This completes the proof. If A is m x m Hermitian, i.e., A = A*, and non-negative definite (nnd), i.e., x* Ax 2: 0 or positive definite (pd), i.e., x* Ax > 0 for all complex vectors x, then the decomposition takes a finer hue as established in P 5.3.5 and P 5.3.9.


173

P 5.3.5 (Cbolesky Decomposition) Let A be a non-negative definite matrix of order m x m. Then A can be factorized as

(5.3.6)

A=KK*,

where K is a lower-triangular matrix with non-negative entries in the diagonal. If A is positive definite, then all the entries in the diagonal of K are positive and the decomposition is unique. PROOF. The result is obviously true when m = 1. We will establish the result by induction. Assume that the result is true for any nonnegative definite matrix of order (m - 1) x (m - 1) . Let A be a nonnegative definite matrix of order m x m. Partition A as

A =

[a

2

aa

aa*]

Al

'

where a is a real scalar, a is a column vector of order (m - 1) x 1 and the matrix Al is of order (m - 1) x (m - I). (Check that it is possible to partition A the way we did.) Suppose a i= 0. Let x be any vector of order m x 1 partitioned as x' = (xlix'), where Xl is a scalar and x is a column vector of order (m - 1) x 1. Since A is non-negative definite,

°: ; =

X* Ax

= a21xl12 + aXlx*a + aXla*x + x* Alx

-*(A I X +X IaXI + a *-12

'- aa *)-x.

Since this inequality is valid for all complex vectors x, it follows that At - aa* is non-negative definite. By the induction hypothesis, there exists a lower-triangular matrix L of order (m - 1) x (m - 1) with nonnegative diagonal entries such that Al - aa* = LL *. Observe that

[~ ~] [~

a* ] =

L*

[a

2

aa

Take

If a

aa*

K=

[~

K=

[~

= 0, choose

aa*

+ LL*

] _

-

[a

2

aa

aa*]_

Al

- A.

174


where Al = M M* with M being a lower-triangular matrix with nonnegative diagonal elements. If A is positive definite, IKI 1= 0 which means that every diagonal entry of A is positive. In this case, suppose H H* is an alternative factorization of A. By directly comparing the elements of Hand K, one can show that H = K. P 5.3.6 (A General Decomposition Theorem) Let A be a square matrix of order m x m. Then A can be factorized as

A = prp*, where P is unitary and

r

(5 .3.7)

is upper-triangular.

We prove this result by induction. The case m = 1 is clear. Assume that the result is true for all matrices of order (m-l) x (m-1). Let A be a matrix of order m x m. Consider the equation Ax = AX in unknown A, a scalar, and x, a vector. Choose a A satisfying the determinantal equation IA - AIm I = 0, which is a polynomial in A of m-th degree. Let Al be one of the roots of the polynomial. Let Xl be a vector of unit length satisfying AXI = AlXI. Choose a matrix X of order m x (m - 1) such that the partitioned matrix (XliX) is unitary. One can verify that PROOF.

XjAX] X*AX . Note that X* AX is of order (m - 1) x (m - 1). hypothesis, we can write

By the induction

X* AX = QrlQ* with Q*Q

= Im -

l

and

rl

upper-triangular. Note that

[~l i~~]=[~ ~][~l Xj~~Q][~ Consequently,

XjAX] X*AX

[Xi] X* XjAXQ]

rl

[1

0


Let

175

~]

and

xtAXQ] rl Clearly, P is unitary and

r

.

is upper-triangular.

COROLLARY 5.3.7. Let A be of order m x m. In the factorization A = prp* in (5.3.7), the diagonal entries of r are the roots of the polynomial IA - >.Iml = 0 in >.. The diagonal entries of r in Corollary 5.3.7 have a special name. They are the eigenvalues of A. See Section 5.4 that follows. P 5.3.6 is also called Schur's Decomposition Theorem. We will discuss the roots of the polynomial IA - >'11 = 0 in >. in the next section. Some inequalities concerning these roots will be presented in a later chapter. We want to present two more decompositions of matrices before we close this section. First, we need a definition. A square matrix A is said to be normal if A* A = AA*. P 5.3.8 (Decomposition of a Normal Matrix) A normal matrix A can be factorized as

A

where

r

= prp*

(5.3.8)

is diagonal and P unitary.

PROOF. Let us use the general decomposition result (5.3.7) on A. Write A = pr P*, where r is upper-triangular and P unitary. Note that

prr* P* A* A = pr* P* prp* = pr*rp*.

AA* = prp* pr* P* = =

Consequently, rr* = diagonal. (Why?)

r*r.

If

r

is upper-triangular, then

r

has to be

P 5.3.9 (Decomposition of a Hermitian Matrix) A Hermitian matrix A can be factorized as A = prp*,

(5.3.9)

176

where


r

is diagonal with real entries and P unitary.

PROOF. Let us use the general decomposition result (5.3.7) again. Write A = pr P*, where r is upper-triangular and P unitary. Note that A" = pr* P* = A = Pf p... Consequently, r = r* and hence r is diagonal with real entries.

Complements 5.3.1 Let E(w) be the Householder matrix based on the vector w of unit length. Show that: E(w)a = a, if a and ware orthogonal, i.e., a'"w = 0, and E(w)w = -w. 5.3.2 Let o 2 o 4 o 6 Obtain QR factorization of A following the method outlined in the proof of P 5.3.3. Obtain also QR factorization of A following the GramSchmidt as well as the modified Gram-Schmidt process. Comment on the computational stability of the two methods. 5.3.3 Let

A=

[all a12 J a21 a22

be non-negative definite. Spell out explicitly a Cholesky decomposition of A. If A is non-negative definite but not positive definite, explore the source of non-uniqueness in the Cholesky decomposition of A. 5.3.4 The QR factorization is very useful in solving systems of linear equations. Suppose Ax = b is a system of linear equations in unknown x of order n xl, where A of order m x nand b of order m x 1 are known. Suppose a QR decomposition of A, i.e., A = QR is available. Rewriting the equations as Rx = Q*b, demonstrate a simple way of solving the linear equations. 5.3.5 (SQ Factorization) If A is an m x n matrix, m ~ Tt, show that A can be factorized as A = SQ where S is m x m lower triangular and Q is m x n matrix with orthonormal rows. 5.3.6 Work out the SVD (singular value decomposition) of the matrix A in the example 5.3.2 above.


177

5.4. Eigenvalues and Eigenvectors In many of the factorization results on matrices, the main objective is to reduce a given matrix to a diagonal matrix. The question is whether one can attach some meaning to the numbers that appear in the diagonal matrix. In this section, we embark on such an investigation. DEFINITION 5.4.1. Let A be a square matrix of order m x m with complex entries. A complex number A is said to be an eigenvector of A if there exists a non-zero vector x such that

Ax

= AX.

(5.4.1)

We have come across the word "eigenvalue" before in a different context in Chapter 2. There is a connection between what was presented in Chapter 2 in the name of "eigenvalue" and what we are discussing now. This connection is explored in the complements at the end of the section. Rewriting (5.4.1) as (5.4.2) we see that any A for which A - AIm is singular would produce a nonzero solution to the system (A - AIm)x = 0 of homogeneous linear equations in x. The matrix A - AIm being singular would imply that the determinant IA - AIml = O. But IA - AIml is a polynomial in A of degree m. Consequently, one can conclude that every matrix of order m x m has m eigenvalues which are the roots of the polynomial equation IA - AIml = 0 in A. Suppose A is an eigenvalue of a matrix A. Let x be an eigenvector of A corresponding to A. If a is a non-zero scalar, ax is also an eigenvector corresponding to the eigenvalue A. For some special matrices eigenvalues are real.

P 5.4.2 real.

Let A be a Hermitian matrix. Then all its eigenvalues are

Let A be an eigenvalue and x a corresponding eigenvector (aij), i.e., Ax = AX. Take x to be of unit length. This implies

PROOF.

of A that

=

m

x· Ax

m

= L: L: aij XiXj = AX·X = A, i=l j=l

178


where x* = {x}, X2,'" ,xm). We show that x" Ax is real. It is clear that aiixixi is real for all i. If i 1= j,

is real. Hence A is real. In Section 5.3, we presented a decomposition of a Hermitian matrix. The entries in the diagonal matrix involved have a special meaning. p 5.4.3 Let A be a Hermitian matrix and A = prP* its decomposition with P unitary and r = Diag{A}, A2,'" ,Am}. Then A}, A2, .. . ,Am are the eigenvalues of A. PROOF. Look at the determinantal equation IA-AIml = 0 for eigenvalues of A. Note that

IA - AIml = Iprp* - AIml = Iprp* - APP*I -

IP{r - AIm)P*1 = !PI jr - AIm I IP*I

= Ir -

AIml

=

{AI - A){A2 - A)'" (Am - A).

The proof is complete. It can also be verified that the i-th column vector of P is an eigenvector corresponding to the eigenvalue Ai of A. We now spend some time on eigenvectors. Let A and J.L be two distinct eigenvalues of a Hermitian matrix A and x and y the corresponding eigenvectors. Then x and yare orthogonal. Observe that Ax

= AX ~ y* Ax = AY*x,

Ay = J.Ly ~ x* Ay

= J.Lx*y.

Since x* Ay = y* Ax and x*y = y*x, we have (A - J.L)Y*x = 0,

from which we conclude that y*x = 0, i.e., x and yare orthogonal. If A is any matrix and A and J.L are two distinct eigenvalues of A with corresponding eigenvectors x and y, the good thing we can say about x and y is that they are linearly independent. Suppose they are linearly dependent. Then there exist two scalars a and f3 such that

ax + f3y = O.

(5.4.3)


Since the vectors x and yare non-zero, both zero. Note that

Q

179

and f3 have to be non-

0= A(QX + f3y) = QAx + f3Ay = QAX + f3J.LY.

(5.4.4)

Multiplying (5.4.3) by J.L and then subtracting it from (5.4.4), we note that QAX - QJ.LX = 0 Q(A - J.L)x = O.

Since Q =1= 0, A =1= J.L, and x =1= 0, we have a contradiction to the assumption of linear dependence of x and y. We will now discuss the notion of multiplicity in the context of eigenvalues and eigenvectors. The mUltiplicity aD of a root AD of the equation IA - All = 0 is called its algebraic multiplicity. The number 90 of linearly independent solutions of the system (A - AOI)X = 0 of equations is called the geometric multiplicity of the root AD. If A is Hermitian, we show that 90 = aD. This result follows from the decomposition theorem for Hermitian matrices reported in P 5.3.8. From this theorem, we have a unitary matrix P = (XI,X2"" ,xm ) such that A = Pt:::,.P*, where t:::,. = Diag{A}, A2,'" ,Am}, Xi is the i-th column of P, and Ai'S are the eigenvalues of A. Let A(I),A(2),'" ,A(8) be the distinct values among AI, A2, ... ,Am with multiplicities aI, a2, ... ,as respectively. Clearly, al + a2 + ... + a 8 = m. Assume, without loss of generality, that the first al numbers among AI, A2, ... ,Am are each equal to A(I)' the next a2 numbers are each equal to A(2) and so on. Since

and Xi, i = 1,2, ... ,al are orthogonal, it follows that al ~ 91' A similar argument yields that ai ~ 9i for i = 2, 3, ... ,s. Consequently,

m=

8

8

i=1

i=1

L ai :::; L 9i :::; m.

(We cannot have more than m linearly independent vectors.) Hence ai = 9i for all i. This result is not true for any matrix. As an example, let

A=

[~

~].

180


Zero is an eigenvalue of A of multiplicity 2. The corresponding eigenvector is of the form with x i= 0. Consequently, the geometric multiplicity of the zero eigenvalue is one. We will now present what is called the spectral decomposition of a Hermitian matrix A. This is essentially a rehash of P 5.3.8. From the decomposition A = P 6.P* we can write

A = >'IXIXr + >'2X2XZ + ... + where P

>'mxmx~

= (Xl, X2, ... ,xm ), Xi is the i-th column of P, 6. = Diag{>.l, >'2,.·. ,>'m}.

(5.4.5) and

Without loss of generality, assume that >'i i= 0, i = 1,2, ... ,r and >'i = 0, i = r+ 1,1'+2, ... ,m. Let >'(1),>'(2), ... ,>'(5) be the distinct values among >'1, ... ,>'m. We can rewrite (5.4.5) as

A = >'(1) EI + >'(2) E2 + ... + >'(5) E s ,

(5.4.6)

where Ei is the sum of all the matrices XjX; associated with the same eigenvalue >'i. Note that E; = E i , Ei = Ei for all i, and EiEj = for all i i= j. FUrther, p(Et} is the multiplicity of the root >'(1). The form (5.4.6) is the spectral decomposition of A. The spectral decomposition (5.4.6) is unique. We will demonstrate its uniqueness as follows. Suppose

°

(5.4.7) is another decomposition of A with the properties F? = Fi for all i and FiFj = for all i i= j. Subtracting (5.4.7) from (5.4.6), we note that

°

>'(1)(E1

-

FI ) + >'(2)(E2 - F2) + ... + >'(s)(Es - F 5 )

= 0.

(5.4.8)

MUltiplying (5.4.8) by Ei on the left and Fj on the right, for i i= j, we have EiFj = 0. Multiplying (5.4.8) by Ei on the left, we note that Ei = EiFi. Using a similar argument, we can show that EiFi = Ei. Thus we have Ei = Fi for all i. Now we take up the case of an important class of matrices, namely non-negative definite (also called positive senli-definite) matrices.


181

DEFINITION 5.4.4. A Hermitian matrix A is said to be non-negative definite (nnd) if x· Ax ~ 0 for all column vectors x E em. Non-negative definite matrices are like non-negative numbers. For example, we can take the square root of an nnd matrix which is also nnd. We shall now identify a special subclass of nnd matrices.

5.4.5. A Hermitian matrix A is said to be positive definite (abbreviated as pd) if A is nnd and x· Ax = 0 if and only if x = O. What we need is a tangible criterion to check whether or not a given Hermitian matrix is nnd. In the following result, we address this problem. DEFINITION

p 5.4.6

(1) (2) (3) (4)

Let A be a Hermitian matrix. The matrix A is nnd if and only if all its eigenvalues are nonnegative. For the matrix A to be nnd, it is necessary (not sufficient) that all its leading principal minors are non-negative. The matrix A is positive definite if and only if all its eigenvalues are positive. The matrix A is positive definite if and only if all its leading principal minors are positive.

(1) Let A be an eigenvalue of A and x a corresponding eigenvector, i.e., Ax = AX. Note that 0 ::; x· Ax = AX·X from which we have A ~ O. Conversely, suppose every eigenvalue of A is nonnegative. By P 5.3.8, there exists a unitary P such that P* AP = !:::,. = Diag{AI, A2, ... , Am}, where AI, A2, ... , Am are the eigenvalues of A. Let x E em be any given vector. Let y = p. x = (Yll . .. , Ym)'. Then PROOF.

x* Ax = y* P* APy = y. !:::,.y m

=

L AilYil

2

~ O.

i=l

(2) Suppose A is nnd. Then the determinant IAI of A is nonnegative. This follows from P 5.3.8. As a matter of fact,

Observe that any principal submatrix of A is also non-negative definite. Consequently, the leading principal minors of A are non-negative.

182


(3) A proof of this assertion can be built based on the proof of (I). (4) If A is positive definite, it is clear that every leading principal minor of A is positive. Conversely, suppose every leading principal minor of A is positive. We want to show that A is positive definite. Let A = (aij). By hypothesis, au > o. Using au as a pivot, sweep out the first colunm and first row of A. Let B be the resultant matrix. Write

We note the following. 1. Every principal minor of A and the corresponding principal minor of B which includes the first row are equal. 2. Any leading principal minor of Bl of order k x k is equal to (a!l) times the leading principal minor of Al of order (k + 1) x (k + 1). These facts are useful in proving the result. We use induction. The result is obviously true for 1 x 1 matrices. Assume that the assertion holds for any matrix of order (m - 1) x (m - 1). The given matrix A is assumed to be of order m x m and for which all the leading principal minors are positive. Let B be the matrix obtained from A as above. It now follows that every leading principal minor of BI is positive. By the induction hypothesis, BI is positive definite. Consequently, B is positive definite. Hence A is positive definite. (Why?) This completes the proof. Note the difference between the assertions (2) and (4) of P 5.4.6. If all the principal leading minors of A are non-negative, it does not follow that A is non-negative definite. For an example, look at

However, if all the principal minors of the Hermitian matrix are nonnegative, then A is non-negative definite. Complements

5.4.1 Let AI, A2, ... ,Am be the eigenvalues of a matrix A of order m x m. Show that m

Tr A

= LAi i=l

and

IAI = n~lAi.


183

If A is an eigenvalue of A, show that A2 is an eigenvalue of A2. 5.4.3 Let A be a Hermitian matrix of order m X m with complex entries. For row vectors x, y E em, let K(x, y) = xAy*. Show that K(·,·) is a Hermitian conjugate bilinear functional on the vector space

5.4.2

em.

5.4.4 Let K(-'·) be as defined in Complement 5.4.3. Let < ',' > be the usual inner product on em. Let Al 2:: A2 2:: ••. 2:: Am be the eigenvalues of K(·, .). Show that AI, A2, ... ,Am are the eigenvalues of A. 5.4.5 Let A be a Hermitian matrix of order m x m and Ai the i-th leading principal minor of A, i = 1,2, ... ,m. If Al > 0, ... ,Am-I> and Am 2:: 0, show that A is nnd.

° =°

5.4.6 Let A = (aij) be an nnd matrix. If aii = 0, show that aij for all j. 5.4.7 Let A be an nnd matrix. Show that there exists an nnd matrix B such that B2 = A. Show also that B is unique. (It is customary to denote B by Al/2.)

5.4.8 If A is nnd, show that A 2 is nnd. 5.4.9 If A is pd, show that A-I is pd. 5.4.10 If A is positive definite, show that these exists a non-singular matrix C such that C* AC = I. 5.4.11 If A = (aij) is nnd, show that IAI ::; rr~Iaii. 5.4.12 For any matrix B of order m X n, show that BB* is nnd. What is the relationship between eigenvalues of B B* and singular values of B? 5.4.13 Let B = (b ij ) be a matrix of order m x m with real entries. Show that

Hint. Look at A = B' B. 5.4.14 Let A be an n X n nonsingular matrix with complex entries. Show that (A -1)' = A-I if A' = A, where A' is the transpose of A. 5.4.15 Show that a complex symmetric matrix need not be diagonalizable. 5.4.16

Show that every square matrix is similar to its transpose.

184


5.4.17

Let A be a matrix of order m x m given by

A=

[ ~ i : : : :i]. p

p

p

p

1 p

... .. .

Show that A is positive definite if and only if - m~l < p < l. 5.4.18 Let A and B be two Hermitian matrices of the same order. Say A ~ B if A - B is nnd. Prove the following. (1) (2) (3) (4)

c.

If A ~ Band B ~ C, then A ~ If A ~ Band B ~ A, then A = B . If A and Bare nnd and A ~ B, then IAI ~ IBI· If A and B are positive definite, A ~ B, and IAI = IBI, then

A=B. 5.4.19 A Hermitian matrix A is said to be negative semi-definite if x· Ax ~ 0 for all column vectors x E em. A Hermitian matrix A is said to be negative definite if A is negative semi-definite and x'" Ax = 0 only if x = o. Formulate P 5.4.6 for these matrices. 5.5. Simultaneous Reduction of Two Matrices The principal goal of this section is to investigate under what circumstances two given matrices can be factorized in such a way that some factors are common. First, we tackle Hermitian matrices. Before this, we need a result which is useful in our quest. P 5.5.1 Let A be a square matrix of order n x n with complex entries. Let x be a non-zero column vector of order n x 1. Then there exists an eigenvector y of A belonging to the span of x, Ax, A 2x, .... PROOF. The vectors x, Ax, A2 x , ... cannot all be linearly independent. Let k be the smallest integer such that

(5.5.1) for some scalars bo, bl , ... ,b k of the polynomial,

1.

(Why?) Let ILl, IL2,

. .. ,ILk

be the roots


185

of degree k in z, Le., we can write

z k + bk-lZ k-l + bk-2Z k-2 + ... + blZ+ b0 = (z - JLd(z - JL2) ... (z - JLk). Consequently,

0= Akx + bk_lAk-ix + bk_ 2A k- 2x + ... + blAx + box = (A - JLlI)(A - JL2I) ... (A - JLkI)X. Let Y = (A - JL2I)(A - JL3I) ... (A - JLkI)X. It is clear that Y =1= 0 (Why?) and Y is an eigenvector of A. Further, Y belongs to the span of the vectors x, Ax, A2x, ... , Ak-lx. This completes the proof. p 5.5.2 Let A and B be two Hermitian matrices of the same order n X n. Then a necessary and sufficient condition that A and B have factorizations, A = P /::"1 P* and B = P /::"2P*, with P unitary and /::"1 and /::"2 diagonal matrices is that A and B commute, Le., AB = BA. If A and B have the stipulated factorizations, it is clear that AB = BA. On the other hand, let A and B commute. Let Yl be an eigenvector of B. For any positive integer 1·, we show that ArYl is also an eigenvector of B provided that it is non-zero. Since BYI = AYI for some scalar A, we have B(Ar yl ) = Ar BYI = AAr yl , from which the avowed assertion follows. We now look at the sequence Yl, AYl, A2 yI , ... of vectors. There is a vector PI in the span of the sequence which is an eigenvector of A. The vector PI is obviously an eigenvector of B. Thus we are able to find a common eigenvector of both A and B. We can take PI to be of unit length. Let Y2 be an eigenvector of B orthogonal to Pl. We claim that PI is orthogonal to every vector in the span of Y2, AY2, A2y2 , .. .. For any r 2: 1, PROOF.

* (A r)* Y2 PI = Y2*Ar PI = Y2*( a rPI) = a r Y2PI = 0,

where a is the eigenvalue of A associated with the eigenvector PI of A. Following the argument used earlier in the proof, we can find a vector P2 in the span of Y2, AY2, A 2Y2, ... which is a common eigenvector of both A and B. It is clear that P2 and PI are orthogonal. Take P2 to be of unit length. Continuing this way, we obtain orthonormal eigenvectors

186


Pt.P2,'" ,Pn common to both A and B. Let P = (PI,P2,'" ,Pn)' Note that P is unitary and both P* AP and P* B P are diagonal. This completes the proof. Thus the above result clearly identifies the situation in which two Hermitian matrices are diagonalizable by the same unitary matrix. This result can be extended for any number of matrices. The diagonal matrices involved in the decomposition consist of eigenvalues of their respective matrices in the diagonals.

COROLLARY 5.5.3. Let AI, A 2 , .•. ,Ak be a finite number of Hermitian matrices of the same order. Then there exists a unitary matrix P such that P* AiP is diagonal for every i if and only if the matrices commute pairwise, i.e., AiAj = AjAi for all i =1= j. Finally, we present a result similar to the one presented in P 5.5.2 for special matrices. p 5.5.4 Let A and B be two Hermitian matrices at least one of which is positive definite. Then there exists a non-singular matrix C such that C* AC and C* BC are both diagonal matrices. PROOF. Assume that A is positive definite. Then there exists a noosingular matrix D such that D* AD = I. See Complements 5.4.10. Since D* BD is Hermitian, there exists a unitary matrix V such that V*(D* BD)V is diagonal. Take C = DV. Then C* AC = V* D* ADV = V" IV = I, which is obviously diagonal. This completes the proof. It will be instructive to explore the nature of entries in the diagonal matrix C* BC in P 5.5.4. Let C" BC = 6. = diag{ aI, a2, ... ,am}. The equation C* AC = I implies that C" = C-I A -I. Consequently,

6. = C* BC = C- I A-I BC from which we have A-IBC = C6..

(5.5.2)

Let Xi be the i-th column vector of C which is obviously non-zero. From (5.5.2), we have

A - I Bx't --

...

""X ' \.At t,

which means that ai is an eigenvalue of A - I B and Xi the corresponding eigenvector for every i. It remains to be seen that all the eigenvalues of A-I are accounted by al,a2, ... ,am, i.e., al,a2, ... ,am are the roots


187

of the detenninantal equation IA-l B - all = O. The linear independence of Xi'S settles this question. The Xi'S have an additional nice property: xi AXj = 0 for all i j. The detenninantal equation IA-I B - all = 0 makes an interesting reading. This equation is equivalent to the equation IB - aAI = O. The roots of this equation can be called the eigenvalues of B with respect to the positive definite matrix A! The usual eigenvalues of B as we know them traditionally can now be called the eigenvalues of B with respect to the positive definite matrix I.

t=

Complements 5.5.1 If A is positive definite and B is Hermitian, show that the eigenvalues of B with respect to A are real. 5.5.2 Let AI, A 2 , .•. ,Ak be k Hermitian matrices with Al positive definite. Show that there exists a non-singular matrix C such that C* AiC is diagonal for every i if and only if AiAII Aj = AjAII Ai for all i and j. 5.5.3 Let

A=

[: :-:I 3

-1

and B =

13

Determine a non-singular matrix which diagonalizes A and B simultaneously. 5.5.4 Suppose A and B are Hermitian and commute. Let >q, A2, ... ,Am be the eigenvalues of A and J.Ll, J.L2, ••. ,J.Lm those of B. Show that the eigenvalues of A + Bare

for some permutation iI, i 2, ... ,im of 1,2, ... ,m. 5.5.5 (Polar Decomposition) Let A be an n xn nOllsingular matrix. Then there exist a positive definite matrix C and an orthogonal matrix H such that A = CH. (Hint: Consider the positive definite matrix AA' and take C = (AA')t the positive definite square root of AA'. Then take H = C-I A. Verify that H is orthogonal.)

188


5.6. A Review of Matrix Factorizations Because of the importance of matrix factorization theorems in applications, all major results of Chapter 5 and some additional propositions not proved in the chapter are recorded in this section for ready reference. The reader may consult the references given to books and papers for further details. We use the following notations. The class of m x n matrices is represented by Mm,n and the matrices of order n x n by Mn. Triangular Matrices. A matrix A = (aij) E Mn is called upper triangular if aij = 0 whenever j < i and lower triangular if aij = 0 whenever j > i. A unit triangular matrix is a triangular matrix which has unities in the diagonal. Permutation Matrices. A matrix P E Mn is called a permutation matrix if exactly one entry in each row and column is equal to 1, and all other entries are zero. Premultiplication by such a matrix interchanges the rows and postmultiplication, the columns. Hessenberg Matrices. A matrix A E Mn is said to be an upper Hessenberg matrix if aij = 0 for i > j + 1, and its transpose is called a lower Hessenberg matrix. if

Tridiagonal Matrices. A E Mn is said to be a tridiagonal matrix = 0, whenever Ii - jl > 1. For example

aij

a12

a13

a12

0

a22

a23

a22

a23

a32

a33

a32

a33

o

a43

o

a43

are upper Hessenberg and tridiagonal matrices respectively. Givens Matrices. A matrix A(l, m; c, s) E Mn is said to be a Givens matrix if aii alm

= =

= a mm = c and aij = 0 elsewhere.

1, i =1= l, i =1= m; -S, aml

= s,

all

We may choose c = cosO and s = sinO. Geometrically, the Givens matrix A = (l, m; c, s) rotates the l-th and m-th coordinate axes in the (l, m)-th plane through an angle O.


189

Other matrices. A E Mn is said to be Hennitian if A = A*, positive (negative) definite if x* Ax > 0« 0) for all nonzero x E en and non-negative (nonpositive) definite if x* Ax ;::: O(x* Ax ::; 0) for all x E en. The abbreviations pd is used for positive definite and nnd for non-negative definite. An alternative term for non-negative definite used in books on algebra is positive semi-definite abbreviated as psd. P 5.6.1 (Rank Factorization) Let A E Mm,n and rank p(A) = k. Then there exist matrices R E Mm,k, FE Mk,n and p(R) = p(F) = k such that A = RF. In the following propositions A E Mn represents a general matrix, L E M n , a lower triangular matrix and U E M n , an upper triangular matrix, all with complex entries unless otherwise stated. P 5.6.2

(LU Factorization Theorems)

(1) If all the leading principal minors of A are nonzero, then A can be factorized as A = LU, where the diagonal entries of L can all be chosen as unities. (2) If A is nonsingular, there exists a permutation matrix P E Mn such that P A = LU, where the diagonal entries of L can all be chosen as unities. (3) In any case there exist permutation matrices P1 , P2 E Mn such that A = P1 LU P2 . If A is nonsingular, it may be written as A = P1 LU. P 5.6.3

(Schur'S Triangulation Theorems)

(1) Let At, ... ,An be eigenvalues of A in any prescribed order. Then there exists a unitary matrix Q E Mn such that Q* AQ = U or A = QUQ*, where U is upper triangular with the eigenvalues A1, ... ,An as diagonal entries. That is, every square matrix is unitarily equivalent to a triangular matrix whose diagonal entries are in a prescribed order. If A is real and if all its eigenvalues are real, then U may be chosen to be real orthogonal. (2) Given a real A E Mn with k real eigenvalues, A1, ... ,Ak, and Xj + iYj as complex eigenvalues for j > k, there exists a real orthogonal matrix Q E Mn such that A = QRQ' where R is a quasi diagonal n x n matrix

190


R=

and m = (n + k)/2, with blocks

R OJ·· of size

[

= [

xj -Cj

j b ], Jbjcj Xj

k,j ~ k

if i

1x2 2x 1

ifi~k,j>k

> k,j ~ k if i > k, j > k

if i

2x2 Zj

~

1x 1

=

Yj,

for j > kj

bj

2:

Cj, b j Cj

> O.

Note that ~i = Ai, i = 1, ... ,k. For an application of this result to probability theory see Edelman (1997). P 5.6.4 (QR Factorization) . Let A E Mm,n. Then A can be factorized as A = QR, where Q E Mm is unitary and

R= {

[~]

[R1 : 8 11

if m > n, if m ~ n,

where Ro E Mn and R1 E Mm are upper triangular and 8 1 E Mm ,n-m' P 5.6.5 (Upper Hessenberg Reduction) For any A, there exists a unitary matrix Q E Mn such that QAQ* = Hu (upper Hessenberg). P 5.6.6 (Tridiagonal Reduction). If A is Hermitian, there exists a unitary matrix Q E Mn such that QAQ* = HT (Tridiagonal). P 5.6.7 (Normal Matrix Decomposition) If A is normal, i.e., AA* = A* A, then there exists a unitary matrix Q E Mn such that A = QAQ*, where A E Mn is a diagonal matrix with the eigenvalues of A as diagonal elements. (Spectral Decomposition) Let A be Hermitian. Then A = QAQ*, where Q E Mn is unitary and A E Mn is diagonal with real entries, which are the eigenvalues of A. If A is real symmetric, then P 5.6.8


A

= QAQ' where Q

191

is orthogonal.

P 5.6.9 (Singular Value Decomposition) For A E Mm,n, we have A = V D. W*, where V E Mm and W E Mn are unitary matrices and D. E Mm,n has non-negative elements in the main diagonal and zeros elsewhere. If p(A) = k, then A = VoD.oWO', where Vo E Mm,k and Wo E Mn,k are such that Vo*Vo = h, WO'Wo = hand D.o E Mk is a diagonal matrix with positive elements in the main diagonal. P 5.6.10 (Hermitian Matrix Decomposition) If A is Hermitian, we have the factorization A = SD.S*, where S E Mn is nonsingular and D. E Mn is diagonal with +1,-1 or 0 as diagonal entries. The number of +l's and -l's are same as the number of positive and negative eigenvalues of A and the number of zeros is n - p(A). P 5.6.11 (Symmetric Matrix Decomposition) If A is real symmetric, then A has the factorization A = SD.S', where S E Mn is nonsingular and D. is diagonal with +1 or 0 as diagonal entries and p(D.) = p(A). P 5.6.12 (Cholesky Decomposition) If A is Hermitian and nnd, then it can be factorized as A = LL *, where L E Mn is lower triangular with non-negative diagonal entries. The factorization is unique if A is nonsingular. P 5.6.13 (General Matrix Decomposition) Any A can be factorized as A = SQ'L,Q*S-I, where S E Mn is nonsingular, Q E Mn is unitary and 'L, is diagonal with non-negative entries. P 5.6.14 (Polar Decomposition) Any A with rank k can be factorized as A = SQ, where S E Mn is nnd with rank k and Q E Mn is unitary. If A is nonsingular, then A can be factorized as A = CQ, where QQ' = I and C = C'. P 5.6.15 (Jordan Canonical Form) Let A be a given complex matrix. Then there is a nonsingular matrix S E Mn such that A = SJS-l where J is a block diagonal matrix with the r-th diagonal block as Jnr(Ar) E M nr , r = 1, ... , k and nl + .. .+nk = n. The Ai'S are eigenvalues of A which are not necessarily distinct. The matrix Jnr(Ar) = (aij) is defined as follows (see Horn and Johnson (1985) for a detailed proof):

192


\ ~• -- 1, ... , n r,. at,t+l · . -- 1, ~• -- 1, a it. -- An

.•• ,

n r - 1·,a · . = 0 elsewhere. tJ

P 5.6.16 (Takagi Factorization) If A E Mn is symmetric (i.e., A = A'), then there exists a unitary Q E Mn and a real non-negative diagonal matrix E such that A = QEQ'. The columns of Q are an orthogonal set of eigenvectors of AA and the corresponding diagonal entries of E are the non-negative square roots of the corresponding eigenvalues of AA. [If A = (aij), A = (aij), where aij is the complex conjugate of aij.] P 5.6.17 Let A E Mn be given. There exists a unitary U E Mn and an upper triangular ~ E Mn such that A = U ~U' if and only if all the eigenvalues of AA are real and non-negative. Under this condition all the main diagonal entries of ~ may be chosen to be non-negative. P 5.6.18 (Complete Orthogonal Theorem) Given A E Mm,n with p(A) = k, there exist unitary matrices Q E Mm and W E Mn such that Q* AW

=

(~ ~)

where U E Mk is upper triangular.

P 5.6.19 (Similarity of Matrices) Every A E Mn is similar to a symmetric matrix. [A is similar to B if there exists a nonsingular S E Mn such that B = S-1 AS.]

P 5.6.20 (Simultaneous Singular Value Decomposition) Let E Mm,n. Then there exist unitary matrices P E Mm and Q E Mn such that A = PEl Q* and B = PE2Q* with both El and E2 E Mm,n and diagonal if and only if AB* and B* A are both normal [G is said to be normal if GG* = G*G].

A, B

P 5.6.21 For a set :F = {Ai, i E I} c Mm,n, there exist unitary matrices P and Q such that Ai = PAiQ* for all i E I and Ai are all diagonal if and only if each Ai Aj E Mn is normal and 9 = {AiAj : i,j E I} c Mm is a commuting family. P 5.6.22 Let A = A*, B = B* and AB = BA. Then there exists a unitary matrix U such that U AU* and U BU* are both diagonal. P 5.6.23 The Hermitian matrices, AI, A 2, ... , are simultaneously diagonalizable by the same unitary matrix U if they commute pairwise.

Note: The main references for this Chapter are: Bhatia (1991), Datta (1995), Golub and van Loan (1989), Horn and Johnson (1985, 1990), and Rao (1973c).

CHAPTER 6 OPERATIONS ON MATRICES Matrix multiplication is at the core of a substantial number of developments in Matrix Algebra. In this chapter, we look at other multiplicative operations on matrices and their applications. 6.1. Kronecker Product

Let A = (aij) and B = (b ij ) be two matrices of order m x nand p x q, respectively. The Kronecker product of A and B is denoted by

A ® B and defined by

A®B=

For a specific example, take m = 3, n = 2, p = 2, and q = 3. The Kronecker product of A and B spreads out to be a 6 x 6 matrix, allb 13 1 a12 b l1 allb23 1 a12 b 21

a12 b 12 a12 b 13 a12 b 22 a12 b 23

a21 b 21 a21 b 22

a21 b23 1 a22 b ll a21 b 23 1 a22 b 21

a22 b 12 a22 b 13 a22 b 22 a22 b 23

a31 b ll a31 b 12 a31 b 21 a31 b 22

a31 b 13 1 a32 b ll a31 b 23 1 a32 b 21

a32 b 12 a32 b 13 a32 b 22 a32~3

allb ll allb 12 allb 21 allb22

- - -- - --- - --1- - --- - --- - A®B= 3x2 2x3

a21 bl l a21 b 12

- - -- - --- - --1- - --- - --- - -

In the general case, the Kronecker product, A ® B, of A and B is of order mp x nq. From the very definition of the Kronecker product, it is 193

194


clear that there are no restrictions on the numbers m, n, p, and q for the product to be meaningful. Notice also that the equality A 0 B = B 0 A rarely holds just as it is the case for the usual multiplication of matrices. However, the Kronecker product has one distinctive feature. The matrix B 0 A can be obtained from A 0 B by interchanging rows and columns of A 0 B. This feature is absent in the usual multiplication of matrices. We list some of the salient properties of this operation in the following proposition. Most of these properties stem directly from the definition of the product. P 6.1.1 (1) The operation of performing Kronecker product on matrices is associative. More precisely, if A, B, and C are any three matrices, then (A ®B) ®C = A0 (B 0C).

It is customary to denote (A ® B) 0 C by A ® B 0 C. (2) If A, B, and C are three matrices with Band C being of the same order, then

A0(B +C) = A®B +A®C. (3)

If a is a scalar and A is any matrix, then

o0A

= oA = A0o.

(In the Kronecker product operation, view a as a matrix of order 1 X 1.) (4) If A, B, C, and D are four matrices such that each pair A and C and Band D is conformable for the usual multiplication, then

(A®B)(C®D) = AC0BD. (5)

If A and B are any two matrices, then

(A 0 B)' = A' 0 B'. (6)

If A and B are any two matrices, then

(A 0 B)'" = A* 0B*.

Operations on Matrices

(7) then

195

If A and B are square matrices not necessarily of the same order, tr(A ® B) = [tr(A)][tr(B)J.

(8) If A and B are non-singular matrices not necessarily of the same order, then One of the most important questions concerning Kronecker products is about the relationship of the eigenvalues of the Kronecker product of two matrices and the eigenvalues of the constituent matrices of the product. We will address this question now. Let A and B be two square matrices with eigenvalues A1,A2, ... ,Am and Jl1,Jl2, ... ,Jln, respectively. Then the AiJlj, i = 1,2, ... ,m and j = 1,2, ... ,n are the eigenvalues of A ® B. P 6.1.2

PROOF.

If A is an eigenvalue of A with a corresponding eigenvector

x, and Jl is an eigenvalue of B with a corresponding eigenvector y, it is easy to show that AJl is an eigenvalue of A ® B with a corresponding eigenvector x ® y. As a matter of fact, note that (A ® B)(x ® y) = (Ax) ® (By) = (AX) ® (JlY) = AJl(X ® V),

which settles the avowed assertion. This does not prove that the eigenvalues of A®B are precisely AilLj, i E {1,2, ... ,m}andj E {1,2, ... ,n}. (Why?) Let us prove the assertion invoking the General Decomposition Theorem (P 5.3.6). There exist unitary matrices U of order m x m and V of order nxn such that U AU* = 6 1 and V BV* = 6 2 , where 6 1 and 6 2 are upper-triangular, the diagonal entries of 6 1 are the eigenvalues of A, and the diagonal entries of 6 2 are the eigenvalues of B. Note that (U ® V)(A ® B)(U ® V)* = (U ® V)(A ® B)(U* ® V*) = (U AU*) ® (V BV*) = 6

1

®6

2.

We also note that U ® V is unitary and 6 1 ® 6 2 upper-triangular. The diagonal entries of 6 1 ® 6 2 should exhaust all the eigenvalues of A ® B. (Why?) This completes the proof.

196


We now look at the status of eigenvectors. A proof of the following result has already been included in the proof of P 6.1.2.

P 6.1.3 Let x be an eigenvector corresponding to some eigenvalue of a matrix A and y an eigenvector corresponding to some eigenvalue of a matrix B. Then x ® y is an eigenvector of A ® B. A word of warning. Not all eigenvectors of A ® B do arise the way it was described in P 6.1.3. (See the contrast in the statements of P 6.1.2 and P 6.1.3.) As a counterexample, look at the following 2 x 2 matrices:

A=B=[~ ~].

Let x' = (1,0). The eigenvalues of A are Al = A2 = 0 and the eigenvalues of Bare J.Ll = J.L2 = O. Only non-zero multiples of x are the eigenvectors of A. But A ® B has only three linearly independent eigenvectors, written as rows,

u'

= (I, 0, 0, 0),

v'

= (O, 1,0,0)

and w'

= (0,0,1,0).

There is no way we can write u, v and w in the way expostulated in P 6.1.3! P 6.1.2 has a number of interesting implications. We chronicle some of these in the following Proposition. P 6.1.4 (I) If A and B are non-negative definite matrices, so is

A®B. (2) If A and B are positive definite, so is A ® B. (3) If A and B are matrices of order m x m and n x n, respectively, then (using the notation 1· 1for determinant),

(4)

If A and B are two matrices not necessarily square, then rank(A ® B) = [rank{A)][rank{B)].

Hint: Look at (AA*) ® (BB*). It is time to look at the usefulness of Kronecker products. Consider linear equations AX = B,

(6.1.1)


197

where A and B are known matrices of orders m x nand m x p, respectively, and X is of order n x p and unknown. As an example, look at (6.1.2) There are two ways we can write these equations in the format we are familiar with. One way is: a12

a22

o o

0 0 all

(6.1.3)

a21

Let x' = (Xl,X2,X3,X4) and b' = (b ll ,b2.,b 12,b22). The system (6.1.3) can be rewritten as (6.1.4) (12 ® A)x = b. Another way is:

o (6.1.5)

Let y' = (X},X2,X3,X4) and c' = (bll,b12,b21,b22). The system (6.1.5) can be rewritten as (6.1.6)

In the general case of (6.1.1), let x be the column vector of order np xl obtained from X = (Xij) by stacking the rows of X one by one, i.e.,

This vector has a special name. We will introduce this concept in the next section. Let b be the column vector of order mp x 1 obtained from B = (b ij ) in the same way x was obtained from X. The system of equations (6.1.1) can be rewi·itten as (6.1.7)

198


Suppose m = n, i.e., A is a square matrix. Then the system (6.1.7) has a Wlique solution if A is non-singular. In the general case, a discussion of the consistency of the system AX = B of matrix equation now becomes easy in (6.1.7), courtesy of Kronecker products! Another matrix equation of importance is given by (6.1.8)

AX+XB=C,

where A is of order m x m, B of order n x n, C of order m x n, and X of order m x n. The matrices A, B, and C are known and X is Wlknown. Let x be the column vector of order mn x 1 obtained from X by stacking the rows of X, and c is obtained analogously from C. It can be shown that the system (6.1.8) is equivalent to

(A ® In

+ 1m ® B'}x =

c.

(6.1.9)

Now we can say that the system (6.1.8) admits a unique solution if and only if the matrix (6.1.10) D = A ® In + 1m ® B' of order mn x mn is non-singular. The matrix D is very special. It will be certainly of interest to know when it is non-singular. First, we would like to say something about the eigenvalues of D. P 6.1.5 Let D be as specified in (6.1.10) and AI, Az, ... ,Am be the eigenvalues of A and J.ll' J.l2, ... ,J.ln be those of B. Then the eigenvalues of D are Ai + !Lj, i = 1,2, ... ,m and j = 1,2, ... ,n. PROOF.

Let

E

> 0 be

any number. Look at the product

(Im + EA) ® (In + EB') = 1m ® In + E(A ® In + 1m ® B'} + EZ A ® B' = 1m ® In

+ ED + EZ A ® B'.

The eigenvalues of 1m + EA are 1 + EAl, 1 + EAz, . " , 1 + EAm and those of In+EB are 1+EJ.ll' 1+EJ.l2, ... ,1+EJ.ln. Consequently, the eigenvalues (Im + EA) ® (In + EB') are all given by

Opemtions on Matrices

199

for i = 1, 2, ... ,m and j = 1,2, ... ,n. Since t is arbitrary, it now follows that the eigenvalues of D are all given by (Ai + /lj) for i = 1, 2, . .. ,m and j = 1,2, ... ,n. (Why?) We can reap some benefits out of P 6.1.5. The non-singularity of D can be settled. COROLLARY 6.1.6. Let D be as defined in (6.1.9) and Ai'S be the eigenvalues of A and /l/s those of B. Then D is non-singular if and only if Ai + /lj =1= 0 for all i and j.

Complements 6.1.1 The matrices A and B are of the same order and so are the matrices C and D. Show that (A+B)®(C+D) =A®C+A®D+B®C+B®D.

6.1.2 Let x and y be two column vectors not necessarily of the same order. Show that x' ® y = yx' = y ® x'. 6.1.3 Let A and B be matrices of orders m x nand n x p, respectively. Let x be a column vector of order q x 1. Show that (A ®x)B = (AB) ® x.

6.1.4 Let A be a matrix of order m x n. Define the Kronecker powers of A by etc. Let A and C be two matrices of orders m x nand n x p, respectively. Show that (AC)12J = AI2JCI2J. Hence show that (AC)lkJ = AlkJClkJ for all positive integers k. 6.1.5 Let AX = B be a matrix equation. Provide a criterion for the existence of a solution to the equation. If the matrix equation is consistent, describe the set of all solutions of the equation. 6.1.6 Obtain a solution to the matrix equation AX = B, where A=

4

2

5

3

[ 3 -1

-!-1]

and B =

[-!1!3] .

200


6.1.7 By looking at the eigenvalues of A and B, show that the following matrix equation has a unique solution:

[~

-1]

2 X+X

[-3

1

Determine the unique solution. 6.1.S By looking at the eigenvalues of A and B, show that the following matrix equation has more than one solution:

[01 -1]

2 X+X

[-3

0

Determine all solutions of the equation. 6.1.9 Show that the matrix equation AX - X B = C has a unique solution if and only if A and B have no common eigenvalues. 6.1.10 For what values of j.L, the matrix equation

AX -XA =ILX has a non-'t rivial solution in X. If j.L to the equation.

= -2, obtain a

non-trivial solution

6.2. The Vee Operation One of the most common problems that occurs in many fields of scientific endeavor is solving a system of linear equations. Typically, a system of linear equations can be written in the form Ax = b, where the matrices A and b of orders m x nand m xl, respectively, are known and x of order n x 1 unknown. Another problem of similar nature is solving matrix equations of the form AX = B or AX + X B = C, where A, B, and C are known matrices and X unknown. These matrix equations can be recast in the traditional linear equations format as explained in Section 6.1. The vec operation and Kronecker products playa key role. Let A = (aij) be a matrix of order m x n. One can create a single column vector comprising all the entries of A. This can be done in two ways. One way is to stack the entries of all the rows of A one after another starting from the first row. Another way, which is more popular, is to stack the coluIIUls of A one underneath the other. Let us follow the popular way. Let ai be the i-th column of A, i = 1,2, ... ,n.


201

Fonnally, we define the vec of A as the column vector of order mn x 1 given by

v~(A) ~ [ ] . The notation vee A is an abbreviation of the operation of creating a single column vector comprising all the entries of the matrix A in a systematic way as outlined above. In the new notation, the matrix equation AX = B, where A is of order m x n, X of order n x p, and B of order m x p, can be rewritten as

(Ip ® A)veeX = veeB. This is just a system of linear equations. If the system is consistent, i.e., admits a solution, one can write down all solutions to the system. We will now examine some properties of the vec operation. One thing we would like to emphasize is that the vec operation can be defined for any matrix not necessarily square. Another point to note is that if vee(A) = vec(B), it does not mean that A = B. The matrices A and B could be of different orders and yet vec(A) = vec(B) is possible. P 6.2.1 (1) If x and yare two column vectors not necessarily of the same order, then vec(xy') = y ® x. (2)

If A and B are matrices of the same order, then

tr(A' B) (3)

= [vee(A)l'vec(B).


vee(A + B) = vec(A)

+ vec(B).

(4) If A, B, and C are three matrices such that the product ABC makes sense, then

vec(ABC) = (C' ® A)veeB.


202

(5) If A and B are two matrices of orders m x nand n x p, respectively, then vec(AB) = (B' ® Im)vecA = (Ip ® A)vecB. PROOF. The assertions (1), (2), and (3) are easy to verify. We tackle (4). Let A = (aij), B = (b ij ), and C = (Cij) be of orders m x n, n x p, and p x q, respectively. Let bI , b2, .. . ,bp be the columns of B. Let ell e2, ... ,e p be the columns of the identity matrix Ip. We can write p

B

= Blp = (bI'~'· ..

,bp)(el' e2,· .. ,ep)'

=L

bjej.

j=1

Consequently, by (3) and (4), vec (ABC) p

p

p

=vec(A(Lbjej)C) = Lvec(AbjejC) = Lvec«Abj)(C'ej)') j=1 j=1 j=1 p

p

= L(C'ej) ® (Abj) = L(C' ® A)(ej ® bj ) j=1 j=1 P

= (C' ® A) L(ej ® bj ) j=I P

= (C' ® A) Lvec(bjej) j=1 p

= (C' ® A)vec(L bjej) = (C' ® A)vecB. j=1 The assertion (5) follows from (4) directly if we note that the matrix AB can be written in two ways, namely, AB = ImAB = AB/p. Complements

6.2.1 Let A, B, C, and D be four matrices such that ABC D is square. Show that tr(ABCD) = (vecD')'(C' ® A)vecB = (vec(D'))'(A ® C')vecB'. 6.2.2 Give a necessary and sufficient condition for the existence of a solution to the matrix equation AX B = C, where A, B, and C are all matrices of the same order m x m. Hint: Use P 6.2.1(4).


203

6.3. The Hadamard-Schur Product Let A = (aij) and B = (bij) be two matrices of the same order. The Hadamard-Schur (hereafter abbreviated as HS) product of A and B is again a matrix of the same order whose (i,j)-th entry is given by aijbij. Symbolically, the product is denoted by A . B = (aijb ij ). The HS product is precisely the entry-wise product of A and B. Let Mm,n be the collection of all matrices of order m x n with complex entries. We have already known that Mm,n is a vector space with respect to the operations of addition of matrices and scalar multiplication of matrices. (If m = n, we denote Mm ,n by Mn.) The HS product is associative and distributive over matrix addition. The identity element with respect to HS multiplication is the matrix J in which every entry is equal to 1. In short, Mm,n is a commutative algebra, i.e., a commutative ring with a multiplicative identity. If m and n are different, the usual matrix multiplication of matrices does not make sense. However, if m = n, the usual matrix multiplication is operational in M n , and Mn is indeed a non-commutative algebra. In this section, we study some of the properties of HS multiplication and present some statistical applications. Some of the properties mentioned earlier are chronicled below.

P 6.3.1

(1)


A·B=B · A. (2)

If A, B, and C are three matrices of the same order, then

A · (B · C) = (A· B)· C. (Now the brackets in multiplication involving three or more matrices can be deleted.) (3) If A, B, C, and D are four matrices of the same order, then

(A

+ B)· (C + D) =

A ·C

+ A · D + B · C + B · D.

(4) If A is any matrix and 0 is the zero matrix of the same order, then A·O = O. (5) If A is any matrix and J is the matrix of the same order each entry of which is 1, then A . J = A.

204


(6) If m = n, A = (aij) is any matrix, and 1m is the identity matrix, then A · l m =diag(all,a22, .. . ,amm ). (7) If A and B are any two matrices of the same order, then

(A · B)' = A' . B'. (8) If A = (aij) is any matrix with the property that each aij =1= 0, and B = (l/aij), then A . B = J. (The matrix B is the (HS) multiplicative inverse of A.) There seems to be no universal agreement on an appropriate name for entry-wise product of matrices of the same order. In some research papers and books, the product is called Schur product. In 1911, Schur conducted a systematic study of what we call HS multiplication. In 1899, Hadamard studied properties of three power series J{z) = Eanz n , g{z) = Ebnz n , and h{z) = Eanbnz n , and obtained some remarkable results. Even though he never mentioned entry-wise multiplication of matrices in his study but the idea was mute when he undertook the study of coefficient-wise multiplication of two power senes. The following is one of the celebrated results of Schur. It can be proved in a number of different ways. We will concentrate just on the one which is statistical! P 6.3.2 (Schur's Theorem) If A and B are two non-negative definite matrices of the same order, then A . B is also non-negative definite. If A and B are both positive definite, then so is A . B . PROOF. Let X and Y be two independent random vectors with mean vector 0 and dispersion matrices A and B, respectively. The random vector X . Y has mean vector 0 and dispersion matrix A . B. It is clear that every dispersion matrix is non-negative definite. The nonstatistical proof is as follows using Kronecker product A (8) B. The HS product A · B of two matrices can be regarded as a submatrix of A ® B of A and B. Let A and B be two square matrices of the same order m . Consider the submatrix of A (8) B by retaining its rows by numbers 1, m + 2, 2m + 3, ... ,{m - l)m + m = m 2 and the columns by numbers 1, m + 2, 2m + 3, ... , (m - l)m + m = m 2 and chucking out the rest. This submatrix is precisely A . B and moreover, it is indeed a principal submatrix of A (8) B. If A and B are non-negative definite,


205

then so is A ® B. Consequently, any principal submatrix of A ® B is also non-negative definite. This is another proof of P 6.3.2. In the general case when m and n are different, the HS product A· B is still a submatrix of the Kronecker product A ® B, and the same proof holds. Schur's theorem has a converse! If A is a non-negative definite matrix, is it possible to write A as a HS product of two non-negative definite matrices? The answer is, trivially, yes. Write A = A· J. Recall that J is the matrix in which every entry is equal to 1. If A is positive definite, is it possible to write A as a HS product of two positive definite matrices? The answer is yes. See the complements at the end of the section. Rank and HS product are the next items to be considered jointly. The following result provides an inequality. P 6.3.3 Let A and B be two matrices of the same order m x n. Then rank(A· B) ~ [rank(A)][rank(B)]. PROOF. Let us use rank factorization of matrices. Let A and B have ranks a and b, respectively. Then there exist matrices X =

(Xl,X2, ... ,xa)ofordermxa, Y=(Yl,Y2, ... ,Ya)ofordernxa, Z= (Zl, Z2, ... ,Zb) of order m x b, and U = (Ul' U2, ... , Ub) of order n x b

such that A

=

a

XY'

= 2:= xiyi

and B

=

b

ZU'

= 2:= ziui.

i=l

The matrices

i=l

X and Y each has rank a and Z and U each has rank b. Note that a

b

a

b

A· B = (LxiyD' (LZjuj) = LL(XiyD· (Zjuj) i=l

a

j=l

i=l j=l

b

= LL(Xi· Zj)(Yi· Uj)'. i=l j=l

Consequently, p(A - B) ~ ab = [p(A)][P(B)]. See Complement 6.3.1 at the end of this section. The stipulated inequality follows if we observe that each matrix within the summation symbols is of rank 1 at most. The inequality stated in P 6.3.3 seems to be very crude. On one hand, the rank of A· B cannot exceed min{ m, n} and on the other hand, [rank(A)][rank(B)] could be [min[m, n]j2. However, equality in P 6.3.3

206


is possible. Let

A

=

[~ ~

H]

0011

and B

=

[~ ! ~

!]

0101

Note that rank(A) = rank (B) = 2 and rank(A . B) = 4. Next we concentrate in obtaining some bounds for the eigenvalues of HS products of matrices. For any Hermitian matrix A of order m x m, let AI(A) 2: A2(A) 2: ... 2: Am(A) be the eigenvalues of A arranged in decreasing order. P 6.3.4 Let A and B be two non-negative definite matrices of order m x m. Let b1 and bm be the largest and smallest entries respectively among the diagonal entries of B. Then

PROOF. This is virtually a consequence of variational characterization of the largest and smallest eigenvalues of a Hermitian matrix. We will see more of this in a later chapter. We need to note right away that for any Hermitian matrix A and vector x,

(6.3.1) One can establish this inequality by appealing to P 5.3.8. Note that

Next we note that A - Am(A)Im is non-negative definite. Use (6.3.1). By P 6.3.2, both B·(A-Am(A)Im ) and [Am(A)]B·lm are non-negative definite. If x is a vector of unit length, then

(Why?) Since x is arbitrary, it follows that Am(A . B) 2: bm[Am(A)]. In a similar vein, the inequality Al (A . B) $ bt[A2(A)] follows. This completes the proof.


207

Let us see what we can get out of this inequality. If B = (b ij ) is Hermitian, then Am(B) :::; bii :::; Al (B) for all i. This follows from (6.3.1). Take x to be the unit vector with the i-th component being 1 and all other components being zeros. This simple observation yields the following result. COROLLARY 6.3.5. If A and B are two non-negative definite matrices of the same order m, then for all j

This corollary resembles the result on the rank of an HS product (see P 6.3.3). For the next result, we need the notion of a correlation matrix. A non-negative definite matrix R = (Pij) is said to be a correlation matrix if Pii = 1 for all i, i.e., every diagonal entry of R is equal to unity. If R is a correlation matrix, then Ipijl :::; 1 for all i and j. (Why?) If the correlation matrix R is non-singular, then Ipijl < 1 for all i i= j. Correlation matrices arise naturally in Multivariate Analysis. Let X be a random vector with dispersion matrix ~ = (O"ij). Define Pij = correlation between Xi and Xj = O"ij/[O"iiO"jjj1/2= [covariance between Xi and XjJl[(standard deviation of Xi) (standard deviation of Xj)], where Xi is the i-th component of X. Let R = (Pij). Then R is a correlation matrix. The manner in which we arrived at the correlation matrix begs an apology. What happens when one of the variances O"ii'S is zero. If O"ii is zero, the entire i-th row of ~ is zero. In such an event, we could define Pij = 1 for all j. With this convention, we can proclaim that every correlation matrix arises this way. The following is a trivial corollary of P 6.3.4. COROLLARY 6.3.6. Let A be a non-negative definite matrix of order m x m and R any correlation matrix. Then for all j

Let us examine some implications of this corollary. If we take R

=

1m, we get the result that every diagonal entry of A is sandwiched between the smallest and largest eigenvalues of A. Let ~ = (O"ij) be any non-negative definite matrix. It can be regarded as a dispersion matrix of some random vector X with components Xl, X 2 ,.·· , X m .

208


Let R = (Pij) be the correlation matrix associated with E. Write O'ii = variance of Xi = aT for all i. Observe that aij = Pijaiaj for all i and j. Define A = aa', where the vector a' = (aI,'" ,am)' Note that A is of rank 1. Further, E = A . R . Corollary 6.3.6 provides bounds on the eigenvalues of E in terms of the eigenvalues of A. Since A is of rank 1, (m - 1) eigenvalues of A are all equal to zero. The other eigenvalue is m

m

i=I

i=I

~ a'f. (Why?) Corollary 6.3.6 offers the inequality 0 ::; Aj(E) ::; ~ a'f. Corollary 6.3.6 offers a good insight as to the magnitude of the eigenvalues when the correlations associated with a dispersion matrix are modified. Suppose we have a dispersion matrix E = (aij) with the associated correlation matrix R = (Pij). Suppose we reduce the correlations in R in absolute value by a systematic factor keeping the variances the same. The question is how the eigenvalues of the modified dispersion matrix are affected. Let us make this a little more concrete. Let 1 P P ... P] pIp ... P

I4J=

[

~~~:::~' P P P ... 1

where -1/( m - 1) ::; P ::; 1 is fixed. Clearly, Ro is a correlation matrix. Let Eo = E· Ro. It is clear that Eo is non-negative definite and can be regarded as a dispersion matrix of some random vector. The variances in E and Eo are identical. The correlations Pij'S associated with the dispersion matrix Eo are a constant multiple of the correlations Pij'S associated with E. The gist of Corollary 6.3.6 is that the eigenvalues of the modified dispersion matrix Eo are sandwiched between the smallest and largest eigenvalues of the dispersion matrix E. Let us now deal with some deterrninantal inequalities. The following result is very useful in this connection. P 6.3.7 Let A = (aij) be a non-negative definite matrix of order m x m and Al the submatrix of A obtained from A by deleting the first row and first column of A. Let e' = (1,0,0, ... ,0); n = 0, if IAI = 0; n = IAI/IAII if IAI =1= 0 and

A2 = A - nee'. Then the following are valid.


209

(1) The matrix A2 is non-negative definite. (2) If A is positive definite, A-I satisfies A 2A-I A2 = A 2. (In the nomenclature of a later chapter, A-I is a g-inverse of A 2 .) (3) If A is positive definite and A-I = (a ij ), then alla ll ~ 1. (4) IAI ~ alla22··· amm o

PROOF. (1) and (2). If IAI = 0, A2 = A and hence A2 is nonnegative definite. Suppose IAI i= o. Let us determine what 0 precisely is. Write

A = [all

a~

al Al

] ,

where (all, a~) is the first row of A. Observe that

Check the material on Schur complements. Consequently, 0 = all a~ All al. There is another way to identify o . Recall how the inverse of a matrix is computed using its minors. The determinant of Al is the cofactor of all. Therefore, the (1, I}-th entry all in A-I is given by all = IAII/IAI = 1/0 = e' A-Ie. Note that

A 2A- I A2

= (A - oee')A-I(A - oee') = A + 02ee' A-lee' - oee' = A + oee' - oee' - oee' =

oee' A2 .

It is clear that A2 is symmetric and hence A 2A-I A2 is non-negative definite. This means that A2 is non-negative definite. With one stroke, we are able to establish both (1) and (2). (3) Since A2 is non-negative definite, its (l,l)-th element must be non-negative, i.e., all - 0 ~ o. But 0 = I/a ll . Consequently, alla ll ~ 1. As a matter of fact, aiiaii ~ 1 for all i. (4) The inequality alla ll ~ 1 can be rewritten as IAI ~ auiAII. Let B be the submatrix of A obtained by deleting the first two rows and first two columns of A. In an analogous way, we find that we have IAII ~ a221BI . If we keep pushing this inequality to its utmost capacity, we have the inequality that IAI ~ au a22 .. . ammo This inequality goes under the name of HS determinantal inequality.

210


If A is positive definite, it is not necessary that A2 is positive definite. For a counterexample, take A = 12. The correlation matrix is an important landmark in Multivariate Analysis. The above result provides a good understanding on the makeup of a correlation matrix. COROLLARY 6.3.8. If R = (Pij) is a non-singular correlation matrix then the diagonal entries of R- 1 = (pij) satisfy the inequality pii ~ l. Further, IRI ~ 1. The case when IRI = 1 is of interest. If R = 1m , then IRI = 1. In fact, this is the only situation we have determinant equal to 1. This can be shown as follows. Let AI, A2, . . . ,Am be the eigenvalues of R. By the Arithmetic-Geometric mean inequality,

The equality holds if and only of all Ai's are equal. In our case, 1 = m

IRI = I1~1 Ai and

L

Ai = tr(R)

=

m . Thus equality holds in the

i=l

Arithmetic-Geometric mean inequality. Hence all Ai'S are equal and in fact, they are all equal to unity. Hence R = 1m. (Why?) Now we come to an interesting phase of HS multiplication. The following result involves detenninants. P 6.3.9 Let A = (aij) and B = (b ij ) be two non-negative definite matrices of the same order m x m. Then

IA· BI PROOF.

~ IAI IBI·

(6.3.2)

First, we establish the following inequality: (6.3.3)

If A is singular or one of the diagonal entries of B is zero, the inequality (6.3.3) is crystal clear. Ass1lllle that A is non-singular and none of the diagonal entries of B is zero. Let R be the correlation matrix associated with the dispersion matrix B. Now observe that

IA· BI

= bll b22 o. obmmlA RI. 0


211

(Why?) In order to establish (6.3.3), it suffices to prove that

IA·RI ~IAI

(6.3.4)

for any correlation matrix R = (Pij). Let A2 = A - aee', where a and e are as defined in P 6.3.7. Let A-I = (a ij ). Observe that

Os IA2 . RI = I(A -

aee')· RI

= IA· R - (i/all)ee'l.

The computation of the last determinant requires some tact. We need to borrow a trick or two from the theory of determinants. We note that the determinant IA . R - alII ee'l = IA· RI- allllA 1 . RII where Al is the submatrix of A obtained by deleting the first row and first column of A and RI is created analogously. From these deliberations, we obtain the inequality,

This cess. rows from

is an interesting inequality begging for a continuation of the pr~ Let B be the submatrix obtained from A by deleting the first two and first two columns. Let R2 be the correlation matrix obtained R in a similar fashion. Continuing the work, we have

Pushing the chain of inequalities to the end, we obtain IA·RI ~ IAI. This establishes (6.3.3). Now from (6.3.3), it is clear that IA . BI ~ IAI IBI· Use P 6.3.7 (4). Let us examine what HS multiplication means in certain quarters of Statistics. Suppose X(l), X(2), ... is a sequence of independent identically distributed random vectors with common mean vector 0 and dispersion matrix E. Assume that E is non-singular. Let R be the correlation matrix associated with E. Let, for each n ~ 1, y(n)

= X(l) . X(2) .•... x(n),

i.e., y(n) is the HS product of the random vectors X(1), X(2), ... ,x(n). The correlation matrix R{n) of the random vector y(n) is precisely the HS product of R with itself n times. Let us denote this product by

212


R(n) = (p~j») . If IRI = 1, the components of y(n) are clearly uncorrelatOO. If 0 < IRI < 1, the components of y(n) are nearly uncorrelatOO if n is large. (Why?) The determinantal inequality referred to in P 6.3.9, namely IR(n)1 :2: (IRl)n, is not informative. The determinant IR(n)1 is nearly equal to one if n is large, whereas the quantity (IRl)n is nearly equal to zero. One can easily improve the lower bound provided by P 6.3.9 on the determinant of the HS product of two non-negative definite matrices. This is what we do next. P 6.3.10 If A = (aij) and B = (bij) are two non-negative definite matrices of order m X m, then (6.3.5) PROOF . Note that the determinantal inequality (6.3.2) is a special case of (6.3.5). The inequality (6.3.5) leads to

from which (6.3.2) follows . As for the validity of (6.3.5), if A or B is singular, (6.3.5) is essentially the inequality (6.3.3). Assume that both A and B are non-singular. Let Q and R be the correlation matrices associated with A and B, respectively. It suffices to prove that

IQ· RI + IQI IRI :2: IQI + IRI· To prove this inequality, we can employ the trick we have used in the proof of P 6.3.9 by looking at the relationship between the minors of a matrix. Let Qi be the submatrix of Q obtained by deleting the first i rows and i columns of Q, i = 0, 1,2, ... ,m - 1. Let Ri stand for the submatrix of R likewise. Let

Our objective is to show that £1 :2: O. We shall scrutinize £1 and £2 a little closely. We need to put in some additional work before the scrutinization. Let R-l = (pij). Recall the vector e' = (1,0,0, . . . ,0)


213

we have used in the proof of P 6.3.9. By P 6.3.7, R- = R- (II pll )ee' is non-negative definite and hence Q. Ii is non-negative definite. Further, by (6.3.3),

IQI (1 - 1I p11) ~ IQ . RI = IQ . RI (1 - I~~ ~~~11 ),

(6.3.6)

from which we have (6.3.7) (The equality in (6.3.6) is fashioned after (6.3.6).) By (6.3.7),

£1 -

IQ . RI - IQ1 . RIll pll + IQI IRI -IQ11 IR111 pll + IQ111pll -IQI + IR111pll -IRI Z IQI-IQII pll + IQI IRI-IQ11 IR111 pll + IQ111pll -IQI + IR111pll -IRI = (Ilpll)(IQ11-IQI) + IQI IRI-IQ11 IRI + IRI-IRI = (II pll -IRI)(IQ11-IQI).

£21 pll =

In these deliberations, we have used the fact that pll = IRII/IRI. Observe that (Ilpll) -Ipl = (1 -lp11)lpll Z 0 as the determinant of a correlation matrix is ~ 1. Note also that IQ11 - IQI = qlllQI - IQI = (qll _ I)IQI 0, where Q-1 = (qii). Consequently, we observe that £1 - £21 pll z 0, which implies that £1 Z £2(IRI/IR11). This inequality sets a chain reaction. It now follows that £2 Z £3(IR11/IR21). Proceeding inductively, we achieve that £1 Z o. This completes the proof. Let us spend some time on HS multiplication and ranks. For any two matrices A and B, we have seen that rank(A · B) ~ [rank(A)][rank(B)J. If the matrices are non-negative definite, we can do a better job.

z

P 6.3.11

If A is positive definite and B is non-negative definite with r non-zero diagonal entries, then rank(A . B) = r. PROOF . Observe that A . B is non-negative definite and has r nonzero diagonal entries. Consequently. rank(A . B) ~ 1·. Consider the principal submatrix of A . B of order r x r whose diagonal entries are

214


precisely these non-zero numbers. By (6.3.3), the determinant o~ t~s submatrix is non-zero. Thus we now have a minor of order r whIch IS non-zero. Hence rank(A . B) 2: r. This completes the proof. From P 6.3.11, the following interesting results emerge. (1) If A is pd and B is nnd with diagonal entries nonzero, then A· B is nnd even if B is singular. (2) If p(A) = p(B) = 1, then p(A . B) = 1. (3) If A is pd and B is nd, then A· B is nd. (4) It is feasible for A . B to have full rank even if A and B are not of full rank. For instance, each of the matrices

A

~

[:

:

nand ~ B

[:

i ~1

has rank 2, but A . B has rank 3. P 6.3.12 (Fejer's Theorem) . Let A = (aij) be an n X n matrix. Then A is nnd if and only if tr(A . B) 2: for all nnd matrices B of order n x n.

°

PROOF . To establish the only if part, let A and B be both nnd and consider a vector x E en, with all its components unity. Then A . B is nnd and x"(A· B)x 2: 0, i.e., tr(A · B) 2: 0. Conversely let tr(A · B) 2: for all nnd B. Choose B = (bij) = (XiXj) for' any x E en. Then B is nnd and tr{A . B) = x" Ax 2: which implies that A is nnd. As a corollary to Schur's product theorem we have the following.

°

°

COROLLARY 6.3.13. Let A be an n X n nnd matrix. Then: (1) The matrix A . A ..... A, with any number of terms is nnd. (2) If J{z) = ao + alz + a2z2 + ... is an analytic function with non-negative coefficients and radius of convergence R > 0, then the matrix (f{aij)) is nnd if alllaiji < R.

Complements 6.3.1 Let A and B be matrices of the same order m x n and the same unit rank, i.e., A = xy' and B = uv' for some non-zero column vectors


215

x and u each of order m x 1 and non-zero column vectors y and veach of order n x 1. Show that

A· B = (x· u)(y· v)'. Show also that A . B is at most of rank 1. 6.3.2 Let A = (aij) be a square matrix of order m x m. Show that

6.3.3 (An alternative proof of Schur's Theorem P 6.3.2) . Since B is non-negative definite, write B = TT*. Let tk be the k-th column of T = (tij), k = 1,2, ... ,m. Let x' = (Xl, X2 , '" ,xm ) be any vector with complex entries. Then m

x*(A· B)x

=L

m

LaijbijXiXj

i=I j=I

m

=L

m

m

Laij(Ltikljk)XiXj

i=I j=I

k=I

m

= L(x, tk)* A(x· tk) ;::: 0,

as A is non-negative definite.

k=I

6.3.4 Let A and B be two matrices of the same order m x m. Let 1m be a column vector of order m x 1 in which each entry is equal to 1. Show that tr(AB) = l~(A· B')l m • 6.3.5 Show that every positive definite matrix is the HS product of two positive definite matrices. Explore the uniqueness of the factorization. 6.3.6 If A and B are positive definite, show that A . B = AB if and only if A and B are both diagonal matrices. Hint: Use P 6.3.7 (4) and (6.3.3). Is it necessary that A and B have to be positive definite? 6.3.7 Let A be a symmetric non-singular matrix. Show that 1 is an eigenvalue of A . A-I with corresponding eigenvector e, where e' =

(1,1, ... ,1). Hint: Observe that each row sum of A . A-I is unity.

216


6.4. Khatri-Rao Product In this section, we introduce another product of matrices known as Khatri-Rao product and examine some of its ramifications. Let A and B be two matrices of orders p x nand m x n, respectively. Let ai be the i-th column of A and f3i, be the i-th column of B, i = 1,2, ... ,n. The Khatri-Rao product A 0 B of A and B is the partitioned matrix of order pm x n given by

(6.4.1) We will establish some useful results stemming out of this type of matrix. P 6.4.1 Let A, B, C , and D be four matrices of orders p x n, m x n, m x p, and n x m, respectively. Then (C ® D)(A 0 B) = (CA) 0 (DB), PROOF. Let ai be the i-th column of A and f3i, the i-th column of B, i = 1,2, .. . ,n. Then the i-th column of CA is Cai and that of DB is Df3i. Consequently, the i-th column of (CA) 0 (DB) is Cai ® Df3i = (C® D)(ai ®f3i), which is precisely the i-th column of (C® D)(A0B) . In the next result, we rope HS multiplication into the process of Khatri-Rao product.

P 6.4.2 Let A and B be two non-negative definite matrices each of order n x n. Let A = r'r and B = 11'11 be the Gram-matrix representations of A and B, respectively, for some matrices r of order r x n and 11 of order s x n. Then the HS product of A and B is related to the Khatri-Rao product by

A · B = (r 0 11)'(r 0 11). PROOF. Let ai, a2, .. . ,an be the columns of rand f31, f32, ... , f3n be those of 11. If A = (aij) and B = (b ij ), note that aij = aiaj and bij = f3:f3j for all i and j. The (i, j)-th entry of (r 0 11)' (r 011) is given by (ai ® f3i)'(aj ® f3j) = aiaj ® f3:f3j = (aiaj)(f3ff3j) = aijb ij , which is the (i,j)-th entry of the HS product A . B. From P 6.4.2, we can draw the important conclusion that the HS product A . B is also non-negative definite. We have arrived at this conclusion in Section 6.3 by a different route.


217

P 6.4.3 Let A and B be two non-negative definite matrices of the same order n x n. Let A = rlf and B = 0'0 be the Gram-matrix representations of A and B, respectively, for some matrices r of order r X nand 0 of order s x n. If the HS product A . B is not of full rank, then there exists a non-null diagonal matrix 6. such that r 6.0' = O. PROOF. By P 6.4.2, we can write A ·B = (r00)'(r00). Since A ·B is not of full rank, there exists a non-null vector x such that (A· B)x = O. This implies that (r 0 O)x = O. (Why?) Let r = (,ij), 0 = (Wij), and x' = (x}, X2,'" , x n ). Let 6. = diag(xI' X2, ••• ,xn ). The statement that (r 0 O)x = 0 implies that

for 1 ~ i ~ 7' and 1 ~ j ~ s. This is equivalent to r 6.0' = O. This completes the proof. Let us explore the world of estimation of heteroscedastic variances in linear models as an application of the results presented in this section. Suppose Y1 , Y2,'" ,Yn(n > 2) are pairwise uncorrelated random varirespecables all with the same mean IL but with variances lT~, lT~, ... tively. The objective is to estimate each of the variances using the data

,IT;

n

-

Y1 , Y2, ... ,Yn . Let Y =

n1 L

2

Yi and S =

1 n-l

i=l

n

-2

.

L(Yi - Y) . The estli=l

mati on problem arises when one wants to ascertain precisions of some n instruments. All instruments measure the same quantitative phenomenon IL but with error characterized by the variances lT~, lTi, .. . Based on the measurements Y1 , Y2," . ,Yn made one on each instrument, the objective is to estimate the precisions lT~, lTi, ... of the instruments. Suppose we want an unbiased estimator of lT~. It is natural to seek a quadratic function of the data as an unbiased estimator of

,IT;.

,IT;

n

lT~. Any quadratic function of the data can be written as L

n

L aij Yi Yj

i=l j=l

for some constants aij's. Setting n

lT~ =

n

n

n

2

ELLaijYiYj =1L L L aij i=l j=l

i=l j=l

n

+ LaiilTT, i=l

218


we note that the coefficients aij's have to satisfy the conditions: n

i=2,3, ...

all=l,aii=O,

,nj and

n

L::L::aij=O. i=l j=l

As an example y2 - Y1Y2 is an unbiased estimator of O'r· One can jot down any numbe: of estimators of O'r. We would like to focus on one particular estimator of O'r and demonstrate that it .is optimal in a sense to be made precise later. First, we note the followmg: E(Y1

-

Yi = E((Y1 -

(Y -

JL) -

= E[( - 1 )(Y1

-

n

JL))2

~(Y2 -

JL) -

n

JL) - . .. -

~(Yn -

JL)]2

n

n

= ( - 1 )20'r + (~)2 2: O'~j n

n

i=2

1

n

ES2

n

2:

= (_l_)(n - 1)2 L::O'~ + (_l_)(n - 1)( _)2 O'~ n- 1 n i=l n- 1 n i=l 1

n

2

=-L::O'i. n i=l Now it is simple to verify that

n - 2 1 2 Tl = --(Yl - Y) - - - S

n-2

n-2

is an unbiased estimator of O'i. What is special about this estimator? One thing is that it uses all the data. One could provide other estimators of O'r which make use of all the data. The estimator Tl enjoys a certain invariance property. More precisely, if each Yi is replaced by Yi + c for some fixed constant c, the value of the statistic Tl computed using the new data Y1 + c, Y2 + C, • •• , Yn + c is the same as the value of the statistic Tl computed using the data Y1 , Y2, . . . , Yn . This invariance property is a desirable property for an estimator to possess because the parameters O'['S enjoy such a property. One could provide different invariant estimators of O'i. Let us try a different track. The estimator Tl is a quadratic form in Y1 , Y2, ... , Yn . More precisely, T1=Y'AY.

219


where Y'

= (Yt, 1'2, ... , Yn ), all

= 1, ail =

ali

ai2 aij

and A =

= (aij)

-(n -

where

1)-1, i = 2, ...

,n

= ... = ann = 0,

= (n -1)-I(n - 2)-1 for i =f j, i,j = 2, . .. ,n.

(6.4.2)

(1r

Let Q be the collection of all quadratic unbiased estimators of which are invariant under translations of the data. Any quadratic estimator is of the form Y' BY for some symmetric matrix B = (b ij ) of order n x n. The estimator Y' BY is invariant means that

(Y

+ c1)' B(Y + c1) =

y' BY

for all real numbers c and for all realizations of the data vector Y, where 1 is a column vector of order n x 1 in which every entry is equal to one. The data vector Y + c1 is simply a translation of the data vector Y. The demand that the estimator be invariant is equivalent to the condition that c2 1' B1 + 2c1' BY = 0 for all real numbers and for all realizations of the data vector Y. This is equivalent to the condition that

B1 =0.

(6.4.3)

This condition can be rephrased as the condition that every row sum of B is equal zero. In the presence of invariance condition, the estimator Y' BY is unbiased for if

(1r

bll = 1 and

bii = 0, i = 2,3, ... ,n.

(6.4.4)

(Why?) The matrix A spelled out in (6.4.2) meets all these requirements (6.4.3) and (6.4.4). The matrix A has the optimum property that among n

all matrices B

=

(b ij ) satisfying (6.4.3) and (6.4.4),

n

IIBI12 = L: L: b~j i=1 j=1

is a minimum when B = A. One can easily verify this using the Lagrangian multipliers method. There is yet another way of deriving the

220


estimator T J • The random vector Y falls into the mould of a linear model, i.e., (6.4.5) with E(E) = 0 and Disp(E) = Diag(O"r,O"~, ... ,0";), where f.' = (EJ,E2, ... , En), the unobservable error random vector associated with the da!a vector Y, and X' = (1,1, ... ,1). The least squares estimator of J.L is Y. The projection matrix F associated with the linear model is given by

= (mij) = (In

F

- X ( X , X ) -J X ')

=

1I , In - n n

where I n is the matrix of order n X n in which every entry is equal to one. The vector € of residuals is given by

€' Let

a2' =

= (FY)' = (YJ

(ar, a~,

... , a;).

-

Y, Y2 - Y, ... , Yn

-

Y).

Look at the system of linear equations (6.4.6)

in the unknown a2 , where the symbol . stands for HS multiplication. The matrix F· F is non-singular and its inverse is given by

b

...

~ ~

:::

b

...

a

(F . F) -J =

[

b

where b = [-l/{n - l)(n - 2)J and a - b = n/(n - 2). The solution to the linear equations (6.4.6) is given by a2 = (F. F)-J (€ . E). After a round of routine simplification, we note that

~2

O"i

n

(

-)2

S2

.

= --2 Yi - Y - --2' z = 1,2, ... nn-

a?

,no

For i = 1, is precisely equal to the statistic T} we have been harboring and nurturing all along! There are a couple of things to be sorted out before we can conclude this section. The first is about the significance of the linear equations (6.4.6) we have jotted down. The

221


second is the significance of the optimization problem in arriving at the estimator TI = Y' AY. Let us explain why TI is optimal. When we think about optimality of a certain estimator, we would like to phrase the optimality of the estimator in terms of variance. Declare an unbiased estimator of a parameter to be optimal if its variance is the least among all unbiased estimators of the same parameter. If the variance were to be the criterion of optimality, we need to assume some structure on the fourth moments of the random variables Y 1 , Y2 , ••• ,Yn . If we do not want to assume beyond what we imposed on the data, namely pairwise uncorrelatedness and finite second moments, variance criterion is beyond our reach. We need to seek other optimality criteria intuitively justifiable and acceptable. Suppose J1 is known. A reasonable estimator of (7~ is (Y1 - J1)2. If J1 is known, the residuals EI, E2, ... ,En in the linear model (6.4.5) are observable. In terms of the residuals, the reasonable estimator can be rewritten as E'GE, where G = diag(l, 0, 0, ... ,0). Can we do better than this? Is it possible to find an invariant unbiased estimator Y' BY of (7~ which is close to the reasonable estimator? The conditions on the estimator, especially the invariance property, imply that Y' BY = (Y - J11)' B(Y - J11) = E' BE. The problem is to determine the matrix B such that E'GE - E' BE is as small as possible. This is tantamount to choosing the matrix B with all the constraints so that IIG - BII is minimum. This is equivalent to minimizing IIBII subject to the constraints of invariance and unbiasedness. This is the story behind the estimator TI = Y' AY. One can also justify the estimator TI on the ground that the variation exhibited by the invariant unbiased estimator Y' BY be reduced as much as possible. We can achieve this by choosing B as small as possible. This is the s~called Minimum Norm Quadratic Unbiased Estimation, with the acronym MINQUE principle of, C.R. Rao (1972a). Now we come to the significance of the linear equations (6.4.6). Let

be a general linear model with Disp(E)

= Diag«(7i,(7~, ...

,(7~),

222


where the matrix X of order n x m is known and the parameter vector {3 and variances are unknown. We need good estimators of the variances based on the data vector Y. Assume that the variances are all distinct and the rank of X is m . (These assumptions can be relaxed.) When we say that the variances are all distinct we mean that the vector «(j2)' = «(jr, (j~ , . .. ,(j;) of variances has the parameter space (0, (0) x (0, (0) x .. . x (0, (0). Let P1(jr + P2(j~ + ... + Pn(j~ = p'(j2 be a linear function of the variances, where the vector P' = (P1, P2, ... ,Pn) is known. As per the MINQUE principle, we seek a quadratic estimator Y' BY of P' (j2 such that B = (b ij ) satisfies the conditions

(jr

BX=O, n

L bii(j; = L Pi(j;, i=1 and

(6.4.7)

n

(6.4.8)

i=1 n

n

IIBII2 = LLb~j i=1 j=1 is a minimum. The condition (6.4.7) implies that the estimator Y' BY is invariant, i.e.,

Y' BY

= (Y - X{3o)' B(Y - X{3o)

for all vectors {3o, and condition (6.4.8) implies that the estimator Y' BY is unbiased for P' (j2 . A solution to this problem has been discussed in C.R. Rao (1972). Let F = (In - X(X' X)-1 X') be the projection matrix and £ = (In - X(X' X)-1 X')Y be the vector of residuals. Let (&2)' = (&r, &~, . . . ,&~). Consider the system (F . F) &2 -= €. € of linear equations in the unknown vector &2. If F · F is non-singular, the MINQ UE of (j2 is given by &2 = (F · F) -1 (£ . €). This is the story behind (6.4.6). The next line of inquiry is to understand when the HS product F · F is non-singular. Hartley, Rao, and Kiefer (1969) and Rao (1972a) throw some light on this problem. Complements 6.4.1 Let Y1, Y2, . .. ,Yn be n pairwise uncorrelated random variables with the same mean J-L. Let kll k 2 , ... ,kr be positive integers such that

223


kl

+ k2 + ... + kr = n. Suppose Var(Yi) = Var(Y2) = ... = Var(Yk1 ) = ur, Var(Yk1+I) = Var(Yk1+2) = ... = Var(Ykl+k2) = u~, Var(Ykl+ ... +kr_1+d

= Var(Ykl+ ... +kr_l+2) = ... = Var(Yn ) = u;.

The mean and variances are all unknown. Develop MINQUE's of the variances. 6.4.2 Let }ij, i = 1,2, ... ,p and j = 1,2, ... ,q be pairwise uncorrelated random variables with the following structure. E(}ij) = Var(}ij) =

ai

+ f3j

for all i and j,

u;, j = 1,2, ... ,q

and for all i.

The variances, ai's and f3/s are all unknown. Show that the residuals are given by fij =}ij -

where Yi . =

iii. -

)q

-E

q j=1

Yj

+ Y., i = 1,2, ...

-)p

}ij, Yj

=- E

P i=1

,p -

}ij, and Y . =

the MINQUE of u~ is given by

and

j

= 1,2, ...

IPq

- E E

,q,

}ij. Show that

pq i=1 j=1

t.

a(t. .) = x' Ax + 2>.'(Bx - p), x E R m , >. E RT. The stationary values of the function L are obtained by setting separately the vector derivatives of L with respect to x and>' equal to zero. Using P 6.5.2, we have

aL I aL ax = 2Ax + 2B >. = 0, a>. = 2(Bx - p) =

o.

228


These equations which are linear in x and A can be rewritten as

Solving these equations is another story. If rank(B) = r and A is positive definite, then the system of equations admits a unique solution. From the equations, the following series of computations follow:

x

=

_A- 1 B' A, Bx

= -BA- I B' A = p;

A = -(BA- I B')-Ip,

X

= A-I B'(BA- I B')-Ip.

This type of optimization problem arises in Linear Models. Suppose y is a random vector of m components whose distribution could be anyone of the distributions indexed by a finite dimensional parameter () E Rr. Suppose under each () E Rr, Y has the same dispersion matrix A but the expected value is given by Eo Y = B'() for some known matrix B of order r x m . (The expected value of each component of Y is a known linear combination of the components of ().) One of the important problems in Linear Models is to estimate a linear function p'() of the parameter vector unbiasedly with minimum variance, where the vector p of order r x 1 is known. In order to make the estimation problem simple, we seek only linear functions of the data Y which are unbiased estimators of p' () and in this collection of estimators we search for one with minimum variance. One can show that a linear function x'Y of the data Y is unbiased for p'() if Bx = p. For such x, the variance of x'Y is x' Ax. Now the objective is to minimize x' Ax over all x but subject to the condition Bx = p. If B is of rank r and A is of full rank, then the linear unbiased estimator of p'() with minimum variance (Best Linear Unbiased Estimator with the acronym BLUE) is given by

Let us look at another optimization problem. Let A be a symmetric and B a positive definite matrix with real entries. Let f(x) = x' Ax and g(x) = x' Bx, x E R:" . We would like to determine the stationary values of the function f(x)/g(x), x E R:" - {O} . We equate the vector


229

derivative of this ratio with respect to x to zero. Using P 6.5.2, we have 8(1/g) = _2_Ax _ 2x'Ax Bx = O.

8x

x' Bx

(x' Bx)2

This equation leads to the equation

(A - AB)x = 0, where A = x' Ax/x' Bx. Thus the stationary value x in Rrn - {O} of the ratio of quadratic forms has to satisfy the equation (A - AB)x = 0 for some A. (But A will be automatically equal to x' Ax/x' Bx. Why?) A non-zero solution to the equation exists if the determinant IA - ABI = o. This determinantal equations has exactly m roots. Thus the stationary values of the ratio of the quadratic forms of interest are at most m in number. We now focus on matrix derivatives. The function I is a real valued function of a matrix variable X of order m x m. The domain of definition of I need not be the space of all matrices. For the determinant function, we will consider the collection Mrn(ns) of all non-singular matrices of order m x m with real entries. This set is an open subset of the collection of all matrices of order m x m in its usual topology. The set {X E Mrn(ns) : IXI > O} is also an open set and we will consider functions having this set as their domain. Differentiability of the determinant function IXI of X in its domain should pose no problems. P 6.5.3 (1) If I(X) = lXI, X E Mrn(ns), then 81/8X = IXI(X-I),. (2) If I(X) = log lXI, IXI > 0, then = (X-I),. (3) If I(X) = IXlr, IXI > 0 fixed, then 81/8X = rIXn(X- 1 ),.

U

(1) We use (6.5.3). Let X = (Xij) E Mrn(ns). Let Y = (Yij) be any arbitrary matrix of order m x m. For small values of t, X +tY will be non-singular. Let us embark on finding the determinant of X + tY. Let Xc = (xij) be the matrix of cofactors of X. After expanding IX + tYI and omitting terms of the order t 2 , we have PROOF.

m

IX + tYI = IXI + t L

m

L YijX ij = IXI + t trey' XC).

i=l j=l

Consequently,

230


lim !X + tY!-!X! t-O

t

. I.e.,

= tr(Y' XC) = tr(Y' af ) ax af = Xc = !X!(X-1), . ax

This completes the proof. The proofs of (2) and (3) are now trivial. The case of symmetric matrices requires some caution. The space Mm(s) of all symmetric matrices of order m x m is no longer an m 2 _ dimensional vector space. In fact, it is an m(m + 1)/2-dimensional vector space. Now we consider the subset Mm(s,ns) of all non-singular matrices in Mm(s). [The letters ns stand for nonsingular.j This subset is an open set in Mm (s) in its usual topology. The determinant function on this subset is under focus. As a simple example, consider the case of m = 2. Any matrix X in Mm(s, ns) is of the form, X =

with the determinant

XUX22 -

[xu X12 Xi2

X12] , X22

i= O.

Observe that

= 2 [X22 -X12] _ [X22 0 ] = !X![2X-l _ diag(X-l )1. -X12 Xu 0 Xu

This formula holds in general too. Before we jot it down let us discuss the problem of taking derivatives of functions whose domain of definition is the set of all symmetric matrices. Let f be a scalar valued function of a matrix variable X. It is clear that af lax' = (af lax)'. Let f be a scalar valued function of a matrix variable X, where X is symmetric. What we need is a formula analogous to (6.5.3) operational in the case of a symmetric argument. We do have a direct formula which in conjunction with (6.5.3) can be used to solve the symmetric problem. The formula for X symmetric is

af _ {af(Y) af(Y) . (af(Y))} ax - -ay- + ay' - dlag -ay- !y=x.

(6.5.4)

231


In working out the derivative 8f(Y)/8Y, the function f(-} is pretended to have been defined on the class of all matrices Y, i.e., all the components of Yare regarded as independent variables, and then the derivative formed. Let us illustrate the mechanics of this formula with a simple example. Let f(X) = lXI, where X is of order 2 x 2, IXI =1= 0, and X symmetric. Regard f( ·) as a function of Y = (Yij), where Y is of order 2 x 2 and IYI =1= O. More precisely, f(Y) = IYI = YllY22 - Y12Y21· Note that

8f(Y) _ 8Y

[~~~~

Y22

81YI

[

8Y21

8f(Y) 8Y' =

.

diag

(8f (Y)) _ 8Y -

8Y21

81YI

8 1YI

8Y12

8Y22

[~ 0

-Y1l

-Y21

0]

8Yl1

Yll

-Y12

[~ ~] [ Y22 8Yl1

-Y21] ,

81YI

[Y22

o

8Y22

Yll

1

Yll

and for X symmetric,

8f =

8X

P 6.5.4

{8f(Y) 8Y

+ 8f(~) 8Y

_

diag

(8f(Y))}1 _ 8Y

Y-X

(1) If f(X) = lXI, X E Mm(s, ns), then

:i

= IXI[2X- 1

-

diag(X- 1 )].

232

MATRIX A LGEBRA THEORY AND APPLICATIONS

(2) If f(X) = log lXI , X E Mm(s , ns), IXI > 0, then

af = [2X-1 - diag(X- 1)]. ax (3) If f(X) = IXlr, X E Mm(s, ns) , IXI > 0, then

:~ = r1Xn2X- 1 -

diag(X- 1)].

We will now outline some useful formulas on matrix derivatives. Let U and V be two matrix functions of a matrix variable X, where U = (Uij) and V = (Vij) ar.e of orders p x q and X is of order m x n. Applying P 6.5.1 (1) to each term Uij(X)Vji(X), we deduce

a

ax tr(U(X)V(X))

a

a

+ ax tr(U(Y)V(X))IY=x.

= ax tr(U(X)V(Y))IY=x

(6.5.5)

Instead of the trace function dealt in (6.5.5), we could deal with any scalar valued function f of U(X) and V(X). Accordingly, we have

a:f(U(X), V(X)) =

[a~f(U, V)] [a:U(X)] +

[a~ f(U, V) 1[a: V(X)].

(6 .5.6)

Using (6.5.5) or (6.5.6), one can establish the validity of the following proposition. P 6.5.5 (1) Let U(X) be a matrix valued function of a matrix variable X, where U(X) is of order p x p, non-singular, and X is of order m x n. Then

a

ax tr(U-1(X))

a

= - ax tr(U- 2 (Y)U(X))ly=x .

(2) Let A be a constant matrix of order p x p and U(X) a matrix valued function of a matrix argument X, where U(X) is of order p x p, non-singular, and X is of order m x n . Then

a

ax tr(U-1(X)A)

a

= - ax tr(U - 1(Y)AU- 1 (Y)U(X))ly=x.


(3) Let A and B be constant matrices each of order m f(X) = tr(AX-I B), X E Mm(ns). Then

:~

233 X

m and

= -(X-1BAX- 1 )'.

(4) Let U(X) be a matrix valued function of a matrix variable X, where U(X) is of order p x p, non-singular, and X is of order m x n. Then

a

ax IU(X)I

a

= IU(X)I ax tr(U-I(Y)U(X))ly=x.

(5) Let A be a constant matrix of order m x m and f(X) = IAXI, X is of order m X m and AX non-singular. Then

:~

= IAXI

a~ tr«Ay)-l AX)ly=x

= IAXI«AX)-l A)'.

At the beginning of this section, we toyed with another idea of writing the matrix of partial derivatives. More precisely, let F(X) be a matrix valued function of a matrix variable X. We defined

*aF avec(F) ax = a(vecX)'· Even though the entries of *aFjaX are simply a rearrangement of the entries of aF/aX, it is useful to compile *aFjaX for some standard functions F of X. This is what we do in the following proposition. All these results can be derived from first principles. P 6.5.6 (1) Let F(X) = AX, where A is a constant matrix of order p x m and X of order m x n. Then

(2) Let F(X) = X B, where B is a constant matrix of order n x q and X of order m x n. Then

*aF ax = B' ®Im.

234


(3) Let F(X) = AXB, where A and B are constant matrices of orders p x m and n x q, respectively, and X of order m x n. Then

*oF oX =B'®A. (4) Let F(X) = AX' B, where A and B are constant matrices of orders p x nand m x q, respectively, and X of order m x n. Then

*oF 8X

= (A ® B')P,

where P is the permutation matrix which transforms the vector vec(X) into vec(X'), i.e., vec(X') = Pvec(X). (5) Let U(X) and VeX) be matrix valued functions of a matrix variable X, where U(X) is of order p x q, VeX) of order q x r, and X of order m x n. Then

;~U(X)V(X) = (V(X) ® Ir)' ;~ U(X) + (I ® U(X)) ;~ VeX). (6) Let F(X) = X' AX, where A is a constant matrix of order m x m and X of order m x n. Then

*oF oX = ('A' X ® In ) P

+ ( In ® X , A ) .

(7) Let F(X) = AX- l B, where A and B are constant matrices of orders p x m and m x q, respectively, X of order m x m and non-singular. Then *oF oX = _(X- l B)' ® (AX-I) . (8) Let U(X) and Z(X) be two matrix valued functions of a matrix variable X, where U(·) is of order p x q, Z( ·) of order 1 x 1 and X of order m x n . Let f(X) = Z(X)U(X). Then

*8f 8X

= vec(U(X)) *8Z(X) + Z(X) *8U(X) 8X

oX

235


(9) Let U(X) be a matrix valued function of a matrix variable X, where U(·) is of order p x p and non-singular, and X is of order m x n. Let J(X) = [U(X)]-l. Then

*8J = «U-I(X))' ® U-I(X)) *8U(X) . 8X

8X

(10) Let Y(X) be a matrix valued function of a matrix variable X, where Y (.) is of order p x q and X of order m x n. Let Z (V) be a matrix valued function of a matrix variable V, where Z(·) is of order r x sand V of order p x q. Let J(X) = Z(Y(X)), X E Mm,n. Then

*8J = (*8Z(V) 8X 8V

I

) (*8Y(X)).

v=y(X)

8X

(11) Let Z(X} and Y(X} be two matrix valued functions of a matrix variable X, where Z(X} and Y(X} are of the same order p x q and X of order m x n. Let J(X} = Z(X} . Y(X), X E Mm,n, where the symbol . denotes HS multiplication. Then

where for any matrix Z = (Zij) of order p x q, D(Z} =diag(Zn,ZI2, . .. ,Zlq,Z2J,Z22, ••• ,Z2q, ••• ,Zpl,Zp2, • •• ,Zpq). (12) Let Z(X} be a matrix valued function of a matrix variable X, where Z(X} is of order p x q and X of order m x n. Let B be a constant matrix of order p x q and J(X} = Z(X} . B, X E Smn . Then

*8J = D(B) *8Z(X}. 8X

8X

As has been indicated earlier, the matrix derivative defined as * 8 J/ 8X is very useful in evaluating the Jacobian of a transformation. Suppose J(X} is a matrix valued function of a matrix variable X, where both X and J(X) are of the same order m x n. The Jacobian J of the transformation JO is given by J

=

*8J I 8X +' 1

236


where the suffix + inclicates the positive value of the deterrillnant of the matrix *8f18X of order mn X mn. Suppose f(X) = AXB , where A and B are constant non-singular matrices of orders m X m and n X n, respectively, and X E M m •n . The Jacobian of the transformation fO is given by

Complements

6.5.1 Let F(X) = X be the identity transformation of the matrix variable X of order m X n . Show that 8FI8X = (vec{Im)) (8) (vec(In))' . 6.5.2 Let F(X) = X be the identity transformation of the matrix variable X of order m X n. Show that *8F18X = I. 6.5.3 Let F be a matrix valued function of order p X q of a matrix variable X = (Xij) of order m X n. Show that

*8F 8X

=L m

L n

.=1 3=1

(

8F ) ( )' vec 8Xi . VeCEij , 3

where Eij is a matrix of order m X n whose (i,j)-th entry is unity and the rest of its entries zeros. 6.5.4 Let A be a constant matrix of order m X nand f(X) = tr(AX), X E M m •n , the vector space of all matrices X of order n X m. Show that (a/ / ax) A'. If m n and the domain of definition of / is the collection M(s) of all m X m symmetric matrices, show that 8/18X = 2A' - diag(A). 6.5.5 Let f(X) = tr(X2), X E M m , the space of all m x m matrices. Show that (8f 18X) = 2X'. If the domain of definition of / is the collection of all symmetric matrices, how does the matrix of partial derivatives change? 6.5.6 Let A and B be two constant matrices of orders m X m and n X n, respectively. Let f(X) = tr(X' AX B), X E Mm .n, the space of all m X n matrices, Show that (8/18X) = AX B + A' X B'. If m = n and the domain of definition of f is the space of all m X m matrices , show that

=

!{

= AXB

=

+ A'XB' + BXA +B'XA' -cliag(AXB + A'XB').

Opemtions on Matrices

237

Let A and B be two constant matrices of the same order m x m and I(X) = tr(X AX B), X E Sm. Show that

6.5.7

: { = A'X'B'

If the domain of definition of show that : { = A'XB'

I

+ B'X'A'.

is the space of all symmetric matrices,

+ B'XA' + AXB + BXA -diag(A'XB' + B'XA').

6.5.8 Let A be a constant matrix of order mxm and f(X) = tr(X' AX), X E Mm,n' Show that (af laX) = (A + A')X. If m = n and the domain of definition of f is the collection of all symmetric matrices, show that

:{ = (A + A')X + X(A + A') -

diag((A

+ A')X).

6.5.9 Let I{X) = tr(Xn), X E M m , n ~ 1. Show that (af laX) = nxn-I. If the domain of definition of I is the space of all symmetric matrices, show that

af ax

= 2n X n- I _ nd'lag (xn-I) .

6.5.10 Let x and y be two fixed column vectors of orders m x 1 and n x 1, and I{X) = x' Xy, X E Mm,n' Show that (al laX) = xy'. If m = n and the domain of definition of f is the set of all symmetric matrices, show that (af laX) = xy' + yx'. 6.5.11 Let A be a constant matrix of order m x m and f(X) = tr(AX-1),X E Mm(ns), the set of all non-singular matrices of order m x m. Show that al = _(X-I AX-I)'.

ax

If the domain of definition of f is confined to the collection of all nonsingular symmetric matrices, show that

238


6.5.12

Let I(X} = IX x'l, X E Mm,n and rank(X} = m . Show that

:~ = 2IXX'I(XX'}-IX. 6.5.13 Let a and b be two constant colunm vectors of orders m X 1 and n xl, respectively. Determine the matrix derivative of each of the scalar valued functions h(X} = a'Xb, and heX) = a'XX'a, X E Mm,n, the collection of all matrices of order m x n, with respect to X. 6.5.14 Let a be a constant colunm vector of order m x 1 and I(X) = a' X-I a, X E Mm (ns), the collection of all m x m non-singular matrices of order m x m. Determine the matrix derivative of the scalar valued function I with respect to X. 6.5.15 Let p be any positive integer and I(X} = XP, X E Mm. Show that *81 P 8X = :L)X')P-j ® Xj-I.

j=I 6.5.16 Find the Jacobian of each of the following transformations, where A and B are constant matrices of order m x m, and X E Mm. (1) I(X) = AX-I B, X non-singular. (2) I(X} = X AX'. (3) I(X) = X' AX. (4) I(X} = XAX, X E Mm.

(5) I(X) = X' AX'. Notes: The following papers and books have been consulted for developing the material in this chapter. Hartley, Rao, and Kiefer (1969), Rao and Mitra (1971b), Rao (1973c), Srivastava and Khatri (1979), Rao and Kleffe (1980), Graham (1981), Barnett (1990), Rao (1985a), Rao and Kleffe (1988), Magnus and Neudecker (1991), Liu (1995), among others.

CHAPTER 7 PROJECTORS AND IDEMPOTENT OPERATORS

The notion of an orthogonal projection has been introduced in Section 2.2 in the context of inner product spaces. Look up Definition 2.2.11 and the ensuing discussion. In this chapter, we will introduce projectors in the general context of vector spaces. Under a particular mixture of circumstances, an orthogonal projection is seen to be a special kind of projector. We round up the chapter with some examples and complements. 7.1. Projectors DEFINITION 7.1.1. Let a vector space V be the direct sum of two

subspaces VI and V2, VI nV 2 = {O}, Le., V = VI E9 V 2. (See P 1.5.5 and the discussion preceding P 1.5.7.) Then any vector x in V has a unique decomposition x = Xl + X2 with Xl E V I and X2 E V 2. The transformation X - Xl is called the projection of X on VI along V2. The operator or map P defined on the vector space V by Px = Xl is called a projector from the vector space V to the subspace VI along the subspace V 2. The first thing we would like to point out is that the map P is welldefined. Further, the map P is an onto map from V to VI. It is also transparent that the projector P restricted to the subspace V I is the identity transformation from VI to VI, Le., Px = X if X E VI' If V is an inner product space and X E VI, Y E V 2 implies that X and yare orthogonal, i.e., V 2 is the orthogonal complement of VI, the map P is precisely the orthogonal projection from the space V to the space VIas enunciated in Definition 2.2.11. Suppose VI is a subspace of V. There could be any number of subspaces V 2 of V such that V I E9 V 2 = V. Each such subspace V 2 gives a projector P from V onto VI along V 2. 239

240


No two such projectors are the same! In the following proposition, we record a simple property of projectors. P 7.1.2

A projector P is a linear transformation.

PROOF. Let x = Xl + X2 and Y = YI + Y2 be the unique decompositions of two vectors x and Y in V, respectively, with respect to the subspaces VI and V2 of V. The decomposition of the vector x + y works out to be x + Y = {Xl + yd + (X2 + Y2)

with {Xl

+ yd

E VI and (X2

+ Y2)

E V2. By definition,

P{X+Y) =XI +YI = Px+Py. This shows that the map P is additive. For a scalar a, ax = aXI + aX2 with aXI E VI and aX2 E V2. Consequently, P{ax) = aXI = a{Px). Hence P is linear. The definition of a projector involves two subspaces with only zero vector common between them. In the following proposition, we characterize projectors abstractly without alluding to the underlying subspaces. P 7.1.3 A linear transformation P from a vector space V into itself is a projector from V onto some subspace of V along some complementary subspace of V if and only if it is idempotent, i.e., p 2 = pPROOF. Let P be a projector from V onto a subspace V I along a subspace V 2 of V. Let X = Xl + X2 be the unique decomposition of any vector X in V with Xl in VI and X2 in V2. The unique decomposition of Xl in V is given by Xl = Xl +0 with Xl E V I and 0 E V 2. Consequently,

p2X = P{Px) = PXI = Xl = Px . Hence it follows that P is idempotent. Conversely, let p2 = P and

= {x E V V 2 = {X E V

VI

= x} : Px = O}.

: Px

Since P is a linear transformation, V I and V 2 are subspaces of V . FUrther, VI nV2 = {O}. For any given X E V, write X = Px + (I _

Projectors and Idempotent Opemtors

241

P)x, where I is the identity transformation from V to V. Since P is idempotent, Px = p2x = 0, from which we have (I - P}x E V 2. Thus x = Xl + X2, where Xl = Px and X2 = (I - P}x E V2, is the unique decomposition of X with respect to the subspaces V I and V 2. Hence P is a projector from V onto V I along V 2. In view of P 7.1.3, we can safely omit mentioning the subspaces that define a projector. Once we recognize the projector as an idempotent operator, the associated subspaces VI and V2 of the projector can be recovered via the formulas presented in the proof of P 7.1.3. These subspaces are explicitly identified in P 7.1.4 below. In order to show that a particular linear map is a projector, in many cases, it is easier to show that it is an idempotent operator. We now jot down several equivalent characterizations of projectors. Let P be a linear transformation from a vector space V to V. Let R(P) and K(P} be the range and kernel of the transformation P, respectively. See Section 3.1.

P 7.1.4 The following statements are equivalent. (I) The map P is a projector. (2) The map (I - P) is a projector. (3) The range R(P) of P is given by R(P} (4) R(P) = K(I - P). (5) R(I - P} = K(P).

= {x

E V:

Px = x}.

(6) R(P) nR(I - P} = {O}. (7) K(P} K(I - P} = {O}.

n

Proving the equivalence of these statements is left to the reader. In view of P 7.1.4, if P is a projector we can say that P is a projector from V onto R(P} along R(I - P}. In the following we look at sums of projectors. P 7.1.5 Let PI, P2, ... ,Pk be projectors such that PiPj = 0 for all i =1= j. Then:

(1) P = PI + P2 + ... + Pk is a projector. (2) R(Pi) nR(Pj} = {O} for all i =1= j and R(P)

= R(Pt}61R(P2} $

... $ R(Pk }.

(I) It is easy to establish that P is idempotent. (2) Let i =1= j and z E R(Pi} nR(Pj). Then z = PiX = PjY for some vectors X and yin V. Observe that PiX = plx = Pi(Pi X} = Pi(PjY} = PROOF.

242


~PjY = o. Hence z = O. This proves that R(~) nR(Fj) = {O}. By the very definition of the projector P, R(P) C R(Pt}EB .. . Ej1 R(Pk). On the other hand, note that R(~) C R(P) for each i. For , if x E R(Pi ), then x = PiY for some y in V and P PiY = ply = ~y = x , from which it follows that x E R(P). Consequently, R(Pt) EBR(P2) EB.· .EBR(Pk) C R( P) . This completes the proof. The following result is complimentary to P 7.1.5. A given projector can be written as a sum of projectors under the right mixture of circumstances. P 7.1.6 Let P be a projector defined on a vector space V onto a subspace VI along a subspace V 2 of V. Suppose the subspace VI is a direct sum of subspaces, Le., VI = V ll EB V 12 EB ... EB VIr for some subspaces Vlj's of V. Then there exist unique projectors ~ from V onto V Ii along an appropriate subspace of V such that P = PI + P2 + ... + Pr and PiPj = 0 for all i =1= j. One can always bring into existence a projector as long as we have two subspaces whose direct sum is the underlying vector space. In order to identify Pi we need two subspaces. We already have one, namely, VIi. In order to avoid an appropriate subspace complementary to the subspace VIi, let us define the map ~ directly. Let x E V. We can write x = Xu + X12 + . .. + Xlr + y with Xli E VI i and y E V 2 . Define PiX = Xli. The map Pi is obviously a linear transformation and idempotent. Consequently, Pi is a projector. (Can you identify the subspace V 2i such that F1 is a projector from V onto VIi along V 2i?) It is clear that P = PI + P 2 + ... + P r and PiPj = 0 for all i =1= j. To prove uniqueness, let P = Ql + Q2 + .. . + Qr be an alternative representation of P as a sum of projectors. Then for any X in V, 0 = Px - Px = (PI - Ql)X + (P2 - Q2)X + ... + (Pr - Qr)x. This implies that (Pi - Qi)X = 0 for each i in view of the fact that (Pi - Qi)X E VIi. If (Pi - Qi)X = 0 for every x, then Pi = Qi. This proves P 7.1.6. We now look at a familiar problem that crops up in Statistics. Suppose YI, Y2, ... , Ym are m real random variables whose joint distribution depends on a vector parameter ()' = ((}l, (}2, ... ,(}k) E R k, with m > k. Suppose PROOF .

Eo"Yi = Xil(}l + Xi2(}2 + ... + Xik(}k, i = 1, 2, . .. ,m , where Xij'S are known. In the language of linear models, the random

Projectors and Idempotent Operators

243

variables Y1 , Y2, ... , Ym constitute a linear model. These models lie at the heart of multiple regression analysis and design of experiments pro~ lems. Let V be the collection of all linear functions of Yi, Y2, ... , Ym . It is clear that V is a real space of dimension m. As a matter of fact, we can identify the vector space V with R m in the obvious manner. Let V be the collection of all linear unbiased estimators of zero. A linear unbiased estimator of zero is any linear function i I YI +i2 Y2 +... +im Ym of YI , Y2, ... ,Ym such that E9(i IY I + i 2Y 2 + ... + imYm) = 0 for all () in R k. Such linear functions constitute a subspace V I of V. The space VI can be identified explicitly. Let X = (Xij). The matrix X of order m x k is called the design matrix of the linear model. One can check that VI = {i' = (i},i 2, ... ,im ) E R m : i'x = O}. Every vector i' in V I is orthogonal to every colUlnn vector of the matrix X. Let V 2 be the collection of all vectors in Rm each of which is a linear combination of the columns of X. The space V 2 is clearly a subspace of V. Further, V = VI a1 V2. As a matter of fact, each vector in VI and each vector in V 2 are orthogonal. The next target is to identify explicitly the projector from the vector space V onto VI along V2. To simplify the argument, assume that the rank of the matrix X is k. This ensures the matrix X' X to be non-singular. Let

A

= X(X'X)-IX'.

It is clear that the matrix A is of order m x m. One can also check that it is symmetric and idempotent, Le., A2 = A. Let i' be any vector in V = Rm. Observe that i = (Irn - A)i + Ai, where 1m is the identity matrix of order m x m. We claim that the vector ((Im - A)i)' = i'(Im - A) belongs to VI. For i'(Im - A)X = i'(X - X) = O. FUrther, it is clear that Ai = X(X'X)-IX'i is a linear combination of the colmnns of X. Thus we have demonstrated practically how the vector space V is the direct sum of the subspaces VI and V 2 . Let P be the projector from V onto V I along V 2. The explicit formula for the computation of P is given by (7.1.1) Pi = (Im - X(X'X)-IX')i. IT V = Rm is equipped with its usual inner product, the projector P is indeed an orthogonal projection.

244


There is one benefit that accrues from the explicit formula (7.1.1) of the projector P. Suppose Y l , Y2, . .. ,Ym are pairwise uncorrelated with common variance (1"2 > o. Let i l Y l + i2 Y2 + ... + im Ym = i'Y be a linear function of Y l , Y2,.·· ,Ym , where i' = (fl, i2, ... ,im ) and Y' = (YI , Y2, . .. ,Ym ). Let i = i(l) + i(2) be the decomposition of l with respect to the subspaces VIand V 2. One can verify that under each () E Rk,

i.e., l(l) Y and i(2) Yare uncorrelated. The celebrated Gauss-Markov theorem unfolds in a simple way in this environment. Let fO be a linear parametric function, i.e., f(()) = PI()l +P2()2 + ... +Pk()k for some known numbers PI, P2, ... ,Pk, () E Rk. We now seek the best linear unbiased estimator (BLUE) of fO. The estimator should be of the form l I Y I + i2 Y2 + ... + im Ym, unbiased, and has minimum variance among all linear unbiased estimators of fO. To begin with, cook up any linear unbiased estimator i l Yl + l2 Y2 + ... + lmYm = i'Y of fO. Obtain the decomposition i = l(l) + i(2), with respect to the subs paces VI and V 2. Then i(2)Y is the desired BLUE of fO. To see this, let s'Y be any linear unbiased estimator of f( ·). Write s'y = (s - l(2) )'Y + l(2) Y . Note that s - l(2) E VI. (How?) Consequently, for each () E R k, Varianceo(s'Y)

= Varianceo((s -

=> Varianceo(s'Y)

i(2»)'Y)

+ Varianceo(i(2) Y),

~ Variance(i(2) Y).

Complements 7.1.1 If P is a projector defined on a vector space V onto a subspace VI of V along a subspace V2 of V, identify the subspaces V h and V 2• such that the operator 1- P is a projector from V onto V h along V 2•. 7.1.2 Let V = R2, VI = {(Xl,O) E R2 : Xl real}, and V 2 = {(Xl, X2) E R2 : Xl + X2 = O}. Identify the projector PI from V onto VI along V 2. Under the usual inner product on the real vector space V, is PI an orthogonal projector? Explain fully.


245

7.1.3 Let V = R2, VI = {(Xl, O) E R2 : Xl real}, and V 2 = {(Xl,X2) E R2 : 2XI + X2 = O}. Identify the projector P 2 from V onto VI along V2. 7.1.4 Let P = PI +P2, where PI is the projector identified in Complement 7.1.2 and P2 the projector in Complement 7.1.3. Is P a projector? Explain fully. 7.1.5 Let Pt, P2, ... ,Pk be projectors defined on a vector space V such that PiPj = 0 for all i =I j. Identify the subspaces V I and V 2 such that P = PI + P2 + ... + Pk is projector from V onto VI along V2. 7.1.6 Let F = {O, I} be the two-element field and V = F2, a vector space over the field F. Let V I = {(O, O), (1, O)} and V 2 = {(O, O), (0, I)}. Show that V = VI EB V 2. Let PI be the projector from V onto VI along V 2 • Show that PI + PI = 0, the map which maps every element of V to the vector (O,O). Let P2 be the projector from V onto V along the subspace {(O,O)}. Show that PI + P2 is a projector but PI P2 =I O. Comment on P 7.1.5 in this connection. 7.1.7 Let V be a vector space over a field F. Suppose that the field F has the property that 1 + 1 =I 0 in F. Let PI and P2 be two projectors defined on V. Show that P = PI + P2 is a projector if and only if PI P2 = P2P I = O. If P is a projector, show that P is a projector from V onto R(PI } EB R(P2} along K(Pt} K(P2}. Hint: First, show that P I P2+P2PI = 0 and then PI P2P I +PIP2PI =

n

O. 7.1.8 Let PI and P2 be two projectors. Show that P I -P2 is a projector if and only if PI P2 = P2P I = P2, in which case PI - P 2 is a projector from V onto R(PI } K(P2} along K(PI }EBR(P2 }. The condition on the underlying field F stipulated in Complement 7.1.7 is still operational. 7.1.9 Let PI and P2 be two projectors such that PI P2 = P2PI. Show that P = P I P2 is a projector. Identify the subspaces VI and V2 so that P is a projector from V onto VI along V 2.

n

7.2. Invariance and Reducibility In this section we explore the world of invariant subspaces. We begin with some basic definitions and notions. Let V represent a generic symbol for a vector space in the following deliberations. DEFINITION 7.2.1. A subspace W of V is said to be invariant under a linear transformation T from V if Tx E W whenever X E W.

246


In other words, what this definition indicates is that if the map T is restricted to the space W, then it is a linear map from W to W. The notion of invariance can be extended to cover two subspaces as in the following definition. DEFINITION 7.2.2. A linear transformation T from V to V is said to be reduced by a pair of subspaces VI and V 2 if VI and V2 are both invariant under T and V = V I EI1 V 2. It will be instructive to examine the notion of invariance in the realm of projectors. Suppose P is a projector from V onto a subspace VI along a subspace V 2. It is clear that V I is invariant under the linear transformation P- It not only maps elements of V I into V I but also all the elements of V. It is also clear that V 2 is also invariant under P. As a matter of fact, every element of V 2 is mapped into O. We will now determine the structure of the matrix associated with a linear transformation with respect to a basis in the context of invariance.

P 7.2.3 Let W be a subspace of V which is invariant under a given transformation T from V to V. Then there exists a basis of V with respect to which the matrix A of the transformation T can be written in the triangular form

A = mXm

(ic\ 0

(m-k) xk

where m

kX~2_k») A3 '

(7.2.1)

(m-k) x(m-k)

= dim(V) and k = dim(W).

For an exposition on matrices that are associated with linear transformations, see Section 3.4. Let Xl, X2, ... ,Xm be a basis of the vector space V so that XI, X2, ••. ,Xk form a basis for the subspace W. Let A = (aij) be the matrix associated with the linear transformation T with respect to this basis. As a matter of fact, PROOF.

m

AXi = LajiXj, i= 1,2, ... ,m. j=1

Since W is an invariant subspace under T, we must have k

AXi= LajiXj, i= 1,2, ... ,k. j=1


247

This implies that Qji = 0 for j = k + 1, k + 2, .. . ,m and i = 1,2, .. . , k. Hence the matrix A must be of the fonn (7.2.1). IT the linear transformation T is reduced by a pair of subspaces, then the matrix associated with T is more elegant as we demonstrate in the following proposition.

P 7.2.4 Suppose a linear transformation T is reduced by a pair of subspaces VI and V 2 of V. Then there exists a basis of V such that the matrix A of the transformation T with respect to the basis is of the fonn AI

A mxm -

where m

= dim(V),

k

kxk

(

0

(m-k)xk

= dim(Vd,

o

k x (m-k)

A3

)

'

(7.2.2)

(m-k) x (m-k)

and m - k

= dim(V2) '

PROOF. Let XI, X2, •.• ,Xm be a basis of V such that XI, X2,'" ,Xk fonn a basis for VIand Xk+I, Xk+2, •.. ,Xm form a basis for V 2. Following the argument presented in the proof of P 7.2.3, we can discern that A must be of the form (7.2.2). Projectors onto an invariant subspace of some linear transformation have an intimate relationship with the transformation. In the following propositions we bring out the connection.

P 7.2.5 If a subspace W of V is invariant under a linear transformation T from V to V, then PTP = TP for every projector P from V onto W. Conversely, if PTP = TP for some projector P from V onto W, then W is invariant under T. Let P be a projector from V onto W . Then for every X in P)x with Px E W. IT W is invariant under T, then TPx = Py for some y in V. Here we use the fact that W = R(P). Further, PTPx = p2 y = Py = TPx. Consequently, PTP = TP. Conversely, let PT P = T P for some projector P from V onto W. For every x in V, the statement that PT Px = T Px implies that T Px E R(P) = W. If yEW = R(P), then y = Px for some x in V . Consequently, Ty = T Px E W. This shows that W is invariant under PROOF.

V,

T.

X

= Px + (1 -

248


P 7.2.6 A linear transformation T from V to V is reduced. by a pair subspaces VI and V 2 if and only if PT = TP, where P is the projector from V onto V I along V 2·

TP = PT. If x E VI, then Px = x . Note that which implies that Tx E R{P) = VI' This shows that VI is invariant under T. If y E V2, PTy = TPy = TO = O. This shows that Ty E V 2. Hence V 2 is invariant under T. Conversely, suppose that T is reduced by V I and V 2. Since V I is invariant under T, we have PTP = TP by P 7.2.5. Since V2 is invariant under T and (I - P) is a projector from V onto V 2 along VI, we have (I P)T{I - P) = T{I - P) by the same proposition. This simplifies to T + PTP - PT - TP = T - TP, from which we have PT P = PT. The result now follows. PROOF. Suppose

PTx

= TPx = Tx

Complements Develop a result analogous to P 7.2.3 for projectors. Develop a result analogous to P 7.2.4 for projectors. Let V = R2, VI = {(XI,O) E V : Xl real}, and V2 = {(X},X2) : Xl + X2 = O}. Let P be the projector from V onto VI along V2. Let (1,0) and (0,1) constitute a basis for the vector space V. Construct the matrix of the linear transformation P with respect to the given basis. Let (1,0) and (1,-1) constitute another basis for the vector space. Construct the matrix of the linear transformation P with respect to the new basis. 7.2.4 Let W be a subspace of a vector space V. Let dim(V) = m and dim{W) = k. Let T be a linear transformation from V to V and W be invariant under T. Choose a basis Xl, X2, .•. , Xm such that x}, X2, .•• , Xk form a basis for W. Let P be any projector from V onto W. Let AT and Ap be the matrices of the transformations T and P, respectively, with respect to the given basis. Show that ApATAp = ATAp directly. Place this assertion vis-a-vis with P 7.2.5.

7.2.1 7.2.2 7.2.3

7.3. Orthogonal Projection

In Section 2.2 we touched upon the orthogonal projection briefly. See Definition 2.2.12 and the ensuing discussion. In this section, we will spend some time with the orthogonal projection and learn some more. We will be working in the environment of inner product spaces. Let


249

V be an inner product space equipped with an inner product < -, ' >. Let W be a subspace of V and W.l its orthogonal complement. Recall that the projector P from V onto W along W.l is called the orthogonal projection on W. Before we proceed with some characteristic properties of orthogonal projections, we need to brush up our knowledge on the adjoint of a transformation. If T is a linear transformation from an inner product space V into itself, then there exists a linear transformation T* from V to V such that

< x,Ty > = < T*x,y >

for all x and yin V.

The transformation T* is called the adjoint of T. If T* = T , T is called self-adjoint.

P 7.3.1 A linear map P from V to V is an orthogonal projection if and only if P is idempotent and self-adjoint, i.e.,

p2 = P

and P*

=

P.

PROOF. Suppose P is an orthogonal projection. Since it is a projector, it is idempotent. See P 6.1.3. We can identify the relevant subspaces involved. The map P is a projector from V onto R{P}, the range of P, along R(I - P}. Since P is an orthogonal projection, the subspaces R{P} and R(I - P} must be orthogonal, i.e.,

¢:}

'*

< u, v> = 0 for all u E R{P} and v E R(I < (I - P}x, Py >= 0 for all x and y in V < P*(I - P}x, y >= 0 for all x and y in V.

P}

Consequently, P* (I - P)

=0

or P* P

= P* .

(Why?) Observe that P* = P* P = {P* P}* = {P*} * = P. See the complements under Section 3.6. Conversely, suppose that p2 = P and P* = p _ It is clear that P is a projector from R{P} along R(I - P}. See P 6.1.3. We need to show that the subspaces R{P} and R(I - P} are orthogonal. For x and y in V, observe that

< (1 -P}x, Py >=
=< P(I -P}x, Y >=< Ox, Y >= O.

Hence R{ P) and R( I - P} are orthogonal.

250


7.3.2. Let V = em, a complex vector space of dimension m. The standard inner product on V is given by EXAMPLE

m

< X,y >

=

L Xi1Ji, i=1

where x = (xt, X2, ... ,xm ) and y = (yt, Y2,··· ,Ym) E V. Let P = (Pij) be a matrix of order m X m with complex entries. The matrix P can be regarded as a linear transformation from V to V. The complex conjugate of P is the matrix p. = (qij) (abuse of notation?), where % = fiji for all i and j. Recall that P is Hermitian if p. = P - One can verify that for any two vectors x and Y in V,

< x,py > = < p·x,y >, with the understanding that when we write Py we view y as a column vec~or and then carry out matrix multiplication of P and y. The matrix p. is after all the adjoint of P when they are viewed as transformations. The matrix P viewed as a linear transformation is a projector if and only if p2 = P and P is Hermitian. Sums of orthogonal projections are easy to handle. The following proposition handles this situation, which is easy to prove. P 7.3.3 Let Vt, V 2 , ... ,Vr be pairwise orthogonal subspaces of V. Let Vo = VI $ V 2 Ea ... $ Yr. Let Pi be the orthogonal projection on Vi, i = 1,2, ... ,T. Then P = PI + P2 + ... + Pr is an orthogonal projection on Vo.

7.4. Idempotent Matrices Every linear transformation T from a finite-dimensional vector space V to V has a matrix associated with it under a given basis of V. See Section 3.4. In particular, the matrices associated with projectors are of special interest. In this section, we focus on matrices with entries as complex numbers. A square matrix A is said to be idempotent if A2 = A. This definition is analogous to the one we introduced for linear transformations. In brief, transformations and matrices associated with them are hardly distinguishable and using the same word "idempotent" in both the contexts should not cause confusion.


251

The definition of idempotent matrix is also operational when the entries of the matrix come from any field. Some of the results stated below make sense in the general framework of matrices with entries coming from any field. P 7.4.1 Let A be an idempotent matrix of order m x m. The following are valid. (1) The eigenvalues of A can only be zeros and ones. (2) The matrix A admits a factorization A = QR* with Q and R being of order m x k and R*Q = I k , where k = p(A), the rank of A. (3) The matrix A is diagonalizable, i.e., there exists a non-singular matrix L and a diagonal matrix D. such that A = LAL -1, the diagonal entries of A being zeros and ones. (4) p( A) = Tr( A). (The trace operation is discussed in Complement 3.4.7.) (5) There exists a positive definite matrix C such that A = C-I A*C. (6) A is a projection matrix, i.e., there exist two subspaces VI and V2 on em such that VI nV2 = {O}, em = VI EB V2, and if x = Xl + X2 with Xl E VI and X2 E V2, then Ax = Xl. (If we view A as a transformation from em to em, then A is a projector from em onto VI along V 2 , in the usual jargon. As usual, members of em are viewed as column vectors.) PROOF. (1) Let A be an eigenvalue of A with an associated eigenvector x. Then Ax = AX implies that AX = Ax = A2x = A(Ax) = .x(Ax) = .x(.xx) = .x2x. Since X =1= O,.x = or 1. (2) By the Singular Value Decomposition Theorem (P 5.3.4), we can write A = QD.P*, where Q is of order m x k with the property that Q*Q = Ik, P is of order m x k with the property that P* P = h, and D. is a diagonal matrix of order k x k with positive entries in the diagonal. Since A2 = A, we have QD.P*QD.P* = QD.P*, from which we have D.P*QD. = D. or D.P*Q = I k • Take R* = D.P*. Thus we have A = QR* with R*Q = I k • (3) Choose a matrix S of order m x (m - k) so that the augmented matrix L = (QIS) is non-singular and R*S = 0, where Q and Rare the matrices that appear in the representation (2) of A above. (How?) Now choose a matrix U of order m x (m - k) such that U*S = Im-k and U*Q = 0. (How?) One can verify that L- I = (RIU)*. (Verify that

°

252


L- I L = 1m.) Observe that

A

= (QIS)

[~ ~]

(RIU)*

= LAL-I,

(7.4.1)

where A is the diagonal matrix given by

A=

[~ ~] .

(4) From (7.4.1), Tr(A) = Tr(LAL-I) = Tr(AL -1 L) = Tr(A) = k = p(A) . See Complement 3.4.7. (5) Note that 1m - A is also idempotent. Consequently, p(lm A) = m - k. Consider the rank factorizations of A and 1m - A . See Corollary 5.2.3. Write A = DI EI and 1m - A = D2E2, where DI is order m x k with rank k, EI of order k x m with rank k, D2 of order m x (m - k) with rank (m - k), and E2 of order (m - k) x m with rank m-k. Let FI = (DIID2) and F2 = [~~]. Then FIF2 = DIEI +D2E2 = A + (1m - A) = 1m. It now follows that FI = F2- I . Let C = F2 F2. It is clear that C is non-singular and Hermitian. Further, C is positive definite. Note that FI = (DIID2) = F2- I = C-I F2 = C-I(EjIE2'), from which we have DI = C-I Ej or equivalently, CD I = Ej. Finally,

This completes the proof. (6) Take VI = R(A), the range of the matrix A , and V 2 = R(lmA). It is clear that VI nV2 = {o}. For every x in em, note that x = Ax + (1m - A)x, Ax E VI = R(A), and (1m - A)x E V 2 = R(Im - A). This implies that VI EI1 V 2 = em and the projector P from em onto V 1 along V 2 is precisely equal to A. COROLLARY 7.4.2 . If A is idempotent and Hermitian, one can write

A

= TT*

with

T*T

= h,

where k = p(A). PROOF. We use P 5.4.3. Since A is Hermitian, there exists a unitary matrix P such that A = pr P*, where r is a diagonal matrix with


253

diagonal entries constituting the eigenvalues of A. Since A is idempotent, each of the eigenvalues of A is either zero or one. Assume, without loss of generality, that r is of the form

r=[~ ~J. Write P = (TIS), where T is of order m x k . We can now check that

A=TT* andT*T=h. The idempotency of a matrix can be characterized solely based on the additive property of the ranks of two matrices A and 1m - A. The following proposition is concerned with this feature. P 7.4.3 Let A be a matrix of order m x m . Then A is idempotent if and only if p(A) + p(Im - A) = m . PROOF. We have already seen that if A is idempotent then p(A) + p(Im - A) = m . The argument is buried somewhere in the proof of P 7.4.1. Suppose p(A) + p(Im - A) = m . Observe that

m = p(Im) = p(A+(Im-A)) = p(A)+p(Im-A)-dim(R(A)nR(Im- A )). This identity requires some explanation. It is clear that c m = R(A) + R(Im - A). (Why?) A look at P 1.5.7 might help the reader to understand the meaning of the symbol + in the orientation defined above. We do not know that C m = R(A) EB R{Im - A) . Consequently, m

= dim(Cm) =

dim(R(A)

= dim(R(A))

+ dim(R{Im

= p(A)

+ p{Im - A) -

= m - dim(R(A)

+ R{Im -

A))

- A)) - dim(R(A) n R{Im - A))

dim(R(A) n R{Im - A))

n R{Im - A)).

This implies that dim(R(A) n R{Im - A))

= 0 from which we have

R(A) n R{Im - A) = {O}. Thus c m = R(A) (1) R{Im - A) indeed. We claim that A(Im - A) = o. Suppose not. Then there exist non-zero vectors x and y in C m such that A{Im - A)x = y. This implies that y E R(A). Note that A(Im - A) =


254

(Im - A}A. It is true that (Im - A}Ax = y. This implies that y E R(Im - A). This is a contradiction. Hence A(Im - A) = 0 or A2 = A. This completes the proof. The following result is analogous to the one stated in Complement 7.1.8. The statement is couched in terms of matrices with complex entries. P 7.4.4 Let At and A2 be two square matrices of the same order and A = At + A2. Then the following statements are equivalent.

(1) A2 = A and p(A} = p(At} + p(A2). = At, A~ = A2, and AtA2 = A2At =

(2) A~

o.

Suppose (2) is true. It is obvious that A2 = A. Since A, At, and A2 are idempotent, p(A) = Tr(A t } + Tr(A2) = p(At) + p(A2}. Suppose (1) is true. By P 7.4.3, m = p(A}+p(Im-A) = p(A t )+p(A2)+ p(Im - A) ~ p(At} + p(A2 + 1m - A} = p(At} + p(lm - At} ~ p(At + 1m - At} = p(lm} = m. Consequently, p(At) + p(Im - A) = m. Again, by P 7.4.3, At is idempotent. In a similar vein, one can show that A2 is idempotent. The fact that A, At, and A2 are idempotent implies that AtA2 + A2At = O. The information that p(A) = P(Al) + p(A2) implies that R(At} n R(A2) = {O}. This coupled with AIA2 = -A 2A t gives AtA2 = o. Follow the argument crafted in the proof of P 7.4.3. A generalization of P 7.4.4 is in order involving more than two matrices. PROOF.

P 7.4.5 Let At, A2, ... ,Ak be any k square matrices of the same order and A = At + A2 + ... + A k. Consider the following statements. (1) Each Ai is idempotent. (2) AiAj = 0 for all i i= j and p(An = p(Ai) for all i. (3) A is idempotent.

(4) p(A} = p(At} + p(A2} + ... + p(A k }. Then the validity of any two of the statements (1), (2), and (3) imply the validity of all the four statements. Further, the validity of statements (3) and (4) imply the validity of the rest of the statements. PROOF. Suppose (I) and (2) are true. It is clear that (3) is true. Since A and At, A2, ... ,Ak are all idempotent, p(A} = Tr(A} = Tr(At} + Tr(A 2} + ... + Tr(Ak) = p(At} + p(A2} + ... + p(A k}. Thus (4) is true.


255

Suppose (2) and (3) are true. A computation of A2 yields A2 = k

E At·

Fix 1 ::; i ::; k. We show that Ai is idempotent. Note that

i=l

AAi = AiA = At and A2 Ai = AiA2 = Ar Since A is idempotent, we have At = A~, which implies that Ar(Im - Ai) = O. The condition p(Ai) = p(At) is equivalent to the statement that dim(R(Ai)) = dim(R(A;)). Since R(A;) C R(A i ), we must have R(Ai) = R(At). Consequently, there exists a nonsingular matrix D such that Ai = DAr. Hence A;(Im - A) = 0 implies that Ai(Im - A) = 0 from which we conclude that Ai is idempotent. Thus (1) is true. Now (4) follows. j. Let B = Ai + Aj and Suppose (3) and (4) are valid. Fix i C = A-B. By (4),

t=

k

k

LP(Ar)

= p(A) = p(B + C)

::; p(B)

+ p(C)

::; LP(Ar).

r=l

r=l

From this, we have p(A) (Why?) Observe that

=

p(B)

+ p(C)

and p(B)

=

p(Ai)

+ p(A j ).

+ 1m - B) ::; p(B) + p(Im - B) p(B) + p(lm - A + C) = p(B) + p(lm - A) + p(C) p(A) + p(Im - A) = m.

m = p(lm) = p(B

= =

Hence p{B) + p{Im - B) = m. By P 7.4.3, B is idempotent. Thus we have Ai + Aj idempotent and p(B) = p(Ai) + p(Aj ). By P 7.4.4, AiAj = 0 and Ai and Aj idempotent. Thus (1) and (2) follow in one stroke. Suppose (1) and (2) are valid. It is obvious that (4) follows exploiting the connection between rank and trace for idempotent matrices. Since we have (4) valid, (2) follows now from what we have established above. This completes the proof. The condition in (2) of P 7.4.5 that p(Ai) = p(A;) is somewhat p(B2) for a matrix B. As an intriguing. It could happen that p(B) example, try

t=

B=

[~

This will not happen if B is Hermitian or nonsingular.

256


Complements 7.4.1 Let Yb Y2, ... , Ym be m real random variables whose joint distribution depends on a vector parameter ()' = (()I, ()2,··· ,()k) E Rk. Suppose E(J~ = Xil()1 +Xi2()2 + ... +Xik()k, i = 1,2, ... ,m, where Xi/ S are known. Let X = (Xij). Assume that p(X) = k. Let g(()) = (), () E Rk. The random variables constitute a linear model and one can rewrite the expectations as E(JY = X(),() E Rk, where Y' = (YI , Y2, ... ,Ym ). The least squares estimator 9 of g( .) is given by 9 = (X' X)-I X'Y. The residual sum of squares (RS S) is given by

RSS = (Y - Xg)'(Y - Xg). Show that

(1) (2) (3) (4)

X(X' X)-I X' and (Im - X(X' X)-I X') are idempotent; p(X(X' X)-I X') = k; E(Jg = () for every () E Rk; E(JRSS = (m - k)(J'2 by assuming that YI , Y2, . .. ,Ym are pairwise uncorrelated with common variance

(J'2

> 0;

(5) 9 and RSS are independently distributed by assuming that YI , ... , Ym have a multivariate normal distribution with variance covariance matrix (J'21m. (Hint: A linear form LY and a quadratic form Y' AY are independently distributed if LA = 0.)

7.4.2 7.4.3

If A is idempotent and non-singular, show that A = I. Let

A=[~

g],

where Band D are square matrices. Show that A is idempotent if and only if Band D are idempotent, BCD = 0, and (I - B)C(I - D) = o.

7.5. Matrix Representation of Projectors In this section, we consider a finite dimensional vector space V over the field of complex numbers equipped with an inner product < .,. >. Let be an ordered collection of vectors from V. We define some algebraic operations on the space of all ordered sets of the form S .


257

Addition of ordered sets Let SI = (aI, a2, . .. ,an) and S2 = (bl, b2, ... ,bn ) be two ordered collections of vectors from V. Define the sum of SI and S2 by (7.5.1) Multiplication of ordered sets Let Sl = (aI, a2, ... ,am) and S2 = (bt,~, ... ,bn ) be two ordered collections of vectors from V. Define the product of SI and S2 to be the matrix of order m X n (7.5.2)

Multiplication of an ordered set and a matrix Let S = (aI, a2, . .. ,an) be an ordered set of vectors from V and (mij) a matrix of order n x k with complex entries. Define the product of Sand M to be the ordered set

M =

n

S x M =

(2: j=1

n

mjl a j,

2:

n

mj2 a j, ...

j=1

,2:

mjkaj),

(7.5.3)

j=1

which is an ordered set of k vectors. IT M is a column vector with entries mt, m2, .. ' ,mn , then S x M is simply the linear combination ml al + m2a2 + ... + mnan of the vectors at, a2,· .. ,an, The operation 0 in (7.5.2) is a little fascinating. Let us examine what this means when the vectors ai's and bi's come from the coordinate vector space Ck. The set SI turns out to be a matrix of order k x m with complex entries and S2 is of order k x n with complex entries. It can be verified that SI 0 S2 = S;S2. A special case of this is when SI

= S2 = S,

SoS = S*S.

say, in which case

258


The operation x simplifies the usual matrix multiplication in coordinate vector spaces. It turns out that S x M = SM, the usual matrix multiplication of Sand M. We record some of the properties of these operations in the following proposition which can be verified easily. In the following the symbol S with subscripts stand for ordered sets of vectors and M with SUbscripts stand for matrices. P 7.5.1

The following are valid.

(1) If S = (aI, a2,' " ,am), M1 = (mD») and M2 = (m~~») each of

order m x k, then S x (M1 + M 2) = (S x Md + (S x M2) ' (2) If Sl = (aI,a2,'" ,am ),S2 = (b 1,b2, ... ,bm ), and M = (mij) of order m x k, then (Sl + S2) x M = (Sl X M) + (S2 X M) . (3) If Sl = (ab a2,·· . ,am), S2 = (bb b2 , ... , bn ), and M = (mij) of order n x k, then Sl 0 (S2 X M) = (Sl 0 S2)M. (4) If Sl = (aI, a2, . .. ,am), M = (mij) of order m x k, and S2 = (b1'~" " ,bn ), then (Sl X M) 0 S2 = M*(Sl 0 S2). (5) If Sl = (al, a2 , ··. ,am), S2 = (bb~, .. . ,bn ) , and S3 = (C1, C2 , ... ,cr ), then Sl 0 (S2 + S3) = Sl 0 S2 +Sl 0 S3. Having an explicitly workable expression for the projector from a vector space V onto a subspace VI along a subspace V 2 seems to be difficult. Using the new operations developed at the beginning of this section, we will attempt to provide an explicit formula. P 7.5.2 Let S = (aI, a2, . .. ,am) be an ordered collection of vectors from a vector space. Let R(S) be the vector space spanned by the vectors of S. Then the orthogonal projection P from V onto R(S) has the representation, P=Sx M(So .) (7.5.4) so that the orthogonal projection of a vector x in V is given by Px = S x M(Sox),

(7.5.5)

where M is any matrix of order mXm satisfying (SoS)M(SoS) = SoS. PROOF. Let bI,~, .. . ,b k be a basis to the R(S).l , the orthogonal complement of R(S). Take Sl = (bb~, ... ,bk ). It is clear that

So Sl = O.


259

X E V. Since R(8) ffi R(8)l. = V, we can write x = Xl + X2 with E R(8) and X2 E R(8)l.. The vector Xl is the one we are going after, i.e., Px = Xl. The vector Xl is a linear combination of the vectors in 8 and X2 is a linear combination of the vectors in 8 1 . Consequently, there are vectors y and z of orders m X 1 and k xl, respectively, such that, in terms of our new algebraic operations,

Let Xl

Xl

= 8 x y

and

X2

= 81

IT we know y, we will know the projection X

= 8 x y

Xl

X

z.

of x. Thus we have

+ 81 X z

(7.5.6)

for some column vectors of complex munbers. Premultiply (7.5.6) by 8 with respect to the operation 0

80 X

= 80 (8 x y) + 8 0 (81 X z) = (80 8)y + (80 8t}z = (80 8)y.

We can view (8 0 8)y = 8 0 X as a linear equation in unknown y. Let M be a generalized inverse of the matrix (808), i.e., M satisfies the equation, (80 8)M(8 08) = (808). A solution of the linear equation is given by y = M(8 0 x). We are jumping the gun again! We will see in Chapter 8 why this is so. Thus

Px

= Xl = 8

xy

=8

x M(8o x).

This completes the proof. We will specialize this result for coordinate spaces. Suppose V = en. Then 8 is an n x m matrix. Take the inner product < -, - > to be the standard inner product in V, i.e., for X and y in V, < X, y >= y*x. Note that 808 = 8*8 and 80 X = 8*x. Finally,

8 x M(8ox) = 8 x M8*x = 8M8*x, where M is a generalized inverse of the Hermitian matrix 8*8. Let us enshrine this result in the form of a corollary. COROLLARY 7.5.3. Let V = en and 8 a matrix of order n X m with complex entries. Let R(8) be the vector space spanned by the

260


columns of S. Then the projection operator P from V into R(S) has the representation, (7.5.7) P= SMS* where M is any matrix satisfying (S* S)M(S* S) = S* S. The expression (7.5.7) for the projection operator first given by Rao (1967) was useful in the discussion of linear models under a very general setup. Now we take up the general case of projectors. Let Sl and S2 be two ordered sets of vectors from an inner product space V. Suppose R(Sl) and R(S2), the vector spaces spanned by Sl and S2, respectively, satisfy the conditions that R(SdnR(S2) = {O} and R(Sl)El1R(S2). The spaces R(Sl) and R(S2) need not be orthogonal. Let P be the projector from V onto R(Sd along R(S2)' We need an explicit representation of P. Let S3 be an ordered set of vectors from V such that R(S2) and R(S3) are orthogonal and V = R( S2) El1 R( S3)' (How will you find S3?) In particular, we will have S3 0 S2 = O. P 7.5.4 In the framework described above, the projector P has the representation Px = Sl X M(S3 ox)

for any x in V, where M is a generalized inverse of the matrix S3 i.e., M satisfies (S3 0 Sl)M(S3 0 Sd = S3 0 Sl.

0

S1,

Let x E V. Write x = Xl + X2 with Xl E R(Sl) and X2 E R(S2). Since Xl is a linear combination of the vectors in S}, we can write Xl = Sl X Y for some column vector of complex entries. In a similar vein, we can write X2 = Sl X z for some column vector z. Premultiplying X = Sl X Y + S2 X z by S3 under the operation 0, we have PROOF.

S3

0

X = S3

(Sl

0

= (S3

0

X

y)

+ S3 0

Sl)y + (S3

0

(S2

S2)Z

X

z)

= (S3 0

A solution to the system of linear equations (S3 unknown y is given by y = M(S3 0 x). Thus

Px

= Sl

This completes the proof.

X

Y = Sl

X

M(S3

0

x).

0

St}y. St}y -

S3

0

X in

Pmjectora and Idempotent Operatorn

261

In the above proof, we roped the ordered set S3 in the representation of the projector. We could avoid this. P 7.5.5 Let the framework of P 7.5.4 be operational here. Let G be any generalized inverse of the matrix

Partitioned as

where the order of the matrix C1is the same as the order of S1o S1 and the order of C4is the same as the order of S2o S2. Then for any x in

Vl Px

= Cl(S10 x)

+ C2 (S2

0

x).

PROOF.As in the proof of P 7.5.4, write for some column vectors y and z with complex entries. Premultiply (7.5.8) by S1 under the operation o. We will have

Premultiplying (7.5.8) by S2 under the operation o, we have

Equations (7.5.9) and (7.5.10) can be written as

[

]

Y = SlOX. [Z]

[S20x]

This is a system of linear equations in the unknowns y and Z. A solution is given by

Consequently, y = Cl(Sl o x ) +C2(S2

o'x), 2 = C3(Si 0 X) + C4(S2 O

262


Finally, Px = S} proof.

X Y

= C}(S}

0

x)

+ C2(S2 0

x). This completes the

Complements 7.5.1 For vectors x and y in a vector space en, define the inner product by < x, y >= y*Ex, where E is a positive definite matrix. Then P is an orthogonal projector if and only if

(1)

p2 = P,

and

(2)

EP is Hermitian.

7.5.2 (Rao (1967)) Let C} be a subspace of cn spanned by columns of an n X k matrix A. Show that the orthogonal projector onto C} is

P

=

A(A*EA)- A*E

where (A*EA)- is a generalized inverse of (A*EA), i.e., any matrix B satisfying the property (A*EA)B(A*EA) = A*EA. The expression for P is unique for any choice of the generalized inverse. (For a discussion of generalized inverses, see Chapter 8.) 7.5.3 Let A be n x p and B be n x q real matrices and denote their Kronecker product by A ® B. Denote by PA, PB and P A0B the orthogonal projectors on R(A), R(B) and R(A ® B) respectively. Then

(1) PA0B = PA ®PB

(2) PA0I=PA®I (3) QA0B = QA ® QB

+ QA ® PB + PA ® QB,

where QA

=

1-

PA, QB = I -QB.

Note: The references for this Chapter are: Rao (1967), Rao and Mitra (1971b), Rao (1974), Rao and Mitra (1974), Rao (1978c), Roo and Yanai (1979) and standard books on Linear Algebra.

CHAPTER 8 GENERALIZED INVERSES

In Section 3.3, we explored the concept of an inverse of a linear transformation T from a vector space V to a vector space W . We found that the nature of the inverse depends upon what kind of properties T has. A summary of the discussion that had been carried out earlier is given below. (1) Suppose T is bijective, i.e., T is injective (one-to-one) and surjective (onto) . Then there exists a unique transformation S from W to V such that ST = I and T S = I with the identity transformation acting on the appropriate vector space. (2) Suppose T is surjective. Then there exist a transformation S (called a right inverse of T) from W to V such that TS = I. (3) Suppose T is injective. Then there exists a transformation S (called a left inverse of T) from W to V such that ST = I . (4) There always exists a transformation S from W to V such that TST = T . Such a transformation S is called a g-inverse of T.

In this chapter, we focus on matrices. We reenact the entire scenario detailing inverses of transformation in the realm of matrices. Special attention will be paid to finding simple criteria for the existence of every type of inverse. The source material for this Chapter is Rao and Mitra (1971). Before we proceed with the details, we need to set up the notation. Let Mm ,n denote the collection of all matrices A of order m x n with entries coming from the field of real or complex numbers. The symbol Mm,n (r) denotes the collection of all matrices A in Mm,n with rank r. The rank of a matrix A is denoted by p{A). The vector space spanned by the columns of a matrix A is denoted by Sp{A) . An equivalent 263

264


notation is R(A), the range of A when A is viewed as a transformation. See Section 4.1. [Sp(A) is more suggestive when A is a matrix as the vector space generated by the column vectors of A.] 8.1. Right and Left Inverses

In this section we characterize right and left inverses of matrices. In addition, the structure of a right inverse as well as left inverse of a matrix is described. P 8.1.1 Let A E Mm,n. There exists a matrix G E Mn,m such that AG = 1m if and only if p(A) = m. In such a case a choice of G is given by (8.1.1) A general solution of G is given by G = VA*(AVA*)-l,

(8.1.2)

where V is any arbitrary matrix satisfying p(A) = p(AV A*) . Suppose p(A) = m. Then m = p(A) = p(AA*) . The matrix AA * is of order m x m and has rank m. Consequently, AA * is nonsingular. The matrix G = A*(AA*)-l indeed satisfies AG = AA*(AA*)-l = 1m. Conversely, suppose there exists a matrix G E Mn ,m such that AG = 1m. Note that m = p(Im) = p(AG) ::; p(A) ::; m. Hence p(A) = m. As for the general structure of G, if V is any matrix satisfying p(AV A*) = p(A) , then G = V A*(AV A*)-l certainly satisfies the condition AG = 1m. On the other hand, if G is any matrix satisfying AG = 1m, it can be put in the form G = V A*(AV A*)-l for some suitable choice of V. Take V = GG*. (How?) The matrix G that figures in P 8.1.1 can rightly be called a right inverse of A. One can also say that a right inverse of A exists if the rows of A are linearly independent. Incidentally, p( G) = m. A similar result can be crafted for left inverses of A. PROOF.

P 8.1.2 Let A E Mm,n . Then there exists a matrix G E M .. ,m such that GA = In if and only if p(A) = n . In such a case one choke of G is given by G=(A*A)-lA*. (8.1.3) A general solution of G satisfying G A = In is given by G = (A*V A)-l A*V (8.1.4)

Generalized Inverses

265

for any matrix V satisfying p(A) = p(A*V A). The matrix G that figures in P 8.1.2 can be rightly called a left inverse of A . The existence of a left inverse of A is guaranteed if the columns of A are linearly independent. Incidentally, p( G) = n. The right and left inverses have some bearing on solving linear equations. Suppose Ax = y is a consistent system of linear equations in an unknown vector x, where A E Mm,n and y E M m ,1 are known. Consistency means that the system admits a solution in x. Suppose p(A) = n. Let G be any left inverse of A . Then x = Gy is a solution to the linear equations. This can be seen as follows . Since Ax = y is consistent, y must be a linear combination of the column of A. In other words, we can write y = Aa for some column vector a. We now proceed to verify that Gy is indeed a solution to the system Ax = y of equations. Note that A(Gy) = AGAa = A1na = Aa = y. Let us look at the other possibility where we have a consistent system Ax = y of linear equations with p(A) = m. Let G be a right inverse of A. Then Gy is a solution to the system Ax = y . This can be verified directly. Incidentally, note that if p(A) = m, Ax = y is always consistent whatever may be the nature of the vector y!

Complements 8.1.1 Let A be a matrix of order m x 1 with at least one non-zero entry. Exhibit a left inverse of A. Obtain a general form of left inverses of A. 8.1.2 Let A be a matrix of order 1 x n with at least one non-zero entry. Exhibit a right inverse of A . Obtain a general form of right inverse of A. Show that Ax = {J is consistent for any number (J. 8.1.3 Let A E Mm ,n. Show that Ax = y is consistent for all y E M m,1 if and only if p(A) = m. 8.1.4 Let

A=

[~

2

3

Obtain a right inverse of A.

8.2. Generalized Inverse (g-inverse) One of the basic problems in Linear Algebra is to determine solutions to a system Ax = y of consistent linear equations, where A E Mm,n.

266


If the matrix is of full rank, i.e., p(A) = m or n, we have seen in the last section how the right and left inverses of A, as the case may be, help to obtain a solution. It is time to make some progress in the case

when A is not of full rank. Generalized Inverses or g-inverses of A are the matrices needed to solve consistent linear equations. They can be introduced in a variety of ways. We follow the linear equations angle. DEFINITION 8.2.1. Let A E Mm,n be of arbitrary rank. A matrix G E Mn,m is said to be a generalized inverse (g-inverse) of A if x = Gy is a solution of Ax = y for any y for which the equation is consistent. This is not a neat definition. It is a goal-oriented definition. Later, we will provide some characterizations of g-inverses, one of which could give us a crisp mathematical definition. The customary notation for a g-inverse of A is A -, if it exists. First, we settle the question of existence.

P 8.2.2

For any matrix A E Mm,n, A- E Mn,m exists.

PROOF . If A = 0, take G = o. Assume that A :f. O. Let us make use of the rank factorization of A . Write A = RF, where R is of order m x a with rank a and F of order a x n with rank a, where a = p(A). See Corollary 5.2.3. Let B be a left inverse of R, i.e., BR = la, and C a right inverse of F, i.e., FC = I. Let A- = CB. We show that Ais a g-inverse of A. Let y E Sp(A), the vector space spanned by the columns of A. Then the system Ax = y is consistent. Also, y = Aa for some column vector a. We show that A-y is a solution of Ax = y.

AA-y = (RF)(CB)y = R(FC)By = RBy = RBAa = (RB)(RF)a = R(BR)Fa = RFa

which shows that x

= Aa = y,

= A-y satisfies the equation Ax = y.

P 8.2.3 Let A E Mm,n and G E Mn,m. The following statements are equivalent. (1) G is a g-inverse of A. (2) AG is an identity on Sp(A), i.e., AGA = A. (3) AG is idempotent and p(A) = p(AG).


267

PROOF. We show that (1) => (2) => (3) => (1). Suppose (1) is true. Let aI, a2, . .. ,an be the columns of A . It is clear that Ax = ai is a consistent system of equations for each i. Since G is a g-inverse of A, Gai is a solution to the system Ax = ai, i.e., AGai = ai· This is true for each i . Combining all these equations, we obtain AGA = A. The statement that AG is an identity on Sp(A) is a restatement of the fact that AGy = y for y E Sp(A). Thus (2) follows. Suppose (2) is true. Post-multiply the equation AGA = A by G. Thus we have AGAG = (AG)2 = AG, which means that AG is idempotent. Note that p(A) = p(AGA) S; p(AG) S; p(A). Hence p(A) = p(AG) . Thus (3) follows. Suppose (3) is true. Let Ax = y be a consistent system of equations. Consistency means that y E Sp(A), i.e., y = Aa for some column vector a. Since p(A) = p(AG), Sp(A) = Sp(AG) . Consequently, y = AG{3 for some column vector {3. We will show that Gy is a solution of Ax = y. Note that AGy = AGAG{3 = AG{3 = y, since AG is idempotent. Thus (1) follows. The characteristic property that a g- inverse G of A satisfies AG A = A can be taken as a definition of g-inverse. The fact that for any g-inverse G of A, AG is idempotent puts us right into the ambit of projectors. In fact, AG is a projector from em onto Sp(A)(= Sp(AG)) along Sp(ImAG). (Why?) It is interesting to note that GA is also idempotent. It is also interesting to note that G E Mn,m is a g-inverse of A if and only if p(A) + p(In - GA) = n . This is reminiscent of the result P 7.4.3. Let G E Mn ,m be a g-inverse of A E Mm,n' The matrix AG behaves almost like an identity matrix. What we mean by this is (AG)A = A, i.e., AG behaves like an identity matrix when multiplying A from the left. In the following proposition we examine under what circumstances AGB=B. P 8.2.4 (1) For a matrix B of order m x k, (AG)B = B if and only if Sp(B) c Sp(A), i.e., B = AD for some matrix D. (2) For a matrix B of order k xn, B(GA) = B if and only if Sp(B') Sp(A'), i.e. B = DA for some matrix D.

c

PROOF. (1) We have already seen that AG is an identity on Sp(A). Consequently, AG is an identity on any subspace of Sp(A) . Thus if B = AD, then AGB = B. Note that AG is a projector from em onto

268


Sp(A) along Sp(Im - AG) . Consequently, if y is a non-zero vector with (AG)y = y, then y better be a linear combination of the columns of A, i.e., y E Sp(A). Hence, if (AG)B = B, then Sp(B) c Sp(A).

(2) This is similar to (1). A number of corollaries can be deduced from P 8.2.4. COROLLARY 8.2 .5. Let A E Mm,n and G E Mn ,m a g-inverse of A. If a is a column vector consisting of n entries such that a E Sp(A') and f3 is a column vector consisting of m entries such that f3 E Sp(A), then a'Gf3 is invariant (i.e., a constant) for any choice of G. PROOF. The conditions on a and f3 mean that a = A', for some column vector, and f3 = Ab" for some column vector 15. Consequently, a ' Gf3 = " AGAf3 = ,'Ab", a constant independent of the choice of G. Corollary 8.2.5 can be generalized. COROLLARY 8 .2.6 . Let G E Mn,m stand as a generic symbol for a g-inverse of A. Suppose Band G are matrices of orders p X nand m x q, respectively, such that Sp(B') c Sp(A') and Sp(C) c Sp(A). Then BGG is invariant for any choice of G. PROOF. A proof can be crafted along the lines of the proof of Corollary 8.2.5. COROLLARY 8.2.7. Let A E Mm,n ' Let (A* A)- E Mn stand for aginverse of A* A. Then A(A* A)-(A* A) = A and (A* A)(A* A)- A* = A*. PROOF. First, we observe that Sp(A*) = Sp(A* A). Consequently, A* = (A* A)D for some matrix D. Therefore, A = D* A* A and A(A* A)(A* A) = D* A* A(A* A)-(A* A) = D* A* A = A . As for the second identity, note that (A* A)(A* A)- A* = (A* A)(A* A)- A* AD = A* AD = A* . Corollary 8.2.7 can be strengthened. Let V be any matrix such that p(A*V A) = p(A). If V is positive definite, this rank condition is definitelysatisfied. ThenA(A*VA)-(A*VA) = A and (A*VA)-(A*VA)A* = A*. The matrix A(A* A)- A* for any A plays a crucial role in Linear Models. Linear Models provide a very general framework embodying Multiple Regression Models and Analysis of Variance Models. In the following proposition, we demonstrate that this matrix has some special properties.


269

P 8.2.8 Let A E Mm,n ' Then A(A* A)- A* is Hermitian, idempotent, and invariant whatever may be the choice of (A* A)- . PROOF. Idempotency is easy to settle: (A(A* A)- A*)A(A* A)- A* = (A(A* A)- A* A)((A* A)- A*) = A(A* A)- A*, by Corollary 8.2.7. Let us look at the invariance property. Since Sp(A*) = Sp(A* A), we can write A* = (A* A)D for some matrix D. Consequently,

A(A* A)- A* = D* A* A(A* A)- A* AD = D* A* AD, which is a constant whatever g-inverse of A * A we use. Incidentally, we note that D* A * AD is Hermitian. This completes the proof. P 8.2.8 can be strengthened. Let V be any positive definite matrix such that p(A*V A) = p(A) . Then A(A*V A)- A* is invariant for any choice of (A*V A)-. Further, if A*V A is Hermitian, so is A(A*V A)- A*. We now focus on a consistant system Ax = y of linear equations. Using a single g-inverse of A, we will demonstrate how all the solutions of the system of equations can be generated. (See Rao (1962) .) P 8.2.9 Let A E Mm,n and G E Mn ,m be a fixed g-inverse of A. (1) A general solution of the homogeneous system Ax = 0 of equations is given by x = (In - GA)z, where z is an arbitrary vector. (2) A general solution to the system Ax = y of consistent equations is given by x = Gy + (In - GA)z where z is an arbitrary vector. (3) Let q be any column vector consisting of n components. Then q' x is a constant for all solutions x of the consistent system Ax = y of equations if and only if q'(In - GA) = 0 or equivalently, q E Sp(A'). (4) A necessary and sufficient condition that the system Ax = y is consistent is that AGy = y. PROOF. (1) First we note that for any vector z, (In - GA)z is a solution of the system Ax = 0 of equations. On the other hand, let x be any solution of the system Ax = O. Then

x

= GAx + (In -

GA)x

= 0 + (In -

GA)x

= (In

- GA)z,

with z = x. Thus we are able to write x in the form (In - GA)z for some vector z.

270


(2) This result follows since a general solution of Ax = y is the sum of a particular solution of Ax = y and a general solution of Ax = O. Note that Gy can be taken as a particular solution. (3) Suppose q'(In - GA) = o. Any solution of Ax = y is of the form Gy+ (In - GA) z for some vector z . We compute q'(Gy+ (In - GA)z) = q'Gy. Since Ax = y is consistent, y E Sp(A). Since q'(In -GA) = O,q E Sp(GA) = Sp(A') . (Why?) By Corollary 8.2.5, q'Gy remains the same regardless of the choice of G. Conversely, suppose q'x remains the same for all solutions x of Ax = y. This means that q'(Gy + (In - GA)z) = constant for all z in en . This implies that q'(In - GA)z = 0 for all z E en . (Why?) Hence q'(In - GA) = O. (4) If Ax = y is consistent, we have seen that x = Gy is a solution. Then AGy = y. The converse is obvious. If A is a nonsingular matrix, g-inverse of A is unique and is given by A-l. This means that AA-l = A-lA = I. If A is a singular square matrix or rectangular matrix, AA - and A - A are, in general, not identity matrices. We shall investigate how exactly AA - and A-A differ from identity matrices and what is common to different g-inverses of A. The answer is contained in the following result.

p 8.2.10 Let A E Mm ,n and G E Mn,m be any two matrices. Partition A and G as

A=

[p~:l ' qXn

G = (G l : G 2 ) nxp nXq

with p + q = m. Then Sp(A~)

n Sp(A;)

=

{O} and G is a g-inverse of A

if and only if

PROOF. Suppose G is a g-inverse of A. The equation AGA = A yields


271

These equations can be rewritten as

Taking transposes, we obtain

If Sp(A~) n Sp(A2) = {O}, the above equations cannot hold unless each expression is zero. Hence

Thus (8.2.1) follows. Conversely, suppose (8 .2.1) holds and (8.2.2)

°

for some vectors x and y. We will show that A~x = A 2y = which would then imply that Sp(A~) n Sp(A2) = {O} . Pre-multiply (8.2.2) by A~ G~. Note that A~ G~A~x = A~ G~A2Y = 0, which implies that A~ x = 0, since A~ G~ A2 = 0. It is simple to check that G is a g-inverse of A when (8.2.1) holds. This completes the proof. This result has a cousin. The matrices could be partitioned in a different way.

P 8.2.11

Let A E Mm ,n and G E Mn ,m be partitioned as

A

= (AI

mxp

: A2

),

G

mxq

=

p Xm

GI [ G2

1

(8.2.3)

qXm

with p + q = n . Then Sp(Ad if and only if

n Sp(A 2 )

= {O} and G is a g-inverse of A

These are useful results. Under some conditions, a g-inverse of a submatrix of A can be picked up from the corresponding submatrix of a

272


g-inverse G of A . Note that the condition that Sp(A I )nSp(A2) = {O} is equivalent to the condition that p(A) = p(Ad + p(A2)' There is another way to look at the result. Pick up some g-inverses G I and G 2 of Al and A 2 , respectively. Build the matrix (8.2.3). We could wish that G be a g-inverse of A. In order to have our wish realized, we must have as a pre-condition that the ranks be additive, i.e., p(A) = p(Al) + P(A2)' We will now derive a number of corollaries from P 8.2.10. COROLLARY 8.2.12. Let matrices. Partition

A=

A

E

All [A2 pxn

Mm ,n

and G E

Mn,m

be any two

, G = ( G I IG 2 ) nXp n x q

qxn

with p + q = m . Then Sp(A~) n and P(AI) = P if and only if

Sp(A~) =

{O}, G is a g-inverse of A,

PROOF. Suppose Sp(AD n Sp(A~) = {O}, G is a g-inverse of A, and p(Ad = p. Then AIGIAI = AI, A2GIAI = 0, A 2G 2A 2 = A 2, and A I G 2A2 = O. See P 8.2.10. If P(AI) = p, we can cancel Al on the right from both sides of the equations AIGIAI = Al and A2GIAl = O. (How?) The converse is straightforward. Look at the product under the stipulated conditions of Corollary 8.2.12.

The matrices AG and 1m have the same first p columns. COROLLARY 8 .2.13. Let A E Mm ,n and G E Mn ,m be any g-inverse of A. A necessary and sufficient condition that i-th column vector of AG is the same as the i-th column vector of 1m is that the i-th row vector of A is non-zero, and cannot be written as a linear combination of the remaining row vectors of A.


273

PROOF. One simply checks the condition of Corollary 8.2.12. One of the interesting implications of Corollary 8.2.13 is that if all the row vectors of A are linearly independent and G is a g-inverse of A, then AG = In. In other words, if all row vectors of A are linearly independent, then p{A) = m and any g-inverse G of A is indeed a right inverse of A confirming the results of Section 8.1. Corollary 8.2.11 can be rehashed in a different way. COROLLARY 8.2.14. Let A E Mm,n be partitioned as A = (AI: A 2 ) mXp

with p Then

+q

mxq

= n. Let G E Mn,m be a matrix partitioned as in (8.2.3).

n Sp(A2) = {O}, G is a g-inverse of A, and p(Ad = p if and only if G1AI = I p , G 1A2 = 0, A 2G 2A 2 = A 2, and A 2G 2A 1 = O. Sp(AI)

The implication of this result is that the matrices G A and In have the same first p rows. Corollary 8.2.13 has a mate. COROLLARY 8 .2.15. Let A E Mm,n and G E Mn,m be any g-inverse of A. Then a necessary and sufficient condition that the i-th row vector of GA is the same as the i-th row vector of In is that the i-th column vector of A is non-zero, and cannot be written as a linear combination of the remaining columns of A. One of the implications of Corollary 8.2.15 is that if all the column vectors of A are linearly independent, i.e., p(A) = n, then GA = In. This means that G is a left inverse of A. The notions of generalized inverse and left inverse coincide in this special setting. We can rephrase these comments in another way. Suppose A E Mm,n and p(A) = m. We have seen in Section 8.1 that A admits a right inverse G. From the very definition of right inverse, it is transparent that G is indeed a g-inverse of A. Corollary 8.2.13 implies that every g-inverse of A is indeed a right inverse of A. Similar remarks apply when p{A) = n. COROLLARY 8.2.16. Let A E Mm,n be partitioned as

A=

[1J~ A21 sxq

1J;] A22 8Xr

274


with p + s = m and q + r = n. Let G E Mn,m be any g-inverse of A. Partition G accordingly, i.e. ,

G11

G=

qx s G12]

qx p

21 [G rxp

G 22 r Xs

Suppose each of the first q columns of A is non-zero and is not a linear combination of all other columns of A and each of the first p rows of A is non-zero and is not a linear combination of all other rows of A. Then G 11 remains the same regardless of the choice of G. PROOF. What the conditions of the proposition mean are

Sp{{A 11 IA 12 )') Sp

n Sp{{A 21 IA 22 )') = {O}

[~~~J nsp [~~~]

= {O}

and p

and

P{AllIA12) = p,

[~~~]

= q.

(8.2.4)

Under (8.2.4), by Corollary 8.2.12,

These equations are equivalent to (8.2.5) Suppose F is another g-inverse of A partitioned in the same style as of G. Let Fij's stand for the blocks of F. Thus we must have (8.2 .6) By subtracting (8.2.6) from (8 .2.5), we observe that

All{Gll-Fll)+A12{G21-F2d

= 0,

We can rewrite these equations as

A21{Gll-Fu)+A22{G21-F2d

= O.


275

This equation means that some linear combinations of the first q columns of A are the same as some linear combinations of the last n - q columns of A. In view of (8.2.4), we must have

Since p

[Au] = q, A21

G u - Fu = O. (Why?) This completes the proof. The following is a special case of Corollary 8.2.16. COROLLARY 8.2.17. If the i-th row of a matrix A E M m •n is nonzero and is not a linear combination of the remaining rows of A and the j-th column of A is non-zero and is not a linear combination of the remaining columns of A, then the (j, i)-th element of a generalized inverse G of A is a constant regardless of the choice of G. We now focus on non-negative definite (nnd) matrices. COROLLARY 8.2.18. Let A E Mm be an nnd matrix partitioned as

Au A= [A21 pxp qxp

with p + q = m. Let G E Mm be any g-inverse of A. Partition G accordingly, i.e.,

GU

G=

pXp

pxq G12]

[ G 21 Gn qXp

qxq

Suppose each of the first p rows of A is non-zero and is not a linear combination of the remaining rows of A. Then

for any g-inverse of A22 of A 22 .


276

PROOF. The conditions of Corollary 8.2.12 are met. Consequently, (8.2.7) and (8.2.8) Choose and fix a g-inverse A22 of A 22 .

Pre-multiplying (8.2.8) by

A 12 A 22 , we obtain

(8.2.9) Since A is nnd, A12A22A22 = A 12 . (Why?) (Since A is nnd, Sp(A21 ) C Sp(A22)') Thus we have from (8.2.9), (8.2.10) Subtracting (8.2.10) from (8.2.7), we obtain (Au - A12A;A.21)G U = Ip.

Hence G u = (Au - A12A22A21)-1. (Note that G u is unique. See Corollary 8.2.16.) COROLLARY 8.2.19. Let A E Mm,n be partitioned as A = (Al

I A2 )

mxp mxq .

with p + q = n. Let G E Mn,m be any g-inverse of A. Let G be partitioned accordingly as in (8.2.3). If Sp(Ad n Sp(A 2) = {O} and Sp(Ad ED Sp(A2) = em, then p = Al G 1 is a projector from em onto Sp(Ad along Sp(A2)' Suppose A = (A1IA2) is a partitioned matrix with Al being of order m x p and A2 of order m x q. Suppose we have g-inverses G 1 and G 2 of Al and A 2, respectively, available. We string G 1 and G 2 as in (8.2.3) . Under what conditions G is a g-inverse of A? P 8.2.11 provides an answer. In the following, we provide a sufficient condition. COROLLARY 8.2 .20. Let Al and A2 be two matrices of orders m x p and m x q, respectively. Let A = (A1IA2) and G 1 and G 2 be g-inverses


of Al and A 2, respectively. If AlGI F of A, then G

=

[g~]

+ A 2G 2 =

277

AF for some g-inverse

is a g-inverse of A.

PROOF. It suffices to show that A I G I A2 = 0, and A 2G 2A I = 0. Post-multiply both sides of the equation AlGI + A 2G 2 = AF by A = (AIA2). This operation leads to

(AI GI

+ A 2G 2) (AIIA2) = AF A = A = (AIIA2) = (AIGIAI + A2G2AIIAIGIA2 + A 2G 2A 2 ),

°

which gives Al GIAI = and A 2 G 2 A I = 0, and the result is proved. In P 8.2.9, we have seen how a single g-inverse of a matrix A generates all solutions of a consistent system Ax = y of linear equations. In the following result, we demonstrate how a single generalized inverse of A generates all g-inverses of A .

P 8.2.21 Let G be a fixed g-inverse of a given matrix A. Any g-inverse G I of A has one of the following forms. (1) G I = G + U - GAU AG for some matrix U. (2) G I = G + V(I - AG) + (I - GA)W for some matrices V and W . PROOF . The first thing we can check is that AGIA = A if G I has anyone of the forms (1) and (2). Conversely, let G I be a given g- inverse of A. If we wish to write G I in the form (1), take U = G I - G. If we wish to write G I in the form (2), take V = G I - G and W = GIAG. We now introduce a special notation. For any matrix A , let {A-} denote the collection of all g-inverses of A. In the following result we demonstrate that A is essentially determined by the class {A -} of all its g-inverses.

P 8.2.22 Let A and B be two matrices of the same order m x n such that {A-} = {B-}. Then A = B . PROOF. What the condition of the theorem means is that if G is a g-inverse of A then it is also a g-inverse of B and vice versa. Let G be a g- inverse of A and

G I = G + (In - A*(AA* )- A)B* .

278


We note that G I is also a g-inverse of A. For, AGIA = AGA + A(In - A*(AA*)- A)B* A = A

+ (A -

(AA*)(AA*)- A)B* A = A

as A = AA*(AA*)- A by Corollary 8.2.7 using A* in place of A. By the hypothesis of the proposition, G I is also a g-inverse of B. Thus B

= BGIB = BGB + B(In - A*(AA*)- A)B* B = B + B(In - A*(AA*)- A)(In - A*(AA*)- A)* B* B,

since (In - A*(AA*)- A) is Hermitian and idempotent. See P 8.2.8. This implies that B(In - A*(AA*)- A)(In - A*(AA*)- A)* B* B

= o.

Pre-multiplying the above by B*, we have B* B(In - A*(AA*)- A)(In - A*(AA*)- A)* B* B =

o.

Consequently, B* B(In -A*(AA*)- A) = O. (Why?) From this, we have B(In - A*(AA*)- A) = 0, or equivalently, B = BA*(AA*)- A. (Why?) Thus we are able to write B = CA for some matrix C. Following the same line of reasoning, we can write A = DB. By focusing now on a variation of G, given above by G 2 = G + B*(Im - A(A* A)- A*),

one can show that B = AE for some matrix E. In a similar vein, one can show that A = BF for some matrix F. (Try.) Now, B

= BGB = CAGAE = CAE = BE,

which implies DB = DBE = AE = B. Hence B = DB = A. This completes the proof. Suppose G is a g-inverse of A, i.e., AGA = A. It need not imply that A is a g-inverse of G, i.e., GAG = G. We will now introduce a special terminology.


279

DEFINITION 8.2.23. Let A be a matrix of order m x n. A matrix G of order n x m is said to be a reflexive g-inverse of A if AGA = A and GAG = G. We use the notation A; for a reflexive g-inverse of A. We now demonstrate the existence of reflexive g-inverses.

P 8.2.24 exists.

For any matrix A E Mm,n, a reflexive g-inverse of A

PROOF. Let p(A) = a. By the Rank Factorization Theorem, write A = RF with R of order m x a with rank a and F of order a x n with rank a. Let C and D be the right and left inverses of F and R, respectively, i.e., FC = Ia and DR = Ia. Choose G = CD . Note that AGA = RFCDRF = RlalaF = RF = A. On the other hand, GAG = CDRFCD = ClalaD = CD = G. If G is a g-inverse of A, one can show that p(A) ::; p(G). The equality p(A) = p(G) does indeed characterize reflexive g-inverses of A. We demonstrate this in the following proposition.

P 8.2.25 The following statements are equivalent for any two matrices A E Mm ,n and G E Mn ,m. (1) AGA = A and GAG = G. (2) AGA = A and p(A) = p(G). PROOF . (1) :::} (2). The statement AGA = A implies that p(A) ::; p(G) and the statement GAG = G implies that p(G) ::; p(A). Then we must have p(A) = p(G). To prove (2):::}(1), note that p(G) = p(A) (by (2))

::; p(GA) ::; p(G) ,

which implies that p(GA) = p(G). The matrix GA is idempotent. Then by P 8.2.3 , A is a g-inverse of G. The computation of a reflexive g-inverse of A can be done in several ways. Making use of the rank factorization of A is one possibility. Suppose we already have a g-inverse G of A. Let G 1 = GAG. One can check that G 1 is a reflexive g-inverse of A. Every reflexive gTinverse of A can be written in the form GAG for some g-inverse of G of A. Try. It is clear that if G is a g-inverse of A, then p(G) ~ p(A). For a given integer s satisfying p(A) ::; s ::; min{ m, n}, is it possible to find a

280


g-inverse G of A such that p( G) = s? In the following result we answer this question.

P 8.2.26 Let A E Mm,n have the decomposition given by

where P and Q are nonsingular matrices of orders m x m and n x n, respectively, ~ isa diagonal matrix of order a x a with rank a, where a = p(A). Then: (1) For any three matrices E 1 , E 2 , and E3 of appropriate orders,

is a g-inverse of Ai (2) For any two matrices El and E2 of appropriate orders,

is a reflexive generalized inverse of A. PROOF. One simply verifies that they are doing their intended jobs. Note that the matrix G of (1) has the property that Ll-l p(G) = p ( [ E2

Given any integer s such that p(A) ~ s ~ min{m,n}, one can choose E 1 , E 2 , and E3 carefully so that p( G) = s. You can experiment with these matrices. One can directly verify that p(G r ) = a = p(A) .

Complements 8.2.1

Cancellation Laws. Prove the following.

(1) If AB = AC, Sp(B) c Sp(A*), and Sp(C) C Sp(A*), then B=C. (Hint: The matrices Band C can be written as B = A* D and C = A * E for some matrices D and E. Therefore, AA * D =


(2) (3) (4) (5)

281

AA * E or AA * (D - E) = 0 or (D - E) * AA * (D - E) = 0 or A*(D - E) = 0.) (Thus A can be cancelled in AB = AC.) If A E Mm ,n, p(A) = n , and AB = AC, then B = C. If A E Mm ,n, p(A) = m , and BA = CA, then B = C. If ABC = ABD and p(AB) = p(B), then BC = BD. If CAB = DAB and p(AB) = p(A), then CA = DA.

8.2.2 If A = 0, determine the structure of g-inverses of A. 8.2.3 Let I n be a matrix of order n x n in which every entry is equal to unity. Let a i= b and a - b + n = O. Show that (a - b)-l In is a g-inverse of (a - b)In + I n . 8.2.4 Let A be a matrix of order m x n and a and (3 column vectors of orders m x 1 and n x 1, respectively. Let G be any g-inverse of A. If either a E Sp(A) or (3 E Sp(A') , show that

= G _ ((Ga)((3'G))

G 1

(1

+ (3'Ga)

is a g-inverse of A + a(3', provided 1 + (3'Ga i= O. 8.2.5 Let A be a matrix of order n x m with rank rand B a matrix of order s x m with rank m - r. Suppose Sp(A*) n Sp(B *) = {O} . Then (1) A* A + B* B is nonsingular j (2) (A* A + B* B)-l is a g-inverse of A* Aj

(3)

A ~*]

[A~

is nonsingular provided that s = m - r .

8.2.6 Show that a Hermitian matrix has a Hermitian g-inverse. 8.2.7 Show that a non-negative definite matrix has a non-negative g-inverse. 8.2.8 Show that a Hermitian matrix A has a non-negative definite g-inverse if and only if A is non-negative definite. 8.2.9 If G 1 and G 2 are two g-inverses of A, show that aG l + (l-a)G 2 is a g-inverse of A for any a. 8.2.10 If G is a g-inverse of a square matrix A , is G 2 a g-inverse of A2? Explain fully. 8.2.11 Let T = A + XU X' where A is nnd, U is symmetric such that Sp(A) C Sp(T) and Sp(X) C Sp(T). Then Sp(X'T- X) = Sp(X') for any g-inverse T- .

282


8.2.12 Let A X = B and X C = D be two consistent system of matrix equations in the unknown matrix X. Show that they have a common solution in X if and only if A D = B C . If this condition holds, show that X = A-B+DC- -ADCis a common solution.

8.3. Geometric Approach: LMN-inverse In defining a g-inverse G of A 6 M,,, in Section 8.2, emphasis has been laid on its use in providing a solution to a consistent system Ax = y of linear equations in the form x = Gy. A necessary and s a c i e n t condition for this is that the operator AG is an identity on the subspace Sp(A) c Cm. Nothing has been specified about the values of AGy or Gy for y 6 Sp(A). The basic theme of this section is that we want to determine a g-inverse G E M,,, such that Sp(G) is a specified subspace in Cn and the kernel of G is a specified subspace in Cm. The concept of g-inverse introduced in Section 8.2 makes perfect sense in the realm of general vector spaces and transformations. Let V and W be two vector spaces and A a transformation from V and W. (We retain the matrix symbol A for transformation too.) A transformation G from W to V is said to be a generalized inverse of A if AGA = A. The entire body of results of Section 8.2 carries over to this general framework. In this section, it is in this general framework we work on.

+ V be a g-inverse of a transformation A + W , i.e., satisfies the equation AGA = A. Define P 8.3.1 Let G : W

V

:

where R denotes the range and K the kernel of a transformation. Then: (1) A N = 0, N A = 0. (2) M n K(A) = (0) and M $ K(A) = V. (3) L n R(A) = (0) and L $ R(A) = W.

PROOF.(1) is clearly satisfied. To prove (2) let a: E V , belong to both M and K(A). Then, obviously Ax = 0, and x = (G - N ) y for some y. Then A(G - N ) y = AGAGy = AGy = 0. This implies that 0 = GAGy = (G - N ) y = x . Hence M n K(A) = (0). Let


283

x E V. Obviously X = GAx + (1 - GA)x, where (1 - GA)x E K(A) since A(1 - GA)x = 0, and GAx = (G - N)Ax E R(G - N). Thus M $ K(A) = V. (3) is proved in an analogous manner. Now we raise the following question. Let MeV be any given complement of K(A), LeW be any given complement of R(A), and N be a given transformation such that N A = 0, AN = o. Does there exist a g-inverse G of A such that the conditions (8.3.1), i.e., N = G - GAG, M = R(G - N), L = K(G - N) hold? The answer is yes and the following propositions give a construction of such an inverse and also establish its uniqueness. We call this inverse the LMN-inverse, which was proposed in Rao and Yanai (1985).

P 8.3.2 Let A be a transformation from a vector space V into a vector space W. Let L be a subspace of W such that L n R(A) = {O} and L $ R(A) = W. Let M be a subspace of V such that M n K(A) = {O} and M61K(A) = V. Let N be a transformation from W to V such that AN = 0 and N A = O. Then there exists an LMN-inverse. PROOF.

Let H be any g-inverse of A. Then we show that G

= PM .K(A)HPR(A).L + N

(8.3.2)

is the desired LMN-inverse, where PM oK(A) (abbreviated as PM in the proof) is the projection operator on M along K(A) and PR(A) oL (abbreviated as P A) is the projection operator on R(A) along L. Consider x(E V) = xdE M) + x2(E K(A)). Then APMX

= Ax! = Ax for every x

E V ::} APM

= A.

(8.3.3)

Obviously, (8.3.4) It is easy to verify using (8.3.3) and (8.3.4) that AGA = A, where G is as in (8.3.2), i.e., G is a g-inverse of A. Let x E M and write HAx = x + Xo. Then AHAx = Ax + Axo ::} Axo = 0 since AHAx = Ax, i.e., Xo E K, and PMHAx = PMX + PMXO = x . Then using (8.3.4) (8.3.5)

284


so that any x E M belongs to R{PMHP A) = R{G - N). Obviously any vector in R{PMHP A) belongs to M, which shows that (8.3.6)

R{G-N) = M.

Let l{E W) = ll{E R{A)) + 12{E L) = Am + 12 for some m . Then (G - N)l = PMHP A{Am + 12) = PMHAm = 0 ~ APMHAm = O. From (8.3.3) , APM = A so that APMH Am = AH Am = Am which is zero. Hence l = 12 E L . Further, for any l E L, (G - N)l = O. Hence

= L.

(8.3.7)

N=G-GAG

(8.3.8)

K{G - N) It is easy to verify that

so that all the conditions for an LMN-inverse are satisfied, and P 8.3.2 is proved. Now we settle the uniqueness problem. P 8.3.3 Let A be a transformation from V to W. The LMNinverse of A, which is a transformation from W to V is unique. Let G 1 and G 2 be two transformations satisfying the conditions (8.3.1). Note that R{G i - N) = M ~ (G i - N)y E M for all yEW and i = 1,2. Taking the difference for i = 1 and 2, (8.3.9)

Similarly K(G i - N) = L ~ {G i - N)y Taking the difference for i = 1 and 2 {G 1

-

G 2 )y = {G 1

A(G 1

-

G 2 )y = A{G 1

= (G i

-

N)P AY for yEW.

G 2 )P AY

-

G 2 )P AY = 0 since PAY = Az for some z

-

~ (G 1

-

G 2 )y E K{A).

Since M n K{A) = 0, (8.3.9) and (8.3.10) ~ {G 1 yEW. Hence G 1 = G 2 .

(8.3.1O) -

G 2 )y

=

0 for all


285

Complements 8.3.1 Let L, M, N be as defined in P 8.3.2. Then the LMN-inverse G has the alternative definitions:

(1) AG = PR(A) .L, GA = PM ' K(A), G - GAG = N. (8.3.11) (2) AGA = A, R(G - N) = M, K(G - N) = L. (8.3.12) 8.3.2 Let G be any g-inverse of A and M is any complement of K(A) in V. Then the transformation AIM: M -+ R(A) is bijective, and the true inverse (AIM)-l : R(A) -+ M exists.

FIGURE. LMN-inverse A:V-+W AIM

GIR(A) M=R(G-N)

R(A)

"

.- '

K(A) N

G:W-+V The figure given above shows the range and kernel spaces of the given transformation A and any associated g-inverse G for given subspaces L, M and a transformation N. P 8.3.4 shows that all generalized inverses can be generated by different choices of L, M and N .

286


P 8.3.4 Let A be a transformation from a vector space V to a vector space W. Let F be the collection of all entities (L, M, N) satisfying the following: (1) LisasubspaceofWsatisfyingLnR(A) = {O} andLEElR(A) = W. (2) M is a subspace of V satisfying M n K(A) = {O} and M EEl K(A) = V . (3) N is a transformation from W to V such that N A = 0 and AN=O. Let G be the collection of all g-inverses of A. Then there exists a bijection from F to G. Given L, M, and N satisfying (1) , (2), and (3), let G LMN be the unique LMN-inverse of A. The map 1(L, M, N) = GLMN is a bijection from F to G. This proves P 8.3.4. The case N = 0 is of special interest. The conditions (8.3.11) and (8.3.12) reduce to PROOF.

AG =

G = GAG

(8.3.13)

AGA = A, R(G) = M, K(G) = L.

(8.3.14)

PR(A).L,

GA =

PM.K(A),

and In other words, when N = 0, G is a reflexive g-inverse of A with range M and kernel L. This special case deserves a mention. COROLLARY 8.3.5. Let A be a transformation from a vector space V to a vector space W. Let Lr be the collection of all entities (L, M) with the following properties: (1) L is a subspace ofW such that LnR(A) = {O} and LEElR(A) = W. (2) M is a subspace of V such that M n K(A) = {O} and M EEl K(A) = V. Let G r be the collection of all reflexive g-inverses of A. Then there exists a bijection from Lr to Gr. A special case of this is of interest. Assume that the vector spaces V and Ware equipped with inner products. We take L to be the orthogonal complement of R(A) and M to be the orthogonal complement of K(A). Let PM be the orthogonal projection from V onto M and


287

the orthogonal projection from W onto R(A) . Take N = O. The conditions (8.3.13) and (8.3.14) reduce to PR(A)

AG

= PR(A) , GA = PM, GAG = G

(8.3.15)

= A, R(G) = M, K(G) = L,

(8.3.16)

and AGA

respectively. If the projection operators are self-adjoint, the equivalent sets (8.3.15) and (8.3.16) of conditions reduce to AGA

= A, GAG = G, AG = (AG)*, GA = (GA)*,

(8.3.17)

where (AG)* is the adjoint of the transformation AG in the inner product space V and (GA)* is the adjoint of the transformation GA in the inner product space W. We now introduce a definition. DEFINITION 8.3.6. Let A be a transformation from an inner product space V to an inner product space W. A transformation G from W to V is said to be a Moore-Penrose inverse of A if (8.3.17) holds. The existence of Moore-Penrose inverse is guaranteed and it is unique. It is after all an LMN-inverse for some suitable choice of L, M, and N. Thus the concept of LMN-inverse is more general than the MoorePenrose inverse. Another special case of interest is when we work with matrices. Let A E Mmn , and N E Mnm , be such that AN = 0 and NA = O. Take V = en and W = em. We now specify some subspaces of V and W. Let B be a matrix of order m x p such that Sp(B) n Sp(A) = {O} and Sp(B) EB Sp(A) = W . Let C be a matrix of order q x n such that K(C)nK(A) = {O} and K(C)EBK(A) = V. The subspaces under focus are Sp(B) and K(C). In the terminology of Land M, M = K(C) and L = Sp(B). We now seek a matrix G E Mn,m such that

AGA

= A, Sp(G - N) = K(C), K(G - N) = Sp(B) .

An explicit form of G is given by

288


8.4. Minimum Norm Solution

In this section, we take up again the problem of solving a system Ax = y of consistent linear equations in the unknown x . The entity A

is a transformation from an inner product space V to a vector space W. For the equation to be consistent, we ought to have y E R(A). Let < ., . > be the inner product and II· II the associated norm on V. The problem we focus on is: minimize

IIxll

over the set {x E V : Ax = y}.

(8.4.1)

The following proposition solves this problem. P 8.4.1 Let PK(A) be the orthogonal projection from V onto K(A), the null space of A. Let G be a transformation from W to V such that GA = J - PK(A) .

(8.4.2)

(We will settle the existence of G a little later.) Then x(n) = Gy solves the minimization problem (8.4.1) . (We use the superscript (n) to indicate that the vector x(n) is a minimum norm solution.) PROOF. Since y E R(A), y = Ab for some bE V. Then

x(n)

= Gy = GAb = (I -

PK(A»)b,

(8.4.3)

which means that x(n) E [K(A)J.l, the orthogonal complement of K(A) . Note that G is a g-inverse of A, i.e., AGA = A. Let x E V, and write x = Xl +X2 with Xl E K(A) and X2 E [K(A)J.l. Then Ax = Ax! +AX2 = AX2. Further, AGAx = A(J -PK(A»)X = A(X-PK(A)X) = A(x-xr) = AX2 = Ax. Hence AGA = A. Since G is a g-inverse of A , Gy = x(n) is a solution to the system of equations. Now let x be any solution to the system of equations. It is clear that x - x(n) E K(A) . Consequently,

<x

- x(n) ,x(n)

>=

o.

(8.4.4)

Observe that

IIxl1 2=< x, x > =< x =< =

x(n)

+ x(n), x _

x - x(n), x - x(n)

x(n)

+ x(n) >

>+<x

_ x(n), x(n)

+ < x(n) ,x - x(n) > + < x(n) ,x(n) > Ilx - x(n) 112 + Ilx(n) 112 2: Ilx(n) 112

>

289


using (8.4.4) in the last but one step. This completes the proof. There is another way to look at the transformation I - PK(A). Let A # be the adjoint of :A. Then I - PK(A) is the orthogonal projection from V onto R(A#). In our suggestive notation, 1- PK(A) = PR(A#) . The existence of G is not a problem. Look back at (8.3.5). The operator PM is precisely our I - PK(A) here. There is a G satisfying (8.3.5). Put any inner product on the vector space W. We could call G a minimum norm inverse. A minimum norm inverse could be defined in general vector spaces not necessarily equipped with inner products. Let A be a transformation from a vector space V to a vector space W. Let M be any subspace of V which is a complement ofK(A), i.e., MnK(A) = {O} and MEBK(A) = V. Let PM.K(A) be a projector from V onto M along K(A). One could define a transformation G from W to Vasa minimum norm inverse if GA = PM .K(A) . Such a G is necessarily a g-inverse of A. (Prove this.)

8.5. Least Squares Solution Let A be a transformation from a vector space V into an inner product space W. Let < .,. > be the inner product on Wand II . II the corresponding norm. Let yEW and Ax = y be a system of equations possibly inconsistent. We focus on the problem: minimize Ily - Axil over all x E V.

(8.5.1)

If the system Ax = y is consistent, any solution of the system, obviously, solves the problem (8.5.1). It is inconsistency that poses a challenge.

P 8.5.1 Let PR(A) be the orthogonal projection from W onto R(A). Let G be any transformation from W to V such that AG =

(8.5.2)

PR(A) .

(Is the existence of G a problem?) Then x(l) = Gy solves the problem (8.5.1). (We use the suggestive superscript l to indicate that x(l) is a least squares solution.) PROOF. Note that y - Ax(l) =

Y -

AGy = (I -

PR(A))Y E

[R(A)J.l.

290


Since A(x U') - x) E R(A) for any x E V, we have

< A(x(l)

- x) , y - Ax(l)

>= O.

(8.5.3)

For any x E V,

Ily -

Axl12

=< Y - Ax,y - Ax> =< (y - Ax(l») + A(x(l) - x), (y =< Y _ AX(l) , y - Ax(l) > + < y -

Ax(l»)

+ A(x(l)

- x) >

AX(l) , A(x(l) - x) >

+ < (x(l) - x), y - Ax(l) > + < A(x(l) - x), A(x(l) = lIy - AX(l) 112 + IIA(x(l) - x)1I2 ~ lIy - AX(l) 112.

- x) >

This completes the proof. The label that x(l) is a least squares solution stems legitimately. Suppose A = (aij) is a matrix of order m x n complex entries. Let y' = (Yl, Y2, ... , Ym) E em. Let the vector space em be equipped with the standard inner product. Then for x' = (Xl,X2, '" ,xm ) E em, m

i=l

This expression appears routinely in many optimization problems. In the context of Linear Models in Statistics, Yi is a realization of a random variable Vi in the model and (ailxl + ai2X2 + ... + ainXn) its expected value. The difference (Yi - (ailxl + ai2x2 + ... + ainXn)) is labelled as the error. The objective is to minimize the sum of squared errors with respect to Xi'S. Any solution to this problem is called a least squares solution. The transformation G satisfying AG = PR(A) could be labelled as a least squares solution. Such a G is necessarily a g-inverse of A . This notion can be defined in general vector spaces without any reference to inner products. Let A be a transformation from a vector space V to a vector space W. Let L be any subspace of W which is a complement of R(A), i.e., R(A)

nL

= {O} and R(A) E9 L = W.

Let PR(A) "L be a projector from W onto R(A) along L . We say that a transformation G from W to V a least squares inverse if AG = PR(A) .L '


291

8.6. Minimum Norm Least Squares Solution Let A be a transformation from an inner product space {V, < -, - >d to an inner product space (W, < .,. >2) with the corresponding norms II . III and 11·112. Let YEW. We focus on the problem: minimize {x(l) E V :

Ily -

Ilx(i)lIl

over the set

Ax(l) 112 = min Ily xEV

-

AxIl2}.

(8.6.1)

First, we gather all the least squares solutions x(l) each of which minimizes lIy - Axl12 over V. Among these solutions, we seek one x(i) for which IIx(l) III is minimum. p 8.6.1 Let PR(A) be the orthogonal projection from W onto R{A) and PK(A) the orthogonal projection from V onto K{A). Let G be a transformation from W to V such that AG

= PR(A),

GA

= (I -

PK(A»),

GAG

= G.

(8.6.2)

(Is the existence of G a problem? Look up (8.3.6) .) Then x(nl) = Gy solves the problem (8.6.1). PROOF.

Let x(i) be any solution to the problem: minimize

Ily -

Axlll over all x E V.

(8.6.3)

From the proof of P 8.5.1, it is clear that x(l) satisfies the equation AX(l) = PR(A)Y.

(8.6.4)

The conditions (8.6.2) subsume the conditions (8.5.2). Consequently, x(nl) is also a solution to the minimization problem (8.6.3). By (8.6.4), AX(l) - Ax(nl)

= PR(A)Y -

PR(A)Y

=0

from which we have x(i) - x(nl) E K{A). Observe that x(ni)

= Gy = GAGy = GA{Gy)

E R{GA)

= R{J -

PK(A»).

292


Since [ - PK(A) is an orthogonal projection from V onto [K(A)j1·, we have x(nl) E [K(A)J-L, from which it follows that

< x(l) Thus for any solution

x(l)

_ x(nl) ,x(nl)

>= O.

of the minimization problem (8.6.3),

Ilx(l) II~ =< x(l), x(l) >=< x(nl) + (x(l) _ x(nl)), x(nl) + (x(l) _ x(nl)) > = Ilx(nl) II~+ < x(nl), (x(l) _ x(nl)) > + < (x(l) _ x(nl)), x(nl) > + Ilx(l) - x(nl) II~ = Ilx(nl) II~ + IIx(l) - x(nl) II~ ~ Ilx(nl) II~. This completes the proof The transformation G satisfying (8.6.2) can be called as a minimum norm least squares solution. This notion can be introduced in general vector spaces without involving inner products. Let A be a transformation from a vector space V to W. Let M be a complement of the subspace K(A) in V and L a complement of R(A) in W. Let PM .K(A) and PR(A) .L stand for the projectors in the usual connotation. A transformation G from W to V is said to be a minimum norm least squares inverse if AG = PR(A).L, AG = PM .K(A), GAG = G. 8.7. Various Types of g-inverses

We have come across a variety of g-inverses in our sojourn. It is time to take stock of what has been achieved and then provide a summary of the deliberations. In the following V and W stand for vector spaces. We look at three possible scenarios. Scenario 1 (1) V and Ware general vector spaces. (2) A stands for a transformation from V to W. (3) G stands for a transformation from W to V. (4) R(A) and K(A) stand for the range and kernel of the transformation A, respectively. (5) M stands for any subspace of V which is a complement of the subspace K(A) of V, i.e., M n K(A) = {O} and M E9 K(A) = V.


293

(6) PM .K(A) stands for the projector from V onto M along K(A). (7) L stands for any subspace of W which is a complement of the subspace R(A). (8) PR(A) .L stands for the projector from W onto R(A) along L.

Scenario 2 (1) V and W stand for general inner product spaces. (2), (3) and (4) are the same as those under Scenario l. (5) A # stands for the adjoint of the transformation A . (The map A # is a transformation from W to V.) (6) PR(A) stands for the orthogonal projection from W onto R(A). (7) P[K(A)J-l stands for the orthogonal projection from V onto [K(A)].l. (8) PR(A#) stands for the orthogonal projection from V onto R(A#) . Scenario 3 (1) V and Ware unitary spaces, i.e., V = cn and W = C m , equipped with their usual inner products. (2) A is a matrix of order m x n. (3) G is a matrix of order n x m . (4) R(A) is the vector space spanned by the columns of A and K(A) the null space of A. (5) A* is the conjugate of A. (6) PA' = A* MiA, where Mi is any matrix satisfying (AA*)Mi(AA*) = AA*. (7) PA = AM2A*, where M2 is any matrix satisfying (A* A)M2(A* A) = A*A.

Complements 8.7.1 Under scenario 1, we have defined the LMN-inv~rse in Section 8.3. (1) Show that the LMN-inverse G can also be characterized as follows GIR(A) = (AIM)-i and GIL = NIL .

[Note that AIM: M -+ R(A) is bijective so that (AIM)-i R(A) -+ M is well defined and unique.] (2) The LMN-inverse can also be defined as Gy = (AIM)-lYi

where Y(E W)

+ NY2

= ydE R(A)) + Y2(E L) .

294


(3) If instead of N, suppose we are given R(G - GAG), an M and an L. Show that G is not unique in such a case. In the following table, we provide a summary of the properties that various types of g-inverses should satisfy under each of the scenarios. TABLE. A catalogue of different types of g-inverses Description of G

Scenario 1

Scenario 2

Scenario 3

g-inverse

AGA=A

AGA=A

AGA=A

r-inverse

GAG=G

GAG=G

GAG=G

r , g-inverse

AGA=A GAG=G

AGA=A GAG=G

AGA=A GAG=G

min norm g-inverse

GA= PM -K(A)

GA = PR(A#)

GA=PA·

min norm r, g-inverse

GA= PM -K(A) GAG=G

GA= PR(A# ) GAG=G

GA = PA. GAG=G

least squares g-inverse

AG = PR(A) -L

AG=PA

AG=PA

least squares r, g- inverse

AG = PR(A) -L GAG=G

AG=PA GAG=G

AG=PA GAG=G

pre min norm least squares g-inverse

AG = PR(A) -L GA=PM -K(A)

AG=PA GA == PA#

AG=PA GA = PA.

min norm least squares g-inverse

AG = PR(A) -L GA= PM -K(A) GAG=G

AG=PA GA=Pc

AG=PA GA=PG

(1)

(2)

g-inverse=generalized inverse, r-inverse=reflexive inverse _ Equivalent conditions for (1) are

AGA=A, GAG=G , (AG)# = AG, (GA)# = GA Equivalent conditions for (2) are

AGA = A, GAG = G , (AGr = AG, (GAr = GA

Gen eralized Inverses

295

Some comments are in order on the above table. (1) If G is a minimum norm inverse of A, then it is also a g-inverse of A . (2) If G is a least squares inverse of A, then it is also a g-inverse of A. (3) Suppose A is a matrix of order m x n and rank a. In Section 8.2, we presented the structure of a g-inverse G of A. Let

be singular value decomposition of A , where P and Q are unitary matrices of orders m x m and n x n, respectively, and ~ is a diagonal matrix of order a x a with p(A) = a. Then a g-inverse G of A is of the form

for any arbitrary matrices E 1 ,E2 , and E 3 . A reflexive g-inverse G of A isOf the form

for any matrices El and E 2 . A matrix G of order n x m is a minimum norm g-inverse of A if G is of the form

for any matrices El and E 3 . A matrix G of order n x m is a minimum norm and reflexive g-inverse of A if G is of the form

for any matrix E 1 . A matrix G of order n x m is a least squares g- inverse of A if G is of the form

296


for any matrices E2 and E 3 . A matrix G of order n x m is a least squares and reflexive inverse of A if G is of the form

for any matrix E 2 • A matrix G of order n x m is a pre-minimum-normleast-squares-inverse of A if G is of the form b. -1

G-Q [0

for any matrix E 3 . A matrix G of order n x m is a minimum norm least squares inverse of A, i.e., Moore-Penrose inverse of A if G is of the form

B.B. G-inverses Through Matrix Approximations

Given a matrix A of order m x n with complex entries, there may not exist a matrix G of order n x m such that GA = In . In such an event, we may try to find a matrix G such that AG and G A are close to 1m and In , respectively. Such a G may be called an approximate inverse of A. The underlying theme of this section is to pursue this idea of determining approximate inverses of A. It turns out that the g-inverses introduced and examined above are after all approximate inverses in some sense. (See Rao (1980) .) In order to develop the theory of approximate inverses, we need a general criterion to decide which one of the two given matrices is closer to the null matrix. DEFINITION 8.8.1. Let Sand R be two matrices of the same order m x n. Assume that R is closer to a null matrix than S if SS* ~ RR* or S" S 2: R* R. Assume that R is strongly closer to a null matrix tlian S if S S" 2: RR* and S* S 2: R* R. Some comments are in order on the definition.

(1) The notation C ~L D (or C ~ D for convenience of notation) for two matrices C and D means that C - D is non-negative definite. (The subscript stands for Lowner ordering.)


297

(2) If SS* ~ RR*, it does not imply that S* S ~ R* R. (3) Let O"dS) ~ 0"2(S) ~ ... ~ O"t(S) ~ 0 and O"dR) ~ 0"2(R) ~ .. . 2: O"t(R) ~ 0 be the singular values of Sand R, respectively, where t = min{m,n}. If O"i(S) ~ O"i(R) for all i, it does not follow that SS* ~ RR* or S* S ~ R* R. But the reverse is true. (4) If SS* ~ RR* or S* S ~ R* R, then it follows that IISII ~ IIRII for any unitarily invariant norm II . II on the space of all matrices of order m x n. We are jumping the gun again. We will not come across unitarily invariant norms until Chapter 11. The converse is not true. Thus the concept of closeness in the sense of Definition 8.6.1 is stronger than having a smaller norm. We are now ready to establish results on matrix approximations via g-inverses. P 8.8.2 Let A be a matrix of order m x nand G be any g-inverse of A, and L a least squares inverse of A . Then

PROOF. Let PA be the orthogonal projection from W = em onto R(A). We have seen that AL = PA. This means that AL is idempotent and Hermitian. Further, A* AL = A*. (Why?) We check that

(AL - AG)*(Im - AL) = AL(Im - AL) - G* A*(Im - AL) =O-G*(A* -A*AL) =0. Finally,

(Im = (Im = (Im + (Im

AG)*(Im - AG) AL + AL - AG)* (Im - AL + AL - AG) AL)*(Im - AL) + (AL - AG)*(lm - AL) - AL)*(AL - AG) + (AL - AG)*(AL - AG)

= (Im - AL)*(Im - AL) ~

(1m - AL)*(Im - AL).

+ 0 + 0 + (AL -

AG)*(AL - AG)

Why?

This completes the proof. We discuss some implications of this result. The matrix 1m - AL is closest to the null matrix among all matrices 1m - AG, with G being

298


a g-inverse of A. Further, for any unitarily invariant norm space of all m x m matrices,

II . II

on the

minllIm - AGII = 111m - ALII, G

where the minimum is taken over all g-inverses G of A. In particular, specializing in the Euclidean norm on the space of all matrices of order m x m, we have min[Tr(Im - AG)*(Im - AG)] = Tr[(Im - AL)*(Im - AL)] G

= Tr(Im - AL).

(Why?)

Note that for any matrix C of order m x m its Euclidean norm is defined by [Tr( C* C)]1/2 . The following results can be established analogously. P 8.8.3 For a given matrix A of order m x n, let M be any minimum norm inverse of A . Then for any g-inverse G of A, (In - GA)*(In - GA) 2 (In - MA)*(In - MA),

so that In - M A is the closest to the null matrix among all matrices In - GA with G being a g-inverse of A. P 8.8.4 For a given matrix A of order m x n, let Q be any preminimum-norm-least-squares-inverse of A. Then for any g-inverse G of A, (Im - AG)*(Im - AG) 2 (Im - AQ)*(Im - AQ) and (In - GA)*(In - GA) 2 (In - QA)* (In - QA).

We now focus on the stronger notion of closeness. Recall that a matrix C is strongly closer to a null matrix than a matrix D if CC* 2 DD* and C*C 2 D* D. P 8.8.5

Let A be a matrix of order m x n. The following hold.

(1) Let L be any least squares inverse as well as a reflexive inverse of A. Then for any g-inverse G of A, (Im - AG)*(Im - AG) 2 (Im - AL)*(Im - AL), (Im - AG)(Im - AG)* 2 (Im - AL)(Im - AL)*,


299

i.e., 1m - AL is strongly closest to a null matrix among all matrices 1m - AG with G being a g-inverse of A . (2) Let M be any minimum norm inverse as well as a reflexive inverse of A. Then for any g-inverse G of A,

(In - GA)*(In - GA) ;::: (In - MA)*(In - MA), (In - GA)(In - GA)* ;::: (In - MA)(In - MA)*, i.e., In - M A is strongly closest to a null matrix among all matrices In - GA with G being a g-inverse of A. (3) Let Q be a Moore-Penrose inverse of A. Then for any g-inverse G of A,

(Im - AG)*{Im - AG) ;::: (Im - AQ)*(Im - AQ), (Im - AG)(Im - AG)* ;::: (1m - AQ)(Im - AQ)*, (In - GA)*(In - GA) ;::: (In -

Q~)*{In -

QA),

(In - GA)(In - GA)* ;::: (In - QA)(In - QA)*, i.e., both 1m - AQ and In - QA are strongly closest to a null matrix among all matrices 1m - AG and In - GA, respectively, with G being a g-inverse of A. PROOF. A proof can be hammered out by imitating the theme in the proof of P 8.8.2. All these results can be restated in the framework of unitarily invariant norms in the manner presented right after the proof of P 8.8.2.

Complements 8.8.1

Let A=

[~ o1 ~]

and B =

[~ o1

Show that AA' 2: BB' but A' A > B' B is not true. Compare the singular values of A and B. 8.8.2 Construct two matrices A and B such that the i-th singular value of A is greater than or equal to the i-th singular value of B for every i, but neither AA* 2: BB* nor A* A ;::: B* B.

300


8.9. Gauss-Markov Theorem The focus in this section is on a random vector Y consisting of m real components with some joint distribution which, among other things, depends on a parameter vector f3 E R n and a scalar a E R+ = (0,00) in the following way,

E{3,u(Y) = Xf3, D{3,u(Y) = E{3,u(Y - X(3)(Y - X(3)' = a21m,

(8.9.1)

where E stands for expectation and D for dispersion matrix, X is a matrix of order m x n with known entries. The model (8.9.1) specifying the mean, E(Y) and dispersion matrix, D(Y) (variance-covariance) matrix is called the Gauss-Markov model and is represented by the triplet (Y, X f3, a 2 1m). This model is widely used in statistics and the associated methodology is called the regression theory. The problems usually considered are the estimation of the unknown parameters f3 and a and tests of hypotheses concerning them. We have touched upon this model earlier but now we show how the concepts of projection and g-inverses are useful in solving these problems. We begin the proceedings with a definition. 8.9.1. Let f(f3) = Qf3, f3 ERn, where Q is a given matrix of order k x n. A statistic CY with C of order k x m is said to be a Linear Unbiased Minimum Dispersion (L UMD) estimator of f (.) if DEFINITION

E{3,u(CY) = f(f3), D{3,u(FY) 2: D{3,u(CY) for all f3 E Rn and a E R +, where FY is any statistic satisfying E{3,u(FY) = f(f3) for all f3 E Rn and a E R+. The parametric function Qf3 is a set of k linear functions of the components of f3. The statistic CY is a set of k linear functions of the components of Y. The estimator CY is LUMD if it is an unbiased estimator of the parametric function and has the least dispersion matrix among all linear unbiased estimators of the parametric function, i.e., that D{3,u(CY) is closest to a null matrix among all matrices D{3,u(FY) for all f3 ERn and a E R +, where FY is an unbiased estimator of f (. ). There is no guarantee that we will have at least one statistic FY which is an unbiased estimator of f (.). The following result answers some of the questions one confronts in the realm of Gauss-Markov models.


301

P 8.9.2 Let P be the orthogonal projector from Rm onto R(X) under the inner product < u, v > = u'v for u. v E R m. The following statements hold. (1) There exists an unbiased estimator of f('), i.e., f(·) is estimable, if and only if Sp(Q') c Sp(X'), i.e., Q = AX for some matrix A. (2) If f{-) is estimable, i.e., Q = AX for some matrix A, APY is the LU M D of f (-). Further,

D/3,u(APY) = a 2 APA' (3) Let g(a) g(.) is given by

= a 2 ,a

for all

/3 ERn and a

E R+ . If p(X)

= r,

E R+.

(8.9.2)

an unbiased estimator of

[l/(m - r)]Y'{I - P)Y.

(8.9.3)

PROOF. (1) Suppose AY is an unbiased estimator of f( ·). means that for every /3 ERn and a E R + ,

This

Q/3 = E{3,u(AY) = AX/3, which implies that Q = AX. Conversely, if Q = AX for some matrix A, the AY is an unbiased estimator of f (.). (2) Suppose f(·) is estimable. Then Q = AX for some matrix A. Further,

E/3,u(APY)

= APX/3 = AX/3 = Q/3

for all /3 and a. This demonstrates that APY is an unbiased estimator of f (.). Let FY be an alternative unbiased estimator of f (. ). This means that FX = Q = AX which implies that FP = AP. Now for all /3 and a,

D/3,u(FY) - D{3,u(APY) = a 2 FF' - a 2 APP' A' = a 2 (FF' - FPP'F') = a 2 (FF' - FPF'), 2

since PP' = P, 2

= a F(I - P)F' = a F(I - P)(I - P)' F',

which is clearly non-negative definite; D{3,u(APY)

= a 2 AP A.

302


(3) Note that for all (3 and a ,

Ep,u[Y'(I - P)Y] = Ep,u [(Y - X(3)'(I - P)(Y - X(3)] = E p,u[Tr(I - P)(Y - X(3)(Y - X(3)'] =

Tr[(I - P)Ep ,u(Y - X(3)(Y - X(3)']

=

a 2 Tr(I - P) = a 2 (m - Tr(P))

= a 2 (m - p(P))

= a 2 (m - r),

from which the result (3) follows. This completes the proof. (What about uniqueness?) We state some of the consequences of the main result. COROLLARY 8.9.3 . If Q = X , then PY is the LUMD of X{3. COROLLARY 8 .9.4. If Q = X , the least squares estimator of X{3 is

PY . COROLLARY 8.9.5. Suppose f(·) is estimable. The LUMD estimator of f(-) is given by Q/J, where /J = GX'Y and G is a g-inverse of X' X. Further, Dp ,u(Q/J) = a 2 QGQ' for all (3 and a. PROOF. To establish the result, we make use of the explicit representation of the projection operator P, namely, P = XGX' , where G is any g-inverse of X'X. Now the LUMD estimator of f(·) is given by

APY = AXGX'Y = QGX'Y = Q/J in our new terminology. Further, for all (3 and a,

Dp ,u(Q/J) == a 2 APA'

= a 2 AXGX'A' = a 2 QGQ' .

The expression /J is not unique when X' X is a singular and depends on the particular choice of the g- inverse G used. However, Q /J and QG Q' are unique for any choice of g-inverse G provided f(·) is estimable. COROLLARY 8.9.6. The unbiased estimator of g(-) given in P 8.9.2 (3) can be rewritten as

(m - r)-lY'(I - P)Y = (m - r)-l(y'Y - /J' X'Y). Note: The material for this Chapter is drawn from the references: Rao(1945a, 1945b, 1946b, 1951 , 1955, 1968, 1971, 1972b, 1973a, 1973b, 1973c, 1975, 1976a, 1976b, 1978a, 1978b, 1979a, 1980, 1981, 1985b), Rao and Mitra (1971a, 1971b, 1973, 1975), Rao, Bhimasankaram and Mitra (1972) and Rao and Yanai (1985a, 1985b).

CHAPTER 9 MAJORIZATION

In this chapter we introduce the notion of majorization and examine some of its ramifications. We also focus on the role of doubly stochastic matrices in characterizing majorization. The chain of ideas presented here will be made use of in understanding the relationship between the eigenvalues and singular values of a square matrix. 9.1. Majorization For any n given numbers Xl, X2, ... ,Xn , let x(1) ~ X(2) ~ ... ~ x(n) be the arrangement of Xl, X2, ... ,Xn in decreasing order of magnitude. Let X = (Xl, X2, . .. ,Xn )' and Y = (Yt, Y2, . .. ,Yn)' be two vectors in R n . Keeping up with the statistical tradition, we identify members of Rn as column vectors. DEFINITION 9.1.1. We say that the vector vector Y (or Y majorizes x), and use the symbol X(1)+ ... +X(i) ::;Y(I)+ ... +Y(i), Xl

X

is majorized by the y, if

X«

i=I, ... ,n-l,

+ X2 + ... + Xn = YI + Y2 + ... + Yn·

(9.1.1) (9.1.2)

REMARKS.

A. The relationship

«

defined above has the following properties.

(1)

X«X for every X in Rn. (2) If X « Y and Y « z, then x«z. (3) The properties stated in (1) and (2) above make the relation « only a pre-order. The crucial element missing for the relation « to be a partial order on Rn is that it does not have the property that x«y and y«x do imply that X = y. However, one can show 303


304

that the components of Y is a permutation of the components of x. See (4) below for a precise statement. (4) Let 7r be a permutation on {I, 2, ... ,n}, i.e., 7r is a one-toone map from {I, 2, . .. ,n} onto {I, 2, . . . ,n}. For x in R n, let X-rr = (X-rr(I),X-rr(2), ... ,x-rr(n»)" The components of X-rr are just a permutation of the components of x. For any x in R n, x«x-rr for every permutation 7r of {I, 2, ... ,n}. The above property can be rephrased as follows. With every permutation 7r of {I, 2, ... ,n}, one can associate a permutation matrix P-rr = (P-rrij) of order n x n defined as follows for 1 ~ i, j ~ n: if 7r(i)=j, otherwise. Only one entry of every row and column of P-rr is unity and the remaining (n-l) entries consist of zeros. One can verify that for any x in Rn and any permutation 7r of {I, 2, ... ,n}, X-rr = P-rrx. Thus we have x«P-rrx for every permutation 7r of {I, 2, ... ,n}. We will elaborate the significance of this observation in the next section. Suppose x«y and y«x. Then one can verify that y = X-rr for some permutation 7r of {I, 2, . .. ,n}. B. The notion of majorization introduced above can be described in a slightly different fashion. For any x = (X},X2,'" ,xn )' in R n , let X[I) ~ X[2) ~ ••. ~ X[n) be the arrangement of Xl> X2, •.. ,X n in increasing order of magnitude. One can show that x«y if and only if X[I)

+ ... + Xli) 2: Y[I) + ... + Y[i» i = 1, ... ,n Xl + X2 + ... + Xn = Yl + Y2 + ... + Yn'

To prove this assertion, one could use the fact that i = 1,2, ... ,no

Xli)

=

1,

X(n-i+l)

for

C. The notion of majorization can be described in yet another form. For

x

and

Y

in

Rn,

let
=

L

XiYi.

For any subset I of

i=l

{I, 2, ... ,n}, let £[ be the column vector in Rn whose i-th entry is given by if i E I, otherwise.

M ajorization

305

For example, if I = {I, 2}, then £] = (1,1,0,0, ... ,0)'. Let #1 denote the cardinality of 1. For any x in R n, it can be verified that k

LX(i)

= max{ < x,£]

>: #1

= k}, k = 1,2, ...

, n.

i=l

The following characterization of majorization is valid. For x and Y in R n , x«y if and only if the following two conditions hold.

(1) For any I C {I, 2, ... ,n} with #1 ~ n - 1, there exists J C {I, 2, ... ,n} such that #1 = #J and < x, £] > ~ < y, cJ >. (2) Xl + ... + Xn = YI + ... + Yn. D. Another equivalent of majorization can be described as follows. A vector x is majorized by a vector Y if and only if n

n

n

n

L{Xi - a)+ ~ L(Yi - a)+ for every a E R and LXi = LYi. i=l i=l i=l i=l EXAMPLES.

1. Let Xl, ... ,Xn be n random variables with some joint distribution

function. Let X(1) 2: ... 2: X(n) be the order statistics of Xt, ... ,Xn arranged in decreasing order of magnitude. Assume that E!Xi ! < 00 for every i. Let X = (Xl, X2, ... , Xn)', X = (EXt, EX2, ... , EXn )' = (X},X2, ... ,xn)', say, and Y = (EX(I), EX(2), ... , EX(n»)'. Then x«y. This can be proved as follows: for any 1 ~ k ~ n, k

LX(i) i=l

= max{ < x,£] >: #1 = k} = max{E{ < X,£] >: #1 = k)} ~ E(max{ < X,£] k

= E(LX(i») = i=l

>: #1 = k}) k

:LEX(i). i=l

In view of Remark C, this essentially completes the proof of the above assertion.

306


2. Let AI, A 2, ... ,An be n events in a probability space with Pr( Ai) = ai, i = 1,2, .. . ,n. For each 1 ~ j ~ n, let Bj denote the event that at least j of At, A2, ... ,An occur. Equivalently, Bj = U(Ail n Ai2 n . .. n Ai;),

where the union is taken over all 1 ~ i l < i2 < .. . < ij ~ n . Let Pr(Bj ) = bj,j = 1,2, ... ,n. Let x = (aI, a2,'" ,an) and y = (bt,~, . " ,bn ). Then x«y. This can be shown as follows. Let Xi = I(A i }, i = 1,2, ... ,n, where I(A} is the indicator function of the event A. It is clear that EXi = ai for every i. Let XU) be the j-th order statistic of Xi's. Observe that for each 1 ~ j ~ n, X(j) = 1 if and only if at least j of Xt, ... ,Xn are each equal to unity, and otherwise equal to O. Consequently, I(Bj} = XU), The required assertion follows from Example 1 above. 3. Let Sn-l be the simplex in R n , i.e., Sn-l = {(PI,P2,'" ,Pn): each n

Pi 2:: 0 and

L Pi

= I}. There is a unique smallest element in Sn-l

i=l

according to the pre-order « on Sn-l given by (lin, lin, ... ,lin), i.e., (lin, lin, ... ,lin) « P for every P in Sn-l. There are n largest elements in Sn-l in the pre-order « on Sn-l. One of them is given by (1,0,0, ... ,0) which majorizes P for every P in Sn-l'

In the following definition we introduce an idea whose defining condition is slightly weaker than that of majorization. This notion plays a useful role in the formulation of some inequalities for eigenvalues. DEFINITION 9.1.2. We say that a vector x = (Xt,X2, ... ,xn )' is weakly majorized by a vector y = (yt, Y2, .. . ,Yn)' and denote the relationship by x «w y, if

I>u) ~ LYU),i = 1,2, . . . ,n. j=l

j=l

Complements 9.1.1 IT x and yare any two vectors of the same order and zany vector, show that

(~) « (~)

if and only if x «y.

Majorization

9.1.2 9.1.3

307

If x « z, Y « z, and 0 ~ 0 ~ 1, show that Ox + (1 - O)y « z. If x «y, x« z, and 0 ~ 0 ~ 1, show that x « Oy + (1 - O)z.

9.2. A Gallery of Functions

The notion of a function f from R m to R n preserving the pre-order « is part of a natural progression of ideas which advance the majorization concept. In this section, we will introduce this notion more formally and study some examples. DEFINITION 9.2.1. Let f be a map from Rm to Rn. It is said to be Schur-convex if x, y E R m and x «y ::::} f(x) «w f(y).

There are variations of Schur-convexity worth reporting. The function f is said to be strongly Schur-convex if x, y E R m and x «w y ::::} f(x) «w f(y). The function

f is said to be strictly Schur-convex if x, y E R m and x «y ::::} f(x)« f(y).

There are a host of other ways one could classify functions. We will now review some of these and examine their connection to Schurconvexity. Let ~ be the usual partial order on R n . More precisely, let x = (X},X2,'" ,xn )' and y = (Y},Y2,'" ,Yn)' be any two members of Rn. Say that x ~e Y iff Xi ~ Yi for all i. [suffix e stands for entrywise.] DEFINITION 9.2.2. Let f be a map from Rm to Rn. We define be monotonically increasing if

f to

x, Y E R m and x ~e Y ::::} f(x) ~e f(y),

f to be monotonically decreasing if - f is monotonically increasing, and f to be monotone if f is either monotonically increasing or monotonically decreasing. The notion that f is monotonically increasing is equivalent to the idea that f is coordinatewise increasing, i.e., if ~ and are real numbers such that ~ ~ then f(Xl,X2, •.. ,Xi-},~,Xi+}"" ,xn ) ~ f(X},X2,"" Xi-l,e,Xi+l,." ,xn ) for 1 ~ i ~ nand Xl,X2, ... ,Xi-l,Xi+l,'" ,Xn inR. Another notion of the same type is "convexity." This notion also uses the usual partial order ~ on R n .

e,

e


308

DEFINITION 9.2.3. Let f be a map from Rm to Rn. We define f to be convex if f(px + (1- p)y) ::;e pf(x) + (1 - p)f(y), for every 0 ::; p ::; 1 and x, y in Rm, and f to be concave if - f is convex.

The notion of "symmetry" can also be introduced for functions. The notion of a real-valued symmetric function is easy to guess. It is simply a function which is symmetric in its arguments. With some innovation, this notion can be extended to multi-valued functions. DEFINITION 9.2.4. Let f be a map from Rm to Rn. Say that f is symmetric if for every permutation 1r of {I, 2, . .. , m} there exists a permutation 1r' of {I, 2, ... , n} such that

(See Remark A(4) above for the definition of x 7r .) In the case when n = 1, f is symmetric if and only if f(x7r) = f(x) for all permutations 1r of {I, 2, ... , m} and x E Rm. A simple example of a real valued symmetric function is given by f(Xl,X2, .•. ,xm) = m

L

Xi, X

= (Xl,X2, •••

,X m )' E

Rm.

i=l

One of the pressing needs now is to identify Schur-convex functions among other classes of functions introduced above. We need some more machinery to move in this direction. We will take up this issue in the next section.

9.3. Basic Results We will now discuss inter-relations between various entities introduced in Section 9.2. At the core of the discussion lies doubly stochastic matrices and permutation matrices. We start with doubly stochastic matrices. DEFINITION 9.3.1. A matrix P = (Pij) of order n x n is said to be a doubly stochastic matrix if (1) Pij ~ 0 for all i and j, n

(2)

L i=l

n

Pij = 1 for all j, and (3)

L

Pij = 1 for all i.

j=l

Every permutation matrix of order nXn is a doubly stochastic matrix. See Remark A(4) of Section 9.2.

Majorization

309

We will now discuss the structure of doubly stochastic matrices. Let Dn be the collection of all doubly stochastic matrices of order n X n. 2 The set Dn can be viewed as a subset of Rn and shown to be a compact convex subset ofRn2. The convexity ofthe set is obvious. If P and Q are members ofD n and 0 ~ P ~ 1, then it is clear that pP+(l-p)Q E Dn. In the context of convex sets, the notion of an extreme point plays a useful role. If A is a convex subset of some vector space, then a member x of A is an extreme point of A if x cannot be written as a strict convex combination of two distinct members of A, i.e., if x = py + (1 - p)z for some 0 < p < 1 and y, z E A, then y = z. It is natural to enquire about the extreme points of Dn. Let P n be the collection of all pennutation matrices of order n X n. See Remark A(4) above. It is obvious that P n C Dn. The following result characterizes the extreme points of Dn.

P 9.3.2

The set of all extreme points of Dn is precisely P n'

P be a permutation matrix from P n and suppose that P = pDl + (1 - p)D2 for some 0 < p < 1 and D 1 ,D2 E Dn. Let P = (Pij), Dl = (dij(l») and D2 = (d ij (2»)' Look at the first row of P. Note that Plj = 1 for exactly one j E {I, 2, ... ,n} and the rest PROOF. Let

of the entries in the first row are all equal to zero. Let Pljl = 1 for some jl E {1,2, ... ,n}. Since Pljl = pd1jdl) + (1 - p)d1j1 (2) and o < P < 1, it follows that d1j1 (1) = d 1j1 (2) = 1. For j i= jt, 0 = Plj = pd1j(1) + (1 - p)d1j (2), which implies that d1j(1) = d1j(2) = O. Consequently, the first rows of P, Dl and D2 are identical. A similar argument can be used to show that the i-th rows of P, Dl and D2 are identical for any i = 2,3, ... ,n. Hence P = Dl = D 2 • Thus P is an extreme point of Dn. Conversely, let D = (d ij ) be an extreme point of Dn. Suppose D is not equal to any of the permutation matrices in P n' Then there are some rows in D such that in each of the rows there are at least two positive entries. Start with one such row, i1-th, say. Then 0 < d ili1 < 1 and 0 < d i li2 < 1 for some jl < h. Look at the j2-th column. There must be another entry di2i2 such that 0 < d i2i2 < 1. There must be an entry di2jS in the i 2-th row such that 0 < d i2i3 < 1. If we continue this way, we will obtain an infinite sequence (this process never ends!)

310


in D such that each entry in the sequence is positive and less than Wlity. But we have only a finite munber of subscripts (i, j) 's for d's with i,j E {I, 2, .. . ,n}. Consequently, a subscript either of the form (ir,jr) or of the form (ir,jr+I) must repeat itself. Assume, without loss of generality, that (it,j1) repeats itself. We now look at the following segment of the above sequence. (9.3.1) with j5+I = j1 . A possible scenario with s = 4 could look like as one depicted below with the entries forming a loop. Entries of the matrix D

did1

T T

d i3i4

L L L L L ! ! di4i4

----

---d i3i3

T T T T T T T di2i3 T T T

--

d id2

L L L L L L d i2i2

- - d i4i5

We now form two distinct matrices. Let 8 be the minimum of all the entries in the loop (9.3.1). One matrix C 1 is created from D by retaining all the entries of Din C 1 with the exception of those in the loop (9.3.1). The entries in the loop are replaced by

respectively and inserted into C 1 • The other matrix C 2 is created in a similar way the only change being the replacement of 8 by -8. We can now verify that C 1 and C 2 are distinct members of Dn and

Majorization

311

D = (1/2)C1 + (1/2)C2 . This contradicts the supposition that D is an extreme point of Dn. This completes the proof. A complete knowledge of extreme points of a convex set is helpful in recreating the set. Every member of the set can be written as a mixture of extreme points. This result is true in a more general framework but, for us, proving it in the environment of doubly stochastic matrices is rewarding and fulfilling. Before we establish this celebrated result, due to G.Birkhoff, we explore the notion of permanent of a matrix. DEFINITION 9.3.3. Let A = (ai j) be a square matrix of order n x n with real entries. The permanent of A, per(A), is defined by n

per(A) =

L II 7r

(9.3.2)

ai7r(i)

i=l

where the summation is taken over all permutations

7r

of {I, 2, ... ,n}.

There is one essential difference between per(A) and det(A). In the n

definition of the determinant, every product of the type multiplied by either +1 or -1 depending on whether odd permutation. As an example, let

A= [

n

ai7r(i)

is

i=l 7r

is an even or

~ -3 -~ -~l4

-2

Check that per(A) = 1 and det(A) = -1. Another critical property of the permanent is that it is permutation-invariant. If we permute the rows and/or columns of A, the value of the permanent remains the same. We will introduce a special notation for submatrices. Let A = (aij) be a square matrix of order n x n. Let I, J C {1, 2, ... ,n}. The submatrix A(I, J) of A is a matrix of order #1 x #J whose rows are indexed by the members of I and columns by J with entries plucked from A correspondingly. (Recall that #1 stands for the cardinality of the set I.) Symbolically, one writes the submatrix as A(I, J) = (aij)iEI,jEJ·

312


For example,

A

=

-1] 1 0 [ 2 -2 -2 -3

=}

1 , I 4

A(I,J)

= {2, I}, =

and J

[-2 1] 0 -1

= {2, 3},

.

The submatrix A(I, J) is technically different from the one given by I' = {I, 2} and J' = {2, 3}, i.e.,

A(I',J' ) =

[-~ -~] .

The permanents of these two submatrices are the same. Since we are predominantly dealing with permanents in this sojourn, we are not concerned about how the members of I and J are arranged. If I and J do not have the same cardinality, perA(I, J) does not make sense. Let us record a simple result on permanents which follows directly from the definition of the permanent. P 9.3.4 If all the entries of a square matrix A of order n x n are non-negative, and I, J are subsets of {I, 2, ... , n} having the property that #1 + #J = n, then per(A) ~ per(A(I, JC)) x per(A(Ic, J)).

(9.3.3)

It is worth noting that, in P 9.3.4, I and JC, the complement of set J, have the same cardinality. The following result plays a crucial role in many of the subsequent results.

P 9.3.5 Let A = (aij) be a square matrix of order n x n with nonnegative entries. Then per(A) = 0 if and only if there are subsets I, J of {I, 2, . . . , n} such that

#1

+

#J ~ n

+1

and

A(I, J) = O.

(9.3.4)

PROOF . Suppose (9.3.4) holds. Since the permanent is permutationinvariant, we can take I = {I, 2, .. . , k} and J = {k, k + 1, ... , n} for

313

M ajorization

some k. Note that #1 + #Ic = nand A(I, IC) = O. Further, the last column of A(I, I) is zero. Consequently,

pereA) = per(A(I, 1)) x per(A(Ic, I C)) =

o.

See Complement 9.3.4. The converse can be established using induction. IT n = 1, the validity of the converse is transparent. Suppose that the converse is true for all non-negative matrices of order m x m with m ~ n, n fixed. Let A = (aij) be a matrix of order (n + 1) x (n + 1) with nonnegative entries and per(A)= O. IT all the entries of A are zeros, the proof ends right there. Assume that aij is positive for some i and j. Without loss of generality, one can take an+I,n+I > O. (Why? Remember that we are dealing with permanents.) Take 10 = {I, 2, ... ,n} and J o = {n+l}. By (9.3.3),

0= pereA)

~

per(A(Io, (Jo)C)) x per(A((Io)C, Jo)) = per(A(Io, (Jo)C)) x an+I,n+1 ~ O.

This implies that per(A(Io, (Jo)C)) = O. Let B = A(Io, (Jo)C). Thus B is a matrix of order n x n with permanent equal to zero. The rows and columns of B are indexed by the same set. By the induction hypothesis, there are subsets It and J 1 of {I, 2, ... ,n} such that

#11 + #J1 ~ n + 1 and B(lt, Jt} = A(Ib Jt} = O. IT #It + #J1 ~ n + 2, the proof ends. Take I = It and J = J1 in (9.3.3). Suppose #11 + #J1 = n + 1. By (9.3.3), in the realm of (n + 1) x (n + 1)-order matrices,

which implies that either per(A(lt, (J1)C))= 0 or per(A((It}c, J 1)) = O. Assume, without loss of generality, that A(lt, (J1)C) = o. By the induction hypothesis, there are sets 12 C It and h C (J1)C such that #12 + #J2 ~ #11 + 1 and A(h, J2) = O. Note that

#h + #(J1 UJ2) = #h + #J1 +#h ~(#I1

+

1)+((n+l)-#Id=n+2.

314


Further, the fact that A(I2, J 1 U J 2) = 0 follows from A(I2, h) = 0 and A{12, J 1 ) is a submatrix of A{lt, Jt} which is equal to zero. Take 1 = 12 and J = J 1 U h in (9.3.4). A useful consequence of this result is the following. P 9.3.6 If A then per{A) > O.

= (Aij)

is a doubly stochastic matrix of order n x n,

Suppose per{A) = O. By P 9.3.5, there are subsets 1 and J of {I, 2, ... ,n} such that #1 + #J ~ n + 1 and A{1, J) = o. After a suitable permutation of its rows and columns, the matrix A can be brought to the form PROOF .

where 0 is of order #1 x #J and the transformed matrix is also doubly stochastic. Note that n = sum of all entries in A* ~

sum of all entries in C

= #1

+

#J

~

+ sum of all entries in B

n+ 1,

which is a contradiction. We are now in a position to describe the structure of the set of all doubly stochastic matrices of the same order. P 9.3.7 Every matrix in On is a convex combination of members of P n , i.e., given any D in On, there exist non-negative numbers Ap, P E P n such that L Ap = 1 and D = LApP. pep,..

Pep,..

PROOF. We furnish a proof using the induction argument. For any matrix D, let n{D) be the total number of non-zero entries in D. It is a well-defined map on On. Obviously, n{D) ~ n. If n{D) = n, it is clear that D ought to be a permutation matrix. The induction method is used on the number n{D). If n(D) = n, the proposition is evidently true. Assume that the proposition is true for all doubly stochastic matrices B for which n(B) < n{D). We will show that the result is also valid

Majorization

315

for the doubly stochastic matrix D = (d ij ). Since per(D) is positive, there exists a permutation 7r on {I, 2, ... ,n} such that d i 7l'(i) > 0 for all i. (Why?) Let Po be the permutation matrix in P n which corresponds to 7r. Let () = minl~i~n d i 7l'(i)' Two cases arise. If () = 1, the matrix D is precisely equal to the permutation matrix Po. The result is true. IT () < 1, let B =

1

(1 _ (})(D - (}Po).

The following features emerge from the definition of the matrix B. (1) B is doubly stochastic. (2) n(B) < n(D). (3) D = (1 - (})B + (}Po. By the induction hypothesis, B is a convex combination of permutation matrices. Hence D is a convex combination of permutation matrices. This completes the proof. REMARK. It is clear that the cardinality of the set P n of all permutation matrices of order n x n is n! Given D in D n , a natural question one can ask is whether one requires all members of P n in the representation of D in terms of members of P n in the form given above. The answer is no. In fact, a crude upper bound for the number of members of P n required in the representation of D above is n 2 • (Why? Each number of P n can be viewed as a member of Rn2.) But a good upper bound is given by n 2 - 2n + 2.

Majorization and doubly stochastic matrices are intimately related. One interesting feature of majorization is that if a vector y is a permutation of a vector x, then one can traverse from x to y through a series of intermediate vectors x

= v(O) «

V(l)

« ... «

v(m)

= y,

where any two consecutive vectors differ only in two coordinates. One merely flips two coordinates of a vector to move on to the next vector in moving from x to y. In the following result we demonstrate that this can be achieved in the environment of majorization. P 9.3.8

Let x, y E Rn. The following are equivalent.

(1) x«y.

316


(2) There is a finite number of vectors u(O), U(l), .. . , u(m) in Rn such that x = u(m) « u(m-l) « .. . « u(O) = y, and for all k , u(k) and U(k+l) differ only in two coordinates. (3) x = Dy for some doubly stochastic matrix. We show that (1) => (2). We use the induction argwnent on the dimension n. If n = 1, there is nothing to show. Assume that the implication is true for all vectors of order m x 1 with m less than n. Let x, Y E Rn. We can assume, without loss of generality, that the components of each of the vectors x = (Xl> X2, ... ,Xn )' and Y = (YI, Y2, . .. , Yn)' are arranged in the decreasing order of magnitude. (Why?) We move from Y to x successively the way outlined in (2). We produce, first, u(l) . Note that Yn :::; Xl :::; YI . One can find k such that Yk-l :::; Xl :::; Yk . Write Xl = tYI + (1 - t)Yk for some 0 :::; t :::; 1. Let PROOF.

u(1)

= (Xl> Y2, ...

, Yk-l,

(1 -

t)YI

+ tYk, Yk+l, .. . ,Yn)'.

The vectors U(l) and u(O) could possibly differ in the first and k-th coordinates. We will show that u(1) « u(O). Towards this end, we will show, first, that (Xl, (1 - t)YI + tYk)' is equal to (9.3.5) It is clear that Xl +(l-t)YI +tYk = YI +Yk and both Xl and (l-t)YI +tYk are less than or equal to YI. Consequently, (9.3.5) follows. (Why?) If we adjoin the vector (Xl> (1 - t)YI + tYk) to (Y2, Y3, ... ,Yk-l, Yk+l, ... ,Yn) and the vector (Yl> Yk) to (Y2 , Y3, ... ,Yk-l> Yk+l, .. . ,Yn), it now follows that u(l) « u(O). See Complement 9.1.1. The next critical step is to demonstrate that

p

«

= (X2' X3, . . .

(Y2, Y3, . .. ,Yk-l,

(1 - t)YI

,X n )'

+ tYk, Yk+l, . ..

,Yn)'

= q. (9.3.6)

By design, we have

(1) Y2 ~ Y3 ~ ... ~ Yk-l ~ Xl ~ X2 ~ . . . , (2) (1 - t)YI + tYk ~ Yk+l ~ Yk+2 · .. ~ Yn . The number (1 - t)YI + tYk is intermediate to the numbers Y2 and Yk+l. The following inequalities are clear.

(1)

X2

+ X3 + ... + Xr

:::; Y2

+ Y3 + ... + Yr

for any 2 :::; r :::; k - 1.

Majorization

317

(2) (Y2 +Y3 + ... +Yk-d + (1- t)Yl +tYk + (Yk+l +Yk+2 + ... +Yr) = (Yl + Y2 + ... + Yr) - Xl ?: (Xl + X2 + .. . + Xr ) - Xl = X2

+ X3 + ... + Xr for any

k ::; r ::; n.

In (2), equality occurs when r = n. No matter where the number (1 - t)Yl + tYk occurs in the chain Y2 ?: Y3 ?: ... Yk-l ?: Yk+t, (9.3.6) follows. (Why?) Let us invoke the induction hypothesis. There are vectors w(l), w(2), ... ,w(m) in Rn-l such that p

= w(m) «

w(m-l)

« ... «

w(2)

«

w(l)

= q.

Let u(k) = (~1») for k = 1,2, ... , m and u(O) = y. These vectors meet all the requirements stipulated in (2). We now proceed with the implication (2) ::} (3). In the chain of vectors, we note that u(k+l) is obtainable from u(k) by pre-multiplying u(k) by a suitable doubly stochastic matrix. More precisely, u(k+I) = (tJ +(I-t)Pk)u(k), where I is the identity matrix and Pk is some suitable permutation matrix which swaps two coordinates only. Stringing all these doubly stochastic matrices in a multiplicative way, we obtain a doubly stochastic matrix D such that X = Dy. Finally, we tackle the implication (3) ::} (1). Let X = Dy for some doubly stochastic matrix D = (d ij ). Assume, without loss of generality, that the components of X = (Xl, X2, ... , Xn)' are in decreasing order of magnitude, Le., Xl ?: X2 ?: ... ?: Xn . Likewise, assume that the components of Y = (yt, Y2, .. . , Yn) are in decreasing order of magnitude. (If not, let PI and P2 be the permutation matrices so that the components of each of the vectors PIX and P2Y are in decreasing order of magnitude. The equation X = Dy can be rewritten as PIX = (Pl )D(P2 )-1(P2Y). Note that (Pt}D(P2)-1 is doubly stochastic.) Note that for any 1 ::; k ::; n, k

k

k

n

k

LXi - LYi = LLdijYj - LYi i=l

i=l

i=l j=l n

i=l

= LtjYj - LYi j=l

n

k

+ Yk(k -

Ltj)

i=l

j=l n

k

= L(Yj - Yk)(tj - 1) j=1

+

L j=k+l

tj(Yj - Yk) ::; 0,

318


where 0 ~

tj

k

L d ij

=

~

1. These numbers are non-negative and sum

i=l

up to k. This completes the proof. We would like to identify Schur-convex functions among convex functions. The following result flows in this direction. P 9.3.9 If a function J from Rm to R n is convex and symmetric, then it is Schur-convex. If, in addition, J is monotonically increasing, then J is strongly Schur-convex. PROOF. Let x, y E R m be such that x «y. By P 9.3.8, there exists

a doubly stochastic matrix D such that x = Dy. By P 9.3.7, we can write D = LApP, a convex combination of permutation matrices PEP",

with 0 ~ Ap and

L

AP = 1. Since J is convex,

PEP",

J(x) = J(

L

APPy) ~

PEP",

L

ApJ(Py) .

PEP",

J is symmetric, for each PEP m, there exists QpEPn such that J(Py) = QpJ(y) for all y E Rm. Consequently,

Since

J(x) ~

L

ApQpJ(y) .

PEP",

Let Do

= L

PEP",

ApQp . Clearly, Do is a doubly stochastic matrix of

order n x n. From the proof of P 9.3.8, it is clear that J(x) ~e DoJ(y) implying that J(x) «w J(y). Suppose, in addition, J is monotonically increasing. Let x, y E Rm be such that x «w y. There exists z in Rm such that x ~ z« y . See Complement 9.2.2. By what we have assumed about J, we have

J(x) ~e J(z) «w J(y), ~ J(x) «w J(z) «w J(y) and J(x) «w J(y). This shows that J is strongly Schur-convex. One can jot down a number of consequences of P 9.3.9.

Majorization COROLLARY

319

9.3.10. Let I be a real-valued function of a real variable

and

If I is convex, then Is is Schur-convex. If, in addition, I is monotonically increasing, then Is is strongly Schur-convex. PROOF. It is clear that Is is convex. As for the symmetry of Is, note that for any permutation II of {I, 2, ... ,n}, I(xn) = (J(x))n for all xERn.

Let us introduce some additional operations on vectors. For any vector x ERn, let u define x+ = (max{xt, O}, max{x2' O}, ... ,max{xn, O})" and Ixl = (lxII, IX21,··· ,Ixn !). 9.3.1l. (a) If x «y, then Ixl «w Iyl. (b) If x« y, then x+ «w y+. (c) If x« y, then (xi,x~, ... ,x~) «w (Yi,y~, ... ,y;).

COROLLARY

For (a), take I(t) = Itl, t E R, in Corollary 9.3.10. For (b), take I(t) = max{t,O}, t E R, in Corollary 9.3.10. For (c), take PROOF.

I(t) = t 2 , t E R. Complements

9.3.1

Let A = (aij) be an orthogonal matrix with real entries, i.e.,

A' A = I. Show that D = (a~j) is a doubly stochastic matrix.

Let x «y. Show that there exists an orthogonal matrix A = (aij) such that x = Dy, where D = (a~j)' 9.3.3 Let A = (aij) be a Hermitian matrix, x = (AI, A2, ... ,An)', the vector of eigenvalues of A, y = (all, a22, ... ,ann), and A = U Diag {AI, A2,'" ,An}U*, the spectral decomposition of A, i.e., U = (Uij) is a unitary matrix. Let D = (luijI2). Show that y = Dx. Hence, or otherwise, show that y « X.

9.3.2

n

9.3.4

Let l(xttX2, ... ,Xn) = LXilogxi; XttX2, ... ,Xn > O. Show i=l

that I is Schur-convex on the set (R+)n. 9.3.5 Let A be a square matrix of order n X n and I, J subsets of {I, 2, ... ,n}. If #1 + #J = nand A(I, J) = 0, show that pereA) = per(A(I, JC)) x per(A(Ic, J)).

320


9.3.6

Let I(XI, X2, .. . ,xn )

=

~

n

L (Xi-x)2 = variance of the numbers i=l

Xl!

X2, ... ,Xn , X

=

(XI, X2, ... ,xn ) ERn, where x

n

= *- L

Xi· Show

i=l

that I is Schur-convex. 9.3.7 Let I be a real-valued. function defined. on Rn whose first-order partial derivatives exist. Let I(i) = 8~i I, i = 1,2, ... ,n. Show that I is Schur-convex if and only if the following hold.

(1) I is symmetric. (2) (Xi - Xj)(J(i) (x) - l(j)(x))

~

0 for all X and i,j. 9.3.8 The k-th elementary symmetric polynomial, for any fixed. 1 k ~ n, is defined by

~

where the summation is taken over alII ~ i l < i2 < ... < i k ~ n. Show that -Sk is Schur-convex over (R+)n. 9.3.9 Let the components of the vectors x' = (Xl! X2, ... ,xn ) and y' = (yI, Y2,· .. ,Yn) be non-negative. If X « y, show that

n Xi n Yi· n

i=l

9.3.10

n

~

i=l

Let A = (aij) be a positive definite matrix of order nXn. Show n

that det(A) ~

n aii. (Use Complements 9.3.3 and 9.3.9.) i=l

Note: For a historical and comprehensive treatment of majorization, Marshall and Olkin (1979) is a good source. A substantial portion of this chapter is inspired. by the work of Ando (1982).

CHAPTER 10 INEQUALITIES FOR EIGENVALUES Eigenvalues and singular values of a matrix playa dominant role in a variety of problems in Statistics, especially in Multivariate Analysis. In this chapter we will present a nucleus of results on the eigenvalues of a square matrix. Corresponding results for the singular values will also be presented. To begin with, we make some general remarks. Let A be a matrix of order n x n with complex numbers as entries. Let AI! A2,· .. ,An be the eigenvalues of A. In order to indicate the dependence of the eigenvalues on the underlying matrix A, the eigenvalues of A are usually denoted by Al (A), A2(A), ... ,An(A). The eigenvalues of A can be obtained as the roots of the determinantal polynomial equation,

IA -

AInl = 0,

in A of degree n. Let O"~ 2 O"~ 2 . . . 2 0"; (2 0) be the eigenvalues of the Hermitian matrix A * A, or equivalently of AA *. The numbers 0"1 2 0'2 2 ... 2 O'n(2 0) are called the singular values of A. More precisely, the singular values are denoted by 0'1 (A) 2 0'2(A) ~ ... ~ O"n(A). Now, let A be a matrix of order m x n. If m i= n, the concept of eigenvalues of A makes no sense. However, singular values of A can be defined in a legitimate manner as the square roots of the eigenvalues of AA * or of A * A. We usually write the singular values of a matrix in decreasing order of magnitude. For a square matrix, some of the eigenvalues could be complex. But if the matrix is Hermitian, the eigenvalues are always real. In such a case, the eigenvalues are written in decreasing order of magnitude. Thus two sets of numbers, namely the set of eigenvalues and the set of singular values, can be defined for a square matrix. The relationship between these two sets will be pinpointed later in this chapter. 321

322


All the results in this chapter are cast in the environment of the vector space en with its usual inner product < ., . > given by n

< x,Y >= LXdii = y*x, i=l

for x' = (Xl, X2, ... ,xn ) and y' = (Yl, Y2, ... ,Yn) E en. Two vectors X and yare orthogonal (x.ly in notation) if < x, Y > = 0 or y*x = o. 10.1. Monotonicity Theorem Let A and B be two Hermitian matrices of the same order n x n, and C = A+B. In this section, some inequalities connecting the eigenvalues of the three matrices A, B, and C will be presented. P 10.1.1

(Monotonicity Theorem for Eigenvalues) Let

be the eigenvalues of the matrices A, B, and C, respectively. Then

(1)

>

1'1

1'2

(2)

}

>

(3)

}

> ?,

01

(n)

+ f3n

02

:.~n-l

On

+ f3l

}

>

1'n

>

323

Inequalities for Eigenvalues

PROOF. The inequalities on the left quoted above can be written succinctly as follows: for each i = 1,2, . . . , n, OJ +f3i-j+l ~ Ii for j

= 1,2, . ..

,i; i

= 1,2, ... ,no

(10.1.1)

The inequalities quoted above on the right can be written as

+ f3n-j+i, j = i,i + 1, ... ,n; i = 1,2, ... ,no

Ii ~ OJ

(10.1.2)

Using a very simple argument, we will show that the inequalities (10.1.2) follow from (10.1.1). Let Ul,U2, ... ,Un; Vl,V2,· .. ,Vn; Wl,W2,.·. ,Wn

be the corresponding orthonormal eigenvectors of A, B, and C, respectively. Fix 1 :S: i :S: nand 1 :S: j :S: i. Let SI = span{Uj,Uj+l, ... ,un}, S2 = span{ Vi-j+l, Vi-j+2, .. . , v n }, S3 = span{ WIt W2, ... , Wi}. Note that dim(St} = n - j + 1, dim(S2) = n - i Using the dimensional identity (see P 1.5.7),

+ j,

and dim(S3)

= i.

+ dim(S2 n S3) - dim(St + (S2 n S3)) = dim(Sl) + dim(S2) + dim(S3) - dim(S2 + S3) - dim(Sl + (S2 n S3)) ~ dim(Sl) + dim(S2) + dim(S3) - n - n

dim(SI n S2 n S3) = dim(St}

=n-j+l+n-i+j+i-n-n=1. Consequently, there exists a vector x in Sl n S2 n S3 such that x*x = 1. Since x E SI, we can write x = ajuj + ... + an Un for some complex numbers aj,aj+It· . . ,an . The property that x*x = 1 implies that E;=j lar l2 = 1. Further, n

x* Ax

=

n

n

n

n

(2: ilru;)A(L arur ) = (2: ilru;)(L arorur ) = L r=j

r=j

r=j

r=j

r=j

lar l2 or .

324


We have used. the fact that ur's are orthonormal eigenvectors. Since the eigenvalues ai's are written in decreasing order of magnitude, it now follows that aj 2: x* Ax 2: an. In our argument we need only the inequality aj 2: x* Ax. In a similar vein, one can show that f3i-j+1 2: x* Bx 2: f3n, II 2: x*Cx 2: Ii, ~ aj

+ f3i-j+1 2: x* Ax + x* Bx =

x*Cx 2: Ii.

Thus (10.1.1) follows. The validity of the inequalities on the right hand side of the Monotonicity Theorem can be effected as follows. Note that

(-A)

+ (-B) = -C,

and the eigenvalues of (-A), (-B) and (-C) are respectively

By what we have proved earlier, it follows that for every i = 1,2, ... , n and j = i, i + 1, ... , n,

from which we have Ii

2: aj + f3n-U-i).

(The map T: {1,2, ... ,n} ---? {n,n-l, ... ,I} involved here is given by T(r) = n - r + 1, r E {I, 2, ... , n}.) This completes the proof. The plethora of inequalities stated in P 10.1.1 can be recast utilizing the majorization concept. Let ,x(C) = (,1" ... "n);,x(A) = (a1, ... ,an), and ,x(B) = (f31, ... ,f3n).

It follows from P 10.1.1 that ,x(C) « (,x(A) + ,x(B)). We would like to venture into the realm of singular values. The question we pop is what is the relationship between the singular values of A, B, and the sum A + B. We need a trick. Let A E Mm nand q = min{m,n}. Let 0"1 2: 0"22: ••• 2: O"q{2: 0) be the singular vaiues of A. Construct a new matrix by


325

A*] o . Note that the matrix Hermitian.

P 10.1.2

A is

of order (m

1m - nl

x (m

+ n).

The eigenvalues of the Hermitian matrix

0"1 2: 0"2 2: ... 2: O"q 2: 0 = (with

+ n)

... = 0 2:

Further,

A is

A are:

-lTq 2: -lTq-l 2: ... 2: -lTl

zeros in the middle).

PROOF. Using a singular value decomposition of A, we will obtain a spectral decomposition of A. Assume, for simplicity, that m ~ n. Let A = Pt:::.Q be a singular value decomposition of A, where P and Q are unitary matrices of appropriate order and t:::. = (DIO) with D = diag{lTb lT2, ... ,lTm} and 0 is the zero matrix of order m x (n - m). Partition Q as: Q = (~~), where Ql is of order mXn and Q2 of order (n-m) Xn. The fact that Q*Q = In implies that QiQl + Q 2Q2 = In. The singular value decomposition of A can be rewritten as: A = P DQl. Construct a matrix

/J2]

P* P*/J2

o

.

Note that U is a matrix of order (m + n) x (m + n). More interestingly, U is indeed a unitary matrix. Further,

o -D

o

Thus we have a spectral decomposition of the Hermitian matrix A. The diagonal entries of the matrix ~n the middle of the representation above are indeed the eigenvalues of A. This completes the proof. The above result will be instrumental in establishing a monotonicity theorem for singular values.

326


P 10.1.3 (Monotonicity Theorem for Singular Values). Let

be the singular values of the matrices A, B, and C, respectively, where A and B are each of order m x n, C = A + B, and q = min {m, n}. Then we have the following results:

(1)

Iq

~

!"'+iJ,

a2.~./3q-l a q + /31

( "I +!i,-I (2)

Iq-l ~

a2 + /3q-2 a q-l

+ /31

( "I +!i,-2 Iq-2~

(3)

a2 + /3q-3 aq_~

(q)

II

T. The recasting is carried out as follows. Let A E Mm,n be given. Let Al be the submatrix obtained from A by deleting either one row of A or one column of A. Then

Suppose A2 is a matrix obtained from A by deleting either two rows of A or two columns of A or one row and one column of A. We would like to explore the connection between the singular values of A and A 2 . Suppose A2 is obtained from A by deleting two columns, say jl-th and

332


h-th, of A. Let Al be the matrix obtained from A by deleting the jrth column of A. It is clear that the matrix A2 can be obtained from Al by deleting an appropriate column of AI. The relationship between the singular values of Al and those of A2 is clear. More precisely,

O"i(A I ) 2: O"i(A 2) 2:

O"i+I (Ad,

i

= 1,2, ...

,min{ m, n - I}.

These inequalities can be stretched a little further. We can say that

O"i(A I ) 2: O"i(A 2) 2: O"i+I(Ad, i = 1,2, ... ,{m, n}. We do have a good grip on the connection between the singular values of Al and those of A. Finally, we can jot down that

The argument can be pushed further. We have now an analogue of Theorem 10.2.1 for singular values. 10.2.3 Let A E Mm,n' Let Ar be any matrix obtained from A by deleting a total of r rows and columns. Show that

10.3. Courant-Fischer Theorem The Monotonicity Theorem examines relationship between the singular values of a sum of two matrices and those of its constituents. The Interlace Theorem explores the connection between the singular values of a matrix and those of a submatrix. It is time to be a little introspective. The Courant-Fischer Theorem characterizes the singular values via optimization.

P 10.3.1 (Courant-Fischer Theorem for Eigenvalues) Let A E Mn be a Hermitian matrix with eigenvalues

Then for each 1 :::; k :::; n, o'k

=

=

max

min

see" ,di m (S }=k xES ,x' x= I

min

(x* Ax)

max

Tee" ,dim(T}=n-k+I xET,x' x=l

(x* Ax).

(10.3.1) (10.3.2)


333

In particular, 0t

= xECn,x'x=l max x" Ax,

On

= xECn,x'x=l min x " Ax.

(The entities Sand Tare subspaces of en.) PROOF . Let U1, U2, . .. ,un be the orthonormal eigenvectors corresponding to the eigenvalues, 01, ... ,On, of A. Fix 1 ::; k ::; n. Let

Clearly, dim(Sl) = n - k Note that

en.

+ 1.

Let S be any k-dimensional subspace of

dim(S n St} = dim(S)

+ dim(Sl) -

dim(S

+ Sl)

2:k+n-k+l-n=1. Choose any y in sns 1 such that y*y = 1. Following by now the familiar argument, since y E Sl, we have Ok 2: y" Ay. Consequently, since YES, Ok

2:

min

xES ,x' x=l

x" Ax.

This inequality is valid for any k-dimensional subspace S of en. Therefore, Ok 2: max min (x" Ax). secn ,dirn(S)=k

xES ,x' x=l

We need to show now the reverse inequality. Let S=span{Ut,U2 , . ..

,ud.

Note that S is a k-dimensional subspace of x .. x = 1, then

en.

Further, if xES and

(Why?) It is clear that max

secn ,dim(S)=k

min xES ,x' x=l

(x* Ax) >

min xES ,x' x=l

x* Ax >

Ok .

334


Thus the identity (10.3.1) follows. As for the second identity, let

It is clear that dim(Sl) = k. Let T be any (n - k + I)-dimensional subspace of en. It is clear that dim(SI n T) 2': 1. Choose any vector y in SI n T such that y*y = 1. Since y E Sl, y* Ay 2': Cik. Since yET, max

xET,x·x=l

(x* Ax)

(10.3.3)

2': Cik.

Let us recast this observation in the following way: for every (n - k + 1)dimensional subspace T of en, (10.3.3) holds. Consequently, min

TCCn,dim(T)=n-k+1

(x* Ax) 2':

max

xET,x· x=l

Cik .

To establish the reverse inequality, take

FUrther, if x E Cik

2':

and x*x = 1, then x" Ax ~

'1',

!llax

xET ,x.x=l

(x* Ax)

2':

Cik .

Hence

min

max

TCCn,dim(T)=n-k+1 xET,x·x=l

(x* Ax).

This establishes the identity (10.3.2). This completes the proof. There is another angle at which one can look at the Fischer-Courant theorem. Let A E Mn be a Hermitian matrix. Let Ci1 2': Ci2 2': ... 2': Ci n be the eigenvalues of A with the corresponding orthogonal eigenvectors Ul, U2, ... ,Un. Define a function p : en - {O} ~ R by u*Au p(u) = - - , u E u*u

en,

U:f: O.

The function p(.) is called the Raleigh quotient and has the following properties. (1)

(2)

Ci1

2': p(u) 2': min

uEC",u;tO

Cin,U E en,U:f:

p(u)

= Ci n

o.

with the minimum attaining at u

= u n·


(3)

max

uEC",u;i:O

p(u)

= 01

335

with the maximum attaining at

U

= Ul .

(4) The function p(.) is stationary at, and only at, the eigenvectors of A, i.e., for example,

8p(u)

~ IU=Ui = 0

for each i.

(5) Let 1 ~ k ~ n be fixed. Let Sk = span{uk, Uk+b

..• ,

un}. Then

and the maximum is attained at U = Uk . (6) LetTk =span{u1,u2, ... ,ud. Then

and the minimum is attained at

= Uk .

U

Look at the Courant-Fischer theorem after going through the above properties of the Raleigh quotient. The properties of the Raleigh quotient involve both the eigenvalues and the chosen eigenvectors. The Courant-Fischer theorem characterizes the eigenvalues of A without involving eigenvectors. An analogue of P 10.3.1 for the singular values of a matrix is not hard to obtain. In the formulation of the relevant result, an expression like x* Ax does not make sense for non-square matrices. We need to find an appropriate analogue for such an expression.

P 10.3.2 (Courant-Fischer Theorem for Singular Values) Let A E Mrn ,n and be its singular values. Then for each k

O'k(A) =

min

~

max

1,

(x*(A* A)X)1/2

(10.3.4)

SCC",dim{S)=n-k+1 xES,x' x=1

=

max

min

TCC" ,dim{T)=k xET,x' x=1

(x*(A* A)x)1/2.

(10.3.5)

336


PROOF. If a subspace is not available with a specified dimension, we take the min-max and max-min numbers of (10.3.4) and (10.3.5) to be equal to zero. The result follows from the corresponding CourantFischer theorem for the eigenvalues of a matrix if we keep in mind the fact that the eigenvalues of A * A are:

(7~(A) ~ (7~(A) ~ ...

One can obtain the Interlace Theorem from the Courant-Fischer Theorem. First, we deal with the eigenvalues of a square matrix. COROLLARY 10.3.3. (Interlace Theorem) Let A E Mn be Hermitian and 01 ~ 02 ~ . .. ~ On its eigenvalues. Let B be a submatrix of A obtained by deleting some n - r rows of A and the corresponding columns of A. Let 131 ~ 132 ~ ... ~ f3r be the eigenvalues of B. Then Ok ~ 13k ~ 0k+n-r,

k

= 1,2, ...

,r.

PROOF. Note that B is a Hermitian matrix of order r x r. Assume, without loss of generality, that B was obtained from A by deleting the last n - r rows and last n - r columns of A. For 1 ~ k ~ r, by the Courant-Fischer theorem, Ok

= ~

=

min

Tee n ,dim(T)=n-k+1

max

xET,x' x=l

(x* Ax)

min

max

(z* Az)

min

max

(y* By) =

Tee n ,dim(T)=n-k+1 zET,z' z=l Tee r ,dim(T)=r-k+1 yET,y'y=l

13k

where z in the middle equation is a vector with the last (n - r) components as zero. The last step requires some deft handling. It is left as an exercise to the reader. On the other hand, for each 1 ~ k ~ r, by the Courant-Fischer theorem, 0k+n-r

= See",dim(S)=k+n-r max ~

max

min

xES,x' x=l

min

seen ,dim(S)=k+n-r zES,z' z=l

(x* Ax)

(z* Az)

= seermax min (y* By) = 13k ,dim(S)=k yES,y' y=l . This completes the proof.


337

Complements

10.3.1 Let A, B E Mn be such that A is Hennitian and B is nonnegative definite. Let

be the eigenvalues of A and A + B, respectively. Using the CourantFischer theorem, show that ak ~ {3k, k = 1, 2, . .. ,n. (These inequalities can also be obtained using the Monotonicity Theorem.) 10.3.2 Obtain the Interlace Theorem for the singular values of a matrix from the corresponding Courant-Fischer theorem. 10.3.3 (Sturmiall Separation Theorem) Let Ar be the submatrix obtained by deleting the last n - 7' rows and columns of a Hermitian matrix A, r = 1, ... ,n. Then Ak+l (A+d ~ Ak(A i ) ~ Ak{Ai+d.

10.4. Poincare Separation Theorem The Monotonicity Theorem, Interlace Theorem and Courant-Fischer Theorem form a triumvirate of results on the eigenvalues and the singular values of matrices. The monotonicity theorem compares the eigenvalues of two Hermitian matrices A and B with their sum. The Interlace Theorem compares the eigenvalues of a Hennitian matrix and its principal submatrices. The Courant-Fischer Theorem characterizes the eigenvalues of a Hermitian matrix. Anyone of these results is deducible from anyone of the other results. Some of these implications have already been alluded earlier. The Poincare Separation Theorem, which is the subject of discussion in this section, also falls into the same genre. P 10.4.1 (Poincare Separation Theorem for Eigenvalues) Let A E Mn be a Hermitian matrix with eigenvalues 0.1 ~ 0.2 ~ ... ~ an. Let B be any matrix of order n x k such that B* B = I k , i.e., the columns of B constitute a set of orthonormal vectors. Let {31 ~ {32 ~ ... ~ {3k be the eigenvalues of the matrix B* AB. Then ai ~ (3i ~ ai+n-k, i = 1,2, ... ,k. PROOF. Note that B* AB is Hermitian. This result can be deduced from the Interlace Theorem. Let B = (u}, U2, ... ,Uk), where Ui is the

338


i-th column of B. Determine orthonormal vectors Uk+}, Uk+2, ... ,Un such that V = (u}, U2, ... ,un) is unitary. Observe that the matrices V* AV and A have the same set of eigenvalues. Further, B* AB is a principal submatrix of V* AV obtained by deleting the last n - k rows and columns. Now the Interlace Theorem takes over. The result follows. Now the question arises as to when 0i = f3i for i = 1, ... ,k in the Poincare Separation Theorem. Let U}, U2, .. . ,Un be the corresponding orthonormal eigenvectors of A. Let V = (u}, ... ,Uk) and take B = VT for some unitary matrix T of order k x k. Then 0i = f3i, i = 1,2, ... ,k. Let us see what happens to B* AB. Since V* AV = Diag(o}, ... ,Ok), B* AB = T*V* AVT = T*Diag{o}, 02, ... ,0dT. Thus the eigenvalues of B* AB are precisely O}, 02, . . . ,Ok. This establishes that 0i = f3i for i = 1,2, ... ,k. One might ask in a similar vein as to when f3i = Oi+n-k holds for i = 1,2, ... ,k. This is left to the reader as an exercise. We now need to launch an analogue of the Poincare separation theorem for the singular values of a matrix. The following result which is easy to prove covers such a contingency.

P 10.4.2 (Poincare Separation Theorem for Singular Values) Let A E Mm,n with singular values

O"}(A)

~

0"2(A)

~

....

Let V and V be two matrices of order m x p and n x q, respectively, such that V*V = Ip and V*V = I q. Let B = V* AV with singular values

O"}(B) Then, with r

~

0"2(B)

~

....

= (m - p) + (n - q),

O"i(A)

~

O"i(B)

~

O"i+r(A), i = 1,2, ... , min {m, n}.

Complements Let A,B E Mm,n, E = B - A and q = min{m,n}. If O"} ~ 0"q are the singular values of A, Tl ~ ... ~ Tq are the singular values of B, and 8 is the spectral norm of E, then (1) 10"i - Ti I s 8 for all i = 1, . .. ,qj and (2) (O"} - T})2 + ... + (O"q - Tq)2 s IIEII~.

10.4.1

. ..

~


339

10.5. Singular Values and Eigenvalues For a square matrix, one can detennine its singular values as well as its eigenvalues. If the matrix is Hermitian, a precise relationship exists between its singular values and eigenvalues. In this section, we establish a set of inequalities connecting the singular values and eigenvalues of a general matrix.

P 10.5.1 Let A E Mn . Let a}, 02, .. . ,an be the eigenvalues of A arranged in such a way that

Let

0"1

~ 0"2 ~ ••• ~ O"n

be the singular values of A. Then

with equality for i = n. PROOF. By the Schur triangularization theorem, there exist unitary matrices U and Veach of order n x n such that U* AV = 6, where 6 is an upper triangular matrix with diagonal entries 01, 02, . .. , an. Fix 1 ~ i ~ n. Write U = (U1IU2) and V = (VI !V2), where U I and VI are of order n x i. Note that

U*AV=

[~}] A(VI !V2) = [~}~~ = 6 = [ ~i

Ui A V2 J

U2AU2

~] , say.

It is now clear that Ui AVI is upper triangular with entries in the diagonal being 01, 02, ... ,oi. It is indeed a sub matrix of U* AV. By the Interlace Theorem, O"j(U; AVt} ~ O"j(U* AV), j = 1,2, ... ,i.

Let us compute the singular values of U* AV. For this we need to find the eigenvalues of (U* AV)*(U* AV) = V* A* AV. The eigenvalues of A* A and V* A* AV are identical. Consequently, O"j(U* AV) = O"j(A) = O"j, j = 1,2, ... ,n. Finally,

340


Idet(U; AVdl

= Idet(6i)1 = Irr;=lajl = rr;=IO"j(U; AVn

:::; rr;=IO"j·

When i = n, equality holds. (See Complement 10.5.1.) This completes the proof. The multiplicative inequalities presented in P 10.6.2 for the eigenvalues and singular values of a square matrix have an additive analogue. More precisely, if none of the eigenvalues and singular values is zero, then (£nla11, £nla21,··· ,£nlanD

«

(£nO"l, £n0"2, ... ,£nO"n),

where £n denotes the natural logarithm. COlnplements

10.5.1 Let A E Mn and 0"1 2: 0"2 2: ... 2: O"n (2: 0) its singular values. Show that Idet(A)1 = 0"10"2·.· O"n· 10.6. Products of Matrices, Singular Values and Horn's Theorem One important operation on matrices is the operation of multiplication. In this section, we will present some results on a connection between the singular values of the product of two matrices and the singular values of its constituent matrices. P 10.6.1 (Horn's Theorem) Let A E Mm ,n and B E Mn,p. Let q = min{m,n,p},r = min{m,n},s = min{n,p} and t = min{m,p}. Let

O"I(A) 2: 0"2(A) 2: ... 2: O"r(A) (2: 0), O"l(B) 2: 0"2(B) 2: ... 2: O"s(B) (2: 0), O"l(AB) 2: 0"2(AB) 2: . .. 2: O"t(AB) (2: 0), be the singular values of A, B, and AB respectively. Then

rr;=IO"j(AB) :::; rr;=IO"j(A)CTj(B), i = 1,2, ... ,q.

(10.6.1)

If A and B are square matrices of the same order, i.e., m = n = p, then equality holds in (10.6.1) for i = n. PROOF.

The proof is outlined in a series of steps.


341

1. Let AB = P 6Q* be a singular value decomposition of AB for some unitary matrices P and Q of orders m x m and p x p, respectively. The (k, k)-th entry of the matrix 6 is equal to O"k(AB), k = 1,2, ... ,t, and the rest of the entries of 6 are zeros. 2. Fix 1 :::; i :::; q. Write P = (Pl IP2) and Q = (QlIQ2) , where PI and Ql are of orders m x i and p x i, respectively. Observe that Pi(AB)Ql is an i x i principal submatrix of P*(AB)Q. Therefore, Pi(AB)Ql = diag{ 0"1 (AB), 0"2(AB), ... ,O"i(AB)}, and

det(P: ABQ1)

= n;=IO"j(AB).

3. We want to focus on the matrix BQ1, which is of order nxi. By the polar decomposition theorem, we can find two matrices X of order n x i and W of order i x i such that X has orthonormal columns, W is nonnegative definite, and BQl = XW. Note that W 2 = (BQ.)*(BQ.) = Qi B* BQl . Hence, det(W2) = the product of the squares of the eigenvalues of W = the product of the eigenvalues of Qi B* BQ1 = n;=10"j(Qi B *BQ.).

(The singular values and eigenvalues are the same for a positive semidefinite matrix.) By the Poincare separation theorem,

4. Let us focus on the matrix Pi AX. This is a square matrix of order i x i. By Complement 10.5.1 and Poincare Separation Theorem,

5. Combining all the steps we have carried out so far, we have n;=IO"j(AB) = det(P: ABQ.) = Idet(Pi ABQ1)1 = Idet((Pi AX)(W))I = Idet(Pi AX)lldet(W)1 :::; n;=lO"j(A)O"j(B).

6. If m

= n = p, then

342


This completes the proof. The multiplicative inequalities presented in P 10.6.1 have an additive analogue too. Within the same scenario, we have

, L1nui(AB) ~ Lln(ui(A)uj(B)), i j=l

= 1,2, ...

,q.

j=l

IT A and B are square matrices of the same order n x n, then (lnUI (AB), 1nu2(AB), ... ,lnun(AB))

«

(In(ul (A)U2(B)), in(u2(A)u2(B)), ... ,in( un(A)un(B) )).

10.7. Von Neumann's Theorem For a square matrix, the trace of the matrix and the sum of all eigenvalues of the matrix are the same. Suppose we have two matrices A and B such that AB is a square matrix. The trace of the matrix AB is easy to compute. How is the trace related to the singular values of the individual matrices A and B? The von Neumann's Theorem is an answer to this question. But first, we need to prepare the reader for the von Neumann's Theorem (von Neumann (1937)). P 10.7.1 Let A E M m •n . Then tr(AX) = 0 for every matrix X of order n x m if and only if A = o. A stronger result than P 10.7.1 is as follows. P 10.7.2 Let A E Mn. Then tr(AX) = 0 for all Hermitian matrices X if and only if A = O. PROOF.

Note that any matrix X of order m x m can be written as

with both Xl and X 2 Hermitian. More precisely, one can take

Xl

= (1/2}(X + X*)

and

X2

= (1/2i}(X -

X") .

P 10.7.3 Let A E Mn. Then tr(AX) is real for all Hermitian matrices X if and only if A is Hermitian.


PROOF. Suppose

343

A = (aij) and X = (Xij) are Hermitian. Observe

that m

tr(AX) =

m

L L aijXji. i=l j=l

Since A and X are Hermitian, aii and Xii are real for each i. Consequently, aiiXii is real for each i. Let i =1= j. Write aij = a + ib and Xji = e + id, where a, b, e and d are real numbers. Then

aijXji

+ ajiXij = =

+ o'ijXji = (a + ib)(e + id) + (a - ib)(e - id) 2(ae - bd) + i(ad + be) - i(ad + be) = 2(ae - bd)

aijXji

which is clearly real. Consequently, tr( AX) is real. Conversely, suppose, tr(AX) is real for all Hennitian matrices X. Then tr(AX) = tr(AX) = tr«AX)*) = tr(X* A*) = tr(X A*) = tr(A* X)

=> tr«A - A*)X) = 0 for all Hermitian matrices X. By P 10.7.2, we have A - A* A is Hermitian.

= 0,

i.e.,

P 10.7.4 Let A E Mm be Hermitian. If tr(A) 2: Retr(AU) for all unitary matrices U, then A is non-negative definite. (The symbol Re stands as an abbreviation for "the real of part of.") PROOF. Let us perform a spectral decomposition on

A. Write

where At, A2, ..• ,Am are the eigenvalues of A and U1, U2,'" ,Um the corresponding orthonormal eigenvectors of A. Our goal is to prove that each Ai 2: O. Suppose not. Some eigenvalues are negative. Assume, without loss of generality, that

for some 1

~

r

~

n. Let

+ A2U2U2 + ... + ArUrU;,

B =

A1u1Uj

C=

-Ar+1Ur+1U;+1 - Ar+2Ur+2U;+2 - ... - AmUmU~,

344


U = UIU1..

+ U2U2.. + .. , + UrU.r -

Ur+l Ur*+l - Ur+2 Ur* +2 - ... - u m u·m'

Check that Band C are non-negative definjte, C =1= 0, A = B - C, U is Hermitian, and U is unitary. Further, AU = B + C. Observe that, by hypothesis,

tr(A) = tr(B) - tr(C) ~ tr(AU) = tr(B)

+ tr(C).

This inequality is possible only if tr( C) = O. Since C is non-negative definite, this is possible only if C = O. This is a contradiction. This completes the proof. Now we take up a certain optimization problem. This has an important bearing on the proof of von Neumann's Theorem. Before we present the optimization result, let us take a detour. Let B E Mm. The problem is to investigate under what condition the series

1m +B+B2

+ ...

converges absolutely, and if it converges, identify the limit. A full discussion was undertaken in Chapter 11. But we are looking for a simple, sufficient, and easily verifiable condition. Let M = maXl::;i,i::;m Ibijl. Observe that

(1) every entry in B is::; M in absolute value; (2) every entry in B2 is ::; mM2 in absolute value; (3) every entry in B3 is ::; m 2 M3 in absolute value, and so on. Consequently, the series 1m + B + B2 + ... converges absolutely if Ek>1 m k - 1Mk converges, or equivalently, if Ek>1 (mM)k converges. The geometric series converges if mM 0 such that both

1m

+ ifX

and

l::;i,i::;m

1m - iEX

are invertible for every -EO < E < EO. To see this, take B = iEX = (bij) with E real. Note that m~x Ibijl = IEIM. The series 1::;1,J::;m


345

1m +B+B2 + . . . converges absolutely to (1m - B)-1 = (1m -i€X)-1 if 1€lmM < 1. One can take €o = l/mM. In a similar vein, one can show that (1m + i€X) is invertible if lEI < Eo. Fix -EO < E < EO. Then it transpires that

and further, that V is unitary. Let us calculate

(1m

+ iEX)(lm -

+ iEX)(lm + iEX + (iEX)2 + ... ) 1m + 2iEX + 2(iEX)2 + 2(iEX)3 + .. .

iEX)-1 = (1m =

= 2(1m - iEX)-1 - 1m.

(10.7.1)

Likewise,

Observe that

which shows that V is unitary. We are now ready to establish the desired optimization result. P 10.7.5 Let A E Mm. Let U m be the collection of all unitary matrices of order m x m. Then sup Re tr(AU) UEU",

is attained at some matrix Uo E U m non-negative definite.

.

Further, AUo turns out to be

PROOF. The fact that the supremum is attained at some matrix in U m is not difficult to prove. This is a topological result. But the battle to show that AUo is non-negative definite is hard. We will present the proof in a series of steps.

346


1. Observe that the set U m is compact when it is viewed as a subset of an appropriate unitary space with the usual topology. Further, the real valued maps tr(AU) and Re tr(AU) as functions of U E U m are continuous. By a standard result in topology that the supremum of a continuous function on a compact is attained, it follows that the desired supremum is attained at some Uo E U m • 2. The next objective is to show that AUo is indeed Hermitian. By P 10.7.3, it suffices to show that tr(AUoX) is real for every Hermitian matrix X. Start with any Hermitian matrix X. Look up the discussion that followed P 10.7.4. There exists to > 0, such that 1m + itX

and

1m - itX

are both nonsingular for every -to < t < to. Further, the matrix

is unitary, where f(t, X) is a series of matrices involving t and X . See (10.7.1). Note that Re tr(AUoV) = Re tr(AUo) + 2tRe[i tr(AUoX)] + t 2 Re tr(AUof(t, X)) .::; Re tr(AUo), by the very definition of Uo, and V is unitary. This implies that

2tRe[itr(AUoX)] for every -to

< t < to.

If 0

+ t 2 Re tr(AUof(t, X))

< t < to,

.::; 0

we indeed have the inequality

2Re[itr(AUoX)] +tRe tr(AUof(t,X))'::;

o.

Taking the limit as t ! 0, we observe that Re[i tr(AUoX)] .::; o. Arguing in a similar vein for -to < t < 0, we can conclude that Re[i tr(AUoX)] ;::: O. Hence Re[i tr(AUoX)] = 0, from which it follows that tr(AUoX) is real. (Why?) This shows that AUo is Hermitian. 3. The final step is to show that AUo is non-negative definite. Note that by the very definition of Uo. Re tr(AUo) ~ Re tr(AUoV) for every


347

Hermitian matrix V . By P 10.7.4, AUa is non-negative definite. This completes the proof. This result is a little unsatisfactory. It uses a topological result to show the existence of the optimal unitary matrix Ua. But we do not know what exactly it is. Secondly, we do not know what the supremum of Re tr(AU), U E U fn actually is. We will ameliorate the deficiencies now. P 10.7.6

Let A E Mfn with the singular values

and singular value decomposition A = Pb.Q, where P and Q are unitary matrices and b. = diag{O"l (A), 0"2(A), ... ,00m(A)}. Let U m be the collection of all unitary matrices of order m x m. Then max Re tr(AU) = 2:ai(A),

UEU",

i=l

and the maximum is attained at Ua = Q* P*. (Ua need not be unique.) PROOF. Let U E U fn . Let us compute m

Re tr(AU) = Re tr(Pb.QU) = Re tr(b.QU P) = Re

2: ai(A)[QU P]ii, i=l

where [QU P]ii is the i-th diagonal entry of the unitary matrix QU P. Being a unitary matrix, I[QU P] iil 1 for each i. Consequently,

: =;

IRe tr(AU)1

m

m

i=l

i=l

::=;2: O"i(A)I[QU P]iil ::=;2: O"i(A) .

Let us compute specifically Re tr(AUa)

= Re tr(Pb.QQ* P*) = Re tr(b.QQ* P* P) = Re tr(b.) =

2: O"i(A). i=l

348


Incidentally, AUo = Pjj.P*, which is clearly non-negative definite. The proof is complete. P 10.7.5 and P 10.7.6 solve the same optimization problem. The proof of P 10.7.5 is originally due to von Neumann (1937). His proof is non-constructive but the methods used are fascinating. The proof provided under P 10.7.6 is constructive. Another nugget emerges from P 10.7.5 and P 10.7.6. Suppose VI is a unitary matrix such that AVt is non-negative definite. Then VI maximizes Re tr(AV) over all V E Um! This can be seen as follows. Since AUt is non-negative definite, its eigenvalues and singular values are identical. Consequently, m

tr(AVt} =

L CTi(AVt ). i=t

The singular values of AVt are precisely the positive square roots of the eigenvalues of (AVI)*(AVI) = Ui A* AUt. The eigenvalues of Vi A* AVt are identical to the eigenvalues of A * A. The singular vales of A are the positive square roots of the eigenvalues of A* A. Hence m

m

i=t

i=t

The avowed assertion now follows from P 10.7.6. We now come to the main result of this section. P 10.7.7 (Von Neumann's Theorem) Let A E Mm nand B E Mn,m be such that AB and BA are non-negative definit~. Let p = minim, n}, q = maxim, n}, and

be the singular values of A and B, respectively. Set

CTp+I(A) = CTp+2(A) = . .. = CTq(A) = 0 CTp+1(B) = CTp+2(B) = ... = CTq(B) = O.


Then there exists a permutation

T

349

of {I, 2, ... ,q} such that q

tr(AB) = tr(BA) = LCTi(A)CTr(i)(B). i=l

PROOF. The proof is carried out in several steps.

1. We work out a special case. Assume that m = n, A and B are nonnegative definite, and AB = BA. This implies that AB is non-negative definite. (Why?) We can diagonalize both A and B simultaneously. There exists a unitary matrix U such that

A ~1

where that

Qi'S

= U~lU*

= diag{ Q}, Q2,· ..

= U~2U*, ~2 = diag{.B}' .B2,'"

and B

,Qn},

,.Bn},

and .Bi's are the eigenvalues of A and B respectively. Note

n

= tr(~1~2) =

L

Qi.Bi.

i=l

Since the eigenvalues and singular values are identical for a non-negative definite matrix, we can write n

n

LQi.Bi = L CTi(A)CTr(i) (B), i=l

i=l

for some permutation T of {I, 2, ... ,n}. The statement of the theorem is valid for this special case. 2. The strategy for the general case is as follows . Assume, without loss of generality, that m ~ n. Let A and B be given as stipulated in the theorem. We will construct two matrices Ao and Bo each of order n x n with the following properties. a. Ao and Bo are non-negative definite. b. Ao and Bo commute. c. The eigenvalues of Ao are CT1 (A), CT2(A), . .. ,CTm(A), 0, 0, ... ,0 (n - m zeros) and the eigenvalues of Bo are CT1(B), CT2(B), ... , CTm(B), 0, 0, ... ,0 (n - m zeros). d. tr(AB) = tr(AoBo).

350


Now the conclusion of the theorem will follow from Step 1. 3. We want to simplify the hypothesis of the theorem. We would like to assume that m = n and that A and B commute. Let us see how this can be done. Let AI, A2, . .. ,Am be the eigenvalues of the m x m positive definite matrix AB. We claim that the eigenvalues of the n x n matrix BA are: AI, A2, .. ' ,Am, 0, 0, ... ,0 (n - m zeros). Let us prove a general result. Let A be a non-zero eigenvalue of the non-negative definite matrix AB with multiplicity t with the corresponding linearly independent eigenvectors Ul, U2, ... ,Ut. Then A is also an eigenvalue of BA with multiplicity at least t and BUl, BU2, ... ,But are the corresponding linearly independent eigenvectors. In fact, it will come out later that the multiplicity is exactly equal to t. First, check that

We now check that BUl, BU2, ... ,But are linearly independent. Suppose that (h (Bul) + ... + Ot(But) = for some scalars 01 , O2, ... ,Ot. Then we have

°

i=l

t

t

t

= l::0i(ABui)

= l::0i(AUi) = A l::0iUi,

i=l

i=l

i=l

which implies n

l::0iUi

= 0,

i=l

as A =1= 0. Since Ut, U2, ... ,Ut are linearly independent, it follows that 01 = O2 = ... = Ot = 0, from which we have the desired linear independence of But, BU2, ... ,But. In order to establish the claim , let PI, P2, ... ,Pr be the distinct eigenvalues of AB among AI, A2, ... ,Am. If no Pi = 0, the argument will be a little easier. We will take up the more difficult case: one of the eigenvalues of AB is zero. Assume that Pr = 0. We will produce two tables. The first table summarizes all the information about the eigenvalues of AB and the second table deals with the matrix BA, where m = tl + ... + t r .


Eigenvalues of AB

Multiplicity

351

Linearly independent eigenvectors

J..LI J..L2

(r-I)

(r-I)

(r-I)

J..Lr-I

t r- I

J..Lr = 0

tr

Eigenvalues of B A

Multiplicity

J..LI

tl

(I) (1) (1) BU I , BU 2 , .•• , BU t1

J..L2

t2

(2) (2) (2) BU I , BU 2 , • .. , BU t2

ul

" " , u tr _ 1

Immaterial Linearly independent eigenvectors

B U (r-I) ,u2 B (r-I) , • . ., B U (r-I) _

J..Lr-1 J..Lr = 0

,u2

I

tr

+ (n -

m)

tr

1

Immaterial

Since Rank(AB)=Rank(BA)=number of non-zero eigenvalues of AB= number of non-zero eigenvalues of BA, the second table follows. Consequently, the eigenvalues of BA are AI, A2, ... ,Am, 0, 0, ... ,0 (n - m zeros). There may be more zeros among A}, A2, . .. ,Am. Since AB is Hermitian, AB = U t::..U*, where t::.. = diag{>.I' A2,'" ,Am} and U is unitary. Also, there exists a unitary matrix V such that BA=V

[~

~]

V*.

Partition V = (VI !V2) where VI is of order n x m. Note that VI has orthonormal columns, i.e., "\.t;*VI = 1m.

352


Now

~

= U* ABU, which gives

BA

= VI~Vt = VIU* ABUVt·

Now define Al = Y* A and A2 = BY with Y Let us list the properties of Al and A2.

= UVt·

a. Each of Al and A2 is of order n x n. b. Al and A2 commute. As a matter of fact,

AIA2

= Y* ABY = BA

and

A2AI

= BYY* A = BA.

c. A}A 2 and A2Al are non-negative definite. d. The singular values of Al are 'A-I) = >'U"A-l which it follows that Bo is non-negative definite.

= >'(AU)-l,

from

{3. Ao and Bo commute. Note that AoBo = AB = >.Im = BoAo. "Y. The singular values of A and Ao are identical. The singular values of Band Bo are identical. 6. trCAB) = tr(AoBo). The goal is achieved.

Case 2. >. = O. We can proceed in exactly the same way as in Case 1. We will have the matrices Ao and Bo with the properties a, {3, "Y, and /j but with one exception. We do not know whether or not Bo is non-negative definite. We need to do some more work. Let J.Ll! J.L2, ... ,J.Lr be the distinct eigenvalues of Ao with multiplicities tl! t2, . " ,tr . If none of the eigenvalues of Ao is zero, then Ao is nonsingular and Bo = 0, which is non-negative definite. The desired objective is achieved. Assume that one of the eigenvalues of Ao is zero. Take J.Lr = 0, for example. Let us summarize some of this information in the following table, where m = tl + ... + t r . Eigenvalues of A

J.Lr

Multiplicity

Orthonormal eigenvectors

Space spanned

(r) (r) (r) u 1 ,u2 ,..., u tr

Let u be any eigenvector of A that corresponds to a non-zero eigenvalue J.L of A. We claim that Bu = O. Note that

0= ABu = B(Au)

= B(JLU) =

J.L(Bu).

356


Since J-L i= 0, it follows that Bu = O. Let W be the matrix composed of the orthonormal eigenvectors of A listed above in the order they were written down as columns. It is clear that W is a unitary matrix. Note that by what we have noted down about Bu for eigenvectors u of A that correspond to non-zero eigenvalues of A, W* Ao W is a block diagonal matrix with J-L i It. in the i-th diagonal block, i = 1, . .. ,r, and

for some matrix Eo of order tr x tr and appropriate dimensions for 0 matrices. The matrices W* Ao Wand W* Bo W would have met the requirements of Q, {3, 'Y, and {j under Case 1 but for the matrix Eo, which need not be non-negative definite. By P 10.7.5, there exists a unitary matrix Wo of order tr x tr such that Eo Wo is non-negative definite. Define

Aoo = W* AoW, and

0

v=[r Note that V is unitary. Let

Aoo

It2

0 0

0 0

0

I t"_ l

~J

= W* AoWV and Boo = W* BoWV.

The following properties of Aoo and Boo flow easily now.

Aoo and Boo are non-negative definite. {3. Aoo and Boo commute. In fact, AooBoo = O. 'Y. The singular values of Aoo and Ao are identical. The singular values of Ao and A are identical. The singular values of Boo and Bo are identical. The singular values of Bo and B are identical. Q.

Our mission is successful.


357

As a consequence of von Neumann's theorem, one can solve another optimization problem. The details follow. Let A E Mrn,n and B E Mn,rn.

P 10.7.8

(1)

Then sup

Re tr(AU BV)

UEUn,VEU",

is attained at some unitary matrices U (2)

= Uo

E Un and V

= Vo

E Urn.

further, p

max

UEUn,VEU",

Re tr(AU BV) =

L O"i(A)O"i(B), i=l

where p

= min{m, n}.

PROOF. One can offer a topological proof of (1). One needs to observe that the spaces of unitary matrices are all compact. We will prove (2). Let Uo and Vo be a solution to the problem in (1). Since

Re tr«AUoB)Vo) ~ Re tr«AUoB)V) for all unitary matrices V, it follows that AUoBVo is non-negative definite. See P 10.7.5. In a similar vein, since Re tr(AUoBVo) = Re tr«BVoA)Uo) ~ Re tr«BVoA)U) for all unitary matrices U, it follows that BVoAUo is non-negative definite. Let us focus on the matrices AUo and BVo. The two matrices (AUo)(BVo) and (BVo)(AUo) are non-negative definite. By von Neumann's theorem, there exists a permutation T of {I, 2, ... ,p} such that max

UEUn,VEU",

Re tr(AU BV)

= Re tr(AUoBVo) = tr(AUoBVo) p

=

L O"i(AUo)O"T(i) (BVo) i=l p

=

L O"i(A)O"T(i)(B). i=l

The proof will be complete if we can show that the identity permutation.

T

can be taken to be

358


Assume, without loss of generality, that m = n. In Step 2 of the proof of von Neumann's theorem, for the matrices AVo and BUo we can find two matrices Ao and Bo each of order m X m with the following properties.

(1) (2) (3) values

Ao and Bo are non-negative definite. Ao and Bo commute. The singular values of A and Ao are identical. The singular of Band Bo are identical.

Since Ao and Bo commute, we can find a unitary matrix W such that

W" AoW = diag{O'I (A), 0'2(A), ... 'O'rrI(A)}, W" BoW = diag{O'1'(I) (B), 0'1'(2) (B), ... 'O'1'(m)(B)}, for some permutation matrix T of {1,2, ... ,m}. Let Wi be the i-th column vector of W. Each Wi is a common eigenvector for both Ao and Bo for the eigenvalues O'i(Ao) and O'1'(i) (Bo), respectively. Let Wo be a unitary matrix with the following properties.

(Exercise: Construct Wo explicitly.) Let us determine the eigenvalues of AoWO' BoWo. Observe that

AoWO'BoWOWI

= AoWO'Bow1'-l(I) = AoWO' (0'1 (BO)W1'-l(I)) = 0'1 (Bo)AoWI = 0'1 (BO)'O'I(Ao)WI'

Thus O'I(Ao)O'I(Bo) is an eigenvalue of AoBo with an eigenvector WI. Likewise, it follows that the eigenvalues of AoBo are: O'i(Ao)O'i(Bo), i = 1,2, ... ,m. Thus m

L O'i(Ao)O'i(Bo) = tr(AoWO' BoWo) i=I

~

=

max

UEU""VEU",

Re tr(AoV BoV)

m

m

i=I

i=I

L O'i(Ao)O'1'(i) (Bo) ~ L O'i(Ao)ai(Bo).


359

Consequently,


Complements 10.7.1 Let A E Mn have singular values O'n ::; ... ::; 0'1. Denote by ri the Euclidean norm of the i-th row of A, i = 1, ... , n, and by Rl ::; R2 ::; .. . ::; Rn the ordered values of rio Show that k

k

i=l

i=l

L O'~-i+l ::; L R; for k = ~,2, ... , n. Write down a similar upper bound for norms of columns. 10.7.2 (Majorization Theorem) Let A E Mn be Hermitian. The vector of diagonal entries of A majorizes the vector of eigenvalues of A. [Hint: The theorem can be proved by induction by assuming that the result is valid for Hermitian matrices of dimension k for all k ::; n - 1 and using the Interlace Theorem (P 10.2.1). 10.7.3 Let A and B E Mn be Hermitian matrices with ordered eigenvalues >\} 2': ... 2': An and /-Ll 2': ... 2': /-Ln· Further let VI 2': ... 2': Vn be the eigenvalues of A-B. Show that the vector (AI - ILl, ... , An - /-Ln) is majorized by the vector (VI, ... , vn ). 10.7.4 (Weyl's Theorem) Let A, BE Mn be Hermitian and pCB), the rank of B, be less than or equal to 1' . Let the eigenvalues Ai(A), Ai (B) and Ai(A + B) be arranged in decreasing order. Then (1) Ai(A + B) ::; Ai-r(A) ::; Ai-2r(A + B), i = 2r + 1, ... , n. (2) Ai(A) ::; Ai-r(A + B) ::; Ai-2r(A), i = 21' + 1, ... , n . (3) If A = UAU· with U = (ull· . . Iu n ) E Mn unitary and A = diag(Ab'" , An) with Al 2': ... 2': An and if

360


10.7.5 (Analogue of Weyl's result of Complement 10.1.2 for singular values). Let A,B E Mm,n, q = min{m,n} and ai denote singular values. Show that, using 10.7.4,

In particular,

+ B) ::; al (A) + al (B) aq(A + B) ::; min{aq(A) + al(B),

al (A

10.7.6

al(A)

+ aq(B)}.

Let A, BE Mm,n and q = min{m, n} . Show that

10.7.7 Let A E Mn be a Hermitian matrix with non-negative eigenvalues >'1 ~ ... ~ >'n ~ O. Show that for each r = 1, ... ,n, the product >'n>'n-t ... >'n- r+ t is less than or equal to the product of the r smallest main diagonal entries of A.

Note: References to material in this Chapter are: Horn and Johnson (1985, 1991), Rao (1973c) and von Neumann (1937).

CHAPTER 11 MATRIX APPROXIMATIONS

The basic line of inquiry in this chapter proceeds along the following lines. Let A be a given collection of matrices each of order m x n. Let B be a given matrix of order m x n. Determine a matrix A in A such that A is closest to B. The notion of one matrix being too close to another can be sanctified with the introduction of the notion of a norm on the spaces of matrices. In this chapter, we will introduce the notion of a norm on a general vector space and establish a variety of matrix approximation theorems.

11.1. Norm on a Vector Space Let V be a vector space either over the field C of complex numbers or the field R of real numbers. The notion of a norm on the vector space V is at the heart of many a development in this chapter. In what follows, we will assume that the underlying field is the field C of complex numbers. If R is the underlying field of the vector space, we simply replace C by R in all the deliberations with obvious modifications. In Chapter 2, we have already introduced the concept of a norm arising out of an inner product. The definition of norm introduced here is more general which need not be generated by an inner product. DEFINITION 11.1.1 . A map II . II from V to R is said to be a vector norm on V if it has the following properties.

(1) IIxll 2: 0 for all x in V and IIxll = 0 if and only if x = O. (2) II ax II = lalllxll for all x in V and a E C. (3) IIx + yll ::; IIxll + Ilyll for all x and yin V. The pair (V, 11 · 11) is called a normed vector space. Using the norm II . II on the vector space V, one can define a distance function d(·, .) 361

362


between any two vectors of V. More precisely,

d(x,y) = I\x -

ylI,

X,y E V.

In view of the definition of the distance d(·, .), one can interpret IIxli as the distance between the vectors x and 0, or simply, the length of x. Property (3) is often called the triangle inequality satisfied by the norm and it implies that d(x, y) :S d(x, z) + d(z, y) for any three vectors x, y, and z in the vector space V. Some examples are in order. EXAMPLE 11.1.2. Take V = en. Let x = (XI, ... following maps are all norms on the vector space V:

(1) (2) (3) (4)

,xn) E V. The

IIxll oo = max1.BII = I>.IIIBII, that Ps(A) ~ IIAII. ~

A matrix norm is useful in investigating the existence of an inverse of a matrix. First, we need to examine the precise relationship between the individual entries of a matrix and its matrix norm. Then we need a criterion for the convergence of a matrix series. Let Iij be the matrix of order n x n such that the (i,j)-th entry of Iij is unity and the rest of the entries are all zeros. Let II ·11 be a matrix norm on Mn. Let Let A = (aij) E Mn. Then one can verify that IijA1ij andj. P 11.2.5

laijl ~ BIIAII

Let A E Mn and for all i and j.

II . II

= aijlij

for all i

a matrix norm on Mn. Then

366 PROOF.


Note that

laijlllIijll = IlaijIijll = IIIijAIijll ::; II I ijIl2I1AII, => laijl ::; II I ijllllAIl ::; OIlAII· P 11.2.6 Let A E M n , 11·11 a matrix nonn on M n , and {ad, k ~ 0 a sequence of scalars. Then the series Lk>O akAk converges if the series Lk~O laklllAlik of real numbers converge;' (By convention, AO = In.) PROOF.

Let Ak = (a~~») for each k ~

o.

By P 11.2.5,

foralliandj,andk~ 1. SincellAkll::; IIAllk,theseriesLk~olaklla~7)1 of real numbers converges. Hence Lk~O aka~7) converges. Now we can settle the existence of inverse of a matrix.

P 11.2.7 Let A E Mn and 11·11 a matrix norm on Mn. If IIIn -All < 1, then A-I exists and is given by the series

A-I = :l)In - A)k. k~O

PROOF. From P 11.2.6, note that the series Lk>o(In - A)k converges. Let N be any positive integer. Then -

N

N

A(2:(In - A)k) = (In - (In - A)) 2:(In - A)k k=O k=O = In - (In - A)N+I. Since IIIn - All < 1, (In - A)N+I converges to 0 as N quently, A(2:(In - A)k) = In, k~O

from which the desired result follows.

--+ 00.

Conse-

Matrix Approximations

367

Not all norms are matrix norms. It will be instructive to check how many of the norms introduced in Example 11.2.1 pass muster. See Complements 11.2.1 to 11.2.3. We need more examples of matrix norms. Instead of plunking down some examples, it would be useful to develop a method of generating a variety of norms on Mn using norms on the vector space en. Start with any norm II . II on en. For each A E M n, define

IIAllin =

IIAxl1 -II -II . xECn,x#O X

(11.2.1)

sup

As usual, the vectors x E en are regarded as column vectors so that matrix multiplication Ax makes sense. If we can show that IIAllin is finite, we can say that (11.2.2) for every vector x E en. The eventual goal is to demonstrate that the map II . lIin is a matrix norm on Mn. If this is the case, one can call 1I·lIin, the induced matrix norm in Mn induced by the norm 11·11 on en. The letters "in" in the norm are an abbreviation of the phrase "induced norm". It is clear that IIAllin is non-negative. Observe that

IIAllin

sup

=

xEcn,ilxil=l

IIAxll·

Using topological arguments, one can show that there exists a vector Xo E en (depending on A) such that

IIxoll = 1

and

IIAllin = IIAxoll·

(11 .2.3)

This demonstrates that IIAllin is finite. (The topological arguments use the facts that the map II . II is a continuous function from en to Rand that the set {x E en : Ilxll = 1} is a compact subset of en in the usual topology of en.) If A = 0, then IIAllin = O. Conversely, suppose IIAllin = O. This implies that Ax = 0 for every vector x E en with IIxll = 1. Hence A = O. (Why?) If a is any complex number, then

lIaAllin

=

sup xEcn,ilxil=l

lIaAxl1

=

lal

sup xEcn,ilxil=l

IIAxl1 = lalllAII·

368


We now set upon the triangle inequality. Let A and B E Mn· Note that for each vector x E en, II(A + B)xlI ::; IIAxll + IIBxlI. From this inequality, it follows that IIA + Bllin ::; IIAIlin + IIBllin. Finally, we need to show that IIABllin ::; (IiAllin)(IIBllin). By what we have pointed out in (11.2.3), there exists a vector Xo (of course, depending on AB) such that Ilxoll = 1 and

= IIABxoll = IIA(Bxo)1I ::; IIAllinllBxoll ::; IIAllinllBllinlixoll = IIAllinllBllin.

IIABllin

Thus we have shown that II· II in is indeed a matrix norm. The definition of the induced norm is something one introduces routinely in functional analysis. The matrix A from Mn can be viewed as a linear operator from the normed linear space (en, 11·11) to the normed linear space (en, II . II). The definition of the induced norm of A is precisely the operator norm of A. There is no dearth of matrix norms. Every norm on en induces a matrix norm on Mn. Some examples are included below.

P 11.2.8

For A = (aij) E M n, define n

IIAlloo,in

= l~l!fn L

- - j=l

laijl·

(First we form the row sums of absolute values of entries of A and then take the maximum of the row sums to compute II Alloo,in') Then 1I'lIoo,in is a matrix norm on Mn. PROOF. The main idea of the proof is to show that II . lloo,in is the induced matrix norm on Mn induced by the Loo-norm, II . 1100 on en. For any given matrix A E M n, let us compute IIAxiloo on en. Let x' = (Xl, X2, ... ,Xn) E en be such that Ilxli oo = maxl~i~n IXil = l. Then n

IIAxii oo

n

= l~tfn I L aijxjl ::; l~~~ L - -

j=l

- - j=l

laijllxjl


369

(11.2.4)

Let

n

n

m~ "!aij! = "!akj! for some L.J L.J - - j=1 j=1

l'2, .. . ,>'n be the eigenvalues of A. By Schur triangularization theorem (See Chapter 5), there exists a unitary matrix U and an upper triangular matrix D = (d ij ) such that A = U DU* and dii = >'i for all i. For t > 0, let G t = diag{t, t 2, . .. ,tn}. Observe that GtDG t l is upper triangular and in fact, given by

GtDG t l

dl2/t

0

0

>'3

. .. / d2n /t n- 2 . . . d 3n /t n- 3

o

0

0

...

o >'2

= [

d 13/ t2 d23 /t

>'1

.

.

.

...

dIn/tn_II

.

.

>'n

Choose t > 0 such that the sum of all off-diagonal entries of GtDG t l is < E. Define a new matrix norm II . II on Mn by

Recall the structure of the norm 1I-III,in from P 11.2.11. By P 11.2.14, II . II is indeed a matrix norm on Mn. Let us compute

IIAII = =

l l II{UG t )-1 A(UG t ) IIt. in

= IIGtU*AUGtllh ,in

IIGtDGtllh,in < l~a.2'n I>'jl + E = _J_


Ps(A)

+ E.


373

We are now ready to provide the connection between the spectral radius of a matrix and the limiting behavior of powers of the matrix. P 11.2.16

Let A E Mn. Then Ak

-+

0 as k

-+ 00

if and only if

Ps(A) < 1. PROOF. Suppose Ps(A) < 1. By P 11.2.15, there exists a matrix norm II . II on Mn such that IIAII < 1. (Why?) Consequently, Ak converges to zero as k -+ 00. See the discussion following Definition 11.2.2. Conversely, suppose Ak converges to 0 as k -+ 00. There is nothing to prove if Ps(A) = o. Assume that Ps(A) > o. Let A be any non-zero eigenvalue of A. Then there exists a non-zero vector x in such that Ax = AX. Consequently, AkX = Akx. Since Akx converges to o as k -+ 00, and x i= 0, it follows that Ak converges to 0 as k -+ 00. But this is possible only when IAI < 1. Hence Ps(A) < 1. The spectral radius of a matrix A has a close connection with the asymptotic behavior of IIAkll for matrix norms II . II. The following result spells out the precise connection.

en

P 11.2.17

Let

II . II

be any matrix norm on Mn. Then for any

A2, ... ,An are the eigenvalues of A, then At, A~, ... ,A~ are the eigenvalues of Ak for any positive integer k. Consequently, Ps(Ak) = [Ps(A)]k. By P 11.2.4, Ps(Ak) ~ IIAkll. Hence Ps(A) ~ (IIAkll)l/k. Let c > 0 and B = (Ps(A) + c)-l A. The matrix B has spectral radius Ps(B) < 1. By P 11.2.16, IIBkl1 -+ 0 as k -+ 00. We can find m ~ 1 such that IIBkll < 1 for all k ~ m. Equivalently, IIAkll < (Ps(A) + cl for all k ~ m. This means that (IIAkll)l/k ~ Ps(A) + c for all k ~ m. Thus we have Ps(A) ~ (IIAkll)l/k ~ Ps(A) + c for all k ~ m. PROOF. If AI,

Since c

> 0 is arbitrary, the desired conclusion follows.

Complements 11.2.1 Show that the Leo-norm on Mn is not a matrix norm (n ~ 2). However, show that the norm defined by

374

MATRIX. ALGEBRA THEORY AND APPLICATIONS

for A = (aij) E Mn is a matrix norm on Mn. See Example 11.2.1. 11.2.2 Show that the F'robenius norm on Mn is a matrix norm on Mn. See Example 11.2.1. 11.2.3 Show that the Lp-norm on Mn is a matrix norm on Mn if and only if 1 ~ P ~ 2. (An outline of the proof. The objective of this exercise is to examine which norms in Example 11.2.1 are matrix norms. If p = 1 or 2, it is easy to demonstrate that the Lp-norm is indeed a matrix norm. Let 1 < p < 2. Determine q such that 1P + 1q = 1. Note that 0 < p - 1 = ~ < 1. Let A = (aij) and B = (b ij ) E Mn. Then

n

IIABII~ =

n

n

L L IL airbrJIP i=1 j=1 r=1

$

ttl (t.lai.IP)! (~?';Iq) l tt.lai.lpt(t(lb';IP)!)'

$

IIAII~

$

(by HOlder's inequality)

(t t Ib,;IP)

$

(IIAII~) (II BII~ )

The inequality (a 8 + b8) ~ (a + b)8 for any a ~ 0, b ~ 0, and () ~ 1 is at the heart of the last step above.) 11.2.4 For any norm 11 · 11 on en, show that IIInllin = 1. 11.2.5 For any matrix A = (aij) in Mn, show that IIAllt.in = IIA* lloo,in, where A * is the adjoint of A. 11.2.6 If A is symmetric, show that Ak converges to 0 if and only if Ps(A) < 1 using the spectral decomposition of A.

11.3. Unitarily Invariant Norms Let m and n be two positive integers. Let Mm,n be the collection of all matrices of order m x n with complex entries. In many statistical applications, we will be concerned with data matrices with m being the sample size and n the number of variables. Generally, m will be much


375

larger than n. We do not have the convenience of matrix multiplication being operational in the space Mm,n' Consequently, the idea of a matrix norm does not make sense in such a general space. In this section, we will look at a particular class of norms on the space Mm,n and find a way to determine the structure of such norms. First, we start with a definition. 11.3.1. A real valued function II . II on the vector space M m •n is said to be a unitarily invariant norm and denoted by 1I·lIui, if it has the following properties. DEFINITION

(1) IIAII ~ 0 for all A E Mm ,n' (2) IIAII = 0 if and only if A = O. (3) lIoAIl = 101 II All for every 0 E C and A E Mm,n . (4) IIA + BII ~ IIAII + IIBII for all A and Bin Mm ,n' (5) IIU AVII = IIAII for all A E Mm,n and unitary matrices U and V of orders m x m and n x n, respectively. The first four properties are the usual properties of a norm. The fifth property is the one which adds spice to the theme of this section. IT M m •n is a real vector space we use the term orthogonally in the place of unitarily invariant norm. We will discuss in an informal way how such an invariant norm looks like. Let A E Mm,n ' Let CTl(A) ~ CT2(A) ~ . . . ~ CTr(A) be the singular values of A, where l' = min{m, n}. By the singular value decomposition theorem, there exist two unitary matrices P and Q of orders m x m and n x n, respectively, such that A = PDQ , where if l' = m, if r = n, with Dl = diag{CTl(A), CT2(A), ... ,CTr(A)} and O's are the appropriate zero matrices. Then IIAllui = IIPDQllui = IIP* PDQQ*lIui = IIDllui . Note that IIDII is purely a function of the singular values of A. Let us denote this function by cp, i.e.,

IIDllui = cp(CTl (A), CT2(A), ... ,CTr(A)). The question then arises as what kind of properties the function cp( .) should possess. One thing is pretty clear. The function cpO must be a

376


symmetric function of its arguments. We will make these notions more precise shortly. Before determining the structure of unitarily invariant norms, let us look at some examples. EXAMPLE 11.3.2. The F'robenius norm on the vector space Mm,n is unitarily invariant. Let A = (aij) and 0"1 2: 0"2 2: . . . 2: 0" r 2: 0 be singular values of A, where r = min{m,n}. Then m

IIAIIF

n

= (L L i=1

laijI2)1/2

= (Tr(A* A))1/2

j=1 r

= (Sum of all the eigenvalues of A* A)I/2 = (LO"~)1/2.

i=1

If U and V are unitary matrices of order m x m and n x n, respectively, then

IIU AVIIF

= [tr((U AV)*(U AV)W/2 = [tr(V* A*U*U AVW/2 = [tr(V* A* AVW/ 2 = [tr(A* AW/2.

The eigenvalues of A* A and V* A* AV are the same for any unitary matrix V. (Why?) Thus the F'robenius norm on Mm,n is seen to be unitarily invariant. EXAMPLE 11.3.3. The spectral norm on Mm,n is also unitarily invariant. In Section 11.2, we have defined the spectral norm on the vector space Mn. It can be defined in two equivalent ways on the space Mm,n too. Let A E Mm,n. One way is:

IIAlis

II Axll2

= xEC",xtO sup -11-11-· x 2

Another way is: IIAlis = 0"1 = max{O"I, 0"2, ... ,O"r} = (Ps(A* A))1/2. One can check that both approaches lead to the same answer. Note that Ps(A* A) is the spectral radius of the matrix A* A. One can also check that the spectral norm is unitarily invariant. Now we take up the case of determining the structure of unitarily invariant norms. Let P n be the group of all permutations of the


377

set {I, 2, ... ,n}. Every member 7r of P n is a one-to-one map from {I, 2, . .. ,n} to {I, 2, ... ,n} and is called a permutation. For each x' = (Xl, X2, ... ,Xn ) E Rn and 7r E P n, let x~ = (X 1T (I) , X1T(2) , ... ,X1T (n»). The vector X 1T is an n-tuple which permutes the components of x. We want to introduce another entity. Let I n denote the collection of all n X n diagonal matrices with each diagonal entry being equal to either +1 or -1. We are ready to introduce a special class of functions. DEFINITION 11.3.4. A real valued function cP from Rn to R is said to be a symmetric gauge function if it has the following properties. (1) cp(x) > 0 for all X E Rn with X =1= o. (2) cp(ax) = lalcp(x) for all X E R n and a E R. (3) cp(x + y) :s; cp(x) + cp(y) for all X and yin Rn. (4) CP(X1T) = cp(x) for all X in R n and 7r E P n . (5) cp(Jx) = cp(x) for all X in Rn and J E I n . The first three properties merely stipulate that the function cp be a norm on the real vector space Rn. The fourth property stipulates that the function cp be symmetrical in its arguments (permutation invariance). The fifth property exhorts the function to remain the same if signs are changed at any number of arguments. The usual Lp-norms on the vector space Rn are some good examples of symmetric gauge functions. The sum of any two symmetric gauge functions is one of the family. A positive multiple of a symmetric gauge function retains all the properties of a symmetric gauge function. Here are some more examples. EXAMPLE n R , let

11.3.5. Let 1 :s; k:S; n be fixed. For x' = (XI,X2, ... ,xn ) E

One can check that CPk(·) is indeed a symmetric gauge function. If k = 1, then CPkO is the usualioo-norm on Rn,. If k = n, then CPkO is the usual iI-norm on Rn. We establish some simple properties of symmetric gauge functions . P 11.3.6 Let cP be a symmetric gauge function. (1) If x' = (X},X2, ... ,xn ) ERn and O:S; PI,P2,··· ,Pn

CP(PIXI,P2X2, ... ,Pnxn)

:s; cp(XI,X2, ...

,X n ).

:s;

1, then

378


(2) If Xi ~ 0 for i = 1, ... ,n and YI ~ XI,Y2 ~ X2,' " ,Yn ~ Xn, then cp(Xl, X2, · · . , xn) :::; cp(YI' Y2, ... , Yn). (3) There exists a constant k > 0 such that for all (Xl, X2, .. , , Xn)' E

Rn,

As a matter of fact, k = cp(l, 0, 0, ... ,0). (4) The function cP is continuous. PROOF. (1) Assume that 0 :::; Pi < 1 for exactly one i and Pj = 1 for the rest of j's. For simplicity, take i = 1 and write PI = p. We will show that

The general result would then follow in an obvious way. Let

Note that u + v = (pXl, X2, X3, ... ,xn). By Properties (2), (3), and (5) of a symmetric gauge function,

CP(PXI, X2, ... , xn) = cp(u + v) :::; cp(u) + cp(v) l+p I-p = (-2-)CP(x) + (-2-)CP(x)

= cp(x) .

(2) Since 0 :::; Xi :::; Yi, we can write Xi = PiYi for some 0 :::; Pi :::; l. Now (2) follows from (1). (3)

Observe that

cp(Xl,X2, ... ,xn) =CP(XI +0,0+X2,0+X3, ... ,0+Xn ) :::; CP(XI, 0, 0, ... ,0) + cp(O, X2, X3, ... , xn)

:::; IX llcp(1, 0,0, ...

,0) + cp(O, X2, 0, ... ,0)

+cp(0,0,X3,X4, ... ,xn)


379

n

~ k(Llxil). i=l

Also, since for each i, (0,0, ... ,0, IXil. 0, ... ,0) ~ (lXII, IX21, ... ,Ixnl} coordinate-wise, by (2),

IXil cp(O, 0, ... ,0,1,0, ... ,0) = cp(O, 0, ... ,0, lXii, 0, ... ,0) ~ CP(IXII, IX21,··· , IXnl) = cp(xt, X2, ... , xn), which implies that CP(XI, X2, ... , xn) ~ klxi I for each i. Hence

CP(XI,X2, ... ,xn) ~ k( m~x IXil). l~t~n

(4)

Since cP is a norm,

ICP(XI,X2, ... ,x n ) -CP(YI,Y2, .. . ,Yn)1 n

~ CP(XI - YI, X2 - Y2,··· , Xn - Yn) ~ k(L IXi - Yil). i=l

The continuity of cP now follows. We have already indicated that a unitarily invariant norm of a matrix is basically a function of the singular values of the matrix. We will show that any such norm is generated by a symmetric gauge function.

P 11.3.1 Let 1I·lIui be a unitarily invariant norm on the vector space Mm,n' Assume, for simplicity, m ~ n. For each (Xl, X2, ... , Xm) E Rm, let

X

° ),

= (mxm D: mx(n-m)

and CP(XI,X2, ... ,xm)

= IIXllui,

where D = diag{ Xl, X2, ... ,xm }. Then cP is a symmetric gauge function on Rm. Conversely, if cP is a symmetric gauge function on Rm, then the map II . II defined by

IIAII =

cp(uI(A), u2(A), ... , um(A)), A

E

Mm,n

380


is a unitarily invariant norm on the vector space Mm,n· Note that the eigenvalues of X X* are IX112, IX212, ... ,l xmI 2. Consequently, the singular values of X are IX11,lx21, ... ,Ixml. It is a routine job to check that cp is a symmetric gauge function. Conversely, let cp be a given symmetric gauge function. The singular values of a matrix can be written in any order. Since cp is symmetric, it is clear that the map II . II induced by cp is well defined and has the following properties. PROOF.

(1) IIAII ~ 0 for all A E Mm,n. (2) IIAII = 0 if and only if A = o. (3) lIaAII = lalllAl1 for all a E C and A E Mm,n. (The singular values of aA are laIO"I(A), ... ,laIO"m(A).) (4) If U and V are unitary matrices of order m x m and n x n, respectively, then IIU AVII = IIAII. The critical step would be to show that the map II . II satisfies the triangle inequality. Let A and B E Mm,n with singular values

Define

O"(A) = (0"1(A)'0"2(A), ... ,00m(A))', O"(B) = (O"I(B), 0"2(B), ... ,00m(B))', O"(A + B) = (0"1 (A + B), 0"2(A + B), ... , O"m(A + B)). By P 10. 1.3, the vector O"(A + B) is weakly majorized by the vector O"(A) + O"(B). By P 9.3.8 and Complement 9.3.2, there exists a doubly stochastic matrix S of order m x m such that

O"(A + B) :S S(O"(A)

+ O"(B))

coordinate-wise. Every doubly stochastic matrix can be written as a convex combination of some permutation matrices, i.e.,

where O/s are non-negative, 2:;=10i matrices. Observe that

IIA + BII =

=

cp(O"(A + B)) :S cp(S(O"(A)

1, and Pi'S are pennutation

+ O"(B)))

(by P 11.3.6 (2))


~ 1'. This proves (1). The sufficiency part of (2) is trivial. But the proof of necessity is a bit involved. For details the reader referred to Rao (1980).

11.3.6 If A, B and A - B are Hermitian and non-negative definite matrices of order m and if B is utmost rank k, then (1) Ai(A - B) ~ Ak+i(A).

(2) A necessary and sufficient condition for equality in (1) to hold for all i is that B = Al (A)VI V{ + ... + Ak(A)Vk V£ where VI, ... , Vk are the first k eigenvectors of A.


383

11.4. Some Matrix Optimization Problems

In this section we consider some matrix optimization problems which are useful in solving matrix approximations considered in the next section. For simplicity of notation, we will consider matrices with real entries only. The results readily extend to complex matrices if we replace the phrase "transpose" by "conjugate transpose" , and the phrase "symmetric" by "Hermitian." P 11.4.1 Let A E Mn be a symmetric matrix with eigenvalues Al ~ A2 ~ ... ~ An and Xl, ... ,Xn be some orthonormal vectors. (Note that we are writing the eigenvalues of A in decreasing order of magnitude.) Then k

k

(1) Lx~Axi ~ LAi' k = 1, ... ,n - 1, i=l n

i=l n

(2) Lx~Axi = LAi. i=l

i=l

PROOF. Let Xk = (xII .. . Ixk) and A = PAP', be the spectral decomposition of A, where A = diag(AI,'" ,An) and P an orthogonal matrix. Then for any 1 ~ k ~ n, k

Lx~Axi = tr(X~AXk) = tr(X~P A P'Xk) i=l n

= tr(AP' XkX~P)

= tr A (QQ') = LAiqii I

where Q = P' Xk so that Q'Q = I (of order k x k) and qii is the i-th diagonal element of the idempotent matrix QQ'. When k = n, QQ' = I (of order n x n), which proves (2). When k ~ n - 1, we have n

k

k

L Aiqii ~ L Aiqii + (k - L Qii)Ak+1 i=l

i=l

i=l

k

= L(Ai - Ak+dQii i=l

+ kAk+1

384

MATRIX ALGEBRA THEORY AND APPLICATIONS k

k

~ 2)Ai - Ak+1) + kAk+1 i=l

= LAi' I

n

2: qii = 1, tr (QQ') = tr(Q'Q) = k,

and q;; ~ 1. i=l In view of results {l) and (2) of P 11.4.1, we may say that the eigenvalues Ai majorize X~AXi and hence, in particular, that Ai rnajorize the diagonal elements of the symmetric matrix A. More formally, let x' = (x~ AX1, X~AX2' . .. ,x~Axn), y' = (AI, A2, . .. ,An), and z' = (all, a22, ... ,ann), where aii is the i-th diagonal entry of A. Then since

x

«y

and

z« y.

The matrix A need not be non-negative definite. The equality in (1) of P 11.4.1 is attained for a given k, when xl. ... ,Xk are the eigenvectors of A corresponding to the eigenvalues At, ... ,Ak. The result of P 11.4.1 readily extends to the maximization of a bilinear form as stated in the next proposition. P 11.4.2 Let zi = (xi: yD be mutually orthonormal vectors, i = 1, ... ,p, where Xi is an m-vector and Yi is an n-vector. Further, let A E Mm,n have singular values a1 ~ a2 ~ ... ~ a r > a r +1 = a r +2 = ... - a p = 0, where p = min(m, n) and A has rank 1·. Then k

k

L2x~AYi < Lai, k = 1, ... ,po i=l

i=l

The maximum is attained when Xi and Yi are the singular vectors of A associated with the singular value ai, i = 1, ... ,k. PROOF.

Let M =

[~, ~]

which is symmetric of order (m + n) x (m + n). Note that a1, ... , a p are the p largest eigenvalues of the symmetric matrix M. Then for any 1 ~ k ~ p, k k k L2x~AYi = Lz~Mzi < Lai' i=l i=l i=l


385

by P 11.4.1, which proves the first part of the result. The second part is left to the reader as an exercise. (It will be instructive to write down all the eigenvalues of M.) P 11.4.3 Let A = (aij) E M m •n with singular values a1 ;::: a2 ;::: ... ;::: a r > a r +1 = a r +2 = ... = a p = 0, where p = min{m,n}. Then k

k

(1) La2n , < La;, i=l i=l k

k = 1, ... ,p,

k

(2) Laii < Lai, i=l

k=l, ... ,p.

i=l

Equality in (1) holds if and only if the leading k x k principal submatrix of A is diagonal and laiil = ai, i = 1, ... , k. Equality in (2) holds when equality holds in (1) and aii ;::: 0, i = 1, ... ,k. PROOF.

We have k

k

k

n

"" a~, < ""("" a ,) ~U-~~tJ 2

i=l

i=l j=l

k

= "" e~AA'e;,-~t' < "" a~ ~t i=l

i=l

where the second inequality follows from P 11.4.1 and ei is the i-th column of 1m. The inequality (1) is proved. The inequality (2) follows similarly from P 11.4.2 setting Xi = 2- 1/ 2 ei and Yi equal to 2- 1 / 2 (i-th column of In). P 11.4.4 (Abel's identity). Let a1, ... ,an and b., ... ,bn be two finite sequences of scalars. Then

PROOF.

The right-hand side of the above identity may be written as n-1 i n-1 i n L(ai Lbj ) - L(aH1 Lb j ) + an Lbj i=l j=l i=l j=l j=l

386


n-l i n i-I n = L(ai Lbj ) - L(ai Lbj ) + an Lbj j=l i=l j=l i=2 j=l n-l i i-I n-l n = Lai(Lbj - Lbj ) + al bl - an L bj + an Lbj i=2 j=l j=l j=l j=l n-l n = L aibi + alb l + anbn = L aibi' i=2 i=l

A more general version of the following result has been established in Chapter 10. In the special case reported here, the proof is simple and instructive.

P 11.4.5 (von Neumann (1937)). Let A and B E Mn be both symmetric with eigenvalues Al 2: ... 2: An and J-LI 2: ... 2: J-Ln. Then n

n

i=l

i=l

Equality holds on the right when B n

left when B =

=

n

E J-LiPiP!

E J-Ln-i+l Pi P!, where Pi

i=l eigenvalue Ai, i = 1, ... ,n.

and equality on the

i=l

is an eigenvector of A for the

PROOF. Let A = P /\ pi be the spectral decomposition of A with /\ = diag(Ab ... ,An) and P orthogonal. Then

tr(AB} = tr(P /\ pi B) = tr(/\p l BP) = EAibii, where bii is the i-th diagonal element of pi BP. Using Abel's identity (P 11.4.4), we obtain


387

where the inequality follows from P 11.4.1, since P' BP and B have the same eigenvalues. The inequality on the left-hand side follows by replacing B by -B in the inequality on the right-hand side. P 11.4.6 Let A E Mm,n and B E Mn,m, with singular values 01 ;:::: and {31 ;:::: {32 ;:::: •.. ;:::: {3p, respectively, where p = min{m, n}. Then .•. ~ op

p

-L

p

oi{3i

~ tr{AB} ~

i=1

L

{11.4.1}

oi{3i.

i=1 p

Equality holds on the right when B

= 2: {3iQiP:

p

left when B =

2: {3i {-Qi}P:,

i=1 A for the singular value PROOF.

and equality on the

i=1

Oi,

i

where Pi and Qi are singular vectors of

= 1, ...

, p.

Note that the eigenvalues of the symmetric matrix

[~, ~]

°

°;: :

are 01 ;:::: ... ;:::: Op ;:::: = ... = -op ;:::: ... > number of O's is 1m - nl. By P 11.4.5, we have

-01,

where the

[0 °A] [0B °B']

2tr{AB} = tr A' p

~ LOi{3i

p

+ L{-Oi)(-{3i) =

p

2L o i{3i.

{11.4.2}

1

The rest of the results are easily established.

Complements 11.4.1 Let A E Mm,n and B E Mm,n with singular value decompositions P 6. 1 Q' and R6. 2 S', singular values 01 ;:::: ... ;:::: Or and {31 ;:::: ... ;:::: {3rJ respectively, where r = min{m, n}. Show that

where U and V are any unitary matrices. Further, show that the upper bound is attained when U = QR' and V = SP'. Show also that, if

388


m = n, tr(AU) ~ 01 + ... + Or for any unitary matrix U with the upper bound attaining when U = QPl. (The results follow from von Neumann's propositions P 11.4.5 and P 11.4.6.) 11.4.2 (Ky Fan and Hoffman (1955)) Let A be a square matrix. Show that ,X; (A + A') ~ 20"i(A) = 2[Ai(A' AW/ 2 where Ai(A + A') is the i-th largest eigenvalue of (A + A') and O"i(A) is the i-th singular value of A. 11.4.3 Let A E Mm,n and q = min{m, n}, and Nk(A) = 0"1 (A) + ... + O"k(A) denote the sum of the k largest singular values of A. Show that NkO is a norm on Mm,n for k = 1, ... ,q, and that when m = n, NkO is a matrix norm on Mn for k = 1, ... ,n. The function Nk(A) is called Ky Fan norm. 11.4.4 Let Al and A2 E Mn symmetric with Al - A2 being nonnegative definite. Show that Ai(Ad ~ Ai(A 2), i = 1, ... ,n. 11.5. Matrix Approximations In this section, we will look at some problems of approximating a given matrix by a matrix with specific structural properties. Let A E Mn be a given matrix. First we consider the problem of finding a symmetric matrix B E M n , which is closest to A with respect to a given norm. In the following we provide an explicit solution to this problem. P 11.5.1 Let A E Mn and B = (A+A')/2. Then B is a symmetric matrix closest to A with respect to any orthogonally invariant norm II·lI. PROOF.

Let C be any symmetric matrix in Mn. Note that

IIA - BII = II(A - A')/211 = (1/2)II(A - C) - (A' - C) II = (1/2)II(A - C) - (A - C),II

+ II (A - c)'lIl C) II + II(A - C) III =

~ (1/2)[11 (A - C) II

= (1/2)[II(A -

II(A - C)II.

A matrix B of order n X n with real entries is said to be skewsymmetric if B = - B'. We can find explicitly a skew-symmetric matrix closest to a given matrix.


389

P 11.5.2 Let A E Mn and B = (A - A')/2. Then B is a skewsymmetric matrix closest to A with respect to any orthogonally invariant norm 11·11 . These results can be formulated for matrices with complex entries. If A is a given matrix of order n X n with complex entries, the objective is to find a hermitian matrix B closest to A. Likewise, one can find an anti-hermitian (A matrix B is anti-hermitian if B = -B*.) matrix closest to A. Our next objective is to find an orthogonal matrix closest to A. In Complements 11.5.1 and 11.5.3, we address this problem. In the following, we prove very general results useful in solving statistical problems. P 11.5.3 Let A and B E Mm,n with singular value decompositions A = P 6 1 Q' and B = R 62 S', respectively. Then

where Ok is the class of all k X k orthogonal matrices , U* = RP' and T* = SQ', and II . IIF is the Frobenius norm. PROOF. Consider

IIU A - BTI\} = tr[(U A - BT)'(U A - BT)] = tr(A' A) + tr(B' B) - 2tr(A'U' BT). We have to find the minimum of the last term in the above expression, when U and T roam over orthogonal matrices. Using von Neumann's result stated in the Complement 11.4.1, we obtain the desired result. The result may not hold for all unitarily invariant norms. A number of results follow from P 11.5.3. These consequences are listed as Complements 11.5.1 to 11.5.6. Next in line is the non-negative definiteness property. For a given matrix A of order n X n, we seek a non-negative definite matrix B closest to A under the Frobenius norm. Let A be a matrix of order n x n with real entries. Let Let B = QH be a polar decomposition of B with Q being orthogonal and H non-negative definite. Then C = (B + H)/2 is

P 11.5.4

B

= (A + A')/2.

390


non-negative definite and closest to A under the Frobenius norm Moreover, C is unique.

II . II·

PROOF. First, we show that any non-negative definite matrix closest to A is also closest to B, the symmetric part of A . For this, we note a special property of the Frobenius norm. Let D = (d ij ) and E = (eij) be symmetric and skew-symmetric matrices, i.e., D = D' and E = - E', respectively. Then nn

nn

nn

n

liD + EII~ = LL(dij + eij)2 = LLd~j + Le~j +2 LLdijeij i=l j=l n

=

n

i=l j=l n

i=l

i=l j=l

n

LLd~j + LLe~j +0= IIDII~ + IIEII~ · i=l j=l

i=l j=l

Let X be any non-negative definite matrix of order n x n. Then

IIA - XII~ = II(A + A')/2 - X

+ (A -

A')/211~

= liB - XII~ + II(A - A')/211~, in view of the facts that B - X is symmetric and (A - A')/2 is skewsymmetric. Thus minimizing IIA - XIIF over all non-negative definite matrices X is equivalent to minimizing liB - XIIF over all non-negative definite matrices X. Let us work with the symmetric matrix B. Let B = ZAZ' be the spectral decomposition of B , where Z is an orthogonal matrix, A = diag{Al' A2, .. . ,An}, and AI, A2 , . .. , An are the eigenvalues of B. Let us replace each negative eigenvalue by zero in the spectral decomposition of B. More precisely, let for each 1 ::; i ::; n , if Ai 2:: 0, if Ai < 0, and C = ZDZ' , where D = diag{d l ,d2 , ... ,dn }. It is clear that Cis non-negative definite. Let us see whether C is closest to B . Let X be any non-negative definite matrix. Note that

liB - XII~ = IIZAZ' - ZZ'XZZ'II~ =

IIA -

Z'XZII~ .


391

The last equality follows from the fact that the F'robenius norm is orthogonally invariant. Let Y = Z'XZ = (Yij). Then n

2

liB - XII = LYTj

n

+ L(Ai -

ii:j

Yii)2

~ L(Ai - Yii)2 ~ LAT.

i= 1

i= 1

Ai 0)

The matrix C is the right choice. Two facts emerge from these deliberations. The first one is that the matrix C which minimizes IIA - XII 2 over all non-negative definite matrices X is unique. The second one is that this minimum can be explicitly spelled out, i.e., min IIA

- XII}

=

L AT + II(A - A')/211

2

.

Ai k) dimensional space. They are called canonical coordinates and used in graphical representation of multivariate data as discussed in Rao (1948).

CHAPTER 12 OPTIMIZATION PROBLEMS IN STATISTICS AND ECONOMETRICS

12.1. Linear Models In this chapter, we consider some general optimization problems which are useful in the statistical analysis of linear models. A linear model is of the form y=X

nxl

nXm

/3+E

mxl

nxl

(12.1.1)

where Y is an n-vector random variable, X is an n x m matrix of rank p{X) = r ::; m, /3 is an m-vector of unknown parameters and E is an n-vector of error variables with E{ E) = 0 and covariance matrix D{E) = 0-2V an n x n non-negative definite matrix. The problems of interest are the estimation of the fixed parameters /3 and 0- 2 and the estimation or the prediction of the random component E. A mixed linear model is of the form (12.1.2) where Y,X,{3 and E are as in (12.1.1), Ui is an n x Pi matrix, ~i is a Pi-vector random variable such that E{~i) = 0 and

COV{~i) = 0-7Ipi' COV{~i' ~j) = 0, i

=1=

j, COV{~i' E) = 0

(12.1.3)

o-r,

for i = 1, ... ,k. The problems of interest are the estimation of /3, 0- 2 , ... ,o-~ and the prediction of the random components 6, ... '~k and

E.

12.2. Some Useful Lemmas In this section, we consider some general results used in optimization problems in statistics. The following notation and assumptions are used throughout this chapter. 403

404


(1) Sp(A) and p(A) stand for the space generated by the columns of the matrix A and the rank of A, respectively. Al. is a matrix of maximum rank such that A' A l. = o. (2) V is n x n non-negative definite matrix, X is n x m matrix and W = (VIX). Note that p(W) = p(V + XX') and Sp(VIX) = Sp(V +XX').

P 12.2.1 The linear equations in matrices L of order n x k and A of order m x k, VL+XA= G1 X'L

(12.2.1)

admit solutions for any G 1 of order n x k and G2 of order m x k such that Sp(G1 ) C Sp(W) = Sp(VIX) and Sp(G2 ) C Sp(X'), respectively. PROOF. We only have to show that

Le. , if there exists a vector (a' : b') such that (12.2.2) then

[a'WI

[g~] = o.

(12.2.3)

Equation (12.2.2) is equivalent to

+ b'X' = 0, a'X = 0 => a'Va = 0, Va = 0, Xb = 0, X' a = 0 => a'W = 0, b' X' = o. a'V

=>

Then a'G' = 0 since Sp(Gt} C Sp(W) and b'G2 Sp(X'), which proves Lemma 12.2.l.

=0

since Sp(G2 ) C

The next lemma due to Rao (1989) is concerned with the minimization of a matrix function in the sense of LOwner.

Optimization Problems in Statistics and Econometrics

405

P 12.2.2 Let Lo and Ao be a solution of the matrix equations in L of order n X k and A of order m X k given by

VL+XA= F, X'L =P, where Sp(F)

c

(12.2.4)

Sp(V\X) and Sp(P) C Sp(X'). Then

L'V L - L' F - F'L '?:.L -F'Lo - p' Ao = -L~F - A~P

(12.2.5)

for all L such that X'L = P. PROOF. A general solution of X'L = P is L = Lo + X 1. A, where A is an arbitrary matrix. Substituting in the expression on the left hand side of (12.2.5), we find (using the equation V Lo = F - X Ao)

L'V L - L' F - F'L = L~ V Lo - L~F - F'Lo

+ (Xl. A)'V(Xl. A)

?:.L L~ V Lo - L~F - F'Lo = - F' Lo - p' Ao since (X 1. A)'V(X 1. A) is non-negative definite.

Complements 12.2.1 (A special case of Lemma 12.2.1) Consider the nonhomogeneous quadratic form Q(f) = f'Vf - 2f'w where f is an n-vector and w is an n-vector. Let p be an m-vector such that p E R(X'). Then min Q(f) =

X'f.=p

-00 {

-fow - p' >'0

if w rJ. R(V\X) if wE R(V\X)

where fo, >'0 is a solution of Vf

12.2.2

+ X>. = w,

X' f

= p.

Consider the matrix A and a g-inverse A-of A partitioned as

A-- [VX'

1 A- = [ClC

X 0'

3


406

where V E Mn and nnd; and X E Mm ,n' Show that: 1a. lb. 2a. 2b. 2c. 2d. 2e. 3a.

peA) ~ m + n; peA) = m + n if p(V) = nand p(X) = m. peA) = p(VIX) + p(X). XC~X = X, XC3 X = X. XC4 X' = XC~X' = VC~X' = XC3 V = VC2X' = XC~V. X'CIX, X'CI V, VCIX are null matrices. VCI VCI V = VCI V = VC~VCI V = VC~V. TrVCI = p(VIX) - p(X) = TrVC~. If Sp(X) c Sp(V), then one choice of C ll C 2, C 3 , C 4 is

3b.

In general

where T = (X'(V + X'V X)- X)-, w = (V + X'V X)- and V is any matrix such that Sp(X) C Sp(V + XV X') and Sp(V) C Sp(V + XV X').

12.3. Estimation in a Linear Model We apply the result of P 12.2.1 to estimate X (3, E and characterizing the linear model

(72,

the triplet

(12.3.1) where we use the symbol D to denote the dispersion (variance covariance) matrix. The reason for choosing X (3 as a more natural parameter for estimation rather than (3 will be made clear once we define estimability of a linear parametric function. DEFINITION 12.3.1. A set of

k parametric functions P'(3 where Pis

an m x k matrix, is said to be unbiasedly estimable by linear functions of Y if there exists a k x n matrix L such that E( L'Y) = P' (3, a sufficient condition for which is Sp(P) C Sp(X'). Note that the entire m-vector parameter (3 is estimable only if Sp(I) = Sp(X'), i.e., when p(X') = m. The function X (3 is always estimable.


407

LoY is said to. be the minimum dispersion unbiased estimator (MOUE) of an estimable function P' (3 if LoX = P' and DEFINITION 12.3.2. A linear function

D(L~Y)

= (j2 L~V Lo ~L (j2 L'V L = D(L'Y)

for all L such that L' X = P'. The following theorem provides the main result for computing the MOUE of estimable parametric functions of (3.

P 12.3.1

Let Lo and Ao be a solution of the equations

VL+XA=O X'L

=X'.

(12.3.2)

Then LoY is the MOUE of X{3 with the dispersion matrix, D(LoY) = -(j2XA o. PROOF. The covariance matrix of L'Y is (j2 L'V L. Using the result of P 12.2.1, the minimum of L'V L with L subject to X'L = X' is attained as stated in the theorem. The following results are consequences of P 12.3.1. (1) If ClI C 2, C3, -C4 are the partitions of any g-inverse of the matrix of the equations in (12.3.2), then Lo = C2X' and Ao = -C4X' and the estimate of X (3 is

and

D(X{3) A

xj3 = L~Y = XC2Y,

(12.3.3)

= (j 2 (-XAo) = (j 2 XC4 X' .

(12.3.4)

(2) Let P be an m x k matrix such that Sp(P) C Sp(X') in which case the function P' {3 is estimable and the MOUE of P' {3 is P' j3 where j3 = C 2Y with the covariance matrix (j2 P'C4 PTo establish this result, we note the following. (1) There exists an n x k matrix A such that P = X' A. (2) If T is MOUE of a vector parameter (), then A'T is MOUE of A'(} (Why?).

408


Now we choose T = x/3 = XC2Y (Le., defining /3 = C 2Y) which is the MOUE of X {3. Then using (2), the MOUE of A' X (3( = P'{3} is A'X/3 = A'XC2Y = P'C2Y = P'/3. Further

D(P'/3) = D(A'X/3)

= A'D(X/3}A = a 2 A' XC4 X' A

using (12.3.3)

= a 2 P'C4 P.

(12.3.5)

The above results show that we may consider /3 as formally an estimate of {3 and a 2 C 4 as the dispersion of /3, in the sense that the MOUE of an estimable function P' (3 is P' /3 and D(P' /3) = a 2 P'Cov(/3)P = a 2 P'C4 P, although {3 as such may not be estimable. As we have estimated X {3 by X /3, it would appear natural to estimate 10, the error vector, by the residual Y - x/3. Is there a direct way of predicting 10 by minimizing a suitable criterion function? We attempt to do this by considering linear estimators of the form L'Y, where L is an n x n matrix.

P 12.3.2 The linear predictor L'Y of 10 has a bounded mean dispersion error (MOE) for all {3 only if L is such that X'L = O. Let (Le, Ve) be any solution of the equation

VL+XA= V, X'L

=0.

(12.3.6)

Then L~Y is the unbiased predictor of 10 with the minimum mean dispersion error of a 2 X Ae. PROOF.

The MOE of predicting 10 by L'Y is

E[(L'Y - E}(L'Y - 10)'] = E[(L' X{3 + (L' - I)E)(L' X{3 + (L' - I)E)'] = L' X{3{3' X' L+a 2(L-I)'V(L-I). (12.3.7) For any given L, the elements of (12.3.7) tend to infinity as the components of {3 tend to infinity. For boundedness, it is therefore necessary that X'L = O. Then the MOE is

(12(L - I)'V(L - I) = (12(L'V L - L'V - V L'

+ V)

~L E[(L~Y - E}(L~Y - E)']


409

using the result of P 12.2.2, establishing that MMDE of E is L~Y with the dispersion matrix (apart from the multiplier 0"2) (L~VLe - L~V - VLe

+ V) =

(L~(V - XAe) - L~V - V

+ XAe + V)

= (I - Le)'XA e = 0"2 XA e,

where Le and Ae are any solutions of (12.3.6), and P 12.3.2 is proved. From P 12.3.1 and P 12.3.2, the estimates of Xj3 and E are L~Y and L~ Y, respectively. Do they add up to Y? Let us consider

+ L~Y - Y) 0"2(Lo + Le - I)'V(L o + Le - I) 0"2(Lo + Le - I)'(V Lo + V Le - V) 0"2(Lo + Le - I)' X( -Ae - Ao) 0"2(L~X + L~X - X)( -Ae - Ao) =

D(L~Y

= = = =

(12.3.8) 0

(12.3.9)

using the equations (12.3.2) and (12.3.6) and substituting for V Lo and V Le in (12.3.8) and for X'L o and X'L e in (12.3.9). The result shows

that L~Y + L~Y = Y with probability one. This result can also be established by using the explicit solutions to Lo and Le in terms of the Ci matrices.

12.4. A Trace Minimization Problem Let C be the collection of all matrices C of order m x n with real entries, X a given matrix of order n x r, Vb V2, ... , Vk given matrices each of order m x n, and PI, P2, ... ,Pk given real scalars. Let

C1 = {C E C : CX = 0, Tr(C\ti') = Pi for each i}.

(12.4.1)

The objective of this section is to minimize Tr( CC') over all C E CI . If the collection CI is empty, we have no case to answer. Assume that C1 is non-empty. Let Px be the projection operator from Rn onto the subspace Sp(X), which has an explicit representation Px

= X(X' X)- X',

(12.4.2)

410


where (X' X)- is any g-inverse of X' X. Let Q = I - Px and note that 0 =} C = DQ for some matrix D of order m x n. We are now ready to solve the problem. Consider the following system of linear equations

ex =

k

2)tr(ViQVj)]Ai i=1

= Pj, j = 1,2, ...

,k

(12.4.3)

in unknown A1, A2, ... ,Ak and check on their consistency. Let F be the symmetric matrix of order k x k given by F = (tr(ViQVj)),

(12.4.4)

i.e., the (i,j)-th entry of F is given by tr(ViQVj). The strategy is the following standard one. Let a be any vector of order k x 1 orthogonal to the columns of F. Then show that it is also orthogonal to the vector p, where P' = (P1,P2,'" ,Pk)' This would then imply that P E Sp(F) and hence the system (12.4.3) is solvable. Let a' = (a1' a2, ... ,ak)' If a is orthogonal to the columns of F, then we have k

L aitr(ViQVj) = 0 for each

= 1,2, ...

j

,k.

(12.4.5)

i=1 The operation trace is linear, so that from (12.4.5), we obtain k

=0

tr«L ai Vi)QVj)

for each

j

= 1,2, ... , k.

(12.4.6)

i=1 Multiplying (12.4.6) byaj and summing over j, we obtain k

k

tr«L ai Vi)Q(L ai Vi)') = O.

i=1

Since the matrix involved in the trace operation of (12.4.7) negative definite, we have k

k

«L ai Vi)Q(L ai Vi)') = O.

i=1

(12.4.7)

i=1

i=1

IS

non-


411

Since Q is non-negative definite, k

k

Q(LaNi)' = 0 = LaiQ~'. i=l

We now show that a'p =

o.

k

Let C E C1 • Observe that k

a'p = L aiPi

k

aitr(C~')

=L

i=l

(12.4.8)

i=l

=L

i=l

aitr(DQ~')

i=l

k

= tr(D(L aiQ~')) = 0

(from (12.4.8)).

i=I

Thus the solvability of the system (12.4.3) of equations is assured. We are now ready to state the result which will solve the optimization problem.

P 12.4.1 The minimum of tr(CC') over all C E Cl is attained at any matrix C. given by k

C. = LAiViQ,

(12.4.9)

i=l

where AI, A2, ... ,Ak is a solution to the following system of linear equations: k

L[tr(ViQVj)]Ai

= Pi, j = 1,2, ...

,k.

i=l

PROOF. First, let us check whether C. belongs to Cl. We need to

verify, first, that C*X = O. This follows from the fact that

QX

= (I -Px)X = X

-X(X'X)-X'X

=X

-X

= O.

Second, we need to verify that tr(C* Vj) = Pi for each j. From the stipulation that Ai'S satisfy the linear equations above, it follows that k

tr(C* Vj)

= L[tr(ViQVj)]Ai = Pi· i=l

412


Thus C* E CI . The next objective is to show that C* is optimal. Let C be any matrix in CI . One can always write C = C* + G, for a suitable matrix G . The fact that C E CI implies, at the outset, that 0= CX = (C* +G)X = C*X +GX = GX,

or more relevantly, that (12.4.10)

GX=O. This implies that GPx = O. (Why?) Further,

GQ

= G(I - Px) = G.

(12.4.11)

Additionally,

Pi = tr(CVj) = tr((C*

+ G)Vj) =

tr(C* Vj)

+ tr(GVj) =

Pi

+ tr(GVj)

for each j implies that tr( CVj) = 0

for each j.

(12.4.12)

Crucially, from (12.4.11) and (12.4.12), we observe that k

tr(C*G')

k

= tr((LAiViQ)C') = tr(LAiViQG') i=I

i=I

k

k

= tr(L Ai ViC') = L i=I

Aitr(ViG') = O.

(12.4.13)

i=I

Finally, we observe that

+ G)(C* + G)'] tr(C*C~) + tr(GG') + tr(C*G') + tr(GC~) tr(C*C~) + tr(GC') ~ tr(C*C~).

tr(CC') = tr[(C* = =

This completes the proof. We now focus on another problem of the same type as the one proposed at the beginning of this section. We assume that m = n. Let VI, V2 , ••• , Vk be a finite collection of known symmetric matrices of order m x m. Let X be a given matrix of order m x T. Let PI, P2, ... , Pk be real scalars. Let C2 = {C : C = C' of order m x m, CX = 0, tr (CVj) = Pi, 'lij}.

The objective is to rrunimize tr(C2 ) over all C E C2. The following result provides an explicit solution to the problem.


P 12.4.2 given by

413

The minimum of tr( C2) is attained at any matrix C* k

C*

=L

(12.4.14)

AiQViQ,

i=l

where AI, A2, ... ,Ak is a solution to the system of linear equations: k

L[tr(QViQ\tj)]Ai

= Pj, j = 1,2, ...

,k.

(12.4.15)

i=l

We will not give a proof of this result. One can imitate the proof of P 12.4.1 to establish this result. One can notice right away that C* is symmetric and that C* belongs to C2. The consistency of the system of equations (12.4.15) follows in a similar vein. (Try it.) The solution to the optimization problems posed above may not be unique. Nonuniqueness arises if the matrices involved in equations (12.4.3) and (12.4.15) are nonsingular. In practice, we are not unduly concerned with the uniqueness problem.

12.5. Estimation of Variance Let us get back to the basic model: Y = X{3 + € with E€ = 0 and O(€) = (72V where (72 > 0 is unknown. In this section, we assume that V = I. The main objective of this section is the estimation of (72. Let Y'CY be a quadratic form in the data Y, for some symmetric matrix C, which we would like to propose as an estimator of (72. The basic requirement is that it be unbiased. Let us record a very general result on the expected value of a quadratic form.

P 12.5.1

Let Y be a random vector of order n x 1 with mean vector

IL and dispersion matrix (72V. Let C be any symmetric matrix of order n x n. Then (12.5.1) E(Y' CY) = (72 tr(CV) + IL' C IL. PROOF.

Observe that V

= E(Y -

IL)(Y - IL)'

= EYY' -

ILIt'. Also,

E(Y'CY) = Etr(Y'CY) = Etr(CYY') = tr(E(CYY'))

= tr(CE(YY'))

= tr(C«(72V + ILIL'}} = (72 tr(CV) + IL'CIL·

414


In our model, we identify that V = I and J.L = X (3. Consequently,

E(Y'CY)

= 0'2tr(C) + {3X'CX{3.

Since we would like to have the quadratic form to be unbiased, we need to nullify the nuisance element present in its expected value, namely (3X'CX{3. We impose the following conditions on the matrix C: trC

=1

and

CX

= o.

(12.5.2)

Next, we would like the unbiased estimator Y'CY to have minimum variance. But we have to venture into the realm of fourth order moments of the random variables Yi's. The variance of Y'CY can be computed but it is complicated. One way to get out of this rigmarole is to assume that the random variables involved in the linear model have a multivariate normal distribution. Under this assumption Var(Y'CY) = 20'2tr(C2). In order to find a minimum variance estimator Y'CY of 0'2, we need to minimize trC2 subject to the conditions: trC = 1 and CX = O. This problem falls into the orbit of P 12.4.2. We identify that k = 1; Vl = I; and Pl = 1. The optimal matrix C .. is given by C .. = AlQVlQ = AlQ2 = AlQ, where Al satisfies the equation tr(QVlQVt}Al = Pl = 1, from which we obtain, Al = l/tr(Q). Consequently, the minimum variance quadratic unbiased estimator of 0'2 is given by

(trQ)-lY'QY = (trQ)-lY'(I - X(X'X)- X')Y.

(12.5.3)

Let us make a few comments on the above derivation. There is a standard approach available to solve this problem. One uses the method of least squares to estimate {3. As a matter of fact, a solution to the resultant normal equations in fj is given by fj = (X'X)- X'Y. Then one shows that the residual sum of squares

(Y - xfj)'(Y - xfj) = Y'(I - X(X'X)- X')Y has expectation (n - p(X))0'2, from which one obtains an unbiased estimator of 0'2. This estimator is precisely what we have obtained in (12.5.3). Our approach has given us something more. We showed that the estimator (12.5.3) has minimum variance under the normality assumption. The next goal is to dispense away with the normality assumption. This is the objective in the next section.


415

12.6. The Method of MINQUE: A Prologue The acronym in the title of this section stands for minimum norm quadratic unbiased estimation. Let us work in the realm of the linear model: Y = X f3 + c with Ec = 0 and D(c) = (J2 I. In the last section, under the pretext of multivariate normality, we obtained the minimum variance quadratic unbiased estimator of (J2. In this section, we will make an attempt to obtain an optimal quadratic estimator of (J2 without invoking the normality assumption. If the error vector c' = (cl' C2, ... ,cn) is observable, then a natural estimator of (J2 is n

I

(lin) LC; = (l/n)c'c = c'(-I)c. i=l

n

But only the data vector Y is observable. We need to build an estimator of (J2 based on Y. As professed earlier, we will entertain only quadratic estimators of (J2. Let C be a symmetric matrix of order n x n. We have seen, in the last section, that Y'CY is an unbiased estimator of (J2 if CX = 0 and tr(C) = 1. Under these restrictions, the quadratic form Y'CY simplifies to

+ c)'C(Xf3 + c) f3'X'CXf3 + c'Cc + f3'X'Cc + c'CXf3 =

Y'CY = (Xf3 =

c'Cc.

Consequently, the difference between what is contrived and what is ideal works out to be: Y'CY - c'(n- 1 I)c = c'(C - n- 1 I)c.

The critical idea behind the method of MINQUE is to choose C such that it is close to (1/n)I. Choosing the Frobenius norm (IIBIIF = tr(BB')), the objective is to minimize IIC - n- 1 IIiF subject to unbiasedness conditions that CX = 0 and tr (C) = 1. Observe that

Consequently, the above problem reduces to minimizing tr(C2) subject to the conditions that CX = 0 and tr(C) = 1. This is precisely what we

416


have done in Section 12.5. The solution was identified and we obtained the estimator of (12 the usual one based on the residuals. There are two aspects of this method worth recording. One is that the usual estimator of (12 is optimal based on the underlying criterion of the method of MINQUE. The other aspect is that the method is amenable to generalization. This is what we are proposing to carry out in subsequent sections.

12.7. Variance Components Models and Unbiased Estimation Variance components models (random and mixed effects models) are routinely postulated for a variety of datasets. They also play a useful role in analyzing repeated measurements data. Any such model can be described by a linear model, Y = X f3 + c, with Ec = 0 and (12.7.1) where VI, V2 , •• • ,Vk are known non-negative definite matrices, O}, O2 , ... ,Ok are unknown real numbers and 0' = (O}, O2 , . .. ,Ok)' There may be additional conditions imposed on Oi'S. For example, Oi'S are bound by the restriction that V(O) be non-negative definite. For the time being, let us not put too much emphasis on the range of values for the vector 0, except for the minor technical condition that the range of values for the vector 0 has a non-empty interior, which is a topological condition. The quantities 0 1 , O2 , .. • ,Ok are called the variance components of the model. There may be a certain element of ambiguity in the specification of the matrices V}, V2 , .• • ,Vk . We can rewrite, for example,

The matrices 2- 1 VI, 2- 1 V2 , ••• ,2- 1 Vk are known non-negative definite matrices. Shall we call 201,202, ... ,20k as the variance components of the model? If we fix the matrices V}, V2 , ••• , Vk , variance components are probably uniquely determined. Another complication could arise. It may be possible that there are two distinct vectors (0 1 ,02 , ... ,Ok) and (Oi, O2, ... ,O'iJ such that 0 1VI + ... + Ok Vk = 0iVl + ... + OZVk. In such a contingency, estimation of all the variance components is not possible. We now assume that the matrices VI, V2 , ••. , Vk are fixed and the variance components are unambiguously defined. The main objective of

Optimization Problems in Statistic8 and Econometrics

417

this section is to find some good estimators of the variance components based on the data Y. More generally, let (12.7.2) be a linear combination of the variance components, where PI, P2, ... ,Pk are known. As usual, we will entertain only quadratic estimators Y'CY for the linear combination of the variance components. We need to impose conditions on the matrix of the quadratic form in order that it be unbiased. Note that k

PIOI + P2 02 + ... + PkOk = E(Y'CY) =

L Oitr(CV;) + f3' X'CXf3. i=1

A set of necessary and sufficient conditions for Y'CY to be an unbiased estimator of the given linear combination of the variance components are: X'CX = 0 and tr(CV;) = Pi, i = 1,2, ... ,k. (12.7.3) Will there be a symmetric matrix C at all satisfying (12.7.3)? We need a criterion for the existence of a solution to (12.7.3) in terms of the matrices VI, V2, ... , Vk and X. Construct the following symmetric matrix of order k x k, H = (tr(V;Vj - PV;PVj)),

(12.7.4)

where P denotes, as usual, the projection operator X(X'X)- X'. The matrix H is computable once we know V;'s and X. We will show that there is a matrix C satisfying (12.7.3) if and only if the coefficient vector P consisting of the components PI, P2, ... , Pk belongs to Sp( H}. We will now prove this assertion. A general solution to the equation X'CX = 0 in unknown C is given by B - P B P for some symmetric matrix B. (How?) Suppose there exists a matrix C satisfying (12.7.:3). Then C = B - PBP for some symmetric matrix B. Note that for each i,

Pi = tr(CV;) = tr(BV; - PBPV;) = tr(BV; - BPV;P) = tr(B(V; - PV;P)) = (vec(Vi - PV;P))'(vec(B)). Let us take a critical look at the last step. We use the vec operation to express the trace of a product of two symmetric matrices. Recall

418


the vec operation from Section 6.2. The vec operation performed on a matrix stacks all the entries of the matrix column by column into a single column vector. IT E = (eij) and F = (Iij) are two symmetric matrices of the same order, then tr(EF} = Li Lj eij!ij = (vec(E))'(vec(F)). (Try it.) Note that

[VeC(Vl - PVIP)]']

[vec(V2 - ~~2P)]'

=

[

[vec(B)] .

(12.7.5)

[vec(Vk - PVkP)]'

Let T be the matrix of order k 2 x k given by

Then (12.7.5) can be rewritten as

p = T'vec(B). What this means is that the vector p is a linear combination of the columns ofT'. Consequently, pE Sp(T'), the range space of the matrix T'. But Sp(T') = Sp(T'T). Let us compute T'T. The (i,j)-th entry of T'T is given by, in view of what we have observed about the relationship between the vec operation and trace, [vec(Vi - PViP)]'[vec(Vj - PVjP)] = tr«Vi - PViP)(Vj - PVjP))

= tr(ViVj - ViPVjP - PViPVj + PViPPVjP) = tr(ViVj - PViPVj), since the projection operator is idempotent. This entry is precisely the (i,j}-th entry of H. Consequently, T'T = H, and hence p E Sp(H). Conversely, suppose p E Sp(H} = Sp(T'}. Then there exists a column vector l of order k 2 x 1 such that p = T'l. Now undo the vec operation. Let B be the matrix of order k x k such that vec(B} = l. (How?) The matrix B may not be symmetric. Let

c=

(1/2}[B

+ B' -

P(B + B'}P].

(12.7.6)


419

This matrix G satisfies all the constraints of (12.7.3) . The symmetric matrix G defined in (12.7.6) satisfies X'GX = O. In order to establish this, one merely uses the fact that PX = X and X'P = X'. Using (12.7.6) and after some simplification, we get tr(GVi) = (1/2)tr[(B

+ B')Vi -

P(B

+ B')PVi]

= [vec(Vi - PViP)]'[vec(B)] .

(l2.7.7)

A critical look at (12.7.7) yields that tr(GVi) = Pi for every i. The above presentation clearly spells out when one can estimate a given linear combination of the variance components unbiasedly using quadratic estimators. Let us enshrine this in the following proposition. P 12.7.1 Let PI(h + ... + PkOk be a linear combination of the variance components. Then there exists a quadratic unbiased estimator of the combination if and only if the vector of coefficients P E Sp(H). 12.8. Normality Assumption and Invariant Estimators Let us get back to the main track. In what follows, we assume that there are quadratic unbiased estimators available to estimate a given linear combination of the variance components. We need to venture beyond unbiasedness. One approach would be to seek a symmetric matrix G such that Y'GY is unbiased and has minimum variance in the class of all unbiased quadratic estimators of the given linear combination of the variance components. But the computation of the variance of the quadratic form involves third and fourth order moments of the error random vector and it is complicated. However, the variance of an unbiased quadratic estimator has a simple form of expression provided we assume normality for the distribution of the error random vector. Under normality, var(Y'GY) = 2tr((GV)2) + 4{3' X'GVGX{3. But the minimization of var(Y' GY) over all quadratic unbiased estimators is fraught with difficulties. The variance involves not only the dispersion matrix V but also the parameter vector {3. The supposition that Y'GY is an unbiased estimator of p'O imposes the condition that X'GX = O. This will not make the term involving {3 to go away. One way to get rid of this term is to impose the stronger condition that GX = O. We not only want our estimators to be unbiased but have a little extra to guarantee that G X = O. Is there a statistical or mathematical interpretation of

420


the condition CX = o? We will provide one interpretation of the condition. Let /30 be an arbitrary but fixed vector of order m x 1. Consider the new data vector Y* = Y + X /30, which is observable as soon as Y is observed. Let /3* = /3 + /30. Since /3 is an unknown parameter vector in our original linear model, /3* is also unknown. Consider the linear model: Y* = X /3* + c. Structurally, this model is no different from our original model. Either of them could be used to estimate the parameter vector /3 and the variance components. Any estimator one provides to estimate the variance components whether one uses the data Y or Y* should lead to the same value. This is the idea behind in the following definition. DEFINITION 12.8.1. A quadratic form Y'CY in the data is said to be an invariant unbiased estimator of a linear combination p'e of variance components if it is unbiased and Y'CY = (Y + X/3o)'C(Y + /30) for all /30 E Rffl . There is nothing new in the idea of invariance. In a substantial number of decision theoretic procedures, invariance property plays a crucial role in deriving some optimal procedures. The invariance property introduced in Definition 12.8.1 is akin to shift invariance. The collection of shift transformations, Y ---. Y + X /30, is indexed by /30, where /30 roams freely over the Euclidean space R ffl • The condition of unbiasedness imposes restrictions on the matrix C. We have already seen that we must have that X'CX = 0 and tr(C\Ii} = Pi for all i. The additional property of invariance imposes also a restriction on C. The following,

+ X/3o}'C(Y + X/3o) = Y'CY + 2Y'CX/30 + /3hX'CX/3o ,

Y'CY = (Y

(12.8.1)

is valid for all /30 if and only if CX = O. Let us paraphrase the above in the following proposition. P 12.8.2. A quadratic form Y'CY in the data is an invariant unbiased estimator of a linear combination p'e of the variance components if and only if

CX = 0

and tr(C\Ii} = Pi, i = 1,2, . . . ,k.

(12.8.2)


421

Understandably, the conditions (12.8.2) are stronger than the conditions (12.7.3). The next task is to mount an inquiry under what conditions there exists a symmetric matrix C meeting the requirements of (12.8.2)? Let us provide an outline how this can be done. The equations tr(CVi) = Pi, i = 1,2, ... ,k, can be viewed as a bunch of linear equations in the entries of C using the vec operation. But C needs to satisfy an additional restriction that CX = o. In order to accommodate this restriction, determine the general form of C satisfying CX = 0, and accommodate this in the other set of constraints which can be rewritten as tr(CVi) = [vec(Vi)]'[vec(C)] = Pi, i = 1,2, .. . , k. This is the technique we followed in Section 12.7 towards the buildup of P 12.7.1. We will not pursue this natural approach. We will try to exploit P 12.7.1 to determine precise conditions under which we can find an invariant unbiased quadratic estimator of p'(}. The key idea is to build another linear model in the framework of P 12.7.1. Any symmetric matrix C satisfying C X = 0 must be of the form C = BAB' for some symmetric matrix A and for some matrix B of maximal rank satisfying B' B = I, B' X = 0 and BB' = Q = I - P, where P is the projection operator X(X'X)- X'. (Show this.) Introduce a transformation Y. = B'Y. Observe that EY. = B' X f3 = 0 and D(Y.) = (}l VI- + ... + (}k Vk ., where Vi* = B'ViB. Thus we have a linear model: Y. = c. with the design matrix X. = 0, Ec. = 0 and D(c*) = D(Y*). Further, the new on the vector space A. For any two matrices A = (aij) and B = (b ij ) in A, let n

< A, B > =

n

L L aijbij = tr(BA). i=l j=l

The following properties of the inner product are transparent. n

n

L L

a;j ~ o. i=l j=l (2) For any A in A, < A, A > = 0 if and only if A = O. (3) For any A and B in A, < A, B > = < B, A >. (4) The inner product < -, . > on the product space A x A is bilinear, i.e., for any A, Band C in A and 0: and {3 real numbers,

(1) For any A in A, < A, A > =

< o:A +{3B,C > < A,o:B + {3C > -

0: 0:

< A,C > + {3 < B,C >, < A, B > + {3 < A, C > . 433

434


In view of the properties enunciated above, the phrase "inner product" is apt for the entity < .,. >. It is time to introduce a special subset of the vector space A . DEFINITION 13.1.1. A subset B of A is called a quadratic subspace of A if the following hold.

(1) B is a subspace of A. (2) If B E B, then B2 E B. Some examples are in order. EXAMPLE

13.1.2 .

(1) At one end of the spectrum, the whole collection A is a quadratic subspace of A. At the other end, B = {O}, the set consisting of only the zero matrix, is a quadratic subspace of A. (2) Let A be a fixed symmetric idempotent matrix of order n x n. Let B = {aA : a real} . Then B is a quadratic subspace of A. (3) The idea behind Example (2) can be extended. Let A and B be two symmetric idempotent matrices of order n x n satisfying AB = o. (The condition AB = 0 implies that B A = o. Why?) Let B = {aA + {3B: a and {3 real}. Then B is a quadratic subspace of A. It is clear that B is a subspace of A. Observe that

(4) Example (3) can be further generalized to handle more than two idempotent matrices. This is one way of providing many examples of quadratic subspaces. These quadratic subspaces have a certain additional property which make them stand out as a special breed. This will be apparent when the notion of commutative quadratic subspaces is introduced. (5) Let us look at the case n = 2. The vector space A is threedimensional. Let A be any matrix in A. By the spectral decomposition theorem, one can write A = CDC' for some orthogonal

Quadratic Sub spaces

435

matrix C of order 2 x 2 and a diagonal matrix D Every orthogonal matrix C is of the form

C = [cos('t. >'2, ... ,>'m) be a given vector. Let P = X(X' X)-l X', and S = tr(~(In - P)Vj). The matrix: S is of order m x m with (i,j)-th element given by tr(~(In - P)Yj). We have seen that there exists an lll1biased quadratic estimator of >.'B if >. E Sp(S), i.e., there exists a vector a such that>. = Sa. (See Sections 12.7 - 12.10.) P 13.4.2 Let Y = X{3 + € be a variance components linear model satisfying all the conditions stipulated in P 13.4.1. Let >.'B be an estimable linear function of the variance components, i.e., >. = Sa for some vector a' = (a1' a2, . .. ,am)' Then: (1) The UMVUE of X{3 is X(X'X)-1 X'Y. m

(2) The UMVUE of >.'B is

2: aiY'(~ -

PViP)Y.

i=1

(1) It is enough to show that X(X'X)-1 X'Y is an lll1biased estimator of X {3. (This is one of the benefits of complete sufficiency of the vector statistic of P 13.4.1.) Note that E(X(X' X)-1 X'Y) = X{3. (2) In a similar vein, it suffices to show that the estimator proposed in (2) is an lll1biased estimator of >.'B. Before proceeding with the task, we do a simple computation to show that PVjP = YjP for each j. Since Sp(VjX) c Sp(X), there exists a matrix C such that YjX = XC. Hence PROOF.

PYjP = X(X' X)-1 X'YjX(X' X)-1 X' = X(X'X)-lC'X'

= X(X' X)-1C' X' X(X' X)-l X'

= X(X'X)-lX'Yj = PYj.

For the evaluation of the desired expectation, we need to recall the formula for the expectation of quadratic form of a random vector. Note that m

ELaiY'(Vi - PViP)Y i=1 m

m

= Laitr((Vi - PViP)Disp(Y))

+L

i=1 m

m

= Laitr((Vi - ViP) LBjYj) i=l

a i{3'X'(Vi - PViP)X{3

i=l

j=1

m

+L i=l

a i{3'(X'ViX - X'PViPX){3

Qu.adratic Su.bspaces

=

m

m

j=1

i=1

m

m

L OJ L

= LOj j=1

447

m

°i

LOi

tr((\Ii - \liP)Vj)

+L

0i

,B'(X'ViX - X'\liPX),B

i=1 m

tr(\Ii(In - P)Vj)

i=1

+0 =

LOjAj, j=1

in view of the fact that A = So. This completes the proof. We would like to indicate the use of P 13.4.2 in the context of design of experiments. We will focus specifically on Balanced Incomplete Block Designs (BIBD). Suppose we wish to compare the performance of some 1/ treatments, T 1 , T2 , ••• ,Til. We select b blocks each containing k experimental units. The treatments are assigned to the experimental units subject to the following conditions:

(1) All treatments are replicated the same number of times

T.

(2) Every pair of treatments occur the same number of times A together in the blocks. Such a design is called a BIBD. The word "incomplete" in BIBD stems from the fact that k < 1/. Consider one particular BIBD adopted for the purported problem of comparing the performance of the treatments. Let Yij be the response from the j-th unit of the i-th block and i(j) the subscript of the treatment that has been assigned to the j-th unit of the i-th block, j = 1,2, ... ,k and i = 1,2, ... ,b. The following is a model with the parameters defined in a natural way that would explain the variation one observes in the responses:

Yij = JL + 0i

+ ,Bi(j) + €ij,

i = 1,2, ... ,b; j = 1,2, ... ,k,

(1) JL is the general effect, (2)

(3) (4)

(5) (6)

(7) (8)

is the effect of the i-th block, ,Bj is the effect of the j-th treatment, €ij is the random error, It and ,Bj's are deterministic unknown parameters, Oi'S are independent identically distributed normal random variables with common mean zero and variance €ij 's are independent identically distributed normal random variables with common mean zero and variance a 2 , Oi'S and €i;'S are mutually independent. 0i

a;,

448


The model is an example of a mixed linear model. The goal now is to put the mixed linear model in the format of Y = Xf3 + c. Let Y be the column vector of all the responses stacked block by block. It is of order n x 1, where n = bk. Let f3' = (iL, f3b f32, .. · ,f3v), a' = (0.1,0.2, ... ,ab),

where lk is a column vector of order k x 1 in which each entry is unity. The matrix X is a block matrix of order n X (1/ + I) and each Xi is of order k x 1/. The first row of Xi has 1 in the i(I}-th column and zeros elsewhere. Remember that the first unit of the i-th block received the treatment Ti(l) and Yil = iL + ai + f3i(l) + Cil. The secon 0 for every non-zero x E cn. The following propositions characterize nnd and pd matrices.

P 14.1.1 A is nnd if and only if there exists a matrix U such that A = U*U, where U is such that Rank(A) = Rank(U). In fact, we can choose U such that U = U* and write A = U 2 = UU . A is pd if Rank(U) = n. The proposition follows easily from the spectral decomposition of A. In fact U can be chosen to be nnd. In that event we say that U is a square root of A. An nnd matrix is also called Gramian. A square root of A is computed as follows. Write the spectral decomposition of A in the style A = .x~ PiP; + .x~P2P2 + ... + .x~PkPk, where k = Rank(A), .xi, .x~, ... ,.x% are positive eigenvalues of A and Pi, P2, ... , Pk the corresponding orthonormal eigenvectors of A. An nnd (also called a Gramian) square root U of A can be taken as (14.1.1) 449

450


P 14.1.2 Let Y be n x r matrix and denote by y.L, the matrix whose columns are orthogonal to those of Y and such that Rank(Y) + Rank(Y.L) = n. Then Py

= y(y*y)-y*

and 1- Py ,

(14.1.2)

are nnd, where (y*y)- is any g-inverse. PROOF. Let a E en be arbitrary. Write a = Ya1 vectors a1 and a2. (How?) Note that

+ y.La2

for some

a* Pya = a*Y(Y*Y)-Y*a = a~Y*Y(Y*Y)-Y*Yal = a~(Y*Y)al = (a~Y*)(Yad ~

o.

Consequently, Py is nnd. In a similar manner,

from which we conclude that 1- Py is nnd. We have seen the matrix Py before. It is the orthogonal projector from en onto Sp(Y). See Chapter 7. P 14.1.3

Let A be nnd and consider a partition

(14.1.3) where Au and A22 are square matrices. Then Au - A12A22A21 is nnd and unique for any g-inverse A22 of A 22 . PROOF. By P 14.1.1, one can write A = U*U for some matrix U. Partition U = (XIY) , where the number of columns of X is the same as the number of columns of Au . Note that A=U*U= [X*] [XIY]= [x*x x*y] = [All A 12 J y* y* X y*y A21 A22 ' Au - A12A22A21 = X* X - X*Y(Y*Y)-Y* X = X*(I - Py)X.

(14.1.4)

Since l - Py is nnd, Au - A12A22A21 is nnd, and unique (Why?).

Inequalities with Applications in Statistics

451

The matrix All - A12A2"2A21 is called. the Schur complement of A with respect to A 22 . P 14.1.4

Let A be pd and partition A-I as

A-I = [All A2l

A12]-1 _ [All A22 A2l

A12] A22

(14.1.5)

where All is a square matrix. Then

(1) (2) PROOF. From

All = (All - A12A2"l A2d-t, All - All is und.

(14.1.6) (14.1.7)

(14.1.5), we have the equations

+ A12A2l = I, All A12 + A12 A22 = o. AllAll

Eliminating A12 we have an equation in All which gives (14.1.6). To prove (14.1.7), we note that All - (All)-l = A12A2"2lA2l' which is und. This shows that All ~ (All)-l which implies (All)-l ::; All. In particular, from (14.1.7), we have that if A = (aij) and A-I = (a ij ), then all ~ l/all. Complements

14.1.1 If A = B+C, B is pd and C is skew symmetric, then IAI ~ IBI. 14.1.2 If A and B are real pd and of order n x n, show that (1) IAA + (1 - A)BI ~ IAIAIBll-A for 0 ::; A ::; 1; (2) IA + Bil/n ~ IAll/n + IBll/n (Minkowski's inequality). 14.1.3 If B is pd and A - B is und, then (1) IA - ABI = 0 has all its roots A ~ 1; (2) IAI ~ IBI; (3) (B-1 - A-I) is nnd; and (4) if Ar is a principal minor of A of order rand Br is the corn~ sponding minor of B, then Ar - Br is und. 14.1.4 Let x be a column vector of order k x 1 and A be a square symmetric matrix of order k x k, with all its elements non-negative. Then for any integer n,

452


with equality if and only if x is an eigenvector of A . This inequality occurs in genetical theory (Mulholland and Smith (1959)). 14.1.5 If II Ax II ::; IIAx + Byll for all x and y, show that A* B = 0, where II . II is the Euclidean norm. 14.1.6 If A is nnd and a i i = 0, show that a i j = 0 for all j. 14.1.7 Let A and B be two pd matrices with eigenvalues contained in the interval [m, MJ, where M ~ m > O. Let 0 ::; A ::; 1. Show that

This is the reverse of the following convex matrix inequality

14.1.8

Let A

= (ai j) be an n

Xn

complex matrix and U

= (Uij ) be an

n x n tulitary matrix. Denote the Hadamard-Schur product of A and U by A . U =

(ai jUij)

and the largest singular value by

(T max'

Show that

14.1.9 For any square complex matrices X and Y of the same size, show that I det{X

14.1.10

A

= TT',

+ Y)1 2

::;

det{I + X X*) det(I + Y*Y).

Show that an nnd matrix A has a unique decomposition, where T is a triangular matrix with diagonal elements non-

negative. 14.1.11 Let H = A + i B be a Hermitian pd, where A and B are real matrices. Show that IAI > IBI (Robertson inequality) and IHI ::; IAI (Taussky inequality). FUrther, if ai's are the eigenvalues of A and fNs of -iB, both in decreasing order, show that ai > f3i. 14.1.12 If A and Bare nnd matrices, show that AB has only nonnegative eigenvalues. 14.1.13 Schur's majorization theorem: If A is a symmetric n x n matrix, show that the eigenvalues of A majorizes its diagonal elements.


14.1.14

453

Let A E Mn and pd be partitioned as

where An and A22 are square matrices. Show that

which is Fischer's inequality. 14.1.15 If A is nnd partitioned as in complement 14.1.14, show that

is nnd. 14.1.16 Let A be n x n positive definite matrix partitioned as in complement 14.1.14. Show that

where A 22 is the partition conformal to A22 in the reciprocal matrix A-I. As a consequence of the above result, show that Al(A · B) ~ Al(AB) when A and Bare n x n nnd matrices, where Al is a generic symbol for the largest eigenvalue. 14.1.17 Let A and B be non-negative definite Hermitian matrices of order n. Show that 1. Amin(A . B) ~ Amin(AB'), 2. Amin(A . B) ~ Amin(AB). 14.1.18 Oppenheim's inequality: Let A and B be n x n nnd matrices. Show that IA . BI ~ IBI an ... ann. 14.1.19 Fiedler's inequality: If A is pd, show that A· A-I - I is nnd. 14.1.20 Let A and B be n x n positive definite matrices. Show that IAIIBI ::; IA· BI ::; (an a22··· ann )(b ll b22 · ·· bnn ). 14.1.21 Let A and B be nxri symmetric matrices and let the elements of B be non-negative. Show that (Al(A · B), .. . , An(A · B))' majorizes

454


(AI (A), ... ,An(A))'. If A also has all non-negative entries, show that IIf=I Ai(A . B) 2: II~l Ai(A). 14.1.22 Consider the projection operator Py = y(y*y)-y* on the space Sp(Y). Let H be a matrix whose columns form an orthonormal basis of Sp(Y). Show that Py = PH = H H*. Get an explicit form of H by using the s.v.d. of Y. 14.1.23 Let B = (BIIB2) be such that B~B2 = O. Show that PB-PB1 is nnd. 14.1.24 (Ostrowski-Taussky Theorem). Let A E M n , and H = 2- 1 (A+ A*) be pd. Show that IH(A)I ::; IAI. 14.2. Cauchy-Schwartz and Related Inequalities Cauchy-Schwartz inequality has numerous applications in statistics. Some well-known applications are in showing that the correlation coefficient lies between -1 and 1 and in deriving a lower bound to the variance of an unbiased estimator such as the Cramer-Rao lower bound. P 14.2.1

Let x and y be two vectors of real elements. Then

(X'y)2 ::; (x'x)(y'y) with equality if and only if y scalar a. Let x

PROOF.

i= o.

(14.2.1)

= ax for some scalar a or x = ay

for some

A simple demonstration, of (14.2.1) is as follows:

o -< ~( y,, _ ~XiYi2 x,.)2 = 't"

~xi

~ y,2 _ (~XiYi)2 't" 2 ~xi

(14.2.2)

For equality in (14.2.2), we must have Yi = [(~xiYi)/~xrlxi for all i, i.e., (~XiYi)/(~Xn is constant, say a. If PI, ... ,Pn are non-negative numbers, it follows from (14.2.1) that the weighted version

(14.2.3) is also true. P 14.2.2 Then

(1)

Let x and y be two real vectors and A be an nnd matrix.

(x'Ay)2::; (x'Ax}(Y'Ay),

(14.2.4)


(2) (3)

(x'y)2 ~ (x' Ax)(y' A-I y ), (x'x)2 ~ (x' Ax)(x' A-Ix),

if A-I exists, if A-I exists.

455

(14.2.5) (14.2.6)

P 14.2.3 Let x and y be two complex vectors and A be a Hermitian nnd matrix. Then (1) Ix*yl2 ~ (x*x)(y*y), (14.2.7) (2) Ix* Ayl2 ~ (x* Ax)(y* Ay), (14.2.8) (3) Ix*yl2 ~ (x* Ax)(y* A-I y), if A-I exists, (14.2.9) where I . I represents the absolute value. P 14.2.4 Constrained Cauchy-Schwartz inequality. Let x and y be two vectors of order n X 1 and B be a matrix of order n x n. Then (1) (x'y)2 ~ (x' PBX)(Y'y) if Y E Sp(B), where PB is the projection operator on Sp(B), (2) (x'y)2 ~ (y'B-y)(x'Bx) if B is nnd, y E Sp(B), and B- is any g-inverse of B.

P 14.2.5

The integral version of the Cauchy-Schwartz inequality is ( / /g dv)2 = / A

A

/2

dv / g2 dv, A

where / and 9 are real functions defined on a set A, and / and 9 are square integrable with respect to a measure v.

P 14.2.6 The Cauchy-Schwartz inequality for non-null vectors x and y can be stated in the form x*x - (x*y)(y*y)-I(y*X)

2:

o.

A more general version of the inequality is obtained when x and yare replaced by matrices X and Y, yielding that X* X - (X*y)(y*y)-(y* X) 2:L 0,

(14.2.10)

i.e., the matrix in (14.2.10) is nnd. Using Complement 14.1.3, we also have the weaker version in terms of determinants that IX* XI 2: IX*Y(Y*Y)-Y* XI·

(14.2.11)

456


The Cauchy-Schwartz inequality gives an upper bound to x'y. We mention without proof some complementary inequalities which give a lower bound to x'y = (XIYl + ... + XnYn). P 14.2.7 Let 0 < ml ::; Xi ::; M 1, 0 < m2 ::; Yi ::; M 2 , and o < m < "Yi ::; M for every i = 1, 2, . .. ,n. Further, let ~i (i = 1, . .. ,n) be real numbers. Then the following inequalities hold. (1)

(E"Yd(E~)::; (~~ m)2 n 2.

( 2)

(E

(3) ( 4)

"Yi

m

(Schweitzer)

~)(EY.2) < -

(MIM2 + mlm2)2 (E . .)2 (Polya-Szego) 4 M M X.Y.· mlm2 I 2 ~7 (M +m)2 (E"Yi~t)(E-1-)::; M (E~t)2. (Kantorovich) "Yi 4 m

(E

x.

~C2)(EY.2C~) < (MIM2 +Mmlm2)2 (E . . c 2 )2 X.... ... 4 M x.Y....· mlm2 I 2 (Greub-Rheinboldt)

The inequality (4) can written in a form to include the lower and upper bounds to the inner product. (See Diaz and Metcalf (1964).) 1

< (Ex~~;)(Ey;~;) < (MIM2 + mlm2)2 -

(EXiYi~n2

(14.2.12)

4M1M2mlm2

14.3. Hadamard Inequality P 14.3.1

For a nonsingular real n x n matrix B = (b ij ) IBI2 ::; rri=I(Ek=lb~k).

(14.3.1)

As special cases, we have (1) IBI::; 1, if Ej=lbrj = 1, i = 1, ... , n, (2) IBI::;Mn n n/2, if Ibijl::;M foralliandj.

(14.3.2) (14.3.3)

PROOF. To prove (14.3.1), we consider the pd matrix A = BB' and apply the inequality IAI ::; au ... ann, the product of the diagonal elements of A which is proved as follows. Consider the expansion

IAI

= au

+

o a21

a12 a22

al n a2n


457

Since A is pd, the matrix (aiih::;i::;n,2::;i::;n is also pd, and the second term is negative (Why?). Hence

The result IAI :::; alla21 ... ann follows by induction. The result (14.3.1) follows by observing that aii = bTl + ... + bTn.

14.4. Holder's Inequality P 14.4.1 p

Let

Xi, Yi ~

0, i

=

1, ... ,n and (l/p)

+ (l/q) =

1 with

> 1. Then (using E to indicate summation over i = 1, . . . ,n), (14.4.1)

with equality if and only if y/s are proportional to xf-1,s. The integral version of Holder's inequality for functions I, 9 ~ 0 is given by

(14.4.2) PROOF.

First, we establish by differentiation or otherwise min {t(x) = x

P

p

x~O

and the minimum is attained at X U,v ~ 0 in t(x) ~ 1, we find

+ x-

q }

q

=

= 1,

1. Substituting X

uP v q uv 1 such that 1

1

1

PI

P2

p""

- + - + ... +-:S l. Then

n

n

Lrr~lx~j) :S rrT=I[L(x~j»)PjP/Pj.

(14.4.4)

i=1

i=1

The result is easily established. - From Holder's inequality, it is easy to deduce that if P 2:: 1, then

Xi,

Yi 2:: 0 and (14.4.5)

which is Minkowski's inequality, where all summations are from 1 to n.

14.5. Inequalities in Information Theory

(1) Let Eai and Eb i be convergent sequences of positive numbers such that Eai 2:: Ebi. Then bEai log -.:. a •-

:s 0

with equality being attained if and only if and bi :S 1 for all i, then 2Ea•-Iog -ai b i

(14.5.1) ai

= bi . Further, if

2 < Ea •-(a•- - b-) •.

-

To prove the inequalities, note that for at x = 1 yields logx = (x -1) - (x _1)2(2y2)-1

X

> 0,

ai

:S 1

(14.5.2)

the expansion of log x

with y E (l,x) .

(14.5.3)

Using the expansion (14.5.3) for each term in (14.5.1), we have bi Eai log ai = (Eb i - Eai) - Eai(b i - ai)2(2a~y~)-1 :S 0

(14.5.4)


459

thus proving (14.5.1). In (14.5.4), ary'f E (ar,bn, and if ai ~ l,bi ~ 1, the maximwn value of y'f is not greater than unity. Hence (14.5.5) Combining (14.5.4) and (14.5.5), we obtain (14.5.2). The results (14.5.1) and (14.5.2) are true if ai and bi are non-negative and the summations are extended over values of i for which ai > O. In other words we admit the possibility of some of the bi being zero (but not the ai). (2) Let f and 9 be non-negative and integrable functions with respect to a measure J.L and S be the region in which f > O. Then

{ (f - g)

Js

with equality only when

f

~ 0 ~ (f log £dJ.L ~ 0

Js

(14.5.6)

9

= 9 a.e·lfL]·

The proof is the same as that of (14.5.1) with summations replaced by integrals. The reader may directly deduce the inequalities (14.5.1) and (14.5.6) by applying Jensen's inequality (14.6.6), given in the next section. 14.6. Convex Functions and Jensen's Inequality

A function f(x) is said to be convex, if for a, j3

f(ax + j3y)

~

af(x) + j3f(y)

> 0, a + j3 = 1

for all x, y.

(14.6.1)

For such a function, we shall first show that at any point xo,J~(xo), f!....(xo), the right and left derivatives, respectively, exist. Let Xo < Xl < X2. Choosing a = (X2 - XI)/(X2 - xo), j3 = (Xl - XO)/(X2 - xo), X = Xo, and y = X2 so that ax + j3y = Xl, and using (14.6.1), we see that (14.6.2) after multiplication by (X2 - xo). Adding xof(xo) to both sides and rearranging the terms in (14.6.2), we have (14.6.3)

460


which shows that (J(x) - f(xo)lI(x - xo) decreases as x ---- Xo· By adding xd(xt} to both sides of (14.6.2) and rearranging the terms, we have ( ) -.f(.'-....:xo::.!..)_-...::....f~(x.::..!...d < f(X2) - f(xt} . 14.6.4

Xo

-Xl

Equation (14.6.4) in terms of X-I

< Xo < Xl becomes

f(x-t} - f(xo) < f(XI) - f(xo) , X-I - Xo Xl - Xo which shows that [f(x) - f(xo)lI(x - xo) is bounded from below, and since it decreases as X ---- Xo, the right-hand derivative f~(xo) exists. Similarly, f~(xo) exists and obviously f~(xo) ~ f~(xo). Let L be such that f~(xo) ~ L ~ f~(xo). Then, for all x,

f(x) ;::: f(xo) for if Xl

+ L(x -

xo),

(14.6.5)

> Xo,

and the reverse relation is true when Xl < Xo . The inequality (14.6.5) has important applications; it leads to Jensen's inequality. P 14.6.1 (Jensen's Inequality). If X is a random variable such that E(X) = J.L and fO is a convex function, then

E[f(X)] ;::: f[E(X)]

(14.6.6)

with equality if and only if X has a degenerate distribution at J.L or f is a linear function. PROOF.

Consider the inequality (14.6.5) with J.L for Xo ,

f(X) ;::: f(J.L)

+ L(X -

J.L),

(14.6.7)

and take expectations of both sides. The expectation of the second term on the right-hand side of (14.6.7) is zero, yielding the inequality (14.6.6).


461

Let f (x) be a convex function of a vector x E Rn. Then a result analogous to (14.6.7) is

f(x) ~ f(f-L)

+ L'(x -

f-L)

(14.6.8)

where L ERn. Using (14.6.8), we find that the result (14.6.6) is true for a convex function of a vector variable. P 14.6.2 Let Xl, ... , Xn be positive real numbers and f-LI ... , f-Ln be arbitrary real numbers. Then

n n n (Lxil)-l ::; (L Xif-Li 2)(Lf-Li l )-2. i=l

PROOF.

i=l

(14.6.9)

i=l

By the Cauchy-Schwartz inequality (14.2.1) n

n

n

(Lxil)(LXif-Li2) ~ (Lf-Lil)2 i=l

i=l

i=l

from which (14.6.9) follows . P 14.6.3 (Inequality on Harmonic Mean) Let Xl, .. . , Xn be positive random variables. Then the expected value of their harmonic mean is not greater than the harmonic mean of their expected values. Let E(Xi) = f-Li, i = 1, ... ,no Considering the inequality (14.6.9) after multiplying both sides by n and taking expectations, we get PROOF.

n

n

(14.6.10)

i=l

i=l

which is the desired result. See Rao (1996).

14.7. Inequalities Involving Moments Let X be a random variable such that E(X) = f-L. Assume that the moments E(X - f-Lr = f-Lr, r = 1, ... ,2N

462

MATRIX ALGEBRA THEORY AND APPLICATIONS N+I N+I

exist. The quadratic form

L: L:

J.Li+j-2 YiYj in Yb · · · ,YN+I

i=1 j=1

Hence the matrix whose (i,j)-th element is J.Li+j-2 is nnd. This, however, is not a sufficient condition for J.Lr to be a moment sequence of a random variable. The condition is necessary whether the moments are of a discrete or a continuous distribution, or calculated from a given set of observations. In particular, for N = 2,

o1

J.L2] 0 J.L3 J.L2 [ J.L2 J.L3 J.L4

2: 0 or J.L~

(J.L~ 2 J.L~ ) 2: - 1

-

J.L2

0

(14.7.1)

J.L2

i.e., f32 2: 1 + f31, where f31 = J.LV J.L~ and f32 skewness and kurtosis of a distribution.

=

J.L4/ J.L~ are measures of

14.8. Kantorovich Inequality and Extensions In this section, we focus on Kantorovich inequality and some of its variants. Application to some statistical problems is also broached.

P 14.8.1 then

If A is n x n Hermitian pd with eigenvalues Al 2: ... 2: An, x* Ax x* A-Ix

1 0 and define Wi

Then

= (c5i + c5n_i+d 2/4c5ic5n_i+l' (x' Ay)(y' A-Ix) (x'x)(y'y) -

"":"'--:--':""';-:---:----'- < wI· 14.8.2 (Khatri and Rao (1981).) Let A and 14.8.1. Consider

Wi

be as in Complement

= IX' APy A-I XI/IX' XI I{X, Y) = tr(PxAPyA-1), g{X, Y)

466


where X and Yare nxk and nxs matrices ofranks k and s, respectively, with s ~ k, Px = X(X'X)-IX' and Py = y(y'y)-ly'. Then min(k,n-k)

II

g(X, Y) :::;

i=1

Wi,

k

I(X, Y) :::; LWi if n ~ 2k, i=1

n-k

: :; L Wi + 2k -

rt

if n < 2k.

i=1

14.8.3 (Greub and Rheinboldt (1959).) Let G and H be pd commuting matrices with eigenvalues >'1 ~ ... ~ >'n > 0 and J-Ll ~ ••• ~ J-Ln > 0 respectively. Then

14.8.4 IT >'1

~

>'2

~

...

~ >'n

> 0, show that

14.8.5 (Khatri and Rao (1982)) A measure of inefficiency alternative to (14.8.8) is

14.8.6 If B(X, Y) = X'A-IY(Y'A-Iy)-lY'A-IX and A(X) = X'A-IX - X'X(X'AX)-IX'X, then

sup IB(X, Y)I = IA(X)I. y

CHAPTER 15 NON-NEGATIVE MATRICES In this chapter, we will examine some of the features of the world of non-negative matrices. non-negative matrices occur naturally in several areas of application. From the statistical side, non-negative matrices figure prominently in Markov Chains. Some models in Genetics are based on non-negative matrices. Leontief models in Economics derive sustenance from non-negative matrices. We will touch upon these applications. One of the most prominent results in the area of non-negative matrices is the Perron-Frobenius Theorem. In the next section, we will dwell upon this remarkable result.

15.1. Perron-Frobenius Theorem We need to set some definitions in place to pave the way for an enunciation of the Perron-Frobenius Theorem. The concept of irreducible matrix is central to the development of this section. DEFINITION 15.1.1.

Let A = (aij) E Mn.

(1) The matrix A is said to be non-negative if aij 2': 0 for all i and j. (If A is non-negative, we use the symbol A 2':e 0 or 0 ~e A, the suffix e denotes entry wise.) In the general theory of matrices, the symbol A 2':L 0, alternatively A 2': 0, is used to indicate that A is non-negative definite. (2) The matrix A is said to be positive if aij > 0 for all i and j. (The symbol that is used in this context is A >e 0.) The concepts of nonnegativity and positivity perfectly make sense for matrices not necessarily square. If A and B are two matrices of the same order; we say that A 2':e B or A - B 2':e 0 to mean that if A = (aij) and B = (bij), then aij 2': bij for all i and j . 467

468


DEFINITION 15.1.2. A non-negative matrix A E Mn(n 2: 2) is said to be reducible if there exists a permutation matrix P E Mn such that PAP' is of the form

PAP' =

[~ ~],

(15.1.1)

where B E Mr and D E M n- r , and 0 E Mr,n-r is the null matrix, and

r> 1. What reducibility means is that if we can find some rows of A such that these rows and the corresponding columns of A are permuted, the resultant matrix has a structure stipulated in (15.1.1). The next question is how to identify reducible matrices. In the following proposition, we focus on this problem. We take n 2: 2 in all the propositions. P 15.1.3 Let A E Mn be a non-negative matrix. The matrix A is reducible if and only if there exists a nonempty proper subset [ of {I, 2, ... , n} such that aij = 0 for every i ~ I and j =f:. [.

Sufficiency. Let [ = {il < i2 < ... < id and Ie = {I, 2, ... , n} - I = {il < j2 < ... < jn-k}. Let u be the permutation map from {1,2, ... ,n} to {1,2, ... ,n} defined by u(t) = it, for t = 1,2, ... , k, and u(t) = jt-k, for t = k + 1, k + 2, .. . , n. Let P be the permutation matrix associated with the permutation map u . One can verify that PAP' is of the form (15.1.1). The necessity is clear. One of the characteristic features of reducible matrices is the following. Suppose A is a matrix already in the reduced form (15.1.1). Then Ak, for any positive integer k, is also reducible. More generally, if A is reducible then Ak is reducible for any positive integer k. (Why?) The notion of reducibility can be defined for matrices not necessarily nonnegative. We do not need the definition in generality. The negation of reducibility is one of the key concepts in this section. PROOF.

DEFINITION 15.1.4. A non-negative matrix A E Mn, n 2: 2, is said to be irreducible if it is not reducible. A trivial example of an irreducible matrix is any positive matrix. Another example is a 2 x 2 matrix with main diagonal elements zero and off diagonal elements unity. Matrices of order 1 x 1 are summarily excluded from discussion in this chapter. For us, n is always ~ 2. The following is a characterization of irreducible matrices.

Non-negative

Matrices

469

P 15.1.5 Let A = (aij) E Mn be a non-negative matrix. The following statements are equivalent.

(1) A is irreducible. (2) (/ + A)n-l is positive. (3) For any i and j with 1 ~ i ~ nand i ~ j ~ n, there exists a positive integer k = k(i,j) such that k ~ n and a~;) > 0, where Ak = (a~~»). PROOF. (1) ~ (2). Let y ~ 0 be a non-zero vector of order n x 1 and z = y + Ay = (I + A)y. Let us compute how many non-zero elements z has. Since Ay ~ 0, z has at least as many non-zero elements as y. Could it be possible that y and z have exactly the same number of non-zero

elements? Suppose it is possible. By rearranging the elements of y, if necessary, we can write y' = (u', 0) with u > O. Perforce, the vector z partitions as, z' = (v', 0) with u and v being of the same order and v > O. Partition A accordingly, i.e.,

Z=[V]=[U]+[All o 0 A21

A12] [u]. A22 0

This implies that 0= 0 + A 21 U. Since U > 0, we have A21 = O. Consequently, A is reducible. (Why?) This contradiction shows that z has more non-zero elements than y. Repeat this argument by taking z in the place of y. We find a vector s such that s = (I + A)z = (I + A)2y has more nonzero entries than z. Repeating this argument at most (n - 1) times, we find that (/ + A)n-l >e O. This is true for every non-zero vector y ~ O. Hence (/ + A)n-l >e O. (Why?) (2) ~ (3). Since (I + A)n-l > 0 and A ~e 0, we have

Consequently, for any (i, j), the (i, j) th entry of A, A2 , •• , positive.

,

or An is

470


(3) ~ (1). Suppose A is reducible. There exists a permutation matrix P such that

for every k ~ 1. (Why?) Consequently, we can find i =1= j such that the (i,j)th-entry, (PA k P')ij = 0 for all k ~ 1. Let P = (Pij). Thus we have n n L

LPira~~)pjs = 0 for all k ~ 1.

r=ls=l

For some rand s, Pir = 1 = Pjs. Therefore, a~~) = 0 for all k ~ 1. This is a contradiction to (3). This completes the proof. There is another concept closely related to irreducibility. For primitive matrices, one could obtain a stronger version of Perron-Frobenius Theorem. DEFINITION 15.1.6. A non-negative matrix A is said to be primitive if Ak is positive for some positive integer k. It is clear that every primitive matrix is irreducible. The converse is not true. Look at the case of 2 x 2 matrix with diagonal elements zero and off diagonal elements unity. We now introduce the notion of the modulus of a matrix. Let A = (aij) be a matrix of any size. We define m(A) = (Iaijl). The following properties of the operation of modulus are easy to establish.

P 15.1.7 (1) If A and B are two matrices such that AB is defined, then m(AB) ~e [m(A)][m(B)]. (2) If A is a square matrix, then m(Ak) ~e [m(A)]k for all positive integers k. (3) If A and B are square matrices of the same order such that m(A) ~e m(B), then Ilm(A)IIF ~ IIm(B)IIF, where II·IIF stands for the Frobenius norm. We now focus on the spectral radii of matrices. Recall that the spectral radius Ps(A) of a square matrix A is the maximum of the absolute values of the eigenvalues of A.

Non-negative Matrices

471

P 15.1.8 If A, B E Mn and m(A) ~ B, then Ps(A) ~ ps[m(A)] ~ Ps[(B)]. (In other words, the spectral radius Ps(-) is monotonically increasing on the set of all non-negative matrices in Mn.) PROOF. (1) Note that for any positive integer k,

m(Ak)

~e

[m(A)]k

~e Bk.

It now follows that (IIm(Ak)IIF)I/k ~ (IIm(Ak)II)I/k ~ (IIBkll)l/k. By taking the limit as k ---. 00 now, we note that Ps(A) ~ ps[m(A)] ~ Ps(B). See P 11.2.17.

P 15.1.9 Let A = (aij) E Mn be a non-negative matrix, and B a principal submatrix of A. Then Ps(B) ~ Ps(A). In particular, max aii ~ Ps(A). l$i$n PROOF. Define a matrix C E Mn as follows. Place the entries of B in C in exactly the same position wherever they come from A. The remaining entries of C are zeros. We note that Ps(B) = Ps(C) (Why?) and C ~e A. [We use the result 0 ~e Al ~e A2 ~ Ps(A 1 ) ~ Ps(A2)']

P 15.1.10 Let A = (aij) be a non-negative matrix such that all the row sums of A are equal to the same number 0:. Then Ps(A) = 0:. If all the column sums of A are equal to the same number /3, then Ps(A) = /3. PROOF. Recall the form of the induced leo-norm on

(bij ), IIBlleo,in = l~t'tn I:j=l

Ibijl·

Mn. For B =

See P 11.2.8. This norm is a matrix

norm. Further, r;;:ll that Ps(B) ~ IIBII for any matrix norm II . II. Thus we observe that IIAlleo,in = 0: ~ Ps(A). On the other hand, note that 0: is an eigenvalue of A with eigenvector (1,1, ... ,1)'. Therefore, 0: ~ Ps(A). This proves the first part of the proposition. For the second part, use the matrix norm, 1I·lkin. See P 11.2.11.

P 15.1.11 Let A = (aij) E Mn be a non-negative matrix with row TI, T2, . .. , Tn and column sums ClI C2, . . . ,en. Then

sums

min Ti ~ Ps(A) ~ max Ti,

(1)

~n q ~

(2) PROOF. (1) Let

0:

l~i$n

l$i$n l$~$n

=

~n

l o. (2) Ps(A) is an eigenvalue of A. (3) There exists a positive eigenvector of A corresponding to the eigenvalue Ps(A). (4) If J.L is any other eigenvalue of A, then IJ.LI < Ps(A). (What this means, in particular, is that we cannot find an eigenvalue J.L of A such that IJ.LI = Ps(A).) (5) The multiplicity of the eigenvalue Ps(A) is one, i.e., Ps(A) is not a repeated eigenvalue. PROOF.

(1) Since A is positive, it follows that Ps(A)

> 0, by P

15.1.11. To prove (2) and (3), let J.L be an eigenvalue of A such that IJ.LI = Ps(A). Let x be a corresponding eigenvector and Ixl = m(x). Then

Ps(A)lxl = IJ.LIIXI = IJ.LXI = IAxl ~e IAllxl = Alxl· Let y = Alxl - Ps(A)lxl. It is clear that y ~e o. If y = 0, then Ps(A) is an eigenvalue of A with a corresponding eigenvector Ixl. Since Ixl ~e 0, Ixl # 0, and A is positive, Alxl >e o. FUrther, Ixl = [Ps(A)]-1 Alxl >e

O. This establishes (2) and (3) in case y = O. Suppose y # O. Set u = Alxl which is, obviously, positive. Note that, since A is positive,

o <e Ay = A(Alxl- Ps(A)lxi) = Az - Ps(A)z, which means that Az >e Ps(A)z. By P 15.1.12, Ps(A) > Ps(A), which is not possible. Hence y = o. Thus (2) and (3) are established. (4) Let J.L be an eigenvalue of A such that J.L # Ps(A). By the very definition of spectral radius, IJ.LI ~ Ps(A). We claim that IJ.LI < Ps(A). ~uppose IJ.LI = Ps(A). Let x be an eigenvector of A corresponding to the eigenvalue J.L. Following the argument presented in the proof of (2) and (3) above, it follows that Ixl >e 0 and Ixl is an eigenvector of A corr~ponding to the eigenvalue Ps(A). Let Xi be the i-th component of x. The equation Ax = J.LX implies that n

Ps(A)lxil =

n

IJ.L II Xi I = IJ.LXil = I"LaijXil ~ "Laijlxjl = Ps(A)lxiJ, j=1

j=1

474


for each i. Thus equality must prevail throughout in the above. This means that the complex numbers aijXj, j = 1,2, ... ,n must lie on the same ray in the complex plane. Let () be their common argument. Then e-i8aijXj > 0 for all j. Since aij > 0, we have w = e-ilix >e O. The vector w is also an eigenvector of A corresponding to the eigenvalue J-L of A, i.e., Aw = J-LW. Since W >e 0, J-L ~ O. (Why?) Trivially, J.LW :Se Aw :Se J.LW. By P 6.1.12, J-L :S Ps(A) :S J-L. This contradiction establishes the claim. (5) First, we establish the following result. If 0 <e C :Se D and Ps(C) = Ps(D), then C = D. We now know that there exist positive eigenvectors x and y such that Cx = Ps(C)x and Dy = Ps(D)y. We would like to show that x is also an eigenvector of D corresponding to the eigenvalue Ps(C) of D. Note that 0 <e Ps(C)x = Ps(D)x = Cx :Se Dx. Let z = Dx - Ps(D)x. It is clear that z ~e O. We claim that z = O. Suppose z i= o. Let y = Dx. Note that 0 <e Dz = D(y - Ps(d)x) = Dy - Ps(D)y leading to the inequality Dy >e Ps(D)y. By P 15.1.12, Ps(D) < Ps(D). This contradiction establishes that z = O. Thus the vector x is a common eigenvector of C and D corresponding to the same eigenvalue Ps(C), i.e., Cx = Dx. Since x >e 0 and C :Se D, we have C= D. (Why?) Let p(x) = IxI -AI. Let Aii be the principal submatrix of A obtained from A by deleting the i-th row and i-th column of A. Write

where J-LI! J-L2, ••• ,J-Ln-l are the eigenvalues of A ii . Since Ps(Aii) :S Ps(A), IJ-Lil :S Ps(A) for every i. See P 15.1.9. We can say something stronger. Since A is positive, Ps(Aii) < Ps(A), by what we have seen at the beginning of the proof of (5). Equivalently, IJ-Lil < Ps(A) for every i. Consequently, IxI - Aiil > 0 if x ~ Ps(A). In particular, IPs(A)I - Aiil > 0 for all i. Observe that

This shows that the eigenvalue Ps(A) is simple. This completes the proof.


475

The spectral radius Ps(A) of a positive matrix is called the Perron root of A. The associated positive eigenvector x' = (Xl, X2, ••. ,x n ), i.e.,

Ax

= [Ps(A)]x,

X

>e 0, with

n

L: Xi = 1 is called the right Perron vector i=l

of A. Note that A' is also positive. The spectral radius remains the same. The right Perron vector y of A' is called the left Perron vector of

A. P 15.1.14 is usually called Perron's theorem. A similar statement has been established by Frobenius in the environment of irreducible matrices. We now focus on irreducible matrices. The extension of P 15.1.14 revolves around comparing the eigenvalues of A and those of I + A. Some results in this connection are worth noting. P 15.1.15 Let A E Mn with eigenvalues )'1, A2, ... ,An. Then the eigenvalues of I + A are 1 + All 1 + A2, ... ,1 + An. Further, Ps(I + A) :::; 1 + Ps(A). If A is non-negative, then Ps(I + A) = 1 + Ps(A).

The first part of the result follows easily. Note that Ps(I + m!lX 11 + Ail:::; 1 + max IAil = 1 + Ps(A). If A ~e 0, then

PROOF.

A)

=

l~t~n

l~.~n

1 + Ps(A) is an eigenvalue of I

+ A.

P 15.1.16 Let A be a non-negative matrix such that Ak is positive for some positive integer k, i.e., A is primitive. Then the assertions of P 15.1.14 hold. [This is easy to establish.] P 15.1.17 (Perron-Frobenius Theorem) Let A E Mn be a nonnegative irreducible matrix. Then:

(1) Ps(A) > 0. (2) Ps(A) is an eigenvalue of A. (3) There exists a positive eigenvector eigenvalue Ps(A) of A. (4) The eigenvalue Ps(A) is simple.

X

of A corresponding to the

PROOF. Since A is irreducible, (I + A)n-l is positive. P 15.1.14 becomes operational for the matrix (I + A)n-l. Now, (1), (2), (3), and (4) follow. Use P 15.1.15 and P 15.1.16. There is one crucial difference between P 15.1.14 and P 15.1.17. If the matrix A is irreducible, it is possible that there is eigenvalue A of A such that IAI = Ps(A).

476


Complements 15.1.1 Let A = (aij) be a matrix of order 3 x 3 with exactly one entry of A equal to zero. Characterize the position of this single zero in the matrix so that A becomes reducible. 15.1.2 Let A = (aij) be a matrix of order 3 x 3 with exactly two entries of A equal to zero. Characterize the position of these zeros in the matrix so that A becomes reducible. 15.1.3 Let A = (aij) be a non-negative matrix such that aii > 0 for all i. Show that A is primitive. 15.1.4 If A is irreducible, show that A' is irreducible. 15.1.5 Let A >e O. Suppose x and yare eigenvectors of A corresponding to the eigenvalue Ps(A) of A. Show that x = ay for some number a. (The eigenvalue Ps(A) is of geometric multiplicity one.) 15.1.6 Let A

=

I-a [

a

f3 f3 ] ,0 < a, f3 < 1.

1_

Determine the Perron root and Perron right vector of A. Examine the asymptotic behavior of A k as k --+ 00. 15.1.7 Let A be a positive matrix. Assess the asymptotic behavior of Ak as k --+ 00. More precisely, show that [AI Ps(A)]k converges as k --+ 00. Show that the limit matrix L is given by L = xy', where Ax = [Ps(A)]x, x >e 0, y' A = [Ps(A)]y', y >e 0, and x'y = 1. (The proof of P 15.3.2 can be adapted in this case by taking L in the place of Q.) Show that exactly the same conclusion is valid for a primitive matrix A. 15.1.8 Let A be a positive matrix and x the right Perron vector of A. Show that Ps(A) = ~~aijXj, the summation being over i,j = 1, ... ,n. 15.1.9 If A is a positive non-singular matrix, demonstrate that A-I cannot be non-negative. 15.1.10 Establish a statement analogous to P 15.1.14 for primitive matrices. Prove this statement using P 15.1.14. 15.1.11 Let In, n 2: 1 be the Fibonacci sequence, i.e., It = h = 1, In = In-I + In-2, n 2: 3. One can show that lim ~f n = I-2,ft the n~oo n-l

'

golden ratio. For any odd positive integer n = 2m + 1, define a matrix


477

An = (a~j») of order n x n by

a~~) lj

= {

~ o

if

Ii - jl =

1,

ifi =j = m+ 1, otherwise.

Show that Ps(An) :::; J5. Hint: Use P 15.1.12. 15.1.12 Let A be a non-negative matrix. Show that Ps(A) is an eigenvalue of A. Show that there exists a vector x ~ 0 such that Ax = (Ps(A)Jx. 15.1.13 For the following matrices, examine which of the properties (1), (2), (3), (4), and (5) of Theorem 15.1.14 are violated:

15.2. Leontief Models in Economics

We begin with a description of Leontief's Model for an economic system involving inputs and outputs of the industries comprising the economy. Suppose an economy has n industries and each industry produces (output) only one commodity. Each industry requires commodities (inputs) from all the industries, including itself, of the economy for the production of its commodity. No input from outside the economy under focus is needed. This is an example of a closed economy. The problem is to determine suitable ''prices'' to be charged for these commodities so that to each industry total expenditure equals total income. Such a price structure represents equilibrium for the economy. Let us fix: the notation and formulate the problem. Let aij

= the fraction of the total output of the j-th industry purchased by the i-th industry, i, j = 1,2, ... , n.

It is clear that aij ~ O. Further, alj + ... + anj = 1,j = 1, ... ,n. Let A = (aij). Thus A is a non-negative matrix and each column of A sums up to unity. Consequently, the spectral radius Ps(A) of A is unity. (Why?) \Ve assume that A is known. The matrix A is called

478


the input-output matrix of the economy. Let Pi = price for the i-th industry for its total output, i = 1,2, .. . ,n. The equilibrium condition can be stated as follows. Total expenditure incurred by the i-th industry is equal to the total income of the i-th industry, i.e., ailPl

+ ... + ainPn = Pi, i = 1, ...

,n.

These equations can be rewritten as Ap = p, where pi = (pt, P2, ... ,Pn)' The objective is to determine the price vector p. We are back in the realm of non-negative matrices. Since Ps(A) = 1, we are looking for the eigenvector of A corresponding to the eigenvalue Ps(A) of A. This problem falls into the realm of the Perron-Frobenius theorem. In practice, one looks for a positive solution (viable solution) of Ap = p. If A is irreducible, we know that P exists and is positive. As an example, suppose an economy has four industries: a steel plant, an electricity generating plant, a coal mine, and an iron ore mining facility. Twenty percent of the output of the steel plant is used by itself, 30 percent of the output of the steel plant is used by the electricity generating plant, 15 percent by the coal mine, and 35 percent by the iron ore facility. Twenty percent of the electricity generated is used by the steel plant, 25 percent by itself, 25 percent by the coal mine, and 30 percent by the iron ore facility. Thirty percent of the coal produced by the coal mine is used by the steel plant, 30 percent by the electricity generating plant, 20 percent by itself, and 20 percent by the iron ore facility. Finally, 80 percent of iron ore produced by the iron ore mining facility is used by the steel plant, 10 percent by the electricity generating plant, 10 percent by the coal mine, and nothing for itself. The corresponding input-output matrix works out to be A=

0.20 0.30 [ 0.15 0.35

0.20 0.25 0.25 0.30

0.30 0.30 0.20 0.20

0.80 0.10 0.10 0.00

1 .

(15.2.1)

The basic problem is to determine how much the total output of each industry is to be priced so that total expenditure equals total income for each industry. Note that the matrix A is irreducible. There exists a positive vector p satisfying Ap = p. As a matter of fact, any multiple of p constitutes a solution to the equilibrium equation.


479

Now we begin with a description of an open economy. Suppose there are n industries in an economy each producing only one type of commodity. Portions of these outputs are to be used in the industries within the economy but there is also some demand for these commodities outside the economy. The prices of units of these commodities are fixed and known. Let us introduce some notation. Let d i = monetary value of the output of the i-th industry demanded by sources outside the economy, i = 1,2, ... ,n. Let d' = (d 1 , d2 , ••• ,dn ). Since the prices are known, the sources outside the economy can compute how much is the monetary value of the commodities they are seeking from the economy. The vector d is known. Denote by Cij, the monetary value of the output of the i-th industry needed by the j-th industry in order to produce one unit of monetary value of its output, i,j = 1,2, ... ,n. Let C = (Cij). Clearly, C is non-negative. The matrix C is called the consumption matrix of n

the economy. Normally,

E Cij

~

1 for each i. If the sum is equal to 1,

i=1

the industry is not profitable. Finally, let Xi be the monetary value of the total output of the i-th industry, i = 1,2, ... ,n. The objective is to determine the values of Xi'S so that the needs of the industries within the economy and the demands from sources outside the economy are n

exactly met. Let x' = (x}, X2,'" ,xn ). Note that

E CijXj is the monej=1

tary value of the output of the i-th industry needed by all the industries inside the economy. Consequently, x - Cx represents monetary values of excess outputs of the industries. We set x - Cx = d to match the excess output with the demand. The objective is to determine x 2: 0 so that (15.2.2) (I - C)x = d is satisfied. If (I - C) is nonsingular, there is a unique solution to the system (15.2.2) of equations. The solution may not be non-negative. If (1 - C)-l is non-negative, we will then have a unique non-negative solution x satisfying (15.2.2) for any given demand vector d 2: O. The following results throw some light on solutions to the system (15.2.2).

P 15.2.1 Let C E Mn be a non-negative matrix. Then (I - C)-1 exists and non-negative if and only if there exists a non-negative vector x such that x > Cx. (The condition x > Cx means that there is some production schedule x such that each industry produces more than it

480


consumes.)

If C = 0, the result trivially holds. Assume that C i= 0. Suppose (1 _C)-1 exists and non-negative. There exists a non-negative non-zero vector x such that (I - C)-Ix = dx, where d = Ps[(I - C)-I]. See Complement 15.1.12. We will show that d > 1. The equation PROOF.

(I -C)-Ix

= dx implies that x = ( d~1 )cx.

Since x

~ 0, x i= 0, C ~ 0,

it follows that d > 1. Moreover, [d/(d-1)] > 1. Hence x> Cx. Suppose for some vector x ~ 0, x > Cx. It means that x better be positive. If this is the case, we can find < .x < 1 such that Cx < .xx. (Why?) This implies that Ckx < .xkx for all k ~ 1. Consequently, lim C k = o.

°

k ..... oo

Since (1 - C)(1 + C + C2 + ... + cm) = 1 - cm+l, which converges to as m _ 00, it follows that the series 1 + C + C 2 + ... is convergent and is equal to (1 - C)-I. Thus I - C is invertible. It is clear that (I - C)-1 ~ 0. This completes the proof.

°

The following results are consequences of P 15.2.1. COROLLARY 15.2.2. Let C be a non-negative matrix such that each of its row sums is less than one. Then (I - C)-1 exists and is nonnegative.

15.2.3. Let C be a non-negative matrix such that each of its column sums is less than one. Then (I - C)-1 exists and is nonnegative. The essential difference between the closed model and open model are the following. (1) In the closed model, the outputs of the industries are distributed among themselves. In the open model, an attempt is made to satisfy an outside demand for the outputs. (2) In the closed model, the outputs are fixed and the problem is to detenmne a price structure for the outputs so that the total expenditure and total income for each industry are equal. In the open model, the prices are fixed and the problem is to determine a production schedule meeting the internal and external demands. Complements 15.2.1 For the input-output matrix A of (15.2.1), examine equilibrium solutions and interpret them. COROLLARY

Non-negative Matrice8

481

15.2.2 Three neighbors have backyard vegetable gardens. Neighbor A grows tomatoes, n~ighbor B grows corn, and neighbor C lettuce. They agree to divide their crops among themselves. Neighbor A keeps half of the tomatoes he produces, gives a quarter of his tomatoes to neighbor B, and a quarter to neighbor C. Neighbor B shares his crop equally among themselves. Neighbor C gives one-sixth of his crop to neighbor A, one-sixth to neighbor B, and the rest he keeps himself. What prices the neighbors should assign to their respective crops if the equilibrium condition for a closed economy is to be satisfied, and if the lowest-priced crop is to realize $75? 15.2.3 A town has three main industries: a coal mine, an electricity generating plant, and a local railroad. To mine $1 worth of coal, the mining operation needs 25 cents worth of electricity and makes use of 25 cents worth of transportation facilities. To produce $1 worth of electricity, the generating plant requires 65 cents worth of coal, 5 cents worth of its own electricity, and 5 cents worth of transportation needs. To provide $1 worth of transportation, the railroad requires 55 cents of coal for fuel and 10 cents worth of electricity. In a particular period of operation, the coal mine receives orders for $50,000 of coal from outside and the generating plant receives orders for $25,000 of electricity from outside. Determine how much each of these industries should produce in the period under focus so that internal and external demands are exactly satisfied. (Source: C. Rorres and H. Anton (1984).) 15.3. Markov Chains Let X o, Xl, . .. be a stochastic process, i.e., a sequence of random variables. Assume that each random variable takes values in a finite set {I, 2, ... ,k}. The set {I, 2, ... ,k} is called the state space of the process and members of the set are called the states of the process. In this case, it is easy to explain what a stochastic process means. For every n ~ 0 and i o, it, i2, ... ,in E {I, 2, ... ,k}, the probabilities, Pr{Xo = i o, Xl = i}, ... ,Xn = in}, are spelled out. Let Pi = Pr{Xo = i}, i = 1,2, ... ,k. The vector p' = (PI, P2, ... ,Pk) is called the initial distribution of the process, i.e., P is the distribution of Xo.

482


A physical evolution of a stochastic process can be described as follows. Suppose a particle moves over the states at times 0,1,2, .. . in a random manner. Let Xn be the state in which the particle is at time n, n ~ O. The joint distribution of the process X n , n ~ 0 describes the random movement of the particle over time. Let us introduce the notion of a Markov chain. ~

DEFINITION 15.3.1. The process X n , n if the conditional probability

Pr{Xn+1

0 is called a Markov chain

= jlXo = io, Xl = i l , . . . ,Xn- l = in-I, Xn = i} Pr{Xo = io, Xl = iI, ... ,Xn- l = in-I, Xn = i, X n+ l = j} = Pr{Xo = i o, Xl = iI, ... ,Xn- l = in-I. Xn = i} .

.

= Pr{Xn+1 = JIXn = z} =

Pr{Xn = i, X n +1 Pr{X = i} n

= j}

= Pij,

(say)

for all io, iI. " . ,in-I. i, and j E {I, 2, ... ,k} and n ~ O. The definition means several things: the conditional probability that X n+1 = j given the past {Xo = io, Xl = i l , . . . ,Xn - l = in-I, Xn = i} depends only on the immediate past {Xn = i}; the conditional probability does not depend on n. The numbers Pij'S are called one-step transition probabilities. The number Pij is the conditional probability of moving from the state i to state j in one step from time n to time

(n + 1). Let P = (pij). The matrix P is called the transition probability k

L

matrix. It has the special property that

Pij = 1 for every i. The

j=1

matrix P is an example of what is called a stochastic matrix. In the case of a Markov chain, the knowledge of the initial distribution P and the transition matrix P is enough to determine the joint distribution of any finite subset of the random variables X n , n ~ O. For example, the distribution of Xn is (p')pn. The joint distribution of X2 and X3 is given by Pr{X2 = i, X3

= j} = Pr{X2 = i}

Pr{X3

= jlX2 = i}

= Pr{X2 = i}Pij, for any i and j in the state space. The entry Pr{X2 component of (p')P2.

=

i} is the i-th


483

One of the basic problems in Markov chain is to assess the asymptotic behavior of the process. One determines lim Pr{ Xn = i} for every state n~oo

i. Asswne that P is irreducible. Irreducibility in the context of Markov chains has a nice physical interpretation. With positive probability one can move from any state to any state in a finite number of steps, Le., p~j) > 0 for some n ~ 1, where pn = (p~j»). See P 15.1.5. Note that the spectral radius P8(P) = 1. The conclusion of the main result of this section, i.e., P 15.3.2 is not valid for irreducible transition matrices. Let us assume that P is primitive. Look up Complement 15.1.10. Let q' = (ql. q2.··· • qk) be the left Perron vector of P, i.e., q> 0, q' P = q', k

and

l: q8 = 1.

The right Perron vector of A is x'

= (1,1, ...• 1),

i.e.,

8=1

Px = x. We want to show that the limiting distribution of Xn is q'. P 15.3.2 If the transition matrix P is primitive, then lim pn = Q, n~oo

where all the rows of Q are identical and equal to q'. Consequently, the limiting distribution of Xn is q', i.e., lim p' pn = q'. n~oo

PROOF.

Observe the following properties of the matrices P and Q.

(1) Q is idempotent. (2) PQ

= Qpm = Q for all m

~

(3) Q(P - Q) = O. (4) (P - Q)m = pm - Q for all M

1.

~ 1. (5) Every non-zero eigenvalue of P - Q is also an eigenvalue of P. This can be proved as follows. Let A be a non-zero eigenvalue of P - Q. Let w be a non-zero vector such that (P - Q)w = AW. Then Q(P - Q)w = AQW = o. This implies that Qw = 0 and Pw = AW. (6) P8(P) = 1 is not an eigenvalue of P - Q, i.e., I - (P - Q) is invertible. This can be proved as follows. Suppose 1 is an eigenvalues of P - Q. Then there exists a non-zero vector W such that (P - Q)w = w. This implies that Qw = 0 and w is an eigenvector of P corresponding to the eigenvalue 1 of A. Since the algebraic multiplicity of the eigenvalue 1 is one, w = ax for

484


some non-zero a, where x is the right Perron vector of P. Since Qw = 0, we must have Qx = o. This is not possible. (7) Let AIA2, ... ,Ak-l' 1 be the eigenvalues of P. Assume that JAIl ::; IA21 ::; ... ::; IAk-11 < 1. Then Ps(P - Q) ::; IAk-11 < 1. From Property 5 above, Ps(P - Q) = IAsl for some s, or = 1, or = O. From Property 6 above, we must have Ps(P - Q) ::; IAk-ll. (8) pm = Q + (P - Q)m for all m ;::: 1. Since Ps(P - Q) < 1, lim (P - Q)m = o. Consequently, lim pm = Q. m--+oo

m~oo

The last property proves the professed assertion. P 15.3.2 is the fundamental theorem of Markov chains. It asserts that whatever the initial distribution p may be, the limiting distribution of Xn is q', the left Perron vector of P. For primitive stochastic matrices, obtaining the left Perron vector of P is tantamount to solving the equations q' P = q' in unknown q.

Complements 15.3.1

Let P =

[~

~ J.

Show that for the transition matrix P,

the limit of pm as m - 00 does not exist. 15.3.2 If P is a stochastic matrix, show that pm is also stochastic for any positive integer m. 15.3.3 An urn contains a black and b red balls. At time n, draw a ball at random from the urn, note its color, put the ball back into the urn, and add c > 0 balls of the same color to the urn. Let Xn be the color of the ball drawn at n-th time, n ;::: 1. Obtain the joint distribution of Xl, X2, and X 3 • Evaluate the conditional probabilities, Pr{X3 = black IXt=black, X 2=black}, Pr{X3 = black IXI=red, X 2=black}, Pr{X3 = black IX2=black}. Show that X n , n ;::: 1 is not a Markov chain. 15.3.4 Show that the transition matrix

0.0 0.5 [ 0.5

0.5 0.5] 0.5 0.0 0.0 0.5


485

is primitive. Obtain the limiting distribution of the Markov chain driven by the above transition matrix. 15.3.5 A country is divided into three demographic regions. It is found that each year 5% of the residents of Region 1 move to Region 2 and 5% to Region 3. Of the residents of Region 2, 15% move to Region 2 and 10% to Region 3. Finally, of the residents of Region 3, 10% move to Region 1 and 5% to Region 2. Obtain the limiting distribution of the underlying Markov chain. (Source: Rorres and Anton (1984).) 15.4. Genetic Models Gregor Mendel is generally credited with the formulation of laws of inheritance of traits from parents to their offspring. One of the basic problems in genetics is to examine the propagation of traits over several generations of a population. Each inherited trait such as eye color, hair color, is usually governed by a set of two genes, designated by the generic symbols A and a. Plants and animals are composed of cells. Each cell has a collection of chromosomes. Chromosomes carry hereditary genes. Each hwnan being has roughly 100,000 pairs of genes. Each individual in the population has one of the pairings AA, Aa, or aa. These pairings are called genotypes. If the genes correspond to color of eyes in hwnan beings, the human being with genotype AA or Aa will have brown eyes, and the one with aa will have blue eyes. In such a case, we say that the gene A dominates a, or equivalently, the gene a is recessive. In what is called autosomal inheritance, an offspring will receive one gene from each parent. If the father is of the genotype AA, the offspring will receive the gene A from the father. If the father is of the genotype Aa, the offspring will receive either A or a from the father with equal probability. Similar considerations do apply to mothers. If the father is of the genotype AA and the mother is of the genotype Aa, the offspring will receive the gene A from the father and either gene A or gene a from the mother with equal probability. Consequently the genotype of the offspring is either AA with probability 1/2 or Aa with probability 1/2. If the genes correspond to eye color, the offspring will have brown eyes with probability one. (Why?) If both father and mother are of the same genotype Aa, the offspring is of genotype AA, Aa, or aa with probabilities 1/4, 1/2, or 1/4, respectively, If the genes correspond to

486


eye color, the offspring will have brown eyes with probability 3/4 or blue eyes with probability 1/4. In the following table, we list the possible genotype of offspring along with their probabilities. Genotmes of Qarents AA&AA AA&Aa AA&aa Aa &Aa Aa & aa aa & aa

GenotYQe of offsQring aa AA Aa 1 1/2 0 1/4 0 0

0 1/2 1 1/2 1/2 0

0 0 0 1/4 1/2 1

No distinction is made of (father, mother) pairings of genotypes (AA, Aa) and (Aa, AA). Let us look at a simple inbreeding model. Suppose in the O-th generation, the population consists of a proportion of Po individuals of genotype AA, qo of genotype Aa, and TO of genotype aa. Clearly, Po + qo + TO = 1. Suppose mating takes place between individuals of the same genotype only. Let in the population, Pn = proportion of genotype AA in the n-th generation, qn = proportion of genotype Aa in the n-th generation, and Tn = proportion of genotype aa in the n-th generation. We would like to determine the limiting behavior of these proportions. In the first generation, we will have PI = Po + (1/4)qo, ql = (1/2)qo, 1'1 = TO + (1/4)qo. These equations can be rewritten as 1/4 1/2 1/4 Let x~ = (Pn, qn, Tn), n ~ 0 and A the 3 X 3 matrix that appears in the above linear equations. It is clear that the vectors Xn's are governed by the equation,

Note that A is non-negative matrix. Let us determine the limiting behavior of the sequence An, n ~ O. The eigenvalues of A are Al =


1, >'2

487

= 1, and >'3 = 1/2 with corresponding eigenvectors chosen to be

Let

P=

[~ ~ o

- ; ] , and p-l = (1/2)

1

1

[~ ~ ~]. 0

-1

0

Note that A = P!:1p- l , where!:1 = diag{l, 1, 1/2}. It now transpires that An = P!:1 n p- l for all n ~ O. Consequently, for any n ~ 0,

An =

1 0

1/2 - (1/2)n+1

(1/2)n

[ o 1/2 - (1/2)n+1

More explicitly,

Pn

1 - (l)n+l] = Po + [2 2 qo, qn = (l)n 2 qo, Tn = TO + [12 - (l)n+l] 2 qo·

Consequently, lim Pn = Po

n-..oo

+ (~)qO, lim qn = 0, lim Tn = TO + (-2 )qo. 2 n--+oo n--+oo 1

In the long fUll, individuals of genotype Aa will disappear! Only pure genotypes AA and aa will remain in the population. Let us look at another model called selective breeding model. In the O-th generation, a population has a proportion po of individuals of genotype AA, a proportion qo of genotype Aa, and a proportion TO of genotype aa. In a special breeding program, all individuals are mated with individuals of genotype AA only. Let in the population Pn = proportion of genotype AA in the n-th generation, qn = proportion of genotype Aa in the n-th generation, and Tn = proportion of genotype aa in the n-th generation. We would like to examine the limiting behaviQr of these proportions. In the first generation, we have Pl

= Po

1

+ (2)qO,

ql

= TO

1

+ (2)qO,

Tl

= O.


488

These equations can be rewritten as 1/2 1/2

o

Let X~ = (Pn, qn, Tn), n ;::: 0 and A the 3 x 3 matrix that appears in the above linear equations. It is clear that the vectors Xn's are governed by the equation, Xn = An xo , n;::: o. Note that A is non-negative matrix. Let us determine the limiting behavior of the sequence An, n;::: o. The eigenvalues of A are >q = 1, >'2 = ~, and >'3 = 0 with corresponding eigenvectors chosen to be

m'[-i] ,

and

[-~]

Let

Note that A = PD.p-l, where D. = diag{1, 1/2, O}. It now transpires that An = pD.n p-l for all n ;::: o. Consequently, for any n ;::: 1,

An~ [~

1 - {1/2)n

{1/2)n

o

More explicitly,

Pn

(1)n-l TO, = 1 - ( :21)n qo -:2

Consequently, lim Pn n~~

= 1,

lim qn n~~

qn

(1)n_l TO, = (1)n :2 qo +:2

= 0,

lim Tn n~~

Tn

= O.

= O.

In the long run, individuals of genotype Aa and aa will disappear! Only the pure genotypes AA will remain in the population. As a matter of fact, individuals of genotype aa will disappear in the fist generation itself.


489

Let us look at a simple restrictive breeding model. In the O-th generation, a proportion Po of individuals is of genotype AA, a proportion qo of genotype Aa, and a proportion TO of genotype aa. Suppose only (AA, AA) and (AA, Aa) matings are allowed. This model is feasible if one wishes to eliminate certain genetic diseases from the population. In many genetic diseases such as cystic fibrosis (predominant among Caucasians), sickle cell anemia (predominant among Blacks), and Tay-Sachs disease (predominant among East European Jews), the relevant normal gene A dominates the recessive gene a. If an individual is of genotype Aa, he or she will be normal but a carrier of the disease. If an individual is of genotype aa, he or she will have the disease and the offspring will be at least a carrier of the disease. One would like to see the effect of the policy of preventing the sufferers of the disease to mate. Let Po be the proportion of the population of genotype AA and qo the proportion of the population of genotype Aa. Since the mating is restricted, the population is taken to be those individuals of genotype AA or Aa. Consequently, Po + qo = 1. Let in the population Pn = proportion of genotype AA in the n-th generation, and qn = proportion of genotype Aa in the n-th generation. One can check that for every n 2: 0,

[::] = An [ :~ ] ,with A =

[~

1/2 ] 1/2 .

A direct computation of powers of A yields

Consequently, lim Pn n--+oo

= 1 and n--+oo lim qn = O.

In the long run, the carriers

of the disease will disappear! 15.5. Population Growth Models One of the important areas in Demography is a critical examination of how population grows over a period of interest. The s~called "Leslie Model" describes the growth of the female portion of a human or animal population. In this section, we will describe the mechanics of this model, in which a non-negative matrix appears. We will outline the limiting


490

behavior of the powers of this matrix in order to shed light on the longterm growth of the population. Suppose the maximum age attained by any female in the population is M years. We classify the female population into some k age classes of equal length. Say, the age classes are: [0, M/k), [M/k, 2M/k), ... , [(k - l)M/k, MJ. When the study is initiated, note down

p~O) = number of females in the population falling into the age group [(i - l)M/k, iM/k), i = 1,2, ... ,k. The vector p(O}1 = (piO) , p~O} , ... ,p~O}) is called the initial age distribution vector. The main objective is to examine how the age distribution changes over time. We would like to examine the age distribution of the female population at times to = 0, tl = M / k, t2 = 2M/ k, and so on. As time progresses, the composition of the classes varies because of three biological processes: birth, death, and aging. These biological processes may be described by the following demographic parameters.

Let = the average number of daughters born to a single female during the time she is in the i-th age class, i = 1,2, ... ,k, bi = the proportion of females in the i-th class expected to survive and pass into the next class, i = 1, ... ,k.

ai

It is clear that ai ~ 0 for every i. Assume that 0 < bi ::; 1 for every i = 1,2, ... ,k - 1. Assume that at least one ai is positive. Let p(m}1 = i·b · f h C I . ( PI(m) ,P2(m) , ... , Pk(m)) b e th e age dstn utlOn 0 t e lema es at tIme t m , where p~m} is the number of females in the i-th age class, i = 1,2, ... ,k. The vectors p(m} 's satisfy the following recurrent relation: (m) PI

(m-l)

= alPI

+ a2P2(m-l) + ... + akPk(m-l)

(m) _ b. (m-l) . _ - t-IPi-1 ,Z -

Pi

2,3, ... ,k.

These equations can be put in a succinct form: - 1 2 3 P(m) -- L P(m-l) , m - " , ...

,


491

where

L= 000 the so-called Leslie matrix. Clearly, L is a non-negative matrix. It now follows that p(m) = Lmp(O). In order to facilitate the computation of powers of L, finding eigenvalues and eigenvectors of L is helpful. The eigenvalues of L are the roots of the determinantal equation

0= p(A) =

IL -

All

= Ak - alA k-

1

-

a2bl Ak - 2 - a3blb2Ak-3 - ... - akbl ... bk- 1 •

Since at least one ai is positive, there is at least one non-zero root of the polynomial equation. Consequently, the spectral radius Ps(L) > o. We record, without proof, some facts concerning the Leslie matrix. For some details, the reader may refer to Rorres and Anton (1984). (1) The eigenvalue Ps(L) is simple. (2) The vector x, given below, is positive and is an eigenvector corresponding to the eigenvalue Ps(L):

x' = (1, btl Ps(L), b1 b2/[Ps(L)f, ... , b1 b2 ... bk_tl(Ps(L)]k-l). (3) If two successive entries ai and ai+l are positive, then IAI < Ps(L) for any eigenvalue A of L different from Ps(L). Assume that ai and ai+1 are positive for some i. It now follows that (Ps(L)]-m Lmp(O) converges to a constant vector q which depends only on p(O). If m is large, p(m) ~ [ps(L)]mq ~ (Ps(L)]p(m-l). What this means is that the age distribution is a scalar multiple of the preceding age distribution. Complements

15.5.1

Comment on the eigenvalues of the following Leslie matrix:

4 2]

00. 1/8

0

492


= 15 and = (1000, 900,

Let M p(O)

15.5.2

k = 3. If the initial age distribution is given by 800), examine the age distribution after 15 years. Comment on the eigenvalues of the following Leslie matrix:

o 02] . 0 1/3 0 15.5.3 (Fibonacci numbers) In 1202 Leonardo of Pisa, also called Fibonacci, posed the following problem. A pair of rabbits do not produce any offspring during their first month of life. However, starting with the second month, each pair of rabbits produces one pair of offsprings per month. Suppose we start with one pair of rabbits and none of the rabbits produced from this pair die. How many pairs of rabbits will be there at tpe beginning of each month. Let Un be the pair of rabbits at the beginning of the n-th month. Establish the recurrence relation Un = U n - l + U n -2 and show that

where Al and A2 are eigenvalues of the matrix

Notes: The books by Horn and Johnson (1985, 1991) provide a comprehensive treatment on matrices. Some sections of this chapter are inspired by these works. The book by Rorres and Anton (1984) gives a good account of many applications of matrices to real world problems. Their influence is discernible in some of the examples presented here.

CHAPTER 16 MISCELLANEOUS COMPLEMENTS

Some topics, not covered under the different themes of the earlier chapters, which have applications in statistics and econometrics are assembled in this chapter. The proofs are omitted in most cases, but references to original papers and books where proofs and other details can be found are given.

16.1. Sixnultaxleous Decolnpositioxl of Matrices In Section 5.5 we have given a number of results on simultaneous diagonalization of matrices. We consider more general results in this section.

D E F I N ~ T16.1.1. ~ ~ N Two matrices A, B E M, are said to be simultaneously diagonalizable if there exists a nonsingular transformation T such that T*AT and T*BT are both diagonal. D E F I N I T I ~16.1.2. N Two matrices A, B E M, are said to be diagonalizable by contragredient transformations if there exists a nonsingular transformation T such that T*BT and T-'A(T-')* are diagonal.

A typical example where both the definitions hold is the case of two Hermitian commuting matrices which are simultaneously diagonalizable by a unitary transformation. We quote here a number of theorems on simultaneous diagonalizability of two matrices under various conditions given in Rao and Mitra (1971b).

P 16.1.3 Let A E M, be a Hermitian matrix and B E M, be an nnd matrix with rank r 5 n and N E M,,,-, with p(N) = n - r be such that N*B = 0. Then the following hold.

494


(1) There exists a matrix L E M n ,. such that L*BL = J(E Mr) and L* AL = ~(E Mr) diagonal. (2) A necessary and sufficient condition that there exists a ?onsin~ lar transformation T such that T* AT and T* BT are both dIagonal IS (using the notation p(B) = rank of B) p(N* A) = p(N* AN).

(3) A necessary and sufficient condition that there exists a nonsingular transformation such that T* BT and T- 1A(T-1)* are diagonal (i.e., A and B are reducible by contragredient transformations) is p(BA)

= p(BAB) .

(4) If in addition A is nnd, then there exists a nonsingular transformation T such that T* AT and T* BT are both diagonal. (5) If in addition A is nnd, then there exists a nonsingular transformation T such that T*BT and T- 1A(T- 1)* are diagonal.

P 16.1.4 Let A and B be Hermitian and B nonsingular. Then, there exists a nonsingular transformation T such that T* AT and T* BT are both diagonal if and only if there exists a matrix L such that LAB- 1 L -1 is diagonal with real diagonal elements (i.e., AB- I is semisimpIe or similar to a diagonal matrix). For details regarding the above theorems and further results, the reader is referred to Chapter 6 in Rao and Mitra (1971b). 16.2. More on Inequalities In Chapters 10 and 14 we have discussed a number of inequalities which are useful in solving optimization problems and in establishing boWlds to certain functions. We quote here some results from a recent thesis by Liu (1995). P 16.2.1 (Matrix-trace versions of Cauchy-Schwartz inequality) Let B E Mn be nnd, A E Mn,m be such that Sp(A) c Sp(B), Z E Mn,m be arbitrary and B+ be the Moore-Penrose inverse of B. Then (1) (trZ' A)2 ~ (trZ' BZ)(trA' B+ A) with equality if and only if B Z and A are proportional.

495

Miscellaneous Complements

(2) tr[(Z' A)2] ~ tr[Z' BZA' B+ A] with equality if and only if BZA' is symmetric (3) A+ B+(A+), ;::: (A' BA)+ (in Lowner sense) where A+ and B+ are Moore-Penrose inverses of A and B respectively. As special cases of (1) we have

(vecA)(vecA)' ~ [(vecA),(I ® B+)vecA](I ® B) ee'

~

(e'B-e)B for e

E

Sp(B).

For an application of the last result in statistics and econometrics see Toutenburg (1992, pp.286-287).

P 16.2.2 Let BE Mn be nnd, A E Mn symmetric and T E Mn,k such that peT) = k, Sp(T) c Sp(B) and T' BT = h. Further let B+ be the Moore-Penrose inverse of B and Al ;::: . .. ;::: An be the eigenvalues of B+ A. Then the following maxima or minima with respect to T hold.

+ ... + Ak min(trT' AT) = An-k+l + ... + An max [tr(T' AT)2] = Ai + ... + A~ min[tr(T' AT)2] = A~-k+l + ... + A~ max [tr(T' AT)-I] = A;-2k+l + ... + A;l for A > 0, min[tr(T' AT)-l] = All + ... + A;I for A > O.

(1) max(trT' AT) = Al (2) (3) (4) (5) (6)

r = pCB)

The optimum values are reached when T = (tIl·· · Itk), where Bl/2ti are orthonormal eigenvectors of (B+)1/2 A(B+)1/2 associated with the eigenvalues Ai(B+ A), i = 1, ... ,k.

P 16.2.3 For A E Mm,n, Bl E Mm and B2 E Mn are nnd, and Mm,k such that Sp(T) c Sp(Bd and T' BIT = hand W E Mn,k such that SpeW) c Sp(B2) and W'B 2W = I k , we have

T

E

max tr(T' AW) = Al T,W

max tr(T' AW)2 = T,W

+ ... + Ak

Ai + ... + A~

496


where A~ are the eigenvalues of Bi ABt A'. This theorem is useful in the study of canonical correlations in multivariate analysis. (See Lin (1990) and Yanai and Takane (1992).) P 16.2.4 (Matrix Kalltorovich-type inequalities) Let B E Mn and nnd with p(B) = b and A E Mn,r such that Sp(A) c Sp(B) and p(A) = a :::; min(b, r). Further let Al 2:: ... 2: Ab > 0 be the eigenvalues of B. Then:

(1) A+ B+(A+), :::; (Al~~:)2 (A' BA)+ with equality if and only if A = 0 or A'BA = AI+AbA'A and A'B+A = AI+AbA'A. 2 2Al Ab (2) A+ B(A+)' - (A' B+ A)+ :::; (~ - v'Xb)2(A' A)+ with equality if and only if A = 0 or Al = Ab or A' BA = (AI +Ab-v'AIAb)A' A and A' B+ A = (AIAb)-1/2 A' A.

(3) A'B2A < -

(Al+ Ab)2 4AI Ab

A'BAA+BA with equality if and only if A = 0 or A' BA = ~ AI+Ab A' A and A' B2 A = A1 Ab A' A.

(4) A'B 2A-A'BAA+BA:::; HAl -Ab)2A'A with equality if and only if A = 0 or Al = Ab or ,2 2 A'BA = AltAbA'A and A'B2A = "1~AbA'A. Liu (1995) established the following inequalities from the above inequalities, where C . D is the Hadamard-Schur product. P 16.2.5 Let Al and Ab the maximum and minimum of eigenvalues of C ® D where C and Dare pd matrices. Then

( 1) (C· D)-1 -< C-l . D- 1 < -

(AI+Ab)2(C. 4AI Ab

(2) C · D - (C- 1 . D- 1)-1 :::;

(~_

(3) (C· D)2 < -

C2 . D2

< -

yIXb)2[

(AI +Ab)2 (C . 4AI Ab

D)2

(4) (C· D)2 - c2. D2:::; HAl - Ab)2[

(5) (C · D) :::; (C2 . D2)1/2 :::; 2A.)t:!b C· D (6) (C2.

D2

r/

2 - C · D

< -

(AI-Ab) [ 4(AI +Ab)

D)-1

497


P 16.2.6 (Schopf (1960)) Let A E Mn be pd with eigenvalues Al 2: ... 2: An and let for y =1= 0, J.Lt = y* Aty. Then

16.3. Miscellaneous Results on Matrices P 16.3.1

(Toeplitz-Hausdorff Theorem) The set

W(A)

= {< x,Ax >:

IIxll

= 1}

is a closed convex set. Note that W(U AU*) = W(A) for any unitary U and W(aA + bI) aW(A) + bW(I) for all a, bE C. Also A(A) E W(A), for any A(A). P 16.3.2

=

For any matrix A, the series

exp A = I

1

2

1

2

1

n

+ A + 2'. A + ... + ,A + ... + ,A + ... n. n.

converges. This is called the exponential of A. Then exp A is invertible and (expA)-I = exp(-A). Conversely every invertible matrix can be expressed as the exponential of a matrix. COROLLARY 16.3.3. Every unitary matrix can be expressed as the exponential of a skew-symmetric matrix. P 16.3.4

Let w(A)

= sup I < x, Ax>

lover IIxll

= 1.

Then

(1) w(A) defines a matrix norm, (2) w(UAU*) = w(A) for all unitary U, and (3) w(A) :S IIAII :S 2w(A) for all A, where IIAII = sup IIAxll· P 16.3.5 (Weyl's Majorant Theorem) Let A E Mn with singular value (TI 2: ., . 2: (Tn and eigenvalues AI, ... ,An arranged in such a way that IA 11 2: ... 2: IAn I. Then for every function ¢ : R+ ---+ R+ such that ¢(et ) is convex and monotonic increasing in t, we have

498


In particular

for all p 2: O. P 16.3.6 (Converse ofWeyl's Majorant Theorem) If )'1,'" ,>'n are complex numbers and Ut, ... ,Un are positive real numbers ordered as 1>'11 2: ... 2: I>'nl and Ul 2: ... 2: Un and if

1>'11 .. ·I>'kl:::; SI·· 'Sk 1>'11·· ·I>'nl = SI··· Sn

for 1:::; k:::; n

then there exists a matrix A E Mn with eigenvalues gular values U}, .. . ,Un'

>'1 '"

>'n and sin-

P 16.3.7 (Fischer's Inequality) Let PI, ... ,Pr be a family of mutually orthogonal projectors in en such that PI El1 ... El1 Pr = In. Then for A 2: 0

[The matrix PIAPI

+ ... + PrAPr

is called the pinching of A.I

P 16.3.8 (Aronszaju's Inequality) Let A E Mn be Hermitian matrix partitioned as

A=

(~*~)

where B E M k • Let the eigenvalues of B, C and A be 131 > 13k; II 2: ... 2: In-k; and al 2: ... 2: an respectively. Then

ai+j-I

+ an :::; 13i + Ij for all

i,j with i

+j

- 1 :::; n.

P 16.3.9 Let A and B be pd matrices. Then the following hold. (1) IIA8 B 811 :::; IIABII8 for 0 :::; S :::; 1. (2) IIABW:::; IIAt Btll for t 2: 1. (3)

>'1 (A8 B8)

:::; >.f(AB) for 0 :::;

(4) [>'I(AB)l t :::;

>'1 (AtBt)

S :::;

for t 2: 1.

l.

>

MiscellaneoU8 Complements

499

(6) (Araki-Lieb-Thirring Inequality)

where A ~ 0, B ~ 0 and sand t are positive real numbers with

t

~

1.

(7) (Lieb-Thirring Inequality) Let A and B be nnd matrices and m, k be positive integers with m ~ k. Then

In particular

P 16.3.10 (n-dimensional Pythagorean Theorem) Let Xl,." , be orthogonal vectors in Rn, and 0 denote the origin. Let the volume of the simplex (Xl, ••. ,xn ) be V and that of the simplex (0, Xl,'" ,Xi, Xi+b' .. ,xn ) be Vi. Then Xn

A formal proof is given by S. Ramanan and K.R. Parthasarathy. For proofs of the propositions in this section, and further results, . reference may be made to Bhatia (1991). P 16.3.11 Then

Let AI, ... ,An be pairwise commuting and pd matrices. A(A) ~ Q(A) ~ 1t(A)

where A(A) = (AI + ... + An)/n, Q(A) = (AI ... An)l/n and 1t(A) = n(A1I + ... +A~l )-1. This is a generalization of the classical inequality connecting arithmetic, geometric and harmonic means. (Maher (1994).)

500


P 16.3.12 (Gersgorill Theorem) Let A E Mn and aij E C be the elements of A, i, j = 1, ... ,n and

Then every eigenvalue of A lies in at least one of the disks {z: Iz - aiil :S ~d, i

= 1, ...

,n

in the complex z-plane. FUrthermore, a set of m disks having no point in common with the remaining (n - m) disks, contains m and only m eigenvalues of A. P 16.3.13

Let A E Mn as in P 16.3.12 and

Then the eigenvalues of A lie in the disk

{z E C : Izl :S min(v, ()}. FUrthermore I det AI :S min(1}n, (n). P 16.3.14 Let A E Mn as in P 16.3.12, dj = lajjl - ~j, j 1, ... ,n and d = min{d}, ... ,dn } > O. Then IAil ~ d, i = 1, ... where AI, ... ,An are the eigenvalues of A. Hence I det AI ~ ~.

= ,n,

P 16.3.15 (Schur Theorem) Let A E M n , II· II denote the Euclidean matrix norm and AI,'" ,An, the eigenvalues of A. Then

+ ... + IAnl2 :S (ReA I)2 + ... + (ReA n )2 :S ( ImA d2 + ... + (ImAn)2 :S IAll2

IIAII2 IIBII2 IICll2

where Re and 1m are real and imaginary parts, B = (A + A*)j2 and C = (A-A*)j2. Equality in anyone of the above three relations implies equality in all three and occurs if and only if A is normal.

501


16.4. Toeplitz Matrices Matrices whose entries are constant along each diagonal arise in many applications and are called Toeplitz matrices. Formally T E Mn is Toeplitz if there exist scalars c-p+l, ... ,Co, ... ,Cp-l such that tij, the {i,j}-th element of Tis Cj-i. Thus

co

T=

[ C-I

C-2

Cl

C2

CO

Cl

C-l

CO

CP-'I Cp -2 Cp-3

C-~+l C-~+2 C-~+3

:::

{16.4.1}

CO

is Toeplitz. The special case of (16.4.1) when C-i = Ci and the matrix is positive definite arises in a linear prediction problem using an autoregressive model in time series, (16.4.2) where, considering {16.4.2} as a stationary process, we have

E{td = 0,

E{Xt}

E{XtXt_j}

= 0,

=

°

= E{XtXt+j} = Cj.

Multiplying both sides of (16.4.2) by have Cj

E{Xttt}

Xt-j

and taking expectations, we

+ alcli-ll + ... + apclj_pl = 0, j = 1, ...

,p

which can be written using a special case of (16.4.1),

(16.4.3)

If Ci are known, we can estimate ai by solving equation (16.4.:3) and use the estimates formula

al, ... ,aj

in predicting

Xt

given

Xt-l, .. . , X t-p

by the {16.4.4}

502


If ..)] d>...

/I

The appropriate interval for F is [min,X f(>"), max,X f(>..)]. In this case, one can show that the eigenvalues of Tn lie in this appropriate interval. For further exposition in this connection, the reader can refer to Gray (1972) and Grenander and Szego (1958).

Complements 16.4.1 Let V be the collection of all n x n Toeplitz matrices. Show that V is a vector space and its dimension is 2n - 1. Obtain a basis of

V. 16.4.2

Determine the inverse of the following Toeplitz matrix of order

nx n.

T=

1 0 0

-1 1 0

0 -1 1

0 0 0

0 0 0

0 0

0 0

0 0

1 0

-1 1

For this matrix, ao = 1, al = -1, and all other ai = O.

506


16.4.3 nx n.

Determine the inverse of the following Toeplitz matrix of order

T=

1 -2 1

0 0

-2

0 0 1

0 0 0

0 0 0

0 0 0

0 0

0 0

-2 1

1 -2

0

0 1

1

In this example, ao = 1, a_I = -2, a_2 = 1, and all other ai = O. Hint: T- I is lower triangular and Toeplitz. 16.4.4 Determine the inverse of the following Toeplitz matrix T of order n x n.

T=

2

0 0 0

0 0 0

0 0 0

0 0

-1 0

2 -1

-1 2

0

2 -1 0

-1 2 -1

-1

0 0

0 0

In this example, ao = 2, al = a_I = -1, and all other ai matrix T is an example of a tri-diagonal matrix. Hint: Study the pattern of inverses for n = 1,2,3, and 4.

= O.

The

16.5. Restricted Eigenvalue Problem

In statistical applications, it is sometimes necessary to find the optimum values of a quadratic form x' Ax subject to the conditions x' Bx = 1, where B is pd, and G'x = t (see Rao (1964b)) . A simple solution exists when t = O. P 16.5.1 The stationary values of x' Ax when x is restricted to x' Bx = 1 and G'x = 0 is attained at the eigenvectors of (I - P)A with respect to B , where P is the projection operator

PROOF.

Introducing Lagrangian multipliers>. and jL, we consider the

expression

x' Ax - >.(x' Bx - 1) - 2x'GjL


507

and equate its derivative to zero Ax - )"Bx - CJ.L = 0 C'x=O x'Bx

= 1.

(16.5.1)

Multiplying the first equation in (16.5.1) by I -P, we have the equation

(I - P}Ax = )"Bx

(16.5.2)

which is required to be proved. In the special case A = aa' where a is a vector, x' Ax has the maximum value when x ex B- 1 (I - P}a which is an impOi"tant result in problems of genetic selection. Another problem of interest in this connection is to find the maximum of x' Ax when x is restricted by the inequality condition C'x ~e 0 in addition to x' Bx = 1. This leads to a quadratic programming problem as shown in Rao (1964a). In the general case of the condition C' x = t, a solution is given by Gander, Golub and Matt (1989). A more general eigenvalue problem which occurs in statistical problems is to find the stationary values of

Matrix algebra and its applications to statistics and econometrics

Statistics and Econometrics: Methods and Applications

Introduction to Statistics and Econometrics

Linear algebra and its applications

Linear algebra and its applications

Linear Algebra and Its Applications

Matrix Algebra: Theory, Computations, and Applications in Statistics

Linear Algebra and Its Applications

Linear algebra and its applications

Linear Algebra and Its Applications

Matrix Algebra: Theory, computations, and applications in statistics

Matrix Algebra: Theory, Computations, and Applications in Statistics

Linear algebra and its applications

Linear Algebra and Its Applications

Boolean Algebra and Its Applications

Boolean Algebra and Its Applications

Linear Algebra and its Applications

Statistics and econometrics

Schaum's Statistics and Econometrics

A Combinatorial Approach to Matrix Theory and Its Applications (Discrete Mathematics and Its Applications)

Introduction to Matrix Algebra

A combinatorial approach to matrix theory and its applications

Linear Algebra and Its Applications (4ed)

Linear Algebra and Its Applications, Fourth Edition

Linear Algebra and Its Applications (4th Edition)

Contemporary Bayesian Econometrics and Statistics

Schaum's Outlines - Statistics And Econometrics

Matrix Algebra: Exercises and Solutions

Introduction to Mathematical Statistics and Its Applications, An (4th Edition)

Matrix algebra and its applications to statistics and econometrics

Statistics and Econometrics: Methods and Applications

Introduction to Statistics and Econometrics

Linear algebra and its applications

Linear algebra and its applications

Linear Algebra and Its Applications

Matrix Algebra: Theory, Computations, and Applications in Statistics

Linear Algebra and Its Applications

Linear algebra and its applications

Linear Algebra and Its Applications

Matrix Algebra: Theory, computations, and applications in statistics

Matrix Algebra: Theory, Computations, and Applications in Statistics

Linear algebra and its applications

Linear Algebra and Its Applications

Boolean Algebra and Its Applications

Boolean Algebra and Its Applications

Linear Algebra and its Applications

Statistics and econometrics

Schaum's Statistics and Econometrics

A Combinatorial Approach to Matrix Theory and Its Applications (Discrete Mathematics and Its Applications)

Introduction to Matrix Algebra

A combinatorial approach to matrix theory and its applications

Linear Algebra and Its Applications (4ed)

Linear Algebra and Its Applications, Fourth Edition

Linear Algebra and Its Applications (4th Edition)

Contemporary Bayesian Econometrics and Statistics

Schaum's Outlines - Statistics And Econometrics

Matrix Algebra: Exercises and Solutions

Introduction to Mathematical Statistics and Its Applications, An (4th Edition)

Recommend Documents