Matrix Algebra From a Statistician's Perspective

Matrix Algebra From a Statistician’s Perspective David A. Harville Matrix Algebra From a Statistician’s Perspective ...

Author: David A. Harville

504 downloads 3615 Views 4MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Matrix Algebra From a Statistician’s Perspective

David A. Harville

Matrix Algebra From a Statistician’s Perspective

13

David A. Harville IBM T. J. Watson Research Center Mathematical Sciences Department Yorktown Heights, NY 10598-0218 USA [email protected]

ISBN 978-0-387-78356-7

e-ISBN 978-0-387-22677-4

Library of Congress Control Number: 2008927514 c 2008 Springer Science+Business Media, LLC All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. Printed on acid-free paper. 9 8 7 6 5 4 3 2 1 springer.com

Preface

Matrix algebra plays a very important role in statistics and in many other disciplines. In many areas of statistics, it has become routine to use matrix algebra in the presentation and the derivation or verification of results. One such area is linear statistical models; another is multivariate analysis. In these areas, a knowledge of matrix algebra is needed in applying important concepts, as well as in studying the underlying theory, and is even needed to use various software packages (if they are to be used with confidence and competence). On many occasions, I have taught graduate-level courses in linear statistical models. Typically, the prerequisites for such courses include an introductory (undergraduate) course in matrix (or linear) algebra. Also typically, the preparation provided by this prerequisite course is not fully adequate. There are several reasons for this. The level of abstraction or generality in the matrix (or linear) algebra course may have been so high that it did not lead to a “working knowledge” of the subject, or, at the other extreme, the course may have emphasized computations at the expense of fundamental concepts. Further, the content of introductory courses on matrix (or linear) algebra varies widely from institution to institution and from instructor to instructor. Topics such as quadratic forms, partitioned matrices, and generalized inverses that play an important role in the study of linear statistical models may be covered inadequately if at all. An additional difficulty is that several years may have elapsed between the completion of the prerequisite course on matrix (or linear) algebra and the beginning of the course on linear statistical models. This book is about matrix algebra. A distinguishing feature is that the content, the ordering of topics, and the level of generality are ones that I consider appropriate for someone with an interest in linear statistical models and perhaps also

The content of the paperback version is essentially the same as that of the earlier, hardcover version. The paperback version differs from the earlier version in that a number of (mostly minor) corrections and alterations have been incorporated. In addition, the typography has been improved—as a side effect, the content and the numbering of the individual pages differ somewhat from those in the earlier version.

vi

Preface

for someone with an interest in another area of statistics or in a related discipline. I have tried to keep the presentation at a level that is suitable for anyone who has had an introductory course in matrix (or linear) algebra. In fact, the book is essentially self-contained, and it is hoped that much, if not all, of the material may be comprehensible to a determined reader with relatively little previous exposure to matrix algebra. To make the material readable for as broad an audience as possible, I have avoided the use of abbreviations and acronyms and have sometimes adopted terminology and notation that may seem more meaningful and familiar to the non-mathematician than those favored by mathematicians. Proofs are provided for essentially all of the results in the book. The book includes a number of results and proofs that are not readily available from standard sources and many others that can be found only in relatively high-level books or in journal articles. The book can be used as a companion to the textbook in a course on linear statistical models or on a related topic—it can be used to supplement whatever results on matrices may be included in the textbook and as a source of proofs. And, it can be used as a primary or supplementary text in a second course on matrices, including a course designed to enhance the preparation of the students for a course or courses on linear statistical models and/or related topics. Above all, it can serve as a convenient reference book for statisticians and for various other professionals. While the motivation for the writing of the book came from the statistical applications of matrix algebra, the book itself does not include any appreciable discussion of statistical applications. It is assumed that the book is being read because the reader is aware of the applications (or at least of the potential for applications) or because the material is of intrinsic interest—this assumption is consistent with the uses discussed in the previous paragraph. (In any case, I have found that the discussions of applications that are sometimes interjected into treatises on matrix algebra tend to be meaningful only to those who are already knowledgeable about the applications and can be more of a distraction than a help.) The book has a number of features that combine to set it apart from the more traditional books on matrix algebra—it also differs in significant respects from those matrix-algebra books that share its (statistical) orientation, such as the books of Searle (1982), Graybill (1983), and Basilevsky (1983). The coverage is restricted to real matrices (i.e., matrices whose elements are real numbers)—complex matrices (i.e., matrices whose elements are complex numbers) are typically not encountered in statistical applications, and their exclusion leads to simplifications in terminology, notation, and results. The coverage includes linear spaces, but only linear spaces whose members are (real) matrices—the inclusion of linear spaces facilitates a deeper understanding of various matrix concepts (e.g., rank) that are very relevant in applications to linear statistical models, while the restriction to linear spaces whose members are matrices makes the presentation more appropriate for the intended audience. The book features an extensive discussion of generalized inverses and makes heavy use of generalized inverses in the discussion of such standard topics as the solution of linear systems and the rank of a matrix. The discussion of eigenvalues

Preface

vii

and eigenvectors is deferred until the next-to-last chapter of the book—I have found it unnecessary to use results on eigenvalues and eigenvectors in teaching a first course on linear statistical models and, in any case, find it aesthetically displeasing to use results on eigenvalues and eigenvectors to prove more elementary matrix results. And the discussion of linear transformations is deferred until the very last chapter—in more advanced presentations, matrices are regarded as subservient to linear transformations. The book provides rather extensive coverage of some nonstandard topics that have important applications in statistics and in many other disciplines. These include matrix differentiation (Chapter 15), the vec and vech operators (Chapter 16), the minimization of a second-degree polynomial (in n variables) subject to linear constraints (Chapter 19), and the ranks, determinants, and ordinary and generalized inverses of partitioned matrices and of sums of matrices (Chapter 18 and parts of Chapters 8, 9, 13 16, 17, and 19). An attempt has been made to write the book in such a way that the presentation is coherent and non-redundant but, at the same time, is conducive to using the various parts of the book selectively. With the obvious exception of certain of their parts, Chapters 12 through 22 (which comprise approximately three-quarters of the book’s pages) can be read in arbitrary order. The ordering of Chapters 1 through 11 (both relative to each other and relative to Chapters 12 through 22) is much more critical. Nevertheless, even Chapters 1 through 11 include sections or subsections that are prerequisites for only a small part of the subsequent material. More often than not, the less essential sections or subsections are deferred until the end of the chapter or section. The book does not address the computational aspects of matrix algebra in any systematic way, however it does include descriptions and discussion of certain computational strategies and covers a number of results that can be useful in dealing with computational issues. Matrix norms are discussed, but only to a limited extent. In particular, the coverage of matrix norms is restricted to those norms that are defined in terms of inner products. In writing the book, I was influenced to a considerable extent by Halmos’s (1958) book on finite-dimensional vector spaces, by Marsaglia and Styan’s (1974) paper on ranks, by Henderson and Searle’s (1979, 1981b) papers on the vec and vech operators, by Magnus and Neudecker’s (1988) book on matrix differential calculus, and by Rao and Mitra’s (1971) book on generalized inverses. And I benefited from conversations with Oscar Kempthorne and from reading some notes (on linear systems, determinants, matrices, and quadratic forms) that he had prepared for a course (on linear statistical models) at Iowa State University. I also benefited from reading the first two chapters (pertaining to linear algebra) of notes prepared by Justus F. Seely for a course (on linear statistical models) at Oregon State University. The book contains many numbered exercises. The exercises are located at (or near) the ends of the chapters and are grouped by section—some exercises may require the use of results covered in previous sections, chapters, or exercises. Many of the exercises consist of verifying results supplementary to those included in the body of the chapter. By breaking some of the more difficult exercises into parts

viii

Preface

and/or by providing hints, I have attempted to make all of the exercises appropriate for the intended audience. I have prepared solutions to all of the exercises, and it is my intention to make them available on at least a limited basis. The origin and historical development of many of the results covered in the book are difficult (if not impossible) to discern, and I have not made any systematic attempt to do so. However, each of Chapters 15 through 21 ends with a short section entitled Bibliographic and Supplementary Notes. Sources that I have drawn on more-or-less directly for an extensive amount of material are identified in that section. Sources that trace the historical development of various ideas, results, and terminology are also identified. And, for certain of the sections in a chapter, some indication may be given of whether that section is a prerequisite for various other sections (or vice versa). The book is divided into (22) numbered chapters, the chapters into numbered sections, and (in some cases) the sections into lettered subsections. Sections are identified by two numbers (chapter and section within chapter) separated by a decimal point—thus, the third section of Chapter 9 is referred to as Section 9.3. Within a section, a subsection is referred to by letter alone. A subsection in a different chapter or in a different section of the same chapter is referred to by referring to the section and by appending a letter to the section number—for example, in Section 9.3, Subsection b of Section 9.1 is referred to as Section 9.1b. An exercise in a different chapter is referred to by the number obtained by inserting the chapter number (and a decimal point) in front of the exercise number. Certain of the displayed “equations” are numbered. An equation number comprises two parts (corresponding to section within chapter and equation within section) separated by a decimal point (and is enclosed in parentheses). An equation in a different chapter is referred to by the “number” obtained by starting with the chapter number and appending a decimal point and the equation number—for example, in Chapter 6, result (2.5) of Chapter 5 is referred to as result (5.2.5). For purposes of numbering (and referring to) equations in the exercises, the exercises in each chapter are to be regarded as forming Section E of that chapter. Preliminary work on the book dates back to the 1982–1983 academic year, which I spent as a visiting professor in the Department of Mathematics at the University of Texas at Austin (on a faculty improvement leave from my then position as a professor of statistics at Iowa State University). The actual writing began after my return to Iowa State and continued on a sporadic basis (as time permitted) until my departure in December 1995. The work was completed during the first part of my tenure in the Mathematical Sciences Department of the IBM Thomas J. Watson Research Center. I am indebted to Betty Flehinger, Emmanuel Yashchin, Claude Greengard, and Bill Pulleyblank (all of whom are or were managers at the Research Center) for the time and support they provided for this activity. The most valuable of that support (by far) came in the form of the secretarial help of Peggy Cargiulo, who entered

The solutions were published by Springer-Verlag in 2001 in a volume entitled Matrix Algebra: Exercises and Solutions (ISBN 0-387-95318-3).

Preface

ix

the last six chapters of the book in LATEX and was of immense help in getting the manuscript into final form. I am also indebted to Darlene Wicks (of Iowa State University), who entered Chapters 1 through 16 in LATEX. I wish to thank John Kimmel, who has been my editor at Springer-Verlag. He has been everything an author could hope for. I also wish to thank Paul Nikolai (formerly of the Air Force Flight Dynamics Laboratory of Wright-Patterson Air Force Base, Ohio) and Dale Zimmerman (of the Department of Statistics and Actuarial Science of the University of Iowa), whose careful reading and marking of the manuscript led to a number of corrections and improvements. These changes were in addition to ones stimulated by the earlier comments of two anonymous reviewers (and by the comments of the editor).

Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

v

1 Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Basic Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Basic Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Some Basic Types of Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 1 2 5 10

2 Submatrices and Partitioned Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Some Terminology and Basic Results . . . . . . . . . . . . . . . . . . . . . . . 2.2 Scalar Multiples, Transposes, Sums, and Products of Partitioned Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Some Results on the Product of a Matrix and a Column Vector . . 2.4 Expansion of a Matrix in Terms of Its Rows, Columns, or Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

13 13 17 19

3 Linear Dependence and Independence . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Some Basic Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

23 23 23 25

4 Linear Spaces: Row and Column Spaces . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Some Definitions, Notation, and Basic Relationships and Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Bases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Rank of a Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Some Basic Results on Partitioned Matrices and on Sums of Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

27

20 21

27 29 31 36 41 46

xii

Contents

5 Trace of a (Square) Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Definition and Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Trace of a Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Some Equivalent Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

49 49 50 52 52

6 Geometrical Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Definitions: Norm, Distance, Angle, Inner Product, and Orthogonality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Orthogonal and Orthonormal Sets . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Schwarz Inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Orthonormal Bases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

55 55 61 62 63 68

7 Linear Systems: Consistency and Compatibility . . . . . . . . . . . . . . . . . 7.1 Some Basic Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 Linear Systems of the Form A0 AX D A0 B . . . . . . . . . . . . . . . . . . . Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

71 71 72 73 74 77

8 Inverse Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 8.1 Some Definitions and Basic Results . . . . . . . . . . . . . . . . . . . . . . . . 79 8.2 Properties of Inverse Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 8.3 Premultiplication or Postmultiplication by a Matrix of Full Column or Row Rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 8.4 Orthogonal Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 8.5 Some Basic Results on the Ranks and Inverses of Partitioned Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 9 Generalized Inverses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1 Definition, Existence, and a Connection to the Solution of Linear Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Some Alternative Characterizations . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 Some Elementary Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Invariance to the Choice of a Generalized Inverse . . . . . . . . . . . . . 9.5 A Necessary and Sufficient Condition for the Consistency of a Linear System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.6 Some Results on the Ranks and Generalized Inverses of Partitioned Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7 Extension of Some Results on Systems of the Form AX D B to Systems of the Form AXC D B . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

107 107 109 117 119 120 121 126 126

Contents

xiii

10 Idempotent Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.1 Definition and Some Basic Properties . . . . . . . . . . . . . . . . . . . . . . . 10.2 Some Basic Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

133 133 134 136

11 Linear Systems: Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Some Terminology, Notation, and Basic Results . . . . . . . . . . . . . . 11.2 General Form of a Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Number of Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4 A Basic Result on Null Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5 An Alternative Expression for the General Form of a Solution . . 11.6 Equivalent Linear Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.7 Null and Column Spaces of Idempotent Matrices . . . . . . . . . . . . . 11.8 Linear Systems With Nonsingular Triangular or Block-Triangular Coefficient Matrices . . . . . . . . . . . . . . . . . . . . . . 11.9 A Computational Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.10 Linear Combinations of the Unknowns . . . . . . . . . . . . . . . . . . . . . . 11.11 Absorption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.12 Extensions to Systems of the Form AXC D B . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

139 139 140 142 144 144 145 146

12 Projections and Projection Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1 Some General Results, Terminology, and Notation . . . . . . . . . . . . 12.2 Projection of a Column Vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3 Projection Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.4 Least Squares Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5 Orthogonal Complements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

161 161 163 166 170 172 177

13 Determinants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.1 Some Definitions, Notation, and Special Cases . . . . . . . . . . . . . . . 13.2 Some Basic Properties of Determinants . . . . . . . . . . . . . . . . . . . . . 13.3 Partitioned Matrices, Products of Matrices, and Inverse Matrices 13.4 A Computational Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5 Cofactors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.6 Vandermonde Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.7 Some Results on the Determinant of the Sum of Two Matrices . . 13.8 Laplace’s Theorem and the Binet-Cauchy Formula . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

179 179 183 187 191 191 195 197 200 205

14 Linear, Bilinear, and Quadratic Forms . . . . . . . . . . . . . . . . . . . . . . . . . . 14.1 Some Terminology and Basic Results . . . . . . . . . . . . . . . . . . . . . . . 14.2 Nonnegative Definite Quadratic Forms and Matrices . . . . . . . . . . 14.3 Decomposition of Symmetric and Symmetric Nonnegative Definite Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

209 209 212

146 149 150 152 157 158

218

xiv

Contents

14.4 Generalized Inverses of Symmetric Nonnegative Definite Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.5 LDU, U0 DU, and Cholesky Decompositions . . . . . . . . . . . . . . . . . 14.6 Skew-Symmetric Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.7 Trace of a Nonnegative Definite Matrix . . . . . . . . . . . . . . . . . . . . . 14.8 Partitioned Nonnegative Definite Matrices . . . . . . . . . . . . . . . . . . . 14.9 Some Results on Determinants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.10 Geometrical Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.11 Some Results on Ranks and Row and Column Spaces and on Linear Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.12 Projections, Projection Matrices, and Orthogonal Complements . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Matrix Differentiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.1 Definitions, Notation, and Other Preliminaries . . . . . . . . . . . . . . . . 15.2 Differentiation of (Scalar-Valued) Functions: Some Elementary Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.3 Differentiation of Linear and Quadratic Forms . . . . . . . . . . . . . . . 15.4 Differentiation of Matrix Sums, Products, and Transposes (and of Matrices of Constants) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.5 Differentiation of a Vector or (Unrestricted or Symmetric) Matrix With Respect to Its Elements . . . . . . . . . . . . . . . . . . . . . . . . 15.6 Differentiation of a Trace of a Matrix . . . . . . . . . . . . . . . . . . . . . . . 15.7 The Chain Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.8 First-Order Partial Derivatives of Determinants and Inverse and Adjoint Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.9 Second-Order Partial Derivatives of Determinants and Inverse Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.10 Differentiation of Generalized Inverses . . . . . . . . . . . . . . . . . . . . . . 15.11 Differentiation of Projection Matrices . . . . . . . . . . . . . . . . . . . . . . . 15.12 Evaluation of Some Multiple Integrals . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bibliographic and Supplementary Notes . . . . . . . . . . . . . . . . . . . . . 16 Kronecker Products and the Vec and Vech Operators . . . . . . . . . . . . 16.1 The Kronecker Product of Two or More Matrices: Definition and Some Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16.2 The Vec Operator: Definition and Some Basic Properties . . . . . . . 16.3 Vec-Permutation Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16.4 The Vech Operator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16.5 Reformulation of a Linear System . . . . . . . . . . . . . . . . . . . . . . . . . . 16.6 Some Results on Jacobian Matrices . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bibliographic and Supplementary Notes . . . . . . . . . . . . . . . . . . . . .

222 223 239 240 243 247 255 259 260 277 289 290 296 298 300 303 304 306 308 312 314 319 324 327 335 337 337 343 347 354 367 368 371 377

Contents

17 Intersections and Sums of Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.1 Definitions and Some Basic Properties . . . . . . . . . . . . . . . . . . . . . . 17.2 Some Results on Row and Column Spaces and on the Ranks of Partitioned Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.3 Some Results on Linear Systems and on Generalized Inverses of Partitioned Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.4 Subspaces: Sum of Their Dimensions Versus Dimension of Their Sum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.5 Some Results on the Rank of a Product of Matrices . . . . . . . . . . . 17.6 Projections Along a Subspace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.7 Some Further Results on the Essential Disjointness and Orthogonality of Subspaces and on Projections and Projection Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bibliographic and Supplementary Notes . . . . . . . . . . . . . . . . . . . . . 18 Sums (and Differences) of Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.1 Some Results on Determinants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.2 Some Results on Inverses and Generalized Inverses and on Linear Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.3 Some Results on Positive (and Nonnegative) Definiteness . . . . . . 18.4 Some Results on Idempotency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.5 Some Results on Ranks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bibliographic and Supplementary Notes . . . . . . . . . . . . . . . . . . . . . 19 Minimization of a Second-Degree Polynomial (in n Variables) Subject to Linear Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.1 Unconstrained Minimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.2 Constrained Minimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.3 Explicit Expressions for Solutions to the Constrained Minimization Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.4 Some Results on Generalized Inverses of Partitioned Matrices . . 19.5 Some Additional Results on the Form of Solutions to the Constrained Minimization Problem . . . . . . . . . . . . . . . . . . . . . . . . . 19.6 Transformation of the Constrained Minimization Problem to an Unconstrained Minimization Problem . . . . . . . . . . . . . . . . . . . . . . . 19.7 The Effect of Constraints on the Generalized Least Squares Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bibliographic and Supplementary Notes . . . . . . . . . . . . . . . . . . . . .

xv

379 379 385 392 396 398 402

409 411 417 419 419 423 437 439 444 450 458 459 460 463 468 476 483 489 491 492 495

20 The Moore-Penrose Inverse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497 20.1 Definition, Existence, and Uniqueness (of the Moore-Penrose Inverse) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497 20.2 Some Special Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 499

xvi

Contents

20.3 20.4 20.5 20.6 20.7 20.8

Special Types of Generalized Inverses . . . . . . . . . . . . . . . . . . . . . . Some Alternative Representations and Characterizations . . . . . . . Some Basic Properties and Relationships . . . . . . . . . . . . . . . . . . . . Minimum Norm Solution to the Least Squares Problem . . . . . . . . Expression of the Moore-Penrose Inverse as a Limit . . . . . . . . . . . Differentiation of the Moore-Penrose Inverse . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bibliographic and Supplementary Notes . . . . . . . . . . . . . . . . . . . . .

500 507 508 512 512 514 517 519

21 Eigenvalues and Eigenvectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.1 Definitions, Terminology, and Some Basic Results . . . . . . . . . . . . 21.2 Eigenvalues of Triangular or Block-Triangular Matrices and of Diagonal or Block-Diagonal Matrices . . . . . . . . . . . . . . . . . . . . . . . 21.3 Similar Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.4 Linear Independence of Eigenvectors . . . . . . . . . . . . . . . . . . . . . . . 21.5 Diagonalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.6 Expressions for the Trace and Determinant of a Matrix . . . . . . . . 21.7 Some Results on the Moore-Penrose Inverse of a Symmetric Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.8 Eigenvalues of Orthogonal, Idempotent, and Nonnegative Definite Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.9 Square Root of a Symmetric Nonnegative Definite Matrix . . . . . . 21.10 Some Relationships . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.11 Eigenvalues and Eigenvectors of Kronecker Products . . . . . . . . . . 21.12 Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.13 Simultaneous Diagonalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.14 Generalized Eigenvalue Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.15 Differentiation of Eigenvalues and Eigenvectors . . . . . . . . . . . . . . 21.16 An Equivalence (Involving Determinants and Polynomials) . . . . . Appendix: Some Properties of Polynomials . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bibliographic and Supplementary Notes . . . . . . . . . . . . . . . . . . . . .

521 522

22 Linear Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.1 Some Definitions, Terminology, and Basic Results . . . . . . . . . . . . 22.2 Scalar Multiples, Sums, and Products of Linear Transformations 22.3 Inverse Transformations and Isomorphic Linear Spaces . . . . . . . . 22.4 Matrix Representation of a Linear Transformation . . . . . . . . . . . . 22.5 Terminology and Properties Shared by a Linear Transformation and Its Matrix Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.6 Linear Functionals and Dual Transformations . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

528 530 534 537 545 546 548 550 551 554 556 566 569 571 574 580 582 588 589 589 595 598 601 609 612 616

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 621 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625

1 Matrices

Matrix algebra is a branch of mathematics in which numbers are dealt with collectively (as rectangular arrays of numbers called matrices) rather than individually, as in “ordinary” algebra. The term matrix is formally defined in Section 1.1. Section 1.1 also includes an introduction to various basic terminology used in referring to matrices. And some basic matrix operations (scalar multiplication, matrix addition and subtraction, matrix multiplication, and transposition) are defined and their properties discussed in Section 1.2—not all of the properties of the multiplication of ordinary numbers extend to matrix multiplication. There are many different types of matrices that are sometimes singled out in the literature for special attention. Some of the most basic of these are introduced in Section 1.3. Various other types of matrices are introduced later in the book (as the need arises). In addition, there are many types of matrices that have received considerable attention in the literature but were regarded as too specialized to be considered here.

1.1 Basic Terminology A rectangular array of real numbers is called a matrix. That is, a matrix is a collection of real numbers, a11 ; a12 , : : : ; a1n , : : : ; am1 ; am2 , : : : ; amn , arranged in the general form 1 0 a11 a12 : : : a1n B a21 a22 : : : a2n C C B B :: :: :: C : @ : : : A am1 am2 : : : amn For example,

is a matrix.

2 3 1 4

0 2

2

1 Matrices

Our definition of a matrix is less general than that sometimes adopted. The more general definition applies to any rectangular arrangement of elements, called scalars, of a field. (A field is a set, whose elements can be added or multiplied in accordance with rules that have certain properties.) For example, the more general definition covers complex matrices, which are rectangular arrays of complex numbers. Here, unless otherwise indicated, the use of the word matrix is confined to real matrices, that is, to rectangular arrays of real numbers. Also, scalar is used synonymously with real number. A matrix having m rows and n columns is referred to as an mn matrix, and m and n are called the dimensions of the matrix. The scalar located at the intersection of the i th row and the j th column of a matrix is called the ij th element or entry of the matrix. Boldface capital letters (e.g., A) are used to represent matrices. The notation A D faij g is used in introducing a matrix, the ij th element of which is aij . Two matrices A and B of the same dimensions are said to be equal if each element of A equals the corresponding element of B, in which case we write A D B (and are said to be unequal otherwise, i.e., if some element of A differs from the corresponding element of B, in which case we write A ¤ B).

1.2 Basic Operations a. Scalar multiplication Scalar multiplication is defined for an arbitrary scalar k and an arbitrary m n matrix A D faij g. The product of k and A is written as kA (or, less commonly, as Ak) and is defined to be the mn matrix whose ij th element is kaij . For example, 2 3 0 8 12 0 4 D : 1 4 2 4 16 8 The matrix kA is said to be a scalar multiple of A. For any matrix A, 1A D A: And, for any scalars c and k and any matrix A, c.kA/ D .ck/A D .kc/A D k.cA/:

(2.1)

It is customary to refer to the scalar product .1/A of 1 and A as the negative of A and to abbreviate .1/A to A. b. Matrix addition and subtraction Matrix addition and subtraction are defined for any two matrices A D faij g and B D fbij g that have the same number of rows, say m, and the same number of

1.2 Basic Operations

3

columns, say n. The sum of the two m n matrices A and B is denoted by the symbol ACB and is defined to be the mn matrix whose ij th element is aij Cbij . For example, 2 3 0 6 1 3 8 2 3 C D : 1 4 2 1 0 4 0 4 6 Matrix addition is commutative, that is, A C B D B C A:

(2.2)

Matrix addition is also associative, that is, taking C to be a third m n matrix, A C .B C C/ D .A C B/ C C:

(2.3)

The symbol A C B C C is used to represent the common value of the left and right sides of equality (2.3), and this value is referred to as the sum of A, B, and C. This notation and terminology extend in an obvious way to any finite number of m n matrices. For any scalar c, we have that c.A C B/ D cA C cB;

(2.4)

and, for any scalars c and k, we have that .c C k/A D cA C kA:

(2.5)

Let us write A B for the sum A C .B/ or equivalently for the m n matrix whose ij th element is aij bij and refer to this matrix as the difference between A and B. Matrices having the same number of rows and the same number of columns are said to be conformal for addition (or subtraction). c. Matrix multiplication Let A D faij g represent an m n matrix, and B D fbij g a p q matrix. When n D p, that is, when A has the same number of columns as B has rows, the matrix product AB is defined to be the m q matrix whose ij th element is n X

aik bkj D ai1 b1j C ai2 b2j C C ai n bnj :

kD1

For example, 2 3 1 4

0 1 3 4 0 @ 2 0A 2 1 2 2.3/ C .3/.2/ C 0.1/ 2.4/ C .3/.0/ C 0.2/ D 1.3/ C 4.2/ C 2.1/ 1.4/ C 4.0/ C 2.2/ 12 8 D : 3 8

4

1 Matrices

The formation of the matrix product AB is referred to as the premultiplication of B by A or the postmultiplication of A by B. When n ¤ p, the matrix product AB is undefined. Matrix multiplication is associative. Thus, introducing a third matrix C, A.BC/ D .AB/C;

(2.6)

provided that p D n and that C has q rows so that the matrix multiplications required to form the left and right sides of the equality are defined. The symbol ABC is used to represent the common value of the left and right sides of equality (2.6), and this value is referred to as the product of A, B, and C. This notation and terminology extend in an obvious way to any finite number of matrices. Matrix multiplication is distributive with respect to addition, that is, A.B C C/ D AB C AC; .A C B/C D AC C BC;

(2.7) (2.8)

where, in each equality, it is assumed that the dimensions of A, B, and C are such that all multiplications and additions are defined. Results (2.7) and (2.8) extend in an obvious way to the postmultiplication or premultiplication of a matrix A or C by the sum of any finite number of matrices. In general, matrix multiplication is not commutative. That is, AB is not necessarily identical to BA. In fact, when n D p but m ¤ q or when m D q but n ¤ p, one of the matrix products AB and BA is defined, while the other is undefined. When n D p and m D q, AB and BA are both defined, but the dimensions .mm/ of AB are the same as those of BA only if m D n. Even when n D p D m D q, in which case A and B are both n n matrices and the two matrix products AB and BA are both defined and have the same dimensions, it is not necessarily true that AB D BA. For example, 1 0 0 1 0 1 0 2 0 1 1 0 D ¤ D : 0 2 1 0 2 0 1 0 1 0 0 2 Two nn matrices A and B are said to commute if AB D BA. More generally, a collection of n n matrices A1 ; A2 ; : : : ; Ak is said to commute in pairs if Ai Aj D Aj Ai for j > i D 1; 2; : : : ; k. For any scalar c, m n matrix A, and n p matrix B, it is customary to write cAB for the product c.AB/ of c and the matrix product AB. Note that cAB D .cA/B D A.cB/

(2.9)

(as is evident from the very definitions of scalar and matrix multiplication). More generally, for any scalar c and say k matrices A1 ; A2 ; : : : ; Ak (for which the product A1 A2 Ak is defined) c.A1 A2 Ak / is typically abbreviated to cA1 A2 Ak , and we have that cA1 A2 Ak D .cA1 /A2 Ak D A1 .cA2 /A3 Ak D A1 Ak1 .cAk /:

(2.10)

1.3 Some Basic Types of Matrices

5

d. Transposition The transpose of an m n matrix A, to be denoted by the symbol A0 , is the n m matrix whose ij th element is the j i th element of A. For example, 0 1 0 2 1 2 3 0 D @3 4A : 1 4 2 0 2 Observe that the transpose of a matrix can be formed by rewriting its rows as columns (or its columns as rows). For any matrix A, (2.11) .A0 /0 D A and, for any two matrices A and B (that are conformal for addition), .A C B/0 D A0 C B0 ;

(2.12)

as is easily verified. Further, for any two matrices A and B (for which the product AB is defined), .AB/0 D B0 A0 :

(2.13)

To see this, observe that the ij th element of .AB/0 equals the j i th element of AB, which in turn equals X X aj k bki D bki aj k : (2.14) k

k

Since bki is the i kth element of B0 and aj k the kj th element of A0 , the ij th element of B0 A0 , like that of .AB/0 , equals expression (2.14). By repeated application of result (2.13), we obtain the more general result that .A1 A2 Ak /0 D A0k A02 A01 ;

(2.15)

for any k matrices A1 ; A2 ; : : : ; Ak of appropriate dimensions.

1.3 Some Basic Types of Matrices a. Square matrices A matrix having the same number of rows as columns is called a square matrix. An n n square matrix is said to be of order n. Those n elements a11 ; a22 ; : : : ; ann of an n n square matrix 0 1 a11 a12 : : : a1n Ba21 a22 : : : a2n C B C ADB : :: : : : C @ :: : :: A : an1 an2 : : : ann

6

1 Matrices

that fall on an imaginary diagonal line extending from the upper left to the lower right corners of the matrix are known as the (first, second, etc.) diagonal elements. The diagonal line itself is sometimes referred to simply as the diagonal. Those elements of a square matrix other than the diagonal elements (i.e., those elements that lie above and to the right or below and to the left of the diagonal) are called the off-diagonal elements. Note that the product AA of a matrix A and itself is defined if and only if A is square. For a square matrix A, the symbol A2 is used to represent AA, A3 to represent AA2 D AAA, and, more generally, Ak to represent AAk1 D AAAk2 D D AAA A (k D 2; 3; : : :). b. Symmetric matrices A matrix A is said to be symmetric if A0 D A. Thus, a symmetric matrix is a square matrix whose ij th element equals its j i th element. For example, 1 0 3 1 4 @ 1 0 2A 4 2 5 is a symmetric matrix. c. Diagonal matrices A diagonal matrix is a a square matrix whose off-diagonal elements are all equal to 0, that is, a matrix of the general form 0 1 d1 0 : : : 0 B 0 d2 0C B C B :: C; :: @: A : 0

0

dn

where d1 ; d2 ; : : : ; dn are scalars. The notation D D fdi g is sometimes used to introduce a diagonal matrix, the i th diagonal element of which is di . Also, we sometimes write diag.d1 ; d2 ; : : : ; dn / for such a matrix. Note that, for any m n matrix A D faij g and any diagonal matrix D D fdi g (of order m), the ij th element of the matrix product DA equals di aij . Thus, the effect of premultiplying an m n matrix A by a diagonal matrix D is simply to multiply each element of the i th row of A by the i th diagonal element of D (i D 1; 2; : : : ; m). Similarly, the effect of postmultiplying A by a diagonal matrix D (of order n) is to multiply each element of the j th column of A by the j th diagonal element of D (j D 1; 2; : : : ; n). Note also that the effect of multiplying an m n matrix A by a scalar k is the same as that of premultiplying or postmultiplying A by the m m or n n matrix diag.k; k; : : : ; k/.


7

d. Identity matrices A diagonal matrix

1 1 0 ::: 0 B0 1 0C C B diag.1; 1; : : : ; 1/ D B : : :: C A @ :: 0 0 1 0

whose diagonal elements are all equal to 1 is called an identity matrix. The symbol In is used to represent an identity matrix of order n. In cases where the order is clear from the context, we may simply write I for an identity matrix. Clearly, for an arbitrary matrix A, IA D AI D A: (3.1) e. Matrices of ones The symbol Jmn is used to represent an m n matrix whose elements are all equal to 1. Or, when the dimensions are clear from the context or are to be left unspecified, we may simply write J for such a matrix. Thus, 1 0 1 1 ::: 1 B1 1 : : : 1C C B J D B: : :: C : @ :: :: :A 1 1 ::: 1 Also, Jn is sometimes written for Jnn . Note that Jmn Jnp D nJmp :

(3.2)

f. Null matrices A matrix whose elements are all equal to 0 is called a null matrix. A null matrix is denoted by the symbol 0. Thus, 1 0 0 0 ::: 0 B0 0 : : : 0C C B 0 D B: : :: C : @ :: :: :A 0 0 ::: 0 Letting A represent an arbitrary matrix and k an arbitrary scalar, we find that 0A D 0;

k0 D 0;

A C 0 D 0 C A D A; A A D 0; 0A D 0;

A0 D 0

(where the dimensions of each null matrix 0 can be ascertained from the context).

8

1 Matrices

g. Triangular matrices If all of the elements of a square matrix that are located below and to the left of the diagonal are 0, the matrix is called an upper triangular matrix. Thus, the general form of an upper triangular matrix is 1 0 a11 a12 a13 : : : a1n B 0 a22 a23 : : : a2n C C B B 0 0 a33 : : : a3n C C: B B :: :: : C :: @ : : :: A : 0

0

0

ann

Similarly, if all of the elements that are located above and to the right of the diagonal are 0, the matrix is called a lower triangular matrix. More formally, an n n matrix A D faij g is upper triangular if aij D 0 for j i D 1; : : : ; n. By a triangular matrix, we mean a (square) matrix that is upper triangular or lower triangular. Observe that a (square) matrix is a diagonal matrix if and only if it is both upper and lower triangular. The transpose of an upper triangular matrix is a lower triangular matrix—and vice versa—as is easily verified. Further, the sum of two upper triangular matrices (of the same order) is upper triangular and the sum of two lower triangular matrices is lower triangular, as is also easily verified. Some basic properties of the product of two upper or two lower triangular matrices are described in the following lemma. Lemma 1.3.1. Let A D faij g and B D fbij g represent nn matrices. If A and B are both upper triangular, then their product AB is upper triangular. Similarly, if A and B are both lower triangular, then AB is lower triangular. Further, if A and B are either both upper triangular or both lower triangular, then the i th diagonal element of AB is the product ai i bi i of the i th diagonal elements ai i and bi i of A and B (i D 1; : : : ; n). P Proof. By definition, the ij th element of AB is nkD1 aik bkj . Suppose that A and B are both upper triangular and hence that aik D 0, for k j . If j i , then clearly bkj D 0 for k > i . Thus, for j i , n X

aik bkj D

kD1

In particular,

i1 X

.0/bkj C ai i bij C

kD1

n X kDiC1

n X

aik bki D ai i bi i

kD1

and, for j < i ,

n X kD1

aik bkj D ai i .0/ D 0:

aik .0/ D ai i bij :


9

We conclude that the i th diagonal element of AB is ai i bi i (i D 1; : : : ; n) and that AB is upper triangular. Suppose now that A and B are both lower triangular, rather than upper triangular. Then, since B0 and A0 are both upper triangular with i th diagonal elements bi i and ai i , respectively, B0 A0 is upper triangular with i th diagonal element ai i bi i , implying that AB D .B0 A0 /0 is lower triangular with i th diagonal element Q.E.D. ai i bi i . An (upper or lower) triangular matrix is called a unit (upper or lower) triangular matrix if all of its diagonal elements equal one. In the special case of unit triangular matrices, Lemma 1.3.1 reduces to the following result. Corollary 1.3.2. The product of two unit upper triangular matrices (of the same order) is unit upper triangular, and the product of two unit lower triangular matrices is unit lower triangular. h. Row and column vectors A matrix that has only one column, that is, a matrix of the form 0 1 a1 B a2 C B C B :: C @ : A am is called a column vector. Similarly, a matrix that has only one row is called a row vector. A row or column vector having m elements may be referred to as an m-dimensional row or column vector. Clearly, the transpose of an m-dimensional column vector is an m-dimensional row vector, and vice versa. Lowercase boldface letters (e.g., a) are customarily used to represent column vectors. This notation is helpful in distinguishing column vectors from matrices that may have more than one column. No further notation is introduced for row vectors. Instead, row vectors are represented as the transposes of column vectors. For example, a0 represents the row vector whose transpose is the column vector a. The notation a D fai g or a0 D fai g is used in introducing a column or row vector whose i th element is ai . The symbol 1m is generally used in place of Jm1 , that is, 1m is used to represent an m-dimensional column vector whose elements are all equal to 1. If its dimension is clear from the context or is to be left unspecified, we may simply write 1 for such a vector. Note that J1n , or equivalently an n-dimensional row vector whose elements are all equal to 1, may be written as 10n or simply as 10 . Observe that (3.3) 1m 10n D Jmn : Null row or column vectors (i.e., null matrices having one row or one column) are sometimes referred to simply as null vectors, and (like null matrices in general) are represented by the symbol 0.

10

1 Matrices

i. 1 1 matrices Note that there is a distinction between a scalar, say k, and the 1 1 matrix .k/ whose sole element is k. For any column vector a, .k/a0 D ka0 :

a.k/ D ka;

However, in general, the product kA obtained by multiplying an m n matrix A by the scalar k is not the same as the product .k/A or A.k/ of the two matrices .k/ and A. In fact, .k/A or A.k/ is undefined unless m D 1 or n D 1, while kA is defined for all m and n. For convenience, we sometimes use the same symbol to represent both a 1 1 matrix and the element of that matrix. Which use is intended is discernible from the context.

Exercises Section 1.2 1. Show that, for any matrices A, B, and C (of the same dimensions), .A C B/ C C D .C C A/ C B: 2. Let A represent an mn matrix and B an np matrix, and let c and k represent arbitrary scalars. Using results (2.9) and (2.1) (or other means), show that .cA/.kB/ D .ck/AB: 3. (a) Verify result (2.6) (on the associativeness of matrix multiplication); that is, show that, for any m n matrix A D faij g, n q matrix B D fbj k g, and q r matrix C D fcks g, A.BC/ D .AB/C. (b) Verify result (2.7) (on the distributiveness with respect to addition of matrix multiplication); that is, show that, for any m n matrix A D faij g and n q matrices B D fbj k g and C D fcj k g, A.B C C/ D AB C AC. 4. Let A D faij g represent an m n matrix and B D fbij g a p m matrix. (a) Let x D fxi g represent an n-dimensional column vector. Show that the i th element of the p-dimensional column vector BAx is m X j D1

bij

n X kD1

aj k xk

(E.1)

Exercises

11

(b) Let X D fxij g represent an n q matrix. Generalize formula (E.1) by expressing the i rth element of the p q matrix BAX in terms of the elements of A, B, and X. (c) Let x D fxi g represent an n-dimensional column vector and C D fcij g a q p matrix. Generalize formula (E.1) by expressing the i th element of the q-dimensional column vector CBAx in terms of the elements of A, B, C, and x. (d) Let y D fyi g represent a p-dimensional column vector. Express the i th element of the n-dimensional row vector y 0 BA in terms of the elements of A, B, and y. Section 1.3 5. Let A and B represent n n matrices. Show that .A C B/.A B/ D A2 B2 if and only if A and B commute. 6. (a) Show that the product AB of two n n symmetric matrices A and B is itself symmetric if and only if A and B commute. (b) Give an example of two symmetric matrices (of the same order) whose product is not symmetric. 7. Verify (a) that the transpose of an upper triangular matrix is lower triangular and (b) that the sum of two upper triangular matrices (of the same order) is upper triangular. 8. Let A D faij g represent an n n upper triangular matrix, and suppose that the diagonal elements of A equal zero (i.e., that a11 D a22 D D ann D 0/. Further, let p represent an arbitrary positive integer. (a) Show that, for i D 1; : : : ; n and j D 1; : : : ; min.n; i C p 1/, the ij th element of Ap equals zero. (b) Show that, for i n p C 1, the i th row of Ap is null. (c) Show that, for p n, Ap D 0.

2 Submatrices and Partitioned Matrices

Two very important (and closely related) concepts are introduced in this chapter: that of a submatrix and that of a partitioned matrix. These concepts arise very naturally in statistics (especially in multivariate analysis and linear models) and in many other disciplines that involve probabilistic ideas. And results on submatrices and partitioned matrices, which can be found in Chapters 8, 9, 13, and 14 (and other of the subsequent chapters), have proved to be very useful. In particular, such results are almost indispensable in work involving the multivariate normal distribution—refer, for example, to Searle (1971, sec. 2.4f).

2.1 Some Terminology and Basic Results A submatrix of a matrix A is a matrix that can be obtained by striking out rows and/or columns of A. For example, if we strike out the second row of the matrix 0 1 2 4 3 6 @ 1 5 7 9A ; 1 0 2 2 we obtain the 2 4 submatrix

2 4 3 6 : 1 0 2 2

Alternatively, if we strike out the first submatrix 0 4 @5 0

and third columns, we obtain the 3 2 1 6 9A I 2

or, if we strike out the second row and the first and third columns, we obtain the 2 2 submatrix

14


4 6 : 0 2 Note that any matrix is a submatrix of itself; it is the submatrix obtained by striking out zero rows and zero columns. Submatrices of a row or column vector, that is, of a matrix having one row or column, are themselves row or column vectors and are customarily referred to as subvectors. Let A represent an r s submatrix of an m n matrix A obtained by striking out the i1 ; : : : ; imr th rows and j1 ; : : : ; jns th columns (of A), and let B represent the s r submatrix of A0 obtained by striking out the j1 ; : : : ; jns th rows and i1 ; : : : ; imr th columns (of A0 ). Then, B D A0 ;

(1.1)

as is easily verified. A submatrix of an n n matrix is called a principal submatrix if it can be obtained by striking out the same rows as columns (so that the i th row is struck out whenever the i th column is struck out, and vice versa). The r r (principal) submatrix of an n n matrix obtained by striking out its last n r rows and columns is referred to as a leading principal submatrix (r D 1; : : : ; n). A principal submatrix of a symmetric matrix is symmetric, a principal submatrix of a diagonal matrix is diagonal, and a principal submatrix of an upper or lower triangular matrix is respectively upper or lower triangular, as is easily verified. A matrix can be divided or partitioned into submatrices by drawing horizontal or vertical lines between various of its rows or columns, in which case the matrix is called a partitioned matrix and the submatrices are sometimes referred to as blocks (as in blocks of elements). For example, 1 0 1 0 1 0 2 4 3 6 2 4 3 6 2 4 3 6 @ 1 5 7 9A ; @ 1 5 7 9A @ 1 5 7 9A ; 1 0 2 2 1 0 2 2 1 0 2 2 are various partitionings of the same matrix. In effect, a partitioned m n matrix is an m n matrix A D faij g that has been reexpressed in the general form 1 0 A11 A12 : : : A1c BA21 A22 : : : A2c C C B B :: :: :: C : @ : : : A Ar1 Ar2 : : : Arc Here, Aij is an mi nj matrix (i D 1; : : : ; r; j D 1; : : : ; c), where m1 , : : : ; mr and n1 , : : : ; nc are positive integers such that m1 C Cmr D m and n1 C Cnc D n. Or, more explicitly,

2.1 Some Terminology and Basic Results

0

am1 CCmi 1 C1; n1 CCnj 1 C1 : : : B Aij D @ ::: am1 CCmi ; n1 CCnj 1 C1

15

1 am1 CCmi 1 C1; n1 CCnj C :: A: :

: : : am1 CCmi ; n1 CCnj

(When i D 1 or j D 1, interpret the degenerate sum m1 C Cmi1 or n1 C C nj 1 as zero.) Thus, a partitioned matrix can be regarded as an array or “matrix” of matrices. Note that a matrix that has been divided by “staggered” lines, for example, 1 0 2 4 3 6 @ 1 5 7 9A ; 1 0 2 2 does not satisfy our definition of a partitioned matrix. Thus, if a matrix, say 0 1 A11 A12 : : : A1c BA21 A22 : : : A2c C B C B :: :: :: C ; @ : : : A Ar1 Ar2 : : : Arc is introduced as a partitioned matrix, there is an implicit assumption that each of the submatrices Ai1 ; Ai2 ; : : : ; Aic in the i th “row” of submatrices contains the same number of rows (i D 1; 2; : : : ; r) and similarly that each of the submatrices A1j ; A2j ; : : : ; Arj in the j th “column” of submatrices contains the same number of columns. It is customary to identify each of the blocks in a partitioned matrix by referring to the row of blocks and the column of blocks in which it appears. Thus, the submatrix Aij is referred to as the ij th block of the partitioned matrix 0 1 A11 A12 : : : A1c BA21 A22 : : : A2c C B C B :: :: :: C : @ : : : A Ar1 Ar2 : : : Arc In the case of a partitioned m n matrix A of the form 0 1 A11 A12 : : : A1r BA21 A22 : : : A2r C B C ADB : :: :: C : : : @ : : : : A Ar1 Ar2 : : : Arr

(1.2)

(for which the number of rows of blocks equals the number of columns of blocks), the ij th block Aij of A is called a diagonal block if j D i and an off-diagonal block if j ¤ i . If all of the off-diagonal blocks of A are null matrices, that is, if

16


0

1 A11 0 : : : 0 B 0 A22 0 C B C ADB : C; : : : @ : A : 0 0 Arr then A is called a block-diagonal matrix, and sometimes diag.A11 ; A22 ; : : : ; Arr / is written for A. If Aij D 0 for j i D 1; : : : ; r, that is, if 0 1 A11 0 : : : 0 BA21 A22 0 C B C ADB : C; : : :: :: @ :: A Arr Ar1 Ar2 then A is called a lower block-triangular matrix. To indicate that A is either upper or lower block-triangular (without being more specific), A is referred to simply as block-triangular. Note that a partitioned m n matrix A of the form (1.2) is block-diagonal if and only if it is both upper block-triangular and lower block-triangular. Note also that, if m D n D r (in which case each block of A consists of a single element), saying that A is block diagonal or upper or lower block triangular is equivalent to saying that A is diagonal or upper or lower triangular. Partitioned matrices having one row or one column are customarily referred to as partitioned (row or column) vectors. Thus, a partitioned m-dimensional column vector is an m 1 vector a D fa t g that has been reexpressed in the general form 0 1 a1 Ba2 C B C B :: C : @:A ar Here, ai is an mi 1 vector with elements am1 CCmi 1 C1 ; : : : ; am1 CCmi 1 Cmi , respectively (i D 1; : : : ; r), where m1 ; : : : ; mr are positive integers such that m1 C C mr D m. Similarly, a partitioned m-dimensional row vector is a 1 m vector a0 D fa t g that has been reexpressed in the general form .a01 ; a02 ; : : : ; a0r /.

2.2 Scalar Multiples, Transposes, Sums, and Products of Partitioned Matrices

17

2.2 Scalar Multiples, Transposes, Sums, and Products of Partitioned Matrices Let

0 1 A11 A12 : : : A1c BA21 A22 : : : A2c C B C ADB : :: :: C @ :: : : A Ar1 Ar2 : : : Arc

represent a partitioned mn matrix whose ij th block Aij is of dimensions mi nj . Clearly, for any scalar k, 1 0 kA11 kA12 : : : kA1c BkA21 kA22 : : : kA2c C C B (2.1) kA D B : :: :: C : @ :: : : A kAr1 kAr2 : : : kArc In particular,

0 1 A11 A12 : : : A1c BA21 A22 : : : A2c C B C A D B : :: :: C : @ :: : : A Ar1 Ar2 : : : Arc

Further, it is a simple exercise to show that 0 0 1 A11 A021 : : : A0r1 BA0 A0 : : : A0 C 22 r2 C B 12 A0 D B : :: :: C I @ :: : : A 0 0 A1c A2c : : : A0rc

(2.2)

(2.3)

that is, A0 can be expressed as a partitioned matrix, comprising c rows and r columns of blocks, the ij th of which is the transpose Aj0 i of the j i th block Aj i of A. Now, let 0 1 B11 B12 : : : B1v BB21 B22 : : : B2v C B C BDB : :: :: C @ :: : : A Bu1 Bu2 : : : Buv represent a partitioned p q matrix whose ij th block Bij is of dimensions pi qj . The matrices A and B are conformal (for addition) provided that p D m and q D n. If u D r, v D c, pi D mi (i D 1; : : : ; r), and qj D nj (j D 1; : : : ; c), that is, if (besides A and B being conformal for addition) the rows and columns of B are partitioned in the same way as those of A, then

18


0

A11 C B11 A12 C B12 : : : A1c BA21 C B21 A22 C B22 : : : A2c B ACB DB :: :: @ : :

1 C B1c C B2c C C C; :: A :

(2.4)

Ar1 C Br1 Ar2 C Br2 : : : Arc C Brc and the partitioning of A and B is said to be conformal (for addition). This result and terminology extend in an obvious way to the addition of any finite number of partitioned matrices. If A and B are conformal (for addition), then 0 1 A11 B11 A12 B12 : : : A1c B1c BA21 B21 A22 B22 : : : A2c B2c C B C (2.5) AB DB C; :: :: :: A @ : : : Ar1 Br1 Ar2 Br2 : : : Arc Brc as is evident from results (2.4) and (2.2). The matrix product AB is defined provided that n D p. If c D u and nk D pk (k D 1; : : : ; c) [in which case all of the products Aik Bkj (i D 1; : : : ; r; j D 1; : : : ; v; k D 1; : : : ; c), as well as the product AB, exist], then 0 1 F11 F12 : : : F1v BF21 F22 : : : F2v C B C (2.6) AB D B : :: :: C ; @ :: : : A Fr1 Fr2 : : : Frv Pc

where Fij D kD1 Aik Bkj D Ai1 B1j C Ai2 B2j C C Aic Bcj , and the partitioning of A and B is said to be conformal (for the premultiplication of B by A). In the special case where r D c D u D v D 2, that is, where B11 B12 A11 A12 and B D ; AD A21 A22 B21 B22 result (2.6) simplifies to A11 B11 C A12 B21 A11 B12 C A12 B22 AB D : A21 B11 C A22 B21 A21 B12 C A22 B22

(2.7)

If A D .A1 ; A2 ; : : : ; Ac / is an m n matrix that has been partitioned only by columns (for emphasis, we sometimes insert commas between the submatrices of such a partitioned matrix), then 0 01 A1 BA0 C B 2C (2.8) A0 D B : C ; @ :: A A0c

2.3 Some Results on the Product of a Matrix and a Column Vector

19

0

1 B1 BB2 C B C BDB : C @ :: A

and further if

Bc is an n q partitioned matrix that has been partitioned only by rows (in a way that is conformal for its premultiplication by A), then AB D

c X

Ak Bk D A1 B1 C A2 B2 C C Ac Bc :

(2.9)

kD1

Similarly, if

0

1 A1 BA 2 C B C ADB : C @ :: A Ar

is an m n matrix that has been partitioned only by rows, then A0 D .A01 ; A02 ; : : : ; A0r /;

(2.10)

and further if B D .B1 ; B2 ; : : : ; Bv / is an n q matrix that has been partitioned only by columns, then 0 1 A1 B1 A1 B2 : : : A1 Bv BA 2 B 1 A 2 B 2 : : : A 2 B v C B C (2.11) AB D B : :: :: C : : @ : : : A Ar B1 Ar B2 : : : Ar Bv

2.3 Some Results on the Product of a Matrix and a Column Vector Let A represent an m n matrix and x an n 1 vector. Writing A as A D .a1 ; a2 ; : : : ; an /, where a1 ; a2 ; : : : ; an are the columns of A, and x as x D .x1 ; x2 ; : : : ; xn /0 , where x1 ; x2 ; : : : ; xn are the elements of x, we find, as a special case of result (2.9), that Ax D

n X

xk ak D x1 a1 C x2 a2 C C xn an :

(3.1)

kD1

Thus, the effect of postmultiplying a matrix by a column vector is to form a linear combination of the columns of the matrix, where the coefficients in the linear combination are the elements of the column vector. Similarly, the effect of

20


premultiplying a matrix by a row vector is to form a linear combination of the rows of the matrix, where the coefficients in the linear combination are the elements of the row vector. Representation (3.1) is helpful in establishing the elementary results expressed in the following two lemmas. Lemma 2.3.1. For any column vector y and nonnull column vector x, there exists a matrix A such that y D Ax. Proof. Since x is nonnull, one of its elements, say xj , is nonzero. Take A to be the matrix whose j th column is .1=xj /y and whose other columns are null. Then, y D Ax, as is evident from result (3.1). Q.E.D. Lemma 2.3.2. For any two m n matrices A and B, A D B if and only if Ax D Bx for every n 1 vector x. Proof. It is obvious that, if A D B, then Ax D Bx for every vector x. To prove the converse, suppose that Ax D Bx for every x. Taking x to be the n 1 vector whose i th element is 1 and whose other elements are 0, and letting ai and bi represent the i th columns of A and B, respectively, it is clear from result (3.1) that ai D Ax D Bx D bi (i D 1; : : : ; n). We conclude that A D B. Q.E.D. Note that Lemma 2.3.2 implies, in particular, that A D 0 if and only if Ax D 0 for every x.

2.4 Expansion of a Matrix in Terms of Its Rows, Columns, or Elements An m n matrix A D faij g can be expanded in terms of its rows, columns, or elements by making use of formula (2.9). Denote the i th row of A by ri0 and the i th column of Im by ei (i D 1; 2; : : : ; m). Then, writing Im as Im D .e1 ; e2 ; : : : ; em / and A as 0 01 r1 B r0 C B 2C ADB : C @ :: A 0 rm and applying formula (2.9) to the product Im A, we obtain the expansion AD

m X

0 ei ri0 D e1 r10 C e2 r20 C C em rm :

(4.1)

iD1

Similarly, denote the j th column of A by aj and the j th row of In by uj0 (j D 1; 2; : : : ; n). Then, writing A as A D .a1 ; a2 ; : : : ; an / and In as

Exercises

21

0

1 u01 Bu0 C B 2C In D B : C @ :: A u0n

and applying formula (2.9) to the product AIn , we obtain the alternative expansion AD

n X

aj uj0 D a1 u01 C a2 u02 C C an u0n :

(4.2)

j D1

Moreover, the application of formula (3.1) to the product Im aj gives the expansion m X aij ei : aj D iD1

Upon substituting this expansion into expansion (4.2), we obtain the further expansion m X n X AD aij Uij ; (4.3) iD1 j D1

ei uj0

is an m n matrix whose ij th element equals 1 and whose where Uij D remaining mn 1 elements equal 0. In the special case where n D m (i.e., where A is square), uj D ej and hence Uij D ei ej0 , and in the further special case where A D Im , expansion (4.3) reduces to Im D

m X

ei e0i :

(4.4)

iD1

Note that, as a consequence of result (4.3), we have that ! n n m X m X X X 0 0 0 ei Auj D ei aks ek us uj D aks e0i ek u0s uj ; kD1 sD1

kD1 sD1

which (since e0i ek equals 1, if k D i , and equals 0, if k ¤ i , and since u0s uj equals 1, if s D j , and equals 0, if s ¤ j ) simplifies to e0i Auj D aij :

Exercises Section 2.1

(4.5)

22


1. Verify result (1.1). 2. Verify (a) that a principal submatrix of a symmetric matrix is symmetric, (b) that a principal submatrix of a diagonal matrix is diagonal, and (c) that a principal submatrix of an upper triangular matrix is upper triangular. 3. Let

0

1 A11 A12 : : : A1r B 0 A22 : : : A2r C B C ADB : :: C :: @ :: : : A 0 0 : : : Arr

represent an n n upper block-triangular matrix whose ij th block Aij is of dimensions ni nj (j i D 1; : : : ; r). Show that A is upper triangular if and only if each of its diagonal blocks A11 ; A22 ; : : : ; Arr is upper triangular. Section 2.2 4. Verify results (2.3) and (2.6).

3 Linear Dependence and Independence

While short in length, the following material on linear dependence and independence is of fundamental importance—so much so that it forms a separate chapter.

3.1 Definitions Any finite set of row or column vectors, or more generally any finite set of matrices, is either linearly dependent or linearly independent. A nonempty finite set fA1 ; A2 ; : : : ; Ak g of m n matrices is said to be linearly dependent if there exist scalars x1 ; x2 ; : : : ; xk , not all zero, such that x1 A1 C x2 A2 C C xk Ak D 0: If no such scalars exist, the set is called linearly independent. The empty set is considered to be linearly independent. Note that if any subset of a finite set of matrices is linearly dependent, then the set itself is linearly dependent. While technically linear dependence and independence are properties of sets of matrices, it is customary to speak of “a set of linearly dependent (or independent) matrices” or simply of “linearly dependent (or independent) matrices” instead of “a linearly dependent (or independent) set of matrices.” In particular, in the case of row or column vectors, it is customary to speak of “linearly dependent (or independent) vectors.”

3.2 Some Basic Results Note that a set consisting of a single matrix is linearly dependent if the matrix is null, and is linearly independent if the matrix is nonnull. An elementary result on the linear dependence or independence of two or more matrices is expressed in the following lemma.

24


Lemma 3.2.1. A set fA1 ; A2 ; : : : ; Ak g of two or more m n matrices is linearly dependent if and only if at least one of the matrices is expressible as a linear combination of the others, that is, if and only if, for some integer j (1 j k), Aj is expressible as a linear combination of A1 ; : : : ; Aj 1 ; Aj C1 ; : : : ; Ak . Proof. Suppose that, for some j , Aj is expressible as a linear combination Aj D x1 A1 C C xj 1 Aj 1 C xj C1 Aj C1 C C xk Ak of the other k 1 m n matrices. Then, .x1 /A1 C C .xj 1 /Aj 1 C Aj C .xj C1 /Aj C1 C C .xk /Ak D 0; implying that fA1 ; A2 ; : : : ; Ak g is a linearly dependent set. Conversely, suppose that fA1 ; A2 ; : : : ; Ak g is linearly dependent, in which case x 1 A1 C x2 A2 C C x k Ak D 0 for some scalars x1 ; x2 ; : : : ; xk , not all zero. Let j represent any integer for which xj ¤ 0. Then, Aj D .x1 =xj /A1 C C .xj 1 =xj /Aj 1 C .xj C1 =xj /Aj C1 C C .xk =xj /Ak : Q.E.D. Another, more profound result on the linear dependence or independence of two or more matrices is expressed in the following lemma. Lemma 3.2.2. A set fA1 ; A2 ; : : : ; Ak g of two or more m n matrices, the first of which is nonnull, is linearly dependent if and only if at least one of the matrices is expressible as a linear combination of the preceding ones, that is, if and only if, for some integer j (2 j k), Aj is expressible as a linear combination of A1 ; : : : ; Aj 1 . (Or, equivalently, fA1 ; A2 ; : : : ; Ak g is linearly independent if and only if none of the matrices is expressible as a linear combination of its predecessors, that is, if and only if, for every integer j (2 j k), Aj is not expressible as a linear combination of A1 ; : : : ; Aj 1 .) Proof. Suppose that, for some j , Aj is expressible as a linear combination of A1 ; : : : ; Aj 1 . Then, it follows immediately from Lemma 3.2.1 that the set fA1 ; : : : ; Ak g is linearly dependent. Conversely, suppose that fA1 ; : : : ; Ak g is linearly dependent. Define j to be the first integer for which fA1 ; : : : ; Aj g is a linearly dependent set. Then, x1 A1 C C xj Aj D 0 for some scalars x1 ; : : : ; xj , not all zero. Moreover, xj ¤ 0, since otherwise fA1 ; : : : ; Aj 1 g would be a linearly dependent set, contrary to the definition of j . Thus,

Exercises

25

Aj D .x1 =xj /A1 C C .xj 1 =xj /Aj 1 : Q.E.D. It is a simple exercise to verify the following corollary of Lemma 3.2.2. Corollary 3.2.3. Suppose that fA1 ; : : : ; Ak g is a (nonempty) linearly independent set of m n matrices and that A is another m n matrix. Then, the set fA1 ; : : : ; Ak ; Ag is linearly independent if and only if A is not expressible as a linear combination of the matrices A1 ; : : : ; Ak . The following lemma gives a basic result on the linear dependence or independence of linear combinations of matrices. Lemma 3.2.4. Let A1 ; A2 ; : : : ; Ak represent m n matrices. Further, for j D 1; : : : ; r, let Cj D x1j A1 C x2j A2 C C xkj Ak (where x1j ; x2j ; : : : ; xkj are scalars), and let xj D .x1j ; x2j ; : : : ; xkj /0 . If A1 ; A2 ; : : : ; Ak are linearly independent and if x1 ; x2 ; : : : ; xr are linearly independent, then C1 ; C2 ; : : : ; Cr are linearly independent. If x1 ; x2 ; : : : ; xr are linearly dependent, then C1 ; C2 ; : : : ; Cr are linearly dependent. Proof. Observe that, for any scalars y1 ; y2 ; : : : ; yr , r X

yj Cj D

j D1

and

r X

r r X X yj x1j A1 C yj x2j A2 C C yj xkj Ak (2.1)

j D1 r X j D1

yj xj D

j D1 r X j D1

yj x1j ;

j D1 r X j D1

yj x2j ; : : : ;

r X

0 yj xkj :

(2.2)

j D1

Now, suppose that x1 ; x2 ; : : : ; xr are linearly dependent. P Then, there exist scalars y1 ; y2 ; : : : ; yr , not all of which are zero, such that jr D1 yj xj D 0 and P hence [in light of equalities (2.2) and (2.1)] such that jr D1 yj Cj D 0. Thus, C1 ; C2 ; : : : ; Cr are linearly dependent. Alternatively, suppose that A1 ; A2 ; : : : ; Ak are linearly independent and x1 ; x2 ; : : : ; xP r are linearly independent. And let y1 ; y2 ; : : : ; yr represent any scalars such that jr D1 yj Cj D 0. Then, in light of equality (2.1) (and the linear indePr pendence of A1 ; A2 ; : : : ; Ak ), j D1 yj xij D 0 (for i D 1; 2; : : : ; r) and hence Pr y x D 0, implying (in light of the linear independence of x1 ; x2 ; : : : ; j D1 j j xr ) that y1 D y2 D D yr D 0. Thus, C1 ; C2 ; : : : ; Cr are linearly independent. Q.E.D.


26


1. For what values of the scalar k are the three row vectors .k; 1; 0/, .1; k; 1/, and .0; 1; k/ linearly dependent, and for what values are they linearly independent? Describe your reasoning. Section 3.2 2. Let A, B, and C represent three linearly independent mn matrices. Determine whether or not the three pairwise sums A C B, A C C, and B C C are linearly independent. (Hint. Take advantage of Lemma 3.2.4.)

4 Linear Spaces: Row and Column Spaces

Associated with any matrix is a very important characteristic called the rank. The rank of a matrix is the subject of Section 4.4. There are several (consistent) ways of defining the rank. The most fundamental of these is in terms of the dimension of a linear space. Linear spaces and their dimensions are discussed in Sections 4.1 through 4.3. Any matrix has two characteristics that are even more basic than its rank; these are two linear spaces that are respectively known as the row and column spaces of the matrix—discussion of row and column spaces is included in the coverage of Sections 4.1 through 4.3. It is shown in Section 4.4 that the column space of a matrix is of the same dimension as its row space; the rank of the matrix equals this dimension. The final section of Chapter 4 (Section 4.5) gives some basic results on the ranks and row and column spaces of partitioned matrices and of sums of matrices.

4.1 Some Definitions, Notation, and Basic Relationships and Properties a. Column spaces The column space of an m n matrix A is the set whose elements consist of all m-dimensional column vectors that are expressible as linear combinations of the n columns of A. Thus, the elements of the column space of A consist of all m-dimensional column vectors of the general form x1 a1 C x2 a2 C C xn an ; where x1 ; x2 ; : : : ; xn are scalars and a1 ; a2 ; : : : ; an represent the columns of A, or equivalently of the general form Ax, where x is an n 1 column vector. For example, the column space of the 3 4 matrix

28


0

2 4 @1 2 0 0

0 0 1

1 0 0A 2

includes the column vector 0 1 0 1 0 1 0 1 0 1 4 2 4 0 0 @2A D 2@1A C 0@ 2A 3@0A C 0@0A ; 3 0 0 1 2 but not the column vector .1; 0; 0/0 . b. Linear spaces A nonempty set, say V, of matrices (all of which have the same dimensions) is called a linear space if: (1) for every matrix A in V and every matrix B in V, the sum A C B is in V; and (2) for every matrix A in V and every scalar k, the product kA is in V. [The definition of a linear space can be extended to sets whose elements may not be matrices—refer, for example, to Halmos (1958) for a more general discussion.] If A1 ; : : : ; Ak are matrices in a linear space V and x1 ; : : : ; xk are scalars, then it follows from the definition of a linear space that the linear combination x1 A1 C C xk Ak is also in V. Clearly, the column space of any m n matrix is a linear space (comprising various m1 matrices). And the set consisting of all mn matrices is a linear space. Further, since sums and scalar multiples of symmetric matrices are symmetric, the set of all n n symmetric matrices is a linear space. Note that every linear space contains the null matrix 0 (of appropriate dimensions), and that the set f0g, whose only element is a null matrix, is a linear space. Note also that if a linear space contains a nonnull matrix, then it contains an infinite number of nonnull matrices. c. Row spaces The row space of an m n matrix A is the set whose elements consist of all ndimensional row vectors that are expressible as linear combinations of the m rows of A. Thus, the elements of the row space of A consist of all n-dimensional row vectors of the general form x1 a01 C x2 a02 C C xm a0m ; where x1 ; x2 ; : : : ; xm are scalars and a01 ; a02 ; : : : ; a0m represent the rows of A, or equivalently of the general form x0 A, where x0 is an m-dimensional row vector. Clearly, a row space, like a column space, is a linear space.

4.2 Subspaces

29

d. Notation The symbol C.A/ will be used to denote the column space of a matrix A, and R.A/ will be used to denote the row space of A. The symbol Rmn will be used to denote the linear space whose elements consist of all m n matrices. The symbol Rn will often be used (in place of Rn1 or R1n ) to represent the set of all n-dimensional column vectors or, depending on the context, the set of all n-dimensional row vectors. Note that R.In / D Rn (where Rn is to be interpreted as the set of all n-dimensional row vectors), and C.In / D Rn (where Rn is to be interpreted as the set of all n-dimensional column vectors). Sometimes, x 2 S will be written as an abbreviation for the statement that x is an element of a set S (or to introduce x as an element of S ). Similarly, x 62 S will sometimes be written as an abbreviation for the statement that x is not an element of S . e. Relationship between row and column spaces The following lemma relates the column space of a matrix to the row space of its transpose. Lemma 4.1.1. For any matrix A, y 2 C.A/ if and only if y 0 2 R.A0 /. Proof. If y 2 C.A/, then y D Ax for some column vector x, implying that y 0 D .Ax/0 D x0 A0 and hence that y 0 2 R.A0 /. An analogous argument can be Q.E.D. used to show that if y 0 2 R.A0 /, then y 2 C.A/. f. A basic property of linear spaces The following lemma gives a basic property of linear spaces. Lemma 4.1.2. Let B represent an m n matrix, and let V represent a linear space of m n matrices. Then, for any matrix A in V, A C B 2 V if and only if B 2 V. Proof. If B 2 V, then it is apparent from the very definition of a linear space that A C B 2 V. Conversely, if A C B 2 V, then, since B D .A C B/ C .1/A, it is clear from the definition of a linear space that B 2 V. Q.E.D.

4.2 Subspaces A subset U of a linear space V is called a subspace of V if U is itself a linear space. Trivial examples of a subspace of a linear space V are: (1) the set f0g, whose only element is the null matrix, and (2) the entire set V. The column space C.A/ of an m n matrix A is a subspace of Rm (when Rm is interpreted as the set of all m-dimensional column vectors), and R.A/ is a subspace of Rn (when Rn is interpreted as the set of all n-dimensional row vectors).

30


Some additional terminology and notation are required in our discussion of subspaces. For any two subsets S and T of a given set (and, in particular, for any two subspaces of Rmn ), S is said to be contained in T if every element of S is an element of T . Subsequently, S T (or T S ) is written as an abbreviation for the statement that S is contained in T . Note that, if S T and T S , then S D T , that is, the subsets S and T are identical. Some basic results on row and column spaces are expressed in the following two lemmas. Lemma 4.2.1. Let A represent an m n matrix. Then, for any subspace U of Rm , C.A/ U if and only if every column of A belongs to U. Similarly, for any subspace V of Rn , R.A/ V if and only if every row of A belongs to V. Proof. Let a1 ; : : : ; an represent the columns of A. If C.A/ U, then obviously ai 2 U (i D 1; : : : ; n). Conversely, suppose that ai 2 U (i D 1; : : : ; n). For y 2 C.A/, there exist scalars x1 ; : : : ; xn such that y D x1 a1 C C xn an , implying (since U is a linear space) that y 2 U. That R.A/ V if and only if every row of A belongs to V can be proved in a similar fashion. Q.E.D. Lemma 4.2.2. For any m n matrix A and m p matrix B, C.B/ C.A/ if and only if there exists an n p matrix F such that B D AF . Similarly, for any m n matrix A and q n matrix C, R.C/ R.A/ if and only if there exists a q m matrix L such that C D LA. Proof. Let b1 ; : : : ; bp represent the columns of B. Clearly, there exists a matrix F such that B D AF if and only if there exist p column vectors f1 ; : : : ; fp (of dimension n) such that bi D Afi (i D 1; : : : ; p), that is, if and only if (for i D 1; : : : ; p) bi 2 C.A/. We conclude on the basis of Lemma 4.2.1 that C.B/ C.A/ if and only if there exists a matrix F such that B D AF . That R.C/ R.A/ if and only if there exists a matrix L such that C D LA follows from an analogous argument. Q.E.D. The “if” part of Lemma 4.2.2 can be restated as the following corollary. Corollary 4.2.3. For any mn matrix A and np matrix F , C.AF / C.A/. Similarly, for any m n matrix A and q m matrix L, R.LA/ R.A/. As a further consequence of Lemma 4.2.2, we have the following result. Corollary 4.2.4. Let A represent an m n matrix, E an n k matrix, F an n p matrix, L a q m matrix, and T an s m matrix. (1) If C.E/ C.F /, then C.AE/ C.AF /; and if C.E/ D C.F /, then C.AE/ D C.AF /. (2) If R.L/ R.T /, then R.LA/ R.T A/; and if R.L/ D R.T /, then R.LA/ D R.T A/. Proof. (1) If C.E/ C.F /, then (according to Lemma 4.2.2) E D F B for some matrix B, implying that AE D AF B and hence (according to Lemma 4.2.2) that C.AE/ C.AF /. Similarly, if C.F / C.E/, then C.AF / C.AE/. Thus, if C.E/ D C.F / [in which case C.E/ C.F / and C.F / C.E/], then C.AE/ C.AF / and C.AF / C.AE/ and hence C.AE/ D C.AF /. (2) The proof of Part (2) is analogous to that of Part (1). Q.E.D.

4.3 Bases

31

Finally, we have the following lemma, which can be easily verified by making use of either Lemma 4.1.1 or Lemma 4.2.2. Lemma 4.2.5. Let A represent an m n matrix and B an m p matrix. Then, (1) C.A/ C.B/ if and only if R.A0 / R.B0 /, and (2) C.A/ D C.B/ if and only if R.A0 / D R.B0 /.

4.3 Bases a. Some definitions (and notation) The span of a finite set of matrices (having the same dimensions) is defined as follows: the span of a finite nonempty set fA1 ; : : : ; Ak g is the set consisting of all matrices that are expressible as linear combinations of A1 ; : : : ; Ak , and the span of the empty set is the set f0g, whose only element is the null matrix. The span of a finite set S of matrices will be denoted by the symbol sp.S /; sp.fA1 ; : : : ; Ak g/, which represents the span of the set fA1 ; : : : ; Ak g, will generally be abbreviated to sp.A1 ; : : : ; Ak /. Clearly, sp.S / is a linear space. A finite set S of matrices in a linear space V is said to span V if sp.S / D V. Or, equivalently [since sp.S / V], S spans V if V sp.S /. A basis for a linear space V is a finite set of linearly independent matrices in V that spans V. Note that the empty set is the (unique) basis for the linear space f0g. Clearly, the column space C.A/ of an m n matrix A is spanned by the set whose members consist of the n columns of A. If this set is linearly independent, then it is a basis for C.A/; otherwise, it is not. Similarly, R.A/ is spanned by the set whose members are the m rows of A, and depending on whether this set is linearly independent or linearly dependent, it may or may not be a basis for R.A/. b. Natural bases for Rmn and for the linear space of all n n symmetric matrices For i D 1; : : : ; m and j D 1; : : : ; n, let Uij represent the m n matrix whose ij th element equals 1 and whose remaining mn 1 elements equal 0. The set whose members consist of the mn matrices U11 ; U21 ; : : : ; Um1 ; : : : ; U1n ; U2n ; : : : ; Umn spans the linear space Rmn [as is evident from result (2.4.3)] and is linearly independent (as can be easily verified) and hence is a basis for Rmn . The set fU11 ; U21 ; : : : ; Um1 ; : : : ; U1n ; U2n ; : : : ; Umn g is sometimes called the natural basis for Rmn . When m D 1 or n D 1 (in which case the members of Rmn are n-dimensional row vectors or m-dimensional column vectors), the natural basis for Rmn is the set whose members consist of the n rows .1; 0; : : : ; 0/; : : : ; .0; : : : ; 0; 1/ of In or the m columns of Im . For i D 1; : : : ; n, let Uii represent the nn matrix whose i th diagonal element equals 1 and whose remaining n2 1 elements equal 0. And, for j < i D 1; : : : ; n,

32


let Uij represent the n n matrix whose ij th and j i th elements equal 1 and whose remaining n2 2 elements equal 0. Then, for any nn symmetric matrix A D faij g, AD

n X i X

aij Uij

(3.1)

iD1 j D1

[as is readily apparent from result (2.4.3) or upon comparing the corresponding elements of the left and right sides of equality (3.1)]. And it follows that the ; U21 ; : : : ; Un1 ;:::; set whose members consist of the n.n C 1/=2 matrices U11 ; : : : ; Uni ; : : : ; Unn spans the linear space of all n n symmetric Uii ; UiC1;i matrices. Moreover, this set is linearly independent (as can be easily verified). ; U21 ; : : : ; Un1 ; : : : ; Uii ; UiC1;i ; : : : ; Uni ; : : : ; Unn g is a basis Thus, the set fU11 for the linear space of all n n symmetric matrices. c. Existence of a basis It was determined in Subsection b that the linear space Rmn has a basis and that the linear space of all nn symmetric matrices has a basis. Does every linear space (of m n matrices) have a basis? This question is answered (in the affirmative) as the culmination of a series of basic results—presented in the form of a lemma and two theorems—on linearly independent sets and spanning sets. Lemma 4.3.1. Let A1 ; : : : ; Ap and B1 ; : : : ; Bq represent matrices in a linear space V. Then, if the set fA1 ; : : : ; Ap g spans V, so does the set fA1 ; : : : ; Ap ; B1 ; : : : ; Bq g. Moreover, if the set fA1 ; : : : ; Ap ; B1 ; : : : ; Bq g spans V and if B1 ; : : : ; Bq are expressible as linear combinations of A1 ; : : : ; Ap , then the set fA1 ; : : : ; Ap g spans V. Lemma 4.3.1 can be proved rather simply by showing that if B1 ; : : : ; Bq are expressible as linear combinations of A1 ; : : : ; Ap , then any linear combination of the matrices A1 ; : : : ; Ap ; B1 ; : : : ; Bq is expressible as a linear combination of A1 ; : : : ; Ap and vice versa. The details are left as Exercise 6. Theorem 4.3.2. Suppose that V is a linear space that is spanned by a set of r matrices, and let S represent any set of k linearly independent matrices in V. Then, k r, and if k D r, S is a basis for V. Proof. Let us restrict attention to the case r > 0. (The proof for the case r D 0 is trivial: if r D 0, then V D f0g, and the only linearly independent set of matrices in V is the empty set, which contains 0 members and is the basis for V.) Denote by A1 ; : : : ; Ak the k matrices in the linearly independent set S , and let fB1 ; : : : ; Br g represent a set of r matrices that spans V. To prove that k r, assume—with the intent of establishing a contradiction— the contrary, that is, assume that k > r. Consider the set fA1 ; B1 ; : : : ; Br g, obtained by inserting the matrix A1 at the beginning of the set fB1 ; : : : ; Br }. According to Lemma 4.3.1, this set spans V. Moreover, since A1 is expressible as a linear combination of B1 ; : : : ; Br , the

4.3 Bases

33

set fA1 ; B1 ; : : : ; Br g is (according to Lemma 3.2.1) linearly dependent, implying (in light of Lemma 3.2.2) that, among the matrices B1 ; : : : ; Br , there exists a matrix, say Bp , that is expressible as a linear combination of the matrices that precede it in the sequence A1 ; B1 ; : : : ; Br . Thus, it follows from Lemma 4.3.1 that the set fA1 ; B1 ; : : : ; Bp1 ; BpC1 ; : : : ; Br g, obtained by deleting Bp from the set fA1 ; B1 ; : : : ; Br g, spans V. Consider now the set fA1 ; A2 ; B1 ; : : : ; Bp1 ; BpC1 ; : : : ; Br g obtained by inserting the matrix A2 after A1 in the set fA1 ; B1 ; : : : ; Bp1 ; BpC1 ; : : : ; Br g. This set spans V. Moreover, this set is linearly dependent, so that, among the matrices B1 ; : : : ; Bp1 ; BpC1 ; : : : ; Br , there exists a matrix that is expressible as a linear combination of the matrices that precede it in the sequence A1 ; A2 ; B1 ; : : : ; Bp1 ; BpC1 ; : : : ; Br . Deleting this matrix from the set fA1 ; A2 ; B1 ; : : : ; Bp1 ; BpC1 ; : : : ; Br g, we obtain a subset of r matrices that spans V. Continuing this process of inserting “A matrices” and deleting “B matrices,” we eventually (after a total of r steps) find that the set fA1 ; : : : ; Ar g spans V. It follows that the matrices ArC1 ; : : : ; Ak are expressible as linear combinations of A1 ; : : : ; Ar , which, since S is a linearly independent set, results in a contradiction. We conclude that k r. If k D r, then by inserting “A matrices” and deleting “B matrices” via the same algorithm employed in proving that k r, we find that the set S spans V and hence, since S is linearly independent, that S is a basis for V. Q.E.D. As discussed in Subsection b, the linear space Rmn (of all m n matrices) is spanned by mn matrices, and the linear space of all symmetric matrices of order n is spanned by n.n C 1/=2 matrices. Thus, we obtain, as a corollary of Theorem 4.3.2, the following result. Corollary 4.3.3. The number of matrices in a linearly independent set of mn matrices cannot exceed mn, and the number of matrices in a linearly independent set of n n symmetric matrices cannot exceed n.n C 1/=2. Theorem 4.3.4. Every linear space (of m n matrices) has a basis. Proof. Let V represent an arbitrary linear space of mn matrices. The proof of Theorem 4.3.4 consists of a description of a stepwise procedure for constructing a (finite) set B of linearly independent matrices that spans V, that is, for constructing a basis B for V. If V D f0g, that is, if there are no nonnull matrices in V, then terminate the procedure immediately and take B to be the empty set. Otherwise, locate a nonnull matrix A1 in V. If the linearly independent set fA1 g, whose only member is A1 , spans V, then terminate the procedure and take B D fA1 g. If this set does not span V, then locate a matrix A2 that cannot be expressed as a scalar multiple of A1 . Necessarily (according to Corollary 3.2.3), the set fA1 ; A2 g is linearly independent. If the set fA1 ; A2 g spans V, then terminate the procedure, taking B D fA1 ; A2 g. If fA1 ; A2 g does not span V, then locate a matrix A3 in V that is not expressible as a linear combination of A1 and A2 . Necessarily, the set fA1 ; A2 ; A3 g is linearly independent.

34


Continuing, on the kth step, either we terminate the procedure, taking B D fA1 ; : : : ; Ak g, or else we locate a matrix AkC1 in V such that the set fA1 ; : : : ; AkC1 g is linearly independent. The stepwise procedure terminates after no more than mn steps, since otherwise there would exist a set of more thn mn linearly independent matrices, contradicting Corollary 4.3.3. Q.E.D. d. Uniqueness of representation Any particular basis for a linear space has the following property. Lemma 4.3.5. A matrix A in a linear space V has a unique representation in terms of any particular basis fA1 ; : : : ; Ak g; that is, the coefficients x1 ; : : : ; xk in the linear combination A D x1 A 1 C C x k A k are uniquely determined. Proof. Let x1 ; : : : ; xk and y1 ; : : : ; yk represent any scalars such that A D Pk Pk iD1 xi Ai and A D iD1 yi Ai . Then, k X iD1

.yi xi /Ai D

k X iD1

yi Ai

k X

xi Ai D A A D 0:

iD1

Since A1 ; : : : ; Ak are linearly independent matrices, we conclude that yi xi D 0 or equivalently that yi D xi (i D 1; : : : ; k). Q.E.D. e. Dimension With the exception of the linear space f0g, for which the only basis is the empty set, every linear space of m n matrices has an infinite number of bases. However, it follows from Theorem 4.3.2 that if two sets, one containing k1 matrices and the other containing k2 matrices, are both bases for the same linear space, then k1 k2 and k2 k1 . Thus, we have the following result. Theorem 4.3.6. Any two bases for the same linear space contain the same number of matrices. The number of matrices in a basis for a linear space V is called the dimension of V and is denoted by the symbol dim V or dim.V/. (Note that the term dimension is used, not only in reference to the number of matrices in a basis, but also in reference to the number of rows or columns in a matrix—which usage is intended can be determined from the context.) As an essentially immediate consequence of Theorem 4.3.2, we obtain the results expressed in the following three theorems. Theorem 4.3.7. If a linear space V is spanned by a set of r matrices, then dim V r, and if there is a set of k linearly independent matrices in V, then dim V k.

4.3 Bases

35

Theorem 4.3.8. If U is a subspace of a linear space V, then dim U dim V. Theorem 4.3.9. Any set of r linearly independent matrices in an r-dimensional linear space V is a basis for V. A further, related result is expressed in the following theorem. Theorem 4.3.10. Let U and V represent linear spaces of m n matrices. If U V (that is, if U is a subspace of V) and if in addition dim U D dim V, then U D V. Proof. Let r D dim U, and let S represent any set of r linearly independent matrices that is a basis for U. Suppose that U V, in which case all r of the matrices in S are in V. If dim U D dim V, then (according to Theorem 4.3.9) S is a basis for V, implying that V D sp.S / D U. Q.E.D. It follows from the discussion of Subsection b that dim.Rmn / D mn:

(3.2)

This result, together with Theorem 4.3.8, implies that dim U mn

(3.3)

for any linear space U of m n matrices (as is also deducible from Corollary 4.3.3). f. Extracting a basis from a spanning set A basis for a linear space can be extracted from any spanning set, as indicated by the following theorem. Theorem 4.3.11. Any set S that spans a linear space V (of m n matrices) contains a subset that is a basis for V. Proof. Assume that S contains at least one nonnull matrix. (The proof for the case where S is empty or consists entirely of null matrices is trivial—in that case, V D f0g, and the empty set is a basis for V.) Let A1 ; : : : ; Ak represent the matrices in S . Define S to be the subset of S obtained by successively applying to each of the matrices A1 ; : : : ; Ak the following algorithm: include the matrix as a member of S if it is nonnull and if it is not expressible as a linear combination of any matrices already included in S . It is clear from Lemma 3.2.2 that the subset S formed in this way is linearly independent. Moreover, every matrix in S either belongs to S or is expressible as a linear combination of matrices that belong to S , implying (according to Lemma Q.E.D. 4.3.1) that S spans V. We conclude that S is a basis for V. Given any particular set that spans a linear space V, the algorithm described in the proof of Theorem 4.3.11 can be used to construct a basis for V and, in the process, to determine the dimension of V.

36


g. Inclusion of any particular linearly independent set in a basis Any set of linearly independent matrices in a linear space can be augmented to form a basis, as indicated by the following theorem, which generalizes Theorem 4.3.4. Theorem 4.3.12. For any set S of r linearly independent matrices in a kdimensional linear space V, there exists a basis for V that includes all r of the matrices in S (and k r additional matrices). Proof. Assume that r > 0. (In the special case r D 0, the theorem essentially reduces to Theorem 4.3.4.) Denote by A1 ; : : : ; Ar the matrices in S , and let fB1 ; : : : ; Bk g represent any basis for V. An alternative basis for V can be extracted from the set fA1 ; : : : ; Ar ; B1 ; : : : ; Bk g, which (according to Lemma 4.3.1) spans V, by applying the algorithm described in the proof of Theorem 4.3.11. It follows from Lemma 3.2.2 (and from the method of extraction) that the alternative basis includes all r of the Q.E.D. matrices A1 ; : : : ; Ar . Given any particular set that spans a k-dimensional linear space V, the procedure described in the proof of Theorem 4.3.12 can be used to construct a basis that includes as a subset any particular set fA1 ; : : : ; Ar g of linearly independent matrices. An alternative approach, not requiring knowledge of a spanning set, is to adopt the procedure described in the proof of Theorem 4.3.4. In the alternative approach, a basis is constructed in k r steps, the i th of which consists of locating a matrix ArCi in V that is not expressible as a linear combination of A1 ; : : : ; ArCi1 . Clearly, the set fA1 ; : : : ; Ak g is a basis for V.

4.4 Rank of a Matrix a. Some definitions and basic results The row rank of a matrix A is defined to be the dimension of the row space of A, and the column rank of A is defined to be the dimension of the column space of A. A fundamental result on the row and column spaces of a matrix is expressed in the following theorem. Theorem 4.4.1. The row rank of any matrix A equals the column rank of A. Preliminary to proving Theorem 4.4.1, it is convenient to prove the following result, which is of some interest in its own right. Theorem 4.4.2. Let A represent an m n nonnull matrix of row rank r and column rank c. Then, there exists an m c matrix B and a c n matrix L such that A D BL. Similarly, there exists an m r matrix K and an r n matrix T such that A D KT . Proof. According to Theorem 4.3.4, there exists a set fb1 ; : : : ; bc g of c vectors that is a basis for C.A/. Take B to be the m c matrix whose columns are b1 ; : : : ; bc . Then, C.B/ D sp.b1 ; : : : ; bc / D C.A/, and it follows from Lemma 4.2.2 that

4.4 Rank of a Matrix

37

there exists a c n matrix L such that A D BL. The existence of an m r matrix K and an r n matrix T such that A D KT can be established via an analogous argument. Q.E.D. Proof (of Theorem 4.4.1). Let r represent the row rank and c the column rank of a matrix A. Assume that A is nonnull. (The proof for the case A D 0 is trivial: If A D 0, then r D 0 D c.) According to Theorem 4.4.2, there exists an m c matrix B and a c n matrix L such that A D BL, and similarly there exists an m r matrix K and an r n matrix T such that A D KT . It follows from Lemma 4.2.2 that R.A/ R.L/ and C.A/ C.K/. Thus, making use of Theorems 4.3.8 and 4.3.7 [and observing that R.L/ is spanned by the c rows of L and C.K/ by the r columns of K], we find that r dimŒR.L/ c and similarly that c dimŒC.K/ r: We conclude that r D c. Q.E.D. In light of Theorem 4.4.1, it is not necessary to distinguish between the row and column ranks of a matrix A. Their common value is called the rank of A and is denoted by the symbol rank A or rank.A/. b. Restatement of results on dimensions Our results on the dimensions of linear spaces and subspaces can be specialized to row and column spaces and restated in terms of ranks. Result (3.3) implies that the row rank of a matrix cannot exceed the number of columns and that its column rank cannot exceed the number of rows, leading to the following conclusion. Lemma 4.4.3. For any m n matrix A, rank.A/ m and rank.A/ n. Theorem 4.3.8 has the following implication. Theorem 4.4.4. Let A represent an m n matrix, B an m p matrix, and C a q n matrix. If C.B/ C.A/, then rank.B/ rank.A/. Similarly, if R.C/ R.A/, then rank.C/ rank.A/. In light of Corollary 4.2.3, we have the following corollary of Theorem 4.4.4. Corollary 4.4.5. For any m n matrix A and n p matrix F , rank.AF / rank.A/ and rank.AF / rank.F /. Theorem 4.3.10 has the following implications. Theorem 4.4.6. Let A represent an m n matrix, B an m p matrix, and C a q n matrix. If C.B/ C.A/ and rank.B/ D rank.A/, then C.B/ D C.A/. Similarly, if R.C/ R.A/ and rank.C/ D rank.A/, then R.C/ D R.A/. Corollary 4.4.7. Let A represent an m n matrix and F an n p matrix. If rank.AF / D rank.A/, then C.AF / D C.A/. Similarly, if rank.AF / D rank.F/, then R.AF / D R.F /.

38


c. Nonsingular matrices and matrices of full row or column rank Among m n matrices, the maximum rank is min.m; n/, as indicated by Lemma 4.4.3. The minimum rank is 0, which is achieved by the m n null matrix 0 (and by no other m n matrix). An m n matrix A is said to have full row rank if rank.A/ D m, that is, if its rank equals the number of rows, and to have full column rank if rank.A/ D n. Clearly, an m n matrix can have full row rank only if m n, that is, only if the number of rows does not exceed the number of columns, and can have full column rank only if n m. A matrix is said to be nonsingular if it has both full row rank and full column rank. Clearly, any nonsingular matrix is square. By definition, an n n matrix A is nonsingular if and only if rank.A/ D n. An n n matrix of rank less than n is said to be singular. An mn matrix can be expressed as the product of a matrix having full column rank and a matrix having full row rank as indicated by the following theorem, which is a restatement and extension of Theorem 4.4.2. Theorem 4.4.8. Let A represent an mn nonnull matrix of rank r. Then, there exist an mr matrix B and an r n matrix T such that A D BT . Moreover, for any m r matrix B and r n matrix T such that A D BT , rank.B/ D rank.T / D r, that is, B has full column rank and T has full row rank. Proof. The existence of an m r matrix B and an r n matrix T such that A D BT follows from Theorem 4.4.2. Moreover, for any such m r matrix B and r n matrix T , it follows from Lemma 4.4.3 that rank.B/ r and rank.T / r and from Corollary 4.4.5 that rank.B/ r and rank.T / r, leading to the conclusion that rank.B/ D r and rank.T / D r. Q.E.D. d. A decomposition of a rectangular matrix The decomposition of a rectangular matrix given in Theorem 4.4.8 can be expressed in an alternative form, as described in the following theorem. Theorem 4.4.9. Let A represent an m n nonnull matrix of rank r. Then, there exist an m m nonsingular matrix B and an n n nonsingular matrix K such that Ir 0 ADB K: 0 0 Proof. According to Theorem 4.4.8, there exist an m r matrix B1 of full column rank and an r n matrix K1 of full row rank such that A D B1 K1 . Let b1 ; : : : ; br represent the first, : : : ; rth columns, respectively, of B1 and k01 ; : : : ; k0r the first, : : : ; rth rows, respectively, of K1 . Clearly, b1 ; : : : ; br are linearly independent, and k01 ; : : : ; k0r are linearly independent. Then, according to Theorem 4.3.12, there exist m r vectors brC1 ; : : : ; bm such that b1 ; : : : ; br ; brC1 ; : : : ; bm form a basis for Rm , and, similarly, there exist n r vectors k0rC1 ; : : : ; k0n such that k01 ; : : : ; k0r ; k0rC1 ; : : : ; k0n form a basis for Rn .

4.4 Rank of a Matrix

39

K1 , where B2 D .brC1 ; : : : ; bm / and K2 D Take B D .B1 ; B2 / and K D K2 0 0 1 krC1 B :: C @ : A. Then, clearly, B and K are nonsingular, and k0n

A D B1 K1 D .B1 ; B2 /

Ir 0 0 0

K1 K2

DB

Ir 0 K: 0 0 Q.E.D.

e. Some equivalent definitions The following theorem can be used to characterize the rank of a matrix in terms of the ranks of its submatrices. Theorem 4.4.10. Let A represent any mn matrix of rank r. Then, A contains r linearly independent rows and r linearly independent columns, and, for any r linearly independent rows and r linearly independent columns of A, the r r submatrix, obtained by striking out the other m r rows and n r columns, is nonsingular. Moreover, any set of more than r rows or more than r columns (of A) is linearly dependent, and there exists no submatrix of A whose rank exceeds r. Proof. Since the m rows and n columns of A span R.A/ and C.A/, respectively, it follows from Theorem 4.3.11 that A contains r linearly independent rows and r linearly independent columns. Moreover, it follows from Theorem 4.3.7 that any set of more than r rows or more than r columns is linearly dependent. Consider now the r r submatrix B of A obtained by striking out all of the rows and columns of A other than r linearly independent rows, say rows i1 ; : : : ; ir , respectively, and r linearly independent columns, say columns j1 ; : : : ; jr , respectively. Letting aij represent the ij th element of A, the t sth element of B is by definition b ts D ai t js : Let U represent the m r matrix whose columns are columns j1 ; : : : ; jr of A. According to Theorem 4.3.9, rows i1 ; : : : ; ir of A form a basis for R.A/, implying that every row of A is expressible as a linear combination of rows i1 ; : : : ; ir or, equivalently, that r X a tj D f tk aik j .j D 1; : : : ; n/ kD1

for some scalars f t1 ; : : : ; f t r (t D 1; : : : ; m). Thus, the t sth element of U is X X u ts D a tjs D f tk aik js D f tk bks k

(t D 1; : : : ; m; s D 1; : : : ; r), so that

k

40


U D F B; where F is the m r matrix whose t kth element is f tk . Since (according to Lemma 4.4.3 and Corollary 4.4.5) rank.B/ r and rank.B/ rank.U / D r, we conclude that rank.B/ D r or, equivalently, that B is nonsingular. It remains to show that there exists no submatrix of A whose rank exceeds r. Let H represent a matrix obtained by striking out all of the rows and columns of A except rows t1 ; : : : ; tk and columns s1 ; : : : ; sp . Define W to be the m p matrix whose columns are columns s1 ; : : : ; sp of A. It is clear that C.W / C.A/ and (since the rows of H are rows t1 ; : : : ; tk of W ) that R.H/ R.W /. Thus, applying Theorem 4.4.4, rank.H/ rank.W / r: Q.E.D. As applied to symmetric matrices, Theorem 4.4.10 has the following implication. Corollary 4.4.11. Any symmetric matrix of rank r contains an r r nonsingular principal submatrix. The rank of a matrix was defined in terms of the dimension of the row and column spaces. Theorem 4.4.10 suggests some equivalent definitions. The rank of a matrix A is interpretable as the size of the largest linearly independent set that can be formed from the rows of A. Similarly, it is interpretable as the size of the largest linearly independent set that can be formed from the columns of A. The rank of A is also interpretable as the size (number of rows or columns) of the largest nonsingular (square) submatrix of A. Clearly, an m n matrix has full row rank if and only if all m of its rows are linearly independent and has full column rank if and only if all n of its columns are linearly independent. An n n matrix is nonsingular if and only if all of its rows are linearly independent; similarly, it is nonsingular if and only if all of its columns are linearly independent. f. Some elementary equalities Let A represent an arbitrary matrix. Then, it is a simple exercise to show that rank.A0 / D rank.A/

(4.1)

and that, for any nonzero scalar k, rank.kA/ D rank.A/:

(4.2)

As a special case of result (4.2), we have that rank.A/ D rank.A/:

(4.3)

4.5 Some Basic Results on Partitioned Matrices and on Sums of Matrices

41

4.5 Some Basic Results on Partitioned Matrices and on Sums of Matrices Inwhat B/ is used as an abbreviation for CŒ.A; B/ and follows, the symbol C.A; A A R as an abbreviation for R . Thus, C.A; B/ represents the column C C space of a partitioned matrix .A; B/ comprising two blocks A and B arranged in a A A row, and R represents the row space of a partitioned matrix comprising C C two blocks A and C arranged in a column. The following five lemmas (and two corollaries) give some basic results on the ranks and row and column spaces of matrices that have been partitioned into two blocks. Lemma 4.5.1. Let A represent an m n matrix, B an m p matrix, and C a q n matrix. Then, C.A/ C.A; B/; A R.A/ R ; C

C.B/ C.A; B/; A R.C/ R : C

Moreover, C.A/ D C.A; B/ , C.B/ C.A/; A R.A/ D R , R.C/ R.A/; C

C.B/ D C.A; B/ , C.A/ C.B/; A R.C/ D R , R.A/ R.C/: C A Proof. That C.A/ C.A; B/, C.B/ C.A; B/, R.A/ R , and C A R.C/ R follows from Lemma 4.2.2 upon observing that C I 0 A D .A; B/ ; B D .A; B/ ; 0 I A A A D .I; 0/ ; C D .0; I/ : C C

Suppose now that C.B/ C.A/. Then, according to Lemma 4.2.2, there exists a matrix F such that B D AF and hence such that .A; B/ D A.I; F /. Thus, C.A; B/ C.A/, implying [since C.A/ C.A; B/] that C.A/ D C.A; B/. Conversely, suppose that C.A/ D C.A; B/. Then, since C.B/ C.A; B/, C.B/ C.A/. Thus, we have established that C.A/ D C.A; B/ , C.B/ C.A/. A That C.B/ D C.A; B/ , C.A/ C.B/, that R.A/ D R , R.C/ C A R.A/, and that R.C/ D R , R.A/ R.C/ can be established via similar C arguments. Q.E.D.

42


Corollary 4.5.2. Let A represent an m n matrix, B an m p matrix, and C a q n matrix. Then, (1) rank.A/ rank.A; B/, with equality holding if and only if C.B/ C.A/; (2) rank.B/ rank.A; B/, with equality holding if and only if C.A/ C.B/; A (3) rank.A/ rank , with equality holding if and only if R.C/ R.A/; C A (4) rank.C/ rank , with equality holding if and only if R.A/ R.C/. C Proof. Since (according to Lemma 4.5.1) C.A/ C.A; B/, it follows from Theorem 4.4.4 that rank.A/ rank.A; B/. Moreover, if C.B/ C.A/, then (according to Lemma 4.5.1) C.A/ D C.A; B/ and, consequently, rank.A/ D rank.A; B/. And, conversely, if rank.A/ D rank.A; B/, then, since C.A/ C.A; B/, it follows from Theorem 4.4.6 that C.A/ D C.A; B/ and hence (in light of Lemma 4.5.1) that C.B/ C.A/. Thus, Part (1) of the corollary is valid. The validity of Parts (2), (3), and (4) can be established in similar fashion. Q.E.D. Lemma 4.5.3. For any m n matrix A, m p matrix B, and q n matrix C, C.B; A/ D C.A; B/; C A R DR ; A C Proof. Since

0 In .B; A/ D .A; B/ ; Ip 0

rank.B; A/ D rank.A; B/; C A rank D rank : A C 0 Ip .A; B/ D .B; A/ ; In 0

we have (as a consequence of Lemma 4.2.2) that C.B; A/ C.A; B/ and C.A; B/ C.B; A/ and hence that C.B; A/ D C.A; B/ [which implies that C A C rank.B; A/ D rank.A; B/]. That R D R [and that rank D A C A A rank ] follows from an analogous argument. Q.E.D. C Lemma 4.5.4. For any m n matrix A, m p matrix B, and n p matrix L, C.A; B/ D C.A; B AL/;

rank.A; B/ D rank.A; B AL/:

Similarly, for any m n matrix A, q n matrix B, and q m matrix L, A A A A R DR ; rank D rank : B B LA B B LA Proof. To establish the first part of the lemma, observe that I L I L .A; B AL/ D .A; B/ ; .A; B/ D .A; B AL/ ; 0 I 0 I


43

implying (in light of Lemma 4.2.2) that C.A; B AL/ C.A; B/ and C.A; B/ C.A; BAL/ and, consequently, that C.A; B/ D C.A; BAL/ and rank.A; B/ D rank.A; B AL/. The second part of the lemma can be established in similar fashion. Q.E.D. Lemma 4.5.5. (1) If A is an m n matrix and E an m q matrix such that C.E/ C.A/ and if B is an m p matrix and F an m r matrix such that C.F / C.B/, then C.E; F / C.A; B/;

rank.E; F / rank.A; B/:

(2) Similarly, if A is an mn matrix and E a p n matrix such that R.E/ R.A/ and if B is a q n matrix and F an r n matrix such that R.F / R.B/, then E A E A R R ; rank rank : F B F B Proof. (1) Suppose that C.E/ C.A/ and C.F / C.B/. Then, according to Lemma 4.2.2, there exist matrices K and L such that E D AK and F D BL. Thus, .E; F / D .A; B/ diag.K; L/, implying that C.E; F / C.A; B/ and further (in light of Theorem 4.4.4) that rank.E; F / rank.A; B/. (2) The proof of Part (2) is analogous to that of Part (1). Q.E.D. Corollary 4.5.6. (1) If A is an m n matrix and E an m q matrix such that C.E/ D C.A/, and if B is an m p matrix and F an m r matrix such that C.F / D C.B/, then C.E; F / D C.A; B/;

rank.E; F / D rank.A; B/:

(2) Similarly, if A is an mn matrix and E a p n matrix such that R.E/ D R.A/, and if B is a q n matrix and F an r n matrix such that R.F / D R.B/, then E A E A R DR ; rank D rank : F B F B Proof. (1) Suppose that C.E/ D C.A/ and C.F / D C.B/. Then, as a consequence of Lemma 4.5.5, we have that C.E; F / C.A; B/ and C.A; B/ C.E; F /. Thus, C.E; F / D C.A; B/, and further rank.E; F / D rank.A; B/. (2) The proof of Part (2) is analogous to that of Part (1). Q.E.D. Lemma 4.5.7. For any m n matrix A, m p matrix B, and q n matrix C, rank.A; B/ rank.A/ C rank.B/; A rank rank.A/ C rank.C/: C

(5.1) (5.2)

Proof. For purposes of establishing inequality (5.1), assume that both A and B are nonnull—if either A or B is null, then, clearly, inequality (5.1) holds as an equality. Let A represent an m r matrix whose columns form a basis for C.A/

44


and B an m s matrix whose columns form a basis for C.B/. Then, clearly, C.A / D C.A/ and C.B / D C.B/. Thus, in light of Corollary 4.5.6 and Lemma 4.4.3, we have that rank.A; B/ D rank.A ; B / r C s D rank.A/ C rank.B/; which establishes inequality (5.1). Inequality (5.2) can be established via an analogous argument. Q.E.D. By repeated application of result (5.1), we obtain the more general result that, for any matrices A1 ; A2 ; : : : ; Ak having m rows, rank.A1 ; A2 ; : : : ; Ak / rank.A1 / C rank.A2 / C C rank.Ak /:

(5.3)

Similarly, by repeated application of result (5.2), we find that, for any matrices A1 ; A2 ; : : : ; Ak having n columns, 0 1 A1 B A2 C B C (5.4) rank B : C rank.A1 / C rank.A2 / C C rank.Ak /: @ :: A Ak The following lemma establishes a basic relationship between the rank or the row or column space of a sum of two matrices and the rank or the row or column space of a certain partitioned matrix. Lemma 4.5.8. For any m n matrices A and B, C.A C B/ C.A; B/; A R.A C B/ R ; B

rank.A C B/ rank.A; B/; A rank.A C B/ rank : B

(5.5) (5.6)

Proof. We have that

I A A C B D .A; B/ D .I; I/ ; I B

implying (in light of Lemma 4.2.2) that C.A C B/ C.A; B/ and R.A C B/ A R and further (in light of Theorem 4.4.4) that rank.A C B/ rank.A; B/ B A and rank.A C B/ rank . Q.E.D. B In light of Lemma 4.5.7, we have the following corollary of Lemma 4.5.8. Corollary 4.5.9. For any m n matrices A and B, rank.A C B/ rank.A/ C rank.B/:

(5.7)


45

By repeated application of results (5.5) through (5.7), we obtain the more general results that, for any m n matrices A1 ; A2 ; : : : ; Ak , C.A1 C A2 C C Ak / C.A1 ; A2 ; : : : ; Ak /;

(5.8a)

rank.A1 C A2 C C Ak / rank.A1 ; A2 ; : : : ; Ak /; 0 1 A1 B A2 C B C R.A1 C A2 C C Ak / RB : C ; @ :: A Ak 0 1 A1 B A2 C B C rank.A1 C A2 C C Ak / rank B : C ; @ :: A

(5.8b)

(5.9a)

(5.9b)

Ak rank.A1 C A2 C C Ak / rank.A1 / C rank.A2 / C C rank.Ak /: (5.10) A further result on the row and column spaces of a sum of matrices is given by the following lemma. Lemma 4.5.10. Let A and B represent m n matrices. Then, for any m p matrix E such that C.A/ C.E/, C.A C B/ C.E/ , C.B/ C.E/: And, similarly, for any q n matrix F such that R.A/ R.F /, R.A C B/ R.F / , R.B/ R.F /: Proof. According to Lemma 4.2.2, A D ER for some matrix R. Further, if C.B/ C.E/, B D ES for some matrix S, implying that A C B D E.R C S/ and hence (in light of Lemma 4.2.2) that C.A C B/ C.E/. Conversely, if C.A C B/ C.E/, then A C B D ET for some matrix T , implying that B D .A C B/ A D E.T R/ and hence that C.B/ C.E/. It can be established in similar fashion that R.A C B/ R.F / if and only if Q.E.D. R.B/ R.F /. The following lemma gives the rank of a block-diagonal matrix in terms of the ranks of the diagonal blocks. Lemma 4.5.11. For any matrices A and B, A 0 rank D rank.A/ C rank.B/: 0 B

(5.11)

Proof. Suppose that both A and B are nonnull—if A or B is null, then equality (5.11) is clearly valid. Let A represent an m r matrix whose columns form a basis for C.A/ and B an n s matrix whose columns form a basis for C.B/.

46


Then, C.A / D C.A/, and C.B / D C.B/, implying (in light of Lemma 4.2.2) that A D AK and B D BL for some matrices K and L and hence that A 0 K 0 A 0 D : 0 B 0 B 0 L Moreover, for any r 1 vector c and s 1 vector d such that c A 0 D 0; 0 B d we find that A c D 0 and B d D 0, implying (since the columns of A are linearly independent) that c D 0 and, similarly, that d D 0. It follows that the columns of diag.A ; B / are linearly independent. Thus, A 0 A 0 D r C s D rank.A/ C rank.B/: (5.12) rank rank 0 B 0 B Further, it follows from Lemma 4.5.7 that A 0 A 0 rank rank C rank rank.A/ C rank.B/: 0 B 0 B

(5.13)

Together, inequalities (5.12) and (5.13) imply equality (5.11). Q.E.D. Repeated application of result (5.11) gives the following formula for the rank of an arbitrary block-diagonal matrix: rankŒdiag.A1 ; A2 ; : : : ; Ak / D rank.A1 / C rank.A2 / C C rank.Ak /:

(5.14)

Note that result (5.14) implies in particular that the rank of a diagonal matrix D equals the number of nonzero diagonal elements in D.

Exercises Section 4.1 1. Which of the following two sets are linear spaces: (a) the set of all n n upper triangular matrices; (b) the set of all n n nonsymmetric matrices? Section 4.2 2. Verify Lemma 4.2.5. 3. Let U and W represent subspaces of a linear space V. Show that if every matrix in V belongs to U or W, then U D V or W D V.

Exercises

47

Section 4.3 4. Let A, B, and C represent three matrices (having the same dimensions) such that A C B C C D 0. Show that sp.A; B/ D sp.A; C/. 5. Let A1 ; : : : ; Ak represent any matrices in a linear space V. Show that sp.A1 ; : : : ; Ak / is a subspace of V and that, among all subspaces of V that contain A1 ; : : : Ak , it is the smallest [in the sense that, for any subspace U (of V) that contains A1 ; : : : ; Ak , sp.A1 ; : : : ; Ak / U]. 6. Prove Lemma 4.3.1. 7. Suppose that fA1 ; : : : ; Ak g is a set of matrices that spans a linear space V but is not a basis for V. Show that, for any matrix A in V, the representation of A in terms of A1 ; : : : ; Ak is nonunique. Section 4.4 8. Let

0 0 1 0 3 B0 2 0 6 ADB @0 2 2 5 0 4 2 1

1 2 2C C: 2A 0

(a) Show that each of the two column vectors .2; 1; 3; 4/0 and .0; 9; 3; 12/0 is expressible as a linear combination of the columns of A [and hence is in C.A/]. (b) Find a basis for C.A/ by applying the algorithm described in the proof of Theorem 4.3.11. (In applying the algorithm, take the spanning set to be the set consisting of the columns of A.) (c) What is the value of rank.A/? Explain your reasoning. (d) Find a basis for C.A/ that includes the two column vectors from Part (a). Do so by applying the algorithm described in the proof of Theorem 4.3.12. (In applying the algorithm, take the spanning set to be the set consisting of the columns of A.) 9. Let A represent a q p matrix, B a p n matrix, and C an m q matrix. Show that (a) if rank.CAB/ D rank.C/, then rank.CA/ D rank.C/ and (b) if rank.CAB/ D rank.B/, then rank.AB/ D rank.B/.

Following the first printing, this exercise was relocated (to better reflect its content) and was renumbered accordingly. In the first printing, it was located under Section 4.2 and ahead of what is now Exercise 3 (formerly Exercise 4).

48


10. Let A represent an m n matrix of rank r. Show that A can be expressed as the sum of r matrices of rank 1. Section 4.5 11. Let A represent an m n matrix and C a q n matrix. (a) Add to the proof of Lemma 4.5.1 by confirming that A R.C/ D R , R.A/ R.C/: C (b) Add to the proof of Corollary 4.5.2 by confirming that rank.C/ A rank , with equality holding if and only if R.A/ R.C/. C

5 Trace of a (Square) Matrix

The primary subject of this relatively short chapter is a basic characteristic of a square matrix called the trace. There are many areas of statistics in which the trace of a matrix is encountered. For example, in the design of experiments, one of the criteria (A-optimality) that is used for comparing designs involves the trace of a matrix (Fedorov 1972). The trace of a matrix is defined and its basic properties described in Section 5.1. Some very useful results on the trace of a product of matrices are covered in Section 5.2. One of these results gives rise to some very useful mathematical equivalences, which are presented in Section 5.3. That the results on the trace of a matrix and the related equivalences are placed in a separate chapter is indicative not only of their importance but of the fact that they don’t fit particularly well into any of the other chapters.

5.1 Definition and Basic Properties The trace of a square matrix A D faij g of order n is defined to be the sum of the n diagonal elements of A and is to be denoted by the symbol tr.A/. Thus, tr.A/ D a11 C a22 C C ann : In particular, tr.In / D n;

tr.Jn / D n;

(1.1)

and, in the case of a 1 1 matrix .k/, whose only element is the scalar k, trŒ.k/ D k:

(1.2)

Clearly, for any scalar k and any n n matrices A and B, tr.kA/ D k tr.A/; tr.A C B/ D tr.A/ C tr.B/; tr.A0 / D tr.A/:

(1.3) (1.4) (1.5)

50


Further, for any r scalars k1 ; k2 ; : : : ; kr and for any r matrices A1 ; A2 ; : : : ; Ar of dimensions n n, r r X X tr ki A i D ki tr.Ai /; (1.6) iD1

iD1

as can be easily verified by, for example, the repeated application of results (1.4) and (1.3). If the diagonal blocks A11 ; A22 ; : : : ; Akk of the partitioned matrix 0 1 A11 A12 : : : A1k BA21 A22 : : : A2k C B C ADB : :: :: C :: @ :: : : : A Ak1 Ak2 : : : Akk are square (in which case A itself is square), then clearly tr.A/ D tr.A11 / C tr.A22 / C C tr.Akk /:

(1.7)

5.2 Trace of a Product Let A D faij g represent an m n matrix and B D fbj i g an n m matrix. Then, tr.AB/ D

m X n X

aij bj i ;

(2.1)

iD1 j D1

P as is evident upon observing that the i th diagonal element of AB is jnD1 aij bj i . Thus, since the j i th element of B is the ij th element of B0 , the trace of the matrix product AB can be formed by multiplying the ij th element of A by the corresponding (ij th) element of B0 and by then summing (over i and j ). It follows from results (1.5) and (1.2.13) that, for any m n matrix A and n m matrix B, (2.2) tr.AB/ D tr.B0 A0 /: A further, very basic result on the product of two matrices is expressed in the following lemma. Lemma 5.2.1. For any m n matrix A and n m matrix B, tr.AB/ D tr.BA/:

(2.3)

Proof. Let aij represent the ij th element of A and Pbj i the j i th element of B, and observe that the j th diagonal element of BA is m iD1 bj i aij . Thus, making use of result (2.1), we find that tr.AB/ D

m X n X iD1 j D1

aij bj i D

n X m X j D1 iD1

aij bj i D

n X m X j D1 iD1

bj i aij D tr.BA/:

5.2 Trace of a Product

51

Q.E.D. Now, let A D faij g and B D fbij g represent m n matrices, and let aj D .a1j ; : : : ; amj /0 and bj D .b1j ; : : : ; bmj /0 represent the j th columns of A and B. Then, since the j i th element of B0 is the ij th element of B and since bj0 aj D P aj0 bj D m iD1 aij bij , it follows from results (2.3) and (2.1) that tr.AB0 / D tr.B0 A/ D

n X

bj0 aj D

j D1

n X

aj0 bj D

j D1

m X n X

aij bij :

(2.4)

iD1 j D1

In particular, tr.AA0 / D tr.A0 A/ D

n X

aj0 aj D

j D1

m X n X

2 aij 0:

(2.5)

iD1 j D1

Thus, both tr.AA0 / and tr.A0 A/ equal the sum of squares of the elements of A (and both are nonnegative). As special cases of results (2.4) and (2.5), we have that, for any m-dimensional column vectors a D fai g and b D fbi g, tr.ab0 / D b0 a D a0 b D

m X

ai bi ;

(2.6)

iD1

tr.aa0 / D a0 a D

m X

ai2 0:

(2.7)

iD1

Note that, together, results (2.2) and (2.3) imply that for any m n matrix A and n m matrix B, tr.AB/ D tr.B0 A0 / D tr.BA/ D tr.A0 B0 /:

(2.8)

Thus, in evaluating the trace of the product of two matrices, the matrices can be permuted or replaced by their transposes or both. Consider now the trace of the product ABC of an m n matrix A, an n p matrix B, and a p m matrix C. Since ABC can be regarded as the product of the two matrices AB and C or, alternatively, of A and BC, it follows from Lemma 5.2.1 that tr.ABC/ D tr.CAB/ D tr.BCA/: (2.9) More generally, for any k matrices A1 ; A2 ; : : : ; Ak for which the matrix product A1 A2 Ak is defined, tr.A1 A2 Ak / D tr.Aj C1 Ak A1 Aj /

(2.10)

(j D 1; : : : ; k 1). Results (2.9) and (2.10) indicate that, for purposes of evaluating the trace of a product of matrices, the matrices can be permuted. However, aside from special cases, they can be permuted only in a certain way. For example, even if the three matrices A, B, and C are square and of the same order so that BAC, like ABC, is defined and square, tr.BAC/ is not necessarily equal to tr.ABC/.

52


5.3 Some Equivalent Conditions The following lemma gives a basic equivalence. Lemma 5.3.1. For any m n matrix A D faij g, A D 0 if and only if tr.A0 A/ D 0. Proof. If A D 0, then obviously tr.A0 A/ D 0. Conversely, if tr.A0 A/ D 0, 2 D 0 and hence that aij D 0 (i D 1; : : : ; m; then it follows from result (2.5) that aij j D 1; : : : ; n) or equivalently that A D 0. Q.E.D. As an essentially immediate consequence of Lemma 5.3.1, we have the following corollary. Corollary 5.3.2. For any m n matrix A, A D 0 if and only if A0 A D 0. The following corollary generalizes Corollary 5.3.2. Corollary 5.3.3. (1) For any m n matrix A and n p matrices B and C, AB D AC if and only if A0 AB D A0 AC. (2) Similarly, for any m n matrix A and p n matrices B and C, BA0 D CA0 if and only if BA0 A D CA0 A. Proof (of Corollary 5.3.3). (1) If AB D AC, then obviously A0 AB D A0 AC. Conversely, if A0 AB D A0 AC, then .AB AC/0 .AB AC/ D .B0 C 0 /.A0 AB A0 AC/ D 0; and it follows from Corollary 5.3.2 that AB AC D 0 or, equivalently, that AB D AC. (2) To establish Part (2), simply take the transpose of each side of the two equivalent equalities AB0 D AC 0 and A0 AB0 D A0 AC 0 . [The equivalence of these two equalities follows from part (1).] Q.E.D. Note that, as a special case of Part (1) of Corollary 5.3.3 (the special case where C D 0), we have that AB D 0 if and only if A0 AB D 0, and, as a special case of Part (2), we have that BA0 D 0 if and only if BA0 A D 0.

Exercises Section 5.2 1. Show that for any m n matrix A, n p matrix B, and p m matrix C, tr.ABC/ D tr.B0 A0 C 0 / D tr.A0 C 0 B0 /: 2. Let A, B, and C represent n n matrices. (a) Using the result of Exercise 1 (or otherwise), show that if A, B, and C are symmetric, then tr.ABC/ D tr.BAC/. (b) Show that [aside from special cases like that considered in Part (a)] tr.BAC/ is not necessarily equal to tr.ABC/.

Exercises

Section 5.3 3. Let A represent an n n matrix such that A0 A D A2 . (a) Show that trŒ.A A0 /0 .A A0 / D 0. (b) Show that A is symmetric.

53

6 Geometrical Considerations

This chapter provides an introduction (at a somewhat general level) to some rather fundamental geometrical ideas and results. Some knowledge of this material is an important prerequisite for assimilating ideas in several areas of statistics, including linear statistical models and multivariate analysis. Among those who teach and write about the more theoretical aspects of linear statistical models, there is a considerable difference of opinion about the extent to which geometrical ideas should be emphasized (relative to “algebraic” ideas). Those who prefer a “geometrical approach” (e.g., Scheffé 1959, Christensen 1996) argue that it is more general, more elegant, and (perhaps most important) more intuitive. Those who prefer a more algebraic approach (e.g., Searle 1971) find it to be more rigorous, more concrete, more palatable to those with a limited mathematical background, and (perhaps most important) more suggestive of computational approaches.

6.1 Definitions: Norm, Distance, Angle, Inner Product, and Orthogonality Certain definitions from plane and solid geometry can be extended in a natural way to an arbitrary linear space and can be otherwise generalized, as is now discussed. Let us begin by reviewing the usual definitions in the familiar settings of R2 and R3 . a. Usual definitions in R2 and R3 Each point in the plane can be identified by its coordinates (relative to a horizontal line and a vertical line—known as the coordinate axes—labeled so that their point of intersection—known as the origin—has coordinates of zero). A vector x D .x1 ; x2 /0 in R2 can be regarded as representing a point in the plane; namely, the point whose coordinates are x1 and x2 . Similarly, each point in 3-dimensional

56


space can be identified by its coordinates (relative to 3 mutually perpendicular axes whose intersection, called the origin, has coordinates of zero); and a vector x D .x1 ; x2 ; x3 /0 in R3 can be regarded as representing the point in space whose coordinates are x1 ; x2 , and x3 . (For definiteness, it is supposed that R2 D R21 and R3 D R31 , so that the members of R2 and R3 are column vectors rather than row vectors.) In addition to representing a point, a vector in R2 or R3 is also used to represent the directed line segment that starts at the origin and ends at that point. The intended usage can be ascertained from the context. The usual definition of the norm kxk (also known as the length or magnitude) of a vector x D .x1 ; x2 /0 in R2 is kxk D .x12 C x22 /1=2 : The distance ı.x; y/ between two vectors x D .x1 ; x2 /0 and y D .y1 ; y2 /0 in R2 is usually taken to be ı.x; y/ D kx yk D Œ.x1 y1 /2 C .x2 y2 /2 1=2 : Similarly, the norm kxk of a vector x D .x1 ; x2 ; x3 /0 in R3 is usually defined to be kxk D .x12 C x22 C x32 /1=2 ; and the distance ı.x; y/ between two vectors x D .x1 ; x2 ; x3 /0 and y D .y1 ; y2 ; y3 /0 (in R3 ) to be ı.x; y/ D kx yk D Œ.x1 y1 /2 C .x2 y2 /2 C .x3 y3 /2 1=2 : Clearly, for any vector x in R2 or R3 , kxk D .x0 x/1=2 ; and kxk D ı.x; 0/, that is, the norm of x is the distance between x and 0. Consider now the angle that two nonnull vectors x and y in R2 or R3 form with each other. There are actually two such angles, as depicted in Figure 6.1. The first angle 1 is that determined by rotating the directed line segment x in a counterclockwise direction (about the origin and within the plane containing x and y) until the directed line segment y is encountered. The second angle 2 D 2 1 is that determined by rotating y in a counterclockwise direction until x is encountered. (Unless otherwise indicated, angles are to be expressed in radians.) Let us arbitrarily define the angle between x and y to be the smaller of the two angles 1 and 2 (which is the usual convention). Thus, D 1 ; D 2 D 2 1 ; Clearly, 0 .

if 0 1 ; if < 1 < 2:

6.1 Definitions: Norm, Distance, Angle, Inner Product, and Orthogonality

57

FIGURE 6.1: The angles formed by two vectors in R2 or R3 with each other.

Let us now consider the cosine of the angle between two nonnull vectors x and y in R2 or R3 . Applying a well-known trigonometric identity, called the law of cosines, to the triangle whose vertices are 0, x and y, we obtain the equality kx yk2 D kxk2 C kyk2 2kxkkyk cos :

(1.1)

Then, substituting the expressions kx yk2 D .x y/0 .x y/ D x0 x 2x0 y C y 0 y; kxk2 D x0 x;

kyk2 D y 0 y

in equality (1.1) and canceling terms common to both sides, we find that 2x0 y D 2.x0 x/1=2 .y 0 y/1=2 cos and hence that cos D

x0 y .x0 x/1=2 .y 0 y/1=2

:

The usual definition of the inner (or dot) product x y of two vectors x D .x1 ; x2 /0 and y D .y1 ; y2 /0 in R2 is x y D x1 y1 C x2 y2

(1.2)

and that of two vectors x D .x1 ; x2 ; x3 /0 and y D .y1 ; y2 ; y3 /0 in R3 is x y D x1 y1 C x2 y2 C x3 y3 : Clearly, for any two vectors x and y in R2 or R3 ,

(1.3)

58


x y D x0 y: Note that the norm of a vector x in R2 or R3 is expressible as kxk D .x x/1=2 ; and the cosine of the angle between two nonnull vectors x and y in R2 or R3 is expressible as xy xy cos D : D 1=2 kxk kyk .x x/ .y y/1=2 Two vectors x and y in R2 or R3 are said to be orthogonal (or perpendicular) if x y D 0. Clearly, nonnull vectors x and y are orthogonal if and only if the angle between them is =2 (90ı ) or, equivalently, the cosine of that angle is 0. b. General definitions The usual definitions of norm, distance, angle, inner product, and orthogonality can be extended, in a natural way, from R2 and R3 to an arbitrary linear space and can be otherwise generalized, though—except for linear spaces of 2- or 3-dimensional row or column vectors—the extended definitions have no obvious geometrical interpretations. To do so, it suffices to extend and generalize the definition of inner product. The other definitions can then be extended and generalized by supposing that the concepts of norm, distance, angle, and orthogonality are related to each other and to the concept of inner product in the same way as (under the usual definitions) in R2 and R3 . The usual definition of the inner (or dot) product x y of two n-dimensional (column) vectors x D .x1 ; x2 ; : : : ; xn /0 and y D .y1 ; y2 ; : : : ; yn /0 in Rn , or, more generally, in a subspace of Rn , is x y D x0 y D x1 y1 C x2 y2 C : : : C xn yn :

(1.4)

Formula (1.4) is the natural extension of formulas (1.2) and (1.3) for the usual inner product of two vectors in R2 or R3 . More generally, the usual definition of the inner (or dot) product A B of two m n matrices A and B in a linear space V of m n matrices is A B D tr.A0 B/

(1.5)

or, equivalently [in light of result (5.2.8)], A B D tr.AB0 /:

(1.6)

In the special case of two n-dimensional row vectors x0 D .x1 ; x2 ; : : : ; xn / and y 0 D .y1 ; y2 ; : : : ; yn /, formula (1.6) for the usual inner product simplifies to x0 y 0 D x0 y D x1 y1 C x2 y2 C C xn yn ;

6.1 Definitions: Norm, Distance, Angle, Inner Product, and Orthogonality

59

which is the same as formula (1.4) for the usual inner product of the corresponding column vectors x and y. The usual inner product A B of two matrices A and B in a linear space V of m n matrices can be regarded as the value assigned to A and B by a function whose domain consists of all (ordered) pairs of matrices in V. As is evident from results (5.1.3), (5.1.4), (5.2.3), and (5.2.5) and from Lemma 5.3.1, this function has the following four basic properties: (1) A B D B A; (2) A A 0, with equality holding if and only if A D 0; (3) .kA/ B D k.A B/; (4) .A C B/ C D .A C/ C .B C/ (where A, B, and C represent arbitrary matrices in V and k represents an arbitrary scalar). It is sometimes useful to adopt a different definition for the inner product. Subsequently, we define the inner product of two matrices A and B in a linear space V (of m n matrices) to be the value A B assigned to A and B by any designated function f (whose domain consists of all pairs of matrices in V) having Properties (1)–(4). The term inner product is applied to the function f , as well as to its values. When f is chosen to be the function whose values are given by formula (1.5) or (1.6), we obtain the usual inner product. Other properties of the inner product can be derived from the four basic properties. We find, in particular, that (for any A in V) 0 A D 0;

(1.7)

as is evident from Property (3) upon observing that 0 A D .0 A/ A D 0 .A A/ D 0: Further (for any matrices A1 ; : : : ; Ak , and B in V and any scalars x1 ; : : : ; xk ), .x1 A1 C C xk Ak / B D x1 .A1 B/ C C xk .Ak B/;

(1.8)

as can be readily verified by the repeated application of Properties (3) and (4). Note that if a function f qualifies as an inner product for a linear space V of m n matrices, then f also qualifies as an inner product for any subspace of V— or, more precisely, the function obtained by restricting the domain of f (which consists of pairs of matrices in V) to pairs of matrices in the subspace qualifies as an inner product. In fact, if f is the designated inner product for a linear space V, then (unless otherwise indicated) it is implicitly assumed that the same function f is the designated inner product for any subspace of V. The norm kAk of a matrix A D faij g in a linear space V of m n matrices is kAk D .A A/1=2 ;

60


which, in the case of the usual inner product, gives m X n 1=2 1=2 X 2 kAk D tr.A0 A/ D aij :

(1.9)

iD1 j D1

If V is a linear space of m-dimensional column vectors, then, letting x D fxi g represent any vector in V, formula (1.9) can be restated as 2 1=2 kxk D .x0 x/1=2 D .x12 C C xm / :

(1.10)

The norm defined by formula (1.10) or more generally by formula (1.9) is called the usual norm. Letting A represent an arbitrary matrix in a linear space V and k an arbitrary scalar, it is clear that kAk > 0; D 0;

if A ¤ 0; if A D 0;

(1.11a) (1.11b)

and that kkAk D jkjkAk:

(1.12)

As a special case of equality (1.12) (that where k D 1), we have that kAk D kAk:

(1.13)

The distance ı.A; B/ between two matrices A and B in a linear space V is ı.A; B/ D kA Bk:

(1.14)

If V is a linear space of n-dimensional column vectors, then, letting x D fxi g represent any vector in V, formula (1.14) simplifies, in the case of the usual norm, to ı.x; y/ D Œ.x y/0 .x y/1=2 D Œ.x1 y1 /2 C C .xn yn /2 1=2 : (1.15) For two nonnull vectors x D fxi g and y D fyi g in a linear space V of ndimensional column vectors, the angle (0 ) between x and y is defined (indirectly in terms of its cosine) by cos D

xy kxk kyk

or equivalently, in the case of the usual inner product and norm, by cos D

.x12

x1 y1 C C xn yn : C C xn2 /1=2 .y12 C C yn2 /1=2

More generally, the angle (0 ) between two nonnull matrices A and B in a linear space V of m n matrices is defined by

6.2 Orthogonal and Orthonormal Sets

cos D

61

AB : kAk kBk

The Schwarz inequality, which is to be introduced in Section 6.3, implies that 1

AB 1 kAk kBk

(1.16)

and hence insures that this angle is well-defined. Two vectors x and y in a linear space V of n-dimensional column vectors are said to be orthogonal if x y D 0. More generally, two matrices A and B in a linear space V are said to be orthogonal if A B D 0. The statement that two matrices A and B are orthogonal is sometimes abbreviated to A ? B. Clearly, two nonnull matrices are orthogonal if and only if the angle between them is =2 or, equivalently, the cosine of that angle is 0. Whether two matrices are orthogonal depends on the choice of inner product; that is, two matrices that are orthogonal with respect to one inner product (one choice for the function f ) may not be orthogonal with respect to another.

6.2 Orthogonal and Orthonormal Sets A finite set of matrices in a linear space V is said to be orthogonal if every pair of matrices in the set are orthogonal. Thus, the empty set and any set containing only one matrix are orthogonal sets, and a finite set {A1 ; : : : ; Ak } of two or more matrices in V is an orthogonal set if Ai Aj D 0 for j ¤ i D 1; : : : ; k. A finite set of matrices in V is said to be orthonormal if the set is orthogonal and if the norm of every matrix in the set equals 1. In the special case of a set of (row or column) vectors, the expression “set of orthogonal (or orthonormal) vectors,” or simply “orthogonal (or orthonormal) vectors,” is often used in lieu of the technically more correct expression “orthogonal (or orthonormal) set of vectors.” Note that if A is any nonnull matrix in a linear space V, then the norm of the matrix (2.1) B D kAk1 A equals 1. The expression “normalizing a (nonnull) matrix” is often used in referring to the conversion—via formula (2.1)—of the matrix into a matrix whose norm is 1. If {A1 ; : : : ; Ak } is an orthogonal set of nonnull matrices in a linear space V, then the matrices B1 D kA1 k1 A1 ; : : : ; Bk D kAk k1 Ak form an orthonormal set. Thus, any orthogonal set of nonnull matrices can be converted into an orthonormal set by normalizing each of the matrices. There is a relationship between orthogonality and linear independence, which is described in the following lemma.

62


Lemma 6.2.1. An orthogonal set of nonnull matrices is linearly independent. Proof. If the orthogonal set is the empty set, then the result is clearly true (since, by convention, the empty set is linearly independent). Suppose then that {A1 ; : : : ; Ak } is any nonempty orthogonal set of nonnull matrices. Let x1 ; : : : ; xk represent arbitrary scalars such that x1 A1 C C xk Ak D 0: For i D 1; : : : ; k, we find that 0 D 0 Ai D .x1 A1 C C xk Ak / Ai D x1 .A1 Ai / C C xk .Ak Ai / D xi .Ai Ai /; implying (since Ai is nonnull) that xi D 0. We conclude that the set {A1 ; : : : ; Ak } is linearly independent. Q.E.D. Note that Lemma 6.2.1 implies in particular that any orthonormal set of matrices is linearly independent. Note also that the converse of Lemma 6.2.1 is not necessarily true; that is, a linearly independent set is not necessarily orthogonal. For example, the set consisting of the two 2-dimensional row vectors .1; 0/ and .1; 1/ is linearly independent but is not orthogonal (with respect to the usual inner product).

6.3 Schwarz Inequality The following theorem gives a famous inequality, known as the Schwarz inequality. Theorem 6.3.1 (Schwarz inequality). For any two matrices A and B in a linear space V, (3.1) jA Bj kAk kBk; with equality holding if and only if B D 0 or A D kB for some scalar k. Proof. Suppose first that kBk D 0 or equivalently that B D 0. Then, jA Bj D 0 D kAk kBk, and inequality (3.1) holds as an equality. Now, suppose that kBk > 0. Then (repeatedly making use of the basic properties of inner products), we find that, for any scalars x and y, 0 .xA yB/ .xA yB/ D x 2 kAk2 2xy.A B/ C y 2 kBk2 : Choosing x D kBk and y D .A B/=kBk, we obtain the inequality

This inequality is also known as the Cauchy-Schwarz (or Cauchy-BunyakovskySchwarz) inequality.

6.4 Orthonormal Bases

63

0 kBk2 kAk2 .A B/2 ; which is equivalent to the inequality .A B/2 kAk2 kBk2 and hence to inequality (3.1). If A D kB, then kAk D jkjkBk, so that jA Bj D jkj.B B/ D jkjkBk2 D kAk kBk: Conversely, suppose that jA Bj D kAk kBk. Then, letting C D kBk2 A .A B/B, we find that C C D 0, implying that C D 0, that is, that kBk2 A D .A B/B: We conclude that either B D 0 or A D Œ.A B/=kBk2 B: Q.E.D. As a special case of Theorem 6.3.1, we have that, for any two n-dimensional column vectors x and y, jx0 yj .x0 x/1=2 .y 0 y/1=2 ;

(3.2)

with equality holding if and only if y D 0 or x D ky for some scalar k. Note that the result of Theorem 6.3.1 can be reexpressed as follows. Corollary 6.3.2. Let V represent a linear space of m n matrices. Then, for any matrix B in V, jA Bj max D kBk: fA2V W A¤0g kAk Moreover, if B D 0, the maximum is attained at every nonnull A 2 V; and if B ¤ 0, the maximum is attained at every nonnull A 2 V that is proportional to B.

6.4 Orthonormal Bases An orthonormal basis for a linear space V is an orthonormal set of matrices in V that is a basis for V. The technique of Gram-Schmidt orthogonalization, which is described in Subsection a below, is used in Subsection b to establish the existence of an orthonormal basis.

64


a. Gram-Schmidt orthogonalization A nonempty linearly independent set of matrices can be transformed into an orthogonal or orthonormal set of matrices by applying the results of the following theorem and corollary. Theorem 6.4.1. Let {A1 ; : : : ; Ak } represent a nonempty linearly independent set of matrices in a linear space V. Then, there exist unique scalars xij (i < j D 1; : : : ; k) such that the set comprising the k matrices B1 D A 1 ; B2 D A2 x12 B1 ; :: : Bj D Aj xj 1;j Bj 1 x1j B1 ; :: : Bk D Ak xk1;k Bk1 x1k B1 is orthogonal. Further, B1 ; B2 ; : : : ; Bk are nonnull, and (for i < j D 1; : : : ; k) xij D

Aj Bi : Bi Bi

(4.1)

Proof. The proof is by mathematical induction. The theorem is obviously true for k D 1. Suppose now that the theorem is true for a set of k 1 linearly independent matrices. Then, there exist unique scalars xij (i < j D 1; : : : ; k 1) such that the set comprising the k 1 matrices B1 ; : : : ; Bk1 is orthogonal. Further, B1 ; : : : ; Bk1 are nonnull, and (for i < j D 1; : : : ; k 1) xij D

Aj Bi : Bi Bi

And, for i D 1; : : : ; k 1, we find that Bk Bi D Ak Bi xk1;k .Bk1 Bi / x1k .B1 Bi / D Ak Bi xik .Bi Bi / and thus that Bk Bi D 0 if and only if Ak Bi xik .Bi Bi / D 0 or, equivalently (since Bi is nonnull), if and only if xik D

Ak Bi : Bi Bi

We conclude that there exist unique scalars xij (i < j D 1; : : : ; k) such that the set comprising the k matrices B1 ; : : : ; Bk is orthogonal. We further conclude that B1 ; : : : ; Bk are nonnull and that (for i < j D 1; : : : ; k) xij is given by formula (4.1). (The matrix Bk D Ak xk1;k Bk1 x1k B1 is nonnull since, by repeated substitution, it can be rewritten as a nontrivial linear combination of the Q.E.D. linearly independent matrices A1 ; : : : ; Ak .)


65

Corollary 6.4.2. Let {A1 ; : : : ; Ak } represent a nonempty linearly independent set of matrices in a linear space V. Then, the set comprising the k matrices C1 D kB1 k1 B1 ; : : : ; Ck D kBk k1 Bk ; obtained by normalizing the k matrices B1 ; : : : ; Bk defined in Theorem 6.4.1, is orthonormal. Corollary 6.4.2 is an essentially immediate consequence of Theorem 6.4.1. The process of constructing an orthonormal set {C1 ; : : : ; Ck } from a linearly independent set {A1 ; : : : ; Ak } in accordance with the formulas of Theorem 6.4.1 and Corollary 6.4.2 is known as Gram-Schmidt orthogonalization. b. Existence of orthonormal bases and a basic property of orthonormal bases According to Theorem 4.3.4, every linear space (of m n matrices) has a basis. The basis of the linear space {0}, whose only member is the m n null matrix 0, is the empty set, which is an orthonormal set. The basis of any other linear space V is a nonempty (linearly independent) set, say {A1 ; : : : ; Ar }. By applying Gram-Schmidt orthogonalization to the set {A1 ; : : : ; Ar }, we can construct an orthonormal set {C1 ; : : : ; Cr } of r matrices in V. It follows from Lemma 6.2.1 that this set is linearly independent and hence—in light of Theorem 4.3.9— that it is a basis for V. Thus, we have the following theorem. Theorem 6.4.3. Every linear space (of m n matrices) has an orthonormal basis. The following theorem describes a basic property of orthonormal bases. Theorem 6.4.4. Let A1 ; A2 ; : : : ; Ar represent matrices that form an orthonormal basis for a linear space V. Then, for any matrix A in V, A D .A A1 /A1 C .A A2 /A2 C C .A Ar /Ar : Proof. There exist (unique) scalars x1 ; x2 ; : : : ; xr such that A D And, for j D 1; : : : ; r, A Aj D

r X iD1

Thus, A D

Pr

iD1

.A Ai /Ai .

Pr

iD1

xi Ai .

r X xi Ai Aj D xi .Ai Aj / D xj : iD1

Q.E.D.

c. Inclusion of any particular orthonormal set in an orthonormal basis Suppose that {A1 ; : : : ; Ar } is an orthonormal set of matrices in a k-dimensional linear space V. Then, in light of Lemma 6.2.1, the matrices A1 ; : : : ; Ar are linearly independent, and it follows from Theorem 4.3.12 that there exists a basis for

66


V that includes A1 ; : : : ; Ar ; that is, there exist k r additional matrices, say ArC1 ; : : : ; Ak , such that the set {A1 ; : : : ; Ar ; ArC1 ; : : : ; Ak } is a basis for V. Does there exist an orthonormal basis for V that includes A1 ; : : : ; Ar ? This question can be answered (in the affirmative) by observing that an orthonormal set {C1 ; : : : ; Cr ; CrC1 ; : : : ; Ck } of k matrices in V can be constructed by applying Gram-Schmidt orthogonalization to the set {A1 ; : : : ; Ar ; ArC1 ; : : : ; Ak }. Since the subset comprising the first r of the matrices in the set {A1 ; : : : ; Ar ; ArC1 ; : : : ; Ak } is orthonormal, the matrices in this subset are not altered by the Gram-Schmidt orthogonalization; that is, Ci D A i

.i D 1; : : : ; r/:

Moreover, it follows from Lemma 6.2.1 and Theorem 4.3.9 that the orthonormal set {C1 ; : : : ; Cr ; CrC1 ; : : : ; Ck } is a basis for V. Thus, we have the following theorem. Theorem 6.4.5. For any orthonormal set S of r matrices in a k-dimensional linear space V, there exists an orthonormal basis for V that includes all r of the matrices in S (and k r additional matrices). d. QR decomposition Let A represent an m k matrix of full column rank, that is, of rank k. And denote the first, : : : ; kth columns of A by a1 ; : : : ; ak , respectively. Then, according to Theorem 6.4.1, there exist unique scalars xij (i < j D 1; : : : ; k) such that the k column vectors b1 ; : : : ; bk defined recursively by the equalities b1 D a1 ; b2 D a2 x12 b1 ; :: : bj D aj xj 1;j bj 1 x1j b1 ; :: : bk D ak xk1;k bk1 x1k b1 or, equivalently, by the equalities a1 D b1 ; a2 D b2 C x12 b1 ; :: : aj D bj C xj 1;j bj 1 C C x1j b1 ; :: : ak D bk C xk1;k bk1 C C x1k b1 form an orthogonal set. Further, b1 ; : : : ; bk are nonnull, and (for i < j D 1; : : : ; k)


xij D

67

aj bi : bi bi

Now, let B represent the m k matrix whose first, : : : ; kth columns are b1 ; : : : ; bk , respectively, and let X represent the k k unit upper triangular matrix whose ij th element is (for i < j D 1; : : : ; k) xij . Then, A D BX;

(4.2)

as is evident upon observing that the first column of BX is b1 and that (for j D 2; : : : ; k) the j th column of BX is bj C xj 1;j bj 1 C C x1j b1 . Thus, A can be expressed as the product of an m k matrix with (nonnull) orthogonal columns and of a k k unit upper triangular matrix. Moreover, this decomposition or factorization of A is unique; that is, if B is an m k matrix with orthogonal columns and X is a k k unit upper triangular matrix such that A D B X , then B D B and X D X. To see this, denote the first, : : : ; kth columns of B by b1 ; : : : ; bk , respectively, denote the ij th element of X by xij , and observe that (by definition) a1 D b1 and (for j D 2; : : : ; k) aj D bj C xj1;j bj1 C C x1j b1 ;

or, equivalently, that b1 D a1 and (for j D 2; : : : ; k) bj D aj xj1;j bj1 x1j b1 :

Then, it follows from Theorem 6.4.1 that (for i < j D 1; : : : ; k) xij D xij and, as a consequence, that (for all j ) bj D bj , so that X D X and B D B. A variation on decomposition (4.2) can be obtained by defining Q D BD, where D D diag.kb1 k1 ; : : : ; kbk k1 /, and R D EX, where E D diag.kb1 k; : : : ; kbk k/; or equivalently by defining Q to be the m k matrix with j th column qj D kbj k1 bj and R D frij g to be the k k matrix with 8 ˆ i , rij D kbi k; for j D i , ˆ : 0; for j < i . Then, clearly, the columns of Q are orthonormal (they are the orthonormal vectors obtained by applying Gram-Schmidt orthogonalization to the columns of A), R is an upper triangular matrix with positive diagonal elements, and A D QR:

(4.3)

Moreover, decomposition (4.3) is unique; that is, if Q is an m k matrix with orthonormal columns q1 ; : : : ; qk , respectively, and if R D frij g is a k k upper triangular matrix (with positive diagonal elements) such that A D Q R , then Q D Q and R D R. To see this, let B D Q E , where E D 1 1 diag.r11 ; : : : ; rkk /, and let X D D R , where D D diag.r11 ; : : : ; rkk /;

68


or, equivalently, let B represent the m k matrix with j th column bj D rjj qj , 1 and let X represent the k k matrix with ij th element xij D ri i rij . Then, the columns of B are orthogonal, X is unit upper triangular, and A D B X . Thus, in light of the uniqueness of decomposition (4.2), B D B and X D X, so that 1 qj D bj D bj D kbj kqj (for all j ) and ri1 rjj i rij D xij D xij D kbi k rij (for all j and i < j ), implying that

1=2 2 0 1=2 rjj D rjj qj qj D kbj k2 qj0 qj D kbj k .D rjj /

(for all j ), and hence that qj D qj (for all j ) and rij D rij (for all j and i j ), or, equivalently, that Q D Q and R D R. Note that decompositions (4.2) and (4.3) depend on the choice of inner product (with respect to which the columns of B and the columns of Q form orthogonal or orthonormal sets). Let us refer to the version of decomposition (4.3) corresponding to the usual inner product as the QR decomposition—for that version, Q0 Q D Ik . This usage of the term QR decomposition conforms to that of Stewart (1973, p. 214) and Lancaster and Tismenetsky (1985, p. 111), but differs somewhat from that of Golub and Van Loan (1989, sec. 5.2), who refer to decomposiiton (4.3) as the “skinny” QR decomposition. In principle, the QR decomposition of the matrix A could be computed by Gram-Schmidt orthogonalization. However, as a computational algorithm, GramSchmidt orthogonalization is numerically unstable. Numerically more stable algorithms for computing the QR decomposition are discussed by Golub and Van Loan (1989, sec. 5.2)—one such algorithm is a variant on Gram-Schmidt orthogonalization called modified Gram-Schmidt orthogonalization.

Exercises Section 6.3 1. Use the Schwarz inequality to show that, for any two matrices A and B in a linear space V, kA C Bk kAk C kBk; with equality holding if and only if B D 0 or A D kB for some nonnegative scalar k. (This inequality is known as the triangle inequality.) 2. Letting A, B, and C represent arbitrary matrices in a linear space V, show that (a) ı.B; A/ D ı.A; B/, that is, the distance between B and A is the same as that between A and B; (b) ı.A; B/ > 0; D 0;

if A ¤ B; if A D B;

Exercises

69

that is, the distance between any two matrices is greater than zero, unless the two matrices are identical, in which case the distance between them is zero; (c) ı.A; B/ ı.A; C/Cı.C; B/, that is, the distance between A and B is less than or equal to the sum of the distances between A and C and between C and B; (d) ı.A; B/ D ı.ACC; BCC/, that is, distance is unaffected by a translation of “axes.” [For Part (c), use the result of Exercise 1, i.e., the triangle inequality.] Section 6.4 3. Let w10 ; w20 , and w30 represent the three linearly independent 4-dimensional row vectors .6; 0; 2; 3/, .2; 4; 4; 2/, and .0; 5; 1; 2/, respectively, in the linear space R4 , and adopt the usual definition of inner product. (a) Use Gram-Schmidt orthogonalization to find an orthonormal basis for the linear space sp.w10 ; w20 ; w30 /. (b) Find an orthonormal basis for R4 that includes the three orthonormal vectors from Part (a). Do so by extending the results of the Gram-Schmidt orthogonalization [from Part (a)] to a fourth linearly independent row vector such as .0; 1; 0; 0/. 4. Let {A1 ; : : : ; Ak } represent a nonempty (possibly linearly dependent) set of matrices in a linear space V. (a) Generalize Theorem 6.4.1 by showing (1) that there exist scalars xij (i < j D 1; : : : ; k) such that the set comprising the k matrices B1 D A 1 ; B2 D A2 x12 B1 ; :: : Bj D Aj xj 1;j Bj 1 x1j B1 ; :: : Bk D Ak xk1;k Bk1 x1k B1 is orthogonal; (2) that, for j D 1; : : : ; k and for those i < j such that Bi is nonnull, xij is given uniquely by xij D

Aj Bi I Bi Bi

and (3) that the number of nonnull matrices among B1 ; : : : ; Bk equals dimŒsp.A1 ; : : : ; Ak /. (b) Describe a procedure for constructing an orthonormal basis for sp.A1 ; : : : ; Ak /.

70


5. Let A represent an m k matrix of rank r (where r is possibly less than k). Generalize result (4.3) by using the results of Exercise 4 to obtain a decomposition of the form A D QR1 , where Q is an m r matrix with orthonormal columns and R1 is an r k submatrix whose rows are the r nonnull rows of a k k upper triangular matrix R having r positive diagonal elements and k r null rows.

7 Linear Systems: Consistency and Compatibility

In many instances, the solution to a problem encountered in statistics (or in some other discipline) can be reduced to a problem of solving a system of linear equations. For example, the problem of obtaining the least squares estimates of the parameters in a linear statistical model can be reduced to the problem of solving a system of linear equations called the normal equations or (if the parameters are subject to linear constraints) a system of linear equations sometimes called the constrained normal equations (e.g., Searle 1971). The focus in this chapter is on questions about the existence of (one or more) solutions to a system of linear equations—questions about the solutions themselves (when solutions exist) are deferred until Chapter 11. Such questions of existence arise in the theory of linear statistical models in determining which parametric functions are estimable (i.e., which parametric functions can be estimated from the data) and in the design of experiments (e.g., Searle 1971). The results of Sections 7.2 and 7.3 are general (i.e., applicable to any system of linear equations), while those of Section 7.4 are specific to systems of linear equations of the form of the normal equations or constrained normal equations—some of the terminology employed in discussing systems of linear equations is introduced in Section 7.1.

7.1 Some Basic Terminology Consider a set of m equations of the general form a11 x1 C a12 x2 C C a1n xn D b1 :: : am1 x1 C am2 x2 C C amn xn D bm ; where a11 ; a12 ; : : : ; a1n ; : : : ; am1 ; am2 ; : : : ; amn and b1 ; : : : ; bm represent fixed scalars, and x1 ; x2 ; : : : ; xn are scalar-valued unknowns or variables. Each of these equations is linear in the unknowns x1 ; x2 ; : : : ; xn . Collectively, these equations

72


are called a system of linear equations (in unknowns x1 ; x2 ; : : : ; xn ) or simply a linear system (in x1 ; x2 ; : : : ; xn ). The linear system can be rewritten in matrix form as Ax D b;

(1.1)

where A is the m n matrix whose ij th element is aij (i D 1; : : : ; m; j D 1; : : : ;n), b D .b1 ; : : : ; bm /0 , and x D .x1 ; x2 ; : : : ; xn /0 . The matrix A is called the coefficient matrix of the linear system, and b is called the right side. Any value of the vector x of unknowns that satisfies Ax D b is called a solution to the linear system, and the process of finding a solution (when one exists) is called solving the linear system. There may be occasion to solve the linear system for more than one right side, that is, to solve each of p linear systems Axk D bk .k D 1; : : : ; p/

(1.2)

(in vectors x1 ; : : : ; xp , respectively, of unknowns) that have the same coefficient matrix A but right sides b1 ; : : : ; bp , respectively, that may differ. By forming an n p matrix X whose first, : : : ; pth columns are x1 ; : : : ; xp , respectively, and an m p matrix B whose columns are b1 ; : : : ; bp , respectively, the p linear systems (1.2) can be rewritten collectively as AX D B:

(1.3)

As in the special case (1.1) where p D 1, AX D B is called a linear system (in X), A and B are called the coefficient matrix and the right side, respectively, and any value of X that satisfies AX D B is called a solution. In the special case AX D 0; where the right side B of linear system (1.3) is a null matrix, the linear system is said to be homogeneous. If B is nonnull, linear system (1.3) is said to be nonhomogeneous.

7.2 Consistency A linear system is said to be consistent if it has one or more solutions. If no solution exists, the linear system is said to be inconsistent. Every homogeneous linear system is consistent—one solution to a homogeneous linear system is the null matrix (of appropriate dimensions). A nonhomogeneous linear system may be either consistent or inconsistent. Some necessary and sufficient conditions for a linear system to be consistent are given by the following theorem. Theorem 7.2.1. Each of the following conditions is necessary and sufficient for a linear system AX D B (in X) to be consistent:

7.3 Compatibility

73

C.B/ C.A/; every column of B belongs to C.A/; C.A; B/ D C.A/; rank.A; B/ D rank.A/. Proof. That Condition (1) is necessary and sufficient for the consistency of AX D B is an immediate consequence of Lemma 4.2.2. Further, it follows from Lemma 4.2.1 that Condition (2) is equivalent to Condition (1), from Lemma 4.5.1 that Condition (3) is equivalent to Condition (1), and from Corollary 4.5.2 that Condition (4) is equivalent to Condition (1). Thus, each of Conditions (2) through (4) is also necessary and sufficient for the consistency of AX D B. Q.E.D. A sufficient (but in general not a necessary) condition for the consistency of a linear system is given by the following theorem.

(1) (2) (3) (4)

Theorem 7.2.2. If the coefficient matrix A of a linear system AX D B (in X) has full row rank, then AX D B is consistent. Proof. Let m represent the number of rows in A, and suppose that A has full row rank or, equivalently, that rank.A/ D m. Then, making use of Lemma 4.4.3, we find that rank.A; B/ m D rank.A/: Moreover, since C.A/ C.A; B/, it follows from Theorem 4.4.4 that rank.A; B/ rank.A/. Thus, rank.A; B/ D rank.A/, and the consistency of AX D B follows from Theorem 7.2.1. Q.E.D.

7.3 Compatibility A linear system AX D B (in X) is said to be compatible if every linear relationship that exists among the rows of A also exists among the rows of B. That is, AX D B is compatible if k0 B D 0 for every row vector k0 (of appropriate dimension) such that k0 A D 0. Compatibility is equivalent to consistency as indicated by the following theorem. Theorem 7.3.1. A linear system AX D B (in X) is consistent if and only if it is compatible. Proof. Suppose that the linear system AX D B is consistent. Then, there exists a matrix X such that AX D B, and, for every row vector k0 such that k0 A D 0, we have that k0 B D k0 AX D 0: Thus, AX D B is compatible. To prove the converse, let a01 ; : : : ; a0m and b01 ; : : : ; b0m represent the rows of A and B, respectively, and observe that if a value of X is a solution to all m of the equations a0i X D b0i .i D 1; : : : ; m/;

74


then it is a solution to AX D B. According to Theorem 4.3.11, there exists a subset S D fk1 ; : : : ; kr g of the first m positive integers such that the set fa0k1 ; : : : ; a0kr g is a basis for R.A/. Taking A1 to be a matrix whose rows are a0k1 ; : : : ; a0kr and B1 to be a matrix whose rows are b0k1 ; : : : ; b0kr , it follows from Theorem 7.2.2 that the linear system A1 X D B1 (in X) is consistent. Thus, there exists a matrix X such that A1 X D B1 or, equivalently, such that a0kj X D b0kj

.j D 1; : : : ; r/:

To complete the proof, it suffices to show that if AX D B is compatible, then a0i X D b0i for i … S . For each i … S , there exist scalars i1 ; : : : ; ir such that a0i D

r X j D1

or, equivalently, such that a0i

r X j D1

ij a0kj

ij a0kj D 0:

If AX D B is compatible, then, for each i … S , b0i

r X j D1

or, equivalently, b0i D

ij b0kj D 0

r X j D1

in which case a0i X D

r X j D1

ij b0kj ;

ij a0kj X D

r X j D1

ij b0kj D b0i : Q.E.D.

7.4 Linear Systems of the Form A0 AX D A0 B a. A basic result Theorem 7.3.1 can be used to establish the following result. Theorem 7.4.1. For any mn matrix A and mp matrix B, the linear system A0 AX D A0 B (in X) is consistent.

7.4 Linear Systems of the Form A0 AX D A0 B

75

Proof. To prove Theorem 7.4.1, it suffices (in light of Theorem 7.3.1) to show that the linear system A0 AX D A0 B is compatible. Let k0 represent any row vector such that k0 A0 A D 0. Then, according to Corollary 5.3.3, we have that k0 A0 D 0 and hence that k0 A0 B D 0. Thus, the Q.E.D. linear system A0 AX D A0 B is compatible. In the special case where B D I, Theorem 7.4.1 reduces to the following result. Corollary 7.4.2. For any matrix A, the linear system A0 AX D A0 (in X) is consistent. b. Some results on row and column spaces and on ranks Corollary 7.4.2 can be used to establish the following result. Theorem 7.4.3. For any m n matrix A and n s matrix T , C.T 0 A0 A/ D C.T 0 A0 / and R.A0 AT / D R.AT /. Proof. According to Corollary 7.4.2, there exists a matrix X such that A0 AX D 0 A . Thus, making use of Corollary 4.2.3, we find that C.T 0 A0 / D C.T 0 A0 AX/ C.T 0 A0 A/ and also that C.T 0 A0 A/ C.T 0 A0 /, implying that C.T 0 A0 A/ D C.T 0 A0 / and further (in light of Lemma 4.2.5) that R.A0 AT / D R.AT /. Q.E.D. One implication of Theorem 7.4.3 is given by the following corollary. Corollary 7.4.4. For any mn matrix A and ns matrix T , rank.T 0 A0 A/ D rank.T 0 A0 / and rank.A0 AT / D rank.AT /. In the special case where T D I, the results of Theorem 7.4.3 and Corollary 7.4.4 reduce to the results given by the following corollary. Corollary 7.4.5. For any matrix A, C.A0 A/ D C.A0 /, R.A0 A/ D R.A/, and rank.A0 A/ D rank.A/. One implication of Corollary 7.4.5 is described in the following, additional corollary. Corollary 7.4.6. For any matrix A of full column rank, A0 A is nonsingular. In light of Corollary 4.5.6, the following theorem is an immediate consequence of Theorem 7.4.3 (and Corollary 7.4.4). Theorem 7.4.7. Let A represent an m n matrix, T an n s matrix, and K a q s matrix. Then, C.T 0 A0 A; K0 / D C.T 0 A0 ; K0 /; 0 AT A AT DR ; R K K

rank.T 0 A0 A; K0 / D rank.T 0 A0 ; K0 /; 0 A AT AT rank D rank : K K

c. Extensions The following theorem extends Theorem 7.4.1.

76


Theorem 7.4.8. For any m n matrix A, m p matrix B, and n q matrix L, and for any q p matrix C such that C.C/ C.L0 /, the linear system 0 0 AA L X AB D (4.1) Y L0 0 C (in X and Y ) is consistent. Proof. It suffices (in light of Theorem 7.3.1) to show that linear system (4.1) is compatible. Let k01 and k02 represent any n 1 and q 1 vectors such that 0 AA L .k01 ; k02 / D 0: L0 0 Then, implying that

k01 A0 A D k02 L0 ;

L0 k1 D 0;

(4.2)

.Ak1 /0 Ak1 D k01 A0 Ak1 D k02 L0 k1 D 0

and hence (in light of Corollary 5.3.2) that Ak1 D 0:

(4.3)

Moreover, there exists a matrix F such that C D L0 F , implying—in light of results (4.2) and (4.3)—that k02 C D k02 L0 F D k01 A0 AF D .Ak1 /0 AF D 0:

(4.4)

Together, results (4.3) and (4.4) imply that 0 AB .k01 ; k02 / D .Ak1 /0 B C k02 C D 0: C Thus, linear system (4.1) is compatible. Theorem 7.4.8 is generalized in the following corollary.

Q.E.D.

Corollary 7.4.9. For any m n matrix A, m p matrix B, n q matrix L, and q p matrix R, and for any q p matrix C such that C.C/ C.L0 /, the linear system 0 0 X A B C LR AA L D (4.5) Y L0 0 C (in X and Y ) is consistent. Proof. According to Theorem 7.4.8, the linear system 0 0 X AB AA L D Y L0 0 C X X . Then, clearly, the matrix is a (in X and Y ) has a solution, say Y Y C R solution to linear system (4.5). Thus, linear system (4.5) is consistent. Q.E.D.

Exercise

77

Exercise Section 7.4 1. (a) Let A represent an m n matrix, C an n q matrix, and B a q p matrix. Show that if rank.AC/ D rank.C/, then R.ACB/ D R.CB/ and

rank.ACB/ D rank.CB/

and that if rank.CB/ D rank.C/, then C.ACB/ D C.AC/ and

rank.ACB/ D rank.AC/;

thereby extending the results of Corollary 4.4.7, Theorem 7.4.3, and Corollary 7.4.4. (b) Let A and B represent mn matrices. (1) Show that if C is an r q matrix and D a q m matrix such that rank.CD/ D rank.D/, then CDA D CDB implies DA D DB, thereby extending the result of Part (1) of Corollary 5.3.3. {Hint. To show that DA D DB, it suffices to show that rankŒD.A B/ D 0:g (2) Similarly, show that if C is an n q matrix and D a q p matrix such that rank.CD/ D rank.C/, then ACD D BCD implies AC D BC, thereby extending the result of Part (2) of Corollary 5.3.3.

8 Inverse Matrices

Corresponding to any nonsingular matrix is another nonsingular matrix that is referred to as the inverse of the first matrix. Inverse matrices are introduced and defined in Section 8.1 and are discussed in the subsequent sections of this chapter. Inverse matrices are of interest because there is a close relationship between the solution to a linear system having a nonsingular coefficient matrix and the inverse of the coefficient matrix—this relationship is described in Section 8.2a. Moreover, there are many situations in statistics and in other disciplines that give rise to inverse matrices that are of interest in their own right.

8.1 Some Definitions and Basic Results A right inverse of an m n matrix A is an n m matrix R such that AR D Im : Similarly, a left inverse of an m n matrix A is an n m matrix L such that LA D In or, equivalently, such that

A 0 L 0 D In :

A matrix may or may not have a right or left inverse, as indicated by the following lemma. Lemma 8.1.1. An m n matrix A has a right inverse if and only if rank.A/ D m (i.e., if and only if A has full row rank) and has a left inverse if and only if rank.A/ D n (i.e., if and only if A has full column rank). Proof. If rank.A/ D m, then it follows from Theorem 7.2.2 that there exists a matrix R such that AR D Im , that is, that A has a right inverse. Conversely, if there exists a matrix R such that AR D Im , then

80

8 Inverse Matrices

rank.A/ rank.AR/ D rank.Im / D m; implying [since, according to Lemma 4.4.3, rank.A/ m] that rank.A/ D m. Thus, A has a right inverse if and only if rank.A/ D m. That A has a left inverse if and only if rank.A/ D n is evident upon observing that A has a left inverse if and only if A0 has a right inverse [and that rank.A0 / D rank.A/]. Q.E.D. As an almost immediate consequence of Lemma 8.1.1, we have the following corollary. Corollary 8.1.2. A matrix A has both a right inverse and a left inverse if and only if A is a (square) nonsingular matrix. If there exists a matrix B that is both a right and left inverse of a matrix A (so that AB D I and BA D I), then A is said to be invertible and B is referred to as an inverse of A. Only a (square) nonsingular matrix can be invertible, as is evident from Corollary 8.1.2. The following lemma and theorem include some basic results on the existence and uniqueness of inverse matrices. Lemma 8.1.3. If a square matrix A has a right or left inverse B, then A is nonsingular and B is an inverse of A. Proof. Suppose that A has a right inverse R. Then, it follows from Lemma 8.1.1 that A is nonsingular and further that A has a left inverse L. Observing that L D LI D LAR D IR D R and hence that RA D I, we conclude that R is an inverse of A. A similar argument can be used to show that if A has a left inverse L, then A is nonsingular and L is an inverse of A. Q.E.D. Theorem 8.1.4. A matrix is invertible if and only if it is a (square) nonsingular matrix. Further, any nonsingular matrix has a unique inverse B and has no right or left inverse other than B. Proof. Suppose that A is a nonsingular matrix. Then, it follows from Lemma 8.1.1 that A has a right inverse B and from Lemma 8.1.3 that B is an inverse of A. Thus, A is invertible. Moreover, for any inverse C of A, we find that C D CI D CAB D IB D B; implying that A has a unique inverse and further—in light of Lemma 8.1.3—that A has no right or left inverse other than B. That any invertible matrix is nonsingular is (as noted earlier) evident from Corollary 8.1.2. Q.E.D. The symbol A1 is used to denote the inverse of a nonsingular matrix A. By definition, AA1 D A1 A D I: For an n n nonsingular diagonal matrix D D fdi g,

8.2 Properties of Inverse Matrices

D 1 D diag.1=d1 ; 1=d2 ; : : : ; 1=dn /:

81

(1.1)

(Clearly, the diagonal elements of a nonsingular diagonal matrix are nonzero.) a a For any 2 2 nonsingular matrix A D 11 12 , a21 a22 A

1

a22 a12 ; D .1=k/ a21 a11

(1.2)

where k D a11 a22 a12 a21 , as is easily verified. [Necessarily, k ¤ 0. To see this, assume—for purposes of establishing a contradiction—the contrary (i.e., assume that k D 0). Then, if a11 D 0, we find that a12 a21 D 0, or, equivalently, that a12 D 0 or a21 D 0, and hence that A has a null row or column. If a11 ¤ 0, we find that a22 D a12 a21 =a11 and hence that (a21 ; a22 / D .a21 =a11 /.a11 ; a12 ), so that the second row of A is a scalar multiple of the first. Thus, in either case (a11 D 0 or a11 ¤ 0), the assumption that k D 0 leads to the conclusion that A is singular, thereby establishing the sought-after contradiction.]

8.2 Properties of Inverse Matrices a. Relationship of inverse matrices to the solution of linear systems There is an intimate relationship between the inverse A1 of a nonsingular matrix A and the solution of linear systems whose coefficient matrix is A. This relationship is described in the following theorem. Theorem 8.2.1. Let A represent any n n nonsingular matrix, G any n n matrix, and p any positive integer. Then, GB is a solution to a linear system AX D B (in X) for every n p matrix B if and only if G D A1 . Proof. If G D A1 , then, for every n p matrix B, A.GB/ D AA1 B D IB D B: Conversely, suppose that GB is a solution to AX D B (i.e., that AGB D B) for every n p matrix B and, in particular, for every n p matrix of the form B D .0; : : : ; 0; b; 0; : : : ; 0/: Then, AGb D b D Ib for every n 1 vector b. We conclude (on the basis of Lemma 2.3.2) that AG D I and hence (in light of Lemma 8.1.3) that G D A1 . Q.E.D. b. Some elementary properties For any nonsingular matrix A and any nonzero scalar k, kA is nonsingular and

82

8 Inverse Matrices

.kA/1 D .1=k/A1 ;

(2.1)

as is easily verified. In the special case k D 1, equality (2.1) reduces to .A/1 D A1 :

(2.2)

It is easy to show that, for any nonsingular matrix A, A0 is nonsingular, and .A0 /1 D .A1 /0 :

(2.3)

In the special case of a symmetric matrix A, equality (2.3) reduces to A1 D .A1 /0 :

(2.4)

Thus, the inverse of any nonsingular symmetric matrix is symmetric. The inverse A1 of an nn nonsingular matrix A is invertible, or, equivalently (in light of Theorem 8.1.4), (2.5) rank.A1 / D n; and

.A1 /1 D A;

(2.6)

that is, the inverse of A1 is A (as is evident from the definition of A1 ). For any two n n nonsingular matrices A and B, rank.AB/ D n;

(2.7)

.AB/1 D B1 A1 :

(2.8)

that is, AB is nonsingular, and

Results (2.7) and (2.8) can be easily verified by observing that ABB1 A1 D I (so that B1 A1 is a right inverse of AB) and applying Lemma 8.1.3. (If either or both of two n n matrices A and B are singular, then their product AB is singular, as is evident from Corollary 4.4.5.) Repeated application of results (2.7) and (2.8) leads to the conclusion that, for any k n n nonsingular matrices A1 ; A2 ; : : : ; Ak , rank.A1 A2 Ak / D n and

1 1 .A1 A2 Ak /1 D A1 k A2 A1 :

(2.9) (2.10)

8.3 Premultiplication or Postmultiplication by a Matrix of Full Column or Row Rank Lemma 8.1.1 is useful in investigating the effects of premultiplication or postmultiplication by a matrix having full column or row rank. In particular, it can be used to establish the following two lemmas.

8.3 Premultiplication or Postmultiplication by a Matrix of Full Column or Row Rank

83

Lemma 8.3.1. Let A and B represent m n matrices. Then, for any r m matrix C of full column rank (i.e., of rank m) and any n p matrix D of full row rank (i.e., of rank n), (1) CA D CB implies A D B, (2) AD D BD implies A D B, and (3) CAD D CBD implies A D B. Proof. Parts (1) and (2) are special cases of Part (3) (those where D D I and C D I, respectively). Thus, it suffices to prove Part (3). According to Lemma 8.1.1, C has a left inverse L and D a right inverse R. Consequently, if CAD D CBD, then A D IAI D LCADR D LCBDR D IBI D B: Q.E.D. Lemma 8.3.2. Let A represent an m n matrix and B an n p matrix. If A has full column rank, then R.AB/ D R.B/

and

rank.AB/ D rank.B/:

Similarly, if B has full row rank, then C.AB/ D C.A/

and

rank.AB/ D rank.A/:

Proof. It is clear from Corollary 4.2.3 that R.AB/ R.B/ and C.AB/ C.A/. If A has full column rank, then (according to Lemma 8.1.1) it has a left inverse L, implying that R.B/ D R.IB/ D R.LAB/ R.AB/ and hence that R.AB/ D R.B/ [which implies, in turn, that rank.AB/ D rank.B/]. Similarly, if B has full row rank, then it has a right inverse R, implying that C.A/ D C.ABR/ C.AB/ and hence that C.AB/ D C.A/ [and rank.AB/ D rank.A/]. Q.E.D. As an immediate consequence of Lemma 8.3.2, we have the following corollary. Corollary 8.3.3. If A is an n n nonsingular matrix, then, for any n p matrix B, R.AB/ D R.B/ and rank.AB/ D rank.B/: Similarly, if B is an n n nonsingular matrix, then, for any m n matrix A, C.AB/ D C.A/

and

rank.AB/ D rank.A/:

A further consequence of Lemma 8.3.2 is described in the following additional corollary. Corollary 8.3.4. Let A represent an m n nonnull matrix, and let B represent an m r matrix of full column rank r and T an r n matrix of full row rank r such that A D BT . Then, r D rank.A/. Proof (of Corollary 8.3.4). In light of Lemma 8.3.2, we have that rank.A/ D rank.BT / D rank.T / D r: Q.E.D.

84

8 Inverse Matrices

8.4 Orthogonal Matrices a. Definition and basic properties and results A (square) matrix A is said to be an orthogonal matrix if A0 A D AA0 D I or, equivalently, if A is nonsingular and A1 D A0 . To show that a (square) matrix A is orthogonal, it suffices (in light of Lemma 8.1.3) to demonstrate that A0 A D I or, alternatively, that AA0 D I. The 2 2 matrix p 1 1 .1= 2/ 1 1 is an example of an orthogonal matrix. More generally, for any angle , the 2 2 matrix cos sin sin cos is an orthogonal matrix. Further, the n n identity matrix In is an orthogonal matrix. Clearly, for an n n matrix A, A0 A D I if and only if the columns a1 ; : : : ; an of A are such that a0i aj D 1; for j D i D 1; : : : ; n; D 0; for j ¤ i D 1; : : : ; n: Thus, a square matrix is orthogonal if and only if its columns form an orthonormal (with respect to the usual inner product) set of vectors. Similarly, a square matrix is orthogonal if and only if its rows form an orthonormal set of vectors. Note that if A is an orthogonal matrix, then its transpose A0 is also orthogonal. If P and Q are both n n orthogonal matrices, then it follows from results (2.7) and (2.8) that P Q is nonsingular and that .P Q/1 D Q1 P 1 D Q0 P 0 D .P Q/0 :

(4.1)

Thus, the product of two n n orthogonal matrices is an (n n) orthogonal matrix. Repeated application of this result leads to the following extension. Lemma 8.4.1. If each of the matrices Q1 ; Q2 ; : : : ; Qk is an n n orthogonal matrix, then their product Q1 Q2 Qk is an (n n) orthogonal matrix. A further result on orthogonal matrices is given by the following lemma. Lemma 8.4.2. Let A represent an m n matrix. Then, when the norm is taken to be the usual norm, kP AQk D kAk (4.2) for any m m orthogonal matrix P and any n n orthogonal matrix Q.

8.4 Orthogonal Matrices

85

Proof. Making use of Lemma 5.2.1, we find that kP AQk2 D trŒ.P AQ/0 P AQ D tr.Q0 A0 P 0 P AQ/ D tr.Q0 A0 AQ/ D tr.A0 AQQ0 / D tr.A0 A/ D kAk2 : Q.E.D. Note that, as special cases of result (4.2) (those obtained by setting Q D In or P D Im ), we have that (for any m m orthogonal matrix P ) kP Ak D kAk and (for any n n orthogonal matrix Q) kAQk D kAk. b. Helmert matrix Let a0 D .a1 ; a2 ; : : : ; an / represent any row vector such that ai ¤ 0 (i D 1; 2; : : : ; n), that is, any row vector whose elements are all nonzero. Suppose that we require an n n orthogonal matrix, one row of which is proportional to a0 . In what follows, one such matrix P is derived. Let p01 ; : : : ; p0n represent the rows of P , and take the first row p01 to be the row of P that is proportional to a0 . Take the second row p02 to be proportional to the n-dimensional row vector .a1 ; a12 =a2 ; 0; 0; : : : ; 0/; the third row p03 proportional to Œa1 ; a2 ; .a12 C a22 /=a3 ; 0; 0; : : : ; 0; and more generally the second through nth rows p02 ; : : : ; p0n proportional to P 2 .k D 2; : : : ; n/; (4.3) .a1 ; a2 ; : : : ; ak1 ; k1 iD1 ai =ak ; 0; 0; : : : ; 0/ respectively. It is easy to confirm that the n 1 vectors (4.3) are orthogonal to each other and to the vector a0 . To obtain explicit expressions for p01 ; : : : ; p0n , it remains to normalize a0 and the vectors (4.3). Observing that the (usual) norm of the kth of the vectors (4.3) is 2 1=2 Pk1 2 Pk1 2 2 2 1=2 Pk1 2 Pk 2 D ; iD1 ai C iD1 ai =ak iD1 ai iD1 ai =ak we find that

p01 D

Pn

iD1

ai2

1=2

.a1 ; a2 ; : : : ; an /;

(4.4)

and that, for k D 2; : : : ; n, " p0k

D

ak2 Pk1 2 Pk . iD1 ai /. iD1 ai2 /

#1=2

P 2 .a1 ; a2 ; : : : ; ak1 ; k1 iD1 ai =ak ; 0; 0; : : : ; 0/:

(4.5)

86

8 Inverse Matrices

When a0 D .1; 1; : : : ; 1/, formulas (4.4) and (4.5) simplify to p01 D n1=2 .1; 1; : : : ; 1/; p0k D Œk.k 1/1=2 .1; 1; : : : ; 1; 1 k; 0; 0; : : : ; 0/ (k D 2; : : : ; n), and P reduces to a matrix known as the Helmert matrix (of order n). (In some presentations, it is the transpose of this matrix that is called the Helmert matrix.) For example, the Helmert matrix of order 4 is 0 1 1=2 1=2 1=2 1=2 p p B 1= 2 1= 2 0p 0 C B p C: p @ 1= 6 1=p 6 2=p 6 0 A p p 1= 12 1= 12 1= 12 3= 12 c. Permutation matrices A permutation matrix is a square matrix whose columns can be obtained by permuting (rearranging) the columns of an identity matrix. Thus, letting u1 ; u2 ; : : : ; un represent the first, second, : : : ; nth columns, respectively, of In , an n n permutation matrix is a matrix of the general form .uk1 ; uk2 ; : : : ; ukn /; where k1 ; k2 ; : : : ; kn represents any permutation of the first n positive integers 1; 2; : : : ; n. For example, one permutation matrix of order n D 3 is the 3 3 matrix 0 1 0 1 0 .u3 ; u1 ; u2 / D @0 0 1A ; 1 0 0 whose columns are the third, first, and second columns, respectively, of I3 . It is clear that the columns of any permutation matrix form an orthonormal (with respect to the usual inner product) set and hence that any permutation matrix is an orthogonal matrix. Thus, if P is a permutation matrix, then P is nonsingular and P 1 D P 0 . Clearly, the j th element of the kj th row of the n n permutation matrix .uk1 ; uk2 ; : : : ; ukn / is 1 and its other n 1 elements are 0. That is, the j th row uj0 of In is the kj th row of .uk1 ; uk2 ; : : : ; ukn / or, equivalently, the j th column uj of In is the kj th column of .uk1 ; uk2 ; : : : ; ukn /0 . Thus, the transpose of any permutation matrix is itself a permutation matrix. Further, the rows of any permutation matrix are a permutation of the rows of an identity matrix, and conversely any matrix whose rows can be obtained by permuting the rows of an identity matrix is a permutation matrix. The effect of postmultiplying an mn matrix A by an nn permutation matrix P is to permute the columns of A in the same way that the columns of In were permuted in forming P . Thus, if a1 ; a2 ; : : : ; an are the first, second, : : : ; nth columns

8.4 Orthogonal Matrices

87

of A, the first, second, : : : ; nth columns of the product A.uk1 ; uk2 ; : : : ; ukn / of A and the n n permutation matrix .uk1 ; uk2 ; : : : ; ukn / are .ak1 ; ak2 ; : : : ; akn /, respectively. When n D 3, we have, for example, that 1 0 0 1 0 A.u3 ; u1 ; u2 / D .a1 ; a2 ; a3 / @0 0 1A D .a3 ; a1 ; a2 /: 1 0 0 Further, the first, second : : : ; nth columns a1 ; a2 ; : : : ; an of A are the k1 ; k2 ; : : : ; kn th columns, respectively, of the product A.uk1 ; uk2 ; : : : ; ukn /0 of A and the permutation matrix .uk1 ; uk2 ; : : : ; ukn /0 . When n D 3, we have, for example, that 0 1 0 0 1 A.u3 ; u1 ; u2 /0 D .a1 ; a2 ; a3 / @1 0 0A D .a2 ; a3 ; a1 /: 0 1 0 Similarly, the effect of premultiplying an nm matrix A by an nn permutation matrix is to permute the rows of A. If the first, second, : : : ; nth rows of A are a01 ; a02 ; : : : ; a0n , respectively, then the first, second, : : : ; nth rows of the product .uk1 ; uk2 ; : : : ; ukn /0 A of the permutation matrix .uk1 ; uk2 ; : : : ; ukn /0 and A are .a0k1 ; a0k2 ; : : : ; a0kn /, respectively; and a01 ; a02 ; : : : ; a0n are the k1 ; k2 ; : : : ; kn th rows, respectively, of .uk1 ; uk2 ; : : : ; ukn /A. When k D 3, we have, for example, that 0 10 0 1 0 0 1 a3 0 0 1 a1 .u3 ; u1 ; u2 /0 A D @1 0 0A @a02 A D @a01 A ; 0 1 0 a03 a02 10 0 1 0 0 1 0 a1 a2 0 1 0 .u3 ; u1 ; u2 /A D @0 0 1A @a02 A D @a03 A : 1 0 0 a03 a01 Letting e1 ; e2 ; : : : ; em represent the first, second, : : : ; mth columns of Im and letting r1 ; r2 ; : : : ; rm represent any permutation of the first m positive integers 1; 2; : : : ; m, the ij th element of the matrix .er1 ; er2 ; : : : ; erm /0 A.uk1 ; uk2 ; : : : ; ukn /; obtained by premultiplying and postmultiplying an m n matrix A by the permutation matrices .er1 ; er2 ; : : : ; erm /0 and .uk1 ; uk2 ; : : : ; ukn /, respectively, is the ri kj th element of A. And the ij th element of A is the ri kj th element of the matrix .er1 ; er2 ; : : : ; erm /A.uk1 ; uk2 ; : : : ; ukn /0 ; obtained by premultiplying and postmultiplying A by the permutation matrices .er1 ; er2 ; : : : ; erm / and .uk1 ; uk2 ; : : : ; ukn /0 , respectively. Note that the product of two or more n n permutation matrices is another n n permutation matrix.

88

8 Inverse Matrices

8.5 Some Basic Results on the Ranks and Inverses of Partitioned Matrices a. A lemma The following lemma is useful in deriving results on the ranks of partitioned matrices. Lemma 8.5.1. Let T represent an m p matrix, U an m q matrix, V an n p matrix, and W an n q matrix. Then, T U U T V W W V rank D rank D rank D rank : (5.1) V W W V T U U T Proof. Lemma 8.5.1 is obtained by successively applying Lemma 4.5.3 to U T U T V W [taking A D and B D ], to [taking A D .V ; W / W V W V T U W V W V and C D .T ; U /], and to [taking A D and B D or taking U T U T A D .W ; V / and C D .U ; T /]. Q.E.D.

b. Block-diagonal matrices Let T represent an mm matrix and W an nn matrix. Then, the .mCn/.mCn/ T 0 block-diagonal matrix is nonsingular if and only if both T and W 0 W are nonsingular, as is evident from Lemma 4.5.11. Moreover, if T and W are nonsingular, then 1 1 T 0 T 0 ; (5.2) D 0 W 0 W 1 as is easily verified. More generally, for any square matrices A1 ; A2 ; : : : ; Ak , the block-diagonal matrix diag.A1 ; A2 ; : : : ; Ak / is nonsingular if and only if A1 ; A2 ; : : : ; Ak are all nonsingular [as is evident from result (4.5.14)], in which case 1 1 Œdiag.A1 ; A2 ; : : : ; Ak /1 D diag.A1 1 ; A2 ; : : : ; Ak /:

(5.3)

c. Block-triangular matrices

Im 0 In V Consider now block-triangular matrices of the form or , V In 0 Im where V is an n m matrix. Note that for any m p matrix A and n p matrix B,

8.5 Some Basic Results on the Ranks and Inverses of Partitioned Matrices

Im V In 0

A A D ; B B C VA B B C VA V D A A Im 0 In

and similarly that, for any p m matrix A and p n matrix B, Im 0 .A; B/ D .A C BV ; B/; V In I V D .B; A C BV /: .B; A/ n 0 Im

89

(5.4a) (5.4b)

(5.5a) (5.5b)

Further, recalling (from Theorem 8.1.4) that an invertible matrix is nonsingular, and observing that I 0 I 0 I 0 D ; V I V I 0 I I V I V I 0 D ; 0 I 0 I 0 I we obtain the following result. Lemma 8.5.2. For any n m matrix V , the .m C n/ .m C n/ partitioned Im 0 In V matrices and are nonsingular, and V In 0 Im

Im V In 0

0 In V Im

1 1

Im 0 ; D V In I V : D n 0 Im

(5.6a) (5.6b)

Formula (4.5.11) for the rank of a block-diagonal matrix can be extended to certain block-triangular matrices, as indicated by the following lemma. Lemma 8.5.3. Let T represent an m p matrix, V an n p matrix, and W an n q matrix. If T has full column rank or W has full row rank, that is, if rank.T / D p or rank.W / D n, then T 0 W V rank D rank D rank.T / C rank.W /: (5.7) V W 0 T Proof. Suppose that rank.T / D p. Then, according to Lemma 8.1.1, there exists a matrix L that is a left inverse of T , in which case I V L W V W 0 D : 0 I 0 T 0 T

90

8 Inverse Matrices

Since (according to Lemma 8.5.2)

I V L is nonsingular, we conclude (on 0 I

the basis of Corollary 8.3.3) that W V W 0 rank D rank 0 T 0 T and hence (in light of Lemmas 8.5.1 and 4.5.11) that T 0 W V rank D rank D rank.T / C rank.W /: V W 0 T That result (5.7) holds if rank.W / D n can be established via an analogous argument. Q.E.D. The results of Lemma 8.5.2 can be extended to additional block-triangular matrices, as detailed in the following lemma. Lemma 8.5.4. Let T represent an m m matrix, V an n m matrix, and W T 0 an n n matrix. Then, the .m C n/ .m C n/ partitioned matrix , or V W W V equivalently , is nonsingular if and only if both T and W are nonsingular, 0 T in which case 1 T 0 T 1 0 ; (5.8a) D V W W 1 V T 1 W 1 1 1 W V W W 1 V T 1 : (5.8b) D 0 T 0 T 1

T 0 V W 4.4.3, we find that Proof. If

is nonsingular, then, making use of Lemmas 4.5.7 and

T 0 m C n D rank V W

rank.T ; 0/ C rank.V ; W / rank.T / C rank.0/ C rank.V ; W / rank.T / C n m C n; implying that m rank.T / m and hence that rank.T / D m or, equivalently, T 0 that T is nonsingular. It can likewise be shown that if is nonsingular, V W then W is nonsingular. Suppose now that both T and W are nonsingular. Then, T 0 Im 0 T 0 Im 0 D ; V W 0 In 0 W W 1 V T 1 In


91

I 0 I 0 , as is easily verified, and (in light of Lemma 8.5.3) , 0 W W 1 V T 1 I T 0 T 0 and are nonsingular. Thus, it follows from result (2.9) that is 0 I V W nonsingular and from results (2.10), (5.2), and (5.6) that

T 0 V W

1

1 1 1 I 0 I 0 T 0 D W 1 V T 1 I 0 W 0 I 1 0 I 0 I 0 T D 0 I W 1 V T 1 I 0 W 1 T 1 0 : D 1 W V T 1 W 1

The second part of result (5.8) can be derived in similar fashion.

Q.E.D.

The following theorem extends the first part of Lemma 8.5.4. Theorem 8.5.5. Let 0 1 A11 A12 : : : A1r B 0 A22 : : : A2r C B C ADB : :: C :: @ :: : : A 0 0 Arr

0

and

1 B11 0 : : : 0 BB21 B22 0 C B C BDB : C : : :: :: @ :: A Br1 Br2 : : : Brr

represent, respectively, an nn upper block-triangular matrix whose ij th block Aij is of dimensions ni nj (j i D 1; 2; : : : ; r) and an n n lower block-triangular matrix whose ij th block Bij is of dimensions ni nj (j i D 1; 2; : : : ; r). Then, A is nonsingular if and only if its diagonal blocks A11 ; A22 ; : : : ; Arr are all nonsingular. Likewise, B is nonsingular if and only if its diagonal blocks B11 ; B22 ; : : : ; Brr are all nonsingular. Proof. Observe that the transpose A0 of the upper block-triangular matrix A is lower block-triangular with diagonal blocks A011 ; A022 ; : : : ; A0rr , that rank.A0 / D rank.A/ and rank.A0i i / D rank.Ai i / (i D 1; 2; : : : ; r), and that, as a consequence, it suffices to prove the part of the theorem pertaining to the lower block-triangular matrix B. The proof is by mathematical induction. Lemma 8.5.4 implies that the theorem is valid for r D 2. Suppose then that the theorem is valid for r D k 1. Defining 0 1 B11 0 ::: 0 C B B21 B22 0 B C T DB : C and V D .Bk1 ; Bk2 ; : : : ; Bk;k1 /; : : :: :: @ :: A Bk1;1 Bk1;2 : : : Bk1;k1 T 0 is nonsingular if and only if the matrices we need to show that V Bkk B11 ; B22 ; : : : ; Bkk are all nonsingular.

92

8 Inverse Matrices

If B11 ; B22 ; : : : ; Bkk are nonsingular, then, by supposition, T is nonsingular T 0 is nonsingular. and, based on Lemma 8.5.4, we conclude that V Bkk T 0 is nonsingular, then, according to Lemma 8.5.4, T Conversely, if V Bkk and Bkk are both nonsingular. Moreover, if T is nonsingular, then, by supposition, Q.E.D. B11 ; : : : ; Bk1;k1 are nonsingular. In the special case of a triangular matrix (i.e., the special case where r D n), Theorem 8.5.5 can be restated as follows. Corollary 8.5.6. A triangular matrix is nonsingular if and only if its diagonal elements are all nonzero. Certain of the results given in Lemma 8.5.4 on the inverse of a block-triangular matrix with two rows and columns of blocks are extended in the following theorem to a block-triangular matrix with an arbitrary number r of rows and columns of blocks. Theorem 8.5.7. Let

0

A11 A12 BA21 A22 B ADB : :: @ :: : Ar1 Ar2

1 : : : A1r : : : A2r C C :: C :: : : A : : : Arr

represent a nonsingular partitioned matrix whose ij th block Aij is of dimensions ni nj (i; j D 1; : : : ; r). Partition A1 as 0 1 F11 F12 : : : F1r BF21 F22 : : : F2r C B C F DB : :: : : :: C ; @ :: : : : A Fr1 Fr2 : : : Frr where Fij is of the same dimensions as Aij (i; j D 1; : : : ; r). If A is upper blocktriangular, then A1 is upper block-triangular; that is, if Aij D 0 for j < i D 1; : : : ; r, then Fij D 0 for j < i D 1; : : : ; r. Similarly, if A is lower blocktriangular, then A1 is lower block-triangular. Further, if A is (lower or upper) 1 equals block-triangular, then Fi i D A1 i i ; that is, the i th diagonal block of A the inverse of the i th diagonal block of A (i D 1; : : : ; r). Proof. The proof is by mathematical induction. Lemma 8.5.4 implies that the theorem is valid for r D 2. Suppose then that the theorem is valid for r D k 1. Define 0 0 1 1 A1k A11 A12 : : : A1;k1 B A2k C B A21 A22 : : : A2;k1 C B B C C ; U D T DB : B :: C ; C : : :: :: :: @ : A @ :: A : Ak1;1 Ak1;2 : : : Ak1;k1

Ak1;k


and V D .Ak1 ; Ak2 ; : : : ; Ak;k1 /, so that 0 A11 A12 BA T U B 21 A22 DB : :: V Akk @ :: : Ak1 Ak2 1 T U and partition as V Akk 0 B11 B12 1 BB T U B 21 B22 DB : :: V Akk @ :: : Bk1 Bk2

93

1 : : : A1k : : : A2k C C :: C ; :: : : A : : : Akk

1 : : : B1k : : : B2k C C :: C ; :: : : A : : : Bkk

where Bij is of dimensions ni nj (i; j D 1; : : : ; k). It suffices to show that if Aij D 0 for j i D 1; : : : ; k, then Bi i D A1 ii (i D 1; : : : ; k). If Aij D 0 for j < i D 1; : : : ; k, then V D 0, so that (according to Lemma , and 8.5.4) Bkj D 0 for j D 1; : : : ; k 1, Bkk D A1 kk 1 0 B11 B12 : : : B1;k1 B B21 B22 : : : B2;k1 C C B C D T 1 I B :: :: :: :: A @ : : : : Bk1;1 Bk1;2 : : : Bk1;k1 1 A11 A12 : : : A1;k1 B 0 A22 : : : A2;k1 C B C and T D B : C is upper block-triangular, so that (by :: :: @ :: A : : 0 0 Ak1;k1 supposition) Bij D 0 for j < i D 1; : : : ; k 1 and Bi i D A1 i i (i D 1; : : : ; k 1). Similarly, if Aij D 0 for i < j D 1; : : : ; k, then U D 0, so that (according to , and Lemma 8.5.4) Bik D 0 for i D 1; : : : ; k 1, Bkk D A1 kk 1 0 B11 B12 : : : B1;k1 B B21 B22 : : : B2;k1 C C B C D T 1 I B :: :: :: :: A @ : : : : Bk1;1 Bk1;2 : : : Bk1;k1 0 1 A11 0 ::: 0 B A21 C A22 0 B C and T D B : C is lower block-triangular, so that : :: :: @ :: A : 0

Ak1;1 Ak1;2 : : : Ak1;k1

94

8 Inverse Matrices

(by supposition) Bij D 0 for i < j D 1; : : : ; k 1 and Bi i D A1 i i (i D 1; : : : ; k 1). Q.E.D. In the special case of a triangular matrix (i.e., the special case where r D n), Theorem 8.5.7 can be restated as follows. Corollary 8.5.8. Let A D faij g represent an n n nonsingular matrix. If A is upper triangular, then A1 is also upper triangular. Similarly, if A is lower triangular, then A1 is lower triangular. Further, if A is (lower or upper) triangular, then the i th diagonal element of A1 is the reciprocal 1=ai i of the i th diagonal element ai i of A (i D 1; : : : ; n). In the special case of a unit triangular matrix, Corollary 8.5.8 reduces to the following result. Corollary 8.5.9. The inverse of a unit upper triangular matrix is unit upper triangular. Similarly, the inverse of a unit lower triangular matrix is unit lower triangular. d. A recursive algorithm for finding the inverse of a (nonsingular) triangular or block-triangular matrix 0 1 A11 A12 : : : A1r B 0 A22 : : : A2r C B C ADB : :: C :: @ :: : : A 0 0 Arr

Let

represent an n n upper block-triangular matrix whose ij th block Aij is of dimensions ni nj (j i D 1; 2; : : : ; r). Suppose that the diagonal blocks A11 ; A22 ; : : : ; Arr of A are all nonsingular, or, equivalently (according to Theorem 8.5.5), that A itself is nonsingular. We now describe a procedure, derived by repeated application of result (5.8), for building up the inverse of A one “row” of blocks at a time, or in the special case of an upper triangular matrix, one row (of elements) at a time. According to result (5.8), the inverse of the submatrix Ar1;r1 Ar1;r 0 Arr is expressible as

Fr1;r1 Fr1;r ; 0 Frr

where Frr D A1 rr and Fr1;r1 D A1 r1;r1 ;

Fr1;r D A1 r1;r1 Ar1;r Frr :

By using this representation in an obvious way, the inverse of the submatrix


95

Ar1;r1 Ar1;r can be generated from that of the sub-submatrix Arr . 0 Arr More generally, result (5.8) indicates that the inverse of the submatrix 1 0 Ai i Ai;iC1 : : : Air B 0 AiC1;iC1 : : : AiC1;r C C B B :: :: C :: @ : : : A 0

0

Arr

can be generated by bordering the inverse matrix 0 1 FiC1;iC1 FiC1;iC2 : : : FiC1;r B 0 FiC2;iC2 : : : FiC2;r C B C B :: :: C :: @ : : : A 0 0 Frr 0 11 AiC1;iC1 AiC1;iC2 : : : AiC1;r B 0 AiC2;iC2 : : : AiC2;r C B C DB :: :: C :: @ : : : A 0 0 Arr

(5.9a)

(5.9b)

with the matrices Fi i D

A1 ii ;

to form the matrix

Fij D

j X 1 Ai i Aik Fkj kDiC1

.j D i C 1; : : : ; r/;

1 Fi i Fi;iC1 : : : Fir B 0 FiC1;iC1 : : : FiC1;r C C B B :: :: C :: @ : : : A 0 0 Frr

(5.10)

0

(5.11)

(i D r 1; r 2; : : : ; 1). Note that formulas (5.10) are in terms of the entries of the inverse matrix (5.9) and of the submatrices Ai i ; Ai;iC1 ; : : : ; Air , belonging to the i th “row” of A. Note also that, for i D 1, matrix (5.11) equals A1 . These observations suggest an algorithm for computing A1 in r steps. The first step is to compute the matrix Frr D A1 rr . The .r i C 1/th step is to compute the matrices Fi i ; Fi;iC1 ; : : : ; Fir from formulas (5.10) (i D r 1; r 2; : : : ; 1). Note that if the matrices Fi;iC1 ; : : : ; Fir are computed in reverse order, that is, in the order Fir ; : : : ; Fi;iC1 , then, once the matrix Fij (j i ) has been computed, the submatrix Aij is no longer needed (at least not for purposes of computing A1 ), a feature that allows us to save storage space in implementing the algorithm on a computer. The savings is achieved by storing the elements of Fij in the locations previously occupied by those of Aij . Then, at the completion of the algorithm, the locations that were previously occupied by the elements of A are occupied by the corresponding elements of A1 .

96

8 Inverse Matrices

A possible variation on this algorithm is to use formulas (5.10) to build-up A1 one “column” at a time, rather than one “row” at a time. As a first step, we could compute the diagonal blocks Frr , Fr1;r1 ; : : : ; F11 . As the .r j C 2/th step, we could compute in succession the submatrices Fj 1;j ; Fj 2;j ; : : : ; F1j (j D r; r 1; : : : ; 2). Note that this alternative scheme, like the original, is such that the elements of Fij can share storage locations with those of Aij . For purposes of illustration, we apply the algorithm to the 33 upper triangular matrix 0 1 8 1 6 A D @0 5 4A ; 0 0 2 which has an inverse of the form

0

1 f11 f12 f13 F D @ 0 f22 f23 A : 0 0 f33

On the first step of the algorithm, we compute f33 D 1=2 D 0:5I on the second step, f22 D 1=5 D 0:2;

f23 D .0:2/.4/.0:5/ D 0:4I

and, on the third (and final) step, f11 D 1=8 D 0:125; f12 giving

f13 D .0:125/Œ.1/.0:4/ C .6/.0:5/ D 0:325; D .0:125/.1/.0:2/ D 0:025I

1 0:125 0:025 0:325 0:2 0:4 A : DF D@ 0 0 0 0:5 0

A1

Let us now switch our attention from the upper block-triangular matrix A to a lower block-triangular matrix 0 1 B11 0 : : : 0 BB21 B22 0 C B C BDB : C; : : :: :: @ :: A Br1 Br2 : : : Brr whose ij th block Bij is of dimensions ni nj (j i D 1; : : : ; r). Suppose that the diagonal blocks B11 ; B22 ; : : : ; Brr of B are all nonsingular, or, equivalently, that B itself is nonsingular. According to result (5.8), the inverse of the submatrix


97

0

1 B11 0 : : : 0 BB21 B22 0 C B C B :: C :: : : @ : A : : Bi1 Bi2 : : : Bi i can be generated by bordering the inverse matrix 0 1 G11 0 ::: 0 B G21 C G22 0 B C B :: C :: :: @ : A : : Gi1;1 Gi2;2 : : : Gi1;i1 0 B11 0 B B21 B 22 B DB : :: : @ : : Bi1;1 Bi1;2

:::

(5.12a)

0 0

::

: : : : Bi1;i1

11 C C C A

(5.12b)

with the matrices Gi i D Bi1 i ;

Gij D Bi1 i

i1 X

Bik Gkj

.j D 1; : : : ; i 1/;

(5.13)

kDj

to form the matrix

1 0 G11 0 : : : 0 BG21 G22 0 C C B C: B :: :: : :: A @ : : Gi1 Gi2 : : : Gi i

(5.14)

We see that formulas (5.13) are in terms of the entries of the inverse matrix (5.12) and of the submatrices Bi1 ; Bi2 ; : : : ; Bi i , belonging to the i th “row” of B. We see also that, for i D r, matrix (5.14) equals B1 . These results suggest an algorithm for computing B1 in r steps. The first step 1 . The i th step is to compute the matrices Gi1 ; Gi2 ; : : : ; is to compute G11 D B11 Gi i from formulas (5.13) (i D 2; : : : ; r). Note that if Gi1 ; Gi2 ; : : : ; Gi;i1 are computed in the order listed, then, once the matrix Gij (j i ) has been computed, the submatrix Bij is no longer required and hence, in implementing the algorithm on a computer, storage can be saved by “overwriting” the matrix Bij with the matrix Gij . e. Matrices partitioned into two rows and two columns of submatrices The ranks and inverses of block-diagonal and block-triangular matrices were considered in Subsections a through d. In this subsection, we consider the ranks and inverses of partitioned matrices that are not necessarily block-diagonal or blocktriangular. The following theorem can (when applicable) be used to express the rank of a partitioned matrix in terms of the rank of a matrix of smaller dimensions.

98

8 Inverse Matrices

Theorem 8.5.10. Let T represent an m m matrix, U an m q matrix, V an n m matrix, and W an n q matrix. If rank.T / D m, that is, if T is nonsingular, then T U U T V W rank D rank D rank V W W V T U W V D rank U T D m C rank.W V T 1 U /:

(5.15)

Proof. Suppose that rank.T / D m. Then, in light of Lemma 8.5.1, it suffices to show that T U rank D m C rank.W V T 1 U /: V W It is easy to verify that Im T U T U 0 D : V W 0 W V T 1 U V T 1 In I 0 is nonsingular, it follows Thus, since (according to Lemma 8.5.2) V T 1 I from Corollary 8.3.3 and Lemma 8.5.3 that T U T U rank D rank V W 0 W V T 1 U D rank.T / C rank.W V T 1 U / D m C rank.W V T 1 U /: Q.E.D. Result (5.8) on the inverse of a block-triangular matrix can be extended to other partitioned matrices, as detailed in the following theorem. Theorem 8.5.11. Let T represent an m m matrix, U an m n matrix, V an n matrix. Suppose that T is nonsingular. Then, mmatrix, and W an n n T U W V , or equivalently , is nonsingular if and only if the nn matrix V W U T Q D W V T 1 U is nonsingular, in which case

T U V W

1

T 1 C T 1 U Q1 V T 1 T 1 U Q1 D Q1 V T 1 Q1 1 0 T 1 U T Q1 .V T 1 ; In /; C D 0 0 In

(5.16a) (5.16b)


1 W V Q1 Q1 V T 1 D U T T 1 U Q1 T 1 C T 1 U Q1 V T 1 In 0 0 C D Q1 .In ; V T 1 /: 0 T 1 T 1 U

99

(5.17a) (5.17b)

T U Proof. That is nonsingular if and only if Q is nonsingular is an V W immediate consequence of Theorem 8.5.10. Suppose now that Q is nonsingular, and observe that T 0 T U I T 1 U D : (5.18) V Q V W 0 I T 0 T U Then (in light of Lemma 8.5.4), , as well as , is nonsingular. V Q V W 1 T U Premultiplying both sides of equality (5.18) by and postmultiplying V W 1 T 0 both sides by and making further use of Lemma 8.5.4, we find that V Q

T U V W

1

1 I T 1 U T 0 0 I V Q 1 I T U T 1 0 D 0 I Q1 V T 1 Q1 1 T C T 1 U Q1 V T 1 T 1 U Q1 ; D Q1 V T 1 Q1 D

which establishes formula (5.16). Formula (5.17) can be derived in an analogous fashion. (As an alternative to deriving these formulas from results on the inverse of a block-triangular matrix, which is the approach taken here, their validity could be established simply by verifying that the product of expression (5.16) or (5.17) T U W V Q.E.D. and or , respectively, equals ImCn .) V W U T When T is nonsingular, the matrix Q D W V T 1 U , which appears in formulas (5.16) and (5.17) for the inverse of a partitioned matrix and also in formula (5.15) for the rank of a partitioned matrix, is called the Schur complement T U W V of T in or , respectively. Moreover, when the context is clear, V W U T it is sometimes referred to simply as the Schur complement of T or even more T U W V simply as the Schur complement. Note that if or is symmetric V W U T (in which case T 0 D T , W 0 D W , and V D U 0 ), then the Schur complement of T is symmetric.

100

8 Inverse Matrices

Among other things, the following corollary (of Theorem 8.5.11) expresses the inverse of a diagonal block of a partitioned matrix in terms of the inverse of the partitioned matrix. Corollary 8.5.12. Let T represent an m m matrix, U an m n matrix, V an T U nm matrix, and W an nn matrix. Suppose that the partitioned matrix V W 1 T U B11 B12 is nonsingular, define B D , , and partition B as B D V W B21 B22 where the dimensions of B11 ; B12 ; B21 ; and B22 are the same as those of T , U , V , and W , respectively. If T is nonsingular, then B22 is nonsingular, 1 T 1 D B11 B12 B22 B21

(5.19)

(i.e., T 1 equals the Schur complement of B22 ), 1 T 1 U D B12 B22 ;

and

1 V T 1 D B22 B21 ;

1 W V T 1 U D B22 1 equals B22 ). Similarly, if W

(i.e., the Schur complement of T B11 is nonsingular, 1 B12 W 1 D B22 B21 B11

(5.20) (5.21)

is nonsingular, then (5.22)

(i.e., W 1 equals the Schur complement of B11 ), 1 ; W 1 V D B21 B11

and

1 U W 1 D B11 B12 ;

1 T U W 1 V D B11

(5.23) (5.24)

1 B11 ).

(i.e., the Schur complement of W equals Proof. Suppose that T is nonsingular, and let Q D W V T 1 U . Then, it follows from Theorem 8.5.11 that Q is nonsingular, that B22 D Q1 , implying 1 D Q, and that [in light of results (2.5) and (2.6)] that B22 is nonsingular and B22 1 D .T 1 U Q1 /Q D T 1 U ; B12 B22 1 B21 D Q.Q1 V T 1 / D V T 1 ; B22

and 1 B21 D T 1 C T 1 U Q1 V T 1 C T 1 U .Q1 V T 1 / D T 1 : B11 B12 B22

The second part of the corollary can be established via an analogous argument. Q.E.D. Suppose that we wish to find the rank of a matrix A; or if A is nonsingular, that we wish to find A1 or a submatrix of A1 , or that we wish to find the inverse


101

of a nonsingular submatrix of A. Then, depending on the nature of A, Theorems 8.5.10 and 8.5.11 and Corollary 8.5.12 can be helpful, as will now be discussed. A11 A12 represent an arbitrary partitioning of A into two rows Let A D A21 A22 and columns of submatrices. If A11 or A22 is nonsingular, then, by making use of Theorem 8.5.10, the problem of finding the rank of A can be replaced by that of (1) inverting A11 or A22 , (2) forming the Schur complement A22 A21 A1 11 A12 or A11 A12 A1 A , and (3) finding the rank of the Schur complement. This can 22 21 be advantageous if the inverse of A11 or A22 is known—as would be the case, for example, if A11 or A22 were a (nonsingular) diagonal matrix—and if A11 or A22 , respectively, is of relatively large order. Thus, whether Theorem 8.5.10 is helpful in finding the rank of A depends on whether the partitioning of A can be carried out in such a way that A11 or A22 is a relatively large order, nonsingular matrix with a known inverse. Now, suppose that A is nonsingular, let B D A1 , and partition B as B11 B12 , where the dimensions of B11 ; B12 ; B21 ; and B22 are the same BD B21 B22 as those of A11 ; A12 ; A21 ; and A22 , respectively. If A11 or A22 is nonsingular, then, by making use of Theorem 8.5.11, the problem of constructing A1 can 1 be reformulated as one of (1) finding A1 11 or A22 , (2) forming and inverting the 1 Schur complement A22 A21 A11 A12 or A11 A12 A1 22 A21 , and (3) determining, based on formula (5.16) or (5.17), respectively, B11 ; B12 ; B21 ; and B22 . This can be advantageous if A11 or A22 has a known inverse and is of relatively large order and/or if only the lower-right or upper-left part (B22 or B11 ) of A1 is of interest. Thus, whether Theorem 8.5.11 is helpful in constructing A1 depends on whether A can be partitioned in such a way that A11 or A22 is a relatively large order matrix with a known inverse and possibly on whether only part of A1 is of interest. Finally, suppose that the upper-left or lower-right part (A11 or A22 ) of A is 1 nonsingular and that we wish to construct A1 11 or A22 . If A itself is nonsingular, 1 1 then, by making use of Corollary 8.5.12, we can compute A1 11 or A22 from A , 1 which can be advantageous if A is known and if A11 or A22 , respectively, is of relatively large order. f. Use of permutation matrices to extend results on partitioned matrices One important use of permutation matrices is to extend results on partitioned matrices. Consider, for example, result (5.16) on the inverse of a partitioned matrix. This result gives the inverse of an .mCn/.mCn/ matrix, say A, in terms of a submatrix T formed from the first m rows and columns of A, a submatrix U formed from the first m rows and last n columns of A, a submatrix V formed from the last n rows and first m columns of A, and a submatrix W formed from the last n rows and columns of A. Taking i1 ; i2 ; : : : ; imCn and j1 ; j2 ; : : : ; jmCn to be any two (not necessarily different) permutations of the first m C n positive integers 1; 2; : : : ; m C n, result (5.16) can be extended to the case where the submatrix T is formed from rows

102

8 Inverse Matrices

i1 ; : : : ; im , respectively, and columns j1 ; : : : ; jm , respectively (of A), U is formed from rows i1 ; : : : ; im , respectively, and columns jmC1 ; : : : ; jmCn , respectively, V from rows imC1 ; : : : ; imCn , respectively, and columns j1 ; : : : ; jm , respectively, and W from rows imC1 ; : : : ; imCn , respectively, and columns jmC1 ; : : : ; jmCn , respectively. In fact, result (5.17) can be viewed as one particularly simple extension of this type. To present explicitly a version of result (5.16) that would be applicable to arbitrary permutations i1 ; i2 ; : : : ; imCn and j1 ; j2 ; : : : ; jmCn would require very elaborate and cumbersome notation. However, by explicitly or implicitly making use of appropriately chosen permutation matrices, result (5.16), which is for the special case where both i1 ; i2 ; : : : ; imCn and j1 ; j2 ; : : : ; jmCn are the sequence 1; 2; : : : ; m C n, can be applied to any other choice for i1 ; i2 ; : : : ; imCn and j1 ; j2 ; : : : ; jmCn . Letting u1 ; u2 ; : : : ; umCn represent the first, second, : : : ; .m C n/th columns of ImCn , define B D P AQ; where P D .ui1 ; ui2 ; : : : ; uimCn /0 Partition B as

and

Q D .uj1 ; uj2 ; : : : ; ujmCn /:

T U BD V W

;

where T is mm, U is mn, V is nm, and W is nn. Note that T is the matrix formed from rows i1 ; : : : ; im , respectively, and columns j1 ; : : : ; jm , respectively, of A; the submatrices U , V , and W have analogous interpretations. Suppose that we invert B by, for example, making use of result (5.16) or (5.17) (assuming that A, and hence B, is invertible). Then, to obtain A1 from B1 , we observe that A D P 0 BQ0 and, consequently, that

A1 D QB1 P :

(5.25)

Thus, to form A1 from B1 , it suffices to permute the rows of B1 so that its first row becomes the j1 th row, its second row becomes the j2 th row, and, in general, its rth row becomes the jr th row, and to then permute the columns so that the first column becomes the i1 th column, the second column becomes the i2 th column, and, in general, the sth column becomes the is th column. Or, combining the row and column operations, we can form A1 from B1 simply by rearranging the elements of B1 so that the rsth element becomes the jr is th element. Consider, for example, the matrix 0 1 0 1 0 1 B6 2 4 8C C A DB @ 0 5 1 0A : 1 3 0 0

Exercises

103

Observe that the 3 3 matrix formed from the fourth, third, and first rows, respectively, and the first, third, and fourth columns, respectively, of A is the 3 3 identity matrix, whose inverse is known. Suppose then that we take B D .u4 ; u3 ; u1 ; u2 /0 A.u1 ; u3 ; u4 ; u2 / 0 1 1 0 0 3 B 0 1 0 5C C DB @ 0 0 1 1A : 6 4 8 2 Applying result (5.16) with T D I3 , U 0 D .3; 5; 1/, V D .6; 4; 8/, and W D .2/, we find easily that 0 1 3:25 1:5 3 0:375 B 3:75 1:5 5 0:625C C: B1 D B @ 0:75 0:5 0 0:125A 0:75 0:5 1 0:125 Rearranging the elements of B1 in accordance with formula (5.25), we obtain 0 1 3 0:375 1:5 3:25 B 1 0:125 0:5 0:75C C: A1 D B @5 0:625 1:5 3:75A 0 0:125 0:5 0:75

Exercises Section 8.1 1. Let A represent an m n matrix. Show that (a) if A has a right inverse, then n m and (b) if A has a left inverse, then m n. 2. An n n matrix A is said to be involutory if A2 D I, that is, if A is invertible and is its own inverse. (a) Show that an nn matrix A is involutory if and only if .IA/.ICA/ D 0. a b (b) Show that a 2 2 matrix A D is involutory if and only if (1) c d a2 C bc D 1 and d D a or (2) b D c D 0 and d D a D ˙1. Section 8.3 3. Let A represent an n n nonnull symmetric matrix, and let B represent an n r matrix of full column rank r and T an r n matrix of full row rank r such that A D BT . Show that the r r matrix T B is nonsingular. (Hint. Observe that A0 A D A2 D BT BT .)

104

8 Inverse Matrices

Section 8.4 4. Let A represent an n n matrix, and partition A as A D .A1 ; A2 /. B1 (a) Show that if A is invertible and A1 is partitioned as A1 D (where B2 B1 has the same number of rows as A1 has columns), then B1 A1 D I;

B1 A2 D 0;

A1 B 1 D I A 2 B 2 ;

B2 A1 D 0;

B2 A2 D I;

A2 B2 D I A1 B1 :

(E.1) (E.2)

(b) Show that if A is orthogonal, then A01 A1 D I;

A1 A01

A01 A2 D 0; DI

A2 A02 ;

A02 A1 D 0; A2 A02

A02 A2 D I;

DI

A1 A01 :

(E.3) (E.4)

5. Let A represent an m n nonnull matrix of rank r. Show that there exists an m m orthogonal matrix whose first r columns span C.A/. Section 8.5 6. Let T represent an n n triangular matrix. Show that rank.T / is greater than or equal to the number of nonzero diagonal elements in T . 7. Let 0

1 A11 A12 : : : A1r B 0 A22 : : : A2r C B C ADB : :: C ; :: @ :: : : A 0 0 Arr

0

1 B11 0 : : : 0 BB21 B22 0 C B C BDB : C :: : : @ :: A : : Br1 Br2 : : : Brr

represent, respectively, an n n upper block-triangular matrix whose ij th block Aij is of dimensions ni nj (j i D 1; : : : ; r) and an n n lower block-triangular matrix whose ij th block Bij is of dimensions ni nj (j i D 1; : : : ; r). (a) Assuming that A and B are invertible, show that 0 0 1 1 F11 F12 : : : F1r G11 0 : : : 0 B 0 F22 : : : F2r C BG21 G22 0 C B B C C 1 ; B D A1 D B : B C C; : : : :: :: :: @ :: @ :: A : : :: A Gr1 Gr2 : : : Grr 0 0 Frr where

Exercises 1 Fjj D Ajj ;

1 jX 1 Fij D Fik Akj Ajj

105

.i < j D 1; : : : r/; (E.5)

kDi 1 Gjj D Bjj ;

i X 1 Gij D Gik Bkj Bjj .i > j D 1; : : : r/; (E.6) kDj C1

Do so by applying the results of Section 8.5d to A0 and B0 [as opposed to mimicking the derivations of formulas (5.10) and (5.13)]. (b) Describe how the formulas in Part (a) can be used to devise r-step algorithms for computing A1 and B1, and indicate how these algorithms differ from those described in Section 8.5d.

9 Generalized Inverses

In statistics (and in many other disciplines), it is common to encounter linear systems whose coefficient matrices are square but not nonsingular (i.e., they are singular) and hence are not invertible (i.e., they lack inverses). For a linear system that has a nonsingular coefficient matrix, there is an intimate relationship—refer to Section 8.2a—between the solution of a linear system and the inverse of the coefficient matrix. What about a linear system whose coefficient matrix is not nonsingular? Is there a matrix comparable to the inverse (i.e., that relates to the solution of the linear system in a similar way)? The answer is yes! In fact, there are an infinite number of such matrices. These matrices, which are called generalized inverses, are the subject of this chapter. Over time, the use of generalized inverses in statistical discourse (especially that related to linear statistical models and multivariate analysis) has become increasingly routine. There is a particular generalized inverse, known as the Moore-Penrose inverse, that is sometimes singled out for special attention. Discussion of the MoorePenrose inverse is deferred to Chapter 20. For many purposes, one generalized inverse is as good as another.

9.1 Definition, Existence, and a Connection to the Solution of Linear Systems A generalized inverse of an m n matrix A is any n m matrix G such that AGA D A: (The use of the term generalized inverse to refer to any such matrix is widespread, but not universal—among the alternative names found in the literature are pseudoinverse and conditional inverse.) For example, each of the two 3 2 matrices 0 1 0 1 1 0 42 1 @0 0A @ 5 3A and 0 0 2 2

108


is a generalized inverse of the 2 3 matrix 1 3 2 : 2 6 4 For a nonsingular matrix A, it is clear that AA1 A D A and that if G is a generalized inverse of A, then G D A1 AGAA1 D A1 AA1 D A1 . Thus, we have the following lemma. Lemma 9.1.1. The inverse A1 of a nonsingular matrix A is a generalized inverse of A, and a nonsingular matrix has no generalized inverse other than its inverse. There is an intimate relationship between any generalized inverse of an m n matrix A and the solution of linear systems whose coefficient matrix is A. This relationship is described in the following theorem, which generalizes Theorem 8.2.1. Theorem 9.1.2. Let A represent any m n matrix, G any n m matrix, and p any positive integer. Then, GB is a solution to a linear system AX D B (in X) for every m p matrix B for which the linear system is consistent if and only if G is a generalized inverse of A. Proof. Suppose that G is a generalized inverse of A. Let B represent any m p matrix for which AX D B is consistent, and take X to be any solution to AX D B. Then, A.GB/ D .AG/B D AGAX D AX D B: Conversely, suppose that GB is a solution to AX D B (i.e., that AGB D B) for every m p matrix B for which AX D B is consistent. Letting ai represent the i th column of A, observe that AX D B is consistent in particular for B D .ai ; 0; : : : ; 0/ —for this B, one solution to AX D B is the matrix .ui ; 0; : : : ; 0/, where ui is the i th column of In (i D 1; : : : ; n). It follows that AG.ai ; 0; : : : ; 0/ D .ai ; 0; : : : ; 0/ and hence that AGai D ai (i D 1; : : : ; n), implying that AGA D A. Q.E.D. Let us now consider the existence of generalized inverses. Does every matrix have at least one generalized inverse? The answer to this question is yes, as can be shown by making use of the following theorem, which is of interest in its own right. Theorem 9.1.3. Let B represent an m r matrix of full column rank and T an r n matrix of full row rank. Then, B has a left inverse, say L, and T has a right inverse, say R; and RL is a generalized inverse of BT .

9.2 Some Alternative Characterizations

109

Proof. That B has a left inverse L and T a right inverse R is an immediate consequence of Lemma 8.1.1. Moreover, BT .RL/BT D B.T R/.LB/T D BIIT D BT ; that is, RL is a generalized inverse of BT . Q.E.D. Now, consider an arbitrary m n matrix A. If A D 0, then clearly any n m matrix is a generalized inverse of A. If A ¤ 0, then (according to Theorem 4.4.8) there exist a matrix B of full column rank and a matrix T of full row rank such that A D BT , and hence, according to Theorem 9.1.3, A has a generalized inverse. Thus, we arrive at the following conclusion. Corollary 9.1.4. Every matrix has at least one generalized inverse. The symbol A is used to denote an arbitrary generalized inverse of an m n matrix A. By definition, AA A D A:

9.2 Some Alternative Characterizations a. A generalized inverse in terms of the inverse of a nonsingular submatrix For an m n matrix A of the general form A11 0 AD ; 0 0 where A11 is an r r nonsingular matrix, and an n m matrix G11 G12 GD G21 G22 (where G11 is of dimensions r r), we find that A11 G11 A11 0 ; AGA D 0 0 implying that G is a generalized inverse of A if and only if A11 G11 A11 D A11 ; or equivalently (in light of Lemma 9.1.1) if and only if G11 D A1 11 , and hence that G is a generalized inverse of A if and only if 1 A11 X GD Y Z for some matrices X, Y , and Z (of appropriate dimensions). This result can be generalized as follows:

110

as


Theorem 9.2.1. Let A represent an m n matrix of rank r, and partition A A11 A12 ; AD A21 A22

where A11 is of dimensions r r. Suppose that A11 is nonsingular. Then, an n m matrix G is a generalized inverse of A if and only if 1 1 1 1 A11 XA21 A1 11 A11 A12 Y A11 A12 ZA21 A11 X GD (2.1) Y Z for some matrices X, Y , and Z (of appropriate dimensions). To prove Theorem 9.2.1, we require the following lemma, which is of interest in its own right. Lemma 9.2.2. Let

A11 A12 AD A21 A22

represent an m n partitioned matrix of rank r, where A11 is of dimensions r r. If rank.A11 / D r, then A22 D A21 A1 11 A12 : Proof (of Lemma 9.2.2). Suppose that rank.A11 / D r. Then, applying Theorem 8.5.10 (with T D A11 , U D A12 , V D A21 , and W D A22 ), we find that r D rank.A/ D r C rank.A22 A21 A1 11 A12 /; implying that and hence that or equivalently that

rank.A22 A21 A1 11 A12 / D 0 A22 A21 A1 11 A12 D 0 A22 D A21 A1 11 A12 : Q.E.D.

Proof (of Theorem 9.2.1). It follows from Lemma 9.2.2 that A11 A1 AD 11 .A11 ; A12 /: A21 By definition, G is a generalized inverse of A if and only if AGA D A, or, equivalently, if and only if A11 A11 1 A11 .A11 ; A12 /G A1 11 .A11 ; A12 / A21 A21 A11 A1 D 11 .A11 ; A12 /: A21

(2.2)


Partition G as

111

G11 G12 GD G21 G22

(where G11 is of dimensions r r). It is clear that A rank 11 D r and rank.A11 ; A12 / D r A21 and hence (in light of Lemma 8.3.1) that G satisfies condition (2.2) if and only if A11 1 A1 .A ; A /G A1 11 12 11 11 D A11 ; A21 or, equivalently [since A1 11 .A11 ;

A11 A1 A12 /G 11 A21

1 1 1 D G11 C A1 11 A12 G21 C G12 A21 A11 C A11 A12 G22 A21 A11 ];

if and only if 1 1 1 1 G11 D A1 11 A11 A12 G21 G12 A21 A11 A11 A12 G22 A21 A11 :

Q.E.D. Any m n matrix A of rank r can be partitioned as A11 A12 ; AD A21 A22 where A11 is of dimensions r r. [If m D r or n D r, then A22 and A21 or A12 , respectively, are regarded as degenerate, that is A11 ; or AD A D .A11 ; A12 / A21 respectively.] Theorem 9.2.1 describes (for the special case where A11 is nonsingular) the general form of a generalized inverse of A (in terms of the submatrices A11 , A12 , and A21 ). [If m D r or n D r, then, in Theorem 9.2.1, Z and X or Y , respectively, are regarded as degenerate, and—assuming rank.A11 / D r—we have respectively that G is a generalized inverse of A D .A11 ; A12 / if and only if 1 A11 A1 11 A12 Y GD Y for some Y , or that G is a genealized inverse of A11 AD A21

112


if and only if

1 G D .A1 11 XA21 A11 ; X/

for some X.] Note that Theorem 9.2.1 implies, in particular, that, if A11 is nonsingular, then one generalized inverse of A is the n m matrix 1 A11 0 : GD 0 0 The r r submatrix A11 may be singular. For example, if 0 1 1 0 5 A D @0 0 0A ; 5 0 2 then r D 2 and A11 D

1 0 : 0 0

If A11 is singular or equivalently (in light of Theorem 4.4.10) if the first r rows or first r columns of A are linearly dependent, then the expression given by Theorem 9.2.1 for the general form of a generalized inverse is not applicable to A. The following theorem, which generalizes Theorem 9.2.1, gives an expression for the general form of a generalized inverse of an m n matrix A that is applicable even if the first r rows or columns of A are linearly dependent. Theorem 9.2.3. Let A represent an m n matrix of rank r. Let i1 ; : : : ; ir represent integers, chosen from the first m positive integers 1; : : : ; m in such a way that the i1 ; : : : ; ir th rows of A are linearly independent, and let j1 ; : : : ; jr represent integers, chosen from the first n positive integers 1; : : : ; n in such a way that the j1 ; : : : ; jr th columns of A are linearly independent. Take P to be any m m permutation matrix that has as its first r rows the i1 ; : : : ; ir th rows of the m m identity matrix, and Q to be any n n permutation matrix that has as its first r columns the j1 ; : : : ; jr th columns of the n n identity matrix. Let B D P AQ, and partition B as B11 B12 ; BD B21 B22 where B11 is of dimensions r r. Then, an n m matrix G is a generalized inverse of A if and only if 1 1 1 1 1 B11 B12 Y B11 B12 ZB21 B11 X B11 XB21 B11 P (2.3) GDQ Y Z for some matrices X, Y , and Z (of appropriate dimensions). With regard to Theorem 9.2.3, it follows from Theorem 4.4.10 that A contains r linearly independent rows and r linearly independent columns. Furthermore, if i1 < < ir and j1 < < jr , then B11 is a submatrix of A, namely, the


113

r r submatrix obtained by striking out all of the rows and columns of A except the i1 ; : : : ; ir th rows and j1 ; : : : ; jr th columns—more generally, B11 is an r r matrix obtained from that submatrix by permuting its rows and columns. That B11 is invertible is a consequence of Theorem 4.4.10. If the first r rows and first r columns of A are linearly independent, then i1 ; : : : ; ir and j1 ; : : : ; jr can be chosen to be the first r positive integers 1; : : : ; r, and P and Q to be identity matrices, in which case Theorem 9.2.3 reduces to Theorem 9.2.1. It is convenient, in proving Theorem 9.2.3, to make use of the following lemma, which generalizes result (8.2.8) and is of interest in its own right. Lemma 9.2.4. Let B represent an m n matrix and G an n m matrix. Then, for any m m nonsingular matrix A and n n nonsingular matrix C, (1) G is a generalized inverse of AB if and only if G D HA1 for some generalized inverse H of B, (2) G is a generalized inverse of BC if and only if G D C 1 H for some generalized inverse H of B, and (3) G is a generalized inverse of ABC if and only if G D C 1 HA1 for some generalized inverse H of B. Proof (of Lemma 9.2.4). Parts (1) and (2) are special cases of Part (3) (those where C D I and A D I, respectively). Thus, it suffices to prove Part (3). By definition, G is a generalized inverse of ABC if and only if ABCGABC D ABC or, equivalently (in light of Lemma 8.3.1), if and only if BCGAB D B: Thus, G is a generalized inverse of ABC if and only if CGA D H for some generalized inverse H of B or, equivalently, if and only if G D C 1 HA1 for some generalized inverse H of B. Q.E.D. Proof (of Theorem 9.2.3). Since P and Q are permutation matrices, they are orthogonal. Thus, A D P 0 P AQQ0 D P 0 BQ0 ; and it follows from Lemma 9.2.4 that G is a generalized inverse of A if and only if G D QHP for some generalized inverse H of B. Moreover, it follows from Theorem 9.2.1 that an n m matrix H is a generalized inverse of B if and only if 1 1 1 1 1 B11 XB21 B11 B11 B12 Y B11 B12 ZB21 B11 X HD Y Z for some matrices X, Y , and Z. Q.E.D. By setting the matrices X, Y , and Z in expression (2.3) equal to null matrices, we find that one generalized inverse of an m n matrix A of rank r is 1 B11 0 P (2.4) GDQ 0 0 (where B11 , P , and Q—and i1 ; : : : ; ir and j1 ; : : : ; jr —are as defined in Theorem 9.2.3). Note that this generalized inverse can be formed by carrying out the following 5-step procedure:

114


(1) Find the rank r of A (by, e.g., determining the size of the largest linearly independent set that can be formed from the rows or columns of A). (2) Determine values for i1 ; : : : ; ir and j1 ; : : : ; jr (such that i1 < < ir and j1 < < jr ); that is, locate r linearly independent rows of A and r linearly independent columns. (3) Form a submatrix of A by striking out all of the rows and columns of A except the i1 ; : : : ; ir th rows and j1 ; : : : ; jr th columns—this matrix is the matrix B11 . 1 of B11 . (4) Form the inverse B11

(5) Write out the generalized inverse G of A, taking the js i t th element of G to 1 (for s D 1; : : : ; r and t D 1; : : : ; r) and taking the be the st th element of B11 other elements of G to be zero. Suppose, for example, that 0 1 6 2 2 3 5 2A : A D @ 3 1 3 1 3 1 The first and second rows of A are linearly independent (as is easily verified), and the third row is the sum of the first two, so that r D 2. Let us choose i1 D 1; i2 D 3; j1 D 2, and j2 D 4—clearly, the first and third rows and the second and fourth columns of A are linearly independent. Then, 2 3 : B11 D 1 1 Applying formula (8.1.2) for the inverse of a 2 2 nonsingular matrix, we find that 1 3 1 B11 D : 1 2 Thus, one generalized inverse of A is 0

0 B1 GDB @ 0 1

0 0 0 0

1 0 3C C: 0A 2

How many generalized inverses does an m n matrix A of rank r possess? If A is nonsingular, then (according to Theorem 8.1.4) it has a unique generalized inverse, namely A1 . Suppose that A is not nonsingular (i.e., that A is either not square or is square but singular). Then, in expression (2.3) for the general form of a generalized inverse, at least one of the three matrices X, Y , and Z is nondegenerate, and it follows from Lemma 8.3.1 that distinct choices for X, Y , and Z produce distinct values of G, that is, distinct generalized inverses. Since there are an infinite number of choices for these matrices, we conclude that A has an infinite number of generalized inverses.


115

b. Generalized inverses of symmetric matrices The expression given by Theorem 9.2.3 for the general form of a generalized inverse of an m n matrix A of rank r is, of course, applicable to a symmetric matrix. If the i1 ; : : : ; ir th rows of a symmetric matrix A are linearly independent, then clearly the corresponding (i1 ; : : : ; ir th) columns of A are likewise linearly independent. Thus, for symmetric matrices, we have the following special case of Theorem 9.2.3. Theorem 9.2.5. Let A represent an n n symmetric matrix of rank r. Let i1 ; : : : ; ir represent integers chosen from the first n positive integers 1; : : : ; n in such a way that the i1 ; : : : ; ir th rows of A are linearly independent, and take P to be any n n permutation matrix that has as its first r rows the i1 ; : : : ; ir th rows of the n n identity matrix. Let B D P AP 0 , and partition B as B11 B21 BD ; B21 B22 where B11 is of dimensions r r. Then, an n n matrix G is a generalized inverse of A if and only if 1 1 1 1 1 B XB21 B11 B11 B12 Y B11 B12 ZB21 B11 X P (2.5) G D P 0 11 Y Z for some matrices X, Y , and Z (of appropriate dimensions). As indicated in Section 8.2b, the inverse of any nonsingular symmetric matrix is symmetric. Is a generalized inverse of a singular symmetric matrix necessarily symmetric? To answer this question, observe that the generalized inverses of an n n symmetric matrix A of rank r consist of all matrices of the form G D P 0 CP , where 1 1 1 1 1 B11 B12 Y B11 B12 ZB21 B11 X B11 XB21 B11 CD Y Z and the matrices X, Y , and Z are arbitrary. (Here, P and B—and the submatrices B11 ; B12 ; B21 , and B22 of B—are as defined in Theorem 9.2.5.) Since G 0 D P 0 C 0 P , it is clear from Lemma 8.3.1 that G 0 D G (i.e., that G is symmetric) if and only if C 0 D C. Observing that B0 D P A0 P 0 D P AP 0 D B, that is, B is symmetric (so that 1 0 is symmetric—and B21 D B12 ), we B11 is symmetric—and consequently B11 find that 1 1 1 1 1 B11 B11 B12 X0 Y 0 B21 B11 B11 B12 Z0 B21 B11 Y0 : C0 D X0 Z0 It follows that C 0 D C if and only if Y 0 D X and Z0 D Z. We conclude that, if n > r > 0 (in which case A is singular and the matrices X; Y , and Z are nondegenerate),

116


then A has both symmetric and nonsymmetric generalized inverses. For example, if we take X; Y , and Z all to be null, then G is symmetric; however if we take X to be null but Y to be nonnull, then G is nonsymmetric. In conclusion, we have the following lemma. Lemma 9.2.6. Every singular symmetric matrix (of order two or more) has both symmetric and nonsymmetric generalized inverses. c. Every generalized inverse in terms of any particular generalized inverse The general form of a generalized inverse of an m n matrix A can be expressed in terms of any particular generalized inverse of A, as described in the following theorem. Theorem 9.2.7. Let A represent an m n matrix, and G any particular generalized inverse of A. Then, an n m matrix G is a generalized inverse of A if and only if (2.6) G D G C Z GAZAG for some n m matrix Z. Also, G is a generalized inverse of A if and only if G D G C .I GA/T C S.I AG/

(2.7)

for some n m matrices T and S. Proof. It is a simple exercise to verify that any matrix G that is expressible in the form (2.6) or the form (2.7) is a generalized inverse of A. Conversely, if G is any generalized inverse of A, then G D G C .G G/ GA.G G/AG D G C Z GAZAG; where Z D G G, and G D G C .I GA/G AG C .G G/.I AG/ D G C .I GA/T C S.I AG/; Q.E.D. where T D G AG and S D G G. All generalized inverses of the mn matrix A can be generated from expression (2.6) by letting Z range over all n m matrices. Alternatively, all generalized inverses of A can be generated from expression (2.7) by letting both T and S range over all n m matrices. Note that distinct choices for Z or for T or S may not result in distinct generalized inverses. d. Generalized inverses of matrices having full column or row rank The following lemma extends the results of Lemma 8.1.3.

9.3 Some Elementary Properties

117

Lemma 9.2.8. Let A represent a matrix of full column rank and B a matrix of full row rank. Then, (1) a matrix G is a generalized inverse of A if and only if G is a left inverse of A. And (2) a matrix G is a generalized inverse of B if and only if G is a right inverse of B. Proof. Let L represent any left inverse of A— that A has a left inverse is guaranteed by Lemma 8.1.1. Then, ALA D AI D A, so that L is a generalized inverse of A. And the proof of Part (1) of Lemma 9.2.8 is complete upon observing that A A D IA A D LAA A D LA D I and hence that A is a left inverse of A. The validity of Part (2) can be established via an analogous argument. Q.E.D. If a matrix A has full column rank, then (according to Corollary 7.4.6) A0 A is nonsingular, and similarly if A has full row rank, then AA0 is nonsingular. Further, if A has full column or row rank, then, by making use of the following (easily verifiable) lemma, a generalized inverse of A can (in light of Lemma 9.2.8) be obtained from the ordinary inverse of A0 A or AA0 . Lemma 9.2.9. If a matrix A has full column rank, then the matrix .A0 A/1 A0 is a left inverse of A. Similarly, if A has full row rank, then A0 .AA0 /1 is a right inverse of A.

9.3 Some Elementary Properties Let us begin by considering the extension (to generalized inverses) of various elementary properties of inverse matrices. It is easy to verify the following lemma, which is a generalization of result (8.2.1). Lemma 9.3.1. For any matrix A and any nonzero scalar k, .1=k/A is a generalized inverse of kA. Setting k D 1 in Lemma 9.3.1, we obtain the following corollary, which generalizes result (8.2.2). Corollary 9.3.2. For any matrix A, A is a generalized inverse of A. For any matrix A, we find that A0 .A /0 A0 D .AA A/0 D A0 : Thus, we have the following lemma, which generalizes result (8.2.3). Lemma 9.3.3. For any matrix A, .A /0 is a generalized inverse of A0 . While (as discussed in Section 9.2b) the transpose .A /0 of a generalized inverse A of a singular symmetric matrix A is not necessarily the same as A , Lemma 9.3.3 implies that .A /0 has the following weaker property. Corollary 9.3.4. For any symmetric matrix A, .A /0 is a generalized inverse of A.

118


For a nonsingular matrix A, we have (by definition) that AA1 D A1 A D I. The following two lemmas describe, for an arbitrary matrix A, some properties of AA and A A. Lemma 9.3.5. Let A represent an m n matrix. Then, for any m p matrix B, C.B/ C.A/ if and only if B D AA B, or, equivalently, if and only if .I AA /B D 0. And, for any q n matrix C, R.C/ R.A/ if and only if C D CA A, or, equivalently, if and only if C.I A A/ D 0. Proof. If B D AA B, then it follows immediately from Lemma 4.2.2 that C.B/ C.A/. Conversely, if C.B/ C.A/, then, according to the same lemma, there exists a matrix F such that B D AF , implying that B D AA AF D AA B: Thus, C.B/ C.A/ if and only if B D AA B. That R.C/ R.A/ if and only Q.E.D. if C D CA A follows from an analogous argument. In the special case where p D 1 and q D 1, Lemma 9.3.5 reduces to the following corollary. Corollary 9.3.6. Let A represent an mn matrix. Then, for any m-dimensional column vector x, x 2 C.A/ if and only if x D AA x, and, for any n-dimensional row vector y 0 , y 0 2 R.A/ if and only if y 0 D y 0 A A. Lemma 9.3.7. For any matrix A, R.A A/ D R.A/

and

C.AA / D C.A/:

Proof. It follows from Corollary 4.2.3 that R.A A/ R.A/ and also, since A D A.A A/, that R.A/ R.A A/. Thus, R.A A/ D R.A/. That Q.E.D. C.AA / D C.A/ follows from an analogous argument. As an immediate consequence of Lemma 9.3.7, we have the following corollary. Corollary 9.3.8. For any matrix A, rank.A A/ D rank.AA / D rank.A/: It follows from this corollary, together with Corollary 4.4.5, that, for any matrix A,

rank.A / rank.A A/ D rank.A/:

Thus, we have the following lemma, which extends result (8.2.5). Lemma 9.3.9. For any matrix A, rank.A / rank.A/: The following two lemmas give some elementary results on generalized inverses of partitioned matrices.

9.4 Invariance to the Choice of a Generalized Inverse

119

Lemma 9.3.10. Let A and B represent m n partitioned matrices of the form W G1 (where A D .T ; 0/ and B D . Then, an nm partitioned matrix G D G2 0 the number of rows in G1 equals the number of columns in T ) is a generalized inverse of A if and only if G1 is a generalized inverse of T . Similarly, an n m partitioned matrix H D .H1 ; H2 / (where the number of columns in H1 equals the number of rows in W ) is a generalized inverse of B if and only if H1 is a generalized inverse of W . Proof. The lemma becomes obvious upon observing that AGA D .T G1 T ; 0/ W H1 W and BHB D . Q.E.D. 0 Lemma 9.3.11. Let A represent an m n matrix, B an m p matrix, and C a A is a generalized inverse of the q n matrix. Then, the .n C p/ m matrix B partitioned matrix .A; B/ if and only if AA B D 0 and BB A D 0. Similarly, the n .m C q/ matrix .A ; C / is a generalized inverse of the partitioned matrix A if and only if CA A D 0 and AC C D 0. C Proof. The proof becomes obvious upon observing that A .A; B/ D .AA A C BB A; AA B C BB B/ .A; B/ B D .A C BB A; AA B C B/ and that A C AC C A AA A C AC C A D : D .A ; C / C CA A C CC C CA A C C C Q.E.D.

9.4 Invariance to the Choice of a Generalized Inverse Suppose that a generalized inverse A of a matrix A is premultiplied by a matrix B and postmultiplied by a matrix C. Under certain conditions, the product BA C is invariant to the choice of A . These conditions are given by the following theorem. Theorem 9.4.1. Let A represent an m n matrix, B a p n matrix, and C an m q matrix. If R.B/ R.A/ and C.C/ C.A/, then BA C is invariant to the choice of the generalized inverse A . Conversely, if BA C is invariant to the choice of A and if in addition C is nonnull, then R.B/ R.A/; and, if BA C is invariant to the choice of A and if in addition B is nonnull, then C.C/ C.A/. Proof. Suppose that R.B/ R.A/ and C.C/ C.A/. Then, there exist matrices L and R such that B D LA and C D AR, in which case

120


BA C D LAA AR D LAR: Thus, BA C is invariant to the choice of A . Conversely, suppose that BA C is invariant to the choice of A . Then, letting G represent any particular choice for A , it follows from Theorem 9.2.7 that BGC D BŒG C .I GA/T C S.I AG/C for all n m matrices T and S, or equivalently that B.I GA/T C C BS.I AG/C D 0 for all T and S, and hence that B.I GA/T C D 0 for all T and BS.I AG/C D 0 for all S. Suppose further that C is nonnull. Then, some column, say the kth column ck , of C is nonnull. And, letting ui represent the i th column of In , there exists an n m matrix Ti such that Ti ck D ui (i D 1; : : : ; n). We have (for i D 1; : : : ; n) that B.IGA/Ti C D 0. Moreover, the kth column of B.I GA/Ti C is B.I GA/Ti ck D B.I GA/ui , which is the i th column of B.I GA/. Thus, the i th column of B.I GA/ is null (i D 1; : : : ; n). We conclude that B.I GA/ D 0 and (in light of Lemma 9.3.5) that R.B/ R.A/. That C.C/ C.A/ if B is nonnull (and BA C is invariant to the choice of Q.E.D. A ) can be established in similar fashion. Theorem 9.4.1 has the following implication. Corollary 9.4.2. Let A represent a q p matrix, B a p n matrix, and C an mq matrix. If rank.CAB/ D rank.C/ D rank.B/, then B.CAB/ C is invariant to the choice of the generalized inverse .CAB/ . Proof. Suppose that rank.CAB/ D rank.C/ D rank.B/. Then, it follows from Corollary 4.4.7 that R.B/ D R.CAB/ and C.C/ D C.CAB/. Based on Theorem 9.4.1, we conclude that B.CAB/ C is invariant to the choice of Q.E.D. .CAB/ .

9.5 A Necessary and Sufficient Condition for the Consistency of a Linear System According to Theorem 7.2.1, a linear system AX D B (in X) is consistent if and only if C.B/ C.A/. In light of Lemma 9.3.5, this result can be restated as follows:

9.6 Some Results on the Ranks and Generalized Inverses of Partitioned Matrices

121

Lemma 9.5.1. A linear system AX D B (in X) is consistent if and only if AA B D B or, equivalently, if and only if .I AA /B D 0: According to Lemma 9.5.1, either of the two matrices AA or I AA can be used to determine whether any linear system having A as a coefficient matrix is consistent or inconsistent. If the right side of the linear system is unaffected by premultiplication by AA , then the linear system is consistent; otherwise, it is inconsistent. Similarly, if the premultiplication of the right side by I AA produces a null matrix, then the linear system is consistent; otherwise, it is inconsistent. Consider, for example, the linear system Ax D b (in x), where 0 1 6 2 2 3 5 2A : A D @ 3 1 3 1 3 1 As determined in Section 9.2a, one generalized inverse of A is 0 1 0 0 0 B1 0 3C C GDB @ 0 0 0A : 1 0 2 Clearly,

If b D .3; 2; 5/0 , then

0

1 1 0 0 AG D @1 0 1A : 0 0 1 AGb D .3; 2; 5/0 D b;

in which case the linear system Ax D b is consistent. However, if b D .1; 2; 1/0 , then AGb D .1; 0; 1/0 ¤ b; in which case Ax D b is inconsistent.

9.6 Some Results on the Ranks and Generalized Inverses of Partitioned Matrices a. Basic results Consider the partitioned matrix A D

T U . In the special case where T is V W

122


nonsingular, Theorem 8.5.10 gives rank.A/ in terms of the Schur complement Q D W V T 1 U , which is of smaller dimensions than A. Moreover, in the special case where A, as well as T , is nonsingular, Theorem 8.5.11 gives A1 in terms of T 1 and Q1 (and U and V ). The following theorem extends the results of Theorems 8.5.10 and 8.5.11 to a larger class of partitioned matrices. Theorem 9.6.1. Let T represent an m p matrix, U an m q matrix, V an n p matrix, and W an n q matrix, and define Q D W V T U . Suppose that C.U / C.T / and R.V / R.T /. Then, T U W V rank D rank D rank.T / C rank.Q/: (6.1) V W U T Further, the partitioned matrices T C T U Q V T T U Q Q V T Q T 0 T U Q .V T ; In / D C 0 0 Iq

(6.2a) (6.2b)

and

Q Q V T T U Q T C T U Q V T Iq 0 0 C Q .In ; V T / D T U 0 T

T U are generalized inverses of V W

(6.3a) (6.3b)

W V and , respectively. U T

In the special case of a block-triangular matrix, Theorem 9.6.1 can be restated as the following result, which generalizes results (8.5.7) and (8.5.8). Corollary 9.6.2. Let T represent an m p matrix, V an n p matrix, and W an n q matrix. If R.V / R.T / or C.V / C.W /, then T 0 W V rank D rank D rank.T / C rank.W /; V W 0 T and the block-triangular matrices T 0 W V T W

T 0 are generalized inverses of V W

and

W W V T 0 T

W V and , respectively. 0 T

As a special case of Corollary 9.6.2, we obtain the following generalization of result (8.5.2).


123

T 0 Corollary 9.6.3. For any matrices T and W , the partitioned matrix 0 W T 0 is a generalized inverse of the block-diagonal matrix . 0 W To prove Theorem 9.6.1, let us first prove the following result, which is of some interest in its own right. Theorem 9.6.4. Let T represent an m p matrix, R a p q matrix, L an n m matrix, and W an n q matrix, and define Q D W LT R. Then, T TR W LT rank D rank D rank.T / C rank.Q/: (6.4) LT W TR T Furthermore, the partitioned matrices T C RQ L RQ T 0 R D Q .L; In / C Q L Q 0 0 Iq and

(6.5)

Q Iq 0 0 Q L C D Q .In ; L/ (6.6) 0 T R RQ T C RQ L T TR W LT are generalized inverses of and , respectively. LT W TR T Proof (of Theorem 9.6.4). Observe that I 0 T TR I R T 0 D (6.7) L I LT W 0 I 0 Q I 0 I R and that (according to Lemma 8.5.2) the matrices and are L I 0 I nonsingular. Thus, making use of Corollary 8.3.3 and Lemma 4.5.11, we find that T TR T 0 rank D rank D rank.T / C rank.Q/; LT W 0 Q which (in light of Lemma 8.5.1) establishes result (6.4). Equality (6.7) further implies that 1 1 T 0 I R T TR I 0 : D 0 Q 0 I LT W L I Observing that T 0 T T 0 T 0 0 D 0 Q 0 Q 0 Q 0 Q T T 0 0 and hence that is a generalized inverse of , we conclude, on 0 Q 0 Q

124


the basis of Lemma 9.2.4, that the matrix I R T I 0 T C RQ L RQ 0 D 0 I L I Q L Q 0 Q T TR is a generalized inverse of . That matrix (6.6) is a generalized inverse LT W W LT of follows from a similar argument. (An alternative way of estabTR T T TR lishing that matrices (6.5) and (6.6) are generalized inverses of and LT W W LT , respectively, is by direct verification; that is, by premultiplying and TR T T TR postmultiplying matrix (6.5) by and premultiplying and postmultiLT W W LT plying matrix (6.6) by and by verifying that the resultant products TR T T TR W LT equal and , respectively.) Q.E.D. LT W TR T Proof (of Theorem 9.6.1). According to Lemma 9.3.5, U D T T U and V D V T T . It follows that T U T T T U W V W V T T D and D : V W W U T T V T T T T U Thus, Theorem 9.6.1 can be obtained from Theorem 9.6.4 by setting R D T U and L D V T and observing that LT R D V T T T U D V T U : Q.E.D. The matrix Q D W V T U , which appears in formulas (6.1)–(6.3) for the rank and generalized inverse matrix, is called the Schur comple of a partitioned T U W V ment of T in or relative to T . Moreover, when the context V W U T is clear, Q is sometimes referred to simply as the Schur complement or the Schur complement of T . In general, the Schur complement varies with the choice of the generalized inverse T . However, in the special case where the conditions of Theorem 9.6.1 are satisfied [i.e., where C.U / C.T / and R.V / R.T /], it follows from Theorem 9.4.1 that V T U is invariant to the choice of the generalized of T is invariant to the choice inverse T and hencethat theSchurcomplement T U W V of T . Moreover, if or is symmetric (or, equivalently, if V W U T T 0 D T , W 0 D W , and V D U 0 ), then, in the special case where C.U / C.T /, the Schur complement of T is symmetric (as well as invariant to the choice of


125

T ), as is evident upon observing that (in this special case) there exists a matrix R such that U D T R and hence such that Q D W R0 T T T R D W R0 T R: b. A further result According to Theorem 9.6.1, there exists a generalized inverse of a partitioned T U matrix A D [where C.U / C.T / and R.V / R.W /] whose V W lower right corner is a generalized inverse of the Schur complement Q D W V T U . The following theorem indicates that every generalized inverse of A has this property. Theorem 9.6.5. Let T represent an m p matrix, U an m q matrix, V an n p matrix, and W an n q matrix, and define Q D W V T U . Suppose that C.U / C.T / and R.V / R.T /. Then, for any generalized inverse B D T U B11 B12 of the partitioned matrix , the (q n) submatrix B22 is a V W B21 B22 C11 C12 generalized inverse of Q. Similarly, for any generalized inverse C D C21 C22 W V of the matrix , the (q n) submatrix C11 is a generalized inverse of Q. U T T U Proof. Let A D , and take G to be the generalized inverse of A V W given by expression (6.2). Then, according to Theorem 9.2.7, B D G C Z GAZAG for some .p C q/ .m C n/ matrix Z. Thus, partitioning Z as Z11 Z12 ZD Z21 Z22 (where the dimensions of Z22 are q n) and observing (in light of Lemma 9.3.5) that U D T T U and V D V T T , we find that T U Q B22 D Q C Z22 .Q V T ; Q /AZA Q 0 D Q C Z22 .0; Q Q/Z QQ D Q C Z22 Q QZ22 QQ and hence that QB22 Q D QQ Q C QZ22 Q QQ QZ22 QQ Q D Q: We conclude that B22 is a generalized inverse of Q. That C11 is a generalized inverse of Q can be established via an analogous argument. Q.E.D.

126


9.7 Extension of Some Results on Systems of the Form AX D B to Systems of the Form AXC D B It was established in Section 9.5 that a linear system AX D B (with an m n coefficient matrix A, m p right side B, and n p matrix of unknowns X) is consistent if and only if AA B D B, in which case A B is a solution to AX D B. Consider now the generalization of this result to a system of the form AXC D B, where A is an m n matrix, C a p q matrix, B an m q matrix, and X an n p matrix of unknowns. A system of this form (like one of the form AX D B) is said to be consistent if it has one or more solutions. The desired generalization is given by the following theorem. Theorem 9.7.1. The system AXC D B (in X) is consistent if and only if AA BC C D B, or equivalently if and only if AA B D B and BC C D B, in which case A BC is a solution to AXC D B. Proof. If AA BC C D B, then clearly A BC is a solution to the system AXC D B, and this system is consistent. Conversely, suppose that the system AXC D B is consistent. Then, by definition, this system has a solution, say X . Thus, AA BC C D AA AX CC C D AX C D B: It remains to show that AA BC C D B if and only if AA B D B and BC C D B. If AA B D B and BC C D B, then clearly AA BC C D BC C D B. Conversely, if AA BC C D B, then clearly AA B D AA AA BC C D AA BC C D B and similarly BC C D AA BC CC C D AA BC C D B: Q.E.D. In light of Lemma 9.3.5, we have the following corollary, which generalizes Part (1) of Theorem 7.2.1. Corollary 9.7.2. The system AXC D B (in X) is consistent if and only if C.B/ C.A/ and R.B/ R.C/.

Exercises Section 9.1 1. Let A represent any m n matrix and B any m p matrix. Show that if AHB D B for some n m matrix H, then AGB D B for every generalized inverse G of A.

Exercises

127

2. (a) Let A represent an m n matrix. Show that any n m matrix X such that A0 AX D A0 is a generalized inverse of A and similarly that any n m matrix Y such that AA0 Y 0 D A is a generalized inverse of A. (b) Use Part (a), together with Corollary 7.4.2, to arrive at the same conclusion as Corollary 9.1.4. 3. Let A represent an m n nonnull matrix, let B represent a matrix of full column rank and T a matrix of full row rank such that A D BT , and let L represent a left inverse of B and R a right inverse of T . (a) Show that the matrix R.B0 B/1 R0 is a generalized inverse of the matrix A0 A and that the matrix L0 .T T 0 /1 L is a generalized inverse of the matrix AA0 . (b) Show that if A is symmetric, then the matrix R.T B/1 L is a generalized inverse of the matrix A2 . (If A is symmetric, then it follows from the result of Exercise 8.3 that T B is nonsingular.) Section 9.2 4. Use formula (2.4) to find a generalized inverse of the matrix 0 1 0 0 0 B0 4 2C B C C: 0 2 1 ADB B C @0 3 3A 0 8 4 5. Let A represent an m n nonnull matrix of rank r. Take B and K to be nonsingular matrices (of orders m and n, respectively) such that I 0 ADB r K 0 0 (the existence of which is guaranteed by Theorem 4.4.9). Show (a) that an n m matrix G is a generalized inverse of A if and only if G is expressible in the form I U (E.1) G D K1 r B1 V W for some r .m r/ matrix U , .n r/ r matrix V , and .n r/ .m r/ matrix W , and (b) that distinct choices for U , V , and/or W lead to distinct generalized inverses. Section 9.3 6. Let A represent an m n matrix, B an m p matrix, and C a q n matrix, and let k represent a nonzero scalar. Generalize Lemma 9.3.1 by showing (a) that,

128


G1 of the partitioned matrix .A; B/ (where G1 for any generalized inverse G 2 G1 is a generalized inverse of .A; kB/ and is of dimensions n m), k 1 G2 A (b) that, for any generalized inverse .H1 ; H2 / of the partitioned matrix C (where H is of dimensions n m), .H1 ; k 1 H2 / is a generalized inverse of 1 A . kC Section 9.6 7. Let T represent an m p matrix and W an n q matrix. (a) Show that, unless T and W are both nonsingular, there exist generalized T 0 T 0 . [Hint. Make inverses of that are not of the form 0 W 0 W use of result (2.7).] (b) Take U to be an mq matrix and V an np matrix such that C.U / C.T / and R.V / R.T /. Generalize the result of Part (a) by showing that, unless T and the matrix Q D W V T U are both nonsingular, there T U exist generalized inverses of that are not of the form (6.2). [Hint. V W Use Part (a), together with Lemma 9.2.4.] 8. Let T represent an m p matrix, U an m q matrix, V an n p matrix, and T U W an n q matrix, take A D , and define Q D W V T U . V W (a) Show that the matrix T C T U Q V T T U Q GD Q V T Q is a generalized inverse of the matrix A if and only if (1) .I T T /U .I Q Q/ D 0, (2) .I QQ /V .I T T / D 0, and (3) .I T T /U Q V .I T T / D 0. (b) Verify that the conditions [C.U / C.T / and R.V / R.T /] of Theorem 9.6.1 imply Conditions (1)–(3) of Part (a). (c) Exhibit matrices T , U , V , and W that (regardless of how the generalized inverses T and Q are chosen) satisfy Conditions (1)–(3) of Part (a) but do not satisfy (both of) the conditions of Theorem 9.6.1. 9. (a) Suppose that a matrix A is partitioned as

Exercises

129

1 A11 A12 A13 A D @A21 A22 A23 A A31 A32 A33 0

and that C.A12 / C.A11 / and R.A21 / R.A11 /. Take Q to be the Schur complement of A11 in A relative to A 11 , and partition Q as Q11 Q12 QD Q21 Q22 (where Q11 , Q12 , Q21 , and Q22 are of the same dimensions as A22 , A23 , A32 , and A33 , respectively), so that Q11 D A22 A21 A 11 A12 , Q12 D A23 A21 A 11 A13 , Q21 D A32 A31 A11 A12 , and Q22 D A33 A31 A 11 A13 . Let A11 C A 11 A12 Q11 A21 A11 A11 A12 Q11 GD : Q Q 11 A21 A11 11 A13 A11 A12 , U D , and V D .A31 ; A32 /, or Define T D A21 A22 A23 equivalently define T , U , and V to satisfy T U AD : V A33 Show that: (1) G is a generalized inverse of T ; (2) the Schur complement Q22 Q21 Q 11 Q12 of Q11 in Q relative to Q 11 equals the Schur complement A33 V GU of T in A relative to G; A11 A13 A 11 A12 Q11 Q12 , and (3) GU D Q 11 Q12 V G D .A31 A 11 Q21 Q11 A21 A11 ; Q21 Q11 /. (b) Let A represent an n n matrix (where n 2), let n1 ; : : : ; nk represent positive integers such that n1 C C nk D n (where k 2), and (for i D 1; : : : ; k) let ni D n1 C C ni . Define (for i D 1; : : : ; k) Ai to be the leading principal submatrix of A of order ni and define (for i D 1; : : : ; k 1) Ui to be the ni .n ni / matrix obtained by striking out all of the rows and columns of A except the first ni rows and the last n ni columns, Vi to be the .n ni / ni matrix obtained by striking out all of the rows and columns of A except the last n ni rows and first ni columns, and Wi to be the .n ni / .n ni / submatrix obtained by striking out all of the rows and columns of A except the last n ni rows and columns, so that (for i D 1; : : : ; k 1) Ai U i : AD Vi Wi

130


Suppose that (for i D 1; : : : ; k 1) C.Ui / C.Ai / and R.Vi / R.Ai /. Let ! .i/ .i/ B11 B12 .i/ B D .i/ .i/ B21 B22 .k/ .1/ .1/ (i D 1; : : : ; k 1) and B.k/ D B11 , where B11 D A 1 , B12 D A1 U1 , .1/ .1/ .i/ B21 D V1 A 1 , and B22 D W1 V1 A1 U1 , and where (for i 2) B11 , .i/ .i/ .i/ .i1/ .i1/ B12 , B21 , and B22 are defined recursively by partitioning B12 , B21 , .i1/ and B22 as ! Y1.i1/ .i1/ .i1/ .i1/ .i1/ B21 D ; B12 D .X1 ; X2 /; Y2.i1/

and .i1/ B22

D

Q.i1/ Q.i1/ 11 12

!

Q.i1/ Q.i1/ 21 22

(in such a way that X.i1/ has ni columns, Y1.i1/ has ni rows, and Q.i1/ 1 11 is of dimensions ni ni ) and (using Q.i1/ to represent a generalized 11 inverse of Q.i1/ ) by taking 11 ! .i1/ C X.i1/ Q.i1/ Y1.i1/ X.i1/ Q.i1/ B11 .i/ 1 11 1 11 B11 D ; Q.i1/ Y1.i1/ Q.i1/ 11 11 ! X.i1/ Q.i1/ Q.i1/ X.i1/ .i/ 2 1 11 12 B12 D ; Q.i1/ Q.i1/ 11 12 .i/ D .Y2.i1/ Q.i1/ Q.i1/ Y1.i1/ ; Q.i1/ Q.i1/ /; B21 21 11 21 11

and

.i/ B22 D Q.i1/ Q.i1/ Q.i1/ Q.i1/ : 22 21 11 12

Show that: .i/ is a generalized inverse of Ai (i D 1; : : : ; k); (1) B11 .i/ .i/ is the Schur complement of Ai in A relative to B11 (i D (2) B22 1; : : : ; k 1); .i/ .i/ .i/ .i/ D B11 Ui , and B21 D Vi B11 (i D 1; : : : ; k 1). (3) B12

[Note. The recursive formulas given in Part (b) for the sequence of matrices B.1/ ; : : : ; B.k1/ ; B.k/ can be used to generate B.k1/ in k 1 steps or to generate B.k/ in k steps—the formula for generating B.i/ from B.i1/ involves a generalized inverse of the ni ni matrix Q.i1/ . The various 11 .k1/ of Ak1 , the Schur parts of B.k1/ consist of a generalized inverse B11

.k1/ .k1/ .k1/ of Ak1 in A relative to B11 , a solution B12 of complement B22

Exercises

131

.k1/ the linear system Ak1 X D Uk1 (in X), and a solution B21 of the .k/ linear system Y Ak1 D Vk1 (in Y ). The matrix B is a generalized inverse of A. In the special case where ni D 1, the process of generating the elements of the n n matrix B.i/ from those of the n n matrix B.i1/ is called a sweep operation—see, e.g., Goodnight (1979).]

10. Let T represent an m p matrix and W an n q matrix, and let G D G11 G12 (where G11 is of dimensions p m) represent an arbitrary G21 G22 generalized inverse of the .m C n/ .p C q/ block-diagonal matrix A D T 0 . Show that G11 is a generalized inverse of T and G22 a generalized 0 W inverse of W . Show also that T G12 W D 0 and W G21 T D 0. 11. Let T represent an mp matrix, U an mq matrix, V an np matrix, and W an n q matrix, and define Q D W V T U . Suppose that C.U / C.T / and R.V / R.T /. Devise an alternative proof of the first part of Theorem 9.6.5 by showing that I 0 T U I T U T 0 D V T I V W 0 I 0 Q and by then using Lemma 9.2.4 and the result of Exercise 10 to show that, G11 G12 of the partitioned matrix for any generalized inverse G D G21 G22 T U , the (q n) submatrix G22 is a generalized inverse of Q. V W 12. Let T represent an m p matrix, U an m q matrix, V an n p matrix, and T U W an n q matrix, and take A D . V W G11 G12 , where G11 D (a) Define Q D W V T U , and let G D G21 G22 T CT U Q V T , G12 D T U Q , G21 D Q V T , and G22 D Q . Show that the matrix G21 G11 G12 G22

is a generalized inverse of T . {Note. If the conditions [C.U / C.T / and R.V / R.T /] of Theorem 9.6.1 are satisfied or, more generally, if Conditions (1)–(3) of Part (a) of Exercise 8 are satisfied, then G is a generalized inverse of A.} (b) Show by example that, forsome values of T , U , V , and W , there exists a G11 G12 of A (where G11 is of dimensions generalized inverse G D G21 G22

132


p m, G12 of dimensions p n, G21 of dimensions q m, and G22 of dimensions q n) such that the matrix G11 G12 G22 G21

is not a generalized inverse of T .

10 Idempotent Matrices

This chapter is devoted to a very important class of matrices called idempotent matrices. It provides coverage of some basic properties of idempotent matrices and also of some basic results pertaining to idempotent matrices. Idempotent matrices play an important role in the theory of linear statistical models (especially in connection with the theory of least squares and the analysis of variance) and (not coincidentally) appear prominently in several of the ensuing chapters of this book (including Chapters 12 and 17). Making idempotent matrices the subject of a separate chapter (even though this results in a very short chapter) is convenient and serves to emphasize their importance.

10.1 Definition and Some Basic Properties A square matrix A is said to be idempotent if A2 D A: Examples of n n idempotent matrices are the identity matrix In , the n n null matrix 0, and the matrix .1=n/Jn , each element of which equals 1=n. As indicated by the following lemma, n n idempotent matrices are, with one exception, singular. Lemma 10.1.1. The only n n idempotent matrix of rank n is the n n identity matrix In . Proof. Suppose that A is an n n idempotent matrix of rank n. Then, A D In A D A1 AA D A1 A D In : Q.E.D. If a square matrix A is idempotent, then .A0 /2 D .AA/0 D A0

134

and


.I A/2 D I 2A C A2 D I 2A C A D I A:

Thus, upon observing that A D .A0 /0 and A D I .I A/, we have the following lemma. Lemma 10.1.2. Let A represent a square matrix. Then, (1) A0 is idempotent if and only if A is idempotent, and (2) I A is idempotent if and only if A is idempotent. For any idempotent matrix A, we have that AAA D .AA/A D AA D A, giving rise to the following lemma. Lemma 10.1.3. If a (square) matrix A is idempotent, then A is a generalized inverse of itself (i.e., of A).

10.2 Some Basic Results Suppose that A is a square matrix such that A2 D kA for some nonzero scalar k. Or, equivalently, suppose that Œ.1=k/A2 D .1=k/A; that is, .1=k/A is an idempotent matrix (so that, depending on whether k D 1 or k ¤ 1, A is either an idempotent matrix or a scalar multiple of an idempotent matrix). Then, in light of the following theorem, rank.A/ D .1=k/ tr.A/; and consequently the rank of A is determinable from the trace of A. Theorem 10.2.1. For any square matrix A such that A2 D kA for some scalar k, tr.A/ D k rank.A/: Proof. Let us restrict attention to the case where A is nonnull. [The case where A D 0 is trivial—if A D 0, then tr.A/ D 0 D k rank.A/.] Let n denote the order of A, and let r D rank.A/. Then, according to Theorem 4.4.8, there exists an n r matrix B and an r n matrix L such that A D BL. Moreover, rank.B/ D rank.L/ D r. We have that BLBL D A2 D kA D kBL D B.kI/L; implying (in light of Lemma 8.3.1) that LB D kI:

10.2 Some Basic Results

135

Thus, making use of Lemma 5.2.1, we find that tr.A/ D tr.BL/ D tr.LB/ D tr.kI/ D k tr.Ir / D kr: Q.E.D. In the special case where k D 1, Theorem 10.2.1 can be restated as follows. Corollary 10.2.2. For any idempotent matrix A, rank.A/ D tr.A/: Upon observing that only a null matrix can be of rank zero, we obtain the following, additional corollary (of Theorem 10.2.1). Corollary 10.2.3. If the trace of an idempotent matrix A equals 0, then A D 0. Making use of Lemma 10.1.2 and Corollary 10.2.2, we find that, for any n n idempotent matrix A, rank.I A/ D tr.I A/ D tr.In / tr.A/ D n rank.A/; thereby establishing the following lemma. Lemma 10.2.4. For any n n idempotent matrix A, rank.I A/ D tr.I A/ D n rank.A/: For any matrix A, A AA A D A A and AA AA D AA . Thus, we have the following lemma. Lemma 10.2.5. Let A represent an m n matrix. Then, the n n matrix A A and the m m matrix AA are both idempotent. Note that this lemma, together with Corollary 9.3.8 and Corollary 10.2.2, implies that, for any matrix A, rank.A/ D rank.A A/ D rank.AA / D tr.A A/ D tr.AA /:

(2.1)

In light of Lemma 10.2.4 [and of result (2.1)], a further implication of Lemma 10.2.5 is as follows. Lemma 10.2.6. Let A represent an m n matrix. Then, rank.I A A/ D tr.I A A/ D n rank.A/; rank.I AA / D tr.I AA / D m rank.A/:

(2.2) (2.3)

The following theorem expands on the results of Lemma 10.2.5 and of Corollary 9.3.8. Theorem 10.2.7. Let A represent an mn matrix and B an nm matrix. Then, B is a generalized inverse of A if and only if BA is idempotent and rank.BA/ D rank.A/. Similarly, B is a generalized inverse of A if and only if AB is idempotent and rank.AB/ D rank.A/.

136


Proof. In light of Lemma 10.2.5 and Corollary 9.3.8, it suffices to show that B is a generalized inverse of A if BA is idempotent and rank.BA/ D rank.A/ or if AB is idempotent and rank.AB/ D rank.A/. Suppose that BA is idempotent and rank.BA/ D rank.A/. Then, according to Corollary 4.4.7, R.A/ D R.BA/, and it follows from Lemma 4.2.2 that A D SBA for some matrix S. Thus, ABA D SBABA D SBA D A: Alternatively, suppose that AB is idempotent and rank.AB/ D rank.A/. Then, according to Corollary 4.4.7, C.A/ D C.AB/, and consequently A D ABT for some matrix T . Thus, ABA D ABABT D ABT D A: Q.E.D.

Exercises Section 10.1 1. Show that if an n n matrix A is idempotent, then (a) for any n n nonsingular matrix B, B1 AB is idempotent; (b) for any integer k greater than or equal to 2, Ak D A. 2. Let P represent an m n matrix (where m n) such that P 0 P D In , or equivalently an m n matrix whose columns are orthonormal (with respect to the usual inner product). Show that the m m symmetric matrix P P 0 is idempotent. 3. Show that, for any symmetric idempotent matrix A, the matrix I 2A is orthogonal. 4. Let A represent an m n matrix. Show that if A0 A is idempotent, then AA0 is idempotent. 5. Let A represent a symmetric matrix and k an integer greater than or equal to 1. Show that if AkC1 D Ak , then A is idempotent. 6. Let A represent an n n matrix. Show that .1=2/.I C A/ is idempotent if and only if A is involutory (where involutory is as defined in Exercise 8.2). 7. Let A and B represent n n symmetric idempotent matrices. Show that if C.A/ D C.B/, then A D B.

Exercises

137

8. Let A represent an r m matrix and B an m n matrix. (a) Show that B A is a generalized inverse of AB if and only if A ABB is idempotent. (b) Show that if A has full column rank or B has full row rank, then B A is a generalized inverse of AB. Section 10.2 9. Let T represent an m p matrix, U an m q matrix, V an n p matrix, and W an n q matrix, and define Q D W V T U . Using Part (a) of Exercise 9.8, together with result (2.1), show that if (1) .I T T /U .I Q Q/ D 0, (2) .I QQ /V .I T T / D 0, and (3) .I T T /U Q V .I T T / D 0,

then rank

T U V W

D rank.T / C rank.Q/:

10. Let T represent an m p matrix,U an m q matrix, V an n p matrix, T U and W an n q matrix, take A D , and define Q D W V T U . V W Further, let ET D I T T ; FT D I T T ; X D ET U ; Y D V FT ; EY D I Y Y ; FX D I X X; Z D EY QFX ; and Q D FX Z EY : (a) (Meyer 1973, Theorem 3.1) Show that the matrix G D G1 C G 2 ; where

0

B B G1 D B @

1 T T U .I Q Q/X ET FT Y .I QQ /V T FT Y .I QQ / C C C FT Y .I QQ /QX ET A .I Q Q/X ET

and

(E.1)

0

T U G2 D Q .V T ; In /; Iq

is a generalized inverse of A. (b) (Meyer 1973, Theorem 4.1) Show that rank.A/ D rank.T / C rank.X/ C rank.Y / C rank.Z/: [Hint. Use Part (a) together with result (2.1).]

(E.2)

138


(c) Show that if C.U / C.T / and R.V / R.T /, then formula (E.2) for rank.A/ reduces to formula (9.6.1), and formula (9.6.2) for a generalized inverse of A can be obtained as a special case of formula (E.1).

11 Linear Systems: Solutions

The subject of this chapter is linear systems. This subject was considered previously in Chapter 7. The emphasis in Chapter 7 was on whether the linear system is consistent, that is, whether it has solutions, whereas the emphasis in the present chapter is on the solutions themselves. The presentation makes heavy use of the results of Chapters 9 and 10 (on generalized inverses and idempotent matrices).

11.1 Some Terminology, Notation, and Basic Results The collection of all solutions to a linear system AX D B (in X) is called the solution set of the linear system. Clearly, a linear system is consistent if and only if its solution set is nonempty. Is the solution set of a linear system AX D B (in X) a linear space? The answer depends on whether the linear system is homogeneous or nonhomogeneous, that is, on whether the right side B is null or nonnull. Consider first the solution set of a homogeneous linear system AX D 0. Since a homogeneous linear system is consistent, the solution set of AX D 0 is nonempty. Furthermore, if X1 and X2 are solutions to AX D 0 and k is a scalar, then A.X1 C X2 / D AX1 C AX2 D 0, and A.kX1 / D k.AX1 / D 0, so that X1 C X2 and kX1 are also solutions to AX D 0. Thus, the solution set of a homogeneous linear system is a linear space. Accordingly, the solution set of a homogeneous linear system AX D 0 may be called the solution space of AX D 0. The solution space of a homogeneous linear system Ax D 0 (in a column vector x) is called the null space of the matrix A and is denoted by the symbol N.A/. Thus, for any m n matrix A, N.A/ D fx 2 Rn1 W Ax D 0g: Note that, for any n n matrix A, N.A/ D fx 2 Rn1 W .I A/x D xg C.I A/; n1

N.I A/ D fx 2 R

W Ax D xg C.A/:

(1.1) (1.2)

140


The solution set of a nonhomogeneous linear system is not a linear space (as can be easily seen by, e.g., observing that the solution set does not contain the null matrix).

11.2 General Form of a Solution a. Homogeneous linear systems The following theorem gives an expression for the general form of a solution to a homogeneous linear system in terms of any particular generalized inverse of the coefficient matrix. Theorem 11.2.1. A matrix X is a solution to a homogeneous linear system AX D 0 (in X) if and only if X D .I A A/Y for some matrix Y . Proof. If X D .I A A/Y for some matrix Y , then AX D .A AA A/Y D .A A/Y D 0; so that X is a solution to AX D 0. Conversely, if X is a solution to AX D 0, then X D X A .AX / D .I A A/X ; so that X D .I A A/Y for Y D X . Q.E.D. According to Theorem 11.2.1, all solutions to a homogeneous linear system AX D 0 can be generated by setting X D .I A A/Y and allowing Y to range over all matrices (of the appropriate dimensions). As a special case of Theorem 11.2.1, we have that a column vector x is a solution to a homogeneous linear system Ax D 0 (in a column vector x) if and only if x D .I A A/y for some column vector y. Thus, we have the following corollary of Theorem 11.2.1. Corollary 11.2.2. For any matrix A, N.A/ D C.I A A/:

11.2 General Form of a Solution

141

b. Nonhomogeneous linear systems The following theorem relates the solutions of an arbitrary (consistent) linear system AX D B to those of the linear homogeneous system AZ D 0 (in Z) having the same coefficient matrix. (To avoid confusion, the matrix of unknowns in the homogeneous linear system is now denoted by Z, rather than X.) Theorem 11.2.3. Let X0 represent any particular solution to a consistent linear system AX D B (in X). Then, a matrix X is a solution to AX D B if and only if X D X0 C Z for some solution Z to the homogeneous linear system AZ D 0 (in Z). Proof. If X D X0 C Z for some solution Z to AZ D 0, then AX D AX0 C AZ D B C 0 D B; so that X is a solution to AX D B. Conversely, if X is a solution to AX D B, then, defining Z D X X0 , we find that X D X0 C Z and, since

AZ D AX AX0 D B B D 0;

that Z is a solution to AZ D 0. Q.E.D. The upshot of Theorem 11.2.3 is that all of the matrices in the solution set of a consistent linear system AX D B can be generated from any particular solution X0 by setting X D X0 C Z and allowing Z to range over all the matrices in the solution space of the linear homogeneous system AZ D 0. It follows from Theorem 9.1.2 that one solution to a consistent linear system AX D B is the matrix A B. Thus, in light of Theorem 11.2.3, we have the following extension of Theorem 11.2.1. Theorem 11.2.4. A matrix X is a solution to a consistent linear system AX D B (in X) if and only if X D A B C .I A A/Y for some matrix Y . As a special case of Theorem 11.2.4, we have that a column vector x is a solution to a consistent linear system Ax D b (in a column vector x) if and only if (2.1) x D A b C .I A A/y for some column vector y.

142


Consider, for example, expression (2.1) as applied to the linear system 0 1 0 1 6 2 2 3 3 @ 3 1 5 2A x D @2A ; (2.2) 3 1 3 1 5 whose consistency was established in Section 9.5. Taking A and b to be the coefficient matrix and right side of linear system (2.2), choosing 0 1 0 0 0 B1 0 3C C A D B @ 0 0 0A ; 1 0 2 and denoting the elements of y by y1 , y2 , y3 , and y4 , respectively, we find that 1 0 y1 B12 C 3y1 11y3 C C: (2.3) A b C .I A A/y D B A @ y3 7 8y3 Thus, the members of the solution set of linear system (2.2) consist of all vectors of the general form (2.3).

11.3 Number of Solutions a. Homogeneous linear systems What is the dimension of the solution space of a homogeneous linear system AX D 0 (in X)? The following lemma answers this question in the special case of a homogeneous linear system Ax D 0 (in a column vector x). Lemma 11.3.1. Let A represent an m n matrix. Then, dimŒN.A/ D n rank.A/: That is, the dimension of the solution space of the homogeneous linear system Ax D 0 (in an n-dimensional column vector x) equals n rank.A/. Proof. Recalling (from Corollary 11.2.2) that N.A/ D C.I A A/ and making use of Lemma 10.2.6, we find that dimŒN.A/ D dimŒC.I A A/ D rank.I A A/ D n rank.A/: Q.E.D. The solution space of a homogeneous linear system Ax D 0 (in an ndimensional column vector x) is a subspace of the linear space Rn of all ndimensional column vectors. According to Lemma 11.3.1, the dimension of this

11.3 Number of Solutions

143

subspace equals n rank.A/. Thus, if rank.A/ D n, that is, if A has full column rank, then the homogeneous linear system Ax D 0 has a unique solution, namely, the null vector 0. And, if rank.A/ < n, then Ax D 0 has an infinite number of solutions. Consider now the dimension of the solution space of the homogeneous linear system AX D 0 (in an n p matrix X). Clearly, a matrix X is a solution to AX D 0 if and only if each of its columns is a solution to the homogeneous linear system Ax D 0 (in the n-dimensional column vector x). Suppose that rank.A/ D n. Then, the only solution to Ax D 0 is the ndimensional null vector, and hence the only solution to AX D 0 is the n p null matrix. Alternatively, suppose that rank.A/ < n. Let s D n rank.A/, and take x1 ; : : : ; xs to be any s linearly independent solutions to Ax D 0. (According to Lemma 11.3.1, the solution space of Ax D 0 is s-dimensional.) Then, it is a straightforward exercise to show that the ps n p matrices .x1 ; 0; : : : ; 0/; : : : ; .xs ; 0; : : : ; 0/; : : : ; .0; : : : ; 0; x1 /; : : : ; .0; : : : ; 0; xs / form a basis for the solution space of AX D 0 and hence that the dimension of the solution space of AX D 0 equals ps. Thus, we have the following generalization of Lemma 11.3.1. Lemma 11.3.2. The dimension of the solution space of a homogeneous linear system AX D 0 (in an n p matrix X) equals pŒn rank.A/. The solution space of a homogeneous linear system AX D 0 (in an n p matrix X) is a subspace of the linear space Rnp of all n p matrices. According to Lemma 11.3.2, the dimension of this subspace equals pŒn rank.A/. Thus, if rank.A/ D n, then the homogeneous linear system AX D 0 has a unique solution, namely, the null matrix 0. And, if rank.A/ < n, then AX D 0 has an infinite number of solutions. b. Nonhomogeneous linear systems Consider the size of the solution set of a possibly nonhomogeneous linear system AX D B (in an n p matrix X). If AX D B is inconsistent, then, by definition, it has zero solutions. If AX D B is consistent, then it follows from Theorem 11.2.3 that there is a one-to-one correspondence between solutions to AX D B and solutions to the corresponding homogeneous linear system AZ D 0. Thus, in light of the discussion of Subsection a, we conclude (1) that if the linear system AX D B (in the n p matrix X) is consistent and if the coefficient matrix A has full column rank (i.e., rank n), then AX D B has a unique solution, and (2) that if AX D B is consistent but A is of less than full column rank, then AX D B has an infinite number of solutions. Moreover, on the basis of Theorem 7.2.2, we reach the following, additional conclusion. Theorem 11.3.3. If the coefficient matrix A of the linear system AX D B (in X) is nonsingular, then AX D B has a solution and that solution is unique.

144


11.4 A Basic Result on Null Spaces The following lemma gives a basic result on null spaces. Lemma 11.4.1. Let A represent an m n matrix and X an n p matrix. (1) If AX D 0, then C.X/ N.A/. (2) If AX D 0 and rank.X/ D n rank.A/, then C.X/ D N.A/. Proof. (1) Suppose that AX D 0. Then, for any n 1 vector x in C.X/, x D Xr for some p 1 vector r, and consequently Ax D AXr D 0, so that x 2 N.A/. Thus, C.X/ N.A/. (2) Suppose that AX D 0 and that rank.X/ D n rank.A/. Then, in light of Part (1) and Lemma 11.3.1, we have that C.X/ N.A/ and {since dimŒC.X/ D rank.X/} that dimŒC.X/ D dimŒN.A/. Thus, it follows from Theorem 4.3.10 that C.X/ D N.A/. Q.E.D.

11.5 An Alternative Expression for the General Form of a Solution It follows from Theorem 9.1.2 that one solution to a consistent linear system AX D B (in X) is A B. If X is a solution to AX D B, is it necessarily the case that X D GB for some generalized inverse G of A? That is, can all solutions to AX D B be generated by setting X D GB and letting G range over all generalized inverses of A? The answer to this question depends on A and B. Consider, for example, the special case of a consistent linear system Ax D b (in an n 1 column vector x). If rank.A/ < n and b D 0, then (as discussed in Section 11.3a) Ax D b has an infinite number of solutions; however, only one of them is expressible in the form Gb for some generalized inverse G of A, namely, the n 1 null vector 0. The following theorem gives conditions on A and B under which every solution to AX D B is expressible in the form GB for some generalized inverse G of A. Theorem 11.5.1. Suppose that AX D B is a consistent linear system (in an n p matrix X) such that rank.A/ D n or rank.B/ D p. Then, a matrix X is a solution to AX D B if and only if X D GB for some generalized inverse G of A. In the special case where p D 1, Theorem 11.5.1 reduces to the following result. Corollary 11.5.2. Suppose that Ax D b is a consistent linear system (in an n-dimensional column vector x) such that rank.A/ D n or b ¤ 0. Then, x is a solution to Ax D b if and only if x D Gb for some generalized inverse G of A.

11.6 Equivalent Linear Systems

145

Proof (of Theorem 11.5.1). If X D GB for some generalized inverse G of A, then X is a solution to AX D B, as is evident from Theorem 9.1.2. Conversely, suppose that X is a solution to AX D B. By assumption, rank.A/ D n or rank.B/ D p. Assume that rank.A/ D n. Then, AX D B has a unique solution (as discussed in Section 11.3), and hence (since—according to Theorem 9.1.2—A B is a solution to AX D B) X D A B; that is, X D GB for any generalized inverse G of A. Alternatively, assume that rank.B/ D p. According to Theorem 11.2.4, there exists a matrix Y such that X D A B C .I A A/Y ; and, since the matrix B0 is of full row rank (i.e., rank p), there exists (in light of Theorem 7.2.2) a matrix K such that B0 K D Y 0 : Furthermore, according to Lemma 9.5.1, AA B D B: Thus, X D A B C .I A A/.B0 K/0 D A B C K0 B A AK0 B D .A C K0 A AK0 AA /B; and hence, in light of Theorem 9.2.7, X D GB for some generalized inverse G of A. Q.E.D. Theorem 11.5.1 gives an expression—alternative to that given by Theorem 11.2.4—for the general form of a solution to a consistent linear system AX D B (in an np matrix X). However, Theorem 11.5.1 is applicable only if rank.A/ D n or rank.B/ D p, whereas Theorem 11.2.4 is applicable to any consistent linear system.

11.6 Equivalent Linear Systems If two linear systems have the same solution set, they are said to be equivalent. The linear system CAX D CB obtained by premultiplying both sides of a linear system AX D B (in X) by a matrix C may or may not be equivalent to AX D B. The following two lemmas, which are immediate consequences of Lemma 8.3.1 and Corollary 5.3.3, respectively, give sufficient conditions for the two linear systems to be equivalent. Lemma 11.6.1. Let A represent an m n matrix and B an m p matrix. If C is an r m matrix of full column rank (i.e., of rank m), then the linear system CAX D CB is equivalent to the linear system AX D B (in X).

146


Lemma 11.6.2. For any m n matrix A and n p matrix F , the linear system A0 AX D A0 AF is equivalent to the linear system AX D AF (in X). Note that, as a special case of Lemma 11.6.2 (the special case where F D 0), we have that the linear system A0 AX D 0 is equivalent to the linear system AX D 0.

11.7 Null and Column Spaces of Idempotent Matrices It was noted earlier [in results (1.1) and (1.2)] that, for any square matrix A, N.A/ C.I A/ and N.I A/ C.A/. Is it possible for N.A/ to equal C.I A/ and/or for N.I A/ to equal C.A/? And, if so, under what condition(s)? These questions are answered in the following theorem and corollary. Theorem 11.7.1. Let A represent an n n matrix. Then, N.A/ D C.I A/ if and only if A is idempotent. Proof. Suppose that A is idempotent. Then, since (according to Lemma 10.1.3) A is a generalized inverse of itself, it follows from Corollary 11.2.2 that N.A/ D C.I AA/ D C.I A/: Conversely, suppose that N.A/ D C.I A/. Then, for every n 1 vector x, .I A/x 2 N.A/ or equivalently A.I A/x D 0. Thus, in light of Lemma 2.3.2, Q.E.D. we have that A A2 D A.I A/ D 0 and hence that A2 D A. Corollary 11.7.2. Let A represent an n n matrix. Then, C.A/ D N.I A/ if and only if A is idempotent. Proof. According to Theorem 11.7.1, N.I A/ D CŒI .I A/ if and only if I A is idempotent. The proof is complete upon observing that I .I A/ D A and further (in light of Lemma 10.1.2) that I A is idempotent if and only if A is idempotent. Q.E.D.

11.8 Linear Systems With Nonsingular Triangular or Block-Triangular Coefficient Matrices Let

0

1 A11 A12 : : : A1r B 0 A22 : : : A2r C B C ADB : :: C :: @ :: : : A 0 0 Arr

11.8 Linear Systems With Nonsingular Block-Triangular Coefficient Matrices

147

represent an n n upper block-triangular matrix whose ij th block Aij is of dimensions ni nj (j i D 1; 2; : : : ; r). Suppose that the diagonal blocks A11 ; A22 ; : : : ; Arr of A are all nonsingular, or, equivalently (in light of Theorem 8.5.5), that A itself is nonsingular. Let B represent an n p matrix, and consider the linear system AX D B (in the n p matrix X). Partition X and B as 0 1 0 1 B1 X1 BB 2 C BX2 C B C B C and B D B : C ; XDB : C @ :: A @ :: A Xr

Br

where Xi and Bi are of dimensions ni p (i D 1; : : : ; r), and rewrite the linear system as 10 1 0 1 0 X1 B1 A11 A12 : : : A1r B 0 A22 : : : A2r C BX2 C BB2 C CB C B C B (8.1) B :: :: C B :: C D B :: C :: @ : : : A@ : A @ : A 0

0

Arr

Xr

Br

or, equivalently, as Ai i Xi C

r X

Aij Xj D Bi

.i D 1; : : : ; r 1/;

j DiC1

Arr Xr D Br : In solving linear system (8.1), the block-triangularity of the coefficient matrix can be exploited by proceeding in r steps. The first step is to solve the linear system Arr Xr D Br for Xr —one representation for the solution is Xr D A1 rr Br . The second step is to solve the linear system Ar1;r1 Xr1 C Ar1;r Xr D Br1 for Xr1 in terms of Xr —one representation for this solution is Xr1 D A1 r1;r1 .Br1 Ar1;r Xr /: More generally, the .r i C 1/th step is to solve the linear system r X

Ai i Xi C

Aij Xj D Bi

j DiC1

for Xi in terms of the quantities Xr ; Xr1 ; : : : ; XiC1 determined during the first, second, : : : ; .r i /th steps, respectively—one representation for this solution is Xi D

A1 ii

Bi

r X j DiC1

Aij Xj

(8.2)

148


(i D r 1; r 2; : : : ; 1). This approach to the solution of linear system (8.1) is called back substitution. Let us now specialize by taking U D fuij g to be an n n nonsingular upper triangular matrix and b D fbi g to be an n 1 vector, and by considering the linear system U x D b in the n 1 vector x D fxi g. As applied to this linear system, back substitution consists of determining xn ; xn1 ; : : : ; x1 , recursively, from the n formulas xn D bn =unn ; n X uij xj =ui i .i D n 1; : : : ; 1/: x i D bi

(8.3a) (8.3b)

j DiC1

For example, in the case of the linear system 0 10 1 0 1 8 1 6 x1 9 @0 5 4A @x2 A D @ 3A ; 0 0 2 6 x3 these formulas give x3 D 6=2 D 3; x2 D Œ3 4.3/=5 D 3; x1 D Œ9 1.3/ 6.3/=8 D 0:75: Suppose now that, instead of being upper block-triangular, the coefficient matrix of the linear system AX D B is a lower block-triangular matrix 0 1 A11 0 : : : 0 BA21 A22 0 C B C ADB : C : :: :: @ :: A : Ar1 Ar2 : : : Arr (with nonsingular diagonal blocks A11 ; A22 ; : : : ; Arr ), in which case the linear system is expressible as 0 10 1 0 1 X1 B1 A11 0 : : : 0 BA21 A22 C BX2 C BB2 C 0 B CB C B C (8.4) B :: C B :: C D B :: C :: :: @ : A@ : A @ : A : : Ar1 Ar2 : : : Arr

Xr

Br

or, equivalently, as A11 X1 D B1 ; Ai i Xi C

i1 X j D1

Aij Xj D Bi

.i D 2; 3; : : : ; r/:

11.9 A Computational Approach

149

In solving linear system (8.4), the block-triangularity of the coefficient matrix can (as in the case of an upper block-triangular coefficient matrix) be exploited by proceeding in r steps. The first step is to solve the linear system A11 X1 D B1 for X1 —one representation for this solution is X1 D A1 11 B1 . The i th step is to solve the linear system i1 X Aij Xj D Bi Ai i Xi C j D1

for Xi in terms of the quantities X1 ; : : : ; Xi1 determined during the first, second, : : : ; .i 1/th steps, respectively—one representation for this solution is i1 X B A X Xi D A1 i ij j ii

(8.5)

j D1

(i D 2; 3; : : : ; r). This approach to the solution of linear system (8.4) is called forward elimination. Let us now specialize by taking L D fìj g to be an n n nonsingular lower triangular matrix and b D fbi g to be an n 1 vector and by considering the linear system Lx D b in the n 1 vector x D fxi g. As applied to this linear system, forward elimination consists of determining x1 ; x2 ; : : : ; xn , recursively, from the n formulas x1 D b1 =`11 ; x i D bi

i1 X

ìj xj =ì i

.i D 2; 3; : : : ; n/:

(8.6a) (8.6b)

j D1

11.9 A Computational Approach Consider the problem of computing a solution to a consistent linear system AX D B ( in X). Depending on the nature of the coefficient matrix A, this problem can be relatively easy or relatively difficult. One case where it is realtively easy is that where A is an orthogonal matrix or more generally where the columns of A are orthonormal (with respect to the usual inner product). In that special case, the (unique) solution to AX D B is X D A0 B. Another case where the computation of a solution to AX D B is relatively easy is that where A is a triangular matrix. If A is upper or lower triangular, then a solution to the linear system can be computed by back substitution or forward elimination—refer to Section 11.8—or (in the event A is singular) by a variant of back substitution or forward elimination. More generally, the computation of a solution to AX D B can be approached by (implicitly or explicitly) decomposing A into the product A D KT of two matrices K and T , where T is of full row rank, and by proceeding in two steps as follows: (1) compute a solution Y to the linear system KY D B (in a matrix Y ) and then (2) compute a solution X to the linear system T X D Y (in X).

150


[Upon observing that C.B/ C.A/ C.K/, it follows from Theorem 7.2.1 that the linear system KY D B is consistent; and it follows from Theorem 7.2.2 that the linear system T X D Y is consistent.] Clearly, AX D KT X D KY D B; so that the matrix X computed in Step (2) is a solution to AX D B. Here, the decomposition A D KT should be such that the solutions of the linear systems KY D B and T X D Y are relatively easy to compute. One choice for this decomposition (in the special case where A is of full column rank) is the QR decomposition, considered in Section 6.4d. Other decompositions that could (depending on the nature of A) be suitable choices will be considered subsequently (in Chapters 14 and 21).

11.10 Linear Combinations of the Unknowns Suppose that Ax D b is a consistent linear system (in an n-dimensional column vector x). Often, k0 x, where k is an n-dimensional column vector, or more generally K0 x, where K is an n q matrix, may be of interest rather than x itself—note that k0 x is a linear combination of the unknowns (the elements of x) and that K0 x is a q-dimensional column vector, each element of which is a linear combination of the unknowns. For instance, in the case of the so-called normal equations (which arise from fitting a linear statistical model by least squares), interest may be confined to those linear combinations of the unknowns that correspond to estimable functions—refer, for example, to Searle (1971, sec. 5.4). More generally, suppose that AX D B is a consistent linear system (in an np matrix X). Then, K0 X may be of interest. a. Invariance to choice of solution Under what circumstances is the value of K0 X invariant to the choice of solution to AX D B? The answer to this question is provided by the following theorem. Theorem 11.10.1. Suppose that AX D B is a consistent linear system (in an n p matrix X), and let K represent an n q matrix. Then, the value of K0 X is the same for every solution to AX D B if and only if R.K0 / R.A/ or equivalently if and only if every row of K0 belongs to R.A/. Proof. Let X0 represent any particular solution to AX D B. Suppose that R.K0 / R.A/. Then, according to Lemma 4.2.2, there exists a matrix L such that K0 D LA. Letting X represent any solution to AX D B, we find that K0 X D LAX D LB D LAX0 D K0 X0 : Thus, the value of K0 X is the same for every solution to AX D B. Conversely, suppose that the value of K0 X is the same for every solution to AX D B. Then, it follows from Theorem 11.2.4 that the value of

11.10 Linear Combinations of the Unknowns

151

K0 ŒA B C .I A A/Y is the same for every matrix Y (of appropriate dimensions) or equivalently that K0 .I A A/Y D 0 for every matrix Y . In particular, K0 .I A A/Y D 0 for every matrix Y of the general form .y; 0; : : : ; 0/. Thus, K0 .I A A/y D 0 for every n1 column vector y, or equivalently K0 y D K0 A Ay for every y, implying (in light of Lemma 2.3.2) that K0 D K0 A A. We conclude, on the basis of Lemma Q.E.D. 9.3.5, that R.K0 / R.A/. As a special case of Theorem 11.10.1, we have the following corollary. Corollary 11.10.2. Suppose that Ax D b is a consistent linear system (in an n-dimensional column vector x), and let k0 represent an n-dimensional row vector. Then, the value of k0 x is the same for every solution to Ax D b if and only if k0 2 R.A/. b. An alternative representation As described in the following theorem, the solution set of the linear system AX D B is related to that of a linear system A0 Y D K (in Y ). Theorem 11.10.3. Suppose that AX D B and A0 Y D K are consistent linear systems (in X and Y , respectively). Then, K0 X0 D Y00 B

(10.1)

for any solutions X0 and Y0 to AX D B and A0 Y D K, respectively. Proof. We have that K0 X0 D .A0 Y0 /0 X0 D Y00 AX0 D Y00 B: Q.E.D. Suppose that AX D B is a consistent linear system (in X) and that, for some n q matrix K such that R.K0 / R.A/, the value of K0 X is of interest. (It follows from Theorem 11.10.1 that the value of K0 X is the same for every solution to AX D B.) An obvious way to determine the value of K0 X is to find a solution X0 to AX D B and to then form the product K0 X0 . According to Theorem 11.10.3, there is an alternative approach—the value of K0 X can be determined by finding a solution Y0 to A0 Y D K and then forming the product Y00 B. c. Two augmented linear systems It is easy to verify the following two theorems. Theorem 11.10.4. Let A represent an m n matrix, B an m p matrix, and K an n q matrix. If X and L are the first and second parts, respectively, of any solution to the linear system A 0 X B D (10.2) K0 I L 0

152


(in X and L), then X is a solution to the linear system AX D B (in X), and L D K0 X . Conversely, if X is any solution to AX D B, then X and K0 X are the first and second parts, respectively, of some solution to linear system (10.2). Theorem 11.10.5. Let A represent an n n matrix, B an n p matrix, and K an n q matrix. If X and L are the first and second parts, respectively, of any solution to the linear system A C KK0 K X B D (10.3) I L 0 K0 (in X and L), then X is a solution to the linear system AX D B (in X) and L D K0 X . Conversely, if X is any solution to AX D B, then X and K0 X are the first and second parts, respectively, of some solution to linear system (10.3). Note that Theorem 11.10.4 implies that linear system (10.2) is consistent if and only if the linear system AX D B is consistent. Similarly, Theorem 11.10.5 implies that linear system (10.3) is consistent if and only if the linear system AX D B is consistent. Theorems 11.10.4 and 11.10.5 indicate that a solution X to a consistent linear system AX D B (in X) and the corresponding value K0 X of K0 X can both be obtained as parts of the solution to a single linear system, namely linear system (10.2) or (in the special case where A is square) linear system (10.3).

11.11 Absorption Consider the problem of determining whether a linear system a11 x1 C a12 x2 D b1 ;

(11.1a)

a21 x1 C a22 x2 D b2

(11.1b)

of two equations in two unknowns (x1 and x2 ) is consistent and, if it is consistent, of solving it. Suppose that a11 ¤ 0. Then, one approach to the problem is to solve the first equation for x1 in terms of x2 , obtaining 1 1 x1 D a11 b1 a11 a12 x2 ;

(11.2)

and to eliminate x1 from the second equation by replacing x1 with expression (11.2), which gives 1 1 .a22 a21 a11 a12 /x2 D b2 a21 a11 b1 :

(11.3)

Clearly, linear system (11.1) is consistent if and only if equation (11.3) has a solution (for x2 ), say x2 , in which case a solution to linear system (11.1) is x1 and x2 , where 1 1 b1 a11 a12 x2 x1 D a11

11.11 Absorption

153

is the value of x1 obtained by setting x2 D x2 in expression (11.2). Moreover, if 1 a12 , equation (11.3) has a unique solution, namely, a22 ¤ a21 a11 1 1 x2 D .a22 a21 a11 a12 /1 .b2 a21 a11 b1 /: 1 1 If a22 D a21 a11 a12 , then, depending on whether or not b2 D a21 a11 b1 , either any value of x2 can serve as a solution to equation (11.3), or there is no solution to equation (11.3). Let us now consider the extension of this approach to the problem of solving linear system (11.1) to that of solving a linear system comprising an arbitrary number of equations in an arbitrary number of unknowns. The basis for such an extension is provided by the following theorem.

Theorem 11.11.1. Let A represent an m n matrix and B an m p matrix, and consider the linear system AX D B (in an n p matrix X). Partition A, B, and X as B1 X1 A11 A12 ; BD ; and X D ; AD A21 A22 B2 X2 where A11 ; A12 , and B1 have m1 rows, A21 ; A22 , and B2 have m2 rows, A11 and A21 have n1 columns, A12 and A22 have n2 columns, and X1 and X2 have n1 rows and n2 rows, respectively. Suppose that C.A12 / C.A11 /; C.B1 / C.A11 /;

and R.A21 / R.A11 /;

and let C D A 11 B1 and K D A11 A12 or, more generally, let C represent any matrix such that A11 C D B1 and K any matrix such that A11 K D A12 . Then, (1) X1 (where X1 has n1 rows) is a solution to the linear system the matrix X D X2 AX D B if and only if X2 is a solution to the linear system

.A22 A21 K/X2 D B2 A21 C .in X2 /

(11.4)

and X1 and X2 form a solution to the linear system A11 X1 C A12 X2 D B1 .in X1 and X2 /I

(11.5)

(2) the linear system AX D B is consistent if and only if linear system (11.4) is C KX2 consistent; (3) for any particular value of X2 , is a solution to linear X2 system (11.5), that is A11 .C KX2 / C A12 X2 D B1 : In connection with Theorem 11.11.1, observe that the linear system AX D B can be reexpressed in two parts as follows:

154


A11 X1 C A12 X2 D B1 ; A21 X1 C A22 X2 D B2 : Observe also (in light of Lemma 4.2.2) that the existence of matrices C and K such that A11 C D B1 and A11 K D A12 is insured by the assumptions that C.B1 / C.A11 / and C.A12 / C.A11 /. Proof (of Theorem 11.11.1). It is clear that, for any particular value of X2 , A11 .C KX2 / C A12 X2 D B1 A12 X2 C A12 X2 D B1 ; which establishes Part (3) of the theorem. For purposes of proving Part (1), observe (in light of Lemma 4.2.2) that, since R.A21 / R.A11 /, there exists a matrix L such that A21 D LA11 : Thus, it follows from Part (3) that, for any solution X1 and X2 to linear system (11.5), A21 X1 D LA11 X1 D L.B1 A12 X2 / D LA11 .C KX2 / D A21 .C KX2 /:

(11.6)

X1 is a solution to AX D B. Then, it is clear that X2 X1 and X2 form a solution to linear system (11.5) and, in light of result (11.6), that A21 .C KX2 / C A22 X2 D B2 ; Suppose now that X D

or equivalently that

.A22 A21 K/X2 D B2 A21 C;

and hence that X2 is a solution to linear system (11.4). Conversely, suppose that X1 and X2 form a solution to linear system (11.5) and additionally that X2 is a solution to linear system (11.4). Then, A21 .C KX2 / C A22 X2 D B2 ; implying [in light of result (11.6)] that A21 X1 C A22 X2 D B2

X1 is a solution to AX D B, which completes the proof of Part X2 (1) of the theorem. Finally, consider Part (2). That the consistency of AX D B implies the consistency of linear system (11.4) is an obvious consequence of Part (1). Conversely, if linear system (11.4) is consistent, then by definition this linear system has a

and hence that

11.11 Absorption

155

solution, say X2 , and, letting X1 D C KX2 , it follows from Part (3) that X1 and X2 form a solution to linear system (11.5) and hence, in light of Part (1), that X1 is a solution to AX D B, implying that AX D B is consistent. Q.E.D. X2 Theorem 11.11.1 can be translated into the following four-step procedure for solving the linear system AX D B: (1) find matrices C and K such that A11 C D B1 and A11 K D A12 ; (2) form the coefficient matrix A22 A21 K and the right side B2 A21 C of linear system (11.4); (3) obtain a solution X2 to linear system (11.4); and X1 to be the solution to AX D B. (4) form the matrix X1 D CKX2 , and take X2 In intuitive terms, this procedure consists of: (1) solving the first m1 equations of the linear system AX D B for X1 in terms of X2 ; (2) “absorbing” the first m1 equations into the last m2 equations by substituting this solution for X1 into the latter equations, thereby eliminating X1 from these equations and in effect “reducing” them to linear system (11.4); (3) solving linear system (11.4); and (4) “back-solving” for X1 , using the expression for X1 obtained in solving the first m1 equations for X1 in terms of X2 . Accordingly, in the context of this procedure, linear system (11.4) is sometimes called the reduced linear system, and the procedure itself is sometimes referred to as absorption. The circumstances under which the use of absorption is advantageous are similar to those under which the use of formula (8.5.16) for the inverse of a partitioned matrix is advantageous, or more generally under which the use of formula (9.6.2) for the generalized inverse of a partioned matrix is advantageous. More specifically, the use of absorption can be advantageous if matrices C and K such that A11 C D B1 and A11 K D A12 are known and if the dimensions of A11 are relatively large. Under these circumstances, the effect of absorption is to replace the problem of solving the linear system AX D B, which comprises m equations in n unknowns, with the problem of solving linear system (11.4), which comprises “only” m2 equations in “only” n2 unknowns. In describing absorption, it has been supposed that the first n1 rows of X are to be eliminated by absorbing the first m1 equations of the linear system into the last m2 equations. By modifying the results of Theorem 11.11.1 in a straightforward way, we could, in principle, obtain formulas for eliminating an arbitrary set of n1 rows of X by absorbing an arbitrary set of m1 equations of the linear system into the remaining equations. Another means to the same end is to rewrite any particular linear system in such a way that the unknowns to be eliminated are those in the

156


first n1 rows and the equations to be absorbed are the first m1 equations, which can be accomplished by permuting the rows and columns of the coefficient matrix and also the rows of the right side and of the matrix of unknowns. Consider, for example, the linear system 0 10 1 0 1 2 0 1 0 1 x1 B6 2 4 8C Bx2 C B 6C B CB C B C (11.7) @ 0 5 1 0A @x3 A D @1A : 5 x4 1 3 0 0 By permuting the rows and columns of the coefficient matrix, the elements of the right side, and the unknowns, this linear system can be rewritten as 0 10 1 0 1 1 0 0 3 5 x1 B 0 1 0 5C Bx3 C B1C B CB C B C @ 0 0 1 1A @x4 A D @2A : 6 4 8 2 x2 6 Now, let us apply Theorem 11.11.1, taking 0 1 0 1 0 1 1 0 0 3 5 A11 D @0 1 0A ; A12 D @5A ; B1 D @1A ; 0 0 1 1 2

0

1 x1 X1 D @x3 A ; x4

A21 D .6; 4; 8/; A22 D .2/; B2 D .6/; and X2 D .x2 /: It is immediately apparent that the unique choices for C and K (such that A11 C D B1 and A11 K D A12 ) are 0 1 0 1 5 3 C D @1A and K D @5A : 2 1 Thus, A22 A21 K D 8 and B2 A21 C D 56; and the unique solution to linear system (11.4) is x2 D 56=.8/ D 7. Moreover, for x2 D 7, 0 1 26 C KX2 D @34A : 5 We conclude that the unique solution to linear system (11.7) is x1 D 26, x2 D 7, x3 D 34, and x4 D 5. In solving some linear systems, it may be helpful to make repeated use of absorption. That is, the linear system (11.4) obtained by absorbing the first m1 equations of the linear system AX D B into the last m2 equations could itself possibly be solved by absorption.

11.12 Extensions to Systems of the Form AXC D B

157

For some linear statistical models, the use of absorption can greatly facilitate the solution of the so-called normal equations (which arise in fitting the model by least squares). One linear statistical model for which this is the case is the additive version of the two-way crossed-classification model—refer to Searle (1971, sec. 7.1).

11.12 Extensions to Systems of the Form AXC D B Let X0 D A B or, more generally, let X0 represent any particular solution to a consistent linear system AX D B (with an m n coefficient matrix A, m p right side B, and n p matrix of unknowns X). It was established in Section 11.2 that a matrix X is a solution to AX D B if and only if X D X0 C Z for some solution Z to the homogeneous linear system AZ D 0 (in Z). Also, in light of Theorem 11.2.1, X is a solution to AX D B if and only if X D X0 C .I A A/Y for some matrix Y . The following theorem generalizes these results to any consistent system of the form AXC D B (where A is an m n matrix, C a p q matrix, and B an m q matrix, and where X is an n p matrix of unknowns). Theorem 11.12.1. Let X0 D A BC or, more generally, let X0 represent any particular solution to a consistent system AXC D B (in X). Then, (1) a matrix X is a solution to AXC D B if and only if X D X0 C Z for some solution Z to the system AZC D 0 (in Z); (2) X is a solution to AXC D B if and only if X D X0 C Y A AY CC for some matrix Y (of suitable dimensions); and (3) X is a solution to AXC D B if and only if X D X0 C A AR.I CC / C .I A A/SCC C .I A A/T .I CC / for some matrices R, S, and T (of suitable dimensions). Proof. Note that it follows from Theorem 9.7.1 that A BC is a solution to AXC D B. (1) If X D X0 C Z for some solution Z to AZC D 0, then AX C D AX0 C C AZ C D B C 0 D B; so that X is a solution to AXC D B. Conversely, suppose that X is a solution to AXC D B. Then, X D X0 C Z , where Z D X X0 . And, since AZ C D AX C AX0 C D B B D 0;

158


Z is a solution to AZC D 0. (2) If X D X0 C Y A AY CC for some matrix Y , then AX C D AX0 C C AY C AA AY CC C D B C AY C AY C D B; so that X is a solution to AXC D B. Conversely, if X is a solution to AXC D B, then [as a consequence of Part (1)] X D X0 C Z for some solution Z to AZC D 0 and hence X D X0 C Z A 0C D X0 C Z A AZ CC ; so that X D X0 C Y A AY CC for Y D Z . (3) If X D X0 C A AR.I CC / C .I A A/SCC C .I A A/T .I CC / for some matrices R, S, and T , then AX C D AX0 C C AA AR.C CC C/ C .A AA A/SCC C C .A AA A/T .C CC C/ D B C 0 C 0 C 0 D B; so that X is a solution to AXC D B. Conversely, if X is a solution to AXC D B, then [as a consequence of Part (2)] X D X0 C Y A AY CC for some matrix Y , implying {since Y D ŒA A C .I A A/Y ŒCC C .I CC / D A AY CC C A AY .I CC / C .I A A/Y CC C .I A A/Y .I CC / and hence since Y A AY CC D A AY .I CC / C .I A A/Y CC C .I A A/Y .I CC /} that X D X0 C A AR.I CC / C .I A A/SCC C .I A A/T .I CC / for R D S D T D Y .


Q.E.D.

Exercises

159

1. Show that, for any matrix A, C.A/ D N.I AA /: 2. Show that if X1 ; : : : ; Xk are solutions to a linear system AX D B (in X) and P P c1 ; : : : ; ck are scalars such that kiD1 ci D 1, then the matrix kiD1 ci Xi is a solution to AX D B. Section 11.3 3. Let A and Z represent n n matrices. Suppose that rank.A/ D n 1, and let x and y represent nonnull n-dimensional column vectors such that Ax D 0 and A0 y D 0. (a) Show that AZ D 0 if and only if Z D xk0 for some n-dimensional row vector k0 . (b) Show that AZ D ZA D 0 if and only if Z D cxy 0 for some scalar c. 4. Suppose that AX D B is a nonhomogeneous linear system (in an n p matrix X). Let s D pŒn rank.A/, and take Z1 ; : : : ; Zs to be any s n p matrices that form a basis for the solution space of the homogeneous linear system AZ D 0 (in an n p matrix Z). Define X0 to be any particular solution to AX D B, and let Xi D X0 C Zi (i D 1; : : : ; s). (a) Show that the s C 1 matrices X0 ; X1 ; : : : ; Xs are linearly independent solutions to AX D B. (b) Show that every solution to AX D B is expressible as a linear combination of X0 ; X1 ; : : : ; Xs . P (c) Show that a linear combination siD0 ki Xi of X0 ; X1 ; : : : ; Xs is a solution to AX D B if and only if the scalars k0 ; k1 ; : : : ; ks are such that P s iD0 ki D 1. (d) Show that the solution set of AX D B is a proper subset of the linear space sp.X0 ; X1 ; : : : ; Xs /. Section 11.5 5. Suppose that AX D B is a consistent linear system (in an n p matrix X). Show that if rank.A/ < n and rank.B/ < p, then there exists a solution X to AX D B that is not expressible as X D GB for any generalized inverse G of A. Section 11.6 6. Use the result of Part (b) of Exercise 7.1 to generalize Lemma 11.6.1.

160


7. Let A represent an m n matrix, B an m p matrix, and C a q m matrix, and suppose that AX D B and CAX D CB are linear systems (in X). (a) Show that if rankŒC.A; B/ D rank.A; B/, then CAX D CB is equivalent to AX D B, thereby generalizing Lemmas 11.6.1 and 11.6.2. (b) Show that if rankŒC.A; B/ < rank.A; B/ and if CAX D CB is consistent, then the solution set of AX D B is a proper subset of that of CAX D CB (i.e., there exists a solution to CAX D CB that is not a solution to AX D B). (c) Show, by example, that if rankŒC.A; B/ < rank.A; B/ and if AX D B is inconsistent, then CAX D CB can be either consistent or inconsistent. Section 11.10 8. Let A represent a q n matrix, B an m p matrix, and C an m q matrix; and suppose that the linear system CAX D B (in an n p matrix X) is consistent. Show that the value of AX is the same for every solution to CAX D B if and only if rank.CA/ D rank.A/. 9. Verify (1) Theorem 11.10.4 and (2) Theorem 11.10.5.

12 Projections and Projection Matrices

Projections and projection matrices, which are introduced and discussed in this chapter, are frequently encountered in discourse on linear statistical models related to the estimation of parameters and to the analysis of variance. Their appearance in such discourse can be attributed to their connection to the so-called least squares problem—one long-standing approach to the estimation of the parameters of a linear statistical model is based on “fitting” the model by least squares. Their connection to the least squares problem is described and discussed in Section 12.4.

12.1 Some General Results, Terminology, and Notation If a matrix Y in a linear space of matrices is orthogonal to every matrix in a subspace U, Y is said to be orthogonal to U. The statement that Y is orthogonal to U is sometimes abbreviated to Y ? U. Similarly, to indicate that every matrix in a subspace U is orthogonal to every matrix in a subspace W, one says that U is orthogonal to W or writes U ? W. The following lemma is easy to verify. Lemma 12.1.1. Let Y represent a matrix in a linear space V, let U and W represent subspaces of V, and take fX1 ; : : : ; Xs g to be a set of matrices that spans U and fZ1 ; : : : ; Z t g to be a set that spans W. Then, Y ? U if and only if Y Xi D 0 for i D 1; : : : ; s; that is, Y is orthogonal to U if and only if Y is orthogonal to each of the matrices X1 ; : : : ; Xs . And, similarly, U ? W if and only if Xi Zj D 0 for i D 1; : : : ; s and j D 1; : : : ; t ; that is, U is orthogonal to W if and only if each of the matrices X1 ; : : : ; Xs is orthogonal to each of the matrices Z1 ; : : : ; Z t . By applying Lemma 12.1.1 in the special case where V D Rm1 and U and W are the column spaces of two matrices (each of which has m rows), we obtain the following corollary.

162


Corollary 12.1.2. Let y represent an m-dimensional column vector, and let X represent an m n matrix and Z an m p matrix. Then, y is orthogonal to C.X/ (with respect to the usual inner product) if and only if X0 y D 0 (or equivalently if and only if y 0 X D 0). Similarly, C.X/ is orthogonal to C.Z/ (with respect to the usual inner product) if and only if X0 Z D 0 (or equivalently if and only if Z0 X D 0). Consider the following theorem. Theorem 12.1.3. Let Y represent a matrix in a linear space V, and let U represent an r-dimensional subspace of V. Then, there exists a unique matrix Z in U such that .Y Z/ ? U, that is, such that the difference between Y and Z is orthogonal to every matrix in U. If r D 0, then Z D 0, and, if r > 0, Z is expressible as Z D c1 X1 C C cr Xr ; (1.1) where fX1 ; : : : ; Xr g is any orthonormal basis for U and cj D Y Xj (j D 1; : : : ; r). Moreover, Z D Y if and only if Y 2 U. Proof. Consider first the case where r D 0. In this case, the only matrix in U is the null matrix 0. Clearly, Y 0 is orthogonal to 0. Thus, there exists a unique matrix Z in U such that .Y Z/ ? U, namely, Z D 0. Moreover, it is clear that Y D Z if and only if Y 2 U. be any orthonormal Consider now the case where r > 0. Take fX1 ; : : : ; Xr g toP basis for U, and define cj D Y Xj (j D 1; : : : ; r). Clearly, j cj Xj 2 U, and X Y cj Xj Xi D .Y Xi / ci D 0 j

P for i D 1; : : : ; r, implying (in light of Lemma 12.1.1) that .Y j cj Xj / ? U. Moreover, P for any matrix X such that X 2 U and .Y X/ ? U, we find that .X j cj Xj / 2 U and hence that X X X cj Xj X cj Xj j

D Y

X j

j

cj Xj

X X X cj Xj .Y X/ X cj Xj j

j

D 0 0 D 0; P P so that X j cj Xj D 0 or equivalently X D j cj Xj . We conclude P that there exists a unique matrix Z in U such that .Y Z/ ? U, namely, Z D j cj Xj . To complete the proof, observe that, if Y D Z, then obviously Y 2 U, and conversely, if Y 2 U, then, since Y Y D 0 is orthogonal to U, Y D Z. Q.E.D. Suppose that U is a subspace of a linear space V. Then, it follows from Theorem 12.1.3 that, corresponding to each matrix Y in V, there exists a unique matrix Z in U such that Y Z is orthogonal to U or, equivalently (if Y … U), such that Y Z forms an angle of =2 with every nonnull matrix in U. The matrix

12.2 Projection of a Column Vector

163

Z is called the orthogonal projection of Y on U or simply the projection of Y on U. Note that, for any matrix Y in U, Y itself is the projection of Y on U. The following theorem relates the projection of a linear combination of p matrices Y1 ; : : : ; Yp (on a subspace U of a linear space V) to the projections of Y1 ; : : : ; Yp (on U). Theorem 12.1.4. Let Y1 ; : : : ; Yp represent matrices in a linear space V, let U represent a subspace of V, and let Z1 ; : : : ; Zp represent the projections of Y1 ; : : : ; Yp , respectively, on U. Then, for any scalars k1 ; : : : ; kp , the projection of the linear combination k1 Y1 C C kp Yp (on U) is the corresponding linear combination k1 Z1 C C kp Zp of Z1 ; : : : ; Zp . Proof. By definition, Zi 2 U and .Yi Zi / ? U (i D 1; : : : ; p). Thus, .k1 Z1 C C kp Zp / 2 U. Moreover, for every matrix X in U, Œk1 Y1 C C kp Yp .k1 Z1 C C kp Zp / X D Œk1 .Y1 Z1 / C C kp .Yp Zp / X D k1 Œ.Y1 Z1 / X C C kp Œ.Yp Zp / X D k1 0 C C kp 0 D 0; so that Œk1 Y1 C C kp Yp .k1 Z1 C C kp Zp / ? U. We conclude that Q.E.D. k1 Z1 C C kp Zp is the projection of k1 Y1 C C kp Yp on U.

12.2 Projection of a Column Vector a. Main results The following theorem gives an algebraic expression for the projection (with respect to the usual inner product) of an n-dimensional column vector on a subspace of Rn . Theorem 12.2.1. Let z represent the projection (with respect to the usual inner product) of an n-dimensional column vector y on a subspace U of Rn , and let X represent any n p matrix whose columns span U. Then, z D Xb

(2.1)

for any solution b to the (consistent) linear system X0 Xb D X0 y (in b):

(2.2)

Proof. Suppose that b is a solution to the (consistent) linear system X0 Xb D X y. (The consistency of this linear system is evident from Theorem 7.4.1.) Then, X0 .y Xb / D 0, implying (in light of Corollary 12.1.2) that y Xb is orthogonal to C.X/ and hence [since C.X/ D U] to U. Since Xb 2 U, we conclude that Xb is the projection of y on U and hence, by definition, that z D Xb . Q.E.D. 0

164


Linear system (2.2) comprises p equations. These p equations are known as the normal equations. When the columns, say x1 ; : : : ; xp , of the matrix X are taken to be an orthonormal basis for U, we find that X0 X D I and hence that the normal equations (2.2) have a unique solution b D .x01 y; : : : ; xp0 y/0 , in which case expression (2.1) for the projection z of y on U reduces to a special case of expression (1.1). Since the vector .X0 X/ X0 y is one solution to the normal equations (2.2), we have the following corollary of Theorem 12.2.1. Corollary 12.2.2. Let z represent the projection (with respect to the usual inner product) of an n-dimensional column vector y on a subspace U of Rn , and let X represent any n p matrix whose columns span U. Then, z D X.X0 X/ X0 y:

(2.3)

A further implication of Theorem 12.2.1 is as follows: Corollary 12.2.3. Let y represent an n-dimensional column vector, X an np matrix, and W any n q matrix such that C.W / D C.X/. Then, W a D Xb for any solution a to the linear system W 0 W a D W 0 y (in a) and any solution b to the linear system X0 Xb D X0 y (in b). As a special case of Corollary 12.2.3, we have the following corollary. Corollary 12.2.4. Let y represent an n-dimensional column vector, and X an n p matrix. Then, Xb1 D Xb2 for any two solutions b1 and b2 to the linear system X0 Xb D X0 y (in b). The following theorem gives a converse of Theorem 12.2.1. Theorem 12.2.5. Let z represent the projection (with respect to the usual inner product) of an n-dimensional column vector y on a subspace U of Rn , and let X represent any n p matrix whose columns span U. Then, any p 1 vector b such that z D Xb is a solution to the linear system X0 Xb D X0 y (in b). Proof. In light of Theorem 12.2.1, X0 Xb D X0 y has a solution, say a, and z D Xa. Thus, X0 Xb D X0 z D X0 Xa D X0 y: Q.E.D. b. Two-dimensional example Let us find the projection (with respect to the usual inner product) of an ndimensional column vector y on a subspace U of Rn in the special case where n D 2, y D .4; 8/0 , and U D sp.x/, with x D .3; 1/0 . Upon taking X to be the 2 1 matrix whose only column is x, the linear system X0 Xb D X0 y becomes .10/b D .20/, which has the unique solution b D .2/. Thus, the projection of y on U is

12.2 Projection of a Column Vector

165

FIGURE 12.1: The projection z of a two-dimensional column vector y on a one-dimensional subspace U of R2 .

zD

3 6 .2/ D ; 1 2

as depicted in Figure 12.1. c. Three-dimensional example Let us find the projection (with respect to the usual inner product) of an ndimensional column vector y on a subspace U of Rn in the special case where n D 3, y D .3; 38=5; 74=5/0 , and U D sp.x1 ; x2 ; x3 /, with 0 1 0 1 0 1 0 2 2 x1 D @3A ; x2 D @ 2A ; x3 D @ 1A : (2.4) 6 4 2 Clearly, x1 and x2 are linearly independent, and x3 D x2 .1=3/x1 . Thus, dim.U/ D 2. Upon taking X to be the 3 3 matrix whose columns are x1 ; x2 and x3 , respectively, the normal equations X0 Xb D X0 y become 1 0 1 0 66 45 30 15 @30 24 14Ab D @38A : 16 15 14 9

166


FIGURE 12.2: The projection z of a three-dimensional column vector y on a twodimensional subspace U of R3 .

One solution to these equations is the vector .32=15; 1=2; 1/0 . Thus, the projection of y on U is 0 10 1 0 1 0 2 2 32=15 3 2 1A@ 1=2 A D @22=5A ; z D @3 6 4 2 1 44=5 as depicted in Figure 12.2.

12.3 Projection Matrices Subsequently, for any matrix X, the symbol PX is used to represent the square matrix X.X0 X/ X0 . Note that PX is invariant to the choice of the generalized inverse .X0 X/ , as (in light of Corollary 7.4.5) is evident from Theorem 9.4.1 and as is implicit in the following theorem.

12.3 Projection Matrices

167

Theorem 12.3.1. Let U represent any subspace of the linear space Rn of all n-dimensional column vectors. Then, there exists a unique matrix A (of dimensions n n) such that Ay is the projection (with respect to the usual inner product) of y on U for every column vector y in Rn . Moreover, A D PX for any matrix X such that C.X/ D U. Proof. Let X represent any matrix such that C.X/ D U. Then, it follows from Corollary 12.2.2 that, for every y 2 Rn , PX y is the projection of y on U. Moreover, if a matrix A is such that Ay is the projection of y on U for every y 2 Rn , then Ay D PX y for every y, implying (in light of Lemma 2.3.2) that A D PX . We conclude that there exists a unique matrix A such that Ay is the Q.E.D. projection of y on U for every y 2 Rn and that A D PX . n Suppose that U is a subspace of the linear space R of all n-dimensional column vectors. Then, according to Theorem 12.3.1, there exists a unique matrix A (of dimensions n n) such that Ay is the projection (with respect to the usual inner product) of y on U for every y 2 Rn . This matrix is called the orthogonal projection matrix for U or simply the projection matrix for U. The meaning of a statement that an n n matrix is a projection matrix is that there exists some subspace of Rn for which that matrix is the projection matrix. The last part of Theorem 12.3.1 can now be restated as follows: Corollary 12.3.2. Let A represent the projection matrix for a subspace U of the linear space Rn of all n-dimensional column vectors. Then, A D PX for any matrix X such that C.X/ D U. For any matrix X, it is clear that PX is the projection matrix for C.X/. Thus, we have the following additional corollary of Theorem 12.3.1. Corollary 12.3.3. A matrix A is a projection matrix if and only if A D PX for some matrix X. Certain basic properties of projection matrices are listed in the following two theorems. Theorem 12.3.4. Let X represent any n p matrix. Then, (1) PX X D X; that is, X.X0 X/ X0 X D X; that is, .X0 X/ X0 is a generalized inverse of X; (2) PX D XB for any solution B to the (consistent) linear system X0 XB D X0 (in B); (3) PX0 D PX ; that is, PX is symmetric; that is, XŒ.X0 X/ 0 X0 D X.X0 X/ X0 ; (4) XŒ.X0 X/ 0 X0 X D X; that is, Œ.X0 X/ 0 X0 is a generalized inverse of X; (5) X0 PX D X0 PX0 D X0 ; that is, X0 X.X0 X/ X0 D X0 XŒ.X0 X/ 0 X0 D X0 ; that is, X.X0 X/ and XŒ.X0 X/ 0 are generalized inverses of X0 ; (6) PX2 D PX ; that is, PX is idempotent; (7) C.PX / D C.X/, and R.PX / D R.X0 /; (8) rank.PX / D rank.X/; (9) .I PX /2 D I PX D .I PX /0 ; that is, I PX is symmetric and idempotent;

168


(10) rank.I PX / D n rank.X/. Proof. (1) It follows from the very definition of a generalized inverse that X0 X.X0 X/ X0 X D X0 X D X0 XI, and, upon applying Corollary 5.3.3, we conclude that X.X0 X/ X0 X D XI D X. [An alternative proof of (1) is as follows: Let x1 ; : : : ; xp represent the columns of X. Then, according to Theorem 12.3.1, PX xi is the projection of xi on C.X/, and, since xi 2 C.X/, it follows from Theorem 12.1.3 that the projection of xi on C.X/ is xi . Thus, PX X D .PX x1 ; : : : ; PX xp / D .x1 ; : : : ; xp / D X.] (2) If B is a solution to the linear system X0 XB D X0 (whose consistency was established in Corollary 7.4.2), then (according to Theorem 11.2.4) B D .X0 X/ X0 C ŒI .X0 X/ X0 XY for some matrix Y , implying [in light of Part (1)] that XB D PX C .X PX X/Y D PX : (3) According to Corollary 9.3.4, Œ.X0 X/ 0 is a generalized inverse of X0 X, so that Œ.X0 X/ 0 X0 is a solution to the (consistent) linear system X0 XB D X0 (in B). Thus, applying Part (2), we find that X.X0 X/ X0 D PX D XŒ.X0 X/ 0 X0 : (4) Making use of Parts (3) and (1), we find that XŒ.X0 X/ 0 X0 X D PX0 X D PX X D X: (5) Making use of Parts (3) and (1), we find that X0 PX D X0 PX0 D .PX X/0 D X0 : (6) Making use of Part (1), we find that PX2 D PX X.X0 X/ X0 D X.X0 X/ X0 D PX : (7) Recalling Corollary 4.2.3, it is clear from the definition of PX that C.PX / C.X/ and from Part (1) that C.X/ C.PX /. Thus, C.PX / D C.X/, and (in light of Lemma 4.2.5) R.PX0 / D R.X0 / or, equivalently [in light of Part (3)], R.PX / D R.X0 /. (8) That rank.PX / D rank.X/ is an immediate consequence of Part (7). (9) It follows from Part (3) that the matrix I PX is symmetric. And, in light of Lemma 10.1.2, it follows from Part (6) that I PX is idempotent. (10) Making use of Lemma 10.2.4 and Parts (6) and (8), we find that rank.I PX / D n rank.PX / D n rank.X/: Q.E.D.

12.3 Projection Matrices

169

Theorem 12.3.5. Let X represent an n p matrix, and let W represent any n q matrix such that C.W / C.X/. Then, (1) PX W D W , and W 0 PX D W 0 ; (2) PX PW D PW PX D PW . Proof. (1) According to Lemma 4.2.2, there exists a matrix F such that W D XF . Thus, making use of Parts (1) and (3) of Theorem 12.3.4, we find that PX W D PX XF D XF D W and

W 0 D .PX W /0 D W 0 PX0 D W 0 PX : (2) Making use of Part (1), we find that PX PW D PX W .W 0 W / W 0 D W .W 0 W / W 0 D PW

and similarly that PW PX D W .W 0 W / W 0 PX D W .W 0 W / W 0 D PW : Q.E.D. In the special case where W D X, the results of Theorem 12.3.5 reduce to results included in Parts (1), (5), and (6) of Theorem 12.3.4. Note that if X and W are any two matrices such that C.W / D C.X/, then it follows from Part (2) of Theorem 12.3.5 that PW D PX PW D PX : (That PW D PX can also be deduced from Theorem 12.3.1.) Thus, we have the following corollary of Theorem 12.3.5. Corollary 12.3.6. For any two matrices X and W such that C.W / D C.X/, PW D PX . Another result that can be deduced from Part (2) of Theorem 12.3.5 is given by the following corollary. Corollary 12.3.7. Let X represent an n p matrix. Then, for any matrix A that is the projection matrix for some subspace U of C.X/, PX A D APX D A: Proof. Let W represent any matrix whose columns span U. Then, according to Corollary 12.3.2, A D PW . Thus, since C.W / C.X/, it follows from Part (2) of Theorem 12.3.5 that PX A D APX D A: Q.E.D.

170


Suppose that a matrix A is the projection matrix for a subspace U of the linear space Rn of all n-dimensional column vectors. Let X represent any matrix whose columns span U. Then, according to Corollary 12.3.2, A D PX . Thus, making use of Part (7) of Theorem 12.3.4, we find that U D C.X/ D C.PX / D C.A/ [in which case dim.U/ D rank.A/], thereby establishing the following theorem. Theorem 12.3.8. If a matrix A is the projection matrix for a subspace U of the linear space Rn of all n-dimensional column vectors, then U D C.A/, and dim.U/ D rank.A/. Projection matrices can be characterized as follows: Theorem 12.3.9. A matrix is a projection matrix if and only if it is symmetric and idempotent. Proof. Together, Corollary 12.3.3 and Parts (3) and (6) of Theorem 12.3.4 imply that every projection matrix is symmetric and idempotent. Thus, to prove Theorem 12.3.9, it suffices to show that every symmetric idempotent matrix is a projection matrix. Suppose that A is a symmetric idempotent matrix. Then, A0 A D A2 D A, so that any generalized inverse of A0 A is a generalized inverse of A. Thus, PA D A.A0 A/ A0 D A.A0 A/ A D A: We conclude, on the basis of Corollary 12.3.3, that A is a projection matrix. Q.E.D. Note that, together, Theorems 12.3.9 and 12.3.8 imply that if a matrix A is symmetric and idempotent, then it is the projection matrix for C.A/.

12.4 Least Squares Problem Suppose that Y is a matrix in a linear space V and that U is a subspace of V. Then, as indicated by Theorem 12.1.3, there exists a unique matrix Z in U, called the projection of Y on U, such that the difference Y Z between Y and Z is orthogonal to U. A fundamental property of the projection of Y on U is described in the following theorem. Theorem 12.4.1. Let Y represent a matrix in a linear space V, and let U represent a subspace of V. Then, for W 2 U, the distance kY W k between Y and W is minimized uniquely by taking W to be the projection Z of Y on U. Moreover, kY Zk2 D Y .Y Z/: Proof. For any matrix W in U, kY W k2 D k.Y Z/ .W Z/k2 D .Y Z/ .Y Z/ 2.Y Z/ .W Z/ C .W Z/ .W Z/:

12.5 Orthogonal Complements

171

Further, W Z is in U and, by definition, Y Z is orthogonal to every matrix in U. Thus, .Y Z/ .W Z/ D 0, and hence kY W k2 D .Y Z/ .Y Z/ C .W Z/ .W Z/ D kY Zk2 C kW Zk2 kY Zk2 ; with equality holding if and only if W D Z. It follows that, for W 2 U, kY W k2 and, consequently, kY W k are minimized uniquely by taking W D Z. That kY Zk2 D Y .Y Z/ is clear upon observing that Z 2 U and hence Q.E.D. that Z .Y Z/ D 0. Suppose that y D fyi g is an n-dimensional column vector and that U is a subspace of the linear space Rn of all n-dimensional column vectors. Consider the problem of minimizing, for w D fwi g in U, the sum of squares .y w/0 .y w/ D P n 2 0 1=2 (with respect iD1 .yi wi / or, equivalently, the distance Œ.y w/ .y w/ to the usual norm) between y and w. This minimization problem is known as the least squares problem. The solution of the least squares problem can be obtained as a special case of Theorem 12.4.1. Theorem 12.4.1 implies that, for w 2 U, the distance between y and w is minimized uniquely by taking w to be the projection of y on U. Thus, in light of Theorem 12.2.1, we reach the following conclusion. Theorem 12.4.2. Let U represent a subspace of the linear space Rn of all ndimensional column vectors, take X to be an np matrix such that C.X/ D U, and let y represent a vector in Rn . Then, for w 2 U, the sum of squares .y w/0 .y w/ of the elements of the difference y w between y and w is minimized uniquely by taking w D Xb , where b is any solution to the normal equations X0 Xb D X0 y, or, equivalently, by taking w D PX y. Further, the minimum value of the sum of squares is expressible as .y Xb /0 .y Xb / D y 0 .y Xb / D y 0 .I PX /y: In connection with Theorem 12.4.2, observe that U comprises every n 1 vector w that is expressible as w D Xb for some vector b. Thus, in light of Theorem 12.2.5, we have the following variant of Theorem 12.4.2. Theorem 12.4.3. Let X D fxij g represent an n p matrix, and y D fyi g p an n-dimensional column Pnvector. Then, Pp for b D2fbi g in R , the sum of squares 0 .y Xb/ .y Xb/ D iD1 .yi j D1 xij bj / has a minimum at a point b if and only if b is a solution to the normal equations X0 Xb D X0 y, in which case Xb D PX y and .y Xb /0 .y Xb / D y 0 .y Xb / D y 0 .I PX /y:

172


12.5 Orthogonal Complements a. Some general results, terminology, and notation The set of all matrices in a linear space V that are orthogonal to a subspace U of V is called the orthogonal complement of U relative to V or simply the orthogonal complement of U. The symbol U? is used to represent the orthogonal complement of U. Note that, by definition, the orthogonal complement of a subspace U of a linear space V depends on V as well as U (and also on the choice of inner product). Since this dependence is not evident in the symbol U? , the use of this symbol is restricted to settings where the identity of the linear space V, relative to which the orthogonal complement of U is defined, is clear from the context. If U is a subspace of a linear space V, then U? is also a subspace of V, as is easily verified. As an immediate consequence of Lemma 12.1.1, we have the following lemma. Lemma 12.5.1. Let U represent a subspace of a linear space V, and let fB1 ; : : : ; Bk g represent any set of matrices that spans U. Then, a matrix A in V belongs to U? if and only if A is orthogonal to each of the matrices B1 ; : : : ; Bk . For any n p matrix X, the symbol C? .X/ is used to represent the orthogonal complement of C.X/ relative to the linear space Rn of all n-dimensional column vectors (taking the inner product to be the usual inner product for Rn ). Similarly, the symbol R? .X/ is used to represent the orthogonal complement of R.X/ relative to the linear space Rp of all p-dimensional row vectors. It follows from Corollary 12.1.2 that C? .X/ is the set of all solutions to the homogeneous linear system X0 z D 0 (in z). Thus, recalling (from Theorem 12.3.4) that X.X0 X/ is a generalized inverse of X0 , we have (in light of Corollary 11.2.2) the following lemma. Lemma 12.5.2. For any n p matrix X, C? .X/ D N.X0 / D C.I PX /: As a consequence of Lemma 11.3.1 or, alternatively, Theorem 12.3.4, we have the following corollary of Lemma 12.5.2. Corollary 12.5.3. For any n p matrix X, dimŒC? .X/ D n rank.X/ D n dimŒC.X/:

(5.1)

The following theorem describes a fundamental property of the orthogonal complement of a subspace U of a linear space V. Theorem 12.5.4. A matrix A in a linear space V belongs to a subspace U of V if and only if A is orthogonal to every matrix in U? [i.e., if and only if A belongs to .U? /? ]. Or, equivalently, U D .U? /? .


173

Proof. Suppose that A 2 U. Then, it follows immediately from the definition of U? that A is orthogonal to every matrix in U? . Conversely, suppose that A is orthogonal to every matrix in U? . Let B represent the projection of A on U. Then, by definition, .A B/ ? U or, equivalantly, .A B/ 2 U? . Thus, .A B/ .A B/ D A .A B/ B .A B/ D 0 0 D 0; implying that A B D 0, or equivalently that A D B, and hence that A 2 U. Q.E.D. Various implications of Theorem 12.5.4 are set forth in the following three corollaries. Corollary 12.5.5. A subspace W of a linear space V is contained in a subspace U of V if and only if W ? U? (i.e., if and only if W is orthogonal to U? ). Proof. Corollary 12.5.5 follows from Theorem 12.5.4 upon observing that Q.E.D. W ? U? , W .U? /? . Corollary 12.5.6. Let U and W represent subspaces of a linear space V. Then, (1) W U if and only if U? W ? , and (2) W D U if and only if W ? D U? . Proof. (1) Making use of Corollary 12.5.5 and Theorem 12.5.4, we find that W U , W ? U? , .W ? /? ? U? , U? ? .W ? /? , U? W ? : (2) If W ? D U? , then W ? U? and U? W ? , implying [in light of Part (1)] that U W and W U and hence that W D U. Conversely, if W D U, then W U and U W, implying that U? W ? and W ? U? and hence Q.E.D. that W ? D U? . Corollary 12.5.7. Let X represent an n p matrix, and let Z represent any n s matrix whose columns span C? .X/ or, equivalently, span N.X0 /. Then, for any n q matrix Y , C.Y / C.X/ if and only if Z0 Y D 0 or, equivalently, if and only if Y 0 Z D 0. Proof. In light of Corollary 12.1.2 and Corollary 12.5.5, we have that Z0 Y D 0 , C.Y / ? C.Z/ , C.Y / ? C? .X/ [since, by definition, C.Z/ D C? .X/] , C.Y / C.X/: Q.E.D. Note that, for any matrix Y in the orthogonal complement U of a subspace U of a linear space V of m n matrices, the m n null matrix 0 is the projection of Y on U. ?

174


b. Projection on an orthogonal complement For any matrix Y in a linear space V, there is a simple relationship between the projection of Y on a subspace U and the projection of Y on the orthogonal complement U? of U. This relationship is described in the following theorem. Theorem 12.5.8. Let Y represent a matrix in a linear space V, and let U represent a subspace of V. Then, the projection of Y on U? equals Y Z, where Z is the projection of Y on U. Proof. By definition, Y Z is orthogonal to U, and hence .Y Z/ 2 U? . Thus, to show that Y Z is the projection of Y on U? , it suffices to show that Y .Y Z/ is orthogonal to U? . That is, it suffices to show that Z is orthogonal to every matrix in U? or, equivalently (in light of Theorem 12.5.4), that Z 2 U. That Z 2 U is clear from the definition of Z. Q.E.D. In light of Theorem 12.2.1, we have the following two corollaries of Theorem 12.5.8. Corollary 12.5.9. Let X represent an n p matrix, and let z represent the projection (with respect to the usual inner product) of a column vector y on C? .X/. Then, z D y Xb for any solution b to the normal equations X0 Xb D X0 y (in b). In particular, z D y PX y D .I PX /y: Corollary 12.5.10. Let X represent an n p matrix. Then, the projection matrix for C? .X/ is I PX . Various of the results included in Theorems 12.1.3 and 12.5.8 can be combined and restated as follows: Theorem 12.5.11. Let Y represent a matrix in a linear space V, and let U represent a subspace of V. Then, U contains a unique matrix Z and U? a unique matrix W such that Y D Z C W . Moreover, Z is the projection of Y on U, and W is the projection of Y on U? . c. Two-dimensional example revisited Let us find, for the two-dimensional example considered previously in Section 12.2b, the projection (with respect to the usual inner product) of an n-dimensional column vector y on the orthogonal complement U? of a subspace U of Rn . Recall that y D .4; 8/0 , U D sp.x/ D C.X/, where x D .3; 1/0 and X is the 2 1 matrix whose only column is x, and that the projection of y on U is z D .6; 2/0 . It follows from Corollary 12.5.3 and Lemma 12.5.2 that dim.U? / D dimŒC? .X/ D n rank.X/ D 2 1 D 1 and that U? D C? .X/ D sp.v /, where v is any nonnull solution to the honogeneous linear system X0 v D 0 (in v), for example, v D .1; 3/0 . Furthermore, the projection of y on U? is w D y z D .2; 6/0 , as depicted in Figure 12.3.


175

FIGURE 12.3: The projection w of a two-dimensional column vector y on the orthogonal complement U? of a one-dimensional subspace U of R2.

d. Three-dimensional example revisited Let us find, for the three-dimensional example considered previously in Section 12.2c, the projection (with respect to the usual inner product) of an n-dimensional column vector y on the orthogonal complement U? of a subspace U of Rn . Recall that y D .3; 38=5; 74=5/0 , U D sp.x1 ; x2 ; x3 / D C.X/, where x1 , x2 , and x3 are given by expressions (2.4) and X is the 3 3 matrix with columns x1 , x2 , and x3 , and the projection of y on U is z D .3; 22=5; 44=5/0 . We find that dim.U? / D n rank.X/ D 3 2 D 1 and that U? D C? .X/ D sp.v /, where v is any nonnull solution to the homogeneous linear system X0 v D 0, for example, v D .0; 2; 1/0 . Furthermore, the projection of y on U? is w D y z D .0; 12; 6/0 , as depicted in Figure 12.4. e. Dimension of an orthogonal complement Formula (5.1) for the dimension of the orthogonal complement C? .X/ of the column space C.X/ of an n p matrix X can be extended to the orthogonal complement of any subspace of any linear space. To do so, suppose that U is an rdimensional subspace of an n-dimensional linear space V, define k D dim.U? /, and let M D fA1 ; : : : ; Ar g and T D fB1 ; : : : ; Bk g represent orthonormal bases for U and U? , respectively. In addition, take S D fA1 ; : : : ; Ar ; B1 ; : : : ; Bk g to be

176


FIGURE 12.4: The projection w of a three-dimensional column vector y on the orthogonal complement U? of a two-dimensional subspace U of R3.

the set whose members consist of the r matrices in M together with the k matrices in T . Theorem 12.5.11 implies that every matrix Y in V is expressible as Y D ZCW for some matrix Z in U and some matrix W in U? and hence that the set S spans V. Moreover, since Ai 2 U and Bj 2 U?, Ai and Bj are orthogonal (i D 1, : : : ; r; j D 1; : : : ; k), implying that S is orthonormal and consequently linearly independent. We conclude that S is a basis for V and hence that n D r Ck, thereby establishing the following generalization of Corollary 12.5.3. Theorem 12.5.12. Let U represent a subspace of a linear space V. Then, dim.V/ D dim.U/ C dim.U? / or, equivalently, dim.U? / D dim.V/ dim.U/.

Exercises

177

Exercises Section 12.1 1. Verify Lemma 12.1.1. 2. Let U and V represent subspaces of Rmn . Show that if dim.V/ > dim.U/, then V contains a nonnull matrix that is orthogonal to U. Section 12.2 3. Let U represent a subspace of the linear space Rm of all m-dimensional column vectors. Take M to be the subspace of Rmn defined by W 2 M if and only if W D .w1 ; : : : ; wn / for some vectors w1 ; : : : ; wn in U. Let Z represent the projection (with respect to the usual inner product) of an m n matrix Y on M, and let X represent any m p matrix whose columns span U. Show that Z D XB for any solution B to the linear system X0 XB D X0 Y (in B): 4. For the example of Section 12.2c, recompute the projection of y on U by taking X to be the 3 2 matrix 0 1 0 2 @3 2A 6 4 and carrying out the following two steps: (1) compute the solution to the normal equations X0 Xb D X0 y; (2) postmultiply X by the solution you computed in Step (1). Section 12.3 5. Let X represent any n p matrix. It follows from Parts (1)–(3) of Theorem 12.3.4 that if a p n matrix B is a solution to the linear system X0 XB D X0 (in B), then B is a generalized inverse of X, and XB is symmetric. Show that, conversely, if a p n matrix G is a generalized inverse of X, and XG is symmetric, then X0 XG D X0 (i.e., G is a solution to X0 XB D X0 ). 6. Using the result of Part (b) of Exercise 9.3 (or otherwise), show that, for any nonnull symmetric matrix A, PA D B.T B/1 T ; where B is any matrix of full column rank and T any matrix of full row rank such that A D BT . (That T B is nonsingular follows from the result of Exercise 8.3.)

178


7. Let V represent a k-dimensional subspace of the linear space Rn of all ndimensional column vectors. Take X to be any n p matrix whose columns span V, let U represent a subspace of V, and define A to be the projection matrix for U. Show (1) that a matrix B (of dimensions n n) is such that By is the projection of y on U for every y 2 V if and only if B D A C Z0 for some solution Z to the honogeneous linear system X0 Z D 0 (in an n n matrix Z), and (2) that, unless k D n, there is more than one matrix B such that By is the projection of y on U for every y 2 V. Section 12.5 8. Let fA1 ; : : : ; Ak g represent a nonempty linearly independent set of matrices in a linear space V, and define B1 ; : : : ; Bk as in Theorem 6.4.1. Show that Bj is the (orthogonal) projection of Aj on some subspace Uj (of V) and describe Uj (j D 2; : : : ; k).

13 Determinants

Determinants are encountered with considerable frequency in the statistics literature (and in the literature of various other disciplines that involve the notion of randomness). A determinant appears in the “normalizing constant” of the probability density function of the all-important multivariate normal distribution (e.g., Searle 1971, sec. 2.4f). And the definition of the generalized variance (or generalized dispersion) of a random vector involves a determinant—in the design of experiments, D-optimal designs are obtained by minimizing a generalized variance (e.g., Fedorov 1972).

13.1 Some Definitions, Notation, and Special Cases a. Negative versus positive pairs Associated with any square matrix is a scalar that is known as the determinant of the matrix. As a preliminary to defining the determinant, it is convenient to introduce a convention for classifying various pairs of matrix elements as either positive or negative. Let A D faij g represent an arbitrary nn matrix. Consider any pair of elements of A that do not lie either in the same row or the same column, say aij and ai 0 j 0 (where i 0 ¤ i and j 0 ¤ j ). The pair is said to be a negative pair if one of the elements is located above and to the right of the other, or, equivalently, if either i 0 > i and j 0 < j or i 0 j . Otherwise (if one of the elements is located above and to the left of the other, or, equivalently, if either i 0 > i and j 0 > j or i 0 j j0 < j

i0 > i C

i0 < i C

180

13 Determinants

For example (supposing that n 4), the pair a34 and a22 is positive, whereas the pair a34 and a41 is negative. Note that whether the pair aij and ai 0 j 0 is positive or negative is completely determined by the relative locations of aij and ai 0 j 0 and has nothing to do with whether aij and ai 0 j 0 are positive or negative numbers. Now, consider n elements of A, no two of which lie either in the same row or the same column, say the i1 j1 ; : : : ; in jn th elements (where both i1 ; :: : ; in and j1 ; : : : ; jn are permutations of the first n positive integers). A total of n2 pairs can be formed from these n elements. The symbol n .i1 ; j1 I : : : I in ; jn / is to be used to represent the number of these n2 pairs that are negative pairs. Observe that n .i1 ; j1 I : : : I in ; jn / has the following two properties: (1) the value of n .i1 ; j1 I : : : I in ; jn / is not affected by permuting its n pairs of arguments; in particular, it is not affected if the n pairs are permuted so that they are ordered by row number or by column number [e.g., 3 .2; 3I 1; 2I 3; 1/ D 3 .1; 2I 2; 3I 3; 1/ D 3 .3; 1I 1; 2I 2; 3/]; (2)

n .i1 ; j1 I : : : I in ; jn / D n .j1 ; i1 I : : : I jn ; in / [e.g., 3 .2; 3I 1; 2I 3; 1/ D 3 .3; 2I 2; 1I 1; 3/]. For any sequence of n distinct integers i1 ; : : : ; in , define n .i1 ; : : : ; in / D p1 C C pn1 ;

where pk represents the number of integers in the subsequence ikC1 ; : : : ; in that are smaller than ik (k D 1; : : : ; n 1). For example, 5 .3; 7; 2; 1; 4/ D 2 C 3 C 1 C 0 D 6: Then, clearly, for any permutation i1 ; : : : ; in of the first n positive integers, n .1; i1 I : : : I n; in / D n .i1 ; 1I : : : I in ; n/ D n .i1 ; : : : ; in /:

(1.1)

b. Definition of determinant The determinant of an n n matrix A D faij g, to be denoted by jAj or (to avoid confusion with the absolute value of a scalar) by det A or det.A/, is defined by X (1.2a) jAj D .1/n .1;j1 I:::In;jn / a1j1 anjn or equivalently by jAj D

X

.1/n .j1 ;:::;jn / a1j1 anjn ;

(1.2b)

where j1 ; : : : ; jn is a permutation of the first n positive integers and the summation is over all such permutations. Thus, the determinant of an nn matrix A can (at least in principle) be obtained via the following process:

13.1 Some Definitions, Notation, and Special Cases

181

(1) Form all possible products, each of n factors, that can be obtained by picking one and only one element from each row and column of A.

(2) In each product, count the number of negative pairs among the n2 pairs of elements that can be generated from the n elements that contribute to this particular product. If the number of negative pairs is an even number, attach a plus sign to the product; if it is an odd number, attach a minus sign. (3) Sum the signed products. In particular, the determinant of a 1 1 matrix A D .a11 / is jAj D a11 I

(1.3)

the determinant of a 2 2 matrix A D faij g is jAj D .1/0 a11 a22 C .1/1 a12 a21 D a11 a22 a12 a21 I

(1.4)

and the determinant of a 3 3 matrix A D faij g is jAj D .1/0 a11 a22 a33 C .1/1 a11 a23 a32 C .1/1 a12 a21 a33 C .1/2 a12 a23 a31 C .1/2 a13 a21 a32 C .1/3 a13 a22 a31 D a11 a22 a33 C a12 a23 a31 C a13 a21 a32 a11 a23 a32 a12 a21 a33 a13 a22 a31 :

(1.5)

A11 A12 , it is customary to write In the case of a partitioned matrix, say A21 A22 ˇ ˇ ˇ ˇ ˇA11 A12 ˇ ˇ A11 A12 ˇ ˇ ˇ ˇ ˇ ˇ A21 A22 ˇ in the abbreviated form ˇA21 A22 ˇ. An alternative definition of the determinant of an n n matrix A D faij g is X jAj D .1/n .i1 ;1I:::Iin ;n/ ai1 1 ain n (1.6a) X (1.6b) D .1/n .i1 ;:::;in / ai1 1 ain n ; where i1 ; : : : ; in is a permutation of the first n positive integers and the summation is over all such permutations. Definition (1.6) is equivalent to definition (1.2). To see this, observe that the product a1j1 anjn , which appears in definition (1.2), can be reexpressed by permuting the n factors a1j1 ; : : : ; anjn so that they are ordered by column number, giving a1j1 anjn D ai1 1 ain n ; where i1 ; : : : ; in is a permutation of the first n positive integers that is defined uniquely by ji1 D 1; : : : ; jin D n:

182

13 Determinants

Further, n .1; j1 I : : : I n; jn / D n .i1 ; ji1 I : : : I in ; jin / D n .i1 ; 1I : : : I in ; n/; so that .1/n .1;j1 I:::In;jn / a1j1 anjn D .1/n .i1 ;1I:::Iin ;n/ ai1 1 ain n : Thus, we can establish a one-to-one correspondence between the terms of the sum (1.6) and the terms of the sum (1.2) such that the corresponding terms are equal. We conclude that the two sums are themselves equal. The number of terms in the sum (1.2) or (1.6) equals nŠ. As n increases, the number of terms grows rapidly. As a consequence, the computations required for the direct numerical evaluation of this sum can be extensive and (for even moderately large values of n) prohibitive. Representations that are better suited for computational purposes can be devised by making use of various results to be presented subsequently. c. Diagonal, triangular, and permutation matrices There is a simple expression for the determinant of a triangular matrix, which is given by the following lemma. Lemma 13.1.1. If an n n matrix A D faij g is (upper or lower) triangular, then (1.7) jAj D a11 a22 ann ; that is, the determinant of a triangular matrix equals the product of its diagonal elements. Proof. Consider a lower triangular matrix 1 0 a11 0 : : : 0 Ba21 a22 0 C C B ADB : C: : : : : : @ : A : : an1 an2 : : : ann That jAj D a11 a22 ann follows immediately upon observing that the only term in the sum (1.2) that can be nonzero is that corresponding to the permutation j1 D 1; : : : ; jn D n [and that n .1; 2; : : : ; n/ D 0]. (To verify formally that only this one term can be nonzero, let j1 ; : : : ; jn represent an arbitrary permutation of the first n positive integers, and suppose that a1j1 anjn ¤ 0 or equivalently that aiji ¤ 0 for i D 1; : : : ; n. Then, it is clear that j1 D 1 and that, if j1 D 1; : : : ; ji1 D i 1, then ji D i . We conclude, on the basis of mathematical induction, that j1 D 1; : : : ; jn D n). The validity of formula (1.7) as applied to an upper triangular matrix follows from a similar argument. Q.E.D.

13.2 Some Basic Properties of Determinants

183

Corollary 13.1.2. The determinant of a diagonal matrix equals the product of its diagonal elements. As obvious special cases of Corollary 13.1.2, we have that j0j D 0;

(1.8)

jIj D 1:

(1.9)

The following lemma gives a simple expression for the determinant of a permutation matrix. Lemma 13.1.3. For an n n permutation matrix P whose first, : : : ; nth rows or columns are respectively the k1 , : : : ; kn th rows or columns of the n n identity matrix, jP j D .1/n .k1 ;:::;kn / : Proof. Putting A D P , we find that all of the terms in sum (1.6) are zero except the term corresponding to the permutation i1 D k1 ; : : : ; in D kn , which Q.E.D. equals .1/n .k1 ;:::;kn / .

13.2 Some Basic Properties of Determinants The following lemma relates the determinant of a matrix to the determinant of its transpose. Lemma 13.2.1. For any n n matrix A, jA0 j D jAj:

(2.1)

Proof. Let aij and bij represent the ij th elements of A and A0 , respectively (i; j D 1; : : : ; n). Then, in light of the equivalence of definitions (1.6) and (1.2), X .1/n .j1 ;:::;jn / b1j1 bnjn jA0 j D X D .1/n .j1 ;:::;jn / aj1 1 ajn n D jAj; where j1 ; : : : ; jn is a permutation of the first n positive integers and the summations are over all such permutations. Q.E.D. As an immediate consequence of the definition of a determinant, we have the following lemma. Lemma 13.2.2. If an n n matrix B is formed from an n n matrix A by multiplying all of the elements of one row or one column of A by the same scalar k (and leaving the elements of the other n 1 rows or columns unchanged), then jBj D kjAj:

184

13 Determinants

As a corollary of Lemma 13.2.2, we obtain the following generalization of result (1.8). Corollary 13.2.3. If one or more rows (or columns) of an n n matrix A are null, then jAj D 0: Proof. Suppose that the i th row of A is null, and let B represent an n n matrix formed from A by multiplying all of the elements of the i th row of A by zero. Then, A D B, and we find that jAj D jBj D 0jAj D 0: Q.E.D. The following corollary (of Lemma 13.2.2) relates the determinant of a scalar multiple of a matrix A to that of A itself. Corollary 13.2.4. For any n n matrix A and any scalar k, jkAj D k n jAj:

(2.2)

Proof. This result follows from Lemma 13.2.2 upon observing that kA can be formed from A by successively multiplying each of the n rows of A by k. Q.E.D. As a special case of Corollary 13.2.4, we have the following, additional corollary. Corollary 13.2.5. For any n n matrix A, jAj D .1/n jAj:

(2.3)

The following two theorems describe how the determinant of a matrix is affected by permuting its rows or columns in certain ways. Theorem 13.2.6. If an n n matrix B D fbij g is formed from an n n matrix A D faij g by interchanging two rows or two columns of A, then jBj D jAj: Proof. Consider first the case where B is formed from A by interchanging two adjacent rows, say the i th and .i C 1/th rows. Then, X jBj D .1/n .j1 ;:::;jn / b1j1 bnjn X D .1/n .j1 ;:::;jn / a1j1 ai1;ji 1 aiC1;ji ai;ji C1 aiC2;ji C2 anjn X D .1/n .j1 ;:::;ji 1 ;ji C1 ;ji ;ji C2 ;:::;jn / a1j1 ai1;ji 1 ai;ji C1 aiC1;ji aiC2;ji C2 anjn Œsince n .j1 ; : : : ; ji1 ; jiC1 ; ji ; jiC2 : : : ; jn / D n .j1 ; : : : ; jn / C 1; if jiC1 > ji ; D n .j1 ; : : : ; jn / 1; if jiC1 < ji D jAj;

13.2 Some Basic Properties of Determinants

185

where j1 ; : : : ; jn (and hence j1 ; : : : ; ji1 ; jiC1 ; ji ; jiC2 ; : : : ; jn ) is a permutation of the first n positive integers and the summation is over all such permutations. Consider now the case where B is formed from A by interchanging two notnecessarily-adjacent rows, say the i th and kth rows where k > i . Suppose that we successively interchange the kth row of A with the k i rows immediately preceding it, putting the n rows of A in the order 1; : : : ; i 1; k; i; i C 1; : : : ; k 1; k C1; : : : ; n. Suppose that we then further reorder the rows of A by successively interchanging what was originally the i th row with the k i 1 rows immediately succeeding it, putting the n rows in the order 1; : : : ; i 1; k; i C1; : : : ; k 1; i; k C 1; : : : ; n. Thus, by executing 2.k i /1 successive interchanges of adjacent rows, we have in effect interchanged the i th and kth rows of A. Since each interchange of adjacent rows changes the sign of the determinant, we conclude that jBj D .1/2.ki/1 jAj D jAj: By employing an analogous argument, we find that the interchange of any two columns of A likewise changes the sign of the determinant. Q.E.D. Theorem 13.2.7. If B is an n p matrix (where p < n) and C an n q matrix (where q D n p), then jB; Cj D .1/pq jC; Bj: Similarly, if B is a p n matrix and C a q n matrix, then ˇ ˇ ˇ ˇ ˇ ˇ ˇB ˇ ˇ ˇ D .1/pq ˇC ˇ : ˇB ˇ ˇC ˇ

(2.4)

(2.5)

Proof. Suppose that the dimensions of B are n p and those of C are n q. Let b1 ; : : : ; bp represent the columns of B and c1 ; : : : ; cq the columns of C. Then, .C; B/ D .c1 ; : : : ; cq ; b1 ; : : : ; bp /. Suppose that, in the matrix .C; B/, we successively interchange column b1 with the columns cq ; : : : ; c1 , producing the matrix .b1 ; c1 ; : : : ; cq ; b2 ; : : : ; bp /. Suppose then that, in the latter matrix, we successively interchange column b2 with the columns cq ; : : : ; c1 , producing the matrix .b1 ; b2 ; c1 ; : : : ; cq ; b3 ; : : : ; bp /. Continuing in this fashion, we produce (after p steps) the matrix .b1 ; : : : ; bp ; c1 ; : : : ; cq / D .B; C/. It is now clear that we can obtain the matrix .B; C/ from the matrix .C; B/ via a total of pq successive interchanges of columns. Thus, it follows from Theorem 13.2.6 that jB; Cj D .1/pq jC; Bj: Result (2.5) can be derived via an analogous approach. Q.E.D. A (square) matrix that has one or more null rows or columns has (according to Corollary 13.2.3) a zero determinant. Other matrices whose determinants are zero are described in the following two lemmas. Lemma 13.2.8. If two rows or two columns of an nn matrix A are identical, then jAj D 0.

186

13 Determinants

Proof. Suppose that two rows of A are identical, say the i th and kth rows, and let B represent a matrix formed from A by interchanging its i th and kth rows. Obviously, B D A and hence jBj D jAj. Moreover, according to Theorem 13.2.6, jBj D jAj. Thus, jAj D jBj D jAj, implying that jAj D 0. That the determinant of a (square) matrix having two identical columns equals zero can be proved via an analogous argument. Q.E.D. Lemma 13.2.9. If a row or column of an n n matrix A is a scalar multiple of another row or column, then jAj D 0. Proof. Let a01 ; : : : ; a0n represent the rows of A. Suppose that one row is a scalar multiple of another, that is, that a0s D ka0i for some s and i (with s ¤ i ) and some scalar k. Let B represent a matrix formed from A by multiplying the i th row of A by the scalar k. Then, according to Lemmas 13.2.2 and 13.2.8, kjAj D jBj D 0:

(2.6)

If k ¤ 0, then it follows from equality (2.6) that jAj D 0. If k D 0, then a0s D 0, and it follows from Corollary 13.2.3 that jAj D 0. Thus, in either case, jAj D 0. An analogous argument shows that if one column of a (square) matrix is a scalar multiple of another, then again the determinant of the matrix equals zero. Q.E.D. The transposition of a (square) matrix does not (according to Lemma 13.2.1) affect its determinant. Other operations that do not affect the determinant of a matrix are described in the following two theorems. Theorem 13.2.10. Let B represent a matrix formed from an n n matrix A by adding, to any one row or column of A, scalar multiples of one or more other rows or columns. Then, jBj D jAj: Proof. Let a0i D .ai1 ; : : : ; ai n / and b0i D .bi1 ; : : : ; bi n / represent the i th rows of A and B, respectively (i D 1; : : : ; n). For some integer s (1 s n) and some scalars k1 ; : : : ; ks1 ; ksC1 ; : : : ; kn , X b0s D a0s C ki a0i and b0i D a0i .i ¤ s/: i¤s

Thus, jBj D D

X X

.1/n .j1 ;:::;jn / b1j1 bnjn .1/n .j1 ;:::;jn /

D jAj C

X

a1j1 as1;js1 asjs C ki aijs asC1;jsC1 anjn

XX

i¤s

.1/

i¤s

D jAj C

X i¤s

n .j1 ;:::;jn /

a1j1 as1;js1 .ki aijs /asC1;jsC1 anjn jBi j;

13.3 Partitioned Matrices, Products of Matrices, and Inverse Matrices

187

where Bi is a matrix formed from A by replacing the sth row of A with ki a0i and where j1 ; : : : ; jn is a permutation of the first n positive integers and the (unlabeled) summations are over all such permutations. Since (according to Lemma 13.2.9) jBi j D 0 .i ¤ s/; we conclude that jBj D jAj. An analogous argument shows that jBj D jAj when B is formed from A by adding, to a column of A, scalar multiples of other columns. Q.E.D. Theorem 13.2.11. For any n n matrix A and any (upper or lower) unit triangular matrix T , jAT j D jT Aj D jAj: (2.7) Proof. Consider the case where A is postmultiplied by T and T is unit lower triangular. Define Ti to be a matrix formed from In by replacing the i th column of In with the i th column of T (i D 1; : : : ; n). It is easy to verify that T D T1 T2 Tn and hence that AT D AT1 T2 Tn : Define B0 D A, and Bi D AT1 T2 Ti (i D 1; : : : ; n 1). Clearly, to show that jAT j D jAj, it suffices to show that, for i D 1; : : : ; n, the postmultiplication of Bi1 by Ti does not alter the determinant of Bi1 . Observe that the columns of Bi1 Ti are the same as those of Bi1 , except for the i th column of Bi1 Ti , which consists of the i th column of Bi1 plus scalar multiples of the .i C 1/, : : : ; nth columns of Bi1 . Thus, it follows from Theorem 13.2.10 that jBi1 Ti j D jBi1 j: We conclude that jAT j D jAj. That result (2.7) is valid for the cases of postmultiplication by a unit upper triangular matrix, and premultiplication by a unit upper or lower triangular matrix can be established via similar arguments. Q.E.D.

13.3 Partitioned Matrices, Products of Matrices, and Inverse Matrices Formula (1.7) for the determinant of a triangular matrix can be extended to a block-triangular matrix, as indicated by the following theorem. Theorem 13.3.1. Let T represent an m m matrix, V an n m matrix, and W an n n matrix. Then, ˇ ˇ ˇ ˇ ˇ T 0 ˇ ˇW V ˇ ˇ ˇDˇ ˇ (3.1) ˇV W ˇ ˇ 0 T ˇ D jT jjW j:

188

13 Determinants

Proof. Let

T 0 AD V W

;

and let aij represent the ij th element of A (i; j D 1; : : : ; m C n). By definition, X jAj D .1/mCn .j1 ;:::;jmCn / a1j1 amCn;jmCn ; (3.2) where j1 ; : : : ; jmCn is a permutation of the first m C n positive integers and the summation is over all such permutations. Clearly, the only terms of the sum (3.2) that can be nonzero are those for which j1 ; : : : ; jm constitutes a permutation of the first m positive integers and thus for which jmC1 ; : : : ; jmCn constitutes a permutation of the integers mC1; : : : ; mCn. For any such permutation, we have that a1j1 amCn;jmCn D t1j1 tmjm w1;jmC1 m wn;jmCn m D t1j1 tmjm w1k1 wnkn ; where tij represents the ij th element of T (i; j D 1; : : : ; m) and wij the ij th element of W (i; j D 1; : : : ; n) and where k1 D jmC1 m, : : : ; kn D jmCn m, and that mCn .j1 ; : : : ; jmCn / D m .j1 ; : : : ; jm / C n .jmC1 ; : : : ; jmCn / D m .j1 ; : : : ; jm / C n .jmC1 m; : : : ; jmCn m/ D m .j1 ; : : : ; jm / C n .k1 ; : : : ; kn /: Thus, XX

.1/m .j1 ;:::;jm /Cn .k1 ;:::;kn / t1j1 tmjm w1k1 wnkn X X D .1/m .j1 ;:::;jm / t1j1 tmjm .1/n .k1 ;:::;kn / w1k1 wnkn

jAj D

D jT jjW j; where j1 ; : : : ; jm is a permutation of the first m positive integers and k1 ; : : : ; kn a permutation of the first n positive integers, and where the summations are over all such permutations. ˇ ˇ ˇW V ˇ ˇ D jT jjW j can be established via a similar argument. Q.E.D. ˇ That ˇ 0 Tˇ The repeated application of Theorem 13.3.1 leads to the following formulas for the determinant of an arbitrary (square) upper or lower block-triangular matrix (with square diagonal blocks): ˇ ˇ ˇA11 A12 : : : A1r ˇ ˇ ˇ ˇ 0 A22 : : : A2r ˇ ˇ ˇ (3.3) ˇ :: :: ˇ D jA11 jjA22 j jArr jI :: ˇ : ˇ : : ˇ ˇ ˇ 0 0 Arr ˇ

13.3 Partitioned Matrices, Products of Matrices, and Inverse Matrices

ˇ ˇ ˇB11 0 : : : 0 ˇ ˇ ˇ ˇB21 B22 0 ˇˇ ˇ ˇ :: ˇ D jB11 jjB22 j jBrr j: :: : : ˇ : ˇ : : ˇ ˇ ˇ Br1 Br2 : : : Brr ˇ

189

(3.4)

In the special case of a block-diagonal matrix, formula (3.3) becomes diag.A11 ; A22 ; : : : ; Arr / D jA11 jjA22 j jArr j:

(3.5)

By making use of Theorem 13.2.7, we obtain the following corollary of Theorem 13.3.1. Corollary 13.3.2. Let T represent an m m matrix, V an n m matrix, and W an n n matrix. Then, ˇ ˇ ˇ ˇ ˇ 0 T ˇ ˇV W ˇ mn ˇ ˇDˇ ˇ (3.6) ˇW V ˇ ˇT 0 ˇ D .1/ jT jjW j: The following corollary gives a simplified version of formula (3.6) for the special case where m D n and T D In . Corollary 13.3.3. For n n matrices W and V , ˇ ˇ ˇ ˇ ˇ 0 In ˇ ˇ V W ˇ ˇ ˇDˇ ˇ D jW j: ˇW V ˇ ˇ In 0 ˇ

(3.7)

Proof (of Corollary 13.3.3). Corollary 13.3.3 can be derived from the special case of Corollary 13.3.2 where m D n and T D In by observing that .1/nn jIn jjW j D .1/nn .1/n jW j D .1/n.nC1/ jW j and that either n or n C 1 is an even number and consequently n.n C 1/ is an even number. Q.E.D. By using Theorem 13.3.1 (and Corollary 13.3.3), together with Theorem 13.2.11, we find that, for n n matrices A and B, ˇ ˇ ˇ ˇ ˇ ˇ ˇ A 0ˇ ˇ A 0 I B ˇˇ ˇˇ A ABˇˇ ˇDˇ D jABj; D jAjjBj D ˇˇ 0 I ˇ ˇ I 0 ˇ I Bˇ ˇ I B thereby establishing the following, very important result. Theorem 13.3.4. For n n matrices A and B, jABj D jAjjBj:

(3.8)

The repeated application of Theorem 13.3.4 leads to the following formula for the determinant of the product of an arbitrary number of nn matrices A1 ; A2 ; : : : ; Ak : (3.9) jA1 A2 Ak j D jA1 jjA2 j jAk j:

190

13 Determinants

As a special case of this formula, we obtain the following formula for the determinant of the kth power of an n n matrix A: jAk j D jAjk

(3.10)

(k D 1; 2; : : :). In light of Lemma 13.2.1, we have the following corollary of Theorem 13.3.4. Corollary 13.3.5. For any n n matrix A, jA0 Aj D jAj2 :

(3.11)

According to Lemma 13.1.3, a permutation matrix has a determinant whose absolute value is one. The following corollary indicates that every orthogonal matrix has this property. Corollary 13.3.6. For an orthogonal matrix P , jP j D ˙1: Proof (of Corollary 13.3.6). Using Corollary 13.3.5, together with result (1.9), we find that jP j2 D jP 0 P j D jIj D 1: Q.E.D. Corollary 13.2.3 and Lemmas 13.2.8 and 13.2.9 identify certain matrices that have zero determinants. These results are generalized as part of the following theorem. Theorem 13.3.7. Let A represent an n n matrix. Then, A is nonsingular (or, equivalently, A is invertible) if and only if jAj ¤ 0, in which case jA1 j D 1=jAj:

(3.12)

Proof. It suffices to show that, if A is nonsingular, then jAj ¤ 0 and jA1 j D 1=jAj and that, if A is singular, then jAj D 0. Suppose that A is nonsingular. Then, according to Theorem 13.3.4 and result (1.9), jA1 jjAj D jA1 Aj D jIj D 1; implying that jAj ¤ 0 and further that jA1 j D 1=jAj. Suppose now that A is singular. Then, some column of A, say the sth column as , can be expressed as a linear combination of the other n 1 columns a1 ; : : : ; as1 ; asC1 ; : : : ; an ; that is, X ki ai as D i¤s

for some scalars k1 ; : : : ; ks1 ; kP sC1 ; : : : ; kn . Now, let B represent a matrix formed from A by adding the vector i¤s ki ai to the sth column of A. Clearly, the sth

13.5 Cofactors

191

column of B is null, and it follows from Corollary 13.2.3 that jBj D 0. Moreover, it follows from Theorem 13.2.10 that jAj D jBj. Thus, jAj D 0. Q.E.D. The following theorem gives formulas for the determinant of a partitioned matrix that are analogous to formulas (8.5.16) and (8.5.17) for the inverse of a partitioned matrix. Theorem 13.3.8. n m matrix, and W ˇ ˇT ˇ ˇV

Let T represent an m m matrix, U an m n matrix, V an an n n matrix. If T is nonsingular, then ˇ ˇ ˇ U ˇˇ ˇˇW V ˇˇ D jT jjW V T 1 U j: D (3.13) W ˇ ˇU Tˇ

Proof. Suppose that T is nonsingular. Then, T U I 0 T U D : V W V T 1 W V T 1 U 0 I Applying Theorems 13.3.4 and 13.3.1, we find that ˇ ˇ ˇT U ˇ 1 ˇ ˇ ˇV W ˇ D jT jjW V T U j: ˇ ˇ ˇW V ˇ ˇ D jT jjW V T 1 U j can be proved in similar fashion. Q.E.D. ˇ That ˇ U Tˇ

13.4 A Computational Approach Formula (3.8) and, more generally, formula (3.9) (for the determinant of a product of matrices) can be very useful from a computational standpoint. Suppose that A is an n n matrix whose determinant jAj is of interest and that A can be decomposed into the product of some number of n n matrices, each of whose determinants is easy to “evaluate.” Then, formula (3.9) can be used to evaluate jAj. Suppose, for instance, that we have obtained the QR decomposition A D QR (where Q is orthogonal and R triangular) of an n n matrix A. Then, in light of Corollary 13.3.6, jAj D ˙jRj; and, in light of Lemma 13.1.1, jAj can be determined (up to its absolute value) by forming the product of the diagonal elements of R.

13.5 Cofactors Let A D faij g represent an n n matrix. Let Aij represent the .n 1/ .n 1/ submatrix of A obtained by striking out the row and the column that contain the element aij , that is, the i th row and the j th column. The determinant jAij j of this submatrix is called the minor of the element aij . The “signed” minor .1/iCj jAij j is called the cofactor of aij .

192

13 Determinants

The determinant of A can be expanded in terms of the cofactors of the n elements of any particular row or column of A, as described in the following theorem. Theorem 13.5.1. Let aij represent the ij th element of an n n matrix A, and let ˛ij represent the cofactor of aij (i; j D 1; : : : ; n). Then, for i D 1; : : : ; n, jAj D

n X

aij ˛ij D ai1 ˛i1 C C ai n ˛i n ;

(5.1)

aij ˛ij D a1j ˛1j C C anj ˛nj :

(5.2)

j D1

and, for j D 1; : : : ; n, jAj D

n X iD1

Proof. Let Aij represent the matrix obtained by striking out the i th row and the j th column of A (i; j D 1; : : : ; n). Consider first result (5.1) for the case / i D 1. Denote by a.j ts the t sth element of the matrix A1j (t; s D 1; : : : ; n 1I j D 1; : : : ; n). We find that X jAj D .1/n .k1 ;:::;kn / a1k1 ankn (where k1 ; : : : ; kn is a permutation of the first n positive integers and the summation is over all such permutations) X D a11 .1/n1 .k2 ;:::;kn / a2k2 ankn C X C a1j .1/j 1Cn1 .k2 ;:::;kn / a2k2 ankn C X C a1n .1/n1Cn1 .k2 ;:::;kn / a2k2 ankn (where, in the j th of the n sums, k2 ; : : : ; kn is a permutation of the n 1 integers 1; : : : ; j 1; j C 1; : : : ; n and the summation is over all such permutations) X .1/ .1/ an1;s C D a11 .1/n1 .s1 ;:::;sn1 / a1s 1 n1 X .j / j 1 n1 .s1 ;:::;sn1 / .j / a1s1 an1;s C C a1j .1/ .1/ n1 X .n/ n1 n1 .s1 ;:::;sn1 / .n/ a1s1 an1;sn1 C anj .1/ .1/ (where, in each of the n sums, s1 ; : : : ; sn1 is a permutation of the first n 1 positive integers and the summation is over all such permutations) D

n X

a1j .1/j 1 jA1j j

j D1

D

n X j D1

a1j .1/j C1 jA1j j

13.5 Cofactors

D

n X

193

a1j ˛1j :

j D1

Consider now result (5.1) for the case i > 1. Let B represent the n n matrix whose first row is the i th row of A, whose second, : : : ; i th rows are the first, : : : ; .i 1/th rows, respectively, of A, and whose .i C 1/, : : : ; nth rows are the same as those of A. Observe that A can be obtained from B by successively interchanging the first row of B with its second, : : : ; i th rows, so that, according to Theorem 13.2.6, jAj D .1/i1 jBj: Let B1j represent the matrix obtained by striking out the first row and the j th column of B, and let b1j represent the j th element of the first row of B (j D 1; : : : ; n). Then, B1j D Aij (j D 1; : : : ; n), and we find that jAj D .1/i1 jBj D .1/i1

n X

b1j .1/1Cj jB1j j

j D1

D .1/i1

n X

aij .1/1Cj jAij j

j D1

D

n X

aij ˛ij :

j D1

Finally, consider result (5.2). Observe that the matrix obtained by striking out the j th row and the i th column of A0 is A0ij and hence that the cofactor of the j i th element of A0 is .1/j Ci jA0ij j D .1/iCj jAij j D ˛ij (i; j D 1; : : : ; n). Thus, since the j i th element of A0 is the ij th element of A (i; j D 1; : : : ; n), it follows from result (5.1) (and from Lemma 13.2.1) that jAj D jA0 j D

n X

aij ˛ij

iD1

(j D 1; : : : ; n). Q.E.D. Note that if the j th element aij of the i th row of A equals zero, then the j th term in formula (5.1) also equals zero. Similarly, if aij D 0, then the i th term in formula (5.2) equals zero. Thus, if the n n matrix A contains a row or column that includes many zeroes, then formula (5.1) or (5.2) can be used to reexpress jAj in terms of the determinants of a relatively small number of .n 1/ .n 1/ submatrices of A. The following theorem can be regarded as a companion to Theorem 13.5.1. Theorem 13.5.2. Let aij represent the ij th element of an n n matrix A, and let ˛ij represent the cofactor of aij (i; j D 1; : : : ; n). Then, for i 0 ¤ i D 1; : : : ; n,

194

13 Determinants n X

aij ˛i 0 j D ai1 ˛i 0 1 C C ai n ˛i 0 n D 0;

(5.3)

j D1

and, for j 0 ¤ j D 1; : : : ; n, n X

aij ˛ij 0 D a1j ˛1j 0 C C anj ˛nj 0 D 0:

(5.4)

iD1

Proof. Consider result (5.3). Let B represent a matrix whose i 0 th row equals the i th row of A and whose first, : : : ; .i 0 1/th, .i 0 C 1/th, : : : ; nth rows are identical to those of A (where i 0 ¤ i ). Observe that the i 0 th row of B is a duplicate of its i th row and hence, according to Lemma 13.2.8, that jBj D 0. Let bkj represent the kj th element of B (k; j D 1; : : : ; n). Clearly, the cofactor of bi 0 j is the same as that of aij (j D 1; : : : ; n). Thus, making use of Theorem 13.5.1, we find that n X

aij ˛i 0 j D

j D1

n X

bi 0 j ˛i 0 j D jBj D 0;

j D1

which establishes result (5.3). Result (5.4) can be proved via an analogous argument. Q.E.D. For any n n matrix A D faij g, the n n matrix whose ij th element is the cofactor ˛ij of aij is called the matrix of cofactors (or cofactor matrix) of A. The transpose of this matrix is called the adjoint matrix of A and is denoted by the symbol adj A or adj.A/. Thus, 10 0 1 ˛11 : : : ˛1n ˛11 : : : ˛n1 C B C B adj A D @ ::: : : : ::: A D @ ::: : : : ::: A : ˛n1 : : : ˛nn ˛1n : : : ˛nn 0

If A is symmetric, then the matrix of cofactors is also symmetric (or equivalently the adjoint matrix equals the matrix of cofactors), as is easily verified. There is a close relationship between the adjoint of a nonsingular matrix A and the inverse of A, as is evident from the following theorem and is made explicit in the corollary of this theorem. Theorem 13.5.3. For an n n matrix A, A adj.A/ D .adj A/A D jAjIn :

(5.5)

Proof. Let aij represent the ij th element of A, and let ˛ij represent the 0 cofactor of aP ij (i; j D 1; : : : ; n). Then, the i i th element of the matrix product A adj.A/ is jnD1 aij ˛i 0 j (i; i 0 D 1; : : : ; n). According to Theorems 13.5.1 and 13.5.2,

13.6 Vandermonde Matrices n X

aij ˛i 0 j D jAj;

195

if i 0 D i;

j D1

D 0;

if i 0 ¤ i:

Thus, A adj.A/ D jAjI. That .adj A/A D jAjI can be established via a similar argument.

Q.E.D.

Corollary 13.5.4. If A is an n n nonsingular matrix, then

or equivalently

adj.A/ D jAjA1

(5.6)

A1 D .1=jAj/ adj.A/:

(5.7)

13.6 Vandermonde Matrices Consider the matrix

0

1 1 x1 x12 : : : x1n1 B1 x2 x 2 : : : x n1 C 2 2 B C ; V D B: : :: :: C @ :: :: : : A 1 xn xn2 : : : xnn1

where x1 ; x2 ; : : : ; xn are arbitrary scalars. A matrix of this general form is called a Vandermonde matrix, after Alexandre Theophile Vandermonde (1735–1796). Vandermonde matrices are encountered in statistics in fitting (by least squares) a polynomial (in an explanatory variable x) to the observed values of a “dependent” variable y. Let us find the determinant of V . Observe that 1n1 A ; (6.1) VT D 1 0 0

where

and

1 xn 0 : : : B0 1 xn B B0 0 1 B T D B: :: : B: : B @0 0 0 0 0 0 0

x1 xn x2 xn :: :

x1 .x1 xn / x2 .x2 xn / :: :

1 0 0 C C C C C C C 1 xn A 0 1

0 0

::: ::: :: :

x1n2 .x1 xn / x2n2 .x2 xn / :: :

1

B C B C ADB C: @ A n2 xn1 xn xn1 .xn1 xn / : : : xn1 .xn1 xn /

196

13 Determinants

Note that the effect of postmultiplying V by T is to add, to the j th column of V , a scalar multiple of the preceding column (j D 2; : : : ; n), thereby creating a matrix whose last row is .1; 0; 0; : : : ; 0/. Observe also that A is expressible as A D DW ;

(6.2)

where D D diag.x1 xn ; x2 xn ; : : : ; xn1 xn / and 0 1 1 x1 x12 : : : x1n2 B1 x 2 x22 : : : x2n2 C B C : W D B: : :: :: C :: @ :: : : A 2 n2 : : : xn1 1 xn1 xn1

Thus, A is expressible as a product of a diagonal matrix and of the .n 1/ .n 1/ submatrix of V obtained by deleting the last row and column of V . Note that this submatrix (i.e., the matrix W ) is an .n 1/ .n 1/ Vandermonde matrix. Making use of results (6.1) and (6.2) and of various properties of determinants (described in Theorems 13.2.11 and 13.3.4 and Corollaries 13.3.2, 13.2.5, and 13.1.2), we find that 1 A jV j D jV T j D n1 D .1/n1 jAj 1 0 D .1/n1 jDjjW j D jDjjW j D .xn x1 /.xn x2 / .xn xn1 /jW j:

(6.3)

Formula (6.3) serves to relate the determinant of an n n Vandermonde matrix to that of an .n 1/ .n 1/ Vandermonde matrix, and its repeated application allows us to evaluate the determinant of any Vandermonde matrix. Clearly, when n D 2, jV j D x2 x1 I when n D 3, jV j D .x3 x1 /.x3 x2 /.x2 x1 /I and, in general, Y .xi xj / jV j D i;j .j n for some integer s (1 s m), then at least one column of C1fj1 ;:::;jm g is null. Therefore, in light of Corollary 13.2.3, we have that X .1/1CCmCi1 CCim jAfi1 ;:::;im g jjIN fi1 ;:::;im g; Bj; (8.5) jCj D fi1 ;:::;im g

N fi1 ;:::;im g

where I is the n .n m/ matrix obtained from In by striking out the i1 , : : : ; im th columns (and where fi1 ; : : : ; im g is an m-dimensional subset of the first n positive integers). Now, applying result (8.2), we find that X .1/k1 CCkm C.nmC1/CC.n1/Cn jIN fi1 ;:::;im g; Bj D fk1 ;:::;km g

m jBfk1 ;:::;km g jjIN ki11;:::;i j; ;:::;km

where fk1 ; : : : ; km g is an m-dimensional integers

n subset of the first n positivei1 ;:::;i such subsets] and where IN k1 ;:::;kmm is 1; : : : ; n [and the summation is over all m the .n m/ .n m/ submatrix of In obtained by striking out the k1 , : : : ; km th rows and i1 , : : : ; im th columns. Moreover, unless fk1 ; : : : ; km g D fi1 ; : : : ; im g, at m is null, so that least one row of IN ki11;:::;i ;:::;km m jIN ki11;:::;i j D jInm j; ;:::;km

D 0;

if fk1 ; : : : ; km g D fi1 ; : : : ; im g; otherwise:

Therefore, jIN fi1 ;:::;im g; Bj D .1/i1 CCim C.nmC1/CC.n1/Cn jBfi1 ;:::;im g jjInm j D .1/i1 CCim C.nmC1/CC.n1/Cn jBfi1 ;:::;im g j.1/nm D .1/i1 CCim C.nm/CC.n1/Cn jBfi1 ;:::;im g j: Finally, substituting from result (8.6) in expression (8.5), we obtain X .1/1CCmC2.i1 CCim /C.nm/CC.n1/Cn jCj D fi1 ;:::;im g

D

X

jAfi1 ;:::;im g jjBfi1 ;:::;im g j

.1/2.i1 CCim /Cn.mC1/ jAi1 ;:::;im g jjBfi1 ;:::;im g j;

fi1 ;:::;im g

(8.6)

Exercises

205

which, together with result (8.4), implies that X .1/2.i1 CCim / jAfi1 ;:::;im g jjBfi1 ;:::;im g j jABj D fi1 ;:::;im g

D

X

jAfi1 ;:::;im g jjBfi1 ;:::;im g j:

fi1 ;:::;im g

Q.E.D. Formula (8.3) is sometimes called the Binet-Cauchy formula. In the special case where m D n, it reduces to the formula jABj D jAjjBj, given by Theorem 13.3.4. Note (in light of Lemma 4.4.3 and Corollary 4.4.5) that the rank of the product AB of an m n matrix A and an n m matrix B is at most n, implying (in light of Theorem 13.3.7) that jABj D 0 if m > n.

Exercises Section 13.1 1. Let

0 B B ADB @

a11 a21 a31 a41

a12 a22 a32 a42

a13 a23 a33 a43

a14 a24 a34 a44

1 C C C: A

(a) Write out all of the pairs that can be formed from the four boxed elements of A. (b) Indicate which of the pairs from Part (a) are positive and which are negative. (c) Use formula (1.1) to compute the number of pairs from Part (a) that are negative, and check that the result of this computation is consistent with your answer to Part (b). Section 13.2 2. Consider the n n matrix 0

xC x ::: B x x C B ADB : :: @ :: : x

x

1 x x C C C: A xC

Use Theorem 13.2.10 to show that jAj D n1 .nx C /:

206

13 Determinants

(Hint. Add the last n 1 columns of A to the first column, and then subtract the first row of the resultant matrix from each of the last n 1 rows.) Section 13.3 3. Let A represent an n n nonsingular matrix. Show that if the elements of A and A1 are all integers, then jAj D ˙1. 4. Let T represent an m m matrix, U an m n matrix, V an n m matrix, and W an n n matrix. Show that if T is nonsingular, then ˇ ˇ ˇ ˇ ˇV W ˇ ˇ U T ˇ mn 1 ˇ ˇDˇ ˇ ˇT U ˇ ˇW V ˇ D .1/ jT jjW V T U j: Section 13.5 5. Compute the determinant of the 4 4 matrix 0 1 0 4 0 5 B1 0 1 2C C ADB @0 3 0 2A 0 0 6 0 in each of the following two ways: (a) by finding and summing the nonzero terms in expression (1.2) or (1.6); (b) by repeated expansion in terms of cofactors [use formula (5.1) or (5.2) to expand jAj in terms of the determinants of 3 3 matrices, to expand the determinants of the 3 3 matrices in terms of the determinants of 2 2 matrices, and finally to expand the determinants of the 2 2 matrices in terms of the determinants of 1 1 matrices]. 6. Let A D faij g represent an n n matrix. Verify that if A is symmetric, then the matrix of cofactors (of A) is also symmetric. 7. Let A represent an n n matrix. (a) Show that if A is singular, then adj.A/ is singular. (b) Show that detŒadj.A/ D Œdet.A/n1 . 8. Use formula (5.7) to verify formula (8.1.2) for the inverse of a 22 nonsingular matrix. 9. Let

0

1 2 0 1 3 1A : A D @1 0 4 5

Exercises

207

(a) Compute the cofactor of each element of A. (b) Compute jAj by expanding jAj in terms of the cofactors of the elements of the second row of A, and then check your answer by expanding jAj in terms of the cofactors of the elements of the second column of A. (c) Use formula (5.7) to compute A1 . 10. Let A D faij g represent an n n matrix (where n 2), and let ˛ij represent the cofactor of aij . (a) Show [by, for instance, making use of the result of Part (b) of Exercise 11.3] that if rank.A/ D n 1, then there exists a scalar c such that adj.A/ D cxy 0 , where x D fxj g and y D fyi g are any nonnull ndimensional column vectors such that Ax D 0 and A0 y D 0. Show also that c is nonzero and is expressible as c D ˛ij =.yi xj / for any i and j such that yi ¤ 0 and xj ¤ 0. (b) Show that if rank.A/ n 2, then adj.A/ D 0. 11. Let A represent an n n nonsingular matrix and b an n 1 vector. Show that the solution to the linear system Ax D b (in x) is the n 1 vector whose j th component is jAj j=jAj; where Aj is a matrix formed from A by substituting b for the j th column of A (j D 1; : : : ; n). [This result is called Cramer’s rule, after Gabriel Cramer (1704–1752).] 12. Let c represent a scalar, let x and y represent n 1 vectors, and let A represent an n n matrix. (a) Show that

ˇ ˇ ˇA y ˇ 0 ˇ 0 ˇ ˇ x c ˇ D cjAj x adj.A/y:

(E.1)

(b) Show that, in the special case where A is nonsingular, result (E.1) can be reexpressed as ˇ ˇ ˇA y ˇ 0 1 ˇ 0 ˇ ˇ x c ˇ D jAj.c x A y/; in agreement with result (3.13). Section 13.6 13. Let Vk represent the .n 1/ .n 1/ submatrix of the n n Vandermonde matrix

208

13 Determinants

0

1 1 x1 x12 : : : x1n1 B1 x2 x 2 : : : x n1 C 2 2 B C V D B: : :: :: C @ :: :: : : A 2 1 xn xn : : : xnn1 (where x1 ; x2 ; : : : ; xn are arbitrary scalars) obtained by striking out the kth row and the nth (last) column (of V ). Show that Y jV j D jVk j.1/nk .xk xi /: i¤k

Section 13.8 14. Show that, for n n matrices A and B, adj.AB/ D adj.B/ adj.A/: [Hint. Use the Binet-Cauchy formula to establish that the ij th elements of adj.AB/ and adj.B/adj.A/ are equal.]

14 Linear, Bilinear, and Quadratic Forms

This chapter is devoted to certain functions (of an n-dimensional vector) that are defined in Section 14.1 and that are known as linear, bilinear, and quadratic forms. Those bilinear forms that have certain properties (known as symmetry and positive definiteness) are of interest (at least in part) because they qualify as inner products for Rn (as is discussed in Section 14.10). Quadratic (and bilinear) forms are of considerable relevance in statistics; the “sums of squares (or products)” in an analysis of variance (or covariance) table are quadratic (or bilinear) forms.

14.1 Some Terminology and Basic Results Let a D .a1 ; : : : ; an /0 represent an arbitrary n-dimensional column vector, and considerPthe function that assigns to each vector x D .x1 ; : : : ; xn /0 in Rn the value a0 x D i ai xi . A function of x (with domain Rn ) that is expressible as a0 x for some vector a is called a linear form (in x). Thus, a linear form is a homogeneous polynomial of degree one. Note that the linear form a0 x can be reexpressed as x0 a. It is customary to refer to a1 ; : : : ; an as the coefficients of the linear form and to the row or column vector a0 or a as the coefficient vector. A function, say f .x/, of x (with domain Rn / is said to be linear if (1) f .x1 C x2 / D f .x1 / C f .x2 / for all vectors x1 and x2 in Rn and (2) f .cx/ D cf .x/ for every scalar c and every vector x in Rn . Or, equivalently, f .x/ is linear if f .c1 x1 C c2 x2 C C ck xk / D c1 f .x1 / C c2 f .x2 / C C ck f .xk / for all scalars c1 ; c2 ; : : : ; ck and all (n-dimensional) vectors x1 ; x2 ; : : : ; xk (where k is an integer greater than Por equal to 2). A linear form a0 x D i ai xi is a linear function of x D .x1 ; : : : ; xn /0 , as is easily verified. The converse is also true; that is, a linear function f .x/ of x is expressible as a linear form (in x). To see this, take ai D f .ei /, where ei is the i th column of In (i D 1; : : : ; n), and observe that

210


f .x/ D f .x1 e1 C x2 e2 C C xn en / D x1 f .e1 / C x2 f .e2 / C C xn f .en / D a1 x1 C a2 x2 C C an xn D a0 x: If the coefficient vectors a D .a1 ; : : : ; an /0 and b D .b1 ; : : : ; bn /0 of two linear forms a0 x and b0 x (in x) are equal, then clearly the two linear forms are identically equal (i.e., equal for all x). Conversely, if the two linear forms a0 x and b0 x are identically equal, then a D b, as is evident from Lemma 2.3.2. Now, let A D faij g represent an arbitrary m n matrix, and consider the func; : : : ; xm /0 and y D .y1 ; : : : ; yn /0 tion that assigns to each pair of vectors x D .x1P m n 0 (in R and R , respectively) the value x Ay D i;j aij xi yj . A function of x and y that is expressible in the form x0 Ay is called a bilinear form (in x and y). Thus, a bilinear form is a special kind of second-degree homogeneous polynomial (in the m C n variables x1 ; : : : ; xm ; y1 ; : : : ; yn )—one which, for fixed x, is homogeneous of first degree in y1 ; : : : ; yn and which, for fixed y, is homogeneous of first degree in x1 ; : : : ; xm . It is customary to refer to A as the matrix of the bilinear form x0 Ay. Note that the bilinear form x0 Ay can be reexpressed as y 0 A0 x—a bilinear form in y and x whose matrix is A0 . A function, say f .x; y/, of x and y is said to be bilinear if, for every fixed x, it is a linear function of y and if, for every fixed y, it is a linear function of x. For every fixed y, the bilinear form x0 Ay is a linear form in x—one with coefficient vector Ay. (And, for every fixed x, x0 Ay is a linear form in y.) Since (as noted earlier) a linear form is a linear function, a bilinear form is a bilinear function. Conversely, any bilinear function is expressible as a bilinear form. To see this, let f .x; y/ represent an arbitrary bilinear function in x D .x1 ; : : : ; xm /0 and y D .y1 ; : : : ; yn /0 , and take A to be the m n matrix with ij th element aij D f .ei ; uj /, where ei denotes the i th column of Im , and uj the j th column of In . Then, X X x i ei ; y D xi f .ei ; y/ f .x; y/ D f i

i

D

X

X x i f ei ; yj uj

i

D

X i

xi

X j

j

yj f .ei ; uj / D

X

aij xi yj D x0 Ay:

i;j

If the matrices A D faij g and B D fbij g of two bilinear forms x0 Ay and x0 By (in x and y) are equal, then clearly the two bilinear forms are identically equal (i.e., equal for all x and y). Conversely, if the two bilinear forms are identically equal, then A D B, as is evident upon observing that, for x D ei (the i th column of Im ) and y D uj (the j th column of In ), aij D x0 Ay D x0 By D bij (i D 1; : : : ; m; j D 1; : : : ; n). A bilinear form x0 Ay in vectors x D .x1 ; : : : ; xn /0 and y D .y1 ; : : : ; yn /0 (of the same dimensions) is said to be symmetric if x0 Ay D y 0 Ax for all x and y.

14.1 Some Terminology and Basic Results

211

Since y 0 Ax D .y 0 Ax/0 D x0 A0 y, the bilinear form x0 Ay is symmetric if and only if x0 Ay D x0 A0 y for all x and y and hence if and only if A D A0 . Thus, the bilinear form is symmetric if and only if the matrix of the bilinear form is symmetric. Finally, let A D faij g represent an arbitrary n n matrix, and consider the function that assigns to each vector x D .x1 ; : : : ; xn /0 (in Rn ) the value X X X x0 Ax D aij xi xj D ai i xi2 C aij xi xj : i;j

i

i;j ¤i

A function of x that is expressible in the form x0 Ax is called a quadratic form (in x). Thus, a quadratic form is a homogeneous polynomial of degree two. It is customary to refer to A as the matrix of the quadratic form x0 Ax. Clearly, the quadratic form x0 Ax can be obtained from the bilinear form x0 Ay (in x and y) simply by setting y D x. Let B D fbij g represent a second n n matrix. Under what circumstances are the two quadratic forms x0 Ax and x0 Bx identically equal? Clearly, a sufficient condition for x0 Ax x0 Bx is that A D B. However, except in the special case n D 1, A D B is not a necessary condition. For the purpose of establishing a necessary condition, suppose that x0 Ax 0 x Bx. Setting x equal to the i th column of In , we find that ai i D x0 Ax D x0 Bx D bi i .i D 1; : : : ; n/:

(1.1)

That is, the diagonal elements of A are the same as those of B. Consider now the off-diagonal elements of A and B. Setting x equal to the n-dimensional column vector whose i th and j th elements equal one and whose remaining elements equal zero, we find that ai i C aij C aj i C ajj D x0 Ax D x0 Bx D bi i C bij C bj i C bjj .j ¤ i D 1; : : : ; n/:

(1.2)

Together, results (1.1) and (1.2) imply that ai i D bi i ; aij C aj i D bij C bj i .j ¤ i D 1; : : : ; n/; or equivalently that A C A0 D B C B 0 :

(1.3) 0

0

Thus, condition (1.3) is a necessary condition for x Ax and x Bx to be identically equal. It is also a sufficient condition. To see this, observe that (since a 1 1 matrix is symmetric) condition (1.3) implies that x0 Ax D .1=2/Œx0 Ax C .x0 Ax/0 D .1=2/.x0 Ax C x0 A0 x/ D .1=2/x0 .A C A0 /x D .1=2/x0 .B C B0 /x D .1=2/Œx0 Bx C .x0 Bx/0 D x0 Bx: In summary, we have the following lemma.

212


Lemma 14.1.1. Let A D faij g and B D fbij g represent arbitrary n n matrices. The two quadratic forms x0 Ax and x0 Bx (in x) are identically equal if and only if, for j ¤ i D 1; : : : ; n, ai i D bi i and aij C aj i D bij C bj i or, equivalently, if and only if A C A0 D B C B0 . Note that Lemma 14.1.1 implies in particular that x0 A0 x D x0 Ax for all x. When B is symmetric, the condition A C A0 D B C B0 is equivalent to the condition B D .1=2/.A C A0 /, and, when both A and B are symmetric, the condition A C A0 D B C B0 is equivalent to the condition A D B. Thus, we have the following two corollaries of Lemma 14.1.1. Corollary 14.1.2. Corresponding to any quadratic form x0 Ax, there is a unique symmetric matrix B such that x0 Bx D x0 Ax for all x, namely, the matrix B D .1=2/.A C A0 /. Corollary 14.1.3. For any pair of n n symmetric matrices A and B, the two quadratic forms x0 Ax and x0 Bx (in x) are identically equal (i.e., x0 Ax D x0 Bx for all x) if and only if A D B. As a special case of Corollary 14.1.3 (that where B D 0), we have the following, additional corollary. Corollary 14.1.4. Let A represent an n n symmetric matrix. If x0 Ax D 0 for every (n 1) vector x, then A D 0.

14.2 Nonnegative Definite Quadratic Forms and Matrices a. Definitions Let x0 Ax represent an arbitrary quadratic form [in an n-dimensional vector x D .x1 ; : : : ; xn /0 ]. The quadratic form x0 Ax is said to be nonnegative definite if x0 Ax 0 for every x in Rn . Note that x0 Ax D 0 for at least one value of x, namely, x D 0. If x0 Ax is nonnegative definite and if, in addition, the null vector 0 is the only value of x for which x0 Ax D 0, then x0 Ax is said to be positive definite. That is, x0 Ax is positive definite if x0 Ax > 0 for every x except x D 0. A quadratic form that is nonegative definite, but not positive definite, is called positive semidefinite. Thus, x0 Ax is positive semidefinite if x0 Ax 0 for every x 2 Rn and x0 Ax D 0 for some nonnull x. Consider, for example, the two quadratic forms x0 Ix D x12 C x22 C C xn2 and x0 J x D x0 110 x D .10 x/0 10 x D .x1 C x2 C C xn /2 . Clearly, x0 Ix and x0 J x are both nonnegative definite. Moreover, x0 Ix > 0 for all nonnull x, while (assuming that n 2) x0 J x D 0 for the nonnull vector x D .1 n; 1; 1; : : : ; 1/0 . Thus, x0 Ix is positive definite, while x0 J x is positive semidefinite. The quadratic form x0 Ax is said to be nonpositive definite, negative definite, or negative semidefinite if x0 Ax is nonnegative definite, positive definite, or positive semidefinite, respectively. Thus, x0 Ax is nonpositive definite if x0 Ax 0 for every

14.2 Nonnegative Definite Quadratic Forms and Matrices

213

x in Rn , is negative definite if x0 Ax < 0 for every nonnull x in Rn , and is negative semidefinite if x0 Ax 0 for every x in Rn and x0 Ax D 0 for some nonnull x. A quadratic form that is neither nonnegative definite nor nonpositive definite is said to be indefinite. Thus, x0 Ax is indefinite if x0 Ax < 0 for some x and x0 Ax > 0 for some (other) x. The terms nonnegative definite, positive definite, positive semidefinite, nonpositive definite, negative definite, negative semidefinite, and indefinite are applied to matrices as well as to quadratic forms. An n n matrix A is said to be nonnegative definite, positive definite, or positive semidefinite if the quadratic form x0 Ax (in x) is nonnegative definite, positive definite, or positive semidefinite, respectively. Similarly, A is said to be nonpositive definite, negative definite, or negative semidefinite if the quadratic form x0 Ax is nonpositive definite, negative definite, or negative semidefinite, respectively, or equivalently if A is nonnegative definite, positive definite, or positive semidefinite. Further, A is said to be indefinite if the quadratic form x0 Ax is indefinite or equivalently if A is neither nonnegative definite nor nonpositive definite. (Symmetric) nonnegative definite matrices are encountered with considerable frequency in linear statistical models and in other areas of statistics. In particular, the variance-covariance matrix of any random vector is inherently nonnegative definite (and symmetric). Our usage of the terms nonnegative definite, positive definite, positive semidefinite, nonpositive definite, negative definite, negative semidefinite, and indefinite differs somewhat from that employed in various other presentations. In particular, we apply these terms to both symmetric and nonsymmetric matrices, whereas in many other presentations their application to matrices is confined to symmetric matrices. Moreover, our usage of the terms positive semidefinite and negative semidefinite is not completely standard. In some presentations, these terms are used in the same way that nonnegative definite and nonpositive definite are being used here. It is instructive to consider the following lemma, which characterizes the concepts of nonnegative definiteness, positive definiteness, and positive semidefiniteness as applied to diagonal matrices and which is easy to verify. Lemma 14.2.1. Let D D fdi g represent an nn diagonal matrix. Then, (1) D is nonnegative definite if and only if d1 ; : : : ; dn are nonnegative; (2) D is positive definite if and only if d1 ; : : : ; dn are positive; and (3) D is positive semidefinite if and only if di 0 for i D 1; : : : ; n with equality holding for one or more values of i . Suppose that an n n symmetric matrix A is both nonnegative definite and nonpositive definite. Then, for every n 1 vector x, x0 Ax 0 and x0 Ax 0 (or equivalently x0 Ax 0). We conclude that x0 Ax D 0 for every x and hence (in light of Corollary 14.1.4) that A D 0. Thus, we have the following lemma. Lemma 14.2.2. The only n n symmetric matrix that is both nonnegative definite and nonpositive definite is the n n null matrix.

214


b. Some basic properties of nonnegative definite matrices We now consider some of the more elementary properties of nonnegative definite matrices. We do not explicitly consider the properties of nonpositive definite matrices. However, since A is nonpositive definite, negative definite, or negative semidefinite if and only if A is nonnegative definite, positive definite, or positive semidefinite, respectively, the properties of nonpositive definite matrices can be readily deduced from those of nonnegative definite matrices. The following two lemmas give some basic results on scalar multiples and sums of nonnegative definite matrices. Lemma 14.2.3. Let k > 0 represent a (positive) scalar, and A an n n matrix. If A is positive definite, then kA is also positive definite; if A is positive semidefinite, then kA is also positive semidefinite. Proof. Consider the quadratic forms x0 Ax and x0 .kA/x (in x). Clearly, 0 x .kA/x D kx0 Ax. Thus, if x0 Ax is positive definite, then x0 .kA/x is positive definite; similarly, if x0 Ax is positive semidefinite, then x0 .kA/x is positive semidefinite. Or, equivalently, if A is positive definite, then kA is positive definite; and if A is positive semidefinite, then kA is positive semidefinite. Q.E.D. Lemma 14.2.4. Let A and B represent n n matrices. If A and B are both nonnegative definite, then A C B is nonnegative definite. Moreover, if either A or B is positive definite and the other is nonnegative definite (i.e., positive definite or positive semidefinite), then A C B is positive definite. Proof. Suppose that one of the two matrices, say A, is positive definite and that the other (B) is nonnegative definite. Then, for every nonnull vector x in Rn , x0 Ax > 0 and x0 Bx 0, and hence x0 .A C B/x D x0 Ax C x0 Bx > 0: Thus, A C B is positive definite. A similar argument shows that if A and B are both nonnegative definite, then A C B is nonnegative definite. Q.E.D. The repeated application of Lemma 14.2.4 leads to the following generalization. Corollary 14.2.5. Let A1 ; : : : ; Ak represent nn matrices. If A1 ; : : : ; Ak are all nonnegative definite, then their sum A1 C C Ak is also nonnegative definite. Moreover, if one or more of the matrices A1 ; : : : ; Ak are positive definite and the others are nonnegative definite, then A1 C C Ak is positive definite. As an essentially immediate consequence of Lemma 14.1.1, we have the following lemma and corollary. Lemma 14.2.6. Let A represent an n n matrix, and take B to be any n n matrix such that B C B0 D A C A0 . Then, A is positive definite if and only if B is positive definite, is positive semidefinite if and only if B is positive semidefinite, and is nonnegative definite if and only if B is nonnegative definite.


215

Corollary 14.2.7. An n n matrix A is positive definite if and only if A0 is positive definite and if and only if .1=2/.A C A0 / is positive definite; is positive semidefinite if and only if A0 is positive semidefinite and if and only if .1=2/.A C A0 / is positive semidefinite; and is nonnegative definite if and only if A0 is nonnegative definite and if and only if .1=2/.A CA0 / is nonnegative definite. If A is a symmetric matrix, then A0 D A and .1=2/.A C A0 / D A. Thus, it is only in the case of a nonsymmetric positive definite, positive semidefinite, or nonnegative definite matrix that Corollary 14.2.7 is meaningful. A basic property of positive definite matrices is described in the following lemma. Lemma 14.2.8. Any positive definite matrix is nonsingular. Proof. Let A represent an n n positive definite matrix. For purposes of establishing a contradiction, suppose that A is singular or, equivalently, that rank.A/ < n. Then, the columns of A are linearly dependent, and hence there exists a nonnull vector x such that Ax D 0. We find that x0 Ax D x0 .Ax / D x0 0 D 0; which (since A is positive definite) establishes the desired contradication. Q.E.D. Lemma 14.2.8 implies that singular nonnegative definite matrices are positive semidefinite. However, the converse is not necessarily true. That is, there exist nonsingular positive semidefinite matrices as well as singular positive semidefinite matrices. Additional basic properties of nonnegative definite matrices are described in the following theorem and corollaries. Theorem 14.2.9. Let A represent an n n matrix, and P an n m matrix. (1) If A is nonnegative definite, then P 0 AP is nonnegative definite. (2) If A is nonnegative definite and rank.P / < m, then P 0 AP is positive semidefinite. (3) If A is positive definite and rank.P / D m, then P 0 AP is positive definite. Proof. Suppose that A is nonnegative definite (either positive definite or positive semidefinite). Then, y 0 Ay 0 for every y in Rn and in particular for every y that is expressible in the form P x. Thus, for every m-dimensional vector x, (2.1) x0 .P 0 AP /x D .P x/0 AP x 0; which establishes that P 0 AP is nonnegative definite, thereby completing the proof of Part (1). If rank.P / < m, then rank.P 0 AP / rank.P / < m; which (in light of Lemma 14.2.8) establishes that P 0 AP is not positive definite and hence (since P 0 AP is nonnegative definite) that P 0 AP is positive semidefinite, thereby completing the proof of Part (2). If A is positive definite, then equality is attained in inequality (2.1) only when P x D 0. Moreover, if rank.P / D m, then (in light of Lemma 11.3.1) P x D 0

216


implies x D 0. Thus, if A is positive definite and rank.P / D m, then equality is attained in inequality (2.1) only when x D 0, implying (since P 0 AP is nonnegative Q.E.D. definite) that P 0 AP is positive definite. Corollary 14.2.10. Let A represent an nn matrix and P an nn nonsingular matrix. (1) If A is positive definite, then P 0 AP is positive definite. (2) If A is positive semidefinite, then P 0 AP is positive semidefinite. Proof. (1) That P 0 AP is positive definite if A is positive definite is a direct consequence of Part (3) of Theorem 14.2.9. (2) Suppose that A is positive semidefinite. Then, according to Part (1) of Theorem 14.2.9, P 0 AP is nonnegative definite. Further, there exists a nonnull vector y such that y 0 Ay D 0. Let x D P 1 y. Then, y D P x, and we find that x ¤ 0 (since, otherwise, we would have that y D 0) and that x0 .P 0 AP /x D .P x/0 AP x D y 0 Ay D 0: We conclude that P 0 AP is positive semidefinite.

Q.E.D.

Corollary 14.2.11. (1) A positive definite matrix is invertible, and its inverse is positive definite. (2) If a positive semidefinite matrix is nonsingular, then it is invertible and its inverse is positive semidefinite. Proof. (1) Let A represent a positive definite matrix. Then, according to Lemma 14.2.8, A is nonsingular and hence (according to Theorem 8.1.4) invertible. Since .A1 /0 D .A1 /0 AA1 , it follows from Part (1) of Corollary 14.2.10 [together with result (8.2.5)] that .A1 /0 is positive definite and hence (in light of Corollary 14.2.7) that A1 is positive definite. (2) By employing similar reasoning, it can be shown that Part (2) of Corollary 14.2.11 follows from Part (2) of Corollary 14.2.10. Q.E.D. Corollary 14.2.12. Any principal submatrix of a positive definite matrix is positive definite; any principal submatrix of a positive semidefinite matrix is nonnegative definite. Proof. Let A represent an n n matrix, and consider the principal submatrix of A obtained by striking out all of its rows and columns except its i1 , i2 , : : : ; im th rows and columns (where i1 < i2 < < im ). This submatrix is expressible as P 0 AP , where P is the n m matrix whose columns are the i1 , i2 , : : : ; im th columns of In . Since rank.P / D m, it follows from Part (3) of Theorem 14.2.9 that P 0 AP is positive definite if A is positive definite. Further, it follows from Part (1) of Theorem 14.2.9 that P 0 AP is nonnegative definite if A is nonnegative definite (and, in particular, if A is positive semidefinite). Q.E.D. Corollary 14.2.13. The diagonal elements of a positive definite matrix are positive; the diagonal elements of a positive semidefinite matrix are nonnegative. Proof. This corollary follows immediately from Corollary 14.2.12 upon observing (1) that the i th diagonal element of a (square) matrix A is the element of a 1 1 principal submatrix (that obtained by striking out all of the rows and columns of A except the i th row and column) and (2) that the element of a 1 1 positive


217

definite matrix is positive and the element of a 1 1 nonnegative definite matrix is nonnegative. Q.E.D. Corollary 14.2.14. Let P represent an arbitrary n m matrix. The m m matrix P 0 P is nonnegative definite. If rank.P / D m, P 0 P is positive definite; otherwise [if rank.P / < m], P 0 P is positive semidefinite. Proof. This corollary follows from Theorem 14.2.9 upon observing that P 0 P D P 0 IP and that (as demonstrated in Subsection a) I is positive definite. Q.E.D. Corollary 14.2.15. Let P represent an nn nonsingular matrix and D D fdi g an n n diagonal matrix. Then, (1) P 0 DP is nonnegative definite if and only if d1 ; : : : ; dn are nonnegative; (2) P 0 DP is positive definite if and only if d1 ; : : : ; dn are positive; and (3) P 0 DP is positive semidefinite if and only if di 0 for i D 1; : : : ; n with equality holding for one or more values of i . Proof. Let A D P 0 DP . Then, clearly, D D .P 1 /0 AP 1 . Thus, it follows from Part (1) of Theorem 14.2.9 that A is nonnegative definite if and only if D is nonnegative definite; and it follows, respectively, from Parts (1) and (2) of Corollary 14.2.10 that A is positive definite if and only if D is positive definite and that A is positive semidefinite if and only if D is positive semidefinite. In light of Lemma 14.2.1, the proof is complete. Q.E.D. Corollary 14.2.16. Let A represent an n n symmetric matrix and D D fdi g an n n diagonal matrix such that P 0 AP D D for some n n nonsingular matrix P . Then, (1) A is nonnegative definite if and only if d1 ; : : : ; dn are nonnegative; (2) A is positive definite if and only if d1 ; : : : ; dn are positive; and (3) A is positive semidefinite if and only if di 0 for i D 1; : : : ; n with equality holding for one or more values of i . Proof. Corollary 14.2.16 follows from Corollary 14.2.15 upon observing that Q.E.D. A D .P 1 /0 DP 1 . Note that Corollaries 14.2.15 and 14.2.16 can be regarded as generalizations of Lemma 14.2.1. c. The nonnegative definiteness of symmetric idempotent matrices Suppose that A is a symmetric idempotent matrix. Then, A D AA D A0 A, and it follows from Corollary 14.2.14 that A is nonnegative definite. Thus, we have the following lemma. Lemma 14.2.17. Every symmetric idempotent matrix is nonnegative definite. Note (in light of Lemmas 14.2.8 and 10.1.1) that the n n identity matrix In is the only n n idempotent matrix (symmetric or otherwise) that is positive definite.

218


14.3 Decomposition of Symmetric and Symmetric Nonnegative Definite Matrices According to Corollary 14.2.14, every matrix A that is expressible in the form A D P 0 P is a (symmetric) nonnegative definite matrix. Is the converse true? That is, is every symmetric nonnegative definite matrix A expressible in the form A D P 0 P ? In Subsection b, it is established that the answer to this question is yes. As a preliminary, consideration is given to a closely related question, which is of some interest in its own right. Is a symmetric matrix A expressible in the form A D P 0 DP , where P is a nonsingular matrix and D a diagonal matrix? In Subsection a, it is established that the answer to this question is also yes. In what follows, it will be convenient to have at our disposal the following lemma. Lemma 14.3.1. Let A represent an n n matrix and D an n n diagonal matrix such that A D P DQ for some n n nonsingular matrices P and Q. Then, rank.A/ equals the number of nonzero diagonal elements in D. Proof. Making use of Corollary 8.3.3, we find that rank.A/ D rank.P DQ/ D rank.DQ/ D rank.D/: Moreover, rank.D/ equals the number of nonzero diagonal elements in D. Q.E.D. a. Decomposition of symmetric matrices Let us begin by establishing the following two lemmas. Lemma 14.3.2. Let A D faij g represent an n n symmetric matrix (where n 2). Define B D fbij g D L0 AL, where L is an n n unit lower triangular a11 a0 1 0 . Partition A as A D . Suppose matrix of the form L D ` In1 a A22 that a11 D 0, but that a ¤ 0 or, equivalently, that a1j ¤ 0 for some j (greater than 1), say j D k. Then, the .n 1/-dimensional vector ` can be chosen so that b11 ¤ 0; this can be done by taking the .k 1/th element of ` to be any nonzero scalar c such that cakk ¤ 2a1k and by taking the other n 2 elements of ` to be zero. Proof. Since a1k ¤ 0, there exists (whether or not akk D 0) a nonzero scalar c such that cakk ¤ 2a1k . Taking the .k 1/th element of ` to be any such nonzero scalar c and the remaining elements of ` to be zero, we find that b11 D a11 C ` 0 a C a0 ` C ` 0 A22 ` D 0 C cak1 C ca1k C c 2 akk D c.cakk C 2a1k / ¤ 0: Q.E.D.

14.3 Decomposition of Symmetric and Symmetric Nonnegative Definite Matrices

219

Lemma 14.3.3. Let A D faij g represent an n n symmetric matrix (where n 2). Define B D fbij g D U 0 AU , where U is an n n unit upper triangular 1 u0 b11 b0 matrix of the form U D . Partition B and A as B D and 0 In1 b B22 a11 a0 AD . Suppose that a11 ¤ 0. Then, the .n 1/-dimensional vector a A22 1 a. u can be chosen so that b D 0; this can be done by taking u D a11 Proof. We find that 1 b D .u; I/A D a11 u C a; 0 1 a, gives b D a a D 0. which, for u D a11 We are now in a position to establish the following theorem.

Q.E.D.

Theorem 14.3.4. Corresponding to any nn symmetric matrix A, there exists a nonsingular matrix Q such that Q0 AQ is a diagonal matrix. Proof. The proof is by mathematical induction. The theorem is clearly true for any 1 1 matrix. Suppose now that it is true for any .n 1/ .n 1/ symmetric matrix, and consider an arbitrary n n symmetric matrix A D faij g. For purposes of establishing the existence of a nonsingular matrix Q such that Q0 AQ is diagonal, a11 a0 and to proceed case-by-case. it is convenient to partition A as A D a A22 Case (1): a D 0. The submatrix A22 is an .n 1/ .n 1/ symmetric matrix. Thus, by supposition, there exists a nonsingular matrix Q such that Q0 A22 Q is a diagonal matrix. Take Q D diag.1; Q /. Then, Q is nonsingular, and Q0 AQ D diag.a11 ; Q0 A22 Q /, which (like Q0 A22 Q ) is a diagonal matrix. Case (2): a ¤ 0 and a11 ¤ 0. According to Lemma 14.3.3, there exists a unit upper triangular matrix U such that U 0 AU D diag.b11 ; B22 / for some scalar b11 and some .n 1/ .n 1/ matrix B22 . Moreover, by supposition, there exists a nonsingular matrix Q such that Q0 B22 Q is a diagonal matrix. Take Q D U diag.1; Q /. Then, Q is nonsingular (since Q is the product of two nonsingular matrices), and Q0 AQ D diag.1; Q0 / diag.b11 ; B22 / diag.1; Q / D diag.b11 ; Q0 B22 Q /; which (like Q0 B22 Q ) is a diagonal matrix. Case (3): a ¤ 0, but a11 D 0. Let B D fbij g D L0 AL, where L is a unit lower triangular matrix chosen so that b11 ¤ 0—the existence of such a choice is guaranteed by Lemma 14.3.2. Then, according to Lemma 14.3.3, there exists a unit upper triangular matrix U such that U 0 BU D diag.c11 ; C22 / for some scalar c11 and some .n 1/ .n 1/ matrix C22 . Moreover, by supposition, there exists a nonsingular matrix Q such that Q0 C22 Q is a diagonal matrix. Take Q D LU diag.1; Q /. Then, Q is nonsingular (since Q is the product of three nonsingular matrices), and

220


Q0 AQ D diag.1; Q0 /U 0 BU diag.1; Q / D diag.1; Q0 / diag.c11 ; C22 / diag.1; Q / D diag.c11 ; Q0 C22 Q /; which (like Q0 C22 Q ) is a diagonal matrix. Q.E.D. 0 Note that if Q is a nonsingular matrix such that Q AQ D D for some diagonal matrix D, then A D .Q1 /0 Q0 AQQ1 D .Q1 /0 DQ1 : Thus, we have the following corollary of Theorem 14.3.4. Corollary 14.3.5. Corresponding to any nn symmetric matrix A, there exist a nonsingular matrix P and a diagonal matrix D such that A D P 0 DP . Corollary 14.3.5 leads to the following result on quadratic forms. Corollary 14.3.6. Let A represent an n n matrix. Then, there exist a non0 singular matrix P and n scalars d1 ; : : : ; dn such that the quadratic form Pn x Ax2 (in an n-dimensional vector x) is expressible as a linear combination iD1 di yi of the squares of the elements y1 ; : : : ; yn of the transformed vector y D P x. Proof. According to Corollary 14.1.2, there is a unique symmetric matrix B such that x0 Ax D x0 Bx for all x, namely, the matrix B D .1=2/.A C A0 /. And, according to Corollary 14.3.5, there exist a nonsingular matrix P and a diagonal matrix D such that B D P 0 DP . Thus, letting d1 ; : : : ; dn represent the diagonal elements of D and y1 ; : : : ; yn the elements of the vector y D P x, we find that X x0 Ax D x0 P 0 DP x D .P x/0 DP x D di yi2 : i

Q.E.D. b. Decomposition of symmetric nonnegative definite matrices Let us now specialize to symmetric nonnegative definite matrices, beginning with the following theorem. Theorem 14.3.7. An n n (nonnull) matrix A is a symmetric nonnegative definite matrix of rank r if and only if there exists an r n matrix P of rank r such that A D P 0 P . Proof. Suppose that A is a symmetric nonnegative definite matrix of rank r. Then, according to Corollary 14.3.5, there exist a nonsingular matrix T and a diagonal matrix D such that A D T 0 DT . Let d1 ; : : : ; dn represent the first, : : : ; nth diagonal elements of D, and t10 , : : : ; 0 tn the first, : : : ; nth rows of T . It follows from Lemma 14.3.1 that r of the diagonal elements of D, say the k1 , : : : ; kr th diagonal elements, are nonzero (and hence, in light of Corollary 14.2.15, are positive). Now, take P to be the r n matrix whose q 0 i th row is dki tki . Then, AD

n X kD1

dk tk tk0 D

r X iD1

dki tki tk0 i D

r q q X dki tki dki tk0 i D P 0 P ; iD1

14.3 Decomposition of Symmetric and Symmetric Nonnegative Definite Matrices

221

and rank.P / D rank.P 0 P / D rank.A/ D r. Conversely, if there exists an r n matrix P of rank r such that A D P 0 P , then rank.A/ D rank.P 0 P / D rank.P / D r; A is symmetric, and (according to Corollary 14.2.14) A is nonnegative definite. Q.E.D. 0 In light of Corollary 14.2.14 (and upon observing tht 0 D 0 0), we have the following corollary of Theorem 14.3.7. Corollary 14.3.8. An n n matrix A is a symmetric nonnegative definite matrix if and only if there exists a matrix P (having n columns) such that A D P 0 P . As a further corollary of Theorem 14.3.7, we have the following result on quadratic forms. Corollary 14.3.9. Let A represent an n n matrix, and let r D rank.A C A0 /. Then, assuming that r > 0, the quadratic form x0 Ax (in an n-dimensional vector x) is nonnegative definite if and only if, for some r n matrix P of rank r, x0 Ax D .P x/0 P x for all x (i.e., x0 Ax is expressible as the sum of squares of the elements of the vector P x). Proof. According to Corollary 14.1.2, there is a unique symmetric matrix B such that x0 Ax D x0 Bx for all x, namely, the matrix B D .1=2/.A C A0 /. Suppose that the quadratic form x0 Ax, or equivalently the quadratic form x0 Bx, is nonnegative definite, in which case B is a nonnegative definite matrix. Then, it follows from Theorem 14.3.7 that there exists an r n matrix P of rank r such that B D P 0 P and hence such that x0 Bx D .P x/0 P x for all x or, equivalently, such that x0 Ax D .P x/0 P x for all x. Conversely, suppose that, for some r n matrix P of rank r, x0 Ax D .P x/0 P x for all x or, equivalently, x0 Ax D x0 P 0 P x for all x. Since (according to Theorem 14.3.7 or, alternatively, according to Corollary 14.2.14) P 0 P is a nonnegative definite matrix, x0 P 0 P x, or equivalently x0 Ax, is a nonnegative definite quadratic form. Q.E.D. Further implications of Theorem 14.3.7 are given in Corollaries 14.3.10 to 14.3.12. Corollary 14.3.10. Let A represent an n n matrix, let r D rank.A/, and take m to be any positive integer greater than or equal to r. If A is symmetric and nonnegative definite, then there exists an m n matrix P such that A D P 0 P . Proof. Suppose that A is symmetric and nonnegative definite. And assume that r > 0. (When r D 0, A D 00 0.) According to Theorem 14.3.7, there exists an r n P1 . Then, clearly, A D P 0 P . Q.E.D. matrix P1 such that A D P10 P1 . Take P D 0 Corollary 14.3.11. For any n m matrix X and any n n symmetric nonnegative definite matrix A, AX D 0 if and only if X0 AX D 0.

222


Proof. According to Corollary 14.3.8, there exists a matrix P such that A D P 0 P and hence such that X0 AX D .P X/0 P X. Thus, if X0 AX D 0, then (in light of Corollary 5.3.2) P X D 0, implying that AX D P 0 .P X/ D 0. Q.E.D. That X0 AX D 0 if AX D 0 is obvious. Corollary 14.3.12. A symmetric nonnegative definite matrix is positive definite if and only if it is nonsingular (or, equivalently, is positive semidefinite if and only if it is singular). Proof. Let A represent an n n symmetric nonnegative definite matrix. If A is positive definite, then we have, as an immediate consequence of Lemma 14.2.8, that A is nonsingular. Suppose now that the symmetric nonnegative definite matrix A is nonsingular, and consider the quadratic form x0 Ax (in x). If x0 Ax D 0, then, according to Corollary 14.3.11, Ax D 0, and consequently x D A1 Ax D 0. Thus, the quadratic form x0 Ax is positive definite and hence the matrix A is positive definite. Q.E.D. As a special case of Theorem 14.3.7 (that where r D n), we have (in light of Corollary 14.3.12) the following corollary. Corollary 14.3.13. An n n matrix A is a symmetric positive definite matrix if and only if there exists a nonsingular matrix P such that A D P 0 P . Corollary 14.3.13 is a variant of Theorem 14.3.7 that is specific to symmetric positive definite matrices. Similarly, the following corollary is a variant of Corollary 14.3.9 that is specific to positive definite quadratic forms. Corollary 14.3.14. A quadratic form x0 Ax (in an n-dimensional vector x) is positive definite if and only if, for some n n nonsingular matrix P , x0 Ax D .P x/0 P x for all x (i.e., x0 Ax is expressible as the sum of squares of the elements of the vector P x). Proof. If x0 Ax is positive definite, then (according to Corollary 14.2.7) .1=2/.A C A0 / is positive definite and hence (in light of Lemma 14.2.8) is of rank n, so that it follows from Corollary 14.3.9 that, for some n n nonsingular matrix P , x0 Ax D .P x/0 P x for all x. Conversely, if for some n n nonsingular matrix P , x0 Ax D .P x/0 P x for all x, then, since (according to Corollary 14.2.14 or Corollary 14.3.13) P 0 P is positive definite, x0 Ax (which equals x0 P 0 P x) is positive definite. Q.E.D.

14.4 Generalized Inverses of Symmetric Nonnegative Definite Matrices A symmetric positive definite matrix is invertible, and its inverse is positive definite and symmetric [Corollary 14.2.11 and result (8.2.4)]. A symmetric positive semidefinite matrix is not invertible (Corollary 14.3.12). What is the nature of a generalized inverse of a symmetric positive semidefinite matrix? In particular, are

14.5 LDU, U0 DU, and Cholesky Decompositions

223

some or all of the generalized inverses of a symmetric positive semidefinite matrix nonnegative definite? Let A represent an n n matrix of rank r, and suppose that A D P DQ;

(4.1)

where P and Q are (n n) nonsingular matrices and D D fdi g is an n n diagonal matrix. According to Lemma 14.3.1, r of the diagonal elements of D, say the i1 , : : : ; ir th diagonal elements, are nonzero and the other n r diagonal elements of D equal zero. Let S D fi1 ; i2 ; : : : ; ir g and denote by SN the set whose elements consist of those n r of the first n positive integers 1; 2; : : : ; n not contained in S . Let D represent an n n matrix of the general form D D diag.d1 ; d2 ; : : : ; dn /; N d is an arbitrary scalar. Note that where, for i 2 S , di D 1=di and, for i 2 S, i D is a generalized inverse of D. Define G D Q1 D P 1 and G D Q1 D P 1 . According to Lemma 9.2.4, G is a generalized inverse of A. In particular, G is a generalized inverse of A. Suppose now that A is symmetric and that decomposition (4.1) is such that P D Q0 , so that decomposition (4.1) is of the form A D Q0 DQ:

(4.2)

Then, in light of result (8.2.3), G D Q1 D .Q1 /0 :

(4.3) di

Thus, G is a symmetric generalized inverse of A. Moreover, if is chosen to N then G is a (symmetric) nonsingular generalized be nonzero for every i in S, inverse of A. Next, suppose that A is nonnegative definite, as well as symmetric—and continue to suppose that P D Q0 and hence that decomposition (4.1) is of the form (4.2). Then, it follows from Corollary 14.2.15 that di > 0 for every i in S . Thus, N then (in light of Corollary if di is chosen to be greater than zero for every i in S, 14.2.15) G is a symmetric positive definite generalized inverse of A. (If di is chosen to be greater than or equal to zero for all i in SN and equal to zero for at least N then G is positive semidefinite; if di is chosen to be less than zero one i in S, for at least one i in SN , then—assuming that A is nonnull—G is indefinite.) Since (according to Corollary 14.3.5) every symmetric matrix has a decomposition of the form (4.2), we have the following lemma. Lemma 14.4.1. Every symmetric nonnegative definite matrix has a symmetric positive definite generalized inverse.

14.5 LDU, U0 DU, and Cholesky Decompositions It was established in Section 14.3 that every symmetric nonnegative definite matrix A has a decomposition of the form A D P 0 P . Stronger versions of this result are

224


possible. In fact, it is shown in the present section that every symmetric nonnegative definite matrix A has a decomposition of the form A D T 0 T , where T is an upper triangular matrix. [And, in a subsequent chapter (Section 21.9), it is shown that A has a (unique) decomposition of the form A D R2 , where R is a symmetric nonnegative definite matrix.] It is convenient to begin by considering some general questions about the decomposition of (not necessarily nonnegative definite) matrices. Is a square matrix A expressible in the form A D LDU , where L is a unit lower triangular matrix, D is a diagonal matrix, and U is a unit upper triangular matrix? And, more specifically, is a symmetric matrix A expressible in the form A D U 0 DU ? Subsequently, the term LDU decomposition is used for any decomposition of the form A D LDU and the term U 0DU decomposition for any decomposition of the form A D U 0 DU . a. Some preliminary results The following lemma relates the LDU decomposition of an n n matrix A to that of the leading .n 1/ .n 1/ principal submatrix of A. Lemma 14.5.1. Let A represent an n n matrix, L an n n unit lower triangular matrix, U an n n unit upper triangular matrix, and D an n n diagonal matrix (where n 2). Partition A, L, U , and D as A a L 0 U u D 0 AD ; L D ; U D ; and D D ; b0 c `0 1 0 1 0 k where A , L , U , and D are of dimensions .n 1/ .n 1/. Then, A D LDU if and only if

and

L D U D A ; L D u D a; U0 D ` D b;

(5.1a) (5.1b) (5.1c)

k D c ` 0 D u:

(5.1d)

Lemma 14.5.1 can be verified simply by equating the four submatrices A , a, b0 , and c to the corresponding submatrices of the matrix product LDU . With regard to condition (5.1a), note that L is a principal submatrix of a unit lower triangular matrix and hence is itself unit lower triangular, and similarly that U is unit upper triangular and D is diagonal. Thus, condition (5.1a) states that L D U is an LDU decomposition of A . The following theorem relates the existence and construction of an LDU or U0 DU decomposition of an n n matrix to the existence and construction of an LDU or U0 DU decomposition of the leading .n 1/ .n 1/ principal submatrix of A.


225

Theorem 14.5.2. Let A represent an nn matrix (where n 2), and partition A as A a ; AD b0 c where A is of dimensions .n 1/ .n 1/. (1) If A has an LDU decomposition, say A D L D U , and if a 2 C.A / and b0 2 R.A /, then there exist vectors ` and u such that U0 D ` D b and L D u D a, and, taking ` and u to be any such vectors and taking k D c ` 0 D u, an LDU decomposition of A is L 0 D 0 U u AD : (5.2) `0 1 0 k 0 1 (10 ) If A is symmetric (in which case A0 D A and b D a), if A has a U0 DU decomposition, say A D U0 D U , and if a 2 C.A /, then there exists a vector u such that U0 D u D a, and, taking u to be any such vector and taking k D c u0 D u, a U0 DU decomposition of A is 0 D 0 U u U u : (5.3) AD 0 1 0 k 0 1 (2) The matrix A has an LDU decomposition only if A has an LDU decomposition, a 2 C.A /, and b0 2 R.A /. (20 ) The matrix A has a U0 DU decomposition only if A is symmetric, A has a U0 DU decomposition, and a 2 C.A /. Proof. (1) Suppose that A has an LDU decomposition, say A D L D U , and that a 2 C.A / and b0 2 R.A /. Then, there exist vectors r and s such that a D A r and b0 D s0 A . Since U0 D .L0 s/ D A0 s D b, and L D .U r/ D A r D a, there exist vectors ` and u such that U0 D ` D b and L D u D a. Moreover, equality (5.2) is an immediate consequence of Lemma 14.5.1. (10 ) If A0 D A , then a 2 C.A / implies a0 2 R.A /. Thus, Part (10 ) can be obtained as a special case of Part (1) by putting b D a and L D U0 and setting ` D u. (2) and (20 ). Suppose that A has an LDU decomposition, say A D LDU . Partition L, U , and D as U u D 0 L 0 ; U D ; and D D ; LD `0 1 0 1 0 k where L , U , and D are of dimensions .n 1/ .n 1/. Then, it follows from Lemma 14.5.1 that A D L D U and hence that A has an LDU decomposition. As a further consequence, we have that a D L D u D A U1 u 2 C.A / and b0 D ` 0 D U D ` 0 L1 A 2 R.A /. In the special case where L D U 0 (i.e., where A D LDU is a U0 DU decomposition), we have that L D U0 and hence that A D U0 D U . Thus, if A has

226


a U0 DU decomposition, then A also has a U0 DU decomposition. Moreover, if A Q.E.D. has a U0 DU decomposition, then clearly A is symmetric. Note that, together, Parts (1) and (2) of Theorem 14.5.2 imply, in particular, that A has an LDU decomposition if and only if A has an LDU decomposition, a 2 C.A /, and b0 2 R.A /. Similarly, Parts (10 ) and (20 ) imply that A has a U0 DU decomposition if and only if A is symmetric, A has a U0 DU decomposition, and a 2 C.A /. The following theorem relates the LDU decomposition of any leading principal submatrix of an n n matrix A to that of A itself. Theorem 14.5.3. Let A represent an n n matrix (where n 2), and let A11 represent the k k leading principal submatrix of A (where 1 k n 1). Suppose that A has an LDU decomposition, say A D LDU , and partition L, U , and D as U11 U12 D1 0 L11 0 ; UD ; and D D ; LD L21 L22 0 U22 0 D2 where the dimensions of L11 , U11 , and D1 are k k. Then, an LDU decomposition of A11 is A11 D L11 D1 U11 . Theorem 14.5.3 can be verified simply by equating A11 to the k k leading principal submatrix of the matrix product LDU . Note that, in the special case of Theorem 14.5.3 where A D LDU is a U0 DU decomposition (i.e., where L D U 0 ), 0 and hence A11 D L11 D1 U11 is a U0 DU decomposition of A11 . L11 D U11 Theorem 14.5.3 implies in particular that if A has an LDU or U0 DU decomposition, then each of its leading principal submatrices has, respectively, an LDU or U0 DU decomposition. b. Existence, recursive construction, and uniqueness of an LDU or U0 DU decomposition It is easy to see that a 11 matrix A has a (unique) LDU (and U0 DU) decomposition, namely, A D .1/A.1/. The following theorem gives results on the existence and recursive construction of an LDU or U0 DU decomposition of a (square) matrix of order two or more. Theorem 14.5.4. Let A D faij g represent an n n matrix (where n 2). Denote by Ai the leading principal submatrix of A of order i (i D 1; : : : ; n). For i D 2; : : : ; n, take ai D .a1i ; : : : ; ai1;i /0 and b0i D .ai1 ; : : : ; ai;i1 /, or equivalently define ai and b0i by Ai1 ai : Ai D b0i ai i (1) Let L1 D .1/, U1 D .1/, and D1 D .a11 /, and suppose that, for i D 2; : : : ; n, ai 2 C.Ai1 / and b0i 2 R.Ai1 /. Then, for i D 2; : : : ; n, there exist a unit


227

lower triangular matrix Li , a unit upper triangular matrix Ui , and a diagonal matrix Di such that Li1 0 Ui1 ui Di1 0 Li D ; (5.4) ; U D ; D D i i ì0 1 0 1 0 di 0 where Ui1 Di1 ì D bi , Li1 Di1 ui D ai , and di D ai i ì0 Di1 ui ; and, taking Li , Ui , and Di to be any such matrices, an LDU decomposition of Ai is Ai D Li Di Ui . In particular, an LDU decomposition of A is A D Ln Dn Un .

(10 ) Let U1 D .1/ and D1 D .a11 /, and suppose that A is symmetric (in which case A0i1 D Ai1 and bi D ai for i D 2; : : : ; n) and that, for i D 2; : : : ; n, ai 2 C.Ai1 /. Then, for i D 2; : : : ; n, there exist a unit upper triangular matrix Ui and a diagonal matrix Di such that Ui1 ui Di1 0 ; (5.5) ; Di D Ui D 0 1 0 di 0 where Ui1 Di1 ui D ai and di D ai i u0i Di1 ui ; and taking Ui and Di to be any such matrices, a U0 DU decomposition of Ai is Ai D Ui0 Di Ui . In particular, a U0 DU decomposition of A is A D Un0 Dn Un .

(2) The matrix A has an LDU decomposition only if, for i D 2; : : : ; n, ai 2 C.Ai1 / and b0i 2 R.Ai1 /. (20 ) The matrix A has a U0 DU decomposition only if A is symmetric and, for i D 2; : : : ; n, ai 2 C.Ai1 /. Proof. (1) For purposes of proving Part (1), suppose that, for i D 2; : : : ; n, ai 2 C.Ai1 / and b0i 2 R.Ai1 /. The proof is by mathematical induction. It follows from Theorem 14.5.2 that, for i D 2, there exist a unit lower triangular matrix Li , a unit upper triangular matrix Ui , and a diagonal matrix Di of the form (5.4) and that A2 D L2 D2 U2 for any such matrices L2 , U2 , and D2 . Suppose now that, for i D k 1 (where 3 k n), there exist a unit lower triangular matrix Li , a unit upper triangular matrix Ui , and a diagonal matrix Di of the form (5.4) and, taking Lk1 , Uk1 , and Dk1 to be any such matrices, that Ak1 D Lk1 Dk1 Uk1 . The proof is complete upon observing that, for i D k, there exist (as a consequence of Theorem 14.5.2) a unit lower triangular matrix Li , a unit upper triangular matrix Ui , and a diagonal matrix Di of the form (5.4) and that Ak D Lk Dk Uk for any such matrices Lk , Uk , and Dk . (10 ) A proof of Part (10 ), analogous to that of Part (1), can be constructed by making use of Part (10 ) of Theorem 14.5.2. (2) Suppose that A has an LDU decomposition. Then, it follows from Theorem 14.5.3 that the leading principal submatrices A2 ; A3 ; : : : ; An1 of orders two through n 1 (as well as the leading principal submatrix An D A of order n) have LDU decompositions. Based on Part (2) of Theorem 14.5.2, we conclude that, for i D 2; : : : ; n, ai 2 C.Ai1 / and b0i 2 R.Ai1 /. (20 ) A proof of Part (20 ), analogous to that of Part (2), can be constructed by making use of Theorem 14.5.3 and Part (20 ) of Theorem 14.5.2. Q.E.D.

228


Note that, together, Parts (1) and (2) of Theorem 14.5.4 imply in particular that A has an LDU decomposition if and only if, for i D 2; : : : ; n, ai 2 C.Ai1 / and b0i 2 R.Ai1 /. Similarly, Parts (10 ) and (20 ) imply that A has a U0 DU decomposition if and only if A is symmetric and, for i D 2; : : : ; n, ai 2 C.Ai1 /. For purposes of illustration, consider a 2 2 matrix A D faij g and suppose that a11 D 0. Then, A has an LDU decomposition if and only if a21 D a12 D 0, that is, if and only if A is of the form 0 0 AD : 0 a22 When A is of this form, an LDU decomposition of A is 1 u 1 0 0 0 ; AD 0 1 ` 1 0 a22 where ` and u are arbitrary scalars. To what extent is the LDU decomposition of an n n matrix unique? This question is addressed in the following theorem. Theorem 14.5.5. Let A represent an n n matrix of rank r that has an LDU decomposition. Let L D fìj g represent an n n unit lower triangular matrix, U D fuij g an n n unit upper triangular matrix, and D D fdi g an n n diagonal matrix such that A D LDU ; that is, such that A D LDU is an LDU decomposition of A. (1) The diagonal matrix D is unique. (2) Suppose that the i1 , : : : ; ir th diagonal elements of D (where i1 < < ir ) are nonzero (and the remaining diagonal elements of D are zero). Then, for i 2 fi1 ; : : : ; ir g and j > i , uij and `j i are unique, and, for i … fi1 ; : : : ; ir g and j > i , uij and `j i are completely arbitrary. That is, of the elements of U above the diagonal, those in the i1 , : : : ; ir th rows are unique and those in the remaining rows are completely arbitrary; and, similarly, of the elements of L below the diagonal, those in the i1 , : : : ; ir th columns are unique and those in the remaining columns are completely arbitrary. Proof. Let L D fìj g represent a unit lower triangular matrix, U D fuij g a unit upper triangular matrix, and D D fdi g a diagonal matrix such that A D L D U , so that A D L D U , like A D LDU , is an LDU decomposition of A. Then, L1 LD D L1 .LDU /U 1 D L1 .L D U /U 1 D D U U 1 : (5.6) Since (according to Corollary 8.5.9) L1 is unit lower triangular and U 1 is unit upper triangular, it follows from Lemma 1.3.1 that the diagonal elements of L1 LD are d1 ; : : : ; dn , respectively, and that the diagonal elements of D U U 1 are d1 ; : : : ; dn , respectively, implying—in light of equality (5.6)—that di D di (i D 1; : : : ; n) or equivalently that D D D, which establishes Part (1) of Theorem 14.5.5. For purposes of proving Part (2), take A11 to be the r r submatrix obtained by striking out all of the rows and columns of A except the i1 , : : : ; ir th rows and


229

columns. Let D1 D diag.di1 ; : : : ; dir /. Denote by L1 and L1 the r n submatrices obtained by striking out all of the rows of L and L , respectively, except the i1 , : : : ; ir th rows, and by L11 and L11 the submatrices obtained by striking out all of the rows and columns of L and L , respectively, except the i1 , : : : ; ir th rows and columns. Similarly, denote by U1 and U1 the n r submatrices obtained by striking out all of the columns of U and U , respectively, except the i1 , : : : ; the r r submatrices obtained by striking out ir th columns, and by U11 and U11 all of the rows and columns of U and U , respectively, except the i1 , : : : ; ir th rows and columns. Then, A11 D L1 DU1 D L11 D1 U11 and (since D D D) , implying that A11 D L1 D U1 D L1 DU1 D L11 D1 U11 L11 D1 U11 D L11 D1 U11 :

(5.7)

The matrices L11 and L11 are principal submatrices of unit lower triangular matrices and hence are themselves unit lower triangular. Similarly, U11 and U11 1 are unit upper triangular. Premultiplying both sides of equality (5.7) by L11 and 1 1 D1 , we find that postmultiplying both sides by U11 1 1 L1 11 L11 D D1 U11 U11 D1 :

(5.8)

1 According to Corollary 8.5.9, L1 11 is unit lower triangular and U11 is unit upper 1 triangular. Observe that (according to Lemma 1.3.1) L11 L11 is unit lower tri 1 1 U11 D1 is upper triangular, it follows from equality (5.8) angular. Since D1 U11 1 that L11 L11 D I, or, equivalently, that

L11 D L11 ; and—in light of equality (5.7) and the nonsingularity of L11 and D1 —that U11 D U11 :

Now, take A1 to be the n r submatrix obtained by striking out all of the columns of A except the i1 , : : : ; ir th columns, and take A2 to be the r n submatrix obtained by striking out all of the rows of A except the i1 , : : : ; ir th rows. Denote by L2 and L2 the n r submatrices obtained by striking out all of the columns of L and L , respectively, except the i1 , : : : ; ir th columns. Similarly, denote by U2 and U2 the r n submatrices obtained by striking out all of the rows of U and U , respectively, except the i1 , : : : ; ir th rows. Then, A1 D LDU1 D L2 D1 U11 and D U11 ) A1 D L D U1 D L DU1 D L2 D1 U11 D (since D D D and U11 L2 D1 U11 , implying that L2 D1 U11 D L2 D1 U11 , and hence (since D1 and U11 are nonsingular) that (5.9) L2 D L2 : Similarly, A2 D L1 DU D L11 D1 U2 and A2 D L1 D U D L1 DU D L11 D1 U2 D L11 D1 U2 , implying that L11 D1 U2 D L11 D1 U2 , and hence that U2 D U2 :

(5.10)

230


The kj th elements of U2 and U2 are uik j and uik j , respectively, and the j kth elements of L2 and L2 are `j ik and `jik , respectively (k D 1; : : : ; r; j D 1; : : : ; n). Thus, it follows from equalities (5.9) and (5.10) that, for i 2 fi1 ; : : : ; ir g and for j D 1; : : : ; n (and in particular for j D i C1; : : : ; n) that uij D uij and `ji D `j i . To complete the proof of Part (2) of Theorem 14.5.5, observe that A D LDU D L2 D1 U2 and hence that A D LDU even if those elements of L (below the diagonal) that are not elements of L2 and/or those elements of U (above the Q.E.D. diagonal) that are not elements of U2 are changed arbitrarily. Under what circumstances does an n n matrix A have a unique LDU decomposition? This question is addressed in the following two corollaries. Corollary 14.5.6. If an n n matrix A has an LDU decomposition and if the leading principal submatrix (of A) of order n 1 is nonsingular, then the LDU decomposition of A is unique. Proof. Let A11 represent the .n 1/ .n 1/ leading principal submatrix of A. Suppose that A has an LDU decomposition, say A D LDU , and that A11 is nonsingular. Partition L, U , and D as L11 0 U11 u D1 0 LD ; U D ; and D D `0 1 0 1 0 d [where L11 , U11 , and D1 are of dimensions .n 1/ .n 1/]. According to Lemma 14.5.1, an LDU decomposition of A11 is A11 D L11 D1 U11 . Thus (since A11 is nonsingular), it follows from Lemma 14.3.1 that all n 1 diagonal elements of D1 are nonzero or, equivalently, that the first n 1 diagonal elements of D are nonzero. Based on Theorem 14.5.5, we conclude that the LDU decomposition of A is unique. Q.E.D. Corollary 14.5.7. Let A represent an n n matrix (where n 2), and, for i D 1; 2; : : : ; n1, let Ai represent the leading principal submatrix (of A) of order i . If A1 ; A2 ; : : : ; An1 are nonsingular, then A has a unique LDU decomposition. Proof. Let aij represent the ij th element of A (i; j D 1; : : : ; n); and, for i D 2; : : : ; n, define ai D .a1i ; : : : ; ai1;i /0 and b0i D .ai1 ; : : : ; ai;i1 /. Suppose that A1 ; A2 ; : : : ; An1 are nonsingular. Then, for i D 2; : : : ; n, ai 2 C.Ai1 / and b0i 2 R.Ai1 /, and it follows from Part (1) of Theorem 14.5.4 that A has an LDU decomposition. That this decomposition is unique is (since An1 is nonsingular) an immediate consequence of Corollary 14.5.6. Q.E.D. For a symmetric matrix, the uniqueness of an LDU decomposition has the following implication. Lemma 14.5.8. If a symmetric matrix A has a unique LDU decomposition, say A D LDU , then L D U 0 , that is, the unique LDU decomposition is a U0 DU decomposition. Proof. Suppose that the symmetric matrix A has the unique LDU decomposition A D LDU . Then, A D A0 D U 0 DL0 , implying (since U 0 is unit lower triangular and L0 is unit upper triangular) that A D U 0 DL0 is an LDU decomposi-


231

tion of A and hence—in light of the uniqueness of the LDU decomposition—that Q.E.D. L D U 0 (or equivalently U D L0 ). Note that if a (square) matrix A has an LDU decomposition, say A D LDU , then, letting L D LD and U D DU , it can also be decomposed as A D L U , which is the product of a lower triangular matrix and a unit upper triangular matrix, or as A D LU , which is the product of a unit lower triangular matrix and an upper triangular matrix. The term LU decomposition is sometimes used in referring to any decomposition of a (square) matrix A of the general form A D LU , where L is a lower triangular matrix and U an upper triangular matrix—see, e.g., Stewart (1973). In the special case where L is unit lower triangular, Stewart refers to such a decomposition as a Doolittle decomposition, and in the special case where U is unit upper triangular, he refers to such a decomposition as a Crout decomposition. Golub and Van Loan (1989) restrict their use of the term LU decomposition to the special case where L is unit lower triangular, that is, to what Stewart calls a Doolittle decomposition. c. Decomposition of positive definite matrices Let us now consider the implications of the results of Subsection b as applied to positive definite matrices. Does a positive definite matrix necessarily have an LDU decomposition, and is the LDU decomposition of a positive definite matrix unique? These questions are answered (in the affirmative) by the following theorem, which also provides some information about the nature of the LDU decomposition of a positive definite matrix. Theorem 14.5.9. An n n positive definite matrix A has a unique LDU decomposition, say A D LDU , and the diagonal elements of the diagonal matrix D are positive. Proof. According to Corollary 14.2.12, any principal submatrix of a positive definite matrix is positive definite and hence (according to Lemma 14.2.8) is nonsingular. Thus, it follows from Corollary 14.5.7 that A has a unique LDU decomposition, say A D LDU . It remains to show that the diagonal elements, say d1 ; : : : ; dn , of the diagonal matrix D are positive. For this purpose, consider the matrix B D DU .L1 /0 . Since (in light of Corollary 8.5.9) .L1 /0 (like U ) is unit upper triangular, it follows from Lemma 1.3.1 that the diagonal elements of B are the same as the diagonal elements d1 ; : : : ; dn of D. Moreover, B D L1 .LDU /.L1 /0 D L1 A.L1 /0 ; implying (in light of Corollary 14.2.10) that B is positive definite. We conclude (on the basis of Corollary 14.2.13) that d1 ; : : : ; dn are positive. Q.E.D. In the special case of a symmetric positive definite matrix, the conclusions of Theorem 14.5.9 can (as a consequence of Lemma 14.5.8) be made more specific, as described in the following corollary.

232


Corollary 14.5.10. An nn symmetric positive definite matrix A has a unique U0 DU decomposition, say A D U 0 DU , and the diagonal elements of the diagonal matrix D are positive. Note that, as an immediate consequence of Corollary 14.2.15, we have the following converse to Corollary 14.5.10: If an n n matrix A has a U0 DU decomposition, say A D U 0 DU , and if the diagonal elements of the diagonal matrix D are positive, then A is positive definite (and symmetric). An alternative decomposition of a symmetric positive definite matrix is described in the following theorem. Theorem 14.5.11. For any n n symmetric positive definite matrix A, there exists a unique upper triangular matrix T with positive diagonal elements such that (5.11) A D T 0T : Moreover, taking U to be the unique unit upper triangular matrix and D D fdi g to be the unique diagonal matrix such that A D U 0 DU , T D D 1=2 U ; p p p where D 1=2 D diag. d1 ; d2 ; : : : ; dn /. Proof. Note that Corollary 14.5.10 guarantees the existence and uniqueness of the unit upper triangular matrix U and the diagonal matrix D and also guarantees 1=2 that the diagonal elements of D are positive. Note further p that p D Upis upper 1=2 triangular, that the diagonal elements of D U are d1 ; d2 ; : : : ; dn , and that A D .D 1=2 U /0 D 1=2 U . Thus, there exists an upper triangular matrix T with positive diagonal elements such that A D T 0 T . It remains to show that there is only one such upper triangular matrix with positive diagonal elements. Suppose that T D ftij g is an n n upper triangular matrix with positive 2 2 2 ; t22 ; : : : ; tnn / and diagonal elements such that A D T 0 T . Define D D diag.t11 U D Œdiag.t11 ; t22 ; : : : ; tnn /1 T D diag.1=t11 ; 1=t22 ; : : : ; 1=tnn /T : Then, U is unit upper triangular, and A D U0 D U , so that A D U0 D U is a 14.5.10 that D D D U0 DU decomposition of A. Thus, it follows from Corollary p or, equivalently (since ti i is positive), that ti i D di (i D 1; : : : ; n), and that U D U . Consequently, T D D 1=2 U D D 1=2 U . We conclude that the only upper triangular matrix T with positive diagonal elements such that A D T 0 T is Q.E.D. T D D 1=2 U . Decomposition (5.11) of the symmetric positive definite matrix A is known as the Cholesky decomposition. d. Decomposition of symmetric nonnegative definite matrices Let us now extend the results of Corollary 14.5.10 and Theorem 14.5.11 (which apply to symmetric positive definite matrices) to symmetric nonnegative definite matrices that may not be positive definite. The following theorem provides the basis for these extensions.


233

Theorem 14.5.12. Corresponding to any nn symmetric nonnegative definite matrix A, there exists a unit upper triangular matrix U such that U 0 AU is a diagonal matrix. For purposes of proving this theorem, it is helpful to establish the following lemma. Lemma 14.5.13. Let A D faij g represent an nn nonnegative definite matrix. If ai i D 0, then, for j D 1; : : : ; n, aij D aj i ; that is, if the i th diagonal element of A equals 0, then .ai1 ; : : : ; ai n /, which is the i th row of A, equals .a1i ; : : : ; ani /, which is 1 times the transpose of the i th column of A (i D 1; : : : ; n). Proof (of Lemma 14.5.13). Suppose that ai i D 0, and take x D fxk g to be an n-dimensional column vector such that xi < ajj , xj D aij C aj i , and xk D 0 for k other than k D i and k D j (where j ¤ i ). Then, x0 Ax D ai i xi2 C .aij C aj i /xi xj C ajj xj2 D .aij C aj i /2 .xi C ajj / 0;

(5.12)

with equality holding only if aij C aj i D 0 or, equivalently, only if aij D aj i . Moreover, since A is nonnegative definite, x0 Ax 0, which—together with inequality (5.12)—implies that x0 Ax D 0. We conclude that aij D aj i . Q.E.D. If the nonnegative definite matrix A in Lemma 14.5.13 is symmetric, then aij D aj i , 2aij D 0 , aij D 0. Thus, we have the following corollary. Corollary 14.5.14. Let A D faij g represent an n n symmetric nonnegative definite matrix. If ai i D 0, then, for j D 1; : : : ; n, aj i D aij D 0; that is, if the i th diagonal element of A equals zero, then the i th column .a1i ; : : : ; ani /0 of A and the i th row .ai1 ; : : : ; ai n / of A are null. Moreover, if all n diagonal elements a11 ; a22 ; : : : ; ann of A equal zero, then A D 0. Proof (of Theorem 14.5.12). The proof is by mathematical induction and is similar to the proof of Theorem 14.3.4. The theorem is clearly true for any 1 1 matrix. Suppose now that it is true for any .n 1/ .n 1/ symmetric nonnegative definite matrix, and consider an arbitrary n n symmetric nonnegative definite matrix A D faij g. For purposes of establishing the existence of a unit upper triangular matrix U such that U 0 AU is diagonal, it is convenient to consider separately the case a11 D 0 and the case a11 ¤ 0. Case (1): a11 D 0. It follows from Corollary 14.5.14 that the first row and column of A are null or, equivalently, that A D diag.0; A22 / for some .n 1/ .n 1/ matrix A22 . Moreover, since A22 is a principal submatrix of A, A22 is symmetric and nonnegative definite, and hence, by supposition, there exists a unit upper triangular matrix U such that U0 A22 U is a diagonal matrix. Take U D diag.1; U /. Then, U is unit upper triangular, and U 0 AU D diag.0; U0 A22 U /, which (like U0 A22 U ) is a diagonal matrix. Case (2): a11 ¤ 0. According to Lemma 14.3.3, there exists a unit upper triangular matrix U1 such that U10 AU1 D diag.b11 ; B22 / for some scalar b11 and some .n 1/ .n 1/ matrix B22 . Moreover, U10 AU1 is (in light of Theorem

234


14.2.9) symmetric and nonnegative definite, and, since B22 is a principal submatrix of U10 AU1 , B22 is symmetric and nonnegative definite. Thus, by supposition, there exists a unit upper triangular matrix U2 such that U20 B22 U2 is a diagonal matrix. Take U D U1 diag.1; U2 /. Then, U is a unit upper triangular matrix [since diag.1; U2 / is unit upper triangular and the product of two unit upper triangular matrices is unit upper triangular], and U 0 AU D diag.1; U20 / diag.b11 ; B22 / diag.1; U2 / D diag.b11 ; U20 B22 U2 /; Q.E.D. which (like U20 B22 U2 ) is a diagonal matrix. Corollary 14.5.10 indicates that, in the special case of a symmetric positive definite matrix A, the nonsingular matrix P in Corollary 14.3.5 can be chosen to be a unit upper triangular matrix, in which case the decomposition A D P 0 DP is a U0 DU decomposition. The following corollary (of Theorem 14.5.12) indicates that P can be chosen to be unit upper triangular even if A is only (symmetric) positive semidefinite. Corollary 14.5.15. An n n symmetric nonnegative definite matrix A has a U0 DU decomposition, and, for any such decomposition A D U 0 DU , the diagonal elements of the diagonal matrix D are nonnegative. Proof. In light of Corollary 14.2.15, it suffices to show that A has a U0 DU decomposition. According to Theorem 14.5.12, there exists a unit upper triangular matrix T such that T 0 AT D D for some diagonal matrix D. Take U D T 1 . Then, A D U 0 DU , and (according to Corollary 8.5.9) U is unit upper triangular. Q.E.D. Thus, A D U 0 DU is a U0 DU decomposition of A. Note that, as an immediate consequence of Corollary 14.2.15, we have the following converse to Corollary 14.5.15: If an n n matrix A has a U0 DU decomposition, say A D U 0 DU , and if the diagonal elements of the diagonal matrix D are nonnegative, then A is nonnegative definite (and symmetric). Theorem 14.5.11 establishes that any symmetric positive definite matrix A has a unique decomposition of the form (5.11) and relates this decomposition to the U0 DU decomposition of A. The following theorem extends these results to any symmetric nonnegative definite matrix. Theorem 14.5.16. Let A represent an n n symmetric nonnegative definite matrix, and let r D rank.A/. Then, there exists a unique upper triangular matrix T with r positive diagonal elements and with n r null rows such that A D T 0T :

(5.13)

Moreover, taking U to be a unit upper triangular matrix and D D fdi g to be the unique diagonal matrix such that A D U 0 DU , (5.14) T D D 1=2 U ; p p p where D 1=2 D diag. d1 ; d2 ; : : : ; dn /. Proof. Note that Corollary 14.5.15 and Theorem 14.5.5 guarantee the existence of the unit upper triangular matrix U and the existence and uniqueness


235

of the diagonal matrix D and also guarantee that the diagonal elements of D are nonnegative. Note further (on the basis of Lemma 14.3.1) that r of the diagonal elements of D, say the i1 , : : : ; ir th diagonal elements (where i1 < < ir ), are nonzero and hence positive, and that the remaining n r diagonal elements of D 1=2 of D 1=2 U equal pD U is upper triangular, the diagonal elements pzero.pMoreover, 1=2 are d1 , d2 , : : : ; dn (r of which are positive), n r rows of D U are null (those rows other than the i1 , : : : ; ir th rows), and A D .D 1=2 U /0 D 1=2 U . Thus, there exists an upper triangular matrix T with r positive diagonal elements and with n r null rows such that A D T 0 T . It remains to show that there is only one such upper triangular matrix with r positive diagonal elements and with n r null rows. Suppose that T D ftij g is an n n upper triangular matrix with r positive diagonal elements—say the k1 , : : : ; kr th diagonal elements (where k1 < < kr )—and with n r null rows such that A D T 0 T . Take D to be the n n diagonal matrix whose k1 , : : : ; kr th diagonal elements are tk21 k1 ; : : : ; tk2r kr , respectively, and whose other n r diagonal elements equal zero. Further, let tk0 1 ; : : : ; tk0 r represent the k1 , : : : ; kr th rows of T , and take U to be any unit upper triangular matrix whose k1 , : : : ; kr th rows are .1=tk1 k1 /tk0 1 ; : : : ; .1=tkr kr /tk0 r , respectively—clearly, such a unit upper triangular matrix exists. Then, A D tk1tk0 1 C C tkrtk0 r D U0 D U ; so that A D U0 D U is a U0 DU decomposition of A. Thus, it follows from Theorem 14.5.5 thatp D D D, in which case k1 D i1 , : : : ; kr D ir and ti1 i1 D p di1 , : : : ; tir ir D dir , and that the i1 , : : : ; ir th rows of U are respectively equal to the i1 , : : : ; ir th rows of U . Consequently, T D D 1=2 U D D 1=2 U : We conclude that the only upper triangular matrix T with r positive diagonal elements and with n r null rows such that A D T 0 T is T D D 1=2 U . Q.E.D. In the special case where the symmetric nonnegative definite matrix A is positive definite, decomposition (5.13) simplifies to decomposition (5.11). And, as in that special case, the decomposition is called the Cholesky decomposition. e. Recursive formulas for LDU and Cholesky decompositions Let A D faij g represent an n n matrix. Consider the problem of finding a unit upper triangular matrix U D fuij g, a unit lower triangular matrix L D f`j i g, and a diagonal matrix D D fdi g such that A D LDU (when they exist). The formulas given in Part (1) of Theorem 14.5.4 can be used to construct L, D, and U in n steps. At each step, an additional column of U , an additional row of L, and an additional diagonal element of D are obtained by solving linear systems with triangular coefficient matrices. By making use of the results of Section 11.8 (on the solution of linear systems with nonsingular triangular or block-triangular

236


coefficient matrices), the formulas given in Part (1) of Theorem 14.5.4 can be reexpressed as follows: d1 D a11 ; d1 u1j D a1j ; di uij D aij

i1 X

ìk dk ukj .i D 2; 3; : : : ; j 1/;

kD1

d1 `j1 D aj1 ; di `j i D aj i

i1 X

uki dk `j k .i D 2; 3; : : : ; j 1/;

kD1

dj D ajj

jX 1

ukj dk `j k

(5.15)

kD1

(j D 2; 3; : : : ; n). The first step in the n-step procedure for constructing U , L, and D consists of setting d1 D a11 . The j th step (2 j n) consists of successively determining uij and `j i for i D 1; 2; : : : ; j 1, and of then determining [from formula (5.15)] dj . If di D 0, then uij and `j i can be chosen arbitrarily; for example, choose uij D `j i D 0. If di ¤ 0, then, depending on whether i D 1 or i > 1, take uij D aij =di and `j i D aj i =di or

P Pj 1 u d ` =di ; uij D aij i1 kD1 ìk dk ukj =di and `j i D aj i kD1 ki k j k respectively. There is an alternative n-step procedure for constructing U , L, and D in which U is formed row-by-row, rather than column-by-column, and L is formed columnby-column, rather than row-by-row. The first step consists of setting d1 D a11 and, depending on whether d1 D 0 or d1 ¤ 0, choosing u12 ; : : : ; u1n and `21 ; : : : ; `n1 arbitrarily or (for j D 2; : : : ; n) setting u1j D a1j =d1 and `j1 D aj1 =d1 . The i th step (2 i n1) consists of determining [from formula (5.15)] di and, depending on whether di D 0 or di ¤ 0, choosing ui;iC1 ; : : : ; ui n and ìC1;i ; : : : ; `ni arbitrarily or (for j D i C 1; : : : ; n) setting

P Pi1 uij D aij i1 kD1 ìk dk ukj =di and `j i D aj i kD1 uki dk `j k =di : The final (nth) step consists of setting dn D ann

n1 X

uk n dk `nk :

kD1

Note that (with either of these two n-step procedures), once dj has been determined, ajj is no longer needed (at least not for purposes of constructing an LDU decomposition). Similarly, once uij has been determined, aij is no longer


237

required, and, once `j i has been determined, aj i is no longer required. Thus, in implementing either n-step procedure on a computer, storage can be saved if, as dj , uij , and `j i are determined, they are placed in the locations previously occupied by ajj , aij , and aj i , respectively. So far, it has been presumed that A has an LDU decomposition. Suppose that there is uncertainty about the existence of an LDU decomposition. The nonexistence of an LDU decomposition can (at least in principle) be determined during the course of the n-step procedures by incorporating certain tests. If d1 D 0, then— prior to assigning values to u1j and `j1 —we should test whether a1j D 0 and aj1 D 0; similarly, if di D 0 (2 i n 1), then—prior to assigning values to uij and `j i —we should test whether aij

i1 X

ìk dk ukj D 0

and

aj i

kD1

i1 X

uki dk `j k D 0

kD1

or, equivalently, whether aij D

i1 X

ìk dk ukj

and

aj i D

kD1

i1 X

uki dk `j k :

kD1

If the result of any test is negative, we terminate the n-step procedure and conclude (on the basis of Theorem 14.5.4) that A does not have an LDU decomposition. If the result of every test is positive, then the n-step procedure is carried to completion and produces an LDU decomposition. Suppose now that A is symmetric, and consider the problem of determining whether A has a U0 DU decomposition and of constructing a U0 DU decomposition. This problem can be “solved” by employing a simplified version of the procedure for checking for the existence of an LDU decomposition and for constructing an LDU decomposition. The simplification comes from setting `j i D uij , for j D 2; : : : ; n and i D 1; : : : ; j 1, and more specifically from the resulting reduction in computational and storage requirements. Finally, suppose that A is a symmetric nonnegative definite matrix, and consider the problem of finding the Cholesky decomposition of A. That is, consider the problem of finding the unique upper triangular matrix T D ftij g with r positive diagonal elements and n r null rows [where r D rank.A/] such that A D T 0 T . One approach is to find the U0 DU decomposition of A and to then determine T from formula (5.14). However, there is a more direct method, which is sometimes called the square root method. The square root method is based on the following formulas, which can be obtained from the formulas of Part (10 ) of Theorem 14.5.4 by applying the results of Section 11.8 and making use of relationship (5.14): p t11 D a11 ; (5.16) t1j D a1j =t11 ; D 0;

if t11 > 0; if t11 D 0;

(5.17a) (5.17b)

238


P tij D aij i1 kD1 tki tkj =ti i ; D 0; (i D 2; 3; : : : ; j 1),

if ti i > 0; if ti i D 0

P 1 2 1=2 tkj tjj D ajj jkD1

(5.18a) (5.18b) (5.19)

(j D 2; 3; : : : ; n). In the square root method, T is constructed row-by-row or column-by-column in n steps, one row or column per step. For example, the first step of the row-by-row version consists of determining [from formulas (5.16) and (5.17)] t11 , t12 , : : : ; t1n ; the i th step (2 i n 1) consists of determining [from formulas (5.19) and (5.18)] ti i , ti;iC1 , : : : ; ti n ; and the final (nth) step consists of determining [from formula (5.19)] tnn . If there is uncertainty about whether A is nonnegative definite, then, in applying the square root method to A, various modifications should be incorporated. We should (prior to determining t11 ) test whether a11 0, and if a11 D 0, we should (prior to setting t1j D 0) test whether a1j D 0. Similarly, for 2 i P 2 n 1, we should (prior to determining ti i ) test whether ai i i1 kD1 tki 0, Pi1 2 and if ai i kD1 tki D 0, we should (prior to setting tij D 0) test whether P aij i1 t t D 0. Finally, we should (prior to determining tnn ) test whether PkD1 ki2 kj ann n1 kD1 tk n 0. If the result of any test is negative, it follows from Theorems 14.5.4 and 14.5.16 that either A does not have a U0 DU decomposition or, for a U0 DU decomposition (of A), say A D U 0 DU , one or more of the diagonal elements of the diagonal matrix D are negative. Thus, if the result of any test is negative, we terminate the square root procedure and conclude (on the basis of Corollary 14.5.15) that A is not nonnegative definite. If the result of every test is positive, then A is nonnegative definite, and the procedure is carried to completion and produces the Cholesky decomposition. Suppose, for example, that 0 1 4 2 2 1 1A : AD@ 2 2 1 10 Let us apply the modified square p root method (the row-by-row version). Observing that a11 > 0, we find that t11 D 4 D 2, t12 D 2=2 D 1, and t13 D 2=2 D 1. 2 D 0, we find that t22 D 0, and, after Further, upon observing that a22 t12 t t D 0, we set t23 p D 0. Finally, upon observing that observing that a 23 12 13 P 2 D 9, we find that t33 D 9 D 3. We conclude that A is a33 2kD1 tk3 nonnegative definite—since t22 D 0, it is positive semidefinite—and that the Cholesky decomposition of A is 1 0 10 0 2 1 1 2 1 1 0 0A : 0 0A @0 A D @0 0 0 3 0 0 3

14.6 Skew-Symmetric Matrices

239

Note that if a23 D a32 had equaled any number other than 1, it would have been the case that a23 t12 t13 ¤ 0, and A would not have been nonnegative definite. Or, if the value of a22 had been smaller than one, we would have had that 2 < 0, and A would not have been nonnegative definite. a22 t12 f. Use of an LDU, U0 DU, or Cholesky decomposition in obtaining a generalized inverse Suppose that A is a (square) matrix that has an LDU decomposition, say A D LDU . Then, it follows from the discussion in Section 14.4 that the matrix G D U 1 D L1

(5.20)

is a generalized inverse of A and that one choice for D is the diagonal matrix obtained by replacing the nonzero diagonal elements of the (diagonal) matrix D with their reciprocals. Thus, one way to obtain a generalized inverse of A is to obtain the LDU decomposition of A and to then apply formula (5.20). In this regard, note that the recursive procedure described in Section 8.5d can be used to invert U and L. Note also that, in the special case where L D U 0 , formula (5.20) simplifies to G D U 1 D .U 1 /0 :

(5.21)

Suppose now that A is an n n symmetric nonnegative definite matrix of rank r. Then, we can obtain a generalized inverse of A in terms of its U0 DU decomposition [using formula (5.21)]. We can also obtain a generalized inverse of A directly in terms of its Cholesky decomposition. To see this, let A D T 0 T represent the Cholesky decomposition of A, so that, by definition, T is an upper triangular matrix with r positive diagonal elements, say its i1 , i2 , : : : ; ir th diagonal elements (where i1 < i2 < < ir ), and with n r null rows. Take T1 to be the r n matrix obtained by striking out the null rows of T . Then, clearly, A D T10 T1 , and rank.T1 / D rank.A/ D r. It follows from Lemma 8.1.1 that T1 has a right inverse, say R. And, since ARR0 A D T10 T1 R.T1 R/0 T1 D T10 T1 D A, RR0 is a generalized inverse of A. A right inverse of T1 can be obtained by making use of the procedure described in Section 8.5d for inverting a nonsingular triangular matrix. To see this, let T11 represent the r r submatrix obtained by striking out all of the columns of T1 except the i1 , i2 , : : : ; ir th columns. Note that T11 is a principal submatrix of T and hence (like T ) is upper triangular and (since the diagonal elements of T11 are nonzero) that T11 is nonsingular. Take R to be the n r matrix whose i1 , i2 , : : : ; 1 and whose other n r rows ir th rows are the first, second, : : : ; rth rows of T11 1 are null. Then, clearly, T1 R D T11 T11 D I, so that R is a right inverse of T1 .

14.6 Skew-Symmetric Matrices An nn matrix A D faij g is said to be skew-symmetric if A0 D A or, equivalently (since ai i D ai i , 2ai i D 0 , ai i D 0), if ai i D 0 for i D 1; : : : ; n and

240


aj i D aij for j ¤ i D 1; : : : ; n. For example, the 2 2 matrix

0 1 is 1 0

skew-symmeric. A principal submatrix of a skew-symmetric matrix is skew-symmetric, as is easily verified. Other basic properties of skew-symmetric matrices are described in the following two lemmas. Lemma 14.6.1. The only n n matrix that is both symmetric and skewsymmetric is the n n null matrix. Proof. If an n n matrix A is both symmetric and skew-symmetric, then Q.E.D. A D A0 D A, implying that 2A D 0 and hence that A D 0. Lemma 14.6.2. Let A represent an n n matrix and P an n m matrix. If A is skew-symmetric, then P 0 AP is skew-symmetric. Proof. If A is skew-symmetric, then .P 0 AP /0 D P 0 A0 P D P 0 .A/P D P 0 AP : Q.E.D. The following lemma, which is an immediate consequence of Lemma 14.1.1, characterizes the skew-symmetry of a (square) matrix A in terms of the quadratic form whose matrix is A. Lemma 14.6.3. An n n matrix A is skew-symmetric if and only if, for every n-dimensional column vector x, x0 Ax D 0. There is a relationship between skew-symmetry and nonnegative definiteness, which is described in the following lemma. Lemma 14.6.4. An n n matrix A D faij g is skew-symmetric if and only if it is nonnegative definite and its diagonal elements a11 ; a22 ; : : : ; ann equal zero. Proof. Suppose that A is skew-symmetric. Then, by definition, ai i D 0 (i D 1; 2; : : : ; n). Moreover, it follows from Lemma 14.6.3 that A is nonnegative definite. Conversely, suppose that A is nonnegative definite and that its diagonal elements equal zero. Then, it follows from Lemma 14.5.13 that aj i D aij for i; j D 1; 2; : : : ; n. Thus, A is skew-symmetric. Q.E.D.

14.7 Trace of a Nonnegative Definite Matrix a. Basic results The following result is an immediate consequence of Corollary 14.2.13. Theorem 14.7.1. For any positive definite matrix A, tr.A/ > 0. The following theorem extends the result of Theorem 14.7.1 to nonnegative definite matrices that are not positive definite, that is, to positive semidefinite matrices.

14.7 Trace of a Nonnegative Definite Matrix

241

Theorem 14.7.2. Let A D faij g represent an n n nonnegative definite matrix. Then, tr.A/ 0, with equality holding if and only if the diagonal elements a11 ; a22 ; : : : ; ann of A equal zero or, equivalently, if and only if A is skew-symmetric. Proof. According to Corollary 14.2.13, ai i 0 (i D 1; 2; : : : ; n). Thus, tr.A/ D a11 C a22 C C ann 0; with tr.A/ D 0 if and only if a11 ; a22 ; : : : ; ann equal zero. That the nonnegative definite matrix A being skew-symmetric is equivalent to the diagonal elements of A being zero is an immediate consequence of Lemma 14.6.4. Q.E.D. In the special case where A is a symmetric nonnegative definite matrix, Theorem 14.7.2 can (in light of Lemma 14.6.1) be restated as follows. Corollary 14.7.3. Let A D faij g represent an n n symmetric nonnegative definite matrix. Then, tr.A/ 0, with equality holding if and only if the diagonal elements a11 ; a22 ; : : : ; ann of A equal zero or, equivalently, if and only if A D 0. b. Extensions to products of nonnegative definite matrices The following theorem extends Theorem 14.7.1. Theorem 14.7.4. For any n n positive definite matrix A and any n n nonnull symmetric nonnegative definite matrix B, tr.AB/ > 0. Proof. Let r D rank.B/. Then, according to Theorem 14.3.7, B D Q0 Q for some r n matrix Q of rank r. Since (according to Theorem 14.2.9) QAQ0 is positive definite, we find (using Lemma 5.2.1 and Theorem 14.7.1) that tr.AB/ D tr.AQ0 Q/ D tr.QAQ0 / > 0: Q.E.D. As an essentially immediate consequence of Theorem 14.7.4, we have the following corollary, which extends Corollary 14.7.3. Corollary 14.7.5. Let A represent an n n positive definite matrix and B an n n symmetric nonnegative definite matrix. Then, tr.AB/ 0, with equality holding if and only if B D 0. In the special case where B D I, Theorem 14.7.4 reduces to Theorem 14.7.1, and, in the special case where A D I, Corollary 14.7.5 reduces to Corollary 14.7.3. The following theorem extends Theorem 14.7.2. Theorem 14.7.6. Let A represent an n n nonnegative definite matrix and B an n n symmetric nonnegative definite matrix. Then, tr.AB/ 0, with equality holding if and only if BAB is skew-symmetric. Proof. The theorem is true in the special case where B D 0, since in that special case tr.AB/ D 0 and BAB D 0. Thus, in proving the theorem, it suffices to restrict attention to the case where B ¤ 0.

242


Suppose that B ¤ 0, and let r D rank.B/. Then, according to Theorem 14.3.7, B D Q0 Q for some r n matrix Q of rank r. Observe [in light of result (5.2.3)] that (7.1) tr.AB/ D tr.AQ0 Q/ D tr.QAQ0 /: According to Theorem 14.2.9, QAQ0 is nonnegative definite. Thus, it follows from Theorem 14.7.2 that tr.QAQ0 / 0, or, equivalently, that tr.AB/ 0, with equality holding if and only if QAQ0 is skew-symmetric or, equivalently, if and only if (7.2) QA0 Q0 D QAQ0 : Since Q is of full row rank and Q0 of full column rank, we have (as a consequence of Lemma 8.3.1) that condition (7.2) is equivalent to the condition Q0 QA0 Q0 Q D Q0 QAQ0 Q:

(7.3)

Moreover, condition (7.3) can be reexpressed as .BAB/0 D BAB: Thus, condition (7.3) is satisfied if and only if BAB is skew-symmetric. Q.E.D. In the special case where B D I, Theorem 14.7.6 reduces to Theorem 14.7.2. If A is symmetric (as well as nonnegative definite), then the results of Theorem 14.7.6 can be sharpened, as described in the following corollary, which generalizes (in a different way than Corollary 14.7.5) Corollary 14.7.3. Corollary 14.7.7. Let A and B represent nn symmetric nonnegative definite matrices. Then, tr.AB/ 0, with equality holding if and only if AB D 0. Proof. Clearly, .BAB/0 D B0 A0 B0 D BAB; that is, BAB is symmetric, implying (in light of Lemma 14.6.1) that BAB is skew-symmetric if and only if BAB D 0. Thus, it follows from Theorem 14.7.6 that tr.AB/ 0, with equality holding if and only if BAB D 0: (7.4) To complete the proof, it suffices to show that condition (7.4) is equivalent to the condition AB D 0. In the special case where A D 0, these two conditions are clearly equivalent. Thus, it suffices to restrict attention to the case where A ¤ 0. Suppose that A ¤ 0, and let r D rank.A/. According to Theorem 14.3.7, A D Q0 Q for some r n matrix Q of rank r. Thus, condition (7.4) can be reexpressed as .QB/0 QB D 0, which (according to Corollary 5.3.2) is equivalent to the condition QB D 0: (7.5) Moreover, since Q0 is of full column rank, it follows from Lemma 8.3.1 that condition (7.5) is equivalent to the condition Q0 QB D Q0 0; which is reexpressible as AB D 0. We conclude that condition (7.4) is equivalent to the condition AB D 0. Q.E.D.

14.8 Partitioned Nonnegative Definite Matrices

243

In the special case where A D I or B D I, Corollary 14.7.7 reduces to Corollary 14.7.3.

14.8 Partitioned Nonnegative Definite Matrices a. Applicability of formulas for ranks, determinants, inverses, and generalized inverses Consider a (square) partitioned matrix A of the form T U AD ; V W where T is an m m (square) matrix, W is an n n (square) matrix, U is of dimensions m n, and V is of dimensions n m, or consider a partitioned matrix B of the form W V BD : U T Suppose that T is nonsingular. Then, formula (13.3.13) can be used to express jAj or jBj in terms of the determinant of T and the determinant of the Schur complement W V T 1 U of T , and formula (8.5.15) can be used to express rank.A/ or rank.B/ in terms of the order m of T and the rank of the Schur complement of T . Suppose further that A or B is nonsingular. Then, formula (8.5.16) or (8.5.17) can be used to express A1 or B1 in terms of the inverse of T and the inverse of the Schur complement of T . Is A or B nonsingular—and is T nonsingular—if A or B is nonnegative definite? If A or B is positive definite, then it is clear from Lemma 14.2.8 that A or B, respectively, is nonsingular and from Corollary 14.2.12 that T is nonsingular. If A or B is a symmetric positive semidefinite matrix, then it is clear from Corollary 14.3.12 that A or B, respectively, is singular. If A or B is a nonsymmetric positive semidefinite matrix, then A or B, respectively, may or may not be nonsingular. Moreover, even if A or B is a nonsingular positive semidefinite matrix, T may be singular. Suppose now that T is possibly singular. If C.U / C.T / and R.V / R.T /, then formula (9.6.2) or (9.6.3) gives a generalized inverse of A or B, respectively, in terms of an arbitrary generalized inverse T of T and an arbitrary generalized inverse of the Schur complement W V T U of T , and formula (9.6.1) can be used to express rank.A/ or rank.B/ in terms of the rank of T and the rank of the Schur complement of T . Are the conditions C.U / C.T / and R.V / R.T / satisfied if A or B is positive semidefinite? If A or B is symmetric as well as positive semidefinite, the answer to this question is yes, as is evident from the corollary of the following lemma.

244


Lemma 14.8.1. Let A represent a symmetric nonnegative definite matrix that has been partitioned as T U AD ; V W where T (and hence W ) is square. Then, there exist matrices R and S such that T D R0 R;

U D R0 S;

V D S0 R;

W D S0 S:

Proof. According to Corollary 14.3.8, there exists a matrix X such that A D X0 X. Partition X as X D .R; S/ (where R has the same number of columns as T ). Then, 0 0 R R R0 S R .R; S/ D : A D X0 X D S0 S0 R S0 S Since R0 R, R0 S, S0 R, and S0 S are of the same dimensions as T , U , V , and W , Q.E.D. respectively, T D R0 R, U D R0 S, V D S0 R, and W D S0 S. Corollary 14.8.2. Let A represent a symmetric nonnegative definite matrix that has been partitioned as T U AD ; V W where T (and hence W ) is square. Then, C.U / C.T / and R.V / R.T /. Similarly, C.V / C.W / and R.U / R.W /. Proof. According to Lemma 14.8.1, there exist matrices R and S such that T D R0 R;

U D R0 S;

V D S0 R;

W D S0 S:

Thus, making use of Corollaries 4.2.3 and 7.4.5, we find that C.U / C.R0 / D C.T / and R.V / R.R/ D R.T /: It can be shown in similar fashion that C.V / C.W / and R.U / R.W /. Q.E.D. b. Block-diagonal matrices Lemma 14.2.1 characterizes the concepts of nonnegative definiteness, positive definiteness, and positive semidefiniteness as applied to diagonal matrices. The following result extends the results of Lemma 14.2.1 to block-diagonal matrices. Lemma 14.8.3. Let Ai represent an ni ni matrix (i D 1; 2; : : : ; k), let n D n1 C n2 C C nk , and define A to be the n n block-diagonal matrix A D diag.A1 ; A2 ; : : : ; Ak /: Then, (1) A is nonnegative definite if and only if A1 ; A2 ; : : : ; Ak are nonnegative definite; (2) A is positive definite if and only if A1 ; A2 ; : : : ; Ak are positive definite; and (3) A is positive semidefinite if and only if the diagonal blocks A1 ; A2 ; : : : ; Ak are nonnegative definite with at least one of the diagonal blocks being positive semidefinite.

14.8 Partitioned Nonnegative Definite Matrices

245

Proof. Consider the quadratic form x0 Ax (in an n-dimensional column vector x) whose matrix is A. Partition x0 as x0 D .x01 ; x02 ; : : : ; x0k /, where (for i D 1; 2; : : : ; k) xi is an ni -dimensional (column) vector. Then, clearly, x0 Ax D x01 A1 x1 C x02 A2 x2 C C x0k Ak xk : (1) If A1 ; A2 ; : : : ; Ak are nonnegative definite, then, by definition, x0i Ai xi 0 for every xi (i D 1; 2; : : : ; k), implying that x0 Ax 0 for every x and hence that A is nonnegative definite. Conversely, if A is nonnegative definite, then it follows from Corollary 14.2.12 that A1 ; A2 ; : : : ; Ak are nonnegative definite. (2) If A1 ; A2 ; : : : ; Ak are positive definite, then, by definition, x0i Ai xi > 0 for every nonnull xi (i D 1; 2; : : : ; k), implying that x0 Ax > 0 for every nonnull x and hence that A is positive definite. Conversely, if A is positive definite, then it follows from Corollary 14.2.12 that A1 ; A2 ; : : : ; Ak are positive definite. (3) Suppose that A1 ; A2 ; : : : ; Ak are nonnegative definite and that, for some i , say i D i , Ai is positive semidefinite. Then, by definition, x0i Ai xi 0 for every xi (i D 1; 2; : : : ; k), and there exists some nonnull value of xi , say xi D xQ i , for which x0i Ai xi D 0. It follows that x0 Ax 0 for every x, with equality holding for x0 D .0; : : : ; 0; xQ 0i ; 0; : : : ; 0/. Thus, A is positive semidefinite. Conversely, suppose that A is positive semidefinite. Then, it follows from Part (1) that A1 ; A2 ; : : : ; Ak are nonnegative definite. Moreover, it follows from Part (2) that not all of the matrices A1 ; A2 ; : : : ; Ak are positive definite and hence (since they are nonnegative definite) that at least one of them is positive semidefinite. Q.E.D. c. Nonnegative definiteness of a Schur complement Sufficient conditions for the positive definiteness or positive semidefiniteness of a Schur complement are given by the following theorem. Theorem 14.8.4. Let

AD

T U V W

;

where T is of dimensions m m, U of dimensions m n, V of dimensions n m, and W of dimensions n n. (1) If A is positive definite, then the Schur complement W V T 1 U of T and the Schur complement T U W 1 V of W are positive definite. (2) If A is symmetric positive semidefinite, then the Schur complement W V T U of T and the Schur complement T U W V of W are nonnegative definite. (3) If A is positive semidefinite (but not necessarily symmetric) and if C.U / C.T / or R.V / R.T /, then the Schur complement W V T U of T (relative to T ) is nonnegative definite. 0 (3 ) If A is positive semidefinite and if C.V / C.W / or R.U / R.W /, then the Schur complement T U W V of W (relative to W ) is nonnegative definite.

246


Proof. Let us confine our attention to Result (3) and to the parts of Results (1) and (2) that pertain to the Schur complement of T . Result (30 ) and the parts of Results (1) and (2) that pertain to the Schur complement of W can be established via analogous arguments. Define Im T U : PD 0 In According to Lemma 8.5.2, P is nonsingular. Further, T .I T T /U P 0 AP D ; V .T U /0 T Q where Q D W V T U .T U /0 .I T T /U . Suppose that A is positive definite. Then, according to Corollary 14.2.10, P 0 AP is positive definite. Thus, it follows from Corollary 14.2.12 that Q is positive definite. A further implication of Corollary 14.2.12 is that T is positive definite and hence nonsingular, so that Q D W V T 1 U . We conclude that if A is positive definite, then the Schur complement of T is positive definite. Suppose now that A is positive semidefinite. Then, according to Corollary 14.2.10, P 0 AP is positive semidefinite. Thus, it follows from Corollary 14.2.12 that Q is nonnegative definite. If C.U / C.T /—which (in light of Corollary 14.8.2) would be the case if A were symmetric—then (according to Lemma 9.3.5) .I T T /U D 0 and hence Q D W V T U . We conclude that if A is symmetric positive semidefinite or, more generally, if A is positive semidefinite and C.U / C.T /, then the Schur complement of T is nonnegative definite. That the Schur complement of T is nonnegative definite if A is positive semidefinite and R.V / R.T / can be established via a similar argument. Q.E.D. As a partial “converse” to Theorem 14.8.4, we have the following result. Theorem 14.8.5. Let

AD

T U U0 W

;

where T is of dimensions m m, U of dimensions m n, and W of dimensions n n. (1) If T is symmetric positive definite and the Schur complement W U 0 T 1 U of T is positive definite, or if W is symmetric positive definite and the Schur complement T U W 1 U 0 of W is positive definite, then A is positive definite. (2) Suppose that T is symmetric nonnegative definite and that the Schur complement W U 0 T U of T is nonnegative definite. Suppose further that C.U / C.T /. Then, A is nonnegative definite. (20 ) Suppose that W is symmetric nonnegative definite and that the Schur complement T U W U 0 of W is nonnegative definite. Suppose further that R.U / R.W /. Then, A is nonnegative definite.

14.9 Some Results on Determinants

247

Proof. Suppose that T is symmetric positive definite (in which case T is nonsingular) or more generally that T is symmetric nonnegative definite and C.U / C.T /. Define I T U : PD m 0 In According to Lemma 8.5.2, P is nonsingular. Moreover, P 0 AP D diag.T ; W U 0 T U /; as can be easily verified by making use of Lemma 9.3.5, or equivalently A D .P 1 /0 diag.T ; W U 0 T U /P 1 :

(8.1)

If W U 0 T U is nonnegative definite, then it follows from Lemma 14.8.3 that diag.T ; W U 0 T U / is nonnegative definite, implying [in light of equality (8.1) and Corollary 14.2.10] that A is nonnegative definite. Similarly, if T is positive definite (and hence invertible) and W U 0 T 1 U is positive definite, then diag.T ; W U 0 T 1 U / is positive definite, implying that A is positive definite. We conclude that if T is symmetric nonnegative definite, W U 0 T U is nonnegative definite, and C.U / C.T /, then A is nonnegative definite, and that if T is symmetric positive definite and W U 0 T 1 U is positive definite, then A is positive definite. It can be shown, in similar fashion, that if W is symmetric nonnegative definite, T U W U 0 is nonnegative definite, and R.U / R.W /, then A is nonnegative definite, and that if W is symmetric positive definite and Q.E.D. T U W 1 U 0 is positive definite, then A is positive definite. In the special case of a symmetric partitioned matrix, we have (in light of Theorem 14.8.4 and Corollary 14.2.12) the following corollary of Theorem 14.8.5. Corollary 14.8.6. Suppose that a symmetric matrix A is partitioned as T U AD U0 W (where T and W are square). Then, A is positive definite if and only if T and the Schur complement W U 0 T 1 U of T are both positive definite. Similarly, A is positive definite if and only if W and the Schur complement T U W 1 U 0 of W are both positive definite.

14.9 Some Results on Determinants a. Determinant of a symmetric nonnegative definite matrix Let A represent an n n symmetric positive definite matrix. Then, according to Corollary 14.3.13, there exists a nonsingular matrix P such that A D P 0 P . Since (in light of Theorem 13.3.7) jP j ¤ 0, it follows that

248


jAj D jP 0 jjP j D jP j2 > 0: Thus, recalling Corollary 14.3.12, we have the following theorem. Lemma 14.9.1. The determinant of a symmetric positive definite matrix is (strictly) positive. The determinant of a symmetric positive semidefinite matrix equals zero. b. Determinant of a (not necessarily symmetric) nonnegative definite matrix Those nonsymmetric nonnegative definite matrices that are nonsingular have nonzero determinants—refer to Theorem 13.3.7. Can anything further be said about their determinants? In particular, is the determinant of a nonsingular nonsymmetric nonnegative definite matrix, like that of a symmetric positive definite matrix, necessarily positive? In what follows, the answer to this question is shown to be yes. Preliminary to considering the determinant of a possibly nonsymmetric nonnegative definite matrix, it is convenient to establish the following result on the determinants of skew-symmetric matrices—recall (from Lemma 14.6.4) that a matrix is skew-symmetric if and only if it is nonnegative definite and its diagonal elements equal zero. Theorem 14.9.2. Let A represent an n n skew-symmetric matrix. (1) If n is an odd number, then det.A/ D 0. (2) If n is an even number, then det.A/ 0. Proof. (1) By definition, A0 D A, so that, according to Lemma 13.2.1 and Corollary 13.2.5, jAj D jA0 j D jAj D .1/n jAj: Consequently, if n is an odd number, jAj D jAj, implying that 2jAj D 0 and hence that jAj D 0. (2) The proof that det.A/ 0 when n is an even number is by mathematical induction. When n D 2, 0 c AD c 0 for some scalar c, in which case jAj D c 2 0: Suppose now that the determinant of every .n 2/ .n 2/ skew-symmetric matrix is greater than or equal to zero, where n is an arbitrary even number (greater than or equal to 4). To complete the induction argument, we must show that the determinant of an arbitrary n n skew-symmetric matrix A D faij g is greater than or equal to zero. For this purpose, it suffices to restrict attention to the case where at least one of the last n 1 elements of the first row of A is nonzero. (The case where all n 1 of these elements are zero is trivial, since in that case the first row of A is null and


249

hence it follows from Corollary 13.2.3 that jAj D 0.) Assume then that a1m ¤ 0 for some integer m (2 m n). Let B D P 0 AP , where P is the permutation matrix obtained by interchanging the second and mth columns of In . (If m D 2, take P D I.) Observe (in light of Theorem 13.3.4 and Corollary 13.3.6) that jBj D jP j2 jAj D jAj; (9.1) B11 B12 and partition B as B D , where the dimensions of B11 are 2 2. B21 B22 Accordingto Lemma 14.6.2, B is skew-symmetric, implying that B11 D 0 a1m 0 0 , B12 D B21 , and B22 D B22 . Since a1m ¤ 0, we have that a1m 0 2 > 0; jB11 j D a1m

and it follows from Theorem 13.3.8 that 2 1 jBj D a1m jB22 B21 B11 B12 j:

(9.2)

Moreover, 1 0 0 0 1 0 .B22 B21 B11 B12 /0 D B22 B12 .B11 / B21 1 D B22 .B21 /.B11 /.B12 / 1 D .B22 B21 B11 B12 /; 1 B12 [which is of dimensions .n 2/ .n 2/] is skewso that B22 B21 B11 symmetric. Thus, by supposition, 1 jB22 B21 B11 B12 j 0:

(9.3)

Together, results (9.2) and (9.3) imply that jBj 0 and hence [in light of result (9.1)] that jAj 0. Q.E.D. We are now in a position to establish the following result on the determinant of a nonnegative definite matrix. Theorem 14.9.3. For any nonnegative definite matrix A, det.A/ 0. Proof. The proof is by mathematical induction. Clearly, the determinant of any 1 1 nonnegative definite matrix is nonnegative. Suppose now that the determinant of every .n 1/ .n 1/ nonnegative definite matrix is nonnegative, and consider the determinant of an nn nonnegative definite matrix A D faij g. To complete the induction argument, we must show that det.A/ 0. In light of Theorem 14.9.2, it suffices to restrict attention to the case where at least one of the diagonal elements a11 ; a22 ; : : : ; ann of A is nonzero and hence positive. [If all n of the diagonal elements of A are zero, then (according to Lemma

250


14.6.4) A is skew-symmetric, and we have, as an immediate consequence of Theorem 14.9.2, that det.A/ 0.] Assume then that amm > 0 for some integer m (1 m n). Let B D P 0 AP , where P is the permutation matrix obtained by interchanging the first and mth columns of In . (If m D 1, take P D I.) Observe (in light of Theorem 14.2.9) that B is nonnegative definite and that jBj D jP j2 jAj D jAj: (9.4) B11 B12 Partition B as B D , where the dimensions of B11 are 1 1. B21 B22 Clearly, B11 D .amm /, and consequently (since amm ¤ 0) it follows from Theorem 13.3.8 that 1 B12 j: (9.5) jBj D amm jB22 B21 B11

1 Moreover, according to Parts (1) and (3) of Theorem 14.8.4, B22 B21 B11 B12 [which is of dimensions .n 1/ .n 1/] is nonnegative definite, so that, by supposition, 1 B12 j 0: (9.6) jB22 B21 B11

Since amm > 0, we conclude [on the basis of results (9.5) and (9.6)] that jBj 0 and hence [in light of result (9.4)] that jAj 0. Q.E.D. In light of Lemma 14.2.8 and Corollary 14.3.12, we have, as a corollary of Theorem 14.9.3, the following result, which generalizes Lemma 14.9.1. Corollary 14.9.4. The determinant of a positive definite matrix or of a nonsingular nonsymmetric positive semidefinite matrix is (strictly) positive. The determinant of a symmetric positive semidefinite matrix or of a singular nonsymmetric positive semidefinite matrix equals 0. c. A necessary and sufficient condition for the positive definiteness of a symmetric matrix Whether a symmetric matrix is positive definite can be ascertained from the determinants of its leading principal submatrices. The following theorem provides the basis for doing so. Theorem 14.9.5. Let A D faij g represent an n n symmetric matrix, and, for k D 1; 2; : : : ; n, let 0 1 a11 a12 : : : a1k Ba21 a22 : : : a2k C B C Ak D B : :: : : : C @ :: : :: A : ak1 ak2 : : : akk represent the leading principal submatrix of A of order k. Then, A is positive definite if and only if, for k D 1; 2; : : : ; n, det.Ak / > 0, that is, if and only if the determinants of all n of the leading principal submatrices A1 ; A2 ; : : : ; An of A are positive.


251

In proving Theorem 14.9.5, it is convenient to make use of the following result, which is of some interest in its own right. Lemma 14.9.6. Let A represent an n n symmetric matrix (where n 2), and partition A as A a AD ; a0 c where the dimensions of A are .n 1/ .n 1/. Then, A is positive definite if and only if A is positive definite and jAj > 0. Proof (of Lemma 14.9.6). If A is positive definite, then it is clear from Corollary 14.2.12 that A is positive definite and from Lemma 14.9.1 that jAj > 0. Conversely, suppose that A is positive definite (and hence nonsingular) and that jAj > 0. Then, according to Theorem 13.3.8, jAj D jA j.c a0 A1 a/: Since (according to Lemma 14.9.1) jA j > 0 (and since jAj > 0), we conclude that the Schur complement c a0 A1 a of A (like A itself) is positive definite and hence [in light of Part (1) of Theorem 14.8.5] that A is positive definite. Q.E.D. Proof (of Theorem 14.9.5). That the determinants of A1 ; A2 ; : : : ; An are positive if A is positive definite is an immediate consequence of Corollary 14.2.12 and Lemma 14.9.1. For purposes of proving the converse, suppose that the determinants of A1 , A2 , : : : ; An are positive. The proof consists of establishing, via a mathematical induction argument, that A1 ; A2 ; : : : ; An are positive definite, which (since A D An ) implies in particular that A is positive definite. Clearly, A1 is positive definite. Suppose now that Ak1 is positive definite (where 2 k n), and partition Ak as Ak1 ak ; Ak D a0k akk where ak D .a1k ; a2k ; : : : ; ak1;k /0 . Since Ak1 is (by supposition) positive definite (and since jAk j > 0), it follows from Lemma 14.9.6 that Ak is positive definite. We conclude, on the basis of the induction argument, that A1 ; A2 ; : : : ; An are positive definite and that A in particular is positive definite. Q.E.D. d. A necessary and sufficient condition for the nonnegative definiteness of a symmetric matrix Is it the case that an nn symmetric matrix is nonnegative definite if and only if the determinants of all n of its leading principal submatrices are nonnegative (as might be conjectured on the basis of Theorem 14.9.5)? It is clear (from Corollary 14.2.12 and Lemma 14.9.1) that the nonnegativity of the determinants of the n leading

252


principal submatrices is a necessary condition (for nonnegative definiteness). It can be shown, however, that this condition is not sufficient. In this section, we establish a necessary and sufficient condition for the nonnegative definiteness of a symmetric matrix. This condition is analogous to the necessary and sufficient condition for positive definiteness given by the following theorem (which differs from the necessary and sufficient condition for positive definiteness given by Theorem 14.9.5). Theorem 14.9.7. Let A represent an n n symmetric matrix. Then, A is positive definite if and only if (1) A is nonsingular and (2) the determinant of every principal submatrix of A is nonnegative. To prove Theorem 14.9.7, we require the following result, which is of some interest in its own right. Theorem 14.9.8. Let A represent an n n nonsingular symmetric matrix (where n 3), and partition A as B a ; AD a0 c where the dimensions of B are .n 1/ .n 1/. Then, B is nonsingular or B contains an .n 2/ .n 2/ nonsingular principal submatrix. Moreover, if B is singular and B is any .n 2/ .n 2/ nonsingular principal submatrix of B, then jB jjAj < 0 (i.e., jB j and jAj are opposite in sign). Proof (of Theorem 14.9.8). Suppose that B is singular, and let A1 D .B; a/. Since the rows of A are linearly independent, the rows of A1 are also linearly independent, and hence rank.A1 / D n 1. Thus, A1 contains n 1 linearly independent columns, implying that B contains (at least) n2 linearly independent columns. Since (by supposition) B is singular, we have that rank.B/ D n 2: Moreover, B is symmetric. We conclude (in light of Corollary 4.4.11) that B contains an .n 2/ .n 2/ nonsingular principal submatrix. Let B represent an .n 2/ .n 2/ nonsingular principal submatrix of B, say that obtained by striking out the kth row and column (of B). Let U D .U ; u/, where U is the .n 1/ .n 2/ submatrix of In1 obtained by striking out the kth column (of In1 ) and u is the kth column of In1 . Then, U is a permutation matrix, and B U0 Bu 0 U BU D u0 BU u0 Bu [so that the .n 2/ .n 2/ leading principal submatrix of U 0 BU is B ]. Take P to be the n n matrix U 0 : PD 00 1 Then, P is a permutation matrix, and


253

0 1 B U0 Bu U0 a U 0 BU U 0 a P AP D D @u0 BU u0 Bu u0 a A : c a0 U a0 u c a0 U 0

Thus, making use of Theorem 13.3.8, we find that

f11 where F D f12

jAj D jP j2 jAj D jP 0 AP j D jB jjF j; f12 with f22

(9.7)

f11 D u0 Bu u0 BU B1 U0 Bu; f12 D u0 a u0 BU B1 U0 a; f22 D c a0 U B1 U0 a: Further, jBj D jU j2 jBj D jU 0 BU j D jB j.u0 Bu u0 BU B1 U0 Bu/ D jB jf11 : (9.8) Since (by supposition) B is singular and since (by definition) B is nonsingular, we have that jBj D 0 and jB j ¤ 0, implying [in light of result (9.8)] that f11 D 0 2 . Thus, making use of result (9.7), we find that and hence that jF j D f12 2 jAj D f12 jB j:

We conclude that jB j and jAj are opposite in sign. Q.E.D. Proof (of Theorem 14.9.7). If A is positive definite, then it is clear from Lemma 14.2.8, Corollary 14.2.12, and Lemma 14.9.1 that A is nonsingular and that the determinant of every principal submatrix of A is positive (and hence nonnegative). Thus, it suffices to prove that if A is nonsingular and the determinant of every principal submatrix of A is nonnegative, then A is positive definite. The proof of this is by mathematical induction. Consider first the relatively trivial case of a 1 1 matrix A D .a11 / that is nonsingular and whose only principal submatrix (A itself) has a nonnegative determinant. We have that a11 ¤ 0 and a11 0, implying that a11 > 0 and hence that A is positive definite. a11 a12 that Next, consider the case of a 2 2 symmetric matrix A D a12 a22 is nonsingular and whose three principal submatrices [.a11 /, .a22 /, and A itself] have nonnegative determinants. We have that jAj ¤ 0, implying (since jAj 0) that jAj > 0. And a11 0, so that a11 > 0 (since if a11 D 0, it would be the case 2 0). Based on Theorem 14.9.5, we conclude that A is positive that jAj D a12 definite. Now, suppose that every .n 1/ .n 1/ symmetric matrix that is nonsingular and whose principal submatrices all have nonnegative determinants is positive definite, and consider an n n symmetric matrix A that is nonsingular and whose

254


principal submatrices all have nonnegative determinants. To complete the induction argument, we must show that A is positive definite. Observe that jAj ¤ 0 (since A is nonsingular) and that jAj 0 (since A is a principal submatrix of itself) and hence that jAj > 0. Partition A as B a AD ; a0 c where the dimensions of B are .n 1/ .n 1/. Observe that every principal submatrix of B is a principal submatrix of A and hence that every principal submatrix of B has a nonnegative determinant. Observe also that B is nonsingular [since if B were singular, B would (according to Theorem 14.9.8) contain an .n 2/ .n 2/ principal submatrix whose determinant would be opposite in sign to jAj and hence negative]. Accordingly, it follows from our supposition that B is positive definite. We conclude (on the basis of Lemma 14.9.6) that A is positive definite. Q.E.D. For purposes of establishing a necessary and sufficient condition for the nonnegative definiteness of a symmetric matrix, analogous to the necessary and sufficient condition given by Theorem 14.9.7 for the positive definiteness of a symmetric matrix, it is helpful to introduce the following two lemmas. Lemma 14.9.9. Let A represent an n n symmetric matrix of rank r. Then, corresponding to any r r nonsingular principal submatrix A of A, there exists an r n matrix Q (of rank r) such that A D Q0 A Q. Proof. Suppose that A is the r r nonsingular principal submatrix obtained by striking out all of the rows and columns of A except the i1 , i2 , : : : ; ir th rows and columns (where i1 < i2 < < ir ). Take P to be any permutation matrix whose first, second, : : : ; rth columns are the i1 , i2 , : : : ; ir th columns, respectively, of In . Let B D P 0 AP , and partition P as P D .P1 ; P2 /, where P1 is of dimensions n r. Then, A D P10 AP1 and hence A B12 BD ; 0 B12 B22 where B12 D P10 AP2 and B22 D P20 AP2 . Since rank.B/ D rank.A/ D r, it 0 A1 follows from Lemma 9.2.2 that B22 D B12 B12 . Thus, 0 1 B D .Ir ; A1 B12 / A .Ir ; A B12 /;

so that A D P BP 0 D Q0 A Q; 0 where Q D .Ir ; A1 B12 /P . Moreover, rank.Q/ rank.A/ D r, and (since Q is of dimensions r n) rank.Q/ r. Thus, rank.Q/ D r. Q.E.D.

Lemma 14.9.10. . Let A represent an n n symmetric matrix of rank r, and let A represent an r r nonsingular principal submatrix of A. Then, A is nonnegative definite if and only if A is positive definite.

14.10 Geometrical Considerations

255

Proof. Suppose that A is nonnegative definite. Then, according to Corollary 14.2.12, A is nonnegative definite. Moreover, since A is symmetric, A is symmetric. Thus, it follows from Corollary 14.3.12 that A is positive definite. Conversely, suppose that A is positive definite. Then, according to Lemma 14.9.9, there exists a matrix Q such that A D Q0 A Q. Thus, it follows from Theorem 14.2.9 that A is nonnegative definite. Q.E.D. The following theorem gives a necessary and sufficient condition for the nonnegative definiteness of a symmetric matrix, analogous to the necessary and sufficient condition given by Theorem 14.9.7 for the positive definiteness of a symmetric matrix. Theorem 14.9.11. Let A represent an n n symmetric matrix. Then, A is nonnegative definite if and only if the determinant of every principal submatrix of A is nonnegative. Proof. Suppose that the symmetric matrix A is nonnegative definite. Then (in light of Corollary 14.2.12), every principal submatrix of A is nonnegative definite (and symmetric). And we conclude (on the basis of Lemma 14.9.1) that the determinant of every principal submatrix of A is nonnegative. Conversely, suppose that the determinant of every principal submatrix of A is nonnegative. Let r D rank.A/. Then, according to Corollary 4.4.11, A contains an r r nonsingular principal submatrix, say A . Clearly, every principal submatrix of A is a principal submatrix of A, and hence (by supposition) the determinant of every principal submatrix of A is nonnegative, implying (in light of Theorem 14.9.7) that A is positive definite. We conclude (on the basis of Lemma 14.9.10) that A is nonnegative definite. Q.E.D.

14.10 Geometrical Considerations As discussed in Section 6.1, the inner product of two m n matrices A and B (in a linear space V of m n matrices) is the value A B assigned to A and B by a designated function f whose domain consists of all pairs of matrices in V and that has certain basic properties. The usual inner product of A and B is that given by the formula A B D tr.A0 B/: In the special case of n-dimensional column vectors x D fxi g and y D fyi g, the formula for the usual inner product simplifies to x y D x0 y D x1 y1 C x2 y2 C C xn yn : In the present section, we consider alternative inner products for the linear space Rn1 of all n-dimensional column vectors. a. Expression of inner products as bilinear forms Consider a bilinear form x0 Ay (in n-dimensional column vectors x and y). If the matrix A is symmetric positive definite, then x0 Ay qualifies as an inner product for

256


Rn1 . To see this, let x, y, and z represent arbitrary n-dimensional column vectors and k an arbitrary scalar, and observe that if A is symmetric positive definite, then the bilinear form has the following four properties: (1) x0 Ay D y 0 Ax; (2) x0 Ax 0, with equality holding if and only if x D 0; (3) .kx/0 Ay D k.x0 Ay/; (4) .x C y/0 Az D x0 Az C y 0 Az. Are the symmetry and positive definiteness of A necessary as well as sufficient for the bilinear form x0 Ay to qualify as an inner product for Rn1 ? The answer is yes. If A were not symmetric, then (as discussed in Section 14.1) the bilinear form x0 Ay would not be symmetric; that is, there would exist n-dimensional vectors x and y such that x0 Ay ¤ y 0 Ax. Moreover, if A were not positive definite, then x0 Ax < 0 for some x or x0 Ax D 0 for some nonnull x. Note (in light of Corollary 14.1.3) that distinct (symmetric positive definite) choices for the matrix A of the bilinear form x0 Ay produce distinct inner products. Is every inner product for Rn1 expressible as a bilinear form? The answer is yes. To verify this, let x y represent the value assigned to arbitrary n-dimensional column vectors x and y by an inner product for Rn1 ; denote by e1 ; : : : ; en the first, : : : ; nth columns, respectively, of In ; and take A to be the n n matrix whose ij th element is aij D ei ej . Then, X xy D x i ei y i

D

X i

D

X

xi .ei y/ h X i x i ei yj ej

i

D

X i

xi

X j

j

yj .ei ej / D

X

xi yj aij D x0 Ay:

i;j

In summary, a bilinear form (in n-dimensional vectors) qualifies as an inner product for Rn1 if and only if the matrix of the bilinear form is symmetric and positive definite; distinct (symmetric positive definite) choices for the matrix of the bilinear form produce distinct inner products for Rn1 ; and every inner product for Rn1 is expressible as a bilinear form. b. Schwarz inequality In light of the discussion of Subsection a, we obtain, as a direct consequence of Theorem 6.3.1, the following version of the Schwarz inequality. Theorem 14.10.1. Let W represent an n n symmetric positive definite matrix. Then, for any n-dimensional column vectors x and y,

14.10 Geometrical Considerations

.x0 W y/2 .x0 W x/.y 0 W y/;

257

(10.1)

with equality holding if and only if y D 0 or x D ky for some scalar k. The following corollary, which is essentially a special case of Corollary 6.3.2, expresses the result of Theorem 14.10.1 in an alternative form. Corollary 14.10.2. Let W represent an n n symmetric positive definite matrix. Then, for any n-dimensional column vector y, .x0 W y/2 D y 0 W y: fx2Rn W x¤0g x0 W x max

(10.2)

Moreover, if y D 0, the maximum is attained at every nonnull x 2 Rn , and if y ¤ 0, the maximum is attained at every nonnull x 2 Rn that is proportional to y. By replacing the vector y in Theorem 14.10.1 and Corollary 14.10.2 with the vector W 1 y, we obtain the following two corollaries. Corollary 14.10.3. Let W represent an n n symmetric positive definite matrix. Then, for any n-dimensional column vectors x and y, .x0 y/2 .x0 W x/.y 0 W 1 y/;

(10.3)

with equality holding if and only if y D 0 or x D kW 1 y for some scalar k. Corollary 14.10.4. Let W represent an n n symmetric positive definite matrix. Then, for any n-dimensional column vector y, .x0 y/2 D y 0 W 1 y: fx2Rn W x¤0g x0 W x max

(10.4)

Moreover, if y D 0, the maximum is attained at every nonnull x 2 Rn , and if y ¤ 0, the maximum is attained at every nonnull x 2 Rn that is proportional to W 1 y. c. Quasi-inner products Let V represent a linear space of m n matrices, let f represent a function whose domain consists of all pairs of matrices in V, and, for A and B in V, denote by A B the value assigned by f to the pair A and B. The four basic requirements that would (for all matrices A, B, and C in V and for every scalar k) have to be satisfied by f for f to qualify as an inner product are: (1) A B D B A; (2) A A 0, with equality holding if and only if A D 0; (3) .kA/ B D k.A B/; (4) .A C B/ C D .A C/ C .B C/. A less restrictive requirement than requirement (2) is the requirement

258


(20 ) A A 0, with equality holding if A D 0; obtained by allowing A A to equal 0 even if A ¤ 0. If f satisfies requirement (20 ) [as well as requirements (1), (3), and (4)], but not necessarily requirement (2), let us use the term quasi-inner product in referring to f and to the values assigned by f to the various pairs of matrices. Subsequently, the symbol A B will sometimes be used to represent the quasiinner product of two matrices A and B, although this will never be done without calling attention to the (temporary) departure from our standard usage of A B as a symbol for a “true” inner product. In using the dot notation to represent a quasiinner product, the symbol kAk will continue to be used to represent the quantity .A A/1=2 . In the case of a quasi-inner product, let us refer to this quantity as a quasi-norm, rather than a norm. It is a straightforward exercise to generalize the results of Subsection a to quasiinner products. We find that a bilinear form (in n-dimensional vectors) qualifies as a quasi-inner product for Rn1 if and only if the matrix of the bilinear form is symmetric and nonnegative definite; that distinct (symmetric nonnegative definite) choices for the matrix of the bilinear form produce distinct quasi-inner products for Rn1 ; and that every quasi-inner product for Rn1 is expressible as a bilinear form. The Schwarz inequality (6.3.1) applies to quasi-inner products as well as “true” inner products. To see this, let V represent a linear space of m n matrices, and let A B represent the value assigned by a quasi-inner product to any pair of matrices A and B (in V). That the Schwarz inequality jA Bj kAkkBk

(10.5)

is valid for all A and B in V such that kBk > 0 can be established in essentially the same way as in the case of a “true” inner product—refer to the proof of Theorem 6.3.1. And it can be established via an analogous argument that this inequality is valid for all A and B in V such that kAk > 0. It remains to establish the validity of inequality (10.5) for those A and B in V such that kAk D kBk D 0—this can be accomplished by observing that if kAk D kBk D 0, then 0 .A C B/ .A C B/ D 2A B and 0 .A B/ .A B/ D 2A B; and, as a consequence, A B D 0. As in the case of a “true” inner product, a sufficient condition for equality to hold in inequality (10.5), that is, for jA Bj D kAkkBk;

(10.6)

is that B D 0 or A D kB for some scalar k. However, this condition is no longer necessary. That is, in general, equality (10.6) can be satisfied without B being null or A being proportional to B.

14.11 Some Results on Ranks and Row and Column Spaces and on Linear Systems

259

In the case of a quasi-inner product, kAk D 0 need not imply that A D 0. However, even in the case of a quasi-inner product, kAk D 0

)

AB D 0

(10.7)

(for every B in V), as is evident from the Schwarz inequality.

14.11 Some Results on Ranks and Row and Column Spaces and on Linear Systems The following theorem generalizes Theorem 7.4.1. Theorem 14.11.1. Let W represent an m m symmetric nonnegative definite matrix. Then, for any m n matrix A and m p matrix B, the linear system A0 W AX D A0 W B (in X)

(11.1)

is consistent. Proof. According to Corollary 14.3.8, W D P 0 P for some matrix P . Linear system (11.1) can be rewritten as A0 A X D A0 B ; where A D P A and B D P B. Thus, it follows from Theorem 7.4.1 that linear system (11.1) is consistent. Q.E.D. Some results on the row and column spaces and the rank of the coefficient matrix of linear system (11.1) are given by the following lemma, which generalizes Corollary 7.4.5. Lemma 14.11.2. For any m n matrix A and any m m symmetric nonnegative definite matrix W , C.A0 W A/ D C.A0 W /, R.A0 W A/ D R.W A/, and rank.A0 W A/ D rank.W A/. Proof. According to Corollary 14.3.8, W D P 0 P for some matrix P . Observing that A0 W A D .P A/0 P A and making use of Corollary 7.4.5 and Theorem 7.4.3, we find that R.A0 W A/ D R.P A/ D R.P 0 P A/ D R.W A/: That C.A0 W A/ D C.A0 W / follows from Lemma 4.2.5. Q.E.D. Since any positive definite matrix is nonsingular, we have (in light of Corollary 8.3.3) the following corollary of Lemma 14.11.2, which (like Lemma 14.11.2 itself) can be regarded as a generalization of Corollary 7.4.5. Corollary 14.11.3. For any m n matrix A and any m m symmetric positive definite matrix W , C.A0 W A/ D C.A0 /, R.A0 W A/ D R.A/, and rank.A0 W A/ D rank.A/.

260


14.12 Projections, Projection Matrices, and Orthogonal Complements In this section, further consideration is given to the projection of an n-dimensional column vector on a subspace of Rn1 . By definition, the projection depends on the choice of inner product. A rather extensive discourse on the projection of a column vector was presented in Chapter 12. The results included in that discourse are, for the most part, specific to the case where the inner product is the usual inner product. In this section, these results are generalized to an arbitrary inner product. a. Some terminology, notation, and basic results Let W represent an n n symmetric positive definite matrix. As discussed in Section 14.10a, the bilinear form x0 W y (in n-dimensional column vectors x and y) qualifies as an inner product for Rn1 . (And every inner product is expressible as a bilinear form.) When the bilinear form x0 W y is chosen to be the inner product for Rn1, x ? y (i.e., two n-dimensional column vectors x and y are orthogonal) if and only if x0 W y D 0, x ? U (i.e., x is orthogonal to a subspace U of Rn1 ) if and only if x0 W y D 0 for every y in U, and U ? V (i.e., U is orthogonal to a subspace V) if and only if x0 W y D 0 for every x in U and every y in V. To indicate that x0 W y D 0 for two n-dimensional vectors x and y, let us (for convenience and/or to emphasize that the orthogonality of two vectors depends on the choice of inner product) write x ?W y or say that x and y are orthogonal with respect to W . Similarly, to indicate that x0 W y D 0 for every vector y in a subspace U, let us write x ?W U or say that x and U are orthogonal with respect to W . And, to indicate that x0 W y D 0 for every x in a subspace U and every y in a subspace V, let us write U ?W V or say that U and V are orthogonal with respect to W . Clearly, saying that x and y, x and U, or U and V are orthogonal with respect to I is synonymous with saying that they are orthogonal with respect to the usual inner product. Furthermore, let us extend our use of this notation and terminology to the case where W is positive semidefinite rather than positive definite. Thus, for any n n symmetric nonnegative definite matrix W , let us write x ?W y or say that x and y are orthogonal with respect to W to indicate that x0 W y D 0; let us write x ?W U or say that x and U are orthogonal with respect to W to indicate that x0 W y D 0 for every y in U; and let us write U ?W V or say that U and V are orthogonal with respect to W to indicate that x0 W y D 0 for every x in U and every y in V. Note that if W is only positive semidefinite, then the bilinear form x0 W y does not qualify as an inner product (as it does when W is positive definite)—although it still qualifies as what (in Section 14.10c) is called a quasi-inner product. By applying Lemma 12.1.1 (in the special case where V D Rm1 and U and W are column spaces of matrices), we obtain the following generalization of Corollary 12.1.2. Lemma 14.12.1. Let y represent an m-dimensional column vector, X an mn matrix, and Z an m p matrix, and let W represent an m m symmetric positive

14.12 Projections, Projection Matrices, and Orthogonal Complements

261

definite matrix. Then, y ?W C.X/ if and only if X0 W y D 0 (or equivalently if and only if y 0 W X D 0). Similarly, C.X/ ?W C.Z/ if and only if X0 W Z D 0 (or equivalently if and only if Z0 W X D 0). The restriction in Lemma 14.12.1 that W be positive definite can be relaxed. The lemma is valid for any m m symmetric nonnegative definite matrix W , as can be easily verified. b. Approach To extend (to inner products other than the usual inner product) the results presented in Chapter 12 on the projection of a column vector, we could proceed in either of two ways. One approach is to generalize and (when necessary) modify the derivations given in Chapter 12. An alternative approach, which is sometimes employed here, uses the following lemma to take advantage of the results already established (in Chapter 12) in the special case of the usual inner product. Lemma 14.12.2. Let L represent an arbitrary matrix of dimensions m n, and let U represent an arbitrary subspace of Rn1. Further, define V D fv 2 Rm W v D Lx for some x 2 Ug; define W D L0 L, and let X represent an n p matrix whose columns span U. Then, (1) the set V is a subspace of Rm ; (2) two vectors x and y in Rn are orthogonal with respect to W if and only if Lx and Ly are orthogonal with respect to the usual inner product (for Rm ); (3) a vector x (in Rn ) and the subspace U are orthogonal with respect to W if and only if Lx and V are orthogonal with respect to the usual inner product (for Rm ); (4) the subspace U and a subspace M (of Rn1 ) are orthogonal with respect to W if and only if V and the subspace N D fz 2 Rm W z D Ly for some y 2 Mg are orthogonal with respect to the usual inner product (for Rm ); (5) the columns of LX span V. Proof. (1) By definition, V is a subset of Rm . Thus, it suffices to show that V is a linear space. Let v and w represent arbitrary vectors in V, and let k be an arbitrary scalar. Then, v D Lx and w D Ly for some vectors x and y in U, and (since U is a linear space) x C y 2 U and kx 2 U. It follows that v C w D L.x C y/ 2 V and kv D L.kx/ 2 V and hence that V is a linear space. (2) Observe that x0 W y D .Lx/0 Ly, and hence that x0 W y D 0 if and only if .Lx/0 Ly D 0. (3) In light of Part (2), x and U are orthogonal with respect to W , or, equivalently, x and y are orthogonal (with respect to W ) for every y in U, if and only if Lx and Ly are orthogonal (with respect to the usual inner product) for every y

262


in U. Thus, x and U are orthogonal (with respect to W ) if and only if Lx and v are orthogonal (with respect to the usual inner product) for every v in V, or, equivalently, if and only if Lx and V are orthogonal (with respect to the usual inner product). (4) In light of Part (2), U and M are orthogonal with respect to W , or, equivalently, x and y are orthogonal (with respect to W ) for every x in U and every y in M, if and only if Lx and Ly are orthogonal (with respect to the usual inner product) for every x in U and every y in M. Thus, U and M are orthogonal (with respect to W ) if and only if v and z are orthogonal (with respect to the usual inner product) for every v in V and every z in N, or, equivalently, if and only if V and N are orthogonal (with respect to the usual inner product). (5) Let v represent an arbitrary vector in V. Then, v D Lx for some x 2 U. Moreover, x D Xr for some p-dimensional column vector r. Thus, v D LXr, so that v is expressible as a linear combination of the columns of LX. We conclude that the columns of LX span V. Q.E.D. c. Projections Let W represent an n n symmetric positive definite matrix, let y represent an n-dimensional column vector, and let U represent a subspace of Rn1. Define z to be the projection of y on U when the inner product (for Rn1 ) is taken to be the bilinear form x0 W y. Let us refer to this projection as the orthogonal projection of y on U with respect to W , or simply as the projection of y on U with respect to W. Formula (12.2.1) gives the projection of y on U with respect to the usual inner product (i.e., with respect to I) in terms of an arbitrary solution to the normal equations (12.2.2). For purposes of generalizing this formula, let L represent a nonsingular matrix such that W D L0 L, define V D fv 2 Rn W v D Lx for some x 2 Ug; and take X to be any n p matrix whose columns span U. Let b represent any solution to the linear system X0 W Xb D X0 W y (in b): (The existence of such a solution follows from Theorem 14.11.1.) Note that this linear system can be rewritten as .LX/0 LXb D .LX/0 Ly: By definition, z is the (unique) vector in U such that y z ?W U. Thus, it follows from Part (3) of Lemma 14.12.2 that L.y z/, or, equivalently, Ly Lz, is orthogonal to V with respect to the usual inner product, implying that Lz is the projection of Ly on V with respect to the usual inner product. Since [according to Part (5) of Lemma 14.12.2] the columns of LX span V, we have, as a consequence


263

of Theorem 12.2.1, that Lz D LXb and hence (since L is nonsingular) that z D Xb . In summary, we have the following generalization of Theorem 12.2.1. Theorem 14.12.3. Let z represent the projection of an n-dimensional column vector y on a subspace U (of Rn ) with respect to a symmetric positive definite matrix W , and let X represent any n p matrix whose columns span U. Then, z D Xb for any solution b to the (consistent) linear system X0 W Xb D X0 W y (in b): The following three corollaries (of Theorem 14.12.3) generalize Corollaries 12.2.2–12.2.4, respectively. Corollary 14.12.4. Let z represent the projection of an n-dimensional column vector y on a subspace U (of Rn ) with respect to a symmetric positive definite matrix W , and let X represent any n p matrix whose columns span U. Then, z D X.X0 W X/ X0 W y: Corollary 14.12.5. Let y represent an n-dimensional column vector, X an n p matrix, and U any n q matrix such that C.U / D C.X/. Then, U a D Xb for any solution a to the linear system U 0 W U a D U 0 W y (in a) and any solution b to the linear system X0 W Xb D X0 W y (in b). Corollary 14.12.6. Let y represent an n-dimensional column vector, and X an n p matrix. Then, Xb1 D Xb2 for any two solutions b1 and b2 to the linear system X0 W Xb D X0 W y (in b). The following theorem generalizes Theorem 12.2.5. Theorem 14.12.7. Let z represent the projection of an n-dimensional column vector y on a subspace U (of Rn ) with respect to a symmetric positive definite matrix W , and let X represent any np matrix whose columns span U. Then, any p 1 vector b such that z D Xb is a solution to the linear system X0 W Xb D X0 W y (in b). Proof. As a consequence of Theorem 14.12.3, X0 W Xb D X0 W y has a solution, say a, and z D Xa. Thus, X0 W Xb D X0 W z D X0 W Xa D X0 W y: Q.E.D.

264


d. Projection matrices Subsequently, for any n p matrix X and for any n n matrix W , the symbol PX;W is used to represent the n n matrix X.X0 W X/ X0 W . Note that PX;I D X.X0 X/ X0 D PX . Note also that if W is symmetric and positive definite, then PX;W is invariant to the choice of the generalized inverse .X0 W X/ as (in light of Corollary 14.11.3) is evident from Theorem 9.4.1 and as is implicit in the following theorem. Theorem 14.12.8. Let U represent any subspace of the linear space Rn of all n-dimensional column vectors, and let W represent any n n symmetric positive definite matrix. Then, there exists a unique matrix A (of dimensions nn) such that, for every y in Rn , Ay is the projection of y on U with respect to W . Moreover, A D PX;W for any matrix X such that C.X/ D U. Theorem 14.12.8 generalizes Theorem 12.3.1 and can be proved in essentially the same way as Theorem 12.3.1. Let us refer to the matrix A (from Theorem 14.12.8) as the orthogonal projection matrix for U with respect to W or simply as the projection matrix for U with respect to W . And let us speak of an n n matrix as a projection matrix with respect to W when we wish to indicate that there exists some subspace of Rn for which that matrix is the projection matrix with respect to W . Moreover, for consistency with our previous terminology, let us refer to the projection matrix for U with respect to I simply as the projection matrix for U and to a projection matrix with respect to I simply as a projection matrix. The following two corollaries (of Theorem 14.12.8) generalize Corollaries 12.3.2 and 12.3.3. Corollary 14.12.9. Let A represent the projection matrix for a subspace U (of Rn1 ) with respect to an n n symmetric positive definite matrix W . Then, A D PX;W for any matrix X such that C.X/ D U. Corollary 14.12.10. An n n matrix A is a projection matrix with respect to an n n symmetric positive definite matrix W if and only if A D PX;W for some matrix X. The basic properties of projection matrices encompassed in Theorems 12.3.4 and 12.3.5 are generalized in the following two theorems. Theorem 14.12.11. Let X represent any n p matrix, and W any n n symmetric positive definite matrix. Then, (1) PX;W X D X; that is, X.X0 W X/ X0 W X D X; implying that W PX;W X D W X; (2) PX;W D XB for any solution B to the linear system X0 W XB D X0 W (in B); (3) W PX;W is symmetric; that is, W XŒ.X0 W X/ 0 X0 W D W X.X0 W X/ X0 W ; (30 ) PX;WW 1 is symmetric; that is, XŒ.X0 W X/ 0 X0 D X.X0 W X/ X0 ;


265

0 (4) XŒ.X0 W X/ 0 X0 W X D X; implying that PX;W W X D W X; 0 D X0 ; (5) X0 W X.X0 W X/ X0 D X0 W XŒ.X0 W X/ 0 X0 D X0 PX;W 0 0 implying that X W PX;W D X W ; 2 D PX;W ; that is, PX;W is idempotent; (6) PX;W 0 W PX;W D W PX;W ; (60 ) PX;W

(7) C.PX;W / D C.X/, and R.PX;W / D R.X0 W /; (8) rank.PX;W / D rank.X/; (9) .I PX;W /2 D I PX;W ; that is, I PX;W is idempotent; (90 ) .I PX;W /0 W .I PX;W / D W .I PX;W /; (10) rank.I PX;W / D n rank.X/. Proof. According to Corollary 14.3.13, there exists a nonsingular matrix L such that W D L0 L. (1) Making use of Part (1) of Theorem 12.3.4, we find that LX.X0 W X/ X0 W X D LXŒ.LX/0 LX .LX/0 LX D LX: Since L is nonsingular, we conclude that X.X0 W X/ X0 W X D X. (2) If B is a solution to X0 W XB D X0 W , then (according to Theorem 11.2.4) B D .X0 W X/ X0 W C ŒI .X0 W X/ X0 W XY for some matrix Y , implying [in light of Part (1)] that XB D PX;W C ŒX X.X0 W X/ X0 W XY D PX;W : (3) and (30 ). It follows from Part (3) of Theorem 12.3.4 that LXŒ.X0 W X/ 0 X0 L0 D LXfŒ.LX/0 LX g0 .LX/0 D LXŒ.LX/0 LX .LX/0 D LX.X0 W X/ X0 L0 : Thus, W XŒ.X0 W X/ 0 X0 W D L0 LXŒ.X0 W X/ 0 X0 L0 L D L0 LX.X0 W X/ X0 L0 L D W X.X0 W X/ X0 W ; and XŒ.X0 W X/ 0 X0 D L1 LXŒ.X0 W X/ 0 X0 L0 .L0 /1 D L1 LX.X0 W X/ X0 L0 .L0 /1 D X.X0 W X/ X0 : (4) Making use of Parts (3) and (1), we find that XŒ.X0 W X/ 0 X0 W X D W 1 fW XŒ.X0 W X/ 0 X0 W gX D W 1 ŒW X.X0 W X/ X0 W X D X.X0 W X/ X0 W X D X:

266


(5) It follows from Part (4) that X0 D fXŒ.X0 W X/ 0 X0 W Xg0 D X0 W X.X0 W X/ X0 and from Part (1) that 0 X0 D .PX;W X/0 D X0 PX;W D X0 W XŒ.X0 W X/ 0 X0 :

(6) Making use of Part (5), we find that 2 PX;W D X.X0 W X/ X0 W PX;W D X.X0 W X/ X0 W D PX;W :

(60 ) Making use of Part (4), we find that 0 0 PX;W W PX;W D PX;W W X.X0 W X/ X0 W

D W X.X0 W X/ X0 W D W PX;W : (7) Recalling Corollary 4.2.3, it is clear from the definition of PX;W that C.PX;W / C.X/ and from Part (1) that C.X/ C.PX;W /, so that C.PX;W / D C.X/. Similarly, it is clear from the definition of PX;W that R.PX;W / R.X0 W / and from Part (5) that R.X0 W / R.PX;W /, so that R.PX;W / D R.X0 W /. (8) That rank.PX;W / D rank.X/ is an immediate consequence of Part (7). (9) Since [according to Part (6)] PX;W is idempotent, it follows from Lemma 10.1.2 that I PX;W is idempotent. (90 ) Making use of Parts (3) and (60 ), we find that 0 .I PX;W /0 W .I PX;W / D W .W PX;W /0 W PX;W C PX;W W PX;W

D W W PX;W W PX;W C W PX;W D W .I PX;W /: (10) Making use of Lemma 10.2.4 and Parts (6) and (8), we find that rank.I PX;W / D n rank.PX;W / D n rank.X/: Q.E.D. Theorem 14.12.12. Let X represent an n p matrix, U an n q matrix such that C.U / C.X/, and W an n n symmetric positive definite matrix. Then, (1) PX;W U D U , implying that W PX;W U D W U , and U 0 W X.X0 W X/ X0 D U 0 , implying that U 0 W PX;W D U 0 W ; 0 (2) PX;W PU;W D PU;W PX;W D PU;W , and PX;W W PU;W D W PU;W .

Proof. (1) According to Lemma 4.2.2, there exists a matrix F such that U D XF . Thus, making use of Parts (1) and (5) of Theorem 14.12.11, we find that PX;W U D PX;W XF D XF D U


267

and U 0 W X.X0 W X/ X0 D F 0 X0 W X.X0 W X/ X0 D F 0 X0 D U 0 : (2) Making use of Part (1), we find that PX;W PU;W D PX;W U .U 0 W U / U 0 W D U .U 0 W U / U 0 W D PU;W and similarly that PU;W PX;W D U .U 0 W U / U 0 W X.X0 W X/ X0 W D U .U 0 W U / U 0 W D PU;W : Further, making use of Part (3) of Theorem 14.12.11, we find that 0 W PU;W D .W PX;W /0 PU;W D W PX;W PU;W D W PU;W : PX;W

Q.E.D. The following two corollaries generalize Corollaries 12.3.6 and 12.3.7 and can be deduced from Theorem 14.12.12 in essentially the same way that Corollaries 12.3.6 and 12.3.7 were deduced from Theorem 12.3.5. Corollary 14.12.13. Let W represent an n n symmetric positive definite matrix. Then, for any n p matrix X and any n q matrix U such that C.U / D C.X/, PU;W D PX;W . Corollary 14.12.14. Let W represent an n n symmetric positive definite matrix and X an n p matrix. Then, for any matrix A that is the projection matrix for some subspace U of C.X/ with respect to W , PX;W A D APX;W D A: The following theorem generalizes Theorem 12.3.8 and can be derived in essentially the same way as Theorem 12.3.8. Theorem 14.12.15. If an n n matrix A is the projection matrix for some subspace U of Rn1 with respect to some nn symmetric positive definite matrix, then U D C.A/ and dim.U/ D rank.A/. The following theorem characterizes matrices that are projection matrices with respect to any particular symmetric positive definite matrix. Theorem 14.12.16. An n n matrix A is a projection matrix with respect to an n n symmetric positive definite matrix W if and only if A0 W A D W A or, equivalently, if and only if .I A/0 W A D 0. Proof. Suppose that A is a projection matrix with respect to W . Then, according to Corollary 14.12.10, A D PX;W for some matrix X. Thus, it follows from Part (60 ) of Theorem 14.12.11 that A0 W A D W A. Conversely, suppose that A0 W A D W A or, equivalently, that A0 W D .W A/0 D .A0 W A/0 D A0 W A:

268


Then, making use of Part (1) of Theorem 14.12.11, we find that A D PA;W C A A.A0 W A/ A0 W D PA;W C A A.A0 W A/ A0 W A D PA;W C A A D PA;W ; which is the projection matrix for C.A/ with respect to W . Q.E.D. The following corollary, which generalizes Theorem 12.3.9, gives an alternative characterization of matrices that are projection matrices with respect to any particular symmetric positive definite matrix. Corollary 14.12.17. An n n matrix A is a projection matrix with respect to an n n symmetric positive definite matrix W if and only if W A is symmetric and A is idempotent. Proof. In light of Theorem 14.12.16, it suffices to show that A0 W A D W A if and only if W A is symmetric and A is idempotent or, equivalently (since W is nonsingular), if and only if W A is symmetric and W A2 D W A. If W A is symmetric and W A2 D W A, then A0 W A D .W A/0 A D W AA D W A2 D W A: Conversely, if A0 W A D W A, then .W A/0 D .A0 W A/0 D A0 W A D W A (i.e., W A is symmetric), and W A2 D W AA D .W A/0 A D A0 W A D W A: Q.E.D. e. When are projections with respect to a symmetric positive definite matrix W projections with respect to another symmetric positive definite matrix V? Suppose that U is a subspace of the linear space Rn of all n-dimensional column vectors. Let X represent an n p matrix whose columns span U, and let W and V represent n n symmetric positive definite matrices. Under what circumstances is the projection PX;W y of y on U with respect to W the same (for every y in Rn ) as the projection PX;V y of y on U with respect to V ? Or, equivalently (in light of Lemma 2.3.2), under what circumstances is PX;W D PX;V ; that is, under what circumstances is PX;W the projection matrix for U with respect to V as well as the projection matrix for U with respect to W ? These circumstances can be characterized in terms of the conditions given by the following theorem.


269

Theorem 14.12.18. Let U represent a subspace of Rn1, let X represent an n p matrix whose columns span U, and let W and V represent n n symmetric positive definite matrices. Then, each of the following four conditions is necessary and sufficient for the projection PX;W y of y on U with respect to W to be the same (for every y in Rn ) as the projection PX;V y of y on U with respect to V (or, equivalently, for PX;W D PX;V ): (1) X0 V PX;W D X0 V , or, equivalently, X0 V .I PX;W / D 0; 0 V PX;W D V PX;W , or, equivalently, .I PX;W /0 V PX;W D 0; (2) PX;W

(3) V PX;W is symmetric; (4) there exists a p p matrix Q such that V X D W XQ, or, equivalently, C.V X/ C.W X/. Proof. (1) It follows from Lemma 14.12.1 that the projection PX;W y of y on U with respect to W is the same (for every y in Rn ) as the projection of y on U with respect to V if and only if, for every y, X0 V .y PX;W y/ D 0; or, equivalently, if and only if, for every y, X0 V .I PX;W /y D 0; and hence if and only if X0 V .I PX;W / D 0: (2) It suffices to show that Condition (2) is equivalent to Condition (1). If Condition (1) is satisfied, then 0 PX;W V PX;W D .X0V PX;W /0 .X0 W X/ X0 W

D .X0 V /0 .X0 W X/ X0 W D V PX;W : Conversely, if Condition (2) is satisfied, then, making use of Part (5) of Theorem 14.12.11, we find that 0 X0 V D X0 PX;W V D X0 .V PX;W /0 0 0 D X0 .PX;W V PX;W /0 D X0 PX;W V PX;W D X0 V PX;W :

(3) It suffices to show that Condition (3) is equivalent to Condition (2). If Condition (2) is satisfied, then 0 0 .V PX;W /0 D .PX;W V PX;W /0 D PX;W V PX;W D V PX;W ;

that is, V PX;W is symmetric. Conversely, if Condition (3) is satisfied, then [in light of Part (6) of Theorem 14.12.11] 0 PX;W V PX;W D .V PX;W /0 PX;W D V PX;W PX;W D V PX;W :

270


(4) It suffices to show that Condition (4) is equivalent to Condition (1) or, 0 V X D .X0 V PX;W /0 ], to the condition equivalently [since V X D .X0 V /0 and PX;W 0 V X D PX;W V X:

(12.1)

If Condition (12.1) is satisfied, then V X D W XŒ.X0 W X/ 0 X0 V X D W XQ for Q D Œ.X0 W X/ 0 X0 V X. Conversely, if V X D W XQ for some matrix Q, then, making use of Part (4) of Theorem 14.12.11, we find that 0 0 V X D PX;W W XQ D W XQ D V XI PX;W

that is, condition (12.1) is satisifed. Q.E.D. Condition (4) of Theorem 14.12.18 has appeal relative to Conditions (1)–(3) in that it does not explicitly involve PX;W . In the special case where W D I, PX;W reduces to the usual projection matrix PX D X.X0 X/ X0 , and (in this special case) the four conditions of Theorem 14.12.18 reduce, respectively, to (1) (2) (3) (4)

X0 V PX D X0 V , or, equivalently, X0 V .I PX / D 0; PXV PX D V PX , or, equivalently, .I PX /V PX D 0; V PX is symmetric; there exists a p p matrix Q such that V X D XQ, or, equivalently, C.V X/ C.X/.

f. Generalized or weighted least squares problem Suppose that y D fyi g is an n-dimensional column vector and that U is a subspace of the linear space Rn of all n-dimensional column vectors. Let W D fwij g represent an n n symmetric positive definite matrix. Consider the problem of minimizing, for u D fui g in U, the quadratic form .y u/0 W .y u/ (in the difference y u) or, equivalently, the quantity Œ.y u/0 W .y u/1=2 , which is the distance between y and u when the inner product is taken to be the bilinear form x0 W y. This minimization problem is known as the generalized or weighted least squares problem. [Typically, the use of the term weighted least squares is reserved for the special Pcase where W is a diagonal matrix—in this special case, .y u/0 W .y u/ D niD1 wi i .yi ui /2 , which is a weighted sum of the squared deviations of the elements of u from the corresponding elements of y (with weights w11 , w22 , : : : ; wnn , respectively).] In the special case where W D I, the generalized or weighted least squares problem reduces to the least squares problem considered in Section 12.4. It is customary (and convenient) to refer to this special case as simple least squares, ordinary least squares, or unweighted least squares. The solution of the generalized least squares problem, like that of the simple least squares problem, can be obtained as a special case of Theorem 12.4.1. Since


271

Œ.y u/0 W .y u/1=2 is the distance between y and u (when the inner product is taken to be the bilinear form x0 W y), Theorem 12.4.1 implies that Œ.y u/0 W .y u/1=2 is minimized uniquely by taking u to be the projection of y on U with respect to W . Thus, based on Theorem 14.12.3, we arrive at the following generalization of Theorem 12.4.2. Theorem 14.12.19. Let U represent a subspace of the linear space Rn of all n-dimensional column vectors, let X be an n p matrix such that C.X/ D U, let W be an n n symmetric positive definite matrix, and let y represent a vector in Rn . Then, for u 2 U, the quadratic form .y u/0 W .y u/ (in the difference y u) is minimized uniquely by taking u D Xb, where b is any solution to the linear system X0 W Xb D X0 W y (in b), or, equivalently, by taking u D PX;W y. Furthermore, the minimum value of .y u/0 W .y u/ (for u 2 U) is expressible as .y Xb /0 W .y Xb / D y 0 W .y Xb / D y 0 W .I PX;W /y: In light of Theorem 14.12.7, we have the following variant of Theorem 14.12.19, which generalizes Theorem 12.4.3. Theorem 14.12.20. Let X represent an n p matrix, W an n n symmetric positive definite matrix, and y an n-dimensional column vector. Then, for b 2 Rp, the quadratic form .y Xb/0 W .y Xb/ has a minimum at a point b if and only if b is a solution to the linear system X0 W Xb D X0 W y, in which case Xb D PX;W y and .y Xb /0 W .y Xb / D y 0 W .y Xb / D y 0 W .I PX;W /y: The results encompassed in Theorems 14.12.19 and 14.12.20 are generally attributed to Aitken (1935). In fact, the p equations that compose the linear system X0 W Xb D X0 W y have come to be known as the Aitken equations. g. Orthogonal complements of row and column spaces Let X represent an n p matrix, and take W to be an n n symmetric positive definite matrix. In Section 12.5, consideration was given to C? .X/ and R? .X/, which are the orthogonal complements of C.X/ and R.X/, respectively, relative to the linear space Rn (of all n-dimensional column or row vectors) when the inner product is taken to be the usual inner product for Rn . Let us now consider the orthogonal complements of C.X/ and R.X/ relative to Rn when the inner product is taken to be the bilinear form x0 W y. Let us denote these orthogonal ? .X/ and R? complements by CW W .X/, respectively, and speak of them as the orthogonal complements of C.X/ and R.X/ with respect to W . Note that CI? .X/ D ? C? .X/ and R? I .X/ D R .X/. Since [according to Part (5) of Theorem 14.12.11] X.X0 W X/ is a generalized inverse of X0 W , we have (in light of Lemma 14.12.1 and Corollary 11.2.2) the following generalization of Lemma 12.5.2.

272


Lemma 14.12.21. For any n p matrix X and any n n symmetric positive definite matrix W , ? .X/ D N.X0 W / D C.I PX;W /: CW

(12.2)

As a consequence of Lemma 11.3.1 or, alternatively, Theorem 14.12.11, we have the following corollary, which is a generalization of Corollary 12.5.3. Corollary 14.12.22. For any np matrix X and any nn symmetric positive definite matrix W , ? dimŒCW .X/ D n rank.X/ D n dimŒC.X/:

(12.3)

h. Projection on the orthogonal complement of a column space In light of Theorems 12.5.8 and 14.12.3, we have the following theorem and corollary, which are generalizations of Corollaries 12.5.9 and 12.5.10. Theorem 14.12.23. Let X represent an n p matrix, W an n n symmetric positive definite matrix, and y an n-dimensional column vector, and take z to be ? .X/ with respect to W . Then, the projection of y on CW z D y Xb for any solution b to the linear system X0 W Xb D X0 W y (in b). In particular, z D y PX;W y D .I PX;W /y: Corollary 14.12.24. Let X represent an np matrix and W an nn symmetric ? .X/ with respect to W positive definite matrix. Then, the projection matrix for CW is I PX;W . i. Extensions Let W represent an n n symmetric nonnegative definite matrix, and let U represent a subspace of Rn1. In what follows, a vector z in U is said to be a projection of an n-dimensional column vector y on U with respect to W if .y z/ ?W U; and an n n matrix A is said to be a projection matrix for U with respect to W if, for every y in Rn1, Ay is a projection of y on U with respect to W . (To say simply that an n n matrix A is a projection matrix with respect to W is to indicate that there exists some subspace of Rn1 for which A is a projection matrix with respect to W .) Our previous use of this terminology was restricted to the case where W is positive definite. Recall that, for any n p matrix X and any n n matrix W , PX;W D X.X0 W X/ X0 W . Theorem 14.12.11 set forth some basic properties of the matrix PX;W in the special case where W is symmetric and positive definite. The following theorem gives generalizations of those properties for the less restrictive special case where W is symmetric and nonnegative definite.


273

Theorem 14.12.25. Let X represent any n p matrix, and W any n n symmetric nonnegative definite matrix. Then, (1) W PX;W X D W X; that is, W X.X0 W X/ X0 W X D W X; (2) W PX;W D W XB for any solution B to the linear system X0 W XB D X0 W (in B); (3) W PX;W is symmetric; that is, W XŒ.X0 W X/ 0 X0 W D W X.X0 W X/ X0 W ; 0 (4) PX;W W X D W X; that is, W XŒ.X0 W X/ 0 X0 W X D W X; 0 W D X0 W ; (5) X0 W PX;W D X0 PX;W 0 0 that is, X W X.X W X/ X0 W D X0 W XŒ.X0 W X/ 0 X0 W D X0 W ; 2 (6) PX;W D PX;W ; that is, PX;W is idempotent; 0 W PX;W D W PX;W ; (60 ) PX;W

(7) C.W PX;W / D C.W X/, and R.PX;W / D R.X0 W /; (8) rank.PX;W / D rank.W PX;W / D rank.W X/; (9) .I PX;W /2 D I PX;W ; that is, I PX;W is idempotent; (90 ) .I PX;W /0 W .I PX;W / D W .I PX;W /; (10) rank.I PX;W / D n rank.W X/. The various parts of Theorem 14.12.25 can be established via arguments that are essentially the same as the arguments employed in establishing the corresponding parts of Theorem 14.12.11 or via simple variants of those arguments. The following theorem extends the result (on projections) given by Theorem 14.12.3. Theorem 14.12.26. Let y represent an n-dimensional column vector, let U represent a subspace of Rn1, let W represent an n n symmetric nonnegative definite matrix, and let X represent any n p matrix whose columns span U. Then, (1) there exists a projection of y on U with respect to W ; (2) W z has the same value for every vector z that is a projection of y on U with respect to W ; and (3) a vector z is a projection of y on U with respect to W if and only if z D Xb for some solution b to the (consistent) linear system X0 W Xb D X0 W y (in b):

(12.4)

Proof. Let L represent a matrix such that W D L0 L, denote by m the number of rows in L, define V D fv 2 Rm W v D Lx for some x 2 Ug; and let v represent the (unique) projection of Ly on V (with respect to I). Then, it follows from Part (3) of Lemma 14.12.2 that a vector z in U is a projection of y on U with respect to W if and only if L.y z/ (or, equivalently, Ly Lz) is orthogonal to V with respect to the usual inner product and hence if and only if

274


Lz D v :

(12.5)

Let b represent any solution to linear system (12.4). (The existence of such a solution follows from Theorem 14.11.1.) According to Part (5) of Lemma 14.12.2, the columns of LX span V. Moreover, linear system (12.4) can be rewritten as .LX/0 LXb D .LX/0 Ly: Thus, it follows from Theorem 12.2.1 that v D LXb:

(12.6)

Clearly, Xb is in U. Moreover, it follows from result (12.6) that condition (12.5) can be satisfied by taking z D Xb. Thus, Xb is a projection of y on U with respect to W . Furthermore, since a vector z is a projection of y on U with respect to W only if z satisfies condition (12.5), W z (which equals L0 Lz) has the same value, namely L0 v , for every vector z that is a projection of y on U with respect to W . Suppose now that z is a projection of y on U with respect to W . Then, z D Xa for some vector a. Further, W Xa D W z D L0 v D L0 LXb D W Xb; implying that X0 W Xa D X0 W Xb D X0 W y: Thus, a is a solution to linear system (12.4). Q.E.D. The following three corollaries extend the results of Corollaries 14.12.4– 14.12.6. Corollary 14.12.27. Let y represent an n-dimensional column vector, let U represent a subspace of Rn1, let W represent an n n symmetric nonnegative definite matrix, and let X represent any n p matrix whose columns span U. Then, a vector z is a projection of y on U with respect to W if and only if z D PX;W y C .I PX;W /Xk for some p 1 vector k. Proof. Corollary 14.12.27 follows from Part (3) of Theorem 14.12.26 upon observing that (for any k) PX;W y C .I PX;W /Xk D Xf.X0 W X/ X0 W y C ŒI .X0 W X/ X0 W Xkg and that (in light of Theorem 11.2.4) a vector b is a solution to linear system (12.4) if and only if b D .X0 W X/ X0 W y C ŒI .X0 W X/ X0 W Xk for some k.

Q.E.D.


275

Corollary 14.12.28. Let y represent an n-dimensional column vector, W an n n symmetric nonnegative definite matrix, X an n p matrix, and U an n q matrix such that C.U / D C.X/. Then, W U a D W Xb for any solution a to the linear system U 0 W U a D U 0 W y (in a) and any solution b to the linear system X0 W Xb D X0 W y (in b). Corollary 14.12.29. Let y represent an n-dimensional column vector, X an n p matrix, and W an n n symmetric nonnegative definite matrix. Then, W Xb1 D W Xb2 for any two solutions b1 and b2 to the linear system X0 W Xb D X0 W y (in b). Note that, for any vector y in a subspace U of Rn1 and for any symmetric nonnegative definite matrix W, y itself is a projection of y on U with respect to W (as is evident from the very definition of a projection of y on U with respect to W ). The following theorem generalizes Theorem 14.12.7. Theorem 14.12.30. Let y represent an n-dimensional column vector, let U represent a subspace of Rn1 , let W represent an n n symmetric nonnegative definite matrix, and let X represent any n p matrix whose columns span U. Then, any p 1 vector b such that Xb is a projection of y on U with respect to W is a solution to the linear system X0 W Xb D X0 W y (in b). Proof. According to Theorem 14.12.26, Xb D Xa for some solution a to 0 X W Xb D X0 W y. Thus, X0 W Xb D X0 W Xa D X0 W y: Q.E.D. Corollary 14.12.27 leads to the following lemma, which extends the result of Corollary 14.12.9. Lemma 14.12.31. Let U represent a subspace of Rn1, and let W represent an n n symmetric nonnegative definite matrix. Then, for any matrix X such that C.X/ D U, PX;W is a projection matrix for U with respect to W . Thus, PX;W is a projection matrix for U ŒD C.X/ with respect to W when W is positive semidefinite as well as when W is positive definite. However, when W is positive semidefinite, there may be more than one projection matrix for U with respect to W . The following theorem, whose proof parallels that of Theorem 12.4.1, provides a basis for extending the results of Theorem 14.12.19. Theorem 14.12.32. Let U represent a subspace of the linear space Rn of all n-dimensional column vectors, take W to be an n n symmetric nonnegative definite matrix, and let y represent a vector in Rn. Then, for u 2 U, the quadratic form .y u/0 W .y u/ (in the difference y u) attains its minimum value at u D z if and only if z is a projection of y on U with respect to W , in which case

276


.y z/0 W .y z/ D y 0 W .y z/: Proof. Suppose that z is a projection of y on U with respect to W . Then, for any vector u in U, .y u/0 W .y u/ D Œy z .u z/0 W Œy z .u z/ D .y z/0 W .y z/ 2.y z/0 W .u z/ C .u z/0 W .u z/: Moreover, u z is in U and, by definition, y z is orthogonal to every vector in U. Thus, .y z/0 W .u z/ D 0, and hence .y u/0 W .y u/ D .y z/0 W .y z/ C .u z/0 W .u z/ .y z/0 W .y z/;

(12.7)

which implies that .y u/0 W .y u/ attains its minimum value at u D z. Furthermore, since z is in U, we have that z0 W .y z/ D 0, with the consequence that .y z/0 W .y z/ D y 0 W .y z/: To complete the proof, it suffices to show that equality is attained in inequality (12.7), or, equivalently, that .u z/0 W .u z/ D 0, only if u is a projection of y on U with respect to W . If .u z/0 W .u z/ D 0, then we find (in light of Corollary 14.3.11) that W .u z/ D 0, or, equivalently, that W u D W z, implying that, for every vector x in U, x0 W .y u/ D x0 W .y z/ D 0 and hence that u, like z, is a projection of y on U with respect to W . Q.E.D. The following theorem, which extends the results of Theorem 14.12.19, is obtained by combining the results of Theorem 14.12.32 with those of Theorem 14.12.26 and Corollary 14.12.27. Theorem 14.12.33. Let U represent a subspace of the linear space Rn of all n-dimensional column vectors, take X to be an n p matrix such that C.X/ D U, take W to be an n n symmetric nonnegative definite matrix, and let y represent a vector in Rn. Then, for u 2 U, the quadratic form .y u/0 W .y u/ (in the difference y u) has a minimum at a point u if and only if u D Xb for some solution b to the linear system X0 W Xb D X0 W y (in b) or, equivalently, if and only if u D PX;W y C .I PX;W /Xk for some p 1 vector k. Furthermore, the minimum value of .y u/0 W .y u/ (for u 2 U) is expressible as .y Xb /0 W .y Xb / D y 0 W .y Xb / D y 0 W .I PX;W /y (where b is any solution to X0 W Xb D X0 W y). In light of Theorem 14.12.30, we have the following variant of Theorem 14.12.33, which generalizes Theorem 14.12.20.

Exercises

277

Theorem 14.12.34. Let X represent an n p matrix, W an n n symmetric nonnegative definite matrix, and y an n-dimensional column vector. Then, for b 2 Rp, the quadratic form .y Xb/0 W .y Xb/ has a minimum at a point b if and only if b is a solution to the linear system X0 W Xb D X0 W y (in b), in which case Xb D PX;W y C .I PX;W /Xk for some p 1 vector k and .y Xb /0 W .y Xb / D y 0 W .y Xb / D y 0 W .I PX;W /y:

Exercises Section 14.1 1. Show that a symmetric bilinear form x0 Ay (in n-dimensional vectors x and y) can be expressed in terms of the corresponding quadratic form, that is, the quadratic form whose matrix is A. Do so by verifying that x0 Ay D .1=2/Œ.x C y/0 A.x C y/ x0 Ax y 0 Ay: 2. Show that corresponding to any quadratic form x0 Ax (in the n-dimensional vector x) there exists a unique upper triangular matrix B such that x0 Ax and x0 Bx are identically equal, and express the elements of B in terms of the elements of A. Section 14.2 3. Show, by example, that the sum of two positive semidefinite matrices can be positive definite. 4. Show, via an example, that there exist (nonsymmetric) nonsingular positive semidefinite matrices. 5. Show, by example, that there exist an n n positive semidefinite matrix A and an n m matrix P (where m < n) such that P 0 AP is positive definite. 6. Convert results (1)–(3) of Theorem 14.2.9, which are for nonnegative definite (positive definite or positive semidefinite) matrices, into equivalent results for nonpositive definite matrices. 7. Let fX1 ; : : : ; Xr g represent a set of matrices from a linear space V, and let A D faij g represent the r r matrix whose ij th element is Xi Xj —this matrix is referred to as the Gram matrix (or the Gramian) of the set fX1 ; : : : ; Xr g and its determinant is referred to as the Gramian (or the Gram determinant) of fX1 ; : : : ; Xr g.

278


(a) Show that A is symmetric and nonnegative definite. (b) Show that X1 ; : : : ; Xr are linearly independent if and only if A is nonsingular. 8. Let A D faij g represent an n n symmetric positive definite matrix, and let B D fbij g D A1. Show that, for i D 1; : : : ; n, bi i 1=ai i ; with equality holding if and only if, for all j ¤ i , aij D 0. Section 14.3 9. Let A represent an m n matrix and D a diagonal matrix such that A D P DQ for some matrix P of full column rank and some matrix Q of full row rank. Extend the result of Lemma 14.3.1 by showing that rank.A/ equals the number of nonzero diagonal elements in D. 10. Let A represent an n n symmetric idempotent matrix and V an n n symmetric positive definite matrix. Show that rank.AV A/ D tr.A/. 11. Show that if an n n matrix A is such that x0 Ax ¤ 0 for every n 1 nonnull vector x, then A is either positive definite or negative definite. 12. (a) Let A represent an n n symmetric matrix of rank r. Take P to be an n n nonsingular matrix and D an nn diagonal matrix such that A D P 0 DP — the existence of such matrices is guaranteed by Corollary 14.3.5. The number, say m, of diagonal elements of D that are positive is called the index of inertia of A (or of the quadratic form x0 Ax whose matrix is A). Show that the index of inertia is well-defined in the sense that m does not vary with the choice of P or D. That is, show that if P1 and P2 are nonsingular matrices and D1 and D2 are diagonal matrices such that A D P10 D1 P1 D P20 D2 P2 , then D2 contains the same number of positive diagonal elements as D1 . Show also that the number of diagonal elements of D that are negative equals r m. (b) Let A represent an n n symmetric matrix. Show that A D P 0 diag.Im , Irm ; 0/P for some n n nonsingular matrix P and some nonnegative integers m and r. Show further that m equals the index of inertia of the matrix A and that r D rank.A/. (c) An nn symmetric matrix B is said to be congruent to an nn symmetric matrix A if there exists an nn nonsingular matrix P such that B D P 0 AP . (If B is congruent to A, then clearly A is congruent to B.) Show that B is congruent to A if and only if B has the same rank and the same index of inertia as A. This result is called Sylvester’s law of inertia, after James Joseph Sylvester (1814–1897).

Exercises

279

(d) Let A represent an n n symmetric matrix of rank r with index of inertia m. Show that A is nonnegative definite if and only if m D r, and is positive definite if and only if m D r D n. 13. Let A represent an n n symmetric nonnegative definite matrix of rank r. Then, according to Theorem 14.3.7, there exists an n r matrix B (of rank r) such that A D BB0 . Let X represent any n m matrix (where m r) such that A D XX0 . (a) Show that X D PB X. (b) Show that X D .B; 0/Q for some orthogonal matrix Q. Section 14.4 14. Show that if a symmetric matrix A has a nonnegative definite generalized inverse, then A is nonnegative definite. Section 14.5 15. Suppose that an n n matrix A has an LDU decomposition, say A D LDU , and let d1 ; d2 ; : : : ; dn represent the diagonal elements of the diagonal matrix D. Show that jAj D d1 d2 dn : 16. (a) Suppose that an n n matrix A (where n 2) has a unique LDU decomposition, say A D LDU , and let d1 ; d2 ; : : : ; dn represent the first, second, : : : ; nth diagonal elements of D. Show that di ¤ 0 (i D 1; 2; : : : ; n 1) and that dn ¤ 0 if and only if A is nonsingular. (b) Suppose that an n n (symmetric) matrix A (where n 2) has a unique U0 DU decomposition, say A D U 0 DU , and let d1 ; d2 ; : : : ; dn represent the first, second, : : : ; nth diagonal elements of D. Show that di ¤ 0 (i D 1; 2; : : : ; n 1) and that dn ¤ 0 if and only if A is nonsingular. 17. Suppose that an nn (symmetric) matrix A has a unique U0 DU decomposition, say A D U 0 DU . Use the result of Part (b) of Exercise 16 to show that A has no LDU decompositions other than A D U 0 DU . 18. Show that if a nonsingular matrix has an LDU decomposition, then that decomposition is unique. 19. Let A represent an nn matrix (where n 2). By for instance using the results of Exercises 16, 17, and 18, show that if A has a unique LDU decomposition or (in the special case where A is symmetric) a unique U0 DU decomposition, then the leading principal submatrices (of A) of orders 1; 2; : : : ; n 1 are

280


nonsingular (thereby establishing the converse of Corollary 14.5.7) and have unique LDU decompositions. 20. (a) Let A D faij g represent an m n nonnull matrix of rank r. Show that there exist an m m permutation matrix P and an n n permutation matrix Q such that B11 B12 ; P AQ D B21 B22 where B11 is an r r nonsingular matrix whose leading principal submatrices (of orders 1; 2; : : : ; r 1) are nonsingular. B11 B12 represent any m n nonnull matrix of rank r such (b) Let B D B21 B22 that B11 is an r r nonsingular matrix whose leading principal submatrices (of orders 1; 2; : : : ; r 1) are nonsingular. Show that there exists a unique decomposition of B of the form L1 BD D.U1 ; U2 /; L2 where L1 is an r r unit lower triangular matrix, U1 is an r r unit upper triangular matrix, and D is an r r diagonal matrix. Show further that this decomposition is such that B11 D L1 DU1 is the unique LDU decomposition of B11 , D is nonsingular, L2 D B21 U11 D 1 and U2 D D 1 L1 1 B12 . 21. Show, by example, that there exist n n (nonsymmetric) positive semidefinite matrices that do not have LDU decompositions. 22. Let A represent an n n nonnegative definite (possibly nonsymmetric) matrix that has an LDU decomposition, say A D LDU . Show that the diagonal elements of the diagonal matrix D are nonnegative (thereby extending part of Theorem 14.5.9 and part of Corollary 14.5.15). 23 Let A represent an m k matrix of full column rank. And let A D QR represent the QR decomposition of A; that is, let Q represent the unique m k matrix whose columns are orthonormal with respect to the usual inner product and let R represent the unique k k upper triangular matrix with positive diagonal elements such that A D QR. Show that A0 A D R0 R (so that A0 A D R0 R is the Cholesky decomposition of A0 A). 24. Let A represent an m k matrix of rank r (where r is possibly less than k). Consider the decomposition A D QR1 , where Q is an m r matrix with orthonormal columns and R1 is an r k submatrix whose rows are the r nonnull rows of a k k upper triangular matrix R having r positive diagonal

Exercises

281

elements and k r null rows. (Such a decomposition can be obtained by using the results of Exercise 6.4—refer to Exercise 6.5.) Generalize the result of Exercise 23 by showing that if the inner product with respect to which the columns of Q are orthonormal is the usual inner product, then A0 A D R0 R (so that A0 A D R0 R is the Cholesky decomposition of A0 A). 25. Let A D faij g represent an n n matrix that has an LDU decomposition, say A D LDU . And define G D U 1 D L1 (which, as discussed in Section 14.5f, is a generalized inverse of A). (a) Show that G D D L1 C .I U /G D U 1 D C G.I L/: (b) For i D 1; : : : ; n, let di represent the i th diagonal element of the diagonal matrix D; and, for i; j D 1; : : : ; n, let ìj , uij , and gij represent the ij th elements of L, U , and G, respectively. Take D D diag.d1 ; : : : ; dn /, where di D 1=di , if di ¤ 0, and di is an arbitrary scalar, if di D 0. Show that n n X X uik gki D di gik `ki (E.1) gi i D di kDiC1

and that

gij D

kDiC1

8 n X ˆ ˆ ˆ gik `kj ; ˆ
i

(E.2b)

Pn

uik gki are to

kDj C1 n X

uik gkj ;

kDiC1

(where the degenerate sums be interpreted as 0).

Pn

kDnC1

gik `ki and

kDnC1

(c) Devise a recursive procedure that uses the formulas from Part (b) to generate a generalized inverse of A. Section 14.6 26. Verify that a principal submatrix of a skew-symmetric matrix is skewsymmetric. 27. (a) Show that the sum of skew-symmetric matrices is skew-symmetric. (b) Show that the sum A1 C A2 C C Ak of n n nonnegative definite matrices A1 ; A2 ; : : : ; Ak is skew-symmetric if and only if A1 ; A2 ; : : : ; Ak are skew-symmetric. (c) Show that the sum A1 CA2 C CAk of nn symmetric nonnegative definite matrices A1 ; A2 ; : : : ; Ak is a null matrix if and only if A1 ; A2 ; : : : ; Ak are null matrices.

282


Section 14.7 28. (a) Let A1 ; A2 ; : : : ; Ak represent n n nonnegative definite matrices. Show P P that tr. kiD1 Ai / 0, with equality holding if and only if kiD1 Ai is skew-symmetric or, equivalently, if and only if A1 ; A2 ; : : : ; Ak are skewP symmetric, thereby generalizing Theorem 14.7.2. [Note. That kiD1 Ai being skew-symmetric is equivalent to A1 , A2 , : : : ; Ak being skewsymmetric is the result of Part (b) of Exercise 27.] (b) Let A1 ; A2 ; : : : ; Ak represent n n symmetric nonnegative definite maP trices. Show that tr. kiD1 Ai / 0, with equality holding if and only if Pk iD1 Ai D 0 or, equivalently, if and only if A1 ; A2 ; : : : ; Ak are null matrices, thereby generalizing Corollary 14.7.3. 29. Show, via an example, that (for n > 1) there exist n n (nonsymmetric) positive definite matrices A and B such that tr.AB/ < 0. 30. (a) Show, via an example, that (for n > 1) there exist nn symmetric positive definite matrices A and B such that the product AB has one or more negative diagonal elements (and hence such that AB is not nonnegative definite). (b) Show, however, that the product of two n n symmetric positive definite matrices cannot be nonpositive definite. 31. Let A D faij g and B D fbij g represent n n matrices, and take C to be the n n matrix whose ij th element cij D aij bij is the product of the ij th elements of A and B. Show that if A is nonnegative definite and B is symmetric nonnegative definite, then C is nonnegative definite. Show further that if A is positive definite and B is symmetric positive definite, then C is positive definite. [Hint. Taking x D .x1 ; : : : ; xn /0 to be an arbitrary n 1 vector and F D .f1 ; : : : ; fn / to be a matrix such that B D F 0 F , begin by showing that x0 Cx D tr.AH/, where H D G 0 G with G D .x1 f1 ; : : : ; xn fn /.] 32. Let A1 ; A2 ; : : : ; Ak and B1 ; B2 ; : : : ; Bk represent n n symmetric nonnegP ative definite matrices. Show that tr. kiD1 Ai Bi / 0, with equality holding if and only if, for i D 1; 2; : : : ; k, Ai Bi D 0, thereby generalizing Corollary 14.7.7 and the results of Part (b) of Exercise 28. Section 14.8 33. Let A represent a symmetric nonnegative definite matrix that has been partitioned as T U AD ; V W

Exercises

283

where T (and hence W ) is square. Show that V T U and U W V are symmetric and nonnegative definite. 34. Show, via an example, that there exists an .m C n/ .m C n/ (nonsymmetric) T U positive semidefinite matrix A of the form A D , where T is of V W dimensions m m, W of dimensions n n, U of dimensions m n, and V of dimensions nm, for which C.U / 6 C.T / and/or R.V / 6 R.T /, expression (9.6.1) does not necessarily equal rank.A/ [i.e., rank.T /Crank.W V T U / does not necessarily equal rank.A/], and formula (9.6.2) does not necessarily give a generalized inverse of A. 35. Show, via an example, that there exists an .m C n/ .m C n/ symmetric T U partitioned matrix A of the form A D , where T is of dimensions U0 W m m, U of dimensions m n, and W of dimensions n n, such that T is nonnegative definite and (depending on the choice of T ) the Schur complement W U 0 T U of T relative to T is nonnegative definite, but A is not nonnegative definite. 36. An n n matrix A P D faij g is said to be diagonally dominant if, for i D 1; 2; : : : ; n, jai i j > jnD1.j ¤i/ jaij j. (In the degenerate special case where n D 1, A is said to be diagonally dominant if it is nonnull.) (a) Show that a principal submatrix of a diagonally dominant matrix is diagonally dominant. (b) Let A D faij g represent an n n diagonally dominant matrix, partition A11 a A as A D [so that A11 is of dimensions .n 1/ .n 1/], b0 ann and let C D A11 .1=ann /ab0 represent the Schur complement of ann . Show that C is diagonally dominant. (c) Show that a diagonally dominant matrix is nonsingular. (d) Show that a diagonally dominant matrix has a unique LDU decomposition. (e) Let A D faij g represent an n n symmetric matrix. Show that if A is diagonally dominant and if the diagonal elements a11 ; a22 ; : : : ; ann of A are all positive, then A is positive definite. Section 14.9 represent an n n symmetric positive definite matrix. Show 37. Let A D faij gQ that det.A/ niD1 ai i , with equality holding if and only if A is diagonal.

284


38. Let A D

a b , where a, b, c, and d are scalars. c d

(a) Show that A is positive definite if and only if a > 0, d > 0, and jbCcj=2 0, d > 0, and jbj < ad . 39. By, for example, making use of the result of Exercise 38, show that if an n n matrix A D faij g is symmetric positive definite, then, for j ¤ i D 1; : : : ; n, p jaij j < ai i ajj max.ai i ; ajj /: 40. Show, by example, that it is possible for the determinants of both leading principal submatrices of a 2 2 symmetric matrix to be nonnegative without the matrix being nonnegative definite and that, for n 3, it is possible for the determinants of all n leading principal submatrices of an n n symmetric matrix to be nonnegative and for the matrix to be nonsingular without the matrix being nonnegative definite. Section 14.10 41. Let V represent a subspace of Rn1 of dimension r (where r 1). Take B D .b1 ; b2 ; : : : ; br ) to be any n r matrix whose columns b1 ; b2 ; : : : ; br form a basis for V, and let L represent any left inverse of B. Let g represent a function that assigns the value x y to an arbitrary pair of vectors x and y in V. (a) Let f represent an arbitrary inner product for Rr1, and denote by s t the value assigned by f to an arbitrary pair of r-dimensional vectors s and t. Show that g is an inner product (for V) if and only if there exists an f such that (for all x and y in V) x y D .Lx/ .Ly/: (b) Show that g is an inner product (for V) if and only if there exists an r r symmetric positive definite matrix W such that (for all x and y in V) x y D x0 L0 W Ly: (c) Show that g is an inner product (for V) if and only if there exists an n n symmetric positive definite matrix W such that (for all x and y in V) x y D x0 W y:

Exercises

285

42. Let V represent a linear space of m n matrices, and let A B represent the value assigned by a quasi-inner product to any pair of matrices A and B (in V). Show that the set U D fA 2 V W A A D 0g; which comprises every matrix in V with a zero quasi-norm, is a linear space. 43. Let W represent an m m symmetric positive definite matrix and V an n n symmetric positive definite matrix. (a) Show that the function that assigns the value tr.A0 W BV / to an arbitrary pair of m n marices A and B qualifies as an inner product for the linear space Rmn. (b) Show that the function that assigns the value tr.A0 W B/ to an arbitrary pair of m n matrices A and B qualifies as an inner product for Rmn. (c) Show that the function that assigns the value tr.A0 W BW / to an arbitrary pair of m m matrices A and B qualifies as an inner product for Rmm. Section 14.12 44. Let A represent a q p matrix, B a p n matrix, and C an mq matrix. Show that (a) CAB.CAB/ C D C if and only if rank.CAB/ D rank.C/, and (b) B.CAB/ CAB D B if and only if rank.CAB/ D rank.B/ [thereby (in light of Corollary 14.11.3) extending the results of Parts (5) and (1) of Theorem 14.12.11]. 45. Let U represent a subspace of Rn1, let X represent an n p matrix whose columns span U, and let W and V represent n n symmetric positive definite matrices. Show that each of the following two conditions is necessary and sufficient for the projection PX;W y of y on U with respect to W to be the same (for every y in Rn ) as the projection PX;V y of y on U with respect to V: 0 (a) V D PX;W V PX;W C .I PX;W /0 V .I PX;W /;

(b) there exist a scalar c, a p p matrix K, and an n n matrix H such that V D cW C W XKX0 W C .I PX;W /0 H.I PX;W /: 46. Let y represent an n-dimensional column vector, let U represent a subspace of Rn1, and let X represent an n p matrix whose columns span U. It follows from Corollary 12.1.2 that y ?I U if and only if X0 y D 0 and, more generally, it follows from Lemma 14.12.1 that, for any nn symmetric positive definite matrix W , y ?W U if and only if X0 W y D 0. Extend this result (in the direction indicated in the discussion following Lemma 14.12.1) by using

286


Parts (3) and (5) of Lemma 14.12.2 to show that, for any n n symmetric nonnegative definite matrix W , y ?W U if and only if X0 W y D 0. 47. Let W represent an n n symmetric nonnegative definite matrix. (a) Generalize Theorem 14.12.12 by showing that, for any n p matrix X and any n q matrix U such that C.U / C.X/, (1) W PX;W U D W U , and U 0 W PX;W D U 0 W ; 0 W PU;W D W PX;W PU;W D W PU;W . (2) PU;W PX;W D PU;W , and PX;W (b) Generalize Corollary 14.12.13 by showing that, for any n p matrix X and any n q matrix U such that C.U / D C.X/, W PU;W D W PX;W : 48. Let U represent a subspace of Rn1, let A represent an n n matrix and W an n n symmetric nonnegative definite matrix, and let X represent any n p matrix whose columns span U. (a) Show that A is a projection matrix for U with respect to W if and only if A D PX;W C .I PX;W /XK for some p n matrix K. (b) Show that if A is a projection matrix for U with respect to W , then W A D W PX;W . 49. Let A represent an n n matrix and W an n n symmetric nonnegative definite matrix. (a) Show (by, e.g., using the results of Exercise 48) that if A0 W A D W A [or, equivalently, if .I A/0 W A D 0], then A is a projection matrix with respect to W , and in particular A is a projection matrix for C.A/ with respect to W , and, conversely, show that if A is a projection matrix with respect to W , then A0 W A D W A (thereby generalizing Theorem 14.12.16). (b) Show that if A is a projection matrix with respect to W , then in particular A is a projection matrix for C.A/ with respect to W . (c) Show that A is a projection matrix with respect to W if and only if W A is symmetric and W A2 D W A (thereby generalizing Corollary 14.12.17). 50. Let U represent a subspace of Rn1, let X represent an n p matrix whose columns span U, and let W and V represent n n symmetric nonnegative definite matrices. Show (by, e.g., making use of the result of Exercise 46) that each of the following two conditions is necessary and sufficient for every

Exercises

287

projection of y on U with respect to W to be a projection (for every y in Rn ) of y on U with respect to V : (a) X0 V PX;W D X0 V , or, equivalently, X0 V .I PX;W / D 0; (b) there exists a p p matrix Q such that V X D W XQ, or, equivalently, C.V X/ C.W X/ [thereby generalizing Parts (1) and (4) of Theorem 14.12.18]. 51. Let X represent an n p matrix and W an n n symmetric nonnegative definite matrix. As in the special case where W is positive definite, let ? CW .X/ D fy 2 Rn1 W y ?W C.X/g:

(a) By, for example, making use of the result of Exercise 46, show that ? CW .X/ D N.X0 W / D C.I PX;W /

(thereby generalizing Lemma 14.12.21). (b) Show that ? dimŒCW .X/ D n rank.W X/ n rank.X/ D n dimŒC.X/

(thereby generalizing Corollary 14.12.22). (c) By, for example, making use of the result of Exercise 46, show that, for any solution b to the linear system X0 W Xb D X0 W y (in b), the vector ? .X/ with respect to W (thereby y Xb is a projection of y on CW extending the result of Theorem 14.12.23).

15 Matrix Differentiation

It is natural and convenient to use matrix notation and terminology in defining and discussing certain functions of one or more variables. Earlier (in Chapter 14), matrix notation and terminology were used in defining and discussing linear, bilinear, and quadratic forms. In some cases, the use of matrix notation and terminology is essentially unavoidable—consider, for example, a case where the determinant of an m m matrix is regarded as a function of its m2 elements. Matrix differentiation is the derivation of the first-, second-, or higher-order partial derivatives of a function or functions that have been expressed in terms of matrices. In deriving, presenting, and discussing the partial derivatives of such functions, it is natural, convenient, and in some cases necessary to employ matrix notation and terminology. Not only may the functions be expressed in terms of matrices, but the functions may comprise the elements of a vector or matrix, as would be the case if the elements of the inverse of an m m (nonsingular) matrix A were regarded as functions of the elements of A. Matrix differentiation is of considerable importance in statistics. It is especially useful in connection with the maximum likelihood estimation of the parameters in a statistical model. The maximum likelihood estimates of the model’s parameters satisfy the equations (known as the likelihood equations) obtained by equating to zero the first-order partial derivatives (with respect to the model’s parameters) of the logarithm of the so-called likelihood function—in many important cases, the likelihood function involves the determinant and/or inverse of a matrix. Further, an approximation (suitable for large samples) to the variance-covariance matrix of the maximum likelihood estimators can be obtained by inverting the matrix (known as the information matrix) whose ij th element is 1 times the second-order partial derivative (with respect to the i th and j th parameters) of the logarithm of the likelihood function.

290


15.1 Definitions, Notation, and Other Preliminaries a. Neighborhoods, interior points, and open sets Let c represent an arbitrary m-dimensional column vector. Taking the norm for Rm1 to be the usual norm, a neighborhood of c is (by definition) a set of the general form fx 2 Rm1 W kx ck < rg; where r is a positive number called the radius of the neighborhood. Thus, a neighborhood of c of radius r is the set of all m-dimensional column vectors whose distance from c is less than r. Geometrically, a neighborhood of c is (depending on whether m is 1, 2, 3, or greater than 3) the interior of an interval, circle, sphere, or “hypersphere” centered at c. The definition of neighborhood can be extended to row vectors. In fact, it can be extended to matrices of any dimensions. Taking the norm for Rmn to be the usual norm, a neighborhood of an arbitrary m n matrix C is a set of the general form fX 2 Rmn W kX Ck < rg: Let S represent an arbitrary set of m-dimensional column vectors (or more generally of m n matrices), that is, let S represent a subset of Rm1 (or more generally of Rmn ). Then, a vector x (or matrix X) in S is said to be an interior point of S if there exists a neighborhood of x (or X), all of whose points belong to S . The set S is said to be open if all of its points are interior points. It can be shown that every neighborhood of a point x (or X) in Rm (or Rmn ) is an open set. b. Functions and continuity In what follows, the word function is to be used restrictively to mean a scalarvalued function whose domain is a set (of m-dimensional column vectors) in Rm or more generally a set (of m n matrices) in Rmn. The value of a function f corresponding to an arbitrary vector x or matrix X is to be denoted by the symbol f .x/ or f .X/. Further, if f D ffs g is a p1 vector or, more generally, if F D ffst g is a p q matrix, each element of which is a function defined on some set in Rmn, then, for an arbitrary matrix X, the p 1 vector Œf1 .X/; : : : ; fp .X/0 or the p q matrix with st th element fst .X/ is to be denoted by f .X/ or F .X/ and referred to as the value of f or F at X. At times, attention will be restricted to functions whose domains are sets of column vectors. This can be done without loss of generality in the sense that the elements of an m n matrix can be rearranged in the form of an mn-dimensional column vector. Let f represent a function whose domain is a set S in Rm1. Then, f is said to be continuous at an interior point c of S if

15.1 Definitions, Notation, and Other Preliminaries

291

lim f .x/ D f .c/

x!c

—by definition, limx!c f .x/ is a scalar b such that, for every positive scalar , there exists a neighborhood N of c such that jf .x/ bj < for all x in N other than c. A p q matrix F of functions, each of whose domains is the same set S in Rm1, will be said to be continuous at an interior point c of S if all pq elements of F are continuous at c. c. First-order partial derivatives and continuous differentiability Let f represent a function, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Suppose that S contains at least some interior points, and let c D .c1 ; : : : ; cm /0 represent an arbitrary one of those points. Further, let uj represent the j th column of Im . Consider the limit f .c C t uj / f .c/ : lim t!0 t When this limit exists, it is called the jth (first-order) partial derivative of f at c and is denoted by Djf .c/. Note that c C t uj D .c1 ; : : : ; cj 1 ; cj C t; cj C1 , : : : ; cm /0 , so that Djf .c/ can be regarded as the ordinary derivative (at cj ) of the function of one variable obtained from f .x/ by fixing x1 ; : : : ; xj 1 ; xj C1 , : : : ; xm at c1 ; : : : ; cj 1 ; cj C1 , : : : ; cm , respectively. The scalar Djf .c/ can be regarded as the value assigned to the point c by a function. This function is denoted by the symbol Djf (and is referred to as the j th partial derivative of f ). Its domain consists of those interior points (of S ) at which the j th partial derivative (of f ) is defined. The symbol Df represents the row vector .D1f; : : : ; Dmf / whose elements are the partial derivatives of f , and accordingly the symbol Df .c/ represents the row vector ŒD1f .c/; : : : ; Dmf .c/ whose elements are the values of the functions D1f; : : : ; Dmf at c—note that Df .c/ is defined only if c is such that all m of the partial derivatives of f at c exist. The column vector .Df /0 is referred to as the gradient (or gradient vector) of f . An alternative notation is obtained by writing @f .x/=@xj for the j th partial derivative of f at x, writing @f .x/=@x0 for the row vector Œ@f .x/=@x1 , : : : ; @f .x/=@xm of partial derivatives of f at x, and writing @f .x/=@x for the column vector Œ@f .x/=@x1 ; : : : ; @f .x/=@xm 0 of partial derivatives of f at x. In this context, @f .x/=@x0 may be called the derivative of f .x/ with respect to x0 , and @f .x/=@x may be called the derivative of f .x/ with respect to x. The symbols @f .x/=@xj , @f .x/=@x0 , and @f .x/=@x have the same interpretations as Dj f .x/, Df .x/, and ŒDf .x/0 , respectively, and are sometimes abbreviated to @f =@xj , @f =@x0 , and @f =@x. On certain occasions (to be ascertained from the context), these symbols may be used to represent the function Djf and the vectors of functions Df and .Df /0 (rather than their values at x). The function f (with domain S in Rm1 ) will be said to be continuously differentiable at the interior point c (of S ) if D1f .x/; : : : ; Dmf .x/ exist and are

292


continuous at every point x in some neighborhood of c. It can be shown that if f is continuously differentiable at c, then lim

x!c

f .x/ Œf .c/ C Df .c/.x c/ D 0; kx ck

(1.1)

which indicates that, for x sufficiently close to c, the first-order Taylor formula f .c/ C Df .c/.x c/ approximates f .x/ with an error that is of smaller order than kx ck (e.g., Magnus and Neudecker, 1988, chap. 5). If the function f is such that result (1.1) holds, then f .x/ ! f .c/, as is evident upon writing f .x/ D f .c/ C Df .c/.x c/ C kx ck

f .x/ Œf .c/ C Df .c/.x c/ : kx ck

Thus, we have the following lemma. Lemma 15.1.1. If a function f , with domain S in Rm1, is continuously differentiable at an interior point c of S , then f is continuous at c. A function f for which result (1.1) holds is said to be differentiable at c. It is worth mentioning that, while f being continuously differentiable at c is a sufficient condition for f to be differentiable at c, it is not in general necessary— as demonstrated by, for example, Magnus and Neudecker (1988, secs. 5.9 and 5.10). d. Second- and higher-order partial derivatives As in Subsection c, let f represent a function, defined on a set S , of a vector x D .x1 ; : : : xm /0 of m variables; suppose that S contains at least some interior points; and let c represent an arbitrary one of those points. Suppose that D1f .x/; : : : ; Dmf .x/ exist for all x in some neighborhood of c (so that c is an interior point of the domains of the functions D1f; : : : ; Dmf ). When the i th (first-order) partial derivative of Djf at c exists, it is called the ijth second-order partial derivative of f at c and is denoted by Dij2 f .c/. The scalar Dij2 f .c/ can be regarded as the value assigned to the point c by a function. This function is denoted by the symbol Dij2 f (and is referred to as the ij th second-order partial derivative of f ). The symbol Hf represents the m m matrix whose ij th element is Dij2 f , and accordingly the symbol Hf .c/ represents the value of Hf at the point c, that is, the m m matrix whose ij th element is Dij2 f .c/. The matrix Hf is called the Hessian matrix of f . An alternative notation is obtained by writing @2f .x/=@xi @xj —or, in the special case where j D i , @2f .x/=@xi2 —for the ij th second-order partial derivative Dij2 f .x/ of f at x and writing @2f .x/=@x@x0 for the Hessian matrix Hf .x/ of f at x. The symbols @2f .x/=@xi @xj , @2f .x/=@xi2 , and @2f .x/=@x@x0 are sometimes


293

abbreviated to @2f =@xi @xj , @2f =@xi2 , and @2f =@x@x0 , respectively, and are sometimes used to represent the functions Dij2 f and Di2i f and the matrix of functions Hf (rather than their values at x). The function f will be said to be twice (or 2 times) continuously differentiable at c if f and all of its first-order partial derivatives are continuously differentiable at c or, equivalently, if all of the first- and second-order partial derivatives of f exist and are continuous at every point in some neighborhood of c. It can be shown that if f is twice continuously differentiable at c, then the matrix Hf is symmetric at c [i.e., Hf .c/ is symmetric, or, equivalently, Dj2i f .c/ D Dij2 f .c/ for all i and j < i ]—refer to, for example, Magnus and Neudecker (1988, sec. 6.7). It can be further shown that if f is twice continuously differentiable at c, then f .x/ Œf .c/ C Df .c/.x c/ C .1=2/.x c/0 Hf .c/.x c/ D 0; (1.2) x!c kx ck2 lim

which indicates that, for x sufficiently close to c, the second-order Taylor formula f .c/ C Df .c/.x c/ C .1=2/.x c/0 Hf .c/.x c/ approximates f .x/ with an error that is of smaller order than kx ck2 (e.g., Magnus and Neudecker, 1988, sec. 6.9). Partial derivatives of f of order k (where k 2) can be defined recursively. Suppose that the (k 1)th-order partial derivatives of f at x exist for all x in some neighborhood of c. For j D 1; : : : ; k, let ij represent an arbitrary integer between 1 and m, inclusive. When the i1 th partial derivative of the i2 ik th (k 1)th-order partial derivative of f at c exists, it is called the i1 i2 ik th kthorder partial derivative of f at c. The function whose value at c is the i1 i2 ik th kth-order partial derivative of f at c is referred to as the i1 i2 ik th kth-order partial derivative of f . The symbol @kf .x/=@xi1 @xik (or in abbreviated form @kf =@xi1 @xik ) can be used to represent the i1 ik th kth-order partial derivative of f at x. The function f will be said to be k times continuously differentiable at c if f and all of its first- through (k 1)th-order partial derivatives are continuously differentiable at c or, equivalently, if all of the first- through kth-order partial derivatives of f exist and are continuous at every point in some neighborhood of c. It can be shown that if f is k times continuously differentiable at c, then, for any permutation j1 ; : : : ; jk of the sequence i1 ; : : : ; ik , the j1 jk th and i1 ik th kthorder partial derivatives of f are identical, and, letting i1 ; : : : ; is (i1 < < is ) denote the distinct integers represented among i1 ; : : : ; ik and letting kj represent the number of integers in the sequence i1 ; : : : ; ik that equal ij , we may write

@kf .x/=@xik1 @xiks (or simply @kf =@xik1 @xiks ) for @kf .x/=@xi1 @xik . It s s 1 1 can be further shown that if f is k times continuously differentiable at c, then, for x sufficiently close to c, the kth-order Taylor formula approximates f .x/ with an error that is of smaller order than kx ckk. Note that if f is k times continuously differentiable at c, then f is continuously differentiable at c and is also 2; : : : ; k 1 times continuously differentiable at c.

294


Moreover, if f is k times continuously differentiable at c, then f is k times continuously differentiable at every point in some neighborhood of c, as can be easily verified. e. Partial derivatives of a function of an unrestricted or symmetric matrix Suppose that the domain of the function f to be differentiated is the set Rmn of all m n matrices or, more generally, is a set S in Rmn that contains at least some interior points. Then, f can be regarded as a function of an m n matrix X D fxij g of mn “independent” variables. And, for purposes of differentiating f , the elements of X can be rearranged in the form of an mn-dimensional column vector x, and f can be reinterpreted as a function of x, in which case the domain of f is the set, say S , obtained by rearranging the elements of each m n matrix in S in the form of a column vector. (It should be noted that an mn-dimensional column vector is an interior point of S if and only if it is a rearrangement of an m n matrix that is an interior point of S .) By definition, the elements @f =@xij (i D 1; : : : ; mI j D 1; : : : ; n) of the mn-dimensional column vector @f =@x are the first-order partial derivatives of f at x (and the elements of the mn mn matrix @2f =@x@x0 are the second-order partial derivatives of f at x). However, instead of presenting the first-order partial derivatives of f at x (or, equivalently, X) in the form of the vector @f =@x, it is natural and in many cases convenient to present them in the form of the m n matrix whose ij th element is @f =@xij . This matrix is to be denoted by the symbol @f .X/=@X (or in abbreviated form @f =@X) and is to be called the derivative of f .X/ with respect to X. Further, let us write @f .X/=@X0 (or @f =@X0 ) for the n m matrix Œ@f .X/=@X0 and refer to this matrix as the derivative of f .X/ with respect to X0 . Suppose now that the function f of interest is one whose domain is restricted to all m m symmetric matrices or, more generally, to a subset S of such matrices. Then, f can still be regarded as a function of an m m matrix X D fxij g of m2 variables. However, because of the restriction to symmetric matrices, xj i D xij for all i and j > i , so that the m2 elements of X can no longer be regarded as m2 independent variables. A closely related point is that S cannot contain any interior points. As a consequence, the previous development—that leading to the introduction of the symbol @f .X/=@X and to the definition of the derivative of f .X/ with respect to X—is not applicable. In the present case—that where S contains only symmetric matrices—the symbol @f .X/=@X and the term “derivative of f .X/ with respect to X” are to be defined differently. (When not explicitly indicated, the intended interpretation must be ascertained from the context.) For purposes of differentiating the function f of the symmetric matrix X, f is to be interpreted as a function of an [m.m C 1/=2]-dimensional column vector x whose elements are xij (j i D 1; : : : ; m) or, alternatively, xij (j i D 1; : : : ; m) [i.e., whose elements are the “independent” elements of X, which are those on and below (or alternatively above) the diagonal]. When f is reinterpreted in this way, the domain of f is the set S of [m.m C 1/=2]-dimensional column


295

vectors obtained by transforming the m m matrices in S from X-values into x-values. Suppose that S contains at least some interior points. (The set S can contain interior points, even though S cannot.) By definition, the elements @f =@xij (j i D 1; : : : ; m or, alternatively, j i D 1; : : : ; m) of the [m.m C 1/=2]dimensional column vector @f =@x are the first-order partial derivatives of f at x {and the elements of the Œm.m C 1/=2 Œm.m C 1/=2 matrix @f =@x@x0 are the second-order partial derivatives of f at x}. However, instead of presenting the first-order partial derivatives of f at x (or, equivalently, X) in the form of the vector @f =@x, it is natural, and in many cases convenient, to present them in the form of the m m symmetric matrix whose ij th and j i th elements are @f =@xij . In the present case (that where the domain S of f is restricted to symmetric matrices), it is this matrix that is denoted by the symbol @f .X/=@X (or @f =@X) and is called the derivative of f with respect to X. f. Differentiation of a vector or matrix of functions Suppose that there is a (column) vector f D .f1 ; : : : ; fp /0 of p functions to be differentiated and that the domain of all of these functions is a set S in Rm1. Suppose further that S contains at least some interior points, and let c represent an arbitrary one of those points. The symbol Dj f is used to represent the p-dimensional column vector whose sth element is the j th partial derivative Djfs of fs , and the symbol Df is used to represent the p m matrix whose sj th element is Djfs or, equivalently, the p m matrix whose j th column is Dj f . Accordingly, Dj f .c/ D ŒDjf1 .c/; : : : ; Djfp .c/0 , and Df .c/ D ŒD1 f .c/; : : : ; Dm f .c/ (provided that each of the m partial derivatives of each of the p functions f1 ; : : : ; fp at c exists). The matrix Df is called the Jacobian matrix of f , and its transpose .Df /0 is called the gradient (or gradient matrix) of f . Note that .Df /0 D Œ.Df1 /0 ; : : : ; .Dfp /0 . In the special case where p D m, the determinant of Df is called the Jacobian (or Jacobian determinant) of f . An alternative notation is obtained by letting x D .x1 ; : : : ; xm /0 represent a vector of m variables and writing @ f .x/=@x0 (or @ f =@x0 ) for the p m matrix whose sj th element is @fs .x/=@xj and @ f 0 .x/=@x (or @ f 0 =@x) for the mp matrix Œ@ f .x/=@x0 0 whose jsth element is @fs .x/=@xj . In this context, @ f .x/=@x0 may be called the derivative of f .x/ with respect to x0 , and @ f 0 .x/=@x may be called the derivative of f 0 .x/ with respect to x. The symbol @ f .x/=@x0 has the same interpretation as Df .x/ (or alternatively Df ). Suppose now that there is a p q matrix F D ffst g of pq functions to be differentiated and that the domain of all of these functions is a set S in Rm1 (that contains at least some interior points). As in the special case where q D 1, each of the elements of F can be regarded as a function of a vector x D .x1 ; : : : ; xm /0 of m “independent” variables. All mpq (first-order) partial derivatives (of the elements of F ) can be presented in the form of a single pq m (or m pq) matrix by rearranging the elements of

296


F in the form of a column vector f and by then forming the Jacobian (or gradient) matrix of f . However, with the possible exception of the special case where p D 1 or q D 1 (i.e., where F is a row or column vector), it is sometimes preferable to present the j th (first-order) partial derivatives of the elements of F in a separate p q matrix whose st th element is @fst .x/=@xj (j D 1; : : : ; m). This matrix is to be denoted by the symbol @F .x/=@xj (or, in abbrevated form, @F =@xj ) and is to be referred to as the j th partial derivative of F —in the special case where q D 1 (i.e., where F is a column vector), @F .x/=@xj has the same interpretation as Dj F .x/. And the p q matrix whose st th element is @kfst .x/=@xjk11 @xjkrr is to be denoted by the symbol @k F .x/=@xjk11 @xjkrr or @k F =@xjk11 xjkrr (where j1 ; : : : ; jr is a subsequence of the first m positive integers; k1 ; : : : ; kr are positive integers; and k D k1 C C kr ). The matrix F will be said to be continuously differentiable at an interior point c (of S ) if all pq of its elements are continuously differentiable at c, and to be twice (or 2 times) continuously differentiable at c if all pq of its elements are twice continuously differentiable at c. More generally, F is said to be k times continuously differentiable at c if all pq of its elements are k times continuously differentiable at c.

15.2 Differentiation of (Scalar-Valued) Functions: Some Elementary Results The techniques employed in matrix differentiation can be regarded as generalizations or extensions of those employed in nonmatrix differentiation. Some elementary results on nonmatrix differentiation are stated (without proof) in the following two lemmas. Lemma 15.2.1. Let f represent a function, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables, and suppose that (for x 2 S ) f .x/ is constant or (more generally) does not vary with xj . Then, for any interior point c of S , Djf .c/ D 0. Lemma 15.2.2. Let f and g represent functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. And let a and b represent constants or (more generally) functions (defined on S ) that are continuous at every interior point of S and are such that a.x/ and b.x/ do not vary with xj . Define ` D af C bg;

h D fg;

and

r D f =g;

so that ` and h are functions, each of whose domain is S , and r is a function whose domain is S D fx 2 S W g.x/ ¤ 0g. If f and g are continuously differentiable at an interior point c of S , then ` and h are also continuously differentiable at c, and (2.1) Dj `.c/ D a.c/Djf .c/ C b.c/Dj g.c/ and

15.2 Differentiation of (Scalar-Valued) Functions: Some Elementary Results

Dj h.c/ D f .c/Dj g.c/ C g.c/Djf .c/:

297

(2.2)

And if f and g are continuously differentiable at an interior point c of S , then r is also continuously differentiable at c, and Dj r.c/ D Œg.c/Djf .c/ f .c/Dj g.c/=Œg.c/2 :

(2.3)

Formulas (2.1)–(2.3) can be expressed less formally and more succinctly as @f @g @.af C bg/ Da Cb ; @xj @xj @xj @fg @g @f Df Cg ; @xj @xj @xj @.f =g/ @f @g =g 2 : D g f @xj @xj @xj

and

(2.4) (2.5)

(2.6)

Special cases of formula (2.4) include: @f @.af / Da ; @xj @xj

@.f C g/ @f @g D C ; @xj @xj @xj

@.f g/ @f @g D : (2.7) @xj @xj @xj

And, as a special case of formula (2.6), we have (in light of Lemma 15.2.1) that @.1=g/ @g D .1=g 2 / : @xj @xj

(2.8)

Results (2.1) and (2.2) can be extended (by repeated application) to a linear combination or a product of an arbitrary number of functions. Let f1 ; f2 ; : : : ; fk represent k functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables, and let a1 ; a2 ; : : : ; ak represent constants or (more generally) functions (defined on S ) that are continuous at every interior point of S and are such that a1 .x/; a2 .x/; : : : ; ak .x/ do not vary with xj . Define ` D a1 f1 C a2 f2 C C ak fk

and

h D f1 f2 fk :

If f1 ; f2 ; : : : ; fk are continuously differentiable at an interior point c of S , then ` and h are also continuously differentiable at c, and Dj `.c/ D a1 .c/Djf1 .c/ C a2 .c/Djf2 .c/ C C ak .c/Djfk .c/ and Dj h.c/ D

k hY X iD1

i fs .c/ Djfi .c/:

(2.9)

(2.10)

s¤i

Formulas (2.9) and (2.10) can be rewritten as @.a1 f1 C a2 f2 C ak fk / @f1 @f2 @fk D a1 C a2 C C ak @xj @xj @xj @xj

(2.11)

298


and

k Y X @fi @.f1 f2 fk / D fs : @xj @xj iD1

(2.12)

s¤i

Note that formulas (2.6) and (2.12) for the partial derivatives of a ratio of two functions or for a product of two or more functions can be recast in vector notation as @.f =g/ @g @f 2 Dg f (2.13) g @x @x @x and

k Y X @.f1 f2 fk / @fi D : fs @x @x iD1

(2.14)

s¤i

Similarly, formula (2.11) can (in the special case where a1 ; a2 ; : : : ; ak are constants) be recast as @.a1 f1 C a2 f2 C C ak fk / @f1 @f2 @fk D a1 C a2 C C ak : @x @x @x @x

(2.15)

Note also that by setting each of the k functions f1 ; : : : ; fk in formula (2.12) equal to the same function, say f , we obtain the result that (for any positive integer k) @f @f k D kf k1 : (2.16) @xj @xj Since clearly @xj D 1; @xj we have in particular [upon taking f .x/ D xj ] that @xjk @xj

D kxjk1 :

(2.17)

15.3 Differentiation of Linear and Quadratic Forms The results of Section 15.2 can be used to derive formulas for the partial derivatives of a linear form a0 x or a quadratic form x0 Ax (in an unconstrained m-dimensional column vector x). Let ai and xi represent the i th elements of a and x, respectively, and let aik represent the i kth element of A. Then, X X a0 x D ai xi and x0 Ax D aik xi xk : i

i;k

In light of Lemma 15.2.1 and results (2.17) and (2.7), we have that

15.3 Differentiation of Linear and Quadratic Forms

( @xi 1; D @xj 0; and that

8 2xj ; ˆ ˆ ˆ < @.xi xk / xi ; D ˆ @xj xk ; ˆ ˆ : 0;

if i D j , if i ¤ j ,

(3.1a) (3.1b)

if i D k D j , if k D j but i ¤ j , if i D j but k ¤ j , otherwise (i.e., if i ¤ j and k ¤ j ).

Using these results in combination with result (2.11), we find that

P X @xi @ i ai xi @.a0 x/ D D ai D aj @xj @xj @xj i and @.x0 Ax/ @xj D D

(3.2a) (3.2b) (3.2c) (3.2d)

(3.3)

P @ i;k aik xi xk @xj

P P P @ ajj xj2 C i¤j aij xi xj C k¤j aj k xj xk C i¤j;k¤j aik xi xk

D ajj

@xj2 @xj

C

D 2ajj xj C D

299

X

@xj X @.xi xj / @.xj xk / aij C aj k @xj @xj

X i¤j

X

k¤j

aij xi C

i¤j

aij xi C

i

X

X

C

X i¤j;k¤j

aik

@.xi xk / @xj

aj k xk C 0

k¤j

aj k xk :

(3.4)

k

Result (3.3) can be recast in vector notation as @.a0 x/ Da @x

(3.5)

or alternatively as

@.a0 x/ D a0 : @x0 And result (3.4) can be recast in matrix notation as

(3.6)

@.x0 Ax/ D .A C A0 /x; (3.7) @x P as is evident P upon observing that k aj k xk0 is the j th element of the column vector Ax and i aij xi is the j th element of A x.

300


The sj th second-order partial derivative of x0 Ax can be obtained by finding the sth partial derivative of expression (3.4), giving X X @2 .x0 Ax/ @xi @xk D aij C aj k D asj C ajs : @xs @xj @xs @xs i

(3.8)

k

Result (3.8) can be recast in matrix notation as @2 .x0 Ax/ D A C A0 : @x@x0

(3.9)

Note that all partial derivatives of a0 x of order higher than one are zero, and that all partial derivatives of x0 Ax of order higher than two are zero. Note also that, in the special case where A is symmetric, results (3.7) and (3.9) simplify to @.x0 Ax/ D 2Ax; @x

@2 .x0 Ax/ D 2A: @x@x0

Formulas (3.5) and (3.6) for the derivative of the linear form a0 x with respect to x or x0 can be extended to a vector of linear forms. Let B D fbij g represent a p m matrix of constants, and consider the column vector Bx. The i th element of Bx is the linear form b0i x, whose coefficient vector is b0i D .bi1 ; : : : ; bim /. According to result (3.3), the j th partial derivative of this linear form is bij . Thus, the partial derivative of Bx with respect to x0 (which is the Jacobian matrix of Bx) is @.Bx/ D B; (3.10) @x0 and the partial derivative of .Bx/0 with respect to x (which is the gradient matrix of Bx) is @.Bx/0 D B0 : (3.11) @x Note that, in the special case where B D Im , results (3.10) and (3.11) reduce to @x @x0 D Im : D 0 @x @x

(3.12)

15.4 Differentiation of Matrix Sums, Products, and Transposes (and of Matrices of Constants) The results presented in Section 15.2 (on the differentiation of functions) can be extended to the differentiation of matrices of functions. The following lemma generalizes (and follows from) Lemma 15.2.1. Lemma 15.4.1. Let F D ffis g represent a p q matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables; and suppose that (for x 2 S ) F .x/ is constant or (more generally) does not vary with xj . Then, at any interior point of S , @F =@xj D 0.

15.4 Differentiation of Matrix Sums, Products, and Transposes

301

A generalization of formula (2.4) is given by the following lemma. Lemma 15.4.2. Let F D ffis g and G D fgis g represent p q matrices of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. And let a and b represent constants or (more generally) functions (defined on S ) that are continuous at every interior point of S and are such that a.x/ and b.x/ do not vary with xj . Then, at any interior point c (of S ) at which F and G are continuously differentiable, aF C bG is continuously differentiable and @F @G @.aF C bG/ Da Cb : @xj @xj @xj

(4.1)

Proof. Let L D aF C bG. The i sth element of L is ìs D afis C bgis : The functions fis and gis are continuously differentiable at c, implying (in light of Lemma 15.2.2) that ìs is continuously differentiable at c and that (at x D c) @fis @gis @ìs Da Cb : @xj @xj @xj It follows that L is continuously differentiable at c, and (since @ìs =@xj , @fis =@xj , and @gis =@xj are the i sth elements of @L=@xj , @F =@xj , and @G=@xj , respectively) that (at x D c) @L @F @G Da Cb : @xj @xj @xj Q.E.D. We obtain, as special cases of formula (4.1), the following generalization of result (2.7): @.aF / @F Da ; @xj @xj

@.F C G/ @F @G D C ; @xj @xj @xj

@.F G/ @F @G D : (4.2) @xj @xj @xj

Formula (2.5) is generalized in the following lemma. Lemma 15.4.3. Let F D ffis g and G D fgis g represent p q and q r matrices of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Then, at any interior point c (of S ) at which F and G are continuously differentiable, F G is continuously differentiable and @G @F @F G DF C G: @xj @xj @xj Proof. Let H D F G. The i t th element of H is hit D

q X sD1

fis gst :

(4.3)

302


The functions fis and gst are continuously differentiable at c, implying (in light of Lemma 15.2.2) that fis gst is continuously differentiable at c and that (at x D c) @fis gst @gst @fis D fis C gst : @xj @xj @xj Thus, hit is continuously differentiable at c, and (at x D c) q

q

q

X @.fis gst/ X X @fis @gst @hit D D fis C gst : @xj @xj @xj @xj sD1 sD1 sD1 We conclude that H Pis continuously differentiable at c and [since P q q sD1 fis .@gst =@xj / and sD1 .@fis =@xj /gst are the i t th elements of F .@G=@xj / and .@F =@xj /G, respectively] that (at x D c) @F G @G @F DF C G: @xj @xj @xj Q.E.D. In the special case where (for x 2 S ) F .x/ is constant or (more generally) does not vary with xj , formula (4.3) simplifies to @F G @G DF : @xj @xj

(4.4)

And, in the special case where (for x 2 S ) G.x/ is constant or (more generally) does not vary with xj , formula (4.3) simplifies to @.F G/ @F D G: @xj @xj

(4.5)

The results of Lemma 15.4.3 can be extended (by repeated application) to the product of three or more matrices. Let F , G, and H represent p q, q r, and r v matrices of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Then, at any interior point (of S ) at which F , G, and H are continuously differentiable, F GH is continuously differentiable and @F GH @H @G @F D FG CF HC GH: @xj @xj @xj @xj

(4.6)

In the special case where (for x 2 S ) F .x/ and H.x/ are constant or (more generally) do not vary with xj , formula (4.6) simplifies to @.F GH/ @G DF H: @xj @xj

(4.7)

More generally, let F1 ; F2 ; : : : ; Fk represent matrices of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Then, at any interior

15.5 Differentiation of a Vector or Matrix With Respect to Its Elements

303

point (of S ) at which F1 ; F2 ; : : : ; Fk are continuously differentiable, F1 F2 Fk is continuously differentiable and @.F1 F2 Fk / @xj D F1 Fk1

@Fk @Fk1 @F1 C F1 Fk2 Fk C C F2 Fk : @xj @xj @xj

(4.8)

(assuming that the dimensions of F1 ; F2 ; : : : ; Fk are such that the product F1 F2 Fk is defined). Note that if g is a function, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables and if F is a p q matrix of functions (defined on S ) of x, then, at any interior point (of S ) at which g and F are continuously differentiable, gF is continuously differentiable and @.gF / @g @F D F Cg ; @xj @xj @xj

(4.9)

as is evident from Lemma 15.4.3 upon taking G D gI. Note also that if F is a p q matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables, then, at any interior point at which F is continuously differentiable, F 0 is continuously differentiable and @F 0 @F 0 D : (4.10) @xj @xj

15.5 Differentiation of a Vector or (Unrestricted or Symmetric) Matrix With Respect to Its Elements Let x D fxs g represent an m-dimensional column vector, and let uj represent the j th column of an identity matrix (of unspecified dimensions). The elements of x can be regarded as functions, defined on Rm, of x. The j th partial derivatives of these functions are given by result (3.1). This result can be reexpressed in matrix notation as @x D uj : (5.1) @xj Result (5.1) can be generalized to an (unrestricted) m n matrix X D fxst g. The elements of X can be regarded as functions, defined on Rmn, of X. For purposes of differentiating xst , the elements of X can be rearranged in the form of an mn-dimensional column vector x, and xst can be reinterpreted as a function (defined on Rmn ) of x (as discussed in Section 15.1e). Then [in light of result (3.1)], ( @xst 1; if s D i and t D j , (5.2a) D @xij 0; otherwise. (5.2b)

304


Or, in matrix notation, @X D ui uj0 : @xij

(5.3)

Suppose now that X D fxst g is a symmetric (but otherwise unrestricted) matrix (of dimensions m m). Then, the elements of X can still be regarded as functions of X; however, the domain of these functions now comprises all m m symmetric matrices, which (unless m D 1) is a proper subset of Rmm. For purposes of differentiating xst , xst is (by convention) interpreted as a function (defined on Rm.mC1/=2 ) of an [m.m C 1/=2]-dimensional column vector x whose elements are xij (j i D 1; : : : ; m) or alternatively xij (j i D 1; : : : ; m). This approach gives ( @xst 1; if s D t D i , (5.4a) D @xi i 0; otherwise; (5.4b) and, for j i ), ( @xst 1; if s D i and t D j or if s D j and t D i , D @xij 0; otherwise.

(5.5a) (5.5b)

Or, in matrix notation, @X D ui u0i ; @xi i and, for j i ), @X D ui uj0 C uj u0i : @xij

(5.6)

(5.7)

15.6 Differentiation of a Trace of a Matrix Let F D ffis g represent a p p matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Then, at any interior point c (of S ) at which F is continuously differentiable, tr.F / is continuously differentiable and @ tr.F / @F ; (6.1) D tr @xj @xj as is evident upon observing that tr.F / D f11 Cf22 C Cfpp [which establishes that tr.F / is continuously differentiable at c] and that (at x D c) @ tr.F / @f11 @f22 @fpp @F : D C C C D tr @xj @xj @xj @xj @xj Suppose now that F is a p q matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables, and suppose that A is a q p matrix of constants or (more generally) a q p matrix of functions (defined on S ) that

15.6 Differentiation of a Trace of a Matrix

305

is continuous at every interior point of S and such that A.x/ does not vary with xj . Then, making use of results (6.1) and (4.5) (along with Lemma 5.2.1), we find that, at any interior point (of S ) at which F is continuously differentiable, tr.AF / or, equivalently, tr.F A/ is continuously differentiable and @ tr.AF / @ tr.F A/ @F @F : (6.2) D D tr A D tr A @xj @xj @xj @xj Result (6.2) can be generalized to the trace of the product of “any” two matrices of functions. Let F and G represent p q and q p matrices of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Then, in combination with result (6.1) and Lemma 5.2.1, Lemma 15.4.3 implies that, at any interior point at which F and G are continuously differentiable, tr.F G/ or, equivalently, tr.GF / is continuously differentiable and @F @ tr.F G/ @ tr.GF / @G C tr G : (6.3) D D tr F @xj @xj @xj @xj Note that alternative versions of formula (6.3) can be obtained by making either or both of the two substitutions @G @G @F @F tr F D tr D tr F and tr G G : @xj @xj @xj @xj Let us now consider a special case of result (6.2). Take X D fxst g to be an mn matrix of “independent” variables, and suppose that the range of X comprises all of Rmn. Define A D fa ts g to be an nm matrix of constants, and let uj represent the j th column of an identity matrix (of unspecified dimensions). Then, @ tr.AX/ @ tr.XA/ D D aj i ; @xij @xij

(6.4)

as is evident from result (6.2) upon observing [in light of result (5.3)] that @X tr A D tr.Aui uj0 / D tr.uj0 aui / D uj0 aui D aj i : @xij Result (6.4) can be restated as @ tr.AX/ @ tr.XA/ D D A0 : @X @X

(6.5)

Suppose now that X is a symmetric (but otherwise unrestricted) matrix (of dimensions m m). Then, for purposes of differentiating a function of X, the function is reinterpreted as a function of an [m.m C 1/=2]-dimensional column vector x whose elements are xij (j i D 1; : : : ; m). Consequently, result (6.4) is no longer applicable. Instead, we have that ( @ tr.AX/ @ tr.XA/ ai i ; if j D i , (6.6a) D D @xij @xij (6.6b) aij C aj i ; if j < i .

306


To verify this, apply result (6.2) and observe [in light of results (5.6) and (5.7)] that @X D tr.Aui u0i / D u0i Aui D ai i tr A @xi i and that (for j < i ) @X tr A D tr.Aui uj0 / C tr.Auj u0i / D uj0 Aui C u0i Auj D aj i C aij : @xij Result (6.6) can be restated as @ tr.XA/ @ tr.AX/ D D A C A0 diag.a11 ; a22 ; : : : ; amm /: @X @X

(6.7)

Note that, regardless of whether X is unrestricted (but square) or symmetric, we have that ( @ tr.X/ 1; if j D i , D @xij 0; if j ¤ i , or, equivalently, that @ tr.X/ D I; @X as is evident from results (6.4)–(6.7) upon taking A D I.

15.7 The Chain Rule The chain rule can be very helpful in deriving the partial derivatives of a function f that is the composite of a function g and a vector of functions h, that is, of a function f whose value at an arbitrary point x is given by the formula f .x/ D gŒh.x/. The chain rule is stated (without proof) in the following theorem. Theorem 15.7.1 (chain rule). Let h D fhi g represent an n 1 vector of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Let g represent a function, defined on a set T , of a vector y D .y1 ; : : : ; yn /0 of n variables. Suppose that h.x/ 2 T for every x in S , and take f to be the composite function defined (on S ) by f .x/ D gŒh.x/. If h is continuously differentiable at an interior point c of S and if [assuming that h.c/ is an interior point of T ] g is continuously differentiable at h.c/, then f is continuously differentiable at c and Djf .c/ D

n X

Di gŒh.c/Dj hi .c/ D DgŒh.c/Dj h.c/:

(7.1)

iD1

Formula (7.1) can be reexpressed less formally as n

X @g @hi @f @g @h D D 0 ; @xj @yi @xj @y @xj iD1

(7.2)

15.7 The Chain Rule

307

where @g=@yi and @g=@y 0 are to be interpreted as having been evaluated at y D h.x/. Formula (7.1) or (7.2) can be recast as Df .c/ D

n X

Di gŒh.c/Dhi .c/ D DgŒh.c/Dh.c/

(7.3)

iD1

or

n

X @g @hi @f @g @h D D 0 : 0 0 @x @yi @x @y @x0

(7.4)

iD1

The result of Theorem 15.7.1 can be generalized by taking g D fgs g to be a p 1 vector of functions (defined on T ) of y and f D ffs g to be a p 1 vector of composite functions defined (on S ) by fs .x/ D gs Œh.x/ (s D 1; : : : ; p) or, equivalently, by f .x/ D gŒh.x/. If (in this more general context) h is continuously differentiable at an interior point c of S and g is continuously differentiable at h.c/, then f is continuously differentiable at c and Dj f .c/ D

n X

Di gŒh.c/Dj hi .c/ D DgŒh.c/Dj h.c/

(7.5)

iD1

or, equivalently,

n

X @g @hi @f @g @h D D 0 @xj @yi @xj @y @xj

(7.6)

iD1

[where @g=@yi and @g=@y 0 are to be interpreted as having been evaluated at y D h.x/]. Formula (7.5) or (7.6) can be recast as Df .c/ D

n X

Di gŒh.c/Dhi .c/ D DgŒh.c/Dh.c/

(7.7)

iD1

or

n

X @g @hi @f @g @h D D 0 : @x0 @yi @x0 @y @x0

(7.8)

iD1

Let us now consider a different generalization of the result of Theorem 15.7.1— one where H D fhis g is an n r matrix of functions, defined on S , of x; where g is a function, defined on a set T , of an n r matrix Y D fyis g of nr variables; where H.x/ 2 T for every x in S ; and where f is the composite function defined (on S ) by f .x/ D gŒH.x/. Suppose that the elements of H and Y are rearranged in the form of column vectors h and y, respectively, and that, for purposes of differentiation, g is reinterpreted as a function of y. If h or, equivalently, H is continuously differentiable at an interior point c of S and if [assuming that H.c/ is an interior point of T ] g is continuously differentiable at h.c/ or, equivalently, H.c/, then f is continuously differentiable at c and (at x D c) n

r

X X @g @his @f D @xj @yis @xj sD1 iD1

(7.9)

308


(as is evident from Theorem 15.7.1). Formula (7.9) can be rewritten as 0 @f @g @H : D tr @xj @Y @xj

(7.10)

[The quantities @g=@yis and @g=@Y , which appear in formulas (7.9) and (7.10), are to be interpreted as having been evaluated at Y D H.x/.]

15.8 First-Order Partial Derivatives of Determinants and Inverse and Adjoint Matrices Let X D fxij g represent an m m matrix of m2 “independent” variables (where m 2), and suppose that the range of X comprises all of Rmm . Denote by ij the cofactor of xij . Then, the function f of X defined (on Rmm ) by f .X/ D det.X/ is continuously differentiable at every X and @ det.X/ D ij : @xij

(8.1)

To see this, rearrange the elements of X in the form of an m2 -dimensional column vector x, and regard f as a function of x. Since each element of X is a continuously differentiable function of x, and since (according to Lemma 15.2.2) sums and products of continuously differentiable functions are continuously differentiable, it follows from the very definition of the determinant [given by expression (13.1.2)] Pthat f is continuously differentiable. Moreover, expanding det.X/ as det.X/ D m tD1 xit it [in accordance with result (13.5.1)], observing that i1 ; : : : ; im do not vary with xij , and recalling result (5.2), we find that m

X @xit @ det.X/ D it D ij : @xij @xij tD1

(8.2)

Result (8.1) indicates that the derivative of det.X/ with respect to X is the matrix of cofactors (of X) or, equivalently, that @ det.X/ D Œadj.X/0 : @X

(8.3)

Let us now extend our results to the differentiation of the determinant of a p p matrix F D ffis g of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Thus, the function to be differentiated is the function h of x defined (on S ) by h.x/ D detŒF .x/. It is convenient (for purposes of differentiating h) to introduce a function g of a p p matrix Y D fyis g of p 2 variables, defined (on Rpp ) by g.Y / D det.Y /, and to express h as the composite of g and F , so that h.x/ D gŒF .x/. Clearly (in light of our earlier results), g is continuously differentiable at every Y and

15.8 First-Order Partial Derivatives of Determinants and Inverse Matrices

309

@g D Œadj.Y /0 : @Y Thus, it follows from the chain rule [and in particular from result (7.10)] that if F is continuously differentiable at an interior point c of S , then h is continuously differentiable at c and (at x D c) @F @ det.F / : (8.4) D tr adj.F / @xj @xj Moreover, if F is nonsingular as well as continuously differentiable at c, then [in light of result (13.5.6)] @ det.F / @F : (8.5) D jF j tr F 1 @xj @xj Suppose now that S is the set of all x-values for which detŒF .x/ > 0 or is a subset of that set, and consider the differentiation of the function ` of x defined (on S ) by `.x/ D log detŒF .x/. To facilitate the differentiation, let g represent a function of a variable y defined (for y > 0) by g.y/ D log y, and express ` as the composite of g and h, so that `.x/ D gŒh.x/. Recall (from the calculus of a single variable) that g is continuously differentiable (at all points in its domain) and that (for y > 0) 1 @ log y D : @y y Applying the chain rule, we find [in light of results (8.4) and (8.5)] that if F is continuously differentiable at an interior point c of S (in which case h is continuously differentiable at c), then ` is continuously differentiable at c and (at x D c) @ log det.F / @F 1 @ det.F / 1 1 @F tr adj.F / D tr F : (8.6) D D @xj jF j @xj jF j @xj @xj Let us now consider result (8.6) in the special case where x is an m2 dimensional column vector whose elements are the same as those of an m m matrix X D fxij g of m2 “independent” variables and where F .x/ D X. Let c represent an interior point of S [where S consists of some or all x-values for which det.X/ > 0], and let uj represent the j th column of Im . It follows from result (5.3) that X is continuously differentiable at c and that (at x D c) @X=@xij D ui uj0 . We conclude that the function ` defined (on S ) by `.x/ D log det.X/ is continuously differentiable at c and that (at x D c) @ log det.X/ D tr.X1 ui uj0 / D uj0 X1 ui D yj i ; @xij

(8.7)

where yj i is the j i th element of X1 or, equivalently, the ij th element of .X1 /0 . Result (8.7) can be recast as

310


@ log det.X/ D .X1 /0 : @X

(8.8)

Formulas (8.3) and (8.8) [or (8.2) and (8.7)] are applicable when the m2 elements of X are independent variables. Suppose now that X D fxij g is an m m symmetric matrix, in which case formulas (8.3) and (8.8) are not applicable. For purposes of differentiation, the function f of X defined (on a set S comprising some or all m m symmetric matrices) by f .X/ D det.X/ and the function ` of X defined (on a set S comprising some or all m m symmetric matrices with positive determinants) by `.X/ D log det.X/ are by convention to be interpreted as functions of the [m.m C 1/=2]-dimensional column vector x whose elements xij (j i D 1; : : : ; m) are “independent” elements of X. Let c represent an interior point of S and c an interior point of S . Then, it follows from the results of Section 15.5 that X is continuously differentiable at c and c , implying that f is continuously differentiable at c and that ` is continuously differentiable at c . Moreover, we have [as a special case of result (8.4)] that (at x D c) @ det.X/ @X : D tr adj.X/ @xij @xij Making use of results (5.6) and (5.7) and denoting the j th column of Im by uj , we find that @X D trŒadj.X/ui u0i D u0i adj.X/ui tr adj.X/ @xi i and (for j < i ) @X D trŒadj.X/ui uj0 C trŒadj.X/uj u0i tr adj.X/ @xij D uj0 adj.X/ui C u0i adj.X/uj : Thus, denoting the cofactor of xij by ij (and recalling that the cofactor matrix of a symmetric matrix is symmetric), we have that ( @ det.X/ i i ; if j D i , (8.9a) D @xij (8.9b) 2 ij ; if j < i , or, equivalently, that @ det.X/ D 2 adj.X/ diag. 11 ; 22 ; : : : ; mm /: @X

(8.10)

Further, letting yij represent the ij th element of X1 , we find [in light of results (8.6) and (13.5.7)] that (at x D c ) ( @ log det.X/ yi i ; if j D i , (8.11a) D @xij (8.11b) 2yij ; if j < i ,

15.8 First-Order Partial Derivatives of Determinants and Inverse Matrices

311

or, equivalently, that @ log det.X/ D 2X1 diag.y11 ; y22 ; : : : ; ymm /: @X

(8.12)

Consider now the differentiation of the adjoint matrix of a p p matrix F D ffis g of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Let Fsi represent the .p 1/ .p 1/ submatrix of F obtained by striking out the sth row and the i th column (of F ), and let si D .1/sCi det.Fsi /. Then, by definition, si is the cofactor of fsi and hence the i sth element of adj.F /. It follows from our discussion of the differentiation of determinants [including result (8.4)] that if F is continuously differentiable at an interior point c of S , then si is continuously differentiable at c and (at x D c) @si @Fsi sCi : (8.13) D .1/ tr adj.Fsi / @xj @xj In other words, if F is continuously differentiable at c, then adj.F / is continuously differentiable at c, and the i sth element of the matrix @ adj.F /=@xj is given by expression (8.13). Suppose now that S is the set of all x-values for which F .x/ is nonsingular or is a subset of that set, and consider the differentiation of the inverse matrix F 1 . Denote by c any interior point (of S ) at which F is continuously differentiable. Then, si and det.F / are continuously differentiable at c, implying (in light of Lemma 15.2.2) that si =det.F / is continuously differentiable at c. Since (according to Corollary 13.5.4) si =det.F / equals the i sth element of F 1 , we conclude that F 1 is continuously differentiable at c. Moreover, F F 1 D I, implying (in light of Lemmas 15.4.3 and 15.4.1) that (at x D c) F

@F 1 @I @F 1 C F D D0 @xj @xj @xj

and hence F

@F 1 @F 1 D F : @xj @xj

(8.14)

Premultiplying both sides of equality (8.14) by F 1 , we obtain @F 1 @F 1 D F 1 F @xj @xj

(8.15)

as a formula for @F 1 =@xj . An alternative expression for the elements of @ adj.F /=@xj can [when S is restricted to x-values for which F .x/ is nonsingular] be obtained by making use of formula (4.9). Since (according to Corollary 13.5.4) adj.F / D jF jF 1, we have [in light of formula (8.5)] that

312


@ adj.F / @jF j 1 @F 1 D F C jF j @xj @xj @xj @F @F 1 F 1 C jF j F 1 D jF j tr F 1 F @xj @xj @F @F 1 D jF j tr F 1 F 1 F 1 : F @xj @xj

(8.16)

Formula (8.15) can be generalized by making use of result (4.6). Let A and B represent k p and p r matrices of functions (defined on the same set S as the elements of F ) of x. Suppose that A and B (as well as F ) are continuously differentiable at c. Then, AF 1 B is continuously differentiable at c and (at x D c) @.AF 1 B/ @B @F 1 @A 1 D AF 1 AF 1 F BC F B: @xj @xj @xj @xj

(8.17)

In the special case where (for x 2 S ) A.x/ and B.x/ are constant or (more generally) do not vary with xj , formula (8.17) simplifies to @F 1 @.AF 1 B/ D AF 1 F B: @xj @xj

(8.18)

15.9 Second-Order Partial Derivatives of Determinants and Inverse Matrices Let F D ffis g represent a p p matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Suppose that F .x/ is nonsingular for every x in S , and denote by c any interior point (of S ) at which F is twice continuously differentiable. Then, F is continuously differentiable at c. In fact, F is continuously differentiable at every point in some neighborhood N of c. In light of the results of Section 15.8, we have that det.F / and F 1 are continuously differentiable at every point in N and that (for x 2 N ) @ det.F / 1 @F D jF j tr F @xj @xj and

@F 1 @F 1 D F 1 F : @xj @xj

Moreover, @F =@xj is continuously differentiable at x D c. Consequently, @ det.F /=@xj and @F 1=@xj are continuously differentiable at c, and hence det.F / and F 1 are twice continuously differentiable at c. Further, making use of results (2.5) and (6.3), we find that (at x D c)

15.9 Second-Order Partial Derivatives of Determinants and Inverse Matrices

313

@2 det.F / @fjF j trŒF 1 .@F =@xj /g D @xi @xj @xi 2 1 @ F 1 @F 1 @F C tr F D jF j tr F F @xi @xj @xi @xj 1 @F 1 @F C jF j tr F tr F @xi @xj 2 @ F @F @F C tr F 1 tr F 1 D jF j tr F 1 @xi @xj @xi @xj @F 1 1 @F : (9.1) tr F F @xi @xj And, making use of result (4.6), we find that (at x D c) @2 F 1 @ŒF 1 .@F =@xj /F 1 D @xi @xj @xi @F @F 1 @2 F @F 1 @F 1 F 1 F 1 F @xj @xi @xi @xj @xi @xj @2 F 1 @F 1 @F 1 F F 1 D F F F 1 @xj @xi @xi @xj 1 @F 1 @F F F F 1 @xi @xj D F 1

D F 1

@2 F @F 1 @F 1 F 1 C F 1 F F @xi @xj @xi @xj @F 1 @F 1 C F 1 F F : @xj @xi

(9.2)

Suppose now that detŒF .x/ > 0 for every x in S . Then, in light of our previous results, log det.F / is continuously differentiable at every point in N and (for x 2 N ) @F @ log det.F / : D tr F 1 @xj @xj Thus (since F 1 and @F =@xj are continuously differentiable at c), @ log det.F /=@xj is continuously differentiable at c, and hence log det.F / is twice continuously differentiable at c. Moreover, making use of result (6.3), we find that (at x D c) @2 log det.F / @ trŒF 1 .@F =@xj / D @xi @xj @xi @2 F @F 1 @F C tr F 1 D tr F 1 F @xi @xj @xi @xj 2 @ F @F 1 @F tr F 1 : F D tr F 1 @xi @xj @xi @xj

(9.3)

314


15.10 Differentiation of Generalized Inverses Let F represent a p q matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Consider the q p matrix G of functions defined (on S ) by taking G.x/ to be any generalized inverse of F .x/. In the special case where q D p and S is composed exclusively of nonsingular matrices (and hence where G D F 1 ), G is continuously differentiable at every interior point of S at which F is continuously differentiable, and @G=@xj D G.@F =@xj /G (as discussed in Section 15.8). In this section, we consider the extent to which these results (and various other results on the differentiation of ordinary inverses) can be extended to the case where F is not necessarily nonsingular (and q not necessarily equal to p). One such extension is given by the following theorem. Theorem 15.10.1. Let F represent a p q matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Let c represent any interior point of S at which F is continuously differentiable. Suppose that F has constant rank, say r, on some neighborhood of c. Take i1 ; : : : ; ir to be integers chosen from the first p positive integers, 1; : : : ; p, in such a way that the i1 , : : : ; ir th rows of F .c/ are linearly independent, and take j1 ; : : : ; jr to be integers chosen from the first q positive integers, 1; : : : ; q, in such a way that the j1 , : : : ; jr th columns of F .c/ are linearly idependent. Further, take P to be any p p permutation matrix whose first r rows are the i1 , : : : ; ir th rows of Ip and Q to be any q q permutation matrix whose first r columns are the j1 , : : : ; jr th columns of Iq . Let B D P F Q, and partition B as B11 B12 ; BD B21 B22 where B11 is of dimensions r r. Then, rank.B11 / D rank.F / D r on some neighborhood N of c. And there exists a generalized inverse G of F such that 1 B 0 G D Q 11 P 0 0 on N . Moreover, G is continuously differentiable at c, and (at x D c) 1 1 @G @F B .@B11 =@xj /B11 0 D Q 11 G: P D G 0 0 @xj @xj With regard to Theorem 15.10.1, note (in light of the discussion of Section 9.2a) that F .c/ necessarily contains r linearly independent rows and r linearly independent columns, that B11 is the submatrix of F obtained by striking out all of the rows and columns of F except the i1 , : : : ; ir th rows and j1 , : : : ; jr th columns (or is an r r matrix obtained from that submatrix by permuting its rows and columns), and that B11 .c/ is nonsingular. In proving Theorem 15.10.1, it is convenient to make use of the following lemma.

15.10 Differentiation of Generalized Inverses

315

Lemma 15.10.2. Let F represent a p p matrix of functions, defined on a set S , of an m-dimensional column vector x. Suppose that F is nonsingular at an interior point c of S and that F is continuous at c. Then, F is nonsingular at all points in some neighborhood of c. Proof (of Lemma 15.10.2). It follows from the very definition of a determinant [given by formula (13.1.2)] and from standard results on the continuity of functions that det.F / is a continuous function of x (at x D c) and hence that lim detŒF .x/ D detŒF .c/:

x!c

Since detŒF .c/ ¤ 0, S contains a neighborhood N of c such that jdetŒF .x/ detŒF .c/j < jdetŒF .c/j for x 2 N or, equivalently, such that jdetŒF .c/j jdetŒF .c/ detŒF .x/j > 0 for x 2 N . And, since jdetŒF .c/j jdetŒF .x/j C jdetŒF .c/ detŒF .x/j or, equivalently, jdetŒF .x/j jdetŒF .c/j jdetŒF .c/ detŒF .x/j, we have that jdetŒF .x/j > 0 for x 2 N . We conclude that F .x/ is nonsingular for x 2 N . Q.E.D. Proof (of Theorem 15.10.1). Since (in light of Lemma 15.1.1) F is continuous at c (implying that B11 is continuous at c), it follows from Lemma 15.10.2 that B11 is nonsingular at all points in some neighborhood N1 of c. Let N2 represent a neighborhood of c on which rank.F / D r, and take N to be whichever of the two neighborhoods N1 and N2 has the smallest radius. Then, clearly, rank.B11 / D rank.F / D r for x 2 N . The existence of a generalized inverse G of F such that 1 B 0 P G D Q 11 0 0 on N follows from Theorem 9.2.3. That G is continuously differentiable at c and that (at x D c) 1 1 @G B .@B11 =@xj /B11 0 D Q 11 P 0 0 @xj follows from the results of Sections 15.4 and 15.8. To complete the proof, observe that @B=@xj D P .@F =@xj /Q, implying that @B 0 @F @B11 =@xj @B12 =@xj Q0 D P0 Q D P0 @B21 =@xj @B22 =@xj @xj @xj and hence that

1 @F B 0 @B11 =@xj G G D Q 11 @B21 =@xj 0 0 @xj 1 1 B .@B11 =@xj /B11 D Q 11 0

@B12 =@xj @B22 =@xj 0 P: 0

1 B11 0 P 0 0

Q.E.D.

316


According to Theorem 15.10.1, a sufficient condition for the existence of a generalized inverse of F that is continuously differentiable at c (where c is an interior point at which F is continuously differentiable) is that F have constant rank on some neighborhood of c. Is this condition necessary as well as sufficient? This question is answered (in the affirmative) by the corollary of the following lemma. Lemma 15.10.3. Let F represent a p q matrix of functions, defined on a set S , of an m-dimensional column vector x. And let c represent any interior point of S at which F is continuous. If there exists a generalized inverse of F that is continuous at x D c, then F has constant rank on some neighborhood of c. Proof. Suppose that there exists a generalized inverse, say G, of F that is continuous at x D c. Then, since sums and products of continuous functions are continuous, tr.F G/ is continuous at x D c. Moreover, according to result (10.2.1), tr.F G/ D rank.F /. Thus, rank.F / is continuous at x D c. Since rank.F / is integer-valued, we conclude that rank.F / is constant on some neighborhood of c. Q.E.D. Corollary 15.10.4. Let F represent a p q matrix of functions, defined on a set S , of an m-dimensional column vector x. Let c represent any interior point of S at which F is continuously differentiable. If there exists a generalized inverse of F that is continuously differentiable at x D c, then F has constant rank on some neighborhood of c. Proof. Suppose that there exists a generalized inverse, say G, of F that is continuously differentiable at c. Since (according to Lemma 15.1.1) continuously differentiable functions are continuous, F and G are both continuous at x D c. Thus, it follows from Lemma 15.10.3 that F has constant rank on some neighborhood of c. Q.E.D. Let F represent a p q matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Further, let c represent an interior point of S at which F is continuously differentiable, and suppose that F has constant rank on some neighborhood of c. Then, according to Theorem 15.10.1, there exists a generalized inverse G of F that is continuously differentiable at c and whose partial derivatives at x D c are given by the formula @F @G D G G: @xj @xj

(10.1)

Is formula (10.1) applicable to every generalized inverse G (of F ) that is continuously differentiable at c? Except in special cases, the answer is no. Suppose, for example, that F D 0 for all x in some neighborhood N of c and that G is chosen in such a way that G is continuously differentiable at c and @G=@xj is nonnull— since any q p matrix is a generalized inverse of a p q null matrix, such a choice is clearly possible. Then, the right side of equality (10.1) is null at x D c, while the left side is nonnull. The following lemma relates the partial derivatives (at x D c) of a generalized inverse of F to the partial derivatives of F itself—it does so in a less definitive way

15.10 Differentiation of Generalized Inverses

317

than formula (10.1), but in a way that is applicable to any generalized inverse (of F ) that is continuously differentiable at c. Lemma 15.10.5. Let F represent a p q matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Further, let c represent an interior point of S at which F is continuously differentiable, and (assuming that F has constant rank on some neighborhood of c) let G represent any generalized inverse of F that is continuously differentiable at c. Then (at x D c), F

@F @G F D F G GF : @xj @xj

(10.2)

Proof. Differentiating both sides of the equality F D F GF [with the help of result (4.6)], we find that (at x D c) @F @F @G @F D FG CF FC GF : @xj @xj @xj @xj

(10.3)

Premultiplying and postmultiplying both sides of equality (10.3) by F G and GF , respectively, gives FG

@F @G @F @F GF D F GF G GF C F GF F GF C F G GF GF @xj @xj @xj @xj @F @G @F D FG GF C F F C FG GF @xj @xj @xj

or, equivalently, F

@F @G F D F G GF : @xj @xj Q.E.D.

The following theorem generalizes result (10.1). Theorem 15.10.6. Let F , A, and B represent p q, k q, and p r matrices of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Let c represent an interior point of S at which F , A, and B are continuously differentiable. Suppose that there exists a neighborhood N of c such that (1) F has constant rank on N and (2) R.A/ R.F / and C.B/ C.F / for all x in N . Then, for any generalized inverse G of F , AGB is continuously differentiable at c, and (at x D c) @B @F @A @AGB D AG AG GB C GB: @xj @xj @xj @xj

(10.4)

Theorem 15.10.6 indicates that, even if the generalized inverse G is not continuously differentiable (at x D c) or the formula @G @F D G G @xj @xj

(10.5)

318


is not applicable to G, we can for purposes of obtaining the partial derivatives of AGB (at x D c) proceed as though G is continuously differentiable and formula (10.5) is valid. In the special case where (for x 2 S ) A.x/ and B.x/ are constant or (more generally) do not vary with xj , formula (10.4) simplifies to @.AGB/ @F D AG GB: @xj @xj

(10.6)

Preliminary to proving Theorem 15.10.6, it is convenient to establish the following theorem, which is of some interest in its own right. Theorem 15.10.7. Let F represent a p q matrix of functions, defined on a set S , of an m-dimensional column vector x, and let G represent a generalized inverse of F . Further, let c represent any interior point (of S ) at which F is continuously differentiable, and suppose that F has constant rank on some neighborhood of c. Then, there exists a second generalized inverse G (of F ) such that G is continuously differentiable (at c) and G .c/ D G.c/. Proof (of Theorem 15.10.7). According to Theorem 15.10.1, there exists a generalized inverse, say H, of F that is continuously differentiable at c. Let G D H C Z HF ZF H; where Z D G.c/ H.c/. Then, it follows from Theorem 9.2.7 that G is a generalized inverse of F and from the results of Section 15.4 that G is continuously differentiable (at c). Moreover, G .c/ D H.c/ C Z H.c/F .c/ŒG.c/ H.c/F .c/H.c/ D H.c/ C Z H.c/F .c/H.c/ C H.c/F .c/H.c/ D H.c/ C Z D G.c/ Q.E.D. Proof (of Theorem 15.10.6). According to Theorem 15.10.7, there exists a generalized inverse G of F such that G is continuously differentiable (at c) and G .c/ D G.c/. Moreover, according to Theorem 9.4.1, AGB D AG B for every x in N . Thus, it follows from the results of Section 15.4 that AGB is continuously differentiable at c and that (at x D c) @B @.AG B/ @G @A @.AGB/ D D AG CA BC G B @xj @xj @xj @xj @xj @B @G @A D AG CA BC GB: @xj @xj @xj To complete the proof, observe that A.c/ D LF .c/ and B.c/ D F .c/R for some matrices L and R and hence (in light of Lemma 15.10.5) that (at x D c)

15.11 Differentiation of Projection Matrices

A

319

@F @G @G B D LF F R D LF G G F R @xj @xj @xj @F @F D AG G B D AG GB: @xj @xj Q.E.D.

15.11 Differentiation of Projection Matrices Recall (from Section 14.12d) that PX;W represents the matrix X.X0 W X/ X0 W (where X is an n p matrix and W an n n matrix) and that, for an n n symmetric positive definite matrix W , PX;W is the projection matrix for C.X/ with respect to W . If the elements of W and/or X are functions of a vector, say z, of variables, then the differentiation (with respect to the elements of z) of PX;W may be of interest. The following theorem gives some results on the differentiation of PX;W and also on the differentiation of W PX;W and W W PX;W . Theorem 15.11.1. Let X represent an np matrix and W an nn symmetric positive definite matrix, and suppose that the elements of X and W are functions, defined on a set S , of a vector z D .z1 ; : : : ; zm /0 of m variables. Further, let c represent any interior point (of S ) at which X and W are continuously differentiable, and suppose that X has constant rank on some neighborhood of c. Then, PX;W , W PX;W , and W W PX;W are continuously differentiable at c, and (at z D c) @PX;W @zj

@.W PX;W / @zj

@.W W PX;W / @zj

@X 0 .X W X/ X0 W @zj 0 0 @X C X.X W X/ W .I PX;W / @zj @W C X.X0 W X/ X0 .I PX;W /; @zj

D .I PX;W /

D

@W @W 0 .I PX;W / .I PX;W / @zj @zj @X 0 C W .I PX;W / .X W X/ X0 W @zj 0 0 @X C W X.X W X/ W .I PX;W /; @zj @W .I PX;W / @zj @X 0 W .I PX;W / .X W X/ X0 W @zj 0 0 @X W X.X W X/ W .I PX;W /: @zj

(11.1)

(11.2)

0 D .I PX;W /

(11.3)

320


The results of Theorem 15.11.1 are special cases of various results encompassed in the following theorem {as is evident upon recalling (from Corollary 14.11.3) that rank.X0 W X/ D rank.X/ and (from Theorem 14.12.11) that 0 }. XŒ.X0 W X/ 0 X0 D X.X0 W X/ X0 and hence that W X.X0 W X/ X0 D PX;W Theorem 15.11.2. Let W , X, and Y represent r n, np, and q r matrices of functions, defined on a set S , of a vector z D .z1 ; : : : ; zm /0 of m variables. Let c represent any interior point of S at which W , X, and Y are continuously differentiable. Suppose that there exists a neighborhood N of c such that (1) Y W X has constant rank on N and (2) rank.X/ D rank.Y / D rank.Y W X/ for all z in N . Define K D X.Y W X/ Y . Then, K, KW , W KW , and W W KW are continuously differentiable at c, and (at z D c) @Y @K @X @W D .IKW / .Y W X/ Y CX.Y W X/ .IW K/K K; (11.4) @zj @zj @zj @zj @.KW / @X D .I KW / .Y W X/ Y W @zj @zj @Y @W C X.Y W X/ W .I KW / C K .I KW /; (11.5) @zj @zj @.W KW / @W @W D .I W K/ .I KW / @zj @zj @zj @X C W .I KW / .Y W X/ Y W @zj @Y C W X.Y W X/ W .I KW /; @zj

(11.6)

@.W W KW / @W D .I W K/ .I KW / @zj @zj @W .Y W X/ Y W @zj @Y W X.Y W X/ W .I KW /: @zj

W .I KW /

(11.7)

Proof. In light of the results of Section 15.4, we have that Y W X is continuously differentiable at c. And, in light of Corollary 4.4.7, we have that R.X/ D R.Y W X/ and C.Y / D C.Y W X/ for all z in N . Applying Theorem 15.10.6 (with F D Y W X, A D X, and B D Y ), we find that K is continuously differentiable at c and that (at z D c) @Y @K D X.Y W X/ @zj @zj X.Y W X/

@.Y W X/ @X .Y W X/ Y C .Y W X/ Y : (11.8) @zj @zj


321

Moreover, @X @W @Y @.Y W X/ D YW CY XC W X: @zj @zj @zj @zj

(11.9)

Upon substituting expression (11.9) for @.Y W X/=@zj in expression (11.8), we obtain @Y @K @X @W D X.Y W X/ KW .Y W X/ Y K K @zj @zj @zj @zj @Y @X X.Y W X/ WKC .Y W X/ Y @zj @zj @Y @X @W D .I KW / .Y W X/ Y C X.Y W X/ .I W K/ K K; @zj @zj @zj thereby validating formula (11.4). Further, since K is continuously differentiable at c, it follows from the results of Section 15.4 that KW , W KW , and W W KW are continuously differentiable at c and that (at z D c) @W @K @.KW / DK C W; @zj @zj @zj @.W KW / @.KW / @W DW C KW ; @zj @zj @zj @.W W KW / @W @.W KW / D : @zj @zj @zj

(11.10) (11.11) (11.12)

Upon substituting expression (11.4) for @K=@zj in expression (11.10), we obtain (after a little simplification) expression (11.5). Similarly, upon substituting expression (11.5) for @.KW /=@zj in expression (11.11), we obtain expression (11.6), which when substituted for @.W KW /=@zj in expression (11.12) gives expression (11.7). Q.E.D. According to Theorem 15.11.1, a sufficient condition for PX;W to be continuously differentiable at c (where c is an interior point at which X and W are continuously differentiable) is that X have constant rank on some neighborhood of c. Is this condition necessary as well as sufficient? This question is answered (in the affirmative) by the corollary of the following lemma. Lemma 15.11.3. Let X represent an n p matrix and W an n n symmetric positive definite matrix, and suppose that the elements of X and W are functions, defined on a set S , of an m-dimensional column vector z. And let c represent any interior point of S at which X and W are continuous. If PX;W is continuous at z D c, then X has constant rank on some neighborhood of c. Proof. Suppose that PX;W is continuous at z D c. Then, tr.PX;W / is continuous at z D c. Moreover, since (according to Theorem 14.12.11) PX;W is idempotent and rank.X/ D rank.PX;W /, we have (in light of Corollary 10.2.2) that rank.X/ D tr.PX;W /. Thus, rank.X/ is continuous at z D c. Since rank.X/ is integer-valued, we conclude that rank.X/ is constant on some neighborhood of c. Q.E.D.

322


Corollary 15.11.4. Let X represent an np matrix and W an nn symmetric positive definite matrix, and suppose that the elements of X and W are functions, defined on a set S , of an m-dimensional column vector z. Let c represent any interior point of S at which X and W are continuously differentiable. If PX;W is continuously differentiable at c, then X has constant rank on some neighborhood of c. Proof. Suppose that PX;W is continuously differentiable at c. Since (according to Lemma 15.1.1) continuously differentiable functions are continuous, X, W , and PX;W are all continuous at z D c. Thus, it follows from Lemma 15.11.3 that X has constant rank on some neighborhood of c. Q.E.D. In the special case where X is a matrix of constants, or, more generally, where (for z 2 S ) X.z/ does not vary with zj , formulas (11.1)–(11.3) reduce to @PX;W @zj @.W PX;W / @zj @.W W PX;W / @zj

D X.X0 W X/ X0 D

@W .I PX;W /; @zj

@W @W 0 .I PX;W / .I PX;W /; @zj @zj

0 D .I PX;W /

@W .I PX;W /: @zj

(11.13) (11.14) (11.15)

Formulas (11.1)–(11.3) can be expressed in a more general form by making use of the following lemma. Lemma 15.11.5. Let X represent an n p matrix and W an n n symmetric positive definite matrix, and suppose that the elements of X and W are functions, defined on a set S , of a vector z D .z1 ; : : : ; zm /0 of m variables. Let c represent any interior point (of S ) at which W and X are continuously differentiable, and suppose that X has constant rank on some neighborhood of c. Further, let B represent any p n matrix such that X0 W XB D X0 W . Then, at z D c,

@PX;W @zj

0 @W @X 0 W PX;W D .I PX;W / P C W .I PX;W / B: @zj X;W @zj

(11.16)

Proof. Recall (from Theorem 14.12.11) that 0 PX;W WX D WX

(for all z in S ) and (from Theorem 15.11.1) that PX;W is continuously differentiable at c. Thus (at z D c) 0 @.PX;W @PX;W 0 W X/ @.W X/ @.W X/ 0 D D PX;W C W X; @zj @zj @zj @zj so that


323

@PX;W 0 @.W X/ 0 W X D .I PX;W / @zj @zj @W @X 0 0 D .I PX;W / X C .I PX;W /W ; @zj @zj implying that

@PX;W @zj

0 @W @X 0 0 W XB D .I PX;W / XB C .I PX;W /W B: @zj @zj

(11.17)

Moreover, by making use of Parts (2) and (3) of Theorem 14.12.11, equality (11.17) can be reexpressed as equality (11.16). Q.E.D. 0 /.@W =@zj /PX;W involves Since neither .@PX;W =@zj /0 W PX;W nor .I PX;W B, it follows from Lemma 15.11.5 that W .I PX;W /.@X=@zj /B is invariant to the choice of B. Thus, since the choices for B include .X0 W X/ X0 W and Œ.X0 W X/ 0 X0 W , we have that W .I PX;W /

@X 0 @X .X W X/ X0 W D W .I PX;W / B @zj @zj

and W .I PX;W /

(11.18)

@X @X Œ.X0 W X/ 0 X0 W D W .I PX;W / B; @zj @zj

which [since W .I PX;W / is symmetric] is equivalent to W X.X0 W X/

@X 0 @X 0 W .I PX;W / D W .I PX;W / B : @zj @zj

(11.19)

Moreover, upon premultiplying both sides of equalities (11.18) and (11.19) by W 1, we find that .I PX;W /

@X 0 @X .X W X/ X0 W D .I PX;W / B @zj @zj

(11.20)

and X.X0 W X/

@X @zj

0

@X 0 W .I PX;W / D W 1 W .I PX;W / B : @zj

(11.21)

By using results (11.18)–(11.21) [and writing PX;W W 1 for X.X0 W X/ X0 ], formulas (11.1)–(11.3) can be reexpressed as @PX;W @zj

@X @X 0 1 W .I PX;W / D .I PX;W / BCW B @zj @zj @W C PX;W W 1 .I PX;W /; @zj

(11.22)

324


@.W PX;W / @zj

D

@W @W 0 .I PX;W / .I PX;W / @zj @zj @X @X 0 C W .I PX;W / B C W .I PX;W / B ; (11.23) @zj @zj

@.W W PX;W / @zj

0 D .I PX;W /

@W .I PX;W / @zj W .I PX;W /

@X B @zj

@X 0 W .I PX;W / B @zj

(11.24)

(where B is any p n matrix such that X0 W XB D X0 W ).

15.12 Evaluation of Some Multiple Integrals Let x D .x1 ; : : : ; xn /0 represent a (column) vector of n variables x1 ; : : : ; xn . Further, let W represent an nn symmetric positive definite matrix, let A represent an n n matrix, and let c and k represent n 1 vectors. Consider the evaluation of the following three integrals: Z expŒ.1=2/.x c/0 W .x c/ dx; Rn Z .k0 x/ expŒ.1=2/.x c/0 W .x c/ dx; n R Z .x c/0 A.x c/ expŒ.1=2/.x c/0 W .x c/ dx Rn

R [where the symbol Rn f .z/ dz is used to denote the integral of a function f .z/ of a (column) vector z of n variables over all of Rn ]. These three integrals are of considerable importance in probability and statistics, where they arise in connection with the multivariate normal distribution (e.g., Searle 1971, chap. 2). They can be evaluated by introducing a suitable change of variables and by recalling (from the integral calculus of a single variable) that Z 1 2 e x =2 dx D .2/1=2; 1 Z 1 2 xe x =2 dx D 0; Z 1 1 2 x 2 e x =2 dx D .2/1=2: 1

According to Corollary 14.3.13, there exists an n n nonsingular matrix P such that W D P 0 P . Define

15.12 Evaluation of Some Multiple Integrals

325

z D P .x c/; 1

so that x D P z C c. Further, let J represent the Jacobian of P 1 z C c (where P 1 z C c is regarded as a vector of functions of z). Then, observing [in light of result (3.10)] that @.P 1 z C c/ D P 1; @ z0 we find that J D jP 1 j D jP j1: And, since jW j D jP 0 P j D jP j2, we have that J D ˙jW j1=2 : Thus, upon making use of standard results on changes of variables (e.g., Bartle 1976, sec. 45), we find that Z expŒ.1=2/.x c/0 W .x c/ dx Rn Z D expŒ.1=2/z0 z jJ j dz Rn Z 1 Z 1 2 2 e z1 =2 dz1 e zn =2 dzn D jW j1=2 1

1

D .2/n=2 jW j1=2:

(12.1)

Further, letting di represent the i th element of the 1 n vector k0 P 1 [and R1 2 observing that (for i D 1; : : : ; n) 1 zi e zi =2 dzi D 0], we find that Z .k0 x/ expŒ.1=2/.x c/0 W .x c/ dx Rn Z 0 D .k c/ expŒ.1=2/.x c/0 W .x c/ dx n R Z C k0 .x c/ expŒ.1=2/.x c/0 W .x c/ dx Rn Z 0 n=2 1=2 D .k c/.2/ jW j C .k0 P 1 z/ expŒ.1=2/z0 z jJ j dz Rn

0

D .k c/.2/

n=2

jW j

C jW j

1=2

1=2

n X iD1

D .k0 c/.2/n=2 jW j1=2:

Z di

1 1

zi e

zi2 =2

dzi

YZ s¤i

1 1

2

e zs =2 dzs (12.2)

And, letting bij represent the ij th element of the n n matrix .P 1 /0 AP 1 and recalling Lemma 5.2.1 [and observing that .P 1 /0 D .P 0 /1 ], we find that

326


Z Rn

.x c/0 A.x c/ expŒ.1=2/.x c/0 W .x c/ dx Z D Œz0 .P 1 /0 AP 1 z expŒ.1=2/z0 z jJ j dz n R Z X D jW j1=2 bij zi zj expŒ.1=2/z0 z dz Rn

i;j

D jW j1=2 C

X i;j ¤i

D jW j1=2

X i

Z

bij X

Z bi i

1 1

1

1

zi e

YZ

2

zi2 e zi =2 dzi

zi2 =2

Z dzi

s¤i 1

1

zj e

1 1

zj2 =2

2

e zs =2 dzs dzj

Y Z s¤i;j

1

e

zs2 =2

1

dzs

bi i .2/1=2 .2/.n1/=2

i

D .2/

n=2

jW j1=2 trŒ.P 1 /0 AP 1

D .2/n=2 jW j1=2 trŒAP 1 .P 1 /0 D .2/n=2 jW j1=2 tr.AW 1 /:

(12.3)

Formulas (12.1)–(12.3) can be regarded as special cases of the formula given by the following theorem. Theorem 15.12.1. Let B represent an nn symmetric positive definite matrix and A an n n matrix, let b represent an n 1 vector, and let b0 and a0 represent scalars, Then, Z .a0 C a0 x C x0 Ax/ expŒ.b0 C b0 x C x0 Bx/ dx Rn

D .1=2/ n=2 jBj1=2 expŒ.1=4/b0 B1 b b0 Œtr.AB1 / a0 B1 b C .1=2/b0 B1AB1 b C 2a0 :

(12.4)

Proof. It is easy to show that b0 C b0 x C x0 Bx D .1=2/.x c/0 W .x c/ C f; where W D 2B, c D .1=2/B1 b, and f D b0 .1=4/b0 B1 b. It is also easy to show that a0 C a0 x C x0 Ax D .x c/0 A.x c/ C k0 x C g; where k D a .1=2/.A C A0 /B1 b and g D a0 .1=4/b0 B1AB1 b. Thus, making use of results (12.1)–(12.3) [and observing that b0 B1A0 B1 b D 0 1 0 1 0 .b B A B b/ D b0 B1AB1 b], we find that

Exercises

Z Rn

327

.a0 C a0 x C x0 Ax/ expŒ.b0 C b0 x C x0 Bx dx Z D e f Œg C k0 x C .x c/0 A.x c/ Rn

expŒ.1=2/.x c/0 W .x c/ dx

D e f Œg.2/n=2 jW j1=2 C .k0 c/.2/n=2 jW j1=2 C .2/n=2 jW j1=2 tr.AW 1 / D .2/n=2 jW j1=2 e f Œg C .k0 c/ C tr.AW 1 / D .2/n=2 2n=2 jBj1=2 expŒ.1=4/b0 B1 b b0 Œa0 .1=4/b0 B1AB1 b .1=2/a0 B1 b C .1=4/b0 B1.A C A0 /B1 b C .1=2/ tr.AB1 / D .1=2/ n=2 jBj1=2 expŒ.1=4/b0 B1 b b0 Œtr.AB1 / a0 B1 b C .1=2/b0 B1AB1 b C 2a0 : Q.E.D.

Exercises Section 15.1 1. Using the result of Part (c) of Exercise 6.2, show that every neighborhood of a point x in Rm1 is an open set. 2. Let f represent a function, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables, suppose that S contains at least some interior points, and let c represent an arbitrary one of those points. Verify that if f is k times continuously differentiable at c, then f is k times continuously differentiable at every point in some neighborhood of c. 3. Let X D fxij g represent an m n matrix of mn variables, and let x represent an mn-dimensional column vector obtained by rearranging the elements of X (in the form of a column vector). Further, let S represent a set of X-values, and let S represent the corresponding set of x-values (i.e., the set obtained by rearranging the elements of each m n matrix in S in the form of a column vector). Verify that an mn-dimensional column vector is an interior point of S if and only if it is a rearrangement of an m n matrix that is an interior point of S . 4. Let f represent a function whose domain is a set S in Rm1 (that contains at least some interior points). Show that the Hessian matrix Hf of f is the gradient matrix of the gradient vector .Df /0 of f .

328


Section 15.2 5. Let g represent a function, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables, let S D fx 2 S W g.x/ ¤ 0g, and let c represent any interior point of S at which g is continuously differentiable. Use Lemma 15.2.2 and the ensuing discussion [including formulas (2.16) and (2.8)] to show that (for any positive integer k) g k is continuously differentiable at c and @g k @g D kg k1 ; @xj @xj thereby generalizing formula (2.8) and establishing that formula (2.16) is valid for negative (as well as positive) values of k. Section 15.4 6. Let F represent a p p matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Let c represent any interior point of S at which F is continuously differentiable. Show that if F is idempotent at all points in some neighborhood of c, then (at x D c) F

@F F D 0: @xj

7. Let g represent a function, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables, and let f represent a p 1 vector of functions (defined on S ) of x. Let c represent any interior point (of S ) at which g and f are continuously differentiable. Show that gf is continuously differentiable at c and that (at x D c) @g @f @.gf / D f 0 C g 0: @x0 @x @x Section 15.6 8. (a) Let X D fxij g represent an m n matrix of mn “independent” variables, and suppose that X is free to range over all of Rmn. (1) Show that, for any p m and n p matrices of constants A and B, @ tr.AXB/ D A0 B 0 : @X [Hint. Observe that tr.AXB/ D tr.BAX/.] (2) Show that, for any m- and n-dimensional column vectors a and b, @.a0 Xb/ D ab0 : @X [Hint. Observe that a0 Xb D tr.a0 Xb/.]

Exercises

329

(b) Suppose now that X is a symmetric (but otherwise unrestricted) matrix (of dimensions m m). (1) Show that, for any p m and m p matrices of constants A and B, @ tr.AXB/ D C C C 0 diag.c11 ; c22 ; : : : ; cmm /; @X where C D fcij g D BA. (2) Show that, for any m-dimensional column vectors a D fai g and b D fbi g, @.a0 Xb/ D ab0 C ba0 diag.a1 b1 ; a2 b2 ; : : : ; am bm /: @X 9. (a) Let X D fxst g represent an m n matrix of “independent” variables, and suppose that X is free to range over all of Rmn. Show that, for any n m matrix of constants A, @ trŒ.AX/2 D 2.AXA/0 : @X (b) Let X D fxst g represent an m m symmetric (but otherwise unrestricted) matrix of variables. Show that, for any m m matrix of constants A, @ trŒ.AX/2 D 2ŒB C B0 diag.b11 ; b22 ; : : : ; bmm /; @X where B D fbst g D AXA. 10. Let X D fxij g represent an m m matrix of “independent” variables, and suppose that X is free to range over all of Rmm. Show that, for k D 2; 3; : : : ; @ tr.Xk / D k.X0 /k1; @X thereby generalizing result (2.17). 11. Let X D fxst g represent an m n matrix of “independent” variables, and suppose that X is free to range over all of Rmn. (a) Show that, for any m m matrix of constants A, @ tr.X0 AX/ D .A C A0 /X: @X (b) Show that, for any n n matrix of constants A, @ tr.XAX0 / D X.A C A0 /: @X

330


(c) Show (in the special case where n D m) that, for any m m matrix of constants A, @ tr.XAX/ D .AX/0 C .XA/0 : @X (d) Use the formulas from Parts (a)–(c) to devise simple formulas for @ tr.X0 X/=@X, @ tr.XX0 /=@X, and (in the special case where n D m) @ tr.X2 /=@X. Section 15.7 12. Let X D fxij g represent an m m matrix. Let f represent a function of X defined on a set S comprising some or all m m symmetric matrices. Suppose that, for purposes of differentiation, f is to be interpreted as a function of the [m.m C 1/=2]-dimensional column vector x whose elements are xij (j i D 1; : : : ; m). Suppose further that there exists a function g, whose domain is a set T of not-necessarily-symmetric matrices that contains S as a proper subset, such that g.X/ D f .X/ for X 2 S , so that g is a function of X and f is the function obtained by restricting the domain of g to S . Define S D fx W X 2 S g. Let c represent an interior point of S , and let C represent the corresponding value of X. Show that if C is an interior point of T and if g is continuously differentiable at C, then f is continuously differentiable at c and that (at x D c) 0 @g @g @g @g @g @f D C : diag ; ;:::; @X @X @X @x11 @x22 @xmm 13. Let h D fhi g represent an n 1 vector of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Let g represent a function, defined on a set T , of a vector y D .y1 ; : : : ; yn /0 of n variables. Suppose that h.x/ 2 T for every x in S , and take f to be the composite function defined (on S ) by f .x/ D gŒh.x/. Show that if h is twice continuously differentiable at an interior point c of S and if [assuming that h.c/ is an interior point of T ] g is twice continuously differentiable at h.c/, then f is twice continuously differentiable at c and Hf .c/ D ŒDh.c/0 HgŒh.c/Dh.c/ C

n X

Di gŒh.c/Hhi.c/:

iD1

Section 15.8 14. Let X D fxij g represent an m m matrix of m2 “independent” variables (where m 2), and suppose that the range of X comprises all of Rmm. Show that (for any positive integer k) the function f defined (on Rmm ) by f .X/ D jXjk is continuously differentiable at every X and that

Exercises

331

@jXjk D kjXjk1 Œadj.X/0 : @X 15. Let F D ffis g represent a p p matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Let c represent any interior point (of S ) at which F is continuously differentiable. Use the results of Exercise 13.10 to show that (a) if rankŒF .c/ D p 1, then (at x D c) @F @ det.F / D ky 0 z; @xj @xj where z D fzs g and y D fyi g are any nonnull p-dimensional vectors such that F .c/z D 0 and ŒF .c/0 y D 0 and where [letting is represent the cofactor of fis .c/] k is a scalar that is expressible as k D is =.yi zs / for any i and s such that yi ¤ 0 and zs ¤ 0; and (b) if rankŒF .c/ p 2, then (at x D c) @ det.F / D 0: @xj 16. Let X D fxst g represent an m n matrix of “independent” variables, let A represent an m m matrix of constants, and suppose that the range of X is a set S comprising some or all X-values for which det.X0 AX/ > 0. Show that log det.X0 AX/ is continuously differentiable at any interior point C of S and that (at X D C) @ log det.X0 AX/ D AX.X0 AX/1 C Œ.X0 AX/1 X0 A0 : @X 17. (a) Let X represent an m n matrix of “independent” variables, let A and B represent q m and nq matrices of constants, and suppose that the range of X is a set S comprising some or all X-values for which det.AXB/ > 0. Show that log det.AXB/ is continuously differentiable at any interior point C of S and that (at X D C) @ log det.AXB/ D ŒB.AXB/1A0 : @X (b) Suppose now that X D fxij g is an m m symmetric matrix; that A and B are q m and m q matrices of constants; that, for purposes of differentiating any function of X, the function is to be interpreted as a function of the column vector x whose elements are xij (j i D 1; : : : ; m); and that the range of x is a set S comprising some or all xvalues for which det.AXB/ > 0. Show that log det.AXB/ is continuously differentiable at any interior point c (of S ) and that (at x D c) @ log det.AXB/ D K C K0 diag.k11 ; k22 ; : : : ; kqq /; @X where K D fkij g D B.AXB/1A.

332


18. Let F D ffis g represent a p p matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables, and let A and B represent q p and p q matrices of constants. Suppose that S is the set of all x-values for which F .x/ is nonsingular and detŒAF 1.x/B > 0, or is a subset of that set. Show that if F is continuously differentiable at an interior point c of S , then log det.AF 1 B/ is continuously differentiable at c and (at x D c) @ log det.AF 1 B/ 1 1 1 1 @F : D tr F B.AF B/ AF @xj @xj 19. Let A and B represent q m and m q matrices of constants. (a) Let X represent an m m matrix of m2 “independent” variables, and suppose that the range of X is a set S comprising some or all X-values for which X is nonsingular and det.AX1 B/ > 0. Use the result of Exercise 18 to show that log det.AX1 B/ is continuously differentiable at any interior point C of S and that (at X D C) @ log det.AX1 B/ D ŒX1 B.AX1 B/1AX1 0 : @X (b) Suppose now that X D fxij g is an m m symmetric matrix; that, for purposes of differentiating any function of X, the function is to be interpreted as a function of the column vector x whose elements are xij (j i D 1; : : : ; m); and that the range of x is a set S comprising some or all x-values for which X is nonsingular and det.AX1 B/ > 0. Use the result of Exercise 18 to show that log det.AX1 B/ is continuously differentiable at any interior point c of S and that (at x D c) @ log det.AX1 B/ D K K0 C diag.k11 ; k22 ; : : : ; kqq /; @X where K D fkij g D X1 B.AX1 B/1AX1. 20. Let F D ffis g represent a p p matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Let c represent any interior point (of S ) at which F is continuously differentiable. By, for instance, using the result of Part (b) of Exercise 13.10, show that if rankŒF .c/ p 3, then @ adj.F / D 0: @xj 21. (a) Let X D fxij g represent an m m matrix of m2 “independent” variables, and suppose that the range of X is a set S comprising some or all Xvalues for which X is nonsingular. Show that (when the elements of X1

Exercises

333

are regarded as functions of X) X1 is continuously differentiable at any interior point C of S and that (at X D C) @X1 D yi zj0 ; @xij where yi represents the i th column and zj0 the j th row of X1. (b) Suppose now that X D fxij g is an m m symmetric matrix; that, for purposes of differentiating a function of X, the function is to be interpreted as a function of the column vector x whose elements are xij (j i D 1; : : : ; m); and that the range of x is a set S comprising some or all x-values for which X is nonsingular. Show that X1 is continuously differentiable at any interior point c of S and that (at x D c) ( @X1 yi yi0 ; if j D i , D @xij yi yj0 yj yi0 ; if j < i (where yi represents the i th column of X1 ). Section 15.9 22. Let X D fxij g represent an m m matrix of m2 “independent” variables. Suppose that the range of X is a set S comprising some or all X-values for which X is nonsingular, and let C represent an interior point of S . Denote the ij th element of X1 by yij , the j th column of X1 by yj , and the i th row of X1 by z0i . (a) Show that X1 is twice continuously differentiable at C and that (at X D C) @2 X1 D yjs yi z0t C y ti ys zj0 : @xij @xst (b) Suppose that det.X/ > 0 for every X in S . Show that log det.X/ is twice continuously differentiable at C and that (at X D C) @2 log det.X/ D y ti yjs : @xij @xst 23. Let F D ffis g represent a p p matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. For any nonempty set T D ft1 ; : : : ; ts g, whose members are integers between 1 and m, inclusive, define D.T / D @s F =@x t1 @x ts . Let k represent a positive integer and, for i D 1; : : : ; k, let ji represent an arbitrary integer between 1 and m, inclusive. (a) Suppose that F is nonsingular for every x in S , and denote by c any interior point (of S ) at which F is k times continuously differentiable. Show that F 1 is k times continuously differentiable at c and that (at x D c)

334


@k F 1 @xj1 @xjk D

k X X

.1/r F 1 D.T1 /F 1 D.T2 / F 1 D.Tr /F 1;

(E.1)

rD1 T1 ;:::;Tr

where T1 ; : : : ; Tr are r nonempty mutually exclusive and exhaustive subsets of fj1 ; : : : ; jk g (and where the second summation is over all possible choices for T1 ; : : : ; Tr ). (b) Suppose that det.F / > 0 for every x in S , and denote by c any interior point (of S ) at which F is k times continuously differentiable. Show that log det.F / is k times continuously differentiable at c and that (at x D c) @k log det.F / @xj1 @xjk D

k X X

.1/rC1 trŒF 1 D.T1 /F 1 D.T2 / F 1 D.Tr /;

(E.2)

rD1 T1 ;:::;Tr

where T1 ; : : : ; Tr are r nonempty mutually exclusive and exhaustive subsets of fj1 ; : : : ; jk g with jk 2 Tr (and where the second summation is over all possible choices for T1 ; : : : ; Tr ). Section 15.10 24. Let X D fxij g represent an m m symmetric matrix, and let x represent the [m.m C 1/=2]-dimensional column vector whose elements are xij (j i D 1; : : : ; m). Define S to be the set of all x-values for which X is nonsingular and S to be the set of all x-values for which X is positive definite. Show that S and S are both open sets. Section 15.11 25. Let X represent an n p matrix of constants, and let W represent an n n symmetric positive definite matrix whose elements are functions, defined on a set S , of a vector z D .z1 ; : : : ; zm /0 of m variables. Further, let c represent any interior point (of S ) at which W is twice continuously differentiable. Show that W W PX;W is twice continuously differentiable at c and that (at x D c)

Bibliographic and Supplementary Notes

335

@2 .W W PX;W / @zi @zj @2 W .I PX;W / @zi @zj @W @W 0 .I PX;W / X.X0 W X/ X0 .I PX;W / @zi @zj @W @W 0 Œ.I PX;W / X.X0 W X/ X0 .I PX;W /0 : @zi @zj

0 D .I PX;W /

26. Provide an alternative derivation of formula (11.23) for @.W PX;W /=@zj by using Part (60 ) of Theorem 14.12.11 to obtain the representation @.W PX;W / @zj

0 D PX;W W

@PX;W @zj

0 C PX;W

@PX;W 0 @W PX;W C W PX;W @zj @zj

and by then making use of result (11.16).

Bibliographic and Supplementary Notes Dwyer (1967) played a leading role in the development of matrix differentiation. Book-length treatments of matrix differentiation have been authored by Rogers (1980) and by Magnus and Neudecker (1988). §1. For a more general and complete discussion of the terminology and alternative notations employed in matrix differentiation, refer to Magnus and Neudecker (1988, chaps. 4–10). §2 and §3. Sections 2 and 3 can be read without first reading Section 1e. §10 and §11. Many of the results presented in Sections 10 and 11 are adaptations of results presented by Hearon and Evans (1968), Golub and Pereyra (1973), and Rogers (1980, chap. 16). §12. The presentation in Section 12 is modeled after that of Graybill (1983, secs. 10.3–10.5).

16 Kronecker Products and the Vec and Vech Operators

Some (though by no means all) of the partitioned matrices encountered in statistics are expressible in terms of two (or more) matrices (of relatively small dimensions) in the form of something called a Kronecker product. In Section 16.1, the definition of a Kronecker product of matrices is given, and a number of results on Kronecker products are presented. These results can (when applicable) be exploited for computational (and other) purposes. In subsequent sections of this chapter, the notion introduced in Chapter 15 (in connection with the differentiation of a function of a matrix) of rearranging the (nonredundant) elements of a matrix in the form of a column vector is formalized by introducing something called the vec or vech operator. A number of results on the vec or vech operator are presented. Kronecker products appear in many of these results.

16.1 The Kronecker Product of Two or More Matrices: Definition and Some Basic Properties The Kronecker product of two matrices, say an m n matrix A D faij g and a p q matrix B D fbij g, is denoted by the symbol A ˝ B and is defined to be the mp nq matrix 0 1 a11 B a12 B : : : a1n B B a21 B a22 B : : : a2n B C B C A˝B DB : :: :: C @ :: : : A am1 B am2 B : : : amn B obtained by replacing each element aij of A with the p q matrix aij B. Thus, the Kronecker product of A and B is a partitioned matrix, comprising m rows and n columns of p q dimensional blocks, the ij th of which is aij B. Clearly, each element of A ˝ B is the product of an element of A and an element of B. Specifically, the element that appears in the [p.i 1/ C r]th row and [q.j 1/ C s]th column of A ˝ B is the rsth element aij brs of aij B.

338


Note that, unlike the “ordinary” product AB of A and B (which is defined only in the special case where p D n), the Kronecker product A ˝ B is defined regardless of the dimensions of A and B. Note also that, while the Kronecker product B ˝ A of B and A is of the same dimensions (mp nq) as A ˝ B, it is only in special cases that B ˝ A equals A ˝ B. For example, if m D n D q D 2 and p D 3, then 0

a11 b11 B a11 b21 B B a11 b31 A˝B DB B a21 b11 B @ a21 b21 a21 b31 0 b11 a11 B b11 a21 B B b21 a11 B˝ADB B b21 a21 B @ b31 a11 b31 a21

a11 b12 a11 b22 a11 b32 a21 b12 a21 b22 a21 b32

a12 b11 a12 b21 a12 b31 a22 b11 a22 b21 a22 b31

b11 a12 b11 a22 b21 a12 b21 a22 b31 a12 b31 a22

b12 a11 b12 a21 b22 a11 b22 a21 b32 a11 b32 a21

1 a12 b12 a12 b22 C C a12 b32 C C; a22 b12 C C a22 b22 A a22 b32

1 b12 a12 b12 a22 C C b22 a12 C C: b22 a22 C C b32 a12 A b32 a22

In some presentations, A ˝ B is referred to as the direct product or the tensor product of A and B rather than the Kronecker product. And A ˝ B and B ˝ A are sometimes referred to, respectively, as the right and left Kronecker (or direct or tensor) products of A and B. For any scalar k and any m n matrix A, k ˝ A D A ˝ k D kA;

(1.1)

as is evident from the very definition of a Kronecker product. And, letting a D fai g and b D fbj g represent column vectors of dimensions m and n, respectively, it is easy to verify that 0 1 a1 b B a2 b C B C (1.2) a˝bDB : C @ :: A am b (i.e., that a ˝ b is an mn-dimensional partitioned column vector comprising m subvectors, the i th of which is ai b); that a0 ˝ b0 D .a1 b0 ; a2 b0 ; : : : ; am b0 /

(1.3)

(i.e., that a0 ˝ b0 is an mn-dimensional partitioned row vector comprising m subvectors, the i th of which is ai b0 ); and that a ˝ b0 D b0 ˝ a D ab0 :

(1.4)

16.1 The Kronecker Product: Definition and Some Basic Properties

339

In connection with result (1.4), recall that ab0 is an m n matrix whose ij th element is ai bj . Let A D faij g represent a p q matrix. Then, clearly, 0 ˝ A D A ˝ 0 D 0:

(1.5)

Further, for any diagonal matrix D D fdi g of order m, D ˝ A D diag.d1 A; d2 A; : : : ; dm A/;

(1.6)

which, in the special case where A is a diagonal matrix (of order p), is the diagonal matrix of order mp whose [p.i 1/ C r]th diagonal element is di arr (as is easily verified). In the special case where D D Im , result (1.6) simplifies to I ˝ A D diag.A; A; : : : ; A/;

(1.7)

and in the further special case where A D Ip , it simplifies to Im ˝ Ip D Imp : Note that, aside from certain special cases, the Kronecker product 0 1 a11 I a12 I : : : a1q I Ba21 I a22 I : : : a2q I C B C A˝I DB : :: :: C @ :: : : A ap1 I ap2 I : : : apq I

(1.8)

(1.9)

of A and Im differs from the Kronecker product I ˝ A of Im and A. Letting k represent an arbitrary scalar and A and B arbitrary matrices, it is easy to see that the effect on the Kronecker product A ˝ B of replacing either A or B by the scalar multiple kA or kB is as follows: .kA/ ˝ B D A ˝ .kB/ D k.A ˝ B/:

(1.10)

Another easily verified property of Kronecker products is that, for any m n matrices A and B and any p q matrix C, .A C B/ ˝ C D .A ˝ C/ C .B ˝ C/;

(1.11)

C ˝ .A C B/ D .C ˝ A/ C .C ˝ B/:

(1.12)

Results (1.11) and (1.12) can be used to show that, for any m n matrices A and B and any p q matrices C and D, .A C B/ ˝ .C C D/ D .A ˝ C/ C .A ˝ D/ C .B ˝ C/ C .B ˝ D/: (1.13) More generally, results (1.11) and (1.12) can be used to show that, for any m n matrices A1 ; A2 ; : : : ; Ar and p q matrices B1 ; B2 ; : : : ; Bs ,

340

16 Kronecker Products and the Vec and Vech Operators r X iD1

s r X s X X Ai ˝ Bj D .Ai ˝ Bj /: j D1

(1.14)

iD1 j D1

Note that, in the special case where r D s D 2, result (1.14) reduces to what is essentially result (1.13). Consider now the transpose .A ˝ B/0 of the Kronecker product of an m n matrix A D faij g and a p q matrix B. It follows from result (2.2.3) (together with the very definition of a Kronecker product) that .A ˝ B/0 is a partitioned matrix, comprising n rows and m columns of q p dimensional blocks, the ij th of which is aj i B0. Since aj i B0 is also the ij th block of A0 ˝ B0 , .A ˝ B/0 D A0 ˝ B0 :

(1.15)

Note that, in contrast to what might have been conjectured from the formula [.AB/0 D B0 A0 ] for the transpose of the “ordinary” product of two matrices, .A ˝ B/0 is not in general equal to B0 ˝ A0 . Note also that if A and B are both symmetric, then it follows from result (1.15) that .A ˝ B/0 D A ˝ BI that is, the Kronecker product of symmetric matrices is symmetric. The following lemma provides a basis for extending the definition of a Kronecker product to three or more matrices. Lemma 16.1.1. For any m n matrix A D faij g, p q matrix B D fbij g, and u v matrix C, A ˝ .B ˝ C/ D .A ˝ B/ ˝ C: (1.16) Proof. Let G D A ˝ .B ˝ C/ and H D .A ˝ B/ ˝ C. We wish to show that G D H. Partition G into m rows and n columns of pu qv dimensional blocks, and denote the ij th of these blocks by Gij . By definition, Gij D aij .B ˝ C/. For each i and j , partition Gij into p rows and q columns of u v dimensional blocks, thereby defining a further partitioning of G into mp rows and nq columns of (u v dimensional) blocks. Denote the rsth block of Gij by Gijrs and the xzth of the (u v dimensional) blocks of G by Gxz . Then, Gp.i1/Cr;q.j 1/Cs D Gijrs D aij brs C:

(1.17)

Now, observe that H D F ˝ C, where F D ffxz g D A ˝ B. Partition H into mp rows and nq columns of u v dimensional blocks, and denote the xzth of these blocks by Hxz . Then, fp.i1/Cr;q.j 1/Cs D aij brs , and consequently Hp.i1/Cr;q.j 1/Cs D fp.i1/Cr;q.j 1/Cs C D aij brs C:

(1.18)

Together, results (1.17) and (1.18) imply that (for arbitrary i , j , r, and s) Gp.i1/Cr;q.j 1/Cs D Hp.i1/Cr;q.j 1/Cs :

16.1 The Kronecker Product: Definition and Some Basic Properties

341

We conclude that G D H. Q.E.D. The symbol A ˝ B ˝ C is used to represent the common value of the left and right sides of equality (1.16), and this value is referred to as the Kronecker product of A, B, and C. This notation and terminology extend in an obvious way to any finite number of matrices. The following lemma establishes a connection between the “ordinary” product of Kronecker products and the Kronecker product of ordinary products. Lemma 16.1.2. For any m n matrix A D faij g, p q matrix B D fbij g, n u matrix C D fcij g, and q v matrix D D fdij g, .A ˝ B/.C ˝ D/ D .AC/ ˝ .BD/:

(1.19)

Proof. By definition, A ˝ B is a partitioned matrix, comprising m rows and n columns of p q dimensional blocks, the ij th of which is aij B; and C ˝ D is a partitioned matrix, comprising n rows and u columns of q v dimensional blocks, the jrth of which is cjr D. Thus, .A ˝ B/.C ˝ D/ is a partitioned matrix, comprising m rows and u columns of p v dimensional blocks, the i rth of which is the matrix n n X X .aij B/.cjr D/ D aij cjr BD: j D1

j D1

By way of comparison, .AC/ ˝ .BD/ is a partitioned matrix, comprising m rows and u columns of p v dimensional blocks, the i rth of which is the matrix fir BD; where P fir is the i rth element of AC. The proof is complete upon observing that fir D jnD1 aij cjr and hence that the i rth block of .AC/ ˝ .BD/ equals the i rth block of .A ˝ B/.C ˝ D/. Q.E.D. One implication of result (1.19) is that the Kronecker product A ˝ B of an m n matrix A and a p q matrix B can be decomposed in either of the following two ways: (1.20) A ˝ B D .A ˝ Ip /.In ˝ B/ D .Im ˝ B/.A ˝ Iq /: And, in light of result (1.4), a further implication is that, for any m n matrix A, p q matrix C, p 1 vector b, and n 1 vector d, .A ˝ b0 /.d ˝ C/ D .b0 ˝ A/.C ˝ d/ D Adb0 C:

(1.21)

Result (1.19) can be extended (by repeated application) to the ordinary product of an arbitrary number of Kronecker products. We have that .A1 ˝ B1 /.A2 ˝ B2 / .Ak ˝ Bk / D .A1 A2 Ak / ˝ .B1 B2 Bk /; (1.22) where (for i D 1; 2; : : : ; k) Ai is an mi miC1 dimensional matrix and Bi is a pi piC1 dimensional matrix.

342


The Kronecker product A ˝ B of any m m nonsingular matrix A and any p p nonsingular matrix B is invertible, and .A ˝ B/1 D A1 ˝ B1;

(1.23)

as is evident upon observing [in light of results (1.19) and (1.8)] that .A ˝ B/.A1 ˝ B1 / D .AA1 / ˝ .BB1 / D Im ˝ Ip D Imp : More generally, A ˝ B is a generalized inverse of the Kronecker product A ˝ B of any m n matrix A and any p q matrix B, as is evident upon observing that .A ˝ B/.A ˝ B /.A ˝ B/ D .AA A/ ˝ .BB B/ D A ˝ B:

(1.24)

Consider now the trace of the Kronecker product A ˝ B of an m m matrix A D faij g and a p p matrix B. Making use of formula (5.1.7), we find that tr.A ˝ B/ D tr.a11 B/ C tr.a22 B/ C C tr.amm B/ D .a11 C a22 C C amm / tr.B/: Thus,

tr.A ˝ B/ D tr.A/ tr.B/:

(1.25)

Next, consider the rank of the Kronecker product of an m n matrix A and a p q matrix B. Since A ˝ B is a generalized inverse of A ˝ B, it follows from results (10.2.1), (1.19), and (1.25) that rank.A˝B/ D trŒ.A˝B/.A ˝B / D trŒ.AA /˝.BB / D tr.AA / tr.BB /; and [again making use of result (10.2.1)] we conclude that rank.A ˝ B/ D rank.A/ rank.B/:

(1.26)

Note that result (1.26) implies that the mpnq dimensional Kronecker product A ˝ B of an m n matrix A and a p q matrix B has full row rank if and only if both A and B have full row rank, has full column rank if and only if both A and B have full column rank, and hence is nonsingular if and only if both A and B are nonsingular. The Kronecker product of an m n partitioned matrix 0 1 A11 A12 : : : A1c BA21 A22 : : : A2c C B C ADB : :: :: C @ :: : : A Ar1 Ar2 : : : Arc and a p q matrix B equals the mp nq matrix obtained by replacing each block Aij of A with the Kronecker product of Aij and B; that is,

16.2 The Vec Operator: Definition and Some Basic Properties

1 A11 A12 : : : A1c BA21 A22 : : : A2c C B C B :: :: :: C ˝ B @ : : : A 0 1 Ar1 Ar2 : : : Arc A11 ˝ B A12 ˝ B : : : A1c ˝ B BA21 ˝ B A22 ˝ B : : : A2c ˝ BC B C DB C; :: :: :: @ A : : : Ar1 ˝ B Ar2 ˝ B : : : Arc ˝ B

343

0

(1.27)

as is easily verified. And the Kronecker product of an m-dimensional column vector a D fai g and a p q matrix B that has been partitioned as B D .B1 ; B2 ; : : : ; Bk / is the mp q matrix obtained by replacing each block Bj of B with the Kronecker product of a and Bj ; that is, a ˝ .B1 ; B2 ; : : : ; Bk / D .a ˝ B1 ; a ˝ B2 ; : : : ; a ˝ Bk /; as is also easily verified. Suppose now that

(1.28)

0 1 A11 A12 : : : A1c BA21 A22 : : : A2c C B C ADB : :: :: C @ :: : : A Ar1 Ar2 : : : Arc

is a partitioned matrix whose rc blocks A11 ; A12 ; : : : ; Arc are all of the same size, say m n (in which case A is of dimensions rm cn). Let Uij represent an r c matrix whose ij th element is 1 and whose remaining rc 1 elements are 0. Then, AD

r X c X

Uij ˝ Aij ;

(1.29)

iD1 j D1

as is evident upon observing that Uij ˝ Aij is an rm cn partitioned matrtix comprising r rows and c columns of m n dimensional blocks, the pqth of which equals Apq , if i D p and j D q, and equals 0, if i ¤ p or j ¤ q.

16.2 The Vec Operator: Definition and Some Basic Properties It is sometimes convenient (as in the discussion of matrix differentiation in Chapter 15) to rearrange the elements of an m n matrix A D faij g in the form of an mndimensional column vector. The conventional way of doing so is to successively stack the first, second, : : : ; nth columns a1 ; a2 ; : : : ; an of A one under the other, giving the mn-dimensional column vector 0 1 a1 Ba2 C B C (2.1) B :: C; @:A an

344


that is, giving a partitioned column vector comprising n subvectors of dimension m, the i th of which is ai . The mn-dimensional column vector (2.1) is referred to as the vec of A—think of vec as being an abbreviation for vector. It is denoted by the symbol vec.A/ or sometimes (when the parentheses are not needed for clarity) by vec A. Vec.A/ can be regarded as the value assigned to A by a vector-valued function or operator whose domain is Rmn —this operator is known as the vec operator. By definition, the ij th element of A is the [.j 1/m C i ]th element of vec.A/. Suppose, for example, that m D 2 and n D 3. Then, 0 1 a11 Ba21 C B C Ba12 C C vec.A/ D B Ba22 C : B C @a13 A a23 For any column vector a, vec.a0 / D vec.a/ D a;

(2.2)

as is evident from the very definition of the vec operator. And, for any m-dimensional column vector a D fai g and n-dimensional column vector b, vec.ba0 / D a ˝ b;

(2.3)

as is evident from result (1.2) upon observing that the i th (of the m columns) of ba0 is ai b. Clearly, for any scalar c and any matrix A, vec.cA/ D c vec.A/;

(2.4)

and, for any two matrices A and B (of the same size), vec.A C B/ D vec.A/ C vec.B/:

(2.5)

More generally, for any k scalars c1 ; c2 ; : : : ; ck and for any k matrices A1 ; A2 , : : : ; Ak (of the same size), vec

k X iD1

k X ci A i D ci vec.Ai/;

(2.6)

iD1

as can be easily established by the repeated application of results (2.5) and (2.4). Let A represent an m n matrix and B an n p matrix, and denote the first, second, : : : ; pth columns of B by b1 ; b2 ; : : : ; bp , respectively. Then, 0 1 Ab1 BAb2 C B C (2.7) vec.AB/ D B : C ; @ :: A Abp

16.2 The Vec Operator: Definition and Some Basic Properties

345

as is evident upon observing that the j th column of AB is Abj . Note that result (2.7) implies that vec.AB/ D diag.A; A; : : : ; A/ vec.B/ (2.8) or equivalently [in light of result (1.7)] that vec.AB/ D .Ip ˝ A/ vec.B/:

(2.9)

Result (2.9) is generalized in the following theorem. Theorem 16.2.1. For any m n matrix A, n p matrix B, and p q matrix C,

vec.ABC/ D .C 0 ˝ A/ vec.B/:

(2.10)

Proof. According to result (2.4.2), B is expressible as BD

p X

bj uj0 ;

j D1

where (for j D 1; : : : ; p) bj is the j th column of B and uj0 is the j th row of Ip . Thus, making use of results (2.3) and (1.19), we find that i h X bj uj0 C vec.ABC/ D vec A D

X

j

vec.Abj uj0 C/

j

D

X

Œ.C 0 uj / ˝ .Abj /

j

D

X

.C 0 ˝ A/.uj ˝ bj /

j

D

X

.C 0 ˝ A/ vec.bj uj0 /

j

D .C 0 ˝ A/ vec

X

bj uj0 D .C 0 ˝ A/ vec B:

j

Q.E.D. Three alternative expressions for the vec of the product AB of an m n matrix A and an n p matrix B can be obtained as special cases of result (2.10). One of these expressions is given by result (2.9). The other two are vec.AB/ D .B0 ˝ Im / vec.A/; vec.AB/ D .B0 ˝ A/ vec.In /:

(2.11) (2.12)

Consider now the product ABx of an m n matrix A, an n p matrix B, and a p 1 vector x. According to result (2.2),

346


ABx D vec.ABx/ D vec.x0 B0A0 /: Thus, it follows from result (2.10) that ABx D .x0 ˝ A/ vec.B/ D .A ˝ x0 / vec.B0 /:

(2.13)

For any m n matrices A and B, tr.A0 B/ D .vec A/0 vec B:

(2.14)

To see this, let aj and bj represent the j th columns of A and B, respectively, and observe that the j th diagonal element of A0 B equals aj0 bj , so that 1 0 10 0 1 a1 b1 b1 n Bb2 C Ba2 C Bb2 C X B C B CB C aj0 bj D .a01 ; a02 ; : : : ; a0n / B : C D B : C B : C tr.A0 B/ D @ :: A @ :: A @ :: A 0

j D1

bn

an

bn 0

D .vec A/ vec B: Note that result (2.14) implies that the (usual) inner product of any two m n matrices equals the (usual) inner product of their vecs. Note also that, since [according to result (5.2.8)] tr.A0 B/ D tr.B0A/ D tr.BA0 / D tr.AB0 /, alternative versions of formula (2.14) can be obtained by replacing tr.A0 B/ with tr.B0A/, tr.BA0 /, or tr.AB0 /. Result (2.14) is generalized in the following theorem. Theorem 16.2.2. For any m n matrix A, m p matrix B, p q matrix C, and n q matrix D, tr.A0 BCD 0 / D .vec A/0 .D ˝ B/ vec C:

(2.15)

Proof. Making use of results (2.14) and (2.10), we find that tr.A0 BCD 0 / D .vec A/0 vec.BCD 0 / D .vec A/0 .D ˝ B/ vec C: Q.E.D. Note [in light of results (5.2.3) and (5.1.5)] that tr.A BCD / D tr.D A BC/ D tr.CD 0A0 B/ D tr.BCD 0A0 / and that tr.A0 BCD 0 / D trŒ.A0 BCD 0 /0 D tr.DC 0 B0A/ D tr.ADC 0 B0 / D tr.B0ADC 0 / D tr.C 0 B0AD/. Thus, alternative versions of formula (2.15) can be obtained by replacing tr.A0 BCD 0 / with tr.D 0A0 BC/, tr.CD 0A0 B/, tr.BCD 0A0 /, tr.DC 0 B0A/, tr.ADC 0 B0 /, tr.B0ADC 0 /, or tr.C 0 B0AD/. Note also [in light of result (1.7)] that, in the special case where q D n and D D In , result (2.15) reduces in effect to 0

0

tr.A0 BC/ D .vec A/0 diag.B; B; : : : ; B/ vec C:

0 0

(2.16)

Let A D .A1 ; A2 ; : : : ; Ak / represent an m n partitioned matrix comprising a single row of k blocks, and let nj represent the number of columns in Aj . Then,

16.3 Vec-Permutation Matrix

347

0

1 vec A1 Bvec A2 C B C vec.A1 ; A2 ; : : : ; Ak / D B : C I : @ : A

(2.17)

vec Ak that is, vec.A1 ; A2 ; : : : ; Ak / equals an mn-dimensional column vector, comprising k subvectors, the j th of which is the mnj -dimensional column vector vec.Aj /. To see this, observe that both vec.A1 ; A2 ; : : : ; Ak / and 0 1 vec A1 Bvec A2 C B C B :: C @ : A vec Ak are equal to an mn-dimensional partitioned column vector, comprising n mP 1 dimensional subvectors, the ( jsD1 ns C r)th of which is the rth column of Aj P (where 0sD1 ns D 0).

16.3 Vec-Permutation Matrix a. Definition and some alternative descriptions Let A D faij g represent an m n matrix, and denote the first, : : : ; nth columns 0 , of A by a1 ; : : : ; an , respectively, and the first, : : : ; mth rows of A by r10 ; : : : ; rm 0 respectively. Then, r1 ; : : : ; rm are the m columns of A , and, by definition, 0 1 0 1 a1 r1 Ba2 C B r2 C B C B C vec.A/ D B : C and vec.A0 / D B : C : @ :: A @ :: A an rm Both vec.A0 / and vec.A/ are obtained by rearranging the elements of A in the form of an mn-dimensional column vector; however, they are arranged row by row in vec.A0 / instead of column by column [as in vec.A/]. For example, when m D 2 and n D 3, 0 1 0 1 a11 a11 Ba12 C Ba21 C B C B C Ba13 C Ba12 C 0 C C B vec.A/ D B C and vec.A / D B Ba21 C : a B C B 22 C @a22 A @a13 A a23 a23 Clearly, vec.A0 / can be obtained by permuting the elements of vec.A/. Accordingly, there exists an mn mn permutation matrix, to be denoted by the symbol Kmn , such that

348


vec.A0 / D Kmn vec.A/:

(3.1)

(This matrix depends on m and n, but not on the values of a11 ; a12 ; : : : ; amn .) For example, 1 0 1 0 0 0 0 0 B0 0 1 0 0 0 C C B B0 0 0 0 1 0 C C: B K23 D B C B0 1 0 0 0 0 C @0 0 0 1 0 0 A 0 0 0 0 0 1 The matrix Kmn is referred to as a vec-permutation matrix (e.g., Henderson and Searle 1979) or, more commonly and for reasons that will become evident in Subsection c, as a commutation matrix (e.g., Magnus and Neudecker 1979). As discussed in Section 16.2, the ij th element of the m n matrix A is the [.j 1/m C i ]th element of vec.A/. And the ij th element of A is the j i th element of the n m matrix A0 and accordingly is the [.i 1/n C j ]th element of vec.A0 /. Thus, the [.i 1/n C j ]th row of the vec-permutation matrix Kmn is the [.j 1/m C i ]th row of Imn (i D 1; : : : ; mI j D 1; : : : ; n). Note that, since the transpose A0 of the m n matrix A is of dimensions n m, it follows from the very definition of a vec-permutation matrix that vec.A/ D vecŒ.A0 /0 D Knm vec.A0 /:

(3.2)

The following theorem gives a useful and informative representation for the vec-permutation matrix Kmn . Theorem 16.3.1. Let m and n represent positive integers, and (for i D 1; : : : ; m and j D 1; : : : ; n) let Uij D ei uj0 , where ei is the i th column of Im and uj the j th column of In (in which case Uij is an m n matrix whose ij th element is 1 and whose remaining elements are 0). Then, Kmn D

m X n X

.Uij ˝ Uij0 /:

(3.3)

iD1 j D1

Proof. Using results (3.1), (2.4.3), (2.4.5), and (2.10), we find that, for any m n matrix A, X Kmn vec.A/ D vec.A0 / D vec aij Uij0 D

X

i;j

vec.aij uj e0i /

i;j

D

X

vec.uj aij e0i /

i;j

D

X i;j

vecŒuj .e0i Auj /e0i


D

X

349

vec.Uij0 AUij0 /

i;j

D

X

.Uij ˝ Uij0 / vec.A/

i;j

and hence that, for any mn-dimensional column vector a, X Kmn a D Uij ˝ Uij0 a: i;j

Based on Lemma 2.3.2, we conclude that X Kmn D .Uij ˝ Uij0 /: i;j

Q.E.D. In light of result (1.29), result (3.3) can be reexpressed as 0 0 1 0 0 U11 U12 : : : U1n 0 0 C BU0 B 21 U22 : : : U2n C : Kmn D B : : : :: :: C @ :: A

(3.4)

0 0 0 Um2 : : : Umn Um1

That is, Kmn can be regarded as an mn mn partitioned matrix, comprising m rows and n columns of n m dimensional blocks, the ij th of which is the matrix Uij0 D uj e0i (whose j i th element is 1 and whose other mn 1 elements are 0). For example, 0

K23 D

0 0 U11 U12 0 0 U21 U22

1 B0 B 0 B0 U13 DB 0 B0 U23 B @0 0

0 0 0 1 0 0

0 1 0 0 0 0

0 0 0 0 1 0

0 0 1 0 0 0

1 0 0C C 0C C: 0C C 0A 1

Let a represent an m-dimensional column vector. Then, upon setting A D a in equality (3.1) and result (3.2), we find [in light of result (2.2)] that a D Km1 a;

a D K1m a:

It follows (in light of Lemma 2.3.2) that Km1 D K1m D Im :

(3.5)

350


b. Some basic properties It follows from result (3.2) that, for any m n matrix A, vec.A/ D Knm vec.A0 / D Knm Kmn vec.A/ and hence that, for any mn-dimensional column vector a, a D Knm Kmn a; implying (in light of Lemma 2.3.2) that Imn D Knm Kmn : Thus (in light of Lemma 8.1.3), Kmn is nonsingular and K1 mn D Knm . Moreover, since Kmn is a permutation matrix, it is orthogonal, and consequently K0mn D K1 mn . In summary, we have that Kmn is nonsingular and that K0mn D K1 mn D Knm :

(3.6)

Note that result (3.6) implies that K0mm D Kmm I

(3.7)

that is, Kmm is symmetric (as well as orthogonal). As a special case of a formula for tr.Kmn / given by, for example, Magnus and Neudecker (1979, p. 383), we have that tr.Kmm / D m:

(3.8)

To verify formula (3.8), let ei represent the i th column of Im , let Uij D ei ej0 , and observe that ej0 ei equals 1, if j D i , and equals 0, if j ¤ i . Then, making use of results (3.3), (1.25), and (5.2.6), we find that X Uij ˝ Uij0 tr.Kmm / D tr D

X

i;j

tr.Uij ˝ Uij0 /

i;j

D

X

tr.Uij / tr.Uij0 /

i;j

D

X

Œtr.Uij /2

i;j

D

X

Œtr.ei ej0 /2

i;j

D

X i;j

.ej0 ei /2 D

X i

.e0i ei /2 D

m X iD1

.1/2 D m:


351

c. Kronecker product of two matrices: the effect of premultiplying or postmultiplying by a vec-permutation matrix As discussed in Section 16.1, the Kronecker product B ˝ A of two matrices B and A is not in general equal to the Kronecker product A ˝ B of A and B. However, B ˝ A can be “made equal to” A ˝ B by permuting the columns of B ˝ A and the rows of A ˝ B, as indicated by the following theorem. Theorem 16.3.2. For any m n matrix A and p q matrix B, .B ˝ A/Kq n D Kpm .A ˝ B/; B ˝ A D Kpm .A ˝ B/Knq :

(3.9) (3.10)

Proof. Making use of Theorem 16.2.1, we find that, for any q n matrix X, .B ˝ A/Kq n vec X D .B ˝ A/ vec.X0 / D vec.AX0 B0 / D vecŒ.BXA0 /0 D Kpm vec.BXA0 / D Kpm .A ˝ B/ vec X: Thus, .B ˝ A/Kq n x D Kpm .A ˝ B/x for every nq 1 vector x, implying (in light of Lemma 2.3.2) that .B ˝ A/Kq n D Kpm .A ˝ B/: Further, since K1 q n D Knq , we have that B ˝ A D .B ˝ A/Kq n Knq D Kpm .A ˝ B/Knq : Q.E.D. As indicated by result (3.10), the effect on A ˝ B of premultiplication and postmultiplication by the vec-permutation matrices Kpm and Knq , respectively, is to interchange or “commute” A and B. It is because of this effect that vecpermutation matrices are often referred to as commutation matrices. In the various special cases where A or B is a row or column vector, result (3.9) or (3.10) of Theorem 16.3.2 can be stated more simply, as indicated by the following corollary. Corollary 16.3.3. For any m n matrix A and p 1 vector b, b ˝ A D Kpm .A ˝ b/; A ˝ b D Kmp .b ˝ A/; b0 ˝ A D .A ˝ b0 /Knp ;

(3.11) (3.12) (3.13)

A ˝ b0 D .b0 ˝ A/Kpn :

(3.14)

352


Note that in the special case where B is a row vector, say b0, and A is a column vector, say a, equality (3.13) or (3.14) reduces to the equality b0 ˝ a D a ˝ b0 ; given previously in result (1.4). The trace of the matrix obtained by premultiplying or postmultiplying a Kronecker product (of an n m matrix and an m n matrix) by a vec-permutation matrix is given by the following theorem. Theorem 16.3.4. For any m n matrices A and B, trŒ.A0 ˝ B/Kmn D trŒKmn .A0 ˝ B/ D tr.A0 B/ D .vec A0 /0 Kmn vec B: (3.15) Proof. Let ei represent the i th column of Im and uj the j th column of In , and define Uij D ei uj0 . That trŒ.A0 ˝ B/Kmn D trŒKmn .A0 ˝ B/ is an immediate consequence of Lemma 5.2.1. Further, making use of results (3.3), (1.19), (1.25), (5.2.6), (2.4.5), and (5.2.4), we find that hX i .Uij ˝ Uij0 /.A0 ˝ B/ trŒKmn .A0 ˝ B/ D tr i;j

D

X

trŒ.Uij ˝ Uij0 /.A0 ˝ B/

i;j

D

X

trŒ.Uij A0 / ˝ .Uij0 B/

i;j

D

X

tr.Uij A0 / tr.Uij0 B/

i;j

D

X

tr.ei uj0 A0 / tr.uj e0i B/

i;j

D

X

.uj0 A0 ei /e0i Buj

i;j

D

X

bij aij

i;j

D tr.A0 B/: And, in light of results (2.14), (3.2), and (3.6), we have that tr.A0 B/ D .vec A/0 vec B D .Knm vec A0 /0 vec B D .vec A0 /0 K0nm vec B

D .vec A0 /0 Kmn vec B: Q.E.D.


353

d. Relationship between the vec of the Kronecker product of two matrices and the Kronecker product of their vecs The following theorem relates the vec of the Kronecker product of two matrices A and B to the Kronecker product of the vecs of A and B. Theorem 16.3.5. For any m n matrix A and any p q matrix B, vec.A ˝ B/ D .In ˝ Kq m ˝ Ip /Œvec.A/ ˝ vec.B/:

(3.16)

Proof. Let ai represent the i th column of A, and ei the i th column of In (i D 1; : : : ; n). And let bj represent the j th column of PB, and uj the j thPcolumn of Iq (j D 1; : : : ; q). Then, writing A and B as A D i ai e0i and B D j bj uj0 and making use of results (1.14), (2.6), (1.15), (1.19), (2.3), (1.16), and (3.11), we find that i hX .ai e0i / ˝ .bj uj0 / vec.A ˝ B/ D vec D

X

i;j

vecŒ.ai e0i / ˝ .bj uj0 /

i;j

D

X

vecŒ.ai ˝ bj /.ei ˝ uj /0

i;j

D

X

.ei ˝ uj / ˝ .ai ˝ bj /

i;j

D

X

ei ˝ .uj ˝ ai / ˝ bj

i;j

D

X

.In ˝ Kq m ˝ Ip /.ei ˝ ai ˝ uj ˝ bj /

i;j

D

X

.In ˝ Kq m ˝ Ip /Œvec.ai e0i / ˝ vec.bj uj0 /

i;j

D .In ˝ Kq m ˝ Ip /

hX h

i

D .In ˝ Kq m ˝ Ip / vec

vec.ai e0i / ˝

X

ai e0i

X

i vec.bj uj0 /

j

˝ vec.

i

X

i bj uj0

j

D .In ˝ Kq m ˝ Ip /Œvec.A/ ˝ vec.B/: Q.E.D. e. Determinant of a Kronecker product Preliminary to deriving a formula for the determinant of a Kronecker product of two matrices, observe [in light of results (1.20) and (3.10)] that, for any m n matrix A and any p q matrix B,

354


A ˝ B D .A ˝ Ip /.In ˝ B/ D Kmp .Ip ˝ A/Kpn .In ˝ B/ D Kmp diag.A; A; : : : ; A/Kpn diag.B; B; : : : ; B/:

(3.17)

Consider now the determinant of the Kronecker product A ˝ B of an m m matrix A and a p p matrix B. Applying result (3.17) (in the special case where n D m and q D p), we find [in light of results (13.3.9) and (13.3.5)] that jA ˝ Bj D jKmp jjAjp jKpm jjBjm : And, in light of result (3.6), we have that jKmp jjKpm j D jKmp Kpm j D jIj D 1: We conclude that (for any m m matrix A and any p p matrix B) jA ˝ Bj D jAjp jBjm :

(3.18)

16.4 The Vech Operator a. Definition and basic characteristics The values of all n2 elements of an n n symmetric matrix A can be determined from the values of those n.n C 1/=2 elements that are on or below the diagonal [or alternatively from those n.n C 1/=2 elements that are on or above the diagonal]. In rearranging the elements of A in the form of a vector, we may wish to exclude the n.n 1/=2 “duplicate” elements. This would be the case if, for example, A were a (symmetric) matrix of variables and our objective were the differentiation of a function of A. Let A D faij g represent an n n (symmetric or nonsymmetric) matrix. To obtain the vec of A, we successively stack the first, second, : : : ; nth columns a1 ; a2 ; : : : ; an of A one under the other. Consider a modification of this process in which (before or after the stacking) the n.n 1/=2 “supradiagonal” elements of A are eliminated from a1 ; a2 ; : : : ; an . The result is the [n.n C 1/=2]-dimensional vector 0 1 a1 Ba C B 2C (4.1) B : C; @ :: A an

where (for i D 1; 2; : : : ; n) ai D .ai i ; aiC1;i ; : : : ; ani /0 is the subvector of ai obtained by striking out its first i 1 elements. Thus, by definition, the vector (4.1) is a subvector of vec A obtained by striking out a particular set of what are, in the special case where A is symmetric, duplicate or redundant elements. Following Henderson and Searle (1979), let us refer to the vector (4.1) as the vech of A— think of vech as being an abbreviation for “vector-half.” And denote

16.4 The Vech Operator

355

this vector by the symbol vech.A/ or possibly (when the parentheses are not needed for clarity) by vech A. Like vec A, vech A can be regarded as the value assigned to A by a vector-valued function or operator—the domain of this function is Rnn. For n D 1, n D 2, and n D 3, 0 1 a11 Ba21 C 0 1 C B a11 Ba31 C B vech A D .a11 /; vech A D @a21 A; and vech A D B C C; respectively: Ba22 C a22 @a32 A a33 By way of comparison,

0

0 1 a11 Ba21 C B vec A D .a11 /; vec A D @ C ; a12 A a22

is

1 a11 Ba21 C B C Ba31 C B C Ba12 C B C C and vec A D B Ba22 C; respectively: Ba32 C B C Ba13 C B C @a23 A a33

Observe that the total number of elements in the j vectors a1 ; a2 ; a3 ; : : : ; aj

n C .n 1/ C .n 2/ C C n .j 1/ D nj .0 C 1 C 2 C C j 1/ D nj j.j 1/=2 and that, of the n .j 1/ D n j C 1 elements of aj , there are (for i j ) n i elements that come after aij . Since nj j.j 1/=2.ni / D .j 1/.nj=2/Ci , it follows that (for i j ) the ij th element of A is the [.j 1/.n j=2/ C i ]th element of vech A. By way of comparison, the ij th element of A is (as discussed in Section 16.2) the [.j 1/n C i ]th element of vec A, so that (for i j ) the [.j 1/n C i ]th element of vec A is the [.j 1/.n j=2/ C i ]th element of vech A. For example, when n D 8, we find [upon setting i D 7 and j D 5 and observing that .j 1/n C i D 39 and that .j 1/.n j=2/ C i D 29] that a75 is the 39th element of vec A and the 29th element of vech A. Let c1 ; : : : ; ck represent any k scalars, and let A1 ; : : : ; Ak represent any k square matrices (of the same order). Then, vech

k X iD1

k X ci A i D ci vech.Ai /;

(4.2)

iD1

as is evident from result (2.6) upon observing that the vech of any (square) matrix is a subvector of its vec.

356


b. Duplication matrix Every element of an n n symmetric matrix A, and hence every element of vec A, is either an element of vech A or a “duplicate” of an element of vech A. Thus, there exists a unique {[n2 n.n C 1/=2]-dimensional} matrix, to be denoted by the symbol Gn , such that (4.3) vec A D Gn vech A for every n n symmetric matrix A. This matrix is called the duplication matrix. Clearly, 1 0 1 0 0 0 0 0 B0 1 0 0 0 0 C C B B0 0 1 0 0 0 C 0 1 C B 1 0 0 B0 1 0 0 0 0 C C B B0 1 0C C B C G1 D .1/; G2 D B @0 1 0A; and G3 D B0 0 0 1 0 0C: B0 0 0 0 1 0 C C B 0 0 1 B0 0 1 0 0 0 C C B @0 0 0 0 1 0 A 0 0 0 0 0 1 More generally (for an arbitrary positive integer n), the matrix Gn can be described in terms of its rows or columns. For i j , the [.j 1/n C i ]th and [.i 1/n C j ]th rows of Gn equal the [.j 1/.n j=2/ C i ]th row of In.nC1/=2 , that is, they equal the [n.n C 1/=2]-dimensional row vector whose [.j 1/.n j=2/ C i ]th element is 1 and whose remaining elements are 0. And (for i j ) the [.j 1/.n j=2/ C i ]th column of Gn is an n2 -dimensional column vector whose [.j 1/n C i ]th and [.i 1/n C j ]th elements are 1 and whose remaining n2 1 (if i D j ) or n2 2 (if i > j ) elements are 0. The matrix Gn is of full column rank; that is, rank.Gn / D n.n C 1/=2:

(4.4)

To see this, observe that the columns of Gn are orthogonal (since every row of Gn contains only one nonzero element) and are nonnull. Or, alternatively, observe that every row of In.nC1/=2 is a row of Gn and hence that Gn contains n.n C 1/=2 linearly independent rows. Now, for purposes of deriving a recursive formula for Gn , let B represent an arbitrary .n C 1/ .n C 1/ matrix, and partition B as c b0 BD a A (where A is n n). Then, there exists an .n C 1/2 .n C 1/2 permutation matrix, to be denoted by the symbol QnC1 , such that (for every choice of B) 0 1 c B b C C Q0nC1 vec B D B @ a A: vec A


357

Further, letting u represent the first column of InC1 and letting U represent the .n C 1/ n submatrix of InC1 obtained by striking out u, QnC1 is expressible as .2/ QnC1 D .Q.1/ nC1 ; QnC1 /, where Q.1/ nC1 D diag.u; u; : : : ; u/ D InC1 ˝ u; Q.2/ nC1 D diag.U ; U ; : : : ; U / D InC1 ˝ U : Clearly,

0

1 c vech B D @ a A: vech A

Thus, if B is symmetric (in which case A is symmetric and b D a), we have that 1 0 1 0 c 1 0 0 B b C B0 In 0 C C B C B @ a A D @0 In 0 A vech B 0 0 Gn vec A or, equivalently, that 0

1 B0 0 B QnC1 vec B D @ 0 0

1 0 0 In 0 C C vech B: In 0 A 0 Gn

We conclude that if B is symmetric, then 0 1 0 B0 In vec B D QnC1 B @0 I n 0 0

1 0 0 C C vech B: 0 A Gn

This conclusion gives rise to the recursive formula 0 1 1 0 0 B0 In 0 C C GnC1 D QnC1 B @0 In 0 A: 0 0 Gn

(4.5)

Since the duplication matrix Gn is of full column rank, it has a left inverse. In fact, except in the special case where n D 1, Gn has an infinite number of left inverses. (G1 is nonsingular and consequently has a unique left inverse.) In what follows, the symbol Hn is used to represent an arbitrary left inverse of Gn . Thus, by definition, Hn is an n.n C 1/=2 n2 matrix such that Hn Gn D I:

358


One choice for Hn is

Hn D .Gn0 Gn /1 Gn0 :

(Since Gn is of full column rank, Gn0 Gn is nonsingular.) The premultiplication of the vech of an n n symmetric matrix A by the duplication matrix Gn transforms vech A to vec A. Upon premultiplying both sides of equality (4.3) by Hn , we find that vech A D Hn vec A

(4.6)

for every n n symmetric matrix A. Thus, the premultiplication of the vec of an n n symmetric matrix A by the left inverse Hn transforms vec A to vech A. Are there matrices other than left inverses (of Gn ) that have this property? That is, does there exist an n.n C 1/=2 n2 matrix L such that vech A D L vec A

(4.7)

for every n n symmetric matrix A that does not satisfy the condition LGn D I? The answer is no, as is evident upon observing that, together with equality (4.3), equality (4.7) implies that vech A D LGn vech A for every n n symmetric matrix A, or, equivalently, that a D LGn a for every [n.n C 1/=2]-dimensional column vector a, and hence that LGn D I: Thus, as an alternative to characterizing the matrix Hn as an arbitrary left inverse of Gn , it could be characterized as any n.n C 1/=2 n2 matrix L such that equality (4.7) holds for every n n symmetric matrix A. In fact, as noted by Henderson and Searle (1979), it is the latter characterization that provides the motivation for denoting this matrix by a capital h—h added to vec gives vech. We find that 0 1 1 0 0 0 H2 D @0 c 1 c 0A; (4.8a) H1 D .1/; 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 B0 c1 0 1 c1 0 0 0 0 0C C B B0 0 c2 0 0 0 1 c2 0 0C C; (4.8b) H3 D B B0 0 0 0 1 0 0 0 0C B C @0 0 0 0 1 c3 0 A 0 0 c3 0 0 0 0 0 0 0 0 1 where c, c1 , c2 , and c3 are arbitrary scalars. More generally (for an arbitrary positive integer n), the matrix Hn can be described in terms of its rows. The [.i 1/.n i=2/ C i ]th row of Hn is the [.i 1/n C i ]th row of In2 , and (for


359

i > j ) the [.j 1/.n j=2/ C i ]th row of Hn is any n2 -dimensional row vector whose [.j 1/nCi ]th and [.i 1/nCj ]th elements sum to 1 and whose remaining n2 2 elements are 0. Note that Hn is of full row rank, that is, rank.Hn / D n.n C 1/=2

(4.9)

(as is evident from Lemma 8.1.1 upon observing that Gn is a right inverse of Hn ). Note also that (4.10) vec A D Gn Hn vec A for every n n symmetric matrix A [as is evident upon substituting expression (4.6) in the right side of equality (4.3)]. Consider now the matrix .Gn0 Gn /1 Gn0 (which is one choice for Hn ). The n.n C 1/=2 n.n C 1/=2 matrix Gn0 Gn is diagonal (since the columns of Gn are orthogonal). Further, the [.i 1/.ni=2/Ci ]th diagonal element of Gn0 Gn equals 1, and (for i > j ) the [.j 1/.nj=2/Ci ]th diagonal element of Gn0 Gn equals 2. Consequently, the [.i 1/.ni=2/Ci ]th row of .Gn0 Gn /1 Gn0 is the [.i 1/nCi ]th row of In2 , and (for i > j ) the [.j 1/.n j=2/ C i ]th row of .Gn0 Gn /1 Gn0 is an n2 -dimensional row vector whose [.j 1/n C i ]th and [.i 1/n C j ]th elements equal 1=2 and whose remaining n2 2 elements equal 0. For example, G10 G1 D .1/, G20 G2 D diag.1; 2; 1/, and G30 G3 D diag.1; 2; 2; 1; 2; 1/; and .G10 G1 /1 G10 , .G20 G2 /1 G20 , and .G30 G3 /1 G30 are the special cases of H1 , H2 , and H3 , respectively, obtained by setting c D c1 D c2 D c3 D 1=2 in result (4.8). Recursive formulas for Gn0 Gn and for Hn can be obtained from formula (4.5). Recalling that QnC1 is orthogonal, we find that 1 0 1 0 0 0 0 A (4.11) GnC1 D @0 2In GnC1 0 0 Gn0 Gn 0 0 and that, for HnC1 D .GnC1 GnC1 /1 GnC1 and Hn D .Gn0 Gn /1 Gn0 , 1 0 1 0 0 0 HnC1 D @0 .1=2/In .1=2/In 0 A Q0nC1 : 0 0 0 Hn

(4.12)

Note that, for Hn D .Gn0 Gn /1 Gn0 , Gn Hn D Gn .Gn0 Gn /1 Gn0 D PGn : c. Some relationships involving the duplication and vec-permutation matrices For every n n symmetric matrix A, Gn vech A D vec A D vec.A0 / D Knn vec A D Knn Gn vech A:

(4.13)

360


Thus, Gn a D Knn Gn a for every [n.n C 1/=2]-dimensional column vector a, leading to the conclusion that Knn Gn D Gn :

(4.14)

Further, since K0nn D Knn , .Gn0 Gn /1 Gn0 Knn D .Gn0 Gn /1 .Knn Gn /0 D .Gn0 Gn /1 Gn0 ; so that, for Hn D .Gn0 Gn /1 Gn0 , Hn Knn D Hn :

(4.15)

Note that result (4.14) implies that .1=2/.In2 C Knn /Gn D Gn :

(4.16)

Similarly, result (4.15) implies that, for Hn D .Gn0 Gn /1 Gn0 , Hn Œ.1=2/.In2 C Knn / D Hn :

(4.17)

Let us now consider (for future reference) some of the properties of the matrix .1=2/.In2 C Knn /, which enters in results (4.16) and (4.17). Since K2nn D I, we have that .1=2/.In2 C Knn /Knn D .1=2/.In2 C Knn / D Knn Œ.1=2/.In2 C Knn /: (4.18) Further, recalling that Knn is symmetric [and using result (4.18)], we find that Œ.1=2/.In2 C Knn /0 D .1=2/.In2 C Knn / D Œ.1=2/.In2 C Knn /2 ;

(4.19)

that is, .1=2/.In2 C Knn / is symmetric and idempotent. And, in light of result (3.8), we have that trŒ.1=2/.In2 C Knn / D n.n C 1/=2

(4.20)

or, equivalently [since .1=2/.In2 C Knn / is idempotent], that rankŒ.1=2/.In2 C Knn / D n.n C 1/=2:

(4.21)

Now, observe [in light of results (4.19), (4.16), and (4.17)] that, for Hn D .Gn0 Gn /1 Gn0 , Œ.1=2/.In2 C Knn / Gn Hn 2 D Œ.1=2/.In2 C Knn /2 .1=2/.In2 C Knn /Gn Hn D .1=2/.In2

Gn Hn Œ.1=2/.In2 C Knn / C Gn Hn Gn Hn C Knn / Gn Hn Gn Hn C Gn Hn

D .1=2/.In2 C Knn / Gn Hn :


361

Moreover, as a consequence of results (5.2.3) and (4.20), we have that trŒ.1=2/.In2 C Knn / Gn Hn D trŒ.1=2/.In2 C Knn / tr.Gn Hn / D trŒ.1=2/.In2 C Knn / tr.Hn Gn / D trŒ.1=2/.In2 C Knn / tr.In.nC1/=2 / D n.n C 1/=2 n.n C 1/=2 D 0: Thus, for Hn D .Gn0 Gn /1 Gn0 , the matrix .1=2/.In2 CKnn /Gn Hn is idempotent and its trace equals 0. We conclude (on the basis of Corollary 10.2.3) that, for Hn D .Gn0 Gn /1 Gn0 , .1=2/.In2 C Knn / Gn Hn D 0 or, equivalently, that [for Hn D .Gn0 Gn /1 Gn0 ] Gn Hn D .1=2/.In2 C Knn /:

(4.22)

d. Vech of a product of matrices Let A represent an n n matrix and X an n n symmetric matrix. Observe (in light of Theorem 16.2.1) that vec.AXA0 / D .A ˝ A/ vec X D .A ˝ A/Gn vech X:

(4.23)

Observe also that (since AXA0 is symmetric) vech.AXA0 / D Hn vec.AXA0 /:

(4.24)

Together, results (4.23) and (4.24) imply that vech.AXA0 / D Hn .A ˝ A/Gn vech X:

(4.25)

Result (4.25) can be regarded as the vech counterpart of formula (2.10) for the vec of a product of matrices. According to this result, the premultiplication of the vech of X by the n.n C 1/=2 n.n C 1/=2 matrix Hn .A ˝ A/Gn transforms vech X into vech.AXA0 /. Let us now consider various properties of the matrix Hn .A ˝ A/Gn . Observe [in light of results (4.23) and (4.10)] that (for any n n symmetric matrix X) .A ˝ A/Gn vech X D vec.AXA0 / D Gn Hn vec.AXA0 / D Gn Hn .A ˝ A/Gn vech X: Thus, .A ˝ A/Gn x D Gn Hn .A ˝ A/Gn x for every [n.n C 1/=2]-dimensional column vector x, leading to the conclusion that Gn Hn .A ˝ A/Gn D .A ˝ A/Gn :

(4.26)

362


And, for Hn D .Gn0 Gn /1 Gn0 , Gn Hn .A ˝ A/ D .A ˝ A/Gn Hn ;

(4.27)

as is evident from result (4.22) upon observing [in light of result (3.9)] that .1=2/.In2 C Knn /.A ˝ A/ D .1=2/Œ.A ˝ A/ C Knn .A ˝ A/ D .1=2/Œ.A ˝ A/ C .A ˝ A/Knn D .A ˝ A/Œ.1=2/.In2 C Knn /: Further, premultiplying both sides of equality (4.27) by Hn (and recalling that Hn Gn D I), we find that [for Hn D .Gn0 Gn /1 Gn0 ] Hn .A ˝ A/Gn Hn D Hn .A ˝ A/;

(4.28)

analogous to result (4.26). Several properties of the matrix Hn .A ˝ A/Gn are given in the following theorem. (1) (2) (3) (4) (5)

Theorem 16.4.1. For any n n matrix A, Hn .A ˝ A/Gn is invariant to the choice of Hn ; Hn .A ˝ A /Gn is a generalized inverse of Hn .A ˝ A/Gn ; trŒHn .A ˝ A/Gn D .1=2/Œtr.A/2 C .1=2/ tr.A2 /; rankŒHn .A ˝ A/Gn D .1=2/Œrank.A/2 C .1=2/ rank.A/; Hn .A ˝ A/Gn is nonsingular if and only if A is nonsingular.

.2/ Proof. (1) Let H.1/ n and Hn represent any two choices for Hn , that is, any two left inverses of Gn . Then, using result (4.26), we find that .2/ .1/ .1/ H.2/ n .A ˝ A/Gn D Hn Gn Hn .A ˝ A/Gn D Hn .A ˝ A/Gn :

(2) Using result (4.26) and observing (in light of the discussion in Section 16.1) that A ˝ A is a generalized inverse of A ˝ A, we find that Hn .A ˝ A/Gn Hn .A ˝ A /Gn Hn .A ˝ A/Gn D Hn .A ˝ A/.A ˝ A /Gn Hn .A ˝ A/Gn D Hn .A ˝ A/.A ˝ A /.A ˝ A/Gn D Hn .A ˝ A/Gn : (3) Set Hn D .Gn0 Gn /1 Gn0 —in light of Part (1), it suffices to verify Part (3) for a single choice of Hn . Then, using results (5.2.3), (4.22), (1.25), and (3.15), we find that trŒHn .A ˝ A/Gn D trŒ.A ˝ A/Gn Hn D trf.A ˝ A/Œ.1=2/.In2 C Knn /g D .1=2/ tr.A ˝ A/ C .1=2/ trŒ.A ˝ A/Knn D .1=2/Œtr.A/2 C .1=2/ tr.A2 /:


363

(4) Since [according to Part (2)] Hn .A ˝ A /Gn is a generalized inverse of Hn .A ˝ A/Gn , it follows from result (10.2.1) that rankŒHn .A ˝ A/Gn D trŒHn .A ˝ A/Gn Hn .A ˝ A /Gn : Further, using result (4.26), Part (3), and result (10.2.1), we find that trŒHn .A ˝ A/Gn Hn .A ˝ A /Gn D trŒHn .A ˝ A/.A ˝ A /Gn D trfHn Œ.AA / ˝ .AA /Gn g D .1=2/Œtr.AA /2 C .1=2/ tr.AA AA / D .1=2/Œtr.AA /2 C .1=2/ tr.AA / D .1=2/Œrank.A/2 C .1=2/ rank.A/: (5) Since Hn .A ˝ A/Gn is a square matrix of order n.n C 1/=2, Part (5) is an immediate consequence of Part (4). Q.E.D. For purposes of deriving a recursive formula for Hn .A˝A/Gn , let B represent an arbitrary .n C 1/ .n C 1/ matrix, partition B as c b0 BD a A .2/ (where A is nn), and define QnC1 D .Q.1/ nC1 ; QnC1 /, u, and U as in Subsection b. Then, ! 0 .1/ .1/ 0 .2/ Q.1/ nC1 .B ˝ B/QnC1 QnC1 .B ˝ B/QnC1 0 : QnC1 .B ˝ B/QnC1 D 0 .1/ .2/ 0 .2/ Q.2/ nC1 .B ˝ B/QnC1 QnC1 .B ˝ B/QnC1

Further, since 0 c b0 u Bu u0 BU 0 0 D InC1 BInC1 D .u; U / B.u; U / D ; a A U 0 Bu U 0 BU we find that 0 .1/ 0 Q.1/ nC1 .B ˝ B/QnC1 D .InC1 ˝ u /.B ˝ B/.InC1 ˝ u/ D B ˝ .u0 Bu/ D B ˝ c D cB; 0 .2/ 0 Q.1/ nC1 .B ˝ B/QnC1 D .InC1 ˝ u /.B ˝ B/.InC1 ˝ U / D B ˝ .u0 BU / D B ˝ b0 ; 0 .1/ 0 Q.2/ nC1 .B ˝ B/QnC1 D .InC1 ˝ U /.B ˝ B/.InC1 ˝ u/ 0 D B ˝ .U Bu/ D B ˝ a; 0 .2/ 0 Q.2/ nC1 .B ˝ B/QnC1 D .InC1 ˝ U /.B ˝ B/.InC1 ˝ U / D B ˝ .U 0 BU / D B ˝ A:

364


Thus, in light of results (1.27), (1.4), (3.14), and (3.12), we have that 1 0 2 c cb0 cb0 b0 ˝ b0 B ca cA ab0 .b0 ˝ A/Knn C C: Q0nC1 .B ˝ B/QnC1 D B 0 @ ca ab cA b0 ˝ A A A˝A a ˝ a Knn .a ˝ A/ a ˝ A And, in light of results (4.12), (4.5), (4.15), and (4.16) [as well as Part (1) of Theorem 16.4.1], it follows that, for Hn D .Gn0 Gn /1 Gn0 , HnC1 .B ˝ B/GnC1 0 1 1 0 1 0 0 1 0 0 0 B0 In 0 C C D @0 .1=2/In .1=2/In 0 A Q0nC1 .B ˝ B/QnC1 B @0 I n 0 A 0 0 0 Hn 0 0 Gn 0 1 c2 cb0 cb0 b0 ˝ b0 0 B C ˝ A/ D@ ca .1=2/.cA C ab0 / .1=2/.cA C ab0 / .bŒ.1=2/.I A n2 C Knn / Hn .a ˝ A/ Hn .A ˝ A/ Hn .a ˝ a/ Hn .a ˝ A/ 1 0 1 0 0 B0 In 0 C C B @0 In 0 A 0 0 Gn 0 1 c2 2cb0 .b0 ˝ b0 /Gn ca cA C ab0 .b0 ˝ A/Gn A: D@ (4.29) Hn .a ˝ a/ 2Hn .a ˝ A/ Hn .A ˝ A/Gn Result (4.29) provides what is in effect a recursive formula for Hn .A ˝ A/Gn . Let 2 0 c b ˝ b0 2cb0 SD Gn Hn .A ˝ A /Gn Hn Œa ˝ a; 2.a ˝ A/ ca cA C ab0 b0 ˝ A [where Hn D .Gn0 Gn /1 Gn0 ]. And observe [in light of Part (2) of Theorem 16.4.1] that S is a Schur complement of the n.n C 1/=2 n.n C 1/=2 submatrix Hn .A ˝ A/Gn in the lower right corner of partitioned matrix (4.29). Further, using results (4.26), (4.22), and (3.12), we find that Gn Hn .A ˝ A /Gn Hn Œa ˝ a; 2.a ˝ A/ D .A ˝ A /Gn Hn Œa ˝ a; 2.a ˝ A/ D .A ˝ A /Œ.1=2/.In2 C Knn /Œa ˝ a; 2.a ˝ A/ D .A ˝ A /Œa ˝ a; .a ˝ A/ C .A ˝ a/:

Thus,


365

1 2cb0 .b0 A a/ ˝ .b0 A A/ 2 0 0 c .b A a/ ˝ .b A a/ 0 0 B .b A A/ ˝ .b A a/ C C SDB @ cA C ab0 .b0 A a/ ˝ A A 0 ca .b A a/ ˝ .AA a/ .b0 A A/ ˝ .AA a/ c 2 .b0 A a/2 2cb0 2.b0 A a/b0 A A D ; ca .b0 A a/AA a .c b0 A a/A C ab0 AA ab0 A A 0

and, in the special case where A is nonsingular, c C b0 A1 a 2b0 S D .c b0 A1 a/ : a A

(4.30)

Consider now the determinant of the matrix HnC1 .B ˝ B/GnC1 . By applying result (13.3.13) to partitioned matrix (4.29), we find that if A is nonsingular [in which case it follows from Part (5) of Theorem 16.4.1 that Hn .A ˝ A/Gn is also nonsingular], then jHnC1 .B ˝ B/GnC1 j D jHn .A ˝ A/Gn jjSj:

(4.31)

Further, using result (4.30), Corollary 13.2.4, and result (13.3.13), we find that if A is nonsingular, then ˇ ˇ 0 1 0ˇ ˇ 0 1 nC1 ˇc C b A a 2b ˇ jSj D .c b A a/ ˇ a Aˇ D .c b0 A1 a/nC1 jAj.c C b0 A1 a 2b0 A1 a/ D .c b0 A1 a/nC2 jAj:

(4.32)

Together, results (4.31) and (4.32) imply that (in the special case where A is nonsingular) jHnC1 .B ˝ B/GnC1 j D .c b0 A1 a/nC2 jAjjHn .A ˝ A/Gn j:

(4.33)

Result (4.33), which provides what is in effect a recursive formula for the determinant of Hn .A ˝ A/Gn , can be used to establish the following theorem. Theorem 16.4.2. For any n n matrix A, jHn .A ˝ A/Gn j D jAjnC1:

(4.34)

Proof. The proof is by mathematical induction. Clearly, equality (4.34) holds for every 1 1 matrix A. Suppose that equality (4.34) holds for every n n matrix A. Let B represent an arbitrary .n C 1/ .n C 1/ matrix, and partition B as c b0 BD a A (where A is n n). To complete the induction argument, it suffices to show that

366


jHnC1 .B ˝ B/GnC1 j D jBjnC2 :

(4.35)

In establishing the validity of this equality, it is convenient to consider successively three different cases. Case (1): B singular. In this case, both the left and right sides of equality (4.35) equal 0—refer to Part (5) of Theorem 16.4.1. Case (2): Both A and B nonsingular. Since (by supposition) jHn .A˝A/Gn j D jAjnC1, it follows from result (4.33) [along with result (13.3.13)] that jHnC1 .B ˝ B/GnC1 j D .c b0 A1 a/nC2 jAjnC2 D jBjnC2: Case (3): B nonsingular, but A singular. Denote the first, : : : ; nth columns of A by a1 ; : : : ; an , respectively. Since B is nonsingular, the rows of B are linearly independent, implying that rank.a; A/ D n and also (since A is singular) that rank.A/ D n 1. Consequently, A contains n 1 linearly independent columns, say a1 ; : : : ; ai1 ; aiC1 ; : : : ; an , and a is not expressible as a linear combination of the columns of A. Let e represent the i th column of In . Then, the matrix A C ae0 , whose columns are a1 ; : : : ; ai1 ; ai Ca; aiC1 ; : : : ; an , respectively, is nonsingular (since Potherwise there would exist scalars c1 ; : : : ; ci1 ; ciC1 ; : : : ; cn such that ai Ca D j ¤i cj aj , which would imply that a is expressible as a linear combination of the columns of A). Now, let 1 e0 : TD 0 In Then,

c b0 C ce0 BT D : a A C ae0

Further, jT j D 1. Since In and A C ae0 are nonsingular, we have [from the same reasoning as in Case (2)] that jHnC1 .T ˝ T /GnC1 j D jT jnC2 D 1; jHnC1 Œ.BT / ˝ .BT /GnC1 j D jBT j

nC2

D jBj

nC2

jT j

nC2

(4.36) D jBj

nC2

: (4.37)

And, using result (4.26), we find that jHnC1 Œ.BT / ˝ .BT /GnC1 j D jHnC1 .B ˝ B/.T ˝ T /GnC1 j D jHnC1 .B ˝ B/GnC1 HnC1 .T ˝ T /GnC1 j D jHnC1 .B ˝ B/GnC1 jjHnC1 .T ˝ T /GnC1 j; which [in combination with results (4.36) and (4.37)] implies that jHnC1 .B ˝ B/GnC1 j D jBjnC2: Q.E.D.

16.5 Reformulation of a Linear System

367

16.5 Reformulation of a Linear System Consider a linear system AX D B;

(5.1)

with an m n coefficient matrix A, m p right side B, and n p matrix of unknowns X. This linear system can be reformulated as a linear system whose right side is an mp 1 vector and whose unknowns are arranged in an np 1 vector. This process is facilitated by the use of the vec operator (and the use of Kronecker products). Specifically, by applying the vec operator to each side of equation (5.1) and recalling result (2.9), we obtain the equivalent linear system .Ip ˝ A/x D b;

(5.2)

where b D vec B and where x D vec X is an np 1 vector of unknowns—the equivalence is in the sense that X is a solution to linear system (5.1) if and only if vec X is a solution to linear system (5.2). Note that linear system (5.2) can be rewritten as diag.A; A; : : : ; A/x D b: Let us now generalize from linear system (5.1) to the system AXC D B;

(5.3)

where A is an m n matrix, C a p q matrix, B an m q matrix, and X an n p matrix of unknowns. Upon applying the vec operator to each side of equation (5.3) and recalling result (2.10), we obtain the equivalent (linear) system .C 0 ˝ A/x D b; where b D vec B and where x D vec X is an np 1 vector of unknowns. A further generalization is possible. Consider the system r X

Ai XCi C

iD1

s X

Lj X0 Tj D B;

(5.4)

j D1

where A1 ; : : : ; Ar are m n matrices, C1 ; : : : ; Cr are p q matrices, L1 ; : : : ; Ls are m p matrices, T1 ; : : : ; Ts are n q matrices, B is an m q matrix, and X is an n p matrix of unknowns. Using results (2.6) and (2.10), we find that X X vec Ai XCi C Lj X0 Tj i

j

D

X

vec.Ai XCi / C

i

D

X

j

.Ci0

˝ Ai / vec X C

i

D

nX i

X

.Ci0

˝ Ai / C

hX j

vec.Lj X0 Tj / X j

.Tj0 ˝ Lj / vec X0

o i .Tj0 ˝ Lj / Knp vec X: (5.5)

368


Consequently, system (5.4) is equivalent to the (linear) system o nX hX i .Ci0 ˝ Ai / C .Tj0 ˝ Lj / Knp x D b; i

(5.6)

j

where b D vec B, and where x D vec X is an np 1 vector of unknowns. As a special case (that where “s D 0”), we have that the system r X

Ai XCi D B

(5.7)

iD1

is equivalent to the (linear) system hX i .Ci0 ˝ Ai / x D b:

(5.8)

i

In our consideration of system (5.4) [and systems (5.1), (5.3), and (5.7)], it has been implicitly assumed that there are no restrictions on X (other than those imposed by the system)—if there were restrictions on X, system (5.4) would be equivalent to system (5.6) only if equivalent restrictions were imposed on x. Suppose now that X is restricted to be symmetric [in which case p D n and system (5.4) is no more general than system (5.7)]. Then, since vec X D Gn vech X, system (5.7) is equivalent to the (linear) system i hX .Ci0 ˝ Ai / Gn x D b; (5.9) i

where now x D vech X is an n.n C 1/=2 1 (unrestricted) vector of unknowns [and where b D vec B]—the equivalence is in the sense that a symmetric matrix X is a solution to system (5.7) if and only if vech X is a solution to system (5.9).

16.6 Some Results on Jacobian Matrices In this section, attention is given to some Jacobian matrices of a kind encountered in multivariate statistical analysis. Let F represent a p q matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Then, vec F is a pq-dimensional column vector obtained by rearranging the elements of F (in a particular way). By definition, the Jacobian matrix of vec F is the pq m matrix @ vec.F /=@x0 , whose j th column is @ vec.F /=@xj . Note that @ vec F @F (6.1) D vec @xj @xj (as is evident from the very definition of the vec operator). Thus, the Jacobian matrix of vec F is the pq m matrix whose j th column is vec.@F =@xj /.

16.6 Some Results on Jacobian Matrices

369

Consider now the special case where q D p. If F is symmetric (for all x in S ), then p.p 1/=2 of the elements of vec F are redundant. In contrast, vech F contains no redundancies (that are attributable to symmetry). By definition, the Jacobian matrix of vech F is the p.p C 1/=2 m matrix @ vech.F /=@x0, whose j th column is @ vech.F /=@xj . Note that [analogous to result (6.1)] @ vech F @F : (6.2) D vech @xj @xj Thus, the Jacobian matrix of vech F is the p.p C 1/=2 m matrix whose j th column is vech.@F =@xj /. If F is symmetric, then clearly [in light of result (4.6)] @ vec F @ vech F D Gp ; @xj @xj

@ vec F @ vech F D Hp @xj @xj

(6.3)

@ vec F @ vech F D Hp : 0 @x @x0

(6.4)

(j D 1; : : : ; m), and consequently @ vec F @ vech F D Gp ; 0 @x @x0

Next, consider the Jacobian matrix of vec.AF B/, where F is a p q matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables and where A and B are r p and q s matrices of constants. Making use of formula (2.10), we find that @ vec.AF B/ @ vec F D .B0 ˝ A/ (6.5) @xj @xj (j D 1; : : : ; m) and hence that @ vec.AF B/ @ vec F D .B0 ˝ A/ : 0 @x @x0

(6.6)

Formula (6.6) relates the Jacobian matrix of vec.AF B/ to that of vec F . When s D q, r D p, and m D pq, it follows from results (6.6) and (3.18) that ˇ ˇ ˇ ˇ ˇ ˇ ˇ @ vec.AF B/ ˇ ˇ D jAjq jBjp ˇ @ vec F ˇ; ˇ (6.7) ˇ ˇ ˇ @x0 @x0 ˇ which relates the Jacobian of vec.AF B/ to that of vec F . In the special case where x D vec X and F .x/ D X for some p q matrix X of variables, result (6.6) simplifies [in light of result (15.3.12)] to @ vec.AXB/ D B0 ˝ A; @.vec X/0 and (when s D q and r D p) result (6.7) simplifies to ˇ ˇ ˇ @ vec.AXB/ ˇ q p ˇ ˇ ˇ @.vec X/0 ˇ D jAj jBj :

(6.8)

(6.9)

370


Suppose now that F is symmetric and that A is square (in which case both F and A are p p). Then, @ vech F @ vech.AF A0 / D Hp .A ˝ A/Gp ; @x0 @x0

(6.10)

as is evident from result (6.6) upon setting B D A0 and observing [in light of result (4.6)] that @ vech.AF A0 / @ vec.AF A0 / D H ; p @x0 @x0

@ vech F @ vec F D Gp : 0 @x @x0

When m D p.p C 1/=2, it follows from results (6.10) and (4.34) that ˇ ˇ ˇ ˇ ˇ ˇ ˇ @ vech.AF A0 / ˇ ˇ D jAjpC1 ˇ @ vech F ˇ : ˇ ˇ ˇ @x0 ˇ ˇ @x0

(6.11)

In the special case where x D vech X and F .x/ D X for some pp symmetric matrix X, results (6.10) and (6.11) simplify to @ vech.AXA0 / D Hp .A ˝ A/Gp ; @.vech X/0 ˇ ˇ ˇ @ vech.AXA0 / ˇ pC1 ˇ ˇ : ˇ @.vech X/0 ˇ D jAj Finally, consider the Jacobian matrix of vec.F 1 /, where F is a p p matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables [with F .x/ nonsingular for every x in S ]. Making use of results (15.8.15) and (2.10) [as well as result (6.1)], we find that 1 @F @ vec.F 1 / 1 @F 1 D vec F D vec F @xj @xj @xj @F D Œ.F 1 /0 ˝ F 1 vec @xj @ vec F D Œ.F 1 /0 ˝ F 1 (6.12) @xj (j D 1; : : : ; m), which implies that @ vec.F 1 / @ vec F D Œ.F 1 /0 ˝ F 1 ; @x0 @x0

(6.13)

thereby relating the Jacobian matrix of vec.F 1 / to that of vec F . Moreover, when m D p 2 , it follows from result (6.13), together with Corollary 13.2.5, result (3.18), 2 and Theorem 13.3.7 [and the fact that .1/p D .1/p ], that ˇ ˇ ˇ ˇ ˇ ˇ ˇ @ vec.F 1 / ˇ ˇ D .1/p jF j2p ˇ @ vec F ˇ ; ˇ (6.14) ˇ ˇ ˇ 0 0 @x @x ˇ

Exercises

371

which relates the Jacobian of vec.F 1 / to that of vec F . In the special case where x D vec X and F .x/ D X for some pp nonsingular matrix X of variables, results (6.13) and (6.14) simplify to @ vec.X1 / D .X1 /0 ˝ X1; @.vec X/0 ˇ ˇ ˇ @ vec.X1 / ˇ p 2p ˇ ˇ ˇ @.vec X/0 ˇ D .1/ jXj :

(6.15) (6.16)

Suppose now that the (p p) matrix F is symmetric. Then, @ vech.F 1 / @ vech F D Hp .F 1 ˝ F 1 /Gp ; (6.17) @x0 @x0 as is evident from result (6.13) upon observing [in light of result (4.6)] that @ vech.F 1 / @ vec.F 1 / D H ; p @x0 @x0

@ vech F @ vec F D Gp : 0 @x @x0

When m D p.p C 1/=2, it follows from Corollary 13.2.5, result (4.34), and Theorem 13.3.7 that ˇ ˇ ˇ ˇ ˇ ˇ ˇ @ vech.F 1 / ˇ ˇ D .1/p.pC1/=2 jF j.pC1/ ˇ @ vech F ˇ: ˇ (6.18) ˇ ˇ ˇ 0 0 @x @x ˇ In the special case where x D vech X and F .x/ D X for some p p nonsingular symmetric matrix X, results (6.17) and (6.18) simplify to @ vech.X1 / D Hp .X1 ˝ X1 /Gp ; @.vech X/0 ˇ ˇ ˇ @ vech.X1 / ˇ p.pC1/=2 ˇ ˇ jXj.pC1/: ˇ @.vech X/0 ˇ D .1/

(6.19) (6.20)

Exercises Section 16.1 1. (a) Verify result (1.13); that is, verify that, for any m n matrices A and B and any p q matrices C and D, .A C B/ ˝ .C C D/ D .A ˝ C/ C .A ˝ D/ C .B ˝ C/ C .B ˝ D/: (b) Verify result (1.14); that is, verify that, for any m n matrices A1 , A2 , : : : ; Ar and p q matrices B1 ; B2 ; : : : ; Bs , r X iD1

s r X s X X Ai ˝ Bj D .Ai ˝ Bj /: j D1

iD1 j D1

372


2. Show that, for any m n matrix A and p q matrix B, A ˝ B D .A ˝ Ip / diag.B; B; : : : ; B/: 3. Show that, for any m1 vector a and any p1 vector b, (1) a˝b D .a˝Ip /b and (2) a0 ˝ b0 D b0 .a0 ˝ Ip /. 4. Let A and B represent square matrices. (a) Show that if A and B are orthogonal, then A ˝ B is orthogonal. (b) Show that if A and B are idempotent, then A ˝ B is idempotent. 5. Letting m, n, p, and q represent arbitrary positive integers, show (a) that, for any p q matrix B (having p > 1 or q > 1), there exists an m n matrix A such that A ˝ B has generalized inverses that are not expressible in the form A ˝ B ; and (b) that, for any m n matrix A (having m > 1 or n > 1), there exists a p q matrix B such that A ˝ B has generalized inverses that are not expressible in the form A ˝ B. 6. Let X D A ˝ B, where A is an m n matrix and B a p q matrix. Show that PX D PA ˝ PB . 7. Show that the Kronecker product A ˝ B of an m m matrix A and an n n matrix B is (a) symmetric nonnegative definite if A and B are both symmetric nonnegative definite or both symmetric nonpositive definite, and (b) symmetric positive definite if A and B are both symmetric positive definite or both symmetric negative definite. 8. Let A and B represent mm symmetric matrices and C and D nn symmetric matrices. Using the result of Exercise 7 (or otherwise), show that if A B, C D, B, and C are nonnegative definite, then A˝C B˝D is symmetric nonnegative definite. 9. Let A represent an m n matrix and B a p q matrix. Show that, in the case of the usual norm, kA ˝ Bk D kAkkBk: 10. Verify result (1.27). 11. Show that (a) if T and U are both upper triangular matrices, then T ˝ U is an upper triangular matrix, and (b) if T and L are both lower triangular matrices, then T ˝ L is a lower triangular matrix.

Exercises

373

12. Let A represent an m m matrix and B an n n matrix. Suppose that A and B have LDU decompositions, say A D L1 D1 U1 and B D L2 D2 U2 . Using the results of Exercise 11, show that A ˝ B has the LDU decomposition A ˝ B D LDU , where L D L1 ˝ L2 , D D D1 ˝ D2 , and U D U1 ˝ U2 . Section 16.2 13. Let A1 ; A2 ; : : : ; Ak represent k matrices (of the same size). Show that A1 , A2 , : : : ; Ak are linearly independent if and only if vec.A1 /, vec.A2 /, : : : ; vec.Ak / are linearly independent. 14. Let m represent a positive integer, let ei represent the i th column of Im (i D 1; : : : ; m), and (for i; j D 1; : : : ; m) let Uij D ei ej0 (in which case Uij is an m m matrix whose ij th element is 1 and whose remaining m2 1 elements are 0). (a) Show that vec.Im / D

m X

e i ˝ ei :

iD1

(b) Show that (for i; j; r; s D 1; : : : ; m) vec.Uri /Œvec.Usj /0 D Uij ˝ Urs : (c) Show that

m X m X

Uij ˝ Uij D vec.Im /Œvec.Im /0 :

iD1 j D1

15. Let A represent an n n matrix. (a) Show that if A is orthogonal, then .vec A/0 vec A D n. (b) Show that if A is idempotent, then Œvec.A0 /0 vec A D rank.A/. 16. Show that for any m n matrix A, p n matrix X, p p matrix B, and n m matrix C, tr.AX0 BXC/ D .vec X/0 Œ.A0 C 0 / ˝ B vec X D .vec X/0 Œ.CA/ ˝ B0 vec X: 17. (a) Let V represent a linear space of m n matrices, and let g represent a function that assigns the value A B to each pair of matrices A and B in V. Take U to be the linear space of mn 1 vectors defined by U D fx 2 Rmn1 W x D vec.A/ for some A 2 Vg; and let x y represent the value assigned to each pair of vectors x and y in U by an arbitrary inner product f . Show that g is an inner product (for V) if and only if there exists an f such that (for all A and B in V)

374


A B D vec.A/ vec.B/: (b) Let g represent a function that assigns the value A B to an arbitrary pair of matrices A and B in Rmn . Show that g is an inner product (for Rmn ) if and only if there exists an mn mn partitioned symmetric positive definite matrix 0 1 W11 W12 : : : W1n BW21 W22 : : : W2n C B C W DB : :: :: C :: @ :: : : : A Wn1 Wn2 : : : Wnn (where each submatrix is of dimensions m m) such that (for all A and B in Rmn ) X a0i Wij bj ; A B D i;j

where a1 ; a2 ; : : : ; an and b1 ; b2 ; : : : ; bn represent the first, second, : : : ; nth columns of A and B, respectively. (c) Let g represent a function that assigns the value x0 y 0 to an arbitrary pair of (row) vectors in R1n . Show that g is an inner product (for R1n ) if and only if there exists an n n symmetric positive definite matrix W such that (for every pair of n-dimensional row vectors x0 and y 0 ) x0 y 0 D x0 W y: Section 16.3 18. (a) Define (for m 2) P to be the mn mn permutation matrix such that, for every m n matrix A, vec A D P vec A; r where A is the .m 1/ n matrix whose rows are respectively the first, 0 : : : ; .m 1/th rows of A and r is the mth row of A [and hence where A and A0 D .A0 ; r/]. AD r0 Km1;n 0 P. (1) Show that Kmn D 0 In (2) Show that jP j D .1/.m1/n.n1/=2. (3) Show that jKmn j D .1/.m1/n.n1/=2 jKm1;n j. (b) Show that jKmn j D .1/m.m1/n.n1/=4. (c) Show that jKmm j D .1/m.m1/=2. 19. Show that, for any m n matrix A, p 1 vector a, and q 1 vector b,

Exercises

375

(1) b0 ˝ A ˝ a D Kmp Œ.ab0 / ˝ A; (2) a ˝ A ˝ b0 D Kpm ŒA ˝ .ab0 /. 20. Let m and n represent positive integers, and let ei represent the i th column of Im (i D 1; : : : ; m) and uj represent the j th column of In (j D 1; : : : ; n). Show that Kmn D

n X

uj0 ˝ Im ˝ uj D

j D1

m X

ei ˝ In ˝ e0i :

iD1

21. Let m, n, and p represent positive integers. Using the result of Exercise 20, show that (a) Kmp;n D Kp;mn Km;np ; (b) Kmp;n Knp;m Kmn;p D I; (c) Kn;mp D Knp;m Kmn;p ; (d) Kp;mn Km;np D Km;np Kp;mn ; (e) Knp;m Kmn;p D Kmn;p Knp;m ; (f) Km;np Kmp;n D Kmp;n Km;np . [Hint. Begin P by letting uj represent the j th column of In and showing that Kmp;n D j .uj0 ˝ Ip / ˝ .Im ˝ uj / and then making use of result (3.10).] 22. Let A represent an m n matrix, and define B D Kmn .A0 ˝ A/. Show (a) that B is symmetric, (b) that rank.B/ D Œrank.A/2, (c) that B2 D .AA0 / ˝ .A0 A/, and (d) that tr.B/ D tr.A0 A/. 23. Show that, for any m n matrix A and any p q matrix B, vec.A ˝ B/ D .In ˝ G/ vec A D .H ˝ Ip / vec B; where G D .Kq m ˝ Ip /.Im ˝ vec B/ and H D .In ˝ Kq m /Œvec.A/ ˝ Iq . Section 16.4 24. Show that, for Hn D .Gn0 Gn /1 Gn0 , Gn Hn H0n D H0n : 25. There exists a unique matrix Ln such that vech A D Ln vec A for every n n matrix A (symmetric or not). [The matrix Ln is one choice for the matrix Hn discussed in Section 16.4b. It is referred to by Magnus and

376


Neudecker (1980) as the elimination matrix—the effect of premultiplying the vec of an nn matrix A by Ln is to eliminate (from vec A) the “supradiagonal” elements of A.] (a) Write out the elements of L1 , L2 , and L3 . (b) For an arbitrary positive integer n, describe Ln in terms of its rows. 26. Let A represent an n n matrix and b an n 1 vector. (a) Show that .1=2/Œ.A ˝ b0 / C .b0 ˝ A/Gn D .A ˝ b0 /Gn . (b) Show that, for Hn D .Gn0 Gn /1 Gn0 , (1) .1=2/Hn Œ.b ˝ A/ C .A ˝ b/ D Hn .b ˝ A/, (2) .A ˝ b0 /Gn Hn D .1=2/Œ.A ˝ b0 / C .b0 ˝ A/, (3) Gn Hn .b ˝ A/ D .1=2/Œ.b ˝ A/ C .A ˝ b/. 27. Let A D faij g represent an n n (possibly nonsymmetric) matrix. (a) Show that, for Hn D .Gn0 Gn /1 Gn0 , Hn vec A D .1=2/ vech.A C A0 /: (b) Show that Gn0 Gn vech A D vechŒ2A diag.a11 ; a22 ; : : : ; ann /: (c) Show that Gn0 vec A D vechŒA C A0 diag.a11 ; a22 ; : : : ; ann /: 28. Let A represent an n n matrix. Show that, for Hn D .Gn0 Gn /1 Gn0 , Gn Hn .A ˝ A/H0n D .A ˝ A/H0n : 29. Show that if an n n matrix A D faij g is upper triangular, lower triangular, or diagonal, then Hn .A˝A/Gn is respectively upper triangular, lower triangular, or diagonal with diagonal elements ai i ajj (i D 1; : : : ; n; j D i; : : : ; n). Section 16.5 30. Let A1 ; : : : ; Ak , and B represent m n matrices, and let b D vec B. P (a) Show that the matrix equation kiD1 xi Ai D B (in unknowns x1 ; : : : ; xk ) is equivalent to a linear system of the form Ax D b, where x D .x1 ; : : : ; xk /0 is a vector of unknowns. (b) Show that if A1 ; : : : ; Ak , and B are symmetric, then the matrix equation Pk iD1 xi Ai D B (in unknowns x1 ; : : : ; xk ) is equivalent to a linear system


377

of the form A x D b, where b D vech B and x D .x1 ; : : : ; xk /0 is a vector of unknowns. Section 16.6 31 Let F represent a p p matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Show that, for k D 2; 3; : : : ; k

X @ vec.F k / @ vec F D Œ.F s1 /0 ˝ F ks @x0 @x0 sD1 (where F 0 D Ip ). 32. Let F D ffis g and G represent p q and r s matrices of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. (a) Show that (for j D 1; : : : ; m) @.F ˝ G/ @F @G C D F˝ ˝G : @xj @xj @xj (b) Show that (for j D 1; : : : ; m) @ vec.F ˝ G/ @xj

@ vec G @ vec F D .Iq ˝ Ksp ˝ Ir / .vec F /˝ C ˝.vec G/ : @xj @xj

(c) Show that @ vec.F ˝ G/ @x0

@ vec G @ vec F D .Iq ˝ Ksp ˝ Ir / .vec F /˝ C ˝.vec G/ : @x0 @x0

(d) Show that, in the special case where x0 D Œ.vec X/0 ; .vec Y /0 , F .x/ D X, and G.x/ D Y for some p q and r s matrices X and Y of variables, the formula in Part (c) simplifies to @ vec.X ˝ Y / D .Iq ˝ Ksp ˝ Ir /.Ipq ˝.vec Y /; .vec X/ ˝ Irs /: @x0

Bibliographic and Supplementary Notes §1. For some historical notes on Kronecker products, refer to Henderson and Searle (1981b, sec. 2.2). Warning: There exist presentations (e.g., Graybill 1983) where

378


the left Kronecker product of two matrices A and B, which is denoted herein by B ˝A, is referred to as the Kronecker (or direct) product of A and B and is denoted by A ˝ B. §2. The idea of rearranging the elements of a matrix in the form of a vector dates back at least to Sylvester (1884). Refer to Henderson and Searle (1979, sec. 1; 1981b, sec. 2.1) for some historical notes on the development of this idea. In some presentations (e.g., Rogers 1980), the vec of a matrix A is called the pack of A. §3. Refer to Henderson and Searle (1981b) and to Magnus and Neudecker (1986) for some historical notes on vec-permutation matrices and their properties. Their papers include discussions of the alternative names that have been used in referring to vec-permutation matrices and of the various symbols that have been used to denote these matrices. §4. Henderson and Searle (1979) and Magnus and Neudecker (1986) provide references to early work on the vech operator (and on related concepts) and review the properties of the vech operator. They indicate that the idea of including only the distinct elements of a symmetric matrix in its vector representation dates back at least to Aitken (1949). This idea can be extended from symmetric matrices to other matrices that contain “redundant” elements—refer, for example, to Henderson and Searle (1979). §6. The results of Section 6 depend to a very significant extent on the material on matrix differentiation presented in Chapter 15.

17 Intersections and Sums of Subspaces

The orthogonality of a pair of subspaces was defined and discussed in Chapter 12. There are pairs of subspaces that are not orthogonal but that have a weaker property called essential disjointness—orthogonal subspaces are essentially disjoint, but essentially disjoint subspaces are not necessarily orthogonal. Unlike orthogonality, essential disjointness does not depend on the choice of inner product. The essential disjointness of a pair of subspaces is defined and discussed in the present chapter. The concept of essential disjointness arises in a very natural and fundamental way in results (like those of Sections 17.2, 17.3, and 17.5) on the ranks, row and column spaces, and generalized inverses of partitioned matrices and of sums and products of matrices. It was found in Chapter 12 that a subspace U of a linear space V and its orthogonal complement U? (which is a subspace of V that is by definition orthogonal to U and for which dim U C dim U? D dim V) have the property that every matrix Y in V can be uniquely expressed as the sum of a matrix in U (which is the projection of Y on U) and a matrix in U? (which is the projection of Y on U? )—refer to Theorem 12.5.11. It is shown in Section 17.6 that this property extends to any subspaces U and W of V that are essentially disjoint and that have dim UCdim W D dim V. That is, for any such subspaces U and W, every matrix in V can be uniquely expressed as the sum of a matrix in U and a matrix in W.

17.1 Definitions and Some Basic Properties Let U and V represent subsets of the linear space Rmn of all m n matrices. The intersection of U and V is the subset (of Rmn ) comprising all m n matrices that are common to U and V (i.e., that belong to both U and V) and is denoted by the symbol U \ V. If U and V are subspaces (of Rmn ), then their intersection U \ V is also a subspace. To see this, suppose that U and V are subspaces. Then, the m n null matrix 0 belongs to both U and V and hence to U \ V, so that U \ V is nonempty. Further, for any matrix A in U \ V and any scalar k, A is in U and

380


also in V, implying that kA is in both U and V and consequently in U \ V. And, for any matrices A and B in U \ V, both A and B are in U and both are also in V, implying that A C B is in U and also in V and hence that A C B is in U \ V. More generally, the intersection of k subsets U1 ; : : : ; Uk of Rmn (where k 2) is the subset (of Rmn ) comprising all m n matrices that are common to U1 ; : : : ; Uk and is denoted by the symbol U1 \ \ Uk (or alternatively by \kiD1 Ui or simply \i Ui ). And, if U1 ; : : : ; Uk are subspaces (of Rmn ), then U1 \ \ Uk is also a subspace. Note that if (for subsets U, V, and W of Rmn ) W U and W V, then W U\V. Note also that if U W and V W, then U \ V W. The sum of two (nonempty) subsets U and V of Rmn is defined to be the subset fU CV W U 2U; V 2Vg comprising every (mn) matrix that is expressible as the sum of a matrix in U and a matrix in V. It is denoted by the symbol U C V. Note that, except for special cases, U C V differs from the union U [ V of U and V, which by definition is the subset (of Rmn ) comprising all m n matrices that belong to U or V. The following lemma provides a useful characterization of the sum of two subspaces of Rmn . Lemma 17.1.1. Let U and V represent subspaces of Rmn. If V D f0g (i.e., if there are no nonnull matrices in V), then UCV D U, and similarly if U D f0g, then U C V D V. Further, if U is spanned by a (finite nonempty) set of (m n) matrices U1 ; : : : ; Us and V is spanned by a set of matrices V1 ; : : : ; V t , then U C V D sp.U1 ; : : : ; Us ; V1 ; : : : ; V t /: Proof. That U C V D U if V D f0g and that U C V D V if U D f0g is obvious. Suppose now that U is spanned by the set fU1 ; : : : ; Us g and V by the set fV1 ; : : : ; V t g. Then, for any matrix U in U and P any matrix V in V,Pthere exist scalars c1 ; : : : ; cs and k1 ; : : : ; k t such that U D siD1 ci Ui and V D jt D1 kj Vj and hence such that s t X X ci U i C kj Vj ; U CV D iD1

j D1

implying that U C V 2 sp.U1 ; : : : ; Us ; V1 ; : : : ; V t /. Thus, U C V sp.U1 ; : : : ; Us ; V1 ; : : : ; V t /:

(1.1)

Further, for any matrix A in sp.U1P ; : : : ; Us ; V1 ; : P : : ; V t /, there exist scalars c1 , : : : ; cs , k1 , : : : ; k t such that A D siD1 ci Ui C jt D1 kj Vj , implying that A U in U and some matrix V in V (namely, U D Pmatrix PsD U C V for some t iD1 ci Ui and V D j D1 kj Vj ). It follows that sp.U1 ; : : : ; Us ; V1 ; : : : ; V t / U C V, leading [in light of result (1.1)] to the conclusion that U C V D Q.E.D. sp.U1 ; : : : ; Us ; V1 ; : : : ; V t /. As an immediate consequence of Lemma 17.1.1, we have the following corollary.

17.1 Definitions and Some Basic Properties

381

Corollary 17.1.2. The sum U C V of two subspaces U and V (of Rmn ) is itself a subspace. Further, if both U and V are contained in a subspace W (of Rmn ), then U C V is contained in W. The addition of (nonempty) subsets of Rmn (like the addition of matrices in ) is commutative and associative (as is easily verified). Thus, taking U and R V to be two (nonempty) subsets of Rmn , mn

U C V D V C U;

(1.2)

and, taking W to be a third (nonempty) subset, U C .V C W/ D .U C V/ C W:

(1.3)

The symbol U C V C W is used to represent the subset that is common to the left and right sides of equality (1.3), and this subset is referred to as the sum of U, V, and W. Further, U C V C W D fU C V C W W U 2 U; V 2 V; W 2 Wg: This notation, terminology, and property extend in an obvious way to any finite number of (nonempty) subsets. Moreover, the sum of any (finite) number of subspaces is itself a subspace. The column space C.A/ and the row space R.A/ of an m n matrix A are subspaces of Rm and Rn , respectively. Further, C.A/ is spanned by the n columns of A, and R.A/ is spanned by the m rows of A. Thus, it follows from Lemma 17.1.1 that, for any m n matrix A and any m p matrix B, C.A; B/ D C.A/ C C.B/ and, similarly, that, for any m n matrix A and any p n matrix C, A R D R.A/ C R.C/: C

(1.4)

(1.5)

Results (1.4) and (1.5) can be extended (by repeated application). For any matrices A1 ; A2 ; : : : ; Ak having the same number of rows, C.A1 ; A2 ; : : : ; Ak / D C.A1 / C C.A2 / C C C.Ak /I

(1.6)

and similarly, for any matrices A1 ; A2 ; : : : ; Ak having the same number of columns, 0 1 A1 B A2 C B C RB : C D R.A1 / C R.A2 / C C R.Ak /: (1.7) @ :: A Ak Let U and V represent subspaces of the linear space Rmn . If U \ V D f0g, that is, if the only matrix that U and V have in common is the m n null matrix,

382


then U and V are said to be essentially disjoint. Note that the intersection of essentially disjoint subspaces of Rmn is not the empty set; rather, it is a set with one member (the m n null matrix)—this is the reason for referring to essentially disjoint subspaces as essentially disjoint rather than as disjoint. (Since the m n null matrix is a member of every subspace of Rmn, it is not possible for the intersection of two subspaces to be the empty set.) The following lemma gives an important property of essentially disjoint subspaces. Lemma 17.1.3. Let U and V represent subspaces (of dimension 1 or more) of Rmn . And, let fU1 ; : : : ; Ur g represent any linearly independent set of matrices in U, and fV1 ; : : : ; Vs g any linearly independent set of matrices in V. If U and V are essentially disjoint, then the combined set fU1 ; : : : ; Ur ; V1 ; : : : ; Vs g is linearly independent. Proof. It suffices to show that if fU1 ; : : : ; Ur ; V1 ; : : : ; Vs g is linearly dependent, then U and V are not essentially disjoint. Suppose that fU1 ; : : : ; Ur ; V1 ; : : : ; Vs g is linearly dependent. Then, for some scalars c1 ; : : : cr and k1 ; : : : ; ks (not all of which are 0), r X iD1

or, equivalently,

r X iD1

ci U i C

s X

kj Vj D 0

j D1

ci U i D

s X

kj Vj :

(1.8)

j D1

Pr Pr We have that iD1 ci Ui ¤ 0—since if iD1 ci Ui D 0 (in which case P s k V D 0), we would be led to the contradictory conclusion that fU1 ; : : : ; j j j D1 Pr Ur g or fV1 ; : : : ; Vs g is linearly dependent. Thus, since iD1 ci Ui 2 U and [in Pr c U 2 V, we conclude that U \ V contains a light of equality (1.8)] iD1 i i nonnull matrix and hence that U and V are not essentially disjoint. Q.E.D. mn , then, for any Note that if U and V are essentially disjoint subspaces of R subspace W of U and any subspace X of V, W and V are essentially disjoint, U and X are essentially disjoint, and more generally W and X are essentially disjoint. The following lemma gives two alternative characterizations of essentially disjoint subspaces. Lemma 17.1.4. Let U and V represent subspaces of Rmn. Then, (1) U and V are essentially disjoint if and only if, for matrices U 2 U and V 2 V, the only solution to the matrix equation U CV D0

(1.9)

is U D V D 0; and (2) U and V are essentially disjoint if and only if, for every nonnull matrix U in U and every nonnull matrix V in V, U and V are linearly independent.

17.1 Definitions and Some Basic Properties

383

Proof. (1) It suffices to show that U and V are not essentially disjoint if and only if, for U 2 U and V 2 V, equation (1.9) or, equivalently, the equation U D V ;

(1.10)

has a solution other than U D V D 0. Suppose that (for U 2 U and V 2 V) equation (1.10) has a solution other than U D V D 0, say U D U and V D V . Then, U is nonnull. Further, since V 2 V, U is a member of V (as well as of U), and hence U 2 U\V. Thus, U and V are not essentially disjoint. Conversely, suppose that U and V are not essentially disjoint. Then, U \ V contains a nonnull matrix U . And, since U D .U /, U 2 U, and U 2 V, a solution to equation (1.10) (that satisfies the constraints U 2 U and V 2 V and differs from U D V D 0) can be obtained by taking U D U and V D U . (2) Suppose that U and V are essentially disjoint. Then, upon applying Lemma 17.1.3 (in the special case where r D s D 1), we find that, for every nonnull matrix U in U and every nonnull matrix V in V, U and V are linearly independent. Conversely, suppose that, for every nonnull U 2 U and every nonnull V 2 V, U and V are linearly independent. Then, there does not exist any nonnull matrix belonging to both U and V, since any such matrix would not be linearly independent of itself. Q.E.D. When two subspaces U and V of Rmn are essentially disjoint, their sum U C V is sometimes referred to as a direct sum and is sometimes denoted by the symbol U ˚ V. By stating that a subspace W is the direct sum of two subspaces U and V or by writing W D U ˚ V, we can simultaneously indicate that W is the sum of U and V and that U and V are essentially disjoint. Or, even if it has already been indicated that two subspaces U and V are essentially disjoint, we can emphasize their essential disjointness by referring to their sum as a direct sum and/or by using the notation U ˚ V. Direct sums have an important property, which is described in the following theorem. Theorem 17.1.5. Let U represent a p-dimensional subspace and V a qdimensional subspace of Rmn. Further, let S1 and S2 represent bases for U and V, respectively, and define S to be the set of p C q matrices obtained by combining the p matrices in S1 with the q matrices in S2 . If U and V are essentially disjoint (or equivalently if U C V is a direct sum), then S is a basis for U C V; if U and V are not essentially disjoint, then S contains a proper subset that is a basis for U C V. Proof. It is clear from Lemma 17.1.1 that S spans U C V. Now, suppose that U and V are essentially disjoint. Then, it is evident from Lemma 17.1.3 that S is a linearly independent set. Thus, S is a basis for U C V. Alternatively, suppose that U and V are not essentially disjoint. Then, U \ V P contains a nonnull matrix, say U . Further, U D piD1 ci Ui , where c1 ; : : : ; cp are scalars (not all of which can be zero) and U1 ; : : : ; Up are the matrices in S1 .

384


P Similarly, U D jqD1 kj Vj , where k1 ; : : : ; kq are scalars and V1 ; : : : ; Vq are the matrices in S2 . Thus, p X iD1

ci U i C

q X

.kj /Vj D U U D 0;

j D1

implying that S is a linearly dependent set. We conclude that S itself is not a basis and consequently (in light of Theorem 4.3.11) that S contains a proper subset that is a basis for U C V. Q.E.D. The following corollary is an immediate consequence of Theorem 17.1.5. Corollary 17.1.6. Let U and V represent subspaces of Rmn . If U and V are essentially disjoint (or equivalently if U C V is a direct sum), then dim.U ˚ V/ D dim.U/ C dim.V/I

(1.11)

and if U and V are not essentially disjoint, then dim.U C V/ < dim.U/ C dim.V/: Or, equivalently, dim.U C V/ dim.U/ C dim.V/;

(1.12)

with equality holding if and only if U and V are essentially disjoint. Corollary 17.1.6 leads to the following result. Corollary 17.1.7. Let U and V represent subspaces of a linear space W. If U and V are essentially disjoint and if dim.U/ C dim.V/ D dim.W/, then U ˚ V D W. Proof (of Corollary 17.1.7). Suppose that U and V are essentially disjoint and that dim.U/ C dim.V/ D dim.W/. Then, it follows from Corollary 17.1.6 that dim.U ˚ V/ D dim.W/. Since (according to Corollary 17.1.2) U ˚ V W, we conclude (on the basis of Theorem 4.3.10) that U ˚ V D W. Q.E.D. As a special case of Corollary 17.1.7, we have the following corollary. Corollary 17.1.8. Let U and V represent subspaces of Rmn . If U and V are essentially disjoint and if dim.U/ C dim.V/ D mn, then U ˚ V D Rmn. The following lemma establishes an important relationship. Lemma 17.1.9. Let U and V represent subspaces of Rmn . If U ? V (i.e., if U and V are orthogonal), then U \ V D f0g (i.e., U and V are essentially disjoint). Proof. Suppose that U ? V. Then, for any matrix U in U\V (i.e., in both U and V), U ? U , or, equivalently, U U D 0, implying that U D 0. We conclude that U \ V D f0g. Q.E.D. The converse of Lemma 17.1.9 is not necessarily true. That is, essentially disjoint subspaces are not necessarily orthogonal. Suppose, for example, that U

17.2 Results on Row and Column Spaces and on Ranks of Partitioned Matrices

385

is the one-dimensional subspace spanned by the vector .1; 0/ and V is the onedimensional subspace spanned by .1; 1/. Then, U and V are essentially disjoint; however, U and V are not orthogonal (with respect to the usual inner product). As an immediate consequence of Lemma 17.1.9, we have the following corollary. Corollary 17.1.10. Let U and V represent subspaces of Rmn. If U and V are orthogonal, then their sum is a direct sum.

17.2 Some Results on Row and Column Spaces and on the Ranks of Partitioned Matrices The following lemma is an almost immediate consequence of Lemma 4.1.1. Lemma 17.2.1. For any m n matrix A and m p matrix B, C.A/ and C.B/ are essentially disjoint if and only if R.A0 / and R.B0 / are essentially disjoint. The following lemma can be useful in establishing that a matrix equals a null matrix. Lemma 17.2.2. (1) Let A represent an m n matrix and B an m p matrix whose column spaces C.A/ and C.B/ are essentially disjoint. And let X represent an m q matrix. If C.X/ C.A/ and C.X/ C.B/, or, equivalently, if X D AK and X D BL for some matrices K and L, then X D 0. (2) Similarly, let A represent an m n matrix and B a p n matrix whose row spaces R.A/ and R.B/ are essentially disjoint. And let X represent a q n matrix. If R.X/ R.A/ and R.X/ R.B/, or, equivalently, if X D KA and X D LB for some matrices K and L, then X D 0. Proof. (1) If C.X/ C.A/ and C.X/ C.B/, then C.X/ C.A/\C.B/ D f0g; implying that rank.X/ D dimŒC.X/ D 0 and hence that X D 0. (2) The proof of Part (2) is analogous to that of Part (1). As a variation on Lemma 17.2.2, we have the following corollary.

Q.E.D.

Corollary 17.2.3. (1) Let A represent an m n matrix and B an m p matrix whose column spaces C.A/ and C.B/ are essentially disjoint. And, let K represent an n q matrix and L a p q matrix. If AK D BL, then AK D 0 and BL D 0. (2) Similarly, let A represent an m n matrix and B a p n matrix whose row spaces R.A/ and R.B/ are essentially disjoint. And let K represent a q m matrix and L a q p matrix. If KA D LB, then KA D 0 and LB D 0. By applying Corollary 17.1.6 in the special case where U and V are the column spaces of an m n matrix A and an m p matrix B, and in the special case where U and V are the row spaces of an m n matrix A and a q n matrix C, we obtain [in light of equalities (1.4) and (1.5)] the following theorem, which sharpens the results of Lemma 4.5.7.

386


Theorem 17.2.4. (1) Let A represent an m n matrix and B an m p matrix. If C.A/ and C.B/ are essentially disjoint, then rank.A; B/ D rank.A/ C rank.B/; and if C.A/ and C.B/ are not essentially disjoint, then rank.A; B/ < rank.A/ C rank.B/: Or, equivalently, rank.A; B/ rank.A/ C rank.B/;

(2.1)

with equality holding if and only if C.A/ and C.B/ are essentially disjoint. (2) Let A represent an m n matrix and C a q n matrix. If R.A/ and R.C/ are essentially disjoint, then A rank D rank.A/ C rank.C/; C and if R.A/ and R.C/ are not essentially disjoint, then A rank < rank.A/ C rank.C/: C Or, equivalently,

A rank rank.A/ C rank.C/; C

(2.2)

with equality holding if and only if R.A/ and R.C/ are essentially disjoint. A further result on row and column spaces is given by the following lemma. Lemma 17.2.5. (1) Let A represent an m n matrix and B an m p matrix such that C.A/ and C.B/ are essentially disjoint and rank.A/ C rank.B/ D m. Then, corresponding to any m q matrix C, there exist an n q matrix R and a p q matrix S such that C D AR C BS: (2) Similarly, let A represent an m n matrix and B a p n matrix such that R.A/ and R.B/ are essentially disjoint and rank.A/Crank.B/ D n. Then, corresponding to any q n matrix C, there exist a q m matrix R and a q p matrix S such that C D RA C SB: Proof. (1) It follows from Corollary 17.1.8 that C.A/ C C.B/ D Rm or, equivalently [in light of result (1.4)], that C.A; B/ D Rm. Thus, C.C/ C.A; B/, implying (in light of Lemma 4.2.2) that CD .A; B/L for some .nCp/q matrix R L. And, upon partitioning L as L D (where R has n rows), we find that S C D AR C BS. (2) The proof of Part (2) is analogous to that of Part (1). Q.E.D. The following lemma characterizes the idempotency of an n n matrix A in terms of the essential disjointness of two subspaces of Rn.


387

Lemma 17.2.6. (1) An n n matrix A is idempotent if and only if C.A/ and C.I A/ are essentially disjoint. (2) An n n matrix A is idempotent if and only if R.A/ and R.I A/ are essentially disjoint. Proof. (1) Since AA2 D A.I A/ and AA2 D .I A/A, it follows from Lemma 17.2.2 that if C.A/ and C.I A/ are essentially disjoint, then A A2 D 0 or, equivalently, A is idempotent. Conversely, suppose that A is idempotent. Let x represent an arbitrary vector in C.A/ \ C.I A/. Then, x D Ay and x D .I A/z for some vectors y and z, implying that x D AAy D Ax D A.I A/z D .A A2 /z D 0: Thus, C.A/ and C.I A/ are essentially disjoint. (2) The proof of Part (2) is analogous to that of Part (1). The following lemma has some useful consequences.

Q.E.D.

Lemma 17.2.7. For any mn matrix A, C.A/ and C.IAA / are essentially disjoint, and R.A/ and R.I A A/ are essentially disjoint. Proof. Since (according to Lemma 10.2.5) AA is idempotent, it follows from Lemma 17.2.6 that C.AA / and C.I AA / are essentially disjoint and hence (in light of Lemma 9.3.7) that C.A/ and C.I AA / are essentially disjoint. That R.A/ and R.I A A/ are essentially disjoint follows from an analogous argument. Q.E.D. In light of Lemma 4.2.2 and Theorem 17.2.4, we have the following corollary (of Lemma 17.2.7). Corollary 17.2.8. Let A represent an m n matrix, B an m q matrix, and C a q n matrix. Then, C.A/ and CŒ.I AA /B are essentially disjoint, and rankŒA; .I AA /B D rank.A/ C rankŒ.I AA /B: Similarly, R.A/ and RŒC.I A A/ are essentially disjoint, and A D rank.A/ C rankŒC.I A A/: rank C.I A A/

(2.3)

(2.4)

More generally, for any m p matrix F such that C.F / C.A/ (or, equivalently, for any m p matrix F that is expressible as F D AK for some n p matrix K), C.F / and CŒ.I AA /B are essentially disjoint, and rankŒF ; .I AA /B D rank.F / C rankŒ.I AA /B:

(2.5)

And, for any p n matrix H such that R.H/ R.A/ (or, equivalently, for any p n matrix H that is expressible as H D LA for some p m matrix L), R.H/ and RŒC.I A A/ are essentially disjoint, and H rank D rank.H/ C rankŒC.I A A/: (2.6) C.I A A/

388


Upon combining Lemma 4.5.4 with Corollary 17.2.8 [and recalling results (1.4) and (1.5)], we obtain as a second corollary (of Lemma 17.2.7) the following result. Corollary 17.2.9. For any m n matrix A and m p matrix B, C.A; B/ D CŒA; .I AA /B D C.A/ ˚ CŒ.I AA /B;

(2.7)

rank.A; B/ D rankŒA; .I AA /B D rank.A/ C rankŒ.I AA /B: (2.8) Similarly, for any m n matrix A and q n matrix C, A A D R.A/ ˚ RŒC.I A A/; R DR C C.I A A/ A A D rank.A/ C rankŒC.I A A/: rank D rank C C.I A A/

(2.9) (2.10)

Three additional corollaries (of Lemma 17.2.7) are as follows. Corollary 17.2.10. Let A represent an m n matrix, B an m p matrix, and C a q n matrix. Then, rankŒ.I AA /B D rank.B/ if and only if C.A/ and C.B/ are essentially disjoint, and, similarly, rankŒC.I A A/ D rank.C/ if and only if R.A/ and R.C/ are essentially disjoint. Proof. According to Corollary 17.2.9, rankŒ.I AA /B D rank.A; B/ rank.A/: Moreover, according to Theorem 17.2.4, rank.A; B/ D rank.A/ C rank.B/, or, equivalently, rank.A; B/ rank.A/ D rank.B/, if and only if C.A/ and C.B/ are essentially disjoint. We conclude that rankŒ.I A A/B D rank.B/ if and only if C.A/ and C.B/ are essentially disjoint. That rankŒC.I A A/ D rank.C/ if and only if R.A/ and R.C/ are essentially disjoint follows from a similar argument. Q.E.D. Corollary 17.2.11. Let A represent an m n matrix, B an m p matrix, and C a q n matrix. Then, RŒ.I AA /B D R.B/ if and only if C.A/ and C.B/ are essentially disjoint, and, similarly, CŒC.I A A/ D C.C/ if and only if R.A/ and R.C/ are essentially disjoint. Proof. Corollary 17.2.11 follows from Corollary 17.2.10 upon observing (in light of Corollary 4.4.7) that RŒ.I AA /B D R.B/ if and only if rankŒ.I AA /B D rank.B/ and that CŒC.I A A/ D C.C/ if and only if rankŒC.I Q.E.D. A A/ D rank.C/. Corollary 17.2.12. Let A represent an m n matrix, B an m p matrix, and C a q n matrix. If AA B D 0 or BB A D 0, then C.A/ and C.B/ are essentially disjoint. Similarly, if CA A D 0 or AC C D 0, then R.A/ and R.C/ are essentially disjoint. Proof. If AA B D 0 or BB A D 0, then clearly rankŒ.I AA /B D rank.B/ or rankŒ.I BB /A D rank.A/, and it follows from Corollary 17.2.10


389

that C.A/ and C.B/ are essentially disjoint. Similarly, if CA A D 0 or AC C D 0, then clearly rankŒC.I A A/ D rank.C/ or rankŒA.I C C/ D rank.A/, and it follows from Corollary 17.2.10 that R.A/ and R.C/ are essentially disjoint. Q.E.D. In connection with Lemma 17.2.7 (and its corollaries), recall [from Part (1) of Theorem 12.3.4] that one choice for A is .A0 A/A0 , and observe that, for A D .A0 A/A0 , AA D PA . Similarly, recall [from Part (5) of Theorem 12.3.4] that another choice for A is A0 .AA0 / , and observe that, for A D A0 .AA0 /, AA D PA0 . As a consequence of Corollary 17.2.9, we have that, for any m n matrix A and m p matrix B, there exists a matrix K (having m rows) such that C.A/ and C.K/ are essentially disjoint and C.A; B/ D C.A; K/ or, equivalently, such that C.A; B/ equals the direct sum of C.A/ and C.K/. One choice for K (that given by Corollary 17.2.9) is K D .I AA /B. The following theorem gives, for the special case where B is an (m m) symmetric nonnegative definite matrix, an alternative choice for K. Theorem 17.2.13. For any m n matrix A and any m m symmetric nonnegative definite matrix V , C.A/ and CfV ŒI .A0 /A0 g are essentially disjoint and C.A; V / D CfA; V ŒI .A0 /A0 g, or, equivalently, C.A; V / D C.A/ ˚ CfV ŒI .A0 /A0 g:

(2.11)

Proof. Let Z D I .A0 /A0 , and observe that A0 Z D 0 and hence that Z A D .A0 Z/0 D 0. Let x represent an arbitrary (m 1) vector in C.A/ \ C.V Z/. Then, x D Ar for some vector r, and x D V Zs for some vector s. Moreover, 0

Z0 V Zs D Z0 x D Z0 Ar D 0; so that

.Zs/0 V Zs D s0 Z0 V Zs D 0;

implying (in light of Corollary 14.3.11) that V Zs D 0 or equivalently that x D 0. Thus, C.A/ and C.V Z/ are essentially disjoint. Now, it is clear from Lemma 4.5.5 that C.A; V Z/ C.A; V /: And, making use of Theorem 17.2.4 and result (2.8) and observing that rank.V Z/ D rankŒ.V Z/0 D rank.fI AŒ.A0 / 0 gV / and that Œ.A0 / 0 is a generalized inverse of A, we find that rank.A; V Z/ D rank.A/ C rank.V Z/ D rank.A/ C rank.fI AŒ.A0 / 0 gV / D rank.A; V /:

390


We conclude, on the basis of Theorem 4.4.6, that C.A; V Z/ D C.A; V /. Q.E.D. In connection with Theorem 17.2.13, recall (from Corollary 11.2.2) that CŒI .A0 /A0 D N.A0 /, and observe (in light of Corollaries 4.2.4 and 4.5.6) that, for any matrix Z such that C.Z/ D N.A0 /, CfV ŒI .A0 /A0 g D C.V Z/ and CfA; V ŒI .A0 /A0 g D C.A; V Z/. Thus, we have the following corollary. Corollary 17.2.14. Let A represent an m n matrix, and let V represent an m m symmetric nonnegative definite matrix. Then, for any matrix Z such that C.Z/ D N.A0 / (i.e., any matrix Z whose columns span the null space of A0 ), C.A/ and C.V Z/ are essentially disjoint and C.A; V / D C.A; V Z/, or, equivalently, C.A; V / D C.A/ ˚ C.V Z/:

(2.12)

The following lemma gives a basic result on the essential disjointness of the row or column spaces of partitioned matrices. Lemma 17.2.15. (1) Let A D .A1 ; A2 / represent an mn partitioned matrix and B D .B1 ; B2 / a p n partitioned matrix, where B1 has the same number of columns, say n1 , as A1 . If R.A1 / and R.B1 / are essentially disjoint and if R.A2 / and R.B2 / are essentially disjoint, then R.A/ and R.B/ are essentially A1 represent an m n partitioned matrix, and disjoint. (2) Similarly, let A D A2 B1 BD an m p partitioned matrix, where B1 has the same number of rows, B2 say m1 , as A1 . If C.A1 / and C.B1 / are essentially disjoint and if C.A2 / and C.B2 / are essentially disjoint, then C.A/ and C.B/ are essentially disjoint. Proof. (1) Suppose that R.A1 / and R.B1 / are essentially disjoint and that R.A2 / and R.B2 / are essentially disjoint. Let x0 represent an arbitrary (1 n) vector in R.A/ \ R.B/. Then, x0 D r 0 A and x0 D s0 B for some (row) vectors r 0 and s0 . Now, partitioning x0 as x0 D .x01 ; x02 / (where x01 is of dimensions 1 n1 ), we find that .x01 ; x02 / D r 0 .A1 ; A2 / D .r 0 A1 ; r 0 A2 / and, similarly, that .x01 ; x02 / D s0 .B1 ; B2 / D .s0 B1 ; s0 B2 /: Thus, x01 D r 0 A1 and x01 D s0 B1 , implying (in light of Lemma 17.2.2) that x01 D 0; and similarly x02 D r 0 A2 and x02 D s0 B2 , implying that x02 D 0. It follows that x0 D .0; 0/ D 0. We conclude that R.A/ and R.B/ are essentially disjoint. (2) The proof of Part (2) is analogous to that of Part (1). Q.E.D. In the special case where one of the submatrices A1 , A2 , B1 , and B2 is a null matrix, Lemma 17.2.15 can be restated as the following corollary. Corollary 17.2.16. Let T represent an m p matrix, U an m q matrix, V an n p matrix, and W an n q matrix. (1) If R.T / and R.V / are essentially disjoint, then R.T ; U / and R.V ; 0/ are essentially disjoint. Similarly, if R.U / and R.W / are essentially disjoint, then R.T ; U / and R.0; W / are essentially


391

T U and C V 0 are essentially disjoint. Similarly, if C.V / and C.W / are essentially disjoint, then T 0 C and C are essentially disjoint. V W

disjoint. (2) If C.T / and C.U / are essentially disjoint, then C

Let A D .A1 ; A2 / represent an m n partitioned matrix, and B D .B1 ; B2 / a p n partitioned matrix (where B1 has the same number of columns as A1 ). For R.A/ and R.B/ to be essentially disjoint, it is (according to Lemma 17.2.15) sufficient that R.A1 / and R.B1 / be essentially disjoint and that R.A2 / and R.B2 / be essentially disjoint. Are these conditions necessary as well as sufficient? The answer is no, as is easily verified. Results (2.10) and (2.8) are extended in the following theorem. Theorem 17.2.17. Let T represent an m p matrix, U an m q matrix, and V an n p matrix. Then, T U 0 V rank D rank V 0 U T D rank.V / C rankŒU ; T .I V V / V D rank.U / C rank .I U U /T

(2.13) (2.14)

D rank.U / C rank.V / C rankŒ.I U U /T .I V V /: (2.15)

0 Proof. Clearly, is a generalized inverse of .0; V /, and V I 0 0 .0; V / D I : V 0 I V V Thus, making use of result (2.10), we find that 0 V 0 rank D rank.0; V / C rank .U ; T / I .0; V / U T V D rank.V / C rankŒU ; T .I V V /: 0 V It can be established in similar fashion that rank equals expression (2.14). U T T U 0 V That rank D rank is evident from Lemma 8.5.1. And it V 0 U T follows from result (2.8) that rankŒU ; T .I V V / D rank.U / C rankŒ.I U U /T .I V V /: Substitution of this expression for rankŒU ; T .I V V / into expression (2.13) gives expression (2.15). Q.E.D. The following theorem can be regarded as a generalization of Corollary 17.2.10.

392


Theorem 17.2.18. Let T represent an m p matrix, U an m q matrix, and V an n p matrix. Then, rankŒ.I U U /T .I V V / rank.T /;

(2.16)

with equality holding if and only if C.T / and C.U / are essentially disjoint and R.T / and R.V / are essentially disjoint. Proof. It follows from Corollary 4.4.5 that rankŒ.I U U /T .I V V / rankŒT .I V V / rank.T /: Further, in light of Corollary 17.2.10, rankŒ.I U U /T .I V V / D rank.T / , rankŒ.I U U /T .I V V / D rankŒT .I V V / D rank.T / , C.U / and CŒT .I V V / are essentially disjoint and R.V / and R.T / are essentially disjoint. And, according to Corollary 17.2.11, R.V / and R.T / are essentially disjoint , CŒT .I V V / D C.T /; and consequently C.U / and CŒT .I V V / are essentially disjoint and R.V / and R.T / are essentially disjoint , C.U / and C.T / are essentially disjoint and R.V / and R.T / are essentially disjoint. Q.E.D. Upon combining Theorem 17.2.18 with result (2.15), we obtain the following theorem, which extends the results of Theorem 17.2.4. V,

Theorem 17.2.19. For any mp matrix T , mq matrix U , and np matrix T U rank rank.T / C rank.U / C rank.V /; (2.17) V 0

with equality holding if and only if R.T / and R.V / are essentially disjoint and C.T / and C.U / are essentially disjoint.

17.3 Some Results on Linear Systems and on Generalized Inverses of Partitioned Matrices The following theorem gives a basic result on a linear system whose left and right sides are the sums of the left and right sides of two other linear systems.

17.3 Results on Linear Systems and Generalized Inverses of Partitioned Matrices

393

Theorem 17.3.1. Let A1 represent an m n1 matrix, A2 an m n2 matrix, and B1 and B2 m p matrices. Further, let n D n1 C n2 , A D .A1 ; A2 /, and B D B1 C B2 . Suppose that C.A1 / and C.A2 / are essentially disjointand that X1 C.B1 / C.A1 / and C.B2 / C.A2 /. Then, for any solution X D to X2 the linear system AX D B (in an n p matrix X), or, equivalently, to the linear system A1 X1 C A2 X2 D B1 C B2 (in n1 p and n2 p matrices X1 and X2 ), X1 is a solution to the linear system A1 X1 D B1 , and X2 is a solution to the linear system A2 X2 D B2 . Proof. According to Lemma 4.2.2, B1 D A1 F1 and B2 D A2 F2 for some matrices F1 and F2 . Thus, we have that A1 X1 C A2 X2 D B1 C B2 D A1 F1 C A2 F2 ; implying that

A1 .X1 F1 / D A2 .F2 X2 /:

Since C.A1 / and C.A2 / are essentially disjoint, it follows (in light of Corollary 17.2.3) that A1 .X1 F1 / D 0 and A2 .F2 X2 / D 0 and hence that A1 X1 D A1 F1 D B1 and A2 X2 D A2 F2 D B2 . We conclude that X1 is a solution to Q.E.D. A1 X1 D B1 and that X2 is a solution to A2 X2 D B2 . In connection with Theorem 17.3.1, note (in light of Lemmas 4.5.8 and 4.5.5) that the supposition that C.B1 / C.A1 / and C.B2 / C.A2 / implies that C.B/ C.A/ and hence that the linear system AX D B is consistent. Suppose that a linear system A1 X D B1 (in a matrix X of unknowns) is conconsistent. Under sistent and that a second linear system A2 X D B2 (in X)is also A1 B1 what circumstances is the combined linear system XD consistent, A2 B2 or, equivalently, under what circumstances does A2 X D B2 have solutions in common with A1 X D B1 ? This question is addressed in the following theorem. Theorem 17.3.2. Let A1 represent an m1 n matrix, A2 an m2 n matrix, B1 an m1 p matrix, and B2 an m2 p matrix. And suppose that the linear systems A1 X D B1 and A2 X D B2 (in X) are both consistent [or, equivalently, that C.B1 / C.A1 / and C.B2 / C.A2 /]. If R.A1 / and R.A2 / are essentially B1 A1 XD (in X) is consistent. disjoint, then the combined linear system A2 B2 Proof. Suppose that R.A1 / and R.A2 /are essentially disjoint. Let k represent A any .m1 C m2 / 1 vector such that k0 1 D 00 . Then, partitioning k as A2 k1 (where k1 has m1 elements), k01 A1 C k02 A2 D 00 , or, equivalently, kD k2 k01 A1 D k02 A2 . Thus, k01 A1 D 00 , and k02 A2 D 00 .

394


Moreover, each of the linear systems A1 X D B1 and A2 X D B2 is (in light of Theorem 7.3.1) compatible, so that k01 B1 D 00 and k02 B2 D 00 and hence B1 A1 0 B1 0 D 0 . We conclude that the combined linear system XD is k B2 A2 B2 compatible and hence (in light of Theorem 7.3.1) consistent. Q.E.D. and G of a generalized inverse Under what condition(s) are the blocks G 1 2 G1 of a partitioned matrix .A; B/ generalized inverses of the blocks (A and B) G2 of .A; B/? And, similarly, under what condition(s) aretheblocks H1 and H2 of A a generalized inverse .H1 ; H2 / of a partitioned matrix generalized inverses C A of the blocks of ? These questions are answered in the following theorem. C Theorem 17.3.3. Let A represent an m n matrix, B an m p matrix, and G1 C a q n matrix. (1) For any generalized inverse G D (where G1 has n G2 rows) of the m .n C p/ partitioned matrix .A; B/, G1 is a generalized inverse of A, and G2 a generalized inverse of B, if and only if C.A/ and C.B/ are essentially disjoint. (2) Similarly, for any generalized inverse HD .H1 ; H2 / (where H1 A has m columns) of the .m C q/ n partitioned matrix , H1 is a generalized C inverse of A, and H2 a generalized inverse of C, if and only if R.A/ and R.C/ are essentially disjoint. Proof. (1) Clearly, .AG1 A C BG2 A; AG1 B C BG2 B/ D .A; B/G.A; B/ D .A; B/; so that BG2 A D A AG1 A;

AG1 B D B BG2 B:

(3.1)

Now, suppose that C.A/ and C.B/ are essentially disjoint. Then, rewriting equalities (3.1) as BG2 A D A.I G1 A/;

AG1 B D B.I G2 B/

and applying Corollary 17.2.3, we find that A.I G1 A/ D 0;

B.I G2 B/ D 0;

or, equivalently, that AG1 A D A and BG2 B D B. Thus, G1 is a generalized inverse of A and G2 a generalized inverse of B. Conversely, suppose that G1 is a generalized inverse of A and G2 a generalized inverse of B. Then, it follows from result (3.1) that BG2 A D 0, and we conclude (on the basis of Corollary 17.2.12) that C.A/ and C.B/ are essentially disjoint. (2) The proof of Part (2) is analogous to that of Part (1). Q.E.D.

17.3 Results on Linear Systems and Generalized Inverses of Partitioned Matrices

395

The following theorem gives sufficient for the blocks G11 , G21 , conditions G11 G12 G12 , and G22 of a generalized inverse of a partitioned matrix A D G21 G22 T U to be generalized inverses of the blocks (T , U , V , and W ) of A. V W Theorem 17.3.4. Let T represent an m p matrix, U an m q matrix, V G11 G12 an n p matrix, and W an n q matrix. And let (where G11 is G21 G22 of p m) represent a generalized inverse of the partitioned matrix dimensions T U . V W (1) If C.T / and C.U / are essentially disjoint and R.T / and R.V / are essentially disjoint, then G11 is a generalized inverse of T ; (2) if C.U / and C.T / are essentially disjoint and R.U / and R.W / are essentially disjoint, then G21 is a generalized inverse of U ; (3) if C.V / and C.W / are essentially disjoint and R.V / and R.T / are essentially disjoint, then G12 is a generalized inverse of V ; and (4) if C.W / and C.V / are essentially disjoint and R.W / and R.U / are essentially disjoint, then G22 is a generalized inverse of W . Proof. Clearly, 0

T G11 T C U G21 T B CT G12 V C U G22 V B @V G T C W G T 11

21

1 T G11 U C U G21 U CT G12 W C U G22 W C C A VG U C WG U 11

21

CV G12 V C W G22 V CV G12 W C W G22 W T U G11 G12 T U T U D D : V W V W V W G21 G22

(3.2)

(1) It follows from result (3.2) that T G11 T C U G21 T C T G12 V C U G22 V D T :

(3.3)

Now, suppose that C.T / and C.U / are essentially disjoint and R.T / and R.V / are essentially disjoint. Then, rewriting equality (3.3) as U .G21 T C G22 V / D T .I G11 T G12 V / and applying Corollary 17.2.3, we find that T .I G11 T G12 V / D 0: Further, rewriting equality (3.4) as .I T G11 /T D T G12 V

(3.4)

396


and again applying Corollary 17.2.3, we find that .I T G11 /T D 0 or, equivalently, that T D T G11 T . Thus, G11 is a generalized inverse of T . (2), (3), and (4). The proofs of Parts (2), (3), and (4) are analogous to the proof of Part (1). Q.E.D. Note that the conditions of Parts (1) and (4) of Theorem 17.3.4 [as well as those of Parts (2) and (3)] are satisfied in particular if U D 0 and V D 0—if U D 0 and T U V D 0, reduces to the block-diagonal matrix diag.T ; W /. V W

17.4 Subspaces: Sum of Their Dimensions Versus Dimension of Their Sum The following theorem quantifies the difference between the right and left sides of inequality (1.12). Theorem 17.4.1. For any subspaces U and V of Rmn, dim.U C V/ D dim.U/ C dim.V/ dim.U \ V/;

(4.1)

or equivalently dim.U \ V/ D dim.U/ C dim.V/ dim.U C V/:

(4.2)

Proof. Let r D dim.U/, s D dim.V/, and t D dim.U \ V/. If t D 0, U and V are essentially disjoint, and result (4.1) follows from result (1.11). Consider now the case where t > 0. Let fF1 ; : : : ; F t g represent a basis for the subspace U \ V. Since U\V U, F1 ; : : : ; F t are contained in U, and it follows from Theorem 4.3.12 that there exist r t additional matrices A1 ; : : : ; Art such that the set fF1 ; : : : ; F t ; A1 ; : : : ; Art g is a basis for U—if r D t , the set fF1 ; : : : ; F t ; A1 ; : : : ; Art g reduces to the set fF1 ; : : : ; F t g. Similarly, F1 ; : : : ; F t are contained in V, and there exist s t additional matrices B1 ; : : : ; Bst such that the set fF1 ; : : : ; F t ; B1 ; : : : ; Bst g is a basis for V. To validate result (4.1), it suffices to establish that the set fF1 ; : : : ; F t , A1 , : : : ; Art , B1 ; : : : ; Bst g spans UCV and is linearly independent [thereby establishing that this (r C s t )-dimensional set is a basis for U C V]. Clearly, if X 2 U and Y 2 V, then each of the matrices X and Y is expressible as a linear combination of the matrices F1 ; : : : ; F t ; A1 ; : : : ; Art ; B1 ; : : : ; Bst and consequently X C Y is expressible as a linear combination of these matrices. Thus, the set fF1 , : : : ; F t , A1 , : : : ; Art , B1 , : : : ; Bst g spans U C V. To complete the proof, we must show that this set is linearly independent. For this purpose, suppose that s > t —if s D t , the set fF1 ; : : : ; F t ; A1 ; : : : ; Art , B1 , : : : ; Bst g reduces to the set fF1 ; : : : ; F t ; A1 ; : : : ; Art g, which is a basis for U and is therefore linearly independent. Let k1 ; : : : ; k t ; c1 ; : : : ; crt ; d1 ; : : : ; dst represent any scalars such that

17.4 Subspaces: Sum of Their Dimensions Versus Dimension of Their Sum

397

k1 F1 C C k t F t C c1 A1 C C crt Art C d1 B1 C C dst Bst D 0: (4.3) And define H D k1 F1 C C k t F t C c1 A1 C C crt Art :

(4.4)

H D d1 B1 dst Bst :

(4.5)

Then, Equality (4.4) implies that H 2 U, and equality (4.5) implies that H 2 V. Thus, H 2 U\V, and, consequently, there exist scalars z1 ; : : : ; z t such that H D z1 F 1 C C z t F t :

(4.6)

Together, equalities (4.5) and (4.6) imply that z1 F1 C C z t F t C d1 B1 C C dst Bst D H H D 0: Since fF1 ; : : : ; F t ; B1 ; : : : ; Bst g is a basis for V and hence is linearly independent, it follows that (4.7) d1 D D dst D 0 (and that z1 D D z t D 0) and [in light of equality (4.3)] that k1 F1 C C k t F t C c1 A1 C C crt Art D 0:

(4.8)

And since fF1 ; : : : ; F t ; A1 ; : : : ; Art g is a basis for U and hence is linearly independent, result (4.8) implies that k1 D D k t D c1 D D crt D 0:

(4.9)

We conclude [on the basis of results (4.7) and (4.9)] that the set fF1 , : : : ; F t , Q.E.D. A1 , : : : ; Art , B1 ; : : : ; Bst g is linearly independent. By applying Theorem 17.4.1 in the special case where U and V are the column spaces of an m n matrix A and an m p matrix B and in the special case where U and V are the row spaces of an m n matrix A and a q n matrix C, we obtain [in light of equalities (1.4) and (1.5)] the following theorem, which quantifies the difference between the right and left sides of inequality (2.1) and the difference between the right and left sides of inequality (2.2). Theorem 17.4.2. Let A represent an m n matrix, B an m p matrix, and C a q n matrix. Then, rank.A; B/ D rank.A/ C rank.B/ dimŒC.A/ \ C.B/; A rank D rank.A/ C rank.C/ dimŒR.A/ \ R.C/: C

(4.10) (4.11)

Or, equivalently, dimŒC.A/ \ C.B/ D rank.A/ C rank.B/ rank.A; B/; A dimŒR.A/ \ R.C/ D rank.A/ C rank.C/ rank : C

(4.12) (4.13)

398


Upon combining result (4.10) with result (2.8) and result (4.11) with result (2.10), we obtain the following corollary of Theorem 17.4.2. Corollary 17.4.3. Let A represent an m n matrix, B an m p matrix, and C a q n matrix. Then, rankŒ.I AA /B D rank.B/ dimŒC.A/ \ C.B/;

(4.14)

rankŒC.I AA/ D rank.C/ dimŒR.A/ \ R.C/:

(4.15)

Result (4.15) can be restated in terms of the following set: U D fx 2 Rq1 W x D Cz for some n 1 vector z in N.A/g: We have that

U D CŒC.I AA/;

(4.16)

as is evident upon observing (in light of Corollary 11.2.2) that x 2 CŒC.I AA/ , x D C.I AA/y for some n 1 vector y , x D Cz for some z in C.I AA/ , x D Cz for some z in N.A/: Thus, U is a linear space, and, as a consequence of result (4.15), dim.U/ D rankŒC.I AA/ D rank.C/ dimŒR.A/ \ R.C/:

(4.17)

17.5 Some Results on the Rank of a Product of Matrices a. Initial presentation of results Upper bounds for the product of an m n matrix A and an n p matrix B are, as previously indicated (in Corollary 4.4.5): rank.AB/ rank.A/;

rank.AB/ rank.B/:

(5.1)

Note that inequalities (5.1) can be combined and restated as rank.AB/ minŒrank.A/; rank.B/:

(5.2)

For purposes of obtaining a lower bound for the rank of a product of matrices, let A represent an mn matrix, C an nq matrix, and B a q p matrix. According to Theorem 9.6.4, C CB rank D rank.C/ C rank.ACB/ D rank.C/ C rank.ACB/; (5.3) AC 0 and, according to Theorem 17.2.17,

17.5 Some Results on the Rank of a Product of Matrices

399

C CB D rank.AC/ C rank.CB/ AC 0 C rankfŒI CB.CB/ CŒI .AC/ACg:

(5.4)

rank

Equating expression (5.4) to expression (5.3), we obtain the following result on the rank of a product of three matrices. Theorem 17.5.1. Let A represent an m n matrix, C an n q matrix, and B a q p matrix. Then, rank.ACB/ D rank.AC/ C rank.CB/ rank.C/ C rankfŒI CB.CB/ CŒI .AC/ACg:

(5.5)

Further, rank.ACB/ rank.AC/ C rank.CB/ rank.C/;

(5.6)

with equality holding if and only if ŒI CB.CB/ CŒI .AC/AC D 0:

(5.7)

In the special case where C D In , Theorem 17.5.1 reduces to the following corollary on the rank of a product of two matrices. Corollary 17.5.2. Let A represent an m n matrix and B an n p matrix. Then, rank.AB/ D rank.A/ C rank.B/ n C rankŒ.I BB /.I A A/:

(5.8)

Further, rank.AB/ rank.A/ C rank.B/ n;

(5.9)

with equality holding if and only if .I BB /.I A A/ D 0:

(5.10)

Inequality (5.9) is called Sylvester’s law of nullity, after James Joseph Sylvester (1814–1897), and inequality (5.6) is called the Frobenius inequality, after Ferdinand Georg Frobenius (1849–1917). According to Corollary 17.5.2, the lower bound (5.9) (on the rank of a product of two matrices) is attained if and only if equality (5.10) holds. Under what condition(s) are the upper bounds [given by result (5.1)] attained? To answer this question, observe (in light of Corollary 17.2.9) that rank.B; I A A/ D rank.B/ C rankŒ.I BB /.I A A/; A D rank.A/ C rankŒ.I BB /.I A A/: rank I BB

(5.11) (5.12)

A comparison of expressions (5.11) and (5.12) with expression (5.8) leads to the following theorem.

400


Theorem 17.5.3. Let A represent an m n matrix and B an n p matrix. Then, rank.AB/ D rank.A/ n C rank.B; I A A/ A : D rank.B/ n C rank I BB

(5.13) (5.14)

Further, equality is attained in the inequality rank.AB/ rank.A/ if and only if rank.B; I A A/ D n or, equivalently, if and only if .B; I A A/ is of full row rank.And equality if and only if is attained in the inequality rank.AB/ rank.B/ A A D n or, equivalently, if and only if is of full column rank I BB I BB rank. b. Reexpression of results in more “geometrically meaningful” terms Let A represent an m n matrix and B an n p matrix. Then, making use of Theorem 17.4.2, Lemma 10.2.6, and Corollary 11.2.2, we find that rank.B; I A A/ D rank.B/ C rank.I A A/ dimŒC.B/ \ C.I A A/ D rank.B/ C n rank.A/ dimŒC.B/ \ N.A/:

(5.15)

Further, recalling that .B /0 is a generalized inverse of B0 , we find in similar fashion that 0 A A D rank rank I BB I BB D rankŒA0 ; I .B /0 B0 D rank.A/ C rankŒI .B /0 B0 dimfC.A0 / \ CŒI .B /0 B0 g D rank.A/ C n rank.B/ dimŒC.A0 / \ N.B0 /:

(5.16)

Substituting expressions (5.15) and (5.16) into expressions (5.13) and (5.14), respectively, we obtain the following alternative to Theorem 17.5.3. Theorem 17.5.4. Let A represent an m n matrix and B an n p matrix. Then, rank.AB/ D rank.B/ dimŒC.B/ \ N.A/ D rank.A/ dimŒC.A0 / \ N.B0 /:

(5.17) (5.18)

Further, equality is attained in the inequality rank.AB/ rank.B/ if and only if C.B/ and N.A/ are essentially disjoint. And equality is attained in the inequality rank.AB/ rank.A/ if and only if C.A0 / and N.B0 / are essentially disjoint.

17.5 Some Results on the Rank of a Product of Matrices

401

By equating expression (5.8) for the rank of a product of two matrices to each of the alternative expressions (5.17) and (5.18), we find (in light of Lemma 11.3.1) that (for any m n matrix A and any n p matrix B) rankŒ.I BB /.I A A/ D n rank.A/ dimŒC.B/ \ N.A/ D dimŒN.A/ dimŒC.B/ \ N.A/

(5.19)

and similarly that rankŒ.I BB /.I A A/ D n rank.B0 / dimŒC.A0 / \ N.B0 / D dimŒN.B0 / dimŒC.A0 / \ N.B0 /:

(5.20)

Based on equalities (5.19) and (5.20), we obtain the following theorem, which allows us to reexpress condition (5.10) [under which equality holds in inequality (5.9)] in more “geometrically meaningful” terms. Theorem 17.5.5. Let A represent an m n matrix and B an n p matrix. Then, .I BB /.I A A/ D 0 , N.A/ C.B/ , N.B0 / C.A0 /: Let A represent an m n matrix, C an n q matrix, and B a q p matrix. Then, making use of result (5.17), we find that rank.ACB/ D rankŒA.CB/ D rank.CB/ dimŒC.CB/ \ N.A/

(5.21)

and, similarly, making use of result (5.18), that rank.ACB/ D rankŒ.AC/B D rank.AC/ dimfCŒ.AC/0 \ N.B0 /g: (5.22) Further, equating expression (5.5) [for rank.ACB/] to expression (5.21) and making use of result (5.17), we find that rankfŒI CB.CB/ CŒI .AC/ACg D rank.C/ rank.AC/ dimŒC.CB/ \ N.A/ D dimŒC.C/ \ N.A/ dimŒC.CB/ \ N.A/

(5.23)

and, similarly, equating expression (5.5) to expression (5.22) and making use of result (5.18), that rankfŒI CB.CB/ CŒI .AC/ACg D rank.C/ rank.CB/ dimŒC.C 0 A0 / \ N.B0 / D dimŒC.C 0 / \ N.B0 / dimŒC.C 0 A0 / \ N.B0 /: (5.24) Based on equalities (5.23) and (5.24), we obtain the following theorem, which allows us to reexpress condition (5.7) [under which equality holds in the Frobenius inequality] in more geometrically meaningful terms.

402


Theorem 17.5.6. Let A represent an m n matrix, C an n q matrix, and B a q p matrix. Then, ŒI CB.CB/ CŒI .AC/AC D 0 , C.CB/ \ N.A/ D C.C/ \ N.A/ , C.C 0 A0 / \ N.B0 / D C.C 0 / \ N.B0 /:

17.6 Projections Along a Subspace a. Some general results and terminology Let V represent a linear space of m n matrices, and let U represent a subspace of V. The subspace U and its orthogonal complement U? (which is also a subspace of V) are orthogonal (as is evident from the very definition of U? ) and hence are (as a consequence of Lemma 17.1.9) essentially disjoint. Moreover, it follows from Corollary 17.1.2 that UCU? V and from Theorem 12.5.11 that V UCU?. Thus, (6.1) V D U ˚ U? (i.e., V equals the direct sum of U and U? ). Let X represent an mp matrix, and take W to be an mm symmetric positive definite matrix. Then, as a special case of result (6.1) [that where V D Rm and U D C.X/ and where the inner product for Rm is taken to be the bilinear form whose matrix is W ], we have that ? Rm D C.X/ ˚ CW .X/;

(6.2)

which, in the further special case where W D I, reduces to Rm D C.X/ ˚ C? .X/:

(6.3)

According to Theorem 12.1.3, a subspace U (of a linear space V of m n matrices) and its orthogonal complement U? have the following property: corresponding to any mn matrix Y in V, there exists a unique matrix Z in U such that Y Z 2 U?. The following theorem indicates that any two essentially disjoint subspaces whose sum is V have this property. Theorem 17.6.1. Let Y represent a matrix in a linear space V of m n matrices, and let U and W represent essentially disjoint subspaces whose sum is V. Then, U contains a unique matrix Z such that Y Z 2 W. Proof. Since V D U C W, it follows from the very definition of a sum (of subspaces) that U contains a matrix Z and W a matrix W such that Y D Z C W . Clearly, Y Z D W , and hence Y Z 2 W. For purposes of establishing the uniqueness of Z, let Z represent a matrix (potentially different from Z) in U such that Y Z 2 W. Then, Z Z 2 U and Y Z .Y Z/ 2 W. And

17.6 Projections Along a Subspace

403

Z Z C ŒY Z .Y Z/ D 0: We conclude, on the basis of Part (1) of Lemma 17.1.4, that Z Z D 0 [and that Y Z .Y Z/ D 0] and hence that Z D Z, thereby establishing the uniqueness of Z. Q.E.D. Theorem 17.6.1 can be restated in the form of the following corollary. Corollary 17.6.2. Let Y represent a matrix in a linear space V of m n matrices, and let U and W represent essentially disjoint subspaces whose sum is V. Then, U contains a unique matrix Z and W a unique matrix W such that Y D ZCW. Suppose that a linear space V is the direct sum U ˚ W of two subspaces U and W (as in Theorem 17.6.1). Then, as indicated by the theorem, there exists a unique matrix Z in U such that the difference between Y and Z is in W. This matrix (the matrix Z) is referred to as the projection of Y on U along W. The following theorem, which is easy to verify, relates the projection of Y on W along U to the projection of Y on U along W. Theorem 17.6.3. Let Y represent a matrix in a linear space V, and let U and W represent essentially disjoint subspaces whose sum is V. Then, the projection of Y on W along U equals Y Z, where Z is the projection of Y on U along W. The following theorem gives an explicit expression for the projection of a matrix Y (in a linear space V) on a subspace U along a subspace W (where V D U ˚ W) and also for the projection of Y on W along U. Theorem 17.6.4. Let Y represent a matrix in a linear space V of m n matrices, and let U and W represent essentially disjoint subspaces whose sum is V. Further, let fU1 ; : : : ; Us g represent any set of matrices that spans U, let fW1 ; : : : ; W t g represent any set of matrices that spans W, and define c1 ; : : : ; cs and k1 ; : : : ; k t to be any scalars such that c1 U1 C C cs Us C k1W1 C C k tW t D Y :

(6.4)

Then, the projection of Y on U along W equals c1 U1 C C cs Us , and the projection of Y on W along U equals k1W1 C C k tW t . In connection with Theorem 17.6.4, note that the existence of scalars c1 ; : : : ; cs and k1 ; : : : ; k t that satisfy condition (6.4)Pis guaranteed by Lemma 17.1.1. Proof (of Theorem 17.6.4). Clearly, siD1 ci Ui 2 U, and Y

s X iD1

Ps

ci U i D

t X

kj Wj 2 W:

j D1

is (by definition) the projection of Y on U along W, and is (in light of Theorem 17.6.3) the projection of Y on W along U. Q.E.D. In the special case where n D 1 (i.e., where V is a linear space of column vectors), Theorem 17.6.4 can be restated as the following corollary. Thus, P t

iD1 ci Ui

j D1 kj Wj

404


Corollary 17.6.5. Let y represent a vector in a linear space V of m-dimensional column vectors, and let U and W represent essentially disjoint subspaces whose sum is V. Further, let U D .u1 ; : : : ; us / and W D .w1 ; : : : ; w t /, where u1 ; : : : ; us are any vectors that span U and w1 ; : : : ; w t any vectors that span W, or, equivalently, let U represent any m s matrix such that C.U / D U and W any m t matrix such that C.W / D W. And define x to be any solution to the linear system .U ; W /x D y [in an (s C t )-dimensional column vector x], and x 1 (where x1 has s elements). Then, the projection of y partition x as x D x2 on U along W equals U x1 , and the projection of y on W along U equals W x2 . The following theorem, whose validity is apparent from result (6.1) and Theorems 12.5.8 and 17.6.3, establishes a connection to the results of Chapter 12 (on orthogonal projections). Theorem 17.6.6. Let Y represent a matrix in a linear space V, and let U represent a subspace of V. Then, the projection of Y on U along U? equals the orthogonal projection of Y on U. And the projection of Y on U? along U equals the orthogonal projection of Y on U? . The orthogonal projection of an m n matrix Y on a subspace U (of a linear space V of m n matrices) equals Y if and only if Y 2 U—refer to the last part of Theorem 12.1.3. An extension of this result is given in the following lemma. Lemma 17.6.7. Let Y represent a matrix in a linear space V, let U and W represent essentially disjoint subspaces whose sum is V, and let Z represent the projection of Y on U along W. Then, Z D Y if and only if Y 2 U. (And Z D 0 if and only if Y 2 W.) Proof. If Z D Y , then (since, by definition, Z 2 U), Y 2 U. Conversely, if Y 2 U, then, since Y Y D 0 2 W, Z D Y . (If Z D 0, then Y D Y Z 2 W; conversely, if Y 2 W, then clearly Z D 0.) Q.E.D. The following theorem can be regarded as an extension of Theorem 12.1.4. Theorem 17.6.8. Let Y1 ; : : : ; Yp represent matrices in a linear space V, let U and W represent essentially disjoint subspaces whose sum is V, and let Z1 ; : : : ; Zp represent the projections of Y1 ; : : : ; Yp , respectively, on U along W. Then, for any scalars k1 ; : : : ; kp , the projection of the linear combination k1Y1 C C kpYp on U along W is the corresponding linear combination k1 Z1 C C kp Zp of Z1 ; : : : ; Zp . Proof. By definition, Zi 2 U and Yi Zi 2 W (i D 1; : : : ; p). Thus, k1 Z1 C C kp Zp 2 U, and k1Y1 C C kpYp .k1 Z1 C C kp Zp / D k1 .Y1 Z1 / C C kp .Yp Zp / 2 W: We conclude that k1 Z1 C C kp Zp is the projection of k1Y1 C C kpYp on U along W. Q.E.D.


405

b. Two-dimensional example Let us find the projection of an m-dimensional column vector y on a subspace U of Rm along a subspace W of Rm in the special case where m D 2, y D .4; 8/0 , U D spfug, and W D spfwg, with u D .3; 1/0 and w D .1; 4/0. (Clearly, R2 D U ˚ W.) It follows from Corollary 17.6.5 that the projection of y on U along W equals x1 u and the projection of y on W along U equals x2 w, where x D .x1; x2 /0 is any solution to the linear system 3 1 x1 4 .u; w/x D y , D (6.5) 1 4 x2 8 [in the 2 1 vector x D .x1 ; x2 /0 ]. The (unique) solution to linear system (6.5) is x D .8=11; 20=11/0 , as is easily verified. Thus, the projection of y on U along W is the vector 3 24=11 .8=11/ D ; 1 8=11 and the projection of y on W along U is the vector 1 20=11 .20=11/ D : 4 80=11 Both of these vectors are displayed in Figure 17.1. It is informative to compare Figure 17.1 with Figures 12.1 and 12.3; the latter figures depict the orthogonal projection of y on U (which is identical to the projection of y on U along U? ) and the orthogonal projection of y on U? (which is identical to the projection of y on U? along U). c. Three-dimensional example Let us find the projection of an m-dimensional column vector y on a subspace U of Rm along a subspace W of Rm in the special case where m D 3, y D .3; 38=5; 74=5/0 , U D spfu1 ; u2 ; u3 g, and W D spfwg, with 0 1 0 1 0 1 0 1 1 0 2 2 u1 D @3A; u2 D @ 2A; u3 D @ 1A; w D @3A: 6 6 4 2 {It is easy to show that u3 D u2 .1=3/u1 and that u1 , u2 , and w are linearly independent, implying (in light of Lemma 17.1.1) that dim.U/ C dim.W/ D 2 C 1 D 3 D dimŒsp.u1 ; u2 ; u3 ; w/ D dim.U C W/ and hence (in light of Corollaries 17.1.6 and 17.1.8) that R3 D U ˚ W.} Let U D .u1 ; u2 ; u3 /, and take x to be any solution to the linear system

406


FIGURE 17.1: The projection of a two-dimensional column vector y on a one-dimensional subspace U of R2 along a one-dimensional subspace W of R2.

0

1 0 1 0 2 2 1 3 2 1 3A x D @38=5A .U ; w/x D y , @3 6 4 2 6 74=5

(6.6)

(in the 4 1 vector x). Then, partitioning x0 as x0 D .x0 1 ; x2 / (where x1 has 3 elements), it follows from Corollary 17.6.5 that the projection of y on U along W equals U x1 and that the projection of y on W along U equals x2 w. It is easy to verify that one solution to linear system (6.6) is x D .9=5; 11=4; 0; 5=2/0 . Thus, the projection of y on U along W is the vector 0 10 1 0 1 0 2 2 9=5 11=2 @3 2 1A@11=4A D @1=10A; 6 4 2 0 1=5

and the projection of y on W along U is the vector 0 1 0 1 1 5=2 .5=2/ @3A D @15=2A: 6 15 Both of these vectors are displayed in Figure 17.2. It is informative to compare Figure 17.2 with Figures 12.2 and 12.4; the latter figures depict the orthogonal projection of y on U (which is identical to the pro-


407

FIGURE 17.2: The projection of a three-dimensional column vector y on a two-dimensional subspace U of R3 along a one-dimensional subspace W of R3.

jection of y on U along U? ) and the orthogonal projection of y on U? (which is identical to the projection of y on U? along U). d. Projection matrices The concept of a projection matrix was introduced in Section 12.3 and was extended in Section 14.12d. The following theorem provides a basis for a further extension of this concept. Theorem 17.6.9. Let U and V represent essentially disjoint subspaces (of Rn1 ) whose sum is Rn1. Then, there exists a unique matrix A (of dimensions n n) such that Ay is the projection of y on U along V for every y in Rn . Moreover, A D .z1 ; : : : ; zn /, where, letting ej represent the j th column of In , zj is the projection of ej on U along V (j D 1; : : : ; n). Proof. Define Z D .z1 ; : : : ; zn /. Then, P for every n-dimensional column vector y D .y1 ; : : : ; yn /0 , we have that y D jnD1 yj ej , implying (in light of Theorem 17.6.8) that the projection of y on U along V equals

408

17 Intersections and Sums of Subspaces n X

yj zj D Zy:

j D1

Moreover, for any matrix A such that Ay is the projection of y on U along V for every y in Rn, we have that Ay D Zy for every y, implying (in light of Lemma 2.3.2) that A D Z. We conclude that there exists a unique matrix A such that Ay is the projection of y on U along V for every y 2 Rn and that A D Z. Q.E.D. Suppose that U and V are essentially disjoint subspaces (of Rn1 ) whose sum is Rn1. Then, according to Theorem 17.6.9, there exists a unique matrix A (of dimensions n n) such that Ay is the projection of y on U along V for every y 2 Rn. This matrix is called the projection matrix for U along V. As an almost immediate consequence of Theorem 17.6.3, we have the following result. Theorem 17.6.10. Let U and V represent essentially disjoint subspaces (of Rn1 ) whose sum is Rn1. Then, the projection matrix for V along U equals I A, where A is the projection matrix for U along V. In light of Theorems 17.6.6, 17.6.10, 12.3.1, and 14.12.8, we have the following theorem, which establishes a connection to the results of Chapter 12 and Section 14.12. Theorem 17.6.11. Let X represent an np matrix. Then, the projection matrix for C.X/ along C? .X/ equals PX , which is the orthogonal projection matrix for C.X/, and the projection matrix for C? .X/ along C.X/ equals I PX . More generally, for any n n symmetric positive definite matrix W , the projection ? .X/ equals PX;W , which is the orthogonal projection matrix for C.X/ along CW ? .X/ along matrix for C.X/ with respect to W, and the projection matrix for CW C.X/ equals I PX;W . The following theorem extends the results of Theorems 12.3.8 and 14.12.15. Theorem 17.6.12. If an n n matrix A is the projection matrix for a subspace U of Rn1 along a subspace V of Rn1 (where U˚V D Rn1 ), then U D C.A/ and V D C.I A/. Proof. Suppose that A is the projection matrix for U along V. Then, for any y 2 Rn, Ay is the projection of y on U along V and hence is in U. Thus, C.A/ U. Moreover, as a consequence of Lemma 17.6.7, we have that, for any y 2 U, y D Ay 2 C.A/. It follows that U C.A/ and hence [since C.A/ U] that U D C.A/. Further, since (in light of Theorem 17.6.10) I A is the projection matrix for V along U, it follows from a similar argument that V D C.I A/. Q.E.D. The following theorem expands on the results of Theorem 12.3.9 and Corollary 14.12.17. Theorem 17.6.13. An n n matrix A is the projection matrix for some subspace U of Rn1 along some subspace V of Rn1 (where U ˚ V D Rn1 ) if and only if A is idempotent.

17.7 Some Further Results

409

Proof. Suppose that A is idempotent. Then, according to Lemma 17.2.6, C.A/ and C.I A/ are essentially disjoint. And every y in Rn is expressible as y D Ay C .I A/y. Thus, C.A/ ˚ C.I A/ D Rn. Further, for every y in Rn, y Ay D .I A/y, implying [since Ay 2 C.A/ and .I A/y 2 C.I A/] that Ay is the projection of y on C.A/ along C.I A/. It follows that A is the projection matrix for C.A/ along C.I A/. Conversely, suppose that A is the projection matrix for some subspace U along some subspace V (where U˚V D Rn1 ). Then, it follows from Theorem 17.6.12 that C.A/ D U and C.IA/ D V, implying that C.A/ and C.IA/ are essentially disjoint and hence (in light of Lemma 17.2.6) that A is idempotent. Q.E.D. The results of Theorems 17.6.12 and 17.6.13 can be combined and (in light of Theorem 11.7.1) restated as the following theorem. Theorem 17.6.14. If an n n matrix A is the projection matrix for a subspace U of Rn1 along a subspace V of Rn1 (where U ˚ V D Rn1 ), then A is idempotent, U D C.A/, and V D C.I A/ D N.A/. And, if an n n matrix A is idempotent, then C.A/˚C.IA/ D Rn [or, equivalently, C.A/˚N.A/ D Rn ], and A is the projection matrix for C.A/ along C.I A/ [or, equivalently, along N.A/].

17.7 Some Further Results on the Essential Disjointness and Orthogonality of Subspaces and on Projections and Projection Matrices Let U and W represent two subspaces of Rn1. Recall (from Section 17.1) that if U and W are orthogonal, then they are essentially disjoint. Recall also that U and W can be essentially disjoint without being orthogonal with respect to the usual inner product (or with respect to whatever other inner product may have been adopted). Are essentially disjoint subspaces of Rn1 necessarily orthogonal with respect to some inner product? This question is answered (in the affirmative) by the following theorem. Theorem 17.7.1. Let U and W represent essentially disjoint subspaces of Rn1. Further, letting s D dim.U/ and t D dim.W/ (and supposing that s > 0 and t > 0), take U to be any n s matrix such that C.U / D U, W to be any n t matrix such that C.W / D W, and Z to be any n .n s t / matrix such that the 1 n n partitioned matrix 1 ; W ; Z/ is nonsingular. Define B D .U ; W ; Z/ , 0 .U B1 partition B as B D @B2 A (where B1 has s rows and B2 has t rows), and let B3 H D B0 B or (more generally) let H represent any matrix of the form H D B10 A1 B1 C B20 A2 B2 C B30 A3 B3 ; where A1 ; A2 , and A3 are symmetric positive definite matrices. Then, U and W are orthogonal with respect to H.

410


In connection with Theorem 17.7.1, observe (in light of Theorem 17.2.4) that rank.U ; W / D s C t [or, equivalently, that the columns of .U ; W / are linearly independent], insuring (in light of Theorem 4.3.12) the existence of n s t n-dimensional column vectors (the columns of Z) that, together with the columns of .U ; W /, form a basis for Rn (and hence form a set of n linearly independent vectors). Observe also that H D B0 diag.A1 ; A2 ; A3 /B and that (according to Lemma 14.8.3) diag.A1 ; A2 ; A3 / is positive definite, insuring (in light of Corollary 14.2.10) that H is positive definite. Further, in the degenerate special case of Theorem 17.7.1 where s C t D n (or, equivalently, where U ˚ W D Rn1 ), 0 1 B1 B1 interpret .U ; W ; Z/ as .U ; W /, @B2 A as , and B10 A1 B1 C B20 A2 B2 C B2 B3 B30 A3 B3 as B10 A1 B1 C B20 A2 B2 . Proof (of Theorem 17.7.1). In light of Lemma 14.12.1, it suffices to show that U 0 HW D 0. By definition, 0 0 1 1 Is 0 B1 U B 1 W B 1 Z 0 @B2 U B2 W B2 ZA D B.U ; W ; Z/ D In D @ 0 I t 0 A; B3 U B3 W B3 Z 0 0 Inst implying in particular that B1 W D 0, B2 U D 0, and B3 W D 0. Thus, U 0 HW D U 0 B10 A1 B1 W C .B2 U /0 A2 B2 W C U 0 B30 A3 B3 W D 0 C 0 C 0 D 0: Q.E.D. In discussing the implications of Theorem 17.7.1, it will be convenient to have at our disposal the following lemma. Lemma 17.7.2. Let U and W represent subspaces of a linear space V of m n matrices. If U ? W and if U C W D V, then W D U? and U D W ?. Proof. Suppose that U ? W and that U C W D V. Then, clearly, W U?, and U W ?. Moreover, V and W are (in light of Lemma 17.1.9) essentially disjoint, implying (in light of Corollary 17.1.6) that dim.V/ D dim.U/ C dim.W/: Thus, making use of Theorem 12.5.12, we find that dim.W/ D dim.V/ dim.U/ D dim.U? / and, similarly, that dim.U/ D dim.V/ dim.W/ D dim.W ? /: We conclude (on the basis of Theorem 4.3.10) that W D U? and that U D Q.E.D. W?. Let V represent a linear space of n 1 vectors, and let U and W represent essentially disjoint subspaces whose sum is V. Then, according to Theorem 17.7.1,

Exercises

411

U and V are orthogonal with respect to some symmetric positive definite matrix H, implying (in light of Lemma 17.7.2) that W equals the orthogonal complement U? of U (where the orthogonality in the orthogonal complement is with respect to the bilinear form x0 Hy). Thus, it follows from Theorem 17.6.6 that, for any vector y in V, the projection of y on U along W equals the orthogonal projection of y on U with respect to H. Further, letting X represent any matrix such that C.X/ D U, it follows from Theorem 17.6.11 that, in the special case where V D Rn1 (and hence where U ˚ W D Rn1 ), the projection matrix for U along W equals the orthogonal projection matrix PX;H for U with respect to H.

Exercises Section 17.1 1. Let U and V represent subspaces of Rmn. (a) Show that U[V UCV. (b) Show that UCV is the smallest subspace (of Rmn ) that contains U[V, or, equivalently [in light of Part (a)], show that, for any subspace W such that U[V W, UCV W. 0 1 0 1 1 0 0 2 2. Let A D @0 1A and B D @1 1A. Find (a) a basis for C.A/ C C.B/, 0 0 2 3 (b) a basis for C.A/ \ C.B/, and (c) a vector in C.A/ C C.B/ that is not in C.A/ [ C.B/. 3. Let U, W, and X represent subspaces of a linear space V of matrices, and let Y represent an arbitrary matrix in V. (a) Show (1) that if Y ? W and Y ? X, then Y ? .W C X/, and (2) that if U ? W and U ? X, then U ? .W C X/. (b) Show (1) that .U C W/? D U? \ W ? and (2) that .U \ W/? D U? C W ? . 4. Let U, W, and X represent subspaces of a linear space V of matrices. (a) Show that .U \ W/ C .U \ X/ U \ .W C X/. (b) Show (via an example) that U \ W D f0g and U \ X D f0g does not necessarily imply that U \ .W C X/ D f0g. (c) Show that if W U, then (1) U C W D U and (2) U \ .W C X/ D W C .U \ X/.

412


5. Let U1 ; U2 ; : : : ; Uk represent subspaces of Rmn. Show that if, for j D 1, 2, : : : ; k, Uj is spanned by a (finite nonempty) set of (m n) matrices U1.j /, : : : ; Ur.jj /, then U1 C U2 C C U k D ; U1.2/; : : : ; Ur.2/ ; : : : ; U1.k/; : : : ; Ur.k/ /: sp.U1.1/; : : : ; Ur.1/ 1 2 k 6. Let U1 ; : : : ; Uk represent subspaces of Rmn. The k subspaces U1 ; : : : ; Uk are said to be independent if, for matrices U1 2 U1 , : : : ; Uk 2 Uk , the only solution to the matrix equation U1 C C U k D 0

(E.1)

is U1 D D Uk D 0. (a) Show that U1 ; : : : ; Uk are independent if and only if, for i D 2; : : : ; k, Ui and U1 C C Ui1 are essentially disjoint. (b) Show that U1 ; : : : ; Uk are independent if and only if, for i D 1; : : : ; k, Ui and U1 C C Ui1 C UiC1 C C Uk are essentially disjoint. (c) Use the results of Exercise 3 [along with Part (a) or (b)] to show that if U1 ; : : : ; Uk are (pairwise) orthogonal, then they are independent. (d) Assuming that U1 ; U2 ; : : : ; Uk are of dimension one or more and letting fU1.j /; : : : ; Ur.jj / g represent any linearly independent set of matrices in Uj (j D 1; 2; : : : ; k), show that if U1 ; U2 ; : : : ; Uk are independent, then the combined set fU1.1/; : : : ; Ur.1/ ; U1.2/; : : : ; Ur.2/ ; : : : ; U1.k/; : : : ; Ur.k/ g is 1 2 k linearly independent. (e) Assuming that U1 ; U2 ; : : : ; Uk are of dimension one or more, show that U1 ; U2 ; : : : ; Uk are independent if and only if, for every nonnull matrix U1 in U1 , every nonnull matrix U2 in U2 , : : : ; and every nonnull matrix Uk in Uk , U1 ; U2 ; : : : ; Uk are linearly independent. (f) For j D 1; : : : ; k, let pj D dim.Uj /, and let Sj represent a basis for P Uj (j D 1; : : : ; k). Define S to be the set of jkD1 pj matrices obtained by combining all of the matrices in S1 ; : : : ; Sk into a single set. Use the result of Exercise 5 [along with Part (d)] to show that (1) if U1 ; : : : ; Uk are independent, then S is a basis for U1 C C Uk ; and (2) if U1 ; : : : ; Uk are not independent, then S contains a proper subset that is a basis for U1 C C Uk . (g) Show that (1) if U1 ; : : : ; Uk are independent, then dim.U1 C C Uk / D dim.U1 / C C dim.Uk /I and (2) if U1 ; : : : ; Uk are not independent, then dim.U1 C C Uk / < dim.U1 / C C dim.Uk /:

Exercises

413

Section 17.2 7. Let A1 ; : : : ; Ak represent matrices having the same number of rows, and let B1 ; : : : ; Bk represent matrices having the same number of columns. Adopting the terminology of Exercise 6, use Part (g) of that exercise to show (a) that if C.A1 /; : : : ; C.Ak / are independent, then rank.A1 ; : : : ; Ak / D rank.A1 / C C rank.Ak /; and if C.A1 /; : : : ; C.Ak / are not independent, then rank.A1 ; : : : ; Ak / < rank.A1 / C C rank.Ak /; and (b) that if R.B1 /; : : : ; R.Bk / are independent, then 0 1 B1 B :: C rank @ : A D rank.B1 / C C rank.Bk /; Bk and if R.B1 /; : : : ; R.Bk / are not independent, then 0 1 B1 B :: C rank @ : A < rank.B1 / C C rank.Bk /: Bk 8. Show [by, e.g., using the result of Part (c)-(2) of Exercise 4 in combination with Corollary 17.2.9] that, for any m n matrix A and any m p matrix B, (a) CŒ.I AA /B D C.I AA / \ C.A; B/ and (b) CŒ.I PA /B D N.A0 / \ C.A; B/. 9. Let A D .T ; U / and B D .V ; 0/, where T is an m p matrix, U an m q matrix, and V an n p matrix, and suppose that U is of full row rank. Show that R.A/ and R.B/ are essentially disjoint [even if R.T / and R.V / are not essentially disjoint]. 10. To what extent does formula (2.15) simplify in (a) the special case where C.T / and C.U / are essentially disjoint [but R.T / and R.V / are not necessarily essentially disjoint] and (b) the special case where R.T / and R.V / are essentially disjoint. Section 17.3 11. Let T represent an m p matrix, U an m q matrix, and V an n p matrix. U , and Y D V FT . Further, define ET D I T T , FT DI T T , X D ET T T U X ET Show (a) that the partitioned matrix is a generalized X ET

414


inverse of the partitioned matrix .T ; U / and (b) that the partitioned matrix .T FT Y V T ; FT Y / is a generalized inverse of the partitioned matrix T . Do so by applying formula (E.1) from Part (a) of Exercise 10.10 to the V T U T 0 partitioned matrices and and by making use of Theorem 0 0 V 0 17.3.3. 12. Let T representan m p matrix, U an m q matrix, and V an n p G11 G12 matrix. And let (where G11 is of dimensions p m) represent G21 G22 T U a generalized inverse of the partitioned matrix . Show that (a) if G11 V 0 is a generalized inverse of T and G12 a generalized inverse of V , then R.T / and R.V / are essentially disjoint, and (b) if G11 is a generalized inverse of T and G21 a generalized inverse of U , then C.T / and C.U / are essentially disjoint. Section 17.4 13. (a) Generalize Theorem 17.4.1 by showing that, for any subspaces U1 , : : : ; Uk of Rmn, dim.U1 C C Uk / D dim.U1 / C C dim.Uk / k X dimŒ.U1 C C Ui1 / \ Ui :

(E.2)

iD2

(b) Generalize Theorem 17.4.2 by showing that, for any matrices A1 ; : : : ; Ak having the same number of rows, rank.A1 ; : : : ; Ak / D rank.A1 / C C rank.Ak / k X dimŒC.A1 ; : : : Ai1 / \ C.Ai / iD2

and, for any matrices B1 ; : : : ; Bk having the same number of columns, 0 1 B1 B :: C rank @ : A D rank.B1 / C C rank.Bk / 1 3 B1 C 6 B 7 dim4R@ ::: A \ R.Bi /5: Bi1 2 0

Bk

k X iD2

Exercises

415

Section 17.5 14. Show that, for any m n matrix A, n q matrix C, and q p matrix B, rankfŒI CB.CB/ CŒI .AC/ACg D rank.A/ C rank.C/ rank.AC/ n C rankfŒI CB.CB/ .I A A/g: [Hint. Apply equality (5.8) to the product A.CB/, and make use of equality (5.5).] Section 17.6 15. Show that if an n n matrix A is the projection matrix for a subspace U of Rn1 along a subspace V of Rn1 (where U ˚ V D Rn1 ), then A0 is the projection matrix for V ? along U? [where U? and V ? are the orthogonal complements (with respect to the usual inner product and relative to Rn1 ) of U and V, respectively]. 16. Show that, for any n p matrix X, XX is the projection matrix for C.X/ along N.XX /. 17. Let Y represent a matrix in a linear space V of mn matrices, and let U1 , : : : ; Uk represent subspaces of V. Adopting the terminology and using the results of Exercise 6, show that if U1 ; : : : ; Uk are independent and if U1 C C Uk D V, then (a) there exist unique matrices Z1 ; : : : ; Zk in U1 ; : : : ; Uk , respectively, such that Y D Z1 C CZk , and (b) for i D 1; : : : ; k, Zi equals the projection of Y on Ui along U1 C C Ui1 C UiC1 C C Uk . 18. Let U and W represent essentially disjoint subspaces (of Rn1 ) whose sum is Rn1 , and let U represent any n s matrix such that C.U / D U and W any n t matrix such that C.W / D W. (a) Show that the n .s C t / partitioned matrix .U ; W / has a right inverse. (b) TakingR to be an arbitrary right inverse of .U ; W / and partitioning R as R1 RD (where R1 has s rows), show that the projection matrix for R2 U along W equals U R1 and that the projection matrix for W along U equals W R2 . 19. Let A represent the .n n/ projection matrix for a subspace U of Rn1 along a subspace V of Rn1 (where U ˚ V D Rn1 ), let B represent the .n n/ projection matrix for a subspace W of Rn1 along a subspace X of Rn1 (where W ˚ X D Rn1 ), and suppose that A and B commute (i.e., that BA D AB).

416


(a) Show that AB is the projection matrix for U \ W along V C X. (b) Show that A C B AB is the projection matrix for U C W along V \ X. [Hint for Part (b). Observe that I .A C B AB/ D .I A/.I B/, and make use of Part (a).] 20. Let V represent a linear space of n-dimensional column vectors, and let U and W represent essentially disjoint subspaces whose sum is V. Then, an n n matrix A is said to be a projection matrix for U along W if Ay is the projection of y on U along W for every y 2 V—this represents an extension of the definition of a projection matrix for U along W given in Section 17.6d (in the special case where V D Rn ). Further, let U represent an n s matrix such that C.U / D U, and let W represent an n t matrix such that C.W / D W. (a) Show that an n n matrix A is a projection matrix for U along W if and only if AU D U and AW D 0 or, equivalently, if and only if A0 is a 0 0 U U solution to the linear system BD (in an n n matrix B). W0 0 (b) Establish the existence of a projection matrix for U along W. (c) Show that if A is a projection matrix for U along W, then I A is a projection matrix for W along U (thereby generalizing Theorem 17.6.10). (d) Let X represent any n p matrix whose columns span N.W 0 / or, equivalently, W ?. Show that an n n matrix A is a projection matrix for U along W if and only if A0 D XR for some solution R to the linear system U 0 XR D U 0 (in a p n matrix R). Section 17.7 21. Let U1 ; : : : ; Uk represent independent subspaces of Rn1 such that U1 C C Uk D Rn1 (where the independence of subspaces is as defined in Exercise 6). Further, letting si D dim.Ui / (and supposing that si > 0), take Ui to be any n si matrix such that C.Ui / D Ui (i D 1; : : : ; k). And define 0 1 B1 B :: C 1 B D .U1 ; : : : ; Uk / , partition B as B D @ : A (where, for i D 1; : : : ; k, Bk Bi has si rows), and let H D B0 B or (more generally) let H represent any matrix of the form H D B10 A1 B1 C B20 A2 B2 C C Bk0 Ak Bk ;

(E.3)

where A1 ; A2 ; : : : ; Ak are symmetric positive definite matrices. (a) Using the result of Part (g)-(1) of Exercise 6 (or otherwise), verify that the partitioned matrix .U1 ; : : : ; Uk / is nonsingular (i.e., is square and of rank n). (b) Show that H is positive definite.


417

(c) Show that (for j ¤ i D 1; : : : ; k) Ui and Uj are orthogonal with respect to H. (d) Using the result of Part(a)-(2) of Exercise 3 (or otherwise), show that, for i D 1; : : : ; k, (1) U1 C C Ui1 C UiC1 C C Uk equals the orthogonal complement U?i of Ui (where the orthogonality in the orthogonal complement is with respect to the bilinear form x0 Hy) and (2) the projection of any n 1 vector y on Ui along U1 C C Ui1 C UiC1 C CUk equals the orthogonal projection of y on Ui with respect to H. (e) Show that if, for j ¤ i D 1; : : : ; k, Ui and Uj are orthogonal with respect to some symmetric positive definite matrix H , then H is expressible in the form (E.3).

Bibliographic and Supplementary Notes §2. Theorem 17.2.13 and its corollary (Corollary 17.2.14) are based on results given by C. R. Rao—refer, for example, to Section 1.3 of Rao’s (1989) paper. §5. The approach taken in Section 5 is essentially that taken by Marsaglia and Styan (1974) in their Section 3. Section 4 is a prerequisite for Subsection b. §7. Section 7 (and Exercise 21) are based on results given by Rao and Mitra (1971, sec. 5.3).

18 Sums (and Differences) of Matrices

It is common in such areas of statistics as linear statistical models and Bayesian statistics to encounter matrices of the form R C ST U , where the dimensions (number of rows and/or number of columns) of T are small relative to those of R and where R and possibly T are diagonal matrices or are of some other form that, for example, makes them easy to “invert.” Typically, there is a need to find the determinant, ordinary or generalized inverse, or rank of a matrix of this form and/or to solve a linear system having a coefficient matrix of this form. This chapter includes (in Sections 18.1a, 18.2d-e, and 18.5a) formulas that can be very useful in such a situation. It also includes (in the remainder of Sections 18.1, 18.2, and 18.5 and in Section 18.3) a wide variety of relatively basic results (having a myriad of applications in statistics) on the determinants, ordinary or generalized inverses, and ranks of sums, differences, or linear combinations of matrices and on the solution of a linear system whose coefficient matrix is a sum of matrices. Section 18.4 gives necessary and sufficient conditions for (square) matrices whose sum is idempotent to be individually idempotent. These conditions can be used to establish a classical result on the statistical distribution of quadratic forms that is due to Cochran (1934) and is known as Cochran’s theorem. This result can in turn be used to establish the conditions under which the sums of squares in a statistical analysis of variance are distributed independently as scalar multiples of central or noncentral chi-square random variables.

18.1 Some Results on Determinants a. Determinants of matrices of the form R C ST U The following theorem gives a very useful formula for the determinant of the sum R C ST U of an n n nonsingular matrix R and the product ST U of an n m matrix S, m m nonsingular matrix T , and m n matrix U . Theorem 18.1.1. Let R represent an n n matrix, S an n m matrix, T an m m matrix, and U an m n matrix. If R and T are nonsingular, then

420


jR C ST U j D jRjjT jjT 1 C U R1 Sj:

(1.1)

Proof. Suppose that R and T are nonsingular. Then, making use of Theorem 13.3.8, we find that ˇ ˇ ˇ R S ˇ ˇ ˇ D jT 1 jjR .S/.T 1 /1 U j D jT 1 jjR C ST U j ˇU T 1 ˇ and also that ˇ ˇ ˇ R S ˇ ˇ ˇ D jRjjT 1 U R1 .S/j D jRjjT 1 C U R1 Sj: ˇU T 1 ˇ Thus,

jT 1 jjR C ST U j D jRjjT 1 C U R1 Sj;

or, equivalently (since jT 1 j D 1=jT j), jR C ST U j D jRjjT jjT 1 C U R1 Sj: Q.E.D. In the special case where R D In and T D Im , Theorem 18.1.1 simplifies to the following result. Corollary 18.1.2. For any n m matrix S and any m n matrix U , jIn C SU j D jIm C U Sj:

(1.2)

In the special case where m D 1, Corollary 18.1.2 can be restated as the following corollary. Corollary 18.1.3. For any n-dimensional column vectors s D fsi g and u D fui g, n X si ui : (1.3) jIn C su0 j D 1 C u0 s D 1 C s0 u D 1 C iD1

Corollary 18.1.3 reduces the problem of evaluating the determinant of an n n matrix of the form In C su0 to one of evaluating a scalar-valued expression. More generally, Corollary 18.1.2 converts the problem of evaluating the determinant of an n n matrix of the form In C SU into one of evaluating the determinant of an mm matrix (where m equals the number of columns in S and the number of rows in U ), which can be advantageous if m < n. Still more generally, Theorem 18.1.1 converts the problem of evaluating the determinant of an n n matrix of the form R C ST U into one of evaluating the determinant and inverse of the n n matrix R, the determinant and inverse of the m m matrix T , and the determinant of a further m m matrix—this can be advantageous if m < n and if jRj, jT j, R1, and T 1 are easy to obtain (as would be the case if, e.g., R and T were diagonal). In connection with Theorem 18.1.1, note that


R C ST U D R C .ST /T 1 .T U /:

421

(1.4)

Upon applying formula (1.1) to the right side of equality (1.4) [i.e., applying formula (1.1) with ST , T 1, and T U in place of S, T , and U , respectively] and observing that .T 1 /1 D T and jT 1 j D 1=jT j, we obtain (as an additional corollary of Theorem 18.1.1) the following, alternative formula. Corollary 18.1.4. Let R represent an n n matrix, S an n m matrix, T an m m matrix, and U an m n matrix. If R and T are nonsingular, then jR C ST U j D jRjjT C T U R1 ST j=jT j:

(1.5)

Finally, as a consequence of Theorem 18.1.1 and Corollary 18.1.4, we have the following corollary. Corollary 18.1.5. Let R represent an n n matrix, S an n m matrix, T an m m matrix, and U an m n matrix. Suppose that R and T are nonsingular. Then, R C ST U is nonsingular if and only if T 1 C U R1 S is nonsingular or, equivalently, if and only if T C T U R1 ST is nonsingular. b. Some inequalities The following theorem gives a basic result on the determinant jA C Bj of the sum of a symmetric positive definite matrix A and a symmetric nonnegative definite matrix B. Theorem 18.1.6. For any n n symmetric positive definite matrix A and any n n symmetric nonnegative definite matrix B, jA C Bj jAj; with equality holding if and only if B D 0. Proof. According to Corollary 14.3.8, there exists a matrix K such that B D K0 K. Denote by r the number of rows in K; and (for i D 1; : : : ; r) let k0i represent the i th row of K, and let Ki represent the i n submatrix (of K) obtained by striking out the last r i rows of K. Further, define A0 D A and (for i D 1; : : : ; r) Ai D A C K0i Ki . Observe (in light of Lemma 14.2.4 and Corollary 14.2.14) that Ai is symmetric and positive definite. Ki1 It is clear that (for i D 2; : : : ; r) Ki D and hence that K0i Ki D k0i K0i1 Ki1 C ki k0i . Thus, for i D 1; : : : ; r, Ai D Ai1 C ki k0i D Ai1 C ki .1/k0i ; and, applying formula (1.1), we find that 0 1 jAi j D jAi1 jj.1/jj.1 C k0i A1 i1 ki /j D jAi1 j.1 C ki Ai1 ki /:

422


Since (according to Corollary 14.2.11) A1 i1 is positive definite, we have (for i D 1; : : : ; r) that jAi j jAi1 j; with equality holding if and only if k0i A1 i1 ki D 0 or, equivalently, if and only if ki D 0. It is now clear that jAr j jAr1 j jA1 j jA0 j;

(1.6)

jAr j jA0 j:

(1.7)

implying that Further, equality holds in inequality (1.7) if and only if equality holds in all r of the inequalities (1.6). Thus, equality holds in inequality (1.7) if and only if k1 ; : : : ; kr are null, or, equivalently, if and only if K D 0, and hence (in light of Corollary 5.3.2) if and only if B D 0. Since clearly Ar D ACB (and A0 D A), we conclude that jA C Bj jAj; with equality holding if and only if B D 0. Q.E.D. Theorem 18.1.6 can be restated in the form of the following corollary. Corollary 18.1.7. For any n n symmetric positive definite matrix A and for any n n symmetric matrix C such that C A is nonnegative definite, jCj jAj; with equality holding if and only if C D A. Proof. Upon setting B D C A in Theorem 18.1.6 (and observing that C A D 0 , C D A), we obtain Corollary 18.1.7. Q.E.D. A variation on Corollary 18.1.7 (in which the requirement that A be symmetric positive definite is replaced by the weaker requirement that A be symmetric nonnegative definite) is given by the following corollary. Corollary 18.1.8. For any n n symmetric nonnegative definite matrix A and for any n n symmetric matrix C such that C A is nonnegative definite, jCj jAj;

(1.8)

with equality holding if and only if C is singular or C D A. Proof. Let us begin by observing (in light of Lemma 14.9.1) that jAj 0

(1.9)

and that, since C D A C .C A/ (implying that C is nonnegative definite), jCj 0:

(1.10)

18.2 Some Results on Inverses and Generalized Inverses and on Linear Systems

423

Now, consider the inequality jCj jAj. Either A is positive definite, or A is positive semidefinite. If A is positive definite, then it follows from Corollary 18.1.7 that jCj jAj. Alternatively, if A is positive semidefinite, then jAj D 0, and it follows from inequality (1.10) that jCj jAj. It remains to show that jCj D jAj if and only if C is singular or C D A. If C D A, then obviously jCj D jAj. If C is singular, then jCj D 0, which together with inequalities (1.8) and (1.9) implies that jCj D jAj. Suppose now that jCj D jAj. If A is positive definite, then it follows from Corollary 18.1.7 that C D A. Alternatively, if A is positive semidefinite, then (in light of Lemma 14.9.1) jAj D 0 and hence jCj D 0, implying that C is singular. Q.E.D.

18.2 Some Results on Inverses and Generalized Inverses and on Linear Systems a. Some basic results on (ordinary) inverses The following lemma gives some very basic (but useful) results on the inverse of a (nonsingular) sum of two (square) matrices. Lemma 18.2.1. Let A and B represent nn matrices. If ACB is nonsingular, then .A C B/1A D I .A C B/1 B; 1

1

D I B.A C B/

1

1

A.A C B/

B.A C B/ A D A.A C B/

;

(2.1) (2.2)

B:

Proof. Suppose that A C B is nonsingular. Then, clearly, .A C B/1A C .A C B/1 B D .A C B/1 .A C B/ D I; and hence .A C B/1A D I .A C B/1 B. Similarly, A.A C B/1 C B.A C B/1 D .A C B/.A C B/1 D I; and hence A.A C B/1 D I B.A C B/1. That B.A C B/1A D A.A C B/1 B is evident upon premultiplying both sides of equality (2.1) and postmultiplying both sides of equality (2.2) by B. Q.E.D. Note that Lemma 18.2.1 implies in particular that, for nn symmetric matrices A and B (such that A C B is nonsingular), B.A C B/1A is symmetric. In the special case where A D I, Lemma 18.2.1 reduces to the following result. Corollary 18.2.2. For any (square) matrix B such that I C B is nonsingular, .I C B/1 D I .I C B/1 B; 1

1

D I B.I C B/ ;

.I C B/

1

B.I C B/

1

D .I C B/

B:

(2.3) (2.4)

424


A further basic result on inverse matrices is given by the following lemma. Lemma 18.2.3. Let A represent an mn matrix and B an nm matrix. Then, In C BA is nonsingular if and only if Im C AB is nonsingular, in which case .In C BA/1 B D B.Im C AB/1:

(2.5)

Proof. That I C BA is nonsingular if and only if I C AB is nonsingular is evident from Corollary 18.1.5 (upon setting R D In , S D B, T D Im , and U D A). Suppose now that I C AB is nonsingular (in which case I C BA is also nonsingular), and observe that B.I C AB/ D .I C BA/B:

(2.6)

Then, upon premultiplying both sides of equality (2.6) by .I C BA/1 and postmultiplying both sides by .I C AB/1, we obtain the equality .I C BA/1 B D B.I C AB/1: Q.E.D. The following theorem establishes a relationship between the inverse of a (nonsingular) difference A B between two nonsingular matrices A and B and the inverse of the difference A1 B1 between A1 and B1. Theorem 18.2.4. Let A and B represent n n nonsingular matrices. Then, rank.A1 B1 / D rank.B A/ D rank.A B/:

(2.7)

Further, A1 B1 is nonsingular if and only if B A is nonsingular (or, equivalently, if and only if A B is nonsingular), in which case .A1 B1 /1 D A C A.B A/1A 1

D A A.A B/ A:

(2.8) (2.9)

Proof. Clearly, A1 B1 D .I B1A/A1; 1

B A D B.I B A/:

(2.10) (2.11)

Thus, recalling (from Corollary 8.3.3) that the premultiplication or postmultiplication of a matrix by a nonsingular matrix does not affect its rank [and observing that A B D .B A/], we find that rank.A1 B1 / D rank.I B1A/ D rank.B A/ D rank.A B/: It follows in particular that A1 B1 is nonsingular if and only if B A is nonsingular (or equivalently if and only if A B is nonsingular).


425

Suppose now that B A is nonsingular (in which case I B1A is also nonsingular). Then, observing [in light of equality (2.11)] that .B A/1 D ŒB.I B1A/1 D .I B1A/1 B1; we find [in light of equality (2.10)] that .A1 B1 /ŒA C A.B A/1A D .I B1A/A1 ŒA C A.I B1A/1 B1A D I B1A C B1A D I; thereby validating result (2.8) and also {since .A B/1 D Œ.B A/1 D Q.E.D. .B A/1 } result (2.9). b. Some basic results on generalized inverses Under what conditions is a generalized inverse .A C B/ of the sum of two m n matrices A and B a generalized inverse of A and a generalized inverse of B? This question is addressed in the following theorem. Theorem 18.2.5. Let A and B represent m n matrices. If R.A/ and R.B/ are essentially disjoint and if C.A/ and C.B/ are essentially disjoint, then any generalized inverse of A C B is a generalized inverse of A and a generalized inverse of B. Theorem 18.2.5 is an immediate consequence of the following result. Theorem 18.2.6. For any m n matrices A and B, A A 0 .A C B/ .A; B/ D B 0 B

(2.12)

[or, equivalently, A.A C B/ A D A, B.A C B/ B D B, A.A C B/ B D 0, and B.A C B/ A D 0] if and only if R.A/ and R.B/ are essentially disjoint and C.A/ and C.B/ are essentially disjoint. Proof (of Theorem 18.2.6). Suppose that R.A/ and R.B/ are essentially disjoint and C.A/ and C.B/ are essentially disjoint. And observe that A.A C B/ A C A.A C B/ B C B.A C B/ A C B.A C B/ B D .A C B/.A C B/ .A C B/ D A C B and hence that A.A C B/ A C B.A C B/ A A D B A.A C B/ B B.A C B/ B: (2.13) Since the left side of equality (2.13) is expressible in the form KA for some matrix K and the right side is expressible in the form LB for some matrix L, it follows from Part (2) of Corollary 17.2.3 that

426


A.A C B/ A C B.A C B/ A A D 0; B A.A C B/ B B.A C B/ B D 0; or, equivalently, that B.A C B/ A D A A.A C B/ A;

(2.14)

A.A C B/ B D B B.A C B/ B:

(2.15)

And, since the right side of equality (2.14) is expressible in the form AK for some matrix K and the right side of equality (2.15) is expressible in the form BL for some matrix L, it follows from Part (1) of Corollary 17.2.3 that B.A C B/ A D 0; A.A C B/ B D 0;

A A.A C B/ A D 0; B B.A C B/ B D 0;

or, equivalently, that A A 0 .A C B/ .A; B/ D : B 0 B Conversely, suppose that A A 0 .A C B/ .A; B/ D : B 0 B Then, making use of Corollary 4.4.5 and inequality (4.5.1), we find that A 0 A rank.A/ C rank.B/ D rank D rank .A C B/ .A; B/ 0 B B rank.A; B/ rank.A/ C rank.B/ and hence that rank.A; B/ D rank.A/ C rank.B/: It can be established in similar fashion that A rank D rank.A/ C rank.B/: B We conclude, on the basis of Theorem 17.2.4, that C.A/ and C.B/ are essentially disjoint and R.A/ and R.B/ are essentially disjoint. Q.E.D. c. A result on linear systems of the form .A C B/X D C C D (in X) Under what conditions is a solution to a linear system of the form .A C B/X D C C D (in X) a solution to the linear system AX D C and to the linear system BX D D? This question is addressed in the following theorem.


427

Theorem 18.2.7. Let A and B represent m n matrices, and let C and D represent m p matrices such that C.C/ C.A/ and C.D/ C.B/. (1) If R.A/ and R.B/ are essentially disjoint, then the linear system .A C B/X D C C D (in X) is consistent. (2) If C.A/ and C.B/ are essentially disjoint, then any solution to the linear system .A C B/X D C C D is a solution to the linear system AX D C and also to the linear system BX D D. disjoint. Proof. (1) Suppose that R.A/ and R.B/ are essentially Then, it A C follows from Theorem 17.3.2 that the linear system X D (in X) is B D consistent and hence has a solution, say X . Thus, AX D C and BX D D, implying that .A C B/X D C C D, so that X is a solution to the linear system .A C B/X D C C D. And we conclude that the linear system .A C B/X D C C D is consistent. (2) Suppose that C.A/ and C.B/ are essentially disjoint, and let X represent any solution to the linear system .ACB/X D CCD. Then, AX CBX D CCD, and a solution to the linear system AX1 C BX2 D C C D (in n p matrices X1 and X2 ) is obtained by setting X1 D X2 D X . We conclude, on the basis of Theorem 17.3.1, that X is a solution to the linear system AX D C and also to the linear system BX D D. Q.E.D. d. (Ordinary) inverses of matrices of the form R C ST U Let R represent an n n matrix, S an n m matrix, T an m m matrix, and U an m n matrix. And suppose that R and T are nonsingular. Then, result (1.1) [or (1.5)] can be used to express the determinant of R C ST U in terms of the determinants of R, T , and T 1 C U R1 S [or T C T U R1 ST ]. Can the inverse of R C ST U (assuming that R C ST U is nonsingular) be expressed in a comparable way? In what follows, the answer to this question is shown to be yes. Suppose now that R C ST U is nonsingular, and let A D T U R1. Then, since R C ST U D .I C SA/R (and since, e.g., the premultiplication or postmultiplication of a matrix by a nonsingular matrix does not affect its rank), I C SA is nonsingular, and (2.16) .R C ST U /1 D R1.I C SA/1: Further, it follows from Corollary 18.2.2 that .I C SA/1 D I .I C SA/1 SA:

(2.17)

And it follows from Lemma 18.2.3 that I C AS (D I C T U R1 S) is nonsingular and that (2.18) .I C SA/1 S D S.I C AS/1: Substituting expression (2.18) into expression (2.17) and then substituting the resultant expression into expression (2.16), we find that .R C ST U /1 D R1 ŒI S.I C AS/1A D R1 R1 S.I C AS/1 T U R1:

(2.19)

428


And, since I C AS D T .T 1 C U R1 S/ and since also I C AS D .T C T U R1 ST /T 1, T 1 C U R1 S and T C T U R1 ST are nonsingular, and .I C AS/1 D .T 1 C U R1 S/1 T 1 1

D T .T C T U R

1

ST / :

(2.20) (2.21)

Substituting expression (2.20) into expression (2.19), we obtain .R C ST U /1 D R1 R1 S.T 1 C U R1 S/1 U R1: Alternatively, substituting expression (2.21) into expression (2.19), we obtain .R C ST U /1 D R1 R1 ST .T C T U R1 ST /1 T U R1: In summary, we have the following theorem (which incorporates the content of Corollary 18.1.5). Theorem 18.2.8. Let R represent an n n matrix, S an n m matrix, T an m m matrix, and U an m n matrix. Suppose that R and T are nonsingular. Then, R C ST U is nonsingular if and only if T 1 C U R1 S is nonsingular, or, equivalently, if and only if T C T U R1 ST is nonsingular, in which case .R C ST U /1 D R1 R1 S.T 1 C U R1 S/1 U R1 D R1 R1 ST .T C T U R1 ST /1 T U R1:

(2.22) (2.23)

Some special cases of Theorem 18.2.8 are given in the following three corollaries. Corollary 18.2.9. Let S represent an n m matrix and U an m n matrix. Then, In C SU is nonsingular if and only if Im C U S is nonsingular, in which case (2.24) .In C SU /1 D In S.Im C U S/1 U : Corollary 18.2.10. Let R represent an n n nonsingular matrix, and let s and u represent n-dimensional column vectors. Then, R C su0 is nonsingular if and only if u0 R1 s ¤ 1, in which case .R C su0 /1 D R1 .1 C u0 R1 s/1 R1 su0 R1:

(2.25)

Corollary 18.2.11. Let s D fsi g and u D fui g represent n-dimensional col0 0 umn vectors. Then, P I Csu is nonsingular if and only if u s ¤ 1 (or, equivalently, if and only if i si ui ¤ 1), in which case 1 X si ui su0 : .I C su0 /1 D I .1 C u0 s/1 su0 D I 1 C

(2.26)

i

Formula (2.25) is generally attributed to Sherman and Morrison (1949 and 1950) and/or to Bartlett (1951). The more general formulas (2.22) and (2.23) are


429

often attributed to Woodbury (1950), and one or the other of them is sometimes referred to as Woodbury’s formula [although a formula that is essentially the same as formula (2.22) was given earlier by Duncan (1944) and by Guttman (1946)]. Refer to Henderson and Searle (1981a) for additional information about the history of formulas (2.22) and (2.23). Depending upon the circumstances, formula (2.22) or (2.23) can be used to great advantage in the inversion of a (nonsingular) matrix of the form R C ST U . The circumstances under which the use of formula (2.22) can be advantageous are similar to the circumstances (discussed in Section 18.1a) under which the use of formula (1.1) (for the determinant of RCST U ) can be advantageous. Specifically, the use of formula (2.22) can be advantageous if the dimensions of T 1 C U R1 S (or, equivalently, of T ) are small relative to those of R C ST U (or, equivalently, of R) and if R1 and T 1 are easy to obtain. Formula (2.23) calls for more matrix multiplications than formula (2.22), but does not call for the inversion of T . Formulas (2.22) and (2.23) were derived by making use of Corollary 18.2.2 and Lemma 18.2.3. An alternative derivation of these formulas is obtained by applying Theorem 8.5.11 (on the inverse of a partitioned matrix) to the partitioned matrices R ST R S and . TU T U T 1 1 1 If T C U R S is nonsingular, then (according to Theorem 8.5.11) R S is nonsingular, R C ST U is also nonsingular, and U T 1 .R C ST U /1 .R C ST U /1 ST T U .R C ST U /1 T T U .R C ST U /1 ST 1 R S D U T 1 1 R R1 S.T 1 C U R1 S/1 U R1 R1 S.T 1 C U R1 S/1 ; D .T 1 C U R1 S/1 U R1 .T 1 C U R1 S/1 implying in particular that .R C ST U /1 D R1 R1 S.T 1 C U R1 S/1 U R1: R ST 1 Similarly, if T C T U R ST is nonsingular, then is nonsingular, TU T R C ST U is also nonsingular, and .R C ST U /1 .R C ST U /1 S U .R C ST U /1 T 1 U .R C ST U /1 S 1 R ST D TU T 0 1 1 R R1 ST .T R1 ST .T C T U R1 ST /1 1 1 1 A; C T U R ST / T U R D@ 1 1 1 1 1 .T C T U R ST / T U R .T C T U R ST /

430


implying in particular that .R C ST U /1 D R1 R1 ST .T C T U R1 ST /1 T U R1: Either of the two formulas (2.22) and (2.23) can be obtained as a “special case” of the other. To see this, reexpress R C ST U as R C ST U D R C .ST /T 1 .T U /: Then, applying formula (2.22) with ST , T 1, and T U in place of S, T , and U , respectively, we obtain .R C ST U /1 D R1 R1 ST Œ.T 1 /1 C T U R1 ST 1 T U R1 D R1 R1 ST .T C T U R1 ST /1 T U R1: Similarly, applying formula (2.23) with ST , T 1, and T U in place of S, T , and U , respectively, we obtain .R C ST U /1 D R1 R1 ST T 1 .T 1 C T 1 T U R1 ST T 1 /1 T 1 T U R1 D R1 R1 S.T 1 C U R1 S/1 U R1: e. Generalized inverses of matrices of the form R C ST U Let R represent an n q matrix, S an n m matrix, T an m p matrix, and U a pq matrix. In the special case where R, T , and T CT U R1 ST are nonsingular, R C ST U is nonsingular, and formula (2.23) gives the inverse of R C ST U in terms of the inverses of R and T C T U R1 ST . Consider now the general case where R and T may not be nonsingular or even square. Does the expression obtained from formula (2.23) by replacing R1 and .T C T U R1 ST /1 with generalized inverses R and .T C T U R ST / give a generalized inverse of R C ST U ? That is, is the matrix R R ST .T C T U R ST / T U R necessarily a generalized inverse of R C ST U ? To answer this question, let D D ST U and Q D T CT U R ST , and observe that .R C D/R ST D RR ST C S.Q T / D SQ .I RR /ST ; T U R .R C D/ D T U R R C .Q T /U D QU T U .I R R/; SQU D D C DR D; so that


431

.R C D/.R R ST Q T U R /.R C D/ D R C DR R C RR D C DR D .R C D/R ST Q T U R .R C D/ D R C DR R C RR D C DR D ŒSQ .I RR /ST Q ŒQU T U .I R R/ D R C DR R C RR D C DR D SQU C SQQ T U .I R R/ C .I RR /ST Q QU .I RR /ST Q T U .I R R/ D R C DR R C RR D C DR D D DR D C SQQ T U .I R R/ C .I RR /ST Q QU .I RR /ST Q T U .I R R/ D R C D ST U .I R R/ .I RR /ST U C SQQ T U .I R R/ C .I RR /ST Q QU .I RR /ST Q T U .I R R/ D R C D S.I QQ /T U .I R R/ .I RR /ST .I Q Q/U .I RR /ST Q T U .I R R/: Thus, we have the following theorem. Theorem 18.2.12. Let R represent an n q matrix, S an n m matrix, T an m p matrix, and U a p q matrix, and define Q D T C T U R ST . Then, the matrix (2.27) R R ST Q T U R is a generalized inverse of the matrix R C ST U if and only if S.I QQ /T U .I R R/ C .I RR /ST .I Q Q/U C .I RR /ST Q T U .I R R/ D 0:

(2.28)

It is clear (in light of Lemma 9.3.5) that condition (2.28) is satisfied if R.T U / R.R/ and C.ST / C.R/. In fact, condition (2.28) is satisfied if R.ST U / R.R/ and C.ST U / C.R/, as is evident from the following lemma. Lemma 18.2.13. Let R represent an n q matrix, S an n m matrix, T an m p matrix, and U a p q matrix, and define Q D T C T U R ST . If R.ST U / R.R/ and C.ST U / C.R/, then .I QQ /T U .I R R/ D 0;

.I RR /ST .I Q Q/ D 0; .I RR /ST Q T U .I R R/ D 0:

(2.29) (2.30) (2.31)

Proof. Let D D ST U , and observe that QQ .T C T U R ST / D T C T U R ST , or, equivalently, that .I QQ /T D .I QQ /T U R ST , and hence that (2.32) .I QQ /T U D .I QQ /T U R D: It can be shown in similar fashion that ST .I Q Q/ D DR ST .I Q Q/:

(2.33)

432


Further, .T C T U R ST /Q .T C T U R ST / D T C T U R ST ; or, equivalently, T Q T D T C T U R ST T U R ST Q T T Q T U R ST T U R ST Q T U R ST ; implying that ST Q T U D D C DR D DR ST Q T U ST Q T U R D DR ST Q T U R D:

(2.34)

And, if R.D/ R.R/ and C.D/ C.R/ [in which case D.I R R/ D 0 and .I RR /D D 0], results (2.29)–(2.31) follow from results (2.32)–(2.34). Q.E.D. In light of Lemma 18.2.13, we have (as a consequence of Theorem 18.2.12) the following result. Theorem 18.2.14. Let R represent an n q matrix, S an n m matrix, T an m p matrix, and U a p q matrix. If R.ST U / R.R/ and C.ST U / C.R/, then the matrix R R ST .T C T U R ST / T U R is a generalized inverse of the matrix R C ST U . Observe (in connection with Theorem 18.2.14) that R C ST U can be reexpressed as R C ST U D R C .ST /T .T U /: Thus, applying Theorem 18.2.14 with ST , T , and T U in place of S, T , and U , respectively, we obtain the following corollary, which generalizes result (2.22). Corollary 18.2.15. Let R represent an n q matrix, S an n m matrix, T an m p matrix, and U a p q matrix. If R.ST U / R.R/ and C.ST U / C.R/, then the matrix R R ST T .T C T T U R ST T / T T U R is a generalized inverse of the matrix R C ST U . f. An infinite-series representation of the inverse of a matrix of the form IA Let A1 ; A2 ; : : : represent a sequence of m n matrices, and (for i D 1; : : : ; m, .k/ j D 1; : : : ; n, and k D 1; 2; : : :) let aij represent the ij th element of Ak . If for every i and j (i.e., if for i D 1; : : : ; m and j D 1; : : : ; n) there exists a scalar .1/ .2/ ; aij ; : : : , we say that the m n aij such that aij is the limit of the sequence aij


433

matrix A, whose ij th element is aij , is the limit of the sequence A1 ; A2 ; : : : (or that the sequence A1 ; A2 ; : : : converges to A), and write limk!1 Ak D A (or Ak ! A). If the sequence A1 ; A2 ; : : : has a limit, it is said to be convergent. If the sequence .1/ .2/ A1 ; A2 ; : : : does not have a limit (i.e., if for some i and j the sequence aij ; aij ;::: does not have a limit), it is said to be divergent. Four basic properties of sequences of matrices (which are almost immediate consequences of basic properties of sequences of scalars) are as follows: (1) For any mn matrix A, the sequence A; A; : : : (each of whose members equals A) converges to A. (2) For any scalar c (including c D 1) and for any sequence A1 ; A2 ; : : : of m n matrices that converge to an (m n) matrix A, limk!1 .cAk / D cA. (3) If A1 ; A2 ; : : : and B1 ; B2 ; : : : are sequences of m n matrices that converge to (m n) matrices A and B, respectively, then lim .Ak C Bk / D A C B:

k!1

(2.35)

(4) If A1 ; A2 ; : : : is a sequence of m n matrices that converges to an (m n) matrix A and B1 ; B2 ; : : : a sequence of n p matrices that converges to an (n p) matrix B, then (2.36) lim Ak Bk D AB: k!1

Results (2.35) and (2.36) extend in a straightforward way to an arbitrary (finite) number of sequences of matrices. Corresponding to any sequence A1 ; A2 ; : : : of m n matrices is another sequence S1 ; S2 ; : : : , where S1 D A1 , S2 D A1 C A2 , and more generally (for P k D 1; 2; : : :) Sk D kiD1 Ai . The sequence S1 ; S2 ; : : : is referred to as the infinite series (or simply as the series) generated by the sequence A1 ; A2 ; : : : . Further, the matrix Ak is referred to as the kth term of the infinite series, and the matrix Sk is referred to as the kth partial sum. And, if the infinite series S1 ; S2 ; : : : converges, its limit is referred to as the sum of the infinite series and is denoted by the symbol P1 A2 C . kD1 Ak or by the expression A1 CP It is convenient to use the symbol 1 kD1 Ak , or the expression A1 C A2 C , not only to represent the sum of the infinite series generated by the sequence A1 ; A2 ; : : : (when the infinite series converges) but also to represent the infinite series itself—the P intended usage must be determined from the context. Further, the symbol 1 kDp Ak (or the expression Ap C ApC1 C ) is used to represent an infinite series (Ap ; Ap C ApC1 ; ) generated by a sequence Ap ; ApC1 ; : : : of m n matrices and also to represent the sum of that infinite series (where p is an integer, not necessarily equal to 1). The following theorem gives a useful result on the inversion of a matrix of the form I A. Theorem 18.2.16. Let A represent an n n matrix. Then, the infinite series I C A C A2 C A3 C converges if and only if limk!1 Ak D 0, in which case

434


I A is nonsingular and .I A/1 D

1 X

A k D I C A C A2 C A 3 C

(2.37)

kD0

(where A0 D I). Preliminary to proving Theorem 18.2.16, it is convenient to establish the following lemma. Lemma 18.2.17.PLet A0 ; A1 ; A2 ; : : : represent a sequence of m n matrices. lim A D 0. If the infinite series 1 kD0 Ak converges, then P k!1 k A converges. And, for k D Proof (of Lemma 18.2.17). Suppose that 1 k kD0 Pk 0; 1; 2; : : : , let Sk D iD0 Ai . Then (in light of our supposition), the sequence S0 ; S1 ; S2 ; : : : has a limit, say S. Thus, observing that (for k D 1; 2; : : :) Ak D Sk Sk1 and that limk!1 Sk1 D S, it follows that lim Ak D lim .Sk Sk1 / D lim Sk lim Sk1 D S S D 0:

k!1

k!1

k!1

k!1

Q.E.D. Proof (of Theorem 18.2.16). If the infinite series I C A C A C A C converges, then it is clear from Lemma 18.2.17 that limk!1 Ak D 0. Conversely, suppose that limk!1 Ak D 0. And observe that 2

and that

3

.I C A C A2 C C Ak /.I A/ D I AkC1

(2.38)

lim .I AkC1 / D I lim AkC1 D I 0 D I:

(2.39)

k!1

k!1

Then, for any n 1 vector x such that .I A/x D 0, we find that .I AkC1 /x D .I C A C A2 C C Ak /.I A/x D 0 (k D 0; 1; 2; : : :) and consequently that x D Ix D Œ lim .I AkC1 /x D lim .I AkC1 /x D lim 0 D 0: k!1

k!1

k!1

Thus, I A is nonsingular. Further, postmultiplying both sides of equality (2.38) by .I A/1, we obtain the equality I C A C A2 C C Ak D .I AkC1 /.I A/1: P k And, in light of result (2.39), we conclude that the infinite series 1 kD0 A converges and that 1 X kD0

Ak D lim .I AkC1 /.I A/1 k!1

D Œ lim .I AkC1 /.I A/1 D I.I A/1 D .I A/1: k!1


435

Q.E.D. Theorem 18.2.16 can be generalized as follows. Theorem 18.2.18. Let A and B represent n n matrices. Suppose that B is nonsingular, and define F D B1A. Then, the infinite series B1 C F B1 C F 2 B1 C F 3 B1 C converges if and only if limk!1 F k D 0, in which case B A is nonsingular and .B A/1 D

1 X

F k B1 D B1 C F B1 C F 2 B1 C F 3 B1 C

(2.40)

kD0

(where F 0 D I). P k 1 converges, then, as a consequence Proof. If the infinite series 1 kD0 F B k 1 of Lemma 18.2.17, limk!1 F B D 0, and hence lim F k D lim F k B1 B D . lim F k B1 /B D 0B D 0:

k!1

k!1

k!1

Conversely, suppose that limk!1 F k D 0. Then, it follows from Theorem 18.2.16 that I F is nonsingular and that .I F /1 D

1 X

F k:

kD0

Further, B A D B.I F /, implying (since B is nonsingular) that B A is nonsingular and [in light of result (2.36)] that .B A/1 D .I F /1 B1 D

F k B1

1 X kD0

D

lim

p X

p!1

D lim

p!1

F k B1

kD0

p X kD0

F k B1 D

1 X

F k B1:

kD0

Q.E.D. Formula (2.37) is applicable if (and only if) limk!1 A D 0, and, more generally, formula (2.40) is applicable if (and only if) limk!1 F k D 0. Depending on the nature of A (or F ), it may be difficult to determine whether the condition limk!1 Ak D 0 (or the condition limk!1 F k D 0) is satisfied. A condition that is more stringent, but that is typically easier to check, can be obtained from the following theorem. k

Theorem 18.2.19. Let A represent an n n matrix. If kAk < 1 (where the norm is the usual norm), then limk!1 Ak D 0.

436


Preliminary to proving Theorem 18.2.19, it is convenient to establish the following two lemmas, which are of some interest in their own right. Lemma 18.2.20. Let A1 ; A2 ; : : : represent a sequence of m n matrices. Then, A1 ; A2 ; : : : converges to an (m n) matrix A if and only if kAk Ak ! 0 (where the norm is the usual norm). Proof (of Lemma 18.2.20). Let aij represent the ij th element of A, and (for .k/ represent the ij th element of Ak . Then, it is clear from the k D 1; 2; : : :) let aij very definition of the usual norm that (for k D 1; 2; : : :) .k/ .k/ max jaij aij j kAk Ak .mn/1=2 max jaij aij j: i;j

i;j

(2.41)

Suppose now that kAk Ak ! 0. Then, corresponding to each positive scalar , there exists a positive integer p such that, for k > p, kAk Ak < and hence [in light of result (2.41)] such that, for k > p (and for i D 1; : : : ; m and .k/ .k/ aij k < . Thus, limk!1 aij D aij (for i D 1; : : : ; m j D 1; : : : ; n), kaij and j D 1; : : : ; n), and consequently Ak ! A. .k/ ! aij (for Conversely, suppose that Ak ! A. Then, by definition, aij i D 1; : : : ; m and j D 1; : : : ; n). Thus, corresponding to any positive scalar , there exists a positive integer p such that, for k > p (and for i D 1; : : : ; m and .k/ aij j < .mn/1=2 and hence [in light of result (2.41)] such j D 1; : : : ; n), jaij Q.E.D. that, for k > p, kAk Ak < . Thus, kAk Ak ! 0. Lemma 18.2.21. For any m n matrix A and n p matrix B, kABk kAkkBk (where the norms are the usual norms). Proof (of Lemma 18.2.21). Denote the first, : : : ; pth columns of B by b1 , : : : ; bp , respectively. Then, the j th column of AB is Abj , and [in light of result (5.2.5)] we have that 2

0

kABk D trŒ.AB/ AB D

p X

.Abj /0 Abj :

(2.42)

j D1

Further, letting a0i represent the i th row of A (for i D 1; : : : ; m) and making use of inequality (6.3.2) (the Schwarz inequality), we find that (for j D 1; : : : ; p) .Abj /0 Abj D

m X

.a0i bj /2

iD1

m X

.a0i ai /.bj0 bj / D kAk2 bj0 bj :

iD1

And, upon combining result (2.43) with result (2.42), we obtain 2

2

kABk kAk

p X j D1

bj0 bj D kAk2 kBk2;

(2.43)

18.3 Some Results on Positive (and Nonnegative) Definiteness

437

or, equivalently, kABk kAkkBk: Q.E.D. Proof (of Theorem 18.2.19). By making repeated use of Lemma 18.2.21, we find that (for any positive integer k) kAk k kAkk :

(2.44)

Suppose now that kAk < 1. Then, it follows from a basic result on limits [which is example 14.8(c) in Bartle’s (1976) book] that kAkk ! 0, implying [in light of inequality (2.44)] that kAk k ! 0. We conclude, on the basis of Lemma Q.E.D. 18.2.20, that limk!1 Ak D 0.

18.3 Some Results on Positive (and Nonnegative) Definiteness a. Matrices of the form V C

Pk

i D1

xi Ai

P The following theorem gives a basic result on matrices of the form V C kiD1 xi Ai (where V is symmetric and positive definite and A1 ; : : : ; Ak are symmetric). Theorem 18.3.1. Let V represent an nn symmetric positive definite matrix, and let A1 ; : : : ; Ak represent k symmetric matrices of order n. And let x D fxi g represent a k 1 vector. Then, there exists a neighborhood N of the k 1 null P vector 0 such that, for x 2 N , V C kiD1 xi Ai is (symmetric) positive definite.

.s/ .s/ represent the s s leading Proof. For s D 1; : : : ; n, let A.s/ 1 ; : : : ; Ak , and V principal submatrices of A1 ; : : : ; Ak and V , respectively. Then, when regarded as P a function of x, jV .s/ C kiD1 xi A.s/ i j is continuous at 0 (and at every other point in Rk ), as is evident from the very definition of a determinant [given by formula (13.1.2) or (13.1.6)]. Thus,

lim jV .s/ C

x!0

k X iD1

.s/ xi A.s/ C i j D jV

k X iD1

.s/ 0A.s/ j: i j D jV

And, since (according to Theorem 14.9.5) jV .s/ j > 0, there exists a neighborhood P Ns of 0 such that, for x 2 Ns , jV .s/ C kiD1 xi A.s/ i j > 0. Now, take N to be the smallest of the n neighborhoods N1 ; : : : ; Nn . Then, P P .n/ for x 2 N , jV .1/ C kiD1 xi A.1/ C kiD1 xi A.n/ i j > 0, : : : ; jV i j > 0. And, Pk Pk .1/ .n/ .1/ .n/ observing that V C iD1 xi Ai , : : : ; V C iD1 xi Ai are the leading P principal submatrices of V C kiD1 xi Ai , we conclude (on the basis of Theorem P Q.E.D. 14.9.5) that, for x 2 N , V C kiD1 xi Ai is positive definite. In the special case where k D 1, Theorem 18.3.1 can be restated as the following corollary.

438


Corollary 18.3.2. Let A represent an n n symmetric matrix, and let V represent an n n symmetric positive definite matrix. Then, there exists a (strictly) positive scalar c such that V C xA is (symmetric) positive definite for every scalar x in the interval c < x < c. The following corollary gives a variation on Corollary 18.3.2. Corollary 18.3.3. Let A represent an n n symmetric matrix, and let V represent an n n symmetric positive definite matrix. Then, there exists a scalar c such that A C xV is (symmetric) positive definite for every (scalar) x > c. Proof. Let w represent an arbitrary scalar. Then, it follows from Corollary 18.3.2 that there exists a (strictly) positive scalar d such that V C wA is positive definite for d < w < d . And, since x > 1=d ) 0 < 1=x < d and since (for x ¤ 0) A C xV D xŒV C .1=x/A, it follows from Lemma 14.2.3 that A C xV is positive definite for x > 1=d . Q.E.D. b. Matrices of the form A1 B1 The following theorem gives sufficient conditions for the difference A1 B1 between the inverses of a symmetric positive definite matrix A and a (nonsingular) matrix B to be positive definite or positive semidefinite. Theorem 18.3.4. Let A represent an n n symmetric positive definite matrix, and let B represent an n n matrix. (1) If B A is positive definite or, more generally, if B A is nonnegative definite and nonsingular (in which case B is positive definite), then A1 B1 is positive definite. (2) If B A is symmetric and positive semidefinite (in which case B is symmetric and positive definite), then A1 B1 is positive semidefinite. Proof. (1) Suppose that B A is nonnegative definite and nonsingular [which, since B D AC.BA/, implies (in light of Lemma 14.2.4) that B is positive definite (and hence nonsingular)]. Then, it follows from Theorem 18.2.4 that A1 B1 is nonsingular and that .A1 B1 /1 D A C A.B A/1 A D A C A0 .B A/1 A: Moreover, it follows from Corollary 14.2.11 that .B A/1 is nonnegative definite, implying (in light of Theorem 14.2.9) that A0 .B A/1 A is nonnegative definite and hence (in light of Lemma 14.2.4) that A C A0 .B A/1 A is positive definite. Thus, .A1 B1 /1 is positive definite. Since A1 B1 D Œ.A1 B1 /1 1 , we conclude (on the basis of Corollary 14.2.11) that A1 B1 is positive definite. (2) Suppose that B A is symmetric and positive semidefinite [which, since B D A C .B A/, implies that B is symmetric and (in light of Lemma 14.2.4) that B is positive definite (and hence nonsingular)]. And let r D rank.B A/ [and observe (in light of Corollary 14.3.12) that r < n]. Then, according to Theorem 14.3.7, there exists an r n matrix P (of rank r) such that B A D P 0 P . Thus,

18.4 Some Results on Idempotency

439

B D A C .B A/ D A C P 0 P D A C P 0 Ir P I and it follows from Theorem 18.2.8 that Ir C P A1 P 0 is nonsingular and that B1 D A1 A1 P 0 .Ir C P A1 P 0 /1 P A1; or, equivalently, that A1 B1 D .P A1 /0 .Ir C P A1 P 0 /1 P A1: Moreover, since A1 is positive definite, Ir C P A1 P 0 is positive definite and hence .Ir C P A1 P 0 /1 is positive definite. And, since rank.P A1 / D r .< n/, it follows from Theorem 14.2.9 that .P A1 /0 .Ir C P A1 P 0 /1 P A1 is positive semidefinite. Q.E.D.

18.4 Some Results on Idempotency a. Basic results Suppose that the sum of k (square) matrices A1 ; : : : ; Ak is idempotent. Under what condition(s) are A1 ; : : : ; Ak idempotent? This question is answered in the following theorem. Theorem 18.4.1. Let A1 ; : : : ; Ak represent n n matrices, and define A D A1 C C Ak . Suppose that A is idempotent. Then, each of the following three conditions implies the other two: (1) Ai Aj D 0 (for j ¤ i D 1; : : : ; k) and rank.A2i / D rank.Ai / (for i D 1; : : : ; k); (2) A2i D Ai (for i D 1; : : : ; k); (3) rank.A1 / C C rank.Ak / D rank.A/. Preliminary to proving Theorem 18.4.1, it is convenient to establish the following two lemmas. Lemma 18.4.2. An n n matrix A is idempotent if and only if rank.A/ C rank.I A/ D n. Proof (of Lemma 18.4.2). According to Theorem 11.7.1, A is idempotent if and only if N.A/ D C.I A/. Moreover, since [according to result (11.1.1)] N.A/ C.I A/, it follows from Theorem 4.3.10 that N.A/ D C.I A/ if and only if dimŒN.A/ D dimŒC.I A/. Since (according to Lemma 11.3.1) dimŒN.A/ D nrank.A/ and since (by definition) dimŒC.IA/ D rank.IA/, we conclude that A is idempotent if and only if n rank.A/ D rank.I A/ or, equivalently, if and only if rank.A/ C rank.I A/ D n. Q.E.D. Lemma 18.4.3. For any idempotent matrices A and B (of the same size), A C B is idempotent if and only if BA D AB D 0.

440


Proof (of Lemma 18.4.3). Clearly, .A C B/2 D A2 C B2 C AB C BA D A C B C AB C BA: Thus, A C B is idempotent if and only if AB C BA D 0. Suppose now that BA D AB D 0. Then, obviously, AB C BA D 0, and consequently A C B is idempotent. Conversely, suppose that AB C BA D 0. Then, AB C ABA D A2 B C ABA D A.AB C BA/ D 0 and

ABA C BA D ABA C BA2 D .AB C BA/A D 0;

implying that AB BA D AB C ABA .ABA C BA/ D 0 0 D 0 and hence that AB D BA: Further, AB D .1=2/.AB C AB/ D .1=2/.AB C BA/ D 0: Q.E.D. Proof (of Theorem 18.4.1). The proof consists of successively showing that Condition (1) implies Condition (2), Condition (2) implies Condition (3), and Condition (3) implies Condition (1). (1) ) (2). Suppose that Condition (1) is satisfied. Then (for i D 1; : : : ; k) A2i D Ai A D Ai AA D A2i A D A3i : Moreover, since rank.A2i / D rank.Ai /, it follows from Corollary 4.4.7 that C.A2i / D C.Ai /, so that Ai D A2i Li for some matrix Li . Thus, Ai D A2i Li D A3i Li D Ai .A2i Li / D A2i : (2) ) (3). Suppose that Condition (2) is satisfied. Then, making use of Corollary 10.2.2, we find that k X iD1

rank.Ai / D

k X iD1

tr.Ai / D tr

k X

Ai D tr.A/ D rank.A/:

iD1

(3) ) (1). Suppose that Condition (3) is satisfied. And define A0 D In A. P Then, kiD0 Ai D I. Moreover, it follows from Lemma 10.2.4 that rank.A0 / D P n rank.A/, so that kiD0 rank.Ai / D n. Thus, making use of result (4.5.10), we find (for i D 1; : : : ; k) that


rank.I Ai / D rank

Am

k X mD0 .m¤i/

k X

441

rank.Am / D n rank.Ai /;

mD0 .m¤i/

so that rank.Ai / C rank.I Ai / n, implying [since rank.Ai / C rank.I Ai / rank.Ai C I Ai / D rank.In / D n] that rank.Ai / C rank.I Ai / D n and hence implying (in light of Lemma 18.4.2) that Ai is idempotent [and that rank.A2i / D rank.Ai /]. Similarly, again making use of result (4.5.10), we find (for j ¤ i D 1; : : : ; k) that rank.I Ai Aj / D rank

k X

Am

mD0 .m¤i;j /

k X

rank.Am / D n rank.Ai / rank.Aj /

mD0 .m¤i;j /

n rank.Ai C Aj /; so that rank.Ai C Aj / C rank.I Ai Aj / n, implying [since rank.Ai C Aj / C rank.I Ai Aj / rank.Ai C Aj C I Ai Aj / D rank.In / D n] that rank.Ai C Aj / C rankŒI .Ai C Aj / D n and hence (in light of Lemma 18.4.2) that Ai C Aj is idempotent. We conclude, Q.E.D. on the basis of Lemma 18.4.3, that Ai Aj D 0 for j ¤ i D 1; : : : ; k. The following theorem complements Theorem 18.4.1. Theorem 18.4.4. Let A1 ; : : : ; Ak represent n n matrices, and define A D A1 C C Ak . If A1 ; : : : ; Ak are idempotent and if Ai Aj D 0 for all i and j ¤ i , then A is idempotent and rank.A1 / C C rank.Ak / D rank.A/. Proof. Suppose that A1 ; : : : ; Ak are idempotent and that Ai Aj D 0 for all i and j ¤ i . Then, A is idempotent, as is evident upon observing that X X X A2i C Ai Aj D Ai D A: A2 D i

i;j ¤i

i

Further, it follows from Theorem 18.4.1 that rank.A1 / C C rank.Ak / D rank.A/. Q.E.D. In connection with Theorem 18.4.1 (and Theorem 18.4.4), observe that, in the special case where A D In , rank.A/ D n. Observe also that if Ai is a symmetric matrix, then A2i D A0i Ai , implying (in light of Corollary 7.4.5) that rank.A2i / D rank.Ai /. Thus, in the special case where A1 ; : : : ; Ak are symmetric matrices, Condition (1) of Theorem 18.4.1 reduces to the condition Ai Aj D 0 .for j ¤ i D 1; : : : ; k/: In light of these observations, we obtain, as a special case of Theorem 18.4.1, the following theorem.

442


Theorem 18.4.5. Let A1 ; : : : ; Ak represent n n symmetric matrices such that A1 C C Ak D I. Then, each of the following three conditions implies the other two: (1) Ai Aj D 0 (for j ¤ i D 1; : : : ; k); (2) A2i D Ai (for i D 1; : : : ; k); (3) rank.A1 / C C rank.Ak / D n. Theorem 18.4.5 provides the underpinnings for Cochran’s (1934) theorem (on the statistical distribution of quadratic forms) and is itself sometimes referred to as Cochran’s theorem. b. Extensions Theorems 18.4.1 and 18.4.4 are generalized in the following two theorems. Theorem 18.4.6. Let A1 ; : : : ; Ak represent nn matrices, and let V represent an n n symmetric nonnegative definite matrix. Define A D A1 C C Ak , and take R to be any matrix such that V D R0 R. Suppose that RAR0 is idempotent. Then, each of the following three conditions implies the other two: (1) RAi V Aj R0 D 0 (for j ¤ i D 1; : : : ; k) and rank.RAi V Ai R0 / D rank.RAi R0 / (for i D 1; : : : ; k); (2) RAi V Ai R0 D RAi R0 (for i D 1; : : : ; k); (3) rank.RA1 R0 / C C rank.RAk R0 / D rank.RAR0 /. Theorem 18.4.7. Let A1 ; : : : ; Ak represent nn matrices, and let V represent an n n symmetric nonnegative definite matrix. Define A D A1 C C Ak , and take R to be any matrix such that V D R0 R. If RAi V Ai R0 D RAi R0 for all i and if RAi V Aj R0 D 0 for all i and j ¤ i , then RAR0 is idempotent and rank.RA1 R0 / C C rank.RAk R0 / D rank.RAR0 /. In connection with Theorems 18.4.6 and 18.4.7, note that the existence of a matrix R such that V D R0 R is guaranteed by Corollary 14.3.8. Note also that, in the special case where A1 ; : : : ; Ak are symmetric, Condition (1) of Theorem 18.4.6 reduces to the condition RAi V Aj R0 D 0 .for j ¤ i D 1; : : : ; k/ {since, in that special case, rank.RAi V Ai R0 / D rankŒ.RAi R0 /0 RAi R0 D rank.RAi R0 /}. And note that, in the special case where V D I (and where R is chosen to be I), Theorems 18.4.6 and 18.4.7 reduce to Theorems 18.4.1 and 18.4.4. Theorems 18.4.6 and 18.4.7 can be derived from Theorems 18.4.1 and 18.4.4— upon applying Theorems 18.4.1 and 18.4.4 with RA1 R0 ; : : : ; RAk R0 in place of A1 ; : : : ; Ak , respectively, we obtain Theorems 18.4.6 and 18.4.7. Theorems 18.4.6 and 18.4.7 can be restated in terms that do not involve the matrix R. For this purpose, observe that


443

.RAR0 /2 D RAR0 ) R0 RAV AR0 R D R0 RAR0 R; RAi V Ai R0 D RAi R0 ) R0 RAi V Ai R0 R D R0 RAi R0 R; RAi V Aj R0 D 0 ) R0 RAi V Aj R0 R D 0; and conversely (in light of Corollary 5.3.3) that R0 RAV AR0 R D R0 RAR0 R ) RAV AR0 R D RAR0 R ) .RAR0 /2 D RAR0 ; R0 RAi V Ai R0 R D R0 RAi R0 R ) RAi V Ai R0 R D RAi R0 R ) RAi V Ai R0 D RAi R0 ; R0 RAi V Aj R0 R D 0 ) RAi V Aj R0 R D 0 ) RAi V Aj R0 D 0: Thus,

.RAR0 /2 D RAR0 , V AV AV D V AV ; RAi V Ai R0 D RAi R0 , V Ai V Ai V D V Ai V ;

(4.1) (4.2)

RAi V Aj R0 D 0 , V Ai V Aj V D 0:

(4.3)

Further, in light of Corollary 7.4.4, we have that rank.RAR0 / D rank.R0 RAR0 / D rank.R0 RAR0 R/ D rank.V AV /; 0

0

0

0

(4.4)

0

rank.RAi R / D rank.R RAi R / D rank.R RAi R R/ D rank.V Ai V /; (4.5) rank.RAi V Ai R0 / D rank.R0 RAi V Ai R0 / D rank.R0 RAi V Ai R0 R/ D rank.V Ai V Ai V /:

(4.6)

Based on results (4.1)–(4.6), Theorems 18.4.6 and 18.4.7 can be restated as the following two theorems. Theorem 18.4.8. Let A1 ; : : : ; Ak represent n n matrices, let V represent an n n symmetric nonnegative definite matrix, and define A D A1 C C Ak . Suppose that V AV AV D V AV . Then, each of the following three conditions implies the other two: (1) V Ai V Aj V D 0 (for j ¤ i D 1; : : : ; k) and rank.V Ai V Ai V / D rank.V Ai V / (for i D 1; : : : ; k); (2) V Ai V Ai V D V Ai V (for i D 1; : : : ; k); (3) rank.V A1 V / C C rank.V Ak V / D rank.V AV /. Theorem 18.4.9. Let A1 ; : : : ; Ak represent n n matrices, let V represent an n n symmetric nonnegative definite matrix, and define A D A1 C C Ak . If V Ai V Ai V D V Ai V for all i and if V Ai V Aj V D 0 for all i and j ¤ i , then V AV AV D V AV and rank.V A1 V / C C rank.V Ak V / D rank.V AV /. Note that, in the special case where A1 ; : : : ; Ak are symmetric, Condition (1) of Theorem 18.4.8 reduces to the condition V Ai V Aj V D 0 .for j ¤ i D 1; : : : ; k/

444


{since, in that special case, rank.V Ai V Ai V / D rank.RAi V Ai R0 / D rankŒ.RAi R0 /0 RAi R0 D rank.RAi R0 / D rank.V Ai V /}. Note also that, in the special case where V is positive definite (and hence nonsingular), simplifications can be effected in Theorem 18.4.8 (and in Theorem 18.4.9) by observing that V AV AV D V AV , .AV /2 D AV , AV A D A; V Ai V Ai V D V Ai V , .Ai V /2 D Ai V , Ai V Ai D Ai ; V Ai V Aj V D 0 , Ai V Aj V D 0 , Ai V Aj D 0; and that rank.V AV / D rank.AV / D rank.A/; rank.V Ai V / D rank.Ai V / D rank.Ai /; rank.V Ai V Ai V / D rankŒ.Ai V /2 D rank.Ai V Ai /:

18.5 Some Results on Ranks a. Ranks of matrices of the form R C ST U In Sections 18.1a and 18.2d–e, expressions were obtained for the determinant and for the ordinary or generalized inverse of a matrix of the form R C ST U . These expressions (when applicable) give the determinant of R C ST U in terms of the determinants of R, T , and T C T U R1 ST or the determinants of R, T , and T 1 C U R1 S and give the ordinary or generalized inverse of R C ST U in terms of the ordinary or generalized inverses of R and T C T U R1 ST (or T C T U R ST ) or the ordinary or generalized inverses of R, T , and T 1 C U R1 S (or T C T T U R ST T ). The corollary of the following theorem gives a comparable expression for the rank of R C ST U . Theorem 18.5.1. For any n q matrix R, n m matrix S, m p matrix T , and p q matrix U , rank.T / C rank.R C RR ST U R R/ D rank.R/ C rank.T C T U R RR ST /:

R RR ST Proof. Let A D . Then, observing that T T U R R I 0 R C RR ST U R R RR ST A D ; U R R I 0 T I 0 R RR ST A D T U R I 0 T CT U R RR ST and making use of Lemma 8.5.2 and Corollary 9.6.2, we find that

(5.1)

18.5 Some Results on Ranks

445

R C RR ST U R R RR ST rank.A/ D rank 0 T D rank.T / C rank.R C RR ST U R R/; R RR ST rank.A/ D rank 0 T CT U R RR ST D rank.R/ C rank.T C T U R RR ST /; implying that rank.T /Crank.RCRR ST U R R/ D rank.R/Crank.T CT U R RR ST /: Or, alternatively, Theorem 18.5.1 can be proved by applying result (9.6.1)— observing that C.T U R R/ C.T / and R.RR ST / R.T / and that C.RR ST / C.R/ and R.T U R R/ R.R/, we find that rank.T / C rank.R C RR ST U R R/ D rank.A/ D rank.R/ C rank.T C T U R RR ST /: Q.E.D. Corollary 18.5.2. Let R represent an n q matrix, S an n m matrix, T an m p matrix, and U a p q matrix. If R.ST U / R.R/ and C.ST U / C.R/, then rank.R C ST U / D rank.R/ C rank.T C T U R ST / rank.T /:

(5.2)

Proof. Suppose that R.ST U / R.R/ and C.ST U / C.R/. Then, in light of Lemma 9.3.5, R C RR ST U R R D R C RR ST U D R C ST U : Now, let Q D T C T U R ST and H D T C T U R RR ST , and observe that

Q H D T U .I R R/R ST :

Clearly, to complete the proof, it suffices [in light of result (5.1)] to show that rank.H/ D rank.Q/. In light of result (2.29), .I QQ /.Q H/ D .I QQ /T U .I R R/R ST D 0; implying (in light of Lemma 9.3.5) that C.Q H/ C.Q/, so that Q H D QK for some matrix K. Moreover, upon substituting R RR (which, like R itself, is a generalized inverse of R) for R in result (2.29), we find that

446


.I HH /T U .I R R/ D .I HH /T U .I R RR R/ D 0; implying that .I HH /.Q H/ D .I HH /T U .I R R/R ST D 0 and hence (in light of Lemma 9.3.5) that C.Q H/ C.H/, so that Q H D HL for some matrix L. Thus, H D Q .Q H/ D Q.I K/;

Q D H C .Q H/ D H.I C L/;

implying that C.H/ C.Q/ and C.Q/ C.H/ and hence that C.H/ D C.Q/, and it follows that rank.H/ D rank.Q/: Q.E.D. In connection with Corollary 18.5.2, note that equality (5.2) [which is applicable if R.ST U / R.R/ and C.ST U / C.R/] is equivalent to the equality rank.R/ rank.R C ST U / D rank.T / rank.T C T U R ST /:

(5.3)

Thus, if R.ST U / R.R/ and C.ST U / C.R/, then the difference in rank between R and RCST U is the same as that between T and T CT U R ST . Note also that this difference in rank is nonnegative [as is evident, e.g., upon observing that T C T U R ST D T .I C U R ST /]. b. Lower (and upper) bounds on the rank of a sum of two matrices Let A and B represent m n matrices. In Section 4.5, it was established that rank.A C B/ rank.A; B/ rank.A/ C rank.B/; A rank.A C B/ rank rank.A/ C rank.B/: B

(5.4) (5.5)

A Thus, rank.A C B/ is bounded from above by rank.A; B/, by rank , and B by rank.A/ C rank.B/. Further, in Section 17.2, it was established that equality holds in the inequality rank.A; B/ rank.A/ C rank.B/ if and only if C.A/ and A C.B/ are essentially disjoint, and that equality holds in the inequality rank B rank.A/ C rank.B/ if and only if R.A/ and R.B/ are essentially disjoint. In this section, these results are augmented with results on lower bounds for rank.A C B/ and with results on the conditions under which these lower bounds are attained [and alsowith results on the conditions under which the upper bounds A rank.A; B/ and rank are attained]. These additional results are derived from B the following theorem.


447

Theorem 18.5.3. Let A and B represent m n matrices. And let c D dimŒC.A/ \ C.B/, d D dimŒR.A/ \ R.B/, and A A A 0 HD I (5.6) ŒI .A; B/ .A; B/: B B 0 B Then, rank.A C B/

A D rank.A; B/ C rank rank.A/ rank.B/ C rank.H/ B D rank.A; B/ d C rank.H/ A D rank c C rank.H/ B D rank.A/ C rank.B/ c d C rank.H/:

(5.7) (5.8) (5.9) (5.10)

Proof. Equality (5.7) is the special case of equality (17.5.5) obtained by A 0 In in place of B, and in place of putting .Im ; Im / in place of A, 0 B I n A A 0 A 0 In D , C and by observing that .Im ; Im / D .A; B/, B In 0 B 0 B A 0 A 0 In .Im ; Im / D A C B, and rank D rank.A/ C rank.B/. 0 B In 0 B Further, equalities (5.8) and (5.9) are obtained from equality (5.7) by substitutA ing for rank or rank.A; B/, respectively, on the basis of formula (17.4.11) B or (17.4.10). Similarly, equality (5.10) isobtained from equality (5.8) or (5.9) A by substituing for rank.A; B/ or rank , respectively, on the basis of formula B (17.4.10) or (17.4.11). Q.E.D. Lower and upper bounds for the rank of the sum of two matrices A and B can be obtained from the following theorem, which gives various inequalities relating A rank.A C B/ to rank.A/ C rank.B/ and to rank.A; B/ or rank and which B also gives the conditions under which these inequalities hold as equalities. Theorem 18.5.4. Let A and B represent m n matrices. And let c D dimŒC.A/ \ C.B/, d D dimŒR.A/ \ R.B/, and A A A 0 HD I ŒI .A; B/ .A; B/: (5.11) B B 0 B (1) Then,

448


rank.A/ C rank.B/ c d D rank.A; B/ d

(5.12a)

rank.A C B/ rank.A; B/ D rank.A/ C rank.B/ c rank.A/ C rank.B/;

(5.12b) (5.12c)

with equality holding under the following conditions: rank.A C B/ D rank.A; B/ d Œor, equivalently, rank.A C B/ D rank.A/ C rank.B/ c d , rank.H/ D 0I rank.A C B/ D rank.A; B/ , rank.H/ D d I

(5.13) (5.14)

rank.A; B/ D rank.A/ C rank.B/ Œor, equivalently, rank.A/ C rank.B/ c D rank.A/ C rank.B/ , c D 0I

(5.15)

(2) Similarly, rank.A/ C rank.B/ c d A D rank c B

A rank.A C B/ rank D rank.A/ C rank.B/ d B rank.A/ C rank.B/;

(5.16a) (5.16b) (5.16c)

with equality holding under the following conditions: A rank.A C B/ D rank c B Œor, equivalently, rank.A C B/ D rank.A/ C rank.B/ c d , rank.H/ D 0I A rank.A C B/ D rank , rank.H/ D cI B

(5.17) (5.18)

A D rank.A/ C rank.B/ rank B

Œor, equivalently, rank.A/ C rank.B/ d D rank.A/ C rank.B/ , d D 0I

(5.19)

Proof. (1) That rank.A/ C rank.B/ c d D rank.A; B/ d is an immediate consequence of result (17.4.10). That rank.A; B/ d rank.A C B/, with equality holding if and only if rank.H/ D 0, is evident from result (5.8).


449

That rank.A C B/ rank.A; B/ was established earlier (in Lemma 4.5.8). That rank.A C B/ D rank.A; B/ if and only if rank.H/ D d is clear from result (5.8). That rank.A; B/ D rank.A/ C rank.B/ c was established earlier [as result (17.4.10)]. And that rank.A/ C rank.B/ c rank.A/ C rank.B/, with equality holding if and only if c D 0, is obvious. (2) The proof of Part (2) of Theorem 18.5.4 is analogous to that of Part (1). Q.E.D. Together, results (5.12) and (5.16) lead to the following corollary. Corollary 18.5.5. For any matrices A and B (of the same size), rank.A/ C rank.B/ c d rank.A C B/ rank.A/ C rank.B/ max.c; d /; (5.20) where c D dimŒC.A/ \ C.B/ and d D dimŒR.A/ \ R.B/. The matrix H, defined by expression (5.11) [or by expression (5.6)], plays a prominent role in Theorems 18.5.4 and 18.5.3. The following theorem gives some results on the rank of this matrix. Theorem 18.5.6. Let A and B represent m n matrices. And let c D dimŒC.A/ \ C.B/, d D dimŒR.A/ \ R.B/, and A 0 A A ŒI .A; B/ .A; B/: HD I 0 B B B Then, A 0 rank.H/ D c rank rank.A C B/ c; B 0 rank.H/ D d Œrank.A; B/ rank.A C B/ d: A Proof. The equalities rank.H/ D c rank rank.A C B/ and B rank.H/ D d Œrank.A; B/ rank.A C B/ are equivalent to (5.9) and equalities A (5.8). Further, since [according to results (5.5) and (5.4)] rank rank.ACB/ B A and rank.A; B/ rank.ACB/, it is clear that c rank rank.ACB/ c B and that d Œrank.A; B/ rank.A C B/ d . Q.E.D. In connection with Theorem 18.5.6, note that the condition rank.H/ D d , which appears in Part (1) of Theorem 18.5.4 [as a necessary and sufficient condition for rank.A C B/ D rank.A; B/, is satisfied if d D 0—if d D 0, then rank.H/ D 0 D d . And, similarly, the condition rank.H/ D c, which appears in Part (2) of Theorem 18.5.4 [as a necessary and sufficient condition for rank.A C B/ D A rank ], is satisfied if c D 0—if c D 0, then rank.H/ D 0 D c. B

450


c. Rank additivity The following theorem gives necessary and sufficient conditions for rank additivity, that is, for the rank of the sum of two matrices to equal the sum of their ranks. Theorem 18.5.7. For any two matrices A and B (of the same size), rank.A C B/ D rank.A/ C rank.B/ if and only if R.A/ and R.B/ are essentially disjoint and C.A/ and C.B/ are essentially disjoint. Proof. Let c D dimŒC.A/ \ C.B/ and d D dimŒR.A/ \ R.B/. Suppose that R.A/ and R.B/ are essentially disjoint and C.A/ and C.B/ are essentially disjoint (or, equivalently, that c D 0 and d D 0). Then, it follows from Corollary 18.5.5 that rank.A/ C rank.B/ rank.A C B/ rank.A/ C rank.B/ and hence that rank.A C B/ D rank.A/ C rank.B/. Conversely, suppose that rank.A C B/ D rank.A/ C rank.B/. Then, in light of inequalities (5.4) and (5.5), we have that rank.A; B/ D rank.A/ C rank.B/ and A rank D rank.A/ C rank.B/ and hence [according to results (17.4.12) and B (17.4.13)] that c D 0 and d D 0. Q.E.D.

Exercises Section 18.1 1. Provide an alternative ˇ of formula (1.5) by applying the results of ˇ derivation ˇ R ST ˇ ˇ (and by proceeding as in the proof of Theorem Theorem 13.3.8 to ˇˇ TU T ˇ 18.1.1). 2. Let R represent an n n matrix, S an n m matrix, T an m m matrix, and U an m n matrix. Use Theorem 18.1.1 (or Corollary 18.1.2 or 18.1.4) to show that if R is nonsingular, then jR C ST U j D jRjjIm C U R1 ST j D jRjjIm C T U R1 Sj: 3. Let A represent an n n symmetric nonnegative definite matrix. Show that if I A is nonnegative definite and if jAj D 1, then A D I. 4. Show that, for any n n symmetric nonnegative definite matrix B and for any n n symmetric matrix C such that C B is nonnegative definite, jCj jC Bj; with equality holding if and only if C is singular or B D 0.

Exercises

451

5. Let A represent a symmetric nonnegative definite matrix that has been partitioned as T U AD ; U0 W where T is of dimensions m m and W of dimensions n n (and where U is of dimensions m n). Define Q D W U 0 T U (which is the Schur complement of T ). (a) Using Theorem 14.8.4 and the result of Exercise 14.33 (or otherwise), show that jW j jU 0 T U j; with equality holding if and only if W is singular or Q D 0. (b) Suppose that n D m and that T is nonsingular. Show that jW jjT j jU j2 ; with equality holding if and only if W is singular or rank.A/ D m. (c) Suppose that n D m and that A is positive definite. Show that jW jjT j > jU j2 : 6. Show that, for any n p matrix X and any symmetric positive definite matrix W, (E.1) jX0 W XjjX0 W 1 Xj jX0 Xj2 : [Hint. Begin by showing that the matrices X0 X.X0 W X/ X0 X and X0 W 1 X X0 X.X0 W X/ X0 X are symmetric and nonnegative definite.] 7. (a) Show that, for any n n skew-symmetric matrix C, jIn C Cj 1; with equality holding if and only if C D 0. (b) Generalize the result of Part (a) by showing that, for any n n symmetric positive definite matrix A and any n n skew-symmetric matrix B, jA C Bj jAj; with equality holding if and only if B D 0. Section 18.2 8. (a) Let R represent an n n nonsingular matrix, and let B represent an n n matrix of rank one. Show that R C B is nonsingular if and only if tr.R1 B/ ¤ 1, in which case .R C B/1 D R1 Œ1 C tr.R1 B/1 R1 BR1:

452


(b) To what does the result of Part (a) simplify in the special case where R D In ? 9. Let R represent an n n matrix, S an n m matrix, T an m m matrix, and U an m n matrix. Suppose that R is nonsingular. Use Theorem 18.2.8 to show (a) that R C ST U is nonsingular if and only if Im C U R1 ST is nonsingular, in which case .R C ST U /1 D R1 R1 ST .Im C U R1 ST /1 U R1; and (b) that R C ST U is nonsingular if and only if Im C T U R1 S is nonsingular, in which case .R C ST U /1 D R1 R1 S.Im C T U R1 S/1 T U R1: [Hint. Reexpress R C ST U as R C ST U D R C .ST /Im U and as R C ST U D R C SIm T U .] 10. Let R represent an n q matrix, S an n m matrix, T an m p matrix, and U a p q matrix. Extend the results of Exercise 9 by showing that if R.ST U / R.R/ and C.ST U / C.R/, then the matrix R R ST .Ip C U R ST / U R and the matrix

R R S.Im C T U R S/ T U R

are both generalized inverses of the matrix R C ST U . 11. Let R represent an n q matrix, S an n m matrix, T an m p matrix, and U a p q matrix. (a) Let a generalized inverse partitioned matrix G represent of the R ST G11 G12 , and partition G as G D (where G11 is of TU T G21 G22 dimensions q n). Use Theorem 9.6.5 to show that G11 is a generalized inverse of the matrix R C ST U . (b) Let ER D I RR, FR D I R R, X D ER ST , Y D T U FR , EY D I Y Y , FX D I X X, Q D T C T U R ST , Z D EY QFX , and Q D FX Z EY . Use the result of Part (a) of Exercise 10.10 to show that the matrix R R ST Q T U R R ST .I Q Q/X ER FR Y .I QQ /T U R C FR Y .I QQ /QX ER is a generalized inverse of the matrix R C ST U .

(E.2)

Exercises

453

(c) Show that if R.T U / R.R/ and C.ST / C.R/, then formula (2.27) for a generalized inverse of R C ST U can be obtained as a special case of formula (E.2). 12. Let A1 ; A2 ; : : : represent a sequence of m n matrices, and let A represent another m n matrix. (a) Using the result of Exercise 6.1 (i.e., the triangle inequality), show that if kAk Ak ! 0, then kAk k ! kAk. (b) Show that if Ak ! A, then kAk k ! kAk (where the norms are the usual norms). 13. Let A represent an n n matrix. Using the results of Exercise 6.1 and of Part (b) of Exercise 12, show that if kAk < 1, then (for k D 0; 1; 2; : : :) k.I A/1 .I C A C A2 C C Ak /k kAkkC1 =.1 kAk/ (where the norms are the usual norms). (Note. It follows from Theorems 18.2.16 and 18.2.19 that if kAk < 1, then I A is nonsingular.) 14. Let A and B represent n n matrices. Suppose that B is nonsingular, and define F D B1 A. Using the result of Exercise 13, show that if kF k < 1, then (for k D 0; 1; 2; : : :) k.B A/1 .B1 C F B1 C F 2 B1 C C F k B1 /k kB1 kkF kkC1 =.1 kF k/ (where the norms are the usual norms). (Note. It follows from Theorems 18.2.18 and 18.2.19 that if kF k < 1, then B A is nonsingular.) Section 18.3 15. Let A represent an n n symmetric nonnegative definite matrix, and let B represent an n n matrix. Show that if B A is nonnegative definite (in which case B is also nonnegative definite), then R.A/ R.B/ and C.A/ C.B/. Section 18.4 16. Let A represent an n n symmetric idempotent matrix, and let B represent an n n symmetric nonnegative definite matrix. Show that if I A B is nonnegative definite, then BA D AB D 0. (Hint. Show that A0 .IAB/A D A0 BA, and then consider the implications of this equality.) 17. Let A1 ; : : : ; Ak represent n n symmetric matrices, and let A D A1 C C Ak . Suppose that A is idempotent. Suppose further that A1 ; : : : ; Ak1 are idempotent and that Ak is nonnegative definite. Using the result of Exercise

454


16 (or otherwise), show that Ai Aj D 0 (for j ¤ i D 1; : : : ; k), that Ak is P idempotent, and that rank.Ak / D rank.A/ k1 iD1 rank.Ai /. 18. Let A1 ; : : : ; Ak represent nn symmetric matrices, and define A D A1 C C Ak . Suppose that A is idempotent. Show that if A1 ; : : : ; Ak are nonnegative P definite and if tr.A/ kiD1 tr.A2i /, then Ai Aj DP0 (for j ¤ i D 1; : : : ; k) and A1 ; : : : ; Ak are idempotent. (Hint. Show that i;j ¤i tr.Ai Aj / 0 and then make use of Corollary 14.7.7.) 19. Let A1 ; : : : ; Ak represent nn symmetric matrices such that A1 C CAk D I. Show that if rank.A1 /C Crank.Ak / D n, then, for any (strictly) positive scalars c1 ; : : : ; ck , the matrix c1 A1 C C ck Ak is positive definite. 20. Let A1 ; : : : ; Ak represent n n symmetric idempotent matrices such that Ai Aj D 0 for j ¤ i D 1; 2; : : : ; k. Show that, for any (strictly) positive P scalar c0 and any nonnegative scalars c1 ; : : : ; ck , the matrix c0 I C kiD1 ci Ai is positive definite (and hence nonsingular), and k k 1 X X c0 I C ci A i D d0 I C di A i ; iD1

iD1

where d0 D 1=c0 and (for i D 1; : : : ; k) di D ci =Œc0 .c0 C ci /. 21. Let A1 ; : : : ; Ak represent n n symmetric idempotent matrices such that (for j ¤ i D 1; : : : ; k) Ai Aj D 0, and let A represent an n n symmetric idempotent matrix such that (for i D 1; : : : ; k) C.Ai / C.A/. Show that if rank.A1 / C C rank.Ak / D rank.A/, then A1 C C Ak D A. 22. Let A represent an m n matrix and B an n m matrix. If B is a generalized inverse of A, then it follows from Lemma 10.2.6 that rank.I BA/ D n rank.A/. Show that the converse is also true; that is, show that if rank.I BA/ D n rank.A/, then B is a generalized inverse of A. 23. Let A represent the (nn) projection matrix for a subspace U of Rn1 along a subspace V of Rn1 (where U ˚ V D Rn1 ), and let B represent the (n n) projection matrix for a subspace W of Rn1 along a subspace X of Rn1 (where W ˚ X D Rn1 ). (a) Show that A C B is the projection matrix for some subspace L of Rn1 along some subspace M of Rn1 (where L ˚ M D Rn1 ) if and only if BA D AB D 0, in which case L D U ˚ W and M D V \ X. (b) Show that A B is the projection matrix for some subspace L of Rn1 along some subspace M of Rn1 (where L ˚ M D Rn1 ) if and only if BA D AB D B, in which case L D U \ X and M D V ˚ W. [Hint.

Exercises

455

Observe (in light of Theorem 17.6.13 and Lemma 10.1.2) that A B is the projection matrix for some subspace L along some subspace M if and only if I .A B/ D .I A/ C B is the projection matrix for some subspace L along some subspace M, and then make use of Part (a) and Theorem 17.6.10.] 24. (a) Let B represent an n n symmetric matrix, and let W represent an n n symmetric nonnegative definite matrix. Show that W BW BW D W BW , .BW /3 D .BW /2 , trŒ.BW /2 D trŒ.BW /3 D trŒ.BW /4 : (b) Indicate how, in the special case of Theorems 18.4.8 and 18.4.9 where A1 ; : : : ; Ak are symmetric, the conditions V AV AV D V AV and V Ai V Ai V D V Ai V (which appear in both theorems) can be reexpressed by applying the results of Part (a) (of the current exercise). Section 18.5 25. Let R represent an n q matrix, S an n m matrix, T an m p matrix, and U a p q matrix. (a) Show that

R ST rank.R C ST U / D rank TU T

rank.T /:

(E.3)

(b) Let ER D I RR , FR D I R R, X D ER ST , Y D T U FR , EY D IY Y , FX D IX X, Q D T CT U R ST , and Z D EY QFX . Use the result of Part (b) of Exercise 10.10 to show that rank.R C ST U / D rank.R/ C rank.X/ C rank.Y / C rank.Z/ rank.T /: 26. Show that, for any m n matrices A and B, rank.A C B/ jrank.A/ rank.B/j: 27. Show that, for any n n symmetric nonnegative definite matrices A and B, A C.A C B/ D C.A; B/; R.A C B/ D R ; B A : rank.A C B/ D rank.A; B/ D rank B 28. Let A and B represent m n matrices.

456


(a) Show that (1) C.A/ C.ACB/ if and only if rank.A; B/ D rank.ACB/ A and (2) R.A/ R.A C B/ if and only if rank D rank.A C B/. B (b) Show that (1) if R.A/ and R.B/ are essentially disjoint, then C.A/ C.A C B/ and (2) if C.A/ and C.B/ are essentially disjoint, then R.A/ R.A C B/. 29. Let A and B represent m n matrices. Show that each of the following five conditions is necessary and sufficient for rank additivity [i.e., for rank.A C B/ D rank.A/ C rank.B/]: A (a) rank.A; B/ D rank D rank.A/ C rank.B/; B (b) rank.A/ D rankŒA.I B B/ D rankŒ.I BB /A; (c) rank.B/ D rankŒB.I A A/ D rankŒ.I AA /B; (d) rank.A/ D rankŒA.I B B/ and rank.B/ D rankŒ.I AA /B; (e) rank.A/ D rankŒ.I BB /A and rank.B/ D rankŒB.I A A/. 30. Let A and B represent m n matrices. And let A 0 A A ŒI .A; B/ .A; B/: HD I 0 B B B (a) Using result (5.7) (with B in place of B), show that rank.A B/ D rank.A/ rank.B/ C Œrank.A; B/ rank.A/ A C rank rank.A/ C rank.H/: B (b) Show that A and B are rank subtractive [in the sense that B/ D rank.A A rank.A/ rank.B/] if and only if rank.A; B/ D rank D rank.A/ B and H D 0. A D rank.A/, then (1) .A; 0/ and (c) Show that if rank.A; B/ D rank B A A are generalized inverses of and .A; B/, respectively, and (2) 0 B A 0 0 A D .A; 0/ and .A; B/ D , HD . for 0 0 BA B B B (d) Show that each of the following three conditions is necessary and sufficient for rank subtractivity [i.e., for rank.A B/ D rank.A/ rank.B/]: A (1) rank.A; B/ D rank D rank.A/ and BA B D B; B

Exercises

457

(2) C.B/ C.A/, R.B/ R.A/, and BA B D B; (3) AA B D BA A D BA B D B. (e) Using the result of Exercise 29 (or otherwise), show that rank.A B/ D rank.A/ rank.B/ if and only if rank.A B/ D rankŒA.I B B/ D rankŒ.I BB /A. 31. Let A1 ; : : : ; Ak represent m n matrices. Adopting the terminology of Exercise 17.6, use Part (a) of that exercise to show that if R.A1 /; : : : ; R.Ak / are independent and C.A1 /; : : : ; C.Ak / are independent, then rank.A1 C C Ak / D rank.A1 / C C rank.Ak /. 32. Let T represent an m p matrix, U an m q matrix, V an n p matrix, and W an nq matrix, and define Q D W V T U . Further, let ET D I T T , FT D I T T , X D ET U , and Y D V FT . (a) Show that rank

T U V W

D rank.T / C rank

0 X : Y Q

(E.4)

(b) Show that 0 V rank U T

V T U Y D rank.T / C rank : X 0

0 V [Hint. Observe (in light of Lemma 8.5.1) that rank D U T T U rank , and make use of Part (a).] V 0 (c) Show that 0 V rank D rank.T / C rank.X/ C rank.Y / C rank.EY V T U FX /; U T where EY D I Y Y and FX D I X X. [Hint. Use Theorem 17.2.17 in combination with Part (b).] (d) Show that

T U rank V W

D rank.T / C rank.Q/ C rank.A/ C rank.B/ C rankŒ.I AA /XQ Y .I B B/;

where A D X.I Q Q/ and B D .I QQ /Y . [Hint. Use Part (c) in combination with Part (a).]

458


33. Let R represent an n q matrix, S an n m matrix, T an m p matrix, and U a p q matrix, and define Q D T C T U R ST . Further, let ER D I RR, FR D I R R, X D ER ST , Y D T U FR , A D X.I Q Q/, and B D .I QQ /Y . Use the result of Part (d) of Exercise 32 in combination with the result of Part (a) of Exercise 25 to show that rank.R C ST U / D rank.R/ C rank.Q/ rank.T / C rank.A/ C rank.B/ C rankŒ.I AA /XQ Y .I B B/; thereby generalizing formula (5.2).

Bibliographic and Supplementary Notes §2. The presentation in Subsection a (and in the first part of Subsection d) is similar to that of Miller (1987, pp. 10–11). The material in Subsection e (and in Exercises 10 and 11) includes a number of the results of Harville (1997). The presentation in Subsection f requires of the reader some familiarity with basic results on sequences of scalars and on the infinite series generated by such sequences. For a more general and more complete treatment of the subject of Subsection f, refer, e.g., to Faddeeva (1959, sec. 5) or to Golub and Van Loan (1989, sec. 2.3). §3. Exercise 15 is based on Lemma 2.4 of Milliken and Akdeniz (1977). §4. The result of Exercise 16 is given by Loynes (1966) and is referred to by Searle (1971, p. 61) as Loynes’ lemma. §5. The results of Corollary 18.5.2 and of Exercises 25 and 33 were given by Harville (1997). The results covered in Subsections b and c (and in Exercises 28, 29, 30, and 32) are adaptations of results presented by Marsaglia and Styan (1974) in their Sections 4, 5, 7, and 8.

19 Minimization of a Second-Degree Polynomial (in n Variables) Subject to Linear Constraints

The subject of this chapter is the minimization of a quadratic form, or, more generally, a second-degree polynomial, in some number, say n, of variables that may be subject to linear constraints. Special cases of this minimization problem are encountered in various areas of statistics and in many related disciplines. In particular, they are encountered in estimating the parameters of a linear statistical model. One approach to the estimation goes by the acronym BLUE (for best linear unbiased estimation); in this approach, consideration is restricted to estimators that are linear (i.e., that are expressible as linear combinations of the data) and that are unbiased (i.e., whose “expected values” equal the parameters), and the estimator of each parameter is chosen to have minimum variance among all estimators that are linear and unbiased. The minimization problem encompassed in this approach can be formulated as one of minimizing a quadratic form (in the coefficients of the linear combination) subject to linear constraints—the constraints arise from the restriction to unbiased estimators. Another approach is to regard the estimation of the parameters as a least squares problem—the least squares problem was considered (from a geometrical perspective) in Chapter 12. This approach consists of minimizing the sum of the squared deviations between the data points and the linear combinations of the parameters (in the model) that correspond to those points. The sum of the squared deviations is a second-degree polynomial (in the parameters). In some cases, the parameters of the model may be subject to linear constraints—knowledge of the process that gave rise to the data may suggest such constraints (along with the other aspects of the model). In the presence of such constraints, the minimization problem encompassed in the least squares appraoch is one of minimizing a second-degree polynomial (in the parameters) subject to linear constraints (on the parameters).

460

19 Minimization of a Second-Degree Polynomial

19.1 Unconstrained Minimization As a preliminary to considering the minimization of a second–degree polynomial (in n variables) subject to linear constraints, it is instructive to consider the unconstrained minimization of a second–degree polynomial. a. Basic results Let V D fvij g represent an n n symmetric nonnegative definite matrix, and let b D fbi g represent an n 1 vector such that b 2 C.V /. Further, let a D .a1 ; : : : ; an /0 represent a (column) vector of n variables, and consider the minimization of the function f of a defined (for a 2 Rn ) by f .a/ D a0 V a 2b0 a: Note that f .a/ can be reexpressed as f .a/ D a0 V a 2a0 b D a0 V a b0 a a0 b or in nonmatrix form as the second-degree polynomial X X f .a/ D vij ai aj 2 bi ai : i;j

i

For f to have a minimum at a point a , it is necessary that a be a stationary point of f , that is, it is necessary that @f =@a D 0 at a D a (e.g., Magnus and Neudecker 1988, pp. 119–120). As a consequence of formulas (15.3.5) and (15.3.7), we have that @f D 2V a 2b D 2.V a b/: (1.1) @a Thus, for f to have a minimum at a point a , it is necessary that V a D b or, equivalently, that a be a solution to the linear system V a D b (in a). The following theorem makes a stronger claim—for f to have a minimum at a point a , it is sufficient, as well as necessary, that V a D b. Theorem 19.1.1. Let a represent an n 1 vector of (unconstrained) variables, and define f .a/ D a0 V a 2b0 a, where V is an n n symmetric nonnegative definite matrix and b is an n1 vector such that b 2 C.V /. Then, the linear system V a D b (in a) is consistent. Further, f .a/ attains its minimum value at a point a if and only if a is a solution to V a D b, in which case f .a / D b0 a D a0 b. Theorem 19.1.1 is a special case of the following result (that where s D 1). Theorem 19.1.2. Let A represent an ns matrix of (unconstrained) variables, and define F .A/ D A0 V AB0 AA0 B, where V is an nn symmetric nonnegative definite matrix and B is an n s matrix such that C.B/ C.V /. Then, the linear system V A D B (in A) is consistent. Further, F .A/ attains its minimum value at a matrix A (in the sense that F .A/ F .A / is nonnegative definite for every A 2 Rns ) if and only if A is a solution to V A D B, in which case F .A / D B0 A D A0 B.

19.1 Unconstrained Minimization

461

Proof (of Theorem 19.1.2). Since C.B/ C.V /, the consistency of the linear system V A D B follows from Theorem 7.2.1. Now, let A represent any solution to V A D B, and let A represent an arbitrary n s matrix. Then, A0 V A D ŒA C .A A /0 V ŒA C .A A / D A0 V A C A0 V .A A / C .A A /0 V A C .A A /0 V .A A / D A0 V A C B0 .A A / C .A A /0 B C .A A /0 V .A A /: Thus, F .A/ F .A / D A0 V A A0 V A B0 .A A / .A A /0 B D .A A /0 V .A A /;

(1.2)

implying (in light of Theorem 14.2.9) that F .A/ F .A / is nonnegative definite. Further, let A0 represent any n s matrix such that F .A/ F .A0 / is nonnegative definite for every A 2 Rns. Then, F .A / F .A0 / is nonnegative definite, or, equivalently, F .A0 / F .A / {which D ŒF .A / F .A0 /} is nonpositive definite, and, since F .A0 / F .A / is nonnegative definite, it follows from Lemma 14.2.2 that F .A0 / F .A / D 0 and hence [upon applying equality (1.2) with A D A0 ] that .A0 A /0 V .A0 A / D 0. Thus, as a consequence of Corollary 14.3.11, V .A0 A / D 0, implying that V A0 D V A D B and hence that A0 is a solution to V A D B. Finally, observe that F .A / D A0 V A B0 A A0 B D A0 B B0 A A0 B D B0 A and similarly that F .A / D .V A /0 A B0 A A0 B D B0 A B0 A A0 B D A0 B: Q.E.D. Note, in connection with Theorem 19.1.1, that in the special case where V is positive definite (and hence nonsingular), the linear system V a D b has a unique solution, namely, a D V 1 b, and that V 1 b is the only value of a at which f .a/ attains its minimum value. Similarly (and more generally), note, in connection with Theorem 19.1.2, that, in the special case where V is positive definite, the linear system V A D B has a unique solution, namely, A D V 1 B, and that there is only one n s matrix A such that F .A/ F .A / is nonnegative definite for every A 2 Rns, namely, A D V 1 B. b. Least squares approach to derivation of main result Let a represent an n 1 vector of (unconstrained) variables, and define f .a/ D a0 V a 2b0 a, where V is an n n symmetric nonnegative definite matrix and b is an n 1 vector such that b 2 C.V /. Further, let a represent any solution

462


to the (consistent) linear system V a D b. It was established in Subsection a (specifically in Theorem 19.1.1) that f .a/ attains its minimum value at a point a if and only if a is a solution to the (consistent) linear system V a D b, in which case f .a / D b0 a . This result can be derived from the results of Section 12.4 (on the least squares problem), thereby providing an alternative to the proof given in Subsection a—actually, the proof given in Subsection a is for the more general result given by Theorem 19.1.2. Let R represent a matrix such that V D R0 R—the existence of such a matrix is guaranteed by Corollary 14.3.8—let t represent an n 1 vector such that b D V t, and let y D Rt. Then, f .a/ D a0 V a 2t 0 V a D a0 R0 Ra 2y 0 Ra D .y Ra/0 .y Ra/ y 0 y: Since y 0 y does not depend on a, the minimization of f .a/ is equivalent to the minimization of .y Ra/0 .y Ra/. Moreover, since R0 y D R0 Rt D V t D b, the linear system V a D b is equivalent to the linear system R0 Ra D R0 y of normal equations. Thus, it follows from Theorem 12.4.3 that .y Ra/0 .y Ra/ attains its minimum value at a point a if and only if a is a solution to V a D b, in which case .y Ra /0 .y Ra / D y 0 y y 0 Ra D y 0 y b0 a : We conclude that f .a/ attains its minimum value at a point a if and only if a is a solution to V a D b, in which case f .a / D .y Ra /0 .y Ra / y 0 y D b0 a : c. Use of convexity to establish main result Let a represent an n 1 vector of (unconstrained) variables, and define f .a/ D a0 V a 2b0 a, where V is an n n symmetric nonnegative definite matrix and b is an n 1 vector such that b 2 C.V /. Further, let a represent any solution to the (consistent) linear system V a D b. Then, f .a/ attains its minimum value at a D a , as was established in Subsection a and as was reconfirmed (via a second approach) in Subsection b. There is yet another way of verifying that f .a/ attains its minimum value at a D a . Using the results of Section 15.3, we find that @2 f D 2V : @a@a0 Thus, the Hessian matrix of f is nonnegative definite (for all a), and consequently f is a convex function—refer, for example, to Theorem 7.7 of Magnus and Neudecker (1988). And, recalling (from Subsection a) that a is a stationary point of f , it follows from a well-known result on convex functions [which is Theorem 7.8 of Magnus and Neudecker (1988)] that f .a/ attains its minimum value at a .

19.2 Constrained Minimization

463

19.2 Constrained Minimization a. Basic results Let a D .a1 ; : : : ; an /0 represent an n 1 vector of variables, and impose on a the constraint X0 a D d, where X D fxij g is an n p matrix and d D fdj g is a p 1 vector such that d 2 C.X0 /. Further, define (for a 2 Rn ) f .a/ D a0 V a 2b0 a, where V is an n n symmetric nonnegative definite matrix and b is an n 1 vector such that b 2 C.V; X/. And consider the constrained (by X0 a D d) minimization of f .a/. Note that the (vector-valued) constraint X0 a D d can be rewritten as p scalarvalued constraints .j D 1; : : : ; p/; xj0 a D dj where xj D .x1j ; : : : ; xnj /0 is the j th column of X, or in completely nonmatrix notation as n X xij ai D dj .j D 1; : : : ; p/: iD1

Letting r D frj g represent an arbitrary p 1 vector, the Lagrangian function for our constrained minimization problem can be expressed as g.a/ D f .a/ 2r 0 .d X0 a/ or, equivalently, as g.a/ D f .a/ C

p X

.2rj /.dj xj0 a/:

j D1

The p scalars 2r1 ; : : : ; 2rp are the Lagrange multipliers. (Expressing the Lagrangian function so that the vector of Lagrange multipliers is represented by 2r rather than by r will be convenient in what follows.) Suppose now that rank.X/ D p. Then, upon applying Lagrange’s theorem [which is Theorem 7.10 of Magnus and Neudecker (1988)], we find that, for f .a/ to have a minimum at a point a under the constraint X0 a D d, it is necessary that there exist a p 1 vector r such that, for r D r , a is a stationary point of g (i.e., such that, for r D r , @g=@a D 0 at a D a ). Of course, if is also necessary that a satisfy the constraint (i.e., that X0 a D d). Making use of result (1.1) and of formula (15.3.5), we find that @f @.r 0 X0 a/ @g D C2 D 2.V a b C Xr/: @a @a @a Thus, for f to have a minimum at a point a under the constraint X0 a D d, it is necessary that there exist a p 1 vector r such that V a C Xr D b (and that X0 a D d) or, equivalently, it is necessary that there exist a p 1 vector r such

464


that a and r are, respectively, the first and second parts of a solution to the linear system comprising the two equations V a C Xr D b; 0

XaDd

(2.1) (2.2)

(in a and r). Note that equations (2.1) and (2.2) can be reformulated as V X a b D : X0 0 r d Are the conditions obtained by applying Lagrange’s theorem sufficient as well as necessary for f .a/ to have a minimum at a point a under the constraint X0 a D d? And does their necessity extend to the degenerate case where rank.X/ < p? The following theorem answers these questions (in the affirmative). Theorem 19.2.1. Let a represent an n 1 vector of variables; let X represent an n p matrix and d a p 1 vector such that d 2 C.X0 /; and define f .a/ D a0 V a 2b0 a, where V is an n n symmetric nonnegative definite matrix and b is an n 1 vector such that b 2 C.V; X/. Then, the linear system V X a b D (2.3) X0 0 r d (in a and a p 1 vector r) is consistent, and the values of V a and Xr are invariant to the choice of solution to this linear system. Further, f .a/ attains its minimum value at a point a under the constraint X0 a D d if and only if a is the first part of some solution to linear system (2.3). And, for the first and second parts a and r of any solution to linear system (2.3), f .a / D b0 a d0 r D a0 b r0 d:

(2.4)

Theorem 19.2.1 is a special case of the following result (that where s D 1). Theorem 19.2.2. Let A represent an n s matrix of variables; let X represent an n p matrix and D a p s matrix such that C.D/ C.X0 /; and define F .A/ D A0 V A B0 A A0 B, where V is an n n symmetric nonnegative definite matrix and B is an ns matrix such that C.B/ C.V; X/. Then, the linear system V X A B D (2.5) X0 0 R D (in A and a p s matrix R) is consistent, and the values of V A and XR are invariant to the choice of solution to this linear system. Further, an n s matrix A satisfies the constraint X0 A D D and is such that F .A/ F .A / is nonnegative definite for all A satisfying X0 A D D if and only if A is the first part of some solution to linear system (2.5). And, for the first and second parts A and R of any solution to linear system (2.5), F .A / D B0 A D 0 R D A0 B R0 D:

(2.6)


465

Proof (of Theorem 19.2.2). There exists a matrix L such that V D L0 L. And there exist matrices T and U such that B D V T C XU or, equivalently, such that B D L0 LT C XU . Thus, the consistency of linear system (2.5) follows from Corollary 7.4.9. Now, let A and R represent the first and second parts of any solution to linear system (2.5), and let A represent an arbitrary n s matrix satisfying X0 A D D. Then, clearly, X0 A D D. And .A A /0 V A D .A A /0 .B XR / D .A A /0 B .X0 A/0 R C .X0 A /0 R D .A A /0 B D 0 R C D 0 R D .A A /0 B; and consequently A0 V A D ŒA C .A A /0 V ŒA C .A A / D A0 V A C .A A /0 V A C Œ.A A /0 V A 0 C .A A /0 V .A A / D A0 V A C .A A /0 B C B0 .A A / C .A A /0 V .A A /: Thus, F .A/ F .A / D A0 V A A0 V A B0 .A A / .A A /0 B D .A A /0 V .A A /;

(2.7)

implying (in light of Theorem 14.2.9) that F .A/ F .A / is nonnegative definite. Further, let A0 represent any n s matrix that satisfies the constraint X0 A D D and is such that F .A/ F .A0 / is nonnegative definite for all A satisfying X0 A D D. Then, F .A /F .A0 / is nonnegative definite, or, equivalently, F .A0 / F .A / {which equals ŒF .A / F .A0 /} is nonpositive definite, and, since F .A0 /F .A / is nonnegative definite, it follows from Lemma 14.2.2 that F .A0 / F .A / D 0 and hence [upon applying equality (2.7) with A D A0 ] that .A0 A /0 V .A0 A / D 0. Thus, as a consequence of Corollary 14.3.11, V .A0 A / D 0, or, equivalently, (2.8) V A0 D V A ; implying that V A0 C XR D V A C XR D B 0

and hence (since X A0 D D) that A0 and R are the first and second parts of a solution to linear system (2.5). Next, let A1 and R1 represent the first and second parts of a possibly different (from A and R ) solution to linear system (2.5). Then, X0 A1 D D, and (in light of what has already been established) F .A/ F .A1 / is nonnegative definite for every A satisfying X0 A D D. Thus, as a consequence of result (2.8), we have that

466


V A1 D V A and that XR1 D B V A1 D B V A D XR ; implying that V A and XR are invariant to the choice of solution to linear system (2.5). Finally, observe that F .A / D A0 V A B0 A A0 B D A0 .B XR / B0 A A0 B D .X0 A /0 R B0 A D D 0 R B0 A and, similarly, that F .A / D .V A /0 A B0 A A0 B D .B XR /0 A B0 A A0 B D R0 X0 A A0 B D R0 D A0 B: Q.E.D. b. Interpretation of the Lagrange multipliers (in the special case where X is of full column rank) Let a represent an n 1 vector of variables; impose on a the constraint X0 a D d, where X is an n p matrix and d is a p 1 vector such that d 2 C.X0 /; define f .a/ D a0 V a 2b0 a, where V is an n n symmetric nonnegative definite matrix and b is an n 1 vector such that b 2 C.V; X/; and consider the constrained (by X0 a D d) minimization of f .a/. According to Theorem 19.2.1, f .a/ attains its minimum value at a point a (under the constraint X0 a D d) if and only if a is the first (n 1) part of a solution to the linear system V X a b D : (2.9) X0 0 r d What (if any) significance can be attributed to the second (p 1) part of a solution to linear system (2.9)? Suppose that rank.X/ D p (i.e., that X is of full column rank). Then, the second part of the solution to linear system (2.9) is unique—if r1 and r2 are the second parts of two solutions to linear system (2.9), then (according to Theorem 19.2.1) Xr1 D Xr2 , or, equivalently, X.r1 r2 / D 0, implying (since, by supposition, the columns of X are linearly independent) that r1 r2 D 0, or, equivalently, that r1 D r2 . Further, defining (for d 2 Rp1 ) h.d/ D

min

fa2Rn W X0 aDdg

f .a/;


467

it follows from well-known results on Lagrange multipliers [described, for example, by Magnus and Neudecker (1988, sec. 7.16)] that @h D 2r ; @d

(2.10)

where r is the (unique) second part of the solution to linear system (2.9). Thus, the elements of the vector 2r indicate how h is affected by small changes in the elements of d. An alternative way of arriving at formula (2.10) isby direct differentia V X tion. Let G represent a generalized inverse of , and partition G as X0 0 G11 G12 b G D (where G11 is of dimensions n n). Then, G D G21 G22 d G11 b C G12 d is a solution to linear system (2.9). Moreover, since [in light of G21 b C G22 d V X V X 0 0 b the symmetry of D ] G , G is a generalized inverse of X0 0 d X0 0 0 0 G11 b C G21 d is also a solution to linear system (2.9). Further, as a conse0 0 G12 b C G22 d quence of result (2.4), we have that h.d/ D b0 G11 b d0 G21 b b0 G12 d d0 G22 d 0 D b0 G11 b .G21 b/0 d .G12 b/0 d d0 G22 d:

(2.11)

And, using formulas (15.3.5) and (15.3.7) to differentiate expression (2.11), we find that @h 0 0 D G21 b G12 b .G22 C G22 /d @d 0 0 b C G22 d/ D r r D 2r : D .G21 b C G22 d/ .G12 c. Use of convexity to establish main result Let a represent an n 1 vector of variables; impose on a the constraint X0 a D d, where X is an n p matrix and d is a p 1 vector such that d 2 C.X0 /; and define f .a/ D a0 V a 2b0 a, where V is an n n symmetric nonnegative definite matrix and b is an n 1 vector such that b 2 C.V; X/. Further, let a and r represent parts of any solution to the (consistent) linear system the first and second V X a b D . X0 0 r d As discussed in Subsection a, the Lagrangian function for the problem of minimizing f .a/ under the constraint X0 a D d is g.a/ D f .a/ 2r 0 .d X0 a/

468


(where 2r is the vector of Lagrange multipliers). Using the results of Section 15.3, we find that @2 g D 2V : (2.12) @a@a0 Thus, the Hessian matrix of g is nonnegative definite (for all a), and, consequently, g is a convex function (regardless of the value of r)—refer, for example, to Theorem 7.7 of Magnus and Neudecker (1988). And, recalling (from Subsection a) that (for r D r ) @g=@a D 0 at a D a and that X0 a d D 0, it follows from a well-known result on convex functions [which is Theorem 7.13 of Magnus and Neudecker (1988)] that f .a/ attains its minimum value (under the constraint X0 a D d) at a D a .

19.3 Explicit Expressions for Solutions to the Constrained Minimization Problem In Section 19.2a (specifically, in Theorem 19.2.2), we considered a constrained “minimization” problem that includes as a special case the problem of minimizing a second-degree polynomial (in n variables) subject to linear constraints. And we found that this problem “reduces” to the problem of finding the first (n s) part of a solution to a (consistent) linear system of the form A B V X D (3.1) R D X0 0 (in an n s matrix A and a p s matrix R), where V is an n n symmetric nonnegative definite matrix, X is an n p matrix, B is an n s matrix such that C.B/ C.V; X/, and D is a p s matrix such that C.D/ C.X0 /. This linear system comprises the two equations V A C XR D B; 0

X A D D:

(3.2) (3.3)

The objective in the present section is to obtain “explicit” expressions for solutions to linear system (3.1). Let us begin by attacking the important special case where V is positive definite and then the less restrictive special case where C.X/ C.V /, before proceeding to the general case. a. Special case: V positive definite Suppose that the matrix V is positive definite and hence nonsingular. Then, linear system (3.1) is equivalent to the system comprising the two equations A D V 1 B V 1 XR; 0

XV

1

0

XR D X V

1

B D:

(3.4) (3.5)

19.3 Explicit Expressions for Solutions to the Constrained Minimization Problem

469

To verify the equivalence of these two systems, suppose that A and R satisfy equation (3.1), or, equivalently, equations (3.2) and (3.3). Then, clearly, A and R satisfy equation (3.4) and, upon replacing A in equation (3.3) with the right side of equation (3.4), we find that X0 V 1 B X0 V 1 XR D D or, equivalently, that R satisfies equation (3.5). Conversely, suppose that A and R satisfy equations (3.4) and (3.5). Then, clearly, A and R satisfy equation (3.2). Further, X0 A D X0 .V 1 B V 1 XR/ D X0 V 1 B X0 V 1 XR D X0 V 1 B .X0 V 1 B D/ D D; so that A satisfies equation (3.3). The equivalence of linear system (3.1) to the system comprising equations (3.4) and (3.5) suggests a “computational” scheme for solving this system. First, solve equation (3.5) for R. Then, substitute this solution in expression (3.4) to obtain the value of A. If the inverse of V is easy to obtain (as would be the case if, e.g., V were diagonal), then in effect this scheme reduces the problem of solving linear system (3.1) (which comprises n C p equations in n C p rows of unknowns) to one of solving a linear system comprising p equations in p rows of unknowns. According to Corollary 14.11.3, rank.X0 V 1 X/ D rank.X/. Thus, aside from the special case where rank.X/ D p (i.e., where X is of full column rank), equation (3.5) has an infinite number of solutions (for R). However, in light of Theorem 19.2.2 [and of the equivalence of linear system (3.1) and the system comprising equations (3.4) and (3.5)], the matrix XR is invariant to the choice of solution. Further, the right side of equation (3.4) is invariant to the choice of solution (for R), which implies that the first (n s) part of the solution to linear system (3.1) is unique. [The uniqueness of the first part of the solution to linear system (3.1) can also be seen to follow from the invariance of V A to the choice of solution to linear system (3.1)—if A1 and A2 are the first parts of two solutions to linear system (3.1), then V A1 D V A2 , implying that A1 D V 1 V A1 D V 1 V A2 D A2 .] b. Special case: C.X/ C.V / The results obtained in Subsection a [on the solution of linear system (3.1)], which are for the special case where V is positive definite, can be extended in a relatively straightforward way to the less restrictive special case where C.X/ C.V /. Suppose that C.X/ C.V /. Then, the matrices A and R satisfy the two equations (3.2) and (3.3), which compose linear system (3.1), if and only if R satisfies the equation (3.6) X0 V XR D X0 V B D and

A D V B V XR C .I V V /L

(3.7)

470


for some (n s) matrix L. For purposes of verifying this, observe (in light of Lemma 9.3.5 and Lemma 4.2.5) that V V X D X X0 V V D X0 :

(3.8) (3.9)

Observe also that B D V E C XF for some matrices E and F and hence that V V B D V V V E C V V XF D V E C XF D B:

(3.10)

Now, suppose that A and R satisfy equations (3.2) and (3.3). Then, V A D B XR, and it follows from Theorem 11.2.4 that equality (3.7) holds for some L. Moreover, upon replacing A in equation (3.3) with the right side of equality (3.7), we find [in light of result (3.9)] that X0 V B X0 V XR D D or, equivalently, that R satisfies equation (3.6). Conversely, suppose that R satisfies equation (3.6) and that equality (3.7) holds for some L. Then, making use of results (3.8)–(3.10), we find that V A C XR D V V B V V XR C V .I V V /L C XR D B XR C 0 C XR D B and X0 A D X0 V B X0 V XR C X0 .I V V /L D X0 V B X0 V XR D X0 V B .X0 V B D/ D D; so that A and R satisfy equations (3.2) and (3.3). Our results suggest a “computational” scheme for solving linear system (3.1). First, solve equation (3.6) for R. Then, substitute this solution in expression (3.7) to obtain a value for A—the matrix L in expression (3.7) can be chosen arbitrarily; in particular, L can be set equal to 0. We have that (3.11) rank.X0 V X/ D rank.X/: To see this, observe that X D V K for some matrix K and hence that X0 V X D K0 V V V K D K0 V K, implying [in light of Part (1) of Theorem 14.12.25] that X.X0 V X/ X0 V X D V K.K0 V K/ K0 V K D V K D X:

(3.12)

And result (3.12) implies result (3.11), as is evident from the following lemma. Lemma 19.3.1. Let W represent an n n matrix and X an n p matrix. Then, the following three conditions are equivalent: (1) rank.X0 W X/ D rank.X/;


471

(2) X.X0 W X/ X0 W X D X; (3) X0 W X.X0 W X/ X0 D X0 . Proof. If Condition (1) is satisfied, then (according to Corollary 4.4.7) R.X0 W X/ D R.X/, implying (in light of Lemma 9.3.5) that Condition (2) is satisfied. Conversely, if Condition (2) is satisfied, then (according to Corollary 4.4.5) rank.X0 W X/ rank.X/, implying [since clearly rank.X0 W X/ rank.X/] that Condition (1) is satisfied. Thus, Conditions (1) and (2) are equivalent. It can be shown in similar fashion that Conditions (1) and (3) are equivalent. Q.E.D. It is clear from result (3.11) that, aside from the special case where rank.X/ D p, equation (3.6) has an infinite number of solutions (for R). However, since any solution to equation (3.6) is the second part of a solution to linear system (3.1), it follows from Theorem 19.2.2 that the value of XR is invariant to the choice of a solution. Since a solution (for R) to equation (3.6) can be obtained by premultiplying the right side of the equation by .X0 V X/, we find [upon substituting this solution in expression (3.7)] that A is the first part of some solution to linear system (3.1) if and only if A D ŒV V X.X0 V X/ X0 V B C V X.X0 V X/ D C .I V V /L for some matrix L, or, equivalently, if and only if A D V .I PX;V /B C V X.X0 V X/ D C .I V V /L

(3.13)

for some L. Clearly, the first part of the solution to linear system (3.1) is unique if and only if V is nonsingular (which is the special case considered in Subsection a). c. General case The following, easily verifiable lemma provides a basis for extending the results of Subsection b on the solution of linear system (3.1) [which are for the special case where C.X/ C.V /] to the general case. Lemma 19.3.2. Let V represent an m n matrix, X an m p matrix, Y a q n matrix, B an m s matrix, D a q s matrix, and U a p q matrix. Then, A and R are, respectively, the first (n s) and second (p s) parts of a solution to the linear system V X A B D Y 0 R D (in an n s matrix A and a p s matrix R) if and only if A and R U D are, respectively, the first (n s) and second (p s) parts of a solution to the linear system V C XU Y X A B D Y 0 T D

472


(in an n s matrix A and a p s matrix T ). The following theorem gives “explicit” expressions for the first and second parts of solutions to linear system (3.1). Theorem 19.3.3. Let V represent an n n symmetric nonnegative definite matrix, X an n p matrix, B an n s matrix such that C.B/ C.V; X/, and D a p s matrix such that C.D/ C.X0 /. Further, let U represent any p p matrix such that C.X/ C.V C XU X0 /, and let W represent an arbitrary generalized inverse of V C XU X0 . Then, A and R are, respectively, the first (n s) and second (p s) parts of a solution to the (consistent) linear system V X A B D (3.14) X0 0 R D (in an n s matrix A and a p s matrix R) if and only if

and

R D T C U D

(3.15)

A D W B W XT C ŒI W .V C XU X0 /L

(3.16)

for some solution T to the (consistent) linear system X0 W XT D X0 W B D

(3.17)

(in a p s matrix T ) and for some n s matrix L. A natural question (in connection with Theorem 19.3.3) is whether a p p matrix U such that C.X/ C.V C XU X0 / necessarily exists. The answer is yes. To see this, note that (according to Corollary 14.3.8) V D S0 S for some matrix S, implying (in light of Corollary 7.4.5) that, for any nonzero scalar k, C.X/ D C.kX/ C.S0; kX/ D CŒ.S0; kX/.S0; kX/0 D C.V C k 2 XX0 /: Thus, C.X/ C.V C XU X0 / for U D k 2 I and in particular for U D I. In the special case where C.X/ C.V /, the condition C.X/ C.V CXU X0 / can be satisfied by setting U D 0. When U D 0, formula (3.16) reduces to formula (3.7). Preliminary to proving Theorem 19.3.3, it is convenient to establish the following lemma. Lemma 19.3.4. Let V represent an nn symmetric matrix, X an np matrix, and U a p p matrix. Then, the following ten conditions are equivalent: (1) C.X/ C.V C XU X0 /; (2) C.V C XU X0 / D C.V; X/; (3) rank.V C XU X0 / D rank.V; X/; (4) rank.V C XU 0 X0 / D rank.V; X/; (5) C.V C XU 0 X0 / D C.V; X/;


473

(6) C.X/ C.V C XU 0 X0 /; (7) R.X0 / R.V C XU X0 /; (8) R.X0 / R.V C XU 0 X0 /; (9) X D .V C XU X0 /.V C XU X0 / X; (10) X0 D X0 .V C XU X0 / .V C XU X0 /. Proof (of Lemma 19.3.4). Clearly,

I V C XU X D .V; X/ ; U X0 0

so that

C.V C XU X0 / C.V; X/: 0

(3.18) 0

Now, if C.X/ C.V C XU X /, then X D .V C XU X /L for some matrix L, in which case V D V C XU X0 XU X0 D V C XU X0 .V C XU X0 /LU X0 D .V C XU X0 /.I LU X0 / and consequently .V; X/ D .V C XU X0 /.ILU X0; L/; implying that C.V; X/ C.V C XU X0 / and hence [in light of result (3.18)] that C.V C XU X0 / D C.V; X/. Thus, (1) ) (2), and, since clearly (2) ) (1), (1) , (2). Moreover, in light of Theorem 4.4.6, it follows from result (3.18) that (3) ) (2), and, since clearly (2) ) (3), (2) , (3). We conclude that Conditions (1), (2), and (3) are equivalent. And, based on essentially the same line of reasoning (with U 0 in place of U ), we further conclude that Conditions (4), (5), and (6) are equivalent. Further, observe that V CXU 0 X0 D .V CXU X0 /0 . Thus, rank.V CXU 0 X0 / D rank.V C XU X0 /, and consequently (4) , (3). And, in light of Lemma 4.2.5, (7) , (6), and (8) , (1). Finally, in light of Lemma 9.3.5, (9) , (1), and (10) , (7). Q.E.D. Proof (of Theorem 19.3.3). Observe (in light of Lemma 19.3.4) that

and

.V C XU X0 /W X D X

(3.19)

X0 W .V C XU X0 / D X0 :

(3.20)

Observe also that B D V E C XF for some matrices E and F [so that B D .V C XU X0 /E C X.F U X0 E/] and hence that .V C XU X0 /W B D .V C XU X0 /W .V C XU X0 /E C .V C XU X0 /W X.F U X0 E/ D .V C XU X0 /E C X.F U X0 E/ D B: (3.21)

474


Now, suppose that A and R are the first and second parts of a solution to linear system (3.14), and let T0 D R U D. Then, R D T0 C U D. And we have, as a consequence of Lemma 19.3.2, that V C XU X0 X B A D D 0 T0 X0 or, equivalently, that .V C XU X0 /A D B XT0 X0 A D D:

(3.22)

Thus, it follows from Theorem 11.2.4 that A D W B W XT0 C ŒI W .V C XU X0 /L

(3.23)

for some matrix L. Moreover, upon replacing A in equality (3.22) with expression (3.23), we find [in light of result (3.20)] that X0 W B X0 W XT0 D D; or, equivalently, that X0 W XT0 D X0 W B D, so that T0 is a solution to linear system (3.17). Conversely, suppose that R D T C U D and A D W B W XT C ŒI W .V C XU X0 /L for some solution T to linear system (3.17) and for some matrix L. Then, making use of results (3.19)–(3.21), we find that .V C XU X0 /A C XT D .V C XU X0 /W B .V C XU X0 /W XT C .V C XU X0 /ŒI W .V C XU X0 /L C XT D B XT C 0 C XT D B and X0 A D X0 W B X0 W XT C X0 ŒI W .V C XU X0 /L D X0 W B X0 W XT D X0 W B .X0 W B D/ D D; so that

V C XU X0 X B A D : D 0 T X0

Since T D R U D, it follows from Lemma 19.3.2 that A and R are the first and second parts of a solution to linear system (3.14). Finally, note that the consistency of linear system (3.14) was established in Theorem 19.2.2 and that its consistency implies the consistency of linear system (3.17). Q.E.D.


475

Theorem 19.3.3 suggests a “computational” scheme for solving linear system (3.1) or, equivalently, linear system (3.14). First, obtain a solution T to linear system (3.17). Then, use expressions (3.16) and (3.15) to obtain the first and second parts of a solution to linear system (3.14)—the matrix L in expression (3.16) can be chosen arbitrarily; in particular, L can be set equal to 0. Result (3.11), which is for the special case where C.X/ C.V /, is generalized in the following theorem. Theorem 19.3.5. Let V represent an n n symmetric nonnegative definite matrix and X an n p matrix. Further, let U represent any p p matrix such that C.X/ C.V C XU X0 /, and let W represent an arbitrary generalized inverse of V C XU X0 . Then, (3.24) rank.X0 W X/ D rank.X/: Proof. Observe (in light of Lemma 19.3.4) that R.X0 / R.V C XU X0 / and hence (in light of Lemma 9.3.5) that X0 W .V C XU X0 / D X0 : Now, let Z D I .X0 / X0 . Then, X0 W V Z D X0 W V Z C X0 W XU X0 Z D X0 W .V C XU X0 /Z D X0 Z D 0: Thus, since (as a consequence of Theorem 17.2.13 and Lemma 19.3.4) C.X; V Z/ D C.X; V / D C.V C XU X0 /; we find (in light of Corollary 4.2.4) that C.X0 W X/ D C.X0 W X; 0/ D C.X0 W X; X0 W V Z/ D CŒX0 W .X; V Z/ D CŒX0 W .V C XU X0 / D C.X0 /; Q.E.D. implying that rank.X0 W X/ D rank.X/. In light of Lemma 19.3.1, we have the following corollary of Theorem 19.3.5. Corollary 19.3.6. Let V represent an n n symmetric nonnegative definite matrix and X an n p matrix. Further, let U represent any p p matrix such that C.X/ C.V C XU X0 /, and let W represent an arbitrary generalized inverse of V C XU X0 . Then, X.X0 W X/ X0 W X D X; X0 W X.X0 W X/ X0 D X0 :

(3.25) (3.26)

In connection with Theorem 19.3.3, it is clear from result (3.24) that, aside from the special case where rank.X/ D p, linear system (3.17) has an infinite number of solutions (for T ). However, since, for any solution T to linear system (3.17), T C U D is the second part of a solution to linear system (3.14), it follows from

476


Theorem 19.2.2 that the value of X.T C U D/ is invariant to the choice of solution and hence that XT is invariant to the choice of solution—since R.X0 W X/ D R.X/ (as is evident from Theorem 19.3.5 and Corollary 4.4.7), the invariance of XT is also apparent from Theorem 11.10.1. Since the matrix .X0 W X/ .X0 W BD/ is one solution to linear system (3.17), we find [upon substituting this solution in expression (3.16)] that the n s matrix A is the first part of some solution to linear system (3.14) if and only if A D ŒW W X.X0 W X/ X0 W B C W X.X0 W X/ D C ŒI W .V C XU X0 /L for some matrix L, or, equivalently, if and only if A D W .I PX;W /B C W X.X0 W X/ D C ŒI W .V C XU X0 /L

(3.27)

for some L. Clearly, the first part of the solution to linear system (3.14) is unique if and only if V C XU X0 is nonsingular or, equivalently (in light of Lemma 19.3.4), if and only if rank.V; X/ D n. The matrix W in Theorem 19.3.3 (and in Theorem 19.3.5 and Corollary 19.3.6) is by definition a generalized inverse of the matrix V C XU X0 —recall that V is a symmetric nonnegative definite matrix and that U is any matrix such that C.X/ C.V C XU X0 /. There is no requirement that W be a generalized inverse of V . Nevertheless, U can be chosen in such a way that W will be a generalized inverse of V as well as of V C XU X0 . To verify this, it suffices (in light of Theorem 18.2.5) to show that U can be chosen in such a way that C.V / and C.XU X0 / are essentially disjoint and R.V / and R.XU X0 / are essentially disjoint [and also, of course, in such a way that C.X/ C.V C XU X0 /]. For purposes of constructing such a U , let R represent any matrix such that V D RR0 —the existence of which follows from Corollary 14.3.8—let T represent any matrix whose columns span the null space of V, and observe (in light of Corollaries 7.4.5, 4.5.6, and 17.2.14) that C.V; X/ D C.V; XX0 / D C.V; XX0 T / D C.R; XX0 T / D CŒ.R; XX0 T /.R; XX0 T /0 D C.V C XX0 T T 0 XX0 /: Now, set U D X0 T T 0 X. Then, C.V C XU X0 / D C.V; X/, or, equivalently (in light of Lemma 19.3.4), C.X/ C.V C XU X0 /. Moreover, C.XU X0 / D CŒXX0 T .XX0 T /0 D C.XX0 T /, implying (in light of Corollary 17.2.14) that C.V / and C.XU X0 / are essentially disjoint. And, since clearly XU X0 is symmetric, it follows from Lemma 17.2.1 that R.V / and R.XU X0 / are also essentially disjoint.

19.4 Some Results on Generalized Inverses of Partitioned Matrices In Subsection a, some expressions are obtained for generalized inverses of par-

19.4 Some Results on Generalized Inverses of Partitioned Matrices

477

V X (where V is symmetric and nonnegative X0 0 definite), and some basic properties of such generalized inverses are described. Actually, some of these results are applicable to generalized inverses of partitioned V X matrices of the more general form (where Y is not necessarily equal Y 0 to X0 ) and do not require that V be symmetric or nonnegative definite (or even square). Then, in Subsection b, an alternative derivation of the results of Section 19.3c [on the solution of linear system (3.1)] is devised by applying the results of Subsection a. titioned matrices of the form

a. Some expressions and basic properties It is easy to verify the following lemma. Lemma 19.4.1. Let V represent an m n matrix, X an m p matrix, and Y a q n matrix. And let G11 represent an n m matrix, G12 an n q matrix, G21 a p m matrix, and G22 a p q matrix. Then, G11 G12 V X G11 V CG12 Y G11 X D ; (4.1) Y 0 G21 G22 G21 V CG22 Y G21 X V X G11 G12 V G11 CXG21 V G12 C XG22 D : (4.2) Y 0 G21 G22 Y G11 Y G12 V X G11 G12 is a generalized inverse of the partitioned matrix Further, Y 0 G21 G22 if and only if V .G11 V C G12 Y / C X.G21 V C G22 Y / D V Œor, equivalently, .V G11 C XG21 /V C .V G12 C XG22 /Y D V ;

(4.3a)

.V G11 C XG21 /X D X; Y .G11 V C G12 Y / D Y ; and

(4.3b) (4.3c)

Y G11 X D 0:

(4.3d)

V X Some basic properties of generalized inverses of matrices of the form X0 0 (where V is symmetric and nonnegative definite) are established in the following theorem. Theorem 19.4.2. Let V represent an n n symmetric nonnegative definite G11 G12 matrix and X an n p matrix. And let G D (where G11 is of G21 G22 dimensions n n) represent an arbitrary generalized inverse of the partitioned V X . Then, matrix X0 0

478


(1) X0 G12 X0 D X0 and XG21 X D X (i.e., G12 is a generalized inverse of X0 and G21 a generalized inverse of X); (2) V G12 X0 D XG21 V D XG22 X0 ; (3) V G11 X D 0, X0 G11 V D 0, and X0 G11 X D 0; (4) V D V G11 V XG22 X0 ; and (5) V G11 V , V G12 X0 , XG21 V , and XG22 X0 are symmetric and are invariant to the choice of the generalized inverse G. Proof. (1)–(2) Consider the linear system A 0 V X D R X0 X0 0 (in an n n matrix A and a p n matrix R). Since (according to Corollary 14.3.8) V D K0 K for some matrix K, it follows from Theorem 7.4.8 that this linear system is consistent. Thus, 0 0 V X G11 G12 D ; (4.4) X0 X0 X0 0 G21 G22 V X and also [since, in light of the symmetry of , G 0 is a generalized inverse X0 0 V X of ] X0 0 0 0 V X 0 0 G11 G21 D : (4.5) 0 0 X0 0 X0 X0 G12 G22 Moreover, V X0 V X0

X G11 0 G21 0 X G11 0 0 G12

0 V G12 X0 C XG22 X0 D ; X0 X0 G12 X0 0 0 0 0 V G21 G21 X0 C XG22 X0 D 0 0 X0 G22 X0 G21 X0

G12 G22

(4.6) (4.7)

[as can be easily verified by taking advantage of result (4.2)]. Clearly, results (4.6) and (4.4) imply that V G12 X0 C XG22 X0 D 0, or, equivalently, that V G12 X0 D XG22 X0 , and that X0 G12 X0 D X0 . And results (4.7) and (4.5) imply 0 0 0 X0 C XG22 X0 D 0 and X0 G21 X0 D X0 , and hence that XG21 V D that V G21 0 0 0 0 0 0 0 0 X0 /0 D .X0 /0 D .V G21 X / D .XG22 X / D XG22 X and XG21 X D .X0 G21 X. (3) That X0 G11 X D 0 is evident from Lemma 19.4.1. And, as a further consequence of Lemma 19.4.1, we have that .V G11 C XG21 /X D X and X0 .G11 V C G12 X0 / D X0 , implying [in light of Part (1)] that V G11 X D X XG21 X D 0 and X0 G11 V D X0 X0 G12 X0 D 0. (4) It follows from Lemma 19.4.1 that V D V G11 V C V G12 X0 C XG21 V C XG22 X0 :


479

And, upon applying Part (2), we find that V D V G11 V XG22 X0 XG22 X0 C XG22 X0 D V G11 V XG22 X0 : G11 G12 represent an arbitrary generalized inverse of (5) Let G D G21 G22 V X (where G11 is of dimensions n n and where G may differ from X0 0 G). Then, making use of Parts (3) and (4), we find that V G11 V D V G11 .V G11 V XG22 X0 /

D V G11 V G11 V V G11 XG22 X0 D V G11 V G11 V .since V G11 X D 0/ 0 D V G11 V G11 V XG22 X G11 V .since X0 G11 V D 0/ D .V G11 V XG22 X0 /G11 V D V G11 V :

0 V D V G11 V And, upon setting G D G 0 , we find, in particular, that V G11 0 or, equivalently, that .V G11 V / D V G11 V . Thus, V G11 V is symmetric and is invariant to the choice of the generalized inverse G. Further, since [as a consequence of Part (4)] XG22 X0 D V G11 V V, XG22 X0 is symmetric and is invariant to the choice of G, and, in light of Part (2), V G12 X0 and XG21 V are symmetric and are invariant to the choice of G. Q.E.D. Let V represent an nn symmetric nonnegative definite matrix and X an np matrix. In the special case where C.X/ C.V /, Theorem 9.6.1 can be used to V X in terms of V obtain a generalized inverse for the partitioned matrix X0 0 and .X0 V X/. Upon applying formula (9.6.2) [and observing that .X0 V X/ is a generalized inverse of X0 V X], we find that [in the special case where C.X/ C.V /] the partitioned matrix V V X.X0 V X/ X0 V V X.X0 V X/ (4.8) .X0 V X/ X0 V .X0 V X/ V X is a generalized inverse of . X0 0 A generalization of formula (4.8) that applies without restriction [i.e., that applies even if C.X/ is not contained in C.V /] is given by the following theorem.

Theorem 19.4.3. Let V represent an n n symmetric nonnegative definite matrix and X an n p matrix. Further, let U represent any p p matrix such that C.X/ C.V C XU X0 /, and let W represent an arbitrary generalized inverse of V C XU X0 . Then, the partitioned matrix W W X.X0 W X/ X0 W W X.X0 W X/ .X0 W X/ X0 W .X0 W X/ C U

480


V X . X0 0 Preliminary to proving Theorem 19.4.3, it is convenient to establish the following theorem, which is of some interest in its own right.

is a generalized inverse of the partitioned matrix

Theorem 19.4.4. Let V represent an mn matrix, X an mp matrix, Y a qn G11 G12 matrix, and U a p q matrix. (1) Then, for any generalized inverse G21 G22 V X of the partitioned matrix (where G11 is of dimensions n m), the partiY 0 G11 G12 tioned matrix is a generalized inverse of the partitioned matrix G21 G22 U V C XU Y X H11 H12 . (2) Conversely, for any generalized inverse of Y 0 H21 H22 V C XU Y X the partitioned matrix (where H11 is of dimensions n m), the Y 0 H11 H12 partitioned matrix is a generalized inverse of the partitioned H21 H22 C U V X matrix . Y 0 Proof (of Theorem 19.4.4). (1) Let V C XU Y X G11 G12 V C XU Y X CD : Y 0 G21 G22 U Y 0 C11 C12 (where C11 is of dimensions m n). and partition C as C D C21 C22 Then, making use of the results of Lemma 19.4.1, we find that C22 D Y G11 X D 0; C21 D Y .G11 V C G11 XU Y C G12 Y / D Y .G11 V C G12 Y / C Y G11 XU Y D Y .G11 V C G12 Y / .since Y G11 X D 0/ D Y; C12 D .V G11 C XU Y G11 C XG21 /X D .V G11 C XG21 /X C XU Y G11 X D .V G11 C XG21 /X D X;


481

C11 D .V G11 C XU Y G11 C XG21 /.V C XU Y / C .V G12 C XU Y G12 C XG22 XU /Y D .V G11 C XG21 /V C XU Y G11 V C .V G11 C XG21 /XU Y C XU Y G11 XU Y C .V G12 C XG22 /Y C XU Y G12 Y XU Y D .V G11 C XG21 /V C XU Y G11 V C XU Y C 0 C .V G12 C XG22 /Y C XU Y G12 Y XU Y .since Y G11 X D 0/ D .V G11 C XG21 /V C .V G12 C XG22 /Y C XU Y G11 V C XU Y G12 Y D V C XU Y G11 V C XU Y G12 Y D V C XU Y .G11 V C G12 Y / D V C XU Y : V C XU Y X G12 G11 is a generalized inverse of . Thus, G21 G22 U Y 0 V X .V C XU Y / C X.U /Y X (2) Rewriting as and applying Part Y 0 Y 0 (1) (with that the partitioned V and U in place of U ), we find V CXU Y in place of H11 H12 H12 H11 , which equals , is a generalized matrix H21 H22 .U / H21 H22 C U V X inverse of . Q.E.D. Y 0 Proof (of Theorem 19.4.3). Since (in light of Lemma 19.3.4) R.X0 / R.V C XU X0 /, formula (9.6.2) (for the generalized inverse of a partitioned ma V C XU X0 X trix) is applicable to the partitioned matrix . Upon applying that 0 X0 formula [and observing that .X0 W X/ is a generalized inverse of X0 W X], we find that the partitioned matrix W W X.X0 W X/ X0 W W X.X0 W X/ .X0 W X/ X0 W .X0 W X/ V C XU X0 X . Thus, it follows fom Part (2) of is a generalized inverse of 0 X0 Theorem 19.4.4 that the partitioned matrix W W X.X0 W X/ X0 W W X.X0 W X/ .X0 W X/ X0 W .X0 W X/ C U V X . Q.E.D. is a generalized inverse of the partitioned matrix X0 0 b. Alternative derivation of the results of Section 19.3c For purposes of obtaining an alternative derivation of the results of Section 19.3c, let V represent an n n symmetric nonnegative definite matrix and X an n p

482


matrix. Further, let U represent any pp matrix such that C.X/ C.V CXU X0 /, and let W represent an arbitrary generalized inverse of V C XU X0 . Then, upon taking the generalized inverse G [of the partitioned matrix V X ] in Theorem 19.4.2 to be the generalized inverse given by Theorem X0 0 19.4.3, it follows from Part (1) of Theorem 19.4.2 that X0 W X.X0 W X/ X0 D X0 ; 0

0

X.X W X/ X W X D X:

(4.9) (4.10)

And, in light of Lemma 19.3.1, it is evident from equality (4.9) or (4.10) that rank.X0 W X/ D rank.X/:

(4.11)

Now, let B represent an n s matrix such that C.B/ C.V; X/ and D a p s matrix such that C.D/ C.X0 /. And let G11 D W W X.X0 W X/ X0 W ; G12 D W X.X0 W X/; G21 D .X0 W X/ X0 W ; G22 D .X0 W X/ C U :

G11 G12 Then, according to Theorem 19.4.3, the partitioned matrix is a genG21 G22 V X eralized inverse of the partitioned matrix , and it follows from Theorem X0 0 11.2.4 that A and R are, respectively, the first (n s) and second (p s) parts of a solution to the (consistent) linear system A B V X D (4.12) R D X0 0 (in an n s matrix A and a p s matrix R) if and only if G11 G12 B In 0 G11 G12 V X L A D C D X0 0 R G21 G22 G21 G22 M 0 Ip for some n s matrix L and some p s matrix M. Note that, as a consequence of Lemma 19.3.4, X0 D X0 W .V C XU X0 /, or, equivalently, that X0 .I W V / D X0 W XU X0 . Thus, making use of results (4.1) and (4.10), we find that G11 G12 V X X0 0 G21 G22 0 W V C W X.X0 W X/ X0 .I W V / D .X0 W X/ X0 .I W V / C U X0 .X0 W X/ X0 W X 0 W .V C XU X0 / : D ŒI .X0 W X/ X0 W XU X0 .X0 W X/ X0 W X

19.5 Some Additional Results on the Form of Solutions

483

We conclude that A and R are, respectively, the first and second parts of a solution to linear system (4.12) if and only if A D ŒW W X.X0 W X/ X0 W B C W X.X0 W X/ D C ŒI W .V C XU X0 /L; R D .X0 W X/ .X0 W B D/ C U D C ŒI .X0 W X/ X0 W X.M U X0 L/ for some matrices L and M, or, equivalently, if and only if A D ŒW W X.X0 W X/ X0 W B C W X.X0 W X/ D C ŒI W .V C XU X0 /L; R D .X0 W X/ .X0 W B D/ C U D C ŒI .X0 W X/ X0 W XZ

(4.13) (4.14)

for some matrices L and Z. This result can be reexpressed in terms of the linear system X0 W XT D X0 W B D

(4.15)

(in a p s matrix T ). Since clearly D D X0 K for some matrix K, it follows from result (4.9) that X0 W X.X0 W X/ .X0 W B D/ D X0 W B D and hence (in light of Lemma 9.5.1) that linear system (4.15) is consistent. Further, in light of Corollary 4.4.7, it follows from result (4.11) that R.X0 W X/ D R.X/, implying (in light of Theorem 11.10.1) that XT is invariant to the choice of solution to linear system (4.15). Then, upon recalling Theorem 11.2.4 and observing that the matrix .X0 W X/ .X0 W B D/ is a solution to linear system (4.15), we find that equalities (4.13) and (4.14) hold for some matrices L and Z if and only if R D T C U D and A D W B W XT C ŒI W .V C XU X0 /L for some solution T to linear system (4.15) and for some matrix L.

19.5 Some Additional Results on the Form of Solutions to the Constrained Minimization Problem Let a represent an n 1 vector of variables. And consider the minimization of the quadratic form a0 V a subject to the constraint X0 a D d, where V is an n n symmetric nonnegative definite matrix, X is an n p matrix, and d is a p 1 vector such that d 2 C.X0 /. This minimization problem is a special case of the minimization problem considered in Sections 19.2–19.4. It follows from the results of Sections 19.2–19.4 that the problem of minimizing a0 V a subject to X0 a D d has a solution or solutions of the form

484


a D W X.X0 W X/ d;

(5.1)

where W is an nn matrix. For the vector (5.1) to be a solution to this minimization problem, it is sufficient (in light of the results of Section 19.3c) that W equal any generalized inverse of V C XU X0 , where U is any p p matrix such that C.X/ C.V C XU X0 /. In this section, conditions are given that are necessary (as well as sufficient) for the vector (5.1) to solve, for every d 2 C.X0 /, the problem of minimizing a0 V a subject to X0 a D d (when W does not vary with d). For fixed V , these conditions characterize those vectors of the form (5.1) that are solutions [for every d 2 C.X0 /] to this minimization problem. For fixed W , these conditions characterize those nonnegative definite quadratic forms (in a) for which the vector (5.1) is a solution to the constrained minimization problem. The following theorem gives what are perhaps the most basic of the necessary and sufficient conditions. Theorem 19.5.1. Let V represent an n n symmetric nonnegative definite matrix, W an n n matrix, X an n p matrix, and d a p 1 vector. Then, for the vector W X.X0 W X/ d to be a solution, for every d 2 C.X0 /, to the problem of minimizing the quadratic form a0 V a (in a) subject to X0 a D d, it is necessary and sufficient (assuming that the elements of W are not functionally dependent on d) that C.V W X/ C.X/ (5.2) and

rank.X0 W X/ D rank.X/:

(5.3)

Proof. It follows from Theorem 19.2.1 that a0 V a has a minimum at W X.X0 W X/ d under the constraint X0 a D d [where d 2 C.X0 /] if and only if V W X.X0 W X/ d C Xr D 0 for some vector r and X0 W X.X0 W X/ d D d, or, equivalently, if and only if V W X.X0 W X/ d 2 C.X/ and X0 W X.X0 W X/ d D d. Thus, for W X.X0 W X/ d to be a solution, for every d 2 C.X0 /, to the problem of minimizing a0 V a subject to X0 a D d, it is necessary and sufficient that, for every n 1 vector u, V W X.X0 W X/ X0 u 2 C.X/ and X0 W X.X0 W X/ X0 u D X0 u or, equivalently, that (5.4) CŒV W X.X0 W X/ X0 C.X/ and

X0 W X.X0 W X/ X0 D X0 :

(5.5)

It remains to show that conditions (5.4) and (5.5) are equivalent to conditions (5.2) and (5.3). If condition (5.2) is satisfied, then, since CŒV W X.X0 W X/ X0 C.V W X/, condition (5.4) is satisfied; and if condition (5.3) is satisfied, then, as a consequence of Lemma 19.3.1, condition (5.5) is satisfied. Conversely, suppose that conditions (5.4) and (5.5) are satisfied. Then, it follows from Lemma 19.3.1 that condition (5.3) is satisfied. And, as a further consequence of Lemma 19.3.1, we have that


485

V W X.X0 W X/ X0 W X D V W X: Thus,

C.V W X/ CŒV W X.X0 W X/ X0 ;

and hence condition (5.2) is satisfied. Q.E.D. In connection with Theorem 19.5.1, note that the condition C.V W X/ C.X/ is equivalent to the condition V W X D XQ for some matrix Q:

(5.6)

And, letting Z represent any matrix whose columns span C? .X/ or, equivalently, span N.X0 /, it is also equivalent to the condition Z0 V W X D 0

(5.7)

(as is evident from Corollary 12.5.7). Moreover, the condition rank.X0 W X/ D rank.X/ is equivalent to each of the following conditions: X.X0 W X/ X0 W X D X; X0 W X.X0 W X/ X0 D X0 ;

(5.8) (5.9)

R.X0 W X/ D R.X/; C.X0 W X/ D C.X0 /

(5.10) (5.11)

(as is evident from Lemma 19.3.1 and Corollary 4.4.7). Thus, variations on Theorem 19.5.1 can be obtained by replacing condition (5.2) with condition (5.6) or (5.7) and/or by replacing condition (5.3) with any one of conditions (5.8)–(5.11). Note (in light of Corollary 14.11.3) that one circumstance under which the condition rank.X0 W X/ D rank.X/ is satisfied is that where W is symmetric and positive definite. The following theorem provides alternatives to the necessary and sufficient conditions given by Theorem 19.5.1 [conditions (5.2) and (5.3)] and to the variations on those conditions [conditions (5.6)–(5.7) and (5.8)–(5.11)]. Theorem 19.5.2. Let V represent an n n symmetric nonnegative definite matrix, W an n n matrix, X an n p matrix, and d a p 1 vector. Then, for the vector W X.X0 W X/ d to be a solution, for every d 2 C.X0 /, to the problem of minimizing the quadratic form a0 V a (in a) subject to X0 a D d, it is necessary and sufficient that there exist a p p matrix U such that .V C XU X0 /W X D X:

(5.12)

It is also necessary and sufficient that W be expressible in the form W D G C K ;

(5.13) 0

where, for some p p matrix U such that C.X/ C.V C XU X /, G is an arbitrarygeneralized inverse of V C XU X0, and where K is some solution to the V KX D 0 (in an n n matrix K). system X0

486


Preliminary to proving Theorem 19.5.2, it is convenient to prove the following lemma. Lemma 19.5.3. For any nn symmetric nonnegative definite matrix V, nn matrix W, and n p matrix X, C.V W X/ C.X/ ) R.V W X/ R.X0 W X/: Proof (of Lemma 19.5.3). Suppose that C.V W X/ C.X/. Then, as a consequence of Corollary 4.2.4, C.X0 W 0 V W X/ C.X0 W 0 X/; and (in light of Lemma 4.2.5) we have that RŒ.W X/0 V W X D RŒ.X0 W 0 V W X/0 RŒ.X0 W 0 X/0 D R.X0 W X/: Since (in light of Lemma 14.11.2) RŒ.W X/0 V W X D R.V W X/, we conclude Q.E.D. that R.V W X/ R.X0 W X/. Proof (of Theorem 19.5.2). To prove the first part of the theorem, it suffices (in light of Theorem 19.5.1) to show that there exists a matrix U that satisfies condition (5.12) if and only if C.V W X/ C.X/ and rank.X0 W X/ D rank.X/ or, equivalently (in light of Corollary 4.4.7), if and only if C.V W X/ C.X/ and R.X0 W X/ D R.X/. Upon observing that condition (5.12) is equivalent to the condition XU X0 W X D X V W X; it is clear from Corollary 9.7.2 that there exists a matrix U that satisfies condition (5.12) if and only if C.X V W X/ C.X/ and R.X V W X/ R.X0 W X/ or, equivalently (in light of Lemma 4.5.10), if and only if C.V W X/ C.X/ and R.X V W X/ R.X0 W X/. Now, suppose that C.V W X/ C.X/, implying (according to Lemma 19.5.3) that R.V W X/ R.X0 W X/. Then, if R.X0 W X/ D R.X/, it follows from Lemma 4.5.10 that R.XV W X/ R.X0 W X/. Conversely, if R.XV W X/ R.X0 W X/, it follows from Lemma 4.5.10 that R.X/ R.X0 W X/ or, equivalently [since clearly R.X0 W X/ R.X/], that R.X0 W X/ D R.X/. Thus, C.V W X/ C.X/ and R.X V W X/ R.X0 W X/ if and only if C.V W X/ C.X/ and R.X0 W X/ D R.X/. And we conclude that there exists a matrix U that satisfies condition (5.12) if and only if C.V W X/ C.X/ and R.X0 W X/ D R.X/. To complete the proof of Theorem 19.5.2, it suffices to show that W is expressible in the form (5.13) if and only if there exists a matrix U that satisfies condition (5.12). Suppose that W is expressible in the form (5.13). Then, making use of Lemma 19.3.4, we find that


487

.V C XU X0 /W X D .V C XU X0 /GX C V K X C XU X0 K X D X C 0 C XU 0 D X: Thus, there exists a matrix U that satisfies condition (5.12). Conversely, suppose that there exists a matrix U that satisfies condition (5.12). Then, clearly, C.X/ C.V C XU X0 /. And it follows from Theorem 11.12.1 that W D .V C XU X0 / XX C Z for some solution Z to the system .V C XU X0 /ZX D 0 (in Z). Now, let K D Z .V C XU X0 / .I XX /. Then, W D .V C XU X0 / XX C .V C XU X0 / .I XX / C K D .V C XU X0 / C K : Moreover, it follows from Lemma 19.3.4 that C.V C XU 0 X0 / D C.V; X/, implying (in light of Lemma 4.2.2) that .V; X/ D .V C XU 0 X0 /L for some matrix L and hence that V D .V; X/0 D L0 .V C XU 0 X0 /0 D L0 .V C XU X0 /: X0 Thus, V V V K X D Z X .V C XU X0 / .I XX /X X0 X0 X0 V 0 0 .V C XU X0 / .X XX X/ D L .V C XU X /Z X X0 V 0 .V C XU X0 / 0 D 0; DL0 X0 V KX D 0. Q.E.D. so that K is a solution to the system X0 As discussed earlier, the first of the necessary and sufficient conditions in Theorem 19.5.1 [condition (5.2)] is equivalent to condition (5.6) and also to condition (5.7), and, consequently, can be replaced by either of those two conditions. It can also be replaced by any condition that in combination with condition (5.3) gives a pair of conditions that is equivalent to conditions (5.2) and (5.3). The following theorem gives some variations on Theorem 19.5.1 obtained by replacing condition (5.2) by a condition of the latter kind. Theorem 19.5.4. Let V represent an n n symmetric nonnegative definite matrix, W an n n matrix, X an n p matrix, and d a p 1 vector. Then, for the vector W X.X0 W X/ d to be a solution, for every d 2 C.X0 /, to the problem of minimizing the quadratic form a0 V a (in a) subject to X0 a D d, it is necessary and sufficient that 0 0 PX;W 0 V .I PX;W 0 / D 0 and rank.X W X/ D rank.X/

488


or, equivalently, that 0 0 PX;W 0 V D PX;W 0 V PX;W 0 and rank.X W X/ D rank.X/:

It is also necessary and sufficient that 0 0 PX;W 0 V D V PX;W 0 and rank.X W X/ D rank.X/

or, equivalently, that PX;W 0 V be symmetric and rank.X0 W X/ D rank.X/. Preliminary to proving Theorem 19.5.4, it is convenient to establish the following lemma. Lemma 19.5.5. Let W represent an n n matrix and X an n p matrix. If rank.X0 W X/ D rank.X/, then (1) PX;W X D X, (2) X0 W PX;W D X0 W , (3) 2 PX;W D PX;W (i.e., PX;W is idempotent), (4) rank.PX;W / D rank.X/, and (5) rank.I PX;W / D n rank.X/. Proof (of Lemma 19.5.5). Suppose that rank.X0 W X/ D rank.X/ [and recall that PX;W D X.X0 W X/ X0 W ]. (1)–(2) That PX;W X D X and X0 W PX;W D X0 W is evident from Lemma 19.3.1. (3) Using Part (1), we find that 2 PX;W D PX;W X.X0 W X/ X0 W D X.X0 W X/ X0 W D PX;W :

(4)–(5) By definition, PX;W D XG, where G D .X0 W X/ X0 W . Since [according to Part (1)] XGX D X (i.e., since G is a generalized inverse of X), it follows from result (10.2.1) that rank.PX;W / D rank.X/ and from Lemma 10.2.6 that rank.I PX;W / D n rank.X/. Q.E.D. Proof (of Theorem 19.5.4). To establish the necessity and sufficiency of the 0 0 conditions PX;W 0 V .I PX;W 0 / D 0 and rank.X W X/ D rank.X/, it suffices (in light of Theorem 19.5.1) to show that these conditions are equivalent to the conditions C.V W X/ C.X/ and rank.X0 W X/ D rank.X/. Suppose that rank.X0 W X/ D rank.X/. Then, since X0 W 0 X D .X0 W X/0 , rank.X0 W 0 X/ D rank.X/. Thus, applying Parts (1) and (2) of Lemma 19.5.5 (with W 0 in place of W ), we find that PX;W 0 X D X and X0 W 0 PX;W 0 D X0 W 0 . 0 Letting Z D I PX;W 0 and applying Part (5) of Lemma 19.5.5, we find also 0 0 0 that X Z D .Z X/ D Œ.I PX;W 0 /X0 D 00 D 0 and rank.Z/ D rank.Z0 / D n rank.X/, implying (in light of Lemma 11.4.1) that C.Z/ D N.X0 /. Now, if C.V W X/ C.X/ or, equivalently, if V W X D XQ for some matrix Q, then 0 0 0 PX;W 0 V .I PX;W /V PX;W 0 / D Œ.I P 0 X;W 0

D f.I PX;W 0 /V W XŒ.X0 W 0 X/ 0 X0 g0 D f.I PX;W 0 /XQŒ.X0 W 0 X/ 0 X0 g0 D f.X X/QŒ.X0 W 0 X/ 0 X0 g0 D 0:

19.6 Transformation of the Constrained Minimization Problem

489

0 Conversely, if PX;W 0 V .I PX;W 0 / D 0, then 0 0 0 0 .V W X/0 Z D X0 W 0 V .I PX;W V .I PX;W 0/ D X W P 0 / D 0; X;W 0

implying (in light of Corollary 12.5.7) that C.V W X/ C.X/. 0 0 We conclude that the conditions PX;W 0 V .I PX;W 0 / D 0 and rank.X W X/ D rank.X/ are equivalent to the conditions C.V W X/ C.X/ and rank.X0 W X/ D rank.X/. To complete the proof of Theorem 19.5.4, it suffices to show that the conditions 0 0 PX;W 0 V D V PX;W 0 and rank.X W X/ D rank.X/ are equivalent to the conditions 0 0 PX;W 0 V D PX;W 0 V PX;W 0 and rank.X W X/ D rank.X/. 0 0 0 0 If PX;W 0 V D V PX;W 0 and rank.X W X/ D rank.X/, then [since X W X D 0 0 0 0 .X W X/ ] rank.X W X/ D rank.X/, implying [in light of Part (3) of Lemma 2 , so that 19.5.5] that PX;W 0 D P X;W 0 0 PX;W 0 V D PX;W 0 .PX;W 0 V / D PX;W 0 V PX;W 0: 0 V is symmetric, so that Conversely, if PX;W 0 V D PX;W 0 V PX;W 0 , then P X;W 0 0 PX;W 0 V D .PX;W 0 V /0 D V PX;W 0: 0 0 Thus, the conditions PX;W 0 V D V PX;W 0 and rank.X W X/ D rank.X/ are

0 0 equivalent to the conditions PX;W 0 V D PX;W 0 V PX;W 0 and rank.X W X/ D rank.X/. Q.E.D.

19.6 Transformation of the Constrained Minimization Problem to an Unconstrained Minimization Problem Let a represent an n1 vector of variables, and impose on a the constraint X0 a D d, where X is an n p matrix and d is a p 1 vector such that d 2 C.X0 /. Define f .a/ D a0 V a 2b0 a, where V is an n n symmetric nonnegative definite matrix and b is an n 1 vector such that b 2 C.V; X/. In Section 19.3, an expression was obtained for the general form of a solution to the problem of minimizing f .a/ (under the constraint X0 a D d). This expression was derived by “solving” the linear system V X a b D X0 0 r d (in a and r), which is the linear system obtained by equating the derivative of the Lagrangian function (with respect to a) to the null vector and by adjoining the constraint.

490


An alternative expression for the general form of a solution to the constrained minimization problem can be derived by transforming this problem to an unconstrained minimization problem (and by then making use of the results of Section 19.1). For purposes of deriving such an expression, let a0 D .X0 / d, and let Z D I .X0 / X0 . Or, more generally, let a0 represent any n 1 vector such that X0 a0 D d, and let Z represent any n k matrix whose columns span the null space of X0 . Then, in light of Theorem 11.2.3, X0 a D d if and only if a D a0 C Zu for some k 1 vector u. Now, for u 2 Rk1, define h.u/ D f .a0 C Zu/. Then, h.u/ D u0 Z0 V Zu C 2a00 V Zu C a00 V a0 2b0 a0 2b0 Zu D u0 Z0 V Zu 2ŒZ0 .b V a0 /0 u C c;

(6.1)

where c D .V a0 2b/0 a0 (which is a constant). And it is clear that the minimization of f .a/ (with respect to a), subject to the constraint X0 a D d, is equivalent to the unconstrained minimization of h.u/ (with respect to u). The equivalence is in the sense that f .a/ attains its minimum value (under the constraint X0 a D d) at a point a if and only if a D a0 C Zu for some k 1 vector u at which h.u/ attains its minimum value. Since b 2 C.V; X/, b D V r C Xs for some vectors r and s, so that Z0 .b V a0 / D Z0 .V r C Xs V a0 / D Z0 V .r a0 / C .X0 Z/0 s D Z0 V .r a0 / C 0 D Z0 V .r a0 /; implying that Z0 .b V a0 / 2 C.Z0 V / and hence (in light of Lemma 14.11.2) that Z0 .b V a0 / 2 C.Z0 V Z/. Thus, it follows from Theorem 19.1.1 that the linear system Z0 V Zu D Z0 .b V a0 / (in u) is consistent and that h.u/ has a minimum value at a point u if and only if u is a solution to Z0 V Zu D Z0 .b V a0 /, in which case h.u / D .b V a0 /0 Zu C c D b0 a0 .b V a0 /0 .a0 C Zu /:

(6.2)

We conclude that f .a/ attains its minimum value (under the constraint X0 a D d) at a point a if and only if a D a0 C Zu for some solution u to Z0 V Zu D Z0 .b V a0 /, or, equivalently (in light of Theorem 11.2.4), if and only if a D a0 C Z.Z0 V Z/ Z0 .b V a0 / C ZŒI .Z0 V Z/ Z0 V Zw

(6.3)

for some k 1 vector w, in which case f .a / D b0 a0 .b V a0 /0 a :

(6.4)

19.7 The Effect of Constraints on the Generalized Least Squares Problem

491

19.7 The Effect of Constraints on the Generalized Least Squares Problem Let X represent an np matrix, W an nn symmetric nonnegative definite matrix, and y an n-dimensional column vector. Further, let b represent a p 1 vector of variables, and consider the minimization of the quadratic form .yXb/0 W .yXb/ (in the difference between y and Xb), subject to the constraint A0 b D d; where A is a p q matrix and d is a q 1 vector in C.A0 /. The unconstrained minimization of .y Xb/0 W .y Xb/ was considered in Section 12.4 (in the special case where W D I), in Section 14.12f (in the special case where W is positive definite), and in Section 14.12i (in the general case). It was found that (in the absence of constraints) .y Xb/0 W .y Xb/ attains its minimum value at a point b if and only if b is a solution to the linear system X0 W Xb D X0 W y (in b), in which case .y Xb /0 W .y Xb / D y 0 W .y Xb /: The results of Section 19.3 can be used to generalize our results on the unconstrained minimization of .y Xb/0 W .y Xb/ to the case where b is subject to the constraint A0 b D d. For this purpose, reexpress .y Xb/0 W .y Xb/ as .y Xb/0 W .y Xb/ D b0 X0 W Xb 2.X0 W y/0 b C y 0 W y; and observe (in light of Theorem 14.2.9) that X0 W X is nonnegative definite (and symmetric) and (in light of Lemma 14.11.2) that X0 W y 2 C.X0 W X/. Now, applying Theorem 19.2.1 (with b, A, X0 W X, and X0 W y in place of a, X, V, and b, respectively), we find that the linear system 0 0 X WX A b X Wy D (7.1) 0 r A0 d (in b and the q 1 vector r) is consistent (and that the values of X0 W Xb and Ar are invariant to the choice of solution to this linear system). Further, .y Xb/0 W .y Xb/ has a minimum value (under the constraint A0 b D d) at a point b if and only if b is the first part of some solution to linear system (7.1). And, for the first and second parts b and r of any solution to linear system (7.1), .y Xb /0 W .y Xb / D y 0 W y .X0 W y/0 b d0 r D y 0 W .y Xb / d0 r : Linear system (7.1) comprises the two equations

(7.2) (7.3)

492


X0 W Xb C Ar D X0 W y; A0 b D d:

(7.4) (7.5)

If C.A/ C.X0 W X/, then (according to the results of Section 19.3b) b and r satisfy equations (7.4) and (7.5) if and only if r satisfies the equation A0 .X0 W X/ Ar D A0 .X0 W X/ X0 W y d

(7.6)

and b D .X0 W X/ X0 W y .X0 W X/ Ar C ŒI .X0 W X/ X0 W Xk

(7.7)

for some p1 vector k. Note (in light of Lemma 14.11.2 and Corollary 14.11.3) that the condition C.A/ C.X0 W X/ is equivalent to the condition C.A/ C.X0 W / and that, in the special case where W is positive definite, it is equivalent to the condition C.A/ C.X0 /. Turning now to the “solution” of linear system (7.1) in the general case [where C.A/ is not necessarily contained in C.X0 W X/], let U represent any q q matrix such that C.A/ C.X0 W X C AU A0 /, and let G represent any generalized inverse of X0 W X C AU A0 . Then, according to Theorem 19.3.3, b and r are, respectively, the first and second parts of a solution to linear system (7.1) if and only if (7.8) r D t C U d and

b D GX0 W y C GAt C ŒI G.X0 W X C AU A0 /k

(7.9)

for some solution t to the (consistent) linear system A0 GAt D A0 GX0 W y d

(7.10)

(in a q 1 vector t) and for some p 1 vector k. In particular, b is the first part of some solution to linear system (7.1) if and only if b D G.I PA;G /X0 W y C GA.A0 GA/ d C ŒI G.X0 W X C AU A0 /k (7.11) for some vector k.

Exercises Section 19.1 1. Let a represent an n1 vector of (unconstrained) variables, and define f .a/ D a0 V a 2b0 a, where V is an n n matrix and b an n 1 vector. Show that if V is not nonnegative definite or if b … C.V /, then f .a/ is unbounded from below; that is, corresponding to any scalar c, there exists a vector a such that f .a / < c.

Exercises

493

Section 19.3 2. Let V represent an n n symmetric matrix and X an n p matrix. Show that, for any p p matrix U such that C.X/ C.V C XU X0 /, (1) .V C XU X0 /.V C XU X0 / V D V ; (2) V .V C XU X0 / .V C XU X0 / D V . 3. In Section 19.3c, Theorem 19.3.3 was proved “from scratch.” Devise a shorter proof of Theorem 19.3.3 by taking advantage of Theorem 11.11.1. 4. Let V represent an n n symmetric matrix and X an n p matrix. Further, let U D X0 T T 0 X, where T is any matrix whose columns span the null space of V. Show that C.X/ C.V C XU X0 / and that C.V / and C.XU X0 / are essentially disjoint and R.V / and R.XU X0 / are essentially disjoint, as has already been established (in Section 19.3c) in the special case where V is nonnegative definite. 5. Let V represent an n n symmetric nonnegative definite matrix and X an n p matrix. Further, let Z represent any matrix whose columns span N.X0 / or, equivalently, C? .X/. And adopt the same terminology as in Exercise 17.20. (a) Using the result of Part (a) of Exercise 17.20 (or otherwise) show that an n n matrix H is a projection matrix for C.X/ along C.V Z/ if and only if H0 is the first (n n) part of a solution to the consistent linear system V X A 0 (E.1) D X0 0 R X0 (in an n n matrix A and a p n matrix R). (b) Letting U represent any p p matrix such that C.X/ C.V C XU X0 / and letting W represent an arbitrary generalized inverse of V C XU X0, show that an n n matrix H is a projection matrix for C.X/ along C.V Z/ if and only if H D PX;W C KŒI .V C XU X0 /W for some n n matrix K. Section 19.5 6. Let V represent an n n symmetric nonnegative definite matrix, W an n n matrix, and X an n p matrix. Show that, for the matrix W X.X0 W X/ X0 to be the first (n n) part of some solution to the (consistent) linear system V X A 0 (E.2) D X0 0 R X0

494


(in an n n matrix A and a p n matrix R), it is necessary and sufficient that C.V W X/ C.X/ and rank.X0 W X/ D rank.X/. 7. Let a represent an n 1 vector of variables, and impose on a the constraint X0 a D d, where X is an n p matrix and d is a p 1 vector such that d 2 C.X0 /. Define f .a/ D a0 V a 2b0 a, where V is an n n symmetric nonnegative definite matrix and b is an n 1 vector such that b 2 C.V; X/. Further, define g.a/ D a0 .V C W /a 2.b C c/0 a, where W is any n n matrix such that C.W / C.X/ and R.W / R.X0 / and where c is any n 1 vector in C.X/. Show that the constrained (by X0 a D d) minimization of g.a/ is equivalent to the constrained minimization of f .a/ [in the sense that g.a/ and f .a/ attain their minimum values at the same points]. 8. Let V represent an n n symmetric nonnegative definite matrix, W an n n matrix, X an n p matrix, f an n 1 vector, and d a p 1 vector. Further, let b represent an n 1 vector such that b 2 C.V; X/. Generalize Theorem 19.5.1 by showing that, for the vector W .I PX;W /f C W X.X0 W X/ d to be a solution, for every d 2 C.X0 /, to the problem of minimizing the seconddegree polynomial a0 V a 2b0 a (in a) subject to X0 a D d, it is necessary and sufficient that

and

V W f b 2 C.X/;

(E.3)

C.V W X/ C.X/;

(E.4)

rank.X0 W X/ D rank.X/:

(E.5)

9. Let V and W represent n n matrices, and let X represent an n p matrix. Show that if V and W are nonsingular, then the condition C.V W X/ C.X/ is equivalent to the condition C.V 1 X/ C.W X/ and is also equivalent to the condition C.W 1 V 1 X/ C.X/. 10. Let V represent an n n symmetric positive definite matrix, W an n n matrix, X an n p matrix, and d a p 1 vector. Show that, for the vector W X.X0 W X/ d to be a solution, for every d 2 C.X0 /, to the problem of minimizing the quadratic form a0 V a (in a) subject to X0 a D d, it is necessary and sufficient that V 1 PX;W 0 be symmetric, and rank.X0 W X/ D rank.X/. 0 1 Show that it is also necessary and sufficient that .I PX;W PX;W 0 D 0 0 /V 0 and rank.X W X/ D rank.X/. 11. Let V represent an n n symmetric nonnegative definite matrix, X an n p matrix, and d a p 1 vector. Show that each of the following six conditions is necessary and sufficient for the vector X.X0 X/ d to be a solution for every d 2 C.X0 / to the problem of minimizing the quadratic form a0 V a (in a) subject to X0 a D d:


495

(a) C.V X/ C.X/ (or, equivalently, V X D XQ for some matrix Q); (b) PX V .I PX / D 0 (or, equivalently, PX V D PX V PX ); (c) PX V D V PX (or, equivalently, PX V is symmetric); (d) C.V PX / C.PX /; (e) C.V PX / D C.V / \ C.PX /; (f) C.V X/ D C.V / \ C.X/. 12. Let V represent an n n symmetric nonnegative definite matrix, W an n n matrix, X an np matrix, and d a p1 vector. Further, let K represent any nq matrix such that C.K/ D C.IPX;W 0 /. Show that if rank.X0 W X/ D rank.X/, then each of the following two conditions is necessary and sufficient for the vector W X.X0 W X/ d to be a solution, for every d 2 C.X0 /, to the problem of minimizing the quadratic form a0 V a (in a) subject to X0 a D d: (a) V D XR1 X0 C .I PX;W 0 /R2 .I PX;W 0 /0

for some p p matrix R1 and some n n matrix R2 ;

(b) V D XS1 X0 C KS2 K0 for some p p matrix S1 and some q q matrix S2 . And show that if rank.X0 W X/ D rank.X/ and W is nonsingular, then another necessary and sufficient condition is: (c) V D t W 1 C XT1 X0 C KT2 K0 for some scalar t , some p p matrix T1 , and some q q matrix T2 . [Hint. To establish the necessity of Condition (a), begin by observing that V D CC 0 for some matrix C and by expressing C as C D PX;W 0 CC.IPX;W 0 /C.]

Bibliographic and Supplementary Notes §2–§5. Many of the results presented in Sections 2–5 are adaptations of results reported by Rao (1989), Baksalary and Puntanen (1989), and Puntanen and Styan (1989). §6. The approach taken in Section 6 is essentially the same as that taken by Albert (1972, sec. 6.1).

20 The Moore-Penrose Inverse

By definition, a generalized inverse of an m n matrix A is any n m matrix G such that AGA D A. Except for the special case where A is a (square) nonsingular matrix, A has an infinite number of generalized inverses (as discussed in Section 9.2a). While for many purposes one generalized inverse is as good as another, there is a unique one of the generalized inverses, known as the Moore-Penrose inverse, that is sometimes singled out for special attention and that is the primary subject of the present chapter.

20.1 Definition, Existence, and Uniqueness (of the Moore-Penrose Inverse) The Moore-Penrose inverse of an m n matrix A can be described in terms of the conditions given by the following theorem. Theorem 20.1.1. Corresponding to any m n matrix A, there is a unique n m matrix G such that (1) AGA D A (i.e., G is a generalized inverse of A); (2) GAG D G (i.e., A is a generalized inverse of G); (3) .AG/0 D AG (i.e., AG is symmetric); and (4) .GA/0 D GA (i.e., GA is symmetric). Moreover, if A D 0, then G D 0; and if A ¤ 0, then G D T 0 .B0 AT 0 /1 B0 ;

(1.1)

where B is any matrix of full column rank and T any matrix of full row rank such that A D BT . In connection with Theorem 20.1.1, note that the existence of a matrix B of full column rank and a matrix T of full row rank such that A D BT follows from Theorem 4.4.8. Note also that B0 AT 0 D B0 BT T 0 and that [since rank.B0 B/ D

498


rank.B/ and rank.T T 0 / D rank.T /] B0 B and T T 0 are nonsingular, implying in particular that B0 AT 0 is nonsingular. Further, .B0 AT 0 /1 D .T T 0 /1 .B0 B/1, so that, as an alternative to formula (1.1), we have the formula G D T 0 .T T 0 /1 .B0 B/1 B0 :

(1.2)

Proof (of Theorem 20.1.1). If A D 0, then it is obvious that Conditions (1)–(4) are satisfied by taking G D 0. Suppose now that A ¤ 0, and take G D T 0 .B0 AT 0 /1 B0 or, equivalently, take G D T 0 .T T 0 /1 .B0 B/1 B0 . Then, AGA D BT T 0 .T T 0 /1 .B0 B/1 B0 BT D BT D A; and GAG D T 0 .T T 0 /1 .B0 B/1 B0 BT T 0 .T T 0 /1 .B0 B/1 B0 D T 0 .T T 0 /1 .B0 B/1 B0 D G; so that Conditions (1) and (2) are satisfied. Further, AG D BT T 0 .T T 0 /1 .B0 B/1 B0 D B.B0 B/1 B0 D PB ; and

GA D T 0 .T T 0 /1 .B0 B/1 B0 BT D PT 0 ;

implying [in light of Part (3) of Theorem 12.3.4] that AG and GA are symmetric. We have established that Conditions (1)–(4) are satisfied by taking G D G , where G is the n m matrix defined as follows: G D 0, if A D 0; G D T 0 .B0 AT 0 /1 B0 , if A ¤ 0. Moreover, for any n m matrix G that satisfies Conditions (1)–(4), G D GAG D G.AG/0 D GG 0 A0 D GG 0 .AG A/0 D GG 0 A0 .AG /0 D GG 0 A0 AG D G.AG/0 AG D GAGAG D GAG D GAG AG D .GA/0 .G A/0 G D A0 G 0 A0 G0 G D .AGA/0 G0 G D A0 G0 G D .G A/0 G D G AG D G : Thus, there is no choice for G other than G D G that satisfies Conditions (1)– (4). Q.E.D. The nm matrix G defined (uniquely) by Conditions (1)–(4) of Theorem 20.1.1 is called the Moore-Penrose inverse (or Moore-Penrose pseudoinverse) of A. (The practice of referring to this matrix as the Moore-Penrose inverse is widespread, although not universal—some other authors refer to this matrix as the generalized inverse or the pseudoinverse.) The existence and uniqueness of the n m matrix G that satisfies Conditions (1)–(4) was established by Penrose (1955). An earlier effort by Moore (1920) to extend the concept of an inverse matrix to (square) singular matrices and to rectangular matrices had led to a different characterization of the same matrix.

20.2 Some Special Cases

499

Conditions (1)–(4) are referred to as the Penrose conditions or (less commonly and perhaps less appropriately) as the Moore-Penrose conditions. The symbol AC is used to denote the Moore-Penrose inverse of a matrix A. Clearly, AC is a particular one of what may be an infinite number of generalized inverses of A.

20.2 Some Special Cases For any n n nonsingular matrix A, AC D A1; as is evident from Lemma 9.1.1 or upon recalling that AA1 D A1 A D In and verifying that A1 satisfies the Penrose conditions. More generally, for any m n matrix A of rank m (i.e., of full row rank), AC D A0 .AA0 /1

(2.1)

—this result is a special case of result (1.1) (that where B D Im and T D A). Similarly, for any m n matrix A of rank n (i.e., of full column rank) AC D .A0 A/1 A0 :

(2.2)

For an n n diagonal matrix D D fdi g, D C D diag.d1C; d2C; : : : ; dnC /; where (for i D 1; : : : ; n) ( diC D

1=di ; 0;

if di ¤ 0, if di D 0.

More generally, for any matrices A1 ; A2 ; : : : ; Ak , C C Œdiag.A1 ; A2 ; : : : ; Ak /C D diag.AC 1 ; A2 ; : : : ; Ak /;

as is easily verified. The following lemmas are easy to verify. Lemma 20.2.1. For any symmetric idempotent matrix A, AC D A. Lemma 20.2.2. For any m n matrix P whose n columns form an orthonormal (with respect to the usual inner product) set (of m-dimensional vectors) or, equivalently, for any m n matrix P such that P 0 P D In , P C D P 0.

500


20.3 Special Types of Generalized Inverses If in choosing a generalized inverse G of a matrix A [or, equivalently, a matrix G that satisfies Penrose Condition (1)] we require that G satisfy Penrose Conditions (2)–(4), we obtain the Moore-Penrose inverse AC of A. A less restrictive approach is to require only that G satisfy a particular one or a particular two of Penrose Conditions (2)–(4). Generalized inverses that satisfy Penrose Condition (2), (4), or (3) are considered in Subsections a, b, and c, respectively. a. Reflexive generalized inverses A generalized inverse G of a matrix A is said to be reflexive if GAG D G. Thus, an n m matrix G is a reflexive generalized inverse of an m n matrix A if it satisfies Penrose Conditions (1) and (2). The following theorem gives an expression for the general form of a reflexive generalized inverse. Theorem 20.3.1. Let A represent an mn nonnull matrix, and let B represent any matrix of full column rank and T any matrix of full row rank such that A D BT . Then, an n m matrix G is a reflexive generalized inverse of A if and only if G D RL for some right inverse R of T and some left inverse L of B. Proof. Suppose that G D RL for some right inverse R of T and some left inverse L of B. Then, according to Theorem 9.1.3, G is a generalized inverse of A. Further, GAG D RLBT RL D RIIL D RL D G; so that G is reflexive. Conversely, suppose that G is a reflexive generalized inverse of A. Then, BT GBT D AGA D A D BT D BIT ; and it follows from Part (3) of Lemma 8.3.1 that T GB D I. Thus, GB is a right inverse of T , and T G is a left inverse of B. Further, G D GAG D .GB/T G: Q.E.D. Any nonreflexive generalized inverse of a matrix A can be converted into a reflexive generalized inverse by making use of the corollary of the following lemma. Lemma 20.3.2. Let G1 and G2 represent any (possibly different) generalized inverses of a matrix A. Then, the matrix G1 AG2 is a reflexive generalized inverse of A. Proof. That G1 AG2 is a reflexive generalized inverse of A is evident upon observing that A.G1 AG2 /A D .AG1 A/G2 A D AG2 A D A;

20.3 Special Types of Generalized Inverses

501

and .G1 AG2 /A.G1 AG2 / D G1 .AG2 A/G1 AG2 D G1 AG1 AG2 D G1 AG2 : Q.E.D. Corollary 20.3.3. Let G represent any generalized inverse of a matrix A. Then, the matrix GAG is a reflexive generalized inverse of A. Proof. The corollary is essentially a special case of Lemma 20.3.2, namely, Q.E.D. the special case where G2 D G1 . Note (as a rather obvious converse of Corollary 20.3.3) that any reflexive generalized inverse B of a matrix A is expressible in the form B D GAG for some generalized inverse G of A—in particular, B D GAG for G D B. A very basic characteristic of reflexive generalized inverses is described in the following theorem. Theorem 20.3.4. A generalized inverse G of a matrix A is reflexive if and only if rank.G/ D rank.A/. Proof. Suppose that G is reflexive. Then, as a consequence of Lemma 9.3.9, rank.G/ rank.A/ and also (since A is a generalized inverse of G) rank.A/ rank.G/. Thus, rank.G/ D rank.A/. Conversely, suppose that rank.G/ D rank.A/. Then, in light of Lemma 10.2.5 and of Corollary 9.3.8 (or in light of Theorem 10.2.7), GA is idempotent, and rank.GA/ D rank.A/ D rank.G/: Thus, it follows from Theorem 10.2.7 that A is a generalized inverse of G (i.e., that GAG D G) and hence that G is a reflexive generalized inverse of A. Q.E.D. If an n m matrix G is a generalized inverse of an m n matrix A, then (according to Lemma 9.3.3) G 0 is a generalized inverse of A0. Similarly, if A is a generalized inverse of G, then A0 is a generalized inverse of G 0. Thus, we have the following lemma. Lemma 20.3.5. If an n m matrix G is a reflexive generalized inverse of an m n matrix A, then G 0 is a reflexive generalized inverse of A0. b. Minimum norm generalized inverses A generalized inverse G of a matrix A is said to be a minimum norm generalized inverse (of A) if .GA/0 D GA. Thus, an n m matrix G is a minimum norm generalized inverse of an m n matrix A if it satisfies Penrose Conditions (1) and (4). The rationale for referring to such a matrix as a minimum norm generalized inverse comes from the following theorem. Theorem 20.3.6. Let A represent an m n matrix and G an n m matrix. If G is a minimum norm generalized inverse of A, then, for every b 2 C.A/, kxk attains its minimum value over the set fx W Ax D bg [comprising all solutions

502


to the linear system Ax D b (in x)] uniquely at x D Gb (where the norm is the usual norm). Conversely, if, for every b 2 C.A/, kxk attains its minimum value over the set fx W Ax D bg at x D Gb, then G is a minimum norm generalized inverse of A. Proof. Let H represent a minimum norm generalized inverse of A. And let x represent an arbitrary solution to Ax D b [where b 2 C.A/]. Then, according to Theorem 11.2.4, x D Hb C .I HA/y for some vector y. Further, kxk2 D ŒHb C .I HA/y ŒHb C .I HA/y D kHbk2 C k.I HA/yk2 C 2.Hb/ .I HA/y (where the inner product is the usual inner product). And, since b D Ar for some vector r and since (according to Lemma 10.2.5) HA is idempotent, .Hb/ .I HA/y D .HAr/ .I HA/y D .HAr/0 .I HA/y D r 0 .HA/0 .I HA/y D r 0 HA.I HA/y D r 0 ŒHA .HA/2 y D r 0 .HA HA/y D 0: Thus,

kxk2 D kHbk2 C k.I HA/yk2 ;

so that kxk2 kHbk2 , or, equivalently, kxk kHbk, with equality holding if and only if k.I HA/yk D 0. Since k.I HA/yk D 0 if and only if .I HA/y D 0, and hence if and only if x D Hb (and since Hb is a solution to Ax D b), we conclude that kxk attains its minimum value over the set fx W Ax D bg uniquely at x D Hb. To complete the proof, suppose that, for every b 2 C.A/, kxk attains its minimum value over the set fx W Ax D bg at x D Gb. Then, clearly, Gb D Hb for every b 2 C.A/, or, equivalently, GAr D HAr for every .n 1/ vector r, implying (in light of Lemma 2.3.2) that GA D HA. Thus, .GA/0 D .HA/0 D HA D GA. And, since (in light of Theorem 9.1.2) G is a generalized inverse of A, we conclude that G is a minimum norm generalized inverse of A. Q.E.D. Two alternative characterizations of a minimum norm generalized inverse are given by the following theorem and corollary. Theorem 20.3.7. An n m matrix G is a minimum norm generalized inverse of an m n matrix A if and only if GAA0 D A0 , or, equivalently, if and only if AA0 G 0 D A. Proof. Suppose that GAA0 D A0 or, equivalently, that AA0 G 0 D A. Then, AGAA0 D AA0 , implying (in light of Corollary 5.3.3) that AGA D A. Moreover,


503

GA D GAA0 G 0 D GA.GA/0, so that GA is symmetric. Thus, G is a minimum norm generalized inverse of A. Conversely, if G is a minimum norm generalized inverse of A, then GAA0 D .GA/0 A0 D A0 G 0 A0 D .AGA/0 D A0 : Q.E.D. Corollary 20.3.8. An n m matrix G is a minimum norm generalized inverse of an m n matrix A if and only if GA D PA0 , or, equivalently, if and only if A0 G 0 D PA0 . Proof. In light of Theorem 20.3.7, it suffices to show that GA D PA0 , GAA0 D A0 . If GA D PA0 , then [in light of Part (1) of Theorem 12.3.4] GAA0 D PA0 A0 D 0 A . Conversely, if GAA0 D A0, then [in light of Part (5) of Theorem 12.3.4] GA D GAA0 .AA0 / A D A0 .AA0 / A D PA0 : Q.E.D. Let A represent an mn matrix. Then, for any generalized inverse G of A, GA is idempotent, R.GA/ D R.A/, and rank.GA/ D rank.A/, as was established earlier (in Lemmas 10.2.5 and 9.3.7 and Corollary 9.3.8). Further, for any minimum norm generalized inverse G of A, GA is (according to the very definition of a minimum norm generalized inverse) symmetric. Two other properties of the matrix product GA obtained by premultiplying A by a minimum norm generalized inverse G (of A) are described in the following, additional corollary of Theorem 20.3.7. Corollary 20.3.9. Let G represent a minimum norm generalized inverse of a matrix A. Then, (1) GA is invariant to the choice of G; and (2) C.GA/ D C.A0 /. Proof. According to Corollary 20.3.8, GA D PA0 . Thus, GA is invariant to the choice of G, and [in light of Part (7) of Theorem 12.3.4] C.GA/ D C.PA0 / D C.A0 /. Q.E.D. A generalized inverse G of a matrix A is said to be a minimum norm reflexive generalized inverse (of A) if it is a minimum norm generalized inverse (of A) and it is reflexive. Thus, an n m matrix G is a minimum norm reflexive generalized inverse of an m n matrix A if it satisfies Penrose Conditions (1), (2), and (4). The following lemma gives a basic property of a minimum norm reflexive generalized inverse. Lemma 20.3.10. For any minimum norm reflexive generalized inverse G of a matrix A, C.G/ D C.A0 /. Proof. Recalling Corollary 4.2.3, it follows from Part (2) of Corollary 20.3.9 that C.A0 / C.G/. Moreover, in light of Theorem 20.3.4, rank.G/ D rank.A/ D rank.A0 /. Thus, as a consequence of Theorem 4.4.6, we have that Q.E.D. C.G/ D C.A0 /.

504


Since [according to Part (1) of Theorem 12.3.4] A0 .AA0 / AA0 D A0, it follows from Theorem 20.3.7 that the matrix A0 .AA0 / is a minimum norm generalized inverse of A. The following theorem makes a stronger statement. Theorem 20.3.11. An n m matrix G is a minimum norm reflexive generalized inverse of an m n matrix A if and only if G D A0 H for some generalized inverse H of AA0 . Proof. According to Part (1) of Theorem 12.3.4, A0 .AA0 / AA0 D A0 . Thus, as previously indicated, it follows from Theorem 20.3.7 that the matrix A0 .AA0 / is a minimum norm generalized inverse of A. Further, ŒA0 .AA0 / AŒA0 .AA0 / D ŒA0 .AA0 / AA0 .AA0 / D A0 .AA0 /; so that A0 .AA0 / is a minimum norm reflexive generalized inverse of A. Now, suppose that G is a minimum norm reflexive generalized inverse of A. Then, it follows from Lemma 20.3.10 that C.G/ D C.A0 / and hence that there exists a matrix H such that G D A0 H. And A D AGA D AA0 HA, implying that Q.E.D. AA0 D AA0 HAA0, so that H is a generalized inverse of AA0 . c. Least squares generalized inverses A generalized inverse G of a matrix A is said to be a least squares generalized inverse (of A) if .AG/0 D AG. Thus, an n m matrix G is a least squares generalized inverse of an m n matrix A if it satisfies Penrose Conditions (1) and (3). Two alternative characterizations of a least squares generalized inverse are given by the following theorem and corollary—these characterizations are analogous to the characterizations of a minimum norm generalized inverse given, respectively, by Theorem 20.3.7 and Corollary 20.3.8. Theorem 20.3.12. An n m matrix G is a least squares generalized inverse of an m n matrix A if and only if A0 AG D A0, or, equivalently, if and only if G 0 A0 A D A. Proof. Suppose that A0 AG D A0 or, equivalently, that G 0 A0 A D A. Then, 0 A AGA D A0 A, implying (in light of Corollary 5.3.3) that AGA D A. Moreover, AG D G 0 A0 AG D .AG/0 AG, so that AG is symmetric. Thus, G is a least squares generalized inverse of A. Conversely, if G is a least squares generalized inverse of A, then A0 AG D A0 .AG/0 D .AGA/0 D A0 : Q.E.D. Corollary 20.3.13. An n m matrix G is a least squares generalized inverse of an m n matrix A if and only if AG D PA . Proof. In light of Theorem 20.3.12, it suffices to show that AG D PA , A0 AG D A0 .


505

If AG D PA , then [in light of Part (5) of Theorem 12.3.4] A0 AG D A0 PA D A . Conversely, if A0 AG D A0 , then [in light of Part (1) of Theorem 12.3.4] 0

AG D A.A0 A/ A0 AG D A.A0 A/ A0 D PA : Q.E.D. The aptness of our use of the term least squares generalized inverse is evident from the following corollary (of Theorem 20.3.12). Corollary 20.3.14. A p n matrix G is a least squares generalized inverse of an n p matrix X if and only if, for every y 2 Rn1, the norm ky Xbk (of the difference between y and Xb) attains its minimum value (with respect to the p 1 vector b) at b D Gy (where the norm is the usual norm), or, equivalently, if and only if, for every y 2 Rn1, the sum of squares .y Xb/0 .y Xb/ (of the elements of the difference between y and Xb) attains its minimum value at b D Gy. Proof. In light of Theorem 20.3.12, it suffices to show that X0 XG D X0 if and only if, for every y 2 Rn1, .y Xb/0 .y Xb/ attains its minimum value at b D Gy. Or, equivalently (in light of Theorem 12.4.3), it suffices to show that X0 XG D X0 if and only if, for every y 2 Rn1, the vector Gy is a solution to the normal equations X0 Xb D X0 y (i.e., if and only if, for every y 2 Rn1, X0 XGy D X0 y). If, for every y 2 Rn1, X0 XGy D X0 y, then it is evident from Lemma 2.3.2 that X0 XG D X0 . Conversely, if X0 XG D X0, then it is obvious that, for every Q.E.D. y 2 Rn1, X0 XGy D X0 y. Let A represent an mn matrix. Then, for any generalized inverse G of A, AG is idempotent, C.AG/ D C.A/, and rank.AG/ D rank.A/, as was established earlier (in Lemmas 10.2.5 and 9.3.7 and Corollary 9.3.8). Further, for any least squares generalized inverse G of A, AG is (according to the very definition of a least squares generalized inverse) symmetric. Two other properties of the matrix product AG obtained by postmultiplying A by a least squares generalized inverse G (of A) are described in the following, additional corollary of Theorem 20.3.12. Corollary 20.3.15. Let G represent a least squares generalized inverse of a matrix A. Then, (1) AG is invariant to the choice of G; and (2) R.AG/ D R.A0 /. Proof. According to Corollary 20.3.13, AG D PA . Thus, AG is invariant to the choice of G, and [in light of Part (7) of Theorem 12.3.4] R.AG/ D R.PA / D R.A0 /. Q.E.D. A generalized inverse G of a matrix A is said to be a least squares reflexive generalized inverse (of A) if it is a least squares generalized inverse (of A) and it is reflexive. Thus, an n m matrix G is a least squares reflexive generalized inverse of an m n matrix A if it satisfies Penrose Conditions (1), (2), and (3). The following lemma gives some basic properties of a least squares reflexive generalized inverse.

506


Lemma 20.3.16. For any least squares reflexive generalized inverse G of a matrix A, R.G/ D R.A0 / and N.G/ D C? .A/ D N.A0 /. Proof. Recalling Corollary 4.2.3, it follows from Part (2) of Corollary 20.3.15 that R.A0 / R.G/. Moreover, in light of Theorem 20.3.4, rank.G/ D rank.A/ D rank.A0 /. Thus, as a consequence of Theorem 4.4.6, we have that R.G/ D R.A0 /. Further, it follows from Part (2) of Lemma 4.2.5 that C.G 0 / D C.A/. And, making use of Lemma 12.5.2, we find that N.G/ D C? .G 0 / D C? .A/ D N.A0 /: Q.E.D. Since [according to Part (5) of Theorem 12.3.4] A A.A A/ A D A , it follows from Theorem 20.3.12 that the matrix .A0 A/ A0 is a least squares generalized inverse of A. The following theorem makes a stronger statement. 0

0

0

0

Theorem 20.3.17. An n m matrix G is a least squares reflexive generalized inverse of an m n matrix A if and only if G D HA0 for some generalized inverse H of A0 A. Proof. According to Part (5) of Theorem 12.3.4, A0 A.A0 A/ A0 D A0 . Thus, as previously indicated, it follows from Theorem 20.3.12 that the matrix .A0 A/ A0 is a least squares generalized inverse of A. Further, Œ.A0 A/ A0 AŒ.A0 A/ A0 D .A0 A/ ŒA0 A.A0 A/ A0 D .A0 A/ A0; so that .A0 A/ A0 is a least squares reflexive generalized inverse of A. Now, suppose that G is a least squares reflexive generalized inverse of A. Then, it follows from Lemma 20.3.16 that R.G/ D R.A0 / and hence that there exists a matrix H such that G D HA0 . And A D AGA D AHA0 A, implying that Q.E.D. A0 A D A0 AHA0 A, so that H is a generalized inverse of A0 A. d. Some equivalences The four Penrose conditions can be reexpressed in accordance with the equivalences given by the following, easily verifiable lemma. Lemma 20.3.18. Let A represent an m n matrix and G an n m matrix. Then, (1) AGA D A , A0 G 0 A0 D A0 ; (2) GAG D G , G 0 A0 G 0 D G 0 ; (3) .AG/0 D AG , .G 0 A0 /0 D G 0 A0 ; and (4) .GA/0 D GA , .A0 G 0 /0 D A0 G 0 . As an immediate consequence of Lemma 20.3.18, we have the following corollary (which expands on Lemma 20.3.5).

20.4 Some Alternative Representations and Characterizations

507

Corollary 20.3.19. An n m matrix G is a reflexive generalized inverse of an m n matrix A if and only if G 0 is a reflexive generalized inverse of A0 . Further, G is a least squares generalized inverse of A if and only if G 0 is a minimum norm generalized inverse of A0 and is a least squares reflexive generalized inverse of A if and only if G 0 is a minimum norm reflexive generalized inverse of A0 .

20.4 Some Alternative Representations and Characterizations The Moore-Penrose inverse can be characterized in the ways described in the following theorem and two corollaries. Theorem 20.4.1. An nm matrix G is the Moore-Penrose inverse of an mn matrix A if and only if G is a least squares generalized inverse of A and A is a least squares generalized inverse of G. Proof. By definition, G is a least squares generalized inverse of A if and only if AGA D A and .AG/0 D AG [which are Penrose Conditions (1) and (3)], and A is a least squares generalized inverse of G if and only if GAG D G and .GA/0 D GA [which, in the relevant context, are Penrose Conditions (2) and (4)]. Thus, G is the Moore-Penrose inverse of A if and only if G is a least squares generalized inverse of A and A is a least squares generalized inverse of G. Q.E.D. Corollary 20.4.2. An n m matrix G is the Moore-Penrose inverse of an m n matrix A if and only if A0 AG D A0 and G 0 GA D G 0 . Proof. Corollary 20.4.2 follows from Theorem 20.4.1 upon observing (in light of Theorem 20.3.12) that G is a least squares generalized inverse of A if and only if A0 AG D A0 and that A is a least squares generalized inverse of G if and only Q.E.D. if G 0 GA D G 0 . Corollary 20.4.3. An n m matrix G is the Moore-Penrose inverse of an m n matrix A if and only if AG D PA and GA D PG . Proof. Corollary 20.4.3 follows from Theorem 20.4.1 upon observing (in light of Corollary 20.3.13) that G is a least squares generalized inverse of A if and only if AG D PA and that A is a least squares generalized inverse of G if and only if Q.E.D. GA D PG . The following theorem expresses the Moore-Penrose inverse of a matrix A in terms of solutions to the two linear systems A0 AX D A0 and AA0 Y D A. Theorem 20.4.4. For any m n matrix A, AC D Y0 AX ;

(4.1)

where X is any solution to the (consistent) linear system A0 AX D A0 (in an nm matrix X) and Y is any solution to the (consistent) linear system AA0 Y D A (in an m n matrix Y ). Proof. Note (in light of Corollary 7.4.2) that the two linear systems A0 AX D 0 A and AA0 Y D A are consistent. And observe (in light of Theorems 20.3.12

508


and 20.3.7) that AX A D A and .AX /0 D AX and that AY0 A D A and .Y0 A/0 D Y0 A. Thus, A.Y0 AX /A D AY0 .AX A/ D AY0 A D AI .Y0 AX /A.Y0 AX / D Y0 .AX A/Y0 AX D Y0 AY0 AX D Y0 AX I ŒA.Y0 AX /0 D Œ.AY0 A/X 0 D .AX /0 D AX D .AY0 A/X D A.Y0 AX /I Œ.Y0 AX /A0 D ŒY0 .AX A/0 D .Y0 A/0 D Y0 A D Y0 .AX A/ D .Y0 AX /A: We conclude that Y0 AX satisfies all four of the Penrose conditions and hence Q.E.D. that AC D Y0 AX . 0 0 0 The matrix .A A/ A is one solution to the linear system A AX D A0 and {since AA0 is symmetric and hence since Œ.AA0 / 0 is a generalized inverse of AA0 } the matrix Œ.AA0 / 0 A is one solution to the linear system AA0 Y D A. Thus, as a special case of formula (4.1) {that obtained by setting X D .A0 A/ A0 and Y D Œ.AA0 / 0 A}, we have that AC D A0 .AA0 / A.A0 A/ A0 :

(4.2)

Note that the right side of equality (4.2) can be reexpressed as A0 .AA0 / AA A.A0 A/ A0, giving rise to the alternative representation AC D PA0 A PA :

(4.3)

The following theorem gives three expressions for the Moore-Penrose inverse of a (nonnull) symmetric nonnegative definite matrix. Theorem 20.4.5. Let A represent an n n (nonnull) symmetric nonnegative definite matrix. Then, for any matrix T of full row rank such that A D T 0 T , AC D T 0 .T AT 0 /1 T 0

0 2

D T .T T / C

T

C 0

D T .T /

(4.4) (4.5) (4.6)

{where .T T 0 /2 D Œ.T T 0 /1 2 }. In connection with Theorem 20.4.5, note that Theorem 14.3.7 guarantees the existence of a matrix T of full row rank such that A D T 0 T . Proof. Results (4.4) and (4.5) are special cases of results (1.1) and (1.2), respectively. Specifically, they are the special cases obtained by setting B D T 0 . And, in light of result (2.1) [and the symmetry of .T T 0 /1 ], result (4.6) follows from result (4.5). Q.E.D.

20.5 Some Basic Properties and Relationships The Moore-Penrose inverse of a matrix A is, by definition, a reflexive generalized inverse, a minimum norm generalized inverse, a least squares generalized inverse,

20.5 Some Basic Properties and Relationships

509

a minimum norm reflexive generalized inverse, and a least squares reflexive generalized inverse (of A). Accordingly, certain properties of Moore-Penrose inverses can be deduced from the properties (discussed in Section 3) of reflexive, minimum norm, least squares, minimum norm reflexive, and least squares reflexive generalized inverses. In particular, the following theorem describes some properties of Moore-Penrose inverses that are immediate consequences of Theorem 20.3.4 and of Corollaries 20.3.8 and 20.3.13 and Lemmas 20.3.10 and 20.3.16. Theorem 20.5.1. For any matrix A, (1) rank.AC / D rank.A/; (2) AC A D PA0 and AAC D PA ;

(3) C.AC / D C.A0 / and R.AC / D R.A0 /; and (4) N.AC / D C? .A/ D N.A0 /. In light of Parts (3) and (6) of Theorem 12.3.4, we have the following corollary of Theorem 20.5.1. Corollary 20.5.2. For any matrix A, AC A and AAC are symmetric and idempotent. Some additional properties of Moore-Penrose inverses are described in the following theorem. Theorem 20.5.3. Let A represent an m n matrix. Then,

(1) .A0 /C D .AC /0 ; (2) if A is symmetric, then AC is symmetric; (3) if A is symmetric and positive semidefinite, then AC is symmetric and positive semidefinite; if A is symmetric and positive definite, then AC is symmetric and positive definite (and if A is symmetric and nonnegative definite, then AC is symmetric and nonnegative definite); (4) .AC /C D A; and (5) for any nonzero scalar c, .cA/C D .1=c/AC. Proof. (1) It is clear from Lemma 20.3.18 that .AC /0 satisfies all four of the Penrose conditions (as applied to the matrix A0 ) and hence that .AC /0 is the Moore-Penrose inverse of A0 . (2) Part (2) is an immediate consequence of Part (1). (3) Suppose that A is symmetric and nonnegative definite. Then, according to Part (2), AC is symmetric. And, since (according to Theorem 20.5.1) rank.AC / D rank.A/, AC is (in light of Corollary 14.3.12) singular if A is positive semidefinite and nonsingular if A is positive definite. Further, AC D AC AAC D .AC /0 AAC. Thus, it follows from Parts (2) and (3) of Theorem 14.2.9 that AC is positive semidefinite if A is positive semidefinite and positive definite if A is positive definite. (4) That .AC /C D A is evident from the definition of the Moore-Penrose inverse (as a matrix that satisfies the four Penrose conditions).

510


(5) Clearly, cAŒ.1=c/AC cA D cAAC A D cA; Œ.1=c/AC cAŒ.1=c/AC D .1=c/AC AAC D .1=c/AC; fcAŒ.1=c/AC g0 D .AAC /0 D AAC D cAŒ.1=c/AC ; and fŒ.1=c/AC cAg0 D .AC A/0 D AC A D Œ.1=c/AC cA; so that .1=c/AC satisfies Penrose Conditions (1)–(4) (as applied to the matrix cA). Q.E.D. Note that, as a special case of Part (5) of Theorem 20.5.3 (that where c D 1), we have that, for any matrix A, .A/C D AC:

(5.1)

The Moore-Penrose inverse of a matrix A is related to minimum norm generalized inverses of A0 A and to least squares generalized inverses of AA0, as described in the following theorem. Theorem 20.5.4. Let A represent an m n matrix. Then, for any least squares generalized inverse F of AA0, AC D A0 F , and, for any minimum norm generalized inverse H of A0 A, AC D HA0 . Proof. According to Theorem 20.3.11, A0 F is a minimum norm reflexive generalized inverse of A, and, according to Theorem 20.3.17, HA0 is a least squares reflexive generalized inverse of A. Moreover, .AA0 F /0 D AA0 F and .HA0 A/0 D HA0 A, so that A0 F satisfies Penrose Condition (3) and HA0 satisfies Penrose Condition (4). Thus, both A0 F and HA0 satisfy all four Penrose conditions, Q.E.D. and hence both equal AC . As an immediate consequence of Theorem 20.5.4, we have the following corollary, which relates the Moore-Penrose inverse of a matrix A to the Moore-Penrose inverse of AA0 and to the Moore-Penrose inverse of A0 A. Corollary 20.5.5. For any matrix A, AC D A0 .AA0 /C D .A0 A/C A0 :

(5.2)

The following theorem relates the Moore-Penrose inverse of a matrix of the form P AQ0, where P and Q are orthogonal matrices (or more generally where P and Q are such that P 0 P D I and Q0 Q D I), to the Moore-Penrose inverse of A itself. Theorem 20.5.6. Let A represent an m n matrix. Then, for any r m matrix P whose columns form an orthonormal (with respect to the usual inner product) set (of r-dimensional vectors) and any s n matrix Q whose columns form an orthonormal (with respect to the usual inner product) set (of s-dimensional vectors) or, equivalently, for any r m matrix P such that P 0 P D Im and any s n matrix Q such that Q0 Q D In , .P AQ0 /C D QAC P 0 :

20.5 Some Basic Properties and Relationships

511

Proof. It suffices to show that QAC P 0 satisfies the four Penrose conditions. Clearly, (1) P AQ0 .QAC P 0 /P AQ0 D P AAC AQ0 D P AQ0 ; (2) QAC P 0 .P AQ0 /QAC P 0 D QAC AAC P 0 D QAC P 0 ; (3) ŒP AQ0 .QAC P 0 /0 D .P AAC P 0 /0 D P .AAC /0 P 0 D P AAC P 0 D P AQ0 .QAC P 0 /; (4) ŒQAC P 0 .P AQ0 /0 D .QAC AQ0 /0 D Q.AC A/0 Q0 D QAC AQ0 D QAC P 0 .P AQ0 /. Q.E.D. For any m n matrix A and any p q matrix B, A ˝ B is a generalized inverse of the Kronecker product A ˝ B (as discussed in Section 16.1). This result is generalized in the following theorem.

Theorem 20.5.7. Let A represent an m n matrix, B a p q matrix, G an n m matrix, and H a q p matrix. Then, G ˝ H satisfies any of the Penrose Conditions (1)–(4) (when the Penrose conditions are applied to the matrix A ˝ B) that are satisfied by both G and H (when the Penrose conditions are applied to A and B, respectively). Proof. (1) If AGA D A and BHB D B, then (as indicated in Section 16.1) .A ˝ B/.G ˝ H/.A ˝ B/ D .AGA/ ˝ .BHB/ D A ˝ B: (2) Similarly, if GAG D G and HBH D H, then .G ˝ H/.A ˝ B/.G ˝ H/ D .GAG/ ˝ .HBH/ D G ˝ H: (3) If .AG/0 D AG and .BH/0 D BH, then Œ.A ˝ B/.G ˝ H/0 D Œ.AG/ ˝ .BH/0 D .AG/0 ˝ .BH/0 D .AG/ ˝ .BH/ D .A ˝ B/.G ˝ H/: (4) Similarly, if .GA/0 D GA and .HB/0 D HB, then Œ.G ˝ H/.A ˝ B/0 D Œ.GA/ ˝ .HB/0 D .GA/0 ˝ .HB/0 D .GA/ ˝ .HB/ D .G ˝ H/.A ˝ B/: Q.E.D. As an obvious consequence of Theorem 20.5.7, we have the following corollary. Corollary 20.5.8. For any matrices A and B, .A ˝ B/C D AC ˝ BC:

512


20.6 Minimum Norm Solution to the Least Squares Problem For any n p matrix X and any n 1 vector y, the (least squares) problem of minimizing the (usual) norm ky Xbk (of the difference between y and Xb) with respect to the p 1 vector b can be solved by taking b D Gy, where G is any least squares generalized inverse of X (as indicated in Corollary 20.3.14). One choice for G is G D XC. The solution (b D XC y) obtained by choosing G D XC has the property described in the following theorem. Theorem 20.6.1. Let X represent an n p matrix and y an n-dimensional column vector. Taking the norm of a vector to be the usual norm, define S to be the set comprising those values of a p 1 vector b at which ky Xbk attains its minimum value or, equivalently, to be the solution set fb W X0 Xb D X0 yg of the normal equations. Then, kbk attains its minimum value over the set S uniquely at b D XC y. Moreover, if G is a p n matrix such that, for every y, kbk attains its minimum value over the set S at b D Gy, then G D XC. In connection with Theorem 20.6.1, note that in general the set S varies with y (although the dependence of S on y is suppressed in the notation). Proof. That the set comprising those values of b at which ky Xbk attains its minimum value is the same as the solution set of the normal equations is evident from Theorem 12.4.3. Now, let H represent a minimum norm generalized inverse of X0 X. Then, since S D fb W X0 Xb D X0 yg, it follows from Theorem 20.3.6 that kbk attains its minimum value over the set S uniquely at b D HX0 y. And, according to Theorem 20.5.4, HX0 D XC. Thus, kbk attains its minimum value over the set S uniquely at b D XC y. Further, if G is a p n matrix such that, for every y, kbk attains its minimum value over the set S at b D Gy, then, for every y, Gy D XC y, implying (in light Q.E.D. of Lemma 2.3.2) that G D XC.

20.7 Expression of the Moore-Penrose Inverse as a Limit Let F D ffij g represent an m n matrix whose elements are functions defined on some set S of scalars. And let fij .x/ represent the value of fij at an arbitrary point x in S . Then, the m n matrix whose ij th element is fij .x/ will be referred to as the value of F at x and will be denoted by the symbol F .x/. Further, if every element of F has a limit at a point c (in R), then the m n matrix whose ij th element is lim fij .x/ (the limit of fij at c) will be referred to as the limit of F at x!c

c and will be denoted by the symbol lim F .x/. x!c

Clearly, if F is an m n matrix of functions defined on some set S of scalars and if all mn of these functions are continuous at an interior point c of S , then lim F .x/ D F .c/:

x!c

20.7 Expression of the Moore-Penrose Inverse as a Limit

513

Various properties of the limits of matrices of functions can be readily deduced from well-known properties of scalar-valued functions. In particular, for any p q matrix G of functions defined on some set S (of scalars) that have limits at a point c and for any m p matrix of constants A and any q n matrix of constants B, the matrix of functions F , defined by F D AGB, has a limit at c, and lim F .x/ D AŒ lim G.x/B:

x!c

x!c

The following theorem expresses the Moore-Penrose inverse of a matrix as a limit. Theorem 20.7.1. For any m n matrix A, AC D lim .A0 A C ı 2 In /1 A0 D lim A0 .AA0 C ı 2 Im /1: ı!0

ı!0

(7.1)

In connection with Theorem 20.7.1, note (in light of Lemmas 14.2.1 and 14.2.4 and Corollary 14.2.14) that, for ı ¤ 0, the matrices A0 A C ı 2 In and AA0 C ı 2 Im are positive definite and hence (in light of Lemma 14.2.8) nonsingular. Proof. Suppose that A ¤ 0—that the theorem is valid for A D 0 is easy to verify—and let B represent an m r matrix of full column rank r and T an r n matrix of full row rank r such that A D BT —the existence of such matrices was established in Theorem 4.4.8. And let L D B0 BT , so that A0 A D T 0 L. Further, let ı represent an arbitrary nonzero scalar [and write ı 2 for .ı 1 /2 ]. Then, since A0 A C ı 2 In D ı 2 .ı 2 T 0 L C In /, ı 2 T 0 L C In (like A0 A C ı 2 In ) is nonsingular, and .A0 A C ı 2 In /1 D ı 2 .ı 2 T 0 L C In /1: And, in light of Lemma 18.2.3, ı 2 LT 0 C Ir is nonsingular, and .A0 A C ı 2 In /1 T 0 D ı 2 .ı 2 T 0 L C In /1 T 0 D ı 2 T 0 .ı 2 LT 0 C Ir /1: Moreover, since LT 0 C ı 2 Ir D ı 2 .ı 2 LT 0 C Ir /, LT 0 C ı 2 Ir is nonsingular, and .LT 0 C ı 2 Ir /1 D ı 2 .ı 2 LT 0 C Ir /1: Thus, .A0 A C ı 2 In /1 A0 D .A0 A C ı 2 In /1 T 0 B0 D T 0 .LT 0 C ı 2 Ir /1 B0 :

(7.2)

Now, let ı represent a completely arbitrary (not necessarily nonzero) scalar. Recalling (from Section 20.1) that LT 0 (D B0 AT 0 ) is nonsingular, observe that LT 0 C ı 2 Ir is nonsingular for ı D 0 as well as for ı ¤ 0. Observe also (in light of Corollary 13.5.4) that (for i; j D 1; : : : ; r) the ij th element of .LT 0 C ı 2 Ir /1 equals ˛j i =jLT 0 C ı 2 Ir j, where ˛j i is the j i th element of the cofactor matrix of LT 0 C ı 2 Ir . It is clear from Corollary 13.7.4 (or, more basically, from the very definition of a determinant) that jLT 0 C ı 2 Ir j and the elements of the cofactor matrix of

514


LT 0 C ı 2 Ir are polynomials in ı. Thus, every element of .LT 0 C ı 2 Ir /1 is a continuous function of ı (at 0 or at any other point)—refer, for example, to Bartle (1976, sec. 20). We conclude that lim .LT 0 C ı 2 Ir /1 D .LT 0 C 02 Ir /1 D .LT 0 /1

ı!0

and hence [in light of results (7.2) and (1.1)] that lim .A0 A C ı 2 In /1A0 D T 0 Œ lim .LT 0 C ı 2 Ir /1 B0

ı!0

ı!0

D T 0 .LT 0 /1 B0 D T 0 .B0 AT 0 /1 B0 D AC: To complete the proof, observe that (for every ı) .A0 A C ı 2 In /A0 D A0 .AA0 C ı 2 Im /;

(7.3)

so that, for ı ¤ 0, A0 .AA0 C ı 2 Im /1 D .A0 A C ı 2 In /1A0; as is evident upon premultiplying both sides of equality (7.3) by .A0 A C ı 2 In /1 and postmultiplying both sides by .AA0 C ı 2 Im /1. It follows that lim A0 .AA0 C ı 2 Im /1 D lim .A0 A C ı 2 In /1A0 :

ı!0

ı!0

Q.E.D. Result (7.1) is of some relevance in statistics. It can be used in particular to relate the estimates of the parameters in a linear statistical model obtained via a Bayesian approach to those obtained by approaching the estimation of the parameters as a least squares problem. More specifically, it can be used to show that, in the limit (as the “amount of prior information decreases”), the vector of Bayesian estimates equals the minimum norm solution to the least squares problem.

20.8 Differentiation of the Moore-Penrose Inverse Earlier (in Section 15.10), some results were given on the differentiation of generalized inverses of a matrix of functions. These results were (for the most part) applicable to “any” generalized inverse. In the present section, some results are given that are specific to the Moore-Penrose inverse. The Moore-Penrose inverse of a matrix of functions is continuously differentiable under the conditions set forth in the following lemma. Lemma 20.8.1. Let F represent a p q matrix of functions, defined on a set S , of an m-dimensional column vector x. And let c represent any interior point of S at which F is continuously differentiable. If F has constant rank on some neighborhood of c, then F C is continuously differentiable at c.

20.8 Differentiation of the Moore-Penrose Inverse

515

Proof. Suppose that F has constant rank on some neighborhood of c. Then, since rank.F F 0 / D rank.F 0 F / D rank.F /, each of the matrices F F 0 and F 0 F has constant rank on some neighborhood of c. And, in light of the results of Section 15.4, each of the matrices F F 0 and F 0 F is continuously differentiable at c. Thus, it follows from Theorem 15.10.1 that there exist generalized inverses G and H of F F 0 and F 0 F , respectively, such that G and H are continuously differentiable at c. Since [according to result (4.2)] F C D F 0 GF HF 0, we conclude (in light of the results of Section 15.4) that F C is continuously differentiable at c. Q.E.D. In connection with Lemma 20.8.1, note (in light of Corollary 15.10.4) that, for F C to be continuously differentiable at c, it is necessary, as well as sufficient, that F have constant rank on some neighborhood of c. In fact, unless F has constant rank on some neighborhood of c, there is (according to Corollary 15.10.4) no generalized inverse of F that is continuously differentiable at c. The following theorem gives a formula for the partial derivatives of the MoorePenrose inverse of a matrix of functions. Theorem 20.8.2. Let F represent a p q matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Further, let c represent an interior point of S at which F is continuously differentiable, and suppose that F has constant rank on some neighborhood of c. Then, at x D c, 0 @F C @F @F C D F C F C F C .F C /0 .I F F C / @xj @xj @xj 0 @F C .I F C F / .F C /0 F C: (8.1) @xj Preliminary to proving Theorem 20.8.2, it is convenient to establish the following theorem. Theorem 20.8.3. Let F represent a p q matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Further, let c represent an interior point of S at which F is continuously differentiable, and suppose that F has constant rank on some neighborhood of c. Then, F C F and F F C are continuously differentiable at c, and (at x D c) 0 @F @F @F C F D FC .I F C F / C F C .I F C F / ; (8.2) @xj @xj @xj @F F C @F C @F C 0 D .I F F C / F C .I F F C / F : (8.3) @xj @xj @xj Proof (of Theorem 20.8.3). That F C F and F F C are continuously differentiable at c follows from Lemmas 15.4.3 and 20.8.1. Now, recall (from Corollary 20.5.2) that F C F and F F C are symmetric and idempotent, and observe (in light of Lemma 15.4.3) that (at x D c)

516


C 0 C C 0 @F C F @F F @F F F F D D @xj @xj @xj C C 0 @F F C C @F F D F F CF F @xj @xj C @F F @F C F 0 D F CF C F CF @xj @xj

(8.4)

and similarly @F F C @F F C 0 @F F C F F C 0 D D @xj @xj @xj 0 C @F F C C @F F C D FF C FF @xj @xj 0 @F F C @F F C 0 C D FF C FF : @xj @xj

(8.5)

Moreover, at x D c, @F @F F C F @F C @F C F D D F F CF ; @xj @xj @xj @xj and similarly

@F @F @F F C F @F F C D D F C FFC ; @xj @xj @xj @xj

implying that (at x D c) @F @F C F D .I F C F / @xj @xj

(8.6)

@F F C @F F D .I F F C / : @xj @xj

(8.7)

F and

Upon substituting expressions (8.6) and (8.7) into expressions (8.4) and (8.5), respectively, we obtain expressions (8.2) and (8.3). Q.E.D. Proof (of Theorem 20.8.2). Observe (in light of Lemmas 20.8.1 and 15.4.3) that F C, F C F , and F F C are continuously differentiable at c. Now, using Lemma 15.4.3, we find that (at x D c) @F C @F C F F C @F C F C @F C D D F C F CF : @xj @xj @xj @xj Moreover, at x D c,

implying that (at x D c)

@F F C @F C @F C D F CF ; @xj @xj @xj

(8.8)

Exercises

F

@F C @F F C @F C D F : @xj @xj @xj

517

(8.9)

And, upon substituting expression (8.9) into the second term of expression (8.8) and then making use of results (8.3) and (8.2), we find that (at x D c) @F C @F C @F F C @F C F C D F C F C FC C F @xj @xj @xj @xj

0 @F C C C @F C C C @F C D F F C F .I F F / F C F .I F F / F @xj @xj @xj 0 C @F C C C @F C CF .I F F /F C F .I F F / F C: @xj @xj C

Thus, since F C .I F F C / D 0 and .I F C F /F C D 0 and since (according to Corollary 20.5.2) F C F and F F C are symmetric, 0 @F C @F @F C D F C F C F C .F C /0 .I F F C / @xj @xj @xj 0 @F C C .I F F / .F C /0 F C: @xj Q.E.D.

Exercises Section 20.2 1. Show that, for any m n matrix B of full column rank and for any n p matrix C of full row rank, .BC/C D C C BC: 2. Show that, for any mn matrix A, AC D A0 if and only if A0 A is idempotent. Section 20.3 3. In connection with Theorem 9.6.1, show that if C.U / C.T / and R.V / R.T / and if the generalized inverses T and Q (of T and Q, respectively) are both reflexive, then partitioned (9.6.2) matrices and (9.6.3) are reflexive T U W V generalized inverses of and , respectively. V W U T

518


4. Determine which of Penrose Conditions (1)–(4) are necessarily satisfied by a left inverse of an m n matrix A (when a left inverse exists). Which of the Penrose conditions are necessarily satisfied by a right inverse of an m n matrix A (when a right inverse exists)? Section 20.4 5. Let A represent an m n matrix and G an n m matrix. (a) Show that G is the Moore-Penrose inverse of A if and only if G is a minimum norm generalized inverse of A and A is a minimum norm generalized inverse of G. (b) Show that G is the Moore-Penrose inverse of A if and only if GAA0 D A0 and AGG 0 D G 0 . (c) Show that G is the Moore-Penrose inverse of A if and only if GA D PA0 and AG D PG0 . 6. (a) Show that, for any m n matrices A and B such that A0 B D 0 and BA0 D 0, .A C B/C D AC C BC. (b) Let A1 ; A2 ; : : : ; Ak represent m n matrices such that, for j > i D 1; : : : ; k 1, A0i Aj D 0 and Aj A0i D 0. Generalize the result of Part (a) C C by showing that .A1 C A2 C C Ak /C D AC 1 C A2 C C Ak . Section 20.5 7. Show that, for any mn matrix A, .AC A/C D AC A, and .AAC /C D AAC . 8. Show that, for any n n symmetric matrix A, AAC D AC A. 9. Let V represent an n n symmetric nonnegative definite matrix, X an n p matrix, and d a p 1 vector. Using the results of Exercises 8 and 19.11 (or otherwise), show that, for the vector X.X0 X/ d to be a solution, for every d 2 C.X0 /, to the problem of minimizing the quadratic form a0 V a (in a) subject to X0 a D d, it is necessary and sufficient that C.V C X/ C.X/. 10. Use Theorem 20.4.5 to devise an alternative proof of Part (3) of Theorem 20.5.3. 11. Let C represent an m n matrix. Show that, for any m m idempotent matrix A, .AC/C A0 D .AC/C and that, for any n n idempotent matrix B, B0 .CB/C D .CB/C. 12. Let a represent an n 1 vector of variables, and impose on a the constraint X0 a D d, where X is an np matrix and d a p 1 vector such that d 2 C.X0 /.


519

Define f .a/ D a0 V a 2b0 a, where V is an n n symmetric nonnegative definite matrix and b is an n 1 vector such that b 2 C.V ; X/. Further, let R represent any matrix such that V D R0 R, let a0 represent any n 1 vector such that X0 a0 D d, and take s to be any n 1 vector such that b D V s C Xt for some p 1 vector t. Using the results of Section 19.6 and of Exercise 11 (or otherwise), show that f .a/ attains its minimum value (under the constraint X0 a D d) at a point a if and only if a D a0 C ŒR.I PX /C R.s a0 / C fI ŒR.I PX /C Rg.I PX /w for some n 1 vector w. 13. Let A represent an n n symmetric nonnegative definite matrix, and let B represent an n n matrix. Suppose that B A is symmetric and nonnegative definite (in which case B is symmetric and nonnegative definite). Using Theorems 18.3.4 and 20.4.5 and the results of Exercises 1 and 18.15 (or otherwise), show that AC BC is nonnegative definite if and only if rank.A/ D rank.B/.

Bibliographic and Supplementary Notes §3(c). The proof of Corollary 20.3.14 makes use of the results of Section 12.4 (on the least squares problem). §6. The result of Exercise 13 is Theorem 3.1 of Milliken and Akdeniz (1977). §8. Formula (8.1) (for the partial derivatives of the Moore-Penrose inverse of a matrix of functions) was given by Golub and Pereyra (1973, thm. 4.3). The derivation of this formula given in Section 8 is patterned after that of Magnus and Neudecker (1988, sec. 8.5).

21 Eigenvalues and Eigenvectors

The decomposition of a matrix A into a product of two or three matrices can (depending on the characteristics of those matrices) be a very useful first step in computing such things as the rank, the determinant, or an (ordinary or generalized) inverse (of A) as well as a solution to a linear system having A as its coefficient matrix. In particular, decompositions like the QR, LDU, U0 DU, and Cholesky decompositions—refer to Sections 6.4 and 14.5—in which the component matrices are diagonal or triangular, or have orthonormal rows or columns, can be very useful for computational purposes. Moreover, establishing that a matrix has a decomposition of a certain type can be instructive about the nature of the matrix. In particular, if it can be shown that a matrix A is expressible as A D RR0 for some matrix R, then it can be concluded—refer to Corollary 14.2.14—that A is nonnegative definite (and symmetric). In the present chapter, consideration is given to a decomposition known as the spectral decomposition, which exists for symmetric matrices, and to a related decomposition known as the singular-value decomposition, which exists for all matrices. In the spectral or singular-value decomposition of a matrix A, A is decomposed into A D QDP, where P and Q are orthogonal matrices and where D is a diagonal matrix or, more generally, where D D diag.D1 ; 0/ for some diagonal matrix D1 —in the case of the spectral decomposition, P D Q0 . The first part of the present chapter is devoted to scalars and (column) vectors that satisfy the definitions of what are, respectively, known as an eigenvalue and an eigenvector of a (square) matrix. And, in subsequent parts of the chapter, it is shown that the existence and construction of the spectral or singular-value decomposition of a matrix A is equivalent to the existence and computation of linearly independent eigenvectors of A or of A0 A or AA0 . The existence of the spectral or singular-value decomposition can (once it has been established) be very useful in establishing various other results on matrices, including a number of the results that were established earlier in the book without resort to the existence of this decomposition. And, once the spectral or singularvalue decomposition of a matrix has been constructed, it can be very useful for computational purposes. However, there are drawbacks in using the existence of

522


the spectral or singular-value decomposition in “theoretical” arguments (when there are other options) and in constructing this decomposition for computational purposes. Any proof of existence implicitly or explicitly involves some relatively deep mathematics, and in general the construction of the spectral or singularvalue decomposition requires iterative methods and is much more computationally intensive than the construction of various other decompositions. Most (if not all) of the more basic theoretical results in linear statistical models and in related areas of statistics can be established without implicit or explicit use of the existence of the spectral or singular-value decomposition. And, in addressing the computational problems encountered in this area of statistics, it is typically not efficient to adopt an approach that requires the construction of the spectral or singular-vlaue decomposition. However, there are exceptions—refer, for example, to Harville and Fenech (1985).

21.1 Definitions, Terminology, and Some Basic Results A scalar (real number) is said to be an eigenvalue of an n n matrix A D faij g if there exists an n 1 nonnull vector x such that Ax D x or, equivalently, such that .A In /x D 0: Clearly, is an eigenvalue of A if and only if the matrix A I is singular. The set comprising the different scalars that are eigenvalues of A is called the spectrum of A. An n 1 nonnull vector x is said to be an eigenvector of the matrix A if there exists a scalar (real number) such that Ax D x, in which case is (by definition) an eigenvalue of A. For any particular eigenvector x (of A), there is only one eigenvalue such that Ax D x, which (since Ax D x ) x0 Ax D x0 x) is x0 Ax D 0 : xx The eigenvector x is said to correspond to (or belong to) this eigenvalue. Note that if x is an eigenvector of A, then, for any nonzero scalar c, the scalar multiple cx is also an eigenvector of A and cx corresponds to the same eigenvalue as x. Clearly, a nonnull vector x is an eigenvector (of the nn matrix A) corresponding to any particular eigenvalue if and only if it is a solution to the homogeneous linear system .A I/z D 0 (in z) or, equivalently, if and only if x 2 N.A I/. Thus, the set comprising all of the eigenvectors (of A) corresponding to the eigenvalue is the set obtained from N.A I/ by deleting the (n 1) null vector. The linear space N.A I/ is sometimes referred to as the eigenspace of the eigenvalue , and the dimension of this space is referred to as the geometric multiplicity of . As a consequence of Lemma 11.3.1,

21.1 Definitions, Terminology, and Some Basic Results

dimŒN.A I/ D n rank.A I/:

523

(1.1)

A scalar (real number) is an eigenvalue of the n n matrix A if and only if jA Ij D 0

(1.2)

(since A I is singular if and only if jA Ij D 0). Let ıij represent the ij th element of In , so that ıij D 1, if j D i , and ıij D 0, if j ¤ i (i; j D 1; : : : ; n). Then, it follows from the very definition of a determinant [given by formula (13.1.2)] that X jA Ij D .1/n .j1 ;:::;jn / .a1j1 ı1j1 / .anjn ınjn /; (1.3) where j1 ; : : : ; jn is a permutation of the first n positive integers and the summation is over all such permutations. Upon inspecting the sum (1.3), we find that the term .a11 /.a22 / .ann /;

(1.4)

corresponding to the permutation j1 D 1; j2 D 2; : : : ; jn D n, is a polynomial (in ) of degree n and that each of the remaining .nŠ 1/ terms is a polynomial (in ) of degree n 2 or less. It follows that

where

jA Ij D p./;

(1.5)

p./ D c0 C c1 C C cn1 n1 C cn n

(1.6)

is a polynomial (in ) whose coefficients c0 ; c1 ; : : : ; cn1 ; cn depend on the elements of A. And the term (1.4) is the only term of the sum (1.3) that contributes to cn and cn1 , so that cn D .1/n;

cn1 D .1/n1

n X

ai i D .1/n1 tr.A/:

(1.7)

iD1

Further, c0 D p.0/ D jAj:

(1.8)

Alternatively, results (1.7) and (1.8) could have been deduced from Corollary 13.7.4. In fact, Corollary 13.7.4 provides expressions for c1 ; c2 ; : : : ; cn2 as well as for c0 ; cn1 , and cn . Condition (1.2) can be reexpressed as p./ D 0

(1.9)

and can be regarded as an equation (in ). This equation is called the characteristic equation of A, and the polynomial p is called the characteristic polynomial of A. [Equation (1.9) and the polynomial p differ by a factor of .1/n from what

524


some authors call the characteristic equation and characteristic polynomial—those authors refer to jI Aj, which equals .1/n jA Ij, as the characteristic polynomial and/or to jI Aj D 0 as the characteristic equation.] Clearly, a scalar (real number) is an eigenvalue of A if and only if it is a root of the characteristic polynomial or, equivalently, if and only if it is a solution to the characteristic equation. Let f1 ; : : : ; k g represent the spectrum of the n n matrix A, that is, the set of (distinct) scalars that are eigenvalues of A. It follows from standard results on polynomials—refer, for example, to Beaumont and Pierce (1963, secs. 9-5 and 9-7)—that the characteristic polynomial p of A has a unique (aside from the order of the factors) representation of the form p./ D .1/n . 1 /1 . k /k q./;

(1.10)

P where 1 ; : : : ; k are positive integers and q is a polynomial (of degree n kiD1 i ) that has no real roots. The exponent i of the factor .i /i in expression (1.10) is referred to as the algebraic multiplicity of the eigenvalue i (i D 1; : : : ; k). Since the characteristic P polynomial p is of degree n, it is clear that the sum kiD1 i of the algebraic multiplicities of the k eigenvalues of the n n matrix A cannot exceed n (and also that the number k of distinct eigenvalues of A cannot exceed n). The sum Pk iD1 i equals n if and only if (for all ) q./ D 1—compare the coefficient of n in expression (1.10) with that in expression (1.6)—in which case expression (1.10) reduces to p./ D .1/n . 1 /1 . k1 /k1 . k /k:

(1.11)

In general, the algebraic multiplicity of an eigenvalue of an n n matrix is not the same as its geometric multiplicity. In fact, it will be shown in Section 3 that the algebraic multiplicity of any eigenvalue is greater than or equal to its geometric multiplicity. 1 1 Suppose, for example, that n D 2 and that A D . Then, the character0 1 istic polynomial of A is ˇ ˇ ˇ1 1 ˇˇ ˇ D .1 /2 D . 1/2 : p./ D ˇ 0 1 ˇ Thus, the spectrum of A comprises a single eigenvalue 1, and the algebraic multiplicity of this eigenvalue is 2. However, 0 1 A .1/I D ; 0 0 so that dimfNŒA .1/Ig D n rankŒA .1/I D 2 1 D 1. Thus, the geometric multiplicity of A is only 1. Does every (square) matrix have at least one eigenvalue?

21.1 Definitions, Terminology, and Some Basic Results

525

The fundamental theorem of algebra guarantees that every polynomial of degree one or more has a possibly complex root. Refer, for example, to Beaumont and Pierce (1963, sec. 9-8 and app. 3) for a formal statement of the fundamental theorem of algebra and for a proof—the proof of this theorem is well beyond the scope and mathematical level of our presentation. Thus, the characteristic polynomial of every (square) matrix has at least one possibly complex root. However, there exist matrices whose characteristic polynomials have no real roots. These matrices have no eigenvalues. 0 1 Consider, for example, the matrix . The characteristic polynomial of 1 0 this matrix is ˇ ˇ ˇ 1 ˇ ˇ D 2 C 1; ˇ p./ D ˇ 1 ˇ which has no realroots (although it has two imaginary roots, D i and D i ). 0 1 Thus, the matrix has no eigenvalues. 1 0 In the degenerate special case where the n n matrix A has no eigenvalues, expression (1.10) for the characteristic polynomial is to be read as p./ D .1/n q./. Every eigenvalue has an algebraic and geometric multiplicity of at least one. An eigenvalue is referred to as a simple eigenvalue or a multiple eigenvalue, depending on whether its algebraic multiplicity equals one or exceeds one. Eigenvectors that correspond to the same simple eigenvalue are scalar multiples of each other—it will be shown in Section 3 that the geometric multiplicity of a simple eigenvalue is necessarily equal to one—and consequently can be distinguished from each other by, for example, specifying the value of any nonzero element. In what follows, scalars that are not eigenvalues will sometimes (for convenience in stating various results) be regarded as eigenvalues whose algebraic and geometric multiplicities equal zero. If the algebraic multiplicities of the eigenvalues of an n n matrix A sum to n, then every root of the characteristic polynomial of A is a real number [as is evident from expression (1.11)]. The fundamental theorem of algebra can be used to establish the converse of this result. If the polynomial q in expression (1.10) is of degree one or more, then (according to the fundamental theorem) q has a root (which by definition is not real). Thus, if every root of the characteristic polynomial of the n n matrix A is real, then the algebraic multiplicities of the eigenvalues of A sum to n. Our use of the terms eigenvalue and eigenvector is more restrictive than that of many authors. In reference to an n n matrix A, they apply the term eigenvalue to any possibly complex number for which there exists a vector x (of possibly complex numbers) such that Ax D x or, equivalently, to any possibly complex number that is a root of the characteristic polynomial. Similarly, they apply the term eigenvector to any vector x (of possibly complex numbers) for which there exists a possibly complex number such that Ax D x. In contrast, we confine

526


our use of these terms to what those other authors call real eigenvalues and real eigenvectors. It should also be noted that some authors use other terms in place of eigenvalue and eigenvector. In the case of eigenvalues, the alternative terms include characteristic value or root, proper value, and latent value. Similarly, the alternatives to eigenvector include characteristic vector, proper vector, and latent vector. In referring to an n n matrix, it is convenient to adopt a terminology that allows us to distinguish between the elements of its spectrum and the elements P of a certain closely related set. Letting m D kiD1 i , expression (1.10) for the characteristic polynomial of an n n matrix A can be reexpressed as p./ D .1/n . d1 /. d2 / . dm /q./; where d1 ; d2 ; : : : ; dm are m scalars, i of which equal i (i.e., equal the i th element of the spectrum of A). Let us refer to the elements of the set fd1 ; d2 ; : : : ; dm g as the eigenvalues of the matrix A or (for emphasis) as the not-necessarily-distinct eigenvalues of A. And, to avoid confusion, let us subsequently refer to the elements of the spectrum of A as the distinct eigenvalues of A. Consider, for example, the 77 diagonal matrix D D diag.3; 1; 5; 5; 1; 1; 0/. Its characteristic polynomial is p./ D . 3/. 1/. C 5/. C 5/. 1/. 1/ D . 3/. 1/3 . C 5/2 : The (not necessarily distinct) eigenvalues of D are 3; 1; 5; 5; 1; 1, and 0; and the distinct eigenvalues of D are 3; 1; 5, and 0. Let A represent an nn matrix. Further, let x1 ; : : : ; xm represent n1 vectors, and let X D .x1 ; : : : ; xm /. Then, all m of the vectors x1 ; : : : ; xm are eigenvectors of A if and only if there exists a diagonal matrix D D fdj g such that AX D XD; in which case d1 ; : : : ; dm are the eigenvalues to which x1 ; : : : ; xm , respectively, correspond. (To see this, observe that the j th columns of AX and XD are, respectively, Axj and dj xj .) The eigenvectors of an nn matrix A have a noteworthy geometrical property. Let represent any nonzero eigenvalue of A, let x represent any eigenvector that corresponds to , and take the inner product for Rn1 to be the usual inner product. Then, the direction of the vector Ax (which, like x, is an n 1 vector) is either the same as that of x or opposite to that of x, depending on whether > 0 or < 0. That is, the angle between Ax and x is either 0 or (180ı ), depending on whether > 0 or < 0. And the length of Ax is jj times that of x. A subspace U of the linear space Rn1 is said to be invariant relative to an n n matrix A if, for every vector x in U, the vector Ax is also in U. Let x1 ; : : : ; xm represent any eigenvectors of A, and let X D .x1 ; : : : ; xm /. Then, there exists a (diagonal) matrix D such that AX D XD. And, corresponding to any vector u in C.X/, there exists an m 1 vector r such that u D Xr, so that

21.2 Eigenvalues of Block-Triangular or Block-Diagonal Matrices

527

Au D AXr D XDr 2 C.X/: Thus, the subspace C.X/ (of Rn1 ) spanned by the eigenvectors x1 ; : : : ; xm is invariant (relative to A). With the help of result (1.1), it is easy to verify the following lemma. Lemma 21.1.1. The scalar 0 is an eigenvalue of an n n matrix A if and only if A is singular, in which case the geometric multiplicity of the eigenvalue 0 equals n rank.A/ or, equivalently, rank.A/ equals n minus the geometric multiplicity of the eigenvalue 0. The following two lemmas relate the eigenvalues of the transpose of an n n matrix A to those of A itself and relate the eigenvalues and eigenvectors of Ak (where k is a positive integer) and of A1 to those of A. Lemma 21.1.2. Let A represent an n n matrix. Then, A0 has the same characteristic polynomial and the same spectrum as A, and every scalar in the spectrum (of A) has the same algebraic multiplicity when regarded as an eigenvalue of A0 as when regarded as an eigenvalue of A. Proof. It suffices to show that A0 and A have the same characteristic polynomial. That their characteristic polynomials are the same is evident upon observing (in light of Lemma 13.2.1) that, for any scalar , jA0 Ij D j.A I/0 j D jA Ij: Q.E.D. Lemma 21.1.3. Let represent an eigenvalue of an n n matrix A, and let x represent any eigenvector (of A) corresponding to . Then, (1) for any positive integer k, k is an eigenvalue of Ak, and x is an eigenvector of Ak corresponding to k ; and (2) if A is nonsingular (in which case ¤ 0), 1= is an eigenvalue of A1, and x is an eigenvector of A1 corresponding to 1=. Proof. (1) The proof is by mathematical induction. By definition, 1 is an eigenvalue of A1, and x is an eigenvector of A1 corresponding to 1. Now, suppose that k1 is an eigenvalue of Ak1 and that x is an eigenvector of Ak1 corresponding to k1 (where k 2). Then, Ak x D Ak1Ax D Ak1.x/ D Ak1 x D k1 x D k x: (2) Suppose that A is nonsingular (in which case is clearly nonzero). Then, x D A1Ax D A1.x/ D A1 x; implying that A1 x D .1=/x. Q.E.D. In connection with Lemma 21.1.2, it should be noted that (unless A is symmetric) an eigenvector of A is not in general an eigenvector of A0 .

528


21.2 Eigenvalues of Triangular or Block-Triangular Matrices and of Diagonal or Block-Diagonal Matrices The eigenvalues of a matrix depend on the elements of the matrix in a very complicated way, and in general their computation requires the use of rather sophisticated and time-consuming numerical procedures. However, for certain types of matrices such as block-triangular or block-diagonal matrices, the eigenvalues can be reexpressed in terms of the eigenvalues of matrices of smaller dimensions or in other relatively simple ways, and the computational requirements can be substantially reduced. The following lemma gives some results on the eigenvalues of block-triangular matrices. Lemma 21.2.1. Let A represent an n n matrix that is partitioned as 1 0 A11 A12 : : : A1r BA21 A22 : : : A2r C C B ADB : :: :: C; :: @ :: : : : A Ar1 Ar2 : : : Arr where (for i; j D 1; : : : ; r) Aij is of dimensions ni nj . Suppose that A is (upper or lower) block-triangular. Then, (1) the Qr characteristic polynomial p./ of A is such that (for 2 R) p./ D iD1 pi ./, where (for i D 1; : : : ; r) pi ./ is the characteristic polynomial of Ai i ; (2) a scalar is an eigenvalue of A if and only if is an eigenvalue of at least one of the diagonal blocks A11 ; : : : ; Arr , or, equivalently, the spectrum of A is the union of the spectra of A11 ; : : : ; Arr ; P (3) the algebraic multiplicity of an eigenvalue of A equals riD1 .i/ , where (for i D 1; : : : ; r) .i/ is the algebraic multiplicity of when is regarded as an eigenvalue of Ai i —if is not an eigenvalue of Ai i , then .i/ D 0; and .1/ (4) the (not necessarily distinct) eigenvalues of A are d1.1/ ; : : : ; dm ; d1.2/ , : : : ; 1 .2/ .r/ .r/ .i/ i/ dm2 ; : : : ; d1 ; : : : ; dmr , where (for i D 1; : : : ; r) d1 ; : : : ; dmi are the (not necessarily distinct) eigenvalues of Ai i .

Proof. (1) For 2 R, the matrix A In is block triangular with diagonal blocks A11 In1 ; : : : ; Arr Inr , respectively. Thus, making use of results (13.3.3) and (13.3.4), we find that (for 2 R) p./ D jA Ij D

r Y iD1

jAi i Ij D

r Y

pi ./:

iD1

(2)–(4). Parts (2)–(4) are almost immediate consequences of Part (1). Q.E.D. Observe that the characteristic polynomial of a 1 1 matrix .c/ (whose only element is the scalar c) is p./ D .1/. c/, that the spectrum of .c/ is the set

21.2 Eigenvalues of Block-Triangular or Block-Diagonal Matrices

529

fcg whose only member is c, and that the algebraic (and geometric) multiplicities of the eigenvalue c of .c/ equal 1. Then, by regarding a triangular matrix as a block-triangular matrix with 1 1 blocks, we obtain the following corollary of Lemma 21.2.1. Corollary 21.2.2. For any nn (upper or lower) triangular matrix A D faij g, Q (1) the characteristic polynomial of A is p./ D .1/n niD1 . ai i /; (2) a scalar is an eigenvalue of A if and only if it equals one or more of the diagonal elements a11 ; : : : ; ann , or, equivalently, the spectrum of A comprises the distinct scalars represented among the diagonal elements of A; (3) the algebraic multiplicity of an eigenvalue of A equals the number of diagonal elements of A that equal ; and (4) the (not necessarily distinct) eigenvalues of A are the diagonal elements of A. In light of Corollary 21.2.2, it is a trivial matter to determine the eigenvalues of a triangular matrix. One need only inspect its diagonal elements. Lemma 21.2.1 gives various results on the characteristic polynomial, the spectrum, and the algebraic multiplicities of the distinct eigenvalues of a block-triangular matrix (with square diagonal blocks). Since block-diagonal matrices are block-triangular, these results are applicable to block-diagonal matrices (or, more precisely, to block-diagonal matrices with square diagonal blocks). The following lemma gives some results on the geometric multiplicities of the distinct eigenvalues of a block-diagonal matrix and on the eigenvectors of a block-diagonal matrix—these results are specific to block-diagonal matrices, that is, do not (in general) apply to block-triangular matrices. Lemma 21.2.3. For any n n block-diagonal matrix A with square diagonal blocks A11 ; : : : ; Arr of order n1 ; : : : ; nr , respectively, P (1) the geometric multiplicity of an eigenvalue of A equals riD1 .i/ , where (for i D 1; : : : ; r) .i/ is the geometric multiplicity of when is regarded as an eigenvalue of Ai i —if is not an eigenvalue of Ai i , then .i/ D 0; and (2) if an ni 1 vector x is an eigenvector of Ai i corresponding to an eigenvalue , P n /th, then the n 1 vector x D .00 ; : : : 00 ; x0 ; 00 ; : : : ; 00 /0 , whose .1 C i1 s sD1 P : : : ; . isD1 ns /th elements are respectively the first, : : : ; ni th elements of x , is an eigenvector of A corresponding to — is an eigenvalue of A as well as of Ai i (1 i r). Proof. (1) Recalling results (1.1) and (4.5.14) and observing that A I D diag.A11 I; : : : ; Arr I/, we find that the geometric multiplicity of equals n rank.A I/ D n

r X

rank.Ai i I/ D

iD1

D

r X iD1 r X iD1

Œni rank.Ai i I/ .i/ :

530


(2) Suppose that x is an eigenvector of Ai i corresponding to an eigenvalue . Then, 0 1 0 1 0 0 B :: C B :: C B : C B : C B C B C B 0 C B 0 C B C B C C B C Ax D B BAi i x C D Bx C D x: B 0 C B 0 C C B C B B :: C B :: C @ : A @ : A 0 0 Q.E.D. By regarding a diagonal matrix as a block-diagonal matrix with 1 1 blocks, we obtain the following corollary of Lemma 21.2.3. Corollary 21.2.4. For any n n diagonal matrix D D fdi g, (1) the geometric multiplicity of an eigenvalue of D equals the number of diagonal elements of D that equal ; and (2) the n 1 vector cui , where ui is the i th column of In and c is an arbitrary nonzero scalar, is an eigenvector of D corresponding to di —di is an eigenvalue of D (1 i n). Corollary 21.2.4 gives results on diagonal matrices that augment those given by Corollary 21.2.2—since diagonal matrices are triangular, the results of Corollary 21.2.2 are applicable to diagonal matrices. Note that the geometric multiplicity of any eigenvalue of a diagonal matrix is the same as its algebraic multiplicity, as evidenced by Part (1) of Corollary 21.2.4 and Part (3) of Corollary 21.2.2.

21.3 Similar Matrices An n n matrix B is said to be similar to an n n matrix A if there exists an n n nonsingular matrix C such that B D C 1AC or, equivalently, such that CB D AC. Upon observing that (for any n n nonsingular matrix C) B D C 1AC ) A D CBC 1 D .C 1 /1 BC 1; it is clear that if B is similar to A, then A is similar to B. Some of the ways in which similar matrices are similar are described in the following theorem. Theorem 21.3.1. For any n n matrix A and any n n nonsingular matrix C, (1) rank.C 1AC/ D rank.A/; (2) det.C 1AC/ D det.A/; (3) tr.C 1AC/ D tr.A/;

21.3 Similar Matrices

531

(4) C 1AC and A have the same characteristic polynomial and the same spectrum; (5) an eigenvalue has the same algebraic and geometric multiplicities when regarded as an eigenvalue of C 1AC as when regarded as an eigenvalue of A; (6) if x is an eigenvector of A corresponding to an eigenvalue , then C 1 x is an eigenvector of C 1AC corresponding to , and, similarly, if y is an eigenvector of C 1AC corresponding to , then Cy is an eigenvector of A corresponding to ; and (7) the (not necessarily distinct) eigenvalues of C 1AC are the same as those of A. Proof. (1) That rank.C 1AC/ D rank.A/ is evident from Corollary 8.3.3. (2) Using results (13.3.9) and (13.3.12), we find that jC 1ACj D jC 1 jjAjjCj D .1=jCj/jAjjCj D jAj: (3) As a consequence of Lemma 5.2.1, tr.C 1AC/ D tr.ACC 1 / D tr.A/: (4) Applying Part (2) (with A I in place of A), we find that, for any scalar ,

jC 1AC Ij D jC 1 .A I/Cj D jA Ij:

Thus, C 1AC and A have the same characteristic polynomial (which implies that they have the same spectrum). (5) Let represent an eigenvalue of A or, equivalently [in light of Part (4)], of C 1AC. Since [according to Part (4)] C 1AC and A have the same characteristic polynomial, has the same algebraic multiplicity when regarded as an eigenvalue of C 1AC as when regarded as an eigenvalue of A. Further, the geometric multiplicities of when regarded as an eigenvalue of C 1AC and when regarded as an eigenvalue of A are, respectively [in light of result (1.1)], n rank.C 1AC I/ and n rank.A I/ and hence {since rank.C 1AC I/ D rankŒC 1.A I/C D rank.A I/} are the same. (6) If Ax D x, then .C 1AC/C 1 x D C 1Ax D C 1 .x/ D C 1 x; and if C 1ACy D y, then ACy D CC 1ACy D C.y/ D Cy: (7) Since [according to Part (4)] C 1AC and A have the same characteristic polynomial, the (not necessarily distinct) eigenvalues of C 1AC are the same as those of A. Q.E.D.

532


Theorem 21.3.1 can be very useful. If, for example, it can be demonstrated that a matrix A is similar to a diagonal or triangular matrix or to some other matrix whose eigenvalues are relatively easy to obtain, then the determination of the eigenvalues of A may be greatly facilitated. Subsequently, we shall have occasion to make use of the following theorem. Theorem 21.3.2. Let A represent an n n matrix, B a k k matrix, and U an nk matrix with orthonormal columns (i.e., with U 0 U D I) such that AU D U B. Then, there exists an n .n k/ matrix V such that the n n matrix .U ; V / is orthogonal. And, for any such matrix V , B U 0 AV 0 .U ; V / A.U ; V / D 0 V 0 AV [so that A is similar to the block-triangular matrix

B U 0 AV ], and, in the 0 V 0 AV

special case where A is symmetric, .U ; V /0 A.U ; V / D

B 0 0 V 0 AV

[so that A is similar to diag.B; V 0 AV /]. Proof. As a consequence of Theorem 6.4.5, there exists an n .n k/ matrix V such that the columns of the n n matrix .U ; V / form an orthonormal (with respect to the usual inner product) basis for Rn or, equivalently, such that the matrix .U ; V / is orthogonal. Now, let V represent any such matrix. Then, observing that U 0 AU D U 0 U B D IB D B and that

V 0 AU D V 0 U B D 0B D 0;

we find that .U ; V /0 A.U ; V / D

B U 0 AV : 0 V 0 AV

And, in the special case where A is symmetric, U 0 AV D U 0 A0 V D .V 0 AU /0 D 00 D 0; so that

.U ; V /0 A.U ; V / D diag.B; V 0 AV /:

Q.E.D. In the special case where k D 1, Theorem 21.3.2 can be restated as the following corollary.

21.3 Similar Matrices

533

Corollary 21.3.3. Let A represent an n n matrix. And let represent any eigenvalue of A (assuming that one exists), and u represent an eigenvector of (usual) norm 1 that corresponds to . Then, there exists an n .n 1/ matrix V such that the n n matrix .u; V / is orthogonal. Moreover, for any such matrix V, u0 AV .u; V /0 A.u; V / D 0 V 0 AV u0 AV ], and, in the special [so that A is similar to the block-triangular matrix 0 V 0 AV case where A is symmetric, 0 0 .u; V / A.u; V / D 0 V 0 AV [so that A is similar to diag.; V 0 AV /]. The following theorem gives an inequality that was mentioned (but not verified) in Section 1. Theorem 21.3.4. The geometric multiplicity of an eigenvalue of an n n matrix A is less than or equal to the algebraic multiplicity of . Proof. Let D dimŒN.A I/, which by definition is the geometric multiplicity of . Then, as a consequence of Theorem 6.4.3, there exists an n matrix U with orthonormal (with respect to the usual inner product) columns that form a basis for N.A I/. And, since .A I/U D 0 and hence since AU D U D U .I /, it follows from Theorem 21.3.2 that A is similar to a matrix that is expressible as I R ; 0 S where R is a .n / matrix and S is an .n / .n / matrix. I R — Now, it follows from Theorem 21.3.1 that is an eigenvalue of 0 S I R that is an eigenvalue of is also evident from Lemma 21.2.1—and that 0 S I R has the same algebraic multiplicity when regarded as an eigenvalue of 0 S as when regarded as an eigenvalue of A. Moreover, it is clear from Lemma 21.2.1 and from Corollary 21.2.2 that the algebraic multiplicity of when regarded as an I R is at least . We conclude that the algebraic multiplicity eigenvalue of 0 S of when regarded as an eigenvalue of A is at least . Q.E.D. The following four corollaries give various implications of Theorem 21.3.4. Corollary 21.3.5. If the algebraic multiplicity of an eigenvalue of an n n matrix equals 1, then the geometric multiplicity of also equals 1. Proof. The validity of Corollary 21.3.5 is evident from Theorem 21.3.4 upon observing that the geometric multiplicity of is at least one. Q.E.D.

534


Corollary 21.3.6. If an n n matrix A has n distinct eigenvalues, then each of them has a geometric (and algebraic) multiplicity of 1. Proof. The algebraic multiplicities of the distinct eigenvalues of A are greater than or equal to 1, and the sum of the algebraic multiplicities cannot exceed n. Thus, if A has n distinct eigenvalues, then each of them has an algebraic multiplicity of 1 and hence (in light of Corollary 21.3.5) a geometric multiplicity of 1. Q.E.D. Corollary 21.3.7. Let A represent an nn matrix that has k distinct eigenvalues 1 ; : : : ; k with algebraic multiplicities 1 ; : : : ; k , respectively, and geometric P P multiplicities 1 ; : : : ; k , respectively. Then, kiD1 i kiD1 i n. And, if Pk Pk Pk Pk iD1 i D n, then iD1 i D n. Moreover, if iD1 i D iD1 i , then (for i D 1; : : : ; k) i D i . P P Proof. That kiD1 i kiD1 i is an immediate consequence of Theorem Pk 21.3.4, and that iD1 i n was established earlier (in Section 1). And, if Pk Pk

D n. iD1 i D n, then together these inequalities imply that Pk Pk Pk iD1 i Now, suppose that iD1 i D iD1 i . Then, iD1 . i i / D 0. And, according to Theorem 21.3.4, i i or, equivalently, i i 0 (i D 1; : : : ; k). Q.E.D. Thus, i i D 0, or, equivalently, i D i (i D 1; : : : ; k). Corollary 21.3.8. Let A represent an n n matrix that has k distinct eigenvalues 1 ; : : : ; k with geometric multiplicities 1 ; : : : ; k , respectively. Then, the sum of the geometric multiplicities of the (k or k 1) nonzero distinct eigenvalues (of A) is less than or equal to rank.A/, with equality holding if and only if Pk iD1 i D n. Proof. It is convenient to consider separately two cases. Case 1: rank.A/ D n. In this case, all k of the distinct eigenvalues 1 ; : : : ; k are nonzero (as is evident from Lemma 11.3.1), and it is apparent from Corollary P 21.3.7 that the sum kiD1 i of the geometric multiplicities is less than or equal to P rank.A/. (And it is obvious that this sum equals rank.A/ if and only if kiD1 i D n.) Case 2: rank.A/ < n. In this case, it follows from Lemma 11.3.1 that one of the distinct eigenvalues, say s , equals 0 and that s D n rank.A/. Thus, making use of Corollary 21.3.7, we find that k X

i D

i¤s

And, clearly,

k X

i Œn rank.A/ n Œn rank.A/ D rank.A/:

iD1

Pk

i¤s

i D rank.A/ if and only if n D

Pk

iD1

i .

21.4 Linear Independence of Eigenvectors A basic property of eigenvectors is described in the following theorem.

Q.E.D.

21.4 Linear Independence of Eigenvectors

535

Theorem 21.4.1. Let A represent an n n matrix, let x1 ; : : : ; xk represent k eigenvectors of A, and let 1 ; : : : ; k represent the eigenvalues (of A) to which x1 ; : : : ; xk correspond. If 1 ; : : : ; k are distinct, then x1 ; : : : ; xk are linearly independent. Theorem 21.4.1 is a special case of the following result. .m /

i Theorem 21.4.2. Let A represent an nn matrix. And let x.1/ repi ; : : : ; xi resent linearly independent eigenvectors of A that correspond to some eigenvalue .m1 / , i (i D 1; : : : ; k). If 1 ; : : : ; k are distinct, then the combined set x.1/ 1 ; : : : ; x1 P .mk / .1/ k of iD1 mi eigenvectors is linearly independent. : : : ; xk , : : : ; xk Proof (of Theorem 21.4.2). Assume that 1 ; : : : ; k are distinct. To show that the combined set is linearly independent, let us proceed by mathematical induction. .m1 / The set of m1 eigenvectors x.1/ is by definition linearly independent. 1 ; : : : x1 P Suppose now that the set comprising the first siD1 mi eigenvectors x.1/ 1 , :::; .m1 / .1/ .ms / x1 ; : : : ; xs ; : : : ; xs is linearly independent (where 1 s k 1). Then, to complete the induction argument, it suffices to show that the set comprising P .msC1 / .1/ .m1 / ; : : : ; x.1/ is linearly the first sC1 iD1 mi eigenvectors x1 ; : : : ; x1 sC1 ; : : : ; xsC1 independent. .m / .1/ ; : : : ; csC1sC1 represent any scalars such that Let c1.1/ ; : : : ; c1.m1 /, : : : ; csC1

.m

/ .m

/

.m1 / .m1 / .1/ .1/ c1.1/ x.1/ x1 C C csC1 xsC1 C C csC1sC1 xsC1sC1 D 0: (4.1) 1 C C c1

And observe that, for j D 1; : : : ; msC1 , / .A sC1 I/x.j sC1 D 0

and that, for i D 1; : : : ; s and j D 1; : : : ; mi , / / / .A sC1 I/x.j D ŒA i I C .i sC1 /Ix.j D .i sC1 /x.j i i i :

Then, premultiplying both sides of equality (4.1) by A sC1 I, we find that .m1 / .m1 / .1 sC1 /.c1.1/ x.1/ x1 / 1 C C c1 .ms / .ms / C C .s sC1 /.cs.1/ x.1/ xs / D 0: s C C cs Ps And, since i sC1 ¤ 0 for i ¤ s C1 and (by supposition) the iD1 mi vectors .m1 / .ms / ; : : : ; x.1/ are linearly independent, it follows that x.1/ s ; : : : ; xs 1 , : : : ; x1

c1.1/ D D c1.m1 / D D cs.1/ D D cs.ms / D 0:

(4.2)

Moreover, substituting from result (4.2) into equality (4.1), we find that .m

/ .m

/

.1/ .1/ xsC1 C C csC1sC1 xsC1sC1 D 0 csC1 .m

/

sC1 and hence, since the msC1 vectors x.1/ are by definition linearly sC1 ; : : : ; xsC1 independent, that

536

21 Eigenvalues and Eigenvectors .m

/

.1/ csC1 D D csC1sC1 D 0: (4.3) PsC1 .m1 / , Together, results (4.2) and (4.3) imply that the iD1 mi vectors x.1/ 1 ; : : : ; x1 .msC1 / .1/ Q.E.D. : : : ; xsC1 , : : : ; xsC1 are linearly independent. How many linearly independent eigenvectors are there? This question is answered (in terms of the geometric multiplicities of the distinct eigenvalues) by the following theorem.

Theorem 21.4.3. Let 1 ; : : : ; k represent the distinct eigenvalues of an n n matrix A (i.e., the members of the spectrum of A), and let 1 ; : : : ; k represent their respective geometric multiplicities. Then, for i D 1; : : : ; k, there exists a .i / of A corresponding set of i linearly independent eigenvectors x.1/ i ; : : : ; xi . / .1 / .1/ to i . The combined set S D fx.1/ ; : : : ; x ; : : : ; x ; : : : ; xk k g, comprising 1 1 k Pk Pk iD1 i eigenvectors, is linearly independent; and any set of more than iD1 i P eigenvectors is linearly dependent. Moreover, if kiD1 i D n, then S is a basis for Rn , and the set S obtained from S by deleting any eigenvectors that correspond to 0 (in the event that 0 is an eigenvalue of A) is a basis for C.A/. Proof. For i D 1; : : : ; k, any basis for the eigenspace N.A i I/ is a set of i linearly independent eigenvectors corresponding to i , so that such a set clearly exists. That the combined set S is linearly independent is an immediate consequence of Theorem 21.4.2. Suppose now (for purposes of establishing a contradiction) P that there existed a linearly independent set of more than kiD1 i eigenvectors. Then, for some integer t (1 t k), the number of eigenvectors in that set that corresponded to t would exceed t . And, as a consequence, N.A t I/ would contain more than t linearly independent vectors, which {since by definition t D dimŒN.A t I/} establishes the desired contradiction. Thus, any set of P more than kiD1 i eigenvectors is linearly dependent. P To complete the proof, suppose that kiD1 i D n. Then, S comprises n (linearly independent, n-dimensional) vectors and hence is a basis for Rn. Further, let r represent the number of vectors in S , let X represent an n r matrix whose columns are the vectors in S , and let represent an r r diagonal matrix whose diagonal elements are the eigenvalues to which the columns of X correspond, so that AX D X, implying (since the diagonal elements of are nonzero) that X D AX1. And, upon observing that rank.X/ D r and (in light of Corollary 21.3.8) that r D rank.A/, it follows from Corollary 4.4.7 that C.X/ D C.A/. Q.E.D. Thus, S is a basis for C.A/. As an immediate consequence of Theorem 21.4.3, we have the following corollary. Corollary 21.4.4. Let 1 ; : : : ; k represent the geometric multiplicities of the distinct eigenvalues of an n n matrix A. Then, there exists a linearly independent P set of n eigenvectors (of A) if and only if kiD1 i D n.

21.5 Diagonalization

537

Eigenvectors that correspond to distinct eigenvalues are linearly independent (as indicated by Theorem 21.4.1). If the eigenvectors are eigenvectors of a symmetric matrix, then we can make the following, stronger statement. Theorem 21.4.5. If two eigenvectors x1 and x2 of an n n symmetric matrix A correspond to different eigenvalues, then x1 and x2 are orthogonal (with respect to the usual inner product). Proof. Let 1 and 2 represent the eigenvalues to which x1 and x2 correspond. Then, by definition, Ax1 D 1 x1 and Ax2 D 2 x2 . Premultiplying both sides of the first of these two equalities by x02 and both sides of the second by x01 , we find that x02 Ax1 D 1 x02 x1 and x01 Ax2 D 2 x01 x2 . And, since A is symmetric, we have that 1 x02 x1 D x02 A0 x1 D .x01 Ax2 /0 D .2 x01 x2 /0 D 2 x02 x1 ; implying that .1 2 /x02 x1 D 0 and hence if 1 ¤ 2 that x02 x1 D 0. Thus, if Q.E.D. 1 ¤ 2 , then x1 and x2 are orthogonal.

21.5 Diagonalization a. Definitions and some basic properties An n n matrix A is said to be diagonalizable (or diagonable) if there exists an n n nonsingular matrix Q such that Q1AQ D D for some diagonal matrix D, in which case Q is said to diagonalize A (or A is said to be diagonalized by Q). Note (in connection with this definition) that Q1AQ D D , AQ D QD , A D QDQ1:

(5.1)

Clearly, an n n matrix is diagonalizable if and only if it is similar to a diagonal matrix. An n n matrix A is said to be orthogonally diagonalizable if it is diagonalizable by an orthogonal matrix; that is, if there exists an n n orthogonal matrix Q such that Q0 AQ is diagonal. The process of constructing an n n nonsingular matrix that diagonalizes an n n matrix A is referred to as the diagonalization of A. This process is intimately related to the process of finding eigenvalues and eigenvectors of A, as is evident from the following theorem. Theorem 21.5.1. Let A represent an n n matrix. And suppose that there exists an n n nonsingular matrix Q such that Q1AQ D D for some diagonal matrix D; denote the first, : : : ; nth columns of Q by q1 ; : : : ; qn , respectively, and the first, : : : ; nth diagonal elements of D by d1 ; : : : ; dn , respectively. Then, (1) rank.A/ equals the number of nonzero diagonal elements of D; (2) det.A/ D d1 d2 dn ;

538


(3) tr.A/ D d1 C d2 C C dn ; (4) the characteristic polynomial of A is p./ D .1/n . d1 /. d2 / . dn /I (5) the spectrum of A comprises the distinct scalars represented among the diagonal elements of D; (6) the algebraic and geometric multiplicities of an eigenvalue of A equal the number of diagonal elements of D that equal ; (7) the (not necessarily distinct) eigenvalues of A are the diagonal elements of D; and (8) the columns of Q are (linearly independent) eigenvectors of A (with the i th column qi corresponding to the eigenvalue di ). Proof. Upon recalling various results on diagonal matrices [specifically, result (4.5.14) and the results given by Corollaries 13.1.2, 21.2.2, and 21.2.4], Parts (1)–(7) of Theorem 21.5.1 follow from Theorem 21.3.1. That q1 ; : : : ; qn are eigenvectors of A [as claimed in Part (8)] is evident upon observing [in light of result (5.1)] that AQ D QD (and that, for i D 1; : : : ; k, the i th columns of AQ and QD Q.E.D. are, respectively, Aqi and di qi ). b. Necessary and sufficient conditions for a matrix to be diagonalizable If an n n matrix A is diagonalizable by an n n nonsingular matrix Q, then, according to Part (8) of Theorem 21.5.1, the columns of Q are (linearly independent) eigenvectors of A. The following theorem makes a stronger statement. Theorem 21.5.2. An nn matrix A is diagonalizable by an nn nonsingular matrix Q if and only if the columns of Q are (linearly independent) eigenvectors of A. Proof. In light of Part (8) of Theorem 21.5.1, it suffices to prove the “if part” of Theorem 21.5.2. Suppose that the columns q1 ; : : : ; qn of Q are eigenvectors of A. Then, there exists a diagonal matrix D D fdi g such that AQ D QD (as discussed in Section 1 and as is evident upon observing that, for i D 1; : : : ; n, the i th columns of AQ and QD are, respectively, Aqi and di qi ). We conclude [in light of result (5.1)] that Q diagonalizes A. Q.E.D. As an immediate consequence of Theorem 21.5.2, we have the following corollary. Corollary 21.5.3. An n n matrix A is diagonalizable if and only if there exists a linearly independent set of n eigenvectors (of A). In light of Corollary 21.4.4 (and of Corollary 21.3.8), we have the following, additional corollary. Corollary 21.5.4. An n n matrix A, whose spectrum comprises k eigenvalues with geometric multiplicities 1 ; : : : ; k , respectively, is diagonalizable if and


539

P only if kiD1 i D n or, equivalently, if and only if the geometric multiplicities of the k or k 1 nonzero distinct eigenvalues (of A) sum to rank.A/. c. A basic result on symmetric matrices In Subsection d, the discussion of the diagonalization of matrices is specialized to the diagonalization of symmetric matrices. The results of Subsection d depend in a fundamental way on the following property of symmetric matrices. Theorem 21.5.5. Every symmetric matrix has an eigenvalue. The conventional way of proving this theorem (e.g., Searle 1982, chap. 11) is to implicitly or explicitly use the fundamental theorem of algebra to establish that the characteristic polynomial of any (square) matrix has a possibly complex root and to then show that if the matrix is symmetric, this root is necessarily real. The resultant proof is accessible only to those who are at least somewhat knowledgeable about complex numbers. Moreover, the fundamental theorem of algebra is a very deep result—its proof is well beyond the scope and mathematical level of our presentation. Here, we take a different approach. This approach consists of establishing the following theorem, of which Theorem 21.5.5 is an immediate consequence. Theorem 21.5.6. Let A represent an n n matrix. Then, there exist nonnull vectors x1 and x2 such that x02 Ax2 x01 Ax1 x0 Ax x01 x1 x0 x x02 x2 for every nonnull vector x in Rn (or, equivalently, such that x01 Ax1 =x01 x1 D minx¤0 x0 Ax=x0 x and x02 Ax2 =x02 x2 D maxx¤0 x0 Ax=x0 x). Moreover, if A is symmetric, then x01 Ax1 =x01 x1 and x02 Ax2 =x02 x2 are eigenvalues of A—in fact, they are, respectively, the smallest and largest eigenvalues of A—and x1 and x2 are eigenvectors corresponding to x01 Ax1 =x01 x1 and x02 Ax2 =x02 x2 , respectively. Proof. Let x represent an n 1 vector of variables, and define S D fx W x0 x D 1g. The quadratic form x0 Ax is a continuous function of x, and the set S is closed and bounded. Since any continuous function attains a maximum value and a minimum value over any closed and bounded subset of its domain (e.g., Bartle 1976, thm. 22.6; Magnus and Neudecker 1988, thm. 7.1), S contains vectors x1 and x2 such that x01 Ax1 x0 Ax x02 Ax2 for every x in S . Moreover, for any nonnull x (in Rn ), .x0 x/1=2 x 2 S and Œ.x0 x/1=2 x0 AŒ.x0 x/1=2 x D x0 Ax=x0 x: Thus,

x01 Ax1 x02 Ax2 x0 Ax 0 0 x D x Ax Ax D 1 2 1 2 x01 x1 x0 x x02 x2

(5.2)

540


for every nonnull x. Now, suppose that A is symmetric. Then, in light of results (15.2.13) and (15.3.7), we find that (for nonnull x) @x0 Ax @x0 x @.x0 Ax=x0 x/ 0 2 0 0 D .x x/ .x x/ .x Ax/ @x @x @x D .x0 x/2 Œ.x0 x/2Ax .x0 Ax/2x: Moreover, x0 Ax=x0 x attains (for x ¤ 0) minimum and maximum values at x1 and x2 , respectively [as is evident from result (5.2)], so that @.x0 Ax=x0 x/=@x D 0 for x D x1 and x D x2 . Thus, for x D x1 and x D x2 , .x0 x/2Ax .x0 Ax/2x D 0; or, equivalently,

Ax D .x0 Ax=x0 x/x:

We conclude that x01 Ax1 =x01 x1 and x02 Ax2 =x02 x2 are eigenvalues of A and that x1 and x2 are eigenvectors that correspond to x01 Ax1 =x01 x1 and x02 Ax2 =x02 x2 , respectively. Q.E.D. d. Orthogonal diagonalization Let A represent an n n matrix, and suppose that there exists an orthogonal matrix Q such that Q0 AQ D D for some diagonal matrix D. Then, since [in light of result (5.1)] A D QDQ0 , A0 D .QDQ0 /0 D QD 0 Q0 D QDQ0 D A: Thus, a necessary condition for A to be orthogonally diagonalizable is that A be symmetric (so that while certain nonsymmetric matrices are diagonalizable, they are not orthogonally diagonalizable). This condition is also sufficient, as indicated by the following theorem. Theorem 21.5.7. Every symmetric matrix is orthogonally diagonalizable. Proof. The proof is by mathematical induction. Clearly, every 1 1 matrix is orthogonally diagonalizable. Suppose now that every .n 1/ .n 1/ symmetric matrix is orthogonally diagonalizable (where n 2). Then, it suffices to show that every n n symmetric matrix is orthogonally diagonalizable. Let A represent an n n symmetric matrix. And let represent an eigenvalue of A (the existence of which is guaranteed by Theorem 21.5.5), and u represent an eigenvector of (usual) norm 1 that corresponds to . Then, according to Corollary 21.3.3, .u; V /0 A.u; V / D diag.; V 0 AV / for some n .n 1/ matrix V such that the n n matrix .u; V / is orthogonal.


541

Clearly, V 0 AV is a symmetric matrix of order n 1, so that (by supposition) there exists an orthogonal matrix R such that R0 .V 0 AV /R D F for some diagonal matrix F . Define S D diag.1; R/ and P D .u; V /S. Then, S0 S D diag.1; R0 R/ D diag.1; In1 / D In ; so that S is orthogonal and hence (according to Lemma 8.4.1) P is orthogonal. Further, P 0 AP D S0 .u; V /0 A.u; V /S D S0 diag.; V 0 AV /S D diag.; R0 V 0 AV R/ D diag.; F /; so that P 0 AP equals a diagonal matrix. Thus, A is orthogonally diagonalizable. Q.E.D. The following corollary (of Theorem 21.5.7) can be deduced from Corollaries 21.5.4 and 21.3.7 (or from Theorem 21.5.1). Corollary 21.5.8. Let A represent an nn symmetric matrix that has k distinct eigenvalues 1 ; : : : ; k with algebraic multiplicities 1 ; : : : ; k , respectively, and P P geometric multiplicities 1 ; : : : ; k , respectively. Then, kiD1 i D kiD1 i D n, and (for i D 1; : : : ; k) i D i . As a further corollary of Theorem 21.5.7, we have the following result. Corollary 21.5.9. Let A represent an n n symmetric matrix, and let d1 , : : : ; dn represent the (not necessarily distinct) eigenvalues of A (in arbitrary order). Then, there exists an n n orthogonal matrix Q such that Q0 AQ D diag.d1 ; : : : ; dn /: Proof. According to Theorem 21.5.7, there exists an n n orthogonal matrix R such that R0 AR D F for some diagonal matrix F D ffi g. It follows from Part (7) of Theorem 21.5.1 that there exists some permutation k1 ; : : : ; kn of the first n positive integers 1; : : : ; n such that (for i D 1; : : : ; n) di D fki . Now, let Q D RP , where P is the n n permutation matrix whose first, : : : ; nth columns are, respectively, the k1 , : : : ; kn th columns of In . Then, since a permutation matrix is orthogonal, Q is (in light of Lemma 8.4.1) orthogonal. Further, Q0 AQ D P 0 R0 ARP D P 0 F P D diag.d1 ; : : : ; dn /: Q.E.D. Let A represent an n n symmetric matrix. Then, according to Theorem 21.5.7, there exists an orthogonal matrix that diagonalizes A. There also exist nonorthogonal matrices that diagonalize A. If A has n distinct eigenvalues, then it follows from Theorem 21.4.5 that the columns of any diagonalizing matrix are orthogonal [since, according to Part (8) of Theorem 21.5.1, they are necessarily eigenvectors]. However, the norms of these columns need not equal one. More generally, if is an eigenvalue of A of multiplicity , then we can include as

542


columns of the diagonalizing matrix any linearly independent eigenvectors that correspond to . We need not choose these linearly independent eigenvectors to be orthonormal or even (if > 1) orthogonal. The (usual) norm of a symmetric matrix can be expressed in terms of its eigenvalues, as described in the following theorem. Theorem 21.5.10. For any n n symmetric matrix A, kAk D

n X

1=2 di2 ;

iD1

where d1 ; : : : ; dn are the (not necessarily distinct) eigenvalues of A (and where the norm is the usual norm). Proof. According to Corollary 21.5.9, there exists an n n orthogonal matrix Q such that Q0 AQ D D, where D D diag.d1 ; : : : ; dn /. Thus, making use of Lemma 5.2.1, we find that kAk2 D tr.A0 A/ D tr.AA/ D tr.AIn AIn / D tr.AQQ0 AQQ0 / D tr.Q0 AQQ0 AQ/ D tr.D 2 / D trŒdiag.d12 ; : : : ; dn2 / D

n X iD1

di2 : Q.E.D.

e. Orthogonal “triangularization” Let A represent an n n matrix. If A is nonsymmetric, then there does not exist any nn orthogonal matrix that diagonalizes A. However, there may exist an nn orthogonal matrix that “triangularizes” A, as indicated by the following theorem. Theorem 21.5.11. Let A represent an nn matrix whose distinct eigenvalues have algebraic multiplicities that sum to n, and let d1 ; : : : ; dn represent the notnecessarily-distinct eigenvalues of A (in arbitrary order). Then, there exists an n n orthogonal matrix Q such that Q0 AQ D T ; where T is an (nn) upper triangular matrix whose first, : : : ; nth diagonal elements are d1 ; : : : ; dn , respectively. Proof. The proof is by mathematical induction. Clearly, the theorem is valid for any 1 1 matrix. Suppose now that it is valid for any .n 1/ .n 1/ matrix whose distinct eigenvalues have algebraic multiplicities that sum to n 1 (where n 2), and consider the validity of the theorem for an n n matrix A whose algebraic multiplicities sum to n and whose (not necessarily distinct) eigenvalues are d1 ; : : : ; dn .


543

Let u represent an eigenvector (of A) of (usual) norm 1 that corresponds to d1 . Then, according to Corollary 21.3.3, d1 u0 AV 0 .u; V / A.u; V / D 0 V 0 AV for some n .n 1/ matrix V such that the n n matrix .u; V / is orthogonal. Moreover, it follows from Theorem 21.3.1 and Lemma 21.2.1 that the algebraic multiplicities of the distinct eigenvalues of the .n 1/ .n 1/ matrix V 0 AV sum to n 1 and that the (not necessarily distinct) eigenvalues of V 0 AV are d2 ; : : : ; dn . Thus, by supposition, there exists an orthogonal matrix R such that R0 .V 0 AV /R D T , where T is an .n 1/ .n 1/ upper triangular matrix whose first, : : : ; .n 1/th diagonal elements are d2 ; : : : ; dn , respectively. Now, define S D diag.1; R/ and Q D .u; V /S. Then, S0 S D diag.1; R0 R/ D diag.1; In1 / D In ; so that S is orthogonal and hence (according to Lemma 8.4.1) Q is orthogonal. Further, d1 u0 AV 1 0 1 0 Q0 AQ D S0 .u; V /0 A.u; V /S D 0 R 0 R0 0 V 0 AV 0 d1 u0 AV R d1 u AV R : D D 0 R0 V 0 AV R 0 T d1 u0 AV R is an (n n) upper triangular matrix And, upon observing that 0 T whose first, : : : ; nth diagonal elements are d1 ; : : : ; dn , respectively, the induction argument is complete. Q.E.D. Let A represent an n n matrix. If there exists an orthogonal matrix Q such that Q0 AQ D T for some upper triangular matrix T , then the diagonal elements of T are the not-necessarily-distinct eigenvalues of A, as is evident from Part (4) of Corollary 21.2.2 and Part (7) of Theorem 21.3.1. Theorem 21.5.11 indicates that a sufficient (as well as necessary) condition for the existence of such an orthogonal matrix Q is that the algebraic multiplicities of the not-necessarilydistinct eigenvalues of A sum to n and also indicates that (when this condition is satisfied) Q can be chosen so that the order in which the eigenvalues (of A) appear as diagonal elements of the triangular matrix T is arbitrary. Note (in light of Corollary 21.5.8) that Theorem 21.5.11 is applicable to any symmetric matrix. In the special case where the matrix A in Theorem 21.5.11 is symmetric, the triangular matrix T is a diagonal matrix (since if A is symmetric, Q0 AQ is symmetric and hence T is symmetric). Thus, in this special case, Theorem 21.5.11 simplifies to Corollary 21.5.9. f. Spectral decomposition (of a symmetric matrix) Let A represent an nn symmetric matrix. And let Q represent an nn orthogonal matrix and D D fdi g an nn diagonal matrix such that Q0 AQ D D—the existence

544


of such matrices is guaranteed by Theorem 21.5.7. Or, equivalently, let Q represent an n n matrix whose columns are orthonormal (with respect to the usual inner product) eigenvectors of A, and take D D fdi g to be the diagonal matrix whose first, : : : ; nth diagonal elements d1 ; : : : ; dn are the eigenvalues (of A) to which the first, : : : ; nth columns of Q correspond—the equivalence is evident from Theorems 21.5.1 and 21.5.2. Further, denote the first, : : : ; nth columns of Q by q1 ; : : : ; qn , respectively. In light of result (5.1), A can be expressed as A D QDQ0 or as AD

n X

di qi q0i :

(5.3) (5.4)

iD1

And, letting 1 ; : : : ; k represent the distinct eigenvalues of A, letting 1 ; : : : ; k represent the (algebraic or geometric) multiplicities of 1 ; : : : ; k , respectively, and (for j D 1; : : : ; k) letting Sj D fi W di D j g represent the set comprising the j values of i such that di D j , A can also be expressed as AD

k X

j Ej ;

(5.5)

j D1

P where (for j D 1; : : : ; k) Ej D i2Sj qi q0i . Expression (5.5) is called the spectral decomposition or spectral representation of the n n symmetric matrix A. [Sometimes, the term spectral decomposition is used also in reference to expression (5.3) or (5.4).] P The decomposition A D jkD1 j Ej is unique (aside from the ordering of P the terms), whereas (in general) the decomposition A D niD1 di qi q0i is not. To P see this (i.e., the uniqueness of the decomposition A D jkD1 j Ej ), suppose that P is an n n orthogonal matrix and D D fdi g an n n diagonal matrix such that P 0 AP D D (where P and D are possibly different from Q and D). Further, denote the first, : : : ; nth columns of P by p1 ; : : : ; pn , respectively, and (for j D 1; : : : ; k) let Sj D fi W di D j g. Then, analogous to the decomposition P A D jkD1 j Ej , we have the decomposition AD

k X

j Fj ;

j D1

P where (for j D 1; : : : ; k) Fj D i2S pi p0i . Now, for j D 1; : : : ; k, let Qj Dj.qi1 ; : : : ; qij / and Pj D .pi1 ; : : : ; pi /, j where i1 ; : : : ; ij and i1 ; : : : ; ij are the elements of Sj and Sj , respectively. Then, C.Pj / D N.A j I/ D C.Qj /, so that Pj D Qj Lj for some j j matrix Lj . Moreover, since clearly Qj0 Qj D Ij and Pj0 Pj D Ij , Lj0 Lj D Lj0 Qj0 Qj Lj D Pj0 Pj D I;

21.6 Expressions for the Trace and Determinant of a Matrix

545

implying that Lj is an orthogonal matrix. Thus, Fj D Pj Pj0 D Qj Lj Lj0 Qj0 D Qj IQj0 D Qj Qj0 D Ej : P We conclude that the decomposition A D jkD1 j Fj is identical to the decomP P position A D jkD1 j Ej , and hence that the decomposition A D jkD1 j Ej is unique (aside from the ordering of terms). The spectral decomposition of the mth power Am of the n n symmetric matrix A (where m is a positive integer) and (if A is nonsingular) the spectral decomposition of A1 are closely related to the spectral decomposition of A itself. It follows from Lemma 21.1.3 that q1 ; : : : ; qn are eigenvectors of Am and (if A is nonsingular) of A1 (as well as of A) and that the eigenvalues to which they correspond are d1m ; : : : ; dnm (in the case of Am ) and d11 ; : : : ; dn1 (in the case of A1 ). Thus, the spectral decompositions of Am and A1 can be obtained from that of A by replacing each eigenvalue of A by its mth power or by its reciprocal. More explicitly, observing that D m D diag.d1m ; : : : ; dnm / and D 1 D diag.d11 ; : : : ; dn1 /, the spectral decompositions of Am and A1 are Am D QD m Q0 D

n X

dim qi q0i D

iD1

A1 D QD 1 Q0 D

n X

k X

jm Ej ;

(5.6)

j D1 k X

di1 qi q0i D

iD1

j1 Ej :

(5.7)

j D1

21.6 Expressions for the Trace and Determinant of a Matrix Theorem 21.5.1 gives various results on n n diagonalizable matrices or, equivalently (in light of Corollary 21.5.4), on n n matrices whose distinct eigenvalues have geometric multiplicities that sum to n. Included among these results are expressions for the determinant and the trace of such a matrix. These expressions [which are Parts (2) and (3) of Theorem 21.5.1] are in terms of the eigenvalues. They can be extended to a broader class of matrices, as indicated by the following theorem. Theorem 21.6.1. Let A represent an n n matrix having distinct eigenvalues 1 ; : : : ; k with algebraic multiplicities 1 ; : : : ; k , respectively, that sum to n or, equivalently, having n not-necessarily-distinct eigenvalues d1 ; : : : ; dn . Then, det.A/ D

n Y

di D

iD1

tr.A/ D

n X iD1

di D

k Y

(6.1)

j j :

(6.2)

j D1 k X j D1

jj;

546


Proof. As a consequence of Theorem 21.5.11, there exists an nn orthogonal matrix Q such that Q0 AQ D T , where T is an (n n) upper triangular matrix whose diagonal elements are d1 ; : : : ; dn , respectively. Then, recalling Corollary 13.3.6, result (13.3.9), and Lemmas 13.2.1 and 13.1.1, we find that jAj D jQ2 jjAj D jQ0 AQj D jT j D

n Y

di :

iD1

And, recalling Lemma 5.2.1, we find that 0

0

tr.A/ D tr.AIn / D tr.AQQ / D tr.Q AQ/ D tr.T / D

n X

di :

iD1

Q.E.D. There is an alternative way of proving Theorem 21.6.1. Again take Q to be an n n orthogonal matrix such that Q0 AQ D T , where T is a triangular matrix whose diagonal elements are d1 ; : : : ; dn . And recall (from Section 1) that the characteristic polynomial of A is p./ D c0 C c1 C C cn1 n1 C cn n; with first and next-to-last coefficients that are expressible as c0 D jAj;

cn1 D .1/n1 tr.A/:

(6.3)

Clearly, T is similar to A and hence (according to Theorem 21.3.1) has the same characteristic polynomial as A. Thus, recalling Lemma 13.1.1, we find that c0 D jT j D

n Y iD1

di ;

cn1 D .1/n1 tr.T / D .1/n1

n X

di :

iD1

And, upon comparing these n1 with those in result (6.3), Q expressions for c0 and cP we conclude that jAj D niD1 di and that tr.A/ D niD1 di .

21.7 Some Results on the Moore-Penrose Inverse of a Symmetric Matrix The following lemma relates the eigenvalues and eigenvectors of the MoorePenrose inverse of an n n symmetric matrix A to those of A itself. Lemma 21.7.1. Let represent an eigenvalue of an n n symmetric matrix A, and let x represent any eigenvector (of A) corresponding to . Further, define ( 1=; if ¤ 0, C D 0; if D 0. Then, C is an eigenvalue of AC, and x is an eigenvector of AC corresponding to C.

21.7 Some Results on the Moore-Penrose Inverse of a Symmetric Matrix

547

Proof. By definition, Ax D x. If ¤ 0, then x D 1 Ax, and, upon observing (in light of Theorem 20.5.3) that AC is symmetric and hence that AC A D .AC A/0 D A0 .AC /0 D AAC, we find that AC x D AC .1 Ax/ D 1 ACAx D 1 AAC x D 1 AAC .1 Ax/ D .1 /2 AAC Ax D .1 /2 Ax D 1 .1 Ax/ D 1 x: Alternatively, if D 0, then Ax D 0, and hence [since, according to Theorem 20.5.1, N.AC / D N.A/] AC x D 0 D 0x. Q.E.D. Let A represent an n n symmetric matrix. And, as in Section 5f, let Q represent an n n orthogonal matrix and D D fdi g an n n diagonal matrix such that Q0 AQ D D; or, equivalently, let Q represent an n n matrix whose columns are orthonormal eigenvectors of A, and take D D fdi g to be the diagonal matrix whose first, : : : ; nth diagonal elements d1 ; : : : ; dn are the eigenvalues (of A) to which the first, : : : ; nth columns of Q correspond. Further, denote the first, : ; k represent the : : : ; nth columns of Q by q1 ; : : : ; qn , respectively; let 1 ; : :P distinct eigenvalues of A; and (for j D 1; : : : ; k) let Ej D i2Sj qi q0i , where Sj D fi W di D j g. Then, by definition, the spectral decomposition of A is A D QDQ0 D

n X

di qi q0i D

iD1

k X

j Ej :

j D1

The spectral decomposition of the Moore-Penrose inverse AC of the n n symmetric matrix A is closely related to that of A itself. Letting ( 1=di ; if di ¤ 0, C di D 0; if di D 0 (for i D 1; : : : ; n), it follows from Lemma 21.7.1 that q1 ; : : : ; qn are eigenvectors of AC and that the eigenvalues to which they correspond are d1C ; : : : ; dnC. Thus, the spectral decomposition of AC can be obtained from that of A by replacing each nonzero eigenvalue of A by its reciprocal. More explicitly, observing (in light of the results of Section 20.2) that D C D diag.d1C ; : : : ; dnC / and letting (for j D 1; : : : ; k) ( 1=j ; if j ¤ 0, jC D 0; if j D 0, the spectral decomposition of AC is AC D QD C Q0 D

n X iD1

diC qi q0i D

k X j D1

jC Ej :

(7.1)

548


21.8 Eigenvalues of Orthogonal, Idempotent, and Nonnegative Definite Matrices The following two theorems characterize the eigenvalues of orthogonal matrices and idempotent matrices. Theorem 21.8.1. If a scalar (real number) is an eigenvalue of an n n orthogonal matrix P , then D ˙1. Proof. Suppose that is an eigenvalue of P . Then, by definition, there exists a nonnull vector x such that P x D x, and, consequently, x0 x D x0 In x D x0 P 0 P x D .P x/0 P x D .x/0 .x/ D 2 x0 x:

(8.1)

Since x ¤ 0, we have that x0 x ¤ 0, and, upon dividing both sides of equality (8.1) Q.E.D. by x0 x, we find that 2 D 1 or, equivalently, that D ˙1. Theorem 21.8.2. An n n idempotent matrix A has no eigenvalues other than 0 or 1. Moreover, 0 is an eigenvalue of A if and only if A ¤ In , in which case it has a geometric and algebraic multiplicity of n rank.A/. Similarly, 1 is an eigenvalue of A if and only if A ¤ 0, in which case it has a geometric and algebraic multiplicity of rank.A/. And the geometric and algebraic multiplicities of the (one or two) distinct eigenvalues of A sum to n. Proof. Suppose that a scalar is an eigenvalue of A. Then, by definition, there exists a nonnull vector x such that Ax D x, and, consequently, x D Ax D A2 x D A.Ax/ D A.x/ D Ax D 2 x: Since x ¤ 0, it follows that 2 D , or, equivalently, that . 1/ D 0, and hence that D 0 or D 1. Moreover, if A D In , then (in light of Lemma 21.1.1) 0 is not an eigenvalue of A. Alternatively, if A ¤ In , then (according to Lemma 10.1.1) A is singular, and it follows from Lemma 21.1.1 that 0 is an eigenvalue of A of geometric multiplicity n rank.A/. Similarly, if A D 0, then there exists no nonnull vector x such that Ax D x, and, consequently, 1 is not an eigenvalue of A. Alternatively, if A ¤ 0, then (in light of Lemma 10.2.4) rank.A 1I/ D rankŒ.I A/ D rank.I A/ D n rank.A/ < n; implying that 1 is an eigenvalue of A and that its geometric multiplicity is n rank.A 1I/ D rank.A/. And if A D In or A D 0, then A has one distinct eigenvalue (1 or 0) of geometric multiplicity n; otherwise, A has two distinct eigenvalues (0 and 1) with geometric multiplicities [n rank.A/ and rank.A/] that sum to n. It remains only to observe (on the basis of Corollary 21.3.7) that the algebraic multiplicities of the distinct eigenvalues of A equal the geometric multiplicities. Q.E.D. Idempotent matrices of order n have a maximum of two distinct eigenvalues, and the geometric multiplicities of their distinct eigenvalues sum to n. Orthogonal

21.8 Eigenvalues of Orthogonal, Idempotent, and Nonnegative Definite Matrices

549

matrices of order n also have a maximum of two distinct eigenvalues, however the geometric (or algebraic) multiplicities of their distinct eigenvalues do not necessarily sum to n. In fact, for any positive integer n that is divisible by 2, there exist n n orthogonal matrices that have no eigenvalues (i.e., whose characteristic polynomials have no real roots). In the special case of an n n symmetric matrix or other n n diagonalizable matrix, the first part of Theorem 21.8.2 can be equipped with a converse and restated as the following theorem. Theorem 21.8.3. An n n symmetric matrix A or, more generally, an n n diagonalizable matrix A is idempotent if and only if it has no eigenvalues other than 0 or 1. Proof. It suffices to prove the “if” part of the theorem—the “only if” part is a direct consequence of the first part of Theorem 21.8.2. Suppose that A has no eigenvalues other than 0 or 1. By definition, there exists an n n nonsingular matrix Q such that Q1 AQ D D for some n n diagonal matrix D. And, according to Part (7) of Theorem 21.5.1, the diagonal elements of D are the (not necessarily distinct) eigenvalues of A. Thus, every diagonal element of D equals 0 or 1, implying that D 2 D D. Further, A D QQ1AQQ1 D QDQ1, and it follows that A2 D QDQ1 QDQ1 D QD 2 Q1 D QDQ1 D A: Q.E.D. The following theorem characterizes the eigenvalues of nonnegative definite matrices and positive definite matrices. Theorem 21.8.4. Every eigenvalue of a nonnegative definite matrix is nonnegative; every eigenvalue of a positive definite matrix is positive. Proof. Let represent an eigenvalue of a (square) matrix A, and let x represent an eigenvector of A corresponding to . Then, by definition, x ¤ 0, and, consequently, x0 x > 0. Further, D

x0 Ax x0 x

(as discussed in Section 1 and as is evident upon observing that Ax D x and hence that x0 Ax D x0 x). Now, if A is nonnegative definite, then x0 Ax 0, and consequently 0. Q.E.D. And if A is positive definite, then x0 Ax > 0, and consequently > 0. The following theorem gives conditions that can be used to determine whether a symmetric matrix is nonnegative definite and, if so, whether the matrix is positive definite or positive semidefinite. Theorem 21.8.5. An n n symmetric matrix A is nonnegative definite if and only if every eigenvalue of A is nonnegative, is positive definite if and only if every eigenvalue of A is (strictly) positive, and is positive semidefinite if and only if every eigenvalue of A is nonnegative and at least one eigenvalue is zero.

550


Proof. The theorem follows immediately from Corollary 14.2.15 upon recalling (from Section 5) that A D QDQ0, where Q is an n n orthogonal matrix and D is an n n diagonal matrix whose diagonal elements are the (not necessarily distinct) eigenvalues of A. Q.E.D.

21.9 Square Root of a Symmetric Nonnegative Definite Matrix A symmetric nonnegative definite matrix can be decomposed in the way described in the following theorem. Theorem 21.9.1. Corresponding to any n n symmetric nonnegative definite matrix A, there exists an n n symmetric nonnegative definite matrix R such that A D R2. Moreover, R is unique and is expressible as p

p (9.1) R D Q diag d1 ; : : : ; dn Q0; where d1 ; : : : ; dn are the (not necessarily distinct) eigenvalues of A and, letting D D diag.d1 ; : : : ; dn /, Q is any n n orthogonal matrix such that Q0 AQ D D or, equivalently, such that A D QDQ0 . Note that in Theorem 21.9.1 it is implicitly assumed that the eigenvalues of A are nonnegative and that there exists an n n orthogonal matrix Q such that Q0 AQ D D. The justification for these assumptions comes from Theorem 21.8.5 and Corollary 21.5.9. p p p p Proof. Let S D diag. d1 ; : : : ; dn /, so that Q diag. d1 ; : : : ; dn / Q0 D QSQ0 . Then, .QSQ0 /2 D QSIn SQ0 D QS2 Q0 D QDQ0 D A: Moreover, QSQ0 is symmetric and nonnegative definite (as is evident from Corollary 14.2.15). It remains to establish the uniqueness of QSQ0 ; that is, to show that there is no n n symmetric nonnegative definite matrix other than QSQ0 with a “squared value” equal to A. Let R represent any n n symmetric nonnegative definite matrix such that A D R2 . Then, .R QSQ0 /0 .R QSQ0 / D .R QSQ0 /.R QSQ0 / D R2 QSQ0 R RQSQ0 C .QSQ0 /2 D 2A QSQ0 R .QSQ0 R/0 : Thus, to establish the uniqueness of QSQ0, it suffices to show that QSQ0 R D A [since then .R QSQ0 /0 .R QSQ0 / D 0, implying that R QSQ0 D 0 or, equivalently, that R D QSQ0 ]. Since R is symmetric, there exists an n n orthogonal matrix P and an n n diagonal matrix T D fti g such that R D P T P 0 . And we find that

21.10 Some Relationships

551

Q 0 P T 2 D Q 0 P T In T In D Q0 P T P 0 P T P 0 P D Q0 R2 P D Q0 AP D Q0 .QSQ0 /2 P D In SIn SQ0 P D S2 Q0 P : Now, for i; j D 1; : : : ; n, let bij represent the ij th element of Q0 P , and observe that the ij th elements of Q0 P T 2 and S2 Q0 P are, respectively, tj2 bij and di bij . So, we have that tj2 bij D di bij . And, since (in light of Corollary 14.2.15) p p tj 0, it follows that tj bij D di bij . (If bij D 0, then tj bij D 0 D p di bij ; alternatively, if bij ¤ 0, then tj2 bij D di bij ) tj2 D di ) tj D di ) p p tj bij D di bij .) Moreover, tj bij and di bij are the ij th elements of Q0 P T and SQ0 P , respectively. Thus, Q0 P T D SQ0 P , and, consequently, QSQ0 R D QSQ0 P T P 0 D QSSQ0 P P 0 D QDQ0 In D A: Q.E.D. The nn symmetric nonnegative definite matrix R given by expression (9.1) is called the square root of the n n symmetric nonnegative definite matrix A and is denoted by the symbol A1=2. Note that if the matrix A is positive definite, then (as a consequence of Theorem 21.8.5) its eigenvalues d1 ; : : : ; dn are (strictly) positive, implying (in light of Corollary 14.2.15) that the matrix R is positive definite. Thus, the square root of a symmetric positive definite matrix is positive definite.

21.10 Some Relationships Let A represent an n n matrix, and let k represent an arbitrary scalar. The eigenvalues and eigenvectors of the difference A kIn and of the scalar multiple kA are related to those of A itself in a relatively simple way. Clearly, the condition Ax D x (where is a scalar and x a vector) is equivalent to the condition .A kI/x D . k/x. Thus, a scalar is an eigenvalue of A if and only if k is an eigenvalue of A kI. And an n 1 vector x is an eigenvector of A corresponding to the eigenvalue (of A) if and only if x is an eigenvector of A kI corresponding to the eigenvalue k (of A kI). Moreover, for any scalar , jA Ij D jA kI . k/Ij; so that the characteristic polynomials, say p./ and q./, of A and A kI, respectively, are such that (for all ) p./ D q. k/. And the algebraic and geometric multiplicities of an eigenvalue of A are the same as those of the corresponding eigenvalue k of A kI. If k D 0, then kA has only one distinct eigenvalue (namely, 0), any nonnull n 1 vector is an eigenvector of kA, the characteristic polynomial of kA is r./ D ./n, and the algebraic and geometric multiplicities of the eigenvalue 0 (of kA) equal n.

552


Suppose now that k ¤ 0. Then, the condition Ax D x (where is a scalar and x a vector) is equivalent to the condition kAx D kx. Thus, a scalar is an eigenvalue of A if and only if k is an eigenvalue of kA. And an n 1 vector x is an eigenvector of A corresponding to the eigenvalue (of A) if and only if x is an eigenvector of kA corresponding to the eigenvalue k (of kA). Moreover, for any scalar , jA Ij D k n jkA kIj; so that the characteristic polynomials, say p./ and r./ of A and kA, respectively, are such that (for all ) p./ D k n r.k/. And the algebraic and geometric multiplicities of an eigenvalue of A are the same as those of the corresponding eigenvalue k of kA. The following theorem relates the eigenvalues and eigenvectors of the n n product BA of an n m matrix B and an m n matrix A to those of the m m product AB. Theorem 21.10.1. Let A represent an m n matrix and B an n m matrix. Then, (1) the characteristic polynomials, say q./ and p./ of BA and AB, respectively, are such that (for all ) q./ D ./nm p./, ./mn q./ D p./, or q./ D p./, depending on whether m < n, m > n, or m D n; (2) the nonzero distinct eigenvalues and nonzero not-necessarily-distinct eigenvalues of BA are the same as those of AB; (3) for any matrix X whose columns are linearly independent eigenvectors of AB corresponding to a nonzero eigenvalue , the columns of BX are linearly independent eigenvectors of BA corresponding to ; and, similarly, for any matrix Y whose columns are linearly independent eigenvectors of BA corresponding to a nonzero eigenvalue , the columns of AY are linearly independent eigenvectors of AB corresponding to ; (4) each nonzero eigenvalue of AB (or, equivalently, of BA) has the same algebraic and geometric multiplicities when regarded as an eigenvalue of BA as when regarded as an eigenvalue of AB; and (5) the algebraic multiplicity of 0 when 0 is regarded as an eigenvalue (or potential eigenvalue) of BA equals n m plus the algebraic multiplicity of 0 when 0 is regarded as an eigenvalue (or potential eigenvalue) of AB. Proof. (1) Making use of Corollaries 13.2.4 and 18.1.2, we find that, for ¤ 0, p./ D jAB Im j D ./m jIm 1ABj and q./ D jBA In j D ./n jIn 1 BAj D ./n jIm 1ABj: Thus, for ¤ 0, q./ D ./nm p./, ./mn q./ D p./, and, in the special case where m D n, q./ D p./. And the proof of Part (1) is complete

21.10 Some Relationships

553

upon observing that if two polynomials in are equal for all in some nondegenerate interval, then they are equal for all —refer to Theorem 21.A.5 (in the Appendix)–or, alternatively, upon observing that if m < n, then BA is singular and hence q.0/ D 0; that if m > n, then AB is singular and hence p.0/ D 0; and that if m D n, then q.0/ D jBAj D jAjjBj D jABj D p.0/. (2) Part (2) is an almost immediate consequence of Part (1). (3) By definition, ABX D X: Thus, rank.X/ rank.BX/ rank.ABX/ D rank.X/ D rank.X/; so that rank.BX/ D rank.X/ and consequently (since BX has the same number of columns as X) the columns of BX are linearly independent. Moreover, BA.BX/ D B.ABX/ D B.X/ D BX; implying (since the columns of BX are linearly independent and hence nonnull) that the columns of BX are eigenvectors of BA corresponding to . That the columns of AY are linearly independent eigenvectors of AB corresponding to follows from an analogous argument. (4) It follows from Part (3) that, for each nonzero eigenvalue of AB (or, equivalently, of BA), dimŒN.BA I/ dimŒN.AB I/ and dimŒN.AB I/ dimŒN.BAI/ and hence dimŒN.BAI/ D dimŒN.ABI/. Thus, has the same geometric multiplicity when regarded as an eigenvalue of BA as when regarded as an eigenvalue of AB. That has the same algebraic multiplicity when regarded as an eigenvalue of BA as when regarded as an eigenvalue of AB is evident from Part (1). (5) Part (5) is an almost immediate consequence of Part (1). Q.E.D. In the special case where B D A0, the results of Theorem 21.10.1 can be augmented by the following result. Lemma 21.10.2. For any mn matrix A, the geometric multiplicity of 0 when 0 is regarded as an eigenvalue (or potential eigenvalue) of A0 A equals n m plus the geometric multiplicity of 0 when 0 is regarded as an eigenvalue (or potential eigenvalue) of AA0 . Proof. Making use of result (1.1) (or Lemma 11.3.1) and of Corollary 7.4.5, we find that dimŒN.A0 A 0In / D n rank.A0 A/ D n rank.A/ and that dimŒN.AA0 0Im / D m rank.AA0 / D m rank.A/: Thus,

dimŒN.A0 A 0In / D n m C dimŒN.AA0 0Im /: Q.E.D.

554


21.11 Eigenvalues and Eigenvectors of Kronecker Products of (Square) Matrices Eigenvalues and eigenvectors of a Kronecker product A˝B of two square matrices A and B can be obtained by combining eigenvalues and eigenvectors of A with eigenvalues and eigenvectors of B in the way described in the following theorem. Theorem 21.11.1. Let represent an eigenvalue of an m m matrix A, and let x represent an eigenvector of A corresponding to . Similarly, let represent an eigenvalue of a p p matrix B, and let y represent an eigenvector of B corresponding to . Then, is an eigenvalue of A ˝ B, and x ˝ y is an eigenvector of A ˝ B corresponding to . Proof. By definition, Ax D x, and By D y. Thus, .A ˝ B/.x ˝ y/ D .Ax/ ˝ .By/ D .x/ ˝ . y/ D .x ˝ y/: Moreover, x ˝ y is clearly nonnull. Q.E.D. In connection with Theorem 21.11.1, it should be noted that A ˝ B may have eigenvectors that are not expressible as the Kronecker product x ˝ y of an eigenvector x of A and an eigenvector y of B. Following and Neudecker Magnus 0 1 1 (1988, p. 29), suppose, for example, that B D A D , and let e1 D 0 0 0 0 . Then, A (or B) has only one distinct eigenvalue, namely 0; and and e2 D 1 every eigenvector of A (or of B) is a scalar multiple of e1 . And, in accordance with Theorem 21.11.1, 0 is an eigenvalue of A ˝ B, and e1 ˝ e1 is an eigenvector of A ˝ B corresponding to 0. However, e1 ˝ e2 and e2 ˝ e1 are also eigenvectors of A ˝ B corresponding to 0 (as is easily verified). The following theorem gives some basic results on the similarity of two Kronecker products. Theorem 21.11.2. Let R represent an mm matrix that is similar to an mm matrix A. Similarly, let S represent a p p matrix that is similar to a p p matrix B. Then, R ˝ S is similar to A ˝ B. And, for any m m nonsingular matrix C such that R D C 1AC and for any p p nonsingular matrix T such that S D T 1 BT , R ˝ S D .C ˝ T /1 .A ˝ B/.C ˝ T /: Proof. Since R is similar to A and S similar to B, there exist nonsingular matrices C and T such that R D C 1AC and S D T 1 BT . Moreover, recalling formula (16.1.23) (and that the Kronecker product of nonsingular matrices is nonsingular), we find that, for any such nonsingular matrices C and T , C ˝ T is nonsingular, and R ˝ S D .C 1 ˝ T 1 /.A ˝ B/.C ˝ T / D .C ˝ T /1 .A ˝ B/.C ˝ T /: Q.E.D.

21.11 Eigenvalues and Eigenvectors of Kronecker Products of (Square) Matrices

555

Upon recalling that (by definition) a (square) matrix A is diagonalizable if and only if there exists a diagonal matrix that is similar to A and that the Kronecker product of diagonal matrices is diagonal and observing [in light of results (16.1.15) and (16.1.23)] that a Kronecker product of orthogonal matrices is orthogonal, we obtain the following corollary. Corollary 21.11.3. Let A represent an m m matrix and B a p p matrix. If both A and B are diagonalizable, then A ˝ B is diagonalizable; if both A and B are orthogonally diagonalizable, then A ˝ B is orthogonally diagonalizable. And if A is diagonalized by an m m nonsingular matrix P and B by a p p nonsingular matrix Q, then A ˝ B is diagonalized by the (nonsingular) matrix P ˝ Q. Moreover, if P and Q are orthogonal, then P ˝ Q is also orthogonal. The following theorem gives a result on the not-necessarily-distinct eigenvalues of a Kronecker product of two (square) matrices. Theorem 21.11.4. Let A represent an m m matrix and B a p p matrix. Suppose that A has m (not necessarily distinct) eigenvalues, say d1 ; : : : ; dm , the last s of which equal 0. And let f1 ; : : : ; fk represent the (not necessarily distinct) eigenvalues of B (where possibly k < p). Then, A ˝ B has .m s/k C sp (not necessarily distinct) eigenvalues: d1 f1 , : : : ; d1 fk , : : : ; dms f1 , : : : ; dms fk , 0; : : : ; 0. Proof. As a consequence of Theorem 21.5.11, there exists an mm orthogonal matrix Q such that Q0 AQ D T , where T is an upper triangular matrix with diagonal elements d1 ; : : : ; dm . Further, Q ˝ Ip is an orthogonal matrix (as is easily verified), and .Q ˝ Ip /0 .A ˝ B/.Q ˝ Ip / D .Q0 AQ/ ˝ B D T ˝ B: Thus, A ˝ B is similar to T ˝ B, and it follows from Part (7) of Theorem 21.3.1 that the (not necessarily distinct) eigenvalues of A ˝ B are the same as those of T ˝ B. And T ˝ B is an (upper) block-triangular matrix with diagonal blocks d1 B, : : : ; dm B, so that [in light of Part (4) of Lemma 21.2.1 and the results of Section 10] T ˝ B has .m s/k C sp (not necessarily distinct) eigenvalues: d1 f1 , Q.E.D. : : : ; d1 fk , : : : ; dms f1 , : : : ; dms fk , 0; : : : ; 0. In the special case where k D p, Theorem 21.11.4 can be restated as the following corollary. Corollary 21.11.5. Let A represent an m m matrix and B a p p matrix. Suppose that A has m (not necessarily distinct) eigenvalues, say d1 ; : : : ; dm , and that B has p (not necessarily distinct) eigenvalues, say f1 ; : : : ; fp . Then, A ˝ B has mp (not necessarily distinct) eigenvalues: difj (i D 1; : : : ; m; j D 1; : : : ; p). Let A represent an m m matrix and B a p p matrix. According to Theorem 21.11.4 (and Corollary 21.11.5), a sufficient condition for A ˝ B to have mp (not necessarily distinct) eigenvalues is that A have m (not necessarily distinct) eigenvalues and B have p (not necessarily distinct) eigenvalues or that A D 0. However, this condition is not necessary. Suppose, for example, that

556


0

1 0 1 0 B D A D @1 0 0 A: 0 0 0 Then, the characteristic polynomial of A (or B) is p./ D jA Ij D .2 C 1/ (as can be easily seen by, e.g., making use of Theorem 13.3.1). Consequently, 0 is an eigenvalue of A and has an algebraic multiplicity of 1. Moreover, A has no eigenvalues other than 0—the characteristic polynomial of A has two imaginary roots, namely, i and i . By way of comparison, the characteristic polynomial of B ˝ A is (in light of Theorem 13.3.1) ˇ ˇ ˇ ˇ ˇ I A 0 ˇˇ ˇI3 ˇ A ˇˇ 0 ˇˇ D ./3 ˇˇ : q./ D jB ˝ A Ij D ˇˇ A I A I3 ˇ ˇ 0 0 I ˇ And, making use of results (13.3.13) and (13.2.2), we find that (for ¤ 0) ˇ ˇ ˇI3 A ˇˇ 1 ˇ ˇ A I3 ˇ D jI3 jjI .A/.I/ Aj D ./3 jI3 1A2 j D j./.I 1A2 /j D j2 I C A2 j ˇ 2 ˇ ˇ 1 0 0 ˇˇ ˇ 2 1 0 ˇˇ D .2 1/2 2 D 2 Œ. C 1/. 1/2 D ˇˇ 0 ˇ 0 0 2 ˇ and hence that (for all ) q./ D ./3 2 Œ. C 1/. 1/2 D .1/9 5 . C 1/2 . 1/2 : So, B ˝A has three distinct eigenvalues: 0, 1, and 1, with algebraic multiplicities of 5, 2, and 2, respectively. Thus, B˝A has 9 (not necessarily distinct) eigenvalues, whereas each of the matrices A and B (which are of order 3) has only one.

21.12 Singular Value Decomposition a. Definition, existence, and some basic properties and relationships Let A represent an n n symmetric nonnegative definite matrix. Then, it follows from the results of Section 5 (and from the nonnegativity of the eigenvalues of a nonnegative definite matrix) that A is expressible in the form

21.12 Singular Value Decomposition

ADQ

D1 0 Q0; 0 0

557

(12.1)

where Q is an n n orthogonal matrix and where D1 is a diagonal matrix with (strictly) positive diagonal elements. In fact, decomposition (12.1) is the spectral decomposition of A. The following theorem can be used to establish a generalization of this decomposition that is applicable to any matrix. Theorem 21.12.1. Let A represent an m n matrix of rank r. And take Q to be any n n orthogonal matrix and D1 to be any r r nonsingular diagonal matrix such that 2 D1 0 (12.2) Q0 A0 AQ D 0 0 2 D1 0 [where, when r D 0 or r D n, equals 0 or D12 , respectively]. Further, 0 0 partition Q as Q D .Q1 ; Q2 /, where Q1 has r columns, and let P D .P1 ; P2 /, where P1 D AQ1 D11 and where P2 is any m .m r/ matrix such that P10 P2 D 0:

(12.3)

(When r D 0, Q D Q2 , P D P2 , and P2 is arbitrary; when r D n, Q D Q1 ; and when r D m, P D P1 .) Then, D1 0 P 0 AQ D 0 0 D1 0 equals 0, .D1 ; 0/, [where, when r D 0, r D m, r D n, or r D m D n, 0 0 D1 , or D1 , respectively]. 0 Proof. Since 0 0 Q1 A AQ1 Q01 A0 AQ2 Q0 A0 AQ D ; Q02 A0 AQ1 Q02 A0 AQ2 we have that Q01 A0 AQ1 D D12 . Further, .AQ2 /0 AQ2 D Q02 A0 AQ2 D 0, implying (in light of Corollary 5.3.2) that AQ2 D 0. Thus, upon observing that P10 D D11 Q01 A0 and that AQ1 D P1 D1 , we find that 1 0 0 0 D1 Q1 A AQ1 P10 0 P1 AQ1 P10 AQ2 0 D P AQ D P20 AQ1 P20 AQ2 P20 0 P20 P1 D1 1 2 D1 0 0 D1 D1 D : D .P10 P2 /0 D1 0 0 0 Q.E.D. There exist an r r diagonal matrix D1 with (strictly) positive diagonal elements and an n n orthogonal matrix Q that satisfy condition (12.2) of Theorem

558


21.12.1, and there exists an m .m r/ matrix P2 that not only satisfies condition (12.3) but is such that P is orthogonal. To see this, take D1 D diag.s1 ; : : : ; sr /, where s1 ; : : : ; sr are the positive square roots of the r nonzero (and hence positive) not-necessarily-distinct eigenvalues of A0 A. [Since A0 A is symmetric and (in light of Corollary 14.2.14) nonnegative definite and since rank.A0 A/ D rank.A/ D r, it follows from Corollaries 21.3.8 and 21.5.8 that A0 A has r nonzero not-necessarilydistinct eigenvalues and from Theorem 21.8.5 that those eigenvalues are positive.] Now, choose the n n orthogonal matrix Q to satisfy condition (12.2)—that this is possible is evident from Corollary 21.5.9. Further, observe (as in the proof of Theorem 21.12.1) that Q01 A0 AQ1 D D12 and hence that P10 P1 D D11 Q01 A0 AQ1 D11 D D11 D12 D11 D Ir : And, observing (in light of Lemma 11.3.1) that dimŒN.P10 / D m rank.P10 / D m rank.P10 P1 / D m r; take P2 to be any m .m r/ matrix whose columns form an orthonormal basis for N.P10 /. Then, 0 Ir 0 P1 P1 P10 P2 0 PPD D D Im : P20 P1 P20 P2 0 Imr In light of this discussion, we have the following corollary of Theorem 21.12.1. Corollary 21.12.2. Corresponding to any m n matrix A of rank r, there exist an m m orthogonal matrix P and an n n orthogonal matrix Q such that D1 0 0 ; P AQ D 0 0 where D1 is an r r diagonal matrix with diagonal elements that are (strictly) positive. Let A represent an m n matrix. And let P represent an m m orthogonal matrix, Q an n n orthogonal matrix, and D1 D fsi g an r r diagonal matrix with (strictly) positive diagonal elements such that D1 0 P 0 AQ D 0 0 —the existence of such matrices is guaranteed by Corollary 21.12.2. Further, partition P and Q as P D .P1 ; P2 / and Q D .Q1 ; Q2 /, where P1 has r columns, say p1 ; : : : ; pr , respectively, and Q1 has r columns, say q1 ; : : : ; qr , respectively. Then, A can be expressed as D1 0 (12.4) Q0 ADP 0 0


or as

A D P1 D1 Q01

or AD

r X

559

(12.5)

si pi q0i :

(12.6)

iD1

Letting ˛1 ; : : : ; ˛k represent the distinct values represented among s1 ; : : : ; sr and (for j D 1; : : : ; k) letting Lj D fi W si D ˛j g, A can also be expressed as AD

k X

˛j Uj ;

(12.7)

j D1

P where (for j D 1; : : : ; k) Uj D i2Lj pi q0i . Expression (12.4) is called the singular-value decomposition of the m n matrix A. Sometimes, the term singular-value decomposition is used also in referring to expression (12.5), (12.6), or (12.7). The following theorem provides some insight into the nature of the various components of the singular-value decomposition and the extent to which those components are unique. Theorem 21.12.3. Let A represent an mn matrix. And take P to be an mm orthogonal matrix, Q an n n orthogonal matrix, and D1 an r r nonsingular diagonal matrix such that D1 0 P 0 AQ D : (12.8) 0 0 Further, partition P and Q as P D .P1 ; P2 / and Q D .Q1 ; Q2 /, where each of the matrices P1 and Q1 has r columns. Then, r D rank.A/; 2 D1 0 0 Q A AQ D 0 2 D1 P 0 AA0 P D 0

0 ; 0 0 ; 0

(12.9) (12.10) (12.11)

P1 D AQ1 D11;

(12.12)

Q1 D A0 P1 D11:

(12.13)

Proof. The veracity of result (12.9) is evident from equality (12.8) upon observing that rank.P 0 AQ/ D rank.A/. Further, 0 2 D1 0 D1 0 D1 0 0 0 0 0 0 0 0 0 D ; Q A AQ D Q A P P AQ D .P AQ/ P AQ D 0 0 0 0 0 0 which verifies equality (12.10). Equality (12.11) can be verified in similar fashion.

560


And, observing that P 0 AQ D ity (12.8)], we find that

0 P1 AQ1 P10 AQ2 [and making use of equalP20 AQ1 P20 AQ2

P1 D P1 D1 D11 D P1 .P10 AQ1 /D11 D .P1 P10 /AQ1 D11 D .Im P2 P20 /AQ1 D11 D AQ1 D11 P2 .P20 AQ1 /D11 D AQ1 D11 P2 0D11 D AQ1 D11: Q.E.D. That Q1 D A0 P1 D11 can be established via an analogous argument. In light of Theorem 21.5.1, it is clear from Theorem 21.12.3 that the scalars s1 , : : : ; sr , which appear in the singular-value decomposition (12.4) (as the diagonal elements of the diagonal matrix D1 ) are the positive square roots of the nonzero (not necessarily distinct) eigenvalues of A0 A (or, equivalently, of the nonzero notnecessarily-distinct eigenvalues of AA0 ). Moreover, they are unique (i.e., they do not vary with the choice of the orthogonal matrices P and Q that appear in the singular-value decomposition), and they are equal in number to rank.A/. The scalars s1 ; : : : ; sr are referred to as the singular values of the matrix A. (In some presentations, the positive square roots of all n or m eigenvalues of A0 A or AA0, including those that equal 0, are regarded as singular values of A.) The scalars ˛1 ; : : : ; ˛k , which appear in decomposition (12.7) and which (by definition) are the distinct singular values of A, are the positive square roots of the distinct nonzero eigenvalues of A0 A (or, equivalently, of the distinct nonzero eigenvalues of AA0 ). Theorem 21.12.3 is also informative about the nature of the orthogonal matrices P and Q, which appear in the singular-value decomposition (12.4). The m columns of P are eigenvectors of AA0, with the first r columns corresponding to the nonzero eigenvalues s12 ; : : : ; sr2 and the remaining m r columns corresponding to the 0 eigenvalues. Similarly, the n columns of Q are eigenvectors of A0 A, with the first r columns corresponding to the nonzero eigenvalues s12 ; : : : ; sr2 and the remaining n r columns corresponding to the 0 eigenvalues. Moreover, once the first r columns of Q are specified, the first r columns of P are uniquely determined [by result (12.12)]. Similarly, once the first r columns of P are specified, the first r columns of Q are uniquely determined [by result (12.13)]. For any fixed ordering of the distinct singular values ˛1 ; : : : ; ˛k , decomposition (12.7) is unique. To see this, observe [in light of result (12.12)] that (for i D 1; : : : ; r) pi D si1Aqi . Thus, for j D 1; : : : ; k, X X pi q0i D si1Aqi q0i D ˛j1AEj ; Uj D i2Lj

P

i2Lj

where Ej D i2Lj qi q0i . Further, since (for i 2 Lj ) qi is an eigenvector of A0 A corresponding to the eigenvalue ˛j2 , it follows from the results of Section 5f (on the uniqueness of the spectral decomposition) that Ej does not vary with the choice


561

of P , Q, and D1 and hence that Uj does not vary with this choice. We conclude that decomposition (12.7) is unique (aside from the ordering of the terms). The (usual) norm of the m n matrix A with singular-value decomposition (12.4) can be expressed in terms of the singular values s1 ; : : : ; sr . Making use of Lemma 5.2.1, we find that kAk2 D tr.A0 A/ 0 D1 0 D1 0 0 PP D tr Q Q0 0 0 0 0 0 D1 0 D1 0 0 0 D tr PP QQ 0 0 0 0 2 0 D1 0 D1 0 D1 0 D tr Im In D tr D s12 C C sr2 : 0 0 0 0 0 0 Thus,

kAk D .s12 C C sr2 /1=2:

(12.14)

Let us now consider the singular-value decomposition of an n n symmetric matrix A. Let d1 ; : : : ; dn represent the (not necessarily distinct) eigenvalues of A, ordered in such a way that d1 ; : : : ; dr are nonzero and drC1 D D dn D 0. And let q1 ; : : : ; qn represent orthonormal eigenvectors of A corresponding to d1 ; : : : ; dn , respectively, and (for i D 1; : : : ; n) let ( C1; if di 0,

i D 1; if di < 0. Further, define D D diag.d1 ; : : : ; dn /, Q D .q1 ; : : : ; qn /, and D diag. 1 , : : : ; n /; and take P D Q to be the n n matrix whose i th column pi is either qi or qi depending on whether di 0 or di < 0. Then, P and Q are orthogonal. And Q0 AQ D D, so that P 0 AQ D Q0 AQ D D D diag. 1 d1 ; : : : ; n dn / D diag.jd1 j; : : : ; jdn j/ D

D1 0 ; 0 0

where D1 D diag.jd1 j; : : : ; jdn j/. Thus, the singular values of A are the absolute values jd1 j; : : : ; jdr j of its nonzero eigenvalues. And the singular-value decomposition of A [decomposition (12.6)] is AD

r X

jdi jpi q0i

iD1

D

r X

jdi j. i qi q0i /:

iD1

By way of comparison, the spectral decomposition of A [decomposition (5.4)] is AD

n X iD1

di qi q0i D

r X iD1

di qi q0i :

562


Note that, in the special case where the symmetric matrix A is nonnegative definite, d1 ; : : : ; dr are positive, so that (for i D 1; : : : ; n) i D 1, jdi j D di , and pi D qi . Thus, in this special case, the singular values of A are its nonzero eigenvalues, and the singular-value decomposition of A is essentially the same as the spectral decomposition of A. b. Singular-value decomposition of the Moore-Penrose inverse Let A represent an m n matrix. And let P represent an m m orthogonal matrix, Q an n n orthogonal matrix, and D1 D fsi g an r r diagonal matrix with (strictly) positive diagonal elements such that D1 0 0 : P AQ D 0 0 Then, by definition, the singular-value decomposition of A is D 0 ADP 1 Q0 : 0 0 According to Theorem 20.5.6, C D1 0 A DQ P 0: 0 0 C

And, in light of the results of Section 20.2, C C D1 0 D1 0 E1 0 D D ; 0 0 0 0 0 0 where E1 D diag.1=s1 ; : : : ; 1=sr /. Thus, E1 0 0 C P; A DQ 0 0 or, equivalently,

(12.15)

E1 0 QA P D : 0 0 0

C

Clearly, expression (12.15) is the singular-value decomposition of AC. And the singular values of AC are the reciprocals of the singular values of A. c. Approximation of a matrix by a matrix of smaller rank The following theorem suggests that if some of the singular values of an m n matrix A are relatively small, the m n matrix obtained from the singular-value decomposition (of A) by setting those singular values equal to zero may provide a “good” approximation to A.


563

Theorem 21.12.4. Let A represent an m n matrix (of rank r) with singular values s1 ; s2 ; : : : ; sr ordered so that s1 s2 sr . Let D1 D diag.s1 ; s2 ; : : : ; sr /, and let P represent an m m orthogonal matrix and Q an n n orthogonal matrix such that P 0 AQ D diag.D1 ; 0/, so that D1 0 ADP Q0 0 0 is the singular-value decomposition of A. Then, for any m n matrix B of rank k or less (where k < r), 2 C C sr2 kB Ak2 skC1

(12.16)

(where the norm is the usual norm). Moreover, equality is attained in inequality (12.16) by taking D 0 BDP 1 Q0 ; 0 0 where D1 D diag.s1 ; : : : ; sk /. As a preliminary to proving Theorem 21.12.4, it is convenient to establish the following theorem, which is of some interest in its own right. Theorem 21.12.5. Let A represent an n n symmetric matrix with (not necessarily distinct) eigenvalues d1 ; d2 ; : : : ; dn ordered so that d1 d2 dn . Then, for any n k matrix X such that X0 X D Ik or, equivalently, for any n k matrix X with orthonormal columns (where k n), tr.X0 AX/

k X

di ;

iD1

with equality holding if the columns of X are orthonormal eigenvectors of A corresponding to d1 ; d2 ; : : : ; dk , respectively. Proof (of Theorem 21.12.5). Let U represent an n n matrix whose first, : : : ; nth columns are orthonormal (with respect to the usual inner product) eigenvectors of A corresponding to d1 ; : : : ; dn , respectively. Then, there exists an n k matrix R D frij g such that X D U R. Further, U 0 AU D D, where D D diag.d1 ; : : : ; dn /. Thus, tr.X0 AX/ D tr.R0 U 0 AU R/ D tr.R0 DR/ D

k X n X j D1 iD1

di rij2

D

n X

w i di ;

iD1

P where (for i D 1; : : : ; n) wi D jkD1 rij2 . The scalars w1 ; : : : ; wn are such that 0 wi 1

(12.17)

564


(for i D 1; : : : ; n) and

To see this, observe that

n X

wi D k:

(12.18)

iD1

R0 R D R0 In R D R0 U 0 U R D X0 X D Ik (which indicates that the columns of R are orthonormal). Then, as a consequence of Theorem 6.4.5, there exists an n .n k/ matrix S such that the columns of the matrix .R; S/ form an orthonormal basis for Rn or, equivalently, such that .R; S/ is an orthogonal matrix. And RR0 C SS0 D .R; S/.R; S/0 D In , so that In RR0 D SS0 ; implying that In RR0 is nonnegative definite. Thus, since clearly the i th diagonal element of In RR0 equals 1 wi , we have that 1 wi 0, which (together with the obvious inequality wi 0) establishes result (12.17). Moreover, n X iD1

wi D

k X n X

rij2 D tr.R0 R/ D tr.Ik / D k;

j D1 iD1

which establishes result (12.18). P P Results (12.17) and (12.18) imply that niD1 wi di kiD1 di (as is easily verified) and hence that k X di : tr.X0 AX/ iD1

Moreover, in the special case where the columns of X are orthonormal eigenvectors of A corresponding to d1 ; : : : ; dk , respectively, AX D X diag.d1 ; : : : ; dk / and hence X0 AX D X0 X diag.d1 ; : : : ; dk / D diag.d1 ; : : : ; dk /: P Thus, in that special case, tr.X0 AX/ D kiD1 di . Q.E.D. Proof (of Theorem 21.12.4). Corresponding to any m n matrix B of rank k or less, there exists an m k matrix U with orthonormal columns such that C.B/ C.U / or, equivalently, such that B D U L for some k n matrix L. Now, let U represent an arbitrary m k matrix with orthonormal columns or, equivalently, an arbitrary m k matrix such that U 0 U D Ik . And let L represent an arbitrary k n matrix. Then, to verify inequality (12.16), it suffices to show that 2 C C sr2 : kU L Ak2 skC1 We have that kU L Ak2 D trŒ.U L A/0 .U L A/ D tr.L0 L A0 U L L0 U 0 A/ C tr.A0 A/: And, since


565

L0 L A0 U L L0 U 0 A C A0 U U 0 A D .L U 0 A/0 .L U 0 A/ and since .L U 0 A/0 .L U 0 A/ is a nonnegative definite matrix, we find (in light of Theorem 14.7.2) that tr.L0 L A0 U L L0 U 0 A C A0 U U 0 A/ 0 and hence that tr.L0 L A0 U L L0 U 0 A/ tr.A0 U U 0 A/ D tr.U 0 AA0 U /: Thus,

kU L Ak2 tr.A0 A/ tr.U 0 AA0 U /:

Moreover, since (in light of the results of Subsection a) s1 ; : : : ; sr are the positive square roots of the nonzero eigenvalues of A0 A (or, equivalently, of the nonzero eigenvalues of AA0 ), it follows from Theorem 21.6.1 that tr.A0 A/ D

r X

si2

iD1

and from Theorem 21.12.5 that tr.U 0 AA0 U /

k X

si2 :

iD1

We conclude that kU L Ak2

r X iD1

si2

k X

2 si2 D skC1 C C sr2 :

iD1

And the proof is complete upon observing (in light of Lemma 8.4.2) that 2 2 D 0 0 P 1 D P D1 0 Q0 P D1 0 Q0 A Q 0 0 0 0 0 0 2 D 0 D1 0 2 2 1 D 0 0 0 0 D skC1 C C sr : Q.E.D. In the special case where the matrix A is symmetric, Theorem 21.12.4 can (in light of the results of Subsection a) be restated in terms related to the spectral decomposition of A, as described in the following corollary. Corollary 21.12.6. Let A represent an nn symmetric matrix (of rank r) with nonzero (not necessarily distinct) eigenvalues d1 ; : : : ; dr ordered so that jd1 j jd2 j jdr j. Let D1 D diag.d1 ; : : : ; dr /, and let Q represent an n n orthogonal matrix such that Q0 AQ D diag.D1 ; 0/, so that D1 0 ADQ Q0 0 0

566


is the spectral decomposition of A. Then, for any n n matrix B of rank k or less (where k < r) 2 C C dr2 : (12.19) kB Ak2 dkC1 Moreover, equality is attained in inequality (12.19) by taking D 0 Q0 ; BDQ 1 0 0 where D1 D diag.d1 ; : : : ; dk /.

21.13 Simultaneous Diagonalization In Section 21.5, we considered the diagonalization of an n n matrix. Now, let A1 ; : : : ; Ak represent k matrices of dimensions n n, and consider the conditions under which there exists a single n n nonsingular matrix Q that diagonalizes all k of these matrices; that is, the conditions under which there exists an n n nonsingular matrix Q such that Q1A1 Q D D1 , : : : ; Q1Ak Q D Dk for some diagonal matrices D1 ; : : : ; Dk . When such a nonsingular matrix Q exists, Q is said to simultaneously diagonalize A1 ; : : : ; Ak (or A1 ; : : : ; Ak are said to be simultaneously diagonalized by Q), and A1 ; : : : ; Ak are referred to as being simultaneously diagonalizable. Suppose that there exists an n n nonsingular matrix Q such that Q1A1 Q D D1 , : : : ; Q1Ak Q D Dk for some diagonal matrices D1 ; : : : ; Dk . Then, for s ¤ i D 1; : : : ; k, Q1As Ai Q D Q1As QQ1Ai Q D Ds Di D Di Ds D Q1Ai QQ1As Q D Q1Ai As Q; implying that As Ai D Q.Q1As Ai Q/Q1 D Q.Q1Ai As Q/Q1 D Ai As : Thus, a necessary condition for A1 ; : : : ; Ak to be simultaneously diagonalizable is that .s > i D 1; : : : ; k/ (13.1) As Ai D Ai As (i.e., that A1 ; : : : ; Ak commute in pairs). For symmetric matrices A1 ; : : : ; Ak , condition (13.1) is sufficient, as well as necessary, for A1 ; : : : ; Ak to be simultaneously diagonalizable. In fact, symmetric matrices A1 ; : : : ; Ak that satisfy condition (13.1) can be simultaneously diagonalized by an orthogonal matrix. To verify this, let us proceed by mathematical induction. In the case of one symmetric matrix A1 , we know (from Theorem 21.5.7) that there exists an orthogonal matrix that “simultaneously” diagonalizes A1 —when

21.13 Simultaneous Diagonalization

567

k D 1, condition (13.1) is vacuous. Suppose now that any k1 symmetric matrices (of the same order) that commute in pairs can be simultaneously diagonalized by an orthogonal matrix (where k 2). And let A1 ; : : : ; Ak represent k symmetric matrices of arbitrary order n that commute in pairs. Then, to complete the induction argument, it suffices to show that A1 ; : : : ; Ak can be simultaneously diagonalized by an orthogonal matrix. Let 1 ; : : : ; r represent the distinct eigenvalues of Ak , and let 1 ; : : : ; r represent their respective multiplicities. Take Qj to be an n j matrix whose columns are orthonormal (with respect to the usual inner product) eigenvectors of Ak corresponding to the eigenvalue j or, equivalently, whose columns form an orthonormal basis for the j -dimensional linear space N.Ak j I/ (j D 1; : : : ; r). And define Q D .Q1 ; : : : ; Qr /. Since (according P to Theorem 21.5.7) Ak is diagonalizable, it follows from Corollary 21.5.4 that jr D1 j D n, so that Q has n (orthonormal) columns and hence is an orthogonal matrix. Further, Q0 Ak Q D diag.1 I1 ; : : : ; r Ir /

(13.2)

(as is evident from Theorems 21.5.2 and 21.5.1). Since Ak commutes with each of the matrices A1 ; : : : ; Ak1 , we find that, for i D 1; : : : ; k 1 and j D 1; : : : ; r, Ak .Ai Qj / D Ai .Ak Qj / D Ai .j Qj / D j Ai Qj : Thus, .Ak j I/Ai Qj D 0, implying (in light of Lemma 11.4.1) that C.Ai Qj / N.Ak j I/. And, since the columns of Qj form a basis for N.Ak j I/, there exists a j j matrix Bij such that Ai Qj D Qj Bij

(13.3)

(i D 1; : : : ; k 1; j D 1; : : : ; r), and we find that (for i D 1; : : : ; k 1) Q0 Ai Q D Q0 .Ai Q1 ; : : : ; Ai Qr / D Q0 .Q1 Bi1 ; : : : ; Qr Bir / D Q0 Q diag.Bi1 ; : : : ; Bir / D diag.Bi1 ; : : : ; Bir /:

(13.4)

Now, using result (13.3), we find that, for j D 1; : : : ; r, Bij D Ij Bij D Qj0 Qj Bij D Qj0 Ai Qj (i D 1; : : : ; k 1) and that Bsj Bij D Qj0 As Qj Bij D Qj0 As Ai Qj D Qj0 Ai As Qj D Qj0 Ai Qj Bsj D Bij Bsj (s > i D 1; : : : ; k 1), so that the k 1 matrices B1j ; : : : ; Bk1;j (which are of order j ) are symmetric and they commute in pairs. Thus, by supposition, B1j ; : : : ; Bk1;j can be simultaneously diagonalized by an orthogonal matrix;

568


that is, there exists a j j orthogonal matrix Sj and j j diagonal matrices D1j ; : : : ; Dk1;j such that (for i D 1; : : : ; k 1) Sj0 Bij Sj D Dij :

(13.5)

Define S D diag.S1 ; : : : ; Sr / and P D QS. Then, clearly, S is orthogonal and, hence, according to Lemma 8.4.1, P is also orthogonal. Further, using results (13.4), (13.5), and (13.2), we find that, for i D 1; : : : ; k 1, P 0 Ai P D S0 Q0 Ai QS D diag.S10 ; : : : ; Sr0 / diag.Bi1 ; : : : ; Bir / diag.S1 ; : : : ; Sr / D diag.S10 Bi1 S1 ; : : : ; Sr0 Bir Sr / D diag.Di1 ; : : : ; Dir /

and that P 0 Ak P D S0 Q0 Ak QS D diag.S10 ; : : : ; Sr0 / diag.1 I1 ; : : : ; r Ir / diag.S1 ; : : : ; Sr / D diag.1 S10 S1 ; : : : ; r Sr0 Sr / D diag.1 I1 ; : : : ; r Ir /;

so that all k of the matrices A1 ; : : : ; Ak are simultaneously diagonalized by the orthogonal matrix P . Summarizing, we have the following theorem. Theorem 21.13.1. If n n matrices A1 ; : : : ; Ak are simultaneously diagonalizable, then they commute in pairs, that is, for s > i D 1; : : : ; k, As Ai D Ai As . If n n symmetric matrices A1 ; : : : ; Ak commute in pairs, then they can be simultaneously diagonalized by an orthogonal matrix; that is, there exist an n n orthogonal matrix P and diagonal matrices D1 ; : : : ; Dk such that, for i D 1; : : : ; k, P 0 A i P D Di : In connection with Theorem 21.13.1, note that symmetric matrices A1 ; : : : ; Ak commute in pairs if and only if each of the k.k 1/=2 matrix products A1 A2 , A1 A3 , : : : ; Ak1 Ak is symmetric. In the special case where k D 2, Theorem 21.13.1 can be restated as the following corollary. Corollary 21.13.2. If two n n matrices A and B are simultaneously diagonalizable, then they commute, that is, BA D AB. If two n n symmetric matrices A and B commute (or, equivalently, if their product AB is symmetric), then they can be simultaneously diagonalized by an orthogonal matrix; that is, there exists an n n orthogonal matrix P such that P 0 AP D D1 and P 0 BP D D2 for some diagonal matrices D1 and D2 .

21.14 Generalized Eigenvalue Problem

569

21.14 Generalized Eigenvalue Problem Let A represent an n n symmetric matrix, and let B represent an n n symmetric positive definite matrix. Then, according to Corollary 14.3.14, there exists an n n nonsingular matrix Q such that the quadratic form x0 Bx (in an n 1 vector x) Pn 2 is expressible as the sum of squares iD1 yi of the elements y1 ; : : : ; yn of the transformed vector y D Q0 x. And, according to Corollary 14.3.6, there exists ; : : : ; dn such that the quadratic an n n nonsingular matrix Q and n scalars d1P form x0 Ax is expressible as a linear combination niD1 di yi2 of the squares of the elements y1 ; : : : ; yn of the transformed vector y D Q0 x. Can the same n n nonsingular matrix Q be used to so express both x0 Bx and 0 x Ax? Or, equivalently, does there exist an n n nonsingular matrix Q such that B D QQ0 and A D QDQ0 for some diagonal matrix D? The answer is yes, as is shown in the following development. Let R represent any n n nonsingular matrix such that B1 D R0 R. And let C D RAR0, and define P to be an n n orthogonal matrix such that P 0 CP D D for some diagonal matrix D D fdi g. Now, take Q D R1 P . Then, QQ0 D R1 P P 0 .R0 /1 D R1 .R0 /1 D B;

(14.1)

and QDQ0 D R1 P P 0 CP P 0 .R0 /1D R1 C.R0 /1D R1 RAR0 .R0 /1D A: (14.2) In addition to establishing the existence of an n n nonsingular matrix Q such that B D QQ0 and A D QDQ0 (for some diagonal matrix D), the preceding development suggests an indirect “strategy” for constructing such a matrix. This strategy comprises four steps: (1) construction of the n n nonsingular matrix R (such that B1 D R0 R); (2) formation of the matrix C D RAR0 ; (3) determination of the n n orthogonal matrix P that diagonalizes C; and (4) the formation of the matrix Q D R1 P . In connection with Step (3), note that the columns of the n n orthogonal matrix P can be any n orthonormal eigenvectors of C. Let us now consider an alternative approach to the construction of an n n nonsingular matrix Q such that B D QQ0 and A D QDQ0 (for some diagonal matrix D). In the alternative approach, the columns of the matrix .Q0 /1 are taken to be orthonormal “generalized eigenvectors” of A, where the orthonormality of these vectors is with respect to an inner product other than the usual inner product. Let X D .Q0 /1 (where Q represents an arbitrary n n nonsingular matrix), and let D D fdi g represent an arbitrary n n diagonal matrix. Then, B D QQ0 and A D QDQ0

, ,

BX D Q and AX D QD BX D Q and AX D BXD

,

X0 BX D I and AX D BXD:

And, letting xi represent the i th column of X (i D 1; : : : ; n),

570


( 0

X BX D I

xj0 Bxi

,

D

1; for j D i D 1; : : : ; n, 0; for j ¤ i D 1; : : : ; n,

and AX D BXD

,

Axi D di Bxi ; for i D 1; : : : ; n.

Accordingly, the alternative approach to the construction of an n n nonsingular matrix Q such that B D QQ0 and A D QDQ0 (for some diagonal matrix D) consists of finding n solutions .d1 ; x1 /; : : : I .dn ; xn / (for a scalar and an n 1 vector x) to the equation Ax D Bx (14.3) such that x1 ; : : : ; xn are orthogonal with respect to the inner product x0 By and of then taking .Q0 /1 to be the matrix whose columns are x1 ; : : : ; xn . Various properties of the “x- and -parts” of solutions to equation (14.3) can be deduced by observing that Ax D Bx

,

C.R1 /0 x D .R1 /0 x

(14.4)

and that (for any n 1 vectors x and y) x0 By D Œ.R1 /0 x0 .R1 /0 y:

(14.5)

Thus, a scalar and an n-dimensional vector x form a solution to equation (14.3) if and only if is an eigenvalue of C and .R1 /0 x is an eigenvector of C corresponding to . Further, an n-dimensional vector x has unit norm with respect to the inner product x0 By if and only if .R1 /0 x has unit norm with respect to the usual inner product. And two n-dimensional vectors x and y are orthogonal with respect to the inner product x0 By if and only if .R1 /0 x and .R1 /0 y are orthogonal with respect to the usual inner product. Recalling Theorem 21.4.5, we conclude in particular that x01 Bx2 D 0 for any two solutions .1 ; x1 / and .2 ; x2 / to equation (14.3) such that 2 ¤ 1 . Since Ax D Bx , .A B/x D 0; (14.6) Ax D Bx for some n-dimensional vector x if and only if jA Bj D 0. The problem of finding the roots of jA Bj (i.e., the solutions for to the equation jA Bj D 0) is called the generalized eigenvalue problem. Note that jA Bj D jRj2 jC Ij D jBjjAB1 Ij D jBjjB1A Ij;

(14.7)

so that the generalized eigenvalue problem is equivalent to the problem of finding the eigenvalues of the symmetric matrix C or of the not-necessarily-symmetric matrix AB1 or B1A. Note also that, in the special case where B D I, the generalized eigenvalue problem reduces to the problem of finding the eigenvalues of A.

21.15 Differentiation of Eigenvalues and Eigenvectors

571

21.15 Differentiation of Eigenvalues and Eigenvectors The following lemma provides an appropriate context for our discussion of the differentiation of eigenvalues and eigenvectors. Lemma 21.15.1. Let F represent a p p symmetric matrix of functions, defined on a set S , of a vector x D .x1 ; : : : ; xm /0 of m variables. Further, let c represent any interior point of S at which F is continuously differentiable. And let represent a simple eigenvalue of F .c/ (i.e., an eigenvalue of multiplicity 1), and let u D .u1 ; : : : ; up /0 represent an eigenvector [of F .c/] with (usual) norm 1 corresponding to . Then, for some neighborhood N of c, there exists a function (of x), defined on N , and a vector u D .u1 ; : : : ; up /0 of functions (of x), defined on N , with the following properties: (1) .c/ D and u.c/ D u ; (2) for x 2 N , .x/ is an eigenvalue of F .x/, and u.x/ is an eigenvector [of F .x/] with (usual) norm 1 corresponding to .x/ [and the signs of the nonzero elements of u.x/ are the same as those of the corresponding elements of u ]; and (3) and u are continuously differentiable at c. Proof. Regarding (for the moment) and u as a variable and a vector of variables, define (for x 2 S , 2 R, and u 2 Rp ) .F .x/ I/u : h.u; I x/ D u0 u 1 Further, let

J .u; I x/ D

@h.u; I x/ @h.u; I x/ : ; @u0 @

And observe [in light of results (15.3.10) and (15.3.7)] that F .x/ I u J .u; I x/ D : 2u0 0 Now, since is a simple eigenvalue of F .c/, rankŒF .c/ I D p 1: And, since ŒF .c/ I0 u D ŒF .c/ Iu D 0, C.u / is (in light of Corollary 12.1.2) orthogonal to CŒF .c/ I, implying (in light of Lemma 17.1.9) that C.u / and CŒF .c/ I are essentially disjoint and also (in light of Lemma 17.2.1) that R.u0 / and RŒF .c/ I are essentially disjoint. Thus, making use of Theorem 17.2.19, we find that rankŒJ .u; I c/ D rankŒF .c/ I C rank.u / C rank.2u0 / D p 1 C 1 C 1 D p C 1: Moreover,

h.u; I c/ D 0;

and h is continuously differentiable at the point .u; I c/.

572


The upshot of these results is that the conditions of the implicit function theorem (e.g., Magnus and Neudecker 1988, ch. 7, thm. A.2; Bartle 1976, p. 384) are satisfied. And it follows from this theorem that, for some neighborhood N of c, there exist a function (of x), defined on N , and a vector u D .u1 ; : : : ; up /0 of functions (of x), defined on N , with the following properties: (1) .c/ D and u.c/ D u ; (2) for x 2 N , .x/ is an eigenvalue of F .x/, and u.x/ is an eigenvector [of F .x/] with (usual) norm 1 corresponding to .x/; and (3) and u are continuously differentiable at c. Moreover, in light of Lemma 15.1.1, the elements of u are continuous at c, so that there exists some neighborhood N of c (where N N ) such that the signs of the nonzero elements of u.x/ are the same Q.E.D. as those of the corresponding elements of u. Let us now consider the differentiation of the functions , u1 ; : : : ; un described in Lemma 21.15.1. For x 2 N , we have that F u D u: Upon differentiating both sides of this equality with respect to xj , we find that (at x D c) @u @F @u @ F C uD C u (15.1) @xj @xj @xj @xj (j D 1; : : : ; m). And, upon premultiplying both sides of equality (15.1) by u0, we obtain @F @u @u @ 0 C u0 u D u0 C u u; u0 F @xj @xj @xj @xj or, equivalently [since u0 F D .F u/0 D u0 and u0 u D 1], u0

@u @F @u @ C u0 u D u0 C : @xj @xj @xj @xj

And it follows that (at x D c) @ @F D u0 u @xj @xj

(15.2)

(j D 1; : : : ; m). As a formula for the j th partial derivative of u, we have that (at x D c) @u @F D HC u; @xj @xj

(15.3)

where H D F I. To verify formula (15.3), rewrite equality (15.1) as H

@ @F @u D u u: @xj @xj @xj

Premultiplying both sides of equality (15.4) by HC, we obtain

(15.4)

21.15 Differentiation of Eigenvalues and Eigenvectors

HC H

@F @u @ C D H u HC u: @xj @xj @xj

573

(15.5)

Since HC H D .HC H/0 D H.HC /0 D HHC and since [observing that Hu D 0 and recalling (from Theorem 20.5.1) that N.HC / D N.H/] HC u D 0, equality (15.5) can be rewritten as HHC

@u @F D HC u: @xj @xj

To complete the verification of formula (15.3), we need to show that (at x D c) HHC

@u @u D : @xj @xj

(15.6)

For this purpose, observe that

@u 0 @u u C u0 D0 @xj @xj

(as is clear upon differentiating both sides of the equality u0 u D 1) and hence {since .@u=@xj /0 u D Œ.@u=@xj /0 u0 D u0 .@u=@xj /} that u0 Thus,

@u D 0: @xj

@u D 00; u0 H; @xj

implying that the p rows of the matrix .H; @u=@xj / are linearly dependent and hence that @u p 1: (15.7) rank H; @xj Moreover, at x D c, @u p 1 D rank.H/ rank H; : @xj

(15.8)

Together, results (15.7) and (15.8) imply that (at x D c) @u rank H; D rank.H/: @xj It follows (in light of Corollary 4.5.2) that @u=@xj 2 C.H/, which (upon applying Corollary 9.3.6) gives result (15.6). Let us now consider formulas (15.2) and (15.3) in the special case where x is an [m.m C 1/=2]-dimensional column vector whose elements consist of those elements of an m m symmetric matrix X D fxij g that lie on or below the

574


diagonal and where F .x/ D X. Assume that S D Rm.mC1/=2 (i.e., that there are no restrictions on X other than symmetry), observe that (in this special case) the interior points of S at which F is continuously differentiable consist of all of S , and let ej represent the j th column of Im . Then, upon substituting from result (15.5.6) or (15.5.7) into equality (15.2), we find that @ D u0 ei e0i u D u2i @xi i and, for j < i , @ D u0 .ei ej0 C ej e0i /u D ui uj C uj ui D 2ui uj : @xij Thus,

( @ u2i ; D @xij 2ui uj ;

if j D i , if j < i .

(15.9a) (15.9b)

Or, equivalently, @ D 2uu0 diag.u21 ; : : : ; u2m /: @X

(15.10)

Further, let gj represent the j th column of HC and gij the ij th element of H . Then, upon substituting from result (15.5.6) or (15.5.7) into equality (15.3), we find that ( @u ui gi ; if j D i , (15.11a) D @xij (15.11b) .uj gi C ui gj /; if j < i . C

And the sth element of the (m-dimensional) vector @u=@xij is expressible as ( @us ui gsi ; if j D i , (15.12a) D @xij (15.12b) .uj gsi C ui gsj /; if j < i . Further, letting xj D .xj1 ; : : : ; xj;j 1 ; xjj ; xj C1;j ; : : : ; xmj /0, @u D uj HC gj u0 C uj gj ej0 : @xj0

(15.13)

21.16 An Equivalence (Involving Determinants and Polynomials) The following theorem is of considerable importance in statistics (where it is used in establishing a fundamental result on the statistical independence of quadratic forms)—refer, for example, to Driscoll and Gundberg (1986) and Reid and Driscoll (1988).

21.16 An Equivalence (Involving Determinants and Polynomials)

575

Theorem 21.16.1. Let A and B represent n n symmetric matrices, and let c and d represent (strictly) positive scalars. Then, a necessary and sufficient condition for jI t A uBj D jI t AjjI uBj (16.1) for all (scalars) t and u satisfying jt j < c and juj < d (or, equivalently, for all t and u) is that AB D 0. Proof. The sufficiency of the condition AB D 0 is evident upon observing that jI t AjjI uBj D j.I t A/.I uB/j D jI t A uB C t uABj: Now, for purposes of establishing the necessity of this condition, suppose that equality (16.1) holds for all t and u satisfying jt j < c and juj < d . And let c ( c) represent a positive scalar such that I t A is positive definite whenever jt j < c —the existence of such a scalar is guaranteed by Corollary 18.3.2—and (for t satisfying jt j < c ) let H D .I t A/1: If jt j < c , then jI uBj D jI uBHj and jI .u/Bj D jI .u/BHj, implying that (16.2) jI u2 B2 j D jI u2 .BH/2 j: Since each side of equality (16.2) is a polynomial in u2 , we have that jI rB2 j D jI r.BH/2 j for all r (and for t such that jt j < c )—refer to Theorem 21.A.5 (in the Appendix). Applying result (15.8.5), we find that (if jt j < c ) ˇ ˇ @jI rB2 j ˇˇ @jI r.BH/2 j ˇˇ D D trŒ.BH/2 : tr.B2 / D ˇ ˇ @r @r rD0 rD0 Thus,

@2 trŒ.BH/2 @2 tr.B2 / D D0 (16.3) 2 @t @t 2 (for t such that jt j < c ). And, using results (15.8.15), (15.9.2), (15.6.1), (15.4.8), and (5.2.3), we find that @H @2 H D HAH; D 2HAHAH; @t @t 2 @ trŒ.BH/2 @.BH/2 @H @H @H D tr D tr B BH C BHB D 2 tr BHB ; @t @t @t @t @t and

576


@2 trŒ.BH/2 @H @H @2 H D 2 tr B B C BHB 2 @t 2 @t @t @t D 2 trŒBHAHBHAH C 2BHBHAHAH:

(16.4)

Combining results (16.4) and (16.3) and setting t D 0 gives 0 D trŒ.BA/2 C 2 tr.B2A2 / D trŒ.BA/2 C B2A2 C tr.B2A2 / D trŒB.AB C BA/A C tr.B2A2 / D .1=2/ trŒB.AB C BA/A C .1=2/ trfŒB.AB C BA/A0 g C tr.BA2 B/ D .1=2/ trŒ.AB C BA/AB C .1=2/ trŒA.AB C BA/0 B C tr.BA2 B/ D .1=2/ trŒ.AB C BA/0 AB C .1=2/ trŒ.AB C BA/0 BA C tr.BA2 B/ D .1=2/ trŒ.AB C BA/0 .AB C BA/ C trŒ.AB/0 AB:

(16.5)

Both terms of expression (16.5) are nonnegative and hence equal to 0. Moreover, Q.E.D. trŒ.AB/0 AB D 0 implies that AB D 0—refer to Lemma 5.3.1. In light of Theorem 18.3.1, we have the following variation on Theorem 21.16.1. Corollary 21.16.2. Let A and B represent n n symmetric matrices. Then, there exist (strictly) positive scalars c and d such that I t A, I uB, and I t A uB are positive definite for all (scalars) t and u satisfying jt j < c and juj < d . And a necessary and sufficient condition for jI t A uBj D0 log jI t AjjI uBj for all t and u satisfying jt j < c and juj < d is that AB D 0. The following theorem (which, like Theorem 21.16.1, can be useful in establishing results on the statistical independence of quadratic forms) generalizes Corollary 21.16.2. Theorem 21.16.3. Let A and B represent n n symmetric matrices. Then, there exist (strictly) positive scalars c and d such that I t A, I uB, and I t A uB are positive definite for all (scalars) t and u satisfying jt j < c and juj < d . And, letting h.t; u/ represent a polynomial (in t and u), necessary and sufficient conditions for jI t A uBj h.t; u/ D (16.6) log jI t AjjI uBj jI t AjjI uBjjI t A uBj for all t and u satisfying jt j < c and juj < d are that AB D 0 and that, for all t and u satisfying jt j < c and juj < d (or, equivalently, for all t and u), h.t; u/ D 0. As a preliminary to proving Theorem 21.16.3, it is convenient to establish the following result on polynomials.


577

Theorem 21.16.4. Let r1 .x/, s1 .x/, and s2 .x/ represent polynomials in a real variable x. And let r2 .x/ D .x 1 /m1 .x 2 /m2 .x k /mk; where k is a nonnegative integer; m1 ; m2 ; : : : ; mk are (strictly) positive integers;

is a nonzero real number; and 1 ; 2 ; : : : ; k are real numbers. [When k D 0, r2 .x/ D .] Suppose that r1 .x/ s1 .x/ D (16.7) log s2 .x/ r2 .x/ for all x in some nondegenerate interval I [that does not include 1 ; 2 ; : : : ; k or any roots of s2 .x/]. Then, there exists a real number ˛ such that r1 .x/ D ˛r2 .x/ for all x. Further, s1 .x/ D e ˛ s2 .x/ for all x. Proof (of Theorem 21.16.4). The proof makes considerable use of various basic properties of polynomials—refer to the Appendix for a review of the relevant properties. Assume (without loss of generality) that 1 ; 2 ; : : : ; k are distinct and that 1 ; 2 ; : : : ; k are not roots of r1 .x/. [To see that the latter assumption can be made without loss of generality, suppose that 1 were a root of r1 .x/. Then, in light of Theorem 21.A.3, there would exist a polynomial p1 .x/ such that r1 .x/ D .x 1 /p1 .x/, so that, for x 2 I , r1 .x/=r2 .x/ D p1 .x/=p2 .x/, with p2 .x/ D .x 1 /m1 1 .x 2 /m2 .x k /mk: Moreover, to show that r1 .x/ D ˛r2 .x/ for some ˛, it would suffice to show that p1 .x/ D ˛p2 .x/ for some ˛. This process of canceling common factors could be continued until the right side of equality (16.7) were expressed as a ratio of two polynomials that have no common roots.] Assume also (again without loss of generality) that i is not a root of both s1 .x/ and s2 .x/ (i D 1; 2; : : : ; k). Making frequent use of an abbreviated notation for polynomials (discussed in the Appendix) and letting r1 .x/, r2 .x/, s1 .x/, and s2 .x/ represent the derivatives of r1 .x/, r2 .x/, s1 .x/, and s2 .x/, respectively, we find [upon differentiating both sides of equality (16.7)] that r1 r2 r1 r2 s1 s2 s1 s2 D s1 s2 r22 for x 2 I . And, in light of Theorem 21.A.5, it follows that .r1 r2 r1 r2 /s1 s2 D r22 .s1 s2 s1 s2 /

(16.8)

for all x. We now come to the heart of the proof, which consists of showing that equality (16.8) implies that k D 0 (i.e., that r2 ). Letting i represent an arbitrary one of the first m positive integers, we can rewrite r2 .x/ as r2 .x/ D .x i /mi t .x/;

578


Q where t .x/ D j ¤i .x j /mj . Then, letting t .x/ represent the derivative of t .x/, we can rewrite equality (16.8) as Œ.x i /.r1 t r1 t / mi r1 t .x i /mi 1 s1 s2 D .x i /2mi t 2 .s1 s2 s1 s2 /; so that (for all x) Œ.x i /.r1 t r1 t / mi r1 t s1 s2 D .x i /mi C1 t 2 .s1 s2 s1 s2 /:

(16.9)

Neither r1 nor t has i as a root, implying that r1 t does not have i as a root and hence that .x i /.r1 t r1 t / mi r1 t does not have i as a root. Thus, in light of Theorem 21.A.4, s1 s2 D .x i /mi C1 b for some polynomial b.x/. By assumption, i is not a root of both s1 and s2 , so that .x i /mi C1 is a factor of either s1 or s2 . Suppose that it is s1 that has i as a root, so that s1 .x/ D .x i /mi C1 d.x/ for some polynomial d.x/. Then, letting d .x/ represent the derivative of d.x/, the right side of equality (16.9) can be rewritten as the polynomial .x i /mi C1 t 2 fŒ.mi C 1/.x i /mi d C .x i /mi C1 d s2 .x i /mi C1 ds2 g; which has .x i /2mi C1 as a factor. It follows that .x i /2mi C1 is a factor of s1 s2 and hence of s1 . This argument can be repeated to establish that .x i /3mi C1 is a factor of s1 . Further repetition reveals that, for an arbitrarily large integer n, .x i /nmi C1 is a factor of s1 . Alternatively, if it is s2 (rather than s1 ) that has i as a root, then we find that, for arbitrarily large n, .x i /nmi C1 is a factor of s2 . Thus, k D 0, that is, r2 (since otherwise we arrive at a contradiction of the fact that s1 and s2 are polynomials of fixed degree). And, upon substituting

for r2 in equality (16.8), we find that (for all x)

r1 s1 s2 D 2 .s1 s2 s1 s2 /:

(16.10)

Now, if r1 were not a constant and if (for x 2 I ) s1 =s2 were of constant value, then (for x 2 I ) the left side of equality (16.7) would be of constant value but the right side would not be of constant value. If r1 were not a constant and if (for x 2 I ) s1 =s2 were not of constant value (in which case s1 or s2 is not a constant), then the degree of the polynomial that forms the left side of equality (16.10) would exceed the degree of the polynomial that forms the right side. In either case, we arrive at a contradiction. Thus, r1 is a constant; that is, r1 .x/ D for some real


579

number , and hence r1 . = /r2 . Further, s1 exp. = /s2 [since, for x 2 I , Q.E.D. s1 =s2 D exp.r1 =r2 / D exp. = /]. Proof (of Theorem 21.16.3). The sufficiency of these conditions is an immediate consequence of Corollary 21.16.2 [as is the existence of positive scalars c and d such that I t A, I uB, and I t A uB are positive definite for all t and u satisfying jt j < c and juj < d ]. For purposes of establishing their necessity, take u to be an arbitrary scalar satisfying juj < d , and observe (in light of Corollaries 14.2.11 and 14.3.13) that there exists an n n nonsingular matrix S such that .I uB/1 D S0 S. And let d1 ; : : : ; dr represent the nonzero (not necessarily distinct) eigenvalues of A and f1 ; : : : ; fr the nonzero (not necessarily distinct) eigenvalues of SAS0 . Then, according to Corollary 21.5.9, there exist nn orthogonal matrices P and Q such that P 0 AP D diag.d1 ; : : : ; dr ; 0; : : : ; 0/ and Q0 SAS0 Q D diag.f1 ; : : : ; fr ; 0; : : : ; 0/. Thus, in light of result (13.3.9), we have that jI t Aj D jP .I t P 0 AP /P 0 j D jP jjdiag.1 t d1 ; : : : ; 1 t dr ; 1; : : : ; 1/jjP 0 j D D

r Y iD1 r Y

.1 t di / .di /.t di1 /

iD1

and that jI t A uBj D jS1 .S0 /1 t Aj D jS1 .I t SAS0 /.S0 /1 j D jSj2 jI t SAS0 j D jSj2 jQ.I t Q0 SAS0 Q/Q0 j D jSj2 jQjjdiag.1 tf1 ; : : : ; 1 tfr ; 1; : : : ; 1/jjQ0 j D jSj2

r Y

.1 tfi / D jSj2

iD1

r Y

.fi /.t fi1 /;

iD1

so that (for fixed u) jI t A uBj and jI t AjjI uBj are polynomials in t . Further, jI t AjjI uBjjI t A uBj D jI uBjjSj2

r Y

di fi .t di1 /.t fi1 /;

(16.11)

iD1

which (for fixed u) is a polynomial in t of degree 2r with roots d11 ; : : : ; dr1 , f11 , : : : ; fr1 . Now, regarding u as fixed, suppose that equality (16.6) holds for all t satisfying jt j < c. Then, in light of equality (16.11), it follows from Theorem 21.16.4 that there exists a real number ˛ such that, for all t ,

580


h.t; u/ D ˛jI t AjjI uBjjI t A uBj and

jI t A uBj D e ˛ jI t AjjI uBj:

(16.12)

[In applying Theorem 21.16.4, take x D t , s1 .t / D jI t A uBj, s2 .t / D jI t AjjI uBj, r1 .t / D h.t; u/, and r2 .t / D jI t AjjI uBjjI t A uBj.] Moreover, upon setting t D 0 in equality (16.12), we find that jI uBj D e ˛ jI uBj; implying that e ˛ D 1 or, equivalently, that ˛ D 0. Thus, for all t , h.t; u/ D 0 and jI t A uBj D jI t AjjI uBj. We conclude that if equality (16.6) holds for all t and u satisfying jt j < c and juj < d , then h.t; u/ D 0 and jI t A uBj D jI t AjjI uBj for all t and u satisfying juj < d , implying (in light of Theorem 21.A.5) that h.t; u/ D 0 for all t and u and (in light of Theorem 21.16.1) that AB D 0. Q.E.D.

Appendix: Some Properties of Polynomials (in a Single Variable) An expression of the form p.x/ D a0 C a1 x C a2 x 2 C C an x n; where x is a real variable, n is a nonnegative integer, and the coefficients a0 , a1 , a2 ; : : : ; an are real numbers, is referred to as a polynomial in x. The polynomial p.x/ is said to be nonzero if one or more of the coefficients a0 ; a1 ; a2 ; : : : ; an is nonzero, in which case the largest nonnegative integer k such that ak ¤ 0 is referred to as the degree of p.x/ and is denoted by the symbol degŒp.x/. When it causes no confusion, p.x/ may be abbreviated to p {and degŒp.x/ to deg.p/}. A polynomial q is said to be a factor of a polynomial p if there exists a polynomial r such that p qr. And a real number c is said to be a root (or a zero) of a polynomial p if p.c/ D 0. The following three theorems give some basic properties of polynomials— refer, for example, to Beaumont and Pierce (1963) for proofs of these theorems, which are equivalent to their Theorems 9-3.3, 9-3.5, and 9-7.5. Theorem 21.A.1 (the division algorithm). Let p and q represent polynomials. Suppose that q is nonzero. Then, there exist unique polynomials b and r such that p bq C r; where either r 0 or else deg.r/ < deg.q/. Theorem 21.A.2. Let p.x/ represent a nonzero polynomial (in x) of degree n. Then, for any real number c, p.x/ has a unique representation of the form p.x/ D b0 C b1 .x c/ C b2 .x c/2 C C bn .x c/n; where b0 ; b1 ; b2 ; : : : ; bn are real numbers.

Appendix

581

Theorem 21.A.3. A real number c is a root of a polynomial p.x/ (in x) if and only if the polynomial x c is a factor of p.x/. The following two theorems give some additional properties of polynomials. Theorem 21.A.4. Let p.x/, q.x/, and r.x/ represent polynomials (in x). And suppose that p.x/q.x/ D .x c/m r.x/; (A.1) where m is a positive integer and where c is a real number that is not a root of p.x/. Then, .x c/m is a factor of q.x/; that is, there exists a polynomial s.x/ such that q.x/ D .x c/m s.x/: Proof. The proof consists of using mathematical induction to show that, for k D 1, : : : ; m, there exists a polynomial sk .x/ such that q.x/ D .x c/k sk .x/. Clearly, c is a root of q.x/. Thus, it follows from Theorem 21.A.3 that there exists a polynomial s1 .x/ such that q.x/ D .x c/1 s1 .x/. Suppose now that there exists a polynomial sk1 .x/ (where 2 k m) such that (A.2) q.x/ D .x c/k1 sk1 .x/: Then, substituting expression (A.2) into expression (A.1), we find that .x c/k1 p.x/sk1 .x/ D .x c/m r.x/; implying that

(A.3)

p.x/sk1 .x/ D .x c/mkC1 r.x/

for x ¤ c and hence that p.c/sk1 .c/ D lim p.x/sk1 .x/ D lim .x c/mkC1 r.x/ x!c

x!c

D .c c/mkC1 r.c/ D 0:

(A.4)

Equality (A.4) implies that c is a root of sk1 .x/. Thus, there exists a polynomial sk .x/ such that (A.5) sk1 .x/ D .x c/sk .x/: And, substituting expression (A.5) into expression (A.2), we find that q.x/ D .x c/k sk .x/; thereby completing the induction argument.

Q.E.D.

Theorem 21.A.5. Let p.x/ and q.x/ represent polynomials (in x). And suppose that p.x/ D q.x/ for all x in some nondegenerate interval I . Then, p.x/ D q.x/ for all x.

582


Proof. Let c represent an interior point of I, and observe (in light of Theorem 21.A.2) that p.x/ and q.x/ are expressible as p.x/ D a0 C a1 .x c/ C a2 .x c/2 C C an .x c/n; q.x/ D b0 C b1 .x c/ C b2 .x c/2 C C bn .x c/n; respectively, where n is a nonnegative integer [with n deg.p/ and n deg.q/] and a0 ; a1 ; a2 ; : : : ; an and b0 ; b1 ; b2 ; : : : ; bn are real numbers. And we find that, for k D 1; : : : ; n, n X @k p.x/ D m.m 1/ .m k C 1/ am .x c/mk; @x k k

@ q.x/ D @x k

mDk n X

m.m 1/ .m k C 1/ bm .x c/mk:

mDk

Thus, b0 D q.c/ D p.c/ D a0 ; and, for k D 1; : : : ; n, ˇ @k q.x/ ˇˇ kŠ bk D ˇ @x k ˇ

xDc

ˇ @k p.x/ ˇˇ D ˇ @x k ˇ

D kŠ ak :

xDc

We conclude that bk D ak for k D 0; 1; : : : ; n and that p.x/ D q.x/ for all x. Q.E.D.

Exercises Section 21.1 1. Show that an n n skew-symmetric matrix A has no nonzero eigenvalues. 2. Let A represent an n n matrix, B a k k matrix, and X an n k matrix such that AX D XB. (a) Show that C.X/ is an invariant subspace (of Rn1 ) relative to A. (b) Show that if X is of full column rank, then every eigenvalue of B is an eigenvalue of A. 3. Let p./ represent the characteristic polynomial of an n n matrix A, and let c0 ; c1 ; c2 ; : : : ; cn represent the respective coefficients of the characteristic polynomial, so that

Exercises

p./ D c0 0 C c1 C c2 2 C C cn n D

n X

583

c s s

sD0

(for 2 R). Further, let P represent the n n matrix obtained from p./ by formally replacing the scalar with the n n matrix A (and by setting A0 D In ). That is, let P D c0 I C c1 A C c2 A2 C C cn An D

n X

cs As:

sD0

Show that P D 0 (a result that is known as the Cayley-Hamilton theorem) by carrying out the following four steps. (a) Letting B./ D A I and letting H./ represent the adjoint matrix of B./, show that (for 2 R) H./ D K0 C K1 C 2 K2 C C n1 Kn1 ; where K0 ; K1 ; K2 ; : : : ; Kn1 are n n matrices (that do not vary with ). (b) Letting T0 D AK0 , Tn D Kn1 , and (for s D 1; : : : ; n 1) Ts D AKs Ks1 , show that (for 2 R) T0 C T1 C 2 T2 C C n Tn D p./In : [Hint. It follows from Theorem 13.5.3 that (for 2 R) B./H./ D jB./jIn D p./In .] (c) Show that, for s D 0; 1; : : : ; n, Ts D cs I. (d) Show that P D T0 C AT1 C A2 T2 C C An Tn D 0: 4. Let c0 ; c1 ; : : : ; cn1 ; cn represent the respective coefficients of the characteristic polynomial p./ of an n n matrix A [so that p./ D c0 C c1 C C cn1 n1 C cn n (for 2 R)]. Using the result of Exercise 3 (the Cayley-Hamilton theorem), show that if A is nonsingular, then c0 ¤ 0, and A1 D .1=c0 /.c1 I C c2 A C C cn An1 /: Section 21.3 5. Show that if an n n matrix B is similar to an n n matrix A, then (1) Bk is similar to Ak (k D 2; 3; : : :) and (2) B0 is similar to A0 . 6. Show that if an n n matrix B is similar to an (n n) idempotent matrix, then B is idempotent.

584


1 0 1 1 and B D . Show that B has the same rank, deter0 1 0 1 minant, trace, and characteristic polynomial as A, but that, nevertheless, B is not similar to A.

7. Let A D

8. Expand on the result of Exercise 7 by showing (for an arbitrary positive integer n) that, for an n n matrix B to be similar to an n n matrix A, it is not sufficient for B to have the same rank, determinant, trace, and characteristic polynomial as A. 9. Let A represent an n n matrix, B a k k matrix, and X an n k matrix such that AX D XB. Show that if X is of fullcolumn rank, then there exists an T T 11 12 orthogonal matrix Q such that Q0 AQ D , where T11 is a k k 0 T22 matrix that is similar to B. 10. Show that if 0 is an eigenvalue of an n n matrix A, then its algebraic multiplicity is greater than or equal to n rank.A/. 11. Let A represent an n n matrix. Show that if a scalar is an eigenvalue of A of algebraic multiplicity , then rank.A I/ n . 12. Let 1 represent the algebraic multiplicity and 1 the geometric multiplicity of 0 when 0 is regarded as an eigenvalue of an n n (singular) matrix A. And let 2 represent the algebraic multiplicity and 2 the geometric multiplicity of 0 when 0 is regarded as an eigenvalue of A2. Show that if 1 D 1 , then 2 D 2 D 1 . Section 21.4 13. Let x1 and x2 represent eigenvectors of an n n matrix A, and let c1 and c2 represent nonzero scalars. Under what circumstances is the vector x D c1 x1 C c2 x2 an eigenvector of A? Section 21.5 14. Let A represent an n n matrix, and suppose that there exists an n n nonsingular matrix Q such that Q1AQ D D for some diagonal matrix D D fdi g. Further, for i D 1; : : : ; n, let ri0 represent the i th row of Q1. Show (a) that A0 is diagonalized by .Q1 /0, (b) that the diagonal elements of D are the (not necessarily distinct) eigenvalues of A0, and (c) that r1 ; : : : ; rn are eigenvectors of A0 (with ri corresponding to the eigenvalue di ). 15. Show that if an n n nonsingular matrix A is diagonalized by an n n nonsingular matrix Q, then A1 is also diagonalized by Q.

Exercises

585

16. Let A represent an n n matrix whose spectrum comprises k eigenvalues 1 ; : : : ; k with algebraic multiplicities 1 ; : : : ; k , respectively, that sum to n. Show that A is diagonalizable if and only if, for i D 1; : : : ; k, rank.Ai I/ D n i . 17. Let A represent an nn symmetric matrix with not-necessarily-distinct eigenvalues d1 ; : : : ; dn that have been ordered so that d1 d2 dn . And let Q represent an nn orthogonal matrix such that Q0 AQ D diag.d1 ; : : : ; dn /— the existence of which is guaranteed by Corollary 21.5.9. Further, for m D 2; : : : ; n 1, define Sm D fx 2 Rn1 W x ¤ 0; Q0m x D 0g and Tm D 0 x D 0g, where Qm D .q1 ; : : : ; qm1 / and fx 2 Rn1 W x ¤ 0; Pm Pm D .qmC1 ; : : : ; qn /. Show that, for m D 2; : : : ; n 1, dm D min

x2Sm

x0 Ax x0 Ax D max : x2Tm x0 x x0 x

18. Let A represent an n n symmetric matrix, and adopt the same notation as in Section 21.5f. (a) Show that the matrices E1 ; : : : ; Ek , which appear in the spectral decomposition (5.5) of A, have the following properties: (1) E1 C C Ek D I; (2) E1 ; : : : ; Ek are nonnull, symmetric, and idempotent; (3) for t ¤ j D 1; : : : ; k, E t Ej D 0; and (4) rank.E1 / C C rank.Ek / D n. (b) Take F1 ; : : : ; Fr to be n n nonnull idempotent matrices such that F1 C C Fr D I. And suppose that, for some distinct scalars 1 ; : : : ; r , A D 1 F1 C C r Fr : Show that r D k and that there exists a permutation t1 ; : : : ; tr of the first r positive integers such that (for j D 1; : : : ; r) j D tj and Fj D E tj . 19. Let A represent an n n symmetric matrix, and let d1 ; : : : ; dn represent the (not necessarily distinct) eigenvalues of A. Show that limk!1 Ak D 0 if and only if, for i D 1; : : : ; n, jdi j < 1. Section 21.7 20. (a) Show that if 0 is an eigenvalue of an n n not-necessarily-symmetric matrix A, then it is also an eigenvalue of AC and that the geometric multiplicity of 0 is the same when it is regarded as an eigenvalue of AC as when it is regarded as an eigenvalue of A. (b) Show (via an example) that the reciprocals of the nonzero eigenvalues of a square nonsymmetric matrix A are not necessarily eigenvalues of AC.

586


Section 21.8 21. Show that, for any positive integer n that is divisible by 2, there exists an n n orthogonal matrix that has no eigenvalues. [Hint. Find a 2 2 orthogonal matrix Q that has no eigenvalues, and then consider the block-diagonal matrix diag.Q; Q; : : : ; Q/.] 22. Let Q represent an n n orthogonal matrix, and let p./ represent the characteristic polynomial of Q. Show that (for ¤ 0) p./ D ˙ np.1=/: 23. Let A represent an nn matrix, and suppose that the scalar 1 is an eigenvalue of A of geometric multiplicity . Show that rank.A/ and that if D rank.A/, then A is idempotent. Section 21.10 24. Let A represent an nn nonsingular matrix. And let represent an eigenvalue of A, and x represent an eigenvector of A corresponding to . Show that jAj= is an eigenvalue of adj.A/ and that x is an eigenvector of adj.A/ corresponding to jAj=. 25. Let A represent an n n matrix, and let p./ represent the characteristic polynomial of A. And let 1 ; : : : ; k represent the distinct eigenvalues of A, and 1 ; : : : ; k represent their respective algebraic multiplicities, so that (for all ) k Y . j /j p./ D .1/n q./ j D1

P for some polynomial q./ (of degree n jkD1 j ) that has no real roots. Further, define B D A 1 U V 0, where U D .u1 ; : : : ; u1 / is an n 1 matrix whose columns u1 ; : : : ; u1 are (not necessarily linearly independent) eigenvectors of A corresponding to 1 and where V D .v1 ; : : : ; v1 / is an n 1 matrix such that V 0 U is diagonal. (a) Show that the characteristic polynomial, say r./, of B is such that (for all ) n

r./ D .1/ q./

1 Y iD1

Œ .1

vi0 ui /1

k Y

. j /j:

(E.1)

j D2

[Hint. Since the left and right sides of equality (E.1) are polynomials (in ), it suffices to show that they are equal for all other than 1 ; : : : ; k .]

Exercises

587

(b) Show that in the special case where U 0 V D c I for some nonzero scalar c, the distinct eigenvalues of B are either 2 ; : : : ; s1 ; s ; sC1 , : : : ; k with algebraic multiplicities 2 ; : : : ; s1 ; s C 1 ; sC1 , : : : ; k , respectively, or .1 c/1 ; 2 ; : : : ; k with algebraic multiplicities 1 ; 2 , : : : ;

k , respectively [depending on whether or not .1 c/1 D s for some s (2 s k)]. (c) Show that, in the special case where 1 D 1, (1) u1 is an eigenvector of B corresponding to the eigenvalue .1 v10 u1 /1 and (2) for any eigenvector x of B corresponding to an eigenvalue [other than .1 v10 u1 /1 ], the vector x 1 .1 /1 .v10 x/u1 is an eigenvector of A corresponding to . Section 21.12 26. Let A represent an m n matrix of rank r. And take P to be any m m orthogonal matrix and D1 to be any r r nonsingular diagonal matrix such that 2 D1 0 0 0 P AA P D : 0 0 Further, partition P as P D .P1 ; P2 /, where P1 has r columns, and let Q D .Q1 ; Q2 /, where Q1 D A0 P1 D11 and where Q2 is any n .n r/ matrix such that Q01 Q2 D 0. Show that D1 0 0 P AQ D : 0 0 27. Show that the matrices U1 ; : : : ; Uk , which appear in decomposition (12.7), are such that Uj Uj0 Uj D Uj (for j D 1; : : : ; k) and U t0 Uj D 0 and U t Uj0 D 0 (for t ¤ j D 1; : : : ; k). 28. Let A represent an m n matrix. And, as in Theorem 21.12.3, take P to be an m m orthogonal matrix, Q an n n orthogonal matrix, and D1 an r r nonsingular diagonal matrix such that D1 0 P 0 AQ D : 0 0 Further, partition P and Q as P D .P1 ; P2 / and Q D .Q1 ; Q2 /, where each of the matrices P1 and Q1 has r columns. Show that C.A/ D C.P1 / and that N.A/ D C.Q2 /. Section 21.13

588


29. Let A1 ; : : : ; Ak represent n n not-necessarily-symmetric matrices, each of which is diagonalizable. Show that if A1 ; : : : ; Ak commute in pairs, then A1 ; : : : ; Ak are simultaneously diagonalizable. 30. Let V represent an n n symmetric nonnegative definite matrix, X an n p matrix of rank r, and d a p 1 vector. Using the results of Exercise 19.11 (or otherwise), show that each of the following three conditions is necessary and sufficient for the vector X.X0 X/ d to be a solution, for every d 2 C.X0 /, to the problem of minimizing the quadratic form a0 V a (in a) subject to X0 a D d: (a) there exists an orthogonal matrix that simultaneously diagonalizes V and PX ; (b) there exists a subset of r orthonormal eigenvectors of V that is a basis for C.X/; (c) there exists a subset of r eigenvectors of V that is a basis for C.X/. Section 21.14 31. Let A represent an n n symmetric matrix, and let B represent an n n symmetric positive definite matrix. And let max and min represent, respectively, the largest and smallest roots of jA Bj. Show that min

x0 Ax max x0 Bx

for every nonnull vector x in Rn. 32. Let A represent an n n symmetric matrix, and let B represent an n n symmetric positive definite matrix. Show that A B is nonnegative definite if and only if all n (not necessarily distinct) roots of jA Bj are greater than or equal to 1 and is positive definite if and only if all n roots are (strictly) greater than 1.

Bibliographic and Supplementary Notes §15. The presentation in Section 15 was heavily influenced by that of Magnus and Neudecker (1988, sec. 8.8). For a more extensive discussion of the differentiation of eigenvalues and eigenvectors, refer, for example, to Sections 8.7–8.12 of Magnus and Neudecker (and to the sources cited by them). Their coverage includes some results on the second-order differentiation of eigenvalues, on the differentiation of multiple eigenvalues, and on the differentiation of eigenvalues and eigenvectors of nonsymmetric matrices. §16. The proof of the result on polynomials given by Theorem 21.16.4 is that of Harville and Kempthorne (1997).

22 Linear Transformations

In more advanced presentations of the topics of Chapters 1–21, matrices typically play a subordinate role. In such presentations, the main results are in terms of something called a linear transformation, which is regarded as a more fundamental concept than that of a matrix. The emphasis on linear transformations results in an approach that is more abstract, more “elegant,” and more conducive to geometrical interpretation. It results in more generality in one sense, although not in another sense. It tends to appeal to the “mathematically more sophisticated,” who typically prefer “geometrical reasoning” to “algebraic arguments.” In the first part of the present (and final) chapter of the book, the concept of a linear transformation is formally introduced, and a number of results on linear transformations are derived from first principles (i.e., “independently” of matrices). And, in the latter parts of the chapter, it is shown that linear transformations can be put into 1–1 correspondence with matrices, and then, based on this correspondence, various results and terminology pertaining to matrices are extended to linear transformations.

22.1 Some Definitions, Terminology, and Basic Results A transformation (also known as a function, operator, map, or mapping) T from a set V (consisting of scalars, row or column vectors, matrices, or other “objects”) into a set W is a correspondence that assigns to each member X of V a unique member of W. This unique member of W is denoted by the symbol T .X / and is referred to as the image of X . The use of the term “into” (in the definition of T ) alludes to the possibility that some members of W may not appear as the image of any member of V. The set V is called the domain of T . And the set fY 2 W W Y D T .X / for some X 2 Vg, consisting of every member of W that appears as the image of one or more members of V, is called the range of T . In the event that the range of T consists of all of W, T is said to be onto and may be referred to as a transformation

590


from V onto W. If each member of the range of T appears as the image of only one member of V, T is referred to as a 1–1 (one to one) transformation. Corresponding to any m n matrix A is the transformation T from Rn1 into m1 that assigns to each n 1 vector x the m 1 vector Ax. Clearly, the range R of T equals C.A/. Further, T is onto if and only if A is of full row rank, that is, if and only if rank.A/ D m. And T is 1–1 if and only if the columns of A are linearly independent, that is, if and only if rank.A/ D n. In what follows, the emphasis is on linear transformations. Let T represent a transformation from a linear space V (of m n matrices) into a linear space W (of p q matrices). Then, T is said to be linear if it satisfies the following two conditions: (1) for all X and Z in V, T .X C Z/ D T .X/ C T .Z/; and (2) for every scalar c and for all X in V, T .cX/ D cT .X/. It is easy to verify that T is linear if and only if, for all scalars c and k and for all X and Z in V, T .cX C kZ/ D cT .X/ C kT .Z/: More generally, letting r represent a positive integer (greater than 1), T is linear if and only if, for all scalars c1 ; : : : ; cr and for all X1 ; : : : ; Xr in V, T

r X iD1

r X ci Xi D ci T .Xi /:

(1.1)

iD1

Clearly, for any m n matrix A, the transformation from the linear space Rn1 into the linear space Rm1 that assigns to each n 1 vector x the m 1 vector Ax is a linear transformation. More generally, for any r p matrix A and any q s matrix B, the transformation from the linear space Rpq into the linear space Rrs that assigns to each p q matrix X the r s matrix AXB is a linear transformation (as is easily verified). Let T represent a transformation from a set V into a set W. Then, corresponding to any subset U of V, is the subset fY 2 W W Y D T .X / for some X 2 Ug, consisting of every member of W that appears as the image of one or more members of U. This subset is denoted by the symbol T .U/ and is referred to as the image of the set U. Note that, by definition, the image T .V/ of the domain V of T is the range of T . Now, suppose that T is a linear transformation from a linear space V into a linear space W and that U is a subspace of V. Then, for any matrices X1 ; : : : ; Xr that span U, T .U/ D spŒT .X1 /; : : : ; T .Xr /; as is evident from result (1.1). Thus, T .U/ is a linear space; it is a subspace of W. In particular, the range T .V/ of T is a subspace of W and, accordingly, may be referred to as the range space of T . A linear transformation T from a linear space V of m n matrices into a linear space W of p q matrices has the property that T .0/ D T .0 0/ D 0T .0/ D 0.

22.1 Some Definitions, Terminology, and Basic Results

591

Thus, T assigns the (p q) null matrix in W to the (m n) null matrix in V. The linear transformation T may also assign the null matrix in W to various nonnull matrices in V. The set fX 2 V W T .X/ D 0g; (1.2) comprising all matrices in V to which T assigns the null matrix, is nonempty. Moreover, for any matrices X and Z in the set (1.2) and for any scalar c, T .X C Z/ D T .X/ C T .Z/ D 0 C 0 D 0 and T .cX/ D cT .X/ D c 0 D 0, so that X C Z and cX are in the set (1.2). Thus, the set (1.2) is a linear space; it is a subspace of V. This subspace is called the null space (or the kernel) of the linear transformation T and is denoted by the symbol N.T /. There is a simple and fundamental relationship between the range and null spaces of a linear transformation. This relationship is described in the following theorem. Theorem 22.1.1. Let T represent a linear transformation from a linear space V into a linear space W. Then, dim.V/ D dim.M/ C dim.N/;

(1.3)

where M and N are, respectively, the range and null spaces of T . Proof. Let k D dim.V/ and s D dim.N/. Further, let r D k s. And assume that r > 0—if r D 0, then (in light of Theorem 4.3.10) V D N, in which case M D f0g and equality (1.3) is clearly valid—and take fZ1 ; : : : ; Zs g to be a basis for N—if s D 0, interpret fZ1 ; : : : ; Zs g as the empty set. Then, it follows from Theorem 4.3.12 that there exist matrices X1 ; : : : ; Xr in V such that fX1 ; : : : ; Xr , Z1 ; : : : ; Zs g is a basis for V—if s D 0, interpret fX1 ; : : : ; Xr , Z1 ; : : : ; Zs g as fX1 ; : : : ; Xr g. Now, for i D 1; : : : ; r, let Yi D T .Xi / (and observe that Yi 2 M). To prove Theorem 22.1.1, it suffices to show that fY1 ; : : : ; Yr g is a basis for M. Let us begin by showing that this set spans M. Let Y represent an arbitrary matrix in M. Then, there exists a matrix X in V such that Y D T .X/. Further, if s > 0, there exist scalars a1 ; : : : ; ar , b1 , : : : ; bs such that r s X X ai Xi C bj Zj ; XD iD1

j D1

in which case Y D T .X/ D

r X iD1

ai T .Xi / C

s X j D1

bj T .Zj / D

r X

ai T .Xi / D

iD1

Alternatively, if s D 0, there exist scalars a1 ; : : : ; ar such that X D in which case r r X X ai T .Xi / D ai Yi : Y D T .X/ D iD1

iD1

r X

ai Yi :

iD1

Pr

iD1

ai Xi ,

592


Thus, whether s > 0 or s D 0, M sp.Y1 ; : : : ; Yr /, and, consequently, fY1 , : : : ; Yr g spans M. ; : : : ; Yr g is linearly independent. Let It remains to show that fY1 P Pc1 ; : : : ; cr represent any scalars such that riD1 ci Yi D 0. And define Z D riD1 ci Xi . Then, we have that T .Z/ D T

r X iD1

r r X X ci Xi D ci T .Xi / D ci Yi D 0; iD1

iD1

implying P that Z belongs to N. Thus, if s > 0, there exist scalars d1 ; : : : ; ds such that Z D js D1 dj Zj , in which case c1 X1 C C cr Xr d1 Z1 ds Zs D Z Z D 0; and (since fX1 ; : : : ; Xr ; Z1 ; : : : ; Zs g is a basis for V and hence is linearly independent) it follows that c1 D D cr D 0 (and that d1 D D ds D 0). Alternatively, if s D 0, then Z D 0, and (since fX1 ; : : : ; Xr g is linearly independent) it follows that c1 D D cr D 0. In either case, we conclude that Q.E.D. fY1 ; : : : ; Yr g is linearly independent (and hence is a basis for M). In connection with Theorem 22.1.1, equality (1.3) can be rewritten to give the following expressions for dim.M/ and dim.N/: dim.M/ D dim.V/ dim.N/;

(1.4)

dim.N/ D dim.V/ dim.M/:

(1.5)

Moreover, as an immediate consequence of Theorem 22.1.1, we have the following corollary. Corollary 22.1.2. For any linear transformation T from a linear space V into a linear space W, the dimension of the range space of T is less than or equal to dim.V/, that is, less than or equal to the dimension of the domain of T . One very basic transformation is the transformation from a linear space V into a linear space W that assigns to every matrix in V the null matrix (in W). This transformation is called the zero transformation and is denoted by the symbol 0. Clearly, the zero transformation is linear. Further, its range space is the set whose only member is the null matrix, and its null space equals its domain V. Another very basic transformation is the identity transformation. The identity transformation is the transformation from a linear space V into (the same linear space) V defined by T .X/ D X; it is denoted by the symbol I . The identity transformation is linear (as is readily apparent). Further, it is onto; that is, its range space equals V. And, clearly, it is 1–1, implying in particular that its null space is the set whose only member is the null matrix. Let T represent a linear transformation from a linear space V into a linear space W, and let U represent any subspace of V. Then, the transformation, say R, from U into W defined by R.X/ D T .X/ (which assigns to each matrix in U the same matrix in W assigned by T ) is called the restriction of T to U. Clearly,

22.1 Some Definitions, Terminology, and Basic Results

593

the restriction of T to U is a linear transformation. Further, its range space equals T .U/, and its null space equals U\N.T /. And if T is 1–1, then so is the restriction of T to U. The following four lemmas give some basic results on linear transformations. Lemma 22.1.3. Let T represent a linear transformation from a linear space V into a linear space W. Then, T is 1–1 if and only if N.T / D f0g [i.e., if and only if N.T / contains no nonnull matrices]. Proof. Suppose that T is 1–1. Then, since the image of the null matrix (in V) is the null matrix (in W), V cannot contain any other matrix (i.e., any nonnull matrix) whose image is the null matrix. Thus, N.T / contains no nonnull matrices. Conversely, suppose that N.T / contains no nonnull matrices. Then, for any matrices Z and X in V for which T .Z/ D T .X/, we have that T .Z X/ D T .Z/ T .X/ D 0, implying that Z X 2 N.T / and hence that Z X D 0 or, equivalently, Z D X. Thus, T is 1–1. Q.E.D. Lemma 22.1.4. Let T represent a linear transformation from a linear space V into a linear space W. And let X1 ; : : : ; Xk represent matrices in V. If X1 ; : : : ; Xk are linearly dependent, then their images T .X1 /; : : : ; T .Xk / are also linearly dependent. Proof. Suppose that X1 ; : : : ; Xk are linearly dependent. Then, there exist P scalars c1 ; : : : ; ck , one or more of which is nonzero, such that kiD1 ci Xi D 0. And k k X X ci T .Xi / D T ci Xi D T .0/ D 0: iD1

iD1

Thus, T .X1 /; : : : ; T .Xk / are linearly dependent.

Q.E.D.

Lemma 22.1.5. Let T represent a 1–1 linear transformation from a linear space V into a linear space W. And let X1 ; : : : ; Xk represent matrices in V. Then, T .X1 /; : : : ; T .Xk / are linearly independent if and only if X1 ; : : : ; Xk are linearly independent. Proof. It suffices to show that T .X1 /; : : : ; T .Xk / are linearly dependent if and only if X1 ; : : : ; Xk are linearly dependent. That T .X1 /; : : : ; T .Xk / are linearly dependent if X1 ; : : : ; Xk are linearly dependent is an immediate consequence of Lemma 22.1.4. Now, suppose that T .X1 /; : : : ; T .Xk / are linearly dependent. Then, there exist P scalars c1 ; : : : ; ck , one or more of which is nonzero, such that kiD1 ci T .Xi / D 0. Pk Pk Further, T . iD1 ci Xi / D iD1 ci T .Xi / D 0. And it follows from Lemma Pk 22.1.3 that iD1 ci Xi D 0. Thus, X1 ; : : : ; Xk are linearly dependent. Q.E.D. Lemma 22.1.6. Let T represent a linear transformation from V into W, where V and W are linear spaces of the same dimension. Then, T is 1–1 if and only if T .V/ D W (i.e., if and only if T is onto). Proof. According to Lemma 22.1.3, T is 1–1 if and only if N.T / D f0g or, equivalently, if and only if dimŒN.T / D 0. Moreover, according to Theorem 22.1.1,

594


dimŒN.T / D dim.V/ dimŒT .V/: Thus, T is 1–1 if and only if dim.V/ dimŒT .V/ D 0 or, equivalently, if and only if dimŒT .V/ D dim.V/. And, since dim.V/ D dim.W/, T is 1–1 if and only if dimŒT .V/ D dim.W/. It is now clear that if T .V/ D W, then T is 1–1. And, conversely, if T is 1–1, then [since T .V/ W] we have [in light of Theorem 4.3.10] that T .V/ D W. Q.E.D. As a special case of Lemma 22.1.6, we have the following corollary. Corollary 22.1.7. Let T represent a linear transformation from V into V. Then, T is 1–1 if and only if T .V/ D V (i.e., if and only if T is onto). The following lemma is sometimes useful. Lemma 22.1.8. Let V and W represent linear spaces. And, denoting dim.V/ by r (and supposing that r > 0), let fX1 ; : : : ; Xr g represent a basis for V, and let Y1 ; : : : ; Yr represent any r (not necessarily distinct) matrices in W. Then, the transformation T (from V into W) defined by T .X/ D

r X

ci Yi ;

iD1

P where c1 ; : : : ; cr are the (unique) scalars that satisfy X D riD1 ci Xi , is linear. And T .V/ D sp.Y1 ; : : : ; Yr /. Moreover, T is 1–1 if and only if Y1 ; : : : ; Yr are linearly independent. Proof. Letting k represent an arbitrary scalar, letting X and Z represent arbitrary matrices in V, and 1 ; : : : ; dr to be the (unique) Pd Prtaking c1 ; : : : ; cr and r c X and Z D di Xi , we find that kX D scalars that satisfy X D i i iD1 iD1 Pr Pr iD1 .kci /Xi and X C Z D iD1 .ci C di /Xi and hence that T .kX/ D

r X

.kci /Yi D k

iD1

r X

ci Yi D kT .X/

iD1

and T .X C Z/ D

r X iD1

.ci C di /Yi D

r X iD1

ci Yi C

r X

di Yi D T .X/ C T .Z/:

iD1

Thus, T is linear. To verify that T .V/ D sp.Y1 ; : : : ; Yr /, let Y represent any P matrix in sp.Y1 , : : : ; Yr /. Then, there exist scalars c1 ; : : : ; cr such that Y D riD1 ci Yi . Clearly, Pr Pr iD1 ci Xi 2 V and T . iD1 ci Xi / D Y , implying that Y 2 T .V/. Thus, sp.Y1 ; : : : ; Yr / T .V/, and [since obviously T .V/ sp.Y1 ; : : : ; Yr /] it follows that T .V/ D sp.Y1 ; : : : ; Yr /. To prove the final part of the lemma, observe P(in light of Lemma 22.1.3) that T is 1–1 if and only if N.T / D f0g and (since riD1 ci Xi D 0 , c1 D D cr D 0) that N.T / D f0g if and only if the only scalars c1 ; : : : ; cr for which

22.2 Scalar Multiples, Sums, and Products of Linear Transformations

595

Pr

iD1 ci Yi D 0 are c1 D D cr D 0. Thus, T is 1–1 if and only if Y1 ; : : : ; Yr are linearly independent. Q.E.D. Let S and T represent transformations from a set V into a set W. If S.X/ D T .X/ for every member X of V, S and T are said to be equal (or identical), and we may write S D T . The following lemma gives a necessary and sufficient condition for two linear transformations to be equal.

Lemma 22.1.9. Let S and T represent linear transformations from a linear space V into a linear space W. And let X1 ; : : : ; Xr represent any matrices (in V) that form a basis for V or, more generally, any matrices such that sp.X1 ; : : : ; Xr / D V. Then (supposing that r > 0), S D T if and only if, for i D 1; : : : ; r, S.Xi / D T .Xi /. Proof. That S.Xi / D T .Xi / (for i D 1; : : : ; r) if S D T is obvious. Now, suppose that, for i D 1; : : : ; r, S.Xi / D T .Xi /. And let X representPan arbitrary matrix in V. Then, there exist scalars c1 ; : : : ; cr such that X D riD1 ci Xi . Further, S.X/ D S

r X

r r r X X X ci Xi D ci S.Xi / D ci T .Xi / D T ci Xi D T .X/:

iD1

iD1

iD1

iD1

Thus, S D T .

Q.E.D.

22.2 Scalar Multiples, Sums, and Products of Linear Transformations a. Scalar multiples Let T represent a transformation from a linear space V into a linear space W, and let k represent an arbitrary scalar. Then, the transformation from V into W that assigns to each matrix X in V the matrix kT .X/ (in W) is referred to as a scalar multiple of T and is denoted by the symbol .kT / or (in the absence of ambiguity) kT . Thus, .kT /.X/ D kT .X/. If the transformation T is linear, then the scalar multiple kT is also linear, as is easily verified. Clearly, 1T D T; (2.1) and letting c (like k) represent a scalar, we find that c.kT / D .ck/T D .kc/T D k.cT /:

(2.2)

In the special case where k D 0 or T D 0, the scalar multiple kT reduces to the zero transformation (from V into W). That is, 0T D 0, and k 0 D 0. Further, it is customary to refer to the scalar multiple .1/T as the negative of T and to abbreviate .1/T to .T / or T .

596


b. Sums Let T and S represent transformations from a linear space V into a linear space W. Then, the transformation from V into W that assigns to each matrix X in V the matrix T .X/ C S.X/ (in W) is referred to as the sum of T and S and is denoted by the symbol .T C S / or T C S . Thus, .T C S /.X/ D T .X/ C S.X/. If the transformations T and S are both linear, then their sum T C S is also linear, as is easily verified. The addition of transformations is commutative, that is, T C S D S C T:

(2.3)

The addition of transformations is also associative, that is, taking R to be a third transformation from V into W, T C .S C R/ D .T C S / C R:

(2.4)

The symbol .T CS CR/ or T CS CR is used to represent the (common) transformation given by the left and right sides of equality (2.4), and this transformation is referred to as the sum of T , S , and R. This notation and terminology extend in an obvious way to any finite number of transformations from V into W. We find (via a straightforward exercise) that, for any scalar c, c.T C S / D .cT / C .cS /

(2.5)

and that, for any scalars c and k, .c C k/T D .cT / C .kT /:

(2.6)

Let us write .T S / or T S for the sum T C .S / or, equivalently, for the transformation (from V into W) that assigns to each matrix X in V the matrix T .X/ S.X/ (in W). And let us refer to this transformation as the difference between T and S . Clearly, T T D 0: (2.7) Further, T C 0 D 0 C T D T:

(2.8)

c. Products Let S represent a transformation from a linear space U into a linear space V, and let T represent a transformation from V into a linear space W. Consider the transformation from U into W that assigns to each matrix X in U the matrix T ŒS.X/ (in W) obtained by applying S to X and by then applying T to S.X/. This transformation is called the product (or the composition) of T and S and is denoted by the symbol .T S / or T S . Thus, .T S /.X/ D T ŒS.X/. If the transformations T and S are both linear, then their product T S is also linear, as is easily verified.

22.2 Scalar Multiples, Sums, and Products of Linear Transformations

597

Let R represent a transformation from a linear space M into a linear space U, S a transformation from U into a linear space V, and T a transformation from V into a linear space W. Then, for any matrix X in M, .T .SR//.X/ D T Œ.SR/.X/ D T fS ŒR.X/g D .T S/ŒR.X/ D ..T S /R/.X/: Thus, T .SR/ D .T S /R:

(2.9)

The symbol .T SR/ or T SR is used to represent the (common) transformation given by the left and right sides of equality (2.9), and this transformation is referred to as the product of T , S , and R—if the transformations R, S , and T are all linear, then clearly the transformation T SR is also linear. This terminology and notation extend in an obvious way to an arbitrary number of transformations. Now, let R and S represent linear transformations from a linear space U into a linear space V, and let T represent a linear transformation from V into a linear space W. Then, for any matrix X in U, .T .S C R//.X/ D T Œ.S C R/.X/ D T ŒS.X/ C R.X/ D T ŒS.X/ C T ŒR.X/ D .T S /.X/ C .TR/.X/: Thus, T .S C R/ D .T S / C .TR/:

(2.10)

[And the transformation T .S C R/ (like R, S , and T ) is linear.] Alternatively, suppose that R is a linear transformation from a linear space U into a linear space V and that S and T are linear transformations from V into a linear space W. Then, employing an argument similar to that used to establish equality (2.10), we find that .T C S /R D .TR/ C .SR/:

(2.11)

[And the transformation .T C S /R is linear.] Result (2.10) extends in an obvious way to the product of the linear transformation T (from V into W) and the sum of an arbitrary number of linear transformations (from U into V). Similarly, result (2.11) extends in an obvious way to the product of the sum of an arbitrary number of linear transformations (from V into W) and the linear transformation R (from U into V). For any transformation S from a linear space U into a linear space V and any transformation T from V into a linear space W, and for any scalar k, k.T S/ D .kT /S D T .kS /;

(2.12)

as is easily verified. The symbol kT S may be used to represent transformation (2.12). (Further, if the transformations S and T are linear, then so is the transformation kT S .) For any transformation T from a linear space V into a linear space W,

598


T I D I T D T; T 0 D 0; 0T D 0

(2.13) (2.14)

(where, depending on the context, I is the identity transformation from V onto V or from W onto W, and 0 is the zero transformation from a linear space U into V or W or from W or V into U). The following theorem gives a basic result on the product of two linear transformations. Theorem 22.2.1. Let S represent a linear transformation from a linear space U into a linear space V, and let T represent a linear transformation from V into a linear space W. Then, dim.MT S / D dim.MS / dim.MS \ NT /;

(2.15)

where MS and MT S are the range spaces of S and T S , respectively, and NT is the null space of T . Proof. Let R represent the restriction of T to MS . And denote the range space and the null space of R by MR and NR , respectively. According to Theorem 22.1.1, dim.MR / D dim.MS / dim.NR /: Moreover, it follows from the discussion (of restrictions) in Section 22.1 that MR D T .MS / D T ŒS.U/ D T S.U/ D MT S and NR D MS \ NT . Thus, dim.MT S / D dim.MS / dim.MS \ NT /: Q.E.D. As an immediate consequence of Theorem 22.2.1, we have the following corollary. Corollary 22.2.2. Let S represent a linear transformation from a linear space U into a linear space V, and let T represent a linear transformation from V into a linear space W. Then, dim.MT S / dim.MS /;

(2.16)

where MT S and MS are the range spaces of T S and S , respectively.

22.3 Inverse Transformations and Isomorphic Linear Spaces Let T represent a transformation from a linear space V into a linear space W. Suppose that T is 1–1 and onto. Then, T is said to be invertible. And there exists

22.3 Inverse Transformations and Isomorphic Linear Spaces

599

a (unique) transformation from W into V that assigns to each matrix Y in W the (unique) matrix X (in V) such that T .X/ D Y . This transformation is called the inverse of T and is denoted by the symbol T 1. Thus, T 1.Y / D X [where X is such that T .X/ D Y ]. The inverse transformation T 1 is 1–1 and onto, as can be easily verified. Moreover, if T is linear, then so is T 1. To see this, suppose that T is linear. Then, letting c represent an arbitrary scalar, letting Y and Z represent arbitrary matrices in W, and letting X and U represent the (unique) matrices (in V) such that T .X/ D Y and T .U / D Z, we find that T .cX/ D cT .X/ D cY and T .X C U / D T .X/ C T .U / D Y C Z, so that T 1.cY / D cX D cT 1.Y / and T 1.Y C Z/ D X C U D T 1.Y / C T 1.Z/: And it follows that T 1 is linear. For any matrix X in V, we have that .T 1T /.X/ D T 1ŒT .X/ D X D I.X/ (where the identity transformation I is that from V onto V). Thus, T 1T D I:

(3.1)

The inverse transformation T 1 is itself invertible (since it is 1–1 and onto). Moreover, upon applying result (3.1) (with T 1 in place of T ), we find that .T 1 /1 T 1 D I

(3.2)

(where the identity transformation I is that from W onto W). It follows that .T 1 /1 D .T 1 /1I D .T 1 /1 .T 1T / D ..T 1 /1 T 1 /T D I T D T: Thus, T is the inverse of T 1. Result (3.2) can be restated as T T 1 D I:

(3.3)

Let V and W represent linear spaces. If there exists a 1–1 linear transformation, say T, from V onto W, then V and W are said to be isomorphic, and T is said to be an isomorphism of V onto W. Note that the identity transformation I from a linear space V onto V is an isomorphism (from V onto V) and hence that any linear space is isomorphic to itself. Note also that I is invertible and that I 1 D I . The following theorem gives a necessary and sufficient condition for two linear spaces to be isomorphic.

600


Theorem 22.3.1. Two linear spaces V and W are isomorphic if and only if dim.V/ D dim.W/. Proof. Suppose that dim.V/ D dim.W/ D r for some integer r. And suppose that r > 0—if r D 0, we have that V D f0g and W D f0g, and these two linear spaces are clearly isomorphic. Then, there exist r linearly independent matrices Y1 ; : : : ; Yr in W (consisting of any r matrices that form a basis for W), and it follows from Lemma 22.1.8 that there exists a 1–1 linear transformation from V onto W. Thus, V and W are isomorphic. Conversely, suppose that V and W are isomorphic. Then, by definition, there exists an isomorphism, say T, of V onto W. And, since (according to Lemma 22.1.3) N.T / D f0g, it follows from Theorem 22.1.1 that dim.V/ D dimŒT .V/ C dimŒN.T / D dim.W/ C 0 D dim.W/: Q.E.D. As an immediate consequence of Theorem 22.3.1, we have the following corollary. Corollary 22.3.2. Every n-dimensional linear space is isomorphic to Rn. Let W represent an n-dimensional linear space, and let B represent a set of matrices Y1 ; : : : ; Yn (in W) that form a basis for W. Subsequently, the symbol LB is used to represent the transformation from Rn1 into W that assigns to each vector x D .x1 ; : : : ; xn /0 in Rn1 the matrix x1Y1 C CxnYn in W. Clearly, LB is 1–1 and onto (and hence invertible) and is linear, so that LB is an isomorphism of Rn onto W. 1 of LB is the (linear) transformation that assigns Note that the inverse LB to each matrix Y in W the column vector x in Rn whose elements (x1 ; : : : ; xn , respectively) are (uniquely) determined by the condition Y D x1Y1 C C xnYn , that is, whose elements are the coefficients of the basis matrices Y1 ; : : : ; Yn when Y is expressed as a linear combination of those matrices. For i D 1; : : : ; m and j D 1; : : : ; n, let Uij represent that m n matrix whose ij th element equals 1 and whose remaining mn 1 elements equal 0. Then, the set, say B, whose members are, respectively, the mn matrices U11 ; U21 ; : : : ; Um1 , : : : ; U1n ; U2n ; : : : ; Umn , is a basis for Rmn. It is easy to see that, for any m n matrix A, 1 .A/ D vec.A/: (3.4) LB Similarly, for i D 1; : : : ; n, let Uii represent the n n matrix whose i th diagonal element equals 1 and whose remaining n2 1 elements equal 0. And, for j < i D 1; : : : ; n, let Uij represent the n n matrix whose ij th and j i th elements equal 1 and whose remaining n2 2 elements equal 0. Then, the set, say B , ; U21 ; : : : ; Un1 , whose members are, respectively, the n.n C 1/=2 matrices U11 : : : ; Ui i ; UiC1;i ; : : : ; Uni , : : : ; Unn is a basis for the linear space of all n n symmetric matrices. Clearly, for any n n symmetric matrix A, 1 LB .A/ D vech.A/:

(3.5)

22.4 Matrix Representation of a Linear Transformation

601

22.4 Matrix Representation of a Linear Transformation a. Background, definition, and some basic properties Any linear transformation T from an n-dimensional linear space V into an mdimensional linear space W can be replaced by a linear transformation from Rn1 into Rm1. Let B represent a set of matrices V1 ; : : : ; Vn (in V) that form a basis for V, and let C represent a set of matrices W1 ; : : : ; Wm (in W) that form a basis for W. And, adopting the notation introduced in Section 22.3, consider the product 1 L1 C T LB of the three linear transformations LC , T, and LB . For any n 1 vector x and any m 1 vector y, .L1 C T LB /.x/ D y

,

T ŒLB .x/ D LC .y/;

(4.1)

as is evident upon observing that 1 1 .L1 C T LB /.x/ D .LC .T LB //.x/ D LC Œ.T LB /.x/

and T ŒLB .x/ D .T LB /.x/ and hence that result (4.1) can be reexpressed as L1 C Œ.T LB /.x/ D y

,

.T LB /.x/ D LC .y/:

And, for any matrix Z in V and any matrix Y in W, T .Z/ D Y

,

1 1 .L1 C T LB /ŒLB .Z/ D LC .Y /;

(4.2)

1 .Z/ and y D L1 as is evident from result (4.1) upon setting x D LB C .Y /. It is now clear that the linear transformation T from V into W can be replaced by the transformation L1 C T LB , which is a linear transformation from Rn1 into Rm1. While T transforms each matrix Z in V into a matrix T .Z/ in W, 0 L1 C T LB transforms the n-dimensional column vector x D .x1 ; : : : ; xn / , whose elements are the coefficients of the basis matrices V1 ; : : : ; Vn in the representation Z D x1V1 C CxnVn , into the m-dimensional column vector y D .y1 ; : : : ; ym /0 , whose elements are the coefficients of the basis matrices W1 ; : : : ; Wm in the representation T .Z/ D y1W1 C C ymWm . 0 The image .L1 C T LB /.x/ of an arbitrary n 1 vector x D .x1 ; : : : ; xn / under 1 the transformation LC T LB can be obtained by premultiplying x by a certain matrix. To see this, observe that (for j D 1; : : : ; n) the image T .Vj / of the j th basis matrix Vj for V (under the transformation T ) can be uniquely expressed as a linear combination of the basis matrices for W; that is, there exist unique scalars a1j ; a2j ; : : : ; amj such that

T .Vj / D a1j W1 C a2j W2 C C amj Wm :

(4.3)

And let A represent the m n matrix whose j th column is .a1j ; a2j ; : : : ; amj /0 or, equivalently, whose ij th element is aij . This matrix, which is dependent on the choice of the bases B and C , is referred to as the matrix representation of the transformation T (or simply as the matrix of T ).

602


P Now, let Z D jnD1 xj Vj [where x D .x1 ; : : : ; xn /0 is an arbitrary n 1 vector]. Then, Z is in V, and there exists a unique vector y D .y1 ; : : : ; ym /0 in Rm1 such that m X yi Wi : (4.4) T .Z/ D iD1

Moreover, it follows from the definition of a linear transformation that T .Z/ D T

n X

n n m X n m X X X X xj Vj D xj T .Vj / D xj aij Wi D xj aij Wi :

j D1

j D1

j D1

iD1

iD1 j D1

And, in light of the uniqueness of the coefficients y1 ; : : : ; ym in representation (4.4), we have that n X xj aij .i D 1; : : : ; m/: (4.5) yi D j D1

The m equalities (4.5) can be rewritten as the matrix equality y D Ax:

(4.6)

Moreover, since clearly T ŒLB .x/ D T .Z/ D LC .y/, it follows from result (4.1) that y D .L1 C T LB /.x/. Thus, .L1 C T LB /.x/ D AxI

(4.7)

that is, the image of x under the transformation L1 C T LB equals Ax. Let S represent a linear transformation from V into W such that S ¤ T. Further, define F D ffij g to be the matrix representation of S (with respect to the bases B and C ), and observe that, as a consequence of Lemma 22.1.9, there exists an integer j (between 1 P and n, inclusive) such that S.Vj / ¤ T .Vj /. Then, since (by definition) S.Vj / D m iD1 fij Wi , it follows that fij ¤ aij for some integer i and hence that F ¤ A. Thus, distinct transformations from V into W have distinct matrix representations (with respect to the same bases). b. Some special cases It is instructive to consider the matrix representation of a linear transformation in the following special cases. 1. Let T represent a linear transformation from an n-dimensional linear space V into Rm1. Further, let B represent a set of matrices V1 ; : : : ; Vn that form a basis for V, and let C represent a set of (m-dimensional) column vectors w1 ; : : : ; wm that form a basis for Rm1. Then, letting W represent the m m (nonsingular) matrix whose first, : : : ; mth columns are w1 ; : : : ; wm , respectively, the matrix representation of T with respect to B and C is (by definition) the unique m n matrix A such that


603

W A D ŒT .V1 /; : : : ; T .Vn /;

(4.8)

W 1 ŒT .V1 /; : : : ; T .Vn /:

(4.9)

that is, the matrix m1

Note that when C is taken to be the natural basis for R (4.9) simplifies to ŒT .V1 /; : : : ; T .Vn /.

, W D I, and the matrix

2. Let A represent an m n matrix. Then, as discussed in Section 22.1, the transformation T from Rn1 into Rm1 defined by T .x/ D Ax is a linear transformation. The matrix representation of T with respect to the natural bases for Rn1 and Rm1 (which consist of the columns of In and Im , respectively) is the matrix A. However, except for special cases, the matrix representation of T with respect to other bases differs from A. 3. Let V represent an n-dimensional linear space. And let B D fV1 ; : : : ; Vn g and C D fW1 ; : : : ; Wn g represent arbitrary bases for V. Then, the matrix representation (with respect to B and C ) of the identity transformation I from V onto V is the unique n n matrix A D faij g such that (for j D 1; : : : ; n) Vj D a1j W1 C a2j W2 C C anj Wn :

(4.10)

Moreover, the columns of A are linearly independent and hence A is nonsingular (since if the columns of A were linearly dependent, then, in light of the linear independence of V1 ; : : : ; Vn , we would arrive at a contradiction of Lemma 3.2.4). Note that if x D .x1 ; : : : ; xn /0 is the n-dimensional column vector whose elements are the coefficients of the basis matrices V1 ; : : : ; Vn when a matrix Z (in V) is expressed as Z D x1V1 C C xnVn , then the first, : : : ; nth elements y1 ; : : : ; yn of the n-dimensional column vector y D Ax are the coefficients of the basis matrices W1 ; : : : ; Wn when Z is expressed as Z D y1W1 C C ynWn . Note also that A D In if (and only if) B D C . Further, for any n n nonsingular matrix A D faij g and for any choice of the basis C , the basis B can be chosen so that A is the matrix representation of the identity transformation I (from V onto V) with respect to B and C . This can be done by choosing (for j D 1; : : : ; n) Vj D a1j W1 C a2j W2 C C anj Wn ; that V1 ; : : : ; Vn are linearly independent (and hence form a basis for V) is evident from Lemma 3.2.4. Now, suppose that V D Rn1. Then, B comprises n column vectors v1 , : : : ; vn , and similarly C comprises n column vectors w1 ; : : : ; wn . Defining V D .v1 ; : : : ; vn / and W D .w1 ; : : : ; wn / (both of which are n n nonsingular matrices), the matrix representation of the identity transformation I from Rn1 onto Rn1 with respect to B and C is the (unique) n n (nonsingular) matrix A such that W A D V [as is evident from equality (4.10)], that is, the matrix W 1 V . 4. Let B represent a set of matrices Y1 ; : : : ; Yn that form a basis for an ndimensional linear space W, and let C represent the set comprising the first, : : : ; nth columns, say e1 ; : : : ; en , of In (which is the natural basis for Rn1 ). Further, let A D faij g denote the matrix representation of the linear transformation LB (with respect to C and B). For j D 1; : : : ; n, we have that

604


Yj D LB .ej / D

n X

aij Yi ;

iD1

implying that ajj D 1 and that, for i ¤ j , aij D 0. Thus, A D In . 5. Let T represent a linear transformation from Rmn into Rpq. Further, let C represent the natural basis for Rmn, comprising the mn matrices U11 ; U21 , : : : ; Um1 , : : : ; U1n ; U2n , : : : ; Umn , where (for i D 1; : : : ; m and j D 1; : : : ; n) Uij is the m n matrix whose ij th element equals 1 and whose remaining mn 1 elements equal 0; and, similarly, let D represent the natural basis for Rpq. Then, making use of result (3.4), we find that, for any mn 1 vector x, 1 1 1 .LD T LC /.x/ D .LD .T LC //.x/ D LD Œ.T LC /.x/

D vecŒ.T LC /.x/ D vecfT ŒLC .x/g: Further, for any m n matrix X D fxij g, 1 T LC /.vec X/ D vecfT ŒLC .vec X/g .LD

D vec.T fLC ŒL1 C .X/g/ D vecŒT .X/;

(4.11)

implying in particular that i h X 1 .LD T LC /.vec X/ D vec T xij Uij i;j

D vec D

hX

X

i xij T .Uij /

i;j

xij vecŒT .Uij /

i;j

D Œvec T .U11 /; : : : ; vec T .Um1 /; : : : ; vec T .U1n /; : : : ; vec T .Umn / vec.X/: And, in light of result (4.7), it follows that the matrix representation of T (with respect to the natural bases C and D) equals the pq mn matrix Œvec T .U11 /; : : : ; vec T .Um1 /; : : : ; vec T .U1n /; : : : ; vec T .Umn /:

(4.12)

Suppose, for instance, that (for every m n matrix X) T .X/ D AXB; where A is a p m matrix and B an n q matrix. Then, the matrix representation of T (with respect to the natural bases C and D) is B0 ˝ A. To see this, observe [in light of result (4.11) and Theorem 16.2.1] that 1 .LD T LC /.vec X/ D vecŒT .X/ D .B0 ˝ A/ vec X;


605

and apply result (4.7). Or, alternatively, letting ai represent the i th column of A and bj0 the j th row of B, observe [in light of result (16.2.3)] that vec T .Uij / D vec.ai bj0 / D bj ˝ ai (i D 1; : : : ; mI j D 1; : : : ; n) and hence [in light of results (16.1.28) and (16.1.27)] that expression (4.12) simplifies to .b1 ˝ a1 ; : : : ; b1 ˝ am ; : : : ; bn ˝ a1 ; : : : ; bn ˝ am / D .b1 ˝ A; : : : ; bn ˝ A/ D .b1 ; : : : ; bn / ˝ A D B0 ˝ A: 6. Let V represent the linear space of all n n symmetric matrices and W the linear space of all p p symmetric matrices, and let T represent a linear transformation from V into W. Further, let B represent the usual basis for V, comprising ; U21 ; : : : ; Un1 ; : : : ; Uii ; UiC1;i ; : : : ; Uni ; : : : ; Unn , the n.n C 1/=2 matrices U11 where (for i D 1; : : : ; n) Ui i is the nn matrix whose i th diagonal element equals 1 and whose remaining n2 1 elements equal 0 and where (for j < i D 1; : : : ; n) Uij is the n n matrix whose ij th and j i th elements equal 1 and whose remaining n2 2 elements equal 0. And let C represent the usual basis for W. Then, making use of result (3.5), we find that, for any n.n C 1/=2 1 vector x, 1 1 .L1 C T LB /.x/ D .LC .T LB //.x/ D LC Œ.T LB /.x/

D vechŒ.T LB /.x/ D vechfT ŒLB .x/g: Further, for any n n symmetric matrix X D fxij g, .L1 C T LB /.vech X/ D vechfT ŒLB .vech X/g 1 .X/g/ D vechŒT .X/; D vech.T fLB ŒLB

(4.13)

implying in particular that i h X .L1 xij Uij C T LB /.vech X/ D vech T i;j

D vech D

X

hX

i xij T .Uij /

i;j

xij vechŒT .Uij /

i;j 0) from an n-dimensional linear space V into an m-dimensional linear space W. Show that there exist a basis E for V and a basis F for W such matrix that the Ik 0 . representation of T with respect to E and F is of the form 0 0 15. Let T represent a linear transformation from an n-dimensional linear space V into an m-dimensional linear space W, and let A represent the matrix representation of T with respect to bases B and C (for V and W, respectively). Use Part (2) of Theorem 22.5.1 to devise a “direct” proof that dimŒN.T / D dimŒN.A/ (as opposed to deriving this equality as a corollary of Theorem 22.5.2). 16. Let T represent a linear transformation from an n-dimensional linear space V into V. (a) Let U represent an r-dimensional subspace of V, and suppose that U is invariant relative to T. Show that there exists a basis B for V such that the matrix representation of T with respect to B and B is of the (upper E F block-triangular) form (where E is of dimensions r r). 0 H (b) Let U and W represent subspaces of V such that U ˚ W D V (i.e., essentially disjoint subspaces of V whose sum is V). Suppose that both U and W are invariant relative to T. Show that there exists a basis B for V such that the matrix representation of T with respect to B and B is of the (block-diagonal) form diag.E; H/ [where the dimensions of E equal dim.U/]. Section 22.6 17. Let V, W, and U represent linear spaces. (a) Show that the dual transformation of the identity transformation I from V onto V is I. (b) Show that the dual transformation of the zero transformation 01 from V into W is the zero transformation 02 from W into V. (c) Let S represent the dual transformation of a linear transformation T from V into W. Show that T is the dual transformation of S. (d) Let k represent a scalar, and let S represent the dual transformation of a linear transformation T from V into W. Show that kS is the dual transformation of kT. (e) Let T1 and T2 represent linear transformations from V into W, and let S1 and S2 represent the dual transformations of T1 and T2 , respectively. Show that S1 C S2 is the dual transformation of T1 C T2 .

620


(f) Let P represent the dual transformation of a linear transformation S from U into V, and let Q represent the dual transformation of a linear transformation T from V into W. Show that PQ is the dual transformation of T S. 18. Let S represent the dual transformation of a linear transformation T from an n-dimensional linear space V into an m-dimensional linear space W. And let A D faij g represent the matrix representation of T with respect to orthonormal bases C and D, and B D fbij g represent the matrix representation of S with respect to D and C . Using the result of Exercise 9 (or otherwise), show that B D A0. 19. Let A represent an m n matrix, let V represent an n n symmetric positive definite matrix, and let W represent an m m symmetric positive definite matrix. And let S represent the dual transformation of the linear transformation T from Rn1 into Rm1 defined by T .x/ D Ax (where x is an arbitrary n 1 vector). Taking the inner product of arbitrary vectors x and z in Rn1 to be x0 V z and taking the inner product of arbitrary vectors u and y in Rm1 to be u0 W y, generalize result (6.2) by obtaining a formula for S.y/. 20. Let S represent the dual transformation of a linear transformation T from a linear space V into a linear space W. (a) Show that ŒS.W/? D N.T / (i.e., that the orthogonal complement of the range space of S equals the null space of T ). (b) Using the result of Part (c) of Exercise 17 (or otherwise), show that ŒN.S /? D T .V/ (i.e., that the orthogonal complement of the null space of S equals the range space of T ). (c) Show that rank S D rank T.

References

Aitken, A. C. (1935), “On Least Squares and Linear Combination of Observations,” Proceedings of the Royal Society of Edinburgh, 55, 42–48. Aitken, A. C. (1949), “On the Wishart Distribution in Statistics,” Biometrika, 36, 59–62. Albert, A. (1972), Regression and the Moore-Penrose Pseudoinverse, New York: Academic Press. Baksalary, J. K., and Puntanen, S. (1989), “Weighted-Least-Squares Estimation in the General Gauss-Markov Model,” in Statistical Data Analysis and Inference, ed. Y. Dodge, Amsterdam: North-Holland, pp. 355–368. Bartle, R. G. (1976), The Elements of Real Analysis (2nd ed.), New York: John Wiley. Bartlett, M. S. (1951), “An Inverse Matrix Adjustment Arising in Discriminant Analysis,” Annals of Mathematical Statistics, 22, 107–111. Basilevsky, A. (1983), Applied Matrix Algebra in the Statistical Sciences, New York: NorthHolland. Beaumont, R. A., and Pierce, R. S. (1963), The Algebraic Foundations of Mathematics, Reading, MA: Addison-Wesley. Christensen, R. (1996), Plane Answers to Complex Questions: The Theory of Linear Models (2nd ed.), New York: Springer-Verlag. Cochran, W. G. (1934), “The Distribution of Quadratic Forms in a Normal System, With Applications to the Analysis of Covariance,” Proceedings of the Cambridge Philosophical Society, 30, 178–191. Driscoll, M. F., and Gundberg, W. R., Jr. (1986), “A History of the Development of Craig’s Theorem,” The American Statistician, 40, 65–70. Duncan, W. J. (1944), “Some Devices for the Solution of Large Sets of Simultaneous Linear Equations (With an Appendix on the Reciprocation of Patterned Matrices),” The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, Seventh Series, 35, 660–670. Dwyer, P. S. (1967), “Some Applications of Matrix Derivatives in Multivariate Analysis,” Journal of the American Statistical Association, 62, 607–625. Faddeeva, V. N. (1959), Computational Methods of Linear Algebra, New York: Dover.

622

References

Fedorov, V. V. (1972), Theory of Optimal Experiments, New York: Academic Press. Golub, G. H., and Pereyra, V. (1973), “The Differentiation of Pseudo-Inverses and Nonlinear Least Squares Problems Whose Variables Separate,” SIAM Journal on Numerical Analysis, 10, 413–432. Golub, G. H., and Van Loan, C. F. (1989), Matrix Computations (2nd ed.), Baltimore: Johns Hopkins University Press. Goodnight, J. H. (1979), “A Tutorial on the SWEEP Operator,” The American Statistician, 33, 149–158. Graybill, F. A. (1983), Matrices With Applications in Statistics (2nd ed.), Belmont, CA: Wadsworth. Guttman, L. (1946), “Enlargement Methods for Computing the Inverse Matrix,” Annals of Mathematical Statistics, 17, 336–343. Halmos, P. R. (1958), Finite-Dimensional Vector Spaces (2nd ed.), Princeton, NJ: Van Nostrand. Harville, D. A. (1996–97), “Generalized Inverses and Ranks of Modified Matrices," Journal of the Indian Society of Agricultural Statistics, 49 (Golden Jubilee Number), 67–78. Harville, D. A., and Fenech, A. P. (1985), “Confidence Intervals for a Variance Ratio, or for Heritability, in an Unbalanced Mixed Linear Model," Biometrics, 41, 137–152. Harville, D. A., and Kempthorne, O. (1997), “An Alternative Way to Establish the Necessity Part of the Classical Result on the Statistical Independence of Quadratic Forms,” Linear Algebra and Its Applications, 264 (Sixth Special Issue on Linear Algebra and Statistics), 205–215. Hearon, J. Z., and Evans, J. W. (1968), “Differentiable Generalized Inverses,” Journal of Research of the National Bureau of Standards, 72B, 109–113. Henderson, H. V., and Searle, S. R. (1979), “Vec and Vech Operators for Matrices, With Some Uses in Jacobians and Multivariate Statistics,” The Canadian Journal of Statistics, 7, 65–81. Henderson, H. V., and Searle, S. R. (1981a), “On Deriving the Inverse of a Sum of Matrices,” SIAM Review, 23, 53–60. Henderson, H. V., and Searle, S. R. (1981b), “The Vec-Permutation Matrix, the Vec Operator and Kronecker Products: A Review,” Linear and Multilinear Algebra, 9, 271–288. Lancaster, P., and Tismenetsky, M. (1985), The Theory of Matrices (2nd ed.), New York: Academic Press. Loynes, R. M. (1966), “On Idempotent Matrices,” Annals of Mathematical Statistics, 37, 295–296. Magnus, J. R., and Neudecker, H. (1979), “The Commutation Matrix: Some Properties and Applications,” The Annals of Statistics, 7, 381–394. Magnus, J. R., and Neudecker, H. (1980), “The Elimination Matrix: Some Lemmas and Applications,” SIAM Journal on Algebra and Discrete Mathematics, 1, 422–449. Magnus, J. R., and Neudecker, H. (1986), “Symmetry, 0 -1 Matrices and Jacobians: A Review,” Econometric Theory, 2, 157–190.

References

623

Magnus, J. R., and Neudecker, H. (1988), Matrix Differential Calculus With Applications in Statistics and Econometrics, New York: John Wiley. Marsaglia, G., and Styan, G. P. H. (1974), “Equalities and Inequalities for Ranks of Matrices,” Linear and Multilinear Algebra, 2, 269–292. Meyer, C. D. (1973), “Generalized Inverses and Ranks of Block Matrices,” SIAM Journal on Applied Mathematics, 25, 597–602. Miller, K. S. (1987), Some Eclectic Matrix Theory, Malabar, FL: Robert E. Krieger Publishing Company. Milliken, G. A., and Akdeniz, F. (1977), “A Theorem on the Difference of the Generalized Inverses of Two Nonnegative Matrices,” Communications in Statistics, Part A—Theory and Methods, 6, 73–79. Moore, E. H. (1920), “On the Reciprocal of the General Algebraic Matrix,” Bulletin of the American Mathematical Society, 26, 394–395. Penrose, R. (1955), “A Generalized Inverse for Matrices,” Proceedings of the Cambridge Philosophical Society, 51, 406–413. Puntanen, S., and Styan, G. P. H. (1989), “The Equality of the Ordinary Least Squares Estimator and the Best Linear Unbiased Estimator” (with discussion), The American Statistician, 43, 153–164. Rao, C. R. (1989), “A Lemma on Optimization of a Matrix Function and a Review of the Unified Theory of Linear Estimation,” in Statistical Data Analysis and Inference, ed. Y. Dodge, Amsterdam: North-Holland, pp. 397–417. Rao, C. R., and Mitra, S. K. (1971), Generalized Inverse of Matrices and Its Applications, New York: John Wiley. Reid, J. G., and Driscoll, M. F. (1988), “An Accessible Proof of Craig’s Theorem in the Noncentral Case,” The American Statistician, 42, 139–142. Rogers, G. S. (1980), Matrix Derivatives, New York: Marcel Dekker. Scheffé, H. (1959), The Analysis of Variance, New York: John Wiley. Searle, S. R. (1971), Linear Models, New York: John Wiley. Searle, S. R. (1982), Matrix Algebra Useful for Statistics, New York: John Wiley. Sherman, J., and Morrison, W. J. (1949), “Adjustment of an Inverse Matrix Corresponding to Changes in the Elements of a Given Column or a Given Row of the Original Matrix” (abstract), Annals of Mathematical Statistics, 20, 621. Sherman, J., and Morrison, W. J. (1950), “Adjustment of an Inverse Matrix Corresponding to a Change in One Element of a Given Matrix,” Annals of Mathematical Statistics, 21, 124–127. Stewart, G. W. (1973), Introduction to Matrix Computations, New York: Academic Press. Sylvester, J. (1884), “Sur la Solution du Cas le Plus Général des Équations Linéares en Quantités Binares, C’est-á-dire en Quaternions ou en Matrices du Second Ordre,” Comptes Rendus des Séances de l’Académie des Sciences, 99, 117–118. Woodbury, M. A. (1950), “Inverting Modified Matrices,” Memorandum Report 42, Princeton University, Statistical Research Group.

Index

absorption, 152–157 adjoint matrix, 194–195, 207 determinant of, see under determinant differentiation of, see under differentiation of a product, 208 Aitken equations, 271, 273–274, 275, 276–277 algebraic multiplicity, 524, 533–534, 538, 551, 552–553, 584 block-triangular matrix, 528 of zero, 552–553, 584 symmetric matrix, 541 triangular matrix, 529 angle, 56–57, 58, 60–61 back substitution, 148 basis, 31–36 existence of, 32–34 natural, 31 orthonormal, 63, 65–66, 69 bilinear form, 210–211 as a quasi-inner product, 258 as an inner product, 255–256 matrix of, 210 symmetric, 210–211, 277 Binet-Cauchy formula, 205 block-diagonal matrix, see under partitioned matrix block-triangular matrix, see under partitioned matrix Cayley-Hamilton theorem, 583 chain rule, 306–308, 330

characteristic equation, 523–524 characteristic polynomial, 523–524, 538, 551, 552–553, 586 coefficients of, 523 of a block-triangular matrix, 528 of a transposed matrix, 527 of a triangular matrix, 529 of an orthogonal matrix, 586 characteristic value (or root) or vector, 526 Cholesky decomposition, see under decomposition Cochran’s theorem, 419, 442 cofactor, 191 cofactor matrix, 194, 206 column rank, 36–38 column space, 27–29, 75, 77, 118, 159, 259 of a partitioned matrix, 41–45, 388, 389 of a product, 83 of a sum, 44–45, 455–456 commutation matrix, 348 conditional inverse, 107 congruence, 278 Cramer’s rule, 207 decomposition Cholesky, 232, 235, 237–238, 239, 280–281 Crout, 231 Doolittle, 231 LDU, 224–231, 235–237, 239, 279, 281 LU, 231 of a symmetric matrix, 218–220, 544–545

626

Index

of a symmetric nonnegative definite matrix, 220–222, 232–235, 237–239 QR, 66–68, 70, 150, 280–281 singular value, 521–522, 556–562, 587 approximation, 562–565 of Moore-Penrose inverse, 562 relationship to spectral decomposition, 561–562 uniqueness of, 559–561 spectral, 521–522, 544, 585 approximation, 565–566 of a power, 545 of an inverse matrix, 545 of Moore-Penrose inverse, 547 uniqueness of, 544–545, 585 U0 DU, 224–234, 237, 239, 279 determinant, 179–208, 279, 537–538, 545–546, 575–580 computation of, 191 differentiation of, see under differentiation effect of adding scalar multiples of rows or columns to other rows or columns, 186–187 effect of multiplication by a unit triangular matrix, 187 effect of row or column interchanges, 184–185 effect of scalar multiplication of a row or column, 183 expansion by cofactors, 192–194, 206 of a block-diagonal matrix, 189 of a block-triangular matrix, 187–189 of a diagonal matrix, 183 of a Kronecker product, see under Kronecker product of a partitioned matrix, 181, 191, 206 of a permutation matrix, 183 of a positive definite matrix, 248–250, 283 of a positive semidefinite matrix, 248–250 of a power, 190 of a product, 189, 203–205 of a scalar multiple, 184 of a singular matrix, 190–191 of a skew-symmetric matrix, 248–249 of a sum, 197–200, 421–422 of a transposed matrix, 183

of a triangular matrix, 182 of an adjoint matrix, 206 of an inverse matrix, 190–191 of an orthogonal matrix, 190 of R C ST U , 419–421, 450 diagonal element of a (square) matrix, see under matrix: square diagonal matrix, see under matrix diagonal of a (square) matrix, see under matrix: square diagonalizability, 537, 538–539, 585 of a Kronecker product, 555 of a symmetric matrix, 540–542 of a transposed matrix, 584 of an inverse matrix, 584 orthogonal, 537 simultaneous, 566–568, 588 diagonalization, 537 differentiation of a constant, 296 of a determinant, 308–311, 313, 331–332, 333, 334 of a generalized inverse, 314–319 of a linear combination of matrices, 301 of scalars, 296–298 of a linear form, 298–299 of a matrix of constants, 300 of a power of a determinant, 330–331 of a scalar, 298, 328 of a product of a scalar and a matrix, 303 of a scalar and a vector, 328 of matrices, 301–303 of scalars, 296–298 of a projection matrix, 319–324, 334–335 of a quadratic form, 298–300 of a ratio of 2 scalars, 296–298 of a reciprocal of a scalar, 297 of a trace, 304–306 of a trace of a power, 329 of a trace of a product, 305–306, 328–330 of a transposed matrix, 303 of a vector of linear forms, 300 of an adjoint matrix, 311–312, 332 of an eigenvalue, 572, 573–574 of an eigenvector, 572–574

Index of an inverse matrix, 311–312, 313, 333, 334 of Moore-Penrose inverse, 514–517 with respect to a matrix or symmetric matrix, 294–295, 303–304, 305–306, 328–330, 331, 332, 333 dimension(s) of a vector or matrix, see under matrix or vector direct product, 338 direct sum (of subspaces), 383–384, 402 dimension of, 384 distance, 56, 60, 68 dot product, 57–58, 59 duplication matrix, 356–366, 375, 376, 618 left inverse of, 357–366 recursive formula for, 359 rank of, 356 recursive formula for, 356–357 relationship to vec-permutation matrix, 359–361 eigenspace, 522 eigenvalue(s), 522–523, 551, 552–553 differentiation of, see under differentiation distinct, 526, 587 existence of, 524–526 for an orthogonal matrix, 586 for a symmetric matrix, 539–540 largest, 539–540 multiple, 525 multiplicity of algebraic, see algebraic multiplicity geometric, see geometric multiplicity not necessarily distinct, 526, 538, 585 block-triangular matrix, 528 Kronecker product, 555–556 triangular matrix, 529 of a block-triangular matrix, 528 of a Kronecker product, 554–556 of a nonnegative definite matrix, 549–550 of a power, 527 of a skew-symmetric matrix, 582 of a transposed matrix, 527 of a triangular matrix, 529 of an adjoint matrix, 586 of an idempotent matrix, 548–549, 586 of an inverse matrix, 527 of an orthogonal matrix, 548–549

627

of Moore-Penrose inverse, 546–547, 585 simple, 525 smallest, 539–540 eigenvector(s), 522–523, 538, 551, 552–553, 587 differentiation of, see under differentiation geometry of, 526 linear combination of, 584 linear independence of, 534–536 of a block-diagonal matrix, 529–530 of a diagonal matrix, 530 of a Kronecker product, 554 of a power, 527 of an adjoint matrix, 586 of an inverse matrix, 527 of Moore-Penrose inverse, 546–547 orthogonality of, 537 elimination matrix, 376 essential disjointness (of subspaces), 382–383 for row and column spaces, 385–391, 409–410, 493 of partitioned matrices, 390–391, 413 forward elimination, 149 Frobenius inequality, 399 function bilinear, 210 continuous, 290–291, 292 continuously differentiable, 291–292, 293–294, 327 differentiable, 292 linear, 209 partial derivative(s) of, 291–295 generalized eigenvalue problem, 569–570, 588 generalized inverse, 107–132, 135–136, 222–223, 239, 281, 454, 498 alternative characterizations for, 109–117, 127 differentiation of, see under differentiation existence of, 108–109 invariance to the choice of, 119–120 least squares, 504–506, 507, 510 least squares reflexive, 505–506, 507 minimum norm, 501–504, 507, 510

628

Index

minimum norm reflexive, 503–504, 507 Moore-Penrose inverse, see MoorePenrose inverse of a block-diagonal matrix, 123, 128, 131 of a block-triangular matrix, 122 of a Kronecker product, see under Kronecker product of a matrix of full row or column rank, 116–117 of a nonsingular matrix, 108 of a partitioned matrix, 122–125, 128, 137–138, 394–396, 413–414, 476–481 of a product, 113, 137 of a scalar multiple, 117 of a sum, 425–426 of a symmetric matrix, 115–116, 117, 223, 239 of a symmetric nonnegative definite matrix, 223, 239 of a transposed matrix, 117 of R C ST U , 430–432, 452–453 rank of, 118 reflexive, 500–501, 507 least squares, see generalized inverse: least squares reflexive minimum norm, see generalized inverse: minimum norm reflexive of a partitioned matrix, 517 rank of, 501 generalized least squares problem, see under least squares problem geometric multiplicity, 522, 533–534, 538, 551, 552–553 block-diagonal matrix, 529 diagonal matrix, 530 of zero, 527, 553, 584, 585 symmetric matrix, 541 gradient, 291, 295 Gram matrix, 277–278 Gram-Schmidt orthogonalization, 64–65, 69 Gramian, 277

image of a matrix, 589, 593 of a set (of matrices), 590 indefiniteness, see under matrix or quadratic form independence of subspaces, 412, 415, 416–417 index of inertia, 278–279 inner product, 57–59, 255, 284, 285, 373–374, 613–616 integration, 324–327 intersection (of subsets), 379–380 intersection of subspaces dimension of, 396–397 for column spaces, 397 for row spaces, 397 orthogonal complement of, 411 inverse, 80–82, 195, 583 differentiation of, see under differentiation of a block-diagonal matrix, 88 of a block-triangular matrix, 89, 90–91, 92–97, 104–105 of a diagonal matrix, 80–81 of a difference between 2 inverses, 424–425 infinite-series representation for, 432–437, 453 of a Kronecker product, see under Kronecker product of a partitioned matrix, 98–101 of a product, 82 of a scalar multiple, 82 of a sum, 423–424 of a symmetric matrix, 82 of a transposed matrix, 82 of a triangular matrix, 94, 96 of an inverse, 82 of R C ST U , 427–430, 452 invertible matrix, see under matrix involutory matrix, see under matrix isomorphism, see under linear transformation

Helmert matrix, 85–86 Hessian matrix, 292–293, 327

Jacobian, 295 of a vec of a product, 369 of an inverse matrix, 370–371 of a vech

idempotent matrix, see under matrix identity matrix, see under matrix

Index of a product, 370 of an inverse matrix, 371 Jacobian matrix, 295 of a vec, 368 of a Kronecker product, 377 of a power, 377 of a product, 369 of an inverse matrix, 370–371 of a vech, 369 of a product, 370 of an inverse matrix, 371 Kronecker product(s), 337–343, 372, 375 determinant of, 353–354 effect of multiplication by a vecpermutation matrix, 351–352 generalized inverse of, 342, 372 inverse of, 342 involving a diagonal matrix, 339 involving a null matrix, 339 involving a partitioned matrix, 342–343 involving a scalar multiple, 339 involving a sum, 339 involving an identity matrix, 339 LDU decomposition of, 373 left, 338 Moore-Penrose inverse of, see under Moore-Penrose inverse nonnegative definiteness of, 372 norm of, 372 of 2 vectors, 338–339, 344, 372 of 3 or more matrices, 340–341 of a scalar and a matrix, 338 of matrix products, 341 of symmetric matrices, 340 positive definiteness of, 372 product of, 341 projection matrix for column space of, 372 rank of, 342 right, 338 trace of, 342 transpose of, 340 triangularity of, 372 Laplace’s theorem, 200–203 latent value or vector, 526 least squares problem, 170–171 constrained, 459, 491–492

629

generalized, 270–271, 275–277 minimum norm solution, 512 ordinary, 270 simple, 270 unweighted, 270 weighted, 270 left inverse, 79–80, 103, 117 linear dependence, 23–26 linear equations, see linear system linear form, 209–210 coefficient vector of, 209–210 coefficients of, 209 linear functional, 612 linear independence, 23–26, 373 linear space(s), 28–29 dimension of, 34–35 isomorphic, 599–600 subspace(s) of, 29–31 essential disjointness of, see essential disjointness invariant, 526–527, 582, 612 sum of, see sum of subspaces linear system(s), 72–77, 81, 126 augmented, 151–152 coefficient matrix of, 72 compatibility of, 73–74 consistency of, 72–75, 76, 121, 259, 393–394, 427 equivalence of, 145–146, 159, 160, 367–368, 376–377 homogeneous, 72, 140, 142–143 inconsistent, 72 nonhomogeneous, 72, 141–142, 143 reduced, 155 right side of, 72 solution set of, 139–145, 157–158 solution to, 72, 108, 126, 139–160, 393, 426–427 invariance to choice of, 150–151, 160 linear combination of elements of, 150–152 linear transformation, 590 adjoint (transformation), 615 coordinate-free approach to, 610 dual (transformation), 615, 620 eigenmatrix (or eigenvector) of, 611–612 eigenvalue of, 611–612 identity (transformation), 592, 598 isomorphism, 599

630

Index

kernel of, 591 matrix representation of, 601–609, 617, 618, 619 effect of changing bases on, 608–609 null space of, 591–592, 610–611 range space of, 590, 591–592, 598, 610 rank of, 611 restriction of, 592–593, 616 zero (transformation), 592, 598 matrix (or matrices), 1–11 addition of, 2–3 block-diagonal, see under partitioned matrix block-triangular, see under partitioned matrix column of, 20–21 commutativity of, 4 conformal for addition (or subtraction), 3 decomposition of, see decomposition determinant of, see determinant diagonal, 6 diagonalizable (or diagonable), see diagonalizability diagonally dominant, 283 difference between, 3 dimensions of, 2 element of, 2, 21 entry of, 2 equality of, 2, 52, 77 idempotent, 133–136, 146, 278, 328, 372, 387, 439–444, 453–454, 518 nonnegative definiteness of, 217 rank of, see under rank identity, 7, 21 indefinite, 213 inverse of, see inverse invertible, 80 involutory, 103, 136 multiplication by a scalar, 2 multiplication by another matrix, 3–4 postmultiplication, 4 premultiplication, 4 negative definite, 213 negative of, 2 negative semidefinite, 213 nonnegative definite, 213–217, 254–255 decomposition of, see under decomposition

effect of subtracting from its MoorePenrose inverse the Moore-Penrose inverse of another matrix, 519 partitioned, 243–246 square root of, see square root sum of, 214 trace of, see under trace nonpositive definite, 213 nonsingular, 38, 80 null, 7 of cofactors, see cofactor matrix of functions continuously differentiable, 296 partial derivative of, 295–296 of ones, 7 orthogonal, 84–87, 372 orthogonally diagonalizable, see under diagonalizability partitioned, see partitioned matrix permutation, see permutation matrix positive definite, 213–217, 250–254 determinant of, see under determinant diagonal element of, 216–217 effect of adding other matrices, 437–438 effect of subtracting from its inverse the inverse of another matrix, 438–439 inverse of, 216, 278 nonsingularity of, 215, 222 principal submatrix of, 216 positive semidefinite, 213–217, 277 determinant of, see under determinant diagonal element of, 216–217 principal submatrix of, 216 product of, 3–4 row of, 20 scalar multiple of, 2 singular, 38 skew-symmetric, 239–240, 281–282, 451 determinant of, see under determinant square, 5–6 diagonal element of, 6 diagonal of, 6 off-diagonal element of, 6 order of, 5 subtraction of, 2–3 sum of, 3 symmetric, 6, 40

Index transpose of, 5, 614, 615–616 triangular, 8–9 lower, 8–9 unit, 9 upper, 8–9 minimization (of a 2nd-degree polynomial), 459–495 subject to linear constraints, 463–476, 481–489, 494–495, 518–519, 588 computational considerations, 469, 470, 475 convexity (of Lagrangian function), 467–468 Lagrange multipliers, 463–467 transformation to unconstrained minimization, 489–490 unconstrained, 460–462 convexity (of the polynomial), 462 least squares approach, 461–462 minor, 191 Moore-Penrose inverse (or pseudoinverse), 497–499, 507–511, 518 differentiation of, see under differentiation expression as a limit, 513–514 multiplication by a matrix with orthonormal rows or columns, 510–511 null space of, 509 of a block-diagonal matrix, 499 of a diagonal matrix, 499 of a Kronecker product, 511 of a matrix of full row or column rank, 499 of a matrix with orthonormal columns, 499 of a nonsingular matrix, 499 of a product, 517 of a scalar multiple, 509–510 of a sum, 518 of a symmetric idempotent matrix, 499 of a symmetric matrix, 509 of a symmetric nonnegative definite matrix, 509 of a transposed matrix, 509 of Moore-Penrose inverse, 509 rank of, 509 row or column space of, 509

631

natural basis, see under basis negative definiteness, see under matrix or quadratic form negative pair (of matrix elements), 179–180 negative semidefiniteness, see under matrix or quadratic form neighborhood, 290, 327 radius of, 290 nonnegative definiteness, see under matrix or quadratic form nonpositive definiteness, see under matrix or quadratic form norm, 56, 58, 59–60, 84–85 usual, 542, 561 normal equations, 164, 166, 171 normalizing a matrix or vector, 61 null matrix, see under matrix null space, 139, 140, 142–143, 144, 146, 159, 172, 272, 287 off-diagonal element of a (square) matrix, see under matrix: square order of a (square) matrix, see under matrix: square ordinary least squares problem, see under least squares problem orthogonal complement, 172–176, 271, 287, 410–411 dimension of, 172, 175–176, 272, 287 orthogonal complement of, 172–173 projection on, 174–176, 272, 287 orthogonal matrix, see under matrix orthogonal projection, 163 orthogonal set (of matrices or vectors), 61–62, 64, 69 orthogonality of 2 matrices or vectors, 58, 61 of 2 subspaces, 161–162, 260–262, 384–385, 409–410 of a matrix and a subspace, 161–162, 260–262 orthonormal basis, see under basis orthonormal set (of matrices or vectors), 61–62, 65 pack, 378 partitioned matrix (or matrices), 14–19, 451 block of, 14–16

632

Index

diagonal, 15 off-diagonal, 15 block-diagonal, 16, 45–46 nonnegative definiteness of, 244–245 positive definiteness of, 244–245 block-triangular, 16, 146–149 lower, 16 upper, 16 column space of, see under column space conformal (partitioning) for addition, 18 for multiplication, 18 determinant of, see under determinant generalized inverse of, see under generalized inverse inverse of, see under inverse nonnegative definiteness of, 246–247, 283 positive definiteness of, 246–247 product of, 18 rank of, see under rank row space of, see under row space scalar multiple of, 17 sum of, 18 trace of, see under trace transpose of, 17 Penrose (or Moore-Penrose) conditions, 499, 500, 506, 511, 518 permutation matrix, 86–87, 101–103 polynomial, 577–579, 580–582 degree of, 580 factor of, 580, 581 nonzero, 580 root (or zero) of, 580, 581 positive definiteness, see under matrix or quadratic form positive pair (of matrix elements), 179–180 positive semidefiniteness, see under matrix or quadratic form postmultiplication, see under matrix: multiplication by another matrix premultiplication, see under matrix: multiplication by another matrix projection, 163, 403–407, 411 of a column vector, 163–166, 262–263, 268–270, 272, 273–275, 285, 286–287, 404, 405–407 projection matrix, 166–170, 174, 177, 264–268, 272–273, 275, 286,

407–409, 411, 415–416, 454–455, 493, 504–505, 509 differentiation of, see under differentiation proper value or vector, 526 pseudoinverse, 107, 498 quadratic form(s), 211 equality of, 211–212 indefinite, 213 matrix of, 211–212 negative definite, 212–213 negative semidefinite, 212–213 nonnegative definite, 212 nonpositive definite, 212–213 positive definite, 212 positive semidefinite, 212 reexpression of, 212, 220, 221, 222 quasi-inner product, 257–259, 285 quasi-norm, 258–259 Rmn , 29 Rn , 29 rank, 36–40, 75, 77, 135, 259, 537–538 additivity of, 439–441, 450, 456, 457 of a block-triangular matrix, 89–92, 122 of a difference, 456–457 between 2 inverses, 424 of a generalized inverse, see under generalized inverse of a Kronecker product, see under Kronecker product of a partitioned matrix, 41–45, 88, 98–99, 101, 122–125, 137–138, 386, 388, 391, 392, 397, 413, 414, 457 of a product (of matrices), 37, 83, 103, 398–402 of a sum, 44–45, 446–449, 455 of a triangular matrix, 92, 104 of an idempotent matrix, 134–135 of R C ST U , 444–446, 455, 458 subtractivity of, 456–457 right inverse, 79–80, 103, 117 row rank, 36–38 row space, 28–29, 75, 77, 118, 259 of a partitioned matrix, 41–45, 388 of a product, 83 of a sum, 44–45, 455–456 scalar, 2

Index Schur complement, 99, 124–125, 129–131 nonnegative definiteness of, 245–246 positive definiteness of, 245–246 Schwarz inequality, 61, 62–63, 68, 256–257, 258 set interior point of, 290 open, 290 similarity, 530–533, 583, 609 of Kronecker products, 554 to a diagonal matrix, 537 to an idempotent matrix, 583 simple least squares problem, see under least squares problem singular value(s), 560 of Moore-Penrose inverse, 562 singular-value decomposition, see under decomposition skew-symmetric matrix, see under matrix solution set, see under linear system span (of a set of matrices), 31 spectral decomposition (or representation), see under decomposition spectrum, 522, 524, 538 square matrix, see under matrix square root (of a symmetric nonnegative definite matrix), 550–551 square root method, 237–238 submatrix, 13–14, 39–40 leading principal, 14 of a transposed matrix, 14 principal, 14, 40 subspace, see under linear space sum of subsets, 380–381 sum of subspaces, 411, 412 dimension of, 384, 396–397, 414 direct, see direct sum orthogonal complement of, 411 sweep operation, 131 Sylvester’s law of inertia, 278 Sylvester’s law of nullity, 399 symmetric matrix, see under matrix system of linear equations, see linear system Taylor formula, 292, 293 tensor product, 338 trace, 49–53, 538, 545–546

633

differentiation of, see under differentiation of a Kronecker product, see under Kronecker product of a nonnegative definite matrix, 240–241 of a partitioned matrix, 50 of a product, 50–52, 282, 346, 373 of nonnegative definite matrices, 241–243 of a scalar multiple, 49 of a sum, 49–50 of a transposed matrix, 49 transformation(s), 589–620 composition of, 596 difference between, 596 domain of, 589 equality of, 595 inverse of, 599, 607–608, 617 invertible, 598, 617 linear, see linear transformation negative of, 595 one to one, 590, 593–595 onto, 589–590, 593–594 product of, 596–597, 598, 607, 616 range of, 589 scalar multiple of, 595, 597–598, 606 sum of, 596, 597, 606, 607 transpose (of a matrix), see under matrix triangle inequality, 68 triangular matrix, see under matrix triangularizability (orthogonal), 542–543 union (of subsets), 380, 411 unweighted least squares problem, see under least squares problem Vandermonde matrix, 195 determinant of, 196–197, 207–208 rank of, 197 vec, 343–347, 600, 604–605 of a Kronecker product, 353, 375 of a linear combination, 344 of a partitioned matrix, 347 of a product of a column vector and a row vector, 344 of a product of matrices, 344–346 of a scalar multiple, 344 of a sum, 344 of a transposed matrix, 347–348

634

Index

of a vector, 344 of an idempotent matrix, 373 of an orthogonal matrix, 373 vec-permutation matrix, 348–354, 374, 375, 617 determinant of, 374 inverse of, 350 trace of, 350 transpose of, 350

vech, 354–358, 600, 605–606, 617–618 of a linear combination, 355 of a product of matrices, 361–366 vector column, 9, 20 dimension of, 9 row, 9 weighted least squares problem, see under least squares problem

Matrix Algebra From a Statistician's Perspective

Harville Matrix algebra from a statiscian's perspective

Matrix Algebra From A Statiscian's Perspective

A matrix handbook for statisticians

Methods of Matrix Algebra

Methods of matrix algebra

Matrix Algebra: An Introduction

Matrix Algebra: An Introduction

Matrix Algebra (Econometric Exercises)

Introduction to Matrix Algebra

Matrix Algebra: Exercises and Solutions

Matrix methods. Applied linear algebra

Islam from a Contemporary Perspective

Philosophy from a Skeptical Perspective

Philosophy from a Skeptical Perspective

Philosophy from a Skeptical Perspective

Numerical analysis for statisticians

Numerical analysis for statisticians

Statistical Inference, Econometric Analysis and Matrix Algebra

Applied Linear Algebra and Matrix Analysis

Matrix analysis & applied linear algebra

Matrix Analysis And Applied Linear Algebra

Matrix Methods: Applied Linear Algebra (Third Edition)

Matrix Analysis and Applied Linear Algebra (solutions)

Einfuehrung in die moderne Matrix-Algebra

Matrix Analysis and Applied Linear Algebra

Matrix Analysis and Applied Linear Algebra

Elementary Linear Algebra - A Matrix Approach (2nd Edition)

Matrices and their roots: a textbook of matrix algebra

Probability for Statisticians

Stereology for Statisticians

Matrix Algebra From a Statistician's Perspective

Harville Matrix algebra from a statiscian's perspective

Matrix Algebra From A Statiscian's Perspective

A matrix handbook for statisticians

Methods of Matrix Algebra

Methods of matrix algebra

Matrix Algebra: An Introduction

Matrix Algebra: An Introduction

Matrix Algebra (Econometric Exercises)

Introduction to Matrix Algebra

Matrix Algebra: Exercises and Solutions

Matrix methods. Applied linear algebra

Islam from a Contemporary Perspective

Philosophy from a Skeptical Perspective

Philosophy from a Skeptical Perspective

Philosophy from a Skeptical Perspective

Numerical analysis for statisticians

Numerical analysis for statisticians

Statistical Inference, Econometric Analysis and Matrix Algebra

Applied Linear Algebra and Matrix Analysis

Matrix analysis & applied linear algebra

Matrix Analysis And Applied Linear Algebra

Matrix Methods: Applied Linear Algebra (Third Edition)

Matrix Analysis and Applied Linear Algebra (solutions)

Einfuehrung in die moderne Matrix-Algebra

Matrix Analysis and Applied Linear Algebra

Matrix Analysis and Applied Linear Algebra

Elementary Linear Algebra - A Matrix Approach (2nd Edition)

Matrices and their roots: a textbook of matrix algebra

Probability for Statisticians

Stereology for Statisticians

Recommend Documents