Statistical Optimization for Geometric Computation: Theory and Practice

Preface One of the central tasks of computer vision and robotics is building a 3-D model of the environment from sensor and image data. An intrinsic difficulty of this task is the fact that sensor and image data are not necessarily accurate. In the past, many techniques have been proposed to cope with this problem. One of the most effective solutions is to take advantage of our prior knowledge about the structure of the problem: we make use of such geometric relationships as collinearity, coplanarity, rigidity, and epipolar relation that should hold in the absence of noise; they are called geometric constraints. This book focuses on two such techniques that are the most fundamental. One is geometric correction: the data are optimally modified so as to satisfy the geometric constraints. The other is parametric fitting: the parameters of the geometric constraints are optimally estimated. In order that a particular method be called "optimal," the theoretical bound on its attainable accuracy must be known; only methods which can attain it are qualified to be called optimal. In this book, we give many synthetic and real data examples to demonstrate that conventional methods are not optimal and how accuracy improves if truly optimal methods are employed. However, computing optimal estimates alone is not sufficient; at the same time, we must evaluate in quantitative terms how reliable the resulting estimates are. The knowledge that a distance is optimally estimated to be 5m is of little use if we do not know whether the value is reliable within 4-10cm or E l m . This reliability issue has not received much attention in the past. In order to compute optimal estimates and evaluate their reliability, we need an efficient numerical algorithm. Since such estimation is based on our knowledge about the structure of the problem, we also need a criterion for testing if the assumed structure, or model, is correct. This book presents rigorous mathematical techniques for these purposes. Thus, the subject of this book is closely related to traditional statistics. However, the main goal of traditional statistics is to infer the structure of a phenomenon from a number of sample data obtained by repeated measurements. In our applications, usually only one set of data is obtained by a measurement. Also, noise in electronic devices can be assumed to be fairly small, so we can apply first order analysis. Since geometric objects are represented by vectors and tensors in two and three dimensions, linear algebra, eigenvalue analysis in particular, plays a central role in our analysis. These features make our treatment very different from traditional statistics. This book first summarizes the mathematics that is flmdamental to our analysis and then presents techniques for optimal estimation and reliability evaluation by assuming that the noise is Gaussian. We also discuss numerical computation schemes called the optimal filter and renormalization and consider computational issues such as convergence of iterations and numerical stability. Then, we derive the geometric information criterion and apply it

vi to such problems as stereo vision and 3-D motion analysis. Finally, the theoretical accuracy bound for a general non-Gaussian noise model is obtained in the form of the Cramer-Rao lower bound expressed in terms of the Fisher information matrix. This book is an elaboration of the author's lecture note for a series of seminars he gave at the Department of Mathematical Engineering and Information Physics, University of Tokyo in 1994. The author thanks Shun-ichi Amari, Kokichi Sugihara, Koichiro Deguchi, and Ken Hayami of the Department of Mathematical Engineering and Information Physics, University of Tokyo and Keisuke Kinoshita of the ATR Human Information Processing Research Laboratories for detailed discussions. He also thanks Azriel Rosenfeld of the University of Maryland, who carefully read the entire manuscript, and Naoya Ohta of Gunma University, Yasushi Kanazawa of Gunma College of Technology, and all of the students in his laboratory for helping him with this laborious project. This work was in part supported by the Ministry of Education, Science, and Culture, Japan under a Grant in Aid for Scientific Research B (No. 07458067) and the Okawa Institute of Information and Telecommunication, Japan. Gunma, Japan January 1996

Kenichi Kanatani

Chapter 1 Introduction This chapter introduces the aims, features, backgrounds, and organization of the subsequent chapters. In particular, the differences between our treatment and existing studies are described in detail. Also, related references are given for each subject. Since the discussion here refers to many technical issues in the subsequent chapters, readers are advised to go through this chapter for a general overview in the first reading and come back here later for detailed information.

1.1

1.1.1

T h e A i m s of This B o o k

Statistical optimization for image and sensor data

Intelligent robots are expected to play a vital role in manufacturing industries and various hazardous environments such as nuclear power stations, outer space, and deep water. For autonomous robotic operations in an unknown environment, robots must first acquire a 3-D model of their workspace. The most fundamental source of 3-D information is vision. Today, the study of extracting 3-D information from video images and building a 3-D model of the scene, called computer vision or image understanding, is one of the research areas that attract the most attention all over the world [5, 6, 12, 47, 57, 65, 85, 90, 130, 135, 151, 185, 211, 227, 229]. Various other sensing techniques have also been developed--tactile sensing and ultrasonic range sensing, for example. The crucial fact about image and sensor data is that they are not necessarily accurate. We discuss errors in range sensing in Section 10.3, but in the rest of the book we exclusively deal with points and lines detected from camera images. Errors in such image data originate from various sources. Camera images undergo optical distortion, called aberration, due to imperfections of the lens. They are also distorted by mechanical causes (e.g., misalignment of the lens and the array sensor) and electronic causes (e.g., difference between the vertical scanning rate and the horizontal sampling rate). Theoretically, such systematic distortions can be corrected by prior calibration. In reality, however, it is very difficult to remove them completely. Even if a perfect camera is used, images are usually inaccurate because of poor lighting, imperfect focusing, limited resolution, and various factors concerning the scene and objects in question (shades, shadows, specularity, reflection, etc.). Hence, points and lines detected by applying an image processing operation to the gray levels are not located where they should be. In this book, such deviations

2

Chapter 1.

Introduction

of points and lines are collectively termed "image noise". Today, coping with the uncertainty of the data, whatever origins they have, is one of the greatest challenges to computer vision and robotics researchers, and many techniques have been proposed for this purpose. They are basically categorized into two types"

9 Traditional statistical approach: the uncertainty of the data is conquered by repeated measurements. 9 Physics-oriented approach: the uncertainty of the data is compensated for by a priori knowledge about the structure of the problem. This book concentrates on the latter approach. Namely, we take advantage of the geometric constraints that measurement values should satisfy if noise did not exist. For point and line data, such constraints include collinearity and coplanarity of points, concurrency of lines, rigidity of motion, and what is known as the epipolar constraint for stereo vision and motion analysis. We focus on two of the most fundamental methods that make use of such constraints:

9 Geometric correction: the data are optimally modified so as to satisfy the geometric constraints exactly. 9 Parametric fitting: the parameters of the geometric constraints are optimally estimated. The latter is not limited to fitting curves and surfaces to point data [20, 200, 160, 228] but covers a much wider range of problems such as 3-D motion analysis, as we will show later. In the following, we primarily consider points and lines in images, but in principle our statistical theory can also apply to other types of image data (e.g., gray levels of pixels) and sensor data obtained from devices other than cameras.

1.1.2

W h a t are the i s s u e s ?

For any type of statistical estimation, we encounter the following issues:

9 Accuracy bound. In order to claim that a particular method is "optimal," one must know the theoretical bound on its attainable accuracy; only those methods which attain that bound are qualified to be called "optimal." If such methods are not known, we can still evaluate the performance of available methods by comparing their accuracy with the theoretical bound. 9 Reliability evaluation. Computing an optimal estimate alone is not sufficient in real applications; we must at the same time evaluate the reliability of the resulting estimate in quantitative terms. If a robot does

1.1.

The Aims of This Book

3

not know about the reliability of the 3-D model according to which it is operating, the robot is unable to judge if its next action can achieve the given task within the required level of accuracy. For example, the knowledge that the distance to an object is optimally estimated to be 5m is of little use if the robot does not know whether the value is reliable within +10cm or + l m .

9 E~ciency of computation. An efficient numerical algorithm must be available for computing optimal estimates and evaluating their reliability. In choosing algorithms, we must also take into consideration various computational issues other than efficiency, such as numerical instability due to rounding and approximation errors involved in the computation. 9 Plausibility of the model. We need a criterion to judge if the assumed structure of the problem on which the estimation is based is correct. Given a sequence of points, for example, we can optimally fit a line to them by an efficient algorithm and evaluate the reliability of the resulting fit, provided the points are assumed to be collinear in the absence of noise. How can we confirm such a presumption? In the past, the traditional statistical approach has been the main tool in dealing with noisy data, and various statistical techniques have been employed in many forms. However, they are mostly used to make qualitative decisions and judgments [35, 36, 77, 108, 111, 139, 204]. In such problems, probabilities are usually adjusted empirically so as to reflect subjective degrees of tolerance rather than physical noise characteristics. A typical example is a technique called probabilistic (or stochastic) relaxation [12, 151]. For numerical estimation problems, on the other hand, statistical aspects have not been considered very much. Rather, much attention has been paid to algebraic aspects--writing down equations that describe the geometric relationship between the assumed 3-D structure of the scene and the 2-D description of its projected image and then solving these equations. In the presence of noise, the equations obtained by substituting the data are often inconsistent with each other, so some kind of optimization is applied. However, the optimization criterion is chosen rather heuristically or merely for convenience of computation, and its performance is evaluated empirically by simulations using real and synthetic data. A typical example is a technique called regularization [158, 159], for which the regularizing parameter is adjusted on a trial-and-error basis. If statistical techniques are employed, they are often transplanted from textbooks on statistics. In this book, we give many synthetic and real data examples to demonstrate that conventional methods are not optimal and how accuracy improves if truly optimal methods are employed. We also show that the reliability of the resulting optimal solution can be evaluated in analytical terms.

4

Chapter 1.

Introduction

Testing the validity of the assumed model is also very crucial in computer vision and robotics applications. However, it has been customary to make judgments based on a heuristic criterion. For example, a sequence of points are judged as collinear if the residual of line fitting is smaller than an arbitrarily set threshold. In this book, we present a systematic procedure for such a judgment in rigorous statistical terms.

1.1.3

Why is a new statistical theory necessary?

The aim of this book is to give a rigorous mathematical foundation to numerical optimization problems for computing 2-D and 3-D geometric quantities from inaccurate image and sensor data. To this end, we need a new theory of statistics, because the basic premises of traditional statistics do not apply to the problems we consider in this book: what is very important in traditional statistics is not so important in our problems, while what is very important in our problems has not been much recognized by statisticians. One of the major reasons why traditional statistical theories are not suitable for our purpose is their rather narrowly defined framework of viewing statistical estimation as inferring a structure by observing multiple data. This is with a view to evaluating and comparing effects and procedures in domains that involve a large degree of uncertainty, such as medicine, biology, agriculture, manufacturing, sociology, economics, and politics [38, 42, 52, 53, 63, 115, 137, 230]. In such domains, the problem is usually translated into the mathematical language as estimating the parameters in-

volved in a probability distribution from multiple independent samples ]rom it. Although this framework is suitable in the above mentioned domains, statistical estimation problems in computer vision and robotics have many non-traditional elements. In traditional statistics, errors are regarded as uncontrollable; the accuracy of estimation can be improved only by repeated measurements. However, repeating measurements is costly. Hence, if the accuracy is the same, those methods which require a smaller number of data are more desirable. In other words, methods whose accuracy improves rapidly as the number of data increases are more desirable than those with slow increase of accuracy. Thus, the study of asymptotic properties in the limit of a large number of data has been one of the central subjects in traditional statistics [18, 137]. In such engineering domains as computer vision and robotics, where electronic sensing devices are used, errors are usually very small and called noise. Moreover, they are controllable: accuracy can be improved, for example, by using high-resolution devices and controlling the environment (lighting, dust, temperature, humidity, vibration, etc.). However, such control is costly. Hence, if the accuracy is the same, those methods which allow higher levels of noise are more desirable. In other words, methods whose accuracy improves rapidly as the noise level decreases are more desirable than those with slow

1.1.

The Aims of This Book

5

increase of accuracy. Thus, the study of the accuracy of estimation in the limit of small noise is very important. In this book, we assume that errors are small and apply first order approximations in various forms. In this sense, our analysis is essentially a perturbation theory. In many engineering domains, repeating measurements under the same condition (which is easy) often produces the same result because the sources of inaccuracy in the device and the environment are fixed (but unknown). In such a domain, the basic premise of traditional statistics that independent

samples from the same distribution can be observed as many times as desired does not hold. How can one do statistical estimation from only one set of data? One cannot take even the "sample average," which is the most fundamental statistic. The answer is, as we mentioned earlier, the use of a priori knowledge about the problem: we can make inferences from the degree to which the data deviate from the the geometric constraints that should hold if the data were accurate. We can also make use of partial information about the accuracy of the image and sensor devices. To a first approximation, the uncertainty of the data supplied by a particular measurement can be characterized by the covariance matrix inherent to the sensor, but we need not know it exactly if we incorporate a priori knowledge about the problem: it suffices to know the covariance matrix up to scale. In other words, all we need is qualitative properties, such as isotropy and homogeneity, of the error distribution. Another difference from traditional statistics is the geometric nature of our problems. Traditional statistics is mainly concerned with individual variables involved in a phenomenon. In computer vision and robotics applications, however, we are concerned with geometric objects such as points, lines, and surfaces in two and three dimensions and their interrelations such as incidence, joins, and intersections. Such geometric objects and interrelations are described by vectors, tensors, and manifolds. Moreover, all procedures for statistical estimation must be written in a form that is invariant to coordinate transformations: the estimate obtained by applying a coordinate transformation to the data must coincide with the value obtained by applying the same coordinate transformation to the estimate. For example, the problem of fitting a line to approximately linearly correlated data is called regression in statistics, but the independent variables (called abscissa variables, explanatory variables, controlled variables, predictor variables, covariate variables, and by many other names) and the dependent variables (called observed variables, data variables, response variables, outcome variables, ordinate variables, and by m a n y other names) must be distinguished. In contrast, the x and y coordinates are completely equivalent when we fit a line to approximately collinear point data; the coordinate axes are chosen merely for computational convenience, and the line to be fitted must be identical whatever coordinate system is used.

6

1.2

1.2.1

Chapter 1.

Introduction

T h e F e a t u r e s of T h i s B o o k

Theoretical accuracy bound

One of the most significant consequences of our analysis is that a theoretical bound on the attainable accuracy is given for geometric correction and parametric fitting problems in the form of the covariance matrix of the parameters to be estimated. The bound is first derived in terms of the covariance matrices of the data by assuming that the noise is Gaussian. Later, a rigorous mathematical proof is given for general non-Gaussian noise, for which the Fisher information matrix plays the role of the covariance matrices of the data. The theoretical bound we derive corresponds to what is known as the Cramer-Rao lower bound in statistics, but our treatment is very different from traditional statistics. It is well known in statistics that as the number N of data increases, the variance/covariance of an estimate generally converges to zero at a rate of O(1/N). Hence, the bound on accuracy is defined by the asymptotic limit of N times the variance/covariance as N ~ c~. However, sophisticated mathematical arguments are necessary for rigorously defining various types of convergence (week convergence, strong convergence, etc.). In this book, we fix the number N of data and analyze the first order behavior in the limit of small noise. Hence, the bound on accuracy is defined as the limit of 1/e 2 times the variance/covariance as e --+ 0 for an appropriately defined noise level e. Such a perturbation analysis does not involve intricate mathematical subtleties about convergence. W i t h our perturbation approach, it can be shown that the ubiquitous least-squares optimization is not optimal; statistical bias exists in the leastsqum'es solution in general. We can also show that the theoretical bound can be attained by maximum likelihood estimation in the first order if the distribution of the noise belongs to the exponential family, in which the Gaussian distribution is a typical example. Just as m a x i m u m likelihood estimation is given a special status in the traditional asymptotic theory of statistics, it also plays an important role in our perturbation analysis.

1.2.2

Evaluation of reliability and testing of hypotheses

The fact that the reliability of m a x i m u m likelihood estimation can be evaluated quantitatively implies, for example, that when we fit a line to a sequence of point data, we can compute not only an optimal fit but also the probability that the fitted line deviates from the true position by a specified amount. This fact provides a means to visualize the most likely deviations. In this book, we present two visualization techniques--the standard confidence region and the primary deviation pair. They are multi-dimensional analogues of the "confidence interval" for point estimation in statistics. Being able to predict the probability of likely deviations means that we

1.2.

The Features of This Book

7

can derive a technique for testing hypotheses. Namely, if a prediction based on a hypothesis is not consistent with its observation to a significant degree, the hypothesis is rejected. For example, when we apply a minimization scheme for fitting a line to a sequence of points which are supposedly deviated from their original linear configuration by image noise, we can compute the probability distribution of the residual of the minimization. If the actual residual value is very large as compared with the predicted noise level of the sensor, the hypothesis that the original configuration of the points is linear is rejected. In this book, we give a rigorous statistical criterion for testing geometric hypotheses in the form of the X~2 test. However, such a .~ test is effective only when the accuracy of the sensor can be estimated a priori. If no information about the noise is available, we cannot tell whether the hypothesis is violated because of the noise or the hypothesis itself is wrong. Also, we need to set the significance level for doing a X2 test, but no theoretical basis exists for its choice (5% or 1%?). In this book, we show that the goodness of the assumed geometric constraint, or model, can be measured by the geometric information criterion obtained by modifying the AIC widely known in statistics [2]. By using the geometric information criterion, we can compare the plausibility of two models without introducing any arbitrarily set threshold.

1.2.3

Geometric models as manifolds

Parametric fitting may sound merely like fitting lines and surfaces to point data, but it has a much wider meaning. In this book, we primarily deals with such simple geometric primitives as points, lines, planes, conics, and quadrics in two and three dimensions, but our theory can be applied to any primitives as long as they can be specified by coordinates and parameters. In fact, any primitive can be viewed as a point in an appropriately defined parameter space; our theory holds in any dimensions. A relationship between primitives can be identified with a parameterized manifold in an abstract parameter space; we call such a manifold a model. The goal of parametric fitting is to find an optimal model that best fits the data points that represent individual primitives. The criterion of optimality is given in terms of the Mahalanobis distance in the parameter space. Thus, inferring a true configuration that satisfies a constraint such as incidence, coincidence, collinearity, concurrency, coplanarity, parallelism, or orthogonality can be viewed as parametric fitting. From this viewpoint, the problem of 3-D reconstruction from stereo and motion images and optical flow is also parametric fitting: a 3-D structure is reconstructed by fitting a relationship called the epipolar constraint [47, 90, 135, 227]. Once a relationship is inferred by parametric fitting, the next stage is geometric correction. For example, after a line is fitted to non-collinear points, the individual points are optimally moved onto the fitted line. If the epipolar

8

Chapter 1.

Introduction

geometry is fitted to stereo or motion data, they are optimally corrected so as to satisfy the fitted geometry. Mathematically, this problem can be viewed as an optimal projection of points that represent primitives onto a manifold in an abstract space, and the criterion of optimality is given in terms of the Mahalanobis distance. This type of geometric correction not only increases the accuracy of 3-D reconstruction but also allows us to evaluate its reliability, since the amount of such correction indicates the degree of inaccuracy of the data. Geometric correction is also important if we adopt a computational technique called linearization for 3-D motion analysis. It has been widely believed that this technique sacrifices accuracy at the cost of computational convenience [189, 191, 207]. In this book, we point out that linearization does not reduce the accuracy if the geometric correction is optimally applied. In the past, attempts have been made to view motion and optical flow analysis as traditional statistical estimation and apply standard statistical techniques [222, 233]. However, explicit and analytically closed expressions are difficult to obtain because what are known in statistics as nuisance parameters are involved. Viewing the problem as parametric fitting and geometric correction in an abstract space, we can systematically derive an optimal estimation criterion, an efficient numerical scheme, a theoretical bound on accuracy, and statistical tests for various types of hypotheses. In this respect, our approach is very different from all existing treatments of motion and optical flow analysis.

1.2.~

Numerical schemes for optimization

Presenting a mathematical theory is not the only purpose of this book. Giving consideration to making the theory applicable in real situations is also a major theme. The use of the rank-constrained generalized inverse to prevent numerical instability and increase the robustness of the solution is one example. Numerical schemes for computing an optimal solution are also studied in detail. Since an optimal solution is usually given by nonlinear optimization, which is time-consuming if solved by numerical search, we devise two simplified schemes for computing an approximately optimal solution: the optimal filter and renormalization. The latter plays a major role in this book. The renormalization procedure requires no a priori knowledge of the noise level; it is estimated a posteriori as a result of renormalization. It consists of iterated computations of eigenvalues and eigenvectors and bias-correction procedures. The accuracy of renormalization is shown to attain the theoretical bound in the first order. Many simulations and real image examples are given to show that renormalization is superior to conventional methods.

1.3.

Organization and Background

1.3 1.3.1

9

Organization and Background Fundamentals of linear algebra

One of the main characteristics of our analysis is the geometric nature of the problem: the data are vectors and tensors in two and three dimensions that represent 2-D and 3-D objects. Hence, linear algebra plays a fundamental role. Chapter 2 summarizes vector and matrix calculus, the eigenvalue problem, the singular value problem, the generalized inverse, and matrix and tensor algebra. Since the materials presented here are well established facts or their easy derivatives, theorems and propositions are stated without proofs. Many of them are proved in Kanatani [90] in the form of the answers to the exercises. The generalized inverse that will be used in this book is only of the MoorePenrose type (see [171] for other types). Readers who want a more advanced treatment of 3-D rotations and related subjects, such as Lie groups and Lie algebras, are advised to read Kanatani [85]. Along with fundamentals of linear algebra, topics concerning linear equations and optimization are also presented in Chapter 2: techniques for robustly solving indeterminate or ill-posed equations are summarized, and analytical solutions are derived for least-squares optimization, constrained quadratic optimization, and optimal rotation fitting. The optimal rotation fitting problem has been studied by many researchers in many different forms [11, 67, 69, 90, 93, 212]. The solution given in Chapter 2 is based on Kanatani

[90, 93]. 1.3.2

Probabilities and statistical estimation

Chapter 3 summarizes basic concepts of probability and statistics such as mean, variance, covariance, Gaussian distribution, X2 distribution, and the X2 test. Also, various principles of statistical estimation including maximum likelihood estimation and maximum a posteriori probability estimation (Bayesian estimation) are described by assuming that the noise is Gaussian. Then, the (discrete-time) Kalman filter is derived by applying maximum a posteriori probability estimation to a linear dynamical system with Gaussian noise. The Kalman filter is often defined as minimum mean square estimation by means of orthogonal projection, since this formalism can be applied to nonGaussian processes as well. However, minimum mean square estimation is identical with maximum a posteriori probability estimation if all variables are Gaussian, and the derivation of the Kalman filter is much easier for Gaussian processes. The details of the Kalman filter as well as the (continuous-time) Kalman-Bucy filter are left to the literature [10, 29, 33, 54, 74, 78, 79, 138, 140]. Chapter 3 also gives a general formulation of statistical estimation that does not depend on Gaussian properties. First, the Cramer-Rao lower bound

10

Chapter 1.

Introduction

is derived for the covariance matrix of an arbitrary unbiased estimator in terms of the Fisher information matrix. Then, asymptotic properties of the maximum likelihood estimator and the role of the exponential family of distributions are discussed without going into details. Finally, we derive the AIC (Akaike information criterion) that measures the goodness of a statistical model. The AIC was proposed by Akaike [2] and has been widely used for selecting a statistical model for a given random phenomenon without introducing an arbitrarily set threshold such as the significance level of the X2 test. Our analysis has two non-traditional elements. Firstly, we consider probability distributions over a manifold defined by geometric constraints, so fundamentals of manifolds and tangent spaces are briefly summarized. Detailed discussions, transversality in particular, are found in books on catastrophe theory [166, 201]. Statistical treatment of random variables that are constrained to be in a manifold is very difficult even for a very simple one such as a sphere in three dimensions [82, 129]. In order to simplify the argument, we consider only local distributions, assuming that the noise is very small. This is related to the second feature of our analysis: we are mainly interested in the statistical behavior of estimates in the limit of small noise as opposed to their behavior in the limit of a large number of data, which is the main concern of traditional statistical analysis. There exists a vast of amount of introductory literature on probability and statistics. Classical textbooks include [37, 38, 49, 52, 53, 62, 63, 104, 107, 115, 170, 230]. Statistical analysis involving generalized linear models, the Fisher information matrix, and the exponential family of distributions can be found in [42, 137]. Recently, much attention of statisticians has been drawn to geometric treatments of statistical distributions" a parameterized probability density is identified with a point in the parameter space, and statistical problems are interpreted in such geometric terms as Riemannian metrics and affine connections. This approach is called statistical geometry [7, 14, 15, 143]. Although this book also takes a geometric approach, we do not need such sophisticated concepts: all we need is tangent spaces to manifolds and projection operations onto them.

1.3.3

Representation of geometric primitives

Chapter 4 discusses mathematical representations of basic geometric primitives in two and three dimensions such as points, lines, planes, conics, and quadrics. These primitives are the main ingredients of projective geometry [90, 183, 184]. However, the representations established in projective geometry are defined with a view to making the mathematical treatment consistent without allowing any anomalies. For example, ordinary points and ideal points (points at infinity) are treated in the same way in terms of homogeneous coordinates. Hence, two lines always meet at a single (ordinary or ideal) point in

1.3.

Organization and Background

11

two dimensions. Another characteristic of projective geometry is its duality: points and lines are treated in an identical manner in two dimensions, and so are points and planes in three dimensions. This implies that theorems and statements concerning points, lines, and planes are automatically extended to their duals. In engineering applications such as computer vision and robotics, ordinary objects and ideal objects need to be distinguished, because only ordinary objects can be measured by sensors in real environments. Also, the duality concerning points, lines, and planes does not hold in real environments, because the error behavior is different for an object and its dual. In image analysis, for example, feature lines are defined by edge detection followed by line fitting, while feature points are usually defined as the intersections of detected feature lines. Since the error behavior of lines thus defined and the error behavior of points computed from them are very different, the mathematical elegance of projective geometry is destroyed, and representations of geometric primitives should reflect the way they are computed from sensor data. In order to describe geometric primitives and their relationships in real environments, one must therefore reformulate projective geometry from a computational point of view. This was done by Kanatani [86, 90, 91, 94, 95], who called the resulting formalism computational projective geometry. The treatment in Chapter 4 is a hybrid of projective geometry and real computation, but the emphasis is shifted more toward computational aspects than toward the role of projective geometry. Moreover, the same primitive is often given multiple representations; one is useful for error analysis, while another is useful for describing geometric relationships to other primitives. Almost all the representations used in this book have inherent constraints in one way or another, and the number of parameters that specify a primitive is generally larger than its true degrees of freedom. As a result, error behavior is described by a singular covariance matrix having a null space, and dealing with null spaces is one of the main characteristics of our analysis. In Chapter 4, we list formulae to convert expressions for geometric properties and error behavior from one representation to another and from one coordinate system to another. Then, perspective projection is introduced as a physical interpretation of the homogeneous coordinates. This is the key relationship between 3-D objects and their 2-D descriptions. In order to apply 3-D analysis based on perspective projection to real camera images, the camera is assumed to be calibrated, i.e., its imaging geometry is known mid modeled as perspective projection. Techniques for camera calibration are found in [13, 32, 44, 56, 59, 57, 88, 98, 110, 116, 117, 136, 156, 199, 210, 213, 214, 215, 218, 223]. Finally. we give a brief account of conics and quadrics. A conic (also referred to as a conic locus or conic section) is a plane curve defined by a quadratic equation in the coordinates. Conics are classified into ellipses (including circles), parabolas, hyperbolas, and their various degeneracies. If a

12

Chapter 1.

Introduction

robot is to operate in an industrial environment (say, in a nuclear power station), it must recognize gauges, meters, dials, and handles, most of which are circular, and circles are perspectively projected into ellipses. Hence, ellipses, or conics in general, are widely recognized as one of the most fundamental features in the study of computer vision and robotics [50, 169, 121, 178]. Detected conics can provide not only clues to object recognition; if the observed conic in an image is known to be a perspective projection of a conic in the scene of a known shape, its 3-D geometry can be computed analytically [51, 90, 97, 128, 180, 181]. A quadric is a space surface defined by a quadratic equation in the coordinates. Quadrics are classified into ellipsoids (including spheres), paraboloids, hyperboloids, and their various degeneracies. As compared with conics, the role of quadrics appears to be minor in computer vision and robotics applications, since ellipsoidal objects are rare in real environments. However, quadrics play an important role for visualizing the reliability of computation in three dimensions. We also present a technique for visualizing the reliability of objects that have more than three degrees of freedom by means of the

primary deviation pair. In Chapter 4, we only deal with points, lines, planes, conics, and quadrics in two and three dimensions, but this is simply because they are the most frequently encountered primitives in robotics applications. Indeed, the statistical estimation theory described in this book can be applied to any primitives if they are identified with points in an abstract parameter space and their interrelationships are represented by manifolds in that space.

.3.4

G e o m e t r i c correction

Chapter 5 discusses the geometric correction problem in general. A typical problem is to move data points so that they are on a particular line, curve, plane, or surface. Such a correction is necessary if the points we are observing are known to be on a specific line, curve, plane, or surface in the absence of noise. A naive idea is to project each point to the "nearest point" on the line, etc. However, this can be justified only when the noise distribution is isotropic. Otherwise, the correction should be such that it compensates for the most likely deviation. This problem is not limited to points; it can also apply to lines, planes, and more complex objects for imposing geometric constraints on their configurations. If the noise is Gaussian, an optimal solution is obtained by minimizing the Mahalanobis distance" the solution is the "nearest point" on the constraint surface in the parameter space measured in the Mahalanobis distance. We derive an explicit expression for the optimal solution, for which the noise level need not be known: it can be estimated a posteriori by analyzing the statistical behavior of the residual of the minimization. We also show that in the course of computing an optimal solution, its a posteriori covariance

1.3.

Organization and Background

13

matrix can be is automatically evaluated. If the noise level can be predicted in advance, we can test the existence of the constraint" the hypothesis that the observed object is in a certain configuration is rejected if the residual of the minimization is very large as compared with the predicted noise level. We formalize this process as the X2 test. After giving a general theory, we apply it to the problem of imposing coincidence and incidence on points, lines, conics, and planes. We also study the problem of imposing orthogonality on orientations. Applications of this type of geometric correction in reconstructing 3-D structures from stereo images for robot navigation are found in [161, 162, 163, 165].

1.3.5

3-D computation by stereo vision

In Chapter 6, we study stereo vision--3-D reconstruction by triangulation from two (or more) images. Since the principle is very simple, stereo vision has become one of the most widely used means of 3-D sensing for autonomous manipulation and navigation by robots. In the past, studies of stereo vision have been almost entirely concentrated on matching of the two stereo images. This is because establishing correspondences between the two images is a very difficult task to automate efficiently, and many matching techniques have been proposed and tested [47, 57, 227]. In contrast to matching, 3-D reconstruction appears rather trivial. However, since image data contain noise, the reconstructed 3-D structure is not necessarily accurate. Hence, it is very important to evaluate the reliability of the reconstructed 3-D structure. Otherwise, robots are unable to take appropriate actions to achieve given tasks with required precision. This issue has been discussed by only a few researchers (e.g., see [19]). We concentrate on this issue and evaluate the covariance matrices of the reconstructed points and lines by applying the theory of geometric correction given in Chapter 5. In Chapter 6, we first discuss the epipolar constraint of stereo vision by introducing the concepts of epipoles and epipolars and show that the degree to which the required epipolar equation is not satisfied provides an estimate of the noise level, from which the reliability of the reconstructed 3-D structure can be evaluated. This result is based on Kanazawa and Kanatani [101]. It turns out that this analysis is equivalent to introducing a Riemannian metric into the 3-D scene by projecting the direct product of two images onto the 3-D manifold defined by the epipolar equation. Applying the theory of geometric correction given in Chapter 5, we also present an optimal scheme for imposing the constraint that the reconstructed points and lines must be on a specified planar surface. Finally, we analyze the errors in 3-D reconstruction due to camera calibration errors (e.g., see [39, 80, 186] for stereo camera calibration).

14

1.3.6

Chapter 1.

Introduction

Parametric fitting

In Chapter 7, we study methods for fitting a geometric object to multiple instances of another geometric object in an optimal manner in the presence of noise. A typical example is fitting a line to a sequence of points. Line fitting is one of the most important steps in computer vision, and various techniques for it have been studied in the past [81,106, 219, 235]. This is because the first step of image analysis is detecting edges, i.e., sequences of pixels constituting boundaries that separate objects from the background. They are detected by applying a filter called an edge operator. Since man-made objects in robotic workspaces usually have linear boundaries, many objects in the image can be located by fitting straight lines to detected edge pixels. Then, their shapes and locations in the scene are inferred by various technique, e.g., by computing vanishing points and focuses of expansion [24, 90]. Since errors in line fitting propagate to the final 3-D reconstruction, the reliability of the reconstructed 3-D shape can be evaluated if the reliability of line fitting is quantitatively evaluated. In Chapter 7, we analyze the statistical behavior of a line fitted to a sequence of edge pixels. See [23, 87, 88, 91, 94, 187, 220] for various types of statistical inference based on line fitting errors. The theory in Chapter 7 is not limited to line fitting. We generalize the theory of parametric fitting to deal with arbitrary geometric objects in abstract terms. The goal is not only obtaining an optimal fit but also evaluating its reliability in statistical terms. First, the criterion for parametric fitting is derived as maximum likelihood estimation by assuming that the noise is Gaussian. The covariance matrix of the resulting estimate is evaluated, and the statistical behavior of the residual of optimization is analyzed. This analysis leads to the X2 test for the hypothesis that the observed objects are in a special configuration. Then, we study various types of fitting problem in two and three dimensions such as finding an optimal average, estimating a common intersection, and fitting a common line or plane.

1.3.7

Optimal filter and renormalization

In Chapters 8 and 9, we study numerical methods for efficiently computing the optimal solution of the parametric fitting problem. In Chapter 8, we construct a filter that optimally updates the estimate each time a new datum is read; we call it simply the optimal filter. The update rule is derived for Gaussian noise by adopting the Bayesian standpoint and applying maximum a posteriori probability estimation. Various assumptions and approximations involved in the derivation m'e elucidated, and the philosophical implications of the Bayesian approach are also discussed. The update rule is simplified by introducing the effective gradient approximation. The Kalman filter was originally derived for dynamical linear systems, but it can also be applied to nonlinear dynamical systems by introducing

1.3.

Organization and Background

15

linear approximation into the system equations. The resulting filter is known as the extended Kalman filter [10, 29, 33, 54, 140]. In Chapter 8, we show that the optimal filter is also obtained from the extended Kalman filter if the parametric fitting problem is identified with a nonlinear dynamical system with no state transition (i.e., a nonlinear "static" system). For this reason, the optimal filter described in Chapter 8 is often called the "extended Kalman filter" [61, 161, 162, 163, 165]. Porrill [164] pointed out the fact that the extended Kalman filter yields a statistically biased solution if the standard procedure for linear approximation is applied. He also proposed a method for correcting the bias, which is a special version of the effective gradient approximation described in Chapter 8. Chapter 9 focuses on the parametric fitting problem for linear equations. To a first approximation, the problem reduces to least-squares fitting, for which the solution can be obtained by solving an eigenvalue problem; we call the resulting solution the eigenvector fit. We first show that this leastsquares approximation introduces statistical bias into the solution. After analyzing the statistical bias in quantitative terms, we present a simple correction scheme for cancelling this bias; we call the resulting solution the unbiased eigenvector fit. In order to compute the unbiased eigenvector fit, however, we need to estimate the noise level precisely, which is usually difficult. In order to avoid this difficulty, we present a scheme for computing an unbiased solution, called the generalized eigenvector fit, without assuming a priori knowledge of the noise level. The computation is further simplified into an iterative form called renorrealization. In contrast to the optimal filter, no initial estimate is necessary for renormalization. The noise level is estimated a posteriori as a result of renormalization, and the covariance matrix of the computed estimate is automatically obtained. We then discuss a procedure called linearization, which allows us to apply renormalization to nonlinear equations. Finally, we define second order renormalization that removes statistical bias up to higher order terms.

1.3.8

Applications of geometric estimation

Chapter 10 illustrates the renormalization technique by solving the problems of line fitting and conic fitting in two dimensions and plane fitting in three dimensions. The computation for line fitting by renormalization is straightforward. If the image noise is uniform and isotropic, the least-squares scheme gives an optimal fit. This means that accuracy is not increased by the use of renormalization any further. However, the advantage of renormalization is that in the course of the fitting computation the reliability of the fit is automatically evaluated in the form of the covariance matrix. We will demonstrate this by showing the standard confidence region and the primary deviation pair that visualize the reliability of the solution computed from simulations and

16

Chapter 1.

Introduction

real image data. Next, we study the problem of conic fitting. As pointed out earlier, conics are important image features for computer vision, and if the observed conic in an image is known to be a perspective projection of a conic in the scene of a known shape, its 3-D geometry can be computed analytically [51, 90, 97, 121, 128, 178, 180, 181]. In order to do such an analysis, the conic must be given a mathematical representation in advance. This is done by applying an edge operator to a gray-level image, detecting object boundaries as edge segments, and fitting a quadratic curve to those edge segments which supposedly constitute a boundary of a conic region. Numerous conic fitting techniques have been proposed in the past [4, 20, 21, 34, 41, 45, 55, 145, 155, 167, 179, 182], but most of them are least-squares schemes with different parameterizations and criteria, and little consideration has been given to the statistical behavior of image noise. An exception is Porrill [164], who devised an iterative filtering scheme, which he called the "extended Kalman filter." Pointing out the existence of statistical bias in the solution if the filter was formulated naively, he proposed a bias correction procedure, which is equivalent to the "effective gradient approximation" introduced in Chapter 8. Kanatani [92] presented a prototype of renormalization, from which the formulation in this book has evolved. As in the case of line fitting, the reliability of the fit is evaluated in the course of the the fitting computation. Finally, we study the problem of fitting a planar surface to two types of data. Planar surface fitting is a very important process in an indoor robotic workspace, where many objects, including walls, ceilings, and floors, have planar surfaces. First, we assume that the 3-D positions of feature points that are known to be coplanar in the scene are measured by a range finder. In this case, the uncertainty of the data has a special form, according to which the covariance matrix of the data is modeled. The computation is straightforward, and the reliability of the fit is automatically evaluated in the form of the covariance matrix in the course of the fitting computation. Line and plane fitting to range data has been studied by many researchers in the past [22], but the reliability of the fit has been evaluated only in an ad hoc manner [197]. The method described here is based on Kanazawa and Kanatani [102]. Next, we consider the problem of reconstructing a planar surface from stereo images of feature points that are known to be coplanar in the scene. An indirect but straightforward method is first computing the 3-D positions of individual feature points separately and then fitting a planar surface to them. However, it is expected that the reliability of the 3-D reconstruction can be enhanced if the knowledge that the feature points are coplanar is incorporated from the beginning. With this motivation, we optimally reconstruct a planar surface directly from stereo correspondence pairs. Applying the theory of geometric correction given in Chapter 5, we optimally correct corresponding

1.3.

Organization and Background

17

pairs of feature points in the stereo images in such a way that they define exactly coplanar points in the scene. Then, the parameters of the plane on which they lie are computed by renormalization. Here again, the reliability of the fit is evaluated in the course of the fitting computation. This result is based on Kanazawa and Kanatani [100].

1.3.9

3-D motion analysis

Mathematical analysis of 3-D rigid motion estimation, known as shape (or structure) from motion, was initiated by Ullman [211], who presented a basic mathematical framework that has had a lasting influence over the subsequent computer vision research. Roach and Aggarwal [177] applied this framework to real images and obtained the solution by numerical search. Nagel [144] presented a semi-analytical formulation, reducing the problem to solving a single nonlinear equation. A complete analytical solution for eight feature points was independently given by Longuet-Higgins [122] and Wsai and Huang [207]. The solution of Longuet-Higgins was based on elementary vector calculus, while the solution of Tsai and Huang involved singular value decomposition. Zhuang et al. [238] combined them into a simplified eight-point algorithm. Zhuang [236] also discussed the uniqueness issue. All these algorithms first compute the essential matrix from the epipolar equation and then compute the motion parameters from it. This technique is called linearization, and algorithms that use it are called linearized algorithms. Huang and Faugeras [73] pointed out that a matrix can be an essential matrix for some motion if and only if it has singular values 1, 1, and 0. Linear algorithms compute the essential matrix without considering this decomposability condition. Since the essential matrix has five degrees of freedom, a 3-D interpretation can be determined, in principle, from five feature points. Using a numerical technique called the homotopy method, Netravali et al. [150] showed the existence of at most ten solutions. Arguing from the standpoint of projective geometry, Faugeras and Maybank [48] also showed that at most ten solutions can be obtained from five feature points. They reduced the problem to solving an algebraic equation of degree ten and solved it by symbolic algebra software. Using the quaternion representation of 3-D rotation, J erian and J ain [75] reduced the problem to solving the resultant of degree 16 of a pair of polynomials of degree 4 in two variables and computed the solution by symbolic algebra software. Other proposed techniques include [113, 103]. Jerian and Jain [76] exhaustively reviewed algorithms known by that time and compared their performances for noisy data. However, all these algorithms are constructed on the assumption that all data are exact. Hence, they are all fragile in the sense that inconsistencies arise in the presence of noise (e.g., the solution becomes different, depending on which of the theoretically equivalent relationships are used). A noise robust algorithm was presented by Weng et al. [226], who estimated the essential

18

Chapter 1.

Introduction

matrix from the epipolar equation by least squares and then computed the motion parameters by least squares. Spetsakis and Aloimonos [188, 189, 190] applied direct optimization to the epipolar equation without computing the essential matrix. Although error analyses have been given for 3-D motion analysis by several researchers, most of the studies were empirical and qualitative, e.g., estimating approximate orders of errors and conducting simulations with noisy data [112, 157]. A notable exception is Weng et al. [226], who analyzed the perturbation of the essential matrix and the resulting motion parameters in detail. The fact that the least-squares solution based on the epipolar equation is statistically biased has also been recognized [3, 189, 190] and analyzed in detail by Kanatani [89, 90]. Spetsakis [191] conducted a statistical error analysis on a heuristic basis. Weng et al. [222] presented an optimal algorithm by identifying the problem with a traditional statistical estimation problem. Planar surface motion has also been studied by many psychologists from the viewpoint of human visual perception, and the fact that multiple 3-D interpretations are possible was pointed out by Hay [58] as early as in 1966. A rigorous mathematical analysis was given by Tsai and Huang [206, 208] and Tsai et al. [209] by using singular value decomposition. A complete solution was presented by Longuet-Higgins [125], and errors analysis was done by Weng et al. [221]. These results are summarized in Kanatani [47, 90, 135, 227]. Hu and Ahuja [72] extended the analysis to multiple planar surface images. Chapter 11 presents a statistically optimal algorithm for computing the 3-D camera motion and the object shape from corresponding points over two views. We first study the case in which the feature points in the scene are in general position and then the case in which they are known to be coplanar. Our analysis is distinct from all existing studies in many respects. First of all, the reliability of the computed motion parameters and the reconstructed depths is evaluated in the form of their covariance matrices, and the theoretical bound on their attainable accuracy is obtained in an explicit form. This is made possible by viewing 3-D motion analysis as a correction and fitting problem and applying the theories established in Chapters 5 and 7. This viewpoint is in a sharp contrast to that of Weng et al. [222], who treated the problem as a traditional statistical estimation problem. A germ of our approach was found in the work of Trivedi [205], but his idea has not been noticed very much: the present theory is in a sense a full-fledged version of his idea. The analysis of general motion given here is based on Kanatani [96], and the analysis of planar surface motion is based on Kanatani and Takeda [99]. Since we first compute the essential matrix and then decompose it into the motion parameters, our algorithm can be classified as a linear algorithm. In the past, the linearization technique has often been rejected as having poor accuracy as compared with direct nonlinear optimization. We point out that this is because statistical aspects have not been considered fully" we demon-

1.3.

Organization and Background

19

strate that linearization does not reduce the accuracy if the essential matrix is optimally computed and optimally corrected. The renormalization procedure produces not only a statistically optimal estimate of the essential matrix but also its covariance tensor, according to which the computed essential matrix is optimally corrected so as to satisfy the decomposability condition. Another new viewpoint given in Chapter 11 is the introduction of two types of statistical test. One is the rotation test. Prior to 3-D analysis, we need to test if the camera motion is a pure rotation, in which case no 3-D information is obtained. This is done by first hypothesizing that the camera motion is a pure rotation and then testing whether the observed images support that hypothesis to a statistically admissible degree. Since no 3-D information is obtained if the camera motion is a pure rotation, the degree to which the observed images support the rotation hypothesis can be viewed as defining the information in the motion images. By computing the information in observed images, we can predict the degree of robustness of the resulting 3-D reconstruction. The other test is the planarity test" we need to test if the object is a planar surface, in which case a separate algorithm must be used. In the past, various ad hoc criteria have been used for this test. For example, since the algorithm for general motion breaks down if the object is exactly planar, it has been customary to switch to the planar surface algorithm only when computational dij~culties are encountered (e.g., if theoretically nonzero quantities approach zero), and the judgment about this has been made arbitrarily. Our approach is the opposite: we first use the planar surface algorithm and switch to the general motion algorithm only when the assumption of planarity is questioned. We make this judgment by the ;~2 test, hypothesizing that the object is a planar surface and testing if the observed data support that hypothesis. 3-D interpretation of the scene becomes more reliable and more realistic if a sequence of image frames is incorporated [71,198]. However, the discussion in this book is limited to two-view analysis, because multi-view analysis raises many mathematical and technical problems that are outside the scope of this book. One approach is the use of the extended Kalman filter with non-linear internal dynamics (the optimal filter we discuss in Chapter 8 has no internal dynamics); see [25, 26, 27, 28, 47, 114, 131, 164, 227, 231]. Thomas et al. [202] presented a related statistical analysis. Another approach includes considering the physics of the 3-D motion in question [225] and using stereo image sequences [224, 234]. Tomasi and Kanade [203] proposed a 3-D analysis method called factorization for a sequence of orthographically projected images. Chapter 11 includes a brief discussion about the critical surface that gives rise to ambiguity of 3-D interpretation. Longuet-Higgins [123] pointed out that all linear algorithms for determining the essential matrix fail to yield a unique solution if and only if all the feature points are on a special quadric surface, which is called the weak critical surface in this book. Horn [68] studied

20

Chapter 1.

Introduction

the problem of two-camera registration, which is different from 3-D motion analysis in appearance but has essentially the same mathematical structure. He also discussed ambiguity of interpretation and showed that the critical surface must be a hyperboloid of one sheet or its degeneracy. A more direct analysis was given by Longuet-Higgins [126], who analyzed the condition that the critical surface degenerates into two planar surfaces or takes on other special forms. Further analyses were done by Negahdaripour [147, 148] and Maybank [133, 134, 135], giving detailed classifications of possible types of the critical surface. Their results are concisely summarized in Kanatani [90]. In this book, we concentrate on 3-D analysis of motion images and do not deal with image processing techniques for tracking feature points, on which research is still in progress [47, 57, 227].

1.3.10

3-D interpretation of optical flow

If a sequence of images is taken by a smoothly moving camera, the difference between two consecutive frames is very small. Small displacements of points define a dense "flow," called optical flow, on the image plane. In general, the computation of 3-D interpretation based on optical flow is likely to be sensitively affected by image noise. Also, detecting accurate optical flow is very difficult. However, optical flow has the advantage that the flow can be detected densely (usually at each pixel) over the entire image by an image processing technique, while it is in general very difficult to detect point-topoint correspondences between two images--particularly so if the interframe motion is large. Hence, the use of optical flow is expected to have practical significance only if an appropriate optimization technique is available. In Chapter 12, we first summarize the principle for optimally detecting optical flow from two gray-level images. A prototype of optical flow detection technique is due to Horn and Schunck [70]. However, it has a flaw in that it uses the smoothness constraint, which is a special case of the heuristics called regularization [158, 159]. If the detected flow is to be used for image segmentation (e.g., for separating moving objects from the stationary scene), the smoothed (or regularized) solution will often blur the motion boundaries. If the purpose is 3-D reconstruction, smoothing does not increase the accuracy of the reconstructed 3-D shape, because smoothing is essentially interpolation in the image. Rather, we should reconstruct only those 3-D points which produce reliable optical flow. We should then interpolate them in the scene when and only when some knowledge about the true shape of the object (e.g., planarity) is available. In Chapter 12, we apply the theory of parametric fitting given in Chapter 7 to optical flow detection. However, since research on optical flow detection is still in progress [47, 57, 227], we do not go into details. A comprehensive review of existing techniques is given in [16]. The discussion in this book is limited to a single optical flow image observed by one camera in motion. 3-D analysis from stereo optical flow images observed

1.3.

Organization and Background

21

by two or more cameras in motion is found in [105, 118, 142, 216]. We then turn to 3-D reconstruction from optical flow, which has also been studied in various forms in the past. If the object is a planar surface, the solution can be computed in an analytical form although multiple solutions exist [83, 84, 124, 192]. If the object surface is expressed as a polynomial or a collection of planar patches, the problem reduces to estimating the coefficients [1, 193, 217]. An analytical solution can be obtained if spatial derivatives of the flow velocity are used [127]. Since the flow due to camera rotation is depth independent [85], thereby globally continuous and smooth, a sudden change of the flow over a small number of pixels implies the existence of a translational motion and a depth discontinuity; this is the phenomenon called motion parallaz, from which the translational velocity can be estimated [127, 172]. More systematically, the translation velocity can be determined by subtracting the effect of camera rotation in such a way that the resulting flow has a common focus of ezpansion [168]. A more direct approach is to do numerical search for minimizing the sum of the squares of the differences between the observed flow and the expected theoretical expression [30, 43, 60, 232]. Zhuang et al. [237] derived a linear algorithm similar to the corresponding finite motion algorithm. As in the case of finite motion, it has been pointed out that the solution based on least-squares optimization is likely to be systematically biased [40]. Tagawa et al. [195, 196] proposed an iterative method to remove statistical bias, which was a prototype of the renormalization procedure described in this book. Endoh et al. [46] discussed the asymptotic accuracy behavior. A theoretical bound on accuracy based on the Cramer-Rao inequality was studied by Young and Chellappa [233], who regarded optical flow analysis as a statistical estimation problem in the traditional form. Various aspects about the accuracy and robustness of the solution have been studied in many different forms [31, 141, 149, 194]. Mathematically, optical flow is simply an infinitesimal limit of a finite image motion, so all procedures for 3-D interpretation of optical flow should be obtained from those for finite motion by taking the limit as the interframe time approaches zero. In fact, the analysis of 3-D interpretation of optical flow given in Chapter 12 is exactly parallel to the finite motion analysis given in Chapter 11" we first study the case in which the object has a general shape and then the case in which the object is a planar surface. We also describe a theoretical bound on the attainable accuracy and present a computational scheme by using renormalization and linearization. The analysis given here is based on Ohta and Kanatani [154]. The rotation test and the planarity test are also discussed in the same way as in the case of finite motion. Finally, we study the error introduced by identifying the interframe displacement by a continuous flow and point out that the ratio of image noise to optical flow is a very deceptive measure for predicting the accuracy of the resulting 3-D interpretation.

22

Chapter 1.

Introduction

A brief description of the critical surface of optical flow is also given here. Its definition and geometric properties are the same as those of finite motion, but historically the existence of the critical surface was first pointed out for optical flow. Maybank [132] showed that such a surface should necessarily be a quach-ic surface passing though the viewpoint. Horn [66] introduced the term "critical surface" and showed that the critical surface must generally be a ruled quadric and hence a hyperboloid of one sheet or its degeneracy. Negahdaripour [146] exhaustively classified the possible types of critical surface. Their results are concisely summarized in Kanatani [90].

1.3.11

Information criterion for model selection

In order to apply the techniques for geometric correction and parametric fitting, one needs to know the geometric model, i.e., the constraints and hypotheses that should hold in the absence of noise. But how can one prefer one geometric model to another? The problem of model selection is very important in many engineering domains where probabilities are involved, and two approaches have attracted attention: one is the AIC (Akaike information criterion) [2]; the other is the MDL (minimum description length) principle [173, 174, 175, 176]. The AIC favors a model whose maximum likelihood estimator has a minimum residual for future data, while the MDL principle favors a model that can be described in a code of minimum length. Since a geometric model is defined as a manifold, the complexity of the model is evaluated not only by its degree of freedom but also by such invariant quantities as the dimension and codimension of the manifold. In Chapter 13, we first summarize what we call the Mahalanobis geometry of maximum likelihood estimation and derive the geometric information criterion by modifying the AIC. This criterion in its original form requires exact knowledge of the noise level. In order to avoid this, we present a comparison criterion for evaluating the goodness of one model relative to another without using any arbitrarily set threshold such as the significance level of the X2 test. We derive explicit expressions for the comparison criterion for point data in two and three dimensions, 3-D reconstruction by stereo vision, 3-D motion analysis, and 3-D interpretation of optical flow. At the same time, we elucidate the geometric structures of these problems and their interrelationships--in particular, the ambiguity and singularity of 3-D reconstruction when the object is a planar surface or the camera motion is a pure rotation.

1.3.12

General theory of geometric estimation

Chapter 14 presents a rigorous mathematical foundation for the theory of geometric correction given in Chapter 5 and the theory of parametric fitting given in Chapter 7. For both, the problem is stated in general terms without assuming Gaussian noise: the role of the covariance matrix for a

1.4.

The Analytical Mind: Strength and Weakness

23

Gaussian distribution is played by the Fisher information matrix. We derive a lower bound, which corresponds to the Cramer-Rao lower bound in traditional statistics, on the covariance matrix of the unbiased estimator of the parameter. Then, the maximum likelihood estimator is proved to attain it in the first order if the problem belongs to the exponential family. Finally, the maximum likelihood estimation process is expressed in a computationally convenient form, where the rank-constrained generalized inverse is used to deal with the ill-posedness of the problem and the numerical instability of the solution. The theme of Chapter 14 is essentially statistical estimation, but our treatment has non-traditional elements for the reasons pointed out in Section 1.1.3. The statistical problem closely related to the parametric fitting problem is what is known as the Neyman-Scott problem [152]: observing multiple data, each having a distribution characterized by a common parameter (called the structure parameter or the parameter of interest) and a distinct parameter (called the nuisance parameter) that depends on each observation, one has to estimate the structural parameter without knowing the nuisance parameters [18]. In the parametric fitting problem, the true values of image/sensor data can be viewed as nuisance parameters, since the goal is to estimate the fitting parameter that characterizes the 2-D/3-D structure without knowing the exact values of the sensor data. However, the probability densities of the sensor data do not involve the fitting parameter. Hence, various mathematical concepts and tools developed for the Neyman-Scott problem [8, 9, 17, 109, 119, 120] cannot be applied directly in the form given in the literature of statistics.

The Analytical Mind: Strength and Weakness

1.4

i. .i

Criticisms of statistical approaches

In concluding this chapter, let us note the fact that there exist strong criticisms of statistical (or in general analytical) theories of robotics and computer vision. Since any theory is essentially an idealization of reality, there is always something in the real world that the theory does not cover, and statistical theories of robotics and computer vision have often been criticized for this very reason. Strong arguments against statistical theories include the following: 9 In order to apply a statistical theory, one must model the noise in a mathematically tractable form, say, assuming that the noise is independent, uniform, homogeneous, isotropic, Gaussian, etc. However, these are mathematical artifacts" noise in reality is more or less correlated, non-uniform, non-homogeneous, non-isotropic, and non-Gaussian. Hence, one does not know how reliable the analytical results based on these assumptions are.

24

Chapter 1.

Introduction

A statistically optimal solution is usually obtained by nonlinear optimization in a very complicated form, requiring a lot of computations. However, a human can easily perceive 3-D structures using vision and can take appropriate actions with sufficient precision. It is difficult to imagine that the brain conducts complicated optimization computations, such as renormalization, each time a human sees or does something. Rather, human perception seems to be based on ingenious combinations of experiences (data) and heuristics (rules). Hence, it should be possible to build an intelligent robot that has a human's versatility without doing rigorous mathematical analysis. We should learn from humans rather than relying on mathematics. Uncertainty in the reaI world is not merely quantitative. There is always a possibility that an unexpected circumstance happens--sudden appem'ance or disappearance of an object, for example. Since it is difficult to model the uncertainty in the real world completely, it should be more realistic to adopt heuristics and avoid rigorous mathematical assumptions. Experience has shown that empirically adjusted systems often outperform those designed by precise mathematical analysis; the latter do well only when the assumptions are exactly met. Those who criticize the statistical approach advocate such non-analytical pm'adigms as artificial intelligence, fuzzy inference, and neuro-computing. In the domain of computer vision, such compromises as qualitative vision and purposive vision have been proposed as alternatives to rigorous mathematical analysis. However, just as the importance of such paradigms cannot be denied, the importance of analytical methods cannot be denied either, because advantages and disadvantages are both sides of the same coin. Rigid and inflexible as it is, an analytically designed system has the advantage that it can faithfully reveal the insufficiency of the model. Namely, its poor behavior implies the existence of new factors in the environment that are not properly modeled in the analysis, and the study of these becomes the next goal, an analysis guiding us to a new analysis. In contrast, such insufficiency is often concealed by adroit behavior of a heuristically designed system, and its longterm performance is unpredictable--it may suddenly break down when its versatility capacity reaches a limit. This is an inescapable destiny of a system that learns from humans. By a cm'eful mathematical analysis, humans can learn nmch about the mechanisms that underlie seemingly simple phenomena.

Gaussian noise ass~tmption and outlier detection In our statistical analysis, the covariance matrix plays a central role as a measure of uncertainty. This is practically equivalent to assuming that the distribution of the noise is either Gaussian or approximately Gaussian. We develop

1.4.

The Analytical Mind: Strength and Weakness

25

a general non-Gaussian theory in Chapter 14, but the noise distribution is required to share many properties with the Gaussian distribution--smoothness and unimodality, for instance. Although assuming Gaussian or Gaussian-like noise nlakes mathematical analysis very easy, this certainly sets a limitation on potential applications of our theory. A typical situation in which this Gaussian noise assumption is violated is when the noise is due to systematic bias in the data acquisition procedure. For example, lens aberration causes a systematic distortion of the image, e.g., symmetric elongation or contraction around its center. The use of wrong camera parameters estimated by poor calibration is another source of systematic errors. In processing images, error characteristics are different from operation to operation. For example, if the epipolar constraint is used for finding point correspondences between stereo images, the error characteristics are very different along the epipolar and in the direction orthogonal to it. In this book, deviations of points and lines from their supposed positions are collectively regarded as "noise" irrespective of their sources. The Gaussian noise assumption is a good approximation if a lot of independent noise sources are involved, each having a small effect. This is a consequence of the central limit theorem,. In contrast, the Gaussian noise assumption poses a difficulty if a small number of very different noise sources coexist. If only one noise source exists, the noise characteristics can be estimated by repeating measurements. If nmltiple sources exist and one source is dominant, data from a dominant source are called inliers; those from other sources are called outliers. In order to do statistical estimation, one must judge which data are inliers and which are outliers. This can be done, for example, by hypothesizing that particular data are inliers and testing that hypothesis by various statistical tests. Such a procedure is called outlier detection,. This book does not deal with outlier detection at all. This is because it requires treatments that are very difficult to generalize. However, it should be empha~sized that the theory of statistical estimation and model selection presented in this book provides a theoretical foundation for developing such techniques. See [139, 204, 219] for outlier detection in computer vision problenls.

Remaining problems In this book, we concentrate only o11 numerical computation by assuming that necessary data are provided by image processing and other sensing operations. We do not go into the details of image processing techniques involving gray levels and colors such as edge detection, stereo matching, and feature point tracking (see [47, 57, 185, 227] for these topics). We also do not consider 3-D analysis based on the gray levels of images such as shape from shading and photometric stereo (see [65]). Since the focus is on numerical computation, we do not discuss paradigms of image understanding and computer vision and

26

Chapter 1.

Introduction

relationships with human perception. See [5, 6, 12, 130, 151, 211, 229] for such arguments.

Chapter 2 F u n d a m e n t a l s of Linear Algebra This chapter presents fundamentals of linear algebra t h a t will be necessary in subsequent chapters. Also, the symbols and terminologies t h a t will be used t h r o u g h o u t this book are defined here. Since the materials presented here are well established facts or their easy derivatives, theorems and propositions are listed without proofs; readers should refer to standard textbooks on mathematics for the details.

2.1

2.1.1

Vector

and

Matrix

Calculus

Vectors and matrices

T h r o u g h o u t this b o o k , g e o m e t r i c q u a n t i t i e s such as v e c t o r s a n d t e n s o r s a r e d e s c r i b e d w i t h r e s p e c t to a Cartesian coordinate system, t h e c o o r d i n a t e axes b e i n g m u t u a l l y o r t h o g o n a l a n d h a v i n g t h e s a m e u n i t of l e n g t h 1. W e also a s s u m e t h a t tile c o o r d i n a t e s y s t e m is right-handed 2. B y a vector, we m e a n a c o l u m n of real n u m b e r s 3. V e c t o r s are d e n o t e d b y l o w e r c a s e b o l d f a c e l e t t e r s s u c h as a, b, u , a n d v; t h e i r c o m p o n e n t s are w r i t t e n in t h e c o r r e s p o n d i n g l o w e r c a s e italic l e t t e r s . A v e c t o r w h o s e c o m p o n e n t s are a l , a2, ..., an is also d e n o t e d by (ai), i - 1, ..., n; t h e n u m b e r n of t h e c o m p o n e n t s is called t h e dimension of this vector. If t h e d i m e n s i o n is u n d e r s t o o d , n o t a t i o n s such as (ai) are used. In t h e following, a n n - d i m e n s i o n a l v e c t o r is r e f e r r e d to as an n-vector. T h e v e c t o r w h o s e c o m p o n e n t s are all 0 is called t h e zero vector a n d d e n o t e d b y 0 ( t h e d i m e n s i o n is u s u a l l y i m p l i e d by the context). A matrix is an a r r a y of real n u m b e r s . M a t r i c e s are d e n o t e d b y u p p e r c a s e b o l d f a c e l e t t e r s s u c h as A , B , S , a n d T ; t h e i r e l e m e n t s are w r i t t e n in t h e c o r r e s p o n d i n g u p p e r c a s e italic l e t t e r s . A m a t r i x is also defined b y its e l e m e n t s as (Aij), i = 1, ..., m, j = 1, ..., n; s u c h a m a t r i x is said to be of type ran. In t h e following, a m a t r i x of t y p e m n is r e f e r r e d to as a n mn-matrix; if m = n, it is also called a square matrix or s i m p l y n-dimensional matrix. If t h e t y p e is ] This is only an intuitive definition, since "orthogonality" and "length" are later defined in terms of coordinates. To be strict, we need to start with axioms of one kind or another (we do not go into the details). 2In three dimensions, a Cartesian coordinate system is right-handed if the x-, y-, and z-axes have the same orientations as the thumb, the forefinger, and the middle finger, respectively, of a right hand. Otherwise, the coordinate system is left-handed. In other dimensions, the handedness, or the parity, can be defined arbitrarily: if a coordinate system is right-handed, its mirror image is left-handed (we do not go into the details). 3We do not deal with complex numbers in this book. 27

28

Chapter 2.

Fundamentals of Linear Algebra

understood, notations such a s ( A i j ) are used. The matrix whose elements are all 0 is called the zero m a t r i x and denoted by O (the type is usually implied by the context). If not explicitly stated, the type is understood to be nn in this chapter but 33 in the rest of this book. The unit m a t r i x is denoted by I; its elements are written a s (~ij (not Iij); the dimension is usually implied by the context. The symbol (~ij, which takes value 1 for i = j and 0 otherwise, is called the K r o n e c k e r delta. Addition and subtraction of matrices and multiplication of a matrix by a scalar, vector, or matrix are defined in the standard way. n The trace of nn-matrix A - ( A i j ) is the sum ~ / = 1 A i i of its diagonal elements and is denoted by trA. Evidently, t r l = n. The transpose of a vector or matrix is denoted by superscript T. A matrix A is s y m m e t r i c if A = A T. We say that a matrix is of type ( n n ) or an ( n n ) - m a t r i x if it is an n-dimensional symmetric matrix. A matrix A is antisyrnmetric (or skews y m m e t r i c ) if A = - A T. We say that a matrix is of type Inn] or [nn]-matriz if it is an n-dimensional antisymmetric matrix. Note the following expression, which is sometimes called the outer product of vectors a and b: albi a2bl .

ab T - ( a i b j ) -

alb2 a2b2 .

.'. "" .

anbl

anb2

.

albn a2bn .

.

(2.1)

.

999

anbn

The following identities are very familiar: (AT) T -

A,

(AB) T-

t r ( A T) - trA,

tr(AB)-

B T A T,

tr(BA).

(2.2)

The i n n e r product of vectors a - (ai) and b - (bi) is defined by n

(a, b) - a T b -- E

aibi.

(2.3)

i=1

Evidently, (a, b) - (b, a). Vectors a and b are said to be orthogonal if (a, b) = 0. The following identities are easily confirmed: (a, T b ) - ( T T a , b),

t r ( a b T) - (a, b).

(2.4)

9 The matrix consisting of vectors a l, a2, ..., an as its columns in that order is denoted by (al, a2, ...., an). If A -

B - (bl, b2, ..., bn),

( a l , a 2 , ..., a n ) ,

the following identities hold: Yt

AB T -

E aibT i , i--1

(2.5)

2.1.

Vector and Matrix Calculus

A_rB -

29

(al,bl)

(al,b2)

-..

(a2,bl)

(a2,b2)

""

9

.

(a,~ b l ) ( a , , b 2 )

.

.

(al,b,~) (a2, b,~) .

...

.

(2.6)

"

(a~,b~)

The n o r m 4 and the n o r m a l i z a t i o n operator N [ . ] are defined as follows"

Ilall- v/(a,a)-

•

'/~1 ._ ai 2,

a N[a]- Ilall"

(2.7)

A unit vector is a vector of unit norm. A set of vectors {Ul, ..., u,.} is said to be o r t h o n o r m a l if its members are all unit vectors and orthogonal to each other: (ui, u j ) - 5ij. The following Schwarz inequality holds:

-Ilall [[bI[ 0, e q u a l i t y h o l d i n g if a n d only if a = 0, (ii)

IIcall- Icl" Ila for any sc~,~ c, a,~d (iii) the triangle inequality (2.9) holds. There exist o t h e r definitions t h a t satisfy t h e s e - - - t h e 1 - n o r m la l1 -- ~'i-1 Jail a n d the e~-noTwn Ilall~ maxi ]ail, for i n s t a n c e . T h e y can be g e n e r a l i z e d into t h e M i n k o w s k i n o r m (or t h e p - n o r m ) y ~ n

Ilal]p -

~ / E ~ = 1 ai[P for 1 < p < ~ ; the 1-norm, t h e 2 - n o r m , a n d t h e ~ - n o r m

cases of t h e M i n k o w s k i n o r m for p -

1, 2, cx>, respectively.

are special

30

Chapter 2.

(~ij -~- cAij

Replacing Aij by

Fundamentals of Linear Algebra

in eq. (2.10) and expanding it in z, we obtain

d e t ( l + cA) = 1 + ctrA + O(c2),

(2.13)

where the order symbol 0 ( . . . ) denotes terms having order the same as or higher than --.. Let A (ij) be the matrix obtained from a square matrix A - (Aij) by removing the ith row and the j t h column. The determinant det A is expanded in the form n

n

dee A - ~ ( - 1 ) i + J A i j

det A (ij) - Z ( - 1 ) i + J A i j

i--1

det A (ij).

(2.14)

j=l

This is called the cofactor expansion formula. matrix A t = (A~j) of A is defined by

The cofactor (or adjugate)

Ai~ - ( - 1 ) i+j det A (ji).

(2.15)

Eq. (2.14) can be rewritten as

A A * = A f A = (det A ) I .

(2.16)

The following identity holds: det(A + cB) = det A + c t r ( A f B ) + O(c2).

(2.17)

The elements of the cofactor matrix A t of nn-matrix A are all polynomials of degree n - 1 in the elements of A. In three dimensions, the cofactor matrix of A = (Aij) has the following form: A12A23 - A22A13 A13A21 - A23All ) 9 AliA22 -- A21A12 (2.18) The inverse A -1 of a square matrix A is defined by

Af

-

A22A33 - A32A23 A23 A31 A 3 3 A21 A21A32 - A31A22 -

-

A32A13 - A12A33 A33A11 - A13A31 A31A12 - All A32

A A -1 = A - 1 A = I,

(2.19)

if such an A - 1 exists. A square matrix is singular if its inverse does not exist, and nonsingular (or of full rank) otherwise. Eq. (2.16) implies that if A is nonsingular, its inverse A -1 is given by At A -1 = ~ det A"

(2.20)

If we define A ~ = I, the following identities hold for nonsingular matrices (k is a nonnegative integer): (A - 1 ) - ' - A,

( A B ) -1 - B - i A -1,

(A-1)k _ ( A k ) - i

2.1.

Vector and Matrix Calculus (A-l-)_ 1 _ ( A _ I ) T

31 detA_ 1 _ 1 - det----A"

'

(2.21)

The third identity implies t h a t m a t r i x (A -1)k can be unambiguously denoted by A -k. Note t h a t the d e t e r m i n a n t and the inverse are defined only for square matrices. Let A be a nonsingular n n - m a t r i x , and B a nonsingular ram-matrix. Let S and T be rim-matrices. The following matrix inversion formula holds, provided that the inverses involved all exist" (A -~- S B T T ) -1 - A -1 - A - 1 S ( B -1 -!- T T A - 1 S ) - I T T A

-1.

(2.22)

If m - 1, the n m - m a t r i c e s S and T are n-vectors, and the r a m - m a t r i x B is a scalar. If we let B - 1 and write S and T as s and t, respectively, the above formula reduces to

A - l s t T A -1 (A + stn-) -1 - A -1 - 1 + ( t , A - l s ) "

(2.23)

For A - I , we obtain st T

(I + stT) -1 - I 2.1.3

1 + (s, t)"

(2.24)

Vector p r o d u c t in three d i m e n s i o n s

In three dimensions, the signature symbol defined by eq. (2.11) is often referred to as the Eddington epsilon 5. It satisfies the following identity:

3

E

eijmeklm - 5ikSjl- 5ilSjk.

(2.25)

m=l

The vector (or exterior) product of 3-vectors a - (ai) and b - (bi) is defined by

--s

a • b-

( a2b3-a3b2)

(j~l,= e i j k a j b k ) -

ala3blb2--a2blalb3

9

(2.26)

Evidently, axb--bxa,

(b, a • b ) -

a•

(a, a • b ) - O.

(2.27)

The following identities, known as the Lagrange formulae, are direct consequences of eq. (2.25)" a x (b • c) - (a, c)b - (a, b)c, 5Some a u t h o r s use different terminologies such as the

Levi-Civitasymbol.

32

Chapter 2.

Fundamentals of Linear Algebra

O a

a

(~)

(b)

Fig. 2.1. (a) Vector product. (b) Scalar triple product.

(a x b) x c = (a, c ) b -

(b, c)a.

(2.28)

The expressions a x (b x c) and (a x b) x c are called vector triple products. The following identities also hold:

(a x b, c x d) = (a, c)(b, d) - (a, d)(b, c),

(2.29)

Ila • bll 2-- Ilall~llbll ~- (a,b) 2.

(2.30)

If 3-vectors a and b make angle 0, we have (a, b) = Ilall" Ilbll ~os0,

lid x bll = Ilall. Ilbll sin 0.

(2.31)

Eq. (2.30) states the well-known trigonometric identity cos20 + sin20 = 1. From eq. (2.26), the third of eqs. (2.27), and the second of eqs. (2.31), we can visualize a x b as a vector normal to the plane defined by a and b; the length of a x b equals the area of the parallelogram made by a and b (Fig. 2.1a). The scalar triple product la, b, c I of 3-vectors a, b, and c is the determinant of the matrix (a, b, c) having a, b, c as its colunms in that order. We say that three 3-vectors {a, b, c} are a right-handed system if la, b, cl > 0 and a left-handed system if ]a, b, c] < 0. The scalar triple product la, b, c] equals the signed volume of the parallelepiped defined by a, b, and c (Fig. 2.1b); the volume is positive if the three vectors are a right-handed system in that order and negative if they are a left-handed system. The equality la, b, c I - 0 holds if and only if a, b, and c are coplanar, i.e., if they all lie on a common plane. We can also write

la, b,c[ = (a x b , c ) = (b x c , a ) =

(c x a , b ) .

(2.32)

Since [a, b, a x b[ - lid x bll 2, th~ vector product a • b is oriented, if it is not O, in such a way that {a, b, a x b} form a right-handed system (Fig. 2.1a). The following identity also holds: (a x b) x (c x d) = ]a, b, d]c - ]a, b,

old --- la, c, dlb - Ib, c, did.

(2.33)

2.1.

Vector and Matrix Calculus

33

Taking the determinant of (a,b, c ) ( a , b , c ) T (see eq. (2.12)), we obtain

(a, b) (a,c)

llall= (b, a) (c,a)

la, b, cl "2

llbll

(b, c)

(c,b)

llcll=

(2.34)

The vector (or exterior) product of 3-vector a mad 33-matrix T - (tl, t2, t3) is defined by a X T - (a X t l , a X t 2 , a X ta). (2.35) From this definition, the following identities are obtained: a x

a x I-

0 a3 -a2

--ct3 0 al

(Tb)-

(a

a2 ) --al , 0

x

T)b,

(a x I ) T - - - - a

X

I.

(2.36)

The matrix a x I is called the antisymmetric matrix associated with the 3-vector a. The following identity is an alternative expression to the Lagrange formulae (2.28): (a x I ) ( b x /)T (a, b ) I - ba T. (2.37) _

The vector (or exterior) product of 33-matrix T and 3-vector b is defined by T x b - T ( b x I)T.

(2.38)

This definition implies the following identities"

( T x b) T - b x T

T x a,

(a x T ) T - T

T

( T x b ) c - T ( c x b).

(2.39)

It is easy to confirm that (axT)

xb-ax

( T x b),

(2.40)

which can be written unambiguously as a x T x b. We also have (a x T x b) T -- b x T T X a.

(2.41)

Eq. (2.37) now reads a x I x b-

(a,b)l-ba

T.

(2.42)

The following identities are also important:

(axb)(cxd) m-ax (a x b , T ( c x d ) ) -

(bd T) x c - b x

(ac T ) x d,

(2.43)

(a, (b x T x d)c) - (b, (a x T x c)d).

(2.44)

34

Chapter 2.

Fundamentals of Linear Algebra

a/ a

:\

l

n

\

0

P.a

(b)

(~)

Fig. 2.2. (a) Projection onto a line. (b) Projection onto a plane. The exterior product [A x B] of 33-matrices A = (Aij) and B = (Bij) is a 33-matrix defined as follows6: 3

[A • S]ij -

E

eiklejmnAkrnBln.

(2.45)

k,l,m,n=l

If A and B are both symmetric, their exterior product [A • B] is also symmetric.

2.1.4

Projection matrices

If a vector a is projected orthogonally onto a line 1 that extends along a unit vector n, it defines on l a segment of signed length (n, a) (Fig. 2.2a); it is positive in the direction n and negative in the direction - n . The vector a is decomposed into the component (n, a)n parallel to I and the component a - ( n , a)n (= ( I - nnq-)a) orthogonal it. Let {n}L be the one-dimensional subspace defined by unit vector n, and {n}L~ its orthogonal complement--the set of all vectors orthogonal to n. The projection of a vector a onto {n}Ll is written as P n a (Fig. 2.2b). The matrix P n is defined by

P n - I - n n q-,

(2.46)

and called the projection matrix onto the plane orthogonal to n, or the projection matrix along n. The following identities are easily confirmed:

pn_pmn, det P n = 0,

p 2 = Pn,

trPn = n - 1,

I]Pn[I = x / n - 1.

Here, the matrix norm II-II is defined by Ilall -

(2.47)

E~=a E~=I A ~ for m,~-

v/ ~ matrix A = (Aij). In three dimensions, eq. (2.42) implies the following identity for unit vector n:

n • I • n-

(n • I ) ( n • I) m - Pn.

6For e x a m p l e , [A • B]11 = A22B33 - A32B23 - A23B32 + A33B22.

(2.48)

2.1.

Vector and Matrix Calculus

35

The projection matrix can be generalized as follows. Let the symbol 7~~ denote the n-dimensional space of all n-vectors. Let $ be an m-dimensional subspace of 7~~, and Af (= $ • its orthogonal c o m p l e m e n t - - t h e set of all vectors that are orthogonal to every vector in $. The orthogonal projection 7 P X onto $ is a linear mapping such that for an arbitrary vector v G T~n P x v E $,

v-

Pxv EAf.

(2.49)

In other words, P • is the operator that removes the component in Af. We also use an alternative notation p S when we want to indicate the space to be projected explicitly. Let { n l .... , n m } be an orthonormal basis of Af. The orthogonal projection P x has the following matrix expression: m

Px - I-

Z

nin i .

(2.50)

i=1

Eqs. (2.47) can be generalized as follows: exdet P~r = O, 2.1.5

Pfr, trPx

= n-

P~m,

PX, [[P~II = x / n - m.

(2.51)

Orthogonal matrices and rotations

Matrix R is orthogonal if one of the following conditions holds (all are equivalent to each other): R R y - I,

RTR-

I,

R -1 - R y.

(2.52)

Equivalently, matrix R = (7"1, ... , l " n ) is orthogonal if and only if its columns form an orthonormal set of vectors: (ri, r j ) = 5ij. For an orthogonal matrix R and vectors a and b, we have (Ra, Rb)

= ( a , b),

IIRall- Ilall,

(2,53)

The second equation implies that the length of a vector is unchanged after multiplication by an orthogonal matrix. The first one together with eqs. (2.31) implies that in three dimensions the angle that two vectors make is also unchanged. Applying eq. (2.12) to eqs. (2.52), we see that det R = • for an orthogonal matrix R. If det R = 1, the orthogonal matrix R is said to be a rotation 7The notation given here is non-traditional: the projection onto subspace S is usually denoted by Ps. Our definition is in conformity to the notation Pn given by eq. (2.46).

36

C h a p t e r 2.

F u n d a m e n t a l s of Linear A l g e b r a

r=o) x r Rr

r

1

O)

O

O

(~)

(b)

Fig. 2.3. (a) Axis and angle of rotation. (b) Instantaneous rotation.

(1) (0)(0)

matrix s. In t h r e e dimensions, t h e o r t h o n o r m a l C a r t e s i a n c o o r d i n a t e basis vectors are i -

0 0

,

j -

1 0

,

k -

0 1

.

(2.54)

T h e c o l u m n s of a t h r e e - d i m e n s i o n a l r o t a t i o n m a t r i x R = ( r l , r 2 , r 3 ) define a r i g h t - h a n d e d o r t h o n o r m a l s y s t e m { r l , r2, ra}. T h e m a t r i x R m a p s t h e c o o r d i n a t e basis {i, j , k} to { r i , r2, r3}. Such a m a p is realized as a r o t a t i o n along an axis 1 by an angle f~ of r o t a t i o n (Euler's theorem; Fig. 2.3a). T h e axis l (unit vector) a n d t h e angle f~ ( m e a s u r e d in t h e screw sense) of r o t a t i o n R are c o m p u t e d as follows:

l

N[

R13 --

/~31

],

=

cos

i

tr

1.

(2.55)

R21 -- R12

Conversely, an axis I a n d an angle ~ define a r o t a t i o n R in t h e following form: cosFt + lie(1 - cosFt) /~ --

1211 ( 1 - cos Ft) + 13 sin fi

/311(1 - cos f~) - 12 sinf~

l112 (1 -- cos f~) -- 13 sin fl cos ~ + 122 (1 -- cos fl) 1312(1 - c o s f~) + ll s i n f l Ill3(1 - cos f~) + 12 sin ~ 1213 (1 - cos ~ ) - 11 sin f~ ) . cos fl + 132 ( 1 - cos f~)

(2.56)

F r o m this equation, we see t h a t a r o t a t i o n a r o u n d unit v e c t o r / by a small angle A~2 is expressed in t h e f o r m R = I + Aal x I + O(Afl2),

(2.57)

SThe set of all n-dimensional rotation matrices forms a group, denoted by SO(n), under matrix multiplication. It is a subgroup of O(n), the group consisting of all n-dimensional orthogonal matrices. The group consisting of all nonsingular nn-matrices is denoted by GL(n), and the group consisting of all nn-matrices of determinant 1 is denoted by SL(n).

2.2.

Eigenvalue Problem

37

which implies that for a continuous rotation there exists a vector w such that Afll = w A t + O ( A t 2) for a short lapse of time At. The vector w is called the rotation velocity: Its orientation N[w] defines the instantaneous axis of rotation; its norm I1~11 d fin s the angular velocity. Eq. (2.57) reads

R = I + w x I A t + O(At2).

(2.58)

Hence, the velocity/" = linl,,xt-+o(Rr- r ) / A t of vector R r at At = 0 has the form /~ = w x r, (2.59) and is orthogonal to both r and w (Fig. 2.3b).

2.2 2.2.1

Eigenvalue Problem Spectral d e c o m p o s i t i o n

An eigenvector of an (nn)-matrix 9 A for eigenvalue )~ is a nonzero vector u such that

A u = )~u. This equation can be rewritten as ( A I solution if and only if the flmction

(2.60)

A ) u = 0, which has a non-trivial

OA()~) = I)~I- AI

(2.61)

has a zero: ~bA ()~) = 0. The function ~bA ()~) is an nth degree polynomial in )~ and called the characteristic polynomial of A. The equation CA()~) = 0 is called the characteristic equation and has n roots {)~/} (with multiplicities counted). The number of nonzero eigenvalues (with multiplicities counted) is called the rank of the (nn)-matrix A. The eigenvalues {~i } of an (nn)-matrix A are all real. The corresponding set {ui} of unit eigenvectors can be chosen to be an orthonormal set. Let us call the set {ui} so defined the eigensystem of the (nn)-matrix A. An (nn)-matrix A is expressed in terms of its eigenvalues {)~i} and eigensystem {ui } in the form

A-

Z

)~iUi ~ ,

(2.62)

i=1

which is called the spectral (or eigenvalue) decomposition of A. In particular, the identity ] , )

~iU?

-- I

(2.63)

i=1 9 Recall that by an ( n n ) - m a t r i x we mean an n-dimensional symmetric matrix (see Section 2.1.1). Eigenvalues and eigenvectors are defined for a general (non-symmetric) matrix in exactly the same way, but in this book we deal with eigenvalues and eigenvectors of symmetric matrices only.

Chapter 2.

38

Fundamentals of Linear Algebra

holds for an arbitrary orthonormal system {ui}. From (x, ~in.=.l Uilt~T,) --" (x, Ix), we obtain the following identity for an arbitrary vector and an arbitrary orthonormal system {ui}: n

~ ( ~ , ~ ) ~ - I1~11~.

(2.64)

i=1

Let {Ai} be the eigenvalues of (nn)-matrix A, and {ui} the corresponding eigensystem. Since {ui} is an orthonormal system, the matrix U = (Ul, u 2 , . . . , u . ) i s orthogonal. Eq. (2.62)is equivalent to

A-

(2.65)

U A U ~,

where A is the diagonal matrix with diagonal elements A1, A2, ..., A~ in that order; we write A = diag(A1, A2, ..., An). (2.66) From eq. (2.65), we obtain

U T A U - A,

(2.67)

which is called the diagonalization of A. Applying the fourth of eqs. (2.2) and eq. (2.12) to eq. (2.65), we obtain the following identities: rt

trA = ~

n

~,

det A = I-[ ~"

i=1

(2.6S)

i=1

From the spectral decomposition (2.62), the kth power A k for an arbitrary integer k > 0 is given by n

Ak - E Aikuiu~"

(2.69)

i=1

This can be extended to an arbitrary polynomial p(x):

p(A)- ~p(~)~uT.

(2.70)

i=1

If A is of full rank, its inverse A -1 is given by

A -1 -- ~1"= -~iltt i u T .

(2.71)

This can be extended to an arbitrary negative power of A (see the third of eqs. (2.21)):

A- ~ _ ~ i=1

1 7~~'~,

(2.72)

2.2.

Eigenvalue Problem

2.2.2

39

Generalized inverse

An ( n n ) - m a t r i x A is positive definite if its eigenvalues are all positive, and is positive semi-definite if its eigenvalues are all nonnegative; it is negative definite if its eigenvalues are all negative, and is negative semi-definite if its eigenvalues are all nonpositive. For a positive semi-definite (nn)-matrix A, eq. (2.69) can be extended to arbitrary non-integer powers A q, q > 0. In particular, the "square root" v ~ of A is defined by n

7-

(2.73)

i=1

It is easy to see t h a t ( v f A ) 2 - A . If A is positive definite, eq. (2.69) can be extended to arbitrary negative non-integer powers such as A -2/3. Let {rl, ..., rl}L denote the linear subspace spanned (or generated) by r l , ..., rl, i.e., the set of all vectors that can be expressed as a linear combination l ~-,i=1 ciri for some real numbers cl, ..., ct. A positive semi-definite (nn)matrix of rank r (_< n) has the following spectral decomposition:

A - ~ Aiuiu-~,

Ai > 0,

i-

1, ..., r.

(2.74)

i=1

Let the symbol T~n denote the n-dimensional space of all n-vectors. r-dimensional subspace 7~A = {Ul,..., U,}L C T~n

The

(2.75)

is called the range (or image space) of A, for which the set {Ul, ..., u~} is an orthonormal basis. The ( n - r)-dimensional subspace A/"A = {u~+l, ..., u,~}n C 7~n

(2.76)

is called the null space of A, for which the set {u~+ 1, ..., un } is an orthonormal basis. The n-dimensional space is the direct sum of T~A and HA, each being the orthogonal complement of the other: T~n - 7~A 9 AfA,

7~A _L N"A .

(2.77)

This definition implies

PXA A = APxA = A.

(2.78)

The (Moore-Penrose) generalized (or pseudo) inverse 1~ A - of A is defined 1~ Moore-Penrose generalized inverse is often denoted by A + in order to distinguish it from the generalized inverse in general, which is defined as the matrix X that satisfies A X A = A and denoted by A - . The generalized inverse we use throughout this book is always the Moore-Penrose type, so we adopt the generic symbol A - . The symbol A + will be given another meaning (see Section 2.2.6).

40

Chapter 2.

Fundamentals of Linear Algebra

by

A-

-

~1 i=1

--~--uiu Ai

7-.

(279)

i

Evidently, the generalized inverse A - coincides with the inverse A -1 if A is of full rank. From this definition, the following relationships are obtained (see eqs. (2.50)and (2.63))"

(A-)--A,

PXA A - - A - P x A - A - , A-A-

(2.so)

A A - - PXA.

From eqs. (2.78)and (2.80), we obtain

AA-A-

A,

A-AA-

- A-.

(2.81)

The rank and the generalized inverse of a matrix are well defined concepts in a mathematical sense only; it rarely occurs in finite precision numerical computation that some eigenvalues are precisely zero. In computing the generalized inverse numerically, the rank of the matrix should be predicted by a theoretical analysis first. Then, the matrix should be modified so that it has the desired rank. Let A be a positive semi-definite (nn)-matrix of rank 7" r; let A - ~i=1 Aiuiu-{, A1 > "" _ A,. > 0, be its spectral decomposition. Its rank-constrained generalized inverse (A)~ of rank r' (_< r ) i s defined by

(A)~, - ~ - " u1i u Ai i=1

i7-.

(2.82)

From this definition, the following identities are obtained:

(A)~A-

A(A)~, - Px(a _~,,

(A)~,A(A)~, - ( A ) , - , .

(2.83)

Let A be an (nn)-matrix, and B an (mm)-matrix. Let S and T be nmmatrices. Even if A and B are not of full rank, the matrix inversion formula (2.22) holds in the form (A + PA;ASBTTPxA) - - A - - A - S ( B - + P x B T T A - S P x B ) - T T A-,

(2.84) provided that matrix A + P x A S B T T P x A has the same rank as A and matrix B - + P x B T T A - S P x 8 has the same rank as B - . We call eq. (2.84) the generalized matrix inversion formula.

2.2.3

R a y l e i g h q u o t i e n t and quadratic f o r m

For an (nn)-matrix A, the expression (u, Au)/[]u[I 2 is called the Rayleigh quotient of vector u for A. Let )~min and Amax be, respectively, the largest

2.2.

Eigenvalue Problem

41

and the smallest eigenvalues of A. arbitrary nonzero vector u:

/~min ~

The following inequality holds for an

(u, Au) ]]U]]2 ~ )kmax.

(2.85)

The left equality holds if u is an eigenvector of A for eigenvalue )~inin; the right equality holds if u is an eigenvector for eigenvalue Am~x. The Rayleigh quotient (u, Au)/llull 2 is invariant to multiplication of u by a constant and hence is a function of the orientation of u" if we put n N[u], then (u, Au)/llull ~ = (n, An), which is called the quadratic form in n for A. Eq. (2.85) implies min (n, Ilnll-1

A n ) - )~min,

max (n, Iln]l=l

A n ) - )~max.

(2.86)

The minimum is attained by any unit eigenvector n of A for eigenvalue/~min; the m a x i n m m is attained by any unit eigenvector n for eigenvalue "~max. It follows that an (nn)-matrix A is positive definite if and only if (r, A r ) > 0 for an arbitrary nonzero vector r; it is positive semi-definite if and only if (r, At) > 0 for an arbitrary n-vector r. For an arbitrary ran-matrix B, the matrix B TB is symmetric (see the second of eq. (2.2)). It is also positive semi-definite since (r, B T B r ) - [ [ B r [ [ 2 > 0 for an arbitrary n-vector r. If B is an n n - m a t r i x of full rank, equality holds if and only if r - 0. For an (nn)-matrix A, its square root ~ is also symmetric (see eq. (2.73)). We can also write A observations, we conclude the following:

v/-A Tv/A.

From these

9 Matrix A is positive semi-definite if and only if there exists a matrix B such that A - B T B . 9 Matrix A is positive definite if and only if there exists a nonsingular matrix B such that A - B TB. 9 If A is a positive semi-definite (nn)-matrix, matrix B

T A B is a positive

semi-definite ( m m ) - m a t r i x for any n m - m a t r i x B.

2.2.~

Nonsingulargeneralized eigenvalue problem

Let A be an (nn)-matrix, and G a positive semi-definite (nn)-matrix. If there exists a nonzero vector u and a scalar/~ such that

Au-

)~Gu,

(2.87)

the scalar/~ is called the generalized eigenvalue of A with respect to G; the vector u is called the corresponding generalized eigenvector. The problem of

42

Chapter 2.

Fundamentals of Linear Algebra

computing such u and )~ is said to be nonsingular if G is of full rank, and singular otherwise. Consider the nonsingular generalized eigenvalue problem. Eq. (2.87) can be rewritten as ()~G - A ) u = 0, which has a nonzero solution u if and only if function CA,G ( , ~ ) - I , ~ G - A I (2.88) has a zero: CA G ('~) - 0. The function CA G("~) is an nth degree polynomial in ,~ and is cali'ed the generalized characteristic polynomial of A with respect to G. The equation CA,G()~) - 0 is called the generalized characteristic equation of A with respect to G and has n roots {,~i} (with multiplicities counted). The generalized eigenvalue problem with respect to I reduces to the usual eigenvalue problem. The generalized eigenvalues {)~i} of A with respect to G are all real. The corresponding generalized eigenvectors {ui} can be chosen so that

(ui, a u j ) = 6ij,

(2.89)

(Ui, A u j ) - )~j~ij.

(2.90)

which implies Let us call the set {ui} so defined the generalized eigensystem of the (nn)matrix with respect to the positive definite (nn)-matrix G. Let U = (Ul,...,un) and A = diag(A1, ..., An), respectively. Eqs. (2.89) and (2.90) can be rewritten as

U - c G U - I,

U - c A U - A.

(2.91)

By multiplying the first equation by G U from the left and U-rG from the right, the following generalized spectral decomposition is obtained: n

A - GUAU-VG - E

)~i(Gui)(Gui)-r"

(2.92)

i=1

The number of nonzero generalized eigenvalues is equal to the rank of A. If A is positive definite, {hi } are all positive; if A is positive semi-definite, {hi} are all nonnegative. The generalized eigenvalue problem A u - )~Gu reduces to an ordinary eigenvalue problem as follows. Let C - G -1/2 and it - C - l u (see eqs. (2.71) and (2.73)). It is easy to see that eq. (2.87) can be written as -

A

=

CAC.

(2.93)

If an eigenvector it of A is computed, the corresponding generalized eigenvector is given by u - C~. (2.94)

2.2.

Eigenvalue Problem

43

The expression (u, A u ) / ( u , Gu) for an (nn)-matrix A and a positive definite (nn)-matrix G is called the generalized Rayleigh quotient of u. It satisfies

(u, Au)

)~min ... n-1

~__ A n _ 1 > A n.

Define matrix

T

A+ _ ~

uiu______~i

._

Ai-

(2 110)

An"

"

This is a positive semi-definite matrix having eigenvalues { 1 / ( A / - An)} for the same eigensystem { u i } . If A~ - 0, the matrix A + coincides with the generalized inverse A - . Eq. (2.109) can be rewritten as u~ - u,~ - c A + D u n + O(e2).

(2.111)

Let A and D be (nn)-matrices, and G a positive definite (nn)-matrix. Let {A/} be the generalized eigenvalues of A with respect to G, and { u i } the corresponding generalized eigensystem: Aui - AiGui,

(ui, V u j ) - (~ij.

(2.112)

If A is perturbed in the form of eq. (2.107), the perturbation theorem holds in the same form. Eq. (2.111) also holds if {Ai} in eq. (2.110) are interpreted as generalized eigenvalues of A with respect to G. Linear

2.3 2.3.1

Systems

Singular

and

Optimization

value decomposition

and generalized

inverse

If A is an ran-matrix, A T A is a positive semi-definite (nn)-matrix, and A A T is a positive semi-definite (mm)-matrix. They share the same nonzero eigenvalues or1 >_ cr2 > ... >_ a~ (> 0), r < rain(m, n). The number r is called the rank of A. Let Ai - v/-~, i - 1, ..., r, and ~i - 0, i - r + 1, ..., max(re, n). It can be shown that orthono~r~i~l systems { u i } , i - 1, ..., n, and {v i}, i 1, ..., m, exist such that 9 A u i - Aivi, i - 1, ..., rain(m, n). 9 { u i } , i - 1, ..., n, is the eigensystem of A T A for eigenvalues {A2}, i 1, ..., n. 9 { v i } , i - 1, ..., m, is the eigensystem of A A T for eigenvalues {A2}, i -

Matrix A is expressed in terms of { u i } , {vi}, and {~i} in the form

A

Aiviu i .

-

(2.113)

i=1

This is called the singular value decomposition of A; the values {Ai}, i - 1, ..., min(m, n), are called the singular values of A. Let us call { u i } , i - 1, ..., n,

Chapter 2.

46

Fundamentals of Linear Algebra

A

~Rm

Fig. 2.4. The range TrA and the null space AfA of linear mapping A. and {vi}, i - 1, ..., m, the right orthonormal system and the left orthonormal system of A, respectively. If we define orthogonal matrices U - ( u l , u 2 , . . . , u n ) and V = (Vl, v2, ..., vm), eq. (2.113)can be rewritten in the form

A-

V A U T,

(2.114)

where A is an mn matrix whose first r diagonal elements a r e )~1, ..., )~r in that order and whose other elements are all zero. If m -- n, matrix A is diagonal. The r-dimensional linear subspace (2.115)

nA = { ~ , "",-~}L C ~e~

is called the range (or image space) of A: for any m-vector y E T~A, there exists an n-vector x such that y = A x (Fig. 2.4). The ( n - r )- dimensional linear subspace

AfA

-

(2.116)

{ U r + I , ..., U n } L C T~ n

is called the null space (or kernel) of A: A x = 0 for any n-vector x E A fA (Fig. 2.4). If A is symmetric, its right and left orthonormal systems coincide with its eigensystem, and its singular value decomposition coincides with its spectral decomposition (see eq. (2.62)). Since {ui} is an orthonormal system, eq. (2.64) holds for an arbitrary nvector x. Let )kma x be the maximum singular value. Since {v i} is also an orthonormal system, we see from eq. (2.113) that

IIAxll = - II

A~(u~, x)v~ll = i:1

A 2 (ui, x) 2 _
_ ""

> _ an > _ 09

(3.21)

i=1

The vector ul indicates the orientation in which the error in x is most likely to occur, and a 2 is the variance in that orientation. In fact, for an arbitrary unit vector u

v[(~. ~)l - E[(~. ~.)~] - (~. E [ A . A . ~ ] u ) - (~. V[.]~).

(3.22)

which is maximized by u l ; the m a x i m u m value is a 2 (see eqs. (2.86)). We can also see that for each i the eigenvalue a i2 indicates the variance of the error A x in orientation ui. Since {ui} is an orthonormal system, the error A x can be expressed in the form Tt

Ax - Z

Axiui,

A x i - (Ax, ui).

(3.23)

i=1

It follows from eq. (3.21) that the distribution in each orientation is uncorre-

fated" E [ A x i A x j ] - (ui, V[xluj) - a~Sij.

(3.24)

If the distribution of the error A x is isotropic, i.e., its occurrence is equally 2 Eq. (2.63) implies that the likely in every orientation, we have a 2 = ... = a n. covariance matrix V[x] has the form

v[~]

-

-~- z ,

~2 _

E[IIAxlI2].

(3.25)

n

Let {ul, ..., un} be an orthonormal basis of 7~n. If the distribution of A x is restricted to the r-dimensional subspace S - {ul, ..., U.}L C 7~n, the covariance matrix V[x] is singular:

V[x] - ~ cr2uiui r.

(3.26)

i=1

The null space of V[x] is the orthogonal complement Af - {U,+l, ..., Un}L C TC~ of S. If the distribution is isotropic in S, the covariance matrix V[x] has the following form (see eqs. ( 2 . 5 0 ) a n d (2.51))"

v[~]

-

-~- P x , r

~2 _

E[llAxl[2].

(3.27)

3.2.

Manifolds a n d Local D i s t r i b u t i o n s

65

%(s)

0 Fig. 3.1. The tangent space Tx(S) and the normal space Tx(,5') to S at x in three-dimensions.

3.2 3.2.1

M a n i f o l d s and Local D i s t r i b u t i o n s M a n i f o l d s and tangent spaces

A (differential) manifold S in an n - d i m e n s i o n a l space T~n is a s u b s e t of 7~ n such t h a t for each p o i n t x C S t h e r e exists a diffeomorphism (a s m o o t h differential m a p p i n g ) b e t w e e n a n e i g h b o r h o o d of x in S a n d an o p e n s u b s e t of 7~ m for some m, which is called t h e dimension of t h e m a n i f o l d S. A m a n i f o l d is a generalization of a s m o o t h c u r v e in two d i m e n s i o n s a n d a s m o o t h surface in t h r e e d i m e n s i o n s (the exact definition of a m a n i f o l d is o m i t t e d ) . A n n - d i m e n s i o n a l space 7~ ~ is itself an n - d i m e n s i o n a l manifold. If an m~-dimensional m a n i f o l d S ' is a s u b s e t of an m - d i m e n s i o n a l m a n i f o l d S, m a n i f o l d S ' is said to be a submanifold of S of codimension m - m'. In p a r t i c u l a r , an m - d i m e n s i o n a l m a n i f o l d in 7~m is a s u b m a n i f o l d of 7~ ~ of c o d i m e n s i o n n - m. Manifolds are often defined by e q u a t i o n s 2. For e x a m p l e , an ( n - 1)d i m e n s i o n a l u n i t sphere in n d i m e n s i o n s c e n t e r e d at t h e c o o r d i n a t e origin, d e n o t e d by S ~-1, is defined by Ilxl] 2 = 1. A n e q u a t i o n f ( x ) = 0 is nonsingular if it defines a m a n i f o l d of c o d i m e n s i o n 1, a n d is singular otherwise. For e x a m p l e , e q u a t i o n Ilxll 2 = 0 is singular a, b e c a u s e t h e m a n i f o l d it defines is t h e c o o r d i n a t e origin O, w h i c h is z e r o - d i m e n s i o n a l (i.e., of c o d i m e n s i o n n). Let x be a p o i n t in an m - d i m e n s i o n a l m a n i f o l d S C g ~ . T h e set of all infinitesimally small vectors Am E T~n such t h a t x + A x E S forms an md i m e n s i o n a l linear space to a first a p p r o x i m a t i o n . T h i s linear space is d e n o t e d by Tx(S) a n d called t h e tangent space 4 to S at x (Fig. 3.1). A t a n g e n t space is a g e n e r a l i z a t i o n of a t a n g e n t line to a c u r v e in two a n d t h r e e d i m e n s i o n s a n d a t a n g e n t p l a n e to a surface in t h r e e d i m e n s i o n s . T h e o r t h o g o n a l c o m p l e m e n t 2A manifold is also called an algebraic variety if it can be defined by polynomial equations F(k)(x) = 0, k = 1, ..., L. 3In this book, we always consider real spaces; in a complex space, equation I]xll2 = 0 defines an (n - 1)-dimensional imaginary surface. 4The collection of all Tx(S), x C S, is called the tangent bundle of S.

66

Chapter 3.

Probabilities and Statistical Estimation

,,,-" ./i

(~)

(b)

(c)

(d)

Fig. 3.2. (a) Two surfaces intersecting transversally. (b)-(d) Two surfaces meeting non-transversally. of T x ( S ) C ~ n , denoted by N x ( S ) , is an ( n - m)-dimensional linear space and is called the normal space to 8. If f ( x ) = 0 is a nonsingular equation that defines a manifold of codimension 1, it has an ( n - 1)-dimensional tangent space T x ( S ) - {Vf}Ll and a one-dimensional normal space N x ( S ) - { V f } L , where V f = (Of/Oxl,...,Of/Oxn) T. If an m-dimensional manifold S and an m~-dimensional manifold 8 ~ meet in 7~" and if m + m ' > n, their intersection 8 f 3 8 ~ is in general an ( m + m ' - n ) dimensional manifold. Equivalently put, if a manifold S of codimension I and a manifold $~ of codimension 1~ meet in T~n and if 1-4-1~ < n, their intersection S gl S~ is in general a manifold of codimension 1 + 1~. If the following condition is satisfied in addition, manifolds 8 and S~ are said to intersect transversally:

T~ (S n S') = T~ (S) n T~ (S').

(3.28)

For example, two surfaces in three dimensions intersect transversally if they cut each other along a curve (Fig. 3.2a), but they do not if they overlap (Fig. 3.2b) or touch each other along a curve (Fig. 3.2b) or at a single point (Fig. 3.2c). If two manifolds intersect transversally, their intersection is structurally stable in the sense that its dimension (or codimension) is preserved if the two manifolds are infinitesimally perturbed in an arbitrary manner.

3.2.

Manifolds a n d Local Distributions

T

67

(S)

x)

0 Fig. 3.3. A local distribution.

3.2.2

Local distributions

Let x be an n-vector r a n d o m variable whose distribution is r e s t r i c t e d to an m - d i m e n s i o n a l manifold $ C 7~n. In general, m a t h e m a t i c a l t r e a t m e n t is very difficult if a r a n d o m variable is constrained to be in a manifold. For example, if x is a unit 3-vector, its probability density p(x) is defined over a unit sphere S 2, but its e x p e c t a t i o n E[x] - fs2 xp(x)dx is generally inside the sphere. In this book, w h e n e v e r we consider a probability distribution of a r a n d o m variable x c o n s t r a i n e d to be in a manifold $, we a s s u m e t h a t x has a local distribution in the sense t h a t the distribution is sufficiently localized a r o u n d some point 5~ E $ and hence to a first a p p r o x i m a t i o n the d o m a i n of the distribution can be identified with the t a n g e n t space T x ( $ ) at 5~ (Fig. 3.3). We choose the point ~ in such a way t h a t

p S E [ x - 5~] - O,

(3.29)

and identify the covariance m a t r i x of x with v[.]-

-

-

(3.30)

where p S is the projection m a t r i x onto the t a n g e n t space T x ( $ ) at 5~. We often call 5~ simply "the t r u e value of x". Eq. (3.30) implies t h a t the range and the null space of V[x] coincide with the t a n g e n t space T x ( S ) and the n o r m a l space N x ( $ ) , respectively. For brevity, we call the null space of the covariance m a t r i x V[x] of x simply "the null space of x".

3.2.3

Covariance matrix of a 3-D rotation

Consider a local distribution of three-dimensional r o t a t i o n s 5 a r o u n d R. Namely, we r e g a r d a t h r e e - d i m e n s i o n a l r o t a t i o n m a t r i x R as a r a n d o m variable p e r t u r b e d from 1~ by a small a m o u n t A R in the form R - R + A R . Since R and R are b o t h rotations, the t r a n s f o r m a t i o n from 1~ to R is also 5A three-dimensional rotation can also be represented by a 4-vector q, called quaternion, constrained to be on a 3-dimensional unit sphere S 3 in four dimensions. Hence, the distribution of three-dimensional rotations can also be thought of as defined over S 3.

68

Chapter 3.

Probabilities and Statistical Estimation

a rotation around some axis by a small angle. Since a small rotation has the form given by eq. (2.57), we can write R - ( I + A ~ l x I + O ( h ~ 2 ) ) R , or

R - / ~ +/',fit x/~ + o(A~2).

(3.31)

Hence, to a first approximation AR-

AFtl •

(3.32)

We define the covariance matrix of rotation R by (3.33) The unit eigenvector of V[R] for the largest eigenvalue indicates the axis around which perturbation is most likely to occur. The corresponding eigenvalue indicates the mean square of the angle of rotation around that axis. The mean square of the angle of perturbed rotation in total is E [ A ~ 2] -- trV[R].

(3.34)

In particular, if the perturbation is equally likely to occur for every axis orientation, the covariance matrix V[R] has the form

v [ n ] - ~-~, 3.3 3.3.1

~ = E[Z~n~].

(3.35)

G a u s s i a n D i s t r i b u t i o n s and )~2 D i s t r i b u t i o n s Gaussian distributions

The most fundamental probability distribution of an n-vector random variable is the multidimensional Gaussian distribution (or normal distribution). We say that n-vector x is a Gaussian random variable if it has a multidimensional Gaussian distribution. If it has mean m and covariance matrix ~ of full rank, the probability density has the form

p(x) -

1

e-(m-m'E-~(m-m))/z

(3.36)

v/(2~.)n i~, I which defines a distribution over the entire n-dimensional space 7~n. It is easy to confirm that

E[,] - ]~o .p(.)d. - m, v[.] - / ~ ( .

- m)(. - m)~p(.)d.

- ~.

(3.37)

3.3.

69

Gaussian Distributions and ~2 Distributions

////2 ....

1

X

0

ml

Fig. 3.4. Contours along which probability density is constant. The probability density is constant on the surface in 7~~ defined by (x-

m , 27-1(x - m ) ) =

c

(3.38)

for a positive constant c (Fig. 3.4). Suppose x is decomposed into the form x = x l | x2, and let m = m l | m2 be the corresponding decomposition of the mean m . If X l and x2 are uncorrelated to each other, the covariance matrix E is decomposed into the form ,~ = •1 | E:2, where -~1 and E2 are the covariance matrices of xl and x2, respectively. Then, eq. (3.36) has the form

1 p(x)-

C

- ( x , - m ~ E~-~(x~-m~))/2

V/(27r)nlEil X

1

e

- ( x 2 - m 2 ' ~,2-~(x2-m2))/2

.

(3.39)

This means that uncorrelated Gaussian random variables are always indepen-

dent of each other. In one dimension, the probability density reduces to

p(x)-

1

e--(x--m)2/2cr2

where cr2 (> 0) is the variance of x (Fig. 3.5). standard deviation. Let us call the pair {m + cr, m - or}

(3.40) The value a is called the

(3.41)

the standard deviation pair, and the interval [m - ~r, m + cr]

(3.42)

the standard confidence interval. The probability that x falls into the standard confidence interval is about 68.27%. If m = 0 and a = 1, the distribution is called the standard Gaussian (or normal) distribution.

70

C h a p t e r 3.

Probabilities and Statistical Estimation

m--o

m

m+o

"------

X

Fig. 3.5. One-dimensional Gaussian distribution. Gaussian distribution of mean m and covariance matrix E of rank r (< n) is defined as follows. Consider the case m - 0 first. Let Tt

-- Z

6ri2Uilti'

0"1 ~ ' ' "

-~ ~7r > O'r-t-1 :

"''-"

f i n ( = 0),

(3.43)

i--1

be the spectral decomposition of 2~. From the discussion in Section 3.1.2, we see t h a t x cannot deviate in the orientations u , + l , ..., Un. Since {ui} is an orthonormal basis of T~n, vector x is expressed in the form

- ~ x,~,,

x,- (,, ~,).

(3.44)

i----1

It follows t h a t the distribution is limited to the range 7"4,E = {Ul, ..., U r } L of the covariance matrix ~ . The components Xl, ..., x . have an r-dimensional Gaussian distribution with density

p(Xl,...,Xr.) __

1 p

C--Z'i=I xi2/2~

(3.45)

v/(2~) ~ 1-i~=l ~, which can be rewritten in the form

p(x) -

1

e-(X'~-x)/2 ,

(3.46)

X/(2~)~l~l+ where I ~ l + - 1-ii=lr ~ is the product of all positive eigenvalues of 2~ Eq. (3.46) defines a probability distribution only in the r-dimensional subspace ~ E . Hence,

v[x]-/~

xxTp(x)dx- ~.

(3.47)

23

T h e Gaussian distribution for m :fi 0 is defined by replacing x by x - m .

3.3.

Gaussian Distributions and )~2 Distributions

71

If the covariance m a t r i x 27 has the spectral decomposition in the form of eq. (3.43), Ul indicates the orientation of the most likely deviation (see Fig. 3.4); or1 is the standard deviation in that orientation. Hence, the probability t h a t - a l < (Ul, x - m ) < al is about 68.27%. Let us call { m + o l U l , m -- crlUl }

(3.48)

the primary deviation pair, which indicates in which orientation the deviation is most likely to occur. The Gaussian distribution plays a special role in statistics for m a n y reasons, of which the most important is the fact t h a t if X1, ..., XN are independent samples from a distribution of mean zero and variance/covariance matrix V[X], the average .'~ - Ec~N__I X ~ / N is asymptotically a Gaussian r a n d o m variable of mean zero and variance/covariance m a t r i x V[X]/N under a mild regularity condition. This fact is known as the central limit theorem 6. Other important properties of the Gaussian distribution include the following: 9 If x is an n-vector Gaussian random variable of mean m and covariance matrix 27, m-vector y - A x for an arbitrary r a n - m a t r i x A is also a Gaussian r a n d o m variable of mean A m and covariance m a t r i x A ~ A T (see eqs. (3.14)). 9 Each component of a vector Gaussian r a n d o m variable x is independent and has the s t a n d a r d Gaussian distribution if and only if m - 0 and ,~ mY.

9 If each component of x is independent and has the s t a n d a r d Gaussian distribution, each component of vector y = A x is independent and has the s t a n d a r d Gaussian distribution if and only if AA-r = I. Since the Gaussian distribution is defined over the entire n-dimensional space 7~n or its linear subspace, the probability tails away infinitely. However, we can define a Gaussian distribution over an arbitrary manifold $ C 7~n if the distribution is sufficiently localized around one point 5~ E ,5 and hence the domain of the distribution can be identified with the tangent space Tx ($) (see Section 3.2.2). Namely, the distribution can be regarded as locally Gaussian if it has a probability density in the form

p(x) - Ce -(x-Jc'E-(x-'2"))/2,

(3.49)

where C is the normalization constant. The mean 5~ and the covariance matrix E are assumed to satisfy the following relations:

~ p(x)dx - l,

P~ fs(x-

5~)p(x)dx - O,

6We omit the exact statement of the theorem and the proof.

72

Chapter 3.

P~ L(.

Probabilities and Statistical Estimation

- ~)(~ - ~ ) T p ( ~ ) d ~ P ~ - E.

(3.50)

Here, P ~ is the projection matrix onto the tangent space T x ( $ ) at 5~.

3.3.2

Moment generating functions and moments

The moment generating function of a scalar random variable x is defined by 7

E[~~~ - Eoo E[xk k! ] Ok.

r

(3.51)

k=0 If x is a Gaussian random variable of mean 0 and variance a2, its moment generating function has the following form:

9 (0)- ~

20~/~= ~0r ~•2k 0 2k .

(3.52)

k=0 Comparing this with eq. (3.51) term by term, we obtain the kth moment E[x k] in the following form:

E[xkl -

k!ak

k - 0, 2, 4,6,..., k - 1,3,5,7, ....

2k/2(k/2)! , 0,

(3.53)

The moment generating function of a vector random variable x is defined by 8 O0

~ o ( 0 ) - E[e (x'O)] - ~

~ E [ ( x , 0)k],

(3.54)

k:0 where the argument 0 is also an n-vector. If x is a Gaussian random variable of mean 0 and covariance matrix E , its characteristic function has the following form: OO

9 (o)- ~(o.~o),= ~ 2_~(o, ~o)~

(3.55)

k=O Comparing eqs. (3.54) and (3.55), we obtain the expressions for the (multidimensional) moments E[xi~xi2"" xi.]. For example,

E[xi] - 0,

E[xixj] - ~ij,

E[xixjxkxl] -- ~ij~kl + ~ik~jl + ~il~jk,

E[xixjxk] -- O, E[xixjxkXlXm] = 0.

(3.56)

7The function r is called the characteristic function (i is the imaginary unit). It is simply the Fourier transform of the probability density p(x) of x. SFunction (~(iO) is also called the characteristic function. It is the multidimensional Fourier transform of the probability density p(x) of x.

3.3.

Gaussian Distributions and X2 Distributions

73

O Fig. 3.6. )~2 distribution with r degrees of freedom.

3.3.3

X 2 distributions

If xl, ..., x~ are i n d e p e n d e n t samples from the s t a n d a r d Gaussian distribution, the distribution of

R-xa

+...+x~

(3.57)

is called the X2 distribution with r degrees of freedom. We call a r a n d o m variable which has the X 2 distribution simply a X2 variable. Its probability density is defined over [0, oc) in the form

p~(R)-

1 Rr/2_le_R/2 2~/2r(r/2)

(3.58)

where F(n) - f o t ~ - l e - t d t is the Gamma function 9 (Fig. 3.6). T h e m e a n and the variance of this distribution are

E[R] = r,

V[R] = 2r.

(3.59)

T h e density p~(R) takes its m a x i m u m at R = r - 2. T h e i m p o r t a n t facts concerning the X~z distribution include the following: 9 If R1, ..., RN are i n d e p e n d e n t ~2 variables with rl, ..., rN degrees of freedom, respectively, the sum R = Ra + " " + RN

(3.60)

is a )i2 variable with rl + ' . . + rn degrees of freedom. 9 If x is a Gaussian r a n d o m variable of m e a n 0 and covariance m a t r i x E of rank r, the q u a d r a t i c form R = (x,1ff-x)

9F(n + 1) - n! and F(n + 1/2) = (2n)Iv/~/22nnI for nonnegative integers n.

(3.61)

74

C h a p t e r 3.

Probabilities and Statistical E s t i m a t i o n

is a X2 variable with r degrees of freedom. 9 T h e probability t h a t a Gaussian r a n d o m variable of m e a n 0 and covariance m a t r i x E of rank r satisfies

(x, ~ - x )

_< 1

(3.62)

to 0 f0 9 If x~, c~ = 1, ..., N , are i n d e p e n d e n t Gaussian r a n d o m variables, each having m e a n 0 and covariance m a t r i x E a of rank ra, the s u m N

R - E

(x~, E ~ x~)

(3.63)

or--1

is a X2 variable w i t h Ec~N__Ir~ degrees of freedom. 9 Let n - v e c t o r x and m - v e c t o r y be Gaussian r a n d o m variables of m e a n 0, and let ~ x and E y be their respective covariance matrices. Let n and r ( ~ n) be the ranks of ~ x and ,!Ty, respectively. If t h e r e exists an r a n - m a t r i x A such t h a t y = A x , t h e difference R - (x, ~7~1x) - (y, E y y ) is a X2 variable with n -

3.3.~

(3.64)

r degrees of freedom (Cochran's theorem 11).

Mahalanobis distance and X 2 test

Let n-vector x be a Gaussian r a n d o m variable of m e a n 0 and covariance m a t r i x E . If E is of full rank, we can define a n o r m 12 of x by

ll ll -

(3.65)

E q u i d i s t a n t points from the origin in this n o r m have equal probability densities, and the probability density at x becomes smaller as IlxllE becomes larger. T h e value IlxllE is called the Mahalanobis distance of x f r o m the origin. If x is r a n d o m l y chosen, Ilxll~ is a ~(2 variable with n degrees of freedom. If E has rank r ( < n), we can define a p s e u d o - n o r m 13

I1 11 -

(3.66)

1~ four decimal digits, this equals 0.6827, 0.3935, 0.1987, 0.0902, 0.0374 for r = 1, 2, 3, 4, 5, respectively. alTo be exact, this is a special case of Cochran's theorem. 12For any positive definite symmetric matrix ~, eq. (3.65) defines a norm in the strict mathematical sense described in Footnote 4 in Section 2.1.1. 13This is not a norm in the strict mathematical sense because the triangle inequality (2.9) does not hold; see eq. (3.67).

3.3.

G a u s s i a n Distributions a n d X2 Distributions

75

a

100

Fig. 3.7. ~:2 test with significance level a%. which is also called the M a h a l a n o b i s distance. Since I1 11 -- 0 for x e N'2~, eq. (3.66) defines a distance in the usual sense only in the range ~ of ~ ; for x l E 7~2~ a n d x2 E N'2~, we have

+ If x is r a n d o m l y chosen,

II ll

= ll lll .

is

(3.67)

variable with r degrees of freedom.

T h e X2 d i s t r i b u t i o n provides a simple m e a n s to test hypotheses. In m a n y problems, we can define a r a n d o m variable R in the form of eq. (3.57), where each x / m a y not have zero mean. T h e e x p e c t a t i o n of R b e c o m e the smallest w h e n all x i have zero means. Suppose all x i have zero m e a n s if and only if some condition is satisfied. This condition is r e g a r d e d as a hypothesis a n d can be t e s t e d by observing a s a m p l e d value of R: the hypothesis is rejected if it is very large to an inadmissible degree. An exact p r o c e d u r e is as follows. Let R be a sample from a X2 d i s t r i b u t i o n with r degrees of f r e e d o m u n d e r t h e hypothesis. T h e hypothesis is rejected with significance level a% (or with confidence level ( 1 0 0 - a ) % ) i f it falls into the rejection region (X2,a, e~) a n d is r e g a r d e d as acceptable 14 otherwise (Fig. 3.7). T h e t h r e s h o l d value Xr,a 2 is called t h e a ~o significance value of X2 with r degrees of f r e e d o m a n d defined in such a way t h a t 15

f

~ p~(R)dR2r,a

a 100

(3.68)

Thus, the hypothesis is rejected with significance level a% if R>

2 X~,a"

(3.69)

This p r o c e d u r e is called the X2 test a n d frequently a p p e a r s in m a n y practical p r o b l e m s - - i n p a r t i c u l a r w h e n least-squares o p t i m i z a t i o n based on the Mahalanobis distance is used, since the residual of o p t i m i z a t i o n is usually a X 2 variable if the noise is G a u s s i a n (see the next section). 14Note that we do not say that the hypothesis is accepted. Being acceptable means that there exists no evidence strong enough to reject it. 15If r is large, say r > 30, the approximation Xr, 2 a "~ (Na +v/2r - 1)2/2 holds, where the number Na is defined in such a way that a standard Gaussian random variable falls in the interval (Na, ~ ) with probability a/100.

76

Chapter 3.

Probabilities and Statistical Estimation

O F i g . 3.8. M o d i f i e d X 2 d i s t r i b u t i o n w i t h r degrees of f r e e d o m .

If R is a X2 variable with r degrees of freedom, the distribution of 8 _

R

/

J-I

p..rr~

\

_

\

----/

r

is sometimes called the modified ~12 distribution16 with r degrees of freedom. The probability density of s is given by rp~(rs), where p r ( R ) is the X2 probability density given by eq. (3.58) (Fig. 3.8). Eq. (3.59) implies that its expectation and variance are

E[s]- 1,

V[s] =

2

-.

(3.71)

r

In terms of the modified k,2 variable s, the X2 test given by eq. (3.69) can be rewritten as >

(3.72)

r

The X2 test usually takes this form when the magnitude of the noise is estimated mid compared with its presumed value, as will be shown in later chapters.

3.4

3.~.1

S t a t i s t i c a l E s t i m a t i o n for G a u s s i a n M o d e l s

Maximum likelihood estimation

Let x be an n-vector, and A an ran-matrix. Let ~ be an m-vector Gaussian r a n d o m variable of mean 0 and covariance matrix E of full rank. Then y -

Ax

+ e

(3.73)

16This terminology is not widely used because the only difference from the X2 distribution is scaling. However, this distribution plays an essential role in the problems we study in this book, as we will see later.

3.4.

Statistical Estimation for Gaussian Models

77

is an m-vector Gaussian random variable with mean A x and covariance matrix ~7. Hence, the probability density of y is given by p(v) =

1

_(y_Ax,,F,-l(y_Ax))/2

(3.74)

Consider the problem of estimating the parameter x from a sampled value y. Namely, we want to find a function ~ ( y ) that gives an estimate of x for a given y. Such a function is called an estimator. Evidently, any value x ~ such that A x ~ = 0 can be added to x. In order to remove this indeterminacy, we assume that x is constrained to be in A f t , the null space of matrix A. Maximum likelihood estimation seeks the value x that maximizes the probability density p(y), which is called the likelihood when viewed as a function of the observed value y. The problem reduces to minimizing the Mahalanobis distance I l Y - AxlI,F,, i.e.,

J[x] = (y - A x , E - l (y - A x ) ) --+ min

(3.75)

under the constraint x E A f t . The solution, which is called the maximum likelihood estimator, is obtained in the following form (see eqs. (2.136) and (2.137)): -(A-r~-IA)-Aq-,F,-ly. (3.76) Its expectation and covariance matrix are E[5~] = x,

V[~] = ( A q - ~ - I A ) -.

(3.77)

An estimator is unbiased if its expectation coincides with the true value. The first of eqs. (3.77) implies that the maximum likelihood estimator 5~ is unbiased. The residual J[~] of the function J[x] given by eq. (3.75) can be written as follows (see eq. 2.138)): J[~] = (y, ~ 7 - 1 y ) - ( ~ , a q - ~ - l A ~ ) .

(3.78)

This is a ~2 variable with n - m ~ degrees of freedom, where m ~ = r a n k A (see eq. (3.64)). If each component of e distributes independently and isotropically with the same root mean square e, the covariance matrix of E has the form V[e] = e2I (see eqs. (3.25)). Hence, eq. (3.75) reduces to the least-squares optimization I l Y - Axll 2 -4 min,

(3.79)

and the maximum likelihood estimator 5: is given as follows (see eq. (2.134)):

= A-y.

(3.S0)

78

Chapter 3.

Probabilities and Statistical Estimation

(.i:-x, 2:- ~(x-x)) = constant

Rn

L

Fig. 3.9. The point ~ E S that minimizes J[R] is the tangent point of the equilikelihood surface to S.

3.4.2

Optimization

with linear constraints

Let x be an n-vector Gaussian random variable with an unknown mean 5~ and a known covariance matrix 22 of full rank. Suppose the mean 2 satisfies a linear constraint A x - b. Consider the problem of estimating 9 from a sampled value x. Since the probability density of x has the form 1

e -(x-x,

~ -1

(x - x))/2

(3.81)

I the m a x i m u m likelihood estimator for 2 is obtained by the minimization J[5~] = (x - ~, 27 -1 (x - ~)) --+ min

(3.82)

under the constraint A ~ = b. In geometric terms, this problem can be interpreted as follows. The constraint A x = b defines an a ~ n e subspace 17 S in 7~~. The minimization (3.82) means that 5~ is a point in $ that has the shortest Mahalanobis distance 115;- 5c]122 from the sampled position x. Since the equilikelihood surface (the set of all 5~ for which J[5~] = constant) is an "ellipsoid" in 7~~, the point 5~ that minimizes J[2] in S is the "tangent point" of this ellipsoid to 8; all other points in S should be outside that ellipsoid (Fig. 3.9). If we let A x = x - ~, eq. (3.82) reduces to the minimization (Ax, 27 - l A x ) --+ min

(3.83)

under the constraint AAx

= A x - b.

(3.84)

If this constraint is satisfiable, the m a x i m u m likelihood estimator ~ (= x - A x ) is obtained as follows (see eq. (2.140)): 5; - x - • A 17An in ~r~.

affine subspace of

q- ( A E A

T-)- ( A x - b).

(3.85)

7~ n is a s u b s e t of 7~ n o b t a i n e d by t r a n s l a t i n g a linear s u b s p a c e

3.4.

Statistical Estimation for Gaussian Models

79

Its expectation and covariance matrix are E[5~]- 3,

V [ : b ] - 2 ~ - IF,A T ( A ~ A - C ) - A E .

(3.86)

Hence, the m a x i m u m likelihood estimator 5: is unbiased. The residual J[~] of the function J[x] given by eq. (3.82) can be written as J[~] - ( A x - b, ( A ~ , A T ) - ( A x

- b)).

(3.87)

This is a X2 variable with m' degrees of freedom, where m' - r a n k ( A E A ]-) (= rankA) (see eq. (3.61)).

3.4.3

Maximum

a posteriori probability e s t i m a t i o n

Let x be an n-vector, and A an ran-matrix. Let y be an m-vector Gaussian random variable of mean 9 and covariance matrix 2~y of full rank. Consider the following linear model: z = A x + y. (3.88) We want to estimate the parameter x from a sampled value of m-vector z. Suppose the parameter x has an a priori distribution with mean 5~ and covariance matrix 2~x of full rank. The a priori probability density (or prior) of x is p(x) 1 - ( x - x , 2 ~ ~(x-x))/2 -

v/(2~)~l~l

~

.

(3.89)

.

(3.90)

The probability density of y is p(u)

=

~

v/(2~)ml~ui

~-r

~r

For a particular value of x, the m-vector z defined by eq. (3.88) is a Gaussian random variable of mean A x + ~) and covariance matrix E y . Hence, the conditional probability density p(zlx ) of z conditioned on x is

p(z]x) -

1 ~/(2~)~l~yl

e - ( z - A x - ~ t ' E ~ t ~(z-Ax-~l))/2 .

(3.91)

The marginal probability density p(z) of z is defined by

p(z) - f p(z[x)p(x)dx,

(3.92)

which is computed indirectly as follows. From eq. (3.88), the expectation and the covariance matrix of z are given by

E[z] - ASc + 9,

V[z] - A ~ x A -r + E y .

(3.93)

80

Chapter 3.

Probabilities and Statistical Estimation

Hence, the marginal probability density p(z) should have the form

p(z)

1

_(z_A~c_ 9 (A,V, x A T+Ey)-I(z_A~c_y))/2

V/ ( 2rr) m lA ,U,x A r + .,Uy l (3.94) The a posteriori probability density (or posterior)p(xlz) of x conditioned on z is defined by

p(z)

'

(3.95)

which is known as the Bayes formula. Maximum a posteriori probability estimation (often called Bayes estimation) seeks the value of x that maximizes the a posteriori probability density p(xlz ). Maximum likelihood estimation is a special case obtained by setting p(x) - constant. If eqs. (3.89), (3.91), and (3.94) are substituted into eq. (3.95), the a posteriori probability density is obtained in the form

P(XlZ) where

~ l ~ x I +A-rE~IIAI -(x-x

(2~)-

e

(E[r~+ATE~/A)(x-x))/2 '

(3.96)

the vector ~ is defined as follows (see the matrix inversion formula

(2.22)): 5; -- 9 + (,E~, x + AT-,~yl A ) - I A T z~yl (Z -- (A~ + 9)) = ~ + r, x A r ( A r , x A T +

~ ) - 1 ( ~ _ (A~ + 9)).

(3.97)

Thus, the a posteriori probability density p(xlz ) defines a Gaussian distribution of x. If we write its mean and covariance matrix as E[x, lz ] and V[xlz ], respectively, we have W[xl2:] _ ( ~ 1

E[xlz]_ ~ +

.~_ A T . ~ I A ) - I ,

V[xlz]Ar.~yl ( Z -

(A~ -~- 9)).

(3.98) (3.99)

Evidently, p(xlz ) is maximized by x = ~ (= E[zlz]) given by eq. (3.97). We also see that E[~] - 2, i.e., the maximum a posteriori probability estimator 5~ is unbiased. The uncertainty of x that still remains after z is observed is described by the a posteriori covariance matrix V[xlz ] given by eq. (3.98). The marginal probability density p(z) in the denominator in the Bayes formula (3.95) does not involve x. Hence, maximizing p(xlz ) is equivalent to maximizing p(zlx)p(x), which in turn is equivalent to maximizing l o g p ( x ) + logp(zlx ). If eqs. (3.89) and (3.91) are substituted into this, the problem can be written in the following form:

J[x] - (x-hc, z ~ l ( x - 5 ; ) ) + ( z - A x - ~ l ,

Eyl(z-Ax-~l))

--+ min. (3.100)

3.4.

Statistical Estimation for Gaussian Models

81

Va_l

state transitiOnr~B ~ ....

Xct_l - - ~ z q k a _ l ~ - ~ Xct----~ ...

....... o bs;;ation

...................

......................

Y~ Fig. 3.10. State transition of a linear dynamical system. Hence, the problem is viewed as minimizing the square sum of the Mahalanobis distances:

d [ x ] - [ I x - 5:[]2E x + I1 -

~ min.

(3.101)

J[&] - (z - AS~ - ~), (A,V, x A T + , ~ y ) - l ( z - AS - ~)).

(3.102)

The residual J[&] can be written as

This is a X2 variable with m degrees of freedom. The marginal probability density p(z) given by eq. (3.94) has the form

p ( z ) - constant • e -J[2]/2. 3.4.4

(3.103)

K a l m a n filter

The Kalman filter is an iterative linear update procedure for maximum a posteriori probability estimation when the parameters to be estimated change as time progresses in the form of a linear dynamical system. Let n-vector x~ be the state vector at time c~, and L-vector y~ the observation vector at time c~. The process of state transition and observation is described by x~ - A ~ - I x ~ - i + B ~ - l V ~ - I ,

(3.104)

y,~ = C~x~ + w,~,

(3.105)

where A ~ - I , B ~ - I , and C a are constant matrices (Fig. 3.10). Vectors v~ and w~ are assumed to be independent Gaussian random variables, and their expectations E[vo~] and E[w(~] and covariance matrices V[vo~] and V[wo~] are assumed to be known. Furthermore, the covariance matrix V[w,~] is assumed to be of full rank. With this setting, the Kalman filter computes the estimator &~ of the state vector x~ at time c~ and its covariance matrix V[&~] by iterating maximum a posteriori probability estimation. The update rule is derived as follows.

Chapter 3.

82

Probabilities and Statistical Estimation

E[Va_l]

--"

~-,-'~-~,~~

i

"~~ ~

"'"

Fig. 3.11. Kalman filter. Assume that x,~-i is a Gaussian random variable of mean X~--I and covariance matrix V[5~_1]. Since eq. (3.104) is linear in x~-1 and V~-l, the state vector xa at time a is also a Gaussian random variable. Let ~ and V[xa] be, respectively, the mean and the covariance matrix of that distribution. They are computed from eqs. (3.88) and (3.93) as follows (read z, A, x, and y in eqs. (3.93) as x~, A ~ - I , x~-1, and B~-lV~-l, respectively):

Xo, -- Ac~-i ~,o~-1 Jr So~-i E[vo~-I ],

(3.106)

T~r[xal-Aa-lV[ ~ o~-1]Ac~_l 7- + B a - 1 V[vot-]1 B aT- l .

(3.107)

If the value y~ determined by eq. (3.105) is observed, the a posteriori probability distribution of x~ is also Gaussian. Let ~ and V [ ~ ] be, respectively, the mean and the covariance matrix of that distribution. They are computed as follows (read z, A, x, and y in eqs. (3.98), and (3.99) as y~, C a , x~, and w~, respectively):

5~ = 5c~ + V[Sc~lC~V[w~]-l(y~ - ( C ~ 5 ~ + E[w~])),

( -

-1 +

T V [ w a ] - l e a ) -1 .

Eqs. (3.106)-(3.109) define the Kalman filter for computing ~ XO~--I V[~c~-l]. Eq. (3.108) can also be written in the following form:

from

and

5~,~ = 5~ + K,~ (y,~ - :~,~),

K~ = V[~,o~lC:V[wcv] -1,

~lc~ --C~5~ + E[w~].

(3.108) (3.109) and V[&~]

(3.110) (3.111)

Since ~ and 9~ are maximum likelihood estimators of x~ and ya before the actual value y~ is observed, eq. (3.110) can be viewed as correcting the predicted value 5~ by feeding back the difference between the actual observation y~ and its estimator 9~ (Fig. 3.11). In this sense, the matrix K ~ is often referred to as the Kalman gain. The difference y~ - 9~ is independent of Y~-I, Y~-2, " " , and has mean 0; it is called the innovation of y~.

3.5.

General Statistical Estimation

83

In the above formulation, the Kalman filter is derived as m a x i m u m a posteriori probability estimation on the assumption that all the variables are Gaussian. However, the same Kalman filter can be obtained without assuming Gaussian distributions: if we adopt the criterion of minimum mean square estimation, we can obtain eqs. (3.106)-(3.109) by orthogonal projection is of the state vector onto the affine subspace defined by the observation vector (we omit the details).

3.5

3.5.1

General Statistical Estimation

Score and Fisher information matrix

We now study the problem of statistical estimation in the general case where the distribution of the data or noise is not necessarily Gaussian and the statistical model is not necessarily linear. In abstract terms, the problem is stated as follows. Let x be a random variable that has a probability density p(x; O) parameterized by 0. The problem is estimating the parameter 0 by observing random samples from that distribution. We assume that x is an n-vector constrained to be in an hi-dimensional manifold X C 7~~ and the parameter 0 is an m-vector constrained to be in an m'-dimensional manifold $ C 7~m. The probability density p(x; 8) is assumed to be continuous and continuously differentiable with respect to both x and 0 an arbitrary number of times. We also assume that p(x; 8) > 0 for all x e X. Furthermore, differentiation V0(. ) (= ( 0 ( . ) / 0 0 1 , . . . , 0 ( 9)lOOn)T) with respect to 0 and integration f dx with respect to x are assumed to be interchangeable for any expression of p(x; 8) (as long as the integration exists). The probability density p(x; 8) is not defined for 0 g S. For the convenience of analysis, however, we extend it to 0 g S in such a way that p(x; 0 + AO) = p(x; 8) + 0 ( 5 8 ) 2 for all 0 e S and A0 e TO(,S)• Here, TO(S ) is the tangent space to manifold $ at 0 (see Section 3.2.1); O(A0) 2 denotes terms of degree 2 or higher in the components of A0. In intuitive terms, the probability density is "constant in the normal direction" to S in ~r~. This assumption implies the identity

v0p(.; 0) e TO(S).

(3.112)

Define an m-vector random variable I by I = V 0 logp(x; 8).

(3.113)

This vector is called the score of x with respect to the parameter 8. Since lSOrthogonality is defined in the statistical sense as having no correlation. We omit the details.

Chapter 3.

84

Probabilities and Statistical Estimation

p(x; 0) is a probability density, the normalization condition

xp(x; O)dx

- 1

(3.114)

holds for any 8 E $. It follows that if 8 is perturbed into 8 + A0 in such a way that 0 + A0 E 8, the first variation of the left-hand side of eq. (3.114) must be 0. The constraint 0 + A0 E 8 requires A0 E TO(S ) to a first approximation. If we use the logarithmic differentiation formula V0P(X; 0) =

p(x;

0)V 0 logp(x; 0),

(3.115)

the first variation of the left-hand side of eq. (3.114) is

fxp(x;

0)(V0 logp(x; 0),

AO)dx -

(El/f, A0),

(3.116)

where we have invoked the assumption that differentiation and integration are interchangeable. Since eq. (3.116) must vanish for an arbitrary A0 E TO(S ), we conclude that E[l] E TO(S) • However, eq. (3.112)implies that E[l] E TO(S ). It follows that the score l is a random variable of mean 0:

E[l] : The

Fisher information matrix

0.

(3.117)

is defined by

j-

(3.11S)

Taking the expectation of the identity 02 log p

OOiOOj

0 log p 0 log p

OOi

OOj

1

02p

I P O0~OOj'

(3.119)

and noting that differentiation and integration are interchangeable, we obtain

/0,.o,. /,lo,.o.o, / o, pdx OOiooPpdx = Ppdx + OOi

OOj

OOiOOj

02 - - fx liljpdx -~ OOTOOj/xPdX - -E[lilj].

(3.120)

Hence, if an (mm)-matrix L is defined by L - - V ~ logp(x; 0),

(3.121)

where V~(. ) denotes a matrix whose (ij) element is 02( 9)/OOiOOj, the Fisher information matrix is expressed in the following form:

j = E[L].

(3.12z)

3.5.

General Statistical Estimation Since

85

I E TO(S ), we have P~J-

J P ~ - J,

(3.123)

where P 0s is the projection matrix onto TO(N ). Hence, the rank of J is at most rn'. We say that the distribution p(x; O) is regular if J has rank m, which means that I can take all orientations in TO(S ) if x is appropriately chosen. In this book, we consider only regular distributions. Since the range of J coincides with TO(S ) for a regular distribution, the following identity holds: J JJ - J - P~. (3.124) -

If the distribution is locally Gaussian and has the probability density given by eq. (3.49}, the score 1 and the Fisher information m a t r i x J have the following form: l = E-(x~), /., = J = 2Y-. (3.125)

3.5.2

Unbiased estimator and Cramer-Rao lower bound

Let x E A" be a sample from a distribution which has a probability density

p(x; O) parameterized by 0 E $. Let ~}(x) be an estimator of 0, i.e., a function of x that returns an estimate of 0. The estimator ~}(x) is assumed to satisfy the constraint on 0: 0(x) E S for any x E X. The estimator {}(x) is unbiased if 19

P~ / ( O ( x ) - O)p(x; O)dx - O, ax

(3.126)

where P0s is the projection matrix onto TO(S) (see eq. (3.29)). The covariance matrix of the estimator 0(x) is defined as follows (see eq. (3.30))"

ViOl = Po~/;(0(~)

- o)(o(x) - o)Tp(x;

o)~P~.

(3.127)

Since eq. (3.126) holds identically for any 0 E $, the first variation of the left-hand side of eq. (3.126) must be 0 if 0 is perturbed into 0 + A0 for AO e TO(S ). If we use the logarithmic differentiation formula (3.115) and interchange differentiation and integration, the first variation of the left-hand side of eq. (3.126) can be written as

= --AO + E[P~O(x)I-C]AO, 19It seems that unbiasedness can be defined by El0] = 0, but is "curved" (see Section 3.2.2).

(3.128)

E[O]may be outside S if S

86

Chapter 3.

Probabilities and Statistical Estimation

where we have employed the approximation P~+AoE[O

-

01 ~ 0. The exact

expression for PO+AoE[O s - 0] involves the second fundamental form of the manifold $ at 0. Roughly speaking, it has the order of a2AO/R 2, where a is the "standard deviation" of 0 in $ and R is the "radius of curvature" of $ (we omit the exact analysis). Here, we simply assume that the manifold $ is sufficiently "flat" in the domain within which the distribution of 0 is localized. Throughout this book, this is always assumed whenever we talk about local distributions. Eq. (3.128) must vanish for an arbitrary A0 e T0($ ). Since E[P~OI T] P~OE[I] T - O, this condition is written as -

A0 e $.

-

(3.129)

If we let A0 - P~Ae, then A0 E S for an arbitrary Ar E 7~m. Hence, the identity E[P~(O - O)(P~I)TIAE - P~Ae (3.130) must hold identically for e E ~m. eq. (3.130) implies

Since I E T0($ ) and hence P~l - l,

E[P~(O - 0)1 T] - P~.

(3.131)

Combining this with eqs. (3.127) and (3.118), we obtain the following relationship:

T ( ]_

V[0] P0s

p S J

)

9

(3 132) .

Since the left-hand side is positive semi-definite, the following matrix is also positive semi-definite (see Section 2.2.3)" (pS

,-

_j-)(

,-)-(

V[0]

V[O]-J-

,-). (3. 33)

Here, we have assumed that the distribution is regular and hence eqs. (3.123) and (3.124) hold. Since J - is positive semi-definite, the positive semidefiniteness of eq. (3.133) implies V[0]_ J - ,

(3.134)

where A __ B means A - B is positive semi-definite. Eq. (3.134) is called the Cramer-Rao inequality and gives a lower bound, called the Cramer-Rao lower bound, on the covariance matrix V[0] of an arbitrary unbiased estimator 0. If this bound is attained, the estimator 0 is said to be efficient.

3.6.

Maxinmm Likelihood Estimation

87

Suppose N independent samples Xl, -.-, X N are observed from a distribution whose probability density is p(x; 0). Let 0(ix, ...,iN) be an estimator of 0. If we consider the direct sum -- Xl @''"

the independence of x l,

..., X N

(3.135)

O iN,

implies that 5: has the probability density

/5(5:; 0) -- p(xl; 0 ) . .

"p(xN; 0).

(3.136)

Since the estimator ~}(Xl, . . . , i x ) can be viewed as an estimator 0(5:) of 5:, the argument described earlier can be applied. The score of 5: is N

l-

V 0 log/5(5:; 0 ) -

N

V0 E

logp(x~; 0 ) -

ct=l

E

l~,

(3.137)

a=l

where l~ is the score of x~. Since {x~} are independent, the Fisher information matrix of 5: is T

[ I - E[

l~

l#

a=l

]- E

/3=1

E[l~l~] - N J,

(3.138)

c~,f~=l

where J is the Fisher information matrix for p(x; 0). Cramer-Rao lower bound is given in the following form:

Consequently, the

1 v[0] ~_NJ-.

(3.139)

In particular, if 0 is a scalar, we have V[0] >

3.6

3.6.1

1

1

NE[(O logp/O0) 2] = -NE[O z logp/O02]"

(3.140)

Maximum Likelihood Estimation

Maximum likelihood estimator and the exponential family

Given a sample x E ,Y from a distribution which has a probability density

p(x; O) parameterized by 0 E 8, the maximum likelihood estimator 0 is the value of 0 that maximizes the likelihood p(x; 0), i.e., the probability density viewed as a function of 0 by substituting the sampled value. Hence, the maximum likelihood estimator 0 is the solution of the minimization JThe probability density in preceding section.

-21ogp(x; 0) --+ min.

(3.141)

p(x; O) is assumed to have the properties described

88

Chapter 3.

Probabilities and Statistical Estimation

In order to distinguish variables fi'om their true values, we regard 0 as a variable and write its true value as 0. With the expectation that the maximum likelihood estimator 0 is close to 0, we write 0-0+A0.

(3.142)

The constraint 0 E S requires A0 E To(S ) to a first approximation. Substituting eq. (3.142) into the fimction J in eq. (3.141) and expanding it in the neighborhood of 0, we obtain _

J = - 2 logp(x; 0) - 2(1, A0) + (A0, LA0) + O(A0) 3,

(3.143)

where I is the score defined by eq. (3.113) and L is the matrix defined by eq. (3.121). The bar indicates that the value is evaluated at 0. The assumption (3.112) implies that P ~ ] , = LPoS - L,

(3.144)

where P ~ is the projection matrix onto T0(S ). It follows that the rank of L is at most the dimension m ~ of the tangent space T0(S ). Here, we assume that the rank of L is exactly rn ~ so as to guarantee the unique existence of the value of 0 E $ that minimizes J. Then, the range of L coincides with T0(S ). If the term O(A0) 3 in eq. (3.143) is ignored, the function J is minimized under the constraint A0 E T0(S ) by the following value (see eq. (2.137)): A0 - - L - i .

(3.145)

It follows that the maximum likelihood estimator is given by 0 - 0 + A0. From eqs. (3.117) and (3.144), we see that P x o E [ O - 0 ] - 0, which means that the maximum likelihood estimator is unbiased in the first order 2~ A probability distribution which has a probability density p(x; O) parameterized by 0 is said to belong to the exponential family if p(x; O) can be expressed in terms of a vector function f ( x ) and scalar functions C(O) and g(x) in the form 21

p(x; O) -- C(O) exp[(f(x), O) + 9(x)].

(3.146)

Many probability distributions that appear in practical applications have probability densities in this form, and the Gaussian distribution is a typical example 22. For a distribution of the exponential family, the matrix L 2~ proviso "in the first order" means t h a t the result is obtained by ignoring high order terms in A0. 21If the exponent on the right-hand side of eq. (3.146) has the form (f(x), h(0)) + g(x) for some vector function h ( . ), the distribution is said to belong to the curved exponential family. If h(0) is taken as a new parameter, i.e., if rI = h(0) can be solved for 0 in the form 0 = t(rl) , the distribution belongs to the exponential family with p a r a m e t e r rI. 22Beside the Gaussian distribution, the exponential family includes such distributions as the Poisson distribution, the binomial distribution, the gamma distribution, the beta distribution, and the Poisson-gamma distribution.

3.6.

Maximum Likelihood Estimation

89

defined by eq. (3.121) does not depend on x and hence is equal to the Fisher information matrix J (see eq. (3.122)). From eq. (3.145), we see that the covariance matrix V[0] of the maximum likelihood estimator 0 is given as follows (see eqs. (3.127)and (3.144)): V [ 0 ] - L-E[~q-]L - - J - J J - - J - .

(3.147)

This means that the Cramer-Rao lower bound is attained (see eq. (3.134)). Thus, the maximum likelihood estimator is efficient in the first order if the distribution belongs to the exponential family.

3.6.2

Asymptotic behavior

If N independent samples X l , . . . , X N are observed from a distribution whose probability density is p(x; 0), the maximum likelihood estimator ~) is the value of 0 that maximizes the likelihood p(xl; O) .. . p ( x s ; O). Namely, 0 is the solution of N

J-

-2 E

logp(x~)--+ min.

(3.148)

c~'-i

In this case, eq. (3.145) is replaced by

(s ) ~=1

(~=1

where l~ and L~ are, respectively, the score l and the matrix L for x~ evaluated at 0. Hence, the covariance matrix V[0] of the maximum likelihood estimator 0 - 0 + A0 is written in the following form:

E

ViOl = E[

c~=l

~=1

3,=1

5=1

-l. (3. 50)

Matrices [,1, ..., I,N are random variables that belong to the same distribution, and their common expectation is the Fisher information matrix J . Hence, the following law of large numbers holds for a sufficiently large N (see eq. (3.9))" N

1 ~ E

t ~ ,,~ J .

(3.151)

c~=l

From this and the independence of each x~, we can write eq. (3.150) as follows (see eqs. (2.81 ) )" N

1

N

[lj~-lJc~,/~=l

N2Z c~--1

J - J J - - -N J - "

(3.152)

90

Chapter 3.

Probabilities and Statistical Estimation

This means that the Cramer-Rao lower bound is attained in the asymptotic limit (see eq. (3.139)). Thus, the maximum likelihood estimator is asymptotically efficient 23. We also observe the following (we omit the details)" 9 An estimator 0 of 8 is said to be consistent if 0 ~ 8 as N -+ co. Since eq. (3.152)implies that V[0] ,.~ O(1/N), the maximum likelihood

estimator consistent. 9 Since 11, ..., IN are independent random variables of mean 0 (see eq. (3.117)) and have the same distribution, the central limit theorem N (see Section 3.3.1) states that ~ = 1 l ~ / N is asymptotically Gaussian. It follows from eq. (3.149) that the maximum likelihood estimator is

asymptotically a Gaussian random variable of mean 0 and covariance matrix J - IN. 3.7

3. 7.1

Akaike

Information

Criterion

Model selection

Suppose N data x l, ..., x N are observed. A statistical test is a procedure for judging if they can be regarded as independent random samples from a particular distribution with probability density p(x); the X2 test is a typical example (see Section 3.3.4). If the data are known to be independent random samples from a distribution with probability density p(x; O) parameterized by O, the procedure for determining the parameter 8 that best explains the data is called statistical estimation; maximum likelihood estimation is a typical example (see Section 3.6.1). But how can we guess a parameterized probability density p(x; 8)? In other words, how can we judge if the data can be regarded as independent random samples from a distribution with probability density pl(x; 8), or with probability density p2(x; 8), or with other probability densities? A parameterized probability density p(x; 8) is called a (statistical) model of the distribution. In order to select a best model, we need a criterion that measures the "goodness" of a particular model. If we adopt a particular model p(x; 8) and apply maximum likelihood estimation, the parameter 8 is determined by maximizing l I N (~=1 p ( x ~ ; 8) o r N 1 log p(x a; 0). Let ~ be the resulting maxiequivalently minimizing - 2 ~ 4= mum likelihood estimator of 0. Let us call n

J[{x~ }, 0] - - 2 ~

logp(x~; ~)

(3.153)

o~--1

23The effect of the neglected high order terms in A0 converges to 0 as N --+ c~ under mild regularity conditions. Hence, the proviso "in the first order" can be dropped. We omit the details.

3.7.

Akaike Information Criterion

91

simply the residual. A good model is expected to have a large likelihood l-I~'~=l p(xa; 0), thereby a small residual J[{x~},0]. Hence, the residual appears to be a good criterion for model selection. However, since 0 is determined so as to minimize the residual for the current data {x a}, the residual can be made arbitrarily small, say, by assuming that x can take only N values x l , .... , x u . Such an artificial model may explain the current data very well but may be unable to predict the data to be observed in the future. This * ... , x N * be independent ranobservation leads to the following idea. Let Xl, dom samples to be observed in the future; they are assumed to have the same distribution as the current data x l, ..., X g. For a good model, the residual n

J[{x,~}, 0] - - 2

logp(x,~, 0)

(3.154)

o~1

for the future data { x ; } should be small. Since the future data { x ; } and the maximum likelihood estimator 0, which is a function of the current data {x~}, are both random variables, the above residual is also a random variable. In order to define a definitive value for the model, we take expectation and consider I - E*[E[J[{x~}, 0]]]

(3.155)

where E*[.] and E[.] denote expectation with respect to the future data {x~} and the current data {xa }, respectively. We call I simply the expected residual and regard a model as better if the expected residual I is smaller.

3. 7.2

Asymptotic expression for the expected residual

As in Sections 3.5 and 3.6, we assume that the data x l, ..., x N are n-vectors sampled from an hi-dimensional manifold X C ~ n . The model parameter 0 is assumed to be an m-vector constrained to be in an ml-dimensional manifold S C 7~m. Hence, the model p(x; O) has rn' degrees of freedom. We also assume that the model p(x; 0) contains the true probability density. Suppose the true model is p(x; 0), and let 0 be the maximum likelihood estimator of 0. Writing - 0 + A0 and expanding logp(x~; 0) in the neighborhood of 0, we obtain -, A 0 ) l o g p ( x ; ; 0) - logp(x~; 0- ) + (1,~,

~1 ( A 0 , L : A 0 )

+ O(A0) 3

(3.156)

where the score l~ and the m a t r i x / ~ are defined as follows (see eqs. (3.113) and (3.121)):

i: -

log p(. ;; 0),

/,~ = - V ~ logp(x*; 0).

(3.157)

92

Chapter 3.

Probabilities and Statistical Estimation

Ignoring higher order terms and noting that A0 does not depend on {x~}, we obtain from eqs. (3.155) and (3.156) N

I -- Z (-2E*tlogp(x*~; 0)1- 2(E*[l*], ElAn}I)-t- El(A0, E*[L*IA~})]). o~--1

(3.158) Since {x;} and {x~} have the same distribution, we have

E*[logp(x*~;0)1 - E[logp(x,~; 0)1.

(3.159)

Recall that E * [ l : ] - 0,

E * [ L : ] - J,

(3.160)

where J is the Fisher information matrix (see eqs. (3.117), (3.118), and (3.122)). It follows that eq. (3.158) can be rewritten as N

I - -2E[Z

logp(x~; 0)] +

E[N(AO, JA0)].

(3.161)

ct=l

Expanding log p(x~; 0) in the neighborhood of 0, we obtain -

1

logp(x~; 0) - logp(x~; 0) - (l~, A0) - ~(A0, s

+ O(A0) 3, (3.162)

where l~ and I,a are defined by 1,~ = V 0 logp(x,~; 0),

L,~ - - V ~ logp(x,:,; 0).

(3.163)

If we put .[,,~ -- - V ~ logp(xo~; 0),

(3.164)

I,~ - L~ + O(A0).

(3.165)

we have Substituting this into eq. (3.162) and summing it over c~ - 1, ..., N, we obtain N

Z

N

N

logp(xa; 0) - Z logp(x~; 0) - ( Z 1~' A0)

c~=l

~=I

2

4=1

~ ~ L~

AO) + O(AO)3.

(3.166)

c~=l

Since the maximum likelihood estimator {} maximizes ~ =N1 log p(x o~; 0 ) , w e see that ( Z lo,, A0) -- (V 0 ot=l

logp(xo,; 0) ce=l

, A0) -- 0

(3.167)

3.7.

Akaike Information Criterion

93

for any admissible variation A~} e T~)($). If we recall that eq. (3.151) holds for N ,,~ oo (the law of large numbers) and ignore higher order terms in A~} in eq. (3.166), we have N

E

N

logp(xa; 0) -~ E

c~=l

logp(x~;/)) - N N ( A / ) , JA/})

(3.168)

c~=l

for N ,,~ oe. Substituting this into eq. (3.161), we obtain N

I ,,~ - 2 E [ E

logp(x~;/})1 + 2NE[(A/},

JAb)l.

(3.169)

ce--1

As shown in Section 3.6.2, the deviation A0 of the maximum likelihood estimator is asymptotically a Gaussian random variable of mean 0 and covariance matrix J - / N (the central limit theorem). We assume that the distribution is regular, so the Fisher information matrix J has rank m ~ (see Section 3.5.1). It follows that (A~), ( J - / N ) - A O ) - N ( A 0 , J A 0 ) is asymptotically a X2 variable of m' degrees of freedom (see eq. (3.61)). Hence, we have E[N(A/}, J A 0 ) ] ,,~ m'

(3.170)

for N ,,~ c~. Thus, eq. (3.169) can be expressed in the form

I ,,, E[J[{x~}, t}]] + 2m'.

(3.171)

This means that if we define

A I C - J[ { x ~ }, 0] + 2m',

(3.172)

this is an unbiased estimator of the expected residual I for N ~- oe. This estimator is called the Akaike information criterion, or A I C for short, and can be used as a measure of the goodness of the model; the predictive capacity of the model is expected to be large if AIC is small. According to this criterion, a good model should not only have a small residual but at the same time have a small degree of freedom; otherwise, one can define a model that fits the current data arbitrarily well by increasing the number of the model parameters.

Chapter 4 Representation of Geometric Objects This chapter discusses representations of geometric objects in two and three dimensions. First, representations of points and lines in two dimensions are described, and their error behavior is characterized in terms of their covariance matrices. Similar analysis is done for points, lines, and planes in three dimensions. Then, incidence relations are described in terms of the representations of the objects involved, and the error behavior of intersections and joins is analyzed. Geometric properties of conics and quadrics are also discussed. Finally, the coordinate transformation of three dimensional objects and their perspective projection onto the image plane are summarized. 4.1

4.1.1

Image

Points

and

Image

Lines

Representation of image points

We call points and lines defined in two-dimensional images image points and image lines, respectively, to distinguish them from points and lines in threedimensional scenes, which we call space points and space lines, respectively. An image point is represented by its image coordinates (x, y) with respect to a fixed Cartesian coordinate system. An alternative representation is the use of the 3-vector x -

(x) y

.

(4.1)

If the image plane is viewed as a two-dimensional projective space 1, the three components of the 3-vector x can be interpreted as the homogeneous coordinates of the image point it represents. By definition, a 3-vector x represents an image point if and only if (k, x) = 1,

(4.2)

where k - (0, 0, 1)T. The distance of the image point x from the image origin o is d - V/x 2 + y2 _ V/llxll2 _ 1. (4.3) Consider an X Y Z Cartesian coordinate system with origin O in the scene. We call the plane placed parallel to the X and Y axes in distance f from the 1A two-dimensional Euclidean space can be identified with a two-dimensional projective space if ideal image points ("image points at infinity") are added. Ideal image points are represented by homogeneous coordinates whose third components are 0. 95

96

Chapter 4.

x

Representation of Geometric Objects

.P(X,Y,Z)

~f-~

(a)

~

P

(b)

Fig. 4.1. (a) Perspective projection. (b) Idealized camera model.

X Y plane the image plane. Define an xy coordinate system on the image plane in such a way that the x- and y-axes are parallel to the X- and Y-axes, respectively, and the origin o is on the Z-axis. If viewed from the origin O, a space point P with coordinates (X, Y, Z) is seen at p on the image plane Z = f with image coordinates (x, y) given by X

Y

(4.4)

We call the origin O and the constant f the viewpoint and the focal length, respectively. Eqs. (4.4) define a mapping, called perspective projection, from the three-dimensional X Y Z scene to the two-dimensional xy image plane (Fig. 4.1a). This is an idealized model of camera imaging geometry (Fig. 4.15)" the viewpoint O corresponds to the center of the lens; the Z-axis corresponds to the optical axis of the lens; the focal length f corresponds to the distance from the center of the lens to the surface of the film 2 (or the photo-cells for video cameras). In the following, we use f as the unit of length, so the image plane can be written as Z - 1. If the above geometry of perspective projection is assumed, the vector representation (4.1) can be thought of as identifying an image point p with its position x in three dimensions (Fig. 4.2a). Let us call the space line that starts from the viewpoint O and passes through image point p the line of sight of p. The vector x that represents image point p indicates the orientation of the line of sight of p. Suppose observation of an image point is susceptible to image noise. Let (x, y) be the observed position. If image noise (Ax, Ay) is randomly added, this position is perturbed into (x + Ax, y + Ay). The image noise is assumed to be very small3: IAxl KK 1, IAyl K< 1. In the vector representation, the observed value x is randomly perturbed into x + A x by image noise Ax. If Ax 2The focal length f thus defined depends on the position of the object on which the camera is focused. Hence, it is generally different from the optical focal length of the lens; the two values coincide only when the object in focus is infinitely far away. 3Since we take f as the unit of length, this means that image noise is very small as

4.1.

Image Points and Image Lines

X

97

x

X

x

n

O Y

(b)

(~)

Fig. 4.2. (a) Representation of an image point. (b) Representation of an image line. is regarded as a random variable of mean 0, the uncertainty of the value x is characterized by the covariance matrix V[x] = E [ A x A x T ] . Since the image noise A x is orthogonal to k, it is constrained to be in the two-dimensional subspace {k}L~, the set of all vectors orthogonal to k. Consequently, the covariance matrix V[x] is singular; its null space is the one-dimensional subspace { k } L generated by k. The fact that rankV[x] - 2 simply states that an image point is represented by a 3-vector but has only two degrees of freedom. In other words, the rank of the covariance matrix indicates the degrees of freedom of the representation. E x a m p l e 4.1 If Ax and Ay are independent Gaussian r a n d o m variables of mean 0 and variance e2, the covariance matrix of x is given as follows (see eqs. (3.27)): V[x] -

Here, P k = I eq. (2.46)).

4.1.2

0 0

e2 0

0 0

- e2Pk .

k k T is the projection matrix onto the X Y

(4.5)

plane (see

Representation of image lines

An image line l is represented by its equation

Ax + By + C = 0.

(4.6)

Since the coefficients A, B, and C are determined only up to scale, we impose the normalization A 2 + B 2 + C 2 = 1. Then, an image line I is represented by compared with the focal length f. The focal length f is usually comparable to or much larger than the physical size of the image. So, this assumption is satisfied if the image noise is much smaller than the size of the image. For an image of 512 • 512 pixels, for example, this assumption is well satisfied if the image noise is less than, say, five pixels.

Chapter 4.

98

Representation of Geometric Objects

a unit 3-vector .

-

B

.

(4.7)

C This representation is not unique: n and - n represent the same image line. If the image plane is viewed as a two-dimensional projective space, the three components of the 3-vector n can be interpreted as the homogeneous coordinates of the image line it represents. Since A and B in eq. (4.6) cannot be zero at the same time 4, an arbitrary 3-vector n represents an image line if and only if I I n ] l - 1,

n ~: + k .

(4.8)

The distance of the image line represented by n from the image origin o is d -

ICl v/A2 + B 2

=

I(k,-)l . X/1 - (k, n) 2

(4.9)

Recall the geometry of perspective projection described in Fig. 4.1a. The lines of sight of all the image points on an image line l define a space plane. The vector representation (4.7) can be thought of as identifying the image line l with the unit surface normal n to that space plane. In fact, the space plane defined by an image line A x + B y + C -- 0 is A X + B Y + C Z = O, which has surface normal n = (A, B, C)7- (Fig. 4.2b). As in the case of an image point, observation of an image line also has uncertainty. Suppose an image line represented by n is randomly perturbed into a position represented by n - n + A n . If the error A n is regarded as a r a n d o m variable of mean 0, the uncertainty of the value n is characterized by the covariance matrix y[n] = E[AnAnT]. Since n is normalized into a unit vector, the error A n is orthogonal to n to a first approximation. Hence, the covariance matrix Y[n] is singular; its null space is {n}L. Thus, although an image line is represented by a 3-vector n, it has only two degrees of freedom: rankV[n] = 2. This description of error behavior implies that we are viewing n as a r a n d o m variable which has a local distribution (see Section 3.2.2). Since n is a unit vector, it has a distribution over S 2, a unit sphere centered at the origin. We are assuming that the distribution is sufficiently localized around n and hence its domain can be identified with the tangent plane T n ( S 2) at n (see Fig. 3.3). 4The set of all ideal image points in a two-dimensional projective space is called the ideal image line ("image line at infinity"). In projective geometry, the ideal image line can be treated just like an ordinary image line and represented by homogeneous coordinates whose first and second components are both 0.

4.1.

Image Points and Image Lines

99

X,Y

n

nl

l, Z

(~)

(b)

Fig. 4.3. (a) Incidence of image point p and image line I. (b) Parallel image lines 11 and 12.

4.1.3

Incidence, intersections, and joins

A. I n c i d e n c e

Image point p and image line 1 are incident to each other if p is on l, or 1 passes through p (Fig. 4.3a). Let (x, y) be the image coordinates of p. If image line 1 is represented by A, B, and C, image point p and image line l are incident to each other if and only if A x + B y + C - O. In other words, an image point p represented by x and an image line 1 represented by n are incident to each other if and only if (n, x) = 0, (4.10) which is simply the equation of image line I if x is regarded as a variable. The distance D(p, l) between an image point p represented by x and an image line 1 represented by n is D(p,l) -

[(n' x)l . V/1 - (k, n) 2

(4.11)

B. I n t e r s e c t i o n s

Two image lines (nl, x) = O and (n2, x) = O are parallel to each other if and only if Ini,n~, k l - 0 .

(4.12)

(Fig. 4.3b.) If image lines (hi,X) = 0 and (n2,x) = 0 are not parallel s, they intersect at a single image point. From Fig. 4.4a, we see that the vector x that represents the intersection must be orthogonal to both n l and n2. Hence, x c< n l x n2. Since (k, x) = 1, the intersection x is obtained in the following form (see eq. (2.32)): nl xll2

-

kl

(4.13)

5parallel distinct image lines can be thought of as intersecting at an ideal image point

("image point at infinity").

100

Chapter 4.

Representation of Geometric Objects

n

P2 Pl

x2

(b)

Fig. 4.4. (a) The intersection p of image lines 11 and 12. (b) The join l of image points pl and p2. Let V[Tll] and V[n2] be the covariance matrices of ~1,1 and n2, respectively. If n l and n2 are perturbed into n l + An1 and n2 -b An2, respectively, the intersection x computed by eq. (4.13) is perturbed into x + Ax accordingly. The error is to a first approximation mx

(IAn , n~, kl § Inl,n2,kl

-- Z~nl X IF~2 -I-7"/,1X Art 2 --

In1, An2, kl)~.

(4.14)

If n l and n2 are statistically independent 6, the covariance matrix V[x] E [ A x A x -r-] of the intersection x is given by

V[m]

=

1 ( n2 Inl,n2,kl

x V[~r~l] x •2

-~- 2S[7~,2 x V [ n l ] ( l o , 2 x k ) x

+ ( n 2 • k, V[nl](n2 • k ) ) x x T + n l + 2 S [ n l x V[nu](nl X k ) x T] + (nl

X V[~o,2] x X

Tl

nl

k, V[n2](nl

x

k))xxT), (4.15)

where S[. ] denotes the symmetrization operator (see eqs. (2.205)). In deriving the above expression, the identity (Ani x n j ) ( A n k x nl) T _ nj x ( A n i A n - [ ) x nt has been used (see eq. (2.43)). C. Joins

An image line (n, x) = 0 that passes through two distinct image points X l and x2 is called the join of X 1 and X 2. Since (n, x l ) - 0 and (n, x2) - 0, the vector n must be orthogonal to both Xl and x2 (Fig. 4.4b). Also, n is normalized into a unit vector. Hence, =

•

(4.16)

6We use an informal expression like this, instead of saying that the noise that arises in the observation of nl and the noise that arises in the observation of n2 are independent random variables.

4.1.

Image Points and Image Lines

101

which defines an image line as long as X l r X2. Let V[Xl] and V[x2] be the covariance matrices of image points Xl and x2, respectively. If Xl and x2 are perturbed into Xl + / k X l and x2 + Ax2, respectively, the vector n computed by (4.16) is perturbed into n + A n accordingly. The error is to a first approximation an

=

4-

Pn(Axl

x

x2 + Xl

X

AX2)

.

(4.17)

Here, P n = I - n n T is the projection matrix along n. If X l and x2 are statistically independent, the covariance matrix V[n] - E [ A n A n T] of n is given by

Pn(xl x

V[n]-

V[x2] x x, + x2 x V[Xl] x I1~1 x ~,~11~

x2)Pn

(4.18)

E x a m p l e 4.2 As shown in Example 4.1, if each coordinate is perturbed by Gaussian noise independently of mean 0 and variance e 2, the covariance matrices of Xl and x2 are V[Xl] - V[x2] - e 2 P k . Note the following identity (see eq. (2.42))" xi

x

Pk

x

xi

-

xi

(I-

x

kk T)

X Xi

= II~,,ll~x - ~,~T-

(~, x k)(~, • k) ~.

(4.19)

Let u be the unit vector that indicates the orientation of the join (n, x) = 0, and w the separation between the two image points. Their midpoint is x c = (Xl + x2)/2, so we can write x l - X c - w u / 2 and x2 - x c + w u / 2 . If the two image points are close to the image origin o, we have ll~,ll ~ 1, ~ , • k o, i - 1, 2, and IIXI X X211 ,-~ W. Using eq. (4.19), we can approximate eq. (4.18) in the form Y[n] '~ - ~

Pn - xcx~-

---~uu T

,

(4.20)

where the identities P n u - u and P n x c - X c are used. If X c is approximately perpendicular to u, we obtain the approximation P n ~ x c x ~ + u u T. Since the distance w is usually very small as compared with the focal length (which we take as the unit of length), we obtain to the following approximation: 2e 2

v[n] ~ ~ - y ~ .

(4.21)

This means that the vector n is most likely to deviate in the direction u (Fig. 4.5a); the p r i m a r y deviation pair 7 is given as follows (see eq. (3.48))"

~+ ~ N[. +

V~ W

~],

~- ~ N [ ~ - vJA~]. W

(4.22)

7The geometric meaning of the primary deviation pair will be discussed in detail in Section 4.5.3.

102

Chapter 4.

n- n

Representation of Geometric Objects

+ i

_

l

n§ ~

n

-

l+

(b)

(~)

Fig. 4.5. (a) P r i m a r y d e v i a t i o n pair. (b) D e v i a t i o n of t h e join. The two image lines 1+ and l - defined by these two vectors intersect at the midpoint of the two points, indicating that the join is very likely to pass near the midpoint in the presence of noise (Fig. 4.5b). It is also seen that the error is approximately proportional to the reciprocal of the separation w between the two image points.

4.2 ~.2.1

Space Points and Space Lines Representation of space points

A space point (X, Y, Z) is represented by a 3-vector r - (X, Y, Z)T. Alternatively, it is represented by a 4-vector

P=

X Y z

= ~el

(4.23)

1 The distance from the coordinate origin O is d = Jx~

+ y 2 + z~ = II,'lI = JIIpll ~

- 1.

(4.24)

A 4-vector p represents a space point if and only if (~, p) = 1,

(4.25)

where ~r - (0, 0, 0, 1)x (__ 0 | 1). If the three-dimensional space is viewed as a three-dimensional projective space, the four components of the 4-vector p can be interpreted as the homogeneous coordinates of the space point 8. The 8As in the two-dimensionM case, a three-dimensional Euclidean space can be identified with a three-dimensional projective space if ideal space points ("space points at infinity") are added. Ideal space points are represented by homogeneous coordinates whose fourth components are zero.

4.2.

Space Points and Space Lines

103

p-representation is very convenient for various computations, but we must be careful about the rounding effect in fixed precision computation: if X, Y, and Z are very large, the fourth component of p may be treated as effectively 0. Hence, an appropriate scaling must be applied to keep the space coordinates in a reasonable range of magnitude. Observation of a space point is susceptible to noise. Suppose the observed value r is randomly perturbed into v + A t . If A t is regarded as a random variable of mean 0, the uncertainty of r is characterized by the covariance matrix Y[r] = E [ A r A r T ] , which we assume is positive definite: rankV[r] = 3. In the 4-vector representation p = r | 1, the covariance matrix V[p] is singular and has the form

V[p]-(

V[r]

0)-V[r]|

(4.26)

Since the error Ap is constrained to be in the three-dimensional subspace {~;}~ C 7~4, the null space of V[p] is {~}L. The fact that rankV[p] = 3 states that a space point has three degrees of freedom even if it is represented by a 4-vector p.

E x a m p l e 4.3 If each component of (X, Y, Z) is perturbed independently by Gaussian noise of mean 0 and variance e2, the covariance matrix V[r] has the following form (see eq. (3.25)): Vir]

(4.27)

= e2 I .

In the 4-vector representation, the covariance matrix

V[p] = e2I | where P ~ - I -

4.2.2

~T

0 = e2P~,

V[p] has

the form (4.28)

is the four-dimensional projection matrix along ~.

Representation of space lines

Consider a space line L. Let H be the point on L closest to the coordinate origin O, and put rH -- OH. Let m be the unit vector that indicates the orientation of L (Fig. 4.6a). The space line L is represented by two 3-vectors {m, rH}; two representations {m, rH} and { - m , rH} define the same space line. A space point r is on L if and only if vector r - rH is parallel to m , or (v

-

rH) X m = 0,

(4.29)

which is simply the equation of space line L if r is regarded as a variable. Two 3-vectors {m, rH} represent a space line if and only if

Ilmll = 1,

(m, rH)

=

0.

(4.30)

104

Chapter 4.

Representation of Geometric Objects

O O

(~)

(b)

Fig. 4.6. (a) The {m, rH}-representation. (b) The {p, n}-representation. Eq. (4.29) can also be written as v x m = rH • m . Hence, the equation of a space line can alternatively be written in the form r

x p = n.

(4.31)

Since both sides can be multiplied by an arbitrary nonzero constant, we normalize the 3-vectors p and n into lip 9 nil = 1 (i.e., ]Jp]l2 + ]]n]l 2 = 1). The signs of p and n are chosen in such a way that {p, n, rH } constitute a righthanded orthogonal system (Fig. 4.6b). With this convention, a space line L is also represented by two 3-vectors {p, n}; two representations {p, n} and { - p , - n } define the same space line. Geometrically, the vector p indicates the orientation of L, and the vector n indicates the surface normal to the space plane defined by L and the coordinate origin O. The distance of L from O is

d = Ilnll = v/1- IlPll ~ _ IlPll Ilvll - r

Ilnll Ilnll ~"

(4.32)

It follows that 1

Ilvll

d

i

v/1 + d 2'

ll~ll

= -x/1 - - - +- ~ d"

(4.33)

The 3-vectors p and n can be interpreted as the homogeneous coordinates of a space line, known as the Pliicker (or Grassmann) coordinates. Two 3-vectors {p, n} represent a space line if and only if

Ilpll = + II~L]~ - 1,

(p, n) = O.

(4.34)

The { m , rH}-representation and the {V, n}-representation are related as follows: m-

N[p]

rH = p • n

'

(4.35)

IlPll ~ '

(p) N,(o rHxmm)1

430,

4.2.

Space Points and Space Lines

105

Suppose the values m and ~'H are randomly perturbed into m + A m and rH + A rH, respectively. If the errors A m and A rH are regarded as random variables of mean 0, the uncertainty of the values {m, rH } is characterized by the six-dimensional covariance matrix V i m | rH] = E [ ( A m @ A r H ) ( A m @ ArH)T], which has the following submatrices (see eqs. (3.11) and (3.12)):

vim 9 ~.] -

(

V[m] v[~.,m]

V[m, r H ] ) v[~]

(4.37)

The constraint (4.30) implies that the errors A m and ArH are constrained by (m, A m ) = 0, ( A m , rH) + (m, ArH) = 0, (4.38) Hence, the null space of V[m | rH] is the two-

to a first approximation. dimensional subspace

AfmerH = { m | 0, rH @ m}L C 7~6.

(4.39)

It follows that rankV[m | rH] -- 4; a space line has four degrees of freedom. The six-dimensional projection matrix onto H mi ~ r H is given by PX~r

-1+11r.112 m.~

I

mm T

'

(4.40)

where P m I - m m T is the projection matrix along m. In the {p, n}-representation, the error behavior is characterized by the six-dimensional covariance matrix

Vtp|

= ( V[n,p] V[pl V[p'n] )V[n] "

(4.41)

The constraint (4.34) implies that the errors Ap and A n are constrained by

(p, z~p) + (~, ~ ) to a first approximation. dimensional subspace

= 1,

(~p, ~) + (p, z ~ ) = 0,

(4.42)

Hence, the null space of V[p @ n] is the two-

H p e n - {v 9 n, n r v} ~ c n ~.

(4.43)

Again, rank V[p | n] = 4, confirming that a space line indeed has four degrees of freedom. The six-dimensional projection matrix onto Afl~en is given by

pxpe

_ ( I - pp T - n n T --2S[pn T]

-2S[p n T] ) I -- ppT _ n n T 9

(4.44)

As in the case of image lines, we are considering a local distribution (see Section 3.2.2). Since a space line is represented by two 3-vectors {m, rH}

106

Chapter 4.

Representation of Geometric Objects

with two constraints given by eqs. (4.30), the set of all space lines defines a

four-dimensional manifold S in ~6. We are assuming that the distribution is sufficiently localized around the value m | r H in S and hence the domain of the distribution can be identified with the tangent space T m e r , ( S ) to S at m | rH. It follows that the null space A/'merH and its orthogonal complement HXmerH coincide, respectively, with the normal space N m e r , ( S ) and the tangent space T m e r , ( S ) (see Section 3.2.1). The same can be said for the {p, n }-representation. The covariance matrix Vim | rH] defined by eq. (4.37) is expressed in terms of the covariance matrix V[p | n] defined by eq. (4.41) as follows (see eqs. (3.18)): 1

V [ m ] - [ipll2PmV[p]Pm,

V[m, rH] --

1 ilPl13 (ProW[p] x n + 2PmV[plprTH - PraY[p, n] x P)

= V[rH,m] T,

WitH]-

,(

IIPlI4 n x V[p] x n -

4S[(n

x

V[p]p)ri~ll

+4(p, V[plp)rHrI~- 2S[n x V[p, n] x P] - 4 S [ ( p x V[p, n]Tp)rnTl + p x V[n I x p ) .

(4.45)

Here, P m - I - m m T is the projection matrix along m. Conversely, the covariance matrix V[p | n] is expressed in terms of the covariance matrix V[m 9 rH] in the form 1

V[p ~

nl - 1 + II~ll2 Pp~n

(

V[m] V[~ x m, m]

v[~, ~ • ~] ) V[r x m]

Ppen,

(4.46)

where v[m, ~ • m]-

V[m x r ] - r

v [ m ] • ~ - v [ m , ~] • m - v [ ~ x m , m] ~,

x V[m] x r - 2 S [ r x V[m, rl x m] + m x V[r] x m. (4.47)

Here, Ppen is the six-dimensional projection matrix along p @ n, which has the form

Ppen-(

I-ppT -np T

--pnT ) T I - nn

(4.48)

4.2.

Space Points and Space Lines

107

L

m m' r I

t

L

O

(~)

(b)

Fig. 4.7. (a) Two intersecting space lines. (b) The join of two space points.

~.2.3

Incidence, intersections, and joins

A. I n c i d e n c e

A space point P and a space line L are incident to each other if P is on L, or L passes through P. If space point P is represented by r and space line L is represented by {m, rH} (or {p, n}), they are incident to each other if and only if (r - rH) X m -- 0 (or r x p -- n). The distance D(P, L) between a space point P represented by r and a space line L represented by ( v - r H ) X m =0 (orr• is

D(P, L) - lit x p -

nil =

IIPmr-

~HII-

(4.49)

Ilpll

B. I n t e r s e c t i o n s

The distance D ( L , L ' ) between a space line L represented by {m, rH} and a ! space line L' represented by { m ' , r H} is given by

lira x m'll

D(L,L')/.

II ,,

t

-

.11.

ifm x m' r (4.50) ifmx

m ' = 0.

It follows that the space lines L and L' intersect if and only if the three vectors m , m ' , and r~ - rH are coplanar and 9 m r + m ' (Fig. 4.7a), i.e., !

Im, m ' , rH -- rHI - 0,

m x m'#

0.

(4.51)

9Parallel distinct space lines (m = =Em' or p x pl = 0) can be thought of as intersecting at an ideal space point ("space point at infinity").

108

Chapter 4.

Representation of Geometric Objects

In the {p, n}-representation, eqs. (4.50) are replaced by

(V,n')+(V',n)

V'

lip • p'll

D(L,L')

'

if p x

# o,

-

nt

pt

(p, n ' ) + (p', n ) = 0,

p x p' # 0.

Illl tl- tl'lLII, n

(4.52)

-o

Eqs. (4.51) are replaced by (4.53)

The intersection r of the space lines L and L' is given by r =

(m • ~.) • (m'• Im, m', ~-I

~h) _ n • n' - (p', n)"

(4.54)

This expression is convenient for theoretical analysis but not suitable for actual computation, since both the numerator and the denominator vanish when Ira, m ' , rHI = 0 or (p', n) = 0. A more convenient expression is

(m. rl,) + (m. m')(m', r.) r - r. +

lira • m'll ~

m

1 /' Ilpll:lp, p', n ' l - (p, p')lp, p', nl = Ilpll ~ [,

IIv • p'll:

v + p • n)

(4.55)

C. J o i n s

The join of two distinct space points r l and r2 is a space line that passes through both rl and r2 (Fig. 4.7b). In the {m, rH}-representation, the join is given by

'm -- N['rl - 'r2],

'rH -

('rrt, 'r 1)'r' 2 -- ('D't, 'r'2)'r 1 IIr~ - ,'~11

(4.56)

In the {p, n}-representation,

(P)-'N[(~t

~2'P1-'P2X,rl)].

(4.57)

If the space points Vl and v2 are statistically independent and have respective covariance matrices V[rl] and V[r2], the covariance matrix of their join is computed from eq. (4.57) in the following form (see eqs. (3.18)): v [ p 9 ~] -

1 I1~1 - ~ II= + I1~ • 'rl II2 P p G n

(

V[?'I]-~-V[/'2]

V[T1] x/'2-~-V[/'2] x/'1

,.~ • v[,-~] + ,-~ • v[,,~]

,-~ • v[,~,] • ,-~ + ,-, • v[,-~] • ,-,

Ppen.

) (4.58)

Here, Pp~n is the six-dimensional projection matrix defined by eq. (4.48).

4.3.

Space Planes

109

d

O Fig. 4.8. Representation of a space plane.

4.3

~.3.1

Space Planes

Representation of space planes

A space plane H is represented by its unit surface normal n and its (signed) distance d from the coordinate origin O (Fig. 4.8): the distance d is positive in the direction n and negative in the direction - n . Hence, {n, d} and { - n , - d } represent the same space plane. The equation of a space plane that has unit surface normal n = (A, B, C) T and distance d is A X + B Y + C Z - d,

(4.59)

(n, r) = d.

(4.60)

which can be written as If the 4-vector representation p = r | 1 is used for space point r, eq. (4.60) is expressed in the form ( v , p ) = 0, (4.61) where v ~ n | ( - 1 ) . The scale indeterminacy can be removed by imposing the normalization I l v l l - 1:

v--

v/1 + a~

-d

"

Conversely, the unit 4-vector v - (Vl, v2, u3,//4) T Can be decomposed into the unit surface normal n and the (signed) distance from the coordinate origin in the form -

1,

x[

d -

-

va

-

/23

where ~ - (0, 0, 0, 1) T. As in the case of space points, we must be careful about the rounding effect in fixed precision computation: if the distance d is very large, the first three components of v may be treated as effectively 0.

110

Chapter 4.

Representation of Geometric Objects

Hence, an appropriate scaling must be applied to keep the distance d in a reasonable range of magnitude. As we did for image lines and space lines, we assume local distributions for all variables; the error behavior of a space plane is characterized by the four-dimensional covariance matrix

V[n | d] - ( V[n] V[n,d] ) V[n,d] V[d]

'

(4.64)

where V[n, d] (- V[d, n] T) is a a-vector and V[d] is a scalar. Since n is a unit vector, the null space of V[n | d] is {n | 0}L. If the v-representation, the covariance matrix V[~,] has the form

1

V[v] - --------gPv 1+ d

(

V[n] -V[n,d]) -V[n, d] V[d] P~'

(4.65)

where Pu - I - u u -r is the four-dimensional projection matrix along u. Since u is a unit vector, the null space of V[u] is {U}L. Whichever representation is used, a space plane has three degrees of freedom: rankV[n | dl - rankV[u]

--3.

If let v -+ v + A v in eqs. (4.63), the unit surface normal n and the distance d are respectively perturbed to a first approximation by

A n - VII + d 2P v

Ab'l)

Au2 A~,3

,

Ad - - v / ( i + d2)aAv4,

(4.66)

where we have used eqs. (3.17). It follows that for a given covariance matrix V[~,] the the corresponding covariance matrix Y[n | d] can be computed as follows"

V[n]-

V[n,d]4.3.2

(1 +

d2)Pv

- ( 1 + d2)2pv

V[V]ll V[v]31 V[v]14 ) V[v]24 , V[V]34

)

V[u]21

V[d]-

(1

+ d2)3V[v]44.

(4.67)

Incidence, intersections, and joins

A. I n c i d e n c e

1. A space point P and a space plane H are incident to each other if P is on H, or H passes through P. A space point r (or p) and a space plane represented by {n, d} (or u) are incident to each other if and only if

4.3.

Space P l a n e s

111 n

L

0 Fig. 4.9. Incidence of space plane II and space line L. (n, r ) = d (or (v, p) = 0). T h e distance D ( P , H) b e t w e e n a space point P r e p r e s e n t e d by r (or p) a n d a space plane r e p r e s e n t e d by {n, d} (or

. ) is D ( P , II) - ](n, r) - d] -

](v, p)]

v/i

.

(4.68)

-

2. A space line L and a space plane H are incident to each other if L is on H, or H passes t h r o u g h L (Fig. 4.9). A space line (r - rH) • m = 0 and a space plane (n, r) = d (or (v, p) = 0) are incident to each other if a n d only if (n, m ) = 0,

(n, rH) = d,

(4.69)

(v, rH | 1) = 0.

(4.70)

or equivalently 1~ (v, m | 0) = 0,

B.

Intersections

1. T w o distinct space planes ( n l , r ) = d l and (n2, r ) = d2 intersect along a space line as long as 11 n 1 ~ -~-n 2 (Fig. 4.10a). In the { m , rH}r e p r e s e n t a t i o n , the intersection is m -- N [ n l • n2],

T'H - -

+

]]n I X n 2]]2

"

(4.71)

l~ first of eqs. (4.70) can be given the interpretation that the ideal space point ("space point at infinity") m | 0 of the space line (r - rH) • rn -- 0 is on the space plane (L,,p) = 0. 11 Parallel distinct space planes can be thought of as intersecting along an ideal space line ("space line at infinity").

112

Chapter 4.

Representation of Geometric Objects

n2

L /

0

0

(a) (b) Fig. 4.10. (a) The intersection L of space planes Ha and 122. (b) The intersection P of space line L and space plane 12. In the {p, n}-representation, \

•

nx - din2

)

(4.72)

Suppose the two space planes (•1, ~') dl and (n2, r) = d2 are statistically independent. Let V[nl | dl] and V[n2 | d2] be their respective covariance matrices. The covariance matrix of the intersection is computed from eq. (4.72) in the form -

vtp |

(\

-

Vnp

')

Vnn /

(4.73)

where

V~p

--"2

X W [ - 1 ] x ~'12 "~KI 1 X V[~'~2] x - 1 ,

V p n -- -d2n2 x V [ n l ] - d l n l • V[n2] + n 2 x V[nx,dx]n: +

~"$1 X

V~l, rl -- d22V[~'ll]--[-- d 1 2 V [ ~ l , 2 ] -

V[n2,d2]n? = VYnp,

2d2S[V[rll,dl]~'lT2]

-2d1S[V[~9,2,d21717] ...[-V [ d l ] t l 2 ~ l , : Here, P p e n eq. (4.48).

--[- V[d2]irll~,l,?.

(4.74)

is the six-dimensional projection matrix defined by

2. A space plane (nn, r) - d and a space line (r - rH) X m -- 0 (or r x p = nL) intersect at a space point unless 12 (nri, rn) -- 0 (or (nn, p) -- 0) 12A space plane and a space line that are parallel and not incident to each other can be thought of as intersecting at an ideal space point ("space point at infinity").

4.4.

Conics

113 t

L /7

O O

O

(b)

(c)

Fig. 4.11. (a) The join II of three space points P0, P1, and P2. (b) The join II of space line L and space point P. (c) The join H of space lines L and L'. (Fig. 4.10b). Their intersection r is given by

d--(nn,rH)

l

(Inn,nL,PlTdllpll 2

)

,r --- ,r H +

(4.75) C. J o i n s

1. Three distinct space points r0, /'1, and r2 define a space plane, called their join (Fig. 4.11a), represented by =

-

•

-

d = (n, r0).

(4.76)

2. A space line ( r - rH) X m -- 0 and a space point r p define a space plane, called their join (Fig. 4.11b), if they are not incident to each other. The join is represented by d = (n, vp).

(4.77)

3. Two intersecting space lines (r - VH) X m = 0 and (v - V~H) X m ' = 0 define a space plane, called their join (Fig. 4.11c), represented by

n = N [ m • m'],

4.4

4.4.1

d = (n, ell).

(4.78)

Conics

C l a s s i f i c a t i o n o f conics

A conic (sometimes referred to as a conic locus or conic section) is a curve on a two-dimensional plane whose equation has the form

Ax 2 + 2 B x y + Cy 2 + 2(Dx + Ey) + F = O.

(4.79)

114

Chapter 4.

Representation of Geometric Objects

If an image point (x,y) is represented by 3-vector x - (x,y, 1) T , eq. (4.79) can be written in the form -0, (4.80) where Q is a (33)-matrix defined by

Q-

B D

C E

E F

.

(4.81)

A conic does not necessarily define a curve. A conic is singular if it defines two (real or imaginary) lines 13 (including one degenerate line); otherwise, the conic is nonsingular. It is easy to prove the following: 9 Conic (x, Q x ) - 0 is nonsingular if and only if the matrix Q is nonsingular. 9 A nonsingular conic (x, Q x ) = 0 defines a real conic (an ellipse, a parabola, or a hyperbola) if and only if the signature of Q is (2,1) or (1,2) (see Section 2.3.4). If the sign of Q are chosen so that d e t Q < 0, the type of conic (x, Q x ) = 0 is classified as follows: 1. If det Q = 0, the conic defines two (real or imaginary) lines (including one degenerate line). 2. If det Q 7~ 0, then (a) if A C - B

2 > 0, then

i. if A + C > 0, the conic is an ellipse, ii. if A + C < 0, the conic is an empty set (an imaginary ellipse), (b) if A C -

B 2 - 0, the conic is a parabola,

(c) if A C -

B 2 < 0, the conic is a hyperbola.

Given a nonsingular conic (x, Q x ) - 0 and an image point xp, the image line (np, x) - 0 for np - :l:N[Qxp] (4.82) is called the polar of the image point xp with respect to the conic (x, Q x ) -- O. Conversely, given a nonsingular conic (x, Q x ) - 0 and an image line (np, x) - O, the image point

xp = (k, Q - l n p )

(4.83)

is called the pole of the image line (np, x) - 0 with respect to the conic (x, Q x ) - O. We observe the following: 13If two imaginary lines intersect, their intersection is an isolated real point.

4.4.

Conics

115 P Ii Pl P2

Fig. 4.12. Image point p is the pole of image line l, and image line l is the polar of image pont p. The tangent points pl and p2 are the poles of the tangents ll and 12, respectively, and the tangents 11 and 12 are the polars of the tangent points pl and p2, respectively. 9 A point xp is on conic (x, Qx) = 0 if and only if its polar (np, x) = 0 is tangent to the conic (x, Qx) = 0 at xp. 9 Let xp be the pole of an image line (np, x) = 0 with respect to conic (x, Qx) = 0. If the image line (np, x) = 0 has two intersections Xl and x2 with the conic (x, Qx) = 0, the join of Xp and xi is tangent to the conic (x, Qx) = 0 at xi, i = 1, 2 (Fig. 4.12).

4.4.2

Canonical forms of conics

A real conic can be reduced to its canonical form by an appropriate translation and rotation of the xy coordinate system. If the coordinate system is rotated by angle 0 and translated by (a, b), the transformation of coordinates can be written in the following form14" cos0

x t-Ax,

A-

sin0 cos0 0

-sinO 0

-acos0-bsin0 asin0-bcos0 1

\ ) .

(4.84)

Hence, x

--

A - 1x ! ,

A

-

1

__

cos0 sin 0 0

-sin0 cos 0 0

a ) b . 1

(4.85)

By this coordinate transformation, conic (x, Q x ) = 0 is transformed to conic (x', Q'x') = 0 for Q' =

(A-~ )-CQA-1.

(4.86)

14The set of matrices that have the form shown in eqs. (4.84) is closed under matrix multiplication and called the group of two-dimensional Euclidean motions.

116

Chapter 4.

Representation of Geometric Objects

This is the congruence transformation of Q by matrix A -1 (see eq. (2.154)). Since the signature is preserved by a congruence transformation (Sylvester's law of inertia; see Section 2.3.4), a real conic is always transformed to a real conic 15 . Consider a nonsingular conic (x, Q x ) - 0. If the scale and the sign of Q are chosen so that det Q - - 1 , the conic has the following canonical form. Let (A + C) 4- v/(A + C) 2 - 4(AC - B 2) 2

)~i, )~2 --

.

(4.87)

I. If A C - B 2 ~= O, then AI and A2 are both nonzero. Let

a-

11'

b-

g.

(a) If p a l > 0 and pA2 > 0, the conic is an ellipse (Fig. canonical form X2

4.13a) with

y2

a--7 + ~-g -- 1.

(4.89)

This ellipse has radii a and b in the x and y directions, respectively. (b) If #A1 and #A2 have opposite signs, the conic is a hyperbola (Fig. 4.13b) with canonical form x2 y2 + ~ ~: ~- - I,

(4.90)

where the upper signs are for p a l > 0 and #A2 < 0, and the lower signs are for p a l < 0 and pA2 > 0. In the former (latter) case, this hyperbola intersects with the x-axis (y-axis) at x - a (y = b) and has two asymptotes y = +(b/a)x. (c) If p a l < 0 and pA2 < 0, the conic is an imaginary ellipse with canonical form x2 y2 -

a--g

b2 = 1.

2. If A C - B 2 = O, either A1 or A2 is zero. (Fig. 4.13c).

(4.91)

The conic is a parabola

(a) If A1 ~ 0 and A2 = 0, the canonical form is y

__.

(A + C)v/A 2 + B 2 2(BD-

AE)

x 2.

(4.92)

aSThis property holds not only for two-dimensional Euclidean motions but also for all two-dimensional projective transformations. A two-dimensional projective transformation has the form x' or Ax, where A is an arbitrary nonsingular 33-matrix. Conics undergo the congruence transformation in the form of eq. (4.86).

4.4.

Conics

117

y

X

O

(~)

(b)

(c)

Fig. 4.13. (a) Canonical form of an ellipse. (b) Canonical form of a hyperbola. (c) Canonical form of a parabola. (b) If

)~1

--

0 and A2 #- 0, the canonical form is g

D

(A + C)~/B 2 + C 2

2(BE-CD)

x 2.

(4.93)

T h e shape of a real conic is characterized by its axes of s y m m e t r y and

eccentricity as follows: 1. An ellipse has two axes of symmetry, called the principal axes, t h a t are orthogonal to each other. In the canonical form of eq. (4.89), the x-axis is called the major (minor) axis if a < b (a > b). Its area is

S-

7~ab.

(4.94)

Its eccentricity e is defined to be via /a 2 - b 2

~ ,

a >_ b,

e m

(4.95)

x/b 2 -a2

~ ,

a 1 for hyperbolas. Thus, 0 1 for hyperbolas (Fig. 4.14).

4.5

4.5.1

Space Conics and Quadrics Representation in three dimensions

A. Space conics

A space conic is a conic defined on a space plane. We represent it by back projection: we regard a space conic as the intersection of a space plane with a "cone" with vertex at the viewpoint O generated by the lines of sight of image points on a conic (Fig. 4.15). It follows that a space conic is represented by the space plane (n, r) - d on which the space conic lies and its projection (x, Q x ) = 0 onto the image plane; a space conic represented by {n, d, Q} is a set of space points that satisfy ( n , r ) - d,

(r, Q r ) - 0.

(4.97)

B. Quadrics

A quadric is a surface defined by a quadratic equation in the coordinates (X, Y, Z). Quadrics are classified into ellipsoids, paraboloids, hyperboloids, and their degeneracies (e.g., cylinders, pairs of space planes, space points, and empty sets). Paraboloids, hyperboloids, and their degeneracies are called centered quadrics, because they have centers of symmetry. Let re be the center of symmetry, which we simply call the center. Then, a centered quadric is represented in the form (r - rc, S(r

- rc))

-

1,

where S is a (33)-matrix. It is easy to see the following: 1. If S is positive definite, the quadric is an ellipsoid.

(4.98)

4.5.

Space Conics and Quadrics

119

(x, Qx)=o 0

(r, Qr)= 0

Fig. 4.15. Representation of a space conic.

O Fig. 4.16. Representation of a quadric. 2. If S is negative definite, the quadric is an empty set. 3. If S has signature (2,1), the quadric is a hyperboloid of one sheet. 4. If S has signature (1,2), the quadric is a hyperboloid of two sheets. 5. If S is singular, various types of degeneracy occur. The quadric given by eq. (4.98) is said to be singular or nonsingular depending on the matrix S is singular or nonsingular. The eigenvectors of S are called its principal axes. The reciprocal of the square root of its positive eigenvalue is the radius of the quadric for the corresponding principal axis (Fig. 4.16). For an ellipsoid that has an orthonormal system {Ul, u2, u3} as its principal axes and (rl, (re, and (re as the corresponding radii, the matrix S has the following form (see eq. (2.62)):

S-

..... uluT1 + u2u~

+

(4.99)

uau[

E x a m p l e 4.4 An ellipsoid centered at (a, b, c) with principal axes in the the coordinate axis orientations and the corresponding radii (rl, a2, and (r2 has the form

(X-a) 2 (r2

(Y-b) 2 +

(r~

(Z-c) 2 +

(r2

= 1.

(4.100)

120

Chapter 4.

Representation of Geometric Objects

(a) (lJ l )

The center re and the matrix S are given as follows:

rc -

4.5.2

b c

,

S -

1/a~

.

(4.101)

1/cr~

Polarity and conjugate direction

A. Poles and polars

Consider a nonsingular quadric given by eq. (4.98). The polar of a space point rp with respect to this quadric is a space plane defined by (r - rc, S(rp - r e ) ) = 1.

(4.102)

Its unit surface normal np and distance d from 0 are

~,

= N[s(~, - ~)1,

1 a~ = I I s ( ~ - ~)11 +

(~' '~')

(4.~0a)

Conversely, the space point rp is called the pole of the space plane (np, r) = dp with respect to the same quadric. From eqs. (4.103), we see that s-in

rp -- dp - (re, rip)"

(4.104)

Eq. (4.102) implies the following: 9 The polar of rp passes through rp if and only if rp is on the quadric (4.98). 9 If rp is on the quadric (4.98), the polar of rp is tangent to the quadric at rp (Fig. 4.17a). 9 The surface normal n to the tangent plane to the quadric (4.98) at r has orientation S (r - re): n ~ s(~-

~).

(4.105)

B. Conjugate direction

Consider a space plane H that passes through the center r e of the quadric (4.98). Let n be its surface normal. The conjugate direction 16 of the space 16If t h e space u n d e r g o e s a general linear t r a n s f o r m a t i o n , vectors n a n d n t t r a n s f o r m differently as a covariant vector and a contravariant vector, respectively. In this sense, t h e t r a n s f o r m a t i o n s of n and n t are contragradient to each oth e r .

4.5.

Space Conics and Quadrics

121

np

O (a) (b) Fig. 4.17. (a) The polar of rp passes through rp if and only if it is tangent to the quadric at rp. (b) The conjugate direction n t of space plane II. plane H with respect to the quadric (4.98) is the orientation of the vector n t that starts from the center r e and points toward a point on the quadric at which the tangent plane has surface normal ~ n (Fig. 4.17b). It follows from eq. (4.105) that n t c( S - i n . (4.106) If the quadric is a sphere, the conjugate direction of a space plane coincides with its surface normal. In this sense, the conjugate direction is a generalization of the surface normal to a space plane. C. G e n e r a l i z a t i o n

Quadrics can also be defined in higher dimensions: eq. (4.98) defines a quadric in n dimensions if r and r e are regarded as n-vectors and S is regarded as an ( n n ) matrix. If S is nonsingular, the polarity and the conjugate direction are defined as in three dimensions straightforwardly. Suppose S is a positive semidefinite symmetric m a t r i x of rank r (< n), and let {ui } be its eigensystem for eigenvalues {)~i}, where )~1 _> ... _> )~ > A~+ 1 = ... = )~n - 0. Then, the center r e and the vectors Ul, ..., u~ define an r-dimensional ajfine subspace 17 $ C 7r n. If the quadric is restricted to this affine subspace (Fig. 4.18), it defines a nonsingular quadric in it, so the polarity and the conjugate direction are defined in S.

.5.3

Visualization of covariance matrices

Conics are i m p o r t a n t image features because m a n y m a n - m a d e objects have circular and spherical parts, and circles and spheres are projected onto ellipses on the image plane. As compared with conics, the role of quadrics appears to be minor, since ellipsoidal objects are rare in real environments. However, quadrics have an important role: a three-dimensional positive semi-definite 17See Footnote 17 in Section 3.4.2.

Chapter 4.

122

Representation of Geometric Objects

Fig. 4.18. A singular quadric defines a nonsingular quadric if restricted to the affine subspace spanned by eigenvectors for positive eigenvalues. symmetric matrix is represented by an ellipsoid. This fact allows us to visualize three-dimensional covariance matrices. The following two techniques serve this purpose. A. Standard confidence region

Let V[~] be the covariance matrix of a space point ~. If it has eigenvalues A1, A2, and A3 and the corresponding eigensystem {ul, u2, u3}, it has the spectral decomposition

V['r]- o2UlU1-I- + 02u2'u: + 02~3u:,

(4.107)

where (ri - ~X~, i = 1, 2, 3. We can regard ei as the standard deviation in direction ui (see eq. (3.21)). 1. Suppose V[~] is of full rank. If the distribution is Gaussian, the surface on which the probability density is constant is an ellipsoid centered at in the form (r - ~, V [ ~ ] - l ( r -/~)) = constant (see eq. (3.38)). If we choose the constant to be 1/v/(27~)31V[i']Ie-t/2 , the ellipsoid is (r - ~, V [ ~ ] - l ( r - ~)) = 1,

(4.108)

which has principal axes {ul, u2, u3} and the corresponding radii al, cr2, and a3. Let us call the region inside this ellipsoid the standard confidence region of ?. This is a natural extension of the standard confidence interval (3.42). The probability that the true value of r falls inside the standard confidence region is 19.87% (see eq. (3.62)). 2. If V[/'] is of rank 2 with a3 = 0, the standard confidence region degenerates into a space conic with principal axes {Ul, u2} and the corresponding radii (71 and a2. It is represented by (u3, r) = (u3, i'),

(x, Q x ) = O,

(4.109)

where Q -- N[(Ul, u2,/')t-rdiag(er2, a~,--1)(Ul, u2, r)*].

(4.110)

4.5.

Space Conics and Quadrics

123

^

)

O Fig. 4.19. The standard confidence region is defined in the tangent space TU(L/) to the manifold U defined by the constraint. The symbol N[.] denotes the normalization for a matrix (see eq. (2.145)), and (Ul, u2, ~)t is the cofactor of the matrix whose columns are ul, u2, and ~ in that order (see eq. (2.18)). The probability that the true value of r falls inside this space conic is 39.35%. 3. If VIii is of rank 1 with a2 = a3 = 0, the standard confidence region degenerates into the line segment connecting ~-crlUl and ~ + a l U l . The probability that the true value of r falls inside this segment is 68.27%. The standard confidence region can be defined in higher dimensions in the same way. Let /~ be an n-vector that represents some object, and V[/~] its covariance matrix. Suppose/~ is constrained to be in an r-dimensional manifold L/. Then, V[/L] is generally a singular matrix of rank r, so the equation (u-/~, V[/~l-(u-/L)) = 1 (4.111) generally defines a singular quadric. However, it defines a nonsingular quadric if it is restricted to the tangent space T/~ (b/) to the manifold U a t / , (Fig. 4.19). We define the standard confidence region of V[/~] to be inside this quadric.

B. Primary deviation pair Covariance matrices can be visualized by means of their standard confidence regions only in three dimensions. In higher dimensions, we use an alternative technique. Note that if Crl is sufficiently large as compared with a2 in eq. (4.107), the error distribution is localized along the major axis (i.e., the principal axis with the largest eigenvalue). This suggests the following visualization technique. Let/~ be an n-vector that represents a geometric object, and VICe] its covariance matrix. Let n

-

>__ i=1

>_...

>_

(4.112)

124

Chapter 4.

Representation of Geometric Objects

~(U)

O Fig. 4.20. The primary deviation pair. be its spectral decomposition. The vector U l Can be interpreted as the orientation of the most likely deviation, and crl as the standard deviation in that orientation. Hence, the reliability o f / t can be visualized by displaying the two objects represented by the primary deviation pair {u +, u - } defined as follows (see eq. (3.48)): U + -- C[~t -[- 0"1~11 ,

~ t - "-- C[~t - o-1~/,1].

(4.113)

Here, the operation C[. ] designates high order correction, such as normalization, to ensure that u + E / d and u - E L/. To a first approximation, the primary deviation pair {u +, u - } indicates diametrically located endpoints in the standard confidence region is in the direction of its major axis (Fig. 4.20).

4.6

~. 6.1

Coordinate Transformation and P r o j e c t i o n

Coordinate transformation

Since objects in three dimensions are represented with respect to a fixed X Y Z coordinate system, the same object has different representations if described with respect to different coordinate systems. Suppose a new X ' Y ' Z ' coordinate system is defined in such a way that its axis orientation is obtained by rotating the original X Y Z coordinate system by R and its origin O' is translated from O by h; the vector h and the rotation m a t r i x / / a r e defined with respect to the original X Y Z coordinate system. We call {h, R} the motion parameters of the coordinate system (Fig. 4.21). This definition implies that if {h, //} are the motion parameters of the X ' Y ' Z ' coordinate system with respect to the X Y Z coordinate system, the motion parameters {h', R'} of the X Y Z coordinate system with respect to the X ' Y ' Z ' coordinate system are given by h'=-RTh,

R ' = R T.

(4.114)

lSNote that the standard confidence region of V[fi] is defined in the tangent space Tta(H), which does not coincide with the manifold///if b/is "curved" (Fig. 4.19).

4.6.

Coordinate Transformation and Projection

!,

125

g

O Fig. 4.21. Coordinate transformation and the motion parameters {h, R}.

A. Space points 1. A space point r defined with respect to the X Y Z coordinate system is represented with respect to the X ~ Y ~Z ~ coordinate system in the form r'-

RT(r-

h).

(4.115)

Hence, r

(4.116)

-- h + R r ' .

2. In the p-representation 19 ,

B. Space lines 1. In the {m, rH }-representation, a space line ( r - - r H ) x m -- 0 defined with respect to the X Y Z coordinate system is represented by ( r~ --?"H)t X m ! = 0 with respect to the X ~ Y ~ Z ~ coordinate system, where m ' -- R T m ,

- h).

(4.118)

rH -- R P m , ( r ~ H + R T h ) .

(4.119)

r 'H -- R T p m ( r ,

Hence, m - Rm',

2. In the {p, n}-representation, a space line r x p - n defined with respect to the X Y Z coordinate system is represented by r ~x ff - n ~with respect to the X ~ Y ~ Z ~ coordinate system, where p' (n')-N[(

RTp RT(n_h

xp))]"

(4.120)

19Eqs. (4.117) define a t r a n s f o r m a t i o n of T~3 and its inverse. T h e set of all such transformations forms the group of three-dimensional E u c l i d e a n m o t i o n s . If the rotation matrix R

in eqs. (4.117) is replaced by an arbitrary nonsingular 33-matrix, the resulting group is the group of three-dimensional affine transformations. If the 44-matrices in eqs. (4.117) are replaced by an arbitrary nonsingular 44-matrix and its inverse, the resulting group is the group of three-dimensional projective transformations of the four-dimensional space T/4, which can be identified with a three-dimensional projective space.

Chapter 4.

126

Representation of Geometric Objects

Hence, n

R(n'+

p')

(RTh)•

I.

(4.121)

C. Space plane 1. In the {n, d}-representation, a space plane (n, r) = d defined with respect to the X Y Z coordinate system is represented by (n', r') - d' with respect to the X ' Y ' Z ' coordinate system, where n' - R Tn,

d' = d - (n, h).

(4.122)

Hence, d - d' + (n, R T h).

n = Rn',

(4.123)

2. In the r,-representation, ~/= N[

(

_hTRR

01

)

v],

~, = N[

hT-

1

v'].

(4.124)

D. Space conics and quadrics 1. A space conic {n, d, Q} defined with respect to the X Y Z coordinate system is represented by {n', d', Q'} with respect to the X ' Y ' Z ' coordinate system, where n ' -- R T n ,

Q' - R T

i + e-

(n, h),

d' = d -

(,~, h)

Q

I + d -- (~i h)

R.

(4.~25)

"

(4.126)

Hence 2~, d - d' + (n, R T h),

n - Rn',

d

RQ'R T

I

d

2. A quadric ( r - rc, S ( r - rc)) - 1 defined with respect to the X Y Z coordinate system is represented by ( r ' - r'c, S ' ( r ' - rb) ) - 1 with respect to the X ' Y ' Z ~ coordinate system, where

~

- R ~(~

- h),

s' - R ~sR.

(4.127)

S - R S ' R T.

(4.128)

Hence, r e -- h + R r ~ ,

2~ (4.125) and (4.126) define a congruence transformation (see eq. (2.154)) and its inverse in the form Q = ATQ'A and Q' = (A-1)TQA -1, where A is the matrix that defines the projective transformation of the image plane (see eq. (4.86) and the matrix inversion formula (2.24)).

4.6.

Coordinate Transformation and Projection

r

127

L

p

0

(~)

(b)

Fig. 4.22. (a) Perspective projection of a space point. (b) Perspective projection of a space fine.

4.6.2

Perspectiveprojection

A. Space points 1. A space point P not on the X Y plane represented by r is perspectively projected onto the intersection p of the image plane Z - 1 with the line of sight of P (Fig. 4.22a). It is represented by x -

r (k, r)"

(4.129)

2. Space points on the X Y plane produce no images on the image plane 21. B. Space lines 1. A space line L that does not pass through the viewpoint O is projected onto the intersection I of the image plane Z = 1 with the space plane that passes through O and L (Fig. 4.22b). The surface normal to that space plane is n in the {p, n}-representation, and rH X m in the {m, rH }-representation (see eq. (4.36)). Hence, the projected image line is represented by (nt, x) = 0, where nl-- N[n]-

N[I"H x m].

(4.130)

If a space point on space line L moves in one direction indefinitely, its projection converges to an image point called the vanishing point of L (Fig. 4.23a); the same vanishing point is defined if the space point moves in the opposite direction. The vanishing point is represented by -

m

_

(k, m)

-

p x n Ik, p ,

hi

(4.131)

21A space point r on the X Y plane different from O can be thought of as projected onto the ideal image point ("image point at infinity") in the direction of r on the image plane; perspective projection of the viewpoint O is not defined.

128

C h a p t e r 4.

R e p r e s e n t a t i o n of Geometric Objects

x

x

O~

(b) Fig. 4.23. (a) Vanishing point of a space line. (b) Perspective projection of a space line that passes through the viewpoint O. T h e vanishing point separates the projected image line into two half lines. One side is the projection of the part of the space line for which Z > 0; the other side is the projection of the part for which Z < 0. T h e part for which Z = 0 is not projected onto the image plane 22. 2. A space line L t h a t the intersection p of (Fig. 4.235). Hence, r x p = 0 for (k, p)

passes t h r o u g h the viewpoint O is projected onto the image plane with L if L is not on the X Y plane space line v x m = 0 for (k, m ) # 0, or space line # 0, is projected onto image point

3. A space line on the X Y

m _ p (k, m ) (k,v)

(4.132)

plane is not projected onto the image plane 23.

C. S p a c e p l a n e s 1. A space plane II t h a t does not pass t h r o u g h the viewpoint O is p r o j e c t e d onto the entire image plane. If the space plane H is not parallel to the image plane, those space points which are on H and infinitely far a w a y 24 from the viewpoint O are projected onto an image line, known as the vanishing line of H (Fig. 4.24a). It is easily seen t h a t the vanishing line of space plane ( n n , r ) = d ( # 0) is ( n n , x) = O.

(4.133)

22The part of a space line for which Z = 0 can be thought of as projected onto the ideal image point ("image point at infinity") of the projected image line. 23A space line on the X Y plane can be thought of as projected onto the ideal image line ("image line at infinity") of the image plane if it does not pass through the viewpoint O; perspective projection of a space line on the X Y plane that passes through O is not defined. 24The set of all space points on a space plane that are infinitely far away from the viewpoint O is called an ideal space line ("space line at infinity"). The ideal space lines defined by mutually parallel space planes are regarded as the same ideal space line.

4.6.

Coordinate Transformation and Projection

n

129

.... .~ . . i

~..............

(b)

(~)

Fig. 4.24. (a) Vanishing line of a space plane. (b) Perspective projection of a space plane that passes through the viewpoint O. One of the two regions on the image plane separated by the vanishing line is the projection of the part for which Z > 0; the other is the projection of the part for which Z < 0. The part for which Z - 0 is not projected 25. If n - -t-k, no vanishing line appears 26. 2. A space plane H that passes through the viewpoint O is projected onto the intersection 1 of the image plane with II if it does not coincide with the X Y plane itself (Fig. 4.24b). Namely, space plane ( n n , r ) - 0 for n n 7~ + k is projected onto image line ( n n , x) = 0. 3. The X Y

plane is not projected onto the the image plane 27.

D. Space conics and quadrics 1. By definition, a space conic {n, d, Q} is projected onto conic ( x , Q x ) = 0 (see Fig. 4.15). 2. If space point r - Z x is on quadric (r - r e , S ( r - r e ) ) - 1, we have

Sx) - 2Z(x, S r c ) + (rc, S r c ) - 1. (4.134) This equation yields two real solutions for Z if the line of sight of x intersects the quadric at two space points; no real solution exists if it does not meet the quadric. The set of those image points whose lines of sight are tangent to an object in the scene is called the (occluding) c o n t o u r of the object (Fig. 4.25). It follows that an image pont x is on the contour of quadric ( r - r e , S ( r - r e ) ) - 1 if and only if eq. (4.134) has one multiple root, i.e., ( Z x - rc, S ( Z x -

rc))-Z2(x,

(x, S r c ) 2 - (x, S x ) ( ( r c , S r c ) - 1) - O,

(4.135)

25The part of a space plane for which Z = 0 can be thought of as projected onto the ideal image line ("image line at infinity").

U6The vanishing line can be thought of as the ideal image line ("image line at infinity"). 27Space plane (k,r) = 0 can be thought of as projected onto the ideal image line ("image line at infinity").

130

Chapter 4.

Representation of Geometric Objects

(r-rc,S(r-rc))=l

(x, Qx)=O o

Fig. 4.25. Perspective projection of a quadric. which defines a conic

(x, Qx) = 0,

(4.136)

O = ((~, s , ~ ) - 1 ) s - s , ~ , J s .

(4.137)

where In other words, quadric ( r - rc, S ( r - r c ) ) = 1 is projected onto the conic given by eq. (4.136). This conic may not be a real conic28; if it is a real conic, it may be degenerate.

28For example, an ellipsoid centered at the viewpoint O is projected onto an "imaginary conic" (an empty set in the real space).

Chapter 5 G e o m e t r i c Correction Multiple geometric objects that axe supposedly interrelated by a constraint may not satisfy it if each object is separately observed in the presence of noise. This chapter presents a statistically optimal way to correct the positions of geometric objects so that they satisfy a required constraint. The fundamental principle is the minimization of the Mahalanobis distance defined in terms of the covariance matrices of the objects. First, a general theory is formulated in abstract terms. Then, it is applied to typical geometric problems in two and three dimensions-optimally imposing coincidence and incidence on image points, image lines, conics, space points, space lines, and space planes. For each problem, explicit expressions for the correction and the a posteriori covariance matrices are derived. Optimal correction for orthogonality constraints is also studied.

5.1 5.1.1

General Theory Basic formulation

Consider N geometric objects in two or t h r e e dimensions, the c~th object being r e p r e s e n t e d by an n ~ - v e c t o r u~, c~ - 1, ..., N . Let n a be t h e dimension of vector u~. T h e N objects are a s s u m e d to be statistically i n d e p e n d e n t , b u t the c o m p o n e n t s of each u ~ m a y be correlated. We a s s u m e t h a t each u a is c o n s t r a i n e d to be in an n~~ -dimensional manifold b/~ C 7Zn~ which we call t h e data space of u~. Let ft~ be the t r u e value we should observe in the absence of noise, a n d write u ~ = ~2~ + A u ~ . T h e error A u a is, to a first a p p r o x i m a t i o n , constrained to be in the tangent space Tu~ (b/a) to the manifold Ha at ~ . Let V[u~] be the covariance m a t r i x of the error A u ~ . W e a s s u m e t h a t no constraint exists on A u a other t h a n A u a E T~t~ (Ha) a n d hence the range of V[ua] coincides with T~to (L/a). It follows t h a t

where P~tu" is the n ~ - d i m e n s i o n a l projection m a t r i x onto Tu~ (b/a). Suppose L s m o o t h functions F ( k ) ( . , . . . , 9)" 7~nl • the t r u e values Ul, .... , u N are k n o w n to satisfy

F(k)(a,,..., ~ ) --0, 131

~ - N _~ 7~ exist and

k = 1, ..., L.

(5.2)

132

C h a p t e r 5.

S(1) '

Geometric Correction

S

S

(~)

(b)

Fig. 5.1. (a) Nonsingular constraint imposed by three equations that are not independent. (b) Singular constraint imposed by two independent equations. We call eq. (5.2) simply the constraint, whereas we refer to the constraint u~ E L/a as the inherent constraint on u a . We now consider the problem of optimally correcting u l , .... , u g so t h a t these L equations are satisfied. Since each u~ is constrained to be in the d a t a space Ha C 7~ n~ , the direct sum ~[~N__1 u~ is constrained to be in its d a t a space

U_(DN ~=1/4~

C ~=~n~.

We say t h a t the constraint imposed by eq. (5.2) is

nonsingular if each of the L equations defines a manifold S (k) of codimension i i n / 4 and the L manifolds S (k), k = 1, ..., L, meet each other transversally in b/ (Fig. 5.1a; see Section 3.2.1); otherwise, the constraint is said to be singular I (Fig. 5.1b). In this chapter, we assume t h a t the constraint (5.2) is nonsingular, although the following theory can also be applied to singular constraints if appropriately modified 2. T h e L equations (5.2) m a y not necessarily be i n d e p e n d e n t (Fig. 5.1a). We call the n u m b e r r of i n d e p e n d e n t equations the rank of the constraint. It L S(k) of codifollows t h a t the constraint (5.2) defines a manifold S - Nk=l mension r in H; we call S the (geometric) modal of th~ constraint (5.2). F r o m the above definition, we see t h a t the rank r equals the dimension of the linear subspace

N

N

-

~N

a=l

(5.a)

ot=l

where V u / ~ ( k ) is the abbreviation of ~7u,~F(k)('Y_tl,...,ftN). Note t h a t the dimension of this subspace m a y not be equal to the dimension of the linear subspace N

v-

us { G P,"oV, oF

c~=l

N

~ 9 ...j

p u s V, o F(L) Us

c

N- ~n~

(5.4)

ct=l

1For example, if the L equations (5.2) are expressed as one equation, say, in the form ~N_ 1F(k)(fil, ..., fiN) 2 = 0, the constraint is singular. 2-'~Vewill see an example of a singular constraint in the motion analysis in Chapter 11.

5.1.

General Theory

133

for u~ # ~t~, where V u F (k) is the abbreviation of VuF(k)(Ul,..., UN). We say that the constraint (5.2) is degenerate 3 if the dimension of )2 is larger that t h e dimension of the subspace 12. Substituting u~ - ~ + Au~ i n t o F(k)(Ul,...,UL) and taking a linear approximation, we can replace eq. (5.2) to a first approximation by _

N

F(a) - E (Vu~/~(a)' A u a ) ,

k - 1, ..., L,

(5.5)

or--1

where F (k) is the abbreviation of F(k)(ul,..., UL). This linearized constraint is assumed to be satisfiable, i.e., there exists at least one set of solutions A u ~ E T~t. (b/a), c~ = 1, ..., N, that satisfies eq. (5.5). If A u ~ is a solution of eq. (5.5), the correction takes the f o r m / t ~ = u~ Au~ to a first approximation. However, infinitely many solutions may exist for Au~, c~ = 1, ..., N. From among them, we choose the one which minimizes the square sum of the Mahalanobis distance ]]Au~[l~,[u~] (see eq. (3.66)), i.e., N

J-

E ( A u ~ , IT[u~l-Au~)--4 min

(5.6)

c~--1

under the inherent constraint A u ~ E T~t. (t4~), c~ = 1, ..., N. N Geometrically, we are projecting the direct sum ( ~ = 1 u~ onto the "closest point" in the model $ determined by eq. (5.2), where the "closeness" is measured in the Mahalanobis distance with respect to the total covariance N matrix (~)~=1 17[u~] (Fig. 5.2). This criterion can be justified as maximum likelihood estimation for Gaussian noise. Namely, if the errors AUl, ..., AUN have the probability density

~=1

(5.7) maximizing the likelihood is equivalent to minimizing the function J given in (5.6) (see eqs. (3.46), ( 3 . 8 1 ) a n d (3.82)).

5.1.2

Optimal solution

1,..., n ~ - n ~ , ' be an orthonormal basis of T~t. (t4~) • Let ~"-( V j(a) } , j inherent constraint A u ~ E T u . (L/a) can be written as 9v j , Au (:'(~)

~

) -

0,

j -

1,

...,

3A more rigorous argument will be given in Chapter 14.

n ~ - n' ~ .

The

(5.8)

134

Chapter 5.

Geometric Correction

N ~ t t

a

a=l

Fig. 5.2. Projecting ~[~=1 N us onto the closest point in the model ,.q measured in the Mahalanobis distance. Introducing Lagrange multipliers Ai and pl ~), differentiating L

N

N

J-2EAkE(Vu k=l

/~ ( k ) , A u ~ ) - 2 E

a=l

a=l

n~-n,~

! 9

E

(~) ,-(~)

t~j (vj

,Au~)

(5.9)

j=l

with respect to each Au~, and setting the result zero, we obtain L

n ~ - - n IoL

-

v[~ol-~uo - Z ~ v ~ o p(~) + ~ k=l

=(~)

.~(o)~

(5.10)

j=l

Multiplying this by V[u~] on both sides and noting that =(~) u~ ~ T~to(U~) • we obtain L

)~kV[u~lVu~T'(k)'

PuU~ Au~ - E

(5.11)

k=l

where eqs. (5.1) have been used. Since Au~ E T~t(b/~), the solution is given by L

~

- ~ ~ [ ~ o l V ~ o ~ (~).

(5.12)

k=l

Substitution of this into eq. (5.5) yields

(v~oF (k), ~?[u~lVuoP (~)) ~ - F(k) /=1

(5.13)

c~=1

Since eq. (5.5) is assumed to be satisfiable, this equation is solvable (see Section 2.3.2); the solution is given in the following form: L

~ - Z w(~')r('). /=1

(5.14)

5.1.

General Theory

135

Here, l~ (kl) is the (kl) element of the (LL)-matrix I ~ - (I~ (kl)) defined by 12V - V - , where V - (~(kl)) is the (LL)-matrix defined by

(~?(~')) - ( L~=1(vuo F(k)' r

P('))) "

(5.15)

In the following, we use the following abbreviation to denote the (LL)-matrix

W-

(w(k~)):

(~v(kl))--(~(~u' ) ~-'a(k)'~/r[ua]vu' = l ~(l))

(5.~6)

It can be shown 4 that the rank of the matrix V (hence of I?V) equals the rank r of the constraint (5.2). It follows that the optimal correction is given in the following form (see eqs. (2.140)and (3.85)): L

(5.17)

k,l--1

This equation has the following geometric interpretation. If the noise is Gaussian, the equiprobability surface for ua has the form

(5.~8)

(u a - u a, V[u ~ ](ua - ~ a ) ) - constant.

As discussed in Section 4.5, this equation defines a nonsingular quadric in the tangent space T~t~ (L/a). Let Sa be the restriction of the model S to L/a obtained by fixing u~ - ~tZ for/3 ~ a. We now show that the optimal correction A u a given by eq. (5.17) is in the conjugate direction of the tangent space T~t~ ($a) to the model Sa at ~a (see Section 4.5.2). In T u , (L/a), the tangent hyperplane to the quadric defined by eq. (5.18) at ua + A u a has the following surface normal (see eq. (4.105)): L

no o~ e [ ~ o ] - ~

- Z

w ,~2,a.

5.2.2

(5.71)

Optimal correction for incidence

A. S i m u l t a n e o u s correction Suppose image point x and image line (n, x) - 0 are, respectively, estimates of an image point p and an image line 1 that should be incident to each other in the absence of noise. Consider the problem of optimally correcting them so as to make them incident. In other words, we want to find A x and A n such that 9 - x - A x and ~ - n - A n satisfy

(~, ~) - 0.

(5.72)

The rank of this constraint is 1. Let V[x] and V[n] be the a priori covariance matrices of x and n, respectively. If the image point and the image line are statistically independent, the problem can be written as the optimization J - (Ax, V [ x ] - A x ) + (An, V [ n ] - A n ) -+ min

(5.73)

under the linearized constraint

(g. ~ )

+ (~. ~ n ) - (n. ~).

(5.74)

k • The first order solution is given by A~ -

(n, ~ ) v [ ~ ] ~

v[~]n) + (~. v[.~]~) '

(~. A,~ -

(n, ~ ) v [ n ] ~

(.. v[~]~) + (~. v [ ~ ] ~ )

(5.75)

A realistic form of the correction is - ~-

A~,

a-

N[n-

An].

(5.76)

The a posteriori covariance matrices of the corrected values & and ~ are V[~] - V[x] -

(V[~']'~)(V[~I'~)T (~, v[~]~) + (~, ~ [ ~ ] ~ ) '

v[,~] - ?In]

(r (~, v[~],~) + (~, ~ [ n ] ~ ) '

-

148

Chapter 5.

( r l ~(~r 1 T ~,v~x~,,_~n~,,

v[~,h]--

(~. v[~]~) + (~. ~[.]~)

Geometric Correction

= V[fi, &]T,

(5.77)

where V [ n ] - P n V[n]Pn.

(5.78)

The residual of J can be written as J -

(5.79)

(n'x)2

(g, V[~lg) + (~, ~[.]~)' which is a X2 variable with one degree of freedom. This fact provides an incidence test for an image point and an image line: the hypothesis that image point x and image line (n, x) - 0 are incident to each other is rejected with significance level a ~ if J > X2,a 9 (5.80)

B. Image point correction If the image line (n, x) - 0 is fixed, the linearized constraint is

(-,~)

• A ~ e {k)L.

- (-,~),

(5.81)

The first-order correction of the image point x is obtained by letting V[n] O in eqs. (5.75)"

~ = (~,~)v[~]~

(~, v [ ~ ] ~ ) "

(5.82)

The a posteriori covariance matrix of the corrected value ~ is

v[~]-

v[~]-

(v[~]')(v[~]~)~

(~. v[~].)

(5.83)

Since V[hc]n = 0, the rank of V[~] is 1; its null space is {k, n } L , which is orthogonal to the orientation m - N[k x n] of the space line (n, x) = 0. The residual

j_

(n, ~)~

(n, V[x]n)

(5.84)

is a X2 variable with one degree of freedom. Hence, and the incidence test given by eq. (5.80) can be applied. E x a m p l e 5.2 If each coordinate is perturbed independently by Gaussian noise of mean 0 and variance e2, the covariance matrix of A x is V[x] e2Pk . The optimal correction (5.82) reduces to

~

- - (~' ~ ) P k ~ 1 - (k, ~ ) ~

(5.s5)

5.2.

Correction of Image Points and Image Lines

149

X

^ x

m

Fig. 5.8. Optimal incidence correction for an image point. Consequently, x is displaced onto the image line ( n , x ) - 0 perpendicularly (Fig. 5.8). The a posteriori covariance matrix of the corrected value 5~ can be written as V[~] = e2....~T. (5.86) where m = X[k x n I is the orientation of the image line (n, x) = 0. The residual (5.84) can be written as

af -

~

(n'~):

-'T E 1 -(k,

n) 2"

(5.87)

Hence, an unbiased estimator of the variance e2 is obtained in the form ~2 __--

( n , X) 2

- (k,.)~"

(5.8s)

The value @thus estimated equals the distance between the image point and the image line (see eq. (4.11)). If the value e is given a priori, the incidence test takes the form ~2 E---~-

> ~1~,o.

(5.s9)

C. Image line correction

If the image point x = 0 is fixed, the linearized constraint is ( A n , x) - (n, x),

A n 6 {fi}LI .

(5.90)

The first-order correction of the image line (n, x) = 0 is obtained by letting V[x] = O in eqs. (5.75):

A,~ - (n, ~ ) v [ n ] ~ (~, v[n]~) "

(5.91)

The a posteriori covariance matrix of the corrected value ~ is

V[n]- ~tT[n]- (~/"[fl']X)(Y[f't]x)T

(5.92)

150

Chapter 5.

Geometric Correction

The rank of V[~] is 1; its null space is {~, X}L. The residual J -- (x, l}'[n]x)

(5.93)

is a X2 variable with one degree of freedom. Hence, the incidence test given by eq. (5.80) can be applied.

5.3

5.3.1

Correction of Space Points and Space Lines

Optimal correction for coincidence

A. Space points Two space points ?'1 and ?'2 that are supposed to coincide can be optimally corrected in the same way as in the case of image points. Let V[?'l] and V[?'2] be their respective a priori covariance matrices. Let r l and r2 be the true positions of ?'1 and ?'2, respectively. The constraint to be imposed is

r, - r2,

(5.94)

which has rank 3. If the two space points are statistically independent, the problem is finding A?'I and A?'2 such that J - (A?'I, V[?'1]-1A?'1) -~-(A?'2, V[?'2]-1A?'2) --+ min

(5.95)

under the linearized constraint

A?'2 -- A?'I -- ?'2 -- ?'1.

(5.96)

The first order solution is given by

A?'I-

V[?'1]W(?'1-?'2),

A?'2 -- V[?'2]W(?'2 - ?'1),

(5.97)

where W is a (33)-matrix defined by

W-

(V[?'l]-~- V[?'2]) -1 9

(5.98)

The a posteriori covariance matrix of the estimate ~ is

V [ ' r ' ] - V [ ? ' l ] - V[?'I]WV[?'I] -- V[?'I]WV[?'2] =

(5.99)

The residual of J can be written as

J-

(?'2-?'1,W(?'2-?'1)),

(5.100)

5.3.

Correction of Space Points and Space Lines

151

which is a X2 variable with three degrees of freedom. This fact provides a coincidence test for space points: the hypothesis that space points r l and ?'2 coincide with each other is rejected with significance level a% if

J >

(5.101)

E x a m p l e 5.3 If each coordinate is perturbed independently by Gaussian noise of mean 0 and variance e 2, the covariance matrices of A r l and At2 are V[rl] = V[r2] = e2I. The optimal estimate ~ is 1

~" -- ~'1 -- A / ' I -- /'2 -- A/'2 -- ~(7"1 ~t- ~'2).

(5.102)

The a posteriori covariance matrix of ~ is s

V[e]-

~-I.

(5.103)

The residual (5.100) can be written as

1

12

J -- ~E2 11~2 -- ~'1[ 9

(5.104)

Hence, an unbiased estimator of the variance e 2 is obtained in the form ~2_ 1 ~llr2 - rill 2.

(5.105)

The value ~ thus estimated equals 1/vf3 times the half-distance between the two space points. If the value e is given a priori, the coincidence test takes the form ~:2

X 2 ,a

J > -i-"

(5.106)

B. Space lines

The same analysis can be done n2. Let V[p 1 | n l] and V[p 2 | signs of {Pl, n l } and {P2, n2} /' X P l - - ~'1 and r x P2 - n2 imposed is

for two space lines r x Pl - n l and r x P2 = n2] be their a priori covariance matrices. The are chosen so that Pl ~ P2 and ~t 1 ~,~ ~t 2. Let be the true space lines. The constraint to be

P l -- P2,

~1 -- fi2,

(5.107)

which has rank 4 because the representations {Pl, fil } and {102, fi2 } have four degrees of freedom 12 (see Section 4.2.2). If the two space lines are statistically independent, the problem is finding Apl , An1, A p 2, and An2 such that J -

( / k p l 9 ATtl, ~r[p 1 9 n l l - ( / k p

I 9 ATt,1))

_

+ ( A p 2 | An2, V[p 2 | n : ] - ( A p 2 | An2)) ~ min 12This constraint is degenerate.

(5.108)

152

Chapter 5.

Geometric Correction

under the linearized constraint Ap 2 | An2 - Ap 1 | An1 - P2 | n2 - Pl | n l , A p l ~ A n l e {Pl 0 ~/,I, ~,1 O P l } f , 2_

(5.109)

Ap2 | An2 E {P2 @ fi2,fi2 @ P2}L" The first order solution is

~XV~ 9 ~X,~ - V[p~ ~ ,~]W(p~ 9 ,~ - p, 9 nl),

(5.110)

where W is a (66)-matrix defined by 13 W-(V[p

1

@rtl] "4-V[p2 O n 2 ] )

(5.111)

4 9

A realistic form of the correction is (lb)

_N•

PI--Apl nl --/knl

h

where the operation N•

N• ( a ) b

rt2 -- Art2

'

(5.112)

] is defined by

{ N[a | PN[a] b] ]-

p2--Ap2 )]

)I-N•

N[PN[b]a | b]

if"a">--"bl"

(5.113)

otherwise.

The a posteriori covariance matrix of the estimate lb @ h is W[i~ @ ~] - l$[pl @ rtl] - V[Pl @ n l]l~rV[Pl 9 n l]

(5.114) Here,

V[Pi | nil = Px~,.~ V[pi | n~]P~r

i = 1, 2,

(5.115)

where PX~.~ is the six-dimensional projection matrix onto N ' p.1_. n

(see

eq. (4.44:)). The matrix I ~ is obtained by replacing V[p i @ hi] by l~[p i @ hi], i - 1, 2, in eq. (15.111). The residual of J can be written as

(5.116) which is a X2 variable with four degrees of freedom. This fact provides a coincidence test for space lines" the hypothesis that space lines r x Pl = n l and r x P2 - n2 coincide with each other is rejected with significance level a% if J > X~,a" (5.117) 13The ranges of V[pl • nl] and V[p2 9 n2] are different from the ranges of l/[pl (~ nl] and 17[p2 @ n2]. Consequently, although IY[pl @ nl] + l?[p2 @ n2] is a singular matrix of rank 4, the matrix V[pl | nl] + V[p2 | he] is generally nonsingular.

5.3.

5.3.2

Correction of Space Points and Space Lines

153

Optimal correction for incidence

A. Simultaneous correction As in two dimensions, a space point r and a space line r • p - n can be optimally corrected so as to make them incident. Let V[r] and V[p | n] be their a priori covariance matrices. The problem is finding A t , Ap, and A n such that ~ = r - A t , p = p - Ap, and ~ = n - A n satisfy

e xp--ff,.

(5.118)

The rank of this constraint is 2 because the three component equations are algebraically dependent 14. If the space point and the space line are statistically independent, the problem can be written as the optimization J - (At,

~r[~,]--I A/')

-~- (Ap | An, fT[p | n ] - ( A p | A n ) ) -+ min

(5.119)

under the linearized constraint Ar

x

p+f"

x

Ap--An=r

x

p_n,

Ap | A n E {p | n, fi @ lb}~.

(5.120)

The first order solution is given by

=-(vH

x r,)w(,, x p - n),

Ap = (V[p] x r -

V[p, n l ) W ( r x p -

A n = ( V t n , p] x r -

V[n])W(r x p-

n), (5.121)

n),

where W is a (33)-matrix defined by 15 W-

(p x V[v] x p + v x V[p] x r -

)-

2S[v x V[p,n]l + V[n] 2 "

(5.122)

The symbol S[. ] denotes the symmetrization operator (see eqs. (2.205)). A realistic form of the correction is f--r-At,

h

n--An

]'

(5.123)

where the operator N• ]is defined by eq. (5.113). The a posteriori covariance matrices of the corrected values ?,/~, and h are

U p ] - V[,']- (V[r] x/,)W(p x V[,']), 14This constraint is degenerate. 15The rank-constrained generalized inverse ( . ) 2 is used because ( . ) is generally nonsingular if evaluated at the data values; it should be a singular matrix of rank 2 if evaluated at the true values.

154

Chapter 5.

Geometric Correction

v[v] - ?[v]- (?Iv] • e - ?[v,,~])W(e • ?Iv]- ?In, v]), V[Ib, ~t] - V[p, n] - (V[p] • ? - lY[p, n])I~(/" • V[p, n] - I7[n]) - V[/t, lb] 7-,

v[,~]- ? [ n l - (?[n,v] x e - ?[n])Cc(e • ?[v, n ] - ?[n]), v[~,/,]- (v[r] x/,)g,(~ x

?[v]- ~[n, v ] ) -

v[/,, ~]T,

V[O, ~t] = (V[r] x p)I~(O x V[p,n]- I Y [ n ] ) - V[~t, ?]T.

(5.124)

The matrices IY[p], ~Z[p, n], and V[n] are obtained as submatrices of

V[p | n] - P~'~e~V[p @ nlPx~e~.

(s.~2s)

The matrix I&r is obtained by replacing r, p, V[p], V[p, n], and V[n] b y / ' , lb, V[p], V[p, n], and V[n], respectively, in eq. (5.122). The residual of J can be written as j - (r • p - n, W ( r x p - n)), (5.126) which is a ~:2 variable with two degrees of freedom. This fact provides an incidence test for a space point and a space line: the hypothesis that space point r and space line r • p - n are incident to each other is rejected with significance level a ~ if j > ~2,a. (5.127) B. Space point correction

If the space line r • p - n is fixed, the linearized constraint is

Arxp=rxp-n.

(5.128)

The first order correction of the space point r is

~

= -(v[~] • v)w(~

• v-

n),

(s.129)

where W is a (33)-matrix given by

The a posteriori covm'iance matrix V[~] of the corrected value ~ is given in the form shown in eqs. (5.124), where lb and ~ are replaced by p and n, respectively. Matrix V[~] has rank 1; its null space is {n, rH}L in the {m, rH}-representation. The residual J is given in the form of eq. (5.126) and is a X2 variable with two degrees of freedom. Hence, the incidence test given by eq. (5.127) can be applied.

5.3.

Correction of Space Points and Space Lines

155

Ar r

O Fig. 5.9. Optimal incidence correction for a space point. E x a m p l e 5.4 If each coordinate is p e r t u r b e d independently by Gaussian noise of mean 0 and variance e2, the covariance m a t r i x of A r is V[r] - e2I. The optimal correction (5.129) reduces to

p

At-

x

(r x p - n ) ilPl12

-- P r o m -

?"H.

(5.131)

Consequently, r is displaced onto the space line r x p - n perpendicularly (Fig. 5.9). The a posteriori covariance matrix of the corrected value ~ has the form V [ ~ ] - e 2 m m -T-, (5.132) where m - N[p] is the orientation of the space line r x p - n. The residual (5.126) can be written as

J-

1

--~

I1~ x p - n i l ~ 1 ilpl12 = -~llPmr-

r.II 2.

(5.133)

Hence, an unbiased estimator of the variance e e is obtained in the form

~2

I1',' x

p-.,112

211pll2

12

1

= ~llPmr-

rill 9

(5.134)

The value ~ thus estimated equals 1 / v ~ times the distance between the space point and the space line (see eq. 4.49)). If the value e is given a priori, the incidence test takes the form ~2 X2,~ e-Y > ---~--.

(5.135)

C. Space line correction If the space point r is fixed, the linearized constraint is r x Ap-

An - v x p-

n,

156

Chapter 5. AV+n

Geometric Correction (5.136)

~ {p+ n,n +p}~.

The optimal correction of the space line r x p - n is Ap~.

V[p, n l ) W ( r

(V[p] x r -

- ((~ • v [ v , . ] ) ~

x p-

n), (5.137)

-),

- V[.l)W(~ • v-

where W is a (33)-matrix given by w

-

(\ ~ • v [ v ] • ~ - ~ • v [ v , . ] -

(~ • v [ v , . ] ) ~

+ vM

.

(5.138)

The a posteriori covariance matrices of the corrected values/5 and fi are given in the form shown in eqs. (5.124), where ~ is replaced by r. The residual is (5.139)

J - (," • v - - , w ( , - • v - -)).

This is a X2 variable with two degrees of freedom. Hence, the incidence test given by eq. (5.127) can be applied. 5.4

5.~.1

Correction

of Space

Planes

Optimal correction for coincidence

Two space planes (v,, p) = 0 and (v2, p) = 0 that are supposed to coincide can also be optimally corrected. Let Y[vl] and Y[v2] be their respective a priori covariance matrices. The signs of the 4-vectors Vl and v2 are chosen so that vl ~-- v2. Let (Pl, P) - 0 and (P2, P) - 0 be the true space planes. The constraint to be imposed is (5.140)

Vl -- V2,

which has rank 3 because both sides are unit vectors 16. If the two space planes are statistically independent, the problem is finding AVl and Av2 such that J - (AVl, ~r[vl]-lAvl)

-[- ( A v 2 , ~ r [ / 2 2 ] - l n v 2 )

--=+m i n

(5.141)

under the linearized constraint A v 2 -- A V l -- /22 -- V l ,

Av2 e {P2}L~. The first order solution is given by ~1 16This constraint is degenerate.

- V[~l]W(~l - ~),

(5.142)

5.4.

Correction of Space Planes

157

AI/2 -- V[I22]W(v 2 - Vl) ,

(5.143)

where W is a (44)-matrix defined by a7 (5.144) A realistic form of the correction is ~' = N[L,1 - A I / 1 ] - N[L,2 - A!21].

(5.145)

The a posteriori covariance matrix of the estimate s is Villi -

?[Vl]W?[Vl]-

?[Vl]W?[v2]

(5.146)

Here, -

i - 1, 2,

(5.147)

where P~,, is the four-dimensional projection matrix along u i. The matrix IrV is obtained by replacing V[vi] by Y[~'i], i - 1, 2, in eq. (5.144). The residual of J can be written as ^

J - - (V2 - V l , W ( v 2 - V l ) ) ,

(5.148)

which is a X2 variable with three degrees of freedom. This fact provides a coincidence test for space planes: the hypothesis that two space planes (Vl, p) = 0 and (L'2, p) = 0 coincide with each other is rejected with significance level a% if J > ~2,a. (5.149)

5.~.2

Optimal incidence with space points

A. S i m u l t a n e o u s c o r r e c t i o n

A space point p and a space plane (v, p) - 0 can be optimally corrected so as to make them incident. Let V[p] and V[v] be their respective a priori covariance matrices. The problem is finding A p and A v such that ~ = p - A p and P - L , - A v satisfy (P, h) - 0. (5.150) The rank of this constraint is 1. If the space point and the space plane are statistically independent, the problem can be written as the optimization J - (Ap, V [ p ] - A p ) + (Av, V [ v ] - A v ) --+ min

(5.151)

aTThe the ranges of V[ua] and V[u2] are different from the ranges of l)'[pa] and V[v2]. Consequently, although ~'[Vl]+ V[u2] is a singular matrix of rank 3, the matrix VIal]+ V[v2] is generally nonsingular.

158

Chapter 5.

Geometric Correction

under the linearized constraint

(~,,/,) + (~, ~p) = (~,, p), tg /

(5.152)

where ~ = (0, 0, 0, 1) T. The first order solution is given by

(~,, p)V[plv ~t, - (~, v[p]v) + (p, v[~,]p)' (~,, p)v[~,]p (v, V[plv) + (p, v[~,lp)

Alp

(5.153)

A realistic form of the correction is /5 - p -

Ap,

~, = N [ ~ , -

(5.154)

A~,].

The a posteriori covariance matrices of the corrected values/~ and 1) are

v[/,]- v [ d -

(v[p]~)(v[p]~) ~ (~,, v[p]~,) + (/,, ?[~,]/,)'

(?[,,,]/,)(?[,,].a) T v[~] = ~ [ ~ ] v[/,, ~,1 -

(~,, v[p]~,) + (/,, ?[~,]/,)'

(v[p]~,) (?[~,]/,)T (~,, v[p]~,) + (h, ~'[~,]h)

-

= v[~,,/,]T

.

(5.155)

Here,

r

p~v[~]p~,

(5.156)

where P/, is the four-dimensional projection matrix along ~,. The residual of J can be written as

.]

-

(u' P)2

(5.157)

(~', V[d~') + (h, ?[~']/')' which is a X2 variable with one degree of freedom. This fact provides an incidence test for a space point and a space plane: the hypothesis that space point p and space plane (u, p) - 0 are incident to each other is rejected with significance level a~0 if j > X2,a .

(5.158)

5.4.

Correction of Space Plmles

159

O Fig. 5.10. Optimal incidence correction for a space point. B. Space point correction

If the space plane (v, p) = O is fixed, the linearized constraint is

(~, Ap)

(~, p),

-

Ap

e {~}~.

(5.159)

The optimal correction of p is

Ap

(v,p)V[p]u

(5.160)

(~, v[p]~) "

-

The a posteriori covariance matrix of the corrected value/5 is V[~b]-

V[p]- (Y[P]v)(Y[P]V)-r (•, V[plv)

"

(5.161)

Since Viably = O, the rank of V[/~] is 2; its null space is {~, V}L. The residual j

_

(v, p)2

(~, v[p]~)

(5.162)

is a X2 variable with one degree of freedom. Hence, the incidence test given by eq. (5.158) can be applied. E x a m p l e 5.5 If each coordinate is perturbed independently by Gaussia~ noise of mean 0 rand variance e2, the covariance matrix of p is V[p] = e2P~ (=

~ I ~ 0). In the (n, d}-represent~tion, the optim~ correctio~ (5.160) reduces to

zx~ = ((n, ~) - d)n.

(5.~63)

Consequently, r is displaced onto the space plane (n, r) = d perpendicularly (Fig. 5.10). The a posteriori covariance matrix of the corrected value ? is V[/'] =

e2pn.

(5.164)

The residual (5.162) can be written as 1 j - j ( ( n , r) - d)2.

(5.165)

160

Chapter 5.

Geometric Correction

Hence, an unbiased estimator of the variance e2 is obtained in the form -

-

d)

(5.166)

The value ~ thus estimated equals the distance between the space plane and the space point (see eq. (4.68)). If the value e is given a priori, the incidence test takes the form ~2 e-~ > X~,a" (5.167) C. Space plane correction

If the space point p is fixed, the linearized constraint is (Av, p ) -

(v,p),

Av e {P}~.

(5.168)

The optimal correction of space plane (v, p) - 0 is (p, V[v]p) "

(5.169)

The a posteriori covariance matrix the corrected value D is V[~]- ?[~l-

(V[~IP)(?[~]P)~ (p, ~[~]p) .

(5.~70)

The rank of V[9] is 2; its null space is {/~, P}L. The residual

J -

P):

(5. 71)

is a X2 variable with one degree of freedom. Hence, the incidence test given by eq. (5.158) can be applied.

5.4.3

Optimal incidence with space lines

A. Simultaneous correction

A space line (r - rH) • m = 0 and a space plane (n, r) = d can be optimally corrected so as to make them incident. Let V[m (DrH] and V[n @d] be their respective a priori covariance matrices, and ( r - OH) • rh -- 0 and (fi, r) d their true equations. The constraint is (~, ,~) - 0,

(~, ~ , ) - d,

(5.172)

which has rank 2. If the space line and the space plane are statistically independent, the problem is finding A m , Art/, An, and Ad such that _

J - (Am + ArH, Vim + rHI-(Am + Art/)) + ( A n | Ad, V[n 9 d]- ( A n | Ad)) -+ min _

(5.173)

5.49

Correction of Space Planes

161

under the linearized constraint

(An,~,)+(~,Am)=(n,m),

(~.. e.) + (~. ~ . ) - ~d = (....) - d. • A m | ArH e { m | 0, OH | vh }L,

An e {n}f.

(5.174)

The first order solution is given by

vimlo

Am

V[rH,m]Tn

Ad

--

(m, V[n, d])

v[...l.

(.. ...) - d

(rH, V[n, d]) - V[d]

W

'

(n, rH) - d ' (5.175)

where W is a (22)-matrix defined by

(.. vim].) + (m. v[.]m) w =

(.. vim. ~.1.) + (~.. v [ . ] m ) (n. V[~?l.. 7".]n) -}- (771..V[n]~'.) - (~11..V[n. d]) ~ -1 (n,

V[rH]n) +

(vii, V[nIrH ) -- (rH, V[n, d])

)

9 (5.176)

A realistic form of the correction is

vh = N [ m -

Am],

if, : N [ n -

An],

d = d - Ad.

(5.177)

The a posteriori covariance matrices of the corrected values rh,/'H, fi, and d

are

v[.~l

v[.~. #~]

#[m]

#[m. ~1

v[+~,m] v[+~])= (~[~,m] ~[~] ) ~[m..'~/].~ ) v[r.].~

viii

(v[d,,~]

v[~. d] vial]

~[.] #[n. d] ~[d. nl V[dl ^ ) _ ( ~[.].~ ~[.]~ - ~[.,d] (.h. e[.. dl) (~.. e[-. dl/- ~[d] ) W ( ?[-l.h e[n]~H-'~[-.dl (.h. e[.. dl) (+.. r d]) - V[~I )

)=(

162

Chapter 5.

v[.~. ,~] ( v[e..a]

v[.~. d] v[~. , d] ) - - (

?[m]a

?[rH, m] .~

i~r ( ?[n]~q~

Geometric Correction

?[m.~.]a

?[rH]/~ )

?[nigH -- ?[n, d]

(rh, ?In, d]) (/~g,?[n, d]) - ?[4 )

, (5.178)

where V[m], V[m, rH], etc. are computed as submatrices of V[m | r , ]

- Px~e~ V[m |

lY[n | d] - ( P n | 1)V[n | Here, P x ~ e ~

rg]Px~e~,

d](Pr~ |

1).

(5.179) • N,i~**. (se~ m, rH, n, V[m],

is the six-dimensional projection matrix onto

eq. (4.40)). The matrix I ~ is obtained by replacing Vim, rH], etc. by rh, rH,/~, ?[m], ? I r a , rH], etc., respectively, in eq. (5.176). The residual of J can be written as J -- ~TV "(11) (1"~,m ) 2 +212V(12) (n, m)((n,

rH)-d)+IV (22)((n, rH)--d) 2,

(5.180)

which is a X2 variable with two degrees of freedom. This fact provides an incidence test for a space line and a space plane: the hypothesis that space line ( r - r H ) • m -- 0 and space plane are incident to each other is rejected with significance level a% if J > X~,a. (5.181) B. Space line correction

If the space plane (n, r) = d is fixed, the linearized constraint is (n, A m ) = (n, m),

(n, Art/) = (n,

rH) -- d,

A m G A~'H e {?~ O 0, ~H O m } f .

(5.182)

The optimal correction of the space line (r - r H ) X m = 0 is

~.

v[~.. m] ~ .

v[~.l.

(~. ~.) - d

'

where W is a (22)-matrix defined by

w-

(n,V[m]n) ((,vim, ~.1~)

(n, V[m, rH]n) )-1 (~. v[~.].) 9

(5.~84)

The a posteriori covariance matrices of the corrected values ~h and rH are given in the form shown in eqs.^(5.178), where/~ and d are replaced by n and d, respectively. The residual J is given in the form of eq. (5.178) and is a X2 variable with two degrees of freedom. Hence, the incidence test given by eq. (5.181) can be applied.

5.5.

Orthogonality Correction

163

C. Space plane correction

If the space line (r - r H ) • m - 0 is fixed, the linearized constraint is (Art, m ) = (n, m),

(An, r g ) -- Ad = (n, rH) -- d, _k A n e {n}L.

(5.185)

The optimal correction of the space plane (n, r) = d is

Ad

=

(m, V[n, d])

(r, V[n, dl) - V[d]

W

(n, rH) - d

' (5.186)

where W is a (22)-matrix defined by

W -

(m, V[n]m) ( ~ , Via]m)

(m, v[,~]rH) - (m, v [ . , d]) ( ~ , v[n]~H) - (~, v[n, d])

-i

)

(5.187)

The a posteriori covariance matrices of the corrected values fi and d are given in the form shown in eqs. (5.178), where ~'H and ~h are replaced by r H and m , respectively. The residual J is given in the form of eq. (5.178) and is a X2 variable with two degrees of freedom. Hence, the incidence test given by eq. (5.181) can be applied.

5.5

Orthogonality Correction

5.5.1

Correction of two orientations

A. Simultaneous correction

Let m l and m2 be unit 3-vectors that indicate orientations supposedly orthogonal. Let V[ml] and Vim2] be their respective a priori covariance matrices. In the presence of noise, m l and m2 are not exactly orthogonal. Consider the problem of optimally correcting them so as to make them orthogonal (Fig. 5.11). In other words, we want to find Am1 and Am2 such that m l = m l -- A m l a n d m 2 - m 2 - A m 2 satisfy (rhl, m2) - 0.

(5.188)

The rank of this constraint is 1. If the two orientations are statistically independent, the problem can be written as the optimization

J

-

(Am1,

V [ m l ] - A m 1) +

(Am2,

V[m2]-Am2) .-+ r a i n

under the linearized constraint

(~ml, m2)+ (rex, ~m2) - (ml, m2),

(5.189)

164

Chapter 5.

Geometric Correction

Am I

ml

m I

Am2 0

n-t2

Fig. 5.11. Orthogonality correction for two orientations.

A m l E {ml}L~,

Am2 E {?~2}~.

(5.19o)

The first order solution is given by

(ml,m2)V[ml]m2 A m i -- (m2,V[mi]~T/,2) ~- (ml, V[m2]mi)'

(mi,m2)V[m2]ml Am2 = (m2, V [ m i ] m 2 ) + (m~, V[m2]mi)"

(5.191)

A realistic form of the correction is

ml

-

-

N[ml

-

-

Am:],

vh2 - N[m2 - Am2].

(5.192)

The a posteriori covariance matrices of the corrected values vhl and ~h2 are

V[mi] -- ? [ m i ] - (T~2,V[ml]m2) 4- ( m l , V [ m 2 ] m l ) ' V[zh2] - ?[m2] - (zh2, V[ml]~~2) -~- (rex, V[/2]~/~l)' ( ? [ m l ]m2 ) ( ? [ m 2 l m i ) T

V[T~I, m21 -- --(m2, V[mi]m2) + (ziz~, V[m2]ziZl)"

(5.193)

Here, V[mi] - P v h V [ m i ] P v i z , ,

i-

1, 2,

(5.194)

where Pviz~ is the projection matrix along zizi. The residual of J can be written as

,] --

(mi, m2) 2 (~TI'2,~/[ml]m2) -}- (#/,1, IY[m2]ziZl)'

(5.195)

which is a )/2 variable with one degree of freedom. This fact provides an orthogonality test for two orientations" the hypothesis that the two orientations m l and m2 are orthogonal to each other is rejected with significance level a% if

J > x i,o.

(5.196)

5.5.

Orthogonality Correction

165

Am3 in3 ~ m3 Am, Am 2 Fig. 5.12. Orthogonality correction for three orientations. B. Correction of one orientation If m2 is fixed, the linearized constraint is (Am1, m2) - ( m l , m2),

A m l e {Vhl }L I.

(5.197)

The optimal correction of m l is =

(m2, U[ml]m2) .

(5.198)

The a posteriori covariance matrix of the corrected value vhl is

V[?~I]-- Y[ml]- (?[ml]m2)(?[ml]m2)T (m2, ? [ m i ] m 2 )

"

(5.199)

Since V[Vhl]m2 -- O, the rank of V[vh]] is 1; its null space is {Vhl,m2}L. The residual

j --

(ml, m2) 2 (m2, V[ml]m2)

(5.200)

is a X2 variable with one degree of freedom. Hence, the orthogonality test given by eq. (5.196) can be applied.

5.5.2

C o r r e c t i o n of three o r i e n t a t i o n s

A. Simultaneous correction The same procedure can be applied to three orientations. Let m l, m2, and m3 be unit 3-vectors that indicate three orientations supposedly orthogonal (Fig. 5.12). The problem is finding A m i such that m i = m i - A m i satisfies

(~'~,i, ~'j) "-- (~ij,

i, j = 1, 2, 3.

(5.201)

The rank of this constraint is 3. Let V[mi] be the a priori covariance matrix of mi. If the three orientations are statistically independent, the problem can be written as the optimization

3 J-

~-~(Ami, < d [ m i ] - A m i ) ~ min

i=1

(5.202)

166

Chapter 5.

Geometric Correction

under the linearized constraint (m3, Am2) + (m2, Am3) = (m2, m3), (rhl, Am3)+ (rn3, Am1) -- ( m 3 , m l ) , (rh2, Am1)+ (ml,Am2) = ( m l , m 2 ) , mi e {rhi}Ll,

i = 1, 2, 3.

(5.203)

The first order solution is given by

( mx) ( o Vmlm3VEmxlm) 2 (,m2,m3,) Am2

--

Am3

V[m2]m3

0

V[m2]m I

V[m3lm2

W[m3lml

0

W

(m3,ml)

,

(ml,m2) (5.204)

where W is a (33)-matrix defined by W

/(o m3m2) ( viral] v[m~] m3 m2

0 ml

ml 0

) v[~]

(

0 m3 m2

m3 0 ml

m2 ml 0

))

(5.205)

A realistic form of the correction is i = 1,2,3.

~'1,i - N [ m i - Ami],

(5.206)

The a posteriori covariance matrices of the corrected values rhi are

V[ml] V['r~2, 'r~,l] V["~3, ~'Y/'I]

V[~,I, vh2] V[rhl, m3] ) V ['rll,2] V["~3, '/~2]

V['rh2, 'r?t,3] V['/~3]

o --

( V[ml]

f'[m~]ma ?[ml]~t2 )

~'[,ml)

V[m2l'rn3

0 V[m2]~'~t1 0 l}'[m3lrh2 ~Z[m3]'m 1 0

V[ml]~,3

?[m~],~ o ~[m~],~ ~[,m],~ where

f'[m~]

-

f~[m~]-Pm,V[,,~,lP,u,,

?[ml]~'h2 ) V[m2]~,l

,

(5.207)

0

i = 1,2,3.

(5.208)

5.5.

Orthogonality Correction

167

V[mi]

The matrix I ~ is obtained by replacing m~ and by ~h~ and IY[mi], respectively, in eq. (5.207). The residual of J can be written as - (

(m3, m l )

,W

(ml,m )

(m3, m l )

),

(5.209)

(ml,m )

which is a ~2 variable with three degrees of freedom. This fact provides an for three orientations: the hypothesis that the three orientations m i , i - 1, 2, 3, are orthogonal to each other is rejected with significance level a% if

orthogonalitytest

J > X ,a. B.

Correction

of one

(5.e10)

orientation

If m l and m2 are fixed in such a way that ( m l, m2) - 0, the rank of the constraint decreases to 2, and the linearized constraint is (ml,Am3)-

( m l , m3),

(m2, A m 3 ) -

(m2,m3),

(5.211)

e The optimal correction of m3 is

Am3-V[m3](ml'm2)W((me'm3))(m2,m3),

(5.212)

where W is a (22)-matrix defined by

W_ ((ml,V[m3]ml) (ml,V[m3]m2))-1 (m2, V [ m 3 ] m l )

(m2,V[m3]m2)

.

(5.213)

It is evident from the underlying geometry that ~4~3 - -~-ml x m2

(5.214)

is the exact solution if the sign is appropriately chosen. Hence, its covariance is V[,~I] = O. (5.215) The residual can be written as ,]

--

w ( l l ) ( m l , m3)2 +

2I~(12)(ml,m3)(m2,m3)+W(22)(m2,m3)2, (5.2 6)

which is a X2 variable with two degrees of freedom. Here, the matrix I~r = (l~ (kl)) is obtained by replacing m l , m2, and Vim3] by ~hl, ~h2, and Vim3], respectively, in eq. (5.213). The orthogonality test takes the form

j >

(5.217)

168

Chapter 5.

Geometric Correction

C. C o r r e c t i o n o f t w o o r i e n t a t i o n s

If m3 is fixed, the rank of the constraint is 3, and the linearized constraint is

(m3, Aml)--(m3, ml),

(m3, Am2) -- (m3, m2),

( A m l , m 2 ) + ( m l , A m 2 ) -- ( m l , m 2 ) ,

AT/%I E {Thl}t ,

Am2 6 {~2}L~.

(5.218)

The optimal correction of m l and m2 is given by

( iml ) _ ( V[~l,1]m3 ~m:

o

O u[m:].%

V[ml]m2 ) W

((m3,~D,1))

V[m~]m~

(m~,m~)

'

(5.2~9)

where W is a (33)-matrix defined by

W = ( ( m 3 , V[ml]m3)

\

(m3,V[m2]m3)

(ml,V[m2lm3) (m3,V[ml]m2) ) -1 (lYt,3,V[m2]m l ) (m2, V[ml]m2) + (ml, V[m2]ml)

(m2, V[~I]TYt3)

The a posteriori covariance matrices of the corrected values

(5.220)

m l and vh2 are

V[~rrt,1]-- Vii1] - w(ll)(i~,r[ml]m3)(~,r[ml]m3)T,

u[,~] VITal,m2]--

-

?Imp] w(~)(?[.~].~)(?[.~]m~) ~, -

-w(i2)(V[ml]m3)(?[m2]m3) T -

U[m2,~l] T,

(5.221)

where the matrix W is obtained by replacing m i and V[mi] by vhi and ? [ m i ] , respectively, in eq. (5.220). The residual can be written as J - Iiv(ll) (m3, m i ) 2 +

21~(12)(m3, mi)(m3,m2)+ Wi22)(m3,m1) 2,

(5.222) which is a ~(2 variable with two degrees of freedom. Hence, the orthogonality test given by eq. (5.217) can be applied.

5.6

Conic Incidence C o r r e c t i o n

Consider a conic (x, Qx) - 0 (see eq. (4.80)). Let x be an image point not on conic (x, Qx) = O. We consider the problem of optimally correcting x so as to make it incident to the conic (x, Qx) = O. In other words, we want to find A x such that 5~ = x - Ax satisfies (~, Q $ ) = 0.

(5.223)

5.6.

Conic Incidence Correction

169

The rank of this constraint is 1. Let V[x] be the a priori covariance matrix of x. The problem can be written as the optimization J-

(Ax, V [ x ] - A x ) ~ min

(5.224)

under the linearized constraint 1

( ~ , Q~) - 5(~, Q~),

(5.225)

The first order solution is given by ~,~

-

(x Q x ) V [ x ] Q x ' 2(x, QV[x]Qx) "

(5.226)

If we put n - N[Qx], eq. (5.226) can be written as

~_

(~,~)v[~]~ 2(., v[~]~)

(5.227)

This problem can be viewed as imposing the incidence constraint on the image point x and its polar (n, x) = 0 with respect to the conic (x, Q x ) = 0 (see eq. (4.82)). The difference in the factor 2 (see eq. (5.82)) is due to the fact that as the image point x approaches, its polar (n, x) - 0 also approaches its pole x by the same distance. The a posteriori covariance matrix of the corrected position :b is

v i i i - v[~]- (v[~]a)(v[~la)~ (a, via]a) '

(5.22s)

i-~ = N[Q~,].

(5.229)

where Eq. (5.228) has the same form as eq. (5.83). Hence, the rank of V[5~] is 1; its null space is {k, n } i , which is orthogonal to the orientation vh - N[k • nl of the polar (~, x) - 0. The residual of J can be written as

j -

(~, Q~)~ 4(~,,QV[x]Q~,)'

(5.230)

which is a X2 variable with one degree of freedom. This fact provides a conic incidence test: the hypothesis that image point x is on conic (x, Q x ) = 0 is rejected with significance level a ~ if

j > ~2,a.

(5.231)

170

Chapter 5.

Geometric Correction

X

Fig. 5.13. Optimal incidence correction for an image point. E x a m p l e 5.6 If each coordinate is perturbed independently by Gaussian noise of mean 0 and variance e 2, the covariance matrix of x is V[x] = e2Pk . The optimal correction (5.226) reduces to

A x - (x, Q X ) P k Q x . 2[[PkQx[[2

(5.232)

Consequently, x is displaced onto the conic perpendicularly (Fig. 5.13). The a posteriori covariance matrix of the corrected value 5~ is

V[5~]- E2ff~vhT,

(5.233)

where rh - N[k x r is the orientation of the polar (r x) = 0. The residual (5.230) can be written as

j-

1 (~, Q~)~ 4e 2 [[pkQS~[[2.

(5.234)

Hence, an unbiased estimator of the variance e2 is obtained in the form

~ = 1 (~, Q:~):

411PkQSc[I 2"

(5.235)

The value ~ thus estimated equals half the distance between the image point and its polar with respect to the conic. If the value e is given a priori, the conic incidence test takes the form

~:2 E--~ > ~2,a.

(5.236)

Chapter 6 3-D Computation by Stereo Vision As seen in the preceding chapter, the covariance matrix plays a fundamental role in any type of optimization. In the beginning, the a priori covariance matrices of raw data are determined by the characteristics of the sensing device. Then, the raw data define geometric objects, and the resulting geometric objects in turn define another class of geometric objects, which define geometric objects in a higher level and so on. In this process, the error characteristics can be traced bottom up: the covariance matrix of an object is computed from the covariance matrices of the objects in the lower hierarchy. In this chapter, we do such an analysis for stereo vision. We first study the epipolar constraint of a stereo system and derive explicit expressions for optimal 3-D reconstruction of points and lines. Then, the error behavior of the reconstructed space points and space lines is analyzed. We also derive expressions for optimal back projection of image points and image lines onto a space plane and analyze the behavior of the errors involved. Finally, we evaluate the effect of camera calibration errors.

6.1

6.1.1

Epipolar

Constraint

Camera imaging geometry

Stereo vision is a m e a n s of r e c o n s t r u c t i n g 3-D s t r u c t u r e s from two-dimensional images by t r i a n g u l a t i o n using two c a m e r a s 1 . To do this, t h e c a m e r a g e o m e t r y m u s t be known, since 3-D s t r u c t u r e s are r e c o n s t r u c t e d by inverting the imaging process. As discussed in Section 4.1.1, the c a m e r a g e o m e t r y is modeled as perspective projection: a space point P is p r o j e c t e d onto the intersection p of the image plane with the line of sight t h a t s t a r t s from the center of the lens and passes t h r o u g h P (see Fig. 4.1). T h e center of the lens is called the viewpoint, and the distance f from the viewpoint to the image plane is called the focal length. Define an X Y Z c o o r d i n a t e s y s t e m by identifying the origin O with the viewpoint and t a k i n g the Z-axis along the optical axis of the lens. T h e unit of length is scaled so t h a t the focal length f is unity. First-order analysis based on covariance matrices is justified if image noise is very small as c o m p a r e d with the focal length, which is usually the case. Given two cameras, define an X Y Z c o o r d i n a t e s y s t e m for the first camera, and an X I Y ~ Z ~ c o o r d i n a t e s y s t e m for the second (Fig. 6.1). T h e two cameras 1More than two cameras can be used to enhance the reliability of the reconstruction. If the number of cameras needs to be specified, such terms as binocular stereo vision and trinocular stereo vision are used. 171

172

Chapter 6.

3-D Computation by Stereo Vision P

X

,

1

""',,

~ O

X'

Z'

Y' O'

Y

Fig. 6.1. Geometry of stereo vision. are assumed to have the same focal length. Let h represent the origin O ~ of the X ~ Y t Z ~ coordinate system with respect to the X Y Z coordinate system; we call it the base-line vector. Let R be the rotation matrix that describes the orientation of the X~Y ~Z ~ coordinate system relative to the X Y Z coordinate system. The relative configuration of the two cameras is specified by the pair {h, R}, which we call the motion parameters (see Section 4.6.1). Consider a space point which has coordinates (X, Y, Z) with respect to the X Y Z coordinate system. Let (X ~, Y~, Z ~) be the coordinates of the same space point with respect to the X~Y ~Z ~ coordinate system. If we put r - (X, Y, Z) 1and r ' = ( X ' , Y ' , Z ' ) -7, the following relationship holds (see eq. (4.116)): r - h + Rr'.

6.1.2

(6.1)

Epipolar equation

Let p be the perspective projection of a space point P onto the image plane of the first camera, and pl that for the second camera. Evidently, the three vectors Op, O 0 ~, and O~p~ must be coplanar (Fig. 6.1). Hence,

(6.2)

Idp, oh,, orp, I - o,

where I', "," I den~ the scalar triple product (see eq. (2.32)). Eq. (6.2) is known as the epipolar constraint; the plane defined by alp, O~9', and Orp' is called the epipolar plane. Let (x,y) be the image coordinates of p, and (x', y') those of p'. The image points p and p~ are respectively represented by the following 3-vectors (see eq. (4.1))" x --

,

x' -

(X) y' 1

.

(6.3)

Since the X~Y~Z! coordinate system is rotated by R relative to the X Y Z coordinate system, vector O~p~ is represented by R x ~ with respect to the

6.1.

Epipolar Constraint

173

P

n = N[h •

(n,x) =0

0

x Fig. 6.2. Epipole and epipolar.

X Y Z coordinate system. Noting that Op = x and 0 0 ' - h, we can express eq. (6.3) with respect to the X Y Z coordinate system in the form Ix, h, Rx'] = O,

(6.4)

which we call the epipolar equation. The epipolar equation provides a strong clue to automatically detecting point-to-point correspondences between the two images. Consider the image plane of the first camera. For a fixed value of x', eq. (6.4) defines an image line if x is regarded as a variable. This line is called the epipolar of x I. Eq. (6.4) states that the epipolar of x' passes through x. Let pe be the image point represented by h xe = (k, h ) ' (6.5) where k = (0, 0, 1)7-. This point is called the epipole 2 of the first image plane (Fig. 6.2). Since Ix~, h, R x ' I = 0 holds irrespective of the value of x', all

epipolars pass through the epipoIe. Consider the image plane of the second camera. For a fixed value of x, eq. (6.4) defines an image line if x' is regarded as a variable. This line is called the epipolar of x. Eq. (6.4) states that the epipolar of x passes through x ~. Let Pie be the image point represented by

x~

Rr h (k, R_Ch)

(6.6)

This point is also called the epipole of the second image plane. Since Ix, h, P~'el - 0 holds irrespective of the value of x, all epipolars passes through

the epipole. The above observations are summarized as follows: 9 The image point p that corresponds to an image point p' in the other image is located on the epipolar of pl, and vice versa. 2In a real camera, the size of the image frame is finite, so the epipole pe may be located outside the image frame, or it can be an ideal image point located at infinity. In the following analysis, the image plane is assumed to be infinitely extended.

174

Chapter 6.

3-D Computation by Stereo Vision Z

X

**.-'~

P

,,""" Z

0

Y

P

x

-~ O'

\

jy'

Fig. 6.3. Parallel stereo system. 9 The epipolar in each image is determined by the location of the corresponding point in the other image and the motion parameters {h,

R}.

9 In each image, epipolars are concurrent, all passing through the epipole. If we define the essential matrix G = h x R,

(6.7)

the epipolar equation (6.4) can be written as = 0.

(6.8)

Hence, the epipolar in the first image is an image line represented by n = ax'.

(n, x) = 0,

(6.9)

The epipolar in the second image is an image line represented by (n', x') - 0 ,

6.1.3

n'--GTx.

(6.10)

Parallel stereo system

Let us call a stereo system parallel if the optical axes of the two cameras are parallel and the base-line vector is perpendicular to them (Fig. 6.3). If the Y-axis is taken in the direction of the base-line vector, the motion parameters have the form h-

(o) (o 01) h 0

,

R

= I.

(6.11)

The essential matrix (6.7) reduces to

G-h

0 0 0 -1 0 0

.

(6.12)

6.2.

Optimal Correction of Correspondence

175

The epipolar equation (6.4) is simply

x-x'

=0,

(6.13)

which defines a line parallel to the y-axis on each image plane. The epipole is an ideal image point located at infinity in the direction of the base-line vector. The epipolar constraint simply states that corresponding points must have the same x coordinate. We now show that any stereo system can be regarded as a parallel stereo system by changing the camera coordinate systems. Consider a stereo system with motion parameters {h, R}. Let j - N [ h ] , and let k be an arbitrary unit vector orthogonal to j . If we define ~ - j x k, the set {i, ), k} is an orthonormal system. Define a new X Y Z coordinate system for the first camera by taking {i, ~, k} as the axis orientations. An image point represented by x with respect to the X Y Z coordinate system is now represented by 3 /~7- x -

,

(6.14)

where

h - (i, j, k). Similarly, define a new . ~ ' Y ' Z ' coordinate system for the second camera by taking the same axis orientations. An image point represented by x' with respect to the X ' Y ' Z ' coordinate system is now represented by I~TRx' -'

-

.

(6.16)

With respect to the X Y Z and X ' Y ' Z ' coordinate systems, the stereo system is parallel with motion parameters {llhllj, I}. 6.2

Optimal

6.2.1

Correction

of Correspondence

Correspondence detection and optimal correction

In order to do error analysis based on corresponding image points, we need to consider the way they are detected. Basically, there are two possibilities:

9 Edge-based method. We first detect corresponding edge segments. Then, point-to-point correspondence is established by computing the intersection of one edge segment with the epipolars of the pixels of the other edge segment. a Eq. (6.14) defines a two-dimensional projective transformation if the image plane is identified with the two-dimensional projective space by adding ideal image points. If R runs over all rotations and k is its third column, the set of the corresponding transformations is a subgroup of the two-dimensional projective transformations.

176

Chapter 6. ."-,..

f" ,, f

3-D Computation by Stereo Vision

,.

,,.

,*

O' O Fig. 6.4. point.

Corresponding image points are corrected so that they define a space

9 Feature matching method. We match feature points directly: for a small neighborhood of each feature point in one image, the other image is searched for a portion that has a maximum correlation with that neighborhood. Many types of correlation measure and searching strategy are conceivable. The edge-based method has the advantage that a large number of corresponding pairs can be obtained easily. However, computational difficulties may arise for those edge segments which are nearly parallel to the epipolars. Moreover, the motion parameters {h, R} must be known accurately for computing the epipolars. In contrast, the feature matching method does not require any knowledge of the motion parameters. If they are known, the epipolar constraint can be used for limiting the search region--only a small neighborhood of each epipolar needs to be searched. In Chapter 11, we will show that the motion parameters {h, R} can be computed if a sufficient number of corresponding feature points are detected. The feature matching method is also suitable if natural or artificial markers are involved or humans intervene by using cursors and mice. In this section, we assume that corresponding points are detected by the feature matching method. The case in which correspondence is given between straight edge segments will be studied in Section 6.4. If corresponding image points x and x' are detected by the feature matching method in the presence of image noise, they may not strictly satisfy the epipolar equation (6.4). This means that their lines of sight do not intersect in the scene. In order that a unique space point be determined, image points x and x' must be corrected so as to satisfy the epipolar equation exactly. In geometric terms, this means modifying x and x' so that their lines of sight meet in the scene (Fig. 6.4). This problem is equivalent to finding A x and A x ' such that ~ = x - A x and 5~' = x ' - A x ' satisfy the epipolar equation

(e, a e ' ) = 0.

(6.17)

The rank of this constraint is 1. Let V[x] and V[x'] be the a priori covariance matrices of x and x', respectively. If x and x' are statistically independent,

6.2.

177

Optimal Correction of Correspondence

the problem can be written as the optimization J-

(Ax,

V[x]-Ax)+ (Ax', V[x']-Ax') ~ min

(6.18)

under the linearized constraint (Ax,

Gx') + (x, GAx') - (x, Gx'), ~, ~'

e {k}~.

(6.19)

The first order solution is given as follows (see eq. (5.17))"

Gx')V[x]Gx' (x', GTV[x]Gx ') + (x, GV[x']G Tx)' (x,

AxAx'=

(x',

(x'Gx')V[x']GTx 9 GTV[x]Gx ') + (x, GV[x']G Tx)

(6.20)

Since the constraint (6.19) is obtained by a linear approximation, corrections x +-- x - Ax and x' +-- x ' - Ax' need to be iterated until (x, Gx') = 0 is sufficiently satisfied (see Section 5.1.3). The a posteriori covariance matrices of the corrected positions ~ - x - Ax and 5:' - x ' - A x ' are computed as follows (see eqs. (5.31)and (5.32)):

(V[~]G~') (V[~]G~') ~

V[~]- V[~]-

(Jc', GTV[x]GSc ') + (5:, GV[x']GTSc) '

v[~']- v[~']-

(v [,'] a ~ ~) (v [~'] a ~ ~) 9 (5:', GTV[x]GSc ')

+ (~c,GV[x']G TS)'

(V[x]GScl)(V[x']G T Sc) T

V[~, ~'] = -(~,, GTV[x]G~,) + (~, GV[x,IGT~) 6.2.2

= V[5:', 5:] q-.

(6.21)

Correspondence test and noise level estimation

The residual of the function J given in eq. (6.18) can be written as follows (see eq. (5.34))"

j --

(~' a~')~

(~', a~v[~]ae ') + (~, av[~']a~e)

.

(6.22)

This is a X2 variable with one degree of freedom if the noise is Gaussian (see Section 5.1.5). This fact provides a correspondence test: the hypothesis that image points x and x' correspond to each other is rejected with significance level a% if

J > ~,a.

(6.23)

178

Chapter 6.

3-D Computation by Stereo Vision

As discussed in Section 5.1.5, it is usually very difficult to estimate the absolute magnitude of image noise, but often its geometric characteristics (e.g., the degree of homogeneity/inhomogeneity and isotropy/anisotropy) can be relatively easily predicted. Let us assume that the covariance matrices V[x] and V[x'] can be written in the form V [ x ] - e2Vo[x],

V [ x ' ] - e2Vo[x'],

(6.24)

where matrices V0[x] and V0[x'] are known but the constant e2 is unknown. We call V0[x] and V0[x'] the normalized covariance matrices and e the noise level. From eq. (6.20), we see that the optimal correction is not affected if the covariance matrices Y[x] and V[x'] are replaced by the normalized covariance matrices V0[x] and V0[x'], respectively. The unknown noise level e can be estimated a posteriori as follows. If Y[x] and V[x'] in eq. (6.22) are replaced by V0[x] and V0[x'], respectively, the right-hand side is multiplied by 1/e 2. Since J is a X2 variable with one degree of freedom, an unbiased estimator of e2 is obtained in the following form (see eq. (5.38)):

-- (5:', GTV0[x]GS: ') + (5:, GVo[x']GTS:) Its expectation and variance are given as follows (see eqs. (5.39)):

E[~ 2] = E2,

V[~ 2] -- 2E4.

(6.26)

If the value e is given a priori, the X2 test (6.23) takes the following form (see eq. (5.40)): ~2 "~ ~" ~2,a. (6.27) E x a m p l e 6.1 If each coordinate is perturbed independently by Gaussian noise of mean 0 and variance e2, the covariance matrices of x and x' are V[x] - Y[x'] - e2Pk . The optimal correction (6.20) reduces to

(x, G x ' ) P k G x ' AT,

--

IIPkGT~II ~ § IIPkG~'II ~' Ax' =

(x'Gx')PkGTx 9 IlekGTxll 2 + IlekGx'll 2

The a posteriori covariance matrices (6.21) become

Viii- ~2 (Pk -

(PkG~')(PkC~')T) IIPkGT~ll2+llPkG~'ll 2 '

(6.28)

3-D Reconstruction of Points

6.3.

(PkGT'x')(PkGT~)T)

v[~']- r ( e k V[5c, 5c'] -

E2

-

179

IIPkGT~II 2+IIPkG~:'II 2

(PkGhCt )(PkGT yc)T

'

= V[5~', 5~1T.

(6.29)

IIPkGT~II 2 + IlekG~'ll 2 An unbiased estimator of the variance e2 is obtained in the form

~2 =

(z, 0~') 2 IIPkGT 5=II2 + IIPkG5='II 2

9

(6.30)

E x a m p l e 6.2 Consider the parallel stereo system described in Section 6.1.3 with the noise characteristics given in Example 6.1. Let (x,y) and (x',y') be the corresponding image points on the first and the second image planes, respectively. The optimal correction (6.28) reduces to

Jc-

(x + x')/2 ) y 1

^,

(x+y,x')/2 )

(6.31)

1

The residual (6.22) is simply

1 (xJ = ~-~=

x') ~

(6.32)

which gives an unbiased estimator of e2 in the form

~2

1 x,)2 =~(x-

(6.33)

The a posteriori covariance matrices (6.29) reduce to ( 1/2

v[~] = v[~'] = ~ ( ~/2

1

, V[~, ~'] - e2 0

0

0)

(6.34)

6.3

6.3.1

3-D Reconstruction

of Points

Depth reconstruction

Let & and &' be the corrected positions of the corresponding image points on the first and the second image planes, respectively. Since the epipolar equations (&, G&') = 0 is satisfied, there exist constants Z and Z' such that

Z& = h + Z'R&'.

(6.35)

180

Chapter 6.

3-D Computation by Stereo Vision i

i

i i

x~_..~f

!

!

i

Rx'

x

~

~

h

O'

0

0 ~

h

O'

(b)

(~)

Fig. 6.5. (a) The feature point is infinitely fax away. (b) The feature point is in the direction of the base-line vector h. We call Z and Z' the depths of x and x', respectively; they are the distances of the reconstructed space point P from the X Y plane of the first camera and from the X ' Y ' plane of the second camera, respectively (see eq. (6.1)). Taking the vector product with RS~' on both sides of eq. (6.35), we obtain Z#c x

RS~'- h

x

RS~'.

(6.36)

Taking the vector product with ~ on both sides of eq. (6.35), we obtain O = #c x h + Z'5: x RS:'.

(6.37)

From these, we obtain the depths Z and Z' in the following form 4" Z-

(h x R5~', ~ x R5~') ]l~xR~'ll 2 '

Z'-

(h x 5, ~ x R ~ ' ) II~xP~'l] ~ "

(6.38)

The corresponding space point is given by r-

Z~.

(6.39)

Eqs. (6.38) have a singularity: the depths Z and Z' cannot be determined when (and only when) 5: x R ~ ' - 0. This configuration occurs in two cases" 9 The space point P is infinitely far away from the two cameras (Fig. 6.5a). 9 The space point P and the two viewpoints O and O' are collinear, i.e., the space point P is projected onto the epipoles of the two image planes (Fig. 6.5b).

4The depths Z and Z' can be equivalently written in various different forms such as Z = llh x R~'II/II~ x R~'I] and Z' = l]h • ~[I/ll~ x R~'II. Among them, eqs. (6.38) are the most convenient. For one thing, this form makes sense even if the depth is negative (i.e., the object is behind the camera).

6.3.

3-D Reconstruction of Points

181

E x a m p l e 6.3 For the parallel stereo system described in Section 6.1.3, the space point r reconstructed from corresponding image points (x, y) and (x', y') has the depths h Z- Z'= (6.40) D' -

-

where D - y - y~ is called the disparity of the two image points.

6.3.2

Error behavior of reconstructed space points

The depths Z and Z ~ are defined in such a way that eq. (6.35) holds. If 5: and ^1 x are perturbed by AS~ and A~ ~, respectively, the resulting perturbations AZ and AZ ~ of Z and Z ~ satisfy to a first approximation

AZYc

AZ'RS~'- ZA~, + ZIRA~, I.

-

(6.41)

Taking the vector product with R ~ ' on both sides, we obtain

~z(n~') • ~- -z(n~') x ~

+ z'(n~') x n ~ ' .

(6.42)

The unit surface normal to the epipolar plane, on which 5:, h, and R ~ ' lie, is given by i ~ - X[h x 5]. (6.43) Taking the inner product with ft on both sides of eq. (6.42), we obtain AZ(/~, (RS~') x 5:) - - Z ( ~ , (RS~') x ASz) + Z'(~, (R~') x R A ~ ' ) .

(6.44)

If we define rh = / ~ x R~',

(6.45)

eq. (6.44) can be written as ~z

= - ('~' z~

- Z'RLx~') (rh, &)

"

(6.46)

Using the relationship (6.47) and noting that E[ASzA~ T] _ V[Sz], E[A~ASz 'T] -- V[5~, 5~'], and E[A~'AS~ 'T] = Y[5~'] (see eqs. (6.21)), we obtain the variance V[Z] - E[(AZ) 2] of Z from eq. (6.46) in the following form:

z~(m, v[~],~) V[Z]

-

2zz'(,~, v[~, ~']R~,~) + z'~(,~, RV[~']RT,~) I1~ x R~'II 2 (6.48)

182

Chapter 6.

3-D Computation by Stereo Vision

The covariance vector V[5~, Z l - E[A&AZ] has the form

v[~, z] - - ( z v [ ~ ] -

z'v[~, ~']R~),~ (,~,~)

(6.49)

The covariance matrix of the reconstructed space point r = Z& is given by

v[,] -

z:v[~] + 2zs[v[~, z]~ ~] + v [ z ] ~ ~,

(6.50)

where S[. ]is the symmetrization operator (see eqs. (2.205)). E x a m p l e 6.4 Consider the parallel stereo system described in Section 6.1.3 with the noise characteristics given in Example 6.1. Eqs. (6.48) and (6.49) reduce to 2e2Z 4 e2Z 2 V[Z] h~ , Y[~, Z] h j' (6.51) where j - (0, 1, 0)7-. Put ^

~* - ~(~ + ~'),

(6.52)

which represents the midpoint of & and &~ if the two image planes are identifled. The covariance matrix V[r] of the reconstructed space point r given by eq. (6.50) reduces to V [ r ] = e 2 Z( 2

P k + - D-Fx4 , x , T ) ,

(6.53)

where D - y-y~ is the disparity. It is geometrically evident that the error in r is proportional to e and Z. Eq. (6.53) also implies that the error in orientation x* is very large. If x* ~ k, for instance, the error is approximately isotropic around the Z-axis but its magnitude along the Z-axis is about 2/D times as large as in the direction orthogonal to it. If the focal length of the camera is 700 pixels and the disparity is 10 pixels, for example, we have 2/D -- 140. If the disparity is 1 pixel, we have 2/D - 1400. Thus, the uncertainty of the Z coordinate is very large as compared with that of the X and Y coordinates.

E x a m p l e 6.5 Fig. 6.6 are simulated stereo images (480 x 680 pixels with focal length f = 600 (pixels)) of a cylindrical grid. Gaussian noise of mean 0 and standard deviation a - 2 (pixels) is independently added to the x- and ycoordinates of each grid point, so the noise level is e = a / f = 1/300. However, the value of e is treated as unknown in the 3-D reconstruction computation and estimated a posteriori by using eq. (6.30) for each grid point. Fig. 6.7 shows the standard confidence regions (see Section 4.5.3) of the grid points computed from eq. (6.50).

6.3.

3-D Reconstruction of Points

183

Fig. 6.6. Simulated stereo images. E x a m p l e 6.6 Fig. 6.8 are real stereo images (512 x 512 pixels with focal length f - 1870 (pixels)) of a surface of a building. Fig. 6.9a shows a grid pattern defined by feature points (corners of the windows) extracted from the left image in Fig. 6.8. Assuming that the image noise is isotropic, we estimate the noise level by eq. (6.30). Fig. 6.9b displays the reconstructed grid and the two shapes that envelop the primary deviation pairs of the grid points (see Section 4.5.3). In this example, the base-line length Ilhll is very short as compared with the distance to the surface (approximately 1/16), so a very small error in camera calibration sensitively affects the computation, thereby reducing the reliability of 3-D reconstruction. Since the noise level is estimated from "the degree to which the epipolar equation is not satisfied", the error in the motion parameters is also treated as "image noise".

6.3.3

M a h a l a n o b i s distance in the scene

If a particular stereo system with known motion parameters {h, R} is fixed and particular characteristics of the image noise are assumed, eq. (6.50) can be viewed as defining an uncertainty field over the 3-D scene. Given an arbitrary space point r, its depths Z and Z ~ are computed by Z = (k, r),

Z' = (Rk, r - h).

(6.54)

The corresponding image points are given by 1

x - 2r,

x'-

1

~-TRT(r -- h).

(6.55)

By construction, the epipolar equation Ix, h, R x ~] = 0 is satisfied. Identifying x and x ~ with & and &~, we can compute the covariance matrix V[r] from eqs. (6.21), (6.48), (6.49), and (6.50). Hence, V[r] is a function s of r, and 5As we noted earlier, singularities exist along the base line OO ~. In reality, 3-D cannot be reconstructed in the part outside either of the views of the two cameras, including the

184

Chapter 6.

3-D Computation by Stereo Vision

Fig. 6.7. 3-D reconstruction and standard confidence regions of grid points.

Fig. 6.8. Real stereo images.

:-:v--,v7~Ti_,~ 1

(~) (b) Fig. 6.9. (a) Extracted feature pattern. (b) 3-D reconstruction (solid lines) and the two shapes that envelop the primary deviation pairs of grid points (dashed lines).

6.3.

3-D R e c o n s t r u c t i o n of Points

185

R2

x .......

x

o,/

............

4

2

........~....

y,

X

/

/

........................

1t

Z

y'

Fig. 6.10. Geometry of 3-D reconstruction by stereo. it defines the Mahalanobis distance 6 t h a t m e a s u r e s the degree of u n c e r t a i n t y in the scene; a p e r t u r b a t i o n in the s a m e M a h a l a n o b i s distance has the same probability density (see Section 3.3.4). T h e following geometric i n t e r p r e t a t i o n is given to this distance. An image plane can be identified with a two-dimensional C a r t e s i a n space 7~2, a n d the direct s u m x | can be viewed as a point in t h e four-dimensional direct sum C a r t e s i a n space ~ 4 _ 7~2 | 7~2. T h e set of all x | x ' t h a t satisfy the epipolar e q u a t i o n (6.4) defines a t h r e e - d i m e n s i o n a l manifold $ in ~ 4 . T h e 3-D r e c o n s t r u c t i o n by eqs. (6.38) can be t h o u g h t of as establishing a one-to-one and onto m a p p i n g from the manifold S to the scene 7~3. Fig. 6.10 schematically illustrates these relations, where the ellipses indicate the stand a r d confidence regions defined by the covariance matrices. T h e covariance m a t r i c e s V[x] and V[x'] define the M a h a l a n o b i s distances t h a t m e a s u r e the u n c e r t a i n t y of x and x ' in the two images. T h e optimization (6.18) m e a n s projecting each direct s u m point x | x ' E 7~4 = 7~2 x ~ 2 onto the "nearest point" & | 5~' in S m e a s u r e d in the M a h a l a n o b i s distance defined by the direct s u m covariance m a t r i x V[x] | Y[x']. Eqs. (6.20) describe this projection, and eqs. (6.21) define the s t a n d a r d confidence region of & @ &' in the tangent space Ta~e~,(S ) to S at 5~ | :b'. Eq. (6.50) can be viewed as defining the M a h a l a n o b i s distance in 7~3 by m a p p i n g 7 the M a h a l a n o b i s distance in $ in accordance with the 3-D r e c o n s t r u c t i o n equation r - Zh~; part behind the cameras, but mathematically the value of V[r] is defined everywhere except at singularities. 6This distance is generally Riemannian (non-Euclidean) and defines a nonzero Riemannian curvature in the scene. Hence, the 3-D scene can be regarded as a Riemannian space. 7Mathematically, this process of projecting the distance in T~4 onto 8 and mapping the distance in 8 to T~3 is defined by a procedure called pull-back of a tensor field.

186

Chapter 6.

3-D Computation by Stereo Vision

""...-~-_,

n r

....~

O "-...~ n

O' Fig. 6.11. 3-D reconstruction of a space line. the standard confidence region of :b | &~ in Tx~x, ($) is mapped to define the standard confidence region of r in 7r 3.

6.4

6.4.1

3-D

Reconstruction

of Lines

Line reconstruction

Image lines are also very important image features. As we noted in Section 6.2.1, they are usually detected as edge segments, i.e., sequences of pixels. If straight lines are fitted to them (the fitting procedure will be discussed in subsequent chapters), point-to-point correspondence is easily established once line-to-line correspondence is obtained: for a given image point p on one image line l, the corresponding image point p~ on the other image line l' is at the intersection s of the epipolar of p with l'. By construction, the corresponding image points p and p' satisfy the epipolar equation. Hence, 3-D can be reconstructed pointwise. However, if the two image lines are represented by (n, x) = 0 and (n', x') = 0, a space line is directly reconstructed from n and n " all we need to do is compute the intersection of the space plane defined by the viewpoint O and the image line 1 with the space plane defined by the viewpoint O' and the image line l' (Fig. 6.11). In other words, the point-to-point correspondence

need not be computed. Note that a space line r x p - n is perspectively projected onto the image plane of the first camera as an image line (n, x) - 0 (see eq. (4.130)). In order to invert this projection, we must be careful about the scaling of the vector n. Recall that we adopted the scale I]nll - 1 in representing an image line (see eqs. (4.8)) while we adopted the scale Ilpll2 + Ilnll 2 = i in representing a space line (see eqs. (4.34)). For the convenience of computation, let us temporarily SAs pointed out in Section 6.2.1, computational difficulties may arise in the intersection computation if the epipolar is nearly parallel to l'.

6.4.

3-D R e c o n s t r u c t i o n of Lines

187

adopt the scale Ilnll - I for the {p, n } - r e p r e s e n t a t i o n of space lines 9. Let the two corresponding image lines 1 and l' be represented by (n, x) = 0,

(n', x ' ) = 0.

(6.56)

T h e space plane defined by the first viewpoint O and image line 1 is (n,r) =0.

(6.57)

T h e space plane defined by the second viewpoint O' and image line l' is

(Rn', r - h) = 0.

(6.58)

T h e intersection of these two space planes define a space line L. Let its equation be r • p = n. T h e vector p is given as follows (see eq. (4.72)): n

P -

x

(h,

Rn' Rn')"

(6.59)

After p is computed, the vectors {p, n } are normalized to if necessary.

6.~.2

Ilvll 2 + Ilnll = 1,

Error behavior of reconstructed space lines

First, assume the scale Ilnll = IIn'll = 1. If n and n ' are p e r t u r b e d by A n and A n ' , respectively, the vector p c o m p u t e d by eq. (6.59) is p e r t u r b e d to a first a p p r o x i m a t i o n by Ap =

A n x Rn' + n x R A n '

(h,

R A n ' ) n x Rn'

m

(h, R n ' )

(h, R n ' ) 2

(h, Rn')

(6.60)

Let V[n] and V[n'] be the covariance matrices of n and n ' , respectively. Assuming t h a t the two image lines are obtained by separately processing the two images, we regard n and n ' as i n d e p e n d e n t r a n d o m variables. T h e covariance matrices V[p] = E[ApAp T] and V[p, n] = E[ApAn T] are given as follows (see eq. (2.43)):

1

V[p] - (h, Rn,)2

( (Rn')

x V[n] x (Rn') + (h, RV[n']RTh)pp T

- 2S[n x RY[n']RThp T] + n x R V [ n ' ] R T

x.)

9If n is scaled to a unit vector, space lines that pass through the coordinate origin O cannot be represented in the form r x p = n. However, such space lines are "invisible" when viewed from O, so we need not consider them for the purpose of 3-D reconstruction.

188

Chapter 6.

v i v , ~] --

3-D Computation by Stereo Vision

-(R~') • v[n] (h, R n ' )

(6.61)

"

Once the covariance matrices V[p], V[p, n] (= V[n, plT), and V[n] are obtained for the vectors {p, n}, Ilnl[ - 1, the corresponding covariance matrices for the rescaled vectors {lb, ~t}, 11151]2 + II~tll2 - 1, are computed as follows (see eqs. (3.18))" (

V[lb]

V[lb,~t])_

1 ( V[p] V [ p , n ] ) ilpll + ilnll Ppeff, Vfn, p] V[n]

PPC~'ff" (6.62)

Here, Plben is the (66)-projection matrix onto {lb | fi}L~ (see eq. (4.48)).

6.5 6.5.1

Optimal Back Projection onto a Space Plane Back projection of a point

A. Image transformation b e t w e e n the two images

If a feature point is known to be on a space plane H whose equation is (nn, r) - d, a single image is sufficient to compute their 3D positions: all we need to do is back project the image point onto the space plane H along its line of sight (see Section 4.5.1). In fact, let x be an image point on the first image plane. If the reconstructed space point r - Zx is on space plane (nn, r) = d, we have (nn,Zx) - d. Hence, the depth Z is given by d

Z-

(nn,x)'

(6.63)

and the space point is reconstructed in the position dx

r = (nn, x~.)

(6.64)

It follows that a pair of stereo images have redundant information. We can take advantage of this fact: the reliability of 3D reconstruction can be enhanced by optimizing this redundancy. Suppose there exists no image noise, and consider a stereo system with motion parameters {h, R}. Let x be an image point in the first image. The corresponding image point x' in the second image satisfies Zx - h + Z ' R x ~ (see eqs. (6.1) and (6.35)). Hence,

--~R =

-

-

ZRT (

Z RT(hn~ -- dI)x. Z'd

X--

(nnd.X)h )

(6.65)

6.5.

Optimal Back Projection onto a Space Plane

189

It follows that the transformation between the two image planes can be written in the form x'-

(6.66)

kAx,

where A

-

(6.67)

-

The constant k is chosen so that (k, x !) = i holds. An image transformation in the form of eq. (6.66)is called a (two-dimensional) projective transformation 1~ or collineation (see Sections 4.4.2 and 4.6.1). Note that eq. (6.66) implies the epipolm" equation (6.4) and hence is a stronger condition than eq. (6.4). In fact, we see from eqs. (6.66) and (6.67) that Ix, h, R x ' [ - k ( x x h, R A m ) -

k ( x x h, ( h n ~ -

dl)x)

(6.68)

= k ( x x h, (nn, x ) h - dx) - O.

E x a m p l e 6.7 Consider the parallel stereo system described in Section 6.1.3. If we write n n - (nii(1),nii(2),niI(3)) -V, the transformation matrix A has the form

(

A -

0

hnn(1) 0

hnn(2) - d 0

0 )

hnii(3) -d

(6.69)

9

It follows that the constant k in eq. (6.66) should be chosen to be k - - 1 / d . In image coordinates, the transformation given by eq. (6.66) has the following form:

(x

(

0

x

0

(6.70) This is an affine transformation. It is easily seen that although the transformation between two images of a space plane is a projective transformation in general, it reduces to an affine transformation if and only if the two optical axes are parallel and the base-line vector is orthogonal to them, i.e., ( k , h ) = 0 and R k - k. As pointed out in Section 6.1.3, any stereo system can be equivalently treated as a parallel stereo system by applying the projective transformations given by eqs. (6.14) and (6.16) to the two images. It follows that the transformation between two images of a space plane reduces to an affine transformation for any stereo system if an appropriate transformation is applied to each image. X~ set of all image transformations in the form of eq. (6.66) for an arbitrary nonsingular matrix A forms a group of two-dimensional projective transformations with respect to the composition operation. Note that eq. (4.126) can be written as Q = A-'-Q~A/d 2 in terms of the matrix A defined in eq. (6.67).

190

C h a p t e r 6.

3-D C o m p u t a t i o n by Stereo Vision

X

X' ~X'

O

O'

Fig. 6.12. Two corresponding image points are back projected onto a space plane.

B. Optimal correction In the presence of noise, corresponding image points x and x ' do not necessarily satisfy eq. (6.66). Hence, we optimally correct them. In geometric terms, this m e a n s modifying x and x so t h a t their lines of sight meet exactly on the space plane H (Fig. 6.12). This problem is equivalent to finding A x and A x ' such t h a t 5~ = x - A x and 3' = x ~ - A x ' satisfy eq. (6.66) or equivalently 11 9 ' x A S = 0.

(6.71)

T h e rank of this constraint 12 is 2, since only two of the three c o m p o n e n t equations are algebraically i n d e p e n d e n t 13. Let Y[x] and Y[x'] be the a priori covariance matrices of x and x ' , respectively. As discussed in Section 5.1.1, the optimal correction is d e t e r m i n e d by the optimization

J = (Ax, V[x]-Ax)+ ( A x ' , V[x'l-Ax') --4 min

(6.72)

u n d e r the linearized constraint

x'

x

A A x - (Ax)

x

Ax'=

x'

x

Ax,

~,, ~,' e {k}f.

(6.73)

The first order solution is given as follows (see eq. (5.17)): A x - ( Y [ x ] A -r x x')W(x' x A x ) ,

~'=-(v[~']

• (A~,))W(~,' • A~).

(6.74)

11If eq. (6.71) satisfied, there exists a value k such that ~' - kAY; it is chosen so that (k, x') - 1 holds. 12This constraint is degenerate (see Section 5.1.3). 13The third one can be obtained by multiplying the first and the second ones by - ~ ' and -~3', respectively, and adding them together.

6.5.

Optimal Back Projection onto a Space Plane

191

Here, W is a (33)-matrix defined as follows (see Section 5.1.3): W-

(x' x A V [ x ] A -r • x' + ( A x )

)

• V[x'] • ( A x ) 2 "

(6.75)

Since the constraint (6.73) is obtained by a linear approximation, corrections x t - x - A x and x' +- x ' - A x ~ need to be iterated until the constraint (6.71) is sufficiently satisfied (see Section 5.1.3). The a posteriori covariance matrices of the corrected positions ~ and ~ are given as follows (see eqs. (5.31) and (5.32)):

V [ ~ l - v [ ~ ] - (V[~]A r • ~')W(V[~]A r • ~')r v[~'] = v [ ~ ' ] - (v[~'] • (A~))W(V[~'] • (A~)) T, V[~, ~ ' 1 - (V[x]A T • 5~t)I~(V[x '] • (A~)) -r - V[~', ~] T.

(6.76)

Here, the matrix I~r is obtained by replacing x and x t by 5~ and 5~~, respectively, in eq. (6.75). The 3-D position r of the back projected point is determined by the depth Z given by eq. (6.63). Its covariance matrix V[r] is computed from eqs. (6.48), (6.49), and (6.50). Since r is constrained to be on H, its covariance matrix V[r] is singular; the surface normal n n is its eigenvector for eigenvalue 0, and the remaining eigenvectors lie on H. C. I n c i d e n c e t e s t a n d noise level e s t i m a t i o n

The residual of J given in eq. (6.72) can be written as follows (see eq. (5.34)):

j-

(~' • A~, W(~' • A~)).

(6.77)

If the noise is Gaussian, the residual J is a X2 variable with two degrees of freedom (see Section 5.1.5). This fact provides an incidence test: the hypothesis that the space point defined by image points x and x ~ is on the space plane (nn, r) - d is rejected with significance level aT0 if

j > /~2,a.

(6.78)

Suppose the covariance matrices V[x] and V[x'] are expressed in terms of the normalized covariance matrices V0[x] and V0[x'] and the noise level e in the form of eqs. (6.24). From eqs. (6.74) and (6.75), we see that the optimal correction is not affected if the covariance matrices V[x] and V[x'] are replaced by the normalized covariance matrices V0[x] and V0[x'], respectively. The unknown noise level c can be estimated a posteriori in the form

~2_ 1

~(~'• A~, W0(~' • A~)),

(6.79)

192

Chapter 6.

where ~r

-

(~ ' x

3-D Computation by Stereo Vision

A V o [ x l A T x ~' + (A~) x V0[x'] x (A~)

)- .

(6.80)

2

The expectation and variance of ~2 are given as follows (see eqs. (5.39)):

E[~ ~] = d,

V[~ ~1 = ~ .

(6.8~)

If the value e is given a priori, the X2 test (6.78) takes the following form (see eq. (5.40)): ~2 X 2,a ~-~ > -5-"

(6.82)

E x a m p l e 6.8 Consider the paraJlel stereo system described in Section 6.1.3 with the noise characteristics given in Example 6.1. Suppose a feature point is known to be on space plane Z = d. Let (x,y) and (x',y ~) be its image coordinates on the first and the second image planes, respectively. According to eqs. (6.74) and (6.75), the two image points are respectively corrected into --

(y + y' + h / d ) / 2 1

,

5~' -

(y + y' - h / d ) / 2 1

.

(6.83)

The a posteriori covariance matrices (6.76) reduce to E2

V [ & ] - V [ & ' ] - V[&, 5~']- ~ P k "

(6.84)

The residual (6.77) is a f - _ 1~ ( ( x - x ' ) 2 + ( y - y ' - ~ h )

2).

(6.85)

Hence, an unbiased estimator of e2 is obtained in the form =~

+(y-

-~)

.

(6.86)

The depths Z and Z ~ computed by eqs. (6.38) are simply Z = Z'=

d.

(6.87)

Eq. (6.84)reduces eqs. (6.48) and (6.49)to

v[z] = o,

v[~, z] = o.

(6.88)

From eq. (6.50), we see that the covariance matrix of the reconstructed space point r is e2d 2

V [ r ] - ----~-Pk,

(6.89)

meaning that errors in r are constrained to be on the space plane Z - 1, on which the distribution is isotropic with variance ed/2 in each orientation.

6.5.

Optimal Back Projection onto a Space Plane

193

Fig. 6.13. Stereo images of a planar grid.

(~)

(b)

Fig. 6.14. (a) Simple back projection. (b) Optimal back projection. E x a m p l e 6.9 Fig. 6.13 shows two simulated stereo images (512 x 512 pixels with focal length f - 600 (pixels)) of a planar grid in the scene. The x and y coordinates of each grid point are independently perturbed by Gaussian noise of standard deviation cr = 5 (pixels), so the noise level is e = (r/f = 1/120, which is treated as unknown in the subsequent computation and estimated a posteriori by using eq. (6.79). The equation of the space plane on which the grid lies is assume to be known. Fig. 6.14a shows the back projected pattern computed by eq. (6.64) viewed from a different angle; Fig. 6.14b is the corresponding result obtained after the feature points are optimally corrected. We can clearly see that the correction enhances the accuracy of 3-D reconstruction.

6.5.2

Back projection of a line

Suppose we observe a space line which is known to lie on a space plane ( n n , r) = d. L e t / a n d l' be its stereo images; let (n, x) - 0 and (n', x') - 0 be their respective equations. The image point x ~ that corresponds to an image point x on 1 must be on l ~. If there is no image noise, the corresponding image points x and x' are related by eq. (6.66). Hence, every x that satisfies (n, x)

194

Chapter 6.

3-D Computation by Stereo Vision

O'

n

Fig. 6.15. T w o c o r r e s p o n d i n g i m a g e lines are b a c k p r o j e c t e d o n t o a s p a c e p l a n e . = 0 must satisfy (n',kAx) - 0 or ( A T n ' , x ) - 0, which implies 14

n - k'ATn',

(6.90)

where the constant k' is chosen so that I]k'ATn'l[ - 1 holds. In the presence of noise, corresponding image lines (n, x) - 0 and (n', x') - 0 do not necessarily satisfy eq. (6.90). Hence, we optimally correct n and n ' so that eq. (6.90) is strictly satisfied. In geometric terms, this means modifying the image lines l and l' so that the space plane defined by the viewpoint O and 1 meet the space plane defined by the viewpoint O' and l' exactly on the space plane H (Fig. 6.15). This problem is equivalent to finding A n and A n ' such that ~ - n - A n and ~' = n ' - A n ' satisfy eq. (6.90) or equivalently n x ATfi' = 0.

(6.91)

As in the case of image points, this constraint 15 has rank is 2. Let V[n] and V[n'] be the a priori covariance matrices of n and n', respectively. The problem can be written as the optimization

J = (An, V [ n ] - A n ) + (An', V[n']-An')--+ min

(6.92)

under the linearized constraint - - A T n ' X A n + n x A T A n t __ n X A T n ' , e

n st

solution is given An--(V[n]

e

onows

•

(6.93)

(5. lr))-

x ( A T n ' ) ) W ( n x ATn'),

a4If we note that ( A T ) -1 -- ( A - l ) T (see eqs. (2.21)), we see from eq. (6.90) t h a t n' -- ( A - 1 ) T n / k '. This is also a projective transformation (or collineation) of image lines. This transformation is contragradient to eq. (6.66). This duality between image points and image lines is a f u n d a m e n t a l property of projective geometry. 15This constraint is degenerate (see Section 5.1.3).

6.6.

Scenes Infinitely Far Away

195

~'-(v[~']A x ~)W(~ x A ~ ' ) .

(6.94)

Here, W is a (33)-matrix defined as follows (see Section 5.1.3): W =

( ( A T n ') x V[n] x ( A T n t) + n x A T V [ n t ] A

Since the constraint (6.93) is obtained by n t-- n + A n and n t ~ n ~ + A n t need (6.91) is sufficiently satisfied (see Section matrices of the corrected values ~ and ht and (5.32)):

v[q = ~[q-

)-2 .

(6.95)

a linear approximation, corrections to be iterated until the constraint 5.1.3). The a posteriori covariance are given as follows (see eqs. (5.31)

(C[q x (A~g'))W(~[q x (A~g')) ~,

V[g']- V[n']- (V[n']A x V[g,g']-

x n

g)W(V[n']Ax g)T,

(V[n] x ( A T g ' ) ) W ( V [ n ' ] A x g ) T = V [ g , , g l T .

(6.96)

r

(6.97)

Here, we define

~[n]- P,~V[n]Pn,

P,~v[n']P,~.

The matrix ~ r is obtained by replacing n and n ~ by ~ and h~, respectively, in eq. (6.95). The space line reconstructed by eq. (6.59) lies exactly on the space plane II. Its covariance matrix V[p | n] is computed from eqs. (6.61). The residual of J can be written as follows (see eq. (5.34)): J-

(~ x ATfi ', ~Z(~ x A T ~ ' ) ) .

(6.98)

If noise is Gaussian, the residual J is a X2 variable with two degrees of freedom (see Section 5.1.5). This fact provides an incidence test: the hypothesis that the space line defined by image lines ( h i , X ) = 0 and ( n 2 , x ) = 0 is on the space plane (nn, r) = d is rejected with significance level a% if

J > )~2,a.

(6.99)

If the noise level in the images is unknown, it can be estimated a posteriori as in the case of image point back projection.

6.6

6.6.1

Scenes Infinitely Far Away

Spacepoints infinitely far away

If x and x ~ are stereo images of a feature point that belongs to an object located practically infinitely far away (e.g., a mountain, a boat on the sea,

196

Chapter 6.

3-D Computation by Stereo Vision

...................... iiiiii............... ax

.......iiiii..................... .....

............

......

............ii....................

y

Ax' .........

Fig. 6.16. Two corresponding image points are corrected so that their lines of sight become parallel. or an airplane in the sky), then lines of sight of x and x' should be parallel. This condition can be written as

x-

kRx'

(6.100)

for some constant k. This equation may not necessarily be satisfied exactly in the presence of noise (Fig. 6.16). Consider the problem of optimally correcting them: we want to find A x and A x ' such that 9 - x - A x and 3' - x ' - A x ' satisfy eq. (6.100) or equivalently x R2'-0.

(6.101)

Like eq. (6.71), this constraint t6 has rank 2. Let Y[x] and Y[x'] be the a priori covariance matrices of x and x', respectively. As discussed in Section 5.1.1, the optimal correction is determined by the optimization J = (Ax, V [ x l - A x ) + (Ax', V[x']-Ax') ~ nfin

(6.102)

under the linearized constraint x x RAx'-

(Rx')

~,

~'

x Ax

-

x x

Rx', (6.103)

e {k}~.

The first order solution is given as follows (see eq. (5.17)): ~

= -(V[~l

~'-(v[~']R

• (R~'))w(~

• R~'),

~ x ~ ) w ( ~ x R~').

(6.104)

Here, W is a (33)-matrix defined as follows: w

=

(

(R~') • v[~] • (R~')+

~ • RV[~']R ~ • ~

16This constraint is degenerate (see Section 5.1.3).

)2

.

(6.105)

6.6.

Scenes Infinitely Far Away

197

Since the constraint (6.103) is obtained by a linear approximation, corrections x +-- x - A x ' and x' +---x ' - A x need to be iterated until the constraint (6.101) is sufficiently satisfied (see Section 5.1.3). The a posteriori covariance matrices of the corrected positions 5: and 5:' are given as follows (see eqs. (5.31) and (5.32)): viii-

•

• 9

•

9

•

V[Sz, Sc'] - (V[x] • ( R ~ c ' ) ) W ( V [ x ' ] R T • 5:)7- - V[Yc',Sc] q-.

(6.106)

The matrix TV is obtained by replacing x and x ~ by 5: and 5~~, respectively, in eq. (6.105). The residual of J given in eq. (6.102) can be written as follows (see eq. (5.34)):

J-

(x • R x ' , W ( x • R x ' ) ) .

(6.107)

If the noise is Gaussian, the residual j is a X2 variable with two degrees of freedom (see Section 5.1.5). This fact provides an infinity test: the hypothesis that the space point defined by image points x and x' is infinitely far away is rejected with significance level a% if Y > ~2,a.

(6.108)

In actual applications of stereo vision, feature points reconstructed to be very far away from the cameras are often disregarded as meaningless, but the judgement as to how far is far enough is usually done ad hoc. The above procedure provides a rigorous statistical criterion. Suppose the covariance matrices V[x] and V[x'] are expressed in terms of the noise level e and the normalized covariance matrices V0[x] and V0[x'] in the form of eqs. (6.24). From eqs. (6.104) and (6.105), we see that the optimal correction is not affected if the covariance matrices V[x] and V[x'] are replaced by the normalized covariance matrices V0[x] and V0[x'], respectively. The unknown noise level e can be estimated a posteriori in the form 42 - ~(x 1 • R x ' , I2r

• Rx')),

(6.109)

where

Wo - ((R:i:') • Vo[x]

• (Rh:') + ~ • R V o [ x ' ] R T •

The expectation and variance of 42 E[~ 21 -.- 62 '

are

5:)2.

(6.110)

given as follows (see eqs. (5.39)): V[~ 2] --- E4.

(6.111)

198

Chapter 6.

3-D Computation by Stereo Vision

If the value e is given a priori, the ;~z test (6.108) takes the following form (see eq. (5.40)): ~2 X2,a e--5- > --~-. (6.112) E x a m p l e 6.10 Consider the parallel stereo system described in Section 6.1.3 with the noise characteristics given in Example 6.1. Suppose a feature point is known to be infinitely far away. Let (x, y) and (x', y') be its image coordinates on the first and the second image planes, respectively. Eqs. (6.104) and (6.105) lead to

- ~'-

(tx+xl ) (y+y')/2 1

.

(6.113)

Namely, they are corrected into their "midpoint" if the two image planes are identified. The a posteriori covariance matrices (6.106) reduce to E2

v [ e ] - v [ e ' l - v[e, e ' ] - -~Pk.

(6.114)

The residual (6.107) is 1

j = ~

((x- x') ~ + ( u - u')~) 9

(6.115)

An unbiased estimator of e2 is obtained in the form ~2

6.6.2

1 =~((x-x') ~+(u- yt)2 ).

(6.116)

Space lines infinitely f a r away

Let 1 and I t be stereo images of a space line located infinitely far away. If ( n , x ) = 0 and (n',x') = 0 are their respective equations, the space planes (n, r) = 0 and (Rn', r) = 0 should be parallel to each other. This condition can be written as n = kRn' (6.117) for some constant k. the presence of noise n and nt: we want n ' - A n ' satisfy eq.

This equation may not necessarily be satisfied exactly in (Fig. 6.17). Consider the problem of optimally correcting to find A n and A n t such that f = n - A n and fit = (6.117) or equivalently fix R~'-0.

As in the case of image points, this constraint 17 has rank 2. 17This constraint is degenerate (see Section 5.1.3).

(6.118)

6.6.

Scenes Infinitely Far Away

199 .~ . - - " "

o

iiiiii! iiiii!!!iiill. . . . . . . .

Fig. 6.17. Two corresponding image lines are corrected so as to define a space line located infinitely far away. Let V[n] and V[n'] be the a priori covariance matrices of n and n', respectively. The problem can be written as the optimization J = (An, V [ n ] - A n ) +

(An', V [ n ' ] - A n ' )

--+ min

(6.119)

under the linearized constraint -Rn'

x An + n x RAn'

~ . e {-}~.

= n x Rn',

~ . ' e { n ' }L" 9

(6.120)

The first order solution is given as follows (see eq. (5.17)):

zx~ = - ( v [ ~ ]

• (R.'))w(.

zx.'-(v[,~'lR ~ x ~)w(n

• R.'), • R.').

(6.1Zl)

Here, W is a (33)-matrix defined as follows (see Section 5.1.3): W -

(

(Rn') x Vial x (Rn')+

n x RV[n']R T x n

)-

2

.

(6.122)

Since the constraint (6.120) is obtained by a linear approximation, corrections n +-- n + A n and n' +-- n ~ + A n ' need to be iterated until the constraint (6.118) is sufficiently satisfied (see Section 5.1.3). The a posteriori covariance matrices of the corrected values/~ a n d / t ' are given as follows (see eqs. (5.31) and (5.32)):

v[,~]- r

(9"[r4 • (m~'))g~(C'[n] x (R,~')) T,

v[~'] = ~ [ n ' ] - (~[n']R ~ x .~)W(9[n']R T x ~)~. V[/~,/~']- (V[n]

x

(R/t'))I~(Ik[n']R T

x

~)T = V[/t',/~]q-.

(6.123)

Here, we define I?[n] = Pr V [ n ] P i z ,

l?[n'] = P n V[n']Po,.

(6.124)

200

Chapter 6.

3-D Computation by Stereo Vision

The matrix I ~ is obtained by replacing n and n ~ by ~ and ~1, respectively, in eq. (6.122). The residual of J can be written as follows (see eq. (5.34)):

(6. 25) If noise is Gaussian, the residual J is a X2 variable with two degrees of freedom (see Section 5.1.5). This fact provides an infinity test: the hypothesis that the space line defined by image lines (nl, x) = 0 and (n2, x) = 0 is infinitely far away is rejected with significance level a% if J > k~,,.

(6.126)

If the noise level is unknown, it can be estimated a posteriori as in the case of image points.

6.7

Camera

6. 7.1

Calibration

Errors

Errors in base-line

So far, our analysis has been based on the assumption that the motion parameters and the focal length are accurately calibrated beforehand. In reality, however, accurately estimating such parameters is very difficult. We now analyze how the accuracy of the camera parameters affects the accuracy of 3-D reconstruction. The effect of calibration errors is very different from the effect of image noise: the statistical characteristics of all space points reconstructed from perturbed camera parameters are correlated, while the effect of image noise is generally independent from point to point. For simplicity, errors in the camera parameters are assumed to be independent of image noise. This means that when we analyze the effect of errors in the camera parameters, we can assume that image noise does not exist, since the combined effect of image noise and calibration errors is, to a first approximation, a superimposition of the effect of image noise with accurate calibration and the effect of calibration errors with no image noise. First, assume that errors in the base-line vector h are independent of errors in other parameters and image noise. For the reason stated above, other parameters and image points can be assumed accurate. Let V[h] = E[AhAhT] be the covariance matrix of h. Since the image points are assumed correct, we have ~ - x and ~1 _ xl. If h is perturbed into h + Ah, the image points ~ and 5~~ are corrected into 5~ + AS~ and ~ + A~ !, respectively, so as to satisfy the perturbed epipolar equation for G ~ = (h + Ah) x R. Noting that (5~, G~') = 0, we put A~-

(~, G ' ~ ) =

(~, Ah x R ~ ) -

-(5~ x

R S ~ , A h ) - - ( d , Ah),

(6.127)

6.7.

Camera Calibration Errors

201

where a - 5 x P.,5'.

(6.128)

According to eqs. (6.20), the correction of the image points has the form

~X~V[~IGe' (5', GTV[x]G5 t) '1- (5, GV[xt]G TS~)' aev[x']GVe A&' = - ( 5 ' , GTV[xlG5 ') + (5c,av[x']G T ge)"

(6.129)

Hence, the a posteriori covariance matrices of the corrected points are given by V[~,] V[~I(VixlG~')(V[x]G~')T

(~', aTv[~]C~ ') + (~, av[~'laT~) V[~']

V[~](V[~'laT~)(VWIOT~)T

--

(~', a~-v[xla~, ') + (~, Grin']aTe) v[~,~'] -

VH(V[~]a~')(V[~']GT ~')T ((5 t, oTv[XlOx 1) + (X, aV[=tl~.Tx)) 2'

(6.130)

where v[a] = E[Ae2I = (a, V[h]&).

(6.131)

We also obtain

v[h, ~} = -

V[h,~](V[x]GS') T (5', GTV[xlG5 ') + (5, GV[xqG TS)'

V[h,~](V[x']GT5) T V[h,~'] - -(5,,GTV[x]GS,) + (x, GV[x,IGTS) ,

(6.132)

where

V[h, ~] = E[AhA~] = -V[hlfi,.

(6.133)

If h in eq. (6.35) is perturbed by Ah, eq. (6.41) is replaced by

AZ5 = AZIR51 + A h - ZA5 + ZIRA5 I,

(6.134)

from which we obtain AZ-

-(rh, ZA5-

Z'RA~,'- Ah) (,h, ~)

(6.~35)

202

Chapter 6.

3-D Computation by Stereo Vision

where rh is defined by eq. (6.45). The variance by

V[Z] =

E[(AZ) 2] of Z is given

1

y[z] =

x R 'll z (

- 2ZZ'(rh, V[~, ~']Rq-rh)

+ Z'2(rh, RV[~'lRq-rh) - 2Z(rh, V[~, hlrh)

+ 2Z'(~z, RV[~',hlrh) + (~h, V [ h ] r h ) ) .

(6.136)

The covariance vector V[5~, Z] = E[AYcAZ] has the form

V[hc Z] '

ZV[~]gn - Z'V[5c, ~']Rq-~h - V[]c, h]~Zn -

(rh, ~)

.

(6.137)

The covariance matrix of the reconstructed space point r - Zh: is given by eq. (6.50). Although the above computation involves the a priori covariance matrices V[x] and V[x'], the result is invariant to multiplication of them by any positive constant. Hence, the covariance matrices V[x] and V[x'] need to be given only up to scale for this analysis.

6. 7.2

Errors in camera orientation

Suppose the rotation matrix R that describes the relative orientation of the two cameras is not accurate. If R is perturbed into R + AR, both R and R + A R are rotation matrices, so the difference is also a small rotation. Since a small rotation is given in the form of eq. (2.57), there exists a small vector A ~ such that to a first approximation A R = A a • R.

(6.138)

This means that the second camera is further rotated around axis A ~ by angle [IA~II, where the vector A ~ is defined with respect to the first camera coordinate system (see eqs. (2.58) and (3.32)). The covariance matrix of R is defined by V [ R ] - E [ A ~ A ~ q-] (see eq. (3.33)). We assume that errors in R are independent of errors in other parameters and the image noise. It follows that other parameters and image points can be assumed accurate, so we let ~ = x and ~ = x ~. If R is perturbed into R + A R , the image points 5~ and 5~~ are corrected into 5: + A~ and 5:~+ A~ ~ respectively, so as to satisfy the perturbed epipolar equation for G ~ = h • ( R + A ~ • R). Put A~ = (5, G ' ~ ) =

(5, h • ( A a • R ~ ) )

= ( h , / ~ ' ) ( 5 : , A a ) - (5, Rhe')(h, A a ) - - ( b , A a ) ,

(6.139)

6.7.

Camera Calibration Errors

203

where

b - (5:, R ~ ' ) h - (h, RS:')']c.

(6.140)

The corrections AS: and AS:' are given in the form of eqs. (6.129). Hence, the a posteriori covariance matrices V[~], V[5:'], and V[~, ~'] are given in the form of eqs. (6.130), where eq. (6.131)is replaced by V[~] -

E[A~ 2] - (b, V[h]b).

(6.141)

Similarly, the covariance matrices V[R, 5:] - E[AaA~7-1 and V[R, 5:'] E[Ag~A~ 'q-] are given in the form

v[~,~] = -

V[n,~](V[.lG~') T

(~c', aTv[x]G~c ') + (~c,aV[x']G TS:)' V[n,~](V[.'IGT~) T

(6.142)

V [ R , ~c'] = - (~,, G T V [ x ] G S , ) + (~c, a V [ x ' ] G T ~) ' where

V[R, ~] - E[AY2A~] - -V[R]b.

(6.143)

If R in eq. (6.35) is perturbed by A R - ~ x R, eq. (6.41) is replaced by

A z ~ - AZ'R~' - Z'(R~') • A ~ -

Z A ~ + Z'RA~',

(6.144)

from which we obtain ~xz -

- ('~'' z ~ x ~

-

z'n~x~')

-

z'll~'ll~(~, A a )

(6.145)

(~h, ~) Hence, the variance v[z] -

V[Z] - E[(AZ) 2] of Z is given by

1

Jl~ • n~'ll ~ ( z~('~' v[~]~h) -

2zz'(,~,

+ Z '~ (,~, R v [ ~ ' l n T , ~ ) -

v[~, ~ ' ] n ~ h )

2z'll~'ll ~(z(m, v[~, Rla)

- Z'(,~,RVK~',R]~)) + z'Zll~'ll4(~, v [ R ] ~ ) ) .

(6.146)

The covariance vector V[5:, Z] has the form v[~,

z]

-

- zv[~]m

-

z'v[~,

~']R wm - Z'll~'llzv[~, R]~ (rh, &)

.

(6.147)

The covariance matrix of the reconstructed space point r - ZS: is given by eq. (6.50). As in the case of base-line vector errors, the covariance matrices V[x] and V[x'] need to be given only up to scale for this analysis.

204

Chapter 6.

3-D Computation by Stereo Vision

..........................................................................: ii/:::............... : ( x 'y

~~~

1

(~,~)

0

Fig. 6.18. Image coordinates (x, y) are computed with respect to a perturbed focal length f.

6.7.3

Errors in focal length

As mentioned in Section 6.1.1, the focal length f is the distance between the image plane and the viewpoint O. In 3-D analysis, we take f as the unit of length (see also Section 4.1.1). However, the focal length f is also difficult to calibrate accurately. Let f be the correct focal length. Consider an image point p whose correct image coordinates are (2, ~). The physical location of p measured in pixels is (f2, f~). If (x, y) are the image coordinates of p computed by using a perturbed value f = f + A f, we have

((/+ ~:)x, (/+ ~:)v) = (/~, /~),

(6.148)

since the physical location of the image point should be the same (Fig. 6.18). From eq. (6.148), we obtain to a first approximation = x + -ATfx ,

~ = ~ + -ATf~ .

(6.149)

In other words, the true location of an image point which appears to be at x is

5c - x + ~ - P k X .

(6.150)

A. I n d e p e n d e n t c a m e r a s

Consider the case in which the two cameras have different focal lengths f and ft, which are separately calibrated. Then, errors in f and errors in f~ are regarded as independent. Suppose f is not accurate but f~ is accurate. Other parameters and image points are also assumed accurate. Let x and x ~ be corresponding image points. Because f is not accurate, x and x ~ do not

6.7.

Camera Calibration Errors

205

exactly satisfy the epipolar equation, but their true locations 2 and 5:' (= x') do. From eq. (6.150), we see that to a first approximation

Hence,

(x, ax')

= - A--L(x,

f

Pk Gx').

(6.152)

The image points x and x' are corrected into 5: = x + Ax and &~ = x ~+ Ax so as to satisfy (~, G ~ ' ) = 0. The corrections Ax and Ax' are given by eqs. (6.20). Let A~ = 5 : - ~. Then,

AS: -

x + A x - 5: - - ~ - P k x +

Ax

.I

= f~ Pkx + (x', GTV-[x-~x--;) --(--(xi-~x']Gq-x)

"

(6.153)

Similarly,

~I

(*.Pka*')v[~']GT*

Let V[f] = E [ A f 2] be the variance of f. The covariance matrices of the corrected position & and &' are given as follows:

(*'Pka~')V[*]G~'

v[f]( v[~]= /~ Pk*+

(pe~+ v[~'] =

(X'PkGx')V[x]Gx' ) GTV[x]Gx ') + (x, GV[xqG T x)

V[f] (x, PkGX')2(V[x']GTx)(V[x']GTx) -r f~ (~,. aTV[.]G. ') + (~. aV[.']a T*)

V[f](

V[Jz, 5:'] -

(x',

)

f2

(

)'

(x, PkGx')V[x]Gx' ) Pk x + (x', GTV[x]Gx ') + -~I-~x'IGTx) (x, Pk Gx')(V[x'IGT x) T (x', GTV[x]Gx ') + (x, GV[x']G vx)" (6.155)

The subsequent analysis is the same as in Section 6.3.2. Namely, the covariance matrix of the reconstructed space point r = Z~ is computed by eqs. (6.48), (6.49), and (6.50). The covariance matrices Y[x] and Y[x'] need to be given only up to scale.

206

Chapter 6.

If f is accurate but f ' is not, leads to

V[f']

v[~] -

3-D Computation by Stereo Vision

V[f'] being

its variance, a similar analysis

(x, GPkx')2(V[x]Gx')(V[x]ax')T (~', GTV[~Ia~ ') + (~, aV[~']GT~)

V[f'] ( x, V[x'l -

(x, GPkx')V[x'] GTx ) Pk + (x ' , G T-~x ]-~x' ) + -~ :-G-~ qG Tx )

if2

(

y [ ~ , ~']

(x, GPkx,)V[x,]GT x

+

V[f']

IT

(x, GPkx')V[x]Gx'

-

f,2 (x', GTv[x]Gx ') + (x, GV[xqG Tx)

(

(x, GPkx,)V[xqGTx Pk ' +

:i

)T 9 (6.~56)

If f and f ' are both inaccurate but independent, the error behavior of 3-D reconstruction is described to a first approximation as the superimposition of t h a t for inaccurate f and accurate f ' and that for accurate f and inaccurate f'. B. Identical

cameras

Consider the case in which the two cameras are identical and have the same focal length but the value f we use is not necessarily correct. Let V[f] be its variance. In this case, eq. (6.152) is modified to

(x, Gx') -

Af s ((*Pk, ax') + (~, GPk,'))

Af

f (x, (Pk G + GPk)x' ).

(6.157)

The corrections AS~ and AS~' are given as follows: (x,

~'=

(Pk G + GPk)x')V[x]Gx'

(x, (Pk G + GPk)x')V[x']GTx ) ~'I Pk~' + f (x', GTV[x]Gx ') + (x, GV[x']GTx)

.

(6.~58)

6.7.

Camera Calibration Errors

207

Hence, viii -

V[f]f2 IRk x + ( Pk~ +

v[~']

V[f] (x'

(x, (PkG + GPk)x')V[x]Gx' ) (x, (Pk G +

GPk)x')V[x]Gx'

(x' (PkG+ GPk)x')V[x/]GTx )

-

T

Pk x' + ( x ; : G T ~ g ~ i

,E:I(

+(-x:

I-,(-~ §

v[~.~']= :.- Pk-+ (.,i~ev-~a-~u165

(

GV[x'IGTx)

)

(x, (PkG + GPk)x')V[x']GTx ) T . (6.159) V[x]

V[x']

Again, the covariance matrices and need to be given only up to scale. The subsequent analysis is the same as given in Section 6.3.2.

Chapter 7 Parametric Fitting This chapter studies methods for fitting a geometric object to multiple instances of another geometric object in an optimal manner in the presence of noise. First, the criterion for optimal fitting is derived in general terms as maximum likelihood estimation. The covariance matrix of the resulting estimate is explicitly obtained, and the statistical behavior of the residual of optimization is analyzed. This analysis provides a criterion for testing the hypothesis that the observed objects are in a special configuration. Then, we study various types of fitting problems in two and three dimensions such as finding an optimal average, estimating a common intersection, and fitting a common line or plane. In two dimensions, this theory predicts the statistical behavior of an image line fitted to an edge segment.

7.1

7.1.1

General

Theory

Parametric fitting

We consider the problem of defining a geometric object from a number of data. Let u be the vector t h a t represents the object to be determined, and let a l , . . . , a N be the data. Suppose the d a t a a l , . . . , a N are vectors t h a t represent N instances of the same type of object (e.g., image point). They are supposed to satisfy the same geometric relationship (e.g., incidence) with the object in question in the absence of noise. We want to determine the value of u by optimally imposing the relationship on the data. This type of problem frequently arises in computer vision and robotics applications. Let n be the dimension of the vector u. Its domain is assumed to be an n ~dimensional manifold b/ C 7~n, which we call the parameter space. Consider a n object represented by an m-vector a. Its domain is assumed to be an m~-dimensional manifold A C 7~m, which we call the data space. Suppose N different instances a l, ..., aN of vector a are observed in the presence of noise. Let 5~ be the true value of am, and write a, = a, + Aa,,

a = 1, ..., N .

(7.1)

Assuming t h a t each am has been obtained by a separate process, we regard the noise A a a as a statistically independent r a n d o m variable of mean 0 and covariance m a t r i x V i a l ] . Since the d a t a are constrained to be in the d a t a space ,4, the noise A a ~ is, to a first approximation, constrained to be in the tangent space Ta~ (A) to the manifold A at ~ . Hence, the domain and 209

210

Chapter 7.

Parametric Fitting

the null space of the covariance matrix ~r[aa] are Ta~ (,4) and Ta~(A) • respectively. Suppose L smooth functions F (k) ( . , 9)" ~ m • 7~" --+ 7~ exist and the true values al, ..., aN satisfy

F (k) (d~, u) - 0,

k - 1, ..., L,

(7.2)

for a - 1, ..., N for some value u. Eq. (7.2) imposes a parameterized constraint on { a s }; we call this the hypothesis, because the value of the parameter u is unknown. Our goal is to optimally estimate the parameter u from the data a l , ..., aN and their covariance matrices V[ai], ..., V[aN]. We assume that the hypothesis (7.2) is nonsingular in the sense discussed in Section 5.1. The L equations (7.2) may not be independent; we call the number of independent equations the rank of the hypothesis (7.2). To be specific, the rank is the codimension of the manifold ,S defined by the L equations F(k)( 9, u) -- 0, k - 1, ..., L, in the data space j[ (see Section 3.2.1). We call S the (geometric) model of the hypothesis (7.2). It can be shown I that the rank of the hypothesis (7.2), or the co dimension of the model S, generally coincides with the dimension of the linear subspace

9a - { P ~ VaF(1)(aa, u),

p Aa. VaF(L)(a,, U)}L

C T~ m

(7.3)

for all a, where PaA is the m-dimensional projection matrix onto the tangent space Ta. (A) at a s . If the dimension of this linear subspace is smaller than the rank of the hypothesis (7.2), we call a s a singular datum; otherwise, it is a nonsingular datum. In the following, we assume that singular data are removed from the fitting data. However, the dimension of the linear subspace defined by eq. (7.3) may not be equal to the dimension of the linear subspace

)]a - {PaA.VaF(i)(a,,u),...,PaA.VaF(L)(aa, u)}L C T~m

(7.4)

for a s # a s (see Section 5.1.1); we say that the hypothesis (7.2) is degenerate if the dimension of the subspace ];a is larger than the dimension of ];a (see Section 5.1.3).

7.1.2

Maximum likelihood estimation

Our approach consists of two stages 2" the correction stage and the estimation stage. A. Correction stage

We first assume a particular value of the parameter u. If the data { a s } do not satisfy the hypothesis for the assumed value u, we optimally corrected them 1Detailed discussions will be given in Chapter 14. 2The proof that the following approach is indeed optimal will be given in Chapter 14.

7.1.

General Theory

211

by applying the theory given in Chapter 5. Namely, we find A a s such that a s = a s - A a s satisfies eq. (7.2). As discussed in Section 5.1.1, this correction is done for each a by minimizing the Mahalanobis distance IIAasl]~[a~], i.e., by the optimization

(7.5)

Js - ( A a s , V [ a s ] - A a s ) --+ min.

This can be justified as maximum likelihood estimation for Gaussian noise (see Section 5.1.1). Eq. (7.2) imposes the following linearized constraint on A a s 6 Ta~(A)"

(VaF(k)(as, u ) , A a s ) - F(k)(as,u),

k-

1,...,L.

(7.6)

The solution of the optimization (7.5) is given as follows (see eq. (5.17)): L

A a s -~?[a~] ~

W-(kl) s (u)F (k)(as, U)VaF (l)(as, u).

(7.7)

k,l=l

Here, I?r

- (l?~(kl) ( u ) ) i s the (LL)-matrix defined by

(17V(kt) ) - ((VaF(k) (as, u), V[as]VaF(l) (as, u))) - .

(7.8)

As in Section 5.1.2, this expression is the abbreviation of I2r - Vs(u)for the (LL)-matrix V s ( u ) - (V(~kl)(u)) defined as follows (see eqs. (5.15) and (5.16))" (7.9) It can be proved 3 that the rank of matrix V s ( u ) (hence of matrix IgVs(u)) equals the rank r of the hypothesis (7.2). B. Estimation stage

If A a s is an independent Gaussian random variable of mean 0 and covariance matrix V[as], the joint probability density of all {Aas}, a - 1, ..., N, has the following form (see eq. (3.46))"

Ir

(7.10)

This probability density can be viewed as the likelihood of the observed values {Aas}, a - 1, ..., N. Let Js be the residual of Js obtained by substituting 3The proof will be given in Chapter 14.

212

Chapter 7.

Parametric Fitting

eq. (7.7) into eq. (7.5). The likelihood takes the form

~--

/3:1

a-'l

v/(2~)"~I?[a~]l+

which is a flmction of u alone. We now seek the value u that maximizes this likelihood. This is equivalent to minimizing the sum of the residuals j~, which we write as Jlul" I. J N

J[u]-

~

Jm ~ min.

(7.12)

or--1

Substituting eq. (7.7) into eq. (7.5), we see that this minimization takes the following form (see eq. (5.35)): N

J[u] - Z m=l

L

~

lrV(~al)(u)F(k)(am,u)F (l) (am, u) -4 min.

(7.13)

k,l=l

C. P r a c t i c a l c o n s i d e r a t i o n s

In practice, the function J[u] cannot be computed from the data alone, since l~(kt)(u) involves the true value a~. Hence, as we did in Section 5.1.3, the (LL)-matrix I~m(u) - (17V(~kt) (u))is approximated by the (LL)-matrix Win(u) - (lVt~kt) (u)) obtained by using the rank-constrained generalized inverse (see eq. (2.82)):

( ( V a F (k) (a~, u), V[am]VaF (l) (am, u)) )- .

(W.(k0 (u)) -

(7.14)

7-

Here, r is the rank of the hypothesis (7.2). Eq. (7.14) is the abbreviation of Win(u) - (V~(u))-~ for the (LL)-matrix Vm(u) - (V(~kz) (u)) defined by (V(kl) (u)) - ( ( V a F (k) (a~, u), V[a,~]VaF (0 (a~, u))).

(7.15)

The rank-constrained generalized inverse is used because matrix Vm(u) has a larger rank than matrix IP~(u) if the hypothesis (7.2) is degenerate (see Section 5.1.3). Thus, the actual computation is N

J[u] - ~ a=l

L

~

W (kt) (u)F (a) (am, u)F (0 (am, u) -+ min.

(7.16)

k,l=l

Since u is constrained to be in the parameter space b/C T~n, the minimization search is done in H. In principle, the solution/~ can be obtained by numerical

7.1.

General Theory

213

computation, and many kinds of numerical software are available for that purpose (e.g., the quasi-Newton method and the conjugate gradient method). In the rest of this chapter, we assume that the optimal s o l u t i o n / t has been obtained by some numerical means; actual computational schemes for this optimization will be given in subsequent chapters.

7.1.3

Covariance matrix of the optimal fit

The optimal e s t i m a t e / t is a random variable, because it is computed from the data {am}. We now study its statistical behavior. To do this, we need to distinguish u regarded as a variable from its true value. Let ~t be the true value that satisfies the hypothesis (7.2), and put

(7.17)

u-~+Au.

Since u is constrained to be in the parameter space b/ C T~~, the variation A u is constrained, to a first approximation, to be in the tangent space T~t (L/) to the manifold L / a t r If we substitute am - aa + A a a and u = ~t + A u into F(k)(aa, u) and expand it in the neighborhood of ~ and ~, we obtain F (k) (am, u) - ( V a / ~(k) , A a ~ ) + (Vu/~(k) , A u ~ ) + O ( A a , , A t ) 2,

(7.18)

where Va/)(k) and Vu/~(k) are the abbreviations of VaF(k) ( ~ , ~) and VuF(k)(da, ~t), respectively. The symbol O(...)P denotes terms of order p or higher i n . . . . Noting that 1 ~ (kl)(u) - 1~(kl)(~) + O ( A u ) , we see from eq. (7.13) that N

L

aao))

:[,,1- Z E c~=1 k , l = l

((Vu/~('), A t ) +

(Va/~(0, A a ~ ) ) +

O(Aa~,Au) a.

(7.19)

Let {vi}, i - 1, ..., n - n', be an orthonormal basis of Tu(b/)• constraint A u E T~t(/d) can be written as (oi, A t ) - 0,

i - 1, ..., n - n'.

The

(7.20)

The minimum of eq. (7.19) is obtained by introducing Lagrange multipliers ~_~/-l__ n I

~i and differentiating g[u]- z-,i=l higher order terms, we obtain -

N

L

c~----1 k , l ' - i

-

Ai(Oi,Au)

with respect to Au. Ignoring n-n

I

i--1

(7.21)

214

Chapter 7.

Parametric Fitting

Let p U be the n-dimensional projection matrix onto the tangent space Tu (H) at ft. Multiplying p u on both sides and noting that p U A u = Au and pUoi , we obtain - 0 , i = 1, .. . , n - n '

~=1

)

k,l=l

u w.(~')(~)P. v ~ G-(k) (VaL(~

+~ a=l

k,l=l

T

)

a ~ = o.

(7.22)

If we define the moment matrix M by N

L

c~=l

k,l=l

W,

(~)

)(puvulV~(O)

,

(7.23)

Aa~.

(7.24)

eq. (7.22) can be written in the following form:

l~Au = - E c~--1

- (kz)(~)(PgVul~(k) )(Va ~(1)) T W~ k,l=l

)

The moment matrix M is an (nn)-matrix, and its range is contained in Tu(H). Hence, the rank o f / ~ is at most n'. Here, we assume that the number of data {a~ } is sufficiently large and no special relationship exists among &l, ..., aN other than eq. (7.2). Then, the range of M generally coincides with Tu(H). It follows that /I5//~- = M M = pU. (7.25) Multiplying / ~ - on both sides of eq. (7.24) and noting that M p U A u = Au, we obtain

Au = - l f / I - E a=l

l~,.(kO(f~)(pUvu[-'(k))(VaF('))-r k,l=l

)

MAu =

Aa,~.

(7.26)

The covariance matrix l?[/~] - E[AuAuT-] of the optimal estimate/~ - ~ + A u is computed as follows: N

L

/ ,-,/4v~ ~(k) o',~= 1

k,l,m,n=l

(Va te~(')) n-E[Aa, Aa~](Val~('))(pU Vu F ( m ) ) r / ( / / N

-M- E c~,fl= 1

L

E k,l,m,n=l

W.

(fi)I~* (m')(~)(PgVuk (k))

7.1.

General Theory

215

(v~ f~') )9 a~, ~[~ 1(v~ L (n))(eg v~ P~m))~ MN

L

-M-E k,l,rn,n=l E s=l

(V~f~'), r = M- ~

(eg V~(m))r

E

-

)

W~') (~)e~ 'n)(~)W(m~)(~)

s=l k,rn=l l,n=l

(eg v ~ L (~))(eg v~ L(~) )9 u - .

(7.27)

(~)~,r(In)(~l,)~v(rnn)(~t)equals

Here, we have used eq. (7.9). Since E l,m=l L l~(kl) the (kn) element of matrix Iizs(~)I~r~ ( f t ) - I ~ s ( f t ) - l ~ ( f t ) we obtain N

r

(see eq. (2.81)),

L

M- Z ~ w~')(~)(egv~r~))(egv~L(')) ~ s=l k,/=l

=M

M M

- M

(7.28)

.

Thus, the covariance matrix of the optimal e s t i m a t e / t is given by

% s=l

(~)(P~Vu

)(PgV~L ('))

(7.29)

, =1

which has rank n'; its null space is Tu(b/) • Eq. (7.29) coincides with the Cramer-Rao lower bound 4 on the attainable accuracy of parametric fitting. Eq. (7.29) is purely a theoretical expression because it involves the true values of u and as. A simple approximation to compute it form the data alone is replacing the true value ft by the optimal estimate it and the true value a s by the optimally corrected value &s - a s - A a s , where A a s is given by eq. (7.7). Then, the covariance matrix V[~] takes the form

W(kt) ( i t ) ( p U Vu/5(k))(pU Vu~,(z))7

V[it] s=l , =1

)

(7.30)

where Vu/~(k) is the abbreviation of V u F (k) (&s, it). However, the data {as } themselves can be used instead of the corrected values {&s } to a first approximation. In whichever form, the resulting covariance matrix V[it] also has rank n'; its null space is T~t(b/) • 4Detailed discussions will be given in Chapter 14.

216

Chapter 7.

Parametric Fitting

Hypothesis testing and noise level estimation

Z1.4

The above estimation procedure is based on the hypothesis that the data { a s } are random deviations from the values { ~ } that satisfy eq. (7.2); minimizing the function J[u] defined by eq. (7.13) can be interpreted as choosing the value/t that make the hypothesis the most likely. If the hypothesis is correct, the residual J[/t] should be 0 for the true values { ~ }. However, the residual is generally positive for the data {as }. This suggests that if the residual J[/t] is much larger than can be accounted for by the statistical behavior of the noise in {a~}, the hypothesis should be rejected. In order to formulate this process as a statistical test, we need to derive the probability distribution of the residual j[/t]. We do this by assuming that the noise is Gaussian. A. Testing of a strong hypothesis

As a preliminary step, consider the residual J[~] for the true value ~t. Letting A u - 0 in eq. (7.19) and neglecting higher order terms, we observe that N

J[u]-

E

L

E

c~--1

ITV(kO(u)(VaF(k)'Aa~)(VaF(O'Aa~)"

(7.31)

k,l--1

If we put

(7.32) the L-vector e~ (e (1) Its covariance matrix is _

e~(L) ) T is a Gaussian random variable of mean 0

V[eo~] = ( E [ e ( k ) e ~ ) ] ) - ((VaF~k),E[Aao, Aa:]VaT'(~O)) = ((Va.F'(k),V[a~]VaF~O))- (12(k0(,R))- (l~(k0(O,)) - '

(7.33)

where eqs. (7.9) and (7.8) are used. Hence, eq. (7.31) can be written as N

=

(7.34) oe=l

The rank of V[ec~] (= W~(~t)-) equals the rank r of the hypothesis (7.2), and each e~ is an independent Gaussian random variable. Hence, the residual J[~] is a X2 variable with rN degrees of freedom (see eq. (3.63)). Its expectation and variance are given as follows (see eq. (3.59)): _

E [ J [ ~ ] ] - rN,

V [ J [ f i ] ] - 2rN.

(7.35)

It follows that if a particular value ~ is given independently of the data {as }, the stronghypothesis that the true value is ~t can be tested by the standard

7.1.

General Theory

217

the k2 test (see Section 3.3.4). Namely, the strong hypothesis is rejected with a% significance level if

J[u] > X~N. (7.36) In practice, however, the function J[u] defined by eq. (7.13) cannot be computed, since I7V~ (kt) involves true value a~. Hence, J[u] is approximated by the function J[u] given by eq. (7.16). B. T e s t i n g of a w e a k h y p o t h e s i s

Next, consider the residual J[/t] for the optimal estimate it. Substituting eq. (7.26) into eq. (7.19) and using eqs. (7.24) and (7.29), we obtain to a first approximation N

L

:t 1.

(7.63)

The residual is 0 when N - 1. For N > 2, we obtain a coincidence test for image points: the hypothesis that image points {x~ } coincide is rejected with significance level a% if J[&] > X~(N-1),a"

(7.64)

The 2 ( N - 1) degrees of freedom can be intuitively interpreted as follows. An image point has two degrees of freedom, so N image points have 2N degrees of freedom for their deviations from the true position 3. But if we consider deviations from the average &, no deviation occurs when N - 1. Hence, the degrees of freedom decrease by two. 6As in C h a p t e r 5, we a s s u m e G a u s s i a n noise a n d do first o r d e r a n a l y s i s w h e n e v e r we refer to X2 d i s t r i b u t i o n s a n d X2 tests.

222

Chapter 7.

Parametric Fitting

P(l

Fig. 7.1. Optimally averaging image points by least squares. E x a m p l e 7.1 If each coordinate is perturbed independently by Gaussian noise of mean 0 a n d variance e2, the covariance matrix of each image point xo" is V[xo'] = e2Pk, so eq. (7.60) reduces to the usual least-squares optimization N

J014- ~

N

I1~ - ~11~ - Z D(PO"P)2 ~ min,

o'=l

(7.65)

o,=1

where D(po',p) is the distance from the a t h image point po" to the image point p to be fitted (Fig. 7.1). The optimal average given by eq. (7.61) reduces to the sample average N

1 5: -- ~ E

xo'.

(7.66)

v[~]- ~Pk-

(7.67)

o'--1

Its covariance matrix is E2

An unbiased estimator of the variance e2 is obtained in the form N

1 ~~iI 1 ~

~2_ 2 ( N - 1)

- ~11 ~.

(7.68)

If the value e is given a priori, the coincidence test takes the form ~2

X2(N_I), a

--~ e > 2(N-

7.2.2

1)"

(7.69)

Image line fitting

Given N image points {xo'}, a = 1, ..., N, consider the problem of fitting an image line (n, x) = 0 to them. We write xo" = ~o" + Axo',

(7.70)

and regard each Axo" as an independent random variable with mean 0 and covariance matrix V[xo']. The hypothesis is (n, ~ )

- 0,

~ - 1, ..., N,

(7.71)

7.2.

Optimal Fitting for Image Points

223

which has rank 1. An optimal estimate of n can be obtained by the optimization N

J[n] - ~

2"1 under the constraint

(I,*ll -

( n , xc~)2

V[x~]n)

--+ min

(7.72)

1. The covariance matrix of the solution ti is

V[fl]-(o~=~l(P~x'~ ~

)

(7.73)

where P n and ~ are the projection matrix along fi and the optimally corrected value of x~, respectively (see eq. (5.82)). The rank of V[fi] is 2; its null space is {fi} L. The residual J[fi] is a ;~2 variable with N - 2 degrees of freedom. Evidently, the number N of image points must be such that N _> 2.

(7.74)

The residual is 0 when N - 2. For N >_ 3, we obtain a collinearity test for image points: the hypothesis that image points {x~ } are collinear is rejected with significance level a~0 if

J[,i] > x 2

(7.75)

The N - 2 degrees of freedom can be intuitively interpreted as follows. A free image point has two degrees of freedom. An image point constrained to be on an image line keeps the incidence if it moves along that image line, so there remains one degree of freedom to break the incidence. Consequently, N image points have N degrees of freedom for their deviations from the true image line (~t, x) = 0. But if we consider their deviations from the fitted image line (fi, x) = 0, no deviations occur when N = 2. Hence, the degrees of freedom decrease by two. E x a m p l e 7.2 If each coordinate is perturbed independently by Gaussian noise of mean 0 and variance e 2, image point x~ has covariance matrix V[x~] = e2Pk , so eq. (7.72) reduces to the least-squares optimization

J~

E a=l

1-(k:n--) 2 = E

D(p~'I)2 --} min,

(7.76)

a=l

where D(p~,l) is the distance fl'om the a t h image point p~ to the image line 1 to be fitted (Fig. 7.2; see eq. (4.11)). The covariance matrix of the solution ti for image points { (x~, y~)} is given by

224

Chapter 7.

Parametric Fitting

P~ Fig. 7.2. Line fitting by least squares. ~2

V[ti]- l+d 2

/

E 4= N I X4 ^2

N 1 :r4Y4 E 4=

E N4=1 ~)4Xa Ec~=l S 024 4=1

E o N--r-f"

(7.80)

The solution of the least-squares optimization (7.76) can be obtained analytically. In fact, let x cos 0 + y sin 0 = d be the image line to be fitted. The function to minimize can be written in the following form: N

J(O,d) - Z ( x 4

cos 0 + y4 s i n 0 - d ) 2.

(7.81)

4=1

Differentiating this with respect to d and setting the result 0, we obtain d in the form d - 2cos0 + t3sin0, (7.82) where -

1

N

Z 4:1

xo,

-

1

N

Z ~:1

(7.83/

7.2.

Optimal Fitting for Image Points

225

Eq. (7.82) states that the image line should pass through the centroid (2, fy) of the data {(x~,y~)}. Substituting eq. (7.82) into eq. (7.81), we obtain a function of 0 to minimize in the form N

J(O)- Z

((x~ - 2) cos O+ (g~ - ~) sin0) 2 .

7( . 8 4 )

ct--1

If we put ii - (cos 0, sin 0)7-, this equation can be rewritten as

J [ n ] - (n, Mn), where M is the two-dimensional N

M ---

(7.85)

moment matrix

)2

E~----I(Xo~ N

-- 2

E~=I(~- 0)(x~- ~)

(7.86)

E N ( ~ - ~)~ c~=l

Since g is a unit vector, J[fi] is minimized by the unit eigenvector of the moment matrix M for the smallest eigenvalue (see eqs. (2.86)). The value of d is given by eq. (7.82). E x a m p l e 7.3 Suppose N is an odd number in Example 7.2. Let w be the distance between ( x l , g l ) and (xN,yN). Put

X(N+I)/2 ) Y(N+I)/2 ,

XC --

(7.87)

1

which represents the midpoint. Let u be the unit vector that indicates the orientation of the image line (/t, x) - 0. If {(x~,g~)}, a - 1, ..., N, are approximately equidistant, we have w

x~ ~ xc + N - 1 (a

N+I

-----~---)u.

(7.88)

Then, eq. (7.77) is approximated by

v[n] ~

e2

X(~+

d~

(

)

~~

+

w 2 ( N + 1)

~2(x-~)

u~

7-).

(7.89)

If the number N of the data points is very large, this expression gives the covariance matrix of the image line fitted to an edge segment having length w, orientation u, and midpoint x c (Fig. 7.3). Define the edge density (the number of edge pixels per unit length) p by N p - --. W

(7.90)

226

Chapter 7.

Parametric Fitting

^

g/

Fig. 7.3. Line fitting to an edge segment. In the limit N ~ cr we have the following asymptotic expression: V[~]

12e2 --

p 3(1 +

( UU

T

+

12

xcx T

)-

(7.91) "

The distance d of the fitted line from the image origin o is usually very small as compared to the focal length (which we take as the unit of length). If the edge segment is near the image origin o, we can assume that (Xc, u) 0 and IIxcII ~ 1. The length w of the edge segment is usually very small as compared with the focal length, so 1 2.

(7.98)

The residual is 0 when N - 2. For N > 3, we obtain a concurrency test for image lines: the hypothesis that image lines {(n~, x) - 0} are concurrent is rejected with significance level a ~ if J[5~]

> )~-2,a"

(7.99)

The N - 2 degrees of freedom can be intuitively interpreted as follows. A free image line has two degrees of freedom. An image line passing through an image point keeps the incidence if it changes its orientation around that image point, so there remains one degree of freedom to break the incidence. Consequently, N image lines have N degrees of freedom for their deviations from the true intersection 3. But if we consider their deviations from the estimated intersection &, no deviations occur when N - 2. Hence, the degrees of freedom decrease by two. E x a m p l e 7.4 Suppose each image line l~ is likely to be translated by noise into a position parallel to la, and suppose the distance of such a parallel translation is an independent random variable for each image line with mean 0 and standard deviation e, which is assumed to be very small. If image line l~ is represented by (n~, x) - 0, the covariance matrix of n~ for this statistical model is V[n,~] - e 2 ( 1 - (k, n~)2)(Pn k)(Pn k) T. (7.100) If image point x is very close to each image line (n~, x) - 0, we see that

(x,V[n~]x) - e 2 ( 1 - (k,n~)2)(x, P n . k ) 2 - e2(1 - ( k , n ~ ) 2 ) ( P n . x , k )

e2(l

-

(k, n,)2)(x,k)

2 -

e2(l - (k, n,)2).

2

(7.i01)

228

Chapter 7.

--

Parametric Fitting

II

Fig. 7.4. Intersection estimation by least squares. Hence, eq. (7.96) reduces to the following least-squares optimization (Fig. 7.4; see eq. (4.11))" J0[x] ~ E

1 -~,~-s

(7.102)

D(p'I~) 2 -+ min.

= E

c~=l

c~=1

The solution can be obtained analytically. In fact, let x cos0~ + y sin0~ d~ be the c~th image line. The function to minimize can be written in the following form: N

J ( x , y ) - E ( x c o s e ~ + y sin0~ - d ~ ) 2.

(7.103)

This is a quadratic polynomial in x and y, so the solution is given by solving the following linear equation: E c~---=1 cOS2 {9c~ N

~=lsin0~

cos0~

E ~=1 N

cos 0~ sin 0~ ~-~N c~--1 sin 2 0~

x g

N -"

d~ cos 0~ d~ sin0~

9

(7.104) However, the statistical model of noise given by eq. (7.100) is very artificial and unnatural (see Examples 4.2 and 7.3). Hence, the least-squares optimization (7.102) for intersection estimation is not appropriate even though the leastsquares optimization (7.77) is appropriate for line fitting.

7.3.2

Image line fitting

Given N image lines {(n~,m) = 0}, c~ = 1, ..., N, consider the problem of fitting an image line (n,m) - 0 to them. This means finding an optimal average n of {n~ }. We write n~ = ~

+ An~,

(7.105)

and regard each An~ as an independent random variable of mean 0 and covariance matrix V[n~]. The hypothesis is ft~ = n,

c~ = 1, ..., N,

(7.106)

7.3.

Optimal Fitting for hnage Lines

229

which has rank 2 because both sides are unit vectors 7. For each n~, the optimal estimate of its true value fi~ is evidently n, so the optimal average/t can be obtained by the optimization N

J[n] = E ( n ~ - n, V[n~l-(n~ - n)) --+ min

(7.107)

c~---1

under the constraint I1 11- 1. The covariance matrix V[na] has the null space {n~}L, so V[n~ln~ - O. Hence, eq. (7.107) can be rewritten as

J[n]-(n,(~V[na]-~=l

n) --+ min.

(7.108)

The solution ti is given by the unit eigenvector of the matrix N

M - E V[n~]-

(7.109)

c~=l

for the smallest eigenvalue (see eqs. (2.86)). The covariance matrix of the solution ~i is

V[Ti] - ( ~ Piz

,

(7.110)

which has rank 2; its null space is {/Z}L. The residual J[,i I is a )(2 variable with 2 ( N - 1) degrees of freedom. Evidently, the number N of image lines must be such that N >_ 1.

(7.111)

The residual is 0 when N - 1. For N > 2, we obtain a coincidence test for image lines: the hypothesis that image lines {(n~, x) - 0} coincide is rejected with significance level a% if J[~] > X2(N_I),a .

(7.112)

The 2 ( N - 1) degrees of freedom can be intuitively interpreted as follows. An image line has two degrees of freedom, so N image lines have 2N degrees of freedom for their deviations from the true position (fi, x) - 0. But if we consider deviations from the average (ti, x) - 0, no deviation occurs when N - 1. Hence, the degrees of freedom decrease by two. 7This hypothesis is degenerate.

230 7.4

7.~. 1

Chapter 7. Optimal

Fitting

for Space

Parametric Fitting

Points

Space point fitting

Given N space points {r~}, a - 1, ..., N, consider the problem of fitting a space point r to them. This means finding an optimal average r of {r~ }. We write r~ - +~ + Ar~ (7.113) and regard each Ar~ as an independent random variable of mean 0 and covariance matrix V[r~]. The hypothesis is +~ - r,

a - 1, ..., N,

which has rank 3. For each r~, the optimal estimate of its true value ~ the optimal average can be obtained by the optimization

(7.114)

is evidently r, so

N

J[r] - ~-~(r~ - r, V[/'~]-I(/'. - / ' ) ) ---+min.

(7.115)

~=1

The solution is given by -1

N

V[rol-lro.

+:

(7.116)

o~--1

Its covariance matrix is -1

vi+l ( o=1 v ro, 1)

(7.1 7)

which has rank 3. The residual J[~] is a X2 variable with 3 ( N - 1) degrees of freedom. Evidently, the number N of space points must be such that N _> 1.

(7.118)

The residual is 0 when N - 1. For N >__ 2, we obtain a coincidence test for space points: the hypothesis that space points {ra} coincide is rejected with significance level a% if 2 J[r] > ~ 3 ( N - 1 ) , a " (7.119) The 3 ( N - 1) degrees of freedom can be intuitively interpreted as follows. A space point has three degrees of freedom, so N space points have 3N degrees of freedom for their deviations from the true position e. But if we consider deviations from the average ~, no deviation occurs when N - 1. Hence, the degrees of freedom decrease by two.

7.4.

Optimal Fitting for Space Points

231

P~

Fig. 7.5. Optimally averaging space points by least squares. E x a m p l e 7.5 If each coordinate is perturbed independently by Gaussian noise of mean 0 and variance e 2, the covariance matrix of each space point r4 is V[r~] = e2I, so eq. (7.115) reduces to the least-squares optimization N

J0[r] = E

N

D(P4, p)2

Ilr4 - vii2 - E

4"-1

__+ min,

(7.120)

4:1

where D(P4, P) is the distance from the c~th space point P~ to the space point P to be fitted (Fig. 7.5). The optimal average given by eq. (7.116) reduces to N

1

- ~ E

r4.

(7.121)

4--1

Its covariance matrix is E2

v i i i - ~x.

(7.~22)

An unbiased estimator of the variance e2 is obtained in the form ~2_ 1 - 3(N- 1)E

N

I 1 ~ - ell ~.

(7.123)

4-'1

If the value e is given a priori, the coincidence test takes the form ~2

2

)~3(N-1),a

J > 3(X- 1)"

7.~.2

(7.124)

Space line fitting

Given N space points {r~}, a - 1, ..., N, consider the problem of fitting a space line r x p - n to them. We write r 4 - ~4 + A t 4 ,

(7.125)

and regard each A t 4 as an independent random variable with mean 0 and covariance matrix V[r4]. The hypothesis is 04 x p -

n,

c ~ - 1, ..., N,

(7.126)

232

Chapter 7.

Parametric Fitting

which has rank 2 because both sides are orthogonal to p. If we let

e (1) - -

(1) 0

,

e (2) - -

(o)(o) 1

0

e (3)

,

--

0

0 1

,

(7.127)

and

r~ ) - r~ x e (k),

k - 1, 2, 3,

(7.128)

the hypothesis (7.126) can be equivalently written in the form

(~(2),v) + (~(~),n) - o,

k = 1, 2, 3.

(7.129)

If we define 6-vectors a~ )

- - r ( ~ k) |

e (k),

u

--

p | n,

(7.130)

eq. (7.129) can be further rewritten in the form (~,~), u)

--

0,

a

--

1, ..., N,

(7.131)

where the bar refers to the true value. Eq. (7.131) has the same form as eq. (7.46), so the result in Section 7.1.5 can be applied. Since e (k) does not incur noise, the covariance matrix Y[a(~k), a~ )] has the form

(')]- v[,~) ,~)] 9 o.

(7.132)

The covariance matrix of r(~k) is given as follows (see eq. (2.43)):

v[,~),,~)]-

~(~) • v[,o] • ~(').

(7.133)

From eqs. (7.128)and (7.130), we see that (a~),

u)

-

-(v~

• p -

(7.x34)

n, e(k)).

Hence, eq. (7.55) reduces to the optimization N

J[p| n ] - E ( r ~

x p - n, W~(p)(r~ x p - n)) -+ min

(7.135)

c~=l

under the constraints (p, n) - 0 and Ilpll2 -4-Ilnll 2 W a ( p ) is given by

W~(p)-

(p • V[t~] x p ) - .

1. The (33)-matrix (7.136)

7.4.

Optimal Fitting for Space Points

233

The covariance matrix of the solution i5 @/~ is given by eq. (7.57), which reduces to V[15 |

- (P~c~e~

E~=I

E~=,~xW~(#)x§

~2 N

~=1

W (#)x~

~2 N

(~=1 W c ~ ( # )

c~

Par~e~

,

(7.137)

where ~ is the optimally corrected value of r~ (see eq. (5.129)). Here, PAGe~ is the six-dimensional projection matrix onto Afpen• (see eqs. (4.43) and (4.44)). The rank of the covariance matrix V[lb |

is 4; its null space is

.N'lbr ~ 9 The residual J[lb | 5] is a X2 variable with 2 ( N - 2)degrees of freedom. Evidently, the number N of space points must be such that N _> 2.

(7.138)

The residual is 0 when N - 2. For N >__ 3, we obtain a collinearity test for space points: the hypothesis that space points {r a} are collinear is rejected with significance level a% if

(7.139)

2

The 2 ( N - 2) degrees of freedom can be intuitively interpreted as follows. A free space point has three degrees of freedom. A space point constrained to be on a space line keeps the incidence if it moves along that space line, so there remain two degrees of freedom to break the incidence. Consequently, N space points have 2 N degrees of freedom for their deviations from the true image line r x p - ft. But if we consider their deviations from the fitted space line r x lb = fi, no deviations occur when N - 2. Hence, the degrees of freedom decrease by four. E x a m p l e 7.6 If each coordinate is perturbed independently by Gaussian noise of mean 0 and variance e2, space point r a has covariance matrix Viva] = e2I, so eq. (7.136) can be rewritten as follows (see eq. (2.42))"

- -

ix

1 (,1 11 21

ppT)

-

, ( =

211pll2

(7.140) I

ilpl12

.

Then, eq. (7.135) reduces to the least-squares optimization

N

Jo[p 9 ~]- ~ I1~ x p~ a=l IlPll

.112

N -- E D(Pa, L)2 --+ min, ~=1

(7.141)

234

Chapter 7.

Parametric Fitting

P~j

~

D(P~, L)

Fig. 7.6. Space line fitting by least squares. where D(P,~, L) is the distance from the a t h space point to the space line L to be fitted (Fig. 7.6; see eq. (4.49)). Eq. (7.1371 reduces to

v[/,e

,~] - ~11/,11~

(P~.

~=1~ N

to\

x I x ~

NI

E~=I(~O • I/T

P,v'~,r

1

)

(7.1421

where ~a is the orthogonal projection of r~ onto the fitted space line (see eq. (5.13111. An unbiased estimator of e2 is obtained in the form

~

-

-

2(N-

~

2)

~~1 IIr~ xi l ~2,,~11~ . = [ ~

(7.143)

If the value e is given a priori, the collinearity test takes the form ~2

X2(N_2),a

e-~ > 2 ( N - 2)"

(7.144/

The solution of the least-squares optimization (7.1411 can be obtained analyticaily. In fact, let ( r - rH) • m -- 0 be the {m, rH}-representation of the space line to be fitted. The function to minimize can be written in the following form (see eq. (4.49))" N

J[m, rH] - E [ I P m r ~ -- Villi2"

(7.1451

o~"-1

Differentiating this with respect to/'H and setting the result 0, we obtain rH in the form

rH -- Pm~',

1

N

r - ~ E

r~.

(7.1461

c~--I

This means that the space line should passes through the centroid f" of the data {ra}. If we note the identity IIPmal[ 2 -Ilall 2 - ( m , a) 2 for an arbitrary

7.4.

Optimal Fitting for Space Points

235

vector a (see Fig. 2.2), eq. (7.145) reduces to a function of m in the form N

N

J[m] - E IlPm(r~ - 0)112 - E c~:l

N

IIr~ - ~ll 2 - ~ ( m ,

c~:l

r ~ - ~)2

(~=1

N

= Z

lira - rll2 - ( m , M m ) ,

(7.147)

o~--1

where M is the moment matrix N

M - Z

(r~ - e ) ( r ~ - ~ ) r .

(7.148)

The function g[m] is minimized if (m, M m ) is maximized. Since m is a unit vector, the solution is obtained as the unit eigenvector of the moment matrix M for the largest eigenvalue (see eqs. (2.86)). The vector rH is given by eq. (7.146).

7.~.3

Space plane fitting

Given N space points {p~}, c~ - 1, ..., N, consider the problem of fitting a space plane (v, p) - 0 to them. We write p~ - ,o~ + Ap,~,

(7.149)

and regard each Ap~ as an independent random variable with mean 0 and covariance matrix V[p~]. The hypothesis is (,, ~)

- 0,

~ = 1, ..., N ,

(7.150)

which has rank 1. An optimal estimate i can be obtained by the optimization J[v] - E

(v, V[p~lv) -+ min

(7.151)

o~'--1

under the constraint []vii - 1. The covariance matrix of the solution/, is

v[~]-(o:~(p~#~)(p~#~)~i~:y~]~)

(7.~52)

w h e r e / ~ is the optimally corrected value of p~ (see eq. (5.160)). The rank of V[/,] is 3; its null space is {/'}L. The residual J[/,] is a )C2 variable with N - 3 degrees of freedom. Evidently, the number N of space points nmst be such that N > 3.

(7.153)

236

Chapter 7.

Parametric Fitting

P~

Fig.

7.7. Space plane fitting by least squares.

The residual is 0 when N - 3. For N _> 4, we obtain a coplanarity test for space points: the hypothesis that space points {r~} are coplanar is rejected with significance level a% if

J[/)] > jX2_3,a .

(7.154)

The N - 3 degrees of freedom can be intuitively interpreted as follows. A free space point has three degrees of freedom. A space point constrained to be on a space plane keeps the incidence if it moves within that space plane, so there remains one degree of freedom to break the incidence. Consequently, N space points have N degrees of freedom for their deviations form the true space plane (p, p) = 0. But if we consider their deviations from the fitted space plane ( i , p) = 0, no deviations occur when N = 3. Hence, the degrees of freedom decrease by three. E x a m p l e 7.7 If each coordinate is perturbed independently by Gaussian noise of mean 0 and variance e2, space point p~ has covariance matrix V[p~] = e2I | - e2(I - ~ - r ) , where ~ - (0,0,0, 1) T. In this case, eq. (7.151) reduces to the least-squares optimization

N N (u,p~)2 _ E J0[v] - E 1 - i~;i ~,]2 ~=i

D ( P ~ ' H ) 2 ~ min,

(7.155)

cr--I

where D(P~, I I) is the distance from the a t h space point to the space line H to be fitted (Fig. 7.7; see eq. (4.68)). The covariance matrix of the solution/, for space points { (X~, Y~, Z~) } is given in the form

ViOl

-- 1+c~ 2

/

o~=1 Xo~ E E N f~2~ a=l E N Xcr

E a = l "Yc~"c~ Eo~=l N y2 EO~-'I E a =Nl ~r Ec~LI Ya N

E c~=l 2o N L EO~=I

/

(7. s6) where d is the distance of the space plane (~,, p) - 0 from the origin O (see eqs. (4.63)). Here, (X~,lY~,Z~)is the orthogonal projection of ( X ~ , Y ~ , Z ~ )

7.4.

237

Optimal Fitting for Space Points

onto the fitted space plane (see eq. (5.163)). If we write ~, - (A,/3, ~, D)T, the residual of the minimization (7.155) can be written as N

J o i n ] - (1 + d 2) E

(_/iX~ +/~Y~ + C;Z~ + / ) ) 2 .

(7.157)

o~1

Hence, an unbiased estimator of the variance e2 is obtained in the form ~2 __ 1 - [ - ( t 2

- x-

N

3 ~

(/i_u176 + BY. + d z ~ + D) ~.

(7.~5s)

If the value e is given a priori, the coplanarity test takes the form ~2

X2_3,a

~-Y > N---:~

(7.159)

The solution of the least-squares optimization (7.155) can be obtained analytically. In fact, let (n, v) - d be the {n, d}-representation of the space plane to be fitted. The function function to minimize can be written in the following form (see eq. (4.68))" N ct--- 1

Differentiating this with respect to d and setting the result 0, we obtain d in the form N 1

- (~, ~1,

~ = ~ Z ~*"

(7.161/

o~'-1

This means that the space plane should pass through the centroid f" of the data {r~}. Substituting eq. (7.161)into eq. (7'.160), we obtain a function of n to minimize in the form N

(n, v~ - ~)2 _ (n, M n ) ,

J[n] - E

(7.162)

where M is the m o m e n t matrix N

M - E (r~ - e)(r~ - e)7-.

(7.163)

o~--'1

Since n is a unit vector, J[n] is minimized by the unit eigenvector of the moment matrix M for the smallest eigenvalue (see eqs. (2.86)). The value of d is given by eq. (7.161).

238

7.5

7.5.1

Chapter 7.

Parametric Fitting

Optimal Fitting for Space Lines Space point fitting

Given N space lines {r x p~ - n~}, a - 1, ..., N, consider the problem of fitting a space point r to them. This means optimally estimating their common intersection. We write p~ = 1~ + Ap~,

n~ - ~,~ + A n ~ ,

(7.164)

and regard A p a and A n a as random variables of mean 0 and covariance matrix V[p~ | n~], independent for each a. The hypothesis 8 is v x p~ - f i ~ ,

a-

1,...,N,

(7.165)

which has rank 2 because both sides are orthogonal to p~. Let p - (X, Y, Z, 1) T, and define e (1) , e (2) , and e (3) by eqs. (7.127). If we define 4-vector (7.166) a(~(k) _ (e(k) • p~) | (e(k) n,~), the hypothesis (7.165) can be rewritten in the form (a(~k) , p) - O,

k - 1, 2, 3,

(7.167)

where the bar refers to the true value. Since e (k) does not incur noise, the covariance matrix V[a (k), a~ )] is given as follows (see eqs. (2.39) and (2.44)):

e(k) • V[po~] • e(l) =

• v[v

l

(

e (k) x V[p~]e(O ) (e (k), V[p~]e(O) "

)lT

(7.168)

From eq. (7.166), we see that (a(2),

p)

-

•

-

(7.169)

Hence, the optimization (7.55) can be rewritten in the form N

J[r]-

E(r

• p~-

n~, W ~ ( r ) ( r • p ~ -

n~))~

min,

(7.170)

o~=1

where the (33)-matrix W ~ ( r ) i s W~(r)-

(r • 2 1 5 2 1 5 2 1 5

given by

I)- . 2

(7.171) 8This hypothesis is degenerate.

7.5.

Optimal Fitting for Space Lines

239

The covariance matrix of the solution ~ is given by eq. (7.57), which reduces to --1

w h e r e / ~ is the optimally corrected value of p~ (see eqs. (5.137)). The residual J[/'] is a ~2 variable with 2 N - 3 degrees of freedom. Evidently, the number N of space lines must be such that N _> 2.

(7.173)

However, the residual is not 0 when N - 2. For N > 3, we obtain a concurrency test for space lines" the hypothesis that space lines {v x p~ - n~} are concurrent is rejected with significance level a% if 2 J[r] > X2N-3,a"

(7.174)

The 2 N - 3 degrees of freedom can be intuitively interpreted as follows. A free space line has four degrees of freedom. If it is constrained to pass through a space point, it can still change its orientation freely around that space point, so there remain two degrees of freedom to break the incidence. Consequently, N space lines have 2 N degrees of freedom for their deviations from the true intersection ~. But we are considering their deviations form the estimated intersection ~. For N - 2, there remains only one degree of freedom to break the concurrency: the freedom of translation orthogonal to the two space lines. Hence, the total degrees of freedom are 2 ( N - 2) + 1 = 2 N - 3.

7.5.2

Space line fitting

Given N space lines {v x p~ - n~}, a - 1, ..., N, consider the problem of fitting a space line r x p - n to them. This means finding optimal averages p and n of {p~} and {n~}, respectively. We write p~ - 1~ + Ap~,

n~ -- gt~ + A n ~ ,

(7.175)

and regard Ap~ and A n ~ as random variables of mean 0 and covariance matrix V[p~ | n~], independent for each a. The hypothesis is p~ - p,

fi~ - n,

a - 1, ..., N,

(7.176)

which has rank 4 because {p, n} have four degrees of freedom 9. For each a, the optimal estimates of the true values i~a and fia are evidently p and n, so the optimal averages can be obtained by the optimization N

g[p, n] - ~ - ~ ( p , | 1 7 4 cr=l 9This hypothesis is degenerate.

V[p,|174174

-+ min (7.177)

240

Chapter 7.

Parametric Fitting

under the constraints (p, n) - 0 and Ilpll + Ilnll 1. Since the covariance matrix V[p, | n~] has null space Afp.en. - {p~ | n,, n , 9 P~}L C Tt 6 (see eq. (4.43)), we have V[p0, | n~](p~ | no,) - O. Hence, the optimization (7.177) can be rewritten in the form N

J[p,n] - E ( p |

|174

~ rain.

(7.178)

or--1

The solution l~ @ h is given by the unit eigenvector of the matrix N

M - E V[p~ | n,~]-

(7.179)

o'--1

for the smallest eigenvalue (see eqs. (2.86)). The covariance matrix of of the solution io | h is

V[P|

)iPa6*~V[p~| c ~ -

(7.180)

i

where Par~ea is the six-dimensional projection matrix onto

• arp.,

(see

eq. (4.4,4)). The covariance matrix V[p | ti] has rank 4; its null space is .A/'pe~ . The residual N

J[fg, ~i] - E (P~ | n~ - p | ~i, V[p~ | n~]- (p, | n~ - p | ~i)) (7.181) 0,--1

is a X2 variable with 4 ( N - 1) degrees of freedom. Evidently, the number N of space lines must be such that N _> 1.

(7.182)

The residual is 0 when N - 1. For N > 2, we obtain a coincidence test for space lines: the hypothesis that space lines {r x p~ = n~ } coincide is rejected with significance level a% if J[lb, hi > X~(N-1),~-

(7.183)

The 4 ( N - 1) degrees of freedom can be intuitively interpreted as follows. A free space line has four degrees of freedom, so N space lines have 4 N degrees of freedom for their deviations from the true position r x p - ~. But if we consider deviations from the average r x la - 1i, no deviation occurs when N = 1. Hence, the degrees of freedom decrease by four.

7.5.

Optimal Fitting for Space Lines

7. 5.3

241

Space plane fitting

Given N space lines { ( r - - r H ) x m ~ = 0}, a = 1, ..., N, consider the problem of fitting a space plane (v, p) = 0 to them. We write m ~ = rh~ + A m ~ ,

I'H~ -- ~ H ~ -!-

Atria,

(7.184)

and regard A r e a and ArHa as random variables of mean 0 and covariance matrix V[m~ @rS~], independent for each a. The hypothesis can be written ~ follows ( ~ ~q. (4.70)): (v, rh~ | 0) = 0,

(v, ~ , ~ | 1) = 0,

a = 1, ..., N.

(7.185)

The rank of this hypothesis is 2. If we define 4-vectors

the hypothesis (7.187) can be rewritten in the form ( a ~ ) , p) - 0,

k - 1, 2,

(7.187)

where the bar refers to the true value. The covariance matrices k, 1 = 1, 2, are given as follows:

Via(i) V[a~

(1)] _ U[frtc~] 9 0. W [ a ~ ) , a (')] - V[m~, the] T @ 0,

) a(~(~)l

V[a~ )

a~ )]

- v [ m , , ~ .c~ ] 9 0

V [ a ~ ) , a ~ )] --

V[rH~] |

0.

(7.188)

Hence, eq (7.55) reduces to the optimization

N 1 J[l~,d]- 1 --~d2 E

ll)(n'd)(~'t"~Tl'c~)2

ct-1

-+-2W(12)(n, d)(n, ~r~o~)((n,rHa ) - d) + W(~22)(n,d)((n, rH~)-d) 2) --~ rain i

W~(n,d)is given (n,V[m~,rH~]n))-1

under the constraint I1~11 = 1, where the (22)-matrix

1 W ~ ( ~ , d ) - 1 + d~

((n,V[m~]n) (~, V [ ~ , o , m ~ l n )

(7.189)

(,,, V [ ~ , ~ ] n )

by

. (7.190)

The covariance matrix of the solution i is given by eq. (7.57), which reduces to V[/,]- (Po

(Ec~% 1 ( w(ll)(~,d) ~^~ o^ T +2w~(~) (n ~)s[,~,~o] +w.(~) EN: 1 (W(12) (~'~'d)'r~ o~+ ~V(22)(~r 0~)'rHo~)T

(~,^^~)~.o~.o^~)

E N (W(12' (.. (~)m -~-./17(22)(?zt.j) ~ Hc~) ) )c~:l (~ ' ' ~ PO , E c~--1 ~ w($~)(n,d)

(7.191)

242

Chapter 7.

Parametric Fitting

where vh~ and /~H~ are the optimally corrected values of m a and rH(~, respectively (see eq. (5.183)). The symbol S[. ] denotes the symmetrization operator (see eqs. (2.205)). The rank of V[~,] is 3; its null space is {U}L. The residual J[fi, d] is a X2 variable with 2 N - 3 degrees of freedom. Evidently, the number N of space lines must be such that N _> 2.

(7.192)

However, the residual is not 0 when N - 2. For N > 3, we obtain a coplanarity test for space lines: the hypothesis that space lines { ( r - rHa) X m ~ -- 0} are coplanar is rejected with significance level a% if J [ n , d] > X~g-3,a.

(7.193)

The 2 N - 3 degrees of freedom can be intuitively interpreted as follows. A free space line has four degrees of freedom. If it is constrained to be on a space plane, it can still translate and rotate freely within that space plane, so there remain two degrees of freedom to break the incidence. Consequently, N space lines have 2N degrees of freedom for their deviations from the true plane (P, p) = 0. But we are considering their deviations from the fitted plane (~, p) - 0. For N - 2, there remains only one degree of freedom to break the coplanarity: the freedom of translation orthogonal to the two space lines. Hence, the total degrees of freedom are 2 ( N - 2) + 1 - 2 N - 3.

7.6

7.6.1

Optimal

Fitting

for Space

Planes

Space point fitting

Given N space planes { ( v a , p ) - 0}, a - 1, ..., N, consider the problem of fitting a space point p to them. This means optimally estimating their common intersection point. We write =

+

(7.194)

and regard each A v a as an independent random variable of mean 0 and covariance matrix Viral. The hypothesis can be written as follows: (p~, p) - 0,

a - 1, ..., N.

(7.195)

The rank of this hypothesis is 1. An optimal e s t i m a t e / , can be obtained by the optimization

J[P]- E (p~(~-~p) --+min

(7.196)

7.6.

243

O p t i m a l Fitting for Space Planes

under the constraint (to, p) - 1, where tr - (0, 0, 0, 1) 7-. The covariance m a t r i x of the solution/5 is

V[~]_(L(Ptci'~)(Ptci', r) ~) a=i

(p, U[vc~]h)

(7.107)

where Oa is the optimally corrected value of v~ (see eq. (5.169)). The rank of V[~] is 3; its null space is Af~ - {~}L. The residual J[~] is a X2 variable with N - 3 degrees of freedom. Evidently, the n u m b e r N of space lines must be such that N _> 3.

(7.198)

The residual is 0 when N - 3. For N > 4, we can test if space planes have a c o m m o n intersection: the hypothesis t h a t space planes {(v~, p) - 0} have a common intersection point is rejected with significance level a% if J[~b] > X~v-3,a"

(7.199)

The N - 3 degrees of freedom can be intuitively interpreted as follows. A free space plane has three degrees of freedom. If it is constrained to pass t h r o u g h a space point, it can still change its orientation freely around that space point, so there remains one degree of freedom to break the incidence. Consequently, N space lines have N degrees of freedom for their deviations from the true space point lb. But if we consider their deviations from the estimated intersection point t), no deviations occur when N - 3. Hence, the degrees of freedom decrease by three.

7. 6.2

Space line fitting

Given N space planes { ( v ~ , p ) = 0}, a = 1, ..., N, consider the problem of fitting a space line ( r - r H ) X m -- 0 to them. This means optimally estimating their c o m m o n intersection line. We write v~ - O~ + A v ~ ,

(7.200)

and regard each A v ~ as an independent r a n d o m variable of mean 0 and covariance m a t r i x V[v~]. The hypothesis can be written as follows (see eq. (4.70))" (v~, m | 0) - 0,

(v~, rH | 1) -- 0,

a -- 1, ..., N.

(7.201)

The rank of this hypothesis is 2. If we define 8-vectors

u--mOOOrHO1,

a (1) - v~ | O,

-

o

(7.202)

244

Chapter 7.

Parametric Fitting

the hypothesis (7.201) can be rewritten in the form (a~), u) - O,

k - 1, 2,

(7.203)

where the bar refers to the true value. The covariance matrices V[a~ ) a~ )] k, l - 1, 2, are given as follows:

V[a(1), ao~ (1)] __

v[~),

(1)

ao I -

( v[..] 0 (

O O

O

'

O )

V[~o]

O

V[a)(i a .(2)] _

( o V[~o] )

'

, V[a~),~

O

O

(2)]_ ( O O

O

V[~]

)(7.204)

From eqs. (7.202), eq. (7.55) reduces to the optimization

N Jim, rill-- ~

l + d1~

(W (11) (m, ~'H)(~'~c~m) 2

+ 2 W (12) (m, rH)(no~, m ) ( ( n ~ , rH) -- da)

-~- W~(22)(m, rH)((no~, under the constraints Ilmll W~,(m, rH) is given by

rH) --

da) 2) -+ min

(7.205)

1 and (m, rH) = 0, where the (22)-matrix

(m 9 0, V[vc~](~ H O 1)) ~ --1 (rH 9 1, V[v~](rH 9 1)) / " (7.206) The covariance matrix of the solution/t is given by eq. (7.57), which reduces to

W,~(m, rH) --

( m | 0, V[v~](m | 0)) (rH 9 1, V[v~](m 9 0))

V[/~]- (PNa

E Nc~=lw(ll)(m, ~'H)~Vc~ ^T EN 1W~12)(m, ~'H)0c~Vc~

E ~ w~(l~)(,~,e.)~ ^~

~= ,~:(~)

9

-

P~) ,

(7.207)

where f,~ is the optimally corrected value of v~ (see eq. (5.186)). Here, PX~ is the eight-dimensional projection matrix onto the orthogonal complement of

Af

~u-~

f

0 o

0

'

~'H

0

0

1

m

0

'

o

0

0 0 '

o

} ~,

(7.208)

1

which is the four-dimensional null space of/t; the covariance matrix V[/t] has rank 4.

7.6.

Optimal Fitting for Space Planes

245

The residual J[rh, rill is a )r variable with 2 ( N - 2) degrees of freedom. Evidently, the number N of space planes must be such that N _~ 2.

(7.209)

The residual is 0 when N - 2. For N _ 3, we can test if space planes have a common intersection line: the hypothesis that space planes {(n~, r) - da} have a common intersection line is rejected with significance level a% if J[rh, rH] > ~2(N-2),a.

(7.210)

The 2 ( N - 1) degrees of freedom can be intuitively interpreted as follows. A free space plane has three degrees of freedom. If it is constrained to pass through a space line, it can still rotate freely around that space line, so there remain two degrees of freedom to break the incidence. Consequently, N space planes have 2 N degrees of freedom for their deviations from the true intersection line ( r - r H) • rh -- 0. But if we consider their deviations from the estimated intersection line (r - / ' H ) • rh - 0, no deviations occur when N = 2. Hence, the degrees of freedom decrease by two.

7. 6.3

Space plane fitting

Given N space planes { ( v ~ , p ) - 0}, a - 1, ..., N, consider the problem of fitting a space plane (v, p) - 0 to them. This means finding an optimal average v of {v~}. We write v~ - O~ + Av~,

(7.211)

and regard each A v a as an independent random variable of mean 0 and covariance matrix V[va]. The hypothesis is O~ - v,

a-

1, ..., N,

(7.212)

which has rank 3 because both sides are unit 4-vectors 1~ For each v~, the optimal estimate of its true value O~ is evidently v, so the optimal average can be obtained by the optimization N

J[u] - E ( u ~ - u, V [ u ~ ] - ( u ~

- v)) --4 man

(7.213)

o~--1

under the constraint I1 11- 1, Sinc the cov~ri~nc~ matrix V[v~] has null space Af~ - {Us}L, we have V[u~]u~ - O. Hence, the optimization (7.213) can be rewritten in the form

J[u] - (u' ( L 1~

hypothesis is degenerate.

V[u~]-) u)) -+

(7.214)

246

Chapter 7.

Parametric Fitting

The solution ~ is given by the unit eigenvector of the matrix N

M - Z

Viva]-

(7.215)

for the smallest eigenvalue (see eqs. (2.86)). The covariance matrix of of the solution ~ is given by

(7.216) which has rank 3; its null space is { h a } L . The residual J[/,] is a X2 variable with 3 ( g - 1) degrees of freedom. Evidently, the number N of space planes must be such that N > 1.

(7.217)

The residual is 0 when N - 1. For N > 2, we obtain a coincidence test for space planes: the hypothesis that space planes {(va, p) - 0} coincide is rejected with significance level a% if

>

(7.218)

The 3 ( N - 1) degrees of freedom can be intuitively interpreted as follows. A space plane has three degrees of freedom, so N space planes have 3N degrees of freedom for their deviations from the true position (P, p) - 0. But if we consider deviations from the average (~, p) - 0, no deviation occurs when N = 1. Hence, the degrees of freedom decrease by three.

Chapter 8 Optimal Filter In the preceding chapter, the statistical characteristics of the solution of the parametric fitting problem were examined by assuming that the solution was computed by some numerical method. This and the next chapters present such numerical methods. In this chapter, we construct a filter that starts from an initial estimate and optimally updates it each time a new datum is read; we call it simply the optimal filter. The update rule is derived from the Bayesian standpoint: we apply the principle of maximum a posteriori probability estimation by assuming that the noise is Gaussian. The assumptions and approximations introduced in the derivation are elucidated, and the philosophical implications of the Bayesian approach are discussed. Then, the update rule is simplified by introducing the effective gradient approximation. The resulting expression is compared with the Kalman filter with no internal dynamics. Finally, the update rule for linear hypotheses is derived.

8.1

8.1.1

General Theory

Bayesian approach

In the preceding chapter, it was implicitly assumed t h a t the entire set of d a t a { a s } , a = 1, ..., N , was available for computing the optimal estimate of p a r a m e t e r u. Suppose the d a t a are a time sequence, the a t h value a s read at time a - 1, 2, 3, .... If the sequence is very long, it is realistic to c o m p u t e an estimate u s of u at time a in such a way t h a t it is optimal for the d a t a a l , ..., a s and u p d a t e it to Us+l at time a + 1 so t h a t it is optimal for a l, ..., a s + l , and so on (Fig. 8.1). If the u p d a t e c o m p u t a t i o n does not require rereading the past data, the estimation process can be efficiently run in real time. In this chapter, we derive a theoretically optimal u p d a t e rule for Gaussian noise. T h e resulting filter not only suits real time d a t a processing but also serves as a numerical algorithm for the parametric fitting problem even when the entire set of d a t a is simultaneously available. If we try to formulate such an u p d a t e rule, however, we immediately encounter a subtle problem, which, although purely philosophical, has beset m a n y theoreticians. A natural form of u p d a t e is c o m p u t i n g the a t h estimate u s and its covariance matrix V[us] at time a and correcting t h e m into Us+l and V[Us+l] by using the (ct + 1)st d a t u m a s + l . As the u p d a t e proceeds, the estimate u s is expected to approach the true value, and its covariance matrix V[u~] is expected to decrease. In this process, we are viewing the estimate u s as a random variable. 247

248

Chapter 8.

Optimal Filter

In the preceding chapter, we derived the covariance matrix V[/t] of the optimal e s t i m a t e / t . The optimal estimate /t is a random variable because it is computed from the data {a~}. The statistical characteristics of the data {aa} reflect the accuracy and reliability of the data acquisition process (e.g., image processing and 3-D sensing), and the noise distribution can be estimated, at least in principle, by repeating the data acquisition process many times. Thus, the statistical characteristics of the optimal e s t i m a t e / t are a mere (and faithful) reflection of the uncertainty of the data acquisition process. On the other hand, starting such an update procedure requires an initial value u0 and its covariance matrix V[u0]. The question is: is Uo a random variable? Since no data exist yet, it has nothing to do with the uncertainty of the data acquisition process. If it is a random variable, how can its distribution be defined? This observation necessitates that we think of u0 as our subjective belief, and its distribution as the relative strength of our belief in particular values. Once this subjective interpretation is accepted, logical consistency dictates that we view the subsequent update process in the same way: each time a new datum is read, the belief is influenced and its subjective distribution is modified. The approach that admits such a subjective interpretation is known as Bayesian 1, whereas that which rejects it is called non-Bayesian. The distinction between the Bayesian and the non-Bayesian approaches is purely philosophical and often not so clear in reality. For example, one can compute the initial estimate u0 and its covariance matrix V[u0] from a small number of data, say a l , a2, and a3, and then start the update process by regarding the subsequent data a4, as, ... as a new sequence a~, a~, .... In this case, the statistical characteristics of u0 reflect the characteristics of the errors in a l, a2, and a3. Hence, the approach is non-Bayesian. On the other hand, if one guesses Uo and V[u0] and applies the same procedure thereafter, the approach is Bayesian. In this chapter, we adopt the Bayesian approach for the sake of formal consistency, but we do not worry about its philosophical foundation any further. The distribution of an estimate before data are observed is called the a priori distribution (or simply the prior), while the distribution updated by the data is called the a posteriori distribution 2 (or simply the posterior).

8.1.2

Maximum a posteriori probability estimation

Given an m-vector datum a, we write a=~+Aa,

(8.1)

1This terminology derives from the Bayes formula (see eq. (3.95)), which plays an essential role in this approach. 2In the preceding chapters, the terms a priori and a posteriori were used in the nonBayesian context.

8.1.

General Theory

249

a~

a 2

//0 I~ V[uol

a 3

UI_~U2 VIu,l

~

VIu~]

U3

a4

V[u3]

Fig. 8.1. The c~th estimate h~ and its covariance matrix V [ ~ ] are updated by the (c~ + 1)th d a t u m a~+l.

and regard A a as a Gaussian random variable of mean 0 and covariance matrix Via]. We assume that a and g are constrained to be in an m~-dimensional manifold .A C 7~m, which we call the data space. It follows that the noise A a is constrained, to a first approximation, to be in the tangent space Ta(A) to the manifold ,4 at g. We also assume that no constraint is imposed on A a other than A a E To,(JI), so the range of the covariance matrix IF[a] coincides with the tangent space Ta(A). Our task is to estimate the n-vector u that satisfies the hypothesis F(k) (a, u) -- 0,

k - 1, ..., L,

(8.2)

given by L smooth functions F(k) ( ", . ): 7~m x 74n ~ 74. We assume that the domain of the parameter u is an n~-dimensional manifold/4 C T/n, which we call the parameter space. As argued in Section 7.1.2, this problem is solved in two stages: the correction stage and the estimation stage. The difference from Section 7.1.2 is that the parameter u is now a random variable that has an a priori probability density. The formulation in Section 7.1.2 is modified as follows.

A. Correction stage We estimate the value ~ - a - A a that satisfies the hypothesis (8.2) for a particular value of u. As shown in Section 5.1.1, the optimal correction A a is determined by the optimization J-

(s.3)

(Aa, l ) [ a ] - A a ) --+ min

under the linearized constraint ( V a F (k) (a, u), A a ) - F (k) (a, u),

k - 1,...,L,

(8.4)

together with A a E Tet(.A). The optimization (8.3), which minimizes the Mahalanobis distance I]Aal]f~[a], can be justified as m a x i m u m likelihood estimation for Gaussian noise (see Section 5.1.1). The first order solution is given as follows (see eq. (5.17))" L

A a - l~[a] E k,l--1

l~V2w(kl) ~ O.

-

(8.44)

Under these approximations, eqs. (8.32) and (8.33) are respectively approximated by L

so

Z

+

,

k,l=l (8.45)

8.3.

Effective Gradient Approximation

257

L

t~

E

W(kt)F(a)VuF(t)"

(8.46)

k,/--1

Let s be the range of the (LL)-matrix W~(u) defined by eq. (8.43); it is a linear subspace of 7~L. Let p C~(u) be the L-dimensional projection matrix onto the linear subspace s Since P C~(U)W~(u) - W~(u), we have

OpC~(u) Oui W~(u) + pLy(u)OW~(U)oui __ OW~(U)oui If W~(u) is very smooth in u, its range s a first approximation. Hence, OPC~(u)/oui ,~ W~(u)W~(u)-, we obtain the approximation

.

(8.47)

does not depend on u to O. Noting that pC~(u) =

L

Z

w(kl)(w(lrn))-v7vu LTr(mn) ,, ~ ~ VuW~a~>T)

Z

W+

c~=l k,l= l

1

N

L

(kt)

(9.18)

c~=l k,l=l

The first term on the right-hand side is O(e/yr~) since it is an average of independent random variables of mean zero, while the second term is O(e2). Since E[Aa(~k)] = 0, we see that e2

N

L

E[ZXM]- ~ Z ~ c~=l k,l=l

w~')Vo[~),~)]

9

(9.19)

9.1.

Eigenvector Fit

271

9 Let ft be the true value of u that satisfies the hypothesis ( ~ ) , u) = 0, k - 1, ..., L, c~ - 1, ..., N. From eq. (9.17), it is immediately seen that Mft

-

O,

(9.20)

meaning that the true value ~ is the unit eigenvector of the unperturbed moment matrix M for eigenvalue 0. _

Now, consider the gradient of J[u] at the true value ~. Differentiating eq. (9.10) with respect to u, n o t i n g / ~ f t - 0 and ( ~ ) , ~) - 0, and using eq. (9.18), we obtain Vu~][~] - 2 M ~ t - 2(_M + A M ) ~ -

2

N

2AM~t

L

a(k) a~)

(kl) a=l

(9.21)

k,l= l

Let P~t be the n-dimensional projection matrix along ~. Since (g(k), ~) _ 0 implies P ~ t ~ ) - ~ ) , we see from eq. (9.21)that N

L

a=l

k,/=l

2 P~tVu3[~t]- ~ ~

~

W,,~ (kl) ( ( A a ~ ) , ~ ) , ~ k ) + Aa~)Aa~)-r~t _(~,, (Aa(k)Aa~)T)~)~)

,

(9.22)

Suppose the estimate u* for computing the weights W (kt) is chosen to be the true value ~. Using eq. (9.13), we observe that 2e 2

N

L

E[P~tVuJ[ft]]- ~ Z Z

lu

(V~

a ~ ) ] f t - (ft, Vo[a~ ), a~))]ft)ft)

or-- 1 k , l = 1

= 2r

~ Z c~--i

Z

W('kl)V~

-

k,l--1

The right-hand side is O(e 2) but not zero in general even in the limit N oc. In other words, P~tVu.][g] is not zero on average however many data we use. This implies that J[u] does not take its minimum at ~ because J[u] can be further reduced without violating the constraint I1 11 - 1 if u is infinitesimally incremented in the direction - P u V u J [ f t ] . It follows that if/t minimizes J[u], the probability that ( / t - ~ , < 0 is larger than the probability that ( / t E[Pu%.J[ ll) > 0. the eigenvector fit is biased (Fig. 9.1).

272

Chapter 9.

Renormalization

~

)[ul fi

ti

Fig. 9.1. Bias of the value fi that minimizes J[u]. On the other hand, no such bias exists if the the original function J[u] is minimized. Differentiating eq. (9.4) with respect to u, we obtain 1

N

L

(~----1 k , l - 1

+2w~(~)(~)(~), ~)~)).

(9.24)

Evaluating this at ~ and substituting eq. (9.2) into this, we observe that N

L

c~:1 k , l : l

+2W.(kt)(~l(a~) + A a (a), ~)(a~ ) + Aa~))) N

_

L

1

(Vu

(kt)(. (Aa~)

,

c r = l k,l= l

+2W. (kt)(~)((Aa(k), ~)a~) + (Aa~), ~)Aa~))) N

L

c~=1 k , l = l

(9.25) From eq. (9.14), we see that

E[VuJ[~,]] - ~

V~w(~') (a)(a, Vo[~), (x:l

~)]a)

k ,l: l

+2 Z w~(~)(~)v0[~),~)]" k,l:l

)

-0.

(9.26)

9.2.

Unbiased Eigenvector Fit

273

Consequently, E[P~tVuJ[ft]] = 0. Since P~tVuJ[ft] is an average of a large number of independent data of mean zero and of order O(e), we have P~tVuJ[ft] = O(e2/v/-N). This means that the minimum of J[u] occurs at the true value ~ in the limit N --+ oc: the optimal solution is a consistent estimator. From this argument, we also see that the statistical bias of the eigenvector fit is solely due to the least-squares approximation W(~kl) (u) ~ W(~kl) (u*), or equivalently VuW(~ kl) ~ O. (9.27) (See eq. (8.87).) Hence, the bias is unavoidable however the estimate u* is chosen--even for u* = ~,.

Unbiased Eigenvector Fit

9.2

9.2.1

Unbiasedleast-squares approximation

The analysis in the preceding section suggests that an unbiased solution could be obtained if

1

N

L

(kl)

E

J[u]--~E

W~

((a~)u)a~ ) ,

(

,u) - V[a~ )

,a~)])

(9.28)

c~=l k , / = l

is minimized instead of eq. (9.7). By the same analysis as shown earlier (see eq. (9.23)), we conclude that E[VuJ[~t]]- 0 and hence E[PuVuJ[~]]o. Since P~t Vu~][~] is an average of a large number of independent data of mean zero and of order O(e), we have P~tVuJ[ft] - O(e2/v/N). Hence, the solution is a consistent estimator. We call eq. (9.28) the unbiased least-squares approximation to eq. (9.4). The unbiased least-squares approximation is computationally convenient. In fact, if we define the unbiased moment matrix

M -

N

1

L

)

Z

a~)T

- V [ a ~ ), a~)]),

(9.29)

o = 1 k,l= l

eq. (9.28) can be rewritten as the quadratic form

J[u] - (u, 1VIu),

(9.30)

which is minimized under the constraint Ilull - 1 by the unit eigenvector/t of / ~ for the smallest eigenvalue. Let us call the s o l u t i o n / t thus obtained the

unbiased eigenvector fit. Assume that the true value ~, which is the unit eigenvector of M for eigenvalue 0 (see eq. (9.20)), is unique, i.e., the eigenvalue 0 is a simple root.

274

Chapter 9.

Renormalization

According to the perturbation theorem, the perturbation A/t - / t - fi of/t from ~ is given in the following form (see eq. (2.111))" A/t - - / f / / - A/~lrf~ + O(e2). The perturbation A / ~ -- M - / ~ N

1

o f / ~ f r o m / ~ has the following form:

L

~M - ~ Z Z w~(~) (~2) ~ (~=1

(9.31)

a~)T

+ ~a(2 )

~,~) T)

k.l=l

N

L

1

(kt)

+~ E ~ wo (~ (~=1

a~)

T

~)

,

- ~V0[a~) ~)l).

(9.32)

k.l--1

Suppose the estimate u* for computing the weights W(~kt) - W (kl) ( u * ) i s chosen to be the true value ft. Ignoring terms of O(e 2) and noting that (d~), ~t) - 0, we obtain from eq. (9.32) 1

N

L

c~=1

k.l=l

~M~ = ~ ~ Z w~,(kt) (~ a~) , f t ),~

(k)

.

(9.33)

Since a ~ ) is independent for each a, we obtain E[(/kMa)

(A/~a)

1

m]

N

L

-- N 2 E a.fl=l 62

N

L

-- N 2 Z c~.~=1

=

N2 E

]~)~

~ w'(kl)W(mn)(fl"'S[Aa~)Aa(n)T~"~'(k)~l'(m) k.l.m.n=l ~ w(~kl)w(mn) (~t. 5~t~Vo[a~) , a~)]~t)a~ ) 5 ~ ) k.l.m.n=l

E

c~=l

~2 N _-- N2 ~ c~=l

k.m=l

w(kl)w(nm)(~l,,

go[a~ ) . a~)].)

l.n=l

)

~.(k)a(m)

L ~ I/V(km) ~t(k) ~t(m) __ ~/iS/, E2 k,m=l

(9.34)

where the identity (9.15) is used. From eq. (9.31), the covariance matrix V[~i] - E[A~iA~i -r] of the unbiased eigenvector fit ~i is obtained to a first approximation in the following form: E2

V [ / t ] - _M-E[(A/I~ft)(A/~'~)T]/~- = ~ M = e2

Z c~----1 k.l= l

)

W(kt)a~)a~) -~

"

M/~-=

E2 _ _

~M

(9.35)

9.2.

Unbiased Eigenvector Fit

275

The rank of this matrix is n - 1 in general; the null space is {R}L. Comparing this with the general result in Section 7.1.5 (see eq. (7.57)), we conclude that the unbiased eigenvector fit has the same covariance matrix as the truly optimal estimate. In other words, the unbiased least-squares approximation (9.28) does not decrease the accuracy of the solution. In actual computation, eq. (9.35) is approximated by

(z N

=

EL

T

k,l=l

)

(9.36)

where P/~ is the n-dimensional projection matrix 2 along/~. Here, h ~ ) is the optimally corrected value of a ~ ) (see Section 5.1.2), but to a first approximation it can be approximated by the data value a ~ ) itself.

9.2.2

Analysis of residual

As discussed in Section 7.1.4, the residual of optimization plays an important role in testing of hypotheses. As in Section 7.1.4, we consider both the residual J[~] for the true value ~ and the residual j[/t] for the estimate/t. A. Residual for the true value Consider the residual J[~] for the true value a. Since ( ~ , / ~ ) that

- 0,

we

see

j[~] -- (~, f ~ u ) - (~, ( M + ~ f ~ ) ~ ) - (~, ~f~u) 1 = NE

N

L

(k~) , a~) W~ (Aa~) R)(A ,R)

~

c~=1 k , l = l 62

N

N ~

L

~

W(kO(~' V~

a~)]u)"

(9.37)

c~=1 k , l = l

We assume that the weights W~ kl) are so chosen that W (kl) - W (k') (~,). Then, the second term on the right-hand side of the above equation is - e 2 r (see the identity (9.13)). If we put -

(9.3s)

the L-vector e~ - (e (1),..., e~)) T is a Gaussian random variable of mean 0. Its covariance matrix is V[e~]- (E[e~)e~)])-

((ft, E[Aa~)Aa~)]~t))

2The projection matrix P u is inserted to ensure t h a t the computed covariance matrix V[fi] has null space {fi}L, which is required by the normalization constraint Ilfi[[ = 1.

276

Chapter 9.

Renormalization

Hence, V[e~]- = W ~ / e 2, so the first term on the right-hand side of eq. (9.37) is l

N

L

-

~.

- X ~(~.,

a--1 k,l--1

vW.]-~.).

(9.40)

c~--I

Since V[e~] (= e2W-~) has rank r and the vector e~ is independent for each c~, the sum ~ g~=l(e~, V[e~]-e~)is a )~2 variable with rN degrees of freedom (see eq. (3.63)). Consequently, eq. (9.37) can be written in the form ~2

J[a]-

7(~,~-

rX),

(9.41)

where X~N : stands for a X 2 variable with r N degrees of freedom. The expectation and variance of the residual J[~] are given as follows (see eqs. (3.59)):

2e4r

E[J[~]] - 0,

v[J[~]] - --~-.

(9.42)

It follows that if a particular value ~t is given independently of the data {a~) }, we can construct the X2 test for the strong hypothesis that the true value is ~: the strong hypothesis is rejected with significance level a% if 62

J[~] > -~(X2~N,, -- rX).

(9.43)

B. R e s i d u a l for t h e e s t i m a t e

Consider the residual j[/t] for the unbiased eigenvector fit/t. We see that

J[~] - ( ~ , . ~ ) - (~ + A~, M(~ + ~ ) ) = (~, M~) + 2(/x~, M ~ ) + (A~, M/x~) = j[~] + 2 ( ~ , (M + ~ M ) ~ ) + (a~, M ~ ) = j['~] + 2(A/t, A / ~ ) +

(A/t,/I/A/t).

(9.44)

Since A/t = O ( A / ~ ) = O(e/v/-N) (see eqs. (9.31) and (9.32)), the last term on the right-hand side is ~3

(A/t, f / A / t ) - (A/t, (AT//+ A / ~ ) A / t ) - (A/t, AT//A/,) + O(Nv/_~).

(9.45)

Consider the second term on the right-hand side of eq. (9.44). According to the perturbation theorem, the smallest eigenvalue A (= j[/t]) o f / ~ has the following form (see eq. (2.108)): 6

-

(/t, A / ~ / t ) + O(AA}I) 2 - O(--~:).

(9.46)

9.2.

Unbiased Eigenvector Fit

277

Since/t is a unit vector, we have (?2

(zx~, ~) - o(zx~) ~ - o ( v ) . Hence,

(9.47)

(?3

(Aa, M a ) - s

(9.48)

~) - O( N .~=).,/j~

On the other hand,

_

(?3

(9.49)

= (A/t, A ] ~ f t ) + (A/t, M A / t ) + O ( N ).~/~ .rr~. Comparing eqs. (9.48) and (9.49), we conclude that _

(A/t, A.~r

(?3

- -(A/t, MA/t) +

O(N"7;)'x/J~/

(9.50)

Substituting eqs. (9.45) and (9.50)into eq. (9.44) and using eqs. (9.37) and (9.40), we obtain (?3

j[,i] - 2[~,] - (zx,i, M ~ , i ) + O ( x v ~ ) (?3

= V

( ~ , v [ ~ o ] - ~ ) - (~x,i, v[,i]- zx,~) - r x c~=1

+ o( x4-~), (9.51)

e21~-/N

where we have used the fact that V[/t] = (see eq. (9.35)). Since we are assuming that there exists no constraint other than II,ill- 1, the rank of V[/t] is n - 1 in general. Hence, the quadratic form (A/t, V [ / t ] - A / t ) i s a X2 variable with n - 1 degrees of freedom (see eq. (3.61)). From eqs. (9.31), (9.33), and (9.38), we have N

~x,~-

X1 ~

L

Z

W~(~)~) M a~) + o(~: ),

(9.52)

a=1 k,l=l

meaning that A/t is obtained, to a first approximation, by a linear mapping from {e~}. Consequently, J[/t] can be written to a first approximation as follows (see eq. (3.64)):

6_2 J[/t]-

--~(~2vN_n+ 1 -- r N ) .

(9.53)

278

Chapter 9.

Renormalization

Its expectation and variance are E2

E[J[/tl]--~(n-

2e4r(n-l) V[j['5]]- ~ 1 rN

1),

"

(9.54)

From eq. (9.53), we can construct the X2 test for the weak hypothesis that eq. (9.1) holds for some value u: the weak hypothesis is rejected with significance level a% if E2

>

XrN-nWl,a - r N + n - 1 ) .

(9.55)

Comparing eqs. (9.42) and (9.54), we observe that the expectation and variance of J[/t] are both smaller than their respective values for J[R] because of the correlation etween/t and the data { a ~ ) }.

9.3 9.3.1

Generalized Eigenvalue Fit N o i s e level e s t i m a t i o n

Although the unbiased eigenvalue fit seems very desirable for its unbiasedness and computational convenience, a difficulty arises if we want to compute it in a real situation: the noise level c must be estimated precisely. If the noise is underestimated, statistical bias still remains, while if it is overestimated, statistical bias arises in the opposite direction. However, accurately predicting the noise level e is very difficult in practice. This problem does not occur for the eigenvector fit ~, since the leastsquares approximation does not involve the noise level e: the covariance matrices V [ a ~ ), a ~ )] need to be estimated only up to scale. Thus, the eigenvalue f i t / t is insensitive to the noise level e, while the unbiased eigenvalue f i t / t is very sensitive to it. As pointed out in Sections 5.1.5 and 7.1.4, it is often easy to predict the form of the covariance matrix from geometric considerations, but its absolute magnitude is very difficult to predict a priori. Hence, it is desirable to solve the problem without knowing the noise level e. In the following, we present such a scheme; the noise level e is estimated a posteriori. Note that eq. (9.29) can be written in the following form: /14 - M 1

N

e2N,

(9.56)

L

N - '-N E E w(kl)V~ (~=1 k,l=l Let u* be the estimate for computing the weights: W (kO

(9.57)

-

9.3.

Generalized Eigenvalue Fit

279

see from the identity (9.13) that N

L

1 E (u*,Nu*) - -~

W~(kl) (u*, Vo[a~) , a~ )] u* ) - r .

E

(9.58)

o~--1 k,l--1

Recall that we have defined the unbiased moment m a t r i x / ~ - M - e2N with the expectation t h a t / ~ should be a better estimate o f / ~ / t h a n M . This suggests that the noise level e should be estimated so that / ~ ~ M - e2 N .

(9.59)

However, exact equality m a y not hold. So, we_ introduce the following cornpromise. Since the true value ~ satisfies (~, M~t) - 0, we seek the value e that satisfies ( u , / ~ u ) - (u, M u ) - e2(u, N u ) - 0 (9.60) for some u. There may exist multiple pairs {u, e} that satisfy eq. (9.60). From among them, we choose the one for which e 2 - (u, M u ) / ( u , N u ) is the smallest. Eq. (9.58) implies that if u ~ u* then (u, N u ) 7s O. Hence, the vector u is determined by minimizing

I[u]

-

(u, (u, M N uu)) "

(9.61)

In other words, we minimize the generalized Rayleigh quotient (see eq. (2.103)). Since M is generally positive definite 3 in the presence of noise, eq. (9.61) is minimized under the constraint Ilull - 1 by the unit generalized eigenvector/~ of the generalized eigenvalue problem

Mu - cNu

(9.62)

for the smallest generalized eigenvalue (see eq. (2.104)). At the same time, the smallest generalized eigenvalue c gives an estimate of e2. We call the resulting solution/~ the generalized eigenvector fit.

9.3.2

Accuracy of generalized eigenvector fit

A. U n b i a s e d n e s s of generalized e i g e n v e c t o r fit The bias analysis in Section 9.1.2 can be applied to I[u] by assuming that the estimate u* for computing the weights W (kt) is chosen to be the true value ft. Differentiating eq. (9.61) with respect to u, we obtain

2Mu -

(u, Nu)

2(u, M u ) N u

(9.63)

-

3The m o m e n t matrix M is positive semi-definite by definition; it is singular if and only if noise does not exist. It follows that if the smallest eigenvalue of M happens to be zero, the corresponding unit eigenvector is the true value fi, so we need not consider this case.

280

Chapter 9.

Renormalization

Since (~, N~t) - r (see eq. (9.58)), we see that 2 (AT//+ A M ) ~ t 2(~t, (_37//+ A M ) ~ t ) N ~ (~t, N ~ ) (~, N ~ ) 2

VuI[~] -

2AM~t

2(~, A M ~ t ) N ~ t

r

r 2

(9.64) "

Eqs. (9.19) and (9.57)imply E[AM] - e2N, so

E[VuI[~]]- 2e2Nu r

2r --

r 2

=

0.

(9.65)

Consequently, E[P~tVuI[~t]] - O. Hence, the generalized eigenvector fit is a consistent estimator by the same argument as in the case of the unbiased eigenvector fit. B. Covariance m a t r i x of generalized eigenvector fit The covarimlce matrix V[/t] of the generalized eigenvector f i t / t is evaluated as follows. If I[u] takes its minimum a t / t , we have PuVuI[it] = 0. Eq. (9.63) implies (u, VuI[u]) - O. Consequently, PitVuI[it] - VuI[/t], and the generalized eigenvector f i t / t satisfies VuI[/t] - 0.

(9.66)

Using eq. (9.63), we can rewrite this equation as (/t, N i t ) M i t - (/t, M i t ) N i t .

(9.67)

_

S u b s t i t u t i n g / t - ft + A/t and M - M + A M into this, we obtain (~, N ~ ) ( / ( f A / ~ + A M ~ ) - (~, A M ~ ) N ~

+ O(e2).

(9.68)

Since (~, N ~ ) - r, we have

l"~Ait

-

-(AM

- ( ~ t ' A M ~ t ) N ) ~ + O(e2).

(9.69)

r

Hence, - -a3-

+ o(d),

(9.70)

where we have defined A / ~ ' - A M - (~' A M ~ t ) N . r

Let A / ~ - A ) I - / ~ .

Eq. (9.56)implies that

A/~ - M-

E2N-/~

- A M - E2N.

(9.72)

9.3.

Generalized Eigenvalue Fit

281

Hence, eq. (9.71) can be written as

AI~' - AI~ - ( (ft, AMft)Nr

- e 2) W -

A / ~ - ~,3[~t]r

(9.73)

where we have used the relation Jilt] - (ft, A/~/~) - (ft, ( A M - e 2 N ) ~ ) - (~, A M f t ) -

Sinc

(9.42) imply J[a] ~ d

V/2r/N,

e2r.

(9.74)

(9.73)is written as 52

A/I)/' - A / ~ / + O ( ~ ) .

(9.75)

This means that the statistical behavior of A / ~ ~ is the same as that of A/I)/if terms of O(e2/v/-N) are ignored. Hence, eq. (9.34) holds for A M ' as well to a first approximation, and the covariance matrix of the generalized eigenvector f i t / t is also given by ~2

L

W..(kOa(k)a~ )T

V [ / t ] - ~ / f / / - - e2 a=l

,

(9.76)

--1

which can be approximated by eq. (9.36) in actual computation.

9.3.3

Analysis of residual

A. Residual

for t h e t r u e v a l u e

Consider the residual I[~] for the true value ~. Here, too, we assume that the estimate u* for computing the weights W (kO is chosen to be the true value ~. Then, I[ft]- (f~,M~)= (~t, N ~ )

(R,/~)+e2(~,N~)= r

(R,/P/R) + e 2 r _ J[f~] ~_ ~2. r r (9.77)

From eq. (9.41), we obtain e2

2

(9.78)

Hence, the expectation and the variance of I[~] are 2(~4

E [ I [ ~ ] ] - ~2

V[I[~]]-

rN"

(9.79)

If the noise level e is given a priori, we can construct the X2 test for testing the strong hypothesis: if a particular value ~ is given independently of the

282

Chapter 9.

Renormalization

data {a~)}, the strong hypothesis that the true value is ~ is rejected with significance level a% if e2 2 (9.80) I[~] > 7 - ~ X r N , a . B. Residual for the estimate Consider the residual I[/~] for the generalized eigenvector fit/~. Substituting /L - ~ + Au -- ~ + O(e) and noting that (~, N ~ ) - r, we obtain (/z, M/~)

(/~, flr/~)+ e2 (/~, N/~)

i[~] - (,~, N a ) = (a, N a ) + O(r (~, M~) + ~((~, N~) + O(~11 r + 0(~)

(a, M a )

+ e2 + O(e3).

(9.81)

Eqs. (9.31), (9.70), and (9.75)imply that the statistical behavior of the generalized eigenvector fit is the same as the statistical behavior of the unbiased eigenvector fit if terms of O(e2/v/-N) are ignored. Hence, the quadratic form (/~,/174r/~) in eq. (9.81) can be identified with the residual j[a] of the unbiased eigenvector fit under the same approximation. It follows from eq. (9.53) that to a first approximation E2

(9.82)

1[~1- 7-~x~_~+~. Hence, its expectation and the variance of I[/~] are rg

, Y[I[/~]] = 7-N

1

rg

"

Comparing eqs. (9.79) and (9.83), we observe that the expectation and variance of I[/~] are both smaller than their respective values for I[~] due to the correlation between/~ and the data {a~)}. From eq. (9.82), an unbiased estimator ~2 of the squared noise level e2 is obtained in the following form: ~2 =

I[/~] 1 - ( n - 1)/rN"

(9.84)

Its expectation and variance are 2E 4

E[~2]_ e2,

V[~:]- rN-

n + 1"

(9.85)

If the noise level e is given a priori, we can construct the X2 test for the weak hypothesis that eq. (9.1) holds for s o m e value u" the weak hypothesis is rejected with significance level a% if E2

2

1[~] > 7-~X~N_~+~,o.

(9.86)

9.4.

Renormalization

283

In terms of the estimate ~2 computed by eq. (9.84), the above X2 test takes the form ~2 2

~rN-n+l,a

--5 e > rN-

9.4

n + 1"

(9.87)

Renormalization

9.~.1

Iterations for generalized eigenvalue problem

In order to compute the generalized eigenvector fit, one must solve the generalized eigenvalue problem for the moment matrix M with respect to the matrix N . If N is of full rank, this is a nonsingular generalized eigenvalue problem (see Section 2.2.4), and the solution is obtained by computing the inverse square root N -1/2 (see eqs. (2.93) and (2.94)). However, the domain of the data { a ~ ) } is usually constrained, i.e., the covariance matrices V [ a ~ ) , a ~ )] are generally singular. As a result, the matrix N is generally singular, so the generalized eigenvalue problem is singular (see Section 2.2.5). It follows that one must solve a generalized eigenvalue problem of a smaller size (see eq. (2.100)). However, there exists a more efficient method if we note the following two facts" 9 Not all the generalized eigenvalues and generalized eigenvectors need to be computed" only the smallest generalized eigenvalue and the corresponding generalized eigenvector are necessary. 9 Computing the smallest generalized eigenvalue c of the problem M u = c N u is equivalent to computing the value c for which the smallest eigenvalue of M - M - c N is 0. ^

The second assertion is proved as follows. If c is the smallest generalized eigenvalue of the problem M u - c N u a n d / t is the corresponding generalized eigenvector, we have f / I i t - M i t - c N i t - 0, (9.88) meaning that ./~ has eigenvalue 0 for eigenvector/t. Suppose M has another eigenvalue -)~' (< 0) for a unit eigenvector u'. Then, M u ' - M u ' - c N u ' = - ) ~ ' u ' . Hence, (u', M u ' ) - c(u', N u ' ) - -)~'. (9.89) If (u', N u ' ) - O, then (u', M u ' ) - -)~', which is a contradiction because M is semi-positive definite. Hence, (u I, N u ~) > 0, but then (u', M u ' )

= c-

)~'

(9.90)

which is also a contradiction, since c should be the minimum of the generalized Rayleigh quotient (u, M u ) / ( u , N u ) .

284

Chapter 9.

Renormalization

Thus, we only need to compute the value c for which the smallest eigenvalue of .l~ - M - c N is 0. I f / t is the corresponding unit eigenvector, we have ( / t , / ~ / t ) - 0. Let u be the unit eigenvector of M for the smallest eigenvalue )~ (~= 0). Then, (u, ( M - c N ) u ) = (u, M u ) - c(u, N u ) = )~ - c(u, N u ) .

(9.91)

It follows that if we define - M-

~----~----N,

we have ( u , / ~ u ) - 0. However, u may not be an eigenvector o f / ~ . iterate this process:

(9.92) So, we

1. Let c = 0. 2. Let u be the unit eigenvector of the matrix

I~/I-M-cN

(9.93)

for the smallest eigenvalue, and let A be that smallest eigenvalue. 3. If A ~ 0, return u and c. Else, update c as follows:

c+--c+

(9.94)

4. Go back to Step 2. The convergence of this process can be confirmed as follows. If u is the unit eigenvector of M for the smallest eigenvalue A, the m a t r i x / ~ is perturbed by - ) ~ N / ( u , N u ) at the next iteration. According to the perturbation theorem, the corresponding eigenvalue A is perturbed in the following form (see eq. (2.108)): AN )( - )~- (u, (u, N u ) u) + O()~2) - O()~2). (9.95) In other words, A converges to 0 quadratically as in the Newton iterations, meaning that three or four iterations are usually sufficient.

9.~.2

Iterations for weight update

So far, we have assumed that the weights W (kl) are approximated by using an appropriate estimate u*. Since we need not worry about the bias any longer, it can be determined by iterations: we guess the initial value u0 and compute the generalized eigenvector fit u x; using u l, we compute the generalized eigenvector fit uz and so on. Since the generalized eigenvector fit is itself computed by iterations, the combined process requires double loops of iterations. However, the two loops can be merged in the following form:

9.4.

Renormalization

285

1. Let c - 0 and W (kl) - Ski, a -- 1, ..., N. 2. Compute the following matrices M and N : N

1

L

M - -~ E E W (~kt)a(~k)a(~)T a=l

1

N

(9.96)

k,l=l L

N - -~ E E W'(kOV~ ), a~)]"

(9.97)

c~=l k , / = l

3. Compute the smallest eigenvalue A of the matrix

M-M-cN

(9.98)

and the corresponding unit eigenvector u. 4. If A ~ 0, return u, c, and 1~. Else, update c and W ~ follows: c +-- c + (u,

Nu)'

(

W~ +--- (u,

)

V0[a (k), a~)]u) ~ .

(9.99)

5. Go back to Step 2. We call this process renormalization. Although the convergence is no longer quadratic, the computation is usually very efficient. Since the returned value u gives the generalized eigenvector fit/t, the covariance matrix o f / t is given by eq. (9.76): E2

V[/t]- v(M)n_I.

(9.100)

Here, the rank-constrained generalized inverse is used b e c a u s e / ~ / t - 0 may not be strictly satisfied if the iterations are pffematurely terminated. Since the eigenspace for the smallest eigenvalue of M is {/t}L, computing ( / ~ ) ~ - 1 is equivalent to applying the projection matrix P / t as in eq. (9.76). After renormalization, the returned constant c equals the residual I[/t] of the generalized eigenvector fit /t (see eqs. (9.61), and (9.62)). Hence, an unbiased estimator ~2 of the squared noise level e2 is obtained from eq. (9.84) in the following form: ~:2 : c

1

1)/rN"

(9.101)

Its expectation and variance are given by eqs. (9.85). It follows that the number N of necessary data for uniquely estimating u must be such that n-1 N > ~ . r

(9.102)

286

Chapter 9.

Renormalization

If N = ( n - 1)/r, the value of u is determined but the noise level e cannot be estimated. The significance of renormalization is that it produces not only an unbiased estimator /t of the parameter u by a simple numerical computation but at the same time its normalized covariance matrix V0[/t], an unbiased estimator ~2 of the squared noise level, and its variance V[~2] as well. Let ~max be the largest eigenvalue of V[/t], and Vmax the corresponding unit eigenvector. The primary deviation pair {u +, u - } is given as follows (see Section 4.5.3): U+ -- N[/t + X/#maxVmax],

U - -- N [ / t -

~/~maxVmax].

(9.103)

If the value e is given a priori, the hypothesis (9.1) can be tested by comparing it with the estimate ~ (see eq. (9.87))" the hypothesis (9.1) is rejected with significance level a ~ if ~2

2

XrN-n+l,a

~e > r N 9.5

n + 1"

(9.104)

Linearization

9.5.1

Linearized algorithm

Renormalization is a numerical means to compute the generalized eigenvector fit, which is the solution of the unbiased least-squares approximation to the original optimization, but the unbiased least-squares approximation and the least-squares approximation are both designed for problems for which 1. the hypothesis is linear, and

2. no constraint is imposed other than normalization. In many application problems, the parameter u is constrained in various ways, even though the hypothesis is linear (see Chapter 5). For example, u may be a unit vector constrained to be in a parameter space b / C 7~n. Then, the procedure for renormalization as described in the preceding section can no longer be applied. We now devise a method for overcoming this restriction by noting that the following generally holds: 9 The constraint u E L/ is compatible with the hypothesis (9.1): there exists a solution fi E/4 that satisfies the hypothesis (9.1). 9 The hypothesis (9.1) is an overspecification: there exists a minimum number No (_< N) such that equations ( ~ ) , u) - 0, k - 1, ..., L, a 1, ..., No, can uniquely determine the solution ~t E/4. It follows that if the noise is small, the solution/t based on the hypothesis (9.1) alone is expected to be a good estimate of the true value ~t. This observation leads us to the following approach:

9.5.

Linearization

287

Fig. 9.2. Optimally projecting fi onto the parameter space/4 C ~n.

1. We apply renormalization without considering the constraint u EL/. 2. We then optimally project the resulting solution onto the parameter space/2 (Fig. 9.2). L e t / t be the solution obtained in Step 1, a n d / t t E L/its projection onto b/. Suppose the manifold/4 is defined by M equations F (m) (u) - 0,

rn - 1, ..., M,

(9.105)

together with the normalization condition Ilull- 1. Substituting/t / - / t - A / t into eq. (9.105) and taking a linear approximation, we obtain (Vu/~(m), A/t) -/~(m),

rn - 1, ..., M,

(9.106)

where Vu/~(rn) and/~(m) are the abbreviations of VuF(m)(it) and F(m)(/t), respectively. The correction A/t is determined by minimizing the Mahalanobis distance IIAullv0[/t] (see eq. (5.6))" J = ( A t , V0[/t]-Au)--+ min,

A/~ E {/L}f.

(9.107)

Here, we use the normalized covariance matrix V0[/~] instead of V[/L], since the solution of the optimization is not affected if the covariance matrix is multiplied by a positive constant. Theoretically, V0[/L] should be evaluated a t / I but we approximate it by the value at/~. Note that the normalized covariance matrix V0[/t] is given as a by-product of renormalization (see eq. (9.100)). Let r be the rank of the constraint (9.105). As shown in Section 5.1.2, the first order solution of the optimization (9.107) under the constraint (9.106) is given in the following form (see eqs. (5.15) and (5.17)): M

A/t--Vo[/I,] E

W(~'~)P(m)VuP("),

(9.108)

re,n--1

(9.109)

288

Chapter 9.

Renormalization

In order to impose the normalization condition {{/t'{] - 1 exactly, the actual correction o f / t and its normalized covariance matrix takes the following form (see eqs. (5.25)and (5.26)): /t'-

~/~[/t]' - Pit~Vo[/t]Pit~.

N [ / t - A/t],

(9.110)

Here, P / tu, is the n-dimensional projection matrix onto the tangent space T/t,(5/) to the manifold b/ at /t'. This correction is iterated until the cons t r a i n t / t ~ E 5/is sufficiently satisfied. The normalized a posteriori covariance m a t r i x of the final v a l u e / t ' is given as follows (see eq.(5.31)): M

Pit~ Y~

Y~

' - E w(rnn) ( gi.L~P~ m,n=l

)(git~ P~

) T"

(9.111)

This matrix has rank 4 n - r - 1. From eq. (9.111), the a posteriori covariance matrix o f / t ' is given in the form V[/t'] = ~2V0[/t'], where ~2 is an estimate of the squared noise level e2; it is given as a by-product of renormalization (see eq. (9.101)). We call the above procedure linearization, and the resulting algorithm the

linearized algorithm. 9.5.2

Decomposability condition

In some applications such as 3-D motion analysis, which we will study in Chapters 11 and 12, the hypothesis is nonlinear in the parameter u but can

be rearranged into the form ( ~ ) , v)

- -

0,

k

- -

1,

. . . ,

L,

(9.112)

where v is obtained from u by a nonlinear mapping v ( . ): 7"4n --+ 7-4m in the form v = v(u). (9.113) As before, the data { a ~ ) } , k - 1, ..., L, c~ - 1, ..., N, are m-vectors and are assumed to have the form

(9.114) where A a ~ ) is a random variable of mean 0, independent for each a. It appears that renormalization and linearization can be applied if v is regarded as a new variable instead of u. However, we must be careful about the degree of freedom of the variable: 4 T h e last " - 1 " in n -

r-

1 is due to the normalization I l f i ' l ] - 1.

9.5.

Linearization

289

R"

R"~

Fig. 9.3. Optimally projecting ~, E ~'~ onto the manifold 12 C 7~m so that the equation v(6) - fi' has a solution fi E b/. 1. If rn < n, the p a r a m e t e r u cannot be d e t e r m i n e d even if v is obtained. In other words, the original p r o b l e m is underspecified; the solution is indeterminate. 2. If m - n, eq. (9.113) d e t e r m i n e s t h e value u from the c o m p u t e d value v in general. 3. If m > n, eq. (9.113) is an general.

overspecification; no solution m a y exist in

Let us consider the case rn > n m o r e closely. If rn > n, vector v(u) is generally c o n s t r a i n e d to be in an n~-dimensional manifold 12 C ~ m when u ranges over the p a r a m e t e r space 5/ C T~~. Let F (m) (v)

-

O,

m

--

1, . . . , m

-

n',

(9.115)

be the e q u a t i o n of t h e manifold Y. In order t h a t eq. (9.113) has a solution u, the o p t i m i z a t i o n for v m u s t be done u n d e r eq. (9.115). In this sense, eq. (9.115) is called the decomposability condition. If the d e c o m p o s a b i l i t y condition (9.115) is imposed, r e n o r m a l i z a t i o n can no longer be applied in the original form. As discussed in the preceding subsection, a simple way to overcome this restriction is c o m p u t i n g an e s t i m a t e / J of v w i t h o u t considering the d e c o m p o s a b i l i t y condition (9.115) and t h e n optimally projecting i~ onto the manifold 1; (Fig. 9.3). As described earlier, this p r o j e c t i o n is done so t h a t the M a h a l a n o b i s distance I I v ( u ) - viIvo[v ] is minimized. T h e p r o c e d u r e is s u m m a r i z e d as follows: 1. C o m p u t e the optimal estimate/~, its n o r m a l i z e d covariance m a t r i x V0[/~], and the u n b i a s e d e s t i m a t o r ~ :~ the s q u a r e d noise level by applying r e n o r m a l i z a t i o n to the hypothesis (9.112). 2. C o m p u t e t h e optimal correction A 9 so t h a t / ~ - / ~ - A 9 satisfies the d e c o m p o s a b i l i t y condition (9.115); do iterations if necessary.

290

Chapter 9.

Renormalization

3. Compute the normalized a posteriori covariance matrix V0[O~] of the resulting/~' E ];. 4. Solve the equation v(/t) - 'O' for/t E ~n. 5. Compute the normalized a posteriori covariance matrix V0[/t] of the resulting solution /t e L/ (see eq. (3.16)); its a posteriori covariance matrix is given in the form Y[/t] - ~2V0[/t].

9.6

9.6.1

S e c o n d Order R e n o r m a l i z a t i o n

Effective value of nonlinear data

In many problems in computer vision and robotics, the hypothesis is nonlinear in the raw data but can be rearranged into the form (b~) , u) -- 0,

k - 1, ..., L,

(9.116)

where b~) is obtained from g~ by a nonlinear mapping b(k)( 9)" T~m --4 7~" in the form b ~ ) - b(k) (a~). (9.117) In this case, if we compute b~ ) - b (k) ( a , )

(9.118)

for given data { a~ }, we can apply renormalization to the converted data { b, }. The deviation Ab~ ) -- b~ ) - E[b~ )] (9.119) is a random variable of mean O. Hence, the covariance matrix of b~ ) is defined by r [ b ~ ) b~)] _ E[Ab~)Ab(/)T]. (9.120) However, such a conversion introduces a small error. As before, let the raw data be a~ - d~ + Aa~, the noise A a a being an independent random variable of mean 0 and covariance matrix V[aa]. In general, the expectation

E[b (k) (a,)] is not equal to b~ ) (Fig. 9.4). In fact, let blk)( 9), b(k).(i), and ~(k),(i) be the ith components of b(k)(-), b(k)(aa), and b(k)(g~), respectively. We observe that

b(~)i) = bl k) ( ~ + Aa~) 3_-- b(~k()i)4-t,rwvaua(i), ~(k) Aa~) + 51'(V2~(k) a o~(i); A a , A a ~ ) + O ( A a , ) 3, (9.121)

9.6.

Second Order Renormalization

E[b~)]

291

b(~)(.) f .

.

.

.

.

.

.

.

.

.

.

.

.

aa

Fig. 9.4. The expectation E[b(k)(a~)] is not equal to ~(k).

Vabl k)(gs). The symbol V a2 ~(k) s(i) denotes (pq) element is c92bIk)(D,`~)/Oapcoaq. Let e be an ap-

where Wva%(i)~(k)is the abbreviation of

the (mm)-matrix whose propriately defined noise level such that A a s - O(e). Taking the expectation of eq. (9.121) and noting that the expectation of the terms of order 3 in A a s vanishes, we obtain

1 (~: ~(k) v [ ~ ] ) + o(~).

(9.122)

Thus, E[b (k) (am)] is not equal to b (k) (as), although the deviation is of order

O(~2).

effective value b*~(k) =

This deviation can be canceled if we define the ( b : ~ ) by

b,(~) s(i) = b(~) s(i) - ~l~v a~b(~) s(i); v[~]),

(9.123)

(k) is the abbreviation of ~2~(k) where V2b a s(i) -avi (as). In fact, if we note that V[a] = O(e 2) and V a2 b(k) s(i) _ V2~(k) a s(i) + O(e), (9.124) we obtain from eqs. (9.119), (9.122), and (9.123) l(w2~(k)

1 v~ ~(k)

Vial])

b(k) v [ ~ ] ) + o(~ 4) (9.125)

Hence, if terms of O(e 3) are ignored, {b~(k) } can be regarded as converted data with the interpretation that the true value b (k) that satisfies the hypothesis (9.116) is perturbed by independent noise Ab~ ) of mean 0 and covariance

292 matrices

Chapter 9.

Renormalization

V[b~),b~)], k, 1 = 1, ..., L. Since E[O(e3)] - O@4), we see that E[b; (k)] - b~) + 0 ( ~ 4 ) .

(9.126)

Following the general theory given in Section 7.1.5, the optimal estimate/t is obtained as the solution of the optimization N

J[u] - E

L

E

W(kO (u)(b*(k), u)(b*(t), u) ~ min,

(9.127)

a = l k ,l= l

(9.128) where r is the rank of the hypothesis (9.116). Suppose u is constrained to be in an n'-dimensional parameter space/4 C 7~n, and l e t / t be the resulting estimate. As discussed in Section 7.1.4, the residual J[/t] is a X2 variable with r N - n' degrees of freedom, where n' is the degree of freedom of u. The covariance matrix o f / t is given in the following form (see eq. (7.57)):

V[/L]-

E

W(kt)(it)(P~b(k))(Pg b~))T

(9.129)

ot=l k,l=l

Here, p U is the n-dimensional projection matrix onto the tangent space T/t(b/) to the manifold b / a t / t .

9.6.2

Second order unbiased estimation

Suppose no constraints are imposed on the n-vector u other than normalization I l u l l - 1. The least-squares approximation to eq. (9.127) is obtained if the functions W~(kl) (u) are replaced by constants W(~kO. Since multiplication of J[u] by a positive constant does not affect the solution, the least-squares approximation to eq. (9.127) can be written in the form J[u] - (u, M * u) ~ min, where M * is the

(9.130)

effective moment matrix defined by 1

M*--~E

N

L

,(k) ,(l)n-

EW(~kOb~

b~

(9.131)

c~=l k,l=l

The solution under the constraint Ilull - 1 is obtained as the unit eigenvector of M * for the smallest eigenvalue. As discussed in Section 9.1, however, the solution of the least-squares approximation is statistically biased whatever weights W (kl) are used.

9.6.

Second Order Renormalization

293

In order to construct an unbiased least-squares approximation, we first consider the moment matrix of the converted data {b~ ) } defined by N

L

1

(kl) b(k) b(1)T

(9.132)

c~----1 k,/----1

L e t / ~ be the unperturbed moment matrix obtained by replacing b~ ) by b~). The hypothesis (9.116) implies that the true value ~ satisfies M ~ - 0, i.e., ft is the unit eigenvector o f / ~ for eigenvalue 0. In general, E[M] is not equal t o / ~ . Suppose we have found (nn)-matrices N (1) and N (2) such that the matrix /~ M - e2N (1) ~- e4N (2) ~- 0(e 6) (9.133) -

has expectation/~/r, where matrices N (1) and N (2) do not involve the noise level e. Then, the unbiased least-squares approximation is obtained in the following form: J[u] - (u, 1Vlu) -+ min. (9.134) The solution under the constraint - x is obtained as the unit eigenvector o f / ~ for the smallest eigenvalue. If the noise level e is not known, we let

f l - M - c N (1) + c 2 N (2) ,

(9.135)

and choose the constant c so that the smallest eigenvalue o f / ~ is 0. Let u be the unit eigenvector o f / ~ for the smallest eigenvalue ~ (7~ 0). If c is incremented by Ac, the m a t r i x / ~ changes into /I;/~ - M - (c + A c ) N (1) + (c + A c ) 2 N (2) -- / ~ Since ( u , / ~ u )

A c ( N (1) -- 2 c N (2)) + A c 2 N (2).

(9.136)

- )~, we see that (9.137)

It follows that we can let (u, fir~u) - 0 by choosing the increment Ac to be Ac

1

2(u,N(2)u) ((u'N(1)u) - 2c(u,N(2)u) _

~/((u,N(i)u)-

2 c ( u , N ( 2 ) u ) ) 2 - 4)~(u,N(2)u)) . (9.138)

The covariance matrix defined by eq. (9.120) generally has the form V[b~ ) , b~ )] - e2Vo(1)[b~), b~ )] + e4Vo(2)[b~),b~ )] q-O(e6),

(9.139)

294

Chapter 9.

Renormalization

where the matrices V0(1)[b (k), b(ak)] and V0(2)[b(ak),b(~k)] do not involve the noise level e. By incorporating the update of the weights W~(kl) , the renormalization procedure is described in the following form: 1. Let c - 0 and W (kl)

-

5ij , k,

l - 1, ..., L, a - 1, ..., N .

2. Compute the matrices M , N (1), and N (2). 3. Compute the smallest eigenvalue )~ of the matrix 2~ - M

-

eN

(1) +

c2N (2)

(9.140)

and the corresponding unit eigenvector u. 4. If )~ ~ 0, return u, c, a n d / ~ . Else, update c and W (kl) as follows: D

--

((u,N(1)u)

_ 2 c ( u , N ( 2 ) u ) ) 2 - 4)~(u, N(2) u),

2 (u, c+-

c+

2(u,N(2)u)

)~

,

i f D > 0, (9.142) ifD 5.

(10.33)

The residual is 0 when N - 5. For N _> 6, we obtain the conic test for image points: the hypothesis that image points {x~ } are on a conic is rejected with significance level a% if 2 J[Q] > XN-5,a(10.34) The theoretical bound on the accuracy of fitting is given by the covariance tensor of Q in the form

(L 1)[Q]-

c~=l

oe )

oe )

)

4(5:c~, QV[x~]Qfc~) + 2(V[x~lQ; QV[x~])

(10.35)

where 79 = (Pijkl) is the projection tensor defined by Pijkl

-- (~ikSjl

--

QijQkl.

( o.a6)

The covariance tensor ])[Q] has rank 5; its null space is {Q}L.

10.2.2

Unbiased estimation and renormalization

Decompose the covariance matrix V[x~] into the noise level e and the normalized covariance matrix V0[x~] in the form V [ x ~ ] - e2V0[x~].

(10.37)

If the denominator in eq. (10.32) is replaced by a constant, we obtain the least-squares approximation J [ Q ] - (Q; fl4*Q)--+ min, 1 In

(10.38)

the denominator, the true values { . ~ } are approximated by the data values {xa }.

10.2.

Conic Fitting

303

where A4* is the effective moment tensor defined by N

1 M* - V ~

w.(~.

o ~ . - v[~o]) o ( ~ . o ~ . - v [ ~ . ] ) .

1

W~

4(x~, Q*Vo[x~]Q* x~) + 2e2(Vo[x~]Q*; Q*Vo[x~]))"

(10.39)

(10.40)

Here, Q* is an appropriate estimate of Q. The minimum is sought under the normalization constraint IIQll = 1. If a 6-vector q and a 66-matrix M * are defined by casting the (33)-matrix Q and the (33)(33)-tensor jVI* in the form q - type6[Q] ,

M * - type6613d*],

(10.41)

we have (Q; Ad*Q) = (q, M ' q ) and IIQ[I = IIq[I (see Section 2.4.2). Hence, the solution of the minimization (10.38) is obtained as the eigenmatrix of norm 1 of tensor A4* (i.e., the matrix Q obtained by cast from the unit eigenvector q of the matrix M ) for the smallest eigenvalue. However, the solution of the least-squares approximation is statistically biased whatever weights W~ are used. Define the moment tensor 1

N

M - -~ E W,~x~ | x~ | x~ | x~,

(10.42)

o~=1

and let M be the unperturbed moment tensor obtained by replacing x~ by 5~. Then, the true value (~ satisfies ,Q(~ - O, i.e., (~ is the eigenmatrix of norm 1 of M for eigenvalue 0. However, E[,M] is not equal to ,M. In fact, _

1

N

E[M,5~,] - -f Z woE[x.(i)xo(j)x.(~)x.(,)] ~=1

1

N

Z W.E[(~.(i) + Ax.(,))(~.(~) + Ax.(j)) c~=1

(~r

1

N

+ Ax~r

+ Ax~t*))]

(

c~--1

+3cc~(i)~(k)E[Axa(j) /kx~(l)] -k Yc~(i)5:~(l)E[Ax~(j) Axa(k)] +Yca(j)YC~(k)E[Axo~(i)Ax~(l)] + x~(j)YCa(l)E[/kx~(i)Axc~(k)] -4- ~a(k)Jca(l)E[Ax~(i)Ax~(j)] -4- E[Ax~(i)Ax~(j)Ax~(k)Ax~(l)])

304

Chapter 10. ~2

Applications of Geometric Estimation

N c~=l ~4

N ~-~1

+ v0[*~l~kV0[~]~ + v0[~l~v0[~l~k). Define (aa)(aa )-tensors N "(1) - t~,N ijkl) (1) and N "(2)

(10.43) (2) ) -(Nijkl

by

N

i(1)

1

( c~--I

+ Vo[Xa]ilXa(j)Xa(k) + Vo[Xa]jkXa(i)Xot(l)

+ Vo[~.lj~/.(~)/.(j) + Vo[~.]k~/.(o/.(~)). 1

(10.44)

N

~ - - ~ Z wo "(Vo[~l,j yo[.o]~, + Vo[~l,kVo[~]j, + yo[~l**Yo[~lj~). ot:l

(10.45) Let sV"(1) be the unperturbed value of N "(1) obtained by replacing x~ by ~ in eq. (10.44). Then, eq. (10.43) can be written in the following form: E[M]

- ./~ -[- ~2 ~ ( 1 ) ..[_ e4/~-(2).

(10.46)

From eqs. (10.44) and (10.45), we immediately see that E[.AS( 1 )] __ ./~r(1) _~_ 2E2./~(2).

(10.47)

It follows that if we define the unbiased moment tensor -- ./~ -- C2Af(1) -~- (~4N'(2) '

(10.48)

we have E[M] - AJt. Hence, we obtain the unbiased least-squares approximation J [ Q ] - (Q; 3~tQ)--+ min. (10.49) The solution under the constraint lIQll- 1 is obtained as the eigenmatrix of 324 of norm 1 for the smallest eigenvalue. If the noise level e is not known, the second order renormalization procedure is given as follows (see Section 9.6.2)" 1. L e t c - 0 a n d W ~ -

1, a -

1,...,N.

2. Compute the (33)(33)-tensors M , N "(1), and Af (2) defined by eqs. (10.42), (10.44), and (10.45), respectively.

10.2.

Conic Fitting

305

3. Compute the smallest eigenvalue )~ of the (33)(33)-tensor

(~0.50)

./~ __ ./~ __ C./~ (1) ..[_ C 2 ./~f(2),

and the corresponding eigenmatrix Q of norm 1. 4. If ~ ,~ 0, return Q, c, and 3~t. Else, update c and W~ as follows: D - ( ( Q ; A f ( 1 ) Q ) - 2c(Q;A/'(2)Q)) 2 - 4s c + (Q; Af(1)Q) - 2c(Q;N'(2)Q)- ~ )~ 2( Q; N'(2)Q)

c +--

Af(2)Q),

,

if D_> O,

(10.51)

(10.52)

ifD N----L-~.

(10.84)

The covariance matrix of the resulting estimate D is obtained in the following form: ~2 ^ V[D]- v(M)~-. (10.85)

10.3.

Space Plane Fitting by Range Sensing

313

Fig. 10.8. Planar surface patch in the scene. Let #max be the largest eigenvalue of V[/,], and ~max the corresponding unit

eigenvector. The primary deviation pair {v +, v - } is given as follows (see Section 4.5.3): V + -- X [ v - t - 4~max~max],

v- - N[i-

V/#max~maxl .

(10.86)

As we pointed out in Section 4.2.1, we must be careful about the scaling of the coordinates when we use the p-representation (X, Y, Z, 1)7-. Namely, we must use an appropriate unit of length for space coordinates so that X, Y, and Z are not too large or too small as compared with 1. A convenient scaling is to regard the average r0 - Ec~N_I Ilr~l[/N as unit length. Once the {v}representation of the fitted space plane is obtained, its {n, d}-representation can be obtained by computing eq. (4.63). The covariance matrix V[fi], the correlation vector V[ti, d], and the variance V[d] are computed from V[/p] by eqs. (4.67). E x a m p l e 10.5 Fig. 10.8 shows a planar surface patch placed in the scene and viewed from the coordinate origin, at which we assume a range finder is fixed. The dots in the figure indicate the orientations in which the distance is measured. We assume that the rays emitted outside the patch return the value oo and are ignored in the fitting computation. We simulated the distance measurement by adding Gaussian random noise to the exact distance to the surface in each orientation independently according to the statistical model given by eq. (10.65) for e = 0.1. Then, a space plane was fitted to them. Let {fi, d} and {~, d} be the true and the computed surface parameters. Since the deviation of ti from fi is orthogonal to ~ to a first approximation, the error in ti can be represented by a 3-vector

Pn(a-

+

d-d d

(10.s7)

where Pft is the projection matrix along ft. The covariance matrix of this

314

Chapter 10.

Applications of Geometric Estimation

oo

e#

9 9

9 9

(~)

(b)

Fig. 10.9. (a) Error distribution for the least-squares solution. (b) Error distribution for renormalization. vector is given by ^

+

J] ~,T] _[..7~,~T.

(lO.8S)

In Figs. 10.9a and 10.9b, A u is plotted in three dimensions for 100 trials, each time using different noise: Fig. 10.9a is for the least-squares solution given by eq. (10.69); Fig. 10.9b is for renormalization. In each figure, the ellipse indicates the standard confidence region computed from eq. (10.88) by using the theoretical expression (10.64) (see Section 4.5.3); the cubes are drawn merely for the sake of reference. We can see that statistical bias exists in the least-squares solution and the bias is removed by renormalization. We can also see from Fig. 10.9b that the theoretical bound on accuracy is almost attained. Fig. 10.10a shows a grid reconstructed by eq. (10.78) for a typical surface fit obtained by renormalization and viewed from an angle (the true position is superimposed in dashed lines); Fig. 10.10b shows its primary deviation pair.

10.4

10.~.1

Space Plane Fitting by Stereo Vision

Optimal space plane fitting

Suppose multiple feature points in the scene are observed by stereo vision, and suppose they are known to be on a planar surface. Although the 3-D structure is uniquely reconstructed by stereo vision alone, we can expect that the reliability of 3-D reconstruction can be enhanced if the knowledge that the feature points are coplanar is incorporated. This problem has practical significance, since many objects in an indoor robotic workspace, such as walls, ceilings, and floors, have planar surfaces. An indirect but simple method is first reconstructing 3-D by stereo vision alone and computing the covariance matrices of the feature points as

10.4.

Space Plane Fitting by Stereo Vision

315

(~)

(b)

Fig. 10.10. (a) An example of a fit computed by renormalization. (b) Its primary deviation pair. described in Section 6.3. Then, a space plane is optimally fitted, as described in Section 7.4.3. Here, we consider a direct method: a space plane is optimally reconstructed directly from stereo correspondence pairs {x~, x ~C~ }, a - 1, " ' ' ~ N. As shown in Section 6.5.1, the necessary and sufficient condition for 5~ and x- a to be projections of a space point on space plane (n, r) - d viewed by a stereo system with motion parameters {h, R} is given as follows (see eqs. (6.67) and (6.71); the scale difference does not affect the result): _,

R

x~ • A h ~ - 0,

A -

T

( hrt T

u

dl)

x/1 + d 2

.

(10.89)

This is the hypothesis from which the surface parameters {n, d} are to be estimated. Define a 34-matrix B ~ and a 4-vector v by

B,~-(

t T T / x,::,,xR hx,:,, x,:,,xR

v -

x/1

1(o) + d2

-d

T x

"

,~ ), (10.90)

Let (~, p) - 0 b~ th~ ~,-repr~s~t~tio~ of the s p ~ p l ~ (n, r) - d. L~t B~ be the u n p e r t u r b e d value of B~ obtained by replacing x~ and x~ by ~ and x~,-' respectively, in the first of eqs. (10.90). Then, the first of eqs. (10.89) can be rewritten as B ~ v - 0. Hence, the problem is stated as follows: Problem

10.4 E s t i m a t e a unit ~-vector v such that /~v

f r o m the data { B ~ } ,

a -

- 0,

1, .... N .

a - i, ..., N,

(10.91)

316

Chapter 10.

Applications of Geometric Estimation

If we let the kth row of the m a t r i x / ) ~ be a ~ )T, eq. (10.91) can be written as ( ~ ) , v) - 0, k - 1, 2, 3, which has the same form as eq. (7.46). Hence, the general theory described in Section 7.1.5 can be applied. Since the true positions 5~ and xa-' satisfy the epipolar equation 15:~,h, -' - - 0 (see eq. (6.4)), vectors ~ , h, and /~x~ -' are coplanar. It n ol -, -, n follows that R 5:~, R T h , and x~ are also coplanar. Hence, vectors x~ x h and x~ - ' x R T ~:~ are collinear, meaning that the m a t r i x / ) ~ has rank 1. Consequently, only one of the three equations ( ~ ) , v) - 0, k - 1, 2, 3, is linearly independent, so the rank of the hypothesis (10.91) appears to be 1. However, the epipolar equation is also an implicit hypothesis, and from the general theory given in Section 5.1.1 it can be shown that the rank 2 of the hypothesis (10.91) is 2. The optimization (7.55) for this problem can be rewritten in the form N

J[v]-

~ ( B ~ v , W~(v)B~v) --+ min,

(10.92)

c~--1

where W ~ ( v ) i s the (33)-matrix given by eq. (7.54). In the present problem, it can be rewritten in the following form: W~(~) - (V[B~,]);. !

Since B a t , - x~ x ABe,

Axa, we have -!

- Ax: x A~

+ x~ x A A x ~ + A x " x AAx~.

-

x'o xAV[x~]A T

(10.94)

V[B~v] - E[(AB~v)(AB~v) T] has the form

Hence, the covariance matrix

Y[B~v]

(10.93)

!

[V[x']xAV[x~]A T],

'

(10.95) where the symbol [. x 9] denotes the exterior product of matrices (see eq. (2.45)). The theoretical lower bound on the covariance matrix of the optimal estim a t e / , is given in the following form (see eq. (7.57))" 12[/,1 -

P~,B~W~(~,)B~Pv

)

.

(10.96)

Here, Pt, is the four-dimensional projection matrix along t,. The rank of V[/,] is 3; its null space is {V}L. Since a space plane has three degrees of freedom, the residual N

J[/,] - ~ ( B ~ / , ,

W~(i,)B~i,)

(10.97)

ct--1 2This hypothesis is degenerate; see Section 7.1.1. We will give a rigorous m a t h e m a t i c a l a r g u m e n t a b o u t the rank of a hypothesis is C h a p t e r 14.

10.4.

Space Plane Fitting by Stereo Vision

317

is a X2 variable with 2 N - 3 degrees of freedom. Evidently, the number N of the pairs of corresponding image points must be such that N > 2.

(10.98)

The residual is not 0 when N - 2. For N >_ 3, we obtain the coplanarity test for feature points" the hypothesis that the observed feature points are projections of coplanar space points is rejected with significance level a% if J[/,]

>

(10.99)

~2N_3, a.

Once the solution t9 is obtained, all the feature points are optimally back projected onto the space plane (/,, p) - 0 by the procedure described in Section 6.5.1.

10.~.2

Unbiasedestimation and renormalization

If the covariance matrices V[x~] and V[x~] are decomposed into the noise level e and the normalized covariance matrices V0[x~] and V0[x~] in the form

V [ x ~ ] - e2Vo[x~],

V [ x ~ ] - eZVo[x~],

(10.100)

the least-squares approximation to the optimization (10.92) can be written in the form a~[v]- (v, M y ) --+ min, (10.101) where M is the (44)-matrix defined by N

i

M-

1 u

T

-N ~~"a:l (h' X'~h)xaxc~ 1

N

~" E c t = l

(Xo~, X o ~ h ) x T

)

1NE : . ot=l

,

"

(10.102)

Here, we have defined the (44)-matrix X~ by

X ~ - R ( z ' ~ x W ~ x x ~,)R

s.

(io.lo3)

The (44)-matrix W~ is defined by wo

-

/

A*

I

+ D

+ e2[Vo[x'~] x A*Vo[x~]A *s] 2 '

(lo.lo4)

where A* is the value of A obtained by replacing n and d by their appropriate estimates rt* and d*, respectively, in the second of eqs. (10.89). The solution of the optimization (10.101) under the constraint I1~11 - 1 is obtained as the unit eigenvector of M for the smallest eigenvalue.

318

Chapter 10.

Applications of Geometric Estimation

It is easy to confirm that

E[Ax~ • W~ • A x e ] - [W~ • V[x~]].

(10.105)

_

Let X~ be the unperturbed value of X~ obtained by replacing x~' b y -x~ ' in eq. (10.103). From eqs. (10.103) and (10.105), we obtain

E[X,] - -X o, + e2Y~,

(10.106)

where we have defined the (44)-matrix Y~ by

Y . - R [ W . • Vo[x~]]R T.

(~o.~o7)

Using eq. (10.106), we obtain the following identities:

El(h, Xo~h)x~x~] - (h, E[Xa]h)E[xo~x~] = (h, (X~ + Y ~ ) h ) ( 5 ~ ~ + e2Vo[x~]) -T e2(h,.~ ~ h) V0[x~]+ e2 (h,Y~h)x~x~ - -T =(h,X~h)Sc,~x~+ +e4 (h, Y~h)Vo[x~], (10.108)

-17E[(h,X~x~)x~] - E[x~x~]E[X,~]h = (~, ~x~ - T + 62 Vo[x~])(X c~ + •2 Y~)h

= (~. X h ) ~ + ~Vo[~]X~h + ~ ( ~ , Y~h)~ +e4 Vo[x ~]Y ~h,

(10.109)

E[(~o, X.~o)] - (E[~o~:]; E[Xo]) = (2~2~ + e2Vo[x~]; . ~ + e2Y~)

= (~.. x . ~ . ) + ~:(yo[~.]; X . ) + ~:(~.. Y.~.) -[-~4(Vo [xa] ; Y~).

(10.110)

Define (44)-matrices/~(1) and N (2) by

N (1) _ ( ~ E N - 1 ((h, io~h))Uo[xo~] + (h, Yo~h)xo~X~)

1

))

~ ~=1

Vo[x~]X~h+ (x~,Y~h)x~ 1 E ~~= I ( (Vo[x,]; X~) + (x~, Y~x~) )

N(2) _

V1 E t =Nl ( h Y , ~h)Vo[x~]

1 E ,N -~ = i (Vo[x~]Y,h) T

IN

; E~=l Vo[x~]Y~h 1 N

~ E~=l(Vo[~o]; Y~)

' (10.111)

) (10.112)

10.4.

Space Plane Fitting by Stereo Vision

319

Let/V(1) be the unperturbed value of N (1) obtained by replacing xa, x , and X,~ by ~,~, x~, -' and X ~ respectively, in eq. (10.111). From eqs. (10.102), (10.10S)-(10.112), we see that _

E [ M ] - M + ~2J~ (1) nt- ~4N(2).

(10.113)

Since the true value of u is the unit eigenvector of M for eigenvalue 0, eq. (10.113) implies that the least-squares solution is statistically unbiased (see Section 9.1.2). From eqs. (10.111) and (10.112), it is easily seen that E [ N (1)]

J~r(1) + 2e2N (2).

(10.114)

/ ~ - M - E2N (1) + e 4 N (2),

(10.115)

-

It follows that if we define

we have E[/I)/] - _]~. Hence, we obtain the unbiased least-squares approximation J[n] - ( v , l ~ v ) ~ min. (10.116) The solution /, is obtained as the unit eigenvector of /~/ for the smallest eigenvalue. If the noise level e is not known, the second order renormalization procedure is given as follows: 1. L e t c - 0 a n d W ~ - I , a -

1,...,N.

2. Compute the (44)-matrices M , N (1), and N (2) defined by eqs. (10.102), (10.111) and (10.112), respectively. 3. Compute the smallest eigenvalue A of the (44)-matrix - M - c N (1) + c2N (2),

(10.117)

and the corresponding unit eigenvector v. 4. If )~ ~ 0, return v - (Ul, u2, ua, u4) 7-, c, a n d / ~ . as follows: D - ((v, N (1) v) - 2c(v, N (2) v)) 2

4A(v, N(2)v),

(v, N(1)v) - 2c(v, N(2)v) - x / ~ , 2(v, N(2)v) A c + (V, N (l)v)' c+

c+-

-

Else, update c and W ~

(10.118)

i f D >_ O, (10.119) i f D < O,

320

Chapter 10. A -

W ~ +-

Applications of Geometric Estimation

RT(h(~'I,

!

x~xAVo[x~]A

T

(m.120)

v2, zJ3) -4- t ' 4 I ) , !

xx~ + (Ax~)xVo[x~]xAx~ _

2 "

+ c[V~176

(10.121)

5. Go back to Step 2. After renormalization, the squared noise level e2 is estimated in the form ~2 ._

C

(10.122)

1 - 3/2N"

Its expectation and variance are 2E 4

E[~21- e2'

V[~21 - 2 N - 3"

(10.123)

If the value e is given a priori, the coplanarity test takes the following form (see eq. (10.99): ~2 2 X2N-a,a (10.124) -g > 2N-----7-~. The covariance matrix of the resulting estimate/, is obtained in the following form: V[i]-

N(M)a.

(10.125)

Let Pmax be the largest eigenvalue of V[/,], and ~max the corresponding unit eigenvector. The primary deviation pair {v +, v - } is given as follows (see Section 4.5.3)"

v + - N[/, + ~ / ~ m a x { m a x ] ,

-

V'max m

]"

(10.126)

Once the v-representation (/,,p) - 0 of the fitted space plane is obtained, its {n,d}-representation (~, r) - a~ is computed by eq. (4.63). The covariance matrix V[/z], the correlation vector V[/z, a~], and the variance V[d] of the parameters {/z, a~} are computed from V[i] in the form of eqs. (4.67) (see Section 4.3.1). E x a m p l e 10.6 Fig. 10.11 shows simulated stereo images (512 x 512 pixels with focal length f = 600 (pixels)) of a planar grid placed in the scene. We added Gaussian noise of mean 0 and standard deviation a = 2 (pixels) to each image coordinate of the grid points independently (so the noise level is e - o / f - 1/300) and fitted a space plane (/z, r) - d. As in Example 10.5, the error in the parameters {/z, a~} is represented by the vector A u given by eq. (10.87), and its covariance matrix is given by eq. (10.88).

10.4.

Space Plane Fitting by Stereo Vision

321

In Fig. 10.12, A u is plotted in three-dimensions for 100 trials, each time using different noise: Fig. 10.12a is for the optimal least-squares approximation (the weights are computed from the true values); Fig. 10.12b is for renormalization. We can see that statistical bias exists in the least-squares solution and the bias is removed by renormalization. The ellipses in the figures indicate the standard confidence regions computed from eq. (10.88) by using the theoretical expression (10.96); the cubes are drawn merely for the sake of reference. We can also see from Fig. 10.12b that the theoretical bound on accuracy is almost attained. Fig. 10.13a shows a reconstructed grid viewed from an angle. The true position is superimposed in dashed lines. Fig. 10.13b shows the primary deviation pairs viewed from a different angle. E x a m p l e 10.7 Fig. 10.14 shows the real stereo images used in Example 6.6 (see Fig. 6.8), where the 3-D shape was computed without assuming any knowledge about the shape of the surface. Here, we incorporate the knowledge that the surface is planar and reconstruct it from the same feature points shown in Fig. 6.9a. Fig. 10.15 shows two views of the surface reconstructed by the procedure described in this section (solid lines) and its primary deviation pair (dashed lines). Comparing this with Fig. 6.9b, we can see that the reliability is indeed increased by the knowledge that the surface is planar.

322

Chapter 10.

Applications of Geometric Estimation

Fig. 10.11. Simulated stereo images of a planar grid.

(~)

(b)

Fig. 10.12. (a) Error distribution for the least-squares approximation. (b) Error distribution for renormalization.

(~)

(b)

Fig. 10.13. (a) Reconstructed grid (solid lines) and its true position (dashed fines). (b) Primary deviation pair of the reconstructed grid.

10.4.

Space Plane Fitting by Stereo Vision

323

Fig. 10.14. Real stereo image (the same as Fig. 6.8).

Fig. 10.15. Reconstructed surface (solid lines) and its primary deviation pair (dashed lines).

Chapter 11 3-D M o t i o n Analysis This chapter presents a statistically optimal algorithm for computing 3-D motion of an object from corresponding image points observed over two views. At the same time, the reliability of the computed motion parameters and the reconstructed depths are evaluated quantitatively. The analysis consists of two separate cases: the case in which the feature points axe in general position in the scene, and the case in which they are known to be coplanax. The statistical properties of the theoretically optimal estimate are described first, providing a theoretical bound on the attainable accuracy. Then, the statistical properties of the solution computed by renormalization and linearization are discussed. We also discuss the critical surface that gives rise to ambiguity of 3-D interpretation. Finally, we formulate a statistical test for testing if the camera motion is a pure rotation or if the object is a planar surface.

11.1

11.1.1

General Theory Camera and object motion

In C h a p t e r 6, we saw how 3-D s t r u c t u r e is recovered from image point correspondences observed by a stereo s y s t e m with known m o t i o n p a r a m e t e r s {h, R } . We now show t h a t the m o t i o n p a r a m e t e r s can also be c o m p u t e d if the correspondence is established for a sufficient n u m b e r of image points. In this chapter, the stereo configuration defined in C h a p t e r 6 is r e i n t e r p r e t e d as a c a m e r a m o t i o n in the scene: the first and second c a m e r a positions of a stereo s y s t e m are identified with the c a m e r a positions before and after the motion, respectively. However, we also use stereo terminologies interchangeably. - ~ their t r u e Let x a and x ~ be corresponding image points, and 5~ and x~ positions. We write x~ -

-

/

x~ + Axe,

x~ -

-/

/

x~ + Axe,

(11.1)

and regard the noise t e r m s A x and A x ~ as i n d e p e n d e n t Gaussian r a n d o m variables of m e a n 0 and covariance m a t r i c e s V[x~] and V[x~], respectively. As discussed in Section 6.1.2, the t r u e positions m u s t satisfy the epipolar equation (6.4). Hence, the p r o b l e m is s t a t e d as follows: Problem

11.1 Estimate the motion parameters {h, R } that satisfy le~, h, n e ' ~ l -

0,

t from the data x~ and x~, c~ - 1, ..., N.

325

~ - 1,..., N,

(11.2)

326

Chapter 11.

...................... ',,

3-D Motion Analysis

-. ...

Fig. 11.1. A large camera motion relative to a large object in the distance is indistinguishable from a small camera motion relative to a small object near the camera. Eq. (11.2) is the hypothesis from which the motion parameters {h, R } are to be estimated. The rank of this hypothesis is 1. It is immediately seen that translation h can be determined only up to scale. This is intuitively interpreted as follows: as long as an image sequence is the only source of information, a large camera motion relative to large objects far away from the camera is indistinguishable from a small camera motion relative to small objects near the camera (Fig. 11.1). In order to remove this indeterminacy, we adopt the scaling Ilhll - 1 if h r 0. It follows that we must first decide whether or not h = O, i.e., whether or not the camera motion is a pure rotation. The procedure for this decision, which we call the r o t a t i o n test, will be discussed in Section 11.7.1. Here, we assume that h r 0 has already been confirmed. In the following, we assume that the camera moves in a stationary scene, but the subsequent analyses can also be applied to a moving object viewed from a stationary camera (Fig. 11.2). If an object rotates around a fixed reference point rG in the object, say its centroid, b y / / o and then translates by ho, a point v in the object moves to r t -- rG

q-

ho + R o ( r -

rG).

(11.3)

Comparing this with eq. (6.1), we find that the resulting image motion is the same as when the object is stationary and the camera moves with motion parameters h - rc - Rro(ho + ra), R - tlor. (11.4) Conversely, the parameters { h o , Ro} of the object motion are computed from the parameters {h, R} of the equivalent camera motion in the following form: ho - R V ( r c -- h ) -

r~,

Ro - R w.

(11.5)

11.1.

General Theory

327

~-~R

FG

Fig. 11.2. Object motion relative to a stationary camera coordinate system.

11.1.2

Optimal estimation of motion parameters

A. Nonlinear optimization For each c~, define the 33-matrix

X ~ - x ~ x ' (~ 7-9

(11.6)

Let X ~ be the unperturbed value of X ~ obtained by replacing x~ and x~ by 5~ and x~,~ respectively. The epipolar equation (11.2) is expressed in the following form: (X~; h x R) - 0. (11.7) This equation has the same form as eq. (10.24) if h x R is identified with Q. Hence, the subsequent analysis takes almost the same form as in Section 10.2.1, except that X a and h x R are no longer symmetric matrices. Since x~ and x~ are independent, we have E [ X a ] - .X Hence, the deviation A X ~ -- X ~ - . ~ is a random variable ofmean O. From eqs. (11.1) and (11.6), we see that

A X ~ - Ax~Sc 'O/ -c + 5 ~ A x ~ 7- + A x ~ A x ~ - C

(11.8)

Let AXa(ij) , xa(i) , and x ~~(i) be the (ij) element of A X ~ and the ith components of x~ and x ~ respectively. The covariance tensor 1;[X~] of X ~ has the following (ijkl) element:

F[X~]~jkl - E[AX~(ij)AX~(kO] = E[Ax~(i)Ax~(k)]-' -t

= V[x~]ikx~(j)

,~,l

-'

'

'

t

I

~(,)+ V[x~]jt2~(i)2~(k)+ V[x~]ikV[x~]jl

.

(11.9)

According to the general theory in Section 7.1.5, the optimal estimate of the motion parameters { h, R} can be obtained by the following minimization (see eq. (7.55))"

J[h,R]-

g E c~----1

(X~; h x R) 2 (h • R; F [ X ~ ] ( h • R)) --+ min.

(11.10)

328

Chapter 11.

3-D Motion Analysis

The minimum is sought under the constraint that h is a unit vector and R is a rotation matrix. If eq. (11.9) is substituted into eq. (11.10), the optimization can be rewritten in the following form: N

J[h, n ] -

Z W(h,

h, n 'l

~ min,

(11.11)

or--1

W~(h,R) - 1 / ( ( h x I:lS:~, V[xa](h x Rfc~)) + (h x 2 ~ , R V [ x ~ ] R T ( h x ~ ) )

+ (V[x~l(h x R); (h x R)V[x~])). In actual computation, the true values ~ data values x~ and x~, ' respectively.

(11.12)

and x~-' are approximated by the

B. Rigidity test

Let {h, R} be the resulting estimate of {h, R}. The motion parameters (h, R} have five degrees of freedom--two for h (unit vector) and three for R (rotation matrix). Hence, the residual J[/z, l~] is a X2 variable with N - 5 degrees of freedom (see Section 7.1.4). It follows that the number N of the pairs of corresponding image points must be such that 1 N ~ 5.

(11.13)

The residual is 0 when N - 5. For N >_ 6, we obtain a rigidity test" the hypothesis that the camera is moving in a stationary scene (or equivalently the object in motion is rigid) is rejected with significance level a% if J[h,

.l~] >

~y-5,a"

(11.14)

C. Focus of expansion

Suppose the camera moves toward or away from the a t h feature point P~ in the scene (Fig. l l.3a). Since h x ~ - 0 and h x R 2 ~ - 0, the first two terms in the denominator on the right-hand side of eq. (11.12) become 0. Consequently, W~ (h, R) becomes very large 2. Geometrically, this is interpreted as follows. The expression (h x R; Y[X~](h x R)) measures how likely the epipolar equation (11.7) is to be violated by noise. The epipolar equation 1Algorithms for computing the motion parameters from five pairs of points are known as five-point algorithms. 2Theoretically, W~(h, R) becomes infinite. The fact that it always remains finite is due to the approximation introduced in the computation.

11.1.

General Theory

329

..

.e

Pc~

..

(~)

(b)

Fig. 11.3. (a) T h e c a m e r a moves toward or away from the a t h feature point. (b) Focus of expansion.

states that the baseline and the two lines of sight that start from the viewpoints O and O' and pass through P~ are coplanar (see eq. (6.2)). However, if the camera moves toward or away from P~, the two lines of sight and the baseline become collinear. The collinearity relation may be destroyed in the presence of noise, but the coplanarity relation still holds to a first approximation. Consequently, P~ is regarded as "robust" to noise, and thereby given a large weight. If the camera orientation does not change (i.e., R = I), the projection of P~ does not move on the image plane; all other feature points seem to move away from or toward it (Fig. ll.3b). Such an image point is known as the focus of expansion. Although this description does not hold if the camera rotates, we still call the projection of P~ the "focus of expansion" if O, O', and P~ are collinear.

11.1.3

Theoretical bound on accuracy

The theoretical bound on the attainable accuracy is obtained by analyzing the behavior of the function J[h, R] near the true values { h , / / } . The translation h is a unit vector, so we can write

h-h+Ah,

(h, Ah) -- 0,

(11.15)

to a first approximation. Since R is a rotation matrix, a small vector 12 exists such that

P,.- R + AP,.,

AP,.- AY2 x R,

(11.16)

to a first approximation (see eqs. (2.57) and (3.31)). Substituting eqs. (11.1), (11.15), and (11.16)into eq. (11.11), we obtain N

J[h,R]

E(W~(h,R)

+ O(Ah, AI2))

Ix~,h, Rx~[ + I~ , A h , - -'

o~:1

+ ] ~ , h, A R x ~ l +

O(Ax~,Ax~,Ah, AR)2) 2

330

Chapter 11.

3-D Motion Analysis

N

=

~

Rx~l ,

-

(5: x Rx~ --,, o

Ah)

oc'-- i

-

((~,~, R- x_,~ ) h

-

( h , R-x- 'o ) ~ , o , ~ x a )

)~

N

+ E W~(ft'R)O(Ax"'Ax~'Ah'At'-2)3'

(11.17)

a=l

where O(.-.)P denotes terms of order p or higher in ---. If we ignore the last term and define vectors a~ -- x~ x R x ~ ,

we obtain

b~ - (e~, Re,'. ) h -

(h, i~5~'. )5~. ,

(

N

J[h,R] - E W,~(ft, R) I~,h,/~21-(&~,Ah)-(b~,Aa)

(11 . lS)

)2

,

(11.19)

(~=1

where we have used the following relationship:

I ~ , h , ~ R ~ I-'

- ( ~ , ( h , R- -~') / ~

-

( h , A ~ ) R- -x'~ )

(11.20)

If we minimize J[h, R] by regarding A h and At? as variables, we need not consider the constraint (h, Ah) -- 0, since the epipolar equation (11.2) implies that (a~, h) - l e o , R e ~ , s = 0, (11.21) _

and hence (as, A h + ch) - (&o,,Ah) for an arbitrary constant c, i.e., the component of A h parallel to h has no effect on the minimization. Differentiating J[h, R] with respect to A h and Ag~ and setting the result 0, we obtain N

N

ct=l

ct=l

(11.22) N

N

c~=l

c~=l

(11.23) If we define the (66)-matrix

A- = (

~-~N EL

ac~

1W~(h,R)b~ -Tas

N

T)

N

-T

(11.24)

eqs. (11.22) and (11.23) are combined into one equation in the form A

,AY2

- E W~(h,/~)[x~, h,/~x~[ o~--1

( )oo ~

.

(11.25)

11.1.

General Theory

331

It is easily seen from eq. (11.21) that the matrix A has rank 5; its null space is {h | 0}L. The solution of eq. (11.25) is obtained in the following form:

= Z wo(h, tZ)l~, h , -R ~ I' , ~

,•

c~=l

--

( ) ac~

~

.

(11.26)

The optimal estimate {h,/~} of the motion parameters {h, R} is given by h - h + Ah,

1 ~ - / ~ + At2 x 1~.

(11.27)

Their covariance matrices are defined as follows (see eq. (3.33))" 17[s -

E[AhAhT],

V[R, s

E[AY2AhT],

1?[/~,/~] - E[AhAI2T], V [ l t / ] - E[AY2AY2T].

(11.28)

From eq. (11.26), we obtain

r

~[h,/~]

N

a,fl--1

=

A

AA

,

-A

(11.29)

where we have used the following relationship:

E[I~, h, R ~ : l - I ~ , h, R ~ I ] - 5~,E[I~, h, R~'I ~] = 5~zE[(X~; h x R)~] - 5~zE[(AX~; h x/~)~] 5~/~ i~)" = &,~(h x i~; v[x,,](h x i ~ ) ) - w~(h,

(11.30)

The covariance matrices f~[h], ~7[h,/~] (= ~7[/t{,h]), and f~[/~] thus obtained give a theoretical bound on the attainable accuracy of estimating {h, R}. In actual computation, eq. (11.29) is approximated by

V[hl V[h,i~] V[&h] _ --

o~=1

E

N, = , o

v[R]

]:

W.(h,i{)(Phd~)(Phh.)T " ., ,, Wo~(h,R)b~(PhD',~)T

)

N N Eo~=, Wo,(h, R)"b~b~^T

(11.31)

where {1%,h} are the estimated motion parameters and

a~-~

xR~,' ~,

go - (~,~ , R' '~ ) h

-

(h,^R x^' ~ ) ~ ~,.

(11.32)

Here, 5~ and 5~'~ are, respectively, the optimally corrected positions of x~ and x~ (see eqs. (6.20)).

332

Chapter 11.

11.2

3-D Motion Analysis

L i n e a r i z a t i o n and R e n o r m a l i z a t i o n

11.2.1

Linearization

The optimization (11.11) is nonlinear, requiring numerical search. However, the hypothesis (11.2) (or equivalently (11.7))is linear in the essential matrix G = h x R (see eq. (6.7)), and the epipolar equation (11.2) is expressed in the form ( ~ , G 2 ~ ) - 0 (see eq. (6.8)). Hence, the linearization technique described in Section 9.5.2 can be applied. To be specific, Problem 11.1 is decomposed into the following two subproblems: P r o b l e m 11.2 Estimate a matrix G such that (~, G~)

- 0,

a - 1, .., N,

(11.33)

from the data x~ and x~, a - 1, ..., N.

P r o b l e m 11.3 Decompose the matrix G into motion parameters {h, R } in such a way that G - h x R. (11.34) Consider Problem 11.2 first. Eq. (11.33) is the hypothesis from which the essential matrix G is to be estimated. The rank of this hypothesis is 1. Since the scale of G is indeterminate, we normalize it so that the resulting translation h is a unit vector. Note the following relationship (see eqs. (2.37) and (2.142))" IIGII 2 - t r

((hx R)(h•

R) T) - t r

= tr(llhll2l - h h T) - 211hll2.

((hx I)RRT(hx

I)T)

(11.35)

Hence, normalizing h to Ilhll- 1 is equivalent to normalizing G to IIGIIv~. If the essential matrix G - h x R is regarded as a variable in eqs. (11.11) and (11.12), it can be optimally estimated by the following optimization"

J[G] - oE~ - - 1 (x~' ' G T V [ x ~ ] G x 'o,, )+(x~ , GY[x~]GTx )+(V[x~]G;GV[x~])

--+ min.

( 11.36)

Let (~ be the resulting estimate. Since G has eight degrees of freedom (the nine elements are normalized), the residual J[(~] is a X2 variable with N - 8 degrees of freedom. It follows that the number N of the pairs of corresponding image points must be such that N > 8.

(11.37)

11.2.

Linearization and Renormalization

333

The residual is 0 when N = 8. In other words, the linearization technique requires three extra pairs of image points a as compared with direct optimization (11.10) (see eq. (ll.la)). If the covariance matrices V[x~] and V[x~] are decomposed into the noise level e and the normalized covariance matrices V0[x~] and V0[x~] in the form

V[xo~]- e2Volxo~],

V[x~]- e2Vo[x~],

(11.38)

the least-squares approximation to (11.36) has the form

J[ G]

-(G;

(11.39)

M G ) -~ min,

where the m o m e n t tensor M is defined by N

1 ~

!

,

(11.40)

M - -~ ~_, W ~ x ~ | x~ | x~ | x~, ~--1

,C

Vo

'

,r

a*Vo[='])).

+

(i .41)

Here, G* is an appropriate estimate of G. The minimum is sought under the normalization [ ] G l l - vr2. If a 9-vector g and a 99-matrix M are defined by casting the 33-matrix G and the 3333-tensor M , respectively, in the form 4 g-

typ%[G],

M-

typ%9[M],

(11.42)

we have (G;lt4G) - (g, M g ) and ] ] G ] [ - ]]g]] (see Section 2.4.2). Hence, the solution of the optimization (11.39) under the constraint lIGII - x/~ is obtained as the eigenmatrix of norm vf2 of tensor M (i.e., the matrix G obtained by cast from the eigenvector g of norm v ~ of the matrix M ) for the smallest eigenvalue. Let fl74 be the unperturbed moment tensor obtained by replacing x~ and x~' by ~ and x~,-' respectively, in eq. (11.40). Eq. (11.33) implies that the true value (~ satisfies M G - O, i.e., (~ is the eigenmatrix fl74 for eigenvalue 0. However, E[M] is generally not equal to M . In fact, _

+ = ~,(i)~,(k)+ E[Ax,(i)Ax,(k)]-

+ Axe(k))] ~,(i)Sc,(k)+ c2Vo[x,]ij,

(11.43)

3Algorithms for computing the motion parameters from eight pairs of points by using the linearization technique are known as eight-point algorithms. 4Here, G is a 33-matrix and M is a 3333-tensor, so they are respectively cast into a 9-vector and a 99-matrix, whereas Q in Section 10.2.2 is a (33)-matrix and M is a (33)(33)tensor, so they are respectively cast into a 6-vector and a 66-matrix; see eq. (10.41).

334

Chapter 11.

3-D Motion Analysis

and similarly -' z ' o~(1)4- 62V0 E[x~(j) x'o~(/)] -- Xo~(j)

(11.44)

'

It follows that N

1

E[Mijk~] - ~ ~ I~E[x.(~)x.(k)]E[x'.(j)x.(t)] c~=1 _

62

N

or--1 64

N

-~--N E

W~176176176

(11.45)

Hence, the solution of the least-squares approximation is statistically biased whatever weights W~ are used.

11.2.2

Unbiased estimation and renormalization

Constructing the unbiased least-squares approximation requires defining an unbiased moment tensor M such that E[3)t] - ~t. Define 3333-tensors 2((1) _ -

-

[ 7V(1)

~,'~'ijkl)and

./~'(2)

{ ~V(2)

- ~ "ijkl] by

N

Ni(1)

=

1

E

(

x'

a=l

'

+

'

) ,

(11.46)

N

Ni(2) i E Wc~Vo[xot]ikVo[x~]jl. jkl --- --N

(11.47)

c~=l

Let .~(1) be the unperturbed value of A: (1) obtained by replacing x~ and x~ by 5~ and x~,-' respectively, in eq. (11.46). Then, eq. (11.45) can be written in the following form: E [ . / ~ ] - ./~ -~- 62./~f (1) -~- 64./~ (2) .

(11.48)

It is immediately seen from eqs. (11.46) and (11.47) that

E[~(1)] -- j~f(') + 2E2N"(2).

(11.49)

It follows that if we define ./~ -- ./~ -- E2./~ (1) ~- 64.]~ (2),

(11.50)

we have E[,M] - ~:4. Hence, we obtain the unbiased least-squares approximation J[G]- (G; A ) / G ) ~ min. (11.51)

11.2.

Linearization and Renormalization

335

The solution under the constraint IIGII - v ~ is obtained as the eigenmatrix of norm ~ of 3~ for the smallest eigenvalue. If the noise level E is not known, the second order renormalization procedure is given as follows (see Section 9.6.2)" 1. L e t c - 0 a n d W ~ -

1, a -

1,...,N.

2. Compute the 3333-tensors M , Af(1), and Af (1) defined by eqs. (11.40), (11.46), and (11.47), respectively. 3. Compute the smallest eigenvalue )~ of the 3333-tensor - - . / ~ - - c N * ( 1 ) _~_ c 2 j ~ f ( 2 ) '

(11.52)

and the corresponding eigenmatrix G of norm v~. 4. If )~ ~ 0, return G, c, and fl;t. Else, update the constant c and the weights 1u as follows"

D - ((a; Az(')a)-

c +-

-

8)~(G; A/"(2)G),

c + (G; N ' ( 1 ) G ) - 2c(G;N'(2)G)- ~ 2~ 2(G; Af(z) G)

,

(11.53)

If D_> 0, IfD 4.

(11.133)

The residual is 0 when N - 4. For N _> 5, we obtain a planarity test" the hypothesis that the feature points are collinear is rejected with significance level a% if

J[~i]

>

~22(N_4),a .

(11.134)

Note that although the general motion algorithm breaks down if all feature points are coplanar, the planar surface algorithm can be applied to general motion as well, always yielding an (incorrect) matrix A. It follows that if the feature points are nearly coplanar, it is safer to apply the planar surface algorithm than the general motion algorithm; it should be switched to the general motion algorithm only when assuming planarity is not compatible with the image data to a convincing degree. In practice, however, the opposite approach has often been adopted: the general motion algorithm is used first by assuming non-planarity and switched to the planar surface algorithm only when assuming non-planarity causes a computational breakdown. To be specific, it has been customary to abandon the general motion algorithm when the second smallest eigenvalue of the moment tensor 3/i given by eq. (11.40) is sufficiently close to its smallest eigenvalue, and the decision criterion has been set arbitrarily. In contrast, the use of the planarity test as described above has a solid statistical foundation. In Chapter 13, we present an alternative criterion which does not require the significance level.

354

Chapter 11.

11.6.2

3-D Motion Analysis

Unbiased estimation and renormalization

The least-squares approximation to the optimization (11.131) based on the normalized covariance matrices V0[x~] and V0[x~] can be written in the following form: J [ A ] - (A; MA)--+ min. (11.135) The moment tensor f14 is defined by N

3

M _ I~r E

E

W

X2 2(g-4),a

(11.152)

We can interpret this as follows: we compare the estimate ~ computed under the hypothesis that the feature points are all coplanar with the noise level e expected from the accuracy of the image processing operations for locating feature points.

11.6.3

Computation of surface and motion parameters

A. Decomposition of the matrix of the image transformation We now solve Problem 11.5. The important fact is that unlike the decomposition of the essential matrix G, no decomposability condition needs to be imposed. In other words, there always exist {n, d} and {h, R} that satisfy eq. (11.125), although the decomposition may not be unique. This is easily understood by counting the degrees of freedom. The matrix A has eight degrees of freedom, since its elements are normalized. The surface parameters {n, d} have three degrees of freedom; the motion parameters {h, R} have five degrees of freedom. Thus, the matrix A has the same degrees of freedom as the surface and motion parameters. In order to solve Problem 11.5, the following two types of camera motion must be distinguished (Fig. 11.12): Case 1 The camera moves on one side of the planar surface; the two images are views from the same side of the surface.

11.6.

3-D Reconstruction from Planar Surface Motion

357

47 (~)

(b)

Fig. 11.12. (a) Case 1: the camera moves on one side of the planar surface. (b) Case 2: the camera penetrates through the planar surface. C a s e 2 The camera penetrates through the planar surface; the second image is a view from the opposite side of the surface. Although the normalization IIAII = i is imposed in the renormalization computation, we rescale A so that det A - 1 Then, the surface and motion parameters {n, d} and {h, R} are computed by the following procedure: 1. Let )~1 >_ )~2 >__A3 (> 0) be the eigenvalues of matrix ATA, and {Ul, u2, u3} the orthonormal set of the corresponding eigenvectors 1~ Let cri = x / ~ (> 0), i = 1, 2, 3. 2. The surface parameters {n, d} in Case 1 are given by

n - ]~[V/Cr2 --O'2Ul-t- V/O"2 --cr2u3],

d = or-------L-2. Or I

~

(11.153)

0- 3

In Case 2, the distance d is replaced by

(11.154)

d -- or---L-2. 0"1 + 0"3

3. The motion parameters {h, R} in Case 1 are given by

(i1.i55) R-

1 (I + o'3ph T)

A T,

(11.156)

o"2 l~ other words, {ui} is the see Section 2.3.1.

left orthonormal system of A

for

singular values {x/~};

358

Chapter 11.

3-D Motion Analysis

Fig. 11.13. A surface behind the camera cast be also observed. and in Case 2 by h--

N[0"3 V/0-12--0"2~ 1 "4-0-1V/0-22- 0"32U3],

R - 1 (-I + a~phT) A m,

(11.157) (11.158)

0"2

where the double signs 4- correspond to that in eqs. (11.153). 4. For each solution, another solution is obtained by changing the signs of n and v simultaneously.

B. G e o m e t r y of ambiguity

The above procedure yields eight solutions in general. This ambiguity is partly explained by noting the following two facts: 9 According to our camera imaging model, a surface behind the camera can be observed as well as a surface in front of the camera 11 (Fig. 11.13). 9 The surface and motion parameters are computed from the matrix A of the image transformation, not individual feature points. Suppose one solution {n, d} and {h, R} is obtained. T y p e 1 If we 1. move the first camera to the other side of the planar surface, and 2. reverse the translation (Fig. 11.14), the new surface and motion parameters are { - n , d} and { - h , R}, respectively. We can see from eq. (11.125) that the matrix A of the image transformation does not change. 11This is not a mere mathematical artifact due to the abstract perspective projection model; this also occurs to a real camera. For example, if the optical axis is oriented upward but not vertical, the horizon can be seen if the field of view is sufficiently large.

11.6.

3-D R e c o n s t r u c t i o n from P l a n a r Surface M o t i o n

,,"

d

"'.,.

359

\ \\

................

m fl

_....~'~tx,~

"

1 -n

~ R

Fig. 11.14. The first camera is moved to the other side of the planar surface, and the camera translation is reversed. Type

2 If we 1. reverse t h e three c o o r d i n a t e axis orientations of the second camera, and 2. move it to the mirror image position with respect to the p l a n a r surface (Fig. 11.15), the new m o t i o n p a r a m e t e r s 12 {h ~, R ~} are given by h' = J n h + 2 d n ,

It'=

-RJn,

(11.159)

where .In is the linear m a p p i n g of the reflection with respect to the plane with surface unit n o r m a l n (Fig. 11.16): it has the expression 9I n = I -

2 n n -r.

(11.160)

T h e m a t r i x of t h e resulting image t r a n s f o r m a t i o n is A ' - l : l ' T ( h ' n -r - d I ) - - F l - r J rn ( ( J n h + 2 d n ) n =

_R, jn (Jnnn"

= -R-r(nn

T -dl)

- -A,

-

-r - d l ) )

-dJn)

(11.161)

where we have used the identities J ~ - .In and j 2 _ I . Thus, t h e two m o t i o n s have the same m a t r i x of image t r a n s f o r m a t i o n up to sign la.

C. Resolution of ambiguity A l t h o u g h eight solutions exist in general, we can reduce the n u m b e r of solutions if we can tell w h e t h e r the m o t i o n belongs to Case 1 or Case 2. In fact, 12The camera orientation obtained after two reflections is described by a (proper) orthogonal matrix. 13Note that multiplication of the matrix A by a nonzero (positive or negative) constant does not affect the 3-D interpretation.

360

Chapter 11.

3-D Motion Analysis

R'

ol ." --....

i

,,'"'""

n l

\

......

~

Fig. 11.15. The second camera is reversed and moved to the mirror image position with respect to the planar surface.

Fig. 11.16. Vectors h and h' are the mirror images of each other with respect to the plane (n,r) - d. if we impose the condition that all the feature points have positive depths before and after the camera motion, the number of solution reduces to at most t w o - - i n most cases one. This is because spurious interpretations are in most cases such that the feature points are all behind the camera or some of them are behind the camera. In the latter case, the vanishing line separates some feature points from the rest, which is impossible because only one side of it is visible (Fig. 11.17; see Section 4.6.2). The true position e~ of the (~th feature point satisfies the following equation (see eq. (11.72)):

(11.162) If this point is on space plane (n, r) - d, the depths 2~ and 2~ can be determined by substituting eq. (11.162)into (n, e~) - d: we obtain

2o -

(~,d~.),

d-(n,h)

2" = ~ : n ~ - )

(11.163)

11.6.

3-D Reconstruction from Planar Surface Motion

361

.---",,, .....

::,:::('::::

......

Fig. 11.17. The vanishing line cannot pass through the projected feature points; only one side of it is visible. Hence, the condition to be imposed is d

(n, ~-) > o,

d - (n, h) > 0.

(11.164)

(~, n ~ - )

If the correct sm~face and motion parameters are chosen, eqs. (11.162) and (11.163) determine the 3-D position of each feature point. However, if the ! data values x~ and x~ are substituted for ~ and x~ -~ in them, eq. (11.162) does not hold exactly. Geometrically, this is equivalent to saying that the two lines of sight defined by x~ and x ~ may not meet; if they do, the intersection may not be on the space plane (n, r) = d. As described in Section 6.5.1, this problem can be resolved by the optimal back projection: the corresponding / image points x~ and x~ are optimally corrected into ~ and x~ ^ so that their lines of sight intersect exactly on the space plane (n, r) = d (see Fig. 6.12). The correction takes the following form (see eqs. (6.74)and (6.75)):

~ ,,/

- x~ -(Y0[x~]A /

v x x'~)W~(x~

x Axe),

/

x~ - x~ + (Vo[x~] x (Ax,~))W,~(x~ x Ax,~). w~

-

( ~'~ • a v o [ ~ , ] a T • ~ ' + ( A ~ )

' • Vo[~o] • (A~)

)-

(11.165) .

(11.166)

This correction is iterated until eq. (11.124) is sufficiently satisfied. Then, the 3-D position of the c~th feature point is given by

r~-

(n,~)

E x a m p l e 11.2 Fig. 11.18 shows two simulated images (512 x 512 pixels) of a planar grid in the scene viewed from a moving camera. The focal length is assumed to be f = 600 (pixels). We added Gaussian noise of mean 0 and standard deviation a = 5 (pixels) to the x and y coordinates of each grid point

362

Chapter 11.

3-D Motion Analysis

Fig. 11.18. Motion images of a planar grid. independently, so the noise level is e - o / f -- 1/120 (this value is not used in the reconstruction computation). We assume that the motion belongs to Case 1. We also assume that the correct solution can be distinguished from the spurious solution. Let {n, d} and {h, R} be the computed surface and motion parameters. Their deviation from the true values {~, d} and {h, R} is measured as follows (see eqs. (10.87), (11.114), and (11.115)): 9 The error in the surface parameters is represented by the 3-vector Au - Pr~(n-

~) + I]h[[d d - d •"

(11.168)

9 The error in translation is represented by the 3-vector A h -- P N [ h ] ( h -

N[h]).

(11.169)

9 The error in rotation is represented by the 3-vector A ~ - AFt/,

(11.170)

where ~ and I are, respectively, the axis and angle of the relative rotation - , (= In Figs. 11.19 and 11.20, A t , Ah, and AY2 are plotted in three dimensions for 100 trials, each time using different noise. Fig. 11.19 is for the least-squares approximation (the weights W(~kl) are computed from the true surface and motion parameters {fi, d} and {h, IR}); Fig. 11.20 is for renormalization. Comparing Figs. 11.19 and 11.20, we can see that the least-squares solution has statistical bias, which is removed by renormalization. Fig. 11.21a shows one example of a grid reconstructed by directly applying eq. (11.167) to x~; Fig. 11.21b shows the corresponding surface reconstructed by the optimal correction of the feature points. In both figures, the true position is superimposed in dashed lines. We can see that the correction enhances the accuracy of 3-D reconstruction.

11.6.

3-D Reconstruction from Planar Surface Motion

9

-~-~

-

(~)

.

363

9

(b)

(~)

Fig. 11.19. Least-squares approximation. (a) Errors in the surface parameters. (b) Errors in translation. (c) Errors in rotation.

/

(~)

(b)

(c)

Fig. 11.20. Renormalization. (a) Errors in the surface parameters. (b) Errors in translation. (c) Errors in rotation.

/

/

(~)

(b)

Fig. 11.21. (a) Simple back projection. (b) Optimal back projection.

364

Chapter 11.

Z'

R

Z

3-D Motion Analysis

p

9

Z'=I

Z=I

~

V

xa

Fig. 11.22. Camera rotation with no translation. 11.7 11.7.1

Camera Rotation

Rotation

and

Information

test

A. R o t a t i o n e s t i m a t i o n

So far we have assumed that h ~ O. If h = O, the camera motion is a pure rotation around the center of the lens, and the incoming rays of light are the same before and after the camera rotation (Fig. 11.22). Consequently, no 3-D information can be obtained. In fact, if no image noise exists, eq. (11.72) implies that the corresponding image points ~ and x-'a are related by Z~

-- Z ' . R ~ ' .

(11.171)

Thus, the depths Z~ and Z~ are completely indeterminate. It follows that in order to do a 3-D motion analysis, we need to test whether h = 0 or not in advance. This r o t a t i o n t e s t can be done in the form of a X2 test by first hypothesizing h = 0 and then testing if the discrepancy of the observation from the hypothesis is small enough to be accounted for by the statistical behavior of the image noise. Eq. (11.171) is equivalent to 9 ~ c< R~'a.

(11.172)

In the presence of noise, the observed image points x~ and x'~, a - 1, ..., N, may not exactly satisfy this condition. So, we consider the following problem: Problem

11.6 E s t i m a t e a r o t a t i o n m a t r i x R s u c h t h a t

-!

x~ x R S ~ -- 0, !

f r o m the d a t a x ~ a n d x ~ , ~ -

1, ..., N .

- 1, ..., N,

(11.173)

11.7.

Camera Rotation and Information

365

Evidently, eq. (11.172) is equivalent to eq. (11.173). This problem is formally identical with Problem 11.4, so the matrix R can be optimally estimated by the following optimization (see eq. (11.131))" N

3

J[R] -- E E W(kl)(R)(x(k); R ) ( X ~ ) ; R ) -+ min.

(11.174)

a=l k,l=l /

w.(R)-

• nv[

:]n T •

+

+[RV[x~]R T x V[x~

•

]])

-

•

(11.175)

2 ~

The only difference from eq. (11.131) is that the minimum is sought under the constraint that R is a rotation matrix. B. X2 test

Let R be the resulting estimate. Since R has three degrees of freedom, the residual J[/~] is a ~2 variable with 2 N - 3 degrees of freedom. It follows that the number N of the pairs of corresponding image points must be such that

(II.176)

N > 2.

However, the residual is not 0 when N = 2. The rotation test can be done in the form of the standard ~2 t e s t : the hypothesis that the camera motion is a pure rotation is rejected with significance level a% if J[R] > X2N_3,,.

(11.177)

Decompose the covariance matrices V[xa] and V[x~] into the noise level e and the normalized covariance matrices V0[xa] and V0[x~l as shown in eqs. (11.38). Let J0[lk] be the normalized residual obtained by replacing V[xa] and V[x~] by V0[xa] and V0[x~], respectively, in the expression for J[IR]. An unbiased estimator of c2 under the hypothesis h - 0 is obtained in the form ~2= [1~______~] J0 2N-3" Its expectation and variance under the hypothesis are E[~2]- e2'

264

V[e2] = 2 N ~ - 3"

(11.178)

(11.179)

In terms of the estimate ~2 the rotation test given by eq. (11.177) takes the following form: ~2 ~2 2N-3,a ~ > 2N------j~. (11.180)

366

Chapter 11.

3-D Motion Analysis

The interpretation is the same as in the case of the planarity test (see eq. (11.152)): we compare the estimate ~ computed under the hypothesis h 0 with the noise level e expected from the accuracy of the image processing operations for locating feature points. C. E s t i m a t i o n o f t h e r e s i d u a l

Since the optimization (11.174) is nonlinear, we need numerical search in the parameter space for R, which is computationally costly. However, an approximately optimal solution can be computed easily. Normalizing both sides of eq. (11.171)into unit vectors and noting that N [ R x ~ ] - R N [ x ~ ] , we obtain

N[~]- RN[x:].

(11.181)

!

If vectors x~ and x~, ~ = 1, ..., N, are normalized into m ~ -- N[x~],

m 's = N [ x ~ ] ,

(11.182)

the rotation R can be estimated by the least-squares optimization 1

J[R]

-

N

K ~

W~llm~ - Rm~ll ~ -~ min,

(11.183)

where W~ are arbitrary positive weights. If we define the correlation m a t r i x N

C -- E

W~mam~7-

(11.184)

o~--1

the optimization (11.183) is equivalent to the following maximization (see

eqs. (2.157)~nd (2 158))" (C; R)--4 max.

(11.185)

C - VAU T

(11.186)

Let

be the singular value decomposition of C (see eq. (2.114)). The solution of the optimization (11.185)is given as follows (see eq. (2.160))" /~ = Vdiag(1, 1, d e t ( V U T ) ) U T.

(11.187)

This may not be the truly optimal solution of (11.174) but can be used to compute a good approximation to the residual, because to a first approximation the function J0[R] is stationary in the neighborhood of/~.

11.7.

11.7.2

C a m e r a Rotation and Information

367

Information in motion images

The rotation test described in the preceding section can also be given the following intuitive interpretation. If I l h l l - 0, no 3-D information can be obtained, and the residual J0[/~] is very small (exactly 0 if noise does not exist). As in the case of stereo vision, the reliability of 3-D reconstruction generally increases as Ilhll increases. If Ilhll increases, the residual J0[/~] also increases. This implies that we can view J0[/~] as the information in the two images. From this viewpoint, we observe the following: 9 The information is m i n i m u m when h - 0; it generally increases as Ilhll increases. 9 If h is the same, the information increases as the variation of the depth becomes larger 14. 9 The information is unchanged if an arbitrary rotation R is added to the camera motionl 5. 9 The information increases as the number N of the feature points increases. In general, the accuracy of 3-D interpretation decreases as the image noise increases. However, the above observation implies that the negative effect of image noise cannot be measured simply by its magnitude: it must be weighed against the information in the two images. In other words, 3-D interpretation can be disrupted by image noise of a very small magnitude if the information is very small, while 3-D interpretation is robust to image noise of a large magnitude if the information is sufficiently large. This viewpoint is further extended in a more general framework in Chapter 13. E x a m p l e 11.3 Fig. 11.23 shows two images of a scene with small depth variance viewed from a translating camera. It is widely known that 3-D interpretation of this type of motion images is very difficult in the presence of noise, because similar images would be observed by a camera rotating around the center of the lens (Fig. 11.24), in which case 3-D interpretation is theoretically impossible. For images such as shown in Fig. 11.23, accurate 3-D interpretation is possible only if the image noise is very small. In other words, the image noise must be small as compared with the information in the images. Thus, the image noise magnitude is a very deceptive measure of the reliability of the 3-D interpretation. 14Humans in motion can perceive 3-D structures from the retinal image motion caused by the difference in depth. Psychologists call this phenomenon motion parallax. 15We a s s u m e that all visible feature points are still visible in the image frame after the rotation.

368

C h a p t e r 11.

3-D M o t i o n Analysis

Fig. 11.23. Motion images of a scene viewed from a translating camera.

Fig. 11.24. Motion images of a scene viewed from a rotating camera.

Chapter 12 3-D Interpretation of Optical Flow In this chapter, we study 3-D interpretation of small (theoretically infinitesimal) image motion called optical flow. We begin with the problem of how to detect optical flow from gray-level images and how to evaluate its reliability. We then turn to 3-D reconstruction from optical flow. The entire analysis is based on the epipolar equation, which is obtained from the epipolar equation for finite motion by taking the limit of infinitesimal motion. The subsequent analysis almost parallels the finite motion analysis given in Chapter 11: we derive a theoretical bound on the attainable accuracy of the motion parameters and present a numerical scheme for 3-D reconstruction by using renormalization and linem'ization. The critical surface of optical flow, the rotation test, and the planarity test are also discussed in the same way as in the case of finite motion.

12.1 12.1.1

Optical Flow D e t e c t i o n Gradient equation

If a smoothly moving camera takes a sequence of images of a stationary object (or equivalently a stationary camera takes a sequence of images of a smoothly moving object), the difference between consecutive image frames is very small and defines interframe displacements of image points called optical flow. As compared with feature matching, which is necessary for 3-D analysis of finite motion, optical flow has the advantage that the flow can be detected densely (usually at each pixel) by a homogeneous image operation. Hence, it is widely expected that optical flow can be used as an i m p o r t a n t source of information for many practical purposes such as real time 3-D interpretation of the environment for robot navigation. The basic assumption for detecting optical flow is that corresponding image points have the same gray level. If the image motion is idealized as continuous, this assumption takes the following form:

Ixu + I~v + It = O.

(12.1)

Here, I and (u, v) are, respectively, the image intensity and the optical flow in position (x, y) at time t. The subscripts x, y, and t denote partial derivatives cg/Ox, O/Oy, and O/Ot, respectively. A differential equation of the form of eq. (12.1) is generally known as the conservation equation, stating that quantity I does not change in the course of motion. Let us call Ix, I v, and

369

370

Chapter 12.

3-D Interpretation of Optical Flow

It the (spatio-temporal) gradient values, and eq. (12.1) the (spatio-temporal) gradient equation. The gradient equation (12.1) alone is insufficient to determine the two unknowns u and v. This indeterminacy can be resolved if the flow is assumed to be constant over a small region in the image. Suppose the flow is constant in an image region that contains N pixels, to which we refer by serial numbers a -- 1, ..., N. We write the gradient values at the a t h pixel as Ixs, Iys, and Its They can be computed by applying a digital filter in the (spatio-temporal) neighborhood of the pixel in question; different computational schemes are obtained depending on what kind of filter is used. Whatever filter is used, however, the computed value is an approximation to the true value. Hence, optical flow detection can be viewed as the following parametric fitting: P r o b l e m 12.1 Estimate the flow components u and v from observed gradient

values {Ixs, Iys, Its} in such a way that their (unknown) true values {Ixs, iys, I--ts} satisfy

L ~ + t~o~ + t~ - o ,

c ~ - 1, ..., N.

(12.2)

Eq. (12.2) is the hypothesis from which the optical flow is to be estimatedi the rank of this hypothesis is 1. Problem 12.1 is formally equivalent to the problem of fitting a space plane u X + v Y + Z - 0 to space points (I~s, Iys, Its), c~ - 1, ..., N, in the X Y Z space. We write the observed gradient values as

I~s - i~s + AI~s,

Ivs - fvs + A i r s ,

Its - its + AI, s,

(12.3)

and regard the noise terms Alxs, Alys, and Airs as Gaussian random variables of means 0, independent for each a. Let their variances and covariances be E [ A I ~ I = ~2 , E [ A I ~ ] - %~2 , E[AI~] - ~,

E[,~L,,~,~Iy~] - %,y~, E[AL,~AI~,~]- ~,~, If we write ~ -

('XO) I~

,

It~

,~-

E[,~I~,~,~I~]- %,~,. (12.4)

~

,

(12.5)

1

the gradient equation (12.2) can be written in the form (~s, u ) - 0 ,

(12.6)

where a s denotes the true value of a s . Vector a s can be interpreted as the space-time gradient of the image intensity I at the a t h pixel in the xyt space time. It is normal to the surface on which the intensity I has a constant value (Fig. 12.1). Vector u can be interpreted as the space-time velocity. It is tangent to the world line in the xyt space time. It follows that the gradient

12.1.

Optical Flow Detection

371

v t

,

. (x, y, t)

stant Y

Fig. 12.1. The image intensity I is constant along the world line in the xyt space time. equation (12.6) (or equivalently eq. (12.2)) states that in the absence of noise

the image intensity I is constant along the world line in the xyt space time. The covariance matrix of vector a~ is given by

(2

O"z c~

V[ a ~ ] -

"Y~ ~/xt(~

"Yz y a 2

"}/z t a

%~

"~t~

"Yytc~

O'2tc~

)

9

(12.7)

As discussed in Section 7.1.1, the optimal estimate of u can be computed by the optimization N (a~, u) 2 J [ u ] - ~ (u~ Yf~]u) ~ min (12.8) under the constraint (k, u) - 1. L e t / t be the resulting estimate. Theoretically, the residual J[/t] is a ~(2 variable of N - 2 degrees of freedom (so, we need at least two pixels), and the hypothesis of the constancy of gray levels in the course of motion can be tested by the standard X2 test, provided the statistical model given by eqs. (12.3) and (12.4) is strictly true, which is very difficult to ascertain for real images.

12.1.2

R e l i a b i l i t y of optical f l o w

According to the general theory in Section 7.1.3, the theoretical covariance matrix of the optimal estimate/t computed by the optimization (12.8) is given by

~--i

(~, V[a~]~t)

'

(12.9)

where ~t is the true value of u. This covariance matrix gives a theoretical bound on the attainable accuracy of optical flow detection. In practice, the

372

Chapter 12. ].

.. },~ n

9

n

9. 9.

.

.i..

9 ."~

" ' : i i . . " ~ '?

5, .

3-D Interpretation of Optical Flow

~,

.

9 " '~

\

..".

9 "\ 9 .\

.

.

: ;i;?": .

,,

9

... 9 . .. 9 .....\

.....'" ...

:.

\ 9 '~\

9

.\\

9 9 ..

".. '!\

9 ?...... L_ .i..3.

Fig. 12.2. The aperture problem: the tangential component is ambiguous when a gray-level image is moving. above expression is approximated by V[/t] -

(Pka~)(Pka~)T (a, v[

(12.10)

16)

Let (~, ~) be the estimated flow, and write VI~ = can be written as 0

)-

0

(Ix~,Iy,~)T 9

Eq.

(12.io)

(12.11)

'

where V[fi, 6] is a (22)-matrix given by --1

-

E cN~ = l W~I2~ Ecr--1

Wo~Yyo~txo~

E cu~ = l

EotN=l Wo~I2o~

,

(i2.i2)

where 1

W~ = o . ~ 2 + 2 7 ~ f i ~ + o ~

2 + 2 ( 7 ~ t ~ + %t,~) + aL"

(12.13)

The inverse matrix in eq. (12.12) may not always exist. It is easily seen that the inverse exists if and only if the rank of {VI~} is two, i.e., the intensity gradient vectors {VI~} are not all parallel. If {VI~} are all parallel, the isointensity contour of the gray level consists of parallel lines, so the motion along the isointensity lines (called tangential flow) is indiscernible; only the motion in the direction of the intensity gradient (called the normal flow) is visible (Fig. 12.2). For such a flow, the covariance matrix diverges to infinity. This anomaly, known as the aperture problem, can be understood easily if we recall the space-time description: since the constraint on the space-time velocity u is its orthogonality to the space-time gradient a~, we need gradient values that have different orientations at least at two pixels (Fig. 12.3).

12.1.

Optical Flow Detection

373

t

t ~ = constant y

/

Fig. 12.3. The space-time velocity u can be determined if and only if the space component of the space-time gradient has different orientations at least at two pixels. The aperture problem always occurs in a region where the image intensity is constant or changes linearly. Hence, the accuracy of the detected flow is generally low. Since the interfi'ame camera motion relative to the object is very small, 3-D interpretation based on optical flow is sensitively affected by image noise. In the following sections, we cope with this difficulty by deriving a theoretically optimal technique for maximizing the accuracy of the 3-D reconstruction. E x a m p l e 12.1 If the noise in the gradient values is isotropic and homogeneous in space and if the time components are independent of the space components, the noise is characterized by two constants o-s and at:

o'~2 _ O.y~2 _ o.~2 a L -

"~u~ --')'~t~ - ' T y t a --O"

(12.14)

Then, the minimization (12.8) reduces to N ( z ~ + I ~ + z,~) 2 ao[u, v ] - E (u 2 + v2 ~ _ - ~ --+ min,

(12.15)

a,=l

where 7 = at/o.s. Eq. (12.12) reduces to N

v[~ ~1- ~ ( ~ + ~ + ~)

N

2

E o:, I ~

'

E~=I I~Iw

E~l~l Iy~Ix~

Ea=l IU~ N

2

)1 "

(12 16) "

An unbiased estimator of o.s2 is obtained in the form ^2 _

J o [ u , ~]

a~

N-2

(12 17) "

Its expectation and variance are given by 2

(12.18)

374

Chapter 12.

3-D Interpretation of Optical Flow

X O)

12

z

Y Fig. 12.4. Instantaneous motion parameters.

12.2

12.2.1

Theoretical Basis of 3-D Interpretation

Optical flow equation

Suppose the camera is smoothly moving in a stationary scene. An instantaneous camera motion is a composition of an instantaneous translation of the center of the lens and an instantaneous rotation around it. Let I (unit vector) be the instantaneous axis of rotation, and w the angular velocity around it. The instantaneous camera motion is specified by the t r a n s l a t i o n v e l o c i t y v and the r o t a t i o n v e l o c i t y ca = w l (see Section 2.1.5); we call {v, w} the (instantaneous) m o t i o n p a r a m e t e r s (Fig. 12.4). Viewed from the camera in motion, a space point r is rotating around the center of the lens with rotation velocity -ca and translating with translation velocity - v . Hence, the velocity of r relative to the camera has the following form (see eq. (2.59)): /~ = - v - ca • r. (12.19) Throughout this chapter, we use dots to denote time derivatives. As in the case of finite motion, an infinitesimal object motion relative to a stationary camera is equivalently treated as an infinitesimal motion of the camera relative to the object. Suppose an object is rotating around a reference point r G arbitrarily fixed in the object, say its centroid, with rotation velocity Wo and translating with translation velocity Vo. The velocity of an arbitrary point r in the object is = v0 +

• (r -

rG).

(12.20)

Comparing this with eq. (12.19), we find that the motion parameters of the equivalent camera motion relative to the object are given as follows (see eqs. (11.4)): v = - V o + Wo x t o , w = -Wo. (12.21) Conversely, the parameters { v o, w o} of the object motion are computed from the parameters {v, w } of the equivalent camera motion in the following form

12.2.

Theoretical Basis of 3-D Interpretation

375

O

Fig. 12.5. Optical flow. (see eqs. (11.5) )" Vo---V-

~o--w.

~ X rG,

Consider an image point x. written in the following form:

(12.22)

The corresponding space point v can be ?-

x-

Since

(12.23)

Z(x)"

(k,x) - 1 for k - (0,0, 1) T, the depth Z(x)is given by Z(x) = (k, r).

(12.24)

Suppose the space point r has velocity/~ relative to the camera (Fig. 12.5). Differentiating eq. (12.23) with respect to time, we obtain

2(~)~ ~ = z(~)~ z(~)

m

z(~)

(k, e)~ Q~e = z ( ~ ) ~ z(~)'

(12.25)

where we have defined Qx - I -

(12.26)

xkT.

Note the following identities:

Qxx-o, Q~ - Q x ,

Qx&-ic,

Pk Qx - Qx,

Qx Pk - Pk"

(12.27)

Substituting eq. (12.19) into eq. (12.25), we obtain the following expression for the image velocity x: -

We call this the optical flow optical flow at x.

-Qx

(v

Z(x) +w

x x

)

.

(12.28)

equation. In the following, we call 5~ simply the

376

Chapter 12.

3-D Interpretation of Optical Flow .... -5:" p

xx)A/

O(t) ~ vat

................... ~ ......... O(t+At)

Fig. 12.6. Geometric interpretation of the epipolar equation.

12.2.2

Epipolar equation for optical flow

Consider an infinitesimal image motion. moves to ~' =

After At seconds, image point x

~ + x~,t + o(At~).

(~2.29)

The motion parameters {h, R} m'e expressed in the following form (see

~q. (2.58)): h = vat + O(At~),

ft = I + ,, • IAt + O(At~).

(12.30)

From eqs. (12.29)and (12.30), we obtain Ix, h, R x ' l = - e ( x ) A t 2 + O(At) 3,

(12.31)

~(~) =

(12.32)

where

I~, x , - I + (~ • ~ , ~ • ~).

Thus, the epipolar equation Ix, h, lZtx~l = 0 reduces to

e(x) = 0.

(12.33)

The epipolar equation Ix, h, Rx'] = 0 states that the camera translation h and the two vectors x and l:tx I, which define the lines of sight of a feature point before and after the camera motion, must be coplanar (see Fig. 6.1). Since eq. (12.33) is obtained as the limit of infinitesimal camera motion, it should describe the condition that the camera translation velocity v, the vector x, which defines the line of sight of a feature point, and the velocity of x must be coplanar (Fig. 12.6). This is easily confirmed if we note that the velocity of vector x relative to a stationary scene is x + v + w • x; the vector x is moving with velocity x relative to the camera, which is translating with translation velocity v and rotating with rotation velocity w. It is immediately seen that eq. (12.32) can be rewritten in the form

e(x) = Ix, ic + v + w • x, vl,

(12.34)

12.2.

Theoretical Basis of 3-D Interpretation

377

stating that three vectors x, x + v + w • x, and v are coplanar. We call eq. (12.33) the epipolar equation for optical flow. The fact that eq. (12.33) is the necessary and sufficient condition for 3-D reconstruction is alternatively confirmed as follows. First, note that the optical flow equation (12.28) can be rewritten in the following form: •

Z(x)"

(12.35)

It follows that for given vectors fi~, w, and v, the depth Z ( x ) that satisfies eq. (12.35) exists if and only if vectors x + Qx(w • x) and Qx v are parallel, i.e.~

/

(5c + Qx(w x x ) ) x Q x v - o.

(12.36)

By definition, 5~ 6 {k}~. The defining equation (12.26) of matrix Qx implies that Qx a 6 {k}L1 for an arbitrary vector a. Hence, the left-hand side of eq. (12.36) has the form ck for some constant c. Since (x, k) = 1, the constant c can be determined by computing the inner product of the left-hand side of eq. (12.36) with x" we obtain

= Ix, ic, v] + (v • x, ,, • x) - e(x).

(12.37)

It follows that eq. (12.36) can be equivalently written as e(x)k = 0, meaning that the depth Z ( x ) exists if and only if e(x) - O.

12.2.3

3-D analysis from optical flow

In the above analysis, we have assumed that no image noise exists. In the presence of noise, the epipolar equation e(x) - 0 does not necessarily hold. Let & be the flow that should be observed in the absence of noise. We write - x + A~,

(12.38)

and regard the noise term A& as a random variable of mean 0 and covariance matrix V[&], independent at each pixel. If the optical flow is detected by the optimization (12.15), the covariance matrix V[&] can be identified with the matrix V[/t] given by eq. (12.11). As in the case of the finite motion analysis, the problem of computing 3-D structures from optical flow is stated as follows" P r o b l e m 12.2 Estimate the motion parameters {v, w} that satisfy •

from the observed flow &.

•

(12.39)

378

Chapter 12.

3-D Interpretation of Optical Flow

Eq. (12.39)is the hypothesis from which the motion parameters {v, w} are to be estimated. The rank of this hypothesis is 1. Since eq. (12.39) is homogeneous in v, the absolute magnitude of the translation velocity v is indeterminate. As in the case of finite motion, this is due to the fact that as long as the image motion is the only source of information, a small camera motion near a small object is indistinguishable from a large camera motion far away from a large object (see Fig. 11.1). The decision criterion (the rotation test) to test whether or not v - O, i.e., whether or not the camera motion is a pure rotation, will be discussed in Section 12.9.2. In the following, we normalize v into [Ivll- 1, assuming that v # 0 has already been confirmed.

12.3

Optimal Estimation of Motion Parameters

12.3.1

Optimal estimation

A. F l o w m a t r i x

Eq. (12.32) can be rewritten in the following form:

= (~,~; ~ • ,) +

(~.~; (~, ~),

-

s[~]).

(12.40)

The symbol S[. ] denotes the symmetrization operator (see eqs. (2.205)). Define 33-matrices X and F by

X - x x n- + d[bcx-r],

F = (~, ~ ) , -

(12.41)

s [ , ~ ~1 + ~ •

(12.42)

,,

where the symbol A[.] denotes the antisymmetrization operator (see eqs. (2.205)). We call F the flow matrix. Let X be the unperturbed value of X obtained by replacing ~ by ~ in eq. (12.41). It is easily seen that eq. (12.39) can be written in the form

( x ; F) - 0.

(12.43)

This equation has the same form as eq. (11.7). Hence, the subsequent analysis takes essentially the same form as in the case of finite motion analysis. Eq. (12.41) is linear in &, so E[X] - X and the deviation A X - X is a random variable of mean O. From eq. (12.41), we see that A X

= A[A~x T]

-

1

(Axx T

_

xAxT )

(12.44)

Let ]2[X] be the covariance tensor of X . It has the following (ijkl) element:

]2[X]ijkt- E[AXijAXkl]1

=

1 E [ ( i ~ g i x j __ x i A ~ j ) ( A ~ , k X -~

l __ XkA~gl) ]

-4(V[ic]ikxjxt- V[x]itxjxk - V[Sc]jkxixt + V[Se]jlxixk).

(12.45)

12.3.

Optimal Estimation of Motion Parameters

379

Since A X is a [33]-matrix, the covariance tensor P[X] is a [33][33J-tensor. According to the general theory in Section 7.1.5, an optimal estimate of the motion parameters {v, w} can be obtained by the following optimization (see eq. (7.55)): g[v, ca] -

f~ ( f ;(X;F) )?[X]F)

d x d y --+ min.

(12.46)

The minimum is sought under the constraint that Ilvll - 1. Here, the integral f s d x d y is a symbolic notation for summation over all the pixels in the region S where the optical flow is defined. From eq. (12.42), we see that (12.47)

A[F] = v x I.

Since 12[X] is a [33][33]-tensor, all (33)-matrices belong to its null space (see Section 2.4.3). Hence, eq. (12.45) implies that (F;)2[X]F) = (S[F] + A[F]; "N[X](S[F] + A[F])) = (A[F]; "P[X]A[F]) (v, (x x V[&] x x ) v ) .

= (x,A[F]TV[&]A[F]x)=

(12.48)

Consequently, the optimization (12.46) can be rewritten in the following form:

J[v,.,]-

J; w(x,v) (Ix, x, v l + ( ~

x x,~ x x

dxdy ~ rain,

1

w(~, ~ ) - (~, (~ • v[,l • ~)~)"

(12.49) (12.50)

E x a m p l e 12.2 If each flow component is perturbed independently by Gaussian noise of mean 0 and variance e2, the covariance matrix of & is V[&] = c : P k. If we define a (33)-matrix S x by 1 Sx - X X Pk X x -

1 --x

--y

x) --y

(12.51)

x 2 + y2

eq. (12.50) can be written in the following form: 1

w(~, ~) - ~(~, s ~ )

(12.52)

B. Rigidity test

Let {~, &} be the optimal estimate of {v, w} determined by the optimization (12.49). The motion parameters {v, w} have five degrees of freedom--two for v (unit vector) and three for w. Hence, the residual J[/J, &] is a X2 variable with S - 5 degrees of freedom, where S is the number of pixels at which

380

Chapter 12.

X

3-D Interpretation of Optical Flow

09

O

z

(~) (b) Fig. 12.7. (a) The image point seen in the direction of the camera translation. (b) Focus of expansion. the optical flow is defined. It follows that optical flow must be observed at least at five pixels. This is in agreement with the case of finite motion, but this is merely a theoretical result. In an actual application, the flow must be observed at a large number of pixels, since it is much more vulnerable to image noise than the point correspondence for finite motion. We can also obtain, at least in principle, a rigidity test" the hypothesis that the camera is moving in a stationary scene (or equivalently the object in motion is rigid) is rejected with significance level a% if

J[V,~.~] > ~2_5, a.

(12.53)

C. Focus of e x p a n s i o n

Let Xv be the image point whose line of sight is in the direction of the translation velocity v (Fig. 12.7a). Since x v c< v, eq. (12.50)implies that W ( x v , v) = c~. It is immediately seen from eq. (12.32) that e(xv) - 0 holds irrespective of the values of x and w. Hence, Xv is a singularity of the optimization (12.49). If the camera is not rotating (i.e., w - 0), the optical flow seems to diverge from or converge to Xv. Such a point is known as the focus of expansion (Fig. 12.7b). Although this description does not hold if the camera is rotating, we still call Xv the focus of expansion. The underlying geometry is the same as in the case of the finite motion analysis in Section 11.1.2 (see Fig. 11.3). In an actual application, a measure must be taken to avoid this singularity. For example, each time a candidate value of v is computed, a neighborhood of the corresponding focus of expansion is excluded or the function W ( x , v) is replaced by a constant in that neighborhood in the next optimization step.

12.3.2

Theoretical bound on accuracy

The theoretical bound on the attainable accuracy is obtained by analyzing the behavior of the function J[v, w] near the true values {0, ~}. Substituting

12.3.

Optimal Estimation of Motion P a r a m e t e r s

381

v - 0 + Av, w - ~ + Aw, and eq. (12.38) into eq. (12.49), we obtain

~(w(~, ~)+ o(~)) (1~, ~, ~1 + I~,~, A~I + (o • ~,~ • ~)

J[v, w]

+ (~x~, • ~,~, • ~ ) + (v • ~, ~x~o • ~ ) + o ( z x e , ~x~,, ~x~o) ~ -

ax@

w(~,o) ~(~)+ (a(~),~x~) + (b(~), ~x~) alas + f W(x, O)O(A[c,Av, Aw)3dxdy, ds

(12.54)

where we have defined

~(~) - I ~ , x, el + (~ • ~, a, • ~),

(~2.55)

a ( ~ ) - ~ • ~ + I1~11=~ - ( ~ , ~ ) ~ , b(~) -II~llZo Ignoring

fs W(x, 0 ) O ( A x ,

Av,

(~, ~)~.

Aw)3dxdy in

(12.56)

eq. (12.54), we obtain (12.57)

Since v is normalized into a unit vector, the deviation Av should be such that (0, Av) - 0 to a first approximation. However, this constraint need not be imposed, because the epipolar equation implies that

(a(~), ~) - (~ • ~ , ~ ) + I 1 ~ 1 1 = ( ~ , ~ ) - ( ~ , ~ ) ( ~ , ~ ) = I~, $, el + (e x ~, a, x ~) - o.

(~2.5s)

Hence, ( a ( x ) , Av + cO) - ( a ( x ) , Av) for an arbitrary constant c. In other words, the component of A v parallel to 0 has no effect on the minimization. Differentiating J[v, w] with respect to Av and Aw and setting the result 0, we obtain

~ w(~, ~) ((a(~), ~.)a(~)+

(~(~), ~ ) a ( ~ ) ) axas

- - f~ w(~, ~,)~(~)~(~)axa~,

(12.59)

f w(~, o) ((a(~), ~.)~(~)+ (~(~), ~)b(~)) alas - - ~ w(~, o)~(~)~(~)axa~.

(12.60)

382

Chapter 12.

3-D Interpretation of Optical Flow

If we define (66)-matrix

~4 -

( ff: W(x, o)a(x)a(x)Tdxdy fs W(x, o)a(x)b(x)Tdxdy ) W(x, o)b(x)a(x)Tdxdy fs W(x, O)b(x)b(x)Tdxdy

, (12.61)

eqs. (12.59) and (12.60) are combined into one equation in the form A(

Av~ ) - - ~ W ( m , ~ ) ~ ( x ) ( a ( x ) A

b(m)

] dxdy.

(12.62)

It is easily seen from eq. (12.58) that the matrix .4 has rank 5; its null space is {0 | 0}L. The solution of eq. (12.62) is obtained in the following form:

~

-

-

w ( ~ , o)~(~)2~-

a(~) ~,(~)

dxa~.

(12.63)

The optimal estimate {0, d~} of the motion parameters {v, w} is given by - o + A~,

d+ - ~ + Acb.

(12.64)

Their covariance matrices are obtained from eq. (12.63) in the following form:

- 2 1 - 2 1 ] - - 21-.

(12.65)

Here, we have used the relationship E [ ~ ( x ) ~ ( x ' ) ] - 5(x - x')E[~(x) ~] - 5 ( x - x ' ) E [ ( X ; t~) ~]

: 5(x - x')E[(AX', ~,)2]

_

5(x -

x')(~'; PIXIe') :

5(Xw(x,-~i) ,

(12.66)

where ~' is the unperturbed flow matrix F obtained by replacing v and w by and ~, respectively, in eq. (12.42). The function 5(x) takes value 1 if x 0 and 0 otherwise. The covariance matrices VIOl, V[/J, r (= V[d~, o]T), and V[cb] thus obtained give a theoretical bound on the attainable accuracy of estimating {v, }. In actual computation, eq. (12.65) is approximated by

v[~,~]

v[~]

fs W(x, i~)b(x)(Podz(x))-rdxdy

fs W(x, i~)b(x)b(x)Tdxdy

' (1~.67)

12.4.

Linearization and Renormalization

383

where {~, &} are the estimated motion parameters and

a(~) - ~ x ~ + II~ll:& - ( ~ ,

&)~,

b(x) - [[x[[2O - (x, 0)x.

(12.68)

Here, x is the optimally corrected flow (this correction will be discussed later).

12.4

12.~. 1

Linearization

and Renormalization

Linearization

The optimization (12.49) is nonlinear, requiring numerical search. However, the hypothesis (12.43) is linear in the flow matrix F. Hence, the linearization technique described in Section 9.5.2 can be applied. To be specific, Problem 12.2 is decomposed into the following two subproblems" P r o b l e m 12.3 Estimate a matrix F such that (X; F ) - 0

(12.69)

from the observed flow &. P r o b l e m 12.4 Decompose the matrix F into motion parameters {v, w} in

such a way that F - (v, w ) Z - S[vw T] + v x *.

(12.70)

Consider Problem 12.3 first. Eq. (12.69) is the hypothesis from which the flow matrix F is to be estimated; the rank of this hypothesis is 1. Eq. (12.47) implies that the translation velocity v is obtained by the following cast (see eqs. (2.182)): v = type3[A[F]]. (12.71) It follows that the normalization malization (see eqs. (2.183))"

Ilvll-

1 is equivalent to the following nor-

IIA[F]II- v~.

(12.72)

If we regard F as a variable in the optimization (12.46) and use eq. (12.45), we see that the flow matrix F can be optimally estimated by J[F] -

f~

(x;F)

(x,A[F]TV[JclA[Flx) dxdy ~ min

(12.73)

under the constraint IIA[F]I]- v/2. Let F be the resulting estimate of F. Since the flow matrix F has eight degrees of freedom (its nine elements are constrained by the normalization (12.72)), the residual J[~'] is a X2 variable

384

Chapter 12.

3-D Interpretation of Optical Flow

with S - 8 degrees of freedom. It follows that the flow matrix can be determined in principle if optical flow is observed at least at eight pixels. This is in agreement with the finite motion analysis, but in an actual application a large number of data are necessary for robust computation, as pointed out earlier. If the covariance matrix V[&] is decomposed into the noise level e and the normalized covariance matrix V0[&] in the form V[&] - e2Vo[&],

(12.74)

the covariance tensor ])[X] is accordingly decomposed in the form FIX]-

e2])0[X].

(12.75)

The least-squares approximation to the optimization (12.73) has the following form: J [ F ] - (f; MR) -+ min. (12.76) Here, the

moment tensor M is defined by 1 ~ W(x)X | Xdxdy,

(12.77)

1

W ( x ) - (x,A[F,]TVo[ic]A[F,]x),

(12.78)

where F* is an appropriate estimate of F. If a 9-vector f and a 99-matrix M are respectively defined by casting the 33-matrix F and the 3333-tensor M in the form f -- type9[F],

M - type99[M],

(12.79)

we have (f; M r ) = (F, MF) (see Section 2.4.2). Hence, the solution of the optimization (12.76) is obtained as the eigenmatrix of tensor M (i.e., the matrix F obtained by cast from the eigenvector f of the matrix M ) for the smallest eigenvalue. The scale of the matrix f is adjusted so that [[AIR][] =

12.4.2

Unbiased estimation and renormalization

Let M be the unperturbed moment tensor obtained by replacing X by X in eq. (12.77). Eq. (12.69) implies that the true flow matrix ~' satisfies ,~7~' = O, i.e., F is the eigenmatrix M for eigenvalue 0. However, the expectation E[uk4] of ,~[ is generally not equal to f14. In fact,

E[M] -

_

~

W(x)E[(X + AX) + (X + AX)]dzdy

12.4.

Linearization and Renormalization

385

_ I_sJfs W ( x ) (.)2 + .X + E [A X | A X ] ) d x d y

6-2 Jfs W(x)'l)o[Xldxdy. = M- + -~

(12.80)

Hence, the solution of the least-squares approximation is statistically biased whatever estimate F* is chosen. Define a [33][33]-tensor iV by

A/-

-~

W ( x ) "l,;o[X l d x d y .

(12.81)

Eq. (12.80) implies that if we define (12.82) we have E[3)t] - 2Q. Hence, we obtain the unbiased least-squares approximation J [ F ] - (F; 3)IF)--+ min. (12.83) The solution is obtained as the eigenmatrix of tensor A)[ for the smallest eigenvalue. The scale of the matrix F is adjusted so that IIA[F]II = V~. If the noise level e is not known, the renormalization procedure is given as follows 1 (see Section 9.4.2): 1. Let c = 0 and W ( x ) = 1. 2. Compute the 3333-tensor 3/[ and the [33][33]-tensor Af defined by eqs. (12.77) and (12.81), respectively.

3. Compute the smallest eigenvalue ~ of the [331133]-tensor -

(12.84)

M-cA;,

and the corresponding eigenmatrix F scaled so that

IIA[FII] = rift.

4. If ,~ ~ 0, return F, c, and 3)t. Else, update the constant c and the function W (x) as follows:

~IIFII 2 c +- c + ( F ; A f F ) '

1

W ( x ) +-- (x,A[F]TVo[x]A[F]x).

(12.85)

5. Go back to Step 2. 1in Section 9.4, r e n o r m a l i z a t i o n was i n t r o d u c e d by a s s u m i n g t h a t no c o n s t r a i n t o t h e r t h a n n o r m a l i z a t i o n existed. It is easy to confirm t h a t t h e s a m e p r o c e d u r e can be applied if t h e n o r m a l i z a t i o n IIA[F]II = v/2 is imposed instead of IIFII = 1.

386

Chapter 12.

3-D Interpretation of Optical Flow

After renormalization, the squared noise level e2 is estimated in the form ~2 _ .

C

( 2.s6)

- s/s

Its expectation and variance are given by 2E4

E[~2] - e2'

V[~2] - S - S"

(12.87)

Since the number S of the pixels at which the optical flow is defined is usually very large 2, the variance V[~ 2] is very small. Hence, ~2 is expected to be a very good estimate of e2. The constraint (12.72) is linearized as follows: (A[F]; A[AF]) - (A[F]; A F ) - 0. Hence, the null space of f onto {A[F]}~ is given by

is

{A[F]}L.

The projection tensor 7) -

Pijkl - 5ik(~jl- 1d[Fij]d[Fkl],

(12.88)

(Pijkt) (12.89)

where A[Fij] - (Fij- Fji)/2 (= the (ij) element of A[F]). Let M - (l~lijkl) be the moment tensor obtained after renormalization. Define a [33][33]-tensor A4'-- (]~[jkl) by A t

Mij a l -

E

PijmnPklPq]~/lmnpq"

(12.90)

m,n,p,q=l

The covariance tensor V[F] of the resulting estimate F is obtained in the following form (see eq. (7.30))"

V[F]12.5 12.5.1

~2

Optimal 3-D Reconstruction Optimal correction and decomposition

A. Decomposability condition We now consider Problem 12.4. Namely, we decompose the flow matrix F into motion parameters {v, w} such that eq. (12.70) holds. As in the case of the essential matrix G for finite motion (see Section 11.3.1), not every matrix 2For example, if optical flow is defined at every pixel of a 512 • 512 image frame, we haves = 512• 512~ 3• l0 s .

12.5.

Optimal 3-D Reconstruction

387

F can be decomposed in the form of eq. (12.70): a (33)-matrix F has the form of eq. (12.70) if and only if 1

K - ~trK(l-

vv

T

)+ 2S[KvvTI,

Ilvll-- x,

(12.92)

where

K = S[F],

v = typea[A[F]].

(12.93)

In fact, if eq. (12.70) holds, it is easy to confirm eqs. (12.92) by direct substitution. Conversely, if eqs. (12.92) hold, it is immediately observed that (v, K v ) = 0.

(12.94)

If we let 1 (trK)v

-

2Kv,

(12.95)

eq. (12.70)can be easily confirmed by direct substitution. We call eqs. (12.92) and (12.93) the decomposability condition for optical flow. We say that a flow matrix F is decomposable if it satisfies the decomposability condition. B. Correction of t h e f l o w matrix The flow matrix F computed by renormalization may not satisfy the decomposability condition (12.92) exactly. Hence, it must be corrected into ~" that satisfies ,/ff - ~ t r K ( l - / ~ / ~

) + 2S[~/('/j/~T]

where = S[~'],

/~ = type3[A[Fl].

(12.97)

Constrained by the normalization IIA[F]II- v~, the flow matrix F has eight degrees of freedom, while the motion parameters {v, w} have five degrees of freedom. Hence, only three of the six 3 component equations of (12.96) are independent,. In other words, the rank of the constraint given by eqs. (12.96) and (12.97) is 3. Substituting F - F - A F into eq. (12.96) and taking a linear approximation, we obtain K-

~1 t r K ( l -

v v T) - 2 S [ K v v T] - t r K S [ A v v T] + 2 S [ K A v v T

+ K v A v T] - A K + 2 t r A g ( l -

v v T) + 2 S [ A g v v T] - O, (12.98)

3Since eq. (12.96) is an equality between (33)-matrices, it gives six element-wise equalities.

388

Chapter 12.

3-D Interpretation of Optical Flow

where A K -- S[AF] and Av -- type3[A[AF]]. If a (33)3-tensor A a (33)(33)-tensor 13 -- (Bijkl), and a (33)-matrix D are defined by 3

1

Aijk -- -~trK(6ikvj + 6jkVi) -- I(ikVj -- I(jkVi - 6jk E

(Aijk), 3

Kilvt - 6ik E

/=1

I(jlvt,

/=1

(12.991 Bijkl - -~l(6ik6jl +6it6jk . 6ij6kt. +6klViVj .

6ikVlVj . . 6jlVkVi

6ilVkVj

6jkVlVi),

(12.1001 D-

K-

ltrK(I-

v v T) - 2 S [ K v v T ] ,

(12.101)

eq. (12.981 can be written in the following form:

AAv + BAK

- D.

(12.102)

If a (33)33-tensor C - (Cijkl) is defined by 1

C -- ~type33133][A ] + B,

(12.1031

or in elements 1

3

Cijkl- --~ E ~SklmAijm+ Bijkl,

(12.104)

m--1

eq. (12.1021 can be rewritten in the following form" CAF-

D.

(12.105)

Hence, the optimal correction A F is determined by minimizing the Mahalanobis distance IIAFIIv[F] (see Section 9.5.2), i.e., (AF, V [ F ] - A F )

~ min,

A F e {A[F]}Ll ,

(12.1061

under the linearized constraint (12.105). The covariance tensor V[F] of the flow matrix F is given as a by-product of the renormalization procedure (see eq. (12.9111. Since multiplication of the covariance tensor V[F] by a positive constant does not affect the solution, it can be replaced by the normalized covariance tensor V0[F] defined by setting ~ = 1 in eq. (12.911. The first order solution is obtained as follows: 3

AFij =

E

Wm"pqCm'klV~

(12.107)

k,l,m,n,p,q--1 Here, "142-- (Wijkt)is a (33)(33)-tensor defined by

W - (V)~-,

(12.108)

12.5.

Optimal 3-D Reconstruction

389

where )2 - (Fijkl) is a (33)(33)-tensor given by

Vijkl --

3 E CijrnnCklpq'12~ m,n,p,q= l

(12.109)

Since the constraint A F e {A[FI} ~ns~r~s the constraint IIA[F]Ib- r only to a first approximation, the actual correction takes the following form: v~(F-

-~ = IIA[F-

AF)

F]II"

(12.110)

This correction is iterated until the matrix D becomes sufficiently close to O. In this process, the normalized covariance tensor l;0[F] needs to be updated, since its null space should change as F changes (see eq. (5.26)). Let 79 = (Pijkl) be the orthogonal projection tensor onto {A[~']}~ (see eq. (12.89)). We let ^

~)o[lP]ijkl - 19ijmnPklpq'l~O[f ]mnpq,

(12.111)

where Vo[F]ijal and 12o[F]ijal are the (ijkl)elements of V0[F] and ))0IF], respectively. C. D e c o m p o s i t i o n into motion parameters

If the corrected flow matrix ~" satisfies the decomposability condition (12.92), the decomposition into motion parameters {v, w} can be done in principle by the second of eqs. (12.93) and eq. (12.95). However, these hold only when ~" is decomposable. As in the case of finite motion, we should use a robust expression that yields a good approximation even if ~" does not satisfy eq. (12.92), say, as a result of a premature termination of the correction iterations described above. We compute the translation velocity v by the second of eqs. (12.97) and determine the rotation velocity w by the least-squares optimization [IS[/v] - ( v , w)I + S[vw-r]ll: -+ min. (12.112) Then, we obtain v - type3[A[/~]],

1

,~ - ~ ( t r F + 3(v,/~v)) v - 2S[F]v.

(12.113)

The second equation is equivalent to eq. (12.95) if F is decomposable.

12.5.2

Optimal correction of op tical flow

A. Epipolar of optical flow

Since the epipolar equation e(x) - 0 is linear in &, it defines a plane in a three-dimensional space if & is regarded as a variable. The constraint (k,&)

390

Chapter 12.

3-D Interpretation of Optical Flow

X Fig. 12.8. Epipolar of optical flow. = 0 also defines a plane. Hence, the set of all vectors s that satisfy the epipolar equation for fixed x, v, and ca is a line: it is the intersection of the plane defined by (k, 5) - 0 with the plane defined by the epipolar equation e ( x ) - 0 (Fig. 12.8). We call this line the epipolar of the optical flow ~b at x. From eqs. (12.36) and (12.37), we see that the equation of the epipolar is x Qxv - -(v

x x,

x x)k.

(12.114)

The orientation of this line is Q x v, and its distance d from the origin O is d-

I(v • x , , , , •

(12.115)

B. C o r r e c t i o n of optical flow

Let {/~, r be the motion parameters computed from optical flow. Although they are estimated so that the flow satisfies the epipolar equation e ( x ) = 0 on the average, individual flow components may not necessary satisfy it. In order that the depth can be defined at each point, the observed flow 5~ must be corrected into the value ~ that is compatible with the epipolar equation for the computed motion parameters {~3, r This correction has the following geometric interpretations (they are equivalent to each other)" 9 The vector & is moved to ~ so that ~ is on the epipolar at x (see Fig. 12.9). 9 The vector x is moved to x so that the lines o/sight o f x ( t ) and x ( t + A t ) meet at a point in the scene to a first approximation (see Fig. 12.6). If we let

~(x) = Ix, x, 'b I + (0 x x , ~ x x),

(12.116)

this is not zero in general. It is easy to see that the right-hand side vanishes if & is replaced by ~ - & - A& in such a way that (~3 x x, A:b) = ~(x).

(12.117)

12.5.

Optimal 3-D Reconstruction

391

Ax

X

Fig. 12.9. Optical correction of optical flow. The rank of this constraint is 1. An optimal value of A5~ is determined by minimizing the Mahalanobis distance IIAS~lly0[x], i.e., (A&, t ~ [ x ] - A x ) ~ min,

•

Aic e {k}L,

(12.118)

under the constraint (12.117). The solution is given by

Ax-

~(x)Vo [&] (O x x) (6 • ~, Vo[x](6 • ~))"

(12.119)

It follows that the correction has the form .~

-z-

~(x)V0[~](~? x x)

(~, (~ x v0[x] x ~)~)"

(12.120)

Since the epipolar equation e(x) - 0 is linear in ~b, no approximation has been introduced into the constraint (12.117). Hence, the above correction need not be iterated. The residual (A~, V0[:b]-A~) of the optimization (12.118) is given by

~(~)~ e(x)2 = (~?, (x x Vo[x] x x)/J)"

(12.121)

From eqs. (12.49) and (12.50), we can see that J0[0, do] - Is ~(x)2dxdy

(12.122)

equals e2 times the residual of the optimization (12.49). Hence, J0[/~, ~b]/e 2 is a ~(2 variable with S - 5 degrees of freedom. An unbiased estimator of the squared noise level e2 is obtained in the form g2= Jo [/~, dJ]. S-5

(12.123)

392

Chapter 12.

3-D Interpretation of Optical Flow

Its expectation and variance are E[~2]_ e2

v[~]-

2E4 s- 5

(12.124)

The rigidity test (12.53) can be rewritten as

12.5.3

~2

X2_5,a

-e7 >

S-5

(12.125)

Computation of depth

A. Depth equation From eq. (12.119), we see that (Q~O V o [ ~ ] - ~ x ) - ~(~)(Q~o' Vo[x]-Vo[x](o • ~)) ' (~ • ~, Vo[x](o • ~)) ~(x)(Q~,,pk(~, • ~)) ~(~)(PkQ~,, ~, • ~) (~ • ~, Vo[~](~ • ~)) (~ • ~, Uo[~](~ • ~)) _

~ ( x ) ( Q x i ~ , i~ x x )

_ ~(x)(iJ-

(~ • ~, Vo[~](o • ~))

( i ~ , k ) x , iJ x x ) - O ,

(12.126)

(~ • ~, Vo[~](o • ~))

where we have used eqs. (12.26) and (12.27). If the corrected flow x - 5~- AS~ satisfies the epipolar equation for the computed motion parameters {0, r there exists a depth Z(x) such that

_ A~-

_Q~

0 + d~ x x ) . 2(~)

(12.127)

Noting the second of eqs. (12.27), we can rewrite the above equation in the following form: ~-Q~

(

~+~•

@ ) ( 1 2 . 1 2 8 ) "

Eq. (12.126) implies (Q~o, v0[~]-Q~

~ + ~ x ~ + 2(~)

) - 0,

(12.129)

from which the depth Z(x) is given in the following form: 2(~1 - -

(~ , Q~~ v0[~l - Q ~ )

(iJ, QTx Vo[&,]-Qx(SC + go x x ) )

We call this the depth equation. We observe the following:

(12 ~30)

12.5.

Optimal 3-D Reconstruction

393

9 Eq. (12.28) implies that Qx(& + d~ x x) = 0 if ~5 = O. Hence, the depth Z(x) is indeterminate if ~5 - O. 9 Let x 0 (<x /J) be the focus of expansion for the computed motion parameters {~, r (see Section 12.3.1). Since Qx~O ~ Qx~xi~ = 0 (see the first of eqs. (12.27)), the depth at the focus of expansion xo is indeterminate. Up to now, there exist two solutions, since renormalization computes the flow matrix F only up to sign. It is easily seen that if one solution is {v, w} and Z(x), the other solution is { - v , w} and - Z ( x ) . Hence, the correct solution is chosen by imposing the constraint

~

sgn(2(x))dxdy > O.

(12.131)

As in the case of finite motion (see eq. (11.94)), the use of a seemingly equivalent condition fs Z(x)dxdy > 0 is dangerous because a large depth (Z(x) oo) can be estimated to be very far behind the camera (Z(x) ~ - o o ) in the presence of image noise. B. Reinterpretation of optimal estimation

Eq. (12.128) implies that

(zx~, y0[~]-~x) = (~ + ~, x ~ + 2(~)' Q g V o [ ~ I - Q ~

~ + ~, x ~ +

2(~) ). (12.132)

If eq. (12.130) is substituted into the right-hand side of this equation, it should be equal to the right-hand side of eq. (12.121). On the other hand, it is easily seen that the depth Z(x) given by eq. (12.130) minimizes the right-hand side of eq. (12.132) viewed as a function of Z(x). Thus, we conclude that

~(.): (~. (.

x

Vo[x] x .)~)

= min ('~ +

z(x)

Z(x)

+ & x x, Q~Vo[x]-Qx

(

)

x + z ( x ) + a, • x ),

(12.133)

and the minimum is attained by the depth Z(x) given by eq. (12.130). It follows that the optimization (12.122) is equivalent to f

Jo[v .,]- I '

min

Js z(x)

& + Qx

(v

Z(x) + w x x

)lq2

v0[x]

dxdy --+ min.

(12.134)

394

Chapter 12.

3-D Interpretation of Optical Flow

E x a m p l e 12.3 If each flow component is perturbed independently by Gaussian noise of mean 0 and variance e2, the normalized covariance matrix of is V0[x] - Pk" From the fourth of eqs. (12.27), we see that

Q~ Pk Q~ - Q~ Q~ - S~ ,

(12.135)

where the matrix Sx is defined by eq. (12.51). Hence, eq. (12.130) can be written as

2(~)--(0,s~(~

(,5, Sx+)

(12 136)

+ ~ • ~1 ) 9

Eq. (12.133) implies the following identity:

min

z(~)

II (~ ~ +

Qx

Z(x)

+ d~ x x

)]1

=

.

(e, s ~ )

(12.137)

The minimum is attained by the depth Z(x) given by eq. (12.136). It follows that the optimization (12.134) with respect to the Mahalanobis distance reduces to the following optimization with respect to the Euclidean distance:

Jo[v' w] - fs z(~) min ll~ + Qx ( z (vx ) + w x x -- JfS e(x)2dxdy (~:~~ . m i n 12.6

12.6.1

)112

dxdy (12.138)

Reliability of 3-D Reconstruction

Effect of image noise

Errors in the 3-D shape reconstructed by the depth equation (12.130) originate from the following two sources: 1. errors in the observed flow 5; 2. errors in the computed motion parameters {/J, &}. Strictly speaking, these two sources of error are correlated, since {/~, d~} are computed from the flow 5~. However, since {/~, r are estimated by optimization over all the pixels at which the flow is defined, the correlation between {~, r and ~ at a particular pixel is expected to be very small. Hence, as in the case of finite motion, errors in 3-D reconstruction can be treated to a first approximation as the sum of the errors from these two sources. First, consider the effect of image noise, assuming that the motion parameters {/~, r are accurate. If the observed flow 5~ is corrected into

12.6.

Reliability of 3-D Reconstruction

395

by eq. (12.120), its a posteriori covariance matrix is given as follows (see eq. (5.46))"

t~[~] -

~ (Vo[x]-

\

(Vo[x](,:, x ~))(Vo[~](~, • ~))' '~ (~ x x, Vo[x](/~ x x)) / "

(12.139)

Here, ~2 is the value estimated by eq. (12.123). The subscript i indicates that we are considering the effect of image noise only. It i.s easily seen that the above matrix has rank 1; its range and null space are {(x • 6) • k } L = { Q x O } L and {x x/~, k } L , respectively (see eq. (12.26)). This is a consequence of the fact that errors in x are constrained to be along the epipolar at x (see Fig. 12.9). It follows that the covariance matrix Viii] has the following form: Vi[x] - constant x (Qxi~)(QxO) T.

(12.140)

Recall that the depth Z(x) has been computed from the following relationship (see eq. (12.127)): [r, - - Q x

Z(x) + ~ x x

.

(12.141)

Since the corrected flow ~ is computed from the observed flow, it is a random variable. If it is perturbed by A~, the depth Z(x) is accordingly perturbed by AZ(x). To a first approximation, we have

Ax - AZ(x)= QxO.

(12.142)

z(.)~

Hence, trVi[~]

-

E[IIA~II 2] - E[AZ(x)2] (~ Q~Q~e) = V~[2(~)] (~, S ~ ) , 2(x)4 ' 2(x)4

(~2.143) where the matrix Sx is defined by eq. (12.51) (see eqs. (12.135)). Consequently, the variance Y~[2(x)] of the computed depth Z(x) is given by V~[2(x)]- 2(x)4trV~[~]

12.6.2

(12.144)

Effect of errors in the motion parameters

Consider the effect of errors in the motion parameters {/~, d~}, assuming that the f l o w x i s accurate. Let / ~ - ~ + A v a n d c b - ~ - ~ + A w . Since the epipolar equation e ( x ) - 0 is satisfied by the true values {0, ~}, we have to a first approximation

~(~) - I~, x, ~ +/',vl + ((~ + i v ) x ~, (,x, + A ~ ) x ~) = I', ~, ,~"1 + ( ~ " x . , ~ x . ) + = (a(~), ,~,,) + (/,(~), ~,.,.,),

(~ x . , ~ . , x . ) (12.145)

Chapter 12.

396

3-D Interpretation of Optical Flow

where 4

a(~) - ~ • x + I1~11~, - ( ~ , ~)~, b(x) -I]x[[2~) - (x, 9)x.

(12.146)

From eq. (12.145), we obtain the following relationships (the subscript m indicates that we are considering errors in the motion parameters only)"

v~[~(.)] - ( a ( . ) . v[~]a(~)) + 2(a(~). v[~. ~ ] b ( . ) ) + (b(.). v [ ~ ] b ( . ) ) .

v~[~. ~ ( . ) ] - E[~(~)Zx~]- V[~]a(.)+ V[~. ~]b(.). Vm [~b, (~(x)]-

E[8(x)Aw]- V[~, O]8(x)+ V[~b]b(x).

(12.147)

The covariance matrices V[~], V[/J, ~b], and V[&] are computed by eq. (12.67). In computing eq. (12.67), we need the value of e2 because W(x, iJ) involves the covariance matrix V[&] = e2V0[x] (see eq. (12.50)). It is estimated by eq. (12.123). If the motion parameters are not accurate, the flow x is corrected by eq. (12.120) even though x itself is accurate. From eq. (12.119), the a posteriori covariance matrix Vm[~] is obtained to a first approximation in the following form:

v~[~]- v~[~(~)](v0[x](~ • .))(v0[~](~ • .))~ (+ •

(12.148)

v0[~](~ • ~))~

We also have

v~[~. ~] - v~[~. ~(.)](v0[~](~ • .)) 9 (~ • .. v0[~](~ • . ) ) ' v~[~. ~ ] - v~[~,. ~(.)](v0[~](~ • .))~ (~3 x x, V0[~b](~3 x x))

"

(12.149)

From eq. (12.141), we see that the perturbations Av, Aw, and A~ are related to the perturbation A Z ( x ) to a first approximation in the form

~

- -q~

2(~)

2(~)~

+ zx~ x ~

.

(12.150)

from which we have

4Strictly speaking, the motion parameters {~,, &} in eqs. (12.146) should be {9, &}, but this approximation introduces only a second order difference.

12.6.

Reliability of 3-D Reconstruction

397

Fig. 12.10. Simulated optical flow. Taking the expectation of the squared norm on both sides, we obtain

Vm[2(~ )] llQ~ ~ll ~ 2(x) 4

-E[

I1( Qx

A~+

~.

^

z(~)

)ll2

+Amxx

(12.152)

],

which can be rewritten as

Vm[2(~)l(i,, Q~Q~ i,) 2(~)~ T

=

(E[ a ~ + 2(~) + ~

• ~

~

+ 2(~)

+~

•

];Q~Q~). (12.153)

From this, the variance Vm[Z(x)] of the depth Z(x) is obtained in the following form:

2(~) ~ ( (V[&I;S~)+ (v[~]; Vm[2(~)]- (~,S~) 2(~)s~) + (~ x v[~] x ~; s~) 2(Vm[/~, x]; S x ) _ 2(x x Vm[&, x]" S x ) - 2(x x V[&,/J]; S x ) ) .

(12.154)

E x a m p l e 12.4 Fig. 12.10 shows a simulated image (512 x 512 pixels with focal length f = 600 (pixels)) of three planar grids in the scene viewed from a moving camera. We synthesized optical flow at the grid points according to eq. (12.28) and added Gaussian noise of mean 0 and standard deviation a = 1 (pixels) to the x and y flow components at each point independently. Hence, the noise level is ~ = ~ / f = 1/600 (this value is not used in the reconstruction computation). Let {~, aJ} and {/J, r be the true and the computed motion

398

Chapter 12.

3-D Interpretation of Optical Flow

....-~ 9

1~ (b)

(~)

Fig. 12.11. Least-squares approximation with optimal weights: (a) errors in translation; (b) errors in rotation.

"; "'~"

IL I ",d

I

(~)

(b)

Fig. 12.12. Renormalization: (a) errors in translation; (b) errors in rotation. parameters, respectively. Since v is a unit vector, the error in translation is measured by Av = PN[v](~) - N[0]). (12.155) The error in rotation is measured by

A~ - c, - ~.

(12.156)

In Figs. 12.11-12.13, vectors Av and Aw are plotted in three dimensions for 100 trials, each time using different noise. The ellipses in these figures indicate the standard confidence regions 5 defined by V[~] and V[d~] (see eq. (12.65)); the cubes are drawn merely for the sake of reference. Fig. 12.11 is for the least-squares approximation with optimal weights (computed from the true motion parameters {0, 3}); Fig. 12.12 is for renormalization; Fig. 12.13 is for renormalization followed by optimal correction of the flow matrix. The corresponding standard confidence regions and reference cubes have the same absolute sizes. From Fig. 12.13, we can see that the accuracy almost attains the theoretical bound given by eq. (12.65) if renormalization and the optimal correction are combined. Fig. 12.14 shows a reconstructed shape viewed from a different angle. The true shape is superimposed in dashed lines. Fig. 12.15a shows the standard confidence regions, which degenerate into line segments. They are computed from the total variance Vt[Z(x)] - Y~[2(x)] + Vm[2(x)] around the reconstructed grid points: Vi[Z(x)] is the variance due to the image noise (see 5Since l?[~,] has rank 2 and ~r[&] has rank 3, the standard confidence region is a space conic for the translation velocity and an ellipsoid for the rotation velocity; see Section 4.5.3.

12.6.

Reliability of 3-D Reconstruction

(~)

399

(b)

Fig. 12.13. Renormalization followed by optimal correction: (a) errors in translation; (b) errors in rotation.

Fig. 12.14. Reconstructed shape. The true shape is superimposed in dashed lines.

(~)

(b)

Fig. 12.15. (a) Reconstructed shape and the standard confidence regions of grid points. (b) Two shapes corresponding to the primary deviation pairs of grid points.

400

Chapter 12.

3-D Interpretation of Optical Flow

eq. (12.144)); Vm[Z(x)] is the variance due to errors in the computed motion parameters (see eq. (12.154)). Fig. 12.15b shows the shapes that envelop the primary deviation pairs of the grid points.

12.7

Critical Surfaces for Optical flow Weak critical surfaces

12.7.1

As in the case of finite motion, the linearization technique described in Section 12.4.1 is based on the fact that the flow matrix F is an eigenmatrix of the unperturbed moment tensor A74 for eigenvalue 0, which we have implicitly assumed to be a simple root. If it is a multiple root, infinitely many eigenmatrices exist. This occurs if and only if the epipolar equation (12.69) is satisfied by a false flow matrix F ( r F). In other words, ambiguity occurs if and only if the object surface r - Z ( 2 ) 2 is such that (JE; ~') - 0

(12.157)

for F r F . For such a surface, the linearization technique in Section 12.4.1 does not work, since any linear combination of F and F is an eigenmatrix of A//for eigenvalue 0. If we put ~ - type3[A[~']], we can write A[~'] - ~ • I. It follows that

(-~; ~') -- (X= T 4- A[x=T]; S[~'] 1L A[F]) - (xxT; S[~']) 4- (~=T; ~ X I ) = ( x , S [ F l x ) + (&, iJ x x).

(12.158)

Substituting eq. (12.28) into this and replacing x by r / Z ( x ) , we obtain

( x ; F) - ( . . S [ P ] . ) - ( Q .

=

(v

z(.) + ~ x ~

)

. ~ x .)

(~.s[/~]~) _ ( ~ ~, x ~ Q=~(~ x ~)1 z(~)~ z ( ~ ) + z(~----?' z(~)

(~. s[Pl~) - (~. ~ • ~1 - (~. ((~. ~), - s [ ~ ] ) z(.)~

~)

(12.159)

where we have used the identity QxT(9 x x) -- ~ • x. Hence, eq. (12.157) holds if and only if the object surface has the form

(r, K ' r ) - (v • ~, r),

(12.160)

where

K'-

S I R ] - (~, ~)* + S [ ~ ] .

(12.161)

12.7.

Critical Surfaces for Optical flow

401

Since eq. (12.160) is identically satisfied by r = 0, the origin 0 of the camera coordinate system is on this surface. If K ~ is nonsingular, eq. (12.160) defines a centered quadric of the following form (see eq. (4.98)):

(r - rc, S ( r -

r e ) ) = 1.

(12.162)

The center r c and the matrix S are respectively given by rc-

12 K ' - l (v • v)

S-

4K I (v x O , K ' - l ( v x ~))"

(12.163)

We call a surface whose equation has the form of eq. (12.160) for some (33)matrix K ~ a weak critical surface.

12. 7.2

S t r o n g critical surfaces

If the object surface is included in a weak critical surface, the epipolar equation (12.69) is satisfied by infinitely many false flow matrices. However, since the true flow matrix F must be decomposable, the correct flow matrix F can be chosen if the false flow matrices are not decomposable. In fact, the correct motion parameters {v, w} can be obtained by the direct optimization (12.49). However, if the epipolar equation (12.69) is satisfied by a false flow matrix that is decomposable, the problem is inherently ambiguous. This occurs if and only if the object surface has the form of eq. (12.160) for which the matrix F in eq. (12.161) can be decomposed into false motion parameters {~, 5~}. This condition is equivalent to saying that there exist vectors ~ (~ v) and ~ (:/: w) such that K ' = (9, r - w ) I - S[~(r - w)T]. (12.164) Let us call a surface that satisfies this condition a strong critical surface. By definition, a strong critical surface is also a weak critical surface and hence a quadric that passes through the origin O of the camera coordinate system. However, the converse does not necessarily hold. In fact, eq. (12.164) implies

= 0,

(

2.165)

and hence eq. (12.160) is identically satisfied by r = c9 for an arbitrary constant c. In other words, the space line r x ~ - 0 is entirely contained in that surface. This implies that a strong critical surface is a hyperboloid of one sheet or its degeneracy (Fig. 12.16a). As in the case of finite motion, all planar surfaces are strong critical surfaces and hence weak critical surfaces as well: an arbitrary space plane H that does not pass through the origin O can be coupled with another space plane H ~ that passes through O and intersects with H perpendicularly in such a way that II and H ~ constitute a degenerate hyperboloid (Fig. 12.16b). Consequently, the 3-D interpretation of a planar surface is inherently ambiguous. In

402

Chapter 12.

3-D Interpretation of Optical Flow

H

O

(~)

(b)

Fig. 12.16. (a) Hyperboloid of one sheet. (b) Two planar surfaces as a degenerate hyperboloid. the following, we construct an algorithm for computing all possible interpretations of a planar surface scene. In order to invoke this algorithm, however, we must first test if the object surface we are observing is a planar surface. This planarity test is derived from a statistical analysis of the image noise.

12.8 12.8.1

Analysis of Planar Surface Optical Flow Optical flow equation for a planar surface

If space plane (n, r) = d is viewed from a camera moving with motion parameters {h, R}, the projected image motion has the following form (see

~qs. (6.66)~d (6.67)): x' =

Ax (k, A x ) '

A - RT(hn T -dI).

(12.166)

If the camera is stationary, we have h - 0 and R - I, so A = - d I . It follows that for an infinitesimal camera motion the matrix A has the form

A = -~(Z + WAt) + O(~t~).

(12.167)

Substitution of this into the first of eqs. (12.166) yields

x + W x A t + O(At 2)

, X

-"

=

(k, =)+ (k, W~)At + O(~t~) +

+ O(At2).

-(k, /

\

(~2.168)

Hence, the optical flow 5: = limAt--~o(X'-- x ) / A t has the form ic =

Wx

-

(k, W x ) x .

(12.169)

12.8.

Analysis of Planar Surface Optical Flow

403

We call W the flow matrix of planar surface motion. Eq. (12.169) implies that the flow 5~ remains the same if the flow matrix W is replaced W + cI for an arbitrary constant c. This indeterminacy originates from the fact that the image motion given by the first of eqs. (12.166) is unchanged if the matrix A is replaced by cA for an arbitrary constant c. In order to remove this indeterminacy, we hereafter impose the following normalization: t r W = 0. (12.170) If the camera motion is infinitesimal, the motion parameters {h, R} have the following form (see eqs. (12.30)): R - I + w x I A t + O(At2),

h - v a t + O(At2).

(12.171)

Substituting these into the second of eqs. (12.166), we obtain x I A t + O(At2)) ( - d l +

A - (I-w =

+

§

•

vnn-At+O(At2))

(12.172)

§

Comparing this with eq. (12.167) and noting the normalization (12.170), we see that the flow matrix W has the following form:

W - - - j 1 ( vn 7- - 3 l ( v , n ) l ) - w

(12.173)

x I.

Alternatively, eq. (12.169) can be obtained from the general optical flow equation (12.28) by noting that if space point r - Z ( x ) x is on space plane (n, r) - d, the depth is given by Z(x)-

d (n,x)"

(12.174)

Substituting this into eq. (12.28) and noting that Q x x - O, we obtain

x

d = QxWx-

12.8.2

Wx-

+ ~'xx

) - - Q x ( vnd

(k, Wx)x.

+wxl)

x

(12.17 )

Estimation of the flow matrix

In the presence of noise, the observed flow ~ does not satisfy eq. (12.169) exactly. As in the case of finite motion analysis, the surface and motion parameters {n, d} and {v, w} can be estimated by solving the following two subproblems:

404

Chapter 12.

Problem

3-D Interpretation of Optical Flow

12.5 Estimate a matrix W of trace 0 such that -

Wx

-

(12.176)

(k, Wx)x

from the observed flow 5. 12.6 Decompose the matrix W into surface and motion parameters {n, d} and {v, w} in such a way that

Problem

W -

1( v,-, T - 5(,,, 1 n)I) - w•

- -~

(~2.177)

Problem 12.5 can be optimally solved by minimizing the integral of the squared Mahalanobis distance, i.e., f

J[W]- I

Js

I1~- w=

+ (k ,

w=)=ll =v 0 [ x ]

dxdy--+ min ,

(12.178)

under the constraint t r W - 0. The first variation of J[W] for an infinitesimal variation W ~ W + 5 W is

5J[W] - 2 ~ (-SWx + (k, 6Wx)x, V0[&] (& - Wx + (k, Wx)x))dxdy

- -2(/~

(Vo[~] (~ -

w=

+ (k, w=)=)=T

- ( x , V0[&] (& - Wx + (k, Wx)x)) kx T) dxdy; 5W).

(12.179)

This must vanish for an arbitrary variation 6 W such that t r 6 W - 0, so we obtain

g Vo[~] ( ( w =

-

(k. w = ) = ) = ~ -

- (=.

Vo[=]( w =

- (k. w = ) = ) ) k = T) ,x,~

where A is the Lagrange multiplier for the constraint t r W - 0. If a 3333tensor A = (Aijkl) and a 33-matrix B are defined by 3

m=l

+ (x, Vo[i~lx)kikk) xjxtdxdy,

B- fs (Void]==T_ (=, y0[=]=)k~T) dxd~,

(12.181) (12.182)

12.8.

Analysis of Planar Surface Optical Flow

405

eq. (12.180) can be written in the following form: (12.183)

. A W - B + AI.

It is easily seen from eq. (12.181) that .41 = O. This is a consequence of the fact that the flow matrix W can be determined only up to a constant multiple of I. It follows that the rank of tensor .4 is 8; its null space is { I } L . Multiplying the generalized inverse A - of tensor A on both sides of eq. (12.183) and noting that A - I = O , we obtain the solution W in the following form: W = A-B. (12.184) The generalized inverse A - is obtained by casting the 3333-tensor ,4 into a 99-matrix, computing its generalized inverse, and casting it back into a 3333-tensor (see Section 2.4.2).

12.8.3

Planarity test

Let I ~ be the optimal estimate given by eq. (12.184). Since V[5~] has rank 2 and the flow matrix W has eight degrees of freedom, the residual f

J[W]- /I1~- w~ + (k, W~)~ll=V0[x]dxdy Js

(12.185)

is a ~2 variable with 2 ( S - 4) degrees of freedom if the noise is Gaussian, where S is the number of the pixels at which the optical flow is defined. This implies that optical flow must be defined at least at four pixels, as expected from the finite motion analysis for a planar surface (see Section 11.6.1). If the covariance matrix V[5~] is decomposed into the noise level e and the normalized covariance matrix V015~] in the form of eq. (12.74), an unbiased estimator of e2 is obtained in the form

~ _ z0[w] -

-

2(S -

4)'

(12.186)

where J0[W] is the normalized residual obtained by replacing V[&] by V0[&] in the expression for J[l/~r]. The expectation and variance of ~2 are given by ~4

E[~2] _

e2,

Vie2] -

S-

4"

(12.187)

Since S is usually very large, ~2 is a very good estimate of e 2. This analysis can be used to test if the observed flow is due to a planar surface motion. Namely, the hypothesis that the object surface is planar is rejected with significance level a% if 6

J[W] >

~2(S_4),a.

(12.188)

2 6Since S is usually very large, we can use the approximation X2(s_4), a ,~ (Na +

v/4S - 17)2/2 ~ (Na + 2v/~)2/2; see Footnote 15 in Section 3.3.4.

406

Chapter 12.

3-D Interpretation of Optical Flow

In terms of the estimate ~2 computed by eq. (12.186), this planarity test takes the following form: ~2 ~22(S_4),a -~ e > 2 ( S - 4)" (12.189) This has the same interpretation as the planarity test for finite motion (see eq. (11.152)). Namely, we compare the estimate ~ computed under the hypothesis that the object surface is planar with the noise level e expected from the accuracy of the image processing operations for detecting optical flow. As in the case of finite motion analysis, the planar surface algorithm can be applied to general optical flow as well. Hence, it is safer to apply the planar surface algorithm than the general optical flow algorithm; it should be switched to the general optical flow algorithm only when assuming planarity is not compatible with the observed optical flow to a convincing degree. The opposite approach has often been adopted in practice: the general optical flow algorithm is used first and abandoned, say, when the second smallest eigenvalue of the moment tensor A// given by eq. (12.77) is sufficiently close to its smallest eigenvalue, and the decision criterion is set arbitrarily. In contrast, the use of the planarity test as described above has a solid statistical foundation. In Chapter 13, we present an alternative criterion which does not involve the significance level.

2.s.4

Computation of surface and motion parameters

We now solve Problem 12.6. Unlike the decomposition of the flow matrix F for general optical flow, no decomposability condition needs to be imposed. In other words, there always exist {n, d} and {v, w} that satisfy eq. (12.176) exactly, although the decomposition may not be unique. This is easily understood by counting the degrees of freedom. The flow matrix W has eight degrees of freedom since its trace is 0. The surface parameters {n, d} have three degrees of freedom; the motion parameters {v, w} have five degrees of freedom. Thus, the matrix W has the same degrees of freedom as the surface and motion parameters. The surface and motion parameters {n, d} and {v, w} are computed by the following procedure: 1. Compute w = -type3[A[W]].

(12.190)

2. If S[W] = O, then v = 0 and w = w. The surface parameters {n, d} are indeterminate. 3. Else, let 0-1 >_ 02 _~ 0"3 be the eigenvalues of S[W], and {Ul, u2, u3 } the orthonormal system of the corresponding eigenvectors.

12.8.

Analysis of Planar Surface Optical Flow

407

d

Fig. 12.17. Two solutions for the same image transformation. 4. The surface parameters are given by -

VO'I - -

0" 2 U 1 -~-

~t

V0-1

V0"2 - -

1

O"3"U3

d -

- - 0-3

'

~

(12.191)

.

0-1 - - 0"3

5. The motion parameters are given by +Vo,

v - -

-

o-

u, -

x/al-aa

Vo-

-

o3

3

n ,

~

-

w

x v

2d

'

(12 192) "

where the double sign =i= corresponds to that in eqs. (12.191). 6. For each solution, another solution is obtained by changing the signs of n and v simultaneously. Thus, four solutions are obtained. As in the case of finite motion, this ambiguity is partly due to the fact that the interpretation is solely based on the flow matrix W , not the flow itself, and the fact that the scene behind the camera can be seen (Fig. 11.13). Suppose one solution {n, d} and {v, w} is obtained. If we 1. move the camera to the other side of the planar surface, and 2. reverse the translation, the new surface and motion parameters are { - n , d} and { - v , w }, respectively (Fig. 12.17). We can see from eq. (12.177) that the flow matrix W does not change. As in the case of finite motion, however, this ambiguity can be reduced if we also consider which part of the surface we are observing. In fact, if the condition that the visible part of the surface has positive depth is imposed, the number of solution reduces to at most t w o - - i n most cases one. This is because spurious interpretations are in most cases such that the surface is all behind the camera or some of part of it is behind the camera. In the latter

408

Chapter 12.

3-D Interpretation of Optical Flow

Fig. 12.18. The vanishing line cannot pass through the visible part of the planar surface; only one side of it is visible. case, the vanishing line passes through the visible part, which is impossible because only one side of it is visible (Fig. 12.18; see Section 4.6.2). From eq. (12.174), the condition to be imposed is d (n, x) > 0.

(12.193)

The 3-D position r is given by dx

r =

(n, x~.)

(12.194)

E x a m p l e 12.5 Fig. 12.19a shows a simulated optical flow image (512 x 512 pixels) of a planar grid in the scene viewed from a moving camera. The focal length is assumed to be f = 600 (pixels). We added Gaussian noise of mean 0 and standard deviation ~ (pixels) to the x and y component of the flow at each point independently (the noise level e = ~ / f is not used in the reconstruction computation). We assume that the correct solution can be distinguished from spurious solutions. Fig. 12.19b shows the reconstructed grid for ~ - 1, 2, 3. The true position is superimposed in dashed lines. We can see that the reconstruction error increases as the noise increases.

12.9

12.9.1

Camera

Rotation

and

Information

Rotation estimation

So far, we have assumed that v r 0. If v = 0, no 3-D information can be obtained from the optical flow. Hence, we must test if v - 0, i.e., if the camera motion is a pure rotation, before attempting 3-D reconstruction. In order to do this rotation test, we first hypothesize that v - 0, i.e., the camera motion is a pure rotation around the center of the lens. If no image noise exists, eq. (12.28) implies that the flow should have the form 5~ = Q x ( x x w).

12.9.

Camera Rotation and Information

409

(~)

(b)

Fig. 12.19. (a) Optical flow of a planar grid in the scene. (b) 3-D reconstruction f o r a = 1,2,3. Since the observed flow may be perturbed by image noise, we consider the following problem. P r o b l e m 12.7 Estimate a vector w such that

= Q x ( x x w)

(12.195)

from the observed flow &. This problem can be solved in the same way as Problem 12.6: we minimizing the integral of the squared Mahalanobis distance, i.e.,

J[~]

-

f~ II~ - Q~(~ • ~)1 Iy[x]dxdy ~

--+ min.

(12.196)

Noting the identities

II&- Qx(x x

o,)11 ~-Exl

- IJxll.lxj2(~, v i i i ~

-

Q~(~ • ~ ) ) + IIQ~(~ • ~)11:v[x], (12.197)

(x, v[x] - Q~(~ x ~)) - (Q~T v [ x ] - x , ~ x ~ ) -

- (~ • Q ~ V [ ~ ] - ~ , ~ ) , (12.198)

IIQ~(~ x ~ ) l l ~ [ x l - (Q~(~ x ~), v [ x ] - Q ~ ( ~ x = (~ • ~, Q ~ V [ x ] - Q ~ ( ~ • ~ ) ) -

(~, (~

•

~))

Q~r[~]-Q~ •

~)~).

(12.~9v)

we can rewrite the function J[w] in the form

J[~] = c -

2(b, ~ ) + (~, A~),

where

A-

fs x/Q~V[Sc]-Qx

x xdxdy,

(~2.2oo)

410

Chapter 12.

b = - jfs x x QTx V[Se,]-~dxdy, Differentiating

J[w]

3-D Interpretation of Optical Flow

C = ~ ll~ll2[fr,]dxdy.

(12.201)

with respect to w and setting the result 0, we obtain

Aw = b.

(12.202)

The matrix A is nonsingular in general 7, so eq. (12.202) gives a unique solution.

12.9.2

Rotation test

Let & be the resulting estimate. Since V[~] generally has rank 2 and the vector w has three degrees of freedom, the residual J[&]-/s

Ilde'-Qx(x x &)ll2v[(r']dxdy - ffs II~CllY[X]dxdy-(d"'A&)"

(12.203)

is a X2 variable with 2S degrees of freedom. If the covariance matrix V[~] is decomposed into the noise level e and the normalized covariance matrix V0[x] in the form V[&] = e2VoI~], (12.204) an unbiased estimator of e2 is obtained in the form ~2_

J0[w] 2S-3'

(12.205)

where Jo[&] is the normalized residual obtained by replacing V[~] by Vo[x] in the expression for J[&]. The expectation and variance of ~2 are given by 2e 4

2S---:-5"

( 2.206)

Since S is usually very large, ~ is a very good estimate of e. The above analysis is based on the hypothesis that v - 0, i.e. the camera motion is a pure rotation. It follows that the rotation hypothesis can be tested by the standard X2 test: the hypothesis v - 0 is rejected with significance level a% if J[&] > X2s_3,~. (12.207) In terms of the estimate ~2 computed by eq. (12.205), this rotation test takes the following form: ~2

X2S_3

-~ e > 2S - 3"

(12.208)

This has the same interpretation as the planarity test (see eq. (12.189)). Namely, we compare the estimate ~ computed under the hypothesis v -- 0 with the noise level e expected from the accuracy of the image processing operations for detecting optical flow. 7The matrix A is singular if the rank of V[~:] is less than 2 or the flow is defined only along a special type of curve on the image plane. We ignore such pathological cases

12.9.

12.9.3

Camera Rotation and Information

411

Information in optical flow

We can define the information of optical flow as in the case of finite motion (see Section 11.7.2). Let d~ be the rotation velocity estimated by the method described in Section 12.9.1. Define x~ = Q x ( x

x d~).

(12.209)

This is the rotational flow that best "mimics" the observed flow ~b; it contains no information about the 3-D structure of the scene. Letting ~bi - ~ b - ~bu, we can write - xi + xu.

(12.210)

This equation can be interpreted as decomposing the observed flow into the part that contains information about the 3-D scene and the part that contains no information. Let us call 5~i the informative part and x~ the uninformative part. In general, the accuracy of 3-D interpretation decreases as the image noise increases. As in the case of finite motion, however, the negative effect of image noise cannot be measured simply by its magnitude but it must be

compared with the magnitude of the informative part of the optical flow. Let us call the normalized residual J0[d~] the information in the optical flow. We observe the following: 9 The information is minimum if v = 0; it generally increases as Ilvll increases. 9 If v is the same, the information increases as the variation of the depth becomes larger. 9 The information is unchanged if an arbitrary rotational velocity w is added to the camera motion. 9 The information increases as the number pixels at which optical flow is defined increases. Thus, 3-D interpretation can be disrupted by image noise of a very small magnitude if the information is very small, while 3-D interpretation is robust to image noise if the information is sufficiently large. If a rotational velocity is added to the camera motion, the flow magnitude increases, thereby decreasing the relative noise magnitude. However, such an apparent decrease of the relative noise magnitude does not increase the accuracy of 3-D interpretation, because camera rotation does not add any 3-D information to the optical flow. Thus, the relative noise magnitude is a very deceptive measure of the accuracy of 3-D interpretation. This viewpoint is further extended in a more general framework in Chapter 13.

412

Chapter 12.

3-D Interpretation of Optical Flow

(~)

(b)

Fig. 12.20. (a) Optical flow of a scene viewed from a translating camera. Optical flow of the same scene viewed from a rotating camera.

(b)

E x a m p l e 12.6 Fig. 12.20a shows an optical flow of a scene with a small depth variance viewed from a translating camera. It is widely known t h a t 3-D interpretation of this type of optical flow is very difficult in the presence of noise, because a similar optical flow would be obtained by a camera rotating around the center of the lens (Fig. 12.20b), in which case 3-D interpretation is theoretically impossible. For flows such as shown in Fig. 12.20a, accurate 3-D interpretation is possible only if the image noise is very small. In other words, the image noise must be small as compared with the information in the

~OW. E x a m p l e 12.7 Fig. 12.21 shows the informative part and the uninformative part of the optical flow given in Fig. 12.19. The informative part of this flow has a sufficiently large magnitude, so we can expect a stable 3-D interpretation.

12.9.4

Midpoint flow approximation

In applying the theory described in this chapter to an optical flow detected from real images, we must note the discrepancy between the theoretical optical flow and the real optical flow. Theoretically, optical flow is defined as an instantaneous velocity: it is defined by differentiation of a continuous image motion. On the other hand, what can be detected by image processing operations is the inter-frame displacement of each pixel. These two definitions differ even if optical flow is exactly detected with subpixel accuracy. This difference becomes apparent when the trajectory of an image point is not straight. In fact, the theoretical optical flow is tangent to the trajectory, while the interframe displacement is a chord, connecting two points on it (Fig. 12.22a). Suppose an image point x moves to x I in the next frame. A simple and effective way of avoiding the above discrepancy is to identify the

12.9.

Camera Rotation and Information

413

I. I I I I I /

I IIllll 9

Ill

/

/

i

i

\

\

\

\

~

,

'

,

I

,

I

/

~

~

" '

/,1~///\ \ \ , ,

"

\\\\\\ \ \ \ \ ,, \\\\\\\\\\\

~,

\\\\\\\\~,,

/

~

I I I , ,

(~)

"

(b)

Fig. 12.21. (a) Informative part of the flow shown in Fig. 12.19a. (b) Uninformative part. displacement x ' - x with the flow at the midpoint (x the midpoint flow approximation (Fig. 12.22a).

+ x ' ) / 2 . Let us call this

E x a m p l e 12.8 Fig. 12.23 shows superimposition of simulated consecutive images of a planar grid in the scene viewed from a moving camera (no image noise is added). The image frame has 512 x 512 pixels, and the focal length is assumed to be f = 600 (pixels). Fig. 12.24a shows the 3-D shape reconstructed by identifying the displacement field with an optical flow. The true shape is superimposed in dashed lines. Fig. 12.24b shows the corresponding result obtained by the midpoint flow approximation. We can see that the midpoint flow approximation dramatically increases the accuracy of 3-D reconstruction.

414

Chapter 12.

3-D Interpretation of Optical Flow X r --X X r

(b)

(~)

Fig. 12.22. (a) The difference between the theoretical optical flow and the real image optical flow. (b) Midpoint flow approximation.

Fig. 12.23. Two consecutive image frames superimposed.

i

(~)

\

(b)

Fig. 12.24. 3-D reconstruction from Fig. 12.23: (a) direct optical flow approximation; (b) midpoint flow approximation.

C h a p t e r 13 I n f o r m a t i o n C r i t e r i o n for M o d e l Selection In order to apply geometric correction and parametric fitting presented in Chapters 5 and 7, we first need to know the geometric model, i.e., the constraints and hypotheses that should hold in the absence of noise. But how can we prefer one geometric model to another? In this chapter, the AIC introduced in Section 3.7 is modified so that it can be used for selecting a plausible geometric model, which is identified with a manifold. We show that the complexity of the model is evaluated by not only its degrees of freedom but also such invariant quantities as the dimension and codimension of the manifold. We also present a procedure for evaluating relative goodness of one model to another without using any arbitrarily set threshold such as the significance level of the X2 test. This comparison criterion is applied to point data in two and three dimensions, 3-D reconstruction by stereo vision, 3-D motion analysis, and 3-D interpretation of optical flow.

13.1 13.1.1

Model Selection Criterion Model estimation

In the preceding chapters, we have formulated m a n y types of statistical test, all of which have the form of comparing the a posteriori noise level ~ estimated under a hypothesis with the a priori value e expected from the accuracy of the d a t a acquisition process (image processing, range sensing, etc.). As has often been pointed out, however, it is very difficult to predict the noise level e a priori in real situations. It can be estimated a posteriori only if the hypothesis is true. Then, is it not possible to test the hypothesis without using the a priori noise level e? In this chapter, we show t h a t although this is impossible in general, we can compare two hypotheses for the same d a t a and determine which hypothesis is more plausible. In order to show this, we must first generalize the problem in abstract terms. Let al, . . . , a N be m-vector d a t a sampled from an m'-dimensional manifold ,4 C 7~m, which we call the data space. We write

(13.1) where ~ is the true position of d a t u m a s . The noise t e r m A a ~ is assumed to be an independent Gaussian r a n d o m variable of mean 0 and covariance matrix V[a~]. We assume t h a t the covariance m a t r i x Vial] has range Ta~ (A) ( 415

416

Chapter 13.

Information Criterion for Model Selection

9 A

aa

.

i

!

9

O

o

Fig. 13.1. Estimating a manifold $ C jt and ~

E $, a -- 1, ..., N, from the data

the tangent space to A at ~ ) and hence has rank m ~. We want to find a manifold $ C A such that the true values ~ are all in it. This problem is stated as follows (Fig. 13.1): P r o b l e m 13.1 Estimate a manifold S C A and m-vectors { ~ } ~ E $, a -- 1, ..., N , f r o m the data { a ~ } .

such that

In order to solve this problem, we parameterize the manifold S by an n-vector u constrained to be in ml d - d i m e n s i o n a l manifold L/ C 74 ~, which we call the parameter space. It follows t h a t the manifold $ has n ~ degrees of freedom, and Problem 13.1 reduces to parametric fitting: we want to compute an optimal value of the p a r a m e t e r u E b/. If $ has 0 degrees of freedom, it is a fixed manifold, and Problem 13.1 can be identified with geometric correction" we want to correct each a~ into &~ E $ in an optimal manner. Thus, Problem 13.1 generalizes both geometric correction and parametric fitting. We call a parameterized manifold a (geometric) model. If it is a ddimensional manifold in an m~-dimensional d a t a space and has d free parameters, we say t h a t the model has dimension d, codimension r - m I - d, and d degrees of/reedom. If the model is specified by L equations in the form F ( k ) ( a , u ) - O,

k - 1, .... , L ,

(13.2)

the co dimension r of the manifold was called the rank of the "hypothesis" (or "constraint" if the parameter u does not exist) in the preceding chapters (see Sections 5.1.1 and 7.1.1). E x a m p l e 13.1 Suppose N image points Xl, ..., xN are observed. The d a t a space is the entire image plane X - {x E 7~3I(k, x) - 1}. Let ~ be the true position of x~. We write x~ - 2~ + A x ~ and regard the noise t e r m A x a as an independent Gaussian random variable of mean 0 and covariance m a t r i x V[x~], which we assume has range X and rank 2. The following are typical models for the image point d a t a {x~}" 1. Image point model: the true positions 5~, a - 1, ..., N , all coincide (Fig. 13.2a). This model is an image point p - {3}; it has dimension

13.1.

Model Selection Criterion

X

417

X

X X a - "~ a @

.................... o / / X ' " ' " .... "".o

O

(~)

(b)

(c)

Fig. 13.2. (a) Image point model p. (b) Image line model I. (c) Image plane model X. 0, codimension 2, and two degrees of freedom (an image point has two degrees of freedom; see Section 4.1.1). If p is constrained to be at the image origin, the constrained model p~ - {k} has zero degrees of freedom.

2. Image line model: the true positions 5~, c~ = 1, ..., N , are all collinear (Fig. 13.2b). This model is an image line 1 = {(n,m) = 0}; it has dimension 1, co dimension 1, and two degrees of freedom (an image line has two degrees of freedom; see Section 4.1.2). If 1 is constrained to pass through the image origin, the constrained model l' = {(n, m) = 0 - - (n, k) = 0} has one degree of freedom.

3. Image plane model" no constraint is imposed on the true positions 5~, - 1, ..., N (Fig. 13.2c). This model is the entire image plane A'; it has dimension 2, codimension 0, and zero degrees of freedom.

E x a m p l e 13.2 Suppose N space points ~'1, . . . , ~ ' N are observed. The data space is 7~3 itself. Let ~ be the true position of r~. We write r~ = ~ + A r ~ and regard the noise term A r ~ as an independent Gaussian random variable of mean 0 and covariance matrix V[r~], which we assume has range 7~3 and rank 3. The following are typical models for the space point data {r~}:

1. Space point model: the true positions ~ ,

a = 1, ..., N, all coincide (Fig. 13.3a). This model is a space point g = {e}; it has dimension 0, codimension 3, and three degrees of freedom (a space point has three degrees of freedom; see Section 4.2.1). If P is constrained to be at the coordinate origin, the constrained model p / _ {0} has zero degrees of freedom.

2. Space line model: the true positions ~ , a - 1, ..., N, are all collinear (Fig. 13.3b). This model is a space line L - {r x p - n}; it has dimension 1, codimension 2, and four degrees of freedom (a space line

418

Chapter 13.

Information Criterion for Model Selection

Z

r(~

Z

e

r a

Y

X I

.,

x

(~)

(b)

Z r a ~ i" a

roe

y

(c)

(d)

Fig. 13.3. (a) Space point model P. (b) Space line model L. (c) Space plane model II. (c) Space model 7~3. has four degrees of freedom; see Section 4.2.2). If L is constrained to pass through the coordinate origin, the constrained model L ~ - {r • p = O} has two degrees of freedom.

3. Space plane model: the true positions ~ , a - 1, ..., N, are all coplanar (Fig. 13.3c). This model is a space plane II - {(n, r ) - d}; it has dimension 2, codimension 1, and three degrees of freedom (a space plane has three degrees of freedom; see Section 4.3.1). If II is constrained to pass through the coordinate origin, the constrained model II' - {(n, r) = 0}; has two degree of freedom.

4. Space model: no constraint is imposed on the true positions ~ , c~ - 1, ..., N (Fig. 13.3d). This model is the entire space 7~3; it has dimension 3, codimension 0, and zero degrees of freedom.

13.1.2

Minimization of expected residual

Given a particular model $, we can compute an optimal solution as described in Chapters 5 and 7. Namely, we can obtain the m a x i m u m likelihood solution

13.1.

Model Selection Criterion

419

of Problem 13.1 by minimizing the sum of the squared Mahalanobis distances N

J - ~-~(a~ - gs, V [ a s l - ( a s - g s ) )

(13.3)

c~=l

under the constraint that a s E $, c~ - 1, ..., N (see eq. (5.6), (7.5), and (7.12)). Let {hs } and S be the resulting maximum likelihood estimators of {as} and S, respectively. Substituting them back into the function J, we write the residual in the following form: N

- Z

- a II E 1.

(13.4)

s--1

This quantity measures the minimum discrepancy between the assumed model $ and the data {as }, so this appears to be a good measure of the goodness of the model. However, because {~is } and S are determined so as to minimize the residual for the current data {as}, the residual can be made arbitrarily small. In fact, if the manifold $ has a sufficient number of parameters, we can make S pass through all the data {as }. Such an artificial model may explain the current data but may be unable to predict occurrence of the data to be observed in the future. In order to measure the predicting capacity of the model, we adopt the criterion introduced in Section 3.7.1. Namely, we consider future data { a ; } that have the same distribution as the current data {as } and require that the residual for the future data N

J[{as},* {ds},S] - E

lids* - &s 112-,,vra~l~__J

(13.5)

be small. Since this is a random variable, we take expectation to define a definitive value for the model" I ( $ ) - E*[E[J[{a~}, {&s},S]]].

(13.6)

Here, E*[. ] and E[. ] denote expectation with respect to the future data {a~ } and the current data {a~}, respectively. We call I ( $ ) the expected residual of model 8 and regard $ as good if I ( $ ) is small. E x a m p l e 13.3 Consider the three models described in Example 13.1. For whatever data {x~ }, the residual of the image line model 1 is always no more than the image point model p, and the residual of the image plane model X is always 0 since the maximum likelihood estimator 5~ of the true value 5~ is the datum x~ itself. The same can be said for Example 13.2: for whatever data {r~}, the residual of the space line model L is always no more than

420

C h a p t e r 13.

I n f o r m a t i o n Criterion for Model Selection

Fig. 13.4. The Mahalanobis projection of a~ onto manifold S is the tangent point of the equilikelihood surface to S. t h e space point model P; the residual of the space plane model H is always no m o r e t h a n the space line m o d e l L; the residual of the space m o d e l g 3 is always 0. Thus, the residual is not a g o o d m e a s u r e of the goodness of the model.

13.2

Mahalanobis Geometry Mahalanobis projection

13.2.1

First, we consider the case in which the model $ has no free p a r a m e t e r s (n' = 0). As discussed in Section 5.1.1, if the t r u e value 6.~ of d a t u m a~ is in S, the m a x i m u m likelihood e s t i m a t o r &~ of 6.~ is the point in S in the shortest distance from a~ m e a s u r e d in the Mahalanobis distance with respect to the covariance m a t r i x V[a~]. Geometrically, &~ is the tangent point of the equilikelihood surface ( a - a~, V [ a ~ ] - ( a - a~)) - constant, a E ,4, to t h e manifold $ (Fig. 13.4). Let us call &~ the Mahalanobis projection of a~ onto $ with respect to V[a~]. In t h e following, the proviso "with respect to V[a~]" is o m i t t e d as u n d e r s t o o d . W e a s s u m e t h a t the noise is sufficiently small, by which we m e a n t h a t V[a~] - O(e2), a - 1, ..., N , for an a p p r o p r i a t e l y defined noise level e, as c o m p a r e d with which the d a t a space ,4 and the manifold $ are b o t h ass u m e d to be sufficiently s m o o t h 1. T h e following p r o p o s i t i o n is easily o b t a i n e d (Fig. 13.5)"

Proposition Ila

-

13.1

The following equality holds to a first approximation:

l12[a ]

-

Ila

-

21VIal] +

-

Iv[a ]2 ,

(13.7)

1A precise statement is that the radius of curvature, which is defined as the reciprocal of the total curvature, of the manifold is sufficiently small. This is the fundamental assumption for defining a (local) Gaussian distribution over a manifold (see Section 3.2.2). This assumption also played a fundamental role in Section 3.5.2.

13.2.

Mahalanobis Geometry

421

aa #

Fig. 13.5. The relation between datum as, its true value fi~ E $, and the Mahalanobis projection g~ of as onto S.

Proof. A point ac~ ^' on t h e line t h a t connects 6,~ and ac~ can be expressed in the form a"~ - a s + t ( ~ - ao,) (13.8) for a real n u m b e r t. If t is small, point a^ s is on 3 to a first a p p r o x i m a t i o n . Hence,

II'~ 0

(14.73)

for all as E A. Furthermore, the integration operation f A das with respect to a~ for any expression of ps(as; ds) is, if the integration exists, interchangeable with the differentiation operation Va, with respect to as. The probability density p~(as; a~) is not defined for ds ~ .A. For the convenience of analysis, however, we extend it to ds ~ ,4 in the same way as in the case of geometric correction. Namely, we assume that

ps(as; ds + A d s ) = ps(as; ds) + O ( A a , ) 2

(14.74)

for all ds E A and A d s E Tao(A) • where T a , ( A ) C T~m is the tangent space to the manifold jt at 5,s. In precise terms, we assume the following: Assumption

14.10

Vamp, E Ta, (A). 1~.~.2

(14.75)

The rank o f the hypothesis

The L equations (14.71) may be algebraically dependent. If only r of them are independent, they generally give only r effective constraints and define an (m ~ - r)-dimensional manifold 8, or a manifold S of codimension r, in the data space jt except at singular points. We assume that all {as } are not singular points and call r the rank of the hypothesis (14.71). A precise statement of this assumption, which also gives a precise definition of the rank r, is as follows: 14.11 The L equations of the hypothesis define a manifold $ of codimension r in the data space A in the neighborhood of as, a = 1, ..., N, for u = ~t. Assumption

14.4.

General Parametric Fitting

469

As in the preceding chapter, let us call the manifold $ the (geometric) model of Problem 14.2. If we write a s = a s + A a ~ , we have

/

§

t

9

__

Va/~(1)...,

VaF(~L)

(

)

Aa~

(14.76)

to a first approximation in A a a , where and hereafter VaF(k)(5~, ~t) is abbreviated as Va/Nk). The inherent constraint a s E ,4 requires that A a s E Tas (,4) to a first approximation. If A a s ranges over the entire tangent space T a , (A), eq. (14.76) defines a linear mapping from Tas (,4) to a linear subspace

z:a

-

{

9

I/Xa

e Tao (A)}

c

n

(14.77)

The dimension of this linem" subspace equals the dimension of the subspace

{ PaA VaF(1) , ..., PaAs VaF(L) } L C Ti m,

(14.78)

where p aA s is the m-dimensional projection matrix onto T a s ( A ) . Consequently, the dimension of s is at most the rank r of the hypothesis (14.71), but we further assume the following3: 14.12 The dimension of the linear subspace s the rank r of the hypothesis for all c~.

Assumption

is equal to

In m a t h e m a t i c a l terms, Assumption 14.12 states that each of the L equations of the hypothesis defines a manifold of codimension 1 in the d a t a space A such that they intersect each other transversally at 5,s, c~ = 1, ..., N. We say that a s is a singular datum if the dimension s is less than r, and an ordinary datum otherwise. Assumption 14.12 requires that singular data should be excluded from the fitting process. We say that the hypothesis (14.71) is nonsingularif Assumption 14.12 is satisfied, and singular otherwise (see Section 7.1.1). The linear s u b s p a c e / : a s defined by eq. (14.77) can alternatively be defined as the tangent space to the manifold

jz _ {

9 F(L) (a, ~t)

e 7"iLia e A} e Ti n

(14.79)

3As a result, the L equations (14.71) cannot be replaced by a single equation, say = {0} for all a.

L Ek--1 F(k)(a' u)2 = 0, which still has rank r but s

470

Chapter 14.

General Theory of Geometric Estimation

at a,m E A. However, am can be a singular point of the mapping (F(1)(a, fi), .... F (L)(a, ~))T: ,4 --+ 7~L. In other words, the linear subspace s defined by replacing gm by am ( r gin) in eq. (14.77) has a higher dimension that r. If this is the case, we say that the hypothesis is degenerate (see Section 7.1.1). We further assume that no effective constraints exist on ~t other than the inherent constraint fi E b/. Putting it differently, we assume that for an arbitrary variation A u of ~ there exists a variation A~m of am such that

F(k)(am + A6,m, ~ + Au) = 0,

k = 1,..., L,

(14.80)

for c~ - 1, ..., N. The above equation is equivalent to k-

1,...,L,

(14.81)

to a first approximation, where and hereafter VuF(k)(~m, ~) is abbreviated as VuF (k). Let Tu(/d) be the tangent space to the manifold/d at ~. The inherent constraint u E ?d requires that A u E T~t (b/) to a first approximation. Similarly, the inherent constraint ~m E A requires that A~m E Ta~ (A) to a first approximation. From the definition of the linear subspace s the above assumption is formally stated as follows: A s s u m p t i o n 14.13 For an arbitrary Au E T~t(Lt),

E s

9

1~.~.3

(14.82)

Cramer-Rao lower bound for parametric fitting

Let Im be the score of am with respect to am: lm - Va~ logpm.

(14.83)

The Fisher information matrix Jm with respect to am is defined by w

-T

gm - E[lmlm ].

(14.84)

The symbol El. ] denotes the expectation with respect to the joint probability density 4 I-I N m=l pro(am; am). Assumption 14.10 implies that lm E Ta~(A). We_ assume that the distribution pro(am; ~tm) is regular with respect to am, i.e., lm takes all orientations in Tam (A) as am ranges over the entire data space A. In precise terms, this assumption is stated as follows: 4Since each aa is assumed to be independent, the expectation E[.] of a quantity that depends on aa alone equals the expectation with respect to pa(aa;~a) alone.

14.4.

General Parametric Fitting

471

A s s u m p t i o n 14.14 The Fisher information matrix J s is positive semi-

definite and has range Tvt. (.A). Define an (LL)-matrix V~ - ( ~ ( k 0 )

by

(g~k,,)_ ((Var(k),j-~Va~(O)), (LL)-m

trix

-

(-Ws(kl) ) -

by

-

Y2,

(14.85)

or

((Va/>(k) , J-~Va />(0 )) - .

(14.86)

Define an (nn)-matrix M by N

- E s=l

L

E

u

-(k)

17Vak')(P~VuF~ ) ( P U V u )

/>(t) T ,

(14.87)

k,l=l

where p U is the n-dimensional projection matrix onto the tangent space Tu (b/) to the manifold gg at R. We c a l l / ~ the moment matrix (see eq. (r.2a)). We assume that the number of the data {as } is sufficiently large and their true values {R~} distribute "generally" in the model $. In other words, we exclude the possibility that { a s } have a special configuration (being identical, collinear, coplanar, etc.) such that the fitting parameter u is indeterminate. In precise terms, this assumption is stated as follows: A s s u m p t i o n 14.15 The moment matrix M is positive semi-definite and has

range Tu (gt). L e t / t - / t ( a l , . . . , aN) be an unbiased estimator of u determined from the data {as}. Since u is constrained to be in the parameter space U, it must satisfy/t(al,..., aN) E ~ for any a~ E A, a - 1, ..., N. Its unbiasedness is defined by P ~ E[/t - R] - 0. (14.88) As in the case of geometric correction, we assume that the distribution of /t is sufficiently localized around ~ in the parameter space b / a n d define the covariance matrix of/t in the following form: V [ / t ] - P-UuE[(it - R ) ( / t - R)q-]Pg.

(14.89)

We now prove the following theorem: T h e o r e m 14.2 V[/t] __>/~-.

(14.90)

Since Theorem 14.2 is an analogue of the Cramer-Rao inequality (3.134) in Section 3.5.2, we call eq. (14.90) and the bound it gives the Cramer-Rao inequality and the Cramer-Rao lower bound, respectively, for parametric fitting.

472

Chapter 14.

14.4.4

General Theory of Geometric Estimation

Proof of the main theorem

L e m m a 14.6 The (LL)-matrix Va is positive semi-definite and has range

Proof. Let v / - (~](k)) be an arbitrary L-vector. We have L

L

(~7, V~7) - ( E q(k)VaF(k)'J-~ E r](l)~af(l))" k=l

(14.91)

/=1

Since J ~ is positive semi-definite, its generalized inverse J ~ is also positive semi-definite. It follows that the right-hand side of eq. (14.91) is nonnegative for an arbitrary ~7, meaning that V~ is positive semi-definite. Suppose v/ belongs to the null space of V~. Then, the left-hand side of eq. (14.91) is 0. Hence, ~kL=l rl(k)VaF(~k) belongs to the null space of J ~ , which is identical with the null space of J a . Assumption 14.14 implies that the null space of J ~ coincides with Tas (A) • Hence, L

A o)

-

o

(14.92)

k--1

for an arbitrary Aa~ E Ta s (A). From the definition of the subspace /:as, this means that v/ E /:_x as" Conversely, if v/ E /:--P as, we obtain (r/, V~v/) - 0 by following eqs. (14.92) and (14.91) backward. Hence, the null space of the (LL)-matrix V~ coincides with the ( L - r)-dimensional subspace s a s C 7~L Consequently, the range of V~ coincides w i t h / : a s . [] Consider the (LL)-matrix ISr~ defined by eq. (14.86) 9 Let p aL be the Ls dimensional projection matrix o n t o / : a s - Lemma 14.6 implies the following: C o r o l l a r y 14.3 The (LL)-matrix 17Va is positive semi-definite and has range f-.(zs, and the following relationship holds: V~I?r

- p_C as

(14.93)

9

The following lemma plays an essential role in proving Theorem 14.2. L e m m a 14.7 The variation Ads of ~

defined by

L

(14.94) k,l=l

satisfies Aa E Tcts(.A) for any A u E T~t(bl), and to a first approximation F(k)(d~ + Aa~, ~ + A u ) - 0,

k-

1,..., L.

(14.95)

14.4.

General Parametric Fitting

473

Proof. Assumption 14.10 implies le E T a : (A). Hence, we have p_u a~ J e - J e from the definition (14.84) of the Fisher information matrix J e . Consequently, p_U a , ~ j-~ _ j-~ ~ and eq. (14.94)implies PU a ~ Age = Age ~ i.e. ~ Age E Ta~ (A) 9 Eq. (14.95) is proved by showing (Vu/~(k) , Age) - -(Va/~(k) , Au).

(14.96)

From eqs. (14.93) and (14.94), we see that

(VaF(k)' A g e ) - - - E

(~TaF(k) , J-~Va~'(rn)) - (ml)W~ (Vu~(1) , Au) /=1

m=l

L

= - E ( P a ~ )kt(Vu/~(/) , Au),

(14.97)

/=1

where (paC)kt is the (kl) element of the projection matrix p_Cas. Assumption 14.13 implies that the L-vector ((Vu/~a(k), Au)) belongs to the s u b s p a c e / : a . Hence, the last term of eq. (14.97) is equal to -(Vaf'(k),Au).

[]

Define an n-vector random variable m e by L

me-

W~-